2024-03-06T07:10:25.930 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:10:25.930 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi --owner scheduled_leonidus@teuthology --verbose -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/orig.config.yaml 2024-03-06T07:10:25.974 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:10:26.051 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583058' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJ3vhpohM9xB5ux8f7iuiyN+Bf6kEPwaNo6LDrS7aDwVfwCdJ+zvK44YvoLEqEN6iG62xG3g1AQhcguxwafKWc= smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVpYiu/YoCYgaV/eQmFohRMqb3IJ5fc5iJmrQsU0co4UgQoF2P0s0/28GIdJ3ak/3iXpY9jZANDyCOyCMhbsro= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:10:26.051 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:10:26.052 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:10:26.052 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:10:26.053 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:10:26.077 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:10:26.077 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:10:26.077 INFO:teuthology.packaging:ref: None 2024-03-06T07:10:26.077 INFO:teuthology.packaging:tag: None 2024-03-06T07:10:26.077 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:10:26.077 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:10:26.077 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:10:26.231 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:10:26.232 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:10:26.238 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:10:26.238 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:10:26.254 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:10:26.269 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:10:26.274 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:10:26.299 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:39.801121', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJ3vhpohM9xB5ux8f7iuiyN+Bf6kEPwaNo6LDrS7aDwVfwCdJ+zvK44YvoLEqEN6iG62xG3g1AQhcguxwafKWc='} 2024-03-06T07:10:26.323 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:39.802255', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVpYiu/YoCYgaV/eQmFohRMqb3IJ5fc5iJmrQsU0co4UgQoF2P0s0/28GIdJ3ak/3iXpY9jZANDyCOyCMhbsro='} 2024-03-06T07:10:26.323 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:10:26.329 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:10:26.329 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:10:26.329 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:10:26.393 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f80aa602f70>, signals=[15]) 2024-03-06T07:10:26.394 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:10:26.399 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:10:26.399 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:10:26.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:26.513 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:10:26.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:26.601 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:10:26.607 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-03-06T07:10:26.625 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-03-06T07:10:26.625 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-03-06T07:10:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-03-06T07:10:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-03-06T07:10:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-03-06T07:10:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-03-06T07:10:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:26.682 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:26.683 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:26.683 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:26.683 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:26.683 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:26.683 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:26.683 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-03-06T07:10:26.709 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-03-06T07:10:26.725 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-03-06T07:10:26.726 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:26.782 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:26.783 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:26.783 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-03-06T07:10:26.800 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:10:26.809 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:10:26.814 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:10:26.814 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:26.816 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:26.838 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:10:26.844 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:10:26.845 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:26.874 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:26.903 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:10:26.953 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:10:26.969 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-06T07:10:26.969 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-06T07:10:26.969 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:26.970 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:26.987 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-03-06T07:10:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:26.988 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-03-06T07:10:27.043 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:27.043 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-361.el9.x86_64 2024-03-06T07:10:27.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2024-03-06T07:10:27.045 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:27.045 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-361.el9.x86_64 2024-03-06T07:10:27.046 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-03-06T07:10:29.093 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-06T07:10:29.174 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-06T07:10:30.616 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-06T07:10:30.769 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-06T07:10:33.017 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-03-06T07:10:33.923 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-03-06T07:10:35.774 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-06T07:10:42.150 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.1 MB 00:02 2024-03-06T07:10:43.584 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 15 kB 00:00 2024-03-06T07:10:46.169 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-03-06T07:10:51.253 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-06T07:10:53.551 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:53.551 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:53.650 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-06T07:10:53.655 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-06T07:10:53.655 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-03-06T07:10:53.656 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:10:53.657 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 100 M 2024-03-06T07:10:53.657 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 123 M 2024-03-06T07:10:53.657 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-06T07:10:54.057 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 963 kB/s | 19 MB 00:19 2024-03-06T07:10:54.904 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 5.0 MB/s | 6.0 MB 00:01 2024-03-06T07:11:00.649 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-03-06T07:11:02.200 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-03-06T07:11:21.886 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-03-06T07:11:22.710 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 736 kB/s | 21 MB 00:29 2024-03-06T07:11:26.740 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-06T07:11:28.989 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:11:28.989 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:11:29.091 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-06T07:11:29.102 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-06T07:11:29.102 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-03-06T07:11:29.102 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-06T07:11:29.102 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-06T07:11:29.102 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:11:29.102 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-06T07:11:29.102 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:11:29.103 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:11:29.103 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:11:29.103 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:11:29.103 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-06T07:11:29.103 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-06T07:11:29.103 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2024-03-06T07:11:29.103 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:11:29.104 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 100 M 2024-03-06T07:11:29.104 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 123 M 2024-03-06T07:11:29.104 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-06T07:11:29.600 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 13 MB/s | 6.0 MB 00:00 2024-03-06T07:11:30.400 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 17 MB/s | 21 MB 00:01 2024-03-06T07:11:31.017 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-03-06T07:11:32.984 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 9.9 MB/s | 33 MB 00:03 2024-03-06T07:11:32.984 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:11:32.984 INFO:teuthology.orchestra.run.smithi069.stdout:Total 26 MB/s | 100 MB 00:03 2024-03-06T07:11:33.868 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-06T07:11:34.226 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-06T07:11:34.226 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-06T07:11:35.889 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-06T07:11:35.890 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-06T07:11:38.801 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-06T07:11:39.455 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:39.592 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:40.341 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 863 kB/s | 39 MB 00:46 2024-03-06T07:11:42.373 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:42.552 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:46.733 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 660 kB/s | 33 MB 00:51 2024-03-06T07:11:46.733 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:11:46.733 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.9 MB/s | 100 MB 00:53 2024-03-06T07:11:47.287 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:47.502 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:47.652 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-06T07:11:48.096 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-06T07:11:48.096 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-06T07:11:49.530 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-06T07:11:49.530 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-06T07:11:52.282 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-06T07:11:53.090 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:53.217 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:56.151 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:12:01.450 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:12:01.558 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:06.685 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:21.477 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:21.477 INFO:teuthology.orchestra.run.smithi069.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:12:21.477 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:12:21.562 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:22.024 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:22.025 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:12:22.025 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:12:22.025 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:12:22.427 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:22.427 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:12:22.427 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-06T07:12:22.427 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:22.428 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:12:22.428 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:12:22.428 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:12:22.428 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:12:22.428 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-06T07:12:22.940 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:12:23.085 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:23.086 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:12:23.086 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:12:23.086 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:12:23.086 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:12:23.086 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-06T07:12:23.086 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:12:23.087 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:12:23.206 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:23.207 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:12:23.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-03-06T07:12:23.252 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:12:23.253 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:12:23.254 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:12:23.255 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:12:23.255 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:12:23.255 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:12:23.889 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-03-06T07:12:25.298 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:12:25.319 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-03-06T07:12:25.321 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-03-06T07:12:25.336 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.dSMOll0bkt 2024-03-06T07:12:25.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dSMOll0bkt 2024-03-06T07:12:25.414 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.dSMOll0bkt 2024-03-06T07:12:25.550 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.dSMOll0bkt is 6KB 2024-03-06T07:12:25.598 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.dSMOll0bkt 2024-03-06T07:12:25.612 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:12:25.677 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:12:25.677 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:12:25.677 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:12:25.677 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:12:25.679 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:12:25.825 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-03-06T07:12:37.526 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:37.526 INFO:teuthology.orchestra.run.smithi027.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:12:37.526 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:12:37.614 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:38.066 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:38.066 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:12:38.066 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:12:38.066 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:12:38.500 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:12:38.501 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:12:38.501 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-06T07:12:38.501 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:38.501 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:12:38.501 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:12:38.501 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:12:38.501 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:12:38.501 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-06T07:12:39.046 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:12:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:39.201 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:12:39.201 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:12:39.201 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:12:39.201 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:12:39.202 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-06T07:12:39.202 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:12:39.202 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:12:39.328 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:12:39.328 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:12:39.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-03-06T07:12:39.376 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-03-06T07:12:39.376 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-03-06T07:12:39.376 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-03-06T07:12:39.376 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 68.el9 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8264254 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-03-06T07:12:39.377 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:12:39.378 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-03-06T07:12:39.378 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-03-06T07:12:39.378 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:12:39.378 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:12:39.378 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:12:39.378 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:12:39.378 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-03-06T07:12:39.378 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:12:39.378 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:12:39.379 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:12:39.379 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:12:39.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:12:40.025 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-03-06T07:12:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:12:41.420 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-03-06T07:12:41.422 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-03-06T07:12:41.437 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.cDE6h9NjF6 2024-03-06T07:12:41.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cDE6h9NjF6 2024-03-06T07:12:41.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.cDE6h9NjF6 2024-03-06T07:12:41.658 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.cDE6h9NjF6 is 6KB 2024-03-06T07:12:41.707 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.cDE6h9NjF6 2024-03-06T07:12:41.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:12:41.798 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:12:41.798 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:12:41.798 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:12:41.798 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:12:41.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:12:41.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-03-06T07:12:55.840 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-06T07:12:55.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-06T07:12:55.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:11.956 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-06T07:13:11.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-06T07:13:11.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:46.557 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-06T07:13:55.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-06T07:13:55.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:55.845 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:13:58.621 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-06T07:14:04.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-06T07:14:04.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:14:06.522 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-03-06T07:14:07.033 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-06T07:14:07.033 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:14:07.033 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-03-06T07:14:07.049 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:14:07.049 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:14:07.049 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:14:07.049 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:14:08.050 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-06T07:14:08.050 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:14:08.150 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-03-06T07:14:08.199 DEBUG:teuthology.parallel:result is None 2024-03-06T07:14:10.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-06T07:14:10.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:15:10.626 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:15:25.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-06T07:15:25.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:15:26.040 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-03-06T07:15:26.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-06T07:15:26.446 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:15:26.447 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-03-06T07:15:26.465 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:15:26.465 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:15:26.465 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:15:26.465 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:15:27.466 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-06T07:15:27.467 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:15:27.575 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-03-06T07:15:27.607 DEBUG:teuthology.parallel:result is None 2024-03-06T07:15:27.608 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:15:27.615 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:15:27.616 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:15:27.618 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:15:27.638 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:15:27.690 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:15:27.755 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:15:27.755 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:15:27.758 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:15:27.796 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:15:27.803 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:15:27.804 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:15:27.822 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:15:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:15:27.872 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:15:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:15:27.886 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:15:27.888 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:15:27.898 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:15:27.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:15:27.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:15:27.956 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:15:27.967 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:15:27.968 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:15:27.998 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:15:28.014 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:15:28.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:15:28.152 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:15:28.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:15:28.283 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:15:28.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:15:28.354 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:15:28.399 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:15:28.455 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:15:28.532 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:15:28.582 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:15:28.582 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:15:28.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-03-06T07:15:28.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-03-06T07:15:28.706 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:15:28.753 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:15:29.155 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:15:29.165 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:15:29.165 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:15:29.176 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:15:29.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-03-06T07:15:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-03-06T07:15:29.255 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-03-06T07:15:29.326 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-03-06T07:15:29.329 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:15:29.329 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-03-06T07:15:29.358 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-03-06T07:15:29.359 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-03-06T07:15:29.406 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-03-06T07:15:29.406 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi069.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:15:29.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:15:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:15:29.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:15:29.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:15:29.468 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:15:29.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:15:29.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:15:29.558 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:15:29.572 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-06T07:15:29.577 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-06T07:15:29.769 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:15:29.818 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:15:29.905 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi069.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:19:48.636 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi069.front.sepia.ceph.com')] 2024-03-06T07:19:48.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-06T07:19:48.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:19:48.748 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-03-06T07:19:48.828 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-06T07:19:48.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-06T07:19:48.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:19:48.927 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-03-06T07:19:49.011 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-06T07:19:49.011 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:19:49.023 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:19:49.023 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:19:49.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:19:49.026 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:19:49.026 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:19:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:19:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:19:49.083 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:19:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:19:49.118 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-03-06T07:19:49.129 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-03-06T07:19:49.135 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-03-06T07:19:49.143 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-03-06T07:19:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:19:49.160 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:19:49.175 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:19:49.176 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:19:49.227 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:19:49.229 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:19:49.230 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-03-06T07:19:49.230 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:19:49.230 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:19:49.230 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:19:49.230 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:19:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:19:49.234 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:19:49.234 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-03-06T07:19:49.234 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:19:49.235 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:19:49.235 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:19:49.235 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:19:49.236 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:19:49.244 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:19:49.245 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:19:49.245 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:19:49.245 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:19:49.250 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:19:49.251 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:19:49.251 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:19:49.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:19:49.254 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:19:49.254 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:19:49.395 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:19:49.395 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:19:49.472 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:19:49.473 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:19:49.652 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:19:49.652 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:19:49.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:19:49.685 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:19:49.685 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:19:49.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:19:49.788 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:19:49.788 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:19:49.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:19:49.818 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:19:49.819 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:19:49.844 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:19:49.874 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-03-06T07:19:49.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-03-06T07:19:49.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:19:49.968 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:19:50.034 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2024-03-06T07:19:50.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2024-03-06T07:19:50.133 INFO:teuthology.orchestra.run.smithi027.stdout:74 files removed 2024-03-06T07:19:50.164 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:19:50.292 INFO:teuthology.orchestra.run.smithi069.stdout:74 files removed 2024-03-06T07:19:50.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:19:50.948 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 185 kB/s | 83 kB 00:00 2024-03-06T07:19:51.167 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 156 kB/s | 83 kB 00:00 2024-03-06T07:19:51.382 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-06T07:19:51.659 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-06T07:19:51.799 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-06T07:19:52.026 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-03-06T07:19:52.393 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:19:52.470 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-06T07:19:53.945 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-06T07:19:53.952 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-06T07:19:56.051 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-06T07:19:56.528 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-03-06T07:19:58.452 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-06T07:19:59.462 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-06T07:20:02.544 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-06T07:20:03.588 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-06T07:20:05.054 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-06T07:20:05.155 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 15 kB 00:00 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-03-06T07:20:06.234 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-06T07:20:10.922 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-06T07:20:11.248 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-06T07:20:13.528 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:20:13.529 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:20:13.545 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:20:13.546 INFO:teuthology.orchestra.run.smithi027.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:20:13.547 INFO:teuthology.orchestra.run.smithi027.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:20:13.548 INFO:teuthology.orchestra.run.smithi027.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:20:13.550 INFO:teuthology.orchestra.run.smithi027.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:20:13.553 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:20:13.555 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:20:13.642 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-06T07:20:13.671 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================================ 2024-03-06T07:20:13.671 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-06T07:20:13.671 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================================ 2024-03-06T07:20:13.671 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:20:13.672 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:20:13.673 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-03-06T07:20:13.674 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:20:13.675 INFO:teuthology.orchestra.run.smithi027.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:20:13.676 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:20:13.677 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:20:13.678 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:20:13.679 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:20:13.680 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:20:13.680 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:20:13.680 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:20:13.680 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:20:13.680 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:20:13.680 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:20:13.681 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:20:13.681 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:20:13.681 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:20:13.681 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:20:13.681 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:20:13.681 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:20:13.682 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:20:13.682 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:20:13.682 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:20:13.682 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:20:13.682 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:20:13.682 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:20:13.683 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:20:13.683 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:20:13.683 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:20:13.683 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:20:13.683 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:20:13.683 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:20:13.683 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:20:13.684 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:20:13.684 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:20:13.684 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:20:13.684 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:20:13.684 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:20:13.684 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:20:13.685 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:20:13.685 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:20:13.685 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:20:13.685 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:20:13.685 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:20:13.685 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:20:13.686 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:20:13.686 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:20:13.686 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:20:13.686 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:20:13.686 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:20:13.686 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:20:13.687 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:20:13.687 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:20:13.687 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:20:13.687 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:20:13.687 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:20:13.687 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:20:13.687 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:20:13.688 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:20:13.688 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:20:13.688 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:20:13.688 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:20:13.688 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:20:13.689 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:20:13.689 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:20:13.689 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:20:13.689 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:20:13.689 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:20:13.689 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:20:13.689 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:20:13.689 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:20:13.689 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:20:13.690 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:20:13.690 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:20:13.690 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:20:13.690 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:20:13.690 INFO:teuthology.orchestra.run.smithi027.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:20:13.690 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:20:13.690 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:20:13.691 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-03-06T07:20:13.691 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:20:13.691 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:20:13.691 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-06T07:20:13.691 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================================ 2024-03-06T07:20:13.691 INFO:teuthology.orchestra.run.smithi027.stdout:Install 150 Packages 2024-03-06T07:20:13.692 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 26 Packages 2024-03-06T07:20:13.692 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:20:13.693 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 224 M 2024-03-06T07:20:13.693 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-06T07:20:13.724 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:20:13.725 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:20:13.739 INFO:teuthology.orchestra.run.smithi069.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:20:13.740 INFO:teuthology.orchestra.run.smithi069.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:20:13.741 INFO:teuthology.orchestra.run.smithi069.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:20:13.742 INFO:teuthology.orchestra.run.smithi069.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:20:13.744 INFO:teuthology.orchestra.run.smithi069.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:20:13.747 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:20:13.748 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:20:13.830 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-06T07:20:13.853 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-03-06T07:20:13.853 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-03-06T07:20:13.853 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-03-06T07:20:13.854 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-06T07:20:13.854 INFO:teuthology.orchestra.run.smithi069.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:20:13.854 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:20:13.854 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:20:13.854 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:20:13.854 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:20:13.855 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:20:13.855 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:20:13.855 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:20:13.855 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:20:13.855 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:20:13.855 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:20:13.855 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:20:13.856 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:20:13.857 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:20:13.857 INFO:teuthology.orchestra.run.smithi069.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:20:13.857 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:20:13.857 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:20:13.857 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:20:13.857 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:20:13.857 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:20:13.857 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:20:13.857 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:20:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:20:13.860 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:20:13.861 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:20:13.862 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:20:13.863 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:20:13.864 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:20:13.865 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:20:13.866 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:20:13.867 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:20:13.868 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2024-03-06T07:20:13.869 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:20:13.870 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:20:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-06T07:20:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================================ 2024-03-06T07:20:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:Install 150 Packages 2024-03-06T07:20:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 26 Packages 2024-03-06T07:20:13.870 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:20:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 224 M 2024-03-06T07:20:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-06T07:20:14.182 INFO:teuthology.orchestra.run.smithi027.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-03-06T07:20:14.708 INFO:teuthology.orchestra.run.smithi069.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-03-06T07:20:14.907 INFO:teuthology.orchestra.run.smithi027.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.2 MB/s | 888 kB 00:00 2024-03-06T07:20:15.116 INFO:teuthology.orchestra.run.smithi027.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 697 kB/s | 145 kB 00:00 2024-03-06T07:20:15.316 INFO:teuthology.orchestra.run.smithi027.stdout:(4/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 15 MB/s | 20 MB 00:01 2024-03-06T07:20:15.442 INFO:teuthology.orchestra.run.smithi027.stdout:(5/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 3.6 MB/s | 5.3 MB 00:01 2024-03-06T07:20:15.483 INFO:teuthology.orchestra.run.smithi027.stdout:(6/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 9.0 MB/s | 1.5 MB 00:00 2024-03-06T07:20:15.500 INFO:teuthology.orchestra.run.smithi069.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.1 MB/s | 888 kB 00:00 2024-03-06T07:20:15.675 INFO:teuthology.orchestra.run.smithi069.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 830 kB/s | 145 kB 00:00 2024-03-06T07:20:15.942 INFO:teuthology.orchestra.run.smithi027.stdout:(7/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 2.8 MB/s | 2.3 MB 00:00 2024-03-06T07:20:16.511 INFO:teuthology.orchestra.run.smithi027.stdout:(8/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 4.4 MB/s | 4.7 MB 00:01 2024-03-06T07:20:16.701 INFO:teuthology.orchestra.run.smithi027.stdout:(9/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x86 14 MB/s | 17 MB 00:01 2024-03-06T07:20:16.727 INFO:teuthology.orchestra.run.smithi027.stdout:(10/176): ceph-selinux-19.0.0-1739.g1bc250c7.el 116 kB/s | 25 kB 00:00 2024-03-06T07:20:16.826 INFO:teuthology.orchestra.run.smithi027.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 2.2 MB/s | 220 kB 00:00 2024-03-06T07:20:16.935 INFO:teuthology.orchestra.run.smithi027.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 249 kB/s | 27 kB 00:00 2024-03-06T07:20:17.035 INFO:teuthology.orchestra.run.smithi027.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 313 kB/s | 31 kB 00:00 2024-03-06T07:20:17.310 INFO:teuthology.orchestra.run.smithi027.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 2.7 MB/s | 747 kB 00:00 2024-03-06T07:20:17.419 INFO:teuthology.orchestra.run.smithi027.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.5 MB/s | 163 kB 00:00 2024-03-06T07:20:17.520 INFO:teuthology.orchestra.run.smithi027.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.2 MB/s | 127 kB 00:00 2024-03-06T07:20:17.558 INFO:teuthology.orchestra.run.smithi069.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.2 MB/s | 2.3 MB 00:01 2024-03-06T07:20:17.684 INFO:teuthology.orchestra.run.smithi069.stdout:(5/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.6 MB/s | 5.3 MB 00:03 2024-03-06T07:20:17.703 INFO:teuthology.orchestra.run.smithi027.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 2.7 MB/s | 501 kB 00:00 2024-03-06T07:20:17.995 INFO:teuthology.orchestra.run.smithi027.stdout:(18/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 4.8 MB/s | 9.8 MB 00:02 2024-03-06T07:20:18.087 INFO:teuthology.orchestra.run.smithi027.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 494 kB/s | 45 kB 00:00 2024-03-06T07:20:18.195 INFO:teuthology.orchestra.run.smithi027.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.2 MB/s | 131 kB 00:00 2024-03-06T07:20:18.312 INFO:teuthology.orchestra.run.smithi027.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.4 MB/s | 162 kB 00:00 2024-03-06T07:20:18.430 INFO:teuthology.orchestra.run.smithi027.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 2.7 MB/s | 321 kB 00:00 2024-03-06T07:20:18.546 INFO:teuthology.orchestra.run.smithi027.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 2.5 MB/s | 298 kB 00:00 2024-03-06T07:20:18.647 INFO:teuthology.orchestra.run.smithi027.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 993 kB/s | 99 kB 00:00 2024-03-06T07:20:18.755 INFO:teuthology.orchestra.run.smithi027.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 789 kB/s | 85 kB 00:00 2024-03-06T07:20:18.939 INFO:teuthology.orchestra.run.smithi027.stdout:(26/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 4.1 MB/s | 5.1 MB 00:01 2024-03-06T07:20:19.034 INFO:teuthology.orchestra.run.smithi069.stdout:(6/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 1.0 MB/s | 1.5 MB 00:01 2024-03-06T07:20:19.056 INFO:teuthology.orchestra.run.smithi027.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 1.4 MB/s | 171 kB 00:00 2024-03-06T07:20:19.139 INFO:teuthology.orchestra.run.smithi027.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 378 kB/s | 31 kB 00:00 2024-03-06T07:20:19.240 INFO:teuthology.orchestra.run.smithi027.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 1.3 MB/s | 138 kB 00:00 2024-03-06T07:20:19.292 INFO:teuthology.orchestra.run.smithi069.stdout:(7/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 4.2 MB/s | 20 MB 00:04 2024-03-06T07:20:19.432 INFO:teuthology.orchestra.run.smithi027.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 4.6 MB/s | 3.1 MB 00:00 2024-03-06T07:20:19.673 INFO:teuthology.orchestra.run.smithi027.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 4.1 MB/s | 1.8 MB 00:00 2024-03-06T07:20:19.782 INFO:teuthology.orchestra.run.smithi027.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 2.2 MB/s | 248 kB 00:00 2024-03-06T07:20:19.882 INFO:teuthology.orchestra.run.smithi027.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 493 kB/s | 49 kB 00:00 2024-03-06T07:20:19.983 INFO:teuthology.orchestra.run.smithi027.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 153 kB/s | 15 kB 00:00 2024-03-06T07:20:20.091 INFO:teuthology.orchestra.run.smithi027.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 2.4 MB/s | 264 kB 00:00 2024-03-06T07:20:20.308 INFO:teuthology.orchestra.run.smithi027.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 3.4 MB/s | 748 kB 00:00 2024-03-06T07:20:20.417 INFO:teuthology.orchestra.run.smithi027.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 169 kB/s | 18 kB 00:00 2024-03-06T07:20:20.726 INFO:teuthology.orchestra.run.smithi069.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 6.8 MB/s | 9.8 MB 00:01 2024-03-06T07:20:20.784 INFO:teuthology.orchestra.run.smithi069.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 433 kB/s | 25 kB 00:00 2024-03-06T07:20:20.834 INFO:teuthology.orchestra.run.smithi027.stdout:(38/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 11 MB/s | 47 MB 00:04 2024-03-06T07:20:20.951 INFO:teuthology.orchestra.run.smithi069.stdout:(10/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x8 1.4 MB/s | 4.7 MB 00:03 2024-03-06T07:20:20.967 INFO:teuthology.orchestra.run.smithi027.stdout:(39/176): ceph-mgr-diskprediction-local-19.0.0- 4.8 MB/s | 7.4 MB 00:01 2024-03-06T07:20:21.043 INFO:teuthology.orchestra.run.smithi069.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 2.4 MB/s | 220 kB 00:00 2024-03-06T07:20:21.102 INFO:teuthology.orchestra.run.smithi069.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 464 kB/s | 27 kB 00:00 2024-03-06T07:20:21.118 INFO:teuthology.orchestra.run.smithi027.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 3.5 MB/s | 920 kB 00:00 2024-03-06T07:20:21.143 INFO:teuthology.orchestra.run.smithi027.stdout:(41/176): automake-1.16.2-8.el9.noarch.rpm 912 kB/s | 662 kB 00:00 2024-03-06T07:20:21.160 INFO:teuthology.orchestra.run.smithi027.stdout:(42/176): boost-program-options-1.75.0-8.el9.x8 645 kB/s | 106 kB 00:00 2024-03-06T07:20:21.161 INFO:teuthology.orchestra.run.smithi069.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 540 kB/s | 31 kB 00:00 2024-03-06T07:20:21.177 INFO:teuthology.orchestra.run.smithi027.stdout:(43/176): elfutils-libelf-devel-0.190-2.el9.x86 350 kB/s | 20 kB 00:00 2024-03-06T07:20:21.227 INFO:teuthology.orchestra.run.smithi027.stdout:(44/176): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-06T07:20:21.244 INFO:teuthology.orchestra.run.smithi027.stdout:(45/176): flexiblas-3.0.4-8.el9.x86_64.rpm 379 kB/s | 32 kB 00:00 2024-03-06T07:20:21.361 INFO:teuthology.orchestra.run.smithi027.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 129 kB/s | 17 kB 00:00 2024-03-06T07:20:21.369 INFO:teuthology.orchestra.run.smithi069.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 3.5 MB/s | 747 kB 00:00 2024-03-06T07:20:21.378 INFO:teuthology.orchestra.run.smithi027.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 75 kB/s | 10 kB 00:00 2024-03-06T07:20:21.436 INFO:teuthology.orchestra.run.smithi027.stdout:(48/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-03-06T07:20:21.462 INFO:teuthology.orchestra.run.smithi027.stdout:(49/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-06T07:20:21.479 INFO:teuthology.orchestra.run.smithi027.stdout:(50/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 446 kB/s | 44 kB 00:00 2024-03-06T07:20:21.485 INFO:teuthology.orchestra.run.smithi069.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.4 MB/s | 163 kB 00:00 2024-03-06T07:20:21.504 INFO:teuthology.orchestra.run.smithi027.stdout:(51/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-06T07:20:21.537 INFO:teuthology.orchestra.run.smithi027.stdout:(52/176): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-06T07:20:21.554 INFO:teuthology.orchestra.run.smithi027.stdout:(53/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 330 kB/s | 24 kB 00:00 2024-03-06T07:20:21.561 INFO:teuthology.orchestra.run.smithi069.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.7 MB/s | 127 kB 00:00 2024-03-06T07:20:21.571 INFO:teuthology.orchestra.run.smithi027.stdout:(54/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 699 kB/s | 46 kB 00:00 2024-03-06T07:20:21.596 INFO:teuthology.orchestra.run.smithi027.stdout:(55/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-06T07:20:21.621 INFO:teuthology.orchestra.run.smithi027.stdout:(56/176): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-06T07:20:21.638 INFO:teuthology.orchestra.run.smithi027.stdout:(57/176): openblas-0.3.21-2.el9.x86_64.rpm 516 kB/s | 34 kB 00:00 2024-03-06T07:20:21.711 INFO:teuthology.orchestra.run.smithi069.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 3.3 MB/s | 501 kB 00:00 2024-03-06T07:20:21.814 INFO:teuthology.orchestra.run.smithi027.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-03-06T07:20:21.880 INFO:teuthology.orchestra.run.smithi027.stdout:(59/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-06T07:20:21.922 INFO:teuthology.orchestra.run.smithi027.stdout:(60/176): postgresql-13.11-1.el9.x86_64.rpm 5.4 MB/s | 1.5 MB 00:00 2024-03-06T07:20:21.947 INFO:teuthology.orchestra.run.smithi027.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 6.1 MB/s | 811 kB 00:00 2024-03-06T07:20:21.973 INFO:teuthology.orchestra.run.smithi027.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-06T07:20:22.065 INFO:teuthology.orchestra.run.smithi027.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 8.7 MB/s | 1.0 MB 00:00 2024-03-06T07:20:22.140 INFO:teuthology.orchestra.run.smithi027.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-06T07:20:22.157 INFO:teuthology.orchestra.run.smithi027.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 427 kB/s | 78 kB 00:00 2024-03-06T07:20:22.266 INFO:teuthology.orchestra.run.smithi027.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-06T07:20:22.282 INFO:teuthology.orchestra.run.smithi027.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 227 kB 00:00 2024-03-06T07:20:22.308 INFO:teuthology.orchestra.run.smithi027.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 165 kB/s | 24 kB 00:00 2024-03-06T07:20:22.333 INFO:teuthology.orchestra.run.smithi027.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 272 kB/s | 18 kB 00:00 2024-03-06T07:20:22.358 INFO:teuthology.orchestra.run.smithi027.stdout:(70/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-06T07:20:22.383 INFO:teuthology.orchestra.run.smithi027.stdout:(71/176): python3-markupsafe-1.1.1-12.el9.x86_6 420 kB/s | 32 kB 00:00 2024-03-06T07:20:22.509 INFO:teuthology.orchestra.run.smithi027.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-06T07:20:22.534 INFO:teuthology.orchestra.run.smithi027.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-06T07:20:22.559 INFO:teuthology.orchestra.run.smithi027.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-06T07:20:22.576 INFO:teuthology.orchestra.run.smithi027.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-06T07:20:22.601 INFO:teuthology.orchestra.run.smithi027.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 983 kB/s | 41 kB 00:00 2024-03-06T07:20:22.627 INFO:teuthology.orchestra.run.smithi027.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-06T07:20:22.652 INFO:teuthology.orchestra.run.smithi027.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-06T07:20:22.677 INFO:teuthology.orchestra.run.smithi027.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-06T07:20:22.702 INFO:teuthology.orchestra.run.smithi027.stdout:(80/176): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-06T07:20:22.727 INFO:teuthology.orchestra.run.smithi027.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 573 kB/s | 43 kB 00:00 2024-03-06T07:20:22.786 INFO:teuthology.orchestra.run.smithi027.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 464 kB/s | 39 kB 00:00 2024-03-06T07:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 4.5 MB/s | 5.1 MB 00:01 2024-03-06T07:20:22.903 INFO:teuthology.orchestra.run.smithi069.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 907 kB/s | 45 kB 00:00 2024-03-06T07:20:22.962 INFO:teuthology.orchestra.run.smithi069.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 2.2 MB/s | 131 kB 00:00 2024-03-06T07:20:23.054 INFO:teuthology.orchestra.run.smithi069.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.7 MB/s | 162 kB 00:00 2024-03-06T07:20:23.086 INFO:teuthology.orchestra.run.smithi027.stdout:(83/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-06T07:20:23.112 INFO:teuthology.orchestra.run.smithi027.stdout:(84/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 107 kB/s | 41 kB 00:00 2024-03-06T07:20:23.137 INFO:teuthology.orchestra.run.smithi027.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 620 kB/s | 215 kB 00:00 2024-03-06T07:20:23.162 INFO:teuthology.orchestra.run.smithi027.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 451 kB/s | 34 kB 00:00 2024-03-06T07:20:23.171 INFO:teuthology.orchestra.run.smithi069.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 2.7 MB/s | 321 kB 00:00 2024-03-06T07:20:23.187 INFO:teuthology.orchestra.run.smithi027.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 872 kB/s | 43 kB 00:00 2024-03-06T07:20:23.212 INFO:teuthology.orchestra.run.smithi027.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-06T07:20:23.229 INFO:teuthology.orchestra.run.smithi027.stdout:(89/176): uuid-1.6.2-55.el9.x86_64.rpm 848 kB/s | 56 kB 00:00 2024-03-06T07:20:23.255 INFO:teuthology.orchestra.run.smithi027.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-03-06T07:20:23.271 INFO:teuthology.orchestra.run.smithi027.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-06T07:20:23.287 INFO:teuthology.orchestra.run.smithi069.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 2.5 MB/s | 298 kB 00:00 2024-03-06T07:20:23.288 INFO:teuthology.orchestra.run.smithi027.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 755 kB/s | 44 kB 00:00 2024-03-06T07:20:23.314 INFO:teuthology.orchestra.run.smithi027.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-06T07:20:23.339 INFO:teuthology.orchestra.run.smithi027.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 186 kB 00:00 2024-03-06T07:20:23.354 INFO:teuthology.orchestra.run.smithi069.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 1.5 MB/s | 99 kB 00:00 2024-03-06T07:20:23.372 INFO:teuthology.orchestra.run.smithi027.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-03-06T07:20:23.389 INFO:teuthology.orchestra.run.smithi027.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 194 kB 00:00 2024-03-06T07:20:23.405 INFO:teuthology.orchestra.run.smithi069.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-06T07:20:23.414 INFO:teuthology.orchestra.run.smithi027.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.9 MB/s | 217 kB 00:00 2024-03-06T07:20:23.440 INFO:teuthology.orchestra.run.smithi027.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-06T07:20:23.475 INFO:teuthology.orchestra.run.smithi027.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-06T07:20:23.499 INFO:teuthology.orchestra.run.smithi027.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-06T07:20:23.524 INFO:teuthology.orchestra.run.smithi027.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.5 MB/s | 124 kB 00:00 2024-03-06T07:20:23.541 INFO:teuthology.orchestra.run.smithi027.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.7 MB/s | 113 kB 00:00 2024-03-06T07:20:23.634 INFO:teuthology.orchestra.run.smithi027.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 1.7 MB/s | 187 kB 00:00 2024-03-06T07:20:23.691 INFO:teuthology.orchestra.run.smithi027.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.4 MB/s | 245 kB 00:00 2024-03-06T07:20:23.800 INFO:teuthology.orchestra.run.smithi027.stdout:(105/176): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-06T07:20:23.947 INFO:teuthology.orchestra.run.smithi069.stdout:(26/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 15 MB/s | 47 MB 00:03 2024-03-06T07:20:24.114 INFO:teuthology.orchestra.run.smithi069.stdout:(27/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 3.3 MB/s | 17 MB 00:05 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi069.stdout:(28/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 932 kB/s | 171 kB 00:00 2024-03-06T07:20:24.156 INFO:teuthology.orchestra.run.smithi069.stdout:(29/176): ceph-grafana-dashboards-19.0.0-1739.g 758 kB/s | 31 kB 00:00 2024-03-06T07:20:24.189 INFO:teuthology.orchestra.run.smithi069.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 2.3 MB/s | 138 kB 00:00 2024-03-06T07:20:24.334 INFO:teuthology.orchestra.run.smithi069.stdout:(31/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 3.3 MB/s | 3.1 MB 00:00 2024-03-06T07:20:24.434 INFO:teuthology.orchestra.run.smithi069.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 2.4 MB/s | 248 kB 00:00 2024-03-06T07:20:24.442 INFO:teuthology.orchestra.run.smithi027.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 481 kB/s | 309 kB 00:00 2024-03-06T07:20:24.492 INFO:teuthology.orchestra.run.smithi069.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 848 kB/s | 49 kB 00:00 2024-03-06T07:20:24.525 INFO:teuthology.orchestra.run.smithi027.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 234 kB/s | 19 kB 00:00 2024-03-06T07:20:24.534 INFO:teuthology.orchestra.run.smithi069.stdout:(34/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 4.7 MB/s | 1.8 MB 00:00 2024-03-06T07:20:24.551 INFO:teuthology.orchestra.run.smithi069.stdout:(35/176): ceph-prometheus-alerts-19.0.0-1739.g1 262 kB/s | 15 kB 00:00 2024-03-06T07:20:24.626 INFO:teuthology.orchestra.run.smithi069.stdout:(36/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 2.8 MB/s | 264 kB 00:00 2024-03-06T07:20:24.677 INFO:teuthology.orchestra.run.smithi069.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 366 kB/s | 18 kB 00:00 2024-03-06T07:20:24.769 INFO:teuthology.orchestra.run.smithi069.stdout:(38/176): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-03-06T07:20:24.836 INFO:teuthology.orchestra.run.smithi069.stdout:(39/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 2.6 MB/s | 748 kB 00:00 2024-03-06T07:20:24.969 INFO:teuthology.orchestra.run.smithi069.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 5.2 MB/s | 920 kB 00:00 2024-03-06T07:20:24.994 INFO:teuthology.orchestra.run.smithi069.stdout:(41/176): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 662 kB 00:00 2024-03-06T07:20:25.011 INFO:teuthology.orchestra.run.smithi069.stdout:(42/176): boost-program-options-1.75.0-8.el9.x8 711 kB/s | 106 kB 00:00 2024-03-06T07:20:25.036 INFO:teuthology.orchestra.run.smithi069.stdout:(43/176): elfutils-libelf-devel-0.190-2.el9.x86 305 kB/s | 20 kB 00:00 2024-03-06T07:20:25.070 INFO:teuthology.orchestra.run.smithi069.stdout:(44/176): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-06T07:20:25.095 INFO:teuthology.orchestra.run.smithi069.stdout:(45/176): flexiblas-3.0.4-8.el9.x86_64.rpm 379 kB/s | 32 kB 00:00 2024-03-06T07:20:25.201 INFO:teuthology.orchestra.run.smithi027.stdout:(108/176): protobuf-compiler-3.14.0-13.el9.x86_ 551 kB/s | 863 kB 00:01 2024-03-06T07:20:25.254 INFO:teuthology.orchestra.run.smithi069.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-03-06T07:20:25.271 INFO:teuthology.orchestra.run.smithi069.stdout:(47/176): flexiblas-openblas-openmp-3.0.4-8.el9 86 kB/s | 17 kB 00:00 2024-03-06T07:20:25.287 INFO:teuthology.orchestra.run.smithi069.stdout:(48/176): libaio-devel-0.3.111-13.el9.x86_64.rp 53 kB/s | 10 kB 00:00 2024-03-06T07:20:25.301 INFO:teuthology.orchestra.run.smithi027.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-06T07:20:25.313 INFO:teuthology.orchestra.run.smithi069.stdout:(49/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-06T07:20:25.330 INFO:teuthology.orchestra.run.smithi069.stdout:(50/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-06T07:20:25.355 INFO:teuthology.orchestra.run.smithi069.stdout:(51/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi069.stdout:(52/176): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-03-06T07:20:25.393 INFO:teuthology.orchestra.run.smithi027.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-03-06T07:20:25.397 INFO:teuthology.orchestra.run.smithi069.stdout:(53/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 368 kB/s | 24 kB 00:00 2024-03-06T07:20:25.414 INFO:teuthology.orchestra.run.smithi069.stdout:(54/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 800 kB/s | 46 kB 00:00 2024-03-06T07:20:25.439 INFO:teuthology.orchestra.run.smithi069.stdout:(55/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-06T07:20:25.465 INFO:teuthology.orchestra.run.smithi069.stdout:(56/176): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-06T07:20:25.468 INFO:teuthology.orchestra.run.smithi027.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 898 kB/s | 67 kB 00:00 2024-03-06T07:20:25.481 INFO:teuthology.orchestra.run.smithi069.stdout:(57/176): openblas-0.3.21-2.el9.x86_64.rpm 513 kB/s | 34 kB 00:00 2024-03-06T07:20:25.535 INFO:teuthology.orchestra.run.smithi027.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-06T07:20:25.715 INFO:teuthology.orchestra.run.smithi069.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-03-06T07:20:25.736 INFO:teuthology.orchestra.run.smithi027.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-06T07:20:25.774 INFO:teuthology.orchestra.run.smithi069.stdout:(59/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-06T07:20:25.807 INFO:teuthology.orchestra.run.smithi069.stdout:(60/176): postgresql-13.11-1.el9.x86_64.rpm 4.7 MB/s | 1.5 MB 00:00 2024-03-06T07:20:25.844 INFO:teuthology.orchestra.run.smithi027.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-03-06T07:20:25.857 INFO:teuthology.orchestra.run.smithi069.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 5.7 MB/s | 811 kB 00:00 2024-03-06T07:20:25.878 INFO:teuthology.orchestra.run.smithi027.stdout:(115/176): abseil-cpp-20211102.0-3.el9.x86_64.r 251 kB/s | 548 kB 00:02 2024-03-06T07:20:25.883 INFO:teuthology.orchestra.run.smithi069.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.2 MB/s | 134 kB 00:00 2024-03-06T07:20:25.931 INFO:teuthology.orchestra.run.smithi027.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 522 kB/s | 43 kB 00:00 2024-03-06T07:20:25.954 INFO:teuthology.orchestra.run.smithi027.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-03-06T07:20:25.958 INFO:teuthology.orchestra.run.smithi069.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-03-06T07:20:26.034 INFO:teuthology.orchestra.run.smithi069.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-06T07:20:26.045 INFO:teuthology.orchestra.run.smithi027.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-03-06T07:20:26.059 INFO:teuthology.orchestra.run.smithi069.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 450 kB/s | 78 kB 00:00 2024-03-06T07:20:26.159 INFO:teuthology.orchestra.run.smithi069.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-06T07:20:26.184 INFO:teuthology.orchestra.run.smithi069.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 227 kB 00:00 2024-03-06T07:20:26.201 INFO:teuthology.orchestra.run.smithi069.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-03-06T07:20:26.227 INFO:teuthology.orchestra.run.smithi069.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 272 kB/s | 18 kB 00:00 2024-03-06T07:20:26.251 INFO:teuthology.orchestra.run.smithi069.stdout:(70/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-06T07:20:26.276 INFO:teuthology.orchestra.run.smithi069.stdout:(71/176): python3-markupsafe-1.1.1-12.el9.x86_6 420 kB/s | 32 kB 00:00 2024-03-06T07:20:26.360 INFO:teuthology.orchestra.run.smithi069.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-06T07:20:26.385 INFO:teuthology.orchestra.run.smithi069.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-06T07:20:26.411 INFO:teuthology.orchestra.run.smithi069.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 174 kB 00:00 2024-03-06T07:20:26.436 INFO:teuthology.orchestra.run.smithi069.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 923 kB/s | 69 kB 00:00 2024-03-06T07:20:26.461 INFO:teuthology.orchestra.run.smithi069.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-06T07:20:26.486 INFO:teuthology.orchestra.run.smithi069.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-06T07:20:26.512 INFO:teuthology.orchestra.run.smithi069.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi069.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-06T07:20:26.562 INFO:teuthology.orchestra.run.smithi069.stdout:(80/176): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-06T07:20:26.587 INFO:teuthology.orchestra.run.smithi069.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 572 kB/s | 43 kB 00:00 2024-03-06T07:20:26.629 INFO:teuthology.orchestra.run.smithi069.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 582 kB/s | 39 kB 00:00 2024-03-06T07:20:26.747 INFO:teuthology.orchestra.run.smithi069.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 264 kB/s | 41 kB 00:00 2024-03-06T07:20:26.896 INFO:teuthology.orchestra.run.smithi069.stdout:(84/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-06T07:20:26.921 INFO:teuthology.orchestra.run.smithi069.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 746 kB/s | 215 kB 00:00 2024-03-06T07:20:26.947 INFO:teuthology.orchestra.run.smithi069.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 171 kB/s | 34 kB 00:00 2024-03-06T07:20:26.972 INFO:teuthology.orchestra.run.smithi069.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 869 kB/s | 43 kB 00:00 2024-03-06T07:20:26.997 INFO:teuthology.orchestra.run.smithi069.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-06T07:20:27.022 INFO:teuthology.orchestra.run.smithi069.stdout:(89/176): uuid-1.6.2-55.el9.x86_64.rpm 753 kB/s | 56 kB 00:00 2024-03-06T07:20:27.039 INFO:teuthology.orchestra.run.smithi069.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-03-06T07:20:27.045 INFO:teuthology.orchestra.run.smithi027.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 157 kB/s | 172 kB 00:01 2024-03-06T07:20:27.064 INFO:teuthology.orchestra.run.smithi069.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 939 kB/s | 63 kB 00:00 2024-03-06T07:20:27.090 INFO:teuthology.orchestra.run.smithi069.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 661 kB/s | 44 kB 00:00 2024-03-06T07:20:27.115 INFO:teuthology.orchestra.run.smithi069.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi069.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2024-03-06T07:20:27.174 INFO:teuthology.orchestra.run.smithi069.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-03-06T07:20:27.199 INFO:teuthology.orchestra.run.smithi069.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 194 kB 00:00 2024-03-06T07:20:27.224 INFO:teuthology.orchestra.run.smithi069.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 217 kB 00:00 2024-03-06T07:20:27.249 INFO:teuthology.orchestra.run.smithi069.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-06T07:20:27.283 INFO:teuthology.orchestra.run.smithi069.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-06T07:20:27.308 INFO:teuthology.orchestra.run.smithi069.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-06T07:20:27.342 INFO:teuthology.orchestra.run.smithi069.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 2.1 MB/s | 124 kB 00:00 2024-03-06T07:20:27.367 INFO:teuthology.orchestra.run.smithi069.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.3 MB/s | 113 kB 00:00 2024-03-06T07:20:27.384 INFO:teuthology.orchestra.run.smithi069.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.4 MB/s | 187 kB 00:00 2024-03-06T07:20:27.518 INFO:teuthology.orchestra.run.smithi069.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.6 MB/s | 245 kB 00:00 2024-03-06T07:20:27.543 INFO:teuthology.orchestra.run.smithi069.stdout:(105/176): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-06T07:20:27.827 INFO:teuthology.orchestra.run.smithi069.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-03-06T07:20:27.852 INFO:teuthology.orchestra.run.smithi069.stdout:(107/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-03-06T07:20:27.885 INFO:teuthology.orchestra.run.smithi069.stdout:(108/176): grpc-data-1.46.7-10.el9.noarch.rpm 334 kB/s | 19 kB 00:00 2024-03-06T07:20:27.936 INFO:teuthology.orchestra.run.smithi069.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-03-06T07:20:27.994 INFO:teuthology.orchestra.run.smithi069.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 835 kB/s | 49 kB 00:00 2024-03-06T07:20:28.061 INFO:teuthology.orchestra.run.smithi069.stdout:(111/176): libarrow-9.0.0-10.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-03-06T07:20:28.086 INFO:teuthology.orchestra.run.smithi069.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-03-06T07:20:28.120 INFO:teuthology.orchestra.run.smithi069.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-06T07:20:28.187 INFO:teuthology.orchestra.run.smithi069.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 8.1 MB/s | 548 kB 00:00 2024-03-06T07:20:28.212 INFO:teuthology.orchestra.run.smithi069.stdout:(115/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2024-03-06T07:20:28.246 INFO:teuthology.orchestra.run.smithi069.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 743 kB/s | 43 kB 00:00 2024-03-06T07:20:28.271 INFO:teuthology.orchestra.run.smithi069.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-03-06T07:20:28.305 INFO:teuthology.orchestra.run.smithi069.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-03-06T07:20:28.338 INFO:teuthology.orchestra.run.smithi069.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-06T07:20:28.372 INFO:teuthology.orchestra.run.smithi069.stdout:(120/176): protobuf-compiler-3.14.0-13.el9.x86_ 875 kB/s | 863 kB 00:00 2024-03-06T07:20:28.398 INFO:teuthology.orchestra.run.smithi069.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-06T07:20:28.423 INFO:teuthology.orchestra.run.smithi069.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-03-06T07:20:28.464 INFO:teuthology.orchestra.run.smithi069.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-06T07:20:28.531 INFO:teuthology.orchestra.run.smithi069.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-06T07:20:28.556 INFO:teuthology.orchestra.run.smithi069.stdout:(125/176): python3-colorama-0.4.4-12.el9.noarch 191 kB/s | 35 kB 00:00 2024-03-06T07:20:28.607 INFO:teuthology.orchestra.run.smithi069.stdout:(126/176): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-03-06T07:20:28.632 INFO:teuthology.orchestra.run.smithi069.stdout:(127/176): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-03-06T07:20:28.657 INFO:teuthology.orchestra.run.smithi069.stdout:(128/176): python3-jaraco-classes-3.2.1-5.el9.n 175 kB/s | 18 kB 00:00 2024-03-06T07:20:28.683 INFO:teuthology.orchestra.run.smithi069.stdout:(129/176): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-03-06T07:20:28.708 INFO:teuthology.orchestra.run.smithi069.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-06T07:20:28.733 INFO:teuthology.orchestra.run.smithi069.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-03-06T07:20:28.775 INFO:teuthology.orchestra.run.smithi069.stdout:(132/176): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-03-06T07:20:28.796 INFO:teuthology.orchestra.run.smithi027.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 130 kB/s | 358 kB 00:02 2024-03-06T07:20:28.800 INFO:teuthology.orchestra.run.smithi069.stdout:(133/176): python3-logutils-0.3.5-21.el9.noarch 502 kB/s | 46 kB 00:00 2024-03-06T07:20:28.821 INFO:teuthology.orchestra.run.smithi027.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 171 kB/s | 303 kB 00:01 2024-03-06T07:20:28.834 INFO:teuthology.orchestra.run.smithi069.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 990 kB/s | 58 kB 00:00 2024-03-06T07:20:28.859 INFO:teuthology.orchestra.run.smithi069.stdout:(135/176): python3-more-itertools-8.12.0-2.el9. 626 kB/s | 79 kB 00:00 2024-03-06T07:20:28.884 INFO:teuthology.orchestra.run.smithi069.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-06T07:20:28.910 INFO:teuthology.orchestra.run.smithi069.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-03-06T07:20:28.935 INFO:teuthology.orchestra.run.smithi069.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-06T07:20:28.960 INFO:teuthology.orchestra.run.smithi069.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 373 kB/s | 28 kB 00:00 2024-03-06T07:20:28.985 INFO:teuthology.orchestra.run.smithi069.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-03-06T07:20:29.019 INFO:teuthology.orchestra.run.smithi069.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-06T07:20:29.044 INFO:teuthology.orchestra.run.smithi069.stdout:(142/176): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-06T07:20:29.070 INFO:teuthology.orchestra.run.smithi069.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 426 kB/s | 36 kB 00:00 2024-03-06T07:20:29.095 INFO:teuthology.orchestra.run.smithi069.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-03-06T07:20:29.096 INFO:teuthology.orchestra.run.smithi027.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 118 kB/s | 35 kB 00:00 2024-03-06T07:20:29.128 INFO:teuthology.orchestra.run.smithi069.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-06T07:20:29.179 INFO:teuthology.orchestra.run.smithi069.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 824 kB/s | 90 kB 00:00 2024-03-06T07:20:29.204 INFO:teuthology.orchestra.run.smithi069.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-03-06T07:20:29.229 INFO:teuthology.orchestra.run.smithi069.stdout:(148/176): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-06T07:20:29.254 INFO:teuthology.orchestra.run.smithi069.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-06T07:20:29.346 INFO:teuthology.orchestra.run.smithi069.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-06T07:20:29.380 INFO:teuthology.orchestra.run.smithi069.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:20:29.455 INFO:teuthology.orchestra.run.smithi069.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 30 MB/s | 2.3 MB 00:00 2024-03-06T07:20:29.522 INFO:teuthology.orchestra.run.smithi069.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-03-06T07:20:29.555 INFO:teuthology.orchestra.run.smithi069.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:20:29.589 INFO:teuthology.orchestra.run.smithi069.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 7.2 MB/s | 247 kB 00:00 2024-03-06T07:20:29.623 INFO:teuthology.orchestra.run.smithi069.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:20:29.648 INFO:teuthology.orchestra.run.smithi069.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.4 MB/s | 35 kB 00:00 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi069.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 7.5 MB/s | 191 kB 00:00 2024-03-06T07:20:29.715 INFO:teuthology.orchestra.run.smithi069.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:20:29.766 INFO:teuthology.orchestra.run.smithi069.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-03-06T07:20:29.808 INFO:teuthology.orchestra.run.smithi069.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi069.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 662 kB/s | 22 kB 00:00 2024-03-06T07:20:29.875 INFO:teuthology.orchestra.run.smithi069.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 107 kB 00:00 2024-03-06T07:20:29.917 INFO:teuthology.orchestra.run.smithi069.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 154 kB 00:00 2024-03-06T07:20:29.976 INFO:teuthology.orchestra.run.smithi069.stdout:(165/176): librados2-19.0.0-1739.g1bc250c7.el9. 4.5 MB/s | 3.4 MB 00:00 2024-03-06T07:20:30.001 INFO:teuthology.orchestra.run.smithi069.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 135 kB 00:00 2024-03-06T07:20:30.028 INFO:teuthology.orchestra.run.smithi069.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 63 kB 00:00 2024-03-06T07:20:30.029 INFO:teuthology.orchestra.run.smithi027.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 168 kB/s | 203 kB 00:01 2024-03-06T07:20:30.051 INFO:teuthology.orchestra.run.smithi069.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 552 kB/s | 28 kB 00:00 2024-03-06T07:20:30.085 INFO:teuthology.orchestra.run.smithi069.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 20 MB/s | 1.2 MB 00:00 2024-03-06T07:20:30.135 INFO:teuthology.orchestra.run.smithi069.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-03-06T07:20:30.186 INFO:teuthology.orchestra.run.smithi069.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 23 MB/s | 2.2 MB 00:00 2024-03-06T07:20:30.211 INFO:teuthology.orchestra.run.smithi069.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 5.9 MB/s | 438 kB 00:00 2024-03-06T07:20:30.236 INFO:teuthology.orchestra.run.smithi069.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 612 kB/s | 30 kB 00:00 2024-03-06T07:20:30.261 INFO:teuthology.orchestra.run.smithi069.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-06T07:20:30.295 INFO:teuthology.orchestra.run.smithi069.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-06T07:20:30.345 INFO:teuthology.orchestra.run.smithi069.stdout:(176/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-03-06T07:20:30.349 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:20:30.349 INFO:teuthology.orchestra.run.smithi069.stdout:Total 14 MB/s | 224 MB 00:16 2024-03-06T07:20:30.399 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-06T07:20:30.711 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-06T07:20:30.711 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-06T07:20:30.888 INFO:teuthology.orchestra.run.smithi027.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 168 kB/s | 144 kB 00:00 2024-03-06T07:20:30.980 INFO:teuthology.orchestra.run.smithi027.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-06T07:20:31.122 INFO:teuthology.orchestra.run.smithi027.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 125 kB/s | 18 kB 00:00 2024-03-06T07:20:31.289 INFO:teuthology.orchestra.run.smithi027.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 139 kB/s | 23 kB 00:00 2024-03-06T07:20:31.448 INFO:teuthology.orchestra.run.smithi027.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 123 kB/s | 19 kB 00:00 2024-03-06T07:20:31.606 INFO:teuthology.orchestra.run.smithi027.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 124 kB/s | 20 kB 00:00 2024-03-06T07:20:32.679 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-06T07:20:32.679 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-06T07:20:35.856 INFO:teuthology.orchestra.run.smithi027.stdout:(130/176): libarrow-9.0.0-10.el9.x86_64.rpm 399 kB/s | 4.4 MB 00:11 2024-03-06T07:20:36.040 INFO:teuthology.orchestra.run.smithi027.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 252 kB/s | 46 kB 00:00 2024-03-06T07:20:36.224 INFO:teuthology.orchestra.run.smithi027.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 430 kB/s | 79 kB 00:00 2024-03-06T07:20:36.349 INFO:teuthology.orchestra.run.smithi027.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 462 kB/s | 58 kB 00:00 2024-03-06T07:20:36.599 INFO:teuthology.orchestra.run.smithi027.stdout:(134/176): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-03-06T07:20:36.674 INFO:teuthology.orchestra.run.smithi027.stdout:(135/176): python3-portend-3.1.0-2.el9.noarch.r 220 kB/s | 16 kB 00:00 2024-03-06T07:20:36.975 INFO:teuthology.orchestra.run.smithi027.stdout:(136/176): python3-kubernetes-26.1.0-2.el9.noar 195 kB/s | 1.0 MB 00:05 2024-03-06T07:20:37.000 INFO:teuthology.orchestra.run.smithi027.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 276 kB/s | 90 kB 00:00 2024-03-06T07:20:37.117 INFO:teuthology.orchestra.run.smithi027.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 198 kB/s | 28 kB 00:00 2024-03-06T07:20:37.251 INFO:teuthology.orchestra.run.smithi027.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 123 kB/s | 31 kB 00:00 2024-03-06T07:20:37.651 INFO:teuthology.orchestra.run.smithi027.stdout:(140/176): python3-rsa-4.9-2.el9.noarch.rpm 148 kB/s | 59 kB 00:00 2024-03-06T07:20:37.684 INFO:teuthology.orchestra.run.smithi027.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 332 kB/s | 188 kB 00:00 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi027.stdout:(142/176): python3-typing-extensions-4.4.0-2.el 292 kB/s | 51 kB 00:00 2024-03-06T07:20:37.918 INFO:teuthology.orchestra.run.smithi027.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 134 kB/s | 36 kB 00:00 2024-03-06T07:20:38.502 INFO:teuthology.orchestra.run.smithi027.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 358 kB/s | 230 kB 00:00 2024-03-06T07:20:38.527 INFO:teuthology.orchestra.run.smithi027.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 147 kB/s | 90 kB 00:00 2024-03-06T07:20:38.702 INFO:teuthology.orchestra.run.smithi027.stdout:(146/176): python3-zc-lockfile-2.0-10.el9.noarc 114 kB/s | 20 kB 00:00 2024-03-06T07:20:39.565 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-06T07:20:39.677 INFO:teuthology.orchestra.run.smithi027.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 363 kB/s | 427 kB 00:01 2024-03-06T07:20:39.691 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:20:39.718 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:20:39.805 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:39.886 INFO:teuthology.orchestra.run.smithi027.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 162 kB/s | 192 kB 00:01 2024-03-06T07:20:40.055 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:40.152 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:20:40.197 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:20:40.255 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:20:40.303 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:20:40.328 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:20:40.345 INFO:teuthology.orchestra.run.smithi027.stdout:(149/176): librados2-19.0.0-1739.g1bc250c7.el9. 7.4 MB/s | 3.4 MB 00:00 2024-03-06T07:20:40.381 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:20:40.612 INFO:teuthology.orchestra.run.smithi027.stdout:(150/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 12 MB/s | 3.1 MB 00:00 2024-03-06T07:20:40.662 INFO:teuthology.orchestra.run.smithi027.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 68 kB 00:00 2024-03-06T07:20:40.724 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:20:40.771 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:40.879 INFO:teuthology.orchestra.run.smithi027.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 10 MB/s | 2.3 MB 00:00 2024-03-06T07:20:40.914 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:40.939 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:40.987 INFO:teuthology.orchestra.run.smithi027.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 17 MB/s | 1.8 MB 00:00 2024-03-06T07:20:41.021 INFO:teuthology.orchestra.run.smithi027.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:20:41.031 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:41.063 INFO:teuthology.orchestra.run.smithi027.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:20:41.070 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:41.088 INFO:teuthology.orchestra.run.smithi027.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.7 MB/s | 69 kB 00:00 2024-03-06T07:20:41.122 INFO:teuthology.orchestra.run.smithi027.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-03-06T07:20:41.164 INFO:teuthology.orchestra.run.smithi027.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:20:41.206 INFO:teuthology.orchestra.run.smithi027.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:20:41.243 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:20:41.256 INFO:teuthology.orchestra.run.smithi027.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-03-06T07:20:41.282 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:20:41.298 INFO:teuthology.orchestra.run.smithi027.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-03-06T07:20:41.309 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:41.331 INFO:teuthology.orchestra.run.smithi027.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 660 kB/s | 22 kB 00:00 2024-03-06T07:20:41.365 INFO:teuthology.orchestra.run.smithi027.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 3.1 MB/s | 107 kB 00:00 2024-03-06T07:20:41.399 INFO:teuthology.orchestra.run.smithi027.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 4.5 MB/s | 154 kB 00:00 2024-03-06T07:20:41.405 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:41.427 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:41.441 INFO:teuthology.orchestra.run.smithi027.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 135 kB 00:00 2024-03-06T07:20:41.466 INFO:teuthology.orchestra.run.smithi027.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 2.4 MB/s | 63 kB 00:00 2024-03-06T07:20:41.492 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:41.532 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:20:41.554 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:41.569 INFO:teuthology.orchestra.run.smithi027.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 271 kB/s | 28 kB 00:00 2024-03-06T07:20:41.570 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:41.755 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:41.785 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:41.792 INFO:teuthology.orchestra.run.smithi027.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 5.2 MB/s | 1.2 MB 00:00 2024-03-06T07:20:41.817 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:20:41.884 INFO:teuthology.orchestra.run.smithi027.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 23 MB/s | 2.2 MB 00:00 2024-03-06T07:20:41.960 INFO:teuthology.orchestra.run.smithi027.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 29 MB/s | 2.2 MB 00:00 2024-03-06T07:20:41.986 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:20:42.001 INFO:teuthology.orchestra.run.smithi027.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 10 MB/s | 438 kB 00:00 2024-03-06T07:20:42.027 INFO:teuthology.orchestra.run.smithi027.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 1.2 MB/s | 30 kB 00:00 2024-03-06T07:20:42.068 INFO:teuthology.orchestra.run.smithi027.stdout:(173/176): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-06T07:20:42.111 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:42.119 INFO:teuthology.orchestra.run.smithi027.stdout:(174/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 21 MB/s | 1.1 MB 00:00 2024-03-06T07:20:42.154 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:42.255 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:42.314 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:42.553 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:42.659 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:20:42.754 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:20:42.794 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:20:42.861 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:20:42.916 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:20:42.952 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:20:42.990 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:20:43.030 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:20:43.094 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:20:43.129 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:20:43.180 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:20:43.223 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:20:43.250 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:20:43.318 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:20:43.355 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:20:43.398 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:20:43.427 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:20:43.495 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:20:43.541 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:20:43.614 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:20:43.756 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:20:43.810 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:20:43.953 INFO:teuthology.orchestra.run.smithi027.stdout:(175/176): thrift-0.15.0-2.el9.x86_64.rpm 379 kB/s | 1.6 MB 00:04 2024-03-06T07:20:43.994 INFO:teuthology.orchestra.run.smithi027.stdout:(176/176): python3-grpcio-1.46.7-10.el9.x86_64. 140 kB/s | 2.0 MB 00:14 2024-03-06T07:20:43.998 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:20:43.999 INFO:teuthology.orchestra.run.smithi027.stdout:Total 7.4 MB/s | 224 MB 00:30 2024-03-06T07:20:44.052 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-06T07:20:44.324 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:20:44.396 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:20:44.399 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-06T07:20:44.399 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-06T07:20:44.474 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:20:44.546 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:44.554 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:44.589 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:20:44.618 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:20:44.662 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:20:44.780 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:20:44.835 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:20:45.094 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:20:45.281 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:20:45.353 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:20:45.405 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:20:45.631 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:20:45.705 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:20:45.763 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:20:45.843 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:20:45.875 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:20:45.912 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:20:45.958 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:20:45.994 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:20:46.040 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:20:46.082 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:20:46.215 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:20:46.262 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:20:46.292 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:20:46.333 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:20:46.428 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:20:46.467 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-06T07:20:46.509 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-06T07:20:46.533 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:20:46.557 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:20:46.644 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:20:46.766 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:20:46.865 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:20:46.916 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:20:46.963 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:20:47.020 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:20:47.204 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:20:47.520 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:20:47.657 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:20:47.702 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:20:47.734 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:20:47.774 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:20:47.809 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:20:47.845 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:20:47.868 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:20:47.902 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:20:47.941 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:20:47.973 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:20:48.003 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:20:48.079 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:20:48.114 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:20:48.147 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:20:48.188 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:20:48.313 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:20:48.393 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:20:48.437 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:20:48.468 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:20:48.507 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:20:48.605 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:20:49.381 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:20:49.434 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:20:49.468 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:20:49.524 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:20:49.603 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:20:49.694 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:20:49.870 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:20:49.900 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:20:49.977 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:20:50.612 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:50.722 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:51.937 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:52.020 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:52.048 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:52.358 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:52.386 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:52.459 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:52.490 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:52.527 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:52.979 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:53.005 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:53.076 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:53.098 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:53.729 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-06T07:20:53.816 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:20:53.842 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:20:53.888 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:53.917 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:53.958 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:20:54.004 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:20:54.061 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:20:54.110 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:20:54.135 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:20:54.187 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:20:54.531 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:20:54.586 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:54.732 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:54.761 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:54.837 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:54.875 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:55.048 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:20:55.109 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:20:55.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:55.228 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:55.241 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:55.262 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:55.270 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:55.335 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:55.375 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:20:55.397 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:55.413 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:55.574 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:55.604 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:55.636 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:20:55.805 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:55.810 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:20:55.849 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:55.920 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:55.926 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:55.926 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:55.926 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:55.926 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:20:55.966 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:55.980 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:56.094 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:56.174 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:56.378 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:56.483 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:20:56.577 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:20:56.612 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:20:56.683 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:20:56.753 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:20:56.818 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:20:56.859 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:20:56.933 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:20:56.965 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:20:57.009 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:20:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:20:57.069 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:20:57.132 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:20:57.174 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:20:57.219 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:20:57.246 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:20:57.317 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:20:57.361 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:20:57.436 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:20:57.577 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:20:57.633 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:20:58.187 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:20:58.270 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:20:58.344 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:20:58.397 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:58.405 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:58.444 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:20:58.482 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:20:58.517 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:20:58.609 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:20:58.674 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:20:58.920 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:20:59.106 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:20:59.175 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:20:59.220 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:20:59.369 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:20:59.414 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:20:59.479 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:20:59.553 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:20:59.589 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:20:59.618 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:20:59.665 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:20:59.701 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:20:59.744 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:20:59.786 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:20:59.927 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:20:59.974 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:21:00.005 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:21:00.047 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:21:00.147 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:21:00.179 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:21:00.220 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:21:00.245 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:21:00.269 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:21:00.349 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:21:00.477 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:21:00.593 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:21:00.637 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:21:00.696 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:21:00.759 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:21:00.951 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:21:01.016 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:21:01.162 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:21:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:21:01.230 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:21:01.261 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:21:01.297 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:21:01.336 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:21:01.364 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:21:01.398 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:21:01.438 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:21:01.469 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:21:01.499 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:21:01.578 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:21:01.609 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:21:01.643 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:21:01.685 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:21:01.719 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:21:01.819 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:21:01.910 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:21:01.959 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:21:01.989 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:21:02.038 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:21:02.148 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:21:02.209 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:21:02.976 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:21:03.029 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:21:03.063 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:21:03.113 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:21:03.203 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:21:03.247 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:21:03.432 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:21:03.461 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:21:03.551 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:21:04.230 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:21:04.339 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:21:05.641 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:21:05.716 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:21:05.743 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:21:06.071 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:21:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:21:06.189 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:21:06.219 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:21:06.734 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:21:06.768 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:21:06.823 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:21:06.853 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:21:07.777 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:21:07.777 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2024-03-06T07:21:07.777 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2024-03-06T07:21:07.777 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2024-03-06T07:21:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2024-03-06T07:21:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2024-03-06T07:21:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2024-03-06T07:21:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2024-03-06T07:21:07.778 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:07.928 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:21:07.944 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:21:07.944 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:07.944 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:21:07.944 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:09.076 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:21:09.107 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:21:09.423 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:21:09.449 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:21:09.449 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:09.449 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:21:09.449 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:21:09.449 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:21:09.449 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:09.676 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:21:09.683 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:21:09.705 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:21:09.705 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:09.706 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:21:09.706 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:21:09.706 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:21:09.706 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:09.711 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:21:09.790 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:21:09.791 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:21:09.791 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:21:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:09.829 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:21:10.151 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:21:10.174 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:21:10.174 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:10.175 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:21:10.175 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:21:10.175 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:21:10.175 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:10.261 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:21:10.670 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:21:10.701 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:21:10.732 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:21:10.793 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:21:10.816 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:21:11.042 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:21:11.075 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:21:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:21:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:21:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:21:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:21:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:21:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:21:12.411 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:12.412 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:21:12.412 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:21:12.412 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:21:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:21:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:21:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:21:13.387 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:21:13.413 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:21:13.436 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:21:13.436 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:13.436 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:21:13.436 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:21:13.436 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:21:13.436 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:13.489 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:21:13.511 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:21:13.511 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:13.511 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:21:13.511 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:13.564 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:21:13.587 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:21:13.587 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:13.587 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:21:13.587 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:21:13.587 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:21:13.588 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:13.890 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:21:13.913 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:21:13.913 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:13.913 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:21:13.913 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:21:13.913 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:21:13.913 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:18.319 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:21:18.360 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:21:18.399 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:21:18.427 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:21:18.618 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:21:18.725 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:21:18.767 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:21:18.822 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:21:18.891 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:21:18.966 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:21:19.010 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:21:19.053 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:21:19.085 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:21:19.114 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:21:19.255 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:21:19.296 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:21:19.893 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:21:19.936 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:21:19.955 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:21:20.757 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:21:20.831 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:21:20.872 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:21:20.963 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:21:20.992 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:21:21.018 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:21:21.043 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:21:21.070 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:21:21.070 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:21:21.154 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:21:21.182 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:21:21.221 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:21:21.221 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:21:21.297 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:21:21.324 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:21:21.357 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:21:21.382 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:21:21.414 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:21:21.441 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:21:21.484 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:21:21.522 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:21:21.555 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:21:21.580 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:21:21.605 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:21:21.630 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:21:21.656 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:21:21.663 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:21:21.663 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:21:21.850 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:21:21.850 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:21:21.942 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:21:21.973 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:21:22.000 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:21:22.003 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:21:22.003 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-03-06T07:21:22.003 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-03-06T07:21:22.003 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-03-06T07:21:22.003 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-03-06T07:21:22.003 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-03-06T07:21:22.003 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-03-06T07:21:22.003 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-03-06T07:21:22.003 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:22.025 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:21:22.051 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:21:22.116 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:21:22.168 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:21:22.184 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:21:22.184 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:22.184 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:21:22.184 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:23.721 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:21:23.748 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:21:23.748 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:23.748 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:21:23.748 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:21:23.748 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:21:23.748 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:23.980 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:21:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:21:24.004 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:24.004 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:21:24.004 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:21:24.004 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:21:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:24.472 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:21:24.496 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:21:24.496 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:24.496 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:21:24.496 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:21:24.496 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:21:24.496 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:24.590 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:21:25.018 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:21:25.047 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:21:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:21:25.142 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:21:25.170 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:21:25.412 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:21:25.438 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:21:26.451 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:21:26.480 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:21:26.586 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:21:26.758 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:21:26.790 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:21:26.815 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:21:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:21:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:21:26.815 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:21:26.815 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:26.866 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:21:26.877 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:21:26.932 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:21:26.982 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:21:26.982 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:21:26.982 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:21:26.982 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:21:26.982 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:21:26.982 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:21:26.982 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:21:26.982 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:21:26.983 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:21:26.984 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:21:27.001 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:21:27.001 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:21:27.001 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:21:27.001 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:21:27.001 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:21:27.002 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:21:27.003 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:21:27.003 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:21:27.003 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:21:27.003 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:21:27.003 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:21:27.003 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:21:27.003 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:21:27.004 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:21:27.005 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:21:27.006 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:21:27.007 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:21:27.008 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:21:27.009 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:21:27.836 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:21:27.868 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:21:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:21:27.892 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:27.892 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:21:27.892 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:21:27.892 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:21:27.892 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:27.935 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:21:27.958 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:21:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:21:27.959 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:28.019 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:21:28.044 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:21:28.044 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:28.044 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:21:28.044 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:21:28.044 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:21:28.044 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:28.370 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:21:28.394 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:21:28.394 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:21:28.394 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:21:28.394 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:21:28.394 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:21:28.394 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:29.671 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:21:29.672 INFO:teuthology.orchestra.run.smithi069.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:21:29.673 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.674 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.675 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:21:29.676 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:21:29.677 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:21:29.678 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:21:29.679 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:21:29.680 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:21:29.681 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:21:29.682 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:21:29.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.684 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:29.685 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:21:29.685 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:21:29.685 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:21:29.685 INFO:teuthology.orchestra.run.smithi069.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:21:29.685 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:21:29.685 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:21:29.685 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:21:29.685 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:29.685 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-06T07:21:30.160 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:33.058 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:21:33.119 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:21:33.146 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:21:33.173 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:21:33.376 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:21:33.493 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:21:33.530 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:21:33.604 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:21:33.663 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:21:33.751 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:21:33.798 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:21:33.842 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:21:33.872 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:21:33.902 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:21:33.975 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:21:34.031 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:21:34.067 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:21:34.669 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:21:34.706 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:21:34.727 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:21:35.615 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:21:35.685 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:21:35.727 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:21:35.823 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:21:35.856 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:21:35.881 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:21:35.931 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:21:35.957 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:21:35.957 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:21:36.042 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:21:36.071 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:21:36.100 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:21:36.100 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:21:36.178 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:21:36.203 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:21:36.239 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:21:36.263 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:21:36.298 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:21:36.331 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:21:36.365 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:21:36.404 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:21:36.436 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:21:36.461 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:21:36.487 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:21:36.511 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:21:36.537 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:21:36.544 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:21:36.544 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:21:36.732 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:21:36.732 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:21:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:21:36.830 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:21:36.856 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:21:36.881 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:21:36.907 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:21:36.976 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:21:41.837 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:21:41.837 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:21:41.837 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:21:41.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:21:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:21:41.857 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:21:41.858 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:21:41.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:21:41.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:21:41.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:21:41.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:21:41.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:21:41.859 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:21:41.860 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:21:41.861 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:21:41.862 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:21:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:21:41.864 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:21:41.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:21:41.866 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:21:41.867 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:21:41.868 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:21:41.869 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:21:41.870 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:21:41.871 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:21:41.871 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:21:41.871 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:21:41.871 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:21:41.871 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:21:41.871 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:21:44.727 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:21:44.728 INFO:teuthology.orchestra.run.smithi027.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:21:44.729 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.730 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.731 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:21:44.732 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.733 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:21:44.734 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:21:44.735 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:21:44.736 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:21:44.737 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:21:44.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:21:44.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:21:44.740 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:21:44.741 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:21:44.742 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:21:44.742 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:44.742 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-06T07:21:45.411 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:45.416 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:21:45.569 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:21:45.593 INFO:teuthology.orchestra.run.smithi027.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:21:45.594 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:21:45.594 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:21:45.595 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:21:45.815 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:21:45.841 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:21:45.842 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:21:45.842 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:21:45.844 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:21:45.844 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:45.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:21:45.882 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:21:45.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:21:45.916 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:21:45.917 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:45.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:21:45.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:21:46.029 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:21:46.029 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:21:46.059 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:21:46.126 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:21:46.127 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:46.127 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:21:46.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:21:46.232 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:21:46.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:21:46.260 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:21:46.328 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:21:46.328 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:46.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:21:46.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:21:46.434 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:21:46.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:21:46.461 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:21:46.529 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:21:46.618 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:21:46.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2024-03-06T07:21:46.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2024-03-06T07:21:46.652 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:21:46.653 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:21:46.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:21:46.725 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:21:46.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:21:46.768 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:21:46.803 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:21:46.803 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:21:46.803 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:21:46.803 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:46.803 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:21:46.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:46.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:47.009 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:21:47.009 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:21:47.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:47.140 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:47.173 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:21:47.173 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:21:47.173 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583058', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJ3vhpohM9xB5ux8f7iuiyN+Bf6kEPwaNo6LDrS7aDwVfwCdJ+zvK44YvoLEqEN6iG62xG3g1AQhcguxwafKWc=', 'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVpYiu/YoCYgaV/eQmFohRMqb3IJ5fc5iJmrQsU0co4UgQoF2P0s0/28GIdJ3ak/3iXpY9jZANDyCOyCMhbsro='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:21:47.174 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:21:47.176 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:21:47.222 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:21:47.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:21:47.289 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:47.289 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:21:47.314 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:21:47.315 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-03-06T07:21:47.370 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:21:47.370 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:47.370 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-06T07:21:47.370 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:47.371 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:47.371 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-06 07:21:37.351546835 +0000 2024-03-06T07:21:47.371 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-06 07:19:26.236361766 +0000 2024-03-06T07:21:47.371 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-06 07:19:26.236361766 +0000 2024-03-06T07:21:47.371 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-06 07:19:26.236361766 +0000 2024-03-06T07:21:47.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:21:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-06T07:21:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-06T07:21:47.439 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000238792 s, 2.1 MB/s 2024-03-06T07:21:47.441 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:21:47.497 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-03-06T07:21:47.553 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:21:47.553 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:47.553 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-06T07:21:47.554 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:47.554 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:47.554 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-06 07:21:37.351546835 +0000 2024-03-06T07:21:47.554 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-06 07:19:26.594352681 +0000 2024-03-06T07:21:47.554 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-06 07:19:26.594352681 +0000 2024-03-06T07:21:47.554 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-06 07:19:26.594352681 +0000 2024-03-06T07:21:47.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:21:47.623 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-06T07:21:47.623 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-06T07:21:47.623 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000202397 s, 2.5 MB/s 2024-03-06T07:21:47.624 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:21:47.680 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-03-06T07:21:47.736 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:21:47.736 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:47.736 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-06T07:21:47.736 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:47.736 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-06 07:21:37.352546810 +0000 2024-03-06T07:21:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-06 07:19:26.951343621 +0000 2024-03-06T07:21:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-06 07:19:26.951343621 +0000 2024-03-06T07:21:47.737 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-06 07:19:26.951343621 +0000 2024-03-06T07:21:47.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-06T07:21:47.805 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000203949 s, 2.5 MB/s 2024-03-06T07:21:47.806 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:21:47.864 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-03-06T07:21:47.919 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:21:47.919 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:47.919 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-06T07:21:47.919 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:47.920 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:47.920 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-06 07:21:37.352546810 +0000 2024-03-06T07:21:47.920 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-06 07:19:27.295334891 +0000 2024-03-06T07:21:47.920 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-06 07:19:27.295334891 +0000 2024-03-06T07:21:47.920 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-06 07:19:27.295334891 +0000 2024-03-06T07:21:47.920 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:21:47.988 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-06T07:21:47.988 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-06T07:21:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000228705 s, 2.2 MB/s 2024-03-06T07:21:47.990 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:21:48.046 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:48.046 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:21:48.046 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:21:48.063 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:21:48.064 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-03-06T07:21:48.120 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:21:48.120 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:48.120 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-06T07:21:48.120 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:48.120 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:48.120 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-06 07:21:22.477055704 +0000 2024-03-06T07:21:48.120 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-06 07:19:20.012900427 +0000 2024-03-06T07:21:48.120 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-06 07:19:20.012900427 +0000 2024-03-06T07:21:48.121 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-06 07:19:20.012900427 +0000 2024-03-06T07:21:48.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:21:48.189 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-06T07:21:48.189 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-06T07:21:48.189 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000233807 s, 2.2 MB/s 2024-03-06T07:21:48.190 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:21:48.248 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-03-06T07:21:48.304 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:21:48.304 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:48.304 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-06T07:21:48.304 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:48.304 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:48.304 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-06 07:21:22.478055681 +0000 2024-03-06T07:21:48.304 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-06 07:19:20.340892772 +0000 2024-03-06T07:21:48.304 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-06 07:19:20.340892772 +0000 2024-03-06T07:21:48.304 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-06 07:19:20.340892772 +0000 2024-03-06T07:21:48.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:21:48.370 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-06T07:21:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-06T07:21:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000212457 s, 2.4 MB/s 2024-03-06T07:21:48.372 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:21:48.429 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-03-06T07:21:48.484 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:21:48.484 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:48.484 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-06T07:21:48.484 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:48.484 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:48.484 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-06 07:21:22.478055681 +0000 2024-03-06T07:21:48.485 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-06 07:19:20.678884884 +0000 2024-03-06T07:21:48.485 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-06 07:19:20.678884884 +0000 2024-03-06T07:21:48.485 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-06 07:19:20.678884884 +0000 2024-03-06T07:21:48.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:21:48.550 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-06T07:21:48.551 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-06T07:21:48.551 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000190592 s, 2.7 MB/s 2024-03-06T07:21:48.552 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:21:48.608 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-03-06T07:21:48.663 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:21:48.663 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:48.663 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-06T07:21:48.663 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:48.663 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:48.663 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-06 07:21:22.478055681 +0000 2024-03-06T07:21:48.664 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-06 07:19:21.011877112 +0000 2024-03-06T07:21:48.664 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-06 07:19:21.011877112 +0000 2024-03-06T07:21:48.664 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-06 07:19:21.011877112 +0000 2024-03-06T07:21:48.664 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:21:48.729 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-06T07:21:48.729 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-06T07:21:48.729 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000178182 s, 2.9 MB/s 2024-03-06T07:21:48.730 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:21:48.787 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:48.787 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:21:48.787 INFO:tasks.ceph:Generating config... 2024-03-06T07:21:48.788 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:21:48.788 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:21:48.788 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:21:48.789 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:21:48.789 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:21:48.789 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:21:48.789 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:21:48.790 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:21:48.790 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:21:48.790 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:21:48.790 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:21:48.790 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:21:48.791 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:21:48.791 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:21:48.791 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:21:48.791 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:21:48.791 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:21:48.791 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:21:48.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:48.854 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:21:48.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:21:48.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:21:48.938 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27'), ('mon.b', '172.21.15.69'), ('mon.c', '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]')] 2024-03-06T07:21:48.938 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:21:48.939 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:48.939 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:21:48.994 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --add b 172.21.15.69 --addv c '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:49.081 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:49.081 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:21:49.081 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-03-06T07:21:49.081 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2024-03-06T07:21:49.081 INFO:teuthology.orchestra.run.smithi027.stdout:fsid ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:21:49.081 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2024-03-06T07:21:49.080192+0000 2024-03-06T07:21:49.081 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-03-06T07:21:49.080192+0000 2024-03-06T07:21:49.081 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 17 (quincy) 2024-03-06T07:21:49.082 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2024-03-06T07:21:49.082 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-03-06T07:21:49.082 INFO:teuthology.orchestra.run.smithi027.stdout:1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.b 2024-03-06T07:21:49.082 INFO:teuthology.orchestra.run.smithi027.stdout:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-03-06T07:21:49.082 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:21:49.083 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:21:49.098 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3... 2024-03-06T07:21:49.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:21:49.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:21:49.191 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-03-06T07:21:49.191 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-03-06T07:21:49.191 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:21:49.191 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-03-06T07:21:49.191 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.196 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:21:49.197 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:21:49.198 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-03-06T07:21:49.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:21:49.201 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:21:49.202 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:21:49.203 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:21:49.203 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-06T07:21:49.203 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-06T07:21:49.203 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:21:49.203 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-06T07:21:49.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.203 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-06T07:21:49.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.203 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.204 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.205 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-06T07:21:49.206 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-06T07:21:49.207 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-06T07:21:49.208 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:21:49.209 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:21:49.210 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:21:49.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:21:49.212 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-03-06T07:21:49.213 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-03-06T07:21:49.214 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:21:49.214 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-03-06T07:21:49.214 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-03-06T07:21:49.214 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-03-06T07:21:49.214 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-03-06T07:21:49.214 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:21:49.214 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:21:49.214 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:21:49.216 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:21:49.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:21:49.316 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:21:49.316 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:49.316 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:21:49.331 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:49.331 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:21:49.386 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:21:49.386 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:49.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:21:49.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:21:49.472 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:21:49.473 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:49.529 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:21:49.529 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:21:49.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:21:49.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:21:49.568 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:21:49.568 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:49.623 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:21:49.623 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:21:49.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:21:49.687 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:21:49.690 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:21:49.754 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:21:49.756 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:21:49.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:21:49.819 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:21:49.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:21:49.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:21:49.955 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:21:49.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:21:49.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:21:50.044 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:21:50.047 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:21:50.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:21:50.168 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:21:50.170 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:21:50.196 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:21:50.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:21:50.244 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:21:50.262 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:21:50.308 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:21:50.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:21:50.369 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:21:50.384 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:21:50.467 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:21:50.482 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:21:50.482 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:21:50.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:50.514 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:50.514 INFO:tasks.ceph:role: osd.0 2024-03-06T07:21:50.514 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:21:50.514 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:21:50.592 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:50.592 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:50.592 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:50.592 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:50.592 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:50.593 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:50.593 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:50.593 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:50.593 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:50.593 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:51.097 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-06T07:21:51.099 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-06T07:21:51.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:51.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:51.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:51.360 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:51.360 INFO:tasks.ceph:role: osd.1 2024-03-06T07:21:51.360 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:21:51.360 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:21:51.439 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:51.439 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:51.439 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:51.440 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:51.440 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-06T07:21:51.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-06T07:21:51.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:51.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:52.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:52.085 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:52.085 INFO:tasks.ceph:role: osd.2 2024-03-06T07:21:52.085 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:21:52.085 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:21:52.162 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:52.163 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:52.163 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:52.163 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:52.163 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:52.646 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-06T07:21:52.648 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-06T07:21:52.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:52.798 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:52.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:52.908 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:52.908 INFO:tasks.ceph:role: osd.3 2024-03-06T07:21:52.908 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:21:52.908 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:21:52.985 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:53.471 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-06T07:21:53.473 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-06T07:21:53.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:53.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:53.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:53.771+0000 7f6c16837640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:21:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:53.771+0000 7f6c16837640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:21:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:53.771+0000 7f6c16837640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:21:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:53.771+0000 7f6c16837640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:21:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:53.771+0000 7f6c16837640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:21:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:53.771+0000 7f6c16837640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:21:56.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:56.663 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:56.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:56.758+0000 7fe50e021640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:21:56.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:56.758+0000 7fe50e021640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:21:56.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:56.758+0000 7fe50e021640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:56.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:56.758+0000 7fe50e021640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:56.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:56.758+0000 7fe50e021640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:56.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:56.758+0000 7fe50e021640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:21:59.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:59.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:59.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:59.737+0000 7fee7e210640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:21:59.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:59.737+0000 7fee7e210640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:21:59.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:59.737+0000 7fee7e210640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:59.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:59.737+0000 7fee7e210640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:59.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:59.737+0000 7fee7e210640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:59.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:21:59.738+0000 7fee7e210640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:22:02.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:22:02.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:22:02.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:02.723+0000 7f22c4349640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:22:02.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:02.723+0000 7f22c4349640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:22:02.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:02.723+0000 7f22c4349640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:22:02.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:02.723+0000 7f22c4349640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:22:02.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:02.723+0000 7f22c4349640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:22:02.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:02.724+0000 7f22c4349640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:22:05.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:22:05.609 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:22:05.641 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:22:05.641 INFO:tasks.ceph:role: osd.4 2024-03-06T07:22:05.641 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:22:05.641 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:22:05.719 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:22:05.719 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:22:05.719 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:22:05.719 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:22:05.719 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:22:05.720 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:22:05.720 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:22:05.720 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:22:05.720 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:22:05.720 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:22:06.212 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-03-06T07:22:06.214 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-03-06T07:22:06.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:22:06.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:22:06.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:22:06.436 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:22:06.437 INFO:tasks.ceph:role: osd.5 2024-03-06T07:22:06.437 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:22:06.437 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:22:06.514 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:22:06.514 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:22:06.514 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:22:06.514 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:22:06.515 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:22:06.515 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:22:06.515 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:22:06.515 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:22:06.515 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:22:06.515 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:22:06.964 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-03-06T07:22:06.966 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-03-06T07:22:06.966 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:22:07.057 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:22:07.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:22:07.156 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:22:07.156 INFO:tasks.ceph:role: osd.6 2024-03-06T07:22:07.156 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:22:07.156 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:22:07.230 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:22:07.729 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-03-06T07:22:07.731 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-03-06T07:22:07.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:22:07.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:22:07.907 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:22:07.975 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:22:07.975 INFO:tasks.ceph:role: osd.7 2024-03-06T07:22:07.975 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:22:07.975 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:22:08.048 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:22:08.049 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:22:08.049 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:22:08.049 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:22:08.049 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:22:08.049 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:22:08.049 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:22:08.049 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:22:08.049 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:22:08.049 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:22:08.562 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-03-06T07:22:08.564 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-03-06T07:22:08.564 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:22:08.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:22:08.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:22:08.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:08.834+0000 7fbaefe14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:22:08.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:08.834+0000 7fbaefe14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:22:08.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:08.834+0000 7fbaefe14640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:22:08.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:08.834+0000 7fbaefe14640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:22:08.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:08.834+0000 7fbaefe14640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:22:08.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:08.835+0000 7fbaefe14640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:22:11.691 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:22:11.722 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:22:11.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:11.811+0000 7fa6a5fe1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:22:11.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:11.811+0000 7fa6a5fe1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:22:11.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:11.811+0000 7fa6a5fe1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:22:11.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:11.811+0000 7fa6a5fe1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:22:11.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:11.811+0000 7fa6a5fe1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:22:11.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:11.812+0000 7fa6a5fe1640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:22:14.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:22:14.691 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:22:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:14.780+0000 7f09e540e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:22:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:14.780+0000 7f09e540e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:22:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:14.780+0000 7f09e540e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:22:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:14.780+0000 7f09e540e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:22:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:14.780+0000 7f09e540e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:22:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:14.780+0000 7f09e540e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:22:17.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:17.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:17.632 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:22:17.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:22:17.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:17.746+0000 7f523301d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:22:17.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:17.747+0000 7f523301d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:22:17.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:17.747+0000 7f523301d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:22:17.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:17.747+0000 7f523301d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:22:17.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:17.747+0000 7f523301d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:22:17.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:17.747+0000 7f523301d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:22:20.600 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:22:20.631 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:22:20.631 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:20.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:22:20.668 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:20.668 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:22:20.743 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:20.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:22:20.819 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:20.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:22:20.889 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:20.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:22:20.961 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:20.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:22:21.033 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:21.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:22:21.104 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:21.104 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:22:21.136 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:21.137 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:22:21.208 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:21.208 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:22:21.277 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:21.277 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:22:21.346 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:21.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:22:21.415 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:21.415 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:22:21.484 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:21.484 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:22:21.553 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:21.553 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:22:21.568 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:21.568 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:22:21.625 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:21.625 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:22:21.643 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:21.643 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:22:21.695 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:22:21.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:22:21.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:22:21.739 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.y] 2024-03-06T07:22:21.739 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg== 2024-03-06T07:22:21.739 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-06T07:22:21.739 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g== 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg== 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw== 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA== 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA== 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg== 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw== 2024-03-06T07:22:21.740 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ== 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ== 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.4] 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A== 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.5] 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw== 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.6] 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA== 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.7] 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A== 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:22:21.741 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:22:21.742 INFO:teuthology.orchestra.run.smithi027.stdout:[client.2] 2024-03-06T07:22:21.742 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ== 2024-03-06T07:22:21.742 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2024-03-06T07:22:21.742 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ== 2024-03-06T07:22:21.742 INFO:teuthology.orchestra.run.smithi027.stdout:[client.3] 2024-03-06T07:22:21.742 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg== 2024-03-06T07:22:21.761 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.y] 2024-03-06T07:22:21.761 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg== 2024-03-06T07:22:21.761 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-03-06T07:22:21.761 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g== 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg== 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw== 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA== 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA== 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.3] 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg== 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw== 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-03-06T07:22:21.762 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ== 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ== 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.4] 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A== 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.5] 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw== 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.6] 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA== 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.7] 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A== 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout:[client.2] 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ== 2024-03-06T07:22:21.763 INFO:teuthology.orchestra.run.smithi069.stdout:[client.1] 2024-03-06T07:22:21.764 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ== 2024-03-06T07:22:21.764 INFO:teuthology.orchestra.run.smithi069.stdout:[client.3] 2024-03-06T07:22:21.764 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg== 2024-03-06T07:22:21.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:22:21.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:22:21.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:22:21.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:22:21.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:22:21.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:22:21.975 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:21.978 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.090 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:22:22.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:22:22.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:22:22.261 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:22:22.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:22:22.320 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:22:22.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.379 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.436 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.491 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.548 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:22:22.602 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:22:22.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:22:22.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:22:22.663 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:22:22.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:22:22.722 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:22:22.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:22:22.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:22:22.838 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:22:22.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:22:22.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:22:22.970 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:22:23.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:22:23.029 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:22:23.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:22:23.153 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:22:23.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:22:23.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:22:23.345 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:22:23.347 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:22:23.399 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:22:23.400 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:22:23.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:22:23.402 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:22:23.402 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:22:23.402 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:22:23.442 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:22:23.442 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:22:23.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:22:23.444 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:22:23.444 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:22:23.445 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:22:23.445 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:22:23.447 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:22:23.447 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:22:23.447 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:22:23.449 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:22:23.449 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:22:23.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.556+0000 7f6ce214b640 1 Processor -- start 2024-03-06T07:22:23.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.556+0000 7f6ce214b640 1 -- start start 2024-03-06T07:22:23.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.556+0000 7f6ce214b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6cdc0b79f0 0x7f6cdc0b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.556+0000 7f6ce214b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0d4960 con 0x7f6cdc071540 2024-03-06T07:22:23.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.556+0000 7f6ce214b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0d36c0 con 0x7f6cdc0b2d30 2024-03-06T07:22:23.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.556+0000 7f6ce214b640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc0d3230 con 0x7f6cdc0b79f0 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.557+0000 7f6cdbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6cdc0b79f0 0x7f6cdc0b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.557+0000 7f6cdb7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6cdc071540 0x7f6cdc071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53298/0 (socket says 172.21.15.27:53298) 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.557+0000 7f6cdb7fe640 1 -- 172.21.15.27:0/1281403144 learned_addr learned my addr 172.21.15.27:0/1281403144 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.557+0000 7f6cdbfff640 1 -- 172.21.15.27:0/1281403144 >> v2:172.21.15.27:3300/0 conn(0x7f6cdc0b79f0 msgr2=0x7f6cdc0b7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.557+0000 7f6cdbfff640 1 -- 172.21.15.27:0/1281403144 >> v2:172.21.15.27:3300/0 conn(0x7f6cdc0b79f0 msgr2=0x7f6cdc0b7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.557+0000 7f6cdbfff640 1 --2- 172.21.15.27:0/1281403144 >> v2:172.21.15.27:3300/0 conn(0x7f6cdc0b79f0 0x7f6cdc0b7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:22:23.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.557+0000 7f6cdbfff640 1 --2- 172.21.15.27:0/1281403144 >> v2:172.21.15.27:3300/0 conn(0x7f6cdc0b79f0 0x7f6cdc0b7ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:22:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.748+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407642690 0 0) 0x7f6cdc0d4960 con 0x7f6cdc071540 2024-03-06T07:22:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.748+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc4003520 con 0x7f6cdc071540 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381726049 0 0) 0x7f6cdc0d36c0 con 0x7f6cdc0b2d30 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cdc0d4960 con 0x7f6cdc0b2d30 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3395340364 0 0) 0x7f6cc4003520 con 0x7f6cdc071540 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 >> v2:172.21.15.27:3300/0 conn(0x7f6cdc0b79f0 msgr2=0x7f6cdc0b7ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 --2- 172.21.15.27:0/1281403144 >> v2:172.21.15.27:3300/0 conn(0x7f6cdc0b79f0 0x7f6cdc0b7ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 >> v1:172.21.15.69:6789/0 conn(0x7f6cdc0b2d30 legacy=0x7f6cdc0b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0d2620 con 0x7f6cdc071540 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f6ccc001670 con 0x7f6cdc071540 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 >> v1:172.21.15.27:6789/0 conn(0x7f6cdc071540 legacy=0x7f6cdc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4003b40 0x7f6cc4003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cc4004800 0x7f6cc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cc40051c0 0x7f6cc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0d3230 con 0x7f6cc4004800 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc400c0f0 con 0x7f6cc4003b40 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc400c2f0 con 0x7f6cc40051c0 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cdaffd640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cc4004800 0x7f6cc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cdbfff640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cc40051c0 0x7f6cc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.749+0000 7f6cdb7fe640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4003b40 0x7f6cc4003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.750+0000 7f6cdaffd640 1 -- 172.21.15.27:0/1281403144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cc40051c0 msgr2=0x7f6cc400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.750+0000 7f6cdaffd640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cc40051c0 0x7f6cc400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.750+0000 7f6cdaffd640 1 -- 172.21.15.27:0/1281403144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4003b40 msgr2=0x7f6cc4003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.750+0000 7f6cdaffd640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4003b40 0x7f6cc4003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.750+0000 7f6cdaffd640 1 -- 172.21.15.27:0/1281403144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0d2620 con 0x7f6cc4004800 2024-03-06T07:22:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.750+0000 7f6cdbfff640 1 -- 172.21.15.27:0/1281403144 reap_dead start 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.751+0000 7f6cdaffd640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cc4004800 0x7f6cc4004c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00018a0 tx=0x7f6cc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c97836b0c0c7f61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.751+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd002f050 con 0x7f6cc4004800 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.751+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd00052d0 con 0x7f6cc4004800 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.751+0000 7f6cda7fc640 1 -- 172.21.15.27:0/1281403144 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00014f0 con 0x7f6cc4004800 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.752+0000 7f6ce214b640 1 -- 172.21.15.27:0/1281403144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cc4004800 msgr2=0x7f6cc4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.752+0000 7f6ce214b640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cc4004800 0x7f6cc4004c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00018a0 tx=0x7f6cc4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.752+0000 7f6ce214b640 1 -- 172.21.15.27:0/1281403144 shutdown_connections 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.752+0000 7f6ce214b640 1 --2- 172.21.15.27:0/1281403144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cc4004800 0x7f6cc4004c80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.752+0000 7f6ce214b640 1 -- 172.21.15.27:0/1281403144 >> 172.21.15.27:0/1281403144 conn(0x7f6cdc0d3d60 msgr2=0x7f6cdc0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.752+0000 7f6ce214b640 1 -- 172.21.15.27:0/1281403144 shutdown_connections 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.752+0000 7f6ce214b640 1 -- 172.21.15.27:0/1281403144 wait complete. 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.753+0000 7f6ce214b640 1 Processor -- start 2024-03-06T07:22:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.753+0000 7f6ce214b640 1 -- start start 2024-03-06T07:22:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6ce214b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4004800 0x7f6cdc156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6ce214b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc0cae30 0x7f6cdc150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6ce214b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc158630 0x7f6cdc158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6ce214b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0d2ec0 con 0x7f6cdc0cae30 2024-03-06T07:22:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6ce214b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0d2be0 con 0x7f6cc4004800 2024-03-06T07:22:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6ce214b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0d22f0 con 0x7f6cdc158630 2024-03-06T07:22:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6cdb7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4004800 0x7f6cdc156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6cdb7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4004800 0x7f6cdc156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:35086/0 (socket says 172.21.15.27:35086) 2024-03-06T07:22:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6cdb7fe640 1 -- 172.21.15.27:0/2565408245 learned_addr learned my addr 172.21.15.27:0/2565408245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.754+0000 7f6cdaffd640 1 --2- 172.21.15.27:0/2565408245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc0cae30 0x7f6cdc150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cdbfff640 1 --2- 172.21.15.27:0/2565408245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc158630 0x7f6cdc158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cdb7fe640 1 -- 172.21.15.27:0/2565408245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc158630 msgr2=0x7f6cdc158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cdb7fe640 1 --2- 172.21.15.27:0/2565408245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc158630 0x7f6cdc158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cdb7fe640 1 -- 172.21.15.27:0/2565408245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc0cae30 msgr2=0x7f6cdc150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cdb7fe640 1 --2- 172.21.15.27:0/2565408245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc0cae30 0x7f6cdc150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cdb7fe640 1 -- 172.21.15.27:0/2565408245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0cbe40 con 0x7f6cc4004800 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cdb7fe640 1 --2- 172.21.15.27:0/2565408245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4004800 0x7f6cdc156d70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0072c0 tx=0x7f6ccc007780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cc3fff640 1 -- 172.21.15.27:0/2565408245 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc01a070 con 0x7f6cc4004800 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cc3fff640 1 -- 172.21.15.27:0/2565408245 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc00f9c0 con 0x7f6cc4004800 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6cc3fff640 1 -- 172.21.15.27:0/2565408245 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc00fd70 con 0x7f6cc4004800 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6ce214b640 1 -- 172.21.15.27:0/2565408245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc0cb2c0 con 0x7f6cc4004800 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.755+0000 7f6ce214b640 1 -- 172.21.15.27:0/2565408245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc0cb880 con 0x7f6cc4004800 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.756+0000 7f6cc3fff640 1 -- 172.21.15.27:0/2565408245 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6ccc01d070 con 0x7f6cc4004800 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.756+0000 7f6cc3fff640 1 -- 172.21.15.27:0/2565408245 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6ccc01f8b0 con 0x7f6cc4004800 2024-03-06T07:22:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.756+0000 7f6ce214b640 1 -- 172.21.15.27:0/2565408245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc072070 con 0x7f6cc4004800 2024-03-06T07:22:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.759+0000 7f6cc3fff640 1 -- 172.21.15.27:0/2565408245 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6ccc024070 con 0x7f6cc4004800 2024-03-06T07:22:23.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.826+0000 7f6ce214b640 1 -- 172.21.15.27:0/2565408245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6cdc04f410 con 0x7f6cc4004800 2024-03-06T07:22:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.835+0000 7f6cc3fff640 1 -- 172.21.15.27:0/2565408245 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6ccc01fb00 con 0x7f6cc4004800 2024-03-06T07:22:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.836+0000 7f6ce214b640 1 -- 172.21.15.27:0/2565408245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4004800 msgr2=0x7f6cdc156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.836+0000 7f6ce214b640 1 --2- 172.21.15.27:0/2565408245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4004800 0x7f6cdc156d70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0072c0 tx=0x7f6ccc007780 comp rx=0 tx=0).stop 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.836+0000 7f6ce214b640 1 -- 172.21.15.27:0/2565408245 shutdown_connections 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.836+0000 7f6ce214b640 1 --2- 172.21.15.27:0/2565408245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc158630 0x7f6cdc158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.836+0000 7f6ce214b640 1 --2- 172.21.15.27:0/2565408245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc0cae30 0x7f6cdc150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.836+0000 7f6ce214b640 1 --2- 172.21.15.27:0/2565408245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cc4004800 0x7f6cdc156d70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.836+0000 7f6ce214b640 1 -- 172.21.15.27:0/2565408245 >> 172.21.15.27:0/2565408245 conn(0x7f6cdc0d3d60 msgr2=0x7f6cdc0b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.836+0000 7f6ce214b640 1 -- 172.21.15.27:0/2565408245 shutdown_connections 2024-03-06T07:22:23.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:23.836+0000 7f6ce214b640 1 -- 172.21.15.27:0/2565408245 wait complete. 2024-03-06T07:22:23.848 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:22:23.848 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:22:23.849 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:23.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:22:23.879 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:23.879 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:22:23.950 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:23.950 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:22:24.021 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:22:24.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:22:24.092 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:24.092 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:22:24.119 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:24.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:22:24.186 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:24.187 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:22:24.217 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:24.214+0000 7f84f2041040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:22:24.231 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:24.230+0000 7f434121a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:22:24.244 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:22:24.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:22:24.273 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:24.271+0000 7f84f2041040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:22:24.287 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:24.286+0000 7f434121a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:22:24.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 4a4d6578-2fd9-4b76-99c2-17fb01916001 0 2024-03-06T07:22:24.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.437+0000 7f39e5c39640 1 Processor -- start 2024-03-06T07:22:24.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.437+0000 7f39e5c39640 1 -- start start 2024-03-06T07:22:24.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39e5c39640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39e0144f50 0x7f39e01604f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39e5c39640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39e00cfeb0 0x7f39e006fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39e5c39640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e00cf0f0 con 0x7f39e0066d00 2024-03-06T07:22:24.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39e5c39640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e00cf3d0 con 0x7f39e00cfeb0 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39e5c39640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e00cf6b0 con 0x7f39e0144f50 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39df7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39e0144f50 0x7f39e01604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39df7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39e0144f50 0x7f39e01604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44004/0 (socket says 172.21.15.69:44004) 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39dffff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f39e0066d00 0x7f39e00719d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57148/0 (socket says 172.21.15.69:57148) 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39df7fe640 1 -- 172.21.15.69:0/2328571189 learned_addr learned my addr 172.21.15.69:0/2328571189 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.438+0000 7f39d7fff640 1 --2- 172.21.15.69:0/2328571189 >> v2:172.21.15.27:3300/0 conn(0x7f39e00cfeb0 0x7f39e006fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.439+0000 7f39df7fe640 1 -- 172.21.15.69:0/2328571189 >> v2:172.21.15.27:3300/0 conn(0x7f39e00cfeb0 msgr2=0x7f39e006fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.439+0000 7f39df7fe640 1 --2- 172.21.15.69:0/2328571189 >> v2:172.21.15.27:3300/0 conn(0x7f39e00cfeb0 0x7f39e006fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.439+0000 7f39df7fe640 1 -- 172.21.15.69:0/2328571189 >> v1:172.21.15.69:6789/0 conn(0x7f39e0066d00 legacy=0x7f39e00719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.439+0000 7f39df7fe640 1 -- 172.21.15.69:0/2328571189 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e00cf990 con 0x7f39e0144f50 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.439+0000 7f39deffd640 1 -- 172.21.15.69:0/2328571189 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744830532 0 0) 0x7f39e00cf0f0 con 0x7f39e0066d00 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.439+0000 7f39df7fe640 1 --2- 172.21.15.69:0/2328571189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0144f50 0x7f39e01604f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f39e006d7f0 tx=0x7f39d002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6578d40e2998df5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.439+0000 7f39deffd640 1 -- 172.21.15.69:0/2328571189 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0034070 con 0x7f39e0144f50 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.439+0000 7f39deffd640 1 -- 172.21.15.69:0/2328571189 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d0030860 con 0x7f39e0144f50 2024-03-06T07:22:24.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.439+0000 7f39deffd640 1 -- 172.21.15.69:0/2328571189 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0030c30 con 0x7f39e0144f50 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.440+0000 7f39e5c39640 1 -- 172.21.15.69:0/2328571189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0144f50 msgr2=0x7f39e01604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.440+0000 7f39e5c39640 1 --2- 172.21.15.69:0/2328571189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0144f50 0x7f39e01604f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f39e006d7f0 tx=0x7f39d002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.440+0000 7f39e5c39640 1 -- 172.21.15.69:0/2328571189 shutdown_connections 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.440+0000 7f39e5c39640 1 --2- 172.21.15.69:0/2328571189 >> v2:172.21.15.27:3300/0 conn(0x7f39e00cfeb0 0x7f39e006fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.440+0000 7f39e5c39640 1 --2- 172.21.15.69:0/2328571189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0144f50 0x7f39e01604f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.440+0000 7f39e5c39640 1 -- 172.21.15.69:0/2328571189 >> 172.21.15.69:0/2328571189 conn(0x7f39e00d09c0 msgr2=0x7f39e00d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.440+0000 7f39e5c39640 1 -- 172.21.15.69:0/2328571189 shutdown_connections 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.440+0000 7f39e5c39640 1 -- 172.21.15.69:0/2328571189 wait complete. 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.441+0000 7f39e5c39640 1 Processor -- start 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39e5c39640 1 -- start start 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39e5c39640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0066d00 0x7f39e015b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39e5c39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e00cfeb0 0x7f39e0144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39e5c39640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e0144f50 0x7f39e01f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39e5c39640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e00ceda0 con 0x7f39e00cfeb0 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39e5c39640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0145330 con 0x7f39e0066d00 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39e5c39640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0067170 con 0x7f39e0144f50 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0066d00 0x7f39e015b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39dffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e0144f50 0x7f39e01f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0066d00 0x7f39e015b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44008/0 (socket says 172.21.15.69:44008) 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39df7fe640 1 -- 172.21.15.69:0/2639642578 learned_addr learned my addr 172.21.15.69:0/2639642578 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.442+0000 7f39d7fff640 1 --2- 172.21.15.69:0/2639642578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e00cfeb0 0x7f39e0144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39df7fe640 1 -- 172.21.15.69:0/2639642578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e0144f50 msgr2=0x7f39e01f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39df7fe640 1 --2- 172.21.15.69:0/2639642578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e0144f50 0x7f39e01f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39df7fe640 1 -- 172.21.15.69:0/2639642578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e00cfeb0 msgr2=0x7f39e0144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39df7fe640 1 --2- 172.21.15.69:0/2639642578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e00cfeb0 0x7f39e0144010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39df7fe640 1 -- 172.21.15.69:0/2639642578 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0138e20 con 0x7f39e0066d00 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39dffff640 1 --2- 172.21.15.69:0/2639642578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e0144f50 0x7f39e01f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39df7fe640 1 --2- 172.21.15.69:0/2639642578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0066d00 0x7f39e015b290 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f39d0000c00 tx=0x7f39d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39dcff9640 1 -- 172.21.15.69:0/2639642578 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0034040 con 0x7f39e0066d00 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39dcff9640 1 -- 172.21.15.69:0/2639642578 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d003ab90 con 0x7f39e0066d00 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39e5c39640 1 -- 172.21.15.69:0/2639642578 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e0139100 con 0x7f39e0066d00 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39dcff9640 1 -- 172.21.15.69:0/2639642578 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0004030 con 0x7f39e0066d00 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39e5c39640 1 -- 172.21.15.69:0/2639642578 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e01389b0 con 0x7f39e0066d00 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39dcff9640 1 -- 172.21.15.69:0/2639642578 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f39d004b060 con 0x7f39e0066d00 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.443+0000 7f39dcff9640 1 -- 172.21.15.69:0/2639642578 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f39d0003c60 con 0x7f39e0066d00 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.444+0000 7f39e5c39640 1 -- 172.21.15.69:0/2639642578 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e0139620 con 0x7f39e0066d00 2024-03-06T07:22:24.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.446+0000 7f39dcff9640 1 -- 172.21.15.69:0/2639642578 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f39e0139620 con 0x7f39e0066d00 2024-03-06T07:22:24.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.517+0000 7f39e5c39640 1 -- 172.21.15.69:0/2639642578 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4a4d6578-2fd9-4b76-99c2-17fb01916001", "id": 0} v 0) v1 -- 0x7f39e0073380 con 0x7f39e0066d00 2024-03-06T07:22:24.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.523+0000 7f39dcff9640 1 -- 172.21.15.69:0/2639642578 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4a4d6578-2fd9-4b76-99c2-17fb01916001", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f39d003f070 con 0x7f39e0066d00 2024-03-06T07:22:24.525 INFO:teuthology.orchestra.run.smithi069.stdout:0 2024-03-06T07:22:24.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.525+0000 7f39e5c39640 1 -- 172.21.15.69:0/2639642578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0066d00 msgr2=0x7f39e015b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.525+0000 7f39e5c39640 1 --2- 172.21.15.69:0/2639642578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0066d00 0x7f39e015b290 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f39d0000c00 tx=0x7f39d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.525+0000 7f39e5c39640 1 -- 172.21.15.69:0/2639642578 shutdown_connections 2024-03-06T07:22:24.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.525+0000 7f39e5c39640 1 --2- 172.21.15.69:0/2639642578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e0144f50 0x7f39e01f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.525+0000 7f39e5c39640 1 --2- 172.21.15.69:0/2639642578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e00cfeb0 0x7f39e0144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.525+0000 7f39e5c39640 1 --2- 172.21.15.69:0/2639642578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39e0066d00 0x7f39e015b290 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.525+0000 7f39e5c39640 1 -- 172.21.15.69:0/2639642578 >> 172.21.15.69:0/2639642578 conn(0x7f39e00d09c0 msgr2=0x7f39e015e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.525+0000 7f39e5c39640 1 -- 172.21.15.69:0/2639642578 shutdown_connections 2024-03-06T07:22:24.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.525+0000 7f39e5c39640 1 -- 172.21.15.69:0/2639642578 wait complete. 2024-03-06T07:22:24.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 69fd2f9d-6cff-4219-9c13-3a1b08bd1649 1 2024-03-06T07:22:24.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.638+0000 7f18288e1640 1 Processor -- start 2024-03-06T07:22:24.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.638+0000 7f18288e1640 1 -- start start 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.639+0000 7f18288e1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f182415f5a0 0x7f18240ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.639+0000 7f18288e1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.639+0000 7f18288e1640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18240cf2a0 con 0x7f18240c5750 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.639+0000 7f18288e1640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18240cf540 con 0x7f182415f5a0 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.639+0000 7f18288e1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240d0280 con 0x7f18240b2c60 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.639+0000 7f1821d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.639+0000 7f1821d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49880/0 (socket says 172.21.15.69:49880) 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.639+0000 7f1821d9b640 1 -- 172.21.15.69:0/1428894459 learned_addr learned my addr 172.21.15.69:0/1428894459 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.640+0000 7f182259c640 1 --2- 172.21.15.69:0/1428894459 >> v2:172.21.15.27:3300/0 conn(0x7f182415f5a0 0x7f18240ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.640+0000 7f1821d9b640 1 -- 172.21.15.69:0/1428894459 >> v2:172.21.15.27:3300/0 conn(0x7f182415f5a0 msgr2=0x7f18240ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.640+0000 7f1821d9b640 1 --2- 172.21.15.69:0/1428894459 >> v2:172.21.15.27:3300/0 conn(0x7f182415f5a0 0x7f18240ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.640+0000 7f1821d9b640 1 -- 172.21.15.69:0/1428894459 >> v1:172.21.15.27:6789/0 conn(0x7f18240c5750 legacy=0x7f18240c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.640+0000 7f1821d9b640 1 -- 172.21.15.69:0/1428894459 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240d0560 con 0x7f18240b2c60 2024-03-06T07:22:24.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.640+0000 7f1821d9b640 1 --2- 172.21.15.69:0/1428894459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240cea10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1814009870 tx=0x7f181402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9058ddc5336a8cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f182159a640 1 -- 172.21.15.69:0/1428894459 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814034070 con 0x7f18240b2c60 2024-03-06T07:22:24.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f182159a640 1 -- 172.21.15.69:0/1428894459 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1814030840 con 0x7f18240b2c60 2024-03-06T07:22:24.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f182159a640 1 -- 172.21.15.69:0/1428894459 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814030c10 con 0x7f18240b2c60 2024-03-06T07:22:24.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f18288e1640 1 -- 172.21.15.69:0/1428894459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 msgr2=0x7f18240cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f18288e1640 1 --2- 172.21.15.69:0/1428894459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240cea10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1814009870 tx=0x7f181402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:24.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f18288e1640 1 -- 172.21.15.69:0/1428894459 shutdown_connections 2024-03-06T07:22:24.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f18288e1640 1 --2- 172.21.15.69:0/1428894459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240cea10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f18288e1640 1 --2- 172.21.15.69:0/1428894459 >> v2:172.21.15.27:3300/0 conn(0x7f182415f5a0 0x7f18240ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f18288e1640 1 -- 172.21.15.69:0/1428894459 >> 172.21.15.69:0/1428894459 conn(0x7f18240b3700 msgr2=0x7f18240b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f18288e1640 1 -- 172.21.15.69:0/1428894459 shutdown_connections 2024-03-06T07:22:24.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.641+0000 7f18288e1640 1 -- 172.21.15.69:0/1428894459 wait complete. 2024-03-06T07:22:24.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f18288e1640 1 Processor -- start 2024-03-06T07:22:24.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f18288e1640 1 -- start start 2024-03-06T07:22:24.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f18288e1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f18288e1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18240c5750 0x7f18240cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f18288e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182415f5a0 0x7f182406af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f18288e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182415f980 con 0x7f182415f5a0 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f18288e1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240cef50 con 0x7f18240c5750 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f18288e1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240cd720 con 0x7f18240b2c60 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f182259c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f182259c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49884/0 (socket says 172.21.15.69:49884) 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f182259c640 1 -- 172.21.15.69:0/1688255955 learned_addr learned my addr 172.21.15.69:0/1688255955 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f1821d9b640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18240c5750 0x7f18240cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.643+0000 7f1822d9d640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182415f5a0 0x7f182406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f182259c640 1 -- 172.21.15.69:0/1688255955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18240c5750 msgr2=0x7f18240cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f182259c640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18240c5750 0x7f18240cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f182259c640 1 -- 172.21.15.69:0/1688255955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182415f5a0 msgr2=0x7f182406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f182259c640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182415f5a0 0x7f182406af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f182259c640 1 -- 172.21.15.69:0/1688255955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240caaf0 con 0x7f18240b2c60 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f1822d9d640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182415f5a0 0x7f182406af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f182259c640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240b4b80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f180c00e870 tx=0x7f180c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f181affd640 1 -- 172.21.15.69:0/1688255955 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c017070 con 0x7f18240b2c60 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f181affd640 1 -- 172.21.15.69:0/1688255955 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c00c940 con 0x7f18240b2c60 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f18288e1640 1 -- 172.21.15.69:0/1688255955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18240cadd0 con 0x7f18240b2c60 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f181affd640 1 -- 172.21.15.69:0/1688255955 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c00cd30 con 0x7f18240b2c60 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f18288e1640 1 -- 172.21.15.69:0/1688255955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18241520a0 con 0x7f18240b2c60 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f181affd640 1 -- 172.21.15.69:0/1688255955 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f180c01a070 con 0x7f18240b2c60 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.644+0000 7f181affd640 1 -- 172.21.15.69:0/1688255955 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f180c01c840 con 0x7f18240b2c60 2024-03-06T07:22:24.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.645+0000 7f18288e1640 1 -- 172.21.15.69:0/1688255955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182409fcf0 con 0x7f18240b2c60 2024-03-06T07:22:24.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.647+0000 7f181affd640 1 -- 172.21.15.69:0/1688255955 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f180c01ca90 con 0x7f18240b2c60 2024-03-06T07:22:24.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.720+0000 7f18288e1640 1 -- 172.21.15.69:0/1688255955 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "69fd2f9d-6cff-4219-9c13-3a1b08bd1649", "id": 1} v 0) v1 -- 0x7f182404f410 con 0x7f18240b2c60 2024-03-06T07:22:24.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.728+0000 7f181affd640 1 -- 172.21.15.69:0/1688255955 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69fd2f9d-6cff-4219-9c13-3a1b08bd1649", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f180c01cd40 con 0x7f18240b2c60 2024-03-06T07:22:24.730 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-03-06T07:22:24.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.729+0000 7f18288e1640 1 -- 172.21.15.69:0/1688255955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 msgr2=0x7f18240b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.729+0000 7f18288e1640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240b4b80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f180c00e870 tx=0x7f180c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:24.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.729+0000 7f18288e1640 1 -- 172.21.15.69:0/1688255955 shutdown_connections 2024-03-06T07:22:24.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.729+0000 7f18288e1640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182415f5a0 0x7f182406af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.729+0000 7f18288e1640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18240c5750 0x7f18240cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.729+0000 7f18288e1640 1 --2- 172.21.15.69:0/1688255955 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18240b2c60 0x7f18240b4b80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.729+0000 7f18288e1640 1 -- 172.21.15.69:0/1688255955 >> 172.21.15.69:0/1688255955 conn(0x7f18240b3700 msgr2=0x7f18240ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.729+0000 7f18288e1640 1 -- 172.21.15.69:0/1688255955 shutdown_connections 2024-03-06T07:22:24.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.729+0000 7f18288e1640 1 -- 172.21.15.69:0/1688255955 wait complete. 2024-03-06T07:22:24.741 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59 2 2024-03-06T07:22:24.839 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:24.838+0000 7f434121a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:22:24.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.840+0000 7f8e77eaf640 1 Processor -- start 2024-03-06T07:22:24.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.840+0000 7f8e77eaf640 1 -- start start 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e77eaf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e700b52f0 0x7f8e700719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e77eaf640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e700722f0 con 0x7f8e70144370 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e77eaf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7015f0d0 con 0x7f8e70071ee0 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e77eaf640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e7015f3b0 con 0x7f8e700b52f0 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e7644c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8e70071ee0 0x7f8e7006e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57160/0 (socket says 172.21.15.69:57160) 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e7644c640 1 -- 172.21.15.69:0/736798181 learned_addr learned my addr 172.21.15.69:0/736798181 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e7544a640 1 --2- 172.21.15.69:0/736798181 >> v2:172.21.15.27:3300/0 conn(0x7f8e700b52f0 0x7f8e700719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482180178 0 0) 0x7f8e7015f0d0 con 0x7f8e70071ee0 2024-03-06T07:22:24.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5c003520 con 0x7f8e70071ee0 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662595906 0 0) 0x7f8e6c001110 con 0x7f8e70144370 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e7015f0d0 con 0x7f8e70144370 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448239806 0 0) 0x7f8e5c003520 con 0x7f8e70071ee0 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 >> v2:172.21.15.27:3300/0 conn(0x7f8e700b52f0 msgr2=0x7f8e700719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 --2- 172.21.15.69:0/736798181 >> v2:172.21.15.27:3300/0 conn(0x7f8e700b52f0 0x7f8e700719a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 >> v1:172.21.15.27:6789/0 conn(0x7f8e70144370 legacy=0x7f8e7015e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7015f690 con 0x7f8e70071ee0 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f8e680019f0 con 0x7f8e70071ee0 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e7544a640 1 --2- 172.21.15.69:0/736798181 >> v2:172.21.15.27:3300/0 conn(0x7f8e700b52f0 0x7f8e700719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 >> v1:172.21.15.69:6789/0 conn(0x7f8e70071ee0 legacy=0x7f8e7006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e5c003b40 0x7f8e5c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c004800 0x7f8e5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7015f3b0 con 0x7f8e5c004800 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c00e250 con 0x7f8e5c003b40 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c00e450 con 0x7f8e5c0051c0 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.841+0000 7f8e75c4b640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e5c003b40 0x7f8e5c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e7644c640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e5c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e7544a640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c004800 0x7f8e5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e7644c640 1 -- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e5c003b40 msgr2=0x7f8e5c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e7644c640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e5c003b40 0x7f8e5c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e7644c640 1 -- 172.21.15.69:0/736798181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c004800 msgr2=0x7f8e5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e7644c640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c004800 0x7f8e5c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e7644c640 1 -- 172.21.15.69:0/736798181 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c003520 con 0x7f8e5c0051c0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e75c4b640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e5c003b40 0x7f8e5c003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e7544a640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c004800 0x7f8e5c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.842+0000 7f8e7644c640 1 -- 172.21.15.69:0/736798181 reap_dead start 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e7644c640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e5c00b9c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800a800 tx=0x7f8e5c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=862a50e46fa2a4b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0016c0 con 0x7f8e5c0051c0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e68034670 con 0x7f8e5c0051c0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e74c49640 1 -- 172.21.15.69:0/736798181 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60009860 con 0x7f8e5c0051c0 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e77eaf640 1 -- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 msgr2=0x7f8e5c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e77eaf640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e5c00b9c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800a800 tx=0x7f8e5c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e77eaf640 1 -- 172.21.15.69:0/736798181 shutdown_connections 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e77eaf640 1 --2- 172.21.15.69:0/736798181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e5c00b9c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e77eaf640 1 -- 172.21.15.69:0/736798181 >> 172.21.15.69:0/736798181 conn(0x7f8e700b2790 msgr2=0x7f8e700b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e77eaf640 1 -- 172.21.15.69:0/736798181 shutdown_connections 2024-03-06T07:22:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.843+0000 7f8e77eaf640 1 -- 172.21.15.69:0/736798181 wait complete. 2024-03-06T07:22:24.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e77eaf640 1 Processor -- start 2024-03-06T07:22:24.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e77eaf640 1 -- start start 2024-03-06T07:22:24.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e77eaf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e700accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e77eaf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e700ad200 0x7f8e700aac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e77eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e700ab190 0x7f8e700cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e77eaf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e701448d0 con 0x7f8e700ab190 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e77eaf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70144750 con 0x7f8e700ad200 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e77eaf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700719a0 con 0x7f8e5c0051c0 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e75c4b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e700accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e75c4b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e700accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49896/0 (socket says 172.21.15.69:49896) 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e75c4b640 1 -- 172.21.15.69:0/2510822429 learned_addr learned my addr 172.21.15.69:0/2510822429 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.845+0000 7f8e7544a640 1 --2- 172.21.15.69:0/2510822429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e700ad200 0x7f8e700aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e7644c640 1 --2- 172.21.15.69:0/2510822429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e700ab190 0x7f8e700cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e75c4b640 1 -- 172.21.15.69:0/2510822429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e700ad200 msgr2=0x7f8e700aac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e75c4b640 1 --2- 172.21.15.69:0/2510822429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e700ad200 0x7f8e700aac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e75c4b640 1 -- 172.21.15.69:0/2510822429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e700ab190 msgr2=0x7f8e700cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e75c4b640 1 --2- 172.21.15.69:0/2510822429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e700ab190 0x7f8e700cfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e75c4b640 1 -- 172.21.15.69:0/2510822429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e700aee40 con 0x7f8e5c0051c0 2024-03-06T07:22:24.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e75c4b640 1 --2- 172.21.15.69:0/2510822429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e700accc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00f7d0 tx=0x7f8e6c00fbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e667fc640 1 -- 172.21.15.69:0/2510822429 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0050d0 con 0x7f8e5c0051c0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e667fc640 1 -- 172.21.15.69:0/2510822429 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c018070 con 0x7f8e5c0051c0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e667fc640 1 -- 172.21.15.69:0/2510822429 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c005490 con 0x7f8e5c0051c0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e77eaf640 1 -- 172.21.15.69:0/2510822429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e700af120 con 0x7f8e5c0051c0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e77eaf640 1 -- 172.21.15.69:0/2510822429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e700af6e0 con 0x7f8e5c0051c0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e667fc640 1 -- 172.21.15.69:0/2510822429 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8e6c00d730 con 0x7f8e5c0051c0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.846+0000 7f8e667fc640 1 -- 172.21.15.69:0/2510822429 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f8e6c003cd0 con 0x7f8e5c0051c0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.847+0000 7f8e77eaf640 1 -- 172.21.15.69:0/2510822429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e700aa7c0 con 0x7f8e5c0051c0 2024-03-06T07:22:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.849+0000 7f8e667fc640 1 -- 172.21.15.69:0/2510822429 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8e700722f0 con 0x7f8e5c0051c0 2024-03-06T07:22:24.854 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:24.849+0000 7f84f2041040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:22:24.896 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:24.895+0000 7f434121a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:22:24.907 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:24.906+0000 7f84f2041040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:22:24.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.922+0000 7f8e77eaf640 1 -- 172.21.15.69:0/2510822429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59", "id": 2} v 0) v1 -- 0x7f8e7004f410 con 0x7f8e5c0051c0 2024-03-06T07:22:24.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.928+0000 7f8e667fc640 1 -- 172.21.15.69:0/2510822429 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8e6c0035f0 con 0x7f8e5c0051c0 2024-03-06T07:22:24.930 INFO:teuthology.orchestra.run.smithi069.stdout:2 2024-03-06T07:22:24.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.929+0000 7f8e77eaf640 1 -- 172.21.15.69:0/2510822429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 msgr2=0x7f8e700accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.929+0000 7f8e77eaf640 1 --2- 172.21.15.69:0/2510822429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e700accc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00f7d0 tx=0x7f8e6c00fbd0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.929+0000 7f8e77eaf640 1 -- 172.21.15.69:0/2510822429 shutdown_connections 2024-03-06T07:22:24.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.929+0000 7f8e77eaf640 1 --2- 172.21.15.69:0/2510822429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e700ab190 0x7f8e700cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.929+0000 7f8e77eaf640 1 --2- 172.21.15.69:0/2510822429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e700ad200 0x7f8e700aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.929+0000 7f8e77eaf640 1 --2- 172.21.15.69:0/2510822429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e5c0051c0 0x7f8e700accc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.929+0000 7f8e77eaf640 1 -- 172.21.15.69:0/2510822429 >> 172.21.15.69:0/2510822429 conn(0x7f8e700b2790 msgr2=0x7f8e7015bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.929+0000 7f8e77eaf640 1 -- 172.21.15.69:0/2510822429 shutdown_connections 2024-03-06T07:22:24.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:24.929+0000 7f8e77eaf640 1 -- 172.21.15.69:0/2510822429 wait complete. 2024-03-06T07:22:24.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new be254672-63a5-4e78-a2f8-596fe2afa104 3 2024-03-06T07:22:24.944 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:24.942+0000 7f434121a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:22:24.955 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:24.954+0000 7f84f2041040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:22:25.000 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:24.998+0000 7f434121a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:22:25.012 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:25.010+0000 7f84f2041040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:22:25.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.040+0000 7fd7360b5640 1 Processor -- start 2024-03-06T07:22:25.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.040+0000 7fd7360b5640 1 -- start start 2024-03-06T07:22:25.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.041+0000 7fd7360b5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd73015f500 0x7fd7300ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.041+0000 7fd7360b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300cf740 0x7fd7300ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd7360b5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7300cd310 con 0x7fd7300ced80 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd7360b5640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73015f8e0 con 0x7fd73015f500 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd7360b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300c56c0 con 0x7fd7300cf740 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd727fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300cf740 0x7fd7300ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd727fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300cf740 0x7fd7300ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49912/0 (socket says 172.21.15.69:49912) 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd727fff640 1 -- 172.21.15.69:0/1298407676 learned_addr learned my addr 172.21.15.69:0/1298407676 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd72f7fe640 1 --2- 172.21.15.69:0/1298407676 >> v2:172.21.15.27:3300/0 conn(0x7fd73015f500 0x7fd7300ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd727fff640 1 -- 172.21.15.69:0/1298407676 >> v2:172.21.15.27:3300/0 conn(0x7fd73015f500 msgr2=0x7fd7300ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd727fff640 1 --2- 172.21.15.69:0/1298407676 >> v2:172.21.15.27:3300/0 conn(0x7fd73015f500 0x7fd7300ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd727fff640 1 -- 172.21.15.69:0/1298407676 >> v1:172.21.15.27:6789/0 conn(0x7fd7300ced80 legacy=0x7fd7300cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.042+0000 7fd727fff640 1 -- 172.21.15.69:0/1298407676 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7300cd5b0 con 0x7fd7300cf740 2024-03-06T07:22:25.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.043+0000 7fd727fff640 1 --2- 172.21.15.69:0/1298407676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300cf740 0x7fd7300ce840 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd718009870 tx=0x7fd71802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af264a62b924a8d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.043+0000 7fd72effd640 1 -- 172.21.15.69:0/1298407676 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd718035020 con 0x7fd7300cf740 2024-03-06T07:22:25.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.043+0000 7fd72effd640 1 -- 172.21.15.69:0/1298407676 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd718031640 con 0x7fd7300cf740 2024-03-06T07:22:25.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.043+0000 7fd72effd640 1 -- 172.21.15.69:0/1298407676 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71802d040 con 0x7fd7300cf740 2024-03-06T07:22:25.046 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:25.043+0000 7f434121a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:22:25.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.043+0000 7fd7360b5640 1 -- 172.21.15.69:0/1298407676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300cf740 msgr2=0x7fd7300ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.043+0000 7fd7360b5640 1 --2- 172.21.15.69:0/1298407676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300cf740 0x7fd7300ce840 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd718009870 tx=0x7fd71802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.044+0000 7fd7360b5640 1 -- 172.21.15.69:0/1298407676 shutdown_connections 2024-03-06T07:22:25.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.044+0000 7fd7360b5640 1 --2- 172.21.15.69:0/1298407676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300cf740 0x7fd7300ce840 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.044+0000 7fd7360b5640 1 --2- 172.21.15.69:0/1298407676 >> v2:172.21.15.27:3300/0 conn(0x7fd73015f500 0x7fd7300ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.044+0000 7fd7360b5640 1 -- 172.21.15.69:0/1298407676 >> 172.21.15.69:0/1298407676 conn(0x7fd7300b3700 msgr2=0x7fd7300b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.044+0000 7fd7360b5640 1 -- 172.21.15.69:0/1298407676 shutdown_connections 2024-03-06T07:22:25.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.044+0000 7fd7360b5640 1 -- 172.21.15.69:0/1298407676 wait complete. 2024-03-06T07:22:25.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.045+0000 7fd7360b5640 1 Processor -- start 2024-03-06T07:22:25.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.046+0000 7fd7360b5640 1 -- start start 2024-03-06T07:22:25.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd7360b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300ced80 0x7fd730156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd7360b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7300cf740 0x7fd7300c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd7360b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015f500 0x7fd730157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd7360b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300b43d0 con 0x7fd7300cf740 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd7360b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300c59a0 con 0x7fd73015f500 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd7360b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300c6e40 con 0x7fd7300ced80 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300ced80 0x7fd730156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300ced80 0x7fd730156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49918/0 (socket says 172.21.15.69:49918) 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72f7fe640 1 -- 172.21.15.69:0/1784241909 learned_addr learned my addr 172.21.15.69:0/1784241909 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72ffff640 1 --2- 172.21.15.69:0/1784241909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015f500 0x7fd730157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd727fff640 1 --2- 172.21.15.69:0/1784241909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7300cf740 0x7fd7300c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72f7fe640 1 -- 172.21.15.69:0/1784241909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015f500 msgr2=0x7fd730157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72f7fe640 1 --2- 172.21.15.69:0/1784241909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015f500 0x7fd730157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72f7fe640 1 -- 172.21.15.69:0/1784241909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7300cf740 msgr2=0x7fd7300c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72f7fe640 1 --2- 172.21.15.69:0/1784241909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7300cf740 0x7fd7300c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72f7fe640 1 -- 172.21.15.69:0/1784241909 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7300c9b40 con 0x7fd7300ced80 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72f7fe640 1 --2- 172.21.15.69:0/1784241909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300ced80 0x7fd730156c10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd72000e8a0 tx=0x7fd72000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72cff9640 1 -- 172.21.15.69:0/1784241909 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720017070 con 0x7fd7300ced80 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.047+0000 7fd72cff9640 1 -- 172.21.15.69:0/1784241909 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72000c940 con 0x7fd7300ced80 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.048+0000 7fd72cff9640 1 -- 172.21.15.69:0/1784241909 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72000cd30 con 0x7fd7300ced80 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.048+0000 7fd7360b5640 1 -- 172.21.15.69:0/1784241909 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7300b4a40 con 0x7fd7300ced80 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.048+0000 7fd7360b5640 1 -- 172.21.15.69:0/1784241909 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7300b5000 con 0x7fd7300ced80 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.048+0000 7fd72cff9640 1 -- 172.21.15.69:0/1784241909 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd72001a070 con 0x7fd7300ced80 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.048+0000 7fd72cff9640 1 -- 172.21.15.69:0/1784241909 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fd720013470 con 0x7fd7300ced80 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.049+0000 7fd7360b5640 1 -- 172.21.15.69:0/1784241909 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73009fcf0 con 0x7fd7300ced80 2024-03-06T07:22:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.051+0000 7fd72cff9640 1 -- 172.21.15.69:0/1784241909 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd720006020 con 0x7fd7300ced80 2024-03-06T07:22:25.059 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:25.058+0000 7f84f2041040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:22:25.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.122+0000 7fd7360b5640 1 -- 172.21.15.69:0/1784241909 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "be254672-63a5-4e78-a2f8-596fe2afa104", "id": 3} v 0) v1 -- 0x7fd73004f410 con 0x7fd7300ced80 2024-03-06T07:22:25.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.128+0000 7fd72cff9640 1 -- 172.21.15.69:0/1784241909 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be254672-63a5-4e78-a2f8-596fe2afa104", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd7200064d0 con 0x7fd7300ced80 2024-03-06T07:22:25.130 INFO:teuthology.orchestra.run.smithi069.stdout:3 2024-03-06T07:22:25.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.129+0000 7fd7360b5640 1 -- 172.21.15.69:0/1784241909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300ced80 msgr2=0x7fd730156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.129+0000 7fd7360b5640 1 --2- 172.21.15.69:0/1784241909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300ced80 0x7fd730156c10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd72000e8a0 tx=0x7fd72000ed60 comp rx=0 tx=0).stop 2024-03-06T07:22:25.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.129+0000 7fd7360b5640 1 -- 172.21.15.69:0/1784241909 shutdown_connections 2024-03-06T07:22:25.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.129+0000 7fd7360b5640 1 --2- 172.21.15.69:0/1784241909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015f500 0x7fd730157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.129+0000 7fd7360b5640 1 --2- 172.21.15.69:0/1784241909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7300cf740 0x7fd7300c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.129+0000 7fd7360b5640 1 --2- 172.21.15.69:0/1784241909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300ced80 0x7fd730156c10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.129+0000 7fd7360b5640 1 -- 172.21.15.69:0/1784241909 >> 172.21.15.69:0/1784241909 conn(0x7fd7300b3700 msgr2=0x7fd7300cc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.129+0000 7fd7360b5640 1 -- 172.21.15.69:0/1784241909 shutdown_connections 2024-03-06T07:22:25.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.129+0000 7fd7360b5640 1 -- 172.21.15.69:0/1784241909 wait complete. 2024-03-06T07:22:25.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9 4 2024-03-06T07:22:25.165 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:25.164+0000 7f434121a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:22:25.180 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:25.178+0000 7f84f2041040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:22:25.239 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:25.237+0000 7f434121a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:22:25.253 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:25.251+0000 7f84f2041040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:22:25.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.277+0000 7f031f884640 1 Processor -- start 2024-03-06T07:22:25.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.278+0000 7f031f884640 1 -- start start 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.279+0000 7f031f884640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03181445b0 0x7f03180b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.279+0000 7f031f884640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031806f1e0 con 0x7f031806ec40 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.279+0000 7f031f884640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031806f4c0 con 0x7f0318072130 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.279+0000 7f031f884640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0318144990 con 0x7f03181445b0 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.279+0000 7f031d620640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03181445b0 0x7f03180b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.279+0000 7f031de21640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0318072130 0x7f03180ce570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57164/0 (socket says 172.21.15.69:57164) 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.279+0000 7f031d620640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03181445b0 0x7f03180b43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49930/0 (socket says 172.21.15.69:49930) 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.279+0000 7f031de21640 1 -- 172.21.15.69:0/457111944 learned_addr learned my addr 172.21.15.69:0/457111944 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.280+0000 7f030ffff640 1 -- 172.21.15.69:0/457111944 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157291864 0 0) 0x7f031806f4c0 con 0x7f0318072130 2024-03-06T07:22:25.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.280+0000 7f030ffff640 1 -- 172.21.15.69:0/457111944 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304003520 con 0x7f0318072130 2024-03-06T07:22:25.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.280+0000 7f031d620640 1 -- 172.21.15.69:0/457111944 >> v1:172.21.15.69:6789/0 conn(0x7f0318072130 legacy=0x7f03180ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.280+0000 7f031d620640 1 -- 172.21.15.69:0/457111944 >> v1:172.21.15.27:6789/0 conn(0x7f031806ec40 legacy=0x7f0318071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.280+0000 7f031d620640 1 -- 172.21.15.69:0/457111944 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031806f7a0 con 0x7f03181445b0 2024-03-06T07:22:25.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.281+0000 7f031d620640 1 --2- 172.21.15.69:0/457111944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03181445b0 0x7f03180b43c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f03140097d0 tx=0x7f031402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ac6b1022a0431d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.281+0000 7f030ffff640 1 -- 172.21.15.69:0/457111944 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314035020 con 0x7f03181445b0 2024-03-06T07:22:25.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.281+0000 7f030ffff640 1 -- 172.21.15.69:0/457111944 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0314031890 con 0x7f03181445b0 2024-03-06T07:22:25.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.281+0000 7f030ffff640 1 -- 172.21.15.69:0/457111944 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314031c60 con 0x7f03181445b0 2024-03-06T07:22:25.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.282+0000 7f031f884640 1 -- 172.21.15.69:0/457111944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03181445b0 msgr2=0x7f03180b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.282+0000 7f031f884640 1 --2- 172.21.15.69:0/457111944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03181445b0 0x7f03180b43c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f03140097d0 tx=0x7f031402b930 comp rx=0 tx=0).stop 2024-03-06T07:22:25.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.282+0000 7f031f884640 1 -- 172.21.15.69:0/457111944 shutdown_connections 2024-03-06T07:22:25.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.282+0000 7f031f884640 1 --2- 172.21.15.69:0/457111944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03181445b0 0x7f03180b43c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.282+0000 7f031f884640 1 -- 172.21.15.69:0/457111944 >> 172.21.15.69:0/457111944 conn(0x7f03180d09c0 msgr2=0x7f03180d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.282+0000 7f031f884640 1 -- 172.21.15.69:0/457111944 shutdown_connections 2024-03-06T07:22:25.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.282+0000 7f031f884640 1 -- 172.21.15.69:0/457111944 wait complete. 2024-03-06T07:22:25.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.284+0000 7f031f884640 1 Processor -- start 2024-03-06T07:22:25.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.284+0000 7f031f884640 1 -- start start 2024-03-06T07:22:25.287 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:25.285+0000 7f434121a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:22:25.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031f884640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f031806ec40 0x7f031815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031f884640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0318072130 0x7f03180afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031f884640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03181445b0 0x7f03180a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031f884640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03180cdc70 con 0x7f03181445b0 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031f884640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03180b4900 con 0x7f0318072130 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031f884640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03180cddf0 con 0x7f031806ec40 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031ce1f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0318072130 0x7f03180afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031d620640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f031806ec40 0x7f031815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031ce1f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0318072130 0x7f03180afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44048/0 (socket says 172.21.15.69:44048) 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031ce1f640 1 -- 172.21.15.69:0/2559126320 learned_addr learned my addr 172.21.15.69:0/2559126320 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031de21640 1 --2- 172.21.15.69:0/2559126320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03181445b0 0x7f03180a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031d620640 1 -- 172.21.15.69:0/2559126320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0318072130 msgr2=0x7f03180afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031d620640 1 --2- 172.21.15.69:0/2559126320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0318072130 0x7f03180afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031d620640 1 -- 172.21.15.69:0/2559126320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03181445b0 msgr2=0x7f03180a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031d620640 1 --2- 172.21.15.69:0/2559126320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03181445b0 0x7f03180a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031d620640 1 -- 172.21.15.69:0/2559126320 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03180aeb60 con 0x7f031806ec40 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.285+0000 7f031ce1f640 1 --2- 172.21.15.69:0/2559126320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0318072130 0x7f03180afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.286+0000 7f031d620640 1 --2- 172.21.15.69:0/2559126320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f031806ec40 0x7f031815a2e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f031806c0c0 tx=0x7f03140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.286+0000 7f030dffb640 1 -- 172.21.15.69:0/2559126320 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314040070 con 0x7f031806ec40 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.286+0000 7f030dffb640 1 -- 172.21.15.69:0/2559126320 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0314030d20 con 0x7f031806ec40 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.286+0000 7f030dffb640 1 -- 172.21.15.69:0/2559126320 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314035040 con 0x7f031806ec40 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.286+0000 7f031f884640 1 -- 172.21.15.69:0/2559126320 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03180aee40 con 0x7f031806ec40 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.286+0000 7f031f884640 1 -- 172.21.15.69:0/2559126320 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03180af400 con 0x7f031806ec40 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.286+0000 7f030dffb640 1 -- 172.21.15.69:0/2559126320 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0314038020 con 0x7f031806ec40 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.286+0000 7f030dffb640 1 -- 172.21.15.69:0/2559126320 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f03140438c0 con 0x7f031806ec40 2024-03-06T07:22:25.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.287+0000 7f031f884640 1 -- 172.21.15.69:0/2559126320 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03180af840 con 0x7f031806ec40 2024-03-06T07:22:25.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.289+0000 7f030dffb640 1 -- 172.21.15.69:0/2559126320 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f031402d040 con 0x7f031806ec40 2024-03-06T07:22:25.305 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:25.303+0000 7f84f2041040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:22:25.342 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:25.340+0000 7f434121a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:22:25.357 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:25.356+0000 7f84f2041040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:22:25.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.360+0000 7f031f884640 1 -- 172.21.15.69:0/2559126320 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9", "id": 4} v 0) v1 -- 0x7f03180733d0 con 0x7f031806ec40 2024-03-06T07:22:25.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.364+0000 7f030dffb640 1 -- 172.21.15.69:0/2559126320 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f031403aba0 con 0x7f031806ec40 2024-03-06T07:22:25.366 INFO:teuthology.orchestra.run.smithi069.stdout:4 2024-03-06T07:22:25.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.365+0000 7f031f884640 1 -- 172.21.15.69:0/2559126320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f031806ec40 msgr2=0x7f031815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.365+0000 7f031f884640 1 --2- 172.21.15.69:0/2559126320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f031806ec40 0x7f031815a2e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f031806c0c0 tx=0x7f03140046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.365+0000 7f031f884640 1 -- 172.21.15.69:0/2559126320 shutdown_connections 2024-03-06T07:22:25.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.365+0000 7f031f884640 1 --2- 172.21.15.69:0/2559126320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03181445b0 0x7f03180a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.365+0000 7f031f884640 1 --2- 172.21.15.69:0/2559126320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0318072130 0x7f03180afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.365+0000 7f031f884640 1 --2- 172.21.15.69:0/2559126320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f031806ec40 0x7f031815a2e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.365+0000 7f031f884640 1 -- 172.21.15.69:0/2559126320 >> 172.21.15.69:0/2559126320 conn(0x7f03180d09c0 msgr2=0x7f031815d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.365+0000 7f031f884640 1 -- 172.21.15.69:0/2559126320 shutdown_connections 2024-03-06T07:22:25.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.365+0000 7f031f884640 1 -- 172.21.15.69:0/2559126320 wait complete. 2024-03-06T07:22:25.377 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new ef198e8f-3cec-4506-bc07-dd1a4f70aceb 5 2024-03-06T07:22:25.391 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:25.390+0000 7f434121a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:22:25.408 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:25.406+0000 7f84f2041040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:22:25.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.475+0000 7f943fab5640 1 Processor -- start 2024-03-06T07:22:25.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.476+0000 7f943fab5640 1 -- start start 2024-03-06T07:22:25.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.476+0000 7f943fab5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f9438165660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.476+0000 7f943fab5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94380b4cc0 con 0x7f94380719d0 2024-03-06T07:22:25.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.476+0000 7f943fab5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94380cdfc0 con 0x7f943806f5b0 2024-03-06T07:22:25.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.476+0000 7f943fab5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94380ce610 con 0x7f94380cfab0 2024-03-06T07:22:25.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.477+0000 7f943e052640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f9438165660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.477+0000 7f943d050640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f943806f5b0 0x7f94380b43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57176/0 (socket says 172.21.15.69:57176) 2024-03-06T07:22:25.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.477+0000 7f943d050640 1 -- 172.21.15.69:0/1569665904 learned_addr learned my addr 172.21.15.69:0/1569665904 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-06T07:22:25.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.477+0000 7f943e052640 1 -- 172.21.15.69:0/1569665904 >> v1:172.21.15.69:6789/0 conn(0x7f943806f5b0 legacy=0x7f94380b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.477+0000 7f943c84f640 1 -- 172.21.15.69:0/1569665904 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850232001 0 0) 0x7f94380cdfc0 con 0x7f943806f5b0 2024-03-06T07:22:25.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.477+0000 7f943e052640 1 -- 172.21.15.69:0/1569665904 >> v1:172.21.15.27:6789/0 conn(0x7f94380719d0 legacy=0x7f9438071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.477+0000 7f943e052640 1 -- 172.21.15.69:0/1569665904 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94380ce2a0 con 0x7f94380cfab0 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943e052640 1 --2- 172.21.15.69:0/1569665904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f9438165660 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9430009dc0 tx=0x7f943002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5b5385322387ec0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943c84f640 1 -- 172.21.15.69:0/1569665904 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430039070 con 0x7f94380cfab0 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943c84f640 1 -- 172.21.15.69:0/1569665904 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943002ccd0 con 0x7f94380cfab0 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943c84f640 1 -- 172.21.15.69:0/1569665904 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430034660 con 0x7f94380cfab0 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943fab5640 1 -- 172.21.15.69:0/1569665904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 msgr2=0x7f9438165660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943fab5640 1 --2- 172.21.15.69:0/1569665904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f9438165660 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9430009dc0 tx=0x7f943002bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943fab5640 1 -- 172.21.15.69:0/1569665904 shutdown_connections 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943fab5640 1 --2- 172.21.15.69:0/1569665904 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f9438165660 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943fab5640 1 -- 172.21.15.69:0/1569665904 >> 172.21.15.69:0/1569665904 conn(0x7f94380d09c0 msgr2=0x7f94380d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943fab5640 1 -- 172.21.15.69:0/1569665904 shutdown_connections 2024-03-06T07:22:25.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.478+0000 7f943fab5640 1 -- 172.21.15.69:0/1569665904 wait complete. 2024-03-06T07:22:25.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.479+0000 7f943fab5640 1 Processor -- start 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.479+0000 7f943fab5640 1 -- start start 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943fab5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f943806f5b0 0x7f943815a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943fab5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94380719d0 0x7f9438139d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943fab5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f943813a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943fab5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94381447c0 con 0x7f94380719d0 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943fab5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9438144640 con 0x7f943806f5b0 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943fab5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9438165ba0 con 0x7f94380cfab0 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943e052640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f943813a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943d851640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f943806f5b0 0x7f943815a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943d851640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f943806f5b0 0x7f943815a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44064/0 (socket says 172.21.15.69:44064) 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943d851640 1 -- 172.21.15.69:0/660585101 learned_addr learned my addr 172.21.15.69:0/660585101 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943e052640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f943813a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49964/0 (socket says 172.21.15.69:49964) 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.480+0000 7f943d050640 1 --2- 172.21.15.69:0/660585101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94380719d0 0x7f9438139d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f943e052640 1 -- 172.21.15.69:0/660585101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f943806f5b0 msgr2=0x7f943815a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f943e052640 1 --2- 172.21.15.69:0/660585101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f943806f5b0 0x7f943815a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f943e052640 1 -- 172.21.15.69:0/660585101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94380719d0 msgr2=0x7f9438139d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f943e052640 1 --2- 172.21.15.69:0/660585101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94380719d0 0x7f9438139d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f943e052640 1 -- 172.21.15.69:0/660585101 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94381f8340 con 0x7f94380cfab0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f943e052640 1 --2- 172.21.15.69:0/660585101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f943813a240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9430033000 tx=0x7f9430004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f942e7fc640 1 -- 172.21.15.69:0/660585101 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430039050 con 0x7f94380cfab0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f942e7fc640 1 -- 172.21.15.69:0/660585101 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943003a040 con 0x7f94380cfab0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f942e7fc640 1 -- 172.21.15.69:0/660585101 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943003e610 con 0x7f94380cfab0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f943fab5640 1 -- 172.21.15.69:0/660585101 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943815a920 con 0x7f94380cfab0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f943fab5640 1 -- 172.21.15.69:0/660585101 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94380b3df0 con 0x7f94380cfab0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f942e7fc640 1 -- 172.21.15.69:0/660585101 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9430047020 con 0x7f94380cfab0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.481+0000 7f942e7fc640 1 -- 172.21.15.69:0/660585101 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f9430040bb0 con 0x7f94380cfab0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.482+0000 7f943fab5640 1 -- 172.21.15.69:0/660585101 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9438073380 con 0x7f94380cfab0 2024-03-06T07:22:25.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.486+0000 7f942e7fc640 1 -- 172.21.15.69:0/660585101 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9430040630 con 0x7f94380cfab0 2024-03-06T07:22:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.558+0000 7f943fab5640 1 -- 172.21.15.69:0/660585101 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ef198e8f-3cec-4506-bc07-dd1a4f70aceb", "id": 5} v 0) v1 -- 0x7f94380cd110 con 0x7f94380cfab0 2024-03-06T07:22:25.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.563+0000 7f942e7fc640 1 -- 172.21.15.69:0/660585101 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ef198e8f-3cec-4506-bc07-dd1a4f70aceb", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9430040810 con 0x7f94380cfab0 2024-03-06T07:22:25.565 INFO:teuthology.orchestra.run.smithi069.stdout:5 2024-03-06T07:22:25.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.564+0000 7f943fab5640 1 -- 172.21.15.69:0/660585101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 msgr2=0x7f943813a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.564+0000 7f943fab5640 1 --2- 172.21.15.69:0/660585101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f943813a240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9430033000 tx=0x7f9430004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:25.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.564+0000 7f943fab5640 1 -- 172.21.15.69:0/660585101 shutdown_connections 2024-03-06T07:22:25.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.564+0000 7f943fab5640 1 --2- 172.21.15.69:0/660585101 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94380cfab0 0x7f943813a240 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.564+0000 7f943fab5640 1 --2- 172.21.15.69:0/660585101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94380719d0 0x7f9438139d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.564+0000 7f943fab5640 1 --2- 172.21.15.69:0/660585101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f943806f5b0 0x7f943815a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.564+0000 7f943fab5640 1 -- 172.21.15.69:0/660585101 >> 172.21.15.69:0/660585101 conn(0x7f94380d09c0 msgr2=0x7f943806df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.564+0000 7f943fab5640 1 -- 172.21.15.69:0/660585101 shutdown_connections 2024-03-06T07:22:25.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.564+0000 7f943fab5640 1 -- 172.21.15.69:0/660585101 wait complete. 2024-03-06T07:22:25.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new 93b9645a-7585-49e2-9c6d-bc4d715946d6 6 2024-03-06T07:22:25.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.715+0000 7fd21f77a640 1 Processor -- start 2024-03-06T07:22:25.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21f77a640 1 -- start start 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21f77a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2180b2130 0x7fd2180d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21f77a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd2180d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21f77a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2180cf490 con 0x7fd2180b2800 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21f77a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2180cf9c0 con 0x7fd2180b2130 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21f77a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180cfca0 con 0x7fd2180b2d50 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21cd15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd2180d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21d516640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2180b2130 0x7fd2180d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21cd15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd2180d0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:49978/0 (socket says 172.21.15.69:49978) 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.716+0000 7fd21cd15640 1 -- 172.21.15.69:0/3947925871 learned_addr learned my addr 172.21.15.69:0/3947925871 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.717+0000 7fd21cd15640 1 -- 172.21.15.69:0/3947925871 >> v2:172.21.15.69:3300/0 conn(0x7fd2180b2130 msgr2=0x7fd2180d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.717+0000 7fd21cd15640 1 --2- 172.21.15.69:0/3947925871 >> v2:172.21.15.69:3300/0 conn(0x7fd2180b2130 0x7fd2180d0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.717+0000 7fd21cd15640 1 -- 172.21.15.69:0/3947925871 >> v1:172.21.15.27:6789/0 conn(0x7fd2180b2800 legacy=0x7fd2180cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.717+0000 7fd21cd15640 1 -- 172.21.15.69:0/3947925871 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180cec90 con 0x7fd2180b2d50 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.717+0000 7fd21cd15640 1 --2- 172.21.15.69:0/3947925871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd2180d0660 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd208009870 tx=0x7fd20802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c34d60be0ce289be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.717+0000 7fd20ffff640 1 -- 172.21.15.69:0/3947925871 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208034070 con 0x7fd2180b2d50 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd20ffff640 1 -- 172.21.15.69:0/3947925871 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd208030840 con 0x7fd2180b2d50 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd20ffff640 1 -- 172.21.15.69:0/3947925871 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208030c10 con 0x7fd2180b2d50 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd21f77a640 1 -- 172.21.15.69:0/3947925871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 msgr2=0x7fd2180d0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd21f77a640 1 --2- 172.21.15.69:0/3947925871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd2180d0660 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd208009870 tx=0x7fd20802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd21f77a640 1 -- 172.21.15.69:0/3947925871 shutdown_connections 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd21f77a640 1 --2- 172.21.15.69:0/3947925871 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd2180d0660 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd21f77a640 1 --2- 172.21.15.69:0/3947925871 >> v2:172.21.15.69:3300/0 conn(0x7fd2180b2130 0x7fd2180d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd21f77a640 1 -- 172.21.15.69:0/3947925871 >> 172.21.15.69:0/3947925871 conn(0x7fd2180b36d0 msgr2=0x7fd2180b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd21f77a640 1 -- 172.21.15.69:0/3947925871 shutdown_connections 2024-03-06T07:22:25.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.718+0000 7fd21f77a640 1 -- 172.21.15.69:0/3947925871 wait complete. 2024-03-06T07:22:25.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.720+0000 7fd21f77a640 1 Processor -- start 2024-03-06T07:22:25.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.720+0000 7fd21f77a640 1 -- start start 2024-03-06T07:22:25.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.721+0000 7fd21f77a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2180b2130 0x7fd21814bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.721+0000 7fd21f77a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180b2800 0x7fd21806f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.721+0000 7fd21f77a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd21806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.721+0000 7fd21f77a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180b2510 con 0x7fd2180b2800 2024-03-06T07:22:25.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.721+0000 7fd21f77a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180c66a0 con 0x7fd2180b2130 2024-03-06T07:22:25.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.721+0000 7fd21d516640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2180b2130 0x7fd21814bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21d516640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2180b2130 0x7fd21814bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44090/0 (socket says 172.21.15.69:44090) 2024-03-06T07:22:25.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.721+0000 7fd21f77a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180cd9b0 con 0x7fd2180b2d50 2024-03-06T07:22:25.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21d516640 1 -- 172.21.15.69:0/1485625579 learned_addr learned my addr 172.21.15.69:0/1485625579 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21dd17640 1 --2- 172.21.15.69:0/1485625579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd21806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21cd15640 1 --2- 172.21.15.69:0/1485625579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180b2800 0x7fd21806f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21dd17640 1 -- 172.21.15.69:0/1485625579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2180b2130 msgr2=0x7fd21814bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21dd17640 1 --2- 172.21.15.69:0/1485625579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2180b2130 0x7fd21814bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21dd17640 1 -- 172.21.15.69:0/1485625579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180b2800 msgr2=0x7fd21806f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21dd17640 1 --2- 172.21.15.69:0/1485625579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180b2800 0x7fd21806f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21dd17640 1 -- 172.21.15.69:0/1485625579 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180b0630 con 0x7fd2180b2d50 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21dd17640 1 --2- 172.21.15.69:0/1485625579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd21806f7d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd21000c6c0 tx=0x7fd21000cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd20dffb640 1 -- 172.21.15.69:0/1485625579 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210003b70 con 0x7fd2180b2d50 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd20dffb640 1 -- 172.21.15.69:0/1485625579 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd210003d10 con 0x7fd2180b2d50 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd20dffb640 1 -- 172.21.15.69:0/1485625579 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2100052f0 con 0x7fd2180b2d50 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21f77a640 1 -- 172.21.15.69:0/1485625579 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2180b08a0 con 0x7fd2180b2d50 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd20dffb640 1 -- 172.21.15.69:0/1485625579 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd210005490 con 0x7fd2180b2d50 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.722+0000 7fd21f77a640 1 -- 172.21.15.69:0/1485625579 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2180b0e60 con 0x7fd2180b2d50 2024-03-06T07:22:25.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.723+0000 7fd20dffb640 1 -- 172.21.15.69:0/1485625579 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fd2100124c0 con 0x7fd2180b2d50 2024-03-06T07:22:25.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.723+0000 7fd21f77a640 1 -- 172.21.15.69:0/1485625579 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21809fcc0 con 0x7fd2180b2d50 2024-03-06T07:22:25.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.725+0000 7fd20dffb640 1 -- 172.21.15.69:0/1485625579 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd21000a640 con 0x7fd2180b2d50 2024-03-06T07:22:25.790 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:25.789+0000 7f434121a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:22:25.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.797+0000 7fd21f77a640 1 -- 172.21.15.69:0/1485625579 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "93b9645a-7585-49e2-9c6d-bc4d715946d6", "id": 6} v 0) v1 -- 0x7fd21804f410 con 0x7fd2180b2d50 2024-03-06T07:22:25.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.803+0000 7fd20dffb640 1 -- 172.21.15.69:0/1485625579 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "93b9645a-7585-49e2-9c6d-bc4d715946d6", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd21000a820 con 0x7fd2180b2d50 2024-03-06T07:22:25.805 INFO:teuthology.orchestra.run.smithi069.stdout:6 2024-03-06T07:22:25.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.804+0000 7fd21f77a640 1 -- 172.21.15.69:0/1485625579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 msgr2=0x7fd21806f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.804+0000 7fd21f77a640 1 --2- 172.21.15.69:0/1485625579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd21806f7d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd21000c6c0 tx=0x7fd21000cb80 comp rx=0 tx=0).stop 2024-03-06T07:22:25.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.804+0000 7fd21f77a640 1 -- 172.21.15.69:0/1485625579 shutdown_connections 2024-03-06T07:22:25.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.804+0000 7fd21f77a640 1 --2- 172.21.15.69:0/1485625579 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2180b2d50 0x7fd21806f7d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.804+0000 7fd21f77a640 1 --2- 172.21.15.69:0/1485625579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2180b2800 0x7fd21806f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.804+0000 7fd21f77a640 1 --2- 172.21.15.69:0/1485625579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2180b2130 0x7fd21814bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.804+0000 7fd21f77a640 1 -- 172.21.15.69:0/1485625579 >> 172.21.15.69:0/1485625579 conn(0x7fd2180b36d0 msgr2=0x7fd2180c7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.804+0000 7fd21f77a640 1 -- 172.21.15.69:0/1485625579 shutdown_connections 2024-03-06T07:22:25.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.804+0000 7fd21f77a640 1 -- 172.21.15.69:0/1485625579 wait complete. 2024-03-06T07:22:25.807 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:25.804+0000 7f84f2041040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:22:25.816 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd new f0686243-4ac5-48eb-97aa-b9b9345b3615 7 2024-03-06T07:22:25.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.914+0000 7ff9dcbac640 1 Processor -- start 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.914+0000 7ff9dcbac640 1 -- start start 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9dcbac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9d806f5b0 0x7ff9d80b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9dcbac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d80ceaa0 0x7ff9d80cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9dcbac640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d80b4cc0 con 0x7ff9d80719d0 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9dcbac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d80ce610 con 0x7ff9d806f5b0 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9dcbac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80cdfc0 con 0x7ff9d80ceaa0 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9d659c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff9d80719d0 0x7ff9d8071db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57182/0 (socket says 172.21.15.69:57182) 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9d6d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d80ceaa0 0x7ff9d80cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9d659c640 1 -- 172.21.15.69:0/1058069757 learned_addr learned my addr 172.21.15.69:0/1058069757 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9cffff640 1 --2- 172.21.15.69:0/1058069757 >> v2:172.21.15.27:3300/0 conn(0x7ff9d806f5b0 0x7ff9d80b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9d5d9b640 1 -- 172.21.15.69:0/1058069757 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357689995 0 0) 0x7ff9d80b4cc0 con 0x7ff9d80719d0 2024-03-06T07:22:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9d5d9b640 1 -- 172.21.15.69:0/1058069757 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c4003520 con 0x7ff9d80719d0 2024-03-06T07:22:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9d6d9d640 1 -- 172.21.15.69:0/1058069757 >> v2:172.21.15.27:3300/0 conn(0x7ff9d806f5b0 msgr2=0x7ff9d80b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9d6d9d640 1 --2- 172.21.15.69:0/1058069757 >> v2:172.21.15.27:3300/0 conn(0x7ff9d806f5b0 0x7ff9d80b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9d6d9d640 1 -- 172.21.15.69:0/1058069757 >> v1:172.21.15.69:6789/0 conn(0x7ff9d80719d0 legacy=0x7ff9d8071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.915+0000 7ff9d6d9d640 1 -- 172.21.15.69:0/1058069757 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80ce2a0 con 0x7ff9d80ceaa0 2024-03-06T07:22:25.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9d6d9d640 1 --2- 172.21.15.69:0/1058069757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d80ceaa0 0x7ff9d80cef30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009d90 tx=0x7ff9d002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32b9862a1948fd78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9d5d9b640 1 -- 172.21.15.69:0/1058069757 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0039070 con 0x7ff9d80ceaa0 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9d5d9b640 1 -- 172.21.15.69:0/1058069757 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d002cc50 con 0x7ff9d80ceaa0 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9d5d9b640 1 -- 172.21.15.69:0/1058069757 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0034440 con 0x7ff9d80ceaa0 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9dcbac640 1 -- 172.21.15.69:0/1058069757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d80ceaa0 msgr2=0x7ff9d80cef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9dcbac640 1 --2- 172.21.15.69:0/1058069757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d80ceaa0 0x7ff9d80cef30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009d90 tx=0x7ff9d002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9dcbac640 1 -- 172.21.15.69:0/1058069757 shutdown_connections 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9dcbac640 1 --2- 172.21.15.69:0/1058069757 >> v2:172.21.15.27:3300/0 conn(0x7ff9d806f5b0 0x7ff9d80b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9dcbac640 1 --2- 172.21.15.69:0/1058069757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d80ceaa0 0x7ff9d80cef30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9dcbac640 1 -- 172.21.15.69:0/1058069757 >> 172.21.15.69:0/1058069757 conn(0x7ff9d80d09c0 msgr2=0x7ff9d80d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9dcbac640 1 -- 172.21.15.69:0/1058069757 shutdown_connections 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.917+0000 7ff9dcbac640 1 -- 172.21.15.69:0/1058069757 wait complete. 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9dcbac640 1 Processor -- start 2024-03-06T07:22:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9dcbac640 1 -- start start 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9dcbac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d806f5b0 0x7ff9d8141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9dcbac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d80719d0 0x7ff9d813cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9dcbac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d80ceaa0 0x7ff9d813d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9dcbac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80cdd60 con 0x7ff9d80719d0 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9dcbac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80cdbe0 con 0x7ff9d80ceaa0 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9dcbac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d8144640 con 0x7ff9d806f5b0 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9d659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d806f5b0 0x7ff9d8141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9d659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d806f5b0 0x7ff9d8141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:50008/0 (socket says 172.21.15.69:50008) 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9d659c640 1 -- 172.21.15.69:0/186299128 learned_addr learned my addr 172.21.15.69:0/186299128 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-03-06T07:22:25.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.919+0000 7ff9d6d9d640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d80ceaa0 0x7ff9d813d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9cffff640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d80719d0 0x7ff9d813cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9d659c640 1 -- 172.21.15.69:0/186299128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d80ceaa0 msgr2=0x7ff9d813d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9d659c640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d80ceaa0 0x7ff9d813d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9d659c640 1 -- 172.21.15.69:0/186299128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d80719d0 msgr2=0x7ff9d813cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9d659c640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d80719d0 0x7ff9d813cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9d659c640 1 -- 172.21.15.69:0/186299128 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d815afb0 con 0x7ff9d806f5b0 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9d6d9d640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d80ceaa0 0x7ff9d813d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9d659c640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d806f5b0 0x7ff9d8141d70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800c430 tx=0x7ff9c800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9ceffd640 1 -- 172.21.15.69:0/186299128 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8017070 con 0x7ff9d806f5b0 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9ceffd640 1 -- 172.21.15.69:0/186299128 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8005ce0 con 0x7ff9d806f5b0 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9dcbac640 1 -- 172.21.15.69:0/186299128 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d815b290 con 0x7ff9d806f5b0 2024-03-06T07:22:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9ceffd640 1 -- 172.21.15.69:0/186299128 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80072f0 con 0x7ff9d806f5b0 2024-03-06T07:22:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9dcbac640 1 -- 172.21.15.69:0/186299128 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d815b850 con 0x7ff9d806f5b0 2024-03-06T07:22:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9ceffd640 1 -- 172.21.15.69:0/186299128 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff9c801a070 con 0x7ff9d806f5b0 2024-03-06T07:22:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.920+0000 7ff9ceffd640 1 -- 172.21.15.69:0/186299128 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7ff9c8012a60 con 0x7ff9d806f5b0 2024-03-06T07:22:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.921+0000 7ff9dcbac640 1 -- 172.21.15.69:0/186299128 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d8073380 con 0x7ff9d806f5b0 2024-03-06T07:22:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.923+0000 7ff9ceffd640 1 -- 172.21.15.69:0/186299128 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff9c8021020 con 0x7ff9d806f5b0 2024-03-06T07:22:25.932 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:25.931+0000 7f434121a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:22:25.948 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:25.946+0000 7f84f2041040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:22:25.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:25.995+0000 7ff9dcbac640 1 -- 172.21.15.69:0/186299128 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f0686243-4ac5-48eb-97aa-b9b9345b3615", "id": 7} v 0) v1 -- 0x7ff9d806c990 con 0x7ff9d806f5b0 2024-03-06T07:22:26.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.001+0000 7ff9ceffd640 1 -- 172.21.15.69:0/186299128 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f0686243-4ac5-48eb-97aa-b9b9345b3615", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff9c8007620 con 0x7ff9d806f5b0 2024-03-06T07:22:26.003 INFO:teuthology.orchestra.run.smithi069.stdout:7 2024-03-06T07:22:26.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.002+0000 7ff9dcbac640 1 -- 172.21.15.69:0/186299128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d806f5b0 msgr2=0x7ff9d8141d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.002+0000 7ff9dcbac640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d806f5b0 0x7ff9d8141d70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800c430 tx=0x7ff9c800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.002+0000 7ff9dcbac640 1 -- 172.21.15.69:0/186299128 shutdown_connections 2024-03-06T07:22:26.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.002+0000 7ff9dcbac640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9d80ceaa0 0x7ff9d813d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.002+0000 7ff9dcbac640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9d80719d0 0x7ff9d813cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.002+0000 7ff9dcbac640 1 --2- 172.21.15.69:0/186299128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9d806f5b0 0x7ff9d8141d70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.002+0000 7ff9dcbac640 1 -- 172.21.15.69:0/186299128 >> 172.21.15.69:0/186299128 conn(0x7ff9d80d09c0 msgr2=0x7ff9d8141300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.002+0000 7ff9dcbac640 1 -- 172.21.15.69:0/186299128 shutdown_connections 2024-03-06T07:22:26.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:22:26.002+0000 7ff9dcbac640 1 -- 172.21.15.69:0/186299128 wait complete. 2024-03-06T07:22:26.015 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:22:26.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:22:26.017 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:22:26.017 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:22:26.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:22:26.019 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:22:26.019 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:22:26.020 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:22:26.022 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:22:26.022 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:22:26.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:22:26.024 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:22:26.024 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:22:26.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:22:26.026 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:22:26.027 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:22:26.027 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:22:26.029 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:22:26.029 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:22:26.029 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:22:26.031 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:22:26.031 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:22:26.031 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:22:26.033 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:22:26.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:26.068 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:26.067+0000 7f434121a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:22:26.091 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:26.089+0000 7f84f2041040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:22:26.139 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:26.137+0000 7f434121a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:22:26.148 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:26.146+0000 7f84f2041040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:22:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.160+0000 7f7a47dbc640 1 Processor -- start 2024-03-06T07:22:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.160+0000 7f7a47dbc640 1 -- start start 2024-03-06T07:22:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.161+0000 7f7a47dbc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a400b4260 0x7f7a400cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.161+0000 7f7a47dbc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7a400d06c0 0x7f7a400b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.161+0000 7f7a47dbc640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a400b0dd0 con 0x7f7a400d02e0 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.161+0000 7f7a47dbc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a400b0b80 con 0x7f7a400d06c0 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.161+0000 7f7a47dbc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a400aeec0 con 0x7f7a400b4260 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.161+0000 7f7a45b58640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7a400d02e0 0x7f7a400b1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53326/0 (socket says 172.21.15.27:53326) 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.161+0000 7f7a45b58640 1 -- 172.21.15.27:0/2759435502 learned_addr learned my addr 172.21.15.27:0/2759435502 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.161+0000 7f7a46359640 1 --2- 172.21.15.27:0/2759435502 >> v2:172.21.15.27:3300/0 conn(0x7f7a400d06c0 0x7f7a400b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.161+0000 7f7a45357640 1 --2- 172.21.15.27:0/2759435502 >> v2:172.21.15.69:3300/0 conn(0x7f7a400b4260 0x7f7a400cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.162+0000 7f7a44b56640 1 -- 172.21.15.27:0/2759435502 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073315394 0 0) 0x7f7a400b0dd0 con 0x7f7a400d02e0 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.162+0000 7f7a45357640 1 -- 172.21.15.27:0/2759435502 >> v2:172.21.15.27:3300/0 conn(0x7f7a400d06c0 msgr2=0x7f7a400b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.162+0000 7f7a45357640 1 --2- 172.21.15.27:0/2759435502 >> v2:172.21.15.27:3300/0 conn(0x7f7a400d06c0 0x7f7a400b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.162+0000 7f7a45357640 1 -- 172.21.15.27:0/2759435502 >> v1:172.21.15.27:6789/0 conn(0x7f7a400d02e0 legacy=0x7f7a400b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.162+0000 7f7a45357640 1 -- 172.21.15.27:0/2759435502 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400b10b0 con 0x7f7a400b4260 2024-03-06T07:22:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a45357640 1 --2- 172.21.15.27:0/2759435502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a400b4260 0x7f7a400cfa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7a40067cb0 tx=0x7f7a3c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6ed190374597276 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a44b56640 1 -- 172.21.15.27:0/2759435502 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c034070 con 0x7f7a400b4260 2024-03-06T07:22:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a44b56640 1 -- 172.21.15.27:0/2759435502 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c030610 con 0x7f7a400b4260 2024-03-06T07:22:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a44b56640 1 -- 172.21.15.27:0/2759435502 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c0309e0 con 0x7f7a400b4260 2024-03-06T07:22:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a47dbc640 1 -- 172.21.15.27:0/2759435502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a400b4260 msgr2=0x7f7a400cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a47dbc640 1 --2- 172.21.15.27:0/2759435502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a400b4260 0x7f7a400cfa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7a40067cb0 tx=0x7f7a3c02b960 comp rx=0 tx=0).stop 2024-03-06T07:22:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a47dbc640 1 -- 172.21.15.27:0/2759435502 shutdown_connections 2024-03-06T07:22:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a47dbc640 1 --2- 172.21.15.27:0/2759435502 >> v2:172.21.15.27:3300/0 conn(0x7f7a400d06c0 0x7f7a400b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a47dbc640 1 --2- 172.21.15.27:0/2759435502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a400b4260 0x7f7a400cfa00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.163+0000 7f7a47dbc640 1 -- 172.21.15.27:0/2759435502 >> 172.21.15.27:0/2759435502 conn(0x7f7a400b36d0 msgr2=0x7f7a400b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.164+0000 7f7a47dbc640 1 -- 172.21.15.27:0/2759435502 shutdown_connections 2024-03-06T07:22:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.164+0000 7f7a47dbc640 1 -- 172.21.15.27:0/2759435502 wait complete. 2024-03-06T07:22:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.165+0000 7f7a47dbc640 1 Processor -- start 2024-03-06T07:22:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.165+0000 7f7a47dbc640 1 -- start start 2024-03-06T07:22:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.165+0000 7f7a47dbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a400b4260 0x7f7a401f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a47dbc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a400d02e0 0x7f7a40152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a47dbc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a400d06c0 0x7f7a40153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a47dbc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400b22d0 con 0x7f7a400d06c0 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a47dbc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400b2150 con 0x7f7a400d02e0 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a47dbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400af350 con 0x7f7a400b4260 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a46359640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a400d06c0 0x7f7a40153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a46359640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a400d06c0 0x7f7a40153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59684/0 (socket says 172.21.15.27:59684) 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a46359640 1 -- 172.21.15.27:0/1380245916 learned_addr learned my addr 172.21.15.27:0/1380245916 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a45357640 1 --2- 172.21.15.27:0/1380245916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a400d02e0 0x7f7a40152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a45b58640 1 --2- 172.21.15.27:0/1380245916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a400b4260 0x7f7a401f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a46359640 1 -- 172.21.15.27:0/1380245916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a400b4260 msgr2=0x7f7a401f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a46359640 1 --2- 172.21.15.27:0/1380245916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a400b4260 0x7f7a401f3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a46359640 1 -- 172.21.15.27:0/1380245916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a400d02e0 msgr2=0x7f7a40152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a46359640 1 --2- 172.21.15.27:0/1380245916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a400d02e0 0x7f7a40152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a46359640 1 -- 172.21.15.27:0/1380245916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a401562f0 con 0x7f7a400d06c0 2024-03-06T07:22:26.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a46359640 1 --2- 172.21.15.27:0/1380245916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a400d06c0 0x7f7a40153440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800efc0 tx=0x7f7a3800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a367fc640 1 -- 172.21.15.27:0/1380245916 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38018020 con 0x7f7a400d06c0 2024-03-06T07:22:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a367fc640 1 -- 172.21.15.27:0/1380245916 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a38014480 con 0x7f7a400d06c0 2024-03-06T07:22:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a47dbc640 1 -- 172.21.15.27:0/1380245916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a40153980 con 0x7f7a400d06c0 2024-03-06T07:22:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a367fc640 1 -- 172.21.15.27:0/1380245916 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38014870 con 0x7f7a400d06c0 2024-03-06T07:22:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.166+0000 7f7a47dbc640 1 -- 172.21.15.27:0/1380245916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a40156010 con 0x7f7a400d06c0 2024-03-06T07:22:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.167+0000 7f7a367fc640 1 -- 172.21.15.27:0/1380245916 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7a38005d10 con 0x7f7a400d06c0 2024-03-06T07:22:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.167+0000 7f7a367fc640 1 -- 172.21.15.27:0/1380245916 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7a38013ab0 con 0x7f7a400d06c0 2024-03-06T07:22:26.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.167+0000 7f7a47dbc640 1 -- 172.21.15.27:0/1380245916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a40068700 con 0x7f7a400d06c0 2024-03-06T07:22:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.171+0000 7f7a367fc640 1 -- 172.21.15.27:0/1380245916 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7a38022070 con 0x7f7a400d06c0 2024-03-06T07:22:26.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.233+0000 7f7a47dbc640 1 -- 172.21.15.27:0/1380245916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a400709b0 con 0x7f7a400d06c0 2024-03-06T07:22:26.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.234+0000 7f7a367fc640 1 -- 172.21.15.27:0/1380245916 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f7a38010040 con 0x7f7a400d06c0 2024-03-06T07:22:26.236 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:26.236 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":10,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:25.997497+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.235+0000 7f7a47dbc640 1 -- 172.21.15.27:0/1380245916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a400d06c0 msgr2=0x7f7a40153440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.235+0000 7f7a47dbc640 1 --2- 172.21.15.27:0/1380245916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a400d06c0 0x7f7a40153440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800efc0 tx=0x7f7a3800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.235+0000 7f7a47dbc640 1 -- 172.21.15.27:0/1380245916 shutdown_connections 2024-03-06T07:22:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.235+0000 7f7a47dbc640 1 --2- 172.21.15.27:0/1380245916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a400d06c0 0x7f7a40153440 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.235+0000 7f7a47dbc640 1 --2- 172.21.15.27:0/1380245916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a400d02e0 0x7f7a40152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.235+0000 7f7a47dbc640 1 --2- 172.21.15.27:0/1380245916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a400b4260 0x7f7a401f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.235+0000 7f7a47dbc640 1 -- 172.21.15.27:0/1380245916 >> 172.21.15.27:0/1380245916 conn(0x7f7a400b36d0 msgr2=0x7f7a40151db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.235+0000 7f7a47dbc640 1 -- 172.21.15.27:0/1380245916 shutdown_connections 2024-03-06T07:22:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:26.235+0000 7f7a47dbc640 1 -- 172.21.15.27:0/1380245916 wait complete. 2024-03-06T07:22:26.247 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:22:26.247 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:22:26.259 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:26.257+0000 7f84f2041040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:22:26.260 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:26.259+0000 7f434121a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:22:26.352 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:26.350+0000 7f84f2041040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:22:26.360 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:26.359+0000 7f434121a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:22:26.505 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:26.503+0000 7f84f2041040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:22:26.571 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:26.545+0000 7f84f2041040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:22:26.571 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:26.521+0000 7f434121a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:22:26.572 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:26.565+0000 7f434121a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:22:26.630 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-03-06T07:22:26.628+0000 7fb11a72a640 -1 Falling back to public interface 2024-03-06T07:22:26.631 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-03-06T07:22:26.629+0000 7fba2e4bd640 -1 Falling back to public interface 2024-03-06T07:22:26.632 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-03-06T07:22:26.630+0000 7f1ecab9a640 -1 Falling back to public interface 2024-03-06T07:22:26.643 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-03-06T07:22:26.641+0000 7f0c5201b640 -1 Falling back to public interface 2024-03-06T07:22:26.644 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-03-06T07:22:26.642+0000 7f692a8d9640 -1 Falling back to public interface 2024-03-06T07:22:26.651 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-03-06T07:22:26.650+0000 7f4edb411640 -1 Falling back to public interface 2024-03-06T07:22:26.651 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-03-06T07:22:26.650+0000 7f989e949640 -1 Falling back to public interface 2024-03-06T07:22:26.658 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-03-06T07:22:26.657+0000 7fb9e676f640 -1 Falling back to public interface 2024-03-06T07:22:26.701 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:26.700+0000 7f84f2041040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:22:26.733 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:26.731+0000 7f434121a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:22:27.074 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:27.167 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.166+0000 7f43cd0e0640 1 Processor -- start 2024-03-06T07:22:27.167 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.166+0000 7f43cd0e0640 1 -- start start 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.166+0000 7f43cd0e0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8147840 0x7f43c8072460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43cd0e0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f43c8062120 0x7f43c815e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43cd0e0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c815ee30 con 0x7f43c815e380 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43cd0e0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c815f3f0 con 0x7f43c8062120 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43cd0e0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c815f110 con 0x7f43c8147840 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c6d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8147840 0x7f43c8072460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c6d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8147840 0x7f43c8072460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47476/0 (socket says 172.21.15.27:47476) 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c6d9d640 1 -- 172.21.15.27:0/3144850623 learned_addr learned my addr 172.21.15.27:0/3144850623 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c759e640 1 --2- 172.21.15.27:0/3144850623 >> v2:172.21.15.69:3300/0 conn(0x7f43c8062120 0x7f43c815e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.169 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c6d9d640 1 -- 172.21.15.27:0/3144850623 >> v2:172.21.15.69:3300/0 conn(0x7f43c8062120 msgr2=0x7f43c815e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.170 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c6d9d640 1 --2- 172.21.15.27:0/3144850623 >> v2:172.21.15.69:3300/0 conn(0x7f43c8062120 0x7f43c815e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.170 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c6d9d640 1 -- 172.21.15.27:0/3144850623 >> v1:172.21.15.69:6789/0 conn(0x7f43c815e380 legacy=0x7f43c8061a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.170 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c5d9b640 1 -- 172.21.15.27:0/3144850623 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641696385 0 0) 0x7f43c815ee30 con 0x7f43c815e380 2024-03-06T07:22:27.170 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c6d9d640 1 -- 172.21.15.27:0/3144850623 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c8071c60 con 0x7f43c8147840 2024-03-06T07:22:27.170 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.167+0000 7f43c759e640 1 --2- 172.21.15.27:0/3144850623 >> v2:172.21.15.69:3300/0 conn(0x7f43c8062120 0x7f43c815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:27.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.168+0000 7f43c6d9d640 1 --2- 172.21.15.27:0/3144850623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8147840 0x7f43c8072460 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009870 tx=0x7f43b802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e219e93f1ddc6345 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.168+0000 7f43c5d9b640 1 -- 172.21.15.27:0/3144850623 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b8035020 con 0x7f43c8147840 2024-03-06T07:22:27.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.168+0000 7f43c5d9b640 1 -- 172.21.15.27:0/3144850623 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b8031640 con 0x7f43c8147840 2024-03-06T07:22:27.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.168+0000 7f43c5d9b640 1 -- 172.21.15.27:0/3144850623 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b802d040 con 0x7f43c8147840 2024-03-06T07:22:27.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.169+0000 7f43cd0e0640 1 -- 172.21.15.27:0/3144850623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8147840 msgr2=0x7f43c8072460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.169+0000 7f43cd0e0640 1 --2- 172.21.15.27:0/3144850623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8147840 0x7f43c8072460 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009870 tx=0x7f43b802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.169+0000 7f43cd0e0640 1 -- 172.21.15.27:0/3144850623 shutdown_connections 2024-03-06T07:22:27.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.169+0000 7f43cd0e0640 1 --2- 172.21.15.27:0/3144850623 >> v2:172.21.15.69:3300/0 conn(0x7f43c8062120 0x7f43c815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.172 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.169+0000 7f43cd0e0640 1 --2- 172.21.15.27:0/3144850623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8147840 0x7f43c8072460 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.172 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.169+0000 7f43cd0e0640 1 -- 172.21.15.27:0/3144850623 >> 172.21.15.27:0/3144850623 conn(0x7f43c80d0a20 msgr2=0x7f43c80d0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.172 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.169+0000 7f43cd0e0640 1 -- 172.21.15.27:0/3144850623 shutdown_connections 2024-03-06T07:22:27.172 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.169+0000 7f43cd0e0640 1 -- 172.21.15.27:0/3144850623 wait complete. 2024-03-06T07:22:27.174 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.170+0000 7f43cd0e0640 1 Processor -- start 2024-03-06T07:22:27.174 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.170+0000 7f43cd0e0640 1 -- start start 2024-03-06T07:22:27.174 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43cd0e0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8062120 0x7f43c815d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43cd0e0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c8147840 0x7f43c80c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43cd0e0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43c815e380 0x7f43c80c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43cd0e0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80729a0 con 0x7f43c815e380 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43cd0e0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c806fd20 con 0x7f43c8147840 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43cd0e0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80b6660 con 0x7f43c8062120 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43c759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43c815e380 0x7f43c80c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43c6d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8062120 0x7f43c815d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43c659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c8147840 0x7f43c80c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43c759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43c815e380 0x7f43c80c5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59686/0 (socket says 172.21.15.27:59686) 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43c6d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8062120 0x7f43c815d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47478/0 (socket says 172.21.15.27:47478) 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.171+0000 7f43c759e640 1 -- 172.21.15.27:0/482635539 learned_addr learned my addr 172.21.15.27:0/482635539 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43c659c640 1 -- 172.21.15.27:0/482635539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8062120 msgr2=0x7f43c815d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43c659c640 1 --2- 172.21.15.27:0/482635539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8062120 0x7f43c815d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43c659c640 1 -- 172.21.15.27:0/482635539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43c815e380 msgr2=0x7f43c80c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43c659c640 1 --2- 172.21.15.27:0/482635539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43c815e380 0x7f43c80c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43c659c640 1 -- 172.21.15.27:0/482635539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c80b5a50 con 0x7f43c8147840 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43c6d9d640 1 --2- 172.21.15.27:0/482635539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8062120 0x7f43c815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43c759e640 1 --2- 172.21.15.27:0/482635539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43c815e380 0x7f43c80c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43c659c640 1 --2- 172.21.15.27:0/482635539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c8147840 0x7f43c80c59c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00c440 tx=0x7f43bc00c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43af7fe640 1 -- 172.21.15.27:0/482635539 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc017070 con 0x7f43c8147840 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43af7fe640 1 -- 172.21.15.27:0/482635539 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43bc0035f0 con 0x7f43c8147840 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43cd0e0640 1 -- 172.21.15.27:0/482635539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c80b5d30 con 0x7f43c8147840 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43af7fe640 1 -- 172.21.15.27:0/482635539 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc0039e0 con 0x7f43c8147840 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43cd0e0640 1 -- 172.21.15.27:0/482635539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c80b62f0 con 0x7f43c8147840 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43af7fe640 1 -- 172.21.15.27:0/482635539 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f43bc01a070 con 0x7f43c8147840 2024-03-06T07:22:27.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.172+0000 7f43af7fe640 1 -- 172.21.15.27:0/482635539 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f43bc003b80 con 0x7f43c8147840 2024-03-06T07:22:27.177 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.173+0000 7f43cd0e0640 1 -- 172.21.15.27:0/482635539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c81374a0 con 0x7f43c8147840 2024-03-06T07:22:27.177 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.176+0000 7f43af7fe640 1 -- 172.21.15.27:0/482635539 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f43c81374a0 con 0x7f43c8147840 2024-03-06T07:22:27.240 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.238+0000 7f43cd0e0640 1 -- 172.21.15.27:0/482635539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43c804f330 con 0x7f43c8147840 2024-03-06T07:22:27.241 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.239+0000 7f43af7fe640 1 -- 172.21.15.27:0/482635539 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f43bc021020 con 0x7f43c8147840 2024-03-06T07:22:27.241 INFO:teuthology.misc.health.smithi027.stdout: 2024-03-06T07:22:27.241 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":10,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:25.997497+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:27.242 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.240+0000 7f43cd0e0640 1 -- 172.21.15.27:0/482635539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c8147840 msgr2=0x7f43c80c59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.242 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.240+0000 7f43cd0e0640 1 --2- 172.21.15.27:0/482635539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c8147840 0x7f43c80c59c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00c440 tx=0x7f43bc00c840 comp rx=0 tx=0).stop 2024-03-06T07:22:27.242 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.240+0000 7f43cd0e0640 1 -- 172.21.15.27:0/482635539 shutdown_connections 2024-03-06T07:22:27.242 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.240+0000 7f43cd0e0640 1 --2- 172.21.15.27:0/482635539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43c815e380 0x7f43c80c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.242 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.240+0000 7f43cd0e0640 1 --2- 172.21.15.27:0/482635539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c8147840 0x7f43c80c59c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.242 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.240+0000 7f43cd0e0640 1 --2- 172.21.15.27:0/482635539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c8062120 0x7f43c815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.242 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.240+0000 7f43cd0e0640 1 -- 172.21.15.27:0/482635539 >> 172.21.15.27:0/482635539 conn(0x7f43c80d0a20 msgr2=0x7f43c80b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.242 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.240+0000 7f43cd0e0640 1 -- 172.21.15.27:0/482635539 shutdown_connections 2024-03-06T07:22:27.242 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:27.240+0000 7f43cd0e0640 1 -- 172.21.15.27:0/482635539 wait complete. 2024-03-06T07:22:27.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:22:27.524 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:27.522+0000 7f84f2041040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:22:27.585 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:27.584+0000 7f434121a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:22:27.800 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:27.798+0000 7f84f2041040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:22:27.850 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:27.848+0000 7f84f2041040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:22:27.877 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:27.875+0000 7f434121a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:22:27.931 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:27.929+0000 7f434121a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:22:27.963 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:27.961+0000 7f84f2041040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:22:28.053 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-06T07:22:28.051+0000 7f434121a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:22:28.992 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:28.990+0000 7f8492d91640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:22:28.992 INFO:tasks.ceph.mgr.x.smithi069.stderr:2024-03-06T07:22:28.990+0000 7f8492d91640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:22:30.108 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-03-06T07:22:30.107+0000 7fba2e4bd640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:22:30.116 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-03-06T07:22:30.114+0000 7fb11a72a640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:22:30.118 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-03-06T07:22:30.116+0000 7f1ecab9a640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:22:30.165 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-03-06T07:22:30.163+0000 7f692a8d9640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:22:30.169 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-03-06T07:22:30.168+0000 7fb9e676f640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:22:30.171 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-03-06T07:22:30.169+0000 7f0c5201b640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:22:30.177 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-03-06T07:22:30.176+0000 7f4edb411640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:22:30.177 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-03-06T07:22:30.176+0000 7f989e949640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:22:33.940 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-03-06T07:22:33.938+0000 7f989a0bc640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:22:33.955 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-03-06T07:22:33.954+0000 7f98926ee640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:22:34.057 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:34.065 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-03-06T07:22:34.063+0000 7f1ec630d640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:22:34.065 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-03-06T07:22:34.064+0000 7f0c4d1c9640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:22:34.067 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-03-06T07:22:34.066+0000 7f4ed64ec640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:22:34.081 INFO:tasks.ceph.osd.4.smithi069.stderr:2024-03-06T07:22:34.079+0000 7f1ebe93f640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:22:34.081 INFO:tasks.ceph.osd.7.smithi069.stderr:2024-03-06T07:22:34.079+0000 7f0c457fb640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:22:34.083 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-03-06T07:22:34.081+0000 7fb115e9d640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:22:34.083 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-03-06T07:22:34.082+0000 7f4eceb1e640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:22:34.087 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-03-06T07:22:34.086+0000 7f692604c640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:22:34.094 INFO:tasks.ceph.osd.6.smithi069.stderr:2024-03-06T07:22:34.093+0000 7fb10e4cf640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:22:34.104 INFO:tasks.ceph.osd.5.smithi069.stderr:2024-03-06T07:22:34.102+0000 7f691e67e640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:22:34.110 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-03-06T07:22:34.109+0000 7fba29c30640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:22:34.117 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-03-06T07:22:34.116+0000 7fb9e371a640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:22:34.127 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-03-06T07:22:34.126+0000 7fba22262640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:22:34.131 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-03-06T07:22:34.130+0000 7fb9da514640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:22:34.168 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.167+0000 7f4b492d4640 1 Processor -- start 2024-03-06T07:22:34.168 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.167+0000 7f4b492d4640 1 -- start start 2024-03-06T07:22:34.170 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b492d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440cf200 0x7f4b44168dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b492d4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b441622e0 con 0x7f4b4415e030 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b492d4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b441625c0 con 0x7f4b440b9640 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b492d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4415e410 con 0x7f4b440cf200 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b42ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4b4415e030 0x7f4b44161a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39748/0 (socket says 172.21.15.27:39748) 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b42ffd640 1 -- 172.21.15.27:0/2557753159 learned_addr learned my addr 172.21.15.27:0/2557753159 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b437fe640 1 --2- 172.21.15.27:0/2557753159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440cf200 0x7f4b44168dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b41ffb640 1 -- 172.21.15.27:0/2557753159 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119966952 0 0) 0x7f4b441622e0 con 0x7f4b4415e030 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b41ffb640 1 -- 172.21.15.27:0/2557753159 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2c003520 con 0x7f4b4415e030 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b41ffb640 1 -- 172.21.15.27:0/2557753159 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831991948 0 0) 0x7f4b441625c0 con 0x7f4b440b9640 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b437fe640 1 -- 172.21.15.27:0/2557753159 >> v1:172.21.15.69:6789/0 conn(0x7f4b440b9640 legacy=0x7f4b440ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b437fe640 1 -- 172.21.15.27:0/2557753159 >> v1:172.21.15.27:6789/0 conn(0x7f4b4415e030 legacy=0x7f4b44161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.168+0000 7f4b437fe640 1 -- 172.21.15.27:0/2557753159 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b441628a0 con 0x7f4b440cf200 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.169+0000 7f4b437fe640 1 --2- 172.21.15.27:0/2557753159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440cf200 0x7f4b44168dc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009dc0 tx=0x7f4b3c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c3cb954bc647030 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.171 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.169+0000 7f4b41ffb640 1 -- 172.21.15.27:0/2557753159 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c039070 con 0x7f4b440cf200 2024-03-06T07:22:34.172 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.169+0000 7f4b41ffb640 1 -- 172.21.15.27:0/2557753159 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c02ccd0 con 0x7f4b440cf200 2024-03-06T07:22:34.172 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.169+0000 7f4b41ffb640 1 -- 172.21.15.27:0/2557753159 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c034660 con 0x7f4b440cf200 2024-03-06T07:22:34.173 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.170+0000 7f4b492d4640 1 -- 172.21.15.27:0/2557753159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440cf200 msgr2=0x7f4b44168dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.173 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.170+0000 7f4b492d4640 1 --2- 172.21.15.27:0/2557753159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440cf200 0x7f4b44168dc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009dc0 tx=0x7f4b3c02bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:34.173 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.170+0000 7f4b492d4640 1 -- 172.21.15.27:0/2557753159 shutdown_connections 2024-03-06T07:22:34.173 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.170+0000 7f4b492d4640 1 --2- 172.21.15.27:0/2557753159 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440cf200 0x7f4b44168dc0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.173 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.170+0000 7f4b492d4640 1 -- 172.21.15.27:0/2557753159 >> 172.21.15.27:0/2557753159 conn(0x7f4b44061a10 msgr2=0x7f4b44061e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.173 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.170+0000 7f4b492d4640 1 -- 172.21.15.27:0/2557753159 shutdown_connections 2024-03-06T07:22:34.173 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.170+0000 7f4b492d4640 1 -- 172.21.15.27:0/2557753159 wait complete. 2024-03-06T07:22:34.173 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.171+0000 7f4b492d4640 1 Processor -- start 2024-03-06T07:22:34.173 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.171+0000 7f4b492d4640 1 -- start start 2024-03-06T07:22:34.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b492d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440b9640 0x7f4b440b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b492d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b440cf200 0x7f4b440b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b492d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4415e030 0x7f4b441f7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b492d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b440d37a0 con 0x7f4b4415e030 2024-03-06T07:22:34.175 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b492d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b440d3620 con 0x7f4b440cf200 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b492d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b440d3920 con 0x7f4b440b9640 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4415e030 0x7f4b441f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b42ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440b9640 0x7f4b440b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4415e030 0x7f4b441f7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49016/0 (socket says 172.21.15.27:49016) 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b437fe640 1 -- 172.21.15.27:0/479070404 learned_addr learned my addr 172.21.15.27:0/479070404 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b427fc640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b440cf200 0x7f4b440b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b437fe640 1 -- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440b9640 msgr2=0x7f4b440b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b437fe640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440b9640 0x7f4b440b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b437fe640 1 -- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b440cf200 msgr2=0x7f4b440b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b437fe640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b440cf200 0x7f4b440b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b437fe640 1 -- 172.21.15.27:0/479070404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b440cb410 con 0x7f4b4415e030 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.172+0000 7f4b42ffd640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440b9640 0x7f4b440b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.173+0000 7f4b437fe640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4415e030 0x7f4b441f7910 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009ee0 tx=0x7f4b3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.176 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.173+0000 7f4b2b7fe640 1 -- 172.21.15.27:0/479070404 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c039050 con 0x7f4b4415e030 2024-03-06T07:22:34.177 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.173+0000 7f4b2b7fe640 1 -- 172.21.15.27:0/479070404 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c045070 con 0x7f4b4415e030 2024-03-06T07:22:34.177 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.173+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b440cae90 con 0x7f4b4415e030 2024-03-06T07:22:34.177 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.173+0000 7f4b2b7fe640 1 -- 172.21.15.27:0/479070404 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c009450 con 0x7f4b4415e030 2024-03-06T07:22:34.177 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.173+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b440cb130 con 0x7f4b4415e030 2024-03-06T07:22:34.177 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.174+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4404f330 con 0x7f4b4415e030 2024-03-06T07:22:34.177 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.174+0000 7f4b2b7fe640 1 -- 172.21.15.27:0/479070404 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f4b3c03e850 con 0x7f4b4415e030 2024-03-06T07:22:34.178 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.174+0000 7f4b2b7fe640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4b3006b470 0x7f4b3006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.178 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.175+0000 7f4b2b7fe640 1 -- 172.21.15.27:0/479070404 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f4b3c0ac2d0 con 0x7f4b4415e030 2024-03-06T07:22:34.178 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.175+0000 7f4b42ffd640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4b3006b470 0x7f4b3006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.178 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.175+0000 7f4b42ffd640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4b3006b470 0x7f4b3006d930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4b34005880 tx=0x7f4b34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.180 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.179+0000 7f4b2b7fe640 1 -- 172.21.15.27:0/479070404 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b440cb410 con 0x7f4b4415e030 2024-03-06T07:22:34.365 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.364+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b440b2b70 con 0x7f4b4415e030 2024-03-06T07:22:34.366 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.365+0000 7f4b2b7fe640 1 -- 172.21.15.27:0/479070404 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f4b3c07ba90 con 0x7f4b4415e030 2024-03-06T07:22:34.366 INFO:teuthology.misc.health.smithi027.stdout: 2024-03-06T07:22:34.366 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":12,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:31.990099+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:34.368 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4b3006b470 msgr2=0x7f4b3006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4b3006b470 0x7f4b3006d930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4b34005880 tx=0x7f4b34009040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4415e030 msgr2=0x7f4b441f7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4415e030 0x7f4b441f7910 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009ee0 tx=0x7f4b3c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 shutdown_connections 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4b3006b470 0x7f4b3006d930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4415e030 0x7f4b441f7910 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b440cf200 0x7f4b440b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 --2- 172.21.15.27:0/479070404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b440b9640 0x7f4b440b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 >> 172.21.15.27:0/479070404 conn(0x7f4b44061a10 msgr2=0x7f4b44069040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 shutdown_connections 2024-03-06T07:22:34.369 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:34.367+0000 7f4b492d4640 1 -- 172.21.15.27:0/479070404 wait complete. 2024-03-06T07:22:34.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:22:41.185 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.287+0000 7fb7a00c3640 1 Processor -- start 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.287+0000 7fb7a00c3640 1 -- start start 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb7a00c3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb79815a3e0 0x7fb798072090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb7a00c3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb79806d940 0x7fb7980b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb7a00c3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79815ed80 con 0x7fb7980725d0 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb7a00c3640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb798072c00 con 0x7fb79806d940 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb7a00c3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb79815e8f0 con 0x7fb79815a3e0 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb79e660640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb79806d940 0x7fb7980b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb79e660640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb79806d940 0x7fb7980b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49024/0 (socket says 172.21.15.27:49024) 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb79e660640 1 -- 172.21.15.27:0/42736995 learned_addr learned my addr 172.21.15.27:0/42736995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb79de5f640 1 --2- 172.21.15.27:0/42736995 >> v2:172.21.15.69:3300/0 conn(0x7fb79815a3e0 0x7fb798072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.294 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb79e660640 1 -- 172.21.15.27:0/42736995 >> v2:172.21.15.69:3300/0 conn(0x7fb79815a3e0 msgr2=0x7fb798072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb79e660640 1 --2- 172.21.15.27:0/42736995 >> v2:172.21.15.69:3300/0 conn(0x7fb79815a3e0 0x7fb798072090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb79e660640 1 -- 172.21.15.27:0/42736995 >> v1:172.21.15.69:6789/0 conn(0x7fb7980725d0 legacy=0x7fb79806d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.288+0000 7fb79e660640 1 -- 172.21.15.27:0/42736995 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7980cd660 con 0x7fb79806d940 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.289+0000 7fb79e660640 1 --2- 172.21.15.27:0/42736995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb79806d940 0x7fb7980b3d40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb79806e080 tx=0x7fb79002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e15fa77d19060c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.289+0000 7fb79ce5d640 1 -- 172.21.15.27:0/42736995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79002cb20 con 0x7fb79806d940 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.289+0000 7fb79ce5d640 1 -- 172.21.15.27:0/42736995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79002ccc0 con 0x7fb79806d940 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.289+0000 7fb79ce5d640 1 -- 172.21.15.27:0/42736995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790034610 con 0x7fb79806d940 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.290+0000 7fb7a00c3640 1 -- 172.21.15.27:0/42736995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb79806d940 msgr2=0x7fb7980b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.290+0000 7fb7a00c3640 1 --2- 172.21.15.27:0/42736995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb79806d940 0x7fb7980b3d40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb79806e080 tx=0x7fb79002bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.290+0000 7fb7a00c3640 1 -- 172.21.15.27:0/42736995 shutdown_connections 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.290+0000 7fb7a00c3640 1 --2- 172.21.15.27:0/42736995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb79806d940 0x7fb7980b3d40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.290+0000 7fb7a00c3640 1 --2- 172.21.15.27:0/42736995 >> v2:172.21.15.69:3300/0 conn(0x7fb79815a3e0 0x7fb798072090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.290+0000 7fb7a00c3640 1 -- 172.21.15.27:0/42736995 >> 172.21.15.27:0/42736995 conn(0x7fb7980744d0 msgr2=0x7fb798061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.290+0000 7fb7a00c3640 1 -- 172.21.15.27:0/42736995 shutdown_connections 2024-03-06T07:22:41.295 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.290+0000 7fb7a00c3640 1 -- 172.21.15.27:0/42736995 wait complete. 2024-03-06T07:22:41.296 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.291+0000 7fb7a00c3640 1 Processor -- start 2024-03-06T07:22:41.296 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.291+0000 7fb7a00c3640 1 -- start start 2024-03-06T07:22:41.298 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb7a00c3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79806d940 0x7fb79814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.298 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb7a00c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7980725d0 0x7fb79815b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.298 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb7a00c3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79815a3e0 0x7fb798165420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.298 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb7a00c3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79815e2a0 con 0x7fb7980725d0 2024-03-06T07:22:41.298 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb7a00c3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980b38e0 con 0x7fb79815a3e0 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb7a00c3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79815e580 con 0x7fb79806d940 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79d65e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7980725d0 0x7fb79815b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79d65e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7980725d0 0x7fb79815b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49028/0 (socket says 172.21.15.27:49028) 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79d65e640 1 -- 172.21.15.27:0/259882475 learned_addr learned my addr 172.21.15.27:0/259882475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79e660640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79815a3e0 0x7fb798165420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79de5f640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79806d940 0x7fb79814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79d65e640 1 -- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79806d940 msgr2=0x7fb79814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79d65e640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79806d940 0x7fb79814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79d65e640 1 -- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79815a3e0 msgr2=0x7fb798165420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79d65e640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79815a3e0 0x7fb798165420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79d65e640 1 -- 172.21.15.27:0/259882475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79814b6d0 con 0x7fb7980725d0 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.292+0000 7fb79d65e640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7980725d0 0x7fb79815b1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00c400 tx=0x7fb78c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.293+0000 7fb78a7fc640 1 -- 172.21.15.27:0/259882475 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c0052d0 con 0x7fb7980725d0 2024-03-06T07:22:41.299 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.293+0000 7fb78a7fc640 1 -- 172.21.15.27:0/259882475 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c005490 con 0x7fb7980725d0 2024-03-06T07:22:41.300 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.293+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb798145fa0 con 0x7fb7980725d0 2024-03-06T07:22:41.300 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.293+0000 7fb78a7fc640 1 -- 172.21.15.27:0/259882475 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c007260 con 0x7fb7980725d0 2024-03-06T07:22:41.300 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.293+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb79815b6f0 con 0x7fb7980725d0 2024-03-06T07:22:41.300 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.294+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7980cc4b0 con 0x7fb7980725d0 2024-03-06T07:22:41.300 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.294+0000 7fb78a7fc640 1 -- 172.21.15.27:0/259882475 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb78c007400 con 0x7fb7980725d0 2024-03-06T07:22:41.300 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.295+0000 7fb78a7fc640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb77c06b4d0 0x7fb77c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.300 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.295+0000 7fb78a7fc640 1 -- 172.21.15.27:0/259882475 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb78c089350 con 0x7fb7980725d0 2024-03-06T07:22:41.300 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.295+0000 7fb79de5f640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb77c06b4d0 0x7fb77c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.300 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.295+0000 7fb79de5f640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb77c06b4d0 0x7fb77c06d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb798165e40 tx=0x7fb794009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.301 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.299+0000 7fb78a7fc640 1 -- 172.21.15.27:0/259882475 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7980cc4b0 con 0x7fb7980725d0 2024-03-06T07:22:41.484 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.482+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb798145d90 con 0x7fb7980725d0 2024-03-06T07:22:41.485 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.482+0000 7fb78a7fc640 1 -- 172.21.15.27:0/259882475 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12812 (secure 0 0 0) 0x7fb78c057ed0 con 0x7fb7980725d0 2024-03-06T07:22:41.485 INFO:teuthology.misc.health.smithi027.stdout: 2024-03-06T07:22:41.485 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":17,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:39.010912+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:41.487 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb77c06b4d0 msgr2=0x7fb77c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.487 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb77c06b4d0 0x7fb77c06d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb798165e40 tx=0x7fb794009190 comp rx=0 tx=0).stop 2024-03-06T07:22:41.487 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7980725d0 msgr2=0x7fb79815b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.487 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7980725d0 0x7fb79815b1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00c400 tx=0x7fb78c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.487 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 shutdown_connections 2024-03-06T07:22:41.487 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb77c06b4d0 0x7fb77c06d990 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.487 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79815a3e0 0x7fb798165420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.488 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7980725d0 0x7fb79815b1b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.488 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 --2- 172.21.15.27:0/259882475 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb79806d940 0x7fb79814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.488 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.484+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 >> 172.21.15.27:0/259882475 conn(0x7fb7980744d0 msgr2=0x7fb798151270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.488 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.485+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 shutdown_connections 2024-03-06T07:22:41.488 INFO:teuthology.misc.health.smithi027.stderr:2024-03-06T07:22:41.485+0000 7fb7a00c3640 1 -- 172.21.15.27:0/259882475 wait complete. 2024-03-06T07:22:41.497 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:22:41.497 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:22:41.497 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:41.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:41.500 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:41.500 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:41.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:41.502 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:41.502 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:41.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:41.505 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:41.505 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:41.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:41.507 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:41.508 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:22:41.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:41.581 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:22:41.584 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:22:41.590 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:22:41.590 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:22:41.590 INFO:tasks.ceph.mds.b.smithi069.stderr:2024-03-06T07:22:41.589+0000 7f99023d9640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:22:41.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.620+0000 7f3f8d7d0640 1 Processor -- start 2024-03-06T07:22:41.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.620+0000 7f3f8d7d0640 1 -- start start 2024-03-06T07:22:41.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f8d7d0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f88071540 0x7f3f880b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f8d7d0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f880ce610 con 0x7f3f880d3400 2024-03-06T07:22:41.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f8d7d0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f880d2920 con 0x7f3f880b2b50 2024-03-06T07:22:41.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f8d7d0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f880d2f70 con 0x7f3f88071540 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f86ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f88071540 0x7f3f880b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f867fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3f880d3400 0x7f3f88071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39768/0 (socket says 172.21.15.27:39768) 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f867fc640 1 -- 172.21.15.27:0/3869436698 learned_addr learned my addr 172.21.15.27:0/3869436698 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f85ffb640 1 -- 172.21.15.27:0/3869436698 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235232327 0 0) 0x7f3f880ce610 con 0x7f3f880d3400 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f85ffb640 1 -- 172.21.15.27:0/3869436698 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f70003520 con 0x7f3f880d3400 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f86ffd640 1 -- 172.21.15.27:0/3869436698 >> v1:172.21.15.69:6789/0 conn(0x7f3f880b2b50 legacy=0x7f3f880b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f86ffd640 1 -- 172.21.15.27:0/3869436698 >> v1:172.21.15.27:6789/0 conn(0x7f3f880d3400 legacy=0x7f3f88071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.621+0000 7f3f86ffd640 1 -- 172.21.15.27:0/3869436698 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f880d2c00 con 0x7f3f88071540 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.622+0000 7f3f86ffd640 1 --2- 172.21.15.27:0/3869436698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 0x7f3f880b30a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3f780097a0 tx=0x7f3f7802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824d88f1577f39dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.622+0000 7f3f85ffb640 1 -- 172.21.15.27:0/3869436698 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78034070 con 0x7f3f88071540 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.622+0000 7f3f85ffb640 1 -- 172.21.15.27:0/3869436698 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f78030610 con 0x7f3f88071540 2024-03-06T07:22:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.622+0000 7f3f85ffb640 1 -- 172.21.15.27:0/3869436698 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f780309e0 con 0x7f3f88071540 2024-03-06T07:22:41.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.622+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/3869436698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 msgr2=0x7f3f880b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.622+0000 7f3f8d7d0640 1 --2- 172.21.15.27:0/3869436698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 0x7f3f880b30a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3f780097a0 tx=0x7f3f7802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.622+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/3869436698 shutdown_connections 2024-03-06T07:22:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.622+0000 7f3f8d7d0640 1 --2- 172.21.15.27:0/3869436698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 0x7f3f880b30a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.622+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/3869436698 >> 172.21.15.27:0/3869436698 conn(0x7f3f880d3ca0 msgr2=0x7f3f880d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.623+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/3869436698 shutdown_connections 2024-03-06T07:22:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.623+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/3869436698 wait complete. 2024-03-06T07:22:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.624+0000 7f3f8d7d0640 1 Processor -- start 2024-03-06T07:22:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.624+0000 7f3f8d7d0640 1 -- start start 2024-03-06T07:22:41.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f8d7d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 0x7f3f880cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f8d7d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f880b2b50 0x7f3f880cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f8d7d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f880d3400 0x7f3f8814ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f8d7d0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8815e370 con 0x7f3f88071540 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f8d7d0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880d4be0 con 0x7f3f880d3400 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f8d7d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f88158720 con 0x7f3f880b2b50 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f86ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 0x7f3f880cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f86ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 0x7f3f880cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49088/0 (socket says 172.21.15.27:49088) 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f86ffd640 1 -- 172.21.15.27:0/1371725625 learned_addr learned my addr 172.21.15.27:0/1371725625 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f867fc640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f880b2b50 0x7f3f880cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f86ffd640 1 -- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f880b2b50 msgr2=0x7f3f880cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f877fe640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f880d3400 0x7f3f8814ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f86ffd640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f880b2b50 0x7f3f880cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f86ffd640 1 -- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f880d3400 msgr2=0x7f3f8814ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f86ffd640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f880d3400 0x7f3f8814ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f86ffd640 1 -- 172.21.15.27:0/1371725625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f880cb470 con 0x7f3f88071540 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f867fc640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f880b2b50 0x7f3f880cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f877fe640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f880d3400 0x7f3f8814ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f86ffd640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 0x7f3f880cad70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3f780098c0 tx=0x7f3f780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f6f7fe640 1 -- 172.21.15.27:0/1371725625 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78034040 con 0x7f3f88071540 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f6f7fe640 1 -- 172.21.15.27:0/1371725625 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f78030410 con 0x7f3f88071540 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f880cb750 con 0x7f3f88071540 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f6f7fe640 1 -- 172.21.15.27:0/1371725625 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78039c50 con 0x7f3f88071540 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.625+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f880cbd10 con 0x7f3f88071540 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.627+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8815deb0 con 0x7f3f88071540 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.627+0000 7f3f6f7fe640 1 -- 172.21.15.27:0/1371725625 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f78037070 con 0x7f3f88071540 2024-03-06T07:22:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.627+0000 7f3f6f7fe640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f7406b4d0 0x7f3f7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.628+0000 7f3f6f7fe640 1 -- 172.21.15.27:0/1371725625 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3f780abe30 con 0x7f3f88071540 2024-03-06T07:22:41.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.628+0000 7f3f867fc640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f7406b4d0 0x7f3f7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.628+0000 7f3f867fc640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f7406b4d0 0x7f3f7406d990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f88157f10 tx=0x7f3f7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.632+0000 7f3f6f7fe640 1 -- 172.21.15.27:0/1371725625 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f780305e0 con 0x7f3f88071540 2024-03-06T07:22:41.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.814+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f88068700 con 0x7f3f88071540 2024-03-06T07:22:41.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.815+0000 7f3f6f7fe640 1 -- 172.21.15.27:0/1371725625 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12812 (secure 0 0 0) 0x7f3f7807b000 con 0x7f3f88071540 2024-03-06T07:22:41.818 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:41.818 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:39.010912+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:41.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f7406b4d0 msgr2=0x7f3f7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f7406b4d0 0x7f3f7406d990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f88157f10 tx=0x7f3f7c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 msgr2=0x7f3f880cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 0x7f3f880cad70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3f780098c0 tx=0x7f3f780046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 shutdown_connections 2024-03-06T07:22:41.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f7406b4d0 0x7f3f7406d990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f880d3400 0x7f3f8814ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f880b2b50 0x7f3f880cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 --2- 172.21.15.27:0/1371725625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f88071540 0x7f3f880cad70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 >> 172.21.15.27:0/1371725625 conn(0x7f3f880d3ca0 msgr2=0x7f3f880b8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 shutdown_connections 2024-03-06T07:22:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.817+0000 7f3f8d7d0640 1 -- 172.21.15.27:0/1371725625 wait complete. 2024-03-06T07:22:41.831 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:41.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.938+0000 7f40179d7640 1 Processor -- start 2024-03-06T07:22:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.939+0000 7f40179d7640 1 -- start start 2024-03-06T07:22:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.939+0000 7f40179d7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f401015dd90 0x7f40100ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.939+0000 7f40179d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40100b6d10 0x7f40100b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.939+0000 7f40179d7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40100d3740 con 0x7f40100d29d0 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.939+0000 7f40179d7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40100d3400 con 0x7f401015dd90 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.939+0000 7f40179d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40100d3120 con 0x7f40100b6d10 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4015f74640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40100b6d10 0x7f40100b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4015773640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f401015dd90 0x7f40100ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4015f74640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40100b6d10 0x7f40100b71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:46994/0 (socket says 172.21.15.27:46994) 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4015f74640 1 -- 172.21.15.27:0/566823153 learned_addr learned my addr 172.21.15.27:0/566823153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4015f74640 1 -- 172.21.15.27:0/566823153 >> v2:172.21.15.69:3300/0 conn(0x7f401015dd90 msgr2=0x7f40100ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4015f74640 1 --2- 172.21.15.27:0/566823153 >> v2:172.21.15.69:3300/0 conn(0x7f401015dd90 0x7f40100ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4015f74640 1 -- 172.21.15.27:0/566823153 >> v1:172.21.15.69:6789/0 conn(0x7f40100d29d0 legacy=0x7f40100d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4007fff640 1 -- 172.21.15.27:0/566823153 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282100217 0 0) 0x7f40100d3740 con 0x7f40100d29d0 2024-03-06T07:22:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4015f74640 1 -- 172.21.15.27:0/566823153 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40100d39e0 con 0x7f40100b6d10 2024-03-06T07:22:41.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.940+0000 7f4015773640 1 --2- 172.21.15.27:0/566823153 >> v2:172.21.15.69:3300/0 conn(0x7f401015dd90 0x7f40100ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:41.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f4015f74640 1 --2- 172.21.15.27:0/566823153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40100b6d10 0x7f40100b71a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4008009e50 tx=0x7f400802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb824fd9e2ac4a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f4007fff640 1 -- 172.21.15.27:0/566823153 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400802cb90 con 0x7f40100b6d10 2024-03-06T07:22:41.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f4007fff640 1 -- 172.21.15.27:0/566823153 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400802cd30 con 0x7f40100b6d10 2024-03-06T07:22:41.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f4007fff640 1 -- 172.21.15.27:0/566823153 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008034610 con 0x7f40100b6d10 2024-03-06T07:22:41.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f40179d7640 1 -- 172.21.15.27:0/566823153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40100b6d10 msgr2=0x7f40100b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f40179d7640 1 --2- 172.21.15.27:0/566823153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40100b6d10 0x7f40100b71a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4008009e50 tx=0x7f400802bff0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f40179d7640 1 -- 172.21.15.27:0/566823153 shutdown_connections 2024-03-06T07:22:41.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f40179d7640 1 --2- 172.21.15.27:0/566823153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40100b6d10 0x7f40100b71a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f40179d7640 1 --2- 172.21.15.27:0/566823153 >> v2:172.21.15.69:3300/0 conn(0x7f401015dd90 0x7f40100ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.941+0000 7f40179d7640 1 -- 172.21.15.27:0/566823153 >> 172.21.15.27:0/566823153 conn(0x7f4010071960 msgr2=0x7f4010071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.942+0000 7f40179d7640 1 -- 172.21.15.27:0/566823153 shutdown_connections 2024-03-06T07:22:41.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.942+0000 7f40179d7640 1 -- 172.21.15.27:0/566823153 wait complete. 2024-03-06T07:22:41.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.943+0000 7f40179d7640 1 Processor -- start 2024-03-06T07:22:41.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.943+0000 7f40179d7640 1 -- start start 2024-03-06T07:22:41.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.943+0000 7f40179d7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40100b6d10 0x7f401014f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.943+0000 7f40179d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40100d29d0 0x7f4010160540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.943+0000 7f40179d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f401015dd90 0x7f40100b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.943+0000 7f40179d7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100d2df0 con 0x7f40100b6d10 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f40179d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401015e170 con 0x7f40100d29d0 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f40179d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40100d48f0 con 0x7f401015dd90 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015773640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40100b6d10 0x7f401014f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015773640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40100b6d10 0x7f401014f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49096/0 (socket says 172.21.15.27:49096) 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015773640 1 -- 172.21.15.27:0/1227705894 learned_addr learned my addr 172.21.15.27:0/1227705894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015f74640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f401015dd90 0x7f40100b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4014f72640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40100d29d0 0x7f4010160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015773640 1 -- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f401015dd90 msgr2=0x7f40100b8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015773640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f401015dd90 0x7f40100b8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015773640 1 -- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40100d29d0 msgr2=0x7f4010160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015773640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40100d29d0 0x7f4010160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015773640 1 -- 172.21.15.27:0/1227705894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401015e6d0 con 0x7f40100b6d10 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4015773640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40100b6d10 0x7f401014f160 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f400c00e960 tx=0x7f400c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4005ffb640 1 -- 172.21.15.27:0/1227705894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c017070 con 0x7f40100b6d10 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4005ffb640 1 -- 172.21.15.27:0/1227705894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c00ca00 con 0x7f40100b6d10 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401015e9b0 con 0x7f40100b6d10 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.944+0000 7f4005ffb640 1 -- 172.21.15.27:0/1227705894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00cdd0 con 0x7f40100b6d10 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.945+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401015ef70 con 0x7f40100b6d10 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.945+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4010072a30 con 0x7f40100b6d10 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.946+0000 7f4005ffb640 1 -- 172.21.15.27:0/1227705894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f400c01a070 con 0x7f40100b6d10 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.946+0000 7f4005ffb640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ff406b500 0x7f3ff406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.946+0000 7f4005ffb640 1 -- 172.21.15.27:0/1227705894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f400c089da0 con 0x7f40100b6d10 2024-03-06T07:22:41.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.946+0000 7f4014f72640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ff406b500 0x7f3ff406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.947+0000 7f4014f72640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ff406b500 0x7f3ff406d9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f40100b98a0 tx=0x7f4000009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:41.951+0000 7f4005ffb640 1 -- 172.21.15.27:0/1227705894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f400c058f70 con 0x7f40100b6d10 2024-03-06T07:22:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.132+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f401015d740 con 0x7f40100b6d10 2024-03-06T07:22:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.133+0000 7f4005ffb640 1 -- 172.21.15.27:0/1227705894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f400c058870 con 0x7f40100b6d10 2024-03-06T07:22:42.135 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:22:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ff406b500 msgr2=0x7f3ff406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ff406b500 0x7f3ff406d9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f40100b98a0 tx=0x7f4000009040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40100b6d10 msgr2=0x7f401014f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40100b6d10 0x7f401014f160 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f400c00e960 tx=0x7f400c00ee20 comp rx=0 tx=0).stop 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 shutdown_connections 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ff406b500 0x7f3ff406d9c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f401015dd90 0x7f40100b8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40100d29d0 0x7f4010160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 --2- 172.21.15.27:0/1227705894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40100b6d10 0x7f401014f160 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 >> 172.21.15.27:0/1227705894 conn(0x7f4010071960 msgr2=0x7f40100ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 shutdown_connections 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.135+0000 7f40179d7640 1 -- 172.21.15.27:0/1227705894 wait complete. 2024-03-06T07:22:42.148 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:42.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.253+0000 7f4a10081640 1 Processor -- start 2024-03-06T07:22:42.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.254+0000 7f4a10081640 1 -- start start 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.254+0000 7f4a10081640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4a080d3400 0x7f4a080b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.254+0000 7f4a10081640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a080ce610 con 0x7f4a080b2b50 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.254+0000 7f4a10081640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a080d2920 con 0x7f4a08071540 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.254+0000 7f4a10081640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a080d2f70 con 0x7f4a080d3400 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0e61e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a080b2b50 0x7f4a080b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39784/0 (socket says 172.21.15.27:39784) 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0e61e640 1 -- 172.21.15.27:0/111785070 learned_addr learned my addr 172.21.15.27:0/111785070 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0d61c640 1 --2- 172.21.15.27:0/111785070 >> v2:172.21.15.69:3300/0 conn(0x7f4a080d3400 0x7f4a080b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846537485 0 0) 0x7f4a080ce610 con 0x7f4a080b2b50 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f4003520 con 0x7f4a080b2b50 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2316534770 0 0) 0x7f49f4003520 con 0x7f4a080b2b50 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 >> v2:172.21.15.69:3300/0 conn(0x7f4a080d3400 msgr2=0x7f4a080b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 --2- 172.21.15.27:0/111785070 >> v2:172.21.15.69:3300/0 conn(0x7f4a080d3400 0x7f4a080b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 >> v1:172.21.15.69:6789/0 conn(0x7f4a08071540 legacy=0x7f4a08071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a080d2c00 con 0x7f4a080b2b50 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f4a000019f0 con 0x7f4a080b2b50 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 >> v1:172.21.15.27:6789/0 conn(0x7f4a080b2b50 legacy=0x7f4a080b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40039b0 0x7f49f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49f4004800 0x7f49f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49f40051c0 0x7f49f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080d2f70 con 0x7f49f40039b0 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f400c060 con 0x7f49f4004800 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f400c260 con 0x7f49f40051c0 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0de1d640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40039b0 0x7f49f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0d61c640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49f4004800 0x7f49f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.255+0000 7f4a0e61e640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49f40051c0 0x7f49f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0de1d640 1 -- 172.21.15.27:0/111785070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49f40051c0 msgr2=0x7f49f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0de1d640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49f40051c0 0x7f49f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0de1d640 1 -- 172.21.15.27:0/111785070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49f4004800 msgr2=0x7f49f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0de1d640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49f4004800 0x7f49f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0de1d640 1 -- 172.21.15.27:0/111785070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f4003520 con 0x7f49f40039b0 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0e61e640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49f40051c0 0x7f49f400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0e61e640 1 -- 172.21.15.27:0/111785070 reap_dead start 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0de1d640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40039b0 0x7f49f4003dd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4a04001750 tx=0x7f49f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d6041b2bf72fd1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04001370 con 0x7f49f40039b0 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a040052d0 con 0x7f49f40039b0 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.256+0000 7f4a0ce1b640 1 -- 172.21.15.27:0/111785070 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04003830 con 0x7f49f40039b0 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.257+0000 7f4a10081640 1 -- 172.21.15.27:0/111785070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40039b0 msgr2=0x7f49f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.257+0000 7f4a10081640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40039b0 0x7f49f4003dd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4a04001750 tx=0x7f49f4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.257+0000 7f4a10081640 1 -- 172.21.15.27:0/111785070 shutdown_connections 2024-03-06T07:22:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.257+0000 7f4a10081640 1 --2- 172.21.15.27:0/111785070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49f40039b0 0x7f49f4003dd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.257+0000 7f4a10081640 1 -- 172.21.15.27:0/111785070 >> 172.21.15.27:0/111785070 conn(0x7f4a080d3ca0 msgr2=0x7f4a080d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.257+0000 7f4a10081640 1 -- 172.21.15.27:0/111785070 shutdown_connections 2024-03-06T07:22:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.257+0000 7f4a10081640 1 -- 172.21.15.27:0/111785070 wait complete. 2024-03-06T07:22:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.258+0000 7f4a10081640 1 Processor -- start 2024-03-06T07:22:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.258+0000 7f4a10081640 1 -- start start 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a10081640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49f40039b0 0x7f4a080ab990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a10081640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a080adf60 0x7f4a080ad0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a10081640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a0815f5c0 0x7f4a080af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a10081640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0815e370 con 0x7f4a080adf60 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a10081640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080d4be0 con 0x7f49f40039b0 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a10081640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080ac050 con 0x7f4a0815f5c0 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0d61c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a080adf60 0x7f4a080ad0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0d61c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a080adf60 0x7f4a080ad0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49116/0 (socket says 172.21.15.27:49116) 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0d61c640 1 -- 172.21.15.27:0/2792399638 learned_addr learned my addr 172.21.15.27:0/2792399638 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0e61e640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a0815f5c0 0x7f4a080af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0de1d640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49f40039b0 0x7f4a080ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0d61c640 1 -- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a0815f5c0 msgr2=0x7f4a080af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0d61c640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a0815f5c0 0x7f4a080af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0d61c640 1 -- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49f40039b0 msgr2=0x7f4a080ab990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0d61c640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49f40039b0 0x7f4a080ab990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0d61c640 1 -- 172.21.15.27:0/2792399638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a080ac330 con 0x7f4a080adf60 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a0d61c640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a080adf60 0x7f4a080ad0f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0119b0 tx=0x7f49fc011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f49fa7fc640 1 -- 172.21.15.27:0/2792399638 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc00faf0 con 0x7f4a080adf60 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f49fa7fc640 1 -- 172.21.15.27:0/2792399638 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49fc00fc90 con 0x7f4a080adf60 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.259+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a080ac610 con 0x7f4a080adf60 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.260+0000 7f49fa7fc640 1 -- 172.21.15.27:0/2792399638 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc014440 con 0x7f4a080adf60 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.260+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a080ad630 con 0x7f4a080adf60 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.260+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a080aca00 con 0x7f4a080adf60 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.261+0000 7f49fa7fc640 1 -- 172.21.15.27:0/2792399638 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49fc012040 con 0x7f4a080adf60 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.261+0000 7f49fa7fc640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f49ec06b4d0 0x7f49ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.261+0000 7f49fa7fc640 1 -- 172.21.15.27:0/2792399638 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f49fc08a5d0 con 0x7f4a080adf60 2024-03-06T07:22:42.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.261+0000 7f4a0de1d640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f49ec06b4d0 0x7f49ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.262+0000 7f4a0de1d640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f49ec06b4d0 0x7f49ec06d990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a04033000 tx=0x7f4a04009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.266+0000 7f49fa7fc640 1 -- 172.21.15.27:0/2792399638 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49fc019030 con 0x7f4a080adf60 2024-03-06T07:22:42.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.484+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a080ac7e0 con 0x7f4a080adf60 2024-03-06T07:22:42.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.484+0000 7f49fa7fc640 1 -- 172.21.15.27:0/2792399638 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f49fc0590c0 con 0x7f4a080adf60 2024-03-06T07:22:42.487 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:42.487 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:22:42.487 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 2 2024-03-06T07:22:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f49ec06b4d0 msgr2=0x7f49ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f49ec06b4d0 0x7f49ec06d990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a04033000 tx=0x7f4a04009030 comp rx=0 tx=0).stop 2024-03-06T07:22:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a080adf60 msgr2=0x7f4a080ad0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a080adf60 0x7f4a080ad0f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0119b0 tx=0x7f49fc011e70 comp rx=0 tx=0).stop 2024-03-06T07:22:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 shutdown_connections 2024-03-06T07:22:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f49ec06b4d0 0x7f49ec06d990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a0815f5c0 0x7f4a080af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a080adf60 0x7f4a080ad0f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 --2- 172.21.15.27:0/2792399638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49f40039b0 0x7f4a080ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 >> 172.21.15.27:0/2792399638 conn(0x7f4a080d3ca0 msgr2=0x7f4a080b86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 shutdown_connections 2024-03-06T07:22:42.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:42.486+0000 7f4a10081640 1 -- 172.21.15.27:0/2792399638 wait complete. 2024-03-06T07:22:44.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:44.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.618+0000 7f111abe2640 1 Processor -- start 2024-03-06T07:22:44.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.618+0000 7f111abe2640 1 -- start start 2024-03-06T07:22:44.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.619+0000 7f111abe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114157f60 0x7f111415b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.619+0000 7f111abe2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111406c7b0 con 0x7f1114157b80 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.619+0000 7f111abe2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111406ca90 con 0x7f11140cbab0 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.619+0000 7f111abe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111406cd70 con 0x7f1114157f60 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.619+0000 7f111897e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1114157b80 0x7f111415b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56550/0 (socket says 172.21.15.27:56550) 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.619+0000 7f111897e640 1 -- 172.21.15.27:0/4137996402 learned_addr learned my addr 172.21.15.27:0/4137996402 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.619+0000 7f1113fff640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114157f60 0x7f111415b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.619+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072823303 0 0) 0x7f111406c7b0 con 0x7f1114157b80 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.619+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10fc004630 con 0x7f1114157b80 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926939288 0 0) 0x7f10fc004630 con 0x7f1114157b80 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114157f60 msgr2=0x7f111415b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114157f60 0x7f111415b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 >> v1:172.21.15.69:6789/0 conn(0x7f11140cbab0 legacy=0x7f1114070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111406d050 con 0x7f1114157b80 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f110c0019f0 con 0x7f1114157b80 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 >> v1:172.21.15.27:6789/0 conn(0x7f1114157b80 legacy=0x7f111415b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114157f60 0x7f10fc004af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc005030 0x7f10fc005910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10fc005e50 0x7f10fc0062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111406cd70 con 0x7f10fc005030 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc00cb30 con 0x7f10fc005e50 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc00cd30 con 0x7f1114157f60 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f1113fff640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc005030 0x7f10fc005910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f111897e640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114157f60 0x7f10fc004af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f111917f640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10fc005e50 0x7f10fc0062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f1113fff640 1 -- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114157f60 msgr2=0x7f10fc004af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f1113fff640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114157f60 0x7f10fc004af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f1113fff640 1 -- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10fc005e50 msgr2=0x7f10fc0062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f1113fff640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10fc005e50 0x7f10fc0062d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.620+0000 7f1113fff640 1 -- 172.21.15.27:0/4137996402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc004630 con 0x7f10fc005030 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.621+0000 7f1113fff640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc005030 0x7f10fc005910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1108009ac0 tx=0x7f10fc006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f6496cb6c56eab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.621+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110803b070 con 0x7f10fc005030 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.621+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110802ee30 con 0x7f10fc005030 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.621+0000 7f11137fe640 1 -- 172.21.15.27:0/4137996402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108009690 con 0x7f10fc005030 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.622+0000 7f111abe2640 1 -- 172.21.15.27:0/4137996402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc005030 msgr2=0x7f10fc005910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.622+0000 7f111abe2640 1 --2- 172.21.15.27:0/4137996402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc005030 0x7f10fc005910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1108009ac0 tx=0x7f10fc006940 comp rx=0 tx=0).stop 2024-03-06T07:22:44.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.622+0000 7f111917f640 1 -- 172.21.15.27:0/4137996402 reap_dead start 2024-03-06T07:22:44.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.622+0000 7f111abe2640 1 -- 172.21.15.27:0/4137996402 shutdown_connections 2024-03-06T07:22:44.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.622+0000 7f111abe2640 1 -- 172.21.15.27:0/4137996402 >> 172.21.15.27:0/4137996402 conn(0x7f1114066d00 msgr2=0x7f11140670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.622+0000 7f111abe2640 1 -- 172.21.15.27:0/4137996402 shutdown_connections 2024-03-06T07:22:44.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.622+0000 7f111abe2640 1 -- 172.21.15.27:0/4137996402 wait complete. 2024-03-06T07:22:44.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.623+0000 7f111abe2640 1 Processor -- start 2024-03-06T07:22:44.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.623+0000 7f111abe2640 1 -- start start 2024-03-06T07:22:44.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111abe2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10fc005030 0x7f11141518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111abe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114143480 0x7f11140cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111abe2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114155170 0x7f1114153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111abe2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1114071250 con 0x7f1114155170 2024-03-06T07:22:44.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111abe2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11140cc260 con 0x7f10fc005030 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111abe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11140713d0 con 0x7f1114143480 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111917f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114155170 0x7f1114153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111917f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114155170 0x7f1114153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58268/0 (socket says 172.21.15.27:58268) 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111917f640 1 -- 172.21.15.27:0/2809514791 learned_addr learned my addr 172.21.15.27:0/2809514791 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111897e640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10fc005030 0x7f11141518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f1113fff640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114143480 0x7f11140cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111917f640 1 -- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114143480 msgr2=0x7f11140cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111917f640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114143480 0x7f11140cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111917f640 1 -- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10fc005030 msgr2=0x7f11141518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111917f640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10fc005030 0x7f11141518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.624+0000 7f111917f640 1 -- 172.21.15.27:0/2809514791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1114157270 con 0x7f1114155170 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.625+0000 7f111917f640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114155170 0x7f1114153580 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f110400f770 tx=0x7f110400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.625+0000 7f11117fa640 1 -- 172.21.15.27:0/2809514791 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1104019070 con 0x7f1114155170 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.625+0000 7f11117fa640 1 -- 172.21.15.27:0/2809514791 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1104005030 con 0x7f1114155170 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.625+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1114156f90 con 0x7f1114155170 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.625+0000 7f11117fa640 1 -- 172.21.15.27:0/2809514791 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1104005420 con 0x7f1114155170 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.625+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11140ce260 con 0x7f1114155170 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.626+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11140cdd70 con 0x7f1114155170 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.626+0000 7f11117fa640 1 -- 172.21.15.27:0/2809514791 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f110401c070 con 0x7f1114155170 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.627+0000 7f11117fa640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f110006b4d0 0x7f110006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.627+0000 7f11117fa640 1 -- 172.21.15.27:0/2809514791 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1104088f30 con 0x7f1114155170 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.627+0000 7f111897e640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f110006b4d0 0x7f110006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.628+0000 7f111897e640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f110006b4d0 0x7f110006d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f11140ae610 tx=0x7f110c006040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.632+0000 7f11117fa640 1 -- 172.21.15.27:0/2809514791 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1104057ab0 con 0x7f1114155170 2024-03-06T07:22:44.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.853+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1114067ff0 con 0x7f1114155170 2024-03-06T07:22:44.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.853+0000 7f11117fa640 1 -- 172.21.15.27:0/2809514791 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7f110405bb50 con 0x7f1114155170 2024-03-06T07:22:44.856 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:44.856 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1135544337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1135544337},{"type":"v1","addr":"172.21.15.27:6833","nonce":1135544337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1920994895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1920994895},{"type":"v1","addr":"172.21.15.69:6837","nonce":1920994895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3933261233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3933261233},{"type":"v1","addr":"172.21.15.27:6835","nonce":3933261233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-06T07:22:44.856 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-03-06T07:22:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.855+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f110006b4d0 msgr2=0x7f110006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.855+0000 7f111abe2640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f110006b4d0 0x7f110006d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f11140ae610 tx=0x7f110c006040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.855+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114155170 msgr2=0x7f1114153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.855+0000 7f111abe2640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114155170 0x7f1114153580 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f110400f770 tx=0x7f110400fc30 comp rx=0 tx=0).stop 2024-03-06T07:22:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.856+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 shutdown_connections 2024-03-06T07:22:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.856+0000 7f111abe2640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f110006b4d0 0x7f110006d990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.856+0000 7f111abe2640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114155170 0x7f1114153580 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.856+0000 7f111abe2640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1114143480 0x7f11140cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.856+0000 7f111abe2640 1 --2- 172.21.15.27:0/2809514791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10fc005030 0x7f11141518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.856+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 >> 172.21.15.27:0/2809514791 conn(0x7f1114066d00 msgr2=0x7f11140ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.856+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 shutdown_connections 2024-03-06T07:22:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.856+0000 7f111abe2640 1 -- 172.21.15.27:0/2809514791 wait complete. 2024-03-06T07:22:44.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.973+0000 7fb59957a640 1 Processor -- start 2024-03-06T07:22:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.974+0000 7fb59957a640 1 -- start start 2024-03-06T07:22:44.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.974+0000 7fb59957a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb594071c50 0x7fb5940b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.974+0000 7fb59957a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940b6f20 con 0x7fb5940d18c0 2024-03-06T07:22:44.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.974+0000 7fb59957a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940d3030 con 0x7fb59415df70 2024-03-06T07:22:44.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.974+0000 7fb59957a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940d3310 con 0x7fb594071c50 2024-03-06T07:22:44.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.974+0000 7fb5927fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb5940d18c0 0x7fb594071540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56554/0 (socket says 172.21.15.27:56554) 2024-03-06T07:22:44.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.974+0000 7fb5927fc640 1 -- 172.21.15.27:0/1539133335 learned_addr learned my addr 172.21.15.27:0/1539133335 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:44.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb5937fe640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb594071c50 0x7fb5940b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782265892 0 0) 0x7fb5940b6f20 con 0x7fb5940d18c0 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb580003520 con 0x7fb5940d18c0 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3602947331 0 0) 0x7fb580003520 con 0x7fb5940d18c0 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb594071c50 msgr2=0x7fb5940b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb594071c50 0x7fb5940b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 >> v1:172.21.15.69:6789/0 conn(0x7fb59415df70 legacy=0x7fb5940d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5940d35f0 con 0x7fb5940d18c0 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fb57c001670 con 0x7fb5940d18c0 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 >> v1:172.21.15.27:6789/0 conn(0x7fb5940d18c0 legacy=0x7fb594071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5800039e0 0x7fb580003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb580004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb594071c50 0x7fb5800050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940d3310 con 0x7fb580004390 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58000b9f0 con 0x7fb5800039e0 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58000bbf0 con 0x7fb594071c50 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb5927fc640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb580004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb592ffd640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5800039e0 0x7fb580003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb5937fe640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb594071c50 0x7fb5800050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.975+0000 7fb5927fc640 1 -- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb594071c50 msgr2=0x7fb5800050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.976+0000 7fb5927fc640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb594071c50 0x7fb5800050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.976+0000 7fb5927fc640 1 -- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5800039e0 msgr2=0x7fb580003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.976+0000 7fb5927fc640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5800039e0 0x7fb580003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.976+0000 7fb5927fc640 1 -- 172.21.15.27:0/1539133335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580003520 con 0x7fb580004390 2024-03-06T07:22:44.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.976+0000 7fb5927fc640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb580004860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c290 tx=0x7fb580005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6516c99fe1c4968a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.976+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c007d50 con 0x7fb580004390 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.976+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c030070 con 0x7fb580004390 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.976+0000 7fb591ffb640 1 -- 172.21.15.27:0/1539133335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c02cd60 con 0x7fb580004390 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.977+0000 7fb59957a640 1 -- 172.21.15.27:0/1539133335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 msgr2=0x7fb580004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.977+0000 7fb59957a640 1 --2- 172.21.15.27:0/1539133335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb580004860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c290 tx=0x7fb580005830 comp rx=0 tx=0).stop 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.977+0000 7fb5937fe640 1 -- 172.21.15.27:0/1539133335 reap_dead start 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.977+0000 7fb59957a640 1 -- 172.21.15.27:0/1539133335 shutdown_connections 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.977+0000 7fb59957a640 1 -- 172.21.15.27:0/1539133335 >> 172.21.15.27:0/1539133335 conn(0x7fb5940d3cf0 msgr2=0x7fb5940d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.977+0000 7fb59957a640 1 -- 172.21.15.27:0/1539133335 shutdown_connections 2024-03-06T07:22:44.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.977+0000 7fb59957a640 1 -- 172.21.15.27:0/1539133335 wait complete. 2024-03-06T07:22:44.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.978+0000 7fb59957a640 1 Processor -- start 2024-03-06T07:22:44.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.978+0000 7fb59957a640 1 -- start start 2024-03-06T07:22:44.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb59957a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb5940cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb59957a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5940ce1c0 0x7fb5941f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb59957a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5941f43e0 0x7fb5941f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb59957a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59415e4d0 con 0x7fb580004390 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb59957a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59415e350 con 0x7fb5941f43e0 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb59957a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940b8410 con 0x7fb5940ce1c0 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb592ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb5940cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb592ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb5940cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58296/0 (socket says 172.21.15.27:58296) 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb592ffd640 1 -- 172.21.15.27:0/4091858776 learned_addr learned my addr 172.21.15.27:0/4091858776 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb5927fc640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5940ce1c0 0x7fb5941f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb5937fe640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5941f43e0 0x7fb5941f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb592ffd640 1 -- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5940ce1c0 msgr2=0x7fb5941f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb592ffd640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5940ce1c0 0x7fb5941f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb592ffd640 1 -- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5941f43e0 msgr2=0x7fb5941f6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb592ffd640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5941f43e0 0x7fb5941f6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb592ffd640 1 -- 172.21.15.27:0/4091858776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5940cc030 con 0x7fb580004390 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.979+0000 7fb5927fc640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5940ce1c0 0x7fb5941f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.980+0000 7fb592ffd640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb5940cdc80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb58400fb60 tx=0x7fb58400ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.980+0000 7fb58b7fe640 1 -- 172.21.15.27:0/4091858776 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb584003b80 con 0x7fb580004390 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.980+0000 7fb58b7fe640 1 -- 172.21.15.27:0/4091858776 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb584018070 con 0x7fb580004390 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.980+0000 7fb58b7fe640 1 -- 172.21.15.27:0/4091858776 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58400d8e0 con 0x7fb580004390 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.980+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5940cc310 con 0x7fb580004390 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.980+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5940cc8d0 con 0x7fb580004390 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.981+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb594067ff0 con 0x7fb580004390 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.981+0000 7fb58b7fe640 1 -- 172.21.15.27:0/4091858776 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb58400da80 con 0x7fb580004390 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.982+0000 7fb58b7fe640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb57406b4d0 0x7fb57406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.982+0000 7fb58b7fe640 1 -- 172.21.15.27:0/4091858776 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb58408a080 con 0x7fb580004390 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.982+0000 7fb5927fc640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb57406b4d0 0x7fb57406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.982+0000 7fb5927fc640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb57406b4d0 0x7fb57406d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb57c034000 tx=0x7fb57c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:44.986+0000 7fb58b7fe640 1 -- 172.21.15.27:0/4091858776 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5840592e0 con 0x7fb580004390 2024-03-06T07:22:45.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.168+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5940ce5a0 con 0x7fb580004390 2024-03-06T07:22:45.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.169+0000 7fb58b7fe640 1 -- 172.21.15.27:0/4091858776 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12812 (secure 0 0 0) 0x7fb584058be0 con 0x7fb580004390 2024-03-06T07:22:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:45.171 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:39.010912+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:45.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb57406b4d0 msgr2=0x7fb57406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb57406b4d0 0x7fb57406d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb57c034000 tx=0x7fb57c009030 comp rx=0 tx=0).stop 2024-03-06T07:22:45.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 msgr2=0x7fb5940cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb5940cdc80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb58400fb60 tx=0x7fb58400ff60 comp rx=0 tx=0).stop 2024-03-06T07:22:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 shutdown_connections 2024-03-06T07:22:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb57406b4d0 0x7fb57406d990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5941f43e0 0x7fb5941f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5940ce1c0 0x7fb5941f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 --2- 172.21.15.27:0/4091858776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb580004390 0x7fb5940cdc80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 >> 172.21.15.27:0/4091858776 conn(0x7fb5940d3cf0 msgr2=0x7fb5940d1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 shutdown_connections 2024-03-06T07:22:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.171+0000 7fb59957a640 1 -- 172.21.15.27:0/4091858776 wait complete. 2024-03-06T07:22:45.184 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:45.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.291+0000 7f9c150df640 1 Processor -- start 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.291+0000 7f9c150df640 1 -- start start 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c150df640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c100ce4f0 0x7f9c100ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c150df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100d0150 0x7f9c100d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c150df640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1015f730 con 0x7f9c101618e0 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c150df640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c100cfce0 con 0x7f9c100ce4f0 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c150df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100ceea0 con 0x7f9c100d0150 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c0f59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100d0150 0x7f9c100d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c0659c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c100ce4f0 0x7f9c100ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c0ed9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9c101618e0 0x7f9c1006b130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:43718/0 (socket says 172.21.15.27:43718) 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c0ed9d640 1 -- 172.21.15.27:0/2604986214 learned_addr learned my addr 172.21.15.27:0/2604986214 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.292+0000 7f9c0f59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100d0150 0x7f9c100d05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:60442/0 (socket says 172.21.15.27:60442) 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.293+0000 7f9c0659c640 1 -- 172.21.15.27:0/2604986214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100d0150 msgr2=0x7f9c100d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.293+0000 7f9c0659c640 1 --2- 172.21.15.27:0/2604986214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100d0150 0x7f9c100d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.293+0000 7f9c0659c640 1 -- 172.21.15.27:0/2604986214 >> v1:172.21.15.69:6789/0 conn(0x7f9c101618e0 legacy=0x7f9c1006b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.293+0000 7f9c0659c640 1 -- 172.21.15.27:0/2604986214 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1015ebb0 con 0x7f9c100ce4f0 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.293+0000 7f9c0659c640 1 --2- 172.21.15.27:0/2604986214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c100ce4f0 0x7f9c100ce960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80097a0 tx=0x7f9bf802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0a8c0a26b5d98df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c0e59c640 1 -- 172.21.15.27:0/2604986214 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8034070 con 0x7f9c100ce4f0 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c0e59c640 1 -- 172.21.15.27:0/2604986214 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf8030610 con 0x7f9c100ce4f0 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c0e59c640 1 -- 172.21.15.27:0/2604986214 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf80309e0 con 0x7f9c100ce4f0 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c150df640 1 -- 172.21.15.27:0/2604986214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c100ce4f0 msgr2=0x7f9c100ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c150df640 1 --2- 172.21.15.27:0/2604986214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c100ce4f0 0x7f9c100ce960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80097a0 tx=0x7f9bf802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c150df640 1 -- 172.21.15.27:0/2604986214 shutdown_connections 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c150df640 1 --2- 172.21.15.27:0/2604986214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c100ce4f0 0x7f9c100ce960 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c150df640 1 --2- 172.21.15.27:0/2604986214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100d0150 0x7f9c100d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c150df640 1 -- 172.21.15.27:0/2604986214 >> 172.21.15.27:0/2604986214 conn(0x7f9c100b3660 msgr2=0x7f9c100b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c150df640 1 -- 172.21.15.27:0/2604986214 shutdown_connections 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.294+0000 7f9c150df640 1 -- 172.21.15.27:0/2604986214 wait complete. 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.296+0000 7f9c150df640 1 Processor -- start 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.296+0000 7f9c150df640 1 -- start start 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.296+0000 7f9c150df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c100ce4f0 0x7f9c100b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.296+0000 7f9c150df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c100d0150 0x7f9c101561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.296+0000 7f9c150df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100c9630 0x7f9c100caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.296+0000 7f9c150df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1015f450 con 0x7f9c100ce4f0 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.296+0000 7f9c150df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1015f170 con 0x7f9c100d0150 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.296+0000 7f9c150df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1015e7d0 con 0x7f9c100c9630 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c100ce4f0 0x7f9c100b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c100ce4f0 0x7f9c100b66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58308/0 (socket says 172.21.15.27:58308) 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0ed9d640 1 -- 172.21.15.27:0/3151277546 learned_addr learned my addr 172.21.15.27:0/3151277546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0f59e640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100c9630 0x7f9c100caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0659c640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c100d0150 0x7f9c101561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0ed9d640 1 -- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100c9630 msgr2=0x7f9c100caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0ed9d640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100c9630 0x7f9c100caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0ed9d640 1 -- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c100d0150 msgr2=0x7f9c101561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0ed9d640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c100d0150 0x7f9c101561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0ed9d640 1 -- 172.21.15.27:0/3151277546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c100c9dd0 con 0x7f9c100ce4f0 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c0ed9d640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c100ce4f0 0x7f9c100b66f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000c3f0 tx=0x7f9c0000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c07fff640 1 -- 172.21.15.27:0/3151277546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00017070 con 0x7f9c100ce4f0 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c07fff640 1 -- 172.21.15.27:0/3151277546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c00005ce0 con 0x7f9c100ce4f0 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c100ca0b0 con 0x7f9c100ce4f0 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c07fff640 1 -- 172.21.15.27:0/3151277546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c000072f0 con 0x7f9c100ce4f0 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.297+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c100c60c0 con 0x7f9c100ce4f0 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.298+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1009fc50 con 0x7f9c100ce4f0 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.299+0000 7f9c07fff640 1 -- 172.21.15.27:0/3151277546 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c0001a070 con 0x7f9c100ce4f0 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.299+0000 7f9c07fff640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bf006b4d0 0x7f9bf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.299+0000 7f9c07fff640 1 -- 172.21.15.27:0/3151277546 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9c000892e0 con 0x7f9c100ce4f0 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.299+0000 7f9c0659c640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bf006b4d0 0x7f9bf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.300+0000 7f9c0659c640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bf006b4d0 0x7f9bf006d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8036000 tx=0x7f9bf803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.304+0000 7f9c07fff640 1 -- 172.21.15.27:0/3151277546 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c00058420 con 0x7f9c100ce4f0 2024-03-06T07:22:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.483+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c1004f410 con 0x7f9c100ce4f0 2024-03-06T07:22:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.483+0000 7f9c07fff640 1 -- 172.21.15.27:0/3151277546 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c00057d20 con 0x7f9c100ce4f0 2024-03-06T07:22:45.486 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bf006b4d0 msgr2=0x7f9bf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bf006b4d0 0x7f9bf006d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8036000 tx=0x7f9bf803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c100ce4f0 msgr2=0x7f9c100b66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c100ce4f0 0x7f9c100b66f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000c3f0 tx=0x7f9c0000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 shutdown_connections 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bf006b4d0 0x7f9bf006d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c100c9630 0x7f9c100caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c100d0150 0x7f9c101561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 --2- 172.21.15.27:0/3151277546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c100ce4f0 0x7f9c100b66f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 >> 172.21.15.27:0/3151277546 conn(0x7f9c100b3660 msgr2=0x7f9c10157fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 shutdown_connections 2024-03-06T07:22:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.485+0000 7f9c150df640 1 -- 172.21.15.27:0/3151277546 wait complete. 2024-03-06T07:22:45.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.603+0000 7f89c4ba2640 1 Processor -- start 2024-03-06T07:22:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.603+0000 7f89c4ba2640 1 -- start start 2024-03-06T07:22:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89c4ba2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89c01612b0 0x7f89c00b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89c4ba2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89c00b5c40 0x7f89c00d19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89c4ba2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c00cf370 con 0x7f89c0161690 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89c4ba2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c00d1040 con 0x7f89c01612b0 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89c4ba2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c00d1320 con 0x7f89c00b5c40 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89bf7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89c01612b0 0x7f89c00b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89bf7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89c01612b0 0x7f89c00b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58320/0 (socket says 172.21.15.27:58320) 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89bf7fe640 1 -- 172.21.15.27:0/3488845062 learned_addr learned my addr 172.21.15.27:0/3488845062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89bffff640 1 --2- 172.21.15.27:0/3488845062 >> v2:172.21.15.69:3300/0 conn(0x7f89c00b5c40 0x7f89c00d19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89bf7fe640 1 -- 172.21.15.27:0/3488845062 >> v2:172.21.15.69:3300/0 conn(0x7f89c00b5c40 msgr2=0x7f89c00d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89bf7fe640 1 --2- 172.21.15.27:0/3488845062 >> v2:172.21.15.69:3300/0 conn(0x7f89c00b5c40 0x7f89c00d19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89bf7fe640 1 -- 172.21.15.27:0/3488845062 >> v1:172.21.15.69:6789/0 conn(0x7f89c0161690 legacy=0x7f89c00b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.604+0000 7f89bf7fe640 1 -- 172.21.15.27:0/3488845062 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c00ced20 con 0x7f89c01612b0 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.605+0000 7f89bf7fe640 1 --2- 172.21.15.27:0/3488845062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c01612b0 0x7f89c00b2990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89c00b6330 tx=0x7f89b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22c32323e79701d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.605+0000 7f89be7fc640 1 -- 172.21.15.27:0/3488845062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0035020 con 0x7f89c01612b0 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.605+0000 7f89be7fc640 1 -- 172.21.15.27:0/3488845062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b0031870 con 0x7f89c01612b0 2024-03-06T07:22:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.605+0000 7f89be7fc640 1 -- 172.21.15.27:0/3488845062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0031c40 con 0x7f89c01612b0 2024-03-06T07:22:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.606+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3488845062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c01612b0 msgr2=0x7f89c00b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.606+0000 7f89c4ba2640 1 --2- 172.21.15.27:0/3488845062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c01612b0 0x7f89c00b2990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89c00b6330 tx=0x7f89b002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.606+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3488845062 shutdown_connections 2024-03-06T07:22:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.606+0000 7f89c4ba2640 1 --2- 172.21.15.27:0/3488845062 >> v2:172.21.15.69:3300/0 conn(0x7f89c00b5c40 0x7f89c00d19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.606+0000 7f89c4ba2640 1 --2- 172.21.15.27:0/3488845062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c01612b0 0x7f89c00b2990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.606+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3488845062 >> 172.21.15.27:0/3488845062 conn(0x7f89c0067120 msgr2=0x7f89c0067530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.606+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3488845062 shutdown_connections 2024-03-06T07:22:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.606+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3488845062 wait complete. 2024-03-06T07:22:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.607+0000 7f89c4ba2640 1 Processor -- start 2024-03-06T07:22:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.607+0000 7f89c4ba2640 1 -- start start 2024-03-06T07:22:45.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89c4ba2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c00b5c40 0x7f89c01f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89c4ba2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89c01612b0 0x7f89c0153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89c4ba2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89c0161690 0x7f89c0154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89c4ba2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c00b3150 con 0x7f89c00b5c40 2024-03-06T07:22:45.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89c4ba2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c00b2fd0 con 0x7f89c0161690 2024-03-06T07:22:45.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89c4ba2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c00d3560 con 0x7f89c01612b0 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c00b5c40 0x7f89c01f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c00b5c40 0x7f89c01f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58332/0 (socket says 172.21.15.27:58332) 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bf7fe640 1 -- 172.21.15.27:0/3543025973 learned_addr learned my addr 172.21.15.27:0/3543025973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89beffd640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89c01612b0 0x7f89c0153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bffff640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89c0161690 0x7f89c0154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bf7fe640 1 -- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89c01612b0 msgr2=0x7f89c0153dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bf7fe640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89c01612b0 0x7f89c0153dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bf7fe640 1 -- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89c0161690 msgr2=0x7f89c0154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bf7fe640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89c0161690 0x7f89c0154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bf7fe640 1 -- 172.21.15.27:0/3543025973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c015d650 con 0x7f89c00b5c40 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89beffd640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89c01612b0 0x7f89c0153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bffff640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89c0161690 0x7f89c0154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89bf7fe640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c00b5c40 0x7f89c01f3d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89b002b950 tx=0x7f89b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89a7fff640 1 -- 172.21.15.27:0/3543025973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0040070 con 0x7f89c00b5c40 2024-03-06T07:22:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89a7fff640 1 -- 172.21.15.27:0/3543025973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b0030d00 con 0x7f89c00b5c40 2024-03-06T07:22:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c0154940 con 0x7f89c00b5c40 2024-03-06T07:22:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.608+0000 7f89a7fff640 1 -- 172.21.15.27:0/3543025973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0035040 con 0x7f89c00b5c40 2024-03-06T07:22:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.609+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c0154f00 con 0x7f89c00b5c40 2024-03-06T07:22:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.610+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c004f330 con 0x7f89c00b5c40 2024-03-06T07:22:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.610+0000 7f89a7fff640 1 -- 172.21.15.27:0/3543025973 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89b004b060 con 0x7f89c00b5c40 2024-03-06T07:22:45.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.610+0000 7f89a7fff640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89ac06b500 0x7f89ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.611+0000 7f89a7fff640 1 -- 172.21.15.27:0/3543025973 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f89b00abc30 con 0x7f89c00b5c40 2024-03-06T07:22:45.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.611+0000 7f89beffd640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89ac06b500 0x7f89ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.611+0000 7f89beffd640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89ac06b500 0x7f89ac06d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f89c00cfc40 tx=0x7f89b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.615+0000 7f89a7fff640 1 -- 172.21.15.27:0/3543025973 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89b007ae10 con 0x7f89c00b5c40 2024-03-06T07:22:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.796+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f89c00b6990 con 0x7f89c00b5c40 2024-03-06T07:22:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.797+0000 7f89a7fff640 1 -- 172.21.15.27:0/3543025973 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f89b007a710 con 0x7f89c00b5c40 2024-03-06T07:22:45.799 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:45.799 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:22:45.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:22:45.800 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:22:45.800 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:22:45.800 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:22:45.800 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.798+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89ac06b500 msgr2=0x7f89ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.798+0000 7f89c4ba2640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89ac06b500 0x7f89ac06d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f89c00cfc40 tx=0x7f89b4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.798+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c00b5c40 msgr2=0x7f89c01f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.798+0000 7f89c4ba2640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c00b5c40 0x7f89c01f3d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89b002b950 tx=0x7f89b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.799+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 shutdown_connections 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.799+0000 7f89c4ba2640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89ac06b500 0x7f89ac06d9c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.799+0000 7f89c4ba2640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89c0161690 0x7f89c0154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.799+0000 7f89c4ba2640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89c01612b0 0x7f89c0153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.799+0000 7f89c4ba2640 1 --2- 172.21.15.27:0/3543025973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89c00b5c40 0x7f89c01f3d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.799+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 >> 172.21.15.27:0/3543025973 conn(0x7f89c0067120 msgr2=0x7f89c00b34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.799+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 shutdown_connections 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.799+0000 7f89c4ba2640 1 -- 172.21.15.27:0/3543025973 wait complete. 2024-03-06T07:22:45.812 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:45.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.920+0000 7f00b3116640 1 Processor -- start 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.920+0000 7f00b3116640 1 -- start start 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.921+0000 7f00b3116640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac1618e0 0x7f00ac06b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.921+0000 7f00b3116640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00ac0d0150 0x7f00ac0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.921+0000 7f00b3116640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac15f730 con 0x7f00ac0ce4f0 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.921+0000 7f00b3116640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ac0cfce0 con 0x7f00ac0d0150 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.921+0000 7f00b3116640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac15ebb0 con 0x7f00ac1618e0 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.921+0000 7f00b16b3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00ac0d0150 0x7f00ac0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.921+0000 7f00b16b3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00ac0d0150 0x7f00ac0d05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58340/0 (socket says 172.21.15.27:58340) 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.921+0000 7f00b16b3640 1 -- 172.21.15.27:0/1087821088 learned_addr learned my addr 172.21.15.27:0/1087821088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.921+0000 7f00b0eb2640 1 --2- 172.21.15.27:0/1087821088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac1618e0 0x7f00ac06b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.922+0000 7f00b16b3640 1 -- 172.21.15.27:0/1087821088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac1618e0 msgr2=0x7f00ac06b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.922+0000 7f00b16b3640 1 --2- 172.21.15.27:0/1087821088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac1618e0 0x7f00ac06b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.922+0000 7f00b16b3640 1 -- 172.21.15.27:0/1087821088 >> v1:172.21.15.69:6789/0 conn(0x7f00ac0ce4f0 legacy=0x7f00ac0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.922+0000 7f00b16b3640 1 -- 172.21.15.27:0/1087821088 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac15ee90 con 0x7f00ac0d0150 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.922+0000 7f00b16b3640 1 --2- 172.21.15.27:0/1087821088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0d0150 0x7f00ac0d05e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f009c009870 tx=0x7f009c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2aa07faa75491d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.922+0000 7f00ab7fe640 1 -- 172.21.15.27:0/1087821088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c02f840 con 0x7f00ac0d0150 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00ab7fe640 1 -- 172.21.15.27:0/1087821088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c02f9e0 con 0x7f00ac0d0150 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00ab7fe640 1 -- 172.21.15.27:0/1087821088 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c02fdb0 con 0x7f00ac0d0150 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00b3116640 1 -- 172.21.15.27:0/1087821088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0d0150 msgr2=0x7f00ac0d05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00b3116640 1 --2- 172.21.15.27:0/1087821088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0d0150 0x7f00ac0d05e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f009c009870 tx=0x7f009c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00b3116640 1 -- 172.21.15.27:0/1087821088 shutdown_connections 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00b3116640 1 --2- 172.21.15.27:0/1087821088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0d0150 0x7f00ac0d05e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00b3116640 1 --2- 172.21.15.27:0/1087821088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac1618e0 0x7f00ac06b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00b3116640 1 -- 172.21.15.27:0/1087821088 >> 172.21.15.27:0/1087821088 conn(0x7f00ac0b3660 msgr2=0x7f00ac0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00b3116640 1 -- 172.21.15.27:0/1087821088 shutdown_connections 2024-03-06T07:22:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.923+0000 7f00b3116640 1 -- 172.21.15.27:0/1087821088 wait complete. 2024-03-06T07:22:45.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.924+0000 7f00b3116640 1 Processor -- start 2024-03-06T07:22:45.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.924+0000 7f00b3116640 1 -- start start 2024-03-06T07:22:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b3116640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac0ce4f0 0x7f00ac0b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b3116640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac0d0150 0x7f00ac14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b3116640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0b5380 0x7f00ac0c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b3116640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac06b670 con 0x7f00ac0b5380 2024-03-06T07:22:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b3116640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac15f450 con 0x7f00ac0d0150 2024-03-06T07:22:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b3116640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac15e7d0 con 0x7f00ac0ce4f0 2024-03-06T07:22:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b16b3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0b5380 0x7f00ac0c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b16b3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0b5380 0x7f00ac0c65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58352/0 (socket says 172.21.15.27:58352) 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b16b3640 1 -- 172.21.15.27:0/906185760 learned_addr learned my addr 172.21.15.27:0/906185760 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00b0eb2640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac0ce4f0 0x7f00ac0b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.925+0000 7f00abfff640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac0d0150 0x7f00ac14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00b16b3640 1 -- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac0ce4f0 msgr2=0x7f00ac0b5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00b16b3640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac0ce4f0 0x7f00ac0b5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00b16b3640 1 -- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac0d0150 msgr2=0x7f00ac14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00b16b3640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac0d0150 0x7f00ac14d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00b16b3640 1 -- 172.21.15.27:0/906185760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0c6ae0 con 0x7f00ac0b5380 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00b16b3640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0b5380 0x7f00ac0c65a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f009c009870 tx=0x7f009c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00a97fa640 1 -- 172.21.15.27:0/906185760 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c037c40 con 0x7f00ac0b5380 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00a97fa640 1 -- 172.21.15.27:0/906185760 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c046070 con 0x7f00ac0b5380 2024-03-06T07:22:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac0c6d80 con 0x7f00ac0b5380 2024-03-06T07:22:45.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00a97fa640 1 -- 172.21.15.27:0/906185760 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c041400 con 0x7f00ac0b5380 2024-03-06T07:22:45.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.926+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac0b4b80 con 0x7f00ac0b5380 2024-03-06T07:22:45.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.927+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac09fc50 con 0x7f00ac0b5380 2024-03-06T07:22:45.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.927+0000 7f00a97fa640 1 -- 172.21.15.27:0/906185760 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f009c040070 con 0x7f00ac0b5380 2024-03-06T07:22:45.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.928+0000 7f00a97fa640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f009806b4d0 0x7f009806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.928+0000 7f00a97fa640 1 -- 172.21.15.27:0/906185760 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f009c0ac470 con 0x7f00ac0b5380 2024-03-06T07:22:45.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.928+0000 7f00b0eb2640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f009806b4d0 0x7f009806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.929+0000 7f00b0eb2640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f009806b4d0 0x7f009806d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f00a40073f0 tx=0x7f00a400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:45.932+0000 7f00a97fa640 1 -- 172.21.15.27:0/906185760 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f009c033030 con 0x7f00ac0b5380 2024-03-06T07:22:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:46.116+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f00ac072cb0 con 0x7f00ac0b5380 2024-03-06T07:22:47.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.045+0000 7f00a97fa640 1 -- 172.21.15.27:0/906185760 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f009c07b790 con 0x7f00ac0b5380 2024-03-06T07:22:47.048 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:47.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.049+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f009806b4d0 msgr2=0x7f009806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.050+0000 7f00b3116640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f009806b4d0 0x7f009806d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f00a40073f0 tx=0x7f00a400b040 comp rx=0 tx=0).stop 2024-03-06T07:22:47.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.050+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0b5380 msgr2=0x7f00ac0c65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.050+0000 7f00b3116640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0b5380 0x7f00ac0c65a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f009c009870 tx=0x7f009c004730 comp rx=0 tx=0).stop 2024-03-06T07:22:47.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.050+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 shutdown_connections 2024-03-06T07:22:47.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.050+0000 7f00b3116640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f009806b4d0 0x7f009806d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.050+0000 7f00b3116640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac0b5380 0x7f00ac0c65a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.050+0000 7f00b3116640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac0d0150 0x7f00ac14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.050+0000 7f00b3116640 1 --2- 172.21.15.27:0/906185760 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00ac0ce4f0 0x7f00ac0b5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.050+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 >> 172.21.15.27:0/906185760 conn(0x7f00ac0b3660 msgr2=0x7f00ac0cc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.051+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 shutdown_connections 2024-03-06T07:22:47.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.051+0000 7f00b3116640 1 -- 172.21.15.27:0/906185760 wait complete. 2024-03-06T07:22:47.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:47.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.185+0000 7fcc4537a640 1 Processor -- start 2024-03-06T07:22:47.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.186+0000 7fcc4537a640 1 -- start start 2024-03-06T07:22:47.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.186+0000 7fcc4537a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc400717b0 0x7fcc40071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.186+0000 7fcc4537a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4015e4b0 con 0x7fcc400b2b10 2024-03-06T07:22:47.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.186+0000 7fcc4537a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc400b97e0 con 0x7fcc400b3060 2024-03-06T07:22:47.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.186+0000 7fcc4537a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc400b7c70 con 0x7fcc400717b0 2024-03-06T07:22:47.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.186+0000 7fcc3effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc400717b0 0x7fcc40071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.186+0000 7fcc3f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcc400b2b10 0x7fcc400b7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56562/0 (socket says 172.21.15.27:56562) 2024-03-06T07:22:47.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.186+0000 7fcc3f7fe640 1 -- 172.21.15.27:0/3696533303 learned_addr learned my addr 172.21.15.27:0/3696533303 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:47.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.187+0000 7fcc3effd640 1 -- 172.21.15.27:0/3696533303 >> v1:172.21.15.69:6789/0 conn(0x7fcc400b3060 legacy=0x7fcc400b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.187+0000 7fcc3dffb640 1 -- 172.21.15.27:0/3696533303 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779318721 0 0) 0x7fcc30001490 con 0x7fcc400b2b10 2024-03-06T07:22:47.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.187+0000 7fcc3effd640 1 -- 172.21.15.27:0/3696533303 >> v1:172.21.15.27:6789/0 conn(0x7fcc400b2b10 legacy=0x7fcc400b7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.187+0000 7fcc3effd640 1 -- 172.21.15.27:0/3696533303 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc400b9a80 con 0x7fcc400717b0 2024-03-06T07:22:47.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc3effd640 1 --2- 172.21.15.27:0/3696533303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400717b0 0x7fcc40071b90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc380097a0 tx=0x7fcc3802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b52dec86e10ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc3dffb640 1 -- 172.21.15.27:0/3696533303 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38035020 con 0x7fcc400717b0 2024-03-06T07:22:47.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc3dffb640 1 -- 172.21.15.27:0/3696533303 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc38031830 con 0x7fcc400717b0 2024-03-06T07:22:47.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc3dffb640 1 -- 172.21.15.27:0/3696533303 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38031c00 con 0x7fcc400717b0 2024-03-06T07:22:47.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc4537a640 1 -- 172.21.15.27:0/3696533303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400717b0 msgr2=0x7fcc40071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc4537a640 1 --2- 172.21.15.27:0/3696533303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400717b0 0x7fcc40071b90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc380097a0 tx=0x7fcc3802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc4537a640 1 -- 172.21.15.27:0/3696533303 shutdown_connections 2024-03-06T07:22:47.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc4537a640 1 --2- 172.21.15.27:0/3696533303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400717b0 0x7fcc40071b90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc4537a640 1 -- 172.21.15.27:0/3696533303 >> 172.21.15.27:0/3696533303 conn(0x7fcc400d3d20 msgr2=0x7fcc400d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc4537a640 1 -- 172.21.15.27:0/3696533303 shutdown_connections 2024-03-06T07:22:47.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.188+0000 7fcc4537a640 1 -- 172.21.15.27:0/3696533303 wait complete. 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.190+0000 7fcc4537a640 1 Processor -- start 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.190+0000 7fcc4537a640 1 -- start start 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.190+0000 7fcc4537a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400717b0 0x7fcc400cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.190+0000 7fcc4537a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400b2b10 0x7fcc401f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.190+0000 7fcc4537a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc400b3060 0x7fcc40156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.190+0000 7fcc4537a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400d4bf0 con 0x7fcc400b2b10 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.190+0000 7fcc4537a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400d2e10 con 0x7fcc400b3060 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.190+0000 7fcc4537a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400d32a0 con 0x7fcc400717b0 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400b2b10 0x7fcc401f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400b2b10 0x7fcc401f6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58384/0 (socket says 172.21.15.27:58384) 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3e7fc640 1 -- 172.21.15.27:0/228335167 learned_addr learned my addr 172.21.15.27:0/228335167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3f7fe640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc400b3060 0x7fcc40156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3effd640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400717b0 0x7fcc400cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3e7fc640 1 -- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400717b0 msgr2=0x7fcc400cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3e7fc640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400717b0 0x7fcc400cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3e7fc640 1 -- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc400b3060 msgr2=0x7fcc40156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3e7fc640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc400b3060 0x7fcc40156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3e7fc640 1 -- 172.21.15.27:0/228335167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc40159bc0 con 0x7fcc400b2b10 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3f7fe640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc400b3060 0x7fcc40156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3effd640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400717b0 0x7fcc400cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc3e7fc640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400b2b10 0x7fcc401f6b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800b3f0 tx=0x7fcc2800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc377fe640 1 -- 172.21.15.27:0/228335167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28015020 con 0x7fcc400b2b10 2024-03-06T07:22:47.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc377fe640 1 -- 172.21.15.27:0/228335167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc28011830 con 0x7fcc400b2b10 2024-03-06T07:22:47.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc40159ea0 con 0x7fcc400b2b10 2024-03-06T07:22:47.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc377fe640 1 -- 172.21.15.27:0/228335167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28011c20 con 0x7fcc400b2b10 2024-03-06T07:22:47.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.191+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc400cd710 con 0x7fcc400b2b10 2024-03-06T07:22:47.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.192+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc400cc800 con 0x7fcc400b2b10 2024-03-06T07:22:47.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.193+0000 7fcc377fe640 1 -- 172.21.15.27:0/228335167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc28018020 con 0x7fcc400b2b10 2024-03-06T07:22:47.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.193+0000 7fcc377fe640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc2006b4d0 0x7fcc2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.193+0000 7fcc377fe640 1 -- 172.21.15.27:0/228335167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fcc28089bc0 con 0x7fcc400b2b10 2024-03-06T07:22:47.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.193+0000 7fcc3effd640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc2006b4d0 0x7fcc2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.194+0000 7fcc3effd640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc2006b4d0 0x7fcc2006d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc38036000 tx=0x7fcc38008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.198+0000 7fcc377fe640 1 -- 172.21.15.27:0/228335167 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc2801f030 con 0x7fcc400b2b10 2024-03-06T07:22:47.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:47.263 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:47.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:47.380+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcc4004f410 con 0x7fcc400b2b10 2024-03-06T07:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.050+0000 7fcc377fe640 1 -- 172.21.15.27:0/228335167 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcc280172c0 con 0x7fcc400b2b10 2024-03-06T07:22:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc2006b4d0 msgr2=0x7fcc2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc2006b4d0 0x7fcc2006d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc38036000 tx=0x7fcc38008030 comp rx=0 tx=0).stop 2024-03-06T07:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400b2b10 msgr2=0x7fcc401f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400b2b10 0x7fcc401f6b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800b3f0 tx=0x7fcc2800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 shutdown_connections 2024-03-06T07:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc2006b4d0 0x7fcc2006d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc400b3060 0x7fcc40156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc400b2b10 0x7fcc401f6b40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 --2- 172.21.15.27:0/228335167 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc400717b0 0x7fcc400cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 >> 172.21.15.27:0/228335167 conn(0x7fcc400d3d20 msgr2=0x7fcc400b81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 shutdown_connections 2024-03-06T07:22:48.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.053+0000 7fcc4537a640 1 -- 172.21.15.27:0/228335167 wait complete. 2024-03-06T07:22:48.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:48.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.197+0000 7f9667a08640 1 Processor -- start 2024-03-06T07:22:48.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.197+0000 7f9667a08640 1 -- start start 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9667a08640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9660072070 0x7f96600b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9667a08640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96600ce900 0x7f96600ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9667a08640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966006dbb0 con 0x7f96600b4700 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9667a08640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96600724f0 con 0x7f9660072070 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9667a08640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966015b980 con 0x7f96600ce900 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f96657a4640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f96600b4700 0x7f966015b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56566/0 (socket says 172.21.15.27:56566) 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f96657a4640 1 -- 172.21.15.27:0/2456407300 learned_addr learned my addr 172.21.15.27:0/2456407300 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9664fa3640 1 --2- 172.21.15.27:0/2456407300 >> v2:172.21.15.69:3300/0 conn(0x7f9660072070 0x7f96600b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9665fa5640 1 --2- 172.21.15.27:0/2456407300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96600ce900 0x7f96600ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9657fff640 1 -- 172.21.15.27:0/2456407300 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667027477 0 0) 0x7f966006dbb0 con 0x7f96600b4700 2024-03-06T07:22:48.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9657fff640 1 -- 172.21.15.27:0/2456407300 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f964c003520 con 0x7f96600b4700 2024-03-06T07:22:48.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9657fff640 1 -- 172.21.15.27:0/2456407300 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 449173313 0 0) 0x7f964c003520 con 0x7f96600b4700 2024-03-06T07:22:48.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9664fa3640 1 -- 172.21.15.27:0/2456407300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96600ce900 msgr2=0x7f96600ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9664fa3640 1 --2- 172.21.15.27:0/2456407300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96600ce900 0x7f96600ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9664fa3640 1 -- 172.21.15.27:0/2456407300 >> v1:172.21.15.27:6789/0 conn(0x7f96600b4700 legacy=0x7f966015b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9664fa3640 1 -- 172.21.15.27:0/2456407300 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966015bc60 con 0x7f9660072070 2024-03-06T07:22:48.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.198+0000 7f9665fa5640 1 --2- 172.21.15.27:0/2456407300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96600ce900 0x7f96600ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:48.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.199+0000 7f9664fa3640 1 --2- 172.21.15.27:0/2456407300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9660072070 0x7f96600b3620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f96500097a0 tx=0x7f965002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c1e6c90b25afa01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.199+0000 7f9657fff640 1 -- 172.21.15.27:0/2456407300 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9650037070 con 0x7f9660072070 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.199+0000 7f9657fff640 1 -- 172.21.15.27:0/2456407300 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9650030510 con 0x7f9660072070 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.199+0000 7f9657fff640 1 -- 172.21.15.27:0/2456407300 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c001670 con 0x7f9660072070 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.200+0000 7f9667a08640 1 -- 172.21.15.27:0/2456407300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9660072070 msgr2=0x7f96600b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.200+0000 7f9667a08640 1 --2- 172.21.15.27:0/2456407300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9660072070 0x7f96600b3620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f96500097a0 tx=0x7f965002b790 comp rx=0 tx=0).stop 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.200+0000 7f9667a08640 1 -- 172.21.15.27:0/2456407300 shutdown_connections 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.200+0000 7f9667a08640 1 --2- 172.21.15.27:0/2456407300 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96600ce900 0x7f96600ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.200+0000 7f9667a08640 1 --2- 172.21.15.27:0/2456407300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9660072070 0x7f96600b3620 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.200+0000 7f9667a08640 1 -- 172.21.15.27:0/2456407300 >> 172.21.15.27:0/2456407300 conn(0x7f9660066d00 msgr2=0x7f9660067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.200+0000 7f9667a08640 1 -- 172.21.15.27:0/2456407300 shutdown_connections 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.200+0000 7f9667a08640 1 -- 172.21.15.27:0/2456407300 wait complete. 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.201+0000 7f9667a08640 1 Processor -- start 2024-03-06T07:22:48.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.201+0000 7f9667a08640 1 -- start start 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9667a08640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9660072070 0x7f96600ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9667a08640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96600b4700 0x7f966013fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9667a08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96600ce900 0x7f9660142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9667a08640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96600b3ce0 con 0x7f96600ce900 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9667a08640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96600b3b60 con 0x7f96600b4700 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9667a08640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966006e0b0 con 0x7f9660072070 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9665fa5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96600ce900 0x7f9660142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9665fa5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96600ce900 0x7f9660142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58396/0 (socket says 172.21.15.27:58396) 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9665fa5640 1 -- 172.21.15.27:0/3837292863 learned_addr learned my addr 172.21.15.27:0/3837292863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f96657a4640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9660072070 0x7f96600ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9664fa3640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96600b4700 0x7f966013fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9665fa5640 1 -- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9660072070 msgr2=0x7f96600ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9665fa5640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9660072070 0x7f96600ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9665fa5640 1 -- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96600b4700 msgr2=0x7f966013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9665fa5640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96600b4700 0x7f966013fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9665fa5640 1 -- 172.21.15.27:0/3837292863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96600ac910 con 0x7f96600ce900 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.202+0000 7f9665fa5640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96600ce900 0x7f9660142de0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f965800efc0 tx=0x7f965800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.203+0000 7f9655ffb640 1 -- 172.21.15.27:0/3837292863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658018030 con 0x7f96600ce900 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.203+0000 7f9655ffb640 1 -- 172.21.15.27:0/3837292863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965800cdf0 con 0x7f96600ce900 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.203+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96600acbf0 con 0x7f96600ce900 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.203+0000 7f9655ffb640 1 -- 172.21.15.27:0/3837292863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9658003bb0 con 0x7f96600ce900 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.203+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96600ad490 con 0x7f96600ce900 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.204+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96600acdc0 con 0x7f96600ce900 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.204+0000 7f9655ffb640 1 -- 172.21.15.27:0/3837292863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f965801b070 con 0x7f96600ce900 2024-03-06T07:22:48.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.205+0000 7f9655ffb640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964406b4d0 0x7f964406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.205+0000 7f9655ffb640 1 -- 172.21.15.27:0/3837292863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f965808aa40 con 0x7f96600ce900 2024-03-06T07:22:48.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.205+0000 7f96657a4640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964406b4d0 0x7f964406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.205+0000 7f96657a4640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964406b4d0 0x7f964406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f965c0013c0 tx=0x7f965c001350 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.209+0000 7f9655ffb640 1 -- 172.21.15.27:0/3837292863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9658010040 con 0x7f96600ce900 2024-03-06T07:22:48.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:48.431+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f96600733c0 con 0x7f96600ce900 2024-03-06T07:22:49.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.054+0000 7f9655ffb640 1 -- 172.21.15.27:0/3837292863 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f96580592d0 con 0x7f96600ce900 2024-03-06T07:22:49.057 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964406b4d0 msgr2=0x7f964406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964406b4d0 0x7f964406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f965c0013c0 tx=0x7f965c001350 comp rx=0 tx=0).stop 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96600ce900 msgr2=0x7f9660142de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96600ce900 0x7f9660142de0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f965800efc0 tx=0x7f965800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 shutdown_connections 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964406b4d0 0x7f964406d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96600ce900 0x7f9660142de0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96600b4700 0x7f966013fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 --2- 172.21.15.27:0/3837292863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9660072070 0x7f96600ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 >> 172.21.15.27:0/3837292863 conn(0x7f9660066d00 msgr2=0x7f96600716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 shutdown_connections 2024-03-06T07:22:49.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.057+0000 7f9667a08640 1 -- 172.21.15.27:0/3837292863 wait complete. 2024-03-06T07:22:49.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:49.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.205+0000 7fa402355640 1 Processor -- start 2024-03-06T07:22:49.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.205+0000 7fa402355640 1 -- start start 2024-03-06T07:22:49.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa402355640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3fc15df70 0x7fa3fc0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa402355640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 0x7fa3fc071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa402355640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc0b6f20 con 0x7fa3fc071a80 2024-03-06T07:22:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa402355640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc0d3030 con 0x7fa3fc15df70 2024-03-06T07:22:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa402355640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0d3310 con 0x7fa3fc0d18c0 2024-03-06T07:22:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa3fbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3fc15df70 0x7fa3fc0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa3fbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3fc15df70 0x7fa3fc0d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58412/0 (socket says 172.21.15.27:58412) 2024-03-06T07:22:49.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa3fbfff640 1 -- 172.21.15.27:0/99545843 learned_addr learned my addr 172.21.15.27:0/99545843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:49.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa3fb7fe640 1 --2- 172.21.15.27:0/99545843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 0x7fa3fc071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa3fbfff640 1 -- 172.21.15.27:0/99545843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 msgr2=0x7fa3fc071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa3fbfff640 1 --2- 172.21.15.27:0/99545843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 0x7fa3fc071540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa3fbfff640 1 -- 172.21.15.27:0/99545843 >> v1:172.21.15.69:6789/0 conn(0x7fa3fc071a80 legacy=0x7fa3fc0b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.206+0000 7fa3fbfff640 1 -- 172.21.15.27:0/99545843 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0d35f0 con 0x7fa3fc15df70 2024-03-06T07:22:49.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.207+0000 7fa3fbfff640 1 --2- 172.21.15.27:0/99545843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc15df70 0x7fa3fc0d27b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009870 tx=0x7fa3e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2d9f5e1bcfa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.207+0000 7fa3faffd640 1 -- 172.21.15.27:0/99545843 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4034070 con 0x7fa3fc15df70 2024-03-06T07:22:49.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.207+0000 7fa3faffd640 1 -- 172.21.15.27:0/99545843 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e4030840 con 0x7fa3fc15df70 2024-03-06T07:22:49.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.207+0000 7fa3faffd640 1 -- 172.21.15.27:0/99545843 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4030c10 con 0x7fa3fc15df70 2024-03-06T07:22:49.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.208+0000 7fa402355640 1 -- 172.21.15.27:0/99545843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc15df70 msgr2=0x7fa3fc0d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.208+0000 7fa402355640 1 --2- 172.21.15.27:0/99545843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc15df70 0x7fa3fc0d27b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009870 tx=0x7fa3e402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:49.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.208+0000 7fa402355640 1 -- 172.21.15.27:0/99545843 shutdown_connections 2024-03-06T07:22:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.208+0000 7fa402355640 1 --2- 172.21.15.27:0/99545843 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 0x7fa3fc071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.208+0000 7fa402355640 1 --2- 172.21.15.27:0/99545843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc15df70 0x7fa3fc0d27b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.208+0000 7fa402355640 1 -- 172.21.15.27:0/99545843 >> 172.21.15.27:0/99545843 conn(0x7fa3fc0d3cf0 msgr2=0x7fa3fc0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.208+0000 7fa402355640 1 -- 172.21.15.27:0/99545843 shutdown_connections 2024-03-06T07:22:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.208+0000 7fa402355640 1 -- 172.21.15.27:0/99545843 wait complete. 2024-03-06T07:22:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.209+0000 7fa402355640 1 Processor -- start 2024-03-06T07:22:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.209+0000 7fa402355640 1 -- start start 2024-03-06T07:22:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa402355640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc071a80 0x7fa3fc14e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa402355640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 0x7fa3fc0cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa402355640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3fc15df70 0x7fa3fc1607d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa402355640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc15e350 con 0x7fa3fc071a80 2024-03-06T07:22:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa402355640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0d2cf0 con 0x7fa3fc15df70 2024-03-06T07:22:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa402355640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc15e4d0 con 0x7fa3fc0d18c0 2024-03-06T07:22:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa3fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc071a80 0x7fa3fc14e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa3fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc071a80 0x7fa3fc14e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58414/0 (socket says 172.21.15.27:58414) 2024-03-06T07:22:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa3fbfff640 1 -- 172.21.15.27:0/2137099676 learned_addr learned my addr 172.21.15.27:0/2137099676 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa4008f2640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3fc15df70 0x7fa3fc1607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa3fb7fe640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 0x7fa3fc0cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa3fbfff640 1 -- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 msgr2=0x7fa3fc0cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa3fbfff640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 0x7fa3fc0cadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa3fbfff640 1 -- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3fc15df70 msgr2=0x7fa3fc1607d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa3fbfff640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3fc15df70 0x7fa3fc1607d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa3fbfff640 1 -- 172.21.15.27:0/2137099676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0cd8e0 con 0x7fa3fc071a80 2024-03-06T07:22:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.210+0000 7fa4008f2640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3fc15df70 0x7fa3fc1607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:49.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.211+0000 7fa3fbfff640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc071a80 0x7fa3fc14e0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4000c00 tx=0x7fa3e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.211+0000 7fa3f8ff9640 1 -- 172.21.15.27:0/2137099676 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4034040 con 0x7fa3fc071a80 2024-03-06T07:22:49.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.211+0000 7fa3f8ff9640 1 -- 172.21.15.27:0/2137099676 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e4039d90 con 0x7fa3fc071a80 2024-03-06T07:22:49.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.211+0000 7fa3f8ff9640 1 -- 172.21.15.27:0/2137099676 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4043610 con 0x7fa3fc071a80 2024-03-06T07:22:49.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.211+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc0cdbc0 con 0x7fa3fc071a80 2024-03-06T07:22:49.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.211+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc0ce180 con 0x7fa3fc071a80 2024-03-06T07:22:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.212+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc067ff0 con 0x7fa3fc071a80 2024-03-06T07:22:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.212+0000 7fa3f8ff9640 1 -- 172.21.15.27:0/2137099676 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3e404a060 con 0x7fa3fc071a80 2024-03-06T07:22:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.212+0000 7fa3f8ff9640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3dc06b4d0 0x7fa3dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.213+0000 7fa3f8ff9640 1 -- 172.21.15.27:0/2137099676 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3e40ac1d0 con 0x7fa3fc071a80 2024-03-06T07:22:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.213+0000 7fa3fb7fe640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3dc06b4d0 0x7fa3dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.213+0000 7fa3fb7fe640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3dc06b4d0 0x7fa3dc06d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc151da0 tx=0x7fa3ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.217+0000 7fa3f8ff9640 1 -- 172.21.15.27:0/2137099676 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3e403e070 con 0x7fa3fc071a80 2024-03-06T07:22:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.398+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3fc072170 con 0x7fa3fc071a80 2024-03-06T07:22:49.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.399+0000 7fa3f8ff9640 1 -- 172.21.15.27:0/2137099676 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16546 (secure 0 0 0) 0x7fa3e402d330 con 0x7fa3fc071a80 2024-03-06T07:22:49.402 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:49.402 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:48.433026+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3dc06b4d0 msgr2=0x7fa3dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3dc06b4d0 0x7fa3dc06d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc151da0 tx=0x7fa3ec008040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc071a80 msgr2=0x7fa3fc14e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc071a80 0x7fa3fc14e0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4000c00 tx=0x7fa3e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 shutdown_connections 2024-03-06T07:22:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3dc06b4d0 0x7fa3dc06d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3fc15df70 0x7fa3fc1607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3fc0d18c0 0x7fa3fc0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 --2- 172.21.15.27:0/2137099676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3fc071a80 0x7fa3fc14e0c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 >> 172.21.15.27:0/2137099676 conn(0x7fa3fc0d3cf0 msgr2=0x7fa3fc0b7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 shutdown_connections 2024-03-06T07:22:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.401+0000 7fa402355640 1 -- 172.21.15.27:0/2137099676 wait complete. 2024-03-06T07:22:49.415 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.519+0000 7fa78451e640 1 Processor -- start 2024-03-06T07:22:49.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.519+0000 7fa78451e640 1 -- start start 2024-03-06T07:22:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa78451e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa77c15e060 0x7fa77c0ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa78451e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c0d0880 0x7fa77c0ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa78451e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0d0dd0 con 0x7fa77c06fe20 2024-03-06T07:22:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa78451e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c0d10b0 con 0x7fa77c15e060 2024-03-06T07:22:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa78451e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0cc0e0 con 0x7fa77c0d0880 2024-03-06T07:22:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa7822ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa77c15e060 0x7fa77c0ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa782abb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c0d0880 0x7fa77c0ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa7822ba640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa77c15e060 0x7fa77c0ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:38330/0 (socket says 172.21.15.27:38330) 2024-03-06T07:22:49.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa781ab9640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa77c06fe20 0x7fa77c0ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:43768/0 (socket says 172.21.15.27:43768) 2024-03-06T07:22:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa7822ba640 1 -- 172.21.15.27:0/258583956 learned_addr learned my addr 172.21.15.27:0/258583956 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.520+0000 7fa782abb640 1 --2- 172.21.15.27:0/258583956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c0d0880 0x7fa77c0ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:60528/0 (socket says 172.21.15.27:60528) 2024-03-06T07:22:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.521+0000 7fa7822ba640 1 -- 172.21.15.27:0/258583956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c0d0880 msgr2=0x7fa77c0ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.521+0000 7fa7822ba640 1 --2- 172.21.15.27:0/258583956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c0d0880 0x7fa77c0ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.521+0000 7fa7822ba640 1 -- 172.21.15.27:0/258583956 >> v1:172.21.15.69:6789/0 conn(0x7fa77c06fe20 legacy=0x7fa77c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.521+0000 7fa7822ba640 1 -- 172.21.15.27:0/258583956 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c0cc3c0 con 0x7fa77c15e060 2024-03-06T07:22:49.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.521+0000 7fa782abb640 1 --2- 172.21.15.27:0/258583956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c0d0880 0x7fa77c0ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:49.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.521+0000 7fa7822ba640 1 --2- 172.21.15.27:0/258583956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c15e060 0x7fa77c0ce310 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa778009870 tx=0x7fa77802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=546b2a463d9da912 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.521+0000 7fa7812b8640 1 -- 172.21.15.27:0/258583956 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778034070 con 0x7fa77c15e060 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa7812b8640 1 -- 172.21.15.27:0/258583956 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778030840 con 0x7fa77c15e060 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa7812b8640 1 -- 172.21.15.27:0/258583956 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778030c10 con 0x7fa77c15e060 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa78451e640 1 -- 172.21.15.27:0/258583956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c15e060 msgr2=0x7fa77c0ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa78451e640 1 --2- 172.21.15.27:0/258583956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c15e060 0x7fa77c0ce310 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa778009870 tx=0x7fa77802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa78451e640 1 -- 172.21.15.27:0/258583956 shutdown_connections 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa78451e640 1 --2- 172.21.15.27:0/258583956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c0d0880 0x7fa77c0ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa78451e640 1 --2- 172.21.15.27:0/258583956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c15e060 0x7fa77c0ce310 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa78451e640 1 -- 172.21.15.27:0/258583956 >> 172.21.15.27:0/258583956 conn(0x7fa77c0d1a90 msgr2=0x7fa77c0d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa78451e640 1 -- 172.21.15.27:0/258583956 shutdown_connections 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.522+0000 7fa78451e640 1 -- 172.21.15.27:0/258583956 wait complete. 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.523+0000 7fa78451e640 1 Processor -- start 2024-03-06T07:22:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.523+0000 7fa78451e640 1 -- start start 2024-03-06T07:22:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa78451e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c06fe20 0x7fa77c157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa78451e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c0d0880 0x7fa77c1600f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa78451e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c15e060 0x7fa77c160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa78451e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0cef60 con 0x7fa77c15e060 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa78451e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c15e440 con 0x7fa77c0d0880 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa78451e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0cf0e0 con 0x7fa77c06fe20 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa782abb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c15e060 0x7fa77c160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa782abb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c15e060 0x7fa77c160630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58422/0 (socket says 172.21.15.27:58422) 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa782abb640 1 -- 172.21.15.27:0/1834411385 learned_addr learned my addr 172.21.15.27:0/1834411385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa7822ba640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c06fe20 0x7fa77c157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa781ab9640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c0d0880 0x7fa77c1600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa782abb640 1 -- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c06fe20 msgr2=0x7fa77c157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa782abb640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c06fe20 0x7fa77c157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa782abb640 1 -- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c0d0880 msgr2=0x7fa77c1600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa782abb640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c0d0880 0x7fa77c1600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa782abb640 1 -- 172.21.15.27:0/1834411385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c1585c0 con 0x7fa77c15e060 2024-03-06T07:22:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.524+0000 7fa781ab9640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c0d0880 0x7fa77c1600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.525+0000 7fa782abb640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c15e060 0x7fa77c160630 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa77400efc0 tx=0x7fa774009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.525+0000 7fa772ffd640 1 -- 172.21.15.27:0/1834411385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774018030 con 0x7fa77c15e060 2024-03-06T07:22:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.525+0000 7fa772ffd640 1 -- 172.21.15.27:0/1834411385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77400cde0 con 0x7fa77c15e060 2024-03-06T07:22:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.525+0000 7fa772ffd640 1 -- 172.21.15.27:0/1834411385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774003bb0 con 0x7fa77c15e060 2024-03-06T07:22:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.525+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c15e810 con 0x7fa77c15e060 2024-03-06T07:22:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.525+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c15edd0 con 0x7fa77c15e060 2024-03-06T07:22:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.526+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c0c9310 con 0x7fa77c15e060 2024-03-06T07:22:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.526+0000 7fa772ffd640 1 -- 172.21.15.27:0/1834411385 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa77401b070 con 0x7fa77c15e060 2024-03-06T07:22:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.527+0000 7fa772ffd640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa76006b4d0 0x7fa76006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.527+0000 7fa772ffd640 1 -- 172.21.15.27:0/1834411385 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa77408ac60 con 0x7fa77c15e060 2024-03-06T07:22:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.527+0000 7fa7822ba640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa76006b4d0 0x7fa76006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.527+0000 7fa7822ba640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa76006b4d0 0x7fa76006d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa778036000 tx=0x7fa77803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.531+0000 7fa772ffd640 1 -- 172.21.15.27:0/1834411385 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa774010040 con 0x7fa77c15e060 2024-03-06T07:22:49.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.716+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa77c067ff0 con 0x7fa77c15e060 2024-03-06T07:22:49.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.717+0000 7fa772ffd640 1 -- 172.21.15.27:0/1834411385 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16546 (secure 0 0 0) 0x7fa774012340 con 0x7fa77c15e060 2024-03-06T07:22:49.720 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:49.720 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:48.433026+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:49.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa76006b4d0 msgr2=0x7fa76006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa76006b4d0 0x7fa76006d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa778036000 tx=0x7fa77803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c15e060 msgr2=0x7fa77c160630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c15e060 0x7fa77c160630 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa77400efc0 tx=0x7fa774009f90 comp rx=0 tx=0).stop 2024-03-06T07:22:49.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 shutdown_connections 2024-03-06T07:22:49.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa76006b4d0 0x7fa76006d990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c15e060 0x7fa77c160630 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa77c0d0880 0x7fa77c1600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 --2- 172.21.15.27:0/1834411385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa77c06fe20 0x7fa77c157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 >> 172.21.15.27:0/1834411385 conn(0x7fa77c0d1a90 msgr2=0x7fa77c0ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 shutdown_connections 2024-03-06T07:22:49.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.719+0000 7fa78451e640 1 -- 172.21.15.27:0/1834411385 wait complete. 2024-03-06T07:22:49.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:22:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.840+0000 7f5c9340e640 1 Processor -- start 2024-03-06T07:22:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.840+0000 7f5c9340e640 1 -- start start 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c9340e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c8c0b3060 0x7f5c8c071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c9340e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0b2b10 0x7f5c8c0d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c9340e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c15e4b0 con 0x7f5c8c0717b0 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c9340e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c0d2e10 con 0x7f5c8c0b3060 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c9340e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0b97e0 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c909a9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c8c0b3060 0x7f5c8c071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c909a9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c8c0b3060 0x7f5c8c071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58432/0 (socket says 172.21.15.27:58432) 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c909a9640 1 -- 172.21.15.27:0/59772624 learned_addr learned my addr 172.21.15.27:0/59772624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c919ab640 1 --2- 172.21.15.27:0/59772624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0b2b10 0x7f5c8c0d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c909a9640 1 -- 172.21.15.27:0/59772624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0b2b10 msgr2=0x7f5c8c0d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c909a9640 1 --2- 172.21.15.27:0/59772624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0b2b10 0x7f5c8c0d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c909a9640 1 -- 172.21.15.27:0/59772624 >> v1:172.21.15.69:6789/0 conn(0x7f5c8c0717b0 legacy=0x7f5c8c0b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.841+0000 7f5c909a9640 1 -- 172.21.15.27:0/59772624 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c0b9a80 con 0x7f5c8c0b3060 2024-03-06T07:22:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.842+0000 7f5c909a9640 1 --2- 172.21.15.27:0/59772624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b3060 0x7f5c8c071b90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c009870 tx=0x7f5c7c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6904d1a9d43f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.842+0000 7f5c83fff640 1 -- 172.21.15.27:0/59772624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c034070 con 0x7f5c8c0b3060 2024-03-06T07:22:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.842+0000 7f5c83fff640 1 -- 172.21.15.27:0/59772624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7c030840 con 0x7f5c8c0b3060 2024-03-06T07:22:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.842+0000 7f5c83fff640 1 -- 172.21.15.27:0/59772624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c030c10 con 0x7f5c8c0b3060 2024-03-06T07:22:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.842+0000 7f5c9340e640 1 -- 172.21.15.27:0/59772624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b3060 msgr2=0x7f5c8c071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.843+0000 7f5c9340e640 1 --2- 172.21.15.27:0/59772624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b3060 0x7f5c8c071b90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c009870 tx=0x7f5c7c02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:49.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.843+0000 7f5c9340e640 1 -- 172.21.15.27:0/59772624 shutdown_connections 2024-03-06T07:22:49.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.843+0000 7f5c9340e640 1 --2- 172.21.15.27:0/59772624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b3060 0x7f5c8c071b90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.843+0000 7f5c9340e640 1 --2- 172.21.15.27:0/59772624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0b2b10 0x7f5c8c0d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.843+0000 7f5c9340e640 1 -- 172.21.15.27:0/59772624 >> 172.21.15.27:0/59772624 conn(0x7f5c8c0d3d20 msgr2=0x7f5c8c0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.843+0000 7f5c9340e640 1 -- 172.21.15.27:0/59772624 shutdown_connections 2024-03-06T07:22:49.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.843+0000 7f5c9340e640 1 -- 172.21.15.27:0/59772624 wait complete. 2024-03-06T07:22:49.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.844+0000 7f5c9340e640 1 Processor -- start 2024-03-06T07:22:49.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.844+0000 7f5c9340e640 1 -- start start 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c9340e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0717b0 0x7f5c8c0ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c9340e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b2b10 0x7f5c8c1e42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c9340e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8c0b3060 0x7f5c8c1e4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c9340e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0b7560 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c9340e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0d4bf0 con 0x7f5c8c0b3060 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c9340e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0b76e0 con 0x7f5c8c0717b0 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c909a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b2b10 0x7f5c8c1e42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c909a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b2b10 0x7f5c8c1e42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58442/0 (socket says 172.21.15.27:58442) 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c909a9640 1 -- 172.21.15.27:0/2705634737 learned_addr learned my addr 172.21.15.27:0/2705634737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c911aa640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0717b0 0x7f5c8c0ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c919ab640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8c0b3060 0x7f5c8c1e4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c909a9640 1 -- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0717b0 msgr2=0x7f5c8c0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c909a9640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0717b0 0x7f5c8c0ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c909a9640 1 -- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8c0b3060 msgr2=0x7f5c8c1e4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c909a9640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8c0b3060 0x7f5c8c1e4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c909a9640 1 -- 172.21.15.27:0/2705634737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c0cd070 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c909a9640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b2b10 0x7f5c8c1e42c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c000c00 tx=0x7f5c7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c81ffb640 1 -- 172.21.15.27:0/2705634737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c034040 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c81ffb640 1 -- 172.21.15.27:0/2705634737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7c039d90 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c0cd350 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.845+0000 7f5c81ffb640 1 -- 172.21.15.27:0/2705634737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c043630 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.846+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c0cd910 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.846+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c06ce40 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.847+0000 7f5c81ffb640 1 -- 172.21.15.27:0/2705634737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c7c04a060 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.847+0000 7f5c81ffb640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c7006b4d0 0x7f5c7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.848+0000 7f5c81ffb640 1 -- 172.21.15.27:0/2705634737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c7c0ac040 con 0x7f5c8c0b2b10 2024-03-06T07:22:49.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.848+0000 7f5c911aa640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c7006b4d0 0x7f5c7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.848+0000 7f5c911aa640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c7006b4d0 0x7f5c7006d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5c88001780 tx=0x7f5c88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:49.852+0000 7f5c81ffb640 1 -- 172.21.15.27:0/2705634737 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c7c03e070 con 0x7f5c8c0b2b10 2024-03-06T07:22:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:50.073+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5c8c06b8e0 con 0x7f5c8c0b2b10 2024-03-06T07:22:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.078+0000 7f5c81ffb640 1 -- 172.21.15.27:0/2705634737 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5c7c07a870 con 0x7f5c8c0b2b10 2024-03-06T07:22:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c7006b4d0 msgr2=0x7f5c7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c7006b4d0 0x7f5c7006d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5c88001780 tx=0x7f5c88009040 comp rx=0 tx=0).stop 2024-03-06T07:22:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b2b10 msgr2=0x7f5c8c1e42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b2b10 0x7f5c8c1e42c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c000c00 tx=0x7f5c7c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 shutdown_connections 2024-03-06T07:22:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c7006b4d0 0x7f5c7006d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8c0b3060 0x7f5c8c1e4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c8c0b2b10 0x7f5c8c1e42c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 --2- 172.21.15.27:0/2705634737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c8c0717b0 0x7f5c8c0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 >> 172.21.15.27:0/2705634737 conn(0x7f5c8c0d3d20 msgr2=0x7f5c8c15c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 shutdown_connections 2024-03-06T07:22:51.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.080+0000 7f5c9340e640 1 -- 172.21.15.27:0/2705634737 wait complete. 2024-03-06T07:22:51.093 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:51.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:51.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.201+0000 7f91d553e640 1 Processor -- start 2024-03-06T07:22:51.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.201+0000 7f91d553e640 1 -- start start 2024-03-06T07:22:51.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91d553e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d00b40c0 0x7f91d0168ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91d553e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d006f070 con 0x7f91d015dbd0 2024-03-06T07:22:51.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91d553e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d0071940 con 0x7f91d00b5840 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91d553e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00b5db0 con 0x7f91d00b40c0 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91ceffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f91d015dbd0 0x7f91d015ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56568/0 (socket says 172.21.15.27:56568) 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91ceffd640 1 -- 172.21.15.27:0/422305753 learned_addr learned my addr 172.21.15.27:0/422305753 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296187682 0 0) 0x7f91d006f070 con 0x7f91d015dbd0 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91cf7fe640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d00b40c0 0x7f91d0168ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b8003520 con 0x7f91d015dbd0 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3532057688 0 0) 0x7f91b8003520 con 0x7f91d015dbd0 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d00b40c0 msgr2=0x7f91d0168ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91cdffb640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d00b40c0 0x7f91d0168ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 >> v1:172.21.15.69:6789/0 conn(0x7f91d00b5840 legacy=0x7f91d00b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.202+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0071c00 con 0x7f91d015dbd0 2024-03-06T07:22:51.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cf7fe640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d00b40c0 0x7f91d0168ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f91c0001690 con 0x7f91d015dbd0 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 >> v1:172.21.15.27:6789/0 conn(0x7f91d015dbd0 legacy=0x7f91d015ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cdffb640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d00b40c0 0x7f91b80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cdffb640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b8003fd0 0x7f91b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cdffb640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91b8004d40 0x7f91b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00b5db0 con 0x7f91b8003fd0 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b800bb20 con 0x7f91b8004d40 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b800bd20 con 0x7f91d00b40c0 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91ce7fc640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b8003fd0 0x7f91b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91ceffd640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d00b40c0 0x7f91b80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91cf7fe640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91b8004d40 0x7f91b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91ce7fc640 1 -- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d00b40c0 msgr2=0x7f91b80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91ce7fc640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d00b40c0 0x7f91b80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91ce7fc640 1 -- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91b8004d40 msgr2=0x7f91b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91ce7fc640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91b8004d40 0x7f91b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.203+0000 7f91ce7fc640 1 -- 172.21.15.27:0/422305753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b8003520 con 0x7f91b8003fd0 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.204+0000 7f91ce7fc640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b8003fd0 0x7f91b8004800 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f91c4001750 tx=0x7f91b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a0e96c8ff8b1d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.204+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c4001370 con 0x7f91b8003fd0 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.204+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c40052d0 con 0x7f91b8003fd0 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.204+0000 7f91cdffb640 1 -- 172.21.15.27:0/422305753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c4003830 con 0x7f91b8003fd0 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.205+0000 7f91d553e640 1 -- 172.21.15.27:0/422305753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b8003fd0 msgr2=0x7f91b8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.205+0000 7f91d553e640 1 --2- 172.21.15.27:0/422305753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b8003fd0 0x7f91b8004800 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f91c4001750 tx=0x7f91b8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.205+0000 7f91cf7fe640 1 -- 172.21.15.27:0/422305753 reap_dead start 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.205+0000 7f91d553e640 1 -- 172.21.15.27:0/422305753 shutdown_connections 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.205+0000 7f91d553e640 1 -- 172.21.15.27:0/422305753 >> 172.21.15.27:0/422305753 conn(0x7f91d00b26b0 msgr2=0x7f91d00b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.205+0000 7f91d553e640 1 -- 172.21.15.27:0/422305753 shutdown_connections 2024-03-06T07:22:51.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.205+0000 7f91d553e640 1 -- 172.21.15.27:0/422305753 wait complete. 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.207+0000 7f91d553e640 1 Processor -- start 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.207+0000 7f91d553e640 1 -- start start 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.208+0000 7f91d553e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91b8003fd0 0x7f91d0148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.208+0000 7f91d553e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d0161820 0x7f91d0161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.208+0000 7f91d553e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d015f830 0x7f91d015fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.208+0000 7f91d553e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00726e0 con 0x7f91d0161820 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.208+0000 7f91d553e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00721c0 con 0x7f91d015f830 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.208+0000 7f91d553e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d0072860 con 0x7f91b8003fd0 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.208+0000 7f91ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d0161820 0x7f91d0161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.208+0000 7f91ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d0161820 0x7f91d0161ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58452/0 (socket says 172.21.15.27:58452) 2024-03-06T07:22:51.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.208+0000 7f91ce7fc640 1 -- 172.21.15.27:0/2204927102 learned_addr learned my addr 172.21.15.27:0/2204927102 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91ceffd640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91b8003fd0 0x7f91d0148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91cf7fe640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d015f830 0x7f91d015fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91ce7fc640 1 -- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91b8003fd0 msgr2=0x7f91d0148430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91ce7fc640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91b8003fd0 0x7f91d0148430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91ce7fc640 1 -- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d015f830 msgr2=0x7f91d015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91ce7fc640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d015f830 0x7f91d015fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91ce7fc640 1 -- 172.21.15.27:0/2204927102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d01f77b0 con 0x7f91d0161820 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91ce7fc640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d0161820 0x7f91d0161ca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91c4008fd0 tx=0x7f91c4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91b77fe640 1 -- 172.21.15.27:0/2204927102 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c4041070 con 0x7f91d0161820 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91b77fe640 1 -- 172.21.15.27:0/2204927102 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c4005480 con 0x7f91d0161820 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d01f7a50 con 0x7f91d0161820 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91b77fe640 1 -- 172.21.15.27:0/2204927102 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c4001370 con 0x7f91d0161820 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.209+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d01f8010 con 0x7f91d0161820 2024-03-06T07:22:51.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.210+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d01621e0 con 0x7f91d0161820 2024-03-06T07:22:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.211+0000 7f91b77fe640 1 -- 172.21.15.27:0/2204927102 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91c4044070 con 0x7f91d0161820 2024-03-06T07:22:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.211+0000 7f91b77fe640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91bc06b4d0 0x7f91bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.211+0000 7f91b77fe640 1 -- 172.21.15.27:0/2204927102 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91c40abd80 con 0x7f91d0161820 2024-03-06T07:22:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.211+0000 7f91ceffd640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91bc06b4d0 0x7f91bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.212+0000 7f91ceffd640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91bc06b4d0 0x7f91bc06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f91d006f240 tx=0x7f91c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.215+0000 7f91b77fe640 1 -- 172.21.15.27:0/2204927102 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91c407ac90 con 0x7f91d0161820 2024-03-06T07:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:51.434+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f91d0068820 con 0x7f91d0161820 2024-03-06T07:22:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.083+0000 7f91b77fe640 1 -- 172.21.15.27:0/2204927102 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f91c407a590 con 0x7f91d0161820 2024-03-06T07:22:52.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91bc06b4d0 msgr2=0x7f91bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91bc06b4d0 0x7f91bc06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f91d006f240 tx=0x7f91c0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d0161820 msgr2=0x7f91d0161ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d0161820 0x7f91d0161ca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91c4008fd0 tx=0x7f91c4001120 comp rx=0 tx=0).stop 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 shutdown_connections 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91bc06b4d0 0x7f91bc06d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d015f830 0x7f91d015fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d0161820 0x7f91d0161ca0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 --2- 172.21.15.27:0/2204927102 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91b8003fd0 0x7f91d0148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 >> 172.21.15.27:0/2204927102 conn(0x7f91d00b26b0 msgr2=0x7f91d015ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 shutdown_connections 2024-03-06T07:22:52.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.085+0000 7f91d553e640 1 -- 172.21.15.27:0/2204927102 wait complete. 2024-03-06T07:22:52.098 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:52.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.203+0000 7f779f19a640 1 Processor -- start 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.203+0000 7f779f19a640 1 -- start start 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f779f19a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77980d0940 0x7f7798162390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f779f19a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 0x7f779806f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f779f19a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77980d1090 con 0x7f77980cebb0 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f779f19a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779806b0a0 con 0x7f77980d0940 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f779f19a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779806b380 con 0x7f779806ef80 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f7797fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77980d0940 0x7f7798162390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f779cf36640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f77980cebb0 0x7f7798161c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56572/0 (socket says 172.21.15.27:56572) 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f779cf36640 1 -- 172.21.15.27:0/2066342913 learned_addr learned my addr 172.21.15.27:0/2066342913 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f779d737640 1 --2- 172.21.15.27:0/2066342913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 0x7f779806f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f7797fff640 1 -- 172.21.15.27:0/2066342913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 msgr2=0x7f779806f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f7797fff640 1 --2- 172.21.15.27:0/2066342913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 0x7f779806f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f7797fff640 1 -- 172.21.15.27:0/2066342913 >> v1:172.21.15.27:6789/0 conn(0x7f77980cebb0 legacy=0x7f7798161c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f7797fff640 1 -- 172.21.15.27:0/2066342913 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779806b660 con 0x7f77980d0940 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.204+0000 7f77977fe640 1 -- 172.21.15.27:0/2066342913 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13375857 0 0) 0x7f77980d1090 con 0x7f77980cebb0 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.205+0000 7f7797fff640 1 --2- 172.21.15.27:0/2066342913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980d0940 0x7f7798162390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f77840097a0 tx=0x7f778402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc2c8de1b1abdb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.205+0000 7f77977fe640 1 -- 172.21.15.27:0/2066342913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784035020 con 0x7f77980d0940 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.205+0000 7f77977fe640 1 -- 172.21.15.27:0/2066342913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7784031830 con 0x7f77980d0940 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.205+0000 7f77977fe640 1 -- 172.21.15.27:0/2066342913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784031c00 con 0x7f77980d0940 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.206+0000 7f779f19a640 1 -- 172.21.15.27:0/2066342913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980d0940 msgr2=0x7f7798162390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.206+0000 7f779f19a640 1 --2- 172.21.15.27:0/2066342913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980d0940 0x7f7798162390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f77840097a0 tx=0x7f778402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.206+0000 7f779f19a640 1 -- 172.21.15.27:0/2066342913 shutdown_connections 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.206+0000 7f779f19a640 1 --2- 172.21.15.27:0/2066342913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 0x7f779806f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.206+0000 7f779f19a640 1 --2- 172.21.15.27:0/2066342913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980d0940 0x7f7798162390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.206+0000 7f779f19a640 1 -- 172.21.15.27:0/2066342913 >> 172.21.15.27:0/2066342913 conn(0x7f77980b35d0 msgr2=0x7f77980b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.206+0000 7f779f19a640 1 -- 172.21.15.27:0/2066342913 shutdown_connections 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.206+0000 7f779f19a640 1 -- 172.21.15.27:0/2066342913 wait complete. 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.207+0000 7f779f19a640 1 Processor -- start 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.207+0000 7f779f19a640 1 -- start start 2024-03-06T07:22:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f779f19a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 0x7f77980c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f779f19a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cebb0 0x7f77980b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f779f19a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77980d0940 0x7f77980afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f779f19a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7798162a50 con 0x7f77980cebb0 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f779f19a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77981628d0 con 0x7f77980d0940 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f779f19a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779806f950 con 0x7f779806ef80 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f7797fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cebb0 0x7f77980b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f7797fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cebb0 0x7f77980b6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58460/0 (socket says 172.21.15.27:58460) 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f7797fff640 1 -- 172.21.15.27:0/2610307319 learned_addr learned my addr 172.21.15.27:0/2610307319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f779cf36640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 0x7f77980c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f779d737640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77980d0940 0x7f77980afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f7797fff640 1 -- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 msgr2=0x7f77980c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f7797fff640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 0x7f77980c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f7797fff640 1 -- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77980d0940 msgr2=0x7f77980afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f7797fff640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77980d0940 0x7f77980afa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.208+0000 7f7797fff640 1 -- 172.21.15.27:0/2610307319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77980c8b90 con 0x7f77980cebb0 2024-03-06T07:22:52.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.209+0000 7f7797fff640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cebb0 0x7f77980b6f10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f77840096f0 tx=0x7f77840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.209+0000 7f77957fa640 1 -- 172.21.15.27:0/2610307319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784040070 con 0x7f77980cebb0 2024-03-06T07:22:52.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.209+0000 7f77957fa640 1 -- 172.21.15.27:0/2610307319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7784039a20 con 0x7f77980cebb0 2024-03-06T07:22:52.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.209+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77980b4b00 con 0x7f77980cebb0 2024-03-06T07:22:52.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.209+0000 7f77957fa640 1 -- 172.21.15.27:0/2610307319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784035040 con 0x7f77980cebb0 2024-03-06T07:22:52.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.209+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77980c8e50 con 0x7f77980cebb0 2024-03-06T07:22:52.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.210+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7798068700 con 0x7f77980cebb0 2024-03-06T07:22:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.210+0000 7f77957fa640 1 -- 172.21.15.27:0/2610307319 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7784038020 con 0x7f77980cebb0 2024-03-06T07:22:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.211+0000 7f77957fa640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777c06b4d0 0x7f777c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.211+0000 7f77957fa640 1 -- 172.21.15.27:0/2610307319 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77840abaa0 con 0x7f77980cebb0 2024-03-06T07:22:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.211+0000 7f779cf36640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777c06b4d0 0x7f777c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.212+0000 7f779cf36640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777c06b4d0 0x7f777c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f778c000f60 tx=0x7f778c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.215+0000 7f77957fa640 1 -- 172.21.15.27:0/2610307319 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f778407a2d0 con 0x7f77980cebb0 2024-03-06T07:22:52.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.436+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f779806ed60 con 0x7f77980cebb0 2024-03-06T07:22:52.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.437+0000 7f77957fa640 1 -- 172.21.15.27:0/2610307319 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f778407e370 con 0x7f77980cebb0 2024-03-06T07:22:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:52.440 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1135544337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1135544337},{"type":"v1","addr":"172.21.15.27:6833","nonce":1135544337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1920994895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1920994895},{"type":"v1","addr":"172.21.15.69:6837","nonce":1920994895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:48.433333+0000","modified":"2024-03-06T07:22:51.436125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/3933261233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3933261233},{"type":"v1","addr":"172.21.15.27:6835","nonce":3933261233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-06T07:22:52.440 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-03-06T07:22:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777c06b4d0 msgr2=0x7f777c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777c06b4d0 0x7f777c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f778c000f60 tx=0x7f778c007d50 comp rx=0 tx=0).stop 2024-03-06T07:22:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cebb0 msgr2=0x7f77980b6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cebb0 0x7f77980b6f10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f77840096f0 tx=0x7f77840046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 shutdown_connections 2024-03-06T07:22:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777c06b4d0 0x7f777c06d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77980d0940 0x7f77980afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cebb0 0x7f77980b6f10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 --2- 172.21.15.27:0/2610307319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f779806ef80 0x7f77980c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 >> 172.21.15.27:0/2610307319 conn(0x7f77980b35d0 msgr2=0x7f77980b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 shutdown_connections 2024-03-06T07:22:52.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.439+0000 7f779f19a640 1 -- 172.21.15.27:0/2610307319 wait complete. 2024-03-06T07:22:52.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.556+0000 7f14530d4640 1 Processor -- start 2024-03-06T07:22:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.556+0000 7f14530d4640 1 -- start start 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f14530d4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f144c0d18c0 0x7f144c071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f14530d4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f144c071a80 0x7f144c0b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f14530d4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144c0b6f20 con 0x7f144c15df70 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f14530d4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144c071eb0 con 0x7f144c0d18c0 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f14530d4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144c0d3030 con 0x7f144c071a80 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f1450e70640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f144c15df70 0x7f144c0d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56576/0 (socket says 172.21.15.27:56576) 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f1450e70640 1 -- 172.21.15.27:0/2850862104 learned_addr learned my addr 172.21.15.27:0/2850862104 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f1443fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f144c0d18c0 0x7f144c071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f1451671640 1 --2- 172.21.15.27:0/2850862104 >> v2:172.21.15.69:3300/0 conn(0x7f144c071a80 0x7f144c0b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f1443fff640 1 -- 172.21.15.27:0/2850862104 >> v2:172.21.15.69:3300/0 conn(0x7f144c071a80 msgr2=0x7f144c0b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f1443fff640 1 --2- 172.21.15.27:0/2850862104 >> v2:172.21.15.69:3300/0 conn(0x7f144c071a80 0x7f144c0b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f1443fff640 1 -- 172.21.15.27:0/2850862104 >> v1:172.21.15.27:6789/0 conn(0x7f144c15df70 legacy=0x7f144c0d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f1443fff640 1 -- 172.21.15.27:0/2850862104 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c0d3310 con 0x7f144c0d18c0 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f14437fe640 1 -- 172.21.15.27:0/2850862104 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977944081 0 0) 0x7f144c0b6f20 con 0x7f144c15df70 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.557+0000 7f1451671640 1 --2- 172.21.15.27:0/2850862104 >> v2:172.21.15.69:3300/0 conn(0x7f144c071a80 0x7f144c0b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.558+0000 7f1443fff640 1 --2- 172.21.15.27:0/2850862104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0d18c0 0x7f144c071540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f144c08d130 tx=0x7f143c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bd07290d50d57c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.558+0000 7f14437fe640 1 -- 172.21.15.27:0/2850862104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c034070 con 0x7f144c0d18c0 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.558+0000 7f14437fe640 1 -- 172.21.15.27:0/2850862104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143c030610 con 0x7f144c0d18c0 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.558+0000 7f14437fe640 1 -- 172.21.15.27:0/2850862104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c0309e0 con 0x7f144c0d18c0 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.559+0000 7f14530d4640 1 -- 172.21.15.27:0/2850862104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0d18c0 msgr2=0x7f144c071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.559+0000 7f14530d4640 1 --2- 172.21.15.27:0/2850862104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0d18c0 0x7f144c071540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f144c08d130 tx=0x7f143c02b960 comp rx=0 tx=0).stop 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.559+0000 7f14530d4640 1 -- 172.21.15.27:0/2850862104 shutdown_connections 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.559+0000 7f14530d4640 1 --2- 172.21.15.27:0/2850862104 >> v2:172.21.15.69:3300/0 conn(0x7f144c071a80 0x7f144c0b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.559+0000 7f14530d4640 1 --2- 172.21.15.27:0/2850862104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c0d18c0 0x7f144c071540 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.559+0000 7f14530d4640 1 -- 172.21.15.27:0/2850862104 >> 172.21.15.27:0/2850862104 conn(0x7f144c0d3cf0 msgr2=0x7f144c0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.559+0000 7f14530d4640 1 -- 172.21.15.27:0/2850862104 shutdown_connections 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.559+0000 7f14530d4640 1 -- 172.21.15.27:0/2850862104 wait complete. 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.560+0000 7f14530d4640 1 Processor -- start 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.560+0000 7f14530d4640 1 -- start start 2024-03-06T07:22:52.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f14530d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f144c071a80 0x7f144c15be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f14530d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f144c0d18c0 0x7f144c0cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f14530d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c15df70 0x7f144c0cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f14530d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c15e350 con 0x7f144c15df70 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f14530d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c0d38d0 con 0x7f144c0d18c0 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f14530d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c15e4d0 con 0x7f144c071a80 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1451671640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c15df70 0x7f144c0cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1451671640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c15df70 0x7f144c0cd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58478/0 (socket says 172.21.15.27:58478) 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1451671640 1 -- 172.21.15.27:0/2823665656 learned_addr learned my addr 172.21.15.27:0/2823665656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1450e70640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f144c071a80 0x7f144c15be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1443fff640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f144c0d18c0 0x7f144c0cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1451671640 1 -- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f144c071a80 msgr2=0x7f144c15be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1451671640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f144c071a80 0x7f144c15be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1451671640 1 -- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f144c0d18c0 msgr2=0x7f144c0cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1451671640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f144c0d18c0 0x7f144c0cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1451671640 1 -- 172.21.15.27:0/2823665656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c15c340 con 0x7f144c15df70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.561+0000 7f1443fff640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f144c0d18c0 0x7f144c0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.562+0000 7f1451671640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c15df70 0x7f144c0cd5f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1444007f20 tx=0x7f144400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.562+0000 7f14417fa640 1 -- 172.21.15.27:0/2823665656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144401a070 con 0x7f144c15df70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.562+0000 7f14417fa640 1 -- 172.21.15.27:0/2823665656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1444003b90 con 0x7f144c15df70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.562+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144c0cc210 con 0x7f144c15df70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.562+0000 7f14417fa640 1 -- 172.21.15.27:0/2823665656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1444004030 con 0x7f144c15df70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.562+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c0cc7d0 con 0x7f144c15df70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.563+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144c067ff0 con 0x7f144c15df70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.564+0000 7f14417fa640 1 -- 172.21.15.27:0/2823665656 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f144401d070 con 0x7f144c15df70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.564+0000 7f14417fa640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f143806b4d0 0x7f143806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.564+0000 7f14417fa640 1 -- 172.21.15.27:0/2823665656 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f144408a480 con 0x7f144c15df70 2024-03-06T07:22:52.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.564+0000 7f1450e70640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f143806b4d0 0x7f143806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.565+0000 7f1450e70640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f143806b4d0 0x7f143806d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1448000f60 tx=0x7f1448009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.568+0000 7f14417fa640 1 -- 172.21.15.27:0/2823665656 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1444010070 con 0x7f144c15df70 2024-03-06T07:22:52.603 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-06T07:22:52.601+0000 7f7e36695640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:52.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.751+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f144c06b870 con 0x7f144c15df70 2024-03-06T07:22:52.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.752+0000 7f14417fa640 1 -- 172.21.15.27:0/2823665656 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f1444059390 con 0x7f144c15df70 2024-03-06T07:22:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:52.755 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:52.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.754+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f143806b4d0 msgr2=0x7f143806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.754+0000 7f14530d4640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f143806b4d0 0x7f143806d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1448000f60 tx=0x7f1448009040 comp rx=0 tx=0).stop 2024-03-06T07:22:52.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.754+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c15df70 msgr2=0x7f144c0cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.754+0000 7f14530d4640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c15df70 0x7f144c0cd5f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1444007f20 tx=0x7f144400d560 comp rx=0 tx=0).stop 2024-03-06T07:22:52.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.755+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 shutdown_connections 2024-03-06T07:22:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.755+0000 7f14530d4640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f143806b4d0 0x7f143806d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.755+0000 7f14530d4640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144c15df70 0x7f144c0cd5f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.755+0000 7f14530d4640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f144c0d18c0 0x7f144c0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.755+0000 7f14530d4640 1 --2- 172.21.15.27:0/2823665656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f144c071a80 0x7f144c15be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.755+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 >> 172.21.15.27:0/2823665656 conn(0x7f144c0d3cf0 msgr2=0x7f144c160c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.755+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 shutdown_connections 2024-03-06T07:22:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.755+0000 7f14530d4640 1 -- 172.21.15.27:0/2823665656 wait complete. 2024-03-06T07:22:52.770 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:52.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.873+0000 7fb2e235e640 1 Processor -- start 2024-03-06T07:22:52.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.873+0000 7fb2e235e640 1 -- start start 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e235e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb2dc0b0e90 0x7fb2dc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e235e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb2dc0ae9b0 0x7fb2dc0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e235e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0cee90 con 0x7fb2dc06b5c0 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e235e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc0cfa00 con 0x7fb2dc0ae9b0 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e235e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc0cfc50 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e08fb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb2dc0ae9b0 0x7fb2dc0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2dbfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb2dc06b5c0 0x7fb2dc0b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56590/0 (socket says 172.21.15.27:56590) 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2dbfff640 1 -- 172.21.15.27:0/1622277040 learned_addr learned my addr 172.21.15.27:0/1622277040 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2db7fe640 1 --2- 172.21.15.27:0/1622277040 >> v2:172.21.15.69:3300/0 conn(0x7fb2dc0b0e90 0x7fb2dc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e08fb640 1 -- 172.21.15.27:0/1622277040 >> v2:172.21.15.69:3300/0 conn(0x7fb2dc0b0e90 msgr2=0x7fb2dc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e08fb640 1 --2- 172.21.15.27:0/1622277040 >> v2:172.21.15.69:3300/0 conn(0x7fb2dc0b0e90 0x7fb2dc06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e08fb640 1 -- 172.21.15.27:0/1622277040 >> v1:172.21.15.27:6789/0 conn(0x7fb2dc06b5c0 legacy=0x7fb2dc0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2e08fb640 1 -- 172.21.15.27:0/1622277040 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0cff30 con 0x7fb2dc0ae9b0 2024-03-06T07:22:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.874+0000 7fb2daffd640 1 -- 172.21.15.27:0/1622277040 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384171551 0 0) 0x7fb2dc0cee90 con 0x7fb2dc06b5c0 2024-03-06T07:22:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.875+0000 7fb2e08fb640 1 --2- 172.21.15.27:0/1622277040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0ae9b0 0x7fb2dc0d03a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0b0e40 tx=0x7fb2d402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577ea4a018dad698 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.875+0000 7fb2daffd640 1 -- 172.21.15.27:0/1622277040 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4039070 con 0x7fb2dc0ae9b0 2024-03-06T07:22:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.875+0000 7fb2daffd640 1 -- 172.21.15.27:0/1622277040 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d402cbf0 con 0x7fb2dc0ae9b0 2024-03-06T07:22:52.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.875+0000 7fb2daffd640 1 -- 172.21.15.27:0/1622277040 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4034440 con 0x7fb2dc0ae9b0 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.876+0000 7fb2e235e640 1 -- 172.21.15.27:0/1622277040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0ae9b0 msgr2=0x7fb2dc0d03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.876+0000 7fb2e235e640 1 --2- 172.21.15.27:0/1622277040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0ae9b0 0x7fb2dc0d03a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0b0e40 tx=0x7fb2d402be40 comp rx=0 tx=0).stop 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.876+0000 7fb2e235e640 1 -- 172.21.15.27:0/1622277040 shutdown_connections 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.876+0000 7fb2e235e640 1 --2- 172.21.15.27:0/1622277040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0ae9b0 0x7fb2dc0d03a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.876+0000 7fb2e235e640 1 --2- 172.21.15.27:0/1622277040 >> v2:172.21.15.69:3300/0 conn(0x7fb2dc0b0e90 0x7fb2dc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.876+0000 7fb2e235e640 1 -- 172.21.15.27:0/1622277040 >> 172.21.15.27:0/1622277040 conn(0x7fb2dc0b36d0 msgr2=0x7fb2dc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.876+0000 7fb2e235e640 1 -- 172.21.15.27:0/1622277040 shutdown_connections 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.876+0000 7fb2e235e640 1 -- 172.21.15.27:0/1622277040 wait complete. 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.877+0000 7fb2e235e640 1 Processor -- start 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.877+0000 7fb2e235e640 1 -- start start 2024-03-06T07:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e235e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2dc06b5c0 0x7fb2dc0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e235e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2dc0ae9b0 0x7fb2dc1dbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e235e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0b0e90 0x7fb2dc0c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e235e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc06c910 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e235e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc06c790 con 0x7fb2dc0ae9b0 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e235e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0aedc0 con 0x7fb2dc06b5c0 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e08fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0b0e90 0x7fb2dc0c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e08fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0b0e90 0x7fb2dc0c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58506/0 (socket says 172.21.15.27:58506) 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e08fb640 1 -- 172.21.15.27:0/498192249 learned_addr learned my addr 172.21.15.27:0/498192249 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2db7fe640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2dc0ae9b0 0x7fb2dc1dbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2dbfff640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2dc06b5c0 0x7fb2dc0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e08fb640 1 -- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2dc06b5c0 msgr2=0x7fb2dc0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e08fb640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2dc06b5c0 0x7fb2dc0c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e08fb640 1 -- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2dc0ae9b0 msgr2=0x7fb2dc1dbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e08fb640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2dc0ae9b0 0x7fb2dc1dbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2e08fb640 1 -- 172.21.15.27:0/498192249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0c9110 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.878+0000 7fb2db7fe640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2dc0ae9b0 0x7fb2dc1dbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.879+0000 7fb2e08fb640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0b0e90 0x7fb2dc0c5d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e80 tx=0x7fb2d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.879+0000 7fb2d8ff9640 1 -- 172.21.15.27:0/498192249 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4044020 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.879+0000 7fb2d8ff9640 1 -- 172.21.15.27:0/498192249 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d403a040 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.879+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc1dc340 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.879+0000 7fb2d8ff9640 1 -- 172.21.15.27:0/498192249 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4039050 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.879+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc0cc770 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.880+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc068700 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.881+0000 7fb2d8ff9640 1 -- 172.21.15.27:0/498192249 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2d4003b90 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.881+0000 7fb2d8ff9640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb2c806b4d0 0x7fb2c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.881+0000 7fb2d8ff9640 1 -- 172.21.15.27:0/498192249 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2d40aca00 con 0x7fb2dc0b0e90 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.881+0000 7fb2dbfff640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb2c806b4d0 0x7fb2c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.881+0000 7fb2dbfff640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb2c806b4d0 0x7fb2c806d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0007fd0 tx=0x7fb2d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:52.890+0000 7fb2d8ff9640 1 -- 172.21.15.27:0/498192249 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2d404e020 con 0x7fb2dc0b0e90 2024-03-06T07:22:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.066+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2dc1e5d60 con 0x7fb2c806b4d0 2024-03-06T07:22:53.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.073+0000 7fb2d8ff9640 1 -- 172.21.15.27:0/498192249 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251876 (secure 0 0 0) 0x7fb2dc1e5d60 con 0x7fb2c806b4d0 2024-03-06T07:22:53.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:53.081 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:22:53.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb2c806b4d0 msgr2=0x7fb2c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb2c806b4d0 0x7fb2c806d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0007fd0 tx=0x7fb2d0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0b0e90 msgr2=0x7fb2dc0c5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0b0e90 0x7fb2dc0c5d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e80 tx=0x7fb2d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 shutdown_connections 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb2c806b4d0 0x7fb2c806d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc0b0e90 0x7fb2dc0c5d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2dc0ae9b0 0x7fb2dc1dbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 --2- 172.21.15.27:0/498192249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2dc06b5c0 0x7fb2dc0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 >> 172.21.15.27:0/498192249 conn(0x7fb2dc0b36d0 msgr2=0x7fb2dc0b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 shutdown_connections 2024-03-06T07:22:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.076+0000 7fb2e235e640 1 -- 172.21.15.27:0/498192249 wait complete. 2024-03-06T07:22:53.094 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:22:51.987161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218988,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755732,"statfs":{"total":773094113280,"available":772869869568,"internally_reserved":0,"allocated":2887680,"data_stored":1739962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.939392"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082744+0000","last_change":"2024-03-06T07:22:49.080233+0000","last_active":"2024-03-06T07:22:49.082744+0000","last_peered":"2024-03-06T07:22:49.082744+0000","last_clean":"2024-03-06T07:22:49.082744+0000","last_became_active":"2024-03-06T07:22:49.079770+0000","last_became_peered":"2024-03-06T07:22:49.079770+0000","last_unstale":"2024-03-06T07:22:49.082744+0000","last_undegraded":"2024-03-06T07:22:49.082744+0000","last_fullsized":"2024-03-06T07:22:49.082744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.083749+0000","last_change":"2024-03-06T07:22:49.081636+0000","last_active":"2024-03-06T07:22:49.083749+0000","last_peered":"2024-03-06T07:22:49.083749+0000","last_clean":"2024-03-06T07:22:49.083749+0000","last_became_active":"2024-03-06T07:22:49.081356+0000","last_became_peered":"2024-03-06T07:22:49.081356+0000","last_unstale":"2024-03-06T07:22:49.083749+0000","last_undegraded":"2024-03-06T07:22:49.083749+0000","last_fullsized":"2024-03-06T07:22:49.083749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.080060+0000","last_change":"2024-03-06T07:22:49.076314+0000","last_active":"2024-03-06T07:22:49.080060+0000","last_peered":"2024-03-06T07:22:49.080060+0000","last_clean":"2024-03-06T07:22:49.080060+0000","last_became_active":"2024-03-06T07:22:49.075901+0000","last_became_peered":"2024-03-06T07:22:49.075901+0000","last_unstale":"2024-03-06T07:22:49.080060+0000","last_undegraded":"2024-03-06T07:22:49.080060+0000","last_fullsized":"2024-03-06T07:22:49.080060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089026+0000","last_change":"2024-03-06T07:22:49.086730+0000","last_active":"2024-03-06T07:22:49.089026+0000","last_peered":"2024-03-06T07:22:49.089026+0000","last_clean":"2024-03-06T07:22:49.089026+0000","last_became_active":"2024-03-06T07:22:49.086475+0000","last_became_peered":"2024-03-06T07:22:49.086475+0000","last_unstale":"2024-03-06T07:22:49.089026+0000","last_undegraded":"2024-03-06T07:22:49.089026+0000","last_fullsized":"2024-03-06T07:22:49.089026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076094+0000","last_change":"2024-03-06T07:22:49.074936+0000","last_active":"2024-03-06T07:22:49.076094+0000","last_peered":"2024-03-06T07:22:49.076094+0000","last_clean":"2024-03-06T07:22:49.076094+0000","last_became_active":"2024-03-06T07:22:49.074662+0000","last_became_peered":"2024-03-06T07:22:49.074662+0000","last_unstale":"2024-03-06T07:22:49.076094+0000","last_undegraded":"2024-03-06T07:22:49.076094+0000","last_fullsized":"2024-03-06T07:22:49.076094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.031324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.083008+0000","last_change":"2024-03-06T07:22:49.081940+0000","last_active":"2024-03-06T07:22:49.083008+0000","last_peered":"2024-03-06T07:22:49.083008+0000","last_clean":"2024-03-06T07:22:49.083008+0000","last_became_active":"2024-03-06T07:22:49.081668+0000","last_became_peered":"2024-03-06T07:22:49.081668+0000","last_unstale":"2024-03-06T07:22:49.083008+0000","last_undegraded":"2024-03-06T07:22:49.083008+0000","last_fullsized":"2024-03-06T07:22:49.083008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089412+0000","last_change":"2024-03-06T07:22:49.088117+0000","last_active":"2024-03-06T07:22:49.089412+0000","last_peered":"2024-03-06T07:22:49.089412+0000","last_clean":"2024-03-06T07:22:49.089412+0000","last_became_active":"2024-03-06T07:22:49.087856+0000","last_became_peered":"2024-03-06T07:22:49.087856+0000","last_unstale":"2024-03-06T07:22:49.089412+0000","last_undegraded":"2024-03-06T07:22:49.089412+0000","last_fullsized":"2024-03-06T07:22:49.089412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:07.781375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084421+0000","last_change":"2024-03-06T07:22:49.074570+0000","last_active":"2024-03-06T07:22:49.084421+0000","last_peered":"2024-03-06T07:22:49.084421+0000","last_clean":"2024-03-06T07:22:49.084421+0000","last_became_active":"2024-03-06T07:22:49.073115+0000","last_became_peered":"2024-03-06T07:22:49.073115+0000","last_unstale":"2024-03-06T07:22:49.084421+0000","last_undegraded":"2024-03-06T07:22:49.084421+0000","last_fullsized":"2024-03-06T07:22:49.084421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.075698+0000","last_change":"2024-03-06T07:22:49.073716+0000","last_active":"2024-03-06T07:22:49.075698+0000","last_peered":"2024-03-06T07:22:49.075698+0000","last_clean":"2024-03-06T07:22:49.075698+0000","last_became_active":"2024-03-06T07:22:49.073434+0000","last_became_peered":"2024-03-06T07:22:49.073434+0000","last_unstale":"2024-03-06T07:22:49.075698+0000","last_undegraded":"2024-03-06T07:22:49.075698+0000","last_fullsized":"2024-03-06T07:22:49.075698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.073661+0000","last_change":"2024-03-06T07:22:49.072470+0000","last_active":"2024-03-06T07:22:49.073661+0000","last_peered":"2024-03-06T07:22:49.073661+0000","last_clean":"2024-03-06T07:22:49.073661+0000","last_became_active":"2024-03-06T07:22:49.072206+0000","last_became_peered":"2024-03-06T07:22:49.072206+0000","last_unstale":"2024-03-06T07:22:49.073661+0000","last_undegraded":"2024-03-06T07:22:49.073661+0000","last_fullsized":"2024-03-06T07:22:49.073661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082251+0000","last_change":"2024-03-06T07:22:49.076507+0000","last_active":"2024-03-06T07:22:49.082251+0000","last_peered":"2024-03-06T07:22:49.082251+0000","last_clean":"2024-03-06T07:22:49.082251+0000","last_became_active":"2024-03-06T07:22:49.076054+0000","last_became_peered":"2024-03-06T07:22:49.076054+0000","last_unstale":"2024-03-06T07:22:49.082251+0000","last_undegraded":"2024-03-06T07:22:49.082251+0000","last_fullsized":"2024-03-06T07:22:49.082251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079175+0000","last_change":"2024-03-06T07:22:49.077078+0000","last_active":"2024-03-06T07:22:49.079175+0000","last_peered":"2024-03-06T07:22:49.079175+0000","last_clean":"2024-03-06T07:22:49.079175+0000","last_became_active":"2024-03-06T07:22:49.076757+0000","last_became_peered":"2024-03-06T07:22:49.076757+0000","last_unstale":"2024-03-06T07:22:49.079175+0000","last_undegraded":"2024-03-06T07:22:49.079175+0000","last_fullsized":"2024-03-06T07:22:49.079175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:52.449738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084528+0000","last_change":"2024-03-06T07:22:49.073467+0000","last_active":"2024-03-06T07:22:49.084528+0000","last_peered":"2024-03-06T07:22:49.084528+0000","last_clean":"2024-03-06T07:22:49.084528+0000","last_became_active":"2024-03-06T07:22:49.072969+0000","last_became_peered":"2024-03-06T07:22:49.072969+0000","last_unstale":"2024-03-06T07:22:49.084528+0000","last_undegraded":"2024-03-06T07:22:49.084528+0000","last_fullsized":"2024-03-06T07:22:49.084528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:51.096875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078810+0000","last_change":"2024-03-06T07:22:49.076207+0000","last_active":"2024-03-06T07:22:49.078810+0000","last_peered":"2024-03-06T07:22:49.078810+0000","last_clean":"2024-03-06T07:22:49.078810+0000","last_became_active":"2024-03-06T07:22:49.074449+0000","last_became_peered":"2024-03-06T07:22:49.074449+0000","last_unstale":"2024-03-06T07:22:49.078810+0000","last_undegraded":"2024-03-06T07:22:49.078810+0000","last_fullsized":"2024-03-06T07:22:49.078810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:51.096875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.077970+0000","last_change":"2024-03-06T07:22:49.072745+0000","last_active":"2024-03-06T07:22:49.077970+0000","last_peered":"2024-03-06T07:22:49.077970+0000","last_clean":"2024-03-06T07:22:49.077970+0000","last_became_active":"2024-03-06T07:22:49.072443+0000","last_became_peered":"2024-03-06T07:22:49.072443+0000","last_unstale":"2024-03-06T07:22:49.077970+0000","last_undegraded":"2024-03-06T07:22:49.077970+0000","last_fullsized":"2024-03-06T07:22:49.077970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088847+0000","last_change":"2024-03-06T07:22:49.085335+0000","last_active":"2024-03-06T07:22:49.088847+0000","last_peered":"2024-03-06T07:22:49.088847+0000","last_clean":"2024-03-06T07:22:49.088847+0000","last_became_active":"2024-03-06T07:22:49.085023+0000","last_became_peered":"2024-03-06T07:22:49.085023+0000","last_unstale":"2024-03-06T07:22:49.088847+0000","last_undegraded":"2024-03-06T07:22:49.088847+0000","last_fullsized":"2024-03-06T07:22:49.088847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.031324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079704+0000","last_change":"2024-03-06T07:22:49.073395+0000","last_active":"2024-03-06T07:22:49.079704+0000","last_peered":"2024-03-06T07:22:49.079704+0000","last_clean":"2024-03-06T07:22:49.079704+0000","last_became_active":"2024-03-06T07:22:49.073105+0000","last_became_peered":"2024-03-06T07:22:49.073105+0000","last_unstale":"2024-03-06T07:22:49.079704+0000","last_undegraded":"2024-03-06T07:22:49.079704+0000","last_fullsized":"2024-03-06T07:22:49.079704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.080226+0000","last_change":"2024-03-06T07:22:49.074912+0000","last_active":"2024-03-06T07:22:49.080226+0000","last_peered":"2024-03-06T07:22:49.080226+0000","last_clean":"2024-03-06T07:22:49.080226+0000","last_became_active":"2024-03-06T07:22:49.074638+0000","last_became_peered":"2024-03-06T07:22:49.074638+0000","last_unstale":"2024-03-06T07:22:49.080226+0000","last_undegraded":"2024-03-06T07:22:49.080226+0000","last_fullsized":"2024-03-06T07:22:49.080226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076698+0000","last_change":"2024-03-06T07:22:49.075473+0000","last_active":"2024-03-06T07:22:49.076698+0000","last_peered":"2024-03-06T07:22:49.076698+0000","last_clean":"2024-03-06T07:22:49.076698+0000","last_became_active":"2024-03-06T07:22:49.074945+0000","last_became_peered":"2024-03-06T07:22:49.074945+0000","last_unstale":"2024-03-06T07:22:49.076698+0000","last_undegraded":"2024-03-06T07:22:49.076698+0000","last_fullsized":"2024-03-06T07:22:49.076698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087015+0000","last_change":"2024-03-06T07:22:49.085348+0000","last_active":"2024-03-06T07:22:49.087015+0000","last_peered":"2024-03-06T07:22:49.087015+0000","last_clean":"2024-03-06T07:22:49.087015+0000","last_became_active":"2024-03-06T07:22:49.084955+0000","last_became_peered":"2024-03-06T07:22:49.084955+0000","last_unstale":"2024-03-06T07:22:49.087015+0000","last_undegraded":"2024-03-06T07:22:49.087015+0000","last_fullsized":"2024-03-06T07:22:49.087015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.090574+0000","last_change":"2024-03-06T07:22:49.081034+0000","last_active":"2024-03-06T07:22:49.090574+0000","last_peered":"2024-03-06T07:22:49.090574+0000","last_clean":"2024-03-06T07:22:49.090574+0000","last_became_active":"2024-03-06T07:22:49.080639+0000","last_became_peered":"2024-03-06T07:22:49.080639+0000","last_unstale":"2024-03-06T07:22:49.090574+0000","last_undegraded":"2024-03-06T07:22:49.090574+0000","last_fullsized":"2024-03-06T07:22:49.090574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079734+0000","last_change":"2024-03-06T07:22:49.074317+0000","last_active":"2024-03-06T07:22:49.079734+0000","last_peered":"2024-03-06T07:22:49.079734+0000","last_clean":"2024-03-06T07:22:49.079734+0000","last_became_active":"2024-03-06T07:22:49.073041+0000","last_became_peered":"2024-03-06T07:22:49.073041+0000","last_unstale":"2024-03-06T07:22:49.079734+0000","last_undegraded":"2024-03-06T07:22:49.079734+0000","last_fullsized":"2024-03-06T07:22:49.079734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088695+0000","last_change":"2024-03-06T07:22:49.076771+0000","last_active":"2024-03-06T07:22:49.088695+0000","last_peered":"2024-03-06T07:22:49.088695+0000","last_clean":"2024-03-06T07:22:49.088695+0000","last_became_active":"2024-03-06T07:22:49.073031+0000","last_became_peered":"2024-03-06T07:22:49.073031+0000","last_unstale":"2024-03-06T07:22:49.088695+0000","last_undegraded":"2024-03-06T07:22:49.088695+0000","last_fullsized":"2024-03-06T07:22:49.088695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088936+0000","last_change":"2024-03-06T07:22:49.086108+0000","last_active":"2024-03-06T07:22:49.088936+0000","last_peered":"2024-03-06T07:22:49.088936+0000","last_clean":"2024-03-06T07:22:49.088936+0000","last_became_active":"2024-03-06T07:22:49.085825+0000","last_became_peered":"2024-03-06T07:22:49.085825+0000","last_unstale":"2024-03-06T07:22:49.088936+0000","last_undegraded":"2024-03-06T07:22:49.088936+0000","last_fullsized":"2024-03-06T07:22:49.088936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084382+0000","last_change":"2024-03-06T07:22:49.071422+0000","last_active":"2024-03-06T07:22:49.084382+0000","last_peered":"2024-03-06T07:22:49.084382+0000","last_clean":"2024-03-06T07:22:49.084382+0000","last_became_active":"2024-03-06T07:22:49.071117+0000","last_became_peered":"2024-03-06T07:22:49.071117+0000","last_unstale":"2024-03-06T07:22:49.084382+0000","last_undegraded":"2024-03-06T07:22:49.084382+0000","last_fullsized":"2024-03-06T07:22:49.084382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:38.976172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082892+0000","last_change":"2024-03-06T07:22:49.077927+0000","last_active":"2024-03-06T07:22:49.082892+0000","last_peered":"2024-03-06T07:22:49.082892+0000","last_clean":"2024-03-06T07:22:49.082892+0000","last_became_active":"2024-03-06T07:22:49.077647+0000","last_became_peered":"2024-03-06T07:22:49.077647+0000","last_unstale":"2024-03-06T07:22:49.082892+0000","last_undegraded":"2024-03-06T07:22:49.082892+0000","last_fullsized":"2024-03-06T07:22:49.082892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088325+0000","last_change":"2024-03-06T07:22:49.085345+0000","last_active":"2024-03-06T07:22:49.088325+0000","last_peered":"2024-03-06T07:22:49.088325+0000","last_clean":"2024-03-06T07:22:49.088325+0000","last_became_active":"2024-03-06T07:22:49.084942+0000","last_became_peered":"2024-03-06T07:22:49.084942+0000","last_unstale":"2024-03-06T07:22:49.088325+0000","last_undegraded":"2024-03-06T07:22:49.088325+0000","last_fullsized":"2024-03-06T07:22:49.088325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084426+0000","last_change":"2024-03-06T07:22:49.071314+0000","last_active":"2024-03-06T07:22:49.084426+0000","last_peered":"2024-03-06T07:22:49.084426+0000","last_clean":"2024-03-06T07:22:49.084426+0000","last_became_active":"2024-03-06T07:22:49.070986+0000","last_became_peered":"2024-03-06T07:22:49.070986+0000","last_unstale":"2024-03-06T07:22:49.084426+0000","last_undegraded":"2024-03-06T07:22:49.084426+0000","last_fullsized":"2024-03-06T07:22:49.084426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:46.459505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.083562+0000","last_change":"2024-03-06T07:22:49.077320+0000","last_active":"2024-03-06T07:22:49.083562+0000","last_peered":"2024-03-06T07:22:49.083562+0000","last_clean":"2024-03-06T07:22:49.083562+0000","last_became_active":"2024-03-06T07:22:49.077017+0000","last_became_peered":"2024-03-06T07:22:49.077017+0000","last_unstale":"2024-03-06T07:22:49.083562+0000","last_undegraded":"2024-03-06T07:22:49.083562+0000","last_fullsized":"2024-03-06T07:22:49.083562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088945+0000","last_change":"2024-03-06T07:22:49.086200+0000","last_active":"2024-03-06T07:22:49.088945+0000","last_peered":"2024-03-06T07:22:49.088945+0000","last_clean":"2024-03-06T07:22:49.088945+0000","last_became_active":"2024-03-06T07:22:49.085793+0000","last_became_peered":"2024-03-06T07:22:49.085793+0000","last_unstale":"2024-03-06T07:22:49.088945+0000","last_undegraded":"2024-03-06T07:22:49.088945+0000","last_fullsized":"2024-03-06T07:22:49.088945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086818+0000","last_change":"2024-03-06T07:22:49.076891+0000","last_active":"2024-03-06T07:22:49.086818+0000","last_peered":"2024-03-06T07:22:49.086818+0000","last_clean":"2024-03-06T07:22:49.086818+0000","last_became_active":"2024-03-06T07:22:49.076610+0000","last_became_peered":"2024-03-06T07:22:49.076610+0000","last_unstale":"2024-03-06T07:22:49.086818+0000","last_undegraded":"2024-03-06T07:22:49.086818+0000","last_fullsized":"2024-03-06T07:22:49.086818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089380+0000","last_change":"2024-03-06T07:22:49.088635+0000","last_active":"2024-03-06T07:22:49.089380+0000","last_peered":"2024-03-06T07:22:49.089380+0000","last_clean":"2024-03-06T07:22:49.089380+0000","last_became_active":"2024-03-06T07:22:49.088352+0000","last_became_peered":"2024-03-06T07:22:49.088352+0000","last_unstale":"2024-03-06T07:22:49.089380+0000","last_undegraded":"2024-03-06T07:22:49.089380+0000","last_fullsized":"2024-03-06T07:22:49.089380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.072012+0000","last_change":"2024-03-06T07:22:49.071168+0000","last_active":"2024-03-06T07:22:49.072012+0000","last_peered":"2024-03-06T07:22:49.072012+0000","last_clean":"2024-03-06T07:22:49.072012+0000","last_became_active":"2024-03-06T07:22:49.070861+0000","last_became_peered":"2024-03-06T07:22:49.070861+0000","last_unstale":"2024-03-06T07:22:49.072012+0000","last_undegraded":"2024-03-06T07:22:49.072012+0000","last_fullsized":"2024-03-06T07:22:49.072012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:20.360478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086951+0000","last_change":"2024-03-06T07:22:48.076442+0000","last_active":"2024-03-06T07:22:49.086951+0000","last_peered":"2024-03-06T07:22:49.086951+0000","last_clean":"2024-03-06T07:22:49.086951+0000","last_became_active":"2024-03-06T07:22:48.076193+0000","last_became_peered":"2024-03-06T07:22:48.076193+0000","last_unstale":"2024-03-06T07:22:49.086951+0000","last_undegraded":"2024-03-06T07:22:49.086951+0000","last_fullsized":"2024-03-06T07:22:49.086951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:48.118117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.080355+0000","last_change":"2024-03-06T07:22:49.067928+0000","last_active":"2024-03-06T07:22:49.080355+0000","last_peered":"2024-03-06T07:22:49.080355+0000","last_clean":"2024-03-06T07:22:49.080355+0000","last_became_active":"2024-03-06T07:22:49.066102+0000","last_became_peered":"2024-03-06T07:22:49.066102+0000","last_unstale":"2024-03-06T07:22:49.080355+0000","last_undegraded":"2024-03-06T07:22:49.080355+0000","last_fullsized":"2024-03-06T07:22:49.080355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:49.124109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086794+0000","last_change":"2024-03-06T07:22:48.072246+0000","last_active":"2024-03-06T07:22:49.086794+0000","last_peered":"2024-03-06T07:22:49.086794+0000","last_clean":"2024-03-06T07:22:49.086794+0000","last_became_active":"2024-03-06T07:22:48.071947+0000","last_became_peered":"2024-03-06T07:22:48.071947+0000","last_unstale":"2024-03-06T07:22:49.086794+0000","last_undegraded":"2024-03-06T07:22:49.086794+0000","last_fullsized":"2024-03-06T07:22:49.086794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.062281+0000","last_change":"2024-03-06T07:22:48.075039+0000","last_active":"2024-03-06T07:22:49.062281+0000","last_peered":"2024-03-06T07:22:49.062281+0000","last_clean":"2024-03-06T07:22:49.062281+0000","last_became_active":"2024-03-06T07:22:48.074175+0000","last_became_peered":"2024-03-06T07:22:48.074175+0000","last_unstale":"2024-03-06T07:22:49.062281+0000","last_undegraded":"2024-03-06T07:22:49.062281+0000","last_fullsized":"2024-03-06T07:22:49.062281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:51.443747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086954+0000","last_change":"2024-03-06T07:22:49.081619+0000","last_active":"2024-03-06T07:22:49.086954+0000","last_peered":"2024-03-06T07:22:49.086954+0000","last_clean":"2024-03-06T07:22:49.086954+0000","last_became_active":"2024-03-06T07:22:49.081307+0000","last_became_peered":"2024-03-06T07:22:49.081307+0000","last_unstale":"2024-03-06T07:22:49.086954+0000","last_undegraded":"2024-03-06T07:22:49.086954+0000","last_fullsized":"2024-03-06T07:22:49.086954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065442+0000","last_change":"2024-03-06T07:22:48.070889+0000","last_active":"2024-03-06T07:22:49.065442+0000","last_peered":"2024-03-06T07:22:49.065442+0000","last_clean":"2024-03-06T07:22:49.065442+0000","last_became_active":"2024-03-06T07:22:48.070591+0000","last_became_peered":"2024-03-06T07:22:48.070591+0000","last_unstale":"2024-03-06T07:22:49.065442+0000","last_undegraded":"2024-03-06T07:22:49.065442+0000","last_fullsized":"2024-03-06T07:22:49.065442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.072890+0000","last_change":"2024-03-06T07:22:49.071108+0000","last_active":"2024-03-06T07:22:49.072890+0000","last_peered":"2024-03-06T07:22:49.072890+0000","last_clean":"2024-03-06T07:22:49.072890+0000","last_became_active":"2024-03-06T07:22:49.070830+0000","last_became_peered":"2024-03-06T07:22:49.070830+0000","last_unstale":"2024-03-06T07:22:49.072890+0000","last_undegraded":"2024-03-06T07:22:49.072890+0000","last_fullsized":"2024-03-06T07:22:49.072890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.064187+0000","last_change":"2024-03-06T07:22:48.073692+0000","last_active":"2024-03-06T07:22:49.064187+0000","last_peered":"2024-03-06T07:22:49.064187+0000","last_clean":"2024-03-06T07:22:49.064187+0000","last_became_active":"2024-03-06T07:22:48.073366+0000","last_became_peered":"2024-03-06T07:22:48.073366+0000","last_unstale":"2024-03-06T07:22:49.064187+0000","last_undegraded":"2024-03-06T07:22:49.064187+0000","last_fullsized":"2024-03-06T07:22:49.064187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.081385+0000","last_change":"2024-03-06T07:22:49.075374+0000","last_active":"2024-03-06T07:22:49.081385+0000","last_peered":"2024-03-06T07:22:49.081385+0000","last_clean":"2024-03-06T07:22:49.081385+0000","last_became_active":"2024-03-06T07:22:49.073115+0000","last_became_peered":"2024-03-06T07:22:49.073115+0000","last_unstale":"2024-03-06T07:22:49.081385+0000","last_undegraded":"2024-03-06T07:22:49.081385+0000","last_fullsized":"2024-03-06T07:22:49.081385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.066295+0000","last_change":"2024-03-06T07:22:48.074598+0000","last_active":"2024-03-06T07:22:49.066295+0000","last_peered":"2024-03-06T07:22:49.066295+0000","last_clean":"2024-03-06T07:22:49.066295+0000","last_became_active":"2024-03-06T07:22:48.074291+0000","last_became_peered":"2024-03-06T07:22:48.074291+0000","last_unstale":"2024-03-06T07:22:49.066295+0000","last_undegraded":"2024-03-06T07:22:49.066295+0000","last_fullsized":"2024-03-06T07:22:49.066295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078759+0000","last_change":"2024-03-06T07:22:49.075698+0000","last_active":"2024-03-06T07:22:49.078759+0000","last_peered":"2024-03-06T07:22:49.078759+0000","last_clean":"2024-03-06T07:22:49.078759+0000","last_became_active":"2024-03-06T07:22:49.075395+0000","last_became_peered":"2024-03-06T07:22:49.075395+0000","last_unstale":"2024-03-06T07:22:49.078759+0000","last_undegraded":"2024-03-06T07:22:49.078759+0000","last_fullsized":"2024-03-06T07:22:49.078759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:38.976172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089292+0000","last_change":"2024-03-06T07:22:48.071600+0000","last_active":"2024-03-06T07:22:49.089292+0000","last_peered":"2024-03-06T07:22:49.089292+0000","last_clean":"2024-03-06T07:22:49.089292+0000","last_became_active":"2024-03-06T07:22:48.071194+0000","last_became_peered":"2024-03-06T07:22:48.071194+0000","last_unstale":"2024-03-06T07:22:49.089292+0000","last_undegraded":"2024-03-06T07:22:49.089292+0000","last_fullsized":"2024-03-06T07:22:49.089292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079453+0000","last_change":"2024-03-06T07:22:49.078266+0000","last_active":"2024-03-06T07:22:49.079453+0000","last_peered":"2024-03-06T07:22:49.079453+0000","last_clean":"2024-03-06T07:22:49.079453+0000","last_became_active":"2024-03-06T07:22:49.077993+0000","last_became_peered":"2024-03-06T07:22:49.077993+0000","last_unstale":"2024-03-06T07:22:49.079453+0000","last_undegraded":"2024-03-06T07:22:49.079453+0000","last_fullsized":"2024-03-06T07:22:49.079453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:53.011841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.095110+0000","last_change":"2024-03-06T07:22:48.074028+0000","last_active":"2024-03-06T07:22:49.095110+0000","last_peered":"2024-03-06T07:22:49.095110+0000","last_clean":"2024-03-06T07:22:49.095110+0000","last_became_active":"2024-03-06T07:22:48.073737+0000","last_became_peered":"2024-03-06T07:22:48.073737+0000","last_unstale":"2024-03-06T07:22:49.095110+0000","last_undegraded":"2024-03-06T07:22:49.095110+0000","last_fullsized":"2024-03-06T07:22:49.095110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076124+0000","last_change":"2024-03-06T07:22:49.075328+0000","last_active":"2024-03-06T07:22:49.076124+0000","last_peered":"2024-03-06T07:22:49.076124+0000","last_clean":"2024-03-06T07:22:49.076124+0000","last_became_active":"2024-03-06T07:22:49.074938+0000","last_became_peered":"2024-03-06T07:22:49.074938+0000","last_unstale":"2024-03-06T07:22:49.076124+0000","last_undegraded":"2024-03-06T07:22:49.076124+0000","last_fullsized":"2024-03-06T07:22:49.076124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.090737+0000","last_change":"2024-03-06T07:22:48.062093+0000","last_active":"2024-03-06T07:22:49.090737+0000","last_peered":"2024-03-06T07:22:49.090737+0000","last_clean":"2024-03-06T07:22:49.090737+0000","last_became_active":"2024-03-06T07:22:48.061816+0000","last_became_peered":"2024-03-06T07:22:48.061816+0000","last_unstale":"2024-03-06T07:22:49.090737+0000","last_undegraded":"2024-03-06T07:22:49.090737+0000","last_fullsized":"2024-03-06T07:22:49.090737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.354486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087317+0000","last_change":"2024-03-06T07:22:49.081035+0000","last_active":"2024-03-06T07:22:49.087317+0000","last_peered":"2024-03-06T07:22:49.087317+0000","last_clean":"2024-03-06T07:22:49.087317+0000","last_became_active":"2024-03-06T07:22:49.080738+0000","last_became_peered":"2024-03-06T07:22:49.080738+0000","last_unstale":"2024-03-06T07:22:49.087317+0000","last_undegraded":"2024-03-06T07:22:49.087317+0000","last_fullsized":"2024-03-06T07:22:49.087317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.067613+0000","last_change":"2024-03-06T07:22:48.076377+0000","last_active":"2024-03-06T07:22:49.067613+0000","last_peered":"2024-03-06T07:22:49.067613+0000","last_clean":"2024-03-06T07:22:49.067613+0000","last_became_active":"2024-03-06T07:22:48.076062+0000","last_became_peered":"2024-03-06T07:22:48.076062+0000","last_unstale":"2024-03-06T07:22:49.067613+0000","last_undegraded":"2024-03-06T07:22:49.067613+0000","last_fullsized":"2024-03-06T07:22:49.067613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.083781+0000","last_change":"2024-03-06T07:22:49.081681+0000","last_active":"2024-03-06T07:22:49.083781+0000","last_peered":"2024-03-06T07:22:49.083781+0000","last_clean":"2024-03-06T07:22:49.083781+0000","last_became_active":"2024-03-06T07:22:49.081400+0000","last_became_peered":"2024-03-06T07:22:49.081400+0000","last_unstale":"2024-03-06T07:22:49.083781+0000","last_undegraded":"2024-03-06T07:22:49.083781+0000","last_fullsized":"2024-03-06T07:22:49.083781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:58.929359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.066207+0000","last_change":"2024-03-06T07:22:48.072725+0000","last_active":"2024-03-06T07:22:49.066207+0000","last_peered":"2024-03-06T07:22:49.066207+0000","last_clean":"2024-03-06T07:22:49.066207+0000","last_became_active":"2024-03-06T07:22:48.071793+0000","last_became_peered":"2024-03-06T07:22:48.071793+0000","last_unstale":"2024-03-06T07:22:49.066207+0000","last_undegraded":"2024-03-06T07:22:49.066207+0000","last_fullsized":"2024-03-06T07:22:49.066207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089620+0000","last_change":"2024-03-06T07:22:49.088880+0000","last_active":"2024-03-06T07:22:49.089620+0000","last_peered":"2024-03-06T07:22:49.089620+0000","last_clean":"2024-03-06T07:22:49.089620+0000","last_became_active":"2024-03-06T07:22:49.088565+0000","last_became_peered":"2024-03-06T07:22:49.088565+0000","last_unstale":"2024-03-06T07:22:49.089620+0000","last_undegraded":"2024-03-06T07:22:49.089620+0000","last_fullsized":"2024-03-06T07:22:49.089620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086942+0000","last_change":"2024-03-06T07:22:48.067604+0000","last_active":"2024-03-06T07:22:49.086942+0000","last_peered":"2024-03-06T07:22:49.086942+0000","last_clean":"2024-03-06T07:22:49.086942+0000","last_became_active":"2024-03-06T07:22:48.067273+0000","last_became_peered":"2024-03-06T07:22:48.067273+0000","last_unstale":"2024-03-06T07:22:49.086942+0000","last_undegraded":"2024-03-06T07:22:49.086942+0000","last_fullsized":"2024-03-06T07:22:49.086942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082705+0000","last_change":"2024-03-06T07:22:49.080955+0000","last_active":"2024-03-06T07:22:49.082705+0000","last_peered":"2024-03-06T07:22:49.082705+0000","last_clean":"2024-03-06T07:22:49.082705+0000","last_became_active":"2024-03-06T07:22:49.080699+0000","last_became_peered":"2024-03-06T07:22:49.080699+0000","last_unstale":"2024-03-06T07:22:49.082705+0000","last_undegraded":"2024-03-06T07:22:49.082705+0000","last_fullsized":"2024-03-06T07:22:49.082705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084318+0000","last_change":"2024-03-06T07:22:48.071049+0000","last_active":"2024-03-06T07:22:49.084318+0000","last_peered":"2024-03-06T07:22:49.084318+0000","last_clean":"2024-03-06T07:22:49.084318+0000","last_became_active":"2024-03-06T07:22:48.070751+0000","last_became_peered":"2024-03-06T07:22:48.070751+0000","last_unstale":"2024-03-06T07:22:49.084318+0000","last_undegraded":"2024-03-06T07:22:49.084318+0000","last_fullsized":"2024-03-06T07:22:49.084318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:35.856218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.081936+0000","last_change":"2024-03-06T07:22:49.077479+0000","last_active":"2024-03-06T07:22:49.081936+0000","last_peered":"2024-03-06T07:22:49.081936+0000","last_clean":"2024-03-06T07:22:49.081936+0000","last_became_active":"2024-03-06T07:22:49.077188+0000","last_became_peered":"2024-03-06T07:22:49.077188+0000","last_unstale":"2024-03-06T07:22:49.081936+0000","last_undegraded":"2024-03-06T07:22:49.081936+0000","last_fullsized":"2024-03-06T07:22:49.081936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088852+0000","last_change":"2024-03-06T07:22:48.074777+0000","last_active":"2024-03-06T07:22:49.088852+0000","last_peered":"2024-03-06T07:22:49.088852+0000","last_clean":"2024-03-06T07:22:49.088852+0000","last_became_active":"2024-03-06T07:22:48.074441+0000","last_became_peered":"2024-03-06T07:22:48.074441+0000","last_unstale":"2024-03-06T07:22:49.088852+0000","last_undegraded":"2024-03-06T07:22:49.088852+0000","last_fullsized":"2024-03-06T07:22:49.088852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082719+0000","last_change":"2024-03-06T07:22:49.080000+0000","last_active":"2024-03-06T07:22:49.082719+0000","last_peered":"2024-03-06T07:22:49.082719+0000","last_clean":"2024-03-06T07:22:49.082719+0000","last_became_active":"2024-03-06T07:22:49.079471+0000","last_became_peered":"2024-03-06T07:22:49.079471+0000","last_unstale":"2024-03-06T07:22:49.082719+0000","last_undegraded":"2024-03-06T07:22:49.082719+0000","last_fullsized":"2024-03-06T07:22:49.082719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.085863+0000","last_change":"2024-03-06T07:22:48.067652+0000","last_active":"2024-03-06T07:22:49.085863+0000","last_peered":"2024-03-06T07:22:49.085863+0000","last_clean":"2024-03-06T07:22:49.085863+0000","last_became_active":"2024-03-06T07:22:48.067390+0000","last_became_peered":"2024-03-06T07:22:48.067390+0000","last_unstale":"2024-03-06T07:22:49.085863+0000","last_undegraded":"2024-03-06T07:22:49.085863+0000","last_fullsized":"2024-03-06T07:22:49.085863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.085967+0000","last_change":"2024-03-06T07:22:49.084081+0000","last_active":"2024-03-06T07:22:49.085967+0000","last_peered":"2024-03-06T07:22:49.085967+0000","last_clean":"2024-03-06T07:22:49.085967+0000","last_became_active":"2024-03-06T07:22:49.083758+0000","last_became_peered":"2024-03-06T07:22:49.083758+0000","last_unstale":"2024-03-06T07:22:49.085967+0000","last_undegraded":"2024-03-06T07:22:49.085967+0000","last_fullsized":"2024-03-06T07:22:49.085967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.067389+0000","last_change":"2024-03-06T07:22:48.072197+0000","last_active":"2024-03-06T07:22:49.067389+0000","last_peered":"2024-03-06T07:22:49.067389+0000","last_clean":"2024-03-06T07:22:49.067389+0000","last_became_active":"2024-03-06T07:22:48.071904+0000","last_became_peered":"2024-03-06T07:22:48.071904+0000","last_unstale":"2024-03-06T07:22:49.067389+0000","last_undegraded":"2024-03-06T07:22:49.067389+0000","last_fullsized":"2024-03-06T07:22:49.067389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076103+0000","last_change":"2024-03-06T07:22:49.074585+0000","last_active":"2024-03-06T07:22:49.076103+0000","last_peered":"2024-03-06T07:22:49.076103+0000","last_clean":"2024-03-06T07:22:49.076103+0000","last_became_active":"2024-03-06T07:22:49.074307+0000","last_became_peered":"2024-03-06T07:22:49.074307+0000","last_unstale":"2024-03-06T07:22:49.076103+0000","last_undegraded":"2024-03-06T07:22:49.076103+0000","last_fullsized":"2024-03-06T07:22:49.076103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065334+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:22:49.065334+0000","last_peered":"2024-03-06T07:22:49.065334+0000","last_clean":"2024-03-06T07:22:49.065334+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:22:49.065334+0000","last_undegraded":"2024-03-06T07:22:49.065334+0000","last_fullsized":"2024-03-06T07:22:49.065334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.066085+0000","last_change":"2024-03-06T07:22:48.073977+0000","last_active":"2024-03-06T07:22:49.066085+0000","last_peered":"2024-03-06T07:22:49.066085+0000","last_clean":"2024-03-06T07:22:49.066085+0000","last_became_active":"2024-03-06T07:22:48.073656+0000","last_became_peered":"2024-03-06T07:22:48.073656+0000","last_unstale":"2024-03-06T07:22:49.066085+0000","last_undegraded":"2024-03-06T07:22:49.066085+0000","last_fullsized":"2024-03-06T07:22:49.066085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084355+0000","last_change":"2024-03-06T07:22:49.074128+0000","last_active":"2024-03-06T07:22:49.084355+0000","last_peered":"2024-03-06T07:22:49.084355+0000","last_clean":"2024-03-06T07:22:49.084355+0000","last_became_active":"2024-03-06T07:22:49.073858+0000","last_became_peered":"2024-03-06T07:22:49.073858+0000","last_unstale":"2024-03-06T07:22:49.084355+0000","last_undegraded":"2024-03-06T07:22:49.084355+0000","last_fullsized":"2024-03-06T07:22:49.084355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065286+0000","last_change":"2024-03-06T07:22:48.069166+0000","last_active":"2024-03-06T07:22:49.065286+0000","last_peered":"2024-03-06T07:22:49.065286+0000","last_clean":"2024-03-06T07:22:49.065286+0000","last_became_active":"2024-03-06T07:22:48.068441+0000","last_became_peered":"2024-03-06T07:22:48.068441+0000","last_unstale":"2024-03-06T07:22:49.065286+0000","last_undegraded":"2024-03-06T07:22:49.065286+0000","last_fullsized":"2024-03-06T07:22:49.065286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:45.086584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078528+0000","last_change":"2024-03-06T07:22:49.075228+0000","last_active":"2024-03-06T07:22:49.078528+0000","last_peered":"2024-03-06T07:22:49.078528+0000","last_clean":"2024-03-06T07:22:49.078528+0000","last_became_active":"2024-03-06T07:22:49.074906+0000","last_became_peered":"2024-03-06T07:22:49.074906+0000","last_unstale":"2024-03-06T07:22:49.078528+0000","last_undegraded":"2024-03-06T07:22:49.078528+0000","last_fullsized":"2024-03-06T07:22:49.078528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:53.011841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.069758+0000","last_change":"2024-03-06T07:22:48.074088+0000","last_active":"2024-03-06T07:22:49.069758+0000","last_peered":"2024-03-06T07:22:49.069758+0000","last_clean":"2024-03-06T07:22:49.069758+0000","last_became_active":"2024-03-06T07:22:48.073794+0000","last_became_peered":"2024-03-06T07:22:48.073794+0000","last_unstale":"2024-03-06T07:22:49.069758+0000","last_undegraded":"2024-03-06T07:22:49.069758+0000","last_fullsized":"2024-03-06T07:22:49.069758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.073127+0000","last_change":"2024-03-06T07:22:49.071324+0000","last_active":"2024-03-06T07:22:49.073127+0000","last_peered":"2024-03-06T07:22:49.073127+0000","last_clean":"2024-03-06T07:22:49.073127+0000","last_became_active":"2024-03-06T07:22:49.071043+0000","last_became_peered":"2024-03-06T07:22:49.071043+0000","last_unstale":"2024-03-06T07:22:49.073127+0000","last_undegraded":"2024-03-06T07:22:49.073127+0000","last_fullsized":"2024-03-06T07:22:49.073127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.068197+0000","last_change":"2024-03-06T07:22:48.071307+0000","last_active":"2024-03-06T07:22:49.068197+0000","last_peered":"2024-03-06T07:22:49.068197+0000","last_clean":"2024-03-06T07:22:49.068197+0000","last_became_active":"2024-03-06T07:22:48.070797+0000","last_became_peered":"2024-03-06T07:22:48.070797+0000","last_unstale":"2024-03-06T07:22:49.068197+0000","last_undegraded":"2024-03-06T07:22:49.068197+0000","last_fullsized":"2024-03-06T07:22:49.068197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079479+0000","last_change":"2024-03-06T07:22:49.077714+0000","last_active":"2024-03-06T07:22:49.079479+0000","last_peered":"2024-03-06T07:22:49.079479+0000","last_clean":"2024-03-06T07:22:49.079479+0000","last_became_active":"2024-03-06T07:22:49.077442+0000","last_became_peered":"2024-03-06T07:22:49.077442+0000","last_unstale":"2024-03-06T07:22:49.079479+0000","last_undegraded":"2024-03-06T07:22:49.079479+0000","last_fullsized":"2024-03-06T07:22:49.079479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:45.617755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.064402+0000","last_change":"2024-03-06T07:22:48.071210+0000","last_active":"2024-03-06T07:22:49.064402+0000","last_peered":"2024-03-06T07:22:49.064402+0000","last_clean":"2024-03-06T07:22:49.064402+0000","last_became_active":"2024-03-06T07:22:48.070893+0000","last_became_peered":"2024-03-06T07:22:48.070893+0000","last_unstale":"2024-03-06T07:22:49.064402+0000","last_undegraded":"2024-03-06T07:22:49.064402+0000","last_fullsized":"2024-03-06T07:22:49.064402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088822+0000","last_change":"2024-03-06T07:22:49.085398+0000","last_active":"2024-03-06T07:22:49.088822+0000","last_peered":"2024-03-06T07:22:49.088822+0000","last_clean":"2024-03-06T07:22:49.088822+0000","last_became_active":"2024-03-06T07:22:49.085124+0000","last_became_peered":"2024-03-06T07:22:49.085124+0000","last_unstale":"2024-03-06T07:22:49.088822+0000","last_undegraded":"2024-03-06T07:22:49.088822+0000","last_fullsized":"2024-03-06T07:22:49.088822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.066623+0000","last_change":"2024-03-06T07:22:48.071472+0000","last_active":"2024-03-06T07:22:49.066623+0000","last_peered":"2024-03-06T07:22:49.066623+0000","last_clean":"2024-03-06T07:22:49.066623+0000","last_became_active":"2024-03-06T07:22:48.071136+0000","last_became_peered":"2024-03-06T07:22:48.071136+0000","last_unstale":"2024-03-06T07:22:49.066623+0000","last_undegraded":"2024-03-06T07:22:49.066623+0000","last_fullsized":"2024-03-06T07:22:49.066623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:30.767002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076564+0000","last_change":"2024-03-06T07:22:49.072607+0000","last_active":"2024-03-06T07:22:49.076564+0000","last_peered":"2024-03-06T07:22:49.076564+0000","last_clean":"2024-03-06T07:22:49.076564+0000","last_became_active":"2024-03-06T07:22:49.071896+0000","last_became_peered":"2024-03-06T07:22:49.071896+0000","last_unstale":"2024-03-06T07:22:49.076564+0000","last_undegraded":"2024-03-06T07:22:49.076564+0000","last_fullsized":"2024-03-06T07:22:49.076564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065042+0000","last_change":"2024-03-06T07:22:48.071003+0000","last_active":"2024-03-06T07:22:49.065042+0000","last_peered":"2024-03-06T07:22:49.065042+0000","last_clean":"2024-03-06T07:22:49.065042+0000","last_became_active":"2024-03-06T07:22:48.070666+0000","last_became_peered":"2024-03-06T07:22:48.070666+0000","last_unstale":"2024-03-06T07:22:49.065042+0000","last_undegraded":"2024-03-06T07:22:49.065042+0000","last_fullsized":"2024-03-06T07:22:49.065042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086916+0000","last_change":"2024-03-06T07:22:49.077332+0000","last_active":"2024-03-06T07:22:49.086916+0000","last_peered":"2024-03-06T07:22:49.086916+0000","last_clean":"2024-03-06T07:22:49.086916+0000","last_became_active":"2024-03-06T07:22:49.077034+0000","last_became_peered":"2024-03-06T07:22:49.077034+0000","last_unstale":"2024-03-06T07:22:49.086916+0000","last_undegraded":"2024-03-06T07:22:49.086916+0000","last_fullsized":"2024-03-06T07:22:49.086916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084620+0000","last_change":"2024-03-06T07:22:48.076685+0000","last_active":"2024-03-06T07:22:49.084620+0000","last_peered":"2024-03-06T07:22:49.084620+0000","last_clean":"2024-03-06T07:22:49.084620+0000","last_became_active":"2024-03-06T07:22:48.076412+0000","last_became_peered":"2024-03-06T07:22:48.076412+0000","last_unstale":"2024-03-06T07:22:49.084620+0000","last_undegraded":"2024-03-06T07:22:49.084620+0000","last_fullsized":"2024-03-06T07:22:49.084620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078857+0000","last_change":"2024-03-06T07:22:49.069020+0000","last_active":"2024-03-06T07:22:49.078857+0000","last_peered":"2024-03-06T07:22:49.078857+0000","last_clean":"2024-03-06T07:22:49.078857+0000","last_became_active":"2024-03-06T07:22:49.068741+0000","last_became_peered":"2024-03-06T07:22:49.068741+0000","last_unstale":"2024-03-06T07:22:49.078857+0000","last_undegraded":"2024-03-06T07:22:49.078857+0000","last_fullsized":"2024-03-06T07:22:49.078857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:45.617755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065154+0000","last_change":"2024-03-06T07:22:48.072458+0000","last_active":"2024-03-06T07:22:49.065154+0000","last_peered":"2024-03-06T07:22:49.065154+0000","last_clean":"2024-03-06T07:22:49.065154+0000","last_became_active":"2024-03-06T07:22:48.071279+0000","last_became_peered":"2024-03-06T07:22:48.071279+0000","last_unstale":"2024-03-06T07:22:49.065154+0000","last_undegraded":"2024-03-06T07:22:49.065154+0000","last_fullsized":"2024-03-06T07:22:49.065154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079512+0000","last_change":"2024-03-06T07:22:49.077623+0000","last_active":"2024-03-06T07:22:49.079512+0000","last_peered":"2024-03-06T07:22:49.079512+0000","last_clean":"2024-03-06T07:22:49.079512+0000","last_became_active":"2024-03-06T07:22:49.076719+0000","last_became_peered":"2024-03-06T07:22:49.076719+0000","last_unstale":"2024-03-06T07:22:49.079512+0000","last_undegraded":"2024-03-06T07:22:49.079512+0000","last_fullsized":"2024-03-06T07:22:49.079512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:49.124109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.064860+0000","last_change":"2024-03-06T07:22:48.071228+0000","last_active":"2024-03-06T07:22:49.064860+0000","last_peered":"2024-03-06T07:22:49.064860+0000","last_clean":"2024-03-06T07:22:49.064860+0000","last_became_active":"2024-03-06T07:22:48.070898+0000","last_became_peered":"2024-03-06T07:22:48.070898+0000","last_unstale":"2024-03-06T07:22:49.064860+0000","last_undegraded":"2024-03-06T07:22:49.064860+0000","last_fullsized":"2024-03-06T07:22:49.064860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084272+0000","last_change":"2024-03-06T07:22:49.069977+0000","last_active":"2024-03-06T07:22:49.084272+0000","last_peered":"2024-03-06T07:22:49.084272+0000","last_clean":"2024-03-06T07:22:49.084272+0000","last_became_active":"2024-03-06T07:22:49.069691+0000","last_became_peered":"2024-03-06T07:22:49.069691+0000","last_unstale":"2024-03-06T07:22:49.084272+0000","last_undegraded":"2024-03-06T07:22:49.084272+0000","last_fullsized":"2024-03-06T07:22:49.084272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:53.011841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065247+0000","last_change":"2024-03-06T07:22:48.074693+0000","last_active":"2024-03-06T07:22:49.065247+0000","last_peered":"2024-03-06T07:22:49.065247+0000","last_clean":"2024-03-06T07:22:49.065247+0000","last_became_active":"2024-03-06T07:22:48.074261+0000","last_became_peered":"2024-03-06T07:22:48.074261+0000","last_unstale":"2024-03-06T07:22:49.065247+0000","last_undegraded":"2024-03-06T07:22:49.065247+0000","last_fullsized":"2024-03-06T07:22:49.065247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087034+0000","last_change":"2024-03-06T07:22:49.082024+0000","last_active":"2024-03-06T07:22:49.087034+0000","last_peered":"2024-03-06T07:22:49.087034+0000","last_clean":"2024-03-06T07:22:49.087034+0000","last_became_active":"2024-03-06T07:22:49.081534+0000","last_became_peered":"2024-03-06T07:22:49.081534+0000","last_unstale":"2024-03-06T07:22:49.087034+0000","last_undegraded":"2024-03-06T07:22:49.087034+0000","last_fullsized":"2024-03-06T07:22:49.087034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087583+0000","last_change":"2024-03-06T07:22:48.069179+0000","last_active":"2024-03-06T07:22:49.087583+0000","last_peered":"2024-03-06T07:22:49.087583+0000","last_clean":"2024-03-06T07:22:49.087583+0000","last_became_active":"2024-03-06T07:22:48.068402+0000","last_became_peered":"2024-03-06T07:22:48.068402+0000","last_unstale":"2024-03-06T07:22:49.087583+0000","last_undegraded":"2024-03-06T07:22:49.087583+0000","last_fullsized":"2024-03-06T07:22:49.087583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:09.655063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086941+0000","last_change":"2024-03-06T07:22:49.082657+0000","last_active":"2024-03-06T07:22:49.086941+0000","last_peered":"2024-03-06T07:22:49.086941+0000","last_clean":"2024-03-06T07:22:49.086941+0000","last_became_active":"2024-03-06T07:22:49.082096+0000","last_became_peered":"2024-03-06T07:22:49.082096+0000","last_unstale":"2024-03-06T07:22:49.086941+0000","last_undegraded":"2024-03-06T07:22:49.086941+0000","last_fullsized":"2024-03-06T07:22:49.086941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079643+0000","last_change":"2024-03-06T07:22:49.073629+0000","last_active":"2024-03-06T07:22:49.079643+0000","last_peered":"2024-03-06T07:22:49.079643+0000","last_clean":"2024-03-06T07:22:49.079643+0000","last_became_active":"2024-03-06T07:22:49.072839+0000","last_became_peered":"2024-03-06T07:22:49.072839+0000","last_unstale":"2024-03-06T07:22:49.079643+0000","last_undegraded":"2024-03-06T07:22:49.079643+0000","last_fullsized":"2024-03-06T07:22:49.079643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.085029+0000","last_change":"2024-03-06T07:22:48.072083+0000","last_active":"2024-03-06T07:22:49.085029+0000","last_peered":"2024-03-06T07:22:49.085029+0000","last_clean":"2024-03-06T07:22:49.085029+0000","last_became_active":"2024-03-06T07:22:48.071560+0000","last_became_peered":"2024-03-06T07:22:48.071560+0000","last_unstale":"2024-03-06T07:22:49.085029+0000","last_undegraded":"2024-03-06T07:22:49.085029+0000","last_fullsized":"2024-03-06T07:22:49.085029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082434+0000","last_change":"2024-03-06T07:22:49.080775+0000","last_active":"2024-03-06T07:22:49.082434+0000","last_peered":"2024-03-06T07:22:49.082434+0000","last_clean":"2024-03-06T07:22:49.082434+0000","last_became_active":"2024-03-06T07:22:49.080487+0000","last_became_peered":"2024-03-06T07:22:49.080487+0000","last_unstale":"2024-03-06T07:22:49.082434+0000","last_undegraded":"2024-03-06T07:22:49.082434+0000","last_fullsized":"2024-03-06T07:22:49.082434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087060+0000","last_change":"2024-03-06T07:22:48.074550+0000","last_active":"2024-03-06T07:22:49.087060+0000","last_peered":"2024-03-06T07:22:49.087060+0000","last_clean":"2024-03-06T07:22:49.087060+0000","last_became_active":"2024-03-06T07:22:48.073981+0000","last_became_peered":"2024-03-06T07:22:48.073981+0000","last_unstale":"2024-03-06T07:22:49.087060+0000","last_undegraded":"2024-03-06T07:22:49.087060+0000","last_fullsized":"2024-03-06T07:22:49.087060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:13.210796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078782+0000","last_change":"2024-03-06T07:22:49.075565+0000","last_active":"2024-03-06T07:22:49.078782+0000","last_peered":"2024-03-06T07:22:49.078782+0000","last_clean":"2024-03-06T07:22:49.078782+0000","last_became_active":"2024-03-06T07:22:49.075254+0000","last_became_peered":"2024-03-06T07:22:49.075254+0000","last_unstale":"2024-03-06T07:22:49.078782+0000","last_undegraded":"2024-03-06T07:22:49.078782+0000","last_fullsized":"2024-03-06T07:22:49.078782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:46.459505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088279+0000","last_change":"2024-03-06T07:22:48.073512+0000","last_active":"2024-03-06T07:22:49.088279+0000","last_peered":"2024-03-06T07:22:49.088279+0000","last_clean":"2024-03-06T07:22:49.088279+0000","last_became_active":"2024-03-06T07:22:48.073181+0000","last_became_peered":"2024-03-06T07:22:48.073181+0000","last_unstale":"2024-03-06T07:22:49.088279+0000","last_undegraded":"2024-03-06T07:22:49.088279+0000","last_fullsized":"2024-03-06T07:22:49.088279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088433+0000","last_change":"2024-03-06T07:22:49.079451+0000","last_active":"2024-03-06T07:22:49.088433+0000","last_peered":"2024-03-06T07:22:49.088433+0000","last_clean":"2024-03-06T07:22:49.088433+0000","last_became_active":"2024-03-06T07:22:49.078927+0000","last_became_peered":"2024-03-06T07:22:49.078927+0000","last_unstale":"2024-03-06T07:22:49.088433+0000","last_undegraded":"2024-03-06T07:22:49.088433+0000","last_fullsized":"2024-03-06T07:22:49.088433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.090890+0000","last_change":"2024-03-06T07:22:48.062906+0000","last_active":"2024-03-06T07:22:49.090890+0000","last_peered":"2024-03-06T07:22:49.090890+0000","last_clean":"2024-03-06T07:22:49.090890+0000","last_became_active":"2024-03-06T07:22:48.062497+0000","last_became_peered":"2024-03-06T07:22:48.062497+0000","last_unstale":"2024-03-06T07:22:49.090890+0000","last_undegraded":"2024-03-06T07:22:49.090890+0000","last_fullsized":"2024-03-06T07:22:49.090890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:51.443747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":660255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":69397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":69465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":660311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:53.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.239+0000 7f0056471640 1 Processor -- start 2024-03-06T07:22:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.239+0000 7f0056471640 1 -- start start 2024-03-06T07:22:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.239+0000 7f0056471640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0050071540 0x7f00500b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.239+0000 7f0056471640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00500d3400 0x7f0050071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.239+0000 7f0056471640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0050071e60 con 0x7f00500b2b50 2024-03-06T07:22:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.239+0000 7f0056471640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00500d2f70 con 0x7f0050071540 2024-03-06T07:22:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.239+0000 7f0056471640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00500d4be0 con 0x7f00500d3400 2024-03-06T07:22:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0050071540 0x7f00500b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00500d3400 0x7f0050071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00500d3400 0x7f0050071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:60628/0 (socket says 172.21.15.27:60628) 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f0054a0e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f00500b2b50 0x7f00500d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:43784/0 (socket says 172.21.15.27:43784) 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004f7fe640 1 -- 172.21.15.27:0/445089141 learned_addr learned my addr 172.21.15.27:0/445089141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0050071540 0x7f00500b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:38418/0 (socket says 172.21.15.27:38418) 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004f7fe640 1 -- 172.21.15.27:0/445089141 >> v2:172.21.15.69:3300/0 conn(0x7f0050071540 msgr2=0x7f00500b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004f7fe640 1 --2- 172.21.15.27:0/445089141 >> v2:172.21.15.69:3300/0 conn(0x7f0050071540 0x7f00500b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004f7fe640 1 -- 172.21.15.27:0/445089141 >> v1:172.21.15.69:6789/0 conn(0x7f00500b2b50 legacy=0x7f00500d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004f7fe640 1 -- 172.21.15.27:0/445089141 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00500ce610 con 0x7f00500d3400 2024-03-06T07:22:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.240+0000 7f004ffff640 1 --2- 172.21.15.27:0/445089141 >> v2:172.21.15.69:3300/0 conn(0x7f0050071540 0x7f00500b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:53.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.241+0000 7f004f7fe640 1 --2- 172.21.15.27:0/445089141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00500d3400 0x7f0050071920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0044009870 tx=0x7f004402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d17832b10eca393 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.241+0000 7f004effd640 1 -- 172.21.15.27:0/445089141 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0044034070 con 0x7f00500d3400 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.241+0000 7f004effd640 1 -- 172.21.15.27:0/445089141 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0044030840 con 0x7f00500d3400 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.241+0000 7f004effd640 1 -- 172.21.15.27:0/445089141 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0044030c10 con 0x7f00500d3400 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.242+0000 7f0056471640 1 -- 172.21.15.27:0/445089141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00500d3400 msgr2=0x7f0050071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.242+0000 7f0056471640 1 --2- 172.21.15.27:0/445089141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00500d3400 0x7f0050071920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0044009870 tx=0x7f004402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.242+0000 7f0056471640 1 -- 172.21.15.27:0/445089141 shutdown_connections 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.242+0000 7f0056471640 1 --2- 172.21.15.27:0/445089141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00500d3400 0x7f0050071920 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.242+0000 7f0056471640 1 --2- 172.21.15.27:0/445089141 >> v2:172.21.15.69:3300/0 conn(0x7f0050071540 0x7f00500b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.242+0000 7f0056471640 1 -- 172.21.15.27:0/445089141 >> 172.21.15.27:0/445089141 conn(0x7f00500d3ca0 msgr2=0x7f00500d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.242+0000 7f0056471640 1 -- 172.21.15.27:0/445089141 shutdown_connections 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.242+0000 7f0056471640 1 -- 172.21.15.27:0/445089141 wait complete. 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.243+0000 7f0056471640 1 Processor -- start 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.243+0000 7f0056471640 1 -- start start 2024-03-06T07:22:53.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0056471640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0050071540 0x7f00500ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0056471640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00500b2b50 0x7f00501f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0056471640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00500d3400 0x7f00500cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0056471640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00500ce8f0 con 0x7f00500d3400 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0056471640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00500d2c00 con 0x7f00500b2b50 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0056471640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005015e370 con 0x7f0050071540 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0054a0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00500d3400 0x7f00500cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0054a0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00500d3400 0x7f00500cd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58520/0 (socket says 172.21.15.27:58520) 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0054a0e640 1 -- 172.21.15.27:0/980832548 learned_addr learned my addr 172.21.15.27:0/980832548 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f004f7fe640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00500b2b50 0x7f00501f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f004ffff640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0050071540 0x7f00500ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0054a0e640 1 -- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0050071540 msgr2=0x7f00500ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0054a0e640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0050071540 0x7f00500ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0054a0e640 1 -- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00500b2b50 msgr2=0x7f00501f62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0054a0e640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00500b2b50 0x7f00501f62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.244+0000 7f0054a0e640 1 -- 172.21.15.27:0/980832548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00500c96f0 con 0x7f00500d3400 2024-03-06T07:22:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.245+0000 7f0054a0e640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00500d3400 0x7f00500cd200 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f004800c780 tx=0x7f004800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.245+0000 7f004cff9640 1 -- 172.21.15.27:0/980832548 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0048017070 con 0x7f00500d3400 2024-03-06T07:22:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.245+0000 7f004cff9640 1 -- 172.21.15.27:0/980832548 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004800a640 con 0x7f00500d3400 2024-03-06T07:22:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.245+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00500c99d0 con 0x7f00500d3400 2024-03-06T07:22:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.245+0000 7f004cff9640 1 -- 172.21.15.27:0/980832548 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004800a9f0 con 0x7f00500d3400 2024-03-06T07:22:53.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.245+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00500c9f90 con 0x7f00500d3400 2024-03-06T07:22:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.246+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0050068700 con 0x7f00500d3400 2024-03-06T07:22:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.246+0000 7f004cff9640 1 -- 172.21.15.27:0/980832548 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f004801a070 con 0x7f00500d3400 2024-03-06T07:22:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.247+0000 7f004cff9640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003c06b4d0 0x7f003c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.247+0000 7f004cff9640 1 -- 172.21.15.27:0/980832548 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00480897c0 con 0x7f00500d3400 2024-03-06T07:22:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.247+0000 7f004ffff640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003c06b4d0 0x7f003c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.248+0000 7f004ffff640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003c06b4d0 0x7f003c06d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0040009560 tx=0x7f0040009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.251+0000 7f004cff9640 1 -- 172.21.15.27:0/980832548 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0048057f60 con 0x7f00500d3400 2024-03-06T07:22:53.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.431+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00500ca370 con 0x7f003c06b4d0 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.438+0000 7f004cff9640 1 -- 172.21.15.27:0/980832548 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251876 (secure 0 0 0) 0x7f00500ca370 con 0x7f003c06b4d0 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:53.444 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:22:53.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003c06b4d0 msgr2=0x7f003c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003c06b4d0 0x7f003c06d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0040009560 tx=0x7f0040009190 comp rx=0 tx=0).stop 2024-03-06T07:22:53.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00500d3400 msgr2=0x7f00500cd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00500d3400 0x7f00500cd200 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f004800c780 tx=0x7f004800cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 shutdown_connections 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003c06b4d0 0x7f003c06d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00500d3400 0x7f00500cd200 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00500b2b50 0x7f00501f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 --2- 172.21.15.27:0/980832548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0050071540 0x7f00500ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 >> 172.21.15.27:0/980832548 conn(0x7f00500d3ca0 msgr2=0x7f0050160ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 shutdown_connections 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.441+0000 7f0056471640 1 -- 172.21.15.27:0/980832548 wait complete. 2024-03-06T07:22:53.454 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:22:51.987161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218988,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755732,"statfs":{"total":773094113280,"available":772869869568,"internally_reserved":0,"allocated":2887680,"data_stored":1739962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.939392"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082744+0000","last_change":"2024-03-06T07:22:49.080233+0000","last_active":"2024-03-06T07:22:49.082744+0000","last_peered":"2024-03-06T07:22:49.082744+0000","last_clean":"2024-03-06T07:22:49.082744+0000","last_became_active":"2024-03-06T07:22:49.079770+0000","last_became_peered":"2024-03-06T07:22:49.079770+0000","last_unstale":"2024-03-06T07:22:49.082744+0000","last_undegraded":"2024-03-06T07:22:49.082744+0000","last_fullsized":"2024-03-06T07:22:49.082744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.083749+0000","last_change":"2024-03-06T07:22:49.081636+0000","last_active":"2024-03-06T07:22:49.083749+0000","last_peered":"2024-03-06T07:22:49.083749+0000","last_clean":"2024-03-06T07:22:49.083749+0000","last_became_active":"2024-03-06T07:22:49.081356+0000","last_became_peered":"2024-03-06T07:22:49.081356+0000","last_unstale":"2024-03-06T07:22:49.083749+0000","last_undegraded":"2024-03-06T07:22:49.083749+0000","last_fullsized":"2024-03-06T07:22:49.083749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.080060+0000","last_change":"2024-03-06T07:22:49.076314+0000","last_active":"2024-03-06T07:22:49.080060+0000","last_peered":"2024-03-06T07:22:49.080060+0000","last_clean":"2024-03-06T07:22:49.080060+0000","last_became_active":"2024-03-06T07:22:49.075901+0000","last_became_peered":"2024-03-06T07:22:49.075901+0000","last_unstale":"2024-03-06T07:22:49.080060+0000","last_undegraded":"2024-03-06T07:22:49.080060+0000","last_fullsized":"2024-03-06T07:22:49.080060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089026+0000","last_change":"2024-03-06T07:22:49.086730+0000","last_active":"2024-03-06T07:22:49.089026+0000","last_peered":"2024-03-06T07:22:49.089026+0000","last_clean":"2024-03-06T07:22:49.089026+0000","last_became_active":"2024-03-06T07:22:49.086475+0000","last_became_peered":"2024-03-06T07:22:49.086475+0000","last_unstale":"2024-03-06T07:22:49.089026+0000","last_undegraded":"2024-03-06T07:22:49.089026+0000","last_fullsized":"2024-03-06T07:22:49.089026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076094+0000","last_change":"2024-03-06T07:22:49.074936+0000","last_active":"2024-03-06T07:22:49.076094+0000","last_peered":"2024-03-06T07:22:49.076094+0000","last_clean":"2024-03-06T07:22:49.076094+0000","last_became_active":"2024-03-06T07:22:49.074662+0000","last_became_peered":"2024-03-06T07:22:49.074662+0000","last_unstale":"2024-03-06T07:22:49.076094+0000","last_undegraded":"2024-03-06T07:22:49.076094+0000","last_fullsized":"2024-03-06T07:22:49.076094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.031324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.083008+0000","last_change":"2024-03-06T07:22:49.081940+0000","last_active":"2024-03-06T07:22:49.083008+0000","last_peered":"2024-03-06T07:22:49.083008+0000","last_clean":"2024-03-06T07:22:49.083008+0000","last_became_active":"2024-03-06T07:22:49.081668+0000","last_became_peered":"2024-03-06T07:22:49.081668+0000","last_unstale":"2024-03-06T07:22:49.083008+0000","last_undegraded":"2024-03-06T07:22:49.083008+0000","last_fullsized":"2024-03-06T07:22:49.083008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089412+0000","last_change":"2024-03-06T07:22:49.088117+0000","last_active":"2024-03-06T07:22:49.089412+0000","last_peered":"2024-03-06T07:22:49.089412+0000","last_clean":"2024-03-06T07:22:49.089412+0000","last_became_active":"2024-03-06T07:22:49.087856+0000","last_became_peered":"2024-03-06T07:22:49.087856+0000","last_unstale":"2024-03-06T07:22:49.089412+0000","last_undegraded":"2024-03-06T07:22:49.089412+0000","last_fullsized":"2024-03-06T07:22:49.089412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:07.781375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084421+0000","last_change":"2024-03-06T07:22:49.074570+0000","last_active":"2024-03-06T07:22:49.084421+0000","last_peered":"2024-03-06T07:22:49.084421+0000","last_clean":"2024-03-06T07:22:49.084421+0000","last_became_active":"2024-03-06T07:22:49.073115+0000","last_became_peered":"2024-03-06T07:22:49.073115+0000","last_unstale":"2024-03-06T07:22:49.084421+0000","last_undegraded":"2024-03-06T07:22:49.084421+0000","last_fullsized":"2024-03-06T07:22:49.084421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.075698+0000","last_change":"2024-03-06T07:22:49.073716+0000","last_active":"2024-03-06T07:22:49.075698+0000","last_peered":"2024-03-06T07:22:49.075698+0000","last_clean":"2024-03-06T07:22:49.075698+0000","last_became_active":"2024-03-06T07:22:49.073434+0000","last_became_peered":"2024-03-06T07:22:49.073434+0000","last_unstale":"2024-03-06T07:22:49.075698+0000","last_undegraded":"2024-03-06T07:22:49.075698+0000","last_fullsized":"2024-03-06T07:22:49.075698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.073661+0000","last_change":"2024-03-06T07:22:49.072470+0000","last_active":"2024-03-06T07:22:49.073661+0000","last_peered":"2024-03-06T07:22:49.073661+0000","last_clean":"2024-03-06T07:22:49.073661+0000","last_became_active":"2024-03-06T07:22:49.072206+0000","last_became_peered":"2024-03-06T07:22:49.072206+0000","last_unstale":"2024-03-06T07:22:49.073661+0000","last_undegraded":"2024-03-06T07:22:49.073661+0000","last_fullsized":"2024-03-06T07:22:49.073661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082251+0000","last_change":"2024-03-06T07:22:49.076507+0000","last_active":"2024-03-06T07:22:49.082251+0000","last_peered":"2024-03-06T07:22:49.082251+0000","last_clean":"2024-03-06T07:22:49.082251+0000","last_became_active":"2024-03-06T07:22:49.076054+0000","last_became_peered":"2024-03-06T07:22:49.076054+0000","last_unstale":"2024-03-06T07:22:49.082251+0000","last_undegraded":"2024-03-06T07:22:49.082251+0000","last_fullsized":"2024-03-06T07:22:49.082251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079175+0000","last_change":"2024-03-06T07:22:49.077078+0000","last_active":"2024-03-06T07:22:49.079175+0000","last_peered":"2024-03-06T07:22:49.079175+0000","last_clean":"2024-03-06T07:22:49.079175+0000","last_became_active":"2024-03-06T07:22:49.076757+0000","last_became_peered":"2024-03-06T07:22:49.076757+0000","last_unstale":"2024-03-06T07:22:49.079175+0000","last_undegraded":"2024-03-06T07:22:49.079175+0000","last_fullsized":"2024-03-06T07:22:49.079175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:52.449738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084528+0000","last_change":"2024-03-06T07:22:49.073467+0000","last_active":"2024-03-06T07:22:49.084528+0000","last_peered":"2024-03-06T07:22:49.084528+0000","last_clean":"2024-03-06T07:22:49.084528+0000","last_became_active":"2024-03-06T07:22:49.072969+0000","last_became_peered":"2024-03-06T07:22:49.072969+0000","last_unstale":"2024-03-06T07:22:49.084528+0000","last_undegraded":"2024-03-06T07:22:49.084528+0000","last_fullsized":"2024-03-06T07:22:49.084528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:51.096875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078810+0000","last_change":"2024-03-06T07:22:49.076207+0000","last_active":"2024-03-06T07:22:49.078810+0000","last_peered":"2024-03-06T07:22:49.078810+0000","last_clean":"2024-03-06T07:22:49.078810+0000","last_became_active":"2024-03-06T07:22:49.074449+0000","last_became_peered":"2024-03-06T07:22:49.074449+0000","last_unstale":"2024-03-06T07:22:49.078810+0000","last_undegraded":"2024-03-06T07:22:49.078810+0000","last_fullsized":"2024-03-06T07:22:49.078810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:51.096875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.077970+0000","last_change":"2024-03-06T07:22:49.072745+0000","last_active":"2024-03-06T07:22:49.077970+0000","last_peered":"2024-03-06T07:22:49.077970+0000","last_clean":"2024-03-06T07:22:49.077970+0000","last_became_active":"2024-03-06T07:22:49.072443+0000","last_became_peered":"2024-03-06T07:22:49.072443+0000","last_unstale":"2024-03-06T07:22:49.077970+0000","last_undegraded":"2024-03-06T07:22:49.077970+0000","last_fullsized":"2024-03-06T07:22:49.077970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088847+0000","last_change":"2024-03-06T07:22:49.085335+0000","last_active":"2024-03-06T07:22:49.088847+0000","last_peered":"2024-03-06T07:22:49.088847+0000","last_clean":"2024-03-06T07:22:49.088847+0000","last_became_active":"2024-03-06T07:22:49.085023+0000","last_became_peered":"2024-03-06T07:22:49.085023+0000","last_unstale":"2024-03-06T07:22:49.088847+0000","last_undegraded":"2024-03-06T07:22:49.088847+0000","last_fullsized":"2024-03-06T07:22:49.088847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.031324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079704+0000","last_change":"2024-03-06T07:22:49.073395+0000","last_active":"2024-03-06T07:22:49.079704+0000","last_peered":"2024-03-06T07:22:49.079704+0000","last_clean":"2024-03-06T07:22:49.079704+0000","last_became_active":"2024-03-06T07:22:49.073105+0000","last_became_peered":"2024-03-06T07:22:49.073105+0000","last_unstale":"2024-03-06T07:22:49.079704+0000","last_undegraded":"2024-03-06T07:22:49.079704+0000","last_fullsized":"2024-03-06T07:22:49.079704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.080226+0000","last_change":"2024-03-06T07:22:49.074912+0000","last_active":"2024-03-06T07:22:49.080226+0000","last_peered":"2024-03-06T07:22:49.080226+0000","last_clean":"2024-03-06T07:22:49.080226+0000","last_became_active":"2024-03-06T07:22:49.074638+0000","last_became_peered":"2024-03-06T07:22:49.074638+0000","last_unstale":"2024-03-06T07:22:49.080226+0000","last_undegraded":"2024-03-06T07:22:49.080226+0000","last_fullsized":"2024-03-06T07:22:49.080226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076698+0000","last_change":"2024-03-06T07:22:49.075473+0000","last_active":"2024-03-06T07:22:49.076698+0000","last_peered":"2024-03-06T07:22:49.076698+0000","last_clean":"2024-03-06T07:22:49.076698+0000","last_became_active":"2024-03-06T07:22:49.074945+0000","last_became_peered":"2024-03-06T07:22:49.074945+0000","last_unstale":"2024-03-06T07:22:49.076698+0000","last_undegraded":"2024-03-06T07:22:49.076698+0000","last_fullsized":"2024-03-06T07:22:49.076698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087015+0000","last_change":"2024-03-06T07:22:49.085348+0000","last_active":"2024-03-06T07:22:49.087015+0000","last_peered":"2024-03-06T07:22:49.087015+0000","last_clean":"2024-03-06T07:22:49.087015+0000","last_became_active":"2024-03-06T07:22:49.084955+0000","last_became_peered":"2024-03-06T07:22:49.084955+0000","last_unstale":"2024-03-06T07:22:49.087015+0000","last_undegraded":"2024-03-06T07:22:49.087015+0000","last_fullsized":"2024-03-06T07:22:49.087015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.090574+0000","last_change":"2024-03-06T07:22:49.081034+0000","last_active":"2024-03-06T07:22:49.090574+0000","last_peered":"2024-03-06T07:22:49.090574+0000","last_clean":"2024-03-06T07:22:49.090574+0000","last_became_active":"2024-03-06T07:22:49.080639+0000","last_became_peered":"2024-03-06T07:22:49.080639+0000","last_unstale":"2024-03-06T07:22:49.090574+0000","last_undegraded":"2024-03-06T07:22:49.090574+0000","last_fullsized":"2024-03-06T07:22:49.090574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079734+0000","last_change":"2024-03-06T07:22:49.074317+0000","last_active":"2024-03-06T07:22:49.079734+0000","last_peered":"2024-03-06T07:22:49.079734+0000","last_clean":"2024-03-06T07:22:49.079734+0000","last_became_active":"2024-03-06T07:22:49.073041+0000","last_became_peered":"2024-03-06T07:22:49.073041+0000","last_unstale":"2024-03-06T07:22:49.079734+0000","last_undegraded":"2024-03-06T07:22:49.079734+0000","last_fullsized":"2024-03-06T07:22:49.079734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088695+0000","last_change":"2024-03-06T07:22:49.076771+0000","last_active":"2024-03-06T07:22:49.088695+0000","last_peered":"2024-03-06T07:22:49.088695+0000","last_clean":"2024-03-06T07:22:49.088695+0000","last_became_active":"2024-03-06T07:22:49.073031+0000","last_became_peered":"2024-03-06T07:22:49.073031+0000","last_unstale":"2024-03-06T07:22:49.088695+0000","last_undegraded":"2024-03-06T07:22:49.088695+0000","last_fullsized":"2024-03-06T07:22:49.088695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088936+0000","last_change":"2024-03-06T07:22:49.086108+0000","last_active":"2024-03-06T07:22:49.088936+0000","last_peered":"2024-03-06T07:22:49.088936+0000","last_clean":"2024-03-06T07:22:49.088936+0000","last_became_active":"2024-03-06T07:22:49.085825+0000","last_became_peered":"2024-03-06T07:22:49.085825+0000","last_unstale":"2024-03-06T07:22:49.088936+0000","last_undegraded":"2024-03-06T07:22:49.088936+0000","last_fullsized":"2024-03-06T07:22:49.088936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084382+0000","last_change":"2024-03-06T07:22:49.071422+0000","last_active":"2024-03-06T07:22:49.084382+0000","last_peered":"2024-03-06T07:22:49.084382+0000","last_clean":"2024-03-06T07:22:49.084382+0000","last_became_active":"2024-03-06T07:22:49.071117+0000","last_became_peered":"2024-03-06T07:22:49.071117+0000","last_unstale":"2024-03-06T07:22:49.084382+0000","last_undegraded":"2024-03-06T07:22:49.084382+0000","last_fullsized":"2024-03-06T07:22:49.084382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:38.976172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082892+0000","last_change":"2024-03-06T07:22:49.077927+0000","last_active":"2024-03-06T07:22:49.082892+0000","last_peered":"2024-03-06T07:22:49.082892+0000","last_clean":"2024-03-06T07:22:49.082892+0000","last_became_active":"2024-03-06T07:22:49.077647+0000","last_became_peered":"2024-03-06T07:22:49.077647+0000","last_unstale":"2024-03-06T07:22:49.082892+0000","last_undegraded":"2024-03-06T07:22:49.082892+0000","last_fullsized":"2024-03-06T07:22:49.082892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088325+0000","last_change":"2024-03-06T07:22:49.085345+0000","last_active":"2024-03-06T07:22:49.088325+0000","last_peered":"2024-03-06T07:22:49.088325+0000","last_clean":"2024-03-06T07:22:49.088325+0000","last_became_active":"2024-03-06T07:22:49.084942+0000","last_became_peered":"2024-03-06T07:22:49.084942+0000","last_unstale":"2024-03-06T07:22:49.088325+0000","last_undegraded":"2024-03-06T07:22:49.088325+0000","last_fullsized":"2024-03-06T07:22:49.088325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084426+0000","last_change":"2024-03-06T07:22:49.071314+0000","last_active":"2024-03-06T07:22:49.084426+0000","last_peered":"2024-03-06T07:22:49.084426+0000","last_clean":"2024-03-06T07:22:49.084426+0000","last_became_active":"2024-03-06T07:22:49.070986+0000","last_became_peered":"2024-03-06T07:22:49.070986+0000","last_unstale":"2024-03-06T07:22:49.084426+0000","last_undegraded":"2024-03-06T07:22:49.084426+0000","last_fullsized":"2024-03-06T07:22:49.084426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:46.459505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.083562+0000","last_change":"2024-03-06T07:22:49.077320+0000","last_active":"2024-03-06T07:22:49.083562+0000","last_peered":"2024-03-06T07:22:49.083562+0000","last_clean":"2024-03-06T07:22:49.083562+0000","last_became_active":"2024-03-06T07:22:49.077017+0000","last_became_peered":"2024-03-06T07:22:49.077017+0000","last_unstale":"2024-03-06T07:22:49.083562+0000","last_undegraded":"2024-03-06T07:22:49.083562+0000","last_fullsized":"2024-03-06T07:22:49.083562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088945+0000","last_change":"2024-03-06T07:22:49.086200+0000","last_active":"2024-03-06T07:22:49.088945+0000","last_peered":"2024-03-06T07:22:49.088945+0000","last_clean":"2024-03-06T07:22:49.088945+0000","last_became_active":"2024-03-06T07:22:49.085793+0000","last_became_peered":"2024-03-06T07:22:49.085793+0000","last_unstale":"2024-03-06T07:22:49.088945+0000","last_undegraded":"2024-03-06T07:22:49.088945+0000","last_fullsized":"2024-03-06T07:22:49.088945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086818+0000","last_change":"2024-03-06T07:22:49.076891+0000","last_active":"2024-03-06T07:22:49.086818+0000","last_peered":"2024-03-06T07:22:49.086818+0000","last_clean":"2024-03-06T07:22:49.086818+0000","last_became_active":"2024-03-06T07:22:49.076610+0000","last_became_peered":"2024-03-06T07:22:49.076610+0000","last_unstale":"2024-03-06T07:22:49.086818+0000","last_undegraded":"2024-03-06T07:22:49.086818+0000","last_fullsized":"2024-03-06T07:22:49.086818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089380+0000","last_change":"2024-03-06T07:22:49.088635+0000","last_active":"2024-03-06T07:22:49.089380+0000","last_peered":"2024-03-06T07:22:49.089380+0000","last_clean":"2024-03-06T07:22:49.089380+0000","last_became_active":"2024-03-06T07:22:49.088352+0000","last_became_peered":"2024-03-06T07:22:49.088352+0000","last_unstale":"2024-03-06T07:22:49.089380+0000","last_undegraded":"2024-03-06T07:22:49.089380+0000","last_fullsized":"2024-03-06T07:22:49.089380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.072012+0000","last_change":"2024-03-06T07:22:49.071168+0000","last_active":"2024-03-06T07:22:49.072012+0000","last_peered":"2024-03-06T07:22:49.072012+0000","last_clean":"2024-03-06T07:22:49.072012+0000","last_became_active":"2024-03-06T07:22:49.070861+0000","last_became_peered":"2024-03-06T07:22:49.070861+0000","last_unstale":"2024-03-06T07:22:49.072012+0000","last_undegraded":"2024-03-06T07:22:49.072012+0000","last_fullsized":"2024-03-06T07:22:49.072012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:20.360478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086951+0000","last_change":"2024-03-06T07:22:48.076442+0000","last_active":"2024-03-06T07:22:49.086951+0000","last_peered":"2024-03-06T07:22:49.086951+0000","last_clean":"2024-03-06T07:22:49.086951+0000","last_became_active":"2024-03-06T07:22:48.076193+0000","last_became_peered":"2024-03-06T07:22:48.076193+0000","last_unstale":"2024-03-06T07:22:49.086951+0000","last_undegraded":"2024-03-06T07:22:49.086951+0000","last_fullsized":"2024-03-06T07:22:49.086951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:48.118117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.080355+0000","last_change":"2024-03-06T07:22:49.067928+0000","last_active":"2024-03-06T07:22:49.080355+0000","last_peered":"2024-03-06T07:22:49.080355+0000","last_clean":"2024-03-06T07:22:49.080355+0000","last_became_active":"2024-03-06T07:22:49.066102+0000","last_became_peered":"2024-03-06T07:22:49.066102+0000","last_unstale":"2024-03-06T07:22:49.080355+0000","last_undegraded":"2024-03-06T07:22:49.080355+0000","last_fullsized":"2024-03-06T07:22:49.080355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:49.124109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086794+0000","last_change":"2024-03-06T07:22:48.072246+0000","last_active":"2024-03-06T07:22:49.086794+0000","last_peered":"2024-03-06T07:22:49.086794+0000","last_clean":"2024-03-06T07:22:49.086794+0000","last_became_active":"2024-03-06T07:22:48.071947+0000","last_became_peered":"2024-03-06T07:22:48.071947+0000","last_unstale":"2024-03-06T07:22:49.086794+0000","last_undegraded":"2024-03-06T07:22:49.086794+0000","last_fullsized":"2024-03-06T07:22:49.086794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.062281+0000","last_change":"2024-03-06T07:22:48.075039+0000","last_active":"2024-03-06T07:22:49.062281+0000","last_peered":"2024-03-06T07:22:49.062281+0000","last_clean":"2024-03-06T07:22:49.062281+0000","last_became_active":"2024-03-06T07:22:48.074175+0000","last_became_peered":"2024-03-06T07:22:48.074175+0000","last_unstale":"2024-03-06T07:22:49.062281+0000","last_undegraded":"2024-03-06T07:22:49.062281+0000","last_fullsized":"2024-03-06T07:22:49.062281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:51.443747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086954+0000","last_change":"2024-03-06T07:22:49.081619+0000","last_active":"2024-03-06T07:22:49.086954+0000","last_peered":"2024-03-06T07:22:49.086954+0000","last_clean":"2024-03-06T07:22:49.086954+0000","last_became_active":"2024-03-06T07:22:49.081307+0000","last_became_peered":"2024-03-06T07:22:49.081307+0000","last_unstale":"2024-03-06T07:22:49.086954+0000","last_undegraded":"2024-03-06T07:22:49.086954+0000","last_fullsized":"2024-03-06T07:22:49.086954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065442+0000","last_change":"2024-03-06T07:22:48.070889+0000","last_active":"2024-03-06T07:22:49.065442+0000","last_peered":"2024-03-06T07:22:49.065442+0000","last_clean":"2024-03-06T07:22:49.065442+0000","last_became_active":"2024-03-06T07:22:48.070591+0000","last_became_peered":"2024-03-06T07:22:48.070591+0000","last_unstale":"2024-03-06T07:22:49.065442+0000","last_undegraded":"2024-03-06T07:22:49.065442+0000","last_fullsized":"2024-03-06T07:22:49.065442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.072890+0000","last_change":"2024-03-06T07:22:49.071108+0000","last_active":"2024-03-06T07:22:49.072890+0000","last_peered":"2024-03-06T07:22:49.072890+0000","last_clean":"2024-03-06T07:22:49.072890+0000","last_became_active":"2024-03-06T07:22:49.070830+0000","last_became_peered":"2024-03-06T07:22:49.070830+0000","last_unstale":"2024-03-06T07:22:49.072890+0000","last_undegraded":"2024-03-06T07:22:49.072890+0000","last_fullsized":"2024-03-06T07:22:49.072890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.064187+0000","last_change":"2024-03-06T07:22:48.073692+0000","last_active":"2024-03-06T07:22:49.064187+0000","last_peered":"2024-03-06T07:22:49.064187+0000","last_clean":"2024-03-06T07:22:49.064187+0000","last_became_active":"2024-03-06T07:22:48.073366+0000","last_became_peered":"2024-03-06T07:22:48.073366+0000","last_unstale":"2024-03-06T07:22:49.064187+0000","last_undegraded":"2024-03-06T07:22:49.064187+0000","last_fullsized":"2024-03-06T07:22:49.064187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.081385+0000","last_change":"2024-03-06T07:22:49.075374+0000","last_active":"2024-03-06T07:22:49.081385+0000","last_peered":"2024-03-06T07:22:49.081385+0000","last_clean":"2024-03-06T07:22:49.081385+0000","last_became_active":"2024-03-06T07:22:49.073115+0000","last_became_peered":"2024-03-06T07:22:49.073115+0000","last_unstale":"2024-03-06T07:22:49.081385+0000","last_undegraded":"2024-03-06T07:22:49.081385+0000","last_fullsized":"2024-03-06T07:22:49.081385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.066295+0000","last_change":"2024-03-06T07:22:48.074598+0000","last_active":"2024-03-06T07:22:49.066295+0000","last_peered":"2024-03-06T07:22:49.066295+0000","last_clean":"2024-03-06T07:22:49.066295+0000","last_became_active":"2024-03-06T07:22:48.074291+0000","last_became_peered":"2024-03-06T07:22:48.074291+0000","last_unstale":"2024-03-06T07:22:49.066295+0000","last_undegraded":"2024-03-06T07:22:49.066295+0000","last_fullsized":"2024-03-06T07:22:49.066295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078759+0000","last_change":"2024-03-06T07:22:49.075698+0000","last_active":"2024-03-06T07:22:49.078759+0000","last_peered":"2024-03-06T07:22:49.078759+0000","last_clean":"2024-03-06T07:22:49.078759+0000","last_became_active":"2024-03-06T07:22:49.075395+0000","last_became_peered":"2024-03-06T07:22:49.075395+0000","last_unstale":"2024-03-06T07:22:49.078759+0000","last_undegraded":"2024-03-06T07:22:49.078759+0000","last_fullsized":"2024-03-06T07:22:49.078759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:38.976172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089292+0000","last_change":"2024-03-06T07:22:48.071600+0000","last_active":"2024-03-06T07:22:49.089292+0000","last_peered":"2024-03-06T07:22:49.089292+0000","last_clean":"2024-03-06T07:22:49.089292+0000","last_became_active":"2024-03-06T07:22:48.071194+0000","last_became_peered":"2024-03-06T07:22:48.071194+0000","last_unstale":"2024-03-06T07:22:49.089292+0000","last_undegraded":"2024-03-06T07:22:49.089292+0000","last_fullsized":"2024-03-06T07:22:49.089292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079453+0000","last_change":"2024-03-06T07:22:49.078266+0000","last_active":"2024-03-06T07:22:49.079453+0000","last_peered":"2024-03-06T07:22:49.079453+0000","last_clean":"2024-03-06T07:22:49.079453+0000","last_became_active":"2024-03-06T07:22:49.077993+0000","last_became_peered":"2024-03-06T07:22:49.077993+0000","last_unstale":"2024-03-06T07:22:49.079453+0000","last_undegraded":"2024-03-06T07:22:49.079453+0000","last_fullsized":"2024-03-06T07:22:49.079453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:53.011841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.095110+0000","last_change":"2024-03-06T07:22:48.074028+0000","last_active":"2024-03-06T07:22:49.095110+0000","last_peered":"2024-03-06T07:22:49.095110+0000","last_clean":"2024-03-06T07:22:49.095110+0000","last_became_active":"2024-03-06T07:22:48.073737+0000","last_became_peered":"2024-03-06T07:22:48.073737+0000","last_unstale":"2024-03-06T07:22:49.095110+0000","last_undegraded":"2024-03-06T07:22:49.095110+0000","last_fullsized":"2024-03-06T07:22:49.095110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076124+0000","last_change":"2024-03-06T07:22:49.075328+0000","last_active":"2024-03-06T07:22:49.076124+0000","last_peered":"2024-03-06T07:22:49.076124+0000","last_clean":"2024-03-06T07:22:49.076124+0000","last_became_active":"2024-03-06T07:22:49.074938+0000","last_became_peered":"2024-03-06T07:22:49.074938+0000","last_unstale":"2024-03-06T07:22:49.076124+0000","last_undegraded":"2024-03-06T07:22:49.076124+0000","last_fullsized":"2024-03-06T07:22:49.076124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.090737+0000","last_change":"2024-03-06T07:22:48.062093+0000","last_active":"2024-03-06T07:22:49.090737+0000","last_peered":"2024-03-06T07:22:49.090737+0000","last_clean":"2024-03-06T07:22:49.090737+0000","last_became_active":"2024-03-06T07:22:48.061816+0000","last_became_peered":"2024-03-06T07:22:48.061816+0000","last_unstale":"2024-03-06T07:22:49.090737+0000","last_undegraded":"2024-03-06T07:22:49.090737+0000","last_fullsized":"2024-03-06T07:22:49.090737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.354486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087317+0000","last_change":"2024-03-06T07:22:49.081035+0000","last_active":"2024-03-06T07:22:49.087317+0000","last_peered":"2024-03-06T07:22:49.087317+0000","last_clean":"2024-03-06T07:22:49.087317+0000","last_became_active":"2024-03-06T07:22:49.080738+0000","last_became_peered":"2024-03-06T07:22:49.080738+0000","last_unstale":"2024-03-06T07:22:49.087317+0000","last_undegraded":"2024-03-06T07:22:49.087317+0000","last_fullsized":"2024-03-06T07:22:49.087317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.067613+0000","last_change":"2024-03-06T07:22:48.076377+0000","last_active":"2024-03-06T07:22:49.067613+0000","last_peered":"2024-03-06T07:22:49.067613+0000","last_clean":"2024-03-06T07:22:49.067613+0000","last_became_active":"2024-03-06T07:22:48.076062+0000","last_became_peered":"2024-03-06T07:22:48.076062+0000","last_unstale":"2024-03-06T07:22:49.067613+0000","last_undegraded":"2024-03-06T07:22:49.067613+0000","last_fullsized":"2024-03-06T07:22:49.067613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.083781+0000","last_change":"2024-03-06T07:22:49.081681+0000","last_active":"2024-03-06T07:22:49.083781+0000","last_peered":"2024-03-06T07:22:49.083781+0000","last_clean":"2024-03-06T07:22:49.083781+0000","last_became_active":"2024-03-06T07:22:49.081400+0000","last_became_peered":"2024-03-06T07:22:49.081400+0000","last_unstale":"2024-03-06T07:22:49.083781+0000","last_undegraded":"2024-03-06T07:22:49.083781+0000","last_fullsized":"2024-03-06T07:22:49.083781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:58.929359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.066207+0000","last_change":"2024-03-06T07:22:48.072725+0000","last_active":"2024-03-06T07:22:49.066207+0000","last_peered":"2024-03-06T07:22:49.066207+0000","last_clean":"2024-03-06T07:22:49.066207+0000","last_became_active":"2024-03-06T07:22:48.071793+0000","last_became_peered":"2024-03-06T07:22:48.071793+0000","last_unstale":"2024-03-06T07:22:49.066207+0000","last_undegraded":"2024-03-06T07:22:49.066207+0000","last_fullsized":"2024-03-06T07:22:49.066207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.089620+0000","last_change":"2024-03-06T07:22:49.088880+0000","last_active":"2024-03-06T07:22:49.089620+0000","last_peered":"2024-03-06T07:22:49.089620+0000","last_clean":"2024-03-06T07:22:49.089620+0000","last_became_active":"2024-03-06T07:22:49.088565+0000","last_became_peered":"2024-03-06T07:22:49.088565+0000","last_unstale":"2024-03-06T07:22:49.089620+0000","last_undegraded":"2024-03-06T07:22:49.089620+0000","last_fullsized":"2024-03-06T07:22:49.089620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086942+0000","last_change":"2024-03-06T07:22:48.067604+0000","last_active":"2024-03-06T07:22:49.086942+0000","last_peered":"2024-03-06T07:22:49.086942+0000","last_clean":"2024-03-06T07:22:49.086942+0000","last_became_active":"2024-03-06T07:22:48.067273+0000","last_became_peered":"2024-03-06T07:22:48.067273+0000","last_unstale":"2024-03-06T07:22:49.086942+0000","last_undegraded":"2024-03-06T07:22:49.086942+0000","last_fullsized":"2024-03-06T07:22:49.086942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082705+0000","last_change":"2024-03-06T07:22:49.080955+0000","last_active":"2024-03-06T07:22:49.082705+0000","last_peered":"2024-03-06T07:22:49.082705+0000","last_clean":"2024-03-06T07:22:49.082705+0000","last_became_active":"2024-03-06T07:22:49.080699+0000","last_became_peered":"2024-03-06T07:22:49.080699+0000","last_unstale":"2024-03-06T07:22:49.082705+0000","last_undegraded":"2024-03-06T07:22:49.082705+0000","last_fullsized":"2024-03-06T07:22:49.082705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084318+0000","last_change":"2024-03-06T07:22:48.071049+0000","last_active":"2024-03-06T07:22:49.084318+0000","last_peered":"2024-03-06T07:22:49.084318+0000","last_clean":"2024-03-06T07:22:49.084318+0000","last_became_active":"2024-03-06T07:22:48.070751+0000","last_became_peered":"2024-03-06T07:22:48.070751+0000","last_unstale":"2024-03-06T07:22:49.084318+0000","last_undegraded":"2024-03-06T07:22:49.084318+0000","last_fullsized":"2024-03-06T07:22:49.084318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:35.856218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.081936+0000","last_change":"2024-03-06T07:22:49.077479+0000","last_active":"2024-03-06T07:22:49.081936+0000","last_peered":"2024-03-06T07:22:49.081936+0000","last_clean":"2024-03-06T07:22:49.081936+0000","last_became_active":"2024-03-06T07:22:49.077188+0000","last_became_peered":"2024-03-06T07:22:49.077188+0000","last_unstale":"2024-03-06T07:22:49.081936+0000","last_undegraded":"2024-03-06T07:22:49.081936+0000","last_fullsized":"2024-03-06T07:22:49.081936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088852+0000","last_change":"2024-03-06T07:22:48.074777+0000","last_active":"2024-03-06T07:22:49.088852+0000","last_peered":"2024-03-06T07:22:49.088852+0000","last_clean":"2024-03-06T07:22:49.088852+0000","last_became_active":"2024-03-06T07:22:48.074441+0000","last_became_peered":"2024-03-06T07:22:48.074441+0000","last_unstale":"2024-03-06T07:22:49.088852+0000","last_undegraded":"2024-03-06T07:22:49.088852+0000","last_fullsized":"2024-03-06T07:22:49.088852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082719+0000","last_change":"2024-03-06T07:22:49.080000+0000","last_active":"2024-03-06T07:22:49.082719+0000","last_peered":"2024-03-06T07:22:49.082719+0000","last_clean":"2024-03-06T07:22:49.082719+0000","last_became_active":"2024-03-06T07:22:49.079471+0000","last_became_peered":"2024-03-06T07:22:49.079471+0000","last_unstale":"2024-03-06T07:22:49.082719+0000","last_undegraded":"2024-03-06T07:22:49.082719+0000","last_fullsized":"2024-03-06T07:22:49.082719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.085863+0000","last_change":"2024-03-06T07:22:48.067652+0000","last_active":"2024-03-06T07:22:49.085863+0000","last_peered":"2024-03-06T07:22:49.085863+0000","last_clean":"2024-03-06T07:22:49.085863+0000","last_became_active":"2024-03-06T07:22:48.067390+0000","last_became_peered":"2024-03-06T07:22:48.067390+0000","last_unstale":"2024-03-06T07:22:49.085863+0000","last_undegraded":"2024-03-06T07:22:49.085863+0000","last_fullsized":"2024-03-06T07:22:49.085863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.085967+0000","last_change":"2024-03-06T07:22:49.084081+0000","last_active":"2024-03-06T07:22:49.085967+0000","last_peered":"2024-03-06T07:22:49.085967+0000","last_clean":"2024-03-06T07:22:49.085967+0000","last_became_active":"2024-03-06T07:22:49.083758+0000","last_became_peered":"2024-03-06T07:22:49.083758+0000","last_unstale":"2024-03-06T07:22:49.085967+0000","last_undegraded":"2024-03-06T07:22:49.085967+0000","last_fullsized":"2024-03-06T07:22:49.085967+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.067389+0000","last_change":"2024-03-06T07:22:48.072197+0000","last_active":"2024-03-06T07:22:49.067389+0000","last_peered":"2024-03-06T07:22:49.067389+0000","last_clean":"2024-03-06T07:22:49.067389+0000","last_became_active":"2024-03-06T07:22:48.071904+0000","last_became_peered":"2024-03-06T07:22:48.071904+0000","last_unstale":"2024-03-06T07:22:49.067389+0000","last_undegraded":"2024-03-06T07:22:49.067389+0000","last_fullsized":"2024-03-06T07:22:49.067389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076103+0000","last_change":"2024-03-06T07:22:49.074585+0000","last_active":"2024-03-06T07:22:49.076103+0000","last_peered":"2024-03-06T07:22:49.076103+0000","last_clean":"2024-03-06T07:22:49.076103+0000","last_became_active":"2024-03-06T07:22:49.074307+0000","last_became_peered":"2024-03-06T07:22:49.074307+0000","last_unstale":"2024-03-06T07:22:49.076103+0000","last_undegraded":"2024-03-06T07:22:49.076103+0000","last_fullsized":"2024-03-06T07:22:49.076103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065334+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:22:49.065334+0000","last_peered":"2024-03-06T07:22:49.065334+0000","last_clean":"2024-03-06T07:22:49.065334+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:22:49.065334+0000","last_undegraded":"2024-03-06T07:22:49.065334+0000","last_fullsized":"2024-03-06T07:22:49.065334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.066085+0000","last_change":"2024-03-06T07:22:48.073977+0000","last_active":"2024-03-06T07:22:49.066085+0000","last_peered":"2024-03-06T07:22:49.066085+0000","last_clean":"2024-03-06T07:22:49.066085+0000","last_became_active":"2024-03-06T07:22:48.073656+0000","last_became_peered":"2024-03-06T07:22:48.073656+0000","last_unstale":"2024-03-06T07:22:49.066085+0000","last_undegraded":"2024-03-06T07:22:49.066085+0000","last_fullsized":"2024-03-06T07:22:49.066085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084355+0000","last_change":"2024-03-06T07:22:49.074128+0000","last_active":"2024-03-06T07:22:49.084355+0000","last_peered":"2024-03-06T07:22:49.084355+0000","last_clean":"2024-03-06T07:22:49.084355+0000","last_became_active":"2024-03-06T07:22:49.073858+0000","last_became_peered":"2024-03-06T07:22:49.073858+0000","last_unstale":"2024-03-06T07:22:49.084355+0000","last_undegraded":"2024-03-06T07:22:49.084355+0000","last_fullsized":"2024-03-06T07:22:49.084355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065286+0000","last_change":"2024-03-06T07:22:48.069166+0000","last_active":"2024-03-06T07:22:49.065286+0000","last_peered":"2024-03-06T07:22:49.065286+0000","last_clean":"2024-03-06T07:22:49.065286+0000","last_became_active":"2024-03-06T07:22:48.068441+0000","last_became_peered":"2024-03-06T07:22:48.068441+0000","last_unstale":"2024-03-06T07:22:49.065286+0000","last_undegraded":"2024-03-06T07:22:49.065286+0000","last_fullsized":"2024-03-06T07:22:49.065286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:45.086584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078528+0000","last_change":"2024-03-06T07:22:49.075228+0000","last_active":"2024-03-06T07:22:49.078528+0000","last_peered":"2024-03-06T07:22:49.078528+0000","last_clean":"2024-03-06T07:22:49.078528+0000","last_became_active":"2024-03-06T07:22:49.074906+0000","last_became_peered":"2024-03-06T07:22:49.074906+0000","last_unstale":"2024-03-06T07:22:49.078528+0000","last_undegraded":"2024-03-06T07:22:49.078528+0000","last_fullsized":"2024-03-06T07:22:49.078528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:53.011841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.069758+0000","last_change":"2024-03-06T07:22:48.074088+0000","last_active":"2024-03-06T07:22:49.069758+0000","last_peered":"2024-03-06T07:22:49.069758+0000","last_clean":"2024-03-06T07:22:49.069758+0000","last_became_active":"2024-03-06T07:22:48.073794+0000","last_became_peered":"2024-03-06T07:22:48.073794+0000","last_unstale":"2024-03-06T07:22:49.069758+0000","last_undegraded":"2024-03-06T07:22:49.069758+0000","last_fullsized":"2024-03-06T07:22:49.069758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.073127+0000","last_change":"2024-03-06T07:22:49.071324+0000","last_active":"2024-03-06T07:22:49.073127+0000","last_peered":"2024-03-06T07:22:49.073127+0000","last_clean":"2024-03-06T07:22:49.073127+0000","last_became_active":"2024-03-06T07:22:49.071043+0000","last_became_peered":"2024-03-06T07:22:49.071043+0000","last_unstale":"2024-03-06T07:22:49.073127+0000","last_undegraded":"2024-03-06T07:22:49.073127+0000","last_fullsized":"2024-03-06T07:22:49.073127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.068197+0000","last_change":"2024-03-06T07:22:48.071307+0000","last_active":"2024-03-06T07:22:49.068197+0000","last_peered":"2024-03-06T07:22:49.068197+0000","last_clean":"2024-03-06T07:22:49.068197+0000","last_became_active":"2024-03-06T07:22:48.070797+0000","last_became_peered":"2024-03-06T07:22:48.070797+0000","last_unstale":"2024-03-06T07:22:49.068197+0000","last_undegraded":"2024-03-06T07:22:49.068197+0000","last_fullsized":"2024-03-06T07:22:49.068197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079479+0000","last_change":"2024-03-06T07:22:49.077714+0000","last_active":"2024-03-06T07:22:49.079479+0000","last_peered":"2024-03-06T07:22:49.079479+0000","last_clean":"2024-03-06T07:22:49.079479+0000","last_became_active":"2024-03-06T07:22:49.077442+0000","last_became_peered":"2024-03-06T07:22:49.077442+0000","last_unstale":"2024-03-06T07:22:49.079479+0000","last_undegraded":"2024-03-06T07:22:49.079479+0000","last_fullsized":"2024-03-06T07:22:49.079479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:45.617755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.064402+0000","last_change":"2024-03-06T07:22:48.071210+0000","last_active":"2024-03-06T07:22:49.064402+0000","last_peered":"2024-03-06T07:22:49.064402+0000","last_clean":"2024-03-06T07:22:49.064402+0000","last_became_active":"2024-03-06T07:22:48.070893+0000","last_became_peered":"2024-03-06T07:22:48.070893+0000","last_unstale":"2024-03-06T07:22:49.064402+0000","last_undegraded":"2024-03-06T07:22:49.064402+0000","last_fullsized":"2024-03-06T07:22:49.064402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088822+0000","last_change":"2024-03-06T07:22:49.085398+0000","last_active":"2024-03-06T07:22:49.088822+0000","last_peered":"2024-03-06T07:22:49.088822+0000","last_clean":"2024-03-06T07:22:49.088822+0000","last_became_active":"2024-03-06T07:22:49.085124+0000","last_became_peered":"2024-03-06T07:22:49.085124+0000","last_unstale":"2024-03-06T07:22:49.088822+0000","last_undegraded":"2024-03-06T07:22:49.088822+0000","last_fullsized":"2024-03-06T07:22:49.088822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.066623+0000","last_change":"2024-03-06T07:22:48.071472+0000","last_active":"2024-03-06T07:22:49.066623+0000","last_peered":"2024-03-06T07:22:49.066623+0000","last_clean":"2024-03-06T07:22:49.066623+0000","last_became_active":"2024-03-06T07:22:48.071136+0000","last_became_peered":"2024-03-06T07:22:48.071136+0000","last_unstale":"2024-03-06T07:22:49.066623+0000","last_undegraded":"2024-03-06T07:22:49.066623+0000","last_fullsized":"2024-03-06T07:22:49.066623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:30.767002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.076564+0000","last_change":"2024-03-06T07:22:49.072607+0000","last_active":"2024-03-06T07:22:49.076564+0000","last_peered":"2024-03-06T07:22:49.076564+0000","last_clean":"2024-03-06T07:22:49.076564+0000","last_became_active":"2024-03-06T07:22:49.071896+0000","last_became_peered":"2024-03-06T07:22:49.071896+0000","last_unstale":"2024-03-06T07:22:49.076564+0000","last_undegraded":"2024-03-06T07:22:49.076564+0000","last_fullsized":"2024-03-06T07:22:49.076564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065042+0000","last_change":"2024-03-06T07:22:48.071003+0000","last_active":"2024-03-06T07:22:49.065042+0000","last_peered":"2024-03-06T07:22:49.065042+0000","last_clean":"2024-03-06T07:22:49.065042+0000","last_became_active":"2024-03-06T07:22:48.070666+0000","last_became_peered":"2024-03-06T07:22:48.070666+0000","last_unstale":"2024-03-06T07:22:49.065042+0000","last_undegraded":"2024-03-06T07:22:49.065042+0000","last_fullsized":"2024-03-06T07:22:49.065042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086916+0000","last_change":"2024-03-06T07:22:49.077332+0000","last_active":"2024-03-06T07:22:49.086916+0000","last_peered":"2024-03-06T07:22:49.086916+0000","last_clean":"2024-03-06T07:22:49.086916+0000","last_became_active":"2024-03-06T07:22:49.077034+0000","last_became_peered":"2024-03-06T07:22:49.077034+0000","last_unstale":"2024-03-06T07:22:49.086916+0000","last_undegraded":"2024-03-06T07:22:49.086916+0000","last_fullsized":"2024-03-06T07:22:49.086916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084620+0000","last_change":"2024-03-06T07:22:48.076685+0000","last_active":"2024-03-06T07:22:49.084620+0000","last_peered":"2024-03-06T07:22:49.084620+0000","last_clean":"2024-03-06T07:22:49.084620+0000","last_became_active":"2024-03-06T07:22:48.076412+0000","last_became_peered":"2024-03-06T07:22:48.076412+0000","last_unstale":"2024-03-06T07:22:49.084620+0000","last_undegraded":"2024-03-06T07:22:49.084620+0000","last_fullsized":"2024-03-06T07:22:49.084620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078857+0000","last_change":"2024-03-06T07:22:49.069020+0000","last_active":"2024-03-06T07:22:49.078857+0000","last_peered":"2024-03-06T07:22:49.078857+0000","last_clean":"2024-03-06T07:22:49.078857+0000","last_became_active":"2024-03-06T07:22:49.068741+0000","last_became_peered":"2024-03-06T07:22:49.068741+0000","last_unstale":"2024-03-06T07:22:49.078857+0000","last_undegraded":"2024-03-06T07:22:49.078857+0000","last_fullsized":"2024-03-06T07:22:49.078857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:45.617755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065154+0000","last_change":"2024-03-06T07:22:48.072458+0000","last_active":"2024-03-06T07:22:49.065154+0000","last_peered":"2024-03-06T07:22:49.065154+0000","last_clean":"2024-03-06T07:22:49.065154+0000","last_became_active":"2024-03-06T07:22:48.071279+0000","last_became_peered":"2024-03-06T07:22:48.071279+0000","last_unstale":"2024-03-06T07:22:49.065154+0000","last_undegraded":"2024-03-06T07:22:49.065154+0000","last_fullsized":"2024-03-06T07:22:49.065154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079512+0000","last_change":"2024-03-06T07:22:49.077623+0000","last_active":"2024-03-06T07:22:49.079512+0000","last_peered":"2024-03-06T07:22:49.079512+0000","last_clean":"2024-03-06T07:22:49.079512+0000","last_became_active":"2024-03-06T07:22:49.076719+0000","last_became_peered":"2024-03-06T07:22:49.076719+0000","last_unstale":"2024-03-06T07:22:49.079512+0000","last_undegraded":"2024-03-06T07:22:49.079512+0000","last_fullsized":"2024-03-06T07:22:49.079512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:49.124109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.064860+0000","last_change":"2024-03-06T07:22:48.071228+0000","last_active":"2024-03-06T07:22:49.064860+0000","last_peered":"2024-03-06T07:22:49.064860+0000","last_clean":"2024-03-06T07:22:49.064860+0000","last_became_active":"2024-03-06T07:22:48.070898+0000","last_became_peered":"2024-03-06T07:22:48.070898+0000","last_unstale":"2024-03-06T07:22:49.064860+0000","last_undegraded":"2024-03-06T07:22:49.064860+0000","last_fullsized":"2024-03-06T07:22:49.064860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.084272+0000","last_change":"2024-03-06T07:22:49.069977+0000","last_active":"2024-03-06T07:22:49.084272+0000","last_peered":"2024-03-06T07:22:49.084272+0000","last_clean":"2024-03-06T07:22:49.084272+0000","last_became_active":"2024-03-06T07:22:49.069691+0000","last_became_peered":"2024-03-06T07:22:49.069691+0000","last_unstale":"2024-03-06T07:22:49.084272+0000","last_undegraded":"2024-03-06T07:22:49.084272+0000","last_fullsized":"2024-03-06T07:22:49.084272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:53.011841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.065247+0000","last_change":"2024-03-06T07:22:48.074693+0000","last_active":"2024-03-06T07:22:49.065247+0000","last_peered":"2024-03-06T07:22:49.065247+0000","last_clean":"2024-03-06T07:22:49.065247+0000","last_became_active":"2024-03-06T07:22:48.074261+0000","last_became_peered":"2024-03-06T07:22:48.074261+0000","last_unstale":"2024-03-06T07:22:49.065247+0000","last_undegraded":"2024-03-06T07:22:49.065247+0000","last_fullsized":"2024-03-06T07:22:49.065247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087034+0000","last_change":"2024-03-06T07:22:49.082024+0000","last_active":"2024-03-06T07:22:49.087034+0000","last_peered":"2024-03-06T07:22:49.087034+0000","last_clean":"2024-03-06T07:22:49.087034+0000","last_became_active":"2024-03-06T07:22:49.081534+0000","last_became_peered":"2024-03-06T07:22:49.081534+0000","last_unstale":"2024-03-06T07:22:49.087034+0000","last_undegraded":"2024-03-06T07:22:49.087034+0000","last_fullsized":"2024-03-06T07:22:49.087034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087583+0000","last_change":"2024-03-06T07:22:48.069179+0000","last_active":"2024-03-06T07:22:49.087583+0000","last_peered":"2024-03-06T07:22:49.087583+0000","last_clean":"2024-03-06T07:22:49.087583+0000","last_became_active":"2024-03-06T07:22:48.068402+0000","last_became_peered":"2024-03-06T07:22:48.068402+0000","last_unstale":"2024-03-06T07:22:49.087583+0000","last_undegraded":"2024-03-06T07:22:49.087583+0000","last_fullsized":"2024-03-06T07:22:49.087583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:09.655063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.086941+0000","last_change":"2024-03-06T07:22:49.082657+0000","last_active":"2024-03-06T07:22:49.086941+0000","last_peered":"2024-03-06T07:22:49.086941+0000","last_clean":"2024-03-06T07:22:49.086941+0000","last_became_active":"2024-03-06T07:22:49.082096+0000","last_became_peered":"2024-03-06T07:22:49.082096+0000","last_unstale":"2024-03-06T07:22:49.086941+0000","last_undegraded":"2024-03-06T07:22:49.086941+0000","last_fullsized":"2024-03-06T07:22:49.086941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.079643+0000","last_change":"2024-03-06T07:22:49.073629+0000","last_active":"2024-03-06T07:22:49.079643+0000","last_peered":"2024-03-06T07:22:49.079643+0000","last_clean":"2024-03-06T07:22:49.079643+0000","last_became_active":"2024-03-06T07:22:49.072839+0000","last_became_peered":"2024-03-06T07:22:49.072839+0000","last_unstale":"2024-03-06T07:22:49.079643+0000","last_undegraded":"2024-03-06T07:22:49.079643+0000","last_fullsized":"2024-03-06T07:22:49.079643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.085029+0000","last_change":"2024-03-06T07:22:48.072083+0000","last_active":"2024-03-06T07:22:49.085029+0000","last_peered":"2024-03-06T07:22:49.085029+0000","last_clean":"2024-03-06T07:22:49.085029+0000","last_became_active":"2024-03-06T07:22:48.071560+0000","last_became_peered":"2024-03-06T07:22:48.071560+0000","last_unstale":"2024-03-06T07:22:49.085029+0000","last_undegraded":"2024-03-06T07:22:49.085029+0000","last_fullsized":"2024-03-06T07:22:49.085029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.082434+0000","last_change":"2024-03-06T07:22:49.080775+0000","last_active":"2024-03-06T07:22:49.082434+0000","last_peered":"2024-03-06T07:22:49.082434+0000","last_clean":"2024-03-06T07:22:49.082434+0000","last_became_active":"2024-03-06T07:22:49.080487+0000","last_became_peered":"2024-03-06T07:22:49.080487+0000","last_unstale":"2024-03-06T07:22:49.082434+0000","last_undegraded":"2024-03-06T07:22:49.082434+0000","last_fullsized":"2024-03-06T07:22:49.082434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.087060+0000","last_change":"2024-03-06T07:22:48.074550+0000","last_active":"2024-03-06T07:22:49.087060+0000","last_peered":"2024-03-06T07:22:49.087060+0000","last_clean":"2024-03-06T07:22:49.087060+0000","last_became_active":"2024-03-06T07:22:48.073981+0000","last_became_peered":"2024-03-06T07:22:48.073981+0000","last_unstale":"2024-03-06T07:22:49.087060+0000","last_undegraded":"2024-03-06T07:22:49.087060+0000","last_fullsized":"2024-03-06T07:22:49.087060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:13.210796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.078782+0000","last_change":"2024-03-06T07:22:49.075565+0000","last_active":"2024-03-06T07:22:49.078782+0000","last_peered":"2024-03-06T07:22:49.078782+0000","last_clean":"2024-03-06T07:22:49.078782+0000","last_became_active":"2024-03-06T07:22:49.075254+0000","last_became_peered":"2024-03-06T07:22:49.075254+0000","last_unstale":"2024-03-06T07:22:49.078782+0000","last_undegraded":"2024-03-06T07:22:49.078782+0000","last_fullsized":"2024-03-06T07:22:49.078782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:46.459505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088279+0000","last_change":"2024-03-06T07:22:48.073512+0000","last_active":"2024-03-06T07:22:49.088279+0000","last_peered":"2024-03-06T07:22:49.088279+0000","last_clean":"2024-03-06T07:22:49.088279+0000","last_became_active":"2024-03-06T07:22:48.073181+0000","last_became_peered":"2024-03-06T07:22:48.073181+0000","last_unstale":"2024-03-06T07:22:49.088279+0000","last_undegraded":"2024-03-06T07:22:49.088279+0000","last_fullsized":"2024-03-06T07:22:49.088279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.088433+0000","last_change":"2024-03-06T07:22:49.079451+0000","last_active":"2024-03-06T07:22:49.088433+0000","last_peered":"2024-03-06T07:22:49.088433+0000","last_clean":"2024-03-06T07:22:49.088433+0000","last_became_active":"2024-03-06T07:22:49.078927+0000","last_became_peered":"2024-03-06T07:22:49.078927+0000","last_unstale":"2024-03-06T07:22:49.088433+0000","last_undegraded":"2024-03-06T07:22:49.088433+0000","last_fullsized":"2024-03-06T07:22:49.088433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:22:49.090890+0000","last_change":"2024-03-06T07:22:48.062906+0000","last_active":"2024-03-06T07:22:49.090890+0000","last_peered":"2024-03-06T07:22:49.090890+0000","last_clean":"2024-03-06T07:22:49.090890+0000","last_became_active":"2024-03-06T07:22:48.062497+0000","last_became_peered":"2024-03-06T07:22:48.062497+0000","last_unstale":"2024-03-06T07:22:49.090890+0000","last_undegraded":"2024-03-06T07:22:49.090890+0000","last_fullsized":"2024-03-06T07:22:49.090890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:51.443747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":660255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":69397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":69465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":660311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:53.458 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:53.458 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:22:53.459 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:22:53.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:22:53.459 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:22:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.599+0000 7f39ce9d7640 1 Processor -- start 2024-03-06T07:22:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.599+0000 7f39ce9d7640 1 -- start start 2024-03-06T07:22:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.600+0000 7f39ce9d7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39c815e080 0x7f39c8066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.600+0000 7f39ce9d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80b00c0 0x7f39c8067240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.600+0000 7f39ce9d7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8161c20 con 0x7f39c815dca0 2024-03-06T07:22:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.600+0000 7f39ce9d7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c81619d0 con 0x7f39c815e080 2024-03-06T07:22:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.600+0000 7f39ce9d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c8161f00 con 0x7f39c80b00c0 2024-03-06T07:22:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.600+0000 7f39c7fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f39c815dca0 0x7f39c806fd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49984/0 (socket says 172.21.15.27:49984) 2024-03-06T07:22:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.600+0000 7f39c7fff640 1 -- 172.21.15.27:0/662648892 learned_addr learned my addr 172.21.15.27:0/662648892 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.600+0000 7f39c77fe640 1 --2- 172.21.15.27:0/662648892 >> v2:172.21.15.69:3300/0 conn(0x7f39c815e080 0x7f39c8066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.600+0000 7f39ccf74640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80b00c0 0x7f39c8067240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120612799 0 0) 0x7f39c8161c20 con 0x7f39c815dca0 2024-03-06T07:22:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b0003520 con 0x7f39c815dca0 2024-03-06T07:22:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2377846510 0 0) 0x7f39b0003520 con 0x7f39c815dca0 2024-03-06T07:22:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80b00c0 msgr2=0x7f39c8067240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80b00c0 0x7f39c8067240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 >> v2:172.21.15.69:3300/0 conn(0x7f39c815e080 msgr2=0x7f39c8066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 --2- 172.21.15.27:0/662648892 >> v2:172.21.15.69:3300/0 conn(0x7f39c815e080 0x7f39c8066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c81621e0 con 0x7f39c815dca0 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f39bc001670 con 0x7f39c815dca0 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c77fe640 1 --2- 172.21.15.27:0/662648892 >> v2:172.21.15.69:3300/0 conn(0x7f39c815e080 0x7f39c8066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 >> v1:172.21.15.27:6789/0 conn(0x7f39c815dca0 legacy=0x7f39c806fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b00039b0 0x7f39b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80b00c0 0x7f39b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c81619d0 con 0x7f39b0004800 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c8161f00 con 0x7f39b00039b0 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b000b9c0 con 0x7f39c80b00c0 2024-03-06T07:22:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c77fe640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39ccf74640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80b00c0 0x7f39b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c7fff640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b00039b0 0x7f39b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c77fe640 1 -- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80b00c0 msgr2=0x7f39b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c77fe640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80b00c0 0x7f39b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c77fe640 1 -- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b00039b0 msgr2=0x7f39b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c77fe640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b00039b0 0x7f39b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.601+0000 7f39c77fe640 1 -- 172.21.15.27:0/662648892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b0003520 con 0x7f39b0004800 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.602+0000 7f39c77fe640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39b0004c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009cc0 tx=0x7f39b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7bab6eb07b20f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.602+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b8039070 con 0x7f39b0004800 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.602+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b802c960 con 0x7f39b0004800 2024-03-06T07:22:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.602+0000 7f39c6ffd640 1 -- 172.21.15.27:0/662648892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b8009860 con 0x7f39b0004800 2024-03-06T07:22:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.603+0000 7f39ce9d7640 1 -- 172.21.15.27:0/662648892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 msgr2=0x7f39b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.603+0000 7f39ce9d7640 1 --2- 172.21.15.27:0/662648892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39b0004c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009cc0 tx=0x7f39b0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.603+0000 7f39ccf74640 1 -- 172.21.15.27:0/662648892 reap_dead start 2024-03-06T07:22:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.603+0000 7f39ce9d7640 1 -- 172.21.15.27:0/662648892 shutdown_connections 2024-03-06T07:22:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.603+0000 7f39ce9d7640 1 -- 172.21.15.27:0/662648892 >> 172.21.15.27:0/662648892 conn(0x7f39c80d0990 msgr2=0x7f39c80d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.603+0000 7f39ce9d7640 1 -- 172.21.15.27:0/662648892 shutdown_connections 2024-03-06T07:22:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.603+0000 7f39ce9d7640 1 -- 172.21.15.27:0/662648892 wait complete. 2024-03-06T07:22:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.604+0000 7f39ce9d7640 1 Processor -- start 2024-03-06T07:22:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.604+0000 7f39ce9d7640 1 -- start start 2024-03-06T07:22:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39ce9d7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39c8160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39ce9d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c80ce620 0x7f39c80ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39ce9d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80cf0b0 0x7f39c80cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39ce9d7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c80673c0 con 0x7f39b0004800 2024-03-06T07:22:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39ce9d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c8067240 con 0x7f39c80ce620 2024-03-06T07:22:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39ce9d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c8067540 con 0x7f39c80cf0b0 2024-03-06T07:22:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39c7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39c8160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39c7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39c8160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51354/0 (socket says 172.21.15.27:51354) 2024-03-06T07:22:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39c7fff640 1 -- 172.21.15.27:0/1592131239 learned_addr learned my addr 172.21.15.27:0/1592131239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39c77fe640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c80ce620 0x7f39c80ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.605+0000 7f39ccf74640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80cf0b0 0x7f39c80cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39c7fff640 1 -- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80cf0b0 msgr2=0x7f39c80cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39c7fff640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80cf0b0 0x7f39c80cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39c7fff640 1 -- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c80ce620 msgr2=0x7f39c80ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39c7fff640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c80ce620 0x7f39c80ae760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39c7fff640 1 -- 172.21.15.27:0/1592131239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c81f3b40 con 0x7f39b0004800 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39c7fff640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39c8160cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f39bc007340 tx=0x7f39bc007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39c4ff9640 1 -- 172.21.15.27:0/1592131239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc01a070 con 0x7f39b0004800 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39c4ff9640 1 -- 172.21.15.27:0/1592131239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc00f860 con 0x7f39b0004800 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c81f3e20 con 0x7f39b0004800 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39c4ff9640 1 -- 172.21.15.27:0/1592131239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00fc30 con 0x7f39b0004800 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.606+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c81f43e0 con 0x7f39b0004800 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.607+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c8068700 con 0x7f39b0004800 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.607+0000 7f39c4ff9640 1 -- 172.21.15.27:0/1592131239 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39bc01d070 con 0x7f39b0004800 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.608+0000 7f39c4ff9640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b4d0 0x7f39b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.608+0000 7f39c4ff9640 1 -- 172.21.15.27:0/1592131239 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39bc089a90 con 0x7f39b0004800 2024-03-06T07:22:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.608+0000 7f39c77fe640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b4d0 0x7f39b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.609+0000 7f39c77fe640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b4d0 0x7f39b406d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f39b8036000 tx=0x7f39b8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.612+0000 7f39c4ff9640 1 -- 172.21.15.27:0/1592131239 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39bc0582c0 con 0x7f39b0004800 2024-03-06T07:22:53.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.869+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f39c815d720 con 0x7f39b0004800 2024-03-06T07:22:53.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.872+0000 7f39c4ff9640 1 -- 172.21.15.27:0/1592131239 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171785 (secure 0 0 0) 0x7f39bc05c360 con 0x7f39b0004800 2024-03-06T07:22:53.877 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b4d0 msgr2=0x7f39b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b4d0 0x7f39b406d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f39b8036000 tx=0x7f39b8030000 comp rx=0 tx=0).stop 2024-03-06T07:22:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 msgr2=0x7f39c8160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39c8160cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f39bc007340 tx=0x7f39bc007800 comp rx=0 tx=0).stop 2024-03-06T07:22:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 shutdown_connections 2024-03-06T07:22:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b4d0 0x7f39b406d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39c80cf0b0 0x7f39c80cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c80ce620 0x7f39c80ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 --2- 172.21.15.27:0/1592131239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b0004800 0x7f39c8160cb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.875+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 >> 172.21.15.27:0/1592131239 conn(0x7f39c80d0990 msgr2=0x7f39c8072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.876+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 shutdown_connections 2024-03-06T07:22:53.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.876+0000 7f39ce9d7640 1 -- 172.21.15.27:0/1592131239 wait complete. 2024-03-06T07:22:53.889 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2286301988},{"type":"v1","addr":"172.21.15.69:6833","nonce":2286301988}]},"active_addr":"172.21.15.69:6833/2286301988","active_change":"2024-03-06T07:22:27.975432+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":4071656377}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":2084020326}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3799252030}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3664094869}]}]} 2024-03-06T07:22:53.892 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:22:53.892 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:22:53.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.997+0000 7f1544295640 1 Processor -- start 2024-03-06T07:22:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.997+0000 7f1544295640 1 -- start start 2024-03-06T07:22:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1544295640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f153c0b0e90 0x7f153c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1544295640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1544295640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0cee90 con 0x7f153c06b5c0 2024-03-06T07:22:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1544295640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c0cfa00 con 0x7f153c0b0e90 2024-03-06T07:22:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1544295640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0cfc50 con 0x7f153c0ae9b0 2024-03-06T07:22:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1541830640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f153c0b0e90 0x7f153c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1541830640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f153c0b0e90 0x7f153c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51368/0 (socket says 172.21.15.27:51368) 2024-03-06T07:22:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1541830640 1 -- 172.21.15.27:0/3691289497 learned_addr learned my addr 172.21.15.27:0/3691289497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1542832640 1 --2- 172.21.15.27:0/3691289497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1541830640 1 -- 172.21.15.27:0/3691289497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 msgr2=0x7f153c0d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1541830640 1 --2- 172.21.15.27:0/3691289497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c0d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1541830640 1 -- 172.21.15.27:0/3691289497 >> v1:172.21.15.69:6789/0 conn(0x7f153c06b5c0 legacy=0x7f153c0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1541830640 1 -- 172.21.15.27:0/3691289497 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0cff30 con 0x7f153c0b0e90 2024-03-06T07:22:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.998+0000 7f1542832640 1 --2- 172.21.15.27:0/3691289497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c0d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.999+0000 7f1541830640 1 --2- 172.21.15.27:0/3691289497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0b0e90 0x7f153c06b9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f152c009870 tx=0x7f152c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44552c7c56639c88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:53.999+0000 7f154102f640 1 -- 172.21.15.27:0/3691289497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c0317c0 con 0x7f153c0b0e90 2024-03-06T07:22:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f154102f640 1 -- 172.21.15.27:0/3691289497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c037070 con 0x7f153c0b0e90 2024-03-06T07:22:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f154102f640 1 -- 172.21.15.27:0/3691289497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c031b80 con 0x7f153c0b0e90 2024-03-06T07:22:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f1544295640 1 -- 172.21.15.27:0/3691289497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0b0e90 msgr2=0x7f153c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f1544295640 1 --2- 172.21.15.27:0/3691289497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0b0e90 0x7f153c06b9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f152c009870 tx=0x7f152c02b860 comp rx=0 tx=0).stop 2024-03-06T07:22:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f1544295640 1 -- 172.21.15.27:0/3691289497 shutdown_connections 2024-03-06T07:22:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f1544295640 1 --2- 172.21.15.27:0/3691289497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0b0e90 0x7f153c06b9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f1544295640 1 --2- 172.21.15.27:0/3691289497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c0d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f1544295640 1 -- 172.21.15.27:0/3691289497 >> 172.21.15.27:0/3691289497 conn(0x7f153c0b36d0 msgr2=0x7f153c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f1544295640 1 -- 172.21.15.27:0/3691289497 shutdown_connections 2024-03-06T07:22:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.000+0000 7f1544295640 1 -- 172.21.15.27:0/3691289497 wait complete. 2024-03-06T07:22:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.002+0000 7f1544295640 1 Processor -- start 2024-03-06T07:22:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.002+0000 7f1544295640 1 -- start start 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.003+0000 7f1544295640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c06b5c0 0x7f153c0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.003+0000 7f1544295640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c1daf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.003+0000 7f1544295640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b0e90 0x7f153c1e7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.003+0000 7f1544295640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c06c910 con 0x7f153c06b5c0 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.003+0000 7f1544295640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c06c790 con 0x7f153c0b0e90 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.003+0000 7f1544295640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0aee00 con 0x7f153c0ae9b0 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.003+0000 7f1542031640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c06b5c0 0x7f153c0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542031640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c06b5c0 0x7f153c0c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51382/0 (socket says 172.21.15.27:51382) 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542031640 1 -- 172.21.15.27:0/1447783569 learned_addr learned my addr 172.21.15.27:0/1447783569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542832640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b0e90 0x7f153c1e7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1541830640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c1daf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542031640 1 -- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 msgr2=0x7f153c1daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542031640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c1daf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542031640 1 -- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b0e90 msgr2=0x7f153c1e7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542031640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b0e90 0x7f153c1e7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542031640 1 -- 172.21.15.27:0/1447783569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0c7ff0 con 0x7f153c06b5c0 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542832640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b0e90 0x7f153c1e7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1541830640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c1daf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1542031640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c06b5c0 0x7f153c0c8bd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f153800c430 tx=0x7f153800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1532ffd640 1 -- 172.21.15.27:0/1447783569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1538011790 con 0x7f153c06b5c0 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1532ffd640 1 -- 172.21.15.27:0/1447783569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1538017070 con 0x7f153c06b5c0 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1532ffd640 1 -- 172.21.15.27:0/1447783569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1538011b70 con 0x7f153c06b5c0 2024-03-06T07:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c0c8290 con 0x7f153c06b5c0 2024-03-06T07:22:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.004+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c1e7670 con 0x7f153c06b5c0 2024-03-06T07:22:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.005+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c1e71b0 con 0x7f153c06b5c0 2024-03-06T07:22:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.006+0000 7f1532ffd640 1 -- 172.21.15.27:0/1447783569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1538011d10 con 0x7f153c06b5c0 2024-03-06T07:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.007+0000 7f1532ffd640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.007+0000 7f1532ffd640 1 -- 172.21.15.27:0/1447783569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1538089fe0 con 0x7f153c06b5c0 2024-03-06T07:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.007+0000 7f1541830640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.007+0000 7f1541830640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f152c036000 tx=0x7f152c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.011+0000 7f1532ffd640 1 -- 172.21.15.27:0/1447783569 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f153c1e71b0 con 0x7f153c06b5c0 2024-03-06T07:22:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.211+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f153c068700 con 0x7f153c06b5c0 2024-03-06T07:22:54.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.212+0000 7f1532ffd640 1 -- 172.21.15.27:0/1447783569 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f1538058ef0 con 0x7f153c06b5c0 2024-03-06T07:22:54.214 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:54.215 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:54.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 msgr2=0x7f152006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f152c036000 tx=0x7f152c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c06b5c0 msgr2=0x7f153c0c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c06b5c0 0x7f153c0c8bd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f153800c430 tx=0x7f153800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 shutdown_connections 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b0e90 0x7f153c1e7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c0ae9b0 0x7f153c1daf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 --2- 172.21.15.27:0/1447783569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c06b5c0 0x7f153c0c8bd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 >> 172.21.15.27:0/1447783569 conn(0x7f153c0b36d0 msgr2=0x7f153c1dc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 shutdown_connections 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.214+0000 7f1544295640 1 -- 172.21.15.27:0/1447783569 wait complete. 2024-03-06T07:22:54.228 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:22:54.228 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.332+0000 7f8d316af640 1 Processor -- start 2024-03-06T07:22:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.332+0000 7f8d316af640 1 -- start start 2024-03-06T07:22:54.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.332+0000 7f8d316af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c06fe20 0x7f8d2c0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.332+0000 7f8d316af640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d2c0d0880 0x7f8d2c0ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.332+0000 7f8d316af640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c0d0dd0 con 0x7f8d2c15e060 2024-03-06T07:22:54.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.332+0000 7f8d316af640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2c0d10b0 con 0x7f8d2c0d0880 2024-03-06T07:22:54.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.332+0000 7f8d316af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0cc0e0 con 0x7f8d2c06fe20 2024-03-06T07:22:54.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.333+0000 7f8d2b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d2c0d0880 0x7f8d2c0ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.333+0000 7f8d2b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d2c0d0880 0x7f8d2c0ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51390/0 (socket says 172.21.15.27:51390) 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.333+0000 7f8d2b7fe640 1 -- 172.21.15.27:0/267050022 learned_addr learned my addr 172.21.15.27:0/267050022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.333+0000 7f8d2a7fc640 1 --2- 172.21.15.27:0/267050022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c06fe20 0x7f8d2c0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.333+0000 7f8d2b7fe640 1 -- 172.21.15.27:0/267050022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c06fe20 msgr2=0x7f8d2c0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.333+0000 7f8d2b7fe640 1 --2- 172.21.15.27:0/267050022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c06fe20 0x7f8d2c0cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.333+0000 7f8d2b7fe640 1 -- 172.21.15.27:0/267050022 >> v1:172.21.15.69:6789/0 conn(0x7f8d2c15e060 legacy=0x7f8d2c0ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.333+0000 7f8d2b7fe640 1 -- 172.21.15.27:0/267050022 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c0cc3c0 con 0x7f8d2c0d0880 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d2b7fe640 1 --2- 172.21.15.27:0/267050022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c0d0880 0x7f8d2c0ccbc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009e50 tx=0x7f8d2402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667e28c7af57918b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d29ffb640 1 -- 172.21.15.27:0/267050022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2402cb00 con 0x7f8d2c0d0880 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d29ffb640 1 -- 172.21.15.27:0/267050022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2402cca0 con 0x7f8d2c0d0880 2024-03-06T07:22:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d29ffb640 1 -- 172.21.15.27:0/267050022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d24034610 con 0x7f8d2c0d0880 2024-03-06T07:22:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d316af640 1 -- 172.21.15.27:0/267050022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c0d0880 msgr2=0x7f8d2c0ccbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d316af640 1 --2- 172.21.15.27:0/267050022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c0d0880 0x7f8d2c0ccbc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009e50 tx=0x7f8d2402bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d316af640 1 -- 172.21.15.27:0/267050022 shutdown_connections 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d316af640 1 --2- 172.21.15.27:0/267050022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c0d0880 0x7f8d2c0ccbc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d316af640 1 --2- 172.21.15.27:0/267050022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c06fe20 0x7f8d2c0cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d316af640 1 -- 172.21.15.27:0/267050022 >> 172.21.15.27:0/267050022 conn(0x7f8d2c0d1a90 msgr2=0x7f8d2c0d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d316af640 1 -- 172.21.15.27:0/267050022 shutdown_connections 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.334+0000 7f8d316af640 1 -- 172.21.15.27:0/267050022 wait complete. 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.336+0000 7f8d316af640 1 Processor -- start 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.336+0000 7f8d316af640 1 -- start start 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.336+0000 7f8d316af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c06fe20 0x7f8d2c0c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.336+0000 7f8d316af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c0d0880 0x7f8d2c1581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.336+0000 7f8d316af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c15e060 0x7f8d2c0c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.336+0000 7f8d316af640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0cef60 con 0x7f8d2c15e060 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d316af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c15e440 con 0x7f8d2c06fe20 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d316af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0cf0e0 con 0x7f8d2c0d0880 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c15e060 0x7f8d2c0c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c15e060 0x7f8d2c0c7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51406/0 (socket says 172.21.15.27:51406) 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2b7fe640 1 -- 172.21.15.27:0/2919391268 learned_addr learned my addr 172.21.15.27:0/2919391268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2affd640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c06fe20 0x7f8d2c0c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2a7fc640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c0d0880 0x7f8d2c1581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2b7fe640 1 -- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c0d0880 msgr2=0x7f8d2c1581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2b7fe640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c0d0880 0x7f8d2c1581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2b7fe640 1 -- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c06fe20 msgr2=0x7f8d2c0c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2b7fe640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c06fe20 0x7f8d2c0c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2b7fe640 1 -- 172.21.15.27:0/2919391268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c0c6960 con 0x7f8d2c15e060 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d2b7fe640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c15e060 0x7f8d2c0c7840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8d24036040 tx=0x7f8d24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d137fe640 1 -- 172.21.15.27:0/2919391268 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d24043020 con 0x7f8d2c15e060 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d137fe640 1 -- 172.21.15.27:0/2919391268 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2402ce50 con 0x7f8d2c15e060 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c0c63a0 con 0x7f8d2c15e060 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d137fe640 1 -- 172.21.15.27:0/2919391268 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d24009510 con 0x7f8d2c15e060 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.337+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c0b5c90 con 0x7f8d2c15e060 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.338+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c15d890 con 0x7f8d2c15e060 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.339+0000 7f8d137fe640 1 -- 172.21.15.27:0/2919391268 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d24046020 con 0x7f8d2c15e060 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.339+0000 7f8d137fe640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d1806b4d0 0x7f8d1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.339+0000 7f8d137fe640 1 -- 172.21.15.27:0/2919391268 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d240ac740 con 0x7f8d2c15e060 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.340+0000 7f8d2affd640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d1806b4d0 0x7f8d1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.340+0000 7f8d2affd640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d1806b4d0 0x7f8d1806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0cf940 tx=0x7f8d1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.349+0000 7f8d137fe640 1 -- 172.21.15.27:0/2919391268 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d2c15d890 con 0x7f8d2c15e060 2024-03-06T07:22:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.529+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d2c067ff0 con 0x7f8d2c15e060 2024-03-06T07:22:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.530+0000 7f8d137fe640 1 -- 172.21.15.27:0/2919391268 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f8d2404d030 con 0x7f8d2c15e060 2024-03-06T07:22:54.533 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:54.533 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d1806b4d0 msgr2=0x7f8d1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d1806b4d0 0x7f8d1806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0cf940 tx=0x7f8d1c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c15e060 msgr2=0x7f8d2c0c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c15e060 0x7f8d2c0c7840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8d24036040 tx=0x7f8d24004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 shutdown_connections 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d1806b4d0 0x7f8d1806d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d2c15e060 0x7f8d2c0c7840 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d2c0d0880 0x7f8d2c1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 --2- 172.21.15.27:0/2919391268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2c06fe20 0x7f8d2c0c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 >> 172.21.15.27:0/2919391268 conn(0x7f8d2c0d1a90 msgr2=0x7f8d2c06c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 shutdown_connections 2024-03-06T07:22:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.532+0000 7f8d316af640 1 -- 172.21.15.27:0/2919391268 wait complete. 2024-03-06T07:22:54.547 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:22:54.547 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:22:54.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:22:54.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:22:54.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:22:54.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:22:54.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:22:54.550 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:22:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.738+0000 7f9c8f59e640 1 Processor -- start 2024-03-06T07:22:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.738+0000 7f3600caf640 1 Processor -- start 2024-03-06T07:22:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.738+0000 7f3600caf640 1 -- start start 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f3600caf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35fc0b3390 0x7f35fc071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f3600caf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35fc06ec90 0x7f35fc06c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f3600caf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35fc16ae00 con 0x7f35fc0cc9a0 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f3600caf640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35fc0709d0 con 0x7f35fc0b3390 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.738+0000 7f9c8f59e640 1 -- start start 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8f59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9c880df100 0x7f9c880e0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8f59e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c88050180 0x7f9c88005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8f59e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c880df4e0 con 0x7f9c880530a0 2024-03-06T07:22:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8f59e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c880535f0 con 0x7f9c880df100 2024-03-06T07:22:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8f59e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c880538d0 con 0x7f9c88050180 2024-03-06T07:22:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7fbaebc8f640 1 Processor -- start 2024-03-06T07:22:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f3600caf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35fc072140 con 0x7f35fc06ec90 2024-03-06T07:22:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f35f9d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35fc0b3390 0x7f35fc071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f35f9d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35fc0b3390 0x7f35fc071c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51408/0 (socket says 172.21.15.27:51408) 2024-03-06T07:22:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f35f9d9b640 1 -- 172.21.15.27:0/2188341395 learned_addr learned my addr 172.21.15.27:0/2188341395 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f35fad9d640 1 --2- 172.21.15.27:0/2188341395 >> v2:172.21.15.69:3300/0 conn(0x7f35fc06ec90 0x7f35fc06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f35f9d9b640 1 -- 172.21.15.27:0/2188341395 >> v2:172.21.15.69:3300/0 conn(0x7f35fc06ec90 msgr2=0x7f35fc06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f35f9d9b640 1 --2- 172.21.15.27:0/2188341395 >> v2:172.21.15.69:3300/0 conn(0x7f35fc06ec90 0x7f35fc06c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f35f9d9b640 1 -- 172.21.15.27:0/2188341395 >> v1:172.21.15.69:6789/0 conn(0x7f35fc0cc9a0 legacy=0x7f35fc0b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f35f9d9b640 1 -- 172.21.15.27:0/2188341395 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc0b3890 con 0x7f35fc0b3390 2024-03-06T07:22:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8e59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9c880df100 0x7f9c880e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8ed9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c88050180 0x7f9c88005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8dd9b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9c880530a0 0x7f9c8804fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:42348/0 (socket says 172.21.15.27:42348) 2024-03-06T07:22:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8ed9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c88050180 0x7f9c88005fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55164/0 (socket says 172.21.15.27:55164) 2024-03-06T07:22:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.740+0000 7f9c8dd9b640 1 -- 172.21.15.27:0/67154121 learned_addr learned my addr 172.21.15.27:0/67154121 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.741+0000 7f9c8ed9d640 1 -- 172.21.15.27:0/67154121 >> v2:172.21.15.27:3300/0 conn(0x7f9c880df100 msgr2=0x7f9c880e0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.741+0000 7f9c8ed9d640 1 --2- 172.21.15.27:0/67154121 >> v2:172.21.15.27:3300/0 conn(0x7f9c880df100 0x7f9c880e0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.741+0000 7f9c8ed9d640 1 -- 172.21.15.27:0/67154121 >> v1:172.21.15.69:6789/0 conn(0x7f9c880530a0 legacy=0x7f9c8804fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.741+0000 7f9c8ed9d640 1 -- 172.21.15.27:0/67154121 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c88053bb0 con 0x7f9c88050180 2024-03-06T07:22:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7f9c8e59c640 1 --2- 172.21.15.27:0/67154121 >> v2:172.21.15.27:3300/0 conn(0x7f9c880df100 0x7f9c880e0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7f35f9d9b640 1 --2- 172.21.15.27:0/2188341395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35fc0b3390 0x7f35fc071c00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f35fc0ceb40 tx=0x7f35f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcccd3b7638ee865 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.741+0000 7fbaebc8f640 1 -- start start 2024-03-06T07:22:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbaebc8f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbae4154400 0x7fbae415a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbaebc8f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbae40aee70 0x7fbae40b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbaebc8f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbae4067820 0x7fbae406ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbaebc8f640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae40af320 con 0x7fbae4154400 2024-03-06T07:22:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbae922a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbae40aee70 0x7fbae40b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbae9a2b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbae4154400 0x7fbae415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbae922a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbae40aee70 0x7fbae40b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55192/0 (socket says 172.21.15.27:55192) 2024-03-06T07:22:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbae922a640 1 -- 172.21.15.27:0/4027664780 learned_addr learned my addr 172.21.15.27:0/4027664780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7f9c8ed9d640 1 --2- 172.21.15.27:0/67154121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c88050180 0x7f9c88005fe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9c88054550 tx=0x7f9c84034550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3685e03044a1f35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbaea22c640 1 --2- 172.21.15.27:0/4027664780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbae4067820 0x7fbae406ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbaebc8f640 1 -- 172.21.15.27:0/4027664780 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae415aae0 con 0x7fbae40aee70 2024-03-06T07:22:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.742+0000 7fbaebc8f640 1 -- 172.21.15.27:0/4027664780 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae415adc0 con 0x7fbae4067820 2024-03-06T07:22:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.743+0000 7f9c8d59a640 1 -- 172.21.15.27:0/67154121 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84008730 con 0x7f9c88050180 2024-03-06T07:22:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.743+0000 7f9c8d59a640 1 -- 172.21.15.27:0/67154121 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c840088d0 con 0x7f9c88050180 2024-03-06T07:22:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.743+0000 7f9c8d59a640 1 -- 172.21.15.27:0/67154121 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84008ca0 con 0x7f9c88050180 2024-03-06T07:22:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.743+0000 7fbae9a2b640 1 -- 172.21.15.27:0/4027664780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbae4067820 msgr2=0x7fbae406ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.743+0000 7fbae9a2b640 1 --2- 172.21.15.27:0/4027664780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbae4067820 0x7fbae406ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.743+0000 7fbae9a2b640 1 -- 172.21.15.27:0/4027664780 >> v2:172.21.15.69:3300/0 conn(0x7fbae40aee70 msgr2=0x7fbae40b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.744+0000 7f9c8f59e640 1 -- 172.21.15.27:0/67154121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c88050180 msgr2=0x7f9c88005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.744+0000 7f9c8f59e640 1 --2- 172.21.15.27:0/67154121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c88050180 0x7f9c88005fe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9c88054550 tx=0x7f9c84034550 comp rx=0 tx=0).stop 2024-03-06T07:22:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.743+0000 7fbae9a2b640 1 --2- 172.21.15.27:0/4027664780 >> v2:172.21.15.69:3300/0 conn(0x7fbae40aee70 0x7fbae40b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.743+0000 7fbae9a2b640 1 -- 172.21.15.27:0/4027664780 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae415b0a0 con 0x7fbae4154400 2024-03-06T07:22:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.744+0000 7fbae9a2b640 1 --2- 172.21.15.27:0/4027664780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae4154400 0x7fbae415a420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbae4070800 tx=0x7fbae002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbc50d181ce5500 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.745+0000 7fbae8a29640 1 -- 172.21.15.27:0/4027664780 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0038070 con 0x7fbae4154400 2024-03-06T07:22:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.745+0000 7fbae8a29640 1 -- 172.21.15.27:0/4027664780 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae002f050 con 0x7fbae4154400 2024-03-06T07:22:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.745+0000 7fbae8a29640 1 -- 172.21.15.27:0/4027664780 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0031950 con 0x7fbae4154400 2024-03-06T07:22:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.746+0000 7f35f959a640 1 -- 172.21.15.27:0/2188341395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0035020 con 0x7f35fc0b3390 2024-03-06T07:22:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.746+0000 7f35f959a640 1 -- 172.21.15.27:0/2188341395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f0031870 con 0x7f35fc0b3390 2024-03-06T07:22:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.746+0000 7fbaebc8f640 1 -- 172.21.15.27:0/4027664780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae4154400 msgr2=0x7fbae415a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.746+0000 7fbaebc8f640 1 --2- 172.21.15.27:0/4027664780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae4154400 0x7fbae415a420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbae4070800 tx=0x7fbae002b7f0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7f35f959a640 1 -- 172.21.15.27:0/2188341395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0031c40 con 0x7f35fc0b3390 2024-03-06T07:22:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7f3600caf640 1 -- 172.21.15.27:0/2188341395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35fc0b3390 msgr2=0x7f35fc071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7f3600caf640 1 --2- 172.21.15.27:0/2188341395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35fc0b3390 0x7f35fc071c00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f35fc0ceb40 tx=0x7f35f002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7fbaebc8f640 1 -- 172.21.15.27:0/4027664780 shutdown_connections 2024-03-06T07:22:54.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7fbaebc8f640 1 --2- 172.21.15.27:0/4027664780 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbae4067820 0x7fbae406ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7fbaebc8f640 1 --2- 172.21.15.27:0/4027664780 >> v2:172.21.15.69:3300/0 conn(0x7fbae40aee70 0x7fbae40b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7fbaebc8f640 1 --2- 172.21.15.27:0/4027664780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae4154400 0x7fbae415a420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7fbaebc8f640 1 -- 172.21.15.27:0/4027664780 >> 172.21.15.27:0/4027664780 conn(0x7fbae4066d00 msgr2=0x7fbae4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7f3600caf640 1 -- 172.21.15.27:0/2188341395 shutdown_connections 2024-03-06T07:22:54.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7fbaebc8f640 1 -- 172.21.15.27:0/4027664780 shutdown_connections 2024-03-06T07:22:54.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7fbaebc8f640 1 -- 172.21.15.27:0/4027664780 wait complete. 2024-03-06T07:22:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7f3600caf640 1 --2- 172.21.15.27:0/2188341395 >> v2:172.21.15.69:3300/0 conn(0x7f35fc06ec90 0x7f35fc06c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7f3600caf640 1 --2- 172.21.15.27:0/2188341395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35fc0b3390 0x7f35fc071c00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7f3600caf640 1 -- 172.21.15.27:0/2188341395 >> 172.21.15.27:0/2188341395 conn(0x7f35fc067120 msgr2=0x7f35fc067530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f0b2f247640 1 Processor -- start 2024-03-06T07:22:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f0b2f247640 1 -- start start 2024-03-06T07:22:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f3600caf640 1 -- 172.21.15.27:0/2188341395 shutdown_connections 2024-03-06T07:22:54.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.750+0000 7f0b2f247640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b280afb30 0x7f0b280719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.750+0000 7f0b2f247640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b28071ef0 0x7f0b280cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.750+0000 7f0b27fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b280afb30 0x7f0b280719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f3600caf640 1 -- 172.21.15.27:0/2188341395 wait complete. 2024-03-06T07:22:54.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7f0b27fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b280afb30 0x7f0b280719b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55198/0 (socket says 172.21.15.27:55198) 2024-03-06T07:22:54.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7f0b27fff640 1 -- 172.21.15.27:0/3321665811 learned_addr learned my addr 172.21.15.27:0/3321665811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7f3600caf640 1 Processor -- start 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7f0b2d7e4640 1 --2- 172.21.15.27:0/3321665811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b28071ef0 0x7f0b280cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7f0b2f247640 1 -- 172.21.15.27:0/3321665811 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280b4120 con 0x7f0b2815ac30 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.750+0000 7f0b2cfe3640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0b2815ac30 0x7f0b280b3940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:42360/0 (socket says 172.21.15.27:42360) 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7f0b2f247640 1 -- 172.21.15.27:0/3321665811 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b28072340 con 0x7f0b280afb30 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7f0b2f247640 1 -- 172.21.15.27:0/3321665811 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280b4400 con 0x7f0b28071ef0 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f0b277fe640 1 -- 172.21.15.27:0/3321665811 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936486133 0 0) 0x7f0b280b4120 con 0x7f0b2815ac30 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f0b277fe640 1 -- 172.21.15.27:0/3321665811 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b18005730 con 0x7f0b2815ac30 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f0b27fff640 1 -- 172.21.15.27:0/3321665811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b28071ef0 msgr2=0x7f0b280cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f0b27fff640 1 --2- 172.21.15.27:0/3321665811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b28071ef0 0x7f0b280cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f0b27fff640 1 -- 172.21.15.27:0/3321665811 >> v1:172.21.15.69:6789/0 conn(0x7f0b2815ac30 legacy=0x7f0b280b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f0b27fff640 1 -- 172.21.15.27:0/3321665811 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b280b46e0 con 0x7f0b280afb30 2024-03-06T07:22:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7f3600caf640 1 -- start start 2024-03-06T07:22:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f3600caf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35fc06ec90 0x7f35fc16a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f3600caf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35fc0b3390 0x7f35fc088b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f3600caf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35fc0cc9a0 0x7f35fc0890d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f35fa59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35fc06ec90 0x7f35fc16a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f35fad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35fc0cc9a0 0x7f35fc0890d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f35fa59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35fc06ec90 0x7f35fc16a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47804/0 (socket says 172.21.15.27:47804) 2024-03-06T07:22:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7f35fa59c640 1 -- 172.21.15.27:0/1599308115 learned_addr learned my addr 172.21.15.27:0/1599308115 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc071400 con 0x7f35fc0cc9a0 2024-03-06T07:22:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc071120 con 0x7f35fc0b3390 2024-03-06T07:22:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0716e0 con 0x7f35fc06ec90 2024-03-06T07:22:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.745+0000 7f9c8f59e640 1 -- 172.21.15.27:0/67154121 shutdown_connections 2024-03-06T07:22:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.745+0000 7f9c8f59e640 1 --2- 172.21.15.27:0/67154121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c88050180 0x7f9c88005fe0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.745+0000 7f9c8f59e640 1 --2- 172.21.15.27:0/67154121 >> v2:172.21.15.27:3300/0 conn(0x7f9c880df100 0x7f9c880e0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.745+0000 7f9c8f59e640 1 -- 172.21.15.27:0/67154121 >> 172.21.15.27:0/67154121 conn(0x7f9c88009fb0 msgr2=0x7f9c8800a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.745+0000 7f9c8f59e640 1 -- 172.21.15.27:0/67154121 shutdown_connections 2024-03-06T07:22:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f35fa59c640 1 -- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35fc0b3390 msgr2=0x7f35fc088b90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f35fa59c640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35fc0b3390 0x7f35fc088b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f35fa59c640 1 -- 172.21.15.27:0/1599308115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35fc0cc9a0 msgr2=0x7f35fc0890d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f35fa59c640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35fc0cc9a0 0x7f35fc0890d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f35fa59c640 1 -- 172.21.15.27:0/1599308115 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc1787a0 con 0x7f35fc06ec90 2024-03-06T07:22:54.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.745+0000 7f9c8f59e640 1 -- 172.21.15.27:0/67154121 wait complete. 2024-03-06T07:22:54.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7f9c8f59e640 1 Processor -- start 2024-03-06T07:22:54.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7f35fa59c640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35fc06ec90 0x7f35fc16a220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f35f400c730 tx=0x7f35f400cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.747+0000 7f9c8f59e640 1 -- start start 2024-03-06T07:22:54.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7f0b27fff640 1 --2- 172.21.15.27:0/3321665811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b280afb30 0x7f0b280719b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009870 tx=0x7f0b1402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a450c138d2426cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f9c8f59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c88050180 0x7f9c880dd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f9c8f59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c880530a0 0x7f9c8802d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f9c8f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c880df100 0x7f9c8802d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f9c8f59e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8800aad0 con 0x7f9c880df100 2024-03-06T07:22:54.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f9c8f59e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88054170 con 0x7f9c880530a0 2024-03-06T07:22:54.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.748+0000 7f9c8f59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8800ac50 con 0x7f9c88050180 2024-03-06T07:22:54.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c880df100 0x7f9c8802d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c880df100 0x7f9c8802d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51436/0 (socket says 172.21.15.27:51436) 2024-03-06T07:22:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8ed9d640 1 -- 172.21.15.27:0/1506024514 learned_addr learned my addr 172.21.15.27:0/1506024514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8e59c640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c88050180 0x7f9c880dd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8dd9b640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c880530a0 0x7f9c8802d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8ed9d640 1 -- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c88050180 msgr2=0x7f9c880dd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8ed9d640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9c88050180 0x7f9c880dd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8ed9d640 1 -- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c880530a0 msgr2=0x7f9c8802d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8ed9d640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c880530a0 0x7f9c8802d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7f9c8ed9d640 1 -- 172.21.15.27:0/1506024514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8802c090 con 0x7f9c880df100 2024-03-06T07:22:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7f9c8ed9d640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c880df100 0x7f9c8802d580 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c84007520 tx=0x7f9c840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7f35eaffd640 1 -- 172.21.15.27:0/1599308115 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4003b70 con 0x7f35fc06ec90 2024-03-06T07:22:54.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35fc178a40 con 0x7f35fc06ec90 2024-03-06T07:22:54.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35fc179000 con 0x7f35fc06ec90 2024-03-06T07:22:54.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7f35eaffd640 1 -- 172.21.15.27:0/1599308115 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f4003d30 con 0x7f35fc06ec90 2024-03-06T07:22:54.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7f0b277fe640 1 -- 172.21.15.27:0/3321665811 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14034070 con 0x7f0b280afb30 2024-03-06T07:22:54.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7f0b277fe640 1 -- 172.21.15.27:0/3321665811 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b14030860 con 0x7f0b280afb30 2024-03-06T07:22:54.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7f0b277fe640 1 -- 172.21.15.27:0/3321665811 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14030c70 con 0x7f0b280afb30 2024-03-06T07:22:54.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7f0b2f247640 1 -- 172.21.15.27:0/3321665811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b280afb30 msgr2=0x7f0b280719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7f0b2f247640 1 --2- 172.21.15.27:0/3321665811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b280afb30 0x7f0b280719b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009870 tx=0x7f0b1402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7f35eaffd640 1 -- 172.21.15.27:0/1599308115 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f400aa90 con 0x7f35fc06ec90 2024-03-06T07:22:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7f0b2f247640 1 -- 172.21.15.27:0/3321665811 shutdown_connections 2024-03-06T07:22:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7f0b2f247640 1 --2- 172.21.15.27:0/3321665811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b280afb30 0x7f0b280719b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7f0b2f247640 1 --2- 172.21.15.27:0/3321665811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b28071ef0 0x7f0b280cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7f0b2f247640 1 -- 172.21.15.27:0/3321665811 >> 172.21.15.27:0/3321665811 conn(0x7f0b280d0990 msgr2=0x7f0b280d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f35fc1793e0 con 0x7f35fc06ec90 2024-03-06T07:22:54.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.756+0000 7f0b2f247640 1 -- 172.21.15.27:0/3321665811 shutdown_connections 2024-03-06T07:22:54.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.756+0000 7f9c7effd640 1 -- 172.21.15.27:0/1506024514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8400fbd0 con 0x7f9c880df100 2024-03-06T07:22:54.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.749+0000 7fbaebc8f640 1 Processor -- start 2024-03-06T07:22:54.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.749+0000 7fbaebc8f640 1 -- start start 2024-03-06T07:22:54.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.750+0000 7fbaebc8f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbae4067820 0x7fbae4140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbaebc8f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae40aee70 0x7fbae414aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbae922a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae40aee70 0x7fbae414aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbae922a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae40aee70 0x7fbae414aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51452/0 (socket says 172.21.15.27:51452) 2024-03-06T07:22:54.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbae922a640 1 -- 172.21.15.27:0/1638672470 learned_addr learned my addr 172.21.15.27:0/1638672470 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbaebc8f640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae4154400 0x7fbae40b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae41547e0 con 0x7fbae40aee70 2024-03-06T07:22:54.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae406c7b0 con 0x7fbae4154400 2024-03-06T07:22:54.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae4154960 con 0x7fbae4067820 2024-03-06T07:22:54.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbae922a640 1 -- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbae4067820 msgr2=0x7fbae4140810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:54.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbae922a640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbae4067820 0x7fbae4140810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbae922a640 1 -- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae4154400 msgr2=0x7fbae40b15a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:54.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbae922a640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae4154400 0x7fbae40b15a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.751+0000 7fbae922a640 1 -- 172.21.15.27:0/1638672470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae41432b0 con 0x7fbae40aee70 2024-03-06T07:22:54.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7fbae922a640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae40aee70 0x7fbae414aa20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00c410 tx=0x7fbadc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.752+0000 7fbada7fc640 1 -- 172.21.15.27:0/1638672470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc0163f0 con 0x7fbae40aee70 2024-03-06T07:22:54.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.753+0000 7fbada7fc640 1 -- 172.21.15.27:0/1638672470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc016590 con 0x7fbae40aee70 2024-03-06T07:22:54.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7fbada7fc640 1 -- 172.21.15.27:0/1638672470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc016940 con 0x7fbae40aee70 2024-03-06T07:22:54.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae4143590 con 0x7fbae40aee70 2024-03-06T07:22:54.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.754+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae4143b50 con 0x7fbae40aee70 2024-03-06T07:22:54.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.755+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbae4143f30 con 0x7fbae40aee70 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.756+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8802c370 con 0x7f9c880df100 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.756+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c880ddb70 con 0x7f9c880df100 2024-03-06T07:22:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.756+0000 7f9c7effd640 1 -- 172.21.15.27:0/1506024514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8400fd70 con 0x7f9c880df100 2024-03-06T07:22:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.757+0000 7f35eaffd640 1 -- 172.21.15.27:0/1599308115 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35f400ac30 con 0x7f35fc06ec90 2024-03-06T07:22:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.757+0000 7f9c7effd640 1 -- 172.21.15.27:0/1506024514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84014490 con 0x7f9c880df100 2024-03-06T07:22:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.757+0000 7f35eaffd640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35dc06b5d0 0x7f35dc06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.757+0000 7f0b2f247640 1 -- 172.21.15.27:0/3321665811 wait complete. 2024-03-06T07:22:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.757+0000 7f35eaffd640 1 -- 172.21.15.27:0/1599308115 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35f4089fe0 con 0x7f35fc06ec90 2024-03-06T07:22:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.758+0000 7f35eaffd640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f35dc0737f0 0x7f35dc075c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.758+0000 7f35f9d9b640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35dc06b5d0 0x7f35dc06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.758+0000 7fa0f61e9640 1 Processor -- start 2024-03-06T07:22:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.758+0000 7fbada7fc640 1 -- 172.21.15.27:0/1638672470 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbadc016ae0 con 0x7fbae40aee70 2024-03-06T07:22:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.758+0000 7fa0f61e9640 1 -- start start 2024-03-06T07:22:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9c74001030 con 0x7f9c880df100 2024-03-06T07:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7fbada7fc640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbac808d5b0 0x7fbac808fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7fbae9a2b640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbac808d5b0 0x7fbac808fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7fa0f61e9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0f006cba0 0x7fa0f00b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7fa0f61e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f015cc10 con 0x7fa0f006c710 2024-03-06T07:22:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7fa0f61e9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f006d010 con 0x7fa0f015c830 2024-03-06T07:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7fa0f61e9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f0072b20 con 0x7fa0f006cba0 2024-03-06T07:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7fbada7fc640 1 -- 172.21.15.27:0/1638672470 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbadc08ac70 con 0x7fbae40aee70 2024-03-06T07:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7fbae9a2b640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbac808d5b0 0x7fbac808fa70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbae0035000 tx=0x7fbae0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7fa0ef7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa0f015c830 0x7fa0f01631c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:42370/0 (socket says 172.21.15.27:42370) 2024-03-06T07:22:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7fa0ef7fe640 1 -- 172.21.15.27:0/681785802 learned_addr learned my addr 172.21.15.27:0/681785802 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212841141 0 0) 0x7fa0f006d010 con 0x7fa0f015c830 2024-03-06T07:22:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d8003520 con 0x7fa0f015c830 2024-03-06T07:22:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7fbada7fc640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fbac80957d0 0x7fbac8097bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7fa0effff640 1 --2- 172.21.15.27:0/681785802 >> v2:172.21.15.27:3300/0 conn(0x7fa0f006cba0 0x7fa0f00b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7fbada7fc640 1 -- 172.21.15.27:0/1638672470 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbadc012ec0 con 0x7fbac80957d0 2024-03-06T07:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7fbada7fc640 1 -- 172.21.15.27:0/1638672470 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbadc08b060 con 0x7fbae40aee70 2024-03-06T07:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7f35eaffd640 1 -- 172.21.15.27:0/1599308115 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f35f4010e50 con 0x7f35dc0737f0 2024-03-06T07:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.759+0000 7f35eaffd640 1 -- 172.21.15.27:0/1599308115 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35f408c5e0 con 0x7f35fc06ec90 2024-03-06T07:22:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7f35fad9d640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f35dc0737f0 0x7f35dc075c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4207549092 0 0) 0x7fa0d8003520 con 0x7fa0f015c830 2024-03-06T07:22:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 >> v2:172.21.15.27:3300/0 conn(0x7fa0f006cba0 msgr2=0x7fa0f00b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 --2- 172.21.15.27:0/681785802 >> v2:172.21.15.27:3300/0 conn(0x7fa0f006cba0 0x7fa0f00b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 >> v1:172.21.15.27:6789/0 conn(0x7fa0f006c710 legacy=0x7fa0f00b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f0072840 con 0x7fa0f015c830 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fa0e0001670 con 0x7fa0f015c830 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 >> v1:172.21.15.69:6789/0 conn(0x7fa0f015c830 legacy=0x7fa0f01631c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0d8003a00 0x7fa0d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0d8004800 0x7fa0d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d80053c0 0x7fa0d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f0072b20 con 0x7fa0d80053c0 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d800c2a0 con 0x7fa0d8003a00 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d800c4a0 con 0x7fa0d8004800 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0ef7fe640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0d8003a00 0x7fa0d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0effff640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d80053c0 0x7fa0d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.761+0000 7fa0eeffd640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0d8004800 0x7fa0d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7fa0ef7fe640 1 -- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0d8004800 msgr2=0x7fa0d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7fa0ef7fe640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0d8004800 0x7fa0d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7fa0ef7fe640 1 -- 172.21.15.27:0/681785802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d80053c0 msgr2=0x7fa0d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7fa0ef7fe640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d80053c0 0x7fa0d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7fa0ef7fe640 1 -- 172.21.15.27:0/681785802 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d8003520 con 0x7fa0d8003a00 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7fa0effff640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d80053c0 0x7fa0d800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7fa0effff640 1 -- 172.21.15.27:0/681785802 reap_dead start 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7f0b2f247640 1 Processor -- start 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.760+0000 7f0b2f247640 1 -- start start 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2f247640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b28071ef0 0x7f0b2814b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2f247640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b280afb30 0x7f0b28157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2f247640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2815ac30 0x7f0b2806d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2f247640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280ce4d0 con 0x7f0b28071ef0 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2f247640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280b4ca0 con 0x7f0b280afb30 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2f247640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280ce650 con 0x7f0b2815ac30 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2cfe3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b28071ef0 0x7f0b2814b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2cfe3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b28071ef0 0x7f0b2814b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51494/0 (socket says 172.21.15.27:51494) 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b27fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b280afb30 0x7f0b28157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2cfe3640 1 -- 172.21.15.27:0/2170679723 learned_addr learned my addr 172.21.15.27:0/2170679723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.762+0000 7f0b2d7e4640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2815ac30 0x7f0b2806d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f0b2cfe3640 1 -- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2815ac30 msgr2=0x7f0b2806d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f0b2cfe3640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b2815ac30 0x7f0b2806d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f0b2cfe3640 1 -- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b280afb30 msgr2=0x7f0b28157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f0b2cfe3640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b280afb30 0x7f0b28157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f0b2cfe3640 1 -- 172.21.15.27:0/2170679723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2814f2f0 con 0x7f0b28071ef0 2024-03-06T07:22:54.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f9c7effd640 1 -- 172.21.15.27:0/1506024514 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c84014630 con 0x7f9c880df100 2024-03-06T07:22:54.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7fbaea22c640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fbac80957d0 0x7fbac8097bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f0b2cfe3640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b28071ef0 0x7f0b2814b220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c02e420 tx=0x7f0b1c02e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7fa0ef7fe640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0d8003a00 0x7fa0d8003e20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c110 tx=0x7fa0d8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31ec11223b0251fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f9c7effd640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9c7006b5a0 0x7f9c7006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f0b257fa640 1 -- 172.21.15.27:0/2170679723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c0072e0 con 0x7f0b28071ef0 2024-03-06T07:22:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f0b257fa640 1 -- 172.21.15.27:0/2170679723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c0074a0 con 0x7f0b28071ef0 2024-03-06T07:22:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7f0b257fa640 1 -- 172.21.15.27:0/2170679723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c004a20 con 0x7f0b28071ef0 2024-03-06T07:22:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.764+0000 7f9c8e59c640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9c7006b5a0 0x7f9c7006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.763+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0007c40 con 0x7fa0d8003a00 2024-03-06T07:22:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.764+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e0030070 con 0x7fa0d8003a00 2024-03-06T07:22:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.764+0000 7fa0ee7fc640 1 -- 172.21.15.27:0/681785802 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0005db0 con 0x7fa0d8003a00 2024-03-06T07:22:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.764+0000 7f9c7effd640 1 -- 172.21.15.27:0/1506024514 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c840acc30 con 0x7f9c880df100 2024-03-06T07:22:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.764+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2814e9c0 con 0x7f0b28071ef0 2024-03-06T07:22:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.764+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2814ef80 con 0x7f0b28071ef0 2024-03-06T07:22:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.764+0000 7f35fad9d640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f35dc0737f0 0x7f35dc075c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.764+0000 7f35f9d9b640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35dc06b5d0 0x7f35dc06da90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f35f0036000 tx=0x7f35f0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f14d03d0640 1 Processor -- start 2024-03-06T07:22:54.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f14d03d0640 1 -- start start 2024-03-06T07:22:54.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b2814e770 con 0x7f0b28071ef0 2024-03-06T07:22:54.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f9c7effd640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f9c700737c0 0x7f9c70075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f9c7effd640 1 -- 172.21.15.27:0/1506024514 --> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c8400fd70 con 0x7f9c700737c0 2024-03-06T07:22:54.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f9c8dd9b640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f9c700737c0 0x7f9c70075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f9c7effd640 1 -- 172.21.15.27:0/1506024514 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c840ad020 con 0x7f9c880df100 2024-03-06T07:22:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f5eb0a8a640 1 Processor -- start 2024-03-06T07:22:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f5eb0a8a640 1 -- start start 2024-03-06T07:22:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7fa0f61e9640 1 -- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0d8003a00 msgr2=0x7fa0d8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7fa0f61e9640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0d8003a00 0x7fa0d8003e20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c110 tx=0x7fa0d8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7fa0f61e9640 1 -- 172.21.15.27:0/681785802 shutdown_connections 2024-03-06T07:22:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7fa0f61e9640 1 --2- 172.21.15.27:0/681785802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0d8003a00 0x7fa0d8003e20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7fa0f61e9640 1 -- 172.21.15.27:0/681785802 >> 172.21.15.27:0/681785802 conn(0x7fa0f0066d00 msgr2=0x7fa0f00670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7f9c8dd9b640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f9c700737c0 0x7f9c70075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.766+0000 7f9c8e59c640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9c7006b5a0 0x7f9c7006da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9c88035d70 tx=0x7f9c80011040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7fa0f61e9640 1 -- 172.21.15.27:0/681785802 shutdown_connections 2024-03-06T07:22:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.766+0000 7f35eaffd640 1 -- 172.21.15.27:0/1599308115 <== osd.3 v2:172.21.15.27:6824/4277055095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f35f4010e50 con 0x7f35dc0737f0 2024-03-06T07:22:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.765+0000 7fa0f61e9640 1 -- 172.21.15.27:0/681785802 wait complete. 2024-03-06T07:22:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.766+0000 7fbaea22c640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fbac80957d0 0x7fbac8097bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.766+0000 7f5eb0a8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 0x7f5eac1631c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eb0a8a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5eac06cb70 0x7f5eac166470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eb0a8a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac163700 con 0x7f5eac06c710 2024-03-06T07:22:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f9c7effd640 1 -- 172.21.15.27:0/1506024514 <== osd.2 v2:172.21.15.27:6808/3892975520 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c8400fd70 con 0x7f9c700737c0 2024-03-06T07:22:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eb0a8a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eac072b20 con 0x7f5eac06cb70 2024-03-06T07:22:54.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eb0a8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac15cc10 con 0x7f5eac15c830 2024-03-06T07:22:54.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eaa59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 0x7f5eac1631c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eaa59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 0x7f5eac1631c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47822/0 (socket says 172.21.15.27:47822) 2024-03-06T07:22:54.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eaad9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5eac06cb70 0x7f5eac166470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eaa59c640 1 -- 172.21.15.27:0/3845258466 learned_addr learned my addr 172.21.15.27:0/3845258466 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f14d03d0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14c8159d50 0x7f14c806c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7fbada7fc640 1 -- 172.21.15.27:0/1638672470 <== osd.6 v2:172.21.15.69:6800/288062332 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbadc012ec0 con 0x7fbac80957d0 2024-03-06T07:22:54.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f14d03d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14c8070f80 0x7f14c80b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f14d03d0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c806ccf0 con 0x7f14c80aef90 2024-03-06T07:22:54.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f14d03d0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c80b28c0 con 0x7f14c8159d50 2024-03-06T07:22:54.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f14d03d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c806cfd0 con 0x7f14c8070f80 2024-03-06T07:22:54.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f14ce16c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14c8159d50 0x7f14c806c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f14ce16c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14c8159d50 0x7f14c806c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51502/0 (socket says 172.21.15.27:51502) 2024-03-06T07:22:54.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7fa0f61e9640 1 Processor -- start 2024-03-06T07:22:54.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.768+0000 7f14ce16c640 1 -- 172.21.15.27:0/2372905785 learned_addr learned my addr 172.21.15.27:0/2372905785 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.768+0000 7f14ce96d640 1 --2- 172.21.15.27:0/2372905785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14c8070f80 0x7f14c80b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.768+0000 7fa0f61e9640 1 -- start start 2024-03-06T07:22:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.768+0000 7f14ce16c640 1 -- 172.21.15.27:0/2372905785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14c8070f80 msgr2=0x7f14c80b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.768+0000 7f14ce16c640 1 --2- 172.21.15.27:0/2372905785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14c8070f80 0x7f14c80b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.768+0000 7f14ce16c640 1 -- 172.21.15.27:0/2372905785 >> v1:172.21.15.27:6789/0 conn(0x7f14c80aef90 legacy=0x7f14c8070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eaad9d640 1 -- 172.21.15.27:0/3845258466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 msgr2=0x7f5eac1631c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eaad9d640 1 --2- 172.21.15.27:0/3845258466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 0x7f5eac1631c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eaad9d640 1 -- 172.21.15.27:0/3845258466 >> v1:172.21.15.69:6789/0 conn(0x7f5eac06c710 legacy=0x7f5eac165d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.767+0000 7f5eaad9d640 1 -- 172.21.15.27:0/3845258466 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac072840 con 0x7f5eac06cb70 2024-03-06T07:22:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.768+0000 7f14ce16c640 1 -- 172.21.15.27:0/2372905785 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c806d2b0 con 0x7f14c8159d50 2024-03-06T07:22:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.769+0000 7f14ce16c640 1 --2- 172.21.15.27:0/2372905785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8159d50 0x7f14c806c710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f14c40083f0 tx=0x7f14c400abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6f7a3fedc96c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7fa0f61e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d8003a00 0x7fa0f0147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7f5eaad9d640 1 --2- 172.21.15.27:0/3845258466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eac06cb70 0x7f5eac166470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5eac070630 tx=0x7f5e9c02b6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f724ab51b4484b59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7f14cd16a640 1 -- 172.21.15.27:0/2372905785 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c4034020 con 0x7f14c8159d50 2024-03-06T07:22:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7f14cd16a640 1 -- 172.21.15.27:0/2372905785 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c4008df0 con 0x7f14c8159d50 2024-03-06T07:22:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7f14cd16a640 1 -- 172.21.15.27:0/2372905785 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c400a2f0 con 0x7f14c8159d50 2024-03-06T07:22:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7f5ea959a640 1 -- 172.21.15.27:0/3845258466 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c037070 con 0x7f5eac06cb70 2024-03-06T07:22:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7f5ea959a640 1 -- 172.21.15.27:0/3845258466 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c030710 con 0x7f5eac06cb70 2024-03-06T07:22:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7f5ea959a640 1 -- 172.21.15.27:0/3845258466 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c030ae0 con 0x7f5eac06cb70 2024-03-06T07:22:54.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7fa0ef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d8003a00 0x7fa0f0147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7fa0ef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d8003a00 0x7fa0f0147c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51514/0 (socket says 172.21.15.27:51514) 2024-03-06T07:22:54.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7fa0f61e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f01619e0 0x7fa0f0162e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.770+0000 7fa0ef7fe640 1 -- 172.21.15.27:0/165352993 learned_addr learned my addr 172.21.15.27:0/165352993 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7fa0f61e9640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f014d470 0x7fa0f014d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7fa0eeffd640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f01619e0 0x7fa0f0162e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f14d03d0640 1 -- 172.21.15.27:0/2372905785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8159d50 msgr2=0x7f14c806c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f0070f60 con 0x7fa0d8003a00 2024-03-06T07:22:54.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f14d03d0640 1 --2- 172.21.15.27:0/2372905785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8159d50 0x7f14c806c710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f14c40083f0 tx=0x7f14c400abe0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7fa0effff640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f014d470 0x7fa0f014d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f5eb0a8a640 1 -- 172.21.15.27:0/3845258466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eac06cb70 msgr2=0x7f5eac166470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f0070c80 con 0x7fa0f014d470 2024-03-06T07:22:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f00b4960 con 0x7fa0f01619e0 2024-03-06T07:22:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f5eb0a8a640 1 --2- 172.21.15.27:0/3845258466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eac06cb70 0x7f5eac166470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5eac070630 tx=0x7f5e9c02b6f0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f14d03d0640 1 -- 172.21.15.27:0/2372905785 shutdown_connections 2024-03-06T07:22:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f14d03d0640 1 --2- 172.21.15.27:0/2372905785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14c8070f80 0x7f14c80b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f14d03d0640 1 --2- 172.21.15.27:0/2372905785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8159d50 0x7f14c806c710 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f14d03d0640 1 -- 172.21.15.27:0/2372905785 >> 172.21.15.27:0/2372905785 conn(0x7f14c8066d00 msgr2=0x7f14c80670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f5eb0a8a640 1 -- 172.21.15.27:0/3845258466 shutdown_connections 2024-03-06T07:22:54.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f5eb0a8a640 1 --2- 172.21.15.27:0/3845258466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eac06cb70 0x7f5eac166470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7f14d03d0640 1 -- 172.21.15.27:0/2372905785 shutdown_connections 2024-03-06T07:22:54.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0eeffd640 1 -- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f014d470 msgr2=0x7fa0f014d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0eeffd640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f014d470 0x7fa0f014d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0eeffd640 1 -- 172.21.15.27:0/165352993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d8003a00 msgr2=0x7fa0f0147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0eeffd640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d8003a00 0x7fa0f0147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7f14d03d0640 1 -- 172.21.15.27:0/2372905785 wait complete. 2024-03-06T07:22:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0eeffd640 1 -- 172.21.15.27:0/165352993 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f014ce50 con 0x7fa0f01619e0 2024-03-06T07:22:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0ef7fe640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d8003a00 0x7fa0f0147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0eeffd640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f01619e0 0x7fa0f0162e20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0069700 tx=0x7fa0e400e8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0d7fff640 1 -- 172.21.15.27:0/165352993 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4019070 con 0x7fa0f01619e0 2024-03-06T07:22:54.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0effff640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f014d470 0x7fa0f014d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:54.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0d7fff640 1 -- 172.21.15.27:0/165352993 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e40052e0 con 0x7fa0f01619e0 2024-03-06T07:22:54.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0d7fff640 1 -- 172.21.15.27:0/165352993 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4005490 con 0x7fa0f01619e0 2024-03-06T07:22:54.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f0160f00 con 0x7fa0f01619e0 2024-03-06T07:22:54.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.771+0000 7f5eb0a8a640 1 --2- 172.21.15.27:0/3845258466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 0x7f5eac1631c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.773+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f0163360 con 0x7fa0f01619e0 2024-03-06T07:22:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.772+0000 7f5eb0a8a640 1 -- 172.21.15.27:0/3845258466 >> 172.21.15.27:0/3845258466 conn(0x7f5eac066d00 msgr2=0x7f5eac0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.773+0000 7f5eb0a8a640 1 -- 172.21.15.27:0/3845258466 shutdown_connections 2024-03-06T07:22:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.774+0000 7f14d03d0640 1 Processor -- start 2024-03-06T07:22:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.774+0000 7f14d03d0640 1 -- start start 2024-03-06T07:22:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.775+0000 7fa0d7fff640 1 -- 172.21.15.27:0/165352993 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0e401c070 con 0x7fa0f01619e0 2024-03-06T07:22:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.775+0000 7fa0d7fff640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa0dc06b5a0 0x7fa0dc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.775+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa0f0148190 con 0x7fa0f01619e0 2024-03-06T07:22:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.774+0000 7f5eb0a8a640 1 -- 172.21.15.27:0/3845258466 wait complete. 2024-03-06T07:22:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.775+0000 7fe66cb43640 1 Processor -- start 2024-03-06T07:22:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7f14d03d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8070f80 0x7f14c81ef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7f14d03d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14c80aef90 0x7f14c814ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.776+0000 7f5eb0a8a640 1 Processor -- start 2024-03-06T07:22:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7f5eb0a8a640 1 -- start start 2024-03-06T07:22:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.775+0000 7fa0d7fff640 1 -- 172.21.15.27:0/165352993 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0e4089700 con 0x7fa0f01619e0 2024-03-06T07:22:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.776+0000 7fa0ef7fe640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa0dc06b5a0 0x7fa0dc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.776+0000 7fa0d7fff640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fa0dc073740 0x7fa0dc075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.776+0000 7fa0d7fff640 1 -- 172.21.15.27:0/165352993 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0e401be50 con 0x7fa0dc073740 2024-03-06T07:22:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.776+0000 7fa0d7fff640 1 -- 172.21.15.27:0/165352993 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0e40541b0 con 0x7fa0f01619e0 2024-03-06T07:22:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.776+0000 7fa0effff640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fa0dc073740 0x7fa0dc075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7fa0ef7fe640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa0dc06b5a0 0x7fa0dc06da60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0001c70 tx=0x7fa0e0009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7fa0effff640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fa0dc073740 0x7fa0dc075b60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.776+0000 7fe66cb43640 1 -- start start 2024-03-06T07:22:54.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fa0d7fff640 1 -- 172.21.15.27:0/165352993 <== osd.5 v2:172.21.15.69:6819/715204514 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa0e401be50 con 0x7fa0dc073740 2024-03-06T07:22:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7f14ce16c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8070f80 0x7f14c81ef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7f14ce16c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8070f80 0x7f14c81ef6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51520/0 (socket says 172.21.15.27:51520) 2024-03-06T07:22:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7f14ce16c640 1 -- 172.21.15.27:0/547744587 learned_addr learned my addr 172.21.15.27:0/547744587 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7f14d03d0640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14c8159d50 0x7f14c815b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7fe66cb43640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe668165c70 0x7fe6680b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7fe66cb43640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c4af0 0x7fe6680c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7fe66cb43640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6680cfde0 con 0x7fe6680b1540 2024-03-06T07:22:54.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7fe66cb43640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6680c54b0 con 0x7fe668165c70 2024-03-06T07:22:54.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.777+0000 7fe66cb43640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66806b6f0 con 0x7fe6680c4af0 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7fe666d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe6680b1540 0x7fe6680b19d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50026/0 (socket says 172.21.15.27:50026) 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7fe666d9d640 1 -- 172.21.15.27:0/1636538401 learned_addr learned my addr 172.21.15.27:0/1636538401 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073715165 0 0) 0x7fe6680cfde0 con 0x7fe6680b1540 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe650003520 con 0x7fe6680b1540 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4289160676 0 0) 0x7fe650003520 con 0x7fe6680b1540 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c4af0 msgr2=0x7fe6680c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe665d9b640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c4af0 0x7fe6680c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c4af0 0x7fe6680c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 >> v2:172.21.15.27:3300/0 conn(0x7fe668165c70 msgr2=0x7fe6680b41f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 --2- 172.21.15.27:0/1636538401 >> v2:172.21.15.27:3300/0 conn(0x7fe668165c70 0x7fe6680b41f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66806b0a0 con 0x7fe6680b1540 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fe6580019f0 con 0x7fe6680b1540 2024-03-06T07:22:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 >> v1:172.21.15.27:6789/0 conn(0x7fe6680b1540 legacy=0x7fe6680b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6500039b0 0x7fe650003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650004800 0x7fe650004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6500053c0 0x7fe65000b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680c54b0 con 0x7fe650004800 2024-03-06T07:22:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66806b6f0 con 0x7fe6500039b0 2024-03-06T07:22:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65000c2f0 con 0x7fe6500053c0 2024-03-06T07:22:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c806e420 con 0x7f14c8070f80 2024-03-06T07:22:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c815a130 con 0x7f14c80aef90 2024-03-06T07:22:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe665d9b640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c4af0 0x7fe6680c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c806e960 con 0x7f14c8159d50 2024-03-06T07:22:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7f14ce96d640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14c8159d50 0x7f14c815b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7f14ce16c640 1 -- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14c8159d50 msgr2=0x7f14c815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7f14ce16c640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14c8159d50 0x7f14c815b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.778+0000 7f14ce16c640 1 -- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14c80aef90 msgr2=0x7f14c814ab80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7f14cd96b640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14c80aef90 0x7f14c814ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7f14ce16c640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14c80aef90 0x7f14c814ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7f14ce16c640 1 -- 172.21.15.27:0/547744587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c8156e50 con 0x7f14c8070f80 2024-03-06T07:22:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7f14ce16c640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8070f80 0x7f14c81ef6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f14c40083f0 tx=0x7f14c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe66659c640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6500039b0 0x7fe650003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe666d9d640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6500053c0 0x7fe65000b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.779+0000 7fe665d9b640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650004800 0x7fe650004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7fe666d9d640 1 -- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6500039b0 msgr2=0x7fe650003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7fe666d9d640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6500039b0 0x7fe650003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7fe666d9d640 1 -- 172.21.15.27:0/1636538401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650004800 msgr2=0x7fe650004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7fe666d9d640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe650004800 0x7fe650004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7fe666d9d640 1 -- 172.21.15.27:0/1636538401 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66806b0a0 con 0x7fe6500053c0 2024-03-06T07:22:54.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.781+0000 7f5eb0a8a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eac06c710 0x7f5eac1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.781+0000 7f5eb0a8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eac06cb70 0x7f5eac1ebb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.781+0000 7f5eb0a8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 0x7f5eac15f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.781+0000 7f5eb0a8a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac070f60 con 0x7f5eac06cb70 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.781+0000 7f5eb0a8a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac070c80 con 0x7f5eac06c710 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.781+0000 7f5eb0a8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac06cfe0 con 0x7f5eac15c830 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.782+0000 7f5ea9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eac06cb70 0x7f5eac1ebb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.782+0000 7f5ea9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eac06cb70 0x7f5eac1ebb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51542/0 (socket says 172.21.15.27:51542) 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.782+0000 7f5ea9d9b640 1 -- 172.21.15.27:0/2271705604 learned_addr learned my addr 172.21.15.27:0/2271705604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.782+0000 7f5eaad9d640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 0x7f5eac15f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f5ea9d9b640 1 -- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 msgr2=0x7f5eac15f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f5ea9d9b640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5eac15c830 0x7f5eac15f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f5ea9d9b640 1 -- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eac06c710 msgr2=0x7f5eac1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f5eaa59c640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eac06c710 0x7f5eac1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f5ea9d9b640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eac06c710 0x7f5eac1552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f5ea9d9b640 1 -- 172.21.15.27:0/2271705604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac15dbf0 con 0x7f5eac06cb70 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7f14beffd640 1 -- 172.21.15.27:0/547744587 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c4034040 con 0x7f14c8070f80 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c8157130 con 0x7f14c8070f80 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7f14beffd640 1 -- 172.21.15.27:0/547744587 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c4030880 con 0x7f14c8070f80 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7f14beffd640 1 -- 172.21.15.27:0/547744587 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c4030c30 con 0x7f14c8070f80 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.780+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c80b0e20 con 0x7f14c8070f80 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.782+0000 7f14beffd640 1 -- 172.21.15.27:0/547744587 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14c404a060 con 0x7f14c8070f80 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f14beffd640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14ac06b4d0 0x7f14ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f14cd96b640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14ac06b4d0 0x7f14ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f14beffd640 1 -- 172.21.15.27:0/547744587 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14c40ac2b0 con 0x7f14c8070f80 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f14cd96b640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14ac06b4d0 0x7f14ac06d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14c815c8b0 tx=0x7f14b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f14d03d0640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f14b0001650 0x7f14b0003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f5ea9d9b640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eac06cb70 0x7f5eac1ebb80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400b480 tx=0x7f5e9400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14b0006bb0 con 0x7f14b0001650 2024-03-06T07:22:54.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7f5ea2ffd640 1 -- 172.21.15.27:0/2271705604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94014070 con 0x7f5eac06cb70 2024-03-06T07:22:54.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.784+0000 7f5eb0a8a640 1 -- 172.21.15.27:0/2271705604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eac15e8b0 con 0x7f5eac06cb70 2024-03-06T07:22:54.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.784+0000 7f14ce96d640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f14b0001650 0x7f14b0003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.784+0000 7f14ce96d640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f14b0001650 0x7f14b0003b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.785+0000 7f14beffd640 1 -- 172.21.15.27:0/547744587 <== osd.0 v2:172.21.15.27:6800/3926946944 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f14b0006bb0 con 0x7f14b0001650 2024-03-06T07:22:54.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.783+0000 7fe666d9d640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6500053c0 0x7fe65000b9e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe65800a930 tx=0x7fe650005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aae854e6ca9d3aa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.784+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65800a500 con 0x7fe6500053c0 2024-03-06T07:22:54.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.784+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe658030070 con 0x7fe6500053c0 2024-03-06T07:22:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.784+0000 7fe66559a640 1 -- 172.21.15.27:0/1636538401 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65800a500 con 0x7fe6500053c0 2024-03-06T07:22:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.786+0000 7f5ea2ffd640 1 -- 172.21.15.27:0/2271705604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e94010890 con 0x7f5eac06cb70 2024-03-06T07:22:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.786+0000 7f5eb0a8a640 1 -- 172.21.15.27:0/2271705604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eac15ee70 con 0x7f5eac06cb70 2024-03-06T07:22:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.787+0000 7fe66cb43640 1 -- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6500053c0 msgr2=0x7fe65000b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.787+0000 7fe66cb43640 1 --2- 172.21.15.27:0/1636538401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6500053c0 0x7fe65000b9e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe65800a930 tx=0x7fe650005830 comp rx=0 tx=0).stop 2024-03-06T07:22:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.787+0000 7fe666d9d640 1 -- 172.21.15.27:0/1636538401 reap_dead start 2024-03-06T07:22:54.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.787+0000 7f5ea2ffd640 1 -- 172.21.15.27:0/2271705604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94010c80 con 0x7f5eac06cb70 2024-03-06T07:22:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.787+0000 7f5ea2ffd640 1 -- 172.21.15.27:0/2271705604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e94017070 con 0x7f5eac06cb70 2024-03-06T07:22:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.787+0000 7f5ea0ff9640 1 -- 172.21.15.27:0/2271705604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e90001030 con 0x7f5eac06cb70 2024-03-06T07:22:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.787+0000 7f5ea2ffd640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e8c06b5d0 0x7f5e8c06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.788+0000 7f5ea2ffd640 1 -- 172.21.15.27:0/2271705604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e940895d0 con 0x7f5eac06cb70 2024-03-06T07:22:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.788+0000 7f5ea2ffd640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f5e8c0737d0 0x7f5e8c075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.788+0000 7f5eaa59c640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e8c06b5d0 0x7f5e8c06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.788+0000 7f5ea2ffd640 1 -- 172.21.15.27:0/2271705604 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e94013ec0 con 0x7f5e8c0737d0 2024-03-06T07:22:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.788+0000 7f5eaad9d640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f5e8c0737d0 0x7f5e8c075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.788+0000 7f5ea2ffd640 1 -- 172.21.15.27:0/2271705604 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e9408e060 con 0x7f5eac06cb70 2024-03-06T07:22:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.788+0000 7fe66cb43640 1 -- 172.21.15.27:0/1636538401 shutdown_connections 2024-03-06T07:22:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.788+0000 7fe66cb43640 1 -- 172.21.15.27:0/1636538401 >> 172.21.15.27:0/1636538401 conn(0x7fe6680b36d0 msgr2=0x7fe6680b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.788+0000 7f5eaa59c640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e8c06b5d0 0x7f5e8c06da90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4009c40 tx=0x7f5ea400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.789+0000 7fe66cb43640 1 -- 172.21.15.27:0/1636538401 shutdown_connections 2024-03-06T07:22:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.789+0000 7fe66cb43640 1 -- 172.21.15.27:0/1636538401 wait complete. 2024-03-06T07:22:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.789+0000 7f5eaad9d640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f5e8c0737d0 0x7f5e8c075bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.790+0000 7f5ea2ffd640 1 -- 172.21.15.27:0/2271705604 <== osd.4 v2:172.21.15.69:6808/1626703870 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5e94013ec0 con 0x7f5e8c0737d0 2024-03-06T07:22:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.793+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 --> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9c74002dd0 con 0x7f9c700737c0 2024-03-06T07:22:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.794+0000 7fe66cb43640 1 Processor -- start 2024-03-06T07:22:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.794+0000 7fe66cb43640 1 -- start start 2024-03-06T07:22:54.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.794+0000 7f9c7effd640 1 -- 172.21.15.27:0/1506024514 <== osd.2 v2:172.21.15.27:6808/3892975520 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9c74002dd0 con 0x7f9c700737c0 2024-03-06T07:22:54.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f9c700737c0 msgr2=0x7f9c70075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7f9c8f59e640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f9c700737c0 0x7f9c70075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe66cb43640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6500053c0 0x7fe6681f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe66cb43640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c9c20 0x7fe668163200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe66cb43640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680ca0a0 0x7fe6681fb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe66cb43640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680cfa00 con 0x7fe6680ca0a0 2024-03-06T07:22:54.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe66cb43640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680d00c0 con 0x7fe6500053c0 2024-03-06T07:22:54.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe66cb43640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680cfb80 con 0x7fe6680c9c20 2024-03-06T07:22:54.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9c7006b5a0 msgr2=0x7f9c7006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7f9c8f59e640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9c7006b5a0 0x7f9c7006da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9c88035d70 tx=0x7f9c80011040 comp rx=0 tx=0).stop 2024-03-06T07:22:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c880df100 msgr2=0x7f9c8802d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7f9c8f59e640 1 --2- 172.21.15.27:0/1506024514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c880df100 0x7f9c8802d580 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c84007520 tx=0x7f9c840046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe665d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c9c20 0x7fe668163200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe665d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c9c20 0x7fe668163200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47894/0 (socket says 172.21.15.27:47894) 2024-03-06T07:22:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe665d9b640 1 -- 172.21.15.27:0/3232679130 learned_addr learned my addr 172.21.15.27:0/3232679130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7fe666d9d640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680ca0a0 0x7fe6681fb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7f9c8ed9d640 1 -- 172.21.15.27:0/1506024514 reap_dead start 2024-03-06T07:22:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 shutdown_connections 2024-03-06T07:22:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.795+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 >> 172.21.15.27:0/1506024514 conn(0x7f9c88009fb0 msgr2=0x7f9c880de430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7fe665d9b640 1 -- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6500053c0 msgr2=0x7fe6681f53b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7fe665d9b640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6500053c0 0x7fe6681f53b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7fe665d9b640 1 -- 172.21.15.27:0/3232679130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680ca0a0 msgr2=0x7fe6681fb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7fe665d9b640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680ca0a0 0x7fe6681fb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7fe665d9b640 1 -- 172.21.15.27:0/3232679130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6680c6390 con 0x7fe6680c9c20 2024-03-06T07:22:54.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 shutdown_connections 2024-03-06T07:22:54.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7f9c8f59e640 1 -- 172.21.15.27:0/1506024514 wait complete. 2024-03-06T07:22:54.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7fe665d9b640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c9c20 0x7fe668163200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00e5c0 tx=0x7fe65c00ea80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7fe656ffd640 1 -- 172.21.15.27:0/3232679130 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c019020 con 0x7fe6680c9c20 2024-03-06T07:22:54.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7fe656ffd640 1 -- 172.21.15.27:0/3232679130 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c015690 con 0x7fe6680c9c20 2024-03-06T07:22:54.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.796+0000 7fe656ffd640 1 -- 172.21.15.27:0/3232679130 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c015a80 con 0x7fe6680c9c20 2024-03-06T07:22:54.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.797+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6681629b0 con 0x7fe6680c9c20 2024-03-06T07:22:54.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.797+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6681fc150 con 0x7fe6680c9c20 2024-03-06T07:22:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.798+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe668162ed0 con 0x7fe6680c9c20 2024-03-06T07:22:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.799+0000 7fe656ffd640 1 -- 172.21.15.27:0/3232679130 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe65c01c020 con 0x7fe6680c9c20 2024-03-06T07:22:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.799+0000 7fe656ffd640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe64806b4d0 0x7fe64806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.800+0000 7fe656ffd640 1 -- 172.21.15.27:0/3232679130 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe65c08aaa0 con 0x7fe6680c9c20 2024-03-06T07:22:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.800+0000 7fe656ffd640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7fe6480736d0 0x7fe648075af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.800+0000 7fe656ffd640 1 -- 172.21.15.27:0/3232679130 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6480761a0 con 0x7fe6480736d0 2024-03-06T07:22:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.800+0000 7fe656ffd640 1 -- 172.21.15.27:0/3232679130 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe65c0554b0 con 0x7fe6680c9c20 2024-03-06T07:22:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.801+0000 7fe66659c640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe64806b4d0 0x7fe64806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.801+0000 7f0b257fa640 1 -- 172.21.15.27:0/2170679723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b1c03c020 con 0x7f0b28071ef0 2024-03-06T07:22:54.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.802+0000 7f0b257fa640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b0c06b520 0x7f0b0c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.802+0000 7fe666d9d640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7fe6480736d0 0x7fe648075af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.802+0000 7f0b27fff640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b0c06b520 0x7f0b0c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.803+0000 7fe666d9d640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7fe6480736d0 0x7fe648075af0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.803+0000 7fe66659c640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe64806b4d0 0x7fe64806d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe6600098a0 tx=0x7fe660006ee0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.803+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f35fc04f330 con 0x7f35dc0737f0 2024-03-06T07:22:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.802+0000 7f0b257fa640 1 -- 172.21.15.27:0/2170679723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b1c078670 con 0x7f0b28071ef0 2024-03-06T07:22:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.802+0000 7f0b257fa640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f0b0c074350 0x7f0b0c0767f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.802+0000 7f0b257fa640 1 -- 172.21.15.27:0/2170679723 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b0c076ec0 con 0x7f0b0c074350 2024-03-06T07:22:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.803+0000 7f0b27fff640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b0c06b520 0x7f0b0c06d9e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b14036000 tx=0x7f0b1403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.803+0000 7f0b2d7e4640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f0b0c074350 0x7f0b0c0767f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.803+0000 7f0b2d7e4640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f0b0c074350 0x7f0b0c0767f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.803+0000 7f0b257fa640 1 -- 172.21.15.27:0/2170679723 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b1c077000 con 0x7f0b28071ef0 2024-03-06T07:22:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.804+0000 7fe656ffd640 1 -- 172.21.15.27:0/3232679130 <== osd.7 v2:172.21.15.69:6816/1392567864 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe6480761a0 con 0x7fe6480736d0 2024-03-06T07:22:54.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f35eaffd640 1 -- 172.21.15.27:0/1599308115 <== osd.3 v2:172.21.15.27:6824/4277055095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f35fc04f330 con 0x7f35dc0737f0 2024-03-06T07:22:54.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f0b257fa640 1 -- 172.21.15.27:0/2170679723 <== osd.1 v2:172.21.15.27:6809/746548704 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0b0c076ec0 con 0x7f0b0c074350 2024-03-06T07:22:54.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa0f0148190 con 0x7fa0dc073740 2024-03-06T07:22:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbae41440e0 con 0x7fbac80957d0 2024-03-06T07:22:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7fa0d7fff640 1 -- 172.21.15.27:0/165352993 <== osd.5 v2:172.21.15.69:6819/715204514 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa0e401be50 con 0x7fa0dc073740 2024-03-06T07:22:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f35dc0737f0 msgr2=0x7f35dc075c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f3600caf640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f35dc0737f0 0x7f35dc075c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35dc06b5d0 msgr2=0x7f35dc06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f3600caf640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35dc06b5d0 0x7f35dc06da90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f35f0036000 tx=0x7f35f0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35fc06ec90 msgr2=0x7f35fc16a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f3600caf640 1 --2- 172.21.15.27:0/1599308115 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35fc06ec90 0x7f35fc16a220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f35f400c730 tx=0x7f35f400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f35fad9d640 1 -- 172.21.15.27:0/1599308115 reap_dead start 2024-03-06T07:22:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 shutdown_connections 2024-03-06T07:22:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.805+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 >> 172.21.15.27:0/1599308115 conn(0x7f35fc067120 msgr2=0x7f35fc0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.806+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fa0dc073740 msgr2=0x7fa0dc075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.806+0000 7fa0f61e9640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fa0dc073740 0x7fa0dc075b60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7fbada7fc640 1 -- 172.21.15.27:0/1638672470 <== osd.6 v2:172.21.15.69:6800/288062332 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbae41440e0 con 0x7fbac80957d0 2024-03-06T07:22:54.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 shutdown_connections 2024-03-06T07:22:54.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7f3600caf640 1 -- 172.21.15.27:0/1599308115 wait complete. 2024-03-06T07:22:54.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa0dc06b5a0 msgr2=0x7fa0dc06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7fa0f61e9640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa0dc06b5a0 0x7fa0dc06da60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0001c70 tx=0x7fa0e0009030 comp rx=0 tx=0).stop 2024-03-06T07:22:54.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f01619e0 msgr2=0x7fa0f0162e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7fa0f61e9640 1 --2- 172.21.15.27:0/165352993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0f01619e0 0x7fa0f0162e20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0069700 tx=0x7fa0e400e8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7fa0effff640 1 -- 172.21.15.27:0/165352993 reap_dead start 2024-03-06T07:22:54.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 shutdown_connections 2024-03-06T07:22:54.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.809+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 >> 172.21.15.27:0/165352993 conn(0x7fa0f0066d00 msgr2=0x7fa0f00b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.810+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 shutdown_connections 2024-03-06T07:22:54.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.810+0000 7fa0f61e9640 1 -- 172.21.15.27:0/165352993 wait complete. 2024-03-06T07:22:54.854 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-03-06T07:22:54.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:54.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.816+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f14b0005c90 con 0x7f14b0001650 2024-03-06T07:22:54.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.817+0000 7f14beffd640 1 -- 172.21.15.27:0/547744587 <== osd.0 v2:172.21.15.27:6800/3926946944 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f14b0005c90 con 0x7f14b0001650 2024-03-06T07:22:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.817+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f14b0001650 msgr2=0x7f14b0003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.817+0000 7f14d03d0640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f14b0001650 0x7f14b0003b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.817+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14ac06b4d0 msgr2=0x7f14ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.817+0000 7f14d03d0640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14ac06b4d0 0x7f14ac06d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14c815c8b0 tx=0x7f14b8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.817+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8070f80 msgr2=0x7f14c81ef6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.817+0000 7f14d03d0640 1 --2- 172.21.15.27:0/547744587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c8070f80 0x7f14c81ef6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f14c40083f0 tx=0x7f14c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.817+0000 7f14ce96d640 1 -- 172.21.15.27:0/547744587 reap_dead start 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.818+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 shutdown_connections 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.818+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 >> 172.21.15.27:0/547744587 conn(0x7f14c8066d00 msgr2=0x7f14c8154960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.818+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 shutdown_connections 2024-03-06T07:22:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.818+0000 7f14d03d0640 1 -- 172.21.15.27:0/547744587 wait complete. 2024-03-06T07:22:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.818+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fbac80957d0 msgr2=0x7fbac8097bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.819+0000 7fbaebc8f640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fbac80957d0 0x7fbac8097bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.819+0000 7f5ea0ff9640 1 -- 172.21.15.27:0/2271705604 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e90002dd0 con 0x7f5e8c0737d0 2024-03-06T07:22:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.819+0000 7f5ea2ffd640 1 -- 172.21.15.27:0/2271705604 <== osd.4 v2:172.21.15.69:6808/1626703870 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5e90002dd0 con 0x7f5e8c0737d0 2024-03-06T07:22:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.819+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbac808d5b0 msgr2=0x7fbac808fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.819+0000 7f5ea0ff9640 1 -- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f5e8c0737d0 msgr2=0x7f5e8c075bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.820+0000 7f5ea0ff9640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f5e8c0737d0 0x7f5e8c075bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.820+0000 7f5ea0ff9640 1 -- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e8c06b5d0 msgr2=0x7f5e8c06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.820+0000 7f5ea0ff9640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e8c06b5d0 0x7f5e8c06da90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4009c40 tx=0x7f5ea400b040 comp rx=0 tx=0).stop 2024-03-06T07:22:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.820+0000 7f5ea0ff9640 1 -- 172.21.15.27:0/2271705604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eac06cb70 msgr2=0x7f5eac1ebb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.820+0000 7f5ea0ff9640 1 --2- 172.21.15.27:0/2271705604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5eac06cb70 0x7f5eac1ebb80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400b480 tx=0x7f5e9400b940 comp rx=0 tx=0).stop 2024-03-06T07:22:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.820+0000 7f5eaad9d640 1 -- 172.21.15.27:0/2271705604 reap_dead start 2024-03-06T07:22:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.820+0000 7f5ea0ff9640 1 -- 172.21.15.27:0/2271705604 shutdown_connections 2024-03-06T07:22:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.820+0000 7f5ea0ff9640 1 -- 172.21.15.27:0/2271705604 >> 172.21.15.27:0/2271705604 conn(0x7f5eac066d00 msgr2=0x7f5eac15a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.822+0000 7fbaebc8f640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbac808d5b0 0x7fbac808fa70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbae0035000 tx=0x7fbae0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.822+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae40aee70 msgr2=0x7fbae414aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.822+0000 7fbaebc8f640 1 --2- 172.21.15.27:0/1638672470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae40aee70 0x7fbae414aa20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00c410 tx=0x7fbadc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.822+0000 7fbaea22c640 1 -- 172.21.15.27:0/1638672470 reap_dead start 2024-03-06T07:22:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.822+0000 7f5ea0ff9640 1 -- 172.21.15.27:0/2271705604 shutdown_connections 2024-03-06T07:22:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.822+0000 7f5ea0ff9640 1 -- 172.21.15.27:0/2271705604 wait complete. 2024-03-06T07:22:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.823+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 shutdown_connections 2024-03-06T07:22:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.823+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 >> 172.21.15.27:0/1638672470 conn(0x7fbae4066d00 msgr2=0x7fbae4070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.823+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 shutdown_connections 2024-03-06T07:22:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.823+0000 7fbaebc8f640 1 -- 172.21.15.27:0/1638672470 wait complete. 2024-03-06T07:22:54.863 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-03-06T07:22:54.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:54.864 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-03-06T07:22:54.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:54.865 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-06T07:22:54.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:54.866 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-03-06T07:22:54.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:54.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.840+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe668068700 con 0x7fe6480736d0 2024-03-06T07:22:54.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.840+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0b28072b30 con 0x7f0b0c074350 2024-03-06T07:22:54.868 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-06T07:22:54.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.840+0000 7fe656ffd640 1 -- 172.21.15.27:0/3232679130 <== osd.7 v2:172.21.15.69:6816/1392567864 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe668068700 con 0x7fe6480736d0 2024-03-06T07:22:54.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.840+0000 7f0b257fa640 1 -- 172.21.15.27:0/2170679723 <== osd.1 v2:172.21.15.27:6809/746548704 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0b28072b30 con 0x7f0b0c074350 2024-03-06T07:22:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7fe6480736d0 msgr2=0x7fe648075af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7fe6480736d0 0x7fe648075af0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f0b0c074350 msgr2=0x7f0b0c0767f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f0b0c074350 0x7f0b0c0767f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b0c06b520 msgr2=0x7f0b0c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b0c06b520 0x7f0b0c06d9e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b14036000 tx=0x7f0b1403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe64806b4d0 msgr2=0x7fe64806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe64806b4d0 0x7fe64806d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe6600098a0 tx=0x7fe660006ee0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c9c20 msgr2=0x7fe668163200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 --2- 172.21.15.27:0/3232679130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6680c9c20 0x7fe668163200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00e5c0 tx=0x7fe65c00ea80 comp rx=0 tx=0).stop 2024-03-06T07:22:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b28071ef0 msgr2=0x7f0b2814b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 --2- 172.21.15.27:0/2170679723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b28071ef0 0x7f0b2814b220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c02e420 tx=0x7f0b1c02e8e0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe666d9d640 1 -- 172.21.15.27:0/3232679130 reap_dead start 2024-03-06T07:22:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2d7e4640 1 -- 172.21.15.27:0/2170679723 reap_dead start 2024-03-06T07:22:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 shutdown_connections 2024-03-06T07:22:54.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 >> 172.21.15.27:0/3232679130 conn(0x7fe6680b36d0 msgr2=0x7fe6680d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 shutdown_connections 2024-03-06T07:22:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 shutdown_connections 2024-03-06T07:22:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 >> 172.21.15.27:0/2170679723 conn(0x7f0b280d0990 msgr2=0x7f0b28159230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 shutdown_connections 2024-03-06T07:22:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7fe66cb43640 1 -- 172.21.15.27:0/3232679130 wait complete. 2024-03-06T07:22:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:54.841+0000 7f0b2f247640 1 -- 172.21.15.27:0/2170679723 wait complete. 2024-03-06T07:22:54.875 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-03-06T07:22:54.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:54.876 INFO:teuthology.orchestra.run.smithi027.stdout:55834574853 2024-03-06T07:22:54.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:55.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.091+0000 7fe20040e640 1 Processor -- start 2024-03-06T07:22:55.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.091+0000 7fe20040e640 1 -- start start 2024-03-06T07:22:55.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.092+0000 7ff78cc15640 1 Processor -- start 2024-03-06T07:22:55.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.092+0000 7fd462866640 1 Processor -- start 2024-03-06T07:22:55.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.092+0000 7ff78cc15640 1 -- start start 2024-03-06T07:22:55.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.093+0000 7fe20040e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1f815a420 0x7fe1f80aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.093+0000 7fe1fe1aa640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe1f8067820 0x7fe1f80b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:42382/0 (socket says 172.21.15.27:42382) 2024-03-06T07:22:55.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.092+0000 7fd462866640 1 -- start start 2024-03-06T07:22:55.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.093+0000 7fe1fe1aa640 1 -- 172.21.15.27:0/1011833827 learned_addr learned my addr 172.21.15.27:0/1011833827 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:55.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fe20040e640 1 -- 172.21.15.27:0/1011833827 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f815aae0 con 0x7fe1f8154400 2024-03-06T07:22:55.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7ff78cc15640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7880cf7c0 0x7ff788146f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fd462866640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c165c90 0x7fd45c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7ff78cc15640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff7880cfba0 0x7ff788147440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fe20040e640 1 -- 172.21.15.27:0/1011833827 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f815adc0 con 0x7fe1f8067820 2024-03-06T07:22:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fe20040e640 1 -- 172.21.15.27:0/1011833827 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f80af3b0 con 0x7fe1f815a420 2024-03-06T07:22:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fd45b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c165c90 0x7fd45c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fd45b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c165c90 0x7fd45c06cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47896/0 (socket says 172.21.15.27:47896) 2024-03-06T07:22:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fd45b7fe640 1 -- 172.21.15.27:0/1558964389 learned_addr learned my addr 172.21.15.27:0/1558964389 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fd460e03640 1 --1- 172.21.15.27:0/1558964389 >> v1:172.21.15.69:6789/0 conn(0x7fd45c0ae700 0x7fd45c162b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:42390/0 (socket says 172.21.15.27:42390) 2024-03-06T07:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fd462866640 1 -- 172.21.15.27:0/1558964389 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45c1626a0 con 0x7fd45c15fb80 2024-03-06T07:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fd462866640 1 -- 172.21.15.27:0/1558964389 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45c166170 con 0x7fd45c0ae700 2024-03-06T07:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fd462866640 1 -- 172.21.15.27:0/1558964389 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c072940 con 0x7fd45c165c90 2024-03-06T07:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7ff78cc15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 0x7ff788162030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7ff785d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff7880cfba0 0x7ff788147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711787147 0 0) 0x7fe1f815adc0 con 0x7fe1f8067820 2024-03-06T07:22:55.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e4003520 con 0x7fe1f8067820 2024-03-06T07:22:55.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7f48ab1ac640 1 Processor -- start 2024-03-06T07:22:55.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7ff78cc15640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff788147980 con 0x7ff7880cf7c0 2024-03-06T07:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7ff78cc15640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7880b2a60 con 0x7ff7880cfba0 2024-03-06T07:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7f48ab1ac640 1 -- start start 2024-03-06T07:22:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7ff78cc15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7880b2d40 con 0x7ff788161be0 2024-03-06T07:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7ff785d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff7880cfba0 0x7ff788147440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55312/0 (socket says 172.21.15.27:55312) 2024-03-06T07:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.094+0000 7ff785d9b640 1 -- 172.21.15.27:0/2699512142 learned_addr learned my addr 172.21.15.27:0/2699512142 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.095+0000 7ff786d9d640 1 --2- 172.21.15.27:0/2699512142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 0x7ff788162030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7f2dbd017640 1 Processor -- start 2024-03-06T07:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.095+0000 7fd45b7fe640 1 -- 172.21.15.27:0/1558964389 >> v1:172.21.15.69:6789/0 conn(0x7fd45c0ae700 legacy=0x7fd45c162b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.095+0000 7fd45b7fe640 1 -- 172.21.15.27:0/1558964389 >> v1:172.21.15.27:6789/0 conn(0x7fd45c15fb80 legacy=0x7fd45c06e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.095+0000 7fd45b7fe640 1 -- 172.21.15.27:0/1558964389 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c166450 con 0x7fd45c165c90 2024-03-06T07:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7f48ab1ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4070f80 0x7f48a406c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 <== mon.1 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3968950406 0 0) 0x7fe1e4003520 con 0x7fe1f8067820 2024-03-06T07:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 >> v2:172.21.15.27:3300/0 conn(0x7fe1f815a420 msgr2=0x7fe1f80aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 --2- 172.21.15.27:0/1011833827 >> v2:172.21.15.27:3300/0 conn(0x7fe1f815a420 0x7fe1f80aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 >> v1:172.21.15.27:6789/0 conn(0x7fe1f8154400 legacy=0x7fe1f80cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f815b0a0 con 0x7fe1f8067820 2024-03-06T07:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 <== mon.1 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fe1ec001670 con 0x7fe1f8067820 2024-03-06T07:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 >> v1:172.21.15.69:6789/0 conn(0x7fe1f8067820 legacy=0x7fe1f80b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 --2- 172.21.15.27:0/1011833827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e40039b0 0x7fe1e4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 --2- 172.21.15.27:0/1011833827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1e4004800 0x7fe1e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 --2- 172.21.15.27:0/1011833827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e40051c0 0x7fe1e40145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80af3b0 con 0x7fe1e40039b0 2024-03-06T07:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4014b00 con 0x7fe1e40051c0 2024-03-06T07:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4014c80 con 0x7fe1e4004800 2024-03-06T07:22:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.095+0000 7ff78659c640 1 --2- 172.21.15.27:0/2699512142 >> v2:172.21.15.27:3300/0 conn(0x7ff7880cf7c0 0x7ff788146f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7f48ab1ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f48a40aef90 0x7f48a40b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7ff78659c640 1 -- 172.21.15.27:0/2699512142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 msgr2=0x7ff788162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7f48a3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4070f80 0x7f48a406c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7ff78659c640 1 --2- 172.21.15.27:0/2699512142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 0x7ff788162030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f48a3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4070f80 0x7f48a406c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47912/0 (socket says 172.21.15.27:47912) 2024-03-06T07:22:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7ff78659c640 1 -- 172.21.15.27:0/2699512142 >> v2:172.21.15.69:3300/0 conn(0x7ff7880cfba0 msgr2=0x7ff788147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7ff78659c640 1 --2- 172.21.15.27:0/2699512142 >> v2:172.21.15.69:3300/0 conn(0x7ff7880cfba0 0x7ff788147440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.096+0000 7ff78659c640 1 -- 172.21.15.27:0/2699512142 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7880b3020 con 0x7ff7880cf7c0 2024-03-06T07:22:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f48a3fff640 1 -- 172.21.15.27:0/2023688761 learned_addr learned my addr 172.21.15.27:0/2023688761 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f48ab1ac640 1 -- 172.21.15.27:0/2023688761 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a406ccf0 con 0x7f48a4159d50 2024-03-06T07:22:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f48ab1ac640 1 -- 172.21.15.27:0/2023688761 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a40b28c0 con 0x7f48a40aef90 2024-03-06T07:22:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f48ab1ac640 1 -- 172.21.15.27:0/2023688761 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a406cfd0 con 0x7f48a4070f80 2024-03-06T07:22:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7fd45b7fe640 1 --2- 172.21.15.27:0/1558964389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c165c90 0x7fd45c06cc60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd454009e80 tx=0x7fd45402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf7898860f051f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f2dbd017640 1 -- start start 2024-03-06T07:22:55.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7fe1fe9ab640 1 --2- 172.21.15.27:0/1011833827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e40051c0 0x7fe1e40145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.098+0000 7fe1fe9ab640 1 -- 172.21.15.27:0/1011833827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1e4004800 msgr2=0x7fe1e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.098+0000 7fe1fe9ab640 1 --2- 172.21.15.27:0/1011833827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1e4004800 0x7fe1e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.098+0000 7fe1fe9ab640 1 -- 172.21.15.27:0/1011833827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e40039b0 msgr2=0x7fe1e4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.098+0000 7fe1fe9ab640 1 --2- 172.21.15.27:0/1011833827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1e40039b0 0x7fe1e4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.098+0000 7fe1fe9ab640 1 -- 172.21.15.27:0/1011833827 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f815b0a0 con 0x7fe1e40051c0 2024-03-06T07:22:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f48a3fff640 1 -- 172.21.15.27:0/2023688761 >> v2:172.21.15.27:3300/0 conn(0x7f48a40aef90 msgr2=0x7f48a40b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f48a3fff640 1 --2- 172.21.15.27:0/2023688761 >> v2:172.21.15.27:3300/0 conn(0x7f48a40aef90 0x7f48a40b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f48a3fff640 1 -- 172.21.15.27:0/2023688761 >> v1:172.21.15.27:6789/0 conn(0x7f48a4159d50 legacy=0x7f48a4070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.097+0000 7f48a3fff640 1 -- 172.21.15.27:0/2023688761 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a406d2b0 con 0x7f48a4070f80 2024-03-06T07:22:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.098+0000 7fd45affd640 1 -- 172.21.15.27:0/1558964389 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45402cb00 con 0x7fd45c165c90 2024-03-06T07:22:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.098+0000 7fd45affd640 1 -- 172.21.15.27:0/1558964389 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45402cca0 con 0x7fd45c165c90 2024-03-06T07:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7fd45affd640 1 -- 172.21.15.27:0/1558964389 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454034610 con 0x7fd45c165c90 2024-03-06T07:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7fe1fe9ab640 1 -- 172.21.15.27:0/1011833827 reap_dead start 2024-03-06T07:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7f48a3fff640 1 --2- 172.21.15.27:0/2023688761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4070f80 0x7f48a406c710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4890009870 tx=0x7f489002db20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c405c1d59be7a21 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7f2dbd017640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2db80d3400 0x7f2db80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7f48a37fe640 1 -- 172.21.15.27:0/2023688761 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489002f8e0 con 0x7f48a4070f80 2024-03-06T07:22:55.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7fe1fe9ab640 1 --2- 172.21.15.27:0/1011833827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e40051c0 0x7fe1e40145c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0001d50 tx=0x7fe1e4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3370f946ae37d9f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7f2db6d9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2db8071540 0x7f2db8071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:42394/0 (socket says 172.21.15.27:42394) 2024-03-06T07:22:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7f48a37fe640 1 -- 172.21.15.27:0/2023688761 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489002fa80 con 0x7f48a4070f80 2024-03-06T07:22:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0001920 con 0x7fe1e40051c0 2024-03-06T07:22:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f0035070 con 0x7fe1e40051c0 2024-03-06T07:22:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7fe1fd1a8640 1 -- 172.21.15.27:0/1011833827 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0003bd0 con 0x7fe1e40051c0 2024-03-06T07:22:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.099+0000 7f2db659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2db80d3400 0x7f2db80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f2db659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2db80d3400 0x7f2db80b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51594/0 (socket says 172.21.15.27:51594) 2024-03-06T07:22:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f2db659c640 1 -- 172.21.15.27:0/4187215283 learned_addr learned my addr 172.21.15.27:0/4187215283 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f2dbd017640 1 --2- 172.21.15.27:0/4187215283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db80b2b50 0x7f2db80b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f48a37fe640 1 -- 172.21.15.27:0/2023688761 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489002fe30 con 0x7f48a4070f80 2024-03-06T07:22:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f2dbd017640 1 -- 172.21.15.27:0/4187215283 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db80ce610 con 0x7f2db8071540 2024-03-06T07:22:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7ff78659c640 1 --2- 172.21.15.27:0/2699512142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cf7c0 0x7ff788146f00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff7700097a0 tx=0x7ff7700340e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a848f7c9fbe6b833 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f48ab1ac640 1 -- 172.21.15.27:0/2023688761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4070f80 msgr2=0x7f48a406c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f48ab1ac640 1 --2- 172.21.15.27:0/2023688761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4070f80 0x7f48a406c710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4890009870 tx=0x7f489002db20 comp rx=0 tx=0).stop 2024-03-06T07:22:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f48ab1ac640 1 -- 172.21.15.27:0/2023688761 shutdown_connections 2024-03-06T07:22:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f48ab1ac640 1 --2- 172.21.15.27:0/2023688761 >> v2:172.21.15.27:3300/0 conn(0x7f48a40aef90 0x7f48a40b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7fe20040e640 1 -- 172.21.15.27:0/1011833827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e40051c0 msgr2=0x7fe1e40145c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7fe20040e640 1 --2- 172.21.15.27:0/1011833827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e40051c0 0x7fe1e40145c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0001d50 tx=0x7fe1e4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f48ab1ac640 1 --2- 172.21.15.27:0/2023688761 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4070f80 0x7f48a406c710 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f48ab1ac640 1 -- 172.21.15.27:0/2023688761 >> 172.21.15.27:0/2023688761 conn(0x7f48a4066d00 msgr2=0x7f48a40670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f2db759e640 1 --2- 172.21.15.27:0/4187215283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db80b2b50 0x7f2db80b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f48ab1ac640 1 -- 172.21.15.27:0/2023688761 shutdown_connections 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.100+0000 7f48ab1ac640 1 -- 172.21.15.27:0/2023688761 wait complete. 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7f2dbd017640 1 -- 172.21.15.27:0/4187215283 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db80d2f70 con 0x7f2db80d3400 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7f2dbd017640 1 -- 172.21.15.27:0/4187215283 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db80d4be0 con 0x7f2db80b2b50 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7fe20040e640 1 -- 172.21.15.27:0/1011833827 shutdown_connections 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7fe20040e640 1 --2- 172.21.15.27:0/1011833827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e40051c0 0x7fe1e40145c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7fe20040e640 1 -- 172.21.15.27:0/1011833827 >> 172.21.15.27:0/1011833827 conn(0x7fe1f8066d00 msgr2=0x7fe1f8067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7f2db5d9b640 1 -- 172.21.15.27:0/4187215283 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515589317 0 0) 0x7f2db80ce610 con 0x7f2db8071540 2024-03-06T07:22:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7f2db5d9b640 1 -- 172.21.15.27:0/4187215283 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da0003520 con 0x7f2db8071540 2024-03-06T07:22:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7ff78559a640 1 -- 172.21.15.27:0/2699512142 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff770041070 con 0x7ff7880cf7c0 2024-03-06T07:22:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7ff78559a640 1 -- 172.21.15.27:0/2699512142 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77003c440 con 0x7ff7880cf7c0 2024-03-06T07:22:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7ff78559a640 1 -- 172.21.15.27:0/2699512142 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77003c850 con 0x7ff7880cf7c0 2024-03-06T07:22:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7fe20040e640 1 -- 172.21.15.27:0/1011833827 shutdown_connections 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.102+0000 7f2db659c640 1 -- 172.21.15.27:0/4187215283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db80b2b50 msgr2=0x7f2db80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.102+0000 7f2db659c640 1 --2- 172.21.15.27:0/4187215283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db80b2b50 0x7f2db80b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.102+0000 7f2db659c640 1 -- 172.21.15.27:0/4187215283 >> v1:172.21.15.69:6789/0 conn(0x7f2db8071540 legacy=0x7f2db8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.102+0000 7f2db659c640 1 -- 172.21.15.27:0/4187215283 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db80d2920 con 0x7f2db80d3400 2024-03-06T07:22:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.102+0000 7ff78cc15640 1 -- 172.21.15.27:0/2699512142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cf7c0 msgr2=0x7ff788146f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.102+0000 7ff78cc15640 1 --2- 172.21.15.27:0/2699512142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cf7c0 0x7ff788146f00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff7700097a0 tx=0x7ff7700340e0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.102+0000 7f48ab1ac640 1 Processor -- start 2024-03-06T07:22:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.102+0000 7f48ab1ac640 1 -- start start 2024-03-06T07:22:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.101+0000 7fe20040e640 1 -- 172.21.15.27:0/1011833827 wait complete. 2024-03-06T07:22:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7fe20040e640 1 Processor -- start 2024-03-06T07:22:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7ff78cc15640 1 -- 172.21.15.27:0/2699512142 shutdown_connections 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7ff78cc15640 1 --2- 172.21.15.27:0/2699512142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 0x7ff788162030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7ff78cc15640 1 --2- 172.21.15.27:0/2699512142 >> v2:172.21.15.69:3300/0 conn(0x7ff7880cfba0 0x7ff788147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7ff78cc15640 1 --2- 172.21.15.27:0/2699512142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cf7c0 0x7ff788146f00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7ff78cc15640 1 -- 172.21.15.27:0/2699512142 >> 172.21.15.27:0/2699512142 conn(0x7ff788066d00 msgr2=0x7ff788067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7f7a81be2640 1 Processor -- start 2024-03-06T07:22:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7f7a81be2640 1 -- start start 2024-03-06T07:22:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7f2db659c640 1 --2- 172.21.15.27:0/4187215283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db80d3400 0x7f2db80b30a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009870 tx=0x7f2dac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6688683f70c85246 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7fd462866640 1 -- 172.21.15.27:0/1558964389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c165c90 msgr2=0x7fd45c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7fd462866640 1 --2- 172.21.15.27:0/1558964389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c165c90 0x7fd45c06cc60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd454009e80 tx=0x7fd45402bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.103+0000 7fe20040e640 1 -- start start 2024-03-06T07:22:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7ff78cc15640 1 -- 172.21.15.27:0/2699512142 shutdown_connections 2024-03-06T07:22:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7ff78cc15640 1 -- 172.21.15.27:0/2699512142 wait complete. 2024-03-06T07:22:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48ab1ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4070f80 0x7f48a415b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f2db5d9b640 1 -- 172.21.15.27:0/4187215283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac035020 con 0x7f2db80d3400 2024-03-06T07:22:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f2db5d9b640 1 -- 172.21.15.27:0/4187215283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac031640 con 0x7f2db80d3400 2024-03-06T07:22:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f2db5d9b640 1 -- 172.21.15.27:0/4187215283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac02d040 con 0x7f2db80d3400 2024-03-06T07:22:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48ab1ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a40aef90 0x7f48a41f8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48ab1ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4159d50 0x7f48a414b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7fe20040e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1e40051c0 0x7fe1f8154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48ab1ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a406e420 con 0x7f48a4070f80 2024-03-06T07:22:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7fe20040e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8142c30 0x7fe1f8159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48ab1ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a415a130 con 0x7f48a40aef90 2024-03-06T07:22:55.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7fe1fe1aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1e40051c0 0x7fe1f8154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7fe1fe1aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1e40051c0 0x7fe1f8154990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47946/0 (socket says 172.21.15.27:47946) 2024-03-06T07:22:55.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48ab1ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a406e960 con 0x7f48a4159d50 2024-03-06T07:22:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7fe1fe1aa640 1 -- 172.21.15.27:0/3369254218 learned_addr learned my addr 172.21.15.27:0/3369254218 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7fe20040e640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f81f2e00 0x7fe1f81f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48a8f48640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4070f80 0x7f48a415b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f7a81be2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a7c15a420 0x7f7a7c0aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f7a81be2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7a7c154190 0x7f7a7c1545a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a81be2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7c15adc0 con 0x7f7a7c067820 2024-03-06T07:22:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f8154810 con 0x7fe1f8142c30 2024-03-06T07:22:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f806c7b0 con 0x7fe1f81f2e00 2024-03-06T07:22:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80aee70 con 0x7fe1e40051c0 2024-03-06T07:22:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a81be2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7c0af3b0 con 0x7f7a7c154190 2024-03-06T07:22:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a81be2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7c0b32e0 con 0x7f7a7c15a420 2024-03-06T07:22:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe1fe9ab640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f81f2e00 0x7fe1f81f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe1fd9a9640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8142c30 0x7fe1f8159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48a8f48640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4070f80 0x7f48a415b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51596/0 (socket says 172.21.15.27:51596) 2024-03-06T07:22:55.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a7affd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a7c15a420 0x7f7a7c0aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a7affd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a7c15a420 0x7f7a7c0aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55342/0 (socket says 172.21.15.27:55342) 2024-03-06T07:22:55.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a7affd640 1 -- 172.21.15.27:0/404585668 learned_addr learned my addr 172.21.15.27:0/404585668 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a7bfff640 1 --2- 172.21.15.27:0/404585668 >> v2:172.21.15.27:3300/0 conn(0x7f7a7c154190 0x7f7a7c1545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48a8f48640 1 -- 172.21.15.27:0/1598244654 learned_addr learned my addr 172.21.15.27:0/1598244654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.104+0000 7f48a9749640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4159d50 0x7f48a414b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f2dbd017640 1 -- 172.21.15.27:0/4187215283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db80d3400 msgr2=0x7f2db80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f2dbd017640 1 --2- 172.21.15.27:0/4187215283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db80d3400 0x7f2db80b30a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009870 tx=0x7f2dac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f2dbd017640 1 -- 172.21.15.27:0/4187215283 shutdown_connections 2024-03-06T07:22:55.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f2dbd017640 1 --2- 172.21.15.27:0/4187215283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db80d3400 0x7f2db80b30a0 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009870 tx=0x7f2dac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f48a3fff640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a40aef90 0x7f48a41f8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f48a8f48640 1 -- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4159d50 msgr2=0x7f48a414b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f48a8f48640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4159d50 0x7f48a414b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f48a8f48640 1 -- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a40aef90 msgr2=0x7f48a41f8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f48a8f48640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a40aef90 0x7f48a41f8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f48a8f48640 1 -- 172.21.15.27:0/1598244654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a40cd820 con 0x7f48a4070f80 2024-03-06T07:22:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f48a3fff640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a40aef90 0x7f48a41f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:55.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a7affd640 1 -- 172.21.15.27:0/404585668 >> v2:172.21.15.27:3300/0 conn(0x7f7a7c154190 msgr2=0x7f7a7c1545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a7affd640 1 --2- 172.21.15.27:0/404585668 >> v2:172.21.15.27:3300/0 conn(0x7f7a7c154190 0x7f7a7c1545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a7affd640 1 -- 172.21.15.27:0/404585668 >> v1:172.21.15.69:6789/0 conn(0x7f7a7c067820 legacy=0x7f7a7c0b2bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:55.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f7a7affd640 1 -- 172.21.15.27:0/404585668 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c15b0a0 con 0x7f7a7c15a420 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7f2dbd017640 1 --2- 172.21.15.27:0/4187215283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db80b2b50 0x7f2db80b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7f2dbd017640 1 -- 172.21.15.27:0/4187215283 >> 172.21.15.27:0/4187215283 conn(0x7f2db80d3ca0 msgr2=0x7f2db80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe1fd9a9640 1 -- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1e40051c0 msgr2=0x7fe1f8154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe1fd9a9640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1e40051c0 0x7fe1f8154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe1fd9a9640 1 -- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f81f2e00 msgr2=0x7fe1f81f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe1fd9a9640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f81f2e00 0x7fe1f81f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7fd462866640 1 -- 172.21.15.27:0/1558964389 shutdown_connections 2024-03-06T07:22:55.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7fd462866640 1 --2- 172.21.15.27:0/1558964389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c165c90 0x7fd45c06cc60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7fd462866640 1 -- 172.21.15.27:0/1558964389 >> 172.21.15.27:0/1558964389 conn(0x7fd45c066d00 msgr2=0x7fd45c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.105+0000 7fe1fd9a9640 1 -- 172.21.15.27:0/3369254218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8155070 con 0x7fe1f8142c30 2024-03-06T07:22:55.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7fe1fd9a9640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8142c30 0x7fe1f8159310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4006fd0 tx=0x7fe1f400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7f48a8f48640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4070f80 0x7f48a415b4a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f489800b410 tx=0x7f489800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7ff78cc15640 1 Processor -- start 2024-03-06T07:22:55.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7ff78cc15640 1 -- start start 2024-03-06T07:22:55.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7fe1eaffd640 1 -- 172.21.15.27:0/3369254218 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4011830 con 0x7fe1f8142c30 2024-03-06T07:22:55.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f8155350 con 0x7fe1f8142c30 2024-03-06T07:22:55.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.106+0000 7fd462866640 1 -- 172.21.15.27:0/1558964389 shutdown_connections 2024-03-06T07:22:55.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.107+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f8155910 con 0x7fe1f8142c30 2024-03-06T07:22:55.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.107+0000 7fe1eaffd640 1 -- 172.21.15.27:0/3369254218 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f4016070 con 0x7fe1f8142c30 2024-03-06T07:22:55.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.107+0000 7f48a17fa640 1 -- 172.21.15.27:0/1598244654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4898015020 con 0x7f48a4070f80 2024-03-06T07:22:55.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.107+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a40cdb00 con 0x7f48a4070f80 2024-03-06T07:22:55.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.107+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a40ce0c0 con 0x7f48a4070f80 2024-03-06T07:22:55.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.107+0000 7f48a17fa640 1 -- 172.21.15.27:0/1598244654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4898011830 con 0x7f48a4070f80 2024-03-06T07:22:55.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.107+0000 7f48a17fa640 1 -- 172.21.15.27:0/1598244654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4898011c20 con 0x7f48a4070f80 2024-03-06T07:22:55.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7fe1eaffd640 1 -- 172.21.15.27:0/3369254218 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4011c10 con 0x7fe1f8142c30 2024-03-06T07:22:55.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7f2dbd017640 1 -- 172.21.15.27:0/4187215283 shutdown_connections 2024-03-06T07:22:55.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.107+0000 7fd462866640 1 -- 172.21.15.27:0/1558964389 wait complete. 2024-03-06T07:22:55.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7ff78cc15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7880cf7c0 0x7ff78806d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7ff78cc15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cfba0 0x7ff78813a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7ff78cc15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 0x7ff788143100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7ff78cc15640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff788162570 con 0x7ff7880cfba0 2024-03-06T07:22:55.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7ff78cc15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7880b3820 con 0x7ff7880cf7c0 2024-03-06T07:22:55.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7ff78cc15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7881626f0 con 0x7ff788161be0 2024-03-06T07:22:55.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7fd462866640 1 Processor -- start 2024-03-06T07:22:55.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.108+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f81f5980 con 0x7fe1f8142c30 2024-03-06T07:22:55.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.109+0000 7f2dbd017640 1 -- 172.21.15.27:0/4187215283 wait complete. 2024-03-06T07:22:55.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4884005290 con 0x7f48a4070f80 2024-03-06T07:22:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.109+0000 7fd462866640 1 -- start start 2024-03-06T07:22:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7ff786d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 0x7ff788143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7ff786d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 0x7ff788143100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47958/0 (socket says 172.21.15.27:47958) 2024-03-06T07:22:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7ff786d9d640 1 -- 172.21.15.27:0/1103067318 learned_addr learned my addr 172.21.15.27:0/1103067318 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7fd462866640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c0ae700 0x7fd45c153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7f48a17fa640 1 -- 172.21.15.27:0/1598244654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4898018020 con 0x7f48a4070f80 2024-03-06T07:22:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7ff78659c640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7880cf7c0 0x7ff78806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7ff785d9b640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cfba0 0x7ff78813a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7fd462866640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd45c15fb80 0x7fd45c0ad360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7fd462866640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c165c90 0x7fd45c0b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7fd462866640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c06d1a0 con 0x7fd45c165c90 2024-03-06T07:22:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7fd462866640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c166a10 con 0x7fd45c15fb80 2024-03-06T07:22:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7fd462866640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c06d320 con 0x7fd45c0ae700 2024-03-06T07:22:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7fd460e03640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c165c90 0x7fd45c0b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7ff785d9b640 1 -- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 msgr2=0x7ff788143100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7ff785d9b640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 0x7ff788143100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7ff785d9b640 1 -- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7880cf7c0 msgr2=0x7ff78806d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7ff785d9b640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7880cf7c0 0x7ff78806d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7ff785d9b640 1 -- 172.21.15.27:0/1103067318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78806da00 con 0x7ff7880cfba0 2024-03-06T07:22:55.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7fd460e03640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c165c90 0x7fd45c0b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51636/0 (socket says 172.21.15.27:51636) 2024-03-06T07:22:55.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7ff785d9b640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cfba0 0x7ff78813a130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff77800efc0 tx=0x7ff77800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7f2dbd017640 1 Processor -- start 2024-03-06T07:22:55.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7fd460e03640 1 -- 172.21.15.27:0/325753643 learned_addr learned my addr 172.21.15.27:0/325753643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7fd460e03640 1 -- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c0ae700 msgr2=0x7fd45c153ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:55.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7fe1eaffd640 1 -- 172.21.15.27:0/3369254218 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1f401b460 con 0x7fe1f8142c30 2024-03-06T07:22:55.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.110+0000 7f7a7affd640 1 --2- 172.21.15.27:0/404585668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7c15a420 0x7f7a7c0aee70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c072d00 tx=0x7f7a6c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0fc8960a34f7e40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7f7a7a7fc640 1 -- 172.21.15.27:0/404585668 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c035020 con 0x7f7a7c15a420 2024-03-06T07:22:55.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7f7a7a7fc640 1 -- 172.21.15.27:0/404585668 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c031870 con 0x7f7a7c15a420 2024-03-06T07:22:55.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.111+0000 7f7a7a7fc640 1 -- 172.21.15.27:0/404585668 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c031c40 con 0x7f7a7c15a420 2024-03-06T07:22:55.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f7a81be2640 1 -- 172.21.15.27:0/404585668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7c15a420 msgr2=0x7f7a7c0aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f7a81be2640 1 --2- 172.21.15.27:0/404585668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7c15a420 0x7f7a7c0aee70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c072d00 tx=0x7f7a6c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:55.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7fd460e03640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c0ae700 0x7fd45c153ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f7a81be2640 1 -- 172.21.15.27:0/404585668 shutdown_connections 2024-03-06T07:22:55.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f7a81be2640 1 --2- 172.21.15.27:0/404585668 >> v2:172.21.15.27:3300/0 conn(0x7f7a7c154190 0x7f7a7c1545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f7a81be2640 1 --2- 172.21.15.27:0/404585668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7c15a420 0x7f7a7c0aee70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f7a81be2640 1 -- 172.21.15.27:0/404585668 >> 172.21.15.27:0/404585668 conn(0x7f7a7c066d00 msgr2=0x7f7a7c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7fe1eaffd640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1dc06b5a0 0x7fe1dc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f7a81be2640 1 -- 172.21.15.27:0/404585668 shutdown_connections 2024-03-06T07:22:55.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f48a17fa640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f488806b5a0 0x7f488806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7fd460e03640 1 -- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd45c15fb80 msgr2=0x7fd45c0ad360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:55.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7fd460e03640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd45c15fb80 0x7fd45c0ad360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7fd460e03640 1 -- 172.21.15.27:0/325753643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c0a96f0 con 0x7fd45c165c90 2024-03-06T07:22:55.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7fd460e03640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c165c90 0x7fd45c0b36d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd45000c3f0 tx=0x7fd45000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7fe1eaffd640 1 -- 172.21.15.27:0/3369254218 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1f4089f20 con 0x7fe1f8142c30 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f48a3fff640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f488806b5a0 0x7f488806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7fe1fe1aa640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1dc06b5a0 0x7fe1dc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f48a17fa640 1 -- 172.21.15.27:0/1598244654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f489808a160 con 0x7f48a4070f80 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f7a81be2640 1 -- 172.21.15.27:0/404585668 wait complete. 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f48a3fff640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f488806b5a0 0x7f488806da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4890038000 tx=0x7f4890008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.112+0000 7f2dbd017640 1 -- start start 2024-03-06T07:22:55.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7fe1fe1aa640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1dc06b5a0 0x7fe1dc06da60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec004be0 tx=0x7fe1ec001ad0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2dbd017640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db8071540 0x7f2db81f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2dbd017640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2db80b2b50 0x7f2db80c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2dbd017640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db81fb230 0x7f2db81fb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2dbd017640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db815e370 con 0x7f2db81fb230 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2dbd017640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db80ce8f0 con 0x7f2db80b2b50 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2dbd017640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db81f2160 con 0x7f2db8071540 2024-03-06T07:22:55.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2db659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2db80b2b50 0x7f2db80c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2db759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db81fb230 0x7f2db81fb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2db6d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db8071540 0x7f2db81f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.114+0000 7f7a81be2640 1 Processor -- start 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2db6d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db8071540 0x7f2db81f1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47974/0 (socket says 172.21.15.27:47974) 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.113+0000 7f2db6d9d640 1 -- 172.21.15.27:0/2075198627 learned_addr learned my addr 172.21.15.27:0/2075198627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.114+0000 7f2db659c640 1 -- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db8071540 msgr2=0x7f2db81f1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.114+0000 7f2db659c640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db8071540 0x7f2db81f1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.114+0000 7f2db659c640 1 -- 172.21.15.27:0/2075198627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db81fb230 msgr2=0x7f2db81fb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.114+0000 7f2db659c640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db81fb230 0x7f2db81fb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.114+0000 7f2db659c640 1 -- 172.21.15.27:0/2075198627 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db80c7fc0 con 0x7f2db80b2b50 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.114+0000 7f7a81be2640 1 -- start start 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f7a81be2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a7c067820 0x7f7a7c0a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f7a81be2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7c154190 0x7f7a7c13b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f7a7b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a7c067820 0x7f7a7c0a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f7a7b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a7c067820 0x7f7a7c0a9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51656/0 (socket says 172.21.15.27:51656) 2024-03-06T07:22:55.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f2db659c640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2db80b2b50 0x7f2db80c7670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2dac033000 tx=0x7f2dac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f7a7b7fe640 1 -- 172.21.15.27:0/481433294 learned_addr learned my addr 172.21.15.27:0/481433294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f7a81be2640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7c15a420 0x7f7a7c158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f7a81be2640 1 -- 172.21.15.27:0/481433294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0cf820 con 0x7f7a7c067820 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f2d9f7fe640 1 -- 172.21.15.27:0/2075198627 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac040070 con 0x7f2db80b2b50 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db80c82a0 con 0x7f2db80b2b50 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.115+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db80c8860 con 0x7f2db80b2b50 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.116+0000 7f2d9f7fe640 1 -- 172.21.15.27:0/2075198627 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac0399f0 con 0x7f2db80b2b50 2024-03-06T07:22:55.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.116+0000 7f2d9f7fe640 1 -- 172.21.15.27:0/2075198627 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac035040 con 0x7f2db80b2b50 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.116+0000 7f7a81be2640 1 -- 172.21.15.27:0/481433294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c06c7b0 con 0x7f7a7c15a420 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.116+0000 7f7a81be2640 1 -- 172.21.15.27:0/481433294 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0cf9a0 con 0x7f7a7c154190 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.116+0000 7f7a7bfff640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7c15a420 0x7f7a7c158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.116+0000 7f7a7affd640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7c154190 0x7f7a7c13b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.117+0000 7f7a7b7fe640 1 -- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7c154190 msgr2=0x7f7a7c13b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.117+0000 7f7a7b7fe640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7c154190 0x7f7a7c13b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.117+0000 7f7a7b7fe640 1 -- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7c15a420 msgr2=0x7f7a7c158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.117+0000 7f7a7b7fe640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7c15a420 0x7f7a7c158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.117+0000 7f7a7b7fe640 1 -- 172.21.15.27:0/481433294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c13bc50 con 0x7f7a7c067820 2024-03-06T07:22:55.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.117+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db8068700 con 0x7f2db80b2b50 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.117+0000 7f7a7b7fe640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a7c067820 0x7f7a7c0a9e60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400ba80 tx=0x7f7a7400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.118+0000 7f2d9f7fe640 1 -- 172.21.15.27:0/2075198627 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dac04b060 con 0x7f2db80b2b50 2024-03-06T07:22:55.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.118+0000 7f8c91355640 1 Processor -- start 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.118+0000 7f8c91355640 1 -- start start 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.118+0000 7f2d9f7fe640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2da406b5a0 0x7f2da406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.118+0000 7f2d9f7fe640 1 -- 172.21.15.27:0/2075198627 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dac0ab9a0 con 0x7f2db80b2b50 2024-03-06T07:22:55.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.119+0000 7f2db6d9d640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2da406b5a0 0x7f2da406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.119+0000 7f8c91355640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c8c0cda40 0x7f8c8c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.119+0000 7f8c91355640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8c067b40 con 0x7f8c8c06c040 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.119+0000 7f8c91355640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8c0d0070 con 0x7f8c8c06b130 2024-03-06T07:22:55.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.119+0000 7f2db6d9d640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2da406b5a0 0x7f2da406da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2da8007630 tx=0x7f2da8007e10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.119+0000 7f8c91355640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8c0cfe40 con 0x7f8c8c0cda40 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.119+0000 7f8c8affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c8c0cda40 0x7f8c8c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.119+0000 7f8c8b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8c8c06c040 0x7f8c8c15d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50066/0 (socket says 172.21.15.27:50066) 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.119+0000 7f8c8b7fe640 1 -- 172.21.15.27:0/2948623033 learned_addr learned my addr 172.21.15.27:0/2948623033 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.120+0000 7f8c8affd640 1 -- 172.21.15.27:0/2948623033 >> v1:172.21.15.69:6789/0 conn(0x7f8c8c06b130 legacy=0x7f8c8c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.120+0000 7f8c8affd640 1 -- 172.21.15.27:0/2948623033 >> v1:172.21.15.27:6789/0 conn(0x7f8c8c06c040 legacy=0x7f8c8c15d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.120+0000 7f8c8affd640 1 -- 172.21.15.27:0/2948623033 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c0d0310 con 0x7f8c8c0cda40 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.120+0000 7f8c8affd640 1 --2- 172.21.15.27:0/2948623033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c0cda40 0x7f8c8c0ced50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0097a0 tx=0x7f8c7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a73fc72ffb7359f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.121+0000 7fe1eaffd640 1 -- 172.21.15.27:0/3369254218 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1f81f5980 con 0x7fe1f8142c30 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.123+0000 7f2d9f7fe640 1 -- 172.21.15.27:0/2075198627 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dac07a800 con 0x7f2db80b2b50 2024-03-06T07:22:55.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7ff77effd640 1 -- 172.21.15.27:0/1103067318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77800cda0 con 0x7ff7880cfba0 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7ff77effd640 1 -- 172.21.15.27:0/1103067318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff778019070 con 0x7ff7880cfba0 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7ff77effd640 1 -- 172.21.15.27:0/1103067318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77800cda0 con 0x7ff7880cfba0 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7880af5b0 con 0x7ff7880cfba0 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78806d640 con 0x7ff7880cfba0 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7fd458ff9640 1 -- 172.21.15.27:0/325753643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4500074e0 con 0x7fd45c165c90 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7fd458ff9640 1 -- 172.21.15.27:0/325753643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd450007680 con 0x7fd45c165c90 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7fd458ff9640 1 -- 172.21.15.27:0/325753643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4500049e0 con 0x7fd45c165c90 2024-03-06T07:22:55.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45c0a99d0 con 0x7fd45c165c90 2024-03-06T07:22:55.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45c0a9f90 con 0x7fd45c165c90 2024-03-06T07:22:55.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7f7a63fff640 1 -- 172.21.15.27:0/481433294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74019070 con 0x7f7a7c067820 2024-03-06T07:22:55.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7f7a63fff640 1 -- 172.21.15.27:0/481433294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7400cc90 con 0x7f7a7c067820 2024-03-06T07:22:55.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7f7a63fff640 1 -- 172.21.15.27:0/481433294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74014680 con 0x7f7a7c067820 2024-03-06T07:22:55.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.131+0000 7f8c8a7fc640 1 -- 172.21.15.27:0/2948623033 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c035020 con 0x7f8c8c0cda40 2024-03-06T07:22:55.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.131+0000 7f8c8a7fc640 1 -- 172.21.15.27:0/2948623033 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c031830 con 0x7f8c8c0cda40 2024-03-06T07:22:55.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.131+0000 7f8c8a7fc640 1 -- 172.21.15.27:0/2948623033 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c031c00 con 0x7f8c8c0cda40 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.131+0000 7f7a81be2640 1 -- 172.21.15.27:0/481433294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c13bf30 con 0x7f7a7c067820 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.130+0000 7f48a17fa640 1 -- 172.21.15.27:0/1598244654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f489801f020 con 0x7f48a4070f80 2024-03-06T07:22:55.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.131+0000 7f7a81be2640 1 -- 172.21.15.27:0/481433294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7c13c4f0 con 0x7f7a7c067820 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.132+0000 7f8c91355640 1 -- 172.21.15.27:0/2948623033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c0cda40 msgr2=0x7f8c8c0ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.132+0000 7f8c91355640 1 --2- 172.21.15.27:0/2948623033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c0cda40 0x7f8c8c0ced50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0097a0 tx=0x7f8c7c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.132+0000 7f8c91355640 1 -- 172.21.15.27:0/2948623033 shutdown_connections 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.132+0000 7f8c91355640 1 --2- 172.21.15.27:0/2948623033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c0cda40 0x7f8c8c0ced50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.132+0000 7f8c91355640 1 -- 172.21.15.27:0/2948623033 >> 172.21.15.27:0/2948623033 conn(0x7f8c8c0d1a40 msgr2=0x7f8c8c0d1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.133+0000 7f8c91355640 1 -- 172.21.15.27:0/2948623033 shutdown_connections 2024-03-06T07:22:55.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.133+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff788067ff0 con 0x7ff7880cfba0 2024-03-06T07:22:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.133+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45c04f330 con 0x7fd45c165c90 2024-03-06T07:22:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.133+0000 7f8c91355640 1 -- 172.21.15.27:0/2948623033 wait complete. 2024-03-06T07:22:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.135+0000 7f8c91355640 1 Processor -- start 2024-03-06T07:22:55.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.135+0000 7f8c91355640 1 -- start start 2024-03-06T07:22:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.135+0000 7fd458ff9640 1 -- 172.21.15.27:0/325753643 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd45001a020 con 0x7fd45c165c90 2024-03-06T07:22:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.135+0000 7fd458ff9640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd44806b520 0x7fd44806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.135+0000 7fd458ff9640 1 -- 172.21.15.27:0/325753643 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4500569a0 con 0x7fd45c165c90 2024-03-06T07:22:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.136+0000 7f7a63fff640 1 -- 172.21.15.27:0/481433294 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a7401c070 con 0x7f7a7c067820 2024-03-06T07:22:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.137+0000 7f7a63fff640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a6806b5a0 0x7f7a6806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.137+0000 7f7a63fff640 1 -- 172.21.15.27:0/481433294 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a7408a020 con 0x7f7a7c067820 2024-03-06T07:22:55.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.137+0000 7f8c91355640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8c06b130 0x7f8c8c151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.137+0000 7fd45bfff640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd44806b520 0x7fd44806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.137+0000 7f8c91355640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c06c040 0x7f8c8c0c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.137+0000 7f8c91355640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c8c0cda40 0x7f8c8c161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7ff77effd640 1 -- 172.21.15.27:0/1103067318 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff778005d10 con 0x7ff7880cfba0 2024-03-06T07:22:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7f8c83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c06c040 0x7f8c8c0c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7f8c83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c06c040 0x7f8c8c0c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51672/0 (socket says 172.21.15.27:51672) 2024-03-06T07:22:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.137+0000 7f8c8affd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8c06b130 0x7f8c8c151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7fd45bfff640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd44806b520 0x7fd44806d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd45c164fd0 tx=0x7fd44c007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7f8c8b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c8c0cda40 0x7f8c8c161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7f8c8b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c8c0cda40 0x7f8c8c161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55394/0 (socket says 172.21.15.27:55394) 2024-03-06T07:22:55.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7ff77effd640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff76806b8d0 0x7ff76806dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7f8c83fff640 1 -- 172.21.15.27:0/1224071155 learned_addr learned my addr 172.21.15.27:0/1224071155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7f8c91355640 1 -- 172.21.15.27:0/1224071155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c0cd0a0 con 0x7f8c8c06c040 2024-03-06T07:22:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7f8c91355640 1 -- 172.21.15.27:0/1224071155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c0d0c40 con 0x7f8c8c0cda40 2024-03-06T07:22:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7f8c91355640 1 -- 172.21.15.27:0/1224071155 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c0d08d0 con 0x7f8c8c06b130 2024-03-06T07:22:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7ff77effd640 1 -- 172.21.15.27:0/1103067318 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff77800cba0 con 0x7ff7880cfba0 2024-03-06T07:22:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.139+0000 7ff78659c640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff76806b8d0 0x7ff76806dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.139+0000 7f8c83fff640 1 -- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8c06b130 msgr2=0x7f8c8c151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.139+0000 7f8c83fff640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8c06b130 0x7f8c8c151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.139+0000 7f8c83fff640 1 -- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c8c0cda40 msgr2=0x7f8c8c161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.139+0000 7f8c83fff640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c8c0cda40 0x7f8c8c161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.139+0000 7f8c83fff640 1 -- 172.21.15.27:0/1224071155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c1515a0 con 0x7f8c8c06c040 2024-03-06T07:22:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.139+0000 7f8c83fff640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c06c040 0x7f8c8c0c7bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400c400 tx=0x7f8c7400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.140+0000 7ff78659c640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff76806b8d0 0x7ff76806dd90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff77003e000 tx=0x7ff770038000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.138+0000 7f7a7affd640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a6806b5a0 0x7f7a6806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.139+0000 7f7a7affd640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a6806b5a0 0x7f7a6806da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c036000 tx=0x7f7a6c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.144+0000 7fd458ff9640 1 -- 172.21.15.27:0/325753643 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd450021070 con 0x7fd45c165c90 2024-03-06T07:22:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.153+0000 7ff77effd640 1 -- 172.21.15.27:0/1103067318 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff778010040 con 0x7ff7880cfba0 2024-03-06T07:22:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.160+0000 7f8c837fe640 1 -- 172.21.15.27:0/1224071155 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c740052d0 con 0x7f8c8c06c040 2024-03-06T07:22:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.160+0000 7f8c837fe640 1 -- 172.21.15.27:0/1224071155 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c74005470 con 0x7f8c8c06c040 2024-03-06T07:22:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.160+0000 7f8c837fe640 1 -- 172.21.15.27:0/1224071155 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c74007220 con 0x7f8c8c06c040 2024-03-06T07:22:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.160+0000 7f8c91355640 1 -- 172.21.15.27:0/1224071155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8c0b4a40 con 0x7f8c8c06c040 2024-03-06T07:22:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.160+0000 7f8c91355640 1 -- 172.21.15.27:0/1224071155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8c0b5000 con 0x7f8c8c06c040 2024-03-06T07:22:55.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.163+0000 7f7a61ffb640 1 -- 172.21.15.27:0/481433294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7c0b10c0 con 0x7f7a7c067820 2024-03-06T07:22:55.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.163+0000 7f8c817fa640 1 -- 172.21.15.27:0/1224071155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8c068700 con 0x7f8c8c06c040 2024-03-06T07:22:55.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.164+0000 7f8c837fe640 1 -- 172.21.15.27:0/1224071155 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c740073c0 con 0x7f8c8c06c040 2024-03-06T07:22:55.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.164+0000 7f8c837fe640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6c06b5a0 0x7f8c6c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.165+0000 7f8c837fe640 1 -- 172.21.15.27:0/1224071155 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c740896c0 con 0x7f8c8c06c040 2024-03-06T07:22:55.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.165+0000 7f8c8affd640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6c06b5a0 0x7f8c6c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.166+0000 7f8c8affd640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6c06b5a0 0x7f8c6c06da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c036000 tx=0x7f8c7c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.174+0000 7f7a63fff640 1 -- 172.21.15.27:0/481433294 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a7c0b10c0 con 0x7f7a7c067820 2024-03-06T07:22:55.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.174+0000 7f679f59e640 1 Processor -- start 2024-03-06T07:22:55.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.174+0000 7f679f59e640 1 -- start start 2024-03-06T07:22:55.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679f59e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67980e0010 0x7f679800b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679f59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 0x7f679803a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679f59e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679800cb20 con 0x7f679803adf0 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679f59e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679800b700 con 0x7f67980e0010 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679f59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67980565c0 con 0x7f6798008f40 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679e59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67980e0010 0x7f679800b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679e59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67980e0010 0x7f679800b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55398/0 (socket says 172.21.15.27:55398) 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679e59c640 1 -- 172.21.15.27:0/2098634990 learned_addr learned my addr 172.21.15.27:0/2098634990 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679e59c640 1 -- 172.21.15.27:0/2098634990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 msgr2=0x7f679803a8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679e59c640 1 --2- 172.21.15.27:0/2098634990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 0x7f679803a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679e59c640 1 -- 172.21.15.27:0/2098634990 >> v1:172.21.15.27:6789/0 conn(0x7f679803adf0 legacy=0x7f6798007c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.178+0000 7f679e59c640 1 -- 172.21.15.27:0/2098634990 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67980568a0 con 0x7f67980e0010 2024-03-06T07:22:55.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.182+0000 7f679e59c640 1 --2- 172.21.15.27:0/2098634990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980e0010 0x7f679800b1c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f67940098a0 tx=0x7f679402b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=711e227b1e5ded2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.183+0000 7f679d59a640 1 -- 172.21.15.27:0/2098634990 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794034070 con 0x7f67980e0010 2024-03-06T07:22:55.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.183+0000 7f679d59a640 1 -- 172.21.15.27:0/2098634990 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6794030880 con 0x7f67980e0010 2024-03-06T07:22:55.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.184+0000 7f679f59e640 1 -- 172.21.15.27:0/2098634990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980e0010 msgr2=0x7f679800b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.184+0000 7f679f59e640 1 --2- 172.21.15.27:0/2098634990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980e0010 0x7f679800b1c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f67940098a0 tx=0x7f679402b9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.184+0000 7f679d59a640 1 -- 172.21.15.27:0/2098634990 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794030c90 con 0x7f67980e0010 2024-03-06T07:22:55.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.184+0000 7f679f59e640 1 -- 172.21.15.27:0/2098634990 shutdown_connections 2024-03-06T07:22:55.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.184+0000 7f679f59e640 1 --2- 172.21.15.27:0/2098634990 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 0x7f679803a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.184+0000 7f679f59e640 1 --2- 172.21.15.27:0/2098634990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980e0010 0x7f679800b1c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.184+0000 7f679f59e640 1 -- 172.21.15.27:0/2098634990 >> 172.21.15.27:0/2098634990 conn(0x7f679801ef60 msgr2=0x7f6798055c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.184+0000 7f679f59e640 1 -- 172.21.15.27:0/2098634990 shutdown_connections 2024-03-06T07:22:55.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.184+0000 7f679f59e640 1 -- 172.21.15.27:0/2098634990 wait complete. 2024-03-06T07:22:55.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.186+0000 7f679f59e640 1 Processor -- start 2024-03-06T07:22:55.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.186+0000 7f679f59e640 1 -- start start 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679f59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 0x7f67980d43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679803adf0 0x7f67981799e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679f59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980e0010 0x7f679817b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679f59e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67980e03f0 con 0x7f679803adf0 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679f59e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6798053490 con 0x7f67980e0010 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679f59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679803b600 con 0x7f6798008f40 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679e59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 0x7f67980d43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679e59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 0x7f67980d43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:48016/0 (socket says 172.21.15.27:48016) 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679e59c640 1 -- 172.21.15.27:0/1024482396 learned_addr learned my addr 172.21.15.27:0/1024482396 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679ed9d640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980e0010 0x7f679817b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679dd9b640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679803adf0 0x7f67981799e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679e59c640 1 -- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980e0010 msgr2=0x7f679817b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.188+0000 7f679e59c640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980e0010 0x7f679817b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.189+0000 7f679e59c640 1 -- 172.21.15.27:0/1024482396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679803adf0 msgr2=0x7f67981799e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.189+0000 7f679e59c640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679803adf0 0x7f67981799e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.189+0000 7f679e59c640 1 -- 172.21.15.27:0/1024482396 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67980d49f0 con 0x7f6798008f40 2024-03-06T07:22:55.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.189+0000 7f679e59c640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 0x7f67980d43a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f67940099c0 tx=0x7f67940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.189+0000 7f679dd9b640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679803adf0 0x7f67981799e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.193+0000 7f678effd640 1 -- 172.21.15.27:0/1024482396 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794034040 con 0x7f6798008f40 2024-03-06T07:22:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.193+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67980d4cd0 con 0x7f6798008f40 2024-03-06T07:22:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.193+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6798033660 con 0x7f6798008f40 2024-03-06T07:22:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.193+0000 7f678effd640 1 -- 172.21.15.27:0/1024482396 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6794039d10 con 0x7f6798008f40 2024-03-06T07:22:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.193+0000 7f678effd640 1 -- 172.21.15.27:0/1024482396 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67940435f0 con 0x7f6798008f40 2024-03-06T07:22:55.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.194+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6798005700 con 0x7f6798008f40 2024-03-06T07:22:55.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.195+0000 7f678effd640 1 -- 172.21.15.27:0/1024482396 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f679404a060 con 0x7f6798008f40 2024-03-06T07:22:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.196+0000 7f678effd640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f678006b5d0 0x7f678006da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.196+0000 7f678effd640 1 -- 172.21.15.27:0/1024482396 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67940ac420 con 0x7f6798008f40 2024-03-06T07:22:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.196+0000 7f679dd9b640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f678006b5d0 0x7f678006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.198+0000 7f679dd9b640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f678006b5d0 0x7f678006da90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f67980d78f0 tx=0x7f6790008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.200+0000 7f678effd640 1 -- 172.21.15.27:0/1024482396 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f679403e070 con 0x7f6798008f40 2024-03-06T07:22:55.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.336+0000 7f8c837fe640 1 -- 172.21.15.27:0/1224071155 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c8c1515a0 con 0x7f8c8c06c040 2024-03-06T07:22:55.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.525+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd45c06b780 con 0x7fd45c165c90 2024-03-06T07:22:55.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.526+0000 7fd458ff9640 1 -- 172.21.15.27:0/325753643 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd450059000 con 0x7fd45c165c90 2024-03-06T07:22:55.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.524+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe1f8067ff0 con 0x7fe1f8142c30 2024-03-06T07:22:55.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.527+0000 7fe1eaffd640 1 -- 172.21.15.27:0/3369254218 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe1f40586a0 con 0x7fe1f8142c30 2024-03-06T07:22:55.530 INFO:teuthology.orchestra.run.smithi027.stdout:55834574853 2024-03-06T07:22:55.532 INFO:teuthology.orchestra.run.smithi027.stdout:55834574852 2024-03-06T07:22:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd44806b520 msgr2=0x7fd44806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd44806b520 0x7fd44806d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd45c164fd0 tx=0x7fd44c007990 comp rx=0 tx=0).stop 2024-03-06T07:22:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c165c90 msgr2=0x7fd45c0b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c165c90 0x7fd45c0b36d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd45000c3f0 tx=0x7fd45000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 shutdown_connections 2024-03-06T07:22:55.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd44806b520 0x7fd44806d9e0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd45c164fd0 tx=0x7fd44c007990 comp rx=0 tx=0).stop 2024-03-06T07:22:55.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c165c90 0x7fd45c0b36d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd45c15fb80 0x7fd45c0ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 --2- 172.21.15.27:0/325753643 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd45c0ae700 0x7fd45c153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.530+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 >> 172.21.15.27:0/325753643 conn(0x7fd45c066d00 msgr2=0x7fd45c06ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.531+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1dc06b5a0 msgr2=0x7fe1dc06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.531+0000 7fe20040e640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1dc06b5a0 0x7fe1dc06da60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec004be0 tx=0x7fe1ec001ad0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.532+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8142c30 msgr2=0x7fe1f8159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.532+0000 7fe20040e640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8142c30 0x7fe1f8159310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4006fd0 tx=0x7fe1f400b8e0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.534+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 shutdown_connections 2024-03-06T07:22:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.534+0000 7fe20040e640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1dc06b5a0 0x7fe1dc06da60 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec004be0 tx=0x7fe1ec001ad0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.534+0000 7fe20040e640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f81f2e00 0x7fe1f81f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.534+0000 7fe20040e640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f8142c30 0x7fe1f8159310 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.534+0000 7fe20040e640 1 --2- 172.21.15.27:0/3369254218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1e40051c0 0x7fe1f8154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.534+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 >> 172.21.15.27:0/3369254218 conn(0x7fe1f8066d00 msgr2=0x7fe1f80cf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.534+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 shutdown_connections 2024-03-06T07:22:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.534+0000 7fd462866640 1 -- 172.21.15.27:0/325753643 wait complete. 2024-03-06T07:22:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.537+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 shutdown_connections 2024-03-06T07:22:55.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.538+0000 7fe20040e640 1 -- 172.21.15.27:0/3369254218 wait complete. 2024-03-06T07:22:55.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.537+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2db8071b30 con 0x7f2db80b2b50 2024-03-06T07:22:55.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.538+0000 7f2d9f7fe640 1 -- 172.21.15.27:0/2075198627 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2dac07a100 con 0x7f2db80b2b50 2024-03-06T07:22:55.542 INFO:teuthology.orchestra.run.smithi027.stdout:55834574853 2024-03-06T07:22:55.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.547+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2da406b5a0 msgr2=0x7f2da406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.547+0000 7f2dbd017640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2da406b5a0 0x7f2da406da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2da8007630 tx=0x7f2da8007e10 comp rx=0 tx=0).stop 2024-03-06T07:22:55.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.547+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2db80b2b50 msgr2=0x7f2db80c7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.547+0000 7f2dbd017640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2db80b2b50 0x7f2db80c7670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2dac033000 tx=0x7f2dac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.554 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-03-06T07:22:55.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.551+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4884005580 con 0x7f48a4070f80 2024-03-06T07:22:55.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.552+0000 7f48a17fa640 1 -- 172.21.15.27:0/1598244654 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f48980172c0 con 0x7f48a4070f80 2024-03-06T07:22:55.555 INFO:teuthology.orchestra.run.smithi027.stdout:55834574853 2024-03-06T07:22:55.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.554+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 shutdown_connections 2024-03-06T07:22:55.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.554+0000 7f2dbd017640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2da406b5a0 0x7f2da406da60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.554+0000 7f2dbd017640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db81fb230 0x7f2db81fb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.554+0000 7f2dbd017640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2db80b2b50 0x7f2db80c7670 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.554+0000 7f2dbd017640 1 --2- 172.21.15.27:0/2075198627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2db8071540 0x7f2db81f1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.554+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 >> 172.21.15.27:0/2075198627 conn(0x7f2db80d3ca0 msgr2=0x7f2db815ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.554+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 shutdown_connections 2024-03-06T07:22:55.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.554+0000 7f2dbd017640 1 -- 172.21.15.27:0/2075198627 wait complete. 2024-03-06T07:22:55.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-03-06T07:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f488806b5a0 msgr2=0x7f488806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f488806b5a0 0x7f488806da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4890038000 tx=0x7f4890008030 comp rx=0 tx=0).stop 2024-03-06T07:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4070f80 msgr2=0x7f48a415b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4070f80 0x7f48a415b4a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f489800b410 tx=0x7f489800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 shutdown_connections 2024-03-06T07:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f488806b5a0 0x7f488806da60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48a4159d50 0x7f48a414b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a40aef90 0x7f48a41f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 --2- 172.21.15.27:0/1598244654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4070f80 0x7f48a415b4a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.555+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 >> 172.21.15.27:0/1598244654 conn(0x7f48a4066d00 msgr2=0x7f48a4153530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.556+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 shutdown_connections 2024-03-06T07:22:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.556+0000 7f48ab1ac640 1 -- 172.21.15.27:0/1598244654 wait complete. 2024-03-06T07:22:55.574 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-03-06T07:22:55.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.571+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff7880cf120 con 0x7ff7880cfba0 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.571+0000 7ff77effd640 1 -- 172.21.15.27:0/1103067318 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff778010040 con 0x7ff7880cfba0 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-03-06T07:22:55.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff76806b8d0 msgr2=0x7ff76806dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff76806b8d0 0x7ff76806dd90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff77003e000 tx=0x7ff770038000 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cfba0 msgr2=0x7ff78813a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cfba0 0x7ff78813a130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff77800efc0 tx=0x7ff77800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 shutdown_connections 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff76806b8d0 0x7ff76806dd90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff788161be0 0x7ff788143100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7880cfba0 0x7ff78813a130 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 --2- 172.21.15.27:0/1103067318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7880cf7c0 0x7ff78806d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 >> 172.21.15.27:0/1103067318 conn(0x7ff788066d00 msgr2=0x7ff7881463a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 shutdown_connections 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.574+0000 7ff78cc15640 1 -- 172.21.15.27:0/1103067318 wait complete. 2024-03-06T07:22:55.591 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-03-06T07:22:55.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.602+0000 7f7a61ffb640 1 -- 172.21.15.27:0/481433294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7a7c071030 con 0x7f7a7c067820 2024-03-06T07:22:55.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.602+0000 7f7a63fff640 1 -- 172.21.15.27:0/481433294 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a740587a0 con 0x7f7a7c067820 2024-03-06T07:22:55.605 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-03-06T07:22:55.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 -- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a6806b5a0 msgr2=0x7f7a6806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a6806b5a0 0x7f7a6806da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c036000 tx=0x7f7a6c03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 -- 172.21.15.27:0/481433294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a7c067820 msgr2=0x7f7a7c0a9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a7c067820 0x7f7a7c0a9e60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400ba80 tx=0x7f7a7400bf40 comp rx=0 tx=0).stop 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 -- 172.21.15.27:0/481433294 shutdown_connections 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a6806b5a0 0x7f7a6806da60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a7c15a420 0x7f7a7c158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a7c154190 0x7f7a7c13b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 --2- 172.21.15.27:0/481433294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a7c067820 0x7f7a7c0a9e60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 -- 172.21.15.27:0/481433294 >> 172.21.15.27:0/481433294 conn(0x7f7a7c066d00 msgr2=0x7f7a7c0b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 -- 172.21.15.27:0/481433294 shutdown_connections 2024-03-06T07:22:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.604+0000 7f7a61ffb640 1 -- 172.21.15.27:0/481433294 wait complete. 2024-03-06T07:22:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.612+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6798049000 con 0x7f6798008f40 2024-03-06T07:22:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.613+0000 7f678effd640 1 -- 172.21.15.27:0/1024482396 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f679404be90 con 0x7f6798008f40 2024-03-06T07:22:55.615 INFO:teuthology.orchestra.run.smithi027.stdout:55834574853 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f678006b5d0 msgr2=0x7f678006da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f678006b5d0 0x7f678006da90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f67980d78f0 tx=0x7f6790008040 comp rx=0 tx=0).stop 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 msgr2=0x7f67980d43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 0x7f67980d43a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f67940099c0 tx=0x7f67940046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 shutdown_connections 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f678006b5d0 0x7f678006da90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67980e0010 0x7f679817b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679803adf0 0x7f67981799e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 --2- 172.21.15.27:0/1024482396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6798008f40 0x7f67980d43a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 >> 172.21.15.27:0/1024482396 conn(0x7f679801ef60 msgr2=0x7f6798006af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 shutdown_connections 2024-03-06T07:22:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.615+0000 7f679f59e640 1 -- 172.21.15.27:0/1024482396 wait complete. 2024-03-06T07:22:55.621 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-03-06T07:22:55.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2024-03-06T07:22:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.797+0000 7f8c817fa640 1 -- 172.21.15.27:0/1224071155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8c8c0b79c0 con 0x7f8c8c06c040 2024-03-06T07:22:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.797+0000 7f8c837fe640 1 -- 172.21.15.27:0/1224071155 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c74057ed0 con 0x7f8c8c06c040 2024-03-06T07:22:55.801 INFO:teuthology.orchestra.run.smithi027.stdout:55834574853 2024-03-06T07:22:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 -- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6c06b5a0 msgr2=0x7f8c6c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6c06b5a0 0x7f8c6c06da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c036000 tx=0x7f8c7c03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 -- 172.21.15.27:0/1224071155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c06c040 msgr2=0x7f8c8c0c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c06c040 0x7f8c8c0c7bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400c400 tx=0x7f8c7400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 -- 172.21.15.27:0/1224071155 shutdown_connections 2024-03-06T07:22:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6c06b5a0 0x7f8c6c06da60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c8c0cda40 0x7f8c8c161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c8c06c040 0x7f8c8c0c7bd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 --2- 172.21.15.27:0/1224071155 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c8c06b130 0x7f8c8c151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 -- 172.21.15.27:0/1224071155 >> 172.21.15.27:0/1224071155 conn(0x7f8c8c0d1a40 msgr2=0x7f8c8c0b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 -- 172.21.15.27:0/1224071155 shutdown_connections 2024-03-06T07:22:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:55.800+0000 7f8c817fa640 1 -- 172.21.15.27:0/1224071155 wait complete. 2024-03-06T07:22:55.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-03-06T07:22:56.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:56.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:56.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:56.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:56.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:56.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:56.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:56.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.705+0000 7f252c9ae640 1 Processor -- start 2024-03-06T07:22:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.706+0000 7f252c9ae640 1 -- start start 2024-03-06T07:22:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.707+0000 7f252c9ae640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f252815dc30 0x7f2528165270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.707+0000 7f252c9ae640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25280b3e70 con 0x7f252815b5a0 2024-03-06T07:22:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.707+0000 7f252c9ae640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252815bb80 con 0x7f252815a660 2024-03-06T07:22:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.707+0000 7f252c9ae640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252815be60 con 0x7f252815dc30 2024-03-06T07:22:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.708+0000 7f2526d9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f252815dc30 0x7f2528165270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.708+0000 7f2526d9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f252815dc30 0x7f2528165270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55418/0 (socket says 172.21.15.27:55418) 2024-03-06T07:22:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.708+0000 7f2526d9d640 1 -- 172.21.15.27:0/2834709074 learned_addr learned my addr 172.21.15.27:0/2834709074 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.708+0000 7f2526d9d640 1 -- 172.21.15.27:0/2834709074 >> v1:172.21.15.69:6789/0 conn(0x7f252815a660 legacy=0x7f25280b3050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:56.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.708+0000 7f2526d9d640 1 -- 172.21.15.27:0/2834709074 >> v1:172.21.15.27:6789/0 conn(0x7f252815b5a0 legacy=0x7f25280b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.708+0000 7f2526d9d640 1 -- 172.21.15.27:0/2834709074 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252815c140 con 0x7f252815dc30 2024-03-06T07:22:56.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.710+0000 7f2526d9d640 1 --2- 172.21.15.27:0/2834709074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f252815dc30 0x7f2528165270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2520009e50 tx=0x7f252002cff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=936750647a030d4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.710+0000 7f252559a640 1 -- 172.21.15.27:0/2834709074 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520039070 con 0x7f252815dc30 2024-03-06T07:22:56.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.710+0000 7f252559a640 1 -- 172.21.15.27:0/2834709074 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252002db90 con 0x7f252815dc30 2024-03-06T07:22:56.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.710+0000 7f252559a640 1 -- 172.21.15.27:0/2834709074 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25200344b0 con 0x7f252815dc30 2024-03-06T07:22:56.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.711+0000 7f252c9ae640 1 -- 172.21.15.27:0/2834709074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f252815dc30 msgr2=0x7f2528165270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.711+0000 7f252c9ae640 1 --2- 172.21.15.27:0/2834709074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f252815dc30 0x7f2528165270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2520009e50 tx=0x7f252002cff0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.713+0000 7f252c9ae640 1 -- 172.21.15.27:0/2834709074 shutdown_connections 2024-03-06T07:22:56.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.713+0000 7f252c9ae640 1 --2- 172.21.15.27:0/2834709074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f252815dc30 0x7f2528165270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.713+0000 7f252c9ae640 1 -- 172.21.15.27:0/2834709074 >> 172.21.15.27:0/2834709074 conn(0x7f2528071650 msgr2=0x7f2528072500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.716+0000 7f252c9ae640 1 -- 172.21.15.27:0/2834709074 shutdown_connections 2024-03-06T07:22:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.716+0000 7f252c9ae640 1 -- 172.21.15.27:0/2834709074 wait complete. 2024-03-06T07:22:56.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.718+0000 7f252c9ae640 1 Processor -- start 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.718+0000 7f252c9ae640 1 -- start start 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.719+0000 7f252c9ae640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f252815a660 0x7f2528152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.719+0000 7f252c9ae640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815b5a0 0x7f25281e4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.719+0000 7f252c9ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f252815dc30 0x7f252814f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.719+0000 7f252c9ae640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25280cc6a0 con 0x7f252815dc30 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f252c9ae640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252815c700 con 0x7f252815a660 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f252c9ae640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2528074010 con 0x7f252815b5a0 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f2526d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f252815dc30 0x7f252814f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f2526d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f252815dc30 0x7f252814f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51698/0 (socket says 172.21.15.27:51698) 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f2526d9d640 1 -- 172.21.15.27:0/189774497 learned_addr learned my addr 172.21.15.27:0/189774497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f2526d9d640 1 -- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815b5a0 msgr2=0x7f25281e4960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:56.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f2526d9d640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815b5a0 0x7f25281e4960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f2526d9d640 1 -- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f252815a660 msgr2=0x7f2528152850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:56.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f2526d9d640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f252815a660 0x7f2528152850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.720+0000 7f2526d9d640 1 -- 172.21.15.27:0/189774497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252814fc10 con 0x7f252815dc30 2024-03-06T07:22:56.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.721+0000 7f2526d9d640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f252815dc30 0x7f252814f6d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2520037040 tx=0x7f2520004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.721+0000 7f2516ffd640 1 -- 172.21.15.27:0/189774497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520039050 con 0x7f252815dc30 2024-03-06T07:22:56.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.721+0000 7f252c9ae640 1 -- 172.21.15.27:0/189774497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2528152e30 con 0x7f252815dc30 2024-03-06T07:22:56.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.721+0000 7f252c9ae640 1 -- 172.21.15.27:0/189774497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25281533f0 con 0x7f252815dc30 2024-03-06T07:22:56.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.722+0000 7f2516ffd640 1 -- 172.21.15.27:0/189774497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2520045070 con 0x7f252815dc30 2024-03-06T07:22:56.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.722+0000 7f2516ffd640 1 -- 172.21.15.27:0/189774497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520009510 con 0x7f252815dc30 2024-03-06T07:22:56.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.724+0000 7f2516ffd640 1 -- 172.21.15.27:0/189774497 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25200079a0 con 0x7f252815dc30 2024-03-06T07:22:56.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.725+0000 7f2516ffd640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f250806b520 0x7f250806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.725+0000 7f2516ffd640 1 -- 172.21.15.27:0/189774497 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25200ae610 con 0x7f252815dc30 2024-03-06T07:22:56.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.725+0000 7f252659c640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f250806b520 0x7f250806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.726+0000 7f252659c640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f250806b520 0x7f250806d9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f251c009670 tx=0x7f251c006bd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.726+0000 7f252c9ae640 1 -- 172.21.15.27:0/189774497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c005290 con 0x7f252815dc30 2024-03-06T07:22:56.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.737+0000 7f2516ffd640 1 -- 172.21.15.27:0/189774497 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f252814fc10 con 0x7f252815dc30 2024-03-06T07:22:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.763+0000 7f11ce87d640 1 Processor -- start 2024-03-06T07:22:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.763+0000 7f11ce87d640 1 -- start start 2024-03-06T07:22:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.764+0000 7f11ce87d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f11c0035d80 0x7f11c0039100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.765+0000 7f11ce87d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c00517d0 con 0x7f11c0054230 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.765+0000 7f11ce87d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c004a2a0 con 0x7f11c0036160 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.765+0000 7f11ce87d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c0051a70 con 0x7f11c0035d80 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.765+0000 7f11ce07c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f11c0054230 0x7f11c0055180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50082/0 (socket says 172.21.15.27:50082) 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.765+0000 7f11ce07c640 1 -- 172.21.15.27:0/780591402 learned_addr learned my addr 172.21.15.27:0/780591402 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73248511 0 0) 0x7f11c00517d0 con 0x7f11c0054230 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b4003520 con 0x7f11c0054230 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4266996258 0 0) 0x7f11b4003520 con 0x7f11c0054230 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 >> v2:172.21.15.69:3300/0 conn(0x7f11c0035d80 msgr2=0x7f11c0039100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 --2- 172.21.15.27:0/780591402 >> v2:172.21.15.69:3300/0 conn(0x7f11c0035d80 0x7f11c0039100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 >> v1:172.21.15.69:6789/0 conn(0x7f11c0036160 legacy=0x7f11c0007c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c0051d50 con 0x7f11c0054230 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f11c804fb80 con 0x7f11c0054230 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 >> v1:172.21.15.27:6789/0 conn(0x7f11c0054230 legacy=0x7f11c0055180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11b40039b0 0x7f11b4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11b4004800 0x7f11b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.766+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c0051a70 con 0x7f11b40039b0 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.767+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b402e150 con 0x7f11b4004800 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.767+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b402e350 con 0x7f11b4005210 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.767+0000 7f11ce07c640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.767+0000 7f11cd07a640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11b4004800 0x7f11b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.768+0000 7f11cd87b640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11b40039b0 0x7f11b4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.768+0000 7f11ce07c640 1 -- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11b4004800 msgr2=0x7f11b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.768+0000 7f11ce07c640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11b4004800 0x7f11b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.768+0000 7f11ce07c640 1 -- 172.21.15.27:0/780591402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11b40039b0 msgr2=0x7f11b4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.768+0000 7f11ce07c640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11b40039b0 0x7f11b4003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.768+0000 7f11ce07c640 1 -- 172.21.15.27:0/780591402 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c0051d50 con 0x7f11b4005210 2024-03-06T07:22:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.769+0000 7f11ce07c640 1 -- 172.21.15.27:0/780591402 reap_dead start 2024-03-06T07:22:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.769+0000 7f11ce07c640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11b400b9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f11c80505a0 tx=0x7f11b4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c8766ba68b67682 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.770+0000 7f1aede7f640 1 Processor -- start 2024-03-06T07:22:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.770+0000 7f1aede7f640 1 -- start start 2024-03-06T07:22:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f1aede7f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ae80cde20 0x7f1ae80b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.771+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c80b4070 con 0x7f11b4005210 2024-03-06T07:22:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.771+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c80a7cf0 con 0x7f11b4005210 2024-03-06T07:22:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.771+0000 7f11cc879640 1 -- 172.21.15.27:0/780591402 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c80752b0 con 0x7f11b4005210 2024-03-06T07:22:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f11ce87d640 1 -- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 msgr2=0x7f11b400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f11ce87d640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11b400b9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f11c80505a0 tx=0x7f11b4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f1aede7f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1ae80b3920 0x7f1ae806e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f1aede7f640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae80ce500 con 0x7f1ae81620f0 2024-03-06T07:22:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f1aede7f640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae81624d0 con 0x7f1ae80cde20 2024-03-06T07:22:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f1aede7f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae80b3490 con 0x7f1ae80b3920 2024-03-06T07:22:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f1aece7d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1ae81620f0 0x7f1ae80cd710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50092/0 (socket says 172.21.15.27:50092) 2024-03-06T07:22:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f1aece7d640 1 -- 172.21.15.27:0/3712663654 learned_addr learned my addr 172.21.15.27:0/3712663654 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f1ae7fff640 1 --2- 172.21.15.27:0/3712663654 >> v2:172.21.15.27:3300/0 conn(0x7f1ae80cde20 0x7f1ae80b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.772+0000 7f1aed67e640 1 --2- 172.21.15.27:0/3712663654 >> v2:172.21.15.69:3300/0 conn(0x7f1ae80b3920 0x7f1ae806e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.773+0000 7f1ae77fe640 1 -- 172.21.15.27:0/3712663654 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406507348 0 0) 0x7f1ae80ce500 con 0x7f1ae81620f0 2024-03-06T07:22:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.773+0000 7f11ce87d640 1 -- 172.21.15.27:0/780591402 shutdown_connections 2024-03-06T07:22:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.773+0000 7f1ae7fff640 1 -- 172.21.15.27:0/3712663654 >> v2:172.21.15.69:3300/0 conn(0x7f1ae80b3920 msgr2=0x7f1ae806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.773+0000 7f1ae7fff640 1 --2- 172.21.15.27:0/3712663654 >> v2:172.21.15.69:3300/0 conn(0x7f1ae80b3920 0x7f1ae806e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.773+0000 7f1ae7fff640 1 -- 172.21.15.27:0/3712663654 >> v1:172.21.15.27:6789/0 conn(0x7f1ae81620f0 legacy=0x7f1ae80cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.773+0000 7f1ae7fff640 1 -- 172.21.15.27:0/3712663654 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae806c9a0 con 0x7f1ae80cde20 2024-03-06T07:22:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.773+0000 7f11ce87d640 1 --2- 172.21.15.27:0/780591402 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11b400b9c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.773+0000 7f11ce87d640 1 -- 172.21.15.27:0/780591402 >> 172.21.15.27:0/780591402 conn(0x7f11c0053740 msgr2=0x7f11c0053b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.774+0000 7f1ae7fff640 1 --2- 172.21.15.27:0/3712663654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80cde20 0x7f1ae80b4830 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80ce6d0 tx=0x7f1ad402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a89ecdcc051fab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.775+0000 7f6bd5e15640 1 Processor -- start 2024-03-06T07:22:56.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.775+0000 7f6bd5e15640 1 -- start start 2024-03-06T07:22:56.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.774+0000 7f11ce87d640 1 -- 172.21.15.27:0/780591402 shutdown_connections 2024-03-06T07:22:56.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.774+0000 7f11ce87d640 1 -- 172.21.15.27:0/780591402 wait complete. 2024-03-06T07:22:56.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f1ae77fe640 1 -- 172.21.15.27:0/3712663654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4035020 con 0x7f1ae80cde20 2024-03-06T07:22:56.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f1ae77fe640 1 -- 172.21.15.27:0/3712663654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad4031830 con 0x7f1ae80cde20 2024-03-06T07:22:56.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f1ae77fe640 1 -- 172.21.15.27:0/3712663654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4031c00 con 0x7f1ae80cde20 2024-03-06T07:22:56.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.777+0000 7f1aede7f640 1 -- 172.21.15.27:0/3712663654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80cde20 msgr2=0x7f1ae80b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.777+0000 7f1aede7f640 1 --2- 172.21.15.27:0/3712663654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80cde20 0x7f1ae80b4830 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80ce6d0 tx=0x7f1ad402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f6bd5e15640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd006c710 0x7f6bd006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f6bd5e15640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd015cc10 con 0x7f6bd00b2c30 2024-03-06T07:22:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f6bd5e15640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0072840 con 0x7f6bd015c830 2024-03-06T07:22:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f6bd5e15640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd0072b20 con 0x7f6bd006c710 2024-03-06T07:22:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f6bceffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd006c710 0x7f6bd006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f6bceffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd006c710 0x7f6bd006cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51718/0 (socket says 172.21.15.27:51718) 2024-03-06T07:22:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f6bceffd640 1 -- 172.21.15.27:0/1174671393 learned_addr learned my addr 172.21.15.27:0/1174671393 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.777+0000 7f6bceffd640 1 -- 172.21.15.27:0/1174671393 >> v1:172.21.15.69:6789/0 conn(0x7f6bd015c830 legacy=0x7f6bd01631c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.777+0000 7f6bceffd640 1 -- 172.21.15.27:0/1174671393 >> v1:172.21.15.27:6789/0 conn(0x7f6bd00b2c30 legacy=0x7f6bd00b30a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.777+0000 7f6bceffd640 1 -- 172.21.15.27:0/1174671393 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd00709a0 con 0x7f6bd006c710 2024-03-06T07:22:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f1aede7f640 1 -- 172.21.15.27:0/3712663654 shutdown_connections 2024-03-06T07:22:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f1aede7f640 1 --2- 172.21.15.27:0/3712663654 >> v2:172.21.15.69:3300/0 conn(0x7f1ae80b3920 0x7f1ae806e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f1aede7f640 1 --2- 172.21.15.27:0/3712663654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80cde20 0x7f1ae80b4830 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f1aede7f640 1 -- 172.21.15.27:0/3712663654 >> 172.21.15.27:0/3712663654 conn(0x7f1ae8066d00 msgr2=0x7f1ae8067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f1aede7f640 1 -- 172.21.15.27:0/3712663654 shutdown_connections 2024-03-06T07:22:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f1aede7f640 1 -- 172.21.15.27:0/3712663654 wait complete. 2024-03-06T07:22:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f11ce87d640 1 Processor -- start 2024-03-06T07:22:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.776+0000 7f11ce87d640 1 -- start start 2024-03-06T07:22:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f11ce87d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11c00d6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f6bceffd640 1 --2- 172.21.15.27:0/1174671393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd006c710 0x7f6bd006cba0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b1a0 tx=0x7f6bc802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198d46182db52872 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f11ce87d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c00d6d00 0x7f11c00e37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f11ce87d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c00e15e0 0x7f11c00e1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.778+0000 7f6bce7fc640 1 -- 172.21.15.27:0/1174671393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc802c9b0 con 0x7f6bd006c710 2024-03-06T07:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd87b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11c00d6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd87b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11c00d6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:48036/0 (socket says 172.21.15.27:48036) 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11ce07c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c00e15e0 0x7f11c00e1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd87b640 1 -- 172.21.15.27:0/1858127277 learned_addr learned my addr 172.21.15.27:0/1858127277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c0039100 con 0x7f11c00d6d00 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c0052310 con 0x7f11c00e15e0 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c0039280 con 0x7f11b4005210 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd07a640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c00d6d00 0x7f11c00e37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f6bce7fc640 1 -- 172.21.15.27:0/1174671393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc802cb50 con 0x7f6bd006c710 2024-03-06T07:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f6bce7fc640 1 -- 172.21.15.27:0/1174671393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc802c9b0 con 0x7f6bd006c710 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd87b640 1 -- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c00e15e0 msgr2=0x7f11c00e1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd87b640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c00e15e0 0x7f11c00e1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd87b640 1 -- 172.21.15.27:0/1858127277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c00d6d00 msgr2=0x7f11c00e37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd87b640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c00d6d00 0x7f11c00e37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd87b640 1 -- 172.21.15.27:0/1858127277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c0178810 con 0x7f11b4005210 2024-03-06T07:22:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.779+0000 7f11cd07a640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c00d6d00 0x7f11c00e37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.780+0000 7f11cd87b640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11c00d6610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f11c400d4e0 tx=0x7f11c400d9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.780+0000 7f11be7fc640 1 -- 172.21.15.27:0/1858127277 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4018020 con 0x7f11b4005210 2024-03-06T07:22:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.781+0000 7f1aede7f640 1 Processor -- start 2024-03-06T07:22:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.781+0000 7f1aede7f640 1 -- start start 2024-03-06T07:22:56.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.780+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c0178af0 con 0x7f11b4005210 2024-03-06T07:22:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.782+0000 7f1aede7f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80b3920 0x7f1ae8200200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.783+0000 7f1aece7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80b3920 0x7f1ae8200200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.783+0000 7f1aece7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80b3920 0x7f1ae8200200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51734/0 (socket says 172.21.15.27:51734) 2024-03-06T07:22:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.783+0000 7f1aece7d640 1 -- 172.21.15.27:0/2986379082 learned_addr learned my addr 172.21.15.27:0/2986379082 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.783+0000 7f1aede7f640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae80cde20 0x7f1ae815f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.783+0000 7f1aede7f640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae81620f0 0x7f1ae81598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f1aede7f640 1 -- 172.21.15.27:0/2986379082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80707a0 con 0x7f1ae80b3920 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f1aede7f640 1 -- 172.21.15.27:0/2986379082 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae806cf60 con 0x7f1ae80cde20 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f1aed67e640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae81620f0 0x7f1ae81598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f1aede7f640 1 -- 172.21.15.27:0/2986379082 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8070920 con 0x7f1ae81620f0 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.781+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c01790b0 con 0x7f11b4005210 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.781+0000 7f11be7fc640 1 -- 172.21.15.27:0/1858127277 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c4014620 con 0x7f11b4005210 2024-03-06T07:22:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.783+0000 7f11be7fc640 1 -- 172.21.15.27:0/1858127277 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40149f0 con 0x7f11b4005210 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f0e2593a640 1 Processor -- start 2024-03-06T07:22:56.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.780+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1174671393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd006c710 msgr2=0x7f6bd006cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.783+0000 7f11be7fc640 1 -- 172.21.15.27:0/1858127277 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11c401b020 con 0x7f11b4005210 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f11be7fc640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11ac06b530 0x7f11ac06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f11be7fc640 1 -- 172.21.15.27:0/1858127277 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11c408a520 con 0x7f11b4005210 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.780+0000 7f6bd5e15640 1 --2- 172.21.15.27:0/1174671393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd006c710 0x7f6bd006cba0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b1a0 tx=0x7f6bc802c010 comp rx=0 tx=0).stop 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.780+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1174671393 shutdown_connections 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.780+0000 7f6bd5e15640 1 --2- 172.21.15.27:0/1174671393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd006c710 0x7f6bd006cba0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.780+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1174671393 >> 172.21.15.27:0/1174671393 conn(0x7f6bd0066d00 msgr2=0x7f6bd00670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.781+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1174671393 shutdown_connections 2024-03-06T07:22:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.781+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1174671393 wait complete. 2024-03-06T07:22:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b0005290 con 0x7f11b4005210 2024-03-06T07:22:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f11cd07a640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11ac06b530 0x7f11ac06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.782+0000 7f6bd5e15640 1 Processor -- start 2024-03-06T07:22:56.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.783+0000 7f6bd5e15640 1 -- start start 2024-03-06T07:22:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f11cd07a640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11ac06b530 0x7f11ac06d9f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f11b8006fd0 tx=0x7f11b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f1aece7d640 1 -- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae81620f0 msgr2=0x7f1ae81598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f1aece7d640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae81620f0 0x7f1ae81598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f1aece7d640 1 -- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae80cde20 msgr2=0x7f1ae815f270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f1aece7d640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae80cde20 0x7f1ae815f270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f1aece7d640 1 -- 172.21.15.27:0/2986379082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae8158ae0 con 0x7f1ae80b3920 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bd5e15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd006c710 0x7f6bd014ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bd5e15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b2c30 0x7f6bd01638d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bd5e15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015c830 0x7f6bd0163e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bd5e15640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00b4960 con 0x7f6bd015c830 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bd5e15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0070f60 con 0x7f6bd00b2c30 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bd5e15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00b4ae0 con 0x7f6bd006c710 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bcf7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd006c710 0x7f6bd014ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bceffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b2c30 0x7f6bd01638d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bceffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b2c30 0x7f6bd01638d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55456/0 (socket says 172.21.15.27:55456) 2024-03-06T07:22:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bceffd640 1 -- 172.21.15.27:0/1771674618 learned_addr learned my addr 172.21.15.27:0/1771674618 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f6bcffff640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015c830 0x7f6bd0163e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f6bceffd640 1 -- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd006c710 msgr2=0x7f6bd014ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f6bceffd640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd006c710 0x7f6bd014ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f6bceffd640 1 -- 172.21.15.27:0/1771674618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015c830 msgr2=0x7f6bd0163e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f6bceffd640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015c830 0x7f6bd0163e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f6bceffd640 1 -- 172.21.15.27:0/1771674618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd0151760 con 0x7f6bd00b2c30 2024-03-06T07:22:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f1aece7d640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80b3920 0x7f1ae8200200 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00c780 tx=0x7f1adc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f6bceffd640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b2c30 0x7f6bd01638d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b2c0 tx=0x7f6bc8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.785+0000 7f6bb7fff640 1 -- 172.21.15.27:0/1771674618 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8043020 con 0x7f6bd00b2c30 2024-03-06T07:22:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f1ae57fa640 1 -- 172.21.15.27:0/2986379082 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc018070 con 0x7f1ae80b3920 2024-03-06T07:22:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f1ae57fa640 1 -- 172.21.15.27:0/2986379082 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1adc0070d0 con 0x7f1ae80b3920 2024-03-06T07:22:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f1ae57fa640 1 -- 172.21.15.27:0/2986379082 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc007540 con 0x7f1ae80b3920 2024-03-06T07:22:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd0165c40 con 0x7f6bd00b2c30 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd0151480 con 0x7f6bd00b2c30 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.787+0000 7f6bb7fff640 1 -- 172.21.15.27:0/1771674618 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc803e450 con 0x7f6bd00b2c30 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.787+0000 7f6bb7fff640 1 -- 172.21.15.27:0/1771674618 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc803e840 con 0x7f6bd00b2c30 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.784+0000 7f0e2593a640 1 -- start start 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f0e2593a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 0x7f0e200aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f0e2593a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2015aae0 con 0x7f0e2015a420 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f0e2593a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2015adc0 con 0x7f0e20154400 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f0e2593a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200af3b0 con 0x7f0e200cff30 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f0e1e7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0e2015a420 0x7f0e200cf820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50106/0 (socket says 172.21.15.27:50106) 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f0e1e7fc640 1 -- 172.21.15.27:0/3079180242 learned_addr learned my addr 172.21.15.27:0/3079180242 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f0e1f7fe640 1 --2- 172.21.15.27:0/3079180242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 0x7f0e200aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.787+0000 7f0e1dffb640 1 -- 172.21.15.27:0/3079180242 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476680272 0 0) 0x7f0e2015aae0 con 0x7f0e2015a420 2024-03-06T07:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.787+0000 7f0e1dffb640 1 -- 172.21.15.27:0/3079180242 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e08003520 con 0x7f0e2015a420 2024-03-06T07:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f1aede7f640 1 -- 172.21.15.27:0/2986379082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae8158dc0 con 0x7f1ae80b3920 2024-03-06T07:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.786+0000 7f1aede7f640 1 -- 172.21.15.27:0/2986379082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae8200740 con 0x7f1ae80b3920 2024-03-06T07:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.787+0000 7f0e1f7fe640 1 -- 172.21.15.27:0/3079180242 >> v1:172.21.15.69:6789/0 conn(0x7f0e20154400 legacy=0x7f0e200b2bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.787+0000 7f0e1f7fe640 1 -- 172.21.15.27:0/3079180242 >> v1:172.21.15.27:6789/0 conn(0x7f0e2015a420 legacy=0x7f0e200cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.787+0000 7f0e1f7fe640 1 -- 172.21.15.27:0/3079180242 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2015b0a0 con 0x7f0e200cff30 2024-03-06T07:22:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.788+0000 7f1ad2ffd640 1 -- 172.21.15.27:0/2986379082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae804f330 con 0x7f1ae80b3920 2024-03-06T07:22:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.788+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb0005290 con 0x7f6bd00b2c30 2024-03-06T07:22:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.790+0000 7f6bb7fff640 1 -- 172.21.15.27:0/1771674618 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bc8046020 con 0x7f6bd00b2c30 2024-03-06T07:22:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.790+0000 7f6bb7fff640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bbc06b520 0x7f6bbc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.790+0000 7f11be7fc640 1 -- 172.21.15.27:0/1858127277 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11c4022020 con 0x7f11b4005210 2024-03-06T07:22:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.790+0000 7f6bb7fff640 1 -- 172.21.15.27:0/1771674618 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bc80acb70 con 0x7f6bd00b2c30 2024-03-06T07:22:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.790+0000 7f6bcf7fe640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bbc06b520 0x7f6bbc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.790+0000 7f1ae57fa640 1 -- 172.21.15.27:0/2986379082 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1adc0047e0 con 0x7f1ae80b3920 2024-03-06T07:22:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.791+0000 7f1ae57fa640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b640 0x7f1ad806db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.791+0000 7f1ae57fa640 1 -- 172.21.15.27:0/2986379082 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1adc0897b0 con 0x7f1ae80b3920 2024-03-06T07:22:56.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.790+0000 7f0e1f7fe640 1 --2- 172.21.15.27:0/3079180242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 0x7f0e200aee70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0e100097d0 tx=0x7f0e1002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=221c69226603701d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.791+0000 7f1ae7fff640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b640 0x7f1ad806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.791+0000 7f6bcf7fe640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bbc06b520 0x7f6bbc06d9e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009930 tx=0x7f6bc0009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.791+0000 7f0e1dffb640 1 -- 172.21.15.27:0/3079180242 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10031780 con 0x7f0e200cff30 2024-03-06T07:22:56.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.791+0000 7f0e1dffb640 1 -- 172.21.15.27:0/3079180242 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e10037070 con 0x7f0e200cff30 2024-03-06T07:22:56.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.791+0000 7f0e1dffb640 1 -- 172.21.15.27:0/3079180242 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10031b40 con 0x7f0e200cff30 2024-03-06T07:22:56.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.793+0000 7f0e2593a640 1 -- 172.21.15.27:0/3079180242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 msgr2=0x7f0e200aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.793+0000 7f0e2593a640 1 --2- 172.21.15.27:0/3079180242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 0x7f0e200aee70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0e100097d0 tx=0x7f0e1002b820 comp rx=0 tx=0).stop 2024-03-06T07:22:56.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.793+0000 7f0e2593a640 1 -- 172.21.15.27:0/3079180242 shutdown_connections 2024-03-06T07:22:56.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.793+0000 7f0e2593a640 1 --2- 172.21.15.27:0/3079180242 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 0x7f0e200aee70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.793+0000 7f0e2593a640 1 -- 172.21.15.27:0/3079180242 >> 172.21.15.27:0/3079180242 conn(0x7f0e20066d00 msgr2=0x7f0e20067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.793+0000 7f1ae7fff640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b640 0x7f1ad806db00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4036000 tx=0x7f1ad4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.796+0000 7f1ae57fa640 1 -- 172.21.15.27:0/2986379082 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1adc058050 con 0x7f1ae80b3920 2024-03-06T07:22:56.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.797+0000 7f0e2593a640 1 -- 172.21.15.27:0/3079180242 shutdown_connections 2024-03-06T07:22:56.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.797+0000 7f6bb7fff640 1 -- 172.21.15.27:0/1771674618 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bc804d030 con 0x7f6bd00b2c30 2024-03-06T07:22:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.797+0000 7f0e2593a640 1 -- 172.21.15.27:0/3079180242 wait complete. 2024-03-06T07:22:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.799+0000 7f0e2593a640 1 Processor -- start 2024-03-06T07:22:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.799+0000 7f0e2593a640 1 -- start start 2024-03-06T07:22:56.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.800+0000 7f0e2593a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 0x7f0e2015c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.800+0000 7f0e2593a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e20154400 0x7f0e2014d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.800+0000 7f0e2593a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e2015a420 0x7f0e202000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e2593a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2006c7b0 con 0x7f0e2015a420 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e2593a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200b32e0 con 0x7f0e20154400 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e2593a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e201547e0 con 0x7f0e200cff30 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e1e7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e20154400 0x7f0e2014d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e1e7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e20154400 0x7f0e2014d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:55470/0 (socket says 172.21.15.27:55470) 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e1e7fc640 1 -- 172.21.15.27:0/1210650629 learned_addr learned my addr 172.21.15.27:0/1210650629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e1e7fc640 1 -- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 msgr2=0x7f0e2015c9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e1e7fc640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 0x7f0e2015c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e1e7fc640 1 -- 172.21.15.27:0/1210650629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e2015a420 msgr2=0x7f0e202000a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.801+0000 7f0e1f7fe640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e2015a420 0x7f0e202000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.802+0000 7f0e1e7fc640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e2015a420 0x7f0e202000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.802+0000 7f0e1e7fc640 1 -- 172.21.15.27:0/1210650629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e20201e50 con 0x7f0e20154400 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.802+0000 7f0e1e7fc640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e20154400 0x7f0e2014d1e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0e18001950 tx=0x7f0e18001e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.802+0000 7f0e1f7fe640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e2015a420 0x7f0e202000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.802+0000 7f0e07fff640 1 -- 172.21.15.27:0/1210650629 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18018070 con 0x7f0e20154400 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.803+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e20202130 con 0x7f0e20154400 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.803+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e202026f0 con 0x7f0e20154400 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.803+0000 7f0e07fff640 1 -- 172.21.15.27:0/1210650629 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e180082e0 con 0x7f0e20154400 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.803+0000 7f0e07fff640 1 -- 172.21.15.27:0/1210650629 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1800d900 con 0x7f0e20154400 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.805+0000 7f0e07fff640 1 -- 172.21.15.27:0/1210650629 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e1801b070 con 0x7f0e20154400 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.806+0000 7f0e07fff640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0c06b5a0 0x7f0e0c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.806+0000 7f0e1effd640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0c06b5a0 0x7f0e0c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.806+0000 7f0e07fff640 1 -- 172.21.15.27:0/1210650629 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e180899d0 con 0x7f0e20154400 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.807+0000 7f0e1effd640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0c06b5a0 0x7f0e0c06da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009670 tx=0x7f0e14006bd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.808+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e00005290 con 0x7f0e20154400 2024-03-06T07:22:56.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:56.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.817+0000 7f0e07fff640 1 -- 172.21.15.27:0/1210650629 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e180588c0 con 0x7f0e20154400 2024-03-06T07:22:56.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.819+0000 7f443e69a640 1 Processor -- start 2024-03-06T07:22:56.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.819+0000 7f443e69a640 1 -- start start 2024-03-06T07:22:56.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.820+0000 7f443e69a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4438067820 0x7f443815a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.820+0000 7f443e69a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44380aee70 0x7f4438154370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.820+0000 7f443e69a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44381548b0 con 0x7f44380b2bd0 2024-03-06T07:22:56.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.820+0000 7f443e69a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443815aae0 con 0x7f44380aee70 2024-03-06T07:22:56.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.820+0000 7f443e69a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443815adc0 con 0x7f4438067820 2024-03-06T07:22:56.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.821+0000 7f443cc37640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f44380b2bd0 0x7f44380cf820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50122/0 (socket says 172.21.15.27:50122) 2024-03-06T07:22:56.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.821+0000 7f443cc37640 1 -- 172.21.15.27:0/4260364961 learned_addr learned my addr 172.21.15.27:0/4260364961 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:56.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.820+0000 7f886b53a640 1 Processor -- start 2024-03-06T07:22:56.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.820+0000 7f886b53a640 1 -- start start 2024-03-06T07:22:56.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.821+0000 7f4437fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4438067820 0x7f443815a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f886b53a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f886415faf0 0x7f88641631b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f886b53a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f88640b4810 0x7f886406c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f886b53a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 0x7f886406cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f886b53a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88641638f0 con 0x7f886415faf0 2024-03-06T07:22:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f886b53a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88641636f0 con 0x7f88640b4810 2024-03-06T07:22:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f886b53a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8864070870 con 0x7f886415fed0 2024-03-06T07:22:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f88692d6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f886415faf0 0x7f88641631b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f88692d6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f886415faf0 0x7f88641631b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51756/0 (socket says 172.21.15.27:51756) 2024-03-06T07:22:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f88692d6640 1 -- 172.21.15.27:0/2581037519 learned_addr learned my addr 172.21.15.27:0/2581037519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f8868ad5640 1 --2- 172.21.15.27:0/2581037519 >> v2:172.21.15.69:3300/0 conn(0x7f88640b4810 0x7f886406c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.822+0000 7f8869ad7640 1 --2- 172.21.15.27:0/2581037519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 0x7f886406cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.823+0000 7f4437fff640 1 -- 172.21.15.27:0/4260364961 >> v2:172.21.15.27:3300/0 conn(0x7f44380aee70 msgr2=0x7f4438154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.823+0000 7f4437fff640 1 --2- 172.21.15.27:0/4260364961 >> v2:172.21.15.27:3300/0 conn(0x7f44380aee70 0x7f4438154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.823+0000 7f4437fff640 1 -- 172.21.15.27:0/4260364961 >> v1:172.21.15.27:6789/0 conn(0x7f44380b2bd0 legacy=0x7f44380cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.823+0000 7f4437fff640 1 -- 172.21.15.27:0/4260364961 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443815b0a0 con 0x7f4438067820 2024-03-06T07:22:56.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.823+0000 7f88692d6640 1 -- 172.21.15.27:0/2581037519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 msgr2=0x7f886406cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.823+0000 7f88692d6640 1 --2- 172.21.15.27:0/2581037519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 0x7f886406cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.823+0000 7f88692d6640 1 -- 172.21.15.27:0/2581037519 >> v2:172.21.15.69:3300/0 conn(0x7f88640b4810 msgr2=0x7f886406c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.823+0000 7f88692d6640 1 --2- 172.21.15.27:0/2581037519 >> v2:172.21.15.69:3300/0 conn(0x7f88640b4810 0x7f886406c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.823+0000 7f88692d6640 1 -- 172.21.15.27:0/2581037519 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8864072470 con 0x7f886415faf0 2024-03-06T07:22:56.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.825+0000 7f88692d6640 1 --2- 172.21.15.27:0/2581037519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f886415faf0 0x7f88641631b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8864067a80 tx=0x7f885c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2979d67cd5fa6354 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.826+0000 7f885bfff640 1 -- 172.21.15.27:0/2581037519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c02ca80 con 0x7f886415faf0 2024-03-06T07:22:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.826+0000 7f885bfff640 1 -- 172.21.15.27:0/2581037519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c02cc20 con 0x7f886415faf0 2024-03-06T07:22:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.826+0000 7f885bfff640 1 -- 172.21.15.27:0/2581037519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c0343f0 con 0x7f886415faf0 2024-03-06T07:22:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.827+0000 7f886b53a640 1 -- 172.21.15.27:0/2581037519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f886415faf0 msgr2=0x7f88641631b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.827+0000 7f886b53a640 1 --2- 172.21.15.27:0/2581037519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f886415faf0 0x7f88641631b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8864067a80 tx=0x7f885c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.827+0000 7f886b53a640 1 -- 172.21.15.27:0/2581037519 shutdown_connections 2024-03-06T07:22:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.827+0000 7f886b53a640 1 --2- 172.21.15.27:0/2581037519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 0x7f886406cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.827+0000 7f886b53a640 1 --2- 172.21.15.27:0/2581037519 >> v2:172.21.15.69:3300/0 conn(0x7f88640b4810 0x7f886406c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.827+0000 7f886b53a640 1 --2- 172.21.15.27:0/2581037519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f886415faf0 0x7f88641631b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.827+0000 7f886b53a640 1 -- 172.21.15.27:0/2581037519 >> 172.21.15.27:0/2581037519 conn(0x7f8864066d00 msgr2=0x7f8864067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.828+0000 7f886b53a640 1 -- 172.21.15.27:0/2581037519 shutdown_connections 2024-03-06T07:22:56.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.828+0000 7f886b53a640 1 -- 172.21.15.27:0/2581037519 wait complete. 2024-03-06T07:22:56.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.830+0000 7f886b53a640 1 Processor -- start 2024-03-06T07:22:56.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.830+0000 7f886b53a640 1 -- start start 2024-03-06T07:22:56.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.831+0000 7f886b53a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88640b4810 0x7f88640a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f88692d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88640b4810 0x7f88640a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f88692d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88640b4810 0x7f88640a97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51772/0 (socket says 172.21.15.27:51772) 2024-03-06T07:22:56.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f886b53a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f886415faf0 0x7f88640a9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f88692d6640 1 -- 172.21.15.27:0/1830143429 learned_addr learned my addr 172.21.15.27:0/1830143429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f886b53a640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 0x7f8864159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f886b53a640 1 -- 172.21.15.27:0/1830143429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8864070b10 con 0x7f88640b4810 2024-03-06T07:22:56.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f886b53a640 1 -- 172.21.15.27:0/1830143429 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8864072a30 con 0x7f886415faf0 2024-03-06T07:22:56.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f886b53a640 1 -- 172.21.15.27:0/1830143429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8864165d20 con 0x7f886415fed0 2024-03-06T07:22:56.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f8868ad5640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f886415faf0 0x7f88640a9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.832+0000 7f8869ad7640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 0x7f8864159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.833+0000 7f88692d6640 1 -- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 msgr2=0x7f8864159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.833+0000 7f88692d6640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 0x7f8864159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.833+0000 7f88692d6640 1 -- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f886415faf0 msgr2=0x7f88640a9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.833+0000 7f88692d6640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f886415faf0 0x7f88640a9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.833+0000 7f88692d6640 1 -- 172.21.15.27:0/1830143429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886415a5c0 con 0x7f88640b4810 2024-03-06T07:22:56.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.834+0000 7f88692d6640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88640b4810 0x7f88640a97d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f885c036040 tx=0x7f885c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.835+0000 7f8859ffb640 1 -- 172.21.15.27:0/1830143429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c043020 con 0x7f88640b4810 2024-03-06T07:22:56.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.835+0000 7f8859ffb640 1 -- 172.21.15.27:0/1830143429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c03e450 con 0x7f88640b4810 2024-03-06T07:22:56.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.835+0000 7f8859ffb640 1 -- 172.21.15.27:0/1830143429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c03e840 con 0x7f88640b4810 2024-03-06T07:22:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.836+0000 7f886b53a640 1 -- 172.21.15.27:0/1830143429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886415a8a0 con 0x7f88640b4810 2024-03-06T07:22:56.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.836+0000 7f886b53a640 1 -- 172.21.15.27:0/1830143429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886415ae60 con 0x7f88640b4810 2024-03-06T07:22:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.835+0000 7f4437fff640 1 --2- 172.21.15.27:0/4260364961 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4438067820 0x7f443815a420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4428009d90 tx=0x7f442802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b282ee9699f2bf4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.836+0000 7f4436ffd640 1 -- 172.21.15.27:0/4260364961 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428039070 con 0x7f4438067820 2024-03-06T07:22:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.837+0000 7f4436ffd640 1 -- 172.21.15.27:0/4260364961 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442802cc50 con 0x7f4438067820 2024-03-06T07:22:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.837+0000 7f4436ffd640 1 -- 172.21.15.27:0/4260364961 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428039070 con 0x7f4438067820 2024-03-06T07:22:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.838+0000 7f443e69a640 1 -- 172.21.15.27:0/4260364961 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4438067820 msgr2=0x7f443815a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.838+0000 7f443e69a640 1 --2- 172.21.15.27:0/4260364961 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4438067820 0x7f443815a420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4428009d90 tx=0x7f442802bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.840+0000 7f8859ffb640 1 -- 172.21.15.27:0/1830143429 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f885c046020 con 0x7f88640b4810 2024-03-06T07:22:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.840+0000 7f8859ffb640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f884806b5a0 0x7f884806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.841+0000 7f443e69a640 1 -- 172.21.15.27:0/4260364961 shutdown_connections 2024-03-06T07:22:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.841+0000 7f443e69a640 1 --2- 172.21.15.27:0/4260364961 >> v2:172.21.15.27:3300/0 conn(0x7f44380aee70 0x7f4438154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.841+0000 7f443e69a640 1 --2- 172.21.15.27:0/4260364961 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4438067820 0x7f443815a420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.841+0000 7f443e69a640 1 -- 172.21.15.27:0/4260364961 >> 172.21.15.27:0/4260364961 conn(0x7f4438066d00 msgr2=0x7f4438067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.841+0000 7f8859ffb640 1 -- 172.21.15.27:0/1830143429 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f885c0acdc0 con 0x7f88640b4810 2024-03-06T07:22:56.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.844+0000 7f88477fe640 1 -- 172.21.15.27:0/1830143429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8864067ff0 con 0x7f88640b4810 2024-03-06T07:22:56.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.844+0000 7f443e69a640 1 -- 172.21.15.27:0/4260364961 shutdown_connections 2024-03-06T07:22:56.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.848+0000 7f443e69a640 1 -- 172.21.15.27:0/4260364961 wait complete. 2024-03-06T07:22:56.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.850+0000 7f443e69a640 1 Processor -- start 2024-03-06T07:22:56.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.850+0000 7f443e69a640 1 -- start start 2024-03-06T07:22:56.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.851+0000 7f443e69a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4438067820 0x7f4438168f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.851+0000 7f443e69a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44380aee70 0x7f4438200bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.851+0000 7f443e69a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44380b2bd0 0x7f443815c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.851+0000 7f443e69a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44380af2f0 con 0x7f44380b2bd0 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.851+0000 7f443e69a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443806c7b0 con 0x7f4438067820 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.851+0000 7f443e69a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44380cff30 con 0x7f44380aee70 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.852+0000 7f443cc37640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44380b2bd0 0x7f443815c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.852+0000 7f443cc37640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44380b2bd0 0x7f443815c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51788/0 (socket says 172.21.15.27:51788) 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.852+0000 7f44377fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44380aee70 0x7f4438200bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.852+0000 7f4437fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4438067820 0x7f4438168f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.852+0000 7f443cc37640 1 -- 172.21.15.27:0/3775317492 learned_addr learned my addr 172.21.15.27:0/3775317492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.853+0000 7f4437fff640 1 -- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44380aee70 msgr2=0x7f4438200bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.853+0000 7f4437fff640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44380aee70 0x7f4438200bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.853+0000 7f4437fff640 1 -- 172.21.15.27:0/3775317492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44380b2bd0 msgr2=0x7f443815c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.853+0000 7f4437fff640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44380b2bd0 0x7f443815c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.853+0000 7f4437fff640 1 -- 172.21.15.27:0/3775317492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4438169630 con 0x7f4438067820 2024-03-06T07:22:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.853+0000 7f443cc37640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44380b2bd0 0x7f443815c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.854+0000 7f4437fff640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4438067820 0x7f4438168f70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4428033000 tx=0x7f4428004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.858+0000 7f4434ff9640 1 -- 172.21.15.27:0/3775317492 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428044030 con 0x7f4438067820 2024-03-06T07:22:56.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.858+0000 7f4434ff9640 1 -- 172.21.15.27:0/3775317492 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4428040660 con 0x7f4438067820 2024-03-06T07:22:56.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.858+0000 7f4434ff9640 1 -- 172.21.15.27:0/3775317492 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428039070 con 0x7f4438067820 2024-03-06T07:22:56.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.861+0000 7f8859ffb640 1 -- 172.21.15.27:0/1830143429 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f885c04d030 con 0x7f88640b4810 2024-03-06T07:22:56.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.863+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4438169910 con 0x7f4438067820 2024-03-06T07:22:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.863+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4438169ed0 con 0x7f4438067820 2024-03-06T07:22:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.868+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4438149370 con 0x7f4438067820 2024-03-06T07:22:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.874+0000 7f4434ff9640 1 -- 172.21.15.27:0/3775317492 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4428047070 con 0x7f4438067820 2024-03-06T07:22:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.874+0000 7f4434ff9640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441806b5a0 0x7f441806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.874+0000 7f4434ff9640 1 -- 172.21.15.27:0/3775317492 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f442802ca50 con 0x7f4438067820 2024-03-06T07:22:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.881+0000 7f44377fe640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441806b5a0 0x7f441806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.883+0000 7f4434ff9640 1 -- 172.21.15.27:0/3775317492 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4438149370 con 0x7f4438067820 2024-03-06T07:22:56.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.913+0000 7f44377fe640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441806b5a0 0x7f441806da60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f443815d5d0 tx=0x7f4420009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.954+0000 7f8868ad5640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f884806b5a0 0x7f884806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.958+0000 7f8868ad5640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f884806b5a0 0x7f884806da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88640cffc0 tx=0x7f8860009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.992+0000 7f4977df6640 1 Processor -- start 2024-03-06T07:22:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.992+0000 7f4977df6640 1 -- start start 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.993+0000 7f4977df6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4970072470 0x7f497016f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.993+0000 7f4977df6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4970072b80 con 0x7f49700cf650 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.993+0000 7f4977df6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49700728e0 con 0x7f497016af10 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.993+0000 7f4977df6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497016f570 con 0x7f4970072470 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.994+0000 7f4976393640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f497016af10 0x7f49700b2f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:42430/0 (socket says 172.21.15.27:42430) 2024-03-06T07:22:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.994+0000 7f4976393640 1 -- 172.21.15.27:0/567442404 learned_addr learned my addr 172.21.15.27:0/567442404 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.994+0000 7f4975391640 1 --2- 172.21.15.27:0/567442404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4970072470 0x7f497016f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.994+0000 7f4975391640 1 -- 172.21.15.27:0/567442404 >> v1:172.21.15.69:6789/0 conn(0x7f497016af10 legacy=0x7f49700b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.994+0000 7f4975391640 1 -- 172.21.15.27:0/567442404 >> v1:172.21.15.27:6789/0 conn(0x7f49700cf650 legacy=0x7f49700cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.994+0000 7f4975391640 1 -- 172.21.15.27:0/567442404 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4970070980 con 0x7f4970072470 2024-03-06T07:22:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:56.995+0000 7f4975391640 1 --2- 172.21.15.27:0/567442404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4970072470 0x7f497016f030 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f496c009750 tx=0x7f496c02c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2c4e3100d1e2ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.003+0000 7f4974b90640 1 -- 172.21.15.27:0/567442404 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c036020 con 0x7f4970072470 2024-03-06T07:22:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.003+0000 7f4974b90640 1 -- 172.21.15.27:0/567442404 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496c032880 con 0x7f4970072470 2024-03-06T07:22:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.004+0000 7f4977df6640 1 -- 172.21.15.27:0/567442404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4970072470 msgr2=0x7f497016f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.004+0000 7f4977df6640 1 --2- 172.21.15.27:0/567442404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4970072470 0x7f497016f030 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f496c009750 tx=0x7f496c02c920 comp rx=0 tx=0).stop 2024-03-06T07:22:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.004+0000 7f4977df6640 1 -- 172.21.15.27:0/567442404 shutdown_connections 2024-03-06T07:22:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.004+0000 7f4977df6640 1 --2- 172.21.15.27:0/567442404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4970072470 0x7f497016f030 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f496c009750 tx=0x7f496c02c920 comp rx=0 tx=0).stop 2024-03-06T07:22:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.004+0000 7f4977df6640 1 -- 172.21.15.27:0/567442404 >> 172.21.15.27:0/567442404 conn(0x7f4970066d00 msgr2=0x7f4970067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.069+0000 7f4977df6640 1 -- 172.21.15.27:0/567442404 shutdown_connections 2024-03-06T07:22:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.070+0000 7f4977df6640 1 -- 172.21.15.27:0/567442404 wait complete. 2024-03-06T07:22:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.072+0000 7f4977df6640 1 Processor -- start 2024-03-06T07:22:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.072+0000 7f4977df6640 1 -- start start 2024-03-06T07:22:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4977df6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49700cf650 0x7f4970153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4977df6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497016af10 0x7f4970164760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4977df6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4970166a60 0x7f49701625c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4977df6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4970070f40 con 0x7f4970166a60 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4977df6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49700b29f0 con 0x7f497016af10 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4977df6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497016b380 con 0x7f49700cf650 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4976393640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4970166a60 0x7f49701625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4976393640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4970166a60 0x7f49701625c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51792/0 (socket says 172.21.15.27:51792) 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4976393640 1 -- 172.21.15.27:0/2555260014 learned_addr learned my addr 172.21.15.27:0/2555260014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.076+0000 7f4975b92640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49700cf650 0x7f4970153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.077+0000 7f4975391640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497016af10 0x7f4970164760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.077+0000 7f4976393640 1 -- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49700cf650 msgr2=0x7f4970153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.077+0000 7f4976393640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49700cf650 0x7f4970153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.077+0000 7f4976393640 1 -- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497016af10 msgr2=0x7f4970164760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.078+0000 7f4976393640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497016af10 0x7f4970164760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.078+0000 7f4976393640 1 -- 172.21.15.27:0/2555260014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4970165060 con 0x7f4970166a60 2024-03-06T07:22:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.078+0000 7f4976393640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4970166a60 0x7f49701625c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f496800dcf0 tx=0x7f496800c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.080+0000 7f4975391640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497016af10 0x7f4970164760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.081+0000 7f49667fc640 1 -- 172.21.15.27:0/2555260014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4968010460 con 0x7f4970166a60 2024-03-06T07:22:57.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.081+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4970150dc0 con 0x7f4970166a60 2024-03-06T07:22:57.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.081+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4970165340 con 0x7f4970166a60 2024-03-06T07:22:57.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.087+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4970067ff0 con 0x7f4970166a60 2024-03-06T07:22:57.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.097+0000 7f49667fc640 1 -- 172.21.15.27:0/2555260014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4968015070 con 0x7f4970166a60 2024-03-06T07:22:57.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.097+0000 7f49667fc640 1 -- 172.21.15.27:0/2555260014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4968010860 con 0x7f4970166a60 2024-03-06T07:22:57.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.097+0000 7f49667fc640 1 -- 172.21.15.27:0/2555260014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4968010ac0 con 0x7f4970166a60 2024-03-06T07:22:57.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.097+0000 7f49667fc640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f495406b5d0 0x7f495406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.098+0000 7f49667fc640 1 -- 172.21.15.27:0/2555260014 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f496808aba0 con 0x7f4970166a60 2024-03-06T07:22:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.098+0000 7f49667fc640 1 -- 172.21.15.27:0/2555260014 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f496808f390 con 0x7f4970166a60 2024-03-06T07:22:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.098+0000 7f4975b92640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f495406b5d0 0x7f495406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.104+0000 7f4975b92640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f495406b5d0 0x7f495406da90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4960000f60 tx=0x7f4960007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.146+0000 7f252c9ae640 1 -- 172.21.15.27:0/189774497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f250c005580 con 0x7f252815dc30 2024-03-06T07:22:57.149 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-03-06T07:22:57.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.147+0000 7f2516ffd640 1 -- 172.21.15.27:0/189774497 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f252007d470 con 0x7f252815dc30 2024-03-06T07:22:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 -- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f250806b520 msgr2=0x7f250806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f250806b520 0x7f250806d9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f251c009670 tx=0x7f251c006bd0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 -- 172.21.15.27:0/189774497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f252815dc30 msgr2=0x7f252814f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f252815dc30 0x7f252814f6d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2520037040 tx=0x7f2520004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 -- 172.21.15.27:0/189774497 shutdown_connections 2024-03-06T07:22:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f250806b520 0x7f250806d9e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f252815dc30 0x7f252814f6d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f252815b5a0 0x7f25281e4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 --2- 172.21.15.27:0/189774497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f252815a660 0x7f2528152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 -- 172.21.15.27:0/189774497 >> 172.21.15.27:0/189774497 conn(0x7f2528071650 msgr2=0x7f252815e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.151+0000 7f2514ff9640 1 -- 172.21.15.27:0/189774497 shutdown_connections 2024-03-06T07:22:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.152+0000 7f2514ff9640 1 -- 172.21.15.27:0/189774497 wait complete. 2024-03-06T07:22:57.171 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.1 2024-03-06T07:22:57.171 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:57.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.179+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f11b0005580 con 0x7f11b4005210 2024-03-06T07:22:57.185 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.179+0000 7f11be7fc640 1 -- 172.21.15.27:0/1858127277 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f11c4059380 con 0x7f11b4005210 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11ac06b530 msgr2=0x7f11ac06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11ac06b530 0x7f11ac06d9f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f11b8006fd0 tx=0x7f11b8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 msgr2=0x7f11c00d6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11c00d6610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f11c400d4e0 tx=0x7f11c400d9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 shutdown_connections 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11ac06b530 0x7f11ac06d9f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c00e15e0 0x7f11c00e1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c00d6d00 0x7f11c00e37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 --2- 172.21.15.27:0/1858127277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11b4005210 0x7f11c00d6610 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.183+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 >> 172.21.15.27:0/1858127277 conn(0x7f11c0053740 msgr2=0x7f11c0053b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.184+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 shutdown_connections 2024-03-06T07:22:57.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.184+0000 7f11ce87d640 1 -- 172.21.15.27:0/1858127277 wait complete. 2024-03-06T07:22:57.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.0 2024-03-06T07:22:57.204 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:57.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.207+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0e00005580 con 0x7f0e20154400 2024-03-06T07:22:57.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.207+0000 7f0e07fff640 1 -- 172.21.15.27:0/1210650629 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0e180581c0 con 0x7f0e20154400 2024-03-06T07:22:57.210 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-06T07:22:57.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.209+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0c06b5a0 msgr2=0x7f0e0c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.209+0000 7f0e2593a640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0c06b5a0 0x7f0e0c06da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009670 tx=0x7f0e14006bd0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e20154400 msgr2=0x7f0e2014d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e20154400 0x7f0e2014d1e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0e18001950 tx=0x7f0e18001e10 comp rx=0 tx=0).stop 2024-03-06T07:22:57.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 shutdown_connections 2024-03-06T07:22:57.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0c06b5a0 0x7f0e0c06da60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e2015a420 0x7f0e202000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e20154400 0x7f0e2014d1e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 --2- 172.21.15.27:0/1210650629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e200cff30 0x7f0e2015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 >> 172.21.15.27:0/1210650629 conn(0x7f0e20066d00 msgr2=0x7f0e2006d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 shutdown_connections 2024-03-06T07:22:57.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.210+0000 7f0e2593a640 1 -- 172.21.15.27:0/1210650629 wait complete. 2024-03-06T07:22:57.223 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.7 2024-03-06T07:22:57.287 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.236+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6bb0005850 con 0x7f6bd00b2c30 2024-03-06T07:22:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.237+0000 7f6bb7fff640 1 -- 172.21.15.27:0/1771674618 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6bd0165c40 con 0x7f6bd00b2c30 2024-03-06T07:22:57.288 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bbc06b520 msgr2=0x7f6bbc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bbc06b520 0x7f6bbc06d9e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009930 tx=0x7f6bc0009060 comp rx=0 tx=0).stop 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b2c30 msgr2=0x7f6bd01638d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b2c30 0x7f6bd01638d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b2c0 tx=0x7f6bc8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 shutdown_connections 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bbc06b520 0x7f6bbc06d9e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015c830 0x7f6bd0163e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b2c30 0x7f6bd01638d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 --2- 172.21.15.27:0/1771674618 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd006c710 0x7f6bd014ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 >> 172.21.15.27:0/1771674618 conn(0x7f6bd0066d00 msgr2=0x7f6bd015af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 shutdown_connections 2024-03-06T07:22:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.239+0000 7f6bd5e15640 1 -- 172.21.15.27:0/1771674618 wait complete. 2024-03-06T07:22:57.291 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.2 2024-03-06T07:22:57.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.264+0000 7f1ad2ffd640 1 -- 172.21.15.27:0/2986379082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1ac8005140 con 0x7f1ae80b3920 2024-03-06T07:22:57.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.264+0000 7f1ae57fa640 1 -- 172.21.15.27:0/2986379082 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1adc05c0f0 con 0x7f1ae80b3920 2024-03-06T07:22:57.291 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:57.291 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-06T07:22:57.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 -- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b640 msgr2=0x7f1ad806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b640 0x7f1ad806db00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4036000 tx=0x7f1ad4008030 comp rx=0 tx=0).stop 2024-03-06T07:22:57.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 -- 172.21.15.27:0/2986379082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80b3920 msgr2=0x7f1ae8200200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80b3920 0x7f1ae8200200 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00c780 tx=0x7f1adc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 -- 172.21.15.27:0/2986379082 shutdown_connections 2024-03-06T07:22:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b640 0x7f1ad806db00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ae81620f0 0x7f1ae81598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae80cde20 0x7f1ae815f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 --2- 172.21.15.27:0/2986379082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae80b3920 0x7f1ae8200200 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.267+0000 7f1ad2ffd640 1 -- 172.21.15.27:0/2986379082 >> 172.21.15.27:0/2986379082 conn(0x7f1ae8066d00 msgr2=0x7f1ae80b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.268+0000 7f1ad2ffd640 1 -- 172.21.15.27:0/2986379082 shutdown_connections 2024-03-06T07:22:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.268+0000 7f1ad2ffd640 1 -- 172.21.15.27:0/2986379082 wait complete. 2024-03-06T07:22:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.275+0000 7f88477fe640 1 -- 172.21.15.27:0/1830143429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f88640b3af0 con 0x7f88640b4810 2024-03-06T07:22:57.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.275+0000 7f8859ffb640 1 -- 172.21.15.27:0/1830143429 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f885c07bb90 con 0x7f88640b4810 2024-03-06T07:22:57.294 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-06T07:22:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 -- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f884806b5a0 msgr2=0x7f884806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f884806b5a0 0x7f884806da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88640cffc0 tx=0x7f8860009190 comp rx=0 tx=0).stop 2024-03-06T07:22:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 -- 172.21.15.27:0/1830143429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88640b4810 msgr2=0x7f88640a97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88640b4810 0x7f88640a97d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f885c036040 tx=0x7f885c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 -- 172.21.15.27:0/1830143429 shutdown_connections 2024-03-06T07:22:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f884806b5a0 0x7f884806da60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f886415fed0 0x7f8864159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f886415faf0 0x7f88640a9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 --2- 172.21.15.27:0/1830143429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88640b4810 0x7f88640a97d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.278+0000 7f88477fe640 1 -- 172.21.15.27:0/1830143429 >> 172.21.15.27:0/1830143429 conn(0x7f8864066d00 msgr2=0x7f8864163af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.296 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.3 2024-03-06T07:22:57.297 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:57.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.279+0000 7f88477fe640 1 -- 172.21.15.27:0/1830143429 shutdown_connections 2024-03-06T07:22:57.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.279+0000 7f88477fe640 1 -- 172.21.15.27:0/1830143429 wait complete. 2024-03-06T07:22:57.297 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.4 2024-03-06T07:22:57.298 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:57.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.332+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f4438067ff0 con 0x7f4438067820 2024-03-06T07:22:57.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.332+0000 7f4434ff9640 1 -- 172.21.15.27:0/3775317492 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f442804e070 con 0x7f4438067820 2024-03-06T07:22:57.334 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441806b5a0 msgr2=0x7f441806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441806b5a0 0x7f441806da60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f443815d5d0 tx=0x7f4420009110 comp rx=0 tx=0).stop 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4438067820 msgr2=0x7f4438168f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4438067820 0x7f4438168f70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4428033000 tx=0x7f4428004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 shutdown_connections 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441806b5a0 0x7f441806da60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44380b2bd0 0x7f443815c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44380aee70 0x7f4438200bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 --2- 172.21.15.27:0/3775317492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4438067820 0x7f4438168f70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 >> 172.21.15.27:0/3775317492 conn(0x7f4438066d00 msgr2=0x7f4438148c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 shutdown_connections 2024-03-06T07:22:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.335+0000 7f443e69a640 1 -- 172.21.15.27:0/3775317492 wait complete. 2024-03-06T07:22:57.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.6 2024-03-06T07:22:57.354 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.369+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f497016d9f0 con 0x7f4970166a60 2024-03-06T07:22:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.369+0000 7f49667fc640 1 -- 172.21.15.27:0/2555260014 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4968059a90 con 0x7f4970166a60 2024-03-06T07:22:57.371 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f495406b5d0 msgr2=0x7f495406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f495406b5d0 0x7f495406da90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4960000f60 tx=0x7f4960007e30 comp rx=0 tx=0).stop 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4970166a60 msgr2=0x7f49701625c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4970166a60 0x7f49701625c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f496800dcf0 tx=0x7f496800c640 comp rx=0 tx=0).stop 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 shutdown_connections 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f495406b5d0 0x7f495406da90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4970166a60 0x7f49701625c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497016af10 0x7f4970164760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 --2- 172.21.15.27:0/2555260014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f49700cf650 0x7f4970153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 >> 172.21.15.27:0/2555260014 conn(0x7f4970066d00 msgr2=0x7f497016c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 shutdown_connections 2024-03-06T07:22:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.371+0000 7f4977df6640 1 -- 172.21.15.27:0/2555260014 wait complete. 2024-03-06T07:22:57.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.5 2024-03-06T07:22:57.384 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:57.384 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:22:57.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:57.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.529+0000 7f8ca1565640 1 Processor -- start 2024-03-06T07:22:57.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.529+0000 7f8ca1565640 1 -- start start 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.530+0000 7f8ca1565640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8c9c15dca0 0x7f8c9c06fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.530+0000 7f8ca1565640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c9c0b00c0 0x7f8c9c0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.530+0000 7f8ca1565640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9c161c20 con 0x7f8c9c15e080 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.530+0000 7f8ca1565640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c9c1619d0 con 0x7f8c9c0b00c0 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.530+0000 7f8ca1565640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c9c161f00 con 0x7f8c9c15dca0 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.530+0000 7f8c9b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c9c0b00c0 0x7f8c9c0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.530+0000 7f8c9b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c9c0b00c0 0x7f8c9c0b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51808/0 (socket says 172.21.15.27:51808) 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.530+0000 7f8c9b7fe640 1 -- 172.21.15.27:0/601940375 learned_addr learned my addr 172.21.15.27:0/601940375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.531+0000 7f8c9affd640 1 --2- 172.21.15.27:0/601940375 >> v2:172.21.15.69:3300/0 conn(0x7f8c9c15dca0 0x7f8c9c06fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.531+0000 7f8c9b7fe640 1 -- 172.21.15.27:0/601940375 >> v2:172.21.15.69:3300/0 conn(0x7f8c9c15dca0 msgr2=0x7f8c9c06fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.531+0000 7f8c9b7fe640 1 --2- 172.21.15.27:0/601940375 >> v2:172.21.15.69:3300/0 conn(0x7f8c9c15dca0 0x7f8c9c06fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.531+0000 7f8c9b7fe640 1 -- 172.21.15.27:0/601940375 >> v1:172.21.15.69:6789/0 conn(0x7f8c9c15e080 legacy=0x7f8c9c066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.531+0000 7f8c9b7fe640 1 -- 172.21.15.27:0/601940375 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c1621e0 con 0x7f8c9c0b00c0 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.531+0000 7f8c9affd640 1 --2- 172.21.15.27:0/601940375 >> v2:172.21.15.69:3300/0 conn(0x7f8c9c15dca0 0x7f8c9c06fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8c9b7fe640 1 --2- 172.21.15.27:0/601940375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b00c0 0x7f8c9c0b3ca0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c06c050 tx=0x7f8c9402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73687dd2d499169e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8c9a7fc640 1 -- 172.21.15.27:0/601940375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9402ca80 con 0x7f8c9c0b00c0 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8c9a7fc640 1 -- 172.21.15.27:0/601940375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9402cc20 con 0x7f8c9c0b00c0 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8c9a7fc640 1 -- 172.21.15.27:0/601940375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c940343f0 con 0x7f8c9c0b00c0 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8ca1565640 1 -- 172.21.15.27:0/601940375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b00c0 msgr2=0x7f8c9c0b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8ca1565640 1 --2- 172.21.15.27:0/601940375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b00c0 0x7f8c9c0b3ca0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c06c050 tx=0x7f8c9402bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8ca1565640 1 -- 172.21.15.27:0/601940375 shutdown_connections 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8ca1565640 1 --2- 172.21.15.27:0/601940375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b00c0 0x7f8c9c0b3ca0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8ca1565640 1 --2- 172.21.15.27:0/601940375 >> v2:172.21.15.69:3300/0 conn(0x7f8c9c15dca0 0x7f8c9c06fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8ca1565640 1 -- 172.21.15.27:0/601940375 >> 172.21.15.27:0/601940375 conn(0x7f8c9c0d0990 msgr2=0x7f8c9c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8ca1565640 1 -- 172.21.15.27:0/601940375 shutdown_connections 2024-03-06T07:22:57.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.532+0000 7f8ca1565640 1 -- 172.21.15.27:0/601940375 wait complete. 2024-03-06T07:22:57.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.533+0000 7f8ca1565640 1 Processor -- start 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.534+0000 7f8ca1565640 1 -- start start 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.534+0000 7f8ca1565640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c9c0b00c0 0x7f8c9c1603f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.534+0000 7f8ca1565640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c9c15dca0 0x7f8c9c15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.534+0000 7f8ca1565640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b1d80 0x7f8c9c1e7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.534+0000 7f8ca1565640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c0703d0 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.534+0000 7f8ca1565640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c070250 con 0x7f8c9c15dca0 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.534+0000 7f8ca1565640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c067410 con 0x7f8c9c0b00c0 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b1d80 0x7f8c9c1e7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b1d80 0x7f8c9c1e7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51820/0 (socket says 172.21.15.27:51820) 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9b7fe640 1 -- 172.21.15.27:0/512107862 learned_addr learned my addr 172.21.15.27:0/512107862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c927fc640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c9c15dca0 0x7f8c9c15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9affd640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c9c0b00c0 0x7f8c9c1603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9b7fe640 1 -- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c9c0b00c0 msgr2=0x7f8c9c1603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9b7fe640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c9c0b00c0 0x7f8c9c1603f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9b7fe640 1 -- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c9c15dca0 msgr2=0x7f8c9c15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9b7fe640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c9c15dca0 0x7f8c9c15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9b7fe640 1 -- 172.21.15.27:0/512107862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c0cdb40 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c9b7fe640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b1d80 0x7f8c9c1e7790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8c94009f40 tx=0x7f8c94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c93fff640 1 -- 172.21.15.27:0/512107862 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94043020 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c93fff640 1 -- 172.21.15.27:0/512107862 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9403e450 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8c93fff640 1 -- 172.21.15.27:0/512107862 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94009510 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c9c15f3c0 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.535+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c9c0ce380 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.537+0000 7f8c93fff640 1 -- 172.21.15.27:0/512107862 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c94046020 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.537+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c9c068700 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.537+0000 7f8c93fff640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7c06b4d0 0x7f8c7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.537+0000 7f8c93fff640 1 -- 172.21.15.27:0/512107862 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c940ac630 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.537+0000 7f8c9affd640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7c06b4d0 0x7f8c7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.538+0000 7f8c9affd640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7c06b4d0 0x7f8c7c06d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c006fd0 tx=0x7f8c8c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.541+0000 7f8c93fff640 1 -- 172.21.15.27:0/512107862 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c9404d070 con 0x7f8c9c0b1d80 2024-03-06T07:22:57.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.719+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c9c15d7b0 con 0x7f8c7c06b4d0 2024-03-06T07:22:57.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.725+0000 7f8c93fff640 1 -- 172.21.15.27:0/512107862 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251836 (secure 0 0 0) 0x7f8c9c15d7b0 con 0x7f8c7c06b4d0 2024-03-06T07:22:57.731 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7c06b4d0 msgr2=0x7f8c7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7c06b4d0 0x7f8c7c06d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c006fd0 tx=0x7f8c8c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b1d80 msgr2=0x7f8c9c1e7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b1d80 0x7f8c9c1e7790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8c94009f40 tx=0x7f8c94004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 shutdown_connections 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7c06b4d0 0x7f8c7c06d990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c9c0b1d80 0x7f8c9c1e7790 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c9c15dca0 0x7f8c9c15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 --2- 172.21.15.27:0/512107862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c9c0b00c0 0x7f8c9c1603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 >> 172.21.15.27:0/512107862 conn(0x7f8c9c0d0990 msgr2=0x7f8c9c0ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 shutdown_connections 2024-03-06T07:22:57.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.728+0000 7f8ca1565640 1 -- 172.21.15.27:0/512107862 wait complete. 2024-03-06T07:22:57.741 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-06T07:22:55.988054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219096,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755624,"statfs":{"total":773094113280,"available":772869758976,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.935264"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079733+0000","last_change":"2024-03-06T07:22:49.080233+0000","last_active":"2024-03-06T07:22:50.079733+0000","last_peered":"2024-03-06T07:22:50.079733+0000","last_clean":"2024-03-06T07:22:50.079733+0000","last_became_active":"2024-03-06T07:22:49.079770+0000","last_became_peered":"2024-03-06T07:22:49.079770+0000","last_unstale":"2024-03-06T07:22:50.079733+0000","last_undegraded":"2024-03-06T07:22:50.079733+0000","last_fullsized":"2024-03-06T07:22:50.079733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:32.043873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077134+0000","last_change":"2024-03-06T07:22:49.081636+0000","last_active":"2024-03-06T07:22:50.077134+0000","last_peered":"2024-03-06T07:22:50.077134+0000","last_clean":"2024-03-06T07:22:50.077134+0000","last_became_active":"2024-03-06T07:22:49.081356+0000","last_became_peered":"2024-03-06T07:22:49.081356+0000","last_unstale":"2024-03-06T07:22:50.077134+0000","last_undegraded":"2024-03-06T07:22:50.077134+0000","last_fullsized":"2024-03-06T07:22:50.077134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677913+0000","last_change":"2024-03-06T07:22:49.076314+0000","last_active":"2024-03-06T07:22:51.677913+0000","last_peered":"2024-03-06T07:22:51.677913+0000","last_clean":"2024-03-06T07:22:51.677913+0000","last_became_active":"2024-03-06T07:22:49.075901+0000","last_became_peered":"2024-03-06T07:22:49.075901+0000","last_unstale":"2024-03-06T07:22:51.677913+0000","last_undegraded":"2024-03-06T07:22:51.677913+0000","last_fullsized":"2024-03-06T07:22:51.677913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670047+0000","last_change":"2024-03-06T07:22:49.086730+0000","last_active":"2024-03-06T07:22:50.670047+0000","last_peered":"2024-03-06T07:22:50.670047+0000","last_clean":"2024-03-06T07:22:50.670047+0000","last_became_active":"2024-03-06T07:22:49.086475+0000","last_became_peered":"2024-03-06T07:22:49.086475+0000","last_unstale":"2024-03-06T07:22:50.670047+0000","last_undegraded":"2024-03-06T07:22:50.670047+0000","last_fullsized":"2024-03-06T07:22:50.670047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:29.588686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079650+0000","last_change":"2024-03-06T07:22:49.074936+0000","last_active":"2024-03-06T07:22:50.079650+0000","last_peered":"2024-03-06T07:22:50.079650+0000","last_clean":"2024-03-06T07:22:50.079650+0000","last_became_active":"2024-03-06T07:22:49.074662+0000","last_became_peered":"2024-03-06T07:22:49.074662+0000","last_unstale":"2024-03-06T07:22:50.079650+0000","last_undegraded":"2024-03-06T07:22:50.079650+0000","last_fullsized":"2024-03-06T07:22:50.079650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:32.401483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669865+0000","last_change":"2024-03-06T07:22:49.081940+0000","last_active":"2024-03-06T07:22:50.669865+0000","last_peered":"2024-03-06T07:22:50.669865+0000","last_clean":"2024-03-06T07:22:50.669865+0000","last_became_active":"2024-03-06T07:22:49.081668+0000","last_became_peered":"2024-03-06T07:22:49.081668+0000","last_unstale":"2024-03-06T07:22:50.669865+0000","last_undegraded":"2024-03-06T07:22:50.669865+0000","last_fullsized":"2024-03-06T07:22:50.669865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:58.929359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079272+0000","last_change":"2024-03-06T07:22:49.088117+0000","last_active":"2024-03-06T07:22:50.079272+0000","last_peered":"2024-03-06T07:22:50.079272+0000","last_clean":"2024-03-06T07:22:50.079272+0000","last_became_active":"2024-03-06T07:22:49.087856+0000","last_became_peered":"2024-03-06T07:22:49.087856+0000","last_unstale":"2024-03-06T07:22:50.079272+0000","last_undegraded":"2024-03-06T07:22:50.079272+0000","last_fullsized":"2024-03-06T07:22:50.079272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:43.569514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.676835+0000","last_change":"2024-03-06T07:22:49.074570+0000","last_active":"2024-03-06T07:22:51.676835+0000","last_peered":"2024-03-06T07:22:51.676835+0000","last_clean":"2024-03-06T07:22:51.676835+0000","last_became_active":"2024-03-06T07:22:49.073115+0000","last_became_peered":"2024-03-06T07:22:49.073115+0000","last_unstale":"2024-03-06T07:22:51.676835+0000","last_undegraded":"2024-03-06T07:22:51.676835+0000","last_fullsized":"2024-03-06T07:22:51.676835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.031324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.080031+0000","last_change":"2024-03-06T07:22:49.073716+0000","last_active":"2024-03-06T07:22:50.080031+0000","last_peered":"2024-03-06T07:22:50.080031+0000","last_clean":"2024-03-06T07:22:50.080031+0000","last_became_active":"2024-03-06T07:22:49.073434+0000","last_became_peered":"2024-03-06T07:22:49.073434+0000","last_unstale":"2024-03-06T07:22:50.080031+0000","last_undegraded":"2024-03-06T07:22:50.080031+0000","last_fullsized":"2024-03-06T07:22:50.080031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:33.120440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620829+0000","last_change":"2024-03-06T07:22:49.072470+0000","last_active":"2024-03-06T07:22:50.620829+0000","last_peered":"2024-03-06T07:22:50.620829+0000","last_clean":"2024-03-06T07:22:50.620829+0000","last_became_active":"2024-03-06T07:22:49.072206+0000","last_became_peered":"2024-03-06T07:22:49.072206+0000","last_unstale":"2024-03-06T07:22:50.620829+0000","last_undegraded":"2024-03-06T07:22:50.620829+0000","last_fullsized":"2024-03-06T07:22:50.620829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620623+0000","last_change":"2024-03-06T07:22:49.076507+0000","last_active":"2024-03-06T07:22:50.620623+0000","last_peered":"2024-03-06T07:22:50.620623+0000","last_clean":"2024-03-06T07:22:50.620623+0000","last_became_active":"2024-03-06T07:22:49.076054+0000","last_became_peered":"2024-03-06T07:22:49.076054+0000","last_unstale":"2024-03-06T07:22:50.620623+0000","last_undegraded":"2024-03-06T07:22:50.620623+0000","last_fullsized":"2024-03-06T07:22:50.620623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078932+0000","last_change":"2024-03-06T07:22:49.077078+0000","last_active":"2024-03-06T07:22:50.078932+0000","last_peered":"2024-03-06T07:22:50.078932+0000","last_clean":"2024-03-06T07:22:50.078932+0000","last_became_active":"2024-03-06T07:22:49.076757+0000","last_became_peered":"2024-03-06T07:22:49.076757+0000","last_unstale":"2024-03-06T07:22:50.078932+0000","last_undegraded":"2024-03-06T07:22:50.078932+0000","last_fullsized":"2024-03-06T07:22:50.078932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677438+0000","last_change":"2024-03-06T07:22:49.073467+0000","last_active":"2024-03-06T07:22:51.677438+0000","last_peered":"2024-03-06T07:22:51.677438+0000","last_clean":"2024-03-06T07:22:51.677438+0000","last_became_active":"2024-03-06T07:22:49.072969+0000","last_became_peered":"2024-03-06T07:22:49.072969+0000","last_unstale":"2024-03-06T07:22:51.677438+0000","last_undegraded":"2024-03-06T07:22:51.677438+0000","last_fullsized":"2024-03-06T07:22:51.677438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677469+0000","last_change":"2024-03-06T07:22:49.076207+0000","last_active":"2024-03-06T07:22:51.677469+0000","last_peered":"2024-03-06T07:22:51.677469+0000","last_clean":"2024-03-06T07:22:51.677469+0000","last_became_active":"2024-03-06T07:22:49.074449+0000","last_became_peered":"2024-03-06T07:22:49.074449+0000","last_unstale":"2024-03-06T07:22:51.677469+0000","last_undegraded":"2024-03-06T07:22:51.677469+0000","last_fullsized":"2024-03-06T07:22:51.677469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076881+0000","last_change":"2024-03-06T07:22:49.072745+0000","last_active":"2024-03-06T07:22:50.076881+0000","last_peered":"2024-03-06T07:22:50.076881+0000","last_clean":"2024-03-06T07:22:50.076881+0000","last_became_active":"2024-03-06T07:22:49.072443+0000","last_became_peered":"2024-03-06T07:22:49.072443+0000","last_unstale":"2024-03-06T07:22:50.076881+0000","last_undegraded":"2024-03-06T07:22:50.076881+0000","last_fullsized":"2024-03-06T07:22:50.076881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669789+0000","last_change":"2024-03-06T07:22:49.085335+0000","last_active":"2024-03-06T07:22:50.669789+0000","last_peered":"2024-03-06T07:22:50.669789+0000","last_clean":"2024-03-06T07:22:50.669789+0000","last_became_active":"2024-03-06T07:22:49.085023+0000","last_became_peered":"2024-03-06T07:22:49.085023+0000","last_unstale":"2024-03-06T07:22:50.669789+0000","last_undegraded":"2024-03-06T07:22:50.669789+0000","last_fullsized":"2024-03-06T07:22:50.669789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670606+0000","last_change":"2024-03-06T07:22:49.073395+0000","last_active":"2024-03-06T07:22:50.670606+0000","last_peered":"2024-03-06T07:22:50.670606+0000","last_clean":"2024-03-06T07:22:50.670606+0000","last_became_active":"2024-03-06T07:22:49.073105+0000","last_became_peered":"2024-03-06T07:22:49.073105+0000","last_unstale":"2024-03-06T07:22:50.670606+0000","last_undegraded":"2024-03-06T07:22:50.670606+0000","last_fullsized":"2024-03-06T07:22:50.670606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:16.151911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677340+0000","last_change":"2024-03-06T07:22:49.074912+0000","last_active":"2024-03-06T07:22:51.677340+0000","last_peered":"2024-03-06T07:22:51.677340+0000","last_clean":"2024-03-06T07:22:51.677340+0000","last_became_active":"2024-03-06T07:22:49.074638+0000","last_became_peered":"2024-03-06T07:22:49.074638+0000","last_unstale":"2024-03-06T07:22:51.677340+0000","last_undegraded":"2024-03-06T07:22:51.677340+0000","last_fullsized":"2024-03-06T07:22:51.677340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079918+0000","last_change":"2024-03-06T07:22:49.075473+0000","last_active":"2024-03-06T07:22:50.079918+0000","last_peered":"2024-03-06T07:22:50.079918+0000","last_clean":"2024-03-06T07:22:50.079918+0000","last_became_active":"2024-03-06T07:22:49.074945+0000","last_became_peered":"2024-03-06T07:22:49.074945+0000","last_unstale":"2024-03-06T07:22:50.079918+0000","last_undegraded":"2024-03-06T07:22:50.079918+0000","last_fullsized":"2024-03-06T07:22:50.079918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:16.151911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620534+0000","last_change":"2024-03-06T07:22:49.085348+0000","last_active":"2024-03-06T07:22:50.620534+0000","last_peered":"2024-03-06T07:22:50.620534+0000","last_clean":"2024-03-06T07:22:50.620534+0000","last_became_active":"2024-03-06T07:22:49.084955+0000","last_became_peered":"2024-03-06T07:22:49.084955+0000","last_unstale":"2024-03-06T07:22:50.620534+0000","last_undegraded":"2024-03-06T07:22:50.620534+0000","last_fullsized":"2024-03-06T07:22:50.620534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677814+0000","last_change":"2024-03-06T07:22:49.081034+0000","last_active":"2024-03-06T07:22:51.677814+0000","last_peered":"2024-03-06T07:22:51.677814+0000","last_clean":"2024-03-06T07:22:51.677814+0000","last_became_active":"2024-03-06T07:22:49.080639+0000","last_became_peered":"2024-03-06T07:22:49.080639+0000","last_unstale":"2024-03-06T07:22:51.677814+0000","last_undegraded":"2024-03-06T07:22:51.677814+0000","last_fullsized":"2024-03-06T07:22:51.677814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.031324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670034+0000","last_change":"2024-03-06T07:22:49.074317+0000","last_active":"2024-03-06T07:22:50.670034+0000","last_peered":"2024-03-06T07:22:50.670034+0000","last_clean":"2024-03-06T07:22:50.670034+0000","last_became_active":"2024-03-06T07:22:49.073041+0000","last_became_peered":"2024-03-06T07:22:49.073041+0000","last_unstale":"2024-03-06T07:22:50.670034+0000","last_undegraded":"2024-03-06T07:22:50.670034+0000","last_fullsized":"2024-03-06T07:22:50.670034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:18.288459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677267+0000","last_change":"2024-03-06T07:22:49.076771+0000","last_active":"2024-03-06T07:22:51.677267+0000","last_peered":"2024-03-06T07:22:51.677267+0000","last_clean":"2024-03-06T07:22:51.677267+0000","last_became_active":"2024-03-06T07:22:49.073031+0000","last_became_peered":"2024-03-06T07:22:49.073031+0000","last_unstale":"2024-03-06T07:22:51.677267+0000","last_undegraded":"2024-03-06T07:22:51.677267+0000","last_fullsized":"2024-03-06T07:22:51.677267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669877+0000","last_change":"2024-03-06T07:22:49.086108+0000","last_active":"2024-03-06T07:22:50.669877+0000","last_peered":"2024-03-06T07:22:50.669877+0000","last_clean":"2024-03-06T07:22:50.669877+0000","last_became_active":"2024-03-06T07:22:49.085825+0000","last_became_peered":"2024-03-06T07:22:49.085825+0000","last_unstale":"2024-03-06T07:22:50.669877+0000","last_undegraded":"2024-03-06T07:22:50.669877+0000","last_fullsized":"2024-03-06T07:22:50.669877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:07.781375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678152+0000","last_change":"2024-03-06T07:22:49.071422+0000","last_active":"2024-03-06T07:22:51.678152+0000","last_peered":"2024-03-06T07:22:51.678152+0000","last_clean":"2024-03-06T07:22:51.678152+0000","last_became_active":"2024-03-06T07:22:49.071117+0000","last_became_peered":"2024-03-06T07:22:49.071117+0000","last_unstale":"2024-03-06T07:22:51.678152+0000","last_undegraded":"2024-03-06T07:22:51.678152+0000","last_fullsized":"2024-03-06T07:22:51.678152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:18.288459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620872+0000","last_change":"2024-03-06T07:22:49.077927+0000","last_active":"2024-03-06T07:22:50.620872+0000","last_peered":"2024-03-06T07:22:50.620872+0000","last_clean":"2024-03-06T07:22:50.620872+0000","last_became_active":"2024-03-06T07:22:49.077647+0000","last_became_peered":"2024-03-06T07:22:49.077647+0000","last_unstale":"2024-03-06T07:22:50.620872+0000","last_undegraded":"2024-03-06T07:22:50.620872+0000","last_fullsized":"2024-03-06T07:22:50.620872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:58.929359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077021+0000","last_change":"2024-03-06T07:22:49.085345+0000","last_active":"2024-03-06T07:22:50.077021+0000","last_peered":"2024-03-06T07:22:50.077021+0000","last_clean":"2024-03-06T07:22:50.077021+0000","last_became_active":"2024-03-06T07:22:49.084942+0000","last_became_peered":"2024-03-06T07:22:49.084942+0000","last_unstale":"2024-03-06T07:22:50.077021+0000","last_undegraded":"2024-03-06T07:22:50.077021+0000","last_fullsized":"2024-03-06T07:22:50.077021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677841+0000","last_change":"2024-03-06T07:22:49.071314+0000","last_active":"2024-03-06T07:22:51.677841+0000","last_peered":"2024-03-06T07:22:51.677841+0000","last_clean":"2024-03-06T07:22:51.677841+0000","last_became_active":"2024-03-06T07:22:49.070986+0000","last_became_peered":"2024-03-06T07:22:49.070986+0000","last_unstale":"2024-03-06T07:22:51.677841+0000","last_undegraded":"2024-03-06T07:22:51.677841+0000","last_fullsized":"2024-03-06T07:22:51.677841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:58.929359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677884+0000","last_change":"2024-03-06T07:22:49.077320+0000","last_active":"2024-03-06T07:22:51.677884+0000","last_peered":"2024-03-06T07:22:51.677884+0000","last_clean":"2024-03-06T07:22:51.677884+0000","last_became_active":"2024-03-06T07:22:49.077017+0000","last_became_peered":"2024-03-06T07:22:49.077017+0000","last_unstale":"2024-03-06T07:22:51.677884+0000","last_undegraded":"2024-03-06T07:22:51.677884+0000","last_fullsized":"2024-03-06T07:22:51.677884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670057+0000","last_change":"2024-03-06T07:22:49.086200+0000","last_active":"2024-03-06T07:22:50.670057+0000","last_peered":"2024-03-06T07:22:50.670057+0000","last_clean":"2024-03-06T07:22:50.670057+0000","last_became_active":"2024-03-06T07:22:49.085793+0000","last_became_peered":"2024-03-06T07:22:49.085793+0000","last_unstale":"2024-03-06T07:22:50.670057+0000","last_undegraded":"2024-03-06T07:22:50.670057+0000","last_fullsized":"2024-03-06T07:22:50.670057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:43.569514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677199+0000","last_change":"2024-03-06T07:22:49.076891+0000","last_active":"2024-03-06T07:22:51.677199+0000","last_peered":"2024-03-06T07:22:51.677199+0000","last_clean":"2024-03-06T07:22:51.677199+0000","last_became_active":"2024-03-06T07:22:49.076610+0000","last_became_peered":"2024-03-06T07:22:49.076610+0000","last_unstale":"2024-03-06T07:22:51.677199+0000","last_undegraded":"2024-03-06T07:22:51.677199+0000","last_fullsized":"2024-03-06T07:22:51.677199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620616+0000","last_change":"2024-03-06T07:22:49.088635+0000","last_active":"2024-03-06T07:22:50.620616+0000","last_peered":"2024-03-06T07:22:50.620616+0000","last_clean":"2024-03-06T07:22:50.620616+0000","last_became_active":"2024-03-06T07:22:49.088352+0000","last_became_peered":"2024-03-06T07:22:49.088352+0000","last_unstale":"2024-03-06T07:22:50.620616+0000","last_undegraded":"2024-03-06T07:22:50.620616+0000","last_fullsized":"2024-03-06T07:22:50.620616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078869+0000","last_change":"2024-03-06T07:22:49.071168+0000","last_active":"2024-03-06T07:22:50.078869+0000","last_peered":"2024-03-06T07:22:50.078869+0000","last_clean":"2024-03-06T07:22:50.078869+0000","last_became_active":"2024-03-06T07:22:49.070861+0000","last_became_peered":"2024-03-06T07:22:49.070861+0000","last_unstale":"2024-03-06T07:22:50.078869+0000","last_undegraded":"2024-03-06T07:22:50.078869+0000","last_fullsized":"2024-03-06T07:22:50.078869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:51.096875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677602+0000","last_change":"2024-03-06T07:22:48.076442+0000","last_active":"2024-03-06T07:22:51.677602+0000","last_peered":"2024-03-06T07:22:51.677602+0000","last_clean":"2024-03-06T07:22:51.677602+0000","last_became_active":"2024-03-06T07:22:48.076193+0000","last_became_peered":"2024-03-06T07:22:48.076193+0000","last_unstale":"2024-03-06T07:22:51.677602+0000","last_undegraded":"2024-03-06T07:22:51.677602+0000","last_fullsized":"2024-03-06T07:22:51.677602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:48.118117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677974+0000","last_change":"2024-03-06T07:22:49.067928+0000","last_active":"2024-03-06T07:22:51.677974+0000","last_peered":"2024-03-06T07:22:51.677974+0000","last_clean":"2024-03-06T07:22:51.677974+0000","last_became_active":"2024-03-06T07:22:49.066102+0000","last_became_peered":"2024-03-06T07:22:49.066102+0000","last_unstale":"2024-03-06T07:22:51.677974+0000","last_undegraded":"2024-03-06T07:22:51.677974+0000","last_fullsized":"2024-03-06T07:22:51.677974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620363+0000","last_change":"2024-03-06T07:22:48.072246+0000","last_active":"2024-03-06T07:22:50.620363+0000","last_peered":"2024-03-06T07:22:50.620363+0000","last_clean":"2024-03-06T07:22:50.620363+0000","last_became_active":"2024-03-06T07:22:48.071947+0000","last_became_peered":"2024-03-06T07:22:48.071947+0000","last_unstale":"2024-03-06T07:22:50.620363+0000","last_undegraded":"2024-03-06T07:22:50.620363+0000","last_fullsized":"2024-03-06T07:22:50.620363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677363+0000","last_change":"2024-03-06T07:22:48.075039+0000","last_active":"2024-03-06T07:22:51.677363+0000","last_peered":"2024-03-06T07:22:51.677363+0000","last_clean":"2024-03-06T07:22:51.677363+0000","last_became_active":"2024-03-06T07:22:48.074175+0000","last_became_peered":"2024-03-06T07:22:48.074175+0000","last_unstale":"2024-03-06T07:22:51.677363+0000","last_undegraded":"2024-03-06T07:22:51.677363+0000","last_fullsized":"2024-03-06T07:22:51.677363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:51.443747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.676975+0000","last_change":"2024-03-06T07:22:49.081619+0000","last_active":"2024-03-06T07:22:51.676975+0000","last_peered":"2024-03-06T07:22:51.676975+0000","last_clean":"2024-03-06T07:22:51.676975+0000","last_became_active":"2024-03-06T07:22:49.081307+0000","last_became_peered":"2024-03-06T07:22:49.081307+0000","last_unstale":"2024-03-06T07:22:51.676975+0000","last_undegraded":"2024-03-06T07:22:51.676975+0000","last_fullsized":"2024-03-06T07:22:51.676975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669896+0000","last_change":"2024-03-06T07:22:48.070889+0000","last_active":"2024-03-06T07:22:50.669896+0000","last_peered":"2024-03-06T07:22:50.669896+0000","last_clean":"2024-03-06T07:22:50.669896+0000","last_became_active":"2024-03-06T07:22:48.070591+0000","last_became_peered":"2024-03-06T07:22:48.070591+0000","last_unstale":"2024-03-06T07:22:50.669896+0000","last_undegraded":"2024-03-06T07:22:50.669896+0000","last_fullsized":"2024-03-06T07:22:50.669896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076880+0000","last_change":"2024-03-06T07:22:49.071108+0000","last_active":"2024-03-06T07:22:50.076880+0000","last_peered":"2024-03-06T07:22:50.076880+0000","last_clean":"2024-03-06T07:22:50.076880+0000","last_became_active":"2024-03-06T07:22:49.070830+0000","last_became_peered":"2024-03-06T07:22:49.070830+0000","last_unstale":"2024-03-06T07:22:50.076880+0000","last_undegraded":"2024-03-06T07:22:50.076880+0000","last_fullsized":"2024-03-06T07:22:50.076880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620725+0000","last_change":"2024-03-06T07:22:48.073692+0000","last_active":"2024-03-06T07:22:50.620725+0000","last_peered":"2024-03-06T07:22:50.620725+0000","last_clean":"2024-03-06T07:22:50.620725+0000","last_became_active":"2024-03-06T07:22:48.073366+0000","last_became_peered":"2024-03-06T07:22:48.073366+0000","last_unstale":"2024-03-06T07:22:50.620725+0000","last_undegraded":"2024-03-06T07:22:50.620725+0000","last_fullsized":"2024-03-06T07:22:50.620725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670451+0000","last_change":"2024-03-06T07:22:49.075374+0000","last_active":"2024-03-06T07:22:50.670451+0000","last_peered":"2024-03-06T07:22:50.670451+0000","last_clean":"2024-03-06T07:22:50.670451+0000","last_became_active":"2024-03-06T07:22:49.073115+0000","last_became_peered":"2024-03-06T07:22:49.073115+0000","last_unstale":"2024-03-06T07:22:50.670451+0000","last_undegraded":"2024-03-06T07:22:50.670451+0000","last_fullsized":"2024-03-06T07:22:50.670451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:04.714438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670425+0000","last_change":"2024-03-06T07:22:48.074598+0000","last_active":"2024-03-06T07:22:50.670425+0000","last_peered":"2024-03-06T07:22:50.670425+0000","last_clean":"2024-03-06T07:22:50.670425+0000","last_became_active":"2024-03-06T07:22:48.074291+0000","last_became_peered":"2024-03-06T07:22:48.074291+0000","last_unstale":"2024-03-06T07:22:50.670425+0000","last_undegraded":"2024-03-06T07:22:50.670425+0000","last_fullsized":"2024-03-06T07:22:50.670425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678055+0000","last_change":"2024-03-06T07:22:49.075698+0000","last_active":"2024-03-06T07:22:51.678055+0000","last_peered":"2024-03-06T07:22:51.678055+0000","last_clean":"2024-03-06T07:22:51.678055+0000","last_became_active":"2024-03-06T07:22:49.075395+0000","last_became_peered":"2024-03-06T07:22:49.075395+0000","last_unstale":"2024-03-06T07:22:51.678055+0000","last_undegraded":"2024-03-06T07:22:51.678055+0000","last_fullsized":"2024-03-06T07:22:51.678055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077137+0000","last_change":"2024-03-06T07:22:48.071600+0000","last_active":"2024-03-06T07:22:50.077137+0000","last_peered":"2024-03-06T07:22:50.077137+0000","last_clean":"2024-03-06T07:22:50.077137+0000","last_became_active":"2024-03-06T07:22:48.071194+0000","last_became_peered":"2024-03-06T07:22:48.071194+0000","last_unstale":"2024-03-06T07:22:50.077137+0000","last_undegraded":"2024-03-06T07:22:50.077137+0000","last_fullsized":"2024-03-06T07:22:50.077137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078999+0000","last_change":"2024-03-06T07:22:49.078266+0000","last_active":"2024-03-06T07:22:50.078999+0000","last_peered":"2024-03-06T07:22:50.078999+0000","last_clean":"2024-03-06T07:22:50.078999+0000","last_became_active":"2024-03-06T07:22:49.077993+0000","last_became_peered":"2024-03-06T07:22:49.077993+0000","last_unstale":"2024-03-06T07:22:50.078999+0000","last_undegraded":"2024-03-06T07:22:50.078999+0000","last_fullsized":"2024-03-06T07:22:50.078999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620480+0000","last_change":"2024-03-06T07:22:48.074028+0000","last_active":"2024-03-06T07:22:50.620480+0000","last_peered":"2024-03-06T07:22:50.620480+0000","last_clean":"2024-03-06T07:22:50.620480+0000","last_became_active":"2024-03-06T07:22:48.073737+0000","last_became_peered":"2024-03-06T07:22:48.073737+0000","last_unstale":"2024-03-06T07:22:50.620480+0000","last_undegraded":"2024-03-06T07:22:50.620480+0000","last_fullsized":"2024-03-06T07:22:50.620480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079495+0000","last_change":"2024-03-06T07:22:49.075328+0000","last_active":"2024-03-06T07:22:50.079495+0000","last_peered":"2024-03-06T07:22:50.079495+0000","last_clean":"2024-03-06T07:22:50.079495+0000","last_became_active":"2024-03-06T07:22:49.074938+0000","last_became_peered":"2024-03-06T07:22:49.074938+0000","last_unstale":"2024-03-06T07:22:50.079495+0000","last_undegraded":"2024-03-06T07:22:50.079495+0000","last_fullsized":"2024-03-06T07:22:50.079495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:43.229187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678067+0000","last_change":"2024-03-06T07:22:48.062093+0000","last_active":"2024-03-06T07:22:51.678067+0000","last_peered":"2024-03-06T07:22:51.678067+0000","last_clean":"2024-03-06T07:22:51.678067+0000","last_became_active":"2024-03-06T07:22:48.061816+0000","last_became_peered":"2024-03-06T07:22:48.061816+0000","last_unstale":"2024-03-06T07:22:51.678067+0000","last_undegraded":"2024-03-06T07:22:51.678067+0000","last_fullsized":"2024-03-06T07:22:51.678067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.354486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677614+0000","last_change":"2024-03-06T07:22:49.081035+0000","last_active":"2024-03-06T07:22:51.677614+0000","last_peered":"2024-03-06T07:22:51.677614+0000","last_clean":"2024-03-06T07:22:51.677614+0000","last_became_active":"2024-03-06T07:22:49.080738+0000","last_became_peered":"2024-03-06T07:22:49.080738+0000","last_unstale":"2024-03-06T07:22:51.677614+0000","last_undegraded":"2024-03-06T07:22:51.677614+0000","last_fullsized":"2024-03-06T07:22:51.677614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670172+0000","last_change":"2024-03-06T07:22:48.076377+0000","last_active":"2024-03-06T07:22:50.670172+0000","last_peered":"2024-03-06T07:22:50.670172+0000","last_clean":"2024-03-06T07:22:50.670172+0000","last_became_active":"2024-03-06T07:22:48.076062+0000","last_became_peered":"2024-03-06T07:22:48.076062+0000","last_unstale":"2024-03-06T07:22:50.670172+0000","last_undegraded":"2024-03-06T07:22:50.670172+0000","last_fullsized":"2024-03-06T07:22:50.670172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079805+0000","last_change":"2024-03-06T07:22:49.081681+0000","last_active":"2024-03-06T07:22:50.079805+0000","last_peered":"2024-03-06T07:22:50.079805+0000","last_clean":"2024-03-06T07:22:50.079805+0000","last_became_active":"2024-03-06T07:22:49.081400+0000","last_became_peered":"2024-03-06T07:22:49.081400+0000","last_unstale":"2024-03-06T07:22:50.079805+0000","last_undegraded":"2024-03-06T07:22:50.079805+0000","last_fullsized":"2024-03-06T07:22:50.079805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:04.714438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620373+0000","last_change":"2024-03-06T07:22:48.072725+0000","last_active":"2024-03-06T07:22:50.620373+0000","last_peered":"2024-03-06T07:22:50.620373+0000","last_clean":"2024-03-06T07:22:50.620373+0000","last_became_active":"2024-03-06T07:22:48.071793+0000","last_became_peered":"2024-03-06T07:22:48.071793+0000","last_unstale":"2024-03-06T07:22:50.620373+0000","last_undegraded":"2024-03-06T07:22:50.620373+0000","last_fullsized":"2024-03-06T07:22:50.620373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620496+0000","last_change":"2024-03-06T07:22:49.088880+0000","last_active":"2024-03-06T07:22:50.620496+0000","last_peered":"2024-03-06T07:22:50.620496+0000","last_clean":"2024-03-06T07:22:50.620496+0000","last_became_active":"2024-03-06T07:22:49.088565+0000","last_became_peered":"2024-03-06T07:22:49.088565+0000","last_unstale":"2024-03-06T07:22:50.620496+0000","last_undegraded":"2024-03-06T07:22:50.620496+0000","last_fullsized":"2024-03-06T07:22:50.620496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.676898+0000","last_change":"2024-03-06T07:22:48.067604+0000","last_active":"2024-03-06T07:22:51.676898+0000","last_peered":"2024-03-06T07:22:51.676898+0000","last_clean":"2024-03-06T07:22:51.676898+0000","last_became_active":"2024-03-06T07:22:48.067273+0000","last_became_peered":"2024-03-06T07:22:48.067273+0000","last_unstale":"2024-03-06T07:22:51.676898+0000","last_undegraded":"2024-03-06T07:22:51.676898+0000","last_fullsized":"2024-03-06T07:22:51.676898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079385+0000","last_change":"2024-03-06T07:22:49.080955+0000","last_active":"2024-03-06T07:22:50.079385+0000","last_peered":"2024-03-06T07:22:50.079385+0000","last_clean":"2024-03-06T07:22:50.079385+0000","last_became_active":"2024-03-06T07:22:49.080699+0000","last_became_peered":"2024-03-06T07:22:49.080699+0000","last_unstale":"2024-03-06T07:22:50.079385+0000","last_undegraded":"2024-03-06T07:22:50.079385+0000","last_fullsized":"2024-03-06T07:22:50.079385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:02.284670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078773+0000","last_change":"2024-03-06T07:22:48.071049+0000","last_active":"2024-03-06T07:22:50.078773+0000","last_peered":"2024-03-06T07:22:50.078773+0000","last_clean":"2024-03-06T07:22:50.078773+0000","last_became_active":"2024-03-06T07:22:48.070751+0000","last_became_peered":"2024-03-06T07:22:48.070751+0000","last_unstale":"2024-03-06T07:22:50.078773+0000","last_undegraded":"2024-03-06T07:22:50.078773+0000","last_fullsized":"2024-03-06T07:22:50.078773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:35.856218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079364+0000","last_change":"2024-03-06T07:22:49.077479+0000","last_active":"2024-03-06T07:22:50.079364+0000","last_peered":"2024-03-06T07:22:50.079364+0000","last_clean":"2024-03-06T07:22:50.079364+0000","last_became_active":"2024-03-06T07:22:49.077188+0000","last_became_peered":"2024-03-06T07:22:49.077188+0000","last_unstale":"2024-03-06T07:22:50.079364+0000","last_undegraded":"2024-03-06T07:22:50.079364+0000","last_fullsized":"2024-03-06T07:22:50.079364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:29.588686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677428+0000","last_change":"2024-03-06T07:22:48.074777+0000","last_active":"2024-03-06T07:22:51.677428+0000","last_peered":"2024-03-06T07:22:51.677428+0000","last_clean":"2024-03-06T07:22:51.677428+0000","last_became_active":"2024-03-06T07:22:48.074441+0000","last_became_peered":"2024-03-06T07:22:48.074441+0000","last_unstale":"2024-03-06T07:22:51.677428+0000","last_undegraded":"2024-03-06T07:22:51.677428+0000","last_fullsized":"2024-03-06T07:22:51.677428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.080122+0000","last_change":"2024-03-06T07:22:49.080000+0000","last_active":"2024-03-06T07:22:50.080122+0000","last_peered":"2024-03-06T07:22:50.080122+0000","last_clean":"2024-03-06T07:22:50.080122+0000","last_became_active":"2024-03-06T07:22:49.079471+0000","last_became_peered":"2024-03-06T07:22:49.079471+0000","last_unstale":"2024-03-06T07:22:50.080122+0000","last_undegraded":"2024-03-06T07:22:50.080122+0000","last_fullsized":"2024-03-06T07:22:50.080122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:21.116229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077260+0000","last_change":"2024-03-06T07:22:48.067652+0000","last_active":"2024-03-06T07:22:50.077260+0000","last_peered":"2024-03-06T07:22:50.077260+0000","last_clean":"2024-03-06T07:22:50.077260+0000","last_became_active":"2024-03-06T07:22:48.067390+0000","last_became_peered":"2024-03-06T07:22:48.067390+0000","last_unstale":"2024-03-06T07:22:50.077260+0000","last_undegraded":"2024-03-06T07:22:50.077260+0000","last_fullsized":"2024-03-06T07:22:50.077260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670254+0000","last_change":"2024-03-06T07:22:49.084081+0000","last_active":"2024-03-06T07:22:50.670254+0000","last_peered":"2024-03-06T07:22:50.670254+0000","last_clean":"2024-03-06T07:22:50.670254+0000","last_became_active":"2024-03-06T07:22:49.083758+0000","last_became_peered":"2024-03-06T07:22:49.083758+0000","last_unstale":"2024-03-06T07:22:50.670254+0000","last_undegraded":"2024-03-06T07:22:50.670254+0000","last_fullsized":"2024-03-06T07:22:50.670254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:11.749715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.676989+0000","last_change":"2024-03-06T07:22:48.072197+0000","last_active":"2024-03-06T07:22:51.676989+0000","last_peered":"2024-03-06T07:22:51.676989+0000","last_clean":"2024-03-06T07:22:51.676989+0000","last_became_active":"2024-03-06T07:22:48.071904+0000","last_became_peered":"2024-03-06T07:22:48.071904+0000","last_unstale":"2024-03-06T07:22:51.676989+0000","last_undegraded":"2024-03-06T07:22:51.676989+0000","last_fullsized":"2024-03-06T07:22:51.676989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079659+0000","last_change":"2024-03-06T07:22:49.074585+0000","last_active":"2024-03-06T07:22:50.079659+0000","last_peered":"2024-03-06T07:22:50.079659+0000","last_clean":"2024-03-06T07:22:50.079659+0000","last_became_active":"2024-03-06T07:22:49.074307+0000","last_became_peered":"2024-03-06T07:22:49.074307+0000","last_unstale":"2024-03-06T07:22:50.079659+0000","last_undegraded":"2024-03-06T07:22:50.079659+0000","last_fullsized":"2024-03-06T07:22:50.079659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:11.749715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079272+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:22:50.079272+0000","last_peered":"2024-03-06T07:22:50.079272+0000","last_clean":"2024-03-06T07:22:50.079272+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:22:50.079272+0000","last_undegraded":"2024-03-06T07:22:50.079272+0000","last_fullsized":"2024-03-06T07:22:50.079272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669826+0000","last_change":"2024-03-06T07:22:48.073977+0000","last_active":"2024-03-06T07:22:50.669826+0000","last_peered":"2024-03-06T07:22:50.669826+0000","last_clean":"2024-03-06T07:22:50.669826+0000","last_became_active":"2024-03-06T07:22:48.073656+0000","last_became_peered":"2024-03-06T07:22:48.073656+0000","last_unstale":"2024-03-06T07:22:50.669826+0000","last_undegraded":"2024-03-06T07:22:50.669826+0000","last_fullsized":"2024-03-06T07:22:50.669826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677513+0000","last_change":"2024-03-06T07:22:49.074128+0000","last_active":"2024-03-06T07:22:51.677513+0000","last_peered":"2024-03-06T07:22:51.677513+0000","last_clean":"2024-03-06T07:22:51.677513+0000","last_became_active":"2024-03-06T07:22:49.073858+0000","last_became_peered":"2024-03-06T07:22:49.073858+0000","last_unstale":"2024-03-06T07:22:51.677513+0000","last_undegraded":"2024-03-06T07:22:51.677513+0000","last_fullsized":"2024-03-06T07:22:51.677513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079180+0000","last_change":"2024-03-06T07:22:48.069166+0000","last_active":"2024-03-06T07:22:50.079180+0000","last_peered":"2024-03-06T07:22:50.079180+0000","last_clean":"2024-03-06T07:22:50.079180+0000","last_became_active":"2024-03-06T07:22:48.068441+0000","last_became_peered":"2024-03-06T07:22:48.068441+0000","last_unstale":"2024-03-06T07:22:50.079180+0000","last_undegraded":"2024-03-06T07:22:50.079180+0000","last_fullsized":"2024-03-06T07:22:50.079180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:45.086584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677256+0000","last_change":"2024-03-06T07:22:49.075228+0000","last_active":"2024-03-06T07:22:51.677256+0000","last_peered":"2024-03-06T07:22:51.677256+0000","last_clean":"2024-03-06T07:22:51.677256+0000","last_became_active":"2024-03-06T07:22:49.074906+0000","last_became_peered":"2024-03-06T07:22:49.074906+0000","last_unstale":"2024-03-06T07:22:51.677256+0000","last_undegraded":"2024-03-06T07:22:51.677256+0000","last_fullsized":"2024-03-06T07:22:51.677256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677178+0000","last_change":"2024-03-06T07:22:48.074088+0000","last_active":"2024-03-06T07:22:51.677178+0000","last_peered":"2024-03-06T07:22:51.677178+0000","last_clean":"2024-03-06T07:22:51.677178+0000","last_became_active":"2024-03-06T07:22:48.073794+0000","last_became_peered":"2024-03-06T07:22:48.073794+0000","last_unstale":"2024-03-06T07:22:51.677178+0000","last_undegraded":"2024-03-06T07:22:51.677178+0000","last_fullsized":"2024-03-06T07:22:51.677178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076918+0000","last_change":"2024-03-06T07:22:49.071324+0000","last_active":"2024-03-06T07:22:50.076918+0000","last_peered":"2024-03-06T07:22:50.076918+0000","last_clean":"2024-03-06T07:22:50.076918+0000","last_became_active":"2024-03-06T07:22:49.071043+0000","last_became_peered":"2024-03-06T07:22:49.071043+0000","last_unstale":"2024-03-06T07:22:50.076918+0000","last_undegraded":"2024-03-06T07:22:50.076918+0000","last_fullsized":"2024-03-06T07:22:50.076918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670270+0000","last_change":"2024-03-06T07:22:48.071307+0000","last_active":"2024-03-06T07:22:50.670270+0000","last_peered":"2024-03-06T07:22:50.670270+0000","last_clean":"2024-03-06T07:22:50.670270+0000","last_became_active":"2024-03-06T07:22:48.070797+0000","last_became_peered":"2024-03-06T07:22:48.070797+0000","last_unstale":"2024-03-06T07:22:50.670270+0000","last_undegraded":"2024-03-06T07:22:50.670270+0000","last_fullsized":"2024-03-06T07:22:50.670270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078575+0000","last_change":"2024-03-06T07:22:49.077714+0000","last_active":"2024-03-06T07:22:50.078575+0000","last_peered":"2024-03-06T07:22:50.078575+0000","last_clean":"2024-03-06T07:22:50.078575+0000","last_became_active":"2024-03-06T07:22:49.077442+0000","last_became_peered":"2024-03-06T07:22:49.077442+0000","last_unstale":"2024-03-06T07:22:50.078575+0000","last_undegraded":"2024-03-06T07:22:50.078575+0000","last_fullsized":"2024-03-06T07:22:50.078575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:46.459505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076919+0000","last_change":"2024-03-06T07:22:48.071210+0000","last_active":"2024-03-06T07:22:50.076919+0000","last_peered":"2024-03-06T07:22:50.076919+0000","last_clean":"2024-03-06T07:22:50.076919+0000","last_became_active":"2024-03-06T07:22:48.070893+0000","last_became_peered":"2024-03-06T07:22:48.070893+0000","last_unstale":"2024-03-06T07:22:50.076919+0000","last_undegraded":"2024-03-06T07:22:50.076919+0000","last_fullsized":"2024-03-06T07:22:50.076919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670301+0000","last_change":"2024-03-06T07:22:49.085398+0000","last_active":"2024-03-06T07:22:50.670301+0000","last_peered":"2024-03-06T07:22:50.670301+0000","last_clean":"2024-03-06T07:22:50.670301+0000","last_became_active":"2024-03-06T07:22:49.085124+0000","last_became_peered":"2024-03-06T07:22:49.085124+0000","last_unstale":"2024-03-06T07:22:50.670301+0000","last_undegraded":"2024-03-06T07:22:50.670301+0000","last_fullsized":"2024-03-06T07:22:50.670301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:32.043873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.080057+0000","last_change":"2024-03-06T07:22:48.071472+0000","last_active":"2024-03-06T07:22:50.080057+0000","last_peered":"2024-03-06T07:22:50.080057+0000","last_clean":"2024-03-06T07:22:50.080057+0000","last_became_active":"2024-03-06T07:22:48.071136+0000","last_became_peered":"2024-03-06T07:22:48.071136+0000","last_unstale":"2024-03-06T07:22:50.080057+0000","last_undegraded":"2024-03-06T07:22:50.080057+0000","last_fullsized":"2024-03-06T07:22:50.080057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:30.767002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077039+0000","last_change":"2024-03-06T07:22:49.072607+0000","last_active":"2024-03-06T07:22:50.077039+0000","last_peered":"2024-03-06T07:22:50.077039+0000","last_clean":"2024-03-06T07:22:50.077039+0000","last_became_active":"2024-03-06T07:22:49.071896+0000","last_became_peered":"2024-03-06T07:22:49.071896+0000","last_unstale":"2024-03-06T07:22:50.077039+0000","last_undegraded":"2024-03-06T07:22:50.077039+0000","last_fullsized":"2024-03-06T07:22:50.077039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077163+0000","last_change":"2024-03-06T07:22:48.071003+0000","last_active":"2024-03-06T07:22:50.077163+0000","last_peered":"2024-03-06T07:22:50.077163+0000","last_clean":"2024-03-06T07:22:50.077163+0000","last_became_active":"2024-03-06T07:22:48.070666+0000","last_became_peered":"2024-03-06T07:22:48.070666+0000","last_unstale":"2024-03-06T07:22:50.077163+0000","last_undegraded":"2024-03-06T07:22:50.077163+0000","last_fullsized":"2024-03-06T07:22:50.077163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678235+0000","last_change":"2024-03-06T07:22:49.077332+0000","last_active":"2024-03-06T07:22:51.678235+0000","last_peered":"2024-03-06T07:22:51.678235+0000","last_clean":"2024-03-06T07:22:51.678235+0000","last_became_active":"2024-03-06T07:22:49.077034+0000","last_became_peered":"2024-03-06T07:22:49.077034+0000","last_unstale":"2024-03-06T07:22:51.678235+0000","last_undegraded":"2024-03-06T07:22:51.678235+0000","last_fullsized":"2024-03-06T07:22:51.678235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:43.569514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620780+0000","last_change":"2024-03-06T07:22:48.076685+0000","last_active":"2024-03-06T07:22:50.620780+0000","last_peered":"2024-03-06T07:22:50.620780+0000","last_clean":"2024-03-06T07:22:50.620780+0000","last_became_active":"2024-03-06T07:22:48.076412+0000","last_became_peered":"2024-03-06T07:22:48.076412+0000","last_unstale":"2024-03-06T07:22:50.620780+0000","last_undegraded":"2024-03-06T07:22:50.620780+0000","last_fullsized":"2024-03-06T07:22:50.620780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678095+0000","last_change":"2024-03-06T07:22:49.069020+0000","last_active":"2024-03-06T07:22:51.678095+0000","last_peered":"2024-03-06T07:22:51.678095+0000","last_clean":"2024-03-06T07:22:51.678095+0000","last_became_active":"2024-03-06T07:22:49.068741+0000","last_became_peered":"2024-03-06T07:22:49.068741+0000","last_unstale":"2024-03-06T07:22:51.678095+0000","last_undegraded":"2024-03-06T07:22:51.678095+0000","last_fullsized":"2024-03-06T07:22:51.678095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079069+0000","last_change":"2024-03-06T07:22:48.072458+0000","last_active":"2024-03-06T07:22:50.079069+0000","last_peered":"2024-03-06T07:22:50.079069+0000","last_clean":"2024-03-06T07:22:50.079069+0000","last_became_active":"2024-03-06T07:22:48.071279+0000","last_became_peered":"2024-03-06T07:22:48.071279+0000","last_unstale":"2024-03-06T07:22:50.079069+0000","last_undegraded":"2024-03-06T07:22:50.079069+0000","last_fullsized":"2024-03-06T07:22:50.079069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078676+0000","last_change":"2024-03-06T07:22:49.077623+0000","last_active":"2024-03-06T07:22:50.078676+0000","last_peered":"2024-03-06T07:22:50.078676+0000","last_clean":"2024-03-06T07:22:50.078676+0000","last_became_active":"2024-03-06T07:22:49.076719+0000","last_became_peered":"2024-03-06T07:22:49.076719+0000","last_unstale":"2024-03-06T07:22:50.078676+0000","last_undegraded":"2024-03-06T07:22:50.078676+0000","last_fullsized":"2024-03-06T07:22:50.078676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:38.976172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077000+0000","last_change":"2024-03-06T07:22:48.071228+0000","last_active":"2024-03-06T07:22:50.077000+0000","last_peered":"2024-03-06T07:22:50.077000+0000","last_clean":"2024-03-06T07:22:50.077000+0000","last_became_active":"2024-03-06T07:22:48.070898+0000","last_became_peered":"2024-03-06T07:22:48.070898+0000","last_unstale":"2024-03-06T07:22:50.077000+0000","last_undegraded":"2024-03-06T07:22:50.077000+0000","last_fullsized":"2024-03-06T07:22:50.077000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677021+0000","last_change":"2024-03-06T07:22:49.069977+0000","last_active":"2024-03-06T07:22:51.677021+0000","last_peered":"2024-03-06T07:22:51.677021+0000","last_clean":"2024-03-06T07:22:51.677021+0000","last_became_active":"2024-03-06T07:22:49.069691+0000","last_became_peered":"2024-03-06T07:22:49.069691+0000","last_unstale":"2024-03-06T07:22:51.677021+0000","last_undegraded":"2024-03-06T07:22:51.677021+0000","last_fullsized":"2024-03-06T07:22:51.677021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076906+0000","last_change":"2024-03-06T07:22:48.074693+0000","last_active":"2024-03-06T07:22:50.076906+0000","last_peered":"2024-03-06T07:22:50.076906+0000","last_clean":"2024-03-06T07:22:50.076906+0000","last_became_active":"2024-03-06T07:22:48.074261+0000","last_became_peered":"2024-03-06T07:22:48.074261+0000","last_unstale":"2024-03-06T07:22:50.076906+0000","last_undegraded":"2024-03-06T07:22:50.076906+0000","last_fullsized":"2024-03-06T07:22:50.076906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677543+0000","last_change":"2024-03-06T07:22:49.082024+0000","last_active":"2024-03-06T07:22:51.677543+0000","last_peered":"2024-03-06T07:22:51.677543+0000","last_clean":"2024-03-06T07:22:51.677543+0000","last_became_active":"2024-03-06T07:22:49.081534+0000","last_became_peered":"2024-03-06T07:22:49.081534+0000","last_unstale":"2024-03-06T07:22:51.677543+0000","last_undegraded":"2024-03-06T07:22:51.677543+0000","last_fullsized":"2024-03-06T07:22:51.677543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079905+0000","last_change":"2024-03-06T07:22:48.069179+0000","last_active":"2024-03-06T07:22:50.079905+0000","last_peered":"2024-03-06T07:22:50.079905+0000","last_clean":"2024-03-06T07:22:50.079905+0000","last_became_active":"2024-03-06T07:22:48.068402+0000","last_became_peered":"2024-03-06T07:22:48.068402+0000","last_unstale":"2024-03-06T07:22:50.079905+0000","last_undegraded":"2024-03-06T07:22:50.079905+0000","last_fullsized":"2024-03-06T07:22:50.079905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:09.655063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620537+0000","last_change":"2024-03-06T07:22:49.082657+0000","last_active":"2024-03-06T07:22:50.620537+0000","last_peered":"2024-03-06T07:22:50.620537+0000","last_clean":"2024-03-06T07:22:50.620537+0000","last_became_active":"2024-03-06T07:22:49.082096+0000","last_became_peered":"2024-03-06T07:22:49.082096+0000","last_unstale":"2024-03-06T07:22:50.620537+0000","last_undegraded":"2024-03-06T07:22:50.620537+0000","last_fullsized":"2024-03-06T07:22:50.620537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670227+0000","last_change":"2024-03-06T07:22:49.073629+0000","last_active":"2024-03-06T07:22:50.670227+0000","last_peered":"2024-03-06T07:22:50.670227+0000","last_clean":"2024-03-06T07:22:50.670227+0000","last_became_active":"2024-03-06T07:22:49.072839+0000","last_became_peered":"2024-03-06T07:22:49.072839+0000","last_unstale":"2024-03-06T07:22:50.670227+0000","last_undegraded":"2024-03-06T07:22:50.670227+0000","last_fullsized":"2024-03-06T07:22:50.670227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:32.401483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079077+0000","last_change":"2024-03-06T07:22:48.072083+0000","last_active":"2024-03-06T07:22:50.079077+0000","last_peered":"2024-03-06T07:22:50.079077+0000","last_clean":"2024-03-06T07:22:50.079077+0000","last_became_active":"2024-03-06T07:22:48.071560+0000","last_became_peered":"2024-03-06T07:22:48.071560+0000","last_unstale":"2024-03-06T07:22:50.079077+0000","last_undegraded":"2024-03-06T07:22:50.079077+0000","last_fullsized":"2024-03-06T07:22:50.079077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670115+0000","last_change":"2024-03-06T07:22:49.080775+0000","last_active":"2024-03-06T07:22:50.670115+0000","last_peered":"2024-03-06T07:22:50.670115+0000","last_clean":"2024-03-06T07:22:50.670115+0000","last_became_active":"2024-03-06T07:22:49.080487+0000","last_became_peered":"2024-03-06T07:22:49.080487+0000","last_unstale":"2024-03-06T07:22:50.670115+0000","last_undegraded":"2024-03-06T07:22:50.670115+0000","last_fullsized":"2024-03-06T07:22:50.670115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:43.229187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079758+0000","last_change":"2024-03-06T07:22:48.074550+0000","last_active":"2024-03-06T07:22:50.079758+0000","last_peered":"2024-03-06T07:22:50.079758+0000","last_clean":"2024-03-06T07:22:50.079758+0000","last_became_active":"2024-03-06T07:22:48.073981+0000","last_became_peered":"2024-03-06T07:22:48.073981+0000","last_unstale":"2024-03-06T07:22:50.079758+0000","last_undegraded":"2024-03-06T07:22:50.079758+0000","last_fullsized":"2024-03-06T07:22:50.079758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:13.210796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677964+0000","last_change":"2024-03-06T07:22:49.075565+0000","last_active":"2024-03-06T07:22:51.677964+0000","last_peered":"2024-03-06T07:22:51.677964+0000","last_clean":"2024-03-06T07:22:51.677964+0000","last_became_active":"2024-03-06T07:22:49.075254+0000","last_became_peered":"2024-03-06T07:22:49.075254+0000","last_unstale":"2024-03-06T07:22:51.677964+0000","last_undegraded":"2024-03-06T07:22:51.677964+0000","last_fullsized":"2024-03-06T07:22:51.677964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678098+0000","last_change":"2024-03-06T07:22:48.073512+0000","last_active":"2024-03-06T07:22:51.678098+0000","last_peered":"2024-03-06T07:22:51.678098+0000","last_clean":"2024-03-06T07:22:51.678098+0000","last_became_active":"2024-03-06T07:22:48.073181+0000","last_became_peered":"2024-03-06T07:22:48.073181+0000","last_unstale":"2024-03-06T07:22:51.678098+0000","last_undegraded":"2024-03-06T07:22:51.678098+0000","last_fullsized":"2024-03-06T07:22:51.678098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677959+0000","last_change":"2024-03-06T07:22:49.079451+0000","last_active":"2024-03-06T07:22:51.677959+0000","last_peered":"2024-03-06T07:22:51.677959+0000","last_clean":"2024-03-06T07:22:51.677959+0000","last_became_active":"2024-03-06T07:22:49.078927+0000","last_became_peered":"2024-03-06T07:22:49.078927+0000","last_unstale":"2024-03-06T07:22:51.677959+0000","last_undegraded":"2024-03-06T07:22:51.677959+0000","last_fullsized":"2024-03-06T07:22:51.677959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:07.781375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678094+0000","last_change":"2024-03-06T07:22:48.062906+0000","last_active":"2024-03-06T07:22:51.678094+0000","last_peered":"2024-03-06T07:22:51.678094+0000","last_clean":"2024-03-06T07:22:51.678094+0000","last_became_active":"2024-03-06T07:22:48.062497+0000","last_became_peered":"2024-03-06T07:22:48.062497+0000","last_unstale":"2024-03-06T07:22:51.678094+0000","last_undegraded":"2024-03-06T07:22:51.678094+0000","last_fullsized":"2024-03-06T07:22:51.678094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:51.443747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:57.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:57.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.887+0000 7efccaeed640 1 Processor -- start 2024-03-06T07:22:57.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.887+0000 7efccaeed640 1 -- start start 2024-03-06T07:22:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.887+0000 7efccaeed640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efcc40d02e0 0x7efcc40cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.887+0000 7efccaeed640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efcc40d06c0 0x7efcc40b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.887+0000 7efccaeed640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc40b0dd0 con 0x7efcc40b4260 2024-03-06T07:22:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.887+0000 7efccaeed640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc40b0b80 con 0x7efcc40d02e0 2024-03-06T07:22:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.887+0000 7efccaeed640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc40aeec0 con 0x7efcc40d06c0 2024-03-06T07:22:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.888+0000 7efcc8c89640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efcc40d02e0 0x7efcc40cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.888+0000 7efcc8c89640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efcc40d02e0 0x7efcc40cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51832/0 (socket says 172.21.15.27:51832) 2024-03-06T07:22:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.888+0000 7efcc8c89640 1 -- 172.21.15.27:0/3497014100 learned_addr learned my addr 172.21.15.27:0/3497014100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:57.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.888+0000 7efcc948a640 1 --2- 172.21.15.27:0/3497014100 >> v2:172.21.15.69:3300/0 conn(0x7efcc40d06c0 0x7efcc40b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.888+0000 7efcc8c89640 1 -- 172.21.15.27:0/3497014100 >> v2:172.21.15.69:3300/0 conn(0x7efcc40d06c0 msgr2=0x7efcc40b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.888+0000 7efcc8c89640 1 --2- 172.21.15.27:0/3497014100 >> v2:172.21.15.69:3300/0 conn(0x7efcc40d06c0 0x7efcc40b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.888+0000 7efcc8c89640 1 -- 172.21.15.27:0/3497014100 >> v1:172.21.15.69:6789/0 conn(0x7efcc40b4260 legacy=0x7efcc40b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.888+0000 7efcc8c89640 1 -- 172.21.15.27:0/3497014100 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc40b10b0 con 0x7efcc40d02e0 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efcc8c89640 1 --2- 172.21.15.27:0/3497014100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40d02e0 0x7efcc40cfa00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efcc40cc0e0 tx=0x7efcb802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ec5f91d93db667 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efcc37fe640 1 -- 172.21.15.27:0/3497014100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8034070 con 0x7efcc40d02e0 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efcc37fe640 1 -- 172.21.15.27:0/3497014100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb8030860 con 0x7efcc40d02e0 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efcc37fe640 1 -- 172.21.15.27:0/3497014100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8030c30 con 0x7efcc40d02e0 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efccaeed640 1 -- 172.21.15.27:0/3497014100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40d02e0 msgr2=0x7efcc40cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efccaeed640 1 --2- 172.21.15.27:0/3497014100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40d02e0 0x7efcc40cfa00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efcc40cc0e0 tx=0x7efcb802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efccaeed640 1 -- 172.21.15.27:0/3497014100 shutdown_connections 2024-03-06T07:22:57.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efccaeed640 1 --2- 172.21.15.27:0/3497014100 >> v2:172.21.15.69:3300/0 conn(0x7efcc40d06c0 0x7efcc40b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efccaeed640 1 --2- 172.21.15.27:0/3497014100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40d02e0 0x7efcc40cfa00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efccaeed640 1 -- 172.21.15.27:0/3497014100 >> 172.21.15.27:0/3497014100 conn(0x7efcc40b36d0 msgr2=0x7efcc40b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efccaeed640 1 -- 172.21.15.27:0/3497014100 shutdown_connections 2024-03-06T07:22:57.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.889+0000 7efccaeed640 1 -- 172.21.15.27:0/3497014100 wait complete. 2024-03-06T07:22:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.891+0000 7efccaeed640 1 Processor -- start 2024-03-06T07:22:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.891+0000 7efccaeed640 1 -- start start 2024-03-06T07:22:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.891+0000 7efccaeed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40b4260 0x7efcc41e4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.891+0000 7efccaeed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcc40d02e0 0x7efcc41e6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efccaeed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc40d06c0 0x7efcc40c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efccaeed640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40b22d0 con 0x7efcc40b4260 2024-03-06T07:22:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efccaeed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40b2150 con 0x7efcc40d02e0 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efccaeed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40af350 con 0x7efcc40d06c0 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc8c89640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40b4260 0x7efcc41e4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc8c89640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40b4260 0x7efcc41e4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51842/0 (socket says 172.21.15.27:51842) 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc8c89640 1 -- 172.21.15.27:0/943637825 learned_addr learned my addr 172.21.15.27:0/943637825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc948a640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc40d06c0 0x7efcc40c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc3fff640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcc40d02e0 0x7efcc41e6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc8c89640 1 -- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc40d06c0 msgr2=0x7efcc40c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc8c89640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc40d06c0 0x7efcc40c8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc8c89640 1 -- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcc40d02e0 msgr2=0x7efcc41e6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc8c89640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcc40d02e0 0x7efcc41e6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc8c89640 1 -- 172.21.15.27:0/943637825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc40c8790 con 0x7efcc40b4260 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc948a640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc40d06c0 0x7efcc40c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc3fff640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcc40d02e0 0x7efcc41e6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:57.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc8c89640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40b4260 0x7efcc41e4cf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efcb8000c00 tx=0x7efcb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc17fa640 1 -- 172.21.15.27:0/943637825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8048070 con 0x7efcc40b4260 2024-03-06T07:22:57.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc17fa640 1 -- 172.21.15.27:0/943637825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb8030660 con 0x7efcc40b4260 2024-03-06T07:22:57.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efcc17fa640 1 -- 172.21.15.27:0/943637825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8034040 con 0x7efcc40b4260 2024-03-06T07:22:57.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc41e52d0 con 0x7efcc40b4260 2024-03-06T07:22:57.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.892+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc41e5890 con 0x7efcc40b4260 2024-03-06T07:22:57.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.893+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc4072d90 con 0x7efcc40b4260 2024-03-06T07:22:57.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.894+0000 7efcc17fa640 1 -- 172.21.15.27:0/943637825 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcb804b070 con 0x7efcc40b4260 2024-03-06T07:22:57.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.894+0000 7efcc17fa640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.894+0000 7efcc17fa640 1 -- 172.21.15.27:0/943637825 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcb80ac340 con 0x7efcc40b4260 2024-03-06T07:22:57.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.894+0000 7efcc3fff640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.895+0000 7efcc3fff640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efcc40cbd20 tx=0x7efcb0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:57.903+0000 7efcc17fa640 1 -- 172.21.15.27:0/943637825 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcc4072d90 con 0x7efcc40b4260 2024-03-06T07:22:58.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.080+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efcc4140450 con 0x7efca806b4d0 2024-03-06T07:22:58.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.086+0000 7efcc17fa640 1 -- 172.21.15.27:0/943637825 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251836 (secure 0 0 0) 0x7efcb00567d0 con 0x7efca806b4d0 2024-03-06T07:22:58.091 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:58.093 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:22:58.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efca806b4d0 msgr2=0x7efca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efcc40cbd20 tx=0x7efcb0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40b4260 msgr2=0x7efcc41e4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40b4260 0x7efcc41e4cf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efcb8000c00 tx=0x7efcb8004730 comp rx=0 tx=0).stop 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 shutdown_connections 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc40d06c0 0x7efcc40c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcc40d02e0 0x7efcc41e6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 --2- 172.21.15.27:0/943637825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc40b4260 0x7efcc41e4cf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 >> 172.21.15.27:0/943637825 conn(0x7efcc40b36d0 msgr2=0x7efcc40cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 shutdown_connections 2024-03-06T07:22:58.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.089+0000 7efccaeed640 1 -- 172.21.15.27:0/943637825 wait complete. 2024-03-06T07:22:58.107 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-06T07:22:57.988730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219096,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755624,"statfs":{"total":773094113280,"available":772869758976,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.930027"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079733+0000","last_change":"2024-03-06T07:22:49.080233+0000","last_active":"2024-03-06T07:22:50.079733+0000","last_peered":"2024-03-06T07:22:50.079733+0000","last_clean":"2024-03-06T07:22:50.079733+0000","last_became_active":"2024-03-06T07:22:49.079770+0000","last_became_peered":"2024-03-06T07:22:49.079770+0000","last_unstale":"2024-03-06T07:22:50.079733+0000","last_undegraded":"2024-03-06T07:22:50.079733+0000","last_fullsized":"2024-03-06T07:22:50.079733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:32.043873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077134+0000","last_change":"2024-03-06T07:22:49.081636+0000","last_active":"2024-03-06T07:22:50.077134+0000","last_peered":"2024-03-06T07:22:50.077134+0000","last_clean":"2024-03-06T07:22:50.077134+0000","last_became_active":"2024-03-06T07:22:49.081356+0000","last_became_peered":"2024-03-06T07:22:49.081356+0000","last_unstale":"2024-03-06T07:22:50.077134+0000","last_undegraded":"2024-03-06T07:22:50.077134+0000","last_fullsized":"2024-03-06T07:22:50.077134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677913+0000","last_change":"2024-03-06T07:22:49.076314+0000","last_active":"2024-03-06T07:22:51.677913+0000","last_peered":"2024-03-06T07:22:51.677913+0000","last_clean":"2024-03-06T07:22:51.677913+0000","last_became_active":"2024-03-06T07:22:49.075901+0000","last_became_peered":"2024-03-06T07:22:49.075901+0000","last_unstale":"2024-03-06T07:22:51.677913+0000","last_undegraded":"2024-03-06T07:22:51.677913+0000","last_fullsized":"2024-03-06T07:22:51.677913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670047+0000","last_change":"2024-03-06T07:22:49.086730+0000","last_active":"2024-03-06T07:22:50.670047+0000","last_peered":"2024-03-06T07:22:50.670047+0000","last_clean":"2024-03-06T07:22:50.670047+0000","last_became_active":"2024-03-06T07:22:49.086475+0000","last_became_peered":"2024-03-06T07:22:49.086475+0000","last_unstale":"2024-03-06T07:22:50.670047+0000","last_undegraded":"2024-03-06T07:22:50.670047+0000","last_fullsized":"2024-03-06T07:22:50.670047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:29.588686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079650+0000","last_change":"2024-03-06T07:22:49.074936+0000","last_active":"2024-03-06T07:22:50.079650+0000","last_peered":"2024-03-06T07:22:50.079650+0000","last_clean":"2024-03-06T07:22:50.079650+0000","last_became_active":"2024-03-06T07:22:49.074662+0000","last_became_peered":"2024-03-06T07:22:49.074662+0000","last_unstale":"2024-03-06T07:22:50.079650+0000","last_undegraded":"2024-03-06T07:22:50.079650+0000","last_fullsized":"2024-03-06T07:22:50.079650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:32.401483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669865+0000","last_change":"2024-03-06T07:22:49.081940+0000","last_active":"2024-03-06T07:22:50.669865+0000","last_peered":"2024-03-06T07:22:50.669865+0000","last_clean":"2024-03-06T07:22:50.669865+0000","last_became_active":"2024-03-06T07:22:49.081668+0000","last_became_peered":"2024-03-06T07:22:49.081668+0000","last_unstale":"2024-03-06T07:22:50.669865+0000","last_undegraded":"2024-03-06T07:22:50.669865+0000","last_fullsized":"2024-03-06T07:22:50.669865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:58.929359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079272+0000","last_change":"2024-03-06T07:22:49.088117+0000","last_active":"2024-03-06T07:22:50.079272+0000","last_peered":"2024-03-06T07:22:50.079272+0000","last_clean":"2024-03-06T07:22:50.079272+0000","last_became_active":"2024-03-06T07:22:49.087856+0000","last_became_peered":"2024-03-06T07:22:49.087856+0000","last_unstale":"2024-03-06T07:22:50.079272+0000","last_undegraded":"2024-03-06T07:22:50.079272+0000","last_fullsized":"2024-03-06T07:22:50.079272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:43.569514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.676835+0000","last_change":"2024-03-06T07:22:49.074570+0000","last_active":"2024-03-06T07:22:51.676835+0000","last_peered":"2024-03-06T07:22:51.676835+0000","last_clean":"2024-03-06T07:22:51.676835+0000","last_became_active":"2024-03-06T07:22:49.073115+0000","last_became_peered":"2024-03-06T07:22:49.073115+0000","last_unstale":"2024-03-06T07:22:51.676835+0000","last_undegraded":"2024-03-06T07:22:51.676835+0000","last_fullsized":"2024-03-06T07:22:51.676835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.031324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.080031+0000","last_change":"2024-03-06T07:22:49.073716+0000","last_active":"2024-03-06T07:22:50.080031+0000","last_peered":"2024-03-06T07:22:50.080031+0000","last_clean":"2024-03-06T07:22:50.080031+0000","last_became_active":"2024-03-06T07:22:49.073434+0000","last_became_peered":"2024-03-06T07:22:49.073434+0000","last_unstale":"2024-03-06T07:22:50.080031+0000","last_undegraded":"2024-03-06T07:22:50.080031+0000","last_fullsized":"2024-03-06T07:22:50.080031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:33.120440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620829+0000","last_change":"2024-03-06T07:22:49.072470+0000","last_active":"2024-03-06T07:22:50.620829+0000","last_peered":"2024-03-06T07:22:50.620829+0000","last_clean":"2024-03-06T07:22:50.620829+0000","last_became_active":"2024-03-06T07:22:49.072206+0000","last_became_peered":"2024-03-06T07:22:49.072206+0000","last_unstale":"2024-03-06T07:22:50.620829+0000","last_undegraded":"2024-03-06T07:22:50.620829+0000","last_fullsized":"2024-03-06T07:22:50.620829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620623+0000","last_change":"2024-03-06T07:22:49.076507+0000","last_active":"2024-03-06T07:22:50.620623+0000","last_peered":"2024-03-06T07:22:50.620623+0000","last_clean":"2024-03-06T07:22:50.620623+0000","last_became_active":"2024-03-06T07:22:49.076054+0000","last_became_peered":"2024-03-06T07:22:49.076054+0000","last_unstale":"2024-03-06T07:22:50.620623+0000","last_undegraded":"2024-03-06T07:22:50.620623+0000","last_fullsized":"2024-03-06T07:22:50.620623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078932+0000","last_change":"2024-03-06T07:22:49.077078+0000","last_active":"2024-03-06T07:22:50.078932+0000","last_peered":"2024-03-06T07:22:50.078932+0000","last_clean":"2024-03-06T07:22:50.078932+0000","last_became_active":"2024-03-06T07:22:49.076757+0000","last_became_peered":"2024-03-06T07:22:49.076757+0000","last_unstale":"2024-03-06T07:22:50.078932+0000","last_undegraded":"2024-03-06T07:22:50.078932+0000","last_fullsized":"2024-03-06T07:22:50.078932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:40.466024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677438+0000","last_change":"2024-03-06T07:22:49.073467+0000","last_active":"2024-03-06T07:22:51.677438+0000","last_peered":"2024-03-06T07:22:51.677438+0000","last_clean":"2024-03-06T07:22:51.677438+0000","last_became_active":"2024-03-06T07:22:49.072969+0000","last_became_peered":"2024-03-06T07:22:49.072969+0000","last_unstale":"2024-03-06T07:22:51.677438+0000","last_undegraded":"2024-03-06T07:22:51.677438+0000","last_fullsized":"2024-03-06T07:22:51.677438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677469+0000","last_change":"2024-03-06T07:22:49.076207+0000","last_active":"2024-03-06T07:22:51.677469+0000","last_peered":"2024-03-06T07:22:51.677469+0000","last_clean":"2024-03-06T07:22:51.677469+0000","last_became_active":"2024-03-06T07:22:49.074449+0000","last_became_peered":"2024-03-06T07:22:49.074449+0000","last_unstale":"2024-03-06T07:22:51.677469+0000","last_undegraded":"2024-03-06T07:22:51.677469+0000","last_fullsized":"2024-03-06T07:22:51.677469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:12.342553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076881+0000","last_change":"2024-03-06T07:22:49.072745+0000","last_active":"2024-03-06T07:22:50.076881+0000","last_peered":"2024-03-06T07:22:50.076881+0000","last_clean":"2024-03-06T07:22:50.076881+0000","last_became_active":"2024-03-06T07:22:49.072443+0000","last_became_peered":"2024-03-06T07:22:49.072443+0000","last_unstale":"2024-03-06T07:22:50.076881+0000","last_undegraded":"2024-03-06T07:22:50.076881+0000","last_fullsized":"2024-03-06T07:22:50.076881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669789+0000","last_change":"2024-03-06T07:22:49.085335+0000","last_active":"2024-03-06T07:22:50.669789+0000","last_peered":"2024-03-06T07:22:50.669789+0000","last_clean":"2024-03-06T07:22:50.669789+0000","last_became_active":"2024-03-06T07:22:49.085023+0000","last_became_peered":"2024-03-06T07:22:49.085023+0000","last_unstale":"2024-03-06T07:22:50.669789+0000","last_undegraded":"2024-03-06T07:22:50.669789+0000","last_fullsized":"2024-03-06T07:22:50.669789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670606+0000","last_change":"2024-03-06T07:22:49.073395+0000","last_active":"2024-03-06T07:22:50.670606+0000","last_peered":"2024-03-06T07:22:50.670606+0000","last_clean":"2024-03-06T07:22:50.670606+0000","last_became_active":"2024-03-06T07:22:49.073105+0000","last_became_peered":"2024-03-06T07:22:49.073105+0000","last_unstale":"2024-03-06T07:22:50.670606+0000","last_undegraded":"2024-03-06T07:22:50.670606+0000","last_fullsized":"2024-03-06T07:22:50.670606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:16.151911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677340+0000","last_change":"2024-03-06T07:22:49.074912+0000","last_active":"2024-03-06T07:22:51.677340+0000","last_peered":"2024-03-06T07:22:51.677340+0000","last_clean":"2024-03-06T07:22:51.677340+0000","last_became_active":"2024-03-06T07:22:49.074638+0000","last_became_peered":"2024-03-06T07:22:49.074638+0000","last_unstale":"2024-03-06T07:22:51.677340+0000","last_undegraded":"2024-03-06T07:22:51.677340+0000","last_fullsized":"2024-03-06T07:22:51.677340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079918+0000","last_change":"2024-03-06T07:22:49.075473+0000","last_active":"2024-03-06T07:22:50.079918+0000","last_peered":"2024-03-06T07:22:50.079918+0000","last_clean":"2024-03-06T07:22:50.079918+0000","last_became_active":"2024-03-06T07:22:49.074945+0000","last_became_peered":"2024-03-06T07:22:49.074945+0000","last_unstale":"2024-03-06T07:22:50.079918+0000","last_undegraded":"2024-03-06T07:22:50.079918+0000","last_fullsized":"2024-03-06T07:22:50.079918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:16.151911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620534+0000","last_change":"2024-03-06T07:22:49.085348+0000","last_active":"2024-03-06T07:22:50.620534+0000","last_peered":"2024-03-06T07:22:50.620534+0000","last_clean":"2024-03-06T07:22:50.620534+0000","last_became_active":"2024-03-06T07:22:49.084955+0000","last_became_peered":"2024-03-06T07:22:49.084955+0000","last_unstale":"2024-03-06T07:22:50.620534+0000","last_undegraded":"2024-03-06T07:22:50.620534+0000","last_fullsized":"2024-03-06T07:22:50.620534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677814+0000","last_change":"2024-03-06T07:22:49.081034+0000","last_active":"2024-03-06T07:22:51.677814+0000","last_peered":"2024-03-06T07:22:51.677814+0000","last_clean":"2024-03-06T07:22:51.677814+0000","last_became_active":"2024-03-06T07:22:49.080639+0000","last_became_peered":"2024-03-06T07:22:49.080639+0000","last_unstale":"2024-03-06T07:22:51.677814+0000","last_undegraded":"2024-03-06T07:22:51.677814+0000","last_fullsized":"2024-03-06T07:22:51.677814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:42.031324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670034+0000","last_change":"2024-03-06T07:22:49.074317+0000","last_active":"2024-03-06T07:22:50.670034+0000","last_peered":"2024-03-06T07:22:50.670034+0000","last_clean":"2024-03-06T07:22:50.670034+0000","last_became_active":"2024-03-06T07:22:49.073041+0000","last_became_peered":"2024-03-06T07:22:49.073041+0000","last_unstale":"2024-03-06T07:22:50.670034+0000","last_undegraded":"2024-03-06T07:22:50.670034+0000","last_fullsized":"2024-03-06T07:22:50.670034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:18.288459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677267+0000","last_change":"2024-03-06T07:22:49.076771+0000","last_active":"2024-03-06T07:22:51.677267+0000","last_peered":"2024-03-06T07:22:51.677267+0000","last_clean":"2024-03-06T07:22:51.677267+0000","last_became_active":"2024-03-06T07:22:49.073031+0000","last_became_peered":"2024-03-06T07:22:49.073031+0000","last_unstale":"2024-03-06T07:22:51.677267+0000","last_undegraded":"2024-03-06T07:22:51.677267+0000","last_fullsized":"2024-03-06T07:22:51.677267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669877+0000","last_change":"2024-03-06T07:22:49.086108+0000","last_active":"2024-03-06T07:22:50.669877+0000","last_peered":"2024-03-06T07:22:50.669877+0000","last_clean":"2024-03-06T07:22:50.669877+0000","last_became_active":"2024-03-06T07:22:49.085825+0000","last_became_peered":"2024-03-06T07:22:49.085825+0000","last_unstale":"2024-03-06T07:22:50.669877+0000","last_undegraded":"2024-03-06T07:22:50.669877+0000","last_fullsized":"2024-03-06T07:22:50.669877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:07.781375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678152+0000","last_change":"2024-03-06T07:22:49.071422+0000","last_active":"2024-03-06T07:22:51.678152+0000","last_peered":"2024-03-06T07:22:51.678152+0000","last_clean":"2024-03-06T07:22:51.678152+0000","last_became_active":"2024-03-06T07:22:49.071117+0000","last_became_peered":"2024-03-06T07:22:49.071117+0000","last_unstale":"2024-03-06T07:22:51.678152+0000","last_undegraded":"2024-03-06T07:22:51.678152+0000","last_fullsized":"2024-03-06T07:22:51.678152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:18.288459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620872+0000","last_change":"2024-03-06T07:22:49.077927+0000","last_active":"2024-03-06T07:22:50.620872+0000","last_peered":"2024-03-06T07:22:50.620872+0000","last_clean":"2024-03-06T07:22:50.620872+0000","last_became_active":"2024-03-06T07:22:49.077647+0000","last_became_peered":"2024-03-06T07:22:49.077647+0000","last_unstale":"2024-03-06T07:22:50.620872+0000","last_undegraded":"2024-03-06T07:22:50.620872+0000","last_fullsized":"2024-03-06T07:22:50.620872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:58.929359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077021+0000","last_change":"2024-03-06T07:22:49.085345+0000","last_active":"2024-03-06T07:22:50.077021+0000","last_peered":"2024-03-06T07:22:50.077021+0000","last_clean":"2024-03-06T07:22:50.077021+0000","last_became_active":"2024-03-06T07:22:49.084942+0000","last_became_peered":"2024-03-06T07:22:49.084942+0000","last_unstale":"2024-03-06T07:22:50.077021+0000","last_undegraded":"2024-03-06T07:22:50.077021+0000","last_fullsized":"2024-03-06T07:22:50.077021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677841+0000","last_change":"2024-03-06T07:22:49.071314+0000","last_active":"2024-03-06T07:22:51.677841+0000","last_peered":"2024-03-06T07:22:51.677841+0000","last_clean":"2024-03-06T07:22:51.677841+0000","last_became_active":"2024-03-06T07:22:49.070986+0000","last_became_peered":"2024-03-06T07:22:49.070986+0000","last_unstale":"2024-03-06T07:22:51.677841+0000","last_undegraded":"2024-03-06T07:22:51.677841+0000","last_fullsized":"2024-03-06T07:22:51.677841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:58.929359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677884+0000","last_change":"2024-03-06T07:22:49.077320+0000","last_active":"2024-03-06T07:22:51.677884+0000","last_peered":"2024-03-06T07:22:51.677884+0000","last_clean":"2024-03-06T07:22:51.677884+0000","last_became_active":"2024-03-06T07:22:49.077017+0000","last_became_peered":"2024-03-06T07:22:49.077017+0000","last_unstale":"2024-03-06T07:22:51.677884+0000","last_undegraded":"2024-03-06T07:22:51.677884+0000","last_fullsized":"2024-03-06T07:22:51.677884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670057+0000","last_change":"2024-03-06T07:22:49.086200+0000","last_active":"2024-03-06T07:22:50.670057+0000","last_peered":"2024-03-06T07:22:50.670057+0000","last_clean":"2024-03-06T07:22:50.670057+0000","last_became_active":"2024-03-06T07:22:49.085793+0000","last_became_peered":"2024-03-06T07:22:49.085793+0000","last_unstale":"2024-03-06T07:22:50.670057+0000","last_undegraded":"2024-03-06T07:22:50.670057+0000","last_fullsized":"2024-03-06T07:22:50.670057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:43.569514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677199+0000","last_change":"2024-03-06T07:22:49.076891+0000","last_active":"2024-03-06T07:22:51.677199+0000","last_peered":"2024-03-06T07:22:51.677199+0000","last_clean":"2024-03-06T07:22:51.677199+0000","last_became_active":"2024-03-06T07:22:49.076610+0000","last_became_peered":"2024-03-06T07:22:49.076610+0000","last_unstale":"2024-03-06T07:22:51.677199+0000","last_undegraded":"2024-03-06T07:22:51.677199+0000","last_fullsized":"2024-03-06T07:22:51.677199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:31.772994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620616+0000","last_change":"2024-03-06T07:22:49.088635+0000","last_active":"2024-03-06T07:22:50.620616+0000","last_peered":"2024-03-06T07:22:50.620616+0000","last_clean":"2024-03-06T07:22:50.620616+0000","last_became_active":"2024-03-06T07:22:49.088352+0000","last_became_peered":"2024-03-06T07:22:49.088352+0000","last_unstale":"2024-03-06T07:22:50.620616+0000","last_undegraded":"2024-03-06T07:22:50.620616+0000","last_fullsized":"2024-03-06T07:22:50.620616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078869+0000","last_change":"2024-03-06T07:22:49.071168+0000","last_active":"2024-03-06T07:22:50.078869+0000","last_peered":"2024-03-06T07:22:50.078869+0000","last_clean":"2024-03-06T07:22:50.078869+0000","last_became_active":"2024-03-06T07:22:49.070861+0000","last_became_peered":"2024-03-06T07:22:49.070861+0000","last_unstale":"2024-03-06T07:22:50.078869+0000","last_undegraded":"2024-03-06T07:22:50.078869+0000","last_fullsized":"2024-03-06T07:22:50.078869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:51.096875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677602+0000","last_change":"2024-03-06T07:22:48.076442+0000","last_active":"2024-03-06T07:22:51.677602+0000","last_peered":"2024-03-06T07:22:51.677602+0000","last_clean":"2024-03-06T07:22:51.677602+0000","last_became_active":"2024-03-06T07:22:48.076193+0000","last_became_peered":"2024-03-06T07:22:48.076193+0000","last_unstale":"2024-03-06T07:22:51.677602+0000","last_undegraded":"2024-03-06T07:22:51.677602+0000","last_fullsized":"2024-03-06T07:22:51.677602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:48.118117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677974+0000","last_change":"2024-03-06T07:22:49.067928+0000","last_active":"2024-03-06T07:22:51.677974+0000","last_peered":"2024-03-06T07:22:51.677974+0000","last_clean":"2024-03-06T07:22:51.677974+0000","last_became_active":"2024-03-06T07:22:49.066102+0000","last_became_peered":"2024-03-06T07:22:49.066102+0000","last_unstale":"2024-03-06T07:22:51.677974+0000","last_undegraded":"2024-03-06T07:22:51.677974+0000","last_fullsized":"2024-03-06T07:22:51.677974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:02.782023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620363+0000","last_change":"2024-03-06T07:22:48.072246+0000","last_active":"2024-03-06T07:22:50.620363+0000","last_peered":"2024-03-06T07:22:50.620363+0000","last_clean":"2024-03-06T07:22:50.620363+0000","last_became_active":"2024-03-06T07:22:48.071947+0000","last_became_peered":"2024-03-06T07:22:48.071947+0000","last_unstale":"2024-03-06T07:22:50.620363+0000","last_undegraded":"2024-03-06T07:22:50.620363+0000","last_fullsized":"2024-03-06T07:22:50.620363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677363+0000","last_change":"2024-03-06T07:22:48.075039+0000","last_active":"2024-03-06T07:22:51.677363+0000","last_peered":"2024-03-06T07:22:51.677363+0000","last_clean":"2024-03-06T07:22:51.677363+0000","last_became_active":"2024-03-06T07:22:48.074175+0000","last_became_peered":"2024-03-06T07:22:48.074175+0000","last_unstale":"2024-03-06T07:22:51.677363+0000","last_undegraded":"2024-03-06T07:22:51.677363+0000","last_fullsized":"2024-03-06T07:22:51.677363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:51.443747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.676975+0000","last_change":"2024-03-06T07:22:49.081619+0000","last_active":"2024-03-06T07:22:51.676975+0000","last_peered":"2024-03-06T07:22:51.676975+0000","last_clean":"2024-03-06T07:22:51.676975+0000","last_became_active":"2024-03-06T07:22:49.081307+0000","last_became_peered":"2024-03-06T07:22:49.081307+0000","last_unstale":"2024-03-06T07:22:51.676975+0000","last_undegraded":"2024-03-06T07:22:51.676975+0000","last_fullsized":"2024-03-06T07:22:51.676975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669896+0000","last_change":"2024-03-06T07:22:48.070889+0000","last_active":"2024-03-06T07:22:50.669896+0000","last_peered":"2024-03-06T07:22:50.669896+0000","last_clean":"2024-03-06T07:22:50.669896+0000","last_became_active":"2024-03-06T07:22:48.070591+0000","last_became_peered":"2024-03-06T07:22:48.070591+0000","last_unstale":"2024-03-06T07:22:50.669896+0000","last_undegraded":"2024-03-06T07:22:50.669896+0000","last_fullsized":"2024-03-06T07:22:50.669896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076880+0000","last_change":"2024-03-06T07:22:49.071108+0000","last_active":"2024-03-06T07:22:50.076880+0000","last_peered":"2024-03-06T07:22:50.076880+0000","last_clean":"2024-03-06T07:22:50.076880+0000","last_became_active":"2024-03-06T07:22:49.070830+0000","last_became_peered":"2024-03-06T07:22:49.070830+0000","last_unstale":"2024-03-06T07:22:50.076880+0000","last_undegraded":"2024-03-06T07:22:50.076880+0000","last_fullsized":"2024-03-06T07:22:50.076880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620725+0000","last_change":"2024-03-06T07:22:48.073692+0000","last_active":"2024-03-06T07:22:50.620725+0000","last_peered":"2024-03-06T07:22:50.620725+0000","last_clean":"2024-03-06T07:22:50.620725+0000","last_became_active":"2024-03-06T07:22:48.073366+0000","last_became_peered":"2024-03-06T07:22:48.073366+0000","last_unstale":"2024-03-06T07:22:50.620725+0000","last_undegraded":"2024-03-06T07:22:50.620725+0000","last_fullsized":"2024-03-06T07:22:50.620725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670451+0000","last_change":"2024-03-06T07:22:49.075374+0000","last_active":"2024-03-06T07:22:50.670451+0000","last_peered":"2024-03-06T07:22:50.670451+0000","last_clean":"2024-03-06T07:22:50.670451+0000","last_became_active":"2024-03-06T07:22:49.073115+0000","last_became_peered":"2024-03-06T07:22:49.073115+0000","last_unstale":"2024-03-06T07:22:50.670451+0000","last_undegraded":"2024-03-06T07:22:50.670451+0000","last_fullsized":"2024-03-06T07:22:50.670451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:04.714438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670425+0000","last_change":"2024-03-06T07:22:48.074598+0000","last_active":"2024-03-06T07:22:50.670425+0000","last_peered":"2024-03-06T07:22:50.670425+0000","last_clean":"2024-03-06T07:22:50.670425+0000","last_became_active":"2024-03-06T07:22:48.074291+0000","last_became_peered":"2024-03-06T07:22:48.074291+0000","last_unstale":"2024-03-06T07:22:50.670425+0000","last_undegraded":"2024-03-06T07:22:50.670425+0000","last_fullsized":"2024-03-06T07:22:50.670425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678055+0000","last_change":"2024-03-06T07:22:49.075698+0000","last_active":"2024-03-06T07:22:51.678055+0000","last_peered":"2024-03-06T07:22:51.678055+0000","last_clean":"2024-03-06T07:22:51.678055+0000","last_became_active":"2024-03-06T07:22:49.075395+0000","last_became_peered":"2024-03-06T07:22:49.075395+0000","last_unstale":"2024-03-06T07:22:51.678055+0000","last_undegraded":"2024-03-06T07:22:51.678055+0000","last_fullsized":"2024-03-06T07:22:51.678055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077137+0000","last_change":"2024-03-06T07:22:48.071600+0000","last_active":"2024-03-06T07:22:50.077137+0000","last_peered":"2024-03-06T07:22:50.077137+0000","last_clean":"2024-03-06T07:22:50.077137+0000","last_became_active":"2024-03-06T07:22:48.071194+0000","last_became_peered":"2024-03-06T07:22:48.071194+0000","last_unstale":"2024-03-06T07:22:50.077137+0000","last_undegraded":"2024-03-06T07:22:50.077137+0000","last_fullsized":"2024-03-06T07:22:50.077137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078999+0000","last_change":"2024-03-06T07:22:49.078266+0000","last_active":"2024-03-06T07:22:50.078999+0000","last_peered":"2024-03-06T07:22:50.078999+0000","last_clean":"2024-03-06T07:22:50.078999+0000","last_became_active":"2024-03-06T07:22:49.077993+0000","last_became_peered":"2024-03-06T07:22:49.077993+0000","last_unstale":"2024-03-06T07:22:50.078999+0000","last_undegraded":"2024-03-06T07:22:50.078999+0000","last_fullsized":"2024-03-06T07:22:50.078999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:46.092575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620480+0000","last_change":"2024-03-06T07:22:48.074028+0000","last_active":"2024-03-06T07:22:50.620480+0000","last_peered":"2024-03-06T07:22:50.620480+0000","last_clean":"2024-03-06T07:22:50.620480+0000","last_became_active":"2024-03-06T07:22:48.073737+0000","last_became_peered":"2024-03-06T07:22:48.073737+0000","last_unstale":"2024-03-06T07:22:50.620480+0000","last_undegraded":"2024-03-06T07:22:50.620480+0000","last_fullsized":"2024-03-06T07:22:50.620480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079495+0000","last_change":"2024-03-06T07:22:49.075328+0000","last_active":"2024-03-06T07:22:50.079495+0000","last_peered":"2024-03-06T07:22:50.079495+0000","last_clean":"2024-03-06T07:22:50.079495+0000","last_became_active":"2024-03-06T07:22:49.074938+0000","last_became_peered":"2024-03-06T07:22:49.074938+0000","last_unstale":"2024-03-06T07:22:50.079495+0000","last_undegraded":"2024-03-06T07:22:50.079495+0000","last_fullsized":"2024-03-06T07:22:50.079495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:43.229187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678067+0000","last_change":"2024-03-06T07:22:48.062093+0000","last_active":"2024-03-06T07:22:51.678067+0000","last_peered":"2024-03-06T07:22:51.678067+0000","last_clean":"2024-03-06T07:22:51.678067+0000","last_became_active":"2024-03-06T07:22:48.061816+0000","last_became_peered":"2024-03-06T07:22:48.061816+0000","last_unstale":"2024-03-06T07:22:51.678067+0000","last_undegraded":"2024-03-06T07:22:51.678067+0000","last_fullsized":"2024-03-06T07:22:51.678067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:19.354486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677614+0000","last_change":"2024-03-06T07:22:49.081035+0000","last_active":"2024-03-06T07:22:51.677614+0000","last_peered":"2024-03-06T07:22:51.677614+0000","last_clean":"2024-03-06T07:22:51.677614+0000","last_became_active":"2024-03-06T07:22:49.080738+0000","last_became_peered":"2024-03-06T07:22:49.080738+0000","last_unstale":"2024-03-06T07:22:51.677614+0000","last_undegraded":"2024-03-06T07:22:51.677614+0000","last_fullsized":"2024-03-06T07:22:51.677614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:52.291557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670172+0000","last_change":"2024-03-06T07:22:48.076377+0000","last_active":"2024-03-06T07:22:50.670172+0000","last_peered":"2024-03-06T07:22:50.670172+0000","last_clean":"2024-03-06T07:22:50.670172+0000","last_became_active":"2024-03-06T07:22:48.076062+0000","last_became_peered":"2024-03-06T07:22:48.076062+0000","last_unstale":"2024-03-06T07:22:50.670172+0000","last_undegraded":"2024-03-06T07:22:50.670172+0000","last_fullsized":"2024-03-06T07:22:50.670172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079805+0000","last_change":"2024-03-06T07:22:49.081681+0000","last_active":"2024-03-06T07:22:50.079805+0000","last_peered":"2024-03-06T07:22:50.079805+0000","last_clean":"2024-03-06T07:22:50.079805+0000","last_became_active":"2024-03-06T07:22:49.081400+0000","last_became_peered":"2024-03-06T07:22:49.081400+0000","last_unstale":"2024-03-06T07:22:50.079805+0000","last_undegraded":"2024-03-06T07:22:50.079805+0000","last_fullsized":"2024-03-06T07:22:50.079805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:04.714438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620373+0000","last_change":"2024-03-06T07:22:48.072725+0000","last_active":"2024-03-06T07:22:50.620373+0000","last_peered":"2024-03-06T07:22:50.620373+0000","last_clean":"2024-03-06T07:22:50.620373+0000","last_became_active":"2024-03-06T07:22:48.071793+0000","last_became_peered":"2024-03-06T07:22:48.071793+0000","last_unstale":"2024-03-06T07:22:50.620373+0000","last_undegraded":"2024-03-06T07:22:50.620373+0000","last_fullsized":"2024-03-06T07:22:50.620373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620496+0000","last_change":"2024-03-06T07:22:49.088880+0000","last_active":"2024-03-06T07:22:50.620496+0000","last_peered":"2024-03-06T07:22:50.620496+0000","last_clean":"2024-03-06T07:22:50.620496+0000","last_became_active":"2024-03-06T07:22:49.088565+0000","last_became_peered":"2024-03-06T07:22:49.088565+0000","last_unstale":"2024-03-06T07:22:50.620496+0000","last_undegraded":"2024-03-06T07:22:50.620496+0000","last_fullsized":"2024-03-06T07:22:50.620496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:26.585666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.676898+0000","last_change":"2024-03-06T07:22:48.067604+0000","last_active":"2024-03-06T07:22:51.676898+0000","last_peered":"2024-03-06T07:22:51.676898+0000","last_clean":"2024-03-06T07:22:51.676898+0000","last_became_active":"2024-03-06T07:22:48.067273+0000","last_became_peered":"2024-03-06T07:22:48.067273+0000","last_unstale":"2024-03-06T07:22:51.676898+0000","last_undegraded":"2024-03-06T07:22:51.676898+0000","last_fullsized":"2024-03-06T07:22:51.676898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079385+0000","last_change":"2024-03-06T07:22:49.080955+0000","last_active":"2024-03-06T07:22:50.079385+0000","last_peered":"2024-03-06T07:22:50.079385+0000","last_clean":"2024-03-06T07:22:50.079385+0000","last_became_active":"2024-03-06T07:22:49.080699+0000","last_became_peered":"2024-03-06T07:22:49.080699+0000","last_unstale":"2024-03-06T07:22:50.079385+0000","last_undegraded":"2024-03-06T07:22:50.079385+0000","last_fullsized":"2024-03-06T07:22:50.079385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:02.284670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078773+0000","last_change":"2024-03-06T07:22:48.071049+0000","last_active":"2024-03-06T07:22:50.078773+0000","last_peered":"2024-03-06T07:22:50.078773+0000","last_clean":"2024-03-06T07:22:50.078773+0000","last_became_active":"2024-03-06T07:22:48.070751+0000","last_became_peered":"2024-03-06T07:22:48.070751+0000","last_unstale":"2024-03-06T07:22:50.078773+0000","last_undegraded":"2024-03-06T07:22:50.078773+0000","last_fullsized":"2024-03-06T07:22:50.078773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:35.856218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079364+0000","last_change":"2024-03-06T07:22:49.077479+0000","last_active":"2024-03-06T07:22:50.079364+0000","last_peered":"2024-03-06T07:22:50.079364+0000","last_clean":"2024-03-06T07:22:50.079364+0000","last_became_active":"2024-03-06T07:22:49.077188+0000","last_became_peered":"2024-03-06T07:22:49.077188+0000","last_unstale":"2024-03-06T07:22:50.079364+0000","last_undegraded":"2024-03-06T07:22:50.079364+0000","last_fullsized":"2024-03-06T07:22:50.079364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:29.588686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677428+0000","last_change":"2024-03-06T07:22:48.074777+0000","last_active":"2024-03-06T07:22:51.677428+0000","last_peered":"2024-03-06T07:22:51.677428+0000","last_clean":"2024-03-06T07:22:51.677428+0000","last_became_active":"2024-03-06T07:22:48.074441+0000","last_became_peered":"2024-03-06T07:22:48.074441+0000","last_unstale":"2024-03-06T07:22:51.677428+0000","last_undegraded":"2024-03-06T07:22:51.677428+0000","last_fullsized":"2024-03-06T07:22:51.677428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.080122+0000","last_change":"2024-03-06T07:22:49.080000+0000","last_active":"2024-03-06T07:22:50.080122+0000","last_peered":"2024-03-06T07:22:50.080122+0000","last_clean":"2024-03-06T07:22:50.080122+0000","last_became_active":"2024-03-06T07:22:49.079471+0000","last_became_peered":"2024-03-06T07:22:49.079471+0000","last_unstale":"2024-03-06T07:22:50.080122+0000","last_undegraded":"2024-03-06T07:22:50.080122+0000","last_fullsized":"2024-03-06T07:22:50.080122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:21.116229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077260+0000","last_change":"2024-03-06T07:22:48.067652+0000","last_active":"2024-03-06T07:22:50.077260+0000","last_peered":"2024-03-06T07:22:50.077260+0000","last_clean":"2024-03-06T07:22:50.077260+0000","last_became_active":"2024-03-06T07:22:48.067390+0000","last_became_peered":"2024-03-06T07:22:48.067390+0000","last_unstale":"2024-03-06T07:22:50.077260+0000","last_undegraded":"2024-03-06T07:22:50.077260+0000","last_fullsized":"2024-03-06T07:22:50.077260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670254+0000","last_change":"2024-03-06T07:22:49.084081+0000","last_active":"2024-03-06T07:22:50.670254+0000","last_peered":"2024-03-06T07:22:50.670254+0000","last_clean":"2024-03-06T07:22:50.670254+0000","last_became_active":"2024-03-06T07:22:49.083758+0000","last_became_peered":"2024-03-06T07:22:49.083758+0000","last_unstale":"2024-03-06T07:22:50.670254+0000","last_undegraded":"2024-03-06T07:22:50.670254+0000","last_fullsized":"2024-03-06T07:22:50.670254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:11.749715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.676989+0000","last_change":"2024-03-06T07:22:48.072197+0000","last_active":"2024-03-06T07:22:51.676989+0000","last_peered":"2024-03-06T07:22:51.676989+0000","last_clean":"2024-03-06T07:22:51.676989+0000","last_became_active":"2024-03-06T07:22:48.071904+0000","last_became_peered":"2024-03-06T07:22:48.071904+0000","last_unstale":"2024-03-06T07:22:51.676989+0000","last_undegraded":"2024-03-06T07:22:51.676989+0000","last_fullsized":"2024-03-06T07:22:51.676989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:45.453513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079659+0000","last_change":"2024-03-06T07:22:49.074585+0000","last_active":"2024-03-06T07:22:50.079659+0000","last_peered":"2024-03-06T07:22:50.079659+0000","last_clean":"2024-03-06T07:22:50.079659+0000","last_became_active":"2024-03-06T07:22:49.074307+0000","last_became_peered":"2024-03-06T07:22:49.074307+0000","last_unstale":"2024-03-06T07:22:50.079659+0000","last_undegraded":"2024-03-06T07:22:50.079659+0000","last_fullsized":"2024-03-06T07:22:50.079659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:11.749715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079272+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:22:50.079272+0000","last_peered":"2024-03-06T07:22:50.079272+0000","last_clean":"2024-03-06T07:22:50.079272+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:22:50.079272+0000","last_undegraded":"2024-03-06T07:22:50.079272+0000","last_fullsized":"2024-03-06T07:22:50.079272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.669826+0000","last_change":"2024-03-06T07:22:48.073977+0000","last_active":"2024-03-06T07:22:50.669826+0000","last_peered":"2024-03-06T07:22:50.669826+0000","last_clean":"2024-03-06T07:22:50.669826+0000","last_became_active":"2024-03-06T07:22:48.073656+0000","last_became_peered":"2024-03-06T07:22:48.073656+0000","last_unstale":"2024-03-06T07:22:50.669826+0000","last_undegraded":"2024-03-06T07:22:50.669826+0000","last_fullsized":"2024-03-06T07:22:50.669826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677513+0000","last_change":"2024-03-06T07:22:49.074128+0000","last_active":"2024-03-06T07:22:51.677513+0000","last_peered":"2024-03-06T07:22:51.677513+0000","last_clean":"2024-03-06T07:22:51.677513+0000","last_became_active":"2024-03-06T07:22:49.073858+0000","last_became_peered":"2024-03-06T07:22:49.073858+0000","last_unstale":"2024-03-06T07:22:51.677513+0000","last_undegraded":"2024-03-06T07:22:51.677513+0000","last_fullsized":"2024-03-06T07:22:51.677513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079180+0000","last_change":"2024-03-06T07:22:48.069166+0000","last_active":"2024-03-06T07:22:50.079180+0000","last_peered":"2024-03-06T07:22:50.079180+0000","last_clean":"2024-03-06T07:22:50.079180+0000","last_became_active":"2024-03-06T07:22:48.068441+0000","last_became_peered":"2024-03-06T07:22:48.068441+0000","last_unstale":"2024-03-06T07:22:50.079180+0000","last_undegraded":"2024-03-06T07:22:50.079180+0000","last_fullsized":"2024-03-06T07:22:50.079180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:45.086584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677256+0000","last_change":"2024-03-06T07:22:49.075228+0000","last_active":"2024-03-06T07:22:51.677256+0000","last_peered":"2024-03-06T07:22:51.677256+0000","last_clean":"2024-03-06T07:22:51.677256+0000","last_became_active":"2024-03-06T07:22:49.074906+0000","last_became_peered":"2024-03-06T07:22:49.074906+0000","last_unstale":"2024-03-06T07:22:51.677256+0000","last_undegraded":"2024-03-06T07:22:51.677256+0000","last_fullsized":"2024-03-06T07:22:51.677256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:10.661054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677178+0000","last_change":"2024-03-06T07:22:48.074088+0000","last_active":"2024-03-06T07:22:51.677178+0000","last_peered":"2024-03-06T07:22:51.677178+0000","last_clean":"2024-03-06T07:22:51.677178+0000","last_became_active":"2024-03-06T07:22:48.073794+0000","last_became_peered":"2024-03-06T07:22:48.073794+0000","last_unstale":"2024-03-06T07:22:51.677178+0000","last_undegraded":"2024-03-06T07:22:51.677178+0000","last_fullsized":"2024-03-06T07:22:51.677178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076918+0000","last_change":"2024-03-06T07:22:49.071324+0000","last_active":"2024-03-06T07:22:50.076918+0000","last_peered":"2024-03-06T07:22:50.076918+0000","last_clean":"2024-03-06T07:22:50.076918+0000","last_became_active":"2024-03-06T07:22:49.071043+0000","last_became_peered":"2024-03-06T07:22:49.071043+0000","last_unstale":"2024-03-06T07:22:50.076918+0000","last_undegraded":"2024-03-06T07:22:50.076918+0000","last_fullsized":"2024-03-06T07:22:50.076918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670270+0000","last_change":"2024-03-06T07:22:48.071307+0000","last_active":"2024-03-06T07:22:50.670270+0000","last_peered":"2024-03-06T07:22:50.670270+0000","last_clean":"2024-03-06T07:22:50.670270+0000","last_became_active":"2024-03-06T07:22:48.070797+0000","last_became_peered":"2024-03-06T07:22:48.070797+0000","last_unstale":"2024-03-06T07:22:50.670270+0000","last_undegraded":"2024-03-06T07:22:50.670270+0000","last_fullsized":"2024-03-06T07:22:50.670270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078575+0000","last_change":"2024-03-06T07:22:49.077714+0000","last_active":"2024-03-06T07:22:50.078575+0000","last_peered":"2024-03-06T07:22:50.078575+0000","last_clean":"2024-03-06T07:22:50.078575+0000","last_became_active":"2024-03-06T07:22:49.077442+0000","last_became_peered":"2024-03-06T07:22:49.077442+0000","last_unstale":"2024-03-06T07:22:50.078575+0000","last_undegraded":"2024-03-06T07:22:50.078575+0000","last_fullsized":"2024-03-06T07:22:50.078575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:46.459505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076919+0000","last_change":"2024-03-06T07:22:48.071210+0000","last_active":"2024-03-06T07:22:50.076919+0000","last_peered":"2024-03-06T07:22:50.076919+0000","last_clean":"2024-03-06T07:22:50.076919+0000","last_became_active":"2024-03-06T07:22:48.070893+0000","last_became_peered":"2024-03-06T07:22:48.070893+0000","last_unstale":"2024-03-06T07:22:50.076919+0000","last_undegraded":"2024-03-06T07:22:50.076919+0000","last_fullsized":"2024-03-06T07:22:50.076919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:52.005850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670301+0000","last_change":"2024-03-06T07:22:49.085398+0000","last_active":"2024-03-06T07:22:50.670301+0000","last_peered":"2024-03-06T07:22:50.670301+0000","last_clean":"2024-03-06T07:22:50.670301+0000","last_became_active":"2024-03-06T07:22:49.085124+0000","last_became_peered":"2024-03-06T07:22:49.085124+0000","last_unstale":"2024-03-06T07:22:50.670301+0000","last_undegraded":"2024-03-06T07:22:50.670301+0000","last_fullsized":"2024-03-06T07:22:50.670301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:32.043873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.080057+0000","last_change":"2024-03-06T07:22:48.071472+0000","last_active":"2024-03-06T07:22:50.080057+0000","last_peered":"2024-03-06T07:22:50.080057+0000","last_clean":"2024-03-06T07:22:50.080057+0000","last_became_active":"2024-03-06T07:22:48.071136+0000","last_became_peered":"2024-03-06T07:22:48.071136+0000","last_unstale":"2024-03-06T07:22:50.080057+0000","last_undegraded":"2024-03-06T07:22:50.080057+0000","last_fullsized":"2024-03-06T07:22:50.080057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:30.767002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077039+0000","last_change":"2024-03-06T07:22:49.072607+0000","last_active":"2024-03-06T07:22:50.077039+0000","last_peered":"2024-03-06T07:22:50.077039+0000","last_clean":"2024-03-06T07:22:50.077039+0000","last_became_active":"2024-03-06T07:22:49.071896+0000","last_became_peered":"2024-03-06T07:22:49.071896+0000","last_unstale":"2024-03-06T07:22:50.077039+0000","last_undegraded":"2024-03-06T07:22:50.077039+0000","last_fullsized":"2024-03-06T07:22:50.077039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:17.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077163+0000","last_change":"2024-03-06T07:22:48.071003+0000","last_active":"2024-03-06T07:22:50.077163+0000","last_peered":"2024-03-06T07:22:50.077163+0000","last_clean":"2024-03-06T07:22:50.077163+0000","last_became_active":"2024-03-06T07:22:48.070666+0000","last_became_peered":"2024-03-06T07:22:48.070666+0000","last_unstale":"2024-03-06T07:22:50.077163+0000","last_undegraded":"2024-03-06T07:22:50.077163+0000","last_fullsized":"2024-03-06T07:22:50.077163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678235+0000","last_change":"2024-03-06T07:22:49.077332+0000","last_active":"2024-03-06T07:22:51.678235+0000","last_peered":"2024-03-06T07:22:51.678235+0000","last_clean":"2024-03-06T07:22:51.678235+0000","last_became_active":"2024-03-06T07:22:49.077034+0000","last_became_peered":"2024-03-06T07:22:49.077034+0000","last_unstale":"2024-03-06T07:22:51.678235+0000","last_undegraded":"2024-03-06T07:22:51.678235+0000","last_fullsized":"2024-03-06T07:22:51.678235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:43.569514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620780+0000","last_change":"2024-03-06T07:22:48.076685+0000","last_active":"2024-03-06T07:22:50.620780+0000","last_peered":"2024-03-06T07:22:50.620780+0000","last_clean":"2024-03-06T07:22:50.620780+0000","last_became_active":"2024-03-06T07:22:48.076412+0000","last_became_peered":"2024-03-06T07:22:48.076412+0000","last_unstale":"2024-03-06T07:22:50.620780+0000","last_undegraded":"2024-03-06T07:22:50.620780+0000","last_fullsized":"2024-03-06T07:22:50.620780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678095+0000","last_change":"2024-03-06T07:22:49.069020+0000","last_active":"2024-03-06T07:22:51.678095+0000","last_peered":"2024-03-06T07:22:51.678095+0000","last_clean":"2024-03-06T07:22:51.678095+0000","last_became_active":"2024-03-06T07:22:49.068741+0000","last_became_peered":"2024-03-06T07:22:49.068741+0000","last_unstale":"2024-03-06T07:22:51.678095+0000","last_undegraded":"2024-03-06T07:22:51.678095+0000","last_fullsized":"2024-03-06T07:22:51.678095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079069+0000","last_change":"2024-03-06T07:22:48.072458+0000","last_active":"2024-03-06T07:22:50.079069+0000","last_peered":"2024-03-06T07:22:50.079069+0000","last_clean":"2024-03-06T07:22:50.079069+0000","last_became_active":"2024-03-06T07:22:48.071279+0000","last_became_peered":"2024-03-06T07:22:48.071279+0000","last_unstale":"2024-03-06T07:22:50.079069+0000","last_undegraded":"2024-03-06T07:22:50.079069+0000","last_fullsized":"2024-03-06T07:22:50.079069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:50.090884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.078676+0000","last_change":"2024-03-06T07:22:49.077623+0000","last_active":"2024-03-06T07:22:50.078676+0000","last_peered":"2024-03-06T07:22:50.078676+0000","last_clean":"2024-03-06T07:22:50.078676+0000","last_became_active":"2024-03-06T07:22:49.076719+0000","last_became_peered":"2024-03-06T07:22:49.076719+0000","last_unstale":"2024-03-06T07:22:50.078676+0000","last_undegraded":"2024-03-06T07:22:50.078676+0000","last_fullsized":"2024-03-06T07:22:50.078676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:38.976172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.077000+0000","last_change":"2024-03-06T07:22:48.071228+0000","last_active":"2024-03-06T07:22:50.077000+0000","last_peered":"2024-03-06T07:22:50.077000+0000","last_clean":"2024-03-06T07:22:50.077000+0000","last_became_active":"2024-03-06T07:22:48.070898+0000","last_became_peered":"2024-03-06T07:22:48.070898+0000","last_unstale":"2024-03-06T07:22:50.077000+0000","last_undegraded":"2024-03-06T07:22:50.077000+0000","last_fullsized":"2024-03-06T07:22:50.077000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:37.970180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677021+0000","last_change":"2024-03-06T07:22:49.069977+0000","last_active":"2024-03-06T07:22:51.677021+0000","last_peered":"2024-03-06T07:22:51.677021+0000","last_clean":"2024-03-06T07:22:51.677021+0000","last_became_active":"2024-03-06T07:22:49.069691+0000","last_became_peered":"2024-03-06T07:22:49.069691+0000","last_unstale":"2024-03-06T07:22:51.677021+0000","last_undegraded":"2024-03-06T07:22:51.677021+0000","last_fullsized":"2024-03-06T07:22:51.677021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.076906+0000","last_change":"2024-03-06T07:22:48.074693+0000","last_active":"2024-03-06T07:22:50.076906+0000","last_peered":"2024-03-06T07:22:50.076906+0000","last_clean":"2024-03-06T07:22:50.076906+0000","last_became_active":"2024-03-06T07:22:48.074261+0000","last_became_peered":"2024-03-06T07:22:48.074261+0000","last_unstale":"2024-03-06T07:22:50.076906+0000","last_undegraded":"2024-03-06T07:22:50.076906+0000","last_fullsized":"2024-03-06T07:22:50.076906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677543+0000","last_change":"2024-03-06T07:22:49.082024+0000","last_active":"2024-03-06T07:22:51.677543+0000","last_peered":"2024-03-06T07:22:51.677543+0000","last_clean":"2024-03-06T07:22:51.677543+0000","last_became_active":"2024-03-06T07:22:49.081534+0000","last_became_peered":"2024-03-06T07:22:49.081534+0000","last_unstale":"2024-03-06T07:22:51.677543+0000","last_undegraded":"2024-03-06T07:22:51.677543+0000","last_fullsized":"2024-03-06T07:22:51.677543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:26.062786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079905+0000","last_change":"2024-03-06T07:22:48.069179+0000","last_active":"2024-03-06T07:22:50.079905+0000","last_peered":"2024-03-06T07:22:50.079905+0000","last_clean":"2024-03-06T07:22:50.079905+0000","last_became_active":"2024-03-06T07:22:48.068402+0000","last_became_peered":"2024-03-06T07:22:48.068402+0000","last_unstale":"2024-03-06T07:22:50.079905+0000","last_undegraded":"2024-03-06T07:22:50.079905+0000","last_fullsized":"2024-03-06T07:22:50.079905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:09.655063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.620537+0000","last_change":"2024-03-06T07:22:49.082657+0000","last_active":"2024-03-06T07:22:50.620537+0000","last_peered":"2024-03-06T07:22:50.620537+0000","last_clean":"2024-03-06T07:22:50.620537+0000","last_became_active":"2024-03-06T07:22:49.082096+0000","last_became_peered":"2024-03-06T07:22:49.082096+0000","last_unstale":"2024-03-06T07:22:50.620537+0000","last_undegraded":"2024-03-06T07:22:50.620537+0000","last_fullsized":"2024-03-06T07:22:50.620537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:40.479593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670227+0000","last_change":"2024-03-06T07:22:49.073629+0000","last_active":"2024-03-06T07:22:50.670227+0000","last_peered":"2024-03-06T07:22:50.670227+0000","last_clean":"2024-03-06T07:22:50.670227+0000","last_became_active":"2024-03-06T07:22:49.072839+0000","last_became_peered":"2024-03-06T07:22:49.072839+0000","last_unstale":"2024-03-06T07:22:50.670227+0000","last_undegraded":"2024-03-06T07:22:50.670227+0000","last_fullsized":"2024-03-06T07:22:50.670227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:32.401483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079077+0000","last_change":"2024-03-06T07:22:48.072083+0000","last_active":"2024-03-06T07:22:50.079077+0000","last_peered":"2024-03-06T07:22:50.079077+0000","last_clean":"2024-03-06T07:22:50.079077+0000","last_became_active":"2024-03-06T07:22:48.071560+0000","last_became_peered":"2024-03-06T07:22:48.071560+0000","last_unstale":"2024-03-06T07:22:50.079077+0000","last_undegraded":"2024-03-06T07:22:50.079077+0000","last_fullsized":"2024-03-06T07:22:50.079077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.460032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.670115+0000","last_change":"2024-03-06T07:22:49.080775+0000","last_active":"2024-03-06T07:22:50.670115+0000","last_peered":"2024-03-06T07:22:50.670115+0000","last_clean":"2024-03-06T07:22:50.670115+0000","last_became_active":"2024-03-06T07:22:49.080487+0000","last_became_peered":"2024-03-06T07:22:49.080487+0000","last_unstale":"2024-03-06T07:22:50.670115+0000","last_undegraded":"2024-03-06T07:22:50.670115+0000","last_fullsized":"2024-03-06T07:22:50.670115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:43.229187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:50.079758+0000","last_change":"2024-03-06T07:22:48.074550+0000","last_active":"2024-03-06T07:22:50.079758+0000","last_peered":"2024-03-06T07:22:50.079758+0000","last_clean":"2024-03-06T07:22:50.079758+0000","last_became_active":"2024-03-06T07:22:48.073981+0000","last_became_peered":"2024-03-06T07:22:48.073981+0000","last_unstale":"2024-03-06T07:22:50.079758+0000","last_undegraded":"2024-03-06T07:22:50.079758+0000","last_fullsized":"2024-03-06T07:22:50.079758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:13.210796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677964+0000","last_change":"2024-03-06T07:22:49.075565+0000","last_active":"2024-03-06T07:22:51.677964+0000","last_peered":"2024-03-06T07:22:51.677964+0000","last_clean":"2024-03-06T07:22:51.677964+0000","last_became_active":"2024-03-06T07:22:49.075254+0000","last_became_peered":"2024-03-06T07:22:49.075254+0000","last_unstale":"2024-03-06T07:22:51.677964+0000","last_undegraded":"2024-03-06T07:22:51.677964+0000","last_fullsized":"2024-03-06T07:22:51.677964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:09.529160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678098+0000","last_change":"2024-03-06T07:22:48.073512+0000","last_active":"2024-03-06T07:22:51.678098+0000","last_peered":"2024-03-06T07:22:51.678098+0000","last_clean":"2024-03-06T07:22:51.678098+0000","last_became_active":"2024-03-06T07:22:48.073181+0000","last_became_peered":"2024-03-06T07:22:48.073181+0000","last_unstale":"2024-03-06T07:22:51.678098+0000","last_undegraded":"2024-03-06T07:22:51.678098+0000","last_fullsized":"2024-03-06T07:22:51.678098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:44.611763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.677959+0000","last_change":"2024-03-06T07:22:49.079451+0000","last_active":"2024-03-06T07:22:51.677959+0000","last_peered":"2024-03-06T07:22:51.677959+0000","last_clean":"2024-03-06T07:22:51.677959+0000","last_became_active":"2024-03-06T07:22:49.078927+0000","last_became_peered":"2024-03-06T07:22:49.078927+0000","last_unstale":"2024-03-06T07:22:51.677959+0000","last_undegraded":"2024-03-06T07:22:51.677959+0000","last_fullsized":"2024-03-06T07:22:51.677959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:48.047522+0000","last_clean_scrub_stamp":"2024-03-06T07:22:48.047522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:07.781375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.678094+0000","last_change":"2024-03-06T07:22:48.062906+0000","last_active":"2024-03-06T07:22:51.678094+0000","last_peered":"2024-03-06T07:22:51.678094+0000","last_clean":"2024-03-06T07:22:51.678094+0000","last_became_active":"2024-03-06T07:22:48.062497+0000","last_became_peered":"2024-03-06T07:22:48.062497+0000","last_unstale":"2024-03-06T07:22:51.678094+0000","last_undegraded":"2024-03-06T07:22:51.678094+0000","last_fullsized":"2024-03-06T07:22:51.678094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:47.041531+0000","last_clean_scrub_stamp":"2024-03-06T07:22:47.041531+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:51.443747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:58.112 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:22:58.112 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:22:58.112 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:22:58.112 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:22:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.251+0000 7f567e8df640 1 Processor -- start 2024-03-06T07:22:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.252+0000 7f567e8df640 1 -- start start 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.252+0000 7f567e8df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 0x7f56780b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.252+0000 7f567e8df640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f56780d3400 0x7f5678071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.252+0000 7f567e8df640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5678071e60 con 0x7f56780b2b50 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.252+0000 7f567e8df640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56780d2f70 con 0x7f56780d3400 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.252+0000 7f567e8df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56780d4be0 con 0x7f5678071540 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.252+0000 7f567ce7c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f56780b2b50 0x7f56780d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50148/0 (socket says 172.21.15.27:50148) 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.252+0000 7f567ce7c640 1 -- 172.21.15.27:0/3851711814 learned_addr learned my addr 172.21.15.27:0/3851711814 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.253+0000 7f56777fe640 1 --2- 172.21.15.27:0/3851711814 >> v2:172.21.15.69:3300/0 conn(0x7f56780d3400 0x7f5678071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.253+0000 7f5677fff640 1 --2- 172.21.15.27:0/3851711814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 0x7f56780b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.253+0000 7f5676ffd640 1 -- 172.21.15.27:0/3851711814 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612780899 0 0) 0x7f5678071e60 con 0x7f56780b2b50 2024-03-06T07:22:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.253+0000 7f5676ffd640 1 -- 172.21.15.27:0/3851711814 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5660003520 con 0x7f56780b2b50 2024-03-06T07:22:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.253+0000 7f56777fe640 1 -- 172.21.15.27:0/3851711814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 msgr2=0x7f56780b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.253+0000 7f56777fe640 1 --2- 172.21.15.27:0/3851711814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 0x7f56780b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.253+0000 7f56777fe640 1 -- 172.21.15.27:0/3851711814 >> v1:172.21.15.27:6789/0 conn(0x7f56780b2b50 legacy=0x7f56780d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.253+0000 7f56777fe640 1 -- 172.21.15.27:0/3851711814 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56780ce610 con 0x7f56780d3400 2024-03-06T07:22:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.253+0000 7f5677fff640 1 --2- 172.21.15.27:0/3851711814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 0x7f56780b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.254+0000 7f56777fe640 1 --2- 172.21.15.27:0/3851711814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780d3400 0x7f5678071920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f566c0097a0 tx=0x7f566c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd8b82524e0746ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.254+0000 7f5676ffd640 1 -- 172.21.15.27:0/3851711814 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c035020 con 0x7f56780d3400 2024-03-06T07:22:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.254+0000 7f5676ffd640 1 -- 172.21.15.27:0/3851711814 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c031830 con 0x7f56780d3400 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.254+0000 7f5676ffd640 1 -- 172.21.15.27:0/3851711814 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c031be0 con 0x7f56780d3400 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.254+0000 7f567e8df640 1 -- 172.21.15.27:0/3851711814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780d3400 msgr2=0x7f5678071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.254+0000 7f567e8df640 1 --2- 172.21.15.27:0/3851711814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780d3400 0x7f5678071920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f566c0097a0 tx=0x7f566c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.255+0000 7f567e8df640 1 -- 172.21.15.27:0/3851711814 shutdown_connections 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.255+0000 7f567e8df640 1 --2- 172.21.15.27:0/3851711814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780d3400 0x7f5678071920 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.255+0000 7f567e8df640 1 --2- 172.21.15.27:0/3851711814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 0x7f56780b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.255+0000 7f567e8df640 1 -- 172.21.15.27:0/3851711814 >> 172.21.15.27:0/3851711814 conn(0x7f56780d3ca0 msgr2=0x7f56780d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.255+0000 7f567e8df640 1 -- 172.21.15.27:0/3851711814 shutdown_connections 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.255+0000 7f567e8df640 1 -- 172.21.15.27:0/3851711814 wait complete. 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.256+0000 7f567e8df640 1 Processor -- start 2024-03-06T07:22:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.256+0000 7f567e8df640 1 -- start start 2024-03-06T07:22:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567e8df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 0x7f56781f5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567e8df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780b2b50 0x7f567814ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567e8df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56780d3400 0x7f56781e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567e8df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56780ce8f0 con 0x7f56780d3400 2024-03-06T07:22:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567e8df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56780d2c00 con 0x7f56780b2b50 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567e8df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567815e370 con 0x7f5678071540 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567ce7c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56780d3400 0x7f56781e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567ce7c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56780d3400 0x7f56781e6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51858/0 (socket says 172.21.15.27:51858) 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567ce7c640 1 -- 172.21.15.27:0/4097887042 learned_addr learned my addr 172.21.15.27:0/4097887042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f56777fe640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780b2b50 0x7f567814ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f5677fff640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 0x7f56781f5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567ce7c640 1 -- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 msgr2=0x7f56781f5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567ce7c640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 0x7f56781f5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567ce7c640 1 -- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780b2b50 msgr2=0x7f567814ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567ce7c640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780b2b50 0x7f567814ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567ce7c640 1 -- 172.21.15.27:0/4097887042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56781e7ba0 con 0x7f56780d3400 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f56777fe640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780b2b50 0x7f567814ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567ce7c640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56780d3400 0x7f56781e6dc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f567000cb40 tx=0x7f567000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f5674ff9640 1 -- 172.21.15.27:0/4097887042 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670017070 con 0x7f56780d3400 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f5674ff9640 1 -- 172.21.15.27:0/4097887042 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567000aca0 con 0x7f56780d3400 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5678150480 con 0x7f56780d3400 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.257+0000 7f5674ff9640 1 -- 172.21.15.27:0/4097887042 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670012620 con 0x7f56780d3400 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.258+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56781f60b0 con 0x7f56780d3400 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.258+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5678068700 con 0x7f56780d3400 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.259+0000 7f5674ff9640 1 -- 172.21.15.27:0/4097887042 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f567001a070 con 0x7f56780d3400 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.260+0000 7f5674ff9640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f566406b500 0x7f566406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.260+0000 7f5674ff9640 1 -- 172.21.15.27:0/4097887042 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5670089fe0 con 0x7f56780d3400 2024-03-06T07:22:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.260+0000 7f5677fff640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f566406b500 0x7f566406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.260+0000 7f5677fff640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f566406b500 0x7f566406d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5668009560 tx=0x7f5668008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.264+0000 7f5674ff9640 1 -- 172.21.15.27:0/4097887042 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56781e7ba0 con 0x7f56780d3400 2024-03-06T07:22:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.482+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f56780b95f0 con 0x7f56780d3400 2024-03-06T07:22:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.482+0000 7f5674ff9640 1 -- 172.21.15.27:0/4097887042 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f5670058e60 con 0x7f56780d3400 2024-03-06T07:22:58.485 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:58.485 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:22:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f566406b500 msgr2=0x7f566406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f566406b500 0x7f566406d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5668009560 tx=0x7f5668008040 comp rx=0 tx=0).stop 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56780d3400 msgr2=0x7f56781e6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56780d3400 0x7f56781e6dc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f567000cb40 tx=0x7f567000a060 comp rx=0 tx=0).stop 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 shutdown_connections 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f566406b500 0x7f566406d9c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56780d3400 0x7f56781e6dc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56780b2b50 0x7f567814ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 --2- 172.21.15.27:0/4097887042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5678071540 0x7f56781f5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 >> 172.21.15.27:0/4097887042 conn(0x7f56780d3ca0 msgr2=0x7f567815f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 shutdown_connections 2024-03-06T07:22:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.484+0000 7f567e8df640 1 -- 172.21.15.27:0/4097887042 wait complete. 2024-03-06T07:22:58.499 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:22:58.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.604+0000 7f0e69fa0640 1 Processor -- start 2024-03-06T07:22:58.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.604+0000 7f0e69fa0640 1 -- start start 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e69fa0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6415f7e0 0x7f0e6415fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e69fa0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e6415cee0 0x7f0e640cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e69fa0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e641605d0 con 0x7f0e640cec00 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e69fa0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e64160100 con 0x7f0e6415cee0 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e69fa0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640cc960 con 0x7f0e6415f7e0 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e6979f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e6415cee0 0x7f0e640cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e6979f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e6415cee0 0x7f0e640cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51864/0 (socket says 172.21.15.27:51864) 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e6979f640 1 -- 172.21.15.27:0/89551351 learned_addr learned my addr 172.21.15.27:0/89551351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e68f9e640 1 --2- 172.21.15.27:0/89551351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6415f7e0 0x7f0e6415fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e6979f640 1 -- 172.21.15.27:0/89551351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6415f7e0 msgr2=0x7f0e6415fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e6979f640 1 --2- 172.21.15.27:0/89551351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6415f7e0 0x7f0e6415fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e6979f640 1 -- 172.21.15.27:0/89551351 >> v1:172.21.15.69:6789/0 conn(0x7f0e640cec00 legacy=0x7f0e640cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e6979f640 1 -- 172.21.15.27:0/89551351 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640ccc40 con 0x7f0e6415cee0 2024-03-06T07:22:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.605+0000 7f0e68f9e640 1 --2- 172.21.15.27:0/89551351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6415f7e0 0x7f0e6415fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.606+0000 7f0e6979f640 1 --2- 172.21.15.27:0/89551351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415cee0 0x7f0e640cfbe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009e50 tx=0x7f0e5c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140189139196f04f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.606+0000 7f0e637fe640 1 -- 172.21.15.27:0/89551351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c02cb00 con 0x7f0e6415cee0 2024-03-06T07:22:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.606+0000 7f0e637fe640 1 -- 172.21.15.27:0/89551351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c02cca0 con 0x7f0e6415cee0 2024-03-06T07:22:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.606+0000 7f0e637fe640 1 -- 172.21.15.27:0/89551351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c034610 con 0x7f0e6415cee0 2024-03-06T07:22:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.607+0000 7f0e69fa0640 1 -- 172.21.15.27:0/89551351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415cee0 msgr2=0x7f0e640cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.607+0000 7f0e69fa0640 1 --2- 172.21.15.27:0/89551351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415cee0 0x7f0e640cfbe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009e50 tx=0x7f0e5c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.607+0000 7f0e69fa0640 1 -- 172.21.15.27:0/89551351 shutdown_connections 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.607+0000 7f0e69fa0640 1 --2- 172.21.15.27:0/89551351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415cee0 0x7f0e640cfbe0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.607+0000 7f0e69fa0640 1 --2- 172.21.15.27:0/89551351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6415f7e0 0x7f0e6415fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.607+0000 7f0e69fa0640 1 -- 172.21.15.27:0/89551351 >> 172.21.15.27:0/89551351 conn(0x7f0e640b3430 msgr2=0x7f0e640b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.607+0000 7f0e69fa0640 1 -- 172.21.15.27:0/89551351 shutdown_connections 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.607+0000 7f0e69fa0640 1 -- 172.21.15.27:0/89551351 wait complete. 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.608+0000 7f0e69fa0640 1 Processor -- start 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.608+0000 7f0e69fa0640 1 -- start start 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e69fa0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e640cec00 0x7f0e640cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e69fa0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e6415cee0 0x7f0e641e7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e69fa0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415f7e0 0x7f0e641e7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e69fa0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640cd4e0 con 0x7f0e6415f7e0 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e69fa0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640cd200 con 0x7f0e6415cee0 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e69fa0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640cc5c0 con 0x7f0e640cec00 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e6979f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415f7e0 0x7f0e641e7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e6979f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415f7e0 0x7f0e641e7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51872/0 (socket says 172.21.15.27:51872) 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e6979f640 1 -- 172.21.15.27:0/4062445455 learned_addr learned my addr 172.21.15.27:0/4062445455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.609+0000 7f0e63fff640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e6415cee0 0x7f0e641e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e68f9e640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e640cec00 0x7f0e640cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e6979f640 1 -- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e640cec00 msgr2=0x7f0e640cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e6979f640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e640cec00 0x7f0e640cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e6979f640 1 -- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e6415cee0 msgr2=0x7f0e641e7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e6979f640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e6415cee0 0x7f0e641e7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e6979f640 1 -- 172.21.15.27:0/4062445455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640c93b0 con 0x7f0e6415f7e0 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e68f9e640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e640cec00 0x7f0e640cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e6979f640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415f7e0 0x7f0e641e7c40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009e50 tx=0x7f0e5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e617fa640 1 -- 172.21.15.27:0/4062445455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c043020 con 0x7f0e6415f7e0 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e617fa640 1 -- 172.21.15.27:0/4062445455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c02ce50 con 0x7f0e6415f7e0 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e617fa640 1 -- 172.21.15.27:0/4062445455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c009510 con 0x7f0e6415f7e0 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e640c9680 con 0x7f0e6415f7e0 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.610+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e640c9c40 con 0x7f0e6415f7e0 2024-03-06T07:22:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.611+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e640ca300 con 0x7f0e6415f7e0 2024-03-06T07:22:58.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.612+0000 7f0e617fa640 1 -- 172.21.15.27:0/4062445455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e5c046020 con 0x7f0e6415f7e0 2024-03-06T07:22:58.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.612+0000 7f0e617fa640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4806b4d0 0x7f0e4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.612+0000 7f0e617fa640 1 -- 172.21.15.27:0/4062445455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e5c0ac720 con 0x7f0e6415f7e0 2024-03-06T07:22:58.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.612+0000 7f0e68f9e640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4806b4d0 0x7f0e4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.613+0000 7f0e68f9e640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4806b4d0 0x7f0e4806d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0e58006fd0 tx=0x7f0e58008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.617+0000 7f0e617fa640 1 -- 172.21.15.27:0/4062445455 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e640ca300 con 0x7f0e6415f7e0 2024-03-06T07:22:58.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.798+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e6404f330 con 0x7f0e6415f7e0 2024-03-06T07:22:58.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.799+0000 7f0e617fa640 1 -- 172.21.15.27:0/4062445455 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f0e5c04d030 con 0x7f0e6415f7e0 2024-03-06T07:22:58.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:22:58.802 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:58.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4806b4d0 msgr2=0x7f0e4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4806b4d0 0x7f0e4806d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0e58006fd0 tx=0x7f0e58008040 comp rx=0 tx=0).stop 2024-03-06T07:22:58.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415f7e0 msgr2=0x7f0e641e7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415f7e0 0x7f0e641e7c40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009e50 tx=0x7f0e5c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:58.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 shutdown_connections 2024-03-06T07:22:58.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4806b4d0 0x7f0e4806d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6415f7e0 0x7f0e641e7c40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e6415cee0 0x7f0e641e7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 --2- 172.21.15.27:0/4062445455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e640cec00 0x7f0e640cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 >> 172.21.15.27:0/4062445455 conn(0x7f0e640b3430 msgr2=0x7f0e640aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 shutdown_connections 2024-03-06T07:22:58.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.801+0000 7f0e69fa0640 1 -- 172.21.15.27:0/4062445455 wait complete. 2024-03-06T07:22:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:58.816 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.923+0000 7f10172d4640 1 Processor -- start 2024-03-06T07:22:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.923+0000 7f10172d4640 1 -- start start 2024-03-06T07:22:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f10172d4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f101015e2f0 0x7f10100b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f10172d4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f101006b4e0 0x7f101006b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f10172d4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101015e6d0 con 0x7f101006be80 2024-03-06T07:22:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f10172d4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10100ccd00 con 0x7f101015e2f0 2024-03-06T07:22:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f10172d4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10100ccfe0 con 0x7f101006b4e0 2024-03-06T07:22:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f1015070640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f101015e2f0 0x7f10100b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f1015070640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f101015e2f0 0x7f10100b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51884/0 (socket says 172.21.15.27:51884) 2024-03-06T07:22:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f1015070640 1 -- 172.21.15.27:0/3087096259 learned_addr learned my addr 172.21.15.27:0/3087096259 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f101486f640 1 --2- 172.21.15.27:0/3087096259 >> v2:172.21.15.69:3300/0 conn(0x7f101006b4e0 0x7f101006b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f1015070640 1 -- 172.21.15.27:0/3087096259 >> v2:172.21.15.69:3300/0 conn(0x7f101006b4e0 msgr2=0x7f101006b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f1015070640 1 --2- 172.21.15.27:0/3087096259 >> v2:172.21.15.69:3300/0 conn(0x7f101006b4e0 0x7f101006b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f1015070640 1 -- 172.21.15.27:0/3087096259 >> v1:172.21.15.69:6789/0 conn(0x7f101006be80 legacy=0x7f10100cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.924+0000 7f1015070640 1 -- 172.21.15.27:0/3087096259 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10100cd2c0 con 0x7f101015e2f0 2024-03-06T07:22:58.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.925+0000 7f1015070640 1 --2- 172.21.15.27:0/3087096259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101015e2f0 0x7f10100b41f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1010071f40 tx=0x7f100002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe8c45af5c407bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.925+0000 7f1007fff640 1 -- 172.21.15.27:0/3087096259 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100002e070 con 0x7f101015e2f0 2024-03-06T07:22:58.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.925+0000 7f1007fff640 1 -- 172.21.15.27:0/3087096259 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1000037070 con 0x7f101015e2f0 2024-03-06T07:22:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.925+0000 7f1007fff640 1 -- 172.21.15.27:0/3087096259 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000031a00 con 0x7f101015e2f0 2024-03-06T07:22:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.926+0000 7f10172d4640 1 -- 172.21.15.27:0/3087096259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101015e2f0 msgr2=0x7f10100b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.926+0000 7f10172d4640 1 --2- 172.21.15.27:0/3087096259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101015e2f0 0x7f10100b41f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1010071f40 tx=0x7f100002b880 comp rx=0 tx=0).stop 2024-03-06T07:22:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.926+0000 7f10172d4640 1 -- 172.21.15.27:0/3087096259 shutdown_connections 2024-03-06T07:22:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.926+0000 7f10172d4640 1 --2- 172.21.15.27:0/3087096259 >> v2:172.21.15.69:3300/0 conn(0x7f101006b4e0 0x7f101006b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.926+0000 7f10172d4640 1 --2- 172.21.15.27:0/3087096259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101015e2f0 0x7f10100b41f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.926+0000 7f10172d4640 1 -- 172.21.15.27:0/3087096259 >> 172.21.15.27:0/3087096259 conn(0x7f10100b36d0 msgr2=0x7f10100b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.926+0000 7f10172d4640 1 -- 172.21.15.27:0/3087096259 shutdown_connections 2024-03-06T07:22:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.926+0000 7f10172d4640 1 -- 172.21.15.27:0/3087096259 wait complete. 2024-03-06T07:22:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.928+0000 7f10172d4640 1 Processor -- start 2024-03-06T07:22:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.928+0000 7f10172d4640 1 -- start start 2024-03-06T07:22:58.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.929+0000 7f10172d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101006b4e0 0x7f101015e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.929+0000 7f10172d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101006be80 0x7f1010159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.929+0000 7f10172d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f101015e2f0 0x7f10100c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.929+0000 7f10172d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101006c450 con 0x7f101006be80 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.929+0000 7f10172d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101006c2d0 con 0x7f101006b4e0 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.929+0000 7f10172d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10100cead0 con 0x7f101015e2f0 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.929+0000 7f101486f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101006be80 0x7f1010159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f101486f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101006be80 0x7f1010159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51898/0 (socket says 172.21.15.27:51898) 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f101486f640 1 -- 172.21.15.27:0/203360082 learned_addr learned my addr 172.21.15.27:0/203360082 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f1015070640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101006b4e0 0x7f101015e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f1015871640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f101015e2f0 0x7f10100c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f101486f640 1 -- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f101015e2f0 msgr2=0x7f10100c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f101486f640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f101015e2f0 0x7f10100c58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f101486f640 1 -- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101006b4e0 msgr2=0x7f101015e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f101486f640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101006b4e0 0x7f101015e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f101486f640 1 -- 172.21.15.27:0/203360082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101015f120 con 0x7f101006be80 2024-03-06T07:22:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f1015871640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f101015e2f0 0x7f10100c58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f101486f640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101006be80 0x7f1010159f70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f100c00e9b0 tx=0x7f100c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f1005ffb640 1 -- 172.21.15.27:0/203360082 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c017070 con 0x7f101006be80 2024-03-06T07:22:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f1005ffb640 1 -- 172.21.15.27:0/203360082 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c00c9a0 con 0x7f101006be80 2024-03-06T07:22:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f1005ffb640 1 -- 172.21.15.27:0/203360082 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c00cd90 con 0x7f101006be80 2024-03-06T07:22:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101015f400 con 0x7f101006be80 2024-03-06T07:22:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.930+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101015f9c0 con 0x7f101006be80 2024-03-06T07:22:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.932+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101009fd30 con 0x7f101006be80 2024-03-06T07:22:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.932+0000 7f1005ffb640 1 -- 172.21.15.27:0/203360082 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f100c01a070 con 0x7f101006be80 2024-03-06T07:22:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.932+0000 7f1005ffb640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ff406b500 0x7f0ff406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.932+0000 7f1005ffb640 1 -- 172.21.15.27:0/203360082 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f100c08a4e0 con 0x7f101006be80 2024-03-06T07:22:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.933+0000 7f1015070640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ff406b500 0x7f0ff406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.933+0000 7f1015070640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ff406b500 0x7f0ff406d9c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1000036000 tx=0x7f1000008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:58.937+0000 7f1005ffb640 1 -- 172.21.15.27:0/203360082 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f100c0593f0 con 0x7f101006be80 2024-03-06T07:22:59.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.120+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10100d0660 con 0x7f101006be80 2024-03-06T07:22:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.120+0000 7f1005ffb640 1 -- 172.21.15.27:0/203360082 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f100c058cf0 con 0x7f101006be80 2024-03-06T07:22:59.123 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:22:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ff406b500 msgr2=0x7f0ff406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ff406b500 0x7f0ff406d9c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1000036000 tx=0x7f1000008030 comp rx=0 tx=0).stop 2024-03-06T07:22:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101006be80 msgr2=0x7f1010159f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101006be80 0x7f1010159f70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f100c00e9b0 tx=0x7f100c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 shutdown_connections 2024-03-06T07:22:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ff406b500 0x7f0ff406d9c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f101015e2f0 0x7f10100c58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101006be80 0x7f1010159f70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 --2- 172.21.15.27:0/203360082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101006b4e0 0x7f101015e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 >> 172.21.15.27:0/203360082 conn(0x7f10100b36d0 msgr2=0x7f101015cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 shutdown_connections 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.122+0000 7f10172d4640 1 -- 172.21.15.27:0/203360082 wait complete. 2024-03-06T07:22:59.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.245+0000 7fbef560f640 1 Processor -- start 2024-03-06T07:22:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.245+0000 7fbef560f640 1 -- start start 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.245+0000 7fbef560f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef00ce9d0 0x7fbef0162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.245+0000 7fbef560f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbef015e160 0x7fbef00cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.245+0000 7fbef560f640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef0162990 con 0x7fbef00b3020 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbef560f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef00d1de0 con 0x7fbef015e160 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbef560f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00cf2f0 con 0x7fbef00ce9d0 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeef7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbef00b3020 0x7fbef0071540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50154/0 (socket says 172.21.15.27:50154) 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeef7fe640 1 -- 172.21.15.27:0/2032283392 learned_addr learned my addr 172.21.15.27:0/2032283392 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/2032283392 >> v2:172.21.15.69:3300/0 conn(0x7fbef015e160 0x7fbef00cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeeeffd640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef00ce9d0 0x7fbef0162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181451293 0 0) 0x7fbef0162990 con 0x7fbef00b3020 2024-03-06T07:22:59.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed8003520 con 0x7fbef00b3020 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3189726684 0 0) 0x7fbed8003520 con 0x7fbef00b3020 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef00ce9d0 msgr2=0x7fbef0162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef00ce9d0 0x7fbef0162450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 >> v2:172.21.15.69:3300/0 conn(0x7fbef015e160 msgr2=0x7fbef00cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 --2- 172.21.15.27:0/2032283392 >> v2:172.21.15.69:3300/0 conn(0x7fbef015e160 0x7fbef00cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef00b97c0 con 0x7fbef00b3020 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fbee80019f0 con 0x7fbef00b3020 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/2032283392 >> v2:172.21.15.69:3300/0 conn(0x7fbef015e160 0x7fbef00cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 >> v1:172.21.15.27:6789/0 conn(0x7fbef00b3020 legacy=0x7fbef0071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef00ce9d0 0x7fbed80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8003f30 0x7fbed8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed8004d40 0x7fbed80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00d1de0 con 0x7fbed8003f30 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00cf2f0 con 0x7fbed8004d40 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed800b9f0 con 0x7fbef00ce9d0 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8003f30 0x7fbed8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeeeffd640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef00ce9d0 0x7fbed80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.246+0000 7fbeef7fe640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed8004d40 0x7fbed80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.247+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef00ce9d0 msgr2=0x7fbed80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.247+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef00ce9d0 0x7fbed80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.247+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed8004d40 msgr2=0x7fbed80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.247+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed8004d40 0x7fbed80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.247+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2032283392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed8003520 con 0x7fbed8003f30 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.247+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8003f30 0x7fbed8004800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009c60 tx=0x7fbed8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb25aff441f7de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4039070 con 0x7fbed8003f30 2024-03-06T07:22:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee402c950 con 0x7fbed8003f30 2024-03-06T07:22:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbeedffb640 1 -- 172.21.15.27:0/2032283392 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4009860 con 0x7fbed8003f30 2024-03-06T07:22:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbef560f640 1 -- 172.21.15.27:0/2032283392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8003f30 msgr2=0x7fbed8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbef560f640 1 --2- 172.21.15.27:0/2032283392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8003f30 0x7fbed8004800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009c60 tx=0x7fbed8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbeef7fe640 1 -- 172.21.15.27:0/2032283392 reap_dead start 2024-03-06T07:22:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbef560f640 1 -- 172.21.15.27:0/2032283392 shutdown_connections 2024-03-06T07:22:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbef560f640 1 -- 172.21.15.27:0/2032283392 >> 172.21.15.27:0/2032283392 conn(0x7fbef00d3c90 msgr2=0x7fbef00d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbef560f640 1 -- 172.21.15.27:0/2032283392 shutdown_connections 2024-03-06T07:22:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.248+0000 7fbef560f640 1 -- 172.21.15.27:0/2032283392 wait complete. 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbef560f640 1 Processor -- start 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbef560f640 1 -- start start 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbef560f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbed8003f30 0x7fbef00cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbef560f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef00ab440 0x7fbef0151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbef560f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef00b5760 0x7fbef00b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbef560f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00d3280 con 0x7fbef00ab440 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbef560f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00d1a70 con 0x7fbef00b5760 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbef560f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00d3400 con 0x7fbed8003f30 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbeee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef00ab440 0x7fbef0151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbeee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef00ab440 0x7fbef0151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51924/0 (socket says 172.21.15.27:51924) 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbeee7fc640 1 -- 172.21.15.27:0/1342822604 learned_addr learned my addr 172.21.15.27:0/1342822604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.250+0000 7fbeeeffd640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbed8003f30 0x7fbef00cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbeef7fe640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef00b5760 0x7fbef00b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbeee7fc640 1 -- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbed8003f30 msgr2=0x7fbef00cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbed8003f30 0x7fbef00cc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbeee7fc640 1 -- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef00b5760 msgr2=0x7fbef00b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef00b5760 0x7fbef00b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbeee7fc640 1 -- 172.21.15.27:0/1342822604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef00ccbc0 con 0x7fbef00ab440 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbeef7fe640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef00b5760 0x7fbef00b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef00ab440 0x7fbef0151070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009bb0 tx=0x7fbee4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbed77fe640 1 -- 172.21.15.27:0/1342822604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4039050 con 0x7fbef00ab440 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbed77fe640 1 -- 172.21.15.27:0/1342822604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee403a040 con 0x7fbef00ab440 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbed77fe640 1 -- 172.21.15.27:0/1342822604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4009860 con 0x7fbef00ab440 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef00ccea0 con 0x7fbef00ab440 2024-03-06T07:22:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.251+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef00cd460 con 0x7fbef00ab440 2024-03-06T07:22:59.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.252+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef006cda0 con 0x7fbef00ab440 2024-03-06T07:22:59.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.253+0000 7fbed77fe640 1 -- 172.21.15.27:0/1342822604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbee4047020 con 0x7fbef00ab440 2024-03-06T07:22:59.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.253+0000 7fbed77fe640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b500 0x7fbedc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.253+0000 7fbed77fe640 1 -- 172.21.15.27:0/1342822604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbee40aba90 con 0x7fbef00ab440 2024-03-06T07:22:59.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.254+0000 7fbeeeffd640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b500 0x7fbedc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.254+0000 7fbeeeffd640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b500 0x7fbedc06d9c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009ae0 tx=0x7fbee0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.258+0000 7fbed77fe640 1 -- 172.21.15.27:0/1342822604 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbee404e020 con 0x7fbef00ab440 2024-03-06T07:22:59.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.439+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbef004f410 con 0x7fbef00ab440 2024-03-06T07:22:59.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.440+0000 7fbed77fe640 1 -- 172.21.15.27:0/1342822604 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbee407a910 con 0x7fbef00ab440 2024-03-06T07:22:59.442 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:22:59.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b500 msgr2=0x7fbedc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b500 0x7fbedc06d9c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009ae0 tx=0x7fbee0009190 comp rx=0 tx=0).stop 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef00ab440 msgr2=0x7fbef0151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef00ab440 0x7fbef0151070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009bb0 tx=0x7fbee4004730 comp rx=0 tx=0).stop 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 shutdown_connections 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b500 0x7fbedc06d9c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef00b5760 0x7fbef00b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef00ab440 0x7fbef0151070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 --2- 172.21.15.27:0/1342822604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbed8003f30 0x7fbef00cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 >> 172.21.15.27:0/1342822604 conn(0x7fbef00d3c90 msgr2=0x7fbef00b70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 shutdown_connections 2024-03-06T07:22:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.442+0000 7fbef560f640 1 -- 172.21.15.27:0/1342822604 wait complete. 2024-03-06T07:22:59.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.567+0000 7f749b6eb640 1 Processor -- start 2024-03-06T07:22:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.567+0000 7f749b6eb640 1 -- start start 2024-03-06T07:22:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f749b6eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f749415df40 0x7f7494066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f749b6eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 0x7f7494067240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f749b6eb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749415e6d0 con 0x7f74940b3ce0 2024-03-06T07:22:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f749b6eb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749415f000 con 0x7f749415df40 2024-03-06T07:22:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f749b6eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749415e9b0 con 0x7f749406e3a0 2024-03-06T07:22:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f7499487640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f749415df40 0x7f7494066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f7499487640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f749415df40 0x7f7494066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51938/0 (socket says 172.21.15.27:51938) 2024-03-06T07:22:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f7499487640 1 -- 172.21.15.27:0/3096795309 learned_addr learned my addr 172.21.15.27:0/3096795309 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f7498c86640 1 --1- 172.21.15.27:0/3096795309 >> v1:172.21.15.27:6789/0 conn(0x7f74940b3ce0 0x7f749406dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50158/0 (socket says 172.21.15.27:50158) 2024-03-06T07:22:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f7499c88640 1 --2- 172.21.15.27:0/3096795309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 0x7f7494067240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f7499487640 1 -- 172.21.15.27:0/3096795309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 msgr2=0x7f7494067240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f7499487640 1 --2- 172.21.15.27:0/3096795309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 0x7f7494067240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f7499487640 1 -- 172.21.15.27:0/3096795309 >> v1:172.21.15.27:6789/0 conn(0x7f74940b3ce0 legacy=0x7f749406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f748bfff640 1 -- 172.21.15.27:0/3096795309 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342307257 0 0) 0x7f749415e6d0 con 0x7f74940b3ce0 2024-03-06T07:22:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.568+0000 7f7499487640 1 -- 172.21.15.27:0/3096795309 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749415ec90 con 0x7f749415df40 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.569+0000 7f7499487640 1 --2- 172.21.15.27:0/3096795309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f749415df40 0x7f7494066d00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7490009870 tx=0x7f749002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bc6037b7b4476a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.569+0000 7f748bfff640 1 -- 172.21.15.27:0/3096795309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490034070 con 0x7f749415df40 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.569+0000 7f748bfff640 1 -- 172.21.15.27:0/3096795309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7490030840 con 0x7f749415df40 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.569+0000 7f748bfff640 1 -- 172.21.15.27:0/3096795309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490030c10 con 0x7f749415df40 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.570+0000 7f749b6eb640 1 -- 172.21.15.27:0/3096795309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f749415df40 msgr2=0x7f7494066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.570+0000 7f749b6eb640 1 --2- 172.21.15.27:0/3096795309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f749415df40 0x7f7494066d00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7490009870 tx=0x7f749002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.570+0000 7f749b6eb640 1 -- 172.21.15.27:0/3096795309 shutdown_connections 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.570+0000 7f749b6eb640 1 --2- 172.21.15.27:0/3096795309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 0x7f7494067240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.570+0000 7f749b6eb640 1 --2- 172.21.15.27:0/3096795309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f749415df40 0x7f7494066d00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.570+0000 7f749b6eb640 1 -- 172.21.15.27:0/3096795309 >> 172.21.15.27:0/3096795309 conn(0x7f74940d09c0 msgr2=0x7f74940d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.570+0000 7f749b6eb640 1 -- 172.21.15.27:0/3096795309 shutdown_connections 2024-03-06T07:22:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.570+0000 7f749b6eb640 1 -- 172.21.15.27:0/3096795309 wait complete. 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.571+0000 7f749b6eb640 1 Processor -- start 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.571+0000 7f749b6eb640 1 -- start start 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f749b6eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 0x7f7494143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f749b6eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74940b3ce0 0x7f749413ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f749b6eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749415df40 0x7f74941e7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f749b6eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7494161ad0 con 0x7f74940b3ce0 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f749b6eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7494161950 con 0x7f749415df40 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f749b6eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7494161c50 con 0x7f749406e3a0 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7498c86640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74940b3ce0 0x7f749413ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7498c86640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74940b3ce0 0x7f749413ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51950/0 (socket says 172.21.15.27:51950) 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7498c86640 1 -- 172.21.15.27:0/3544503308 learned_addr learned my addr 172.21.15.27:0/3544503308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7499487640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 0x7f7494143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7499c88640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749415df40 0x7f74941e7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7498c86640 1 -- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 msgr2=0x7f7494143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7498c86640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 0x7f7494143280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7498c86640 1 -- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749415df40 msgr2=0x7f74941e7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7498c86640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749415df40 0x7f74941e7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7498c86640 1 -- 172.21.15.27:0/3544503308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74941e81a0 con 0x7f74940b3ce0 2024-03-06T07:22:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7499c88640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749415df40 0x7f74941e7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7498c86640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74940b3ce0 0x7f749413ff90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f748400c4b0 tx=0x7f748400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.572+0000 7f7489ffb640 1 -- 172.21.15.27:0/3544503308 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484018070 con 0x7f74940b3ce0 2024-03-06T07:22:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.573+0000 7f7489ffb640 1 -- 172.21.15.27:0/3544503308 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7484007500 con 0x7f74940b3ce0 2024-03-06T07:22:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.573+0000 7f7489ffb640 1 -- 172.21.15.27:0/3544503308 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484004a20 con 0x7f74940b3ce0 2024-03-06T07:22:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.573+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7494140830 con 0x7f74940b3ce0 2024-03-06T07:22:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.573+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7494140df0 con 0x7f74940b3ce0 2024-03-06T07:22:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.574+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74940733d0 con 0x7f74940b3ce0 2024-03-06T07:22:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.574+0000 7f7489ffb640 1 -- 172.21.15.27:0/3544503308 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f748401b070 con 0x7f74940b3ce0 2024-03-06T07:22:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.574+0000 7f7489ffb640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747806b4d0 0x7f747806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.574+0000 7f7489ffb640 1 -- 172.21.15.27:0/3544503308 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74840894b0 con 0x7f74940b3ce0 2024-03-06T07:22:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.575+0000 7f7499487640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747806b4d0 0x7f747806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.575+0000 7f7499487640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747806b4d0 0x7f747806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7490036000 tx=0x7f749003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.579+0000 7f7489ffb640 1 -- 172.21.15.27:0/3544503308 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7484057da0 con 0x7f74940b3ce0 2024-03-06T07:22:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.759+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f74940b5710 con 0x7f74940b3ce0 2024-03-06T07:22:59.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.759+0000 7f7489ffb640 1 -- 172.21.15.27:0/3544503308 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f748405be40 con 0x7f74940b3ce0 2024-03-06T07:22:59.762 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747806b4d0 msgr2=0x7f747806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747806b4d0 0x7f747806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7490036000 tx=0x7f749003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74940b3ce0 msgr2=0x7f749413ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74940b3ce0 0x7f749413ff90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f748400c4b0 tx=0x7f748400c970 comp rx=0 tx=0).stop 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 shutdown_connections 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747806b4d0 0x7f747806d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749415df40 0x7f74941e7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74940b3ce0 0x7f749413ff90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 --2- 172.21.15.27:0/3544503308 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f749406e3a0 0x7f7494143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 >> 172.21.15.27:0/3544503308 conn(0x7f74940d09c0 msgr2=0x7f74940d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 shutdown_connections 2024-03-06T07:22:59.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.761+0000 7f749b6eb640 1 -- 172.21.15.27:0/3544503308 wait complete. 2024-03-06T07:22:59.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:59.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.879+0000 7f0b4cde2640 1 Processor -- start 2024-03-06T07:22:59.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.879+0000 7f0b4cde2640 1 -- start start 2024-03-06T07:22:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.879+0000 7f0b4cde2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b480d02e0 0x7f0b480cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.879+0000 7f0b4cde2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b480b4260 0x7f0b480b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.879+0000 7f0b4cde2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480d06c0 0x7f0b480b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.879+0000 7f0b4cde2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b480b0b80 con 0x7f0b480d02e0 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.879+0000 7f0b4cde2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b480aeec0 con 0x7f0b480b4260 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.879+0000 7f0b4cde2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480b0dd0 con 0x7f0b480d06c0 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.879+0000 7f0b4659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b480d02e0 0x7f0b480cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.880+0000 7f0b4659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b480d02e0 0x7f0b480cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51962/0 (socket says 172.21.15.27:51962) 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.880+0000 7f0b4659c640 1 -- 172.21.15.27:0/2924641187 learned_addr learned my addr 172.21.15.27:0/2924641187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.880+0000 7f0b45d9b640 1 --2- 172.21.15.27:0/2924641187 >> v2:172.21.15.69:3300/0 conn(0x7f0b480b4260 0x7f0b480b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.880+0000 7f0b46d9d640 1 --2- 172.21.15.27:0/2924641187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480d06c0 0x7f0b480b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.880+0000 7f0b4659c640 1 -- 172.21.15.27:0/2924641187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480d06c0 msgr2=0x7f0b480b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.880+0000 7f0b4659c640 1 --2- 172.21.15.27:0/2924641187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480d06c0 0x7f0b480b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.880+0000 7f0b4659c640 1 -- 172.21.15.27:0/2924641187 >> v2:172.21.15.69:3300/0 conn(0x7f0b480b4260 msgr2=0x7f0b480b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.880+0000 7f0b4659c640 1 --2- 172.21.15.27:0/2924641187 >> v2:172.21.15.69:3300/0 conn(0x7f0b480b4260 0x7f0b480b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.880+0000 7f0b4659c640 1 -- 172.21.15.27:0/2924641187 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b480b10b0 con 0x7f0b480d02e0 2024-03-06T07:22:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4659c640 1 --2- 172.21.15.27:0/2924641187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d02e0 0x7f0b480cfa00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0b48067cb0 tx=0x7f0b3802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1cd4d75d3fe379f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4559a640 1 -- 172.21.15.27:0/2924641187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38034070 con 0x7f0b480d02e0 2024-03-06T07:22:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4559a640 1 -- 172.21.15.27:0/2924641187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b38030860 con 0x7f0b480d02e0 2024-03-06T07:22:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4559a640 1 -- 172.21.15.27:0/2924641187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38030c30 con 0x7f0b480d02e0 2024-03-06T07:22:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4cde2640 1 -- 172.21.15.27:0/2924641187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d02e0 msgr2=0x7f0b480cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/2924641187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d02e0 0x7f0b480cfa00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0b48067cb0 tx=0x7f0b3802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4cde2640 1 -- 172.21.15.27:0/2924641187 shutdown_connections 2024-03-06T07:22:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/2924641187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480d06c0 0x7f0b480b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/2924641187 >> v2:172.21.15.69:3300/0 conn(0x7f0b480b4260 0x7f0b480b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/2924641187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d02e0 0x7f0b480cfa00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4cde2640 1 -- 172.21.15.27:0/2924641187 >> 172.21.15.27:0/2924641187 conn(0x7f0b480b36d0 msgr2=0x7f0b480b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4cde2640 1 -- 172.21.15.27:0/2924641187 shutdown_connections 2024-03-06T07:22:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.881+0000 7f0b4cde2640 1 -- 172.21.15.27:0/2924641187 wait complete. 2024-03-06T07:22:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.883+0000 7f0b4cde2640 1 Processor -- start 2024-03-06T07:22:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.883+0000 7f0b4cde2640 1 -- start start 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b4cde2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480b4260 0x7f0b481f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b4cde2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b480d02e0 0x7f0b481439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b4cde2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d06c0 0x7f0b480c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b4cde2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480b2100 con 0x7f0b480d06c0 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b4cde2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480b1f80 con 0x7f0b480d02e0 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b4cde2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480b2280 con 0x7f0b480b4260 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b46d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d06c0 0x7f0b480c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b46d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d06c0 0x7f0b480c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51976/0 (socket says 172.21.15.27:51976) 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b46d9d640 1 -- 172.21.15.27:0/243535215 learned_addr learned my addr 172.21.15.27:0/243535215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b4659c640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480b4260 0x7f0b481f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b45d9b640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b480d02e0 0x7f0b481439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b46d9d640 1 -- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480b4260 msgr2=0x7f0b481f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b46d9d640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480b4260 0x7f0b481f2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b46d9d640 1 -- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b480d02e0 msgr2=0x7f0b481439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b46d9d640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b480d02e0 0x7f0b481439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b46d9d640 1 -- 172.21.15.27:0/243535215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b48144090 con 0x7f0b480d06c0 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b46d9d640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d06c0 0x7f0b480c9830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000efc0 tx=0x7f0b4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b36ffd640 1 -- 172.21.15.27:0/243535215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b40018020 con 0x7f0b480d06c0 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b36ffd640 1 -- 172.21.15.27:0/243535215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b40014480 con 0x7f0b480d06c0 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b36ffd640 1 -- 172.21.15.27:0/243535215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b40014870 con 0x7f0b480d06c0 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b480c8d50 con 0x7f0b480d06c0 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.884+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b480c9310 con 0x7f0b480d06c0 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.886+0000 7f0b36ffd640 1 -- 172.21.15.27:0/243535215 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b40005d10 con 0x7f0b480d06c0 2024-03-06T07:22:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.886+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b48068700 con 0x7f0b480d06c0 2024-03-06T07:22:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.886+0000 7f0b36ffd640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b2806b4d0 0x7f0b2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.886+0000 7f0b36ffd640 1 -- 172.21.15.27:0/243535215 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b4008abe0 con 0x7f0b480d06c0 2024-03-06T07:22:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.886+0000 7f0b4659c640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b2806b4d0 0x7f0b2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.887+0000 7f0b4659c640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b2806b4d0 0x7f0b2806d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0b38036000 tx=0x7f0b38008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:22:59.896+0000 7f0b36ffd640 1 -- 172.21.15.27:0/243535215 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b40022070 con 0x7f0b480d06c0 2024-03-06T07:23:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.184+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b480c94e0 con 0x7f0b480d06c0 2024-03-06T07:23:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.184+0000 7f0b36ffd640 1 -- 172.21.15.27:0/243535215 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f0b40010040 con 0x7f0b480d06c0 2024-03-06T07:23:00.186 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:00.186 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1135544337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1135544337},{"type":"v1","addr":"172.21.15.27:6833","nonce":1135544337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1920994895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1920994895},{"type":"v1","addr":"172.21.15.69:6837","nonce":1920994895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:48.433333+0000","modified":"2024-03-06T07:22:51.436125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/3933261233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3933261233},{"type":"v1","addr":"172.21.15.27:6835","nonce":3933261233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-06T07:23:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-03-06T07:23:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b2806b4d0 msgr2=0x7f0b2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b2806b4d0 0x7f0b2806d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0b38036000 tx=0x7f0b38008030 comp rx=0 tx=0).stop 2024-03-06T07:23:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d06c0 msgr2=0x7f0b480c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d06c0 0x7f0b480c9830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000efc0 tx=0x7f0b4000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 shutdown_connections 2024-03-06T07:23:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b2806b4d0 0x7f0b2806d990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480d06c0 0x7f0b480c9830 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b480d02e0 0x7f0b481439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 --2- 172.21.15.27:0/243535215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b480b4260 0x7f0b481f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 >> 172.21.15.27:0/243535215 conn(0x7f0b480b36d0 msgr2=0x7f0b480cea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 shutdown_connections 2024-03-06T07:23:00.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.185+0000 7f0b4cde2640 1 -- 172.21.15.27:0/243535215 wait complete. 2024-03-06T07:23:00.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.309+0000 7f7e17355640 1 Processor -- start 2024-03-06T07:23:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.309+0000 7f7e17355640 1 -- start start 2024-03-06T07:23:00.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e17355640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 0x7f7e1015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e17355640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e10072660 0x7f7e1006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e17355640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e100b4160 con 0x7f7e100b55d0 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e17355640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e100b5b40 con 0x7f7e10072660 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e17355640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100b3ba0 con 0x7f7e1006e700 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e148f0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e10072660 0x7f7e1006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e148f0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e10072660 0x7f7e1006d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51980/0 (socket says 172.21.15.27:51980) 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e148f0640 1 -- 172.21.15.27:0/1618467699 learned_addr learned my addr 172.21.15.27:0/1618467699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e150f1640 1 --2- 172.21.15.27:0/1618467699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 0x7f7e1015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e148f0640 1 -- 172.21.15.27:0/1618467699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 msgr2=0x7f7e1015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e148f0640 1 --2- 172.21.15.27:0/1618467699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 0x7f7e1015ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e148f0640 1 -- 172.21.15.27:0/1618467699 >> v1:172.21.15.69:6789/0 conn(0x7f7e100b55d0 legacy=0x7f7e10071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e148f0640 1 -- 172.21.15.27:0/1618467699 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100b3e80 con 0x7f7e10072660 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.310+0000 7f7e150f1640 1 --2- 172.21.15.27:0/1618467699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 0x7f7e1015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.311+0000 7f7e148f0640 1 --2- 172.21.15.27:0/1618467699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e10072660 0x7f7e1006d9e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009870 tx=0x7f7e0002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f123e7be13fb0068 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.311+0000 7f7e07fff640 1 -- 172.21.15.27:0/1618467699 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00034070 con 0x7f7e10072660 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.311+0000 7f7e07fff640 1 -- 172.21.15.27:0/1618467699 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e00030840 con 0x7f7e10072660 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.311+0000 7f7e07fff640 1 -- 172.21.15.27:0/1618467699 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00030c10 con 0x7f7e10072660 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.312+0000 7f7e17355640 1 -- 172.21.15.27:0/1618467699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e10072660 msgr2=0x7f7e1006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.312+0000 7f7e17355640 1 --2- 172.21.15.27:0/1618467699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e10072660 0x7f7e1006d9e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009870 tx=0x7f7e0002b980 comp rx=0 tx=0).stop 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.312+0000 7f7e17355640 1 -- 172.21.15.27:0/1618467699 shutdown_connections 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.312+0000 7f7e17355640 1 --2- 172.21.15.27:0/1618467699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e10072660 0x7f7e1006d9e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.312+0000 7f7e17355640 1 --2- 172.21.15.27:0/1618467699 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 0x7f7e1015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.312+0000 7f7e17355640 1 -- 172.21.15.27:0/1618467699 >> 172.21.15.27:0/1618467699 conn(0x7f7e100b2720 msgr2=0x7f7e100b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.312+0000 7f7e17355640 1 -- 172.21.15.27:0/1618467699 shutdown_connections 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.312+0000 7f7e17355640 1 -- 172.21.15.27:0/1618467699 wait complete. 2024-03-06T07:23:00.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.313+0000 7f7e17355640 1 Processor -- start 2024-03-06T07:23:00.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.313+0000 7f7e17355640 1 -- start start 2024-03-06T07:23:00.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e17355640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 0x7f7e10140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e17355640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e10072660 0x7f7e1015b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e17355640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e100b55d0 0x7f7e100ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e17355640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1006ec60 con 0x7f7e100b55d0 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e17355640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1006eae0 con 0x7f7e10072660 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e17355640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10072090 con 0x7f7e1006e700 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e158f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e100b55d0 0x7f7e100ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e158f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e100b55d0 0x7f7e100ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51984/0 (socket says 172.21.15.27:51984) 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e158f2640 1 -- 172.21.15.27:0/735040429 learned_addr learned my addr 172.21.15.27:0/735040429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e150f1640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 0x7f7e10140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e148f0640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e10072660 0x7f7e1015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e158f2640 1 -- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 msgr2=0x7f7e10140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e158f2640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 0x7f7e10140dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e158f2640 1 -- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e10072660 msgr2=0x7f7e1015b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e158f2640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e10072660 0x7f7e1015b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e158f2640 1 -- 172.21.15.27:0/735040429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10141300 con 0x7f7e100b55d0 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e148f0640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e10072660 0x7f7e1015b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.314+0000 7f7e158f2640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e100b55d0 0x7f7e100ce610 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800c6b0 tx=0x7f7e0800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.315+0000 7f7e05ffb640 1 -- 172.21.15.27:0/735040429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08003b70 con 0x7f7e100b55d0 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.315+0000 7f7e05ffb640 1 -- 172.21.15.27:0/735040429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e08003d30 con 0x7f7e100b55d0 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.315+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e100cf500 con 0x7f7e100b55d0 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.315+0000 7f7e05ffb640 1 -- 172.21.15.27:0/735040429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08005310 con 0x7f7e100b55d0 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.315+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e100cfac0 con 0x7f7e100b55d0 2024-03-06T07:23:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.316+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e101e7d80 con 0x7f7e100b55d0 2024-03-06T07:23:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.316+0000 7f7e05ffb640 1 -- 172.21.15.27:0/735040429 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e080054b0 con 0x7f7e100b55d0 2024-03-06T07:23:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.317+0000 7f7e05ffb640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7df406b4d0 0x7f7df406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.317+0000 7f7e05ffb640 1 -- 172.21.15.27:0/735040429 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e08089fd0 con 0x7f7e100b55d0 2024-03-06T07:23:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.317+0000 7f7e150f1640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7df406b4d0 0x7f7df406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.318+0000 7f7e150f1640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7df406b4d0 0x7f7df406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00ab60 tx=0x7f7e0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.321+0000 7f7e05ffb640 1 -- 172.21.15.27:0/735040429 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e08058e50 con 0x7f7e100b55d0 2024-03-06T07:23:00.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.504+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e10067ff0 con 0x7f7e100b55d0 2024-03-06T07:23:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.506+0000 7f7e05ffb640 1 -- 172.21.15.27:0/735040429 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f7e08058750 con 0x7f7e100b55d0 2024-03-06T07:23:00.507 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:00.507 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7df406b4d0 msgr2=0x7f7df406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7df406b4d0 0x7f7df406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00ab60 tx=0x7f7e0c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e100b55d0 msgr2=0x7f7e100ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e100b55d0 0x7f7e100ce610 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800c6b0 tx=0x7f7e0800cb70 comp rx=0 tx=0).stop 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 shutdown_connections 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7df406b4d0 0x7f7df406d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e100b55d0 0x7f7e100ce610 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e10072660 0x7f7e1015b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 --2- 172.21.15.27:0/735040429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e1006e700 0x7f7e10140dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 >> 172.21.15.27:0/735040429 conn(0x7f7e100b2720 msgr2=0x7f7e10071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 shutdown_connections 2024-03-06T07:23:00.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.508+0000 7f7e17355640 1 -- 172.21.15.27:0/735040429 wait complete. 2024-03-06T07:23:00.521 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:00.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.629+0000 7fa6852bb640 1 Processor -- start 2024-03-06T07:23:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.629+0000 7fa6852bb640 1 -- start start 2024-03-06T07:23:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa6852bb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6800c6e40 0x7fa680166b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa6852bb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800cf120 con 0x7fa68015f500 2024-03-06T07:23:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa6852bb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800cd310 con 0x7fa6800cf740 2024-03-06T07:23:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa6852bb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68015f8e0 con 0x7fa6800c6e40 2024-03-06T07:23:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67effd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa68015f500 0x7fa6800ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50172/0 (socket says 172.21.15.27:50172) 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67effd640 1 -- 172.21.15.27:0/4019832368 learned_addr learned my addr 172.21.15.27:0/4019832368 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67f7fe640 1 --2- 172.21.15.27:0/4019832368 >> v2:172.21.15.69:3300/0 conn(0x7fa6800c6e40 0x7fa680166b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648886058 0 0) 0x7fa6800cf120 con 0x7fa68015f500 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa668003520 con 0x7fa68015f500 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816689008 0 0) 0x7fa668003520 con 0x7fa68015f500 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 >> v2:172.21.15.69:3300/0 conn(0x7fa6800c6e40 msgr2=0x7fa680166b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 --2- 172.21.15.27:0/4019832368 >> v2:172.21.15.69:3300/0 conn(0x7fa6800c6e40 0x7fa680166b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 >> v1:172.21.15.69:6789/0 conn(0x7fa6800cf740 legacy=0x7fa6800cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6800cd5b0 con 0x7fa68015f500 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fa674001670 con 0x7fa68015f500 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 >> v1:172.21.15.27:6789/0 conn(0x7fa68015f500 legacy=0x7fa6800ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 0x7fa668003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.630+0000 7fa67dffb640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa668004800 0x7fa668004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67dffb640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680051c0 0x7fa66800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68015f8e0 con 0x7fa6680051c0 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66800c160 con 0x7fa6680039b0 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66800c360 con 0x7fa668004800 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67effd640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 0x7fa668003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67f7fe640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680051c0 0x7fa66800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67e7fc640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa668004800 0x7fa668004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67effd640 1 -- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa668004800 msgr2=0x7fa668004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67effd640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa668004800 0x7fa668004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67effd640 1 -- 172.21.15.27:0/4019832368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680051c0 msgr2=0x7fa66800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67effd640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680051c0 0x7fa66800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67effd640 1 -- 172.21.15.27:0/4019832368 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa668003520 con 0x7fa6680039b0 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67f7fe640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680051c0 0x7fa66800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.631+0000 7fa67f7fe640 1 -- 172.21.15.27:0/4019832368 reap_dead start 2024-03-06T07:23:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa67effd640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 0x7fa668003dd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa67400c240 tx=0x7fa668005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee32ce585a4ab58d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa674007d10 con 0x7fa6680039b0 2024-03-06T07:23:00.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa674030070 con 0x7fa6680039b0 2024-03-06T07:23:00.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa67dffb640 1 -- 172.21.15.27:0/4019832368 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa674005d70 con 0x7fa6680039b0 2024-03-06T07:23:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa6852bb640 1 -- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 msgr2=0x7fa668003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa6852bb640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 0x7fa668003dd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa67400c240 tx=0x7fa668005830 comp rx=0 tx=0).stop 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa6852bb640 1 -- 172.21.15.27:0/4019832368 shutdown_connections 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa6852bb640 1 --2- 172.21.15.27:0/4019832368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 0x7fa668003dd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa6852bb640 1 -- 172.21.15.27:0/4019832368 >> 172.21.15.27:0/4019832368 conn(0x7fa6800b3700 msgr2=0x7fa6800b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.632+0000 7fa6852bb640 1 -- 172.21.15.27:0/4019832368 shutdown_connections 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.633+0000 7fa6852bb640 1 -- 172.21.15.27:0/4019832368 wait complete. 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.634+0000 7fa6852bb640 1 Processor -- start 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.634+0000 7fa6852bb640 1 -- start start 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.634+0000 7fa6852bb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 0x7fa6801ef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.634+0000 7fa6852bb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6801f2600 0x7fa6800c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa6852bb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6800ca1d0 0x7fa6800c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa6852bb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800c59a0 con 0x7fa6801f2600 2024-03-06T07:23:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa6852bb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800c56c0 con 0x7fa6680039b0 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa6852bb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800b43d0 con 0x7fa6800ca1d0 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6801f2600 0x7fa6800c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6801f2600 0x7fa6800c9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52002/0 (socket says 172.21.15.27:52002) 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67e7fc640 1 -- 172.21.15.27:0/3275770508 learned_addr learned my addr 172.21.15.27:0/3275770508 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67effd640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 0x7fa6801ef0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67e7fc640 1 -- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6800ca1d0 msgr2=0x7fa6800c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67e7fc640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6800ca1d0 0x7fa6800c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67e7fc640 1 -- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 msgr2=0x7fa6801ef0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67e7fc640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 0x7fa6801ef0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67e7fc640 1 -- 172.21.15.27:0/3275770508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6800b4b90 con 0x7fa6801f2600 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa67e7fc640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6801f2600 0x7fa6800c9c90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa67000f570 tx=0x7fa67000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa6677fe640 1 -- 172.21.15.27:0/3275770508 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700052e0 con 0x7fa6801f2600 2024-03-06T07:23:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa6677fe640 1 -- 172.21.15.27:0/3275770508 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa670018070 con 0x7fa6801f2600 2024-03-06T07:23:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa6677fe640 1 -- 172.21.15.27:0/3275770508 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa670003810 con 0x7fa6801f2600 2024-03-06T07:23:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6800b4e70 con 0x7fa6801f2600 2024-03-06T07:23:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.635+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6800b5430 con 0x7fa6801f2600 2024-03-06T07:23:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.636+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68009fcf0 con 0x7fa6801f2600 2024-03-06T07:23:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.637+0000 7fa6677fe640 1 -- 172.21.15.27:0/3275770508 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6700039b0 con 0x7fa6801f2600 2024-03-06T07:23:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.637+0000 7fa6677fe640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66c06b500 0x7fa66c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.637+0000 7fa6677fe640 1 -- 172.21.15.27:0/3275770508 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa670089f90 con 0x7fa6801f2600 2024-03-06T07:23:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.637+0000 7fa67effd640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66c06b500 0x7fa66c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.638+0000 7fa67effd640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66c06b500 0x7fa66c06d9c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa674034000 tx=0x7fa674039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.642+0000 7fa6677fe640 1 -- 172.21.15.27:0/3275770508 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa670058e10 con 0x7fa6801f2600 2024-03-06T07:23:00.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.821+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6800cbfa0 con 0x7fa6801f2600 2024-03-06T07:23:00.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.822+0000 7fa6677fe640 1 -- 172.21.15.27:0/3275770508 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa670058710 con 0x7fa6801f2600 2024-03-06T07:23:00.824 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66c06b500 msgr2=0x7fa66c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66c06b500 0x7fa66c06d9c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa674034000 tx=0x7fa674039040 comp rx=0 tx=0).stop 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6801f2600 msgr2=0x7fa6800c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6801f2600 0x7fa6800c9c90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa67000f570 tx=0x7fa67000fa30 comp rx=0 tx=0).stop 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 shutdown_connections 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66c06b500 0x7fa66c06d9c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6800ca1d0 0x7fa6800c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6801f2600 0x7fa6800c9c90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 --2- 172.21.15.27:0/3275770508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680039b0 0x7fa6801ef0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.823+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 >> 172.21.15.27:0/3275770508 conn(0x7fa6800b3700 msgr2=0x7fa68006ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.824+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 shutdown_connections 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.824+0000 7fa6852bb640 1 -- 172.21.15.27:0/3275770508 wait complete. 2024-03-06T07:23:00.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.968+0000 7f3e4c8fb640 1 Processor -- start 2024-03-06T07:23:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.968+0000 7f3e4c8fb640 1 -- start start 2024-03-06T07:23:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.969+0000 7f3e4c8fb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e480b2d30 0x7f3e480b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.969+0000 7f3e4c8fb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b7820 0x7f3e480b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.969+0000 7f3e4c8fb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e480d4960 con 0x7f3e48071540 2024-03-06T07:23:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.969+0000 7f3e4c8fb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e480d3230 con 0x7f3e480b2d30 2024-03-06T07:23:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.969+0000 7f3e4c8fb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480d36c0 con 0x7f3e480b7820 2024-03-06T07:23:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.970+0000 7f3e45d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e480b2d30 0x7f3e480b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.970+0000 7f3e4659c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3e48071540 0x7f3e48071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50186/0 (socket says 172.21.15.27:50186) 2024-03-06T07:23:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.970+0000 7f3e4659c640 1 -- 172.21.15.27:0/2300478688 learned_addr learned my addr 172.21.15.27:0/2300478688 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.970+0000 7f3e46d9d640 1 --2- 172.21.15.27:0/2300478688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b7820 0x7f3e480b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.970+0000 7f3e45d9b640 1 -- 172.21.15.27:0/2300478688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b7820 msgr2=0x7f3e480b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.970+0000 7f3e45d9b640 1 --2- 172.21.15.27:0/2300478688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b7820 0x7f3e480b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.970+0000 7f3e45d9b640 1 -- 172.21.15.27:0/2300478688 >> v1:172.21.15.27:6789/0 conn(0x7f3e48071540 legacy=0x7f3e48071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.970+0000 7f3e4559a640 1 -- 172.21.15.27:0/2300478688 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863144914 0 0) 0x7f3e480d4960 con 0x7f3e48071540 2024-03-06T07:23:00.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.970+0000 7f3e45d9b640 1 -- 172.21.15.27:0/2300478688 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e480d2620 con 0x7f3e480b2d30 2024-03-06T07:23:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e45d9b640 1 --2- 172.21.15.27:0/2300478688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e480b2d30 0x7f3e480b72e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3e380097a0 tx=0x7f3e3802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9ab5ae99b43354 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4559a640 1 -- 172.21.15.27:0/2300478688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38035020 con 0x7f3e480b2d30 2024-03-06T07:23:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4559a640 1 -- 172.21.15.27:0/2300478688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e38031830 con 0x7f3e480b2d30 2024-03-06T07:23:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4559a640 1 -- 172.21.15.27:0/2300478688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38031c00 con 0x7f3e480b2d30 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/2300478688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e480b2d30 msgr2=0x7f3e480b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4c8fb640 1 --2- 172.21.15.27:0/2300478688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e480b2d30 0x7f3e480b72e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3e380097a0 tx=0x7f3e3802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/2300478688 shutdown_connections 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4c8fb640 1 --2- 172.21.15.27:0/2300478688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b7820 0x7f3e480b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4c8fb640 1 --2- 172.21.15.27:0/2300478688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e480b2d30 0x7f3e480b72e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/2300478688 >> 172.21.15.27:0/2300478688 conn(0x7f3e480d3d60 msgr2=0x7f3e480d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.971+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/2300478688 shutdown_connections 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.972+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/2300478688 wait complete. 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.973+0000 7f3e4c8fb640 1 Processor -- start 2024-03-06T07:23:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.973+0000 7f3e4c8fb640 1 -- start start 2024-03-06T07:23:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4c8fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e48071540 0x7f3e4814a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4c8fb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b2d30 0x7f3e481e3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4c8fb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e480b7820 0x7f3e480c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4c8fb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480d2ec0 con 0x7f3e48071540 2024-03-06T07:23:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4c8fb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480d2be0 con 0x7f3e480b7820 2024-03-06T07:23:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4c8fb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480d22f0 con 0x7f3e480b2d30 2024-03-06T07:23:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e48071540 0x7f3e4814a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e48071540 0x7f3e4814a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52016/0 (socket says 172.21.15.27:52016) 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4659c640 1 -- 172.21.15.27:0/3587527790 learned_addr learned my addr 172.21.15.27:0/3587527790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e46d9d640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e480b7820 0x7f3e480c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e45d9b640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b2d30 0x7f3e481e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4659c640 1 -- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b2d30 msgr2=0x7f3e481e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4659c640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b2d30 0x7f3e481e3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4659c640 1 -- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e480b7820 msgr2=0x7f3e480c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4659c640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e480b7820 0x7f3e480c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4659c640 1 -- 172.21.15.27:0/3587527790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e480c7570 con 0x7f3e48071540 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e46d9d640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e480b7820 0x7f3e480c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4659c640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e48071540 0x7f3e4814a030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c430 tx=0x7f3e3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e36ffd640 1 -- 172.21.15.27:0/3587527790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0074e0 con 0x7f3e48071540 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e36ffd640 1 -- 172.21.15.27:0/3587527790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c0047e0 con 0x7f3e48071540 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e36ffd640 1 -- 172.21.15.27:0/3587527790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c004bd0 con 0x7f3e48071540 2024-03-06T07:23:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.974+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e480c7850 con 0x7f3e48071540 2024-03-06T07:23:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.975+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e480c7e10 con 0x7f3e48071540 2024-03-06T07:23:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.976+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e48072070 con 0x7f3e48071540 2024-03-06T07:23:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.976+0000 7f3e36ffd640 1 -- 172.21.15.27:0/3587527790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e3c011070 con 0x7f3e48071540 2024-03-06T07:23:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.976+0000 7f3e36ffd640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3e2806b4d0 0x7f3e2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.976+0000 7f3e36ffd640 1 -- 172.21.15.27:0/3587527790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e3c089e30 con 0x7f3e48071540 2024-03-06T07:23:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.977+0000 7f3e45d9b640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3e2806b4d0 0x7f3e2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.977+0000 7f3e45d9b640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3e2806b4d0 0x7f3e2806d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e38036000 tx=0x7f3e38008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:00.981+0000 7f3e36ffd640 1 -- 172.21.15.27:0/3587527790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e3c058d40 con 0x7f3e48071540 2024-03-06T07:23:01.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.250+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e48071b30 con 0x7f3e48071540 2024-03-06T07:23:01.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.251+0000 7f3e36ffd640 1 -- 172.21.15.27:0/3587527790 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3e3c058640 con 0x7f3e48071540 2024-03-06T07:23:01.252 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3e2806b4d0 msgr2=0x7f3e2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3e2806b4d0 0x7f3e2806d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e38036000 tx=0x7f3e38008030 comp rx=0 tx=0).stop 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e48071540 msgr2=0x7f3e4814a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e48071540 0x7f3e4814a030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c430 tx=0x7f3e3c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 shutdown_connections 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3e2806b4d0 0x7f3e2806d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e480b7820 0x7f3e480c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3e480b2d30 0x7f3e481e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 --2- 172.21.15.27:0/3587527790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e48071540 0x7f3e4814a030 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 >> 172.21.15.27:0/3587527790 conn(0x7f3e480d3d60 msgr2=0x7f3e4815da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 shutdown_connections 2024-03-06T07:23:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.253+0000 7f3e4c8fb640 1 -- 172.21.15.27:0/3587527790 wait complete. 2024-03-06T07:23:01.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.374+0000 7f1098563640 1 Processor -- start 2024-03-06T07:23:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.374+0000 7f1098563640 1 -- start start 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f1098563640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10900ceea0 0x7f10900d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f1098563640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10900d3740 con 0x7f109015dd90 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f1098563640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10900d3120 con 0x7f10900b6d10 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f1098563640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10900d3400 con 0x7f10900ceea0 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f10962ff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f109015dd90 0x7f10900d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50190/0 (socket says 172.21.15.27:50190) 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f10962ff640 1 -- 172.21.15.27:0/825715578 learned_addr learned my addr 172.21.15.27:0/825715578 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f1095afe640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10900ceea0 0x7f10900d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142461029 0 0) 0x7f10900d3740 con 0x7f109015dd90 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107c003520 con 0x7f109015dd90 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.375+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293889734 0 0) 0x7f10900d3120 con 0x7f10900b6d10 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10900d3740 con 0x7f10900b6d10 2024-03-06T07:23:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956211420 0 0) 0x7f107c003520 con 0x7f109015dd90 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10900ceea0 msgr2=0x7f10900d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10900ceea0 0x7f10900d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 >> v1:172.21.15.69:6789/0 conn(0x7f10900b6d10 legacy=0x7f10900d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10900d39e0 con 0x7f109015dd90 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f108c0019f0 con 0x7f109015dd90 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 >> v1:172.21.15.27:6789/0 conn(0x7f109015dd90 legacy=0x7f10900d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f107c003b70 0x7f107c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f107c004800 0x7f107c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f107c0051c0 0x7f107c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10900d3400 con 0x7f107c003b70 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c00bf60 con 0x7f107c004800 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c00c160 con 0x7f107c0051c0 2024-03-06T07:23:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f1095afe640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10900ceea0 0x7f10900d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10962ff640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f107c003b70 0x7f107c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f1096b00640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f107c0051c0 0x7f107c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f1095afe640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f107c004800 0x7f107c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10962ff640 1 -- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f107c0051c0 msgr2=0x7f107c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10962ff640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f107c0051c0 0x7f107c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10962ff640 1 -- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f107c004800 msgr2=0x7f107c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10962ff640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f107c004800 0x7f107c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f10962ff640 1 -- 172.21.15.27:0/825715578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c003520 con 0x7f107c003b70 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.376+0000 7f1095afe640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f107c004800 0x7f107c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.377+0000 7f10962ff640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f107c003b70 0x7f107c003f70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f108c00a750 tx=0x7f107c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef545551560d25b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.377+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c00a370 con 0x7f107c003b70 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.377+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108c034630 con 0x7f107c003b70 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.377+0000 7f10952fd640 1 -- 172.21.15.27:0/825715578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c034a20 con 0x7f107c003b70 2024-03-06T07:23:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.377+0000 7f1098563640 1 -- 172.21.15.27:0/825715578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f107c003b70 msgr2=0x7f107c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.377+0000 7f1098563640 1 --2- 172.21.15.27:0/825715578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f107c003b70 0x7f107c003f70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f108c00a750 tx=0x7f107c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.377+0000 7f1096b00640 1 -- 172.21.15.27:0/825715578 reap_dead start 2024-03-06T07:23:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.378+0000 7f1098563640 1 -- 172.21.15.27:0/825715578 shutdown_connections 2024-03-06T07:23:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.378+0000 7f1098563640 1 -- 172.21.15.27:0/825715578 >> 172.21.15.27:0/825715578 conn(0x7f1090071960 msgr2=0x7f1090071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.378+0000 7f1098563640 1 -- 172.21.15.27:0/825715578 shutdown_connections 2024-03-06T07:23:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.378+0000 7f1098563640 1 -- 172.21.15.27:0/825715578 wait complete. 2024-03-06T07:23:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.379+0000 7f1098563640 1 Processor -- start 2024-03-06T07:23:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.379+0000 7f1098563640 1 -- start start 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.379+0000 7f1098563640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f107c003b70 0x7f1090152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.379+0000 7f1098563640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10900c9160 0x7f10900d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.379+0000 7f1098563640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10900d1170 0x7f10901f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.379+0000 7f1098563640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10900d48f0 con 0x7f10900d1170 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.379+0000 7f1098563640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109015e170 con 0x7f10900c9160 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.379+0000 7f1098563640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10900d4a70 con 0x7f107c003b70 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1096b00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10900d1170 0x7f10901f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1096b00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10900d1170 0x7f10901f3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52032/0 (socket says 172.21.15.27:52032) 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1096b00640 1 -- 172.21.15.27:0/3292691369 learned_addr learned my addr 172.21.15.27:0/3292691369 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f10962ff640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f107c003b70 0x7f1090152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1095afe640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10900c9160 0x7f10900d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1096b00640 1 -- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f107c003b70 msgr2=0x7f1090152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1096b00640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f107c003b70 0x7f1090152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1096b00640 1 -- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10900c9160 msgr2=0x7f10900d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1096b00640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10900c9160 0x7f10900d0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1096b00640 1 -- 172.21.15.27:0/3292691369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10901f4bf0 con 0x7f10900d1170 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1096b00640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10900d1170 0x7f10901f3e70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f108800f770 tx=0x7f108800fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1086ffd640 1 -- 172.21.15.27:0/3292691369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10880050d0 con 0x7f10900d1170 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1086ffd640 1 -- 172.21.15.27:0/3292691369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1088018070 con 0x7f10900d1170 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10900c8680 con 0x7f10900d1170 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1086ffd640 1 -- 172.21.15.27:0/3292691369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108800d650 con 0x7f10900d1170 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.380+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10900c95e0 con 0x7f10900d1170 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.381+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1090072a30 con 0x7f10900d1170 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.382+0000 7f1086ffd640 1 -- 172.21.15.27:0/3292691369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f108800da00 con 0x7f10900d1170 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.382+0000 7f1086ffd640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f107406b4d0 0x7f107406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.382+0000 7f1086ffd640 1 -- 172.21.15.27:0/3292691369 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f108808a140 con 0x7f10900d1170 2024-03-06T07:23:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.383+0000 7f10962ff640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f107406b4d0 0x7f107406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.383+0000 7f10962ff640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f107406b4d0 0x7f107406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f108c034000 tx=0x7f108c02f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.387+0000 7f1086ffd640 1 -- 172.21.15.27:0/3292691369 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f108801a310 con 0x7f10900d1170 2024-03-06T07:23:01.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.565+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f109004f410 con 0x7f10900d1170 2024-03-06T07:23:01.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.566+0000 7f1086ffd640 1 -- 172.21.15.27:0/3292691369 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1088058fc0 con 0x7f10900d1170 2024-03-06T07:23:01.568 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:01.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f107406b4d0 msgr2=0x7f107406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f107406b4d0 0x7f107406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f108c034000 tx=0x7f108c02f000 comp rx=0 tx=0).stop 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10900d1170 msgr2=0x7f10901f3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10900d1170 0x7f10901f3e70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f108800f770 tx=0x7f108800fc30 comp rx=0 tx=0).stop 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 shutdown_connections 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f107406b4d0 0x7f107406d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10900d1170 0x7f10901f3e70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10900c9160 0x7f10900d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 --2- 172.21.15.27:0/3292691369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f107c003b70 0x7f1090152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 >> 172.21.15.27:0/3292691369 conn(0x7f1090071960 msgr2=0x7f10900b71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 shutdown_connections 2024-03-06T07:23:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.568+0000 7f1098563640 1 -- 172.21.15.27:0/3292691369 wait complete. 2024-03-06T07:23:01.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:01.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.687+0000 7f025f9f6640 1 Processor -- start 2024-03-06T07:23:01.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.688+0000 7f025f9f6640 1 -- start start 2024-03-06T07:23:01.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.688+0000 7f025f9f6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0258071540 0x7f02580b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.688+0000 7f025f9f6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02580d3400 0x7f0258071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.688+0000 7f025f9f6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0258071e60 con 0x7f02580b2b50 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.688+0000 7f025f9f6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02580d2f70 con 0x7f02580d3400 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.688+0000 7f025f9f6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02580d4be0 con 0x7f0258071540 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.688+0000 7f025cf91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02580d3400 0x7f0258071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.688+0000 7f025cf91640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02580d3400 0x7f0258071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52048/0 (socket says 172.21.15.27:52048) 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.688+0000 7f025df93640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f02580b2b50 0x7f02580d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50194/0 (socket says 172.21.15.27:50194) 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.689+0000 7f025cf91640 1 -- 172.21.15.27:0/3435489795 learned_addr learned my addr 172.21.15.27:0/3435489795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.689+0000 7f025d792640 1 --2- 172.21.15.27:0/3435489795 >> v2:172.21.15.69:3300/0 conn(0x7f0258071540 0x7f02580b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.689+0000 7f025cf91640 1 -- 172.21.15.27:0/3435489795 >> v2:172.21.15.69:3300/0 conn(0x7f0258071540 msgr2=0x7f02580b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.689+0000 7f025cf91640 1 --2- 172.21.15.27:0/3435489795 >> v2:172.21.15.69:3300/0 conn(0x7f0258071540 0x7f02580b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.689+0000 7f025cf91640 1 -- 172.21.15.27:0/3435489795 >> v1:172.21.15.27:6789/0 conn(0x7f02580b2b50 legacy=0x7f02580d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.689+0000 7f024ffff640 1 -- 172.21.15.27:0/3435489795 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594273769 0 0) 0x7f0258071e60 con 0x7f02580b2b50 2024-03-06T07:23:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.689+0000 7f025cf91640 1 -- 172.21.15.27:0/3435489795 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580ce610 con 0x7f02580d3400 2024-03-06T07:23:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.689+0000 7f025d792640 1 --2- 172.21.15.27:0/3435489795 >> v2:172.21.15.69:3300/0 conn(0x7f0258071540 0x7f02580b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:01.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f025cf91640 1 --2- 172.21.15.27:0/3435489795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580d3400 0x7f0258071920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02580d4da0 tx=0x7f025402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbeb32356cb15609 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f024ffff640 1 -- 172.21.15.27:0/3435489795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254034070 con 0x7f02580d3400 2024-03-06T07:23:01.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f024ffff640 1 -- 172.21.15.27:0/3435489795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0254030860 con 0x7f02580d3400 2024-03-06T07:23:01.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f024ffff640 1 -- 172.21.15.27:0/3435489795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254030c30 con 0x7f02580d3400 2024-03-06T07:23:01.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f025f9f6640 1 -- 172.21.15.27:0/3435489795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580d3400 msgr2=0x7f0258071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f025f9f6640 1 --2- 172.21.15.27:0/3435489795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580d3400 0x7f0258071920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02580d4da0 tx=0x7f025402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f025f9f6640 1 -- 172.21.15.27:0/3435489795 shutdown_connections 2024-03-06T07:23:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f025f9f6640 1 --2- 172.21.15.27:0/3435489795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580d3400 0x7f0258071920 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f025f9f6640 1 --2- 172.21.15.27:0/3435489795 >> v2:172.21.15.69:3300/0 conn(0x7f0258071540 0x7f02580b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f025f9f6640 1 -- 172.21.15.27:0/3435489795 >> 172.21.15.27:0/3435489795 conn(0x7f02580d3ca0 msgr2=0x7f02580d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f025f9f6640 1 -- 172.21.15.27:0/3435489795 shutdown_connections 2024-03-06T07:23:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.690+0000 7f025f9f6640 1 -- 172.21.15.27:0/3435489795 wait complete. 2024-03-06T07:23:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025f9f6640 1 Processor -- start 2024-03-06T07:23:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025f9f6640 1 -- start start 2024-03-06T07:23:01.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025f9f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0258071540 0x7f02580cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025f9f6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b2b50 0x7f02580b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025f9f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02580d3400 0x7f02580ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025f9f6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580ce8f0 con 0x7f02580b2b50 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025f9f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580d2c00 con 0x7f0258071540 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025f9f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025815e370 con 0x7f02580d3400 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025cf91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b2b50 0x7f02580b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025cf91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b2b50 0x7f02580b4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52054/0 (socket says 172.21.15.27:52054) 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.692+0000 7f025cf91640 1 -- 172.21.15.27:0/2013293859 learned_addr learned my addr 172.21.15.27:0/2013293859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025df93640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02580d3400 0x7f02580ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025d792640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0258071540 0x7f02580cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025cf91640 1 -- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02580d3400 msgr2=0x7f02580ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025cf91640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02580d3400 0x7f02580ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025cf91640 1 -- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0258071540 msgr2=0x7f02580cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025cf91640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0258071540 0x7f02580cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025cf91640 1 -- 172.21.15.27:0/2013293859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580cc570 con 0x7f02580b2b50 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025cf91640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b2b50 0x7f02580b4360 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0254000c00 tx=0x7f02540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f024dffb640 1 -- 172.21.15.27:0/2013293859 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254034040 con 0x7f02580b2b50 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f024dffb640 1 -- 172.21.15.27:0/2013293859 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025403ab90 con 0x7f02580b2b50 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02580cc850 con 0x7f02580b2b50 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f024dffb640 1 -- 172.21.15.27:0/2013293859 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254004030 con 0x7f02580b2b50 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.693+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02580cce10 con 0x7f02580b2b50 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.694+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02580cdbb0 con 0x7f02580b2b50 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.695+0000 7f024dffb640 1 -- 172.21.15.27:0/2013293859 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f025404b060 con 0x7f02580b2b50 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.695+0000 7f024dffb640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f023c06b4d0 0x7f023c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.695+0000 7f024dffb640 1 -- 172.21.15.27:0/2013293859 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02540ac0e0 con 0x7f02580b2b50 2024-03-06T07:23:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.695+0000 7f025d792640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f023c06b4d0 0x7f023c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.696+0000 7f025d792640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f023c06b4d0 0x7f023c06d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0248006fd0 tx=0x7f0248008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.699+0000 7f024dffb640 1 -- 172.21.15.27:0/2013293859 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f025403f070 con 0x7f02580b2b50 2024-03-06T07:23:01.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.919+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02580cf790 con 0x7f02580b2b50 2024-03-06T07:23:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.919+0000 7f024dffb640 1 -- 172.21.15.27:0/2013293859 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f025404c3b0 con 0x7f02580b2b50 2024-03-06T07:23:01.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:01.921 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1135544337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1135544337},{"type":"v1","addr":"172.21.15.27:6833","nonce":1135544337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1920994895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1920994895},{"type":"v1","addr":"172.21.15.69:6837","nonce":1920994895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:48.433333+0000","modified":"2024-03-06T07:22:51.436125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/3933261233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3933261233},{"type":"v1","addr":"172.21.15.27:6835","nonce":3933261233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-06T07:23:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-03-06T07:23:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.921+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f023c06b4d0 msgr2=0x7f023c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.921+0000 7f025f9f6640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f023c06b4d0 0x7f023c06d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0248006fd0 tx=0x7f0248008040 comp rx=0 tx=0).stop 2024-03-06T07:23:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.921+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b2b50 msgr2=0x7f02580b4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.921+0000 7f025f9f6640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b2b50 0x7f02580b4360 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0254000c00 tx=0x7f02540046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.922+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 shutdown_connections 2024-03-06T07:23:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.922+0000 7f025f9f6640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f023c06b4d0 0x7f023c06d990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.922+0000 7f025f9f6640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02580d3400 0x7f02580ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.922+0000 7f025f9f6640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b2b50 0x7f02580b4360 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.922+0000 7f025f9f6640 1 --2- 172.21.15.27:0/2013293859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0258071540 0x7f02580cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.922+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 >> 172.21.15.27:0/2013293859 conn(0x7f02580d3ca0 msgr2=0x7f02581606b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.922+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 shutdown_connections 2024-03-06T07:23:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:01.922+0000 7f025f9f6640 1 -- 172.21.15.27:0/2013293859 wait complete. 2024-03-06T07:23:01.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.039+0000 7fb03a247640 1 Processor -- start 2024-03-06T07:23:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.039+0000 7fb03a247640 1 -- start start 2024-03-06T07:23:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.040+0000 7fb03a247640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb03415df00 0x7fb0340d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.040+0000 7fb03a247640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0340cea30 con 0x7fb0340b9a30 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.040+0000 7fb03a247640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0340d2af0 con 0x7fb0340b95a0 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.040+0000 7fb03a247640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0340cf3c0 con 0x7fb03415df00 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.040+0000 7fb033fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0340b9a30 0x7fb0340d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50198/0 (socket says 172.21.15.27:50198) 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.040+0000 7fb033fff640 1 -- 172.21.15.27:0/2842847061 learned_addr learned my addr 172.21.15.27:0/2842847061 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.040+0000 7fb0337fe640 1 --2- 172.21.15.27:0/2842847061 >> v2:172.21.15.69:3300/0 conn(0x7fb03415df00 0x7fb0340d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.041+0000 7fb0327fc640 1 -- 172.21.15.27:0/2842847061 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538247156 0 0) 0x7fb0340cea30 con 0x7fb0340b9a30 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.041+0000 7fb0327fc640 1 -- 172.21.15.27:0/2842847061 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb020003520 con 0x7fb0340b9a30 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.041+0000 7fb0337fe640 1 -- 172.21.15.27:0/2842847061 >> v1:172.21.15.69:6789/0 conn(0x7fb0340b95a0 legacy=0x7fb0340b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.041+0000 7fb0337fe640 1 -- 172.21.15.27:0/2842847061 >> v1:172.21.15.27:6789/0 conn(0x7fb0340b9a30 legacy=0x7fb0340d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.041+0000 7fb0337fe640 1 -- 172.21.15.27:0/2842847061 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0340cf0e0 con 0x7fb03415df00 2024-03-06T07:23:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb0337fe640 1 --2- 172.21.15.27:0/2842847061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb03415df00 0x7fb0340d25b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0097a0 tx=0x7fb01c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fe3a21d73c3c074 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb0327fc640 1 -- 172.21.15.27:0/2842847061 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c034070 con 0x7fb03415df00 2024-03-06T07:23:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb0327fc640 1 -- 172.21.15.27:0/2842847061 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c030610 con 0x7fb03415df00 2024-03-06T07:23:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb0327fc640 1 -- 172.21.15.27:0/2842847061 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c0309e0 con 0x7fb03415df00 2024-03-06T07:23:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb03a247640 1 -- 172.21.15.27:0/2842847061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb03415df00 msgr2=0x7fb0340d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb03a247640 1 --2- 172.21.15.27:0/2842847061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb03415df00 0x7fb0340d25b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0097a0 tx=0x7fb01c02b960 comp rx=0 tx=0).stop 2024-03-06T07:23:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb03a247640 1 -- 172.21.15.27:0/2842847061 shutdown_connections 2024-03-06T07:23:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb03a247640 1 --2- 172.21.15.27:0/2842847061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb03415df00 0x7fb0340d25b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb03a247640 1 -- 172.21.15.27:0/2842847061 >> 172.21.15.27:0/2842847061 conn(0x7fb034071960 msgr2=0x7fb034071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb03a247640 1 -- 172.21.15.27:0/2842847061 shutdown_connections 2024-03-06T07:23:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.042+0000 7fb03a247640 1 -- 172.21.15.27:0/2842847061 wait complete. 2024-03-06T07:23:02.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.044+0000 7fb03a247640 1 Processor -- start 2024-03-06T07:23:02.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.044+0000 7fb03a247640 1 -- start start 2024-03-06T07:23:02.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.044+0000 7fb03a247640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0340b95a0 0x7fb0340cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.044+0000 7fb03a247640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb03415df00 0x7fb0340cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.044+0000 7fb03a247640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341f3ad0 0x7fb0340b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.044+0000 7fb03a247640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0340cec70 con 0x7fb0341f3ad0 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.044+0000 7fb03a247640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0340ce750 con 0x7fb0340b95a0 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.044+0000 7fb03a247640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0340cedf0 con 0x7fb03415df00 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb033fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341f3ad0 0x7fb0340b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb033fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341f3ad0 0x7fb0340b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52060/0 (socket says 172.21.15.27:52060) 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb033fff640 1 -- 172.21.15.27:0/3077914663 learned_addr learned my addr 172.21.15.27:0/3077914663 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb0337fe640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0340b95a0 0x7fb0340cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb032ffd640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb03415df00 0x7fb0340cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb033fff640 1 -- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb03415df00 msgr2=0x7fb0340cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb033fff640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb03415df00 0x7fb0340cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb033fff640 1 -- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0340b95a0 msgr2=0x7fb0340cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb033fff640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0340b95a0 0x7fb0340cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb033fff640 1 -- 172.21.15.27:0/3077914663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0340cbc30 con 0x7fb0341f3ad0 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb032ffd640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb03415df00 0x7fb0340cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb033fff640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341f3ad0 0x7fb0340b4cd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00cb70 tx=0x7fb02c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb02bfff640 1 -- 172.21.15.27:0/3077914663 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c017070 con 0x7fb0341f3ad0 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb02bfff640 1 -- 172.21.15.27:0/3077914663 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02c00ad40 con 0x7fb0341f3ad0 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0341f44b0 con 0x7fb0341f3ad0 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb02bfff640 1 -- 172.21.15.27:0/3077914663 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c012620 con 0x7fb0341f3ad0 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.045+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0340caef0 con 0x7fb0341f3ad0 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.046+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb034067ff0 con 0x7fb0341f3ad0 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.047+0000 7fb02bfff640 1 -- 172.21.15.27:0/3077914663 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb02c01a070 con 0x7fb0341f3ad0 2024-03-06T07:23:02.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.047+0000 7fb02bfff640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.047+0000 7fb02bfff640 1 -- 172.21.15.27:0/3077914663 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb02c08a0a0 con 0x7fb0341f3ad0 2024-03-06T07:23:02.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.048+0000 7fb0337fe640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.048+0000 7fb0337fe640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb01c036000 tx=0x7fb01c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.052+0000 7fb02bfff640 1 -- 172.21.15.27:0/3077914663 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb02c058f20 con 0x7fb0341f3ad0 2024-03-06T07:23:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.235+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0340d2fb0 con 0x7fb0341f3ad0 2024-03-06T07:23:02.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.236+0000 7fb02bfff640 1 -- 172.21.15.27:0/3077914663 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7fb02c058820 con 0x7fb0341f3ad0 2024-03-06T07:23:02.238 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:02.239 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 msgr2=0x7fb01406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb01c036000 tx=0x7fb01c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341f3ad0 msgr2=0x7fb0340b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341f3ad0 0x7fb0340b4cd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00cb70 tx=0x7fb02c00a430 comp rx=0 tx=0).stop 2024-03-06T07:23:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 shutdown_connections 2024-03-06T07:23:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341f3ad0 0x7fb0340b4cd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb03415df00 0x7fb0340cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 --2- 172.21.15.27:0/3077914663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0340b95a0 0x7fb0340cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 >> 172.21.15.27:0/3077914663 conn(0x7fb034071960 msgr2=0x7fb034162250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 shutdown_connections 2024-03-06T07:23:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.239+0000 7fb03a247640 1 -- 172.21.15.27:0/3077914663 wait complete. 2024-03-06T07:23:02.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:02.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.363+0000 7ffb98a57640 1 Processor -- start 2024-03-06T07:23:02.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.363+0000 7ffb98a57640 1 -- start start 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.363+0000 7ffb98a57640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb9406b5c0 0x7ffb940b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb98a57640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940b3340 con 0x7ffb940ae9b0 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb98a57640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940cee90 con 0x7ffb940b0e90 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb98a57640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb940cfa00 con 0x7ffb9406b5c0 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb9259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb9406b5c0 0x7ffb940b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb92d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffb940ae9b0 0x7ffb940d03a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50200/0 (socket says 172.21.15.27:50200) 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb92d9d640 1 -- 172.21.15.27:0/4294171666 learned_addr learned my addr 172.21.15.27:0/4294171666 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb9259c640 1 -- 172.21.15.27:0/4294171666 >> v1:172.21.15.69:6789/0 conn(0x7ffb940b0e90 legacy=0x7ffb9406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb9259c640 1 -- 172.21.15.27:0/4294171666 >> v1:172.21.15.27:6789/0 conn(0x7ffb940ae9b0 legacy=0x7ffb940d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb9159a640 1 -- 172.21.15.27:0/4294171666 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702158581 0 0) 0x7ffb940b3340 con 0x7ffb940ae9b0 2024-03-06T07:23:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.364+0000 7ffb9259c640 1 -- 172.21.15.27:0/4294171666 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940cfc50 con 0x7ffb9406b5c0 2024-03-06T07:23:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.365+0000 7ffb9259c640 1 --2- 172.21.15.27:0/4294171666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb9406b5c0 0x7ffb940b2e00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffb840097a0 tx=0x7ffb8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73398ecc42b7ab0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.365+0000 7ffb9159a640 1 -- 172.21.15.27:0/4294171666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb840315a0 con 0x7ffb9406b5c0 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.365+0000 7ffb9159a640 1 -- 172.21.15.27:0/4294171666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb84037070 con 0x7ffb9406b5c0 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.365+0000 7ffb9159a640 1 -- 172.21.15.27:0/4294171666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84031960 con 0x7ffb9406b5c0 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.365+0000 7ffb98a57640 1 -- 172.21.15.27:0/4294171666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb9406b5c0 msgr2=0x7ffb940b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.365+0000 7ffb98a57640 1 --2- 172.21.15.27:0/4294171666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb9406b5c0 0x7ffb940b2e00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffb840097a0 tx=0x7ffb8402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.366+0000 7ffb98a57640 1 -- 172.21.15.27:0/4294171666 shutdown_connections 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.366+0000 7ffb98a57640 1 --2- 172.21.15.27:0/4294171666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb9406b5c0 0x7ffb940b2e00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.366+0000 7ffb98a57640 1 -- 172.21.15.27:0/4294171666 >> 172.21.15.27:0/4294171666 conn(0x7ffb940b36d0 msgr2=0x7ffb940b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.366+0000 7ffb98a57640 1 -- 172.21.15.27:0/4294171666 shutdown_connections 2024-03-06T07:23:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.366+0000 7ffb98a57640 1 -- 172.21.15.27:0/4294171666 wait complete. 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.367+0000 7ffb98a57640 1 Processor -- start 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.367+0000 7ffb98a57640 1 -- start start 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb98a57640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb9406b5c0 0x7ffb940c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb98a57640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb940ae9b0 0x7ffb940cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb98a57640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb940b0e90 0x7ffb940af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb98a57640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9406c910 con 0x7ffb940ae9b0 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb98a57640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9406c790 con 0x7ffb940b0e90 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb98a57640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940d0ab0 con 0x7ffb9406b5c0 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb940ae9b0 0x7ffb940cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb91d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb940ae9b0 0x7ffb940cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52088/0 (socket says 172.21.15.27:52088) 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb91d9b640 1 -- 172.21.15.27:0/4159791941 learned_addr learned my addr 172.21.15.27:0/4159791941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb92d9d640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb940b0e90 0x7ffb940af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb9259c640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb9406b5c0 0x7ffb940c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb91d9b640 1 -- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb9406b5c0 msgr2=0x7ffb940c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb91d9b640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb9406b5c0 0x7ffb940c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb91d9b640 1 -- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb940b0e90 msgr2=0x7ffb940af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb91d9b640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb940b0e90 0x7ffb940af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb91d9b640 1 -- 172.21.15.27:0/4159791941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9415c8c0 con 0x7ffb940ae9b0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb9259c640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb9406b5c0 0x7ffb940c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb92d9d640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb940b0e90 0x7ffb940af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.368+0000 7ffb91d9b640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb940ae9b0 0x7ffb940cb4b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c430 tx=0x7ffb8800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.369+0000 7ffb82ffd640 1 -- 172.21.15.27:0/4159791941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88011790 con 0x7ffb940ae9b0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.369+0000 7ffb82ffd640 1 -- 172.21.15.27:0/4159791941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb88017070 con 0x7ffb940ae9b0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.369+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb940c5cc0 con 0x7ffb940ae9b0 2024-03-06T07:23:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.369+0000 7ffb82ffd640 1 -- 172.21.15.27:0/4159791941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88011bb0 con 0x7ffb940ae9b0 2024-03-06T07:23:02.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.369+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb940c6280 con 0x7ffb940ae9b0 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.370+0000 7ffb82ffd640 1 -- 172.21.15.27:0/4159791941 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb8801c460 con 0x7ffb940ae9b0 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.370+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb94068700 con 0x7ffb940ae9b0 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.370+0000 7ffb82ffd640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffb7406b4d0 0x7ffb7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.371+0000 7ffb82ffd640 1 -- 172.21.15.27:0/4159791941 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb8808a070 con 0x7ffb940ae9b0 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.371+0000 7ffb9259c640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffb7406b4d0 0x7ffb7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.371+0000 7ffb9259c640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffb7406b4d0 0x7ffb7406d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb84036000 tx=0x7ffb84008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.375+0000 7ffb82ffd640 1 -- 172.21.15.27:0/4159791941 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb88058ef0 con 0x7ffb940ae9b0 2024-03-06T07:23:02.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.662+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb940c69f0 con 0x7ffb940ae9b0 2024-03-06T07:23:02.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.662+0000 7ffb82ffd640 1 -- 172.21.15.27:0/4159791941 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7ffb880587f0 con 0x7ffb940ae9b0 2024-03-06T07:23:02.664 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:02.664 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1135544337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1135544337},{"type":"v1","addr":"172.21.15.27:6833","nonce":1135544337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1920994895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1920994895},{"type":"v1","addr":"172.21.15.69:6837","nonce":1920994895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:48.433333+0000","modified":"2024-03-06T07:22:51.436125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/3933261233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3933261233},{"type":"v1","addr":"172.21.15.27:6835","nonce":3933261233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-06T07:23:02.665 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-03-06T07:23:02.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffb7406b4d0 msgr2=0x7ffb7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffb7406b4d0 0x7ffb7406d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb84036000 tx=0x7ffb84008030 comp rx=0 tx=0).stop 2024-03-06T07:23:02.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb940ae9b0 msgr2=0x7ffb940cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb940ae9b0 0x7ffb940cb4b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c430 tx=0x7ffb8800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 shutdown_connections 2024-03-06T07:23:02.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffb7406b4d0 0x7ffb7406d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb940b0e90 0x7ffb940af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb940ae9b0 0x7ffb940cb4b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 --2- 172.21.15.27:0/4159791941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffb9406b5c0 0x7ffb940c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 >> 172.21.15.27:0/4159791941 conn(0x7ffb940b36d0 msgr2=0x7ffb9415e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.664+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 shutdown_connections 2024-03-06T07:23:02.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:02.665+0000 7ffb98a57640 1 -- 172.21.15.27:0/4159791941 wait complete. 2024-03-06T07:23:03.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:48.433333+0000', 'modified': '2024-03-06T07:22:51.436125+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4290': {'gid': 4290, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/3933261233', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6834', 'nonce': 3933261233}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 3933261233}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4290, 'qdb_cluster': [4290]} 2024-03-06T07:23:03.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:03.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:23:03.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.229+0000 7f0605916640 1 Processor -- start 2024-03-06T07:23:03.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.229+0000 7f0605916640 1 -- start start 2024-03-06T07:23:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.230+0000 7f0605916640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f060015dd70 0x7f06000b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.230+0000 7f0605916640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 0x7f06000b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.230+0000 7f0605916640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06000b8100 con 0x7f06000b5700 2024-03-06T07:23:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.230+0000 7f0605916640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06000b60b0 con 0x7f060015dd70 2024-03-06T07:23:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.230+0000 7f0605916640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06000b8fd0 con 0x7f06000b7c70 2024-03-06T07:23:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.230+0000 7f05feffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f060015dd70 0x7f06000b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.230+0000 7f05feffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f060015dd70 0x7f06000b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52092/0 (socket says 172.21.15.27:52092) 2024-03-06T07:23:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.230+0000 7f05feffd640 1 -- 172.21.15.27:0/3616580109 learned_addr learned my addr 172.21.15.27:0/3616580109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.231+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/3616580109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 0x7f06000b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.231+0000 7f05feffd640 1 -- 172.21.15.27:0/3616580109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 msgr2=0x7f06000b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.231+0000 7f05feffd640 1 --2- 172.21.15.27:0/3616580109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 0x7f06000b5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.231+0000 7f05feffd640 1 -- 172.21.15.27:0/3616580109 >> v1:172.21.15.69:6789/0 conn(0x7f06000b5700 legacy=0x7f06000d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.231+0000 7f05feffd640 1 -- 172.21.15.27:0/3616580109 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000b83c0 con 0x7f060015dd70 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.231+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/3616580109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 0x7f06000b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.231+0000 7f05feffd640 1 --2- 172.21.15.27:0/3616580109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 0x7f06000b9650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009870 tx=0x7f05f002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9b74419d6f7927 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f05fe7fc640 1 -- 172.21.15.27:0/3616580109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f002f830 con 0x7f060015dd70 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f05fe7fc640 1 -- 172.21.15.27:0/3616580109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f002f9d0 con 0x7f060015dd70 2024-03-06T07:23:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f05fe7fc640 1 -- 172.21.15.27:0/3616580109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f002fda0 con 0x7f060015dd70 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f0605916640 1 -- 172.21.15.27:0/3616580109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 msgr2=0x7f06000b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f0605916640 1 --2- 172.21.15.27:0/3616580109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 0x7f06000b9650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009870 tx=0x7f05f002ba10 comp rx=0 tx=0).stop 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f0605916640 1 -- 172.21.15.27:0/3616580109 shutdown_connections 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f0605916640 1 --2- 172.21.15.27:0/3616580109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 0x7f06000b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f0605916640 1 --2- 172.21.15.27:0/3616580109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 0x7f06000b9650 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f0605916640 1 -- 172.21.15.27:0/3616580109 >> 172.21.15.27:0/3616580109 conn(0x7f0600071960 msgr2=0x7f0600071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f0605916640 1 -- 172.21.15.27:0/3616580109 shutdown_connections 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.232+0000 7f0605916640 1 -- 172.21.15.27:0/3616580109 wait complete. 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.233+0000 7f0605916640 1 Processor -- start 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.233+0000 7f0605916640 1 -- start start 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f0605916640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06000b5700 0x7f060015a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f0605916640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 0x7f06000cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f0605916640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 0x7f06000ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f0605916640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06000b8c60 con 0x7f060015dd70 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f0605916640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06000b8980 con 0x7f06000b5700 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f0605916640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06000b9b90 con 0x7f06000b7c70 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 0x7f06000ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 0x7f06000ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52098/0 (socket says 172.21.15.27:52098) 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f05ff7fe640 1 -- 172.21.15.27:0/1989898896 learned_addr learned my addr 172.21.15.27:0/1989898896 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:03.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f05feffd640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06000b5700 0x7f060015a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.234+0000 7f05f7fff640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 0x7f06000cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f05ff7fe640 1 -- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 msgr2=0x7f06000cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 0x7f06000cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f05ff7fe640 1 -- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06000b5700 msgr2=0x7f060015a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06000b5700 0x7f060015a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f05ff7fe640 1 -- 172.21.15.27:0/1989898896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060015a7d0 con 0x7f060015dd70 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 0x7f06000ce2b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f05f800cef0 tx=0x7f05f800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f05fcff9640 1 -- 172.21.15.27:0/1989898896 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f801a070 con 0x7f060015dd70 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f05fcff9640 1 -- 172.21.15.27:0/1989898896 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f800d860 con 0x7f060015dd70 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f0605916640 1 -- 172.21.15.27:0/1989898896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06001f40c0 con 0x7f060015dd70 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f05fcff9640 1 -- 172.21.15.27:0/1989898896 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f800dc50 con 0x7f060015dd70 2024-03-06T07:23:03.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.235+0000 7f0605916640 1 -- 172.21.15.27:0/1989898896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06001f4680 con 0x7f060015dd70 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.236+0000 7f05fcff9640 1 -- 172.21.15.27:0/1989898896 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05f801d070 con 0x7f060015dd70 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.237+0000 7f05fcff9640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05e006b4d0 0x7f05e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.237+0000 7f05fcff9640 1 -- 172.21.15.27:0/1989898896 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05f808aa20 con 0x7f060015dd70 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.237+0000 7f05feffd640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05e006b4d0 0x7f05e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.238+0000 7f05feffd640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05e006b4d0 0x7f05e006d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f05f0036000 tx=0x7f05f0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.238+0000 7f06085cf740 1 Processor -- start 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.238+0000 7f06085cf740 1 -- start start 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f06085cf740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1b4870 0x5649ed1b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f06085cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1b88c0 0x5649ed1ba360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f06085cf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed1e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f06085cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ed11e6b0 con 0x5649ed1b88c0 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f06085cf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ed0c7a60 con 0x5649ed1e4600 2024-03-06T07:23:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f06085cf740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649ed11a750 con 0x5649ed1b4870 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1b88c0 0x5649ed1ba360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1b88c0 0x5649ed1ba360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52100/0 (socket says 172.21.15.27:52100) 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05f7fff640 1 -- 172.21.15.27:0/2657655584 learned_addr learned my addr 172.21.15.27:0/2657655584 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05feffd640 1 --2- 172.21.15.27:0/2657655584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1b4870 0x5649ed1b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/2657655584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed1e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05f7fff640 1 -- 172.21.15.27:0/2657655584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1b4870 msgr2=0x5649ed1b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05f7fff640 1 --2- 172.21.15.27:0/2657655584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1b4870 0x5649ed1b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05f7fff640 1 -- 172.21.15.27:0/2657655584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 msgr2=0x5649ed1e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05f7fff640 1 --2- 172.21.15.27:0/2657655584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed1e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05f7fff640 1 -- 172.21.15.27:0/2657655584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649ed0e5f30 con 0x5649ed1b88c0 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.239+0000 7f05f7fff640 1 --2- 172.21.15.27:0/2657655584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1b88c0 0x5649ed1ba360 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f05e8001820 tx=0x7f05e8007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edac4969dd7e304 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f05f57fa640 1 -- 172.21.15.27:0/2657655584 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e80052e0 con 0x5649ed1b88c0 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f05f57fa640 1 -- 172.21.15.27:0/2657655584 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e8018070 con 0x5649ed1b88c0 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f05f57fa640 1 -- 172.21.15.27:0/2657655584 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e800f9b0 con 0x5649ed1b88c0 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f06085cf740 1 -- 172.21.15.27:0/2657655584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1b88c0 msgr2=0x5649ed1ba360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f06085cf740 1 --2- 172.21.15.27:0/2657655584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1b88c0 0x5649ed1ba360 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f05e8001820 tx=0x7f05e8007e50 comp rx=0 tx=0).stop 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f06085cf740 1 -- 172.21.15.27:0/2657655584 shutdown_connections 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f06085cf740 1 --2- 172.21.15.27:0/2657655584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed1e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f06085cf740 1 --2- 172.21.15.27:0/2657655584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1b88c0 0x5649ed1ba360 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f06085cf740 1 --2- 172.21.15.27:0/2657655584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1b4870 0x5649ed1b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f06085cf740 1 -- 172.21.15.27:0/2657655584 >> 172.21.15.27:0/2657655584 conn(0x5649ed1b68f0 msgr2=0x5649ed1b81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f06085cf740 1 -- 172.21.15.27:0/2657655584 shutdown_connections 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.240+0000 7f06085cf740 1 -- 172.21.15.27:0/2657655584 wait complete. 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.241+0000 7f06085cf740 1 -- start start 2024-03-06T07:23:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.241+0000 7f06085cf740 1 Processor -- start 2024-03-06T07:23:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f06085cf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed2101c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f06085cf740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1d9760 0x5649ed1e0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f06085cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e12a0 0x5649ed1e7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f06085cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ed1b8d50 con 0x5649ed1e12a0 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f06085cf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ed0daea0 con 0x5649ed1e4600 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f06085cf740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649ed1b9030 con 0x5649ed1d9760 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f06085cf740 10 client.0 _ll_register_callbacks cb 0x5649ed0da870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e12a0 0x5649ed1e7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e12a0 0x5649ed1e7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52108/0 (socket says 172.21.15.27:52108) 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05ff7fe640 1 -- 172.21.15.27:0/543903246 learned_addr learned my addr 172.21.15.27:0/543903246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05f7fff640 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1d9760 0x5649ed1e0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05feffd640 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed2101c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05ff7fe640 1 -- 172.21.15.27:0/543903246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1d9760 msgr2=0x5649ed1e0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1d9760 0x5649ed1e0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05ff7fe640 1 -- 172.21.15.27:0/543903246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 msgr2=0x5649ed2101c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed2101c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05ff7fe640 1 -- 172.21.15.27:0/543903246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649ed0db270 con 0x5649ed1e12a0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e12a0 0x5649ed1e7da0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f05f8061440 tx=0x7f05f805d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4caf77e2f51325e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.242+0000 7f05de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f05de7fc640 1 -- 172.21.15.27:0/543903246 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f80616e0 con 0x5649ed1e12a0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f05de7fc640 1 -- 172.21.15.27:0/543903246 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05f8058760 con 0x5649ed1e12a0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f06085cf740 1 -- 172.21.15.27:0/543903246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05e8018070 con 0x5649ed1e12a0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f05de7fc640 1 -- 172.21.15.27:0/543903246 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05f8077050 con 0x5649ed1e12a0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f06085cf740 10 client.4665 fetch_fsmap learned FSMap version 7 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f06085cf740 1 -- 172.21.15.27:0/543903246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5649ece332d0 con 0x5649ed1e12a0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f05de7fc640 1 -- 172.21.15.27:0/543903246 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e8018070 con 0x5649ed1e12a0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f05de7fc640 1 -- 172.21.15.27:0/543903246 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 7) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f05f805f340 con 0x5649ed1e12a0 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f06085cf740 10 client.4665 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:23:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f06085cf740 10 client.4665 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f06085cf740 20 client.4665 populate_metadata read hostname 'smithi027' 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f06085cf740 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] conn(0x5649ed1e6150 0x5649ed21ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f06085cf740 4 client.4665 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f06085cf740 1 -- 172.21.15.27:0/543903246 --> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649ed12bbf0 con 0x5649ed1e6150 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.243+0000 7f05feffd640 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] conn(0x5649ed1e6150 0x5649ed21ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f05feffd640 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] conn(0x5649ed1e6150 0x5649ed21ead0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=184876ca19bc0ef3 server_cookie=5fef0c3b0e5ead0 in_seq=0 out_seq=0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f05de7fc640 10 client.4665 ms_handle_connect on v2:172.21.15.27:6834/3933261233 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f05de7fc640 1 -- 172.21.15.27:0/543903246 <== mds.0 v2:172.21.15.27:6834/3933261233 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5649ed12bbf0 con 0x5649ed1e6150 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f05de7fc640 10 client.4665 handle_command_reply: tid=0 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f06085cf740 1 client.4665 shutdown 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f06085cf740 1 -- 172.21.15.27:0/543903246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e12a0 msgr2=0x5649ed1e7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f06085cf740 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e12a0 0x5649ed1e7da0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f05f8061440 tx=0x7f05f805d530 comp rx=0 tx=0).stop 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f06085cf740 1 -- 172.21.15.27:0/543903246 shutdown_connections 2024-03-06T07:23:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f06085cf740 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] conn(0x5649ed1e6150 0x5649ed21ead0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f06085cf740 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e12a0 0x5649ed1e7da0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f06085cf740 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1d9760 0x5649ed1e0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f05de7fc640 0 client.4665 ms_handle_reset on v2:172.21.15.27:6834/3933261233 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f06085cf740 1 --2- 172.21.15.27:0/543903246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed2101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.244+0000 7f06085cf740 1 -- 172.21.15.27:0/543903246 >> 172.21.15.27:0/543903246 conn(0x5649ed1b4870 msgr2=0x5649ed1b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.245+0000 7f06085cf740 1 -- 172.21.15.27:0/543903246 shutdown_connections 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.245+0000 7f06085cf740 1 -- 172.21.15.27:0/543903246 wait complete. 2024-03-06T07:23:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.245+0000 7f06085cf740 20 client.4665 trim_cache size 0 max 16384 2024-03-06T07:23:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.280+0000 7f06085cf740 1 Processor -- start 2024-03-06T07:23:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.280+0000 7f06085cf740 1 -- start start 2024-03-06T07:23:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f06085cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1d9760 0x5649ed1e3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f06085cf740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 0x5649ed212d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f06085cf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed1d83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f06085cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06001f6420 con 0x5649ed1d9760 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f06085cf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06001f7ec0 con 0x5649ed1e4600 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f06085cf740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06001f5e20 con 0x5649ed1e12a0 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1d9760 0x5649ed1e3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1d9760 0x5649ed1e3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52110/0 (socket says 172.21.15.27:52110) 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05feffd640 1 -- 172.21.15.27:0/2279684497 learned_addr learned my addr 172.21.15.27:0/2279684497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed1d83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05f7fff640 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 0x5649ed212d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05feffd640 1 -- 172.21.15.27:0/2279684497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 msgr2=0x5649ed212d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05feffd640 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 0x5649ed212d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05feffd640 1 -- 172.21.15.27:0/2279684497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 msgr2=0x5649ed1d83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05feffd640 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed1d83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05feffd640 1 -- 172.21.15.27:0/2279684497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649ed1da4e0 con 0x5649ed1d9760 2024-03-06T07:23:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05f7fff640 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 0x5649ed212d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.282+0000 7f05feffd640 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1d9760 0x5649ed1e3ec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009960 tx=0x7f05f0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb39ec30a2ab2a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f05f4ff9640 1 -- 172.21.15.27:0/2279684497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0059070 con 0x5649ed1d9760 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f05f4ff9640 1 -- 172.21.15.27:0/2279684497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f0053770 con 0x5649ed1d9760 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f05f4ff9640 1 -- 172.21.15.27:0/2279684497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0053b20 con 0x5649ed1d9760 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f06085cf740 1 -- 172.21.15.27:0/2279684497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1d9760 msgr2=0x5649ed1e3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f06085cf740 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1d9760 0x5649ed1e3ec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009960 tx=0x7f05f0030000 comp rx=0 tx=0).stop 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f06085cf740 1 -- 172.21.15.27:0/2279684497 shutdown_connections 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f06085cf740 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1e4600 0x5649ed1d83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f06085cf740 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 0x5649ed212d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f06085cf740 1 --2- 172.21.15.27:0/2279684497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1d9760 0x5649ed1e3ec0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f06085cf740 1 -- 172.21.15.27:0/2279684497 >> 172.21.15.27:0/2279684497 conn(0x5649ed1b4870 msgr2=0x5649ed223d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f06085cf740 1 -- 172.21.15.27:0/2279684497 shutdown_connections 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.283+0000 7f06085cf740 1 -- 172.21.15.27:0/2279684497 wait complete. 2024-03-06T07:23:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.284+0000 7f06085cf740 1 -- start start 2024-03-06T07:23:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.284+0000 7f06085cf740 1 Processor -- start 2024-03-06T07:23:03.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f06085cf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1d9760 0x5649ed1b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f06085cf740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 0x5649ed1d83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f06085cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e4600 0x5649ed202ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f06085cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ed210700 con 0x5649ed1e4600 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f06085cf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ed1e75a0 con 0x5649ed1d9760 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f06085cf740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649ed1d9ff0 con 0x5649ed1e12a0 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f06085cf740 10 client.0 _ll_register_callbacks cb 0x7f05f8059310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e4600 0x5649ed202ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e4600 0x5649ed202ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52120/0 (socket says 172.21.15.27:52120) 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05ff7fe640 1 -- 172.21.15.27:0/2893726946 learned_addr learned my addr 172.21.15.27:0/2893726946 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05feffd640 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1d9760 0x5649ed1b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05f7fff640 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 0x5649ed1d83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05ff7fe640 1 -- 172.21.15.27:0/2893726946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 msgr2=0x5649ed1d83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 0x5649ed1d83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05ff7fe640 1 -- 172.21.15.27:0/2893726946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1d9760 msgr2=0x5649ed1b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1d9760 0x5649ed1b82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05ff7fe640 1 -- 172.21.15.27:0/2893726946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649ecefd6b0 con 0x5649ed1e4600 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05feffd640 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1d9760 0x5649ed1b82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05ff7fe640 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e4600 0x5649ed202ad0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f05f800ac10 tx=0x7f05f8088e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41572a6bfde8d0d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05de7fc640 1 -- 172.21.15.27:0/2893726946 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f807b020 con 0x5649ed1e4600 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f05de7fc640 1 -- 172.21.15.27:0/2893726946 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05f8055060 con 0x5649ed1e4600 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.285+0000 7f06085cf740 1 -- 172.21.15.27:0/2893726946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05f0053770 con 0x5649ed1e4600 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f05de7fc640 1 -- 172.21.15.27:0/2893726946 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05f805d160 con 0x5649ed1e4600 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f06085cf740 10 client.4671 fetch_fsmap learned FSMap version 7 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f06085cf740 1 -- 172.21.15.27:0/2893726946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5649ed1ba8d0 con 0x5649ed1e4600 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f05de7fc640 1 -- 172.21.15.27:0/2893726946 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f0053770 con 0x5649ed1e4600 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f05de7fc640 1 -- 172.21.15.27:0/2893726946 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 7) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f05f8055260 con 0x5649ed1e4600 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f06085cf740 10 client.4671 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:23:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f06085cf740 10 client.4671 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f06085cf740 20 client.4671 populate_metadata read hostname 'smithi027' 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f06085cf740 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] conn(0x5649ed264470 0x5649ed264850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f06085cf740 4 client.4671 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f06085cf740 1 -- 172.21.15.27:0/2893726946 --> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5649ed125f40 con 0x5649ed264470 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f05feffd640 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] conn(0x5649ed264470 0x5649ed264850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f05de7fc640 10 client.4671 ms_handle_connect on v2:172.21.15.27:6834/3933261233 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f05feffd640 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] conn(0x5649ed264470 0x5649ed264850 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32c5cb1b641c7fd4 server_cookie=ff8195442adb0b86 in_seq=0 out_seq=0 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f05de7fc640 1 -- 172.21.15.27:0/2893726946 <== mds.0 v2:172.21.15.27:6834/3933261233 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5649ed125f40 con 0x5649ed264470 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f05de7fc640 10 client.4671 handle_command_reply: tid=0 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.286+0000 7f06085cf740 1 client.4671 shutdown 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 -- 172.21.15.27:0/2893726946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e4600 msgr2=0x5649ed202ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e4600 0x5649ed202ad0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f05f800ac10 tx=0x7f05f8088e20 comp rx=0 tx=0).stop 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 -- 172.21.15.27:0/2893726946 shutdown_connections 2024-03-06T07:23:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.27:6834/3933261233,v1:172.21.15.27:6835/3933261233] conn(0x5649ed264470 0x5649ed264850 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649ed1e4600 0x5649ed202ad0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5649ed1e12a0 0x5649ed1d83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f05de7fc640 0 client.4671 ms_handle_reset on v2:172.21.15.27:6834/3933261233 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 --2- 172.21.15.27:0/2893726946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5649ed1d9760 0x5649ed1b82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 -- 172.21.15.27:0/2893726946 >> 172.21.15.27:0/2893726946 conn(0x5649ed1b4870 msgr2=0x5649ed267bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 -- 172.21.15.27:0/2893726946 shutdown_connections 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 1 -- 172.21.15.27:0/2893726946 wait complete. 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f06085cf740 20 client.4671 trim_cache size 0 max 16384 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:23:03.296 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4290, 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 14.231376403000001, 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 7, 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 21, 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 21, 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 21.702310766 2024-03-06T07:23:03.297 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 -- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05e006b4d0 msgr2=0x7f05e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05e006b4d0 0x7f05e006d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f05f0036000 tx=0x7f05f0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 -- 172.21.15.27:0/1989898896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 msgr2=0x7f06000ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 0x7f06000ce2b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f05f800cef0 tx=0x7f05f800a710 comp rx=0 tx=0).stop 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 -- 172.21.15.27:0/1989898896 shutdown_connections 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05e006b4d0 0x7f05e006d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060015dd70 0x7f06000ce2b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06000b7c70 0x7f06000cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 --2- 172.21.15.27:0/1989898896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06000b5700 0x7f060015a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.287+0000 7f0605916640 1 -- 172.21.15.27:0/1989898896 >> 172.21.15.27:0/1989898896 conn(0x7f0600071960 msgr2=0x7f06000b36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.288+0000 7f0605916640 1 -- 172.21.15.27:0/1989898896 shutdown_connections 2024-03-06T07:23:03.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.288+0000 7f0605916640 1 -- 172.21.15.27:0/1989898896 wait complete. 2024-03-06T07:23:03.300 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-06T07:23:03.327 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-06T07:23:03.327 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:23:03.327 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-06T07:23:03.327 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:23:03.327 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:03.327 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-06T07:23:03.327 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:23:03.327 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:23:03.328 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-06T07:23:03.328 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:23:03.328 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:03.328 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-06T07:23:03.328 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:23:03.328 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.328 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-03-06T07:23:03.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.348 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-03-06T07:23:03.421 INFO:teuthology.orchestra.run.smithi069.stderr:Cannot find device "ceph-brx" 2024-03-06T07:23:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:03.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.423 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:23:03.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-03-06T07:23:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2024-03-06T07:23:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:03.515 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-06T07:23:03.515 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:03.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.660+0000 7fe781484640 1 Processor -- start 2024-03-06T07:23:03.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.660+0000 7fe781484640 1 -- start start 2024-03-06T07:23:03.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.661+0000 7fe781484640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe77c15dca0 0x7fe77c06fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.661+0000 7fe781484640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77c161c20 con 0x7fe77c0b00c0 2024-03-06T07:23:03.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.661+0000 7fe781484640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77c161f00 con 0x7fe77c15e080 2024-03-06T07:23:03.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.661+0000 7fe781484640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77c1619d0 con 0x7fe77c15dca0 2024-03-06T07:23:03.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.661+0000 7fe77affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe77c15dca0 0x7fe77c06fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.661+0000 7fe77b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe77c0b00c0 0x7fe77c0b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48212/0 (socket says 172.21.15.27:48212) 2024-03-06T07:23:03.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.661+0000 7fe77b7fe640 1 -- 172.21.15.27:0/3887500735 learned_addr learned my addr 172.21.15.27:0/3887500735 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:03.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.662+0000 7fe77affd640 1 -- 172.21.15.27:0/3887500735 >> v1:172.21.15.69:6789/0 conn(0x7fe77c15e080 legacy=0x7fe77c066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.662+0000 7fe77affd640 1 -- 172.21.15.27:0/3887500735 >> v1:172.21.15.27:6789/0 conn(0x7fe77c0b00c0 legacy=0x7fe77c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.662+0000 7fe77affd640 1 -- 172.21.15.27:0/3887500735 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c1621e0 con 0x7fe77c15dca0 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.662+0000 7fe77affd640 1 --2- 172.21.15.27:0/3887500735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c15dca0 0x7fe77c06fd10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0097a0 tx=0x7fe76c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86718c78f7b6db6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.662+0000 7fe779ffb640 1 -- 172.21.15.27:0/3887500735 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c0315a0 con 0x7fe77c15dca0 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.663+0000 7fe779ffb640 1 -- 172.21.15.27:0/3887500735 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c037070 con 0x7fe77c15dca0 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.663+0000 7fe779ffb640 1 -- 172.21.15.27:0/3887500735 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c031960 con 0x7fe77c15dca0 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.663+0000 7fe781484640 1 -- 172.21.15.27:0/3887500735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c15dca0 msgr2=0x7fe77c06fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.663+0000 7fe781484640 1 --2- 172.21.15.27:0/3887500735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c15dca0 0x7fe77c06fd10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0097a0 tx=0x7fe76c02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.663+0000 7fe781484640 1 -- 172.21.15.27:0/3887500735 shutdown_connections 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.663+0000 7fe781484640 1 --2- 172.21.15.27:0/3887500735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c15dca0 0x7fe77c06fd10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.663+0000 7fe781484640 1 -- 172.21.15.27:0/3887500735 >> 172.21.15.27:0/3887500735 conn(0x7fe77c0d0990 msgr2=0x7fe77c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.663+0000 7fe781484640 1 -- 172.21.15.27:0/3887500735 shutdown_connections 2024-03-06T07:23:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.663+0000 7fe781484640 1 -- 172.21.15.27:0/3887500735 wait complete. 2024-03-06T07:23:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.665+0000 7fe781484640 1 Processor -- start 2024-03-06T07:23:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.666+0000 7fe781484640 1 -- start start 2024-03-06T07:23:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe781484640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c0b00c0 0x7fe77c152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe781484640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe77c15dca0 0x7fe77c15a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe781484640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe77c15ad70 0x7fe77c1f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe781484640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c0703d0 con 0x7fe77c0b00c0 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe781484640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c070250 con 0x7fe77c15dca0 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe781484640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c067410 con 0x7fe77c15ad70 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c0b00c0 0x7fe77c152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c0b00c0 0x7fe77c152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56790/0 (socket says 172.21.15.27:56790) 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77affd640 1 -- 172.21.15.27:0/2343860174 learned_addr learned my addr 172.21.15.27:0/2343860174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77a7fc640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe77c15dca0 0x7fe77c15a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77b7fe640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe77c15ad70 0x7fe77c1f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77affd640 1 -- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe77c15ad70 msgr2=0x7fe77c1f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77affd640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe77c15ad70 0x7fe77c1f3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77affd640 1 -- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe77c15dca0 msgr2=0x7fe77c15a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77affd640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe77c15dca0 0x7fe77c15a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77affd640 1 -- 172.21.15.27:0/2343860174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c1f4790 con 0x7fe77c0b00c0 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77b7fe640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe77c15ad70 0x7fe77c1f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.667+0000 7fe77affd640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c0b00c0 0x7fe77c152230 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0098c0 tx=0x7fe76c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.668+0000 7fe7637fe640 1 -- 172.21.15.27:0/2343860174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c032840 con 0x7fe77c0b00c0 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.668+0000 7fe7637fe640 1 -- 172.21.15.27:0/2343860174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c0329e0 con 0x7fe77c0b00c0 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.668+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77c1f7210 con 0x7fe77c0b00c0 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.668+0000 7fe7637fe640 1 -- 172.21.15.27:0/2343860174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c032dd0 con 0x7fe77c0b00c0 2024-03-06T07:23:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.668+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77c1f7cf0 con 0x7fe77c0b00c0 2024-03-06T07:23:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.669+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77c15b180 con 0x7fe77c0b00c0 2024-03-06T07:23:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.669+0000 7fe7637fe640 1 -- 172.21.15.27:0/2343860174 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe76c04d3f0 con 0x7fe77c0b00c0 2024-03-06T07:23:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.670+0000 7fe7637fe640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe76806b500 0x7fe76806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.670+0000 7fe7637fe640 1 -- 172.21.15.27:0/2343860174 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe76c02f030 con 0x7fe77c0b00c0 2024-03-06T07:23:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.670+0000 7fe77a7fc640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe76806b500 0x7fe76806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.671+0000 7fe77a7fc640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe76806b500 0x7fe76806d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe770006fd0 tx=0x7fe770008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.674+0000 7fe7637fe640 1 -- 172.21.15.27:0/2343860174 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe77c15b180 con 0x7fe77c0b00c0 2024-03-06T07:23:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.897+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe77c1f7000 con 0x7fe77c0b00c0 2024-03-06T07:23:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.897+0000 7fe7637fe640 1 -- 172.21.15.27:0/2343860174 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe76c040070 con 0x7fe77c0b00c0 2024-03-06T07:23:03.899 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe76806b500 msgr2=0x7fe76806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe76806b500 0x7fe76806d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe770006fd0 tx=0x7fe770008040 comp rx=0 tx=0).stop 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c0b00c0 msgr2=0x7fe77c152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c0b00c0 0x7fe77c152230 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0098c0 tx=0x7fe76c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 shutdown_connections 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe76806b500 0x7fe76806d9c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe77c15ad70 0x7fe77c1f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe77c15dca0 0x7fe77c15a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 --2- 172.21.15.27:0/2343860174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe77c0b00c0 0x7fe77c152230 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 >> 172.21.15.27:0/2343860174 conn(0x7fe77c0d0990 msgr2=0x7fe77c06dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 shutdown_connections 2024-03-06T07:23:03.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:03.899+0000 7fe781484640 1 -- 172.21.15.27:0/2343860174 wait complete. 2024-03-06T07:23:03.911 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:03.912 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:03.912 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:23:03.912 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:23:03.912 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:03.912 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:03.912 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:03.912 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:03.912 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:03.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.913 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42706sec preferred_lft 42706sec 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:23:03.930 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:23:03.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.930 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:23:03.930 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:23:03.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:23:03.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-03-06T07:23:03.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-03-06T07:23:03.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:23:03.930 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:23:04.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.102 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:04.135 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:23:04.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.136 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:23:04.207 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:23:04.207 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:04.207 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:23:04.207 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:23:04.207 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:04.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.208 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:23:04.208 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:23:04.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:23:04.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:23:04.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:23:04.208 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:23:04.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.480 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:23:04.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.497 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-03-06T07:23:04.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.552 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:23:04.552 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:23:04.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:04.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:23:04.552 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:23:04.641 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:23:04.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.642 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:23:04.642 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:23:04.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:23:04.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:23:04.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:23:04.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:23:04.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:23:04.642 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:23:04.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:04.758 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:23:04.758 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:23:04.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:23:04.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:23:04.758 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:23:04.808 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:23:04.808 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:04.808 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:04.826 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:04.826 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:04.826 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:04.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:23:04.954 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:05.012 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:23:05.012 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:23:05.012 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:23:05.012 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:23:05.012 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:23:05.012 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:23:05.012 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:23:05.012 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:05.013 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:05.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:05.015 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:23:05.067 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:05.068 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:23:05.109 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:23:05.142 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:05.198 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:23:05.198 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:23:05.198 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:23:05.198 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:05.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:05.201 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:23:05.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:23:05.224+0000 7f9b61036480 -1 init, newargv = 0x556b6d0ac530 newargc=15 2024-03-06T07:23:05.225 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[47025]: starting ceph client 2024-03-06T07:23:05.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[47025]: starting fuse 2024-03-06T07:23:05.241 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:23:05.241 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:23:05.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:05.241 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:23:05.241 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:23:05.241 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:23:05.241 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:05.242 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:05.243 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:05.243 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:23:05.243 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:23:05.243 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:23:05.243 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:05.243 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:23:05.243 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:23:05.243 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:23:05.367 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.47025.asok 2024-03-06T07:23:05.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:05.377 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47025.asok 2024-03-06T07:23:05.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:05.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47025.asok status 2024-03-06T07:23:05.520 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:23:05.520 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:23:05.520 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "47025", 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4689, 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 4689 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2347711800 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2347711800 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.4689 192.168.0.1:0/2347711800", 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/2347711800", 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 7, 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 21, 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:23:05.529 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:05.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.634+0000 7f8e0d15c640 1 Processor -- start 2024-03-06T07:23:05.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.634+0000 7f8e0d15c640 1 -- start start 2024-03-06T07:23:05.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.635+0000 7f8e0d15c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e080b9a10 0x7f8e080d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.635+0000 7f8e0d15c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e080cea30 con 0x7f8e0815df00 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.635+0000 7f8e0d15c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e080cf3c0 con 0x7f8e080b95a0 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.635+0000 7f8e0d15c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080cf0e0 con 0x7f8e080b9a10 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.635+0000 7f8e06d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8e0815df00 0x7f8e080d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48222/0 (socket says 172.21.15.27:48222) 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.635+0000 7f8e06d9d640 1 -- 172.21.15.27:0/313634858 learned_addr learned my addr 172.21.15.27:0/313634858 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.636+0000 7f8e0759e640 1 --2- 172.21.15.27:0/313634858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e080b9a10 0x7f8e080d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.636+0000 7f8e05d9b640 1 -- 172.21.15.27:0/313634858 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426448929 0 0) 0x7f8e080cea30 con 0x7f8e0815df00 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.636+0000 7f8e05d9b640 1 -- 172.21.15.27:0/313634858 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df0003520 con 0x7f8e0815df00 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.636+0000 7f8e05d9b640 1 -- 172.21.15.27:0/313634858 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451586446 0 0) 0x7f8e080cf3c0 con 0x7f8e080b95a0 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.636+0000 7f8e05d9b640 1 -- 172.21.15.27:0/313634858 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e080cea30 con 0x7f8e080b95a0 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.636+0000 7f8e0759e640 1 -- 172.21.15.27:0/313634858 >> v1:172.21.15.69:6789/0 conn(0x7f8e080b95a0 legacy=0x7f8e080b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.636+0000 7f8e0759e640 1 -- 172.21.15.27:0/313634858 >> v1:172.21.15.27:6789/0 conn(0x7f8e0815df00 legacy=0x7f8e080d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.636+0000 7f8e0759e640 1 -- 172.21.15.27:0/313634858 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e080ce470 con 0x7f8e080b9a10 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e0759e640 1 --2- 172.21.15.27:0/313634858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e080b9a10 0x7f8e080d3220 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009dc0 tx=0x7f8e0002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba866e953575443c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e05d9b640 1 -- 172.21.15.27:0/313634858 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00039070 con 0x7f8e080b9a10 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e05d9b640 1 -- 172.21.15.27:0/313634858 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0002ccd0 con 0x7f8e080b9a10 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e05d9b640 1 -- 172.21.15.27:0/313634858 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00034660 con 0x7f8e080b9a10 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e0d15c640 1 -- 172.21.15.27:0/313634858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e080b9a10 msgr2=0x7f8e080d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e0d15c640 1 --2- 172.21.15.27:0/313634858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e080b9a10 0x7f8e080d3220 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009dc0 tx=0x7f8e0002bf20 comp rx=0 tx=0).stop 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e0d15c640 1 -- 172.21.15.27:0/313634858 shutdown_connections 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e0d15c640 1 --2- 172.21.15.27:0/313634858 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e080b9a10 0x7f8e080d3220 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e0d15c640 1 -- 172.21.15.27:0/313634858 >> 172.21.15.27:0/313634858 conn(0x7f8e08071960 msgr2=0x7f8e08071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e0d15c640 1 -- 172.21.15.27:0/313634858 shutdown_connections 2024-03-06T07:23:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.637+0000 7f8e0d15c640 1 -- 172.21.15.27:0/313634858 wait complete. 2024-03-06T07:23:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.639+0000 7f8e0d15c640 1 Processor -- start 2024-03-06T07:23:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.639+0000 7f8e0d15c640 1 -- start start 2024-03-06T07:23:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0d15c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e080b95a0 0x7f8e0815b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0d15c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e080b9a10 0x7f8e0815ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0d15c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0815df00 0x7f8e080b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0d15c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080cedf0 con 0x7f8e080b9a10 2024-03-06T07:23:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0d15c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080cec70 con 0x7f8e080b95a0 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0d15c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b5e10 con 0x7f8e0815df00 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e080b9a10 0x7f8e0815ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e080b9a10 0x7f8e0815ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56798/0 (socket says 172.21.15.27:56798) 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0659c640 1 -- 172.21.15.27:0/1409210034 learned_addr learned my addr 172.21.15.27:0/1409210034 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0759e640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0815df00 0x7f8e080b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e06d9d640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e080b95a0 0x7f8e0815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0659c640 1 -- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0815df00 msgr2=0x7f8e080b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0659c640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0815df00 0x7f8e080b4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0659c640 1 -- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e080b95a0 msgr2=0x7f8e0815b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0659c640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e080b95a0 0x7f8e0815b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0659c640 1 -- 172.21.15.27:0/1409210034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0815b740 con 0x7f8e080b9a10 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.640+0000 7f8e0659c640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e080b9a10 0x7f8e0815ecc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00c500 tx=0x7f8dfc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.641+0000 7f8def7fe640 1 -- 172.21.15.27:0/1409210034 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc0052d0 con 0x7f8e080b9a10 2024-03-06T07:23:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.641+0000 7f8def7fe640 1 -- 172.21.15.27:0/1409210034 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dfc005490 con 0x7f8e080b9a10 2024-03-06T07:23:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.641+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e08160430 con 0x7f8e080b9a10 2024-03-06T07:23:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.641+0000 7f8def7fe640 1 -- 172.21.15.27:0/1409210034 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc012870 con 0x7f8e080b9a10 2024-03-06T07:23:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.641+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e081609f0 con 0x7f8e080b9a10 2024-03-06T07:23:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.642+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e08067ff0 con 0x7f8e080b9a10 2024-03-06T07:23:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.642+0000 7f8def7fe640 1 -- 172.21.15.27:0/1409210034 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8dfc012a10 con 0x7f8e080b9a10 2024-03-06T07:23:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.643+0000 7f8def7fe640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b4d0 0x7f8df406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.643+0000 7f8def7fe640 1 -- 172.21.15.27:0/1409210034 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dfc089da0 con 0x7f8e080b9a10 2024-03-06T07:23:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.643+0000 7f8e06d9d640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b4d0 0x7f8df406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.643+0000 7f8e06d9d640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b4d0 0x7f8df406d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8df80011e0 tx=0x7f8df8007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.647+0000 7f8def7fe640 1 -- 172.21.15.27:0/1409210034 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e0815b740 con 0x7f8e080b9a10 2024-03-06T07:23:05.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.869+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8e08068890 con 0x7f8e080b9a10 2024-03-06T07:23:05.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.869+0000 7f8def7fe640 1 -- 172.21.15.27:0/1409210034 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8dfc0585e0 con 0x7f8e080b9a10 2024-03-06T07:23:05.871 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:05.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b4d0 msgr2=0x7f8df406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b4d0 0x7f8df406d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8df80011e0 tx=0x7f8df8007d30 comp rx=0 tx=0).stop 2024-03-06T07:23:05.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e080b9a10 msgr2=0x7f8e0815ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e080b9a10 0x7f8e0815ecc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00c500 tx=0x7f8dfc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 shutdown_connections 2024-03-06T07:23:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b4d0 0x7f8df406d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e0815df00 0x7f8e080b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e080b9a10 0x7f8e0815ecc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 --2- 172.21.15.27:0/1409210034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e080b95a0 0x7f8e0815b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 >> 172.21.15.27:0/1409210034 conn(0x7f8e08071960 msgr2=0x7f8e08162140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 shutdown_connections 2024-03-06T07:23:05.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:05.872+0000 7f8e0d15c640 1 -- 172.21.15.27:0/1409210034 wait complete. 2024-03-06T07:23:05.884 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:05.884 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:05.884 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:23:05.884 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:23:05.884 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:23:05.884 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:05.884 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:05.884 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:05.884 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:23:05.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:05.885 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:23:05.900 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:05.900 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:05.900 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:05.900 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:05.900 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:05.900 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:05.900 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:05.900 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42704sec preferred_lft 42704sec 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:23:05.901 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link tentative 2024-03-06T07:23:05.902 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:05.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:05.903 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:05.971 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:23:05.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:05.972 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:23:06.028 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:23:06.028 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:06.029 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:23:06.029 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:23:06.029 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:06.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.029 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:23:06.029 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:23:06.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:23:06.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:23:06.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:23:06.029 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:23:06.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.134 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:23:06.149 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:23:06.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.150 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-03-06T07:23:06.206 INFO:teuthology.orchestra.run.smithi027.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:23:06.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.207 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:23:06.207 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:23:06.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:06.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:23:06.207 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:23:06.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:23:06.326 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:06.326 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:06.326 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:06.326 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:06.326 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:06.327 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:06.327 INFO:teuthology.orchestra.run.smithi027.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:06.327 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether d6:e9:40:dc:85:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:23:06.327 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:23:06.327 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:06.327 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::d4e9:40ff:fedc:85a2/64 scope link 2024-03-06T07:23:06.327 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:06.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:23:06.398 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether d6:e9:40:dc:85:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::d4e9:40ff:fedc:85a2/64 scope link 2024-03-06T07:23:06.399 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:06.401 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:23:06.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.401 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:23:06.401 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:23:06.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:23:06.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:23:06.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:23:06.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:23:06.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:23:06.401 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:23:06.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.545 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:23:06.545 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:23:06.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.1 up 2024-03-06T07:23:06.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:23:06.545 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:23:06.595 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-06T07:23:06.595 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:06.595 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:23:06.610 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:23:06.610 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:06.610 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:23:06.663 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:23:06.734 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:06.790 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:23:06.790 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:23:06.791 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:06.792 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:06.793 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:23:06.793 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:06.793 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:06.793 INFO:teuthology.orchestra.run.smithi027.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:06.793 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:06.793 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:06.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.793 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:23:06.845 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:23:06.845 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-06T07:23:06.845 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-06T07:23:06.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:23:06.919 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:06.975 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:23:06.976 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:23:06.977 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:06.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.979 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:23:06.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi027.stderr:2024-03-06T07:23:06.985+0000 7fd8860bb480 -1 init, newargv = 0x5595a2f39700 newargc=15 2024-03-06T07:23:06.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi027.stderr:ceph-fuse[47501]: starting ceph client 2024-03-06T07:23:06.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi027.stderr:ceph-fuse[47501]: starting fuse 2024-03-06T07:23:07.002 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:23:07.002 INFO:teuthology.orchestra.run.smithi027.stdout:44 2024-03-06T07:23:07.003 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:23:07.003 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.2")) 2024-03-06T07:23:07.004 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:23:07.127 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.2.47501.asok 2024-03-06T07:23:07.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:07.139 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.47501.asok 2024-03-06T07:23:07.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.47501.asok status 2024-03-06T07:23:07.281 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:23:07.281 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:23:07.281 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "2", 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "47501", 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4704, 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:23:07.282 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 4704 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3616489793 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:23:07.283 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3616489793 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.4704 192.168.0.2:0/3616489793", 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.2:0/3616489793", 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 7, 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 21, 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:23:07.284 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:23:07.290 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:07.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.394+0000 7fd4e3c88640 1 Processor -- start 2024-03-06T07:23:07.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.395+0000 7fd4e3c88640 1 -- start start 2024-03-06T07:23:07.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.395+0000 7fd4e3c88640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd4dc0b4260 0x7fd4dc0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.395+0000 7fd4e3c88640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc0b2320 con 0x7fd4dc0d06c0 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.395+0000 7fd4e3c88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc0b0dd0 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.395+0000 7fd4e3c88640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc0b0b80 con 0x7fd4dc0b4260 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.395+0000 7fd4e2225640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd4dc0d06c0 0x7fd4dc0b1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48236/0 (socket says 172.21.15.27:48236) 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.395+0000 7fd4e2225640 1 -- 172.21.15.27:0/2266293802 learned_addr learned my addr 172.21.15.27:0/2266293802 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.396+0000 7fd4e1223640 1 --2- 172.21.15.27:0/2266293802 >> v2:172.21.15.69:3300/0 conn(0x7fd4dc0b4260 0x7fd4dc0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.396+0000 7fd4e0a22640 1 -- 172.21.15.27:0/2266293802 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251971413 0 0) 0x7fd4dc0b2320 con 0x7fd4dc0d06c0 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.396+0000 7fd4e0a22640 1 -- 172.21.15.27:0/2266293802 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c8003520 con 0x7fd4dc0d06c0 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.396+0000 7fd4e1223640 1 -- 172.21.15.27:0/2266293802 >> v1:172.21.15.69:6789/0 conn(0x7fd4dc0d02e0 legacy=0x7fd4dc0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.396+0000 7fd4e1223640 1 -- 172.21.15.27:0/2266293802 >> v1:172.21.15.27:6789/0 conn(0x7fd4dc0d06c0 legacy=0x7fd4dc0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.396+0000 7fd4e1223640 1 -- 172.21.15.27:0/2266293802 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0b10b0 con 0x7fd4dc0b4260 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.397+0000 7fd4e1223640 1 --2- 172.21.15.27:0/2266293802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4dc0b4260 0x7fd4dc0cfa00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80097a0 tx=0x7fd4d802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6fb8e26fe77c4bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.397+0000 7fd4e0a22640 1 -- 172.21.15.27:0/2266293802 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d802f810 con 0x7fd4dc0b4260 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.397+0000 7fd4e0a22640 1 -- 172.21.15.27:0/2266293802 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d802f9b0 con 0x7fd4dc0b4260 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.397+0000 7fd4e0a22640 1 -- 172.21.15.27:0/2266293802 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d802fd60 con 0x7fd4dc0b4260 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.398+0000 7fd4e3c88640 1 -- 172.21.15.27:0/2266293802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4dc0b4260 msgr2=0x7fd4dc0cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.398+0000 7fd4e3c88640 1 --2- 172.21.15.27:0/2266293802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4dc0b4260 0x7fd4dc0cfa00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80097a0 tx=0x7fd4d802b9f0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.398+0000 7fd4e3c88640 1 -- 172.21.15.27:0/2266293802 shutdown_connections 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.398+0000 7fd4e3c88640 1 --2- 172.21.15.27:0/2266293802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4dc0b4260 0x7fd4dc0cfa00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.398+0000 7fd4e3c88640 1 -- 172.21.15.27:0/2266293802 >> 172.21.15.27:0/2266293802 conn(0x7fd4dc0b36d0 msgr2=0x7fd4dc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.398+0000 7fd4e3c88640 1 -- 172.21.15.27:0/2266293802 shutdown_connections 2024-03-06T07:23:07.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.398+0000 7fd4e3c88640 1 -- 172.21.15.27:0/2266293802 wait complete. 2024-03-06T07:23:07.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.399+0000 7fd4e3c88640 1 Processor -- start 2024-03-06T07:23:07.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.399+0000 7fd4e3c88640 1 -- start start 2024-03-06T07:23:07.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e3c88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4dc0b4260 0x7fd4dc0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e3c88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4dc0d02e0 0x7fd4dc0ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e3c88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4dc0d06c0 0x7fd4dc153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e3c88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0b0360 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e3c88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0aeec0 con 0x7fd4dc0d06c0 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e3c88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0b04e0 con 0x7fd4dc0b4260 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1223640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4dc0d02e0 0x7fd4dc0ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1223640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4dc0d02e0 0x7fd4dc0ca2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56812/0 (socket says 172.21.15.27:56812) 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1223640 1 -- 172.21.15.27:0/982602441 learned_addr learned my addr 172.21.15.27:0/982602441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e2225640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4dc0d06c0 0x7fd4dc153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1a24640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4dc0b4260 0x7fd4dc0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1223640 1 -- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4dc0b4260 msgr2=0x7fd4dc0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1223640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4dc0b4260 0x7fd4dc0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1223640 1 -- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4dc0d06c0 msgr2=0x7fd4dc153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1223640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4dc0d06c0 0x7fd4dc153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1223640 1 -- 172.21.15.27:0/982602441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0c9220 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.400+0000 7fd4e1223640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4dc0d02e0 0x7fd4dc0ca2c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80097a0 tx=0x7fd4d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.401+0000 7fd4d27fc640 1 -- 172.21.15.27:0/982602441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8047070 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.401+0000 7fd4d27fc640 1 -- 172.21.15.27:0/982602441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d8004250 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.401+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc0c9500 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.401+0000 7fd4d27fc640 1 -- 172.21.15.27:0/982602441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8003610 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.401+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc0c9ac0 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.402+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc068700 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.402+0000 7fd4d27fc640 1 -- 172.21.15.27:0/982602441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4d804a070 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.403+0000 7fd4d27fc640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd4c006b4d0 0x7fd4c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.403+0000 7fd4d27fc640 1 -- 172.21.15.27:0/982602441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4d80ab6d0 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.403+0000 7fd4e1a24640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd4c006b4d0 0x7fd4c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.403+0000 7fd4e1a24640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd4c006b4d0 0x7fd4c006d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc007fd0 tx=0x7fd4cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.407+0000 7fd4d27fc640 1 -- 172.21.15.27:0/982602441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4d8033030 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.629+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd4dc0d2330 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.629+0000 7fd4d27fc640 1 -- 172.21.15.27:0/982602441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd4d807a730 con 0x7fd4dc0d02e0 2024-03-06T07:23:07.631 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd4c006b4d0 msgr2=0x7fd4c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd4c006b4d0 0x7fd4c006d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc007fd0 tx=0x7fd4cc009040 comp rx=0 tx=0).stop 2024-03-06T07:23:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4dc0d02e0 msgr2=0x7fd4dc0ca2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4dc0d02e0 0x7fd4dc0ca2c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80097a0 tx=0x7fd4d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 shutdown_connections 2024-03-06T07:23:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd4c006b4d0 0x7fd4c006d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4dc0d06c0 0x7fd4dc153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4dc0d02e0 0x7fd4dc0ca2c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 --2- 172.21.15.27:0/982602441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd4dc0b4260 0x7fd4dc0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 >> 172.21.15.27:0/982602441 conn(0x7fd4dc0b36d0 msgr2=0x7fd4dc0cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 shutdown_connections 2024-03-06T07:23:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:07.631+0000 7fd4e3c88640 1 -- 172.21.15.27:0/982602441 wait complete. 2024-03-06T07:23:07.644 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:07.644 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:07.644 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-03-06T07:23:07.644 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:23:07.644 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:07.644 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:07.644 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:07.644 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:07.644 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:23:07.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.645 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:07.664 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42658sec preferred_lft 42658sec 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:07.665 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:23:07.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.666 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-03-06T07:23:07.666 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-03-06T07:23:07.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:23:07.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr flush dev ceph-brx 2024-03-06T07:23:07.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set ceph-brx up 2024-03-06T07:23:07.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:23:07.666 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-03-06T07:23:07.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.838 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:07.866 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-03-06T07:23:07.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.867 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-03-06T07:23:07.940 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-03-06T07:23:07.940 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:07.940 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:23:07.940 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:23:07.940 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:07.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.941 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-03-06T07:23:07.941 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-03-06T07:23:07.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:23:07.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:23:07.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:23:07.941 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-03-06T07:23:08.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:08.252 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-03-06T07:23:08.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:08.267 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-03-06T07:23:08.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:08.323 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-03-06T07:23:08.323 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-03-06T07:23:08.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:08.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:23:08.323 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-03-06T07:23:08.405 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:23:08.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:08.405 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-03-06T07:23:08.406 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-03-06T07:23:08.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:23:08.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:23:08.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:23:08.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:23:08.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:23:08.406 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-03-06T07:23:08.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:08.570 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-03-06T07:23:08.570 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-03-06T07:23:08.570 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-03-06T07:23:08.570 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:23:08.570 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-03-06T07:23:08.612 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-06T07:23:08.612 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:08.612 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:08.667 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:23:08.668 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:08.668 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:08.722 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-03-06T07:23:08.790 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:08.846 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-03-06T07:23:08.846 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-03-06T07:23:08.846 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-03-06T07:23:08.847 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:23:08.848 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-03-06T07:23:08.849 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:08.849 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:08.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:08.849 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-03-06T07:23:08.901 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:08.902 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-06T07:23:08.945 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-03-06T07:23:08.973 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:09.028 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-03-06T07:23:09.028 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-03-06T07:23:09.028 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-03-06T07:23:09.028 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-03-06T07:23:09.028 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-03-06T07:23:09.029 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:09.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:09.031 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-03-06T07:23:09.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:2024-03-06T07:23:09.028+0000 7f3a18e13480 -1 init, newargv = 0x559dc2504620 newargc=15 2024-03-06T07:23:09.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[44190]: starting ceph client 2024-03-06T07:23:09.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi069.stderr:ceph-fuse[44190]: starting fuse 2024-03-06T07:23:09.046 INFO:teuthology.orchestra.run.smithi069.stdout:42 2024-03-06T07:23:09.046 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:23:09.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:09.046 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-03-06T07:23:09.047 DEBUG:teuthology.orchestra.run.smithi069:> 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.1")) 2024-03-06T07:23:09.048 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-03-06T07:23:09.167 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.1.44190.asok 2024-03-06T07:23:09.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:23:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:09.177 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44190.asok 2024-03-06T07:23:09.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:09.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44190.asok status 2024-03-06T07:23:09.314 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-03-06T07:23:09.314 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-03-06T07:23:09.314 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "1", 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "44190", 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4597, 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-03-06T07:23:09.315 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 4597 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 952211466 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 952211466 2024-03-06T07:23:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-03-06T07:23:09.317 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.4597 192.168.0.1:0/952211466", 2024-03-06T07:23:09.317 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/952211466", 2024-03-06T07:23:09.317 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-03-06T07:23:09.317 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 7, 2024-03-06T07:23:09.317 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 21, 2024-03-06T07:23:09.317 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:09.317 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-03-06T07:23:09.317 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-03-06T07:23:09.317 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-03-06T07:23:09.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:09.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.433+0000 7f0a0750c640 1 Processor -- start 2024-03-06T07:23:09.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.433+0000 7f0a0750c640 1 -- start start 2024-03-06T07:23:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.434+0000 7f0a0750c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 0x7f0a000b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.434+0000 7f0a0750c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0a000d3400 0x7f0a00071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.434+0000 7f0a0750c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a00071e60 con 0x7f0a000b2b50 2024-03-06T07:23:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.434+0000 7f0a0750c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a000d2f70 con 0x7f0a000d3400 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.434+0000 7f0a0750c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000d4be0 con 0x7f0a00071540 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.434+0000 7f0a05aa9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0a000b2b50 0x7f0a000d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48240/0 (socket says 172.21.15.27:48240) 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.434+0000 7f0a05aa9640 1 -- 172.21.15.27:0/1237026190 learned_addr learned my addr 172.21.15.27:0/1237026190 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.434+0000 7f0a04aa7640 1 --2- 172.21.15.27:0/1237026190 >> v2:172.21.15.69:3300/0 conn(0x7f0a000d3400 0x7f0a00071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.435+0000 7f0a052a8640 1 --2- 172.21.15.27:0/1237026190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 0x7f0a000b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.435+0000 7f09f3fff640 1 -- 172.21.15.27:0/1237026190 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250057827 0 0) 0x7f0a00071e60 con 0x7f0a000b2b50 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.435+0000 7f09f3fff640 1 -- 172.21.15.27:0/1237026190 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ec003520 con 0x7f0a000b2b50 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.435+0000 7f0a052a8640 1 -- 172.21.15.27:0/1237026190 >> v2:172.21.15.69:3300/0 conn(0x7f0a000d3400 msgr2=0x7f0a00071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.435+0000 7f0a052a8640 1 --2- 172.21.15.27:0/1237026190 >> v2:172.21.15.69:3300/0 conn(0x7f0a000d3400 0x7f0a00071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.435+0000 7f0a052a8640 1 -- 172.21.15.27:0/1237026190 >> v1:172.21.15.27:6789/0 conn(0x7f0a000b2b50 legacy=0x7f0a000d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.435+0000 7f09f3fff640 1 -- 172.21.15.27:0/1237026190 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1946262903 0 0) 0x7f09ec003520 con 0x7f0a000b2b50 2024-03-06T07:23:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.435+0000 7f0a052a8640 1 -- 172.21.15.27:0/1237026190 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000ce610 con 0x7f0a00071540 2024-03-06T07:23:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.435+0000 7f0a04aa7640 1 --2- 172.21.15.27:0/1237026190 >> v2:172.21.15.69:3300/0 conn(0x7f0a000d3400 0x7f0a00071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f0a052a8640 1 --2- 172.21.15.27:0/1237026190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 0x7f0a000b30a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0097a0 tx=0x7f09fc02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7a2b02e1531e4f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f09f3fff640 1 -- 172.21.15.27:0/1237026190 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc037070 con 0x7f0a00071540 2024-03-06T07:23:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f09f3fff640 1 -- 172.21.15.27:0/1237026190 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc030510 con 0x7f0a00071540 2024-03-06T07:23:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f09f3fff640 1 -- 172.21.15.27:0/1237026190 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f80019f0 con 0x7f0a00071540 2024-03-06T07:23:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f0a0750c640 1 -- 172.21.15.27:0/1237026190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 msgr2=0x7f0a000b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f0a0750c640 1 --2- 172.21.15.27:0/1237026190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 0x7f0a000b30a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0097a0 tx=0x7f09fc02b790 comp rx=0 tx=0).stop 2024-03-06T07:23:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f0a0750c640 1 -- 172.21.15.27:0/1237026190 shutdown_connections 2024-03-06T07:23:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f0a0750c640 1 --2- 172.21.15.27:0/1237026190 >> v2:172.21.15.69:3300/0 conn(0x7f0a000d3400 0x7f0a00071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f0a0750c640 1 --2- 172.21.15.27:0/1237026190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 0x7f0a000b30a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.436+0000 7f0a0750c640 1 -- 172.21.15.27:0/1237026190 >> 172.21.15.27:0/1237026190 conn(0x7f0a000d3ca0 msgr2=0x7f0a000d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.437+0000 7f0a0750c640 1 -- 172.21.15.27:0/1237026190 shutdown_connections 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.437+0000 7f0a0750c640 1 -- 172.21.15.27:0/1237026190 wait complete. 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.438+0000 7f0a0750c640 1 Processor -- start 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.438+0000 7f0a0750c640 1 -- start start 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.438+0000 7f0a0750c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 0x7f0a001f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.438+0000 7f0a0750c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a000b2b50 0x7f0a000cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.438+0000 7f0a0750c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a000d3400 0x7f0a00159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a0750c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000ce8f0 con 0x7f0a000d3400 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a0750c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000d2c00 con 0x7f0a000b2b50 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a0750c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0015e370 con 0x7f0a00071540 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a05aa9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a000d3400 0x7f0a00159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a05aa9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a000d3400 0x7f0a00159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56816/0 (socket says 172.21.15.27:56816) 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a05aa9640 1 -- 172.21.15.27:0/1497211368 learned_addr learned my addr 172.21.15.27:0/1497211368 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a04aa7640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a000b2b50 0x7f0a000cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a052a8640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 0x7f0a001f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a05aa9640 1 -- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 msgr2=0x7f0a001f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a05aa9640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 0x7f0a001f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a05aa9640 1 -- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a000b2b50 msgr2=0x7f0a000cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a05aa9640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a000b2b50 0x7f0a000cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a05aa9640 1 -- 172.21.15.27:0/1497211368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000cc300 con 0x7f0a000d3400 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a05aa9640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a000d3400 0x7f0a00159c60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f09f8001890 tx=0x7f09f800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f09f1ffb640 1 -- 172.21.15.27:0/1497211368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8012400 con 0x7f0a000d3400 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f09f1ffb640 1 -- 172.21.15.27:0/1497211368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f80125c0 con 0x7f0a000d3400 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a000cc5a0 con 0x7f0a000d3400 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.439+0000 7f09f1ffb640 1 -- 172.21.15.27:0/1497211368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f80129b0 con 0x7f0a000d3400 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.440+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a000ccb60 con 0x7f0a000d3400 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.441+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a00068700 con 0x7f0a000d3400 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.441+0000 7f09f1ffb640 1 -- 172.21.15.27:0/1497211368 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09f8012b50 con 0x7f0a000d3400 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.441+0000 7f09f1ffb640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09e406b4d0 0x7f09e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.442+0000 7f09f1ffb640 1 -- 172.21.15.27:0/1497211368 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09f808aa10 con 0x7f0a000d3400 2024-03-06T07:23:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.442+0000 7f0a052a8640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09e406b4d0 0x7f09e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.442+0000 7f0a052a8640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09e406b4d0 0x7f09e406d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f09fc034000 tx=0x7f09fc02f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.446+0000 7f09f1ffb640 1 -- 172.21.15.27:0/1497211368 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09f80591c0 con 0x7f0a000d3400 2024-03-06T07:23:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.668+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0a000d1bc0 con 0x7f0a000d3400 2024-03-06T07:23:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.668+0000 7f09f1ffb640 1 -- 172.21.15.27:0/1497211368 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f09f805d260 con 0x7f0a000d3400 2024-03-06T07:23:09.670 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09e406b4d0 msgr2=0x7f09e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09e406b4d0 0x7f09e406d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f09fc034000 tx=0x7f09fc02f000 comp rx=0 tx=0).stop 2024-03-06T07:23:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a000d3400 msgr2=0x7f0a00159c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a000d3400 0x7f0a00159c60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f09f8001890 tx=0x7f09f800a430 comp rx=0 tx=0).stop 2024-03-06T07:23:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 shutdown_connections 2024-03-06T07:23:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09e406b4d0 0x7f09e406d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a000d3400 0x7f0a00159c60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a000b2b50 0x7f0a000cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 --2- 172.21.15.27:0/1497211368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a00071540 0x7f0a001f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 >> 172.21.15.27:0/1497211368 conn(0x7f0a000d3ca0 msgr2=0x7f0a000d20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 shutdown_connections 2024-03-06T07:23:09.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:09.670+0000 7f0a0750c640 1 -- 172.21.15.27:0/1497211368 wait complete. 2024-03-06T07:23:09.682 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:09.682 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:09.682 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-03-06T07:23:09.682 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:23:09.683 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:23:09.683 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:09.683 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:09.683 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:09.683 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:23:09.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:09.683 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-03-06T07:23:09.699 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:09.700 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42656sec preferred_lft 42656sec 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0e:7f:9d:b2:e3:1b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:09.701 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a4ed:ddff:fe13:fbb4/64 scope link 2024-03-06T07:23:09.702 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:09.702 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:09.702 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0e:7f:9d:b2:e3:1b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:09.702 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::c7f:9dff:feb2:e31b/64 scope link tentative 2024-03-06T07:23:09.702 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:09.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:09.702 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:09.765 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-03-06T07:23:09.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:09.767 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-03-06T07:23:09.823 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-03-06T07:23:09.823 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:09.823 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:23:09.823 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:23:09.823 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:09.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:09.824 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-03-06T07:23:09.824 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-03-06T07:23:09.824 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:23:09.824 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:23:09.824 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:23:09.824 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-03-06T07:23:09.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:09.918 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-03-06T07:23:09.933 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:23:09.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:09.934 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-03-06T07:23:09.988 INFO:teuthology.orchestra.run.smithi069.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:23:09.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:09.989 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-03-06T07:23:09.989 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-03-06T07:23:09.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:23:09.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:23:09.989 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-03-06T07:23:10.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:23:10.137 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 1a:9c:4c:9d:48:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::189c:4cff:fe9d:48ca/64 scope link 2024-03-06T07:23:10.138 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.140 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:23:10.207 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.208 INFO:teuthology.orchestra.run.smithi069.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 1a:9c:4c:9d:48:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:23:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:23:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::189c:4cff:fe9d:48ca/64 scope link 2024-03-06T07:23:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:10.209 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:23:10.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.209 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-03-06T07:23:10.209 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-03-06T07:23:10.209 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:23:10.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:23:10.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:23:10.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:23:10.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:23:10.210 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-03-06T07:23:10.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.336 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec bash -c ' 2024-03-06T07:23:10.336 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-03-06T07:23:10.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.1 up 2024-03-06T07:23:10.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:23:10.336 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-03-06T07:23:10.377 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-06T07:23:10.378 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:10.378 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:23:10.433 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:23:10.433 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:10.433 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:23:10.487 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-03-06T07:23:10.555 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-03-06T07:23:10.611 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-03-06T07:23:10.612 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-03-06T07:23:10.613 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi069.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:23:10.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.614 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-03-06T07:23:10.666 INFO:teuthology.orchestra.run.smithi069.stdout:42 2024-03-06T07:23:10.666 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-03-06T07:23:10.667 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-06T07:23:10.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-03-06T07:23:10.736 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:10.793 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-03-06T07:23:10.793 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-03-06T07:23:10.793 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-03-06T07:23:10.793 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-03-06T07:23:10.793 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-03-06T07:23:10.793 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-03-06T07:23:10.793 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-03-06T07:23:10.793 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-03-06T07:23:10.794 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-03-06T07:23:10.794 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-03-06T07:23:10.794 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-03-06T07:23:10.794 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-03-06T07:23:10.794 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:10.794 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-03-06T07:23:10.794 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-03-06T07:23:10.794 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-03-06T07:23:10.794 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:23:10.795 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:23:10.796 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:23:10.796 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:23:10.796 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-03-06T07:23:10.796 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:10.796 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:23:10.796 INFO:teuthology.orchestra.run.smithi069.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:10.796 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:23:10.796 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:23:10.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.797 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-03-06T07:23:10.797 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi069.stderr:2024-03-06T07:23:10.792+0000 7f97bec47480 -1 init, newargv = 0x558620c10800 newargc=15 2024-03-06T07:23:10.797 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi069.stderr:ceph-fuse[44642]: starting ceph client 2024-03-06T07:23:10.798 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi069.stderr:ceph-fuse[44642]: starting fuse 2024-03-06T07:23:10.812 INFO:teuthology.orchestra.run.smithi069.stdout:42 2024-03-06T07:23:10.812 INFO:teuthology.orchestra.run.smithi069.stdout:44 2024-03-06T07:23:10.812 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-03-06T07:23:10.812 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:10.812 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-03-06T07:23:10.812 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-03-06T07:23:10.812 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-03-06T07:23:10.812 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-03-06T07:23:10.812 DEBUG:teuthology.orchestra.run.smithi069:> 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:10.813 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-03-06T07:23:10.814 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-03-06T07:23:10.814 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-03-06T07:23:10.814 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:10.814 DEBUG:teuthology.orchestra.run.smithi069:> 2024-03-06T07:23:10.814 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.3")) 2024-03-06T07:23:10.814 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-03-06T07:23:10.931 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.3.44642.asok 2024-03-06T07:23:10.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-03-06T07:23:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:10.942 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44642.asok 2024-03-06T07:23:10.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:10.942 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44642.asok status 2024-03-06T07:23:11.077 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-03-06T07:23:11.077 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-03-06T07:23:11.077 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "3", 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "44642", 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4529, 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-03-06T07:23:11.078 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 4529 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1870846932 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1870846932 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-03-06T07:23:11.079 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.4529 192.168.0.2:0/1870846932", 2024-03-06T07:23:11.080 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.2:0/1870846932", 2024-03-06T07:23:11.080 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-03-06T07:23:11.080 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 7, 2024-03-06T07:23:11.080 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 21, 2024-03-06T07:23:11.080 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:11.080 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-03-06T07:23:11.080 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-03-06T07:23:11.080 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-03-06T07:23:11.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.086 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:23:11.086 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:11.102 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:23:11.103 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:11.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:11.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.175 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:23:11.175 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-06T07:23:11.231 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:23:11.231 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-06T07:23:11.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:23:11.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.304 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-03-06T07:23:11.304 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:11.320 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-03-06T07:23:11.320 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:11.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.320 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:11.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.387 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-03-06T07:23:11.387 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-06T07:23:11.442 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-03-06T07:23:11.442 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-06T07:23:11.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:11.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:23:11.508 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:23:11.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:11.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.629+0000 7fc4b3c65640 1 Processor -- start 2024-03-06T07:23:11.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.630+0000 7fc4b3c65640 1 -- start start 2024-03-06T07:23:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.630+0000 7fc4b3c65640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4ac0b4260 0x7fc4ac0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.630+0000 7fc4b3c65640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac0b2320 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b3c65640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac0b0dd0 con 0x7fc4ac0d02e0 2024-03-06T07:23:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b3c65640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac0b0b80 con 0x7fc4ac0b4260 2024-03-06T07:23:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b1200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4ac0b4260 0x7fc4ac0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b1200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4ac0b4260 0x7fc4ac0cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:37402/0 (socket says 172.21.15.27:37402) 2024-03-06T07:23:11.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b1200640 1 -- 172.21.15.27:0/861803940 learned_addr learned my addr 172.21.15.27:0/861803940 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:11.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b2202640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc4ac0d06c0 0x7fc4ac0b1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48256/0 (socket says 172.21.15.27:48256) 2024-03-06T07:23:11.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b09ff640 1 -- 172.21.15.27:0/861803940 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247768347 0 0) 0x7fc4ac0b2320 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b1200640 1 -- 172.21.15.27:0/861803940 >> v1:172.21.15.69:6789/0 conn(0x7fc4ac0d02e0 legacy=0x7fc4ac0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b1200640 1 -- 172.21.15.27:0/861803940 >> v1:172.21.15.27:6789/0 conn(0x7fc4ac0d06c0 legacy=0x7fc4ac0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.631+0000 7fc4b1200640 1 -- 172.21.15.27:0/861803940 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0b10b0 con 0x7fc4ac0b4260 2024-03-06T07:23:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.632+0000 7fc4b1200640 1 --2- 172.21.15.27:0/861803940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4ac0b4260 0x7fc4ac0cfa00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0097f0 tx=0x7fc49c02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26e8be0e5888c541 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.632+0000 7fc4b09ff640 1 -- 172.21.15.27:0/861803940 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c035020 con 0x7fc4ac0b4260 2024-03-06T07:23:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.632+0000 7fc4b09ff640 1 -- 172.21.15.27:0/861803940 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c031800 con 0x7fc4ac0b4260 2024-03-06T07:23:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.632+0000 7fc4b09ff640 1 -- 172.21.15.27:0/861803940 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c031bd0 con 0x7fc4ac0b4260 2024-03-06T07:23:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.633+0000 7fc4b3c65640 1 -- 172.21.15.27:0/861803940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4ac0b4260 msgr2=0x7fc4ac0cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.633+0000 7fc4b3c65640 1 --2- 172.21.15.27:0/861803940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4ac0b4260 0x7fc4ac0cfa00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0097f0 tx=0x7fc49c02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.633+0000 7fc4b3c65640 1 -- 172.21.15.27:0/861803940 shutdown_connections 2024-03-06T07:23:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.633+0000 7fc4b3c65640 1 --2- 172.21.15.27:0/861803940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4ac0b4260 0x7fc4ac0cfa00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.633+0000 7fc4b3c65640 1 -- 172.21.15.27:0/861803940 >> 172.21.15.27:0/861803940 conn(0x7fc4ac0b36d0 msgr2=0x7fc4ac0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.633+0000 7fc4b3c65640 1 -- 172.21.15.27:0/861803940 shutdown_connections 2024-03-06T07:23:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.633+0000 7fc4b3c65640 1 -- 172.21.15.27:0/861803940 wait complete. 2024-03-06T07:23:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.635+0000 7fc4b3c65640 1 Processor -- start 2024-03-06T07:23:11.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.635+0000 7fc4b3c65640 1 -- start start 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b3c65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4ac0b4260 0x7fc4ac142fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b3c65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4ac0d02e0 0x7fc4ac0cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b3c65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ac0d06c0 0x7fc4ac15dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b3c65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac0b0360 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b3c65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac0aeec0 con 0x7fc4ac0b4260 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b3c65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac0b04e0 con 0x7fc4ac0d02e0 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b2202640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ac0d06c0 0x7fc4ac15dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b2202640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ac0d06c0 0x7fc4ac15dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56832/0 (socket says 172.21.15.27:56832) 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b2202640 1 -- 172.21.15.27:0/2960840405 learned_addr learned my addr 172.21.15.27:0/2960840405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b1200640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4ac0d02e0 0x7fc4ac0cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b1a01640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4ac0b4260 0x7fc4ac142fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b2202640 1 -- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4ac0d02e0 msgr2=0x7fc4ac0cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b2202640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4ac0d02e0 0x7fc4ac0cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b2202640 1 -- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4ac0b4260 msgr2=0x7fc4ac142fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b2202640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4ac0b4260 0x7fc4ac142fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b2202640 1 -- 172.21.15.27:0/2960840405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac143730 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.637+0000 7fc4b2202640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ac0d06c0 0x7fc4ac15dfb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c7a0 tx=0x7fc4a400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.638+0000 7fc4a27fc640 1 -- 172.21.15.27:0/2960840405 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4018070 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.638+0000 7fc4a27fc640 1 -- 172.21.15.27:0/2960840405 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a400a640 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.638+0000 7fc4a27fc640 1 -- 172.21.15.27:0/2960840405 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400aab0 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.638+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac143a10 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.638+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac143fd0 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.639+0000 7fc4a27fc640 1 -- 172.21.15.27:0/2960840405 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4a401b070 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.639+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac068700 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.640+0000 7fc4a27fc640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc49006b4d0 0x7fc49006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.640+0000 7fc4a27fc640 1 -- 172.21.15.27:0/2960840405 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4a4089700 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.640+0000 7fc4b1a01640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc49006b4d0 0x7fc49006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.640+0000 7fc4b1a01640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc49006b4d0 0x7fc49006d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80018f0 tx=0x7fc4a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.644+0000 7fc4a27fc640 1 -- 172.21.15.27:0/2960840405 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4a4057fc0 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.826+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4ac0ce450 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.827+0000 7fc4a27fc640 1 -- 172.21.15.27:0/2960840405 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7fc4a405c060 con 0x7fc4ac0d06c0 2024-03-06T07:23:11.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:11.829 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:11.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc49006b4d0 msgr2=0x7fc49006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc49006b4d0 0x7fc49006d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80018f0 tx=0x7fc4a8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ac0d06c0 msgr2=0x7fc4ac15dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ac0d06c0 0x7fc4ac15dfb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c7a0 tx=0x7fc4a400cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 shutdown_connections 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc49006b4d0 0x7fc49006d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ac0d06c0 0x7fc4ac15dfb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4ac0d02e0 0x7fc4ac0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 --2- 172.21.15.27:0/2960840405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4ac0b4260 0x7fc4ac142fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 >> 172.21.15.27:0/2960840405 conn(0x7fc4ac0b36d0 msgr2=0x7fc4ac0cd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 shutdown_connections 2024-03-06T07:23:11.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.829+0000 7fc4b3c65640 1 -- 172.21.15.27:0/2960840405 wait complete. 2024-03-06T07:23:11.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:11.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.949+0000 7f785bcdf640 1 Processor -- start 2024-03-06T07:23:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.950+0000 7f785bcdf640 1 -- start start 2024-03-06T07:23:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.950+0000 7f785bcdf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 0x7f78540b1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.950+0000 7f785bcdf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78540b2cd0 0x7f78540b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.950+0000 7f785bcdf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7854161a40 con 0x7f78540b1fd0 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.950+0000 7f785bcdf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78540d11e0 con 0x7f78540b2cd0 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.950+0000 7f785bcdf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785406c2d0 con 0x7f78540cdcd0 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.951+0000 7f785927a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78540b2cd0 0x7f78540b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.951+0000 7f7859a7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 0x7f78540b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.951+0000 7f785a27c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f78540b1fd0 0x7f78540ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:50276/0 (socket says 172.21.15.27:50276) 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.951+0000 7f785a27c640 1 -- 172.21.15.27:0/2363884019 learned_addr learned my addr 172.21.15.27:0/2363884019 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.951+0000 7f785927a640 1 -- 172.21.15.27:0/2363884019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 msgr2=0x7f78540b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.951+0000 7f785927a640 1 --2- 172.21.15.27:0/2363884019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 0x7f78540b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.951+0000 7f785927a640 1 -- 172.21.15.27:0/2363884019 >> v1:172.21.15.69:6789/0 conn(0x7f78540b1fd0 legacy=0x7f78540ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.951+0000 7f785927a640 1 -- 172.21.15.27:0/2363884019 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78540b2a20 con 0x7f78540b2cd0 2024-03-06T07:23:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.951+0000 7f7859a7b640 1 --2- 172.21.15.27:0/2363884019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 0x7f78540b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:11.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.952+0000 7f785927a640 1 --2- 172.21.15.27:0/2363884019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78540b2cd0 0x7f78540b1a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78500097a0 tx=0x7f785002b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b75333a8917ef171 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.952+0000 7f7858a79640 1 -- 172.21.15.27:0/2363884019 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7850037070 con 0x7f78540b2cd0 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.952+0000 7f7858a79640 1 -- 172.21.15.27:0/2363884019 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7850030710 con 0x7f78540b2cd0 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.952+0000 7f7858a79640 1 -- 172.21.15.27:0/2363884019 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785002be60 con 0x7f78540b2cd0 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.953+0000 7f785bcdf640 1 -- 172.21.15.27:0/2363884019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78540b2cd0 msgr2=0x7f78540b1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.953+0000 7f785bcdf640 1 --2- 172.21.15.27:0/2363884019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78540b2cd0 0x7f78540b1a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78500097a0 tx=0x7f785002b660 comp rx=0 tx=0).stop 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.953+0000 7f785bcdf640 1 -- 172.21.15.27:0/2363884019 shutdown_connections 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.953+0000 7f785bcdf640 1 --2- 172.21.15.27:0/2363884019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78540b2cd0 0x7f78540b1a90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.953+0000 7f785bcdf640 1 --2- 172.21.15.27:0/2363884019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 0x7f78540b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.953+0000 7f785bcdf640 1 -- 172.21.15.27:0/2363884019 >> 172.21.15.27:0/2363884019 conn(0x7f78540b3700 msgr2=0x7f78540b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.953+0000 7f785bcdf640 1 -- 172.21.15.27:0/2363884019 shutdown_connections 2024-03-06T07:23:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.953+0000 7f785bcdf640 1 -- 172.21.15.27:0/2363884019 wait complete. 2024-03-06T07:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.954+0000 7f785bcdf640 1 Processor -- start 2024-03-06T07:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.954+0000 7f785bcdf640 1 -- start start 2024-03-06T07:23:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f785bcdf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78540b1fd0 0x7f78540b6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f785bcdf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78540b2cd0 0x7f7854157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f785bcdf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 0x7f78541f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f785bcdf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78540d0f00 con 0x7f78540b1fd0 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f785bcdf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78540d0c20 con 0x7f78540b2cd0 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f785bcdf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78540cc1c0 con 0x7f78540cdcd0 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f7859a7b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78540b1fd0 0x7f78540b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f7859a7b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78540b1fd0 0x7f78540b6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56840/0 (socket says 172.21.15.27:56840) 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f7859a7b640 1 -- 172.21.15.27:0/2640783278 learned_addr learned my addr 172.21.15.27:0/2640783278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f785a27c640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 0x7f78541f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f785927a640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78540b2cd0 0x7f7854157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f7859a7b640 1 -- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 msgr2=0x7f78541f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f7859a7b640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 0x7f78541f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f7859a7b640 1 -- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78540b2cd0 msgr2=0x7f7854157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f7859a7b640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78540b2cd0 0x7f7854157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f7859a7b640 1 -- 172.21.15.27:0/2640783278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78540ca300 con 0x7f78540b1fd0 2024-03-06T07:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.955+0000 7f7859a7b640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78540b1fd0 0x7f78540b6180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f784400e960 tx=0x7f784400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.956+0000 7f784a7fc640 1 -- 172.21.15.27:0/2640783278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844017070 con 0x7f78540b1fd0 2024-03-06T07:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.956+0000 7f784a7fc640 1 -- 172.21.15.27:0/2640783278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784400ca00 con 0x7f78540b1fd0 2024-03-06T07:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.956+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78541f4bf0 con 0x7f78540b1fd0 2024-03-06T07:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.956+0000 7f784a7fc640 1 -- 172.21.15.27:0/2640783278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784400cdd0 con 0x7f78540b1fd0 2024-03-06T07:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.956+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78541f79f0 con 0x7f78540b1fd0 2024-03-06T07:23:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.957+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785409fcf0 con 0x7f78540b1fd0 2024-03-06T07:23:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.957+0000 7f784a7fc640 1 -- 172.21.15.27:0/2640783278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f784401a070 con 0x7f78540b1fd0 2024-03-06T07:23:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.958+0000 7f784a7fc640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f783806b4d0 0x7f783806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.958+0000 7f784a7fc640 1 -- 172.21.15.27:0/2640783278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f784408a090 con 0x7f78540b1fd0 2024-03-06T07:23:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.958+0000 7f785927a640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f783806b4d0 0x7f783806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.958+0000 7f785927a640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f783806b4d0 0x7f783806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7850034000 tx=0x7f7850008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:11.962+0000 7f784a7fc640 1 -- 172.21.15.27:0/2640783278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78540ca300 con 0x7f78540b1fd0 2024-03-06T07:23:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.144+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f785415e820 con 0x7f78540b1fd0 2024-03-06T07:23:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.145+0000 7f784a7fc640 1 -- 172.21.15.27:0/2640783278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7844058840 con 0x7f78540b1fd0 2024-03-06T07:23:12.146 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f783806b4d0 msgr2=0x7f783806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f783806b4d0 0x7f783806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7850034000 tx=0x7f7850008030 comp rx=0 tx=0).stop 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78540b1fd0 msgr2=0x7f78540b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78540b1fd0 0x7f78540b6180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f784400e960 tx=0x7f784400ee20 comp rx=0 tx=0).stop 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 shutdown_connections 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f783806b4d0 0x7f783806d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78540cdcd0 0x7f78541f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78540b2cd0 0x7f7854157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 --2- 172.21.15.27:0/2640783278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78540b1fd0 0x7f78540b6180 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 >> 172.21.15.27:0/2640783278 conn(0x7f78540b3700 msgr2=0x7f78540b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 shutdown_connections 2024-03-06T07:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.147+0000 7f785bcdf640 1 -- 172.21.15.27:0/2640783278 wait complete. 2024-03-06T07:23:12.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:12.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.269+0000 7f1111c8a640 1 Processor -- start 2024-03-06T07:23:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.269+0000 7f1111c8a640 1 -- start start 2024-03-06T07:23:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f1111c8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c0cdcd0 0x7f110c0b1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f1111c8a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110c161a40 con 0x7f110c0b2cd0 2024-03-06T07:23:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f1111c8a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110c0b2a20 con 0x7f110c06b2c0 2024-03-06T07:23:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f1111c8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110c0d11e0 con 0x7f110c0cdcd0 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f110c0b2cd0 0x7f110c0b1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48272/0 (socket says 172.21.15.27:48272) 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110affd640 1 -- 172.21.15.27:0/4043922976 learned_addr learned my addr 172.21.15.27:0/4043922976 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110b7fe640 1 --2- 172.21.15.27:0/4043922976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c0cdcd0 0x7f110c0b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110a7fc640 1 -- 172.21.15.27:0/4043922976 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167234109 0 0) 0x7f110c161a40 con 0x7f110c0b2cd0 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110a7fc640 1 -- 172.21.15.27:0/4043922976 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f4003520 con 0x7f110c0b2cd0 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110a7fc640 1 -- 172.21.15.27:0/4043922976 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813414120 0 0) 0x7f110c0b2a20 con 0x7f110c06b2c0 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110a7fc640 1 -- 172.21.15.27:0/4043922976 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110c161a40 con 0x7f110c06b2c0 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110b7fe640 1 -- 172.21.15.27:0/4043922976 >> v1:172.21.15.69:6789/0 conn(0x7f110c06b2c0 legacy=0x7f110c06b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110a7fc640 1 -- 172.21.15.27:0/4043922976 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4107328176 0 0) 0x7f10f4003520 con 0x7f110c0b2cd0 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110b7fe640 1 -- 172.21.15.27:0/4043922976 >> v1:172.21.15.27:6789/0 conn(0x7f110c0b2cd0 legacy=0x7f110c0b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.270+0000 7f110b7fe640 1 -- 172.21.15.27:0/4043922976 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c0d0940 con 0x7f110c0cdcd0 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.271+0000 7f110b7fe640 1 --2- 172.21.15.27:0/4043922976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c0cdcd0 0x7f110c0b1550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0097d0 tx=0x7f10fc02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54760335cdc2ecd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.271+0000 7f110a7fc640 1 -- 172.21.15.27:0/4043922976 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc037070 con 0x7f110c0cdcd0 2024-03-06T07:23:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.271+0000 7f110a7fc640 1 -- 172.21.15.27:0/4043922976 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10fc030510 con 0x7f110c0cdcd0 2024-03-06T07:23:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.271+0000 7f110a7fc640 1 -- 172.21.15.27:0/4043922976 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100001670 con 0x7f110c0cdcd0 2024-03-06T07:23:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.272+0000 7f1111c8a640 1 -- 172.21.15.27:0/4043922976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c0cdcd0 msgr2=0x7f110c0b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.272+0000 7f1111c8a640 1 --2- 172.21.15.27:0/4043922976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c0cdcd0 0x7f110c0b1550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0097d0 tx=0x7f10fc02b770 comp rx=0 tx=0).stop 2024-03-06T07:23:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.272+0000 7f1111c8a640 1 -- 172.21.15.27:0/4043922976 shutdown_connections 2024-03-06T07:23:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.272+0000 7f1111c8a640 1 --2- 172.21.15.27:0/4043922976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c0cdcd0 0x7f110c0b1550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.272+0000 7f1111c8a640 1 -- 172.21.15.27:0/4043922976 >> 172.21.15.27:0/4043922976 conn(0x7f110c0b3700 msgr2=0x7f110c0b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.272+0000 7f1111c8a640 1 -- 172.21.15.27:0/4043922976 shutdown_connections 2024-03-06T07:23:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.272+0000 7f1111c8a640 1 -- 172.21.15.27:0/4043922976 wait complete. 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.273+0000 7f1111c8a640 1 Processor -- start 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.273+0000 7f1111c8a640 1 -- start start 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f1111c8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c06b2c0 0x7f110c15df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f1111c8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110c0b2cd0 0x7f110c0c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f1111c8a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c0cdcd0 0x7f110c0b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f1111c8a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c0cc1c0 con 0x7f110c0b2cd0 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f1111c8a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c0d0f00 con 0x7f110c0cdcd0 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f1111c8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110c0cc340 con 0x7f110c06b2c0 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110c0b2cd0 0x7f110c0c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110c0b2cd0 0x7f110c0c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56854/0 (socket says 172.21.15.27:56854) 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110affd640 1 -- 172.21.15.27:0/2459113063 learned_addr learned my addr 172.21.15.27:0/2459113063 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110b7fe640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c06b2c0 0x7f110c15df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110bfff640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c0cdcd0 0x7f110c0b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110affd640 1 -- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c06b2c0 msgr2=0x7f110c15df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110affd640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c06b2c0 0x7f110c15df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110affd640 1 -- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c0cdcd0 msgr2=0x7f110c0b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110affd640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c0cdcd0 0x7f110c0b3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.274+0000 7f110affd640 1 -- 172.21.15.27:0/2459113063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c15e4a0 con 0x7f110c0b2cd0 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.275+0000 7f110b7fe640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c06b2c0 0x7f110c15df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.275+0000 7f110affd640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110c0b2cd0 0x7f110c0c9ba0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f110000c750 tx=0x7f110000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.275+0000 7f10f3fff640 1 -- 172.21.15.27:0/2459113063 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100005ad0 con 0x7f110c0b2cd0 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.275+0000 7f10f3fff640 1 -- 172.21.15.27:0/2459113063 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1100005c70 con 0x7f110c0b2cd0 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.275+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110c0c8de0 con 0x7f110c0b2cd0 2024-03-06T07:23:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.275+0000 7f10f3fff640 1 -- 172.21.15.27:0/2459113063 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100005e60 con 0x7f110c0b2cd0 2024-03-06T07:23:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.275+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110c0c93a0 con 0x7f110c0b2cd0 2024-03-06T07:23:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.276+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110c09fcf0 con 0x7f110c0b2cd0 2024-03-06T07:23:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.277+0000 7f10f3fff640 1 -- 172.21.15.27:0/2459113063 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1100007290 con 0x7f110c0b2cd0 2024-03-06T07:23:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.278+0000 7f10f3fff640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10f806b4d0 0x7f10f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.278+0000 7f10f3fff640 1 -- 172.21.15.27:0/2459113063 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1100089f30 con 0x7f110c0b2cd0 2024-03-06T07:23:12.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.278+0000 7f110b7fe640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10f806b4d0 0x7f10f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.278+0000 7f110b7fe640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10f806b4d0 0x7f10f806d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f10fc034000 tx=0x7f10fc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.282+0000 7f10f3fff640 1 -- 172.21.15.27:0/2459113063 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f110c15e4a0 con 0x7f110c0b2cd0 2024-03-06T07:23:12.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.465+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f110c0d1690 con 0x7f110c0b2cd0 2024-03-06T07:23:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.465+0000 7f10f3fff640 1 -- 172.21.15.27:0/2459113063 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f11000586e0 con 0x7f110c0b2cd0 2024-03-06T07:23:12.467 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10f806b4d0 msgr2=0x7f10f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10f806b4d0 0x7f10f806d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f10fc034000 tx=0x7f10fc008030 comp rx=0 tx=0).stop 2024-03-06T07:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110c0b2cd0 msgr2=0x7f110c0c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110c0b2cd0 0x7f110c0c9ba0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f110000c750 tx=0x7f110000cc10 comp rx=0 tx=0).stop 2024-03-06T07:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 shutdown_connections 2024-03-06T07:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10f806b4d0 0x7f10f806d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f110c0cdcd0 0x7f110c0b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110c0b2cd0 0x7f110c0c9ba0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 --2- 172.21.15.27:0/2459113063 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f110c06b2c0 0x7f110c15df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 >> 172.21.15.27:0/2459113063 conn(0x7f110c0b3700 msgr2=0x7f110c06efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 shutdown_connections 2024-03-06T07:23:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.467+0000 7f1111c8a640 1 -- 172.21.15.27:0/2459113063 wait complete. 2024-03-06T07:23:12.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.587+0000 7f7e75c78640 1 Processor -- start 2024-03-06T07:23:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.587+0000 7f7e75c78640 1 -- start start 2024-03-06T07:23:12.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.588+0000 7f7e75c78640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e70071540 0x7f7e70071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.588+0000 7f7e75c78640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e70071e60 con 0x7f7e700cee80 2024-03-06T07:23:12.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.588+0000 7f7e75c78640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700d4960 con 0x7f7e700b2d30 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.588+0000 7f7e75c78640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e700d3230 con 0x7f7e70071540 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.588+0000 7f7e6ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e700cee80 0x7f7e70168900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48276/0 (socket says 172.21.15.27:48276) 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.588+0000 7f7e6ffff640 1 -- 172.21.15.27:0/164965735 learned_addr learned my addr 172.21.15.27:0/164965735 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.588+0000 7f7e6f7fe640 1 --2- 172.21.15.27:0/164965735 >> v2:172.21.15.69:3300/0 conn(0x7f7e70071540 0x7f7e70071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.588+0000 7f7e6e7fc640 1 -- 172.21.15.27:0/164965735 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125758525 0 0) 0x7f7e70071e60 con 0x7f7e700cee80 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.588+0000 7f7e6e7fc640 1 -- 172.21.15.27:0/164965735 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e58003520 con 0x7f7e700cee80 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.589+0000 7f7e6e7fc640 1 -- 172.21.15.27:0/164965735 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229775569 0 0) 0x7f7e700d4960 con 0x7f7e700b2d30 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.589+0000 7f7e6e7fc640 1 -- 172.21.15.27:0/164965735 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e70071e60 con 0x7f7e700b2d30 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.589+0000 7f7e6e7fc640 1 -- 172.21.15.27:0/164965735 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4184378660 0 0) 0x7f7e58003520 con 0x7f7e700cee80 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.589+0000 7f7e6f7fe640 1 -- 172.21.15.27:0/164965735 >> v1:172.21.15.69:6789/0 conn(0x7f7e700b2d30 legacy=0x7f7e700ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.589+0000 7f7e6f7fe640 1 -- 172.21.15.27:0/164965735 >> v1:172.21.15.27:6789/0 conn(0x7f7e700cee80 legacy=0x7f7e70168900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.589+0000 7f7e6f7fe640 1 -- 172.21.15.27:0/164965735 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e700d36c0 con 0x7f7e70071540 2024-03-06T07:23:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e6f7fe640 1 --2- 172.21.15.27:0/164965735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 0x7f7e70071920 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7e600097a0 tx=0x7f7e6002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ce83db252d8420 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e6e7fc640 1 -- 172.21.15.27:0/164965735 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60038070 con 0x7f7e70071540 2024-03-06T07:23:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e6e7fc640 1 -- 172.21.15.27:0/164965735 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e60031500 con 0x7f7e70071540 2024-03-06T07:23:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e6e7fc640 1 -- 172.21.15.27:0/164965735 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68001670 con 0x7f7e70071540 2024-03-06T07:23:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e75c78640 1 -- 172.21.15.27:0/164965735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 msgr2=0x7f7e70071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e75c78640 1 --2- 172.21.15.27:0/164965735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 0x7f7e70071920 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7e600097a0 tx=0x7f7e6002b820 comp rx=0 tx=0).stop 2024-03-06T07:23:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e75c78640 1 -- 172.21.15.27:0/164965735 shutdown_connections 2024-03-06T07:23:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e75c78640 1 --2- 172.21.15.27:0/164965735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 0x7f7e70071920 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e75c78640 1 -- 172.21.15.27:0/164965735 >> 172.21.15.27:0/164965735 conn(0x7f7e700d3d60 msgr2=0x7f7e700d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e75c78640 1 -- 172.21.15.27:0/164965735 shutdown_connections 2024-03-06T07:23:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.590+0000 7f7e75c78640 1 -- 172.21.15.27:0/164965735 wait complete. 2024-03-06T07:23:12.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.591+0000 7f7e75c78640 1 Processor -- start 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.592+0000 7f7e75c78640 1 -- start start 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.592+0000 7f7e75c78640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 0x7f7e700c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.592+0000 7f7e75c78640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e700b2d30 0x7f7e700cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.592+0000 7f7e75c78640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e700cee80 0x7f7e700cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.592+0000 7f7e75c78640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700d2be0 con 0x7f7e700b2d30 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.592+0000 7f7e75c78640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700d2900 con 0x7f7e70071540 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.592+0000 7f7e75c78640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700d2ec0 con 0x7f7e700cee80 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e700b2d30 0x7f7e700cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e700b2d30 0x7f7e700cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56870/0 (socket says 172.21.15.27:56870) 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6effd640 1 -- 172.21.15.27:0/3270502245 learned_addr learned my addr 172.21.15.27:0/3270502245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6ffff640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e700cee80 0x7f7e700cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6f7fe640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 0x7f7e700c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6effd640 1 -- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e700cee80 msgr2=0x7f7e700cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6effd640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e700cee80 0x7f7e700cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6effd640 1 -- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 msgr2=0x7f7e700c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6effd640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 0x7f7e700c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6effd640 1 -- 172.21.15.27:0/3270502245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e701e9600 con 0x7f7e700b2d30 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6ffff640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e700cee80 0x7f7e700cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6f7fe640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 0x7f7e700c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e6effd640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e700b2d30 0x7f7e700cd010 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c790 tx=0x7f7e6400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e57fff640 1 -- 172.21.15.27:0/3270502245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64003b70 con 0x7f7e700b2d30 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e57fff640 1 -- 172.21.15.27:0/3270502245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e64003d10 con 0x7f7e700b2d30 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e700c7640 con 0x7f7e700b2d30 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e57fff640 1 -- 172.21.15.27:0/3270502245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400aa70 con 0x7f7e700b2d30 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.593+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e700c8660 con 0x7f7e700b2d30 2024-03-06T07:23:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.594+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e700d05c0 con 0x7f7e700b2d30 2024-03-06T07:23:12.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.595+0000 7f7e57fff640 1 -- 172.21.15.27:0/3270502245 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e6400ac10 con 0x7f7e700b2d30 2024-03-06T07:23:12.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.595+0000 7f7e57fff640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.596+0000 7f7e57fff640 1 -- 172.21.15.27:0/3270502245 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e6408a160 con 0x7f7e700b2d30 2024-03-06T07:23:12.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.596+0000 7f7e6f7fe640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.596+0000 7f7e6f7fe640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e60035000 tx=0x7f7e60008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.600+0000 7f7e57fff640 1 -- 172.21.15.27:0/3270502245 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e700d05c0 con 0x7f7e700b2d30 2024-03-06T07:23:12.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.781+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e7004f410 con 0x7f7e700b2d30 2024-03-06T07:23:12.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.781+0000 7f7e57fff640 1 -- 172.21.15.27:0/3270502245 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7e64059070 con 0x7f7e700b2d30 2024-03-06T07:23:12.783 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e5c06b500 msgr2=0x7f7e5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e60035000 tx=0x7f7e60008030 comp rx=0 tx=0).stop 2024-03-06T07:23:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e700b2d30 msgr2=0x7f7e700cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e700b2d30 0x7f7e700cd010 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c790 tx=0x7f7e6400cc50 comp rx=0 tx=0).stop 2024-03-06T07:23:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 shutdown_connections 2024-03-06T07:23:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e700cee80 0x7f7e700cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e700b2d30 0x7f7e700cd010 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 --2- 172.21.15.27:0/3270502245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e70071540 0x7f7e700c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.783+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 >> 172.21.15.27:0/3270502245 conn(0x7f7e700d3d60 msgr2=0x7f7e7015c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.784+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 shutdown_connections 2024-03-06T07:23:12.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.784+0000 7f7e75c78640 1 -- 172.21.15.27:0/3270502245 wait complete. 2024-03-06T07:23:12.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.903+0000 7f7c95dc7640 1 Processor -- start 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.903+0000 7f7c95dc7640 1 -- start start 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c95dc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c9015df00 0x7f7c900d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c95dc7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c900b95a0 0x7f7c900b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c95dc7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c900cea30 con 0x7f7c900b9a30 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c95dc7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c900d2af0 con 0x7f7c900b95a0 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c95dc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c900cf3c0 con 0x7f7c9015df00 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c8effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c900b95a0 0x7f7c900b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c8ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7c900b9a30 0x7f7c900d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48286/0 (socket says 172.21.15.27:48286) 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c8ffff640 1 -- 172.21.15.27:0/415593837 learned_addr learned my addr 172.21.15.27:0/415593837 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c8f7fe640 1 --2- 172.21.15.27:0/415593837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c9015df00 0x7f7c900d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c8effd640 1 -- 172.21.15.27:0/415593837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c9015df00 msgr2=0x7f7c900d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c8effd640 1 --2- 172.21.15.27:0/415593837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c9015df00 0x7f7c900d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c8effd640 1 -- 172.21.15.27:0/415593837 >> v1:172.21.15.27:6789/0 conn(0x7f7c900b9a30 legacy=0x7f7c900d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.904+0000 7f7c8effd640 1 -- 172.21.15.27:0/415593837 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c900cf0e0 con 0x7f7c900b95a0 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.905+0000 7f7c8effd640 1 --2- 172.21.15.27:0/415593837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c900b95a0 0x7f7c900b5700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7c800097a0 tx=0x7f7c8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3c686168e0c055 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.905+0000 7f7c8e7fc640 1 -- 172.21.15.27:0/415593837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80035020 con 0x7f7c900b95a0 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.905+0000 7f7c8e7fc640 1 -- 172.21.15.27:0/415593837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c80031830 con 0x7f7c900b95a0 2024-03-06T07:23:12.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.905+0000 7f7c8e7fc640 1 -- 172.21.15.27:0/415593837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80031c00 con 0x7f7c900b95a0 2024-03-06T07:23:12.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.906+0000 7f7c95dc7640 1 -- 172.21.15.27:0/415593837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c900b95a0 msgr2=0x7f7c900b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.906+0000 7f7c95dc7640 1 --2- 172.21.15.27:0/415593837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c900b95a0 0x7f7c900b5700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7c800097a0 tx=0x7f7c8002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.906+0000 7f7c95dc7640 1 -- 172.21.15.27:0/415593837 shutdown_connections 2024-03-06T07:23:12.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.906+0000 7f7c95dc7640 1 --2- 172.21.15.27:0/415593837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c900b95a0 0x7f7c900b5700 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.906+0000 7f7c95dc7640 1 --2- 172.21.15.27:0/415593837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c9015df00 0x7f7c900d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.906+0000 7f7c95dc7640 1 -- 172.21.15.27:0/415593837 >> 172.21.15.27:0/415593837 conn(0x7f7c90071960 msgr2=0x7f7c90071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.906+0000 7f7c95dc7640 1 -- 172.21.15.27:0/415593837 shutdown_connections 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.906+0000 7f7c95dc7640 1 -- 172.21.15.27:0/415593837 wait complete. 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.907+0000 7f7c95dc7640 1 Processor -- start 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.907+0000 7f7c95dc7640 1 -- start start 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c95dc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c900b95a0 0x7f7c901f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c95dc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c9015df00 0x7f7c900b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c95dc7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c901f8230 0x7f7c9015f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c95dc7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c900cec70 con 0x7f7c9015df00 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c95dc7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c900ce750 con 0x7f7c901f8230 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c95dc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c900cedf0 con 0x7f7c900b95a0 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c9015df00 0x7f7c900b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c9015df00 0x7f7c900b5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56900/0 (socket says 172.21.15.27:56900) 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8effd640 1 -- 172.21.15.27:0/2761204275 learned_addr learned my addr 172.21.15.27:0/2761204275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8f7fe640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c900b95a0 0x7f7c901f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8effd640 1 -- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c900b95a0 msgr2=0x7f7c901f7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8effd640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c900b95a0 0x7f7c901f7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8effd640 1 -- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c901f8230 msgr2=0x7f7c9015f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8ffff640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c901f8230 0x7f7c9015f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8effd640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c901f8230 0x7f7c9015f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8effd640 1 -- 172.21.15.27:0/2761204275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c901f8740 con 0x7f7c9015df00 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8f7fe640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c900b95a0 0x7f7c901f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.908+0000 7f7c8effd640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c9015df00 0x7f7c900b5400 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7c8002bee0 tx=0x7f7c800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.909+0000 7f7c77fff640 1 -- 172.21.15.27:0/2761204275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80041070 con 0x7f7c9015df00 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.909+0000 7f7c77fff640 1 -- 172.21.15.27:0/2761204275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8003b920 con 0x7f7c9015df00 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.909+0000 7f7c77fff640 1 -- 172.21.15.27:0/2761204275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80035040 con 0x7f7c9015df00 2024-03-06T07:23:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.909+0000 7f7c95dc7640 1 -- 172.21.15.27:0/2761204275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c901f8a20 con 0x7f7c9015df00 2024-03-06T07:23:12.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.909+0000 7f7c95dc7640 1 -- 172.21.15.27:0/2761204275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c900b33e0 con 0x7f7c9015df00 2024-03-06T07:23:12.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.910+0000 7f7c77fff640 1 -- 172.21.15.27:0/2761204275 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c8004c060 con 0x7f7c9015df00 2024-03-06T07:23:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.910+0000 7f7c75ffb640 1 -- 172.21.15.27:0/2761204275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c68005290 con 0x7f7c9015df00 2024-03-06T07:23:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.910+0000 7f7c77fff640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c7c06b4d0 0x7f7c7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.911+0000 7f7c77fff640 1 -- 172.21.15.27:0/2761204275 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c800abf90 con 0x7f7c9015df00 2024-03-06T07:23:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.911+0000 7f7c8f7fe640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c7c06b4d0 0x7f7c7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.911+0000 7f7c8f7fe640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c7c06b4d0 0x7f7c7c06d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7c84006fd0 tx=0x7f7c84008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:12.915+0000 7f7c77fff640 1 -- 172.21.15.27:0/2761204275 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c800af030 con 0x7f7c9015df00 2024-03-06T07:23:13.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.101+0000 7f7c75ffb640 1 -- 172.21.15.27:0/2761204275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c68005850 con 0x7f7c9015df00 2024-03-06T07:23:13.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.102+0000 7f7c77fff640 1 -- 172.21.15.27:0/2761204275 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f7c8002f2e0 con 0x7f7c9015df00 2024-03-06T07:23:13.103 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:13.104 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 -- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c7c06b4d0 msgr2=0x7f7c7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c7c06b4d0 0x7f7c7c06d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7c84006fd0 tx=0x7f7c84008040 comp rx=0 tx=0).stop 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 -- 172.21.15.27:0/2761204275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c9015df00 msgr2=0x7f7c900b5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c9015df00 0x7f7c900b5400 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7c8002bee0 tx=0x7f7c800046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 -- 172.21.15.27:0/2761204275 shutdown_connections 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c7c06b4d0 0x7f7c7c06d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c901f8230 0x7f7c9015f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c9015df00 0x7f7c900b5400 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 --2- 172.21.15.27:0/2761204275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c900b95a0 0x7f7c901f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 -- 172.21.15.27:0/2761204275 >> 172.21.15.27:0/2761204275 conn(0x7f7c90071960 msgr2=0x7f7c900d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 -- 172.21.15.27:0/2761204275 shutdown_connections 2024-03-06T07:23:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.104+0000 7f7c75ffb640 1 -- 172.21.15.27:0/2761204275 wait complete. 2024-03-06T07:23:13.121 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:13.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:13.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.231+0000 7f05c9965640 1 Processor -- start 2024-03-06T07:23:13.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.231+0000 7f05c9965640 1 -- start start 2024-03-06T07:23:13.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.231+0000 7f05c9965640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f05c41618e0 0x7f05c406b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.231+0000 7f05c9965640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c4162b60 con 0x7f05c40ce960 2024-03-06T07:23:13.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.231+0000 7f05c9965640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c415f730 con 0x7f05c40ce4f0 2024-03-06T07:23:13.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.231+0000 7f05c9965640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c40cfce0 con 0x7f05c41618e0 2024-03-06T07:23:13.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c37fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f05c40ce960 0x7f05c4162450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48298/0 (socket says 172.21.15.27:48298) 2024-03-06T07:23:13.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c37fe640 1 -- 172.21.15.27:0/1618240276 learned_addr learned my addr 172.21.15.27:0/1618240276 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c2ffd640 1 --2- 172.21.15.27:0/1618240276 >> v2:172.21.15.69:3300/0 conn(0x7f05c41618e0 0x7f05c406b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792708408 0 0) 0x7f05c4162b60 con 0x7f05c40ce960 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ac003520 con 0x7f05c40ce960 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905359069 0 0) 0x7f05c415f730 con 0x7f05c40ce4f0 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05c4162b60 con 0x7f05c40ce4f0 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1473465490 0 0) 0x7f05ac003520 con 0x7f05c40ce960 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 >> v2:172.21.15.69:3300/0 conn(0x7f05c41618e0 msgr2=0x7f05c406b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 --2- 172.21.15.27:0/1618240276 >> v2:172.21.15.69:3300/0 conn(0x7f05c41618e0 0x7f05c406b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 >> v1:172.21.15.69:6789/0 conn(0x7f05c40ce4f0 legacy=0x7f05c4161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c415ebb0 con 0x7f05c40ce960 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f05bc0019f0 con 0x7f05c40ce960 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 >> v1:172.21.15.27:6789/0 conn(0x7f05c40ce960 legacy=0x7f05c4162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05ac003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05ac004800 0x7f05ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05ac0051c0 0x7f05ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac00bfb0 con 0x7f05ac003b90 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac00c160 con 0x7f05ac004800 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.232+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac00c360 con 0x7f05ac0051c0 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.233+0000 7f05c2ffd640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05ac003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.233+0000 7f05c37fe640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05ac0051c0 0x7f05ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.233+0000 7f05c27fc640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05ac004800 0x7f05ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.233+0000 7f05c2ffd640 1 -- 172.21.15.27:0/1618240276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05ac0051c0 msgr2=0x7f05ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.233+0000 7f05c2ffd640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05ac0051c0 0x7f05ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.233+0000 7f05c2ffd640 1 -- 172.21.15.27:0/1618240276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05ac004800 msgr2=0x7f05ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.233+0000 7f05c2ffd640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05ac004800 0x7f05ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.233+0000 7f05c37fe640 1 -- 172.21.15.27:0/1618240276 reap_dead start 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.233+0000 7f05c2ffd640 1 -- 172.21.15.27:0/1618240276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac003520 con 0x7f05ac003b90 2024-03-06T07:23:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c2ffd640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05ac003f90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f05b8009ac0 tx=0x7f05ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef4709f284efa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8039070 con 0x7f05ac003b90 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b802c6d0 con 0x7f05ac003b90 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c1ffb640 1 -- 172.21.15.27:0/1618240276 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8009690 con 0x7f05ac003b90 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c9965640 1 -- 172.21.15.27:0/1618240276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 msgr2=0x7f05ac003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c9965640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05ac003f90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f05b8009ac0 tx=0x7f05ac005830 comp rx=0 tx=0).stop 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c9965640 1 -- 172.21.15.27:0/1618240276 shutdown_connections 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c9965640 1 --2- 172.21.15.27:0/1618240276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05ac003f90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c9965640 1 -- 172.21.15.27:0/1618240276 >> 172.21.15.27:0/1618240276 conn(0x7f05c40b3660 msgr2=0x7f05c40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c9965640 1 -- 172.21.15.27:0/1618240276 shutdown_connections 2024-03-06T07:23:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.234+0000 7f05c9965640 1 -- 172.21.15.27:0/1618240276 wait complete. 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.236+0000 7f05c9965640 1 Processor -- start 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.236+0000 7f05c9965640 1 -- start start 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.236+0000 7f05c9965640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05c40b74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.236+0000 7f05c9965640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c415c4e0 0x7f05c41e9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.236+0000 7f05c9965640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c41e9f90 0x7f05c41e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.236+0000 7f05c9965640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c415f450 con 0x7f05ac003b90 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.236+0000 7f05c9965640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c415f170 con 0x7f05c415c4e0 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.236+0000 7f05c9965640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40cedb0 con 0x7f05c41e9f90 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05c40b74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05c40b74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56932/0 (socket says 172.21.15.27:56932) 2024-03-06T07:23:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c2ffd640 1 -- 172.21.15.27:0/691742466 learned_addr learned my addr 172.21.15.27:0/691742466 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c37fe640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c41e9f90 0x7f05c41e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c27fc640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c415c4e0 0x7f05c41e9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c2ffd640 1 -- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c41e9f90 msgr2=0x7f05c41e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c2ffd640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c41e9f90 0x7f05c41e7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c2ffd640 1 -- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c415c4e0 msgr2=0x7f05c41e9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c2ffd640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c415c4e0 0x7f05c41e9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c2ffd640 1 -- 172.21.15.27:0/691742466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c41ea530 con 0x7f05ac003b90 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c37fe640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c41e9f90 0x7f05c41e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c27fc640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c415c4e0 0x7f05c41e9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c2ffd640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05c40b74b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f05b8037040 tx=0x7f05b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05abfff640 1 -- 172.21.15.27:0/691742466 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8039050 con 0x7f05ac003b90 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05abfff640 1 -- 172.21.15.27:0/691742466 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8045070 con 0x7f05ac003b90 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05abfff640 1 -- 172.21.15.27:0/691742466 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8003a70 con 0x7f05ac003b90 2024-03-06T07:23:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c41ea810 con 0x7f05ac003b90 2024-03-06T07:23:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.237+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c41eadd0 con 0x7f05ac003b90 2024-03-06T07:23:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.238+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c409fc50 con 0x7f05ac003b90 2024-03-06T07:23:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.239+0000 7f05abfff640 1 -- 172.21.15.27:0/691742466 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05b8003c10 con 0x7f05ac003b90 2024-03-06T07:23:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.239+0000 7f05abfff640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05b006b500 0x7f05b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.240+0000 7f05abfff640 1 -- 172.21.15.27:0/691742466 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05b80abcc0 con 0x7f05ac003b90 2024-03-06T07:23:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.240+0000 7f05c27fc640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05b006b500 0x7f05b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.240+0000 7f05c27fc640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05b006b500 0x7f05b006d9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05c40cb890 tx=0x7f05b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.244+0000 7f05abfff640 1 -- 172.21.15.27:0/691742466 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05b804e020 con 0x7f05ac003b90 2024-03-06T07:23:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.484+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05c4072cb0 con 0x7f05ac003b90 2024-03-06T07:23:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.485+0000 7f05abfff640 1 -- 172.21.15.27:0/691742466 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05b807abd0 con 0x7f05ac003b90 2024-03-06T07:23:13.486 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:13.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05b006b500 msgr2=0x7f05b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05b006b500 0x7f05b006d9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05c40cb890 tx=0x7f05b4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 msgr2=0x7f05c40b74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05c40b74b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f05b8037040 tx=0x7f05b8004730 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 shutdown_connections 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f05b006b500 0x7f05b006d9c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05c41e9f90 0x7f05c41e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c415c4e0 0x7f05c41e9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 --2- 172.21.15.27:0/691742466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05ac003b90 0x7f05c40b74b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 >> 172.21.15.27:0/691742466 conn(0x7f05c40b3660 msgr2=0x7f05c40d1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 shutdown_connections 2024-03-06T07:23:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.488+0000 7f05c9965640 1 -- 172.21.15.27:0/691742466 wait complete. 2024-03-06T07:23:13.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:13.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.614+0000 7fcecc45c640 1 Processor -- start 2024-03-06T07:23:13.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.614+0000 7fcecc45c640 1 -- start start 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fcecc45c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcec4071540 0x7fcec4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fcecc45c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec4071e60 con 0x7fcec40b2d30 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fcecc45c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec40d4960 con 0x7fcec40cee80 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fcecc45c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec40d3230 con 0x7fcec4071540 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fcec99f7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcec40b2d30 0x7fcec40ce770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48266/0 (socket says 172.21.15.27:48266) 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fcec99f7640 1 -- 172.21.15.27:0/3320578065 learned_addr learned my addr 172.21.15.27:0/3320578065 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fceca1f8640 1 --2- 172.21.15.27:0/3320578065 >> v2:172.21.15.69:3300/0 conn(0x7fcec4071540 0x7fcec4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fcec91f6640 1 -- 172.21.15.27:0/3320578065 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087553001 0 0) 0x7fcec4071e60 con 0x7fcec40b2d30 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fcec91f6640 1 -- 172.21.15.27:0/3320578065 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb0003520 con 0x7fcec40b2d30 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fceca1f8640 1 -- 172.21.15.27:0/3320578065 >> v1:172.21.15.69:6789/0 conn(0x7fcec40cee80 legacy=0x7fcec4168900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fceca1f8640 1 -- 172.21.15.27:0/3320578065 >> v1:172.21.15.27:6789/0 conn(0x7fcec40b2d30 legacy=0x7fcec40ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.615+0000 7fceca1f8640 1 -- 172.21.15.27:0/3320578065 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec40d36c0 con 0x7fcec4071540 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.616+0000 7fceca1f8640 1 --2- 172.21.15.27:0/3320578065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4071540 0x7fcec4071920 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fceb40097a0 tx=0x7fceb402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69ecb9af4173ee7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.616+0000 7fcec91f6640 1 -- 172.21.15.27:0/3320578065 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4034070 con 0x7fcec4071540 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.616+0000 7fcec91f6640 1 -- 172.21.15.27:0/3320578065 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb4030610 con 0x7fcec4071540 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.616+0000 7fcec91f6640 1 -- 172.21.15.27:0/3320578065 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb40309e0 con 0x7fcec4071540 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.617+0000 7fcecc45c640 1 -- 172.21.15.27:0/3320578065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4071540 msgr2=0x7fcec4071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.617+0000 7fcecc45c640 1 --2- 172.21.15.27:0/3320578065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4071540 0x7fcec4071920 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fceb40097a0 tx=0x7fceb402b960 comp rx=0 tx=0).stop 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.617+0000 7fcecc45c640 1 -- 172.21.15.27:0/3320578065 shutdown_connections 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.617+0000 7fcecc45c640 1 --2- 172.21.15.27:0/3320578065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4071540 0x7fcec4071920 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.617+0000 7fcecc45c640 1 -- 172.21.15.27:0/3320578065 >> 172.21.15.27:0/3320578065 conn(0x7fcec40d3d60 msgr2=0x7fcec40d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.617+0000 7fcecc45c640 1 -- 172.21.15.27:0/3320578065 shutdown_connections 2024-03-06T07:23:13.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.617+0000 7fcecc45c640 1 -- 172.21.15.27:0/3320578065 wait complete. 2024-03-06T07:23:13.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.618+0000 7fcecc45c640 1 Processor -- start 2024-03-06T07:23:13.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.618+0000 7fcecc45c640 1 -- start start 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fcecc45c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcec4071540 0x7fcec41f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fcecc45c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcec40b2d30 0x7fcec41469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fcecc45c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec40cee80 0x7fcec4146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fcecc45c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec40d2be0 con 0x7fcec4071540 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fcecc45c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec40d2900 con 0x7fcec40cee80 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fcecc45c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec40d2ec0 con 0x7fcec40b2d30 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fceca1f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcec4071540 0x7fcec41f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fceca1f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcec4071540 0x7fcec41f3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47714/0 (socket says 172.21.15.27:47714) 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fceca1f8640 1 -- 172.21.15.27:0/262061228 learned_addr learned my addr 172.21.15.27:0/262061228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fcec99f7640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcec40b2d30 0x7fcec41469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fceca9f9640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec40cee80 0x7fcec4146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fceca1f8640 1 -- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcec40b2d30 msgr2=0x7fcec41469b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fceca1f8640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcec40b2d30 0x7fcec41469b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fceca1f8640 1 -- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec40cee80 msgr2=0x7fcec4146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fceca1f8640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec40cee80 0x7fcec4146ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.619+0000 7fceca1f8640 1 -- 172.21.15.27:0/262061228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec41458c0 con 0x7fcec4071540 2024-03-06T07:23:13.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.620+0000 7fceca1f8640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcec4071540 0x7fcec41f3910 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fceb40098c0 tx=0x7fceb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.620+0000 7fcebaffd640 1 -- 172.21.15.27:0/262061228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4034040 con 0x7fcec4071540 2024-03-06T07:23:13.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.620+0000 7fcebaffd640 1 -- 172.21.15.27:0/262061228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb403aaa0 con 0x7fcec4071540 2024-03-06T07:23:13.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.620+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec4145ba0 con 0x7fcec4071540 2024-03-06T07:23:13.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.620+0000 7fcebaffd640 1 -- 172.21.15.27:0/262061228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb40443f0 con 0x7fcec4071540 2024-03-06T07:23:13.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.620+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec4146160 con 0x7fcec4071540 2024-03-06T07:23:13.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.621+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec4072070 con 0x7fcec4071540 2024-03-06T07:23:13.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.621+0000 7fcebaffd640 1 -- 172.21.15.27:0/262061228 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fceb404b060 con 0x7fcec4071540 2024-03-06T07:23:13.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.622+0000 7fcebaffd640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcea806b500 0x7fcea806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.622+0000 7fcebaffd640 1 -- 172.21.15.27:0/262061228 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fceb40abe40 con 0x7fcec4071540 2024-03-06T07:23:13.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.622+0000 7fcec99f7640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcea806b500 0x7fcea806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.623+0000 7fcec99f7640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcea806b500 0x7fcea806d9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcec41f7680 tx=0x7fcec0007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.626+0000 7fcebaffd640 1 -- 172.21.15.27:0/262061228 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fceb403f070 con 0x7fcec4071540 2024-03-06T07:23:13.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.805+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcec404f410 con 0x7fcec4071540 2024-03-06T07:23:13.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.805+0000 7fcebaffd640 1 -- 172.21.15.27:0/262061228 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fceb407a670 con 0x7fcec4071540 2024-03-06T07:23:13.807 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcea806b500 msgr2=0x7fcea806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcea806b500 0x7fcea806d9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcec41f7680 tx=0x7fcec0007d30 comp rx=0 tx=0).stop 2024-03-06T07:23:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcec4071540 msgr2=0x7fcec41f3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcec4071540 0x7fcec41f3910 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fceb40098c0 tx=0x7fceb40046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 shutdown_connections 2024-03-06T07:23:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcea806b500 0x7fcea806d9c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec40cee80 0x7fcec4146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcec40b2d30 0x7fcec41469b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 --2- 172.21.15.27:0/262061228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcec4071540 0x7fcec41f3910 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 >> 172.21.15.27:0/262061228 conn(0x7fcec40d3d60 msgr2=0x7fcec415b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 shutdown_connections 2024-03-06T07:23:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.807+0000 7fcecc45c640 1 -- 172.21.15.27:0/262061228 wait complete. 2024-03-06T07:23:13.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:13.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.928+0000 7f2e6f179640 1 Processor -- start 2024-03-06T07:23:13.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.929+0000 7f2e6f179640 1 -- start start 2024-03-06T07:23:13.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.929+0000 7f2e6f179640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e68166cb0 0x7f2e681690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.929+0000 7f2e6f179640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e68161a40 con 0x7f2e680b2cd0 2024-03-06T07:23:13.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.929+0000 7f2e6f179640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e680b2370 con 0x7f2e680cdcd0 2024-03-06T07:23:13.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.929+0000 7f2e6f179640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e680d11e0 con 0x7f2e68166cb0 2024-03-06T07:23:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.929+0000 7f2e6d716640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e68166cb0 0x7f2e681690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.930+0000 7f2e67fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2e680b2cd0 0x7f2e680b1c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48268/0 (socket says 172.21.15.27:48268) 2024-03-06T07:23:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.930+0000 7f2e67fff640 1 -- 172.21.15.27:0/2425521279 learned_addr learned my addr 172.21.15.27:0/2425521279 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.930+0000 7f2e677fe640 1 -- 172.21.15.27:0/2425521279 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182678265 0 0) 0x7f2e68161a40 con 0x7f2e680b2cd0 2024-03-06T07:23:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.930+0000 7f2e677fe640 1 -- 172.21.15.27:0/2425521279 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e58003520 con 0x7f2e680b2cd0 2024-03-06T07:23:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.930+0000 7f2e6d716640 1 -- 172.21.15.27:0/2425521279 >> v1:172.21.15.69:6789/0 conn(0x7f2e680cdcd0 legacy=0x7f2e680b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.930+0000 7f2e6d716640 1 -- 172.21.15.27:0/2425521279 >> v1:172.21.15.27:6789/0 conn(0x7f2e680b2cd0 legacy=0x7f2e680b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.930+0000 7f2e6d716640 1 -- 172.21.15.27:0/2425521279 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6806c2d0 con 0x7f2e68166cb0 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.931+0000 7f2e6d716640 1 --2- 172.21.15.27:0/2425521279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 0x7f2e681690a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009d90 tx=0x7f2e6002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ec8d7e96ca54e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.931+0000 7f2e677fe640 1 -- 172.21.15.27:0/2425521279 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60039070 con 0x7f2e68166cb0 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.931+0000 7f2e677fe640 1 -- 172.21.15.27:0/2425521279 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6002cc60 con 0x7f2e68166cb0 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.931+0000 7f2e677fe640 1 -- 172.21.15.27:0/2425521279 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60034640 con 0x7f2e68166cb0 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.931+0000 7f2e6f179640 1 -- 172.21.15.27:0/2425521279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 msgr2=0x7f2e681690a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.931+0000 7f2e6f179640 1 --2- 172.21.15.27:0/2425521279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 0x7f2e681690a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009d90 tx=0x7f2e6002beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.931+0000 7f2e6f179640 1 -- 172.21.15.27:0/2425521279 shutdown_connections 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.931+0000 7f2e6f179640 1 --2- 172.21.15.27:0/2425521279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 0x7f2e681690a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.931+0000 7f2e6f179640 1 -- 172.21.15.27:0/2425521279 >> 172.21.15.27:0/2425521279 conn(0x7f2e680b3700 msgr2=0x7f2e680b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.932+0000 7f2e6f179640 1 -- 172.21.15.27:0/2425521279 shutdown_connections 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.932+0000 7f2e6f179640 1 -- 172.21.15.27:0/2425521279 wait complete. 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.933+0000 7f2e6f179640 1 Processor -- start 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.933+0000 7f2e6f179640 1 -- start start 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6f179640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e680b2cd0 0x7f2e68151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6f179640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e680cdcd0 0x7f2e680b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6f179640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 0x7f2e680b6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6f179640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680d0c20 con 0x7f2e68166cb0 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6f179640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680d0940 con 0x7f2e680b2cd0 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6f179640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680d0f00 con 0x7f2e680cdcd0 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6d716640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 0x7f2e680b6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6d716640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 0x7f2e680b6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47722/0 (socket says 172.21.15.27:47722) 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6d716640 1 -- 172.21.15.27:0/2524633849 learned_addr learned my addr 172.21.15.27:0/2524633849 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e67fff640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e680cdcd0 0x7f2e680b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6d716640 1 -- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e680cdcd0 msgr2=0x7f2e680b6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6d716640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e680cdcd0 0x7f2e680b6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6d716640 1 -- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e680b2cd0 msgr2=0x7f2e68151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6d716640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e680b2cd0 0x7f2e68151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6d716640 1 -- 172.21.15.27:0/2524633849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e680c8d70 con 0x7f2e68166cb0 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e67fff640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e680cdcd0 0x7f2e680b6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6d716640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 0x7f2e680b6bd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009d90 tx=0x7f2e60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e657fa640 1 -- 172.21.15.27:0/2524633849 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60042020 con 0x7f2e68166cb0 2024-03-06T07:23:13.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e657fa640 1 -- 172.21.15.27:0/2524633849 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6003a040 con 0x7f2e68166cb0 2024-03-06T07:23:13.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e680c9050 con 0x7f2e68166cb0 2024-03-06T07:23:13.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e657fa640 1 -- 172.21.15.27:0/2524633849 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60039070 con 0x7f2e68166cb0 2024-03-06T07:23:13.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.934+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e680c9610 con 0x7f2e68166cb0 2024-03-06T07:23:13.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.936+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6809fcf0 con 0x7f2e68166cb0 2024-03-06T07:23:13.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.936+0000 7f2e657fa640 1 -- 172.21.15.27:0/2524633849 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e60045020 con 0x7f2e68166cb0 2024-03-06T07:23:13.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.936+0000 7f2e657fa640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e4c06b4d0 0x7f2e4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.936+0000 7f2e657fa640 1 -- 172.21.15.27:0/2524633849 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e600ac540 con 0x7f2e68166cb0 2024-03-06T07:23:13.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.937+0000 7f2e6cf15640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e4c06b4d0 0x7f2e4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.937+0000 7f2e6cf15640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e4c06b4d0 0x7f2e4c06d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c000f60 tx=0x7f2e5c0079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:13.941+0000 7f2e657fa640 1 -- 172.21.15.27:0/2524633849 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e6004c030 con 0x7f2e68166cb0 2024-03-06T07:23:14.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.123+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2e68166a90 con 0x7f2e68166cb0 2024-03-06T07:23:14.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.123+0000 7f2e657fa640 1 -- 172.21.15.27:0/2524633849 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2e6004e300 con 0x7f2e68166cb0 2024-03-06T07:23:14.125 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e4c06b4d0 msgr2=0x7f2e4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e4c06b4d0 0x7f2e4c06d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c000f60 tx=0x7f2e5c0079d0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 msgr2=0x7f2e680b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 0x7f2e680b6bd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009d90 tx=0x7f2e60004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 shutdown_connections 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e4c06b4d0 0x7f2e4c06d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68166cb0 0x7f2e680b6bd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e680cdcd0 0x7f2e680b6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 --2- 172.21.15.27:0/2524633849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e680b2cd0 0x7f2e68151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 >> 172.21.15.27:0/2524633849 conn(0x7f2e680b3700 msgr2=0x7f2e680ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 shutdown_connections 2024-03-06T07:23:14.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.125+0000 7f2e6f179640 1 -- 172.21.15.27:0/2524633849 wait complete. 2024-03-06T07:23:14.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:14.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.244+0000 7f46364b5640 1 Processor -- start 2024-03-06T07:23:14.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.244+0000 7f46364b5640 1 -- start start 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f46364b5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f46300cebb0 0x7f46300d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f46364b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f463006ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f46364b5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46300d1090 con 0x7f4630161c80 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f46364b5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f463006b0a0 con 0x7f46300cebb0 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f46364b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463006b380 con 0x7f46301627f0 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f4634a52640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f463006ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f462ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f46300cebb0 0x7f46300d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f4634a52640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f463006ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:33000/0 (socket says 172.21.15.27:33000) 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f4634a52640 1 -- 172.21.15.27:0/999517803 learned_addr learned my addr 172.21.15.27:0/999517803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f4634a52640 1 -- 172.21.15.27:0/999517803 >> v2:172.21.15.69:3300/0 conn(0x7f46300cebb0 msgr2=0x7f46300d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f4634a52640 1 --2- 172.21.15.27:0/999517803 >> v2:172.21.15.69:3300/0 conn(0x7f46300cebb0 0x7f46300d0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f4634a52640 1 -- 172.21.15.27:0/999517803 >> v1:172.21.15.69:6789/0 conn(0x7f4630161c80 legacy=0x7f46301620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.245+0000 7f4634a52640 1 -- 172.21.15.27:0/999517803 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463006b660 con 0x7f46301627f0 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.246+0000 7f4634a52640 1 --2- 172.21.15.27:0/999517803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f463006ef80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4628009e50 tx=0x7f462802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c781e0352162f05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.246+0000 7f462effd640 1 -- 172.21.15.27:0/999517803 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462802cb00 con 0x7f46301627f0 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.246+0000 7f462effd640 1 -- 172.21.15.27:0/999517803 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462802cca0 con 0x7f46301627f0 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.246+0000 7f462effd640 1 -- 172.21.15.27:0/999517803 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628034610 con 0x7f46301627f0 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.247+0000 7f46364b5640 1 -- 172.21.15.27:0/999517803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 msgr2=0x7f463006ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.247+0000 7f46364b5640 1 --2- 172.21.15.27:0/999517803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f463006ef80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4628009e50 tx=0x7f462802bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.247+0000 7f46364b5640 1 -- 172.21.15.27:0/999517803 shutdown_connections 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.247+0000 7f46364b5640 1 --2- 172.21.15.27:0/999517803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f463006ef80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.247+0000 7f46364b5640 1 --2- 172.21.15.27:0/999517803 >> v2:172.21.15.69:3300/0 conn(0x7f46300cebb0 0x7f46300d0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.247+0000 7f46364b5640 1 -- 172.21.15.27:0/999517803 >> 172.21.15.27:0/999517803 conn(0x7f46300b35d0 msgr2=0x7f46300b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.247+0000 7f46364b5640 1 -- 172.21.15.27:0/999517803 shutdown_connections 2024-03-06T07:23:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.247+0000 7f46364b5640 1 -- 172.21.15.27:0/999517803 wait complete. 2024-03-06T07:23:14.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.248+0000 7f46364b5640 1 Processor -- start 2024-03-06T07:23:14.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.248+0000 7f46364b5640 1 -- start start 2024-03-06T07:23:14.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f46364b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cebb0 0x7f46300b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f46364b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4630161c80 0x7f46300c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f46364b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f4630151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f46364b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463006f8d0 con 0x7f4630161c80 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f46364b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463006f750 con 0x7f46300cebb0 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f46364b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463006fa50 con 0x7f46301627f0 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f462f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4630161c80 0x7f46300c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f462f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4630161c80 0x7f46300c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47730/0 (socket says 172.21.15.27:47730) 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f462f7fe640 1 -- 172.21.15.27:0/89882546 learned_addr learned my addr 172.21.15.27:0/89882546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f4634a52640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f4630151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.249+0000 7f462ffff640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cebb0 0x7f46300b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f462f7fe640 1 -- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 msgr2=0x7f4630151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f462f7fe640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f4630151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f462f7fe640 1 -- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cebb0 msgr2=0x7f46300b6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f462f7fe640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cebb0 0x7f46300b6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f462f7fe640 1 -- 172.21.15.27:0/89882546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300c7bd0 con 0x7f4630161c80 2024-03-06T07:23:14.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f462f7fe640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4630161c80 0x7f46300c7690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f461800c430 tx=0x7f461800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f462cff9640 1 -- 172.21.15.27:0/89882546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46180052d0 con 0x7f4630161c80 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f462cff9640 1 -- 172.21.15.27:0/89882546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4618005490 con 0x7f4630161c80 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46300b5690 con 0x7f4630161c80 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f462cff9640 1 -- 172.21.15.27:0/89882546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618007260 con 0x7f4630161c80 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.250+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46300b5c50 con 0x7f4630161c80 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.251+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4630068700 con 0x7f4630161c80 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.251+0000 7f462cff9640 1 -- 172.21.15.27:0/89882546 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4618007400 con 0x7f4630161c80 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.252+0000 7f462cff9640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f461006b4d0 0x7f461006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.252+0000 7f462cff9640 1 -- 172.21.15.27:0/89882546 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4618089640 con 0x7f4630161c80 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.252+0000 7f462ffff640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f461006b4d0 0x7f461006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.252+0000 7f462ffff640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f461006b4d0 0x7f461006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f46300c6f20 tx=0x7f4620009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.256+0000 7f462cff9640 1 -- 172.21.15.27:0/89882546 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4618057e70 con 0x7f4630161c80 2024-03-06T07:23:14.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.438+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46300b7a40 con 0x7f4630161c80 2024-03-06T07:23:14.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.439+0000 7f462cff9640 1 -- 172.21.15.27:0/89882546 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f461805bf10 con 0x7f4630161c80 2024-03-06T07:23:14.440 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:14.440 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.440+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f461006b4d0 msgr2=0x7f461006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.440+0000 7f46364b5640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f461006b4d0 0x7f461006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f46300c6f20 tx=0x7f4620009190 comp rx=0 tx=0).stop 2024-03-06T07:23:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.440+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4630161c80 msgr2=0x7f46300c7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.440+0000 7f46364b5640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4630161c80 0x7f46300c7690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f461800c430 tx=0x7f461800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.440+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 shutdown_connections 2024-03-06T07:23:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.440+0000 7f46364b5640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f461006b4d0 0x7f461006d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.440+0000 7f46364b5640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46301627f0 0x7f4630151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.440+0000 7f46364b5640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4630161c80 0x7f46300c7690 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.440+0000 7f46364b5640 1 --2- 172.21.15.27:0/89882546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cebb0 0x7f46300b6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.441+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 >> 172.21.15.27:0/89882546 conn(0x7f46300b35d0 msgr2=0x7f463015cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.441+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 shutdown_connections 2024-03-06T07:23:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.441+0000 7f46364b5640 1 -- 172.21.15.27:0/89882546 wait complete. 2024-03-06T07:23:14.453 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:14.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:14.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.564+0000 7f5756177640 1 Processor -- start 2024-03-06T07:23:14.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.564+0000 7f5756177640 1 -- start start 2024-03-06T07:23:14.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.565+0000 7f5756177640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f575015dd90 0x7f57500ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.565+0000 7f5756177640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f57500d29d0 0x7f57500d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.565+0000 7f5756177640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57500d3740 con 0x7f57500d4720 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.565+0000 7f5756177640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57500d3400 con 0x7f57500d29d0 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.565+0000 7f5756177640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57500d3120 con 0x7f575015dd90 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.565+0000 7f574ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f57500d4720 0x7f57500b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48278/0 (socket says 172.21.15.27:48278) 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.565+0000 7f574ffff640 1 -- 172.21.15.27:0/32780185 learned_addr learned my addr 172.21.15.27:0/32780185 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.565+0000 7f574f7fe640 1 --2- 172.21.15.27:0/32780185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f575015dd90 0x7f57500ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.566+0000 7f574effd640 1 --2- 172.21.15.27:0/32780185 >> v2:172.21.15.69:3300/0 conn(0x7f57500d29d0 0x7f57500d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.566+0000 7f574e7fc640 1 -- 172.21.15.27:0/32780185 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376414752 0 0) 0x7f57500d3740 con 0x7f57500d4720 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.566+0000 7f574e7fc640 1 -- 172.21.15.27:0/32780185 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c003520 con 0x7f57500d4720 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.566+0000 7f574e7fc640 1 -- 172.21.15.27:0/32780185 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1498687435 0 0) 0x7f573c003520 con 0x7f57500d4720 2024-03-06T07:23:14.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.566+0000 7f574f7fe640 1 -- 172.21.15.27:0/32780185 >> v2:172.21.15.69:3300/0 conn(0x7f57500d29d0 msgr2=0x7f57500d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.566+0000 7f574f7fe640 1 --2- 172.21.15.27:0/32780185 >> v2:172.21.15.69:3300/0 conn(0x7f57500d29d0 0x7f57500d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.566+0000 7f574f7fe640 1 -- 172.21.15.27:0/32780185 >> v1:172.21.15.27:6789/0 conn(0x7f57500d4720 legacy=0x7f57500b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.566+0000 7f574f7fe640 1 -- 172.21.15.27:0/32780185 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57500d39e0 con 0x7f575015dd90 2024-03-06T07:23:14.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f574f7fe640 1 --2- 172.21.15.27:0/32780185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f575015dd90 0x7f57500ceea0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f57380097a0 tx=0x7f573802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cae9f0615da4c150 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f574e7fc640 1 -- 172.21.15.27:0/32780185 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738038070 con 0x7f575015dd90 2024-03-06T07:23:14.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f574e7fc640 1 -- 172.21.15.27:0/32780185 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5738031500 con 0x7f575015dd90 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f574e7fc640 1 -- 172.21.15.27:0/32780185 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57480019f0 con 0x7f575015dd90 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f5756177640 1 -- 172.21.15.27:0/32780185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f575015dd90 msgr2=0x7f57500ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f5756177640 1 --2- 172.21.15.27:0/32780185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f575015dd90 0x7f57500ceea0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f57380097a0 tx=0x7f573802b820 comp rx=0 tx=0).stop 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f5756177640 1 -- 172.21.15.27:0/32780185 shutdown_connections 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f5756177640 1 --2- 172.21.15.27:0/32780185 >> v2:172.21.15.69:3300/0 conn(0x7f57500d29d0 0x7f57500d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f5756177640 1 --2- 172.21.15.27:0/32780185 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f575015dd90 0x7f57500ceea0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f5756177640 1 -- 172.21.15.27:0/32780185 >> 172.21.15.27:0/32780185 conn(0x7f5750071960 msgr2=0x7f5750071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f5756177640 1 -- 172.21.15.27:0/32780185 shutdown_connections 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.567+0000 7f5756177640 1 -- 172.21.15.27:0/32780185 wait complete. 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.569+0000 7f5756177640 1 Processor -- start 2024-03-06T07:23:14.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.569+0000 7f5756177640 1 -- start start 2024-03-06T07:23:14.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5756177640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57500d29d0 0x7f57500ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5756177640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57500d4720 0x7f575015cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5756177640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575015dd90 0x7f57500aa890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5756177640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57500d2df0 con 0x7f575015dd90 2024-03-06T07:23:14.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5756177640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575015e170 con 0x7f57500d29d0 2024-03-06T07:23:14.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5756177640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57500b7420 con 0x7f57500d4720 2024-03-06T07:23:14.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575015dd90 0x7f57500aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575015dd90 0x7f57500aa890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47740/0 (socket says 172.21.15.27:47740) 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574ffff640 1 -- 172.21.15.27:0/3730881661 learned_addr learned my addr 172.21.15.27:0/3730881661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574effd640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57500d4720 0x7f575015cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574f7fe640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57500d29d0 0x7f57500ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574ffff640 1 -- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57500d4720 msgr2=0x7f575015cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574ffff640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57500d4720 0x7f575015cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574ffff640 1 -- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57500d29d0 msgr2=0x7f57500ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574ffff640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57500d29d0 0x7f57500ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574ffff640 1 -- 172.21.15.27:0/3730881661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57501f7ea0 con 0x7f575015dd90 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f574ffff640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575015dd90 0x7f57500aa890 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5748001890 tx=0x7f574800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5747fff640 1 -- 172.21.15.27:0/3730881661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748012400 con 0x7f575015dd90 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5747fff640 1 -- 172.21.15.27:0/3730881661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57480125c0 con 0x7f575015dd90 2024-03-06T07:23:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5747fff640 1 -- 172.21.15.27:0/3730881661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57480129b0 con 0x7f575015dd90 2024-03-06T07:23:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.570+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57501f7320 con 0x7f575015dd90 2024-03-06T07:23:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.571+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57501f78e0 con 0x7f575015dd90 2024-03-06T07:23:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.571+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5750072a30 con 0x7f575015dd90 2024-03-06T07:23:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.572+0000 7f5747fff640 1 -- 172.21.15.27:0/3730881661 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5748012b50 con 0x7f575015dd90 2024-03-06T07:23:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.573+0000 7f5747fff640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f573006b4d0 0x7f573006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.573+0000 7f5747fff640 1 -- 172.21.15.27:0/3730881661 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f574808a960 con 0x7f575015dd90 2024-03-06T07:23:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.573+0000 7f574f7fe640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f573006b4d0 0x7f573006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.573+0000 7f574f7fe640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f573006b4d0 0x7f573006d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5738035000 tx=0x7f5738008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.577+0000 7f5747fff640 1 -- 172.21.15.27:0/3730881661 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5748059190 con 0x7f575015dd90 2024-03-06T07:23:14.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.757+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f57501f3a40 con 0x7f575015dd90 2024-03-06T07:23:14.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.758+0000 7f5747fff640 1 -- 172.21.15.27:0/3730881661 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f574805d230 con 0x7f575015dd90 2024-03-06T07:23:14.759 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f573006b4d0 msgr2=0x7f573006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f573006b4d0 0x7f573006d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5738035000 tx=0x7f5738008030 comp rx=0 tx=0).stop 2024-03-06T07:23:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575015dd90 msgr2=0x7f57500aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575015dd90 0x7f57500aa890 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5748001890 tx=0x7f574800a430 comp rx=0 tx=0).stop 2024-03-06T07:23:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 shutdown_connections 2024-03-06T07:23:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f573006b4d0 0x7f573006d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f575015dd90 0x7f57500aa890 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57500d4720 0x7f575015cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 --2- 172.21.15.27:0/3730881661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57500d29d0 0x7f57500ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 >> 172.21.15.27:0/3730881661 conn(0x7f5750071960 msgr2=0x7f57500b4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 shutdown_connections 2024-03-06T07:23:14.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.760+0000 7f5756177640 1 -- 172.21.15.27:0/3730881661 wait complete. 2024-03-06T07:23:14.772 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:14.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.882+0000 7f348dd3e640 1 Processor -- start 2024-03-06T07:23:14.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.882+0000 7f348dd3e640 1 -- start start 2024-03-06T07:23:14.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.882+0000 7f348dd3e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34880b2130 0x7f34880d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.882+0000 7f348dd3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2d50 0x7f34880d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.882+0000 7f348dd3e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f34880b2800 0x7f34880cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.882+0000 7f348dd3e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34880cf490 con 0x7f34880b2130 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.882+0000 7f348dd3e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34880cf9c0 con 0x7f34880b2800 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.882+0000 7f348dd3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34880cfca0 con 0x7f34880b2d50 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f34877fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34880b2130 0x7f34880d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f34877fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34880b2130 0x7f34880d0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47748/0 (socket says 172.21.15.27:47748) 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f34877fe640 1 -- 172.21.15.27:0/4162721659 learned_addr learned my addr 172.21.15.27:0/4162721659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f3486ffd640 1 --2- 172.21.15.27:0/4162721659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2d50 0x7f34880d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f3487fff640 1 --2- 172.21.15.27:0/4162721659 >> v2:172.21.15.69:3300/0 conn(0x7f34880b2800 0x7f34880cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f34877fe640 1 -- 172.21.15.27:0/4162721659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2d50 msgr2=0x7f34880d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f34877fe640 1 --2- 172.21.15.27:0/4162721659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2d50 0x7f34880d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f34877fe640 1 -- 172.21.15.27:0/4162721659 >> v2:172.21.15.69:3300/0 conn(0x7f34880b2800 msgr2=0x7f34880cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f34877fe640 1 --2- 172.21.15.27:0/4162721659 >> v2:172.21.15.69:3300/0 conn(0x7f34880b2800 0x7f34880cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.883+0000 7f34877fe640 1 -- 172.21.15.27:0/4162721659 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34880cec90 con 0x7f34880b2130 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f34877fe640 1 --2- 172.21.15.27:0/4162721659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 0x7f34880d0120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f348806ea10 tx=0x7f347002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f762840d6c10e44b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f34867fc640 1 -- 172.21.15.27:0/4162721659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470035020 con 0x7f34880b2130 2024-03-06T07:23:14.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f34867fc640 1 -- 172.21.15.27:0/4162721659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3470031870 con 0x7f34880b2130 2024-03-06T07:23:14.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f34867fc640 1 -- 172.21.15.27:0/4162721659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470031c40 con 0x7f34880b2130 2024-03-06T07:23:14.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f348dd3e640 1 -- 172.21.15.27:0/4162721659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 msgr2=0x7f34880d0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f348dd3e640 1 --2- 172.21.15.27:0/4162721659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 0x7f34880d0120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f348806ea10 tx=0x7f347002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f348dd3e640 1 -- 172.21.15.27:0/4162721659 shutdown_connections 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f348dd3e640 1 --2- 172.21.15.27:0/4162721659 >> v2:172.21.15.69:3300/0 conn(0x7f34880b2800 0x7f34880cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f348dd3e640 1 --2- 172.21.15.27:0/4162721659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2d50 0x7f34880d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f348dd3e640 1 --2- 172.21.15.27:0/4162721659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 0x7f34880d0120 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.884+0000 7f348dd3e640 1 -- 172.21.15.27:0/4162721659 >> 172.21.15.27:0/4162721659 conn(0x7f34880b36d0 msgr2=0x7f34880b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.885+0000 7f348dd3e640 1 -- 172.21.15.27:0/4162721659 shutdown_connections 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.885+0000 7f348dd3e640 1 -- 172.21.15.27:0/4162721659 wait complete. 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.886+0000 7f348dd3e640 1 Processor -- start 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.886+0000 7f348dd3e640 1 -- start start 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f348dd3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 0x7f3488155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f348dd3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2800 0x7f34880ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f348dd3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34880b2d50 0x7f3488158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f348dd3e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34880cd9b0 con 0x7f34880b2130 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f348dd3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34880b2510 con 0x7f34880b2d50 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f348dd3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34880cdb30 con 0x7f34880b2800 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f34877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 0x7f3488155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f34877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 0x7f3488155110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47750/0 (socket says 172.21.15.27:47750) 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f34877fe640 1 -- 172.21.15.27:0/2196571702 learned_addr learned my addr 172.21.15.27:0/2196571702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f3486ffd640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2800 0x7f34880ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f3487fff640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34880b2d50 0x7f3488158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f34877fe640 1 -- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2800 msgr2=0x7f34880ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f34877fe640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2800 0x7f34880ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f34877fe640 1 -- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34880b2d50 msgr2=0x7f3488158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f34877fe640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34880b2d50 0x7f3488158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f34877fe640 1 -- 172.21.15.27:0/2196571702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34880b4860 con 0x7f34880b2130 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f3487fff640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34880b2d50 0x7f3488158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:14.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f34877fe640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 0x7f3488155110 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f347002b950 tx=0x7f34700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.887+0000 7f347ffff640 1 -- 172.21.15.27:0/2196571702 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347003ad60 con 0x7f34880b2130 2024-03-06T07:23:14.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.888+0000 7f347ffff640 1 -- 172.21.15.27:0/2196571702 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34700443f0 con 0x7f34880b2130 2024-03-06T07:23:14.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.888+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34880caf30 con 0x7f34880b2130 2024-03-06T07:23:14.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.888+0000 7f347ffff640 1 -- 172.21.15.27:0/2196571702 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700447e0 con 0x7f34880b2130 2024-03-06T07:23:14.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.888+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34880cb4f0 con 0x7f34880b2130 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.889+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348809fcc0 con 0x7f34880b2130 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.889+0000 7f347ffff640 1 -- 172.21.15.27:0/2196571702 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3470038020 con 0x7f34880b2130 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.890+0000 7f347ffff640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f346806b500 0x7f346806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.890+0000 7f347ffff640 1 -- 172.21.15.27:0/2196571702 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34700ac260 con 0x7f34880b2130 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.890+0000 7f3486ffd640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f346806b500 0x7f346806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.890+0000 7f3486ffd640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f346806b500 0x7f346806d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f348815f180 tx=0x7f3478008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:14.894+0000 7f347ffff640 1 -- 172.21.15.27:0/2196571702 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f347003f070 con 0x7f34880b2130 2024-03-06T07:23:15.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.074+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f348804f410 con 0x7f34880b2130 2024-03-06T07:23:15.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.075+0000 7f347ffff640 1 -- 172.21.15.27:0/2196571702 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f347007aa90 con 0x7f34880b2130 2024-03-06T07:23:15.076 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:15.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f346806b500 msgr2=0x7f346806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f346806b500 0x7f346806d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f348815f180 tx=0x7f3478008040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 msgr2=0x7f3488155110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 0x7f3488155110 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f347002b950 tx=0x7f34700046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 shutdown_connections 2024-03-06T07:23:15.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f346806b500 0x7f346806d9c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34880b2d50 0x7f3488158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34880b2800 0x7f34880ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 --2- 172.21.15.27:0/2196571702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b2130 0x7f3488155110 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 >> 172.21.15.27:0/2196571702 conn(0x7f34880b36d0 msgr2=0x7f34880af510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 shutdown_connections 2024-03-06T07:23:15.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.077+0000 7f348dd3e640 1 -- 172.21.15.27:0/2196571702 wait complete. 2024-03-06T07:23:15.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:15.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.199+0000 7f40432f4640 1 Processor -- start 2024-03-06T07:23:15.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.199+0000 7f40432f4640 1 -- start start 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f40432f4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f403c0c6e40 0x7f403c166b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f40432f4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0cf120 con 0x7f403c15f500 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f40432f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0cd310 con 0x7f403c0cf740 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f40432f4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403c15f8e0 con 0x7f403c0c6e40 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f4041891640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f403c0c6e40 0x7f403c166b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f4041090640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f403c15f500 0x7f403c0ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48280/0 (socket says 172.21.15.27:48280) 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f4041090640 1 -- 172.21.15.27:0/1645354385 learned_addr learned my addr 172.21.15.27:0/1645354385 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f4041891640 1 -- 172.21.15.27:0/1645354385 >> v1:172.21.15.69:6789/0 conn(0x7f403c0cf740 legacy=0x7f403c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f4041891640 1 -- 172.21.15.27:0/1645354385 >> v1:172.21.15.27:6789/0 conn(0x7f403c15f500 legacy=0x7f403c0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f4041891640 1 -- 172.21.15.27:0/1645354385 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c0cd5b0 con 0x7f403c0c6e40 2024-03-06T07:23:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.200+0000 7f4033fff640 1 -- 172.21.15.27:0/1645354385 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209043744 0 0) 0x7f403c0cf120 con 0x7f403c15f500 2024-03-06T07:23:15.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.201+0000 7f4041891640 1 --2- 172.21.15.27:0/1645354385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 0x7f403c166b30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4034009d90 tx=0x7f403402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0bc5a7618c99cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.201+0000 7f4033fff640 1 -- 172.21.15.27:0/1645354385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034039070 con 0x7f403c0c6e40 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.201+0000 7f4033fff640 1 -- 172.21.15.27:0/1645354385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403402cbd0 con 0x7f403c0c6e40 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.201+0000 7f4033fff640 1 -- 172.21.15.27:0/1645354385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034034440 con 0x7f403c0c6e40 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.202+0000 7f40432f4640 1 -- 172.21.15.27:0/1645354385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 msgr2=0x7f403c166b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.202+0000 7f40432f4640 1 --2- 172.21.15.27:0/1645354385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 0x7f403c166b30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4034009d90 tx=0x7f403402be20 comp rx=0 tx=0).stop 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.202+0000 7f40432f4640 1 -- 172.21.15.27:0/1645354385 shutdown_connections 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.202+0000 7f40432f4640 1 --2- 172.21.15.27:0/1645354385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 0x7f403c166b30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.202+0000 7f40432f4640 1 -- 172.21.15.27:0/1645354385 >> 172.21.15.27:0/1645354385 conn(0x7f403c0b3700 msgr2=0x7f403c0b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.202+0000 7f40432f4640 1 -- 172.21.15.27:0/1645354385 shutdown_connections 2024-03-06T07:23:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.202+0000 7f40432f4640 1 -- 172.21.15.27:0/1645354385 wait complete. 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.203+0000 7f40432f4640 1 Processor -- start 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.203+0000 7f40432f4640 1 -- start start 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f40432f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 0x7f403c0cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f40432f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f403c0cf740 0x7f403c154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f40432f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f403c15f500 0x7f403c154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f40432f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0c59a0 con 0x7f403c0c6e40 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f40432f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0c56c0 con 0x7f403c0cf740 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f40432f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0b43d0 con 0x7f403c15f500 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f4041090640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 0x7f403c0cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f4041090640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 0x7f403c0cb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47762/0 (socket says 172.21.15.27:47762) 2024-03-06T07:23:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f4041090640 1 -- 172.21.15.27:0/1485468350 learned_addr learned my addr 172.21.15.27:0/1485468350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f4041090640 1 -- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f403c15f500 msgr2=0x7f403c154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f4041090640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f403c15f500 0x7f403c154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f4041090640 1 -- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f403c0cf740 msgr2=0x7f403c154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f4041090640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f403c0cf740 0x7f403c154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f4041090640 1 -- 172.21.15.27:0/1485468350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c153160 con 0x7f403c0c6e40 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.204+0000 7f4041090640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 0x7f403c0cb410 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f403800c500 tx=0x7f403800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.205+0000 7f4031ffb640 1 -- 172.21.15.27:0/1485468350 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038018070 con 0x7f403c0c6e40 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.205+0000 7f4031ffb640 1 -- 172.21.15.27:0/1485468350 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4038007580 con 0x7f403c0c6e40 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.205+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403c153440 con 0x7f403c0c6e40 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.205+0000 7f4031ffb640 1 -- 172.21.15.27:0/1485468350 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038004a20 con 0x7f403c0c6e40 2024-03-06T07:23:15.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.205+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c153a00 con 0x7f403c0c6e40 2024-03-06T07:23:15.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.206+0000 7f4031ffb640 1 -- 172.21.15.27:0/1485468350 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f403801b070 con 0x7f403c0c6e40 2024-03-06T07:23:15.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.206+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403c072d60 con 0x7f403c0c6e40 2024-03-06T07:23:15.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.207+0000 7f4031ffb640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f402006b4d0 0x7f402006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.207+0000 7f4031ffb640 1 -- 172.21.15.27:0/1485468350 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4038089480 con 0x7f403c0c6e40 2024-03-06T07:23:15.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.207+0000 7f404088f640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f402006b4d0 0x7f402006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.207+0000 7f404088f640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f402006b4d0 0x7f402006d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f403c15ec20 tx=0x7f402c0079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.211+0000 7f4031ffb640 1 -- 172.21.15.27:0/1485468350 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f403c072d60 con 0x7f403c0c6e40 2024-03-06T07:23:15.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.393+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f403c09fcf0 con 0x7f403c0c6e40 2024-03-06T07:23:15.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.393+0000 7f4031ffb640 1 -- 172.21.15.27:0/1485468350 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4038058390 con 0x7f403c0c6e40 2024-03-06T07:23:15.395 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:15.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.395+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f402006b4d0 msgr2=0x7f402006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.395+0000 7f40432f4640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f402006b4d0 0x7f402006d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f403c15ec20 tx=0x7f402c0079b0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.395+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 msgr2=0x7f403c0cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.395+0000 7f40432f4640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 0x7f403c0cb410 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f403800c500 tx=0x7f403800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.395+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 shutdown_connections 2024-03-06T07:23:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.395+0000 7f40432f4640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f402006b4d0 0x7f402006d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.395+0000 7f40432f4640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f403c15f500 0x7f403c154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.395+0000 7f40432f4640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f403c0cf740 0x7f403c154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.396+0000 7f40432f4640 1 --2- 172.21.15.27:0/1485468350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f403c0c6e40 0x7f403c0cb410 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.396+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 >> 172.21.15.27:0/1485468350 conn(0x7f403c0b3700 msgr2=0x7f403c0cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.396+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 shutdown_connections 2024-03-06T07:23:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.396+0000 7f40432f4640 1 -- 172.21.15.27:0/1485468350 wait complete. 2024-03-06T07:23:15.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:15.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.515+0000 7f9f36466640 1 Processor -- start 2024-03-06T07:23:15.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.515+0000 7f9f36466640 1 -- start start 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.515+0000 7f9f36466640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d02e0 0x7f9f300cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.515+0000 7f9f36466640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f300b4260 0x7f9f300b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.515+0000 7f9f36466640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9f300d06c0 0x7f9f300b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.515+0000 7f9f36466640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f300b0b80 con 0x7f9f300b4260 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.515+0000 7f9f36466640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f300aeec0 con 0x7f9f300d06c0 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.515+0000 7f9f36466640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300b0dd0 con 0x7f9f300d02e0 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f2f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f300b4260 0x7f9f300b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f2f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f300b4260 0x7f9f300b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47766/0 (socket says 172.21.15.27:47766) 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f2f7fe640 1 -- 172.21.15.27:0/266938150 learned_addr learned my addr 172.21.15.27:0/266938150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/266938150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d02e0 0x7f9f300cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f34a03640 1 --2- 172.21.15.27:0/266938150 >> v2:172.21.15.69:3300/0 conn(0x7f9f300d06c0 0x7f9f300b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f2f7fe640 1 -- 172.21.15.27:0/266938150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d02e0 msgr2=0x7f9f300cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f2f7fe640 1 --2- 172.21.15.27:0/266938150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d02e0 0x7f9f300cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f2f7fe640 1 -- 172.21.15.27:0/266938150 >> v2:172.21.15.69:3300/0 conn(0x7f9f300d06c0 msgr2=0x7f9f300b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f2f7fe640 1 --2- 172.21.15.27:0/266938150 >> v2:172.21.15.69:3300/0 conn(0x7f9f300d06c0 0x7f9f300b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f2f7fe640 1 -- 172.21.15.27:0/266938150 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300b10b0 con 0x7f9f300b4260 2024-03-06T07:23:15.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.516+0000 7f9f34a03640 1 --2- 172.21.15.27:0/266938150 >> v2:172.21.15.69:3300/0 conn(0x7f9f300d06c0 0x7f9f300b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:15.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f2f7fe640 1 --2- 172.21.15.27:0/266938150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300b4260 0x7f9f300b1500 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9f30071d50 tx=0x7f9f1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a006f9d1d45562 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f2effd640 1 -- 172.21.15.27:0/266938150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18034070 con 0x7f9f300b4260 2024-03-06T07:23:15.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f2effd640 1 -- 172.21.15.27:0/266938150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18030860 con 0x7f9f300b4260 2024-03-06T07:23:15.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f2effd640 1 -- 172.21.15.27:0/266938150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18030c30 con 0x7f9f300b4260 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f36466640 1 -- 172.21.15.27:0/266938150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300b4260 msgr2=0x7f9f300b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f36466640 1 --2- 172.21.15.27:0/266938150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300b4260 0x7f9f300b1500 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9f30071d50 tx=0x7f9f1802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f36466640 1 -- 172.21.15.27:0/266938150 shutdown_connections 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f36466640 1 --2- 172.21.15.27:0/266938150 >> v2:172.21.15.69:3300/0 conn(0x7f9f300d06c0 0x7f9f300b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f36466640 1 --2- 172.21.15.27:0/266938150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300b4260 0x7f9f300b1500 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f36466640 1 --2- 172.21.15.27:0/266938150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d02e0 0x7f9f300cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.517+0000 7f9f36466640 1 -- 172.21.15.27:0/266938150 >> 172.21.15.27:0/266938150 conn(0x7f9f300b36d0 msgr2=0x7f9f300b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.518+0000 7f9f36466640 1 -- 172.21.15.27:0/266938150 shutdown_connections 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.518+0000 7f9f36466640 1 -- 172.21.15.27:0/266938150 wait complete. 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.519+0000 7f9f36466640 1 Processor -- start 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.519+0000 7f9f36466640 1 -- start start 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.519+0000 7f9f36466640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f300b4260 0x7f9f301f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.519+0000 7f9f36466640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300d02e0 0x7f9f300cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f36466640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d06c0 0x7f9f30140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f36466640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300b2100 con 0x7f9f300d02e0 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f36466640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300b1f80 con 0x7f9f300b4260 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f36466640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300b2280 con 0x7f9f300d06c0 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300d02e0 0x7f9f300cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300d02e0 0x7f9f300cb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47774/0 (socket says 172.21.15.27:47774) 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2f7fe640 1 -- 172.21.15.27:0/2798738508 learned_addr learned my addr 172.21.15.27:0/2798738508 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f34a03640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d06c0 0x7f9f30140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f300b4260 0x7f9f301f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2f7fe640 1 -- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d06c0 msgr2=0x7f9f30140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2f7fe640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d06c0 0x7f9f30140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2f7fe640 1 -- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f300b4260 msgr2=0x7f9f301f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2f7fe640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f300b4260 0x7f9f301f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2f7fe640 1 -- 172.21.15.27:0/2798738508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f30144070 con 0x7f9f300d02e0 2024-03-06T07:23:15.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2f7fe640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300d02e0 0x7f9f300cb620 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9f18000c00 tx=0x7f9f180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2cff9640 1 -- 172.21.15.27:0/2798738508 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18034040 con 0x7f9f300d02e0 2024-03-06T07:23:15.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2cff9640 1 -- 172.21.15.27:0/2798738508 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18039d90 con 0x7f9f300d02e0 2024-03-06T07:23:15.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f2cff9640 1 -- 172.21.15.27:0/2798738508 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18043610 con 0x7f9f300d02e0 2024-03-06T07:23:15.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f30142fd0 con 0x7f9f300d02e0 2024-03-06T07:23:15.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.520+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f30143590 con 0x7f9f300d02e0 2024-03-06T07:23:15.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.521+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f30068700 con 0x7f9f300d02e0 2024-03-06T07:23:15.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.522+0000 7f9f2cff9640 1 -- 172.21.15.27:0/2798738508 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f18037070 con 0x7f9f300d02e0 2024-03-06T07:23:15.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.522+0000 7f9f2cff9640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b4d0 0x7f9f1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.522+0000 7f9f2cff9640 1 -- 172.21.15.27:0/2798738508 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f180ac320 con 0x7f9f300d02e0 2024-03-06T07:23:15.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.522+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b4d0 0x7f9f1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.523+0000 7f9f2ffff640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b4d0 0x7f9f1006d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9f20006fd0 tx=0x7f9f20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.526+0000 7f9f2cff9640 1 -- 172.21.15.27:0/2798738508 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f1803e070 con 0x7f9f300d02e0 2024-03-06T07:23:15.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.860+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f300af380 con 0x7f9f300d02e0 2024-03-06T07:23:15.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.861+0000 7f9f2cff9640 1 -- 172.21.15.27:0/2798738508 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f9f1807b230 con 0x7f9f300d02e0 2024-03-06T07:23:15.863 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:15.863 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:15.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b4d0 msgr2=0x7f9f1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b4d0 0x7f9f1006d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9f20006fd0 tx=0x7f9f20008040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300d02e0 msgr2=0x7f9f300cb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300d02e0 0x7f9f300cb620 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9f18000c00 tx=0x7f9f180046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 shutdown_connections 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b4d0 0x7f9f1006d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f300d06c0 0x7f9f30140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f300d02e0 0x7f9f300cb620 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 --2- 172.21.15.27:0/2798738508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f300b4260 0x7f9f301f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 >> 172.21.15.27:0/2798738508 conn(0x7f9f300b36d0 msgr2=0x7f9f300ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 shutdown_connections 2024-03-06T07:23:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.863+0000 7f9f36466640 1 -- 172.21.15.27:0/2798738508 wait complete. 2024-03-06T07:23:15.875 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:15.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.984+0000 7f5488113640 1 Processor -- start 2024-03-06T07:23:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.984+0000 7f5488113640 1 -- start start 2024-03-06T07:23:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.985+0000 7f5488113640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800717b0 0x7f5480071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.985+0000 7f5488113640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f54800b2b10 0x7f54800d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.985+0000 7f5488113640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548015e4b0 con 0x7f54800b3060 2024-03-06T07:23:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.985+0000 7f5488113640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54800d2e10 con 0x7f54800b2b10 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.985+0000 7f5488113640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54800b97e0 con 0x7f54800717b0 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.985+0000 7f5485eaf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800717b0 0x7f5480071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.985+0000 7f54866b0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f54800b2b10 0x7f54800d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.985+0000 7f5485eaf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800717b0 0x7f5480071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:33070/0 (socket says 172.21.15.27:33070) 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.985+0000 7f5485eaf640 1 -- 172.21.15.27:0/3414173350 learned_addr learned my addr 172.21.15.27:0/3414173350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.986+0000 7f5485eaf640 1 -- 172.21.15.27:0/3414173350 >> v2:172.21.15.69:3300/0 conn(0x7f54800b2b10 msgr2=0x7f54800d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.986+0000 7f5485eaf640 1 --2- 172.21.15.27:0/3414173350 >> v2:172.21.15.69:3300/0 conn(0x7f54800b2b10 0x7f54800d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.986+0000 7f5485eaf640 1 -- 172.21.15.27:0/3414173350 >> v1:172.21.15.69:6789/0 conn(0x7f54800b3060 legacy=0x7f54800b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.986+0000 7f5485eaf640 1 -- 172.21.15.27:0/3414173350 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54800b9a80 con 0x7f54800717b0 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.986+0000 7f5484ead640 1 -- 172.21.15.27:0/3414173350 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198144775 0 0) 0x7f547c001110 con 0x7f54800b3060 2024-03-06T07:23:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.986+0000 7f54866b0640 1 --2- 172.21.15.27:0/3414173350 >> v2:172.21.15.69:3300/0 conn(0x7f54800b2b10 0x7f54800d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.986+0000 7f5485eaf640 1 --2- 172.21.15.27:0/3414173350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800717b0 0x7f5480071b90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5474009870 tx=0x7f547402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61a9655369d26f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.987+0000 7f5484ead640 1 -- 172.21.15.27:0/3414173350 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474034070 con 0x7f54800717b0 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.987+0000 7f5484ead640 1 -- 172.21.15.27:0/3414173350 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5474030840 con 0x7f54800717b0 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.987+0000 7f5484ead640 1 -- 172.21.15.27:0/3414173350 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474030c10 con 0x7f54800717b0 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.987+0000 7f5488113640 1 -- 172.21.15.27:0/3414173350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800717b0 msgr2=0x7f5480071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.987+0000 7f5488113640 1 --2- 172.21.15.27:0/3414173350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800717b0 0x7f5480071b90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5474009870 tx=0x7f547402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.987+0000 7f5488113640 1 -- 172.21.15.27:0/3414173350 shutdown_connections 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.987+0000 7f5488113640 1 --2- 172.21.15.27:0/3414173350 >> v2:172.21.15.69:3300/0 conn(0x7f54800b2b10 0x7f54800d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.987+0000 7f5488113640 1 --2- 172.21.15.27:0/3414173350 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800717b0 0x7f5480071b90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.987+0000 7f5488113640 1 -- 172.21.15.27:0/3414173350 >> 172.21.15.27:0/3414173350 conn(0x7f54800d3d20 msgr2=0x7f54800d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.988+0000 7f5488113640 1 -- 172.21.15.27:0/3414173350 shutdown_connections 2024-03-06T07:23:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.988+0000 7f5488113640 1 -- 172.21.15.27:0/3414173350 wait complete. 2024-03-06T07:23:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.989+0000 7f5488113640 1 Processor -- start 2024-03-06T07:23:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.989+0000 7f5488113640 1 -- start start 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5488113640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54800717b0 0x7f54801e4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5488113640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800b2b10 0x7f54801e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5488113640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54800b3060 0x7f54800c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5488113640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54800b7560 con 0x7f54800717b0 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5488113640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54800d4bf0 con 0x7f54800b3060 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5488113640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54800b76e0 con 0x7f54800b2b10 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5485eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54800717b0 0x7f54801e4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5485eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54800717b0 0x7f54801e4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47778/0 (socket says 172.21.15.27:47778) 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5485eaf640 1 -- 172.21.15.27:0/1910465841 learned_addr learned my addr 172.21.15.27:0/1910465841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f54856ae640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800b2b10 0x7f54801e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f54866b0640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54800b3060 0x7f54800c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5485eaf640 1 -- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800b2b10 msgr2=0x7f54801e7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5485eaf640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800b2b10 0x7f54801e7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5485eaf640 1 -- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54800b3060 msgr2=0x7f54800c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5485eaf640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54800b3060 0x7f54800c9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5485eaf640 1 -- 172.21.15.27:0/1910465841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54801e69a0 con 0x7f54800717b0 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f54866b0640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54800b3060 0x7f54800c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f5485eaf640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54800717b0 0x7f54801e4a90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5474000c00 tx=0x7f54740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f54727fc640 1 -- 172.21.15.27:0/1910465841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474034040 con 0x7f54800717b0 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.990+0000 7f54727fc640 1 -- 172.21.15.27:0/1910465841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5474030640 con 0x7f54800717b0 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.991+0000 7f54727fc640 1 -- 172.21.15.27:0/1910465841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474039d40 con 0x7f54800717b0 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.991+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54801e6c80 con 0x7f54800717b0 2024-03-06T07:23:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.991+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54801e7240 con 0x7f54800717b0 2024-03-06T07:23:15.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.991+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548006ce40 con 0x7f54800717b0 2024-03-06T07:23:15.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.992+0000 7f54727fc640 1 -- 172.21.15.27:0/1910465841 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f547404b060 con 0x7f54800717b0 2024-03-06T07:23:15.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.992+0000 7f54727fc640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f546406b4d0 0x7f546406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.992+0000 7f54727fc640 1 -- 172.21.15.27:0/1910465841 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54740abd50 con 0x7f54800717b0 2024-03-06T07:23:15.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.992+0000 7f54856ae640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f546406b4d0 0x7f546406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.993+0000 7f54856ae640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f546406b4d0 0x7f546406d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f54800ce6d0 tx=0x7f547c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:15.997+0000 7f54727fc640 1 -- 172.21.15.27:0/1910465841 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54740af030 con 0x7f54800717b0 2024-03-06T07:23:16.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.179+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f548004f410 con 0x7f54800717b0 2024-03-06T07:23:16.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.180+0000 7f54727fc640 1 -- 172.21.15.27:0/1910465841 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f547407acf0 con 0x7f54800717b0 2024-03-06T07:23:16.182 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f546406b4d0 msgr2=0x7f546406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f546406b4d0 0x7f546406d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f54800ce6d0 tx=0x7f547c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54800717b0 msgr2=0x7f54801e4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54800717b0 0x7f54801e4a90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5474000c00 tx=0x7f54740046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 shutdown_connections 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f546406b4d0 0x7f546406d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54800b3060 0x7f54800c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54800b2b10 0x7f54801e7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 --2- 172.21.15.27:0/1910465841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54800717b0 0x7f54801e4a90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 >> 172.21.15.27:0/1910465841 conn(0x7f54800d3d20 msgr2=0x7f54800b8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 shutdown_connections 2024-03-06T07:23:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.182+0000 7f5488113640 1 -- 172.21.15.27:0/1910465841 wait complete. 2024-03-06T07:23:16.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.302+0000 7f15b79d6640 1 Processor -- start 2024-03-06T07:23:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.303+0000 7f15b79d6640 1 -- start start 2024-03-06T07:23:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.303+0000 7f15b79d6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f15b0071540 0x7f15b0071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.303+0000 7f15b79d6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f15b00b2d30 0x7f15b00ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.303+0000 7f15b79d6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b0071e60 con 0x7f15b00cecb0 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.303+0000 7f15b79d6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b00d3230 con 0x7f15b0071540 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.303+0000 7f15b79d6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b00d4960 con 0x7f15b00b2d30 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.303+0000 7f15b5772640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f15b0071540 0x7f15b0071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.304+0000 7f15b5772640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f15b0071540 0x7f15b0071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47786/0 (socket says 172.21.15.27:47786) 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.304+0000 7f15b5772640 1 -- 172.21.15.27:0/3876218934 learned_addr learned my addr 172.21.15.27:0/3876218934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.304+0000 7f15b4f71640 1 --2- 172.21.15.27:0/3876218934 >> v2:172.21.15.69:3300/0 conn(0x7f15b00b2d30 0x7f15b00ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.304+0000 7f15b5772640 1 -- 172.21.15.27:0/3876218934 >> v2:172.21.15.69:3300/0 conn(0x7f15b00b2d30 msgr2=0x7f15b00ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.304+0000 7f15b5772640 1 --2- 172.21.15.27:0/3876218934 >> v2:172.21.15.69:3300/0 conn(0x7f15b00b2d30 0x7f15b00ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.304+0000 7f15b5772640 1 -- 172.21.15.27:0/3876218934 >> v1:172.21.15.27:6789/0 conn(0x7f15b00cecb0 legacy=0x7f15b00b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.304+0000 7f15b5772640 1 -- 172.21.15.27:0/3876218934 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b00d36c0 con 0x7f15b0071540 2024-03-06T07:23:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.304+0000 7f15a7fff640 1 -- 172.21.15.27:0/3876218934 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014418179 0 0) 0x7f15b0071e60 con 0x7f15b00cecb0 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15b5772640 1 --2- 172.21.15.27:0/3876218934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b0071540 0x7f15b0071920 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f15b00c7690 tx=0x7f15a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4488df9859fcb6d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15a7fff640 1 -- 172.21.15.27:0/3876218934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0034070 con 0x7f15b0071540 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15a7fff640 1 -- 172.21.15.27:0/3876218934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a0030860 con 0x7f15b0071540 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15a7fff640 1 -- 172.21.15.27:0/3876218934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0030c30 con 0x7f15b0071540 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15b79d6640 1 -- 172.21.15.27:0/3876218934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b0071540 msgr2=0x7f15b0071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15b79d6640 1 --2- 172.21.15.27:0/3876218934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b0071540 0x7f15b0071920 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f15b00c7690 tx=0x7f15a002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15b79d6640 1 -- 172.21.15.27:0/3876218934 shutdown_connections 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15b79d6640 1 --2- 172.21.15.27:0/3876218934 >> v2:172.21.15.69:3300/0 conn(0x7f15b00b2d30 0x7f15b00ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15b79d6640 1 --2- 172.21.15.27:0/3876218934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b0071540 0x7f15b0071920 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15b79d6640 1 -- 172.21.15.27:0/3876218934 >> 172.21.15.27:0/3876218934 conn(0x7f15b00d3d60 msgr2=0x7f15b00d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.305+0000 7f15b79d6640 1 -- 172.21.15.27:0/3876218934 shutdown_connections 2024-03-06T07:23:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.306+0000 7f15b79d6640 1 -- 172.21.15.27:0/3876218934 wait complete. 2024-03-06T07:23:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b79d6640 1 Processor -- start 2024-03-06T07:23:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b79d6640 1 -- start start 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b79d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15b0071540 0x7f15b0156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b79d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b00b2d30 0x7f15b01f7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b79d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b00cecb0 0x7f15b00c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b79d6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b00d2be0 con 0x7f15b00cecb0 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b79d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b00d2900 con 0x7f15b00b2d30 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b79d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b00d2ec0 con 0x7f15b0071540 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b5f73640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b00cecb0 0x7f15b00c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b5f73640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b00cecb0 0x7f15b00c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47788/0 (socket says 172.21.15.27:47788) 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.307+0000 7f15b5f73640 1 -- 172.21.15.27:0/1028034054 learned_addr learned my addr 172.21.15.27:0/1028034054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b5772640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15b0071540 0x7f15b0156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b4f71640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b00b2d30 0x7f15b01f7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b5f73640 1 -- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15b0071540 msgr2=0x7f15b0156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b5f73640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15b0071540 0x7f15b0156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b5f73640 1 -- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b00b2d30 msgr2=0x7f15b01f7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b5f73640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b00b2d30 0x7f15b01f7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b5f73640 1 -- 172.21.15.27:0/1028034054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b01570f0 con 0x7f15b00cecb0 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b5f73640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b00cecb0 0x7f15b00c9510 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f15a800c790 tx=0x7f15a800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15a5ffb640 1 -- 172.21.15.27:0/1028034054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8003da0 con 0x7f15b00cecb0 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15a5ffb640 1 -- 172.21.15.27:0/1028034054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a8005070 con 0x7f15b00cecb0 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b01573d0 con 0x7f15b00cecb0 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15a5ffb640 1 -- 172.21.15.27:0/1028034054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8005460 con 0x7f15b00cecb0 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.308+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b0157990 con 0x7f15b00cecb0 2024-03-06T07:23:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.309+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b0072070 con 0x7f15b00cecb0 2024-03-06T07:23:16.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.310+0000 7f15a5ffb640 1 -- 172.21.15.27:0/1028034054 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15a8005600 con 0x7f15b00cecb0 2024-03-06T07:23:16.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.310+0000 7f15a5ffb640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159406b4d0 0x7f159406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.310+0000 7f15a5ffb640 1 -- 172.21.15.27:0/1028034054 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15a8089fe0 con 0x7f15b00cecb0 2024-03-06T07:23:16.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.311+0000 7f15b5772640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159406b4d0 0x7f159406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.311+0000 7f15b5772640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159406b4d0 0x7f159406d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f15a0036000 tx=0x7f15a003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.315+0000 7f15a5ffb640 1 -- 172.21.15.27:0/1028034054 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15a8058810 con 0x7f15b00cecb0 2024-03-06T07:23:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.495+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f15b004f410 con 0x7f15b00cecb0 2024-03-06T07:23:16.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.495+0000 7f15a5ffb640 1 -- 172.21.15.27:0/1028034054 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f15a805c8b0 con 0x7f15b00cecb0 2024-03-06T07:23:16.497 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159406b4d0 msgr2=0x7f159406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159406b4d0 0x7f159406d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f15a0036000 tx=0x7f15a003b040 comp rx=0 tx=0).stop 2024-03-06T07:23:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b00cecb0 msgr2=0x7f15b00c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b00cecb0 0x7f15b00c9510 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f15a800c790 tx=0x7f15a800cc50 comp rx=0 tx=0).stop 2024-03-06T07:23:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 shutdown_connections 2024-03-06T07:23:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159406b4d0 0x7f159406d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15b00cecb0 0x7f15b00c9510 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b00b2d30 0x7f15b01f7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 --2- 172.21.15.27:0/1028034054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15b0071540 0x7f15b0156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.497+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 >> 172.21.15.27:0/1028034054 conn(0x7f15b00d3d60 msgr2=0x7f15b015cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.498+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 shutdown_connections 2024-03-06T07:23:16.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.498+0000 7f15b79d6640 1 -- 172.21.15.27:0/1028034054 wait complete. 2024-03-06T07:23:16.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:16.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.618+0000 7f95f8c6f640 1 Processor -- start 2024-03-06T07:23:16.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.618+0000 7f95f8c6f640 1 -- start start 2024-03-06T07:23:16.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.619+0000 7f95f8c6f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95f40b2d30 0x7f95f40b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.619+0000 7f95f8c6f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f95f40b7820 0x7f95f40b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.619+0000 7f95f8c6f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f40d4960 con 0x7f95f4071540 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.619+0000 7f95f8c6f640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f40d3230 con 0x7f95f40b2d30 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.619+0000 7f95f8c6f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f40d36c0 con 0x7f95f40b7820 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.619+0000 7f95ebfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95f40b2d30 0x7f95f40b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.619+0000 7f95ebfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95f40b2d30 0x7f95f40b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47792/0 (socket says 172.21.15.27:47792) 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.619+0000 7f95ebfff640 1 -- 172.21.15.27:0/3645550091 learned_addr learned my addr 172.21.15.27:0/3645550091 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.619+0000 7f95f2d9d640 1 --2- 172.21.15.27:0/3645550091 >> v2:172.21.15.69:3300/0 conn(0x7f95f40b7820 0x7f95f40b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.620+0000 7f95ebfff640 1 -- 172.21.15.27:0/3645550091 >> v2:172.21.15.69:3300/0 conn(0x7f95f40b7820 msgr2=0x7f95f40b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.620+0000 7f95ebfff640 1 --2- 172.21.15.27:0/3645550091 >> v2:172.21.15.69:3300/0 conn(0x7f95f40b7820 0x7f95f40b7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.620+0000 7f95ebfff640 1 -- 172.21.15.27:0/3645550091 >> v1:172.21.15.69:6789/0 conn(0x7f95f4071540 legacy=0x7f95f4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.620+0000 7f95ebfff640 1 -- 172.21.15.27:0/3645550091 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f40d2620 con 0x7f95f40b2d30 2024-03-06T07:23:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.620+0000 7f95f2d9d640 1 --2- 172.21.15.27:0/3645550091 >> v2:172.21.15.69:3300/0 conn(0x7f95f40b7820 0x7f95f40b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.621+0000 7f95ebfff640 1 --2- 172.21.15.27:0/3645550091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 0x7f95f40b72e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f95f40ce340 tx=0x7f95dc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d229657deeea387c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.621+0000 7f95f1d9b640 1 -- 172.21.15.27:0/3645550091 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc034070 con 0x7f95f40b2d30 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.621+0000 7f95f1d9b640 1 -- 172.21.15.27:0/3645550091 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc030860 con 0x7f95f40b2d30 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.621+0000 7f95f1d9b640 1 -- 172.21.15.27:0/3645550091 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc030c30 con 0x7f95f40b2d30 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.622+0000 7f95f8c6f640 1 -- 172.21.15.27:0/3645550091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 msgr2=0x7f95f40b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.622+0000 7f95f8c6f640 1 --2- 172.21.15.27:0/3645550091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 0x7f95f40b72e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f95f40ce340 tx=0x7f95dc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.622+0000 7f95f8c6f640 1 -- 172.21.15.27:0/3645550091 shutdown_connections 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.622+0000 7f95f8c6f640 1 --2- 172.21.15.27:0/3645550091 >> v2:172.21.15.69:3300/0 conn(0x7f95f40b7820 0x7f95f40b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.622+0000 7f95f8c6f640 1 --2- 172.21.15.27:0/3645550091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 0x7f95f40b72e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.622+0000 7f95f8c6f640 1 -- 172.21.15.27:0/3645550091 >> 172.21.15.27:0/3645550091 conn(0x7f95f40d3d60 msgr2=0x7f95f40d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.622+0000 7f95f8c6f640 1 -- 172.21.15.27:0/3645550091 shutdown_connections 2024-03-06T07:23:16.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.622+0000 7f95f8c6f640 1 -- 172.21.15.27:0/3645550091 wait complete. 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.623+0000 7f95f8c6f640 1 Processor -- start 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.623+0000 7f95f8c6f640 1 -- start start 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f8c6f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95f4071540 0x7f95f41eedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f8c6f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 0x7f95f415a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f8c6f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95f40b7820 0x7f95f415a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f8c6f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f40d2ec0 con 0x7f95f40b2d30 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f8c6f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f40d2be0 con 0x7f95f40b7820 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f8c6f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f40d22f0 con 0x7f95f4071540 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 0x7f95f415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95ebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 0x7f95f415a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47794/0 (socket says 172.21.15.27:47794) 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95ebfff640 1 -- 172.21.15.27:0/1763882382 learned_addr learned my addr 172.21.15.27:0/1763882382 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f259c640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95f4071540 0x7f95f41eedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f2d9d640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95f40b7820 0x7f95f415a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95ebfff640 1 -- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95f4071540 msgr2=0x7f95f41eedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95ebfff640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95f4071540 0x7f95f41eedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95ebfff640 1 -- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95f40b7820 msgr2=0x7f95f415a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95ebfff640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95f40b7820 0x7f95f415a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95ebfff640 1 -- 172.21.15.27:0/1763882382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f40ca7b0 con 0x7f95f40b2d30 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f259c640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95f4071540 0x7f95f41eedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95f2d9d640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95f40b7820 0x7f95f415a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.624+0000 7f95ebfff640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 0x7f95f415a240 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f95dc02b9e0 tx=0x7f95dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.625+0000 7f95eaffd640 1 -- 172.21.15.27:0/1763882382 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc048070 con 0x7f95f40b2d30 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.625+0000 7f95eaffd640 1 -- 172.21.15.27:0/1763882382 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc030660 con 0x7f95f40b2d30 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.625+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f40caa90 con 0x7f95f40b2d30 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.625+0000 7f95eaffd640 1 -- 172.21.15.27:0/1763882382 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc034040 con 0x7f95f40b2d30 2024-03-06T07:23:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.625+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f41f9a50 con 0x7f95f40b2d30 2024-03-06T07:23:16.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.626+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f40d0600 con 0x7f95f40b2d30 2024-03-06T07:23:16.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.626+0000 7f95eaffd640 1 -- 172.21.15.27:0/1763882382 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95dc04b070 con 0x7f95f40b2d30 2024-03-06T07:23:16.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.627+0000 7f95eaffd640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f95d406b4d0 0x7f95d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.627+0000 7f95eaffd640 1 -- 172.21.15.27:0/1763882382 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95dc0ac1e0 con 0x7f95f40b2d30 2024-03-06T07:23:16.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.627+0000 7f95f259c640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f95d406b4d0 0x7f95d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.627+0000 7f95f259c640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f95d406b4d0 0x7f95d406d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f95f4071510 tx=0x7f95e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.631+0000 7f95eaffd640 1 -- 172.21.15.27:0/1763882382 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95f40d0600 con 0x7f95f40b2d30 2024-03-06T07:23:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.891+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f95f4072070 con 0x7f95f40b2d30 2024-03-06T07:23:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.892+0000 7f95eaffd640 1 -- 172.21.15.27:0/1763882382 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f95dc07b060 con 0x7f95f40b2d30 2024-03-06T07:23:16.893 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f95d406b4d0 msgr2=0x7f95d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f95d406b4d0 0x7f95d406d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f95f4071510 tx=0x7f95e4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 msgr2=0x7f95f415a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 0x7f95f415a240 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f95dc02b9e0 tx=0x7f95dc004730 comp rx=0 tx=0).stop 2024-03-06T07:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 shutdown_connections 2024-03-06T07:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f95d406b4d0 0x7f95d406d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95f40b7820 0x7f95f415a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95f40b2d30 0x7f95f415a240 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 --2- 172.21.15.27:0/1763882382 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f95f4071540 0x7f95f41eedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.893+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 >> 172.21.15.27:0/1763882382 conn(0x7f95f40d3d60 msgr2=0x7f95f415b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.894+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 shutdown_connections 2024-03-06T07:23:16.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:16.894+0000 7f95f8c6f640 1 -- 172.21.15.27:0/1763882382 wait complete. 2024-03-06T07:23:16.916 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:23:16.916 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:23:16.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:17.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.022+0000 7fbadc9a2640 1 Processor -- start 2024-03-06T07:23:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.022+0000 7fbadc9a2640 1 -- start start 2024-03-06T07:23:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.023+0000 7fbadc9a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 0x7fbad815fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.023+0000 7fbadc9a2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbad80cec00 0x7fbad80cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.023+0000 7fbadc9a2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbad80cf060 0x7fbad80cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.023+0000 7fbadc9a2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad81605d0 con 0x7fbad80cec00 2024-03-06T07:23:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.023+0000 7fbadc9a2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad8160100 con 0x7fbad80cf060 2024-03-06T07:23:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.023+0000 7fbadc9a2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80cc960 con 0x7fbad815f7e0 2024-03-06T07:23:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbad80cec00 0x7fbad80cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbad80cec00 0x7fbad80cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47798/0 (socket says 172.21.15.27:47798) 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad6ffd640 1 -- 172.21.15.27:0/3794861204 learned_addr learned my addr 172.21.15.27:0/3794861204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad77fe640 1 --2- 172.21.15.27:0/3794861204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 0x7fbad815fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad7fff640 1 --2- 172.21.15.27:0/3794861204 >> v2:172.21.15.69:3300/0 conn(0x7fbad80cf060 0x7fbad80cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad6ffd640 1 -- 172.21.15.27:0/3794861204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 msgr2=0x7fbad815fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad6ffd640 1 --2- 172.21.15.27:0/3794861204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 0x7fbad815fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad6ffd640 1 -- 172.21.15.27:0/3794861204 >> v2:172.21.15.69:3300/0 conn(0x7fbad80cf060 msgr2=0x7fbad80cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad6ffd640 1 --2- 172.21.15.27:0/3794861204 >> v2:172.21.15.69:3300/0 conn(0x7fbad80cf060 0x7fbad80cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad6ffd640 1 -- 172.21.15.27:0/3794861204 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80ccc40 con 0x7fbad80cec00 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.024+0000 7fbad7fff640 1 --2- 172.21.15.27:0/3794861204 >> v2:172.21.15.69:3300/0 conn(0x7fbad80cf060 0x7fbad80cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbad6ffd640 1 --2- 172.21.15.27:0/3794861204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 0x7fbad80cf4d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbad8073560 tx=0x7fbac802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930c3ddc4534199e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbad67fc640 1 -- 172.21.15.27:0/3794861204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac802d040 con 0x7fbad80cec00 2024-03-06T07:23:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbad67fc640 1 -- 172.21.15.27:0/3794861204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac8004230 con 0x7fbad80cec00 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbad67fc640 1 -- 172.21.15.27:0/3794861204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8003610 con 0x7fbad80cec00 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbadc9a2640 1 -- 172.21.15.27:0/3794861204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 msgr2=0x7fbad80cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/3794861204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 0x7fbad80cf4d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbad8073560 tx=0x7fbac802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbadc9a2640 1 -- 172.21.15.27:0/3794861204 shutdown_connections 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/3794861204 >> v2:172.21.15.69:3300/0 conn(0x7fbad80cf060 0x7fbad80cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/3794861204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 0x7fbad80cf4d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/3794861204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 0x7fbad815fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbadc9a2640 1 -- 172.21.15.27:0/3794861204 >> 172.21.15.27:0/3794861204 conn(0x7fbad80b3430 msgr2=0x7fbad80b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbadc9a2640 1 -- 172.21.15.27:0/3794861204 shutdown_connections 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.025+0000 7fbadc9a2640 1 -- 172.21.15.27:0/3794861204 wait complete. 2024-03-06T07:23:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.027+0000 7fbadc9a2640 1 Processor -- start 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.027+0000 7fbadc9a2640 1 -- start start 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbadc9a2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 0x7fbad814e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbadc9a2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbad80cf060 0x7fbad81f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbadc9a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 0x7fbad815e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbadc9a2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80cd4e0 con 0x7fbad80cec00 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbadc9a2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80cd200 con 0x7fbad80cf060 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbadc9a2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80cc5c0 con 0x7fbad815f7e0 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 0x7fbad814e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 0x7fbad814e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47812/0 (socket says 172.21.15.27:47812) 2024-03-06T07:23:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad77fe640 1 -- 172.21.15.27:0/858503589 learned_addr learned my addr 172.21.15.27:0/858503589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad7fff640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 0x7fbad815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad77fe640 1 -- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 msgr2=0x7fbad815e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad6ffd640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbad80cf060 0x7fbad81f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad77fe640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 0x7fbad815e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad77fe640 1 -- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbad80cf060 msgr2=0x7fbad81f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad77fe640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbad80cf060 0x7fbad81f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad77fe640 1 -- 172.21.15.27:0/858503589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80c89d0 con 0x7fbad80cec00 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbad77fe640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 0x7fbad814e920 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbac000d5f0 tx=0x7fbac000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbacffff640 1 -- 172.21.15.27:0/858503589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac00125f0 con 0x7fbad80cec00 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbacffff640 1 -- 172.21.15.27:0/858503589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac0019070 con 0x7fbad80cec00 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad81f5a30 con 0x7fbad80cec00 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbacffff640 1 -- 172.21.15.27:0/858503589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac00129d0 con 0x7fbad80cec00 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.028+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad80c8c70 con 0x7fbad80cec00 2024-03-06T07:23:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.030+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad804f330 con 0x7fbad80cec00 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.030+0000 7fbacffff640 1 -- 172.21.15.27:0/858503589 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbac0012b70 con 0x7fbad80cec00 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.031+0000 7fbacffff640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbab806b4d0 0x7fbab806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.031+0000 7fbacffff640 1 -- 172.21.15.27:0/858503589 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbac008a610 con 0x7fbad80cec00 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.031+0000 7fbad6ffd640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbab806b4d0 0x7fbab806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.031+0000 7fbad6ffd640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbab806b4d0 0x7fbab806d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbac8035000 tx=0x7fbac8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.035+0000 7fbacffff640 1 -- 172.21.15.27:0/858503589 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbac0059490 con 0x7fbad80cec00 2024-03-06T07:23:17.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.221+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbad80b7880 con 0x7fbad80cec00 2024-03-06T07:23:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.222+0000 7fbacffff640 1 -- 172.21.15.27:0/858503589 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7fbac0058d90 con 0x7fbad80cec00 2024-03-06T07:23:17.224 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:17.225 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbab806b4d0 msgr2=0x7fbab806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbab806b4d0 0x7fbab806d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbac8035000 tx=0x7fbac8008030 comp rx=0 tx=0).stop 2024-03-06T07:23:17.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 msgr2=0x7fbad814e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 0x7fbad814e920 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbac000d5f0 tx=0x7fbac000dab0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 shutdown_connections 2024-03-06T07:23:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbab806b4d0 0x7fbab806d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbad815f7e0 0x7fbad815e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbad80cf060 0x7fbad81f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 --2- 172.21.15.27:0/858503589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad80cec00 0x7fbad814e920 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 >> 172.21.15.27:0/858503589 conn(0x7fbad80b3430 msgr2=0x7fbad80b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 shutdown_connections 2024-03-06T07:23:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.225+0000 7fbadc9a2640 1 -- 172.21.15.27:0/858503589 wait complete. 2024-03-06T07:23:17.237 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:17.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:17.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:17.315 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:17.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.348+0000 7f3ddd271640 1 Processor -- start 2024-03-06T07:23:17.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.348+0000 7f3ddd271640 1 -- start start 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3ddd271640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3dd80c6e40 0x7f3dd8166b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3ddd271640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd80cf120 con 0x7f3dd815f500 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3ddd271640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd80cd310 con 0x7f3dd80cf740 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3ddd271640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd815f8e0 con 0x7f3dd80c6e40 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3dd77fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3dd80c6e40 0x7f3dd8166b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3dd6ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3dd815f500 0x7f3dd80ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48310/0 (socket says 172.21.15.27:48310) 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3dd6ffd640 1 -- 172.21.15.27:0/2723580107 learned_addr learned my addr 172.21.15.27:0/2723580107 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3dd77fe640 1 -- 172.21.15.27:0/2723580107 >> v1:172.21.15.69:6789/0 conn(0x7f3dd80cf740 legacy=0x7f3dd80cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3dd77fe640 1 -- 172.21.15.27:0/2723580107 >> v1:172.21.15.27:6789/0 conn(0x7f3dd815f500 legacy=0x7f3dd80ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3dd77fe640 1 -- 172.21.15.27:0/2723580107 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd80cd5b0 con 0x7f3dd80c6e40 2024-03-06T07:23:17.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.349+0000 7f3dd67fc640 1 -- 172.21.15.27:0/2723580107 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099857330 0 0) 0x7f3dd80cf120 con 0x7f3dd815f500 2024-03-06T07:23:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.350+0000 7f3dd77fe640 1 --2- 172.21.15.27:0/2723580107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80c6e40 0x7f3dd8166b30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009d90 tx=0x7f3dd002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498134c1dd59d813 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.350+0000 7f3dd67fc640 1 -- 172.21.15.27:0/2723580107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0039070 con 0x7f3dd80c6e40 2024-03-06T07:23:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.350+0000 7f3dd67fc640 1 -- 172.21.15.27:0/2723580107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd002cbd0 con 0x7f3dd80c6e40 2024-03-06T07:23:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.350+0000 7f3dd67fc640 1 -- 172.21.15.27:0/2723580107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0034440 con 0x7f3dd80c6e40 2024-03-06T07:23:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.351+0000 7f3ddd271640 1 -- 172.21.15.27:0/2723580107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80c6e40 msgr2=0x7f3dd8166b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.351+0000 7f3ddd271640 1 --2- 172.21.15.27:0/2723580107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80c6e40 0x7f3dd8166b30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009d90 tx=0x7f3dd002be20 comp rx=0 tx=0).stop 2024-03-06T07:23:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.351+0000 7f3ddd271640 1 -- 172.21.15.27:0/2723580107 shutdown_connections 2024-03-06T07:23:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.351+0000 7f3ddd271640 1 --2- 172.21.15.27:0/2723580107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80c6e40 0x7f3dd8166b30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.351+0000 7f3ddd271640 1 -- 172.21.15.27:0/2723580107 >> 172.21.15.27:0/2723580107 conn(0x7f3dd80b3700 msgr2=0x7f3dd80b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.351+0000 7f3ddd271640 1 -- 172.21.15.27:0/2723580107 shutdown_connections 2024-03-06T07:23:17.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.351+0000 7f3ddd271640 1 -- 172.21.15.27:0/2723580107 wait complete. 2024-03-06T07:23:17.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.353+0000 7f3ddd271640 1 Processor -- start 2024-03-06T07:23:17.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.353+0000 7f3ddd271640 1 -- start start 2024-03-06T07:23:17.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.354+0000 7f3ddd271640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dd80c6e40 0x7f3dd8155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.354+0000 7f3ddd271640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dd80cf740 0x7f3dd81524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.354+0000 7f3ddd271640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd815f500 0x7f3dd80b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.354+0000 7f3ddd271640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd80c59a0 con 0x7f3dd815f500 2024-03-06T07:23:17.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.354+0000 7f3ddd271640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd80c56c0 con 0x7f3dd80c6e40 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.354+0000 7f3ddd271640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd80b43d0 con 0x7f3dd80cf740 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd815f500 0x7f3dd80b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd815f500 0x7f3dd80b58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47830/0 (socket says 172.21.15.27:47830) 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd77fe640 1 -- 172.21.15.27:0/1541173540 learned_addr learned my addr 172.21.15.27:0/1541173540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd6ffd640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dd80c6e40 0x7f3dd8155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dcffff640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dd80cf740 0x7f3dd81524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd77fe640 1 -- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dd80cf740 msgr2=0x7f3dd81524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd77fe640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dd80cf740 0x7f3dd81524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd77fe640 1 -- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dd80c6e40 msgr2=0x7f3dd8155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd77fe640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dd80c6e40 0x7f3dd8155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd77fe640 1 -- 172.21.15.27:0/1541173540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd8152a10 con 0x7f3dd815f500 2024-03-06T07:23:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dcffff640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dd80cf740 0x7f3dd81524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dd77fe640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd815f500 0x7f3dd80b58a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009d90 tx=0x7f3dd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dcf7fe640 1 -- 172.21.15.27:0/1541173540 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0044020 con 0x7f3dd815f500 2024-03-06T07:23:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dcf7fe640 1 -- 172.21.15.27:0/1541173540 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd003f470 con 0x7f3dd815f500 2024-03-06T07:23:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3dcf7fe640 1 -- 172.21.15.27:0/1541173540 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0039040 con 0x7f3dd815f500 2024-03-06T07:23:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd81549a0 con 0x7f3dd815f500 2024-03-06T07:23:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.355+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd8154f60 con 0x7f3dd815f500 2024-03-06T07:23:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.356+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd809fcf0 con 0x7f3dd815f500 2024-03-06T07:23:17.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.357+0000 7f3dcf7fe640 1 -- 172.21.15.27:0/1541173540 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3dd0047020 con 0x7f3dd815f500 2024-03-06T07:23:17.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.358+0000 7f3dcf7fe640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3db806b500 0x7f3db806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.358+0000 7f3dcf7fe640 1 -- 172.21.15.27:0/1541173540 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dd00acaf0 con 0x7f3dd815f500 2024-03-06T07:23:17.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.358+0000 7f3dd6ffd640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3db806b500 0x7f3db806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.358+0000 7f3dd6ffd640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3db806b500 0x7f3db806d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8000f60 tx=0x7f3dc8007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.362+0000 7f3dcf7fe640 1 -- 172.21.15.27:0/1541173540 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3dd004e070 con 0x7f3dd815f500 2024-03-06T07:23:17.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.541+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3dd804f410 con 0x7f3dd815f500 2024-03-06T07:23:17.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.542+0000 7f3dcf7fe640 1 -- 172.21.15.27:0/1541173540 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3dd8152a10 con 0x7f3dd815f500 2024-03-06T07:23:17.544 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3db806b500 msgr2=0x7f3db806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3db806b500 0x7f3db806d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8000f60 tx=0x7f3dc8007d50 comp rx=0 tx=0).stop 2024-03-06T07:23:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd815f500 msgr2=0x7f3dd80b58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd815f500 0x7f3dd80b58a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009d90 tx=0x7f3dd0004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 shutdown_connections 2024-03-06T07:23:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3db806b500 0x7f3db806d9c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd815f500 0x7f3dd80b58a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3dd80cf740 0x7f3dd81524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 --2- 172.21.15.27:0/1541173540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dd80c6e40 0x7f3dd8155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 >> 172.21.15.27:0/1541173540 conn(0x7f3dd80b3700 msgr2=0x7f3dd806ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 shutdown_connections 2024-03-06T07:23:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.543+0000 7f3ddd271640 1 -- 172.21.15.27:0/1541173540 wait complete. 2024-03-06T07:23:17.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.670+0000 7fea8a955640 1 Processor -- start 2024-03-06T07:23:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.670+0000 7fea8a955640 1 -- start start 2024-03-06T07:23:17.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea8a955640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 0x7fea841624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea8a955640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea840cf9f0 0x7fea8415d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea8a955640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8406d4d0 con 0x7fea84072a30 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea8a955640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea8406dba0 con 0x7fea840cf9f0 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea8a955640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea840b4950 con 0x7fea840d07c0 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea88ef2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea840cf9f0 0x7fea8415d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea83fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fea84072a30 0x7fea840d00b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48320/0 (socket says 172.21.15.27:48320) 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea83fff640 1 -- 172.21.15.27:0/407457675 learned_addr learned my addr 172.21.15.27:0/407457675 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea837fe640 1 --2- 172.21.15.27:0/407457675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 0x7fea841624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea82ffd640 1 -- 172.21.15.27:0/407457675 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016570407 0 0) 0x7fea8406d4d0 con 0x7fea84072a30 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea82ffd640 1 -- 172.21.15.27:0/407457675 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea6c003520 con 0x7fea84072a30 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea88ef2640 1 -- 172.21.15.27:0/407457675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 msgr2=0x7fea841624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea88ef2640 1 --2- 172.21.15.27:0/407457675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 0x7fea841624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea88ef2640 1 -- 172.21.15.27:0/407457675 >> v1:172.21.15.27:6789/0 conn(0x7fea84072a30 legacy=0x7fea840d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea88ef2640 1 -- 172.21.15.27:0/407457675 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8406d7b0 con 0x7fea840cf9f0 2024-03-06T07:23:17.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.671+0000 7fea837fe640 1 --2- 172.21.15.27:0/407457675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 0x7fea841624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.672+0000 7fea88ef2640 1 --2- 172.21.15.27:0/407457675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea840cf9f0 0x7fea8415d660 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009d90 tx=0x7fea7c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ce85971e124b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.672+0000 7fea82ffd640 1 -- 172.21.15.27:0/407457675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c039070 con 0x7fea840cf9f0 2024-03-06T07:23:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.672+0000 7fea82ffd640 1 -- 172.21.15.27:0/407457675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c02cb30 con 0x7fea840cf9f0 2024-03-06T07:23:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.672+0000 7fea82ffd640 1 -- 172.21.15.27:0/407457675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c034440 con 0x7fea840cf9f0 2024-03-06T07:23:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.673+0000 7fea8a955640 1 -- 172.21.15.27:0/407457675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea840cf9f0 msgr2=0x7fea8415d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.673+0000 7fea8a955640 1 --2- 172.21.15.27:0/407457675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea840cf9f0 0x7fea8415d660 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009d90 tx=0x7fea7c02bd80 comp rx=0 tx=0).stop 2024-03-06T07:23:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.673+0000 7fea8a955640 1 -- 172.21.15.27:0/407457675 shutdown_connections 2024-03-06T07:23:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.673+0000 7fea8a955640 1 --2- 172.21.15.27:0/407457675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea840cf9f0 0x7fea8415d660 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.673+0000 7fea8a955640 1 --2- 172.21.15.27:0/407457675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 0x7fea841624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.673+0000 7fea8a955640 1 -- 172.21.15.27:0/407457675 >> 172.21.15.27:0/407457675 conn(0x7fea8406af20 msgr2=0x7fea8406b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.673+0000 7fea8a955640 1 -- 172.21.15.27:0/407457675 shutdown_connections 2024-03-06T07:23:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.673+0000 7fea8a955640 1 -- 172.21.15.27:0/407457675 wait complete. 2024-03-06T07:23:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.675+0000 7fea8a955640 1 Processor -- start 2024-03-06T07:23:17.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.675+0000 7fea8a955640 1 -- start start 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.676+0000 7fea8a955640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea84072a30 0x7fea84152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.676+0000 7fea8a955640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea840cf9f0 0x7fea841f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.676+0000 7fea8a955640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 0x7fea841f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.676+0000 7fea8a955640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea84072e10 con 0x7fea84072a30 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.676+0000 7fea8a955640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea84071130 con 0x7fea840cf9f0 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.676+0000 7fea8a955640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea84072f90 con 0x7fea840d07c0 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.676+0000 7fea83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea84072a30 0x7fea84152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.676+0000 7fea83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea84072a30 0x7fea84152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47850/0 (socket says 172.21.15.27:47850) 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.676+0000 7fea83fff640 1 -- 172.21.15.27:0/474034749 learned_addr learned my addr 172.21.15.27:0/474034749 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea88ef2640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 0x7fea841f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea83fff640 1 -- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 msgr2=0x7fea841f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea83fff640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 0x7fea841f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea83fff640 1 -- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea840cf9f0 msgr2=0x7fea841f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea83fff640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea840cf9f0 0x7fea841f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea83fff640 1 -- 172.21.15.27:0/474034749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea841f5160 con 0x7fea84072a30 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea83fff640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea84072a30 0x7fea84152950 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea7400c390 tx=0x7fea7400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea80ff9640 1 -- 172.21.15.27:0/474034749 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea74017070 con 0x7fea84072a30 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea80ff9640 1 -- 172.21.15.27:0/474034749 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea74005ce0 con 0x7fea84072a30 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea80ff9640 1 -- 172.21.15.27:0/474034749 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea740072f0 con 0x7fea84072a30 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea841f5920 con 0x7fea84072a30 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.677+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea841f5ee0 con 0x7fea84072a30 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.678+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea8409e320 con 0x7fea84072a30 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.679+0000 7fea80ff9640 1 -- 172.21.15.27:0/474034749 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea7401a070 con 0x7fea84072a30 2024-03-06T07:23:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.679+0000 7fea80ff9640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea7006b4d0 0x7fea7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.679+0000 7fea80ff9640 1 -- 172.21.15.27:0/474034749 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea74089550 con 0x7fea84072a30 2024-03-06T07:23:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.679+0000 7fea837fe640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea7006b4d0 0x7fea7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.680+0000 7fea837fe640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea7006b4d0 0x7fea7006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fea78006e80 tx=0x7fea78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.684+0000 7fea80ff9640 1 -- 172.21.15.27:0/474034749 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea841f5160 con 0x7fea84072a30 2024-03-06T07:23:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.945+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fea8404f410 con 0x7fea84072a30 2024-03-06T07:23:17.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.946+0000 7fea80ff9640 1 -- 172.21.15.27:0/474034749 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fea74057d80 con 0x7fea84072a30 2024-03-06T07:23:17.947 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea7006b4d0 msgr2=0x7fea7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea7006b4d0 0x7fea7006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fea78006e80 tx=0x7fea78008040 comp rx=0 tx=0).stop 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea84072a30 msgr2=0x7fea84152950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea84072a30 0x7fea84152950 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea7400c390 tx=0x7fea7400c850 comp rx=0 tx=0).stop 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 shutdown_connections 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea7006b4d0 0x7fea7006d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea840d07c0 0x7fea841f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea840cf9f0 0x7fea841f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 --2- 172.21.15.27:0/474034749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea84072a30 0x7fea84152950 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 >> 172.21.15.27:0/474034749 conn(0x7fea8406af20 msgr2=0x7fea84160100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 shutdown_connections 2024-03-06T07:23:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:17.948+0000 7fea8a955640 1 -- 172.21.15.27:0/474034749 wait complete. 2024-03-06T07:23:17.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:18.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.069+0000 7f89b24fa640 1 Processor -- start 2024-03-06T07:23:18.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.069+0000 7f89b24fa640 1 -- start start 2024-03-06T07:23:18.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.069+0000 7f89b24fa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89ac15dd90 0x7f89ac0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.069+0000 7f89b24fa640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ac0d3740 con 0x7f89ac0d29d0 2024-03-06T07:23:18.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.069+0000 7f89b24fa640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ac0d3120 con 0x7f89ac0b6d10 2024-03-06T07:23:18.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.069+0000 7f89b24fa640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0d3400 con 0x7f89ac15dd90 2024-03-06T07:23:18.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.070+0000 7f89abfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89ac15dd90 0x7f89ac0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.070+0000 7f89ab7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f89ac0d29d0 0x7f89ac0d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48336/0 (socket says 172.21.15.27:48336) 2024-03-06T07:23:18.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.070+0000 7f89ab7fe640 1 -- 172.21.15.27:0/1084960716 learned_addr learned my addr 172.21.15.27:0/1084960716 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:18.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.070+0000 7f89abfff640 1 --2- 172.21.15.27:0/1084960716 >> v2:172.21.15.27:3300/0 conn(0x7f89ac15dd90 0x7f89ac0ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47860/0 (socket says 172.21.15.27:47860) 2024-03-06T07:23:18.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.070+0000 7f89abfff640 1 -- 172.21.15.27:0/1084960716 >> v1:172.21.15.69:6789/0 conn(0x7f89ac0b6d10 legacy=0x7f89ac0d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.070+0000 7f89abfff640 1 -- 172.21.15.27:0/1084960716 >> v1:172.21.15.27:6789/0 conn(0x7f89ac0d29d0 legacy=0x7f89ac0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.070+0000 7f89abfff640 1 -- 172.21.15.27:0/1084960716 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac0d39e0 con 0x7f89ac15dd90 2024-03-06T07:23:18.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89abfff640 1 --2- 172.21.15.27:0/1084960716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac15dd90 0x7f89ac0ceea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f899c0097a0 tx=0x7f899c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cacf52e1b75aba34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89aaffd640 1 -- 172.21.15.27:0/1084960716 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c02f810 con 0x7f89ac15dd90 2024-03-06T07:23:18.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89aaffd640 1 -- 172.21.15.27:0/1084960716 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899c02f9b0 con 0x7f89ac15dd90 2024-03-06T07:23:18.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89aaffd640 1 -- 172.21.15.27:0/1084960716 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c02fd80 con 0x7f89ac15dd90 2024-03-06T07:23:18.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89b24fa640 1 -- 172.21.15.27:0/1084960716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac15dd90 msgr2=0x7f89ac0ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89b24fa640 1 --2- 172.21.15.27:0/1084960716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac15dd90 0x7f89ac0ceea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f899c0097a0 tx=0x7f899c02b9f0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89b24fa640 1 -- 172.21.15.27:0/1084960716 shutdown_connections 2024-03-06T07:23:18.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89b24fa640 1 --2- 172.21.15.27:0/1084960716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac15dd90 0x7f89ac0ceea0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89b24fa640 1 -- 172.21.15.27:0/1084960716 >> 172.21.15.27:0/1084960716 conn(0x7f89ac071960 msgr2=0x7f89ac071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89b24fa640 1 -- 172.21.15.27:0/1084960716 shutdown_connections 2024-03-06T07:23:18.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.071+0000 7f89b24fa640 1 -- 172.21.15.27:0/1084960716 wait complete. 2024-03-06T07:23:18.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.073+0000 7f89b24fa640 1 Processor -- start 2024-03-06T07:23:18.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.073+0000 7f89b24fa640 1 -- start start 2024-03-06T07:23:18.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.073+0000 7f89b24fa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0b6d10 0x7f89ac1f7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.073+0000 7f89b24fa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89ac0d29d0 0x7f89ac0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.073+0000 7f89b24fa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ac15dd90 0x7f89ac150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89b24fa640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac0d2df0 con 0x7f89ac0b6d10 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89b24fa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac15e170 con 0x7f89ac15dd90 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89b24fa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac0d48f0 con 0x7f89ac0d29d0 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0b6d10 0x7f89ac1f7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0b6d10 0x7f89ac1f7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47876/0 (socket says 172.21.15.27:47876) 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89abfff640 1 -- 172.21.15.27:0/4017777190 learned_addr learned my addr 172.21.15.27:0/4017777190 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89ab7fe640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89ac0d29d0 0x7f89ac0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89b0a97640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ac15dd90 0x7f89ac150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89abfff640 1 -- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89ac0d29d0 msgr2=0x7f89ac0ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89abfff640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89ac0d29d0 0x7f89ac0ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89abfff640 1 -- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ac15dd90 msgr2=0x7f89ac150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89abfff640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ac15dd90 0x7f89ac150ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89abfff640 1 -- 172.21.15.27:0/4017777190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac0c82b0 con 0x7f89ac0b6d10 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89b0a97640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ac15dd90 0x7f89ac150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89abfff640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0b6d10 0x7f89ac1f7ec0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f899c009890 tx=0x7f899c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89a8ff9640 1 -- 172.21.15.27:0/4017777190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c046070 con 0x7f89ac0b6d10 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89a8ff9640 1 -- 172.21.15.27:0/4017777190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899c037c90 con 0x7f89ac0b6d10 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac15e2f0 con 0x7f89ac0b6d10 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89a8ff9640 1 -- 172.21.15.27:0/4017777190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c02fe60 con 0x7f89ac0b6d10 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.074+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac15e870 con 0x7f89ac0b6d10 2024-03-06T07:23:18.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.075+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac072a30 con 0x7f89ac0b6d10 2024-03-06T07:23:18.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.076+0000 7f89a8ff9640 1 -- 172.21.15.27:0/4017777190 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f899c049070 con 0x7f89ac0b6d10 2024-03-06T07:23:18.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.077+0000 7f89a8ff9640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f898c06b4d0 0x7f898c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.077+0000 7f89a8ff9640 1 -- 172.21.15.27:0/4017777190 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f899c0abbd0 con 0x7f89ac0b6d10 2024-03-06T07:23:18.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.077+0000 7f89ab7fe640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f898c06b4d0 0x7f898c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.077+0000 7f89ab7fe640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f898c06b4d0 0x7f898c06d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f89ac15cac0 tx=0x7f8994009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.085+0000 7f89a8ff9640 1 -- 172.21.15.27:0/4017777190 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f899c033030 con 0x7f89ac0b6d10 2024-03-06T07:23:18.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.265+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f89ac15c810 con 0x7f89ac0b6d10 2024-03-06T07:23:18.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.265+0000 7f89a8ff9640 1 -- 172.21.15.27:0/4017777190 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f899c07a400 con 0x7f89ac0b6d10 2024-03-06T07:23:18.268 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:18.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f898c06b4d0 msgr2=0x7f898c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f898c06b4d0 0x7f898c06d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f89ac15cac0 tx=0x7f8994009040 comp rx=0 tx=0).stop 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0b6d10 msgr2=0x7f89ac1f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0b6d10 0x7f89ac1f7ec0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f899c009890 tx=0x7f899c004730 comp rx=0 tx=0).stop 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 shutdown_connections 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f898c06b4d0 0x7f898c06d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ac15dd90 0x7f89ac150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89ac0d29d0 0x7f89ac0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 --2- 172.21.15.27:0/4017777190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0b6d10 0x7f89ac1f7ec0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 >> 172.21.15.27:0/4017777190 conn(0x7f89ac071960 msgr2=0x7f89ac0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 shutdown_connections 2024-03-06T07:23:18.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.269+0000 7f89b24fa640 1 -- 172.21.15.27:0/4017777190 wait complete. 2024-03-06T07:23:18.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:23:18.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.392+0000 7fe2c8f5c640 1 Processor -- start 2024-03-06T07:23:18.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.392+0000 7fe2c8f5c640 1 -- start start 2024-03-06T07:23:18.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c8f5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 0x7fe2c415cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c8f5c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2c40b2460 0x7fe2c40b3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c8f5c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c4074860 con 0x7fe2c41597c0 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c8f5c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c40742a0 con 0x7fe2c40b2460 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c8f5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c4074580 con 0x7fe2c4072c50 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c2d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2c40b2460 0x7fe2c40b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c2d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2c40b2460 0x7fe2c40b3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47892/0 (socket says 172.21.15.27:47892) 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c2d9d640 1 -- 172.21.15.27:0/2954383103 learned_addr learned my addr 172.21.15.27:0/2954383103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c1d9b640 1 --2- 172.21.15.27:0/2954383103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 0x7fe2c415cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c2d9d640 1 -- 172.21.15.27:0/2954383103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 msgr2=0x7fe2c415cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c2d9d640 1 --2- 172.21.15.27:0/2954383103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 0x7fe2c415cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c2d9d640 1 -- 172.21.15.27:0/2954383103 >> v1:172.21.15.69:6789/0 conn(0x7fe2c41597c0 legacy=0x7fe2c4072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.393+0000 7fe2c2d9d640 1 -- 172.21.15.27:0/2954383103 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c415c490 con 0x7fe2c40b2460 2024-03-06T07:23:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.394+0000 7fe2c2d9d640 1 --2- 172.21.15.27:0/2954383103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c40b2460 0x7fe2c40b3050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009e50 tx=0x7fe2bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2d7c51dfc749ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.394+0000 7fe2c159a640 1 -- 172.21.15.27:0/2954383103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc02ca70 con 0x7fe2c40b2460 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.394+0000 7fe2c159a640 1 -- 172.21.15.27:0/2954383103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc02cc10 con 0x7fe2c40b2460 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.394+0000 7fe2c159a640 1 -- 172.21.15.27:0/2954383103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc0343f0 con 0x7fe2c40b2460 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.394+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2954383103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c40b2460 msgr2=0x7fe2c40b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.394+0000 7fe2c8f5c640 1 --2- 172.21.15.27:0/2954383103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c40b2460 0x7fe2c40b3050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009e50 tx=0x7fe2bc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.395+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2954383103 shutdown_connections 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.395+0000 7fe2c8f5c640 1 --2- 172.21.15.27:0/2954383103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c40b2460 0x7fe2c40b3050 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.395+0000 7fe2c8f5c640 1 --2- 172.21.15.27:0/2954383103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 0x7fe2c415cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.395+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2954383103 >> 172.21.15.27:0/2954383103 conn(0x7fe2c4071690 msgr2=0x7fe2c40b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.395+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2954383103 shutdown_connections 2024-03-06T07:23:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.395+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2954383103 wait complete. 2024-03-06T07:23:18.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.396+0000 7fe2c8f5c640 1 Processor -- start 2024-03-06T07:23:18.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.396+0000 7fe2c8f5c640 1 -- start start 2024-03-06T07:23:18.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c8f5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 0x7fe2c4150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c8f5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c40b2460 0x7fe2c414db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c8f5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c41597c0 0x7fe2c409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c8f5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40b28e0 con 0x7fe2c41597c0 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c8f5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c415ca50 con 0x7fe2c40b2460 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c8f5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c4159ba0 con 0x7fe2c4072c50 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c41597c0 0x7fe2c409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c41597c0 0x7fe2c409ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47894/0 (socket says 172.21.15.27:47894) 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c2d9d640 1 -- 172.21.15.27:0/2974338866 learned_addr learned my addr 172.21.15.27:0/2974338866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c1d9b640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c40b2460 0x7fe2c414db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c259c640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 0x7fe2c4150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c2d9d640 1 -- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 msgr2=0x7fe2c4150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c2d9d640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 0x7fe2c4150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c2d9d640 1 -- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c40b2460 msgr2=0x7fe2c414db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c2d9d640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c40b2460 0x7fe2c414db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c2d9d640 1 -- 172.21.15.27:0/2974338866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c414ff50 con 0x7fe2c41597c0 2024-03-06T07:23:18.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2c2d9d640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c41597c0 0x7fe2c409ba00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009e50 tx=0x7fe2bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.397+0000 7fe2b2ffd640 1 -- 172.21.15.27:0/2974338866 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc043020 con 0x7fe2c41597c0 2024-03-06T07:23:18.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.398+0000 7fe2b2ffd640 1 -- 172.21.15.27:0/2974338866 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc03e450 con 0x7fe2c41597c0 2024-03-06T07:23:18.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.398+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c41501c0 con 0x7fe2c41597c0 2024-03-06T07:23:18.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.398+0000 7fe2b2ffd640 1 -- 172.21.15.27:0/2974338866 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc009510 con 0x7fe2c41597c0 2024-03-06T07:23:18.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.398+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c41511e0 con 0x7fe2c41597c0 2024-03-06T07:23:18.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.399+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c41505b0 con 0x7fe2c41597c0 2024-03-06T07:23:18.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.399+0000 7fe2b2ffd640 1 -- 172.21.15.27:0/2974338866 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2bc046020 con 0x7fe2c41597c0 2024-03-06T07:23:18.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.400+0000 7fe2b2ffd640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2a406b4d0 0x7fe2a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.400+0000 7fe2b2ffd640 1 -- 172.21.15.27:0/2974338866 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2bc0ac840 con 0x7fe2c41597c0 2024-03-06T07:23:18.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.400+0000 7fe2c259c640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2a406b4d0 0x7fe2a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.401+0000 7fe2c259c640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2a406b4d0 0x7fe2a406d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40016e0 tx=0x7fe2b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.404+0000 7fe2b2ffd640 1 -- 172.21.15.27:0/2974338866 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2bc04d030 con 0x7fe2c41597c0 2024-03-06T07:23:18.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:18.654+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fe2c4067ff0 con 0x7fe2c41597c0 2024-03-06T07:23:19.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.178+0000 7fe2b2ffd640 1 -- 172.21.15.27:0/2974338866 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v55) v1 ==== 135+0+0 (secure 0 0 0) 0x7fe2bc07b070 con 0x7fe2c41597c0 2024-03-06T07:23:19.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2a406b4d0 msgr2=0x7fe2a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2a406b4d0 0x7fe2a406d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40016e0 tx=0x7fe2b4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:19.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c41597c0 msgr2=0x7fe2c409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c41597c0 0x7fe2c409ba00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009e50 tx=0x7fe2bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:19.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 shutdown_connections 2024-03-06T07:23:19.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2a406b4d0 0x7fe2a406d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2c41597c0 0x7fe2c409ba00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c40b2460 0x7fe2c414db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 --2- 172.21.15.27:0/2974338866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2c4072c50 0x7fe2c4150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 >> 172.21.15.27:0/2974338866 conn(0x7fe2c4071690 msgr2=0x7fe2c4156140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 shutdown_connections 2024-03-06T07:23:19.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.180+0000 7fe2c8f5c640 1 -- 172.21.15.27:0/2974338866 wait complete. 2024-03-06T07:23:19.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:19.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.302+0000 7fa736034640 1 Processor -- start 2024-03-06T07:23:19.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.302+0000 7fa736034640 1 -- start start 2024-03-06T07:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa736034640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa7300b45d0 0x7fa7300ae210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa736034640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa7300b4020 0x7fa7300ae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa736034640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7300b1800 con 0x7fa73015ad50 2024-03-06T07:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa736034640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7300add80 con 0x7fa7300b45d0 2024-03-06T07:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa736034640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7300ad770 con 0x7fa7300b4020 2024-03-06T07:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa726ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa7300b45d0 0x7fa7300ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa726ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa7300b45d0 0x7fa7300ae210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47896/0 (socket says 172.21.15.27:47896) 2024-03-06T07:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa72f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa73015ad50 0x7fa7300b3910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48352/0 (socket says 172.21.15.27:48352) 2024-03-06T07:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa726ffd640 1 -- 172.21.15.27:0/4146030723 learned_addr learned my addr 172.21.15.27:0/4146030723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa72ffff640 1 --2- 172.21.15.27:0/4146030723 >> v2:172.21.15.69:3300/0 conn(0x7fa7300b4020 0x7fa7300ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa726ffd640 1 -- 172.21.15.27:0/4146030723 >> v2:172.21.15.69:3300/0 conn(0x7fa7300b4020 msgr2=0x7fa7300ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa726ffd640 1 --2- 172.21.15.27:0/4146030723 >> v2:172.21.15.69:3300/0 conn(0x7fa7300b4020 0x7fa7300ae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa726ffd640 1 -- 172.21.15.27:0/4146030723 >> v1:172.21.15.27:6789/0 conn(0x7fa73015ad50 legacy=0x7fa7300b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa72effd640 1 -- 172.21.15.27:0/4146030723 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507365274 0 0) 0x7fa7300b1800 con 0x7fa73015ad50 2024-03-06T07:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.303+0000 7fa726ffd640 1 -- 172.21.15.27:0/4146030723 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7300b1aa0 con 0x7fa7300b45d0 2024-03-06T07:23:19.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.304+0000 7fa726ffd640 1 --2- 172.21.15.27:0/4146030723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7300b45d0 0x7fa7300ae210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa7300d0140 tx=0x7fa71802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c063658b79b4d102 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.304+0000 7fa72effd640 1 -- 172.21.15.27:0/4146030723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71803b070 con 0x7fa7300b45d0 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.304+0000 7fa72effd640 1 -- 172.21.15.27:0/4146030723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa718036440 con 0x7fa7300b45d0 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.304+0000 7fa72effd640 1 -- 172.21.15.27:0/4146030723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa718036810 con 0x7fa7300b45d0 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.305+0000 7fa736034640 1 -- 172.21.15.27:0/4146030723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7300b45d0 msgr2=0x7fa7300ae210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.305+0000 7fa736034640 1 --2- 172.21.15.27:0/4146030723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7300b45d0 0x7fa7300ae210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa7300d0140 tx=0x7fa71802e0e0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.305+0000 7fa736034640 1 -- 172.21.15.27:0/4146030723 shutdown_connections 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.305+0000 7fa736034640 1 --2- 172.21.15.27:0/4146030723 >> v2:172.21.15.69:3300/0 conn(0x7fa7300b4020 0x7fa7300ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.305+0000 7fa736034640 1 --2- 172.21.15.27:0/4146030723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7300b45d0 0x7fa7300ae210 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.305+0000 7fa736034640 1 -- 172.21.15.27:0/4146030723 >> 172.21.15.27:0/4146030723 conn(0x7fa7300725b0 msgr2=0x7fa7300729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.305+0000 7fa736034640 1 -- 172.21.15.27:0/4146030723 shutdown_connections 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.305+0000 7fa736034640 1 -- 172.21.15.27:0/4146030723 wait complete. 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.307+0000 7fa736034640 1 Processor -- start 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.307+0000 7fa736034640 1 -- start start 2024-03-06T07:23:19.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa736034640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7300b4020 0x7fa73015ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa736034640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa73015ad50 0x7fa73009b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa736034640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73009db90 0x7fa730152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa736034640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73015b2b0 con 0x7fa73009db90 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa736034640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73015b130 con 0x7fa73015ad50 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa736034640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa730074030 con 0x7fa7300b4020 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73009db90 0x7fa730152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73009db90 0x7fa730152b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47898/0 (socket says 172.21.15.27:47898) 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72ffff640 1 -- 172.21.15.27:0/2562455674 learned_addr learned my addr 172.21.15.27:0/2562455674 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72f7fe640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7300b4020 0x7fa73015ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa726ffd640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa73015ad50 0x7fa73009b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72ffff640 1 -- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7300b4020 msgr2=0x7fa73015ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72ffff640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7300b4020 0x7fa73015ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72ffff640 1 -- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa73015ad50 msgr2=0x7fa73009b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72ffff640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa73015ad50 0x7fa73009b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72ffff640 1 -- 172.21.15.27:0/2562455674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73009a360 con 0x7fa73009db90 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.309+0000 7fa72ffff640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73009db90 0x7fa730152b40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa728009dd0 tx=0x7fa72800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.310+0000 7fa72cff9640 1 -- 172.21.15.27:0/2562455674 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa728018020 con 0x7fa73009db90 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.310+0000 7fa72cff9640 1 -- 172.21.15.27:0/2562455674 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa728014480 con 0x7fa73009db90 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.310+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73009a640 con 0x7fa73009db90 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.310+0000 7fa72cff9640 1 -- 172.21.15.27:0/2562455674 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa728014870 con 0x7fa73009db90 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.310+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73009ac00 con 0x7fa73009db90 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.311+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa730068210 con 0x7fa73009db90 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.311+0000 7fa72cff9640 1 -- 172.21.15.27:0/2562455674 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa728005d10 con 0x7fa73009db90 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.312+0000 7fa72cff9640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa71006b4d0 0x7fa71006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.312+0000 7fa72cff9640 1 -- 172.21.15.27:0/2562455674 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa72808aec0 con 0x7fa73009db90 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.312+0000 7fa72f7fe640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa71006b4d0 0x7fa71006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.312+0000 7fa72f7fe640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa71006b4d0 0x7fa71006d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa720007fd0 tx=0x7fa720007e00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.316+0000 7fa72cff9640 1 -- 172.21.15.27:0/2562455674 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa730068210 con 0x7fa73009db90 2024-03-06T07:23:19.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.497+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa73009e530 con 0x7fa73009db90 2024-03-06T07:23:19.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.500+0000 7fa72cff9640 1 -- 172.21.15.27:0/2562455674 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa728022030 con 0x7fa73009db90 2024-03-06T07:23:19.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.502+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa71006b4d0 msgr2=0x7fa71006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.502+0000 7fa736034640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa71006b4d0 0x7fa71006d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa720007fd0 tx=0x7fa720007e00 comp rx=0 tx=0).stop 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.502+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73009db90 msgr2=0x7fa730152b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.502+0000 7fa736034640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73009db90 0x7fa730152b40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa728009dd0 tx=0x7fa72800c490 comp rx=0 tx=0).stop 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.503+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 shutdown_connections 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.503+0000 7fa736034640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa71006b4d0 0x7fa71006d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.503+0000 7fa736034640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73009db90 0x7fa730152b40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.503+0000 7fa736034640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa73015ad50 0x7fa73009b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.503+0000 7fa736034640 1 --2- 172.21.15.27:0/2562455674 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7300b4020 0x7fa73015ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.503+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 >> 172.21.15.27:0/2562455674 conn(0x7fa7300725b0 msgr2=0x7fa7300afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.503+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 shutdown_connections 2024-03-06T07:23:19.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.503+0000 7fa736034640 1 -- 172.21.15.27:0/2562455674 wait complete. 2024-03-06T07:23:19.516 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:19.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:19.559 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:19.615 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:19.615 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:23:19.615 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:19.636 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:23:19.636 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:19.692 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:19.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.837+0000 7f6ab49ac640 1 Processor -- start 2024-03-06T07:23:19.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.837+0000 7f6ab49ac640 1 -- start start 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6ab49ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ab0066d00 0x7f6ab006e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6ab49ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab00b5840 0x7f6ab00670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6ab49ac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab006fc80 con 0x7f6ab015e140 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6ab49ac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab00b5d20 con 0x7f6ab0066d00 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6ab49ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab006d9f0 con 0x7f6ab00b5840 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aae59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ab0066d00 0x7f6ab006e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aaed9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6ab015e140 0x7f6ab0161a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48356/0 (socket says 172.21.15.27:48356) 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aaed9d640 1 -- 172.21.15.27:0/3178889218 learned_addr learned my addr 172.21.15.27:0/3178889218 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aadd9b640 1 --2- 172.21.15.27:0/3178889218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab00b5840 0x7f6ab00670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aae59c640 1 -- 172.21.15.27:0/3178889218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab00b5840 msgr2=0x7f6ab00670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aae59c640 1 --2- 172.21.15.27:0/3178889218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab00b5840 0x7f6ab00670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aae59c640 1 -- 172.21.15.27:0/3178889218 >> v1:172.21.15.27:6789/0 conn(0x7f6ab015e140 legacy=0x7f6ab0161a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aae59c640 1 -- 172.21.15.27:0/3178889218 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0067620 con 0x7f6ab0066d00 2024-03-06T07:23:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aad59a640 1 -- 172.21.15.27:0/3178889218 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185392676 0 0) 0x7f6ab006fc80 con 0x7f6ab015e140 2024-03-06T07:23:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.838+0000 7f6aadd9b640 1 --2- 172.21.15.27:0/3178889218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab00b5840 0x7f6ab00670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:19.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.839+0000 7f6aae59c640 1 --2- 172.21.15.27:0/3178889218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab0066d00 0x7f6ab006e400 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00097a0 tx=0x7f6aa002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6e13be35ed44f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.839+0000 7f6aad59a640 1 -- 172.21.15.27:0/3178889218 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0035020 con 0x7f6ab0066d00 2024-03-06T07:23:19.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.839+0000 7f6aad59a640 1 -- 172.21.15.27:0/3178889218 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa0031810 con 0x7f6ab0066d00 2024-03-06T07:23:19.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.839+0000 7f6aad59a640 1 -- 172.21.15.27:0/3178889218 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0031be0 con 0x7f6ab0066d00 2024-03-06T07:23:19.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.840+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3178889218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab0066d00 msgr2=0x7f6ab006e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.840+0000 7f6ab49ac640 1 --2- 172.21.15.27:0/3178889218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab0066d00 0x7f6ab006e400 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00097a0 tx=0x7f6aa002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.840+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3178889218 shutdown_connections 2024-03-06T07:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.840+0000 7f6ab49ac640 1 --2- 172.21.15.27:0/3178889218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab00b5840 0x7f6ab00670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.840+0000 7f6ab49ac640 1 --2- 172.21.15.27:0/3178889218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab0066d00 0x7f6ab006e400 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.840+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3178889218 >> 172.21.15.27:0/3178889218 conn(0x7f6ab00d0a20 msgr2=0x7f6ab00d0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.840+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3178889218 shutdown_connections 2024-03-06T07:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.840+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3178889218 wait complete. 2024-03-06T07:23:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.842+0000 7f6ab49ac640 1 Processor -- start 2024-03-06T07:23:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.842+0000 7f6ab49ac640 1 -- start start 2024-03-06T07:23:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6ab49ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0066d00 0x7f6ab0154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6ab49ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab00b5840 0x7f6ab00b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6ab49ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab015e140 0x7f6ab0154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6ab49ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab006dc40 con 0x7f6ab00b5840 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6ab49ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0070200 con 0x7f6ab0066d00 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6ab49ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00ad8b0 con 0x7f6ab015e140 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aadd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab00b5840 0x7f6ab00b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aadd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab00b5840 0x7f6ab00b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47918/0 (socket says 172.21.15.27:47918) 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aadd9b640 1 -- 172.21.15.27:0/3221160979 learned_addr learned my addr 172.21.15.27:0/3221160979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aaed9d640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab015e140 0x7f6ab0154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aae59c640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0066d00 0x7f6ab0154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aadd9b640 1 -- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab015e140 msgr2=0x7f6ab0154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aadd9b640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab015e140 0x7f6ab0154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aadd9b640 1 -- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0066d00 msgr2=0x7f6ab0154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aadd9b640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0066d00 0x7f6ab0154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aadd9b640 1 -- 172.21.15.27:0/3221160979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab01532b0 con 0x7f6ab00b5840 2024-03-06T07:23:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6aadd9b640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab00b5840 0x7f6ab00b1a00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e810 tx=0x7f6aa400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6a9effd640 1 -- 172.21.15.27:0/3221160979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400f040 con 0x7f6ab00b5840 2024-03-06T07:23:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6a9effd640 1 -- 172.21.15.27:0/3221160979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa4004030 con 0x7f6ab00b5840 2024-03-06T07:23:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab0153590 con 0x7f6ab00b5840 2024-03-06T07:23:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.844+0000 7f6a9effd640 1 -- 172.21.15.27:0/3221160979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4004400 con 0x7f6ab00b5840 2024-03-06T07:23:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.845+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab00b1f40 con 0x7f6ab00b5840 2024-03-06T07:23:19.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.846+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab0068700 con 0x7f6ab00b5840 2024-03-06T07:23:19.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.846+0000 7f6a9effd640 1 -- 172.21.15.27:0/3221160979 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6aa400c6d0 con 0x7f6ab00b5840 2024-03-06T07:23:19.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.846+0000 7f6a9effd640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a9006b4d0 0x7f6a9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.846+0000 7f6a9effd640 1 -- 172.21.15.27:0/3221160979 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6aa4089ee0 con 0x7f6ab00b5840 2024-03-06T07:23:19.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.846+0000 7f6aae59c640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a9006b4d0 0x7f6a9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.847+0000 7f6aae59c640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a9006b4d0 0x7f6a9006d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0036000 tx=0x7f6aa0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:19.850+0000 7f6a9effd640 1 -- 172.21.15.27:0/3221160979 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6aa4017020 con 0x7f6ab00b5840 2024-03-06T07:23:20.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.199+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ab0072b00 con 0x7f6ab00b5840 2024-03-06T07:23:20.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.200+0000 7f6a9effd640 1 -- 172.21.15.27:0/3221160979 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f6aa4058710 con 0x7f6ab00b5840 2024-03-06T07:23:20.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:20.203 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a9006b4d0 msgr2=0x7f6a9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a9006b4d0 0x7f6a9006d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0036000 tx=0x7f6aa0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab00b5840 msgr2=0x7f6ab00b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab00b5840 0x7f6ab00b1a00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e810 tx=0x7f6aa400ecd0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 shutdown_connections 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a9006b4d0 0x7f6a9006d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6ab015e140 0x7f6ab0154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab00b5840 0x7f6ab00b1a00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 --2- 172.21.15.27:0/3221160979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ab0066d00 0x7f6ab0154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 >> 172.21.15.27:0/3221160979 conn(0x7f6ab00d0a20 msgr2=0x7f6ab00b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 shutdown_connections 2024-03-06T07:23:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.202+0000 7f6ab49ac640 1 -- 172.21.15.27:0/3221160979 wait complete. 2024-03-06T07:23:20.216 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:20.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:20.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.324+0000 7f8a9633e640 1 Processor -- start 2024-03-06T07:23:20.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.324+0000 7f8a9633e640 1 -- start start 2024-03-06T07:23:20.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a9633e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a9015f500 0x7f8a900ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a9633e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900cf740 0x7f8a900ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a9633e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a900cd310 con 0x7f8a900ced80 2024-03-06T07:23:20.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a9633e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9015f8e0 con 0x7f8a9015f500 2024-03-06T07:23:20.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a9633e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a900c56c0 con 0x7f8a900cf740 2024-03-06T07:23:20.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a948db640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8a900ced80 0x7f8a900cfbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48358/0 (socket says 172.21.15.27:48358) 2024-03-06T07:23:20.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a948db640 1 -- 172.21.15.27:0/2129127664 learned_addr learned my addr 172.21.15.27:0/2129127664 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:20.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a8ffff640 1 --2- 172.21.15.27:0/2129127664 >> v2:172.21.15.69:3300/0 conn(0x7f8a9015f500 0x7f8a900ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a8f7fe640 1 --2- 172.21.15.27:0/2129127664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900cf740 0x7f8a900ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a8effd640 1 -- 172.21.15.27:0/2129127664 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258004471 0 0) 0x7f8a900cd310 con 0x7f8a900ced80 2024-03-06T07:23:20.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a8effd640 1 -- 172.21.15.27:0/2129127664 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a78003520 con 0x7f8a900ced80 2024-03-06T07:23:20.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a8f7fe640 1 -- 172.21.15.27:0/2129127664 >> v2:172.21.15.69:3300/0 conn(0x7f8a9015f500 msgr2=0x7f8a900ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a8f7fe640 1 --2- 172.21.15.27:0/2129127664 >> v2:172.21.15.69:3300/0 conn(0x7f8a9015f500 0x7f8a900ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a8f7fe640 1 -- 172.21.15.27:0/2129127664 >> v1:172.21.15.27:6789/0 conn(0x7f8a900ced80 legacy=0x7f8a900cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a8f7fe640 1 -- 172.21.15.27:0/2129127664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a900cd5b0 con 0x7f8a900cf740 2024-03-06T07:23:20.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.325+0000 7f8a8ffff640 1 --2- 172.21.15.27:0/2129127664 >> v2:172.21.15.69:3300/0 conn(0x7f8a9015f500 0x7f8a900ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.326+0000 7f8a8f7fe640 1 --2- 172.21.15.27:0/2129127664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900cf740 0x7f8a900ce840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8a840097a0 tx=0x7f8a8402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78aa3a167cf01205 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.326+0000 7f8a8effd640 1 -- 172.21.15.27:0/2129127664 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a840315a0 con 0x7f8a900cf740 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.326+0000 7f8a8effd640 1 -- 172.21.15.27:0/2129127664 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a84037070 con 0x7f8a900cf740 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.326+0000 7f8a8effd640 1 -- 172.21.15.27:0/2129127664 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84031960 con 0x7f8a900cf740 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.327+0000 7f8a9633e640 1 -- 172.21.15.27:0/2129127664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900cf740 msgr2=0x7f8a900ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.327+0000 7f8a9633e640 1 --2- 172.21.15.27:0/2129127664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900cf740 0x7f8a900ce840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8a840097a0 tx=0x7f8a8402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.327+0000 7f8a9633e640 1 -- 172.21.15.27:0/2129127664 shutdown_connections 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.327+0000 7f8a9633e640 1 --2- 172.21.15.27:0/2129127664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900cf740 0x7f8a900ce840 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.327+0000 7f8a9633e640 1 --2- 172.21.15.27:0/2129127664 >> v2:172.21.15.69:3300/0 conn(0x7f8a9015f500 0x7f8a900ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.327+0000 7f8a9633e640 1 -- 172.21.15.27:0/2129127664 >> 172.21.15.27:0/2129127664 conn(0x7f8a900b3700 msgr2=0x7f8a900b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.327+0000 7f8a9633e640 1 -- 172.21.15.27:0/2129127664 shutdown_connections 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.327+0000 7f8a9633e640 1 -- 172.21.15.27:0/2129127664 wait complete. 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.328+0000 7f8a9633e640 1 Processor -- start 2024-03-06T07:23:20.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.328+0000 7f8a9633e640 1 -- start start 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a9633e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900ced80 0x7f8a900b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a9633e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a900cf740 0x7f8a90156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a9633e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9015f500 0x7f8a900d0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a9633e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a900b43d0 con 0x7f8a900cf740 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a9633e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a900c59a0 con 0x7f8a9015f500 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a9633e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a900c6e40 con 0x7f8a900ced80 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a900cf740 0x7f8a90156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a900cf740 0x7f8a90156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47930/0 (socket says 172.21.15.27:47930) 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8f7fe640 1 -- 172.21.15.27:0/2613484627 learned_addr learned my addr 172.21.15.27:0/2613484627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8ffff640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900ced80 0x7f8a900b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a948db640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9015f500 0x7f8a900d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8f7fe640 1 -- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900ced80 msgr2=0x7f8a900b63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8f7fe640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900ced80 0x7f8a900b63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8f7fe640 1 -- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9015f500 msgr2=0x7f8a900d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8f7fe640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9015f500 0x7f8a900d0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8f7fe640 1 -- 172.21.15.27:0/2613484627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a901f5df0 con 0x7f8a900cf740 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a8ffff640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900ced80 0x7f8a900b63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.329+0000 7f8a948db640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9015f500 0x7f8a900d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.330+0000 7f8a8f7fe640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a900cf740 0x7f8a90156470 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8a84034040 tx=0x7f8a840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.330+0000 7f8a8cff9640 1 -- 172.21.15.27:0/2613484627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84032630 con 0x7f8a900cf740 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.330+0000 7f8a8cff9640 1 -- 172.21.15.27:0/2613484627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a840327f0 con 0x7f8a900cf740 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.330+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a900b68f0 con 0x7f8a900cf740 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.330+0000 7f8a8cff9640 1 -- 172.21.15.27:0/2613484627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84032be0 con 0x7f8a900cf740 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.330+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a901f5310 con 0x7f8a900cf740 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.331+0000 7f8a8cff9640 1 -- 172.21.15.27:0/2613484627 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a8404e430 con 0x7f8a900cf740 2024-03-06T07:23:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.331+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a901569b0 con 0x7f8a900cf740 2024-03-06T07:23:20.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.332+0000 7f8a8cff9640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.332+0000 7f8a8cff9640 1 -- 172.21.15.27:0/2613484627 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a8402f030 con 0x7f8a900cf740 2024-03-06T07:23:20.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.332+0000 7f8a8ffff640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.333+0000 7f8a8ffff640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8a90159b20 tx=0x7f8a80008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.336+0000 7f8a8cff9640 1 -- 172.21.15.27:0/2613484627 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a84040070 con 0x7f8a900cf740 2024-03-06T07:23:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.535+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a900cbfa0 con 0x7f8a900cf740 2024-03-06T07:23:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.536+0000 7f8a8cff9640 1 -- 172.21.15.27:0/2613484627 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a8407ae90 con 0x7f8a900cf740 2024-03-06T07:23:20.539 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a7c06b4d0 msgr2=0x7f8a7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8a90159b20 tx=0x7f8a80008040 comp rx=0 tx=0).stop 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a900cf740 msgr2=0x7f8a90156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a900cf740 0x7f8a90156470 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8a84034040 tx=0x7f8a840046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 shutdown_connections 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9015f500 0x7f8a900d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a900cf740 0x7f8a90156470 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 --2- 172.21.15.27:0/2613484627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a900ced80 0x7f8a900b63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 >> 172.21.15.27:0/2613484627 conn(0x7f8a900b3700 msgr2=0x7f8a9006ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 shutdown_connections 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.538+0000 7f8a9633e640 1 -- 172.21.15.27:0/2613484627 wait complete. 2024-03-06T07:23:20.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.659+0000 7fceed97a640 1 Processor -- start 2024-03-06T07:23:20.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.659+0000 7fceed97a640 1 -- start start 2024-03-06T07:23:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fceed97a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee815e2f0 0x7fcee80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fceed97a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee815e6d0 con 0x7fcee806b4e0 2024-03-06T07:23:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fceed97a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee80ccd00 con 0x7fcee806c050 2024-03-06T07:23:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fceed97a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80d0a10 con 0x7fcee815e2f0 2024-03-06T07:23:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fcee67fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcee806b4e0 0x7fcee806b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48364/0 (socket says 172.21.15.27:48364) 2024-03-06T07:23:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fcee67fc640 1 -- 172.21.15.27:0/1403689302 learned_addr learned my addr 172.21.15.27:0/1403689302 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fcee6ffd640 1 --2- 172.21.15.27:0/1403689302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee815e2f0 0x7fcee80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fcee5ffb640 1 -- 172.21.15.27:0/1403689302 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360391591 0 0) 0x7fcee815e6d0 con 0x7fcee806b4e0 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fcee5ffb640 1 -- 172.21.15.27:0/1403689302 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced0003520 con 0x7fcee806b4e0 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fcee5ffb640 1 -- 172.21.15.27:0/1403689302 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681801171 0 0) 0x7fcee80ccd00 con 0x7fcee806c050 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fcee5ffb640 1 -- 172.21.15.27:0/1403689302 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee815e6d0 con 0x7fcee806c050 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fcee6ffd640 1 -- 172.21.15.27:0/1403689302 >> v1:172.21.15.69:6789/0 conn(0x7fcee806c050 legacy=0x7fcee80cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.660+0000 7fcee6ffd640 1 -- 172.21.15.27:0/1403689302 >> v1:172.21.15.27:6789/0 conn(0x7fcee806b4e0 legacy=0x7fcee806b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.661+0000 7fcee6ffd640 1 -- 172.21.15.27:0/1403689302 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee80ccfe0 con 0x7fcee815e2f0 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.661+0000 7fcee6ffd640 1 --2- 172.21.15.27:0/1403689302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee815e2f0 0x7fcee80b41f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fced80097d0 tx=0x7fced802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df532f6e1f54b702 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.661+0000 7fcee5ffb640 1 -- 172.21.15.27:0/1403689302 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8031770 con 0x7fcee815e2f0 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.662+0000 7fcee5ffb640 1 -- 172.21.15.27:0/1403689302 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced8037070 con 0x7fcee815e2f0 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.662+0000 7fcee5ffb640 1 -- 172.21.15.27:0/1403689302 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8031b10 con 0x7fcee815e2f0 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.662+0000 7fceed97a640 1 -- 172.21.15.27:0/1403689302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee815e2f0 msgr2=0x7fcee80b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.662+0000 7fceed97a640 1 --2- 172.21.15.27:0/1403689302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee815e2f0 0x7fcee80b41f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fced80097d0 tx=0x7fced802b810 comp rx=0 tx=0).stop 2024-03-06T07:23:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.662+0000 7fceed97a640 1 -- 172.21.15.27:0/1403689302 shutdown_connections 2024-03-06T07:23:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.662+0000 7fceed97a640 1 --2- 172.21.15.27:0/1403689302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee815e2f0 0x7fcee80b41f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.662+0000 7fceed97a640 1 -- 172.21.15.27:0/1403689302 >> 172.21.15.27:0/1403689302 conn(0x7fcee80b36d0 msgr2=0x7fcee80b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.662+0000 7fceed97a640 1 -- 172.21.15.27:0/1403689302 shutdown_connections 2024-03-06T07:23:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.662+0000 7fceed97a640 1 -- 172.21.15.27:0/1403689302 wait complete. 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.663+0000 7fceed97a640 1 Processor -- start 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.663+0000 7fceed97a640 1 -- start start 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fceed97a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcee806b4e0 0x7fcee8150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fceed97a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee806c050 0x7fcee81f8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fceed97a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee815e2f0 0x7fcee80b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fceed97a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80ce1d0 con 0x7fcee806b4e0 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fceed97a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80cd5a0 con 0x7fcee815e2f0 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fceed97a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80ce350 con 0x7fcee806c050 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fcee6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcee806b4e0 0x7fcee8150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fcee6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcee806b4e0 0x7fcee8150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47942/0 (socket says 172.21.15.27:47942) 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fcee6ffd640 1 -- 172.21.15.27:0/3743875921 learned_addr learned my addr 172.21.15.27:0/3743875921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fcee67fc640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee806c050 0x7fcee81f8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.664+0000 7fcee77fe640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee815e2f0 0x7fcee80b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fcee6ffd640 1 -- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee806c050 msgr2=0x7fcee81f8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fcee6ffd640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee806c050 0x7fcee81f8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fcee6ffd640 1 -- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee815e2f0 msgr2=0x7fcee80b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fcee6ffd640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee815e2f0 0x7fcee80b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fcee6ffd640 1 -- 172.21.15.27:0/3743875921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee81f8960 con 0x7fcee806b4e0 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fcee6ffd640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcee806b4e0 0x7fcee8150d50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fced8000c00 tx=0x7fced80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fcecffff640 1 -- 172.21.15.27:0/3743875921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8045400 con 0x7fcee806b4e0 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fcecffff640 1 -- 172.21.15.27:0/3743875921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced80455a0 con 0x7fcee806b4e0 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee81f43d0 con 0x7fcee806b4e0 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fcecffff640 1 -- 172.21.15.27:0/3743875921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8045950 con 0x7fcee806b4e0 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.665+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee81f4980 con 0x7fcee806b4e0 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.666+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee809fd30 con 0x7fcee806b4e0 2024-03-06T07:23:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.667+0000 7fcecffff640 1 -- 172.21.15.27:0/3743875921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fced8031070 con 0x7fcee806b4e0 2024-03-06T07:23:20.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.668+0000 7fcecffff640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fced406b4d0 0x7fced406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.668+0000 7fcecffff640 1 -- 172.21.15.27:0/3743875921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fced80ab970 con 0x7fcee806b4e0 2024-03-06T07:23:20.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.668+0000 7fcee67fc640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fced406b4d0 0x7fced406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.669+0000 7fcee67fc640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fced406b4d0 0x7fced406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcee80c8050 tx=0x7fcedc007d70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.672+0000 7fcecffff640 1 -- 172.21.15.27:0/3743875921 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcee81f8960 con 0x7fcee806b4e0 2024-03-06T07:23:20.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.853+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcee804f410 con 0x7fcee806b4e0 2024-03-06T07:23:20.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.853+0000 7fcecffff640 1 -- 172.21.15.27:0/3743875921 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fced8040070 con 0x7fcee806b4e0 2024-03-06T07:23:20.856 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fced406b4d0 msgr2=0x7fced406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fced406b4d0 0x7fced406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcee80c8050 tx=0x7fcedc007d70 comp rx=0 tx=0).stop 2024-03-06T07:23:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcee806b4e0 msgr2=0x7fcee8150d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcee806b4e0 0x7fcee8150d50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fced8000c00 tx=0x7fced80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 shutdown_connections 2024-03-06T07:23:20.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fced406b4d0 0x7fced406d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee815e2f0 0x7fcee80b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcee806c050 0x7fcee81f8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 --2- 172.21.15.27:0/3743875921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcee806b4e0 0x7fcee8150d50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 >> 172.21.15.27:0/3743875921 conn(0x7fcee80b36d0 msgr2=0x7fcee80cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 shutdown_connections 2024-03-06T07:23:20.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.855+0000 7fceed97a640 1 -- 172.21.15.27:0/3743875921 wait complete. 2024-03-06T07:23:20.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.976+0000 7f344e642640 1 Processor -- start 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.976+0000 7f344e642640 1 -- start start 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344e642640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f344806b570 0x7f344806ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344e642640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34480af970 0x7f34480afde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344e642640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34480ae8d0 con 0x7f344815df70 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344e642640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34480aebb0 con 0x7f34480af970 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344e642640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34480aee90 con 0x7f344806b570 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344cbdf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34480af970 0x7f34480afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344cbdf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34480af970 0x7f34480afde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47948/0 (socket says 172.21.15.27:47948) 2024-03-06T07:23:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344cbdf640 1 -- 172.21.15.27:0/535482874 learned_addr learned my addr 172.21.15.27:0/535482874 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f34477fe640 1 --2- 172.21.15.27:0/535482874 >> v2:172.21.15.69:3300/0 conn(0x7f344806b570 0x7f344806ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344cbdf640 1 -- 172.21.15.27:0/535482874 >> v2:172.21.15.69:3300/0 conn(0x7f344806b570 msgr2=0x7f344806ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344cbdf640 1 --2- 172.21.15.27:0/535482874 >> v2:172.21.15.69:3300/0 conn(0x7f344806b570 0x7f344806ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344cbdf640 1 -- 172.21.15.27:0/535482874 >> v1:172.21.15.69:6789/0 conn(0x7f344815df70 legacy=0x7f34480d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.977+0000 7f344cbdf640 1 -- 172.21.15.27:0/535482874 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34480af170 con 0x7f34480af970 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.978+0000 7f344cbdf640 1 --2- 172.21.15.27:0/535482874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34480af970 0x7f34480afde0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3448071cc0 tx=0x7f344002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5192c105ad33381 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.978+0000 7f3446ffd640 1 -- 172.21.15.27:0/535482874 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344002ca80 con 0x7f34480af970 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.978+0000 7f3446ffd640 1 -- 172.21.15.27:0/535482874 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344002cc20 con 0x7f34480af970 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.978+0000 7f3446ffd640 1 -- 172.21.15.27:0/535482874 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34400343f0 con 0x7f34480af970 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.979+0000 7f344e642640 1 -- 172.21.15.27:0/535482874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34480af970 msgr2=0x7f34480afde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.979+0000 7f344e642640 1 --2- 172.21.15.27:0/535482874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34480af970 0x7f34480afde0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3448071cc0 tx=0x7f344002bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.979+0000 7f344e642640 1 -- 172.21.15.27:0/535482874 shutdown_connections 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.979+0000 7f344e642640 1 --2- 172.21.15.27:0/535482874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34480af970 0x7f34480afde0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.979+0000 7f344e642640 1 --2- 172.21.15.27:0/535482874 >> v2:172.21.15.69:3300/0 conn(0x7f344806b570 0x7f344806ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.979+0000 7f344e642640 1 -- 172.21.15.27:0/535482874 >> 172.21.15.27:0/535482874 conn(0x7f34480d1bd0 msgr2=0x7f34480d1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.979+0000 7f344e642640 1 -- 172.21.15.27:0/535482874 shutdown_connections 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.979+0000 7f344e642640 1 -- 172.21.15.27:0/535482874 wait complete. 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.980+0000 7f344e642640 1 Processor -- start 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.980+0000 7f344e642640 1 -- start start 2024-03-06T07:23:20.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f344e642640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344806b570 0x7f3448158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f344e642640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34480af970 0x7f34481f7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f344e642640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f344815df70 0x7f34480b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f344e642640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344806b150 con 0x7f344806b570 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f344e642640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344815e350 con 0x7f344815df70 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f344e642640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34480d0f50 con 0x7f34480af970 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f3447fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344806b570 0x7f3448158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f3447fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344806b570 0x7f3448158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47956/0 (socket says 172.21.15.27:47956) 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f3447fff640 1 -- 172.21.15.27:0/4209535824 learned_addr learned my addr 172.21.15.27:0/4209535824 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f344cbdf640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f344815df70 0x7f34480b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f34477fe640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34480af970 0x7f34481f7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f3447fff640 1 -- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34480af970 msgr2=0x7f34481f7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f3447fff640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34480af970 0x7f34481f7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f3447fff640 1 -- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f344815df70 msgr2=0x7f34480b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f3447fff640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f344815df70 0x7f34480b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f3447fff640 1 -- 172.21.15.27:0/4209535824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3448159440 con 0x7f344806b570 2024-03-06T07:23:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.981+0000 7f344cbdf640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f344815df70 0x7f34480b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.982+0000 7f3447fff640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344806b570 0x7f3448158f00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f343800b3f0 tx=0x7f343800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.982+0000 7f3444ff9640 1 -- 172.21.15.27:0/4209535824 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438015020 con 0x7f344806b570 2024-03-06T07:23:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.982+0000 7f3444ff9640 1 -- 172.21.15.27:0/4209535824 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3438011830 con 0x7f344806b570 2024-03-06T07:23:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.982+0000 7f3444ff9640 1 -- 172.21.15.27:0/4209535824 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438011c20 con 0x7f344806b570 2024-03-06T07:23:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.982+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34480b1b00 con 0x7f344806b570 2024-03-06T07:23:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.982+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34481548e0 con 0x7f344806b570 2024-03-06T07:23:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.983+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344809fc50 con 0x7f344806b570 2024-03-06T07:23:20.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.983+0000 7f3444ff9640 1 -- 172.21.15.27:0/4209535824 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3438018020 con 0x7f344806b570 2024-03-06T07:23:20.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.984+0000 7f3444ff9640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f343406b4d0 0x7f343406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.984+0000 7f3444ff9640 1 -- 172.21.15.27:0/4209535824 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3438089e20 con 0x7f344806b570 2024-03-06T07:23:20.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.984+0000 7f34477fe640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f343406b4d0 0x7f343406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.984+0000 7f34477fe640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f343406b4d0 0x7f343406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f344814d780 tx=0x7f343c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:20.988+0000 7f3444ff9640 1 -- 172.21.15.27:0/4209535824 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f343801f030 con 0x7f344806b570 2024-03-06T07:23:21.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.173+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34480d1680 con 0x7f344806b570 2024-03-06T07:23:21.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.174+0000 7f3444ff9640 1 -- 172.21.15.27:0/4209535824 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34380172c0 con 0x7f344806b570 2024-03-06T07:23:21.177 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f343406b4d0 msgr2=0x7f343406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f343406b4d0 0x7f343406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f344814d780 tx=0x7f343c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344806b570 msgr2=0x7f3448158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344806b570 0x7f3448158f00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f343800b3f0 tx=0x7f343800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 shutdown_connections 2024-03-06T07:23:21.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f343406b4d0 0x7f343406d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f344815df70 0x7f34480b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34480af970 0x7f34481f7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 --2- 172.21.15.27:0/4209535824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344806b570 0x7f3448158f00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 >> 172.21.15.27:0/4209535824 conn(0x7f34480d1bd0 msgr2=0x7f34480cef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 shutdown_connections 2024-03-06T07:23:21.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.176+0000 7f344e642640 1 -- 172.21.15.27:0/4209535824 wait complete. 2024-03-06T07:23:21.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:21.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.300+0000 7f3b32d5e640 1 Processor -- start 2024-03-06T07:23:21.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.300+0000 7f3b32d5e640 1 -- start start 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b32d5e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b2c071540 0x7f3b2c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b32d5e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3b2c0b2b50 0x7f3b2c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b32d5e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0ce610 con 0x7f3b2c0d3400 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b32d5e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c0d2f70 con 0x7f3b2c071540 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b32d5e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c0d4be0 con 0x7f3b2c0b2b50 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b30afa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b2c071540 0x7f3b2c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b2bfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3b2c0d3400 0x7f3b2c071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48380/0 (socket says 172.21.15.27:48380) 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b2bfff640 1 -- 172.21.15.27:0/3534378708 learned_addr learned my addr 172.21.15.27:0/3534378708 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b312fb640 1 --2- 172.21.15.27:0/3534378708 >> v2:172.21.15.69:3300/0 conn(0x7f3b2c0b2b50 0x7f3b2c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b30afa640 1 -- 172.21.15.27:0/3534378708 >> v2:172.21.15.69:3300/0 conn(0x7f3b2c0b2b50 msgr2=0x7f3b2c0b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b30afa640 1 --2- 172.21.15.27:0/3534378708 >> v2:172.21.15.69:3300/0 conn(0x7f3b2c0b2b50 0x7f3b2c0b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b30afa640 1 -- 172.21.15.27:0/3534378708 >> v1:172.21.15.27:6789/0 conn(0x7f3b2c0d3400 legacy=0x7f3b2c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.301+0000 7f3b30afa640 1 -- 172.21.15.27:0/3534378708 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0d2920 con 0x7f3b2c071540 2024-03-06T07:23:21.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.302+0000 7f3b30afa640 1 --2- 172.21.15.27:0/3534378708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c071540 0x7f3b2c0b30a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0683b0 tx=0x7f3b2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36b7e6adec61129 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.302+0000 7f3b2b7fe640 1 -- 172.21.15.27:0/3534378708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20035020 con 0x7f3b2c071540 2024-03-06T07:23:21.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.302+0000 7f3b2b7fe640 1 -- 172.21.15.27:0/3534378708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b20031830 con 0x7f3b2c071540 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.302+0000 7f3b2b7fe640 1 -- 172.21.15.27:0/3534378708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20031c00 con 0x7f3b2c071540 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.302+0000 7f3b32d5e640 1 -- 172.21.15.27:0/3534378708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c071540 msgr2=0x7f3b2c0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.302+0000 7f3b32d5e640 1 --2- 172.21.15.27:0/3534378708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c071540 0x7f3b2c0b30a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0683b0 tx=0x7f3b2002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.303+0000 7f3b32d5e640 1 -- 172.21.15.27:0/3534378708 shutdown_connections 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.303+0000 7f3b32d5e640 1 --2- 172.21.15.27:0/3534378708 >> v2:172.21.15.69:3300/0 conn(0x7f3b2c0b2b50 0x7f3b2c0b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.303+0000 7f3b32d5e640 1 --2- 172.21.15.27:0/3534378708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c071540 0x7f3b2c0b30a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.303+0000 7f3b32d5e640 1 -- 172.21.15.27:0/3534378708 >> 172.21.15.27:0/3534378708 conn(0x7f3b2c0d3ca0 msgr2=0x7f3b2c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.303+0000 7f3b32d5e640 1 -- 172.21.15.27:0/3534378708 shutdown_connections 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.303+0000 7f3b32d5e640 1 -- 172.21.15.27:0/3534378708 wait complete. 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.304+0000 7f3b32d5e640 1 Processor -- start 2024-03-06T07:23:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.304+0000 7f3b32d5e640 1 -- start start 2024-03-06T07:23:21.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b32d5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b2c071540 0x7f3b2c1522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b32d5e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b2c0b2b50 0x7f3b2c0cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b32d5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c0d3400 0x7f3b2c158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b32d5e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c15e370 con 0x7f3b2c0d3400 2024-03-06T07:23:21.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b32d5e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c0ce8f0 con 0x7f3b2c071540 2024-03-06T07:23:21.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b32d5e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c1527f0 con 0x7f3b2c0b2b50 2024-03-06T07:23:21.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b312fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c0d3400 0x7f3b2c158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b312fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c0d3400 0x7f3b2c158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47966/0 (socket says 172.21.15.27:47966) 2024-03-06T07:23:21.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b312fb640 1 -- 172.21.15.27:0/275668200 learned_addr learned my addr 172.21.15.27:0/275668200 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b312fb640 1 -- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b2c0b2b50 msgr2=0x7f3b2c0cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b312fb640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b2c0b2b50 0x7f3b2c0cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b312fb640 1 -- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b2c071540 msgr2=0x7f3b2c1522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b312fb640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b2c071540 0x7f3b2c1522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b312fb640 1 -- 172.21.15.27:0/275668200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0b3bc0 con 0x7f3b2c0d3400 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b312fb640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c0d3400 0x7f3b2c158d20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3b24007e70 tx=0x7f3b2400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b297fa640 1 -- 172.21.15.27:0/275668200 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24014620 con 0x7f3b2c0d3400 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b297fa640 1 -- 172.21.15.27:0/275668200 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b24019070 con 0x7f3b2c0d3400 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c0b3ea0 con 0x7f3b2c0d3400 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.305+0000 7f3b297fa640 1 -- 172.21.15.27:0/275668200 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24014a20 con 0x7f3b2c0d3400 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.306+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c0b4460 con 0x7f3b2c0d3400 2024-03-06T07:23:21.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.306+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c068700 con 0x7f3b2c0d3400 2024-03-06T07:23:21.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.307+0000 7f3b297fa640 1 -- 172.21.15.27:0/275668200 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b24014bc0 con 0x7f3b2c0d3400 2024-03-06T07:23:21.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.307+0000 7f3b297fa640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3b1006b500 0x7f3b1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.307+0000 7f3b297fa640 1 -- 172.21.15.27:0/275668200 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b2408aa20 con 0x7f3b2c0d3400 2024-03-06T07:23:21.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.308+0000 7f3b30afa640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3b1006b500 0x7f3b1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.308+0000 7f3b30afa640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3b1006b500 0x7f3b1006d9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3b20036000 tx=0x7f3b20030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.312+0000 7f3b297fa640 1 -- 172.21.15.27:0/275668200 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b24059930 con 0x7f3b2c0d3400 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.505+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3b2c071be0 con 0x7f3b2c0d3400 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.505+0000 7f3b297fa640 1 -- 172.21.15.27:0/275668200 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3b24059230 con 0x7f3b2c0d3400 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:23:21.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.507+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3b1006b500 msgr2=0x7f3b1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.507+0000 7f3b32d5e640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3b1006b500 0x7f3b1006d9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3b20036000 tx=0x7f3b20030000 comp rx=0 tx=0).stop 2024-03-06T07:23:21.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.507+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c0d3400 msgr2=0x7f3b2c158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.507+0000 7f3b32d5e640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c0d3400 0x7f3b2c158d20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3b24007e70 tx=0x7f3b2400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.508+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 shutdown_connections 2024-03-06T07:23:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.508+0000 7f3b32d5e640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3b1006b500 0x7f3b1006d9c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.508+0000 7f3b32d5e640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b2c0d3400 0x7f3b2c158d20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.508+0000 7f3b32d5e640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b2c0b2b50 0x7f3b2c0cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.508+0000 7f3b32d5e640 1 --2- 172.21.15.27:0/275668200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b2c071540 0x7f3b2c1522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.508+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 >> 172.21.15.27:0/275668200 conn(0x7f3b2c0d3ca0 msgr2=0x7f3b2c15fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.508+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 shutdown_connections 2024-03-06T07:23:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.508+0000 7f3b32d5e640 1 -- 172.21.15.27:0/275668200 wait complete. 2024-03-06T07:23:21.521 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:23:21.521 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:21.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:21.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.680+0000 7f43d95ac640 1 Processor -- start 2024-03-06T07:23:21.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.680+0000 7f43d95ac640 1 -- start start 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d95ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d40b1020 0x7f43d406af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d95ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43d406ed10 0x7f43d40cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d95ac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d40cc0e0 con 0x7f43d40cd6c0 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d95ac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d40d28b0 con 0x7f43d406ed10 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d95ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40cf320 con 0x7f43d40b1020 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d27fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43d406ed10 0x7f43d40cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d37fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f43d40cd6c0 0x7f43d415d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48390/0 (socket says 172.21.15.27:48390) 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d37fe640 1 -- 172.21.15.27:0/2549359701 learned_addr learned my addr 172.21.15.27:0/2549359701 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d2ffd640 1 --2- 172.21.15.27:0/2549359701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d40b1020 0x7f43d406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d1ffb640 1 -- 172.21.15.27:0/2549359701 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566454727 0 0) 0x7f43d40cc0e0 con 0x7f43d40cd6c0 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d1ffb640 1 -- 172.21.15.27:0/2549359701 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c0003520 con 0x7f43d40cd6c0 2024-03-06T07:23:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d27fc640 1 -- 172.21.15.27:0/2549359701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d40b1020 msgr2=0x7f43d406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d27fc640 1 --2- 172.21.15.27:0/2549359701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d40b1020 0x7f43d406af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d27fc640 1 -- 172.21.15.27:0/2549359701 >> v1:172.21.15.27:6789/0 conn(0x7f43d40cd6c0 legacy=0x7f43d415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.681+0000 7f43d27fc640 1 -- 172.21.15.27:0/2549359701 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d40cc3c0 con 0x7f43d406ed10 2024-03-06T07:23:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.682+0000 7f43d27fc640 1 --2- 172.21.15.27:0/2549359701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d406ed10 0x7f43d40cd180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f43bc0097a0 tx=0x7f43bc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afc472ec55ab274 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.682+0000 7f43d1ffb640 1 -- 172.21.15.27:0/2549359701 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc035020 con 0x7f43d406ed10 2024-03-06T07:23:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.682+0000 7f43d1ffb640 1 -- 172.21.15.27:0/2549359701 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43bc031810 con 0x7f43d406ed10 2024-03-06T07:23:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.682+0000 7f43d1ffb640 1 -- 172.21.15.27:0/2549359701 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc031bc0 con 0x7f43d406ed10 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.683+0000 7f43d95ac640 1 -- 172.21.15.27:0/2549359701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d406ed10 msgr2=0x7f43d40cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.683+0000 7f43d95ac640 1 --2- 172.21.15.27:0/2549359701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d406ed10 0x7f43d40cd180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f43bc0097a0 tx=0x7f43bc02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.683+0000 7f43d95ac640 1 -- 172.21.15.27:0/2549359701 shutdown_connections 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.683+0000 7f43d95ac640 1 --2- 172.21.15.27:0/2549359701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d406ed10 0x7f43d40cd180 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.683+0000 7f43d95ac640 1 --2- 172.21.15.27:0/2549359701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d40b1020 0x7f43d406af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.683+0000 7f43d95ac640 1 -- 172.21.15.27:0/2549359701 >> 172.21.15.27:0/2549359701 conn(0x7f43d40d1a90 msgr2=0x7f43d40d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.683+0000 7f43d95ac640 1 -- 172.21.15.27:0/2549359701 shutdown_connections 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.683+0000 7f43d95ac640 1 -- 172.21.15.27:0/2549359701 wait complete. 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.684+0000 7f43d95ac640 1 Processor -- start 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.684+0000 7f43d95ac640 1 -- start start 2024-03-06T07:23:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d95ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d406ed10 0x7f43d4151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d95ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d40b1020 0x7f43d415fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d95ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d40cd6c0 0x7f43d40b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d95ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40ccc60 con 0x7f43d40cd6c0 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d95ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40cc980 con 0x7f43d40b1020 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d95ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40cdaf0 con 0x7f43d406ed10 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d40cd6c0 0x7f43d40b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d40cd6c0 0x7f43d40b54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47988/0 (socket says 172.21.15.27:47988) 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d37fe640 1 -- 172.21.15.27:0/1999433418 learned_addr learned my addr 172.21.15.27:0/1999433418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d2ffd640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d406ed10 0x7f43d4151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.685+0000 7f43d27fc640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d40b1020 0x7f43d415fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43d37fe640 1 -- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d406ed10 msgr2=0x7f43d4151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43d37fe640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d406ed10 0x7f43d4151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43d37fe640 1 -- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d40b1020 msgr2=0x7f43d415fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43d37fe640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d40b1020 0x7f43d415fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43d37fe640 1 -- 172.21.15.27:0/1999433418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d40ca010 con 0x7f43d40cd6c0 2024-03-06T07:23:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43d37fe640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d40cd6c0 0x7f43d40b54a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00cad0 tx=0x7f43cc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43cb7fe640 1 -- 172.21.15.27:0/1999433418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc017070 con 0x7f43d40cd6c0 2024-03-06T07:23:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43cb7fe640 1 -- 172.21.15.27:0/1999433418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc00ac30 con 0x7f43d40cd6c0 2024-03-06T07:23:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43cb7fe640 1 -- 172.21.15.27:0/1999433418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc012400 con 0x7f43d40cd6c0 2024-03-06T07:23:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d40b5f40 con 0x7f43d40cd6c0 2024-03-06T07:23:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.686+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d40ca590 con 0x7f43d40cd6c0 2024-03-06T07:23:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.687+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d4067ff0 con 0x7f43d40cd6c0 2024-03-06T07:23:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.687+0000 7f43cb7fe640 1 -- 172.21.15.27:0/1999433418 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43cc01a070 con 0x7f43d40cd6c0 2024-03-06T07:23:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.688+0000 7f43cb7fe640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.688+0000 7f43cb7fe640 1 -- 172.21.15.27:0/1999433418 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43cc089f40 con 0x7f43d40cd6c0 2024-03-06T07:23:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.688+0000 7f43d2ffd640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.689+0000 7f43d2ffd640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009b40 tx=0x7f43c4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.692+0000 7f43cb7fe640 1 -- 172.21.15.27:0/1999433418 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43cc058770 con 0x7f43d40cd6c0 2024-03-06T07:23:21.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.941+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43d406f6f0 con 0x7f43d40cd6c0 2024-03-06T07:23:21.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.942+0000 7f43cb7fe640 1 -- 172.21.15.27:0/1999433418 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f43cc05c810 con 0x7f43d40cd6c0 2024-03-06T07:23:21.945 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:21.945 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:21.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 msgr2=0x7f43b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009b40 tx=0x7f43c4009190 comp rx=0 tx=0).stop 2024-03-06T07:23:21.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d40cd6c0 msgr2=0x7f43d40b54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d40cd6c0 0x7f43d40b54a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00cad0 tx=0x7f43cc00cf90 comp rx=0 tx=0).stop 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 shutdown_connections 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d40cd6c0 0x7f43d40b54a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43d40b1020 0x7f43d415fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 --2- 172.21.15.27:0/1999433418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43d406ed10 0x7f43d4151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 >> 172.21.15.27:0/1999433418 conn(0x7f43d40d1a90 msgr2=0x7f43d40d1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 shutdown_connections 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:21.946+0000 7f43d95ac640 1 -- 172.21.15.27:0/1999433418 wait complete. 2024-03-06T07:23:21.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:21.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:22.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.067+0000 7ffacecc9640 1 Processor -- start 2024-03-06T07:23:22.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.067+0000 7ffacecc9640 1 -- start start 2024-03-06T07:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.068+0000 7ffacecc9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffac80b2d30 0x7ffac80b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.068+0000 7ffacecc9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac80d4960 con 0x7ffac8071540 2024-03-06T07:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.068+0000 7ffacecc9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac80d36c0 con 0x7ffac80b7820 2024-03-06T07:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.068+0000 7ffacecc9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac80d3230 con 0x7ffac80b2d30 2024-03-06T07:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.068+0000 7ffacca65640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffac8071540 0x7ffac8071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48392/0 (socket says 172.21.15.27:48392) 2024-03-06T07:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.068+0000 7ffacca65640 1 -- 172.21.15.27:0/2899856188 learned_addr learned my addr 172.21.15.27:0/2899856188 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac7fff640 1 --2- 172.21.15.27:0/2899856188 >> v2:172.21.15.69:3300/0 conn(0x7ffac80b2d30 0x7ffac80b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266285840 0 0) 0x7ffac80d4960 con 0x7ffac8071540 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab8003520 con 0x7ffac8071540 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918310878 0 0) 0x7ffac80d36c0 con 0x7ffac80b7820 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac80d4960 con 0x7ffac80b7820 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3359042813 0 0) 0x7ffab8003520 con 0x7ffac8071540 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 >> v2:172.21.15.69:3300/0 conn(0x7ffac80b2d30 msgr2=0x7ffac80b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 --2- 172.21.15.27:0/2899856188 >> v2:172.21.15.69:3300/0 conn(0x7ffac80b2d30 0x7ffac80b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 >> v1:172.21.15.69:6789/0 conn(0x7ffac80b7820 legacy=0x7ffac80b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac80d2620 con 0x7ffac8071540 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7ffabc001670 con 0x7ffac8071540 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 >> v1:172.21.15.27:6789/0 conn(0x7ffac8071540 legacy=0x7ffac8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab8003b40 0x7ffab8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffab8004800 0x7ffab8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab80051c0 0x7ffab800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80d3230 con 0x7ffab8003b40 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab800c040 con 0x7ffab80051c0 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab800c240 con 0x7ffab8004800 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffacca65640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab8003b40 0x7ffab8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffacd266640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab80051c0 0x7ffab800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.069+0000 7ffac7fff640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffab8004800 0x7ffab8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.070+0000 7ffacca65640 1 -- 172.21.15.27:0/2899856188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffab8004800 msgr2=0x7ffab8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.070+0000 7ffacca65640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffab8004800 0x7ffab8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.070+0000 7ffacca65640 1 -- 172.21.15.27:0/2899856188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab80051c0 msgr2=0x7ffab800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.070+0000 7ffacca65640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab80051c0 0x7ffab800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.070+0000 7ffacd266640 1 -- 172.21.15.27:0/2899856188 reap_dead start 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.070+0000 7ffacca65640 1 -- 172.21.15.27:0/2899856188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab8003520 con 0x7ffab8003b40 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.070+0000 7ffacca65640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab8003b40 0x7ffab8003f40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00c190 tx=0x7ffab8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65c1f757abd946c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc007ce0 con 0x7ffab8003b40 2024-03-06T07:23:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffabc030070 con 0x7ffab8003b40 2024-03-06T07:23:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffac77fe640 1 -- 172.21.15.27:0/2899856188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc005d90 con 0x7ffab8003b40 2024-03-06T07:23:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffacecc9640 1 -- 172.21.15.27:0/2899856188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab8003b40 msgr2=0x7ffab8003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffacecc9640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab8003b40 0x7ffab8003f40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00c190 tx=0x7ffab8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffacecc9640 1 -- 172.21.15.27:0/2899856188 shutdown_connections 2024-03-06T07:23:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffacecc9640 1 --2- 172.21.15.27:0/2899856188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab8003b40 0x7ffab8003f40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffacecc9640 1 -- 172.21.15.27:0/2899856188 >> 172.21.15.27:0/2899856188 conn(0x7ffac80d3d60 msgr2=0x7ffac80d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffacecc9640 1 -- 172.21.15.27:0/2899856188 shutdown_connections 2024-03-06T07:23:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.071+0000 7ffacecc9640 1 -- 172.21.15.27:0/2899856188 wait complete. 2024-03-06T07:23:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.073+0000 7ffacecc9640 1 Processor -- start 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.073+0000 7ffacecc9640 1 -- start start 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacecc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab8003b40 0x7ffac814f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacecc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffac815bb20 0x7ffac81554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacecc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac8155a20 0x7ffac80cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacecc9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80d2ec0 con 0x7ffac8155a20 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacecc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80d2be0 con 0x7ffab8003b40 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacecc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80d22f0 con 0x7ffac815bb20 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacd266640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac8155a20 0x7ffac80cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacd266640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac8155a20 0x7ffac80cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48012/0 (socket says 172.21.15.27:48012) 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacd266640 1 -- 172.21.15.27:0/2117815906 learned_addr learned my addr 172.21.15.27:0/2117815906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffac7fff640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffac815bb20 0x7ffac81554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacca65640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab8003b40 0x7ffac814f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacd266640 1 -- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffac815bb20 msgr2=0x7ffac81554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacd266640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffac815bb20 0x7ffac81554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacd266640 1 -- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab8003b40 msgr2=0x7ffac814f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacd266640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab8003b40 0x7ffac814f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacd266640 1 -- 172.21.15.27:0/2117815906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac814f550 con 0x7ffac8155a20 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacca65640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab8003b40 0x7ffac814f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffac7fff640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffac815bb20 0x7ffac81554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacd266640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac8155a20 0x7ffac80cd1f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffab8004780 tx=0x7ffac000ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffac57fa640 1 -- 172.21.15.27:0/2117815906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac001a070 con 0x7ffac8155a20 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffac57fa640 1 -- 172.21.15.27:0/2117815906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0003c40 con 0x7ffac8155a20 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac81560a0 con 0x7ffac8155a20 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffac57fa640 1 -- 172.21.15.27:0/2117815906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac0005070 con 0x7ffac8155a20 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.074+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac80cb5e0 con 0x7ffac8155a20 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.075+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac80b8bf0 con 0x7ffac8155a20 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.076+0000 7ffac57fa640 1 -- 172.21.15.27:0/2117815906 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffac001d070 con 0x7ffac8155a20 2024-03-06T07:23:22.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.076+0000 7ffac57fa640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffaac06b500 0x7ffaac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.077+0000 7ffac57fa640 1 -- 172.21.15.27:0/2117815906 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffac00899c0 con 0x7ffac8155a20 2024-03-06T07:23:22.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.077+0000 7ffacca65640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffaac06b500 0x7ffaac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.077+0000 7ffacca65640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffaac06b500 0x7ffaac06d9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffabc034000 tx=0x7ffabc039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.084+0000 7ffac57fa640 1 -- 172.21.15.27:0/2117815906 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffac80b8bf0 con 0x7ffac8155a20 2024-03-06T07:23:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.268+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffac814f720 con 0x7ffac8155a20 2024-03-06T07:23:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.269+0000 7ffac57fa640 1 -- 172.21.15.27:0/2117815906 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffac00589b0 con 0x7ffac8155a20 2024-03-06T07:23:22.271 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffaac06b500 msgr2=0x7ffaac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffaac06b500 0x7ffaac06d9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffabc034000 tx=0x7ffabc039040 comp rx=0 tx=0).stop 2024-03-06T07:23:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac8155a20 msgr2=0x7ffac80cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac8155a20 0x7ffac80cd1f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffab8004780 tx=0x7ffac000ff70 comp rx=0 tx=0).stop 2024-03-06T07:23:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 shutdown_connections 2024-03-06T07:23:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffaac06b500 0x7ffaac06d9c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac8155a20 0x7ffac80cd1f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffac815bb20 0x7ffac81554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 --2- 172.21.15.27:0/2117815906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffab8003b40 0x7ffac814f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 >> 172.21.15.27:0/2117815906 conn(0x7ffac80d3d60 msgr2=0x7ffac80b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 shutdown_connections 2024-03-06T07:23:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.271+0000 7ffacecc9640 1 -- 172.21.15.27:0/2117815906 wait complete. 2024-03-06T07:23:22.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:22.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.427+0000 7f41714ec640 1 Processor -- start 2024-03-06T07:23:22.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.427+0000 7f41714ec640 1 -- start start 2024-03-06T07:23:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.428+0000 7f41714ec640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f416c15df40 0x7f416c066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.428+0000 7f41714ec640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c15e6d0 con 0x7f416c0b3ce0 2024-03-06T07:23:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.428+0000 7f41714ec640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c15e9b0 con 0x7f416c06e3a0 2024-03-06T07:23:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.428+0000 7f41714ec640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c15f000 con 0x7f416c15df40 2024-03-06T07:23:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.428+0000 7f416a7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f416c0b3ce0 0x7f416c06dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48406/0 (socket says 172.21.15.27:48406) 2024-03-06T07:23:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.428+0000 7f416a7fc640 1 -- 172.21.15.27:0/1242975776 learned_addr learned my addr 172.21.15.27:0/1242975776 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.428+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381978914 0 0) 0x7f416c15e6d0 con 0x7f416c0b3ce0 2024-03-06T07:23:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.428+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4154003520 con 0x7f416c0b3ce0 2024-03-06T07:23:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.428+0000 7f416affd640 1 --2- 172.21.15.27:0/1242975776 >> v2:172.21.15.69:3300/0 conn(0x7f416c15df40 0x7f416c066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1328280420 0 0) 0x7f4154003520 con 0x7f416c0b3ce0 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 >> v2:172.21.15.69:3300/0 conn(0x7f416c15df40 msgr2=0x7f416c066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 --2- 172.21.15.27:0/1242975776 >> v2:172.21.15.69:3300/0 conn(0x7f416c15df40 0x7f416c066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 >> v1:172.21.15.69:6789/0 conn(0x7f416c06e3a0 legacy=0x7f416c161950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c15ec90 con 0x7f416c0b3ce0 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f415c001670 con 0x7f416c0b3ce0 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 >> v1:172.21.15.27:6789/0 conn(0x7f416c0b3ce0 legacy=0x7f416c06dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4154003a00 0x7f4154003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f4154004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41540051c0 0x7f415400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c15f000 con 0x7f4154004800 2024-03-06T07:23:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415400c0c0 con 0x7f4154003a00 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415400c2c0 con 0x7f41540051c0 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416a7fc640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f4154004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416affd640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4154003a00 0x7f4154003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416b7fe640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41540051c0 0x7f415400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416a7fc640 1 -- 172.21.15.27:0/1242975776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41540051c0 msgr2=0x7f415400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416a7fc640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41540051c0 0x7f415400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416a7fc640 1 -- 172.21.15.27:0/1242975776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4154003a00 msgr2=0x7f4154003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416a7fc640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4154003a00 0x7f4154003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416a7fc640 1 -- 172.21.15.27:0/1242975776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154003520 con 0x7f4154004800 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416affd640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4154003a00 0x7f4154003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.429+0000 7f416b7fe640 1 -- 172.21.15.27:0/1242975776 reap_dead start 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.430+0000 7f416a7fc640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f4154004c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f415c007fd0 tx=0x7f4154005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e948cd49b194ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.430+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c007c20 con 0x7f4154004800 2024-03-06T07:23:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.430+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415c030070 con 0x7f4154004800 2024-03-06T07:23:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.430+0000 7f4169ffb640 1 -- 172.21.15.27:0/1242975776 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c005d70 con 0x7f4154004800 2024-03-06T07:23:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.431+0000 7f41714ec640 1 -- 172.21.15.27:0/1242975776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 msgr2=0x7f4154004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.431+0000 7f41714ec640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f4154004c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f415c007fd0 tx=0x7f4154005830 comp rx=0 tx=0).stop 2024-03-06T07:23:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.431+0000 7f41714ec640 1 -- 172.21.15.27:0/1242975776 shutdown_connections 2024-03-06T07:23:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.431+0000 7f41714ec640 1 --2- 172.21.15.27:0/1242975776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f4154004c80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.431+0000 7f41714ec640 1 -- 172.21.15.27:0/1242975776 >> 172.21.15.27:0/1242975776 conn(0x7f416c0d09c0 msgr2=0x7f416c0d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.431+0000 7f41714ec640 1 -- 172.21.15.27:0/1242975776 shutdown_connections 2024-03-06T07:23:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.431+0000 7f41714ec640 1 -- 172.21.15.27:0/1242975776 wait complete. 2024-03-06T07:23:22.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.432+0000 7f41714ec640 1 Processor -- start 2024-03-06T07:23:22.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.432+0000 7f41714ec640 1 -- start start 2024-03-06T07:23:22.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f41714ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f416c142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f41714ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f416c158320 0x7f416c159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f41714ec640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f416c156280 0x7f416c145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f41714ec640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c06de10 con 0x7f4154004800 2024-03-06T07:23:22.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f41714ec640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c06dc90 con 0x7f416c158320 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f41714ec640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416c06df90 con 0x7f416c156280 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f416affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f416c142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f416affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f416c142fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48024/0 (socket says 172.21.15.27:48024) 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f416affd640 1 -- 172.21.15.27:0/368227991 learned_addr learned my addr 172.21.15.27:0/368227991 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f416affd640 1 -- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f416c156280 msgr2=0x7f416c145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f416affd640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f416c156280 0x7f416c145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f416affd640 1 -- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f416c158320 msgr2=0x7f416c159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f416affd640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f416c158320 0x7f416c159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f416affd640 1 -- 172.21.15.27:0/368227991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c1588b0 con 0x7f4154004800 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.433+0000 7f416affd640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f416c142fc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4160011510 tx=0x7f41600119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.434+0000 7f41537fe640 1 -- 172.21.15.27:0/368227991 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4160015870 con 0x7f4154004800 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.434+0000 7f41537fe640 1 -- 172.21.15.27:0/368227991 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4160015a10 con 0x7f4154004800 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.434+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c158b90 con 0x7f4154004800 2024-03-06T07:23:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.434+0000 7f41537fe640 1 -- 172.21.15.27:0/368227991 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4160015e00 con 0x7f4154004800 2024-03-06T07:23:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.434+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c159150 con 0x7f4154004800 2024-03-06T07:23:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.435+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c066d00 con 0x7f4154004800 2024-03-06T07:23:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.435+0000 7f41537fe640 1 -- 172.21.15.27:0/368227991 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f416001d400 con 0x7f4154004800 2024-03-06T07:23:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.435+0000 7f41537fe640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f415806b500 0x7f415806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.436+0000 7f41537fe640 1 -- 172.21.15.27:0/368227991 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f416008aa30 con 0x7f4154004800 2024-03-06T07:23:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.436+0000 7f416a7fc640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f415806b500 0x7f415806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.436+0000 7f416a7fc640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f415806b500 0x7f415806d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f415c034000 tx=0x7f415c02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.440+0000 7f41537fe640 1 -- 172.21.15.27:0/368227991 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f416c066d00 con 0x7f4154004800 2024-03-06T07:23:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.619+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f416c162710 con 0x7f4154004800 2024-03-06T07:23:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.620+0000 7f41537fe640 1 -- 172.21.15.27:0/368227991 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f416c162710 con 0x7f4154004800 2024-03-06T07:23:22.623 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f415806b500 msgr2=0x7f415806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f415806b500 0x7f415806d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f415c034000 tx=0x7f415c02e000 comp rx=0 tx=0).stop 2024-03-06T07:23:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 msgr2=0x7f416c142fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f416c142fc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4160011510 tx=0x7f41600119d0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 shutdown_connections 2024-03-06T07:23:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f415806b500 0x7f415806d9c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f416c156280 0x7f416c145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f416c158320 0x7f416c159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 --2- 172.21.15.27:0/368227991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4154004800 0x7f416c142fc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 >> 172.21.15.27:0/368227991 conn(0x7f416c0d09c0 msgr2=0x7f416c0ade70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 shutdown_connections 2024-03-06T07:23:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.622+0000 7f41714ec640 1 -- 172.21.15.27:0/368227991 wait complete. 2024-03-06T07:23:22.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.742+0000 7fc2fd33e640 1 Processor -- start 2024-03-06T07:23:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2fd33e640 1 -- start start 2024-03-06T07:23:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2fd33e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2f815dd90 0x7fc2f80ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2fd33e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2f80d29d0 0x7fc2f80d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2fd33e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f80d3740 con 0x7fc2f80d4720 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2fd33e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f80d3400 con 0x7fc2f815dd90 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2fd33e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f80d3120 con 0x7fc2f80d29d0 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2f6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2f815dd90 0x7fc2f80ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2f77fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc2f80d4720 0x7fc2f80b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48418/0 (socket says 172.21.15.27:48418) 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2f6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2f815dd90 0x7fc2f80ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48040/0 (socket says 172.21.15.27:48040) 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.743+0000 7fc2f77fe640 1 -- 172.21.15.27:0/576177046 learned_addr learned my addr 172.21.15.27:0/576177046 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.744+0000 7fc2ee7fc640 1 --2- 172.21.15.27:0/576177046 >> v2:172.21.15.69:3300/0 conn(0x7fc2f80d29d0 0x7fc2f80d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.744+0000 7fc2f6ffd640 1 -- 172.21.15.27:0/576177046 >> v2:172.21.15.69:3300/0 conn(0x7fc2f80d29d0 msgr2=0x7fc2f80d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.744+0000 7fc2f6ffd640 1 --2- 172.21.15.27:0/576177046 >> v2:172.21.15.69:3300/0 conn(0x7fc2f80d29d0 0x7fc2f80d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.744+0000 7fc2f6ffd640 1 -- 172.21.15.27:0/576177046 >> v1:172.21.15.27:6789/0 conn(0x7fc2f80d4720 legacy=0x7fc2f80b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.744+0000 7fc2f6ffd640 1 -- 172.21.15.27:0/576177046 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f80d39e0 con 0x7fc2f815dd90 2024-03-06T07:23:22.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.744+0000 7fc2f67fc640 1 -- 172.21.15.27:0/576177046 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303912653 0 0) 0x7fc2f80d3740 con 0x7fc2f80d4720 2024-03-06T07:23:22.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2f6ffd640 1 --2- 172.21.15.27:0/576177046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f815dd90 0x7fc2f80ceea0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80724d0 tx=0x7fc2e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff3ad0bedfd8be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2f67fc640 1 -- 172.21.15.27:0/576177046 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8034070 con 0x7fc2f815dd90 2024-03-06T07:23:22.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2f67fc640 1 -- 172.21.15.27:0/576177046 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e8030610 con 0x7fc2f815dd90 2024-03-06T07:23:22.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2f67fc640 1 -- 172.21.15.27:0/576177046 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e80309e0 con 0x7fc2f815dd90 2024-03-06T07:23:22.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2fd33e640 1 -- 172.21.15.27:0/576177046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f815dd90 msgr2=0x7fc2f80ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2fd33e640 1 --2- 172.21.15.27:0/576177046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f815dd90 0x7fc2f80ceea0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80724d0 tx=0x7fc2e802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:22.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2fd33e640 1 -- 172.21.15.27:0/576177046 shutdown_connections 2024-03-06T07:23:22.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2fd33e640 1 --2- 172.21.15.27:0/576177046 >> v2:172.21.15.69:3300/0 conn(0x7fc2f80d29d0 0x7fc2f80d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2fd33e640 1 --2- 172.21.15.27:0/576177046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f815dd90 0x7fc2f80ceea0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2fd33e640 1 -- 172.21.15.27:0/576177046 >> 172.21.15.27:0/576177046 conn(0x7fc2f8071960 msgr2=0x7fc2f8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2fd33e640 1 -- 172.21.15.27:0/576177046 shutdown_connections 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.745+0000 7fc2fd33e640 1 -- 172.21.15.27:0/576177046 wait complete. 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.747+0000 7fc2fd33e640 1 Processor -- start 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.747+0000 7fc2fd33e640 1 -- start start 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.747+0000 7fc2fd33e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f80d29d0 0x7fc2f8149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.747+0000 7fc2fd33e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f80d4720 0x7fc2f80c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.747+0000 7fc2fd33e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f815dd90 0x7fc2f814a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.747+0000 7fc2fd33e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80d2df0 con 0x7fc2f80d29d0 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.747+0000 7fc2fd33e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f815e170 con 0x7fc2f815dd90 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.747+0000 7fc2fd33e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80b7420 con 0x7fc2f80d4720 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f80d29d0 0x7fc2f8149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f80d29d0 0x7fc2f8149ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48044/0 (socket says 172.21.15.27:48044) 2024-03-06T07:23:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f6ffd640 1 -- 172.21.15.27:0/2041356191 learned_addr learned my addr 172.21.15.27:0/2041356191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2ee7fc640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f80d4720 0x7fc2f80c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f77fe640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f815dd90 0x7fc2f814a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f6ffd640 1 -- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f80d4720 msgr2=0x7fc2f80c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f6ffd640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f80d4720 0x7fc2f80c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f6ffd640 1 -- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f815dd90 msgr2=0x7fc2f814a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f6ffd640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f815dd90 0x7fc2f814a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f6ffd640 1 -- 172.21.15.27:0/2041356191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f80c8cb0 con 0x7fc2f80d29d0 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2ee7fc640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f80d4720 0x7fc2f80c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2f6ffd640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f80d29d0 0x7fc2f8149ba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009890 tx=0x7fc2e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2effff640 1 -- 172.21.15.27:0/2041356191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8047070 con 0x7fc2f80d29d0 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f80c8f90 con 0x7fc2f80d29d0 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2effff640 1 -- 172.21.15.27:0/2041356191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e8030410 con 0x7fc2f80d29d0 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2effff640 1 -- 172.21.15.27:0/2041356191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8034040 con 0x7fc2f80d29d0 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.748+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f80c9550 con 0x7fc2f80d29d0 2024-03-06T07:23:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.749+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f8072a30 con 0x7fc2f80d29d0 2024-03-06T07:23:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.750+0000 7fc2effff640 1 -- 172.21.15.27:0/2041356191 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2e804a070 con 0x7fc2f80d29d0 2024-03-06T07:23:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.751+0000 7fc2effff640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc2d806b4d0 0x7fc2d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.751+0000 7fc2effff640 1 -- 172.21.15.27:0/2041356191 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2e80abbe0 con 0x7fc2f80d29d0 2024-03-06T07:23:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.751+0000 7fc2ee7fc640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc2d806b4d0 0x7fc2d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.752+0000 7fc2ee7fc640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc2d806b4d0 0x7fc2d806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc2f815d5f0 tx=0x7fc2e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.755+0000 7fc2effff640 1 -- 172.21.15.27:0/2041356191 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2e80305e0 con 0x7fc2f80d29d0 2024-03-06T07:23:22.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.935+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc2f815d740 con 0x7fc2f80d29d0 2024-03-06T07:23:22.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.936+0000 7fc2effff640 1 -- 172.21.15.27:0/2041356191 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc2e807a9e0 con 0x7fc2f80d29d0 2024-03-06T07:23:22.938 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc2d806b4d0 msgr2=0x7fc2d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc2d806b4d0 0x7fc2d806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc2f815d5f0 tx=0x7fc2e0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f80d29d0 msgr2=0x7fc2f8149ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f80d29d0 0x7fc2f8149ba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009890 tx=0x7fc2e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 shutdown_connections 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc2d806b4d0 0x7fc2d806d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2f815dd90 0x7fc2f814a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2f80d4720 0x7fc2f80c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 --2- 172.21.15.27:0/2041356191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2f80d29d0 0x7fc2f8149ba0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 >> 172.21.15.27:0/2041356191 conn(0x7fc2f8071960 msgr2=0x7fc2f80b8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 shutdown_connections 2024-03-06T07:23:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:22.938+0000 7fc2fd33e640 1 -- 172.21.15.27:0/2041356191 wait complete. 2024-03-06T07:23:22.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:23.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.057+0000 7f486b1a0640 1 Processor -- start 2024-03-06T07:23:23.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.057+0000 7f486b1a0640 1 -- start start 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486b1a0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4864146940 0x7f48640ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486b1a0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4864067410 0x7f4864072490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486b1a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48640b5840 con 0x7f4864072030 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486b1a0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f486406f610 con 0x7f4864146940 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486b1a0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48640b5b20 con 0x7f4864067410 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486a19e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4864146940 0x7f48640ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486a19e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4864146940 0x7f48640ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48050/0 (socket says 172.21.15.27:48050) 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486a19e640 1 -- 172.21.15.27:0/21145739 learned_addr learned my addr 172.21.15.27:0/21145739 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486a99f640 1 --2- 172.21.15.27:0/21145739 >> v2:172.21.15.69:3300/0 conn(0x7f4864067410 0x7f4864072490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486a19e640 1 -- 172.21.15.27:0/21145739 >> v2:172.21.15.69:3300/0 conn(0x7f4864067410 msgr2=0x7f4864072490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486a19e640 1 --2- 172.21.15.27:0/21145739 >> v2:172.21.15.69:3300/0 conn(0x7f4864067410 0x7f4864072490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486a19e640 1 -- 172.21.15.27:0/21145739 >> v1:172.21.15.69:6789/0 conn(0x7f4864072030 legacy=0x7f4864066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486a19e640 1 -- 172.21.15.27:0/21145739 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486406d6c0 con 0x7f4864146940 2024-03-06T07:23:23.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.058+0000 7f486a99f640 1 --2- 172.21.15.27:0/21145739 >> v2:172.21.15.69:3300/0 conn(0x7f4864067410 0x7f4864072490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.059+0000 7f486a19e640 1 --2- 172.21.15.27:0/21145739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864146940 0x7f48640ad8b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4860009870 tx=0x7f486002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5821131b6657f972 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.059+0000 7f486919c640 1 -- 172.21.15.27:0/21145739 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860035020 con 0x7f4864146940 2024-03-06T07:23:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486919c640 1 -- 172.21.15.27:0/21145739 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4860031870 con 0x7f4864146940 2024-03-06T07:23:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486919c640 1 -- 172.21.15.27:0/21145739 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860031c40 con 0x7f4864146940 2024-03-06T07:23:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486b1a0640 1 -- 172.21.15.27:0/21145739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864146940 msgr2=0x7f48640ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486b1a0640 1 --2- 172.21.15.27:0/21145739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864146940 0x7f48640ad8b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4860009870 tx=0x7f486002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486b1a0640 1 -- 172.21.15.27:0/21145739 shutdown_connections 2024-03-06T07:23:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486b1a0640 1 --2- 172.21.15.27:0/21145739 >> v2:172.21.15.69:3300/0 conn(0x7f4864067410 0x7f4864072490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486b1a0640 1 --2- 172.21.15.27:0/21145739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864146940 0x7f48640ad8b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486b1a0640 1 -- 172.21.15.27:0/21145739 >> 172.21.15.27:0/21145739 conn(0x7f48640d0820 msgr2=0x7f48640d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486b1a0640 1 -- 172.21.15.27:0/21145739 shutdown_connections 2024-03-06T07:23:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.060+0000 7f486b1a0640 1 -- 172.21.15.27:0/21145739 wait complete. 2024-03-06T07:23:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.061+0000 7f486b1a0640 1 Processor -- start 2024-03-06T07:23:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.061+0000 7f486b1a0640 1 -- start start 2024-03-06T07:23:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486b1a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864067410 0x7f48641e7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486b1a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4864072030 0x7f4864160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486b1a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4864146940 0x7f4864131410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486b1a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48640addf0 con 0x7f4864067410 2024-03-06T07:23:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486b1a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48640b54f0 con 0x7f4864146940 2024-03-06T07:23:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486b1a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48640adf70 con 0x7f4864072030 2024-03-06T07:23:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a19e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864067410 0x7f48641e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a19e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864067410 0x7f48641e7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48060/0 (socket says 172.21.15.27:48060) 2024-03-06T07:23:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a19e640 1 -- 172.21.15.27:0/851509799 learned_addr learned my addr 172.21.15.27:0/851509799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486999d640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4864072030 0x7f4864160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a99f640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4864146940 0x7f4864131410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a19e640 1 -- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4864072030 msgr2=0x7f4864160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a19e640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4864072030 0x7f4864160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a19e640 1 -- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4864146940 msgr2=0x7f4864131410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a19e640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4864146940 0x7f4864131410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a19e640 1 -- 172.21.15.27:0/851509799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4864130370 con 0x7f4864067410 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486999d640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4864072030 0x7f4864160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.062+0000 7f486a99f640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4864146940 0x7f4864131410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.063+0000 7f486a19e640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864067410 0x7f48641e7170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f486002b950 tx=0x7f48600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.063+0000 7f485affd640 1 -- 172.21.15.27:0/851509799 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860035040 con 0x7f4864067410 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.063+0000 7f485affd640 1 -- 172.21.15.27:0/851509799 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4860030d00 con 0x7f4864067410 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.063+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4864130650 con 0x7f4864067410 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.063+0000 7f485affd640 1 -- 172.21.15.27:0/851509799 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48600456c0 con 0x7f4864067410 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.063+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4864130c10 con 0x7f4864067410 2024-03-06T07:23:23.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.064+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486404f330 con 0x7f4864067410 2024-03-06T07:23:23.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.064+0000 7f485affd640 1 -- 172.21.15.27:0/851509799 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4860038020 con 0x7f4864067410 2024-03-06T07:23:23.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.065+0000 7f485affd640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484806b4d0 0x7f484806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.065+0000 7f485affd640 1 -- 172.21.15.27:0/851509799 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48600ac440 con 0x7f4864067410 2024-03-06T07:23:23.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.065+0000 7f486999d640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484806b4d0 0x7f484806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.066+0000 7f486999d640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484806b4d0 0x7f484806d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f48640c5c60 tx=0x7f4854009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.069+0000 7f485affd640 1 -- 172.21.15.27:0/851509799 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f486007b350 con 0x7f4864067410 2024-03-06T07:23:23.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.265+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f48641614f0 con 0x7f4864067410 2024-03-06T07:23:23.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.266+0000 7f485affd640 1 -- 172.21.15.27:0/851509799 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f486007ac50 con 0x7f4864067410 2024-03-06T07:23:23.268 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:23:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484806b4d0 msgr2=0x7f484806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484806b4d0 0x7f484806d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f48640c5c60 tx=0x7f4854009040 comp rx=0 tx=0).stop 2024-03-06T07:23:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864067410 msgr2=0x7f48641e7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864067410 0x7f48641e7170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f486002b950 tx=0x7f48600046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 shutdown_connections 2024-03-06T07:23:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484806b4d0 0x7f484806d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4864146940 0x7f4864131410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4864072030 0x7f4864160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 --2- 172.21.15.27:0/851509799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4864067410 0x7f48641e7170 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.267+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 >> 172.21.15.27:0/851509799 conn(0x7f48640d0820 msgr2=0x7f48640b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.268+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 shutdown_connections 2024-03-06T07:23:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.268+0000 7f486b1a0640 1 -- 172.21.15.27:0/851509799 wait complete. 2024-03-06T07:23:23.281 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:23:23.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:23.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:23.313 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:23.313 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:23:23.375 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:23:23.376 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:23.395 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:23:23.395 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:23.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:23.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.559+0000 7f3c69d83640 1 Processor -- start 2024-03-06T07:23:23.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.559+0000 7f3c69d83640 1 -- start start 2024-03-06T07:23:23.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.559+0000 7f3c69d83640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c6406df30 0x7f3c640ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.559+0000 7f3c69d83640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64071240 con 0x7f3c64158520 2024-03-06T07:23:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.559+0000 7f3c69d83640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64071520 con 0x7f3c640aec40 2024-03-06T07:23:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.559+0000 7f3c69d83640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c640af0f0 con 0x7f3c6406df30 2024-03-06T07:23:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.560+0000 7f3c62ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c6406df30 0x7f3c640ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.560+0000 7f3c637fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3c64158520 0x7f3c6406d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41386/0 (socket says 172.21.15.27:41386) 2024-03-06T07:23:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.560+0000 7f3c637fe640 1 -- 172.21.15.27:0/1817924558 learned_addr learned my addr 172.21.15.27:0/1817924558 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.560+0000 7f3c62ffd640 1 -- 172.21.15.27:0/1817924558 >> v1:172.21.15.69:6789/0 conn(0x7f3c640aec40 legacy=0x7f3c640b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.560+0000 7f3c62ffd640 1 -- 172.21.15.27:0/1817924558 >> v1:172.21.15.27:6789/0 conn(0x7f3c64158520 legacy=0x7f3c6406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.560+0000 7f3c627fc640 1 -- 172.21.15.27:0/1817924558 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984447219 0 0) 0x7f3c64071240 con 0x7f3c64158520 2024-03-06T07:23:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.560+0000 7f3c62ffd640 1 -- 172.21.15.27:0/1817924558 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c64071800 con 0x7f3c6406df30 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.561+0000 7f3c62ffd640 1 --2- 172.21.15.27:0/1817924558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c6406df30 0x7f3c640ae700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3c580097d0 tx=0x7f3c5802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6636150d3964ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.561+0000 7f3c627fc640 1 -- 172.21.15.27:0/1817924558 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58034070 con 0x7f3c6406df30 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.561+0000 7f3c627fc640 1 -- 172.21.15.27:0/1817924558 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c58030610 con 0x7f3c6406df30 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.561+0000 7f3c627fc640 1 -- 172.21.15.27:0/1817924558 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c580309e0 con 0x7f3c6406df30 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.562+0000 7f3c69d83640 1 -- 172.21.15.27:0/1817924558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c6406df30 msgr2=0x7f3c640ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.562+0000 7f3c69d83640 1 --2- 172.21.15.27:0/1817924558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c6406df30 0x7f3c640ae700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3c580097d0 tx=0x7f3c5802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.562+0000 7f3c69d83640 1 -- 172.21.15.27:0/1817924558 shutdown_connections 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.562+0000 7f3c69d83640 1 --2- 172.21.15.27:0/1817924558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c6406df30 0x7f3c640ae700 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.562+0000 7f3c69d83640 1 -- 172.21.15.27:0/1817924558 >> 172.21.15.27:0/1817924558 conn(0x7f3c64066d00 msgr2=0x7f3c640670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.562+0000 7f3c69d83640 1 -- 172.21.15.27:0/1817924558 shutdown_connections 2024-03-06T07:23:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.562+0000 7f3c69d83640 1 -- 172.21.15.27:0/1817924558 wait complete. 2024-03-06T07:23:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.563+0000 7f3c69d83640 1 Processor -- start 2024-03-06T07:23:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.563+0000 7f3c69d83640 1 -- start start 2024-03-06T07:23:23.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.564+0000 7f3c69d83640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c6406df30 0x7f3c64154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.564+0000 7f3c69d83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c640aec40 0x7f3c64158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.564+0000 7f3c69d83640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c64158520 0x7f3c64148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.564+0000 7f3c69d83640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64158900 con 0x7f3c64158520 2024-03-06T07:23:23.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.564+0000 7f3c69d83640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64071b70 con 0x7f3c6406df30 2024-03-06T07:23:23.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.564+0000 7f3c69d83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64158a80 con 0x7f3c640aec40 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.564+0000 7f3c63fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c64158520 0x7f3c64148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.564+0000 7f3c63fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c64158520 0x7f3c64148f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48356/0 (socket says 172.21.15.27:48356) 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.564+0000 7f3c63fff640 1 -- 172.21.15.27:0/3951845448 learned_addr learned my addr 172.21.15.27:0/3951845448 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c637fe640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c6406df30 0x7f3c64154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c62ffd640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c640aec40 0x7f3c64158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c63fff640 1 -- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c640aec40 msgr2=0x7f3c64158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c63fff640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c640aec40 0x7f3c64158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c63fff640 1 -- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c6406df30 msgr2=0x7f3c64154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c63fff640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c6406df30 0x7f3c64154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c63fff640 1 -- 172.21.15.27:0/3951845448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c641f7160 con 0x7f3c64158520 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c63fff640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c64158520 0x7f3c64148f40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00c7a0 tx=0x7f3c5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c4bfff640 1 -- 172.21.15.27:0/3951845448 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c003b70 con 0x7f3c64158520 2024-03-06T07:23:23.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c4bfff640 1 -- 172.21.15.27:0/3951845448 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c003d30 con 0x7f3c64158520 2024-03-06T07:23:23.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c4bfff640 1 -- 172.21.15.27:0/3951845448 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c00aa90 con 0x7f3c64158520 2024-03-06T07:23:23.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c641f7440 con 0x7f3c64158520 2024-03-06T07:23:23.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.565+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c641553e0 con 0x7f3c64158520 2024-03-06T07:23:23.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.567+0000 7f3c4bfff640 1 -- 172.21.15.27:0/3951845448 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c5c00ac30 con 0x7f3c64158520 2024-03-06T07:23:23.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.567+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c64067ff0 con 0x7f3c64158520 2024-03-06T07:23:23.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.567+0000 7f3c4bfff640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c5006b4d0 0x7f3c5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.567+0000 7f3c4bfff640 1 -- 172.21.15.27:0/3951845448 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c5c08a000 con 0x7f3c64158520 2024-03-06T07:23:23.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.567+0000 7f3c637fe640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c5006b4d0 0x7f3c5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.568+0000 7f3c637fe640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c5006b4d0 0x7f3c5006d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3c54007f70 tx=0x7f3c54007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.572+0000 7f3c4bfff640 1 -- 172.21.15.27:0/3951845448 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c5c058830 con 0x7f3c64158520 2024-03-06T07:23:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.753+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c6406dab0 con 0x7f3c64158520 2024-03-06T07:23:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.755+0000 7f3c4bfff640 1 -- 172.21.15.27:0/3951845448 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f3c5c05c8d0 con 0x7f3c64158520 2024-03-06T07:23:23.758 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:23.758 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c5006b4d0 msgr2=0x7f3c5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c5006b4d0 0x7f3c5006d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3c54007f70 tx=0x7f3c54007f00 comp rx=0 tx=0).stop 2024-03-06T07:23:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c64158520 msgr2=0x7f3c64148f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c64158520 0x7f3c64148f40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00c7a0 tx=0x7f3c5c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 shutdown_connections 2024-03-06T07:23:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c5006b4d0 0x7f3c5006d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c64158520 0x7f3c64148f40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c640aec40 0x7f3c64158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 --2- 172.21.15.27:0/3951845448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c6406df30 0x7f3c64154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 >> 172.21.15.27:0/3951845448 conn(0x7f3c64066d00 msgr2=0x7f3c64070c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 shutdown_connections 2024-03-06T07:23:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.757+0000 7f3c69d83640 1 -- 172.21.15.27:0/3951845448 wait complete. 2024-03-06T07:23:23.771 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:23.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.886+0000 7f8390d3a640 1 Processor -- start 2024-03-06T07:23:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.886+0000 7f8390d3a640 1 -- start start 2024-03-06T07:23:23.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.887+0000 7f8390d3a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f838c15f500 0x7f838c0ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.887+0000 7f8390d3a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838c0cd310 con 0x7f838c0c5e30 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.887+0000 7f8390d3a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838c0cd5b0 con 0x7f838c0cf740 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.887+0000 7f8390d3a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838c15f8e0 con 0x7f838c15f500 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838ad9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f838c0c5e30 0x7f838c0c6280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41392/0 (socket says 172.21.15.27:41392) 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838ad9d640 1 -- 172.21.15.27:0/977244142 learned_addr learned my addr 172.21.15.27:0/977244142 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838a59c640 1 --2- 172.21.15.27:0/977244142 >> v2:172.21.15.69:3300/0 conn(0x7f838c15f500 0x7f838c0ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838959a640 1 -- 172.21.15.27:0/977244142 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8927854 0 0) 0x7f838c0cd310 con 0x7f838c0c5e30 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838959a640 1 -- 172.21.15.27:0/977244142 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8374003520 con 0x7f838c0c5e30 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838959a640 1 -- 172.21.15.27:0/977244142 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901092560 0 0) 0x7f838c0cd5b0 con 0x7f838c0cf740 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838959a640 1 -- 172.21.15.27:0/977244142 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f838c0cd310 con 0x7f838c0cf740 2024-03-06T07:23:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838959a640 1 -- 172.21.15.27:0/977244142 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2307552293 0 0) 0x7f8374003520 con 0x7f838c0c5e30 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838a59c640 1 -- 172.21.15.27:0/977244142 >> v1:172.21.15.69:6789/0 conn(0x7f838c0cf740 legacy=0x7f838c0ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838a59c640 1 -- 172.21.15.27:0/977244142 >> v1:172.21.15.27:6789/0 conn(0x7f838c0c5e30 legacy=0x7f838c0c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.888+0000 7f838a59c640 1 -- 172.21.15.27:0/977244142 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0cd890 con 0x7f838c15f500 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.889+0000 7f838a59c640 1 --2- 172.21.15.27:0/977244142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f838c15f500 0x7f838c0ce300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f837c0097a0 tx=0x7f837c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76f017b1bd88b1b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.889+0000 7f838959a640 1 -- 172.21.15.27:0/977244142 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c037070 con 0x7f838c15f500 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.889+0000 7f838959a640 1 -- 172.21.15.27:0/977244142 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c030510 con 0x7f838c15f500 2024-03-06T07:23:23.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.889+0000 7f838959a640 1 -- 172.21.15.27:0/977244142 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384001a10 con 0x7f838c15f500 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.890+0000 7f8390d3a640 1 -- 172.21.15.27:0/977244142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f838c15f500 msgr2=0x7f838c0ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.890+0000 7f8390d3a640 1 --2- 172.21.15.27:0/977244142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f838c15f500 0x7f838c0ce300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f837c0097a0 tx=0x7f837c02b790 comp rx=0 tx=0).stop 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.890+0000 7f8390d3a640 1 -- 172.21.15.27:0/977244142 shutdown_connections 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.890+0000 7f8390d3a640 1 --2- 172.21.15.27:0/977244142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f838c15f500 0x7f838c0ce300 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.890+0000 7f8390d3a640 1 -- 172.21.15.27:0/977244142 >> 172.21.15.27:0/977244142 conn(0x7f838c0b3700 msgr2=0x7f838c0b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.890+0000 7f8390d3a640 1 -- 172.21.15.27:0/977244142 shutdown_connections 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.890+0000 7f8390d3a640 1 -- 172.21.15.27:0/977244142 wait complete. 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.891+0000 7f8390d3a640 1 Processor -- start 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.891+0000 7f8390d3a640 1 -- start start 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f8390d3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f838c0c5e30 0x7f838c0b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f8390d3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f838c0cf740 0x7f838c0b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f8390d3a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f838c15f500 0x7f838c157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f8390d3a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0c59a0 con 0x7f838c15f500 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f8390d3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0c56c0 con 0x7f838c0cf740 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f8390d3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0cef50 con 0x7f838c0c5e30 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f838c15f500 0x7f838c157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f838c15f500 0x7f838c157ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48366/0 (socket says 172.21.15.27:48366) 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838ad9d640 1 -- 172.21.15.27:0/527179047 learned_addr learned my addr 172.21.15.27:0/527179047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f8389d9b640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f838c0cf740 0x7f838c0b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838a59c640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f838c0c5e30 0x7f838c0b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838ad9d640 1 -- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f838c0c5e30 msgr2=0x7f838c0b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838ad9d640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f838c0c5e30 0x7f838c0b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838ad9d640 1 -- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f838c0cf740 msgr2=0x7f838c0b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838ad9d640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f838c0cf740 0x7f838c0b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838ad9d640 1 -- 172.21.15.27:0/527179047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0b5400 con 0x7f838c15f500 2024-03-06T07:23:23.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.892+0000 7f838ad9d640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f838c15f500 0x7f838c157ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f838400cac0 tx=0x7f838400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.893+0000 7f837affd640 1 -- 172.21.15.27:0/527179047 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384018070 con 0x7f838c15f500 2024-03-06T07:23:23.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.893+0000 7f837affd640 1 -- 172.21.15.27:0/527179047 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838400a6a0 con 0x7f838c15f500 2024-03-06T07:23:23.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.893+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838c0b56e0 con 0x7f838c15f500 2024-03-06T07:23:23.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.893+0000 7f837affd640 1 -- 172.21.15.27:0/527179047 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838400ab10 con 0x7f838c15f500 2024-03-06T07:23:23.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.893+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838c0b5ca0 con 0x7f838c15f500 2024-03-06T07:23:23.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.894+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838c09fcf0 con 0x7f838c15f500 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.895+0000 7f837affd640 1 -- 172.21.15.27:0/527179047 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f838401b070 con 0x7f838c15f500 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.895+0000 7f837affd640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f836c06b4d0 0x7f836c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.895+0000 7f837affd640 1 -- 172.21.15.27:0/527179047 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83840899e0 con 0x7f838c15f500 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.895+0000 7f838a59c640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f836c06b4d0 0x7f836c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.896+0000 7f838a59c640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f836c06b4d0 0x7f836c06d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f837c034000 tx=0x7f837c02f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:23.905+0000 7f837affd640 1 -- 172.21.15.27:0/527179047 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83840589b0 con 0x7f838c15f500 2024-03-06T07:23:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.084+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f838c04f410 con 0x7f838c15f500 2024-03-06T07:23:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.085+0000 7f837affd640 1 -- 172.21.15.27:0/527179047 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83840582b0 con 0x7f838c15f500 2024-03-06T07:23:24.088 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f836c06b4d0 msgr2=0x7f836c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f836c06b4d0 0x7f836c06d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f837c034000 tx=0x7f837c02f000 comp rx=0 tx=0).stop 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f838c15f500 msgr2=0x7f838c157ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f838c15f500 0x7f838c157ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f838400cac0 tx=0x7f838400cf80 comp rx=0 tx=0).stop 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 shutdown_connections 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f836c06b4d0 0x7f836c06d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f838c15f500 0x7f838c157ea0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f838c0cf740 0x7f838c0b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 --2- 172.21.15.27:0/527179047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f838c0c5e30 0x7f838c0b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 >> 172.21.15.27:0/527179047 conn(0x7f838c0b3700 msgr2=0x7f838c06ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 shutdown_connections 2024-03-06T07:23:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.087+0000 7f8390d3a640 1 -- 172.21.15.27:0/527179047 wait complete. 2024-03-06T07:23:24.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.208+0000 7fc266d65640 1 Processor -- start 2024-03-06T07:23:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.209+0000 7fc266d65640 1 -- start start 2024-03-06T07:23:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.209+0000 7fc266d65640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2600b2cd0 0x7fc2600b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.209+0000 7fc266d65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc26006b2c0 0x7fc26006b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.209+0000 7fc266d65640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc260161a40 con 0x7fc2600cdcd0 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.209+0000 7fc266d65640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2600d11e0 con 0x7fc2600b2cd0 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.209+0000 7fc266d65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600b2a20 con 0x7fc26006b2c0 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.209+0000 7fc264b01640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc2600cdcd0 0x7fc2600b1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41398/0 (socket says 172.21.15.27:41398) 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.209+0000 7fc264b01640 1 -- 172.21.15.27:0/723724252 learned_addr learned my addr 172.21.15.27:0/723724252 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc265302640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc26006b2c0 0x7fc26006b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491027851 0 0) 0x7fc260161a40 con 0x7fc2600cdcd0 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc248003520 con 0x7fc2600cdcd0 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25ffff640 1 --2- 172.21.15.27:0/723724252 >> v2:172.21.15.69:3300/0 conn(0x7fc2600b2cd0 0x7fc2600b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332781831 0 0) 0x7fc248003520 con 0x7fc2600cdcd0 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc26006b2c0 msgr2=0x7fc26006b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc26006b2c0 0x7fc26006b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 >> v2:172.21.15.69:3300/0 conn(0x7fc2600b2cd0 msgr2=0x7fc2600b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 --2- 172.21.15.27:0/723724252 >> v2:172.21.15.69:3300/0 conn(0x7fc2600b2cd0 0x7fc2600b1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2600d0940 con 0x7fc2600cdcd0 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fc250001670 con 0x7fc2600cdcd0 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 >> v1:172.21.15.27:6789/0 conn(0x7fc2600cdcd0 legacy=0x7fc2600b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc248003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc248004800 0x7fc248004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc26006b2c0 0x7fc2480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600d11e0 con 0x7fc2480039b0 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600b2a20 con 0x7fc248004800 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24800b9c0 con 0x7fc26006b2c0 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc264b01640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc248003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc25ffff640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc248004800 0x7fc248004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.210+0000 7fc265302640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc26006b2c0 0x7fc2480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.211+0000 7fc264b01640 1 -- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc26006b2c0 msgr2=0x7fc2480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.211+0000 7fc264b01640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc26006b2c0 0x7fc2480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.211+0000 7fc264b01640 1 -- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc248004800 msgr2=0x7fc248004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.211+0000 7fc264b01640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc248004800 0x7fc248004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.211+0000 7fc264b01640 1 -- 172.21.15.27:0/723724252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc248003520 con 0x7fc2480039b0 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.211+0000 7fc264b01640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc248003dd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc25000c210 tx=0x7fc248005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724bc0734d62795f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.211+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250007d50 con 0x7fc2480039b0 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.212+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc250030070 con 0x7fc2480039b0 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.212+0000 7fc25f7fe640 1 -- 172.21.15.27:0/723724252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25002cd20 con 0x7fc2480039b0 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.212+0000 7fc266d65640 1 -- 172.21.15.27:0/723724252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 msgr2=0x7fc248003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.212+0000 7fc266d65640 1 --2- 172.21.15.27:0/723724252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc248003dd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc25000c210 tx=0x7fc248005830 comp rx=0 tx=0).stop 2024-03-06T07:23:24.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.212+0000 7fc265302640 1 -- 172.21.15.27:0/723724252 reap_dead start 2024-03-06T07:23:24.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.212+0000 7fc266d65640 1 -- 172.21.15.27:0/723724252 shutdown_connections 2024-03-06T07:23:24.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.212+0000 7fc266d65640 1 -- 172.21.15.27:0/723724252 >> 172.21.15.27:0/723724252 conn(0x7fc2600b3700 msgr2=0x7fc2600b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.212+0000 7fc266d65640 1 -- 172.21.15.27:0/723724252 shutdown_connections 2024-03-06T07:23:24.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.212+0000 7fc266d65640 1 -- 172.21.15.27:0/723724252 wait complete. 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.213+0000 7fc266d65640 1 Processor -- start 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.213+0000 7fc266d65640 1 -- start start 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.214+0000 7fc266d65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc260157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.214+0000 7fc266d65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2600c8180 0x7fc2600c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.214+0000 7fc266d65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2600c7570 0x7fc2600d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.214+0000 7fc266d65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600cc1c0 con 0x7fc2480039b0 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.214+0000 7fc266d65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600d0f00 con 0x7fc2600c8180 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.214+0000 7fc266d65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600cc340 con 0x7fc2600c7570 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.214+0000 7fc264b01640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc260157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.214+0000 7fc264b01640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc260157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48392/0 (socket says 172.21.15.27:48392) 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.214+0000 7fc264b01640 1 -- 172.21.15.27:0/2269388132 learned_addr learned my addr 172.21.15.27:0/2269388132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc25ffff640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2600c8180 0x7fc2600c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc265302640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2600c7570 0x7fc2600d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc264b01640 1 -- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2600c7570 msgr2=0x7fc2600d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc264b01640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2600c7570 0x7fc2600d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc264b01640 1 -- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2600c8180 msgr2=0x7fc2600c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc264b01640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2600c8180 0x7fc2600c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc264b01640 1 -- 172.21.15.27:0/2269388132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc260157950 con 0x7fc2480039b0 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc25ffff640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2600c8180 0x7fc2600c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc264b01640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc260157410 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc25002c550 tx=0x7fc2600d0b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc25d7fa640 1 -- 172.21.15.27:0/2269388132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250045070 con 0x7fc2480039b0 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc25d7fa640 1 -- 172.21.15.27:0/2269388132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc250004bf0 con 0x7fc2480039b0 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2600c7a10 con 0x7fc2480039b0 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc25d7fa640 1 -- 172.21.15.27:0/2269388132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250001cb0 con 0x7fc2480039b0 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.215+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2601585a0 con 0x7fc2480039b0 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.216+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26009fcf0 con 0x7fc2480039b0 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.217+0000 7fc25d7fa640 1 -- 172.21.15.27:0/2269388132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc250007070 con 0x7fc2480039b0 2024-03-06T07:23:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.217+0000 7fc25d7fa640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc24c06b500 0x7fc24c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.217+0000 7fc25d7fa640 1 -- 172.21.15.27:0/2269388132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2500ab270 con 0x7fc2480039b0 2024-03-06T07:23:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.217+0000 7fc25ffff640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc24c06b500 0x7fc24c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.218+0000 7fc25ffff640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc24c06b500 0x7fc24c06d9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc2600cf8a0 tx=0x7fc254008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.222+0000 7fc25d7fa640 1 -- 172.21.15.27:0/2269388132 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc250079aa0 con 0x7fc2480039b0 2024-03-06T07:23:24.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.403+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2600d1690 con 0x7fc2480039b0 2024-03-06T07:23:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.403+0000 7fc25d7fa640 1 -- 172.21.15.27:0/2269388132 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc25007db40 con 0x7fc2480039b0 2024-03-06T07:23:24.406 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.405+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc24c06b500 msgr2=0x7fc24c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.405+0000 7fc266d65640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc24c06b500 0x7fc24c06d9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc2600cf8a0 tx=0x7fc254008040 comp rx=0 tx=0).stop 2024-03-06T07:23:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.405+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 msgr2=0x7fc260157410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.405+0000 7fc266d65640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc260157410 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc25002c550 tx=0x7fc2600d0b10 comp rx=0 tx=0).stop 2024-03-06T07:23:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.406+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 shutdown_connections 2024-03-06T07:23:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.406+0000 7fc266d65640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc24c06b500 0x7fc24c06d9c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.406+0000 7fc266d65640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2600c7570 0x7fc2600d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.406+0000 7fc266d65640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2600c8180 0x7fc2600c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.406+0000 7fc266d65640 1 --2- 172.21.15.27:0/2269388132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2480039b0 0x7fc260157410 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.406+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 >> 172.21.15.27:0/2269388132 conn(0x7fc2600b3700 msgr2=0x7fc26006f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.406+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 shutdown_connections 2024-03-06T07:23:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.406+0000 7fc266d65640 1 -- 172.21.15.27:0/2269388132 wait complete. 2024-03-06T07:23:24.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.525+0000 7fb25c916640 1 Processor -- start 2024-03-06T07:23:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.526+0000 7fb25c916640 1 -- start start 2024-03-06T07:23:24.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.526+0000 7fb25c916640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2580b7420 0x7fb2580d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.526+0000 7fb25c916640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2580d3740 con 0x7fb2580ceea0 2024-03-06T07:23:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.526+0000 7fb25c916640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2580d3120 con 0x7fb25815dd90 2024-03-06T07:23:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.526+0000 7fb25c916640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580d3400 con 0x7fb2580b7420 2024-03-06T07:23:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.527+0000 7fb255d9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb2580ceea0 0x7fb2580b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41404/0 (socket says 172.21.15.27:41404) 2024-03-06T07:23:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.527+0000 7fb255d9b640 1 -- 172.21.15.27:0/1600788895 learned_addr learned my addr 172.21.15.27:0/1600788895 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.527+0000 7fb256d9d640 1 --2- 172.21.15.27:0/1600788895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2580b7420 0x7fb2580d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.527+0000 7fb25559a640 1 -- 172.21.15.27:0/1600788895 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899815053 0 0) 0x7fb2580d3740 con 0x7fb2580ceea0 2024-03-06T07:23:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.527+0000 7fb25559a640 1 -- 172.21.15.27:0/1600788895 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb240003520 con 0x7fb2580ceea0 2024-03-06T07:23:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.527+0000 7fb256d9d640 1 -- 172.21.15.27:0/1600788895 >> v1:172.21.15.69:6789/0 conn(0x7fb25815dd90 legacy=0x7fb2580d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.527+0000 7fb25559a640 1 -- 172.21.15.27:0/1600788895 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911763049 0 0) 0x7fb2580d3120 con 0x7fb25815dd90 2024-03-06T07:23:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.527+0000 7fb256d9d640 1 -- 172.21.15.27:0/1600788895 >> v1:172.21.15.27:6789/0 conn(0x7fb2580ceea0 legacy=0x7fb2580b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.527+0000 7fb256d9d640 1 -- 172.21.15.27:0/1600788895 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2580d39e0 con 0x7fb2580b7420 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.528+0000 7fb256d9d640 1 --2- 172.21.15.27:0/1600788895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2580b7420 0x7fb2580d29d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb250009dc0 tx=0x7fb25002bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de667623b8160706 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.528+0000 7fb25559a640 1 -- 172.21.15.27:0/1600788895 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250039070 con 0x7fb2580b7420 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.528+0000 7fb25559a640 1 -- 172.21.15.27:0/1600788895 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25002cd50 con 0x7fb2580b7420 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.528+0000 7fb25559a640 1 -- 172.21.15.27:0/1600788895 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250034640 con 0x7fb2580b7420 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.529+0000 7fb25c916640 1 -- 172.21.15.27:0/1600788895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2580b7420 msgr2=0x7fb2580d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.529+0000 7fb25c916640 1 --2- 172.21.15.27:0/1600788895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2580b7420 0x7fb2580d29d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb250009dc0 tx=0x7fb25002bfa0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.529+0000 7fb25c916640 1 -- 172.21.15.27:0/1600788895 shutdown_connections 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.529+0000 7fb25c916640 1 --2- 172.21.15.27:0/1600788895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb2580b7420 0x7fb2580d29d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.529+0000 7fb25c916640 1 -- 172.21.15.27:0/1600788895 >> 172.21.15.27:0/1600788895 conn(0x7fb258071960 msgr2=0x7fb258071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.529+0000 7fb25c916640 1 -- 172.21.15.27:0/1600788895 shutdown_connections 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.529+0000 7fb25c916640 1 -- 172.21.15.27:0/1600788895 wait complete. 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.530+0000 7fb25c916640 1 Processor -- start 2024-03-06T07:23:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.530+0000 7fb25c916640 1 -- start start 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25c916640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580ceea0 0x7fb2581534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25c916640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25815dd90 0x7fb25815e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25c916640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25815e8c0 0x7fb25815ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25c916640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580d48f0 con 0x7fb2580ceea0 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25c916640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25815e170 con 0x7fb25815e8c0 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25c916640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580d4a70 con 0x7fb25815dd90 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580ceea0 0x7fb2581534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580ceea0 0x7fb2581534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48404/0 (socket says 172.21.15.27:48404) 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25659c640 1 -- 172.21.15.27:0/3185725336 learned_addr learned my addr 172.21.15.27:0/3185725336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb255d9b640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25815dd90 0x7fb25815e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb256d9d640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25815e8c0 0x7fb25815ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25659c640 1 -- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25815dd90 msgr2=0x7fb25815e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25659c640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25815dd90 0x7fb25815e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25659c640 1 -- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25815e8c0 msgr2=0x7fb25815ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25659c640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25815e8c0 0x7fb25815ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb25659c640 1 -- 172.21.15.27:0/3185725336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2580cc210 con 0x7fb2580ceea0 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.531+0000 7fb256d9d640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25815e8c0 0x7fb25815ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.532+0000 7fb25659c640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580ceea0 0x7fb2581534c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb24800c410 tx=0x7fb24800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.532+0000 7fb246ffd640 1 -- 172.21.15.27:0/3185725336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248017070 con 0x7fb2580ceea0 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.532+0000 7fb246ffd640 1 -- 172.21.15.27:0/3185725336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2480035f0 con 0x7fb2580ceea0 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.532+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2580cc4f0 con 0x7fb2580ceea0 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.532+0000 7fb246ffd640 1 -- 172.21.15.27:0/3185725336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2480039e0 con 0x7fb2580ceea0 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.532+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2580c89d0 con 0x7fb2580ceea0 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.533+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb258072a30 con 0x7fb2580ceea0 2024-03-06T07:23:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.533+0000 7fb246ffd640 1 -- 172.21.15.27:0/3185725336 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb24801a070 con 0x7fb2580ceea0 2024-03-06T07:23:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.534+0000 7fb246ffd640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb23806b4d0 0x7fb23806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.534+0000 7fb246ffd640 1 -- 172.21.15.27:0/3185725336 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb248089620 con 0x7fb2580ceea0 2024-03-06T07:23:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.534+0000 7fb255d9b640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb23806b4d0 0x7fb23806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.535+0000 7fb255d9b640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb23806b4d0 0x7fb23806d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb2580b3ce0 tx=0x7fb24c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.538+0000 7fb246ffd640 1 -- 172.21.15.27:0/3185725336 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb248057dd0 con 0x7fb2580ceea0 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.720+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb25804f410 con 0x7fb2580ceea0 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.720+0000 7fb246ffd640 1 -- 172.21.15.27:0/3185725336 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb24805be70 con 0x7fb2580ceea0 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:24.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb23806b4d0 msgr2=0x7fb23806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb23806b4d0 0x7fb23806d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb2580b3ce0 tx=0x7fb24c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580ceea0 msgr2=0x7fb2581534c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580ceea0 0x7fb2581534c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb24800c410 tx=0x7fb24800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 shutdown_connections 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb23806b4d0 0x7fb23806d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb25815e8c0 0x7fb25815ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb25815dd90 0x7fb25815e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 --2- 172.21.15.27:0/3185725336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580ceea0 0x7fb2581534c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 >> 172.21.15.27:0/3185725336 conn(0x7fb258071960 msgr2=0x7fb2580ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 shutdown_connections 2024-03-06T07:23:24.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.722+0000 7fb25c916640 1 -- 172.21.15.27:0/3185725336 wait complete. 2024-03-06T07:23:24.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:24.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.841+0000 7fc845bf6640 1 Processor -- start 2024-03-06T07:23:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.841+0000 7fc845bf6640 1 -- start start 2024-03-06T07:23:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.842+0000 7fc845bf6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b95a0 0x7fc8400b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.842+0000 7fc845bf6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc8400b9a10 0x7fc8400b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.842+0000 7fc845bf6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8400cea30 con 0x7fc84015df00 2024-03-06T07:23:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.842+0000 7fc845bf6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8400cf3c0 con 0x7fc8400b9a10 2024-03-06T07:23:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.842+0000 7fc845bf6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400cf0e0 con 0x7fc8400b95a0 2024-03-06T07:23:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.842+0000 7fc83effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b95a0 0x7fc8400b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.842+0000 7fc83ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc8400b9a10 0x7fc8400b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.843+0000 7fc83effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b95a0 0x7fc8400b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:54478/0 (socket says 172.21.15.27:54478) 2024-03-06T07:23:24.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.843+0000 7fc83f7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc84015df00 0x7fc8400d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:33376/0 (socket says 172.21.15.27:33376) 2024-03-06T07:23:24.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.843+0000 7fc83effd640 1 -- 172.21.15.27:0/3223014966 learned_addr learned my addr 172.21.15.27:0/3223014966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:24.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.843+0000 7fc83effd640 1 -- 172.21.15.27:0/3223014966 >> v2:172.21.15.69:3300/0 conn(0x7fc8400b9a10 msgr2=0x7fc8400b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.843+0000 7fc83effd640 1 --2- 172.21.15.27:0/3223014966 >> v2:172.21.15.69:3300/0 conn(0x7fc8400b9a10 0x7fc8400b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.843+0000 7fc83effd640 1 -- 172.21.15.27:0/3223014966 >> v1:172.21.15.69:6789/0 conn(0x7fc84015df00 legacy=0x7fc8400d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.843+0000 7fc83effd640 1 -- 172.21.15.27:0/3223014966 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8400ce470 con 0x7fc8400b95a0 2024-03-06T07:23:24.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc83effd640 1 --2- 172.21.15.27:0/3223014966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b95a0 0x7fc8400b5700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc834009870 tx=0x7fc83402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bf7b4f03735c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc83e7fc640 1 -- 172.21.15.27:0/3223014966 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc834034070 con 0x7fc8400b95a0 2024-03-06T07:23:24.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc83e7fc640 1 -- 172.21.15.27:0/3223014966 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc834030840 con 0x7fc8400b95a0 2024-03-06T07:23:24.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc83e7fc640 1 -- 172.21.15.27:0/3223014966 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc834030c10 con 0x7fc8400b95a0 2024-03-06T07:23:24.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc845bf6640 1 -- 172.21.15.27:0/3223014966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b95a0 msgr2=0x7fc8400b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc845bf6640 1 --2- 172.21.15.27:0/3223014966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b95a0 0x7fc8400b5700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc834009870 tx=0x7fc83402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc845bf6640 1 -- 172.21.15.27:0/3223014966 shutdown_connections 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc845bf6640 1 --2- 172.21.15.27:0/3223014966 >> v2:172.21.15.69:3300/0 conn(0x7fc8400b9a10 0x7fc8400b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc845bf6640 1 --2- 172.21.15.27:0/3223014966 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b95a0 0x7fc8400b5700 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc845bf6640 1 -- 172.21.15.27:0/3223014966 >> 172.21.15.27:0/3223014966 conn(0x7fc840071960 msgr2=0x7fc840071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc845bf6640 1 -- 172.21.15.27:0/3223014966 shutdown_connections 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.844+0000 7fc845bf6640 1 -- 172.21.15.27:0/3223014966 wait complete. 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.846+0000 7fc845bf6640 1 Processor -- start 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.846+0000 7fc845bf6640 1 -- start start 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.846+0000 7fc845bf6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8400b95a0 0x7fc840155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.846+0000 7fc845bf6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b9a10 0x7fc8400d3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.846+0000 7fc845bf6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc84015df00 0x7fc8400d3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.846+0000 7fc845bf6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400cedf0 con 0x7fc8400b95a0 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.846+0000 7fc845bf6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400cec70 con 0x7fc84015df00 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc845bf6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400d3220 con 0x7fc8400b9a10 2024-03-06T07:23:24.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8400b95a0 0x7fc840155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8400b95a0 0x7fc840155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48414/0 (socket says 172.21.15.27:48414) 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83f7fe640 1 -- 172.21.15.27:0/1148059378 learned_addr learned my addr 172.21.15.27:0/1148059378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83effd640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b9a10 0x7fc8400d3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83ffff640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc84015df00 0x7fc8400d3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83f7fe640 1 -- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b9a10 msgr2=0x7fc8400d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83f7fe640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b9a10 0x7fc8400d3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83f7fe640 1 -- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc84015df00 msgr2=0x7fc8400d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83f7fe640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc84015df00 0x7fc8400d3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83f7fe640 1 -- 172.21.15.27:0/1148059378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8400cc280 con 0x7fc8400b95a0 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc83f7fe640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8400b95a0 0x7fc840155310 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c3c0 tx=0x7fc83000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc827fff640 1 -- 172.21.15.27:0/1148059378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8300074e0 con 0x7fc8400b95a0 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc827fff640 1 -- 172.21.15.27:0/1148059378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc830007680 con 0x7fc8400b95a0 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84015f360 con 0x7fc8400b95a0 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc827fff640 1 -- 172.21.15.27:0/1148059378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830004a00 con 0x7fc8400b95a0 2024-03-06T07:23:24.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.847+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84015fc00 con 0x7fc8400b95a0 2024-03-06T07:23:24.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.848+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84015f750 con 0x7fc8400b95a0 2024-03-06T07:23:24.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.849+0000 7fc827fff640 1 -- 172.21.15.27:0/1148059378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc83001a020 con 0x7fc8400b95a0 2024-03-06T07:23:24.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.849+0000 7fc827fff640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc82c06b4d0 0x7fc82c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.849+0000 7fc827fff640 1 -- 172.21.15.27:0/1148059378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc830089940 con 0x7fc8400b95a0 2024-03-06T07:23:24.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.850+0000 7fc83effd640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc82c06b4d0 0x7fc82c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.850+0000 7fc83effd640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc82c06b4d0 0x7fc82c06d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc834036000 tx=0x7fc83403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:24.854+0000 7fc827fff640 1 -- 172.21.15.27:0/1148059378 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc830010300 con 0x7fc8400b95a0 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.047+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc8400d4380 con 0x7fc8400b95a0 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.047+0000 7fc827fff640 1 -- 172.21.15.27:0/1148059378 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc830058850 con 0x7fc8400b95a0 2024-03-06T07:23:25.050 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:23:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc82c06b4d0 msgr2=0x7fc82c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc82c06b4d0 0x7fc82c06d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc834036000 tx=0x7fc83403b040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8400b95a0 msgr2=0x7fc840155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8400b95a0 0x7fc840155310 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c3c0 tx=0x7fc83000c880 comp rx=0 tx=0).stop 2024-03-06T07:23:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 shutdown_connections 2024-03-06T07:23:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc82c06b4d0 0x7fc82c06d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc84015df00 0x7fc8400d3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8400b9a10 0x7fc8400d3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 --2- 172.21.15.27:0/1148059378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8400b95a0 0x7fc840155310 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.049+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 >> 172.21.15.27:0/1148059378 conn(0x7fc840071960 msgr2=0x7fc840071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.050+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 shutdown_connections 2024-03-06T07:23:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.050+0000 7fc845bf6640 1 -- 172.21.15.27:0/1148059378 wait complete. 2024-03-06T07:23:25.063 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:23:25.063 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:25.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:25.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.186+0000 7fd9b46c2640 1 Processor -- start 2024-03-06T07:23:25.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.186+0000 7fd9b46c2640 1 -- start start 2024-03-06T07:23:25.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.187+0000 7fd9b46c2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac15df00 0x7fd9ac0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.187+0000 7fd9b46c2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9ac0b9a30 0x7fd9ac0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.187+0000 7fd9b46c2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0cea30 con 0x7fd9ac0b95a0 2024-03-06T07:23:25.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.187+0000 7fd9b46c2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac0d2af0 con 0x7fd9ac0b9a30 2024-03-06T07:23:25.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.187+0000 7fd9b46c2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0cf3c0 con 0x7fd9ac15df00 2024-03-06T07:23:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b245e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac15df00 0x7fd9ac0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b2c5f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9ac0b9a30 0x7fd9ac0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b245e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac15df00 0x7fd9ac0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:54488/0 (socket says 172.21.15.27:54488) 2024-03-06T07:23:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b245e640 1 -- 172.21.15.27:0/2389210549 learned_addr learned my addr 172.21.15.27:0/2389210549 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b1c5d640 1 --1- 172.21.15.27:0/2389210549 >> v1:172.21.15.69:6789/0 conn(0x7fd9ac0b95a0 0x7fd9ac0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:33386/0 (socket says 172.21.15.27:33386) 2024-03-06T07:23:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b245e640 1 -- 172.21.15.27:0/2389210549 >> v2:172.21.15.69:3300/0 conn(0x7fd9ac0b9a30 msgr2=0x7fd9ac0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b245e640 1 --2- 172.21.15.27:0/2389210549 >> v2:172.21.15.69:3300/0 conn(0x7fd9ac0b9a30 0x7fd9ac0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b245e640 1 -- 172.21.15.27:0/2389210549 >> v1:172.21.15.69:6789/0 conn(0x7fd9ac0b95a0 legacy=0x7fd9ac0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b245e640 1 -- 172.21.15.27:0/2389210549 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac0cf0e0 con 0x7fd9ac15df00 2024-03-06T07:23:25.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b145c640 1 -- 172.21.15.27:0/2389210549 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792920049 0 0) 0x7fd9ac0cea30 con 0x7fd9ac0b95a0 2024-03-06T07:23:25.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.188+0000 7fd9b2c5f640 1 --2- 172.21.15.27:0/2389210549 >> v2:172.21.15.69:3300/0 conn(0x7fd9ac0b9a30 0x7fd9ac0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:25.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.189+0000 7fd9b245e640 1 --2- 172.21.15.27:0/2389210549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac15df00 0x7fd9ac0d25b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009870 tx=0x7fd9a802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1720069912a28c99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.189+0000 7fd9b145c640 1 -- 172.21.15.27:0/2389210549 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8034070 con 0x7fd9ac15df00 2024-03-06T07:23:25.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.189+0000 7fd9b145c640 1 -- 172.21.15.27:0/2389210549 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a8030840 con 0x7fd9ac15df00 2024-03-06T07:23:25.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.189+0000 7fd9b145c640 1 -- 172.21.15.27:0/2389210549 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8030c10 con 0x7fd9ac15df00 2024-03-06T07:23:25.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.190+0000 7fd9b46c2640 1 -- 172.21.15.27:0/2389210549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac15df00 msgr2=0x7fd9ac0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.190+0000 7fd9b46c2640 1 --2- 172.21.15.27:0/2389210549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac15df00 0x7fd9ac0d25b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009870 tx=0x7fd9a802b980 comp rx=0 tx=0).stop 2024-03-06T07:23:25.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.190+0000 7fd9b46c2640 1 -- 172.21.15.27:0/2389210549 shutdown_connections 2024-03-06T07:23:25.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.190+0000 7fd9b46c2640 1 --2- 172.21.15.27:0/2389210549 >> v2:172.21.15.69:3300/0 conn(0x7fd9ac0b9a30 0x7fd9ac0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.190+0000 7fd9b46c2640 1 --2- 172.21.15.27:0/2389210549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac15df00 0x7fd9ac0d25b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.190+0000 7fd9b46c2640 1 -- 172.21.15.27:0/2389210549 >> 172.21.15.27:0/2389210549 conn(0x7fd9ac071960 msgr2=0x7fd9ac071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.190+0000 7fd9b46c2640 1 -- 172.21.15.27:0/2389210549 shutdown_connections 2024-03-06T07:23:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.190+0000 7fd9b46c2640 1 -- 172.21.15.27:0/2389210549 wait complete. 2024-03-06T07:23:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.192+0000 7fd9b46c2640 1 Processor -- start 2024-03-06T07:23:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.192+0000 7fd9b46c2640 1 -- start start 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.193+0000 7fd9b46c2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9ac0b95a0 0x7fd9ac151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b46c2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac15df00 0x7fd9ac1e8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b46c2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac1517d0 0x7fd9ac15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b46c2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0cec70 con 0x7fd9ac15df00 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b46c2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0ce750 con 0x7fd9ac0b95a0 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b46c2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0cedf0 con 0x7fd9ac1517d0 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b1c5d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac15df00 0x7fd9ac1e8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b1c5d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac15df00 0x7fd9ac1e8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48422/0 (socket says 172.21.15.27:48422) 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b2c5f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac1517d0 0x7fd9ac15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b245e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9ac0b95a0 0x7fd9ac151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b1c5d640 1 -- 172.21.15.27:0/863505885 learned_addr learned my addr 172.21.15.27:0/863505885 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b1c5d640 1 -- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac1517d0 msgr2=0x7fd9ac15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b1c5d640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac1517d0 0x7fd9ac15aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b1c5d640 1 -- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9ac0b95a0 msgr2=0x7fd9ac151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b1c5d640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9ac0b95a0 0x7fd9ac151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b1c5d640 1 -- 172.21.15.27:0/863505885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac151c80 con 0x7fd9ac15df00 2024-03-06T07:23:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.194+0000 7fd9b1c5d640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac15df00 0x7fd9ac1e8f80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c430 tx=0x7fd99c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.199+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/863505885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c0074e0 con 0x7fd9ac15df00 2024-03-06T07:23:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.199+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/863505885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c0047e0 con 0x7fd9ac15df00 2024-03-06T07:23:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.199+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac1e7880 con 0x7fd9ac15df00 2024-03-06T07:23:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.199+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/863505885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c004bd0 con 0x7fd9ac15df00 2024-03-06T07:23:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.199+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac1e7e40 con 0x7fd9ac15df00 2024-03-06T07:23:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.200+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac067ff0 con 0x7fd9ac15df00 2024-03-06T07:23:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.200+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/863505885 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd99c011070 con 0x7fd9ac15df00 2024-03-06T07:23:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.201+0000 7fd9a2ffd640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.201+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/863505885 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd99c089d30 con 0x7fd9ac15df00 2024-03-06T07:23:25.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.201+0000 7fd9b245e640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.202+0000 7fd9b245e640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8036000 tx=0x7fd9a8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.205+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/863505885 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd99c058560 con 0x7fd9ac15df00 2024-03-06T07:23:25.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.387+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9ac0b6f60 con 0x7fd9ac15df00 2024-03-06T07:23:25.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.388+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/863505885 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7fd99c05c600 con 0x7fd9ac15df00 2024-03-06T07:23:25.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:25.392 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 msgr2=0x7fd99006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8036000 tx=0x7fd9a8008030 comp rx=0 tx=0).stop 2024-03-06T07:23:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac15df00 msgr2=0x7fd9ac1e8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac15df00 0x7fd9ac1e8f80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c430 tx=0x7fd99c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 shutdown_connections 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9ac1517d0 0x7fd9ac15aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac15df00 0x7fd9ac1e8f80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 --2- 172.21.15.27:0/863505885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9ac0b95a0 0x7fd9ac151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 >> 172.21.15.27:0/863505885 conn(0x7fd9ac071960 msgr2=0x7fd9ac15bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 shutdown_connections 2024-03-06T07:23:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.390+0000 7fd9b46c2640 1 -- 172.21.15.27:0/863505885 wait complete. 2024-03-06T07:23:25.404 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:25.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.511+0000 7f032d70c640 1 Processor -- start 2024-03-06T07:23:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.511+0000 7f032d70c640 1 -- start start 2024-03-06T07:23:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f032d70c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 0x7f03280ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f032d70c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f03280b6d10 0x7f03280b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f032d70c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03280d3740 con 0x7f03280d29d0 2024-03-06T07:23:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f032d70c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03280d3400 con 0x7f03280b6d10 2024-03-06T07:23:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f032d70c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03280d3120 con 0x7f032815dd90 2024-03-06T07:23:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f03267fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f03280d29d0 0x7f03280d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41420/0 (socket says 172.21.15.27:41420) 2024-03-06T07:23:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f03267fc640 1 -- 172.21.15.27:0/3719189184 learned_addr learned my addr 172.21.15.27:0/3719189184 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f03277fe640 1 --2- 172.21.15.27:0/3719189184 >> v2:172.21.15.69:3300/0 conn(0x7f03280b6d10 0x7f03280b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f0326ffd640 1 --2- 172.21.15.27:0/3719189184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 0x7f03280ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f0325ffb640 1 -- 172.21.15.27:0/3719189184 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223271553 0 0) 0x7f03280d3740 con 0x7f03280d29d0 2024-03-06T07:23:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.513+0000 7f0325ffb640 1 -- 172.21.15.27:0/3719189184 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310003520 con 0x7f03280d29d0 2024-03-06T07:23:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.514+0000 7f0326ffd640 1 -- 172.21.15.27:0/3719189184 >> v2:172.21.15.69:3300/0 conn(0x7f03280b6d10 msgr2=0x7f03280b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.514+0000 7f0326ffd640 1 --2- 172.21.15.27:0/3719189184 >> v2:172.21.15.69:3300/0 conn(0x7f03280b6d10 0x7f03280b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.514+0000 7f0326ffd640 1 -- 172.21.15.27:0/3719189184 >> v1:172.21.15.27:6789/0 conn(0x7f03280d29d0 legacy=0x7f03280d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.514+0000 7f0326ffd640 1 -- 172.21.15.27:0/3719189184 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03280d39e0 con 0x7f032815dd90 2024-03-06T07:23:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.515+0000 7f0326ffd640 1 --2- 172.21.15.27:0/3719189184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 0x7f03280ceea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f03180097a0 tx=0x7f031802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9775a72bfc903d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.515+0000 7f0325ffb640 1 -- 172.21.15.27:0/3719189184 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318034070 con 0x7f032815dd90 2024-03-06T07:23:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.515+0000 7f0325ffb640 1 -- 172.21.15.27:0/3719189184 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0318030610 con 0x7f032815dd90 2024-03-06T07:23:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.515+0000 7f0325ffb640 1 -- 172.21.15.27:0/3719189184 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03180309e0 con 0x7f032815dd90 2024-03-06T07:23:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.516+0000 7f032d70c640 1 -- 172.21.15.27:0/3719189184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 msgr2=0x7f03280ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.516+0000 7f032d70c640 1 --2- 172.21.15.27:0/3719189184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 0x7f03280ceea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f03180097a0 tx=0x7f031802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.516+0000 7f032d70c640 1 -- 172.21.15.27:0/3719189184 shutdown_connections 2024-03-06T07:23:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.516+0000 7f032d70c640 1 --2- 172.21.15.27:0/3719189184 >> v2:172.21.15.69:3300/0 conn(0x7f03280b6d10 0x7f03280b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.516+0000 7f032d70c640 1 --2- 172.21.15.27:0/3719189184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 0x7f03280ceea0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.516+0000 7f032d70c640 1 -- 172.21.15.27:0/3719189184 >> 172.21.15.27:0/3719189184 conn(0x7f0328071960 msgr2=0x7f0328071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.516+0000 7f032d70c640 1 -- 172.21.15.27:0/3719189184 shutdown_connections 2024-03-06T07:23:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.516+0000 7f032d70c640 1 -- 172.21.15.27:0/3719189184 wait complete. 2024-03-06T07:23:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.517+0000 7f032d70c640 1 Processor -- start 2024-03-06T07:23:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.517+0000 7f032d70c640 1 -- start start 2024-03-06T07:23:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f032d70c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03280b6d10 0x7f03280b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f032d70c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03280d29d0 0x7f03281f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f032d70c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 0x7f03281616d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f032d70c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03280d2df0 con 0x7f03280b6d10 2024-03-06T07:23:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f032d70c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032815e170 con 0x7f03280d29d0 2024-03-06T07:23:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f032d70c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03280d48f0 con 0x7f032815dd90 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f0326ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03280b6d10 0x7f03280b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f0326ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03280b6d10 0x7f03280b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48430/0 (socket says 172.21.15.27:48430) 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f0326ffd640 1 -- 172.21.15.27:0/888600290 learned_addr learned my addr 172.21.15.27:0/888600290 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f03277fe640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 0x7f03281616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.518+0000 7f03267fc640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03280d29d0 0x7f03281f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f0326ffd640 1 -- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 msgr2=0x7f03281616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f0326ffd640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 0x7f03281616d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f0326ffd640 1 -- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03280d29d0 msgr2=0x7f03281f4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f0326ffd640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03280d29d0 0x7f03281f4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f0326ffd640 1 -- 172.21.15.27:0/888600290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03281f4710 con 0x7f03280b6d10 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f03267fc640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03280d29d0 0x7f03281f4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f0326ffd640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03280b6d10 0x7f03280b20a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f03180098c0 tx=0x7f03180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f030f7fe640 1 -- 172.21.15.27:0/888600290 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318034040 con 0x7f03280b6d10 2024-03-06T07:23:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f030f7fe640 1 -- 172.21.15.27:0/888600290 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031803aaa0 con 0x7f03280b6d10 2024-03-06T07:23:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03281f4430 con 0x7f03280b6d10 2024-03-06T07:23:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f030f7fe640 1 -- 172.21.15.27:0/888600290 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318006e10 con 0x7f03280b6d10 2024-03-06T07:23:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.519+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032815e6e0 con 0x7f03280b6d10 2024-03-06T07:23:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.520+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0328072a30 con 0x7f03280b6d10 2024-03-06T07:23:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.520+0000 7f030f7fe640 1 -- 172.21.15.27:0/888600290 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f031804b060 con 0x7f03280b6d10 2024-03-06T07:23:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.521+0000 7f030f7fe640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f031406b4d0 0x7f031406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.521+0000 7f030f7fe640 1 -- 172.21.15.27:0/888600290 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03180abea0 con 0x7f03280b6d10 2024-03-06T07:23:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.521+0000 7f03267fc640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f031406b4d0 0x7f031406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.521+0000 7f03267fc640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f031406b4d0 0x7f031406d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f031c007fd0 tx=0x7f031c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.525+0000 7f030f7fe640 1 -- 172.21.15.27:0/888600290 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f031803f070 con 0x7f03280b6d10 2024-03-06T07:23:25.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.705+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f03280cef40 con 0x7f03280b6d10 2024-03-06T07:23:25.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.705+0000 7f030f7fe640 1 -- 172.21.15.27:0/888600290 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f031807ad20 con 0x7f03280b6d10 2024-03-06T07:23:25.708 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f031406b4d0 msgr2=0x7f031406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f031406b4d0 0x7f031406d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f031c007fd0 tx=0x7f031c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03280b6d10 msgr2=0x7f03280b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03280b6d10 0x7f03280b20a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f03180098c0 tx=0x7f03180046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 shutdown_connections 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f031406b4d0 0x7f031406d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f032815dd90 0x7f03281616d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03280d29d0 0x7f03281f4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 --2- 172.21.15.27:0/888600290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03280b6d10 0x7f03280b20a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 >> 172.21.15.27:0/888600290 conn(0x7f0328071960 msgr2=0x7f03280b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 shutdown_connections 2024-03-06T07:23:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.707+0000 7f032d70c640 1 -- 172.21.15.27:0/888600290 wait complete. 2024-03-06T07:23:25.721 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.828+0000 7fe78e84e640 1 Processor -- start 2024-03-06T07:23:25.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.828+0000 7fe78e84e640 1 -- start start 2024-03-06T07:23:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe78e84e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7880b2d30 0x7fe7880b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe78e84e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7880d4960 con 0x7fe7880b7820 2024-03-06T07:23:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe78e84e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7880d36c0 con 0x7fe788071540 2024-03-06T07:23:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe78e84e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7880d3230 con 0x7fe7880b2d30 2024-03-06T07:23:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe7877fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7880b2d30 0x7fe7880b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe78cdeb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe7880b7820 0x7fe7880b7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41434/0 (socket says 172.21.15.27:41434) 2024-03-06T07:23:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe78cdeb640 1 -- 172.21.15.27:0/624852919 learned_addr learned my addr 172.21.15.27:0/624852919 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe7877fe640 1 -- 172.21.15.27:0/624852919 >> v1:172.21.15.69:6789/0 conn(0x7fe788071540 legacy=0x7fe788071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe7877fe640 1 -- 172.21.15.27:0/624852919 >> v1:172.21.15.27:6789/0 conn(0x7fe7880b7820 legacy=0x7fe7880b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe7877fe640 1 -- 172.21.15.27:0/624852919 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7880d2620 con 0x7fe7880b2d30 2024-03-06T07:23:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.829+0000 7fe786ffd640 1 -- 172.21.15.27:0/624852919 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360531894 0 0) 0x7fe7880d4960 con 0x7fe7880b7820 2024-03-06T07:23:25.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.830+0000 7fe7877fe640 1 --2- 172.21.15.27:0/624852919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 0x7fe7880b72e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe7780097a0 tx=0x7fe77802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de01bdf710c1f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.830+0000 7fe786ffd640 1 -- 172.21.15.27:0/624852919 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe778034070 con 0x7fe7880b2d30 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.830+0000 7fe786ffd640 1 -- 172.21.15.27:0/624852919 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe778030610 con 0x7fe7880b2d30 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.830+0000 7fe786ffd640 1 -- 172.21.15.27:0/624852919 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7780309e0 con 0x7fe7880b2d30 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.831+0000 7fe78e84e640 1 -- 172.21.15.27:0/624852919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 msgr2=0x7fe7880b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.831+0000 7fe78e84e640 1 --2- 172.21.15.27:0/624852919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 0x7fe7880b72e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe7780097a0 tx=0x7fe77802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.831+0000 7fe78e84e640 1 -- 172.21.15.27:0/624852919 shutdown_connections 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.831+0000 7fe78e84e640 1 --2- 172.21.15.27:0/624852919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 0x7fe7880b72e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.831+0000 7fe78e84e640 1 -- 172.21.15.27:0/624852919 >> 172.21.15.27:0/624852919 conn(0x7fe7880d3d60 msgr2=0x7fe7880d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.831+0000 7fe78e84e640 1 -- 172.21.15.27:0/624852919 shutdown_connections 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.831+0000 7fe78e84e640 1 -- 172.21.15.27:0/624852919 wait complete. 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.832+0000 7fe78e84e640 1 Processor -- start 2024-03-06T07:23:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.832+0000 7fe78e84e640 1 -- start start 2024-03-06T07:23:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe78e84e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe788071540 0x7fe78814daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe78e84e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 0x7fe7881e9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe78e84e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7880b7820 0x7fe7881ea2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe78e84e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7880d2be0 con 0x7fe7880b2d30 2024-03-06T07:23:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe78e84e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7880b83e0 con 0x7fe788071540 2024-03-06T07:23:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe78e84e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7880d2ec0 con 0x7fe7880b7820 2024-03-06T07:23:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe7877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 0x7fe7881e9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe7877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 0x7fe7881e9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48446/0 (socket says 172.21.15.27:48446) 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe7877fe640 1 -- 172.21.15.27:0/875046735 learned_addr learned my addr 172.21.15.27:0/875046735 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe787fff640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe788071540 0x7fe78814daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe78cdeb640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7880b7820 0x7fe7881ea2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe7877fe640 1 -- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7880b7820 msgr2=0x7fe7881ea2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe7877fe640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7880b7820 0x7fe7881ea2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe7877fe640 1 -- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe788071540 msgr2=0x7fe78814daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe7877fe640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe788071540 0x7fe78814daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe7877fe640 1 -- 172.21.15.27:0/875046735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7880cc690 con 0x7fe7880b2d30 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.833+0000 7fe78cdeb640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7880b7820 0x7fe7881ea2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.834+0000 7fe7877fe640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 0x7fe7881e9d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe778000c00 tx=0x7fe7780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.834+0000 7fe784ff9640 1 -- 172.21.15.27:0/875046735 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe778034040 con 0x7fe7880b2d30 2024-03-06T07:23:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.834+0000 7fe784ff9640 1 -- 172.21.15.27:0/875046735 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe778004030 con 0x7fe7880b2d30 2024-03-06T07:23:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.834+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe788147c10 con 0x7fe7880b2d30 2024-03-06T07:23:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.834+0000 7fe784ff9640 1 -- 172.21.15.27:0/875046735 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe778004420 con 0x7fe7880b2d30 2024-03-06T07:23:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.834+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7881481d0 con 0x7fe7880b2d30 2024-03-06T07:23:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.835+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe788072070 con 0x7fe7880b2d30 2024-03-06T07:23:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.835+0000 7fe784ff9640 1 -- 172.21.15.27:0/875046735 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe77804c060 con 0x7fe7880b2d30 2024-03-06T07:23:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.836+0000 7fe784ff9640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77406b4d0 0x7fe77406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.836+0000 7fe784ff9640 1 -- 172.21.15.27:0/875046735 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7780abe10 con 0x7fe7880b2d30 2024-03-06T07:23:25.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.836+0000 7fe787fff640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77406b4d0 0x7fe77406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.837+0000 7fe787fff640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77406b4d0 0x7fe77406d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe7880cda60 tx=0x7fe77c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:25.844+0000 7fe784ff9640 1 -- 172.21.15.27:0/875046735 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe77807ae40 con 0x7fe7880b2d30 2024-03-06T07:23:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.023+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe78806b8e0 con 0x7fe7880b2d30 2024-03-06T07:23:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.023+0000 7fe784ff9640 1 -- 172.21.15.27:0/875046735 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe77807a740 con 0x7fe7880b2d30 2024-03-06T07:23:26.026 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77406b4d0 msgr2=0x7fe77406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77406b4d0 0x7fe77406d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe7880cda60 tx=0x7fe77c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 msgr2=0x7fe7881e9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 0x7fe7881e9d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe778000c00 tx=0x7fe7780046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 shutdown_connections 2024-03-06T07:23:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77406b4d0 0x7fe77406d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7880b7820 0x7fe7881ea2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7880b2d30 0x7fe7881e9d80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 --2- 172.21.15.27:0/875046735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe788071540 0x7fe78814daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 >> 172.21.15.27:0/875046735 conn(0x7fe7880d3d60 msgr2=0x7fe7880ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 shutdown_connections 2024-03-06T07:23:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.025+0000 7fe78e84e640 1 -- 172.21.15.27:0/875046735 wait complete. 2024-03-06T07:23:26.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.151+0000 7f18748aa640 1 Processor -- start 2024-03-06T07:23:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.151+0000 7f18748aa640 1 -- start start 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.152+0000 7f18748aa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f187015dd50 0x7f18700ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.152+0000 7f18748aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18700cef20 0x7f1870071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.152+0000 7f18748aa640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1870072cf0 con 0x7f18700724b0 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.152+0000 7f18748aa640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18700cf380 con 0x7f187015dd50 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.152+0000 7f18748aa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1870072fd0 con 0x7f18700cef20 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.152+0000 7f186e59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f187015dd50 0x7f18700ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.152+0000 7f186ed9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f18700724b0 0x7f1870161a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41438/0 (socket says 172.21.15.27:41438) 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.152+0000 7f186ed9d640 1 -- 172.21.15.27:0/3500898456 learned_addr learned my addr 172.21.15.27:0/3500898456 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.153+0000 7f186dd9b640 1 --2- 172.21.15.27:0/3500898456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18700cef20 0x7f1870071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.153+0000 7f186e59c640 1 -- 172.21.15.27:0/3500898456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18700cef20 msgr2=0x7f1870071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.153+0000 7f186e59c640 1 --2- 172.21.15.27:0/3500898456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18700cef20 0x7f1870071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.153+0000 7f186e59c640 1 -- 172.21.15.27:0/3500898456 >> v1:172.21.15.27:6789/0 conn(0x7f18700724b0 legacy=0x7f1870161a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.153+0000 7f186e59c640 1 -- 172.21.15.27:0/3500898456 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18700d1f60 con 0x7f187015dd50 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.153+0000 7f186e59c640 1 --2- 172.21.15.27:0/3500898456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187015dd50 0x7f18700ce9e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f18680097a0 tx=0x7f186802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607d0d74e20a2c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f186d59a640 1 -- 172.21.15.27:0/3500898456 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18680315a0 con 0x7f187015dd50 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f186d59a640 1 -- 172.21.15.27:0/3500898456 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1868037070 con 0x7f187015dd50 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f186d59a640 1 -- 172.21.15.27:0/3500898456 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1868031960 con 0x7f187015dd50 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f18748aa640 1 -- 172.21.15.27:0/3500898456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187015dd50 msgr2=0x7f18700ce9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f18748aa640 1 --2- 172.21.15.27:0/3500898456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187015dd50 0x7f18700ce9e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f18680097a0 tx=0x7f186802b840 comp rx=0 tx=0).stop 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f18748aa640 1 -- 172.21.15.27:0/3500898456 shutdown_connections 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f18748aa640 1 --2- 172.21.15.27:0/3500898456 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18700cef20 0x7f1870071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f18748aa640 1 --2- 172.21.15.27:0/3500898456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187015dd50 0x7f18700ce9e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f18748aa640 1 -- 172.21.15.27:0/3500898456 >> 172.21.15.27:0/3500898456 conn(0x7f18700d3ef0 msgr2=0x7f18700d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f18748aa640 1 -- 172.21.15.27:0/3500898456 shutdown_connections 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.154+0000 7f18748aa640 1 -- 172.21.15.27:0/3500898456 wait complete. 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.156+0000 7f18748aa640 1 Processor -- start 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.156+0000 7f18748aa640 1 -- start start 2024-03-06T07:23:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.156+0000 7f18748aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700724b0 0x7f187014bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.156+0000 7f18748aa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18700cef20 0x7f187015cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.156+0000 7f18748aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f187015dd50 0x7f18701e7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.156+0000 7f18748aa640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1870072a80 con 0x7f18700724b0 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.156+0000 7f18748aa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1870072900 con 0x7f18700cef20 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.156+0000 7f18748aa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187015e130 con 0x7f187015dd50 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700724b0 0x7f187014bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700724b0 0x7f187014bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48466/0 (socket says 172.21.15.27:48466) 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186e59c640 1 -- 172.21.15.27:0/2806253708 learned_addr learned my addr 172.21.15.27:0/2806253708 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186dd9b640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18700cef20 0x7f187015cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186ed9d640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f187015dd50 0x7f18701e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186e59c640 1 -- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f187015dd50 msgr2=0x7f18701e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186e59c640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f187015dd50 0x7f18701e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186e59c640 1 -- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18700cef20 msgr2=0x7f187015cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186e59c640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18700cef20 0x7f187015cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186e59c640 1 -- 172.21.15.27:0/2806253708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187015bb70 con 0x7f18700724b0 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f186e59c640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700724b0 0x7f187014bd80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f18680098c0 tx=0x7f18680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f185effd640 1 -- 172.21.15.27:0/2806253708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1868032840 con 0x7f18700724b0 2024-03-06T07:23:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f185effd640 1 -- 172.21.15.27:0/2806253708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1868032a00 con 0x7f18700724b0 2024-03-06T07:23:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187015be50 con 0x7f18700724b0 2024-03-06T07:23:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f185effd640 1 -- 172.21.15.27:0/2806253708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1868032df0 con 0x7f18700724b0 2024-03-06T07:23:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.157+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187015c410 con 0x7f18700724b0 2024-03-06T07:23:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.158+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187006cdd0 con 0x7f18700724b0 2024-03-06T07:23:26.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.159+0000 7f185effd640 1 -- 172.21.15.27:0/2806253708 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f186804e430 con 0x7f18700724b0 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.159+0000 7f185effd640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b4d0 0x7f185006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.159+0000 7f185effd640 1 -- 172.21.15.27:0/2806253708 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f186802f030 con 0x7f18700724b0 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.160+0000 7f186dd9b640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b4d0 0x7f185006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.160+0000 7f186dd9b640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b4d0 0x7f185006d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f18701e6f10 tx=0x7f1864008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.164+0000 7f185effd640 1 -- 172.21.15.27:0/2806253708 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1868040070 con 0x7f18700724b0 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.345+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f187004f410 con 0x7f18700724b0 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.345+0000 7f185effd640 1 -- 172.21.15.27:0/2806253708 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f186807a890 con 0x7f18700724b0 2024-03-06T07:23:26.348 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b4d0 msgr2=0x7f185006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b4d0 0x7f185006d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f18701e6f10 tx=0x7f1864008040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700724b0 msgr2=0x7f187014bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700724b0 0x7f187014bd80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f18680098c0 tx=0x7f18680046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 shutdown_connections 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b4d0 0x7f185006d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f187015dd50 0x7f18701e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18700cef20 0x7f187015cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 --2- 172.21.15.27:0/2806253708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700724b0 0x7f187014bd80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 >> 172.21.15.27:0/2806253708 conn(0x7f18700d3ef0 msgr2=0x7f18700ca180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 shutdown_connections 2024-03-06T07:23:26.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.347+0000 7f18748aa640 1 -- 172.21.15.27:0/2806253708 wait complete. 2024-03-06T07:23:26.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.468+0000 7fd0dc51e640 1 Processor -- start 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.468+0000 7fd0dc51e640 1 -- start start 2024-03-06T07:23:26.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.469+0000 7fd0dc51e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0d4071540 0x7fd0d40b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.469+0000 7fd0dc51e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40b2b50 0x7fd0d40b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.469+0000 7fd0dc51e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d40ce610 con 0x7fd0d40d3400 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.469+0000 7fd0dc51e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d40d2f70 con 0x7fd0d4071540 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.469+0000 7fd0dc51e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40d4be0 con 0x7fd0d40b2b50 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0da2ba640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0d4071540 0x7fd0d40b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0d9ab9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd0d40d3400 0x7fd0d4071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41448/0 (socket says 172.21.15.27:41448) 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0d9ab9640 1 -- 172.21.15.27:0/1293838509 learned_addr learned my addr 172.21.15.27:0/1293838509 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0daabb640 1 --2- 172.21.15.27:0/1293838509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40b2b50 0x7fd0d40b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0da2ba640 1 -- 172.21.15.27:0/1293838509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40b2b50 msgr2=0x7fd0d40b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0da2ba640 1 --2- 172.21.15.27:0/1293838509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40b2b50 0x7fd0d40b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0da2ba640 1 -- 172.21.15.27:0/1293838509 >> v1:172.21.15.27:6789/0 conn(0x7fd0d40d3400 legacy=0x7fd0d4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0d92b8640 1 -- 172.21.15.27:0/1293838509 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771970323 0 0) 0x7fd0d40ce610 con 0x7fd0d40d3400 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0da2ba640 1 -- 172.21.15.27:0/1293838509 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40d2920 con 0x7fd0d4071540 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.470+0000 7fd0daabb640 1 --2- 172.21.15.27:0/1293838509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40b2b50 0x7fd0d40b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0da2ba640 1 --2- 172.21.15.27:0/1293838509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d4071540 0x7fd0d40b30a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd0d00097a0 tx=0x7fd0d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a78cb8c57150f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0d92b8640 1 -- 172.21.15.27:0/1293838509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0035020 con 0x7fd0d4071540 2024-03-06T07:23:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0d92b8640 1 -- 172.21.15.27:0/1293838509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d0031830 con 0x7fd0d4071540 2024-03-06T07:23:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0d92b8640 1 -- 172.21.15.27:0/1293838509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0031c00 con 0x7fd0d4071540 2024-03-06T07:23:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1293838509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d4071540 msgr2=0x7fd0d40b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0dc51e640 1 --2- 172.21.15.27:0/1293838509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d4071540 0x7fd0d40b30a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd0d00097a0 tx=0x7fd0d002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1293838509 shutdown_connections 2024-03-06T07:23:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0dc51e640 1 --2- 172.21.15.27:0/1293838509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d40b2b50 0x7fd0d40b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0dc51e640 1 --2- 172.21.15.27:0/1293838509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d4071540 0x7fd0d40b30a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.471+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1293838509 >> 172.21.15.27:0/1293838509 conn(0x7fd0d40d3ca0 msgr2=0x7fd0d40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.472+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1293838509 shutdown_connections 2024-03-06T07:23:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.472+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1293838509 wait complete. 2024-03-06T07:23:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.473+0000 7fd0dc51e640 1 Processor -- start 2024-03-06T07:23:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.473+0000 7fd0dc51e640 1 -- start start 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.473+0000 7fd0dc51e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d4071540 0x7fd0d4155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.473+0000 7fd0dc51e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40b2b50 0x7fd0d40cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.473+0000 7fd0dc51e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d40d3400 0x7fd0d40ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.473+0000 7fd0dc51e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d415e370 con 0x7fd0d40d3400 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.473+0000 7fd0dc51e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40ce8f0 con 0x7fd0d40b2b50 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.473+0000 7fd0dc51e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4156020 con 0x7fd0d4071540 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0daabb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d40d3400 0x7fd0d40ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0daabb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d40d3400 0x7fd0d40ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48486/0 (socket says 172.21.15.27:48486) 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0daabb640 1 -- 172.21.15.27:0/1943973445 learned_addr learned my addr 172.21.15.27:0/1943973445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0da2ba640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d4071540 0x7fd0d4155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0d9ab9640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40b2b50 0x7fd0d40cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0daabb640 1 -- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d4071540 msgr2=0x7fd0d4155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0daabb640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d4071540 0x7fd0d4155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0daabb640 1 -- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40b2b50 msgr2=0x7fd0d40cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0daabb640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40b2b50 0x7fd0d40cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0daabb640 1 -- 172.21.15.27:0/1943973445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40cb6e0 con 0x7fd0d40d3400 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0d9ab9640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40b2b50 0x7fd0d40cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0daabb640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d40d3400 0x7fd0d40ccff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00cef0 tx=0x7fd0cc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0caffd640 1 -- 172.21.15.27:0/1943973445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc01a070 con 0x7fd0d40d3400 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0caffd640 1 -- 172.21.15.27:0/1943973445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0cc00d860 con 0x7fd0d40d3400 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d40cb9c0 con 0x7fd0d40d3400 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0caffd640 1 -- 172.21.15.27:0/1943973445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc00dc50 con 0x7fd0d40d3400 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.474+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d40cbf80 con 0x7fd0d40d3400 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.475+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d4068700 con 0x7fd0d40d3400 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.476+0000 7fd0caffd640 1 -- 172.21.15.27:0/1943973445 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0cc01d070 con 0x7fd0d40d3400 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.476+0000 7fd0caffd640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0b806b4d0 0x7fd0b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.476+0000 7fd0caffd640 1 -- 172.21.15.27:0/1943973445 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0cc08aad0 con 0x7fd0d40d3400 2024-03-06T07:23:26.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.477+0000 7fd0da2ba640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0b806b4d0 0x7fd0b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.477+0000 7fd0da2ba640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0b806b4d0 0x7fd0b806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0036000 tx=0x7fd0d0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.481+0000 7fd0caffd640 1 -- 172.21.15.27:0/1943973445 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0cc059a10 con 0x7fd0d40d3400 2024-03-06T07:23:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.675+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd0d40b9920 con 0x7fd0d40d3400 2024-03-06T07:23:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.675+0000 7fd0caffd640 1 -- 172.21.15.27:0/1943973445 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd0cc059310 con 0x7fd0d40d3400 2024-03-06T07:23:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:23:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0b806b4d0 msgr2=0x7fd0b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0b806b4d0 0x7fd0b806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0036000 tx=0x7fd0d0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d40d3400 msgr2=0x7fd0d40ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d40d3400 0x7fd0d40ccff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00cef0 tx=0x7fd0cc00a710 comp rx=0 tx=0).stop 2024-03-06T07:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 shutdown_connections 2024-03-06T07:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0b806b4d0 0x7fd0b806d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d40d3400 0x7fd0d40ccff0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0d40b2b50 0x7fd0d40cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 --2- 172.21.15.27:0/1943973445 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0d4071540 0x7fd0d4155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 >> 172.21.15.27:0/1943973445 conn(0x7fd0d40d3ca0 msgr2=0x7fd0d4071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 shutdown_connections 2024-03-06T07:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:26.677+0000 7fd0dc51e640 1 -- 172.21.15.27:0/1943973445 wait complete. 2024-03-06T07:23:26.691 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-03-06T07:23:26.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:26.691 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-06T07:23:26.724 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:26.724 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:23:26.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:26.724 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:23:26.789 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:23:26.790 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:26.805 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:23:26.805 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:26.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.005+0000 7f2f546a4640 1 Processor -- start 2024-03-06T07:23:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.005+0000 7f2f546a4640 1 -- start start 2024-03-06T07:23:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f546a4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f4c071540 0x7f2f4c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f546a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c0d3400 0x7f2f4c071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f546a4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c071e60 con 0x7f2f4c0b2b50 2024-03-06T07:23:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f546a4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4c0d2f70 con 0x7f2f4c071540 2024-03-06T07:23:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f546a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c0d4be0 con 0x7f2f4c0d3400 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f52440640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f4c071540 0x7f2f4c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f52c41640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2f4c0b2b50 0x7f2f4c0d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41464/0 (socket says 172.21.15.27:41464) 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f52c41640 1 -- 172.21.15.27:0/3885272984 learned_addr learned my addr 172.21.15.27:0/3885272984 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f51c3f640 1 --2- 172.21.15.27:0/3885272984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c0d3400 0x7f2f4c071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f52440640 1 -- 172.21.15.27:0/3885272984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c0d3400 msgr2=0x7f2f4c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f52440640 1 --2- 172.21.15.27:0/3885272984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c0d3400 0x7f2f4c071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f52440640 1 -- 172.21.15.27:0/3885272984 >> v1:172.21.15.27:6789/0 conn(0x7f2f4c0b2b50 legacy=0x7f2f4c0d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.007+0000 7f2f52440640 1 -- 172.21.15.27:0/3885272984 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c0ce610 con 0x7f2f4c071540 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.006+0000 7f2f5143e640 1 -- 172.21.15.27:0/3885272984 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355641247 0 0) 0x7f2f4c071e60 con 0x7f2f4c0b2b50 2024-03-06T07:23:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.007+0000 7f2f51c3f640 1 --2- 172.21.15.27:0/3885272984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c0d3400 0x7f2f4c071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.007+0000 7f2f52440640 1 --2- 172.21.15.27:0/3885272984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c071540 0x7f2f4c0b30a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0097a0 tx=0x7f2f3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f5159120e20bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.007+0000 7f2f5143e640 1 -- 172.21.15.27:0/3885272984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c035020 con 0x7f2f4c071540 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.007+0000 7f2f5143e640 1 -- 172.21.15.27:0/3885272984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c031830 con 0x7f2f4c071540 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.007+0000 7f2f5143e640 1 -- 172.21.15.27:0/3885272984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c031c00 con 0x7f2f4c071540 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.008+0000 7f2f546a4640 1 -- 172.21.15.27:0/3885272984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c071540 msgr2=0x7f2f4c0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.008+0000 7f2f546a4640 1 --2- 172.21.15.27:0/3885272984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c071540 0x7f2f4c0b30a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0097a0 tx=0x7f2f3c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.008+0000 7f2f546a4640 1 -- 172.21.15.27:0/3885272984 shutdown_connections 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.008+0000 7f2f546a4640 1 --2- 172.21.15.27:0/3885272984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c0d3400 0x7f2f4c071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.008+0000 7f2f546a4640 1 --2- 172.21.15.27:0/3885272984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c071540 0x7f2f4c0b30a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.008+0000 7f2f546a4640 1 -- 172.21.15.27:0/3885272984 >> 172.21.15.27:0/3885272984 conn(0x7f2f4c0d3ca0 msgr2=0x7f2f4c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.008+0000 7f2f546a4640 1 -- 172.21.15.27:0/3885272984 shutdown_connections 2024-03-06T07:23:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.008+0000 7f2f546a4640 1 -- 172.21.15.27:0/3885272984 wait complete. 2024-03-06T07:23:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.010+0000 7f2f546a4640 1 Processor -- start 2024-03-06T07:23:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.010+0000 7f2f546a4640 1 -- start start 2024-03-06T07:23:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.011+0000 7f2f546a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c071540 0x7f2f4c0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.011+0000 7f2f546a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f4c0b2b50 0x7f2f4c0b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.011+0000 7f2f546a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c0d3400 0x7f2f4c0ae020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.011+0000 7f2f546a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c0ce8f0 con 0x7f2f4c0d3400 2024-03-06T07:23:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.011+0000 7f2f546a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c0d2c00 con 0x7f2f4c0b2b50 2024-03-06T07:23:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.011+0000 7f2f546a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c15e370 con 0x7f2f4c071540 2024-03-06T07:23:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52c41640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c0d3400 0x7f2f4c0ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52c41640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c0d3400 0x7f2f4c0ae020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48502/0 (socket says 172.21.15.27:48502) 2024-03-06T07:23:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52c41640 1 -- 172.21.15.27:0/3043790787 learned_addr learned my addr 172.21.15.27:0/3043790787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52440640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c071540 0x7f2f4c0ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f51c3f640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f4c0b2b50 0x7f2f4c0b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52c41640 1 -- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c071540 msgr2=0x7f2f4c0ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52c41640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c071540 0x7f2f4c0ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52c41640 1 -- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f4c0b2b50 msgr2=0x7f2f4c0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52c41640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f4c0b2b50 0x7f2f4c0b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52c41640 1 -- 172.21.15.27:0/3043790787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c0b1650 con 0x7f2f4c0d3400 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f51c3f640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f4c0b2b50 0x7f2f4c0b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f52c41640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c0d3400 0x7f2f4c0ae020 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400c7d0 tx=0x7f2f4400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f42ffd640 1 -- 172.21.15.27:0/3043790787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44018070 con 0x7f2f4c0d3400 2024-03-06T07:23:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f42ffd640 1 -- 172.21.15.27:0/3043790787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4400a640 con 0x7f2f4c0d3400 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4c0b1930 con 0x7f2f4c0d3400 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f42ffd640 1 -- 172.21.15.27:0/3043790787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4400aa70 con 0x7f2f4c0d3400 2024-03-06T07:23:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.012+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4c0b1ef0 con 0x7f2f4c0d3400 2024-03-06T07:23:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.013+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4c068700 con 0x7f2f4c0d3400 2024-03-06T07:23:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.014+0000 7f2f42ffd640 1 -- 172.21.15.27:0/3043790787 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f4401b070 con 0x7f2f4c0d3400 2024-03-06T07:23:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.014+0000 7f2f42ffd640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f3006b4d0 0x7f2f3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.014+0000 7f2f42ffd640 1 -- 172.21.15.27:0/3043790787 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f440896e0 con 0x7f2f4c0d3400 2024-03-06T07:23:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.014+0000 7f2f52440640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f3006b4d0 0x7f2f3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.015+0000 7f2f52440640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f3006b4d0 0x7f2f3006d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c036000 tx=0x7f2f3c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.023+0000 7f2f42ffd640 1 -- 172.21.15.27:0/3043790787 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f44057fa0 con 0x7f2f4c0d3400 2024-03-06T07:23:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.205+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f4c0cf3b0 con 0x7f2f4c0d3400 2024-03-06T07:23:27.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.206+0000 7f2f42ffd640 1 -- 172.21.15.27:0/3043790787 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f2f4405c040 con 0x7f2f4c0d3400 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:22:50.070743+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f3006b4d0 msgr2=0x7f2f3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f3006b4d0 0x7f2f3006d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c036000 tx=0x7f2f3c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c0d3400 msgr2=0x7f2f4c0ae020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c0d3400 0x7f2f4c0ae020 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400c7d0 tx=0x7f2f4400cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 shutdown_connections 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f3006b4d0 0x7f2f3006d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f4c0d3400 0x7f2f4c0ae020 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f4c0b2b50 0x7f2f4c0b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 --2- 172.21.15.27:0/3043790787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f4c071540 0x7f2f4c0ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 >> 172.21.15.27:0/3043790787 conn(0x7f2f4c0d3ca0 msgr2=0x7f2f4c15f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 shutdown_connections 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.208+0000 7f2f546a4640 1 -- 172.21.15.27:0/3043790787 wait complete. 2024-03-06T07:23:27.223 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:27.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.330+0000 7f1b772f4640 1 Processor -- start 2024-03-06T07:23:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.331+0000 7f1b772f4640 1 -- start start 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.331+0000 7f1b772f4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b701445b0 0x7f1b700b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.331+0000 7f1b772f4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7006f1e0 con 0x7f1b7006ec40 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.331+0000 7f1b772f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7006f4c0 con 0x7f1b70072130 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.331+0000 7f1b772f4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b70144990 con 0x7f1b701445b0 2024-03-06T07:23:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.331+0000 7f1b75090640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b701445b0 0x7f1b700b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.332+0000 7f1b7488f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1b7006ec40 0x7f1b70071a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41478/0 (socket says 172.21.15.27:41478) 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.332+0000 7f1b7488f640 1 -- 172.21.15.27:0/1826702144 learned_addr learned my addr 172.21.15.27:0/1826702144 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.332+0000 7f1b75090640 1 -- 172.21.15.27:0/1826702144 >> v1:172.21.15.69:6789/0 conn(0x7f1b70072130 legacy=0x7f1b700ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.332+0000 7f1b75090640 1 -- 172.21.15.27:0/1826702144 >> v1:172.21.15.27:6789/0 conn(0x7f1b7006ec40 legacy=0x7f1b70071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.332+0000 7f1b75090640 1 -- 172.21.15.27:0/1826702144 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7006f7a0 con 0x7f1b701445b0 2024-03-06T07:23:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.332+0000 7f1b67fff640 1 -- 172.21.15.27:0/1826702144 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123746080 0 0) 0x7f1b7006f1e0 con 0x7f1b7006ec40 2024-03-06T07:23:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.333+0000 7f1b75090640 1 --2- 172.21.15.27:0/1826702144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b701445b0 0x7f1b700b43c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0097a0 tx=0x7f1b6c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d236517e8f7d6a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.333+0000 7f1b67fff640 1 -- 172.21.15.27:0/1826702144 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c034070 con 0x7f1b701445b0 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.333+0000 7f1b67fff640 1 -- 172.21.15.27:0/1826702144 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6c030610 con 0x7f1b701445b0 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.333+0000 7f1b67fff640 1 -- 172.21.15.27:0/1826702144 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c0309e0 con 0x7f1b701445b0 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.334+0000 7f1b772f4640 1 -- 172.21.15.27:0/1826702144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b701445b0 msgr2=0x7f1b700b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.334+0000 7f1b772f4640 1 --2- 172.21.15.27:0/1826702144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b701445b0 0x7f1b700b43c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0097a0 tx=0x7f1b6c02b960 comp rx=0 tx=0).stop 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.334+0000 7f1b772f4640 1 -- 172.21.15.27:0/1826702144 shutdown_connections 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.334+0000 7f1b772f4640 1 --2- 172.21.15.27:0/1826702144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b701445b0 0x7f1b700b43c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.334+0000 7f1b772f4640 1 -- 172.21.15.27:0/1826702144 >> 172.21.15.27:0/1826702144 conn(0x7f1b700d09c0 msgr2=0x7f1b700d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.334+0000 7f1b772f4640 1 -- 172.21.15.27:0/1826702144 shutdown_connections 2024-03-06T07:23:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.334+0000 7f1b772f4640 1 -- 172.21.15.27:0/1826702144 wait complete. 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.335+0000 7f1b772f4640 1 Processor -- start 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.335+0000 7f1b772f4640 1 -- start start 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b772f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7006ec40 0x7f1b701e5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b772f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b70072130 0x7f1b7013a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b772f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b701445b0 0x7f1b7015ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b772f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700cdc70 con 0x7f1b70072130 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b772f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700b4900 con 0x7f1b701445b0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b772f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700cddf0 con 0x7f1b7006ec40 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b7488f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b70072130 0x7f1b7013a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b7488f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b70072130 0x7f1b7013a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48524/0 (socket says 172.21.15.27:48524) 2024-03-06T07:23:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b7488f640 1 -- 172.21.15.27:0/3797249424 learned_addr learned my addr 172.21.15.27:0/3797249424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b75090640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7006ec40 0x7f1b701e5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b7488f640 1 -- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7006ec40 msgr2=0x7f1b701e5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b7488f640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7006ec40 0x7f1b701e5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b75891640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b701445b0 0x7f1b7015ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b7488f640 1 -- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b701445b0 msgr2=0x7f1b7015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b7488f640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b701445b0 0x7f1b7015ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b7488f640 1 -- 172.21.15.27:0/3797249424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7015bf90 con 0x7f1b70072130 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b75090640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7006ec40 0x7f1b701e5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b75891640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b701445b0 0x7f1b7015ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.336+0000 7f1b7488f640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b70072130 0x7f1b7013a110 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000c4a0 tx=0x7f1b6000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.337+0000 7f1b65ffb640 1 -- 172.21.15.27:0/3797249424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60018070 con 0x7f1b70072130 2024-03-06T07:23:27.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.337+0000 7f1b65ffb640 1 -- 172.21.15.27:0/3797249424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b600074e0 con 0x7f1b70072130 2024-03-06T07:23:27.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.337+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7015c210 con 0x7f1b70072130 2024-03-06T07:23:27.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.337+0000 7f1b65ffb640 1 -- 172.21.15.27:0/3797249424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60004a20 con 0x7f1b70072130 2024-03-06T07:23:27.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.337+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b701e3920 con 0x7f1b70072130 2024-03-06T07:23:27.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.338+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b700733d0 con 0x7f1b70072130 2024-03-06T07:23:27.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.339+0000 7f1b65ffb640 1 -- 172.21.15.27:0/3797249424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b6001b070 con 0x7f1b70072130 2024-03-06T07:23:27.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.339+0000 7f1b65ffb640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b5406b500 0x7f1b5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.339+0000 7f1b65ffb640 1 -- 172.21.15.27:0/3797249424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b60089450 con 0x7f1b70072130 2024-03-06T07:23:27.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.339+0000 7f1b75090640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b5406b500 0x7f1b5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.340+0000 7f1b75090640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b5406b500 0x7f1b5406d9c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c036000 tx=0x7f1b6c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.348+0000 7f1b65ffb640 1 -- 172.21.15.27:0/3797249424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b7015bf90 con 0x7f1b70072130 2024-03-06T07:23:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.629+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b7015e4c0 con 0x7f1b70072130 2024-03-06T07:23:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.630+0000 7f1b65ffb640 1 -- 172.21.15.27:0/3797249424 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b60058360 con 0x7f1b70072130 2024-03-06T07:23:27.632 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b5406b500 msgr2=0x7f1b5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b5406b500 0x7f1b5406d9c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c036000 tx=0x7f1b6c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b70072130 msgr2=0x7f1b7013a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b70072130 0x7f1b7013a110 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000c4a0 tx=0x7f1b6000c960 comp rx=0 tx=0).stop 2024-03-06T07:23:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 shutdown_connections 2024-03-06T07:23:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b5406b500 0x7f1b5406d9c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b701445b0 0x7f1b7015ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b70072130 0x7f1b7013a110 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 --2- 172.21.15.27:0/3797249424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7006ec40 0x7f1b701e5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 >> 172.21.15.27:0/3797249424 conn(0x7f1b700d09c0 msgr2=0x7f1b70142aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 shutdown_connections 2024-03-06T07:23:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.632+0000 7f1b772f4640 1 -- 172.21.15.27:0/3797249424 wait complete. 2024-03-06T07:23:27.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.754+0000 7f73bb9bc640 1 Processor -- start 2024-03-06T07:23:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.755+0000 7f73bb9bc640 1 -- start start 2024-03-06T07:23:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.755+0000 7f73bb9bc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73b415dd90 0x7f73b40ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.755+0000 7f73bb9bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 0x7f73b40b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.755+0000 7f73bb9bc640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b40d3740 con 0x7f73b40d29d0 2024-03-06T07:23:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.755+0000 7f73bb9bc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b40d3400 con 0x7f73b415dd90 2024-03-06T07:23:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.755+0000 7f73bb9bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40d3120 con 0x7f73b40b6d10 2024-03-06T07:23:27.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.756+0000 7f73b9758640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73b415dd90 0x7f73b40ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.756+0000 7f73b8f57640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f73b40d29d0 0x7f73b40d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41482/0 (socket says 172.21.15.27:41482) 2024-03-06T07:23:27.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.756+0000 7f73b8f57640 1 -- 172.21.15.27:0/2019731274 learned_addr learned my addr 172.21.15.27:0/2019731274 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:27.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.756+0000 7f73b9f59640 1 --2- 172.21.15.27:0/2019731274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 0x7f73b40b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.756+0000 7f73b9758640 1 -- 172.21.15.27:0/2019731274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 msgr2=0x7f73b40b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.756+0000 7f73b9758640 1 --2- 172.21.15.27:0/2019731274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 0x7f73b40b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.756+0000 7f73b9758640 1 -- 172.21.15.27:0/2019731274 >> v1:172.21.15.27:6789/0 conn(0x7f73b40d29d0 legacy=0x7f73b40d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.756+0000 7f73a7fff640 1 -- 172.21.15.27:0/2019731274 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336568152 0 0) 0x7f73b40d3740 con 0x7f73b40d29d0 2024-03-06T07:23:27.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.756+0000 7f73b9758640 1 -- 172.21.15.27:0/2019731274 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b40d39e0 con 0x7f73b415dd90 2024-03-06T07:23:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73b9758640 1 --2- 172.21.15.27:0/2019731274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 0x7f73b40ceea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f73b00097a0 tx=0x7f73b002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36060c56b6472e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73a7fff640 1 -- 172.21.15.27:0/2019731274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0034070 con 0x7f73b415dd90 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73a7fff640 1 -- 172.21.15.27:0/2019731274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0030610 con 0x7f73b415dd90 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73a7fff640 1 -- 172.21.15.27:0/2019731274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b00309e0 con 0x7f73b415dd90 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73bb9bc640 1 -- 172.21.15.27:0/2019731274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 msgr2=0x7f73b40ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73bb9bc640 1 --2- 172.21.15.27:0/2019731274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 0x7f73b40ceea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f73b00097a0 tx=0x7f73b002b960 comp rx=0 tx=0).stop 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73bb9bc640 1 -- 172.21.15.27:0/2019731274 shutdown_connections 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73bb9bc640 1 --2- 172.21.15.27:0/2019731274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 0x7f73b40b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73bb9bc640 1 --2- 172.21.15.27:0/2019731274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 0x7f73b40ceea0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.757+0000 7f73bb9bc640 1 -- 172.21.15.27:0/2019731274 >> 172.21.15.27:0/2019731274 conn(0x7f73b4071960 msgr2=0x7f73b4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.758+0000 7f73bb9bc640 1 -- 172.21.15.27:0/2019731274 shutdown_connections 2024-03-06T07:23:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.758+0000 7f73bb9bc640 1 -- 172.21.15.27:0/2019731274 wait complete. 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.759+0000 7f73bb9bc640 1 Processor -- start 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.759+0000 7f73bb9bc640 1 -- start start 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.759+0000 7f73bb9bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 0x7f73b40ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73bb9bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b40d29d0 0x7f73b41589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73bb9bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 0x7f73b40cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73bb9bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40d2df0 con 0x7f73b415dd90 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73bb9bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b415e170 con 0x7f73b40d29d0 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73bb9bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40d48f0 con 0x7f73b40b6d10 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9f59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 0x7f73b40cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9f59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 0x7f73b40cd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48532/0 (socket says 172.21.15.27:48532) 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9f59640 1 -- 172.21.15.27:0/1560139128 learned_addr learned my addr 172.21.15.27:0/1560139128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9758640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 0x7f73b40ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9f59640 1 -- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 msgr2=0x7f73b40ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9f59640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 0x7f73b40ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9f59640 1 -- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b40d29d0 msgr2=0x7f73b41589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9f59640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b40d29d0 0x7f73b41589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9f59640 1 -- 172.21.15.27:0/1560139128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b40cbd80 con 0x7f73b415dd90 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73b9f59640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 0x7f73b40cd2a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f73ac007e60 tx=0x7f73ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73a5ffb640 1 -- 172.21.15.27:0/1560139128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac014420 con 0x7f73b415dd90 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73a5ffb640 1 -- 172.21.15.27:0/1560139128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac019070 con 0x7f73b415dd90 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b40cc060 con 0x7f73b415dd90 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.760+0000 7f73a5ffb640 1 -- 172.21.15.27:0/1560139128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac014820 con 0x7f73b415dd90 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.761+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b40cc620 con 0x7f73b415dd90 2024-03-06T07:23:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.762+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b4072a30 con 0x7f73b415dd90 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.762+0000 7f73a5ffb640 1 -- 172.21.15.27:0/1560139128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73ac0149c0 con 0x7f73b415dd90 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.762+0000 7f73a5ffb640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b4d0 0x7f739806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.762+0000 7f73a5ffb640 1 -- 172.21.15.27:0/1560139128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73ac08aa60 con 0x7f73b415dd90 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.763+0000 7f73b9758640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b4d0 0x7f739806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.763+0000 7f73b9758640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b4d0 0x7f739806d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f73b0036000 tx=0x7f73b0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.767+0000 7f73a5ffb640 1 -- 172.21.15.27:0/1560139128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73ac0598e0 con 0x7f73b415dd90 2024-03-06T07:23:27.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.951+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f73b40b7be0 con 0x7f73b415dd90 2024-03-06T07:23:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.952+0000 7f73a5ffb640 1 -- 172.21.15.27:0/1560139128 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f73ac0591e0 con 0x7f73b415dd90 2024-03-06T07:23:28.005 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b4d0 msgr2=0x7f739806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b4d0 0x7f739806d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f73b0036000 tx=0x7f73b0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 msgr2=0x7f73b40cd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 0x7f73b40cd2a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f73ac007e60 tx=0x7f73ac00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 shutdown_connections 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b4d0 0x7f739806d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415dd90 0x7f73b40cd2a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b40d29d0 0x7f73b41589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 --2- 172.21.15.27:0/1560139128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b6d10 0x7f73b40ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 >> 172.21.15.27:0/1560139128 conn(0x7f73b4071960 msgr2=0x7f73b40d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 shutdown_connections 2024-03-06T07:23:28.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:27.954+0000 7f73bb9bc640 1 -- 172.21.15.27:0/1560139128 wait complete. 2024-03-06T07:23:28.008 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.119+0000 7fa927d19640 1 Processor -- start 2024-03-06T07:23:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.120+0000 7fa927d19640 1 -- start start 2024-03-06T07:23:28.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.120+0000 7fa927d19640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa92015dd90 0x7fa9200ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.120+0000 7fa927d19640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9200b6d10 0x7fa9200b7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.120+0000 7fa927d19640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9200d3740 con 0x7fa9200d29d0 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.120+0000 7fa927d19640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9200d3400 con 0x7fa9200b6d10 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.120+0000 7fa927d19640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9200b7690 con 0x7fa92015dd90 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.121+0000 7fa9262b6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9200b6d10 0x7fa9200b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.121+0000 7fa9252b4640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa9200d29d0 0x7fa9200d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41484/0 (socket says 172.21.15.27:41484) 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.121+0000 7fa9252b4640 1 -- 172.21.15.27:0/1154892491 learned_addr learned my addr 172.21.15.27:0/1154892491 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.121+0000 7fa925ab5640 1 --2- 172.21.15.27:0/1154892491 >> v2:172.21.15.69:3300/0 conn(0x7fa92015dd90 0x7fa9200ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.121+0000 7fa9262b6640 1 -- 172.21.15.27:0/1154892491 >> v2:172.21.15.69:3300/0 conn(0x7fa92015dd90 msgr2=0x7fa9200ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.121+0000 7fa9262b6640 1 --2- 172.21.15.27:0/1154892491 >> v2:172.21.15.69:3300/0 conn(0x7fa92015dd90 0x7fa9200ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.121+0000 7fa9262b6640 1 -- 172.21.15.27:0/1154892491 >> v1:172.21.15.27:6789/0 conn(0x7fa9200d29d0 legacy=0x7fa9200d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.121+0000 7fa9262b6640 1 -- 172.21.15.27:0/1154892491 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9200d3120 con 0x7fa9200b6d10 2024-03-06T07:23:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.121+0000 7fa924ab3640 1 -- 172.21.15.27:0/1154892491 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196684404 0 0) 0x7fa9200d3740 con 0x7fa9200d29d0 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.122+0000 7fa9262b6640 1 --2- 172.21.15.27:0/1154892491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 0x7fa9200b7150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa9200d1f90 tx=0x7fa91802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b282719d59db3453 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.122+0000 7fa924ab3640 1 -- 172.21.15.27:0/1154892491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918039070 con 0x7fa9200b6d10 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.122+0000 7fa924ab3640 1 -- 172.21.15.27:0/1154892491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91802cd00 con 0x7fa9200b6d10 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.122+0000 7fa924ab3640 1 -- 172.21.15.27:0/1154892491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918034660 con 0x7fa9200b6d10 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.123+0000 7fa927d19640 1 -- 172.21.15.27:0/1154892491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 msgr2=0x7fa9200b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.123+0000 7fa927d19640 1 --2- 172.21.15.27:0/1154892491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 0x7fa9200b7150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa9200d1f90 tx=0x7fa91802bf50 comp rx=0 tx=0).stop 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.123+0000 7fa927d19640 1 -- 172.21.15.27:0/1154892491 shutdown_connections 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.123+0000 7fa927d19640 1 --2- 172.21.15.27:0/1154892491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 0x7fa9200b7150 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.123+0000 7fa927d19640 1 --2- 172.21.15.27:0/1154892491 >> v2:172.21.15.69:3300/0 conn(0x7fa92015dd90 0x7fa9200ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.123+0000 7fa927d19640 1 -- 172.21.15.27:0/1154892491 >> 172.21.15.27:0/1154892491 conn(0x7fa920071960 msgr2=0x7fa920071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.123+0000 7fa927d19640 1 -- 172.21.15.27:0/1154892491 shutdown_connections 2024-03-06T07:23:28.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.123+0000 7fa927d19640 1 -- 172.21.15.27:0/1154892491 wait complete. 2024-03-06T07:23:28.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.125+0000 7fa927d19640 1 Processor -- start 2024-03-06T07:23:28.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.125+0000 7fa927d19640 1 -- start start 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.126+0000 7fa927d19640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 0x7fa92015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.126+0000 7fa927d19640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9200d29d0 0x7fa9200d43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.126+0000 7fa927d19640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa92015dd90 0x7fa92014fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.126+0000 7fa927d19640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92015e170 con 0x7fa9200b6d10 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.126+0000 7fa927d19640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9200d3cc0 con 0x7fa9200d29d0 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.126+0000 7fa927d19640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9200d2df0 con 0x7fa92015dd90 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.126+0000 7fa925ab5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 0x7fa92015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.126+0000 7fa925ab5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 0x7fa92015e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48554/0 (socket says 172.21.15.27:48554) 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.126+0000 7fa925ab5640 1 -- 172.21.15.27:0/2639359578 learned_addr learned my addr 172.21.15.27:0/2639359578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa925ab5640 1 -- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa92015dd90 msgr2=0x7fa92014fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa925ab5640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa92015dd90 0x7fa92014fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa925ab5640 1 -- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9200d29d0 msgr2=0x7fa9200d43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa925ab5640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9200d29d0 0x7fa9200d43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa925ab5640 1 -- 172.21.15.27:0/2639359578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92015fd70 con 0x7fa9200b6d10 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa925ab5640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 0x7fa92015e2f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00d5f0 tx=0x7fa91c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa9167fc640 1 -- 172.21.15.27:0/2639359578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c0125f0 con 0x7fa9200b6d10 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa9167fc640 1 -- 172.21.15.27:0/2639359578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c019070 con 0x7fa9200b6d10 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9201f7240 con 0x7fa9200b6d10 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa9167fc640 1 -- 172.21.15.27:0/2639359578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c0129d0 con 0x7fa9200b6d10 2024-03-06T07:23:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.127+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9201f7ae0 con 0x7fa9200b6d10 2024-03-06T07:23:28.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.128+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9201f7630 con 0x7fa9200b6d10 2024-03-06T07:23:28.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.129+0000 7fa9167fc640 1 -- 172.21.15.27:0/2639359578 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa91c012b70 con 0x7fa9200b6d10 2024-03-06T07:23:28.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.129+0000 7fa9167fc640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa90406b4d0 0x7fa90406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.129+0000 7fa9167fc640 1 -- 172.21.15.27:0/2639359578 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa91c08a580 con 0x7fa9200b6d10 2024-03-06T07:23:28.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.129+0000 7fa9252b4640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa90406b4d0 0x7fa90406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.130+0000 7fa9252b4640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa90406b4d0 0x7fa90406d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa910007fd0 tx=0x7fa910007920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.133+0000 7fa9167fc640 1 -- 172.21.15.27:0/2639359578 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa91c059490 con 0x7fa9200b6d10 2024-03-06T07:23:28.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.348+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa9200b5540 con 0x7fa9200b6d10 2024-03-06T07:23:28.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.348+0000 7fa9167fc640 1 -- 172.21.15.27:0/2639359578 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa9200b5540 con 0x7fa9200b6d10 2024-03-06T07:23:28.351 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:28.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa90406b4d0 msgr2=0x7fa90406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa90406b4d0 0x7fa90406d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa910007fd0 tx=0x7fa910007920 comp rx=0 tx=0).stop 2024-03-06T07:23:28.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 msgr2=0x7fa92015e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 0x7fa92015e2f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00d5f0 tx=0x7fa91c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 shutdown_connections 2024-03-06T07:23:28.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa90406b4d0 0x7fa90406d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa92015dd90 0x7fa92014fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9200d29d0 0x7fa9200d43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 --2- 172.21.15.27:0/2639359578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9200b6d10 0x7fa92015e2f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 >> 172.21.15.27:0/2639359578 conn(0x7fa920071960 msgr2=0x7fa9200b96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 shutdown_connections 2024-03-06T07:23:28.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.350+0000 7fa927d19640 1 -- 172.21.15.27:0/2639359578 wait complete. 2024-03-06T07:23:28.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:28.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.470+0000 7fb1d0da0640 1 Processor -- start 2024-03-06T07:23:28.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.470+0000 7fb1d0da0640 1 -- start start 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1d0da0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1cc0ce9f0 0x7fb1cc0d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1d0da0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1cc06bb40 0x7fb1cc06bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1d0da0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 0x7fb1cc161e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1d0da0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1cc0cfea0 con 0x7fb1cc0ce9f0 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1d0da0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1cc0d0180 con 0x7fb1cc06bb40 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1d0da0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc0d0460 con 0x7fb1cc1619c0 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1cb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1cc0ce9f0 0x7fb1cc0d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1cb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1cc0ce9f0 0x7fb1cc0d0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48556/0 (socket says 172.21.15.27:48556) 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1cb7fe640 1 -- 172.21.15.27:0/3551762691 learned_addr learned my addr 172.21.15.27:0/3551762691 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:28.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1cbfff640 1 --2- 172.21.15.27:0/3551762691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 0x7fb1cc161e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1caffd640 1 --2- 172.21.15.27:0/3551762691 >> v2:172.21.15.69:3300/0 conn(0x7fb1cc06bb40 0x7fb1cc06bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1cb7fe640 1 -- 172.21.15.27:0/3551762691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 msgr2=0x7fb1cc161e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1cb7fe640 1 --2- 172.21.15.27:0/3551762691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 0x7fb1cc161e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1cb7fe640 1 -- 172.21.15.27:0/3551762691 >> v2:172.21.15.69:3300/0 conn(0x7fb1cc06bb40 msgr2=0x7fb1cc06bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1cb7fe640 1 --2- 172.21.15.27:0/3551762691 >> v2:172.21.15.69:3300/0 conn(0x7fb1cc06bb40 0x7fb1cc06bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.471+0000 7fb1cb7fe640 1 -- 172.21.15.27:0/3551762691 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc0d0740 con 0x7fb1cc0ce9f0 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.472+0000 7fb1cb7fe640 1 --2- 172.21.15.27:0/3551762691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc0ce9f0 0x7fb1cc0d0f40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc071b00 tx=0x7fb1bc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbaa41b69edb15b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.472+0000 7fb1ca7fc640 1 -- 172.21.15.27:0/3551762691 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc02cb00 con 0x7fb1cc0ce9f0 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.472+0000 7fb1ca7fc640 1 -- 172.21.15.27:0/3551762691 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc02cca0 con 0x7fb1cc0ce9f0 2024-03-06T07:23:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.472+0000 7fb1ca7fc640 1 -- 172.21.15.27:0/3551762691 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc034610 con 0x7fb1cc0ce9f0 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.473+0000 7fb1d0da0640 1 -- 172.21.15.27:0/3551762691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc0ce9f0 msgr2=0x7fb1cc0d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.473+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/3551762691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc0ce9f0 0x7fb1cc0d0f40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc071b00 tx=0x7fb1bc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.473+0000 7fb1d0da0640 1 -- 172.21.15.27:0/3551762691 shutdown_connections 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.473+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/3551762691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 0x7fb1cc161e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.473+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/3551762691 >> v2:172.21.15.69:3300/0 conn(0x7fb1cc06bb40 0x7fb1cc06bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.473+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/3551762691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc0ce9f0 0x7fb1cc0d0f40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.473+0000 7fb1d0da0640 1 -- 172.21.15.27:0/3551762691 >> 172.21.15.27:0/3551762691 conn(0x7fb1cc0b3430 msgr2=0x7fb1cc0b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.473+0000 7fb1d0da0640 1 -- 172.21.15.27:0/3551762691 shutdown_connections 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.473+0000 7fb1d0da0640 1 -- 172.21.15.27:0/3551762691 wait complete. 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.475+0000 7fb1d0da0640 1 Processor -- start 2024-03-06T07:23:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.475+0000 7fb1d0da0640 1 -- start start 2024-03-06T07:23:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.476+0000 7fb1d0da0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc06bb40 0x7fb1cc0b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1d0da0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1cc0ce9f0 0x7fb1cc06fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1d0da0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 0x7fb1cc1584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1d0da0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc15f7c0 con 0x7fb1cc06bb40 2024-03-06T07:23:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1d0da0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc162720 con 0x7fb1cc0ce9f0 2024-03-06T07:23:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1d0da0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc15f940 con 0x7fb1cc1619c0 2024-03-06T07:23:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc06bb40 0x7fb1cc0b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc06bb40 0x7fb1cc0b6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48558/0 (socket says 172.21.15.27:48558) 2024-03-06T07:23:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cb7fe640 1 -- 172.21.15.27:0/866870629 learned_addr learned my addr 172.21.15.27:0/866870629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1caffd640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1cc0ce9f0 0x7fb1cc06fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cbfff640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 0x7fb1cc1584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cb7fe640 1 -- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 msgr2=0x7fb1cc1584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cb7fe640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 0x7fb1cc1584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cb7fe640 1 -- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1cc0ce9f0 msgr2=0x7fb1cc06fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cb7fe640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1cc0ce9f0 0x7fb1cc06fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cb7fe640 1 -- 172.21.15.27:0/866870629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc0c9e70 con 0x7fb1cc06bb40 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cbfff640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 0x7fb1cc1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1cb7fe640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc06bb40 0x7fb1cc0b6a20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc036040 tx=0x7fb1bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1b3fff640 1 -- 172.21.15.27:0/866870629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc043020 con 0x7fb1cc06bb40 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1b3fff640 1 -- 172.21.15.27:0/866870629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc02ce50 con 0x7fb1cc06bb40 2024-03-06T07:23:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1b3fff640 1 -- 172.21.15.27:0/866870629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc009510 con 0x7fb1cc06bb40 2024-03-06T07:23:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1cc0ca150 con 0x7fb1cc06bb40 2024-03-06T07:23:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.477+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1cc0ca9f0 con 0x7fb1cc06bb40 2024-03-06T07:23:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.478+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1cc04f330 con 0x7fb1cc06bb40 2024-03-06T07:23:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.479+0000 7fb1b3fff640 1 -- 172.21.15.27:0/866870629 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1bc046020 con 0x7fb1cc06bb40 2024-03-06T07:23:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.480+0000 7fb1b3fff640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1b806b4d0 0x7fb1b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.480+0000 7fb1b3fff640 1 -- 172.21.15.27:0/866870629 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb1bc0ac920 con 0x7fb1cc06bb40 2024-03-06T07:23:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.480+0000 7fb1caffd640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1b806b4d0 0x7fb1b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.481+0000 7fb1caffd640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1b806b4d0 0x7fb1b806d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc1599c0 tx=0x7fb1c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.489+0000 7fb1b3fff640 1 -- 172.21.15.27:0/866870629 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1bc04d030 con 0x7fb1cc06bb40 2024-03-06T07:23:28.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.682+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb1cc0b7840 con 0x7fb1cc06bb40 2024-03-06T07:23:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.682+0000 7fb1b3fff640 1 -- 172.21.15.27:0/866870629 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb1bc07b720 con 0x7fb1cc06bb40 2024-03-06T07:23:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:23:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1b806b4d0 msgr2=0x7fb1b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1b806b4d0 0x7fb1b806d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc1599c0 tx=0x7fb1c0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc06bb40 msgr2=0x7fb1cc0b6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc06bb40 0x7fb1cc0b6a20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc036040 tx=0x7fb1bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 shutdown_connections 2024-03-06T07:23:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1b806b4d0 0x7fb1b806d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1cc1619c0 0x7fb1cc1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1cc0ce9f0 0x7fb1cc06fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 --2- 172.21.15.27:0/866870629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1cc06bb40 0x7fb1cc0b6a20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 >> 172.21.15.27:0/866870629 conn(0x7fb1cc0b3430 msgr2=0x7fb1cc06c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 shutdown_connections 2024-03-06T07:23:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.684+0000 7fb1d0da0640 1 -- 172.21.15.27:0/866870629 wait complete. 2024-03-06T07:23:28.702 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:23:28.702 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:28.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:28.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.863+0000 7f868c878640 1 Processor -- start 2024-03-06T07:23:28.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.863+0000 7f868c878640 1 -- start start 2024-03-06T07:23:28.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f868c878640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86880b95a0 0x7f86880b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f868c878640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 0x7f86880b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f868c878640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86880cea30 con 0x7f868815df00 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f868c878640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86880cf3c0 con 0x7f86880b95a0 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f868c878640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86880cf0e0 con 0x7f86880b9a10 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f868659c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f868815df00 0x7f86880d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41498/0 (socket says 172.21.15.27:41498) 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f8685d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86880b95a0 0x7f86880b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f868659c640 1 -- 172.21.15.27:0/3362368572 learned_addr learned my addr 172.21.15.27:0/3362368572 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f8686d9d640 1 --2- 172.21.15.27:0/3362368572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 0x7f86880b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f8685d9b640 1 -- 172.21.15.27:0/3362368572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 msgr2=0x7f86880b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f8685d9b640 1 --2- 172.21.15.27:0/3362368572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 0x7f86880b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f8685d9b640 1 -- 172.21.15.27:0/3362368572 >> v1:172.21.15.27:6789/0 conn(0x7f868815df00 legacy=0x7f86880d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f8685d9b640 1 -- 172.21.15.27:0/3362368572 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86880ce470 con 0x7f86880b95a0 2024-03-06T07:23:28.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.864+0000 7f868559a640 1 -- 172.21.15.27:0/3362368572 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596427537 0 0) 0x7f86880cea30 con 0x7f868815df00 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.865+0000 7f8685d9b640 1 --2- 172.21.15.27:0/3362368572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86880b95a0 0x7f86880b5700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f867c0097a0 tx=0x7f867c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c01e9bfb32addd1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.865+0000 7f868559a640 1 -- 172.21.15.27:0/3362368572 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c035020 con 0x7f86880b95a0 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.865+0000 7f868559a640 1 -- 172.21.15.27:0/3362368572 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c031830 con 0x7f86880b95a0 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.865+0000 7f868559a640 1 -- 172.21.15.27:0/3362368572 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c031c00 con 0x7f86880b95a0 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.866+0000 7f868c878640 1 -- 172.21.15.27:0/3362368572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86880b95a0 msgr2=0x7f86880b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.866+0000 7f868c878640 1 --2- 172.21.15.27:0/3362368572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86880b95a0 0x7f86880b5700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f867c0097a0 tx=0x7f867c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.866+0000 7f868c878640 1 -- 172.21.15.27:0/3362368572 shutdown_connections 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.866+0000 7f868c878640 1 --2- 172.21.15.27:0/3362368572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 0x7f86880b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.866+0000 7f868c878640 1 --2- 172.21.15.27:0/3362368572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86880b95a0 0x7f86880b5700 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.866+0000 7f868c878640 1 -- 172.21.15.27:0/3362368572 >> 172.21.15.27:0/3362368572 conn(0x7f8688071960 msgr2=0x7f8688071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.866+0000 7f868c878640 1 -- 172.21.15.27:0/3362368572 shutdown_connections 2024-03-06T07:23:28.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.866+0000 7f868c878640 1 -- 172.21.15.27:0/3362368572 wait complete. 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.867+0000 7f868c878640 1 Processor -- start 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.867+0000 7f868c878640 1 -- start start 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f868c878640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880b95a0 0x7f86881f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f868c878640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 0x7f86880b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f868c878640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868815df00 0x7f868815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f868c878640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86880cedf0 con 0x7f868815df00 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f868c878640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86880cec70 con 0x7f86880b95a0 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f868c878640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86880d3220 con 0x7f86880b9a10 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8686d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868815df00 0x7f868815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8686d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868815df00 0x7f868815e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48566/0 (socket says 172.21.15.27:48566) 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8686d9d640 1 -- 172.21.15.27:0/4070075341 learned_addr learned my addr 172.21.15.27:0/4070075341 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8685d9b640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 0x7f86880b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f868659c640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880b95a0 0x7f86881f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8686d9d640 1 -- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 msgr2=0x7f86880b76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8686d9d640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 0x7f86880b76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8686d9d640 1 -- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880b95a0 msgr2=0x7f86881f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8686d9d640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880b95a0 0x7f86881f45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8686d9d640 1 -- 172.21.15.27:0/4070075341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868815f1a0 con 0x7f868815df00 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f868659c640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880b95a0 0x7f86881f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8685d9b640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 0x7f86880b76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.868+0000 7f8686d9d640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868815df00 0x7f868815e2e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8680007e40 tx=0x7f868000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.869+0000 7f8676ffd640 1 -- 172.21.15.27:0/4070075341 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8680014620 con 0x7f868815df00 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.869+0000 7f8676ffd640 1 -- 172.21.15.27:0/4070075341 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8680019070 con 0x7f868815df00 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.869+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868815eec0 con 0x7f868815df00 2024-03-06T07:23:28.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.869+0000 7f8676ffd640 1 -- 172.21.15.27:0/4070075341 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8680014a20 con 0x7f868815df00 2024-03-06T07:23:28.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.869+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86881f3df0 con 0x7f868815df00 2024-03-06T07:23:28.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.870+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8688067ff0 con 0x7f868815df00 2024-03-06T07:23:28.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.870+0000 7f8676ffd640 1 -- 172.21.15.27:0/4070075341 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8680014bc0 con 0x7f868815df00 2024-03-06T07:23:28.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.870+0000 7f8676ffd640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f866806b4d0 0x7f866806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.871+0000 7f8676ffd640 1 -- 172.21.15.27:0/4070075341 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f868008aad0 con 0x7f868815df00 2024-03-06T07:23:28.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.871+0000 7f868659c640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f866806b4d0 0x7f866806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.871+0000 7f868659c640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f866806b4d0 0x7f866806d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8678000f60 tx=0x7f8678007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:28.875+0000 7f8676ffd640 1 -- 172.21.15.27:0/4070075341 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8680059960 con 0x7f868815df00 2024-03-06T07:23:29.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.058+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86880d1270 con 0x7f868815df00 2024-03-06T07:23:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.060+0000 7f8676ffd640 1 -- 172.21.15.27:0/4070075341 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16783 (secure 0 0 0) 0x7f8680059260 con 0x7f868815df00 2024-03-06T07:23:29.062 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:29.062 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":22,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:28.204701+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:29.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f866806b4d0 msgr2=0x7f866806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f866806b4d0 0x7f866806d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8678000f60 tx=0x7f8678007ca0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868815df00 msgr2=0x7f868815e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868815df00 0x7f868815e2e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8680007e40 tx=0x7f868000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 shutdown_connections 2024-03-06T07:23:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f866806b4d0 0x7f866806d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868815df00 0x7f868815e2e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86880b9a10 0x7f86880b76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 --2- 172.21.15.27:0/4070075341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86880b95a0 0x7f86881f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 >> 172.21.15.27:0/4070075341 conn(0x7f8688071960 msgr2=0x7f868815d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 shutdown_connections 2024-03-06T07:23:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.062+0000 7f868c878640 1 -- 172.21.15.27:0/4070075341 wait complete. 2024-03-06T07:23:29.076 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:29.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.186+0000 7f439414b640 1 Processor -- start 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.187+0000 7f439414b640 1 -- start start 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f439414b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f438c071540 0x7f438c071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f439414b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438c0cee80 0x7f438c168900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f439414b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438c071e60 con 0x7f438c0b2d30 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f439414b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438c0d3230 con 0x7f438c071540 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f439414b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0d4960 con 0x7f438c0cee80 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f43916e6640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f438c0b2d30 0x7f438c0ce770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41502/0 (socket says 172.21.15.27:41502) 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f43916e6640 1 -- 172.21.15.27:0/2020648844 learned_addr learned my addr 172.21.15.27:0/2020648844 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f43926e8640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438c0cee80 0x7f438c168900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f4391ee7640 1 --2- 172.21.15.27:0/2020648844 >> v2:172.21.15.69:3300/0 conn(0x7f438c071540 0x7f438c071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.188+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704866950 0 0) 0x7f438c071e60 con 0x7f438c0b2d30 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4378003520 con 0x7f438c0b2d30 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1743651017 0 0) 0x7f4378003520 con 0x7f438c0b2d30 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438c0cee80 msgr2=0x7f438c168900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438c0cee80 0x7f438c168900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 >> v2:172.21.15.69:3300/0 conn(0x7f438c071540 msgr2=0x7f438c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 --2- 172.21.15.27:0/2020648844 >> v2:172.21.15.69:3300/0 conn(0x7f438c071540 0x7f438c071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c0d36c0 con 0x7f438c0b2d30 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f4380001670 con 0x7f438c0b2d30 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 >> v1:172.21.15.27:6789/0 conn(0x7f438c0b2d30 legacy=0x7f438c0ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43780039b0 0x7f4378003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438c0cee80 0x7f4378004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4378004d40 0x7f43780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0d3230 con 0x7f4378004d40 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0d4960 con 0x7f43780039b0 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437800b9c0 con 0x7f438c0cee80 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f43926e8640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4378004d40 0x7f43780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f4391ee7640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43780039b0 0x7f4378003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f43916e6640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438c0cee80 0x7f4378004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f43926e8640 1 -- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438c0cee80 msgr2=0x7f4378004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f43926e8640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f438c0cee80 0x7f4378004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f43926e8640 1 -- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43780039b0 msgr2=0x7f4378003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f43926e8640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43780039b0 0x7f4378003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.189+0000 7f43926e8640 1 -- 172.21.15.27:0/2020648844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4378003520 con 0x7f4378004d40 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f43926e8640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4378004d40 0x7f43780051c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f438400e170 tx=0x7f4378005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2668471676c596 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384039070 con 0x7f4378004d40 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438402cbf0 con 0x7f4378004d40 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f4390ee5640 1 -- 172.21.15.27:0/2020648844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384009c10 con 0x7f4378004d40 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f439414b640 1 -- 172.21.15.27:0/2020648844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4378004d40 msgr2=0x7f43780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f439414b640 1 --2- 172.21.15.27:0/2020648844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4378004d40 0x7f43780051c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f438400e170 tx=0x7f4378005830 comp rx=0 tx=0).stop 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f43926e8640 1 -- 172.21.15.27:0/2020648844 reap_dead start 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f439414b640 1 -- 172.21.15.27:0/2020648844 shutdown_connections 2024-03-06T07:23:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f439414b640 1 -- 172.21.15.27:0/2020648844 >> 172.21.15.27:0/2020648844 conn(0x7f438c0d3d60 msgr2=0x7f438c0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f439414b640 1 -- 172.21.15.27:0/2020648844 shutdown_connections 2024-03-06T07:23:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.191+0000 7f439414b640 1 -- 172.21.15.27:0/2020648844 wait complete. 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f439414b640 1 Processor -- start 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f439414b640 1 -- start start 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f439414b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4378004d40 0x7f438c1596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f439414b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f438c0c5780 0x7f438c0c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f439414b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438c1e9790 0x7f438c1e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f439414b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0d2be0 con 0x7f438c0c5780 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f439414b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0d2900 con 0x7f438c1e9790 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f439414b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0d2ec0 con 0x7f4378004d40 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f43916e6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f438c0c5780 0x7f438c0c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f43916e6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f438c0c5780 0x7f438c0c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48586/0 (socket says 172.21.15.27:48586) 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.193+0000 7f43916e6640 1 -- 172.21.15.27:0/2700650133 learned_addr learned my addr 172.21.15.27:0/2700650133 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f4391ee7640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4378004d40 0x7f438c1596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f43926e8640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438c1e9790 0x7f438c1e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f43916e6640 1 -- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4378004d40 msgr2=0x7f438c1596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f43916e6640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4378004d40 0x7f438c1596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f43916e6640 1 -- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438c1e9790 msgr2=0x7f438c1e9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f43916e6640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438c1e9790 0x7f438c1e9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f43916e6640 1 -- 172.21.15.27:0/2700650133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c156dd0 con 0x7f438c0c5780 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f43916e6640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f438c0c5780 0x7f438c0c5c00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4380007310 tx=0x7f4380007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f437e7fc640 1 -- 172.21.15.27:0/2700650133 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438000f9c0 con 0x7f438c0c5780 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f437e7fc640 1 -- 172.21.15.27:0/2700650133 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4380018070 con 0x7f438c0c5780 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438c1570b0 con 0x7f438c0c5780 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f437e7fc640 1 -- 172.21.15.27:0/2700650133 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438000fdc0 con 0x7f438c0c5780 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.194+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438c157670 con 0x7f438c0c5780 2024-03-06T07:23:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.195+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438c072070 con 0x7f438c0c5780 2024-03-06T07:23:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.196+0000 7f437e7fc640 1 -- 172.21.15.27:0/2700650133 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f438001d460 con 0x7f438c0c5780 2024-03-06T07:23:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.197+0000 7f437e7fc640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437006b500 0x7f437006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.197+0000 7f437e7fc640 1 -- 172.21.15.27:0/2700650133 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f438008a280 con 0x7f438c0c5780 2024-03-06T07:23:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.197+0000 7f4391ee7640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437006b500 0x7f437006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.198+0000 7f4391ee7640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437006b500 0x7f437006d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4388009bc0 tx=0x7f4388009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.206+0000 7f437e7fc640 1 -- 172.21.15.27:0/2700650133 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4380059110 con 0x7f438c0c5780 2024-03-06T07:23:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.471+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f438c04f410 con 0x7f438c0c5780 2024-03-06T07:23:29.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.471+0000 7f437e7fc640 1 -- 172.21.15.27:0/2700650133 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4380058a10 con 0x7f438c0c5780 2024-03-06T07:23:29.474 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437006b500 msgr2=0x7f437006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437006b500 0x7f437006d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4388009bc0 tx=0x7f4388009190 comp rx=0 tx=0).stop 2024-03-06T07:23:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f438c0c5780 msgr2=0x7f438c0c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f438c0c5780 0x7f438c0c5c00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4380007310 tx=0x7f4380007710 comp rx=0 tx=0).stop 2024-03-06T07:23:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 shutdown_connections 2024-03-06T07:23:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437006b500 0x7f437006d9c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f438c1e9790 0x7f438c1e9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f438c0c5780 0x7f438c0c5c00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 --2- 172.21.15.27:0/2700650133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4378004d40 0x7f438c1596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 >> 172.21.15.27:0/2700650133 conn(0x7f438c0d3d60 msgr2=0x7f438c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 shutdown_connections 2024-03-06T07:23:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.474+0000 7f439414b640 1 -- 172.21.15.27:0/2700650133 wait complete. 2024-03-06T07:23:29.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.595+0000 7f78d3fd6640 1 Processor -- start 2024-03-06T07:23:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.595+0000 7f78d3fd6640 1 -- start start 2024-03-06T07:23:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.596+0000 7f78d3fd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 0x7f78cc071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.596+0000 7f78d3fd6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78cc0b2d30 0x7f78cc0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.596+0000 7f78d3fd6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78cc0cecb0 0x7f78cc0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.596+0000 7f78d3fd6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78cc0d3230 con 0x7f78cc0cecb0 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.596+0000 7f78d3fd6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78cc071e60 con 0x7f78cc0b2d30 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.596+0000 7f78d3fd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc0d4960 con 0x7f78cc071540 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.596+0000 7f78d2573640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78cc0cecb0 0x7f78cc0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.597+0000 7f78d2573640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78cc0cecb0 0x7f78cc0cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48588/0 (socket says 172.21.15.27:48588) 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.597+0000 7f78d2573640 1 -- 172.21.15.27:0/448194556 learned_addr learned my addr 172.21.15.27:0/448194556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.597+0000 7f78d1571640 1 --2- 172.21.15.27:0/448194556 >> v2:172.21.15.69:3300/0 conn(0x7f78cc0b2d30 0x7f78cc0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.597+0000 7f78d1d72640 1 --2- 172.21.15.27:0/448194556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 0x7f78cc071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.597+0000 7f78d2573640 1 -- 172.21.15.27:0/448194556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 msgr2=0x7f78cc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.597+0000 7f78d2573640 1 --2- 172.21.15.27:0/448194556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 0x7f78cc071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.597+0000 7f78d2573640 1 -- 172.21.15.27:0/448194556 >> v2:172.21.15.69:3300/0 conn(0x7f78cc0b2d30 msgr2=0x7f78cc0ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.597+0000 7f78d2573640 1 --2- 172.21.15.27:0/448194556 >> v2:172.21.15.69:3300/0 conn(0x7f78cc0b2d30 0x7f78cc0ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.597+0000 7f78d2573640 1 -- 172.21.15.27:0/448194556 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78cc0d36c0 con 0x7f78cc0cecb0 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.598+0000 7f78d2573640 1 --2- 172.21.15.27:0/448194556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0cecb0 0x7f78cc0cf130 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0b68f0 tx=0x7f78c402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47d79a4a187c180 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.598+0000 7f78d0d70640 1 -- 172.21.15.27:0/448194556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c402ca70 con 0x7f78cc0cecb0 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.598+0000 7f78d0d70640 1 -- 172.21.15.27:0/448194556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c402cc10 con 0x7f78cc0cecb0 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.598+0000 7f78d0d70640 1 -- 172.21.15.27:0/448194556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c40343f0 con 0x7f78cc0cecb0 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.599+0000 7f78d3fd6640 1 -- 172.21.15.27:0/448194556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0cecb0 msgr2=0x7f78cc0cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.599+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/448194556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0cecb0 0x7f78cc0cf130 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0b68f0 tx=0x7f78c402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.599+0000 7f78d3fd6640 1 -- 172.21.15.27:0/448194556 shutdown_connections 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.599+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/448194556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0cecb0 0x7f78cc0cf130 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.599+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/448194556 >> v2:172.21.15.69:3300/0 conn(0x7f78cc0b2d30 0x7f78cc0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.599+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/448194556 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 0x7f78cc071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.599+0000 7f78d3fd6640 1 -- 172.21.15.27:0/448194556 >> 172.21.15.27:0/448194556 conn(0x7f78cc0d3d60 msgr2=0x7f78cc0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.599+0000 7f78d3fd6640 1 -- 172.21.15.27:0/448194556 shutdown_connections 2024-03-06T07:23:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.599+0000 7f78d3fd6640 1 -- 172.21.15.27:0/448194556 wait complete. 2024-03-06T07:23:29.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.600+0000 7f78d3fd6640 1 Processor -- start 2024-03-06T07:23:29.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.600+0000 7f78d3fd6640 1 -- start start 2024-03-06T07:23:29.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.601+0000 7f78d3fd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 0x7f78cc1e8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.601+0000 7f78d3fd6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0b2d30 0x7f78cc0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.601+0000 7f78d3fd6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc0cecb0 0x7f78cc1e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.601+0000 7f78d3fd6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc0d2be0 con 0x7f78cc0b2d30 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.601+0000 7f78d3fd6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc0d2900 con 0x7f78cc0cecb0 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.601+0000 7f78d3fd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc0d2ec0 con 0x7f78cc071540 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.601+0000 7f78d1571640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0b2d30 0x7f78cc0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.601+0000 7f78d1571640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0b2d30 0x7f78cc0cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48596/0 (socket says 172.21.15.27:48596) 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.601+0000 7f78d1571640 1 -- 172.21.15.27:0/149401895 learned_addr learned my addr 172.21.15.27:0/149401895 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d1d72640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 0x7f78cc1e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d2573640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc0cecb0 0x7f78cc1e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d1571640 1 -- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 msgr2=0x7f78cc1e8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d1571640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 0x7f78cc1e8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d1571640 1 -- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc0cecb0 msgr2=0x7f78cc1e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d1571640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc0cecb0 0x7f78cc1e49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d1571640 1 -- 172.21.15.27:0/149401895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78cc1e3910 con 0x7f78cc0b2d30 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d2573640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc0cecb0 0x7f78cc1e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d1571640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0b2d30 0x7f78cc0cda60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f78c800e9b0 tx=0x7f78c800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78c27fc640 1 -- 172.21.15.27:0/149401895 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8017070 con 0x7f78cc0b2d30 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78c27fc640 1 -- 172.21.15.27:0/149401895 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c800c9a0 con 0x7f78cc0b2d30 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78c27fc640 1 -- 172.21.15.27:0/149401895 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c800cd70 con 0x7f78cc0b2d30 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78cc1e3bf0 con 0x7f78cc0b2d30 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.602+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78cc1e41b0 con 0x7f78cc0b2d30 2024-03-06T07:23:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.603+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78cc072070 con 0x7f78cc0b2d30 2024-03-06T07:23:29.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.604+0000 7f78c27fc640 1 -- 172.21.15.27:0/149401895 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78c801a070 con 0x7f78cc0b2d30 2024-03-06T07:23:29.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.604+0000 7f78c27fc640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78b006b4d0 0x7f78b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.604+0000 7f78c27fc640 1 -- 172.21.15.27:0/149401895 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f78c808a4d0 con 0x7f78cc0b2d30 2024-03-06T07:23:29.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.604+0000 7f78d1d72640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78b006b4d0 0x7f78b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.605+0000 7f78d1d72640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78b006b4d0 0x7f78b006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009a20 tx=0x7f78bc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.609+0000 7f78c27fc640 1 -- 172.21.15.27:0/149401895 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78c8058d10 con 0x7f78cc0b2d30 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.790+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78cc04f410 con 0x7f78cc0b2d30 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.791+0000 7f78c27fc640 1 -- 172.21.15.27:0/149401895 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f78c805cdb0 con 0x7f78cc0b2d30 2024-03-06T07:23:29.793 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78b006b4d0 msgr2=0x7f78b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78b006b4d0 0x7f78b006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009a20 tx=0x7f78bc009190 comp rx=0 tx=0).stop 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0b2d30 msgr2=0x7f78cc0cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0b2d30 0x7f78cc0cda60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f78c800e9b0 tx=0x7f78c800ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 shutdown_connections 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78b006b4d0 0x7f78b006d990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc0cecb0 0x7f78cc1e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78cc0b2d30 0x7f78cc0cda60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 --2- 172.21.15.27:0/149401895 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78cc071540 0x7f78cc1e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 >> 172.21.15.27:0/149401895 conn(0x7f78cc0d3d60 msgr2=0x7f78cc15bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 shutdown_connections 2024-03-06T07:23:29.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.793+0000 7f78d3fd6640 1 -- 172.21.15.27:0/149401895 wait complete. 2024-03-06T07:23:29.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:29.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.914+0000 7f095b855640 1 Processor -- start 2024-03-06T07:23:29.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.914+0000 7f095b855640 1 -- start start 2024-03-06T07:23:29.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.915+0000 7f095b855640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09540b2d30 0x7f09540b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.915+0000 7f095b855640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09540b7820 0x7f09540b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.915+0000 7f095b855640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09540d4960 con 0x7f0954071540 2024-03-06T07:23:29.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.915+0000 7f095b855640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09540d3230 con 0x7f09540b7820 2024-03-06T07:23:29.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.915+0000 7f095b855640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09540d36c0 con 0x7f09540b2d30 2024-03-06T07:23:29.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.915+0000 7f0959df2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09540b7820 0x7f09540b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.916+0000 7f0959df2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09540b7820 0x7f09540b7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48604/0 (socket says 172.21.15.27:48604) 2024-03-06T07:23:29.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.916+0000 7f0959df2640 1 -- 172.21.15.27:0/243427191 learned_addr learned my addr 172.21.15.27:0/243427191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:29.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.916+0000 7f0958df0640 1 --2- 172.21.15.27:0/243427191 >> v2:172.21.15.69:3300/0 conn(0x7f09540b2d30 0x7f09540b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.916+0000 7f0959df2640 1 -- 172.21.15.27:0/243427191 >> v2:172.21.15.69:3300/0 conn(0x7f09540b2d30 msgr2=0x7f09540b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.916+0000 7f0959df2640 1 --2- 172.21.15.27:0/243427191 >> v2:172.21.15.69:3300/0 conn(0x7f09540b2d30 0x7f09540b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.916+0000 7f0959df2640 1 -- 172.21.15.27:0/243427191 >> v1:172.21.15.69:6789/0 conn(0x7f0954071540 legacy=0x7f0954071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.916+0000 7f0959df2640 1 -- 172.21.15.27:0/243427191 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09540d2620 con 0x7f09540b7820 2024-03-06T07:23:29.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f0959df2640 1 --2- 172.21.15.27:0/243427191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09540b7820 0x7f09540b7cd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f09540c7690 tx=0x7f094c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e71b2397b36811 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f094bfff640 1 -- 172.21.15.27:0/243427191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c02cb20 con 0x7f09540b7820 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f094bfff640 1 -- 172.21.15.27:0/243427191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c02ccc0 con 0x7f09540b7820 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f094bfff640 1 -- 172.21.15.27:0/243427191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c034610 con 0x7f09540b7820 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f095b855640 1 -- 172.21.15.27:0/243427191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09540b7820 msgr2=0x7f09540b7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f095b855640 1 --2- 172.21.15.27:0/243427191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09540b7820 0x7f09540b7cd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f09540c7690 tx=0x7f094c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f095b855640 1 -- 172.21.15.27:0/243427191 shutdown_connections 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f095b855640 1 --2- 172.21.15.27:0/243427191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09540b7820 0x7f09540b7cd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f095b855640 1 --2- 172.21.15.27:0/243427191 >> v2:172.21.15.69:3300/0 conn(0x7f09540b2d30 0x7f09540b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.917+0000 7f095b855640 1 -- 172.21.15.27:0/243427191 >> 172.21.15.27:0/243427191 conn(0x7f09540d3d60 msgr2=0x7f09540d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.918+0000 7f095b855640 1 -- 172.21.15.27:0/243427191 shutdown_connections 2024-03-06T07:23:29.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.918+0000 7f095b855640 1 -- 172.21.15.27:0/243427191 wait complete. 2024-03-06T07:23:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.919+0000 7f095b855640 1 Processor -- start 2024-03-06T07:23:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.919+0000 7f095b855640 1 -- start start 2024-03-06T07:23:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.919+0000 7f095b855640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0954071540 0x7f09540ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.919+0000 7f095b855640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09540b2d30 0x7f09541f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.919+0000 7f095b855640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09540b7820 0x7f0954153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f095b855640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09540d2ec0 con 0x7f0954071540 2024-03-06T07:23:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f095b855640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09540d2be0 con 0x7f09540b2d30 2024-03-06T07:23:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f095b855640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09540d22f0 con 0x7f09540b7820 2024-03-06T07:23:29.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f09595f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0954071540 0x7f09540ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f09595f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0954071540 0x7f09540ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48614/0 (socket says 172.21.15.27:48614) 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f09595f1640 1 -- 172.21.15.27:0/292569849 learned_addr learned my addr 172.21.15.27:0/292569849 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f0959df2640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09540b7820 0x7f0954153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f0958df0640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09540b2d30 0x7f09541f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f09595f1640 1 -- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09540b7820 msgr2=0x7f0954153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f09595f1640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09540b7820 0x7f0954153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f09595f1640 1 -- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09540b2d30 msgr2=0x7f09541f40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f09595f1640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09540b2d30 0x7f09541f40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f09595f1640 1 -- 172.21.15.27:0/292569849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09541f47f0 con 0x7f0954071540 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f0959df2640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09540b7820 0x7f0954153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f09595f1640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0954071540 0x7f09540ca7a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f095000c3f0 tx=0x7f095000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f0949ffb640 1 -- 172.21.15.27:0/292569849 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09500052d0 con 0x7f0954071540 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f0949ffb640 1 -- 172.21.15.27:0/292569849 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0950005490 con 0x7f0954071540 2024-03-06T07:23:29.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f0949ffb640 1 -- 172.21.15.27:0/292569849 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950007260 con 0x7f0954071540 2024-03-06T07:23:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.920+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09541f4ad0 con 0x7f0954071540 2024-03-06T07:23:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.921+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09541f77d0 con 0x7f0954071540 2024-03-06T07:23:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.922+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09541f4ff0 con 0x7f0954071540 2024-03-06T07:23:29.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.922+0000 7f0949ffb640 1 -- 172.21.15.27:0/292569849 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0950007400 con 0x7f0954071540 2024-03-06T07:23:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.922+0000 7f0949ffb640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f093806b4d0 0x7f093806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.922+0000 7f0949ffb640 1 -- 172.21.15.27:0/292569849 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0950089760 con 0x7f0954071540 2024-03-06T07:23:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.923+0000 7f0958df0640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f093806b4d0 0x7f093806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.923+0000 7f0958df0640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f093806b4d0 0x7f093806d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f09540d0620 tx=0x7f0944008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:29.931+0000 7f0949ffb640 1 -- 172.21.15.27:0/292569849 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0950058560 con 0x7f0954071540 2024-03-06T07:23:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.112+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0954072070 con 0x7f0954071540 2024-03-06T07:23:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.113+0000 7f0949ffb640 1 -- 172.21.15.27:0/292569849 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0950057e60 con 0x7f0954071540 2024-03-06T07:23:30.116 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f093806b4d0 msgr2=0x7f093806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f093806b4d0 0x7f093806d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f09540d0620 tx=0x7f0944008040 comp rx=0 tx=0).stop 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0954071540 msgr2=0x7f09540ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0954071540 0x7f09540ca7a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f095000c3f0 tx=0x7f095000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 shutdown_connections 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f093806b4d0 0x7f093806d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09540b7820 0x7f0954153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09540b2d30 0x7f09541f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 --2- 172.21.15.27:0/292569849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0954071540 0x7f09540ca7a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 >> 172.21.15.27:0/292569849 conn(0x7f09540d3d60 msgr2=0x7f095415b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 shutdown_connections 2024-03-06T07:23:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.115+0000 7f095b855640 1 -- 172.21.15.27:0/292569849 wait complete. 2024-03-06T07:23:30.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:30.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.241+0000 7fa30184e640 1 Processor -- start 2024-03-06T07:23:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.241+0000 7fa30184e640 1 -- start start 2024-03-06T07:23:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.242+0000 7fa30184e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.242+0000 7fa30184e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa2fc0d06c0 0x7fa2fc0b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.242+0000 7fa30184e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc0b0dd0 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.242+0000 7fa30184e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc0b0b80 con 0x7fa2fc0d06c0 2024-03-06T07:23:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.242+0000 7fa30184e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0aeec0 con 0x7fa2fc0b4260 2024-03-06T07:23:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.242+0000 7fa2fa7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.242+0000 7fa2fb7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa2fc0d06c0 0x7fa2fc0b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.242+0000 7fa2fa7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc0cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:54710/0 (socket says 172.21.15.27:54710) 2024-03-06T07:23:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.242+0000 7fa2fa7fc640 1 -- 172.21.15.27:0/3250701000 learned_addr learned my addr 172.21.15.27:0/3250701000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.243+0000 7fa2fa7fc640 1 -- 172.21.15.27:0/3250701000 >> v2:172.21.15.69:3300/0 conn(0x7fa2fc0d06c0 msgr2=0x7fa2fc0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.243+0000 7fa2fa7fc640 1 --2- 172.21.15.27:0/3250701000 >> v2:172.21.15.69:3300/0 conn(0x7fa2fc0d06c0 0x7fa2fc0b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.243+0000 7fa2fa7fc640 1 -- 172.21.15.27:0/3250701000 >> v1:172.21.15.69:6789/0 conn(0x7fa2fc0d02e0 legacy=0x7fa2fc0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.243+0000 7fa2fa7fc640 1 -- 172.21.15.27:0/3250701000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc0b10b0 con 0x7fa2fc0b4260 2024-03-06T07:23:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.243+0000 7fa2fa7fc640 1 --2- 172.21.15.27:0/3250701000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc0cfa00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009e50 tx=0x7fa2f002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c30b67f93789f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa2f9ffb640 1 -- 172.21.15.27:0/3250701000 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f002cb90 con 0x7fa2fc0b4260 2024-03-06T07:23:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa2f9ffb640 1 -- 172.21.15.27:0/3250701000 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f002cd30 con 0x7fa2fc0b4260 2024-03-06T07:23:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa2f9ffb640 1 -- 172.21.15.27:0/3250701000 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0034610 con 0x7fa2fc0b4260 2024-03-06T07:23:30.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa30184e640 1 -- 172.21.15.27:0/3250701000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 msgr2=0x7fa2fc0cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa30184e640 1 --2- 172.21.15.27:0/3250701000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc0cfa00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009e50 tx=0x7fa2f002bff0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa30184e640 1 -- 172.21.15.27:0/3250701000 shutdown_connections 2024-03-06T07:23:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa30184e640 1 --2- 172.21.15.27:0/3250701000 >> v2:172.21.15.69:3300/0 conn(0x7fa2fc0d06c0 0x7fa2fc0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa30184e640 1 --2- 172.21.15.27:0/3250701000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc0cfa00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa30184e640 1 -- 172.21.15.27:0/3250701000 >> 172.21.15.27:0/3250701000 conn(0x7fa2fc0b36d0 msgr2=0x7fa2fc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa30184e640 1 -- 172.21.15.27:0/3250701000 shutdown_connections 2024-03-06T07:23:30.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.244+0000 7fa30184e640 1 -- 172.21.15.27:0/3250701000 wait complete. 2024-03-06T07:23:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.245+0000 7fa30184e640 1 Processor -- start 2024-03-06T07:23:30.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa30184e640 1 -- start start 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa30184e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa30184e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0d02e0 0x7fa2fc0c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa30184e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2fc0d06c0 0x7fa2fc0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa30184e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0b22d0 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa30184e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0b2150 con 0x7fa2fc0d06c0 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa30184e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0af350 con 0x7fa2fc0b4260 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa2fa7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0d02e0 0x7fa2fc0c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa2fa7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0d02e0 0x7fa2fc0c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48622/0 (socket says 172.21.15.27:48622) 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.246+0000 7fa2fa7fc640 1 -- 172.21.15.27:0/328689902 learned_addr learned my addr 172.21.15.27:0/328689902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2faffd640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2fb7fe640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2fc0d06c0 0x7fa2fc0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2fa7fc640 1 -- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 msgr2=0x7fa2fc140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2fa7fc640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc140ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2fa7fc640 1 -- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2fc0d06c0 msgr2=0x7fa2fc0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2fa7fc640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2fc0d06c0 0x7fa2fc0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2fa7fc640 1 -- 172.21.15.27:0/328689902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc0c9110 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2fb7fe640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2fc0d06c0 0x7fa2fc0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2fa7fc640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0d02e0 0x7fa2fc0c82e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0000f80 tx=0x7fa2f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2e3fff640 1 -- 172.21.15.27:0/328689902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0043020 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2e3fff640 1 -- 172.21.15.27:0/328689902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f003e450 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa2e3fff640 1 -- 172.21.15.27:0/328689902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0009510 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc0c93b0 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.247+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc0c9970 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.248+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc072d90 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.249+0000 7fa2e3fff640 1 -- 172.21.15.27:0/328689902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2f0046020 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.249+0000 7fa2e3fff640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2e806b4d0 0x7fa2e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.249+0000 7fa2e3fff640 1 -- 172.21.15.27:0/328689902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa2f00ac930 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.249+0000 7fa2faffd640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2e806b4d0 0x7fa2e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.250+0000 7fa2faffd640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2e806b4d0 0x7fa2e806d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec007fd0 tx=0x7fa2ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.253+0000 7fa2e3fff640 1 -- 172.21.15.27:0/328689902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2fc072d90 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.523+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa2fc068700 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.523+0000 7fa2e3fff640 1 -- 172.21.15.27:0/328689902 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa2f004d030 con 0x7fa2fc0d02e0 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:23:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.525+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2e806b4d0 msgr2=0x7fa2e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.525+0000 7fa30184e640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2e806b4d0 0x7fa2e806d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec007fd0 tx=0x7fa2ec009040 comp rx=0 tx=0).stop 2024-03-06T07:23:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.525+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0d02e0 msgr2=0x7fa2fc0c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.525+0000 7fa30184e640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0d02e0 0x7fa2fc0c82e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0000f80 tx=0x7fa2f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.526+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 shutdown_connections 2024-03-06T07:23:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.526+0000 7fa30184e640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2e806b4d0 0x7fa2e806d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.526+0000 7fa30184e640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2fc0d06c0 0x7fa2fc0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.526+0000 7fa30184e640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0d02e0 0x7fa2fc0c82e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.526+0000 7fa30184e640 1 --2- 172.21.15.27:0/328689902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2fc0b4260 0x7fa2fc140ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.526+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 >> 172.21.15.27:0/328689902 conn(0x7fa2fc0b36d0 msgr2=0x7fa2fc0ccb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.526+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 shutdown_connections 2024-03-06T07:23:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.526+0000 7fa30184e640 1 -- 172.21.15.27:0/328689902 wait complete. 2024-03-06T07:23:30.539 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:23:30.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:30.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-06T07:23:30.570 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:30.570 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:23:30.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:30.571 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:23:30.625 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:23:30.625 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:30.643 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:23:30.643 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:30.702 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.807+0000 7fa6ccf7a640 1 Processor -- start 2024-03-06T07:23:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.807+0000 7fa6ccf7a640 1 -- start start 2024-03-06T07:23:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6ccf7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6ccf7a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6c80b9a10 0x7fa6c80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6ccf7a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c80cea30 con 0x7fa6c815df00 2024-03-06T07:23:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6ccf7a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c80cf3c0 con 0x7fa6c80b9a10 2024-03-06T07:23:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6ccf7a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80cf0e0 con 0x7fa6c80b95a0 2024-03-06T07:23:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c6d9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6c80b9a10 0x7fa6c80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c5d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c659c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa6c815df00 0x7fa6c80d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:33426/0 (socket says 172.21.15.27:33426) 2024-03-06T07:23:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c659c640 1 -- 172.21.15.27:0/2526880429 learned_addr learned my addr 172.21.15.27:0/2526880429 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c6d9d640 1 -- 172.21.15.27:0/2526880429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 msgr2=0x7fa6c80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c6d9d640 1 --2- 172.21.15.27:0/2526880429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c6d9d640 1 -- 172.21.15.27:0/2526880429 >> v1:172.21.15.69:6789/0 conn(0x7fa6c815df00 legacy=0x7fa6c80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c6d9d640 1 -- 172.21.15.27:0/2526880429 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c80ce470 con 0x7fa6c80b9a10 2024-03-06T07:23:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c559a640 1 -- 172.21.15.27:0/2526880429 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176513837 0 0) 0x7fa6c80cea30 con 0x7fa6c815df00 2024-03-06T07:23:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.808+0000 7fa6c5d9b640 1 --2- 172.21.15.27:0/2526880429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.809+0000 7fa6c6d9d640 1 --2- 172.21.15.27:0/2526880429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80b5c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009d90 tx=0x7fa6c002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80d6c36b24168da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.809+0000 7fa6c559a640 1 -- 172.21.15.27:0/2526880429 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0039070 con 0x7fa6c80b9a10 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.809+0000 7fa6c559a640 1 -- 172.21.15.27:0/2526880429 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c002cc50 con 0x7fa6c80b9a10 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.809+0000 7fa6c559a640 1 -- 172.21.15.27:0/2526880429 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0034440 con 0x7fa6c80b9a10 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.810+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/2526880429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b9a10 msgr2=0x7fa6c80b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.810+0000 7fa6ccf7a640 1 --2- 172.21.15.27:0/2526880429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80b5c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009d90 tx=0x7fa6c002bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.810+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/2526880429 shutdown_connections 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.810+0000 7fa6ccf7a640 1 --2- 172.21.15.27:0/2526880429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80b5c40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.810+0000 7fa6ccf7a640 1 --2- 172.21.15.27:0/2526880429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.810+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/2526880429 >> 172.21.15.27:0/2526880429 conn(0x7fa6c8071960 msgr2=0x7fa6c8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.810+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/2526880429 shutdown_connections 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.810+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/2526880429 wait complete. 2024-03-06T07:23:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.811+0000 7fa6ccf7a640 1 Processor -- start 2024-03-06T07:23:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.811+0000 7fa6ccf7a640 1 -- start start 2024-03-06T07:23:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6ccf7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 0x7fa6c8152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6ccf7a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6ccf7a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c815df00 0x7fa6c80ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6ccf7a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80cedf0 con 0x7fa6c80b9a10 2024-03-06T07:23:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6ccf7a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80cec70 con 0x7fa6c815df00 2024-03-06T07:23:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6ccf7a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80d3220 con 0x7fa6c80b95a0 2024-03-06T07:23:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6c5d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6c5d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80c9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48636/0 (socket says 172.21.15.27:48636) 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6c5d9b640 1 -- 172.21.15.27:0/3188391054 learned_addr learned my addr 172.21.15.27:0/3188391054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6c659c640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 0x7fa6c8152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.812+0000 7fa6c6d9d640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c815df00 0x7fa6c80ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6c5d9b640 1 -- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 msgr2=0x7fa6c8152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6c5d9b640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 0x7fa6c8152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6c5d9b640 1 -- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c815df00 msgr2=0x7fa6c80ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6c5d9b640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c815df00 0x7fa6c80ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6c5d9b640 1 -- 172.21.15.27:0/3188391054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c8152640 con 0x7fa6c80b9a10 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6c5d9b640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80c9740 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00ebe0 tx=0x7fa6bc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6b6ffd640 1 -- 172.21.15.27:0/3188391054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc017070 con 0x7fa6c80b9a10 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6b6ffd640 1 -- 172.21.15.27:0/3188391054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc00cc30 con 0x7fa6c80b9a10 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c80ca1e0 con 0x7fa6c80b9a10 2024-03-06T07:23:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6b6ffd640 1 -- 172.21.15.27:0/3188391054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc013470 con 0x7fa6c80b9a10 2024-03-06T07:23:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.813+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c81f50e0 con 0x7fa6c80b9a10 2024-03-06T07:23:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.814+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c80cab20 con 0x7fa6c80b9a10 2024-03-06T07:23:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.815+0000 7fa6b6ffd640 1 -- 172.21.15.27:0/3188391054 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6bc01a070 con 0x7fa6c80b9a10 2024-03-06T07:23:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.815+0000 7fa6b6ffd640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6a806b4d0 0x7fa6a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.815+0000 7fa6b6ffd640 1 -- 172.21.15.27:0/3188391054 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa6bc08a430 con 0x7fa6c80b9a10 2024-03-06T07:23:30.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.815+0000 7fa6c659c640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6a806b4d0 0x7fa6a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.816+0000 7fa6c659c640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6a806b4d0 0x7fa6a806d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8007fd0 tx=0x7fa6b8007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:30.820+0000 7fa6b6ffd640 1 -- 172.21.15.27:0/3188391054 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6c80cab20 con 0x7fa6c80b9a10 2024-03-06T07:23:31.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.098+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6c8067ff0 con 0x7fa6c80b9a10 2024-03-06T07:23:31.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.099+0000 7fa6b6ffd640 1 -- 172.21.15.27:0/3188391054 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16783 (secure 0 0 0) 0x7fa6c8152640 con 0x7fa6c80b9a10 2024-03-06T07:23:31.102 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:31.102 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:29.211570+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:31.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6a806b4d0 msgr2=0x7fa6a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6a806b4d0 0x7fa6a806d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8007fd0 tx=0x7fa6b8007de0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 msgr2=0x7fa6c80c9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80c9740 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00ebe0 tx=0x7fa6bc00c520 comp rx=0 tx=0).stop 2024-03-06T07:23:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 shutdown_connections 2024-03-06T07:23:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6a806b4d0 0x7fa6a806d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c815df00 0x7fa6c80ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80c9740 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 --2- 172.21.15.27:0/3188391054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c80b95a0 0x7fa6c8152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 >> 172.21.15.27:0/3188391054 conn(0x7fa6c8071960 msgr2=0x7fa6c80b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 shutdown_connections 2024-03-06T07:23:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.102+0000 7fa6ccf7a640 1 -- 172.21.15.27:0/3188391054 wait complete. 2024-03-06T07:23:31.120 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:31.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:31.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.229+0000 7f3d325bf640 1 Processor -- start 2024-03-06T07:23:31.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.229+0000 7f3d325bf640 1 -- start start 2024-03-06T07:23:31.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.229+0000 7f3d325bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c067410 0x7f3d2c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.229+0000 7f3d325bf640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c15e6d0 con 0x7f3d2c0b3ce0 2024-03-06T07:23:31.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.229+0000 7f3d325bf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c15e9b0 con 0x7f3d2c15df40 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.229+0000 7f3d325bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c15f000 con 0x7f3d2c067410 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d2c0b3ce0 0x7f3d2c06dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41508/0 (socket says 172.21.15.27:41508) 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2b7fe640 1 -- 172.21.15.27:0/1823863222 learned_addr learned my addr 172.21.15.27:0/1823863222 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920454104 0 0) 0x7f3d2c15e6d0 con 0x7f3d2c0b3ce0 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d30b5c640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c067410 0x7f3d2c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d14003520 con 0x7f3d2c0b3ce0 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928698505 0 0) 0x7f3d14003520 con 0x7f3d2c0b3ce0 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c067410 msgr2=0x7f3d2c06e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c067410 0x7f3d2c06e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 >> v1:172.21.15.69:6789/0 conn(0x7f3d2c15df40 legacy=0x7f3d2c066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c15ec90 con 0x7f3d2c0b3ce0 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f3d20001670 con 0x7f3d2c0b3ce0 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d30b5c640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c067410 0x7f3d2c06e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 >> v1:172.21.15.27:6789/0 conn(0x7f3d2c0b3ce0 legacy=0x7f3d2c06dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d140039e0 0x7f3d14003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c067410 0x7f3d14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d14004d40 0x7f3d140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c15f000 con 0x7f3d14004d40 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1400bb80 con 0x7f3d140039e0 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1400bd80 con 0x7f3d2c067410 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d30b5c640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d14004d40 0x7f3d140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.230+0000 7f3d2bfff640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d140039e0 0x7f3d14003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.231+0000 7f3d2b7fe640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c067410 0x7f3d14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.231+0000 7f3d30b5c640 1 -- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c067410 msgr2=0x7f3d14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.231+0000 7f3d30b5c640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c067410 0x7f3d14004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.231+0000 7f3d30b5c640 1 -- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d140039e0 msgr2=0x7f3d14003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.231+0000 7f3d30b5c640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d140039e0 0x7f3d14003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.231+0000 7f3d30b5c640 1 -- 172.21.15.27:0/1823863222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d14003520 con 0x7f3d14004d40 2024-03-06T07:23:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.232+0000 7f3d30b5c640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d14004d40 0x7f3d140051c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0b3680 tx=0x7f3d14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa014585cb1f7b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.232+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24039070 con 0x7f3d14004d40 2024-03-06T07:23:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.232+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2402ccd0 con 0x7f3d14004d40 2024-03-06T07:23:31.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.232+0000 7f3d2affd640 1 -- 172.21.15.27:0/1823863222 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24009c80 con 0x7f3d14004d40 2024-03-06T07:23:31.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.233+0000 7f3d325bf640 1 -- 172.21.15.27:0/1823863222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d14004d40 msgr2=0x7f3d140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.233+0000 7f3d325bf640 1 --2- 172.21.15.27:0/1823863222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d14004d40 0x7f3d140051c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0b3680 tx=0x7f3d14005830 comp rx=0 tx=0).stop 2024-03-06T07:23:31.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.233+0000 7f3d30b5c640 1 -- 172.21.15.27:0/1823863222 reap_dead start 2024-03-06T07:23:31.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.233+0000 7f3d325bf640 1 -- 172.21.15.27:0/1823863222 shutdown_connections 2024-03-06T07:23:31.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.233+0000 7f3d325bf640 1 -- 172.21.15.27:0/1823863222 >> 172.21.15.27:0/1823863222 conn(0x7f3d2c0d09c0 msgr2=0x7f3d2c0d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.233+0000 7f3d325bf640 1 -- 172.21.15.27:0/1823863222 shutdown_connections 2024-03-06T07:23:31.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.233+0000 7f3d325bf640 1 -- 172.21.15.27:0/1823863222 wait complete. 2024-03-06T07:23:31.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.234+0000 7f3d325bf640 1 Processor -- start 2024-03-06T07:23:31.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.234+0000 7f3d325bf640 1 -- start start 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d325bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c14d680 0x7f3d2c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d325bf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d2c14da60 0x7f3d2c0b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d325bf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d2c0c64c0 0x7f3d2c0c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d325bf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c06e520 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d325bf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c06e3a0 con 0x7f3d2c14da60 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d325bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c06e6a0 con 0x7f3d2c14d680 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d30b5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d2c0c64c0 0x7f3d2c0c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d30b5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d2c0c64c0 0x7f3d2c0c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48666/0 (socket says 172.21.15.27:48666) 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d30b5c640 1 -- 172.21.15.27:0/2741528759 learned_addr learned my addr 172.21.15.27:0/2741528759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d2bfff640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c14d680 0x7f3d2c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d2b7fe640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d2c14da60 0x7f3d2c0b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d30b5c640 1 -- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c14d680 msgr2=0x7f3d2c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d30b5c640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c14d680 0x7f3d2c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d30b5c640 1 -- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d2c14da60 msgr2=0x7f3d2c0b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d30b5c640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d2c14da60 0x7f3d2c0b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.235+0000 7f3d30b5c640 1 -- 172.21.15.27:0/2741528759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c160540 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.236+0000 7f3d30b5c640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d2c0c64c0 0x7f3d2c0c6920 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3d240074a0 tx=0x7f3d24004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.236+0000 7f3d28ff9640 1 -- 172.21.15.27:0/2741528759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24039030 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.236+0000 7f3d28ff9640 1 -- 172.21.15.27:0/2741528759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2403a040 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.236+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2c160820 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.236+0000 7f3d28ff9640 1 -- 172.21.15.27:0/2741528759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24009c80 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.236+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c160de0 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.237+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2c0733d0 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.237+0000 7f3d28ff9640 1 -- 172.21.15.27:0/2741528759 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d24045020 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.238+0000 7f3d28ff9640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1806b4d0 0x7f3d1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.238+0000 7f3d28ff9640 1 -- 172.21.15.27:0/2741528759 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3d240aba40 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.238+0000 7f3d2bfff640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1806b4d0 0x7f3d1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.239+0000 7f3d2bfff640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1806b4d0 0x7f3d1806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0035f0 tx=0x7f3d1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.242+0000 7f3d28ff9640 1 -- 172.21.15.27:0/2741528759 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d2404c020 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.421+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3d2c0b5710 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.421+0000 7f3d28ff9640 1 -- 172.21.15.27:0/2741528759 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3d24080070 con 0x7f3d2c0c64c0 2024-03-06T07:23:31.424 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1806b4d0 msgr2=0x7f3d1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1806b4d0 0x7f3d1806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0035f0 tx=0x7f3d1c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d2c0c64c0 msgr2=0x7f3d2c0c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d2c0c64c0 0x7f3d2c0c6920 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3d240074a0 tx=0x7f3d24004ab0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 shutdown_connections 2024-03-06T07:23:31.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1806b4d0 0x7f3d1806d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d2c0c64c0 0x7f3d2c0c6920 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d2c14da60 0x7f3d2c0b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 --2- 172.21.15.27:0/2741528759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d2c14d680 0x7f3d2c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 >> 172.21.15.27:0/2741528759 conn(0x7f3d2c0d09c0 msgr2=0x7f3d2c06dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 shutdown_connections 2024-03-06T07:23:31.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.423+0000 7f3d325bf640 1 -- 172.21.15.27:0/2741528759 wait complete. 2024-03-06T07:23:31.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.545+0000 7f36c42a4640 1 Processor -- start 2024-03-06T07:23:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.545+0000 7f36c42a4640 1 -- start start 2024-03-06T07:23:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c42a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36bc0b7420 0x7f36bc0d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c42a4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0d3740 con 0x7f36bc15dd90 2024-03-06T07:23:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c42a4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0d3120 con 0x7f36bc0ceea0 2024-03-06T07:23:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c42a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36bc0d3400 con 0x7f36bc0b7420 2024-03-06T07:23:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c2040640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f36bc15dd90 0x7f36bc0d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41514/0 (socket says 172.21.15.27:41514) 2024-03-06T07:23:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c2040640 1 -- 172.21.15.27:0/1782927560 learned_addr learned my addr 172.21.15.27:0/1782927560 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938532187 0 0) 0x7f36bc0d3740 con 0x7f36bc15dd90 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c2841640 1 --2- 172.21.15.27:0/1782927560 >> v2:172.21.15.69:3300/0 conn(0x7f36bc0b7420 0x7f36bc0d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a8003520 con 0x7f36bc15dd90 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4270892233 0 0) 0x7f36a8003520 con 0x7f36bc15dd90 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 >> v2:172.21.15.69:3300/0 conn(0x7f36bc0b7420 msgr2=0x7f36bc0d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c103e640 1 --2- 172.21.15.27:0/1782927560 >> v2:172.21.15.69:3300/0 conn(0x7f36bc0b7420 0x7f36bc0d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 >> v1:172.21.15.69:6789/0 conn(0x7f36bc0ceea0 legacy=0x7f36bc0b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc0d39e0 con 0x7f36bc15dd90 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f36b8001690 con 0x7f36bc15dd90 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.546+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 >> v1:172.21.15.27:6789/0 conn(0x7f36bc15dd90 legacy=0x7f36bc0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c103e640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a80039b0 0x7f36a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c103e640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36a8004800 0x7f36a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c103e640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a80051c0 0x7f36a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0d3400 con 0x7f36a8004800 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a800c060 con 0x7f36a80051c0 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a800c260 con 0x7f36a80039b0 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c183f640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36a8004800 0x7f36a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c2040640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a80039b0 0x7f36a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c2841640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a80051c0 0x7f36a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c183f640 1 -- 172.21.15.27:0/1782927560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a80039b0 msgr2=0x7f36a8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c183f640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36a80039b0 0x7f36a8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c183f640 1 -- 172.21.15.27:0/1782927560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a80051c0 msgr2=0x7f36a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c183f640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a80051c0 0x7f36a800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c183f640 1 -- 172.21.15.27:0/1782927560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a8003520 con 0x7f36a8004800 2024-03-06T07:23:31.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.547+0000 7f36c2841640 1 -- 172.21.15.27:0/1782927560 reap_dead start 2024-03-06T07:23:31.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.548+0000 7f36c183f640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36a8004800 0x7f36a8004c80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f36ac001750 tx=0x7f36a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cd184dfebc4f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.548+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac001370 con 0x7f36a8004800 2024-03-06T07:23:31.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.548+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac0052d0 con 0x7f36a8004800 2024-03-06T07:23:31.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.548+0000 7f36c103e640 1 -- 172.21.15.27:0/1782927560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac003830 con 0x7f36a8004800 2024-03-06T07:23:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.548+0000 7f36c42a4640 1 -- 172.21.15.27:0/1782927560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36a8004800 msgr2=0x7f36a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.548+0000 7f36c42a4640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36a8004800 0x7f36a8004c80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f36ac001750 tx=0x7f36a8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.549+0000 7f36c42a4640 1 -- 172.21.15.27:0/1782927560 shutdown_connections 2024-03-06T07:23:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.549+0000 7f36c42a4640 1 --2- 172.21.15.27:0/1782927560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36a8004800 0x7f36a8004c80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.549+0000 7f36c42a4640 1 -- 172.21.15.27:0/1782927560 >> 172.21.15.27:0/1782927560 conn(0x7f36bc071960 msgr2=0x7f36bc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.549+0000 7f36c42a4640 1 -- 172.21.15.27:0/1782927560 shutdown_connections 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.549+0000 7f36c42a4640 1 -- 172.21.15.27:0/1782927560 wait complete. 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.550+0000 7f36c42a4640 1 Processor -- start 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.550+0000 7f36c42a4640 1 -- start start 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.550+0000 7f36c42a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a8004800 0x7f36bc14d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.550+0000 7f36c42a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc15e2f0 0x7f36bc0c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.550+0000 7f36c42a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36bc162190 0x7f36bc0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c42a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0d29d0 con 0x7f36bc15e2f0 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c42a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc15e170 con 0x7f36a8004800 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c42a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0d2b50 con 0x7f36bc162190 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c183f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc15e2f0 0x7f36bc0c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c183f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc15e2f0 0x7f36bc0c9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48688/0 (socket says 172.21.15.27:48688) 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c183f640 1 -- 172.21.15.27:0/289839819 learned_addr learned my addr 172.21.15.27:0/289839819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c2040640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a8004800 0x7f36bc14d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c2841640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36bc162190 0x7f36bc0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c183f640 1 -- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36bc162190 msgr2=0x7f36bc0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c183f640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36bc162190 0x7f36bc0c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c183f640 1 -- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a8004800 msgr2=0x7f36bc14d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c183f640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a8004800 0x7f36bc14d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c183f640 1 -- 172.21.15.27:0/289839819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc14d610 con 0x7f36bc15e2f0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c183f640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc15e2f0 0x7f36bc0c9980 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f36ac005040 tx=0x7f36ac001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36b2ffd640 1 -- 172.21.15.27:0/289839819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac041020 con 0x7f36bc15e2f0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36b2ffd640 1 -- 172.21.15.27:0/289839819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac005480 con 0x7f36bc15e2f0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36bc1f3c80 con 0x7f36bc15e2f0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36b2ffd640 1 -- 172.21.15.27:0/289839819 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac007a30 con 0x7f36bc15e2f0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.551+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc15ea60 con 0x7f36bc15e2f0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.552+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36bc072a30 con 0x7f36bc15e2f0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.553+0000 7f36b2ffd640 1 -- 172.21.15.27:0/289839819 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36ac044020 con 0x7f36bc15e2f0 2024-03-06T07:23:31.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.554+0000 7f36b2ffd640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a006b500 0x7f36a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.554+0000 7f36b2ffd640 1 -- 172.21.15.27:0/289839819 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f36ac0abbe0 con 0x7f36bc15e2f0 2024-03-06T07:23:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.554+0000 7f36c2040640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a006b500 0x7f36a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.555+0000 7f36c2040640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a006b500 0x7f36a006d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f36bc0b36d0 tx=0x7f36b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.558+0000 7f36b2ffd640 1 -- 172.21.15.27:0/289839819 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36ac04b030 con 0x7f36bc15e2f0 2024-03-06T07:23:31.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.810+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f36bc0b6f20 con 0x7f36bc15e2f0 2024-03-06T07:23:31.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.811+0000 7f36b2ffd640 1 -- 172.21.15.27:0/289839819 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f36bc14d610 con 0x7f36bc15e2f0 2024-03-06T07:23:31.813 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a006b500 msgr2=0x7f36a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a006b500 0x7f36a006d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f36bc0b36d0 tx=0x7f36b8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc15e2f0 msgr2=0x7f36bc0c9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc15e2f0 0x7f36bc0c9980 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f36ac005040 tx=0x7f36ac001120 comp rx=0 tx=0).stop 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 shutdown_connections 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a006b500 0x7f36a006d9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36bc162190 0x7f36bc0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36bc15e2f0 0x7f36bc0c9980 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 --2- 172.21.15.27:0/289839819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36a8004800 0x7f36bc14d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 >> 172.21.15.27:0/289839819 conn(0x7f36bc071960 msgr2=0x7f36bc071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 shutdown_connections 2024-03-06T07:23:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.813+0000 7f36c42a4640 1 -- 172.21.15.27:0/289839819 wait complete. 2024-03-06T07:23:31.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:31.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.933+0000 7f9337e76640 1 Processor -- start 2024-03-06T07:23:31.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.933+0000 7f9337e76640 1 -- start start 2024-03-06T07:23:31.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9337e76640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f933015b130 0x7f933015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9337e76640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f93300ce900 0x7f93300ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9337e76640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933015bc40 con 0x7f9330144640 2024-03-06T07:23:31.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9337e76640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933006df70 con 0x7f93300ce900 2024-03-06T07:23:31.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9337e76640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93300cf270 con 0x7f933015b130 2024-03-06T07:23:31.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9335c12640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9330144640 0x7f933006e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41530/0 (socket says 172.21.15.27:41530) 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9335c12640 1 -- 172.21.15.27:0/4186349566 learned_addr learned my addr 172.21.15.27:0/4186349566 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9336413640 1 --2- 172.21.15.27:0/4186349566 >> v2:172.21.15.27:3300/0 conn(0x7f93300ce900 0x7f93300ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9335411640 1 --2- 172.21.15.27:0/4186349566 >> v2:172.21.15.69:3300/0 conn(0x7f933015b130 0x7f933015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9334c10640 1 -- 172.21.15.27:0/4186349566 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306543072 0 0) 0x7f933015bc40 con 0x7f9330144640 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9334c10640 1 -- 172.21.15.27:0/4186349566 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9318003520 con 0x7f9330144640 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9336413640 1 -- 172.21.15.27:0/4186349566 >> v2:172.21.15.69:3300/0 conn(0x7f933015b130 msgr2=0x7f933015b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9336413640 1 --2- 172.21.15.27:0/4186349566 >> v2:172.21.15.69:3300/0 conn(0x7f933015b130 0x7f933015b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.934+0000 7f9336413640 1 -- 172.21.15.27:0/4186349566 >> v1:172.21.15.27:6789/0 conn(0x7f9330144640 legacy=0x7f933006e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.935+0000 7f9336413640 1 -- 172.21.15.27:0/4186349566 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933015bf20 con 0x7f93300ce900 2024-03-06T07:23:31.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.935+0000 7f9335411640 1 --2- 172.21.15.27:0/4186349566 >> v2:172.21.15.69:3300/0 conn(0x7f933015b130 0x7f933015b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.935+0000 7f9336413640 1 --2- 172.21.15.27:0/4186349566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 0x7f93300ced30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f93300706a0 tx=0x7f932802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0dd513d808f4a99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.935+0000 7f9334c10640 1 -- 172.21.15.27:0/4186349566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328035020 con 0x7f93300ce900 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.935+0000 7f9334c10640 1 -- 172.21.15.27:0/4186349566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9328031830 con 0x7f93300ce900 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.936+0000 7f9334c10640 1 -- 172.21.15.27:0/4186349566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328031be0 con 0x7f93300ce900 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.936+0000 7f9337e76640 1 -- 172.21.15.27:0/4186349566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 msgr2=0x7f93300ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.936+0000 7f9337e76640 1 --2- 172.21.15.27:0/4186349566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 0x7f93300ced30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f93300706a0 tx=0x7f932802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.936+0000 7f9337e76640 1 -- 172.21.15.27:0/4186349566 shutdown_connections 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.936+0000 7f9337e76640 1 --2- 172.21.15.27:0/4186349566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 0x7f93300ced30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.936+0000 7f9337e76640 1 --2- 172.21.15.27:0/4186349566 >> v2:172.21.15.69:3300/0 conn(0x7f933015b130 0x7f933015b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.936+0000 7f9337e76640 1 -- 172.21.15.27:0/4186349566 >> 172.21.15.27:0/4186349566 conn(0x7f9330066d00 msgr2=0x7f9330067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.936+0000 7f9337e76640 1 -- 172.21.15.27:0/4186349566 shutdown_connections 2024-03-06T07:23:31.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.936+0000 7f9337e76640 1 -- 172.21.15.27:0/4186349566 wait complete. 2024-03-06T07:23:31.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.937+0000 7f9337e76640 1 Processor -- start 2024-03-06T07:23:31.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.937+0000 7f9337e76640 1 -- start start 2024-03-06T07:23:31.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9337e76640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 0x7f933015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9337e76640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9330144640 0x7f93300cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9337e76640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f933015b130 0x7f93300cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9337e76640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9330070aa0 con 0x7f93300ce900 2024-03-06T07:23:31.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9337e76640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933006eb10 con 0x7f933015b130 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9337e76640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9330070c20 con 0x7f9330144640 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335c12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 0x7f933015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335c12640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 0x7f933015a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48716/0 (socket says 172.21.15.27:48716) 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335c12640 1 -- 172.21.15.27:0/1322949216 learned_addr learned my addr 172.21.15.27:0/1322949216 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335411640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9330144640 0x7f93300cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9336413640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f933015b130 0x7f93300cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335c12640 1 -- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9330144640 msgr2=0x7f93300cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335c12640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9330144640 0x7f93300cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335c12640 1 -- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f933015b130 msgr2=0x7f93300cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335c12640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f933015b130 0x7f93300cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335c12640 1 -- 172.21.15.27:0/1322949216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93300c6020 con 0x7f93300ce900 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.938+0000 7f9335411640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9330144640 0x7f93300cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:31.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.939+0000 7f9335c12640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 0x7f933015a2f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f932000cb40 tx=0x7f932000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.939+0000 7f932e7fc640 1 -- 172.21.15.27:0/1322949216 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320017070 con 0x7f93300ce900 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.939+0000 7f932e7fc640 1 -- 172.21.15.27:0/1322949216 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932000aca0 con 0x7f93300ce900 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.939+0000 7f932e7fc640 1 -- 172.21.15.27:0/1322949216 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320012620 con 0x7f93300ce900 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.939+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93300b0960 con 0x7f93300ce900 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.939+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93300b0f20 con 0x7f93300ce900 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.940+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93300733c0 con 0x7f93300ce900 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.940+0000 7f932e7fc640 1 -- 172.21.15.27:0/1322949216 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f932001a070 con 0x7f93300ce900 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.941+0000 7f932e7fc640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f931c06b4d0 0x7f931c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.941+0000 7f932e7fc640 1 -- 172.21.15.27:0/1322949216 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f932008a0e0 con 0x7f93300ce900 2024-03-06T07:23:31.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.941+0000 7f9335411640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f931c06b4d0 0x7f931c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.942+0000 7f9335411640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f931c06b4d0 0x7f931c06d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f93300b1600 tx=0x7f9324008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:31.945+0000 7f932e7fc640 1 -- 172.21.15.27:0/1322949216 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9320058ee0 con 0x7f93300ce900 2024-03-06T07:23:32.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.126+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93300b1320 con 0x7f93300ce900 2024-03-06T07:23:32.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.126+0000 7f932e7fc640 1 -- 172.21.15.27:0/1322949216 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93200587e0 con 0x7f93300ce900 2024-03-06T07:23:32.129 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f931c06b4d0 msgr2=0x7f931c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f931c06b4d0 0x7f931c06d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f93300b1600 tx=0x7f9324008040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 msgr2=0x7f933015a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 0x7f933015a2f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f932000cb40 tx=0x7f932000a060 comp rx=0 tx=0).stop 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 shutdown_connections 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f931c06b4d0 0x7f931c06d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f933015b130 0x7f93300cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9330144640 0x7f93300cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 --2- 172.21.15.27:0/1322949216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93300ce900 0x7f933015a2f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 >> 172.21.15.27:0/1322949216 conn(0x7f9330066d00 msgr2=0x7f93301418d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 shutdown_connections 2024-03-06T07:23:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.129+0000 7f9337e76640 1 -- 172.21.15.27:0/1322949216 wait complete. 2024-03-06T07:23:32.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:32.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.248+0000 7fd20cfe9640 1 Processor -- start 2024-03-06T07:23:32.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.248+0000 7fd20cfe9640 1 -- start start 2024-03-06T07:23:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.249+0000 7fd20cfe9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd208071540 0x7fd2080b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.249+0000 7fd20cfe9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2080d3400 0x7fd208071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.249+0000 7fd20cfe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2080b2b50 0x7fd2080d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.249+0000 7fd20cfe9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2080d2f70 con 0x7fd208071540 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.249+0000 7fd20cfe9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2080d4be0 con 0x7fd2080d3400 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.249+0000 7fd20cfe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd208071e60 con 0x7fd2080b2b50 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.249+0000 7fd20659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd208071540 0x7fd2080b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.249+0000 7fd20659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd208071540 0x7fd2080b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48718/0 (socket says 172.21.15.27:48718) 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.249+0000 7fd20659c640 1 -- 172.21.15.27:0/1466962157 learned_addr learned my addr 172.21.15.27:0/1466962157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.250+0000 7fd205d9b640 1 --2- 172.21.15.27:0/1466962157 >> v2:172.21.15.69:3300/0 conn(0x7fd2080d3400 0x7fd208071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.250+0000 7fd206d9d640 1 --2- 172.21.15.27:0/1466962157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2080b2b50 0x7fd2080d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.250+0000 7fd20659c640 1 -- 172.21.15.27:0/1466962157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2080b2b50 msgr2=0x7fd2080d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.250+0000 7fd20659c640 1 --2- 172.21.15.27:0/1466962157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2080b2b50 0x7fd2080d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.250+0000 7fd20659c640 1 -- 172.21.15.27:0/1466962157 >> v2:172.21.15.69:3300/0 conn(0x7fd2080d3400 msgr2=0x7fd208071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.250+0000 7fd20659c640 1 --2- 172.21.15.27:0/1466962157 >> v2:172.21.15.69:3300/0 conn(0x7fd2080d3400 0x7fd208071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.250+0000 7fd20659c640 1 -- 172.21.15.27:0/1466962157 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2080ce610 con 0x7fd208071540 2024-03-06T07:23:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.251+0000 7fd20659c640 1 --2- 172.21.15.27:0/1466962157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd208071540 0x7fd2080b30a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd2080b6860 tx=0x7fd20002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5515c564bcaaeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.251+0000 7fd20559a640 1 -- 172.21.15.27:0/1466962157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd200035020 con 0x7fd208071540 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.251+0000 7fd20559a640 1 -- 172.21.15.27:0/1466962157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd200031870 con 0x7fd208071540 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.251+0000 7fd20559a640 1 -- 172.21.15.27:0/1466962157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd200031c40 con 0x7fd208071540 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.251+0000 7fd20cfe9640 1 -- 172.21.15.27:0/1466962157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd208071540 msgr2=0x7fd2080b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.251+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/1466962157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd208071540 0x7fd2080b30a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd2080b6860 tx=0x7fd20002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.252+0000 7fd20cfe9640 1 -- 172.21.15.27:0/1466962157 shutdown_connections 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.252+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/1466962157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2080b2b50 0x7fd2080d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.252+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/1466962157 >> v2:172.21.15.69:3300/0 conn(0x7fd2080d3400 0x7fd208071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.252+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/1466962157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd208071540 0x7fd2080b30a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.252+0000 7fd20cfe9640 1 -- 172.21.15.27:0/1466962157 >> 172.21.15.27:0/1466962157 conn(0x7fd2080d3ca0 msgr2=0x7fd2080d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.252+0000 7fd20cfe9640 1 -- 172.21.15.27:0/1466962157 shutdown_connections 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.252+0000 7fd20cfe9640 1 -- 172.21.15.27:0/1466962157 wait complete. 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.253+0000 7fd20cfe9640 1 Processor -- start 2024-03-06T07:23:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.253+0000 7fd20cfe9640 1 -- start start 2024-03-06T07:23:32.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd20cfe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd208071540 0x7fd2081e9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd20cfe9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2080b2b50 0x7fd2080b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd20cfe9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2080d3400 0x7fd2080b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd20cfe9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2080ce8f0 con 0x7fd2080b2b50 2024-03-06T07:23:32.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd20cfe9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2080d2c00 con 0x7fd2080d3400 2024-03-06T07:23:32.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd20cfe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20815e370 con 0x7fd208071540 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd205d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2080b2b50 0x7fd2080b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd205d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2080b2b50 0x7fd2080b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48722/0 (socket says 172.21.15.27:48722) 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd205d9b640 1 -- 172.21.15.27:0/3767274710 learned_addr learned my addr 172.21.15.27:0/3767274710 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd206d9d640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2080d3400 0x7fd2080b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd20659c640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd208071540 0x7fd2081e9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd205d9b640 1 -- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd208071540 msgr2=0x7fd2081e9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd205d9b640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd208071540 0x7fd2081e9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd205d9b640 1 -- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2080d3400 msgr2=0x7fd2080b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd205d9b640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2080d3400 0x7fd2080b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd205d9b640 1 -- 172.21.15.27:0/3767274710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2081495c0 con 0x7fd2080b2b50 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.254+0000 7fd205d9b640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2080b2b50 0x7fd2080b88b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000e840 tx=0x7fd1f000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.255+0000 7fd1feffd640 1 -- 172.21.15.27:0/3767274710 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0017070 con 0x7fd2080b2b50 2024-03-06T07:23:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.255+0000 7fd1feffd640 1 -- 172.21.15.27:0/3767274710 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f000c940 con 0x7fd2080b2b50 2024-03-06T07:23:32.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.255+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2081498a0 con 0x7fd2080b2b50 2024-03-06T07:23:32.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.255+0000 7fd1feffd640 1 -- 172.21.15.27:0/3767274710 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f000cd10 con 0x7fd2080b2b50 2024-03-06T07:23:32.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.255+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd208149e60 con 0x7fd2080b2b50 2024-03-06T07:23:32.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.256+0000 7fd1feffd640 1 -- 172.21.15.27:0/3767274710 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1f001a070 con 0x7fd2080b2b50 2024-03-06T07:23:32.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.256+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd208068700 con 0x7fd2080b2b50 2024-03-06T07:23:32.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.257+0000 7fd1feffd640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd1e806b500 0x7fd1e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.257+0000 7fd1feffd640 1 -- 172.21.15.27:0/3767274710 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd1f008a420 con 0x7fd2080b2b50 2024-03-06T07:23:32.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.257+0000 7fd20659c640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd1e806b500 0x7fd1e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.257+0000 7fd20659c640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd1e806b500 0x7fd1e806d9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd200036000 tx=0x7fd20003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.261+0000 7fd1feffd640 1 -- 172.21.15.27:0/3767274710 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1f00592b0 con 0x7fd2080b2b50 2024-03-06T07:23:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.455+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd20806b8e0 con 0x7fd2080b2b50 2024-03-06T07:23:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.456+0000 7fd1feffd640 1 -- 172.21.15.27:0/3767274710 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd1f0058bb0 con 0x7fd2080b2b50 2024-03-06T07:23:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:23:32.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd1e806b500 msgr2=0x7fd1e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd1e806b500 0x7fd1e806d9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd200036000 tx=0x7fd20003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2080b2b50 msgr2=0x7fd2080b88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2080b2b50 0x7fd2080b88b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000e840 tx=0x7fd1f000ed00 comp rx=0 tx=0).stop 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 shutdown_connections 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd1e806b500 0x7fd1e806d9c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2080d3400 0x7fd2080b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2080b2b50 0x7fd2080b88b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 --2- 172.21.15.27:0/3767274710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd208071540 0x7fd2081e9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 >> 172.21.15.27:0/3767274710 conn(0x7fd2080d3ca0 msgr2=0x7fd20815fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 shutdown_connections 2024-03-06T07:23:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.458+0000 7fd20cfe9640 1 -- 172.21.15.27:0/3767274710 wait complete. 2024-03-06T07:23:32.471 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:23:32.471 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:32.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:32.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.594+0000 7ffa998b5640 1 Processor -- start 2024-03-06T07:23:32.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.594+0000 7ffa998b5640 1 -- start start 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.595+0000 7ffa998b5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffa940d3400 0x7ffa940b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.595+0000 7ffa998b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa940b2b50 0x7ffa940b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.595+0000 7ffa998b5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa940ce610 con 0x7ffa94071540 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.595+0000 7ffa998b5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa940d2f70 con 0x7ffa940d3400 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.595+0000 7ffa998b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa940d4be0 con 0x7ffa940b2b50 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.595+0000 7ffa92ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffa94071540 0x7ffa94071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41532/0 (socket says 172.21.15.27:41532) 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.595+0000 7ffa92ffd640 1 -- 172.21.15.27:0/2756150596 learned_addr learned my addr 172.21.15.27:0/2756150596 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa927fc640 1 --2- 172.21.15.27:0/2756150596 >> v2:172.21.15.69:3300/0 conn(0x7ffa940d3400 0x7ffa940b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096833182 0 0) 0x7ffa940ce610 con 0x7ffa94071540 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa937fe640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa940b2b50 0x7ffa940b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa80003520 con 0x7ffa94071540 2024-03-06T07:23:32.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411053742 0 0) 0x7ffa80003520 con 0x7ffa94071540 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa940b2b50 msgr2=0x7ffa940b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa940b2b50 0x7ffa940b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 >> v2:172.21.15.69:3300/0 conn(0x7ffa940d3400 msgr2=0x7ffa940b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 --2- 172.21.15.27:0/2756150596 >> v2:172.21.15.69:3300/0 conn(0x7ffa940d3400 0x7ffa940b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa940d2920 con 0x7ffa94071540 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7ffa8c001670 con 0x7ffa94071540 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 >> v1:172.21.15.27:6789/0 conn(0x7ffa94071540 legacy=0x7ffa94071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa940b2b50 0x7ffa800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa80003f30 0x7ffa80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa80004d40 0x7ffa800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa940d2f70 con 0x7ffa80004d40 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa940d4be0 con 0x7ffa80003f30 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8000b9c0 con 0x7ffa940b2b50 2024-03-06T07:23:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa937fe640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa80004d40 0x7ffa800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa92ffd640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa940b2b50 0x7ffa800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.596+0000 7ffa927fc640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa80003f30 0x7ffa80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.597+0000 7ffa937fe640 1 -- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa940b2b50 msgr2=0x7ffa800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.597+0000 7ffa937fe640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa940b2b50 0x7ffa800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.597+0000 7ffa937fe640 1 -- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa80003f30 msgr2=0x7ffa80004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.597+0000 7ffa937fe640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa80003f30 0x7ffa80004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.597+0000 7ffa937fe640 1 -- 172.21.15.27:0/2756150596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa80003520 con 0x7ffa80004d40 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.597+0000 7ffa937fe640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa80004d40 0x7ffa800051c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffa84009fc0 tx=0x7ffa80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7de0a7ec591d619 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.597+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa84039070 con 0x7ffa80004d40 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.597+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8402cc00 con 0x7ffa80004d40 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.597+0000 7ffa91ffb640 1 -- 172.21.15.27:0/2756150596 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa84009c10 con 0x7ffa80004d40 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.598+0000 7ffa998b5640 1 -- 172.21.15.27:0/2756150596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa80004d40 msgr2=0x7ffa800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.598+0000 7ffa998b5640 1 --2- 172.21.15.27:0/2756150596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa80004d40 0x7ffa800051c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffa84009fc0 tx=0x7ffa80005830 comp rx=0 tx=0).stop 2024-03-06T07:23:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.598+0000 7ffa937fe640 1 -- 172.21.15.27:0/2756150596 reap_dead start 2024-03-06T07:23:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.598+0000 7ffa998b5640 1 -- 172.21.15.27:0/2756150596 shutdown_connections 2024-03-06T07:23:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.598+0000 7ffa998b5640 1 -- 172.21.15.27:0/2756150596 >> 172.21.15.27:0/2756150596 conn(0x7ffa940d3ca0 msgr2=0x7ffa940d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.598+0000 7ffa998b5640 1 -- 172.21.15.27:0/2756150596 shutdown_connections 2024-03-06T07:23:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.598+0000 7ffa998b5640 1 -- 172.21.15.27:0/2756150596 wait complete. 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.599+0000 7ffa998b5640 1 Processor -- start 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.599+0000 7ffa998b5640 1 -- start start 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa998b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa9414ea70 0x7ffa940b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa998b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa940ad100 0x7ffa940ad580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa998b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa9414ee50 0x7ffa9414f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa998b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9415e370 con 0x7ffa940ad100 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa998b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa940ce8f0 con 0x7ffa9414ee50 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa998b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9414f810 con 0x7ffa9414ea70 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa940ad100 0x7ffa940ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa940ad100 0x7ffa940ad580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48730/0 (socket says 172.21.15.27:48730) 2024-03-06T07:23:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa927fc640 1 -- 172.21.15.27:0/1711446356 learned_addr learned my addr 172.21.15.27:0/1711446356 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa92ffd640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa9414ea70 0x7ffa940b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.600+0000 7ffa937fe640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa9414ee50 0x7ffa9414f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa927fc640 1 -- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa9414ea70 msgr2=0x7ffa940b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa927fc640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa9414ea70 0x7ffa940b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa927fc640 1 -- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa9414ee50 msgr2=0x7ffa9414f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa927fc640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa9414ee50 0x7ffa9414f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa927fc640 1 -- 172.21.15.27:0/1711446356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa94152b20 con 0x7ffa940ad100 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa927fc640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa940ad100 0x7ffa940ad580 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c011980 tx=0x7ffa7c011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa8bfff640 1 -- 172.21.15.27:0/1711446356 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c00faf0 con 0x7ffa940ad100 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa8bfff640 1 -- 172.21.15.27:0/1711446356 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c00fc90 con 0x7ffa940ad100 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa940b4540 con 0x7ffa940ad100 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa8bfff640 1 -- 172.21.15.27:0/1711446356 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c014440 con 0x7ffa940ad100 2024-03-06T07:23:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.601+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa940b4b00 con 0x7ffa940ad100 2024-03-06T07:23:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.602+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa94068700 con 0x7ffa940ad100 2024-03-06T07:23:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.603+0000 7ffa8bfff640 1 -- 172.21.15.27:0/1711446356 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa7c012040 con 0x7ffa940ad100 2024-03-06T07:23:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.603+0000 7ffa8bfff640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffa7406b4d0 0x7ffa7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.603+0000 7ffa8bfff640 1 -- 172.21.15.27:0/1711446356 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ffa7c08a890 con 0x7ffa940ad100 2024-03-06T07:23:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.603+0000 7ffa92ffd640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffa7406b4d0 0x7ffa7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.604+0000 7ffa92ffd640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffa7406b4d0 0x7ffa7406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c001d30 tx=0x7ffa8c007970 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.608+0000 7ffa8bfff640 1 -- 172.21.15.27:0/1711446356 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa7c019030 con 0x7ffa940ad100 2024-03-06T07:23:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.860+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa9415fa70 con 0x7ffa940ad100 2024-03-06T07:23:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.861+0000 7ffa8bfff640 1 -- 172.21.15.27:0/1711446356 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16783 (secure 0 0 0) 0x7ffa7c059040 con 0x7ffa940ad100 2024-03-06T07:23:32.864 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:32.864 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:29.211570+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffa7406b4d0 msgr2=0x7ffa7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffa7406b4d0 0x7ffa7406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c001d30 tx=0x7ffa8c007970 comp rx=0 tx=0).stop 2024-03-06T07:23:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa940ad100 msgr2=0x7ffa940ad580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa940ad100 0x7ffa940ad580 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c011980 tx=0x7ffa7c011e40 comp rx=0 tx=0).stop 2024-03-06T07:23:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 shutdown_connections 2024-03-06T07:23:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffa7406b4d0 0x7ffa7406d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa9414ee50 0x7ffa9414f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa940ad100 0x7ffa940ad580 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 --2- 172.21.15.27:0/1711446356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffa9414ea70 0x7ffa940b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.863+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 >> 172.21.15.27:0/1711446356 conn(0x7ffa940d3ca0 msgr2=0x7ffa940cd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.864+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 shutdown_connections 2024-03-06T07:23:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.864+0000 7ffa998b5640 1 -- 172.21.15.27:0/1711446356 wait complete. 2024-03-06T07:23:32.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:32.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.984+0000 7fb81544e640 1 Processor -- start 2024-03-06T07:23:32.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.984+0000 7fb81544e640 1 -- start start 2024-03-06T07:23:32.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.985+0000 7fb81544e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8100717b0 0x7fb810071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.985+0000 7fb81544e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb8100b3060 0x7fb8100d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.985+0000 7fb81544e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81015e4b0 con 0x7fb8100b2b10 2024-03-06T07:23:32.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.985+0000 7fb81544e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8100d2e10 con 0x7fb8100717b0 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.985+0000 7fb81544e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8100b97e0 con 0x7fb8100b3060 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.985+0000 7fb80effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8100717b0 0x7fb810071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.985+0000 7fb80f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb8100b2b10 0x7fb8100b6e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41548/0 (socket says 172.21.15.27:41548) 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.985+0000 7fb80f7fe640 1 -- 172.21.15.27:0/112140866 learned_addr learned my addr 172.21.15.27:0/112140866 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.985+0000 7fb80e7fc640 1 --2- 172.21.15.27:0/112140866 >> v2:172.21.15.69:3300/0 conn(0x7fb8100b3060 0x7fb8100d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.986+0000 7fb80effd640 1 -- 172.21.15.27:0/112140866 >> v2:172.21.15.69:3300/0 conn(0x7fb8100b3060 msgr2=0x7fb8100d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.986+0000 7fb80effd640 1 --2- 172.21.15.27:0/112140866 >> v2:172.21.15.69:3300/0 conn(0x7fb8100b3060 0x7fb8100d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.986+0000 7fb80effd640 1 -- 172.21.15.27:0/112140866 >> v1:172.21.15.27:6789/0 conn(0x7fb8100b2b10 legacy=0x7fb8100b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.986+0000 7fb80effd640 1 -- 172.21.15.27:0/112140866 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100b9a80 con 0x7fb8100717b0 2024-03-06T07:23:32.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.986+0000 7fb80dffb640 1 -- 172.21.15.27:0/112140866 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031097637 0 0) 0x7fb800001490 con 0x7fb8100b2b10 2024-03-06T07:23:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.986+0000 7fb80effd640 1 --2- 172.21.15.27:0/112140866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 0x7fb810071b90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb81006ce10 tx=0x7fb80802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0bde358c243f1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.986+0000 7fb80dffb640 1 -- 172.21.15.27:0/112140866 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808035020 con 0x7fb8100717b0 2024-03-06T07:23:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb80dffb640 1 -- 172.21.15.27:0/112140866 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb808031830 con 0x7fb8100717b0 2024-03-06T07:23:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb80dffb640 1 -- 172.21.15.27:0/112140866 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808031c00 con 0x7fb8100717b0 2024-03-06T07:23:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb81544e640 1 -- 172.21.15.27:0/112140866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 msgr2=0x7fb810071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb81544e640 1 --2- 172.21.15.27:0/112140866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 0x7fb810071b90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb81006ce10 tx=0x7fb80802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb81544e640 1 -- 172.21.15.27:0/112140866 shutdown_connections 2024-03-06T07:23:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb81544e640 1 --2- 172.21.15.27:0/112140866 >> v2:172.21.15.69:3300/0 conn(0x7fb8100b3060 0x7fb8100d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb81544e640 1 --2- 172.21.15.27:0/112140866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 0x7fb810071b90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb81544e640 1 -- 172.21.15.27:0/112140866 >> 172.21.15.27:0/112140866 conn(0x7fb8100d3d20 msgr2=0x7fb8100d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb81544e640 1 -- 172.21.15.27:0/112140866 shutdown_connections 2024-03-06T07:23:32.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.987+0000 7fb81544e640 1 -- 172.21.15.27:0/112140866 wait complete. 2024-03-06T07:23:32.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.988+0000 7fb81544e640 1 Processor -- start 2024-03-06T07:23:32.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.988+0000 7fb81544e640 1 -- start start 2024-03-06T07:23:32.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb81544e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 0x7fb8101e33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb81544e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8100b2b10 0x7fb8100c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb81544e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8100b3060 0x7fb8100cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb81544e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100b7560 con 0x7fb8100717b0 2024-03-06T07:23:32.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb81544e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100d4bf0 con 0x7fb8100b2b10 2024-03-06T07:23:32.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb81544e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100b76e0 con 0x7fb8100b3060 2024-03-06T07:23:32.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb80effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 0x7fb8101e33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb80effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 0x7fb8101e33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48748/0 (socket says 172.21.15.27:48748) 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb80effd640 1 -- 172.21.15.27:0/2925497345 learned_addr learned my addr 172.21.15.27:0/2925497345 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb80effd640 1 -- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8100b3060 msgr2=0x7fb8100cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb80effd640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8100b3060 0x7fb8100cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb80effd640 1 -- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8100b2b10 msgr2=0x7fb8100c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.989+0000 7fb80e7fc640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8100b2b10 0x7fb8100c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.990+0000 7fb80effd640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8100b2b10 0x7fb8100c9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.990+0000 7fb80effd640 1 -- 172.21.15.27:0/2925497345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8101e38e0 con 0x7fb8100717b0 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.990+0000 7fb80effd640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 0x7fb8101e33a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb808009890 tx=0x7fb8080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.990+0000 7fb8077fe640 1 -- 172.21.15.27:0/2925497345 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80803b980 con 0x7fb8100717b0 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.990+0000 7fb8077fe640 1 -- 172.21.15.27:0/2925497345 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80803bb40 con 0x7fb8100717b0 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.990+0000 7fb8077fe640 1 -- 172.21.15.27:0/2925497345 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8080453f0 con 0x7fb8100717b0 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.990+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8100ca300 con 0x7fb8100717b0 2024-03-06T07:23:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.990+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb810147ad0 con 0x7fb8100717b0 2024-03-06T07:23:32.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.991+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81006ce40 con 0x7fb8100717b0 2024-03-06T07:23:32.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.992+0000 7fb8077fe640 1 -- 172.21.15.27:0/2925497345 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb808038020 con 0x7fb8100717b0 2024-03-06T07:23:32.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.992+0000 7fb8077fe640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb7f006b4d0 0x7fb7f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.992+0000 7fb8077fe640 1 -- 172.21.15.27:0/2925497345 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb8080abeb0 con 0x7fb8100717b0 2024-03-06T07:23:32.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.992+0000 7fb80e7fc640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb7f006b4d0 0x7fb7f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.993+0000 7fb80e7fc640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb7f006b4d0 0x7fb7f006d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009970 tx=0x7fb7f8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:32.996+0000 7fb8077fe640 1 -- 172.21.15.27:0/2925497345 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb808040070 con 0x7fb8100717b0 2024-03-06T07:23:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.177+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8100d43c0 con 0x7fb8100717b0 2024-03-06T07:23:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.178+0000 7fb8077fe640 1 -- 172.21.15.27:0/2925497345 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb80807acb0 con 0x7fb8100717b0 2024-03-06T07:23:33.180 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.179+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb7f006b4d0 msgr2=0x7fb7f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.179+0000 7fb81544e640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb7f006b4d0 0x7fb7f006d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009970 tx=0x7fb7f8008040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.179+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 msgr2=0x7fb8101e33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.179+0000 7fb81544e640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 0x7fb8101e33a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb808009890 tx=0x7fb8080046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.180+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 shutdown_connections 2024-03-06T07:23:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.180+0000 7fb81544e640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb7f006b4d0 0x7fb7f006d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.180+0000 7fb81544e640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8100b3060 0x7fb8100cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.180+0000 7fb81544e640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8100b2b10 0x7fb8100c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.180+0000 7fb81544e640 1 --2- 172.21.15.27:0/2925497345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100717b0 0x7fb8101e33a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.180+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 >> 172.21.15.27:0/2925497345 conn(0x7fb8100d3d20 msgr2=0x7fb81015d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.180+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 shutdown_connections 2024-03-06T07:23:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.180+0000 7fb81544e640 1 -- 172.21.15.27:0/2925497345 wait complete. 2024-03-06T07:23:33.198 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.305+0000 7f8c975f7640 1 Processor -- start 2024-03-06T07:23:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.306+0000 7f8c975f7640 1 -- start start 2024-03-06T07:23:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.306+0000 7f8c975f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 0x7f8c900b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.306+0000 7f8c975f7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c900b7820 0x7f8c900b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.306+0000 7f8c975f7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c900d4960 con 0x7f8c90071540 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.306+0000 7f8c975f7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c900d3230 con 0x7f8c900b7820 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.307+0000 7f8c975f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c900d36c0 con 0x7f8c900b2d30 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.307+0000 7f8c95b94640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c900b7820 0x7f8c900b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.307+0000 7f8c95b94640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c900b7820 0x7f8c900b7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48758/0 (socket says 172.21.15.27:48758) 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.307+0000 7f8c95b94640 1 -- 172.21.15.27:0/3485535174 learned_addr learned my addr 172.21.15.27:0/3485535174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.307+0000 7f8c94b92640 1 --2- 172.21.15.27:0/3485535174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 0x7f8c900b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.307+0000 7f8c95b94640 1 -- 172.21.15.27:0/3485535174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 msgr2=0x7f8c900b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.307+0000 7f8c95b94640 1 --2- 172.21.15.27:0/3485535174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 0x7f8c900b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.307+0000 7f8c95b94640 1 -- 172.21.15.27:0/3485535174 >> v1:172.21.15.69:6789/0 conn(0x7f8c90071540 legacy=0x7f8c90071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.307+0000 7f8c95b94640 1 -- 172.21.15.27:0/3485535174 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c900d2620 con 0x7f8c900b7820 2024-03-06T07:23:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c95b94640 1 --2- 172.21.15.27:0/3485535174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 0x7f8c900b7ca0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8c88009e50 tx=0x7f8c8802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1473df1f37ac3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c87fff640 1 -- 172.21.15.27:0/3485535174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8802cb00 con 0x7f8c900b7820 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c87fff640 1 -- 172.21.15.27:0/3485535174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8802cca0 con 0x7f8c900b7820 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c87fff640 1 -- 172.21.15.27:0/3485535174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c88034610 con 0x7f8c900b7820 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c975f7640 1 -- 172.21.15.27:0/3485535174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 msgr2=0x7f8c900b7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c975f7640 1 --2- 172.21.15.27:0/3485535174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 0x7f8c900b7ca0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8c88009e50 tx=0x7f8c8802bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c975f7640 1 -- 172.21.15.27:0/3485535174 shutdown_connections 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c975f7640 1 --2- 172.21.15.27:0/3485535174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 0x7f8c900b7ca0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c975f7640 1 --2- 172.21.15.27:0/3485535174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 0x7f8c900b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.308+0000 7f8c975f7640 1 -- 172.21.15.27:0/3485535174 >> 172.21.15.27:0/3485535174 conn(0x7f8c900d3d60 msgr2=0x7f8c900d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.309+0000 7f8c975f7640 1 -- 172.21.15.27:0/3485535174 shutdown_connections 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.309+0000 7f8c975f7640 1 -- 172.21.15.27:0/3485535174 wait complete. 2024-03-06T07:23:33.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.310+0000 7f8c975f7640 1 Processor -- start 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.310+0000 7f8c975f7640 1 -- start start 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.310+0000 7f8c975f7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c90071540 0x7f8c901f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.310+0000 7f8c975f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 0x7f8c9014e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.310+0000 7f8c975f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 0x7f8c900c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.310+0000 7f8c975f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c900d2ec0 con 0x7f8c900b7820 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.310+0000 7f8c975f7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c900d2be0 con 0x7f8c90071540 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.310+0000 7f8c975f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c900d22f0 con 0x7f8c900b2d30 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95b94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 0x7f8c900c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95b94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 0x7f8c900c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48770/0 (socket says 172.21.15.27:48770) 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95b94640 1 -- 172.21.15.27:0/273908586 learned_addr learned my addr 172.21.15.27:0/273908586 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c94b92640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 0x7f8c9014e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95393640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c90071540 0x7f8c901f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95b94640 1 -- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 msgr2=0x7f8c9014e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95b94640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 0x7f8c9014e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95b94640 1 -- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c90071540 msgr2=0x7f8c901f4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95b94640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c90071540 0x7f8c901f4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95b94640 1 -- 172.21.15.27:0/273908586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c901f5320 con 0x7f8c900b7820 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95393640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c90071540 0x7f8c901f4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c95b94640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 0x7f8c900c9060 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8c88036040 tx=0x7f8c88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c85ffb640 1 -- 172.21.15.27:0/273908586 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c88044020 con 0x7f8c900b7820 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c85ffb640 1 -- 172.21.15.27:0/273908586 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8802ce50 con 0x7f8c900b7820 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c901f5600 con 0x7f8c900b7820 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.311+0000 7f8c85ffb640 1 -- 172.21.15.27:0/273908586 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c88009510 con 0x7f8c900b7820 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.312+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c901f5bc0 con 0x7f8c900b7820 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.313+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c9015e0c0 con 0x7f8c900b7820 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.313+0000 7f8c85ffb640 1 -- 172.21.15.27:0/273908586 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c88047020 con 0x7f8c900b7820 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.313+0000 7f8c85ffb640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7406b4d0 0x7f8c7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.314+0000 7f8c85ffb640 1 -- 172.21.15.27:0/273908586 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8c880aca90 con 0x7f8c900b7820 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.314+0000 7f8c95393640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7406b4d0 0x7f8c7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.314+0000 7f8c95393640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7406b4d0 0x7f8c7406d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c001660 tx=0x7f8c8c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.323+0000 7f8c85ffb640 1 -- 172.21.15.27:0/273908586 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c9015e0c0 con 0x7f8c900b7820 2024-03-06T07:23:33.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.503+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c90072070 con 0x7f8c900b7820 2024-03-06T07:23:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.504+0000 7f8c85ffb640 1 -- 172.21.15.27:0/273908586 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c8804e030 con 0x7f8c900b7820 2024-03-06T07:23:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7406b4d0 msgr2=0x7f8c7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7406b4d0 0x7f8c7406d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c001660 tx=0x7f8c8c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 msgr2=0x7f8c900c9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 0x7f8c900c9060 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8c88036040 tx=0x7f8c88004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 shutdown_connections 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c7406b4d0 0x7f8c7406d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c900b7820 0x7f8c900c9060 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c900b2d30 0x7f8c9014e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 --2- 172.21.15.27:0/273908586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c90071540 0x7f8c901f4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 >> 172.21.15.27:0/273908586 conn(0x7f8c900d3d60 msgr2=0x7f8c9015d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 shutdown_connections 2024-03-06T07:23:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.506+0000 7f8c975f7640 1 -- 172.21.15.27:0/273908586 wait complete. 2024-03-06T07:23:33.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:33.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.626+0000 7f271774b640 1 Processor -- start 2024-03-06T07:23:33.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.627+0000 7f271774b640 1 -- start start 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.627+0000 7f271774b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2710071540 0x7f2710071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.627+0000 7f271774b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27100b2d30 0x7f27100ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.627+0000 7f271774b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27100cecb0 0x7f27100cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f271774b640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27100d3230 con 0x7f27100b2d30 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f271774b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2710071e60 con 0x7f2710071540 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f271774b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27100d4960 con 0x7f27100cecb0 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2714ce6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27100b2d30 0x7f27100ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2714ce6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27100b2d30 0x7f27100ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52382/0 (socket says 172.21.15.27:52382) 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2714ce6640 1 -- 172.21.15.27:0/279514665 learned_addr learned my addr 172.21.15.27:0/279514665 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f27154e7640 1 --2- 172.21.15.27:0/279514665 >> v2:172.21.15.69:3300/0 conn(0x7f2710071540 0x7f2710071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2715ce8640 1 --2- 172.21.15.27:0/279514665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27100cecb0 0x7f27100cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2714ce6640 1 -- 172.21.15.27:0/279514665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27100cecb0 msgr2=0x7f27100cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2714ce6640 1 --2- 172.21.15.27:0/279514665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27100cecb0 0x7f27100cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2714ce6640 1 -- 172.21.15.27:0/279514665 >> v2:172.21.15.69:3300/0 conn(0x7f2710071540 msgr2=0x7f2710071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2714ce6640 1 --2- 172.21.15.27:0/279514665 >> v2:172.21.15.69:3300/0 conn(0x7f2710071540 0x7f2710071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2714ce6640 1 -- 172.21.15.27:0/279514665 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27100d36c0 con 0x7f27100b2d30 2024-03-06T07:23:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.628+0000 7f2715ce8640 1 --2- 172.21.15.27:0/279514665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27100cecb0 0x7f27100cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.629+0000 7f2714ce6640 1 --2- 172.21.15.27:0/279514665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100b2d30 0x7f27100ce770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f271008ca00 tx=0x7f270002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ec959c30f2ade8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.629+0000 7f2707fff640 1 -- 172.21.15.27:0/279514665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2700035020 con 0x7f27100b2d30 2024-03-06T07:23:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f2707fff640 1 -- 172.21.15.27:0/279514665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2700031870 con 0x7f27100b2d30 2024-03-06T07:23:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f2707fff640 1 -- 172.21.15.27:0/279514665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2700031c40 con 0x7f27100b2d30 2024-03-06T07:23:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f271774b640 1 -- 172.21.15.27:0/279514665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100b2d30 msgr2=0x7f27100ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f271774b640 1 --2- 172.21.15.27:0/279514665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100b2d30 0x7f27100ce770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f271008ca00 tx=0x7f270002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f271774b640 1 -- 172.21.15.27:0/279514665 shutdown_connections 2024-03-06T07:23:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f271774b640 1 --2- 172.21.15.27:0/279514665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f27100cecb0 0x7f27100cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f271774b640 1 --2- 172.21.15.27:0/279514665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100b2d30 0x7f27100ce770 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f271774b640 1 --2- 172.21.15.27:0/279514665 >> v2:172.21.15.69:3300/0 conn(0x7f2710071540 0x7f2710071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f271774b640 1 -- 172.21.15.27:0/279514665 >> 172.21.15.27:0/279514665 conn(0x7f27100d3d60 msgr2=0x7f27100d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f271774b640 1 -- 172.21.15.27:0/279514665 shutdown_connections 2024-03-06T07:23:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.630+0000 7f271774b640 1 -- 172.21.15.27:0/279514665 wait complete. 2024-03-06T07:23:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.631+0000 7f271774b640 1 Processor -- start 2024-03-06T07:23:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.631+0000 7f271774b640 1 -- start start 2024-03-06T07:23:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f271774b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2710071540 0x7f27101f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f271774b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27100b2d30 0x7f2710156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f271774b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100cecb0 0x7f27100aa840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f271774b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27100d2be0 con 0x7f27100cecb0 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f271774b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27100d2900 con 0x7f27100b2d30 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f271774b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27100d2ec0 con 0x7f2710071540 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f2715ce8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100cecb0 0x7f27100aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f2715ce8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100cecb0 0x7f27100aa840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52398/0 (socket says 172.21.15.27:52398) 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f2715ce8640 1 -- 172.21.15.27:0/4186537750 learned_addr learned my addr 172.21.15.27:0/4186537750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f2714ce6640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27100b2d30 0x7f2710156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.632+0000 7f27154e7640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2710071540 0x7f27101f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f2715ce8640 1 -- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2710071540 msgr2=0x7f27101f5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f2715ce8640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2710071540 0x7f27101f5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f2715ce8640 1 -- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27100b2d30 msgr2=0x7f2710156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f2715ce8640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27100b2d30 0x7f2710156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f2715ce8640 1 -- 172.21.15.27:0/4186537750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27101f5860 con 0x7f27100cecb0 2024-03-06T07:23:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f27154e7640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2710071540 0x7f27101f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f2715ce8640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100cecb0 0x7f27100aa840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2708007f20 tx=0x7f270800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f2705ffb640 1 -- 172.21.15.27:0/4186537750 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270801a070 con 0x7f27100cecb0 2024-03-06T07:23:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f2705ffb640 1 -- 172.21.15.27:0/4186537750 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2708003b90 con 0x7f27100cecb0 2024-03-06T07:23:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f2705ffb640 1 -- 172.21.15.27:0/4186537750 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2708004030 con 0x7f27100cecb0 2024-03-06T07:23:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27100c8570 con 0x7f27100cecb0 2024-03-06T07:23:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.633+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27100c8b30 con 0x7f27100cecb0 2024-03-06T07:23:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.634+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2710072070 con 0x7f27100cecb0 2024-03-06T07:23:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.635+0000 7f2705ffb640 1 -- 172.21.15.27:0/4186537750 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f270801d070 con 0x7f27100cecb0 2024-03-06T07:23:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.635+0000 7f2705ffb640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26f406b4d0 0x7f26f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.635+0000 7f2705ffb640 1 -- 172.21.15.27:0/4186537750 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f270808a450 con 0x7f27100cecb0 2024-03-06T07:23:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.635+0000 7f27154e7640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26f406b4d0 0x7f26f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.636+0000 7f27154e7640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26f406b4d0 0x7f26f406d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f27101f7df0 tx=0x7f270c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.640+0000 7f2705ffb640 1 -- 172.21.15.27:0/4186537750 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27101f5860 con 0x7f27100cecb0 2024-03-06T07:23:33.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.819+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f271004f410 con 0x7f27100cecb0 2024-03-06T07:23:33.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.819+0000 7f2705ffb640 1 -- 172.21.15.27:0/4186537750 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2708010070 con 0x7f27100cecb0 2024-03-06T07:23:33.822 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26f406b4d0 msgr2=0x7f26f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26f406b4d0 0x7f26f406d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f27101f7df0 tx=0x7f270c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100cecb0 msgr2=0x7f27100aa840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100cecb0 0x7f27100aa840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2708007f20 tx=0x7f270800d560 comp rx=0 tx=0).stop 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 shutdown_connections 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26f406b4d0 0x7f26f406d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27100cecb0 0x7f27100aa840 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27100b2d30 0x7f2710156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 --2- 172.21.15.27:0/4186537750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2710071540 0x7f27101f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.821+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 >> 172.21.15.27:0/4186537750 conn(0x7f27100d3d60 msgr2=0x7f271015d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.822+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 shutdown_connections 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.822+0000 7f271774b640 1 -- 172.21.15.27:0/4186537750 wait complete. 2024-03-06T07:23:33.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:33.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.953+0000 7fdbe89ac640 1 Processor -- start 2024-03-06T07:23:33.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.953+0000 7fdbe89ac640 1 -- start start 2024-03-06T07:23:33.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe89ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbe415df00 0x7fdbe40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe89ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 0x7fdbe40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe89ac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe40cea30 con 0x7fdbe40b9a30 2024-03-06T07:23:33.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe89ac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe40d2af0 con 0x7fdbe415df00 2024-03-06T07:23:33.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe89ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe40cf3c0 con 0x7fdbe40b95a0 2024-03-06T07:23:33.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbe415df00 0x7fdbe40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe2d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdbe40b9a30 0x7fdbe40d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58492/0 (socket says 172.21.15.27:58492) 2024-03-06T07:23:33.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe2d9d640 1 -- 172.21.15.27:0/3560742568 learned_addr learned my addr 172.21.15.27:0/3560742568 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:33.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe1d9b640 1 --2- 172.21.15.27:0/3560742568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 0x7fdbe40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe259c640 1 -- 172.21.15.27:0/3560742568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 msgr2=0x7fdbe40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe259c640 1 --2- 172.21.15.27:0/3560742568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 0x7fdbe40b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe259c640 1 -- 172.21.15.27:0/3560742568 >> v1:172.21.15.27:6789/0 conn(0x7fdbe40b9a30 legacy=0x7fdbe40d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe259c640 1 -- 172.21.15.27:0/3560742568 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe40cf0e0 con 0x7fdbe415df00 2024-03-06T07:23:33.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.955+0000 7fdbe1d9b640 1 --2- 172.21.15.27:0/3560742568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 0x7fdbe40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.956+0000 7fdbe259c640 1 --2- 172.21.15.27:0/3560742568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe415df00 0x7fdbe40d25b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40097a0 tx=0x7fdbd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ef5b1768cf6d8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.957+0000 7fdbe159a640 1 -- 172.21.15.27:0/3560742568 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4035020 con 0x7fdbe415df00 2024-03-06T07:23:33.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.957+0000 7fdbe159a640 1 -- 172.21.15.27:0/3560742568 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd4031830 con 0x7fdbe415df00 2024-03-06T07:23:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.957+0000 7fdbe159a640 1 -- 172.21.15.27:0/3560742568 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4031c00 con 0x7fdbe415df00 2024-03-06T07:23:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.957+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3560742568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe415df00 msgr2=0x7fdbe40d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.957+0000 7fdbe89ac640 1 --2- 172.21.15.27:0/3560742568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe415df00 0x7fdbe40d25b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40097a0 tx=0x7fdbd402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.958+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3560742568 shutdown_connections 2024-03-06T07:23:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.958+0000 7fdbe89ac640 1 --2- 172.21.15.27:0/3560742568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 0x7fdbe40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.958+0000 7fdbe89ac640 1 --2- 172.21.15.27:0/3560742568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe415df00 0x7fdbe40d25b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.958+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3560742568 >> 172.21.15.27:0/3560742568 conn(0x7fdbe4071960 msgr2=0x7fdbe4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.958+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3560742568 shutdown_connections 2024-03-06T07:23:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.958+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3560742568 wait complete. 2024-03-06T07:23:33.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.960+0000 7fdbe89ac640 1 Processor -- start 2024-03-06T07:23:33.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.960+0000 7fdbe89ac640 1 -- start start 2024-03-06T07:23:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.961+0000 7fdbe89ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 0x7fdbe415b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.961+0000 7fdbe89ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbe415df00 0x7fdbe40c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.961+0000 7fdbe89ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe41f38c0 0x7fdbe40b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.961+0000 7fdbe89ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe40cec70 con 0x7fdbe41f38c0 2024-03-06T07:23:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.961+0000 7fdbe89ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe40ce750 con 0x7fdbe415df00 2024-03-06T07:23:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.961+0000 7fdbe89ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe40cedf0 con 0x7fdbe40b95a0 2024-03-06T07:23:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.961+0000 7fdbe2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe41f38c0 0x7fdbe40b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.961+0000 7fdbe2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe41f38c0 0x7fdbe40b6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52428/0 (socket says 172.21.15.27:52428) 2024-03-06T07:23:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.961+0000 7fdbe2d9d640 1 -- 172.21.15.27:0/3846479920 learned_addr learned my addr 172.21.15.27:0/3846479920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe1d9b640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbe415df00 0x7fdbe40c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe259c640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 0x7fdbe415b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe2d9d640 1 -- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 msgr2=0x7fdbe415b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe2d9d640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 0x7fdbe415b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe2d9d640 1 -- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbe415df00 msgr2=0x7fdbe40c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe2d9d640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbe415df00 0x7fdbe40c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe2d9d640 1 -- 172.21.15.27:0/3846479920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe41f4110 con 0x7fdbe41f38c0 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe1d9b640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbe415df00 0x7fdbe40c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe2d9d640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe41f38c0 0x7fdbe40b6e40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00caf0 tx=0x7fdbdc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbdaffd640 1 -- 172.21.15.27:0/3846479920 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc017070 con 0x7fdbe41f38c0 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbdaffd640 1 -- 172.21.15.27:0/3846479920 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbdc00ac40 con 0x7fdbe41f38c0 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe41f43f0 con 0x7fdbe41f38c0 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbdaffd640 1 -- 172.21.15.27:0/3846479920 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc012600 con 0x7fdbe41f38c0 2024-03-06T07:23:33.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.962+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe40b4550 con 0x7fdbe41f38c0 2024-03-06T07:23:33.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.963+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe40b4340 con 0x7fdbe41f38c0 2024-03-06T07:23:33.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.964+0000 7fdbdaffd640 1 -- 172.21.15.27:0/3846479920 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbdc01a070 con 0x7fdbe41f38c0 2024-03-06T07:23:33.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.964+0000 7fdbdaffd640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbc406b500 0x7fdbc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.964+0000 7fdbdaffd640 1 -- 172.21.15.27:0/3846479920 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdbdc089e80 con 0x7fdbe41f38c0 2024-03-06T07:23:33.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.965+0000 7fdbe259c640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbc406b500 0x7fdbc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.965+0000 7fdbe259c640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbc406b500 0x7fdbc406d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4036000 tx=0x7fdbd4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:33.969+0000 7fdbdaffd640 1 -- 172.21.15.27:0/3846479920 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbdc058630 con 0x7fdbe41f38c0 2024-03-06T07:23:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.164+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdbe4067ff0 con 0x7fdbe41f38c0 2024-03-06T07:23:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.164+0000 7fdbdaffd640 1 -- 172.21.15.27:0/3846479920 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdbdc05c6d0 con 0x7fdbe41f38c0 2024-03-06T07:23:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:23:34.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbc406b500 msgr2=0x7fdbc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbc406b500 0x7fdbc406d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4036000 tx=0x7fdbd4008030 comp rx=0 tx=0).stop 2024-03-06T07:23:34.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe41f38c0 msgr2=0x7fdbe40b6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe41f38c0 0x7fdbe40b6e40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00caf0 tx=0x7fdbdc00a060 comp rx=0 tx=0).stop 2024-03-06T07:23:34.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 shutdown_connections 2024-03-06T07:23:34.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbc406b500 0x7fdbc406d9c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbe41f38c0 0x7fdbe40b6e40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbe415df00 0x7fdbe40c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 --2- 172.21.15.27:0/3846479920 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdbe40b95a0 0x7fdbe415b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 >> 172.21.15.27:0/3846479920 conn(0x7fdbe4071960 msgr2=0x7fdbe40cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 shutdown_connections 2024-03-06T07:23:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.166+0000 7fdbe89ac640 1 -- 172.21.15.27:0/3846479920 wait complete. 2024-03-06T07:23:34.180 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-03-06T07:23:34.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:34.180 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-06T07:23:34.213 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:34.214 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:34.214 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:23:34.270 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:23:34.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:34.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.376+0000 7fcded781640 1 Processor -- start 2024-03-06T07:23:34.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.376+0000 7fcded781640 1 -- start start 2024-03-06T07:23:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.377+0000 7fcded781640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcde815ac30 0x7fcde80b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.377+0000 7fcded781640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80b3e80 con 0x7fcde80afb30 2024-03-06T07:23:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.377+0000 7fcded781640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80b4120 con 0x7fcde80720c0 2024-03-06T07:23:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.377+0000 7fcded781640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde80b4400 con 0x7fcde815ac30 2024-03-06T07:23:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.377+0000 7fcde6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcde815ac30 0x7fcde80b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.377+0000 7fcde67fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcde80afb30 0x7fcde80719b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58504/0 (socket says 172.21.15.27:58504) 2024-03-06T07:23:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.377+0000 7fcde67fc640 1 -- 172.21.15.27:0/3279039841 learned_addr learned my addr 172.21.15.27:0/3279039841 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.378+0000 7fcde6ffd640 1 -- 172.21.15.27:0/3279039841 >> v1:172.21.15.69:6789/0 conn(0x7fcde80720c0 legacy=0x7fcde80cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.378+0000 7fcde6ffd640 1 -- 172.21.15.27:0/3279039841 >> v1:172.21.15.27:6789/0 conn(0x7fcde80afb30 legacy=0x7fcde80719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.378+0000 7fcde6ffd640 1 -- 172.21.15.27:0/3279039841 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80b46e0 con 0x7fcde815ac30 2024-03-06T07:23:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.378+0000 7fcde6ffd640 1 --2- 172.21.15.27:0/3279039841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde815ac30 0x7fcde80b3940 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcde00097a0 tx=0x7fcde002db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874546228cec43c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcde5ffb640 1 -- 172.21.15.27:0/3279039841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde003b070 con 0x7fcde815ac30 2024-03-06T07:23:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcde5ffb640 1 -- 172.21.15.27:0/3279039841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde002e920 con 0x7fcde815ac30 2024-03-06T07:23:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcde5ffb640 1 -- 172.21.15.27:0/3279039841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde002ecf0 con 0x7fcde815ac30 2024-03-06T07:23:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcded781640 1 -- 172.21.15.27:0/3279039841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde815ac30 msgr2=0x7fcde80b3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcded781640 1 --2- 172.21.15.27:0/3279039841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde815ac30 0x7fcde80b3940 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcde00097a0 tx=0x7fcde002db70 comp rx=0 tx=0).stop 2024-03-06T07:23:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcded781640 1 -- 172.21.15.27:0/3279039841 shutdown_connections 2024-03-06T07:23:34.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcded781640 1 --2- 172.21.15.27:0/3279039841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde815ac30 0x7fcde80b3940 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcded781640 1 -- 172.21.15.27:0/3279039841 >> 172.21.15.27:0/3279039841 conn(0x7fcde80d0990 msgr2=0x7fcde80d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcded781640 1 -- 172.21.15.27:0/3279039841 shutdown_connections 2024-03-06T07:23:34.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.379+0000 7fcded781640 1 -- 172.21.15.27:0/3279039841 wait complete. 2024-03-06T07:23:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcded781640 1 Processor -- start 2024-03-06T07:23:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcded781640 1 -- start start 2024-03-06T07:23:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcded781640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde80720c0 0x7fcde81e3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcded781640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcde80afb30 0x7fcde81423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcded781640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde815ac30 0x7fcde815b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcded781640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80ce6a0 con 0x7fcde80720c0 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcded781640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80b4ca0 con 0x7fcde815ac30 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcded781640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80ce820 con 0x7fcde80afb30 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcde6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde80720c0 0x7fcde81e3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcde6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde80720c0 0x7fcde81e3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52454/0 (socket says 172.21.15.27:52454) 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.381+0000 7fcde6ffd640 1 -- 172.21.15.27:0/2657504536 learned_addr learned my addr 172.21.15.27:0/2657504536 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcde77fe640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde815ac30 0x7fcde815b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcde67fc640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcde80afb30 0x7fcde81423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcde6ffd640 1 -- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcde80afb30 msgr2=0x7fcde81423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcde6ffd640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcde80afb30 0x7fcde81423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcde6ffd640 1 -- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde815ac30 msgr2=0x7fcde815b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcde6ffd640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde815ac30 0x7fcde815b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcde6ffd640 1 -- 172.21.15.27:0/2657504536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde8142910 con 0x7fcde80720c0 2024-03-06T07:23:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcde77fe640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde815ac30 0x7fcde815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcde6ffd640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde80720c0 0x7fcde81e3280 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcde0039040 tx=0x7fcde00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcdcf7fe640 1 -- 172.21.15.27:0/2657504536 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde003b040 con 0x7fcde80720c0 2024-03-06T07:23:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcdcf7fe640 1 -- 172.21.15.27:0/2657504536 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde002e720 con 0x7fcde80720c0 2024-03-06T07:23:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcdcf7fe640 1 -- 172.21.15.27:0/2657504536 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0036d90 con 0x7fcde80720c0 2024-03-06T07:23:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde81e3860 con 0x7fcde80720c0 2024-03-06T07:23:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.382+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde81e3e20 con 0x7fcde80720c0 2024-03-06T07:23:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.383+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde8068700 con 0x7fcde80720c0 2024-03-06T07:23:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.384+0000 7fcdcf7fe640 1 -- 172.21.15.27:0/2657504536 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcde0048020 con 0x7fcde80720c0 2024-03-06T07:23:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.385+0000 7fcdcf7fe640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdd406b4d0 0x7fcdd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.385+0000 7fcdcf7fe640 1 -- 172.21.15.27:0/2657504536 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcde00add00 con 0x7fcde80720c0 2024-03-06T07:23:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.385+0000 7fcde67fc640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdd406b4d0 0x7fcdd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.386+0000 7fcde67fc640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdd406b4d0 0x7fcdd406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcde815e0f0 tx=0x7fcddc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.389+0000 7fcdcf7fe640 1 -- 172.21.15.27:0/2657504536 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcde8142910 con 0x7fcde80720c0 2024-03-06T07:23:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.757+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcde8071dc0 con 0x7fcde80720c0 2024-03-06T07:23:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.758+0000 7fcdcf7fe640 1 -- 172.21.15.27:0/2657504536 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7fcde0034340 con 0x7fcde80720c0 2024-03-06T07:23:34.760 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:34.760 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1135544337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1135544337},{"type":"v1","addr":"172.21.15.27:6833","nonce":1135544337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1920994895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1920994895},{"type":"v1","addr":"172.21.15.69:6837","nonce":1920994895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:48.433333+0000","modified":"2024-03-06T07:22:51.436125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/3933261233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3933261233},{"type":"v1","addr":"172.21.15.27:6835","nonce":3933261233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-06T07:23:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-03-06T07:23:34.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdd406b4d0 msgr2=0x7fcdd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdd406b4d0 0x7fcdd406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcde815e0f0 tx=0x7fcddc009040 comp rx=0 tx=0).stop 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde80720c0 msgr2=0x7fcde81e3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde80720c0 0x7fcde81e3280 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcde0039040 tx=0x7fcde00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 shutdown_connections 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdd406b4d0 0x7fcdd406d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde815ac30 0x7fcde815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcde80afb30 0x7fcde81423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 --2- 172.21.15.27:0/2657504536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde80720c0 0x7fcde81e3280 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 >> 172.21.15.27:0/2657504536 conn(0x7fcde80d0990 msgr2=0x7fcde806ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 shutdown_connections 2024-03-06T07:23:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.760+0000 7fcded781640 1 -- 172.21.15.27:0/2657504536 wait complete. 2024-03-06T07:23:34.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:34.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.879+0000 7f9e050eb640 1 Processor -- start 2024-03-06T07:23:34.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.879+0000 7f9e050eb640 1 -- start start 2024-03-06T07:23:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9e050eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9e0015df00 0x7f9e000d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9e050eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e000b95a0 0x7f9e000b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9e050eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e000b9a30 0x7f9e000b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9e050eb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e000cea30 con 0x7f9e000b95a0 2024-03-06T07:23:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9e050eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e000d2af0 con 0x7f9e0015df00 2024-03-06T07:23:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9e050eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e000cf3c0 con 0x7f9e000b9a30 2024-03-06T07:23:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfe59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e000b95a0 0x7f9e000b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfe59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e000b95a0 0x7f9e000b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52458/0 (socket says 172.21.15.27:52458) 2024-03-06T07:23:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfe59c640 1 -- 172.21.15.27:0/2435394058 learned_addr learned my addr 172.21.15.27:0/2435394058 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dff59e640 1 --2- 172.21.15.27:0/2435394058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e000b9a30 0x7f9e000b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfed9d640 1 --2- 172.21.15.27:0/2435394058 >> v2:172.21.15.69:3300/0 conn(0x7f9e0015df00 0x7f9e000d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfe59c640 1 -- 172.21.15.27:0/2435394058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e000b9a30 msgr2=0x7f9e000b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfe59c640 1 --2- 172.21.15.27:0/2435394058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e000b9a30 0x7f9e000b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfe59c640 1 -- 172.21.15.27:0/2435394058 >> v2:172.21.15.69:3300/0 conn(0x7f9e0015df00 msgr2=0x7f9e000d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfe59c640 1 --2- 172.21.15.27:0/2435394058 >> v2:172.21.15.69:3300/0 conn(0x7f9e0015df00 0x7f9e000d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfe59c640 1 -- 172.21.15.27:0/2435394058 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e000cf0e0 con 0x7f9e000b95a0 2024-03-06T07:23:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.880+0000 7f9dfed9d640 1 --2- 172.21.15.27:0/2435394058 >> v2:172.21.15.69:3300/0 conn(0x7f9e0015df00 0x7f9e000d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:34.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.881+0000 7f9dfe59c640 1 --2- 172.21.15.27:0/2435394058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e000b95a0 0x7f9e000b5700 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9e000d11b0 tx=0x7f9df402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c56c5db58554524 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.881+0000 7f9dfdd9b640 1 -- 172.21.15.27:0/2435394058 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4035020 con 0x7f9e000b95a0 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.881+0000 7f9dfdd9b640 1 -- 172.21.15.27:0/2435394058 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df4031870 con 0x7f9e000b95a0 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.881+0000 7f9dfdd9b640 1 -- 172.21.15.27:0/2435394058 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4031c40 con 0x7f9e000b95a0 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.882+0000 7f9e050eb640 1 -- 172.21.15.27:0/2435394058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e000b95a0 msgr2=0x7f9e000b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.882+0000 7f9e050eb640 1 --2- 172.21.15.27:0/2435394058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e000b95a0 0x7f9e000b5700 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9e000d11b0 tx=0x7f9df402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.882+0000 7f9e050eb640 1 -- 172.21.15.27:0/2435394058 shutdown_connections 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.882+0000 7f9e050eb640 1 --2- 172.21.15.27:0/2435394058 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e000b9a30 0x7f9e000b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.882+0000 7f9e050eb640 1 --2- 172.21.15.27:0/2435394058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e000b95a0 0x7f9e000b5700 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.882+0000 7f9e050eb640 1 --2- 172.21.15.27:0/2435394058 >> v2:172.21.15.69:3300/0 conn(0x7f9e0015df00 0x7f9e000d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.882+0000 7f9e050eb640 1 -- 172.21.15.27:0/2435394058 >> 172.21.15.27:0/2435394058 conn(0x7f9e00071960 msgr2=0x7f9e00071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.882+0000 7f9e050eb640 1 -- 172.21.15.27:0/2435394058 shutdown_connections 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.882+0000 7f9e050eb640 1 -- 172.21.15.27:0/2435394058 wait complete. 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.883+0000 7f9e050eb640 1 Processor -- start 2024-03-06T07:23:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.883+0000 7f9e050eb640 1 -- start start 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9e050eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e000b95a0 0x7f9e001f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9e050eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e0015df00 0x7f9e0015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9e050eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0015ade0 0x7f9e000c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9e050eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e000cec70 con 0x7f9e0015ade0 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9e050eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e000ce750 con 0x7f9e000b95a0 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9e050eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e000cedf0 con 0x7f9e0015df00 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9dff59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0015ade0 0x7f9e000c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9dff59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0015ade0 0x7f9e000c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52468/0 (socket says 172.21.15.27:52468) 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9dff59e640 1 -- 172.21.15.27:0/483479633 learned_addr learned my addr 172.21.15.27:0/483479633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9dfe59c640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e0015df00 0x7f9e0015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.884+0000 7f9dfed9d640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e000b95a0 0x7f9e001f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9dff59e640 1 -- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e0015df00 msgr2=0x7f9e0015a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9dff59e640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e0015df00 0x7f9e0015a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9dff59e640 1 -- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e000b95a0 msgr2=0x7f9e001f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9dff59e640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e000b95a0 0x7f9e001f4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9dff59e640 1 -- 172.21.15.27:0/483479633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e001f4880 con 0x7f9e0015ade0 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9dfe59c640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e0015df00 0x7f9e0015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9dff59e640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0015ade0 0x7f9e000c77b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9df800efc0 tx=0x7f9df800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9de77fe640 1 -- 172.21.15.27:0/483479633 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df8018020 con 0x7f9e0015ade0 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9de77fe640 1 -- 172.21.15.27:0/483479633 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df8014480 con 0x7f9e0015ade0 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e000c9d50 con 0x7f9e0015ade0 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9de77fe640 1 -- 172.21.15.27:0/483479633 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df8014870 con 0x7f9e0015ade0 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.885+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e000ca310 con 0x7f9e0015ade0 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.886+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e00067ff0 con 0x7f9e0015ade0 2024-03-06T07:23:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.887+0000 7f9de77fe640 1 -- 172.21.15.27:0/483479633 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9df8005d10 con 0x7f9e0015ade0 2024-03-06T07:23:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.887+0000 7f9de77fe640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9dec06b4d0 0x7f9dec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.887+0000 7f9de77fe640 1 -- 172.21.15.27:0/483479633 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9df808b0c0 con 0x7f9e0015ade0 2024-03-06T07:23:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.887+0000 7f9dfed9d640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9dec06b4d0 0x7f9dec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.888+0000 7f9dfed9d640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9dec06b4d0 0x7f9dec06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9df0006fd0 tx=0x7f9df0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:34.896+0000 7f9de77fe640 1 -- 172.21.15.27:0/483479633 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9df8022030 con 0x7f9e0015ade0 2024-03-06T07:23:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.112+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e000b5e80 con 0x7f9e0015ade0 2024-03-06T07:23:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.113+0000 7f9de77fe640 1 -- 172.21.15.27:0/483479633 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16783 (secure 0 0 0) 0x7f9e001f4880 con 0x7f9e0015ade0 2024-03-06T07:23:35.116 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:35.116 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:29.211570+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9dec06b4d0 msgr2=0x7f9dec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9dec06b4d0 0x7f9dec06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9df0006fd0 tx=0x7f9df0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0015ade0 msgr2=0x7f9e000c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0015ade0 0x7f9e000c77b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9df800efc0 tx=0x7f9df800c490 comp rx=0 tx=0).stop 2024-03-06T07:23:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 shutdown_connections 2024-03-06T07:23:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9dec06b4d0 0x7f9dec06d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e0015ade0 0x7f9e000c77b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e0015df00 0x7f9e0015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 --2- 172.21.15.27:0/483479633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e000b95a0 0x7f9e001f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 >> 172.21.15.27:0/483479633 conn(0x7f9e00071960 msgr2=0x7f9e0015d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 shutdown_connections 2024-03-06T07:23:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.115+0000 7f9e050eb640 1 -- 172.21.15.27:0/483479633 wait complete. 2024-03-06T07:23:35.129 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:46.148446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:47.412561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:35.129 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:35.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.237+0000 7f8bea5c7640 1 Processor -- start 2024-03-06T07:23:35.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.237+0000 7f8bea5c7640 1 -- start start 2024-03-06T07:23:35.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8bea5c7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8be415f500 0x7f8be40ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8bea5c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be40cf740 0x7f8be40ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8bea5c7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be40cd310 con 0x7f8be40ced80 2024-03-06T07:23:35.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8bea5c7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be415f8e0 con 0x7f8be415f500 2024-03-06T07:23:35.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8bea5c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be40c56c0 con 0x7f8be40cf740 2024-03-06T07:23:35.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8be8b64640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8be40ced80 0x7f8be40cfbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58520/0 (socket says 172.21.15.27:58520) 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8be8b64640 1 -- 172.21.15.27:0/3048202489 learned_addr learned my addr 172.21.15.27:0/3048202489 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8be3fff640 1 --2- 172.21.15.27:0/3048202489 >> v2:172.21.15.69:3300/0 conn(0x7f8be415f500 0x7f8be40ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8be37fe640 1 --2- 172.21.15.27:0/3048202489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be40cf740 0x7f8be40ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8be2ffd640 1 -- 172.21.15.27:0/3048202489 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428648468 0 0) 0x7f8be40cd310 con 0x7f8be40ced80 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.238+0000 7f8be2ffd640 1 -- 172.21.15.27:0/3048202489 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc003520 con 0x7f8be40ced80 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.239+0000 7f8be37fe640 1 -- 172.21.15.27:0/3048202489 >> v2:172.21.15.69:3300/0 conn(0x7f8be415f500 msgr2=0x7f8be40ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.239+0000 7f8be37fe640 1 --2- 172.21.15.27:0/3048202489 >> v2:172.21.15.69:3300/0 conn(0x7f8be415f500 0x7f8be40ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.239+0000 7f8be37fe640 1 -- 172.21.15.27:0/3048202489 >> v1:172.21.15.27:6789/0 conn(0x7f8be40ced80 legacy=0x7f8be40cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.239+0000 7f8be37fe640 1 -- 172.21.15.27:0/3048202489 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be40cd5b0 con 0x7f8be40cf740 2024-03-06T07:23:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.239+0000 7f8be3fff640 1 --2- 172.21.15.27:0/3048202489 >> v2:172.21.15.69:3300/0 conn(0x7f8be415f500 0x7f8be40ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.240+0000 7f8be37fe640 1 --2- 172.21.15.27:0/3048202489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be40cf740 0x7f8be40ce840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80097a0 tx=0x7f8bd802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2937e3e983c90b93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.240+0000 7f8be2ffd640 1 -- 172.21.15.27:0/3048202489 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80315a0 con 0x7f8be40cf740 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.240+0000 7f8be2ffd640 1 -- 172.21.15.27:0/3048202489 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd8037070 con 0x7f8be40cf740 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.240+0000 7f8be2ffd640 1 -- 172.21.15.27:0/3048202489 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd8031960 con 0x7f8be40cf740 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.241+0000 7f8bea5c7640 1 -- 172.21.15.27:0/3048202489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be40cf740 msgr2=0x7f8be40ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.241+0000 7f8bea5c7640 1 --2- 172.21.15.27:0/3048202489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be40cf740 0x7f8be40ce840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80097a0 tx=0x7f8bd802b840 comp rx=0 tx=0).stop 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.241+0000 7f8bea5c7640 1 -- 172.21.15.27:0/3048202489 shutdown_connections 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.241+0000 7f8bea5c7640 1 --2- 172.21.15.27:0/3048202489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be40cf740 0x7f8be40ce840 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.241+0000 7f8bea5c7640 1 --2- 172.21.15.27:0/3048202489 >> v2:172.21.15.69:3300/0 conn(0x7f8be415f500 0x7f8be40ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.241+0000 7f8bea5c7640 1 -- 172.21.15.27:0/3048202489 >> 172.21.15.27:0/3048202489 conn(0x7f8be40b3700 msgr2=0x7f8be40b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.241+0000 7f8bea5c7640 1 -- 172.21.15.27:0/3048202489 shutdown_connections 2024-03-06T07:23:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.241+0000 7f8bea5c7640 1 -- 172.21.15.27:0/3048202489 wait complete. 2024-03-06T07:23:35.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.242+0000 7f8bea5c7640 1 Processor -- start 2024-03-06T07:23:35.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.242+0000 7f8bea5c7640 1 -- start start 2024-03-06T07:23:35.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8bea5c7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be40ced80 0x7f8be4148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8bea5c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be40cf740 0x7f8be40b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8bea5c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be415f500 0x7f8be40d0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8bea5c7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be40b43d0 con 0x7f8be40ced80 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8bea5c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be40c59a0 con 0x7f8be40cf740 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8bea5c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be40c6e40 con 0x7f8be415f500 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be40ced80 0x7f8be4148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be40ced80 0x7f8be4148900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52480/0 (socket says 172.21.15.27:52480) 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be3fff640 1 -- 172.21.15.27:0/655705323 learned_addr learned my addr 172.21.15.27:0/655705323 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be8b64640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be415f500 0x7f8be40d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be37fe640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be40cf740 0x7f8be40b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be3fff640 1 -- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be415f500 msgr2=0x7f8be40d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be3fff640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be415f500 0x7f8be40d0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be3fff640 1 -- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be40cf740 msgr2=0x7f8be40b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be3fff640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be40cf740 0x7f8be40b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be3fff640 1 -- 172.21.15.27:0/655705323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be414b510 con 0x7f8be40ced80 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be3fff640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be40ced80 0x7f8be4148900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400e990 tx=0x7f8bd400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be0ff9640 1 -- 172.21.15.27:0/655705323 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4017070 con 0x7f8be40ced80 2024-03-06T07:23:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.243+0000 7f8be0ff9640 1 -- 172.21.15.27:0/655705323 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd400c9e0 con 0x7f8be40ced80 2024-03-06T07:23:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.244+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be414b7f0 con 0x7f8be40ced80 2024-03-06T07:23:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.244+0000 7f8be0ff9640 1 -- 172.21.15.27:0/655705323 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd400cdb0 con 0x7f8be40ced80 2024-03-06T07:23:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.244+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be414bdb0 con 0x7f8be40ced80 2024-03-06T07:23:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.245+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be409fcf0 con 0x7f8be40ced80 2024-03-06T07:23:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.245+0000 7f8be0ff9640 1 -- 172.21.15.27:0/655705323 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bd401a070 con 0x7f8be40ced80 2024-03-06T07:23:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.246+0000 7f8be0ff9640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8bd006b4d0 0x7f8bd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.246+0000 7f8be0ff9640 1 -- 172.21.15.27:0/655705323 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8bd408a450 con 0x7f8be40ced80 2024-03-06T07:23:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.246+0000 7f8be37fe640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8bd006b4d0 0x7f8bd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.247+0000 7f8be37fe640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8bd006b4d0 0x7f8bd006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8036000 tx=0x7f8bd8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.255+0000 7f8be0ff9640 1 -- 172.21.15.27:0/655705323 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bd40592e0 con 0x7f8be40ced80 2024-03-06T07:23:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.437+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8be40cbfa0 con 0x7f8be40ced80 2024-03-06T07:23:35.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.437+0000 7f8be0ff9640 1 -- 172.21.15.27:0/655705323 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8bd4058be0 con 0x7f8be40ced80 2024-03-06T07:23:35.440 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:35.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8bd006b4d0 msgr2=0x7f8bd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8bd006b4d0 0x7f8bd006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8036000 tx=0x7f8bd8008030 comp rx=0 tx=0).stop 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be40ced80 msgr2=0x7f8be4148900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be40ced80 0x7f8be4148900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400e990 tx=0x7f8bd400ee50 comp rx=0 tx=0).stop 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 shutdown_connections 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8bd006b4d0 0x7f8bd006d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8be415f500 0x7f8be40d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be40cf740 0x7f8be40b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 --2- 172.21.15.27:0/655705323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be40ced80 0x7f8be4148900 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 >> 172.21.15.27:0/655705323 conn(0x7f8be40b3700 msgr2=0x7f8be406ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 shutdown_connections 2024-03-06T07:23:35.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.440+0000 7f8bea5c7640 1 -- 172.21.15.27:0/655705323 wait complete. 2024-03-06T07:23:35.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.562+0000 7f21191c9640 1 Processor -- start 2024-03-06T07:23:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.562+0000 7f21191c9640 1 -- start start 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.562+0000 7f21191c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21140cdcd0 0x7f21140b1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.562+0000 7f21191c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 0x7f21140b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.562+0000 7f21191c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2114161a40 con 0x7f21140b1fd0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.562+0000 7f21191c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21140d11e0 con 0x7f21140cdcd0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.562+0000 7f21191c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211406c2d0 con 0x7f21140b2cd0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.563+0000 7f2112d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21140cdcd0 0x7f21140b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.563+0000 7f2112d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21140cdcd0 0x7f21140b1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52484/0 (socket says 172.21.15.27:52484) 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.563+0000 7f2112d9d640 1 -- 172.21.15.27:0/3340981680 learned_addr learned my addr 172.21.15.27:0/3340981680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.563+0000 7f211259c640 1 --2- 172.21.15.27:0/3340981680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 0x7f21140b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.563+0000 7f2112d9d640 1 -- 172.21.15.27:0/3340981680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 msgr2=0x7f21140b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.563+0000 7f2112d9d640 1 --2- 172.21.15.27:0/3340981680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 0x7f21140b1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.563+0000 7f2112d9d640 1 -- 172.21.15.27:0/3340981680 >> v1:172.21.15.69:6789/0 conn(0x7f21140b1fd0 legacy=0x7f21140ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.563+0000 7f2112d9d640 1 -- 172.21.15.27:0/3340981680 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21140b2a20 con 0x7f21140cdcd0 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f2112d9d640 1 --2- 172.21.15.27:0/3340981680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140cdcd0 0x7f21140b1550 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2104009870 tx=0x7f210402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12647a14bbb1c16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f2111d9b640 1 -- 172.21.15.27:0/3340981680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040317c0 con 0x7f21140cdcd0 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f2111d9b640 1 -- 172.21.15.27:0/3340981680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2104037070 con 0x7f21140cdcd0 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f2111d9b640 1 -- 172.21.15.27:0/3340981680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2104031b80 con 0x7f21140cdcd0 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f21191c9640 1 -- 172.21.15.27:0/3340981680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140cdcd0 msgr2=0x7f21140b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f21191c9640 1 --2- 172.21.15.27:0/3340981680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140cdcd0 0x7f21140b1550 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2104009870 tx=0x7f210402b860 comp rx=0 tx=0).stop 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f21191c9640 1 -- 172.21.15.27:0/3340981680 shutdown_connections 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f21191c9640 1 --2- 172.21.15.27:0/3340981680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 0x7f21140b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f21191c9640 1 --2- 172.21.15.27:0/3340981680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140cdcd0 0x7f21140b1550 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.564+0000 7f21191c9640 1 -- 172.21.15.27:0/3340981680 >> 172.21.15.27:0/3340981680 conn(0x7f21140b3700 msgr2=0x7f21140b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.565+0000 7f21191c9640 1 -- 172.21.15.27:0/3340981680 shutdown_connections 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.565+0000 7f21191c9640 1 -- 172.21.15.27:0/3340981680 wait complete. 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.566+0000 7f21191c9640 1 Processor -- start 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.566+0000 7f21191c9640 1 -- start start 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f21191c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140b1fd0 0x7f21140cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f21191c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 0x7f21141f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f21191c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21140cdcd0 0x7f21140c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f21191c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21140d0f00 con 0x7f21140b1fd0 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f21191c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21140d0c20 con 0x7f21140cdcd0 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f21191c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21140cc1c0 con 0x7f21140b2cd0 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f2112d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140b1fd0 0x7f21140cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f2112d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140b1fd0 0x7f21140cabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52488/0 (socket says 172.21.15.27:52488) 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f2112d9d640 1 -- 172.21.15.27:0/2690566704 learned_addr learned my addr 172.21.15.27:0/2690566704 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f211259c640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 0x7f21141f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f211359e640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21140cdcd0 0x7f21140c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f2112d9d640 1 -- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 msgr2=0x7f21141f42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f2112d9d640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 0x7f21141f42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f2112d9d640 1 -- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21140cdcd0 msgr2=0x7f21140c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f2112d9d640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21140cdcd0 0x7f21140c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f2112d9d640 1 -- 172.21.15.27:0/2690566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21140c84d0 con 0x7f21140b1fd0 2024-03-06T07:23:35.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f211259c640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 0x7f21141f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f211359e640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21140cdcd0 0x7f21140c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.567+0000 7f2112d9d640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140b1fd0 0x7f21140cabf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f210402b8a0 tx=0x7f21040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.568+0000 7f20fb7fe640 1 -- 172.21.15.27:0/2690566704 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2104032860 con 0x7f21140b1fd0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.568+0000 7f20fb7fe640 1 -- 172.21.15.27:0/2690566704 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2104032a20 con 0x7f21140b1fd0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.568+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21140c5820 con 0x7f21140b1fd0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.568+0000 7f20fb7fe640 1 -- 172.21.15.27:0/2690566704 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2104032e10 con 0x7f21140b1fd0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.568+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21140c5de0 con 0x7f21140b1fd0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.569+0000 7f20fb7fe640 1 -- 172.21.15.27:0/2690566704 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f210404c060 con 0x7f21140b1fd0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.569+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211409fcf0 con 0x7f21140b1fd0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.570+0000 7f20fb7fe640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f210006b4d0 0x7f210006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.570+0000 7f20fb7fe640 1 -- 172.21.15.27:0/2690566704 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f21040abc10 con 0x7f21140b1fd0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.570+0000 7f211259c640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f210006b4d0 0x7f210006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.571+0000 7f211259c640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f210006b4d0 0x7f210006d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2114155f40 tx=0x7f2108008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.574+0000 7f20fb7fe640 1 -- 172.21.15.27:0/2690566704 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21140c84d0 con 0x7f21140b1fd0 2024-03-06T07:23:35.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.797+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21140d1690 con 0x7f21140b1fd0 2024-03-06T07:23:35.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.797+0000 7f20fb7fe640 1 -- 172.21.15.27:0/2690566704 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2104040070 con 0x7f21140b1fd0 2024-03-06T07:23:35.800 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f210006b4d0 msgr2=0x7f210006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f210006b4d0 0x7f210006d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2114155f40 tx=0x7f2108008040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140b1fd0 msgr2=0x7f21140cabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140b1fd0 0x7f21140cabf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f210402b8a0 tx=0x7f21040046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 shutdown_connections 2024-03-06T07:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f210006b4d0 0x7f210006d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21140cdcd0 0x7f21140c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21140b2cd0 0x7f21141f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 --2- 172.21.15.27:0/2690566704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21140b1fd0 0x7f21140cabf0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 >> 172.21.15.27:0/2690566704 conn(0x7f21140b3700 msgr2=0x7f21140b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 shutdown_connections 2024-03-06T07:23:35.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.799+0000 7f21191c9640 1 -- 172.21.15.27:0/2690566704 wait complete. 2024-03-06T07:23:35.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.920+0000 7f8fb4d2b640 1 Processor -- start 2024-03-06T07:23:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.920+0000 7f8fb4d2b640 1 -- start start 2024-03-06T07:23:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fb4d2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb015dd90 0x7f8fb00ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fb4d2b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fb00d29d0 0x7f8fb00d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fb4d2b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb00d3740 con 0x7f8fb00d4720 2024-03-06T07:23:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fb4d2b640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb00d3400 con 0x7f8fb00d29d0 2024-03-06T07:23:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fb4d2b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb00d3120 con 0x7f8fb015dd90 2024-03-06T07:23:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fadd9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fb00d29d0 0x7f8fb00d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8faed9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8fb00d4720 0x7f8fb00b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58536/0 (socket says 172.21.15.27:58536) 2024-03-06T07:23:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8faed9d640 1 -- 172.21.15.27:0/2094350814 learned_addr learned my addr 172.21.15.27:0/2094350814 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fae59c640 1 --2- 172.21.15.27:0/2094350814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb015dd90 0x7f8fb00ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fadd9b640 1 -- 172.21.15.27:0/2094350814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb015dd90 msgr2=0x7f8fb00ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fadd9b640 1 --2- 172.21.15.27:0/2094350814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb015dd90 0x7f8fb00ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.921+0000 7f8fadd9b640 1 -- 172.21.15.27:0/2094350814 >> v1:172.21.15.27:6789/0 conn(0x7f8fb00d4720 legacy=0x7f8fb00b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.922+0000 7f8fadd9b640 1 -- 172.21.15.27:0/2094350814 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb00d39e0 con 0x7f8fb00d29d0 2024-03-06T07:23:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.922+0000 7f8fad59a640 1 -- 172.21.15.27:0/2094350814 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038951394 0 0) 0x7f8fb00d3740 con 0x7f8fb00d4720 2024-03-06T07:23:35.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.923+0000 7f8fadd9b640 1 --2- 172.21.15.27:0/2094350814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb00d29d0 0x7f8fb00d41e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40097a0 tx=0x7f8fa402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b8e628073c6833 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.923+0000 7f8fad59a640 1 -- 172.21.15.27:0/2094350814 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4034070 con 0x7f8fb00d29d0 2024-03-06T07:23:35.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.923+0000 7f8fad59a640 1 -- 172.21.15.27:0/2094350814 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa4030610 con 0x7f8fb00d29d0 2024-03-06T07:23:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.923+0000 7f8fad59a640 1 -- 172.21.15.27:0/2094350814 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa40309e0 con 0x7f8fb00d29d0 2024-03-06T07:23:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.923+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/2094350814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb00d29d0 msgr2=0x7f8fb00d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.923+0000 7f8fb4d2b640 1 --2- 172.21.15.27:0/2094350814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb00d29d0 0x7f8fb00d41e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40097a0 tx=0x7f8fa402b960 comp rx=0 tx=0).stop 2024-03-06T07:23:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.924+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/2094350814 shutdown_connections 2024-03-06T07:23:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.924+0000 7f8fb4d2b640 1 --2- 172.21.15.27:0/2094350814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb00d29d0 0x7f8fb00d41e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.924+0000 7f8fb4d2b640 1 --2- 172.21.15.27:0/2094350814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb015dd90 0x7f8fb00ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.924+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/2094350814 >> 172.21.15.27:0/2094350814 conn(0x7f8fb0071960 msgr2=0x7f8fb0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.924+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/2094350814 shutdown_connections 2024-03-06T07:23:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.924+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/2094350814 wait complete. 2024-03-06T07:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.926+0000 7f8fb4d2b640 1 Processor -- start 2024-03-06T07:23:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.926+0000 7f8fb4d2b640 1 -- start start 2024-03-06T07:23:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8fb4d2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb00d29d0 0x7f8fb01e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8fb4d2b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fb00d4720 0x7f8fb015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8fb4d2b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb015dd90 0x7f8fb00c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8fb4d2b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb00d2df0 con 0x7f8fb015dd90 2024-03-06T07:23:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8fb4d2b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb015e170 con 0x7f8fb00d4720 2024-03-06T07:23:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8fb4d2b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb00b7420 con 0x7f8fb00d29d0 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8faed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb015dd90 0x7f8fb00c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8faed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb015dd90 0x7f8fb00c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52504/0 (socket says 172.21.15.27:52504) 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8faed9d640 1 -- 172.21.15.27:0/719587719 learned_addr learned my addr 172.21.15.27:0/719587719 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8fadd9b640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fb00d4720 0x7f8fb015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.927+0000 7f8fae59c640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb00d29d0 0x7f8fb01e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8faed9d640 1 -- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb00d29d0 msgr2=0x7f8fb01e7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8faed9d640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb00d29d0 0x7f8fb01e7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8faed9d640 1 -- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fb00d4720 msgr2=0x7f8fb015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8faed9d640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fb00d4720 0x7f8fb015c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8faed9d640 1 -- 172.21.15.27:0/719587719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb01e7ca0 con 0x7f8fb015dd90 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8faed9d640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb015dd90 0x7f8fb00c8de0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800c7d0 tx=0x7f8fa800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8f9effd640 1 -- 172.21.15.27:0/719587719 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8018070 con 0x7f8fb015dd90 2024-03-06T07:23:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8f9effd640 1 -- 172.21.15.27:0/719587719 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa800a640 con 0x7f8fb015dd90 2024-03-06T07:23:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb01ea670 con 0x7f8fb015dd90 2024-03-06T07:23:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8f9effd640 1 -- 172.21.15.27:0/719587719 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa800aa70 con 0x7f8fb015dd90 2024-03-06T07:23:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.928+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb01eac30 con 0x7f8fb015dd90 2024-03-06T07:23:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.929+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb0072a30 con 0x7f8fb015dd90 2024-03-06T07:23:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.930+0000 7f8f9effd640 1 -- 172.21.15.27:0/719587719 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fa801b070 con 0x7f8fb015dd90 2024-03-06T07:23:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.930+0000 7f8f9effd640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f9006b4d0 0x7f8f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.930+0000 7f8f9effd640 1 -- 172.21.15.27:0/719587719 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8fa8089720 con 0x7f8fb015dd90 2024-03-06T07:23:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.930+0000 7f8fae59c640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f9006b4d0 0x7f8f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.931+0000 7f8fae59c640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f9006b4d0 0x7f8f9006d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00c72a0 tx=0x7f8fa0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:35.934+0000 7f8f9effd640 1 -- 172.21.15.27:0/719587719 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fa8057f60 con 0x7f8fb015dd90 2024-03-06T07:23:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.117+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8fb00d4270 con 0x7f8fb015dd90 2024-03-06T07:23:36.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.117+0000 7f8f9effd640 1 -- 172.21.15.27:0/719587719 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8fa805c000 con 0x7f8fb015dd90 2024-03-06T07:23:36.120 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:23:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f9006b4d0 msgr2=0x7f8f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f9006b4d0 0x7f8f9006d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00c72a0 tx=0x7f8fa0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb015dd90 msgr2=0x7f8fb00c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb015dd90 0x7f8fb00c8de0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800c7d0 tx=0x7f8fa800cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 shutdown_connections 2024-03-06T07:23:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f9006b4d0 0x7f8f9006d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fb015dd90 0x7f8fb00c8de0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fb00d4720 0x7f8fb015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 --2- 172.21.15.27:0/719587719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fb00d29d0 0x7f8fb01e7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 >> 172.21.15.27:0/719587719 conn(0x7f8fb0071960 msgr2=0x7f8fb00b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 shutdown_connections 2024-03-06T07:23:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.119+0000 7f8fb4d2b640 1 -- 172.21.15.27:0/719587719 wait complete. 2024-03-06T07:23:36.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:36.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.238+0000 7f36f2688640 1 Processor -- start 2024-03-06T07:23:36.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.238+0000 7f36f2688640 1 -- start start 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.239+0000 7f36f2688640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36ec15dd70 0x7f36ec0b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.239+0000 7f36f2688640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f36ec0b7c70 0x7f36ec0b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.239+0000 7f36f2688640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec0b83c0 con 0x7f36ec0b5700 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.239+0000 7f36f2688640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36ec0b60b0 con 0x7f36ec0b7c70 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.239+0000 7f36f2688640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36ec0b8fd0 con 0x7f36ec15dd70 2024-03-06T07:23:36.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.239+0000 7f36f0c25640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f36ec0b7c70 0x7f36ec0b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.239+0000 7f36f0c25640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f36ec0b7c70 0x7f36ec0b5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52518/0 (socket says 172.21.15.27:52518) 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.239+0000 7f36f0c25640 1 -- 172.21.15.27:0/3537476890 learned_addr learned my addr 172.21.15.27:0/3537476890 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.239+0000 7f36ebfff640 1 --2- 172.21.15.27:0/3537476890 >> v2:172.21.15.69:3300/0 conn(0x7f36ec15dd70 0x7f36ec0b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.240+0000 7f36f0c25640 1 -- 172.21.15.27:0/3537476890 >> v2:172.21.15.69:3300/0 conn(0x7f36ec15dd70 msgr2=0x7f36ec0b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.240+0000 7f36f0c25640 1 --2- 172.21.15.27:0/3537476890 >> v2:172.21.15.69:3300/0 conn(0x7f36ec15dd70 0x7f36ec0b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.240+0000 7f36f0c25640 1 -- 172.21.15.27:0/3537476890 >> v1:172.21.15.69:6789/0 conn(0x7f36ec0b5700 legacy=0x7f36ec0d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.240+0000 7f36f0c25640 1 -- 172.21.15.27:0/3537476890 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec0b86a0 con 0x7f36ec0b7c70 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.240+0000 7f36f0c25640 1 --2- 172.21.15.27:0/3537476890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 0x7f36ec0b5b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f36ec0b6800 tx=0x7f36e402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0135f1e2f84088 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36eaffd640 1 -- 172.21.15.27:0/3537476890 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e402cbb0 con 0x7f36ec0b7c70 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36eaffd640 1 -- 172.21.15.27:0/3537476890 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e402cd50 con 0x7f36ec0b7c70 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36eaffd640 1 -- 172.21.15.27:0/3537476890 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4034610 con 0x7f36ec0b7c70 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36f2688640 1 -- 172.21.15.27:0/3537476890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 msgr2=0x7f36ec0b5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36f2688640 1 --2- 172.21.15.27:0/3537476890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 0x7f36ec0b5b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f36ec0b6800 tx=0x7f36e402c010 comp rx=0 tx=0).stop 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36f2688640 1 -- 172.21.15.27:0/3537476890 shutdown_connections 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36f2688640 1 --2- 172.21.15.27:0/3537476890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 0x7f36ec0b5b70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36f2688640 1 --2- 172.21.15.27:0/3537476890 >> v2:172.21.15.69:3300/0 conn(0x7f36ec15dd70 0x7f36ec0b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36f2688640 1 -- 172.21.15.27:0/3537476890 >> 172.21.15.27:0/3537476890 conn(0x7f36ec071960 msgr2=0x7f36ec071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36f2688640 1 -- 172.21.15.27:0/3537476890 shutdown_connections 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.241+0000 7f36f2688640 1 -- 172.21.15.27:0/3537476890 wait complete. 2024-03-06T07:23:36.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.242+0000 7f36f2688640 1 Processor -- start 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.243+0000 7f36f2688640 1 -- start start 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.243+0000 7f36f2688640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36ec0b5700 0x7f36ec1534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.243+0000 7f36f2688640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 0x7f36ec0c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.243+0000 7f36f2688640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36ec15dd70 0x7f36ec0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.243+0000 7f36f2688640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec0b9b90 con 0x7f36ec0b7c70 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.243+0000 7f36f2688640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec0b8c60 con 0x7f36ec0b5700 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.243+0000 7f36f2688640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec0b80e0 con 0x7f36ec15dd70 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.243+0000 7f36eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 0x7f36ec0c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 0x7f36ec0c9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52520/0 (socket says 172.21.15.27:52520) 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36eb7fe640 1 -- 172.21.15.27:0/4191876731 learned_addr learned my addr 172.21.15.27:0/4191876731 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:36.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36ebfff640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36ec0b5700 0x7f36ec1534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36f0c25640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36ec15dd70 0x7f36ec0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36eb7fe640 1 -- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36ec15dd70 msgr2=0x7f36ec0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36eb7fe640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36ec15dd70 0x7f36ec0c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36eb7fe640 1 -- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36ec0b5700 msgr2=0x7f36ec1534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36eb7fe640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36ec0b5700 0x7f36ec1534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36eb7fe640 1 -- 172.21.15.27:0/4191876731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec0cb430 con 0x7f36ec0b7c70 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36eb7fe640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 0x7f36ec0c9980 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c430 tx=0x7f36e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36e8ff9640 1 -- 172.21.15.27:0/4191876731 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e00052d0 con 0x7f36ec0b7c70 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36e8ff9640 1 -- 172.21.15.27:0/4191876731 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e0005490 con 0x7f36ec0b7c70 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec0cb710 con 0x7f36ec0b7c70 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36e8ff9640 1 -- 172.21.15.27:0/4191876731 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0007260 con 0x7f36ec0b7c70 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.244+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec0cbcd0 con 0x7f36ec0b7c70 2024-03-06T07:23:36.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.245+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ec067ff0 con 0x7f36ec0b7c70 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.246+0000 7f36e8ff9640 1 -- 172.21.15.27:0/4191876731 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36e0007400 con 0x7f36ec0b7c70 2024-03-06T07:23:36.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.246+0000 7f36e8ff9640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36d806b500 0x7f36d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.246+0000 7f36e8ff9640 1 -- 172.21.15.27:0/4191876731 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f36e00896d0 con 0x7f36ec0b7c70 2024-03-06T07:23:36.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.247+0000 7f36ebfff640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36d806b500 0x7f36d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.247+0000 7f36ebfff640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36d806b500 0x7f36d806d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009aa0 tx=0x7f36dc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.256+0000 7f36e8ff9640 1 -- 172.21.15.27:0/4191876731 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36e0057e70 con 0x7f36ec0b7c70 2024-03-06T07:23:36.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.476+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36ec0b4290 con 0x7f36ec0b7c70 2024-03-06T07:23:36.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.477+0000 7f36e8ff9640 1 -- 172.21.15.27:0/4191876731 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f36e005bf10 con 0x7f36ec0b7c70 2024-03-06T07:23:36.480 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:36.480 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1135544337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1135544337},{"type":"v1","addr":"172.21.15.27:6833","nonce":1135544337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1920994895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1920994895},{"type":"v1","addr":"172.21.15.69:6837","nonce":1920994895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:48.433333+0000","modified":"2024-03-06T07:22:51.436125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/3933261233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3933261233},{"type":"v1","addr":"172.21.15.27:6835","nonce":3933261233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-06T07:23:36.480 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-03-06T07:23:36.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36d806b500 msgr2=0x7f36d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36d806b500 0x7f36d806d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009aa0 tx=0x7f36dc009190 comp rx=0 tx=0).stop 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 msgr2=0x7f36ec0c9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 0x7f36ec0c9980 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c430 tx=0x7f36e000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 shutdown_connections 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36d806b500 0x7f36d806d9c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36ec15dd70 0x7f36ec0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec0b7c70 0x7f36ec0c9980 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 --2- 172.21.15.27:0/4191876731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36ec0b5700 0x7f36ec1534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 >> 172.21.15.27:0/4191876731 conn(0x7f36ec071960 msgr2=0x7f36ec0d2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 shutdown_connections 2024-03-06T07:23:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.479+0000 7f36f2688640 1 -- 172.21.15.27:0/4191876731 wait complete. 2024-03-06T07:23:36.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:36.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.597+0000 7f762db13640 1 Processor -- start 2024-03-06T07:23:36.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.597+0000 7f762db13640 1 -- start start 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f762db13640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76280d3400 0x7f76280b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f762db13640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f76280b2b50 0x7f76280b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f762db13640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76280ce610 con 0x7f7628071540 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f762db13640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76280d2f70 con 0x7f76280d3400 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f762db13640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76280d4be0 con 0x7f76280b2b50 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f7626ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76280d3400 0x7f76280b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f76277fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7628071540 0x7f7628071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58548/0 (socket says 172.21.15.27:58548) 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f76277fe640 1 -- 172.21.15.27:0/3127591838 learned_addr learned my addr 172.21.15.27:0/3127591838 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f7627fff640 1 --2- 172.21.15.27:0/3127591838 >> v2:172.21.15.69:3300/0 conn(0x7f76280b2b50 0x7f76280b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f7626ffd640 1 -- 172.21.15.27:0/3127591838 >> v2:172.21.15.69:3300/0 conn(0x7f76280b2b50 msgr2=0x7f76280b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f7626ffd640 1 --2- 172.21.15.27:0/3127591838 >> v2:172.21.15.69:3300/0 conn(0x7f76280b2b50 0x7f76280b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f7626ffd640 1 -- 172.21.15.27:0/3127591838 >> v1:172.21.15.27:6789/0 conn(0x7f7628071540 legacy=0x7f7628071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f76267fc640 1 -- 172.21.15.27:0/3127591838 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460581950 0 0) 0x7f76280ce610 con 0x7f7628071540 2024-03-06T07:23:36.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.598+0000 7f7626ffd640 1 -- 172.21.15.27:0/3127591838 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76280d2920 con 0x7f76280d3400 2024-03-06T07:23:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.599+0000 7f7626ffd640 1 --2- 172.21.15.27:0/3127591838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 0x7f76280b30a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f76280683b0 tx=0x7f761c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebdb17f54abeb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f76267fc640 1 -- 172.21.15.27:0/3127591838 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c035020 con 0x7f76280d3400 2024-03-06T07:23:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f76267fc640 1 -- 172.21.15.27:0/3127591838 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c031830 con 0x7f76280d3400 2024-03-06T07:23:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f76267fc640 1 -- 172.21.15.27:0/3127591838 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c031c00 con 0x7f76280d3400 2024-03-06T07:23:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f762db13640 1 -- 172.21.15.27:0/3127591838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 msgr2=0x7f76280b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f762db13640 1 --2- 172.21.15.27:0/3127591838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 0x7f76280b30a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f76280683b0 tx=0x7f761c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f762db13640 1 -- 172.21.15.27:0/3127591838 shutdown_connections 2024-03-06T07:23:36.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f762db13640 1 --2- 172.21.15.27:0/3127591838 >> v2:172.21.15.69:3300/0 conn(0x7f76280b2b50 0x7f76280b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f762db13640 1 --2- 172.21.15.27:0/3127591838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 0x7f76280b30a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f762db13640 1 -- 172.21.15.27:0/3127591838 >> 172.21.15.27:0/3127591838 conn(0x7f76280d3ca0 msgr2=0x7f76280d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f762db13640 1 -- 172.21.15.27:0/3127591838 shutdown_connections 2024-03-06T07:23:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.600+0000 7f762db13640 1 -- 172.21.15.27:0/3127591838 wait complete. 2024-03-06T07:23:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.601+0000 7f762db13640 1 Processor -- start 2024-03-06T07:23:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f762db13640 1 -- start start 2024-03-06T07:23:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f762db13640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7628071540 0x7f76280cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f762db13640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76280b2b50 0x7f76281f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f762db13640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 0x7f76281f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f762db13640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762815e370 con 0x7f76280d3400 2024-03-06T07:23:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f762db13640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280ce8f0 con 0x7f76280b2b50 2024-03-06T07:23:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f762db13640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76281f4640 con 0x7f7628071540 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f7627fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 0x7f76281f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f7627fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 0x7f76281f0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52536/0 (socket says 172.21.15.27:52536) 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f7627fff640 1 -- 172.21.15.27:0/1864169694 learned_addr learned my addr 172.21.15.27:0/1864169694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f76277fe640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7628071540 0x7f76280cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.602+0000 7f7626ffd640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76280b2b50 0x7f76281f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f7627fff640 1 -- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7628071540 msgr2=0x7f76280cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f7627fff640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7628071540 0x7f76280cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f7627fff640 1 -- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76280b2b50 msgr2=0x7f76281f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f7627fff640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76280b2b50 0x7f76281f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f7627fff640 1 -- 172.21.15.27:0/1864169694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76280cb350 con 0x7f76280d3400 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f7627fff640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 0x7f76281f0ba0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f762000cd70 tx=0x7f762000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f760ffff640 1 -- 172.21.15.27:0/1864169694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7620003b90 con 0x7f76280d3400 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f760ffff640 1 -- 172.21.15.27:0/1864169694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7620003d30 con 0x7f76280d3400 2024-03-06T07:23:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76280cb630 con 0x7f76280d3400 2024-03-06T07:23:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f760ffff640 1 -- 172.21.15.27:0/1864169694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7620004240 con 0x7f76280d3400 2024-03-06T07:23:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.603+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76280cbbf0 con 0x7f76280d3400 2024-03-06T07:23:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.604+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7628068700 con 0x7f76280d3400 2024-03-06T07:23:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.605+0000 7f760ffff640 1 -- 172.21.15.27:0/1864169694 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76200043e0 con 0x7f76280d3400 2024-03-06T07:23:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.605+0000 7f760ffff640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f761406b4d0 0x7f761406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.605+0000 7f760ffff640 1 -- 172.21.15.27:0/1864169694 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f762008abe0 con 0x7f76280d3400 2024-03-06T07:23:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.605+0000 7f76277fe640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f761406b4d0 0x7f761406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.606+0000 7f76277fe640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f761406b4d0 0x7f761406d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7618000f60 tx=0x7f7618007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.610+0000 7f760ffff640 1 -- 172.21.15.27:0/1864169694 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7620059420 con 0x7f76280d3400 2024-03-06T07:23:36.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.794+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76280b6580 con 0x7f76280d3400 2024-03-06T07:23:36.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.795+0000 7f760ffff640 1 -- 172.21.15.27:0/1864169694 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16783 (secure 0 0 0) 0x7f762005d4c0 con 0x7f76280d3400 2024-03-06T07:23:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:36.798 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:29.211570+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:36.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f761406b4d0 msgr2=0x7f761406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f761406b4d0 0x7f761406d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7618000f60 tx=0x7f7618007d50 comp rx=0 tx=0).stop 2024-03-06T07:23:36.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 msgr2=0x7f76281f0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 0x7f76281f0ba0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f762000cd70 tx=0x7f762000a660 comp rx=0 tx=0).stop 2024-03-06T07:23:36.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 shutdown_connections 2024-03-06T07:23:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f761406b4d0 0x7f761406d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280d3400 0x7f76281f0ba0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76280b2b50 0x7f76281f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 --2- 172.21.15.27:0/1864169694 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7628071540 0x7f76280cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 >> 172.21.15.27:0/1864169694 conn(0x7f76280d3ca0 msgr2=0x7f76280b8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 shutdown_connections 2024-03-06T07:23:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.797+0000 7f762db13640 1 -- 172.21.15.27:0/1864169694 wait complete. 2024-03-06T07:23:36.811 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:36.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.915+0000 7fa24ffc9640 1 Processor -- start 2024-03-06T07:23:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.916+0000 7fa24ffc9640 1 -- start start 2024-03-06T07:23:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.916+0000 7fa24ffc9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa24815df00 0x7fa2480d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.916+0000 7fa24ffc9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2480b9a30 0x7fa2480d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.916+0000 7fa24ffc9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2480cea30 con 0x7fa2480b95a0 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.916+0000 7fa24ffc9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2480d2af0 con 0x7fa2480b9a30 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.916+0000 7fa24ffc9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2480cf3c0 con 0x7fa24815df00 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.917+0000 7fa24e566640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2480b9a30 0x7fa2480d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.917+0000 7fa24e566640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2480b9a30 0x7fa2480d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52538/0 (socket says 172.21.15.27:52538) 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.917+0000 7fa24e566640 1 -- 172.21.15.27:0/204298762 learned_addr learned my addr 172.21.15.27:0/204298762 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.917+0000 7fa24dd65640 1 --2- 172.21.15.27:0/204298762 >> v2:172.21.15.69:3300/0 conn(0x7fa24815df00 0x7fa2480d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.917+0000 7fa24e566640 1 -- 172.21.15.27:0/204298762 >> v2:172.21.15.69:3300/0 conn(0x7fa24815df00 msgr2=0x7fa2480d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.917+0000 7fa24e566640 1 --2- 172.21.15.27:0/204298762 >> v2:172.21.15.69:3300/0 conn(0x7fa24815df00 0x7fa2480d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.917+0000 7fa24e566640 1 -- 172.21.15.27:0/204298762 >> v1:172.21.15.69:6789/0 conn(0x7fa2480b95a0 legacy=0x7fa2480b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.917+0000 7fa24e566640 1 -- 172.21.15.27:0/204298762 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2480cf0e0 con 0x7fa2480b9a30 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.918+0000 7fa24e566640 1 --2- 172.21.15.27:0/204298762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b9a30 0x7fa2480d3220 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa2480d11b0 tx=0x7fa23802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50054533d42859d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.918+0000 7fa24cd63640 1 -- 172.21.15.27:0/204298762 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa238034070 con 0x7fa2480b9a30 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.918+0000 7fa24cd63640 1 -- 172.21.15.27:0/204298762 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa238030860 con 0x7fa2480b9a30 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.918+0000 7fa24cd63640 1 -- 172.21.15.27:0/204298762 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa238030c30 con 0x7fa2480b9a30 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.919+0000 7fa24ffc9640 1 -- 172.21.15.27:0/204298762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b9a30 msgr2=0x7fa2480d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.919+0000 7fa24ffc9640 1 --2- 172.21.15.27:0/204298762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b9a30 0x7fa2480d3220 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa2480d11b0 tx=0x7fa23802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.919+0000 7fa24ffc9640 1 -- 172.21.15.27:0/204298762 shutdown_connections 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.919+0000 7fa24ffc9640 1 --2- 172.21.15.27:0/204298762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b9a30 0x7fa2480d3220 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.919+0000 7fa24ffc9640 1 --2- 172.21.15.27:0/204298762 >> v2:172.21.15.69:3300/0 conn(0x7fa24815df00 0x7fa2480d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.919+0000 7fa24ffc9640 1 -- 172.21.15.27:0/204298762 >> 172.21.15.27:0/204298762 conn(0x7fa248071960 msgr2=0x7fa248071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.919+0000 7fa24ffc9640 1 -- 172.21.15.27:0/204298762 shutdown_connections 2024-03-06T07:23:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.919+0000 7fa24ffc9640 1 -- 172.21.15.27:0/204298762 wait complete. 2024-03-06T07:23:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.920+0000 7fa24ffc9640 1 Processor -- start 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.920+0000 7fa24ffc9640 1 -- start start 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24ffc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b95a0 0x7fa248154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24ffc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24815df00 0x7fa2480b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24ffc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa248155d50 0x7fa2480b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24ffc9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2480cec70 con 0x7fa2480b95a0 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24ffc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2480ce750 con 0x7fa248155d50 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24ffc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2480cedf0 con 0x7fa24815df00 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24dd65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b95a0 0x7fa248154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24dd65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b95a0 0x7fa248154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52540/0 (socket says 172.21.15.27:52540) 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24dd65640 1 -- 172.21.15.27:0/658379631 learned_addr learned my addr 172.21.15.27:0/658379631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24e566640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa248155d50 0x7fa2480b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24d564640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24815df00 0x7fa2480b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24dd65640 1 -- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24815df00 msgr2=0x7fa2480b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24dd65640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24815df00 0x7fa2480b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24dd65640 1 -- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa248155d50 msgr2=0x7fa2480b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24dd65640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa248155d50 0x7fa2480b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.921+0000 7fa24dd65640 1 -- 172.21.15.27:0/658379631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24815e2e0 con 0x7fa2480b95a0 2024-03-06T07:23:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.922+0000 7fa24dd65640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b95a0 0x7fa248154120 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa24000efc0 tx=0x7fa24000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.922+0000 7fa2467fc640 1 -- 172.21.15.27:0/658379631 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240018020 con 0x7fa2480b95a0 2024-03-06T07:23:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.922+0000 7fa2467fc640 1 -- 172.21.15.27:0/658379631 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa240014480 con 0x7fa2480b95a0 2024-03-06T07:23:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.922+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24815e5c0 con 0x7fa2480b95a0 2024-03-06T07:23:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.922+0000 7fa2467fc640 1 -- 172.21.15.27:0/658379631 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240014850 con 0x7fa2480b95a0 2024-03-06T07:23:36.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.922+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24815eb80 con 0x7fa2480b95a0 2024-03-06T07:23:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.923+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa248067ff0 con 0x7fa2480b95a0 2024-03-06T07:23:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.924+0000 7fa2467fc640 1 -- 172.21.15.27:0/658379631 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa24001b020 con 0x7fa2480b95a0 2024-03-06T07:23:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.924+0000 7fa2467fc640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23406b500 0x7fa23406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.924+0000 7fa2467fc640 1 -- 172.21.15.27:0/658379631 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa24008afc0 con 0x7fa2480b95a0 2024-03-06T07:23:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.924+0000 7fa24d564640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23406b500 0x7fa23406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.925+0000 7fa24d564640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23406b500 0x7fa23406d9c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa248155440 tx=0x7fa23c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:36.929+0000 7fa2467fc640 1 -- 172.21.15.27:0/658379631 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa240022020 con 0x7fa2480b95a0 2024-03-06T07:23:37.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.151+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2480d2650 con 0x7fa2480b95a0 2024-03-06T07:23:37.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.152+0000 7fa2467fc640 1 -- 172.21.15.27:0/658379631 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7fa240010040 con 0x7fa2480b95a0 2024-03-06T07:23:37.154 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:37.154 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1135544337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1135544337},{"type":"v1","addr":"172.21.15.27:6833","nonce":1135544337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1920994895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1920994895},{"type":"v1","addr":"172.21.15.69:6837","nonce":1920994895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:48.433333+0000","modified":"2024-03-06T07:22:51.436125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/3933261233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3933261233},{"type":"v1","addr":"172.21.15.27:6835","nonce":3933261233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-06T07:23:37.155 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-03-06T07:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23406b500 msgr2=0x7fa23406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23406b500 0x7fa23406d9c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa248155440 tx=0x7fa23c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b95a0 msgr2=0x7fa248154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b95a0 0x7fa248154120 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa24000efc0 tx=0x7fa24000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 shutdown_connections 2024-03-06T07:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23406b500 0x7fa23406d9c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa248155d50 0x7fa2480b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24815df00 0x7fa2480b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 --2- 172.21.15.27:0/658379631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b95a0 0x7fa248154120 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.153+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 >> 172.21.15.27:0/658379631 conn(0x7fa248071960 msgr2=0x7fa2480b8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.154+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 shutdown_connections 2024-03-06T07:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.154+0000 7fa24ffc9640 1 -- 172.21.15.27:0/658379631 wait complete. 2024-03-06T07:23:37.170 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:37.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.274+0000 7ff5e3bac640 1 Processor -- start 2024-03-06T07:23:37.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.275+0000 7ff5e3bac640 1 -- start start 2024-03-06T07:23:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.275+0000 7ff5e3bac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc15df00 0x7ff5dc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.275+0000 7ff5e3bac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc0cea30 con 0x7ff5dc0b95a0 2024-03-06T07:23:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.275+0000 7ff5e3bac640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc0d2af0 con 0x7ff5dc0b9a30 2024-03-06T07:23:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.275+0000 7ff5e3bac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0cf3c0 con 0x7ff5dc15df00 2024-03-06T07:23:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e1147640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff5dc0b95a0 0x7ff5dc0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58558/0 (socket says 172.21.15.27:58558) 2024-03-06T07:23:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e1147640 1 -- 172.21.15.27:0/4289456016 learned_addr learned my addr 172.21.15.27:0/4289456016 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e1948640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc15df00 0x7ff5dc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897908134 0 0) 0x7ff5dc0cea30 con 0x7ff5dc0b95a0 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c8003520 con 0x7ff5dc0b95a0 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1828568495 0 0) 0x7ff5c8003520 con 0x7ff5dc0b95a0 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc15df00 msgr2=0x7ff5dc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc15df00 0x7ff5dc0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 >> v1:172.21.15.69:6789/0 conn(0x7ff5dc0b9a30 legacy=0x7ff5dc0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0cf0e0 con 0x7ff5dc0b95a0 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7ff5cc001670 con 0x7ff5dc0b95a0 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 >> v1:172.21.15.27:6789/0 conn(0x7ff5dc0b95a0 legacy=0x7ff5dc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5c80039e0 0x7ff5c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc15df00 0x7ff5c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c8004d40 0x7ff5c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0cf3c0 con 0x7ff5c8004d40 2024-03-06T07:23:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c800b9c0 con 0x7ff5c80039e0 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c800bbc0 con 0x7ff5dc15df00 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e2149640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c8004d40 0x7ff5c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e1948640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5c80039e0 0x7ff5c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.276+0000 7ff5e1147640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc15df00 0x7ff5c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.277+0000 7ff5e2149640 1 -- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc15df00 msgr2=0x7ff5c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.277+0000 7ff5e2149640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc15df00 0x7ff5c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.277+0000 7ff5e2149640 1 -- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5c80039e0 msgr2=0x7ff5c8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.277+0000 7ff5e2149640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5c80039e0 0x7ff5c8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.277+0000 7ff5e2149640 1 -- 172.21.15.27:0/4289456016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c8003520 con 0x7ff5c8004d40 2024-03-06T07:23:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.278+0000 7ff5e2149640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c8004d40 0x7ff5c80051c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4001f30 tx=0x7ff5c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a8e5ceba6c1634 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.278+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4001b30 con 0x7ff5c8004d40 2024-03-06T07:23:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.278+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d402d6e0 con 0x7ff5c8004d40 2024-03-06T07:23:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.278+0000 7ff5e0946640 1 -- 172.21.15.27:0/4289456016 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d402dad0 con 0x7ff5c8004d40 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.279+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4289456016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c8004d40 msgr2=0x7ff5c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.279+0000 7ff5e3bac640 1 --2- 172.21.15.27:0/4289456016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c8004d40 0x7ff5c80051c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4001f30 tx=0x7ff5c8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.279+0000 7ff5e2149640 1 -- 172.21.15.27:0/4289456016 reap_dead start 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.279+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4289456016 shutdown_connections 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.279+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4289456016 >> 172.21.15.27:0/4289456016 conn(0x7ff5dc071960 msgr2=0x7ff5dc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.279+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4289456016 shutdown_connections 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.279+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4289456016 wait complete. 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.280+0000 7ff5e3bac640 1 Processor -- start 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.280+0000 7ff5e3bac640 1 -- start start 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e3bac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5dc155340 0x7ff5dc0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e3bac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc0cc180 0x7ff5dc0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e3bac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5dc0b4ee0 0x7ff5dc0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e3bac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0cec70 con 0x7ff5dc155340 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e3bac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0ce750 con 0x7ff5dc0b4ee0 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e3bac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0cedf0 con 0x7ff5dc0cc180 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1948640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5dc155340 0x7ff5dc0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1948640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5dc155340 0x7ff5dc0ca100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52568/0 (socket says 172.21.15.27:52568) 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1948640 1 -- 172.21.15.27:0/4222513410 learned_addr learned my addr 172.21.15.27:0/4222513410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e2149640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5dc0b4ee0 0x7ff5dc0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1147640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc0cc180 0x7ff5dc0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1948640 1 -- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc0cc180 msgr2=0x7ff5dc0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1948640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc0cc180 0x7ff5dc0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1948640 1 -- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5dc0b4ee0 msgr2=0x7ff5dc0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1948640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5dc0b4ee0 0x7ff5dc0cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1948640 1 -- 172.21.15.27:0/4222513410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0cc9b0 con 0x7ff5dc155340 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.281+0000 7ff5e1948640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5dc155340 0x7ff5dc0ca100 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80119d0 tx=0x7ff5d8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.282+0000 7ff5d27fc640 1 -- 172.21.15.27:0/4222513410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d800fb50 con 0x7ff5dc155340 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.282+0000 7ff5d27fc640 1 -- 172.21.15.27:0/4222513410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d800fcf0 con 0x7ff5dc155340 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.282+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc0ccc90 con 0x7ff5dc155340 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.282+0000 7ff5d27fc640 1 -- 172.21.15.27:0/4222513410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d800fea0 con 0x7ff5dc155340 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.282+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc0cd250 con 0x7ff5dc155340 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.283+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc067ff0 con 0x7ff5dc155340 2024-03-06T07:23:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.283+0000 7ff5d27fc640 1 -- 172.21.15.27:0/4222513410 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5d8012040 con 0x7ff5dc155340 2024-03-06T07:23:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.284+0000 7ff5d27fc640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5c006b500 0x7ff5c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.284+0000 7ff5d27fc640 1 -- 172.21.15.27:0/4222513410 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff5d808a870 con 0x7ff5dc155340 2024-03-06T07:23:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.284+0000 7ff5e1147640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5c006b500 0x7ff5c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.285+0000 7ff5e1147640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5c006b500 0x7ff5c006d9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0b6120 tx=0x7ff5cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.288+0000 7ff5d27fc640 1 -- 172.21.15.27:0/4222513410 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5d8019030 con 0x7ff5dc155340 2024-03-06T07:23:37.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.507+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5dc0b80f0 con 0x7ff5dc155340 2024-03-06T07:23:37.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.507+0000 7ff5d27fc640 1 -- 172.21.15.27:0/4222513410 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4971 (secure 0 0 0) 0x7ff5d8059700 con 0x7ff5dc155340 2024-03-06T07:23:37.510 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:37.510 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1135544337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1135544337},{"type":"v1","addr":"172.21.15.27:6833","nonce":1135544337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3914889362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3914889362},{"type":"v1","addr":"172.21.15.69:6836","nonce":3914889362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1920994895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1920994895},{"type":"v1","addr":"172.21.15.69:6837","nonce":1920994895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:48.433333+0000","modified":"2024-03-06T07:22:51.436125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/3933261233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3933261233},{"type":"v1","addr":"172.21.15.27:6835","nonce":3933261233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-03-06T07:23:37.510 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5c006b500 msgr2=0x7ff5c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5c006b500 0x7ff5c006d9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0b6120 tx=0x7ff5cc009040 comp rx=0 tx=0).stop 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5dc155340 msgr2=0x7ff5dc0ca100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5dc155340 0x7ff5dc0ca100 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80119d0 tx=0x7ff5d8011e90 comp rx=0 tx=0).stop 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 shutdown_connections 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5c006b500 0x7ff5c006d9c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5dc0b4ee0 0x7ff5dc0cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5dc0cc180 0x7ff5dc0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 --2- 172.21.15.27:0/4222513410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5dc155340 0x7ff5dc0ca100 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 >> 172.21.15.27:0/4222513410 conn(0x7ff5dc071960 msgr2=0x7ff5dc0b8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 shutdown_connections 2024-03-06T07:23:37.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.509+0000 7ff5e3bac640 1 -- 172.21.15.27:0/4222513410 wait complete. 2024-03-06T07:23:37.523 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.627+0000 7fee4cfc7640 1 Processor -- start 2024-03-06T07:23:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.627+0000 7fee4cfc7640 1 -- start start 2024-03-06T07:23:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.628+0000 7fee4cfc7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee48071540 0x7fee480b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.628+0000 7fee4cfc7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee480ce610 con 0x7fee480b2b50 2024-03-06T07:23:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.628+0000 7fee4cfc7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee480d2920 con 0x7fee480d3400 2024-03-06T07:23:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.628+0000 7fee4cfc7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee480d2f70 con 0x7fee48071540 2024-03-06T07:23:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.628+0000 7fee4659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee48071540 0x7fee480b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.628+0000 7fee46d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fee480b2b50 0x7fee480b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58572/0 (socket says 172.21.15.27:58572) 2024-03-06T07:23:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.628+0000 7fee46d9d640 1 -- 172.21.15.27:0/2031835994 learned_addr learned my addr 172.21.15.27:0/2031835994 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.629+0000 7fee4659c640 1 -- 172.21.15.27:0/2031835994 >> v1:172.21.15.69:6789/0 conn(0x7fee480d3400 legacy=0x7fee48071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.629+0000 7fee4659c640 1 -- 172.21.15.27:0/2031835994 >> v1:172.21.15.27:6789/0 conn(0x7fee480b2b50 legacy=0x7fee480b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.629+0000 7fee4659c640 1 -- 172.21.15.27:0/2031835994 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee480d2c00 con 0x7fee48071540 2024-03-06T07:23:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.629+0000 7fee4559a640 1 -- 172.21.15.27:0/2031835994 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797569800 0 0) 0x7fee480ce610 con 0x7fee480b2b50 2024-03-06T07:23:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.630+0000 7fee4659c640 1 --2- 172.21.15.27:0/2031835994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee48071540 0x7fee480b30a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fee380097a0 tx=0x7fee3802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac662e7f1f572bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.630+0000 7fee4559a640 1 -- 172.21.15.27:0/2031835994 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee38034070 con 0x7fee48071540 2024-03-06T07:23:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.630+0000 7fee4559a640 1 -- 172.21.15.27:0/2031835994 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee38030610 con 0x7fee48071540 2024-03-06T07:23:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.630+0000 7fee4559a640 1 -- 172.21.15.27:0/2031835994 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee380309e0 con 0x7fee48071540 2024-03-06T07:23:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.631+0000 7fee4cfc7640 1 -- 172.21.15.27:0/2031835994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee48071540 msgr2=0x7fee480b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.631+0000 7fee4cfc7640 1 --2- 172.21.15.27:0/2031835994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee48071540 0x7fee480b30a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fee380097a0 tx=0x7fee3802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.631+0000 7fee4cfc7640 1 -- 172.21.15.27:0/2031835994 shutdown_connections 2024-03-06T07:23:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.631+0000 7fee4cfc7640 1 --2- 172.21.15.27:0/2031835994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee48071540 0x7fee480b30a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.631+0000 7fee4cfc7640 1 -- 172.21.15.27:0/2031835994 >> 172.21.15.27:0/2031835994 conn(0x7fee480d3ca0 msgr2=0x7fee480d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.631+0000 7fee4cfc7640 1 -- 172.21.15.27:0/2031835994 shutdown_connections 2024-03-06T07:23:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.631+0000 7fee4cfc7640 1 -- 172.21.15.27:0/2031835994 wait complete. 2024-03-06T07:23:37.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.633+0000 7fee4cfc7640 1 Processor -- start 2024-03-06T07:23:37.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.633+0000 7fee4cfc7640 1 -- start start 2024-03-06T07:23:37.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee4cfc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee48071540 0x7fee480aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee4cfc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee480b2b50 0x7fee480ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee4cfc7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee480d3400 0x7fee480ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee4cfc7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4815e370 con 0x7fee480b2b50 2024-03-06T07:23:37.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee4cfc7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee480d4be0 con 0x7fee480d3400 2024-03-06T07:23:37.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee4cfc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee480ab120 con 0x7fee48071540 2024-03-06T07:23:37.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee45d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee480b2b50 0x7fee480ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee45d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee480b2b50 0x7fee480ad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52586/0 (socket says 172.21.15.27:52586) 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee45d9b640 1 -- 172.21.15.27:0/3397699363 learned_addr learned my addr 172.21.15.27:0/3397699363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee4659c640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee48071540 0x7fee480aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.634+0000 7fee46d9d640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee480d3400 0x7fee480ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee45d9b640 1 -- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee48071540 msgr2=0x7fee480aabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee45d9b640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee48071540 0x7fee480aabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee45d9b640 1 -- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee480d3400 msgr2=0x7fee480ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee45d9b640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee480d3400 0x7fee480ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee45d9b640 1 -- 172.21.15.27:0/3397699363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4814de90 con 0x7fee480b2b50 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee45d9b640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee480b2b50 0x7fee480ad870 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00c3c0 tx=0x7fee3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee36ffd640 1 -- 172.21.15.27:0/3397699363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c017070 con 0x7fee480b2b50 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee36ffd640 1 -- 172.21.15.27:0/3397699363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3c0052f0 con 0x7fee480b2b50 2024-03-06T07:23:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4814e170 con 0x7fee480b2b50 2024-03-06T07:23:37.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee36ffd640 1 -- 172.21.15.27:0/3397699363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c003830 con 0x7fee480b2b50 2024-03-06T07:23:37.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.635+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4814ea10 con 0x7fee480b2b50 2024-03-06T07:23:37.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.636+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee48068700 con 0x7fee480b2b50 2024-03-06T07:23:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.637+0000 7fee36ffd640 1 -- 172.21.15.27:0/3397699363 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee3c01a070 con 0x7fee480b2b50 2024-03-06T07:23:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.637+0000 7fee36ffd640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee2806b4d0 0x7fee2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.638+0000 7fee36ffd640 1 -- 172.21.15.27:0/3397699363 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fee3c089560 con 0x7fee480b2b50 2024-03-06T07:23:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.638+0000 7fee4659c640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee2806b4d0 0x7fee2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.638+0000 7fee4659c640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee2806b4d0 0x7fee2806d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fee38036000 tx=0x7fee38008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.642+0000 7fee36ffd640 1 -- 172.21.15.27:0/3397699363 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee3c057c80 con 0x7fee480b2b50 2024-03-06T07:23:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.822+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee480cdfb0 con 0x7fee480b2b50 2024-03-06T07:23:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.824+0000 7fee36ffd640 1 -- 172.21.15.27:0/3397699363 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16783 (secure 0 0 0) 0x7fee3c05bd20 con 0x7fee480b2b50 2024-03-06T07:23:37.827 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:37.827 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:29.211570+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:46.148446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:47.412561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee2806b4d0 msgr2=0x7fee2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee2806b4d0 0x7fee2806d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fee38036000 tx=0x7fee38008030 comp rx=0 tx=0).stop 2024-03-06T07:23:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee480b2b50 msgr2=0x7fee480ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee480b2b50 0x7fee480ad870 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00c3c0 tx=0x7fee3c00c880 comp rx=0 tx=0).stop 2024-03-06T07:23:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 shutdown_connections 2024-03-06T07:23:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee2806b4d0 0x7fee2806d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee480d3400 0x7fee480ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee480b2b50 0x7fee480ad870 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 --2- 172.21.15.27:0/3397699363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee48071540 0x7fee480aabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 >> 172.21.15.27:0/3397699363 conn(0x7fee480d3ca0 msgr2=0x7fee4815f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.826+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 shutdown_connections 2024-03-06T07:23:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.827+0000 7fee4cfc7640 1 -- 172.21.15.27:0/3397699363 wait complete. 2024-03-06T07:23:37.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.945+0000 7fbdfdbe4640 1 Processor -- start 2024-03-06T07:23:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.945+0000 7fbdfdbe4640 1 -- start start 2024-03-06T07:23:37.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.946+0000 7fbdfdbe4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.946+0000 7fbdfdbe4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf80ce610 con 0x7fbdf8071540 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.946+0000 7fbdfdbe4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf80d2920 con 0x7fbdf80b2b50 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.946+0000 7fbdfdbe4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80d2f70 con 0x7fbdf80d3400 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.946+0000 7fbdf77fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbdf8071540 0x7fbdf8071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58578/0 (socket says 172.21.15.27:58578) 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.946+0000 7fbdf77fe640 1 -- 172.21.15.27:0/2789047541 learned_addr learned my addr 172.21.15.27:0/2789047541 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.946+0000 7fbdf6ffd640 1 --2- 172.21.15.27:0/2789047541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.946+0000 7fbdf67fc640 1 -- 172.21.15.27:0/2789047541 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051019394 0 0) 0x7fbdf80ce610 con 0x7fbdf8071540 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.946+0000 7fbdf67fc640 1 -- 172.21.15.27:0/2789047541 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde0003520 con 0x7fbdf8071540 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.947+0000 7fbdf67fc640 1 -- 172.21.15.27:0/2789047541 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375702231 0 0) 0x7fbdf80d2920 con 0x7fbdf80b2b50 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.947+0000 7fbdf67fc640 1 -- 172.21.15.27:0/2789047541 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf80ce610 con 0x7fbdf80b2b50 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.947+0000 7fbdf6ffd640 1 -- 172.21.15.27:0/2789047541 >> v1:172.21.15.69:6789/0 conn(0x7fbdf80b2b50 legacy=0x7fbdf80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.947+0000 7fbdf67fc640 1 -- 172.21.15.27:0/2789047541 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 210509927 0 0) 0x7fbde0003520 con 0x7fbdf8071540 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.947+0000 7fbdf6ffd640 1 -- 172.21.15.27:0/2789047541 >> v1:172.21.15.27:6789/0 conn(0x7fbdf8071540 legacy=0x7fbdf8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.947+0000 7fbdf6ffd640 1 -- 172.21.15.27:0/2789047541 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf80d2c00 con 0x7fbdf80d3400 2024-03-06T07:23:37.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.948+0000 7fbdf6ffd640 1 --2- 172.21.15.27:0/2789047541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf80b30a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0097d0 tx=0x7fbdec02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba7afda2d75b09f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.948+0000 7fbdf67fc640 1 -- 172.21.15.27:0/2789047541 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec037070 con 0x7fbdf80d3400 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.948+0000 7fbdf67fc640 1 -- 172.21.15.27:0/2789047541 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec030510 con 0x7fbdf80d3400 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.948+0000 7fbdf67fc640 1 -- 172.21.15.27:0/2789047541 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8001670 con 0x7fbdf80d3400 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.949+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2789047541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 msgr2=0x7fbdf80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.949+0000 7fbdfdbe4640 1 --2- 172.21.15.27:0/2789047541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf80b30a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0097d0 tx=0x7fbdec02b800 comp rx=0 tx=0).stop 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.949+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2789047541 shutdown_connections 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.949+0000 7fbdfdbe4640 1 --2- 172.21.15.27:0/2789047541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf80b30a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.949+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2789047541 >> 172.21.15.27:0/2789047541 conn(0x7fbdf80d3ca0 msgr2=0x7fbdf80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.949+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2789047541 shutdown_connections 2024-03-06T07:23:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.949+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2789047541 wait complete. 2024-03-06T07:23:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.950+0000 7fbdfdbe4640 1 Processor -- start 2024-03-06T07:23:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.950+0000 7fbdfdbe4640 1 -- start start 2024-03-06T07:23:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdfdbe4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf8071540 0x7fbdf8156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdfdbe4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80b2b50 0x7fbdf8154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdfdbe4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf815f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdfdbe4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf815e370 con 0x7fbdf8071540 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdfdbe4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80d4be0 con 0x7fbdf80b2b50 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdfdbe4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf81572d0 con 0x7fbdf80d3400 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf8071540 0x7fbdf8156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf8071540 0x7fbdf8156d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52590/0 (socket says 172.21.15.27:52590) 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf77fe640 1 -- 172.21.15.27:0/2841892889 learned_addr learned my addr 172.21.15.27:0/2841892889 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf7fff640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf815f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf6ffd640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80b2b50 0x7fbdf8154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf77fe640 1 -- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 msgr2=0x7fbdf815f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf77fe640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf815f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf77fe640 1 -- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80b2b50 msgr2=0x7fbdf8154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf77fe640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80b2b50 0x7fbdf8154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf77fe640 1 -- 172.21.15.27:0/2841892889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf8155240 con 0x7fbdf8071540 2024-03-06T07:23:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.951+0000 7fbdf7fff640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf815f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.952+0000 7fbdf77fe640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf8071540 0x7fbdf8156d90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbde800c780 tx=0x7fbde800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.952+0000 7fbddffff640 1 -- 172.21.15.27:0/2841892889 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8017070 con 0x7fbdf8071540 2024-03-06T07:23:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.952+0000 7fbddffff640 1 -- 172.21.15.27:0/2841892889 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8005af0 con 0x7fbdf8071540 2024-03-06T07:23:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.952+0000 7fbddffff640 1 -- 172.21.15.27:0/2841892889 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8007070 con 0x7fbdf8071540 2024-03-06T07:23:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.952+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf8155cf0 con 0x7fbdf8071540 2024-03-06T07:23:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.952+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf81562b0 con 0x7fbdf8071540 2024-03-06T07:23:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.953+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf8068700 con 0x7fbdf8071540 2024-03-06T07:23:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.953+0000 7fbddffff640 1 -- 172.21.15.27:0/2841892889 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbde801a070 con 0x7fbdf8071540 2024-03-06T07:23:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.954+0000 7fbddffff640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbde406b4d0 0x7fbde406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.954+0000 7fbddffff640 1 -- 172.21.15.27:0/2841892889 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbde8089720 con 0x7fbdf8071540 2024-03-06T07:23:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.954+0000 7fbdf6ffd640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbde406b4d0 0x7fbde406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.955+0000 7fbdf6ffd640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbde406b4d0 0x7fbde406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbdec034000 tx=0x7fbdec008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:37.958+0000 7fbddffff640 1 -- 172.21.15.27:0/2841892889 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbde8057f60 con 0x7fbdf8071540 2024-03-06T07:23:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.184+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbdf80ce0c0 con 0x7fbdf8071540 2024-03-06T07:23:38.187 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:23:38.185+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.186+0000 7fbddffff640 1 -- 172.21.15.27:0/2841892889 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbde805c000 con 0x7fbdf8071540 2024-03-06T07:23:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbde406b4d0 msgr2=0x7fbde406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbde406b4d0 0x7fbde406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbdec034000 tx=0x7fbdec008030 comp rx=0 tx=0).stop 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf8071540 msgr2=0x7fbdf8156d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf8071540 0x7fbdf8156d90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbde800c780 tx=0x7fbde800cc40 comp rx=0 tx=0).stop 2024-03-06T07:23:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 shutdown_connections 2024-03-06T07:23:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbde406b4d0 0x7fbde406d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbdf80d3400 0x7fbdf815f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdf80b2b50 0x7fbdf8154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 --2- 172.21.15.27:0/2841892889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf8071540 0x7fbdf8156d90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 >> 172.21.15.27:0/2841892889 conn(0x7fbdf80d3ca0 msgr2=0x7fbdf80ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 shutdown_connections 2024-03-06T07:23:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.189+0000 7fbdfdbe4640 1 -- 172.21.15.27:0/2841892889 wait complete. 2024-03-06T07:23:38.194 INFO:tasks.ceph.mds.c.smithi027.stderr: -3115> 2024-03-06T07:22:52.601+0000 7f7e36695640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:38.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:38.224 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:23:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.313+0000 7f3fc1d20640 1 Processor -- start 2024-03-06T07:23:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.313+0000 7f3fc1d20640 1 -- start start 2024-03-06T07:23:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.313+0000 7f3fc1d20640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fbc0b79f0 0x7f3fbc0b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.313+0000 7f3fc1d20640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0d4960 con 0x7f3fbc071540 2024-03-06T07:23:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.313+0000 7f3fc1d20640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0d36c0 con 0x7f3fbc0b2d30 2024-03-06T07:23:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.313+0000 7f3fc1d20640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc0d3230 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.314+0000 7f3fbbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fbc0b79f0 0x7f3fbc0b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.314+0000 7f3fbb7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3fbc071540 0x7f3fbc071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58592/0 (socket says 172.21.15.27:58592) 2024-03-06T07:23:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.314+0000 7f3fbb7fe640 1 -- 172.21.15.27:0/1966728503 learned_addr learned my addr 172.21.15.27:0/1966728503 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.314+0000 7f3fbbfff640 1 -- 172.21.15.27:0/1966728503 >> v1:172.21.15.69:6789/0 conn(0x7f3fbc0b2d30 legacy=0x7f3fbc0b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.314+0000 7f3fbbfff640 1 -- 172.21.15.27:0/1966728503 >> v1:172.21.15.27:6789/0 conn(0x7f3fbc071540 legacy=0x7f3fbc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.314+0000 7f3fba7fc640 1 -- 172.21.15.27:0/1966728503 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159067420 0 0) 0x7f3fbc0d4960 con 0x7f3fbc071540 2024-03-06T07:23:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.314+0000 7f3fbbfff640 1 -- 172.21.15.27:0/1966728503 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0d2620 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.315+0000 7f3fbbfff640 1 --2- 172.21.15.27:0/1966728503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 0x7f3fbc0b7ea0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4009d90 tx=0x7f3fb402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ebb296a084c8dff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.315+0000 7f3fba7fc640 1 -- 172.21.15.27:0/1966728503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4039070 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.315+0000 7f3fba7fc640 1 -- 172.21.15.27:0/1966728503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb402cd00 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.315+0000 7f3fba7fc640 1 -- 172.21.15.27:0/1966728503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4034660 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.315+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1966728503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 msgr2=0x7f3fbc0b7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.315+0000 7f3fc1d20640 1 --2- 172.21.15.27:0/1966728503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 0x7f3fbc0b7ea0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4009d90 tx=0x7f3fb402bf50 comp rx=0 tx=0).stop 2024-03-06T07:23:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.315+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1966728503 shutdown_connections 2024-03-06T07:23:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.315+0000 7f3fc1d20640 1 --2- 172.21.15.27:0/1966728503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 0x7f3fbc0b7ea0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.315+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1966728503 >> 172.21.15.27:0/1966728503 conn(0x7f3fbc0d3d60 msgr2=0x7f3fbc0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.316+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1966728503 shutdown_connections 2024-03-06T07:23:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.316+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1966728503 wait complete. 2024-03-06T07:23:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.317+0000 7f3fc1d20640 1 Processor -- start 2024-03-06T07:23:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.317+0000 7f3fc1d20640 1 -- start start 2024-03-06T07:23:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fc1d20640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc071540 0x7f3fbc1545f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fc1d20640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fc1d20640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 0x7f3fbc150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fc1d20640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0d2be0 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fc1d20640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0b83e0 con 0x7f3fbc0b2d30 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fc1d20640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0d2ec0 con 0x7f3fbc071540 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 0x7f3fbc150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 0x7f3fbc150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52618/0 (socket says 172.21.15.27:52618) 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbbfff640 1 -- 172.21.15.27:0/1386927491 learned_addr learned my addr 172.21.15.27:0/1386927491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbb7fe640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc071540 0x7f3fbc1545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbaffd640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbbfff640 1 -- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc071540 msgr2=0x7f3fbc1545f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbbfff640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc071540 0x7f3fbc1545f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbbfff640 1 -- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc0b2d30 msgr2=0x7f3fbc0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbbfff640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbbfff640 1 -- 172.21.15.27:0/1386927491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc154b30 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbaffd640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fbbfff640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 0x7f3fbc150350 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4009d90 tx=0x7f3fb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fa3fff640 1 -- 172.21.15.27:0/1386927491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4039050 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.318+0000 7f3fa3fff640 1 -- 172.21.15.27:0/1386927491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb403a040 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.319+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc0cd160 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.319+0000 7f3fa3fff640 1 -- 172.21.15.27:0/1386927491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb403e610 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.319+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc0cb110 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.319+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc0694e0 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.320+0000 7f3fa3fff640 1 -- 172.21.15.27:0/1386927491 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fb4047020 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.320+0000 7f3fa3fff640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa806b4d0 0x7f3fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.320+0000 7f3fa3fff640 1 -- 172.21.15.27:0/1386927491 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f3fb40ac6a0 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.321+0000 7f3fbb7fe640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa806b4d0 0x7f3fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.321+0000 7f3fbb7fe640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa806b4d0 0x7f3fa806d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3fac000f60 tx=0x7f3fac007d20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.325+0000 7f3fa3fff640 1 -- 172.21.15.27:0/1386927491 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fbc0694e0 con 0x7f3fbc0b79f0 2024-03-06T07:23:38.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:38.546+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3fbc072070 con 0x7f3fbc0b79f0 2024-03-06T07:23:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.188+0000 7f3fa3fff640 1 -- 172.21.15.27:0/1386927491 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3fb404e020 con 0x7f3fbc0b79f0 2024-03-06T07:23:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.190+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa806b4d0 msgr2=0x7f3fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.190+0000 7f3fc1d20640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa806b4d0 0x7f3fa806d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3fac000f60 tx=0x7f3fac007d20 comp rx=0 tx=0).stop 2024-03-06T07:23:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.190+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 msgr2=0x7f3fbc150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.190+0000 7f3fc1d20640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 0x7f3fbc150350 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4009d90 tx=0x7f3fb4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.191+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 shutdown_connections 2024-03-06T07:23:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.191+0000 7f3fc1d20640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa806b4d0 0x7f3fa806d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.191+0000 7f3fc1d20640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b79f0 0x7f3fbc150350 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.191+0000 7f3fc1d20640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.191+0000 7f3fc1d20640 1 --2- 172.21.15.27:0/1386927491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc071540 0x7f3fbc1545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.191+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 >> 172.21.15.27:0/1386927491 conn(0x7f3fbc0d3d60 msgr2=0x7f3fbc15b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.191+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 shutdown_connections 2024-03-06T07:23:39.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.191+0000 7f3fc1d20640 1 -- 172.21.15.27:0/1386927491 wait complete. 2024-03-06T07:23:39.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.362+0000 7ff1c1193640 1 Processor -- start 2024-03-06T07:23:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.362+0000 7ff1c1193640 1 -- start start 2024-03-06T07:23:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1c1193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 0x7ff1bc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1c1193640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff1bc0d29d0 0x7ff1bc0d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1c1193640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1bc0d4720 0x7ff1bc0b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1c1193640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1bc0d3400 con 0x7ff1bc0d4720 2024-03-06T07:23:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1c1193640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1bc0d3740 con 0x7ff1bc0d29d0 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1c1193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0d3120 con 0x7ff1bc15dd90 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1bb59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1bc0d4720 0x7ff1bc0b6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1bb59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1bc0d4720 0x7ff1bc0b6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52632/0 (socket says 172.21.15.27:52632) 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1bb59e640 1 -- 172.21.15.27:0/3140039452 learned_addr learned my addr 172.21.15.27:0/3140039452 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1bad9d640 1 --2- 172.21.15.27:0/3140039452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 0x7ff1bc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.363+0000 7ff1ba59c640 1 --2- 172.21.15.27:0/3140039452 >> v2:172.21.15.69:3300/0 conn(0x7ff1bc0d29d0 0x7ff1bc0d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.364+0000 7ff1bb59e640 1 -- 172.21.15.27:0/3140039452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 msgr2=0x7ff1bc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.364+0000 7ff1bb59e640 1 --2- 172.21.15.27:0/3140039452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 0x7ff1bc0ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.364+0000 7ff1bb59e640 1 -- 172.21.15.27:0/3140039452 >> v2:172.21.15.69:3300/0 conn(0x7ff1bc0d29d0 msgr2=0x7ff1bc0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.364+0000 7ff1bb59e640 1 --2- 172.21.15.27:0/3140039452 >> v2:172.21.15.69:3300/0 conn(0x7ff1bc0d29d0 0x7ff1bc0d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.364+0000 7ff1bb59e640 1 -- 172.21.15.27:0/3140039452 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc0d39e0 con 0x7ff1bc0d4720 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.364+0000 7ff1bad9d640 1 --2- 172.21.15.27:0/3140039452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 0x7ff1bc0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.364+0000 7ff1ba59c640 1 --2- 172.21.15.27:0/3140039452 >> v2:172.21.15.69:3300/0 conn(0x7ff1bc0d29d0 0x7ff1bc0d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.364+0000 7ff1bb59e640 1 --2- 172.21.15.27:0/3140039452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 0x7ff1bc0b6d10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0b6890 tx=0x7ff1ac02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f388b05f589343 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1b9d9b640 1 -- 172.21.15.27:0/3140039452 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac02cb00 con 0x7ff1bc0d4720 2024-03-06T07:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1b9d9b640 1 -- 172.21.15.27:0/3140039452 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ac02cca0 con 0x7ff1bc0d4720 2024-03-06T07:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1b9d9b640 1 -- 172.21.15.27:0/3140039452 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac034610 con 0x7ff1bc0d4720 2024-03-06T07:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1c1193640 1 -- 172.21.15.27:0/3140039452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 msgr2=0x7ff1bc0b6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3140039452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 0x7ff1bc0b6d10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0b6890 tx=0x7ff1ac02bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1c1193640 1 -- 172.21.15.27:0/3140039452 shutdown_connections 2024-03-06T07:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3140039452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 0x7ff1bc0b6d10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3140039452 >> v2:172.21.15.69:3300/0 conn(0x7ff1bc0d29d0 0x7ff1bc0d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3140039452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 0x7ff1bc0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1c1193640 1 -- 172.21.15.27:0/3140039452 >> 172.21.15.27:0/3140039452 conn(0x7ff1bc071960 msgr2=0x7ff1bc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1c1193640 1 -- 172.21.15.27:0/3140039452 shutdown_connections 2024-03-06T07:23:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.365+0000 7ff1c1193640 1 -- 172.21.15.27:0/3140039452 wait complete. 2024-03-06T07:23:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.367+0000 7ff1c1193640 1 Processor -- start 2024-03-06T07:23:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.367+0000 7ff1c1193640 1 -- start start 2024-03-06T07:23:39.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.367+0000 7ff1c1193640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1bc0d29d0 0x7ff1bc15f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.367+0000 7ff1c1193640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 0x7ff1bc0c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.367+0000 7ff1c1193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 0x7ff1bc0c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.367+0000 7ff1c1193640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0d2df0 con 0x7ff1bc0d4720 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.367+0000 7ff1c1193640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc15e170 con 0x7ff1bc0d29d0 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.367+0000 7ff1c1193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0b7250 con 0x7ff1bc15dd90 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.367+0000 7ff1ba59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 0x7ff1bc0c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1ba59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 0x7ff1bc0c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52644/0 (socket says 172.21.15.27:52644) 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1ba59c640 1 -- 172.21.15.27:0/3455215346 learned_addr learned my addr 172.21.15.27:0/3455215346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1bb59e640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 0x7ff1bc0c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1bad9d640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1bc0d29d0 0x7ff1bc15f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1ba59c640 1 -- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 msgr2=0x7ff1bc0c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1ba59c640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 0x7ff1bc0c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1ba59c640 1 -- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1bc0d29d0 msgr2=0x7ff1bc15f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1ba59c640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1bc0d29d0 0x7ff1bc15f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1ba59c640 1 -- 172.21.15.27:0/3455215346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc0c75c0 con 0x7ff1bc0d4720 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1ba59c640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 0x7ff1bc0c7080 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e810 tx=0x7ff1a400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1b37fe640 1 -- 172.21.15.27:0/3455215346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a400f040 con 0x7ff1bc0d4720 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1b37fe640 1 -- 172.21.15.27:0/3455215346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a4004030 con 0x7ff1bc0d4720 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc15e690 con 0x7ff1bc0d4720 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1b37fe640 1 -- 172.21.15.27:0/3455215346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4004400 con 0x7ff1bc0d4720 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.368+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc15ec50 con 0x7ff1bc0d4720 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.369+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc072a30 con 0x7ff1bc0d4720 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.370+0000 7ff1b37fe640 1 -- 172.21.15.27:0/3455215346 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1a400c6d0 con 0x7ff1bc0d4720 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.370+0000 7ff1b37fe640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff19c06b500 0x7ff19c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.370+0000 7ff1b37fe640 1 -- 172.21.15.27:0/3455215346 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7ff1a4089ff0 con 0x7ff1bc0d4720 2024-03-06T07:23:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.370+0000 7ff1bad9d640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff19c06b500 0x7ff19c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.371+0000 7ff1bad9d640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff19c06b500 0x7ff19c06d9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4006fd0 tx=0x7ff1b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.375+0000 7ff1b37fe640 1 -- 172.21.15.27:0/3455215346 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1a4017020 con 0x7ff1bc0d4720 2024-03-06T07:23:39.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:39.554+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff1bc159890 con 0x7ff1bc0d4720 2024-03-06T07:23:40.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.253+0000 7ff1b37fe640 1 -- 172.21.15.27:0/3455215346 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff1a4058640 con 0x7ff1bc0d4720 2024-03-06T07:23:40.255 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.257+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff19c06b500 msgr2=0x7ff19c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.257+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff19c06b500 0x7ff19c06d9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4006fd0 tx=0x7ff1b4008040 comp rx=0 tx=0).stop 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.257+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 msgr2=0x7ff1bc0c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.257+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 0x7ff1bc0c7080 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e810 tx=0x7ff1a400ecd0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.258+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 shutdown_connections 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.258+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff19c06b500 0x7ff19c06d9c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.258+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1bc15dd90 0x7ff1bc0c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.258+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1bc0d4720 0x7ff1bc0c7080 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.258+0000 7ff1c1193640 1 --2- 172.21.15.27:0/3455215346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1bc0d29d0 0x7ff1bc15f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.258+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 >> 172.21.15.27:0/3455215346 conn(0x7ff1bc071960 msgr2=0x7ff1bc0b6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.258+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 shutdown_connections 2024-03-06T07:23:40.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.258+0000 7ff1c1193640 1 -- 172.21.15.27:0/3455215346 wait complete. 2024-03-06T07:23:40.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:40.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.388+0000 7fc3442e1640 1 Processor -- start 2024-03-06T07:23:40.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.388+0000 7fc3442e1640 1 -- start start 2024-03-06T07:23:40.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.389+0000 7fc3442e1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc33c071540 0x7fc33c071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.389+0000 7fc3442e1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc33c0b2d30 0x7fc33c0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.389+0000 7fc3442e1640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c071e60 con 0x7fc33c0cecb0 2024-03-06T07:23:40.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.389+0000 7fc3442e1640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33c0d3230 con 0x7fc33c0b2d30 2024-03-06T07:23:40.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.389+0000 7fc3442e1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33c0d4960 con 0x7fc33c071540 2024-03-06T07:23:40.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.389+0000 7fc34187c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc33c0b2d30 0x7fc33c0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.389+0000 7fc34287e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc33c0cecb0 0x7fc33c0b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58602/0 (socket says 172.21.15.27:58602) 2024-03-06T07:23:40.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.389+0000 7fc34287e640 1 -- 172.21.15.27:0/1731378050 learned_addr learned my addr 172.21.15.27:0/1731378050 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:40.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.389+0000 7fc34207d640 1 --2- 172.21.15.27:0/1731378050 >> v2:172.21.15.69:3300/0 conn(0x7fc33c071540 0x7fc33c071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.390+0000 7fc34187c640 1 -- 172.21.15.27:0/1731378050 >> v2:172.21.15.69:3300/0 conn(0x7fc33c071540 msgr2=0x7fc33c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.390+0000 7fc34187c640 1 --2- 172.21.15.27:0/1731378050 >> v2:172.21.15.69:3300/0 conn(0x7fc33c071540 0x7fc33c071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.390+0000 7fc34187c640 1 -- 172.21.15.27:0/1731378050 >> v1:172.21.15.27:6789/0 conn(0x7fc33c0cecb0 legacy=0x7fc33c0b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.390+0000 7fc34187c640 1 -- 172.21.15.27:0/1731378050 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c0d36c0 con 0x7fc33c0b2d30 2024-03-06T07:23:40.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.390+0000 7fc34107b640 1 -- 172.21.15.27:0/1731378050 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870253521 0 0) 0x7fc33c071e60 con 0x7fc33c0cecb0 2024-03-06T07:23:40.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.390+0000 7fc34187c640 1 --2- 172.21.15.27:0/1731378050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c0b2d30 0x7fc33c0ce770 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc33c08ca00 tx=0x7fc33002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c694c8d589caaa66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc34107b640 1 -- 172.21.15.27:0/1731378050 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330035020 con 0x7fc33c0b2d30 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc34107b640 1 -- 172.21.15.27:0/1731378050 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc330031830 con 0x7fc33c0b2d30 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc34107b640 1 -- 172.21.15.27:0/1731378050 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330031c00 con 0x7fc33c0b2d30 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc3442e1640 1 -- 172.21.15.27:0/1731378050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c0b2d30 msgr2=0x7fc33c0ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc3442e1640 1 --2- 172.21.15.27:0/1731378050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c0b2d30 0x7fc33c0ce770 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc33c08ca00 tx=0x7fc33002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc3442e1640 1 -- 172.21.15.27:0/1731378050 shutdown_connections 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc3442e1640 1 --2- 172.21.15.27:0/1731378050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c0b2d30 0x7fc33c0ce770 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc3442e1640 1 --2- 172.21.15.27:0/1731378050 >> v2:172.21.15.69:3300/0 conn(0x7fc33c071540 0x7fc33c071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc3442e1640 1 -- 172.21.15.27:0/1731378050 >> 172.21.15.27:0/1731378050 conn(0x7fc33c0d3d60 msgr2=0x7fc33c0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc3442e1640 1 -- 172.21.15.27:0/1731378050 shutdown_connections 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.391+0000 7fc3442e1640 1 -- 172.21.15.27:0/1731378050 wait complete. 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.392+0000 7fc3442e1640 1 Processor -- start 2024-03-06T07:23:40.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.392+0000 7fc3442e1640 1 -- start start 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc3442e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c071540 0x7fc33c1f87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc3442e1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc33c0b2d30 0x7fc33c143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc3442e1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33c0cecb0 0x7fc33c1e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc3442e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c0d2be0 con 0x7fc33c071540 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc3442e1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c0d2900 con 0x7fc33c0b2d30 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc3442e1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c0d2ec0 con 0x7fc33c0cecb0 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc34207d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c071540 0x7fc33c1f87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc34207d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c071540 0x7fc33c1f87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52664/0 (socket says 172.21.15.27:52664) 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc34207d640 1 -- 172.21.15.27:0/2531380988 learned_addr learned my addr 172.21.15.27:0/2531380988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.393+0000 7fc34187c640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc33c0b2d30 0x7fc33c143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc34287e640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33c0cecb0 0x7fc33c1e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc34207d640 1 -- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33c0cecb0 msgr2=0x7fc33c1e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc34207d640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33c0cecb0 0x7fc33c1e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc34207d640 1 -- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc33c0b2d30 msgr2=0x7fc33c143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc34207d640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc33c0b2d30 0x7fc33c143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc34207d640 1 -- 172.21.15.27:0/2531380988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c0c85e0 con 0x7fc33c071540 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc34187c640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc33c0b2d30 0x7fc33c143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc34287e640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33c0cecb0 0x7fc33c1e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc34207d640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c071540 0x7fc33c1f87f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009d10 tx=0x7fc32c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc33affd640 1 -- 172.21.15.27:0/2531380988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c018020 con 0x7fc33c071540 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc33affd640 1 -- 172.21.15.27:0/2531380988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c014480 con 0x7fc33c071540 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33c0c88c0 con 0x7fc33c071540 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc33affd640 1 -- 172.21.15.27:0/2531380988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c014830 con 0x7fc33c071540 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.394+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33c143e50 con 0x7fc33c071540 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.395+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33c072070 con 0x7fc33c071540 2024-03-06T07:23:40.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.396+0000 7fc33affd640 1 -- 172.21.15.27:0/2531380988 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc32c005d10 con 0x7fc33c071540 2024-03-06T07:23:40.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.396+0000 7fc33affd640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc32806b4d0 0x7fc32806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.396+0000 7fc33affd640 1 -- 172.21.15.27:0/2531380988 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fc32c08ab00 con 0x7fc33c071540 2024-03-06T07:23:40.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.396+0000 7fc34187c640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc32806b4d0 0x7fc32806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.397+0000 7fc34187c640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc32806b4d0 0x7fc32806d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc330036000 tx=0x7fc330008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.400+0000 7fc33affd640 1 -- 172.21.15.27:0/2531380988 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc32c022070 con 0x7fc33c071540 2024-03-06T07:23:40.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:40.586+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc33c04f410 con 0x7fc33c071540 2024-03-06T07:23:41.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.258+0000 7fc33affd640 1 -- 172.21.15.27:0/2531380988 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc32c010040 con 0x7fc33c071540 2024-03-06T07:23:41.259 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc32806b4d0 msgr2=0x7fc32806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc32806b4d0 0x7fc32806d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc330036000 tx=0x7fc330008030 comp rx=0 tx=0).stop 2024-03-06T07:23:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c071540 msgr2=0x7fc33c1f87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c071540 0x7fc33c1f87f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009d10 tx=0x7fc32c00c490 comp rx=0 tx=0).stop 2024-03-06T07:23:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 shutdown_connections 2024-03-06T07:23:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc32806b4d0 0x7fc32806d990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33c0cecb0 0x7fc33c1e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc33c0b2d30 0x7fc33c143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 --2- 172.21.15.27:0/2531380988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33c071540 0x7fc33c1f87f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 >> 172.21.15.27:0/2531380988 conn(0x7fc33c0d3d60 msgr2=0x7fc33c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.262+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 shutdown_connections 2024-03-06T07:23:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:41.263+0000 7fc3442e1640 1 -- 172.21.15.27:0/2531380988 wait complete. 2024-03-06T07:23:41.283 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:41.283 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:41.283 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:41.283 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:41.283 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:41.283 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:41.284 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:41.284 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:41.284 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:41.284 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:41.284 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:41.284 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:41.284 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:41.284 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:41.285 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:41.285 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:41.295 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:41.295 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:41.297 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:41.297 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:41.297 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:41.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:41.338 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:41.338 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:41.339 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:41.339 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:41.339 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:41.340 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:41.426 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:23:41.428 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:23:41.428 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:23:42.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-06T07:23:47.295 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:47.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:47.299 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:47.299 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:47.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:47.392 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:23:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.419+0000 7f2fccc40640 1 Processor -- start 2024-03-06T07:23:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.419+0000 7f2fccc40640 1 -- start start 2024-03-06T07:23:47.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.419+0000 7f2fccc40640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2fc8071540 0x7f2fc80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.419+0000 7f2fccc40640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fc80d3400 0x7f2fc8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.419+0000 7f2fccc40640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc8071e60 con 0x7f2fc80b2b50 2024-03-06T07:23:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.419+0000 7f2fccc40640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc80d2f70 con 0x7f2fc80d3400 2024-03-06T07:23:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.419+0000 7f2fccc40640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc80d4be0 con 0x7f2fc8071540 2024-03-06T07:23:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.420+0000 7f2fbffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fc80d3400 0x7f2fc8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.420+0000 7f2fbffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fc80d3400 0x7f2fc8071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58342/0 (socket says 172.21.15.27:58342) 2024-03-06T07:23:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.420+0000 7f2fbffff640 1 -- 172.21.15.27:0/1276845081 learned_addr learned my addr 172.21.15.27:0/1276845081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.420+0000 7f2fc659c640 1 --2- 172.21.15.27:0/1276845081 >> v2:172.21.15.69:3300/0 conn(0x7f2fc8071540 0x7f2fc80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.420+0000 7f2fbffff640 1 -- 172.21.15.27:0/1276845081 >> v2:172.21.15.69:3300/0 conn(0x7f2fc8071540 msgr2=0x7f2fc80b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.420+0000 7f2fbffff640 1 --2- 172.21.15.27:0/1276845081 >> v2:172.21.15.69:3300/0 conn(0x7f2fc8071540 0x7f2fc80b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.420+0000 7f2fbffff640 1 -- 172.21.15.27:0/1276845081 >> v1:172.21.15.69:6789/0 conn(0x7f2fc80b2b50 legacy=0x7f2fc80d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.420+0000 7f2fbffff640 1 -- 172.21.15.27:0/1276845081 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80ce610 con 0x7f2fc80d3400 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.421+0000 7f2fbffff640 1 --2- 172.21.15.27:0/1276845081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80d3400 0x7f2fc8071920 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80d4da0 tx=0x7f2fb002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d82a419a1d7a81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.421+0000 7f2fc5d9b640 1 -- 172.21.15.27:0/1276845081 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0034070 con 0x7f2fc80d3400 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.421+0000 7f2fc5d9b640 1 -- 172.21.15.27:0/1276845081 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb0030860 con 0x7f2fc80d3400 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.421+0000 7f2fc5d9b640 1 -- 172.21.15.27:0/1276845081 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0030c30 con 0x7f2fc80d3400 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.422+0000 7f2fccc40640 1 -- 172.21.15.27:0/1276845081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80d3400 msgr2=0x7f2fc8071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.422+0000 7f2fccc40640 1 --2- 172.21.15.27:0/1276845081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80d3400 0x7f2fc8071920 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80d4da0 tx=0x7f2fb002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.422+0000 7f2fccc40640 1 -- 172.21.15.27:0/1276845081 shutdown_connections 2024-03-06T07:23:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.422+0000 7f2fccc40640 1 --2- 172.21.15.27:0/1276845081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80d3400 0x7f2fc8071920 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.422+0000 7f2fccc40640 1 --2- 172.21.15.27:0/1276845081 >> v2:172.21.15.69:3300/0 conn(0x7f2fc8071540 0x7f2fc80b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.422+0000 7f2fccc40640 1 -- 172.21.15.27:0/1276845081 >> 172.21.15.27:0/1276845081 conn(0x7f2fc80d3ca0 msgr2=0x7f2fc80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.422+0000 7f2fccc40640 1 -- 172.21.15.27:0/1276845081 shutdown_connections 2024-03-06T07:23:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.422+0000 7f2fccc40640 1 -- 172.21.15.27:0/1276845081 wait complete. 2024-03-06T07:23:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.423+0000 7f2fccc40640 1 Processor -- start 2024-03-06T07:23:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.423+0000 7f2fccc40640 1 -- start start 2024-03-06T07:23:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fccc40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc8071540 0x7f2fc8160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fccc40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80b2b50 0x7f2fc80c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fccc40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fc80d3400 0x7f2fc814cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fccc40640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80ce8f0 con 0x7f2fc80b2b50 2024-03-06T07:23:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fccc40640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80d2c00 con 0x7f2fc8071540 2024-03-06T07:23:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fccc40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc815e370 con 0x7f2fc80d3400 2024-03-06T07:23:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fbffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80b2b50 0x7f2fc80c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fbffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80b2b50 0x7f2fc80c8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58356/0 (socket says 172.21.15.27:58356) 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fbffff640 1 -- 172.21.15.27:0/4185329918 learned_addr learned my addr 172.21.15.27:0/4185329918 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fc659c640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc8071540 0x7f2fc8160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fc6d9d640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fc80d3400 0x7f2fc814cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fbffff640 1 -- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fc80d3400 msgr2=0x7f2fc814cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fbffff640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fc80d3400 0x7f2fc814cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fbffff640 1 -- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc8071540 msgr2=0x7f2fc8160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fbffff640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc8071540 0x7f2fc8160da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.424+0000 7f2fbffff640 1 -- 172.21.15.27:0/4185329918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc81f6fb0 con 0x7f2fc80b2b50 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.425+0000 7f2fbffff640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80b2b50 0x7f2fc80c8000 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2fb002b9e0 tx=0x7f2fb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.425+0000 7f2fbeffd640 1 -- 172.21.15.27:0/4185329918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0034040 con 0x7f2fc80b2b50 2024-03-06T07:23:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.425+0000 7f2fbeffd640 1 -- 172.21.15.27:0/4185329918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb0039d90 con 0x7f2fc80b2b50 2024-03-06T07:23:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.425+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc80c8a50 con 0x7f2fc80b2b50 2024-03-06T07:23:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.425+0000 7f2fbeffd640 1 -- 172.21.15.27:0/4185329918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0043610 con 0x7f2fc80b2b50 2024-03-06T07:23:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.425+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc81f7550 con 0x7f2fc80b2b50 2024-03-06T07:23:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.426+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc8068700 con 0x7f2fc80b2b50 2024-03-06T07:23:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.426+0000 7f2fbeffd640 1 -- 172.21.15.27:0/4185329918 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fb004a060 con 0x7f2fc80b2b50 2024-03-06T07:23:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.427+0000 7f2fbeffd640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fa806b4d0 0x7f2fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.427+0000 7f2fbeffd640 1 -- 172.21.15.27:0/4185329918 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2fb00abd60 con 0x7f2fc80b2b50 2024-03-06T07:23:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.427+0000 7f2fc659c640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fa806b4d0 0x7f2fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.428+0000 7f2fc659c640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fa806b4d0 0x7f2fa806d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009b40 tx=0x7f2fb8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.432+0000 7f2fbeffd640 1 -- 172.21.15.27:0/4185329918 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fb003e070 con 0x7f2fc80b2b50 2024-03-06T07:23:47.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:47.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:47.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:47.624+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2fc815d900 con 0x7f2fc80b2b50 2024-03-06T07:23:48.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-06T07:23:49.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.287+0000 7f2fbeffd640 1 -- 172.21.15.27:0/4185329918 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v31) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2fb007a920 con 0x7f2fc80b2b50 2024-03-06T07:23:49.289 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fa806b4d0 msgr2=0x7f2fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fa806b4d0 0x7f2fa806d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009b40 tx=0x7f2fb8009190 comp rx=0 tx=0).stop 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80b2b50 msgr2=0x7f2fc80c8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80b2b50 0x7f2fc80c8000 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2fb002b9e0 tx=0x7f2fb00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 shutdown_connections 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fa806b4d0 0x7f2fa806d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fc80d3400 0x7f2fc814cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fc80b2b50 0x7f2fc80c8000 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 --2- 172.21.15.27:0/4185329918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc8071540 0x7f2fc8160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 >> 172.21.15.27:0/4185329918 conn(0x7f2fc80d3ca0 msgr2=0x7f2fc80b7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 shutdown_connections 2024-03-06T07:23:49.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.290+0000 7f2fccc40640 1 -- 172.21.15.27:0/4185329918 wait complete. 2024-03-06T07:23:49.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.419+0000 7fdfe3fc9640 1 Processor -- start 2024-03-06T07:23:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.419+0000 7fdfe3fc9640 1 -- start start 2024-03-06T07:23:49.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe3fc9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdfdc072660 0x7fdfdc15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe3fc9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfdc0b4160 con 0x7fdfdc0b55d0 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe3fc9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfdc0b5b40 con 0x7fdfdc06e700 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe3fc9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc0b3ba0 con 0x7fdfdc072660 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe2566640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdfdc0b55d0 0x7fdfdc071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50436/0 (socket says 172.21.15.27:50436) 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe2566640 1 -- 172.21.15.27:0/2671977964 learned_addr learned my addr 172.21.15.27:0/2671977964 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe1564640 1 --2- 172.21.15.27:0/2671977964 >> v2:172.21.15.69:3300/0 conn(0x7fdfdc072660 0x7fdfdc15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165669530 0 0) 0x7fdfdc0b4160 con 0x7fdfdc0b55d0 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc8003520 con 0x7fdfdc0b55d0 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3384746323 0 0) 0x7fdfc8003520 con 0x7fdfdc0b55d0 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 >> v2:172.21.15.69:3300/0 conn(0x7fdfdc072660 msgr2=0x7fdfdc15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.420+0000 7fdfe0d63640 1 --2- 172.21.15.27:0/2671977964 >> v2:172.21.15.69:3300/0 conn(0x7fdfdc072660 0x7fdfdc15ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 >> v1:172.21.15.69:6789/0 conn(0x7fdfdc06e700 legacy=0x7fdfdc06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc0b3e80 con 0x7fdfdc0b55d0 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fdfd40019f0 con 0x7fdfdc0b55d0 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 >> v1:172.21.15.27:6789/0 conn(0x7fdfdc0b55d0 legacy=0x7fdfdc071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfc80039b0 0x7fdfc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc8004800 0x7fdfc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc80051c0 0x7fdfc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc0b3ba0 con 0x7fdfc80051c0 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc800c060 con 0x7fdfc8004800 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc800c260 con 0x7fdfc80039b0 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe2566640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc80051c0 0x7fdfc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe1564640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc8004800 0x7fdfc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe1d65640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfc80039b0 0x7fdfc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe2566640 1 -- 172.21.15.27:0/2671977964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfc80039b0 msgr2=0x7fdfc8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe2566640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfc80039b0 0x7fdfc8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe2566640 1 -- 172.21.15.27:0/2671977964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc8004800 msgr2=0x7fdfc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe2566640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc8004800 0x7fdfc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe2566640 1 -- 172.21.15.27:0/2671977964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc8003520 con 0x7fdfc80051c0 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.421+0000 7fdfe1564640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc8004800 0x7fdfc8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.422+0000 7fdfe2566640 1 -- 172.21.15.27:0/2671977964 reap_dead start 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.422+0000 7fdfe2566640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc80051c0 0x7fdfc800b9c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400a860 tx=0x7fdfc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85179ca48dfdc91d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.422+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc009690 con 0x7fdfc80051c0 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.422+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd4034630 con 0x7fdfc80051c0 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.422+0000 7fdfe0d63640 1 -- 172.21.15.27:0/2671977964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd400a480 con 0x7fdfc80051c0 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.422+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/2671977964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc80051c0 msgr2=0x7fdfc800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.422+0000 7fdfe3fc9640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc80051c0 0x7fdfc800b9c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400a860 tx=0x7fdfc8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:49.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.423+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/2671977964 shutdown_connections 2024-03-06T07:23:49.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.423+0000 7fdfe3fc9640 1 --2- 172.21.15.27:0/2671977964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc80051c0 0x7fdfc800b9c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.423+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/2671977964 >> 172.21.15.27:0/2671977964 conn(0x7fdfdc0b2720 msgr2=0x7fdfdc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.423+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/2671977964 shutdown_connections 2024-03-06T07:23:49.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.423+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/2671977964 wait complete. 2024-03-06T07:23:49.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.424+0000 7fdfe3fc9640 1 Processor -- start 2024-03-06T07:23:49.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.424+0000 7fdfe3fc9640 1 -- start start 2024-03-06T07:23:49.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe3fc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc80051c0 0x7fdfdc15b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe3fc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc0cee80 0x7fdfdc0cf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe3fc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfdc0b0250 0x7fdfdc0b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe3fc9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc15f100 con 0x7fdfdc0cee80 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe3fc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc15ef80 con 0x7fdfc80051c0 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe3fc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc15f280 con 0x7fdfdc0b0250 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1564640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc0cee80 0x7fdfdc0cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1564640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc0cee80 0x7fdfdc0cf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58374/0 (socket says 172.21.15.27:58374) 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1564640 1 -- 172.21.15.27:0/256480263 learned_addr learned my addr 172.21.15.27:0/256480263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1d65640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc80051c0 0x7fdfdc15b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe2566640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfdc0b0250 0x7fdfdc0b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1564640 1 -- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfdc0b0250 msgr2=0x7fdfdc0b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1564640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfdc0b0250 0x7fdfdc0b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1564640 1 -- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc80051c0 msgr2=0x7fdfdc15b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1564640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc80051c0 0x7fdfdc15b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1564640 1 -- 172.21.15.27:0/256480263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc0cf920 con 0x7fdfdc0cee80 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe1564640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc0cee80 0x7fdfdc0cf300 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0119c0 tx=0x7fdfcc011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfd27fc640 1 -- 172.21.15.27:0/256480263 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00fa90 con 0x7fdfdc0cee80 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfd27fc640 1 -- 172.21.15.27:0/256480263 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc00fc30 con 0x7fdfdc0cee80 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc0cfc00 con 0x7fdfdc0cee80 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfd27fc640 1 -- 172.21.15.27:0/256480263 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc014440 con 0x7fdfdc0cee80 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.425+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc0adb60 con 0x7fdfdc0cee80 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.426+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc067ff0 con 0x7fdfdc0cee80 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.427+0000 7fdfd27fc640 1 -- 172.21.15.27:0/256480263 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfcc012040 con 0x7fdfdc0cee80 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.427+0000 7fdfd27fc640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdfc006b500 0x7fdfc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.427+0000 7fdfd27fc640 1 -- 172.21.15.27:0/256480263 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdfcc08a3e0 con 0x7fdfdc0cee80 2024-03-06T07:23:49.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.428+0000 7fdfe1d65640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdfc006b500 0x7fdfc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.428+0000 7fdfe1d65640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdfc006b500 0x7fdfc006d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8000c00 tx=0x7fdfd8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.432+0000 7fdfd27fc640 1 -- 172.21.15.27:0/256480263 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfcc019030 con 0x7fdfdc0cee80 2024-03-06T07:23:49.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.685+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fdfdc072c30 con 0x7fdfdc0cee80 2024-03-06T07:23:49.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.685+0000 7fdfd27fc640 1 -- 172.21.15.27:0/256480263 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fdfcc059670 con 0x7fdfdc0cee80 2024-03-06T07:23:49.687 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:49.687 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:23:49.687 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g==", 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.688 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ==", 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg==", 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:23:49.689 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ==", 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw==", 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA==", 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.691 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA==", 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg==", 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A==", 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:49.693 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:23:49.694 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.694 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.694 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-03-06T07:23:49.694 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw==", 2024-03-06T07:23:49.694 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.694 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:49.694 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA==", 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A==", 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA==", 2024-03-06T07:23:49.696 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ==", 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:23:49.697 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ==", 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:23:49.698 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.699 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-03-06T07:23:49.699 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg==", 2024-03-06T07:23:49.699 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlkc2iEhAA6BuYba0FzXeVjGPOP8sgyQ==", 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:23:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCvGehlnH8kLBAAD+B/DC3pqHp0oTABYWKHaA==", 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:23:49.701 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCvGehlQ4kkLBAAppSpnfDCJtKcTg6ebciyxw==", 2024-03-06T07:23:49.702 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.702 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:23:49.702 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.702 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:23:49.702 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCvGehlXZAkLBAA6YWn/YdMVTc0Cbe0nE5blQ==", 2024-03-06T07:23:49.702 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.702 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCvGehlipckLBAAVC7vOSgLGKEVY5clDgGOgg==", 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCvGehl6p4kLBAAJZXg9UjCPYmWKPxvbCUQ+A==", 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCvGehlRaYkLBAAJYmP84qFgZZGLd7YJmsJrg==", 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw==", 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg==", 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:49.706 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:23:49.707 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:23:49.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdfc006b500 msgr2=0x7fdfc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdfc006b500 0x7fdfc006d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8000c00 tx=0x7fdfd8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:49.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc0cee80 msgr2=0x7fdfdc0cf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc0cee80 0x7fdfdc0cf300 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0119c0 tx=0x7fdfcc011e80 comp rx=0 tx=0).stop 2024-03-06T07:23:49.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 shutdown_connections 2024-03-06T07:23:49.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdfc006b500 0x7fdfc006d9c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfdc0b0250 0x7fdfdc0b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfdc0cee80 0x7fdfdc0cf300 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 --2- 172.21.15.27:0/256480263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc80051c0 0x7fdfdc15b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 >> 172.21.15.27:0/256480263 conn(0x7fdfdc0b2720 msgr2=0x7fdfdc160ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 shutdown_connections 2024-03-06T07:23:49.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.687+0000 7fdfe3fc9640 1 -- 172.21.15.27:0/256480263 wait complete. 2024-03-06T07:23:49.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:23:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.818+0000 7f481e5e9640 1 Processor -- start 2024-03-06T07:23:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.818+0000 7f481e5e9640 1 -- start start 2024-03-06T07:23:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.819+0000 7f481e5e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48180d18c0 0x7f4818071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.819+0000 7f481e5e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180b6f20 con 0x7f4818071a80 2024-03-06T07:23:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.819+0000 7f481e5e9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180d3030 con 0x7f481815df70 2024-03-06T07:23:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.819+0000 7f481e5e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4818071eb0 con 0x7f48180d18c0 2024-03-06T07:23:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.819+0000 7f481cb86640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4818071a80 0x7f48180b2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50446/0 (socket says 172.21.15.27:50446) 2024-03-06T07:23:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.819+0000 7f481cb86640 1 -- 172.21.15.27:0/2309436015 learned_addr learned my addr 172.21.15.27:0/2309436015 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.819+0000 7f48177fe640 1 --2- 172.21.15.27:0/2309436015 >> v2:172.21.15.69:3300/0 conn(0x7f48180d18c0 0x7f4818071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.820+0000 7f4816ffd640 1 -- 172.21.15.27:0/2309436015 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179597761 0 0) 0x7f48180b6f20 con 0x7f4818071a80 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.820+0000 7f4816ffd640 1 -- 172.21.15.27:0/2309436015 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800003520 con 0x7f4818071a80 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.820+0000 7f48177fe640 1 -- 172.21.15.27:0/2309436015 >> v1:172.21.15.69:6789/0 conn(0x7f481815df70 legacy=0x7f48180d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.820+0000 7f48177fe640 1 -- 172.21.15.27:0/2309436015 >> v1:172.21.15.27:6789/0 conn(0x7f4818071a80 legacy=0x7f48180b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.820+0000 7f48177fe640 1 -- 172.21.15.27:0/2309436015 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48180d3310 con 0x7f48180d18c0 2024-03-06T07:23:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f48177fe640 1 --2- 172.21.15.27:0/2309436015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48180d18c0 0x7f4818071540 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48080097a0 tx=0x7f480802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b292f66e52988bce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f4816ffd640 1 -- 172.21.15.27:0/2309436015 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480802f810 con 0x7f48180d18c0 2024-03-06T07:23:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f4816ffd640 1 -- 172.21.15.27:0/2309436015 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480802f9b0 con 0x7f48180d18c0 2024-03-06T07:23:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f4816ffd640 1 -- 172.21.15.27:0/2309436015 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480802fd60 con 0x7f48180d18c0 2024-03-06T07:23:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f481e5e9640 1 -- 172.21.15.27:0/2309436015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48180d18c0 msgr2=0x7f4818071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f481e5e9640 1 --2- 172.21.15.27:0/2309436015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48180d18c0 0x7f4818071540 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48080097a0 tx=0x7f480802b9f0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f481e5e9640 1 -- 172.21.15.27:0/2309436015 shutdown_connections 2024-03-06T07:23:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f481e5e9640 1 --2- 172.21.15.27:0/2309436015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48180d18c0 0x7f4818071540 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f481e5e9640 1 -- 172.21.15.27:0/2309436015 >> 172.21.15.27:0/2309436015 conn(0x7f48180d3cf0 msgr2=0x7f48180d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f481e5e9640 1 -- 172.21.15.27:0/2309436015 shutdown_connections 2024-03-06T07:23:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.821+0000 7f481e5e9640 1 -- 172.21.15.27:0/2309436015 wait complete. 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.822+0000 7f481e5e9640 1 Processor -- start 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481e5e9640 1 -- start start 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481e5e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4818071a80 0x7f481814e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481e5e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48180d18c0 0x7f481814bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481e5e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481815df70 0x7f481814c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481e5e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481815e350 con 0x7f481815df70 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481e5e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48180d38d0 con 0x7f48180d18c0 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481e5e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481815e4d0 con 0x7f4818071a80 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481cb86640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481815df70 0x7f481814c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481cb86640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481815df70 0x7f481814c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58388/0 (socket says 172.21.15.27:58388) 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.823+0000 7f481cb86640 1 -- 172.21.15.27:0/243102536 learned_addr learned my addr 172.21.15.27:0/243102536 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f48177fe640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48180d18c0 0x7f481814bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f4817fff640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4818071a80 0x7f481814e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f481cb86640 1 -- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4818071a80 msgr2=0x7f481814e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f481cb86640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4818071a80 0x7f481814e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f481cb86640 1 -- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48180d18c0 msgr2=0x7f481814bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f481cb86640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48180d18c0 0x7f481814bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f481cb86640 1 -- 172.21.15.27:0/243102536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481814c320 con 0x7f481815df70 2024-03-06T07:23:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f4817fff640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4818071a80 0x7f481814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f481cb86640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481815df70 0x7f481814c5f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f481000cb40 tx=0x7f481000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f4814ff9640 1 -- 172.21.15.27:0/243102536 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810017070 con 0x7f481815df70 2024-03-06T07:23:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f4814ff9640 1 -- 172.21.15.27:0/243102536 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481000aca0 con 0x7f481815df70 2024-03-06T07:23:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f4814ff9640 1 -- 172.21.15.27:0/243102536 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810012620 con 0x7f481815df70 2024-03-06T07:23:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481814cfd0 con 0x7f481815df70 2024-03-06T07:23:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.824+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481814d590 con 0x7f481815df70 2024-03-06T07:23:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.825+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4818067ff0 con 0x7f481815df70 2024-03-06T07:23:49.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.826+0000 7f4814ff9640 1 -- 172.21.15.27:0/243102536 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f481001a070 con 0x7f481815df70 2024-03-06T07:23:49.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.826+0000 7f4814ff9640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480406b500 0x7f480406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.826+0000 7f4814ff9640 1 -- 172.21.15.27:0/243102536 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4810089e20 con 0x7f481815df70 2024-03-06T07:23:49.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.826+0000 7f4817fff640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480406b500 0x7f480406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.827+0000 7f4817fff640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480406b500 0x7f480406d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f480c0015e0 tx=0x7f480c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:49.830+0000 7f4814ff9640 1 -- 172.21.15.27:0/243102536 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4810058ee0 con 0x7f481815df70 2024-03-06T07:23:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.080+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f481814ee40 con 0x7f481815df70 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.082+0000 7f4814ff9640 1 -- 172.21.15.27:0/243102536 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f48100587e0 con 0x7f481815df70 2024-03-06T07:23:50.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480406b500 msgr2=0x7f480406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480406b500 0x7f480406d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f480c0015e0 tx=0x7f480c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:50.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481815df70 msgr2=0x7f481814c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481815df70 0x7f481814c5f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f481000cb40 tx=0x7f481000a060 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 shutdown_connections 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480406b500 0x7f480406d9c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481815df70 0x7f481814c5f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48180d18c0 0x7f481814bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 --2- 172.21.15.27:0/243102536 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4818071a80 0x7f481814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 >> 172.21.15.27:0/243102536 conn(0x7f48180d3cf0 msgr2=0x7f48180b6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 shutdown_connections 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.084+0000 7f481e5e9640 1 -- 172.21.15.27:0/243102536 wait complete. 2024-03-06T07:23:50.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:23:50.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.203+0000 7fddecefb640 1 Processor -- start 2024-03-06T07:23:50.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.203+0000 7fddecefb640 1 -- start start 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fddecefb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdde8157b80 0x7fdde815b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fddecefb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdde80cbab0 0x7fdde8070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fddecefb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde806c7b0 con 0x7fdde8157f60 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fddecefb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde806ca90 con 0x7fdde80cbab0 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fddecefb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde806cd70 con 0x7fdde8157b80 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fdde6d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdde80cbab0 0x7fdde8070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fdde6d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdde80cbab0 0x7fdde8070b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58394/0 (socket says 172.21.15.27:58394) 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fdde6d9d640 1 -- 172.21.15.27:0/2533669092 learned_addr learned my addr 172.21.15.27:0/2533669092 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fdde659c640 1 --2- 172.21.15.27:0/2533669092 >> v2:172.21.15.69:3300/0 conn(0x7fdde8157b80 0x7fdde815b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fdde6d9d640 1 -- 172.21.15.27:0/2533669092 >> v2:172.21.15.69:3300/0 conn(0x7fdde8157b80 msgr2=0x7fdde815b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fdde6d9d640 1 --2- 172.21.15.27:0/2533669092 >> v2:172.21.15.69:3300/0 conn(0x7fdde8157b80 0x7fdde815b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fdde6d9d640 1 -- 172.21.15.27:0/2533669092 >> v1:172.21.15.69:6789/0 conn(0x7fdde8157f60 legacy=0x7fdde815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.204+0000 7fdde6d9d640 1 -- 172.21.15.27:0/2533669092 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde806d050 con 0x7fdde80cbab0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.205+0000 7fdde6d9d640 1 --2- 172.21.15.27:0/2533669092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde80cbab0 0x7fdde8070b40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdde806ed60 tx=0x7fdde002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ea5e4f1c04110d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.205+0000 7fdde5d9b640 1 -- 172.21.15.27:0/2533669092 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde002ca80 con 0x7fdde80cbab0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.205+0000 7fdde5d9b640 1 -- 172.21.15.27:0/2533669092 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde002cc20 con 0x7fdde80cbab0 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.205+0000 7fdde5d9b640 1 -- 172.21.15.27:0/2533669092 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00343f0 con 0x7fdde80cbab0 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.206+0000 7fddecefb640 1 -- 172.21.15.27:0/2533669092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde80cbab0 msgr2=0x7fdde8070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.206+0000 7fddecefb640 1 --2- 172.21.15.27:0/2533669092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde80cbab0 0x7fdde8070b40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdde806ed60 tx=0x7fdde002bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.206+0000 7fddecefb640 1 -- 172.21.15.27:0/2533669092 shutdown_connections 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.206+0000 7fddecefb640 1 --2- 172.21.15.27:0/2533669092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde80cbab0 0x7fdde8070b40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.206+0000 7fddecefb640 1 --2- 172.21.15.27:0/2533669092 >> v2:172.21.15.69:3300/0 conn(0x7fdde8157b80 0x7fdde815b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.206+0000 7fddecefb640 1 -- 172.21.15.27:0/2533669092 >> 172.21.15.27:0/2533669092 conn(0x7fdde8066d00 msgr2=0x7fdde80670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.206+0000 7fddecefb640 1 -- 172.21.15.27:0/2533669092 shutdown_connections 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.206+0000 7fddecefb640 1 -- 172.21.15.27:0/2533669092 wait complete. 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.207+0000 7fddecefb640 1 Processor -- start 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.207+0000 7fddecefb640 1 -- start start 2024-03-06T07:23:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fddecefb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde80cbab0 0x7fdde8156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fddecefb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde8157b80 0x7fdde8150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fddecefb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde8157f60 0x7fdde812e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fddecefb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8071080 con 0x7fdde8157f60 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fddecefb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde80cc260 con 0x7fdde8157b80 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fddecefb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8071200 con 0x7fdde80cbab0 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde8157f60 0x7fdde812e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde8157f60 0x7fdde812e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58404/0 (socket says 172.21.15.27:58404) 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde6d9d640 1 -- 172.21.15.27:0/659320492 learned_addr learned my addr 172.21.15.27:0/659320492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdddffff640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde8157b80 0x7fdde8150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde659c640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde80cbab0 0x7fdde8156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde6d9d640 1 -- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde80cbab0 msgr2=0x7fdde8156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde6d9d640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde80cbab0 0x7fdde8156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde6d9d640 1 -- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde8157b80 msgr2=0x7fdde8150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde6d9d640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde8157b80 0x7fdde8150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde6d9d640 1 -- 172.21.15.27:0/659320492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde814aa80 con 0x7fdde8157f60 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdddffff640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde8157b80 0x7fdde8150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.208+0000 7fdde6d9d640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde8157f60 0x7fdde812e930 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009f40 tx=0x7fdde0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.209+0000 7fdddeffd640 1 -- 172.21.15.27:0/659320492 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0043020 con 0x7fdde8157f60 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.209+0000 7fdddeffd640 1 -- 172.21.15.27:0/659320492 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde003e450 con 0x7fdde8157f60 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.209+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde814ad60 con 0x7fdde8157f60 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.209+0000 7fdddeffd640 1 -- 172.21.15.27:0/659320492 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0009510 con 0x7fdde8157f60 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.209+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde8155f00 con 0x7fdde8157f60 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.210+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde8067ff0 con 0x7fdde8157f60 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.210+0000 7fdddeffd640 1 -- 172.21.15.27:0/659320492 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdde0046020 con 0x7fdde8157f60 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.211+0000 7fdddeffd640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddc806b4d0 0x7fddc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.211+0000 7fdddeffd640 1 -- 172.21.15.27:0/659320492 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdde00ac730 con 0x7fdde8157f60 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.211+0000 7fdde659c640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddc806b4d0 0x7fddc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.211+0000 7fdde659c640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddc806b4d0 0x7fddc806d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fddd8006fd0 tx=0x7fddd8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.215+0000 7fdddeffd640 1 -- 172.21.15.27:0/659320492 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdde004d030 con 0x7fdde8157f60 2024-03-06T07:23:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.466+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fdde80d07a0 con 0x7fdde8157f60 2024-03-06T07:23:50.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.468+0000 7fdddeffd640 1 -- 172.21.15.27:0/659320492 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7fdde007b900 con 0x7fdde8157f60 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddc806b4d0 msgr2=0x7fddc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddc806b4d0 0x7fddc806d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fddd8006fd0 tx=0x7fddd8008040 comp rx=0 tx=0).stop 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde8157f60 msgr2=0x7fdde812e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde8157f60 0x7fdde812e930 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009f40 tx=0x7fdde0004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 shutdown_connections 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddc806b4d0 0x7fddc806d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde8157f60 0x7fdde812e930 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde8157b80 0x7fdde8150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 --2- 172.21.15.27:0/659320492 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde80cbab0 0x7fdde8156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.470+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 >> 172.21.15.27:0/659320492 conn(0x7fdde8066d00 msgr2=0x7fdde8158400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.471+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 shutdown_connections 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.471+0000 7fddecefb640 1 -- 172.21.15.27:0/659320492 wait complete. 2024-03-06T07:23:50.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:23:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.590+0000 7fad0be88640 1 Processor -- start 2024-03-06T07:23:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.590+0000 7fad0be88640 1 -- start start 2024-03-06T07:23:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad0be88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 0x7fad0406af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad0be88640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fad040cd890 0x7fad0415d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad0be88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad040cc0e0 con 0x7fad040b1020 2024-03-06T07:23:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad0be88640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad040d28b0 con 0x7fad040cd890 2024-03-06T07:23:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad0be88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad040cf320 con 0x7fad0406ed10 2024-03-06T07:23:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad0a425640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fad040cd890 0x7fad0415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad0a425640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fad040cd890 0x7fad0415d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:43724/0 (socket says 172.21.15.27:43724) 2024-03-06T07:23:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad09c24640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fad040b1020 0x7fad040cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:39432/0 (socket says 172.21.15.27:39432) 2024-03-06T07:23:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad0a425640 1 -- 172.21.15.27:0/1921835626 learned_addr learned my addr 172.21.15.27:0/1921835626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.591+0000 7fad09423640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 0x7fad0406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.592+0000 7fad09423640 1 -- 172.21.15.27:0/1921835626 >> v2:172.21.15.69:3300/0 conn(0x7fad040cd890 msgr2=0x7fad0415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.592+0000 7fad09423640 1 --2- 172.21.15.27:0/1921835626 >> v2:172.21.15.69:3300/0 conn(0x7fad040cd890 0x7fad0415d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.592+0000 7fad09423640 1 -- 172.21.15.27:0/1921835626 >> v1:172.21.15.69:6789/0 conn(0x7fad040b1020 legacy=0x7fad040cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.592+0000 7fad09423640 1 -- 172.21.15.27:0/1921835626 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad040cc3c0 con 0x7fad0406ed10 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.592+0000 7fad08c22640 1 -- 172.21.15.27:0/1921835626 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499849499 0 0) 0x7fad040cc0e0 con 0x7fad040b1020 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.592+0000 7fad0a425640 1 --2- 172.21.15.27:0/1921835626 >> v2:172.21.15.69:3300/0 conn(0x7fad040cd890 0x7fad0415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad09423640 1 --2- 172.21.15.27:0/1921835626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 0x7fad0406af20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7facf40097a0 tx=0x7facf402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c1d578a64516b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad08c22640 1 -- 172.21.15.27:0/1921835626 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4035020 con 0x7fad0406ed10 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad08c22640 1 -- 172.21.15.27:0/1921835626 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf4031830 con 0x7fad0406ed10 2024-03-06T07:23:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad08c22640 1 -- 172.21.15.27:0/1921835626 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4031c00 con 0x7fad0406ed10 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad0be88640 1 -- 172.21.15.27:0/1921835626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 msgr2=0x7fad0406af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad0be88640 1 --2- 172.21.15.27:0/1921835626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 0x7fad0406af20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7facf40097a0 tx=0x7facf402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad0be88640 1 -- 172.21.15.27:0/1921835626 shutdown_connections 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad0be88640 1 --2- 172.21.15.27:0/1921835626 >> v2:172.21.15.69:3300/0 conn(0x7fad040cd890 0x7fad0415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad0be88640 1 --2- 172.21.15.27:0/1921835626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 0x7fad0406af20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.593+0000 7fad0be88640 1 -- 172.21.15.27:0/1921835626 >> 172.21.15.27:0/1921835626 conn(0x7fad040d1a90 msgr2=0x7fad040d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.594+0000 7fad0be88640 1 -- 172.21.15.27:0/1921835626 shutdown_connections 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.594+0000 7fad0be88640 1 -- 172.21.15.27:0/1921835626 wait complete. 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.595+0000 7fad0be88640 1 Processor -- start 2024-03-06T07:23:50.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.595+0000 7fad0be88640 1 -- start start 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.595+0000 7fad0be88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 0x7fad040b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.595+0000 7fad0be88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad040b1020 0x7fad040c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.595+0000 7fad0be88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad040cd890 0x7fad040c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.595+0000 7fad0be88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad040ccc60 con 0x7fad040b1020 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.595+0000 7fad0be88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad040cc980 con 0x7fad040cd890 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.595+0000 7fad0be88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad040cdce0 con 0x7fad0406ed10 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.595+0000 7fad09423640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad040b1020 0x7fad040c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad09423640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad040b1020 0x7fad040c7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58420/0 (socket says 172.21.15.27:58420) 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad09423640 1 -- 172.21.15.27:0/2363061206 learned_addr learned my addr 172.21.15.27:0/2363061206 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad0a425640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad040cd890 0x7fad040c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad09c24640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 0x7fad040b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad09423640 1 -- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 msgr2=0x7fad040b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad09423640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 0x7fad040b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad09423640 1 -- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad040cd890 msgr2=0x7fad040c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad09423640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad040cd890 0x7fad040c7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad09423640 1 -- 172.21.15.27:0/2363061206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0414eb70 con 0x7fad040b1020 2024-03-06T07:23:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad09423640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad040b1020 0x7fad040c7900 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7facf4000c00 tx=0x7facf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7facfa7fc640 1 -- 172.21.15.27:0/2363061206 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4035070 con 0x7fad040b1020 2024-03-06T07:23:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7facfa7fc640 1 -- 172.21.15.27:0/2363061206 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf4004230 con 0x7fad040b1020 2024-03-06T07:23:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad040c84a0 con 0x7fad040b1020 2024-03-06T07:23:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7facfa7fc640 1 -- 172.21.15.27:0/2363061206 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf403a630 con 0x7fad040b1020 2024-03-06T07:23:50.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.596+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad04159820 con 0x7fad040b1020 2024-03-06T07:23:50.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.597+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad04161020 con 0x7fad040b1020 2024-03-06T07:23:50.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.598+0000 7facfa7fc640 1 -- 172.21.15.27:0/2363061206 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7facf4038020 con 0x7fad040b1020 2024-03-06T07:23:50.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.598+0000 7facfa7fc640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7face806b4d0 0x7face806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.598+0000 7facfa7fc640 1 -- 172.21.15.27:0/2363061206 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7facf40abb70 con 0x7fad040b1020 2024-03-06T07:23:50.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.598+0000 7fad09c24640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7face806b4d0 0x7face806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.599+0000 7fad09c24640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7face806b4d0 0x7face806d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fad00000f60 tx=0x7fad00007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.602+0000 7facfa7fc640 1 -- 172.21.15.27:0/2363061206 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad04161020 con 0x7fad040b1020 2024-03-06T07:23:50.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.854+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fad04067ff0 con 0x7fad040b1020 2024-03-06T07:23:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.856+0000 7facfa7fc640 1 -- 172.21.15.27:0/2363061206 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7facf4040070 con 0x7fad040b1020 2024-03-06T07:23:50.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7face806b4d0 msgr2=0x7face806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7face806b4d0 0x7face806d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fad00000f60 tx=0x7fad00007b50 comp rx=0 tx=0).stop 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad040b1020 msgr2=0x7fad040c7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad040b1020 0x7fad040c7900 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7facf4000c00 tx=0x7facf40046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 shutdown_connections 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7face806b4d0 0x7face806d990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad040cd890 0x7fad040c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad040b1020 0x7fad040c7900 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 --2- 172.21.15.27:0/2363061206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad0406ed10 0x7fad040b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 >> 172.21.15.27:0/2363061206 conn(0x7fad040d1a90 msgr2=0x7fad040d1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 shutdown_connections 2024-03-06T07:23:50.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.858+0000 7fad0be88640 1 -- 172.21.15.27:0/2363061206 wait complete. 2024-03-06T07:23:50.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:23:50.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.976+0000 7faafed1e640 1 Processor -- start 2024-03-06T07:23:50.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.976+0000 7faafed1e640 1 -- start start 2024-03-06T07:23:50.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.977+0000 7faafed1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf8147930 0x7faaf8072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.977+0000 7faafed1e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf8072a70 con 0x7faaf80adfc0 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.977+0000 7faafed1e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf80ae450 con 0x7faaf8161ac0 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.977+0000 7faafed1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf806dbe0 con 0x7faaf8147930 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.977+0000 7faafd2bb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faaf80adfc0 0x7faaf8168df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50450/0 (socket says 172.21.15.27:50450) 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.977+0000 7faafd2bb640 1 -- 172.21.15.27:0/632607053 learned_addr learned my addr 172.21.15.27:0/632607053 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.977+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158125122 0 0) 0x7faaf8072a70 con 0x7faaf80adfc0 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.977+0000 7faafcaba640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf8147930 0x7faaf8072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.977+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae0003520 con 0x7faaf80adfc0 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3679667027 0 0) 0x7faae0003520 con 0x7faaf80adfc0 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf8147930 msgr2=0x7faaf8072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf8147930 0x7faaf8072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 >> v1:172.21.15.69:6789/0 conn(0x7faaf8161ac0 legacy=0x7faaf80ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf8071a50 con 0x7faaf80adfc0 2024-03-06T07:23:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7faae8001670 con 0x7faaf80adfc0 2024-03-06T07:23:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 >> v1:172.21.15.27:6789/0 conn(0x7faaf80adfc0 legacy=0x7faaf8168df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf8147930 0x7faae00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faae0003f20 0x7faae0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae0004d40 0x7faae00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf806dbe0 con 0x7faae0004d40 2024-03-06T07:23:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae000ba20 con 0x7faae0003f20 2024-03-06T07:23:50.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae000bc20 con 0x7faaf8147930 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faafd2bb640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae0004d40 0x7faae00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faafcaba640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf8147930 0x7faae00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faaf7fff640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faae0003f20 0x7faae0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faafd2bb640 1 -- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf8147930 msgr2=0x7faae00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faafd2bb640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf8147930 0x7faae00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faafd2bb640 1 -- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faae0003f20 msgr2=0x7faae0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faafd2bb640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faae0003f20 0x7faae0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.978+0000 7faafd2bb640 1 -- 172.21.15.27:0/632607053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae0003520 con 0x7faae0004d40 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.979+0000 7faafd2bb640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae0004d40 0x7faae00051c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7faae800c240 tx=0x7faae0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aad180ea97ea05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.979+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8007d30 con 0x7faae0004d40 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.979+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae8030070 con 0x7faae0004d40 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.979+0000 7faaf77fe640 1 -- 172.21.15.27:0/632607053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae802cd80 con 0x7faae0004d40 2024-03-06T07:23:50.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.980+0000 7faafed1e640 1 -- 172.21.15.27:0/632607053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae0004d40 msgr2=0x7faae00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.980+0000 7faafed1e640 1 --2- 172.21.15.27:0/632607053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae0004d40 0x7faae00051c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7faae800c240 tx=0x7faae0005830 comp rx=0 tx=0).stop 2024-03-06T07:23:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.980+0000 7faafd2bb640 1 -- 172.21.15.27:0/632607053 reap_dead start 2024-03-06T07:23:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.980+0000 7faafed1e640 1 -- 172.21.15.27:0/632607053 shutdown_connections 2024-03-06T07:23:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.980+0000 7faafed1e640 1 -- 172.21.15.27:0/632607053 >> 172.21.15.27:0/632607053 conn(0x7faaf806cd20 msgr2=0x7faaf806d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.980+0000 7faafed1e640 1 -- 172.21.15.27:0/632607053 shutdown_connections 2024-03-06T07:23:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.980+0000 7faafed1e640 1 -- 172.21.15.27:0/632607053 wait complete. 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.981+0000 7faafed1e640 1 Processor -- start 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.981+0000 7faafed1e640 1 -- start start 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faafed1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf815d630 0x7faaf81605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faafed1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaf815da10 0x7faaf8160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faafed1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf81f4630 0x7faaf81f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faafed1e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8147d10 con 0x7faaf815da10 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faafed1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8072010 con 0x7faaf81f4630 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faafed1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf806d840 con 0x7faaf815d630 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faaf7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaf815da10 0x7faaf8160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faaf7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaf815da10 0x7faaf8160b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58428/0 (socket says 172.21.15.27:58428) 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faaf7fff640 1 -- 172.21.15.27:0/3374305695 learned_addr learned my addr 172.21.15.27:0/3374305695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faafd2bb640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf81f4630 0x7faaf81f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faafcaba640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf815d630 0x7faaf81605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faaf7fff640 1 -- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf815d630 msgr2=0x7faaf81605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faaf7fff640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf815d630 0x7faaf81605e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faaf7fff640 1 -- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf81f4630 msgr2=0x7faaf81f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faaf7fff640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf81f4630 0x7faaf81f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faaf7fff640 1 -- 172.21.15.27:0/3374305695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf8143f20 con 0x7faaf815da10 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.982+0000 7faafcaba640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf815d630 0x7faaf81605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.983+0000 7faaf7fff640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaf815da10 0x7faaf8160b20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7faaec00f7f0 tx=0x7faaec00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.983+0000 7faaf57fa640 1 -- 172.21.15.27:0/3374305695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec0052e0 con 0x7faaf815da10 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.983+0000 7faaf57fa640 1 -- 172.21.15.27:0/3374305695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaec018070 con 0x7faaf815da10 2024-03-06T07:23:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.983+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf8144200 con 0x7faaf815da10 2024-03-06T07:23:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.983+0000 7faaf57fa640 1 -- 172.21.15.27:0/3374305695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec00d970 con 0x7faaf815da10 2024-03-06T07:23:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.983+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf81f5510 con 0x7faaf815da10 2024-03-06T07:23:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.984+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf8067ff0 con 0x7faaf815da10 2024-03-06T07:23:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.984+0000 7faaf57fa640 1 -- 172.21.15.27:0/3374305695 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faaec00db10 con 0x7faaf815da10 2024-03-06T07:23:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.985+0000 7faaf57fa640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faae406b4d0 0x7faae406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.985+0000 7faaf57fa640 1 -- 172.21.15.27:0/3374305695 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faaec089df0 con 0x7faaf815da10 2024-03-06T07:23:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.985+0000 7faafcaba640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faae406b4d0 0x7faae406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.985+0000 7faafcaba640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faae406b4d0 0x7faae406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faaf81e8c60 tx=0x7faaf000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:50.989+0000 7faaf57fa640 1 -- 172.21.15.27:0/3374305695 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faaec058fc0 con 0x7faaf815da10 2024-03-06T07:23:51.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.238+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7faaf80b31c0 con 0x7faaf815da10 2024-03-06T07:23:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.240+0000 7faaf57fa640 1 -- 172.21.15.27:0/3374305695 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7faaec0588c0 con 0x7faaf815da10 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faae406b4d0 msgr2=0x7faae406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faae406b4d0 0x7faae406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faaf81e8c60 tx=0x7faaf000b040 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaf815da10 msgr2=0x7faaf8160b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaf815da10 0x7faaf8160b20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7faaec00f7f0 tx=0x7faaec00fcb0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 shutdown_connections 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faae406b4d0 0x7faae406d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faaf81f4630 0x7faaf81f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaf815da10 0x7faaf8160b20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 --2- 172.21.15.27:0/3374305695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faaf815d630 0x7faaf81605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 >> 172.21.15.27:0/3374305695 conn(0x7faaf806cd20 msgr2=0x7faaf80b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 shutdown_connections 2024-03-06T07:23:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.242+0000 7faafed1e640 1 -- 172.21.15.27:0/3374305695 wait complete. 2024-03-06T07:23:51.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:23:51.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.362+0000 7f29b99d0640 1 Processor -- start 2024-03-06T07:23:51.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.362+0000 7f29b99d0640 1 -- start start 2024-03-06T07:23:51.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b99d0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29b406e700 0x7f29b415ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b99d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 0x7f29b406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b99d0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b40b4160 con 0x7f29b40b55d0 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b99d0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b40b5b40 con 0x7f29b406e700 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b99d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b40b3ba0 con 0x7f29b4072660 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b2ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29b406e700 0x7f29b415ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b37fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f29b40b55d0 0x7f29b4071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50466/0 (socket says 172.21.15.27:50466) 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b37fe640 1 -- 172.21.15.27:0/4232542578 learned_addr learned my addr 172.21.15.27:0/4232542578 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29abfff640 1 --2- 172.21.15.27:0/4232542578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 0x7f29b406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b2ffd640 1 -- 172.21.15.27:0/4232542578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 msgr2=0x7f29b406d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b2ffd640 1 --2- 172.21.15.27:0/4232542578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 0x7f29b406d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b2ffd640 1 -- 172.21.15.27:0/4232542578 >> v1:172.21.15.27:6789/0 conn(0x7f29b40b55d0 legacy=0x7f29b4071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b27fc640 1 -- 172.21.15.27:0/4232542578 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040563549 0 0) 0x7f29b40b4160 con 0x7f29b40b55d0 2024-03-06T07:23:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.363+0000 7f29b2ffd640 1 -- 172.21.15.27:0/4232542578 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b40b3e80 con 0x7f29b406e700 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.364+0000 7f29b2ffd640 1 --2- 172.21.15.27:0/4232542578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b406e700 0x7f29b415ef80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f29a40097a0 tx=0x7f29a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc02125307fd26be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.364+0000 7f29b27fc640 1 -- 172.21.15.27:0/4232542578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4035020 con 0x7f29b406e700 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.364+0000 7f29b27fc640 1 -- 172.21.15.27:0/4232542578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a4031830 con 0x7f29b406e700 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.364+0000 7f29b27fc640 1 -- 172.21.15.27:0/4232542578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4031c00 con 0x7f29b406e700 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.365+0000 7f29b99d0640 1 -- 172.21.15.27:0/4232542578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b406e700 msgr2=0x7f29b415ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.365+0000 7f29b99d0640 1 --2- 172.21.15.27:0/4232542578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b406e700 0x7f29b415ef80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f29a40097a0 tx=0x7f29a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.365+0000 7f29b99d0640 1 -- 172.21.15.27:0/4232542578 shutdown_connections 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.365+0000 7f29b99d0640 1 --2- 172.21.15.27:0/4232542578 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 0x7f29b406d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.365+0000 7f29b99d0640 1 --2- 172.21.15.27:0/4232542578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b406e700 0x7f29b415ef80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.365+0000 7f29b99d0640 1 -- 172.21.15.27:0/4232542578 >> 172.21.15.27:0/4232542578 conn(0x7f29b40b2720 msgr2=0x7f29b40b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.365+0000 7f29b99d0640 1 -- 172.21.15.27:0/4232542578 shutdown_connections 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.365+0000 7f29b99d0640 1 -- 172.21.15.27:0/4232542578 wait complete. 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.366+0000 7f29b99d0640 1 Processor -- start 2024-03-06T07:23:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.366+0000 7f29b99d0640 1 -- start start 2024-03-06T07:23:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b99d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b406e700 0x7f29b41f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b99d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 0x7f29b4152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b99d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b40b55d0 0x7f29b40cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b99d0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b406ec60 con 0x7f29b40b55d0 2024-03-06T07:23:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b99d0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b406eae0 con 0x7f29b406e700 2024-03-06T07:23:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b99d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b4072090 con 0x7f29b4072660 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b40b55d0 0x7f29b40cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b40b55d0 0x7f29b40cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58440/0 (socket says 172.21.15.27:58440) 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b37fe640 1 -- 172.21.15.27:0/2763001189 learned_addr learned my addr 172.21.15.27:0/2763001189 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29abfff640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 0x7f29b4152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b37fe640 1 -- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 msgr2=0x7f29b4152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b37fe640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 0x7f29b4152ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b37fe640 1 -- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b406e700 msgr2=0x7f29b41f10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b2ffd640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b406e700 0x7f29b41f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b37fe640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b406e700 0x7f29b41f10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b37fe640 1 -- 172.21.15.27:0/2763001189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b41f15e0 con 0x7f29b40b55d0 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b2ffd640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b406e700 0x7f29b41f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b37fe640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b40b55d0 0x7f29b40cf3f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00c7d0 tx=0x7f29ac00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b0ff9640 1 -- 172.21.15.27:0/2763001189 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac018070 con 0x7f29b40b55d0 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b0ff9640 1 -- 172.21.15.27:0/2763001189 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ac00a640 con 0x7f29b40b55d0 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b0ff9640 1 -- 172.21.15.27:0/2763001189 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac00aa70 con 0x7f29b40b55d0 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.367+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b4158cf0 con 0x7f29b40b55d0 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.368+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b41592b0 con 0x7f29b40b55d0 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.368+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b4067ff0 con 0x7f29b40b55d0 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.369+0000 7f29b0ff9640 1 -- 172.21.15.27:0/2763001189 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29ac01b070 con 0x7f29b40b55d0 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.369+0000 7f29b0ff9640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f299406b4d0 0x7f299406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.369+0000 7f29b0ff9640 1 -- 172.21.15.27:0/2763001189 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f29ac089400 con 0x7f29b40b55d0 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.370+0000 7f29b2ffd640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f299406b4d0 0x7f299406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.370+0000 7f29b2ffd640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f299406b4d0 0x7f299406d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f29a4036000 tx=0x7f29a4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.374+0000 7f29b0ff9640 1 -- 172.21.15.27:0/2763001189 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29ac058660 con 0x7f29b40b55d0 2024-03-06T07:23:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.623+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f29b415be20 con 0x7f29b40b55d0 2024-03-06T07:23:51.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.625+0000 7f29b0ff9640 1 -- 172.21.15.27:0/2763001189 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f29ac057f60 con 0x7f29b40b55d0 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f299406b4d0 msgr2=0x7f299406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f299406b4d0 0x7f299406d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f29a4036000 tx=0x7f29a4008030 comp rx=0 tx=0).stop 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b40b55d0 msgr2=0x7f29b40cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b40b55d0 0x7f29b40cf3f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00c7d0 tx=0x7f29ac00cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 shutdown_connections 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f299406b4d0 0x7f299406d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29b40b55d0 0x7f29b40cf3f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29b4072660 0x7f29b4152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 --2- 172.21.15.27:0/2763001189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29b406e700 0x7f29b41f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 >> 172.21.15.27:0/2763001189 conn(0x7f29b40b2720 msgr2=0x7f29b4071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 shutdown_connections 2024-03-06T07:23:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.628+0000 7f29b99d0640 1 -- 172.21.15.27:0/2763001189 wait complete. 2024-03-06T07:23:51.640 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:23:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.747+0000 7f20e48eb640 1 Processor -- start 2024-03-06T07:23:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.747+0000 7f20e48eb640 1 -- start start 2024-03-06T07:23:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.748+0000 7f20e48eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f20e015bd20 0x7f20e00cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.748+0000 7f20e48eb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e006dcf0 con 0x7f20e0072810 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.748+0000 7f20e48eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e0071a50 con 0x7f20e006d840 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.748+0000 7f20e48eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e0071d30 con 0x7f20e015bd20 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.748+0000 7f20ddd9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f20e0072810 0x7f20e015c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50472/0 (socket says 172.21.15.27:50472) 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.748+0000 7f20ddd9b640 1 -- 172.21.15.27:0/1851409820 learned_addr learned my addr 172.21.15.27:0/1851409820 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20de59c640 1 --2- 172.21.15.27:0/1851409820 >> v2:172.21.15.69:3300/0 conn(0x7f20e015bd20 0x7f20e00cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939391641 0 0) 0x7f20e006dcf0 con 0x7f20e0072810 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20cc003520 con 0x7f20e0072810 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 599274405 0 0) 0x7f20cc003520 con 0x7f20e0072810 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 >> v2:172.21.15.69:3300/0 conn(0x7f20e015bd20 msgr2=0x7f20e00cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 --2- 172.21.15.27:0/1851409820 >> v2:172.21.15.69:3300/0 conn(0x7f20e015bd20 0x7f20e00cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 >> v1:172.21.15.69:6789/0 conn(0x7f20e006d840 legacy=0x7f20e00cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0072010 con 0x7f20e0072810 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f20c8001670 con 0x7f20e0072810 2024-03-06T07:23:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 >> v1:172.21.15.27:6789/0 conn(0x7f20e0072810 legacy=0x7f20e015c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc003a00 0x7f20cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20cc004800 0x7f20cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc0051c0 0x7f20cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0071d30 con 0x7f20cc003a00 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc00c0c0 con 0x7f20cc0051c0 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc00c2c0 con 0x7f20cc004800 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20de59c640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc003a00 0x7f20cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20ded9d640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc0051c0 0x7f20cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20ddd9b640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20cc004800 0x7f20cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20de59c640 1 -- 172.21.15.27:0/1851409820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20cc004800 msgr2=0x7f20cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20de59c640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20cc004800 0x7f20cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20de59c640 1 -- 172.21.15.27:0/1851409820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc0051c0 msgr2=0x7f20cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20de59c640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc0051c0 0x7f20cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20de59c640 1 -- 172.21.15.27:0/1851409820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc003520 con 0x7f20cc003a00 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20ded9d640 1 -- 172.21.15.27:0/1851409820 reap_dead start 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.749+0000 7f20ddd9b640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20cc004800 0x7f20cc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.750+0000 7f20de59c640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc003a00 0x7f20cc003e20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f20d8009ac0 tx=0x7f20cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83de4b03dea6cc51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.750+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d8039070 con 0x7f20cc003a00 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.750+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d802c6b0 con 0x7f20cc003a00 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.750+0000 7f20dd59a640 1 -- 172.21.15.27:0/1851409820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d8009690 con 0x7f20cc003a00 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.751+0000 7f20e48eb640 1 -- 172.21.15.27:0/1851409820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc003a00 msgr2=0x7f20cc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.751+0000 7f20e48eb640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc003a00 0x7f20cc003e20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f20d8009ac0 tx=0x7f20cc005830 comp rx=0 tx=0).stop 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.751+0000 7f20e48eb640 1 -- 172.21.15.27:0/1851409820 shutdown_connections 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.751+0000 7f20e48eb640 1 --2- 172.21.15.27:0/1851409820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20cc003a00 0x7f20cc003e20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.751+0000 7f20e48eb640 1 -- 172.21.15.27:0/1851409820 >> 172.21.15.27:0/1851409820 conn(0x7f20e006cd20 msgr2=0x7f20e006d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.751+0000 7f20e48eb640 1 -- 172.21.15.27:0/1851409820 shutdown_connections 2024-03-06T07:23:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.751+0000 7f20e48eb640 1 -- 172.21.15.27:0/1851409820 wait complete. 2024-03-06T07:23:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.752+0000 7f20e48eb640 1 Processor -- start 2024-03-06T07:23:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.752+0000 7f20e48eb640 1 -- start start 2024-03-06T07:23:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20e48eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc003a00 0x7f20e01f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20e48eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e0154d10 0x7f20e013e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20e48eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0152d60 0x7f20e01531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20e48eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00cfb60 con 0x7f20e0152d60 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20e48eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00cf9e0 con 0x7f20cc003a00 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20e48eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00cfce0 con 0x7f20e0154d10 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20ded9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0152d60 0x7f20e01531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20ded9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0152d60 0x7f20e01531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58456/0 (socket says 172.21.15.27:58456) 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20ded9d640 1 -- 172.21.15.27:0/2099538851 learned_addr learned my addr 172.21.15.27:0/2099538851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20ddd9b640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e0154d10 0x7f20e013e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20de59c640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc003a00 0x7f20e01f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20ded9d640 1 -- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e0154d10 msgr2=0x7f20e013e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20ded9d640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e0154d10 0x7f20e013e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20ded9d640 1 -- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc003a00 msgr2=0x7f20e01f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20ded9d640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc003a00 0x7f20e01f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.753+0000 7f20ded9d640 1 -- 172.21.15.27:0/2099538851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0155540 con 0x7f20e0152d60 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.754+0000 7f20ddd9b640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e0154d10 0x7f20e013e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.754+0000 7f20de59c640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc003a00 0x7f20e01f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.754+0000 7f20ded9d640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0152d60 0x7f20e01531c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f20e006c770 tx=0x7f20d000fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.754+0000 7f20d6ffd640 1 -- 172.21.15.27:0/2099538851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d001a070 con 0x7f20e0152d60 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.754+0000 7f20d6ffd640 1 -- 172.21.15.27:0/2099538851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d0006d60 con 0x7f20e0152d60 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.754+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e0155820 con 0x7f20e0152d60 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.754+0000 7f20d6ffd640 1 -- 172.21.15.27:0/2099538851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d0003bb0 con 0x7f20e0152d60 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.754+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e0155de0 con 0x7f20e0152d60 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.755+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e0067ff0 con 0x7f20e0152d60 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.755+0000 7f20d6ffd640 1 -- 172.21.15.27:0/2099538851 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20d0005070 con 0x7f20e0152d60 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.756+0000 7f20d6ffd640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f20c006b500 0x7f20c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.756+0000 7f20d6ffd640 1 -- 172.21.15.27:0/2099538851 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f20d0089330 con 0x7f20e0152d60 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.756+0000 7f20de59c640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f20c006b500 0x7f20c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.757+0000 7f20de59c640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f20c006b500 0x7f20c006d9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f20d8036000 tx=0x7f20d8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:51.760+0000 7f20d6ffd640 1 -- 172.21.15.27:0/2099538851 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20d0058500 con 0x7f20e0152d60 2024-03-06T07:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.010+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f20e006dcf0 con 0x7f20e0152d60 2024-03-06T07:23:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.012+0000 7f20d6ffd640 1 -- 172.21.15.27:0/2099538851 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f20d0057e00 con 0x7f20e0152d60 2024-03-06T07:23:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f20c006b500 msgr2=0x7f20c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f20c006b500 0x7f20c006d9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f20d8036000 tx=0x7f20d8030000 comp rx=0 tx=0).stop 2024-03-06T07:23:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0152d60 msgr2=0x7f20e01531c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0152d60 0x7f20e01531c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f20e006c770 tx=0x7f20d000fc90 comp rx=0 tx=0).stop 2024-03-06T07:23:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 shutdown_connections 2024-03-06T07:23:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f20c006b500 0x7f20c006d9c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0152d60 0x7f20e01531c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20e0154d10 0x7f20e013e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 --2- 172.21.15.27:0/2099538851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20cc003a00 0x7f20e01f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 >> 172.21.15.27:0/2099538851 conn(0x7f20e006cd20 msgr2=0x7f20e00cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 shutdown_connections 2024-03-06T07:23:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.014+0000 7f20e48eb640 1 -- 172.21.15.27:0/2099538851 wait complete. 2024-03-06T07:23:52.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.132+0000 7f6cfe80f640 1 Processor -- start 2024-03-06T07:23:52.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.132+0000 7f6cfe80f640 1 -- start start 2024-03-06T07:23:52.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfe80f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6cf806e700 0x7f6cf815ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfe80f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6cf80b55d0 0x7f6cf8071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfe80f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf80b4160 con 0x7f6cf8072660 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfe80f640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf80b5b40 con 0x7f6cf80b55d0 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfe80f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf80b3ba0 con 0x7f6cf806e700 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfcdac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6cf80b55d0 0x7f6cf8071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfcdac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6cf80b55d0 0x7f6cf8071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58466/0 (socket says 172.21.15.27:58466) 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfcdac640 1 -- 172.21.15.27:0/3370509647 learned_addr learned my addr 172.21.15.27:0/3370509647 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cf7fff640 1 --2- 172.21.15.27:0/3370509647 >> v2:172.21.15.69:3300/0 conn(0x7f6cf806e700 0x7f6cf815ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfcdac640 1 -- 172.21.15.27:0/3370509647 >> v2:172.21.15.69:3300/0 conn(0x7f6cf806e700 msgr2=0x7f6cf815ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfcdac640 1 --2- 172.21.15.27:0/3370509647 >> v2:172.21.15.69:3300/0 conn(0x7f6cf806e700 0x7f6cf815ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfcdac640 1 -- 172.21.15.27:0/3370509647 >> v1:172.21.15.69:6789/0 conn(0x7f6cf8072660 legacy=0x7f6cf806d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.133+0000 7f6cfcdac640 1 -- 172.21.15.27:0/3370509647 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf80b3e80 con 0x7f6cf80b55d0 2024-03-06T07:23:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.134+0000 7f6cfcdac640 1 --2- 172.21.15.27:0/3370509647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf80b55d0 0x7f6cf8071980 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6cf806bf60 tx=0x7f6cf002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09613a1b77109b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.134+0000 7f6cf6ffd640 1 -- 172.21.15.27:0/3370509647 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf002ca80 con 0x7f6cf80b55d0 2024-03-06T07:23:52.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.134+0000 7f6cf6ffd640 1 -- 172.21.15.27:0/3370509647 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf002cc20 con 0x7f6cf80b55d0 2024-03-06T07:23:52.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.134+0000 7f6cf6ffd640 1 -- 172.21.15.27:0/3370509647 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf00343f0 con 0x7f6cf80b55d0 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.135+0000 7f6cfe80f640 1 -- 172.21.15.27:0/3370509647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf80b55d0 msgr2=0x7f6cf8071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.135+0000 7f6cfe80f640 1 --2- 172.21.15.27:0/3370509647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf80b55d0 0x7f6cf8071980 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6cf806bf60 tx=0x7f6cf002bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.135+0000 7f6cfe80f640 1 -- 172.21.15.27:0/3370509647 shutdown_connections 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.135+0000 7f6cfe80f640 1 --2- 172.21.15.27:0/3370509647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf80b55d0 0x7f6cf8071980 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.135+0000 7f6cfe80f640 1 --2- 172.21.15.27:0/3370509647 >> v2:172.21.15.69:3300/0 conn(0x7f6cf806e700 0x7f6cf815ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.135+0000 7f6cfe80f640 1 -- 172.21.15.27:0/3370509647 >> 172.21.15.27:0/3370509647 conn(0x7f6cf80b2720 msgr2=0x7f6cf80b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.135+0000 7f6cfe80f640 1 -- 172.21.15.27:0/3370509647 shutdown_connections 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.135+0000 7f6cfe80f640 1 -- 172.21.15.27:0/3370509647 wait complete. 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.136+0000 7f6cfe80f640 1 Processor -- start 2024-03-06T07:23:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.136+0000 7f6cfe80f640 1 -- start start 2024-03-06T07:23:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cfe80f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cf806e700 0x7f6cf813e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cfe80f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf8072660 0x7f6cf80cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cfe80f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf80b55d0 0x7f6cf80cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cfe80f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8072040 con 0x7f6cf8072660 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cfe80f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8071ec0 con 0x7f6cf806e700 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cfe80f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf80721c0 con 0x7f6cf80b55d0 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cf77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf8072660 0x7f6cf80cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cf77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf8072660 0x7f6cf80cfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58480/0 (socket says 172.21.15.27:58480) 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cf77fe640 1 -- 172.21.15.27:0/1011989150 learned_addr learned my addr 172.21.15.27:0/1011989150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cf7fff640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cf806e700 0x7f6cf813e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cfcdac640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf80b55d0 0x7f6cf80cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cf77fe640 1 -- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf80b55d0 msgr2=0x7f6cf80cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cf77fe640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf80b55d0 0x7f6cf80cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cf77fe640 1 -- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cf806e700 msgr2=0x7f6cf813e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cf77fe640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cf806e700 0x7f6cf813e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.137+0000 7f6cf77fe640 1 -- 172.21.15.27:0/1011989150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf81e7ad0 con 0x7f6cf8072660 2024-03-06T07:23:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.138+0000 7f6cf77fe640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf8072660 0x7f6cf80cfa50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800c400 tx=0x7f6ce800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.138+0000 7f6cf4ff9640 1 -- 172.21.15.27:0/1011989150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8011790 con 0x7f6cf8072660 2024-03-06T07:23:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.138+0000 7f6cf4ff9640 1 -- 172.21.15.27:0/1011989150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce8017070 con 0x7f6cf8072660 2024-03-06T07:23:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.138+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf81e7db0 con 0x7f6cf8072660 2024-03-06T07:23:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.138+0000 7f6cf4ff9640 1 -- 172.21.15.27:0/1011989150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8011bb0 con 0x7f6cf8072660 2024-03-06T07:23:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.138+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf81e8370 con 0x7f6cf8072660 2024-03-06T07:23:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.139+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf8067ff0 con 0x7f6cf8072660 2024-03-06T07:23:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.139+0000 7f6cf4ff9640 1 -- 172.21.15.27:0/1011989150 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ce801c460 con 0x7f6cf8072660 2024-03-06T07:23:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.140+0000 7f6cf4ff9640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ce406b4d0 0x7f6ce406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.140+0000 7f6cf4ff9640 1 -- 172.21.15.27:0/1011989150 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6ce8089ba0 con 0x7f6cf8072660 2024-03-06T07:23:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.140+0000 7f6cf7fff640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ce406b4d0 0x7f6ce406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.140+0000 7f6cf7fff640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ce406b4d0 0x7f6ce406d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6cec006fd0 tx=0x7f6cec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.144+0000 7f6cf4ff9640 1 -- 172.21.15.27:0/1011989150 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ce8058720 con 0x7f6cf8072660 2024-03-06T07:23:52.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.326+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cf8157ea0 con 0x7f6cf8072660 2024-03-06T07:23:52.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.327+0000 7f6cf4ff9640 1 -- 172.21.15.27:0/1011989150 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+12819 (secure 0 0 0) 0x7f6ce805c7c0 con 0x7f6cf8072660 2024-03-06T07:23:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:52.329 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:49.286200+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:52.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ce406b4d0 msgr2=0x7f6ce406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ce406b4d0 0x7f6ce406d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6cec006fd0 tx=0x7f6cec008040 comp rx=0 tx=0).stop 2024-03-06T07:23:52.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf8072660 msgr2=0x7f6cf80cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf8072660 0x7f6cf80cfa50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800c400 tx=0x7f6ce800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 shutdown_connections 2024-03-06T07:23:52.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ce406b4d0 0x7f6ce406d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cf80b55d0 0x7f6cf80cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cf8072660 0x7f6cf80cfa50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 --2- 172.21.15.27:0/1011989150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cf806e700 0x7f6cf813e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 >> 172.21.15.27:0/1011989150 conn(0x7f6cf80b2720 msgr2=0x7f6cf80b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 shutdown_connections 2024-03-06T07:23:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.329+0000 7f6cfe80f640 1 -- 172.21.15.27:0/1011989150 wait complete. 2024-03-06T07:23:52.342 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:52.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:52.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.450+0000 7fcddf808640 1 Processor -- start 2024-03-06T07:23:52.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.450+0000 7fcddf808640 1 -- start start 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.451+0000 7fcddf808640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80717b0 0x7fcdd8071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.451+0000 7fcddf808640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcdd80b2b10 0x7fcdd80d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.451+0000 7fcddf808640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd815e4b0 con 0x7fcdd80b3060 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.451+0000 7fcddf808640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd80d2e10 con 0x7fcdd80b2b10 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.451+0000 7fcddf808640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b97e0 con 0x7fcdd80717b0 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.451+0000 7fcdddda5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcdd80b2b10 0x7fcdd80d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.451+0000 7fcddcda3640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcdd80b3060 0x7fcdd80b6e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50474/0 (socket says 172.21.15.27:50474) 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.451+0000 7fcddcda3640 1 -- 172.21.15.27:0/971233543 learned_addr learned my addr 172.21.15.27:0/971233543 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.451+0000 7fcddd5a4640 1 --2- 172.21.15.27:0/971233543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80717b0 0x7fcdd8071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.452+0000 7fcdddda5640 1 -- 172.21.15.27:0/971233543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80717b0 msgr2=0x7fcdd8071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.452+0000 7fcdddda5640 1 --2- 172.21.15.27:0/971233543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80717b0 0x7fcdd8071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.452+0000 7fcdddda5640 1 -- 172.21.15.27:0/971233543 >> v1:172.21.15.27:6789/0 conn(0x7fcdd80b3060 legacy=0x7fcdd80b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.452+0000 7fcdddda5640 1 -- 172.21.15.27:0/971233543 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80b9a80 con 0x7fcdd80b2b10 2024-03-06T07:23:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.452+0000 7fcdd7fff640 1 -- 172.21.15.27:0/971233543 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987237427 0 0) 0x7fcdcc001110 con 0x7fcdd80b3060 2024-03-06T07:23:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.452+0000 7fcdddda5640 1 --2- 172.21.15.27:0/971233543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b2b10 0x7fcdd80d32a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00097a0 tx=0x7fcdd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3669016d76430dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.452+0000 7fcdd7fff640 1 -- 172.21.15.27:0/971233543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0035020 con 0x7fcdd80b2b10 2024-03-06T07:23:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.452+0000 7fcdd7fff640 1 -- 172.21.15.27:0/971233543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd0031830 con 0x7fcdd80b2b10 2024-03-06T07:23:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.453+0000 7fcdd7fff640 1 -- 172.21.15.27:0/971233543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0031c00 con 0x7fcdd80b2b10 2024-03-06T07:23:52.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.453+0000 7fcddf808640 1 -- 172.21.15.27:0/971233543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b2b10 msgr2=0x7fcdd80d32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.453+0000 7fcddf808640 1 --2- 172.21.15.27:0/971233543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b2b10 0x7fcdd80d32a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00097a0 tx=0x7fcdd002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.453+0000 7fcddf808640 1 -- 172.21.15.27:0/971233543 shutdown_connections 2024-03-06T07:23:52.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.453+0000 7fcddf808640 1 --2- 172.21.15.27:0/971233543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b2b10 0x7fcdd80d32a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.453+0000 7fcddf808640 1 --2- 172.21.15.27:0/971233543 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80717b0 0x7fcdd8071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.453+0000 7fcddf808640 1 -- 172.21.15.27:0/971233543 >> 172.21.15.27:0/971233543 conn(0x7fcdd80d3d20 msgr2=0x7fcdd80d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.453+0000 7fcddf808640 1 -- 172.21.15.27:0/971233543 shutdown_connections 2024-03-06T07:23:52.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.453+0000 7fcddf808640 1 -- 172.21.15.27:0/971233543 wait complete. 2024-03-06T07:23:52.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.454+0000 7fcddf808640 1 Processor -- start 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.454+0000 7fcddf808640 1 -- start start 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcddf808640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd80717b0 0x7fcdd8158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcddf808640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80b2b10 0x7fcdd80cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcddf808640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b3060 0x7fcdd80ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcddf808640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b7560 con 0x7fcdd80b3060 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcddf808640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80d4bf0 con 0x7fcdd80717b0 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcddf808640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b76e0 con 0x7fcdd80b2b10 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcdddda5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b3060 0x7fcdd80ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcdddda5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b3060 0x7fcdd80ce080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58490/0 (socket says 172.21.15.27:58490) 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcdddda5640 1 -- 172.21.15.27:0/826117114 learned_addr learned my addr 172.21.15.27:0/826117114 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcddd5a4640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd80717b0 0x7fcdd8158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.455+0000 7fcddcda3640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80b2b10 0x7fcdd80cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcdddda5640 1 -- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80b2b10 msgr2=0x7fcdd80cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcdddda5640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80b2b10 0x7fcdd80cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcdddda5640 1 -- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd80717b0 msgr2=0x7fcdd8158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcdddda5640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd80717b0 0x7fcdd8158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcdddda5640 1 -- 172.21.15.27:0/826117114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd8158970 con 0x7fcdd80b3060 2024-03-06T07:23:52.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcdddda5640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b3060 0x7fcdd80ce080 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00098c0 tx=0x7fcdd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcdd5ffb640 1 -- 172.21.15.27:0/826117114 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0041070 con 0x7fcdd80b3060 2024-03-06T07:23:52.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcdd5ffb640 1 -- 172.21.15.27:0/826117114 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd003a8d0 con 0x7fcdd80b3060 2024-03-06T07:23:52.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd8158bf0 con 0x7fcdd80b3060 2024-03-06T07:23:52.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcdd5ffb640 1 -- 172.21.15.27:0/826117114 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0035040 con 0x7fcdd80b3060 2024-03-06T07:23:52.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.456+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd81591b0 con 0x7fcdd80b3060 2024-03-06T07:23:52.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.457+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd81593f0 con 0x7fcdd80b3060 2024-03-06T07:23:52.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.458+0000 7fcdd5ffb640 1 -- 172.21.15.27:0/826117114 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdd0038020 con 0x7fcdd80b3060 2024-03-06T07:23:52.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.458+0000 7fcdd5ffb640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdc406b4d0 0x7fcdc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.458+0000 7fcdd5ffb640 1 -- 172.21.15.27:0/826117114 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcdd00ab720 con 0x7fcdd80b3060 2024-03-06T07:23:52.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.458+0000 7fcddd5a4640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdc406b4d0 0x7fcdc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.459+0000 7fcddd5a4640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdc406b4d0 0x7fcdc406d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8003b80 tx=0x7fcdc80098d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.462+0000 7fcdd5ffb640 1 -- 172.21.15.27:0/826117114 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdd81593f0 con 0x7fcdd80b3060 2024-03-06T07:23:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.643+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcdd804f410 con 0x7fcdd80b3060 2024-03-06T07:23:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.644+0000 7fcdd5ffb640 1 -- 172.21.15.27:0/826117114 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcdd8158970 con 0x7fcdd80b3060 2024-03-06T07:23:52.646 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:23:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdc406b4d0 msgr2=0x7fcdc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdc406b4d0 0x7fcdc406d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8003b80 tx=0x7fcdc80098d0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b3060 msgr2=0x7fcdd80ce080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b3060 0x7fcdd80ce080 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00098c0 tx=0x7fcdd00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 shutdown_connections 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcdc406b4d0 0x7fcdc406d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd80b3060 0x7fcdd80ce080 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcdd80b2b10 0x7fcdd80cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 --2- 172.21.15.27:0/826117114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd80717b0 0x7fcdd8158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 >> 172.21.15.27:0/826117114 conn(0x7fcdd80d3d20 msgr2=0x7fcdd815c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 shutdown_connections 2024-03-06T07:23:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.646+0000 7fcddf808640 1 -- 172.21.15.27:0/826117114 wait complete. 2024-03-06T07:23:52.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:52.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.765+0000 7f0dda7ae640 1 Processor -- start 2024-03-06T07:23:52.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.765+0000 7f0dda7ae640 1 -- start start 2024-03-06T07:23:52.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dda7ae640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0dd415bde0 0x7f0dd40b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dda7ae640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40b2460 0x7f0dd415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dda7ae640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd40b3c70 con 0x7f0dd41451a0 2024-03-06T07:23:52.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dda7ae640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd40b4840 con 0x7f0dd415bde0 2024-03-06T07:23:52.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dda7ae640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40afad0 con 0x7f0dd40b2460 2024-03-06T07:23:52.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dd3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0dd415bde0 0x7f0dd40b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dd3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0dd415bde0 0x7f0dd40b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58502/0 (socket says 172.21.15.27:58502) 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dd3fff640 1 -- 172.21.15.27:0/3221988718 learned_addr learned my addr 172.21.15.27:0/3221988718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dd8d4b640 1 --2- 172.21.15.27:0/3221988718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40b2460 0x7f0dd415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dd3fff640 1 -- 172.21.15.27:0/3221988718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40b2460 msgr2=0x7f0dd415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dd3fff640 1 --2- 172.21.15.27:0/3221988718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40b2460 0x7f0dd415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.766+0000 7f0dd3fff640 1 -- 172.21.15.27:0/3221988718 >> v1:172.21.15.69:6789/0 conn(0x7f0dd41451a0 legacy=0x7f0dd40b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.767+0000 7f0dd3fff640 1 -- 172.21.15.27:0/3221988718 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd40b3f10 con 0x7f0dd415bde0 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.767+0000 7f0dd8d4b640 1 --2- 172.21.15.27:0/3221988718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40b2460 0x7f0dd415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.767+0000 7f0dd3fff640 1 --2- 172.21.15.27:0/3221988718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd415bde0 0x7f0dd40b4cd0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40b49c0 tx=0x7f0dc802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89357d1f087f603f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.767+0000 7f0dd2ffd640 1 -- 172.21.15.27:0/3221988718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8034070 con 0x7f0dd415bde0 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.767+0000 7f0dd2ffd640 1 -- 172.21.15.27:0/3221988718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc8030610 con 0x7f0dd415bde0 2024-03-06T07:23:52.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.767+0000 7f0dd2ffd640 1 -- 172.21.15.27:0/3221988718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc80309e0 con 0x7f0dd415bde0 2024-03-06T07:23:52.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.768+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3221988718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd415bde0 msgr2=0x7f0dd40b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.768+0000 7f0dda7ae640 1 --2- 172.21.15.27:0/3221988718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd415bde0 0x7f0dd40b4cd0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40b49c0 tx=0x7f0dc802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.768+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3221988718 shutdown_connections 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.768+0000 7f0dda7ae640 1 --2- 172.21.15.27:0/3221988718 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd40b2460 0x7f0dd415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.768+0000 7f0dda7ae640 1 --2- 172.21.15.27:0/3221988718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd415bde0 0x7f0dd40b4cd0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.768+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3221988718 >> 172.21.15.27:0/3221988718 conn(0x7f0dd40b55e0 msgr2=0x7f0dd40b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.768+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3221988718 shutdown_connections 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.768+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3221988718 wait complete. 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.769+0000 7f0dda7ae640 1 Processor -- start 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.769+0000 7f0dda7ae640 1 -- start start 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dda7ae640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40b2460 0x7f0dd41529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dda7ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd41451a0 0x7f0dd415f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dda7ae640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd415bde0 0x7f0dd4151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dda7ae640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40b5210 con 0x7f0dd41451a0 2024-03-06T07:23:52.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dda7ae640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40b44d0 con 0x7f0dd40b2460 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dda7ae640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40b5390 con 0x7f0dd415bde0 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd41451a0 0x7f0dd415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd41451a0 0x7f0dd415f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58516/0 (socket says 172.21.15.27:58516) 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd37fe640 1 -- 172.21.15.27:0/3998488738 learned_addr learned my addr 172.21.15.27:0/3998488738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd3fff640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40b2460 0x7f0dd41529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd8d4b640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd415bde0 0x7f0dd4151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd37fe640 1 -- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd415bde0 msgr2=0x7f0dd4151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd37fe640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd415bde0 0x7f0dd4151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd37fe640 1 -- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40b2460 msgr2=0x7f0dd41529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd37fe640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40b2460 0x7f0dd41529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.770+0000 7f0dd37fe640 1 -- 172.21.15.27:0/3998488738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4150240 con 0x7f0dd41451a0 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.771+0000 7f0dd37fe640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd41451a0 0x7f0dd415f060 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400c400 tx=0x7f0dc400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.771+0000 7f0dd0ff9640 1 -- 172.21.15.27:0/3998488738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc40052d0 con 0x7f0dd41451a0 2024-03-06T07:23:52.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.771+0000 7f0dd0ff9640 1 -- 172.21.15.27:0/3998488738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc4005490 con 0x7f0dd41451a0 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.771+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd41f3b50 con 0x7f0dd41451a0 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.771+0000 7f0dd0ff9640 1 -- 172.21.15.27:0/3998488738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4007260 con 0x7f0dd41451a0 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.771+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd41f4110 con 0x7f0dd41451a0 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.772+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd4067ff0 con 0x7f0dd41451a0 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.772+0000 7f0dd0ff9640 1 -- 172.21.15.27:0/3998488738 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0dc4007400 con 0x7f0dd41451a0 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.773+0000 7f0dd0ff9640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dc006b500 0x7f0dc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.773+0000 7f0dd0ff9640 1 -- 172.21.15.27:0/3998488738 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0dc4089210 con 0x7f0dd41451a0 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.773+0000 7f0dd3fff640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dc006b500 0x7f0dc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.773+0000 7f0dd3fff640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dc006b500 0x7f0dc006d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8036000 tx=0x7f0dc803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.777+0000 7f0dd0ff9640 1 -- 172.21.15.27:0/3998488738 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0dc4057e20 con 0x7f0dd41451a0 2024-03-06T07:23:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.959+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0dd40af890 con 0x7f0dd41451a0 2024-03-06T07:23:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.960+0000 7f0dd0ff9640 1 -- 172.21.15.27:0/3998488738 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0dc405bec0 con 0x7f0dd41451a0 2024-03-06T07:23:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:52.962 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:23:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:23:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:23:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:23:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:23:52.962 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:23:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.961+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dc006b500 msgr2=0x7f0dc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.961+0000 7f0dda7ae640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dc006b500 0x7f0dc006d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8036000 tx=0x7f0dc803b040 comp rx=0 tx=0).stop 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.961+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd41451a0 msgr2=0x7f0dd415f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.961+0000 7f0dda7ae640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd41451a0 0x7f0dd415f060 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400c400 tx=0x7f0dc400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.962+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 shutdown_connections 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.962+0000 7f0dda7ae640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dc006b500 0x7f0dc006d9c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.962+0000 7f0dda7ae640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dd415bde0 0x7f0dd4151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.962+0000 7f0dda7ae640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd41451a0 0x7f0dd415f060 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.962+0000 7f0dda7ae640 1 --2- 172.21.15.27:0/3998488738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0dd40b2460 0x7f0dd41529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.962+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 >> 172.21.15.27:0/3998488738 conn(0x7f0dd40b55e0 msgr2=0x7f0dd40b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.962+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 shutdown_connections 2024-03-06T07:23:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:52.962+0000 7f0dda7ae640 1 -- 172.21.15.27:0/3998488738 wait complete. 2024-03-06T07:23:52.976 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:23:52.976 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:23:53.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.085+0000 7fd700f1e640 1 Processor -- start 2024-03-06T07:23:53.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.085+0000 7fd700f1e640 1 -- start start 2024-03-06T07:23:53.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.086+0000 7fd700f1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15dd90 0x7fd6fc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.086+0000 7fd700f1e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd6fc0d29d0 0x7fd6fc0d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.086+0000 7fd700f1e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc0d3740 con 0x7fd6fc0d4720 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.086+0000 7fd700f1e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6fc0d3400 con 0x7fd6fc0d29d0 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.086+0000 7fd700f1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0d3120 con 0x7fd6fc15dd90 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6fa59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15dd90 0x7fd6fc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6fa59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15dd90 0x7fd6fc0ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:45276/0 (socket says 172.21.15.27:45276) 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6fa59c640 1 -- 172.21.15.27:0/1630219414 learned_addr learned my addr 172.21.15.27:0/1630219414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6f9d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd6fc0d29d0 0x7fd6fc0d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6fad9d640 1 --1- 172.21.15.27:0/1630219414 >> v1:172.21.15.69:6789/0 conn(0x7fd6fc0d4720 0x7fd6fc0b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:39494/0 (socket says 172.21.15.27:39494) 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6fa59c640 1 -- 172.21.15.27:0/1630219414 >> v2:172.21.15.69:3300/0 conn(0x7fd6fc0d29d0 msgr2=0x7fd6fc0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6fa59c640 1 --2- 172.21.15.27:0/1630219414 >> v2:172.21.15.69:3300/0 conn(0x7fd6fc0d29d0 0x7fd6fc0d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6fa59c640 1 -- 172.21.15.27:0/1630219414 >> v1:172.21.15.69:6789/0 conn(0x7fd6fc0d4720 legacy=0x7fd6fc0b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6fa59c640 1 -- 172.21.15.27:0/1630219414 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc0d39e0 con 0x7fd6fc15dd90 2024-03-06T07:23:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.087+0000 7fd6f9d9b640 1 --2- 172.21.15.27:0/1630219414 >> v2:172.21.15.69:3300/0 conn(0x7fd6fc0d29d0 0x7fd6fc0d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd6fa59c640 1 --2- 172.21.15.27:0/1630219414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15dd90 0x7fd6fc0ceea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009870 tx=0x7fd6f402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=449ed3869c28455d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd6f959a640 1 -- 172.21.15.27:0/1630219414 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f402f830 con 0x7fd6fc15dd90 2024-03-06T07:23:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd6f959a640 1 -- 172.21.15.27:0/1630219414 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f402f9d0 con 0x7fd6fc15dd90 2024-03-06T07:23:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd6f959a640 1 -- 172.21.15.27:0/1630219414 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f402fda0 con 0x7fd6fc15dd90 2024-03-06T07:23:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd700f1e640 1 -- 172.21.15.27:0/1630219414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15dd90 msgr2=0x7fd6fc0ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd700f1e640 1 --2- 172.21.15.27:0/1630219414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15dd90 0x7fd6fc0ceea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009870 tx=0x7fd6f402ba10 comp rx=0 tx=0).stop 2024-03-06T07:23:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd700f1e640 1 -- 172.21.15.27:0/1630219414 shutdown_connections 2024-03-06T07:23:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd700f1e640 1 --2- 172.21.15.27:0/1630219414 >> v2:172.21.15.69:3300/0 conn(0x7fd6fc0d29d0 0x7fd6fc0d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd700f1e640 1 --2- 172.21.15.27:0/1630219414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15dd90 0x7fd6fc0ceea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd700f1e640 1 -- 172.21.15.27:0/1630219414 >> 172.21.15.27:0/1630219414 conn(0x7fd6fc071960 msgr2=0x7fd6fc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.088+0000 7fd700f1e640 1 -- 172.21.15.27:0/1630219414 shutdown_connections 2024-03-06T07:23:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.089+0000 7fd700f1e640 1 -- 172.21.15.27:0/1630219414 wait complete. 2024-03-06T07:23:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.090+0000 7fd700f1e640 1 Processor -- start 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.090+0000 7fd700f1e640 1 -- start start 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.090+0000 7fd700f1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0d29d0 0x7fd6fc1f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.090+0000 7fd700f1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc0d4720 0x7fd6fc15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd700f1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15dd90 0x7fd6fc1e8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd700f1e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0d2df0 con 0x7fd6fc15dd90 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd700f1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc15e170 con 0x7fd6fc0d4720 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd700f1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0b7420 con 0x7fd6fc0d29d0 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15dd90 0x7fd6fc1e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15dd90 0x7fd6fc1e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58530/0 (socket says 172.21.15.27:58530) 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fad9d640 1 -- 172.21.15.27:0/638344444 learned_addr learned my addr 172.21.15.27:0/638344444 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fa59c640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0d29d0 0x7fd6fc1f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6f9d9b640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc0d4720 0x7fd6fc15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fad9d640 1 -- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0d29d0 msgr2=0x7fd6fc1f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fad9d640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0d29d0 0x7fd6fc1f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fad9d640 1 -- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc0d4720 msgr2=0x7fd6fc15c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fad9d640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc0d4720 0x7fd6fc15c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fad9d640 1 -- 172.21.15.27:0/638344444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc1e8ac0 con 0x7fd6fc15dd90 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6fad9d640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15dd90 0x7fd6fc1e8170 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c7b0 tx=0x7fd6ec00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6eaffd640 1 -- 172.21.15.27:0/638344444 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec017070 con 0x7fd6fc15dd90 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6eaffd640 1 -- 172.21.15.27:0/638344444 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec00a640 con 0x7fd6fc15dd90 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc0caea0 con 0x7fd6fc15dd90 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd6eaffd640 1 -- 172.21.15.27:0/638344444 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec00a9f0 con 0x7fd6fc15dd90 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.091+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc0cb460 con 0x7fd6fc15dd90 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.092+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc072a30 con 0x7fd6fc15dd90 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.093+0000 7fd6eaffd640 1 -- 172.21.15.27:0/638344444 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6ec01a070 con 0x7fd6fc15dd90 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.093+0000 7fd6eaffd640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6dc06b4d0 0x7fd6dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.093+0000 7fd6eaffd640 1 -- 172.21.15.27:0/638344444 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd6ec0893d0 con 0x7fd6fc15dd90 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.094+0000 7fd6fa59c640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6dc06b4d0 0x7fd6dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.094+0000 7fd6fa59c640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6dc06b4d0 0x7fd6dc06d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4036000 tx=0x7fd6f4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.098+0000 7fd6eaffd640 1 -- 172.21.15.27:0/638344444 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6ec057f60 con 0x7fd6fc15dd90 2024-03-06T07:23:53.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.279+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd6fc0cb630 con 0x7fd6fc15dd90 2024-03-06T07:23:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.628+0000 7fd6eaffd640 1 -- 172.21.15.27:0/638344444 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd6ec05c000 con 0x7fd6fc15dd90 2024-03-06T07:23:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:23:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6dc06b4d0 msgr2=0x7fd6dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6dc06b4d0 0x7fd6dc06d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4036000 tx=0x7fd6f4008030 comp rx=0 tx=0).stop 2024-03-06T07:23:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15dd90 msgr2=0x7fd6fc1e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15dd90 0x7fd6fc1e8170 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c7b0 tx=0x7fd6ec00cc70 comp rx=0 tx=0).stop 2024-03-06T07:23:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 shutdown_connections 2024-03-06T07:23:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6dc06b4d0 0x7fd6dc06d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15dd90 0x7fd6fc1e8170 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc0d4720 0x7fd6fc15c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 --2- 172.21.15.27:0/638344444 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0d29d0 0x7fd6fc1f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 >> 172.21.15.27:0/638344444 conn(0x7fd6fc071960 msgr2=0x7fd6fc0b6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.632+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 shutdown_connections 2024-03-06T07:23:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.633+0000 7fd700f1e640 1 -- 172.21.15.27:0/638344444 wait complete. 2024-03-06T07:23:53.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:53.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.762+0000 7fdf7b05c640 1 Processor -- start 2024-03-06T07:23:53.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.762+0000 7fdf7b05c640 1 -- start start 2024-03-06T07:23:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf7b05c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdf7415f5c0 0x7fdf740cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf7b05c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b33b0 con 0x7fdf740cd720 2024-03-06T07:23:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf7b05c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b0d40 con 0x7fdf740cff40 2024-03-06T07:23:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf7b05c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf740cdb50 con 0x7fdf7415f5c0 2024-03-06T07:23:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf78df8640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdf7415f5c0 0x7fdf740cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf795f9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdf740cd720 0x7fdf740b1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36400/0 (socket says 172.21.15.27:36400) 2024-03-06T07:23:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf795f9640 1 -- 172.21.15.27:0/2590840657 learned_addr learned my addr 172.21.15.27:0/2590840657 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf78df8640 1 -- 172.21.15.27:0/2590840657 >> v1:172.21.15.69:6789/0 conn(0x7fdf740cff40 legacy=0x7fdf740b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf78df8640 1 -- 172.21.15.27:0/2590840657 >> v1:172.21.15.27:6789/0 conn(0x7fdf740cd720 legacy=0x7fdf740b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.763+0000 7fdf737fe640 1 -- 172.21.15.27:0/2590840657 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181529134 0 0) 0x7fdf740b33b0 con 0x7fdf740cd720 2024-03-06T07:23:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.764+0000 7fdf78df8640 1 -- 172.21.15.27:0/2590840657 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740b1020 con 0x7fdf7415f5c0 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.764+0000 7fdf78df8640 1 --2- 172.21.15.27:0/2590840657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf7415f5c0 0x7fdf740cfa00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdf640097a0 tx=0x7fdf6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5142d1cfaeaf2e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.764+0000 7fdf737fe640 1 -- 172.21.15.27:0/2590840657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64035020 con 0x7fdf7415f5c0 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.764+0000 7fdf737fe640 1 -- 172.21.15.27:0/2590840657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf64031830 con 0x7fdf7415f5c0 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.764+0000 7fdf737fe640 1 -- 172.21.15.27:0/2590840657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64031c00 con 0x7fdf7415f5c0 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.765+0000 7fdf7b05c640 1 -- 172.21.15.27:0/2590840657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf7415f5c0 msgr2=0x7fdf740cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.765+0000 7fdf7b05c640 1 --2- 172.21.15.27:0/2590840657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf7415f5c0 0x7fdf740cfa00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdf640097a0 tx=0x7fdf6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.765+0000 7fdf7b05c640 1 -- 172.21.15.27:0/2590840657 shutdown_connections 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.765+0000 7fdf7b05c640 1 --2- 172.21.15.27:0/2590840657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf7415f5c0 0x7fdf740cfa00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.765+0000 7fdf7b05c640 1 -- 172.21.15.27:0/2590840657 >> 172.21.15.27:0/2590840657 conn(0x7fdf740b36d0 msgr2=0x7fdf740b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.765+0000 7fdf7b05c640 1 -- 172.21.15.27:0/2590840657 shutdown_connections 2024-03-06T07:23:53.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.765+0000 7fdf7b05c640 1 -- 172.21.15.27:0/2590840657 wait complete. 2024-03-06T07:23:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.766+0000 7fdf7b05c640 1 Processor -- start 2024-03-06T07:23:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.766+0000 7fdf7b05c640 1 -- start start 2024-03-06T07:23:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf7b05c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf740cd720 0x7fdf7414eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf7b05c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cff40 0x7fdf740c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf7b05c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf7415f5c0 0x7fdf7414b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf7b05c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740ced80 con 0x7fdf740cd720 2024-03-06T07:23:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf7b05c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740b0af0 con 0x7fdf7415f5c0 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf7b05c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7415f9a0 con 0x7fdf740cff40 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf78df8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf740cd720 0x7fdf7414eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf78df8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf740cd720 0x7fdf7414eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48296/0 (socket says 172.21.15.27:48296) 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf78df8640 1 -- 172.21.15.27:0/1883914241 learned_addr learned my addr 172.21.15.27:0/1883914241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf795f9640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf7415f5c0 0x7fdf7414b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf73fff640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cff40 0x7fdf740c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf78df8640 1 -- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cff40 msgr2=0x7fdf740c4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf78df8640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cff40 0x7fdf740c4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf78df8640 1 -- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf7415f5c0 msgr2=0x7fdf7414b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf78df8640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf7415f5c0 0x7fdf7414b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.767+0000 7fdf78df8640 1 -- 172.21.15.27:0/1883914241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740c6df0 con 0x7fdf740cd720 2024-03-06T07:23:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.768+0000 7fdf795f9640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf7415f5c0 0x7fdf7414b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.768+0000 7fdf78df8640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf740cd720 0x7fdf7414eae0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdf64000c00 tx=0x7fdf640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.768+0000 7fdf717fa640 1 -- 172.21.15.27:0/1883914241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64040070 con 0x7fdf740cd720 2024-03-06T07:23:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.768+0000 7fdf717fa640 1 -- 172.21.15.27:0/1883914241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf64039a00 con 0x7fdf740cd720 2024-03-06T07:23:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.768+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf740c70d0 con 0x7fdf740cd720 2024-03-06T07:23:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.768+0000 7fdf717fa640 1 -- 172.21.15.27:0/1883914241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64035040 con 0x7fdf740cd720 2024-03-06T07:23:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.768+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7414bbe0 con 0x7fdf740cd720 2024-03-06T07:23:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.769+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7409fd30 con 0x7fdf740cd720 2024-03-06T07:23:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.770+0000 7fdf717fa640 1 -- 172.21.15.27:0/1883914241 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf6404b060 con 0x7fdf740cd720 2024-03-06T07:23:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.770+0000 7fdf717fa640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf6006b4d0 0x7fdf6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.770+0000 7fdf717fa640 1 -- 172.21.15.27:0/1883914241 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fdf640abdb0 con 0x7fdf740cd720 2024-03-06T07:23:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.770+0000 7fdf73fff640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf6006b4d0 0x7fdf6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.771+0000 7fdf73fff640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf6006b4d0 0x7fdf6006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf7415afd0 tx=0x7fdf68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.774+0000 7fdf717fa640 1 -- 172.21.15.27:0/1883914241 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf6407aef0 con 0x7fdf740cd720 2024-03-06T07:23:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:53.956+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdf7404f410 con 0x7fdf740cd720 2024-03-06T07:23:54.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.635+0000 7fdf717fa640 1 -- 172.21.15.27:0/1883914241 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdf6407a7f0 con 0x7fdf740cd720 2024-03-06T07:23:54.637 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.638+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf6006b4d0 msgr2=0x7fdf6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.638+0000 7fdf7b05c640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf6006b4d0 0x7fdf6006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf7415afd0 tx=0x7fdf68009040 comp rx=0 tx=0).stop 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.638+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf740cd720 msgr2=0x7fdf7414eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.638+0000 7fdf7b05c640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf740cd720 0x7fdf7414eae0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdf64000c00 tx=0x7fdf640046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.639+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 shutdown_connections 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.639+0000 7fdf7b05c640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf6006b4d0 0x7fdf6006d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.639+0000 7fdf7b05c640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf7415f5c0 0x7fdf7414b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.639+0000 7fdf7b05c640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf740cff40 0x7fdf740c4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.639+0000 7fdf7b05c640 1 --2- 172.21.15.27:0/1883914241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf740cd720 0x7fdf7414eae0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.639+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 >> 172.21.15.27:0/1883914241 conn(0x7fdf740b36d0 msgr2=0x7fdf74070180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.643+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 shutdown_connections 2024-03-06T07:23:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.644+0000 7fdf7b05c640 1 -- 172.21.15.27:0/1883914241 wait complete. 2024-03-06T07:23:54.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:23:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.780+0000 7f896f3a0640 1 Processor -- start 2024-03-06T07:23:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.780+0000 7f896f3a0640 1 -- start start 2024-03-06T07:23:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.781+0000 7f896f3a0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8968071810 0x7f8968071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.781+0000 7f896f3a0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896806e5a0 con 0x7f896815baa0 2024-03-06T07:23:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.781+0000 7f896f3a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8968072130 con 0x7f8968162430 2024-03-06T07:23:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.781+0000 7f896f3a0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f896806e2c0 con 0x7f8968071810 2024-03-06T07:23:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.781+0000 7f896db9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f896815baa0 0x7f896806d870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36416/0 (socket says 172.21.15.27:36416) 2024-03-06T07:23:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.781+0000 7f896db9d640 1 -- 172.21.15.27:0/3273750804 learned_addr learned my addr 172.21.15.27:0/3273750804 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896e39e640 1 --2- 172.21.15.27:0/3273750804 >> v2:172.21.15.69:3300/0 conn(0x7f8968071810 0x7f8968071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163734306 0 0) 0x7f896806e5a0 con 0x7f896815baa0 2024-03-06T07:23:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8950003520 con 0x7f896815baa0 2024-03-06T07:23:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2211782373 0 0) 0x7f8950003520 con 0x7f896815baa0 2024-03-06T07:23:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 >> v2:172.21.15.69:3300/0 conn(0x7f8968071810 msgr2=0x7f8968071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 --2- 172.21.15.27:0/3273750804 >> v2:172.21.15.69:3300/0 conn(0x7f8968071810 0x7f8968071bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 >> v1:172.21.15.69:6789/0 conn(0x7f8968162430 legacy=0x7f8968164820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896815afc0 con 0x7f896815baa0 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f895c001670 con 0x7f896815baa0 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 >> v1:172.21.15.27:6789/0 conn(0x7f896815baa0 legacy=0x7f896806d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89500039b0 0x7f8950003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8950004800 0x7f8950004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f895000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896806e2c0 con 0x7f89500051c0 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895000c060 con 0x7f89500039b0 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895000c260 con 0x7f8950004800 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896eb9f640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f895000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896e39e640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89500039b0 0x7f8950003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.782+0000 7f896db9d640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8950004800 0x7f8950004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896eb9f640 1 -- 172.21.15.27:0/3273750804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8950004800 msgr2=0x7f8950004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896eb9f640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8950004800 0x7f8950004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896eb9f640 1 -- 172.21.15.27:0/3273750804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89500039b0 msgr2=0x7f8950003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896eb9f640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89500039b0 0x7f8950003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896eb9f640 1 -- 172.21.15.27:0/3273750804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8950003520 con 0x7f89500051c0 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896e39e640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89500039b0 0x7f8950003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896eb9f640 1 -- 172.21.15.27:0/3273750804 reap_dead start 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896eb9f640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f895000b9c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8958001e20 tx=0x7f8950005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d46315261c14a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8960009690 con 0x7f89500051c0 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958003b70 con 0x7f89500051c0 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.783+0000 7f896d39c640 1 -- 172.21.15.27:0/3273750804 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958001a70 con 0x7f89500051c0 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.784+0000 7f896f3a0640 1 -- 172.21.15.27:0/3273750804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 msgr2=0x7f895000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.784+0000 7f896f3a0640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f895000b9c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8958001e20 tx=0x7f8950005830 comp rx=0 tx=0).stop 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.784+0000 7f896f3a0640 1 -- 172.21.15.27:0/3273750804 shutdown_connections 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.784+0000 7f896f3a0640 1 --2- 172.21.15.27:0/3273750804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f895000b9c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.784+0000 7f896f3a0640 1 -- 172.21.15.27:0/3273750804 >> 172.21.15.27:0/3273750804 conn(0x7f89680d07b0 msgr2=0x7f89680d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.784+0000 7f896f3a0640 1 -- 172.21.15.27:0/3273750804 shutdown_connections 2024-03-06T07:23:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.784+0000 7f896f3a0640 1 -- 172.21.15.27:0/3273750804 wait complete. 2024-03-06T07:23:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.785+0000 7f896f3a0640 1 Processor -- start 2024-03-06T07:23:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.785+0000 7f896f3a0640 1 -- start start 2024-03-06T07:23:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896f3a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f896814e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896f3a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8968150b30 0x7f8968066d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896f3a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8968067240 0x7f89680ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896f3a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89680d03a0 con 0x7f89500051c0 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896f3a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89680d0220 con 0x7f8968067240 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896f3a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8968158670 con 0x7f8968150b30 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896e39e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f896814e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896e39e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f896814e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48310/0 (socket says 172.21.15.27:48310) 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896e39e640 1 -- 172.21.15.27:0/3669167462 learned_addr learned my addr 172.21.15.27:0/3669167462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896eb9f640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8968067240 0x7f89680ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.786+0000 7f896db9d640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8968150b30 0x7f8968066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f896e39e640 1 -- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8968150b30 msgr2=0x7f8968066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f896e39e640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8968150b30 0x7f8968066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f896e39e640 1 -- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8968067240 msgr2=0x7f89680ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f896e39e640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8968067240 0x7f89680ce870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f896e39e640 1 -- 172.21.15.27:0/3669167462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89680cf660 con 0x7f89500051c0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f896e39e640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f896814e4d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8960011930 tx=0x7f8960011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f8966ffd640 1 -- 172.21.15.27:0/3669167462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896000f9d0 con 0x7f89500051c0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f8966ffd640 1 -- 172.21.15.27:0/3669167462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896000fb70 con 0x7f89500051c0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f8966ffd640 1 -- 172.21.15.27:0/3669167462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8960014440 con 0x7f89500051c0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896814fd70 con 0x7f89500051c0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.787+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8968150330 con 0x7f89500051c0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.788+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896804f330 con 0x7f89500051c0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.788+0000 7f8966ffd640 1 -- 172.21.15.27:0/3669167462 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8960012040 con 0x7f89500051c0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.789+0000 7f8966ffd640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f895406b500 0x7f895406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.789+0000 7f8966ffd640 1 -- 172.21.15.27:0/3669167462 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f896008a5e0 con 0x7f89500051c0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.789+0000 7f896db9d640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f895406b500 0x7f895406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.789+0000 7f896db9d640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f895406b500 0x7f895406d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f89680af690 tx=0x7f895c0078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:54.793+0000 7f8966ffd640 1 -- 172.21.15.27:0/3669167462 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8960019030 con 0x7f89500051c0 2024-03-06T07:23:55.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.013+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8968067620 con 0x7f89500051c0 2024-03-06T07:23:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.636+0000 7f8966ffd640 1 -- 172.21.15.27:0/3669167462 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8960059550 con 0x7f89500051c0 2024-03-06T07:23:55.640 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f895406b500 msgr2=0x7f895406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f895406b500 0x7f895406d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f89680af690 tx=0x7f895c0078f0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 msgr2=0x7f896814e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f896814e4d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8960011930 tx=0x7f8960011d30 comp rx=0 tx=0).stop 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 shutdown_connections 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f895406b500 0x7f895406d9c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8968067240 0x7f89680ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8968150b30 0x7f8968066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 --2- 172.21.15.27:0/3669167462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89500051c0 0x7f896814e4d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 >> 172.21.15.27:0/3669167462 conn(0x7f89680d07b0 msgr2=0x7f8968152470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.641+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 shutdown_connections 2024-03-06T07:23:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.642+0000 7f896f3a0640 1 -- 172.21.15.27:0/3669167462 wait complete. 2024-03-06T07:23:55.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:55.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.786+0000 7f45f4a88640 1 Processor -- start 2024-03-06T07:23:55.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.786+0000 7f45f4a88640 1 -- start start 2024-03-06T07:23:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.787+0000 7f45f4a88640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45f015df70 0x7f45f00d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.787+0000 7f45f4a88640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00b6f20 con 0x7f45f0071c50 2024-03-06T07:23:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.787+0000 7f45f4a88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00d3030 con 0x7f45f00d18c0 2024-03-06T07:23:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.787+0000 7f45f4a88640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f00d3310 con 0x7f45f015df70 2024-03-06T07:23:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.787+0000 7f45ee59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45f015df70 0x7f45f00d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.787+0000 7f45eed9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f45f0071c50 0x7f45f00b2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36418/0 (socket says 172.21.15.27:36418) 2024-03-06T07:23:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.787+0000 7f45ee59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45f015df70 0x7f45f00d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48318/0 (socket says 172.21.15.27:48318) 2024-03-06T07:23:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.787+0000 7f45eed9d640 1 -- 172.21.15.27:0/28767921 learned_addr learned my addr 172.21.15.27:0/28767921 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.788+0000 7f45ee59c640 1 -- 172.21.15.27:0/28767921 >> v1:172.21.15.69:6789/0 conn(0x7f45f00d18c0 legacy=0x7f45f0071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.788+0000 7f45ee59c640 1 -- 172.21.15.27:0/28767921 >> v1:172.21.15.27:6789/0 conn(0x7f45f0071c50 legacy=0x7f45f00b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.788+0000 7f45ed59a640 1 -- 172.21.15.27:0/28767921 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378887509 0 0) 0x7f45f00b6f20 con 0x7f45f0071c50 2024-03-06T07:23:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.788+0000 7f45ee59c640 1 -- 172.21.15.27:0/28767921 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f00d35f0 con 0x7f45f015df70 2024-03-06T07:23:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.788+0000 7f45ee59c640 1 --2- 172.21.15.27:0/28767921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 0x7f45f00d27b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f45e80097a0 tx=0x7f45e802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f80f97b3b1c0c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45ed59a640 1 -- 172.21.15.27:0/28767921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e802f810 con 0x7f45f015df70 2024-03-06T07:23:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45ed59a640 1 -- 172.21.15.27:0/28767921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e802f9b0 con 0x7f45f015df70 2024-03-06T07:23:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45ed59a640 1 -- 172.21.15.27:0/28767921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e802fd80 con 0x7f45f015df70 2024-03-06T07:23:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45f4a88640 1 -- 172.21.15.27:0/28767921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 msgr2=0x7f45f00d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45f4a88640 1 --2- 172.21.15.27:0/28767921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 0x7f45f00d27b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f45e80097a0 tx=0x7f45e802b9f0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45f4a88640 1 -- 172.21.15.27:0/28767921 shutdown_connections 2024-03-06T07:23:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45f4a88640 1 --2- 172.21.15.27:0/28767921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 0x7f45f00d27b0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45f4a88640 1 -- 172.21.15.27:0/28767921 >> 172.21.15.27:0/28767921 conn(0x7f45f00d3cf0 msgr2=0x7f45f00d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45f4a88640 1 -- 172.21.15.27:0/28767921 shutdown_connections 2024-03-06T07:23:55.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.789+0000 7f45f4a88640 1 -- 172.21.15.27:0/28767921 wait complete. 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.790+0000 7f45f4a88640 1 Processor -- start 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45f4a88640 1 -- start start 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45f4a88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45f0071c50 0x7f45f00b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45f4a88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45f00d18c0 0x7f45f0158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45f4a88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 0x7f45f01592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45f4a88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f015e350 con 0x7f45f015df70 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45f4a88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00d2cf0 con 0x7f45f0071c50 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45f4a88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f015e4d0 con 0x7f45f00d18c0 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45eed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 0x7f45f01592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45eed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 0x7f45f01592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48324/0 (socket says 172.21.15.27:48324) 2024-03-06T07:23:55.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.791+0000 7f45eed9d640 1 -- 172.21.15.27:0/2715480820 learned_addr learned my addr 172.21.15.27:0/2715480820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45ee59c640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45f0071c50 0x7f45f00b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45edd9b640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45f00d18c0 0x7f45f0158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45eed9d640 1 -- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45f00d18c0 msgr2=0x7f45f0158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45eed9d640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45f00d18c0 0x7f45f0158db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45eed9d640 1 -- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45f0071c50 msgr2=0x7f45f00b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45eed9d640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45f0071c50 0x7f45f00b8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45eed9d640 1 -- 172.21.15.27:0/2715480820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f0159b20 con 0x7f45f015df70 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45ee59c640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45f0071c50 0x7f45f00b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45eed9d640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 0x7f45f01592f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f45e000c780 tx=0x7f45e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45e6ffd640 1 -- 172.21.15.27:0/2715480820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0018070 con 0x7f45f015df70 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45e6ffd640 1 -- 172.21.15.27:0/2715480820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e000a640 con 0x7f45f015df70 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45e6ffd640 1 -- 172.21.15.27:0/2715480820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e000aab0 con 0x7f45f015df70 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f0159e00 con 0x7f45f015df70 2024-03-06T07:23:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.792+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f015a6a0 con 0x7f45f015df70 2024-03-06T07:23:55.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.793+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f0159fd0 con 0x7f45f015df70 2024-03-06T07:23:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.794+0000 7f45e6ffd640 1 -- 172.21.15.27:0/2715480820 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45e001b070 con 0x7f45f015df70 2024-03-06T07:23:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.794+0000 7f45e6ffd640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f45d006b4d0 0x7f45d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.794+0000 7f45e6ffd640 1 -- 172.21.15.27:0/2715480820 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45e0089890 con 0x7f45f015df70 2024-03-06T07:23:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.794+0000 7f45ee59c640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f45d006b4d0 0x7f45d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.795+0000 7f45ee59c640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f45d006b4d0 0x7f45d006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f45e8036000 tx=0x7f45e8039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.798+0000 7f45e6ffd640 1 -- 172.21.15.27:0/2715480820 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45e0058680 con 0x7f45f015df70 2024-03-06T07:23:55.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.985+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45f015a870 con 0x7f45f015df70 2024-03-06T07:23:55.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.986+0000 7f45e6ffd640 1 -- 172.21.15.27:0/2715480820 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16610 (secure 0 0 0) 0x7f45e0057f80 con 0x7f45f015df70 2024-03-06T07:23:55.988 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:55.988 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:55.014242+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:53.309610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:53.987106+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:55.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f45d006b4d0 msgr2=0x7f45d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f45d006b4d0 0x7f45d006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f45e8036000 tx=0x7f45e8039040 comp rx=0 tx=0).stop 2024-03-06T07:23:55.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 msgr2=0x7f45f01592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 0x7f45f01592f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f45e000c780 tx=0x7f45e000cc40 comp rx=0 tx=0).stop 2024-03-06T07:23:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 shutdown_connections 2024-03-06T07:23:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f45d006b4d0 0x7f45d006d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45f015df70 0x7f45f01592f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45f00d18c0 0x7f45f0158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 --2- 172.21.15.27:0/2715480820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45f0071c50 0x7f45f00b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 >> 172.21.15.27:0/2715480820 conn(0x7f45f00d3cf0 msgr2=0x7f45f00b9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 shutdown_connections 2024-03-06T07:23:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:55.989+0000 7f45f4a88640 1 -- 172.21.15.27:0/2715480820 wait complete. 2024-03-06T07:23:56.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.106+0000 7fa9007e9640 1 Processor -- start 2024-03-06T07:23:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.106+0000 7fa9007e9640 1 -- start start 2024-03-06T07:23:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa9007e9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8f815e060 0x7fa8f80ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa9007e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 0x7fa8f80ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa9007e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f80d0dd0 con 0x7fa8f806fe20 2024-03-06T07:23:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa9007e9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f80d10b0 con 0x7fa8f815e060 2024-03-06T07:23:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa9007e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80cc0e0 con 0x7fa8f80d0880 2024-03-06T07:23:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fe585640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8f815e060 0x7fa8f80ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fe585640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8f815e060 0x7fa8f80ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48328/0 (socket says 172.21.15.27:48328) 2024-03-06T07:23:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fdd84640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa8f806fe20 0x7fa8f80ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36422/0 (socket says 172.21.15.27:36422) 2024-03-06T07:23:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fe585640 1 -- 172.21.15.27:0/2796767573 learned_addr learned my addr 172.21.15.27:0/2796767573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fed86640 1 --2- 172.21.15.27:0/2796767573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 0x7fa8f80ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fe585640 1 -- 172.21.15.27:0/2796767573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 msgr2=0x7fa8f80ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fe585640 1 --2- 172.21.15.27:0/2796767573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 0x7fa8f80ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fe585640 1 -- 172.21.15.27:0/2796767573 >> v1:172.21.15.27:6789/0 conn(0x7fa8f806fe20 legacy=0x7fa8f80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fd583640 1 -- 172.21.15.27:0/2796767573 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33046471 0 0) 0x7fa8f80d0dd0 con 0x7fa8f806fe20 2024-03-06T07:23:56.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.107+0000 7fa8fe585640 1 -- 172.21.15.27:0/2796767573 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f80cc3c0 con 0x7fa8f815e060 2024-03-06T07:23:56.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.108+0000 7fa8fe585640 1 --2- 172.21.15.27:0/2796767573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f815e060 0x7fa8f80ce310 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4009870 tx=0x7fa8f402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac52b08a20675e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.108+0000 7fa8fd583640 1 -- 172.21.15.27:0/2796767573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4034070 con 0x7fa8f815e060 2024-03-06T07:23:56.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.108+0000 7fa8fd583640 1 -- 172.21.15.27:0/2796767573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f4030840 con 0x7fa8f815e060 2024-03-06T07:23:56.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.108+0000 7fa8fd583640 1 -- 172.21.15.27:0/2796767573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4030c10 con 0x7fa8f815e060 2024-03-06T07:23:56.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.109+0000 7fa9007e9640 1 -- 172.21.15.27:0/2796767573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f815e060 msgr2=0x7fa8f80ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.109+0000 7fa9007e9640 1 --2- 172.21.15.27:0/2796767573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f815e060 0x7fa8f80ce310 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4009870 tx=0x7fa8f402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:56.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.109+0000 7fa9007e9640 1 -- 172.21.15.27:0/2796767573 shutdown_connections 2024-03-06T07:23:56.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.109+0000 7fa9007e9640 1 --2- 172.21.15.27:0/2796767573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 0x7fa8f80ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.109+0000 7fa9007e9640 1 --2- 172.21.15.27:0/2796767573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f815e060 0x7fa8f80ce310 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.109+0000 7fa9007e9640 1 -- 172.21.15.27:0/2796767573 >> 172.21.15.27:0/2796767573 conn(0x7fa8f80d1a90 msgr2=0x7fa8f80d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.109+0000 7fa9007e9640 1 -- 172.21.15.27:0/2796767573 shutdown_connections 2024-03-06T07:23:56.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.109+0000 7fa9007e9640 1 -- 172.21.15.27:0/2796767573 wait complete. 2024-03-06T07:23:56.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.110+0000 7fa9007e9640 1 Processor -- start 2024-03-06T07:23:56.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.110+0000 7fa9007e9640 1 -- start start 2024-03-06T07:23:56.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa9007e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f806fe20 0x7fa8f814ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa9007e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 0x7fa8f80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa9007e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8f815e060 0x7fa8f80b5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa9007e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80cef60 con 0x7fa8f806fe20 2024-03-06T07:23:56.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa9007e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f815e440 con 0x7fa8f815e060 2024-03-06T07:23:56.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa9007e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80cf0e0 con 0x7fa8f80d0880 2024-03-06T07:23:56.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fe585640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f806fe20 0x7fa8f814ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fe585640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f806fe20 0x7fa8f814ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48336/0 (socket says 172.21.15.27:48336) 2024-03-06T07:23:56.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fe585640 1 -- 172.21.15.27:0/3939494645 learned_addr learned my addr 172.21.15.27:0/3939494645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:56.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fdd84640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 0x7fa8f80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fe585640 1 -- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 msgr2=0x7fa8f80b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fed86640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8f815e060 0x7fa8f80b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fe585640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 0x7fa8f80b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fe585640 1 -- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8f815e060 msgr2=0x7fa8f80b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fe585640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8f815e060 0x7fa8f80b5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fe585640 1 -- 172.21.15.27:0/3939494645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f814f4d0 con 0x7fa8f806fe20 2024-03-06T07:23:56.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fdd84640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 0x7fa8f80b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:56.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fed86640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8f815e060 0x7fa8f80b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:56.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8fe585640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f806fe20 0x7fa8f814ef90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4009870 tx=0x7fa8f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.111+0000 7fa8eeffd640 1 -- 172.21.15.27:0/3939494645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4048070 con 0x7fa8f806fe20 2024-03-06T07:23:56.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.112+0000 7fa8eeffd640 1 -- 172.21.15.27:0/3939494645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f4030640 con 0x7fa8f806fe20 2024-03-06T07:23:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.112+0000 7fa8eeffd640 1 -- 172.21.15.27:0/3939494645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4034040 con 0x7fa8f806fe20 2024-03-06T07:23:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.112+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f80b4a60 con 0x7fa8f806fe20 2024-03-06T07:23:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.112+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f80b5020 con 0x7fa8f806fe20 2024-03-06T07:23:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.112+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f8067ff0 con 0x7fa8f806fe20 2024-03-06T07:23:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.113+0000 7fa8eeffd640 1 -- 172.21.15.27:0/3939494645 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8f4037070 con 0x7fa8f806fe20 2024-03-06T07:23:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.113+0000 7fa8eeffd640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8dc06b4d0 0x7fa8dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.113+0000 7fa8eeffd640 1 -- 172.21.15.27:0/3939494645 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8f40ac000 con 0x7fa8f806fe20 2024-03-06T07:23:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.114+0000 7fa8fdd84640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8dc06b4d0 0x7fa8dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.114+0000 7fa8fdd84640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8dc06b4d0 0x7fa8dc06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80b2810 tx=0x7fa8e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.118+0000 7fa8eeffd640 1 -- 172.21.15.27:0/3939494645 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8f814f4d0 con 0x7fa8f806fe20 2024-03-06T07:23:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.299+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8f80d2130 con 0x7fa8f806fe20 2024-03-06T07:23:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.300+0000 7fa8eeffd640 1 -- 172.21.15.27:0/3939494645 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16610 (secure 0 0 0) 0x7fa8f407af10 con 0x7fa8f806fe20 2024-03-06T07:23:56.302 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:56.302 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:55.014242+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:53.309610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:53.987106+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:56.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8dc06b4d0 msgr2=0x7fa8dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8dc06b4d0 0x7fa8dc06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80b2810 tx=0x7fa8e8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f806fe20 msgr2=0x7fa8f814ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f806fe20 0x7fa8f814ef90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4009870 tx=0x7fa8f4004730 comp rx=0 tx=0).stop 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 shutdown_connections 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8dc06b4d0 0x7fa8dc06d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8f815e060 0x7fa8f80b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8f80d0880 0x7fa8f80b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 --2- 172.21.15.27:0/3939494645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8f806fe20 0x7fa8f814ef90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 >> 172.21.15.27:0/3939494645 conn(0x7fa8f80d1a90 msgr2=0x7fa8f815d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 shutdown_connections 2024-03-06T07:23:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.302+0000 7fa9007e9640 1 -- 172.21.15.27:0/3939494645 wait complete. 2024-03-06T07:23:56.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:23:56.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.423+0000 7f6a7a59a640 1 Processor -- start 2024-03-06T07:23:56.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.423+0000 7f6a7a59a640 1 -- start start 2024-03-06T07:23:56.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a7a59a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7415e2f0 0x7f6a740b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a7a59a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a7406c050 0x7f6a740cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a7a59a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7415e6d0 con 0x7f6a7406b4e0 2024-03-06T07:23:56.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a7a59a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a740d0a10 con 0x7f6a7406c050 2024-03-06T07:23:56.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a7a59a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740ccd00 con 0x7f6a7415e2f0 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a737fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6a7406b4e0 0x7f6a7406b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36430/0 (socket says 172.21.15.27:36430) 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a737fe640 1 -- 172.21.15.27:0/3802187249 learned_addr learned my addr 172.21.15.27:0/3802187249 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a73fff640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7415e2f0 0x7f6a740b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a78b37640 1 --2- 172.21.15.27:0/3802187249 >> v2:172.21.15.69:3300/0 conn(0x7f6a7406c050 0x7f6a740cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835031073 0 0) 0x7f6a7415e6d0 con 0x7f6a7406b4e0 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.424+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a5c003520 con 0x7f6a7406b4e0 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3938272168 0 0) 0x7f6a5c003520 con 0x7f6a7406b4e0 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7415e2f0 msgr2=0x7f6a740b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7415e2f0 0x7f6a740b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 >> v2:172.21.15.69:3300/0 conn(0x7f6a7406c050 msgr2=0x7f6a740cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 --2- 172.21.15.27:0/3802187249 >> v2:172.21.15.69:3300/0 conn(0x7f6a7406c050 0x7f6a740cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a740ccfe0 con 0x7f6a7406b4e0 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f6a64001670 con 0x7f6a7406b4e0 2024-03-06T07:23:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 >> v1:172.21.15.27:6789/0 conn(0x7f6a7406b4e0 legacy=0x7f6a7406b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7415e2f0 0x7f6a5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a5c004d40 0x7f6a5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740d0a10 con 0x7f6a5c0039b0 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740ccd00 con 0x7f6a5c004d40 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c00ba40 con 0x7f6a7415e2f0 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a73fff640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a78b37640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a5c004d40 0x7f6a5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a737fe640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7415e2f0 0x7f6a5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a73fff640 1 -- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7415e2f0 msgr2=0x7f6a5c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a73fff640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7415e2f0 0x7f6a5c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a73fff640 1 -- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a5c004d40 msgr2=0x7f6a5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a73fff640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a5c004d40 0x7f6a5c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.425+0000 7f6a73fff640 1 -- 172.21.15.27:0/3802187249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c003520 con 0x7f6a5c0039b0 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.426+0000 7f6a73fff640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a5c003dd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009ac0 tx=0x7f6a5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0db5b2e7f147d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.426+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68039070 con 0x7f6a5c0039b0 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.426+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6802c6b0 con 0x7f6a5c0039b0 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.426+0000 7f6a72ffd640 1 -- 172.21.15.27:0/3802187249 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68009690 con 0x7f6a5c0039b0 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.427+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3802187249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 msgr2=0x7f6a5c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.427+0000 7f6a7a59a640 1 --2- 172.21.15.27:0/3802187249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a5c003dd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009ac0 tx=0x7f6a5c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.427+0000 7f6a78b37640 1 -- 172.21.15.27:0/3802187249 reap_dead start 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.427+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3802187249 shutdown_connections 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.427+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3802187249 >> 172.21.15.27:0/3802187249 conn(0x7f6a740b36d0 msgr2=0x7f6a740b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.427+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3802187249 shutdown_connections 2024-03-06T07:23:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.427+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3802187249 wait complete. 2024-03-06T07:23:56.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.428+0000 7f6a7a59a640 1 Processor -- start 2024-03-06T07:23:56.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.428+0000 7f6a7a59a640 1 -- start start 2024-03-06T07:23:56.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a7a59a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a740b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a7a59a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a740b47c0 0x7f6a740b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a7a59a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7414e1d0 0x7f6a7414e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a7a59a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740b41f0 con 0x7f6a5c0039b0 2024-03-06T07:23:56.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a7a59a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740cd5a0 con 0x7f6a740b47c0 2024-03-06T07:23:56.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a7a59a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740b4370 con 0x7f6a7414e1d0 2024-03-06T07:23:56.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a73fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a740b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a73fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a740b6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48358/0 (socket says 172.21.15.27:48358) 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a73fff640 1 -- 172.21.15.27:0/3850137737 learned_addr learned my addr 172.21.15.27:0/3850137737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a78b37640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7414e1d0 0x7f6a7414e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a737fe640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a740b47c0 0x7f6a740b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a73fff640 1 -- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7414e1d0 msgr2=0x7f6a7414e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a73fff640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7414e1d0 0x7f6a7414e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a73fff640 1 -- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a740b47c0 msgr2=0x7f6a740b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a73fff640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a740b47c0 0x7f6a740b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a73fff640 1 -- 172.21.15.27:0/3850137737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a740b6da0 con 0x7f6a5c0039b0 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.429+0000 7f6a73fff640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a740b6860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009be0 tx=0x7f6a68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.430+0000 7f6a70ff9640 1 -- 172.21.15.27:0/3850137737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68039050 con 0x7f6a5c0039b0 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.430+0000 7f6a70ff9640 1 -- 172.21.15.27:0/3850137737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a68045070 con 0x7f6a5c0039b0 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.430+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a740b7020 con 0x7f6a5c0039b0 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.430+0000 7f6a70ff9640 1 -- 172.21.15.27:0/3850137737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68003a50 con 0x7f6a5c0039b0 2024-03-06T07:23:56.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.430+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a740b75e0 con 0x7f6a5c0039b0 2024-03-06T07:23:56.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.431+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a740b7ca0 con 0x7f6a5c0039b0 2024-03-06T07:23:56.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.431+0000 7f6a70ff9640 1 -- 172.21.15.27:0/3850137737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a68003bf0 con 0x7f6a5c0039b0 2024-03-06T07:23:56.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.431+0000 7f6a70ff9640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a6006b4d0 0x7f6a6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.432+0000 7f6a70ff9640 1 -- 172.21.15.27:0/3850137737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a680abde0 con 0x7f6a5c0039b0 2024-03-06T07:23:56.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.432+0000 7f6a737fe640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a6006b4d0 0x7f6a6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.432+0000 7f6a737fe640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a6006b4d0 0x7f6a6006d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6a64001120 tx=0x7f6a64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.436+0000 7f6a70ff9640 1 -- 172.21.15.27:0/3850137737 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a740b7ca0 con 0x7f6a5c0039b0 2024-03-06T07:23:56.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:56.660+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6a7409fd30 con 0x7f6a5c0039b0 2024-03-06T07:23:57.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.662+0000 7f6a70ff9640 1 -- 172.21.15.27:0/3850137737 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6a6804e020 con 0x7f6a5c0039b0 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a6006b4d0 msgr2=0x7f6a6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a6006b4d0 0x7f6a6006d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6a64001120 tx=0x7f6a64009040 comp rx=0 tx=0).stop 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 msgr2=0x7f6a740b6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a740b6860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009be0 tx=0x7f6a68004730 comp rx=0 tx=0).stop 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 shutdown_connections 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a6006b4d0 0x7f6a6006d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a7414e1d0 0x7f6a7414e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a740b47c0 0x7f6a740b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 --2- 172.21.15.27:0/3850137737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a5c0039b0 0x7f6a740b6860 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 >> 172.21.15.27:0/3850137737 conn(0x7f6a740b36d0 msgr2=0x7f6a740cdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 shutdown_connections 2024-03-06T07:23:57.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.664+0000 7f6a7a59a640 1 -- 172.21.15.27:0/3850137737 wait complete. 2024-03-06T07:23:57.676 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:57.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:23:57.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.828+0000 7f190515c640 1 Processor -- start 2024-03-06T07:23:57.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.828+0000 7f190515c640 1 -- start start 2024-03-06T07:23:57.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f190515c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19000ceea0 0x7f19000d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f190515c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19000b6d10 0x7f19000b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f190515c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19000d3740 con 0x7f190015dd90 2024-03-06T07:23:57.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f190515c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19000d3400 con 0x7f19000ceea0 2024-03-06T07:23:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f190515c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19000d3120 con 0x7f19000b6d10 2024-03-06T07:23:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f18fe59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19000ceea0 0x7f19000d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f18fe59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19000ceea0 0x7f19000d29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48364/0 (socket says 172.21.15.27:48364) 2024-03-06T07:23:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f18fe59c640 1 -- 172.21.15.27:0/2441256588 learned_addr learned my addr 172.21.15.27:0/2441256588 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f18fe59c640 1 -- 172.21.15.27:0/2441256588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19000b6d10 msgr2=0x7f19000b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f18fe59c640 1 --2- 172.21.15.27:0/2441256588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19000b6d10 0x7f19000b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.829+0000 7f18fe59c640 1 -- 172.21.15.27:0/2441256588 >> v1:172.21.15.69:6789/0 conn(0x7f190015dd90 legacy=0x7f19000d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.830+0000 7f18fe59c640 1 -- 172.21.15.27:0/2441256588 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19000d39e0 con 0x7f19000ceea0 2024-03-06T07:23:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.830+0000 7f18fe59c640 1 --2- 172.21.15.27:0/2441256588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000ceea0 0x7f19000d29d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f19000724d0 tx=0x7f18f402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556cdbcf65f76c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.830+0000 7f18fdd9b640 1 -- 172.21.15.27:0/2441256588 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4034070 con 0x7f19000ceea0 2024-03-06T07:23:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.830+0000 7f18fdd9b640 1 -- 172.21.15.27:0/2441256588 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f4030610 con 0x7f19000ceea0 2024-03-06T07:23:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.830+0000 7f18fdd9b640 1 -- 172.21.15.27:0/2441256588 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f40309e0 con 0x7f19000ceea0 2024-03-06T07:23:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.831+0000 7f190515c640 1 -- 172.21.15.27:0/2441256588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000ceea0 msgr2=0x7f19000d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.831+0000 7f190515c640 1 --2- 172.21.15.27:0/2441256588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000ceea0 0x7f19000d29d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f19000724d0 tx=0x7f18f402b960 comp rx=0 tx=0).stop 2024-03-06T07:23:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.831+0000 7f190515c640 1 -- 172.21.15.27:0/2441256588 shutdown_connections 2024-03-06T07:23:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.831+0000 7f190515c640 1 --2- 172.21.15.27:0/2441256588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000ceea0 0x7f19000d29d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.831+0000 7f190515c640 1 --2- 172.21.15.27:0/2441256588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19000b6d10 0x7f19000b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.831+0000 7f190515c640 1 -- 172.21.15.27:0/2441256588 >> 172.21.15.27:0/2441256588 conn(0x7f1900071960 msgr2=0x7f1900071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.831+0000 7f190515c640 1 -- 172.21.15.27:0/2441256588 shutdown_connections 2024-03-06T07:23:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.831+0000 7f190515c640 1 -- 172.21.15.27:0/2441256588 wait complete. 2024-03-06T07:23:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.833+0000 7f190515c640 1 Processor -- start 2024-03-06T07:23:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.833+0000 7f190515c640 1 -- start start 2024-03-06T07:23:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.833+0000 7f190515c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000b6d10 0x7f19000c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.833+0000 7f190515c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19000ceea0 0x7f190014be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.833+0000 7f190515c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f190015dd90 0x7f19000ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.833+0000 7f190515c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19000d48f0 con 0x7f19000b6d10 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.833+0000 7f190515c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190015e170 con 0x7f19000ceea0 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.833+0000 7f190515c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19000d4a70 con 0x7f190015dd90 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000b6d10 0x7f19000c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000b6d10 0x7f19000c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48380/0 (socket says 172.21.15.27:48380) 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fed9d640 1 -- 172.21.15.27:0/912313221 learned_addr learned my addr 172.21.15.27:0/912313221 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fe59c640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19000ceea0 0x7f190014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18ff59e640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f190015dd90 0x7f19000ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fed9d640 1 -- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f190015dd90 msgr2=0x7f19000ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fed9d640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f190015dd90 0x7f19000ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fed9d640 1 -- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19000ceea0 msgr2=0x7f190014be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fed9d640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19000ceea0 0x7f190014be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fed9d640 1 -- 172.21.15.27:0/912313221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190014c6a0 con 0x7f19000b6d10 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18fed9d640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000b6d10 0x7f19000c93c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f18f000c430 tx=0x7f18f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18e77fe640 1 -- 172.21.15.27:0/912313221 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f0011790 con 0x7f19000b6d10 2024-03-06T07:23:57.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18e77fe640 1 -- 172.21.15.27:0/912313221 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f0017070 con 0x7f19000b6d10 2024-03-06T07:23:57.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190014c9b0 con 0x7f19000b6d10 2024-03-06T07:23:57.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f18e77fe640 1 -- 172.21.15.27:0/912313221 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f0011bb0 con 0x7f19000b6d10 2024-03-06T07:23:57.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.834+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190014c360 con 0x7f19000b6d10 2024-03-06T07:23:57.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.836+0000 7f18e77fe640 1 -- 172.21.15.27:0/912313221 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18f001c460 con 0x7f19000b6d10 2024-03-06T07:23:57.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.836+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1900072a30 con 0x7f19000b6d10 2024-03-06T07:23:57.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.836+0000 7f18e77fe640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18ec06b4d0 0x7f18ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.836+0000 7f18e77fe640 1 -- 172.21.15.27:0/912313221 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18f0089fb0 con 0x7f19000b6d10 2024-03-06T07:23:57.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.837+0000 7f18fe59c640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18ec06b4d0 0x7f18ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.837+0000 7f18fe59c640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18ec06b4d0 0x7f18ec06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f18f4036000 tx=0x7f18f4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:57.841+0000 7f18e77fe640 1 -- 172.21.15.27:0/912313221 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f190014c9b0 con 0x7f19000b6d10 2024-03-06T07:23:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.060+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f190004f410 con 0x7f19000b6d10 2024-03-06T07:23:58.302 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:23:58.301+0000 7fc51b573640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:58.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.665+0000 7f18e77fe640 1 -- 172.21.15.27:0/912313221 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f18f0058e30 con 0x7f19000b6d10 2024-03-06T07:23:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.667+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18ec06b4d0 msgr2=0x7f18ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.667+0000 7f190515c640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18ec06b4d0 0x7f18ec06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f18f4036000 tx=0x7f18f4008030 comp rx=0 tx=0).stop 2024-03-06T07:23:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.667+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000b6d10 msgr2=0x7f19000c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.667+0000 7f190515c640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000b6d10 0x7f19000c93c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f18f000c430 tx=0x7f18f000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.667+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 shutdown_connections 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.667+0000 7f190515c640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18ec06b4d0 0x7f18ec06d990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.667+0000 7f190515c640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f190015dd90 0x7f19000ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.667+0000 7f190515c640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19000ceea0 0x7f190014be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.667+0000 7f190515c640 1 --2- 172.21.15.27:0/912313221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19000b6d10 0x7f19000c93c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.668+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 >> 172.21.15.27:0/912313221 conn(0x7f1900071960 msgr2=0x7f19000b73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.668+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 shutdown_connections 2024-03-06T07:23:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.668+0000 7f190515c640 1 -- 172.21.15.27:0/912313221 wait complete. 2024-03-06T07:23:58.761 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:23:58.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.867+0000 7eff3bd81640 1 Processor -- start 2024-03-06T07:23:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.868+0000 7eff3bd81640 1 -- start start 2024-03-06T07:23:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.868+0000 7eff3bd81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415df00 0x7eff340d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.868+0000 7eff3bd81640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff340cea30 con 0x7eff340b95a0 2024-03-06T07:23:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.868+0000 7eff3bd81640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff340d2af0 con 0x7eff340b9a30 2024-03-06T07:23:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.868+0000 7eff3bd81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff340cf3c0 con 0x7eff3415df00 2024-03-06T07:23:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.868+0000 7eff3931c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7eff340b95a0 0x7eff340b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36434/0 (socket says 172.21.15.27:36434) 2024-03-06T07:23:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.868+0000 7eff3931c640 1 -- 172.21.15.27:0/1026473768 learned_addr learned my addr 172.21.15.27:0/1026473768 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff39b1d640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415df00 0x7eff340d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545377192 0 0) 0x7eff340cea30 con 0x7eff340b95a0 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff20003520 con 0x7eff340b95a0 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278489384 0 0) 0x7eff340d2af0 con 0x7eff340b9a30 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff340cea30 con 0x7eff340b9a30 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3885913472 0 0) 0x7eff20003520 con 0x7eff340b95a0 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415df00 msgr2=0x7eff340d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415df00 0x7eff340d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 >> v1:172.21.15.69:6789/0 conn(0x7eff340b9a30 legacy=0x7eff340d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340cf0e0 con 0x7eff340b95a0 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7eff24001670 con 0x7eff340b95a0 2024-03-06T07:23:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 >> v1:172.21.15.27:6789/0 conn(0x7eff340b95a0 legacy=0x7eff340b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff20003b70 0x7eff20003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415df00 0x7eff20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff20004d40 0x7eff200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340cf3c0 con 0x7eff20004d40 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2000ba70 con 0x7eff20003b70 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2000bc20 con 0x7eff3415df00 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff3a31e640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff20004d40 0x7eff200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff39b1d640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff20003b70 0x7eff20003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.869+0000 7eff3931c640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415df00 0x7eff20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.870+0000 7eff3a31e640 1 -- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415df00 msgr2=0x7eff20004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.870+0000 7eff3a31e640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415df00 0x7eff20004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.870+0000 7eff3a31e640 1 -- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff20003b70 msgr2=0x7eff20003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.870+0000 7eff3a31e640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff20003b70 0x7eff20003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.870+0000 7eff3a31e640 1 -- 172.21.15.27:0/1026473768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff20003520 con 0x7eff20004d40 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.870+0000 7eff3a31e640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff20004d40 0x7eff200051c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7eff2c001fa0 tx=0x7eff20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f337232184ee17cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c001bf0 con 0x7eff20004d40 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c003b70 con 0x7eff20004d40 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff38b1b640 1 -- 172.21.15.27:0/1026473768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0050d0 con 0x7eff20004d40 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff3bd81640 1 -- 172.21.15.27:0/1026473768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff20004d40 msgr2=0x7eff200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff3bd81640 1 --2- 172.21.15.27:0/1026473768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff20004d40 0x7eff200051c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7eff2c001fa0 tx=0x7eff20005830 comp rx=0 tx=0).stop 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff3a31e640 1 -- 172.21.15.27:0/1026473768 reap_dead start 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff3bd81640 1 -- 172.21.15.27:0/1026473768 shutdown_connections 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff3bd81640 1 -- 172.21.15.27:0/1026473768 >> 172.21.15.27:0/1026473768 conn(0x7eff34071960 msgr2=0x7eff34071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff3bd81640 1 -- 172.21.15.27:0/1026473768 shutdown_connections 2024-03-06T07:23:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.871+0000 7eff3bd81640 1 -- 172.21.15.27:0/1026473768 wait complete. 2024-03-06T07:23:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff3bd81640 1 Processor -- start 2024-03-06T07:23:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff3bd81640 1 -- start start 2024-03-06T07:23:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff3bd81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff34155350 0x7eff3415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff3bd81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415e820 0x7eff340b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff3bd81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff340b55c0 0x7eff340cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff3bd81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340cec70 con 0x7eff34155350 2024-03-06T07:23:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff3bd81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340ce750 con 0x7eff340b55c0 2024-03-06T07:23:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff3bd81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff340cedf0 con 0x7eff3415e820 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff39b1d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff34155350 0x7eff3415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff39b1d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff34155350 0x7eff3415e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48406/0 (socket says 172.21.15.27:48406) 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.873+0000 7eff39b1d640 1 -- 172.21.15.27:0/942282473 learned_addr learned my addr 172.21.15.27:0/942282473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff3931c640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415e820 0x7eff340b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff3a31e640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff340b55c0 0x7eff340cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff39b1d640 1 -- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415e820 msgr2=0x7eff340b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff39b1d640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415e820 0x7eff340b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff39b1d640 1 -- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff340b55c0 msgr2=0x7eff340cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff39b1d640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff340b55c0 0x7eff340cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff39b1d640 1 -- 172.21.15.27:0/942282473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34155730 con 0x7eff34155350 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff3931c640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415e820 0x7eff340b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff3a31e640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff340b55c0 0x7eff340cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff39b1d640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff34155350 0x7eff3415e2e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7eff300119d0 tx=0x7eff30011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff2a7fc640 1 -- 172.21.15.27:0/942282473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3000fb50 con 0x7eff34155350 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff2a7fc640 1 -- 172.21.15.27:0/942282473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3000fcf0 con 0x7eff34155350 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3415efb0 con 0x7eff34155350 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff2a7fc640 1 -- 172.21.15.27:0/942282473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3000fea0 con 0x7eff34155350 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.874+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3415f570 con 0x7eff34155350 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.875+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff341f7d50 con 0x7eff34155350 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.876+0000 7eff2a7fc640 1 -- 172.21.15.27:0/942282473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff30012040 con 0x7eff34155350 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.877+0000 7eff2a7fc640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7eff1806b4d0 0x7eff1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.877+0000 7eff2a7fc640 1 -- 172.21.15.27:0/942282473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff3008a830 con 0x7eff34155350 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.877+0000 7eff3931c640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7eff1806b4d0 0x7eff1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.877+0000 7eff3931c640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7eff1806b4d0 0x7eff1806d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff340d2dd0 tx=0x7eff24009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:58.881+0000 7eff2a7fc640 1 -- 172.21.15.27:0/942282473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff30019030 con 0x7eff34155350 2024-03-06T07:23:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.350+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff34067ff0 con 0x7eff34155350 2024-03-06T07:23:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.351+0000 7eff2a7fc640 1 -- 172.21.15.27:0/942282473 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7eff30059740 con 0x7eff34155350 2024-03-06T07:23:59.353 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:59.353 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:23:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7eff1806b4d0 msgr2=0x7eff1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7eff1806b4d0 0x7eff1806d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff340d2dd0 tx=0x7eff24009040 comp rx=0 tx=0).stop 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff34155350 msgr2=0x7eff3415e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff34155350 0x7eff3415e2e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7eff300119d0 tx=0x7eff30011e90 comp rx=0 tx=0).stop 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 shutdown_connections 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7eff1806b4d0 0x7eff1806d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff340b55c0 0x7eff340cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7eff3415e820 0x7eff340b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 --2- 172.21.15.27:0/942282473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff34155350 0x7eff3415e2e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 >> 172.21.15.27:0/942282473 conn(0x7eff34071960 msgr2=0x7eff340b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 shutdown_connections 2024-03-06T07:23:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.353+0000 7eff3bd81640 1 -- 172.21.15.27:0/942282473 wait complete. 2024-03-06T07:23:59.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:59.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.476+0000 7f03d6cf4640 1 Processor -- start 2024-03-06T07:23:59.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.476+0000 7f03d6cf4640 1 -- start start 2024-03-06T07:23:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.476+0000 7f03d6cf4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f03d0071540 0x7f03d00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.476+0000 7f03d6cf4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f03d00b2b50 0x7f03d00b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.476+0000 7f03d6cf4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d00ce610 con 0x7f03d00d3400 2024-03-06T07:23:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.476+0000 7f03d6cf4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d00d2f70 con 0x7f03d0071540 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.476+0000 7f03d6cf4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d00d4be0 con 0x7f03d00b2b50 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.477+0000 7f03d4a90640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f03d0071540 0x7f03d00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.477+0000 7f03d4a90640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f03d0071540 0x7f03d00b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48420/0 (socket says 172.21.15.27:48420) 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.477+0000 7f03d4a90640 1 -- 172.21.15.27:0/4009908116 learned_addr learned my addr 172.21.15.27:0/4009908116 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.477+0000 7f03d5291640 1 --2- 172.21.15.27:0/4009908116 >> v2:172.21.15.69:3300/0 conn(0x7f03d00b2b50 0x7f03d00b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.477+0000 7f03d4a90640 1 -- 172.21.15.27:0/4009908116 >> v2:172.21.15.69:3300/0 conn(0x7f03d00b2b50 msgr2=0x7f03d00b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.477+0000 7f03d4a90640 1 --2- 172.21.15.27:0/4009908116 >> v2:172.21.15.69:3300/0 conn(0x7f03d00b2b50 0x7f03d00b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.477+0000 7f03d4a90640 1 -- 172.21.15.27:0/4009908116 >> v1:172.21.15.69:6789/0 conn(0x7f03d00d3400 legacy=0x7f03d0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.477+0000 7f03d4a90640 1 -- 172.21.15.27:0/4009908116 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d00d2920 con 0x7f03d0071540 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.477+0000 7f03d5291640 1 --2- 172.21.15.27:0/4009908116 >> v2:172.21.15.69:3300/0 conn(0x7f03d00b2b50 0x7f03d00b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.478+0000 7f03d4a90640 1 --2- 172.21.15.27:0/4009908116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d0071540 0x7f03d00b30a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f03d00683b0 tx=0x7f03c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722f526238d7f558 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.478+0000 7f03cf7fe640 1 -- 172.21.15.27:0/4009908116 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4034070 con 0x7f03d0071540 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.478+0000 7f03cf7fe640 1 -- 172.21.15.27:0/4009908116 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c4030860 con 0x7f03d0071540 2024-03-06T07:23:59.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.478+0000 7f03cf7fe640 1 -- 172.21.15.27:0/4009908116 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4030c30 con 0x7f03d0071540 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.478+0000 7f03d6cf4640 1 -- 172.21.15.27:0/4009908116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d0071540 msgr2=0x7f03d00b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.478+0000 7f03d6cf4640 1 --2- 172.21.15.27:0/4009908116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d0071540 0x7f03d00b30a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f03d00683b0 tx=0x7f03c402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.479+0000 7f03d6cf4640 1 -- 172.21.15.27:0/4009908116 shutdown_connections 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.479+0000 7f03d6cf4640 1 --2- 172.21.15.27:0/4009908116 >> v2:172.21.15.69:3300/0 conn(0x7f03d00b2b50 0x7f03d00b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.479+0000 7f03d6cf4640 1 --2- 172.21.15.27:0/4009908116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d0071540 0x7f03d00b30a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.479+0000 7f03d6cf4640 1 -- 172.21.15.27:0/4009908116 >> 172.21.15.27:0/4009908116 conn(0x7f03d00d3ca0 msgr2=0x7f03d00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.479+0000 7f03d6cf4640 1 -- 172.21.15.27:0/4009908116 shutdown_connections 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.479+0000 7f03d6cf4640 1 -- 172.21.15.27:0/4009908116 wait complete. 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.480+0000 7f03d6cf4640 1 Processor -- start 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.480+0000 7f03d6cf4640 1 -- start start 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03d6cf4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03d0071540 0x7f03d00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03d6cf4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d00b2b50 0x7f03d00cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03d6cf4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03d00d3400 0x7f03d014d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03d6cf4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d015e370 con 0x7f03d00b2b50 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03d6cf4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00ce8f0 con 0x7f03d0071540 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03d6cf4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00b5d20 con 0x7f03d00d3400 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d00b2b50 0x7f03d00cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d00b2b50 0x7f03d00cdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48426/0 (socket says 172.21.15.27:48426) 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cffff640 1 -- 172.21.15.27:0/3545495789 learned_addr learned my addr 172.21.15.27:0/3545495789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03d5291640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03d00d3400 0x7f03d014d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03d4a90640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03d0071540 0x7f03d00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cffff640 1 -- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03d00d3400 msgr2=0x7f03d014d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cffff640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03d00d3400 0x7f03d014d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cffff640 1 -- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03d0071540 msgr2=0x7f03d00b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cffff640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03d0071540 0x7f03d00b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cffff640 1 -- 172.21.15.27:0/3545495789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d00ce220 con 0x7f03d00b2b50 2024-03-06T07:23:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03d4a90640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03d0071540 0x7f03d00b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cffff640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d00b2b50 0x7f03d00cdce0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c440 tx=0x7f03bc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cd7fa640 1 -- 172.21.15.27:0/3545495789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0052d0 con 0x7f03d00b2b50 2024-03-06T07:23:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.481+0000 7f03cd7fa640 1 -- 172.21.15.27:0/3545495789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc005470 con 0x7f03d00b2b50 2024-03-06T07:23:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.482+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d00b5110 con 0x7f03d00b2b50 2024-03-06T07:23:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.482+0000 7f03cd7fa640 1 -- 172.21.15.27:0/3545495789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc007260 con 0x7f03d00b2b50 2024-03-06T07:23:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.482+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d00b56d0 con 0x7f03d00b2b50 2024-03-06T07:23:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.483+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d0068700 con 0x7f03d00b2b50 2024-03-06T07:23:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.484+0000 7f03cd7fa640 1 -- 172.21.15.27:0/3545495789 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03bc007400 con 0x7f03d00b2b50 2024-03-06T07:23:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.484+0000 7f03cd7fa640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f03b406b500 0x7f03b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.484+0000 7f03cd7fa640 1 -- 172.21.15.27:0/3545495789 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03bc089580 con 0x7f03d00b2b50 2024-03-06T07:23:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.484+0000 7f03d4a90640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f03b406b500 0x7f03b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.485+0000 7f03d4a90640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f03b406b500 0x7f03b406d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f03c4036000 tx=0x7f03c403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.489+0000 7f03cd7fa640 1 -- 172.21.15.27:0/3545495789 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03bc057db0 con 0x7f03d00b2b50 2024-03-06T07:23:59.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.777+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03d00b7940 con 0x7f03d00b2b50 2024-03-06T07:23:59.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.779+0000 7f03cd7fa640 1 -- 172.21.15.27:0/3545495789 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f03bc05be50 con 0x7f03d00b2b50 2024-03-06T07:23:59.781 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:23:59.781 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":35,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:56.637219+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:53.309610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:53.987106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f03b406b500 msgr2=0x7f03b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f03b406b500 0x7f03b406d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f03c4036000 tx=0x7f03c403b040 comp rx=0 tx=0).stop 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d00b2b50 msgr2=0x7f03d00cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d00b2b50 0x7f03d00cdce0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c440 tx=0x7f03bc00c900 comp rx=0 tx=0).stop 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 shutdown_connections 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f03b406b500 0x7f03b406d9c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f03d00d3400 0x7f03d014d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03d00b2b50 0x7f03d00cdce0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 --2- 172.21.15.27:0/3545495789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03d0071540 0x7f03d00b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 >> 172.21.15.27:0/3545495789 conn(0x7f03d00d3ca0 msgr2=0x7f03d015f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 shutdown_connections 2024-03-06T07:23:59.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.781+0000 7f03d6cf4640 1 -- 172.21.15.27:0/3545495789 wait complete. 2024-03-06T07:23:59.795 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:59.795 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.900+0000 7f562b8c9640 1 Processor -- start 2024-03-06T07:23:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.901+0000 7f562b8c9640 1 -- start start 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f562b8c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5624071540 0x7f56240b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f562b8c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56240b2b50 0x7f56240b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f562b8c9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56240ce610 con 0x7f56240d3400 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f562b8c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56240d2f70 con 0x7f56240b2b50 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f562b8c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56240d4be0 con 0x7f5624071540 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f5629e66640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56240b2b50 0x7f56240b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f5628e64640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f56240d3400 0x7f5624071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36446/0 (socket says 172.21.15.27:36446) 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f5628e64640 1 -- 172.21.15.27:0/2955268113 learned_addr learned my addr 172.21.15.27:0/2955268113 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f5629665640 1 --2- 172.21.15.27:0/2955268113 >> v2:172.21.15.69:3300/0 conn(0x7f5624071540 0x7f56240b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f5629e66640 1 -- 172.21.15.27:0/2955268113 >> v2:172.21.15.69:3300/0 conn(0x7f5624071540 msgr2=0x7f56240b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f5629e66640 1 --2- 172.21.15.27:0/2955268113 >> v2:172.21.15.69:3300/0 conn(0x7f5624071540 0x7f56240b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f5629e66640 1 -- 172.21.15.27:0/2955268113 >> v1:172.21.15.27:6789/0 conn(0x7f56240d3400 legacy=0x7f5624071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f5629e66640 1 -- 172.21.15.27:0/2955268113 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56240d2920 con 0x7f56240b2b50 2024-03-06T07:23:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.902+0000 7f5629665640 1 --2- 172.21.15.27:0/2955268113 >> v2:172.21.15.69:3300/0 conn(0x7f5624071540 0x7f56240b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.903+0000 7f5629e66640 1 --2- 172.21.15.27:0/2955268113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56240b2b50 0x7f56240b6e20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f56240d4da0 tx=0x7f561c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e2665d8dd5c467 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.903+0000 7f561bfff640 1 -- 172.21.15.27:0/2955268113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c039070 con 0x7f56240b2b50 2024-03-06T07:23:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.903+0000 7f561bfff640 1 -- 172.21.15.27:0/2955268113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c02cc80 con 0x7f56240b2b50 2024-03-06T07:23:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.903+0000 7f561bfff640 1 -- 172.21.15.27:0/2955268113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c034660 con 0x7f56240b2b50 2024-03-06T07:23:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.904+0000 7f562b8c9640 1 -- 172.21.15.27:0/2955268113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56240b2b50 msgr2=0x7f56240b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.904+0000 7f562b8c9640 1 --2- 172.21.15.27:0/2955268113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56240b2b50 0x7f56240b6e20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f56240d4da0 tx=0x7f561c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.904+0000 7f562b8c9640 1 -- 172.21.15.27:0/2955268113 shutdown_connections 2024-03-06T07:23:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.904+0000 7f562b8c9640 1 --2- 172.21.15.27:0/2955268113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56240b2b50 0x7f56240b6e20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.904+0000 7f562b8c9640 1 --2- 172.21.15.27:0/2955268113 >> v2:172.21.15.69:3300/0 conn(0x7f5624071540 0x7f56240b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.904+0000 7f562b8c9640 1 -- 172.21.15.27:0/2955268113 >> 172.21.15.27:0/2955268113 conn(0x7f56240d3ca0 msgr2=0x7f56240d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.904+0000 7f562b8c9640 1 -- 172.21.15.27:0/2955268113 shutdown_connections 2024-03-06T07:23:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.904+0000 7f562b8c9640 1 -- 172.21.15.27:0/2955268113 wait complete. 2024-03-06T07:23:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.906+0000 7f562b8c9640 1 Processor -- start 2024-03-06T07:23:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.906+0000 7f562b8c9640 1 -- start start 2024-03-06T07:23:59.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.907+0000 7f562b8c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5624071540 0x7f56240b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.907+0000 7f562b8c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56240b2b50 0x7f56241f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.907+0000 7f562b8c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56240d3400 0x7f5624150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.907+0000 7f562b8c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562415e370 con 0x7f5624071540 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.907+0000 7f562b8c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56240ce8f0 con 0x7f56240d3400 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.907+0000 7f562b8c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56240c7e50 con 0x7f56240b2b50 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629665640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5624071540 0x7f56240b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629665640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5624071540 0x7f56240b4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48448/0 (socket says 172.21.15.27:48448) 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629665640 1 -- 172.21.15.27:0/2924857764 learned_addr learned my addr 172.21.15.27:0/2924857764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5628e64640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56240b2b50 0x7f56241f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629e66640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56240d3400 0x7f5624150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629665640 1 -- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56240b2b50 msgr2=0x7f56241f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629665640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56240b2b50 0x7f56241f3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629665640 1 -- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56240d3400 msgr2=0x7f5624150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629665640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56240d3400 0x7f5624150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629665640 1 -- 172.21.15.27:0/2924857764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56240cd0b0 con 0x7f5624071540 2024-03-06T07:23:59.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629e66640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56240d3400 0x7f5624150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:59.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5629665640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5624071540 0x7f56240b4ee0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f562000c7f0 tx=0x7f562000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5619ffb640 1 -- 172.21.15.27:0/2924857764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620015070 con 0x7f5624071540 2024-03-06T07:23:59.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5619ffb640 1 -- 172.21.15.27:0/2924857764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5620007a30 con 0x7f5624071540 2024-03-06T07:23:59.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56240cd390 con 0x7f5624071540 2024-03-06T07:23:59.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.908+0000 7f5619ffb640 1 -- 172.21.15.27:0/2924857764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620007e00 con 0x7f5624071540 2024-03-06T07:23:59.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.909+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56241f4010 con 0x7f5624071540 2024-03-06T07:23:59.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.910+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5624068700 con 0x7f5624071540 2024-03-06T07:23:59.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.910+0000 7f5619ffb640 1 -- 172.21.15.27:0/2924857764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5620018070 con 0x7f5624071540 2024-03-06T07:23:59.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.910+0000 7f5619ffb640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f560806b500 0x7f560806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.910+0000 7f5619ffb640 1 -- 172.21.15.27:0/2924857764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f562008a120 con 0x7f5624071540 2024-03-06T07:23:59.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.911+0000 7f5628e64640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f560806b500 0x7f560806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.911+0000 7f5628e64640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f560806b500 0x7f560806d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f562415acc0 tx=0x7f5614009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:23:59.915+0000 7f5619ffb640 1 -- 172.21.15.27:0/2924857764 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5620059030 con 0x7f5624071540 2024-03-06T07:24:00.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.127+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56240b3140 con 0x7f560806b500 2024-03-06T07:24:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.133+0000 7f5619ffb640 1 -- 172.21.15.27:0/2924857764 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246634 (secure 0 0 0) 0x7f56240b3140 con 0x7f560806b500 2024-03-06T07:24:00.139 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:00.140 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:24:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f560806b500 msgr2=0x7f560806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f560806b500 0x7f560806d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f562415acc0 tx=0x7f5614009040 comp rx=0 tx=0).stop 2024-03-06T07:24:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5624071540 msgr2=0x7f56240b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5624071540 0x7f56240b4ee0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f562000c7f0 tx=0x7f562000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 shutdown_connections 2024-03-06T07:24:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f560806b500 0x7f560806d9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56240d3400 0x7f5624150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56240b2b50 0x7f56241f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 --2- 172.21.15.27:0/2924857764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5624071540 0x7f56240b4ee0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 >> 172.21.15.27:0/2924857764 conn(0x7f56240d3ca0 msgr2=0x7f562415e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 shutdown_connections 2024-03-06T07:24:00.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.136+0000 7f562b8c9640 1 -- 172.21.15.27:0/2924857764 wait complete. 2024-03-06T07:24:00.149 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-03-06T07:24:00.002784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220172,"kb_used_data":4012,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754548,"statfs":{"total":773094113280,"available":772868657152,"internally_reserved":0,"allocated":4108288,"data_stored":2297617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001489"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646720+0000","last_change":"2024-03-06T07:23:54.646720+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646720+0000","last_undegraded":"2024-03-06T07:23:54.646720+0000","last_fullsized":"2024-03-06T07:23:54.646720+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644574+0000","last_change":"2024-03-06T07:23:54.644574+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644574+0000","last_undegraded":"2024-03-06T07:23:54.644574+0000","last_fullsized":"2024-03-06T07:23:54.644574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643477+0000","last_change":"2024-03-06T07:23:54.643477+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643477+0000","last_undegraded":"2024-03-06T07:23:54.643477+0000","last_fullsized":"2024-03-06T07:23:54.643477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648478+0000","last_change":"2024-03-06T07:23:54.648478+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648478+0000","last_undegraded":"2024-03-06T07:23:54.648478+0000","last_fullsized":"2024-03-06T07:23:54.648478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643699+0000","last_change":"2024-03-06T07:23:54.643699+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643699+0000","last_undegraded":"2024-03-06T07:23:54.643699+0000","last_fullsized":"2024-03-06T07:23:54.643699+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641775+0000","last_change":"2024-03-06T07:23:54.641775+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641775+0000","last_undegraded":"2024-03-06T07:23:54.641775+0000","last_fullsized":"2024-03-06T07:23:54.641775+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645152+0000","last_change":"2024-03-06T07:23:54.645152+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645152+0000","last_undegraded":"2024-03-06T07:23:54.645152+0000","last_fullsized":"2024-03-06T07:23:54.645152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647218+0000","last_change":"2024-03-06T07:23:54.647218+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647218+0000","last_undegraded":"2024-03-06T07:23:54.647218+0000","last_fullsized":"2024-03-06T07:23:54.647218+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640159+0000","last_change":"2024-03-06T07:23:54.640159+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640159+0000","last_undegraded":"2024-03-06T07:23:54.640159+0000","last_fullsized":"2024-03-06T07:23:54.640159+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648487+0000","last_change":"2024-03-06T07:23:54.648487+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648487+0000","last_undegraded":"2024-03-06T07:23:54.648487+0000","last_fullsized":"2024-03-06T07:23:54.648487+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646918+0000","last_change":"2024-03-06T07:23:54.646918+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646918+0000","last_undegraded":"2024-03-06T07:23:54.646918+0000","last_fullsized":"2024-03-06T07:23:54.646918+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643738+0000","last_change":"2024-03-06T07:23:54.643738+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643738+0000","last_undegraded":"2024-03-06T07:23:54.643738+0000","last_fullsized":"2024-03-06T07:23:54.643738+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643236+0000","last_change":"2024-03-06T07:23:54.643236+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643236+0000","last_undegraded":"2024-03-06T07:23:54.643236+0000","last_fullsized":"2024-03-06T07:23:54.643236+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641298+0000","last_change":"2024-03-06T07:23:54.641298+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641298+0000","last_undegraded":"2024-03-06T07:23:54.641298+0000","last_fullsized":"2024-03-06T07:23:54.641298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646515+0000","last_change":"2024-03-06T07:23:54.646515+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646515+0000","last_undegraded":"2024-03-06T07:23:54.646515+0000","last_fullsized":"2024-03-06T07:23:54.646515+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644556+0000","last_change":"2024-03-06T07:23:54.644556+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644556+0000","last_undegraded":"2024-03-06T07:23:54.644556+0000","last_fullsized":"2024-03-06T07:23:54.644556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646653+0000","last_change":"2024-03-06T07:23:54.646653+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646653+0000","last_undegraded":"2024-03-06T07:23:54.646653+0000","last_fullsized":"2024-03-06T07:23:54.646653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644109+0000","last_change":"2024-03-06T07:23:54.644109+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644109+0000","last_undegraded":"2024-03-06T07:23:54.644109+0000","last_fullsized":"2024-03-06T07:23:54.644109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640796+0000","last_change":"2024-03-06T07:23:54.640796+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640796+0000","last_undegraded":"2024-03-06T07:23:54.640796+0000","last_fullsized":"2024-03-06T07:23:54.640796+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643640+0000","last_change":"2024-03-06T07:23:54.643640+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643640+0000","last_undegraded":"2024-03-06T07:23:54.643640+0000","last_fullsized":"2024-03-06T07:23:54.643640+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641081+0000","last_change":"2024-03-06T07:23:54.641081+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641081+0000","last_undegraded":"2024-03-06T07:23:54.641081+0000","last_fullsized":"2024-03-06T07:23:54.641081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.639615+0000","last_change":"2024-03-06T07:23:54.639615+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.639615+0000","last_undegraded":"2024-03-06T07:23:54.639615+0000","last_fullsized":"2024-03-06T07:23:54.639615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646031+0000","last_change":"2024-03-06T07:23:54.646031+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646031+0000","last_undegraded":"2024-03-06T07:23:54.646031+0000","last_fullsized":"2024-03-06T07:23:54.646031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642902+0000","last_change":"2024-03-06T07:23:54.642902+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642902+0000","last_undegraded":"2024-03-06T07:23:54.642902+0000","last_fullsized":"2024-03-06T07:23:54.642902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648463+0000","last_change":"2024-03-06T07:23:54.648463+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648463+0000","last_undegraded":"2024-03-06T07:23:54.648463+0000","last_fullsized":"2024-03-06T07:23:54.648463+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643640+0000","last_change":"2024-03-06T07:23:54.643640+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643640+0000","last_undegraded":"2024-03-06T07:23:54.643640+0000","last_fullsized":"2024-03-06T07:23:54.643640+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647824+0000","last_change":"2024-03-06T07:23:54.647824+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647824+0000","last_undegraded":"2024-03-06T07:23:54.647824+0000","last_fullsized":"2024-03-06T07:23:54.647824+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643326+0000","last_change":"2024-03-06T07:23:54.643326+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643326+0000","last_undegraded":"2024-03-06T07:23:54.643326+0000","last_fullsized":"2024-03-06T07:23:54.643326+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640572+0000","last_change":"2024-03-06T07:23:54.640572+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640572+0000","last_undegraded":"2024-03-06T07:23:54.640572+0000","last_fullsized":"2024-03-06T07:23:54.640572+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645644+0000","last_change":"2024-03-06T07:23:54.645644+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645644+0000","last_undegraded":"2024-03-06T07:23:54.645644+0000","last_fullsized":"2024-03-06T07:23:54.645644+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647662+0000","last_change":"2024-03-06T07:23:54.647662+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647662+0000","last_undegraded":"2024-03-06T07:23:54.647662+0000","last_fullsized":"2024-03-06T07:23:54.647662+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646477+0000","last_change":"2024-03-06T07:23:54.646477+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646477+0000","last_undegraded":"2024-03-06T07:23:54.646477+0000","last_fullsized":"2024-03-06T07:23:54.646477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647114+0000","last_change":"2024-03-06T07:23:54.647114+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647114+0000","last_undegraded":"2024-03-06T07:23:54.647114+0000","last_fullsized":"2024-03-06T07:23:54.647114+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656288+0000","last_change":"2024-03-06T07:23:54.655354+0000","last_active":"2024-03-06T07:23:54.656288+0000","last_peered":"2024-03-06T07:23:54.656288+0000","last_clean":"2024-03-06T07:23:54.656288+0000","last_became_active":"2024-03-06T07:23:54.655053+0000","last_became_peered":"2024-03-06T07:23:54.655053+0000","last_unstale":"2024-03-06T07:23:54.656288+0000","last_undegraded":"2024-03-06T07:23:54.656288+0000","last_fullsized":"2024-03-06T07:23:54.656288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:19.451847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644314+0000","last_change":"2024-03-06T07:23:54.644314+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644314+0000","last_undegraded":"2024-03-06T07:23:54.644314+0000","last_fullsized":"2024-03-06T07:23:54.644314+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663137+0000","last_change":"2024-03-06T07:23:54.662097+0000","last_active":"2024-03-06T07:23:54.663137+0000","last_peered":"2024-03-06T07:23:54.663137+0000","last_clean":"2024-03-06T07:23:54.663137+0000","last_became_active":"2024-03-06T07:23:54.661544+0000","last_became_peered":"2024-03-06T07:23:54.661544+0000","last_unstale":"2024-03-06T07:23:54.663137+0000","last_undegraded":"2024-03-06T07:23:54.663137+0000","last_fullsized":"2024-03-06T07:23:54.663137+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.639780+0000","last_change":"2024-03-06T07:23:54.639780+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.639780+0000","last_undegraded":"2024-03-06T07:23:54.639780+0000","last_fullsized":"2024-03-06T07:23:54.639780+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656144+0000","last_change":"2024-03-06T07:23:54.654853+0000","last_active":"2024-03-06T07:23:54.656144+0000","last_peered":"2024-03-06T07:23:54.656144+0000","last_clean":"2024-03-06T07:23:54.656144+0000","last_became_active":"2024-03-06T07:23:54.654566+0000","last_became_peered":"2024-03-06T07:23:54.654566+0000","last_unstale":"2024-03-06T07:23:54.656144+0000","last_undegraded":"2024-03-06T07:23:54.656144+0000","last_fullsized":"2024-03-06T07:23:54.656144+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:13.482463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647238+0000","last_change":"2024-03-06T07:23:54.647238+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647238+0000","last_undegraded":"2024-03-06T07:23:54.647238+0000","last_fullsized":"2024-03-06T07:23:54.647238+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.661489+0000","last_change":"2024-03-06T07:23:54.650365+0000","last_active":"2024-03-06T07:23:54.661489+0000","last_peered":"2024-03-06T07:23:54.661489+0000","last_clean":"2024-03-06T07:23:54.661489+0000","last_became_active":"2024-03-06T07:23:54.650077+0000","last_became_peered":"2024-03-06T07:23:54.650077+0000","last_unstale":"2024-03-06T07:23:54.661489+0000","last_undegraded":"2024-03-06T07:23:54.661489+0000","last_fullsized":"2024-03-06T07:23:54.661489+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:26.696172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645225+0000","last_change":"2024-03-06T07:23:54.645225+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645225+0000","last_undegraded":"2024-03-06T07:23:54.645225+0000","last_fullsized":"2024-03-06T07:23:54.645225+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654149+0000","last_change":"2024-03-06T07:23:54.652376+0000","last_active":"2024-03-06T07:23:54.654149+0000","last_peered":"2024-03-06T07:23:54.654149+0000","last_clean":"2024-03-06T07:23:54.654149+0000","last_became_active":"2024-03-06T07:23:54.652121+0000","last_became_peered":"2024-03-06T07:23:54.652121+0000","last_unstale":"2024-03-06T07:23:54.654149+0000","last_undegraded":"2024-03-06T07:23:54.654149+0000","last_fullsized":"2024-03-06T07:23:54.654149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:26.696172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643151+0000","last_change":"2024-03-06T07:23:54.643151+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643151+0000","last_undegraded":"2024-03-06T07:23:54.643151+0000","last_fullsized":"2024-03-06T07:23:54.643151+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654078+0000","last_change":"2024-03-06T07:23:54.652687+0000","last_active":"2024-03-06T07:23:54.654078+0000","last_peered":"2024-03-06T07:23:54.654078+0000","last_clean":"2024-03-06T07:23:54.654078+0000","last_became_active":"2024-03-06T07:23:54.652023+0000","last_became_peered":"2024-03-06T07:23:54.652023+0000","last_unstale":"2024-03-06T07:23:54.654078+0000","last_undegraded":"2024-03-06T07:23:54.654078+0000","last_fullsized":"2024-03-06T07:23:54.654078+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:37.112364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640802+0000","last_change":"2024-03-06T07:23:54.640802+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640802+0000","last_undegraded":"2024-03-06T07:23:54.640802+0000","last_fullsized":"2024-03-06T07:23:54.640802+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653032+0000","last_change":"2024-03-06T07:23:54.651953+0000","last_active":"2024-03-06T07:23:54.653032+0000","last_peered":"2024-03-06T07:23:54.653032+0000","last_clean":"2024-03-06T07:23:54.653032+0000","last_became_active":"2024-03-06T07:23:54.651636+0000","last_became_peered":"2024-03-06T07:23:54.651636+0000","last_unstale":"2024-03-06T07:23:54.653032+0000","last_undegraded":"2024-03-06T07:23:54.653032+0000","last_fullsized":"2024-03-06T07:23:54.653032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646552+0000","last_change":"2024-03-06T07:23:54.646552+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646552+0000","last_undegraded":"2024-03-06T07:23:54.646552+0000","last_fullsized":"2024-03-06T07:23:54.646552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655564+0000","last_change":"2024-03-06T07:23:54.653509+0000","last_active":"2024-03-06T07:23:54.655564+0000","last_peered":"2024-03-06T07:23:54.655564+0000","last_clean":"2024-03-06T07:23:54.655564+0000","last_became_active":"2024-03-06T07:23:54.651897+0000","last_became_peered":"2024-03-06T07:23:54.651897+0000","last_unstale":"2024-03-06T07:23:54.655564+0000","last_undegraded":"2024-03-06T07:23:54.655564+0000","last_fullsized":"2024-03-06T07:23:54.655564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:29.745505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642849+0000","last_change":"2024-03-06T07:23:54.642849+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642849+0000","last_undegraded":"2024-03-06T07:23:54.642849+0000","last_fullsized":"2024-03-06T07:23:54.642849+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656057+0000","last_change":"2024-03-06T07:23:54.653695+0000","last_active":"2024-03-06T07:23:54.656057+0000","last_peered":"2024-03-06T07:23:54.656057+0000","last_clean":"2024-03-06T07:23:54.656057+0000","last_became_active":"2024-03-06T07:23:54.653405+0000","last_became_peered":"2024-03-06T07:23:54.653405+0000","last_unstale":"2024-03-06T07:23:54.656057+0000","last_undegraded":"2024-03-06T07:23:54.656057+0000","last_fullsized":"2024-03-06T07:23:54.656057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:05.867698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642632+0000","last_change":"2024-03-06T07:23:54.642632+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642632+0000","last_undegraded":"2024-03-06T07:23:54.642632+0000","last_fullsized":"2024-03-06T07:23:54.642632+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663079+0000","last_change":"2024-03-06T07:23:54.661840+0000","last_active":"2024-03-06T07:23:54.663079+0000","last_peered":"2024-03-06T07:23:54.663079+0000","last_clean":"2024-03-06T07:23:54.663079+0000","last_became_active":"2024-03-06T07:23:54.661153+0000","last_became_peered":"2024-03-06T07:23:54.661153+0000","last_unstale":"2024-03-06T07:23:54.663079+0000","last_undegraded":"2024-03-06T07:23:54.663079+0000","last_fullsized":"2024-03-06T07:23:54.663079+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:13.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640747+0000","last_change":"2024-03-06T07:23:54.640747+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640747+0000","last_undegraded":"2024-03-06T07:23:54.640747+0000","last_fullsized":"2024-03-06T07:23:54.640747+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653444+0000","last_change":"2024-03-06T07:23:54.652357+0000","last_active":"2024-03-06T07:23:54.653444+0000","last_peered":"2024-03-06T07:23:54.653444+0000","last_clean":"2024-03-06T07:23:54.653444+0000","last_became_active":"2024-03-06T07:23:54.652092+0000","last_became_peered":"2024-03-06T07:23:54.652092+0000","last_unstale":"2024-03-06T07:23:54.653444+0000","last_undegraded":"2024-03-06T07:23:54.653444+0000","last_fullsized":"2024-03-06T07:23:54.653444+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:24.364320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643899+0000","last_change":"2024-03-06T07:23:54.643899+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643899+0000","last_undegraded":"2024-03-06T07:23:54.643899+0000","last_fullsized":"2024-03-06T07:23:54.643899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.657787+0000","last_change":"2024-03-06T07:23:54.656641+0000","last_active":"2024-03-06T07:23:54.657787+0000","last_peered":"2024-03-06T07:23:54.657787+0000","last_clean":"2024-03-06T07:23:54.657787+0000","last_became_active":"2024-03-06T07:23:54.656285+0000","last_became_peered":"2024-03-06T07:23:54.656285+0000","last_unstale":"2024-03-06T07:23:54.657787+0000","last_undegraded":"2024-03-06T07:23:54.657787+0000","last_fullsized":"2024-03-06T07:23:54.657787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:08.762797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648064+0000","last_change":"2024-03-06T07:23:54.648064+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648064+0000","last_undegraded":"2024-03-06T07:23:54.648064+0000","last_fullsized":"2024-03-06T07:23:54.648064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654119+0000","last_change":"2024-03-06T07:23:54.651813+0000","last_active":"2024-03-06T07:23:54.654119+0000","last_peered":"2024-03-06T07:23:54.654119+0000","last_clean":"2024-03-06T07:23:54.654119+0000","last_became_active":"2024-03-06T07:23:54.651504+0000","last_became_peered":"2024-03-06T07:23:54.651504+0000","last_unstale":"2024-03-06T07:23:54.654119+0000","last_undegraded":"2024-03-06T07:23:54.654119+0000","last_fullsized":"2024-03-06T07:23:54.654119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:07.864613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642671+0000","last_change":"2024-03-06T07:23:54.642671+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642671+0000","last_undegraded":"2024-03-06T07:23:54.642671+0000","last_fullsized":"2024-03-06T07:23:54.642671+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655764+0000","last_change":"2024-03-06T07:23:54.652295+0000","last_active":"2024-03-06T07:23:54.655764+0000","last_peered":"2024-03-06T07:23:54.655764+0000","last_clean":"2024-03-06T07:23:54.655764+0000","last_became_active":"2024-03-06T07:23:54.652029+0000","last_became_peered":"2024-03-06T07:23:54.652029+0000","last_unstale":"2024-03-06T07:23:54.655764+0000","last_undegraded":"2024-03-06T07:23:54.655764+0000","last_fullsized":"2024-03-06T07:23:54.655764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646070+0000","last_change":"2024-03-06T07:23:54.646070+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646070+0000","last_undegraded":"2024-03-06T07:23:54.646070+0000","last_fullsized":"2024-03-06T07:23:54.646070+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653007+0000","last_change":"2024-03-06T07:23:54.651884+0000","last_active":"2024-03-06T07:23:54.653007+0000","last_peered":"2024-03-06T07:23:54.653007+0000","last_clean":"2024-03-06T07:23:54.653007+0000","last_became_active":"2024-03-06T07:23:54.651584+0000","last_became_peered":"2024-03-06T07:23:54.651584+0000","last_unstale":"2024-03-06T07:23:54.653007+0000","last_undegraded":"2024-03-06T07:23:54.653007+0000","last_fullsized":"2024-03-06T07:23:54.653007+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646049+0000","last_change":"2024-03-06T07:23:54.646049+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646049+0000","last_undegraded":"2024-03-06T07:23:54.646049+0000","last_fullsized":"2024-03-06T07:23:54.646049+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658202+0000","last_change":"2024-03-06T07:23:54.657051+0000","last_active":"2024-03-06T07:23:54.658202+0000","last_peered":"2024-03-06T07:23:54.658202+0000","last_clean":"2024-03-06T07:23:54.658202+0000","last_became_active":"2024-03-06T07:23:54.656681+0000","last_became_peered":"2024-03-06T07:23:54.656681+0000","last_unstale":"2024-03-06T07:23:54.658202+0000","last_undegraded":"2024-03-06T07:23:54.658202+0000","last_fullsized":"2024-03-06T07:23:54.658202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:43.882962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640452+0000","last_change":"2024-03-06T07:23:54.640452+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640452+0000","last_undegraded":"2024-03-06T07:23:54.640452+0000","last_fullsized":"2024-03-06T07:23:54.640452+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658778+0000","last_change":"2024-03-06T07:23:54.657746+0000","last_active":"2024-03-06T07:23:54.658778+0000","last_peered":"2024-03-06T07:23:54.658778+0000","last_clean":"2024-03-06T07:23:54.658778+0000","last_became_active":"2024-03-06T07:23:54.657422+0000","last_became_peered":"2024-03-06T07:23:54.657422+0000","last_unstale":"2024-03-06T07:23:54.658778+0000","last_undegraded":"2024-03-06T07:23:54.658778+0000","last_fullsized":"2024-03-06T07:23:54.658778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645677+0000","last_change":"2024-03-06T07:23:54.645677+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645677+0000","last_undegraded":"2024-03-06T07:23:54.645677+0000","last_fullsized":"2024-03-06T07:23:54.645677+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654312+0000","last_change":"2024-03-06T07:23:54.653106+0000","last_active":"2024-03-06T07:23:54.654312+0000","last_peered":"2024-03-06T07:23:54.654312+0000","last_clean":"2024-03-06T07:23:54.654312+0000","last_became_active":"2024-03-06T07:23:54.651460+0000","last_became_peered":"2024-03-06T07:23:54.651460+0000","last_unstale":"2024-03-06T07:23:54.654312+0000","last_undegraded":"2024-03-06T07:23:54.654312+0000","last_fullsized":"2024-03-06T07:23:54.654312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.793262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642632+0000","last_change":"2024-03-06T07:23:54.642632+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642632+0000","last_undegraded":"2024-03-06T07:23:54.642632+0000","last_fullsized":"2024-03-06T07:23:54.642632+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.662455+0000","last_change":"2024-03-06T07:23:54.661954+0000","last_active":"2024-03-06T07:23:54.662455+0000","last_peered":"2024-03-06T07:23:54.662455+0000","last_clean":"2024-03-06T07:23:54.662455+0000","last_became_active":"2024-03-06T07:23:54.661708+0000","last_became_peered":"2024-03-06T07:23:54.661708+0000","last_unstale":"2024-03-06T07:23:54.662455+0000","last_undegraded":"2024-03-06T07:23:54.662455+0000","last_fullsized":"2024-03-06T07:23:54.662455+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:56.838295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642883+0000","last_change":"2024-03-06T07:23:54.642883+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642883+0000","last_undegraded":"2024-03-06T07:23:54.642883+0000","last_fullsized":"2024-03-06T07:23:54.642883+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655597+0000","last_change":"2024-03-06T07:23:54.653524+0000","last_active":"2024-03-06T07:23:54.655597+0000","last_peered":"2024-03-06T07:23:54.655597+0000","last_clean":"2024-03-06T07:23:54.655597+0000","last_became_active":"2024-03-06T07:23:54.652453+0000","last_became_peered":"2024-03-06T07:23:54.652453+0000","last_unstale":"2024-03-06T07:23:54.655597+0000","last_undegraded":"2024-03-06T07:23:54.655597+0000","last_fullsized":"2024-03-06T07:23:54.655597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:29.530866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642264+0000","last_change":"2024-03-06T07:23:54.642264+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642264+0000","last_undegraded":"2024-03-06T07:23:54.642264+0000","last_fullsized":"2024-03-06T07:23:54.642264+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663035+0000","last_change":"2024-03-06T07:23:54.661476+0000","last_active":"2024-03-06T07:23:54.663035+0000","last_peered":"2024-03-06T07:23:54.663035+0000","last_clean":"2024-03-06T07:23:54.663035+0000","last_became_active":"2024-03-06T07:23:54.661081+0000","last_became_peered":"2024-03-06T07:23:54.661081+0000","last_unstale":"2024-03-06T07:23:54.663035+0000","last_undegraded":"2024-03-06T07:23:54.663035+0000","last_fullsized":"2024-03-06T07:23:54.663035+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:55.914789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642349+0000","last_change":"2024-03-06T07:23:54.642349+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642349+0000","last_undegraded":"2024-03-06T07:23:54.642349+0000","last_fullsized":"2024-03-06T07:23:54.642349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658683+0000","last_change":"2024-03-06T07:23:54.656556+0000","last_active":"2024-03-06T07:23:54.658683+0000","last_peered":"2024-03-06T07:23:54.658683+0000","last_clean":"2024-03-06T07:23:54.658683+0000","last_became_active":"2024-03-06T07:23:54.656238+0000","last_became_peered":"2024-03-06T07:23:54.656238+0000","last_unstale":"2024-03-06T07:23:54.658683+0000","last_undegraded":"2024-03-06T07:23:54.658683+0000","last_fullsized":"2024-03-06T07:23:54.658683+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641604+0000","last_change":"2024-03-06T07:23:54.641604+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641604+0000","last_undegraded":"2024-03-06T07:23:54.641604+0000","last_fullsized":"2024-03-06T07:23:54.641604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.650870+0000","last_change":"2024-03-06T07:23:54.648660+0000","last_active":"2024-03-06T07:23:54.650870+0000","last_peered":"2024-03-06T07:23:54.650870+0000","last_clean":"2024-03-06T07:23:54.650870+0000","last_became_active":"2024-03-06T07:23:54.646838+0000","last_became_peered":"2024-03-06T07:23:54.646838+0000","last_unstale":"2024-03-06T07:23:54.650870+0000","last_undegraded":"2024-03-06T07:23:54.650870+0000","last_fullsized":"2024-03-06T07:23:54.650870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.731853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645622+0000","last_change":"2024-03-06T07:23:54.645622+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645622+0000","last_undegraded":"2024-03-06T07:23:54.645622+0000","last_fullsized":"2024-03-06T07:23:54.645622+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655052+0000","last_change":"2024-03-06T07:23:54.654021+0000","last_active":"2024-03-06T07:23:54.655052+0000","last_peered":"2024-03-06T07:23:54.655052+0000","last_clean":"2024-03-06T07:23:54.655052+0000","last_became_active":"2024-03-06T07:23:54.653706+0000","last_became_peered":"2024-03-06T07:23:54.653706+0000","last_unstale":"2024-03-06T07:23:54.655052+0000","last_undegraded":"2024-03-06T07:23:54.655052+0000","last_fullsized":"2024-03-06T07:23:54.655052+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:28.496649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640781+0000","last_change":"2024-03-06T07:23:54.640781+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640781+0000","last_undegraded":"2024-03-06T07:23:54.640781+0000","last_fullsized":"2024-03-06T07:23:54.640781+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655844+0000","last_change":"2024-03-06T07:23:54.653439+0000","last_active":"2024-03-06T07:23:54.655844+0000","last_peered":"2024-03-06T07:23:54.655844+0000","last_clean":"2024-03-06T07:23:54.655844+0000","last_became_active":"2024-03-06T07:23:54.653015+0000","last_became_peered":"2024-03-06T07:23:54.653015+0000","last_unstale":"2024-03-06T07:23:54.655844+0000","last_undegraded":"2024-03-06T07:23:54.655844+0000","last_fullsized":"2024-03-06T07:23:54.655844+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.500351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645281+0000","last_change":"2024-03-06T07:23:54.645281+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645281+0000","last_undegraded":"2024-03-06T07:23:54.645281+0000","last_fullsized":"2024-03-06T07:23:54.645281+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663119+0000","last_change":"2024-03-06T07:23:54.661828+0000","last_active":"2024-03-06T07:23:54.663119+0000","last_peered":"2024-03-06T07:23:54.663119+0000","last_clean":"2024-03-06T07:23:54.663119+0000","last_became_active":"2024-03-06T07:23:54.661129+0000","last_became_peered":"2024-03-06T07:23:54.661129+0000","last_unstale":"2024-03-06T07:23:54.663119+0000","last_undegraded":"2024-03-06T07:23:54.663119+0000","last_fullsized":"2024-03-06T07:23:54.663119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:19.451847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646653+0000","last_change":"2024-03-06T07:23:54.646653+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646653+0000","last_undegraded":"2024-03-06T07:23:54.646653+0000","last_fullsized":"2024-03-06T07:23:54.646653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.664675+0000","last_change":"2024-03-06T07:23:54.663162+0000","last_active":"2024-03-06T07:23:54.664675+0000","last_peered":"2024-03-06T07:23:54.664675+0000","last_clean":"2024-03-06T07:23:54.664675+0000","last_became_active":"2024-03-06T07:23:54.662778+0000","last_became_peered":"2024-03-06T07:23:54.662778+0000","last_unstale":"2024-03-06T07:23:54.664675+0000","last_undegraded":"2024-03-06T07:23:54.664675+0000","last_fullsized":"2024-03-06T07:23:54.664675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644876+0000","last_change":"2024-03-06T07:23:54.644876+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644876+0000","last_undegraded":"2024-03-06T07:23:54.644876+0000","last_fullsized":"2024-03-06T07:23:54.644876+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.662222+0000","last_change":"2024-03-06T07:23:54.651253+0000","last_active":"2024-03-06T07:23:54.662222+0000","last_peered":"2024-03-06T07:23:54.662222+0000","last_clean":"2024-03-06T07:23:54.662222+0000","last_became_active":"2024-03-06T07:23:54.650963+0000","last_became_peered":"2024-03-06T07:23:54.650963+0000","last_unstale":"2024-03-06T07:23:54.662222+0000","last_undegraded":"2024-03-06T07:23:54.662222+0000","last_fullsized":"2024-03-06T07:23:54.662222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:56.838295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646853+0000","last_change":"2024-03-06T07:23:54.646853+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646853+0000","last_undegraded":"2024-03-06T07:23:54.646853+0000","last_fullsized":"2024-03-06T07:23:54.646853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655850+0000","last_change":"2024-03-06T07:23:54.654189+0000","last_active":"2024-03-06T07:23:54.655850+0000","last_peered":"2024-03-06T07:23:54.655850+0000","last_clean":"2024-03-06T07:23:54.655850+0000","last_became_active":"2024-03-06T07:23:54.653889+0000","last_became_peered":"2024-03-06T07:23:54.653889+0000","last_unstale":"2024-03-06T07:23:54.655850+0000","last_undegraded":"2024-03-06T07:23:54.655850+0000","last_fullsized":"2024-03-06T07:23:54.655850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640921+0000","last_change":"2024-03-06T07:23:54.640921+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640921+0000","last_undegraded":"2024-03-06T07:23:54.640921+0000","last_fullsized":"2024-03-06T07:23:54.640921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653305+0000","last_change":"2024-03-06T07:23:54.649432+0000","last_active":"2024-03-06T07:23:54.653305+0000","last_peered":"2024-03-06T07:23:54.653305+0000","last_clean":"2024-03-06T07:23:54.653305+0000","last_became_active":"2024-03-06T07:23:54.649162+0000","last_became_peered":"2024-03-06T07:23:54.649162+0000","last_unstale":"2024-03-06T07:23:54.653305+0000","last_undegraded":"2024-03-06T07:23:54.653305+0000","last_fullsized":"2024-03-06T07:23:54.653305+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:38.700382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641700+0000","last_change":"2024-03-06T07:23:54.641700+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641700+0000","last_undegraded":"2024-03-06T07:23:54.641700+0000","last_fullsized":"2024-03-06T07:23:54.641700+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655269+0000","last_change":"2024-03-06T07:23:54.652266+0000","last_active":"2024-03-06T07:23:54.655269+0000","last_peered":"2024-03-06T07:23:54.655269+0000","last_clean":"2024-03-06T07:23:54.655269+0000","last_became_active":"2024-03-06T07:23:54.651291+0000","last_became_peered":"2024-03-06T07:23:54.651291+0000","last_unstale":"2024-03-06T07:23:54.655269+0000","last_undegraded":"2024-03-06T07:23:54.655269+0000","last_fullsized":"2024-03-06T07:23:54.655269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644697+0000","last_change":"2024-03-06T07:23:54.644697+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644697+0000","last_undegraded":"2024-03-06T07:23:54.644697+0000","last_fullsized":"2024-03-06T07:23:54.644697+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653133+0000","last_change":"2024-03-06T07:23:54.649706+0000","last_active":"2024-03-06T07:23:54.653133+0000","last_peered":"2024-03-06T07:23:54.653133+0000","last_clean":"2024-03-06T07:23:54.653133+0000","last_became_active":"2024-03-06T07:23:54.649451+0000","last_became_peered":"2024-03-06T07:23:54.649451+0000","last_unstale":"2024-03-06T07:23:54.653133+0000","last_undegraded":"2024-03-06T07:23:54.653133+0000","last_fullsized":"2024-03-06T07:23:54.653133+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:13.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.647941+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:23:54.647941+0000","last_peered":"2024-03-06T07:23:54.647941+0000","last_clean":"2024-03-06T07:23:54.647941+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:23:54.647941+0000","last_undegraded":"2024-03-06T07:23:54.647941+0000","last_fullsized":"2024-03-06T07:23:54.647941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":727628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":727628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:00.154 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:00.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.296+0000 7efc6021c640 1 Processor -- start 2024-03-06T07:24:00.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.296+0000 7efc6021c640 1 -- start start 2024-03-06T07:24:00.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc6021c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc580b95a0 0x7efc580b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc6021c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580cea30 con 0x7efc580b9a10 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc6021c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580cf3c0 con 0x7efc5815df00 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc6021c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc580cf0e0 con 0x7efc580b95a0 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc5d7b7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc580b95a0 0x7efc580b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc5d7b7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc580b95a0 0x7efc580b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48456/0 (socket says 172.21.15.27:48456) 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc5d7b7640 1 -- 172.21.15.27:0/3165258914 learned_addr learned my addr 172.21.15.27:0/3165258914 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc5e7b9640 1 --1- 172.21.15.27:0/3165258914 >> v1:172.21.15.27:6789/0 conn(0x7efc580b9a10 0x7efc580d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36452/0 (socket says 172.21.15.27:36452) 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc5d7b7640 1 -- 172.21.15.27:0/3165258914 >> v1:172.21.15.69:6789/0 conn(0x7efc5815df00 legacy=0x7efc580d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc5d7b7640 1 -- 172.21.15.27:0/3165258914 >> v1:172.21.15.27:6789/0 conn(0x7efc580b9a10 legacy=0x7efc580d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc5d7b7640 1 -- 172.21.15.27:0/3165258914 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc580ce470 con 0x7efc580b95a0 2024-03-06T07:24:00.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.298+0000 7efc5cfb6640 1 -- 172.21.15.27:0/3165258914 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372706344 0 0) 0x7efc580cea30 con 0x7efc580b9a10 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.299+0000 7efc5d7b7640 1 --2- 172.21.15.27:0/3165258914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc580b95a0 0x7efc580b5700 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7efc580cf590 tx=0x7efc4c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431f0b16c9a70373 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.299+0000 7efc5cfb6640 1 -- 172.21.15.27:0/3165258914 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c034070 con 0x7efc580b95a0 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.299+0000 7efc5cfb6640 1 -- 172.21.15.27:0/3165258914 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c030840 con 0x7efc580b95a0 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.299+0000 7efc5cfb6640 1 -- 172.21.15.27:0/3165258914 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c030c10 con 0x7efc580b95a0 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.300+0000 7efc6021c640 1 -- 172.21.15.27:0/3165258914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc580b95a0 msgr2=0x7efc580b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.300+0000 7efc6021c640 1 --2- 172.21.15.27:0/3165258914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc580b95a0 0x7efc580b5700 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7efc580cf590 tx=0x7efc4c02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.300+0000 7efc6021c640 1 -- 172.21.15.27:0/3165258914 shutdown_connections 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.300+0000 7efc6021c640 1 --2- 172.21.15.27:0/3165258914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc580b95a0 0x7efc580b5700 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.300+0000 7efc6021c640 1 -- 172.21.15.27:0/3165258914 >> 172.21.15.27:0/3165258914 conn(0x7efc58071960 msgr2=0x7efc58071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.300+0000 7efc6021c640 1 -- 172.21.15.27:0/3165258914 shutdown_connections 2024-03-06T07:24:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.300+0000 7efc6021c640 1 -- 172.21.15.27:0/3165258914 wait complete. 2024-03-06T07:24:00.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.301+0000 7efc6021c640 1 Processor -- start 2024-03-06T07:24:00.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.301+0000 7efc6021c640 1 -- start start 2024-03-06T07:24:00.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc6021c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc580b95a0 0x7efc58154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc6021c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc580b9a10 0x7efc58155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc6021c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc5815df00 0x7efc580b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc6021c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc580cedf0 con 0x7efc5815df00 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc6021c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc580cec70 con 0x7efc580b9a10 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc6021c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc580b5c40 con 0x7efc580b95a0 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5e7b9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc5815df00 0x7efc580b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5e7b9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc5815df00 0x7efc580b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48470/0 (socket says 172.21.15.27:48470) 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5e7b9640 1 -- 172.21.15.27:0/546392244 learned_addr learned my addr 172.21.15.27:0/546392244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5d7b7640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc580b9a10 0x7efc58155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5dfb8640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc580b95a0 0x7efc58154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5e7b9640 1 -- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc580b95a0 msgr2=0x7efc58154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5e7b9640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc580b95a0 0x7efc58154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5e7b9640 1 -- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc580b9a10 msgr2=0x7efc58155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5e7b9640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc580b9a10 0x7efc58155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.302+0000 7efc5e7b9640 1 -- 172.21.15.27:0/546392244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5815e810 con 0x7efc5815df00 2024-03-06T07:24:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.303+0000 7efc5e7b9640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc5815df00 0x7efc580b44f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c450 tx=0x7efc5000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.303+0000 7efc567fc640 1 -- 172.21.15.27:0/546392244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc50018070 con 0x7efc5815df00 2024-03-06T07:24:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.303+0000 7efc567fc640 1 -- 172.21.15.27:0/546392244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc50007500 con 0x7efc5815df00 2024-03-06T07:24:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.303+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5815eaf0 con 0x7efc5815df00 2024-03-06T07:24:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.303+0000 7efc567fc640 1 -- 172.21.15.27:0/546392244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc50004a20 con 0x7efc5815df00 2024-03-06T07:24:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.303+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5815f0b0 con 0x7efc5815df00 2024-03-06T07:24:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.304+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5815f580 con 0x7efc5815df00 2024-03-06T07:24:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.304+0000 7efc567fc640 1 -- 172.21.15.27:0/546392244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc5001b070 con 0x7efc5815df00 2024-03-06T07:24:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.305+0000 7efc567fc640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc4406b500 0x7efc4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.305+0000 7efc567fc640 1 -- 172.21.15.27:0/546392244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc50089400 con 0x7efc5815df00 2024-03-06T07:24:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.305+0000 7efc5dfb8640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc4406b500 0x7efc4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.306+0000 7efc5dfb8640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc4406b500 0x7efc4406d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efc48009f60 tx=0x7efc480094f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.314+0000 7efc567fc640 1 -- 172.21.15.27:0/546392244 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc50058310 con 0x7efc5815df00 2024-03-06T07:24:00.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.493+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc5815d3a0 con 0x7efc4406b500 2024-03-06T07:24:00.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.499+0000 7efc567fc640 1 -- 172.21.15.27:0/546392244 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246634 (secure 0 0 0) 0x7efc5815d3a0 con 0x7efc4406b500 2024-03-06T07:24:00.504 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:00.505 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc4406b500 msgr2=0x7efc4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc4406b500 0x7efc4406d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efc48009f60 tx=0x7efc480094f0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc5815df00 msgr2=0x7efc580b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc5815df00 0x7efc580b44f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c450 tx=0x7efc5000c910 comp rx=0 tx=0).stop 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 shutdown_connections 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc4406b500 0x7efc4406d9c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc5815df00 0x7efc580b44f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc580b9a10 0x7efc58155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 --2- 172.21.15.27:0/546392244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efc580b95a0 0x7efc58154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 >> 172.21.15.27:0/546392244 conn(0x7efc58071960 msgr2=0x7efc580b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 shutdown_connections 2024-03-06T07:24:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.502+0000 7efc6021c640 1 -- 172.21.15.27:0/546392244 wait complete. 2024-03-06T07:24:00.516 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-03-06T07:24:00.002784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220172,"kb_used_data":4012,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754548,"statfs":{"total":773094113280,"available":772868657152,"internally_reserved":0,"allocated":4108288,"data_stored":2297617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001489"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646720+0000","last_change":"2024-03-06T07:23:54.646720+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646720+0000","last_undegraded":"2024-03-06T07:23:54.646720+0000","last_fullsized":"2024-03-06T07:23:54.646720+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644574+0000","last_change":"2024-03-06T07:23:54.644574+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644574+0000","last_undegraded":"2024-03-06T07:23:54.644574+0000","last_fullsized":"2024-03-06T07:23:54.644574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643477+0000","last_change":"2024-03-06T07:23:54.643477+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643477+0000","last_undegraded":"2024-03-06T07:23:54.643477+0000","last_fullsized":"2024-03-06T07:23:54.643477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648478+0000","last_change":"2024-03-06T07:23:54.648478+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648478+0000","last_undegraded":"2024-03-06T07:23:54.648478+0000","last_fullsized":"2024-03-06T07:23:54.648478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643699+0000","last_change":"2024-03-06T07:23:54.643699+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643699+0000","last_undegraded":"2024-03-06T07:23:54.643699+0000","last_fullsized":"2024-03-06T07:23:54.643699+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641775+0000","last_change":"2024-03-06T07:23:54.641775+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641775+0000","last_undegraded":"2024-03-06T07:23:54.641775+0000","last_fullsized":"2024-03-06T07:23:54.641775+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645152+0000","last_change":"2024-03-06T07:23:54.645152+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645152+0000","last_undegraded":"2024-03-06T07:23:54.645152+0000","last_fullsized":"2024-03-06T07:23:54.645152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647218+0000","last_change":"2024-03-06T07:23:54.647218+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647218+0000","last_undegraded":"2024-03-06T07:23:54.647218+0000","last_fullsized":"2024-03-06T07:23:54.647218+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640159+0000","last_change":"2024-03-06T07:23:54.640159+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640159+0000","last_undegraded":"2024-03-06T07:23:54.640159+0000","last_fullsized":"2024-03-06T07:23:54.640159+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648487+0000","last_change":"2024-03-06T07:23:54.648487+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648487+0000","last_undegraded":"2024-03-06T07:23:54.648487+0000","last_fullsized":"2024-03-06T07:23:54.648487+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646918+0000","last_change":"2024-03-06T07:23:54.646918+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646918+0000","last_undegraded":"2024-03-06T07:23:54.646918+0000","last_fullsized":"2024-03-06T07:23:54.646918+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643738+0000","last_change":"2024-03-06T07:23:54.643738+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643738+0000","last_undegraded":"2024-03-06T07:23:54.643738+0000","last_fullsized":"2024-03-06T07:23:54.643738+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643236+0000","last_change":"2024-03-06T07:23:54.643236+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643236+0000","last_undegraded":"2024-03-06T07:23:54.643236+0000","last_fullsized":"2024-03-06T07:23:54.643236+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641298+0000","last_change":"2024-03-06T07:23:54.641298+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641298+0000","last_undegraded":"2024-03-06T07:23:54.641298+0000","last_fullsized":"2024-03-06T07:23:54.641298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646515+0000","last_change":"2024-03-06T07:23:54.646515+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646515+0000","last_undegraded":"2024-03-06T07:23:54.646515+0000","last_fullsized":"2024-03-06T07:23:54.646515+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644556+0000","last_change":"2024-03-06T07:23:54.644556+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644556+0000","last_undegraded":"2024-03-06T07:23:54.644556+0000","last_fullsized":"2024-03-06T07:23:54.644556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646653+0000","last_change":"2024-03-06T07:23:54.646653+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646653+0000","last_undegraded":"2024-03-06T07:23:54.646653+0000","last_fullsized":"2024-03-06T07:23:54.646653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644109+0000","last_change":"2024-03-06T07:23:54.644109+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644109+0000","last_undegraded":"2024-03-06T07:23:54.644109+0000","last_fullsized":"2024-03-06T07:23:54.644109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640796+0000","last_change":"2024-03-06T07:23:54.640796+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640796+0000","last_undegraded":"2024-03-06T07:23:54.640796+0000","last_fullsized":"2024-03-06T07:23:54.640796+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643640+0000","last_change":"2024-03-06T07:23:54.643640+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643640+0000","last_undegraded":"2024-03-06T07:23:54.643640+0000","last_fullsized":"2024-03-06T07:23:54.643640+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641081+0000","last_change":"2024-03-06T07:23:54.641081+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641081+0000","last_undegraded":"2024-03-06T07:23:54.641081+0000","last_fullsized":"2024-03-06T07:23:54.641081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.639615+0000","last_change":"2024-03-06T07:23:54.639615+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.639615+0000","last_undegraded":"2024-03-06T07:23:54.639615+0000","last_fullsized":"2024-03-06T07:23:54.639615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646031+0000","last_change":"2024-03-06T07:23:54.646031+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646031+0000","last_undegraded":"2024-03-06T07:23:54.646031+0000","last_fullsized":"2024-03-06T07:23:54.646031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642902+0000","last_change":"2024-03-06T07:23:54.642902+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642902+0000","last_undegraded":"2024-03-06T07:23:54.642902+0000","last_fullsized":"2024-03-06T07:23:54.642902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648463+0000","last_change":"2024-03-06T07:23:54.648463+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648463+0000","last_undegraded":"2024-03-06T07:23:54.648463+0000","last_fullsized":"2024-03-06T07:23:54.648463+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643640+0000","last_change":"2024-03-06T07:23:54.643640+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643640+0000","last_undegraded":"2024-03-06T07:23:54.643640+0000","last_fullsized":"2024-03-06T07:23:54.643640+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647824+0000","last_change":"2024-03-06T07:23:54.647824+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647824+0000","last_undegraded":"2024-03-06T07:23:54.647824+0000","last_fullsized":"2024-03-06T07:23:54.647824+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643326+0000","last_change":"2024-03-06T07:23:54.643326+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643326+0000","last_undegraded":"2024-03-06T07:23:54.643326+0000","last_fullsized":"2024-03-06T07:23:54.643326+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640572+0000","last_change":"2024-03-06T07:23:54.640572+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640572+0000","last_undegraded":"2024-03-06T07:23:54.640572+0000","last_fullsized":"2024-03-06T07:23:54.640572+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645644+0000","last_change":"2024-03-06T07:23:54.645644+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645644+0000","last_undegraded":"2024-03-06T07:23:54.645644+0000","last_fullsized":"2024-03-06T07:23:54.645644+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647662+0000","last_change":"2024-03-06T07:23:54.647662+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647662+0000","last_undegraded":"2024-03-06T07:23:54.647662+0000","last_fullsized":"2024-03-06T07:23:54.647662+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646477+0000","last_change":"2024-03-06T07:23:54.646477+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646477+0000","last_undegraded":"2024-03-06T07:23:54.646477+0000","last_fullsized":"2024-03-06T07:23:54.646477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647114+0000","last_change":"2024-03-06T07:23:54.647114+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647114+0000","last_undegraded":"2024-03-06T07:23:54.647114+0000","last_fullsized":"2024-03-06T07:23:54.647114+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656288+0000","last_change":"2024-03-06T07:23:54.655354+0000","last_active":"2024-03-06T07:23:54.656288+0000","last_peered":"2024-03-06T07:23:54.656288+0000","last_clean":"2024-03-06T07:23:54.656288+0000","last_became_active":"2024-03-06T07:23:54.655053+0000","last_became_peered":"2024-03-06T07:23:54.655053+0000","last_unstale":"2024-03-06T07:23:54.656288+0000","last_undegraded":"2024-03-06T07:23:54.656288+0000","last_fullsized":"2024-03-06T07:23:54.656288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:19.451847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644314+0000","last_change":"2024-03-06T07:23:54.644314+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644314+0000","last_undegraded":"2024-03-06T07:23:54.644314+0000","last_fullsized":"2024-03-06T07:23:54.644314+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663137+0000","last_change":"2024-03-06T07:23:54.662097+0000","last_active":"2024-03-06T07:23:54.663137+0000","last_peered":"2024-03-06T07:23:54.663137+0000","last_clean":"2024-03-06T07:23:54.663137+0000","last_became_active":"2024-03-06T07:23:54.661544+0000","last_became_peered":"2024-03-06T07:23:54.661544+0000","last_unstale":"2024-03-06T07:23:54.663137+0000","last_undegraded":"2024-03-06T07:23:54.663137+0000","last_fullsized":"2024-03-06T07:23:54.663137+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.639780+0000","last_change":"2024-03-06T07:23:54.639780+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.639780+0000","last_undegraded":"2024-03-06T07:23:54.639780+0000","last_fullsized":"2024-03-06T07:23:54.639780+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656144+0000","last_change":"2024-03-06T07:23:54.654853+0000","last_active":"2024-03-06T07:23:54.656144+0000","last_peered":"2024-03-06T07:23:54.656144+0000","last_clean":"2024-03-06T07:23:54.656144+0000","last_became_active":"2024-03-06T07:23:54.654566+0000","last_became_peered":"2024-03-06T07:23:54.654566+0000","last_unstale":"2024-03-06T07:23:54.656144+0000","last_undegraded":"2024-03-06T07:23:54.656144+0000","last_fullsized":"2024-03-06T07:23:54.656144+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:13.482463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647238+0000","last_change":"2024-03-06T07:23:54.647238+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647238+0000","last_undegraded":"2024-03-06T07:23:54.647238+0000","last_fullsized":"2024-03-06T07:23:54.647238+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.661489+0000","last_change":"2024-03-06T07:23:54.650365+0000","last_active":"2024-03-06T07:23:54.661489+0000","last_peered":"2024-03-06T07:23:54.661489+0000","last_clean":"2024-03-06T07:23:54.661489+0000","last_became_active":"2024-03-06T07:23:54.650077+0000","last_became_peered":"2024-03-06T07:23:54.650077+0000","last_unstale":"2024-03-06T07:23:54.661489+0000","last_undegraded":"2024-03-06T07:23:54.661489+0000","last_fullsized":"2024-03-06T07:23:54.661489+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:26.696172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645225+0000","last_change":"2024-03-06T07:23:54.645225+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645225+0000","last_undegraded":"2024-03-06T07:23:54.645225+0000","last_fullsized":"2024-03-06T07:23:54.645225+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654149+0000","last_change":"2024-03-06T07:23:54.652376+0000","last_active":"2024-03-06T07:23:54.654149+0000","last_peered":"2024-03-06T07:23:54.654149+0000","last_clean":"2024-03-06T07:23:54.654149+0000","last_became_active":"2024-03-06T07:23:54.652121+0000","last_became_peered":"2024-03-06T07:23:54.652121+0000","last_unstale":"2024-03-06T07:23:54.654149+0000","last_undegraded":"2024-03-06T07:23:54.654149+0000","last_fullsized":"2024-03-06T07:23:54.654149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:26.696172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643151+0000","last_change":"2024-03-06T07:23:54.643151+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643151+0000","last_undegraded":"2024-03-06T07:23:54.643151+0000","last_fullsized":"2024-03-06T07:23:54.643151+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654078+0000","last_change":"2024-03-06T07:23:54.652687+0000","last_active":"2024-03-06T07:23:54.654078+0000","last_peered":"2024-03-06T07:23:54.654078+0000","last_clean":"2024-03-06T07:23:54.654078+0000","last_became_active":"2024-03-06T07:23:54.652023+0000","last_became_peered":"2024-03-06T07:23:54.652023+0000","last_unstale":"2024-03-06T07:23:54.654078+0000","last_undegraded":"2024-03-06T07:23:54.654078+0000","last_fullsized":"2024-03-06T07:23:54.654078+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:37.112364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640802+0000","last_change":"2024-03-06T07:23:54.640802+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640802+0000","last_undegraded":"2024-03-06T07:23:54.640802+0000","last_fullsized":"2024-03-06T07:23:54.640802+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653032+0000","last_change":"2024-03-06T07:23:54.651953+0000","last_active":"2024-03-06T07:23:54.653032+0000","last_peered":"2024-03-06T07:23:54.653032+0000","last_clean":"2024-03-06T07:23:54.653032+0000","last_became_active":"2024-03-06T07:23:54.651636+0000","last_became_peered":"2024-03-06T07:23:54.651636+0000","last_unstale":"2024-03-06T07:23:54.653032+0000","last_undegraded":"2024-03-06T07:23:54.653032+0000","last_fullsized":"2024-03-06T07:23:54.653032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646552+0000","last_change":"2024-03-06T07:23:54.646552+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646552+0000","last_undegraded":"2024-03-06T07:23:54.646552+0000","last_fullsized":"2024-03-06T07:23:54.646552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655564+0000","last_change":"2024-03-06T07:23:54.653509+0000","last_active":"2024-03-06T07:23:54.655564+0000","last_peered":"2024-03-06T07:23:54.655564+0000","last_clean":"2024-03-06T07:23:54.655564+0000","last_became_active":"2024-03-06T07:23:54.651897+0000","last_became_peered":"2024-03-06T07:23:54.651897+0000","last_unstale":"2024-03-06T07:23:54.655564+0000","last_undegraded":"2024-03-06T07:23:54.655564+0000","last_fullsized":"2024-03-06T07:23:54.655564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:29.745505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642849+0000","last_change":"2024-03-06T07:23:54.642849+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642849+0000","last_undegraded":"2024-03-06T07:23:54.642849+0000","last_fullsized":"2024-03-06T07:23:54.642849+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656057+0000","last_change":"2024-03-06T07:23:54.653695+0000","last_active":"2024-03-06T07:23:54.656057+0000","last_peered":"2024-03-06T07:23:54.656057+0000","last_clean":"2024-03-06T07:23:54.656057+0000","last_became_active":"2024-03-06T07:23:54.653405+0000","last_became_peered":"2024-03-06T07:23:54.653405+0000","last_unstale":"2024-03-06T07:23:54.656057+0000","last_undegraded":"2024-03-06T07:23:54.656057+0000","last_fullsized":"2024-03-06T07:23:54.656057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:05.867698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642632+0000","last_change":"2024-03-06T07:23:54.642632+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642632+0000","last_undegraded":"2024-03-06T07:23:54.642632+0000","last_fullsized":"2024-03-06T07:23:54.642632+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663079+0000","last_change":"2024-03-06T07:23:54.661840+0000","last_active":"2024-03-06T07:23:54.663079+0000","last_peered":"2024-03-06T07:23:54.663079+0000","last_clean":"2024-03-06T07:23:54.663079+0000","last_became_active":"2024-03-06T07:23:54.661153+0000","last_became_peered":"2024-03-06T07:23:54.661153+0000","last_unstale":"2024-03-06T07:23:54.663079+0000","last_undegraded":"2024-03-06T07:23:54.663079+0000","last_fullsized":"2024-03-06T07:23:54.663079+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:13.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640747+0000","last_change":"2024-03-06T07:23:54.640747+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640747+0000","last_undegraded":"2024-03-06T07:23:54.640747+0000","last_fullsized":"2024-03-06T07:23:54.640747+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653444+0000","last_change":"2024-03-06T07:23:54.652357+0000","last_active":"2024-03-06T07:23:54.653444+0000","last_peered":"2024-03-06T07:23:54.653444+0000","last_clean":"2024-03-06T07:23:54.653444+0000","last_became_active":"2024-03-06T07:23:54.652092+0000","last_became_peered":"2024-03-06T07:23:54.652092+0000","last_unstale":"2024-03-06T07:23:54.653444+0000","last_undegraded":"2024-03-06T07:23:54.653444+0000","last_fullsized":"2024-03-06T07:23:54.653444+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:24.364320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643899+0000","last_change":"2024-03-06T07:23:54.643899+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643899+0000","last_undegraded":"2024-03-06T07:23:54.643899+0000","last_fullsized":"2024-03-06T07:23:54.643899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.657787+0000","last_change":"2024-03-06T07:23:54.656641+0000","last_active":"2024-03-06T07:23:54.657787+0000","last_peered":"2024-03-06T07:23:54.657787+0000","last_clean":"2024-03-06T07:23:54.657787+0000","last_became_active":"2024-03-06T07:23:54.656285+0000","last_became_peered":"2024-03-06T07:23:54.656285+0000","last_unstale":"2024-03-06T07:23:54.657787+0000","last_undegraded":"2024-03-06T07:23:54.657787+0000","last_fullsized":"2024-03-06T07:23:54.657787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:08.762797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648064+0000","last_change":"2024-03-06T07:23:54.648064+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648064+0000","last_undegraded":"2024-03-06T07:23:54.648064+0000","last_fullsized":"2024-03-06T07:23:54.648064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654119+0000","last_change":"2024-03-06T07:23:54.651813+0000","last_active":"2024-03-06T07:23:54.654119+0000","last_peered":"2024-03-06T07:23:54.654119+0000","last_clean":"2024-03-06T07:23:54.654119+0000","last_became_active":"2024-03-06T07:23:54.651504+0000","last_became_peered":"2024-03-06T07:23:54.651504+0000","last_unstale":"2024-03-06T07:23:54.654119+0000","last_undegraded":"2024-03-06T07:23:54.654119+0000","last_fullsized":"2024-03-06T07:23:54.654119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:07.864613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642671+0000","last_change":"2024-03-06T07:23:54.642671+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642671+0000","last_undegraded":"2024-03-06T07:23:54.642671+0000","last_fullsized":"2024-03-06T07:23:54.642671+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655764+0000","last_change":"2024-03-06T07:23:54.652295+0000","last_active":"2024-03-06T07:23:54.655764+0000","last_peered":"2024-03-06T07:23:54.655764+0000","last_clean":"2024-03-06T07:23:54.655764+0000","last_became_active":"2024-03-06T07:23:54.652029+0000","last_became_peered":"2024-03-06T07:23:54.652029+0000","last_unstale":"2024-03-06T07:23:54.655764+0000","last_undegraded":"2024-03-06T07:23:54.655764+0000","last_fullsized":"2024-03-06T07:23:54.655764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646070+0000","last_change":"2024-03-06T07:23:54.646070+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646070+0000","last_undegraded":"2024-03-06T07:23:54.646070+0000","last_fullsized":"2024-03-06T07:23:54.646070+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653007+0000","last_change":"2024-03-06T07:23:54.651884+0000","last_active":"2024-03-06T07:23:54.653007+0000","last_peered":"2024-03-06T07:23:54.653007+0000","last_clean":"2024-03-06T07:23:54.653007+0000","last_became_active":"2024-03-06T07:23:54.651584+0000","last_became_peered":"2024-03-06T07:23:54.651584+0000","last_unstale":"2024-03-06T07:23:54.653007+0000","last_undegraded":"2024-03-06T07:23:54.653007+0000","last_fullsized":"2024-03-06T07:23:54.653007+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646049+0000","last_change":"2024-03-06T07:23:54.646049+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646049+0000","last_undegraded":"2024-03-06T07:23:54.646049+0000","last_fullsized":"2024-03-06T07:23:54.646049+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658202+0000","last_change":"2024-03-06T07:23:54.657051+0000","last_active":"2024-03-06T07:23:54.658202+0000","last_peered":"2024-03-06T07:23:54.658202+0000","last_clean":"2024-03-06T07:23:54.658202+0000","last_became_active":"2024-03-06T07:23:54.656681+0000","last_became_peered":"2024-03-06T07:23:54.656681+0000","last_unstale":"2024-03-06T07:23:54.658202+0000","last_undegraded":"2024-03-06T07:23:54.658202+0000","last_fullsized":"2024-03-06T07:23:54.658202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:43.882962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640452+0000","last_change":"2024-03-06T07:23:54.640452+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640452+0000","last_undegraded":"2024-03-06T07:23:54.640452+0000","last_fullsized":"2024-03-06T07:23:54.640452+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658778+0000","last_change":"2024-03-06T07:23:54.657746+0000","last_active":"2024-03-06T07:23:54.658778+0000","last_peered":"2024-03-06T07:23:54.658778+0000","last_clean":"2024-03-06T07:23:54.658778+0000","last_became_active":"2024-03-06T07:23:54.657422+0000","last_became_peered":"2024-03-06T07:23:54.657422+0000","last_unstale":"2024-03-06T07:23:54.658778+0000","last_undegraded":"2024-03-06T07:23:54.658778+0000","last_fullsized":"2024-03-06T07:23:54.658778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645677+0000","last_change":"2024-03-06T07:23:54.645677+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645677+0000","last_undegraded":"2024-03-06T07:23:54.645677+0000","last_fullsized":"2024-03-06T07:23:54.645677+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654312+0000","last_change":"2024-03-06T07:23:54.653106+0000","last_active":"2024-03-06T07:23:54.654312+0000","last_peered":"2024-03-06T07:23:54.654312+0000","last_clean":"2024-03-06T07:23:54.654312+0000","last_became_active":"2024-03-06T07:23:54.651460+0000","last_became_peered":"2024-03-06T07:23:54.651460+0000","last_unstale":"2024-03-06T07:23:54.654312+0000","last_undegraded":"2024-03-06T07:23:54.654312+0000","last_fullsized":"2024-03-06T07:23:54.654312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.793262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642632+0000","last_change":"2024-03-06T07:23:54.642632+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642632+0000","last_undegraded":"2024-03-06T07:23:54.642632+0000","last_fullsized":"2024-03-06T07:23:54.642632+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.662455+0000","last_change":"2024-03-06T07:23:54.661954+0000","last_active":"2024-03-06T07:23:54.662455+0000","last_peered":"2024-03-06T07:23:54.662455+0000","last_clean":"2024-03-06T07:23:54.662455+0000","last_became_active":"2024-03-06T07:23:54.661708+0000","last_became_peered":"2024-03-06T07:23:54.661708+0000","last_unstale":"2024-03-06T07:23:54.662455+0000","last_undegraded":"2024-03-06T07:23:54.662455+0000","last_fullsized":"2024-03-06T07:23:54.662455+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:56.838295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642883+0000","last_change":"2024-03-06T07:23:54.642883+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642883+0000","last_undegraded":"2024-03-06T07:23:54.642883+0000","last_fullsized":"2024-03-06T07:23:54.642883+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655597+0000","last_change":"2024-03-06T07:23:54.653524+0000","last_active":"2024-03-06T07:23:54.655597+0000","last_peered":"2024-03-06T07:23:54.655597+0000","last_clean":"2024-03-06T07:23:54.655597+0000","last_became_active":"2024-03-06T07:23:54.652453+0000","last_became_peered":"2024-03-06T07:23:54.652453+0000","last_unstale":"2024-03-06T07:23:54.655597+0000","last_undegraded":"2024-03-06T07:23:54.655597+0000","last_fullsized":"2024-03-06T07:23:54.655597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:29.530866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642264+0000","last_change":"2024-03-06T07:23:54.642264+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642264+0000","last_undegraded":"2024-03-06T07:23:54.642264+0000","last_fullsized":"2024-03-06T07:23:54.642264+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663035+0000","last_change":"2024-03-06T07:23:54.661476+0000","last_active":"2024-03-06T07:23:54.663035+0000","last_peered":"2024-03-06T07:23:54.663035+0000","last_clean":"2024-03-06T07:23:54.663035+0000","last_became_active":"2024-03-06T07:23:54.661081+0000","last_became_peered":"2024-03-06T07:23:54.661081+0000","last_unstale":"2024-03-06T07:23:54.663035+0000","last_undegraded":"2024-03-06T07:23:54.663035+0000","last_fullsized":"2024-03-06T07:23:54.663035+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:55.914789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642349+0000","last_change":"2024-03-06T07:23:54.642349+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642349+0000","last_undegraded":"2024-03-06T07:23:54.642349+0000","last_fullsized":"2024-03-06T07:23:54.642349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658683+0000","last_change":"2024-03-06T07:23:54.656556+0000","last_active":"2024-03-06T07:23:54.658683+0000","last_peered":"2024-03-06T07:23:54.658683+0000","last_clean":"2024-03-06T07:23:54.658683+0000","last_became_active":"2024-03-06T07:23:54.656238+0000","last_became_peered":"2024-03-06T07:23:54.656238+0000","last_unstale":"2024-03-06T07:23:54.658683+0000","last_undegraded":"2024-03-06T07:23:54.658683+0000","last_fullsized":"2024-03-06T07:23:54.658683+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641604+0000","last_change":"2024-03-06T07:23:54.641604+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641604+0000","last_undegraded":"2024-03-06T07:23:54.641604+0000","last_fullsized":"2024-03-06T07:23:54.641604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.650870+0000","last_change":"2024-03-06T07:23:54.648660+0000","last_active":"2024-03-06T07:23:54.650870+0000","last_peered":"2024-03-06T07:23:54.650870+0000","last_clean":"2024-03-06T07:23:54.650870+0000","last_became_active":"2024-03-06T07:23:54.646838+0000","last_became_peered":"2024-03-06T07:23:54.646838+0000","last_unstale":"2024-03-06T07:23:54.650870+0000","last_undegraded":"2024-03-06T07:23:54.650870+0000","last_fullsized":"2024-03-06T07:23:54.650870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.731853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645622+0000","last_change":"2024-03-06T07:23:54.645622+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645622+0000","last_undegraded":"2024-03-06T07:23:54.645622+0000","last_fullsized":"2024-03-06T07:23:54.645622+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655052+0000","last_change":"2024-03-06T07:23:54.654021+0000","last_active":"2024-03-06T07:23:54.655052+0000","last_peered":"2024-03-06T07:23:54.655052+0000","last_clean":"2024-03-06T07:23:54.655052+0000","last_became_active":"2024-03-06T07:23:54.653706+0000","last_became_peered":"2024-03-06T07:23:54.653706+0000","last_unstale":"2024-03-06T07:23:54.655052+0000","last_undegraded":"2024-03-06T07:23:54.655052+0000","last_fullsized":"2024-03-06T07:23:54.655052+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:28.496649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640781+0000","last_change":"2024-03-06T07:23:54.640781+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640781+0000","last_undegraded":"2024-03-06T07:23:54.640781+0000","last_fullsized":"2024-03-06T07:23:54.640781+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655844+0000","last_change":"2024-03-06T07:23:54.653439+0000","last_active":"2024-03-06T07:23:54.655844+0000","last_peered":"2024-03-06T07:23:54.655844+0000","last_clean":"2024-03-06T07:23:54.655844+0000","last_became_active":"2024-03-06T07:23:54.653015+0000","last_became_peered":"2024-03-06T07:23:54.653015+0000","last_unstale":"2024-03-06T07:23:54.655844+0000","last_undegraded":"2024-03-06T07:23:54.655844+0000","last_fullsized":"2024-03-06T07:23:54.655844+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.500351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645281+0000","last_change":"2024-03-06T07:23:54.645281+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645281+0000","last_undegraded":"2024-03-06T07:23:54.645281+0000","last_fullsized":"2024-03-06T07:23:54.645281+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663119+0000","last_change":"2024-03-06T07:23:54.661828+0000","last_active":"2024-03-06T07:23:54.663119+0000","last_peered":"2024-03-06T07:23:54.663119+0000","last_clean":"2024-03-06T07:23:54.663119+0000","last_became_active":"2024-03-06T07:23:54.661129+0000","last_became_peered":"2024-03-06T07:23:54.661129+0000","last_unstale":"2024-03-06T07:23:54.663119+0000","last_undegraded":"2024-03-06T07:23:54.663119+0000","last_fullsized":"2024-03-06T07:23:54.663119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:19.451847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646653+0000","last_change":"2024-03-06T07:23:54.646653+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646653+0000","last_undegraded":"2024-03-06T07:23:54.646653+0000","last_fullsized":"2024-03-06T07:23:54.646653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.664675+0000","last_change":"2024-03-06T07:23:54.663162+0000","last_active":"2024-03-06T07:23:54.664675+0000","last_peered":"2024-03-06T07:23:54.664675+0000","last_clean":"2024-03-06T07:23:54.664675+0000","last_became_active":"2024-03-06T07:23:54.662778+0000","last_became_peered":"2024-03-06T07:23:54.662778+0000","last_unstale":"2024-03-06T07:23:54.664675+0000","last_undegraded":"2024-03-06T07:23:54.664675+0000","last_fullsized":"2024-03-06T07:23:54.664675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644876+0000","last_change":"2024-03-06T07:23:54.644876+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644876+0000","last_undegraded":"2024-03-06T07:23:54.644876+0000","last_fullsized":"2024-03-06T07:23:54.644876+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.662222+0000","last_change":"2024-03-06T07:23:54.651253+0000","last_active":"2024-03-06T07:23:54.662222+0000","last_peered":"2024-03-06T07:23:54.662222+0000","last_clean":"2024-03-06T07:23:54.662222+0000","last_became_active":"2024-03-06T07:23:54.650963+0000","last_became_peered":"2024-03-06T07:23:54.650963+0000","last_unstale":"2024-03-06T07:23:54.662222+0000","last_undegraded":"2024-03-06T07:23:54.662222+0000","last_fullsized":"2024-03-06T07:23:54.662222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:56.838295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646853+0000","last_change":"2024-03-06T07:23:54.646853+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646853+0000","last_undegraded":"2024-03-06T07:23:54.646853+0000","last_fullsized":"2024-03-06T07:23:54.646853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655850+0000","last_change":"2024-03-06T07:23:54.654189+0000","last_active":"2024-03-06T07:23:54.655850+0000","last_peered":"2024-03-06T07:23:54.655850+0000","last_clean":"2024-03-06T07:23:54.655850+0000","last_became_active":"2024-03-06T07:23:54.653889+0000","last_became_peered":"2024-03-06T07:23:54.653889+0000","last_unstale":"2024-03-06T07:23:54.655850+0000","last_undegraded":"2024-03-06T07:23:54.655850+0000","last_fullsized":"2024-03-06T07:23:54.655850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640921+0000","last_change":"2024-03-06T07:23:54.640921+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640921+0000","last_undegraded":"2024-03-06T07:23:54.640921+0000","last_fullsized":"2024-03-06T07:23:54.640921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653305+0000","last_change":"2024-03-06T07:23:54.649432+0000","last_active":"2024-03-06T07:23:54.653305+0000","last_peered":"2024-03-06T07:23:54.653305+0000","last_clean":"2024-03-06T07:23:54.653305+0000","last_became_active":"2024-03-06T07:23:54.649162+0000","last_became_peered":"2024-03-06T07:23:54.649162+0000","last_unstale":"2024-03-06T07:23:54.653305+0000","last_undegraded":"2024-03-06T07:23:54.653305+0000","last_fullsized":"2024-03-06T07:23:54.653305+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:38.700382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641700+0000","last_change":"2024-03-06T07:23:54.641700+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641700+0000","last_undegraded":"2024-03-06T07:23:54.641700+0000","last_fullsized":"2024-03-06T07:23:54.641700+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655269+0000","last_change":"2024-03-06T07:23:54.652266+0000","last_active":"2024-03-06T07:23:54.655269+0000","last_peered":"2024-03-06T07:23:54.655269+0000","last_clean":"2024-03-06T07:23:54.655269+0000","last_became_active":"2024-03-06T07:23:54.651291+0000","last_became_peered":"2024-03-06T07:23:54.651291+0000","last_unstale":"2024-03-06T07:23:54.655269+0000","last_undegraded":"2024-03-06T07:23:54.655269+0000","last_fullsized":"2024-03-06T07:23:54.655269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644697+0000","last_change":"2024-03-06T07:23:54.644697+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644697+0000","last_undegraded":"2024-03-06T07:23:54.644697+0000","last_fullsized":"2024-03-06T07:23:54.644697+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653133+0000","last_change":"2024-03-06T07:23:54.649706+0000","last_active":"2024-03-06T07:23:54.653133+0000","last_peered":"2024-03-06T07:23:54.653133+0000","last_clean":"2024-03-06T07:23:54.653133+0000","last_became_active":"2024-03-06T07:23:54.649451+0000","last_became_peered":"2024-03-06T07:23:54.649451+0000","last_unstale":"2024-03-06T07:23:54.653133+0000","last_undegraded":"2024-03-06T07:23:54.653133+0000","last_fullsized":"2024-03-06T07:23:54.653133+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:13.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.647941+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:23:54.647941+0000","last_peered":"2024-03-06T07:23:54.647941+0000","last_clean":"2024-03-06T07:23:54.647941+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:23:54.647941+0000","last_undegraded":"2024-03-06T07:23:54.647941+0000","last_fullsized":"2024-03-06T07:23:54.647941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":727628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":727628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:00.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.661+0000 7f0bd320f640 1 Processor -- start 2024-03-06T07:24:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.661+0000 7f0bd320f640 1 -- start start 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bd320f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bcc0aee10 0x7f0bcc15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bd320f640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc0729b0 con 0x7f0bcc06d400 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bd320f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc06c920 con 0x7f0bcc15bac0 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bd320f640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bcc06cf70 con 0x7f0bcc0aee10 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bcbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bcc0aee10 0x7f0bcc15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bcbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bcc0aee10 0x7f0bcc15cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48476/0 (socket says 172.21.15.27:48476) 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bcbfff640 1 -- 172.21.15.27:0/2594915011 learned_addr learned my addr 172.21.15.27:0/2594915011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bd17ac640 1 --1- 172.21.15.27:0/2594915011 >> v1:172.21.15.27:6789/0 conn(0x7f0bcc06d400 0x7f0bcc070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36468/0 (socket says 172.21.15.27:36468) 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bcbfff640 1 -- 172.21.15.27:0/2594915011 >> v1:172.21.15.69:6789/0 conn(0x7f0bcc15bac0 legacy=0x7f0bcc0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bcbfff640 1 -- 172.21.15.27:0/2594915011 >> v1:172.21.15.27:6789/0 conn(0x7f0bcc06d400 legacy=0x7f0bcc070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bcbfff640 1 -- 172.21.15.27:0/2594915011 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc06cc00 con 0x7f0bcc0aee10 2024-03-06T07:24:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.662+0000 7f0bcb7fe640 1 -- 172.21.15.27:0/2594915011 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41250626 0 0) 0x7f0bcc0729b0 con 0x7f0bcc06d400 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.663+0000 7f0bcbfff640 1 --2- 172.21.15.27:0/2594915011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 0x7f0bcc15cbe0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009870 tx=0x7f0bb802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41287fe68f293590 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.663+0000 7f0bcb7fe640 1 -- 172.21.15.27:0/2594915011 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb802f840 con 0x7f0bcc0aee10 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.663+0000 7f0bcb7fe640 1 -- 172.21.15.27:0/2594915011 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb802f9e0 con 0x7f0bcc0aee10 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.663+0000 7f0bcb7fe640 1 -- 172.21.15.27:0/2594915011 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb802fdb0 con 0x7f0bcc0aee10 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.664+0000 7f0bd320f640 1 -- 172.21.15.27:0/2594915011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 msgr2=0x7f0bcc15cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.664+0000 7f0bd320f640 1 --2- 172.21.15.27:0/2594915011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 0x7f0bcc15cbe0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009870 tx=0x7f0bb802ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.664+0000 7f0bd320f640 1 -- 172.21.15.27:0/2594915011 shutdown_connections 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.664+0000 7f0bd320f640 1 --2- 172.21.15.27:0/2594915011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 0x7f0bcc15cbe0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.664+0000 7f0bd320f640 1 -- 172.21.15.27:0/2594915011 >> 172.21.15.27:0/2594915011 conn(0x7f0bcc066d00 msgr2=0x7f0bcc0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.664+0000 7f0bd320f640 1 -- 172.21.15.27:0/2594915011 shutdown_connections 2024-03-06T07:24:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.664+0000 7f0bd320f640 1 -- 172.21.15.27:0/2594915011 wait complete. 2024-03-06T07:24:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.665+0000 7f0bd320f640 1 Processor -- start 2024-03-06T07:24:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.665+0000 7f0bd320f640 1 -- start start 2024-03-06T07:24:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bd320f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bcc06d400 0x7f0bcc149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bd320f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 0x7f0bcc0ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bd320f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc15bac0 0x7f0bcc0abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bd320f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc15c020 con 0x7f0bcc0aee10 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bd320f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc15bea0 con 0x7f0bcc15bac0 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bd320f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc072580 con 0x7f0bcc06d400 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bcbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 0x7f0bcc0ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bcbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 0x7f0bcc0ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48484/0 (socket says 172.21.15.27:48484) 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bcbfff640 1 -- 172.21.15.27:0/1478686856 learned_addr learned my addr 172.21.15.27:0/1478686856 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bd17ac640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc15bac0 0x7f0bcc0abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bd0fab640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bcc06d400 0x7f0bcc149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bcbfff640 1 -- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bcc06d400 msgr2=0x7f0bcc149250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bcbfff640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bcc06d400 0x7f0bcc149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bcbfff640 1 -- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc15bac0 msgr2=0x7f0bcc0abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bcbfff640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc15bac0 0x7f0bcc0abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bcbfff640 1 -- 172.21.15.27:0/1478686856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc15a0c0 con 0x7f0bcc0aee10 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.666+0000 7f0bd17ac640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc15bac0 0x7f0bcc0abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.667+0000 7f0bcbfff640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 0x7f0bcc0ade00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80090b0 tx=0x7f0bb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.667+0000 7f0bc97fa640 1 -- 172.21.15.27:0/1478686856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8047070 con 0x7f0bcc0aee10 2024-03-06T07:24:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.667+0000 7f0bc97fa640 1 -- 172.21.15.27:0/1478686856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb8037cc0 con 0x7f0bcc0aee10 2024-03-06T07:24:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.667+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc15a3a0 con 0x7f0bcc0aee10 2024-03-06T07:24:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.667+0000 7f0bc97fa640 1 -- 172.21.15.27:0/1478686856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8004270 con 0x7f0bcc0aee10 2024-03-06T07:24:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.667+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc15a960 con 0x7f0bcc0aee10 2024-03-06T07:24:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.668+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc067ff0 con 0x7f0bcc0aee10 2024-03-06T07:24:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.668+0000 7f0bc97fa640 1 -- 172.21.15.27:0/1478686856 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bb804a070 con 0x7f0bcc0aee10 2024-03-06T07:24:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.669+0000 7f0bc97fa640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bb006b4d0 0x7f0bb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.669+0000 7f0bc97fa640 1 -- 172.21.15.27:0/1478686856 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bb80ac330 con 0x7f0bcc0aee10 2024-03-06T07:24:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.669+0000 7f0bd0fab640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bb006b4d0 0x7f0bb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.670+0000 7f0bd0fab640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bb006b4d0 0x7f0bb006d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000f60 tx=0x7f0bc0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.679+0000 7f0bc97fa640 1 -- 172.21.15.27:0/1478686856 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bb8033030 con 0x7f0bcc0aee10 2024-03-06T07:24:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.958+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0bcc06de50 con 0x7f0bb006b4d0 2024-03-06T07:24:00.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.964+0000 7f0bc97fa640 1 -- 172.21.15.27:0/1478686856 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246634 (secure 0 0 0) 0x7f0bcc06de50 con 0x7f0bb006b4d0 2024-03-06T07:24:00.972 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:24:00.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bb006b4d0 msgr2=0x7f0bb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bb006b4d0 0x7f0bb006d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000f60 tx=0x7f0bc0009040 comp rx=0 tx=0).stop 2024-03-06T07:24:00.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 msgr2=0x7f0bcc0ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 0x7f0bcc0ade00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80090b0 tx=0x7f0bb8004730 comp rx=0 tx=0).stop 2024-03-06T07:24:00.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 shutdown_connections 2024-03-06T07:24:00.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bb006b4d0 0x7f0bb006d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc15bac0 0x7f0bcc0abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bcc0aee10 0x7f0bcc0ade00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 --2- 172.21.15.27:0/1478686856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bcc06d400 0x7f0bcc149250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 >> 172.21.15.27:0/1478686856 conn(0x7f0bcc066d00 msgr2=0x7f0bcc0b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 shutdown_connections 2024-03-06T07:24:00.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:00.968+0000 7f0bd320f640 1 -- 172.21.15.27:0/1478686856 wait complete. 2024-03-06T07:24:00.982 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-03-06T07:24:00.002784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220172,"kb_used_data":4012,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754548,"statfs":{"total":773094113280,"available":772868657152,"internally_reserved":0,"allocated":4108288,"data_stored":2297617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001489"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646720+0000","last_change":"2024-03-06T07:23:54.646720+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646720+0000","last_undegraded":"2024-03-06T07:23:54.646720+0000","last_fullsized":"2024-03-06T07:23:54.646720+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644574+0000","last_change":"2024-03-06T07:23:54.644574+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644574+0000","last_undegraded":"2024-03-06T07:23:54.644574+0000","last_fullsized":"2024-03-06T07:23:54.644574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643477+0000","last_change":"2024-03-06T07:23:54.643477+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643477+0000","last_undegraded":"2024-03-06T07:23:54.643477+0000","last_fullsized":"2024-03-06T07:23:54.643477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648478+0000","last_change":"2024-03-06T07:23:54.648478+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648478+0000","last_undegraded":"2024-03-06T07:23:54.648478+0000","last_fullsized":"2024-03-06T07:23:54.648478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643699+0000","last_change":"2024-03-06T07:23:54.643699+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643699+0000","last_undegraded":"2024-03-06T07:23:54.643699+0000","last_fullsized":"2024-03-06T07:23:54.643699+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641775+0000","last_change":"2024-03-06T07:23:54.641775+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641775+0000","last_undegraded":"2024-03-06T07:23:54.641775+0000","last_fullsized":"2024-03-06T07:23:54.641775+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645152+0000","last_change":"2024-03-06T07:23:54.645152+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645152+0000","last_undegraded":"2024-03-06T07:23:54.645152+0000","last_fullsized":"2024-03-06T07:23:54.645152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647218+0000","last_change":"2024-03-06T07:23:54.647218+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647218+0000","last_undegraded":"2024-03-06T07:23:54.647218+0000","last_fullsized":"2024-03-06T07:23:54.647218+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640159+0000","last_change":"2024-03-06T07:23:54.640159+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640159+0000","last_undegraded":"2024-03-06T07:23:54.640159+0000","last_fullsized":"2024-03-06T07:23:54.640159+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648487+0000","last_change":"2024-03-06T07:23:54.648487+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648487+0000","last_undegraded":"2024-03-06T07:23:54.648487+0000","last_fullsized":"2024-03-06T07:23:54.648487+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646918+0000","last_change":"2024-03-06T07:23:54.646918+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646918+0000","last_undegraded":"2024-03-06T07:23:54.646918+0000","last_fullsized":"2024-03-06T07:23:54.646918+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643738+0000","last_change":"2024-03-06T07:23:54.643738+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643738+0000","last_undegraded":"2024-03-06T07:23:54.643738+0000","last_fullsized":"2024-03-06T07:23:54.643738+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643236+0000","last_change":"2024-03-06T07:23:54.643236+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643236+0000","last_undegraded":"2024-03-06T07:23:54.643236+0000","last_fullsized":"2024-03-06T07:23:54.643236+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641298+0000","last_change":"2024-03-06T07:23:54.641298+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641298+0000","last_undegraded":"2024-03-06T07:23:54.641298+0000","last_fullsized":"2024-03-06T07:23:54.641298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646515+0000","last_change":"2024-03-06T07:23:54.646515+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646515+0000","last_undegraded":"2024-03-06T07:23:54.646515+0000","last_fullsized":"2024-03-06T07:23:54.646515+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644556+0000","last_change":"2024-03-06T07:23:54.644556+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644556+0000","last_undegraded":"2024-03-06T07:23:54.644556+0000","last_fullsized":"2024-03-06T07:23:54.644556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646653+0000","last_change":"2024-03-06T07:23:54.646653+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646653+0000","last_undegraded":"2024-03-06T07:23:54.646653+0000","last_fullsized":"2024-03-06T07:23:54.646653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644109+0000","last_change":"2024-03-06T07:23:54.644109+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644109+0000","last_undegraded":"2024-03-06T07:23:54.644109+0000","last_fullsized":"2024-03-06T07:23:54.644109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640796+0000","last_change":"2024-03-06T07:23:54.640796+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640796+0000","last_undegraded":"2024-03-06T07:23:54.640796+0000","last_fullsized":"2024-03-06T07:23:54.640796+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643640+0000","last_change":"2024-03-06T07:23:54.643640+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643640+0000","last_undegraded":"2024-03-06T07:23:54.643640+0000","last_fullsized":"2024-03-06T07:23:54.643640+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641081+0000","last_change":"2024-03-06T07:23:54.641081+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641081+0000","last_undegraded":"2024-03-06T07:23:54.641081+0000","last_fullsized":"2024-03-06T07:23:54.641081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.639615+0000","last_change":"2024-03-06T07:23:54.639615+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.639615+0000","last_undegraded":"2024-03-06T07:23:54.639615+0000","last_fullsized":"2024-03-06T07:23:54.639615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646031+0000","last_change":"2024-03-06T07:23:54.646031+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646031+0000","last_undegraded":"2024-03-06T07:23:54.646031+0000","last_fullsized":"2024-03-06T07:23:54.646031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642902+0000","last_change":"2024-03-06T07:23:54.642902+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642902+0000","last_undegraded":"2024-03-06T07:23:54.642902+0000","last_fullsized":"2024-03-06T07:23:54.642902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648463+0000","last_change":"2024-03-06T07:23:54.648463+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648463+0000","last_undegraded":"2024-03-06T07:23:54.648463+0000","last_fullsized":"2024-03-06T07:23:54.648463+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643640+0000","last_change":"2024-03-06T07:23:54.643640+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643640+0000","last_undegraded":"2024-03-06T07:23:54.643640+0000","last_fullsized":"2024-03-06T07:23:54.643640+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647824+0000","last_change":"2024-03-06T07:23:54.647824+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647824+0000","last_undegraded":"2024-03-06T07:23:54.647824+0000","last_fullsized":"2024-03-06T07:23:54.647824+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643326+0000","last_change":"2024-03-06T07:23:54.643326+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643326+0000","last_undegraded":"2024-03-06T07:23:54.643326+0000","last_fullsized":"2024-03-06T07:23:54.643326+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640572+0000","last_change":"2024-03-06T07:23:54.640572+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640572+0000","last_undegraded":"2024-03-06T07:23:54.640572+0000","last_fullsized":"2024-03-06T07:23:54.640572+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645644+0000","last_change":"2024-03-06T07:23:54.645644+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645644+0000","last_undegraded":"2024-03-06T07:23:54.645644+0000","last_fullsized":"2024-03-06T07:23:54.645644+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647662+0000","last_change":"2024-03-06T07:23:54.647662+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647662+0000","last_undegraded":"2024-03-06T07:23:54.647662+0000","last_fullsized":"2024-03-06T07:23:54.647662+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646477+0000","last_change":"2024-03-06T07:23:54.646477+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646477+0000","last_undegraded":"2024-03-06T07:23:54.646477+0000","last_fullsized":"2024-03-06T07:23:54.646477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647114+0000","last_change":"2024-03-06T07:23:54.647114+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647114+0000","last_undegraded":"2024-03-06T07:23:54.647114+0000","last_fullsized":"2024-03-06T07:23:54.647114+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656288+0000","last_change":"2024-03-06T07:23:54.655354+0000","last_active":"2024-03-06T07:23:54.656288+0000","last_peered":"2024-03-06T07:23:54.656288+0000","last_clean":"2024-03-06T07:23:54.656288+0000","last_became_active":"2024-03-06T07:23:54.655053+0000","last_became_peered":"2024-03-06T07:23:54.655053+0000","last_unstale":"2024-03-06T07:23:54.656288+0000","last_undegraded":"2024-03-06T07:23:54.656288+0000","last_fullsized":"2024-03-06T07:23:54.656288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:19.451847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644314+0000","last_change":"2024-03-06T07:23:54.644314+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644314+0000","last_undegraded":"2024-03-06T07:23:54.644314+0000","last_fullsized":"2024-03-06T07:23:54.644314+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663137+0000","last_change":"2024-03-06T07:23:54.662097+0000","last_active":"2024-03-06T07:23:54.663137+0000","last_peered":"2024-03-06T07:23:54.663137+0000","last_clean":"2024-03-06T07:23:54.663137+0000","last_became_active":"2024-03-06T07:23:54.661544+0000","last_became_peered":"2024-03-06T07:23:54.661544+0000","last_unstale":"2024-03-06T07:23:54.663137+0000","last_undegraded":"2024-03-06T07:23:54.663137+0000","last_fullsized":"2024-03-06T07:23:54.663137+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.639780+0000","last_change":"2024-03-06T07:23:54.639780+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.639780+0000","last_undegraded":"2024-03-06T07:23:54.639780+0000","last_fullsized":"2024-03-06T07:23:54.639780+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656144+0000","last_change":"2024-03-06T07:23:54.654853+0000","last_active":"2024-03-06T07:23:54.656144+0000","last_peered":"2024-03-06T07:23:54.656144+0000","last_clean":"2024-03-06T07:23:54.656144+0000","last_became_active":"2024-03-06T07:23:54.654566+0000","last_became_peered":"2024-03-06T07:23:54.654566+0000","last_unstale":"2024-03-06T07:23:54.656144+0000","last_undegraded":"2024-03-06T07:23:54.656144+0000","last_fullsized":"2024-03-06T07:23:54.656144+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:13.482463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647238+0000","last_change":"2024-03-06T07:23:54.647238+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647238+0000","last_undegraded":"2024-03-06T07:23:54.647238+0000","last_fullsized":"2024-03-06T07:23:54.647238+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.661489+0000","last_change":"2024-03-06T07:23:54.650365+0000","last_active":"2024-03-06T07:23:54.661489+0000","last_peered":"2024-03-06T07:23:54.661489+0000","last_clean":"2024-03-06T07:23:54.661489+0000","last_became_active":"2024-03-06T07:23:54.650077+0000","last_became_peered":"2024-03-06T07:23:54.650077+0000","last_unstale":"2024-03-06T07:23:54.661489+0000","last_undegraded":"2024-03-06T07:23:54.661489+0000","last_fullsized":"2024-03-06T07:23:54.661489+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:26.696172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645225+0000","last_change":"2024-03-06T07:23:54.645225+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645225+0000","last_undegraded":"2024-03-06T07:23:54.645225+0000","last_fullsized":"2024-03-06T07:23:54.645225+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654149+0000","last_change":"2024-03-06T07:23:54.652376+0000","last_active":"2024-03-06T07:23:54.654149+0000","last_peered":"2024-03-06T07:23:54.654149+0000","last_clean":"2024-03-06T07:23:54.654149+0000","last_became_active":"2024-03-06T07:23:54.652121+0000","last_became_peered":"2024-03-06T07:23:54.652121+0000","last_unstale":"2024-03-06T07:23:54.654149+0000","last_undegraded":"2024-03-06T07:23:54.654149+0000","last_fullsized":"2024-03-06T07:23:54.654149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:26.696172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643151+0000","last_change":"2024-03-06T07:23:54.643151+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643151+0000","last_undegraded":"2024-03-06T07:23:54.643151+0000","last_fullsized":"2024-03-06T07:23:54.643151+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654078+0000","last_change":"2024-03-06T07:23:54.652687+0000","last_active":"2024-03-06T07:23:54.654078+0000","last_peered":"2024-03-06T07:23:54.654078+0000","last_clean":"2024-03-06T07:23:54.654078+0000","last_became_active":"2024-03-06T07:23:54.652023+0000","last_became_peered":"2024-03-06T07:23:54.652023+0000","last_unstale":"2024-03-06T07:23:54.654078+0000","last_undegraded":"2024-03-06T07:23:54.654078+0000","last_fullsized":"2024-03-06T07:23:54.654078+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:37.112364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640802+0000","last_change":"2024-03-06T07:23:54.640802+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640802+0000","last_undegraded":"2024-03-06T07:23:54.640802+0000","last_fullsized":"2024-03-06T07:23:54.640802+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653032+0000","last_change":"2024-03-06T07:23:54.651953+0000","last_active":"2024-03-06T07:23:54.653032+0000","last_peered":"2024-03-06T07:23:54.653032+0000","last_clean":"2024-03-06T07:23:54.653032+0000","last_became_active":"2024-03-06T07:23:54.651636+0000","last_became_peered":"2024-03-06T07:23:54.651636+0000","last_unstale":"2024-03-06T07:23:54.653032+0000","last_undegraded":"2024-03-06T07:23:54.653032+0000","last_fullsized":"2024-03-06T07:23:54.653032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646552+0000","last_change":"2024-03-06T07:23:54.646552+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646552+0000","last_undegraded":"2024-03-06T07:23:54.646552+0000","last_fullsized":"2024-03-06T07:23:54.646552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655564+0000","last_change":"2024-03-06T07:23:54.653509+0000","last_active":"2024-03-06T07:23:54.655564+0000","last_peered":"2024-03-06T07:23:54.655564+0000","last_clean":"2024-03-06T07:23:54.655564+0000","last_became_active":"2024-03-06T07:23:54.651897+0000","last_became_peered":"2024-03-06T07:23:54.651897+0000","last_unstale":"2024-03-06T07:23:54.655564+0000","last_undegraded":"2024-03-06T07:23:54.655564+0000","last_fullsized":"2024-03-06T07:23:54.655564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:29.745505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642849+0000","last_change":"2024-03-06T07:23:54.642849+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642849+0000","last_undegraded":"2024-03-06T07:23:54.642849+0000","last_fullsized":"2024-03-06T07:23:54.642849+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656057+0000","last_change":"2024-03-06T07:23:54.653695+0000","last_active":"2024-03-06T07:23:54.656057+0000","last_peered":"2024-03-06T07:23:54.656057+0000","last_clean":"2024-03-06T07:23:54.656057+0000","last_became_active":"2024-03-06T07:23:54.653405+0000","last_became_peered":"2024-03-06T07:23:54.653405+0000","last_unstale":"2024-03-06T07:23:54.656057+0000","last_undegraded":"2024-03-06T07:23:54.656057+0000","last_fullsized":"2024-03-06T07:23:54.656057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:05.867698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642632+0000","last_change":"2024-03-06T07:23:54.642632+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642632+0000","last_undegraded":"2024-03-06T07:23:54.642632+0000","last_fullsized":"2024-03-06T07:23:54.642632+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663079+0000","last_change":"2024-03-06T07:23:54.661840+0000","last_active":"2024-03-06T07:23:54.663079+0000","last_peered":"2024-03-06T07:23:54.663079+0000","last_clean":"2024-03-06T07:23:54.663079+0000","last_became_active":"2024-03-06T07:23:54.661153+0000","last_became_peered":"2024-03-06T07:23:54.661153+0000","last_unstale":"2024-03-06T07:23:54.663079+0000","last_undegraded":"2024-03-06T07:23:54.663079+0000","last_fullsized":"2024-03-06T07:23:54.663079+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:13.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640747+0000","last_change":"2024-03-06T07:23:54.640747+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640747+0000","last_undegraded":"2024-03-06T07:23:54.640747+0000","last_fullsized":"2024-03-06T07:23:54.640747+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653444+0000","last_change":"2024-03-06T07:23:54.652357+0000","last_active":"2024-03-06T07:23:54.653444+0000","last_peered":"2024-03-06T07:23:54.653444+0000","last_clean":"2024-03-06T07:23:54.653444+0000","last_became_active":"2024-03-06T07:23:54.652092+0000","last_became_peered":"2024-03-06T07:23:54.652092+0000","last_unstale":"2024-03-06T07:23:54.653444+0000","last_undegraded":"2024-03-06T07:23:54.653444+0000","last_fullsized":"2024-03-06T07:23:54.653444+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:24.364320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643899+0000","last_change":"2024-03-06T07:23:54.643899+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643899+0000","last_undegraded":"2024-03-06T07:23:54.643899+0000","last_fullsized":"2024-03-06T07:23:54.643899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.657787+0000","last_change":"2024-03-06T07:23:54.656641+0000","last_active":"2024-03-06T07:23:54.657787+0000","last_peered":"2024-03-06T07:23:54.657787+0000","last_clean":"2024-03-06T07:23:54.657787+0000","last_became_active":"2024-03-06T07:23:54.656285+0000","last_became_peered":"2024-03-06T07:23:54.656285+0000","last_unstale":"2024-03-06T07:23:54.657787+0000","last_undegraded":"2024-03-06T07:23:54.657787+0000","last_fullsized":"2024-03-06T07:23:54.657787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:08.762797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648064+0000","last_change":"2024-03-06T07:23:54.648064+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648064+0000","last_undegraded":"2024-03-06T07:23:54.648064+0000","last_fullsized":"2024-03-06T07:23:54.648064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654119+0000","last_change":"2024-03-06T07:23:54.651813+0000","last_active":"2024-03-06T07:23:54.654119+0000","last_peered":"2024-03-06T07:23:54.654119+0000","last_clean":"2024-03-06T07:23:54.654119+0000","last_became_active":"2024-03-06T07:23:54.651504+0000","last_became_peered":"2024-03-06T07:23:54.651504+0000","last_unstale":"2024-03-06T07:23:54.654119+0000","last_undegraded":"2024-03-06T07:23:54.654119+0000","last_fullsized":"2024-03-06T07:23:54.654119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:07.864613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642671+0000","last_change":"2024-03-06T07:23:54.642671+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642671+0000","last_undegraded":"2024-03-06T07:23:54.642671+0000","last_fullsized":"2024-03-06T07:23:54.642671+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655764+0000","last_change":"2024-03-06T07:23:54.652295+0000","last_active":"2024-03-06T07:23:54.655764+0000","last_peered":"2024-03-06T07:23:54.655764+0000","last_clean":"2024-03-06T07:23:54.655764+0000","last_became_active":"2024-03-06T07:23:54.652029+0000","last_became_peered":"2024-03-06T07:23:54.652029+0000","last_unstale":"2024-03-06T07:23:54.655764+0000","last_undegraded":"2024-03-06T07:23:54.655764+0000","last_fullsized":"2024-03-06T07:23:54.655764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646070+0000","last_change":"2024-03-06T07:23:54.646070+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646070+0000","last_undegraded":"2024-03-06T07:23:54.646070+0000","last_fullsized":"2024-03-06T07:23:54.646070+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653007+0000","last_change":"2024-03-06T07:23:54.651884+0000","last_active":"2024-03-06T07:23:54.653007+0000","last_peered":"2024-03-06T07:23:54.653007+0000","last_clean":"2024-03-06T07:23:54.653007+0000","last_became_active":"2024-03-06T07:23:54.651584+0000","last_became_peered":"2024-03-06T07:23:54.651584+0000","last_unstale":"2024-03-06T07:23:54.653007+0000","last_undegraded":"2024-03-06T07:23:54.653007+0000","last_fullsized":"2024-03-06T07:23:54.653007+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646049+0000","last_change":"2024-03-06T07:23:54.646049+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646049+0000","last_undegraded":"2024-03-06T07:23:54.646049+0000","last_fullsized":"2024-03-06T07:23:54.646049+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658202+0000","last_change":"2024-03-06T07:23:54.657051+0000","last_active":"2024-03-06T07:23:54.658202+0000","last_peered":"2024-03-06T07:23:54.658202+0000","last_clean":"2024-03-06T07:23:54.658202+0000","last_became_active":"2024-03-06T07:23:54.656681+0000","last_became_peered":"2024-03-06T07:23:54.656681+0000","last_unstale":"2024-03-06T07:23:54.658202+0000","last_undegraded":"2024-03-06T07:23:54.658202+0000","last_fullsized":"2024-03-06T07:23:54.658202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:43.882962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640452+0000","last_change":"2024-03-06T07:23:54.640452+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640452+0000","last_undegraded":"2024-03-06T07:23:54.640452+0000","last_fullsized":"2024-03-06T07:23:54.640452+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658778+0000","last_change":"2024-03-06T07:23:54.657746+0000","last_active":"2024-03-06T07:23:54.658778+0000","last_peered":"2024-03-06T07:23:54.658778+0000","last_clean":"2024-03-06T07:23:54.658778+0000","last_became_active":"2024-03-06T07:23:54.657422+0000","last_became_peered":"2024-03-06T07:23:54.657422+0000","last_unstale":"2024-03-06T07:23:54.658778+0000","last_undegraded":"2024-03-06T07:23:54.658778+0000","last_fullsized":"2024-03-06T07:23:54.658778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645677+0000","last_change":"2024-03-06T07:23:54.645677+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645677+0000","last_undegraded":"2024-03-06T07:23:54.645677+0000","last_fullsized":"2024-03-06T07:23:54.645677+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654312+0000","last_change":"2024-03-06T07:23:54.653106+0000","last_active":"2024-03-06T07:23:54.654312+0000","last_peered":"2024-03-06T07:23:54.654312+0000","last_clean":"2024-03-06T07:23:54.654312+0000","last_became_active":"2024-03-06T07:23:54.651460+0000","last_became_peered":"2024-03-06T07:23:54.651460+0000","last_unstale":"2024-03-06T07:23:54.654312+0000","last_undegraded":"2024-03-06T07:23:54.654312+0000","last_fullsized":"2024-03-06T07:23:54.654312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.793262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642632+0000","last_change":"2024-03-06T07:23:54.642632+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642632+0000","last_undegraded":"2024-03-06T07:23:54.642632+0000","last_fullsized":"2024-03-06T07:23:54.642632+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.662455+0000","last_change":"2024-03-06T07:23:54.661954+0000","last_active":"2024-03-06T07:23:54.662455+0000","last_peered":"2024-03-06T07:23:54.662455+0000","last_clean":"2024-03-06T07:23:54.662455+0000","last_became_active":"2024-03-06T07:23:54.661708+0000","last_became_peered":"2024-03-06T07:23:54.661708+0000","last_unstale":"2024-03-06T07:23:54.662455+0000","last_undegraded":"2024-03-06T07:23:54.662455+0000","last_fullsized":"2024-03-06T07:23:54.662455+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:56.838295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642883+0000","last_change":"2024-03-06T07:23:54.642883+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642883+0000","last_undegraded":"2024-03-06T07:23:54.642883+0000","last_fullsized":"2024-03-06T07:23:54.642883+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655597+0000","last_change":"2024-03-06T07:23:54.653524+0000","last_active":"2024-03-06T07:23:54.655597+0000","last_peered":"2024-03-06T07:23:54.655597+0000","last_clean":"2024-03-06T07:23:54.655597+0000","last_became_active":"2024-03-06T07:23:54.652453+0000","last_became_peered":"2024-03-06T07:23:54.652453+0000","last_unstale":"2024-03-06T07:23:54.655597+0000","last_undegraded":"2024-03-06T07:23:54.655597+0000","last_fullsized":"2024-03-06T07:23:54.655597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:29.530866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642264+0000","last_change":"2024-03-06T07:23:54.642264+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642264+0000","last_undegraded":"2024-03-06T07:23:54.642264+0000","last_fullsized":"2024-03-06T07:23:54.642264+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663035+0000","last_change":"2024-03-06T07:23:54.661476+0000","last_active":"2024-03-06T07:23:54.663035+0000","last_peered":"2024-03-06T07:23:54.663035+0000","last_clean":"2024-03-06T07:23:54.663035+0000","last_became_active":"2024-03-06T07:23:54.661081+0000","last_became_peered":"2024-03-06T07:23:54.661081+0000","last_unstale":"2024-03-06T07:23:54.663035+0000","last_undegraded":"2024-03-06T07:23:54.663035+0000","last_fullsized":"2024-03-06T07:23:54.663035+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:55.914789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642349+0000","last_change":"2024-03-06T07:23:54.642349+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642349+0000","last_undegraded":"2024-03-06T07:23:54.642349+0000","last_fullsized":"2024-03-06T07:23:54.642349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658683+0000","last_change":"2024-03-06T07:23:54.656556+0000","last_active":"2024-03-06T07:23:54.658683+0000","last_peered":"2024-03-06T07:23:54.658683+0000","last_clean":"2024-03-06T07:23:54.658683+0000","last_became_active":"2024-03-06T07:23:54.656238+0000","last_became_peered":"2024-03-06T07:23:54.656238+0000","last_unstale":"2024-03-06T07:23:54.658683+0000","last_undegraded":"2024-03-06T07:23:54.658683+0000","last_fullsized":"2024-03-06T07:23:54.658683+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641604+0000","last_change":"2024-03-06T07:23:54.641604+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641604+0000","last_undegraded":"2024-03-06T07:23:54.641604+0000","last_fullsized":"2024-03-06T07:23:54.641604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.650870+0000","last_change":"2024-03-06T07:23:54.648660+0000","last_active":"2024-03-06T07:23:54.650870+0000","last_peered":"2024-03-06T07:23:54.650870+0000","last_clean":"2024-03-06T07:23:54.650870+0000","last_became_active":"2024-03-06T07:23:54.646838+0000","last_became_peered":"2024-03-06T07:23:54.646838+0000","last_unstale":"2024-03-06T07:23:54.650870+0000","last_undegraded":"2024-03-06T07:23:54.650870+0000","last_fullsized":"2024-03-06T07:23:54.650870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.731853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645622+0000","last_change":"2024-03-06T07:23:54.645622+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645622+0000","last_undegraded":"2024-03-06T07:23:54.645622+0000","last_fullsized":"2024-03-06T07:23:54.645622+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655052+0000","last_change":"2024-03-06T07:23:54.654021+0000","last_active":"2024-03-06T07:23:54.655052+0000","last_peered":"2024-03-06T07:23:54.655052+0000","last_clean":"2024-03-06T07:23:54.655052+0000","last_became_active":"2024-03-06T07:23:54.653706+0000","last_became_peered":"2024-03-06T07:23:54.653706+0000","last_unstale":"2024-03-06T07:23:54.655052+0000","last_undegraded":"2024-03-06T07:23:54.655052+0000","last_fullsized":"2024-03-06T07:23:54.655052+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:28.496649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640781+0000","last_change":"2024-03-06T07:23:54.640781+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640781+0000","last_undegraded":"2024-03-06T07:23:54.640781+0000","last_fullsized":"2024-03-06T07:23:54.640781+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655844+0000","last_change":"2024-03-06T07:23:54.653439+0000","last_active":"2024-03-06T07:23:54.655844+0000","last_peered":"2024-03-06T07:23:54.655844+0000","last_clean":"2024-03-06T07:23:54.655844+0000","last_became_active":"2024-03-06T07:23:54.653015+0000","last_became_peered":"2024-03-06T07:23:54.653015+0000","last_unstale":"2024-03-06T07:23:54.655844+0000","last_undegraded":"2024-03-06T07:23:54.655844+0000","last_fullsized":"2024-03-06T07:23:54.655844+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.500351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645281+0000","last_change":"2024-03-06T07:23:54.645281+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645281+0000","last_undegraded":"2024-03-06T07:23:54.645281+0000","last_fullsized":"2024-03-06T07:23:54.645281+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663119+0000","last_change":"2024-03-06T07:23:54.661828+0000","last_active":"2024-03-06T07:23:54.663119+0000","last_peered":"2024-03-06T07:23:54.663119+0000","last_clean":"2024-03-06T07:23:54.663119+0000","last_became_active":"2024-03-06T07:23:54.661129+0000","last_became_peered":"2024-03-06T07:23:54.661129+0000","last_unstale":"2024-03-06T07:23:54.663119+0000","last_undegraded":"2024-03-06T07:23:54.663119+0000","last_fullsized":"2024-03-06T07:23:54.663119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:19.451847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646653+0000","last_change":"2024-03-06T07:23:54.646653+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646653+0000","last_undegraded":"2024-03-06T07:23:54.646653+0000","last_fullsized":"2024-03-06T07:23:54.646653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.664675+0000","last_change":"2024-03-06T07:23:54.663162+0000","last_active":"2024-03-06T07:23:54.664675+0000","last_peered":"2024-03-06T07:23:54.664675+0000","last_clean":"2024-03-06T07:23:54.664675+0000","last_became_active":"2024-03-06T07:23:54.662778+0000","last_became_peered":"2024-03-06T07:23:54.662778+0000","last_unstale":"2024-03-06T07:23:54.664675+0000","last_undegraded":"2024-03-06T07:23:54.664675+0000","last_fullsized":"2024-03-06T07:23:54.664675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644876+0000","last_change":"2024-03-06T07:23:54.644876+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644876+0000","last_undegraded":"2024-03-06T07:23:54.644876+0000","last_fullsized":"2024-03-06T07:23:54.644876+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.662222+0000","last_change":"2024-03-06T07:23:54.651253+0000","last_active":"2024-03-06T07:23:54.662222+0000","last_peered":"2024-03-06T07:23:54.662222+0000","last_clean":"2024-03-06T07:23:54.662222+0000","last_became_active":"2024-03-06T07:23:54.650963+0000","last_became_peered":"2024-03-06T07:23:54.650963+0000","last_unstale":"2024-03-06T07:23:54.662222+0000","last_undegraded":"2024-03-06T07:23:54.662222+0000","last_fullsized":"2024-03-06T07:23:54.662222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:56.838295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646853+0000","last_change":"2024-03-06T07:23:54.646853+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646853+0000","last_undegraded":"2024-03-06T07:23:54.646853+0000","last_fullsized":"2024-03-06T07:23:54.646853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655850+0000","last_change":"2024-03-06T07:23:54.654189+0000","last_active":"2024-03-06T07:23:54.655850+0000","last_peered":"2024-03-06T07:23:54.655850+0000","last_clean":"2024-03-06T07:23:54.655850+0000","last_became_active":"2024-03-06T07:23:54.653889+0000","last_became_peered":"2024-03-06T07:23:54.653889+0000","last_unstale":"2024-03-06T07:23:54.655850+0000","last_undegraded":"2024-03-06T07:23:54.655850+0000","last_fullsized":"2024-03-06T07:23:54.655850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640921+0000","last_change":"2024-03-06T07:23:54.640921+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640921+0000","last_undegraded":"2024-03-06T07:23:54.640921+0000","last_fullsized":"2024-03-06T07:23:54.640921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653305+0000","last_change":"2024-03-06T07:23:54.649432+0000","last_active":"2024-03-06T07:23:54.653305+0000","last_peered":"2024-03-06T07:23:54.653305+0000","last_clean":"2024-03-06T07:23:54.653305+0000","last_became_active":"2024-03-06T07:23:54.649162+0000","last_became_peered":"2024-03-06T07:23:54.649162+0000","last_unstale":"2024-03-06T07:23:54.653305+0000","last_undegraded":"2024-03-06T07:23:54.653305+0000","last_fullsized":"2024-03-06T07:23:54.653305+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:38.700382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641700+0000","last_change":"2024-03-06T07:23:54.641700+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641700+0000","last_undegraded":"2024-03-06T07:23:54.641700+0000","last_fullsized":"2024-03-06T07:23:54.641700+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655269+0000","last_change":"2024-03-06T07:23:54.652266+0000","last_active":"2024-03-06T07:23:54.655269+0000","last_peered":"2024-03-06T07:23:54.655269+0000","last_clean":"2024-03-06T07:23:54.655269+0000","last_became_active":"2024-03-06T07:23:54.651291+0000","last_became_peered":"2024-03-06T07:23:54.651291+0000","last_unstale":"2024-03-06T07:23:54.655269+0000","last_undegraded":"2024-03-06T07:23:54.655269+0000","last_fullsized":"2024-03-06T07:23:54.655269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644697+0000","last_change":"2024-03-06T07:23:54.644697+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644697+0000","last_undegraded":"2024-03-06T07:23:54.644697+0000","last_fullsized":"2024-03-06T07:23:54.644697+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653133+0000","last_change":"2024-03-06T07:23:54.649706+0000","last_active":"2024-03-06T07:23:54.653133+0000","last_peered":"2024-03-06T07:23:54.653133+0000","last_clean":"2024-03-06T07:23:54.653133+0000","last_became_active":"2024-03-06T07:23:54.649451+0000","last_became_peered":"2024-03-06T07:23:54.649451+0000","last_unstale":"2024-03-06T07:23:54.653133+0000","last_undegraded":"2024-03-06T07:23:54.653133+0000","last_fullsized":"2024-03-06T07:23:54.653133+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:13.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.647941+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:23:54.647941+0000","last_peered":"2024-03-06T07:23:54.647941+0000","last_clean":"2024-03-06T07:23:54.647941+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:23:54.647941+0000","last_undegraded":"2024-03-06T07:23:54.647941+0000","last_fullsized":"2024-03-06T07:23:54.647941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":727628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":727628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:00.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2024-03-06T07:24:00.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646720+0000', 'last_change': '2024-03-06T07:23:54.646720+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646720+0000', 'last_undegraded': '2024-03-06T07:23:54.646720+0000', 'last_fullsized': '2024-03-06T07:23:54.646720+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:00.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2024-03-06T07:24:00.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.644574+0000', 'last_change': '2024-03-06T07:23:54.644574+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.644574+0000', 'last_undegraded': '2024-03-06T07:23:54.644574+0000', 'last_fullsized': '2024-03-06T07:23:54.644574+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2024-03-06T07:24:00.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.643477+0000', 'last_change': '2024-03-06T07:23:54.643477+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.643477+0000', 'last_undegraded': '2024-03-06T07:23:54.643477+0000', 'last_fullsized': '2024-03-06T07:23:54.643477+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.648478+0000', 'last_change': '2024-03-06T07:23:54.648478+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.648478+0000', 'last_undegraded': '2024-03-06T07:23:54.648478+0000', 'last_fullsized': '2024-03-06T07:23:54.648478+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.643699+0000', 'last_change': '2024-03-06T07:23:54.643699+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.643699+0000', 'last_undegraded': '2024-03-06T07:23:54.643699+0000', 'last_fullsized': '2024-03-06T07:23:54.643699+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.641775+0000', 'last_change': '2024-03-06T07:23:54.641775+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.641775+0000', 'last_undegraded': '2024-03-06T07:23:54.641775+0000', 'last_fullsized': '2024-03-06T07:23:54.641775+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.645152+0000', 'last_change': '2024-03-06T07:23:54.645152+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.645152+0000', 'last_undegraded': '2024-03-06T07:23:54.645152+0000', 'last_fullsized': '2024-03-06T07:23:54.645152+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.647218+0000', 'last_change': '2024-03-06T07:23:54.647218+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.647218+0000', 'last_undegraded': '2024-03-06T07:23:54.647218+0000', 'last_fullsized': '2024-03-06T07:23:54.647218+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2024-03-06T07:24:00.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.640159+0000', 'last_change': '2024-03-06T07:23:54.640159+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.640159+0000', 'last_undegraded': '2024-03-06T07:23:54.640159+0000', 'last_fullsized': '2024-03-06T07:23:54.640159+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.648487+0000', 'last_change': '2024-03-06T07:23:54.648487+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.648487+0000', 'last_undegraded': '2024-03-06T07:23:54.648487+0000', 'last_fullsized': '2024-03-06T07:23:54.648487+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646918+0000', 'last_change': '2024-03-06T07:23:54.646918+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646918+0000', 'last_undegraded': '2024-03-06T07:23:54.646918+0000', 'last_fullsized': '2024-03-06T07:23:54.646918+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.643738+0000', 'last_change': '2024-03-06T07:23:54.643738+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.643738+0000', 'last_undegraded': '2024-03-06T07:23:54.643738+0000', 'last_fullsized': '2024-03-06T07:23:54.643738+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.643236+0000', 'last_change': '2024-03-06T07:23:54.643236+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.643236+0000', 'last_undegraded': '2024-03-06T07:23:54.643236+0000', 'last_fullsized': '2024-03-06T07:23:54.643236+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2024-03-06T07:24:00.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.641298+0000', 'last_change': '2024-03-06T07:23:54.641298+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.641298+0000', 'last_undegraded': '2024-03-06T07:23:54.641298+0000', 'last_fullsized': '2024-03-06T07:23:54.641298+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2024-03-06T07:24:00.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646515+0000', 'last_change': '2024-03-06T07:23:54.646515+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646515+0000', 'last_undegraded': '2024-03-06T07:23:54.646515+0000', 'last_fullsized': '2024-03-06T07:23:54.646515+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2024-03-06T07:24:00.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.644556+0000', 'last_change': '2024-03-06T07:23:54.644556+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.644556+0000', 'last_undegraded': '2024-03-06T07:23:54.644556+0000', 'last_fullsized': '2024-03-06T07:23:54.644556+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2024-03-06T07:24:00.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646653+0000', 'last_change': '2024-03-06T07:23:54.646653+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646653+0000', 'last_undegraded': '2024-03-06T07:23:54.646653+0000', 'last_fullsized': '2024-03-06T07:23:54.646653+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2024-03-06T07:24:00.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.644109+0000', 'last_change': '2024-03-06T07:23:54.644109+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.644109+0000', 'last_undegraded': '2024-03-06T07:23:54.644109+0000', 'last_fullsized': '2024-03-06T07:23:54.644109+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.640796+0000', 'last_change': '2024-03-06T07:23:54.640796+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.640796+0000', 'last_undegraded': '2024-03-06T07:23:54.640796+0000', 'last_fullsized': '2024-03-06T07:23:54.640796+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.643640+0000', 'last_change': '2024-03-06T07:23:54.643640+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.643640+0000', 'last_undegraded': '2024-03-06T07:23:54.643640+0000', 'last_fullsized': '2024-03-06T07:23:54.643640+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.641081+0000', 'last_change': '2024-03-06T07:23:54.641081+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.641081+0000', 'last_undegraded': '2024-03-06T07:23:54.641081+0000', 'last_fullsized': '2024-03-06T07:23:54.641081+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.639615+0000', 'last_change': '2024-03-06T07:23:54.639615+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.639615+0000', 'last_undegraded': '2024-03-06T07:23:54.639615+0000', 'last_fullsized': '2024-03-06T07:23:54.639615+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646031+0000', 'last_change': '2024-03-06T07:23:54.646031+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646031+0000', 'last_undegraded': '2024-03-06T07:23:54.646031+0000', 'last_fullsized': '2024-03-06T07:23:54.646031+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:00.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.642902+0000', 'last_change': '2024-03-06T07:23:54.642902+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.642902+0000', 'last_undegraded': '2024-03-06T07:23:54.642902+0000', 'last_fullsized': '2024-03-06T07:23:54.642902+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.648463+0000', 'last_change': '2024-03-06T07:23:54.648463+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.648463+0000', 'last_undegraded': '2024-03-06T07:23:54.648463+0000', 'last_fullsized': '2024-03-06T07:23:54.648463+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.643640+0000', 'last_change': '2024-03-06T07:23:54.643640+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.643640+0000', 'last_undegraded': '2024-03-06T07:23:54.643640+0000', 'last_fullsized': '2024-03-06T07:23:54.643640+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.647824+0000', 'last_change': '2024-03-06T07:23:54.647824+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.647824+0000', 'last_undegraded': '2024-03-06T07:23:54.647824+0000', 'last_fullsized': '2024-03-06T07:23:54.647824+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.643326+0000', 'last_change': '2024-03-06T07:23:54.643326+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.643326+0000', 'last_undegraded': '2024-03-06T07:23:54.643326+0000', 'last_fullsized': '2024-03-06T07:23:54.643326+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2024-03-06T07:24:00.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.640572+0000', 'last_change': '2024-03-06T07:23:54.640572+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.640572+0000', 'last_undegraded': '2024-03-06T07:23:54.640572+0000', 'last_fullsized': '2024-03-06T07:23:54.640572+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.645644+0000', 'last_change': '2024-03-06T07:23:54.645644+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.645644+0000', 'last_undegraded': '2024-03-06T07:23:54.645644+0000', 'last_fullsized': '2024-03-06T07:23:54.645644+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.647662+0000', 'last_change': '2024-03-06T07:23:54.647662+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.647662+0000', 'last_undegraded': '2024-03-06T07:23:54.647662+0000', 'last_fullsized': '2024-03-06T07:23:54.647662+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646477+0000', 'last_change': '2024-03-06T07:23:54.646477+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646477+0000', 'last_undegraded': '2024-03-06T07:23:54.646477+0000', 'last_fullsized': '2024-03-06T07:23:54.646477+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.647114+0000', 'last_change': '2024-03-06T07:23:54.647114+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.647114+0000', 'last_undegraded': '2024-03-06T07:23:54.647114+0000', 'last_fullsized': '2024-03-06T07:23:54.647114+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-03-06T07:24:00.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.644314+0000', 'last_change': '2024-03-06T07:23:54.644314+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.644314+0000', 'last_undegraded': '2024-03-06T07:23:54.644314+0000', 'last_fullsized': '2024-03-06T07:23:54.644314+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.639780+0000', 'last_change': '2024-03-06T07:23:54.639780+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.639780+0000', 'last_undegraded': '2024-03-06T07:23:54.639780+0000', 'last_fullsized': '2024-03-06T07:23:54.639780+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.647238+0000', 'last_change': '2024-03-06T07:23:54.647238+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.647238+0000', 'last_undegraded': '2024-03-06T07:23:54.647238+0000', 'last_fullsized': '2024-03-06T07:23:54.647238+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.645225+0000', 'last_change': '2024-03-06T07:23:54.645225+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.645225+0000', 'last_undegraded': '2024-03-06T07:23:54.645225+0000', 'last_fullsized': '2024-03-06T07:23:54.645225+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.643151+0000', 'last_change': '2024-03-06T07:23:54.643151+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.643151+0000', 'last_undegraded': '2024-03-06T07:23:54.643151+0000', 'last_fullsized': '2024-03-06T07:23:54.643151+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-03-06T07:24:00.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.640802+0000', 'last_change': '2024-03-06T07:23:54.640802+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.640802+0000', 'last_undegraded': '2024-03-06T07:23:54.640802+0000', 'last_fullsized': '2024-03-06T07:23:54.640802+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646552+0000', 'last_change': '2024-03-06T07:23:54.646552+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646552+0000', 'last_undegraded': '2024-03-06T07:23:54.646552+0000', 'last_fullsized': '2024-03-06T07:23:54.646552+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.642849+0000', 'last_change': '2024-03-06T07:23:54.642849+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.642849+0000', 'last_undegraded': '2024-03-06T07:23:54.642849+0000', 'last_fullsized': '2024-03-06T07:23:54.642849+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.642632+0000', 'last_change': '2024-03-06T07:23:54.642632+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.642632+0000', 'last_undegraded': '2024-03-06T07:23:54.642632+0000', 'last_fullsized': '2024-03-06T07:23:54.642632+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.640747+0000', 'last_change': '2024-03-06T07:23:54.640747+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.640747+0000', 'last_undegraded': '2024-03-06T07:23:54.640747+0000', 'last_fullsized': '2024-03-06T07:23:54.640747+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.643899+0000', 'last_change': '2024-03-06T07:23:54.643899+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.643899+0000', 'last_undegraded': '2024-03-06T07:23:54.643899+0000', 'last_fullsized': '2024-03-06T07:23:54.643899+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.648064+0000', 'last_change': '2024-03-06T07:23:54.648064+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.648064+0000', 'last_undegraded': '2024-03-06T07:23:54.648064+0000', 'last_fullsized': '2024-03-06T07:23:54.648064+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.642671+0000', 'last_change': '2024-03-06T07:23:54.642671+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.642671+0000', 'last_undegraded': '2024-03-06T07:23:54.642671+0000', 'last_fullsized': '2024-03-06T07:23:54.642671+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646070+0000', 'last_change': '2024-03-06T07:23:54.646070+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646070+0000', 'last_undegraded': '2024-03-06T07:23:54.646070+0000', 'last_fullsized': '2024-03-06T07:23:54.646070+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646049+0000', 'last_change': '2024-03-06T07:23:54.646049+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646049+0000', 'last_undegraded': '2024-03-06T07:23:54.646049+0000', 'last_fullsized': '2024-03-06T07:23:54.646049+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.640452+0000', 'last_change': '2024-03-06T07:23:54.640452+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.640452+0000', 'last_undegraded': '2024-03-06T07:23:54.640452+0000', 'last_fullsized': '2024-03-06T07:23:54.640452+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-03-06T07:24:00.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.645677+0000', 'last_change': '2024-03-06T07:23:54.645677+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.645677+0000', 'last_undegraded': '2024-03-06T07:23:54.645677+0000', 'last_fullsized': '2024-03-06T07:23:54.645677+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.642632+0000', 'last_change': '2024-03-06T07:23:54.642632+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.642632+0000', 'last_undegraded': '2024-03-06T07:23:54.642632+0000', 'last_fullsized': '2024-03-06T07:23:54.642632+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.642883+0000', 'last_change': '2024-03-06T07:23:54.642883+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.642883+0000', 'last_undegraded': '2024-03-06T07:23:54.642883+0000', 'last_fullsized': '2024-03-06T07:23:54.642883+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.642264+0000', 'last_change': '2024-03-06T07:23:54.642264+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.642264+0000', 'last_undegraded': '2024-03-06T07:23:54.642264+0000', 'last_fullsized': '2024-03-06T07:23:54.642264+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.642349+0000', 'last_change': '2024-03-06T07:23:54.642349+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.642349+0000', 'last_undegraded': '2024-03-06T07:23:54.642349+0000', 'last_fullsized': '2024-03-06T07:23:54.642349+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-03-06T07:24:00.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.641604+0000', 'last_change': '2024-03-06T07:23:54.641604+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.641604+0000', 'last_undegraded': '2024-03-06T07:23:54.641604+0000', 'last_fullsized': '2024-03-06T07:23:54.641604+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.645622+0000', 'last_change': '2024-03-06T07:23:54.645622+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.645622+0000', 'last_undegraded': '2024-03-06T07:23:54.645622+0000', 'last_fullsized': '2024-03-06T07:23:54.645622+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.640781+0000', 'last_change': '2024-03-06T07:23:54.640781+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.640781+0000', 'last_undegraded': '2024-03-06T07:23:54.640781+0000', 'last_fullsized': '2024-03-06T07:23:54.640781+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.645281+0000', 'last_change': '2024-03-06T07:23:54.645281+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.645281+0000', 'last_undegraded': '2024-03-06T07:23:54.645281+0000', 'last_fullsized': '2024-03-06T07:23:54.645281+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646653+0000', 'last_change': '2024-03-06T07:23:54.646653+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646653+0000', 'last_undegraded': '2024-03-06T07:23:54.646653+0000', 'last_fullsized': '2024-03-06T07:23:54.646653+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-03-06T07:24:00.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.644876+0000', 'last_change': '2024-03-06T07:23:54.644876+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.644876+0000', 'last_undegraded': '2024-03-06T07:23:54.644876+0000', 'last_fullsized': '2024-03-06T07:23:54.644876+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:00.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-03-06T07:24:00.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.646853+0000', 'last_change': '2024-03-06T07:23:54.646853+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.646853+0000', 'last_undegraded': '2024-03-06T07:23:54.646853+0000', 'last_fullsized': '2024-03-06T07:23:54.646853+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:00.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-03-06T07:24:00.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.640921+0000', 'last_change': '2024-03-06T07:23:54.640921+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.640921+0000', 'last_undegraded': '2024-03-06T07:23:54.640921+0000', 'last_fullsized': '2024-03-06T07:23:54.640921+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:00.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-03-06T07:24:00.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.641700+0000', 'last_change': '2024-03-06T07:23:54.641700+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.641700+0000', 'last_undegraded': '2024-03-06T07:23:54.641700+0000', 'last_fullsized': '2024-03-06T07:23:54.641700+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:00.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-03-06T07:24:00.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:54.644697+0000', 'last_change': '2024-03-06T07:23:54.644697+0000', 'last_active': '2024-03-06T07:23:54.629715+0000', 'last_peered': '2024-03-06T07:23:54.629715+0000', 'last_clean': '2024-03-06T07:23:54.629715+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:54.644697+0000', 'last_undegraded': '2024-03-06T07:23:54.644697+0000', 'last_fullsized': '2024-03-06T07:23:54.644697+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:54.629715+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:00.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:24:01.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.125+0000 7fbafeb3e640 1 Processor -- start 2024-03-06T07:24:01.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.125+0000 7fbafeb3e640 1 -- start start 2024-03-06T07:24:01.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.126+0000 7fbafeb3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf815e350 0x7fbaf80d2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.126+0000 7fbafeb3e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf8071c30 con 0x7fbaf80d3470 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.126+0000 7fbafeb3e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf8071f10 con 0x7fbaf8161eb0 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.126+0000 7fbafeb3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf80721f0 con 0x7fbaf815e350 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.126+0000 7fbaf7fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbaf80d3470 0x7fbaf8072cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36484/0 (socket says 172.21.15.27:36484) 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.126+0000 7fbaf7fff640 1 -- 172.21.15.27:0/3628496816 learned_addr learned my addr 172.21.15.27:0/3628496816 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbafc8da640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf815e350 0x7fbaf80d2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377332455 0 0) 0x7fbaf8071c30 con 0x7fbaf80d3470 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8003520 con 0x7fbaf80d3470 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121230960 0 0) 0x7fbaf8071f10 con 0x7fbaf8161eb0 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf8071c30 con 0x7fbaf8161eb0 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1218526942 0 0) 0x7fbae8003520 con 0x7fbaf80d3470 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf815e350 msgr2=0x7fbaf80d2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf815e350 0x7fbaf80d2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 >> v1:172.21.15.69:6789/0 conn(0x7fbaf8161eb0 legacy=0x7fbaf80b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf80724d0 con 0x7fbaf80d3470 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fbae4001670 con 0x7fbaf80d3470 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 >> v1:172.21.15.27:6789/0 conn(0x7fbaf80d3470 legacy=0x7fbaf8072cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae8003b70 0x7fbae8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf815e350 0x7fbae8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae8004d40 0x7fbae80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf80721f0 con 0x7fbae8004d40 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae800bac0 con 0x7fbae8003b70 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae800bcc0 con 0x7fbaf815e350 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbaf7fff640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf815e350 0x7fbae8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbafc8da640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae8003b70 0x7fbae8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.127+0000 7fbafd0db640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae8004d40 0x7fbae80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.128+0000 7fbafd0db640 1 -- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf815e350 msgr2=0x7fbae8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.128+0000 7fbafd0db640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf815e350 0x7fbae8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.128+0000 7fbafd0db640 1 -- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae8003b70 msgr2=0x7fbae8003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.128+0000 7fbafd0db640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae8003b70 0x7fbae8003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.128+0000 7fbafd0db640 1 -- 172.21.15.27:0/3628496816 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae8003520 con 0x7fbae8004d40 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.128+0000 7fbafc8da640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae8003b70 0x7fbae8003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.129+0000 7fbafd0db640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae8004d40 0x7fbae80051c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0001fd0 tx=0x7fbae8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de1078129e484c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.129+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0001bf0 con 0x7fbae8004d40 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.129+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf0003b70 con 0x7fbae8004d40 2024-03-06T07:24:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.129+0000 7fbaf77fe640 1 -- 172.21.15.27:0/3628496816 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf00050d0 con 0x7fbae8004d40 2024-03-06T07:24:01.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.130+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3628496816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae8004d40 msgr2=0x7fbae80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.130+0000 7fbafeb3e640 1 --2- 172.21.15.27:0/3628496816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae8004d40 0x7fbae80051c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0001fd0 tx=0x7fbae8005830 comp rx=0 tx=0).stop 2024-03-06T07:24:01.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.130+0000 7fbafd0db640 1 -- 172.21.15.27:0/3628496816 reap_dead start 2024-03-06T07:24:01.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.130+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3628496816 shutdown_connections 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.130+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3628496816 >> 172.21.15.27:0/3628496816 conn(0x7fbaf80d4230 msgr2=0x7fbaf80d4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.130+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3628496816 shutdown_connections 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.130+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3628496816 wait complete. 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.131+0000 7fbafeb3e640 1 Processor -- start 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.131+0000 7fbafeb3e640 1 -- start start 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafeb3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8159300 0x7fbaf80b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafeb3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf81f49c0 0x7fbaf80c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafeb3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf815c230 0x7fbaf815c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafeb3e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf80719d0 con 0x7fbaf8159300 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafeb3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf8071850 con 0x7fbaf815c230 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafeb3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf80733e0 con 0x7fbaf81f49c0 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafc8da640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8159300 0x7fbaf80b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafc8da640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8159300 0x7fbaf80b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48516/0 (socket says 172.21.15.27:48516) 2024-03-06T07:24:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafc8da640 1 -- 172.21.15.27:0/3586444442 learned_addr learned my addr 172.21.15.27:0/3586444442 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbaf7fff640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf81f49c0 0x7fbaf80c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafd0db640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf815c230 0x7fbaf815c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafc8da640 1 -- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf81f49c0 msgr2=0x7fbaf80c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafc8da640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf81f49c0 0x7fbaf80c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafc8da640 1 -- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf815c230 msgr2=0x7fbaf815c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafc8da640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf815c230 0x7fbaf815c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.132+0000 7fbafc8da640 1 -- 172.21.15.27:0/3586444442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf80c8b40 con 0x7fbaf8159300 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.133+0000 7fbafc8da640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8159300 0x7fbaf80b48a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbaec011ca0 tx=0x7fbaec00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.133+0000 7fbaf57fa640 1 -- 172.21.15.27:0/3586444442 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec00fd20 con 0x7fbaf8159300 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.133+0000 7fbaf57fa640 1 -- 172.21.15.27:0/3586444442 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec014440 con 0x7fbaf8159300 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.133+0000 7fbaf57fa640 1 -- 172.21.15.27:0/3586444442 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec014830 con 0x7fbaf8159300 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.133+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf80b4e80 con 0x7fbaf8159300 2024-03-06T07:24:01.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.133+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf80b5440 con 0x7fbaf8159300 2024-03-06T07:24:01.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.134+0000 7fbaf57fa640 1 -- 172.21.15.27:0/3586444442 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaec012040 con 0x7fbaf8159300 2024-03-06T07:24:01.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.134+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf815cbf0 con 0x7fbaf8159300 2024-03-06T07:24:01.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.135+0000 7fbaf57fa640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbadc06b4d0 0x7fbadc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.135+0000 7fbaf57fa640 1 -- 172.21.15.27:0/3586444442 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbaec08aa80 con 0x7fbaf8159300 2024-03-06T07:24:01.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.135+0000 7fbaf7fff640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbadc06b4d0 0x7fbadc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.135+0000 7fbaf7fff640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbadc06b4d0 0x7fbadc06d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80b68e0 tx=0x7fbae4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.139+0000 7fbaf57fa640 1 -- 172.21.15.27:0/3586444442 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaec019030 con 0x7fbaf8159300 2024-03-06T07:24:01.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.447+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbaf8068700 con 0x7fbaf8159300 2024-03-06T07:24:01.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.448+0000 7fbaf57fa640 1 -- 172.21.15.27:0/3586444442 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1452 (secure 0 0 0) 0x7fbaec059900 con 0x7fbaf8159300 2024-03-06T07:24:01.450 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:01.450 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":97,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709754,"num_in_osds":8,"osd_in_since":1709709745,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225456128,"bytes_avail":772868657152,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5286}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-06T07:23:43.999166+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:24:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.449+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbadc06b4d0 msgr2=0x7fbadc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbadc06b4d0 0x7fbadc06d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80b68e0 tx=0x7fbae4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8159300 msgr2=0x7fbaf80b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8159300 0x7fbaf80b48a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbaec011ca0 tx=0x7fbaec00f610 comp rx=0 tx=0).stop 2024-03-06T07:24:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 shutdown_connections 2024-03-06T07:24:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbadc06b4d0 0x7fbadc06d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf815c230 0x7fbaf815c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbaf81f49c0 0x7fbaf80c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 --2- 172.21.15.27:0/3586444442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8159300 0x7fbaf80b48a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 >> 172.21.15.27:0/3586444442 conn(0x7fbaf80d4230 msgr2=0x7fbaf80b93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 shutdown_connections 2024-03-06T07:24:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.450+0000 7fbafeb3e640 1 -- 172.21.15.27:0/3586444442 wait complete. 2024-03-06T07:24:01.463 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:24:01.463 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:01.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.567+0000 7f3ab3c4e640 1 Processor -- start 2024-03-06T07:24:01.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.567+0000 7f3ab3c4e640 1 -- start start 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab3c4e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3aac06b5c0 0x7f3aac0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab3c4e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3aac0ae9b0 0x7f3aac0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab3c4e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac0b3340 con 0x7f3aac0b0e90 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab3c4e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aac0cfa00 con 0x7f3aac0ae9b0 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab3c4e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aac0cee90 con 0x7f3aac06b5c0 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab21eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3aac0ae9b0 0x7f3aac0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab21eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3aac0ae9b0 0x7f3aac0d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48530/0 (socket says 172.21.15.27:48530) 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab11e9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3aac0b0e90 0x7f3aac06b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36494/0 (socket says 172.21.15.27:36494) 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab21eb640 1 -- 172.21.15.27:0/412633016 learned_addr learned my addr 172.21.15.27:0/412633016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab19ea640 1 --2- 172.21.15.27:0/412633016 >> v2:172.21.15.69:3300/0 conn(0x7f3aac06b5c0 0x7f3aac0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab21eb640 1 -- 172.21.15.27:0/412633016 >> v2:172.21.15.69:3300/0 conn(0x7f3aac06b5c0 msgr2=0x7f3aac0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab21eb640 1 --2- 172.21.15.27:0/412633016 >> v2:172.21.15.69:3300/0 conn(0x7f3aac06b5c0 0x7f3aac0b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab21eb640 1 -- 172.21.15.27:0/412633016 >> v1:172.21.15.27:6789/0 conn(0x7f3aac0b0e90 legacy=0x7f3aac06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab09e8640 1 -- 172.21.15.27:0/412633016 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872594666 0 0) 0x7f3aac0b3340 con 0x7f3aac0b0e90 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.568+0000 7f3ab21eb640 1 -- 172.21.15.27:0/412633016 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac0cfc50 con 0x7f3aac0ae9b0 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.569+0000 7f3ab21eb640 1 --2- 172.21.15.27:0/412633016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0ae9b0 0x7f3aac0d03a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3aac071b00 tx=0x7f3aa402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f5254694f24044 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.569+0000 7f3ab09e8640 1 -- 172.21.15.27:0/412633016 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa402ca80 con 0x7f3aac0ae9b0 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.569+0000 7f3ab09e8640 1 -- 172.21.15.27:0/412633016 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa402cc20 con 0x7f3aac0ae9b0 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.569+0000 7f3ab09e8640 1 -- 172.21.15.27:0/412633016 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa40343f0 con 0x7f3aac0ae9b0 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.570+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/412633016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0ae9b0 msgr2=0x7f3aac0d03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.570+0000 7f3ab3c4e640 1 --2- 172.21.15.27:0/412633016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0ae9b0 0x7f3aac0d03a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3aac071b00 tx=0x7f3aa402bee0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.570+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/412633016 shutdown_connections 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.570+0000 7f3ab3c4e640 1 --2- 172.21.15.27:0/412633016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0ae9b0 0x7f3aac0d03a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.570+0000 7f3ab3c4e640 1 --2- 172.21.15.27:0/412633016 >> v2:172.21.15.69:3300/0 conn(0x7f3aac06b5c0 0x7f3aac0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.570+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/412633016 >> 172.21.15.27:0/412633016 conn(0x7f3aac0b36d0 msgr2=0x7f3aac0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.570+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/412633016 shutdown_connections 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.570+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/412633016 wait complete. 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.571+0000 7f3ab3c4e640 1 Processor -- start 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.571+0000 7f3ab3c4e640 1 -- start start 2024-03-06T07:24:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab3c4e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3aac06b5c0 0x7f3aac0c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab3c4e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3aac0ae9b0 0x7f3aac0c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab3c4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0b0e90 0x7f3aac1de0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab3c4e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac06c910 con 0x7f3aac0b0e90 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab3c4e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac06c790 con 0x7f3aac06b5c0 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab3c4e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac0aedc0 con 0x7f3aac0ae9b0 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab21eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0b0e90 0x7f3aac1de0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab21eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0b0e90 0x7f3aac1de0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48538/0 (socket says 172.21.15.27:48538) 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab21eb640 1 -- 172.21.15.27:0/2534286505 learned_addr learned my addr 172.21.15.27:0/2534286505 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab19ea640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3aac06b5c0 0x7f3aac0c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab11e9640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3aac0ae9b0 0x7f3aac0c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.572+0000 7f3ab21eb640 1 -- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3aac0ae9b0 msgr2=0x7f3aac0c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3ab21eb640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3aac0ae9b0 0x7f3aac0c7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3ab21eb640 1 -- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3aac06b5c0 msgr2=0x7f3aac0c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3ab21eb640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3aac06b5c0 0x7f3aac0c6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3ab21eb640 1 -- 172.21.15.27:0/2534286505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac0c6ac0 con 0x7f3aac0b0e90 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3ab11e9640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3aac0ae9b0 0x7f3aac0c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3ab21eb640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0b0e90 0x7f3aac1de0d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4036040 tx=0x7f3aa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3aa27fc640 1 -- 172.21.15.27:0/2534286505 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4043030 con 0x7f3aac0b0e90 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3aa27fc640 1 -- 172.21.15.27:0/2534286505 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa403e450 con 0x7f3aac0b0e90 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aac0c6da0 con 0x7f3aac0b0e90 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3aa27fc640 1 -- 172.21.15.27:0/2534286505 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4009510 con 0x7f3aac0b0e90 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.573+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aac0c7360 con 0x7f3aac0b0e90 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.574+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aac0d2a70 con 0x7f3aac0b0e90 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.575+0000 7f3aa27fc640 1 -- 172.21.15.27:0/2534286505 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3aa4046070 con 0x7f3aac0b0e90 2024-03-06T07:24:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.575+0000 7f3aa27fc640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a9006b4d0 0x7f3a9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.575+0000 7f3aa27fc640 1 -- 172.21.15.27:0/2534286505 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3aa40ac640 con 0x7f3aac0b0e90 2024-03-06T07:24:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.575+0000 7f3ab19ea640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a9006b4d0 0x7f3a9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.576+0000 7f3ab19ea640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a9006b4d0 0x7f3a9006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8006fd0 tx=0x7f3aa8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.584+0000 7f3aa27fc640 1 -- 172.21.15.27:0/2534286505 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3aac0d2a70 con 0x7f3aac0b0e90 2024-03-06T07:24:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.834+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3aac068700 con 0x7f3a9006b4d0 2024-03-06T07:24:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.841+0000 7f3aa27fc640 1 -- 172.21.15.27:0/2534286505 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246634 (secure 0 0 0) 0x7f3aac068700 con 0x7f3a9006b4d0 2024-03-06T07:24:01.846 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:01.847 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:24:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a9006b4d0 msgr2=0x7f3a9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a9006b4d0 0x7f3a9006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8006fd0 tx=0x7f3aa8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0b0e90 msgr2=0x7f3aac1de0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0b0e90 0x7f3aac1de0d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4036040 tx=0x7f3aa4004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 shutdown_connections 2024-03-06T07:24:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a9006b4d0 0x7f3a9006d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3aac0b0e90 0x7f3aac1de0d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3aac0ae9b0 0x7f3aac0c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 --2- 172.21.15.27:0/2534286505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3aac06b5c0 0x7f3aac0c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 >> 172.21.15.27:0/2534286505 conn(0x7f3aac0b36d0 msgr2=0x7f3aac0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 shutdown_connections 2024-03-06T07:24:01.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:01.844+0000 7f3ab3c4e640 1 -- 172.21.15.27:0/2534286505 wait complete. 2024-03-06T07:24:01.857 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-03-06T07:24:00.002784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220172,"kb_used_data":4012,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754548,"statfs":{"total":773094113280,"available":772868657152,"internally_reserved":0,"allocated":4108288,"data_stored":2297617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001489"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646720+0000","last_change":"2024-03-06T07:23:54.646720+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646720+0000","last_undegraded":"2024-03-06T07:23:54.646720+0000","last_fullsized":"2024-03-06T07:23:54.646720+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644574+0000","last_change":"2024-03-06T07:23:54.644574+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644574+0000","last_undegraded":"2024-03-06T07:23:54.644574+0000","last_fullsized":"2024-03-06T07:23:54.644574+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643477+0000","last_change":"2024-03-06T07:23:54.643477+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643477+0000","last_undegraded":"2024-03-06T07:23:54.643477+0000","last_fullsized":"2024-03-06T07:23:54.643477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648478+0000","last_change":"2024-03-06T07:23:54.648478+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648478+0000","last_undegraded":"2024-03-06T07:23:54.648478+0000","last_fullsized":"2024-03-06T07:23:54.648478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643699+0000","last_change":"2024-03-06T07:23:54.643699+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643699+0000","last_undegraded":"2024-03-06T07:23:54.643699+0000","last_fullsized":"2024-03-06T07:23:54.643699+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641775+0000","last_change":"2024-03-06T07:23:54.641775+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641775+0000","last_undegraded":"2024-03-06T07:23:54.641775+0000","last_fullsized":"2024-03-06T07:23:54.641775+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645152+0000","last_change":"2024-03-06T07:23:54.645152+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645152+0000","last_undegraded":"2024-03-06T07:23:54.645152+0000","last_fullsized":"2024-03-06T07:23:54.645152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647218+0000","last_change":"2024-03-06T07:23:54.647218+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647218+0000","last_undegraded":"2024-03-06T07:23:54.647218+0000","last_fullsized":"2024-03-06T07:23:54.647218+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640159+0000","last_change":"2024-03-06T07:23:54.640159+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640159+0000","last_undegraded":"2024-03-06T07:23:54.640159+0000","last_fullsized":"2024-03-06T07:23:54.640159+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648487+0000","last_change":"2024-03-06T07:23:54.648487+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648487+0000","last_undegraded":"2024-03-06T07:23:54.648487+0000","last_fullsized":"2024-03-06T07:23:54.648487+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646918+0000","last_change":"2024-03-06T07:23:54.646918+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646918+0000","last_undegraded":"2024-03-06T07:23:54.646918+0000","last_fullsized":"2024-03-06T07:23:54.646918+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643738+0000","last_change":"2024-03-06T07:23:54.643738+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643738+0000","last_undegraded":"2024-03-06T07:23:54.643738+0000","last_fullsized":"2024-03-06T07:23:54.643738+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643236+0000","last_change":"2024-03-06T07:23:54.643236+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643236+0000","last_undegraded":"2024-03-06T07:23:54.643236+0000","last_fullsized":"2024-03-06T07:23:54.643236+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641298+0000","last_change":"2024-03-06T07:23:54.641298+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641298+0000","last_undegraded":"2024-03-06T07:23:54.641298+0000","last_fullsized":"2024-03-06T07:23:54.641298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646515+0000","last_change":"2024-03-06T07:23:54.646515+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646515+0000","last_undegraded":"2024-03-06T07:23:54.646515+0000","last_fullsized":"2024-03-06T07:23:54.646515+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644556+0000","last_change":"2024-03-06T07:23:54.644556+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644556+0000","last_undegraded":"2024-03-06T07:23:54.644556+0000","last_fullsized":"2024-03-06T07:23:54.644556+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646653+0000","last_change":"2024-03-06T07:23:54.646653+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646653+0000","last_undegraded":"2024-03-06T07:23:54.646653+0000","last_fullsized":"2024-03-06T07:23:54.646653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644109+0000","last_change":"2024-03-06T07:23:54.644109+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644109+0000","last_undegraded":"2024-03-06T07:23:54.644109+0000","last_fullsized":"2024-03-06T07:23:54.644109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640796+0000","last_change":"2024-03-06T07:23:54.640796+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640796+0000","last_undegraded":"2024-03-06T07:23:54.640796+0000","last_fullsized":"2024-03-06T07:23:54.640796+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643640+0000","last_change":"2024-03-06T07:23:54.643640+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643640+0000","last_undegraded":"2024-03-06T07:23:54.643640+0000","last_fullsized":"2024-03-06T07:23:54.643640+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641081+0000","last_change":"2024-03-06T07:23:54.641081+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641081+0000","last_undegraded":"2024-03-06T07:23:54.641081+0000","last_fullsized":"2024-03-06T07:23:54.641081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.639615+0000","last_change":"2024-03-06T07:23:54.639615+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.639615+0000","last_undegraded":"2024-03-06T07:23:54.639615+0000","last_fullsized":"2024-03-06T07:23:54.639615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646031+0000","last_change":"2024-03-06T07:23:54.646031+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646031+0000","last_undegraded":"2024-03-06T07:23:54.646031+0000","last_fullsized":"2024-03-06T07:23:54.646031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642902+0000","last_change":"2024-03-06T07:23:54.642902+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642902+0000","last_undegraded":"2024-03-06T07:23:54.642902+0000","last_fullsized":"2024-03-06T07:23:54.642902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648463+0000","last_change":"2024-03-06T07:23:54.648463+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648463+0000","last_undegraded":"2024-03-06T07:23:54.648463+0000","last_fullsized":"2024-03-06T07:23:54.648463+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643640+0000","last_change":"2024-03-06T07:23:54.643640+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643640+0000","last_undegraded":"2024-03-06T07:23:54.643640+0000","last_fullsized":"2024-03-06T07:23:54.643640+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647824+0000","last_change":"2024-03-06T07:23:54.647824+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647824+0000","last_undegraded":"2024-03-06T07:23:54.647824+0000","last_fullsized":"2024-03-06T07:23:54.647824+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643326+0000","last_change":"2024-03-06T07:23:54.643326+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643326+0000","last_undegraded":"2024-03-06T07:23:54.643326+0000","last_fullsized":"2024-03-06T07:23:54.643326+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640572+0000","last_change":"2024-03-06T07:23:54.640572+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640572+0000","last_undegraded":"2024-03-06T07:23:54.640572+0000","last_fullsized":"2024-03-06T07:23:54.640572+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645644+0000","last_change":"2024-03-06T07:23:54.645644+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645644+0000","last_undegraded":"2024-03-06T07:23:54.645644+0000","last_fullsized":"2024-03-06T07:23:54.645644+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647662+0000","last_change":"2024-03-06T07:23:54.647662+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647662+0000","last_undegraded":"2024-03-06T07:23:54.647662+0000","last_fullsized":"2024-03-06T07:23:54.647662+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646477+0000","last_change":"2024-03-06T07:23:54.646477+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646477+0000","last_undegraded":"2024-03-06T07:23:54.646477+0000","last_fullsized":"2024-03-06T07:23:54.646477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647114+0000","last_change":"2024-03-06T07:23:54.647114+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647114+0000","last_undegraded":"2024-03-06T07:23:54.647114+0000","last_fullsized":"2024-03-06T07:23:54.647114+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656288+0000","last_change":"2024-03-06T07:23:54.655354+0000","last_active":"2024-03-06T07:23:54.656288+0000","last_peered":"2024-03-06T07:23:54.656288+0000","last_clean":"2024-03-06T07:23:54.656288+0000","last_became_active":"2024-03-06T07:23:54.655053+0000","last_became_peered":"2024-03-06T07:23:54.655053+0000","last_unstale":"2024-03-06T07:23:54.656288+0000","last_undegraded":"2024-03-06T07:23:54.656288+0000","last_fullsized":"2024-03-06T07:23:54.656288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:19.451847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644314+0000","last_change":"2024-03-06T07:23:54.644314+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644314+0000","last_undegraded":"2024-03-06T07:23:54.644314+0000","last_fullsized":"2024-03-06T07:23:54.644314+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663137+0000","last_change":"2024-03-06T07:23:54.662097+0000","last_active":"2024-03-06T07:23:54.663137+0000","last_peered":"2024-03-06T07:23:54.663137+0000","last_clean":"2024-03-06T07:23:54.663137+0000","last_became_active":"2024-03-06T07:23:54.661544+0000","last_became_peered":"2024-03-06T07:23:54.661544+0000","last_unstale":"2024-03-06T07:23:54.663137+0000","last_undegraded":"2024-03-06T07:23:54.663137+0000","last_fullsized":"2024-03-06T07:23:54.663137+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.639780+0000","last_change":"2024-03-06T07:23:54.639780+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.639780+0000","last_undegraded":"2024-03-06T07:23:54.639780+0000","last_fullsized":"2024-03-06T07:23:54.639780+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656144+0000","last_change":"2024-03-06T07:23:54.654853+0000","last_active":"2024-03-06T07:23:54.656144+0000","last_peered":"2024-03-06T07:23:54.656144+0000","last_clean":"2024-03-06T07:23:54.656144+0000","last_became_active":"2024-03-06T07:23:54.654566+0000","last_became_peered":"2024-03-06T07:23:54.654566+0000","last_unstale":"2024-03-06T07:23:54.656144+0000","last_undegraded":"2024-03-06T07:23:54.656144+0000","last_fullsized":"2024-03-06T07:23:54.656144+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:13.482463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.647238+0000","last_change":"2024-03-06T07:23:54.647238+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.647238+0000","last_undegraded":"2024-03-06T07:23:54.647238+0000","last_fullsized":"2024-03-06T07:23:54.647238+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.661489+0000","last_change":"2024-03-06T07:23:54.650365+0000","last_active":"2024-03-06T07:23:54.661489+0000","last_peered":"2024-03-06T07:23:54.661489+0000","last_clean":"2024-03-06T07:23:54.661489+0000","last_became_active":"2024-03-06T07:23:54.650077+0000","last_became_peered":"2024-03-06T07:23:54.650077+0000","last_unstale":"2024-03-06T07:23:54.661489+0000","last_undegraded":"2024-03-06T07:23:54.661489+0000","last_fullsized":"2024-03-06T07:23:54.661489+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:26.696172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645225+0000","last_change":"2024-03-06T07:23:54.645225+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645225+0000","last_undegraded":"2024-03-06T07:23:54.645225+0000","last_fullsized":"2024-03-06T07:23:54.645225+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654149+0000","last_change":"2024-03-06T07:23:54.652376+0000","last_active":"2024-03-06T07:23:54.654149+0000","last_peered":"2024-03-06T07:23:54.654149+0000","last_clean":"2024-03-06T07:23:54.654149+0000","last_became_active":"2024-03-06T07:23:54.652121+0000","last_became_peered":"2024-03-06T07:23:54.652121+0000","last_unstale":"2024-03-06T07:23:54.654149+0000","last_undegraded":"2024-03-06T07:23:54.654149+0000","last_fullsized":"2024-03-06T07:23:54.654149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:26.696172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643151+0000","last_change":"2024-03-06T07:23:54.643151+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643151+0000","last_undegraded":"2024-03-06T07:23:54.643151+0000","last_fullsized":"2024-03-06T07:23:54.643151+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654078+0000","last_change":"2024-03-06T07:23:54.652687+0000","last_active":"2024-03-06T07:23:54.654078+0000","last_peered":"2024-03-06T07:23:54.654078+0000","last_clean":"2024-03-06T07:23:54.654078+0000","last_became_active":"2024-03-06T07:23:54.652023+0000","last_became_peered":"2024-03-06T07:23:54.652023+0000","last_unstale":"2024-03-06T07:23:54.654078+0000","last_undegraded":"2024-03-06T07:23:54.654078+0000","last_fullsized":"2024-03-06T07:23:54.654078+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:37.112364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640802+0000","last_change":"2024-03-06T07:23:54.640802+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640802+0000","last_undegraded":"2024-03-06T07:23:54.640802+0000","last_fullsized":"2024-03-06T07:23:54.640802+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653032+0000","last_change":"2024-03-06T07:23:54.651953+0000","last_active":"2024-03-06T07:23:54.653032+0000","last_peered":"2024-03-06T07:23:54.653032+0000","last_clean":"2024-03-06T07:23:54.653032+0000","last_became_active":"2024-03-06T07:23:54.651636+0000","last_became_peered":"2024-03-06T07:23:54.651636+0000","last_unstale":"2024-03-06T07:23:54.653032+0000","last_undegraded":"2024-03-06T07:23:54.653032+0000","last_fullsized":"2024-03-06T07:23:54.653032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646552+0000","last_change":"2024-03-06T07:23:54.646552+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646552+0000","last_undegraded":"2024-03-06T07:23:54.646552+0000","last_fullsized":"2024-03-06T07:23:54.646552+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655564+0000","last_change":"2024-03-06T07:23:54.653509+0000","last_active":"2024-03-06T07:23:54.655564+0000","last_peered":"2024-03-06T07:23:54.655564+0000","last_clean":"2024-03-06T07:23:54.655564+0000","last_became_active":"2024-03-06T07:23:54.651897+0000","last_became_peered":"2024-03-06T07:23:54.651897+0000","last_unstale":"2024-03-06T07:23:54.655564+0000","last_undegraded":"2024-03-06T07:23:54.655564+0000","last_fullsized":"2024-03-06T07:23:54.655564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:29.745505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642849+0000","last_change":"2024-03-06T07:23:54.642849+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642849+0000","last_undegraded":"2024-03-06T07:23:54.642849+0000","last_fullsized":"2024-03-06T07:23:54.642849+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.656057+0000","last_change":"2024-03-06T07:23:54.653695+0000","last_active":"2024-03-06T07:23:54.656057+0000","last_peered":"2024-03-06T07:23:54.656057+0000","last_clean":"2024-03-06T07:23:54.656057+0000","last_became_active":"2024-03-06T07:23:54.653405+0000","last_became_peered":"2024-03-06T07:23:54.653405+0000","last_unstale":"2024-03-06T07:23:54.656057+0000","last_undegraded":"2024-03-06T07:23:54.656057+0000","last_fullsized":"2024-03-06T07:23:54.656057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:05.867698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642632+0000","last_change":"2024-03-06T07:23:54.642632+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642632+0000","last_undegraded":"2024-03-06T07:23:54.642632+0000","last_fullsized":"2024-03-06T07:23:54.642632+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663079+0000","last_change":"2024-03-06T07:23:54.661840+0000","last_active":"2024-03-06T07:23:54.663079+0000","last_peered":"2024-03-06T07:23:54.663079+0000","last_clean":"2024-03-06T07:23:54.663079+0000","last_became_active":"2024-03-06T07:23:54.661153+0000","last_became_peered":"2024-03-06T07:23:54.661153+0000","last_unstale":"2024-03-06T07:23:54.663079+0000","last_undegraded":"2024-03-06T07:23:54.663079+0000","last_fullsized":"2024-03-06T07:23:54.663079+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:13.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640747+0000","last_change":"2024-03-06T07:23:54.640747+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640747+0000","last_undegraded":"2024-03-06T07:23:54.640747+0000","last_fullsized":"2024-03-06T07:23:54.640747+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653444+0000","last_change":"2024-03-06T07:23:54.652357+0000","last_active":"2024-03-06T07:23:54.653444+0000","last_peered":"2024-03-06T07:23:54.653444+0000","last_clean":"2024-03-06T07:23:54.653444+0000","last_became_active":"2024-03-06T07:23:54.652092+0000","last_became_peered":"2024-03-06T07:23:54.652092+0000","last_unstale":"2024-03-06T07:23:54.653444+0000","last_undegraded":"2024-03-06T07:23:54.653444+0000","last_fullsized":"2024-03-06T07:23:54.653444+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:24.364320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.643899+0000","last_change":"2024-03-06T07:23:54.643899+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.643899+0000","last_undegraded":"2024-03-06T07:23:54.643899+0000","last_fullsized":"2024-03-06T07:23:54.643899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.657787+0000","last_change":"2024-03-06T07:23:54.656641+0000","last_active":"2024-03-06T07:23:54.657787+0000","last_peered":"2024-03-06T07:23:54.657787+0000","last_clean":"2024-03-06T07:23:54.657787+0000","last_became_active":"2024-03-06T07:23:54.656285+0000","last_became_peered":"2024-03-06T07:23:54.656285+0000","last_unstale":"2024-03-06T07:23:54.657787+0000","last_undegraded":"2024-03-06T07:23:54.657787+0000","last_fullsized":"2024-03-06T07:23:54.657787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:08.762797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.648064+0000","last_change":"2024-03-06T07:23:54.648064+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.648064+0000","last_undegraded":"2024-03-06T07:23:54.648064+0000","last_fullsized":"2024-03-06T07:23:54.648064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654119+0000","last_change":"2024-03-06T07:23:54.651813+0000","last_active":"2024-03-06T07:23:54.654119+0000","last_peered":"2024-03-06T07:23:54.654119+0000","last_clean":"2024-03-06T07:23:54.654119+0000","last_became_active":"2024-03-06T07:23:54.651504+0000","last_became_peered":"2024-03-06T07:23:54.651504+0000","last_unstale":"2024-03-06T07:23:54.654119+0000","last_undegraded":"2024-03-06T07:23:54.654119+0000","last_fullsized":"2024-03-06T07:23:54.654119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:07.864613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642671+0000","last_change":"2024-03-06T07:23:54.642671+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642671+0000","last_undegraded":"2024-03-06T07:23:54.642671+0000","last_fullsized":"2024-03-06T07:23:54.642671+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655764+0000","last_change":"2024-03-06T07:23:54.652295+0000","last_active":"2024-03-06T07:23:54.655764+0000","last_peered":"2024-03-06T07:23:54.655764+0000","last_clean":"2024-03-06T07:23:54.655764+0000","last_became_active":"2024-03-06T07:23:54.652029+0000","last_became_peered":"2024-03-06T07:23:54.652029+0000","last_unstale":"2024-03-06T07:23:54.655764+0000","last_undegraded":"2024-03-06T07:23:54.655764+0000","last_fullsized":"2024-03-06T07:23:54.655764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646070+0000","last_change":"2024-03-06T07:23:54.646070+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646070+0000","last_undegraded":"2024-03-06T07:23:54.646070+0000","last_fullsized":"2024-03-06T07:23:54.646070+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653007+0000","last_change":"2024-03-06T07:23:54.651884+0000","last_active":"2024-03-06T07:23:54.653007+0000","last_peered":"2024-03-06T07:23:54.653007+0000","last_clean":"2024-03-06T07:23:54.653007+0000","last_became_active":"2024-03-06T07:23:54.651584+0000","last_became_peered":"2024-03-06T07:23:54.651584+0000","last_unstale":"2024-03-06T07:23:54.653007+0000","last_undegraded":"2024-03-06T07:23:54.653007+0000","last_fullsized":"2024-03-06T07:23:54.653007+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646049+0000","last_change":"2024-03-06T07:23:54.646049+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646049+0000","last_undegraded":"2024-03-06T07:23:54.646049+0000","last_fullsized":"2024-03-06T07:23:54.646049+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658202+0000","last_change":"2024-03-06T07:23:54.657051+0000","last_active":"2024-03-06T07:23:54.658202+0000","last_peered":"2024-03-06T07:23:54.658202+0000","last_clean":"2024-03-06T07:23:54.658202+0000","last_became_active":"2024-03-06T07:23:54.656681+0000","last_became_peered":"2024-03-06T07:23:54.656681+0000","last_unstale":"2024-03-06T07:23:54.658202+0000","last_undegraded":"2024-03-06T07:23:54.658202+0000","last_fullsized":"2024-03-06T07:23:54.658202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:43.882962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640452+0000","last_change":"2024-03-06T07:23:54.640452+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640452+0000","last_undegraded":"2024-03-06T07:23:54.640452+0000","last_fullsized":"2024-03-06T07:23:54.640452+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658778+0000","last_change":"2024-03-06T07:23:54.657746+0000","last_active":"2024-03-06T07:23:54.658778+0000","last_peered":"2024-03-06T07:23:54.658778+0000","last_clean":"2024-03-06T07:23:54.658778+0000","last_became_active":"2024-03-06T07:23:54.657422+0000","last_became_peered":"2024-03-06T07:23:54.657422+0000","last_unstale":"2024-03-06T07:23:54.658778+0000","last_undegraded":"2024-03-06T07:23:54.658778+0000","last_fullsized":"2024-03-06T07:23:54.658778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645677+0000","last_change":"2024-03-06T07:23:54.645677+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645677+0000","last_undegraded":"2024-03-06T07:23:54.645677+0000","last_fullsized":"2024-03-06T07:23:54.645677+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.654312+0000","last_change":"2024-03-06T07:23:54.653106+0000","last_active":"2024-03-06T07:23:54.654312+0000","last_peered":"2024-03-06T07:23:54.654312+0000","last_clean":"2024-03-06T07:23:54.654312+0000","last_became_active":"2024-03-06T07:23:54.651460+0000","last_became_peered":"2024-03-06T07:23:54.651460+0000","last_unstale":"2024-03-06T07:23:54.654312+0000","last_undegraded":"2024-03-06T07:23:54.654312+0000","last_fullsized":"2024-03-06T07:23:54.654312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.793262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642632+0000","last_change":"2024-03-06T07:23:54.642632+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642632+0000","last_undegraded":"2024-03-06T07:23:54.642632+0000","last_fullsized":"2024-03-06T07:23:54.642632+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.662455+0000","last_change":"2024-03-06T07:23:54.661954+0000","last_active":"2024-03-06T07:23:54.662455+0000","last_peered":"2024-03-06T07:23:54.662455+0000","last_clean":"2024-03-06T07:23:54.662455+0000","last_became_active":"2024-03-06T07:23:54.661708+0000","last_became_peered":"2024-03-06T07:23:54.661708+0000","last_unstale":"2024-03-06T07:23:54.662455+0000","last_undegraded":"2024-03-06T07:23:54.662455+0000","last_fullsized":"2024-03-06T07:23:54.662455+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:56.838295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642883+0000","last_change":"2024-03-06T07:23:54.642883+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642883+0000","last_undegraded":"2024-03-06T07:23:54.642883+0000","last_fullsized":"2024-03-06T07:23:54.642883+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655597+0000","last_change":"2024-03-06T07:23:54.653524+0000","last_active":"2024-03-06T07:23:54.655597+0000","last_peered":"2024-03-06T07:23:54.655597+0000","last_clean":"2024-03-06T07:23:54.655597+0000","last_became_active":"2024-03-06T07:23:54.652453+0000","last_became_peered":"2024-03-06T07:23:54.652453+0000","last_unstale":"2024-03-06T07:23:54.655597+0000","last_undegraded":"2024-03-06T07:23:54.655597+0000","last_fullsized":"2024-03-06T07:23:54.655597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:29.530866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642264+0000","last_change":"2024-03-06T07:23:54.642264+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642264+0000","last_undegraded":"2024-03-06T07:23:54.642264+0000","last_fullsized":"2024-03-06T07:23:54.642264+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663035+0000","last_change":"2024-03-06T07:23:54.661476+0000","last_active":"2024-03-06T07:23:54.663035+0000","last_peered":"2024-03-06T07:23:54.663035+0000","last_clean":"2024-03-06T07:23:54.663035+0000","last_became_active":"2024-03-06T07:23:54.661081+0000","last_became_peered":"2024-03-06T07:23:54.661081+0000","last_unstale":"2024-03-06T07:23:54.663035+0000","last_undegraded":"2024-03-06T07:23:54.663035+0000","last_fullsized":"2024-03-06T07:23:54.663035+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:55.914789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.642349+0000","last_change":"2024-03-06T07:23:54.642349+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.642349+0000","last_undegraded":"2024-03-06T07:23:54.642349+0000","last_fullsized":"2024-03-06T07:23:54.642349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.658683+0000","last_change":"2024-03-06T07:23:54.656556+0000","last_active":"2024-03-06T07:23:54.658683+0000","last_peered":"2024-03-06T07:23:54.658683+0000","last_clean":"2024-03-06T07:23:54.658683+0000","last_became_active":"2024-03-06T07:23:54.656238+0000","last_became_peered":"2024-03-06T07:23:54.656238+0000","last_unstale":"2024-03-06T07:23:54.658683+0000","last_undegraded":"2024-03-06T07:23:54.658683+0000","last_fullsized":"2024-03-06T07:23:54.658683+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641604+0000","last_change":"2024-03-06T07:23:54.641604+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641604+0000","last_undegraded":"2024-03-06T07:23:54.641604+0000","last_fullsized":"2024-03-06T07:23:54.641604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.650870+0000","last_change":"2024-03-06T07:23:54.648660+0000","last_active":"2024-03-06T07:23:54.650870+0000","last_peered":"2024-03-06T07:23:54.650870+0000","last_clean":"2024-03-06T07:23:54.650870+0000","last_became_active":"2024-03-06T07:23:54.646838+0000","last_became_peered":"2024-03-06T07:23:54.646838+0000","last_unstale":"2024-03-06T07:23:54.650870+0000","last_undegraded":"2024-03-06T07:23:54.650870+0000","last_fullsized":"2024-03-06T07:23:54.650870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.731853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645622+0000","last_change":"2024-03-06T07:23:54.645622+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645622+0000","last_undegraded":"2024-03-06T07:23:54.645622+0000","last_fullsized":"2024-03-06T07:23:54.645622+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655052+0000","last_change":"2024-03-06T07:23:54.654021+0000","last_active":"2024-03-06T07:23:54.655052+0000","last_peered":"2024-03-06T07:23:54.655052+0000","last_clean":"2024-03-06T07:23:54.655052+0000","last_became_active":"2024-03-06T07:23:54.653706+0000","last_became_peered":"2024-03-06T07:23:54.653706+0000","last_unstale":"2024-03-06T07:23:54.655052+0000","last_undegraded":"2024-03-06T07:23:54.655052+0000","last_fullsized":"2024-03-06T07:23:54.655052+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:28.496649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640781+0000","last_change":"2024-03-06T07:23:54.640781+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640781+0000","last_undegraded":"2024-03-06T07:23:54.640781+0000","last_fullsized":"2024-03-06T07:23:54.640781+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655844+0000","last_change":"2024-03-06T07:23:54.653439+0000","last_active":"2024-03-06T07:23:54.655844+0000","last_peered":"2024-03-06T07:23:54.655844+0000","last_clean":"2024-03-06T07:23:54.655844+0000","last_became_active":"2024-03-06T07:23:54.653015+0000","last_became_peered":"2024-03-06T07:23:54.653015+0000","last_unstale":"2024-03-06T07:23:54.655844+0000","last_undegraded":"2024-03-06T07:23:54.655844+0000","last_fullsized":"2024-03-06T07:23:54.655844+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.500351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.645281+0000","last_change":"2024-03-06T07:23:54.645281+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.645281+0000","last_undegraded":"2024-03-06T07:23:54.645281+0000","last_fullsized":"2024-03-06T07:23:54.645281+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.663119+0000","last_change":"2024-03-06T07:23:54.661828+0000","last_active":"2024-03-06T07:23:54.663119+0000","last_peered":"2024-03-06T07:23:54.663119+0000","last_clean":"2024-03-06T07:23:54.663119+0000","last_became_active":"2024-03-06T07:23:54.661129+0000","last_became_peered":"2024-03-06T07:23:54.661129+0000","last_unstale":"2024-03-06T07:23:54.663119+0000","last_undegraded":"2024-03-06T07:23:54.663119+0000","last_fullsized":"2024-03-06T07:23:54.663119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:19.451847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646653+0000","last_change":"2024-03-06T07:23:54.646653+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646653+0000","last_undegraded":"2024-03-06T07:23:54.646653+0000","last_fullsized":"2024-03-06T07:23:54.646653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.664675+0000","last_change":"2024-03-06T07:23:54.663162+0000","last_active":"2024-03-06T07:23:54.664675+0000","last_peered":"2024-03-06T07:23:54.664675+0000","last_clean":"2024-03-06T07:23:54.664675+0000","last_became_active":"2024-03-06T07:23:54.662778+0000","last_became_peered":"2024-03-06T07:23:54.662778+0000","last_unstale":"2024-03-06T07:23:54.664675+0000","last_undegraded":"2024-03-06T07:23:54.664675+0000","last_fullsized":"2024-03-06T07:23:54.664675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644876+0000","last_change":"2024-03-06T07:23:54.644876+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644876+0000","last_undegraded":"2024-03-06T07:23:54.644876+0000","last_fullsized":"2024-03-06T07:23:54.644876+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.662222+0000","last_change":"2024-03-06T07:23:54.651253+0000","last_active":"2024-03-06T07:23:54.662222+0000","last_peered":"2024-03-06T07:23:54.662222+0000","last_clean":"2024-03-06T07:23:54.662222+0000","last_became_active":"2024-03-06T07:23:54.650963+0000","last_became_peered":"2024-03-06T07:23:54.650963+0000","last_unstale":"2024-03-06T07:23:54.662222+0000","last_undegraded":"2024-03-06T07:23:54.662222+0000","last_fullsized":"2024-03-06T07:23:54.662222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:56.838295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.646853+0000","last_change":"2024-03-06T07:23:54.646853+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.646853+0000","last_undegraded":"2024-03-06T07:23:54.646853+0000","last_fullsized":"2024-03-06T07:23:54.646853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655850+0000","last_change":"2024-03-06T07:23:54.654189+0000","last_active":"2024-03-06T07:23:54.655850+0000","last_peered":"2024-03-06T07:23:54.655850+0000","last_clean":"2024-03-06T07:23:54.655850+0000","last_became_active":"2024-03-06T07:23:54.653889+0000","last_became_peered":"2024-03-06T07:23:54.653889+0000","last_unstale":"2024-03-06T07:23:54.655850+0000","last_undegraded":"2024-03-06T07:23:54.655850+0000","last_fullsized":"2024-03-06T07:23:54.655850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.640921+0000","last_change":"2024-03-06T07:23:54.640921+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.640921+0000","last_undegraded":"2024-03-06T07:23:54.640921+0000","last_fullsized":"2024-03-06T07:23:54.640921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653305+0000","last_change":"2024-03-06T07:23:54.649432+0000","last_active":"2024-03-06T07:23:54.653305+0000","last_peered":"2024-03-06T07:23:54.653305+0000","last_clean":"2024-03-06T07:23:54.653305+0000","last_became_active":"2024-03-06T07:23:54.649162+0000","last_became_peered":"2024-03-06T07:23:54.649162+0000","last_unstale":"2024-03-06T07:23:54.653305+0000","last_undegraded":"2024-03-06T07:23:54.653305+0000","last_fullsized":"2024-03-06T07:23:54.653305+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:38.700382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.641700+0000","last_change":"2024-03-06T07:23:54.641700+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.641700+0000","last_undegraded":"2024-03-06T07:23:54.641700+0000","last_fullsized":"2024-03-06T07:23:54.641700+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.655269+0000","last_change":"2024-03-06T07:23:54.652266+0000","last_active":"2024-03-06T07:23:54.655269+0000","last_peered":"2024-03-06T07:23:54.655269+0000","last_clean":"2024-03-06T07:23:54.655269+0000","last_became_active":"2024-03-06T07:23:54.651291+0000","last_became_peered":"2024-03-06T07:23:54.651291+0000","last_unstale":"2024-03-06T07:23:54.655269+0000","last_undegraded":"2024-03-06T07:23:54.655269+0000","last_fullsized":"2024-03-06T07:23:54.655269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-03-06T07:23:54.644697+0000","last_change":"2024-03-06T07:23:54.644697+0000","last_active":"2024-03-06T07:23:54.629715+0000","last_peered":"2024-03-06T07:23:54.629715+0000","last_clean":"2024-03-06T07:23:54.629715+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:54.644697+0000","last_undegraded":"2024-03-06T07:23:54.644697+0000","last_fullsized":"2024-03-06T07:23:54.644697+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.653133+0000","last_change":"2024-03-06T07:23:54.649706+0000","last_active":"2024-03-06T07:23:54.653133+0000","last_peered":"2024-03-06T07:23:54.653133+0000","last_clean":"2024-03-06T07:23:54.653133+0000","last_became_active":"2024-03-06T07:23:54.649451+0000","last_became_peered":"2024-03-06T07:23:54.649451+0000","last_unstale":"2024-03-06T07:23:54.653133+0000","last_undegraded":"2024-03-06T07:23:54.653133+0000","last_fullsized":"2024-03-06T07:23:54.653133+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:13.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:23:54.647941+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:23:54.647941+0000","last_peered":"2024-03-06T07:23:54.647941+0000","last_clean":"2024-03-06T07:23:54.647941+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:23:54.647941+0000","last_undegraded":"2024-03-06T07:23:54.647941+0000","last_fullsized":"2024-03-06T07:23:54.647941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":727628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":372736,"data_stored":143527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":137260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":727628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:04.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:04.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.966+0000 7fab971c9640 1 Processor -- start 2024-03-06T07:24:04.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.966+0000 7fab971c9640 1 -- start start 2024-03-06T07:24:04.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab971c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab900d3400 0x7fab900b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab971c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fab900b2b50 0x7fab900b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab971c9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab900ce610 con 0x7fab90071540 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab971c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab900d2f70 con 0x7fab900d3400 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab971c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab900d4be0 con 0x7fab900b2b50 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab8ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab900d3400 0x7fab900b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab8ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab900d3400 0x7fab900b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51290/0 (socket says 172.21.15.27:51290) 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab8ffff640 1 -- 172.21.15.27:0/1161784203 learned_addr learned my addr 172.21.15.27:0/1161784203 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab94f65640 1 --1- 172.21.15.27:0/1161784203 >> v1:172.21.15.27:6789/0 conn(0x7fab90071540 0x7fab90071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36376/0 (socket says 172.21.15.27:36376) 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab95766640 1 --2- 172.21.15.27:0/1161784203 >> v2:172.21.15.69:3300/0 conn(0x7fab900b2b50 0x7fab900b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab8ffff640 1 -- 172.21.15.27:0/1161784203 >> v2:172.21.15.69:3300/0 conn(0x7fab900b2b50 msgr2=0x7fab900b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab8ffff640 1 --2- 172.21.15.27:0/1161784203 >> v2:172.21.15.69:3300/0 conn(0x7fab900b2b50 0x7fab900b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab8ffff640 1 -- 172.21.15.27:0/1161784203 >> v1:172.21.15.27:6789/0 conn(0x7fab90071540 legacy=0x7fab90071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab8ffff640 1 -- 172.21.15.27:0/1161784203 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab900d2920 con 0x7fab900d3400 2024-03-06T07:24:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.967+0000 7fab8f7fe640 1 -- 172.21.15.27:0/1161784203 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103357742 0 0) 0x7fab900ce610 con 0x7fab90071540 2024-03-06T07:24:04.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.968+0000 7fab8ffff640 1 --2- 172.21.15.27:0/1161784203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab900d3400 0x7fab900b30a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fab900683b0 tx=0x7fab7c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76685c30a1593938 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.968+0000 7fab8f7fe640 1 -- 172.21.15.27:0/1161784203 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c034070 con 0x7fab900d3400 2024-03-06T07:24:04.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.968+0000 7fab8f7fe640 1 -- 172.21.15.27:0/1161784203 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7c030860 con 0x7fab900d3400 2024-03-06T07:24:04.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.968+0000 7fab8f7fe640 1 -- 172.21.15.27:0/1161784203 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c030c30 con 0x7fab900d3400 2024-03-06T07:24:04.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.969+0000 7fab971c9640 1 -- 172.21.15.27:0/1161784203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab900d3400 msgr2=0x7fab900b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.969+0000 7fab971c9640 1 --2- 172.21.15.27:0/1161784203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab900d3400 0x7fab900b30a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fab900683b0 tx=0x7fab7c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.969+0000 7fab971c9640 1 -- 172.21.15.27:0/1161784203 shutdown_connections 2024-03-06T07:24:04.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.969+0000 7fab971c9640 1 --2- 172.21.15.27:0/1161784203 >> v2:172.21.15.69:3300/0 conn(0x7fab900b2b50 0x7fab900b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.969+0000 7fab971c9640 1 --2- 172.21.15.27:0/1161784203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab900d3400 0x7fab900b30a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.969+0000 7fab971c9640 1 -- 172.21.15.27:0/1161784203 >> 172.21.15.27:0/1161784203 conn(0x7fab900d3ca0 msgr2=0x7fab900d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.969+0000 7fab971c9640 1 -- 172.21.15.27:0/1161784203 shutdown_connections 2024-03-06T07:24:04.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.969+0000 7fab971c9640 1 -- 172.21.15.27:0/1161784203 wait complete. 2024-03-06T07:24:04.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.971+0000 7fab971c9640 1 Processor -- start 2024-03-06T07:24:04.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.971+0000 7fab971c9640 1 -- start start 2024-03-06T07:24:04.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.972+0000 7fab971c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab90071540 0x7fab901530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.972+0000 7fab971c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab900b2b50 0x7fab900c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.972+0000 7fab971c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab900d3400 0x7fab900c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.972+0000 7fab971c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9015e370 con 0x7fab90071540 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.972+0000 7fab971c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab900ce8f0 con 0x7fab900d3400 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.972+0000 7fab971c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab900c9640 con 0x7fab900b2b50 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.972+0000 7fab94f65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab90071540 0x7fab901530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.972+0000 7fab94f65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab90071540 0x7fab901530c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51292/0 (socket says 172.21.15.27:51292) 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.972+0000 7fab94f65640 1 -- 172.21.15.27:0/2638112779 learned_addr learned my addr 172.21.15.27:0/2638112779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab95766640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab900d3400 0x7fab900c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab8ffff640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab900b2b50 0x7fab900c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab94f65640 1 -- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab900b2b50 msgr2=0x7fab900c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab94f65640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab900b2b50 0x7fab900c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab94f65640 1 -- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab900d3400 msgr2=0x7fab900c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab94f65640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab900d3400 0x7fab900c9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab94f65640 1 -- 172.21.15.27:0/2638112779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab900c8a30 con 0x7fab90071540 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab94f65640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab90071540 0x7fab901530c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fab8400c4d0 tx=0x7fab8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab8d7fa640 1 -- 172.21.15.27:0/2638112779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84018070 con 0x7fab90071540 2024-03-06T07:24:04.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab8d7fa640 1 -- 172.21.15.27:0/2638112779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab84007500 con 0x7fab90071540 2024-03-06T07:24:04.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab8d7fa640 1 -- 172.21.15.27:0/2638112779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84004a20 con 0x7fab90071540 2024-03-06T07:24:04.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab900c8d10 con 0x7fab90071540 2024-03-06T07:24:04.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.973+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab900c92d0 con 0x7fab90071540 2024-03-06T07:24:04.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.974+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab901f3b10 con 0x7fab90071540 2024-03-06T07:24:04.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.975+0000 7fab8d7fa640 1 -- 172.21.15.27:0/2638112779 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab8401b070 con 0x7fab90071540 2024-03-06T07:24:04.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.975+0000 7fab8d7fa640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fab7406b4d0 0x7fab7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.976+0000 7fab8d7fa640 1 -- 172.21.15.27:0/2638112779 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab84089540 con 0x7fab90071540 2024-03-06T07:24:04.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.976+0000 7fab8ffff640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fab7406b4d0 0x7fab7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.976+0000 7fab8ffff640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fab7406b4d0 0x7fab7406d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab7c036000 tx=0x7fab7c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:04.980+0000 7fab8d7fa640 1 -- 172.21.15.27:0/2638112779 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab901f3b10 con 0x7fab90071540 2024-03-06T07:24:05.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.157+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab90068700 con 0x7fab7406b4d0 2024-03-06T07:24:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.164+0000 7fab8d7fa640 1 -- 172.21.15.27:0/2638112779 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251951 (secure 0 0 0) 0x7fab90068700 con 0x7fab7406b4d0 2024-03-06T07:24:05.169 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:24:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fab7406b4d0 msgr2=0x7fab7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fab7406b4d0 0x7fab7406d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab7c036000 tx=0x7fab7c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab90071540 msgr2=0x7fab901530c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab90071540 0x7fab901530c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fab8400c4d0 tx=0x7fab8400c990 comp rx=0 tx=0).stop 2024-03-06T07:24:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 shutdown_connections 2024-03-06T07:24:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fab7406b4d0 0x7fab7406d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab900d3400 0x7fab900c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fab900b2b50 0x7fab900c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 --2- 172.21.15.27:0/2638112779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab90071540 0x7fab901530c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 >> 172.21.15.27:0/2638112779 conn(0x7fab900d3ca0 msgr2=0x7fab9015eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 shutdown_connections 2024-03-06T07:24:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.167+0000 7fab971c9640 1 -- 172.21.15.27:0/2638112779 wait complete. 2024-03-06T07:24:05.180 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-03-06T07:24:04.003719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220548,"kb_used_data":4328,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754172,"statfs":{"total":773094113280,"available":772868272128,"internally_reserved":0,"allocated":4431872,"data_stored":2438372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.365236"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.188075+0000","last_change":"2024-03-06T07:23:55.649969+0000","last_active":"2024-03-06T07:23:57.188075+0000","last_peered":"2024-03-06T07:23:57.188075+0000","last_clean":"2024-03-06T07:23:57.188075+0000","last_became_active":"2024-03-06T07:23:55.649668+0000","last_became_peered":"2024-03-06T07:23:55.649668+0000","last_unstale":"2024-03-06T07:23:57.188075+0000","last_undegraded":"2024-03-06T07:23:57.188075+0000","last_fullsized":"2024-03-06T07:23:57.188075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:29.498899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187799+0000","last_change":"2024-03-06T07:23:55.666583+0000","last_active":"2024-03-06T07:23:57.187799+0000","last_peered":"2024-03-06T07:23:57.187799+0000","last_clean":"2024-03-06T07:23:57.187799+0000","last_became_active":"2024-03-06T07:23:55.665724+0000","last_became_peered":"2024-03-06T07:23:55.665724+0000","last_unstale":"2024-03-06T07:23:57.187799+0000","last_undegraded":"2024-03-06T07:23:57.187799+0000","last_fullsized":"2024-03-06T07:23:57.187799+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:30.155676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187777+0000","last_change":"2024-03-06T07:23:55.654572+0000","last_active":"2024-03-06T07:23:57.187777+0000","last_peered":"2024-03-06T07:23:57.187777+0000","last_clean":"2024-03-06T07:23:57.187777+0000","last_became_active":"2024-03-06T07:23:55.654236+0000","last_became_peered":"2024-03-06T07:23:55.654236+0000","last_unstale":"2024-03-06T07:23:57.187777+0000","last_undegraded":"2024-03-06T07:23:57.187777+0000","last_fullsized":"2024-03-06T07:23:57.187777+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:30.155676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.186790+0000","last_change":"2024-03-06T07:23:55.660633+0000","last_active":"2024-03-06T07:23:57.186790+0000","last_peered":"2024-03-06T07:23:57.186790+0000","last_clean":"2024-03-06T07:23:57.186790+0000","last_became_active":"2024-03-06T07:23:55.660342+0000","last_became_peered":"2024-03-06T07:23:55.660342+0000","last_unstale":"2024-03-06T07:23:57.186790+0000","last_undegraded":"2024-03-06T07:23:57.186790+0000","last_fullsized":"2024-03-06T07:23:57.186790+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:39.377020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.690214+0000","last_change":"2024-03-06T07:23:55.667354+0000","last_active":"2024-03-06T07:23:56.690214+0000","last_peered":"2024-03-06T07:23:56.690214+0000","last_clean":"2024-03-06T07:23:56.690214+0000","last_became_active":"2024-03-06T07:23:55.667045+0000","last_became_peered":"2024-03-06T07:23:55.667045+0000","last_unstale":"2024-03-06T07:23:56.690214+0000","last_undegraded":"2024-03-06T07:23:56.690214+0000","last_fullsized":"2024-03-06T07:23:56.690214+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:14.437878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187516+0000","last_change":"2024-03-06T07:23:55.666237+0000","last_active":"2024-03-06T07:23:57.187516+0000","last_peered":"2024-03-06T07:23:57.187516+0000","last_clean":"2024-03-06T07:23:57.187516+0000","last_became_active":"2024-03-06T07:23:55.665926+0000","last_became_peered":"2024-03-06T07:23:55.665926+0000","last_unstale":"2024-03-06T07:23:57.187516+0000","last_undegraded":"2024-03-06T07:23:57.187516+0000","last_fullsized":"2024-03-06T07:23:57.187516+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:14.320694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642032+0000","last_change":"2024-03-06T07:23:55.657713+0000","last_active":"2024-03-06T07:23:56.642032+0000","last_peered":"2024-03-06T07:23:56.642032+0000","last_clean":"2024-03-06T07:23:56.642032+0000","last_became_active":"2024-03-06T07:23:55.657426+0000","last_became_peered":"2024-03-06T07:23:55.657426+0000","last_unstale":"2024-03-06T07:23:56.642032+0000","last_undegraded":"2024-03-06T07:23:56.642032+0000","last_fullsized":"2024-03-06T07:23:56.642032+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:10.347513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641670+0000","last_change":"2024-03-06T07:23:55.653781+0000","last_active":"2024-03-06T07:23:56.641670+0000","last_peered":"2024-03-06T07:23:56.641670+0000","last_clean":"2024-03-06T07:23:56.641670+0000","last_became_active":"2024-03-06T07:23:55.653467+0000","last_became_peered":"2024-03-06T07:23:55.653467+0000","last_unstale":"2024-03-06T07:23:56.641670+0000","last_undegraded":"2024-03-06T07:23:56.641670+0000","last_fullsized":"2024-03-06T07:23:56.641670+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:44.523151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642029+0000","last_change":"2024-03-06T07:23:55.663300+0000","last_active":"2024-03-06T07:23:56.642029+0000","last_peered":"2024-03-06T07:23:56.642029+0000","last_clean":"2024-03-06T07:23:56.642029+0000","last_became_active":"2024-03-06T07:23:55.663006+0000","last_became_peered":"2024-03-06T07:23:55.663006+0000","last_unstale":"2024-03-06T07:23:56.642029+0000","last_undegraded":"2024-03-06T07:23:56.642029+0000","last_fullsized":"2024-03-06T07:23:56.642029+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:21.909396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.186969+0000","last_change":"2024-03-06T07:23:55.666475+0000","last_active":"2024-03-06T07:23:57.186969+0000","last_peered":"2024-03-06T07:23:57.186969+0000","last_clean":"2024-03-06T07:23:57.186969+0000","last_became_active":"2024-03-06T07:23:55.666195+0000","last_became_peered":"2024-03-06T07:23:55.666195+0000","last_unstale":"2024-03-06T07:23:57.186969+0000","last_undegraded":"2024-03-06T07:23:57.186969+0000","last_fullsized":"2024-03-06T07:23:57.186969+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:35.915350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642400+0000","last_change":"2024-03-06T07:23:55.654042+0000","last_active":"2024-03-06T07:23:56.642400+0000","last_peered":"2024-03-06T07:23:56.642400+0000","last_clean":"2024-03-06T07:23:56.642400+0000","last_became_active":"2024-03-06T07:23:55.653764+0000","last_became_peered":"2024-03-06T07:23:55.653764+0000","last_unstale":"2024-03-06T07:23:56.642400+0000","last_undegraded":"2024-03-06T07:23:56.642400+0000","last_fullsized":"2024-03-06T07:23:56.642400+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:30.155676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187525+0000","last_change":"2024-03-06T07:23:55.656937+0000","last_active":"2024-03-06T07:23:57.187525+0000","last_peered":"2024-03-06T07:23:57.187525+0000","last_clean":"2024-03-06T07:23:57.187525+0000","last_became_active":"2024-03-06T07:23:55.656662+0000","last_became_peered":"2024-03-06T07:23:55.656662+0000","last_unstale":"2024-03-06T07:23:57.187525+0000","last_undegraded":"2024-03-06T07:23:57.187525+0000","last_fullsized":"2024-03-06T07:23:57.187525+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:58.765300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.690342+0000","last_change":"2024-03-06T07:23:55.655110+0000","last_active":"2024-03-06T07:23:56.690342+0000","last_peered":"2024-03-06T07:23:56.690342+0000","last_clean":"2024-03-06T07:23:56.690342+0000","last_became_active":"2024-03-06T07:23:55.654795+0000","last_became_peered":"2024-03-06T07:23:55.654795+0000","last_unstale":"2024-03-06T07:23:56.690342+0000","last_undegraded":"2024-03-06T07:23:56.690342+0000","last_fullsized":"2024-03-06T07:23:56.690342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:14.484713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187547+0000","last_change":"2024-03-06T07:23:55.662886+0000","last_active":"2024-03-06T07:23:57.187547+0000","last_peered":"2024-03-06T07:23:57.187547+0000","last_clean":"2024-03-06T07:23:57.187547+0000","last_became_active":"2024-03-06T07:23:55.662551+0000","last_became_peered":"2024-03-06T07:23:55.662551+0000","last_unstale":"2024-03-06T07:23:57.187547+0000","last_undegraded":"2024-03-06T07:23:57.187547+0000","last_fullsized":"2024-03-06T07:23:57.187547+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:18.394192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187662+0000","last_change":"2024-03-06T07:23:55.651806+0000","last_active":"2024-03-06T07:23:57.187662+0000","last_peered":"2024-03-06T07:23:57.187662+0000","last_clean":"2024-03-06T07:23:57.187662+0000","last_became_active":"2024-03-06T07:23:55.651535+0000","last_became_peered":"2024-03-06T07:23:55.651535+0000","last_unstale":"2024-03-06T07:23:57.187662+0000","last_undegraded":"2024-03-06T07:23:57.187662+0000","last_fullsized":"2024-03-06T07:23:57.187662+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:40.903028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641996+0000","last_change":"2024-03-06T07:23:55.654758+0000","last_active":"2024-03-06T07:23:56.641996+0000","last_peered":"2024-03-06T07:23:56.641996+0000","last_clean":"2024-03-06T07:23:56.641996+0000","last_became_active":"2024-03-06T07:23:55.654504+0000","last_became_peered":"2024-03-06T07:23:55.654504+0000","last_unstale":"2024-03-06T07:23:56.641996+0000","last_undegraded":"2024-03-06T07:23:56.641996+0000","last_fullsized":"2024-03-06T07:23:56.641996+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:18.394192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642804+0000","last_change":"2024-03-06T07:23:55.652341+0000","last_active":"2024-03-06T07:23:56.642804+0000","last_peered":"2024-03-06T07:23:56.642804+0000","last_clean":"2024-03-06T07:23:56.642804+0000","last_became_active":"2024-03-06T07:23:55.652043+0000","last_became_peered":"2024-03-06T07:23:55.652043+0000","last_unstale":"2024-03-06T07:23:56.642804+0000","last_undegraded":"2024-03-06T07:23:56.642804+0000","last_fullsized":"2024-03-06T07:23:56.642804+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:27.621582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187865+0000","last_change":"2024-03-06T07:23:55.668554+0000","last_active":"2024-03-06T07:23:57.187865+0000","last_peered":"2024-03-06T07:23:57.187865+0000","last_clean":"2024-03-06T07:23:57.187865+0000","last_became_active":"2024-03-06T07:23:55.666104+0000","last_became_peered":"2024-03-06T07:23:55.666104+0000","last_unstale":"2024-03-06T07:23:57.187865+0000","last_undegraded":"2024-03-06T07:23:57.187865+0000","last_fullsized":"2024-03-06T07:23:57.187865+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:46.023526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187776+0000","last_change":"2024-03-06T07:23:55.668580+0000","last_active":"2024-03-06T07:23:57.187776+0000","last_peered":"2024-03-06T07:23:57.187776+0000","last_clean":"2024-03-06T07:23:57.187776+0000","last_became_active":"2024-03-06T07:23:55.666426+0000","last_became_peered":"2024-03-06T07:23:55.666426+0000","last_unstale":"2024-03-06T07:23:57.187776+0000","last_undegraded":"2024-03-06T07:23:57.187776+0000","last_fullsized":"2024-03-06T07:23:57.187776+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:42.267058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.690459+0000","last_change":"2024-03-06T07:23:55.660466+0000","last_active":"2024-03-06T07:23:56.690459+0000","last_peered":"2024-03-06T07:23:56.690459+0000","last_clean":"2024-03-06T07:23:56.690459+0000","last_became_active":"2024-03-06T07:23:55.660180+0000","last_became_peered":"2024-03-06T07:23:55.660180+0000","last_unstale":"2024-03-06T07:23:56.690459+0000","last_undegraded":"2024-03-06T07:23:56.690459+0000","last_fullsized":"2024-03-06T07:23:56.690459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:25.366569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642723+0000","last_change":"2024-03-06T07:23:55.652504+0000","last_active":"2024-03-06T07:23:56.642723+0000","last_peered":"2024-03-06T07:23:56.642723+0000","last_clean":"2024-03-06T07:23:56.642723+0000","last_became_active":"2024-03-06T07:23:55.652218+0000","last_became_peered":"2024-03-06T07:23:55.652218+0000","last_unstale":"2024-03-06T07:23:56.642723+0000","last_undegraded":"2024-03-06T07:23:56.642723+0000","last_fullsized":"2024-03-06T07:23:56.642723+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:36.146544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642643+0000","last_change":"2024-03-06T07:23:55.669647+0000","last_active":"2024-03-06T07:23:56.642643+0000","last_peered":"2024-03-06T07:23:56.642643+0000","last_clean":"2024-03-06T07:23:56.642643+0000","last_became_active":"2024-03-06T07:23:55.669372+0000","last_became_peered":"2024-03-06T07:23:55.669372+0000","last_unstale":"2024-03-06T07:23:56.642643+0000","last_undegraded":"2024-03-06T07:23:56.642643+0000","last_fullsized":"2024-03-06T07:23:56.642643+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:39.377020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.640212+0000","last_change":"2024-03-06T07:23:55.645003+0000","last_active":"2024-03-06T07:23:56.640212+0000","last_peered":"2024-03-06T07:23:56.640212+0000","last_clean":"2024-03-06T07:23:56.640212+0000","last_became_active":"2024-03-06T07:23:55.644731+0000","last_became_peered":"2024-03-06T07:23:55.644731+0000","last_unstale":"2024-03-06T07:23:56.640212+0000","last_undegraded":"2024-03-06T07:23:56.640212+0000","last_fullsized":"2024-03-06T07:23:56.640212+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:15.126128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187255+0000","last_change":"2024-03-06T07:23:55.656287+0000","last_active":"2024-03-06T07:23:57.187255+0000","last_peered":"2024-03-06T07:23:57.187255+0000","last_clean":"2024-03-06T07:23:57.187255+0000","last_became_active":"2024-03-06T07:23:55.655942+0000","last_became_peered":"2024-03-06T07:23:55.655942+0000","last_unstale":"2024-03-06T07:23:57.187255+0000","last_undegraded":"2024-03-06T07:23:57.187255+0000","last_fullsized":"2024-03-06T07:23:57.187255+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:12.033369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187096+0000","last_change":"2024-03-06T07:23:55.653510+0000","last_active":"2024-03-06T07:23:57.187096+0000","last_peered":"2024-03-06T07:23:57.187096+0000","last_clean":"2024-03-06T07:23:57.187096+0000","last_became_active":"2024-03-06T07:23:55.653196+0000","last_became_peered":"2024-03-06T07:23:55.653196+0000","last_unstale":"2024-03-06T07:23:57.187096+0000","last_undegraded":"2024-03-06T07:23:57.187096+0000","last_fullsized":"2024-03-06T07:23:57.187096+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:22.861919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187913+0000","last_change":"2024-03-06T07:23:55.669095+0000","last_active":"2024-03-06T07:23:57.187913+0000","last_peered":"2024-03-06T07:23:57.187913+0000","last_clean":"2024-03-06T07:23:57.187913+0000","last_became_active":"2024-03-06T07:23:55.668835+0000","last_became_peered":"2024-03-06T07:23:55.668835+0000","last_unstale":"2024-03-06T07:23:57.187913+0000","last_undegraded":"2024-03-06T07:23:57.187913+0000","last_fullsized":"2024-03-06T07:23:57.187913+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:27.621582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187176+0000","last_change":"2024-03-06T07:23:55.658790+0000","last_active":"2024-03-06T07:23:57.187176+0000","last_peered":"2024-03-06T07:23:57.187176+0000","last_clean":"2024-03-06T07:23:57.187176+0000","last_became_active":"2024-03-06T07:23:55.658506+0000","last_became_peered":"2024-03-06T07:23:55.658506+0000","last_unstale":"2024-03-06T07:23:57.187176+0000","last_undegraded":"2024-03-06T07:23:57.187176+0000","last_fullsized":"2024-03-06T07:23:57.187176+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:29:41.312321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187755+0000","last_change":"2024-03-06T07:23:55.647548+0000","last_active":"2024-03-06T07:23:57.187755+0000","last_peered":"2024-03-06T07:23:57.187755+0000","last_clean":"2024-03-06T07:23:57.187755+0000","last_became_active":"2024-03-06T07:23:55.647257+0000","last_became_peered":"2024-03-06T07:23:55.647257+0000","last_unstale":"2024-03-06T07:23:57.187755+0000","last_undegraded":"2024-03-06T07:23:57.187755+0000","last_fullsized":"2024-03-06T07:23:57.187755+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:42.267058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642805+0000","last_change":"2024-03-06T07:23:55.669517+0000","last_active":"2024-03-06T07:23:56.642805+0000","last_peered":"2024-03-06T07:23:56.642805+0000","last_clean":"2024-03-06T07:23:56.642805+0000","last_became_active":"2024-03-06T07:23:55.669244+0000","last_became_peered":"2024-03-06T07:23:55.669244+0000","last_unstale":"2024-03-06T07:23:56.642805+0000","last_undegraded":"2024-03-06T07:23:56.642805+0000","last_fullsized":"2024-03-06T07:23:56.642805+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:35.915350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642822+0000","last_change":"2024-03-06T07:23:55.657025+0000","last_active":"2024-03-06T07:23:56.642822+0000","last_peered":"2024-03-06T07:23:56.642822+0000","last_clean":"2024-03-06T07:23:56.642822+0000","last_became_active":"2024-03-06T07:23:55.656739+0000","last_became_peered":"2024-03-06T07:23:55.656739+0000","last_unstale":"2024-03-06T07:23:56.642822+0000","last_undegraded":"2024-03-06T07:23:56.642822+0000","last_fullsized":"2024-03-06T07:23:56.642822+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:46.023526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.188125+0000","last_change":"2024-03-06T07:23:55.651809+0000","last_active":"2024-03-06T07:23:57.188125+0000","last_peered":"2024-03-06T07:23:57.188125+0000","last_clean":"2024-03-06T07:23:57.188125+0000","last_became_active":"2024-03-06T07:23:55.651538+0000","last_became_peered":"2024-03-06T07:23:55.651538+0000","last_unstale":"2024-03-06T07:23:57.188125+0000","last_undegraded":"2024-03-06T07:23:57.188125+0000","last_fullsized":"2024-03-06T07:23:57.188125+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:15.126128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641647+0000","last_change":"2024-03-06T07:23:55.655712+0000","last_active":"2024-03-06T07:23:56.641647+0000","last_peered":"2024-03-06T07:23:56.641647+0000","last_clean":"2024-03-06T07:23:56.641647+0000","last_became_active":"2024-03-06T07:23:55.654859+0000","last_became_peered":"2024-03-06T07:23:55.654859+0000","last_unstale":"2024-03-06T07:23:56.641647+0000","last_undegraded":"2024-03-06T07:23:56.641647+0000","last_fullsized":"2024-03-06T07:23:56.641647+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:58.765300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.640437+0000","last_change":"2024-03-06T07:23:55.645941+0000","last_active":"2024-03-06T07:23:56.640437+0000","last_peered":"2024-03-06T07:23:56.640437+0000","last_clean":"2024-03-06T07:23:56.640437+0000","last_became_active":"2024-03-06T07:23:55.645596+0000","last_became_peered":"2024-03-06T07:23:55.645596+0000","last_unstale":"2024-03-06T07:23:56.640437+0000","last_undegraded":"2024-03-06T07:23:56.640437+0000","last_fullsized":"2024-03-06T07:23:56.640437+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:10.307772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.188269+0000","last_change":"2024-03-06T07:23:54.655354+0000","last_active":"2024-03-06T07:23:57.188269+0000","last_peered":"2024-03-06T07:23:57.188269+0000","last_clean":"2024-03-06T07:23:57.188269+0000","last_became_active":"2024-03-06T07:23:54.655053+0000","last_became_peered":"2024-03-06T07:23:54.655053+0000","last_unstale":"2024-03-06T07:23:57.188269+0000","last_undegraded":"2024-03-06T07:23:57.188269+0000","last_fullsized":"2024-03-06T07:23:57.188269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.188265+0000","last_change":"2024-03-06T07:23:55.658058+0000","last_active":"2024-03-06T07:23:57.188265+0000","last_peered":"2024-03-06T07:23:57.188265+0000","last_clean":"2024-03-06T07:23:57.188265+0000","last_became_active":"2024-03-06T07:23:55.657614+0000","last_became_peered":"2024-03-06T07:23:55.657614+0000","last_unstale":"2024-03-06T07:23:57.188265+0000","last_undegraded":"2024-03-06T07:23:57.188265+0000","last_fullsized":"2024-03-06T07:23:57.188265+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:11.040063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.640433+0000","last_change":"2024-03-06T07:23:54.662097+0000","last_active":"2024-03-06T07:23:56.640433+0000","last_peered":"2024-03-06T07:23:56.640433+0000","last_clean":"2024-03-06T07:23:56.640433+0000","last_became_active":"2024-03-06T07:23:54.661544+0000","last_became_peered":"2024-03-06T07:23:54.661544+0000","last_unstale":"2024-03-06T07:23:56.640433+0000","last_undegraded":"2024-03-06T07:23:56.640433+0000","last_fullsized":"2024-03-06T07:23:56.640433+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642357+0000","last_change":"2024-03-06T07:23:55.669596+0000","last_active":"2024-03-06T07:23:56.642357+0000","last_peered":"2024-03-06T07:23:56.642357+0000","last_clean":"2024-03-06T07:23:56.642357+0000","last_became_active":"2024-03-06T07:23:55.669330+0000","last_became_peered":"2024-03-06T07:23:55.669330+0000","last_unstale":"2024-03-06T07:23:56.642357+0000","last_undegraded":"2024-03-06T07:23:56.642357+0000","last_fullsized":"2024-03-06T07:23:56.642357+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:02.846623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187878+0000","last_change":"2024-03-06T07:23:54.654853+0000","last_active":"2024-03-06T07:23:57.187878+0000","last_peered":"2024-03-06T07:23:57.187878+0000","last_clean":"2024-03-06T07:23:57.187878+0000","last_became_active":"2024-03-06T07:23:54.654566+0000","last_became_peered":"2024-03-06T07:23:54.654566+0000","last_unstale":"2024-03-06T07:23:57.187878+0000","last_undegraded":"2024-03-06T07:23:57.187878+0000","last_fullsized":"2024-03-06T07:23:57.187878+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:53.301853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187322+0000","last_change":"2024-03-06T07:23:55.655050+0000","last_active":"2024-03-06T07:23:57.187322+0000","last_peered":"2024-03-06T07:23:57.187322+0000","last_clean":"2024-03-06T07:23:57.187322+0000","last_became_active":"2024-03-06T07:23:55.654778+0000","last_became_peered":"2024-03-06T07:23:55.654778+0000","last_unstale":"2024-03-06T07:23:57.187322+0000","last_undegraded":"2024-03-06T07:23:57.187322+0000","last_fullsized":"2024-03-06T07:23:57.187322+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:12.033369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.690495+0000","last_change":"2024-03-06T07:23:54.650365+0000","last_active":"2024-03-06T07:23:56.690495+0000","last_peered":"2024-03-06T07:23:56.690495+0000","last_clean":"2024-03-06T07:23:56.690495+0000","last_became_active":"2024-03-06T07:23:54.650077+0000","last_became_peered":"2024-03-06T07:23:54.650077+0000","last_unstale":"2024-03-06T07:23:56.690495+0000","last_undegraded":"2024-03-06T07:23:56.690495+0000","last_fullsized":"2024-03-06T07:23:56.690495+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:37.112364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642902+0000","last_change":"2024-03-06T07:23:55.654254+0000","last_active":"2024-03-06T07:23:56.642902+0000","last_peered":"2024-03-06T07:23:56.642902+0000","last_clean":"2024-03-06T07:23:56.642902+0000","last_became_active":"2024-03-06T07:23:55.650951+0000","last_became_peered":"2024-03-06T07:23:55.650951+0000","last_unstale":"2024-03-06T07:23:56.642902+0000","last_undegraded":"2024-03-06T07:23:56.642902+0000","last_fullsized":"2024-03-06T07:23:56.642902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:42.395909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642824+0000","last_change":"2024-03-06T07:23:54.652376+0000","last_active":"2024-03-06T07:23:56.642824+0000","last_peered":"2024-03-06T07:23:56.642824+0000","last_clean":"2024-03-06T07:23:56.642824+0000","last_became_active":"2024-03-06T07:23:54.652121+0000","last_became_peered":"2024-03-06T07:23:54.652121+0000","last_unstale":"2024-03-06T07:23:56.642824+0000","last_undegraded":"2024-03-06T07:23:56.642824+0000","last_fullsized":"2024-03-06T07:23:56.642824+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.500351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187662+0000","last_change":"2024-03-06T07:23:55.647539+0000","last_active":"2024-03-06T07:23:57.187662+0000","last_peered":"2024-03-06T07:23:57.187662+0000","last_clean":"2024-03-06T07:23:57.187662+0000","last_became_active":"2024-03-06T07:23:55.647223+0000","last_became_peered":"2024-03-06T07:23:55.647223+0000","last_unstale":"2024-03-06T07:23:57.187662+0000","last_undegraded":"2024-03-06T07:23:57.187662+0000","last_fullsized":"2024-03-06T07:23:57.187662+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:18.394192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642620+0000","last_change":"2024-03-06T07:23:54.652687+0000","last_active":"2024-03-06T07:23:56.642620+0000","last_peered":"2024-03-06T07:23:56.642620+0000","last_clean":"2024-03-06T07:23:56.642620+0000","last_became_active":"2024-03-06T07:23:54.652023+0000","last_became_peered":"2024-03-06T07:23:54.652023+0000","last_unstale":"2024-03-06T07:23:56.642620+0000","last_undegraded":"2024-03-06T07:23:56.642620+0000","last_fullsized":"2024-03-06T07:23:56.642620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:20.209800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641711+0000","last_change":"2024-03-06T07:23:55.651373+0000","last_active":"2024-03-06T07:23:56.641711+0000","last_peered":"2024-03-06T07:23:56.641711+0000","last_clean":"2024-03-06T07:23:56.641711+0000","last_became_active":"2024-03-06T07:23:55.650514+0000","last_became_peered":"2024-03-06T07:23:55.650514+0000","last_unstale":"2024-03-06T07:23:56.641711+0000","last_undegraded":"2024-03-06T07:23:56.641711+0000","last_fullsized":"2024-03-06T07:23:56.641711+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:05.242501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187324+0000","last_change":"2024-03-06T07:23:54.651953+0000","last_active":"2024-03-06T07:23:57.187324+0000","last_peered":"2024-03-06T07:23:57.187324+0000","last_clean":"2024-03-06T07:23:57.187324+0000","last_became_active":"2024-03-06T07:23:54.651636+0000","last_became_peered":"2024-03-06T07:23:54.651636+0000","last_unstale":"2024-03-06T07:23:57.187324+0000","last_undegraded":"2024-03-06T07:23:57.187324+0000","last_fullsized":"2024-03-06T07:23:57.187324+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:10.037813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.186571+0000","last_change":"2024-03-06T07:23:55.659436+0000","last_active":"2024-03-06T07:23:57.186571+0000","last_peered":"2024-03-06T07:23:57.186571+0000","last_clean":"2024-03-06T07:23:57.186571+0000","last_became_active":"2024-03-06T07:23:55.659187+0000","last_became_peered":"2024-03-06T07:23:55.659187+0000","last_unstale":"2024-03-06T07:23:57.186571+0000","last_undegraded":"2024-03-06T07:23:57.186571+0000","last_fullsized":"2024-03-06T07:23:57.186571+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:21.909396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642124+0000","last_change":"2024-03-06T07:23:54.653509+0000","last_active":"2024-03-06T07:23:56.642124+0000","last_peered":"2024-03-06T07:23:56.642124+0000","last_clean":"2024-03-06T07:23:56.642124+0000","last_became_active":"2024-03-06T07:23:54.651897+0000","last_became_peered":"2024-03-06T07:23:54.651897+0000","last_unstale":"2024-03-06T07:23:56.642124+0000","last_undegraded":"2024-03-06T07:23:56.642124+0000","last_fullsized":"2024-03-06T07:23:56.642124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:13.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187900+0000","last_change":"2024-03-06T07:23:55.667221+0000","last_active":"2024-03-06T07:23:57.187900+0000","last_peered":"2024-03-06T07:23:57.187900+0000","last_clean":"2024-03-06T07:23:57.187900+0000","last_became_active":"2024-03-06T07:23:55.666844+0000","last_became_peered":"2024-03-06T07:23:55.666844+0000","last_unstale":"2024-03-06T07:23:57.187900+0000","last_undegraded":"2024-03-06T07:23:57.187900+0000","last_fullsized":"2024-03-06T07:23:57.187900+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:46.023526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187886+0000","last_change":"2024-03-06T07:23:54.653695+0000","last_active":"2024-03-06T07:23:57.187886+0000","last_peered":"2024-03-06T07:23:57.187886+0000","last_clean":"2024-03-06T07:23:57.187886+0000","last_became_active":"2024-03-06T07:23:54.653405+0000","last_became_peered":"2024-03-06T07:23:54.653405+0000","last_unstale":"2024-03-06T07:23:57.187886+0000","last_undegraded":"2024-03-06T07:23:57.187886+0000","last_fullsized":"2024-03-06T07:23:57.187886+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:04.240251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187937+0000","last_change":"2024-03-06T07:23:55.668954+0000","last_active":"2024-03-06T07:23:57.187937+0000","last_peered":"2024-03-06T07:23:57.187937+0000","last_clean":"2024-03-06T07:23:57.187937+0000","last_became_active":"2024-03-06T07:23:55.666179+0000","last_became_peered":"2024-03-06T07:23:55.666179+0000","last_unstale":"2024-03-06T07:23:57.187937+0000","last_undegraded":"2024-03-06T07:23:57.187937+0000","last_fullsized":"2024-03-06T07:23:57.187937+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:42.395909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.640549+0000","last_change":"2024-03-06T07:23:54.661840+0000","last_active":"2024-03-06T07:23:56.640549+0000","last_peered":"2024-03-06T07:23:56.640549+0000","last_clean":"2024-03-06T07:23:56.640549+0000","last_became_active":"2024-03-06T07:23:54.661153+0000","last_became_peered":"2024-03-06T07:23:54.661153+0000","last_unstale":"2024-03-06T07:23:56.640549+0000","last_undegraded":"2024-03-06T07:23:56.640549+0000","last_fullsized":"2024-03-06T07:23:56.640549+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:05.867698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187765+0000","last_change":"2024-03-06T07:23:55.655108+0000","last_active":"2024-03-06T07:23:57.187765+0000","last_peered":"2024-03-06T07:23:57.187765+0000","last_clean":"2024-03-06T07:23:57.187765+0000","last_became_active":"2024-03-06T07:23:55.654852+0000","last_became_peered":"2024-03-06T07:23:55.654852+0000","last_unstale":"2024-03-06T07:23:57.187765+0000","last_undegraded":"2024-03-06T07:23:57.187765+0000","last_fullsized":"2024-03-06T07:23:57.187765+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:40.903028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187872+0000","last_change":"2024-03-06T07:23:54.652357+0000","last_active":"2024-03-06T07:23:57.187872+0000","last_peered":"2024-03-06T07:23:57.187872+0000","last_clean":"2024-03-06T07:23:57.187872+0000","last_became_active":"2024-03-06T07:23:54.652092+0000","last_became_peered":"2024-03-06T07:23:54.652092+0000","last_unstale":"2024-03-06T07:23:57.187872+0000","last_undegraded":"2024-03-06T07:23:57.187872+0000","last_fullsized":"2024-03-06T07:23:57.187872+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:10.037813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187513+0000","last_change":"2024-03-06T07:23:55.655344+0000","last_active":"2024-03-06T07:23:57.187513+0000","last_peered":"2024-03-06T07:23:57.187513+0000","last_clean":"2024-03-06T07:23:57.187513+0000","last_became_active":"2024-03-06T07:23:55.655096+0000","last_became_peered":"2024-03-06T07:23:55.655096+0000","last_unstale":"2024-03-06T07:23:57.187513+0000","last_undegraded":"2024-03-06T07:23:57.187513+0000","last_fullsized":"2024-03-06T07:23:57.187513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:44.523151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":20,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187753+0000","last_change":"2024-03-06T07:23:54.656641+0000","last_active":"2024-03-06T07:23:57.187753+0000","last_peered":"2024-03-06T07:23:57.187753+0000","last_clean":"2024-03-06T07:23:57.187753+0000","last_became_active":"2024-03-06T07:23:54.656285+0000","last_became_peered":"2024-03-06T07:23:54.656285+0000","last_unstale":"2024-03-06T07:23:57.187753+0000","last_undegraded":"2024-03-06T07:23:57.187753+0000","last_fullsized":"2024-03-06T07:23:57.187753+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:19.886291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.186615+0000","last_change":"2024-03-06T07:23:55.659905+0000","last_active":"2024-03-06T07:23:57.186615+0000","last_peered":"2024-03-06T07:23:57.186615+0000","last_clean":"2024-03-06T07:23:57.186615+0000","last_became_active":"2024-03-06T07:23:55.659654+0000","last_became_peered":"2024-03-06T07:23:55.659654+0000","last_unstale":"2024-03-06T07:23:57.186615+0000","last_undegraded":"2024-03-06T07:23:57.186615+0000","last_fullsized":"2024-03-06T07:23:57.186615+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:02.846623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642895+0000","last_change":"2024-03-06T07:23:54.651813+0000","last_active":"2024-03-06T07:23:56.642895+0000","last_peered":"2024-03-06T07:23:56.642895+0000","last_clean":"2024-03-06T07:23:56.642895+0000","last_became_active":"2024-03-06T07:23:54.651504+0000","last_became_peered":"2024-03-06T07:23:54.651504+0000","last_unstale":"2024-03-06T07:23:56.642895+0000","last_undegraded":"2024-03-06T07:23:56.642895+0000","last_fullsized":"2024-03-06T07:23:56.642895+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187630+0000","last_change":"2024-03-06T07:23:55.663893+0000","last_active":"2024-03-06T07:23:57.187630+0000","last_peered":"2024-03-06T07:23:57.187630+0000","last_clean":"2024-03-06T07:23:57.187630+0000","last_became_active":"2024-03-06T07:23:55.663429+0000","last_became_peered":"2024-03-06T07:23:55.663429+0000","last_unstale":"2024-03-06T07:23:57.187630+0000","last_undegraded":"2024-03-06T07:23:57.187630+0000","last_fullsized":"2024-03-06T07:23:57.187630+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:10.347513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.186699+0000","last_change":"2024-03-06T07:23:54.652295+0000","last_active":"2024-03-06T07:23:57.186699+0000","last_peered":"2024-03-06T07:23:57.186699+0000","last_clean":"2024-03-06T07:23:57.186699+0000","last_became_active":"2024-03-06T07:23:54.652029+0000","last_became_peered":"2024-03-06T07:23:54.652029+0000","last_unstale":"2024-03-06T07:23:57.186699+0000","last_undegraded":"2024-03-06T07:23:57.186699+0000","last_fullsized":"2024-03-06T07:23:57.186699+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:05.867698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642275+0000","last_change":"2024-03-06T07:23:55.651112+0000","last_active":"2024-03-06T07:23:56.642275+0000","last_peered":"2024-03-06T07:23:56.642275+0000","last_clean":"2024-03-06T07:23:56.642275+0000","last_became_active":"2024-03-06T07:23:55.650637+0000","last_became_peered":"2024-03-06T07:23:55.650637+0000","last_unstale":"2024-03-06T07:23:56.642275+0000","last_undegraded":"2024-03-06T07:23:56.642275+0000","last_fullsized":"2024-03-06T07:23:56.642275+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:42.267058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187367+0000","last_change":"2024-03-06T07:23:54.651884+0000","last_active":"2024-03-06T07:23:57.187367+0000","last_peered":"2024-03-06T07:23:57.187367+0000","last_clean":"2024-03-06T07:23:57.187367+0000","last_became_active":"2024-03-06T07:23:54.651584+0000","last_became_peered":"2024-03-06T07:23:54.651584+0000","last_unstale":"2024-03-06T07:23:57.187367+0000","last_undegraded":"2024-03-06T07:23:57.187367+0000","last_fullsized":"2024-03-06T07:23:57.187367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:09.305522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.188298+0000","last_change":"2024-03-06T07:23:55.656662+0000","last_active":"2024-03-06T07:23:57.188298+0000","last_peered":"2024-03-06T07:23:57.188298+0000","last_clean":"2024-03-06T07:23:57.188298+0000","last_became_active":"2024-03-06T07:23:55.656333+0000","last_became_peered":"2024-03-06T07:23:55.656333+0000","last_unstale":"2024-03-06T07:23:57.188298+0000","last_undegraded":"2024-03-06T07:23:57.188298+0000","last_fullsized":"2024-03-06T07:23:57.188298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:10.307772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187238+0000","last_change":"2024-03-06T07:23:54.657051+0000","last_active":"2024-03-06T07:23:57.187238+0000","last_peered":"2024-03-06T07:23:57.187238+0000","last_clean":"2024-03-06T07:23:57.187238+0000","last_became_active":"2024-03-06T07:23:54.656681+0000","last_became_peered":"2024-03-06T07:23:54.656681+0000","last_unstale":"2024-03-06T07:23:57.187238+0000","last_undegraded":"2024-03-06T07:23:57.187238+0000","last_fullsized":"2024-03-06T07:23:57.187238+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:28.496649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641877+0000","last_change":"2024-03-06T07:23:55.659439+0000","last_active":"2024-03-06T07:23:56.641877+0000","last_peered":"2024-03-06T07:23:56.641877+0000","last_clean":"2024-03-06T07:23:56.641877+0000","last_became_active":"2024-03-06T07:23:55.659137+0000","last_became_peered":"2024-03-06T07:23:55.659137+0000","last_unstale":"2024-03-06T07:23:56.641877+0000","last_undegraded":"2024-03-06T07:23:56.641877+0000","last_fullsized":"2024-03-06T07:23:56.641877+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:54.304103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187102+0000","last_change":"2024-03-06T07:23:54.657746+0000","last_active":"2024-03-06T07:23:57.187102+0000","last_peered":"2024-03-06T07:23:57.187102+0000","last_clean":"2024-03-06T07:23:57.187102+0000","last_became_active":"2024-03-06T07:23:54.657422+0000","last_became_peered":"2024-03-06T07:23:54.657422+0000","last_unstale":"2024-03-06T07:23:57.187102+0000","last_undegraded":"2024-03-06T07:23:57.187102+0000","last_fullsized":"2024-03-06T07:23:57.187102+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:43.882962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187660+0000","last_change":"2024-03-06T07:23:55.654426+0000","last_active":"2024-03-06T07:23:57.187660+0000","last_peered":"2024-03-06T07:23:57.187660+0000","last_clean":"2024-03-06T07:23:57.187660+0000","last_became_active":"2024-03-06T07:23:55.653955+0000","last_became_peered":"2024-03-06T07:23:55.653955+0000","last_unstale":"2024-03-06T07:23:57.187660+0000","last_undegraded":"2024-03-06T07:23:57.187660+0000","last_fullsized":"2024-03-06T07:23:57.187660+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:14.320694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641990+0000","last_change":"2024-03-06T07:23:54.653106+0000","last_active":"2024-03-06T07:23:56.641990+0000","last_peered":"2024-03-06T07:23:56.641990+0000","last_clean":"2024-03-06T07:23:56.641990+0000","last_became_active":"2024-03-06T07:23:54.651460+0000","last_became_peered":"2024-03-06T07:23:54.651460+0000","last_unstale":"2024-03-06T07:23:56.641990+0000","last_undegraded":"2024-03-06T07:23:56.641990+0000","last_fullsized":"2024-03-06T07:23:56.641990+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:13.482463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.186812+0000","last_change":"2024-03-06T07:23:55.655733+0000","last_active":"2024-03-06T07:23:57.186812+0000","last_peered":"2024-03-06T07:23:57.186812+0000","last_clean":"2024-03-06T07:23:57.186812+0000","last_became_active":"2024-03-06T07:23:55.655462+0000","last_became_peered":"2024-03-06T07:23:55.655462+0000","last_unstale":"2024-03-06T07:23:57.186812+0000","last_undegraded":"2024-03-06T07:23:57.186812+0000","last_fullsized":"2024-03-06T07:23:57.186812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:36.146544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642504+0000","last_change":"2024-03-06T07:23:54.661954+0000","last_active":"2024-03-06T07:23:56.642504+0000","last_peered":"2024-03-06T07:23:56.642504+0000","last_clean":"2024-03-06T07:23:56.642504+0000","last_became_active":"2024-03-06T07:23:54.661708+0000","last_became_peered":"2024-03-06T07:23:54.661708+0000","last_unstale":"2024-03-06T07:23:56.642504+0000","last_undegraded":"2024-03-06T07:23:56.642504+0000","last_fullsized":"2024-03-06T07:23:56.642504+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:08.762797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.690294+0000","last_change":"2024-03-06T07:23:55.651727+0000","last_active":"2024-03-06T07:23:56.690294+0000","last_peered":"2024-03-06T07:23:56.690294+0000","last_clean":"2024-03-06T07:23:56.690294+0000","last_became_active":"2024-03-06T07:23:55.651446+0000","last_became_peered":"2024-03-06T07:23:55.651446+0000","last_unstale":"2024-03-06T07:23:56.690294+0000","last_undegraded":"2024-03-06T07:23:56.690294+0000","last_fullsized":"2024-03-06T07:23:56.690294+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:39.984259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642464+0000","last_change":"2024-03-06T07:23:54.653524+0000","last_active":"2024-03-06T07:23:56.642464+0000","last_peered":"2024-03-06T07:23:56.642464+0000","last_clean":"2024-03-06T07:23:56.642464+0000","last_became_active":"2024-03-06T07:23:54.652453+0000","last_became_peered":"2024-03-06T07:23:54.652453+0000","last_unstale":"2024-03-06T07:23:56.642464+0000","last_undegraded":"2024-03-06T07:23:56.642464+0000","last_fullsized":"2024-03-06T07:23:56.642464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:05.867698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.690419+0000","last_change":"2024-03-06T07:23:55.659203+0000","last_active":"2024-03-06T07:23:56.690419+0000","last_peered":"2024-03-06T07:23:56.690419+0000","last_clean":"2024-03-06T07:23:56.690419+0000","last_became_active":"2024-03-06T07:23:55.658943+0000","last_became_peered":"2024-03-06T07:23:55.658943+0000","last_unstale":"2024-03-06T07:23:56.690419+0000","last_undegraded":"2024-03-06T07:23:56.690419+0000","last_fullsized":"2024-03-06T07:23:56.690419+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:44.885211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.640340+0000","last_change":"2024-03-06T07:23:54.661476+0000","last_active":"2024-03-06T07:23:56.640340+0000","last_peered":"2024-03-06T07:23:56.640340+0000","last_clean":"2024-03-06T07:23:56.640340+0000","last_became_active":"2024-03-06T07:23:54.661081+0000","last_became_peered":"2024-03-06T07:23:54.661081+0000","last_unstale":"2024-03-06T07:23:56.640340+0000","last_undegraded":"2024-03-06T07:23:56.640340+0000","last_fullsized":"2024-03-06T07:23:56.640340+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:58.747963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.690383+0000","last_change":"2024-03-06T07:23:55.649781+0000","last_active":"2024-03-06T07:23:56.690383+0000","last_peered":"2024-03-06T07:23:56.690383+0000","last_clean":"2024-03-06T07:23:56.690383+0000","last_became_active":"2024-03-06T07:23:55.649496+0000","last_became_peered":"2024-03-06T07:23:55.649496+0000","last_unstale":"2024-03-06T07:23:56.690383+0000","last_undegraded":"2024-03-06T07:23:56.690383+0000","last_fullsized":"2024-03-06T07:23:56.690383+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:09.765047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.188086+0000","last_change":"2024-03-06T07:23:54.656556+0000","last_active":"2024-03-06T07:23:57.188086+0000","last_peered":"2024-03-06T07:23:57.188086+0000","last_clean":"2024-03-06T07:23:57.188086+0000","last_became_active":"2024-03-06T07:23:54.656238+0000","last_became_peered":"2024-03-06T07:23:54.656238+0000","last_unstale":"2024-03-06T07:23:57.188086+0000","last_undegraded":"2024-03-06T07:23:57.188086+0000","last_fullsized":"2024-03-06T07:23:57.188086+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:13.435628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.690247+0000","last_change":"2024-03-06T07:23:55.649533+0000","last_active":"2024-03-06T07:23:56.690247+0000","last_peered":"2024-03-06T07:23:56.690247+0000","last_clean":"2024-03-06T07:23:56.690247+0000","last_became_active":"2024-03-06T07:23:55.649236+0000","last_became_peered":"2024-03-06T07:23:55.649236+0000","last_unstale":"2024-03-06T07:23:56.690247+0000","last_undegraded":"2024-03-06T07:23:56.690247+0000","last_fullsized":"2024-03-06T07:23:56.690247+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:59.750212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641868+0000","last_change":"2024-03-06T07:23:54.648660+0000","last_active":"2024-03-06T07:23:56.641868+0000","last_peered":"2024-03-06T07:23:56.641868+0000","last_clean":"2024-03-06T07:23:56.641868+0000","last_became_active":"2024-03-06T07:23:54.646838+0000","last_became_peered":"2024-03-06T07:23:54.646838+0000","last_unstale":"2024-03-06T07:23:56.641868+0000","last_undegraded":"2024-03-06T07:23:56.641868+0000","last_fullsized":"2024-03-06T07:23:56.641868+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:55.914789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641870+0000","last_change":"2024-03-06T07:23:55.651541+0000","last_active":"2024-03-06T07:23:56.641870+0000","last_peered":"2024-03-06T07:23:56.641870+0000","last_clean":"2024-03-06T07:23:56.641870+0000","last_became_active":"2024-03-06T07:23:55.650802+0000","last_became_peered":"2024-03-06T07:23:55.650802+0000","last_unstale":"2024-03-06T07:23:56.641870+0000","last_undegraded":"2024-03-06T07:23:56.641870+0000","last_fullsized":"2024-03-06T07:23:56.641870+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:14.320694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187175+0000","last_change":"2024-03-06T07:23:54.654021+0000","last_active":"2024-03-06T07:23:57.187175+0000","last_peered":"2024-03-06T07:23:57.187175+0000","last_clean":"2024-03-06T07:23:57.187175+0000","last_became_active":"2024-03-06T07:23:54.653706+0000","last_became_peered":"2024-03-06T07:23:54.653706+0000","last_unstale":"2024-03-06T07:23:57.187175+0000","last_undegraded":"2024-03-06T07:23:57.187175+0000","last_fullsized":"2024-03-06T07:23:57.187175+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:19.886291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.690291+0000","last_change":"2024-03-06T07:23:55.663565+0000","last_active":"2024-03-06T07:23:56.690291+0000","last_peered":"2024-03-06T07:23:56.690291+0000","last_clean":"2024-03-06T07:23:56.690291+0000","last_became_active":"2024-03-06T07:23:55.663298+0000","last_became_peered":"2024-03-06T07:23:55.663298+0000","last_unstale":"2024-03-06T07:23:56.690291+0000","last_undegraded":"2024-03-06T07:23:56.690291+0000","last_fullsized":"2024-03-06T07:23:56.690291+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:06.869948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187187+0000","last_change":"2024-03-06T07:23:54.653439+0000","last_active":"2024-03-06T07:23:57.187187+0000","last_peered":"2024-03-06T07:23:57.187187+0000","last_clean":"2024-03-06T07:23:57.187187+0000","last_became_active":"2024-03-06T07:23:54.653015+0000","last_became_peered":"2024-03-06T07:23:54.653015+0000","last_unstale":"2024-03-06T07:23:57.187187+0000","last_undegraded":"2024-03-06T07:23:57.187187+0000","last_fullsized":"2024-03-06T07:23:57.187187+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:24.364320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187681+0000","last_change":"2024-03-06T07:23:55.654634+0000","last_active":"2024-03-06T07:23:57.187681+0000","last_peered":"2024-03-06T07:23:57.187681+0000","last_clean":"2024-03-06T07:23:57.187681+0000","last_became_active":"2024-03-06T07:23:55.654339+0000","last_became_peered":"2024-03-06T07:23:55.654339+0000","last_unstale":"2024-03-06T07:23:57.187681+0000","last_undegraded":"2024-03-06T07:23:57.187681+0000","last_fullsized":"2024-03-06T07:23:57.187681+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:10.347513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"35'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.640600+0000","last_change":"2024-03-06T07:23:54.661828+0000","last_active":"2024-03-06T07:23:56.640600+0000","last_peered":"2024-03-06T07:23:56.640600+0000","last_clean":"2024-03-06T07:23:56.640600+0000","last_became_active":"2024-03-06T07:23:54.661129+0000","last_became_peered":"2024-03-06T07:23:54.661129+0000","last_unstale":"2024-03-06T07:23:56.640600+0000","last_undegraded":"2024-03-06T07:23:56.640600+0000","last_fullsized":"2024-03-06T07:23:56.640600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:13.482463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.640355+0000","last_change":"2024-03-06T07:23:55.652967+0000","last_active":"2024-03-06T07:23:56.640355+0000","last_peered":"2024-03-06T07:23:56.640355+0000","last_clean":"2024-03-06T07:23:56.640355+0000","last_became_active":"2024-03-06T07:23:55.652693+0000","last_became_peered":"2024-03-06T07:23:55.652693+0000","last_unstale":"2024-03-06T07:23:56.640355+0000","last_undegraded":"2024-03-06T07:23:56.640355+0000","last_fullsized":"2024-03-06T07:23:56.640355+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:20.888541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187014+0000","last_change":"2024-03-06T07:23:54.663162+0000","last_active":"2024-03-06T07:23:57.187014+0000","last_peered":"2024-03-06T07:23:57.187014+0000","last_clean":"2024-03-06T07:23:57.187014+0000","last_became_active":"2024-03-06T07:23:54.662778+0000","last_became_peered":"2024-03-06T07:23:54.662778+0000","last_unstale":"2024-03-06T07:23:57.187014+0000","last_undegraded":"2024-03-06T07:23:57.187014+0000","last_fullsized":"2024-03-06T07:23:57.187014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:08.762797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.188195+0000","last_change":"2024-03-06T07:23:55.652910+0000","last_active":"2024-03-06T07:23:57.188195+0000","last_peered":"2024-03-06T07:23:57.188195+0000","last_clean":"2024-03-06T07:23:57.188195+0000","last_became_active":"2024-03-06T07:23:55.652565+0000","last_became_peered":"2024-03-06T07:23:55.652565+0000","last_unstale":"2024-03-06T07:23:57.188195+0000","last_undegraded":"2024-03-06T07:23:57.188195+0000","last_fullsized":"2024-03-06T07:23:57.188195+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:20.888541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.640476+0000","last_change":"2024-03-06T07:23:54.651253+0000","last_active":"2024-03-06T07:23:56.640476+0000","last_peered":"2024-03-06T07:23:56.640476+0000","last_clean":"2024-03-06T07:23:56.640476+0000","last_became_active":"2024-03-06T07:23:54.650963+0000","last_became_peered":"2024-03-06T07:23:54.650963+0000","last_unstale":"2024-03-06T07:23:56.640476+0000","last_undegraded":"2024-03-06T07:23:56.640476+0000","last_fullsized":"2024-03-06T07:23:56.640476+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.982009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.640373+0000","last_change":"2024-03-06T07:23:55.650627+0000","last_active":"2024-03-06T07:23:56.640373+0000","last_peered":"2024-03-06T07:23:56.640373+0000","last_clean":"2024-03-06T07:23:56.640373+0000","last_became_active":"2024-03-06T07:23:55.650263+0000","last_became_peered":"2024-03-06T07:23:55.650263+0000","last_unstale":"2024-03-06T07:23:56.640373+0000","last_undegraded":"2024-03-06T07:23:56.640373+0000","last_fullsized":"2024-03-06T07:23:56.640373+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:11.040063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.186763+0000","last_change":"2024-03-06T07:23:54.654189+0000","last_active":"2024-03-06T07:23:57.186763+0000","last_peered":"2024-03-06T07:23:57.186763+0000","last_clean":"2024-03-06T07:23:57.186763+0000","last_became_active":"2024-03-06T07:23:54.653889+0000","last_became_peered":"2024-03-06T07:23:54.653889+0000","last_unstale":"2024-03-06T07:23:57.186763+0000","last_undegraded":"2024-03-06T07:23:57.186763+0000","last_fullsized":"2024-03-06T07:23:57.186763+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:13.482463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187248+0000","last_change":"2024-03-06T07:23:55.666407+0000","last_active":"2024-03-06T07:23:57.187248+0000","last_peered":"2024-03-06T07:23:57.187248+0000","last_clean":"2024-03-06T07:23:57.187248+0000","last_became_active":"2024-03-06T07:23:55.666070+0000","last_became_peered":"2024-03-06T07:23:55.666070+0000","last_unstale":"2024-03-06T07:23:57.187248+0000","last_undegraded":"2024-03-06T07:23:57.187248+0000","last_fullsized":"2024-03-06T07:23:57.187248+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:44.523151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641982+0000","last_change":"2024-03-06T07:23:54.649432+0000","last_active":"2024-03-06T07:23:56.641982+0000","last_peered":"2024-03-06T07:23:56.641982+0000","last_clean":"2024-03-06T07:23:56.641982+0000","last_became_active":"2024-03-06T07:23:54.649162+0000","last_became_peered":"2024-03-06T07:23:54.649162+0000","last_unstale":"2024-03-06T07:23:56.641982+0000","last_undegraded":"2024-03-06T07:23:56.641982+0000","last_fullsized":"2024-03-06T07:23:56.641982+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:19.451847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.641756+0000","last_change":"2024-03-06T07:23:55.669760+0000","last_active":"2024-03-06T07:23:56.641756+0000","last_peered":"2024-03-06T07:23:56.641756+0000","last_clean":"2024-03-06T07:23:56.641756+0000","last_became_active":"2024-03-06T07:23:55.669316+0000","last_became_peered":"2024-03-06T07:23:55.669316+0000","last_unstale":"2024-03-06T07:23:56.641756+0000","last_undegraded":"2024-03-06T07:23:56.641756+0000","last_fullsized":"2024-03-06T07:23:56.641756+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:34.742235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.187328+0000","last_change":"2024-03-06T07:23:54.652266+0000","last_active":"2024-03-06T07:23:57.187328+0000","last_peered":"2024-03-06T07:23:57.187328+0000","last_clean":"2024-03-06T07:23:57.187328+0000","last_became_active":"2024-03-06T07:23:54.651291+0000","last_became_peered":"2024-03-06T07:23:54.651291+0000","last_unstale":"2024-03-06T07:23:57.187328+0000","last_undegraded":"2024-03-06T07:23:57.187328+0000","last_fullsized":"2024-03-06T07:23:57.187328+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:14.123878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642083+0000","last_change":"2024-03-06T07:23:55.652981+0000","last_active":"2024-03-06T07:23:56.642083+0000","last_peered":"2024-03-06T07:23:56.642083+0000","last_clean":"2024-03-06T07:23:56.642083+0000","last_became_active":"2024-03-06T07:23:55.652700+0000","last_became_peered":"2024-03-06T07:23:55.652700+0000","last_unstale":"2024-03-06T07:23:56.642083+0000","last_undegraded":"2024-03-06T07:23:56.642083+0000","last_fullsized":"2024-03-06T07:23:56.642083+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:54.629715+0000","last_clean_scrub_stamp":"2024-03-06T07:23:54.629715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:40.903028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:57.188055+0000","last_change":"2024-03-06T07:23:54.649706+0000","last_active":"2024-03-06T07:23:57.188055+0000","last_peered":"2024-03-06T07:23:57.188055+0000","last_clean":"2024-03-06T07:23:57.188055+0000","last_became_active":"2024-03-06T07:23:54.649451+0000","last_became_peered":"2024-03-06T07:23:54.649451+0000","last_unstale":"2024-03-06T07:23:57.188055+0000","last_undegraded":"2024-03-06T07:23:57.188055+0000","last_fullsized":"2024-03-06T07:23:57.188055+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:53.627465+0000","last_clean_scrub_stamp":"2024-03-06T07:23:53.627465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.500351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":198,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.642110+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:23:56.642110+0000","last_peered":"2024-03-06T07:23:56.642110+0000","last_clean":"2024-03-06T07:23:56.642110+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:23:56.642110+0000","last_undegraded":"2024-03-06T07:23:56.642110+0000","last_fullsized":"2024-03-06T07:23:56.642110+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":747060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":405504,"data_stored":157218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":405504,"data_stored":157274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":405504,"data_stored":156706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":413696,"data_stored":158236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":156672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":413696,"data_stored":158236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":991232,"data_stored":746970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:05.184 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:05.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.334+0000 7f6a4b13e640 1 Processor -- start 2024-03-06T07:24:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.334+0000 7f6a4b13e640 1 -- start start 2024-03-06T07:24:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.335+0000 7f6a4b13e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a4415b130 0x7f6a4415b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.335+0000 7f6a4b13e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a440ce900 0x7f6a440ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.335+0000 7f6a4b13e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4415bc40 con 0x7f6a44144640 2024-03-06T07:24:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.335+0000 7f6a4b13e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a4406df70 con 0x7f6a440ce900 2024-03-06T07:24:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.335+0000 7f6a4b13e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4415bf20 con 0x7f6a4415b130 2024-03-06T07:24:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.336+0000 7f6a43fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a4415b130 0x7f6a4415b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.336+0000 7f6a496db640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a440ce900 0x7f6a440ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.336+0000 7f6a43fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a4415b130 0x7f6a4415b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:51678/0 (socket says 172.21.15.27:51678) 2024-03-06T07:24:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.336+0000 7f6a43fff640 1 -- 172.21.15.27:0/1793822399 learned_addr learned my addr 172.21.15.27:0/1793822399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.336+0000 7f6a43fff640 1 -- 172.21.15.27:0/1793822399 >> v2:172.21.15.69:3300/0 conn(0x7f6a440ce900 msgr2=0x7f6a440ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.336+0000 7f6a43fff640 1 --2- 172.21.15.27:0/1793822399 >> v2:172.21.15.69:3300/0 conn(0x7f6a440ce900 0x7f6a440ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.336+0000 7f6a43fff640 1 -- 172.21.15.27:0/1793822399 >> v1:172.21.15.69:6789/0 conn(0x7f6a44144640 legacy=0x7f6a4406e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.336+0000 7f6a43fff640 1 -- 172.21.15.27:0/1793822399 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4406dc00 con 0x7f6a4415b130 2024-03-06T07:24:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.336+0000 7f6a496db640 1 --2- 172.21.15.27:0/1793822399 >> v2:172.21.15.69:3300/0 conn(0x7f6a440ce900 0x7f6a440ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:05.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.337+0000 7f6a43fff640 1 --2- 172.21.15.27:0/1793822399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a4415b130 0x7f6a4415b590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009870 tx=0x7f6a3802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=943d7dd5512d03d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.337+0000 7f6a437fe640 1 -- 172.21.15.27:0/1793822399 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a38034070 con 0x7f6a4415b130 2024-03-06T07:24:05.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.337+0000 7f6a437fe640 1 -- 172.21.15.27:0/1793822399 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a38030840 con 0x7f6a4415b130 2024-03-06T07:24:05.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.337+0000 7f6a437fe640 1 -- 172.21.15.27:0/1793822399 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a38030c10 con 0x7f6a4415b130 2024-03-06T07:24:05.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.338+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1793822399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a4415b130 msgr2=0x7f6a4415b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.338+0000 7f6a4b13e640 1 --2- 172.21.15.27:0/1793822399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a4415b130 0x7f6a4415b590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009870 tx=0x7f6a3802b980 comp rx=0 tx=0).stop 2024-03-06T07:24:05.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.338+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1793822399 shutdown_connections 2024-03-06T07:24:05.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.338+0000 7f6a4b13e640 1 --2- 172.21.15.27:0/1793822399 >> v2:172.21.15.69:3300/0 conn(0x7f6a440ce900 0x7f6a440ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.338+0000 7f6a4b13e640 1 --2- 172.21.15.27:0/1793822399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a4415b130 0x7f6a4415b590 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.338+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1793822399 >> 172.21.15.27:0/1793822399 conn(0x7f6a44066d00 msgr2=0x7f6a44067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.338+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1793822399 shutdown_connections 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.338+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1793822399 wait complete. 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.339+0000 7f6a4b13e640 1 Processor -- start 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.339+0000 7f6a4b13e640 1 -- start start 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a4b13e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a440ce900 0x7f6a44132540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a4b13e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44144640 0x7f6a4413b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a4b13e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4415b130 0x7f6a4413a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a4b13e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a44070c20 con 0x7f6a4415b130 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a4b13e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a44070aa0 con 0x7f6a44144640 2024-03-06T07:24:05.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a4b13e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a440b36d0 con 0x7f6a440ce900 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a496db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4415b130 0x7f6a4413a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a496db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4415b130 0x7f6a4413a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51308/0 (socket says 172.21.15.27:51308) 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a496db640 1 -- 172.21.15.27:0/1920928972 learned_addr learned my addr 172.21.15.27:0/1920928972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a48eda640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a440ce900 0x7f6a44132540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a43fff640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44144640 0x7f6a4413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a496db640 1 -- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a440ce900 msgr2=0x7f6a44132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a496db640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a440ce900 0x7f6a44132540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a496db640 1 -- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44144640 msgr2=0x7f6a4413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a496db640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44144640 0x7f6a4413b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a496db640 1 -- 172.21.15.27:0/1920928972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4412e750 con 0x7f6a4415b130 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.340+0000 7f6a496db640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4415b130 0x7f6a4413a090 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009f00 tx=0x7f6a3c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.341+0000 7f6a417fa640 1 -- 172.21.15.27:0/1920928972 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c018030 con 0x7f6a4415b130 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.341+0000 7f6a417fa640 1 -- 172.21.15.27:0/1920928972 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3c00ce50 con 0x7f6a4415b130 2024-03-06T07:24:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.341+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4412e470 con 0x7f6a4415b130 2024-03-06T07:24:05.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.341+0000 7f6a417fa640 1 -- 172.21.15.27:0/1920928972 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c003bb0 con 0x7f6a4415b130 2024-03-06T07:24:05.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.341+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a441330d0 con 0x7f6a4415b130 2024-03-06T07:24:05.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.342+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a440733c0 con 0x7f6a4415b130 2024-03-06T07:24:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.342+0000 7f6a417fa640 1 -- 172.21.15.27:0/1920928972 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a3c005d10 con 0x7f6a4415b130 2024-03-06T07:24:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.343+0000 7f6a417fa640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3006b500 0x7f6a3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.343+0000 7f6a417fa640 1 -- 172.21.15.27:0/1920928972 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a3c08aa10 con 0x7f6a4415b130 2024-03-06T07:24:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.343+0000 7f6a48eda640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3006b500 0x7f6a3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.344+0000 7f6a48eda640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3006b500 0x7f6a3006d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a34001780 tx=0x7f6a34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.347+0000 7f6a417fa640 1 -- 172.21.15.27:0/1920928972 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a3c022070 con 0x7f6a4415b130 2024-03-06T07:24:05.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.597+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6a4404f410 con 0x7f6a4415b130 2024-03-06T07:24:05.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.599+0000 7f6a417fa640 1 -- 172.21.15.27:0/1920928972 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f6a3c010040 con 0x7f6a4415b130 2024-03-06T07:24:05.602 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:24:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:24:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-03-06T07:24:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-03-06T07:24:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:05.602 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-03-06T07:24:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3006b500 msgr2=0x7f6a3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3006b500 0x7f6a3006d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a34001780 tx=0x7f6a34009040 comp rx=0 tx=0).stop 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4415b130 msgr2=0x7f6a4413a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4415b130 0x7f6a4413a090 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009f00 tx=0x7f6a3c00c490 comp rx=0 tx=0).stop 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 shutdown_connections 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3006b500 0x7f6a3006d9c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4415b130 0x7f6a4413a090 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44144640 0x7f6a4413b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 --2- 172.21.15.27:0/1920928972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a440ce900 0x7f6a44132540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 >> 172.21.15.27:0/1920928972 conn(0x7f6a44066d00 msgr2=0x7f6a4415d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 shutdown_connections 2024-03-06T07:24:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.602+0000 7f6a4b13e640 1 -- 172.21.15.27:0/1920928972 wait complete. 2024-03-06T07:24:05.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:05.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.725+0000 7f424477a640 1 Processor -- start 2024-03-06T07:24:05.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.725+0000 7f424477a640 1 -- start start 2024-03-06T07:24:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f424477a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f424477a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f423c0aee10 0x7f423c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f424477a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423c071240 con 0x7f423c06dd60 2024-03-06T07:24:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f424477a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423c06e1c0 con 0x7f423c0aee10 2024-03-06T07:24:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f424477a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423c071b70 con 0x7f423c158520 2024-03-06T07:24:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4241d15640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f423c06dd60 0x7f423c0ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36386/0 (socket says 172.21.15.27:36386) 2024-03-06T07:24:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4241d15640 1 -- 172.21.15.27:0/2237504614 learned_addr learned my addr 172.21.15.27:0/2237504614 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4242d17640 1 --2- 172.21.15.27:0/2237504614 >> v2:172.21.15.69:3300/0 conn(0x7f423c0aee10 0x7f423c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4242516640 1 --2- 172.21.15.27:0/2237504614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4241514640 1 -- 172.21.15.27:0/2237504614 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297979352 0 0) 0x7f423c071240 con 0x7f423c06dd60 2024-03-06T07:24:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4241514640 1 -- 172.21.15.27:0/2237504614 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4228003520 con 0x7f423c06dd60 2024-03-06T07:24:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4242d17640 1 -- 172.21.15.27:0/2237504614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 msgr2=0x7f423c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4242d17640 1 --2- 172.21.15.27:0/2237504614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4242d17640 1 -- 172.21.15.27:0/2237504614 >> v1:172.21.15.27:6789/0 conn(0x7f423c06dd60 legacy=0x7f423c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4242d17640 1 -- 172.21.15.27:0/2237504614 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c071520 con 0x7f423c0aee10 2024-03-06T07:24:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.726+0000 7f4242516640 1 --2- 172.21.15.27:0/2237504614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:05.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.727+0000 7f4242d17640 1 --2- 172.21.15.27:0/2237504614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c0aee10 0x7f423c0b3640 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f423400bfa0 tx=0x7f423402e0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e777908f179ff19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.727+0000 7f4241514640 1 -- 172.21.15.27:0/2237504614 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423402ea40 con 0x7f423c0aee10 2024-03-06T07:24:05.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.727+0000 7f4241514640 1 -- 172.21.15.27:0/2237504614 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423402ebe0 con 0x7f423c0aee10 2024-03-06T07:24:05.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.727+0000 7f4241514640 1 -- 172.21.15.27:0/2237504614 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4234009540 con 0x7f423c0aee10 2024-03-06T07:24:05.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.728+0000 7f424477a640 1 -- 172.21.15.27:0/2237504614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c0aee10 msgr2=0x7f423c0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.728+0000 7f424477a640 1 --2- 172.21.15.27:0/2237504614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c0aee10 0x7f423c0b3640 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f423400bfa0 tx=0x7f423402e0a0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.728+0000 7f424477a640 1 -- 172.21.15.27:0/2237504614 shutdown_connections 2024-03-06T07:24:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.728+0000 7f424477a640 1 --2- 172.21.15.27:0/2237504614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c0aee10 0x7f423c0b3640 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.728+0000 7f424477a640 1 --2- 172.21.15.27:0/2237504614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.728+0000 7f424477a640 1 -- 172.21.15.27:0/2237504614 >> 172.21.15.27:0/2237504614 conn(0x7f423c066d00 msgr2=0x7f423c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.728+0000 7f424477a640 1 -- 172.21.15.27:0/2237504614 shutdown_connections 2024-03-06T07:24:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.728+0000 7f424477a640 1 -- 172.21.15.27:0/2237504614 wait complete. 2024-03-06T07:24:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.730+0000 7f424477a640 1 Processor -- start 2024-03-06T07:24:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.730+0000 7f424477a640 1 -- start start 2024-03-06T07:24:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.731+0000 7f424477a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c06dd60 0x7f423c155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.731+0000 7f424477a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f423c0aee10 0x7f423c151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.731+0000 7f424477a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.731+0000 7f424477a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423c158900 con 0x7f423c0aee10 2024-03-06T07:24:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.731+0000 7f424477a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423c072000 con 0x7f423c06dd60 2024-03-06T07:24:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.731+0000 7f424477a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423c158a80 con 0x7f423c158520 2024-03-06T07:24:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.731+0000 7f4241d15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f423c0aee10 0x7f423c151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.731+0000 7f4241d15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f423c0aee10 0x7f423c151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51324/0 (socket says 172.21.15.27:51324) 2024-03-06T07:24:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.731+0000 7f4241d15640 1 -- 172.21.15.27:0/516924068 learned_addr learned my addr 172.21.15.27:0/516924068 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4242516640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c06dd60 0x7f423c155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4242d17640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4241d15640 1 -- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 msgr2=0x7f423c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4241d15640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c0cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4241d15640 1 -- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c06dd60 msgr2=0x7f423c155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4241d15640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c06dd60 0x7f423c155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4241d15640 1 -- 172.21.15.27:0/516924068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c0cf950 con 0x7f423c0aee10 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4242516640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c06dd60 0x7f423c155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4242d17640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4241d15640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f423c0aee10 0x7f423c151f70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f422c00c430 tx=0x7f422c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4232ffd640 1 -- 172.21.15.27:0/516924068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422c017070 con 0x7f423c0aee10 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4232ffd640 1 -- 172.21.15.27:0/516924068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422c005ce0 con 0x7f423c0aee10 2024-03-06T07:24:05.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423c154e70 con 0x7f423c0aee10 2024-03-06T07:24:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f4232ffd640 1 -- 172.21.15.27:0/516924068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422c0072f0 con 0x7f423c0aee10 2024-03-06T07:24:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.732+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423c155430 con 0x7f423c0aee10 2024-03-06T07:24:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.733+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423c067ff0 con 0x7f423c0aee10 2024-03-06T07:24:05.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.734+0000 7f4232ffd640 1 -- 172.21.15.27:0/516924068 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f422c01a070 con 0x7f423c0aee10 2024-03-06T07:24:05.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.734+0000 7f4232ffd640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f422006b4d0 0x7f422006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.734+0000 7f4232ffd640 1 -- 172.21.15.27:0/516924068 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f422c0896d0 con 0x7f423c0aee10 2024-03-06T07:24:05.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.734+0000 7f4242516640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f422006b4d0 0x7f422006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.735+0000 7f4242516640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f422006b4d0 0x7f422006d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4238009a20 tx=0x7f4238008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.738+0000 7f4232ffd640 1 -- 172.21.15.27:0/516924068 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f423c0cf950 con 0x7f423c0aee10 2024-03-06T07:24:05.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.958+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f423c0b31f0 con 0x7f423c0aee10 2024-03-06T07:24:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.958+0000 7f4232ffd640 1 -- 172.21.15.27:0/516924068 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f422c058560 con 0x7f423c0aee10 2024-03-06T07:24:05.961 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:05.961 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:05.961 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f422006b4d0 msgr2=0x7f422006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f422006b4d0 0x7f422006d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4238009a20 tx=0x7f4238008040 comp rx=0 tx=0).stop 2024-03-06T07:24:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f423c0aee10 msgr2=0x7f423c151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f423c0aee10 0x7f423c151f70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f422c00c430 tx=0x7f422c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 shutdown_connections 2024-03-06T07:24:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f422006b4d0 0x7f422006d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f423c158520 0x7f423c0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f423c0aee10 0x7f423c151f70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 --2- 172.21.15.27:0/516924068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f423c06dd60 0x7f423c155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.961+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 >> 172.21.15.27:0/516924068 conn(0x7f423c066d00 msgr2=0x7f423c159340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.962+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 shutdown_connections 2024-03-06T07:24:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:05.962+0000 7f424477a640 1 -- 172.21.15.27:0/516924068 wait complete. 2024-03-06T07:24:06.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:55.014569+0000', 'modified': '2024-03-06T07:23:58.060823+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5286}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5286': {'gid': 5286, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.27:6835/136269066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6834', 'nonce': 136269066}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 136269066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5286, 'qdb_cluster': [5286]} 2024-03-06T07:24:06.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:06.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:24:06.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.483+0000 7f74948aa640 1 Processor -- start 2024-03-06T07:24:06.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.483+0000 7f74948aa640 1 -- start start 2024-03-06T07:24:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.484+0000 7f74948aa640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f74900b9a10 0x7f74900d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.484+0000 7f74948aa640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900cea30 con 0x7f74900b95a0 2024-03-06T07:24:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.484+0000 7f74948aa640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900cf3c0 con 0x7f749015df00 2024-03-06T07:24:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.484+0000 7f74948aa640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74900cf0e0 con 0x7f74900b9a10 2024-03-06T07:24:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.484+0000 7f7487fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f74900b95a0 0x7f74900b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36398/0 (socket says 172.21.15.27:36398) 2024-03-06T07:24:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.484+0000 7f7487fff640 1 -- 172.21.15.27:0/479722975 learned_addr learned my addr 172.21.15.27:0/479722975 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.484+0000 7f748ed9d640 1 --2- 172.21.15.27:0/479722975 >> v2:172.21.15.69:3300/0 conn(0x7f74900b9a10 0x7f74900d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.485+0000 7f748dd9b640 1 -- 172.21.15.27:0/479722975 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930969606 0 0) 0x7f74900cea30 con 0x7f74900b95a0 2024-03-06T07:24:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.485+0000 7f748dd9b640 1 -- 172.21.15.27:0/479722975 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f747c003520 con 0x7f74900b95a0 2024-03-06T07:24:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.485+0000 7f748ed9d640 1 -- 172.21.15.27:0/479722975 >> v1:172.21.15.69:6789/0 conn(0x7f749015df00 legacy=0x7f74900d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.485+0000 7f748dd9b640 1 -- 172.21.15.27:0/479722975 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622761772 0 0) 0x7f74900cf3c0 con 0x7f749015df00 2024-03-06T07:24:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.485+0000 7f748ed9d640 1 -- 172.21.15.27:0/479722975 >> v1:172.21.15.27:6789/0 conn(0x7f74900b95a0 legacy=0x7f74900b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.485+0000 7f748ed9d640 1 -- 172.21.15.27:0/479722975 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900ce470 con 0x7f74900b9a10 2024-03-06T07:24:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.486+0000 7f748ed9d640 1 --2- 172.21.15.27:0/479722975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74900b9a10 0x7f74900d3220 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f74900cf590 tx=0x7f748802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a484a6faefcd6b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.486+0000 7f748dd9b640 1 -- 172.21.15.27:0/479722975 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488039070 con 0x7f74900b9a10 2024-03-06T07:24:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.486+0000 7f748dd9b640 1 -- 172.21.15.27:0/479722975 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748802cce0 con 0x7f74900b9a10 2024-03-06T07:24:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.486+0000 7f748dd9b640 1 -- 172.21.15.27:0/479722975 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488034640 con 0x7f74900b9a10 2024-03-06T07:24:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.487+0000 7f74948aa640 1 -- 172.21.15.27:0/479722975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74900b9a10 msgr2=0x7f74900d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.487+0000 7f74948aa640 1 --2- 172.21.15.27:0/479722975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74900b9a10 0x7f74900d3220 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f74900cf590 tx=0x7f748802bf30 comp rx=0 tx=0).stop 2024-03-06T07:24:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.487+0000 7f74948aa640 1 -- 172.21.15.27:0/479722975 shutdown_connections 2024-03-06T07:24:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.487+0000 7f74948aa640 1 --2- 172.21.15.27:0/479722975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74900b9a10 0x7f74900d3220 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.487+0000 7f74948aa640 1 -- 172.21.15.27:0/479722975 >> 172.21.15.27:0/479722975 conn(0x7f7490071960 msgr2=0x7f7490071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.487+0000 7f74948aa640 1 -- 172.21.15.27:0/479722975 shutdown_connections 2024-03-06T07:24:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.487+0000 7f74948aa640 1 -- 172.21.15.27:0/479722975 wait complete. 2024-03-06T07:24:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.488+0000 7f74948aa640 1 Processor -- start 2024-03-06T07:24:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.488+0000 7f74948aa640 1 -- start start 2024-03-06T07:24:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f74948aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b95a0 0x7f749015fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f74948aa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900b9a10 0x7f7490149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f74948aa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749015df00 0x7f749014cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f74948aa640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74900cedf0 con 0x7f74900b95a0 2024-03-06T07:24:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f74948aa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74900cec70 con 0x7f749015df00 2024-03-06T07:24:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f74948aa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74900b5e10 con 0x7f74900b9a10 2024-03-06T07:24:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f748e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b95a0 0x7f749015fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f748e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b95a0 0x7f749015fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51336/0 (socket says 172.21.15.27:51336) 2024-03-06T07:24:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f748e59c640 1 -- 172.21.15.27:0/1096849521 learned_addr learned my addr 172.21.15.27:0/1096849521 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f7487fff640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900b9a10 0x7f7490149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.489+0000 7f748ed9d640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749015df00 0x7f749014cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f748e59c640 1 -- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900b9a10 msgr2=0x7f7490149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f748e59c640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900b9a10 0x7f7490149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f748e59c640 1 -- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749015df00 msgr2=0x7f749014cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f748e59c640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749015df00 0x7f749014cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f748e59c640 1 -- 172.21.15.27:0/1096849521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749014d0b0 con 0x7f74900b95a0 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f748e59c640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b95a0 0x7f749015fcf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f748000c410 tx=0x7f748000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f7486ffd640 1 -- 172.21.15.27:0/1096849521 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7480017070 con 0x7f74900b95a0 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f7486ffd640 1 -- 172.21.15.27:0/1096849521 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74800035f0 con 0x7f74900b95a0 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f74948aa640 1 -- 172.21.15.27:0/1096849521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7490160340 con 0x7f74900b95a0 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f7486ffd640 1 -- 172.21.15.27:0/1096849521 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800039e0 con 0x7f74900b95a0 2024-03-06T07:24:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.490+0000 7f74948aa640 1 -- 172.21.15.27:0/1096849521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7490160900 con 0x7f74900b95a0 2024-03-06T07:24:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.492+0000 7f7486ffd640 1 -- 172.21.15.27:0/1096849521 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f748001a070 con 0x7f74900b95a0 2024-03-06T07:24:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.492+0000 7f7486ffd640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.492+0000 7f7486ffd640 1 -- 172.21.15.27:0/1096849521 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7480089540 con 0x7f74900b95a0 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.492+0000 7f7487fff640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.493+0000 7f7497568740 1 Processor -- start 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.493+0000 7f7487fff640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7490150820 tx=0x7f7478009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.493+0000 7f7497568740 1 -- start start 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7497568740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21baf870 0x560c21bafc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7497568740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bb38c0 0x560c21bb5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7497568740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 0x560c21bdfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7497568740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c21b196b0 con 0x560c21bb38c0 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7497568740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c21ac2a60 con 0x560c21baf870 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7497568740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c21b15750 con 0x560c21bdf600 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7487fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bb38c0 0x560c21bb5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7487fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bb38c0 0x560c21bb5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51340/0 (socket says 172.21.15.27:51340) 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7487fff640 1 -- 172.21.15.27:0/1822862808 learned_addr learned my addr 172.21.15.27:0/1822862808 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f748ed9d640 1 --2- 172.21.15.27:0/1822862808 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 0x560c21bdfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f748e59c640 1 --2- 172.21.15.27:0/1822862808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21baf870 0x560c21bafc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7487fff640 1 -- 172.21.15.27:0/1822862808 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 msgr2=0x560c21bdfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7487fff640 1 --2- 172.21.15.27:0/1822862808 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 0x560c21bdfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7487fff640 1 -- 172.21.15.27:0/1822862808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21baf870 msgr2=0x560c21bafc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7487fff640 1 --2- 172.21.15.27:0/1822862808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21baf870 0x560c21bafc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.494+0000 7f7487fff640 1 -- 172.21.15.27:0/1822862808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c21ae0f30 con 0x560c21bb38c0 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f7487fff640 1 --2- 172.21.15.27:0/1822862808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bb38c0 0x560c21bb5360 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f747800fae0 tx=0x7f7478000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294a9b961d2f6903 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f746ffff640 1 -- 172.21.15.27:0/1822862808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478025070 con 0x560c21bb38c0 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f746ffff640 1 -- 172.21.15.27:0/1822862808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7478015cb0 con 0x560c21bb38c0 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f746ffff640 1 -- 172.21.15.27:0/1822862808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478015e60 con 0x560c21bb38c0 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f7497568740 1 -- 172.21.15.27:0/1822862808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bb38c0 msgr2=0x560c21bb5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f7497568740 1 --2- 172.21.15.27:0/1822862808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bb38c0 0x560c21bb5360 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f747800fae0 tx=0x7f7478000ed0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f7497568740 1 -- 172.21.15.27:0/1822862808 shutdown_connections 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f7497568740 1 --2- 172.21.15.27:0/1822862808 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 0x560c21bdfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f7497568740 1 --2- 172.21.15.27:0/1822862808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bb38c0 0x560c21bb5360 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f7497568740 1 --2- 172.21.15.27:0/1822862808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21baf870 0x560c21bafc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.495+0000 7f7497568740 1 -- 172.21.15.27:0/1822862808 >> 172.21.15.27:0/1822862808 conn(0x560c21bb18f0 msgr2=0x560c21bb31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.496+0000 7f7497568740 1 -- 172.21.15.27:0/1822862808 shutdown_connections 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.496+0000 7f7497568740 1 -- 172.21.15.27:0/1822862808 wait complete. 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.496+0000 7f7497568740 1 -- start start 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.496+0000 7f7497568740 1 Processor -- start 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7497568740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bdf600 0x560c21bd9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7497568740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21be2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7497568740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bf76a0 0x560c21be3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7497568740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c21bb3d50 con 0x560c21bfa540 2024-03-06T07:24:06.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7497568740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c21ad5ea0 con 0x560c21bdf600 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7497568740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c21bb4030 con 0x560c21bf76a0 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7497568740 10 client.0 _ll_register_callbacks cb 0x560c21ad5870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7487fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21be2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7487fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21be2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51356/0 (socket says 172.21.15.27:51356) 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f7487fff640 1 -- 172.21.15.27:0/2194949834 learned_addr learned my addr 172.21.15.27:0/2194949834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f748ed9d640 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bf76a0 0x560c21be3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.497+0000 7f748e59c640 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bdf600 0x560c21bd9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7487fff640 1 -- 172.21.15.27:0/2194949834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bf76a0 msgr2=0x560c21be3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7487fff640 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bf76a0 0x560c21be3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7487fff640 1 -- 172.21.15.27:0/2194949834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bdf600 msgr2=0x560c21bd9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7487fff640 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bdf600 0x560c21bd9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7487fff640 1 -- 172.21.15.27:0/2194949834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c21ad6270 con 0x560c21bfa540 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7487fff640 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21be2b10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7478006890 tx=0x7f7478000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5cac1228d7f717 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f746d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f746d7fa640 1 -- 172.21.15.27:0/2194949834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478030020 con 0x560c21bfa540 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f746d7fa640 1 -- 172.21.15.27:0/2194949834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7478020bd0 con 0x560c21bfa540 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7497568740 1 -- 172.21.15.27:0/2194949834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7478015cb0 con 0x560c21bfa540 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f746d7fa640 1 -- 172.21.15.27:0/2194949834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f747803a050 con 0x560c21bfa540 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7497568740 10 client.5487 fetch_fsmap learned FSMap version 17 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7497568740 1 -- 172.21.15.27:0/2194949834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x560c2182e2d0 con 0x560c21bfa540 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f746d7fa640 1 -- 172.21.15.27:0/2194949834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7478015cb0 con 0x560c21bfa540 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f746d7fa640 1 -- 172.21.15.27:0/2194949834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f747803d020 con 0x560c21bfa540 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7497568740 10 client.5487 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7497568740 10 client.5487 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.498+0000 7f7497568740 20 client.5487 populate_metadata read hostname 'smithi027' 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.499+0000 7f7497568740 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] conn(0x560c21bf9a20 0x560c21bf9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.499+0000 7f7497568740 4 client.5487 mds_command: new command op to 5286 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.499+0000 7f7497568740 1 -- 172.21.15.27:0/2194949834 --> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c21b26bf0 con 0x560c21bf9a20 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.499+0000 7f748e59c640 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] conn(0x560c21bf9a20 0x560c21bf9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.499+0000 7f746d7fa640 10 client.5487 ms_handle_connect on v2:172.21.15.27:6834/136269066 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.499+0000 7f748e59c640 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] conn(0x560c21bf9a20 0x560c21bf9e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8457d107b15b95f6 server_cookie=809bd08c68c43bee in_seq=0 out_seq=0 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f746d7fa640 1 -- 172.21.15.27:0/2194949834 <== mds.0 v2:172.21.15.27:6834/136269066 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560c21b26bf0 con 0x560c21bf9a20 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f746d7fa640 10 client.5487 handle_command_reply: tid=0 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 client.5487 shutdown 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 -- 172.21.15.27:0/2194949834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 msgr2=0x560c21be2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21be2b10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7478006890 tx=0x7f7478000bc0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 -- 172.21.15.27:0/2194949834 shutdown_connections 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] conn(0x560c21bf9a20 0x560c21bf9e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bf76a0 0x560c21be3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21be2b10 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 --2- 172.21.15.27:0/2194949834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bdf600 0x560c21bd9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f746d7fa640 0 client.5487 ms_handle_reset on v2:172.21.15.27:6834/136269066 2024-03-06T07:24:06.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 -- 172.21.15.27:0/2194949834 >> 172.21.15.27:0/2194949834 conn(0x560c21baf870 msgr2=0x560c21bae510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 -- 172.21.15.27:0/2194949834 shutdown_connections 2024-03-06T07:24:06.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 1 -- 172.21.15.27:0/2194949834 wait complete. 2024-03-06T07:24:06.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.500+0000 7f7497568740 20 client.5487 trim_cache size 0 max 16384 2024-03-06T07:24:06.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.517+0000 7f7497568740 1 Processor -- start 2024-03-06T07:24:06.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.517+0000 7f7497568740 1 -- start start 2024-03-06T07:24:06.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f7497568740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bdf600 0x560c21bab040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f7497568740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f7497568740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bfa540 0x560c21bfdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f7497568740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74901fef00 con 0x560c21bdf600 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f7497568740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74900c8fc0 con 0x560c21bf76a0 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f7497568740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74900c95c0 con 0x560c21bfa540 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bdf600 0x560c21bab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bdf600 0x560c21bab040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51368/0 (socket says 172.21.15.27:51368) 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748e59c640 1 -- 172.21.15.27:0/4172625559 learned_addr learned my addr 172.21.15.27:0/4172625559 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748ed9d640 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bfa540 0x560c21bfdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f7487fff640 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748e59c640 1 -- 172.21.15.27:0/4172625559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bfa540 msgr2=0x560c21bfdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748e59c640 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bfa540 0x560c21bfdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748e59c640 1 -- 172.21.15.27:0/4172625559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 msgr2=0x560c21bf41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748e59c640 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748e59c640 1 -- 172.21.15.27:0/4172625559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c218f86b0 con 0x560c21bdf600 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f7487fff640 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f748e59c640 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bdf600 0x560c21bab040 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f748000cdd0 tx=0x7f748001e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2d7c4049bd5af7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.518+0000 7f746f7fe640 1 -- 172.21.15.27:0/4172625559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7480056070 con 0x560c21bdf600 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f746f7fe640 1 -- 172.21.15.27:0/4172625559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74800543e0 con 0x560c21bdf600 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f746f7fe640 1 -- 172.21.15.27:0/4172625559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7480003da0 con 0x560c21bdf600 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f7497568740 1 -- 172.21.15.27:0/4172625559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bdf600 msgr2=0x560c21bab040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f7497568740 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bdf600 0x560c21bab040 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f748000cdd0 tx=0x7f748001e040 comp rx=0 tx=0).stop 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f7497568740 1 -- 172.21.15.27:0/4172625559 shutdown_connections 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f7497568740 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bfa540 0x560c21bfdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f7497568740 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f7497568740 1 --2- 172.21.15.27:0/4172625559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bdf600 0x560c21bab040 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f7497568740 1 -- 172.21.15.27:0/4172625559 >> 172.21.15.27:0/4172625559 conn(0x560c21baf870 msgr2=0x560c21bb3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f7497568740 1 -- 172.21.15.27:0/4172625559 shutdown_connections 2024-03-06T07:24:06.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.519+0000 7f7497568740 1 -- 172.21.15.27:0/4172625559 wait complete. 2024-03-06T07:24:06.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.520+0000 7f7497568740 1 -- start start 2024-03-06T07:24:06.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.520+0000 7f7497568740 1 Processor -- start 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7497568740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 0x560c21bf4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7497568740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7497568740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21bfdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7497568740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c21be0770 con 0x560c21bfa540 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7497568740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c21bb3d50 con 0x560c21bf76a0 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7497568740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c21bf6030 con 0x560c21bdf600 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7497568740 10 client.0 _ll_register_callbacks cb 0x7f747c0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21bfdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21bfdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51384/0 (socket says 172.21.15.27:51384) 2024-03-06T07:24:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748ed9d640 1 -- 172.21.15.27:0/3475033067 learned_addr learned my addr 172.21.15.27:0/3475033067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748e59c640 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 0x560c21bf4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7487fff640 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748ed9d640 1 -- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 msgr2=0x560c21bf4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748ed9d640 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 0x560c21bf4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748ed9d640 1 -- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 msgr2=0x560c21bf5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748ed9d640 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748ed9d640 1 -- 172.21.15.27:0/3475033067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c21bb5350 con 0x560c21bfa540 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748e59c640 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 0x560c21bf4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7487fff640 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f748ed9d640 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21bfdad0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f748802c540 tx=0x7f748802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e600904c46045807 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f746d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f746d7fa640 1 -- 172.21.15.27:0/3475033067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488039040 con 0x560c21bfa540 2024-03-06T07:24:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f746d7fa640 1 -- 172.21.15.27:0/3475033067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7488041b00 con 0x560c21bfa540 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.521+0000 7f7497568740 1 -- 172.21.15.27:0/3475033067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74800543e0 con 0x560c21bfa540 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f746d7fa640 1 -- 172.21.15.27:0/3475033067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7488055050 con 0x560c21bfa540 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f7497568740 10 client.5493 fetch_fsmap learned FSMap version 17 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f7497568740 1 -- 172.21.15.27:0/3475033067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x560c21bf41a0 con 0x560c21bfa540 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f746d7fa640 1 -- 172.21.15.27:0/3475033067 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74800543e0 con 0x560c21bfa540 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f746d7fa640 1 -- 172.21.15.27:0/3475033067 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7488058020 con 0x560c21bfa540 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f7497568740 10 client.5493 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f7497568740 10 client.5493 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f7497568740 20 client.5493 populate_metadata read hostname 'smithi027' 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f7497568740 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] conn(0x560c21c1c6d0 0x560c21c1cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f7497568740 4 client.5493 mds_command: new command op to 5286 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f7497568740 1 -- 172.21.15.27:0/3475033067 --> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560c21bb2050 con 0x560c21c1c6d0 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f748e59c640 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] conn(0x560c21c1c6d0 0x560c21c1cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f746d7fa640 10 client.5493 ms_handle_connect on v2:172.21.15.27:6834/136269066 2024-03-06T07:24:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f748e59c640 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] conn(0x560c21c1c6d0 0x560c21c1cab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89ee8ccdf1047693 server_cookie=3bf61c7dc5b9773a in_seq=0 out_seq=0 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f746d7fa640 1 -- 172.21.15.27:0/3475033067 <== mds.0 v2:172.21.15.27:6834/136269066 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x560c21bb2050 con 0x560c21c1c6d0 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f746d7fa640 10 client.5493 handle_command_reply: tid=0 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.522+0000 7f7497568740 1 client.5493 shutdown 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 -- 172.21.15.27:0/3475033067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 msgr2=0x560c21bfdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21bfdad0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f748802c540 tx=0x7f748802e070 comp rx=0 tx=0).stop 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 -- 172.21.15.27:0/3475033067 shutdown_connections 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.27:6834/136269066,v1:172.21.15.27:6835/136269066] conn(0x560c21c1c6d0 0x560c21c1cab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560c21bfa540 0x560c21bfdad0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c21bf76a0 0x560c21bf5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f746d7fa640 0 client.5493 ms_handle_reset on v2:172.21.15.27:6834/136269066 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 --2- 172.21.15.27:0/3475033067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x560c21bdf600 0x560c21bf4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 -- 172.21.15.27:0/3475033067 >> 172.21.15.27:0/3475033067 conn(0x560c21baf870 msgr2=0x560c21baac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 -- 172.21.15.27:0/3475033067 shutdown_connections 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 1 -- 172.21.15.27:0/3475033067 wait complete. 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f7497568740 20 client.5493 trim_cache size 0 max 16384 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5286, 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:24:06.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.886345846999999, 2024-03-06T07:24:06.534 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 17, 2024-03-06T07:24:06.534 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 35, 2024-03-06T07:24:06.534 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 35, 2024-03-06T07:24:06.534 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 19.130899500000002 2024-03-06T07:24:06.534 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 -- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 msgr2=0x7f747006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7490150820 tx=0x7f7478009040 comp rx=0 tx=0).stop 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 -- 172.21.15.27:0/1096849521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b95a0 msgr2=0x7f749015fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b95a0 0x7f749015fcf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f748000c410 tx=0x7f748000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 -- 172.21.15.27:0/1096849521 shutdown_connections 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f749015df00 0x7f749014cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900b9a10 0x7f7490149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 --2- 172.21.15.27:0/1096849521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b95a0 0x7f749015fcf0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.523+0000 7f74948aa640 1 -- 172.21.15.27:0/1096849521 >> 172.21.15.27:0/1096849521 conn(0x7f7490071960 msgr2=0x7f74900b7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.524+0000 7f74948aa640 1 -- 172.21.15.27:0/1096849521 shutdown_connections 2024-03-06T07:24:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.524+0000 7f74948aa640 1 -- 172.21.15.27:0/1096849521 wait complete. 2024-03-06T07:24:06.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:06.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.640+0000 7f912aba0640 1 Processor -- start 2024-03-06T07:24:06.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.641+0000 7f912aba0640 1 -- start start 2024-03-06T07:24:06.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.641+0000 7f912aba0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9124155080 0x7f912415e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.641+0000 7f912aba0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91240b4050 con 0x7f912415e980 2024-03-06T07:24:06.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.641+0000 7f912aba0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912415efb0 con 0x7f9124154ca0 2024-03-06T07:24:06.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.641+0000 7f912aba0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912415b1c0 con 0x7f9124155080 2024-03-06T07:24:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.642+0000 7f912939d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9124155080 0x7f912415e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.642+0000 7f912a39f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f912415e980 0x7f912406af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36404/0 (socket says 172.21.15.27:36404) 2024-03-06T07:24:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.642+0000 7f912a39f640 1 -- 172.21.15.27:0/231505831 learned_addr learned my addr 172.21.15.27:0/231505831 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.642+0000 7f912939d640 1 -- 172.21.15.27:0/231505831 >> v1:172.21.15.69:6789/0 conn(0x7f9124154ca0 legacy=0x7f91240b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.642+0000 7f912939d640 1 -- 172.21.15.27:0/231505831 >> v1:172.21.15.27:6789/0 conn(0x7f912415e980 legacy=0x7f912406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.642+0000 7f912939d640 1 -- 172.21.15.27:0/231505831 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912415f290 con 0x7f9124155080 2024-03-06T07:24:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.642+0000 7f9128b9c640 1 -- 172.21.15.27:0/231505831 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857542300 0 0) 0x7f91240b4050 con 0x7f912415e980 2024-03-06T07:24:06.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.643+0000 7f912939d640 1 --2- 172.21.15.27:0/231505831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124155080 0x7f912415e440 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f91140097a0 tx=0x7f911402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521275eaabae7011 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.643+0000 7f9128b9c640 1 -- 172.21.15.27:0/231505831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114035020 con 0x7f9124155080 2024-03-06T07:24:06.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.643+0000 7f9128b9c640 1 -- 172.21.15.27:0/231505831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9114031830 con 0x7f9124155080 2024-03-06T07:24:06.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.643+0000 7f9128b9c640 1 -- 172.21.15.27:0/231505831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114031c00 con 0x7f9124155080 2024-03-06T07:24:06.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.644+0000 7f912aba0640 1 -- 172.21.15.27:0/231505831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124155080 msgr2=0x7f912415e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.644+0000 7f912aba0640 1 --2- 172.21.15.27:0/231505831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124155080 0x7f912415e440 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f91140097a0 tx=0x7f911402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.644+0000 7f912aba0640 1 -- 172.21.15.27:0/231505831 shutdown_connections 2024-03-06T07:24:06.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.644+0000 7f912aba0640 1 --2- 172.21.15.27:0/231505831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124155080 0x7f912415e440 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.644+0000 7f912aba0640 1 -- 172.21.15.27:0/231505831 >> 172.21.15.27:0/231505831 conn(0x7f91240b2460 msgr2=0x7f91240b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.644+0000 7f912aba0640 1 -- 172.21.15.27:0/231505831 shutdown_connections 2024-03-06T07:24:06.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.644+0000 7f912aba0640 1 -- 172.21.15.27:0/231505831 wait complete. 2024-03-06T07:24:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.645+0000 7f912aba0640 1 Processor -- start 2024-03-06T07:24:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f912aba0640 1 -- start start 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f912aba0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124154ca0 0x7f9124149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f912aba0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9124155080 0x7f91240c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f912aba0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f912415e980 0x7f91240ce850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f912aba0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912406b630 con 0x7f9124154ca0 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f912aba0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91240b45f0 con 0x7f912415e980 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f912aba0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912406b7b0 con 0x7f9124155080 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f9129b9e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124154ca0 0x7f9124149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f9129b9e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124154ca0 0x7f9124149260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51390/0 (socket says 172.21.15.27:51390) 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.646+0000 7f9129b9e640 1 -- 172.21.15.27:0/429313612 learned_addr learned my addr 172.21.15.27:0/429313612 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f912939d640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9124155080 0x7f91240c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f912a39f640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f912415e980 0x7f91240ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f9129b9e640 1 -- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9124155080 msgr2=0x7f91240c70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f9129b9e640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9124155080 0x7f91240c70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f9129b9e640 1 -- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f912415e980 msgr2=0x7f91240ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f9129b9e640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f912415e980 0x7f91240ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f9129b9e640 1 -- 172.21.15.27:0/429313612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91241497a0 con 0x7f9124154ca0 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f9129b9e640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124154ca0 0x7f9124149260 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f912000c430 tx=0x7f912000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f911a7fc640 1 -- 172.21.15.27:0/429313612 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120011790 con 0x7f9124154ca0 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f911a7fc640 1 -- 172.21.15.27:0/429313612 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9120017070 con 0x7f9124154ca0 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91240cdad0 con 0x7f9124154ca0 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f911a7fc640 1 -- 172.21.15.27:0/429313612 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120011bb0 con 0x7f9124154ca0 2024-03-06T07:24:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.647+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91240cdd70 con 0x7f9124154ca0 2024-03-06T07:24:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.648+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912404f330 con 0x7f9124154ca0 2024-03-06T07:24:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.649+0000 7f911a7fc640 1 -- 172.21.15.27:0/429313612 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f912001c460 con 0x7f9124154ca0 2024-03-06T07:24:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.649+0000 7f911a7fc640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f910806b4d0 0x7f910806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.649+0000 7f911a7fc640 1 -- 172.21.15.27:0/429313612 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f912008a060 con 0x7f9124154ca0 2024-03-06T07:24:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.649+0000 7f912939d640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f910806b4d0 0x7f910806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.650+0000 7f912939d640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f910806b4d0 0x7f910806d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9114036000 tx=0x7f9114008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.653+0000 7f911a7fc640 1 -- 172.21.15.27:0/429313612 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91241497a0 con 0x7f9124154ca0 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.965+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f912406dbb0 con 0x7f9124154ca0 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.966+0000 7f911a7fc640 1 -- 172.21.15.27:0/429313612 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9120058ee0 con 0x7f9124154ca0 2024-03-06T07:24:06.968 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f910806b4d0 msgr2=0x7f910806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f910806b4d0 0x7f910806d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9114036000 tx=0x7f9114008030 comp rx=0 tx=0).stop 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124154ca0 msgr2=0x7f9124149260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124154ca0 0x7f9124149260 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f912000c430 tx=0x7f912000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 shutdown_connections 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f910806b4d0 0x7f910806d990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f912415e980 0x7f91240ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9124155080 0x7f91240c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 --2- 172.21.15.27:0/429313612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9124154ca0 0x7f9124149260 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 >> 172.21.15.27:0/429313612 conn(0x7f91240b2460 msgr2=0x7f9124066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 shutdown_connections 2024-03-06T07:24:06.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:06.969+0000 7f912aba0640 1 -- 172.21.15.27:0/429313612 wait complete. 2024-03-06T07:24:06.987 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:06.987 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:06.987 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:24:06.987 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:06.987 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:06.987 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:06.987 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:06.987 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:06.987 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:06.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:06.988 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:24:07.003 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:07.003 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42643sec preferred_lft 42643sec 2024-03-06T07:24:07.004 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:07.005 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:24:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:07.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:07.006 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:07.072 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:24:07.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:07.074 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:24:07.129 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:24:07.130 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:07.130 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:24:07.130 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:24:07.130 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:07.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:07.130 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:24:07.130 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:24:07.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:24:07.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:24:07.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:24:07.131 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:24:07.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:07.236 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:24:07.251 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:07.251 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:07.251 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:07.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:07.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:24:07.323 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:07.323 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:07.323 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:07.377 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:07.377 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:07.377 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:07.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:24:07.503 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:07.559 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:24:07.559 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:24:07.559 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:24:07.559 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:24:07.559 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:24:07.559 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:24:07.559 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:07.560 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:07.561 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:07.562 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:07.562 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:07.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:07.562 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:24:07.614 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:07.614 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:07.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:24:07.688 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:07.744 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:24:07.744 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:24:07.744 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:24:07.744 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:07.745 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:07.746 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:07.747 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:07.747 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:07.747 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:24:07.747 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:07.747 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:07.747 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:07.747 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:07.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:07.747 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:24:07.758 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:24:07.756+0000 7fbc4424d480 -1 init, newargv = 0x55b3f5d1f9d0 newargc=15 2024-03-06T07:24:07.758 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[52808]: starting ceph client 2024-03-06T07:24:07.763 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[52808]: starting fuse 2024-03-06T07:24:07.774 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:24:07.774 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:24:07.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:07.774 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:24:07.774 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:24:07.774 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:24:07.775 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:07.776 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:07.776 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:07.776 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:24:07.776 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:24:07.776 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:24:07.776 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:07.776 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:07.776 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:07.776 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:24:07.900 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.52808.asok 2024-03-06T07:24:07.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:07.909 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52808.asok 2024-03-06T07:24:07.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:07.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52808.asok status 2024-03-06T07:24:08.051 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "52808", 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5508, 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 5508 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3821075032 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:24:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3821075032 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.5508 192.168.0.1:0/3821075032", 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/3821075032", 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 17, 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 35, 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:24:08.054 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:08.060 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:24:08.060 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:08.075 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:24:08.075 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:08.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:08.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:08.151 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:24:08.151 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:24:08.151 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:08.151 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:08.152 DEBUG:teuthology.orchestra.run.smithi027:> subdir = "delete_me" 2024-03-06T07:24:08.152 DEBUG:teuthology.orchestra.run.smithi027:> size = 1024 2024-03-06T07:24:08.152 DEBUG:teuthology.orchestra.run.smithi027:> file_count = 1000 2024-03-06T07:24:08.152 DEBUG:teuthology.orchestra.run.smithi027:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:24:08.152 DEBUG:teuthology.orchestra.run.smithi027:> for i in range(0, file_count): 2024-03-06T07:24:08.152 DEBUG:teuthology.orchestra.run.smithi027:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:24:08.152 DEBUG:teuthology.orchestra.run.smithi027:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:24:08.152 DEBUG:teuthology.orchestra.run.smithi027:> f.write(size * '"'"'x'"'"') 2024-03-06T07:24:08.152 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:24:10.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:10.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:10.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.291+0000 7f7afba46640 1 Processor -- start 2024-03-06T07:24:10.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.291+0000 7f7afba46640 1 -- start start 2024-03-06T07:24:10.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.292+0000 7f7afba46640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7af415df00 0x7f7af40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.292+0000 7f7afba46640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af40cea30 con 0x7f7af40b9a30 2024-03-06T07:24:10.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.292+0000 7f7afba46640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af40d2af0 con 0x7f7af40b95a0 2024-03-06T07:24:10.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.292+0000 7f7afba46640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af40cf3c0 con 0x7f7af415df00 2024-03-06T07:24:10.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.292+0000 7f7af97e2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7af415df00 0x7f7af40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.292+0000 7f7af9fe3640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7af40b9a30 0x7f7af40d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36412/0 (socket says 172.21.15.27:36412) 2024-03-06T07:24:10.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.292+0000 7f7af9fe3640 1 -- 172.21.15.27:0/3842291091 learned_addr learned my addr 172.21.15.27:0/3842291091 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:10.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.292+0000 7f7af97e2640 1 --2- 172.21.15.27:0/3842291091 >> v2:172.21.15.27:3300/0 conn(0x7f7af415df00 0x7f7af40d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51394/0 (socket says 172.21.15.27:51394) 2024-03-06T07:24:10.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.293+0000 7f7af97e2640 1 -- 172.21.15.27:0/3842291091 >> v1:172.21.15.69:6789/0 conn(0x7f7af40b95a0 legacy=0x7f7af40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.293+0000 7f7af97e2640 1 -- 172.21.15.27:0/3842291091 >> v1:172.21.15.27:6789/0 conn(0x7f7af40b9a30 legacy=0x7f7af40d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.293+0000 7f7aebfff640 1 -- 172.21.15.27:0/3842291091 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222028052 0 0) 0x7f7af40cea30 con 0x7f7af40b9a30 2024-03-06T07:24:10.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.293+0000 7f7af97e2640 1 -- 172.21.15.27:0/3842291091 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af40cf0e0 con 0x7f7af415df00 2024-03-06T07:24:10.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.294+0000 7f7af97e2640 1 --2- 172.21.15.27:0/3842291091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af415df00 0x7f7af40d25b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7af00097a0 tx=0x7f7af002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab35433fbee56aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.294+0000 7f7aebfff640 1 -- 172.21.15.27:0/3842291091 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0034070 con 0x7f7af415df00 2024-03-06T07:24:10.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.294+0000 7f7aebfff640 1 -- 172.21.15.27:0/3842291091 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af0030610 con 0x7f7af415df00 2024-03-06T07:24:10.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.294+0000 7f7aebfff640 1 -- 172.21.15.27:0/3842291091 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af00309e0 con 0x7f7af415df00 2024-03-06T07:24:10.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.295+0000 7f7afba46640 1 -- 172.21.15.27:0/3842291091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af415df00 msgr2=0x7f7af40d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.295+0000 7f7afba46640 1 --2- 172.21.15.27:0/3842291091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af415df00 0x7f7af40d25b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7af00097a0 tx=0x7f7af002b960 comp rx=0 tx=0).stop 2024-03-06T07:24:10.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.295+0000 7f7afba46640 1 -- 172.21.15.27:0/3842291091 shutdown_connections 2024-03-06T07:24:10.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.295+0000 7f7afba46640 1 --2- 172.21.15.27:0/3842291091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af415df00 0x7f7af40d25b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.295+0000 7f7afba46640 1 -- 172.21.15.27:0/3842291091 >> 172.21.15.27:0/3842291091 conn(0x7f7af4071960 msgr2=0x7f7af4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.295+0000 7f7afba46640 1 -- 172.21.15.27:0/3842291091 shutdown_connections 2024-03-06T07:24:10.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.295+0000 7f7afba46640 1 -- 172.21.15.27:0/3842291091 wait complete. 2024-03-06T07:24:10.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.296+0000 7f7afba46640 1 Processor -- start 2024-03-06T07:24:10.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.296+0000 7f7afba46640 1 -- start start 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7afba46640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7af40b95a0 0x7f7af415fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7afba46640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7af415df00 0x7f7af40c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7afba46640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af4151f10 0x7f7af40c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7afba46640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af40cec70 con 0x7f7af4151f10 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7afba46640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af40ce750 con 0x7f7af415df00 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7afba46640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af40cedf0 con 0x7f7af40b95a0 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af9fe3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af4151f10 0x7f7af40c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af9fe3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af4151f10 0x7f7af40c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51410/0 (socket says 172.21.15.27:51410) 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af9fe3640 1 -- 172.21.15.27:0/1106694460 learned_addr learned my addr 172.21.15.27:0/1106694460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af8fe1640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7af415df00 0x7f7af40c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af97e2640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7af40b95a0 0x7f7af415fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af9fe3640 1 -- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7af40b95a0 msgr2=0x7f7af415fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af9fe3640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7af40b95a0 0x7f7af415fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af9fe3640 1 -- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7af415df00 msgr2=0x7f7af40c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af9fe3640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7af415df00 0x7f7af40c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.297+0000 7f7af9fe3640 1 -- 172.21.15.27:0/1106694460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af40c9380 con 0x7f7af4151f10 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.298+0000 7f7af9fe3640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af4151f10 0x7f7af40c7cf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c7a0 tx=0x7f7aec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.298+0000 7f7ae9ffb640 1 -- 172.21.15.27:0/1106694460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec018070 con 0x7f7af4151f10 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.298+0000 7f7ae9ffb640 1 -- 172.21.15.27:0/1106694460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec00a640 con 0x7f7af4151f10 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.298+0000 7f7ae9ffb640 1 -- 172.21.15.27:0/1106694460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00aab0 con 0x7f7af4151f10 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.298+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af4160850 con 0x7f7af4151f10 2024-03-06T07:24:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.298+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af40c9630 con 0x7f7af4151f10 2024-03-06T07:24:10.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.299+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af4067ff0 con 0x7f7af4151f10 2024-03-06T07:24:10.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.299+0000 7f7ae9ffb640 1 -- 172.21.15.27:0/1106694460 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7aec01b070 con 0x7f7af4151f10 2024-03-06T07:24:10.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.300+0000 7f7ae9ffb640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ad806b4d0 0x7f7ad806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.300+0000 7f7ae9ffb640 1 -- 172.21.15.27:0/1106694460 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7aec0896d0 con 0x7f7af4151f10 2024-03-06T07:24:10.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.300+0000 7f7af97e2640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ad806b4d0 0x7f7ad806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.301+0000 7f7af97e2640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ad806b4d0 0x7f7ad806d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7af0036000 tx=0x7f7af0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.304+0000 7f7ae9ffb640 1 -- 172.21.15.27:0/1106694460 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7aec057f90 con 0x7f7af4151f10 2024-03-06T07:24:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.523+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af40d3550 con 0x7f7af4151f10 2024-03-06T07:24:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.523+0000 7f7ae9ffb640 1 -- 172.21.15.27:0/1106694460 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f7aec05c030 con 0x7f7af4151f10 2024-03-06T07:24:10.526 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:10.526 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ad806b4d0 msgr2=0x7f7ad806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ad806b4d0 0x7f7ad806d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7af0036000 tx=0x7f7af0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af4151f10 msgr2=0x7f7af40c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af4151f10 0x7f7af40c7cf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c7a0 tx=0x7f7aec00cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 shutdown_connections 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ad806b4d0 0x7f7ad806d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af4151f10 0x7f7af40c7cf0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7af415df00 0x7f7af40c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 --2- 172.21.15.27:0/1106694460 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7af40b95a0 0x7f7af415fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 >> 172.21.15.27:0/1106694460 conn(0x7f7af4071960 msgr2=0x7f7af40d3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 shutdown_connections 2024-03-06T07:24:10.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:10.525+0000 7f7afba46640 1 -- 172.21.15.27:0/1106694460 wait complete. 2024-03-06T07:24:10.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:24:10.949 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:24:10.949 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:24:10.950 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:24:10.950 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:24:10.950 DEBUG:teuthology.orchestra.run.smithi027:> import stat 2024-03-06T07:24:10.950 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:10.950 DEBUG:teuthology.orchestra.run.smithi027:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:24:10.950 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:24:11.017 INFO:teuthology.orchestra.run.smithi027.stdout:1099511627776 2024-03-06T07:24:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:11.026 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:24:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.105+0000 7f994c03cf40 1 Processor -- start 2024-03-06T07:24:11.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.105+0000 7f994c03cf40 1 -- start start 2024-03-06T07:24:11.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f994c03cf40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75eb3b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f994c03cf40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5320 0x55c75eb3bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f994c03cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5700 0x55c75ebdbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f994c03cf40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c75eab3b70 con 0x55c75eb3d590 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f994c03cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c75eab7ad0 con 0x55c75ebe5700 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f994c03cf40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c75eb0e7b0 con 0x55c75ebe5320 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9949401640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75eb3b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9949401640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75eb3b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51416/0 (socket says 172.21.15.27:51416) 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9949401640 1 -- 172.21.15.27:0/1519484498 learned_addr learned my addr 172.21.15.27:0/1519484498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9948c00640 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5320 0x55c75eb3bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9949c02640 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5700 0x55c75ebdbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9949401640 1 -- 172.21.15.27:0/1519484498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5320 msgr2=0x55c75eb3bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9949401640 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5320 0x55c75eb3bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9949401640 1 -- 172.21.15.27:0/1519484498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5700 msgr2=0x55c75ebdbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9949401640 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5700 0x55c75ebdbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9949401640 1 -- 172.21.15.27:0/1519484498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c75ebdef40 con 0x55c75eb3d590 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.106+0000 7f9948c00640 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5320 0x55c75eb3bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.107+0000 7f9949401640 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75eb3b770 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f993c001e40 tx=0x7f993c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393718af3b8faf38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.107+0000 7f993b7fe640 1 -- 172.21.15.27:0/1519484498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c00be30 con 0x55c75eb3d590 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.107+0000 7f993b7fe640 1 -- 172.21.15.27:0/1519484498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c00f400 con 0x55c75eb3d590 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.107+0000 7f993b7fe640 1 -- 172.21.15.27:0/1519484498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c00f7f0 con 0x55c75eb3d590 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.107+0000 7f994c03cf40 1 -- 172.21.15.27:0/1519484498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 msgr2=0x55c75eb3b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.107+0000 7f994c03cf40 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75eb3b770 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f993c001e40 tx=0x7f993c001e70 comp rx=0 tx=0).stop 2024-03-06T07:24:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.108+0000 7f994c03cf40 1 -- 172.21.15.27:0/1519484498 shutdown_connections 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.108+0000 7f994c03cf40 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5700 0x55c75ebdbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.108+0000 7f994c03cf40 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5320 0x55c75eb3bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.108+0000 7f994c03cf40 1 --2- 172.21.15.27:0/1519484498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75eb3b770 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.108+0000 7f994c03cf40 1 -- 172.21.15.27:0/1519484498 >> 172.21.15.27:0/1519484498 conn(0x55c75eb3a9c0 msgr2=0x55c75ebe3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.108+0000 7f994c03cf40 1 -- 172.21.15.27:0/1519484498 shutdown_connections 2024-03-06T07:24:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.108+0000 7f994c03cf40 1 -- 172.21.15.27:0/1519484498 wait complete. 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.109+0000 7f994c03cf40 1 Processor -- start 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.109+0000 7f994c03cf40 1 -- start start 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f994c03cf40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75ebd2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f994c03cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5320 0x55c75ebeb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f994c03cf40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5700 0x55c75ebe9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f994c03cf40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c75eb54210 con 0x55c75eb3d590 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f994c03cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c75ebe3ca0 con 0x55c75ebe5320 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f994c03cf40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c75eb55470 con 0x55c75ebe5700 2024-03-06T07:24:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9949401640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75ebd2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9949401640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75ebd2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51420/0 (socket says 172.21.15.27:51420) 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9949401640 1 -- 172.21.15.27:0/4039847157 learned_addr learned my addr 172.21.15.27:0/4039847157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9948c00640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5320 0x55c75ebeb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9949c02640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5700 0x55c75ebe9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9949401640 1 -- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5700 msgr2=0x55c75ebe9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9949401640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c75ebe5700 0x55c75ebe9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9949401640 1 -- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5320 msgr2=0x55c75ebeb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9949401640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5320 0x55c75ebeb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9949401640 1 -- 172.21.15.27:0/4039847157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c75ebe8fd0 con 0x55c75eb3d590 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.110+0000 7f9948c00640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c75ebe5320 0x55c75ebeb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.111+0000 7f9949401640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75ebd2030 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f993c00fdf0 tx=0x7f993c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.111+0000 7f993a7fc640 1 -- 172.21.15.27:0/4039847157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0012d0 con 0x55c75eb3d590 2024-03-06T07:24:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.111+0000 7f993a7fc640 1 -- 172.21.15.27:0/4039847157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c021020 con 0x55c75eb3d590 2024-03-06T07:24:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.111+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c75ebdf750 con 0x55c75eb3d590 2024-03-06T07:24:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.111+0000 7f993a7fc640 1 -- 172.21.15.27:0/4039847157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c01c650 con 0x55c75eb3d590 2024-03-06T07:24:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.111+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c75ebe9270 con 0x55c75eb3d590 2024-03-06T07:24:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.112+0000 7f993a7fc640 1 -- 172.21.15.27:0/4039847157 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f993c017070 con 0x55c75eb3d590 2024-03-06T07:24:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.113+0000 7f993a7fc640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f992c069eb0 0x7f992c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.113+0000 7f993a7fc640 1 -- 172.21.15.27:0/4039847157 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f993c08a510 con 0x55c75eb3d590 2024-03-06T07:24:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.113+0000 7f9948c00640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f992c069eb0 0x7f992c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.113+0000 7f994c03cf40 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x55c75ebebf00 0x55c75ebea1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.113+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v9 -- 0x55c75ebea720 con 0x55c75ebebf00 2024-03-06T07:24:11.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.113+0000 7f9949c02640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x55c75ebebf00 0x55c75ebea1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.113+0000 7f9948c00640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f992c069eb0 0x7f992c06c370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55c75ebdcaa0 tx=0x7f9940001770 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.113+0000 7f9949c02640 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x55c75ebebf00 0x55c75ebea1e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f9949c02640 1 -- 172.21.15.27:0/4039847157 <== osd.3 v2:172.21.15.27:6824/4277055095 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7f9944052980 con 0x55c75ebebf00 2024-03-06T07:24:11.120 INFO:teuthology.orchestra.run.smithi027.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:24:10.000000+0000, size 0 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x55c75ebebf00 msgr2=0x55c75ebea1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x55c75ebebf00 0x55c75ebea1e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f992c069eb0 msgr2=0x7f992c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f992c069eb0 0x7f992c06c370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55c75ebdcaa0 tx=0x7f9940001770 comp rx=0 tx=0).stop 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 msgr2=0x55c75ebd2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 --2- 172.21.15.27:0/4039847157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c75eb3d590 0x55c75ebd2030 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f993c00fdf0 tx=0x7f993c000bc0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f9949c02640 1 -- 172.21.15.27:0/4039847157 reap_dead start 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 shutdown_connections 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 >> 172.21.15.27:0/4039847157 conn(0x55c75eb3a9c0 msgr2=0x55c75ebe0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 shutdown_connections 2024-03-06T07:24:11.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:11.114+0000 7f994c03cf40 1 -- 172.21.15.27:0/4039847157 wait complete. 2024-03-06T07:24:11.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:11.123 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:24:13.657 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:13.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.763+0000 7f8a02c0e640 1 Processor -- start 2024-03-06T07:24:13.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.763+0000 7f8a02c0e640 1 -- start start 2024-03-06T07:24:13.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a02c0e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89fc15df00 0x7f89fc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a02c0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0b9a30 0x7f89fc0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a02c0e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89fc0cea30 con 0x7f89fc0b95a0 2024-03-06T07:24:13.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a02c0e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89fc0d2af0 con 0x7f89fc15df00 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a02c0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0cf3c0 con 0x7f89fc0b9a30 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a009aa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89fc15df00 0x7f89fc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a009aa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89fc15df00 0x7f89fc0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50244/0 (socket says 172.21.15.27:50244) 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a009aa640 1 -- 172.21.15.27:0/1861448888 learned_addr learned my addr 172.21.15.27:0/1861448888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a011ab640 1 --2- 172.21.15.27:0/1861448888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0b9a30 0x7f89fc0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a009aa640 1 -- 172.21.15.27:0/1861448888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0b9a30 msgr2=0x7f89fc0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a009aa640 1 --2- 172.21.15.27:0/1861448888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0b9a30 0x7f89fc0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a009aa640 1 -- 172.21.15.27:0/1861448888 >> v1:172.21.15.69:6789/0 conn(0x7f89fc0b95a0 legacy=0x7f89fc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.764+0000 7f8a009aa640 1 -- 172.21.15.27:0/1861448888 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc0cf0e0 con 0x7f89fc15df00 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.765+0000 7f8a009aa640 1 --2- 172.21.15.27:0/1861448888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 0x7f89fc0d25b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009870 tx=0x7f89f002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83740450ddc0c326 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.765+0000 7f89fb7fe640 1 -- 172.21.15.27:0/1861448888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0034070 con 0x7f89fc15df00 2024-03-06T07:24:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.765+0000 7f89fb7fe640 1 -- 172.21.15.27:0/1861448888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f0030840 con 0x7f89fc15df00 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.765+0000 7f89fb7fe640 1 -- 172.21.15.27:0/1861448888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0030c10 con 0x7f89fc15df00 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.766+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1861448888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 msgr2=0x7f89fc0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.766+0000 7f8a02c0e640 1 --2- 172.21.15.27:0/1861448888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 0x7f89fc0d25b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009870 tx=0x7f89f002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.766+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1861448888 shutdown_connections 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.766+0000 7f8a02c0e640 1 --2- 172.21.15.27:0/1861448888 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc0b9a30 0x7f89fc0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.766+0000 7f8a02c0e640 1 --2- 172.21.15.27:0/1861448888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 0x7f89fc0d25b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.766+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1861448888 >> 172.21.15.27:0/1861448888 conn(0x7f89fc071960 msgr2=0x7f89fc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.766+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1861448888 shutdown_connections 2024-03-06T07:24:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.766+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1861448888 wait complete. 2024-03-06T07:24:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.767+0000 7f8a02c0e640 1 Processor -- start 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.767+0000 7f8a02c0e640 1 -- start start 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f8a02c0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc0b95a0 0x7f89fc0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f8a02c0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 0x7f89fc0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f8a02c0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc14cf80 0x7f89fc0c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f8a02c0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0cec70 con 0x7f89fc15df00 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f8a02c0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0ce750 con 0x7f89fc0b95a0 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f8a02c0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0cedf0 con 0x7f89fc14cf80 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f89fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 0x7f89fc0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f89fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 0x7f89fc0c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50256/0 (socket says 172.21.15.27:50256) 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f89fbfff640 1 -- 172.21.15.27:0/1025614716 learned_addr learned my addr 172.21.15.27:0/1025614716 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f8a011ab640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc14cf80 0x7f89fc0c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.768+0000 7f8a009aa640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc0b95a0 0x7f89fc0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f89fbfff640 1 -- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc14cf80 msgr2=0x7f89fc0c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f89fbfff640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc14cf80 0x7f89fc0c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f89fbfff640 1 -- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc0b95a0 msgr2=0x7f89fc0cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f89fbfff640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc0b95a0 0x7f89fc0cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f89fbfff640 1 -- 172.21.15.27:0/1025614716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc0cc8e0 con 0x7f89fc15df00 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f89fbfff640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 0x7f89fc0c77b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f89e800c3c0 tx=0x7f89e800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f89f97fa640 1 -- 172.21.15.27:0/1025614716 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e80052d0 con 0x7f89fc15df00 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f89f97fa640 1 -- 172.21.15.27:0/1025614716 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e8005490 con 0x7f89fc15df00 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f89f97fa640 1 -- 172.21.15.27:0/1025614716 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e8007260 con 0x7f89fc15df00 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89fc0c7cf0 con 0x7f89fc15df00 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.769+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89fc0cce60 con 0x7f89fc15df00 2024-03-06T07:24:13.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.770+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89fc067ff0 con 0x7f89fc15df00 2024-03-06T07:24:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.771+0000 7f89f97fa640 1 -- 172.21.15.27:0/1025614716 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89e8007400 con 0x7f89fc15df00 2024-03-06T07:24:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.771+0000 7f89f97fa640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89e006b4d0 0x7f89e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.772+0000 7f89f97fa640 1 -- 172.21.15.27:0/1025614716 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89e80895a0 con 0x7f89fc15df00 2024-03-06T07:24:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.772+0000 7f8a009aa640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89e006b4d0 0x7f89e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.772+0000 7f8a009aa640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89e006b4d0 0x7f89e006d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f89f0036000 tx=0x7f89f0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.776+0000 7f89f97fa640 1 -- 172.21.15.27:0/1025614716 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89e8057e60 con 0x7f89fc15df00 2024-03-06T07:24:13.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.996+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89fc0b5d70 con 0x7f89fc15df00 2024-03-06T07:24:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.997+0000 7f89f97fa640 1 -- 172.21.15.27:0/1025614716 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f89e805ce20 con 0x7f89fc15df00 2024-03-06T07:24:13.999 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:13.999 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:14.000 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.998+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89e006b4d0 msgr2=0x7f89e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.998+0000 7f8a02c0e640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89e006b4d0 0x7f89e006d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f89f0036000 tx=0x7f89f0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 msgr2=0x7f89fc0c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 0x7f89fc0c77b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f89e800c3c0 tx=0x7f89e800c880 comp rx=0 tx=0).stop 2024-03-06T07:24:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 shutdown_connections 2024-03-06T07:24:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89e006b4d0 0x7f89e006d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89fc14cf80 0x7f89fc0c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc15df00 0x7f89fc0c77b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 --2- 172.21.15.27:0/1025614716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89fc0b95a0 0x7f89fc0cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 >> 172.21.15.27:0/1025614716 conn(0x7f89fc071960 msgr2=0x7f89fc0b3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 shutdown_connections 2024-03-06T07:24:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:13.999+0000 7f8a02c0e640 1 -- 172.21.15.27:0/1025614716 wait complete. 2024-03-06T07:24:14.012 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:24:14.327 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:24:14.328 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:24:14.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:14.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.476+0000 7f3f04df4640 1 Processor -- start 2024-03-06T07:24:14.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.476+0000 7f3f04df4640 1 -- start start 2024-03-06T07:24:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.477+0000 7f3f04df4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f00071540 0x7f3f000b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.477+0000 7f3f04df4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f000d3400 0x7f3f00071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.477+0000 7f3f04df4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3f000b2b50 0x7f3f000d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.477+0000 7f3f04df4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000d2f70 con 0x7f3f00071540 2024-03-06T07:24:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.477+0000 7f3f04df4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000d4be0 con 0x7f3f000b2b50 2024-03-06T07:24:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.477+0000 7f3f04df4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00071e60 con 0x7f3f000d3400 2024-03-06T07:24:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efe59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f00071540 0x7f3f000b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efe59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f00071540 0x7f3f000b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50260/0 (socket says 172.21.15.27:50260) 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efe59c640 1 -- 172.21.15.27:0/459565320 learned_addr learned my addr 172.21.15.27:0/459565320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efed9d640 1 --2- 172.21.15.27:0/459565320 >> v2:172.21.15.69:3300/0 conn(0x7f3f000b2b50 0x7f3f000d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efdd9b640 1 --2- 172.21.15.27:0/459565320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f000d3400 0x7f3f00071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efe59c640 1 -- 172.21.15.27:0/459565320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f000d3400 msgr2=0x7f3f00071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efe59c640 1 --2- 172.21.15.27:0/459565320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f000d3400 0x7f3f00071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efe59c640 1 -- 172.21.15.27:0/459565320 >> v2:172.21.15.69:3300/0 conn(0x7f3f000b2b50 msgr2=0x7f3f000d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efe59c640 1 --2- 172.21.15.27:0/459565320 >> v2:172.21.15.69:3300/0 conn(0x7f3f000b2b50 0x7f3f000d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.478+0000 7f3efe59c640 1 -- 172.21.15.27:0/459565320 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f000ce610 con 0x7f3f00071540 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3efe59c640 1 --2- 172.21.15.27:0/459565320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f00071540 0x7f3f000b30a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3f0006c230 tx=0x7f3ef002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae66de760d957031 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3efd59a640 1 -- 172.21.15.27:0/459565320 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0035020 con 0x7f3f00071540 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3efd59a640 1 -- 172.21.15.27:0/459565320 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef0031870 con 0x7f3f00071540 2024-03-06T07:24:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3efd59a640 1 -- 172.21.15.27:0/459565320 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0031c40 con 0x7f3f00071540 2024-03-06T07:24:14.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3f04df4640 1 -- 172.21.15.27:0/459565320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f00071540 msgr2=0x7f3f000b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3f04df4640 1 --2- 172.21.15.27:0/459565320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f00071540 0x7f3f000b30a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3f0006c230 tx=0x7f3ef002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:14.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3f04df4640 1 -- 172.21.15.27:0/459565320 shutdown_connections 2024-03-06T07:24:14.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3f04df4640 1 --2- 172.21.15.27:0/459565320 >> v2:172.21.15.69:3300/0 conn(0x7f3f000b2b50 0x7f3f000d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3f04df4640 1 --2- 172.21.15.27:0/459565320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f000d3400 0x7f3f00071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3f04df4640 1 --2- 172.21.15.27:0/459565320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f00071540 0x7f3f000b30a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.479+0000 7f3f04df4640 1 -- 172.21.15.27:0/459565320 >> 172.21.15.27:0/459565320 conn(0x7f3f000d3ca0 msgr2=0x7f3f000d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.480+0000 7f3f04df4640 1 -- 172.21.15.27:0/459565320 shutdown_connections 2024-03-06T07:24:14.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.480+0000 7f3f04df4640 1 -- 172.21.15.27:0/459565320 wait complete. 2024-03-06T07:24:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.481+0000 7f3f04df4640 1 Processor -- start 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.481+0000 7f3f04df4640 1 -- start start 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3f04df4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f00071540 0x7f3f00148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3f04df4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2b50 0x7f3f000c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3f04df4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f000d3400 0x7f3f0014b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3f04df4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f000ce8f0 con 0x7f3f000b2b50 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3f04df4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f000d2c00 con 0x7f3f000d3400 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3f04df4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0015e370 con 0x7f3f00071540 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2b50 0x7f3f000c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efdd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2b50 0x7f3f000c9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50274/0 (socket says 172.21.15.27:50274) 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efdd9b640 1 -- 172.21.15.27:0/748479256 learned_addr learned my addr 172.21.15.27:0/748479256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efe59c640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f00071540 0x7f3f00148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efed9d640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f000d3400 0x7f3f0014b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efdd9b640 1 -- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f00071540 msgr2=0x7f3f00148e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efdd9b640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f00071540 0x7f3f00148e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efdd9b640 1 -- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f000d3400 msgr2=0x7f3f0014b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efdd9b640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f000d3400 0x7f3f0014b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efdd9b640 1 -- 172.21.15.27:0/748479256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0014be30 con 0x7f3f000b2b50 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efed9d640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f000d3400 0x7f3f0014b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3efdd9b640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2b50 0x7f3f000c9c30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400d5d0 tx=0x7f3ef400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.482+0000 7f3eeeffd640 1 -- 172.21.15.27:0/748479256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4004240 con 0x7f3f000b2b50 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.483+0000 7f3eeeffd640 1 -- 172.21.15.27:0/748479256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef40043e0 con 0x7f3f000b2b50 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.483+0000 7f3eeeffd640 1 -- 172.21.15.27:0/748479256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4003830 con 0x7f3f000b2b50 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.483+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f00149480 con 0x7f3f000b2b50 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.483+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f00149a40 con 0x7f3f000b2b50 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.484+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f00068700 con 0x7f3f000b2b50 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.484+0000 7f3eeeffd640 1 -- 172.21.15.27:0/748479256 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ef40039d0 con 0x7f3f000b2b50 2024-03-06T07:24:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.485+0000 7f3eeeffd640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ee006b500 0x7f3ee006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.485+0000 7f3eeeffd640 1 -- 172.21.15.27:0/748479256 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ef408a400 con 0x7f3f000b2b50 2024-03-06T07:24:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.485+0000 7f3efe59c640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ee006b500 0x7f3ee006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.486+0000 7f3efe59c640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ee006b500 0x7f3ee006d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0036000 tx=0x7f3ef003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.489+0000 7f3eeeffd640 1 -- 172.21.15.27:0/748479256 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ef4059280 con 0x7f3f000b2b50 2024-03-06T07:24:14.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.742+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f000b8b20 con 0x7f3f000b2b50 2024-03-06T07:24:14.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.743+0000 7f3eeeffd640 1 -- 172.21.15.27:0/748479256 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f3ef4058b80 con 0x7f3f000b2b50 2024-03-06T07:24:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:14.746 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:14.746 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ee006b500 msgr2=0x7f3ee006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ee006b500 0x7f3ee006d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0036000 tx=0x7f3ef003c040 comp rx=0 tx=0).stop 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2b50 msgr2=0x7f3f000c9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2b50 0x7f3f000c9c30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400d5d0 tx=0x7f3ef400da90 comp rx=0 tx=0).stop 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 shutdown_connections 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ee006b500 0x7f3ee006d9c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f000d3400 0x7f3f0014b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2b50 0x7f3f000c9c30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 --2- 172.21.15.27:0/748479256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f00071540 0x7f3f00148e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 >> 172.21.15.27:0/748479256 conn(0x7f3f000d3ca0 msgr2=0x7f3f000b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 shutdown_connections 2024-03-06T07:24:14.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.745+0000 7f3f04df4640 1 -- 172.21.15.27:0/748479256 wait complete. 2024-03-06T07:24:14.759 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:24:14.881 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:24:14.882 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:24:14.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:14.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.986+0000 7f167f53e640 1 Processor -- start 2024-03-06T07:24:14.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.986+0000 7f167f53e640 1 -- start start 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.986+0000 7f167f53e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f167806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167f53e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f167806dd60 0x7f16780ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167f53e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16780af080 con 0x7f16780aec40 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167f53e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167806e1c0 con 0x7f167806dd60 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167f53e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1678071b70 con 0x7f1678158520 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167dadb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f16780aec40 0x7f167815e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51604/0 (socket says 172.21.15.27:51604) 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167dadb640 1 -- 172.21.15.27:0/3956290792 learned_addr learned my addr 172.21.15.27:0/3956290792 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167d2da640 1 --2- 172.21.15.27:0/3956290792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f167806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167cad9640 1 --2- 172.21.15.27:0/3956290792 >> v2:172.21.15.69:3300/0 conn(0x7f167806dd60 0x7f16780ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f166ffff640 1 -- 172.21.15.27:0/3956290792 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258697470 0 0) 0x7f16780af080 con 0x7f16780aec40 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f166ffff640 1 -- 172.21.15.27:0/3956290792 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1664003520 con 0x7f16780aec40 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167d2da640 1 -- 172.21.15.27:0/3956290792 >> v2:172.21.15.69:3300/0 conn(0x7f167806dd60 msgr2=0x7f16780ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167d2da640 1 --2- 172.21.15.27:0/3956290792 >> v2:172.21.15.69:3300/0 conn(0x7f167806dd60 0x7f16780ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167d2da640 1 -- 172.21.15.27:0/3956290792 >> v1:172.21.15.27:6789/0 conn(0x7f16780aec40 legacy=0x7f167815e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167d2da640 1 -- 172.21.15.27:0/3956290792 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1678071240 con 0x7f1678158520 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.987+0000 7f167cad9640 1 --2- 172.21.15.27:0/3956290792 >> v2:172.21.15.69:3300/0 conn(0x7f167806dd60 0x7f16780ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f167d2da640 1 --2- 172.21.15.27:0/3956290792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f167806d820 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f16740097a0 tx=0x7f167402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5da0195cfd435284 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f166ffff640 1 -- 172.21.15.27:0/3956290792 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674035020 con 0x7f1678158520 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f166ffff640 1 -- 172.21.15.27:0/3956290792 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1674031830 con 0x7f1678158520 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f166ffff640 1 -- 172.21.15.27:0/3956290792 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674031c00 con 0x7f1678158520 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f167f53e640 1 -- 172.21.15.27:0/3956290792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 msgr2=0x7f167806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f167f53e640 1 --2- 172.21.15.27:0/3956290792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f167806d820 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f16740097a0 tx=0x7f167402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f167f53e640 1 -- 172.21.15.27:0/3956290792 shutdown_connections 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f167f53e640 1 --2- 172.21.15.27:0/3956290792 >> v2:172.21.15.69:3300/0 conn(0x7f167806dd60 0x7f16780ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f167f53e640 1 --2- 172.21.15.27:0/3956290792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f167806d820 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f167f53e640 1 -- 172.21.15.27:0/3956290792 >> 172.21.15.27:0/3956290792 conn(0x7f1678066d00 msgr2=0x7f16780670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f167f53e640 1 -- 172.21.15.27:0/3956290792 shutdown_connections 2024-03-06T07:24:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.989+0000 7f167f53e640 1 -- 172.21.15.27:0/3956290792 wait complete. 2024-03-06T07:24:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.990+0000 7f167f53e640 1 Processor -- start 2024-03-06T07:24:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167f53e640 1 -- start start 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167f53e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f167806dd60 0x7f1678142e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167f53e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780aec40 0x7f167814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167f53e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f16780ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167f53e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16780b4650 con 0x7f16780aec40 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167f53e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1678071800 con 0x7f167806dd60 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167f53e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1678072000 con 0x7f1678158520 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167cad9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780aec40 0x7f167814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167cad9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780aec40 0x7f167814e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50282/0 (socket says 172.21.15.27:50282) 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167cad9640 1 -- 172.21.15.27:0/2308958400 learned_addr learned my addr 172.21.15.27:0/2308958400 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167dadb640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f16780ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.991+0000 7f167d2da640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f167806dd60 0x7f1678142e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f167cad9640 1 -- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 msgr2=0x7f16780ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f167cad9640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f16780ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f167cad9640 1 -- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f167806dd60 msgr2=0x7f1678142e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f167cad9640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f167806dd60 0x7f1678142e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f167cad9640 1 -- 172.21.15.27:0/2308958400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1678145c60 con 0x7f16780aec40 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f167dadb640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f16780ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f167cad9640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780aec40 0x7f167814e460 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f166800ebe0 tx=0x7f166800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f166dffb640 1 -- 172.21.15.27:0/2308958400 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668017070 con 0x7f16780aec40 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f166dffb640 1 -- 172.21.15.27:0/2308958400 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166800cc30 con 0x7f16780aec40 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16781440a0 con 0x7f16780aec40 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f166dffb640 1 -- 172.21.15.27:0/2308958400 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668013470 con 0x7f16780aec40 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.992+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1678144660 con 0x7f16780aec40 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.994+0000 7f166dffb640 1 -- 172.21.15.27:0/2308958400 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f166801a070 con 0x7f16780aec40 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.994+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1678067ff0 con 0x7f16780aec40 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.994+0000 7f166dffb640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f165c06b4d0 0x7f165c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.994+0000 7f166dffb640 1 -- 172.21.15.27:0/2308958400 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f166808a0e0 con 0x7f16780aec40 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.994+0000 7f167d2da640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f165c06b4d0 0x7f165c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.995+0000 7f167d2da640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f165c06b4d0 0x7f165c06d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1674036000 tx=0x7f167403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:14.999+0000 7f166dffb640 1 -- 172.21.15.27:0/2308958400 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1678145c60 con 0x7f16780aec40 2024-03-06T07:24:15.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.216+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16780b31f0 con 0x7f16780aec40 2024-03-06T07:24:15.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.217+0000 7f166dffb640 1 -- 172.21.15.27:0/2308958400 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f1668058910 con 0x7f16780aec40 2024-03-06T07:24:15.219 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:15.220 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:15.220 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f165c06b4d0 msgr2=0x7f165c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f165c06b4d0 0x7f165c06d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1674036000 tx=0x7f167403c040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780aec40 msgr2=0x7f167814e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780aec40 0x7f167814e460 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f166800ebe0 tx=0x7f166800c520 comp rx=0 tx=0).stop 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 shutdown_connections 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f165c06b4d0 0x7f165c06d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1678158520 0x7f16780ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780aec40 0x7f167814e460 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 --2- 172.21.15.27:0/2308958400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f167806dd60 0x7f1678142e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 >> 172.21.15.27:0/2308958400 conn(0x7f1678066d00 msgr2=0x7f1678158a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 shutdown_connections 2024-03-06T07:24:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:15.219+0000 7f167f53e640 1 -- 172.21.15.27:0/2308958400 wait complete. 2024-03-06T07:24:15.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:24:15.355 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_enqueued":315}} 2024-03-06T07:24:15.355 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 315 } } 2024-03-06T07:24:15.355 DEBUG:tasks.ceph_test_case:wait_until_equal: 315 != 1001, waiting (timeout=600)... 2024-03-06T07:24:17.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:17.688 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:20.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.461+0000 7f6eb650c640 1 Processor -- start 2024-03-06T07:24:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.461+0000 7f6eb650c640 1 -- start start 2024-03-06T07:24:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.462+0000 7f6eb650c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00b9a10 0x7f6eb00d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.462+0000 7f6eb650c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb00cea30 con 0x7f6eb015df00 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.462+0000 7f6eb650c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb00cf3c0 con 0x7f6eb00b95a0 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.462+0000 7f6eb650c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00cf0e0 con 0x7f6eb00b9a10 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.462+0000 7f6eaffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6eb015df00 0x7f6eb00d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51616/0 (socket says 172.21.15.27:51616) 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.462+0000 7f6eaffff640 1 -- 172.21.15.27:0/2326540867 learned_addr learned my addr 172.21.15.27:0/2326540867 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561751219 0 0) 0x7f6eb00cea30 con 0x7f6eb015df00 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e98003520 con 0x7f6eb015df00 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eb4aa9640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00b9a10 0x7f6eb00d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 817085590 0 0) 0x7f6e98003520 con 0x7f6eb015df00 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00b9a10 msgr2=0x7f6eb00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00b9a10 0x7f6eb00d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 >> v1:172.21.15.69:6789/0 conn(0x7f6eb00b95a0 legacy=0x7f6eb00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb00ce470 con 0x7f6eb015df00 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f6ea0001670 con 0x7f6eb015df00 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 >> v1:172.21.15.27:6789/0 conn(0x7f6eb015df00 legacy=0x7f6eb00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e980039e0 0x7f6e98003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6e98004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00b9a10 0x7f6e980050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00cf0e0 con 0x7f6e98004390 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9800b9c0 con 0x7f6e980039e0 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9800bbc0 con 0x7f6eb00b9a10 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaf7fe640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6e98004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaffff640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e980039e0 0x7f6e98003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eb4aa9640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00b9a10 0x7f6e980050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaf7fe640 1 -- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00b9a10 msgr2=0x7f6e980050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaf7fe640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00b9a10 0x7f6e980050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaf7fe640 1 -- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e980039e0 msgr2=0x7f6e98003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaf7fe640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e980039e0 0x7f6e98003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eaf7fe640 1 -- 172.21.15.27:0/2326540867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98003520 con 0x7f6e98004390 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.463+0000 7f6eb4aa9640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00b9a10 0x7f6e980050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.464+0000 7f6eaf7fe640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6e98004860 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4001860 tx=0x7f6e98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e28bcd5708a41b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.464+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4038070 con 0x7f6e98004390 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.464+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea4033410 con 0x7f6e98004390 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.464+0000 7f6eaeffd640 1 -- 172.21.15.27:0/2326540867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4001430 con 0x7f6e98004390 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.465+0000 7f6eb650c640 1 -- 172.21.15.27:0/2326540867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 msgr2=0x7f6e98004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.465+0000 7f6eb650c640 1 --2- 172.21.15.27:0/2326540867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6e98004860 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4001860 tx=0x7f6e98005830 comp rx=0 tx=0).stop 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.465+0000 7f6eb4aa9640 1 -- 172.21.15.27:0/2326540867 reap_dead start 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.465+0000 7f6eb650c640 1 -- 172.21.15.27:0/2326540867 shutdown_connections 2024-03-06T07:24:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.465+0000 7f6eb650c640 1 -- 172.21.15.27:0/2326540867 >> 172.21.15.27:0/2326540867 conn(0x7f6eb0071960 msgr2=0x7f6eb0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.465+0000 7f6eb650c640 1 -- 172.21.15.27:0/2326540867 shutdown_connections 2024-03-06T07:24:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.465+0000 7f6eb650c640 1 -- 172.21.15.27:0/2326540867 wait complete. 2024-03-06T07:24:20.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.466+0000 7f6eb650c640 1 Processor -- start 2024-03-06T07:24:20.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.466+0000 7f6eb650c640 1 -- start start 2024-03-06T07:24:20.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eb650c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6eb00cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eb650c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00c9a80 0x7f6eb00c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eb650c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00c7c20 0x7f6eb00c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eb650c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00cedf0 con 0x7f6e98004390 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eb650c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00cec70 con 0x7f6eb00c9a80 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eb650c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00b5e10 con 0x7f6eb00c7c20 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6eb00cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6eb00cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50302/0 (socket says 172.21.15.27:50302) 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaffff640 1 -- 172.21.15.27:0/161849656 learned_addr learned my addr 172.21.15.27:0/161849656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaf7fe640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00c9a80 0x7f6eb00c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eb4aa9640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00c7c20 0x7f6eb00c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaffff640 1 -- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00c7c20 msgr2=0x7f6eb00c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaffff640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00c7c20 0x7f6eb00c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaffff640 1 -- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00c9a80 msgr2=0x7f6eb00c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaffff640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00c9a80 0x7f6eb00c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaffff640 1 -- 172.21.15.27:0/161849656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb00c89e0 con 0x7f6e98004390 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaf7fe640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00c9a80 0x7f6eb00c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.467+0000 7f6eaffff640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6eb00cad50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007340 tx=0x7f6ea0007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.468+0000 7f6eacff9640 1 -- 172.21.15.27:0/161849656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea001a070 con 0x7f6e98004390 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.468+0000 7f6eacff9640 1 -- 172.21.15.27:0/161849656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea000f860 con 0x7f6e98004390 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.468+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb00c8cc0 con 0x7f6e98004390 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.468+0000 7f6eacff9640 1 -- 172.21.15.27:0/161849656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000fc30 con 0x7f6e98004390 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.468+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb00c9280 con 0x7f6e98004390 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.469+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb0067ff0 con 0x7f6e98004390 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.469+0000 7f6eacff9640 1 -- 172.21.15.27:0/161849656 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ea001d070 con 0x7f6e98004390 2024-03-06T07:24:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.470+0000 7f6eacff9640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e9c06b4d0 0x7f6e9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.470+0000 7f6eacff9640 1 -- 172.21.15.27:0/161849656 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ea0089ab0 con 0x7f6e98004390 2024-03-06T07:24:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.470+0000 7f6eaf7fe640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e9c06b4d0 0x7f6e9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.470+0000 7f6eaf7fe640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e9c06b4d0 0x7f6e9c06d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4001860 tx=0x7f6ea4009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.474+0000 7f6eacff9640 1 -- 172.21.15.27:0/161849656 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ea00589c0 con 0x7f6e98004390 2024-03-06T07:24:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.694+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6eb00cf6a0 con 0x7f6e98004390 2024-03-06T07:24:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.695+0000 7f6eacff9640 1 -- 172.21.15.27:0/161849656 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f6ea00582c0 con 0x7f6e98004390 2024-03-06T07:24:20.696 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:20.696 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e9c06b4d0 msgr2=0x7f6e9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e9c06b4d0 0x7f6e9c06d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4001860 tx=0x7f6ea4009030 comp rx=0 tx=0).stop 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 msgr2=0x7f6eb00cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6eb00cad50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007340 tx=0x7f6ea0007800 comp rx=0 tx=0).stop 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 shutdown_connections 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e9c06b4d0 0x7f6e9c06d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6eb00c7c20 0x7f6eb00c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb00c9a80 0x7f6eb00c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 --2- 172.21.15.27:0/161849656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e98004390 0x7f6eb00cad50 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 >> 172.21.15.27:0/161849656 conn(0x7f6eb0071960 msgr2=0x7f6eb015d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 shutdown_connections 2024-03-06T07:24:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.697+0000 7f6eb650c640 1 -- 172.21.15.27:0/161849656 wait complete. 2024-03-06T07:24:20.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:24:20.832 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:24:20.832 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:24:20.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:20.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.938+0000 7fa6db59e640 1 Processor -- start 2024-03-06T07:24:20.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.938+0000 7fa6db59e640 1 -- start start 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6db59e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6dc0ce9f0 0x7fa6dc0d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6db59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6dc06bb40 0x7fa6dc06bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6db59e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6dc0cfea0 con 0x7fa6dc1619c0 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6db59e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6dc0d0180 con 0x7fa6dc06bb40 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6db59e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6dc0d0460 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6d3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6dc06bb40 0x7fa6dc06bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6dad9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6dc1619c0 0x7fa6dc15f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51622/0 (socket says 172.21.15.27:51622) 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6dad9d640 1 -- 172.21.15.27:0/1605519841 learned_addr learned my addr 172.21.15.27:0/1605519841 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6da59c640 1 --2- 172.21.15.27:0/1605519841 >> v2:172.21.15.69:3300/0 conn(0x7fa6dc0ce9f0 0x7fa6dc0d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6d3fff640 1 -- 172.21.15.27:0/1605519841 >> v2:172.21.15.69:3300/0 conn(0x7fa6dc0ce9f0 msgr2=0x7fa6dc0d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6d3fff640 1 --2- 172.21.15.27:0/1605519841 >> v2:172.21.15.69:3300/0 conn(0x7fa6dc0ce9f0 0x7fa6dc0d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6d3fff640 1 -- 172.21.15.27:0/1605519841 >> v1:172.21.15.27:6789/0 conn(0x7fa6dc1619c0 legacy=0x7fa6dc15f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.939+0000 7fa6d3fff640 1 -- 172.21.15.27:0/1605519841 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc0d0740 con 0x7fa6dc06bb40 2024-03-06T07:24:20.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.940+0000 7fa6d3fff640 1 --2- 172.21.15.27:0/1605519841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc06bb40 0x7fa6dc06bf80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc071000 tx=0x7fa6c402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8086ea82315e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.940+0000 7fa6d9d9b640 1 -- 172.21.15.27:0/1605519841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4034070 con 0x7fa6dc06bb40 2024-03-06T07:24:20.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.940+0000 7fa6d9d9b640 1 -- 172.21.15.27:0/1605519841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c4030610 con 0x7fa6dc06bb40 2024-03-06T07:24:20.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.940+0000 7fa6d9d9b640 1 -- 172.21.15.27:0/1605519841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40309e0 con 0x7fa6dc06bb40 2024-03-06T07:24:20.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.941+0000 7fa6db59e640 1 -- 172.21.15.27:0/1605519841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc06bb40 msgr2=0x7fa6dc06bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.941+0000 7fa6db59e640 1 --2- 172.21.15.27:0/1605519841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc06bb40 0x7fa6dc06bf80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc071000 tx=0x7fa6c402b960 comp rx=0 tx=0).stop 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.941+0000 7fa6db59e640 1 -- 172.21.15.27:0/1605519841 shutdown_connections 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.941+0000 7fa6db59e640 1 --2- 172.21.15.27:0/1605519841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc06bb40 0x7fa6dc06bf80 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.941+0000 7fa6db59e640 1 --2- 172.21.15.27:0/1605519841 >> v2:172.21.15.69:3300/0 conn(0x7fa6dc0ce9f0 0x7fa6dc0d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.941+0000 7fa6db59e640 1 -- 172.21.15.27:0/1605519841 >> 172.21.15.27:0/1605519841 conn(0x7fa6dc0b3430 msgr2=0x7fa6dc0b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.941+0000 7fa6db59e640 1 -- 172.21.15.27:0/1605519841 shutdown_connections 2024-03-06T07:24:20.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.941+0000 7fa6db59e640 1 -- 172.21.15.27:0/1605519841 wait complete. 2024-03-06T07:24:20.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6db59e640 1 Processor -- start 2024-03-06T07:24:20.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6db59e640 1 -- start start 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6db59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6dc06bb40 0x7fa6dc160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6db59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc0ce9f0 0x7fa6dc1f77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6db59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6dc1619c0 0x7fa6dc1f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6db59e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc161e30 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6db59e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc162720 con 0x7fa6dc06bb40 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6db59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc161fb0 con 0x7fa6dc1619c0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6d3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc0ce9f0 0x7fa6dc1f77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6d3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc0ce9f0 0x7fa6dc1f77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50324/0 (socket says 172.21.15.27:50324) 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.943+0000 7fa6d3fff640 1 -- 172.21.15.27:0/662469979 learned_addr learned my addr 172.21.15.27:0/662469979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6dad9d640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6dc1619c0 0x7fa6dc1f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6da59c640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6dc06bb40 0x7fa6dc160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6d3fff640 1 -- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6dc1619c0 msgr2=0x7fa6dc1f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6d3fff640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6dc1619c0 0x7fa6dc1f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6d3fff640 1 -- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6dc06bb40 msgr2=0x7fa6dc160810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6d3fff640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6dc06bb40 0x7fa6dc160810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6d3fff640 1 -- 172.21.15.27:0/662469979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc09e820 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6d3fff640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc0ce9f0 0x7fa6dc1f77a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009880 tx=0x7fa6c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6d2ffd640 1 -- 172.21.15.27:0/662469979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4046020 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6d2ffd640 1 -- 172.21.15.27:0/662469979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c4039a10 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6dc09dc10 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6d2ffd640 1 -- 172.21.15.27:0/662469979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4034040 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.944+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6dc09e1d0 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.945+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6dc04f330 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.946+0000 7fa6d2ffd640 1 -- 172.21.15.27:0/662469979 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6c4037070 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.947+0000 7fa6d2ffd640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.947+0000 7fa6d2ffd640 1 -- 172.21.15.27:0/662469979 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6c40acc20 con 0x7fa6dc0ce9f0 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.947+0000 7fa6da59c640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.947+0000 7fa6da59c640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc006fd0 tx=0x7fa6cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:20.950+0000 7fa6d2ffd640 1 -- 172.21.15.27:0/662469979 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6c4050020 con 0x7fa6dc0ce9f0 2024-03-06T07:24:21.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.220+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6dc072b10 con 0x7fa6dc0ce9f0 2024-03-06T07:24:21.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.221+0000 7fa6d2ffd640 1 -- 172.21.15.27:0/662469979 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fa6dc09e820 con 0x7fa6dc0ce9f0 2024-03-06T07:24:21.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:21.223 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:21.223 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:21.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6bc06b4d0 msgr2=0x7fa6bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc006fd0 tx=0x7fa6cc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc0ce9f0 msgr2=0x7fa6dc1f77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc0ce9f0 0x7fa6dc1f77a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009880 tx=0x7fa6c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 shutdown_connections 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6dc1619c0 0x7fa6dc1f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6dc0ce9f0 0x7fa6dc1f77a0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 --2- 172.21.15.27:0/662469979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6dc06bb40 0x7fa6dc160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 >> 172.21.15.27:0/662469979 conn(0x7fa6dc0b3430 msgr2=0x7fa6dc06e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 shutdown_connections 2024-03-06T07:24:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.223+0000 7fa6db59e640 1 -- 172.21.15.27:0/662469979 wait complete. 2024-03-06T07:24:21.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:24:21.358 INFO:teuthology.orchestra.run.smithi027.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:24:21.358 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:24:21.359 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:24:21.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.399+0000 7f132b237f40 1 Processor -- start 2024-03-06T07:24:21.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.399+0000 7f132b237f40 1 -- start start 2024-03-06T07:24:21.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.400+0000 7f132b237f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525ea9e70 0x563525f7cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.400+0000 7f132b237f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563525eae2f0 0x563525ed9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.400+0000 7f132b237f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.400+0000 7f132b237f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563525eaa7c0 con 0x563525ef59b0 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.400+0000 7f132b237f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x563525e52b70 con 0x563525ea9e70 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.400+0000 7f132b237f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x563525ef4d30 con 0x563525eae2f0 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.400+0000 7f1328e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50330/0 (socket says 172.21.15.27:50330) 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328e25640 1 -- 172.21.15.27:0/1515842420 learned_addr learned my addr 172.21.15.27:0/1515842420 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1327e23640 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563525eae2f0 0x563525ed9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328624640 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525ea9e70 0x563525f7cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328e25640 1 -- 172.21.15.27:0/1515842420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563525eae2f0 msgr2=0x563525ed9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328e25640 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563525eae2f0 0x563525ed9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328e25640 1 -- 172.21.15.27:0/1515842420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525ea9e70 msgr2=0x563525f7cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328e25640 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525ea9e70 0x563525f7cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328e25640 1 -- 172.21.15.27:0/1515842420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563525ef6270 con 0x563525ef59b0 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328624640 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525ea9e70 0x563525f7cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1328e25640 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed9ad0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f132001e140 tx=0x7f1320051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf6966bce4287a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1326e21640 1 -- 172.21.15.27:0/1515842420 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132005b040 con 0x563525ef59b0 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1326e21640 1 -- 172.21.15.27:0/1515842420 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1320066070 con 0x563525ef59b0 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.401+0000 7f1326e21640 1 -- 172.21.15.27:0/1515842420 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132004ee60 con 0x563525ef59b0 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.402+0000 7f132b237f40 1 -- 172.21.15.27:0/1515842420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 msgr2=0x563525ed9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.402+0000 7f132b237f40 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed9ad0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f132001e140 tx=0x7f1320051000 comp rx=0 tx=0).stop 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.402+0000 7f132b237f40 1 -- 172.21.15.27:0/1515842420 shutdown_connections 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.402+0000 7f132b237f40 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed9ad0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.402+0000 7f132b237f40 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563525eae2f0 0x563525ed9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.402+0000 7f132b237f40 1 --2- 172.21.15.27:0/1515842420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525ea9e70 0x563525f7cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.402+0000 7f132b237f40 1 -- 172.21.15.27:0/1515842420 >> 172.21.15.27:0/1515842420 conn(0x563525eab5e0 msgr2=0x563525ed6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.402+0000 7f132b237f40 1 -- 172.21.15.27:0/1515842420 shutdown_connections 2024-03-06T07:24:21.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.402+0000 7f132b237f40 1 -- 172.21.15.27:0/1515842420 wait complete. 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f132b237f40 1 Processor -- start 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f132b237f40 1 -- start start 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f132b237f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563525ea9e70 0x563525f68860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f132b237f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525eae2f0 0x563525f7c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f132b237f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f132b237f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563525f62510 con 0x563525ef59b0 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f132b237f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x563525f60d00 con 0x563525eae2f0 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f132b237f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x563525f60fe0 con 0x563525ea9e70 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f1328e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f1328e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50340/0 (socket says 172.21.15.27:50340) 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.403+0000 7f1328e25640 1 -- 172.21.15.27:0/1920243184 learned_addr learned my addr 172.21.15.27:0/1920243184 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1328624640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563525ea9e70 0x563525f68860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1327e23640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525eae2f0 0x563525f7c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1328e25640 1 -- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563525ea9e70 msgr2=0x563525f68860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1328e25640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x563525ea9e70 0x563525f68860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1328e25640 1 -- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525eae2f0 msgr2=0x563525f7c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1328e25640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525eae2f0 0x563525f7c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1328e25640 1 -- 172.21.15.27:0/1920243184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563525f612c0 con 0x563525ef59b0 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1327e23640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563525eae2f0 0x563525f7c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1328e25640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed6f60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1320063040 tx=0x7f1320003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1325e1f640 1 -- 172.21.15.27:0/1920243184 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1320061730 con 0x563525ef59b0 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1325e1f640 1 -- 172.21.15.27:0/1920243184 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13200618f0 con 0x563525ef59b0 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563525f615a0 con 0x563525ef59b0 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f1325e1f640 1 -- 172.21.15.27:0/1920243184 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1320061ce0 con 0x563525ef59b0 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.404+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563525f608b0 con 0x563525ef59b0 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.405+0000 7f1325e1f640 1 -- 172.21.15.27:0/1920243184 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1320056420 con 0x563525ef59b0 2024-03-06T07:24:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.406+0000 7f1325e1f640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1314069eb0 0x7f131406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.406+0000 7f1325e1f640 1 -- 172.21.15.27:0/1920243184 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13200cf620 con 0x563525ef59b0 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.406+0000 7f132b237f40 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x563525f7c580 0x563525f5fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.406+0000 7f1328624640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1314069eb0 0x7f131406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.406+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v9 -- 0x563525f68da0 con 0x563525f7c580 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.406+0000 7f1327e23640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x563525f7c580 0x563525f5fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f1328624640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1314069eb0 0x7f131406c370 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1318000c00 tx=0x7f1318001880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f1327e23640 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x563525f7c580 0x563525f5fba0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f1327e23640 1 -- 172.21.15.27:0/1920243184 <== osd.3 v2:172.21.15.27:6824/4277055095 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f131c009a30 con 0x563525f7c580 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x563525f7c580 msgr2=0x563525f5fba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x563525f7c580 0x563525f5fba0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1314069eb0 msgr2=0x7f131406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1314069eb0 0x7f131406c370 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1318000c00 tx=0x7f1318001880 comp rx=0 tx=0).stop 2024-03-06T07:24:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 msgr2=0x563525ed6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 --2- 172.21.15.27:0/1920243184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563525ef59b0 0x563525ed6f60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1320063040 tx=0x7f1320003330 comp rx=0 tx=0).stop 2024-03-06T07:24:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f1328e25640 1 -- 172.21.15.27:0/1920243184 reap_dead start 2024-03-06T07:24:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 shutdown_connections 2024-03-06T07:24:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 >> 172.21.15.27:0/1920243184 conn(0x563525eab5e0 msgr2=0x563525f63a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 shutdown_connections 2024-03-06T07:24:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.407+0000 7f132b237f40 1 -- 172.21.15.27:0/1920243184 wait complete. 2024-03-06T07:24:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:21.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:24:21.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.559+0000 7febbe064640 1 Processor -- start 2024-03-06T07:24:21.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.559+0000 7febbe064640 1 -- start start 2024-03-06T07:24:21.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febbe064640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febb8158520 0x7febb806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febbe064640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb806dd60 0x7febb80ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febbe064640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7febb80aec40 0x7febb815e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febbe064640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb806e1c0 con 0x7febb80aec40 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febbe064640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb8071b70 con 0x7febb8158520 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febbe064640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb80af080 con 0x7febb806dd60 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb7fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7febb80aec40 0x7febb815e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb7fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7febb80aec40 0x7febb815e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50352/0 (socket says 172.21.15.27:50352) 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb7fff640 1 -- 172.21.15.27:0/2940802599 learned_addr learned my addr 172.21.15.27:0/2940802599 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb6ffd640 1 --2- 172.21.15.27:0/2940802599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb806dd60 0x7febb80ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb7fff640 1 -- 172.21.15.27:0/2940802599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb806dd60 msgr2=0x7febb80ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb7fff640 1 --2- 172.21.15.27:0/2940802599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb806dd60 0x7febb80ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb77fe640 1 --2- 172.21.15.27:0/2940802599 >> v2:172.21.15.69:3300/0 conn(0x7febb8158520 0x7febb806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb7fff640 1 -- 172.21.15.27:0/2940802599 >> v2:172.21.15.69:3300/0 conn(0x7febb8158520 msgr2=0x7febb806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb7fff640 1 --2- 172.21.15.27:0/2940802599 >> v2:172.21.15.69:3300/0 conn(0x7febb8158520 0x7febb806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb7fff640 1 -- 172.21.15.27:0/2940802599 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8071240 con 0x7febb80aec40 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb6ffd640 1 --2- 172.21.15.27:0/2940802599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb806dd60 0x7febb80ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.560+0000 7febb77fe640 1 --2- 172.21.15.27:0/2940802599 >> v2:172.21.15.69:3300/0 conn(0x7febb8158520 0x7febb806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.561+0000 7febb7fff640 1 --2- 172.21.15.27:0/2940802599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb80aec40 0x7febb815e670 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7febb806e900 tx=0x7febb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53628c38883ecdf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.561+0000 7febb67fc640 1 -- 172.21.15.27:0/2940802599 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb0035020 con 0x7febb80aec40 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.561+0000 7febb67fc640 1 -- 172.21.15.27:0/2940802599 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb0031870 con 0x7febb80aec40 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.561+0000 7febb67fc640 1 -- 172.21.15.27:0/2940802599 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb0031c40 con 0x7febb80aec40 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.562+0000 7febbe064640 1 -- 172.21.15.27:0/2940802599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb80aec40 msgr2=0x7febb815e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.562+0000 7febbe064640 1 --2- 172.21.15.27:0/2940802599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb80aec40 0x7febb815e670 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7febb806e900 tx=0x7febb002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.562+0000 7febbe064640 1 -- 172.21.15.27:0/2940802599 shutdown_connections 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.562+0000 7febbe064640 1 --2- 172.21.15.27:0/2940802599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb80aec40 0x7febb815e670 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.562+0000 7febbe064640 1 --2- 172.21.15.27:0/2940802599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb806dd60 0x7febb80ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.562+0000 7febbe064640 1 --2- 172.21.15.27:0/2940802599 >> v2:172.21.15.69:3300/0 conn(0x7febb8158520 0x7febb806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.562+0000 7febbe064640 1 -- 172.21.15.27:0/2940802599 >> 172.21.15.27:0/2940802599 conn(0x7febb8066d00 msgr2=0x7febb80670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.562+0000 7febbe064640 1 -- 172.21.15.27:0/2940802599 shutdown_connections 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.562+0000 7febbe064640 1 -- 172.21.15.27:0/2940802599 wait complete. 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.563+0000 7febbe064640 1 Processor -- start 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.563+0000 7febbe064640 1 -- start start 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febbe064640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb806dd60 0x7febb8144c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febbe064640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb80aec40 0x7febb81574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febbe064640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb8158520 0x7febb80ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febbe064640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb8072000 con 0x7febb806dd60 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febbe064640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb8071800 con 0x7febb80aec40 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febbe064640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb8158900 con 0x7febb8158520 2024-03-06T07:24:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb806dd60 0x7febb8144c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb806dd60 0x7febb8144c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50356/0 (socket says 172.21.15.27:50356) 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb77fe640 1 -- 172.21.15.27:0/38069645 learned_addr learned my addr 172.21.15.27:0/38069645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb6ffd640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb80aec40 0x7febb81574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb7fff640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb8158520 0x7febb80ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb77fe640 1 -- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb8158520 msgr2=0x7febb80ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb77fe640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb8158520 0x7febb80ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb77fe640 1 -- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb80aec40 msgr2=0x7febb81574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb77fe640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb80aec40 0x7febb81574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.564+0000 7febb77fe640 1 -- 172.21.15.27:0/38069645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8154650 con 0x7febb806dd60 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.565+0000 7febb77fe640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb806dd60 0x7febb8144c00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7feba800ce30 tx=0x7feba800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.565+0000 7feb9ffff640 1 -- 172.21.15.27:0/38069645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba800f890 con 0x7febb806dd60 2024-03-06T07:24:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.565+0000 7feb9ffff640 1 -- 172.21.15.27:0/38069645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba800fa30 con 0x7febb806dd60 2024-03-06T07:24:21.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.565+0000 7feb9ffff640 1 -- 172.21.15.27:0/38069645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba800fe00 con 0x7febb806dd60 2024-03-06T07:24:21.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.565+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb8154930 con 0x7febb806dd60 2024-03-06T07:24:21.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.565+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb8154ef0 con 0x7febb806dd60 2024-03-06T07:24:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.566+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb8067ff0 con 0x7febb806dd60 2024-03-06T07:24:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.567+0000 7feb9ffff640 1 -- 172.21.15.27:0/38069645 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feba8018400 con 0x7febb806dd60 2024-03-06T07:24:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.567+0000 7feb9ffff640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feba406b4d0 0x7feba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.567+0000 7feb9ffff640 1 -- 172.21.15.27:0/38069645 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feba808adc0 con 0x7febb806dd60 2024-03-06T07:24:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.568+0000 7febb6ffd640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feba406b4d0 0x7feba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.568+0000 7febb6ffd640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feba406b4d0 0x7feba406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7febb806d190 tx=0x7febac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.572+0000 7feb9ffff640 1 -- 172.21.15.27:0/38069645 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feba8014030 con 0x7febb806dd60 2024-03-06T07:24:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.822+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7febb80b31f0 con 0x7febb806dd60 2024-03-06T07:24:21.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.823+0000 7feb9ffff640 1 -- 172.21.15.27:0/38069645 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1695 (secure 0 0 0) 0x7feba800ae20 con 0x7febb806dd60 2024-03-06T07:24:21.824 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "total_bytes": 773094113280, 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "total_avail_bytes": 772821311488, 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_bytes": 272801792, 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_bytes": 272801792, 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_ratio": 0.00035287009086459875, 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "num_per_pool_osds": 8, 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "stats_by_class": { 2024-03-06T07:24:21.825 INFO:teuthology.orchestra.run.smithi027.stdout: "ssd": { 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "total_bytes": 773094113280, 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "total_avail_bytes": 772821311488, 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_bytes": 272801792, 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_bytes": 272801792, 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_ratio": 0.00035287009086459875 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "name": ".mgr", 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 1, 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:24:21.826 INFO:teuthology.orchestra.run.smithi027.stdout: "stored": 590368, 2024-03-06T07:24:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: "objects": 2, 2024-03-06T07:24:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: "kb_used": 1160, 2024-03-06T07:24:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 1187840, 2024-03-06T07:24:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: "percent_used": 1.6180507600438432e-06, 2024-03-06T07:24:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: "max_avail": 367058354176 2024-03-06T07:24:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:21.827 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs_metadata", 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: "stored": 3008872, 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: "objects": 24, 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: "kb_used": 5936, 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 6078464, 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: "percent_used": 8.2799006122513674e-06, 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: "max_avail": 367058354176 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:21.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs_data", 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: "stored": 701440, 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: "objects": 685, 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: "kb_used": 5480, 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 5611520, 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: "percent_used": 7.6438482210505754e-06, 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: "max_avail": 367058354176 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:24:21.829 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feba406b4d0 msgr2=0x7feba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feba406b4d0 0x7feba406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7febb806d190 tx=0x7febac008040 comp rx=0 tx=0).stop 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb806dd60 msgr2=0x7febb8144c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb806dd60 0x7febb8144c00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7feba800ce30 tx=0x7feba800a650 comp rx=0 tx=0).stop 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 shutdown_connections 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feba406b4d0 0x7feba406d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febb8158520 0x7febb80ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb80aec40 0x7febb81574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 --2- 172.21.15.27:0/38069645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb806dd60 0x7febb8144c00 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 >> 172.21.15.27:0/38069645 conn(0x7febb8066d00 msgr2=0x7febb80670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 shutdown_connections 2024-03-06T07:24:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:21.825+0000 7febbe064640 1 -- 172.21.15.27:0/38069645 wait complete. 2024-03-06T07:24:21.839 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-06T07:24:26.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:24:26.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.945+0000 7f1425b25640 1 Processor -- start 2024-03-06T07:24:26.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.945+0000 7f1425b25640 1 -- start start 2024-03-06T07:24:26.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.946+0000 7f1425b25640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14200b95a0 0x7f14200b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.946+0000 7f1425b25640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f14200b9a10 0x7f14200b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.946+0000 7f1425b25640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14200cea30 con 0x7f142015df00 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.946+0000 7f1425b25640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14200cf3c0 con 0x7f14200b95a0 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.946+0000 7f1425b25640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14200cf0e0 con 0x7f14200b9a10 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.946+0000 7f1417fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14200b95a0 0x7f14200b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.946+0000 7f141f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f142015df00 0x7f14200d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42910/0 (socket says 172.21.15.27:42910) 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.946+0000 7f141f7fe640 1 -- 172.21.15.27:0/1494837833 learned_addr learned my addr 172.21.15.27:0/1494837833 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.946+0000 7f141ffff640 1 --2- 172.21.15.27:0/1494837833 >> v2:172.21.15.69:3300/0 conn(0x7f14200b9a10 0x7f14200b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.947+0000 7f1417fff640 1 -- 172.21.15.27:0/1494837833 >> v2:172.21.15.69:3300/0 conn(0x7f14200b9a10 msgr2=0x7f14200b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.947+0000 7f1417fff640 1 --2- 172.21.15.27:0/1494837833 >> v2:172.21.15.69:3300/0 conn(0x7f14200b9a10 0x7f14200b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.947+0000 7f1417fff640 1 -- 172.21.15.27:0/1494837833 >> v1:172.21.15.27:6789/0 conn(0x7f142015df00 legacy=0x7f14200d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.947+0000 7f1417fff640 1 -- 172.21.15.27:0/1494837833 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14200ce470 con 0x7f14200b95a0 2024-03-06T07:24:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.947+0000 7f141effd640 1 -- 172.21.15.27:0/1494837833 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673650714 0 0) 0x7f14200cea30 con 0x7f142015df00 2024-03-06T07:24:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.947+0000 7f1417fff640 1 --2- 172.21.15.27:0/1494837833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 0x7f14200b5700 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f14200d4d20 tx=0x7f140802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ccb7782510bddb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:26.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f141effd640 1 -- 172.21.15.27:0/1494837833 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1408035020 con 0x7f14200b95a0 2024-03-06T07:24:26.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f141effd640 1 -- 172.21.15.27:0/1494837833 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1408031830 con 0x7f14200b95a0 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f141effd640 1 -- 172.21.15.27:0/1494837833 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1408031c00 con 0x7f14200b95a0 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f1425b25640 1 -- 172.21.15.27:0/1494837833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 msgr2=0x7f14200b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f1425b25640 1 --2- 172.21.15.27:0/1494837833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 0x7f14200b5700 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f14200d4d20 tx=0x7f140802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f1425b25640 1 -- 172.21.15.27:0/1494837833 shutdown_connections 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f1425b25640 1 --2- 172.21.15.27:0/1494837833 >> v2:172.21.15.69:3300/0 conn(0x7f14200b9a10 0x7f14200b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f1425b25640 1 --2- 172.21.15.27:0/1494837833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 0x7f14200b5700 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f1425b25640 1 -- 172.21.15.27:0/1494837833 >> 172.21.15.27:0/1494837833 conn(0x7f1420071960 msgr2=0x7f1420071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f1425b25640 1 -- 172.21.15.27:0/1494837833 shutdown_connections 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.948+0000 7f1425b25640 1 -- 172.21.15.27:0/1494837833 wait complete. 2024-03-06T07:24:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.949+0000 7f1425b25640 1 Processor -- start 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.949+0000 7f1425b25640 1 -- start start 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.950+0000 7f1425b25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 0x7f14200cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.950+0000 7f1425b25640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14200b9a10 0x7f14201584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.950+0000 7f1425b25640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f142015df00 0x7f14201f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.950+0000 7f1425b25640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200cedf0 con 0x7f14200b95a0 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.950+0000 7f1425b25640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200cec70 con 0x7f142015df00 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.950+0000 7f1425b25640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200d3220 con 0x7f14200b9a10 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.950+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 0x7f14200cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.950+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 0x7f14200cb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40122/0 (socket says 172.21.15.27:40122) 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.950+0000 7f141f7fe640 1 -- 172.21.15.27:0/2360536452 learned_addr learned my addr 172.21.15.27:0/2360536452 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141ffff640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f142015df00 0x7f14201f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f1417fff640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14200b9a10 0x7f14201584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141f7fe640 1 -- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14200b9a10 msgr2=0x7f14201584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141f7fe640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14200b9a10 0x7f14201584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141f7fe640 1 -- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f142015df00 msgr2=0x7f14201f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141f7fe640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f142015df00 0x7f14201f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141f7fe640 1 -- 172.21.15.27:0/2360536452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14201f4310 con 0x7f14200b95a0 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141ffff640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f142015df00 0x7f14201f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f1417fff640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14200b9a10 0x7f14201584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141f7fe640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 0x7f14200cb680 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f141000c430 tx=0x7f141000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141cff9640 1 -- 172.21.15.27:0/2360536452 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14100074e0 con 0x7f14200b95a0 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141cff9640 1 -- 172.21.15.27:0/2360536452 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14100047e0 con 0x7f14200b95a0 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14201f45f0 con 0x7f14200b95a0 2024-03-06T07:24:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f141cff9640 1 -- 172.21.15.27:0/2360536452 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410004bd0 con 0x7f14200b95a0 2024-03-06T07:24:26.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.951+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14200cb160 con 0x7f14200b95a0 2024-03-06T07:24:26.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.952+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14200cb360 con 0x7f14200b95a0 2024-03-06T07:24:26.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.953+0000 7f141cff9640 1 -- 172.21.15.27:0/2360536452 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1410011070 con 0x7f14200b95a0 2024-03-06T07:24:26.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.953+0000 7f141cff9640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f140006b4d0 0x7f140006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:26.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.953+0000 7f141cff9640 1 -- 172.21.15.27:0/2360536452 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1410089e30 con 0x7f14200b95a0 2024-03-06T07:24:26.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.953+0000 7f1417fff640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f140006b4d0 0x7f140006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:26.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.954+0000 7f1417fff640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f140006b4d0 0x7f140006d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1408036000 tx=0x7f1408008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:26.957+0000 7f141cff9640 1 -- 172.21.15.27:0/2360536452 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1410058d40 con 0x7f14200b95a0 2024-03-06T07:24:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.212+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f1420067ff0 con 0x7f14200b95a0 2024-03-06T07:24:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.213+0000 7f141cff9640 1 -- 172.21.15.27:0/2360536452 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1657 (secure 0 0 0) 0x7f1410058640 con 0x7f14200b95a0 2024-03-06T07:24:27.214 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:27.214 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:24:27.214 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "total_bytes": 773094113280, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "total_avail_bytes": 772831199232, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_bytes": 262914048, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_bytes": 262914048, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_ratio": 0.00034008026705123484, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "num_per_pool_osds": 8, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "stats_by_class": { 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "ssd": { 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "total_bytes": 773094113280, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "total_avail_bytes": 772831199232, 2024-03-06T07:24:27.215 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_bytes": 262914048, 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_bytes": 262914048, 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_ratio": 0.00034008026705123484 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: "name": ".mgr", 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 1, 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: "stored": 590368, 2024-03-06T07:24:27.216 INFO:teuthology.orchestra.run.smithi027.stdout: "objects": 2, 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "kb_used": 1160, 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 1187840, 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "percent_used": 1.617974476175732e-06, 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "max_avail": 367075655680 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs_metadata", 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "stored": 3677757, 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "objects": 23, 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "kb_used": 7240, 2024-03-06T07:24:27.217 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 7413760, 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "percent_used": 1.0098307029693387e-05, 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "max_avail": 367075655680 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs_data", 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "stored": 0, 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "objects": 0, 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "kb_used": 0, 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 0, 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "percent_used": 0, 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: "max_avail": 367075655680 2024-03-06T07:24:27.218 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:27.219 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:27.219 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:24:27.219 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f140006b4d0 msgr2=0x7f140006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f140006b4d0 0x7f140006d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1408036000 tx=0x7f1408008030 comp rx=0 tx=0).stop 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 msgr2=0x7f14200cb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 0x7f14200cb680 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f141000c430 tx=0x7f141000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 shutdown_connections 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f140006b4d0 0x7f140006d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f142015df00 0x7f14201f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14200b9a10 0x7f14201584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 --2- 172.21.15.27:0/2360536452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14200b95a0 0x7f14200cb680 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 >> 172.21.15.27:0/2360536452 conn(0x7f1420071960 msgr2=0x7f1420071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 shutdown_connections 2024-03-06T07:24:27.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.215+0000 7f1425b25640 1 -- 172.21.15.27:0/2360536452 wait complete. 2024-03-06T07:24:27.227 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:24:27.228 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:27.228 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:27.270 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:27.322 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:27.322 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:24:27.323 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:27.339 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:27.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.483+0000 7fc7db1eb640 1 Processor -- start 2024-03-06T07:24:27.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.483+0000 7fc7db1eb640 1 -- start start 2024-03-06T07:24:27.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.484+0000 7fc7db1eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b95a0 0x7fc7d40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.484+0000 7fc7db1eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7d40b9a10 0x7fc7d40b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.484+0000 7fc7db1eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d40cea30 con 0x7fc7d415df00 2024-03-06T07:24:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.484+0000 7fc7db1eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d40cf3c0 con 0x7fc7d40b9a10 2024-03-06T07:24:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.484+0000 7fc7db1eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40cf0e0 con 0x7fc7d40b95a0 2024-03-06T07:24:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b95a0 0x7fc7d40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d9788640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7d40b9a10 0x7fc7d40b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d8f87640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc7d415df00 0x7fc7d40d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:53694/0 (socket says 172.21.15.27:53694) 2024-03-06T07:24:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d8f87640 1 -- 172.21.15.27:0/3234773135 learned_addr learned my addr 172.21.15.27:0/3234773135 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d9788640 1 -- 172.21.15.27:0/3234773135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b95a0 msgr2=0x7fc7d40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d9788640 1 --2- 172.21.15.27:0/3234773135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b95a0 0x7fc7d40b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d9788640 1 -- 172.21.15.27:0/3234773135 >> v1:172.21.15.69:6789/0 conn(0x7fc7d415df00 legacy=0x7fc7d40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d9788640 1 -- 172.21.15.27:0/3234773135 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d40ce470 con 0x7fc7d40b9a10 2024-03-06T07:24:27.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d37fe640 1 -- 172.21.15.27:0/3234773135 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129556128 0 0) 0x7fc7d40cea30 con 0x7fc7d415df00 2024-03-06T07:24:27.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.485+0000 7fc7d3fff640 1 --2- 172.21.15.27:0/3234773135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b95a0 0x7fc7d40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:27.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.486+0000 7fc7d9788640 1 --2- 172.21.15.27:0/3234773135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7d40b9a10 0x7fc7d40b5c40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009d90 tx=0x7fc7cc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ee553e0483681bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.486+0000 7fc7d37fe640 1 -- 172.21.15.27:0/3234773135 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc039070 con 0x7fc7d40b9a10 2024-03-06T07:24:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.486+0000 7fc7d37fe640 1 -- 172.21.15.27:0/3234773135 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc02cc60 con 0x7fc7d40b9a10 2024-03-06T07:24:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.486+0000 7fc7d37fe640 1 -- 172.21.15.27:0/3234773135 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc034660 con 0x7fc7d40b9a10 2024-03-06T07:24:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.487+0000 7fc7db1eb640 1 -- 172.21.15.27:0/3234773135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7d40b9a10 msgr2=0x7fc7d40b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.487+0000 7fc7db1eb640 1 --2- 172.21.15.27:0/3234773135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7d40b9a10 0x7fc7d40b5c40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009d90 tx=0x7fc7cc02beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.487+0000 7fc7db1eb640 1 -- 172.21.15.27:0/3234773135 shutdown_connections 2024-03-06T07:24:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.487+0000 7fc7db1eb640 1 --2- 172.21.15.27:0/3234773135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7d40b9a10 0x7fc7d40b5c40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.487+0000 7fc7db1eb640 1 --2- 172.21.15.27:0/3234773135 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b95a0 0x7fc7d40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.487+0000 7fc7db1eb640 1 -- 172.21.15.27:0/3234773135 >> 172.21.15.27:0/3234773135 conn(0x7fc7d4071960 msgr2=0x7fc7d4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.487+0000 7fc7db1eb640 1 -- 172.21.15.27:0/3234773135 shutdown_connections 2024-03-06T07:24:27.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.487+0000 7fc7db1eb640 1 -- 172.21.15.27:0/3234773135 wait complete. 2024-03-06T07:24:27.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.488+0000 7fc7db1eb640 1 Processor -- start 2024-03-06T07:24:27.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.488+0000 7fc7db1eb640 1 -- start start 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7db1eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7d40b95a0 0x7fc7d4152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7db1eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b9a10 0x7fc7d40c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7db1eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d415df00 0x7fc7d40ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7db1eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40cedf0 con 0x7fc7d415df00 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7db1eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40cec70 con 0x7fc7d40b95a0 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7db1eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40d3220 con 0x7fc7d40b9a10 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d9788640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d415df00 0x7fc7d40ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d9788640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d415df00 0x7fc7d40ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40132/0 (socket says 172.21.15.27:40132) 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d9788640 1 -- 172.21.15.27:0/4031023154 learned_addr learned my addr 172.21.15.27:0/4031023154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:27.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d8f87640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7d40b95a0 0x7fc7d4152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d3fff640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b9a10 0x7fc7d40c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d9788640 1 -- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b9a10 msgr2=0x7fc7d40c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d9788640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b9a10 0x7fc7d40c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d9788640 1 -- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7d40b95a0 msgr2=0x7fc7d4152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d9788640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7d40b95a0 0x7fc7d4152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.489+0000 7fc7d9788640 1 -- 172.21.15.27:0/4031023154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4152640 con 0x7fc7d415df00 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.490+0000 7fc7d9788640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d415df00 0x7fc7d40ccf20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009d90 tx=0x7fc7cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.490+0000 7fc7d17fa640 1 -- 172.21.15.27:0/4031023154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc039050 con 0x7fc7d415df00 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.490+0000 7fc7d17fa640 1 -- 172.21.15.27:0/4031023154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc045070 con 0x7fc7d415df00 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.490+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d40ca1e0 con 0x7fc7d415df00 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.490+0000 7fc7d17fa640 1 -- 172.21.15.27:0/4031023154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc009450 con 0x7fc7d415df00 2024-03-06T07:24:27.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.490+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d41f50e0 con 0x7fc7d415df00 2024-03-06T07:24:27.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.491+0000 7fc7d17fa640 1 -- 172.21.15.27:0/4031023154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7cc03e850 con 0x7fc7d415df00 2024-03-06T07:24:27.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.491+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d40cab20 con 0x7fc7d415df00 2024-03-06T07:24:27.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.492+0000 7fc7d17fa640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc7b806b4d0 0x7fc7b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.492+0000 7fc7d17fa640 1 -- 172.21.15.27:0/4031023154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7cc0ad050 con 0x7fc7d415df00 2024-03-06T07:24:27.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.492+0000 7fc7d8f87640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc7b806b4d0 0x7fc7b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.492+0000 7fc7d8f87640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc7b806b4d0 0x7fc7b806d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8007fd0 tx=0x7fc7c8007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.496+0000 7fc7d17fa640 1 -- 172.21.15.27:0/4031023154 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7d40cab20 con 0x7fc7d415df00 2024-03-06T07:24:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.727+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7d41f4ed0 con 0x7fc7d415df00 2024-03-06T07:24:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.728+0000 7fc7d17fa640 1 -- 172.21.15.27:0/4031023154 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fc7d4152640 con 0x7fc7d415df00 2024-03-06T07:24:27.730 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:27.730 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":35,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:56.637219+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:53.309610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:53.987106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:27.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc7b806b4d0 msgr2=0x7fc7b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc7b806b4d0 0x7fc7b806d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8007fd0 tx=0x7fc7c8007de0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d415df00 msgr2=0x7fc7d40ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d415df00 0x7fc7d40ccf20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009d90 tx=0x7fc7cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 shutdown_connections 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc7b806b4d0 0x7fc7b806d990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d415df00 0x7fc7d40ccf20 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7d40b9a10 0x7fc7d40c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 --2- 172.21.15.27:0/4031023154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7d40b95a0 0x7fc7d4152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 >> 172.21.15.27:0/4031023154 conn(0x7fc7d4071960 msgr2=0x7fc7d40b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 shutdown_connections 2024-03-06T07:24:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.730+0000 7fc7db1eb640 1 -- 172.21.15.27:0/4031023154 wait complete. 2024-03-06T07:24:27.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:53.309610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:53.987106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:27.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:27.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.851+0000 7f26c27e9640 1 Processor -- start 2024-03-06T07:24:27.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.851+0000 7f26c27e9640 1 -- start start 2024-03-06T07:24:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26c27e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f26bc0cf740 0x7f26bc0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26c27e9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26bc0c5e30 0x7f26bc0c6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26c27e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26bc0cd310 con 0x7f26bc15f500 2024-03-06T07:24:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26c27e9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26bc15f8e0 con 0x7f26bc0c5e30 2024-03-06T07:24:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26c27e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26bc0c56c0 con 0x7f26bc0cf740 2024-03-06T07:24:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26c0d86640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26bc0c5e30 0x7f26bc0c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26bbfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f26bc15f500 0x7f26bc0ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42926/0 (socket says 172.21.15.27:42926) 2024-03-06T07:24:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26c0d86640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26bc0c5e30 0x7f26bc0c6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40146/0 (socket says 172.21.15.27:40146) 2024-03-06T07:24:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26bbfff640 1 -- 172.21.15.27:0/830659270 learned_addr learned my addr 172.21.15.27:0/830659270 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.852+0000 7f26bb7fe640 1 --2- 172.21.15.27:0/830659270 >> v2:172.21.15.69:3300/0 conn(0x7f26bc0cf740 0x7f26bc0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.853+0000 7f26c0d86640 1 -- 172.21.15.27:0/830659270 >> v2:172.21.15.69:3300/0 conn(0x7f26bc0cf740 msgr2=0x7f26bc0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.853+0000 7f26c0d86640 1 --2- 172.21.15.27:0/830659270 >> v2:172.21.15.69:3300/0 conn(0x7f26bc0cf740 0x7f26bc0cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.853+0000 7f26c0d86640 1 -- 172.21.15.27:0/830659270 >> v1:172.21.15.27:6789/0 conn(0x7f26bc15f500 legacy=0x7f26bc0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.853+0000 7f26c0d86640 1 -- 172.21.15.27:0/830659270 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc0cd5b0 con 0x7f26bc0c5e30 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.853+0000 7f26baffd640 1 -- 172.21.15.27:0/830659270 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964152438 0 0) 0x7f26bc0cd310 con 0x7f26bc15f500 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.853+0000 7f26c0d86640 1 --2- 172.21.15.27:0/830659270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0c5e30 0x7f26bc0c6280 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0d1ab0 tx=0x7f26b402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16b9a8fb935d557 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.853+0000 7f26baffd640 1 -- 172.21.15.27:0/830659270 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b4039070 con 0x7f26bc0c5e30 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26baffd640 1 -- 172.21.15.27:0/830659270 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b402cbf0 con 0x7f26bc0c5e30 2024-03-06T07:24:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26baffd640 1 -- 172.21.15.27:0/830659270 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b4034440 con 0x7f26bc0c5e30 2024-03-06T07:24:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26c27e9640 1 -- 172.21.15.27:0/830659270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0c5e30 msgr2=0x7f26bc0c6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26c27e9640 1 --2- 172.21.15.27:0/830659270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0c5e30 0x7f26bc0c6280 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0d1ab0 tx=0x7f26b402be40 comp rx=0 tx=0).stop 2024-03-06T07:24:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26c27e9640 1 -- 172.21.15.27:0/830659270 shutdown_connections 2024-03-06T07:24:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26c27e9640 1 --2- 172.21.15.27:0/830659270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0c5e30 0x7f26bc0c6280 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26c27e9640 1 --2- 172.21.15.27:0/830659270 >> v2:172.21.15.69:3300/0 conn(0x7f26bc0cf740 0x7f26bc0cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26c27e9640 1 -- 172.21.15.27:0/830659270 >> 172.21.15.27:0/830659270 conn(0x7f26bc0b3700 msgr2=0x7f26bc0b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26c27e9640 1 -- 172.21.15.27:0/830659270 shutdown_connections 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.854+0000 7f26c27e9640 1 -- 172.21.15.27:0/830659270 wait complete. 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.855+0000 7f26c27e9640 1 Processor -- start 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.855+0000 7f26c27e9640 1 -- start start 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26c27e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26bc0c5e30 0x7f26bc0c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26c27e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0cf740 0x7f26bc0c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26c27e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26bc15f500 0x7f26bc0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26c27e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc0b43d0 con 0x7f26bc0cf740 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26c27e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc0c59a0 con 0x7f26bc0c5e30 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26c27e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc0cef50 con 0x7f26bc15f500 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26bb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0cf740 0x7f26bc0c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26bb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0cf740 0x7f26bc0c9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40148/0 (socket says 172.21.15.27:40148) 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26bb7fe640 1 -- 172.21.15.27:0/114683880 learned_addr learned my addr 172.21.15.27:0/114683880 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26bbfff640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26bc0c5e30 0x7f26bc0c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.856+0000 7f26c0d86640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26bc15f500 0x7f26bc0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26bb7fe640 1 -- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26bc15f500 msgr2=0x7f26bc0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26bb7fe640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26bc15f500 0x7f26bc0ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26bb7fe640 1 -- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26bc0c5e30 msgr2=0x7f26bc0c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26bb7fe640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26bc0c5e30 0x7f26bc0c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26bb7fe640 1 -- 172.21.15.27:0/114683880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc070550 con 0x7f26bc0cf740 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26bbfff640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26bc0c5e30 0x7f26bc0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26bb7fe640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0cf740 0x7f26bc0c9c90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00d800 tx=0x7f26ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26b8ff9640 1 -- 172.21.15.27:0/114683880 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac00b6c0 con 0x7f26bc0cf740 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26b8ff9640 1 -- 172.21.15.27:0/114683880 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac019070 con 0x7f26bc0cf740 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26bc0ca8e0 con 0x7f26bc0cf740 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26b8ff9640 1 -- 172.21.15.27:0/114683880 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac00ba80 con 0x7f26bc0cf740 2024-03-06T07:24:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.857+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26bc0cb180 con 0x7f26bc0cf740 2024-03-06T07:24:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.858+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26bc09fcf0 con 0x7f26bc0cf740 2024-03-06T07:24:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.858+0000 7f26b8ff9640 1 -- 172.21.15.27:0/114683880 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26ac00bc20 con 0x7f26bc0cf740 2024-03-06T07:24:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.859+0000 7f26b8ff9640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26a806b500 0x7f26a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.859+0000 7f26b8ff9640 1 -- 172.21.15.27:0/114683880 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26ac08a660 con 0x7f26bc0cf740 2024-03-06T07:24:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.859+0000 7f26bbfff640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26a806b500 0x7f26a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.860+0000 7f26bbfff640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26a806b500 0x7f26a806d9c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f26b0000f60 tx=0x7f26b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:27.863+0000 7f26b8ff9640 1 -- 172.21.15.27:0/114683880 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26bc070550 con 0x7f26bc0cf740 2024-03-06T07:24:28.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.147+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26bc04f410 con 0x7f26bc0cf740 2024-03-06T07:24:28.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.148+0000 7f26b8ff9640 1 -- 172.21.15.27:0/114683880 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f26ac01c360 con 0x7f26bc0cf740 2024-03-06T07:24:28.149 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:24:28.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26a806b500 msgr2=0x7f26a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26a806b500 0x7f26a806d9c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f26b0000f60 tx=0x7f26b0009040 comp rx=0 tx=0).stop 2024-03-06T07:24:28.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0cf740 msgr2=0x7f26bc0c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0cf740 0x7f26bc0c9c90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00d800 tx=0x7f26ac00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 shutdown_connections 2024-03-06T07:24:28.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f26a806b500 0x7f26a806d9c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26bc15f500 0x7f26bc0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26bc0cf740 0x7f26bc0c9c90 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 --2- 172.21.15.27:0/114683880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26bc0c5e30 0x7f26bc0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.149+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 >> 172.21.15.27:0/114683880 conn(0x7f26bc0b3700 msgr2=0x7f26bc06ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.150+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 shutdown_connections 2024-03-06T07:24:28.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.150+0000 7f26c27e9640 1 -- 172.21.15.27:0/114683880 wait complete. 2024-03-06T07:24:28.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.270+0000 7f3994e50640 1 Processor -- start 2024-03-06T07:24:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.270+0000 7f3994e50640 1 -- start start 2024-03-06T07:24:28.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f3994e50640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f399015dd90 0x7f39900ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f3994e50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39900b6d10 0x7f39900b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f3994e50640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900d3740 con 0x7f39900d29d0 2024-03-06T07:24:28.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f3994e50640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39900d3400 con 0x7f399015dd90 2024-03-06T07:24:28.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f3994e50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900d3120 con 0x7f39900b6d10 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f398dd9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f39900d29d0 0x7f39900d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42932/0 (socket says 172.21.15.27:42932) 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f398dd9b640 1 -- 172.21.15.27:0/1978300437 learned_addr learned my addr 172.21.15.27:0/1978300437 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542429777 0 0) 0x7f39900d3740 con 0x7f39900d29d0 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c003520 con 0x7f39900d29d0 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f398ed9d640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39900b6d10 0x7f39900b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.271+0000 7f398e59c640 1 --2- 172.21.15.27:0/1978300437 >> v2:172.21.15.69:3300/0 conn(0x7f399015dd90 0x7f39900ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3827374109 0 0) 0x7f397c003520 con 0x7f39900d29d0 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39900b6d10 msgr2=0x7f39900b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39900b6d10 0x7f39900b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 >> v2:172.21.15.69:3300/0 conn(0x7f399015dd90 msgr2=0x7f39900ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 --2- 172.21.15.27:0/1978300437 >> v2:172.21.15.69:3300/0 conn(0x7f399015dd90 0x7f39900ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900d39e0 con 0x7f39900d29d0 2024-03-06T07:24:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f3980001690 con 0x7f39900d29d0 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 >> v1:172.21.15.27:6789/0 conn(0x7f39900d29d0 legacy=0x7f39900d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f397c0039b0 0x7f397c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f397c004800 0x7f397c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39900b6d10 0x7f397c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900d3400 con 0x7f397c0039b0 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900d3120 con 0x7f397c004800 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397c00b9c0 con 0x7f39900b6d10 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398e59c640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f397c0039b0 0x7f397c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398dd9b640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f397c004800 0x7f397c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398ed9d640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39900b6d10 0x7f397c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398e59c640 1 -- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39900b6d10 msgr2=0x7f397c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398e59c640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39900b6d10 0x7f397c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398e59c640 1 -- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f397c004800 msgr2=0x7f397c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398e59c640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f397c004800 0x7f397c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398e59c640 1 -- 172.21.15.27:0/1978300437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c003520 con 0x7f397c0039b0 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.272+0000 7f398ed9d640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39900b6d10 0x7f397c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.273+0000 7f398e59c640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f397c0039b0 0x7f397c003dd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3978009ac0 tx=0x7f397c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93756da6b7108a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.273+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978039070 con 0x7f397c0039b0 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.273+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397802c710 con 0x7f397c0039b0 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.273+0000 7f398d59a640 1 -- 172.21.15.27:0/1978300437 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978009690 con 0x7f397c0039b0 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.274+0000 7f3994e50640 1 -- 172.21.15.27:0/1978300437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f397c0039b0 msgr2=0x7f397c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.274+0000 7f3994e50640 1 --2- 172.21.15.27:0/1978300437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f397c0039b0 0x7f397c003dd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3978009ac0 tx=0x7f397c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.274+0000 7f398ed9d640 1 -- 172.21.15.27:0/1978300437 reap_dead start 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.274+0000 7f3994e50640 1 -- 172.21.15.27:0/1978300437 shutdown_connections 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.274+0000 7f3994e50640 1 -- 172.21.15.27:0/1978300437 >> 172.21.15.27:0/1978300437 conn(0x7f3990071960 msgr2=0x7f3990071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.274+0000 7f3994e50640 1 -- 172.21.15.27:0/1978300437 shutdown_connections 2024-03-06T07:24:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.274+0000 7f3994e50640 1 -- 172.21.15.27:0/1978300437 wait complete. 2024-03-06T07:24:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.275+0000 7f3994e50640 1 Processor -- start 2024-03-06T07:24:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.275+0000 7f3994e50640 1 -- start start 2024-03-06T07:24:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f3994e50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f397c0039b0 0x7f39900c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f3994e50640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399014f180 0x7f39901f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f3994e50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39900cad70 0x7f39900cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f3994e50640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900d2df0 con 0x7f399014f180 2024-03-06T07:24:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f3994e50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399015e170 con 0x7f39900cad70 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f3994e50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900d48f0 con 0x7f397c0039b0 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399014f180 0x7f39901f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399014f180 0x7f39901f3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40156/0 (socket says 172.21.15.27:40156) 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398dd9b640 1 -- 172.21.15.27:0/1163736021 learned_addr learned my addr 172.21.15.27:0/1163736021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398e59c640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f397c0039b0 0x7f39900c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398ed9d640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39900cad70 0x7f39900cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398dd9b640 1 -- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f397c0039b0 msgr2=0x7f39900c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398dd9b640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f397c0039b0 0x7f39900c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398dd9b640 1 -- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39900cad70 msgr2=0x7f39900cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398dd9b640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39900cad70 0x7f39900cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.276+0000 7f398dd9b640 1 -- 172.21.15.27:0/1163736021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900c77b0 con 0x7f399014f180 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.277+0000 7f398dd9b640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399014f180 0x7f39901f3d70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3980004f40 tx=0x7f39800074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.277+0000 7f3986ffd640 1 -- 172.21.15.27:0/1163736021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000f750 con 0x7f399014f180 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.277+0000 7f3986ffd640 1 -- 172.21.15.27:0/1163736021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980018070 con 0x7f399014f180 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.277+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399014f600 con 0x7f399014f180 2024-03-06T07:24:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.277+0000 7f3986ffd640 1 -- 172.21.15.27:0/1163736021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000fb50 con 0x7f399014f180 2024-03-06T07:24:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.277+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39901f4880 con 0x7f399014f180 2024-03-06T07:24:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.278+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3990072a30 con 0x7f399014f180 2024-03-06T07:24:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.278+0000 7f3986ffd640 1 -- 172.21.15.27:0/1163736021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f398000fcf0 con 0x7f399014f180 2024-03-06T07:24:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.279+0000 7f3986ffd640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f397006b4d0 0x7f397006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.279+0000 7f3986ffd640 1 -- 172.21.15.27:0/1163736021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f398008a0f0 con 0x7f399014f180 2024-03-06T07:24:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.279+0000 7f398e59c640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f397006b4d0 0x7f397006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.279+0000 7f398e59c640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f397006b4d0 0x7f397006d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3978036000 tx=0x7f3978030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.283+0000 7f3986ffd640 1 -- 172.21.15.27:0/1163736021 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39800589b0 con 0x7f399014f180 2024-03-06T07:24:28.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.462+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f399004f410 con 0x7f399014f180 2024-03-06T07:24:28.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.463+0000 7f3986ffd640 1 -- 172.21.15.27:0/1163736021 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v35) v1 ==== 104+0+11 (secure 0 0 0) 0x7f398005ca50 con 0x7f399014f180 2024-03-06T07:24:28.464 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:24:28.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f397006b4d0 msgr2=0x7f397006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f397006b4d0 0x7f397006d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3978036000 tx=0x7f3978030000 comp rx=0 tx=0).stop 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399014f180 msgr2=0x7f39901f3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399014f180 0x7f39901f3d70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3980004f40 tx=0x7f39800074f0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 shutdown_connections 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f397006b4d0 0x7f397006d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39900cad70 0x7f39900cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399014f180 0x7f39901f3d70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 --2- 172.21.15.27:0/1163736021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f397c0039b0 0x7f39900c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 >> 172.21.15.27:0/1163736021 conn(0x7f3990071960 msgr2=0x7f399015be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 shutdown_connections 2024-03-06T07:24:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.465+0000 7f3994e50640 1 -- 172.21.15.27:0/1163736021 wait complete. 2024-03-06T07:24:28.477 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:28.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.592+0000 7fdddaa42640 1 Processor -- start 2024-03-06T07:24:28.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.592+0000 7fdddaa42640 1 -- start start 2024-03-06T07:24:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.593+0000 7fdddaa42640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fddd40cf010 0x7fddd415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.593+0000 7fdddaa42640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd415bc40 con 0x7fddd4144640 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.593+0000 7fdddaa42640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd415bf20 con 0x7fddd415b130 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.593+0000 7fdddaa42640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd406df70 con 0x7fddd40cf010 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.593+0000 7fddd3fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fddd4144640 0x7fddd406e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42934/0 (socket says 172.21.15.27:42934) 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.593+0000 7fddd3fff640 1 -- 172.21.15.27:0/1230717892 learned_addr learned my addr 172.21.15.27:0/1230717892 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.594+0000 7fddd8fdf640 1 --2- 172.21.15.27:0/1230717892 >> v2:172.21.15.69:3300/0 conn(0x7fddd40cf010 0x7fddd415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.594+0000 7fddd2ffd640 1 -- 172.21.15.27:0/1230717892 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868834953 0 0) 0x7fddd415bc40 con 0x7fddd4144640 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.594+0000 7fddd2ffd640 1 -- 172.21.15.27:0/1230717892 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc0003520 con 0x7fddd4144640 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.594+0000 7fddd8fdf640 1 -- 172.21.15.27:0/1230717892 >> v1:172.21.15.69:6789/0 conn(0x7fddd415b130 legacy=0x7fddd40ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.594+0000 7fddd8fdf640 1 -- 172.21.15.27:0/1230717892 >> v1:172.21.15.27:6789/0 conn(0x7fddd4144640 legacy=0x7fddd406e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.594+0000 7fddd8fdf640 1 -- 172.21.15.27:0/1230717892 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd406dc00 con 0x7fddd40cf010 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.595+0000 7fddd8fdf640 1 --2- 172.21.15.27:0/1230717892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddd40cf010 0x7fddd415b5c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009d90 tx=0x7fddcc02e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcdba48c3cb4e926 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.595+0000 7fddd2ffd640 1 -- 172.21.15.27:0/1230717892 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc03b070 con 0x7fddd40cf010 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.595+0000 7fddd2ffd640 1 -- 172.21.15.27:0/1230717892 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc036440 con 0x7fddd40cf010 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.595+0000 7fddd2ffd640 1 -- 172.21.15.27:0/1230717892 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc0367f0 con 0x7fddd40cf010 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.596+0000 7fdddaa42640 1 -- 172.21.15.27:0/1230717892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddd40cf010 msgr2=0x7fddd415b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.596+0000 7fdddaa42640 1 --2- 172.21.15.27:0/1230717892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddd40cf010 0x7fddd415b5c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009d90 tx=0x7fddcc02e160 comp rx=0 tx=0).stop 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.596+0000 7fdddaa42640 1 -- 172.21.15.27:0/1230717892 shutdown_connections 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.596+0000 7fdddaa42640 1 --2- 172.21.15.27:0/1230717892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddd40cf010 0x7fddd415b5c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.596+0000 7fdddaa42640 1 -- 172.21.15.27:0/1230717892 >> 172.21.15.27:0/1230717892 conn(0x7fddd4066d00 msgr2=0x7fddd4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.596+0000 7fdddaa42640 1 -- 172.21.15.27:0/1230717892 shutdown_connections 2024-03-06T07:24:28.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.596+0000 7fdddaa42640 1 -- 172.21.15.27:0/1230717892 wait complete. 2024-03-06T07:24:28.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.598+0000 7fdddaa42640 1 Processor -- start 2024-03-06T07:24:28.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.598+0000 7fdddaa42640 1 -- start start 2024-03-06T07:24:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fdddaa42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddd40cf010 0x7fddd415a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fdddaa42640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddd4144640 0x7fddd41ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fdddaa42640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd415b130 0x7fddd41f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fdddaa42640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4070c20 con 0x7fddd415b130 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fdddaa42640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4070aa0 con 0x7fddd40cf010 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fdddaa42640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd40b36d0 con 0x7fddd4144640 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd8fdf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd415b130 0x7fddd41f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd8fdf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd415b130 0x7fddd41f3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40158/0 (socket says 172.21.15.27:40158) 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd8fdf640 1 -- 172.21.15.27:0/3208617233 learned_addr learned my addr 172.21.15.27:0/3208617233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd37fe640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddd4144640 0x7fddd41ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd3fff640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddd40cf010 0x7fddd415a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd8fdf640 1 -- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddd4144640 msgr2=0x7fddd41ef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd8fdf640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddd4144640 0x7fddd41ef820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd8fdf640 1 -- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddd40cf010 msgr2=0x7fddd415a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd8fdf640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddd40cf010 0x7fddd415a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd8fdf640 1 -- 172.21.15.27:0/3208617233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd413a2d0 con 0x7fddd415b130 2024-03-06T07:24:28.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd37fe640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddd4144640 0x7fddd41ef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:28.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd8fdf640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd415b130 0x7fddd41f3830 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009d90 tx=0x7fddcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd0ff9640 1 -- 172.21.15.27:0/3208617233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc03b050 con 0x7fddd415b130 2024-03-06T07:24:28.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd0ff9640 1 -- 172.21.15.27:0/3208617233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc047070 con 0x7fddd415b130 2024-03-06T07:24:28.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fddd0ff9640 1 -- 172.21.15.27:0/3208617233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc009450 con 0x7fddd415b130 2024-03-06T07:24:28.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.600+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd41f3d70 con 0x7fddd415b130 2024-03-06T07:24:28.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.601+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd41f0190 con 0x7fddd415b130 2024-03-06T07:24:28.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.602+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd40733c0 con 0x7fddd415b130 2024-03-06T07:24:28.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.602+0000 7fddd0ff9640 1 -- 172.21.15.27:0/3208617233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddcc036830 con 0x7fddd415b130 2024-03-06T07:24:28.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.602+0000 7fddd0ff9640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb406b4d0 0x7fddb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.602+0000 7fddd0ff9640 1 -- 172.21.15.27:0/3208617233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddcc0aef60 con 0x7fddd415b130 2024-03-06T07:24:28.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.603+0000 7fddd3fff640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb406b4d0 0x7fddb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.603+0000 7fddd3fff640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb406b4d0 0x7fddb406d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fddc40011e0 tx=0x7fddc4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.612+0000 7fddd0ff9640 1 -- 172.21.15.27:0/3208617233 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddcc07de70 con 0x7fddd415b130 2024-03-06T07:24:28.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.811+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fddd415a9d0 con 0x7fddd415b130 2024-03-06T07:24:28.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.811+0000 7fddd0ff9640 1 -- 172.21.15.27:0/3208617233 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v35) v1 ==== 100+0+11 (secure 0 0 0) 0x7fddcc07d770 con 0x7fddd415b130 2024-03-06T07:24:28.813 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:24:28.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb406b4d0 msgr2=0x7fddb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb406b4d0 0x7fddb406d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fddc40011e0 tx=0x7fddc4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd415b130 msgr2=0x7fddd41f3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd415b130 0x7fddd41f3830 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009d90 tx=0x7fddcc004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 shutdown_connections 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb406b4d0 0x7fddb406d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd415b130 0x7fddd41f3830 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddd4144640 0x7fddd41ef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 --2- 172.21.15.27:0/3208617233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddd40cf010 0x7fddd415a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 >> 172.21.15.27:0/3208617233 conn(0x7fddd4066d00 msgr2=0x7fddd415cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 shutdown_connections 2024-03-06T07:24:28.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.813+0000 7fdddaa42640 1 -- 172.21.15.27:0/3208617233 wait complete. 2024-03-06T07:24:28.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.932+0000 7f724c8c3640 1 Processor -- start 2024-03-06T07:24:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.932+0000 7f724c8c3640 1 -- start start 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724c8c3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f724815df00 0x7f72480d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724c8c3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f72480b9a30 0x7f72480d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724c8c3640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72480cea30 con 0x7f72480b95a0 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724c8c3640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72480d2af0 con 0x7f724815df00 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724c8c3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72480cf3c0 con 0x7f72480b9a30 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f724815df00 0x7f72480d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f7245d9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f72480b95a0 0x7f72480b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42950/0 (socket says 172.21.15.27:42950) 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f7245d9b640 1 -- 172.21.15.27:0/4241885533 learned_addr learned my addr 172.21.15.27:0/4241885533 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f7246d9d640 1 --2- 172.21.15.27:0/4241885533 >> v2:172.21.15.69:3300/0 conn(0x7f72480b9a30 0x7f72480d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724659c640 1 -- 172.21.15.27:0/4241885533 >> v2:172.21.15.69:3300/0 conn(0x7f72480b9a30 msgr2=0x7f72480d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724659c640 1 --2- 172.21.15.27:0/4241885533 >> v2:172.21.15.69:3300/0 conn(0x7f72480b9a30 0x7f72480d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724559a640 1 -- 172.21.15.27:0/4241885533 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294328480 0 0) 0x7f72480cea30 con 0x7f72480b95a0 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724659c640 1 -- 172.21.15.27:0/4241885533 >> v1:172.21.15.27:6789/0 conn(0x7f72480b95a0 legacy=0x7f72480b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.933+0000 7f724659c640 1 -- 172.21.15.27:0/4241885533 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72480cf0e0 con 0x7f724815df00 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.934+0000 7f724659c640 1 --2- 172.21.15.27:0/4241885533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 0x7f72480d25b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f72480d4d20 tx=0x7f723c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628c52614ab9205e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.934+0000 7f724559a640 1 -- 172.21.15.27:0/4241885533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723c035020 con 0x7f724815df00 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.934+0000 7f724559a640 1 -- 172.21.15.27:0/4241885533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723c031830 con 0x7f724815df00 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.934+0000 7f724559a640 1 -- 172.21.15.27:0/4241885533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723c031c00 con 0x7f724815df00 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.935+0000 7f724c8c3640 1 -- 172.21.15.27:0/4241885533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 msgr2=0x7f72480d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.935+0000 7f724c8c3640 1 --2- 172.21.15.27:0/4241885533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 0x7f72480d25b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f72480d4d20 tx=0x7f723c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.935+0000 7f724c8c3640 1 -- 172.21.15.27:0/4241885533 shutdown_connections 2024-03-06T07:24:28.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.935+0000 7f724c8c3640 1 --2- 172.21.15.27:0/4241885533 >> v2:172.21.15.69:3300/0 conn(0x7f72480b9a30 0x7f72480d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.935+0000 7f724c8c3640 1 --2- 172.21.15.27:0/4241885533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 0x7f72480d25b0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.935+0000 7f724c8c3640 1 -- 172.21.15.27:0/4241885533 >> 172.21.15.27:0/4241885533 conn(0x7f7248071960 msgr2=0x7f7248071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.935+0000 7f724c8c3640 1 -- 172.21.15.27:0/4241885533 shutdown_connections 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.935+0000 7f724c8c3640 1 -- 172.21.15.27:0/4241885533 wait complete. 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.936+0000 7f724c8c3640 1 Processor -- start 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.936+0000 7f724c8c3640 1 -- start start 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f724c8c3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72480b95a0 0x7f72480c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f724c8c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 0x7f7248157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f724c8c3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72480c8580 0x7f72480b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f724c8c3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72480cec70 con 0x7f724815df00 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f724c8c3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72480ce750 con 0x7f72480c8580 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f724c8c3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72480cedf0 con 0x7f72480b95a0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f7245d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 0x7f7248157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f7245d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 0x7f7248157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40186/0 (socket says 172.21.15.27:40186) 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f7245d9b640 1 -- 172.21.15.27:0/1310178238 learned_addr learned my addr 172.21.15.27:0/1310178238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f724659c640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72480b95a0 0x7f72480c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.937+0000 7f7246d9d640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72480c8580 0x7f72480b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7245d9b640 1 -- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72480b95a0 msgr2=0x7f72480c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7245d9b640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72480b95a0 0x7f72480c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7245d9b640 1 -- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72480c8580 msgr2=0x7f72480b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7245d9b640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72480c8580 0x7f72480b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7245d9b640 1 -- 172.21.15.27:0/1310178238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72480c90e0 con 0x7f724815df00 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7246d9d640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72480c8580 0x7f72480b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7245d9b640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 0x7f7248157cf0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f723800c4d0 tx=0x7f723800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7236ffd640 1 -- 172.21.15.27:0/1310178238 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238018070 con 0x7f724815df00 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7236ffd640 1 -- 172.21.15.27:0/1310178238 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7238007500 con 0x7f724815df00 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f7236ffd640 1 -- 172.21.15.27:0/1310178238 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238004a20 con 0x7f724815df00 2024-03-06T07:24:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724815e2e0 con 0x7f724815df00 2024-03-06T07:24:28.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.938+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72480c8e00 con 0x7f724815df00 2024-03-06T07:24:28.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.939+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7248067ff0 con 0x7f724815df00 2024-03-06T07:24:28.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.940+0000 7f7236ffd640 1 -- 172.21.15.27:0/1310178238 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f723801b070 con 0x7f724815df00 2024-03-06T07:24:28.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.940+0000 7f7236ffd640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f722806b4d0 0x7f722806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.941+0000 7f7236ffd640 1 -- 172.21.15.27:0/1310178238 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7238089510 con 0x7f724815df00 2024-03-06T07:24:28.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.941+0000 7f724659c640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f722806b4d0 0x7f722806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.941+0000 7f724659c640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f722806b4d0 0x7f722806d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f723c036000 tx=0x7f723c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:28.945+0000 7f7236ffd640 1 -- 172.21.15.27:0/1310178238 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7238058390 con 0x7f724815df00 2024-03-06T07:24:29.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.138+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f72480d0420 con 0x7f724815df00 2024-03-06T07:24:29.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.138+0000 7f7236ffd640 1 -- 172.21.15.27:0/1310178238 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v35) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7238057c90 con 0x7f724815df00 2024-03-06T07:24:29.140 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:24:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f722806b4d0 msgr2=0x7f722806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f722806b4d0 0x7f722806d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f723c036000 tx=0x7f723c03c040 comp rx=0 tx=0).stop 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 msgr2=0x7f7248157cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 0x7f7248157cf0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f723800c4d0 tx=0x7f723800c990 comp rx=0 tx=0).stop 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 shutdown_connections 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f722806b4d0 0x7f722806d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72480c8580 0x7f72480b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724815df00 0x7f7248157cf0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 --2- 172.21.15.27:0/1310178238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f72480b95a0 0x7f72480c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 >> 172.21.15.27:0/1310178238 conn(0x7f7248071960 msgr2=0x7f72481618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 shutdown_connections 2024-03-06T07:24:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.141+0000 7f724c8c3640 1 -- 172.21.15.27:0/1310178238 wait complete. 2024-03-06T07:24:29.153 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:24:29.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:29.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:29.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[52808]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:29.184 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:24:29.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:29.184 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:29.351 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:24:29.351 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:29.368 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:29.369 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:24:29.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:29.369 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:24:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:29.607 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:24:29.607 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:24:29.607 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:29.623 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:29.623 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:24:29.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:29.624 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:24:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:24:29.680 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:24:29.680 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:29.699 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:29.699 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:24:29.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:29.699 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:24:29.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:29.757 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:24:29.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:29.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.862+0000 7f415fab1640 1 Processor -- start 2024-03-06T07:24:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.862+0000 7f415fab1640 1 -- start start 2024-03-06T07:24:29.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.863+0000 7f415fab1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4158071540 0x7f41580b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.863+0000 7f415fab1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580b2b50 0x7f41580b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.863+0000 7f415fab1640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580ce610 con 0x7f41580d3400 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.863+0000 7f415fab1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41580d2f70 con 0x7f4158071540 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.863+0000 7f415fab1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41580d4be0 con 0x7f41580b2b50 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.863+0000 7f415d04c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f41580d3400 0x7f4158071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42952/0 (socket says 172.21.15.27:42952) 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.863+0000 7f415d04c640 1 -- 172.21.15.27:0/1764224770 learned_addr learned my addr 172.21.15.27:0/1764224770 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415d84d640 1 --2- 172.21.15.27:0/1764224770 >> v2:172.21.15.69:3300/0 conn(0x7f4158071540 0x7f41580b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415e04e640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580b2b50 0x7f41580b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244377971 0 0) 0x7f41580ce610 con 0x7f41580d3400 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4144003520 con 0x7f41580d3400 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2463024995 0 0) 0x7f4144003520 con 0x7f41580d3400 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580b2b50 msgr2=0x7f41580b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580b2b50 0x7f41580b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 >> v2:172.21.15.69:3300/0 conn(0x7f4158071540 msgr2=0x7f41580b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 --2- 172.21.15.27:0/1764224770 >> v2:172.21.15.69:3300/0 conn(0x7f4158071540 0x7f41580b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580d2920 con 0x7f41580d3400 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f4154001670 con 0x7f41580d3400 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 >> v1:172.21.15.27:6789/0 conn(0x7f41580d3400 legacy=0x7f4158071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41440039b0 0x7f4144003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580b2b50 0x7f4144004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4144004d40 0x7f41440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41580d2f70 con 0x7f4144004d40 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41580d4be0 con 0x7f41440039b0 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414400b9c0 con 0x7f41580b2b50 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415e04e640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4144004d40 0x7f41440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415d84d640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41440039b0 0x7f4144003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415d04c640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580b2b50 0x7f4144004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415e04e640 1 -- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580b2b50 msgr2=0x7f4144004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.864+0000 7f415e04e640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580b2b50 0x7f4144004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.865+0000 7f415e04e640 1 -- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41440039b0 msgr2=0x7f4144003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.865+0000 7f415e04e640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41440039b0 0x7f4144003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.865+0000 7f415e04e640 1 -- 172.21.15.27:0/1764224770 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4144003520 con 0x7f4144004d40 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.865+0000 7f415e04e640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4144004d40 0x7f41440051c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f415000e170 tx=0x7f4144005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0dfa702d320369c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.865+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150039070 con 0x7f4144004d40 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.865+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415002cbf0 con 0x7f4144004d40 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.865+0000 7f415c84b640 1 -- 172.21.15.27:0/1764224770 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150009c10 con 0x7f4144004d40 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.866+0000 7f415fab1640 1 -- 172.21.15.27:0/1764224770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4144004d40 msgr2=0x7f41440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.866+0000 7f415fab1640 1 --2- 172.21.15.27:0/1764224770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4144004d40 0x7f41440051c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f415000e170 tx=0x7f4144005830 comp rx=0 tx=0).stop 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.866+0000 7f415e04e640 1 -- 172.21.15.27:0/1764224770 reap_dead start 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.866+0000 7f415fab1640 1 -- 172.21.15.27:0/1764224770 shutdown_connections 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.866+0000 7f415fab1640 1 -- 172.21.15.27:0/1764224770 >> 172.21.15.27:0/1764224770 conn(0x7f41580d3ca0 msgr2=0x7f41580d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.866+0000 7f415fab1640 1 -- 172.21.15.27:0/1764224770 shutdown_connections 2024-03-06T07:24:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.866+0000 7f415fab1640 1 -- 172.21.15.27:0/1764224770 wait complete. 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.867+0000 7f415fab1640 1 Processor -- start 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.867+0000 7f415fab1640 1 -- start start 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415fab1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158149c50 0x7f415814d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415fab1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41581505d0 0x7f415814cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415fab1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580b7da0 0x7f41580b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415fab1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415815e370 con 0x7f41580b7da0 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415fab1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41580ce8f0 con 0x7f41581505d0 2024-03-06T07:24:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415fab1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415814b270 con 0x7f4158149c50 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415e04e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580b7da0 0x7f41580b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415e04e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580b7da0 0x7f41580b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40216/0 (socket says 172.21.15.27:40216) 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415e04e640 1 -- 172.21.15.27:0/1011617568 learned_addr learned my addr 172.21.15.27:0/1011617568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415d84d640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158149c50 0x7f415814d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415d04c640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41581505d0 0x7f415814cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415e04e640 1 -- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158149c50 msgr2=0x7f415814d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415e04e640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158149c50 0x7f415814d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415e04e640 1 -- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41581505d0 msgr2=0x7f415814cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415e04e640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41581505d0 0x7f415814cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415e04e640 1 -- 172.21.15.27:0/1011617568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415814d450 con 0x7f41580b7da0 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415d84d640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158149c50 0x7f415814d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.868+0000 7f415d04c640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41581505d0 0x7f415814cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.869+0000 7f415e04e640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580b7da0 0x7f41580b8220 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4150009f90 tx=0x7f4150009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.869+0000 7f414e7fc640 1 -- 172.21.15.27:0/1011617568 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150044070 con 0x7f41580b7da0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.869+0000 7f414e7fc640 1 -- 172.21.15.27:0/1011617568 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415003a040 con 0x7f41580b7da0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.869+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415814a660 con 0x7f41580b7da0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.869+0000 7f414e7fc640 1 -- 172.21.15.27:0/1011617568 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150039030 con 0x7f41580b7da0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.869+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415814ac20 con 0x7f41580b7da0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.870+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41580b8760 con 0x7f41580b7da0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.870+0000 7f414e7fc640 1 -- 172.21.15.27:0/1011617568 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4150047070 con 0x7f41580b7da0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.871+0000 7f414e7fc640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f413c06b4d0 0x7f413c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.871+0000 7f414e7fc640 1 -- 172.21.15.27:0/1011617568 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41500abc00 con 0x7f41580b7da0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.871+0000 7f415d84d640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f413c06b4d0 0x7f413c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.872+0000 7f415d84d640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f413c06b4d0 0x7f413c06d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4148009bc0 tx=0x7f4148008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:29.875+0000 7f414e7fc640 1 -- 172.21.15.27:0/1011617568 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41500b2080 con 0x7f41580b7da0 2024-03-06T07:24:30.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.095+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4158068700 con 0x7f41580b7da0 2024-03-06T07:24:30.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.096+0000 7f414e7fc640 1 -- 172.21.15.27:0/1011617568 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f415007a3b0 con 0x7f41580b7da0 2024-03-06T07:24:30.098 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:30.098 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:30.098 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f413c06b4d0 msgr2=0x7f413c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f413c06b4d0 0x7f413c06d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4148009bc0 tx=0x7f4148008040 comp rx=0 tx=0).stop 2024-03-06T07:24:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580b7da0 msgr2=0x7f41580b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580b7da0 0x7f41580b8220 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4150009f90 tx=0x7f4150009750 comp rx=0 tx=0).stop 2024-03-06T07:24:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 shutdown_connections 2024-03-06T07:24:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f413c06b4d0 0x7f413c06d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580b7da0 0x7f41580b8220 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41581505d0 0x7f415814cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 --2- 172.21.15.27:0/1011617568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158149c50 0x7f415814d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 >> 172.21.15.27:0/1011617568 conn(0x7f41580d3ca0 msgr2=0x7f41580cea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 shutdown_connections 2024-03-06T07:24:30.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.098+0000 7f415fab1640 1 -- 172.21.15.27:0/1011617568 wait complete. 2024-03-06T07:24:30.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:30.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.217+0000 7fb60e525640 1 Processor -- start 2024-03-06T07:24:30.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.217+0000 7fb60e525640 1 -- start start 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.217+0000 7fb60e525640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb60815df00 0x7fb6080d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.217+0000 7fb60e525640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6080b95a0 0x7fb6080b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb60e525640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6080b9a30 0x7fb6080b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb60e525640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6080cea30 con 0x7fb6080b95a0 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb60e525640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6080d2af0 con 0x7fb60815df00 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb60e525640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6080cf3c0 con 0x7fb6080b9a30 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb6077fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6080b95a0 0x7fb6080b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb6077fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6080b95a0 0x7fb6080b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40218/0 (socket says 172.21.15.27:40218) 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb6077fe640 1 -- 172.21.15.27:0/1606291602 learned_addr learned my addr 172.21.15.27:0/1606291602 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb607fff640 1 --2- 172.21.15.27:0/1606291602 >> v2:172.21.15.69:3300/0 conn(0x7fb60815df00 0x7fb6080d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb60cac2640 1 --2- 172.21.15.27:0/1606291602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6080b9a30 0x7fb6080b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb6077fe640 1 -- 172.21.15.27:0/1606291602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6080b9a30 msgr2=0x7fb6080b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb6077fe640 1 --2- 172.21.15.27:0/1606291602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6080b9a30 0x7fb6080b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb6077fe640 1 -- 172.21.15.27:0/1606291602 >> v2:172.21.15.69:3300/0 conn(0x7fb60815df00 msgr2=0x7fb6080d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb6077fe640 1 --2- 172.21.15.27:0/1606291602 >> v2:172.21.15.69:3300/0 conn(0x7fb60815df00 0x7fb6080d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb6077fe640 1 -- 172.21.15.27:0/1606291602 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6080cf0e0 con 0x7fb6080b95a0 2024-03-06T07:24:30.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.218+0000 7fb60cac2640 1 --2- 172.21.15.27:0/1606291602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6080b9a30 0x7fb6080b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.219+0000 7fb6077fe640 1 --2- 172.21.15.27:0/1606291602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6080b95a0 0x7fb6080b5700 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb6080d11b0 tx=0x7fb5fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7391a68af08267c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.219+0000 7fb606ffd640 1 -- 172.21.15.27:0/1606291602 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc035020 con 0x7fb6080b95a0 2024-03-06T07:24:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb606ffd640 1 -- 172.21.15.27:0/1606291602 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc031870 con 0x7fb6080b95a0 2024-03-06T07:24:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb606ffd640 1 -- 172.21.15.27:0/1606291602 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc031c40 con 0x7fb6080b95a0 2024-03-06T07:24:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb60e525640 1 -- 172.21.15.27:0/1606291602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6080b95a0 msgr2=0x7fb6080b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb60e525640 1 --2- 172.21.15.27:0/1606291602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6080b95a0 0x7fb6080b5700 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb6080d11b0 tx=0x7fb5fc02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb60e525640 1 -- 172.21.15.27:0/1606291602 shutdown_connections 2024-03-06T07:24:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb60e525640 1 --2- 172.21.15.27:0/1606291602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6080b9a30 0x7fb6080b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb60e525640 1 --2- 172.21.15.27:0/1606291602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6080b95a0 0x7fb6080b5700 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb60e525640 1 --2- 172.21.15.27:0/1606291602 >> v2:172.21.15.69:3300/0 conn(0x7fb60815df00 0x7fb6080d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb60e525640 1 -- 172.21.15.27:0/1606291602 >> 172.21.15.27:0/1606291602 conn(0x7fb608071960 msgr2=0x7fb608071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb60e525640 1 -- 172.21.15.27:0/1606291602 shutdown_connections 2024-03-06T07:24:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.220+0000 7fb60e525640 1 -- 172.21.15.27:0/1606291602 wait complete. 2024-03-06T07:24:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.221+0000 7fb60e525640 1 Processor -- start 2024-03-06T07:24:30.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.221+0000 7fb60e525640 1 -- start start 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb60e525640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6080b95a0 0x7fb6080cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb60e525640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb60815df00 0x7fb6081f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb60e525640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb608160780 0x7fb608160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb60e525640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6080cec70 con 0x7fb608160780 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb60e525640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6080ce750 con 0x7fb6080b95a0 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb60e525640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6080cedf0 con 0x7fb60815df00 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb60cac2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb608160780 0x7fb608160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb60cac2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb608160780 0x7fb608160c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40222/0 (socket says 172.21.15.27:40222) 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb60cac2640 1 -- 172.21.15.27:0/1911070554 learned_addr learned my addr 172.21.15.27:0/1911070554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb607fff640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6080b95a0 0x7fb6080cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.222+0000 7fb6077fe640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb60815df00 0x7fb6081f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb60cac2640 1 -- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb60815df00 msgr2=0x7fb6081f0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb60cac2640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb60815df00 0x7fb6081f0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb60cac2640 1 -- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6080b95a0 msgr2=0x7fb6080cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb60cac2640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6080b95a0 0x7fb6080cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb60cac2640 1 -- 172.21.15.27:0/1911070554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6081f3c30 con 0x7fb608160780 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb607fff640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6080b95a0 0x7fb6080cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb60cac2640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb608160780 0x7fb608160c10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb600009fd0 tx=0x7fb60000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb604ff9640 1 -- 172.21.15.27:0/1911070554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60001a070 con 0x7fb608160780 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6081f3f10 con 0x7fb608160780 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb604ff9640 1 -- 172.21.15.27:0/1911070554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb600003bb0 con 0x7fb608160780 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb604ff9640 1 -- 172.21.15.27:0/1911070554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb600004030 con 0x7fb608160780 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.223+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6081f44d0 con 0x7fb608160780 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.224+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb608067ff0 con 0x7fb608160780 2024-03-06T07:24:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.224+0000 7fb604ff9640 1 -- 172.21.15.27:0/1911070554 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb60001d070 con 0x7fb608160780 2024-03-06T07:24:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.225+0000 7fb604ff9640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5f406b4d0 0x7fb5f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.225+0000 7fb604ff9640 1 -- 172.21.15.27:0/1911070554 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb60008a4e0 con 0x7fb608160780 2024-03-06T07:24:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.225+0000 7fb607fff640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5f406b4d0 0x7fb5f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.226+0000 7fb607fff640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5f406b4d0 0x7fb5f406d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8009e10 tx=0x7fb5f8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.229+0000 7fb604ff9640 1 -- 172.21.15.27:0/1911070554 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb600010070 con 0x7fb608160780 2024-03-06T07:24:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.410+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb60815cbc0 con 0x7fb608160780 2024-03-06T07:24:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.411+0000 7fb604ff9640 1 -- 172.21.15.27:0/1911070554 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fb600058c90 con 0x7fb608160780 2024-03-06T07:24:30.417 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":35,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:56.637219+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:53.309610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:53.987106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:30.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5f406b4d0 msgr2=0x7fb5f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5f406b4d0 0x7fb5f406d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8009e10 tx=0x7fb5f8009190 comp rx=0 tx=0).stop 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb608160780 msgr2=0x7fb608160c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb608160780 0x7fb608160c10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb600009fd0 tx=0x7fb60000d560 comp rx=0 tx=0).stop 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 shutdown_connections 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5f406b4d0 0x7fb5f406d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb608160780 0x7fb608160c10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb60815df00 0x7fb6081f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 --2- 172.21.15.27:0/1911070554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6080b95a0 0x7fb6080cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.414+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 >> 172.21.15.27:0/1911070554 conn(0x7fb608071960 msgr2=0x7fb6080b8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.415+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 shutdown_connections 2024-03-06T07:24:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.415+0000 7fb60e525640 1 -- 172.21.15.27:0/1911070554 wait complete. 2024-03-06T07:24:30.427 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:53.309610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:53.987106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:30.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.536+0000 7f56ff837640 1 Processor -- start 2024-03-06T07:24:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.536+0000 7f56ff837640 1 -- start start 2024-03-06T07:24:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.537+0000 7f56ff837640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80ceea0 0x7f56f80d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.537+0000 7f56ff837640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56f80b6d10 0x7f56f80b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.537+0000 7f56ff837640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f80d3740 con 0x7f56f815dd90 2024-03-06T07:24:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.537+0000 7f56ff837640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f80d3400 con 0x7f56f80b6d10 2024-03-06T07:24:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.537+0000 7f56ff837640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f80d3120 con 0x7f56f80ceea0 2024-03-06T07:24:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.537+0000 7f56fddd4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56f80b6d10 0x7f56f80b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.537+0000 7f56fddd4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56f80b6d10 0x7f56f80b71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40230/0 (socket says 172.21.15.27:40230) 2024-03-06T07:24:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.537+0000 7f56fddd4640 1 -- 172.21.15.27:0/1594573909 learned_addr learned my addr 172.21.15.27:0/1594573909 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.537+0000 7f56fcdd2640 1 --2- 172.21.15.27:0/1594573909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80ceea0 0x7f56f80d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.538+0000 7f56fddd4640 1 -- 172.21.15.27:0/1594573909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80ceea0 msgr2=0x7f56f80d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.538+0000 7f56fddd4640 1 --2- 172.21.15.27:0/1594573909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80ceea0 0x7f56f80d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.538+0000 7f56fddd4640 1 -- 172.21.15.27:0/1594573909 >> v1:172.21.15.69:6789/0 conn(0x7f56f815dd90 legacy=0x7f56f80d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.538+0000 7f56fddd4640 1 -- 172.21.15.27:0/1594573909 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f80d39e0 con 0x7f56f80b6d10 2024-03-06T07:24:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.538+0000 7f56fcdd2640 1 --2- 172.21.15.27:0/1594573909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80ceea0 0x7f56f80d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.538+0000 7f56fddd4640 1 --2- 172.21.15.27:0/1594573909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80b6d10 0x7f56f80b71a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f56e80098a0 tx=0x7f56e802ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb20f3dbf48a42d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56f7fff640 1 -- 172.21.15.27:0/1594573909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e802f820 con 0x7f56f80b6d10 2024-03-06T07:24:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56f7fff640 1 -- 172.21.15.27:0/1594573909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e802f9c0 con 0x7f56f80b6d10 2024-03-06T07:24:30.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56f7fff640 1 -- 172.21.15.27:0/1594573909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e802fd90 con 0x7f56f80b6d10 2024-03-06T07:24:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56ff837640 1 -- 172.21.15.27:0/1594573909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80b6d10 msgr2=0x7f56f80b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56ff837640 1 --2- 172.21.15.27:0/1594573909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80b6d10 0x7f56f80b71a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f56e80098a0 tx=0x7f56e802ba00 comp rx=0 tx=0).stop 2024-03-06T07:24:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56ff837640 1 -- 172.21.15.27:0/1594573909 shutdown_connections 2024-03-06T07:24:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56ff837640 1 --2- 172.21.15.27:0/1594573909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80b6d10 0x7f56f80b71a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56ff837640 1 --2- 172.21.15.27:0/1594573909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80ceea0 0x7f56f80d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56ff837640 1 -- 172.21.15.27:0/1594573909 >> 172.21.15.27:0/1594573909 conn(0x7f56f8071960 msgr2=0x7f56f8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56ff837640 1 -- 172.21.15.27:0/1594573909 shutdown_connections 2024-03-06T07:24:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.539+0000 7f56ff837640 1 -- 172.21.15.27:0/1594573909 wait complete. 2024-03-06T07:24:30.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.540+0000 7f56ff837640 1 Processor -- start 2024-03-06T07:24:30.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.540+0000 7f56ff837640 1 -- start start 2024-03-06T07:24:30.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56ff837640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80b6d10 0x7f56f8153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56ff837640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80ceea0 0x7f56f80ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56ff837640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815dd90 0x7f56f80b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56ff837640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f80d48f0 con 0x7f56f80ceea0 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56ff837640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f815e170 con 0x7f56f815dd90 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56ff837640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f80d4a70 con 0x7f56f80b6d10 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56fcdd2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80ceea0 0x7f56f80ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56fcdd2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80ceea0 0x7f56f80ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40232/0 (socket says 172.21.15.27:40232) 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56fcdd2640 1 -- 172.21.15.27:0/4165385472 learned_addr learned my addr 172.21.15.27:0/4165385472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.541+0000 7f56fddd4640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815dd90 0x7f56f80b90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56fd5d3640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80b6d10 0x7f56f8153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56fcdd2640 1 -- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80b6d10 msgr2=0x7f56f8153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56fcdd2640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80b6d10 0x7f56f8153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56fcdd2640 1 -- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815dd90 msgr2=0x7f56f80b90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56fcdd2640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815dd90 0x7f56f80b90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56fcdd2640 1 -- 172.21.15.27:0/4165385472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f80cdc00 con 0x7f56f80ceea0 2024-03-06T07:24:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56fcdd2640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80ceea0 0x7f56f80ccfb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00e7e0 tx=0x7f56ec00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56f5ffb640 1 -- 172.21.15.27:0/4165385472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec00f040 con 0x7f56f80ceea0 2024-03-06T07:24:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56f5ffb640 1 -- 172.21.15.27:0/4165385472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ec004030 con 0x7f56f80ceea0 2024-03-06T07:24:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f80cd920 con 0x7f56f80ceea0 2024-03-06T07:24:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56f5ffb640 1 -- 172.21.15.27:0/4165385472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec004400 con 0x7f56f80ceea0 2024-03-06T07:24:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.542+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f80c7b30 con 0x7f56f80ceea0 2024-03-06T07:24:30.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.543+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f8072a30 con 0x7f56f80ceea0 2024-03-06T07:24:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.544+0000 7f56f5ffb640 1 -- 172.21.15.27:0/4165385472 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56ec00c6d0 con 0x7f56f80ceea0 2024-03-06T07:24:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.544+0000 7f56f5ffb640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56e406b4d0 0x7f56e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.544+0000 7f56f5ffb640 1 -- 172.21.15.27:0/4165385472 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56ec089f30 con 0x7f56f80ceea0 2024-03-06T07:24:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.544+0000 7f56fd5d3640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56e406b4d0 0x7f56e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.545+0000 7f56fd5d3640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56e406b4d0 0x7f56e406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f56f0001d40 tx=0x7f56f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.549+0000 7f56f5ffb640 1 -- 172.21.15.27:0/4165385472 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56ec017020 con 0x7f56f80ceea0 2024-03-06T07:24:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.728+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f56f81542b0 con 0x7f56f80ceea0 2024-03-06T07:24:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.728+0000 7f56f5ffb640 1 -- 172.21.15.27:0/4165385472 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f56ec058e40 con 0x7f56f80ceea0 2024-03-06T07:24:30.730 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:24:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56e406b4d0 msgr2=0x7f56e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56e406b4d0 0x7f56e406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f56f0001d40 tx=0x7f56f0009040 comp rx=0 tx=0).stop 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80ceea0 msgr2=0x7f56f80ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80ceea0 0x7f56f80ccfb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00e7e0 tx=0x7f56ec00eca0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 shutdown_connections 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56e406b4d0 0x7f56e406d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f815dd90 0x7f56f80b90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f80ceea0 0x7f56f80ccfb0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 --2- 172.21.15.27:0/4165385472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56f80b6d10 0x7f56f8153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 >> 172.21.15.27:0/4165385472 conn(0x7f56f8071960 msgr2=0x7f56f80b4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 shutdown_connections 2024-03-06T07:24:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.731+0000 7f56ff837640 1 -- 172.21.15.27:0/4165385472 wait complete. 2024-03-06T07:24:30.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:30.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.852+0000 7f18f59c9640 1 Processor -- start 2024-03-06T07:24:30.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.852+0000 7f18f59c9640 1 -- start start 2024-03-06T07:24:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.853+0000 7f18f59c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f015f500 0x7f18f00ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.853+0000 7f18f59c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18f00cf740 0x7f18f00ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.853+0000 7f18f59c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18f00ced80 0x7f18f00cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.853+0000 7f18f59c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f015f8e0 con 0x7f18f00ced80 2024-03-06T07:24:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.853+0000 7f18f59c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f00c56c0 con 0x7f18f00cf740 2024-03-06T07:24:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.853+0000 7f18f59c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00c59a0 con 0x7f18f015f500 2024-03-06T07:24:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.853+0000 7f18ef7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18f00ced80 0x7f18f00cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.853+0000 7f18ef7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18f00ced80 0x7f18f00cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40244/0 (socket says 172.21.15.27:40244) 2024-03-06T07:24:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.853+0000 7f18ef7fe640 1 -- 172.21.15.27:0/2128366328 learned_addr learned my addr 172.21.15.27:0/2128366328 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18ee7fc640 1 --2- 172.21.15.27:0/2128366328 >> v2:172.21.15.69:3300/0 conn(0x7f18f00cf740 0x7f18f00ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18eeffd640 1 --2- 172.21.15.27:0/2128366328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f015f500 0x7f18f00ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18ef7fe640 1 -- 172.21.15.27:0/2128366328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f015f500 msgr2=0x7f18f00ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18ef7fe640 1 --2- 172.21.15.27:0/2128366328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f015f500 0x7f18f00ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18ef7fe640 1 -- 172.21.15.27:0/2128366328 >> v2:172.21.15.69:3300/0 conn(0x7f18f00cf740 msgr2=0x7f18f00ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18ef7fe640 1 --2- 172.21.15.27:0/2128366328 >> v2:172.21.15.69:3300/0 conn(0x7f18f00cf740 0x7f18f00ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18ef7fe640 1 -- 172.21.15.27:0/2128366328 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f00cd310 con 0x7f18f00ced80 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18ee7fc640 1 --2- 172.21.15.27:0/2128366328 >> v2:172.21.15.69:3300/0 conn(0x7f18f00cf740 0x7f18f00ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18eeffd640 1 --2- 172.21.15.27:0/2128366328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f015f500 0x7f18f00ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.854+0000 7f18ef7fe640 1 --2- 172.21.15.27:0/2128366328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f00ced80 0x7f18f00cfbf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f18f0071f10 tx=0x7f18e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f8961666ac4ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.855+0000 7f18edffb640 1 -- 172.21.15.27:0/2128366328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e802e070 con 0x7f18f00ced80 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.855+0000 7f18edffb640 1 -- 172.21.15.27:0/2128366328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e8037070 con 0x7f18f00ced80 2024-03-06T07:24:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.855+0000 7f18edffb640 1 -- 172.21.15.27:0/2128366328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e8031a00 con 0x7f18f00ced80 2024-03-06T07:24:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.855+0000 7f18f59c9640 1 -- 172.21.15.27:0/2128366328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f00ced80 msgr2=0x7f18f00cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.855+0000 7f18f59c9640 1 --2- 172.21.15.27:0/2128366328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f00ced80 0x7f18f00cfbf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f18f0071f10 tx=0x7f18e802b880 comp rx=0 tx=0).stop 2024-03-06T07:24:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.855+0000 7f18f59c9640 1 -- 172.21.15.27:0/2128366328 shutdown_connections 2024-03-06T07:24:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.855+0000 7f18f59c9640 1 --2- 172.21.15.27:0/2128366328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f00ced80 0x7f18f00cfbf0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.855+0000 7f18f59c9640 1 --2- 172.21.15.27:0/2128366328 >> v2:172.21.15.69:3300/0 conn(0x7f18f00cf740 0x7f18f00ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.855+0000 7f18f59c9640 1 --2- 172.21.15.27:0/2128366328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f015f500 0x7f18f00ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.856+0000 7f18f59c9640 1 -- 172.21.15.27:0/2128366328 >> 172.21.15.27:0/2128366328 conn(0x7f18f00b3700 msgr2=0x7f18f00b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.856+0000 7f18f59c9640 1 -- 172.21.15.27:0/2128366328 shutdown_connections 2024-03-06T07:24:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.856+0000 7f18f59c9640 1 -- 172.21.15.27:0/2128366328 wait complete. 2024-03-06T07:24:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.858+0000 7f18f59c9640 1 Processor -- start 2024-03-06T07:24:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.858+0000 7f18f59c9640 1 -- start start 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18f59c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18f00ced80 0x7f18f00b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18f59c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f00cf740 0x7f18f00ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18f59c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f015f500 0x7f18f01eea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18f59c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00b43d0 con 0x7f18f015f500 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18f59c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00cd890 con 0x7f18f00ced80 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18f59c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00d0130 con 0x7f18f00cf740 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18ef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f015f500 0x7f18f01eea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18ef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f015f500 0x7f18f01eea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40252/0 (socket says 172.21.15.27:40252) 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18ef7fe640 1 -- 172.21.15.27:0/3818514268 learned_addr learned my addr 172.21.15.27:0/3818514268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18eeffd640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18f00ced80 0x7f18f00b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.859+0000 7f18ee7fc640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f00cf740 0x7f18f00ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18ef7fe640 1 -- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f00cf740 msgr2=0x7f18f00ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18ef7fe640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f00cf740 0x7f18f00ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18ef7fe640 1 -- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18f00ced80 msgr2=0x7f18f00b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18ef7fe640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18f00ced80 0x7f18f00b6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18ef7fe640 1 -- 172.21.15.27:0/3818514268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f00c8ce0 con 0x7f18f015f500 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18ef7fe640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f015f500 0x7f18f01eea40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f18e80090b0 tx=0x7f18e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18e7fff640 1 -- 172.21.15.27:0/3818514268 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e8032880 con 0x7f18f015f500 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18e7fff640 1 -- 172.21.15.27:0/3818514268 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e8032a20 con 0x7f18f015f500 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18e7fff640 1 -- 172.21.15.27:0/3818514268 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e8032e10 con 0x7f18f015f500 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f00c8fc0 con 0x7f18f015f500 2024-03-06T07:24:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.860+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f00c9580 con 0x7f18f015f500 2024-03-06T07:24:30.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.861+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f009fcf0 con 0x7f18f015f500 2024-03-06T07:24:30.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.861+0000 7f18e7fff640 1 -- 172.21.15.27:0/3818514268 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18e804c060 con 0x7f18f015f500 2024-03-06T07:24:30.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.862+0000 7f18e7fff640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18d006b4d0 0x7f18d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.862+0000 7f18e7fff640 1 -- 172.21.15.27:0/3818514268 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18e80abbe0 con 0x7f18f015f500 2024-03-06T07:24:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.862+0000 7f18eeffd640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18d006b4d0 0x7f18d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.863+0000 7f18eeffd640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18d006b4d0 0x7f18d006d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f18e000f500 tx=0x7f18e0007770 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:30.866+0000 7f18e7fff640 1 -- 172.21.15.27:0/3818514268 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18e8040070 con 0x7f18f015f500 2024-03-06T07:24:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.047+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f18f0070860 con 0x7f18f015f500 2024-03-06T07:24:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.048+0000 7f18e7fff640 1 -- 172.21.15.27:0/3818514268 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v35) v1 ==== 104+0+11 (secure 0 0 0) 0x7f18e807a4a0 con 0x7f18f015f500 2024-03-06T07:24:31.049 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18d006b4d0 msgr2=0x7f18d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18d006b4d0 0x7f18d006d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f18e000f500 tx=0x7f18e0007770 comp rx=0 tx=0).stop 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f015f500 msgr2=0x7f18f01eea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f015f500 0x7f18f01eea40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f18e80090b0 tx=0x7f18e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 shutdown_connections 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18d006b4d0 0x7f18d006d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18f015f500 0x7f18f01eea40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18f00cf740 0x7f18f00ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 --2- 172.21.15.27:0/3818514268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18f00ced80 0x7f18f00b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 >> 172.21.15.27:0/3818514268 conn(0x7f18f00b3700 msgr2=0x7f18f015d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 shutdown_connections 2024-03-06T07:24:31.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.051+0000 7f18f59c9640 1 -- 172.21.15.27:0/3818514268 wait complete. 2024-03-06T07:24:31.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.172+0000 7f02f8f93640 1 Processor -- start 2024-03-06T07:24:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.172+0000 7f02f8f93640 1 -- start start 2024-03-06T07:24:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.173+0000 7f02f8f93640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02f40b2d30 0x7f02f40b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.173+0000 7f02f8f93640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b7820 0x7f02f40b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.173+0000 7f02f8f93640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f40d4960 con 0x7f02f4071540 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.173+0000 7f02f8f93640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f40d3230 con 0x7f02f40b2d30 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.173+0000 7f02f8f93640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f40d36c0 con 0x7f02f40b7820 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.174+0000 7f02f1d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02f40b2d30 0x7f02f40b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.174+0000 7f02f1d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02f40b2d30 0x7f02f40b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:47422/0 (socket says 172.21.15.27:47422) 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.174+0000 7f02f259c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f02f4071540 0x7f02f4071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:53718/0 (socket says 172.21.15.27:53718) 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.174+0000 7f02f1d9b640 1 -- 172.21.15.27:0/739287268 learned_addr learned my addr 172.21.15.27:0/739287268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.174+0000 7f02f2d9d640 1 --2- 172.21.15.27:0/739287268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b7820 0x7f02f40b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.174+0000 7f02f1d9b640 1 -- 172.21.15.27:0/739287268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b7820 msgr2=0x7f02f40b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.174+0000 7f02f1d9b640 1 --2- 172.21.15.27:0/739287268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b7820 0x7f02f40b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.174+0000 7f02f1d9b640 1 -- 172.21.15.27:0/739287268 >> v1:172.21.15.69:6789/0 conn(0x7f02f4071540 legacy=0x7f02f4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.174+0000 7f02f1d9b640 1 -- 172.21.15.27:0/739287268 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40d2620 con 0x7f02f40b2d30 2024-03-06T07:24:31.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.175+0000 7f02f1d9b640 1 --2- 172.21.15.27:0/739287268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f40b2d30 0x7f02f40b72e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009870 tx=0x7f02e802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=831cc2491dbdd622 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.175+0000 7f02f159a640 1 -- 172.21.15.27:0/739287268 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8034070 con 0x7f02f40b2d30 2024-03-06T07:24:31.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.175+0000 7f02f159a640 1 -- 172.21.15.27:0/739287268 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e8030840 con 0x7f02f40b2d30 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.175+0000 7f02f159a640 1 -- 172.21.15.27:0/739287268 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8030c10 con 0x7f02f40b2d30 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.176+0000 7f02f8f93640 1 -- 172.21.15.27:0/739287268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f40b2d30 msgr2=0x7f02f40b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.176+0000 7f02f8f93640 1 --2- 172.21.15.27:0/739287268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f40b2d30 0x7f02f40b72e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009870 tx=0x7f02e802b980 comp rx=0 tx=0).stop 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.176+0000 7f02f8f93640 1 -- 172.21.15.27:0/739287268 shutdown_connections 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.176+0000 7f02f8f93640 1 --2- 172.21.15.27:0/739287268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f40b2d30 0x7f02f40b72e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.176+0000 7f02f8f93640 1 --2- 172.21.15.27:0/739287268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b7820 0x7f02f40b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.176+0000 7f02f8f93640 1 -- 172.21.15.27:0/739287268 >> 172.21.15.27:0/739287268 conn(0x7f02f40d3d60 msgr2=0x7f02f40d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.176+0000 7f02f8f93640 1 -- 172.21.15.27:0/739287268 shutdown_connections 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.176+0000 7f02f8f93640 1 -- 172.21.15.27:0/739287268 wait complete. 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.177+0000 7f02f8f93640 1 Processor -- start 2024-03-06T07:24:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.177+0000 7f02f8f93640 1 -- start start 2024-03-06T07:24:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f8f93640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f4071540 0x7f02f41e3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f8f93640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b2d30 0x7f02f41f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f8f93640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f40b7820 0x7f02f41e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f8f93640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f40d2ec0 con 0x7f02f40b7820 2024-03-06T07:24:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f8f93640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f40d2be0 con 0x7f02f4071540 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f8f93640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f40d22f0 con 0x7f02f40b2d30 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f40b7820 0x7f02f41e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f40b7820 0x7f02f41e4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40264/0 (socket says 172.21.15.27:40264) 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f2d9d640 1 -- 172.21.15.27:0/798680106 learned_addr learned my addr 172.21.15.27:0/798680106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f1d9b640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b2d30 0x7f02f41f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f259c640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f4071540 0x7f02f41e3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f2d9d640 1 -- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b2d30 msgr2=0x7f02f41f3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f2d9d640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b2d30 0x7f02f41f3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f2d9d640 1 -- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f4071540 msgr2=0x7f02f41e3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f2d9d640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f4071540 0x7f02f41e3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.178+0000 7f02f2d9d640 1 -- 172.21.15.27:0/798680106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40c7520 con 0x7f02f40b7820 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.179+0000 7f02f2d9d640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f40b7820 0x7f02f41e4d30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009dd0 tx=0x7f02ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.179+0000 7f02e2ffd640 1 -- 172.21.15.27:0/798680106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec018020 con 0x7f02f40b7820 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.179+0000 7f02e2ffd640 1 -- 172.21.15.27:0/798680106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ec014480 con 0x7f02f40b7820 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.179+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f40c7800 con 0x7f02f40b7820 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.179+0000 7f02e2ffd640 1 -- 172.21.15.27:0/798680106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec014870 con 0x7f02f40b7820 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.179+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f40c7dc0 con 0x7f02f40b7820 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.180+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f4072070 con 0x7f02f40b7820 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.180+0000 7f02e2ffd640 1 -- 172.21.15.27:0/798680106 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02ec005d10 con 0x7f02f40b7820 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.181+0000 7f02e2ffd640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02d406b4d0 0x7f02d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.181+0000 7f02e2ffd640 1 -- 172.21.15.27:0/798680106 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02ec08abf0 con 0x7f02f40b7820 2024-03-06T07:24:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.181+0000 7f02f259c640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02d406b4d0 0x7f02d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.182+0000 7f02f259c640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02d406b4d0 0x7f02d406d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f02f40c9820 tx=0x7f02e4007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.185+0000 7f02e2ffd640 1 -- 172.21.15.27:0/798680106 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02ec022070 con 0x7f02f40b7820 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.364+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02f404f410 con 0x7f02f40b7820 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.365+0000 7f02e2ffd640 1 -- 172.21.15.27:0/798680106 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v35) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02ec010040 con 0x7f02f40b7820 2024-03-06T07:24:31.366 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:24:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02d406b4d0 msgr2=0x7f02d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02d406b4d0 0x7f02d406d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f02f40c9820 tx=0x7f02e4007e80 comp rx=0 tx=0).stop 2024-03-06T07:24:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f40b7820 msgr2=0x7f02f41e4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f40b7820 0x7f02f41e4d30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009dd0 tx=0x7f02ec00c490 comp rx=0 tx=0).stop 2024-03-06T07:24:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 shutdown_connections 2024-03-06T07:24:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02d406b4d0 0x7f02d406d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f40b7820 0x7f02f41e4d30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f40b2d30 0x7f02f41f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 --2- 172.21.15.27:0/798680106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f4071540 0x7f02f41e3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.367+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 >> 172.21.15.27:0/798680106 conn(0x7f02f40d3d60 msgr2=0x7f02f415d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.368+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 shutdown_connections 2024-03-06T07:24:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.368+0000 7f02f8f93640 1 -- 172.21.15.27:0/798680106 wait complete. 2024-03-06T07:24:31.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:31.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.487+0000 7f35aeba1640 1 Processor -- start 2024-03-06T07:24:31.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.487+0000 7f35aeba1640 1 -- start start 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35aeba1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35a8071540 0x7f35a80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35aeba1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a80b2b50 0x7f35a80b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35aeba1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a80ce610 con 0x7f35a80d3400 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35aeba1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a80d2f70 con 0x7f35a8071540 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35aeba1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a80d4be0 con 0x7f35a80b2b50 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35ac93d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35a8071540 0x7f35a80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35a7fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f35a80d3400 0x7f35a8071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:53730/0 (socket says 172.21.15.27:53730) 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35ad13e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a80b2b50 0x7f35a80b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35a7fff640 1 -- 172.21.15.27:0/3696606630 learned_addr learned my addr 172.21.15.27:0/3696606630 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35ac93d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35a8071540 0x7f35a80b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:47448/0 (socket says 172.21.15.27:47448) 2024-03-06T07:24:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35ac93d640 1 -- 172.21.15.27:0/3696606630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a80b2b50 msgr2=0x7f35a80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35ac93d640 1 --2- 172.21.15.27:0/3696606630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a80b2b50 0x7f35a80b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35ac93d640 1 -- 172.21.15.27:0/3696606630 >> v1:172.21.15.69:6789/0 conn(0x7f35a80d3400 legacy=0x7f35a8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35ac93d640 1 -- 172.21.15.27:0/3696606630 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a80d2920 con 0x7f35a8071540 2024-03-06T07:24:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.488+0000 7f35a77fe640 1 -- 172.21.15.27:0/3696606630 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287914875 0 0) 0x7f35a80ce610 con 0x7f35a80d3400 2024-03-06T07:24:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.490+0000 7f35ac93d640 1 --2- 172.21.15.27:0/3696606630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35a8071540 0x7f35a80b30a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f359c0097a0 tx=0x7f359c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed8d493ef7562f04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.490+0000 7f35a77fe640 1 -- 172.21.15.27:0/3696606630 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c035020 con 0x7f35a8071540 2024-03-06T07:24:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.490+0000 7f35a77fe640 1 -- 172.21.15.27:0/3696606630 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359c031830 con 0x7f35a8071540 2024-03-06T07:24:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.490+0000 7f35a77fe640 1 -- 172.21.15.27:0/3696606630 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c031c00 con 0x7f35a8071540 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.491+0000 7f35aeba1640 1 -- 172.21.15.27:0/3696606630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35a8071540 msgr2=0x7f35a80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.491+0000 7f35aeba1640 1 --2- 172.21.15.27:0/3696606630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35a8071540 0x7f35a80b30a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f359c0097a0 tx=0x7f359c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.491+0000 7f35aeba1640 1 -- 172.21.15.27:0/3696606630 shutdown_connections 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.491+0000 7f35aeba1640 1 --2- 172.21.15.27:0/3696606630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a80b2b50 0x7f35a80b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.491+0000 7f35aeba1640 1 --2- 172.21.15.27:0/3696606630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35a8071540 0x7f35a80b30a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.491+0000 7f35aeba1640 1 -- 172.21.15.27:0/3696606630 >> 172.21.15.27:0/3696606630 conn(0x7f35a80d3ca0 msgr2=0x7f35a80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.491+0000 7f35aeba1640 1 -- 172.21.15.27:0/3696606630 shutdown_connections 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.491+0000 7f35aeba1640 1 -- 172.21.15.27:0/3696606630 wait complete. 2024-03-06T07:24:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.492+0000 7f35aeba1640 1 Processor -- start 2024-03-06T07:24:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.492+0000 7f35aeba1640 1 -- start start 2024-03-06T07:24:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35aeba1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a8071540 0x7f35a80cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35aeba1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35a80b2b50 0x7f35a81e8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35aeba1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35a80d3400 0x7f35a81493b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35aeba1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a815e370 con 0x7f35a80d3400 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35aeba1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a80ce8f0 con 0x7f35a80b2b50 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35aeba1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a80cbdd0 con 0x7f35a8071540 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ad13e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35a80d3400 0x7f35a81493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ad13e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35a80d3400 0x7f35a81493b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40274/0 (socket says 172.21.15.27:40274) 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ad13e640 1 -- 172.21.15.27:0/3055074702 learned_addr learned my addr 172.21.15.27:0/3055074702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ac93d640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a8071540 0x7f35a80cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35a7fff640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35a80b2b50 0x7f35a81e8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ad13e640 1 -- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a8071540 msgr2=0x7f35a80cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ad13e640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a8071540 0x7f35a80cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ad13e640 1 -- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35a80b2b50 msgr2=0x7f35a81e8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ad13e640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35a80b2b50 0x7f35a81e8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ad13e640 1 -- 172.21.15.27:0/3055074702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a80cb1c0 con 0x7f35a80d3400 2024-03-06T07:24:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.493+0000 7f35ad13e640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35a80d3400 0x7f35a81493b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009dd0 tx=0x7f35a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.494+0000 7f35a57fa640 1 -- 172.21.15.27:0/3055074702 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0018020 con 0x7f35a80d3400 2024-03-06T07:24:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.494+0000 7f35a57fa640 1 -- 172.21.15.27:0/3055074702 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a0014480 con 0x7f35a80d3400 2024-03-06T07:24:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.494+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a80cb4a0 con 0x7f35a80d3400 2024-03-06T07:24:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.494+0000 7f35a57fa640 1 -- 172.21.15.27:0/3055074702 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0014870 con 0x7f35a80d3400 2024-03-06T07:24:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.494+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a80cba60 con 0x7f35a80d3400 2024-03-06T07:24:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.495+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a814f720 con 0x7f35a80d3400 2024-03-06T07:24:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.495+0000 7f35a57fa640 1 -- 172.21.15.27:0/3055074702 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35a0005d10 con 0x7f35a80d3400 2024-03-06T07:24:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.496+0000 7f35a57fa640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f358c06b4d0 0x7f358c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.496+0000 7f35a57fa640 1 -- 172.21.15.27:0/3055074702 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35a008ac70 con 0x7f35a80d3400 2024-03-06T07:24:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.496+0000 7f35ac93d640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f358c06b4d0 0x7f358c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.496+0000 7f35ac93d640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f358c06b4d0 0x7f358c06d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f359c036000 tx=0x7f359c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.505+0000 7f35a57fa640 1 -- 172.21.15.27:0/3055074702 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35a0022070 con 0x7f35a80d3400 2024-03-06T07:24:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.727+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35a8068700 con 0x7f35a80d3400 2024-03-06T07:24:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.728+0000 7f35a57fa640 1 -- 172.21.15.27:0/3055074702 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f35a0010040 con 0x7f35a80d3400 2024-03-06T07:24:31.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:31.729 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f358c06b4d0 msgr2=0x7f358c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f358c06b4d0 0x7f358c06d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f359c036000 tx=0x7f359c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35a80d3400 msgr2=0x7f35a81493b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35a80d3400 0x7f35a81493b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009dd0 tx=0x7f35a000c490 comp rx=0 tx=0).stop 2024-03-06T07:24:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 shutdown_connections 2024-03-06T07:24:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f358c06b4d0 0x7f358c06d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35a80d3400 0x7f35a81493b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35a80b2b50 0x7f35a81e8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 --2- 172.21.15.27:0/3055074702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35a8071540 0x7f35a80cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 >> 172.21.15.27:0/3055074702 conn(0x7f35a80d3ca0 msgr2=0x7f35a8071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 shutdown_connections 2024-03-06T07:24:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.730+0000 7f35aeba1640 1 -- 172.21.15.27:0/3055074702 wait complete. 2024-03-06T07:24:31.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:31.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.848+0000 7fc657a0e640 1 Processor -- start 2024-03-06T07:24:31.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.848+0000 7fc657a0e640 1 -- start start 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc657a0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d02e0 0x7fc6500cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc657a0e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6500b4260 0x7fc6500b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc657a0e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6500d06c0 0x7fc6500b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc657a0e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6500b0b80 con 0x7fc6500d06c0 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc657a0e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6500aeec0 con 0x7fc6500b4260 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc657a0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500b0dd0 con 0x7fc6500d02e0 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc655fab640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6500d06c0 0x7fc6500b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc655fab640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6500d06c0 0x7fc6500b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40286/0 (socket says 172.21.15.27:40286) 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc655fab640 1 -- 172.21.15.27:0/1429143951 learned_addr learned my addr 172.21.15.27:0/1429143951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc654fa9640 1 --2- 172.21.15.27:0/1429143951 >> v2:172.21.15.69:3300/0 conn(0x7fc6500b4260 0x7fc6500b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc6557aa640 1 --2- 172.21.15.27:0/1429143951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d02e0 0x7fc6500cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc655fab640 1 -- 172.21.15.27:0/1429143951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d02e0 msgr2=0x7fc6500cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc655fab640 1 --2- 172.21.15.27:0/1429143951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d02e0 0x7fc6500cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc655fab640 1 -- 172.21.15.27:0/1429143951 >> v2:172.21.15.69:3300/0 conn(0x7fc6500b4260 msgr2=0x7fc6500b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc655fab640 1 --2- 172.21.15.27:0/1429143951 >> v2:172.21.15.69:3300/0 conn(0x7fc6500b4260 0x7fc6500b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.849+0000 7fc655fab640 1 -- 172.21.15.27:0/1429143951 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6500b10b0 con 0x7fc6500d06c0 2024-03-06T07:24:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.850+0000 7fc655fab640 1 --2- 172.21.15.27:0/1429143951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d06c0 0x7fc6500b1a40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc65006eac0 tx=0x7fc64802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a83c138f62c2c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.850+0000 7fc647fff640 1 -- 172.21.15.27:0/1429143951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64802cb00 con 0x7fc6500d06c0 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.850+0000 7fc647fff640 1 -- 172.21.15.27:0/1429143951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64802cca0 con 0x7fc6500d06c0 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.850+0000 7fc647fff640 1 -- 172.21.15.27:0/1429143951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648034610 con 0x7fc6500d06c0 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.851+0000 7fc657a0e640 1 -- 172.21.15.27:0/1429143951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d06c0 msgr2=0x7fc6500b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.851+0000 7fc657a0e640 1 --2- 172.21.15.27:0/1429143951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d06c0 0x7fc6500b1a40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc65006eac0 tx=0x7fc64802bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.851+0000 7fc657a0e640 1 -- 172.21.15.27:0/1429143951 shutdown_connections 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.851+0000 7fc657a0e640 1 --2- 172.21.15.27:0/1429143951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d06c0 0x7fc6500b1a40 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.851+0000 7fc657a0e640 1 --2- 172.21.15.27:0/1429143951 >> v2:172.21.15.69:3300/0 conn(0x7fc6500b4260 0x7fc6500b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.851+0000 7fc657a0e640 1 --2- 172.21.15.27:0/1429143951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d02e0 0x7fc6500cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.851+0000 7fc657a0e640 1 -- 172.21.15.27:0/1429143951 >> 172.21.15.27:0/1429143951 conn(0x7fc6500b36d0 msgr2=0x7fc6500b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.851+0000 7fc657a0e640 1 -- 172.21.15.27:0/1429143951 shutdown_connections 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.851+0000 7fc657a0e640 1 -- 172.21.15.27:0/1429143951 wait complete. 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.852+0000 7fc657a0e640 1 Processor -- start 2024-03-06T07:24:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.852+0000 7fc657a0e640 1 -- start start 2024-03-06T07:24:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc657a0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6500b4260 0x7fc65009bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc657a0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d02e0 0x7fc65009d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc657a0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d06c0 0x7fc650153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc657a0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500b2100 con 0x7fc6500d02e0 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc657a0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500b1f80 con 0x7fc6500b4260 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc657a0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500b2280 con 0x7fc6500d06c0 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc654fa9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d02e0 0x7fc65009d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc654fa9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d02e0 0x7fc65009d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40300/0 (socket says 172.21.15.27:40300) 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc654fa9640 1 -- 172.21.15.27:0/2773736722 learned_addr learned my addr 172.21.15.27:0/2773736722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc655fab640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d06c0 0x7fc650153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc6557aa640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6500b4260 0x7fc65009bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc654fa9640 1 -- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d06c0 msgr2=0x7fc650153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc654fa9640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d06c0 0x7fc650153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc654fa9640 1 -- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6500b4260 msgr2=0x7fc65009bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc654fa9640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6500b4260 0x7fc65009bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc654fa9640 1 -- 172.21.15.27:0/2773736722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65009c0a0 con 0x7fc6500d02e0 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc6557aa640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6500b4260 0x7fc65009bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.853+0000 7fc655fab640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d06c0 0x7fc650153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.854+0000 7fc654fa9640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d02e0 0x7fc65009d140 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc64000e840 tx=0x7fc64000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.854+0000 7fc645ffb640 1 -- 172.21.15.27:0/2773736722 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640017070 con 0x7fc6500d02e0 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.854+0000 7fc645ffb640 1 -- 172.21.15.27:0/2773736722 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64000c730 con 0x7fc6500d02e0 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.854+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65009c380 con 0x7fc6500d02e0 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.854+0000 7fc645ffb640 1 -- 172.21.15.27:0/2773736722 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64000cb00 con 0x7fc6500d02e0 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.854+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65009c940 con 0x7fc6500d02e0 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.855+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc650074c70 con 0x7fc6500d02e0 2024-03-06T07:24:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.855+0000 7fc645ffb640 1 -- 172.21.15.27:0/2773736722 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc64001a070 con 0x7fc6500d02e0 2024-03-06T07:24:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.856+0000 7fc645ffb640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc63406b4d0 0x7fc63406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.856+0000 7fc645ffb640 1 -- 172.21.15.27:0/2773736722 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc640089bc0 con 0x7fc6500d02e0 2024-03-06T07:24:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.856+0000 7fc6557aa640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc63406b4d0 0x7fc63406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.857+0000 7fc6557aa640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc63406b4d0 0x7fc63406d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009a20 tx=0x7fc64c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:31.860+0000 7fc645ffb640 1 -- 172.21.15.27:0/2773736722 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc650074c70 con 0x7fc6500d02e0 2024-03-06T07:24:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.043+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc650068700 con 0x7fc6500d02e0 2024-03-06T07:24:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.044+0000 7fc645ffb640 1 -- 172.21.15.27:0/2773736722 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fc640058ad0 con 0x7fc6500d02e0 2024-03-06T07:24:32.046 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:32.046 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":35,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:56.637219+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:53.309610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:53.987106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc63406b4d0 msgr2=0x7fc63406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc63406b4d0 0x7fc63406d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009a20 tx=0x7fc64c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d02e0 msgr2=0x7fc65009d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d02e0 0x7fc65009d140 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc64000e840 tx=0x7fc64000ed00 comp rx=0 tx=0).stop 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 shutdown_connections 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc63406b4d0 0x7fc63406d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6500d06c0 0x7fc650153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6500d02e0 0x7fc65009d140 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 --2- 172.21.15.27:0/2773736722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6500b4260 0x7fc65009bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 >> 172.21.15.27:0/2773736722 conn(0x7fc6500b36d0 msgr2=0x7fc6500cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 shutdown_connections 2024-03-06T07:24:32.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.047+0000 7fc657a0e640 1 -- 172.21.15.27:0/2773736722 wait complete. 2024-03-06T07:24:32.060 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:32.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.165+0000 7f4fafe64640 1 Processor -- start 2024-03-06T07:24:32.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.165+0000 7f4fafe64640 1 -- start start 2024-03-06T07:24:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fafe64640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fa806e700 0x7f4fa815ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fafe64640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4fa8072660 0x7f4fa806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fafe64640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 0x7f4fa8071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fafe64640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa80b4160 con 0x7f4fa806e700 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fafe64640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa80b5b40 con 0x7f4fa8072660 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fafe64640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa80b3ba0 con 0x7f4fa80b55d0 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fadc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fa806e700 0x7f4fa815ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fadc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fa806e700 0x7f4fa815ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40308/0 (socket says 172.21.15.27:40308) 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fadc00640 1 -- 172.21.15.27:0/550816944 learned_addr learned my addr 172.21.15.27:0/550816944 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fad3ff640 1 --2- 172.21.15.27:0/550816944 >> v2:172.21.15.69:3300/0 conn(0x7f4fa8072660 0x7f4fa806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.167+0000 7f4fae401640 1 --2- 172.21.15.27:0/550816944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 0x7f4fa8071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.168+0000 7f4fadc00640 1 -- 172.21.15.27:0/550816944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 msgr2=0x7f4fa8071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.168+0000 7f4fadc00640 1 --2- 172.21.15.27:0/550816944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 0x7f4fa8071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.168+0000 7f4fadc00640 1 -- 172.21.15.27:0/550816944 >> v2:172.21.15.69:3300/0 conn(0x7f4fa8072660 msgr2=0x7f4fa806d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.168+0000 7f4fadc00640 1 --2- 172.21.15.27:0/550816944 >> v2:172.21.15.69:3300/0 conn(0x7f4fa8072660 0x7f4fa806d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.168+0000 7f4fadc00640 1 -- 172.21.15.27:0/550816944 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa80b3e80 con 0x7f4fa806e700 2024-03-06T07:24:32.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.168+0000 7f4fae401640 1 --2- 172.21.15.27:0/550816944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 0x7f4fa8071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.169+0000 7f4fadc00640 1 --2- 172.21.15.27:0/550816944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 0x7f4fa815ef80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8147ce0 tx=0x7f4fa402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c005d0b67caef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.169+0000 7f4facbfe640 1 -- 172.21.15.27:0/550816944 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa402ca70 con 0x7f4fa806e700 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.169+0000 7f4facbfe640 1 -- 172.21.15.27:0/550816944 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa402cc10 con 0x7f4fa806e700 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.169+0000 7f4facbfe640 1 -- 172.21.15.27:0/550816944 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40343f0 con 0x7f4fa806e700 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.170+0000 7f4fafe64640 1 -- 172.21.15.27:0/550816944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 msgr2=0x7f4fa815ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.170+0000 7f4fafe64640 1 --2- 172.21.15.27:0/550816944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 0x7f4fa815ef80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8147ce0 tx=0x7f4fa402bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.170+0000 7f4fafe64640 1 -- 172.21.15.27:0/550816944 shutdown_connections 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.170+0000 7f4fafe64640 1 --2- 172.21.15.27:0/550816944 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 0x7f4fa8071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.170+0000 7f4fafe64640 1 --2- 172.21.15.27:0/550816944 >> v2:172.21.15.69:3300/0 conn(0x7f4fa8072660 0x7f4fa806d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.170+0000 7f4fafe64640 1 --2- 172.21.15.27:0/550816944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 0x7f4fa815ef80 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.170+0000 7f4fafe64640 1 -- 172.21.15.27:0/550816944 >> 172.21.15.27:0/550816944 conn(0x7f4fa80b2720 msgr2=0x7f4fa80b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.170+0000 7f4fafe64640 1 -- 172.21.15.27:0/550816944 shutdown_connections 2024-03-06T07:24:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.170+0000 7f4fafe64640 1 -- 172.21.15.27:0/550816944 wait complete. 2024-03-06T07:24:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.172+0000 7f4fafe64640 1 Processor -- start 2024-03-06T07:24:32.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.172+0000 7f4fafe64640 1 -- start start 2024-03-06T07:24:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fafe64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 0x7f4fa80ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fafe64640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fa8072660 0x7f4fa815b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fafe64640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 0x7f4fa80aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fafe64640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa806ec60 con 0x7f4fa806e700 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fafe64640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa806eae0 con 0x7f4fa8072660 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fafe64640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa8071ec0 con 0x7f4fa80b55d0 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fadc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 0x7f4fa80ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fadc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 0x7f4fa80ae5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40322/0 (socket says 172.21.15.27:40322) 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fadc00640 1 -- 172.21.15.27:0/1093410727 learned_addr learned my addr 172.21.15.27:0/1093410727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fad3ff640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fa8072660 0x7f4fa815b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fae401640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 0x7f4fa80aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fadc00640 1 -- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 msgr2=0x7f4fa80aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fadc00640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 0x7f4fa80aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fadc00640 1 -- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fa8072660 msgr2=0x7f4fa815b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fadc00640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fa8072660 0x7f4fa815b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fadc00640 1 -- 172.21.15.27:0/1093410727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa80af610 con 0x7f4fa806e700 2024-03-06T07:24:32.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4fadc00640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 0x7f4fa80ae5e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4036040 tx=0x7f4fa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.174+0000 7f4f9e7fc640 1 -- 172.21.15.27:0/1093410727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4043020 con 0x7f4fa806e700 2024-03-06T07:24:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.175+0000 7f4f9e7fc640 1 -- 172.21.15.27:0/1093410727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa403e450 con 0x7f4fa806e700 2024-03-06T07:24:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.175+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa80adb00 con 0x7f4fa806e700 2024-03-06T07:24:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.175+0000 7f4f9e7fc640 1 -- 172.21.15.27:0/1093410727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4009510 con 0x7f4fa806e700 2024-03-06T07:24:32.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.175+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa80ae0c0 con 0x7f4fa806e700 2024-03-06T07:24:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.176+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa8067ff0 con 0x7f4fa806e700 2024-03-06T07:24:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.176+0000 7f4f9e7fc640 1 -- 172.21.15.27:0/1093410727 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fa4046020 con 0x7f4fa806e700 2024-03-06T07:24:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.176+0000 7f4f9e7fc640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.177+0000 7f4f9e7fc640 1 -- 172.21.15.27:0/1093410727 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fa40ac830 con 0x7f4fa806e700 2024-03-06T07:24:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.177+0000 7f4fad3ff640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.177+0000 7f4fad3ff640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8070080 tx=0x7f4f98009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.181+0000 7f4f9e7fc640 1 -- 172.21.15.27:0/1093410727 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fa404d030 con 0x7f4fa806e700 2024-03-06T07:24:32.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.401+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fa80ae290 con 0x7f4fa806e700 2024-03-06T07:24:32.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.402+0000 7f4f9e7fc640 1 -- 172.21.15.27:0/1093410727 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f4fa407b070 con 0x7f4fa806e700 2024-03-06T07:24:32.404 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:32.404 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:32.404 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4f8c06b4d0 msgr2=0x7f4f8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8070080 tx=0x7f4f98009190 comp rx=0 tx=0).stop 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 msgr2=0x7f4fa80ae5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 0x7f4fa80ae5e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4036040 tx=0x7f4fa4004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 shutdown_connections 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4fa80b55d0 0x7f4fa80aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fa8072660 0x7f4fa815b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 --2- 172.21.15.27:0/1093410727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa806e700 0x7f4fa80ae5e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 >> 172.21.15.27:0/1093410727 conn(0x7f4fa80b2720 msgr2=0x7f4fa8157ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 shutdown_connections 2024-03-06T07:24:32.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.405+0000 7f4fafe64640 1 -- 172.21.15.27:0/1093410727 wait complete. 2024-03-06T07:24:32.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.523+0000 7ff61333e640 1 Processor -- start 2024-03-06T07:24:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.523+0000 7ff61333e640 1 -- start start 2024-03-06T07:24:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff61333e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff60c0b9a10 0x7ff60c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff61333e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60c0cea30 con 0x7ff60c0b95a0 2024-03-06T07:24:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff61333e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60c0cf3c0 con 0x7ff60c15df00 2024-03-06T07:24:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff61333e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c0cf0e0 con 0x7ff60c0b9a10 2024-03-06T07:24:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff6108d9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff60c0b95a0 0x7ff60c0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42962/0 (socket says 172.21.15.27:42962) 2024-03-06T07:24:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff6108d9640 1 -- 172.21.15.27:0/3370658095 learned_addr learned my addr 172.21.15.27:0/3370658095 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff6118db640 1 --2- 172.21.15.27:0/3370658095 >> v2:172.21.15.69:3300/0 conn(0x7ff60c0b9a10 0x7ff60c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024205643 0 0) 0x7ff60c0cea30 con 0x7ff60c0b95a0 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f8003520 con 0x7ff60c0b95a0 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3707069571 0 0) 0x7ff5f8003520 con 0x7ff60c0b95a0 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 >> v2:172.21.15.69:3300/0 conn(0x7ff60c0b9a10 msgr2=0x7ff60c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 --2- 172.21.15.27:0/3370658095 >> v2:172.21.15.69:3300/0 conn(0x7ff60c0b9a10 0x7ff60c0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 >> v1:172.21.15.69:6789/0 conn(0x7ff60c15df00 legacy=0x7ff60c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c0ce470 con 0x7ff60c0b95a0 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7ff5fc001670 con 0x7ff60c0b95a0 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 >> v1:172.21.15.27:6789/0 conn(0x7ff60c0b95a0 legacy=0x7ff60c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f80039b0 0x7ff5f8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.524+0000 7ff603fff640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8004800 0x7ff5f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff603fff640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f80051c0 0x7ff5f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0cf0e0 con 0x7ff5f8004800 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f800c090 con 0x7ff5f80039b0 2024-03-06T07:24:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f800c290 con 0x7ff5f80051c0 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6108d9640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8004800 0x7ff5f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6110da640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f80039b0 0x7ff5f8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6118db640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f80051c0 0x7ff5f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6108d9640 1 -- 172.21.15.27:0/3370658095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f80051c0 msgr2=0x7ff5f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6108d9640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f80051c0 0x7ff5f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6108d9640 1 -- 172.21.15.27:0/3370658095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f80039b0 msgr2=0x7ff5f8003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6108d9640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f80039b0 0x7ff5f8003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6118db640 1 -- 172.21.15.27:0/3370658095 reap_dead start 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6108d9640 1 -- 172.21.15.27:0/3370658095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8003520 con 0x7ff5f8004800 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.525+0000 7ff6110da640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f80039b0 0x7ff5f8003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.526+0000 7ff6108d9640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8004800 0x7ff5f8004c80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00c240 tx=0x7ff5f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f58a228ea9b4340 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.526+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc007cf0 con 0x7ff5f8004800 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.526+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5fc030070 con 0x7ff5f8004800 2024-03-06T07:24:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.526+0000 7ff603fff640 1 -- 172.21.15.27:0/3370658095 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc005d30 con 0x7ff5f8004800 2024-03-06T07:24:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.526+0000 7ff61333e640 1 -- 172.21.15.27:0/3370658095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8004800 msgr2=0x7ff5f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.526+0000 7ff61333e640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8004800 0x7ff5f8004c80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00c240 tx=0x7ff5f8005830 comp rx=0 tx=0).stop 2024-03-06T07:24:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.526+0000 7ff61333e640 1 -- 172.21.15.27:0/3370658095 shutdown_connections 2024-03-06T07:24:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.526+0000 7ff61333e640 1 --2- 172.21.15.27:0/3370658095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8004800 0x7ff5f8004c80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.526+0000 7ff61333e640 1 -- 172.21.15.27:0/3370658095 >> 172.21.15.27:0/3370658095 conn(0x7ff60c071960 msgr2=0x7ff60c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.527+0000 7ff61333e640 1 -- 172.21.15.27:0/3370658095 shutdown_connections 2024-03-06T07:24:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.527+0000 7ff61333e640 1 -- 172.21.15.27:0/3370658095 wait complete. 2024-03-06T07:24:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.528+0000 7ff61333e640 1 Processor -- start 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.528+0000 7ff61333e640 1 -- start start 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.528+0000 7ff61333e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f8004800 0x7ff60c0cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.528+0000 7ff61333e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c158670 0x7ff60c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.528+0000 7ff61333e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff60c154e50 0x7ff60c1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.528+0000 7ff61333e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0cedf0 con 0x7ff60c158670 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.528+0000 7ff61333e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0cec70 con 0x7ff5f8004800 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.528+0000 7ff61333e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0d25b0 con 0x7ff60c154e50 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6108d9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c158670 0x7ff60c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6108d9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c158670 0x7ff60c0b4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40348/0 (socket says 172.21.15.27:40348) 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6108d9640 1 -- 172.21.15.27:0/1749142726 learned_addr learned my addr 172.21.15.27:0/1749142726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6110da640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f8004800 0x7ff60c0cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6118db640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff60c154e50 0x7ff60c1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6108d9640 1 -- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff60c154e50 msgr2=0x7ff60c1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6108d9640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff60c154e50 0x7ff60c1552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6108d9640 1 -- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f8004800 msgr2=0x7ff60c0cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6108d9640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f8004800 0x7ff60c0cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6108d9640 1 -- 172.21.15.27:0/1749142726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c155810 con 0x7ff60c158670 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff6108d9640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c158670 0x7ff60c0b4eb0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00c360 tx=0x7ff60c0d0620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff601ffb640 1 -- 172.21.15.27:0/1749142726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc045070 con 0x7ff60c158670 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff601ffb640 1 -- 172.21.15.27:0/1749142726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5fc004bf0 con 0x7ff60c158670 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c0cbcb0 con 0x7ff60c158670 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.529+0000 7ff601ffb640 1 -- 172.21.15.27:0/1749142726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc007cf0 con 0x7ff60c158670 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.530+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c0cb1d0 con 0x7ff60c158670 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.531+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c067ff0 con 0x7ff60c158670 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.531+0000 7ff601ffb640 1 -- 172.21.15.27:0/1749142726 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5fc007070 con 0x7ff60c158670 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.532+0000 7ff601ffb640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5f006b500 0x7ff5f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.532+0000 7ff601ffb640 1 -- 172.21.15.27:0/1749142726 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5fc0ab310 con 0x7ff60c158670 2024-03-06T07:24:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.532+0000 7ff6110da640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5f006b500 0x7ff5f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.532+0000 7ff6110da640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5f006b500 0x7ff5f006d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff60c050c30 tx=0x7ff608009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.541+0000 7ff601ffb640 1 -- 172.21.15.27:0/1749142726 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5fc079b40 con 0x7ff60c158670 2024-03-06T07:24:32.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.763+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff60c0b2bd0 con 0x7ff60c158670 2024-03-06T07:24:32.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.764+0000 7ff601ffb640 1 -- 172.21.15.27:0/1749142726 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4971 (secure 0 0 0) 0x7ff5fc07dbe0 con 0x7ff60c158670 2024-03-06T07:24:32.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:32.766 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:55.014569+0000","modified":"2024-03-06T07:23:58.060823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/136269066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":136269066},{"type":"v1","addr":"172.21.15.27:6835","nonce":136269066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":2}]} 2024-03-06T07:24:32.766 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-06T07:24:32.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5f006b500 msgr2=0x7ff5f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5f006b500 0x7ff5f006d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff60c050c30 tx=0x7ff608009040 comp rx=0 tx=0).stop 2024-03-06T07:24:32.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c158670 msgr2=0x7ff60c0b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c158670 0x7ff60c0b4eb0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00c360 tx=0x7ff60c0d0620 comp rx=0 tx=0).stop 2024-03-06T07:24:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 shutdown_connections 2024-03-06T07:24:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5f006b500 0x7ff5f006d9c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff60c154e50 0x7ff60c1552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c158670 0x7ff60c0b4eb0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 --2- 172.21.15.27:0/1749142726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f8004800 0x7ff60c0cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 >> 172.21.15.27:0/1749142726 conn(0x7ff60c071960 msgr2=0x7ff60c1618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 shutdown_connections 2024-03-06T07:24:32.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.766+0000 7ff61333e640 1 -- 172.21.15.27:0/1749142726 wait complete. 2024-03-06T07:24:32.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.884+0000 7f828fe39640 1 Processor -- start 2024-03-06T07:24:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.884+0000 7f828fe39640 1 -- start start 2024-03-06T07:24:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.885+0000 7f828fe39640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f82880ae9b0 0x7f82880d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.885+0000 7f828fe39640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82880cee90 con 0x7f828806b5c0 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.885+0000 7f828fe39640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82880cfc50 con 0x7f82880b0e90 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.885+0000 7f828fe39640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82880cfa00 con 0x7f82880ae9b0 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.885+0000 7f828dbd5640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f828806b5c0 0x7f82880b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42964/0 (socket says 172.21.15.27:42964) 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.885+0000 7f828dbd5640 1 -- 172.21.15.27:0/4147124604 learned_addr learned my addr 172.21.15.27:0/4147124604 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828e3d6640 1 --2- 172.21.15.27:0/4147124604 >> v2:172.21.15.69:3300/0 conn(0x7f82880ae9b0 0x7f82880d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653768919 0 0) 0x7f82880cee90 con 0x7f828806b5c0 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8274003520 con 0x7f828806b5c0 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 736222625 0 0) 0x7f8274003520 con 0x7f828806b5c0 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 >> v2:172.21.15.69:3300/0 conn(0x7f82880ae9b0 msgr2=0x7f82880d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 --2- 172.21.15.27:0/4147124604 >> v2:172.21.15.69:3300/0 conn(0x7f82880ae9b0 0x7f82880d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 >> v1:172.21.15.69:6789/0 conn(0x7f82880b0e90 legacy=0x7f828806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82880cff30 con 0x7f828806b5c0 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f8284001670 con 0x7f828806b5c0 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 >> v1:172.21.15.27:6789/0 conn(0x7f828806b5c0 legacy=0x7f82880b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740039b0 0x7f8274003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8274004800 0x7f8274004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740051c0 0x7f827400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82880cfa00 con 0x7f82740039b0 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827400c160 con 0x7f8274004800 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827400c360 con 0x7f82740051c0 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828dbd5640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740039b0 0x7f8274003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828d3d4640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8274004800 0x7f8274004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828e3d6640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740051c0 0x7f827400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.886+0000 7f828dbd5640 1 -- 172.21.15.27:0/4147124604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740051c0 msgr2=0x7f827400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.887+0000 7f828dbd5640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740051c0 0x7f827400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.887+0000 7f828dbd5640 1 -- 172.21.15.27:0/4147124604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8274004800 msgr2=0x7f8274004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.887+0000 7f828dbd5640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8274004800 0x7f8274004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.887+0000 7f828dbd5640 1 -- 172.21.15.27:0/4147124604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274003520 con 0x7f82740039b0 2024-03-06T07:24:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.887+0000 7f828e3d6640 1 -- 172.21.15.27:0/4147124604 reap_dead start 2024-03-06T07:24:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.887+0000 7f828dbd5640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740039b0 0x7f8274003dd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f828400c230 tx=0x7f8274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f009a706534e988 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.887+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284007d30 con 0x7f82740039b0 2024-03-06T07:24:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.887+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284005af0 con 0x7f82740039b0 2024-03-06T07:24:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.887+0000 7f828cbd3640 1 -- 172.21.15.27:0/4147124604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284007070 con 0x7f82740039b0 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.888+0000 7f828fe39640 1 -- 172.21.15.27:0/4147124604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740039b0 msgr2=0x7f8274003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.888+0000 7f828fe39640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740039b0 0x7f8274003dd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f828400c230 tx=0x7f8274005830 comp rx=0 tx=0).stop 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.888+0000 7f828fe39640 1 -- 172.21.15.27:0/4147124604 shutdown_connections 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.888+0000 7f828fe39640 1 --2- 172.21.15.27:0/4147124604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740039b0 0x7f8274003dd0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.888+0000 7f828fe39640 1 -- 172.21.15.27:0/4147124604 >> 172.21.15.27:0/4147124604 conn(0x7f82880b36d0 msgr2=0x7f82880b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.888+0000 7f828fe39640 1 -- 172.21.15.27:0/4147124604 shutdown_connections 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.888+0000 7f828fe39640 1 -- 172.21.15.27:0/4147124604 wait complete. 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.889+0000 7f828fe39640 1 Processor -- start 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.889+0000 7f828fe39640 1 -- start start 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828fe39640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740039b0 0x7f82881cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828fe39640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82880c8bd0 0x7f82880c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828fe39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82881e3630 0x7f82881e3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828fe39640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82880aef40 con 0x7f82881e3630 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828fe39640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82880aedc0 con 0x7f82880c8bd0 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828fe39640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82880b2e00 con 0x7f82740039b0 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828e3d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82881e3630 0x7f82881e3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828e3d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82881e3630 0x7f82881e3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40366/0 (socket says 172.21.15.27:40366) 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828e3d6640 1 -- 172.21.15.27:0/1119066014 learned_addr learned my addr 172.21.15.27:0/1119066014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828d3d4640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82880c8bd0 0x7f82880c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828dbd5640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740039b0 0x7f82881cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828e3d6640 1 -- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740039b0 msgr2=0x7f82881cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.890+0000 7f828e3d6640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740039b0 0x7f82881cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f828e3d6640 1 -- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82880c8bd0 msgr2=0x7f82880c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f828e3d6640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82880c8bd0 0x7f82880c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f828e3d6640 1 -- 172.21.15.27:0/1119066014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82881e2870 con 0x7f82881e3630 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f828dbd5640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740039b0 0x7f82881cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f828e3d6640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82881e3630 0x7f82881e3ab0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8288072aa0 tx=0x7f828000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f827e7fc640 1 -- 172.21.15.27:0/1119066014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828000fd80 con 0x7f82881e3630 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f827e7fc640 1 -- 172.21.15.27:0/1119066014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828001d070 con 0x7f82881e3630 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82881e2b50 con 0x7f82881e3630 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f827e7fc640 1 -- 172.21.15.27:0/1119066014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8280015670 con 0x7f82881e3630 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.891+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82881e3110 con 0x7f82881e3630 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.892+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8288068700 con 0x7f82881e3630 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.892+0000 7f827e7fc640 1 -- 172.21.15.27:0/1119066014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8280015810 con 0x7f82881e3630 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.893+0000 7f827e7fc640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f826c06b4d0 0x7f826c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.893+0000 7f827e7fc640 1 -- 172.21.15.27:0/1119066014 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f828008ab80 con 0x7f82881e3630 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.893+0000 7f828dbd5640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f826c06b4d0 0x7f826c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.894+0000 7f828dbd5640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f826c06b4d0 0x7f826c06d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f828403c000 tx=0x7f8284037000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:32.897+0000 7f827e7fc640 1 -- 172.21.15.27:0/1119066014 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8280027020 con 0x7f82881e3630 2024-03-06T07:24:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.125+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82881e3ff0 con 0x7f82881e3630 2024-03-06T07:24:33.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.126+0000 7f827e7fc640 1 -- 172.21.15.27:0/1119066014 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f8280010040 con 0x7f82881e3630 2024-03-06T07:24:33.128 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:33.128 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":35,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:23:56.637219+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:53.309610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:53.987106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:33.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f826c06b4d0 msgr2=0x7f826c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f826c06b4d0 0x7f826c06d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f828403c000 tx=0x7f8284037000 comp rx=0 tx=0).stop 2024-03-06T07:24:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82881e3630 msgr2=0x7f82881e3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82881e3630 0x7f82881e3ab0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8288072aa0 tx=0x7f828000f580 comp rx=0 tx=0).stop 2024-03-06T07:24:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 shutdown_connections 2024-03-06T07:24:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f826c06b4d0 0x7f826c06d990 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82881e3630 0x7f82881e3ab0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82880c8bd0 0x7f82880c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 --2- 172.21.15.27:0/1119066014 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82740039b0 0x7f82881cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 >> 172.21.15.27:0/1119066014 conn(0x7f82880b36d0 msgr2=0x7f82880b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 shutdown_connections 2024-03-06T07:24:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.128+0000 7f828fe39640 1 -- 172.21.15.27:0/1119066014 wait complete. 2024-03-06T07:24:33.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:24:33.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.253+0000 7fc69d637640 1 Processor -- start 2024-03-06T07:24:33.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.253+0000 7fc69d637640 1 -- start start 2024-03-06T07:24:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc69d637640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc698071540 0x7fc6980b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc69d637640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6980d3400 0x7fc698071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc69d637640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6980b2b50 0x7fc6980d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc69d637640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6980d2f70 con 0x7fc6980b2b50 2024-03-06T07:24:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc69d637640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6980d4be0 con 0x7fc698071540 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc69d637640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc698071e60 con 0x7fc6980d3400 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc6977fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6980b2b50 0x7fc6980d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc6977fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6980b2b50 0x7fc6980d1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40372/0 (socket says 172.21.15.27:40372) 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc6977fe640 1 -- 172.21.15.27:0/3774741947 learned_addr learned my addr 172.21.15.27:0/3774741947 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc696ffd640 1 --2- 172.21.15.27:0/3774741947 >> v2:172.21.15.69:3300/0 conn(0x7fc698071540 0x7fc6980b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc68ffff640 1 --2- 172.21.15.27:0/3774741947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6980d3400 0x7fc698071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc6977fe640 1 -- 172.21.15.27:0/3774741947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6980d3400 msgr2=0x7fc698071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc6977fe640 1 --2- 172.21.15.27:0/3774741947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6980d3400 0x7fc698071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc6977fe640 1 -- 172.21.15.27:0/3774741947 >> v2:172.21.15.69:3300/0 conn(0x7fc698071540 msgr2=0x7fc6980b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc6977fe640 1 --2- 172.21.15.27:0/3774741947 >> v2:172.21.15.69:3300/0 conn(0x7fc698071540 0x7fc6980b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc6977fe640 1 -- 172.21.15.27:0/3774741947 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6980ce610 con 0x7fc6980b2b50 2024-03-06T07:24:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.254+0000 7fc68ffff640 1 --2- 172.21.15.27:0/3774741947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6980d3400 0x7fc698071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc6977fe640 1 --2- 172.21.15.27:0/3774741947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980b2b50 0x7fc6980d1870 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc6980d4da0 tx=0x7fc69002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff01a7790f316d55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc6967fc640 1 -- 172.21.15.27:0/3774741947 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69002ca70 con 0x7fc6980b2b50 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc6967fc640 1 -- 172.21.15.27:0/3774741947 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69002cc10 con 0x7fc6980b2b50 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc6967fc640 1 -- 172.21.15.27:0/3774741947 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6900343f0 con 0x7fc6980b2b50 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc69d637640 1 -- 172.21.15.27:0/3774741947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980b2b50 msgr2=0x7fc6980d1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc69d637640 1 --2- 172.21.15.27:0/3774741947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980b2b50 0x7fc6980d1870 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc6980d4da0 tx=0x7fc69002bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc69d637640 1 -- 172.21.15.27:0/3774741947 shutdown_connections 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc69d637640 1 --2- 172.21.15.27:0/3774741947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980b2b50 0x7fc6980d1870 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc69d637640 1 --2- 172.21.15.27:0/3774741947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6980d3400 0x7fc698071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc69d637640 1 --2- 172.21.15.27:0/3774741947 >> v2:172.21.15.69:3300/0 conn(0x7fc698071540 0x7fc6980b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc69d637640 1 -- 172.21.15.27:0/3774741947 >> 172.21.15.27:0/3774741947 conn(0x7fc6980d3ca0 msgr2=0x7fc6980d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc69d637640 1 -- 172.21.15.27:0/3774741947 shutdown_connections 2024-03-06T07:24:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.256+0000 7fc69d637640 1 -- 172.21.15.27:0/3774741947 wait complete. 2024-03-06T07:24:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.258+0000 7fc69d637640 1 Processor -- start 2024-03-06T07:24:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.258+0000 7fc69d637640 1 -- start start 2024-03-06T07:24:33.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.258+0000 7fc69d637640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc698071540 0x7fc6981f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.258+0000 7fc69d637640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6980b2b50 0x7fc69814ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.258+0000 7fc69d637640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980d3400 0x7fc69814f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.258+0000 7fc69d637640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6980ce8f0 con 0x7fc6980d3400 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.258+0000 7fc69d637640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6980d2c00 con 0x7fc6980b2b50 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.258+0000 7fc69d637640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69815e370 con 0x7fc698071540 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.258+0000 7fc6977fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980d3400 0x7fc69814f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc6977fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980d3400 0x7fc69814f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40386/0 (socket says 172.21.15.27:40386) 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc6977fe640 1 -- 172.21.15.27:0/3863808576 learned_addr learned my addr 172.21.15.27:0/3863808576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc696ffd640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc698071540 0x7fc6981f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc68ffff640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6980b2b50 0x7fc69814ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc6977fe640 1 -- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc698071540 msgr2=0x7fc6981f7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc6977fe640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc698071540 0x7fc6981f7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc6977fe640 1 -- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6980b2b50 msgr2=0x7fc69814ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc6977fe640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6980b2b50 0x7fc69814ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc6977fe640 1 -- 172.21.15.27:0/3863808576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6980b4720 con 0x7fc6980d3400 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc6977fe640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980d3400 0x7fc69814f190 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc690036040 tx=0x7fc690004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc68f7fe640 1 -- 172.21.15.27:0/3863808576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690043020 con 0x7fc6980d3400 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc68f7fe640 1 -- 172.21.15.27:0/3863808576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69003e450 con 0x7fc6980d3400 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc698158c10 con 0x7fc6980d3400 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc68f7fe640 1 -- 172.21.15.27:0/3863808576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690009510 con 0x7fc6980d3400 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.259+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6981591d0 con 0x7fc6980d3400 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.260+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc698068700 con 0x7fc6980d3400 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.261+0000 7fc68f7fe640 1 -- 172.21.15.27:0/3863808576 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc690046020 con 0x7fc6980d3400 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.261+0000 7fc68f7fe640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67806b4d0 0x7fc67806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.261+0000 7fc68f7fe640 1 -- 172.21.15.27:0/3863808576 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6900ac870 con 0x7fc6980d3400 2024-03-06T07:24:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.261+0000 7fc696ffd640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67806b4d0 0x7fc67806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.262+0000 7fc696ffd640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67806b4d0 0x7fc67806d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc688009ac0 tx=0x7fc688009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.266+0000 7fc68f7fe640 1 -- 172.21.15.27:0/3863808576 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc69004d030 con 0x7fc6980d3400 2024-03-06T07:24:33.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.556+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc6981589c0 con 0x7fc6980d3400 2024-03-06T07:24:33.558 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:24:33.557+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:33.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.558+0000 7fc68f7fe640 1 -- 172.21.15.27:0/3863808576 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc69007b6f0 con 0x7fc6980d3400 2024-03-06T07:24:33.560 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.561+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67806b4d0 msgr2=0x7fc67806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.561+0000 7fc69d637640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67806b4d0 0x7fc67806d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc688009ac0 tx=0x7fc688009190 comp rx=0 tx=0).stop 2024-03-06T07:24:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.561+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980d3400 msgr2=0x7fc69814f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.561+0000 7fc69d637640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980d3400 0x7fc69814f190 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc690036040 tx=0x7fc690004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.561+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 shutdown_connections 2024-03-06T07:24:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.562+0000 7fc69d637640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67806b4d0 0x7fc67806d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.562+0000 7fc69d637640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6980d3400 0x7fc69814f190 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.562+0000 7fc69d637640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6980b2b50 0x7fc69814ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.562+0000 7fc69d637640 1 --2- 172.21.15.27:0/3863808576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc698071540 0x7fc6981f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.562+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 >> 172.21.15.27:0/3863808576 conn(0x7fc6980d3ca0 msgr2=0x7fc698160630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.562+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 shutdown_connections 2024-03-06T07:24:33.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.562+0000 7fc69d637640 1 -- 172.21.15.27:0/3863808576 wait complete. 2024-03-06T07:24:33.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:33.601 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:24:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.683+0000 7f5c0a6df640 1 Processor -- start 2024-03-06T07:24:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.683+0000 7f5c0a6df640 1 -- start start 2024-03-06T07:24:33.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.684+0000 7f5c0a6df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 0x7f5c040ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.684+0000 7f5c0a6df640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c040d29d0 0x7f5c040d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.684+0000 7f5c0a6df640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c040d3740 con 0x7f5c040d4720 2024-03-06T07:24:33.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.684+0000 7f5c0a6df640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c040d3400 con 0x7f5c040d29d0 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.684+0000 7f5c0a6df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040d3120 con 0x7f5c0415dd90 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.684+0000 7f5c037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c040d29d0 0x7f5c040d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.684+0000 7f5c037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c040d29d0 0x7f5c040d41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45350/0 (socket says 172.21.15.27:45350) 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.684+0000 7f5c037fe640 1 -- 172.21.15.27:0/1693710918 learned_addr learned my addr 172.21.15.27:0/1693710918 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.685+0000 7f5c03fff640 1 --2- 172.21.15.27:0/1693710918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 0x7f5c040ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.685+0000 7f5c037fe640 1 -- 172.21.15.27:0/1693710918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 msgr2=0x7f5c040ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.685+0000 7f5c037fe640 1 --2- 172.21.15.27:0/1693710918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 0x7f5c040ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.685+0000 7f5c037fe640 1 -- 172.21.15.27:0/1693710918 >> v1:172.21.15.69:6789/0 conn(0x7f5c040d4720 legacy=0x7f5c040b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.685+0000 7f5c037fe640 1 -- 172.21.15.27:0/1693710918 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c040d39e0 con 0x7f5c040d29d0 2024-03-06T07:24:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.685+0000 7f5c037fe640 1 --2- 172.21.15.27:0/1693710918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 0x7f5c040d41e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009870 tx=0x7f5bf802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c756309abd84a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c02ffd640 1 -- 172.21.15.27:0/1693710918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf802f840 con 0x7f5c040d29d0 2024-03-06T07:24:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c02ffd640 1 -- 172.21.15.27:0/1693710918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf802f9e0 con 0x7f5c040d29d0 2024-03-06T07:24:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c02ffd640 1 -- 172.21.15.27:0/1693710918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf802fdb0 con 0x7f5c040d29d0 2024-03-06T07:24:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1693710918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 msgr2=0x7f5c040d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c0a6df640 1 --2- 172.21.15.27:0/1693710918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 0x7f5c040d41e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009870 tx=0x7f5bf802ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1693710918 shutdown_connections 2024-03-06T07:24:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c0a6df640 1 --2- 172.21.15.27:0/1693710918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 0x7f5c040d41e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c0a6df640 1 --2- 172.21.15.27:0/1693710918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 0x7f5c040ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1693710918 >> 172.21.15.27:0/1693710918 conn(0x7f5c04071960 msgr2=0x7f5c04071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1693710918 shutdown_connections 2024-03-06T07:24:33.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.686+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1693710918 wait complete. 2024-03-06T07:24:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.687+0000 7f5c0a6df640 1 Processor -- start 2024-03-06T07:24:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c0a6df640 1 -- start start 2024-03-06T07:24:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c0a6df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 0x7f5c041606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c0a6df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c040d4720 0x7f5c040c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c0a6df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 0x7f5c040c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c0a6df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040d2df0 con 0x7f5c040d29d0 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c0a6df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0415e170 con 0x7f5c040d4720 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c0a6df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040b7420 con 0x7f5c0415dd90 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c03fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 0x7f5c041606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c03fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 0x7f5c041606e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45362/0 (socket says 172.21.15.27:45362) 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.688+0000 7f5c03fff640 1 -- 172.21.15.27:0/1361117090 learned_addr learned my addr 172.21.15.27:0/1361117090 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c08c7c640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 0x7f5c040c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c037fe640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c040d4720 0x7f5c040c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c03fff640 1 -- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 msgr2=0x7f5c040c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c03fff640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 0x7f5c040c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c03fff640 1 -- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c040d4720 msgr2=0x7f5c040c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c03fff640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c040d4720 0x7f5c040c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c03fff640 1 -- 172.21.15.27:0/1361117090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c041f3be0 con 0x7f5c040d29d0 2024-03-06T07:24:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c08c7c640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 0x7f5c040c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c03fff640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 0x7f5c041606e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400e870 tx=0x7f5bf400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c00ff9640 1 -- 172.21.15.27:0/1361117090 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4017070 con 0x7f5c040d29d0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c00ff9640 1 -- 172.21.15.27:0/1361117090 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf400c940 con 0x7f5c040d29d0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c041f3ec0 con 0x7f5c040d29d0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c00ff9640 1 -- 172.21.15.27:0/1361117090 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf400cd30 con 0x7f5c040d29d0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.689+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c041f4480 con 0x7f5c040d29d0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.690+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c04072a30 con 0x7f5c040d29d0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.691+0000 7f5c00ff9640 1 -- 172.21.15.27:0/1361117090 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bf401a070 con 0x7f5c040d29d0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.691+0000 7f5c00ff9640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5bf006b4d0 0x7f5bf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.691+0000 7f5c00ff9640 1 -- 172.21.15.27:0/1361117090 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5bf408a450 con 0x7f5c040d29d0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.691+0000 7f5c037fe640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5bf006b4d0 0x7f5bf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.692+0000 7f5c037fe640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5bf006b4d0 0x7f5bf006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8036000 tx=0x7f5bf8039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.696+0000 7f5c00ff9640 1 -- 172.21.15.27:0/1361117090 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bf4059300 con 0x7f5c040d29d0 2024-03-06T07:24:33.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:33.917+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5c040cf040 con 0x7f5c040d29d0 2024-03-06T07:24:34.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.560+0000 7f5c00ff9640 1 -- 172.21.15.27:0/1361117090 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5bf4058c00 con 0x7f5c040d29d0 2024-03-06T07:24:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5bf006b4d0 msgr2=0x7f5bf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5bf006b4d0 0x7f5bf006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8036000 tx=0x7f5bf8039040 comp rx=0 tx=0).stop 2024-03-06T07:24:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 msgr2=0x7f5c041606e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 0x7f5c041606e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400e870 tx=0x7f5bf400ed30 comp rx=0 tx=0).stop 2024-03-06T07:24:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 shutdown_connections 2024-03-06T07:24:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5bf006b4d0 0x7f5bf006d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c0415dd90 0x7f5c040c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c040d4720 0x7f5c040c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 --2- 172.21.15.27:0/1361117090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c040d29d0 0x7f5c041606e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 >> 172.21.15.27:0/1361117090 conn(0x7f5c04071960 msgr2=0x7f5c04162200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 shutdown_connections 2024-03-06T07:24:34.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.563+0000 7f5c0a6df640 1 -- 172.21.15.27:0/1361117090 wait complete. 2024-03-06T07:24:34.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.748+0000 7fb97a51e640 1 Processor -- start 2024-03-06T07:24:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.748+0000 7fb97a51e640 1 -- start start 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.748+0000 7fb97a51e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb97415e160 0x7fb9740cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb97a51e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9740cf2f0 con 0x7fb9740b3020 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb97a51e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9740b97c0 con 0x7fb9740ce9d0 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb97a51e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9740d1de0 con 0x7fb97415e160 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb978abb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb9740b3020 0x7fb9740d2270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55336/0 (socket says 172.21.15.27:55336) 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb978abb640 1 -- 172.21.15.27:0/3581691095 learned_addr learned my addr 172.21.15.27:0/3581691095 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb9737fe640 1 --2- 172.21.15.27:0/3581691095 >> v2:172.21.15.69:3300/0 conn(0x7fb97415e160 0x7fb9740cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361157444 0 0) 0x7fb9740cf2f0 con 0x7fb9740b3020 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb95c003520 con 0x7fb9740b3020 2024-03-06T07:24:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1250089016 0 0) 0x7fb95c003520 con 0x7fb9740b3020 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 >> v2:172.21.15.69:3300/0 conn(0x7fb97415e160 msgr2=0x7fb9740cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 --2- 172.21.15.27:0/3581691095 >> v2:172.21.15.69:3300/0 conn(0x7fb97415e160 0x7fb9740cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 >> v1:172.21.15.69:6789/0 conn(0x7fb9740ce9d0 legacy=0x7fb974162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740b9aa0 con 0x7fb9740b3020 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fb96c0019f0 con 0x7fb9740b3020 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 >> v1:172.21.15.27:6789/0 conn(0x7fb9740b3020 legacy=0x7fb9740d2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb95c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb95c004800 0x7fb95c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb95c0051c0 0x7fb95c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740d1de0 con 0x7fb95c0039b0 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c00c070 con 0x7fb95c0051c0 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c00c270 con 0x7fb95c004800 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.749+0000 7fb973fff640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb95c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.750+0000 7fb9737fe640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb95c004800 0x7fb95c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.750+0000 7fb978abb640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb95c0051c0 0x7fb95c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.750+0000 7fb973fff640 1 -- 172.21.15.27:0/3581691095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb95c004800 msgr2=0x7fb95c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.750+0000 7fb973fff640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb95c004800 0x7fb95c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.750+0000 7fb973fff640 1 -- 172.21.15.27:0/3581691095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb95c0051c0 msgr2=0x7fb95c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.750+0000 7fb973fff640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb95c0051c0 0x7fb95c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.750+0000 7fb978abb640 1 -- 172.21.15.27:0/3581691095 reap_dead start 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.750+0000 7fb973fff640 1 -- 172.21.15.27:0/3581691095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c003520 con 0x7fb95c0039b0 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb973fff640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb95c003dd0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb964001750 tx=0x7fb95c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3c211eec58377e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964001370 con 0x7fb95c0039b0 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9640052d0 con 0x7fb95c0039b0 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb972ffd640 1 -- 172.21.15.27:0/3581691095 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964003830 con 0x7fb95c0039b0 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb97a51e640 1 -- 172.21.15.27:0/3581691095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 msgr2=0x7fb95c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb97a51e640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb95c003dd0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb964001750 tx=0x7fb95c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb97a51e640 1 -- 172.21.15.27:0/3581691095 shutdown_connections 2024-03-06T07:24:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb97a51e640 1 --2- 172.21.15.27:0/3581691095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb95c003dd0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb97a51e640 1 -- 172.21.15.27:0/3581691095 >> 172.21.15.27:0/3581691095 conn(0x7fb9740d3c90 msgr2=0x7fb9740d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb97a51e640 1 -- 172.21.15.27:0/3581691095 shutdown_connections 2024-03-06T07:24:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.751+0000 7fb97a51e640 1 -- 172.21.15.27:0/3581691095 wait complete. 2024-03-06T07:24:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.752+0000 7fb97a51e640 1 Processor -- start 2024-03-06T07:24:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.753+0000 7fb97a51e640 1 -- start start 2024-03-06T07:24:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.753+0000 7fb97a51e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb974153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.753+0000 7fb97a51e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb97415fca0 0x7fb974160120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.753+0000 7fb97a51e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9741e7ac0 0x7fb9741e7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.753+0000 7fb97a51e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740cef30 con 0x7fb95c0039b0 2024-03-06T07:24:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.753+0000 7fb97a51e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740cedb0 con 0x7fb9741e7ac0 2024-03-06T07:24:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.753+0000 7fb97a51e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740cf0b0 con 0x7fb97415fca0 2024-03-06T07:24:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb973fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb974153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb973fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb974153b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45384/0 (socket says 172.21.15.27:45384) 2024-03-06T07:24:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb973fff640 1 -- 172.21.15.27:0/1756931707 learned_addr learned my addr 172.21.15.27:0/1756931707 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb978abb640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9741e7ac0 0x7fb9741e7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb9737fe640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb97415fca0 0x7fb974160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb973fff640 1 -- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb97415fca0 msgr2=0x7fb974160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb973fff640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb97415fca0 0x7fb974160120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb973fff640 1 -- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9741e7ac0 msgr2=0x7fb9741e7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb973fff640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9741e7ac0 0x7fb9741e7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb973fff640 1 -- 172.21.15.27:0/1756931707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9741e9050 con 0x7fb95c0039b0 2024-03-06T07:24:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb9737fe640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb97415fca0 0x7fb974160120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb978abb640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9741e7ac0 0x7fb9741e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb973fff640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb974153b30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb964000c00 tx=0x7fb964001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb970ff9640 1 -- 172.21.15.27:0/1756931707 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964041020 con 0x7fb95c0039b0 2024-03-06T07:24:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb970ff9640 1 -- 172.21.15.27:0/1756931707 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb964005480 con 0x7fb95c0039b0 2024-03-06T07:24:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb970ff9640 1 -- 172.21.15.27:0/1756931707 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964007a30 con 0x7fb95c0039b0 2024-03-06T07:24:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9741e87b0 con 0x7fb95c0039b0 2024-03-06T07:24:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.754+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9741e8d70 con 0x7fb95c0039b0 2024-03-06T07:24:34.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.755+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97406cda0 con 0x7fb95c0039b0 2024-03-06T07:24:34.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.756+0000 7fb970ff9640 1 -- 172.21.15.27:0/1756931707 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb964044020 con 0x7fb95c0039b0 2024-03-06T07:24:34.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.756+0000 7fb970ff9640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96006b500 0x7fb96006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.756+0000 7fb970ff9640 1 -- 172.21.15.27:0/1756931707 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb9640abcd0 con 0x7fb95c0039b0 2024-03-06T07:24:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.757+0000 7fb9737fe640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96006b500 0x7fb96006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.757+0000 7fb9737fe640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96006b500 0x7fb96006d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb974160a20 tx=0x7fb968008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.761+0000 7fb970ff9640 1 -- 172.21.15.27:0/1756931707 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb96404b030 con 0x7fb95c0039b0 2024-03-06T07:24:34.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:34.943+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb97404f410 con 0x7fb95c0039b0 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.772+0000 7fb970ff9640 1 -- 172.21.15.27:0/1756931707 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v37) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb9640303c0 con 0x7fb95c0039b0 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96006b500 msgr2=0x7fb96006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96006b500 0x7fb96006d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb974160a20 tx=0x7fb968008040 comp rx=0 tx=0).stop 2024-03-06T07:24:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 msgr2=0x7fb974153b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb974153b30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb964000c00 tx=0x7fb964001120 comp rx=0 tx=0).stop 2024-03-06T07:24:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 shutdown_connections 2024-03-06T07:24:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96006b500 0x7fb96006d9c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9741e7ac0 0x7fb9741e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb97415fca0 0x7fb974160120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 --2- 172.21.15.27:0/1756931707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb95c0039b0 0x7fb974153b30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 >> 172.21.15.27:0/1756931707 conn(0x7fb9740d3c90 msgr2=0x7fb974162150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.775+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 shutdown_connections 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.776+0000 7fb97a51e640 1 -- 172.21.15.27:0/1756931707 wait complete. 2024-03-06T07:24:35.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.908+0000 7f6bdd993640 1 Processor -- start 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.908+0000 7f6bdd993640 1 -- start start 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bdd993640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80717b0 0x7f6bd8071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bdd993640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd80b3060 0x7f6bd80d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bdd993640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd815e4b0 con 0x7f6bd80b2b10 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bdd993640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd80d2e10 con 0x7f6bd80b3060 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bdd993640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd80b97e0 con 0x7f6bd80717b0 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd67fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd80b3060 0x7f6bd80d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd77fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6bd80b2b10 0x7f6bd80b6e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55344/0 (socket says 172.21.15.27:55344) 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd77fe640 1 -- 172.21.15.27:0/3645318262 learned_addr learned my addr 172.21.15.27:0/3645318262 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd6ffd640 1 --2- 172.21.15.27:0/3645318262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80717b0 0x7f6bd8071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd67fc640 1 -- 172.21.15.27:0/3645318262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80717b0 msgr2=0x7f6bd8071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd67fc640 1 --2- 172.21.15.27:0/3645318262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80717b0 0x7f6bd8071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd67fc640 1 -- 172.21.15.27:0/3645318262 >> v1:172.21.15.27:6789/0 conn(0x7f6bd80b2b10 legacy=0x7f6bd80b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd67fc640 1 -- 172.21.15.27:0/3645318262 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd80b9a80 con 0x7f6bd80b3060 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd5ffb640 1 -- 172.21.15.27:0/3645318262 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213975778 0 0) 0x7f6bd0001490 con 0x7f6bd80b2b10 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.909+0000 7f6bd6ffd640 1 --2- 172.21.15.27:0/3645318262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80717b0 0x7f6bd8071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.910+0000 7f6bd67fc640 1 --2- 172.21.15.27:0/3645318262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80b3060 0x7f6bd80d32a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0097a0 tx=0x7f6bcc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a166b6ae3ad0e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.910+0000 7f6bd5ffb640 1 -- 172.21.15.27:0/3645318262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc035020 con 0x7f6bd80b3060 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.910+0000 7f6bd5ffb640 1 -- 172.21.15.27:0/3645318262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc031810 con 0x7f6bd80b3060 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.910+0000 7f6bd5ffb640 1 -- 172.21.15.27:0/3645318262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc031be0 con 0x7f6bd80b3060 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.911+0000 7f6bdd993640 1 -- 172.21.15.27:0/3645318262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80b3060 msgr2=0x7f6bd80d32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.911+0000 7f6bdd993640 1 --2- 172.21.15.27:0/3645318262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80b3060 0x7f6bd80d32a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0097a0 tx=0x7f6bcc02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.911+0000 7f6bdd993640 1 -- 172.21.15.27:0/3645318262 shutdown_connections 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.911+0000 7f6bdd993640 1 --2- 172.21.15.27:0/3645318262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80b3060 0x7f6bd80d32a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.911+0000 7f6bdd993640 1 --2- 172.21.15.27:0/3645318262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80717b0 0x7f6bd8071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.911+0000 7f6bdd993640 1 -- 172.21.15.27:0/3645318262 >> 172.21.15.27:0/3645318262 conn(0x7f6bd80d3d20 msgr2=0x7f6bd80d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.911+0000 7f6bdd993640 1 -- 172.21.15.27:0/3645318262 shutdown_connections 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.911+0000 7f6bdd993640 1 -- 172.21.15.27:0/3645318262 wait complete. 2024-03-06T07:24:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.912+0000 7f6bdd993640 1 Processor -- start 2024-03-06T07:24:35.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.912+0000 7f6bdd993640 1 -- start start 2024-03-06T07:24:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bdd993640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80717b0 0x7f6bd815ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bdd993640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80b2b10 0x7f6bd815a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bdd993640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd80b3060 0x7f6bd80b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bdd993640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd80b7560 con 0x7f6bd80717b0 2024-03-06T07:24:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bdd993640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd80d4bf0 con 0x7f6bd80b3060 2024-03-06T07:24:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bdd993640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd80b76e0 con 0x7f6bd80b2b10 2024-03-06T07:24:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80717b0 0x7f6bd815ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80717b0 0x7f6bd815ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45396/0 (socket says 172.21.15.27:45396) 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd6ffd640 1 -- 172.21.15.27:0/2900687568 learned_addr learned my addr 172.21.15.27:0/2900687568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd67fc640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80b2b10 0x7f6bd815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd77fe640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd80b3060 0x7f6bd80b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd6ffd640 1 -- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80b2b10 msgr2=0x7f6bd815a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd6ffd640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80b2b10 0x7f6bd815a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd6ffd640 1 -- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd80b3060 msgr2=0x7f6bd80b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd6ffd640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd80b3060 0x7f6bd80b4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd6ffd640 1 -- 172.21.15.27:0/2900687568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd81f7540 con 0x7f6bd80717b0 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.913+0000 7f6bd77fe640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd80b3060 0x7f6bd80b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.914+0000 7f6bd6ffd640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80717b0 0x7f6bd815ad40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800c7f0 tx=0x7f6bc800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.914+0000 7f6bbffff640 1 -- 172.21.15.27:0/2900687568 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8015070 con 0x7f6bd80717b0 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.914+0000 7f6bbffff640 1 -- 172.21.15.27:0/2900687568 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8007a30 con 0x7f6bd80717b0 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.914+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd81f7820 con 0x7f6bd80717b0 2024-03-06T07:24:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.914+0000 7f6bbffff640 1 -- 172.21.15.27:0/2900687568 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8007e00 con 0x7f6bd80717b0 2024-03-06T07:24:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.914+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd81f7de0 con 0x7f6bd80717b0 2024-03-06T07:24:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.915+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd806ce40 con 0x7f6bd80717b0 2024-03-06T07:24:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.915+0000 7f6bbffff640 1 -- 172.21.15.27:0/2900687568 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bc8018070 con 0x7f6bd80717b0 2024-03-06T07:24:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.915+0000 7f6bbffff640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bc406b500 0x7f6bc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.916+0000 7f6bbffff640 1 -- 172.21.15.27:0/2900687568 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f6bc8089f10 con 0x7f6bd80717b0 2024-03-06T07:24:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.916+0000 7f6bd67fc640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bc406b500 0x7f6bc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.916+0000 7f6bd67fc640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bc406b500 0x7f6bc406d9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036000 tx=0x7f6bcc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:35.920+0000 7f6bbffff640 1 -- 172.21.15.27:0/2900687568 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bc8058930 con 0x7f6bd80717b0 2024-03-06T07:24:36.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.107+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6bd80ca270 con 0x7f6bd80717b0 2024-03-06T07:24:36.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.777+0000 7f6bbffff640 1 -- 172.21.15.27:0/2900687568 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v38) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6bc805c9d0 con 0x7f6bd80717b0 2024-03-06T07:24:36.779 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bc406b500 msgr2=0x7f6bc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bc406b500 0x7f6bc406d9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036000 tx=0x7f6bcc03c040 comp rx=0 tx=0).stop 2024-03-06T07:24:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80717b0 msgr2=0x7f6bd815ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80717b0 0x7f6bd815ad40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800c7f0 tx=0x7f6bc800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 shutdown_connections 2024-03-06T07:24:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bc406b500 0x7f6bc406d9c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd80b3060 0x7f6bd80b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd80b2b10 0x7f6bd815a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 --2- 172.21.15.27:0/2900687568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd80717b0 0x7f6bd815ad40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.781+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 >> 172.21.15.27:0/2900687568 conn(0x7f6bd80d3d20 msgr2=0x7f6bd815c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.782+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 shutdown_connections 2024-03-06T07:24:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.782+0000 7f6bdd993640 1 -- 172.21.15.27:0/2900687568 wait complete. 2024-03-06T07:24:36.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:36.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.909+0000 7f599fdbc640 1 Processor -- start 2024-03-06T07:24:36.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.909+0000 7f599fdbc640 1 -- start start 2024-03-06T07:24:36.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599fdbc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5998146fd0 0x7f599806ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599fdbc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59980b4840 0x7f59980b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599fdbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599806f450 0x7f59980b1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599fdbc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5998161900 con 0x7f59980b4840 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599fdbc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5998161b50 con 0x7f5998146fd0 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599fdbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5998161e30 con 0x7f599806f450 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599d357640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59980b4840 0x7f59980b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599d357640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59980b4840 0x7f59980b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45400/0 (socket says 172.21.15.27:45400) 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599d357640 1 -- 172.21.15.27:0/4275474798 learned_addr learned my addr 172.21.15.27:0/4275474798 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599d357640 1 -- 172.21.15.27:0/4275474798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599806f450 msgr2=0x7f59980b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599e359640 1 --2- 172.21.15.27:0/4275474798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599806f450 0x7f59980b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599db58640 1 --2- 172.21.15.27:0/4275474798 >> v2:172.21.15.69:3300/0 conn(0x7f5998146fd0 0x7f599806ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599d357640 1 --2- 172.21.15.27:0/4275474798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599806f450 0x7f59980b1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599d357640 1 -- 172.21.15.27:0/4275474798 >> v2:172.21.15.69:3300/0 conn(0x7f5998146fd0 msgr2=0x7f599806ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599d357640 1 --2- 172.21.15.27:0/4275474798 >> v2:172.21.15.69:3300/0 conn(0x7f5998146fd0 0x7f599806ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.910+0000 7f599d357640 1 -- 172.21.15.27:0/4275474798 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5998162110 con 0x7f59980b4840 2024-03-06T07:24:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.911+0000 7f599d357640 1 --2- 172.21.15.27:0/4275474798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 0x7f59980b0ee0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f59980cfd00 tx=0x7f598802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16cc10bf038eed3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.911+0000 7f599cb56640 1 -- 172.21.15.27:0/4275474798 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988035020 con 0x7f59980b4840 2024-03-06T07:24:36.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.911+0000 7f599cb56640 1 -- 172.21.15.27:0/4275474798 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5988031870 con 0x7f59980b4840 2024-03-06T07:24:36.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.911+0000 7f599cb56640 1 -- 172.21.15.27:0/4275474798 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988031c40 con 0x7f59980b4840 2024-03-06T07:24:36.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.912+0000 7f599fdbc640 1 -- 172.21.15.27:0/4275474798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 msgr2=0x7f59980b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.912+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4275474798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 0x7f59980b0ee0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f59980cfd00 tx=0x7f598802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.912+0000 7f599fdbc640 1 -- 172.21.15.27:0/4275474798 shutdown_connections 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.912+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4275474798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f599806f450 0x7f59980b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.912+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4275474798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 0x7f59980b0ee0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.912+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4275474798 >> v2:172.21.15.69:3300/0 conn(0x7f5998146fd0 0x7f599806ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.912+0000 7f599fdbc640 1 -- 172.21.15.27:0/4275474798 >> 172.21.15.27:0/4275474798 conn(0x7f59980afac0 msgr2=0x7f5998072d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.912+0000 7f599fdbc640 1 -- 172.21.15.27:0/4275474798 shutdown_connections 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.912+0000 7f599fdbc640 1 -- 172.21.15.27:0/4275474798 wait complete. 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.913+0000 7f599fdbc640 1 Processor -- start 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.913+0000 7f599fdbc640 1 -- start start 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599fdbc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599806f450 0x7f599815d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599fdbc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 0x7f59980cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599fdbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5998146fd0 0x7f5998142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599fdbc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5998147530 con 0x7f59980b4840 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599fdbc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59981473b0 con 0x7f599806f450 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599fdbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59980ad8b0 con 0x7f5998146fd0 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599d357640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 0x7f59980cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599d357640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 0x7f59980cc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45408/0 (socket says 172.21.15.27:45408) 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599d357640 1 -- 172.21.15.27:0/4225981672 learned_addr learned my addr 172.21.15.27:0/4225981672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599e359640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5998146fd0 0x7f5998142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599d357640 1 -- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5998146fd0 msgr2=0x7f5998142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599db58640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599806f450 0x7f599815d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599d357640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5998146fd0 0x7f5998142d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599d357640 1 -- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599806f450 msgr2=0x7f599815d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599d357640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599806f450 0x7f599815d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599d357640 1 -- 172.21.15.27:0/4225981672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599815db40 con 0x7f59980b4840 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599d357640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 0x7f59980cc900 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f598802b950 tx=0x7f59880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f598e7fc640 1 -- 172.21.15.27:0/4225981672 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988040070 con 0x7f59980b4840 2024-03-06T07:24:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f598e7fc640 1 -- 172.21.15.27:0/4225981672 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598803a420 con 0x7f59980b4840 2024-03-06T07:24:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59980ce340 con 0x7f59980b4840 2024-03-06T07:24:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f598e7fc640 1 -- 172.21.15.27:0/4225981672 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988035040 con 0x7f59980b4840 2024-03-06T07:24:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.914+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59980cd860 con 0x7f59980b4840 2024-03-06T07:24:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.915+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5998073410 con 0x7f59980b4840 2024-03-06T07:24:36.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.916+0000 7f598e7fc640 1 -- 172.21.15.27:0/4225981672 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5988038020 con 0x7f59980b4840 2024-03-06T07:24:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.916+0000 7f598e7fc640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f597c06b4d0 0x7f597c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.916+0000 7f598e7fc640 1 -- 172.21.15.27:0/4225981672 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f59880abf20 con 0x7f59980b4840 2024-03-06T07:24:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.917+0000 7f599db58640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f597c06b4d0 0x7f597c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.917+0000 7f599db58640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f597c06b4d0 0x7f597c06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f599400b740 tx=0x7f5994008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:36.921+0000 7f598e7fc640 1 -- 172.21.15.27:0/4225981672 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f599815db40 con 0x7f59980b4840 2024-03-06T07:24:37.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.101+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f599804f410 con 0x7f59980b4840 2024-03-06T07:24:37.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.104+0000 7f598e7fc640 1 -- 172.21.15.27:0/4225981672 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f598807b0a0 con 0x7f59980b4840 2024-03-06T07:24:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f597c06b4d0 msgr2=0x7f597c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f597c06b4d0 0x7f597c06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f599400b740 tx=0x7f5994008040 comp rx=0 tx=0).stop 2024-03-06T07:24:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 msgr2=0x7f59980cc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 0x7f59980cc900 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f598802b950 tx=0x7f59880046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 shutdown_connections 2024-03-06T07:24:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f597c06b4d0 0x7f597c06d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5998146fd0 0x7f5998142d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59980b4840 0x7f59980cc900 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 --2- 172.21.15.27:0/4225981672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f599806f450 0x7f599815d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.106+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 >> 172.21.15.27:0/4225981672 conn(0x7f59980afac0 msgr2=0x7f59981416a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.107+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 shutdown_connections 2024-03-06T07:24:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.107+0000 7f599fdbc640 1 -- 172.21.15.27:0/4225981672 wait complete. 2024-03-06T07:24:37.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:24:37.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.224+0000 7fc5903e9640 1 Processor -- start 2024-03-06T07:24:37.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.224+0000 7fc5903e9640 1 -- start start 2024-03-06T07:24:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.225+0000 7fc5903e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58806e700 0x7fc58815ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.225+0000 7fc5903e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b4160 con 0x7fc5880b55d0 2024-03-06T07:24:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.225+0000 7fc5903e9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b5b40 con 0x7fc588072660 2024-03-06T07:24:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.225+0000 7fc5903e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5880b3ba0 con 0x7fc58806e700 2024-03-06T07:24:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.225+0000 7fc58e986640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc5880b55d0 0x7fc588071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55356/0 (socket says 172.21.15.27:55356) 2024-03-06T07:24:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.225+0000 7fc58e986640 1 -- 172.21.15.27:0/3698163547 learned_addr learned my addr 172.21.15.27:0/3698163547 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.226+0000 7fc58e185640 1 --2- 172.21.15.27:0/3698163547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58806e700 0x7fc58815ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.226+0000 7fc58d183640 1 -- 172.21.15.27:0/3698163547 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723238858 0 0) 0x7fc5880b4160 con 0x7fc5880b55d0 2024-03-06T07:24:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.226+0000 7fc58d183640 1 -- 172.21.15.27:0/3698163547 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc574003520 con 0x7fc5880b55d0 2024-03-06T07:24:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.226+0000 7fc58d183640 1 -- 172.21.15.27:0/3698163547 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130427690 0 0) 0x7fc5880b5b40 con 0x7fc588072660 2024-03-06T07:24:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.226+0000 7fc58d183640 1 -- 172.21.15.27:0/3698163547 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5880b4160 con 0x7fc588072660 2024-03-06T07:24:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.226+0000 7fc58e185640 1 -- 172.21.15.27:0/3698163547 >> v1:172.21.15.69:6789/0 conn(0x7fc588072660 legacy=0x7fc58806d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.226+0000 7fc58e185640 1 -- 172.21.15.27:0/3698163547 >> v1:172.21.15.27:6789/0 conn(0x7fc5880b55d0 legacy=0x7fc588071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.226+0000 7fc58e185640 1 -- 172.21.15.27:0/3698163547 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5880b3e80 con 0x7fc58806e700 2024-03-06T07:24:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.227+0000 7fc58e185640 1 --2- 172.21.15.27:0/3698163547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58806e700 0x7fc58815ef80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc5840097d0 tx=0x7fc58402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53ecf59104298b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.227+0000 7fc58d183640 1 -- 172.21.15.27:0/3698163547 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584035020 con 0x7fc58806e700 2024-03-06T07:24:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.227+0000 7fc58d183640 1 -- 172.21.15.27:0/3698163547 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc584031890 con 0x7fc58806e700 2024-03-06T07:24:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.227+0000 7fc58d183640 1 -- 172.21.15.27:0/3698163547 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584031c40 con 0x7fc58806e700 2024-03-06T07:24:37.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.227+0000 7fc5903e9640 1 -- 172.21.15.27:0/3698163547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58806e700 msgr2=0x7fc58815ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.227+0000 7fc5903e9640 1 --2- 172.21.15.27:0/3698163547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58806e700 0x7fc58815ef80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc5840097d0 tx=0x7fc58402b930 comp rx=0 tx=0).stop 2024-03-06T07:24:37.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.228+0000 7fc5903e9640 1 -- 172.21.15.27:0/3698163547 shutdown_connections 2024-03-06T07:24:37.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.228+0000 7fc5903e9640 1 --2- 172.21.15.27:0/3698163547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc58806e700 0x7fc58815ef80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.228+0000 7fc5903e9640 1 -- 172.21.15.27:0/3698163547 >> 172.21.15.27:0/3698163547 conn(0x7fc5880b2720 msgr2=0x7fc5880b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.228+0000 7fc5903e9640 1 -- 172.21.15.27:0/3698163547 shutdown_connections 2024-03-06T07:24:37.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.228+0000 7fc5903e9640 1 -- 172.21.15.27:0/3698163547 wait complete. 2024-03-06T07:24:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.229+0000 7fc5903e9640 1 Processor -- start 2024-03-06T07:24:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.229+0000 7fc5903e9640 1 -- start start 2024-03-06T07:24:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.229+0000 7fc5903e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc58806e700 0x7fc5881f7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.229+0000 7fc5903e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc588072660 0x7fc5881f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc5903e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5880b55d0 0x7fc5881605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc5903e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc588072210 con 0x7fc588072660 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc5903e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc588072090 con 0x7fc58806e700 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc5903e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5880670e0 con 0x7fc5880b55d0 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58d984640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc588072660 0x7fc5881f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58d984640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc588072660 0x7fc5881f30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45412/0 (socket says 172.21.15.27:45412) 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58d984640 1 -- 172.21.15.27:0/2419746081 learned_addr learned my addr 172.21.15.27:0/2419746081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58e986640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5880b55d0 0x7fc5881605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58e185640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc58806e700 0x7fc5881f7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58d984640 1 -- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5880b55d0 msgr2=0x7fc5881605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58d984640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5880b55d0 0x7fc5881605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58d984640 1 -- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc58806e700 msgr2=0x7fc5881f7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58d984640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc58806e700 0x7fc5881f7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58d984640 1 -- 172.21.15.27:0/2419746081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5881f7d50 con 0x7fc588072660 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc58d984640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc588072660 0x7fc5881f30c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c530 tx=0x7fc57800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc57effd640 1 -- 172.21.15.27:0/2419746081 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5780052d0 con 0x7fc588072660 2024-03-06T07:24:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc57effd640 1 -- 172.21.15.27:0/2419746081 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578005490 con 0x7fc588072660 2024-03-06T07:24:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5880d01a0 con 0x7fc588072660 2024-03-06T07:24:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc57effd640 1 -- 172.21.15.27:0/2419746081 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578012870 con 0x7fc588072660 2024-03-06T07:24:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.230+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5880ce8f0 con 0x7fc588072660 2024-03-06T07:24:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.231+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc588067ff0 con 0x7fc588072660 2024-03-06T07:24:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.232+0000 7fc57effd640 1 -- 172.21.15.27:0/2419746081 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc578012a10 con 0x7fc588072660 2024-03-06T07:24:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.232+0000 7fc57effd640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc56c06b500 0x7fc56c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.233+0000 7fc57effd640 1 -- 172.21.15.27:0/2419746081 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc5780899f0 con 0x7fc588072660 2024-03-06T07:24:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.233+0000 7fc58e185640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc56c06b500 0x7fc56c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.233+0000 7fc58e185640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc56c06b500 0x7fc56c06d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc584036000 tx=0x7fc584008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.237+0000 7fc57effd640 1 -- 172.21.15.27:0/2419746081 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5881f7d50 con 0x7fc588072660 2024-03-06T07:24:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.494+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fc588072c30 con 0x7fc588072660 2024-03-06T07:24:37.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.779+0000 7fc57effd640 1 -- 172.21.15.27:0/2419746081 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v125) v1 ==== 132+0+0 (secure 0 0 0) 0x7fc578058530 con 0x7fc588072660 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc56c06b500 msgr2=0x7fc56c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc56c06b500 0x7fc56c06d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc584036000 tx=0x7fc584008030 comp rx=0 tx=0).stop 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc588072660 msgr2=0x7fc5881f30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc588072660 0x7fc5881f30c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c530 tx=0x7fc57800c9f0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 shutdown_connections 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc56c06b500 0x7fc56c06d9c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5880b55d0 0x7fc5881605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc588072660 0x7fc5881f30c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 --2- 172.21.15.27:0/2419746081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc58806e700 0x7fc5881f7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 >> 172.21.15.27:0/2419746081 conn(0x7fc5880b2720 msgr2=0x7fc58815cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 shutdown_connections 2024-03-06T07:24:37.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.781+0000 7fc5903e9640 1 -- 172.21.15.27:0/2419746081 wait complete. 2024-03-06T07:24:37.794 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:24:37.794 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:24:37.794 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:24:37.795 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.900+0000 7f58d08a4640 1 Processor -- start 2024-03-06T07:24:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.900+0000 7f58d08a4640 1 -- start start 2024-03-06T07:24:37.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.901+0000 7f58d08a4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f58cc06e700 0x7f58cc15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.901+0000 7f58d08a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58cc0b55d0 0x7f58cc071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.901+0000 7f58d08a4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58cc0b4160 con 0x7f58cc072660 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.901+0000 7f58d08a4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58cc0b5b40 con 0x7f58cc06e700 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.901+0000 7f58d08a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58cc0b3ba0 con 0x7f58cc0b55d0 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.902+0000 7f58ca59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f58cc06e700 0x7f58cc15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.902+0000 7f58ca59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f58cc06e700 0x7f58cc15ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45428/0 (socket says 172.21.15.27:45428) 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.902+0000 7f58ca59c640 1 -- 172.21.15.27:0/3401024317 learned_addr learned my addr 172.21.15.27:0/3401024317 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.902+0000 7f58cad9d640 1 --2- 172.21.15.27:0/3401024317 >> v2:172.21.15.69:3300/0 conn(0x7f58cc0b55d0 0x7f58cc071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.902+0000 7f58ca59c640 1 -- 172.21.15.27:0/3401024317 >> v2:172.21.15.69:3300/0 conn(0x7f58cc0b55d0 msgr2=0x7f58cc071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.902+0000 7f58ca59c640 1 --2- 172.21.15.27:0/3401024317 >> v2:172.21.15.69:3300/0 conn(0x7f58cc0b55d0 0x7f58cc071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.902+0000 7f58ca59c640 1 -- 172.21.15.27:0/3401024317 >> v1:172.21.15.69:6789/0 conn(0x7f58cc072660 legacy=0x7f58cc06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.902+0000 7f58ca59c640 1 -- 172.21.15.27:0/3401024317 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc0b3e80 con 0x7f58cc06e700 2024-03-06T07:24:37.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.903+0000 7f58ca59c640 1 --2- 172.21.15.27:0/3401024317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 0x7f58cc15ef80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f58cc071580 tx=0x7f58c402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae1f4fee5b0b288 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.903+0000 7f58c959a640 1 -- 172.21.15.27:0/3401024317 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c402ca80 con 0x7f58cc06e700 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.903+0000 7f58c959a640 1 -- 172.21.15.27:0/3401024317 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c402cc20 con 0x7f58cc06e700 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.903+0000 7f58c959a640 1 -- 172.21.15.27:0/3401024317 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c40343f0 con 0x7f58cc06e700 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.903+0000 7f58d08a4640 1 -- 172.21.15.27:0/3401024317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 msgr2=0x7f58cc15ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.903+0000 7f58d08a4640 1 --2- 172.21.15.27:0/3401024317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 0x7f58cc15ef80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f58cc071580 tx=0x7f58c402bee0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.904+0000 7f58d08a4640 1 -- 172.21.15.27:0/3401024317 shutdown_connections 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.904+0000 7f58d08a4640 1 --2- 172.21.15.27:0/3401024317 >> v2:172.21.15.69:3300/0 conn(0x7f58cc0b55d0 0x7f58cc071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.904+0000 7f58d08a4640 1 --2- 172.21.15.27:0/3401024317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 0x7f58cc15ef80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.904+0000 7f58d08a4640 1 -- 172.21.15.27:0/3401024317 >> 172.21.15.27:0/3401024317 conn(0x7f58cc0b2720 msgr2=0x7f58cc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.904+0000 7f58d08a4640 1 -- 172.21.15.27:0/3401024317 shutdown_connections 2024-03-06T07:24:37.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.904+0000 7f58d08a4640 1 -- 172.21.15.27:0/3401024317 wait complete. 2024-03-06T07:24:37.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.905+0000 7f58d08a4640 1 Processor -- start 2024-03-06T07:24:37.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.905+0000 7f58d08a4640 1 -- start start 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58d08a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 0x7f58cc159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58d08a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58cc072660 0x7f58cc15b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58d08a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58cc0b55d0 0x7f58cc0ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58d08a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc072040 con 0x7f58cc06e700 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58d08a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc071ec0 con 0x7f58cc072660 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58d08a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc0721c0 con 0x7f58cc0b55d0 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58ca59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 0x7f58cc159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58ca59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 0x7f58cc159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45434/0 (socket says 172.21.15.27:45434) 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58ca59c640 1 -- 172.21.15.27:0/1230748351 learned_addr learned my addr 172.21.15.27:0/1230748351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58ca59c640 1 -- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58cc0b55d0 msgr2=0x7f58cc0ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58c9d9b640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58cc072660 0x7f58cc15b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58ca59c640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58cc0b55d0 0x7f58cc0ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58ca59c640 1 -- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58cc072660 msgr2=0x7f58cc15b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58ca59c640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58cc072660 0x7f58cc15b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58ca59c640 1 -- 172.21.15.27:0/1230748351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc15a510 con 0x7f58cc06e700 2024-03-06T07:24:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58ca59c640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 0x7f58cc159fd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f58c4036040 tx=0x7f58c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58c2ffd640 1 -- 172.21.15.27:0/1230748351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4043020 con 0x7f58cc06e700 2024-03-06T07:24:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58c2ffd640 1 -- 172.21.15.27:0/1230748351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c403e450 con 0x7f58cc06e700 2024-03-06T07:24:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.906+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58cc158f30 con 0x7f58cc06e700 2024-03-06T07:24:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.907+0000 7f58c2ffd640 1 -- 172.21.15.27:0/1230748351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4009510 con 0x7f58cc06e700 2024-03-06T07:24:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.907+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58cc1597d0 con 0x7f58cc06e700 2024-03-06T07:24:37.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.908+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58cc067ff0 con 0x7f58cc06e700 2024-03-06T07:24:37.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.908+0000 7f58c2ffd640 1 -- 172.21.15.27:0/1230748351 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58c4046020 con 0x7f58cc06e700 2024-03-06T07:24:37.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.908+0000 7f58c2ffd640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f58ac06b4d0 0x7f58ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.909+0000 7f58c2ffd640 1 -- 172.21.15.27:0/1230748351 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f58c40ac740 con 0x7f58cc06e700 2024-03-06T07:24:37.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.909+0000 7f58c9d9b640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f58ac06b4d0 0x7f58ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.909+0000 7f58c9d9b640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f58ac06b4d0 0x7f58ac06d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0b1d40 tx=0x7f58b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:37.913+0000 7f58c2ffd640 1 -- 172.21.15.27:0/1230748351 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58c404d030 con 0x7f58cc06e700 2024-03-06T07:24:38.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.094+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58cc15be20 con 0x7f58cc06e700 2024-03-06T07:24:38.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.095+0000 7f58c2ffd640 1 -- 172.21.15.27:0/1230748351 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+12946 (secure 0 0 0) 0x7f58c407b950 con 0x7f58cc06e700 2024-03-06T07:24:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:38.097 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":38,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:24:36.773435+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/136269066":"2024-03-07T07:24:33.557473+0000","172.21.15.27:6834/136269066":"2024-03-07T07:24:33.557473+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.097+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f58ac06b4d0 msgr2=0x7f58ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.097+0000 7f58d08a4640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f58ac06b4d0 0x7f58ac06d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0b1d40 tx=0x7f58b4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:38.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.097+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 msgr2=0x7f58cc159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.097+0000 7f58d08a4640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 0x7f58cc159fd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f58c4036040 tx=0x7f58c4004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.098+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 shutdown_connections 2024-03-06T07:24:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.098+0000 7f58d08a4640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f58ac06b4d0 0x7f58ac06d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.098+0000 7f58d08a4640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58cc0b55d0 0x7f58cc0ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.098+0000 7f58d08a4640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58cc072660 0x7f58cc15b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.098+0000 7f58d08a4640 1 --2- 172.21.15.27:0/1230748351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58cc06e700 0x7f58cc159fd0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.098+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 >> 172.21.15.27:0/1230748351 conn(0x7f58cc0b2720 msgr2=0x7f58cc15efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.098+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 shutdown_connections 2024-03-06T07:24:38.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.098+0000 7f58d08a4640 1 -- 172.21.15.27:0/1230748351 wait complete. 2024-03-06T07:24:38.110 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:38.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:38.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.219+0000 7fd727755640 1 Processor -- start 2024-03-06T07:24:38.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.219+0000 7fd727755640 1 -- start start 2024-03-06T07:24:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.220+0000 7fd727755640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd720071540 0x7fd720071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.220+0000 7fd727755640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd720071e60 con 0x7fd7200b2d30 2024-03-06T07:24:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.220+0000 7fd727755640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7200d4960 con 0x7fd7200cee80 2024-03-06T07:24:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.220+0000 7fd727755640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7200d3230 con 0x7fd720071540 2024-03-06T07:24:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.220+0000 7fd7254f1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd720071540 0x7fd720071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.220+0000 7fd724cf0640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd7200b2d30 0x7fd7200ce770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55364/0 (socket says 172.21.15.27:55364) 2024-03-06T07:24:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.220+0000 7fd724cf0640 1 -- 172.21.15.27:0/2415926274 learned_addr learned my addr 172.21.15.27:0/2415926274 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.220+0000 7fd7254f1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd720071540 0x7fd720071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45446/0 (socket says 172.21.15.27:45446) 2024-03-06T07:24:38.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.221+0000 7fd7254f1640 1 -- 172.21.15.27:0/2415926274 >> v1:172.21.15.69:6789/0 conn(0x7fd7200cee80 legacy=0x7fd720168900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.221+0000 7fd7254f1640 1 -- 172.21.15.27:0/2415926274 >> v1:172.21.15.27:6789/0 conn(0x7fd7200b2d30 legacy=0x7fd7200ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.221+0000 7fd71ffff640 1 -- 172.21.15.27:0/2415926274 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433181898 0 0) 0x7fd720071e60 con 0x7fd7200b2d30 2024-03-06T07:24:38.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.221+0000 7fd7254f1640 1 -- 172.21.15.27:0/2415926274 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7200d36c0 con 0x7fd720071540 2024-03-06T07:24:38.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.221+0000 7fd7254f1640 1 --2- 172.21.15.27:0/2415926274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd720071540 0x7fd720071920 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd710009d90 tx=0x7fd71002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d15dcbdb87fe38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd71ffff640 1 -- 172.21.15.27:0/2415926274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710039070 con 0x7fd720071540 2024-03-06T07:24:38.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd71ffff640 1 -- 172.21.15.27:0/2415926274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71002cc60 con 0x7fd720071540 2024-03-06T07:24:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd71ffff640 1 -- 172.21.15.27:0/2415926274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710034660 con 0x7fd720071540 2024-03-06T07:24:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd727755640 1 -- 172.21.15.27:0/2415926274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd720071540 msgr2=0x7fd720071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd727755640 1 --2- 172.21.15.27:0/2415926274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd720071540 0x7fd720071920 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd710009d90 tx=0x7fd71002beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd727755640 1 -- 172.21.15.27:0/2415926274 shutdown_connections 2024-03-06T07:24:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd727755640 1 --2- 172.21.15.27:0/2415926274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd720071540 0x7fd720071920 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd727755640 1 -- 172.21.15.27:0/2415926274 >> 172.21.15.27:0/2415926274 conn(0x7fd7200d3d60 msgr2=0x7fd7200d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd727755640 1 -- 172.21.15.27:0/2415926274 shutdown_connections 2024-03-06T07:24:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.222+0000 7fd727755640 1 -- 172.21.15.27:0/2415926274 wait complete. 2024-03-06T07:24:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.223+0000 7fd727755640 1 Processor -- start 2024-03-06T07:24:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.223+0000 7fd727755640 1 -- start start 2024-03-06T07:24:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd727755640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd720071540 0x7fd7201e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd727755640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7200b2d30 0x7fd7201e60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd727755640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200cee80 0x7fd72014b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd727755640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7200d2be0 con 0x7fd7200cee80 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd727755640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7200d2900 con 0x7fd720071540 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd727755640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7200d2ec0 con 0x7fd7200b2d30 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd725cf2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200cee80 0x7fd72014b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd725cf2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200cee80 0x7fd72014b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45462/0 (socket says 172.21.15.27:45462) 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd725cf2640 1 -- 172.21.15.27:0/2954892134 learned_addr learned my addr 172.21.15.27:0/2954892134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd7254f1640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd720071540 0x7fd7201e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd724cf0640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7200b2d30 0x7fd7201e60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd725cf2640 1 -- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7200b2d30 msgr2=0x7fd7201e60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.224+0000 7fd725cf2640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7200b2d30 0x7fd7201e60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.225+0000 7fd725cf2640 1 -- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd720071540 msgr2=0x7fd7201e4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.225+0000 7fd725cf2640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd720071540 0x7fd7201e4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.225+0000 7fd725cf2640 1 -- 172.21.15.27:0/2954892134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7201e6830 con 0x7fd7200cee80 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.225+0000 7fd725cf2640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200cee80 0x7fd72014b1e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd71800c3c0 tx=0x7fd71800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.225+0000 7fd71dffb640 1 -- 172.21.15.27:0/2954892134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd718017070 con 0x7fd7200cee80 2024-03-06T07:24:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.225+0000 7fd71dffb640 1 -- 172.21.15.27:0/2954892134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7180035f0 con 0x7fd7200cee80 2024-03-06T07:24:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.225+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7201e6b10 con 0x7fd7200cee80 2024-03-06T07:24:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.225+0000 7fd71dffb640 1 -- 172.21.15.27:0/2954892134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7180039e0 con 0x7fd7200cee80 2024-03-06T07:24:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.225+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7201e70d0 con 0x7fd7200cee80 2024-03-06T07:24:38.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.226+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7200ce160 con 0x7fd7200cee80 2024-03-06T07:24:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.226+0000 7fd71dffb640 1 -- 172.21.15.27:0/2954892134 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd71801a070 con 0x7fd7200cee80 2024-03-06T07:24:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.227+0000 7fd71dffb640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd70c06b4d0 0x7fd70c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.227+0000 7fd71dffb640 1 -- 172.21.15.27:0/2954892134 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd718089260 con 0x7fd7200cee80 2024-03-06T07:24:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.227+0000 7fd7254f1640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd70c06b4d0 0x7fd70c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.228+0000 7fd7254f1640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd70c06b4d0 0x7fd70c06d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd710036000 tx=0x7fd710030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.231+0000 7fd71dffb640 1 -- 172.21.15.27:0/2954892134 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7200ce160 con 0x7fd7200cee80 2024-03-06T07:24:38.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.413+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7200cf810 con 0x7fd7200cee80 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.414+0000 7fd71dffb640 1 -- 172.21.15.27:0/2954892134 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7200cf810 con 0x7fd7200cee80 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:24:38.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd70c06b4d0 msgr2=0x7fd70c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd70c06b4d0 0x7fd70c06d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd710036000 tx=0x7fd710030000 comp rx=0 tx=0).stop 2024-03-06T07:24:38.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200cee80 msgr2=0x7fd72014b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200cee80 0x7fd72014b1e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd71800c3c0 tx=0x7fd71800c880 comp rx=0 tx=0).stop 2024-03-06T07:24:38.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 shutdown_connections 2024-03-06T07:24:38.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd70c06b4d0 0x7fd70c06d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200cee80 0x7fd72014b1e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7200b2d30 0x7fd7201e60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 --2- 172.21.15.27:0/2954892134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd720071540 0x7fd7201e4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 >> 172.21.15.27:0/2954892134 conn(0x7fd7200d3d60 msgr2=0x7fd7200d1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 shutdown_connections 2024-03-06T07:24:38.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.415+0000 7fd727755640 1 -- 172.21.15.27:0/2954892134 wait complete. 2024-03-06T07:24:38.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:24:38.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.533+0000 7f5814a81640 1 Processor -- start 2024-03-06T07:24:38.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.533+0000 7f5814a81640 1 -- start start 2024-03-06T07:24:38.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f5814a81640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5810156150 0x7f58100cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f5814a81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 0x7f581006e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f5814a81640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581015c600 con 0x7f581006e470 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f5814a81640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f581015c8e0 con 0x7f5810156150 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f5814a81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581015cbc0 con 0x7f58100d01d0 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f580e59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5810156150 0x7f58100cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f580e59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5810156150 0x7f58100cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45476/0 (socket says 172.21.15.27:45476) 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f580e59c640 1 -- 172.21.15.27:0/1432300692 learned_addr learned my addr 172.21.15.27:0/1432300692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f580ed9d640 1 --2- 172.21.15.27:0/1432300692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 0x7f581006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f580e59c640 1 -- 172.21.15.27:0/1432300692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 msgr2=0x7f581006e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f580e59c640 1 --2- 172.21.15.27:0/1432300692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 0x7f581006e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f580e59c640 1 -- 172.21.15.27:0/1432300692 >> v1:172.21.15.69:6789/0 conn(0x7f581006e470 legacy=0x7f58100cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.534+0000 7f580e59c640 1 -- 172.21.15.27:0/1432300692 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581015cea0 con 0x7f5810156150 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.535+0000 7f580e59c640 1 --2- 172.21.15.27:0/1432300692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810156150 0x7f58100cdbd0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f58100d0700 tx=0x7f580002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7b0960a55f7516 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.535+0000 7f580dd9b640 1 -- 172.21.15.27:0/1432300692 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58000315a0 con 0x7f5810156150 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.535+0000 7f580dd9b640 1 -- 172.21.15.27:0/1432300692 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5800037070 con 0x7f5810156150 2024-03-06T07:24:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.535+0000 7f580dd9b640 1 -- 172.21.15.27:0/1432300692 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800031960 con 0x7f5810156150 2024-03-06T07:24:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.536+0000 7f5814a81640 1 -- 172.21.15.27:0/1432300692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810156150 msgr2=0x7f58100cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.536+0000 7f5814a81640 1 --2- 172.21.15.27:0/1432300692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810156150 0x7f58100cdbd0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f58100d0700 tx=0x7f580002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.536+0000 7f5814a81640 1 -- 172.21.15.27:0/1432300692 shutdown_connections 2024-03-06T07:24:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.536+0000 7f5814a81640 1 --2- 172.21.15.27:0/1432300692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 0x7f581006e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.536+0000 7f5814a81640 1 --2- 172.21.15.27:0/1432300692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5810156150 0x7f58100cdbd0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.536+0000 7f5814a81640 1 -- 172.21.15.27:0/1432300692 >> 172.21.15.27:0/1432300692 conn(0x7f5810066ee0 msgr2=0x7f58100672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.536+0000 7f5814a81640 1 -- 172.21.15.27:0/1432300692 shutdown_connections 2024-03-06T07:24:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.536+0000 7f5814a81640 1 -- 172.21.15.27:0/1432300692 wait complete. 2024-03-06T07:24:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.537+0000 7f5814a81640 1 Processor -- start 2024-03-06T07:24:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.537+0000 7f5814a81640 1 -- start start 2024-03-06T07:24:38.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f5814a81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f581006e470 0x7f581014a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f5814a81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 0x7f58100b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f5814a81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5810156150 0x7f58100b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f5814a81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58101566b0 con 0x7f581006e470 2024-03-06T07:24:38.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f5814a81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5810156530 con 0x7f5810156150 2024-03-06T07:24:38.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f5814a81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5810070c80 con 0x7f58100d01d0 2024-03-06T07:24:38.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f581006e470 0x7f581014a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f581006e470 0x7f581014a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45480/0 (socket says 172.21.15.27:45480) 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580e59c640 1 -- 172.21.15.27:0/221671278 learned_addr learned my addr 172.21.15.27:0/221671278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f5807fff640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 0x7f58100b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580ed9d640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5810156150 0x7f58100b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580e59c640 1 -- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 msgr2=0x7f58100b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580e59c640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 0x7f58100b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580e59c640 1 -- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5810156150 msgr2=0x7f58100b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580e59c640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5810156150 0x7f58100b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580e59c640 1 -- 172.21.15.27:0/221671278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581015de40 con 0x7f581006e470 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.538+0000 7f580e59c640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f581006e470 0x7f581014a450 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5800000c00 tx=0x7f58000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.539+0000 7f5806ffd640 1 -- 172.21.15.27:0/221671278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800032840 con 0x7f581006e470 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.539+0000 7f5806ffd640 1 -- 172.21.15.27:0/221671278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58000329e0 con 0x7f581006e470 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.539+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581015e120 con 0x7f581006e470 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.539+0000 7f5806ffd640 1 -- 172.21.15.27:0/221671278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800032dd0 con 0x7f581006e470 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.539+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58101f4e40 con 0x7f581006e470 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.540+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581015e640 con 0x7f581006e470 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.540+0000 7f5806ffd640 1 -- 172.21.15.27:0/221671278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f580004c020 con 0x7f581006e470 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.541+0000 7f5806ffd640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57f006b4d0 0x7f57f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.541+0000 7f5806ffd640 1 -- 172.21.15.27:0/221671278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f580002f020 con 0x7f581006e470 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.541+0000 7f5807fff640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57f006b4d0 0x7f57f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.541+0000 7f5807fff640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57f006b4d0 0x7f57f006d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f58101f1870 tx=0x7f57f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.545+0000 7f5806ffd640 1 -- 172.21.15.27:0/221671278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5800040070 con 0x7f581006e470 2024-03-06T07:24:38.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:38.795+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f5810067ff0 con 0x7f581006e470 2024-03-06T07:24:39.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.783+0000 7f5806ffd640 1 -- 172.21.15.27:0/221671278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v127) v1 ==== 136+0+0 (secure 0 0 0) 0x7f580007b030 con 0x7f581006e470 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57f006b4d0 msgr2=0x7f57f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57f006b4d0 0x7f57f006d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f58101f1870 tx=0x7f57f8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f581006e470 msgr2=0x7f581014a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f581006e470 0x7f581014a450 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5800000c00 tx=0x7f58000046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 shutdown_connections 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57f006b4d0 0x7f57f006d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5810156150 0x7f58100b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f58100d01d0 0x7f58100b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 --2- 172.21.15.27:0/221671278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f581006e470 0x7f581014a450 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 >> 172.21.15.27:0/221671278 conn(0x7f5810066ee0 msgr2=0x7f58100afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 shutdown_connections 2024-03-06T07:24:39.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.786+0000 7f5814a81640 1 -- 172.21.15.27:0/221671278 wait complete. 2024-03-06T07:24:39.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:39.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.916+0000 7f8eb0301640 1 Processor -- start 2024-03-06T07:24:39.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.916+0000 7f8eb0301640 1 -- start start 2024-03-06T07:24:39.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.917+0000 7f8eb0301640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8ea8144640 0x7f8ea806e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.917+0000 7f8eb0301640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ea806e940 0x7f8ea815b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.917+0000 7f8eb0301640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea815bc40 con 0x7f8ea815b670 2024-03-06T07:24:39.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.917+0000 7f8eb0301640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea806df70 con 0x7f8ea806e940 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.917+0000 7f8eb0301640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea815bf20 con 0x7f8ea8144640 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.917+0000 7f8ead89c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ea806e940 0x7f8ea815b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.917+0000 7f8ead89c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ea806e940 0x7f8ea815b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45488/0 (socket says 172.21.15.27:45488) 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.917+0000 7f8ead89c640 1 -- 172.21.15.27:0/1911051134 learned_addr learned my addr 172.21.15.27:0/1911051134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.918+0000 7f8eae09d640 1 --2- 172.21.15.27:0/1911051134 >> v2:172.21.15.69:3300/0 conn(0x7f8ea8144640 0x7f8ea806e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.918+0000 7f8ead89c640 1 -- 172.21.15.27:0/1911051134 >> v2:172.21.15.69:3300/0 conn(0x7f8ea8144640 msgr2=0x7f8ea806e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.918+0000 7f8ead89c640 1 --2- 172.21.15.27:0/1911051134 >> v2:172.21.15.69:3300/0 conn(0x7f8ea8144640 0x7f8ea806e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.918+0000 7f8ead89c640 1 -- 172.21.15.27:0/1911051134 >> v1:172.21.15.69:6789/0 conn(0x7f8ea815b670 legacy=0x7f8ea80ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.918+0000 7f8ead89c640 1 -- 172.21.15.27:0/1911051134 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea806dc00 con 0x7f8ea806e940 2024-03-06T07:24:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.918+0000 7f8eae09d640 1 --2- 172.21.15.27:0/1911051134 >> v2:172.21.15.69:3300/0 conn(0x7f8ea8144640 0x7f8ea806e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8ead89c640 1 --2- 172.21.15.27:0/1911051134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea806e940 0x7f8ea815b130 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8ea806d360 tx=0x7f8e9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55762a07abab9c93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8ead09b640 1 -- 172.21.15.27:0/1911051134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98035020 con 0x7f8ea806e940 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8ead09b640 1 -- 172.21.15.27:0/1911051134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98031870 con 0x7f8ea806e940 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8ead09b640 1 -- 172.21.15.27:0/1911051134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98031c40 con 0x7f8ea806e940 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8eb0301640 1 -- 172.21.15.27:0/1911051134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea806e940 msgr2=0x7f8ea815b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8eb0301640 1 --2- 172.21.15.27:0/1911051134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea806e940 0x7f8ea815b130 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8ea806d360 tx=0x7f8e9802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8eb0301640 1 -- 172.21.15.27:0/1911051134 shutdown_connections 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8eb0301640 1 --2- 172.21.15.27:0/1911051134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea806e940 0x7f8ea815b130 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8eb0301640 1 --2- 172.21.15.27:0/1911051134 >> v2:172.21.15.69:3300/0 conn(0x7f8ea8144640 0x7f8ea806e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8eb0301640 1 -- 172.21.15.27:0/1911051134 >> 172.21.15.27:0/1911051134 conn(0x7f8ea8066d00 msgr2=0x7f8ea8067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8eb0301640 1 -- 172.21.15.27:0/1911051134 shutdown_connections 2024-03-06T07:24:39.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.919+0000 7f8eb0301640 1 -- 172.21.15.27:0/1911051134 wait complete. 2024-03-06T07:24:39.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8eb0301640 1 Processor -- start 2024-03-06T07:24:39.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8eb0301640 1 -- start start 2024-03-06T07:24:39.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8eb0301640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ea806e940 0x7f8ea8155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8eb0301640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8144640 0x7f8ea813c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8eb0301640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ea815b670 0x7f8ea813cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8eb0301640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea80cf190 con 0x7f8ea8144640 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8eb0301640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea80cf010 con 0x7f8ea815b670 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8eb0301640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea80b36d0 con 0x7f8ea806e940 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8ead89c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8144640 0x7f8ea813c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8ead89c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8144640 0x7f8ea813c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45490/0 (socket says 172.21.15.27:45490) 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.921+0000 7f8ead89c640 1 -- 172.21.15.27:0/760496134 learned_addr learned my addr 172.21.15.27:0/760496134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8eae89e640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ea815b670 0x7f8ea813cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8eae09d640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ea806e940 0x7f8ea8155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8ead89c640 1 -- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ea806e940 msgr2=0x7f8ea8155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8ead89c640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ea806e940 0x7f8ea8155710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8ead89c640 1 -- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ea815b670 msgr2=0x7f8ea813cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8ead89c640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ea815b670 0x7f8ea813cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8ead89c640 1 -- 172.21.15.27:0/760496134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea812f970 con 0x7f8ea8144640 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8eae09d640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ea806e940 0x7f8ea8155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8ead89c640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8144640 0x7f8ea813c710 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8e98000c00 tx=0x7f8e980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8e9effd640 1 -- 172.21.15.27:0/760496134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98040070 con 0x7f8ea8144640 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8e9effd640 1 -- 172.21.15.27:0/760496134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9803a420 con 0x7f8ea8144640 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea812fc50 con 0x7f8ea8144640 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8e9effd640 1 -- 172.21.15.27:0/760496134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98035040 con 0x7f8ea8144640 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.922+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea8130210 con 0x7f8ea8144640 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.923+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea80733c0 con 0x7f8ea8144640 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.924+0000 7f8e9effd640 1 -- 172.21.15.27:0/760496134 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e98038020 con 0x7f8ea8144640 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.924+0000 7f8e9effd640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e8c06b4d0 0x7f8e8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.924+0000 7f8e9effd640 1 -- 172.21.15.27:0/760496134 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8e980abf30 con 0x7f8ea8144640 2024-03-06T07:24:39.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.924+0000 7f8eae09d640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e8c06b4d0 0x7f8e8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.925+0000 7f8eae09d640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e8c06b4d0 0x7f8e8c06d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0006fd0 tx=0x7f8ea0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:39.928+0000 7f8e9effd640 1 -- 172.21.15.27:0/760496134 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e9807b0b0 con 0x7f8ea8144640 2024-03-06T07:24:40.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.113+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8ea815ae10 con 0x7f8ea8144640 2024-03-06T07:24:40.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.116+0000 7f8e9effd640 1 -- 172.21.15.27:0/760496134 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8e9807a9b0 con 0x7f8ea8144640 2024-03-06T07:24:40.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e8c06b4d0 msgr2=0x7f8e8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e8c06b4d0 0x7f8e8c06d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0006fd0 tx=0x7f8ea0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8144640 msgr2=0x7f8ea813c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8144640 0x7f8ea813c710 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8e98000c00 tx=0x7f8e980046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 shutdown_connections 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e8c06b4d0 0x7f8e8c06d990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ea815b670 0x7f8ea813cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea8144640 0x7f8ea813c710 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 --2- 172.21.15.27:0/760496134 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8ea806e940 0x7f8ea8155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 >> 172.21.15.27:0/760496134 conn(0x7f8ea8066d00 msgr2=0x7f8ea8138060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 shutdown_connections 2024-03-06T07:24:40.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.119+0000 7f8eb0301640 1 -- 172.21.15.27:0/760496134 wait complete. 2024-03-06T07:24:40.132 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:40.132 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:40.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:40.216 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:40.216 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:24:40.217 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:40.232 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:40.232 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:24:40.232 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:40.250 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:40.250 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:24:40.250 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:40.287 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:40.287 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:24:40.287 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:40.306 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:40.306 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:24:40.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.431+0000 7f0045f83640 1 Processor -- start 2024-03-06T07:24:40.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.431+0000 7f0045f83640 1 -- start start 2024-03-06T07:24:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f0045f83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 0x7f004006ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f0045f83640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00400b2460 0x7f004006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f0045f83640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f004015e410 con 0x7f004006ef70 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f0045f83640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0040162700 con 0x7f00400b2460 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f0045f83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0040161af0 con 0x7f004015e030 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f003ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00400b2460 0x7f004006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f003ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00400b2460 0x7f004006d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45494/0 (socket says 172.21.15.27:45494) 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f003ffff640 1 -- 172.21.15.27:0/1022603061 learned_addr learned my addr 172.21.15.27:0/1022603061 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f003f7fe640 1 --2- 172.21.15.27:0/1022603061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 0x7f004006ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f003ffff640 1 -- 172.21.15.27:0/1022603061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 msgr2=0x7f004006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f003ffff640 1 --2- 172.21.15.27:0/1022603061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 0x7f004006ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f003ffff640 1 -- 172.21.15.27:0/1022603061 >> v1:172.21.15.69:6789/0 conn(0x7f004006ef70 legacy=0x7f00400b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.432+0000 7f003ffff640 1 -- 172.21.15.27:0/1022603061 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040161dd0 con 0x7f00400b2460 2024-03-06T07:24:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.433+0000 7f003ffff640 1 --2- 172.21.15.27:0/1022603061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00400b2460 0x7f004006d210 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0038009870 tx=0x7f003802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116be06b3d063519 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.433+0000 7f003e7fc640 1 -- 172.21.15.27:0/1022603061 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0038035020 con 0x7f00400b2460 2024-03-06T07:24:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.433+0000 7f003e7fc640 1 -- 172.21.15.27:0/1022603061 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0038031640 con 0x7f00400b2460 2024-03-06T07:24:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.433+0000 7f003e7fc640 1 -- 172.21.15.27:0/1022603061 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003802d040 con 0x7f00400b2460 2024-03-06T07:24:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.434+0000 7f0045f83640 1 -- 172.21.15.27:0/1022603061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00400b2460 msgr2=0x7f004006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.434+0000 7f0045f83640 1 --2- 172.21.15.27:0/1022603061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00400b2460 0x7f004006d210 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0038009870 tx=0x7f003802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.434+0000 7f0045f83640 1 -- 172.21.15.27:0/1022603061 shutdown_connections 2024-03-06T07:24:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.434+0000 7f0045f83640 1 --2- 172.21.15.27:0/1022603061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00400b2460 0x7f004006d210 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.434+0000 7f0045f83640 1 --2- 172.21.15.27:0/1022603061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 0x7f004006ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.434+0000 7f0045f83640 1 -- 172.21.15.27:0/1022603061 >> 172.21.15.27:0/1022603061 conn(0x7f00400b55e0 msgr2=0x7f00400b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.434+0000 7f0045f83640 1 -- 172.21.15.27:0/1022603061 shutdown_connections 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.434+0000 7f0045f83640 1 -- 172.21.15.27:0/1022603061 wait complete. 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.435+0000 7f0045f83640 1 Processor -- start 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.435+0000 7f0045f83640 1 -- start start 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f0045f83640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006ef70 0x7f004015cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f0045f83640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400b2460 0x7f00400b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f0045f83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 0x7f0040155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f0045f83640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00400b31e0 con 0x7f004006ef70 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f0045f83640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0040162390 con 0x7f00400b2460 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f0045f83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00400b3360 con 0x7f004015e030 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006ef70 0x7f004015cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006ef70 0x7f004015cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45504/0 (socket says 172.21.15.27:45504) 2024-03-06T07:24:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003f7fe640 1 -- 172.21.15.27:0/3353645823 learned_addr learned my addr 172.21.15.27:0/3353645823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003effd640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400b2460 0x7f00400b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003f7fe640 1 -- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 msgr2=0x7f0040155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003ffff640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 0x7f0040155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003f7fe640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 0x7f0040155970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003f7fe640 1 -- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400b2460 msgr2=0x7f00400b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003f7fe640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400b2460 0x7f00400b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003f7fe640 1 -- 172.21.15.27:0/3353645823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040154910 con 0x7f004006ef70 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003effd640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400b2460 0x7f00400b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.436+0000 7f003f7fe640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006ef70 0x7f004015cd10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0030007e40 tx=0x7f003000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.437+0000 7f0027fff640 1 -- 172.21.15.27:0/3353645823 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030014620 con 0x7f004006ef70 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.437+0000 7f0027fff640 1 -- 172.21.15.27:0/3353645823 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0030019070 con 0x7f004006ef70 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.437+0000 7f0027fff640 1 -- 172.21.15.27:0/3353645823 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030014a00 con 0x7f004006ef70 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.437+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0040154bb0 con 0x7f004006ef70 2024-03-06T07:24:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.437+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0040155170 con 0x7f004006ef70 2024-03-06T07:24:40.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.438+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0040067ff0 con 0x7f004006ef70 2024-03-06T07:24:40.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.438+0000 7f0027fff640 1 -- 172.21.15.27:0/3353645823 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0030014ba0 con 0x7f004006ef70 2024-03-06T07:24:40.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.439+0000 7f0027fff640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f002c06b4d0 0x7f002c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.439+0000 7f0027fff640 1 -- 172.21.15.27:0/3353645823 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f003008a6c0 con 0x7f004006ef70 2024-03-06T07:24:40.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.439+0000 7f003effd640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f002c06b4d0 0x7f002c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.439+0000 7f003effd640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f002c06b4d0 0x7f002c06d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f00400aeed0 tx=0x7f0034009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.443+0000 7f0027fff640 1 -- 172.21.15.27:0/3353645823 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00300598d0 con 0x7f004006ef70 2024-03-06T07:24:40.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.662+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00400af3e0 con 0x7f004006ef70 2024-03-06T07:24:40.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.663+0000 7f0027fff640 1 -- 172.21.15.27:0/3353645823 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f00300591d0 con 0x7f004006ef70 2024-03-06T07:24:40.666 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:40.666 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1854120093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1854120093},{"type":"v1","addr":"172.21.15.69:6837","nonce":1854120093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1993036537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1993036537},{"type":"v1","addr":"172.21.15.69:6836","nonce":1993036537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/655217111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":655217111},{"type":"v1","addr":"172.21.15.27:6833","nonce":655217111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5679,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4130421607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":4130421607},{"type":"v1","addr":"172.21.15.27:6835","nonce":4130421607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-03-06T07:24:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-03-06T07:24:40.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.664+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f002c06b4d0 msgr2=0x7f002c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.664+0000 7f0045f83640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f002c06b4d0 0x7f002c06d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f00400aeed0 tx=0x7f0034009040 comp rx=0 tx=0).stop 2024-03-06T07:24:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006ef70 msgr2=0x7f004015cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006ef70 0x7f004015cd10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0030007e40 tx=0x7f003000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 shutdown_connections 2024-03-06T07:24:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f002c06b4d0 0x7f002c06d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f004015e030 0x7f0040155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00400b2460 0x7f00400b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 --2- 172.21.15.27:0/3353645823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006ef70 0x7f004015cd10 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 >> 172.21.15.27:0/3353645823 conn(0x7f00400b55e0 msgr2=0x7f0040160930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 shutdown_connections 2024-03-06T07:24:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.665+0000 7f0045f83640 1 -- 172.21.15.27:0/3353645823 wait complete. 2024-03-06T07:24:40.679 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:40.679 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:40.679 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:40.679 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:40.679 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:40.679 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:40.680 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:40.680 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:40.680 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:40.680 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:40.680 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:40.680 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:40.680 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:40.680 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:40.681 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:40.681 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:40.688 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:40.689 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:40.690 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:40.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:40.693 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:40.693 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:40.695 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:40.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:40.732 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:40.733 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:40.734 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:40.734 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:40.737 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:40.737 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:40.737 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:40.738 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:40.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:40.816 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:24:40.816 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:24:40.823 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:24:40.823 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:24:40.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.849+0000 7f0731a15640 1 Processor -- start 2024-03-06T07:24:40.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.849+0000 7f0731a15640 1 -- start start 2024-03-06T07:24:40.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.850+0000 7f0731a15640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f072c15df00 0x7f072c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.850+0000 7f0731a15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.850+0000 7f0731a15640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072c0cea30 con 0x7f072c0b9a30 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.850+0000 7f0731a15640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072c0d2af0 con 0x7f072c15df00 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.850+0000 7f0731a15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c0cf3c0 con 0x7f072c0b95a0 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.850+0000 7f072b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f072c0b9a30 0x7f072c0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55370/0 (socket says 172.21.15.27:55370) 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.850+0000 7f072b7fe640 1 -- 172.21.15.27:0/3271364685 learned_addr learned my addr 172.21.15.27:0/3271364685 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.850+0000 7f072affd640 1 --2- 172.21.15.27:0/3271364685 >> v2:172.21.15.69:3300/0 conn(0x7f072c15df00 0x7f072c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.850+0000 7f072a7fc640 1 --2- 172.21.15.27:0/3271364685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.851+0000 7f0729ffb640 1 -- 172.21.15.27:0/3271364685 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333950163 0 0) 0x7f072c0cea30 con 0x7f072c0b9a30 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.851+0000 7f0729ffb640 1 -- 172.21.15.27:0/3271364685 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0718003520 con 0x7f072c0b9a30 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.851+0000 7f072a7fc640 1 -- 172.21.15.27:0/3271364685 >> v2:172.21.15.69:3300/0 conn(0x7f072c15df00 msgr2=0x7f072c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.851+0000 7f072a7fc640 1 --2- 172.21.15.27:0/3271364685 >> v2:172.21.15.69:3300/0 conn(0x7f072c15df00 0x7f072c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.851+0000 7f072a7fc640 1 -- 172.21.15.27:0/3271364685 >> v1:172.21.15.27:6789/0 conn(0x7f072c0b9a30 legacy=0x7f072c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.851+0000 7f072a7fc640 1 -- 172.21.15.27:0/3271364685 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c0cf0e0 con 0x7f072c0b95a0 2024-03-06T07:24:40.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.851+0000 7f072affd640 1 --2- 172.21.15.27:0/3271364685 >> v2:172.21.15.69:3300/0 conn(0x7f072c15df00 0x7f072c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f072a7fc640 1 --2- 172.21.15.27:0/3271364685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b5700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f07140097a0 tx=0x7f071402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbcfb59413f84844 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0729ffb640 1 -- 172.21.15.27:0/3271364685 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714035020 con 0x7f072c0b95a0 2024-03-06T07:24:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0729ffb640 1 -- 172.21.15.27:0/3271364685 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0714031830 con 0x7f072c0b95a0 2024-03-06T07:24:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0729ffb640 1 -- 172.21.15.27:0/3271364685 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714031be0 con 0x7f072c0b95a0 2024-03-06T07:24:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0731a15640 1 -- 172.21.15.27:0/3271364685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 msgr2=0x7f072c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0731a15640 1 --2- 172.21.15.27:0/3271364685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b5700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f07140097a0 tx=0x7f071402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0731a15640 1 -- 172.21.15.27:0/3271364685 shutdown_connections 2024-03-06T07:24:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0731a15640 1 --2- 172.21.15.27:0/3271364685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b5700 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0731a15640 1 --2- 172.21.15.27:0/3271364685 >> v2:172.21.15.69:3300/0 conn(0x7f072c15df00 0x7f072c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0731a15640 1 -- 172.21.15.27:0/3271364685 >> 172.21.15.27:0/3271364685 conn(0x7f072c071960 msgr2=0x7f072c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0731a15640 1 -- 172.21.15.27:0/3271364685 shutdown_connections 2024-03-06T07:24:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.852+0000 7f0731a15640 1 -- 172.21.15.27:0/3271364685 wait complete. 2024-03-06T07:24:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.853+0000 7f0731a15640 1 Processor -- start 2024-03-06T07:24:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f0731a15640 1 -- start start 2024-03-06T07:24:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f0731a15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f0731a15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f072c15df00 0x7f072c0cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f0731a15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c0b34c0 0x7f072c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f0731a15640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c0cec70 con 0x7f072c0b34c0 2024-03-06T07:24:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f0731a15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c0ce750 con 0x7f072c15df00 2024-03-06T07:24:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f0731a15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c0cedf0 con 0x7f072c0b95a0 2024-03-06T07:24:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f072b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c0b34c0 0x7f072c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f072b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c0b34c0 0x7f072c0cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45552/0 (socket says 172.21.15.27:45552) 2024-03-06T07:24:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.854+0000 7f072b7fe640 1 -- 172.21.15.27:0/1202724883 learned_addr learned my addr 172.21.15.27:0/1202724883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f072a7fc640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f072c15df00 0x7f072c0cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f072affd640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f072b7fe640 1 -- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 msgr2=0x7f072c0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f072b7fe640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f072b7fe640 1 -- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f072c15df00 msgr2=0x7f072c0cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f072b7fe640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f072c15df00 0x7f072c0cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f072b7fe640 1 -- 172.21.15.27:0/1202724883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c0b3990 con 0x7f072c0b34c0 2024-03-06T07:24:40.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f072affd640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f072b7fe640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c0b34c0 0x7f072c0cce40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f072400c430 tx=0x7f072400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f0723fff640 1 -- 172.21.15.27:0/1202724883 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724017070 con 0x7f072c0b34c0 2024-03-06T07:24:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f0723fff640 1 -- 172.21.15.27:0/1202724883 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0724005ce0 con 0x7f072c0b34c0 2024-03-06T07:24:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c0b3c70 con 0x7f072c0b34c0 2024-03-06T07:24:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f0723fff640 1 -- 172.21.15.27:0/1202724883 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240072d0 con 0x7f072c0b34c0 2024-03-06T07:24:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.855+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c0ca7c0 con 0x7f072c0b34c0 2024-03-06T07:24:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.856+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072c067ff0 con 0x7f072c0b34c0 2024-03-06T07:24:40.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.857+0000 7f0723fff640 1 -- 172.21.15.27:0/1202724883 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f072401a070 con 0x7f072c0b34c0 2024-03-06T07:24:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.857+0000 7f0723fff640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f070c06b4d0 0x7f070c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.857+0000 7f0723fff640 1 -- 172.21.15.27:0/1202724883 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0724089210 con 0x7f072c0b34c0 2024-03-06T07:24:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.857+0000 7f072affd640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f070c06b4d0 0x7f070c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.858+0000 7f072affd640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f070c06b4d0 0x7f070c06d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f071c009fd0 tx=0x7f071c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:40.862+0000 7f0723fff640 1 -- 172.21.15.27:0/1202724883 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0724057d40 con 0x7f072c0b34c0 2024-03-06T07:24:41.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.055+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f072c15b790 con 0x7f072c0b34c0 2024-03-06T07:24:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.799+0000 7f0723fff640 1 -- 172.21.15.27:0/1202724883 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v40) v1 ==== 97+0+0 (secure 0 0 0) 0x7f072405bde0 con 0x7f072c0b34c0 2024-03-06T07:24:41.802 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f070c06b4d0 msgr2=0x7f070c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f070c06b4d0 0x7f070c06d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f071c009fd0 tx=0x7f071c00b040 comp rx=0 tx=0).stop 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c0b34c0 msgr2=0x7f072c0cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c0b34c0 0x7f072c0cce40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f072400c430 tx=0x7f072400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 shutdown_connections 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f070c06b4d0 0x7f070c06d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f072c0b34c0 0x7f072c0cce40 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f072c15df00 0x7f072c0cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 --2- 172.21.15.27:0/1202724883 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f072c0b95a0 0x7f072c0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 >> 172.21.15.27:0/1202724883 conn(0x7f072c071960 msgr2=0x7f072c0b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 shutdown_connections 2024-03-06T07:24:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.802+0000 7f0731a15640 1 -- 172.21.15.27:0/1202724883 wait complete. 2024-03-06T07:24:41.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:41.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.928+0000 7f635c6eb640 1 Processor -- start 2024-03-06T07:24:41.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.928+0000 7f635c6eb640 1 -- start start 2024-03-06T07:24:41.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.929+0000 7f635c6eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635415e030 0x7f635406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.929+0000 7f635c6eb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635415e410 con 0x7f63540b2460 2024-03-06T07:24:41.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.929+0000 7f635c6eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6354161af0 con 0x7f635406ef70 2024-03-06T07:24:41.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.929+0000 7f635c6eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6354162700 con 0x7f635415e030 2024-03-06T07:24:41.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.929+0000 7f635ac88640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f63540b2460 0x7f635406d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55376/0 (socket says 172.21.15.27:55376) 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.929+0000 7f635ac88640 1 -- 172.21.15.27:0/3750503719 learned_addr learned my addr 172.21.15.27:0/3750503719 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.929+0000 7f635a487640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635415e030 0x7f635406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.929+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549925606 0 0) 0x7f634c001490 con 0x7f63540b2460 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.929+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6340003520 con 0x7f63540b2460 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2989645602 0 0) 0x7f6340003520 con 0x7f63540b2460 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635415e030 msgr2=0x7f635406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635415e030 0x7f635406ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 >> v1:172.21.15.69:6789/0 conn(0x7f635406ef70 legacy=0x7f63540b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6354161dd0 con 0x7f63540b2460 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f634c001bc0 con 0x7f63540b2460 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 >> v1:172.21.15.27:6789/0 conn(0x7f63540b2460 legacy=0x7f635406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63400039e0 0x7f6340003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6340004800 0x7f6340004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635415e030 0x7f63400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6354162700 con 0x7f63400039e0 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634000ba70 con 0x7f6340004800 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634000bc70 con 0x7f635415e030 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f635a487640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63400039e0 0x7f6340003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f635ac88640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635415e030 0x7f63400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f6359c86640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6340004800 0x7f6340004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f635a487640 1 -- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635415e030 msgr2=0x7f63400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f635a487640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635415e030 0x7f63400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f635a487640 1 -- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6340004800 msgr2=0x7f6340004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f635a487640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6340004800 0x7f6340004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.930+0000 7f635a487640 1 -- 172.21.15.27:0/3750503719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6340003520 con 0x7f63400039e0 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.931+0000 7f635a487640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63400039e0 0x7f6340003e00 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6350009a90 tx=0x7f6340005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbecaf20700b0ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.931+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350039070 con 0x7f63400039e0 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.931+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635002c680 con 0x7f63400039e0 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.931+0000 7f6359485640 1 -- 172.21.15.27:0/3750503719 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350009690 con 0x7f63400039e0 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.931+0000 7f635c6eb640 1 -- 172.21.15.27:0/3750503719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63400039e0 msgr2=0x7f6340003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.931+0000 7f635c6eb640 1 --2- 172.21.15.27:0/3750503719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63400039e0 0x7f6340003e00 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6350009a90 tx=0x7f6340005830 comp rx=0 tx=0).stop 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.931+0000 7f635ac88640 1 -- 172.21.15.27:0/3750503719 reap_dead start 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.932+0000 7f635c6eb640 1 -- 172.21.15.27:0/3750503719 shutdown_connections 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.932+0000 7f635c6eb640 1 -- 172.21.15.27:0/3750503719 >> 172.21.15.27:0/3750503719 conn(0x7f63540b55e0 msgr2=0x7f63540b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.932+0000 7f635c6eb640 1 -- 172.21.15.27:0/3750503719 shutdown_connections 2024-03-06T07:24:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.932+0000 7f635c6eb640 1 -- 172.21.15.27:0/3750503719 wait complete. 2024-03-06T07:24:41.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.933+0000 7f635c6eb640 1 Processor -- start 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.933+0000 7f635c6eb640 1 -- start start 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635c6eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63400039e0 0x7f635415f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635c6eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354155890 0x7f63541f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635c6eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63541f40f0 0x7f63541f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635c6eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63540b31e0 con 0x7f63541f40f0 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635c6eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6354162390 con 0x7f63400039e0 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635c6eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63540b3360 con 0x7f6354155890 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635ac88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63541f40f0 0x7f63541f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635ac88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63541f40f0 0x7f63541f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45568/0 (socket says 172.21.15.27:45568) 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635ac88640 1 -- 172.21.15.27:0/4078028951 learned_addr learned my addr 172.21.15.27:0/4078028951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635a487640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63400039e0 0x7f635415f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f6359c86640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354155890 0x7f63541f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635ac88640 1 -- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354155890 msgr2=0x7f63541f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635ac88640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354155890 0x7f63541f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635ac88640 1 -- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63400039e0 msgr2=0x7f635415f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635ac88640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63400039e0 0x7f635415f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635ac88640 1 -- 172.21.15.27:0/4078028951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6354155d10 con 0x7f63541f40f0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635ac88640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63541f40f0 0x7f63541f4570 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f635406fa70 tx=0x7f634c00af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f634affd640 1 -- 172.21.15.27:0/4078028951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c007980 con 0x7f63541f40f0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f634affd640 1 -- 172.21.15.27:0/4078028951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c019070 con 0x7f63541f40f0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635415b910 con 0x7f63541f40f0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.934+0000 7f634affd640 1 -- 172.21.15.27:0/4078028951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c013630 con 0x7f63541f40f0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.935+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635415f9e0 con 0x7f63541f40f0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.935+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6354067ff0 con 0x7f63541f40f0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.936+0000 7f634affd640 1 -- 172.21.15.27:0/4078028951 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f634c0137d0 con 0x7f63541f40f0 2024-03-06T07:24:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.936+0000 7f634affd640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633806b500 0x7f633806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.936+0000 7f634affd640 1 -- 172.21.15.27:0/4078028951 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f634c08a3d0 con 0x7f63541f40f0 2024-03-06T07:24:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.937+0000 7f635a487640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633806b500 0x7f633806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.937+0000 7f635a487640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633806b500 0x7f633806d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6350036000 tx=0x7f6350030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:41.940+0000 7f634affd640 1 -- 172.21.15.27:0/4078028951 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6354155d10 con 0x7f63541f40f0 2024-03-06T07:24:42.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.191+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f63540b2910 con 0x7f63541f40f0 2024-03-06T07:24:42.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.192+0000 7f634affd640 1 -- 172.21.15.27:0/4078028951 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f635415e410 con 0x7f63541f40f0 2024-03-06T07:24:42.194 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:42.194 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g==", 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.195 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ==", 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg==", 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ==", 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw==", 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA==", 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA==", 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg==", 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.200 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A==", 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw==", 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA==", 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A==", 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA==", 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ==", 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ==", 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg==", 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlkc2iEhAA6BuYba0FzXeVjGPOP8sgyQ==", 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:24:42.206 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw==", 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg==", 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:24:42.208 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.193+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633806b500 msgr2=0x7f633806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.193+0000 7f635c6eb640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633806b500 0x7f633806d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6350036000 tx=0x7f6350030000 comp rx=0 tx=0).stop 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63541f40f0 msgr2=0x7f63541f4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63541f40f0 0x7f63541f4570 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f635406fa70 tx=0x7f634c00af60 comp rx=0 tx=0).stop 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 shutdown_connections 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633806b500 0x7f633806d9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63541f40f0 0x7f63541f4570 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6354155890 0x7f63541f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 --2- 172.21.15.27:0/4078028951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63400039e0 0x7f635415f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 >> 172.21.15.27:0/4078028951 conn(0x7f63540b55e0 msgr2=0x7f63540b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 shutdown_connections 2024-03-06T07:24:42.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.194+0000 7f635c6eb640 1 -- 172.21.15.27:0/4078028951 wait complete. 2024-03-06T07:24:42.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:42.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.315+0000 7fc74972c640 1 Processor -- start 2024-03-06T07:24:42.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.315+0000 7fc74972c640 1 -- start start 2024-03-06T07:24:42.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc74972c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 0x7fc7440d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc74972c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7440b95a0 0x7fc7440b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc74972c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7440b9a30 0x7fc7440b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc74972c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7440cea30 con 0x7fc7440b9a30 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc74972c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7440d2af0 con 0x7fc7440b95a0 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc74972c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440cf3c0 con 0x7fc74415df00 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc7437fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7440b9a30 0x7fc7440b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc7437fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7440b9a30 0x7fc7440b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45570/0 (socket says 172.21.15.27:45570) 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc7437fe640 1 -- 172.21.15.27:0/2765669420 learned_addr learned my addr 172.21.15.27:0/2765669420 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc7427fc640 1 --2- 172.21.15.27:0/2765669420 >> v2:172.21.15.69:3300/0 conn(0x7fc7440b95a0 0x7fc7440b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc742ffd640 1 --2- 172.21.15.27:0/2765669420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 0x7fc7440d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc7437fe640 1 -- 172.21.15.27:0/2765669420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 msgr2=0x7fc7440d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc7437fe640 1 --2- 172.21.15.27:0/2765669420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 0x7fc7440d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc7437fe640 1 -- 172.21.15.27:0/2765669420 >> v2:172.21.15.69:3300/0 conn(0x7fc7440b95a0 msgr2=0x7fc7440b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc7437fe640 1 --2- 172.21.15.27:0/2765669420 >> v2:172.21.15.69:3300/0 conn(0x7fc7440b95a0 0x7fc7440b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.316+0000 7fc7437fe640 1 -- 172.21.15.27:0/2765669420 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7440cf0e0 con 0x7fc7440b9a30 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.317+0000 7fc7437fe640 1 --2- 172.21.15.27:0/2765669420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b9a30 0x7fc7440b5c40 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc74408cbc0 tx=0x7fc73c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66957b6b255f1f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.317+0000 7fc741ffb640 1 -- 172.21.15.27:0/2765669420 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c02ca70 con 0x7fc7440b9a30 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.317+0000 7fc741ffb640 1 -- 172.21.15.27:0/2765669420 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c02cc10 con 0x7fc7440b9a30 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.317+0000 7fc741ffb640 1 -- 172.21.15.27:0/2765669420 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c0343f0 con 0x7fc7440b9a30 2024-03-06T07:24:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.318+0000 7fc74972c640 1 -- 172.21.15.27:0/2765669420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b9a30 msgr2=0x7fc7440b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.318+0000 7fc74972c640 1 --2- 172.21.15.27:0/2765669420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b9a30 0x7fc7440b5c40 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc74408cbc0 tx=0x7fc73c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.318+0000 7fc74972c640 1 -- 172.21.15.27:0/2765669420 shutdown_connections 2024-03-06T07:24:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.318+0000 7fc74972c640 1 --2- 172.21.15.27:0/2765669420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b9a30 0x7fc7440b5c40 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.318+0000 7fc74972c640 1 --2- 172.21.15.27:0/2765669420 >> v2:172.21.15.69:3300/0 conn(0x7fc7440b95a0 0x7fc7440b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.318+0000 7fc74972c640 1 --2- 172.21.15.27:0/2765669420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 0x7fc7440d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.318+0000 7fc74972c640 1 -- 172.21.15.27:0/2765669420 >> 172.21.15.27:0/2765669420 conn(0x7fc744071960 msgr2=0x7fc744071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.318+0000 7fc74972c640 1 -- 172.21.15.27:0/2765669420 shutdown_connections 2024-03-06T07:24:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.318+0000 7fc74972c640 1 -- 172.21.15.27:0/2765669420 wait complete. 2024-03-06T07:24:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.319+0000 7fc74972c640 1 Processor -- start 2024-03-06T07:24:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.319+0000 7fc74972c640 1 -- start start 2024-03-06T07:24:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc74972c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b95a0 0x7fc74414d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc74972c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 0x7fc7440c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc74972c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc744149cc0 0x7fc74415f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc74972c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440cec70 con 0x7fc7440b95a0 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc74972c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440ce750 con 0x7fc744149cc0 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc74972c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440cedf0 con 0x7fc74415df00 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc742ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b95a0 0x7fc74414d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc742ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b95a0 0x7fc74414d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45584/0 (socket says 172.21.15.27:45584) 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc742ffd640 1 -- 172.21.15.27:0/4059634526 learned_addr learned my addr 172.21.15.27:0/4059634526 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc7437fe640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc744149cc0 0x7fc74415f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc7427fc640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 0x7fc7440c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc742ffd640 1 -- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 msgr2=0x7fc7440c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc742ffd640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 0x7fc7440c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc742ffd640 1 -- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc744149cc0 msgr2=0x7fc74415f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc742ffd640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc744149cc0 0x7fc74415f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc742ffd640 1 -- 172.21.15.27:0/4059634526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7440c7cf0 con 0x7fc7440b95a0 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc7427fc640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 0x7fc7440c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.320+0000 7fc742ffd640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b95a0 0x7fc74414d1b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc73400e9b0 tx=0x7fc73400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.321+0000 7fc72b7fe640 1 -- 172.21.15.27:0/4059634526 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc734017070 con 0x7fc7440b95a0 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.321+0000 7fc72b7fe640 1 -- 172.21.15.27:0/4059634526 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73400c9a0 con 0x7fc7440b95a0 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.321+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7440b5b30 con 0x7fc7440b95a0 2024-03-06T07:24:42.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.321+0000 7fc72b7fe640 1 -- 172.21.15.27:0/4059634526 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73400cd90 con 0x7fc7440b95a0 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.321+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7440b3000 con 0x7fc7440b95a0 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.321+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc744067ff0 con 0x7fc7440b95a0 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.322+0000 7fc72b7fe640 1 -- 172.21.15.27:0/4059634526 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc73401a070 con 0x7fc7440b95a0 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.322+0000 7fc72b7fe640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc73006b4d0 0x7fc73006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.322+0000 7fc72b7fe640 1 -- 172.21.15.27:0/4059634526 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc734089d40 con 0x7fc7440b95a0 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.323+0000 7fc7427fc640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc73006b4d0 0x7fc73006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.323+0000 7fc7427fc640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc73006b4d0 0x7fc73006d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc74415c3d0 tx=0x7fc738008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.327+0000 7fc72b7fe640 1 -- 172.21.15.27:0/4059634526 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7340588c0 con 0x7fc7440b95a0 2024-03-06T07:24:42.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.511+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc74406b780 con 0x7fc7440b95a0 2024-03-06T07:24:42.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.512+0000 7fc72b7fe640 1 -- 172.21.15.27:0/4059634526 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+12819 (secure 0 0 0) 0x7fc73405c960 con 0x7fc7440b95a0 2024-03-06T07:24:42.514 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:42.515 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":42,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:24:41.810609+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:42.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc73006b4d0 msgr2=0x7fc73006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc73006b4d0 0x7fc73006d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc74415c3d0 tx=0x7fc738008040 comp rx=0 tx=0).stop 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b95a0 msgr2=0x7fc74414d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b95a0 0x7fc74414d1b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc73400e9b0 tx=0x7fc73400ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 shutdown_connections 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc73006b4d0 0x7fc73006d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc744149cc0 0x7fc74415f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc74415df00 0x7fc7440c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 --2- 172.21.15.27:0/4059634526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7440b95a0 0x7fc74414d1b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 >> 172.21.15.27:0/4059634526 conn(0x7fc744071960 msgr2=0x7fc744071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 shutdown_connections 2024-03-06T07:24:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.514+0000 7fc74972c640 1 -- 172.21.15.27:0/4059634526 wait complete. 2024-03-06T07:24:42.527 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:42.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:42.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.635+0000 7fddd37f7640 1 Processor -- start 2024-03-06T07:24:42.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.635+0000 7fddd37f7640 1 -- start start 2024-03-06T07:24:42.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddd37f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddd37f7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc070980 con 0x7fddcc0b4830 2024-03-06T07:24:42.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddd37f7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc142f60 con 0x7fddcc0b29a0 2024-03-06T07:24:42.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddd37f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc1458e0 con 0x7fddcc142170 2024-03-06T07:24:42.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddd0d92640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fddcc0b4830 0x7fddcc160280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55390/0 (socket says 172.21.15.27:55390) 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddd0d92640 1 -- 172.21.15.27:0/205496054 learned_addr learned my addr 172.21.15.27:0/205496054 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddd1593640 1 --2- 172.21.15.27:0/205496054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddcbfff640 1 -- 172.21.15.27:0/205496054 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799605005 0 0) 0x7fddcc070980 con 0x7fddcc0b4830 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddcbfff640 1 -- 172.21.15.27:0/205496054 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb4003520 con 0x7fddcc0b4830 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddcbfff640 1 -- 172.21.15.27:0/205496054 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664514131 0 0) 0x7fddcc142f60 con 0x7fddcc0b29a0 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.636+0000 7fddcbfff640 1 -- 172.21.15.27:0/205496054 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddcc070980 con 0x7fddcc0b29a0 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.637+0000 7fddd1593640 1 -- 172.21.15.27:0/205496054 >> v1:172.21.15.69:6789/0 conn(0x7fddcc0b29a0 legacy=0x7fddcc0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.637+0000 7fddcbfff640 1 -- 172.21.15.27:0/205496054 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547904163 0 0) 0x7fddb4003520 con 0x7fddcc0b4830 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.637+0000 7fddd1593640 1 -- 172.21.15.27:0/205496054 >> v1:172.21.15.27:6789/0 conn(0x7fddcc0b4830 legacy=0x7fddcc160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.637+0000 7fddd1593640 1 -- 172.21.15.27:0/205496054 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc070bf0 con 0x7fddcc142170 2024-03-06T07:24:42.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.637+0000 7fddd1593640 1 --2- 172.21.15.27:0/205496054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc142a20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009dc0 tx=0x7fddc402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4685a18ac44be078 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddcbfff640 1 -- 172.21.15.27:0/205496054 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4039070 con 0x7fddcc142170 2024-03-06T07:24:42.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddcbfff640 1 -- 172.21.15.27:0/205496054 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc402c910 con 0x7fddcc142170 2024-03-06T07:24:42.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddcbfff640 1 -- 172.21.15.27:0/205496054 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc0001670 con 0x7fddcc142170 2024-03-06T07:24:42.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddd37f7640 1 -- 172.21.15.27:0/205496054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 msgr2=0x7fddcc142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddd37f7640 1 --2- 172.21.15.27:0/205496054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc142a20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009dc0 tx=0x7fddc402bde0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddd37f7640 1 -- 172.21.15.27:0/205496054 shutdown_connections 2024-03-06T07:24:42.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddd37f7640 1 --2- 172.21.15.27:0/205496054 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc142a20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddd37f7640 1 -- 172.21.15.27:0/205496054 >> 172.21.15.27:0/205496054 conn(0x7fddcc066d00 msgr2=0x7fddcc067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddd37f7640 1 -- 172.21.15.27:0/205496054 shutdown_connections 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.638+0000 7fddd37f7640 1 -- 172.21.15.27:0/205496054 wait complete. 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.639+0000 7fddd37f7640 1 Processor -- start 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.639+0000 7fddd37f7640 1 -- start start 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd37f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddcc0b29a0 0x7fddcc139930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd37f7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc0b4830 0x7fddcc15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd37f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc13c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd37f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc0b4df0 con 0x7fddcc0b29a0 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd37f7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc0711b0 con 0x7fddcc0b4830 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd37f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc142550 con 0x7fddcc142170 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd1593640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddcc0b29a0 0x7fddcc139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd1593640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddcc0b29a0 0x7fddcc139930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45596/0 (socket says 172.21.15.27:45596) 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd1593640 1 -- 172.21.15.27:0/675279103 learned_addr learned my addr 172.21.15.27:0/675279103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd0d92640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc0b4830 0x7fddcc15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.640+0000 7fddd1d94640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc13c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddd1593640 1 -- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 msgr2=0x7fddcc13c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddd1593640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc13c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddd1593640 1 -- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc0b4830 msgr2=0x7fddcc15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddd1593640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc0b4830 0x7fddcc15b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddd1593640 1 -- 172.21.15.27:0/675279103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc1f55a0 con 0x7fddcc0b29a0 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddd1d94640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc13c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddd1593640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddcc0b29a0 0x7fddcc139930 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009ee0 tx=0x7fddc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddc9ffb640 1 -- 172.21.15.27:0/675279103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4039050 con 0x7fddcc0b29a0 2024-03-06T07:24:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddc9ffb640 1 -- 172.21.15.27:0/675279103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc403a040 con 0x7fddcc0b29a0 2024-03-06T07:24:42.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc1f5880 con 0x7fddcc0b29a0 2024-03-06T07:24:42.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddc9ffb640 1 -- 172.21.15.27:0/675279103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4009450 con 0x7fddcc0b29a0 2024-03-06T07:24:42.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.641+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddcc1f5e40 con 0x7fddcc0b29a0 2024-03-06T07:24:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.642+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddcc0733c0 con 0x7fddcc0b29a0 2024-03-06T07:24:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.643+0000 7fddc9ffb640 1 -- 172.21.15.27:0/675279103 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddc4003bd0 con 0x7fddcc0b29a0 2024-03-06T07:24:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.643+0000 7fddc9ffb640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb806b4d0 0x7fddb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.643+0000 7fddc9ffb640 1 -- 172.21.15.27:0/675279103 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fddc40ac2f0 con 0x7fddcc0b29a0 2024-03-06T07:24:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.643+0000 7fddd0d92640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb806b4d0 0x7fddb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.644+0000 7fddd0d92640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb806b4d0 0x7fddb806d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fddcc137eb0 tx=0x7fddc0001150 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.647+0000 7fddc9ffb640 1 -- 172.21.15.27:0/675279103 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddc407b610 con 0x7fddcc0b29a0 2024-03-06T07:24:42.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.827+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fddcc04f410 con 0x7fddcc0b29a0 2024-03-06T07:24:42.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.827+0000 7fddc9ffb640 1 -- 172.21.15.27:0/675279103 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7fddc407af10 con 0x7fddcc0b29a0 2024-03-06T07:24:42.830 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:24:42.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb806b4d0 msgr2=0x7fddb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb806b4d0 0x7fddb806d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fddcc137eb0 tx=0x7fddc0001150 comp rx=0 tx=0).stop 2024-03-06T07:24:42.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddcc0b29a0 msgr2=0x7fddcc139930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddcc0b29a0 0x7fddcc139930 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009ee0 tx=0x7fddc4004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:42.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 shutdown_connections 2024-03-06T07:24:42.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddb806b4d0 0x7fddb806d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fddcc142170 0x7fddcc13c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc0b4830 0x7fddcc15b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 --2- 172.21.15.27:0/675279103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddcc0b29a0 0x7fddcc139930 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 >> 172.21.15.27:0/675279103 conn(0x7fddcc066d00 msgr2=0x7fddcc0ae610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 shutdown_connections 2024-03-06T07:24:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.829+0000 7fddd37f7640 1 -- 172.21.15.27:0/675279103 wait complete. 2024-03-06T07:24:42.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:42.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.947+0000 7f7f0ccb5640 1 Processor -- start 2024-03-06T07:24:42.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.947+0000 7f7f0ccb5640 1 -- start start 2024-03-06T07:24:42.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f0ccb5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f080aee10 0x7f7f0815cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f0ccb5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f0806d400 0x7f7f08070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f0ccb5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f080729b0 con 0x7f7f0815bac0 2024-03-06T07:24:42.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f0ccb5640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0806cf70 con 0x7f7f0806d400 2024-03-06T07:24:42.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f0ccb5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0806c920 con 0x7f7f080aee10 2024-03-06T07:24:42.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f06d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f0806d400 0x7f7f08070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f06d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f0806d400 0x7f7f08070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45612/0 (socket says 172.21.15.27:45612) 2024-03-06T07:24:42.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f06d9d640 1 -- 172.21.15.27:0/1410823995 learned_addr learned my addr 172.21.15.27:0/1410823995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:42.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f05d9b640 1 --2- 172.21.15.27:0/1410823995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f080aee10 0x7f7f0815cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f06d9d640 1 -- 172.21.15.27:0/1410823995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f080aee10 msgr2=0x7f7f0815cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.948+0000 7f7f06d9d640 1 --2- 172.21.15.27:0/1410823995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f080aee10 0x7f7f0815cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.949+0000 7f7f06d9d640 1 -- 172.21.15.27:0/1410823995 >> v1:172.21.15.69:6789/0 conn(0x7f7f0815bac0 legacy=0x7f7f080ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.949+0000 7f7f06d9d640 1 -- 172.21.15.27:0/1410823995 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0806cc00 con 0x7f7f0806d400 2024-03-06T07:24:42.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.949+0000 7f7f06d9d640 1 --2- 172.21.15.27:0/1410823995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 0x7f7f08070870 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009e50 tx=0x7f7ef802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85cebb8977efb41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.949+0000 7f7f0559a640 1 -- 172.21.15.27:0/1410823995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef802ca70 con 0x7f7f0806d400 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.949+0000 7f7f0559a640 1 -- 172.21.15.27:0/1410823995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef802cc10 con 0x7f7f0806d400 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.949+0000 7f7f0559a640 1 -- 172.21.15.27:0/1410823995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80343f0 con 0x7f7f0806d400 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.950+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/1410823995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 msgr2=0x7f7f08070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.950+0000 7f7f0ccb5640 1 --2- 172.21.15.27:0/1410823995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 0x7f7f08070870 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009e50 tx=0x7f7ef802bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.950+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/1410823995 shutdown_connections 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.950+0000 7f7f0ccb5640 1 --2- 172.21.15.27:0/1410823995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 0x7f7f08070870 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.950+0000 7f7f0ccb5640 1 --2- 172.21.15.27:0/1410823995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f080aee10 0x7f7f0815cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.950+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/1410823995 >> 172.21.15.27:0/1410823995 conn(0x7f7f08066d00 msgr2=0x7f7f080670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.950+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/1410823995 shutdown_connections 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.950+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/1410823995 wait complete. 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.951+0000 7f7f0ccb5640 1 Processor -- start 2024-03-06T07:24:42.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.951+0000 7f7f0ccb5640 1 -- start start 2024-03-06T07:24:42.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f0ccb5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 0x7f7f08152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f0ccb5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f080aee10 0x7f7f081482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f0ccb5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0815bac0 0x7f7f080ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f0ccb5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0815c020 con 0x7f7f0806d400 2024-03-06T07:24:42.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f0ccb5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0815bea0 con 0x7f7f080aee10 2024-03-06T07:24:42.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f0ccb5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f08072580 con 0x7f7f0815bac0 2024-03-06T07:24:42.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f0659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 0x7f7f08152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f0659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 0x7f7f08152f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45622/0 (socket says 172.21.15.27:45622) 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f0659c640 1 -- 172.21.15.27:0/2175877027 learned_addr learned my addr 172.21.15.27:0/2175877027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f06d9d640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0815bac0 0x7f7f080ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.952+0000 7f7f05d9b640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f080aee10 0x7f7f081482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7f0659c640 1 -- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0815bac0 msgr2=0x7f7f080ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7f0659c640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0815bac0 0x7f7f080ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7f0659c640 1 -- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f080aee10 msgr2=0x7f7f081482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7f0659c640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f080aee10 0x7f7f081482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7f0659c640 1 -- 172.21.15.27:0/2175877027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f08153490 con 0x7f7f0806d400 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7f05d9b640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f080aee10 0x7f7f081482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7f0659c640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 0x7f7f08152f50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000b3f0 tx=0x7f7f0000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7efeffd640 1 -- 172.21.15.27:0/2175877027 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00015020 con 0x7f7f0806d400 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7efeffd640 1 -- 172.21.15.27:0/2175877027 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f00011830 con 0x7f7f0806d400 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f080b2dc0 con 0x7f7f0806d400 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7efeffd640 1 -- 172.21.15.27:0/2175877027 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f00011c20 con 0x7f7f0806d400 2024-03-06T07:24:42.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.953+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f080b3600 con 0x7f7f0806d400 2024-03-06T07:24:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.954+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f08067ff0 con 0x7f7f0806d400 2024-03-06T07:24:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.954+0000 7f7efeffd640 1 -- 172.21.15.27:0/2175877027 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f00018020 con 0x7f7f0806d400 2024-03-06T07:24:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.955+0000 7f7efeffd640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ee806b500 0x7f7ee806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.955+0000 7f7efeffd640 1 -- 172.21.15.27:0/2175877027 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7f00089a20 con 0x7f7f0806d400 2024-03-06T07:24:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.955+0000 7f7f05d9b640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ee806b500 0x7f7ee806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.956+0000 7f7f05d9b640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ee806b500 0x7f7ee806d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7f0815d530 tx=0x7f7ef0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:42.959+0000 7f7efeffd640 1 -- 172.21.15.27:0/2175877027 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f0001f030 con 0x7f7f0806d400 2024-03-06T07:24:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.143+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7f0806de50 con 0x7f7f0806d400 2024-03-06T07:24:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.144+0000 7f7efeffd640 1 -- 172.21.15.27:0/2175877027 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v42) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7f080b2dc0 con 0x7f7f0806d400 2024-03-06T07:24:43.146 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:24:43.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:24:43.147 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:24:43.147 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:24:43.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:24:43.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ee806b500 msgr2=0x7f7ee806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ee806b500 0x7f7ee806d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7f0815d530 tx=0x7f7ef0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:43.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 msgr2=0x7f7f08152f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 0x7f7f08152f50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000b3f0 tx=0x7f7f0000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 shutdown_connections 2024-03-06T07:24:43.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ee806b500 0x7f7ee806d9c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0815bac0 0x7f7f080ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f080aee10 0x7f7f081482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 --2- 172.21.15.27:0/2175877027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0806d400 0x7f7f08152f50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.145+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 >> 172.21.15.27:0/2175877027 conn(0x7f7f08066d00 msgr2=0x7f7f080b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.146+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 shutdown_connections 2024-03-06T07:24:43.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.146+0000 7f7f0ccb5640 1 -- 172.21.15.27:0/2175877027 wait complete. 2024-03-06T07:24:43.159 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:43.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.268+0000 7fda5d066640 1 Processor -- start 2024-03-06T07:24:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.268+0000 7fda5d066640 1 -- start start 2024-03-06T07:24:43.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.269+0000 7fda5d066640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda580b2d30 0x7fda580b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.269+0000 7fda5d066640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fda580b7820 0x7fda580b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.269+0000 7fda5d066640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda580d4960 con 0x7fda58071540 2024-03-06T07:24:43.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.269+0000 7fda5d066640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580d3230 con 0x7fda580b2d30 2024-03-06T07:24:43.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.269+0000 7fda5d066640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580d36c0 con 0x7fda580b7820 2024-03-06T07:24:43.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.269+0000 7fda5659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda580b2d30 0x7fda580b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.269+0000 7fda56d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fda58071540 0x7fda58071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55396/0 (socket says 172.21.15.27:55396) 2024-03-06T07:24:43.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.269+0000 7fda5659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda580b2d30 0x7fda580b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45628/0 (socket says 172.21.15.27:45628) 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.269+0000 7fda56d9d640 1 -- 172.21.15.27:0/2032365710 learned_addr learned my addr 172.21.15.27:0/2032365710 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.270+0000 7fda5759e640 1 --2- 172.21.15.27:0/2032365710 >> v2:172.21.15.69:3300/0 conn(0x7fda580b7820 0x7fda580b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.270+0000 7fda5659c640 1 -- 172.21.15.27:0/2032365710 >> v2:172.21.15.69:3300/0 conn(0x7fda580b7820 msgr2=0x7fda580b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.270+0000 7fda5659c640 1 --2- 172.21.15.27:0/2032365710 >> v2:172.21.15.69:3300/0 conn(0x7fda580b7820 0x7fda580b7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.270+0000 7fda5659c640 1 -- 172.21.15.27:0/2032365710 >> v1:172.21.15.27:6789/0 conn(0x7fda58071540 legacy=0x7fda58071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.270+0000 7fda55d9b640 1 -- 172.21.15.27:0/2032365710 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006631880 0 0) 0x7fda580d4960 con 0x7fda58071540 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.270+0000 7fda5659c640 1 -- 172.21.15.27:0/2032365710 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580d2620 con 0x7fda580b2d30 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.270+0000 7fda5659c640 1 --2- 172.21.15.27:0/2032365710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda580b2d30 0x7fda580b72e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fda580ce340 tx=0x7fda4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a416858b74dbb254 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda55d9b640 1 -- 172.21.15.27:0/2032365710 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40035020 con 0x7fda580b2d30 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda55d9b640 1 -- 172.21.15.27:0/2032365710 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda40031830 con 0x7fda580b2d30 2024-03-06T07:24:43.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda55d9b640 1 -- 172.21.15.27:0/2032365710 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40031c00 con 0x7fda580b2d30 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda5d066640 1 -- 172.21.15.27:0/2032365710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda580b2d30 msgr2=0x7fda580b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda5d066640 1 --2- 172.21.15.27:0/2032365710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda580b2d30 0x7fda580b72e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fda580ce340 tx=0x7fda4002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda5d066640 1 -- 172.21.15.27:0/2032365710 shutdown_connections 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda5d066640 1 --2- 172.21.15.27:0/2032365710 >> v2:172.21.15.69:3300/0 conn(0x7fda580b7820 0x7fda580b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda5d066640 1 --2- 172.21.15.27:0/2032365710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda580b2d30 0x7fda580b72e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda5d066640 1 -- 172.21.15.27:0/2032365710 >> 172.21.15.27:0/2032365710 conn(0x7fda580d3d60 msgr2=0x7fda580d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda5d066640 1 -- 172.21.15.27:0/2032365710 shutdown_connections 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.271+0000 7fda5d066640 1 -- 172.21.15.27:0/2032365710 wait complete. 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.273+0000 7fda5d066640 1 Processor -- start 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.273+0000 7fda5d066640 1 -- start start 2024-03-06T07:24:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.273+0000 7fda5d066640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda58071540 0x7fda580c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.273+0000 7fda5d066640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda580b2d30 0x7fda581e3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.273+0000 7fda5d066640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda580b7820 0x7fda5814a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.273+0000 7fda5d066640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580d2ec0 con 0x7fda58071540 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.273+0000 7fda5d066640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580d2be0 con 0x7fda580b7820 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.273+0000 7fda5d066640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda580d22f0 con 0x7fda580b2d30 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda58071540 0x7fda580c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda58071540 0x7fda580c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45642/0 (socket says 172.21.15.27:45642) 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda56d9d640 1 -- 172.21.15.27:0/3364321425 learned_addr learned my addr 172.21.15.27:0/3364321425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda5659c640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda580b2d30 0x7fda581e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda5759e640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda580b7820 0x7fda5814a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda56d9d640 1 -- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda580b2d30 msgr2=0x7fda581e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda56d9d640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda580b2d30 0x7fda581e3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda56d9d640 1 -- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda580b7820 msgr2=0x7fda5814a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda56d9d640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda580b7820 0x7fda5814a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda56d9d640 1 -- 172.21.15.27:0/3364321425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda581482a0 con 0x7fda58071540 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda5759e640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda580b7820 0x7fda5814a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda56d9d640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda58071540 0x7fda580c7cf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c430 tx=0x7fda5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda4f7fe640 1 -- 172.21.15.27:0/3364321425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda500074e0 con 0x7fda58071540 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda4f7fe640 1 -- 172.21.15.27:0/3364321425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda500047e0 con 0x7fda58071540 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda58148580 con 0x7fda58071540 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda4f7fe640 1 -- 172.21.15.27:0/3364321425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda50004bd0 con 0x7fda58071540 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.274+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda58148b40 con 0x7fda58071540 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.275+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda58072070 con 0x7fda58071540 2024-03-06T07:24:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.276+0000 7fda4f7fe640 1 -- 172.21.15.27:0/3364321425 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda50011070 con 0x7fda58071540 2024-03-06T07:24:43.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.276+0000 7fda4f7fe640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.276+0000 7fda4f7fe640 1 -- 172.21.15.27:0/3364321425 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fda50089b70 con 0x7fda58071540 2024-03-06T07:24:43.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.276+0000 7fda5659c640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.277+0000 7fda5659c640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3806b4d0 0x7fda3806d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fda40036000 tx=0x7fda40008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.280+0000 7fda4f7fe640 1 -- 172.21.15.27:0/3364321425 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda50058d40 con 0x7fda58071540 2024-03-06T07:24:43.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.461+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fda5804f410 con 0x7fda58071540 2024-03-06T07:24:43.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.813+0000 7fda4f7fe640 1 -- 172.21.15.27:0/3364321425 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v43) v1 ==== 138+0+0 (secure 0 0 0) 0x7fda50058640 con 0x7fda58071540 2024-03-06T07:24:43.816 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:43.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.815+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3806b4d0 msgr2=0x7fda3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.815+0000 7fda5d066640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3806b4d0 0x7fda3806d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fda40036000 tx=0x7fda40008030 comp rx=0 tx=0).stop 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.815+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda58071540 msgr2=0x7fda580c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.815+0000 7fda5d066640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda58071540 0x7fda580c7cf0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c430 tx=0x7fda5000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.816+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 shutdown_connections 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.816+0000 7fda5d066640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3806b4d0 0x7fda3806d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.816+0000 7fda5d066640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda580b7820 0x7fda5814a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.816+0000 7fda5d066640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda580b2d30 0x7fda581e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.816+0000 7fda5d066640 1 --2- 172.21.15.27:0/3364321425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda58071540 0x7fda580c7cf0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.816+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 >> 172.21.15.27:0/3364321425 conn(0x7fda580d3d60 msgr2=0x7fda5815c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.816+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 shutdown_connections 2024-03-06T07:24:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.816+0000 7fda5d066640 1 -- 172.21.15.27:0/3364321425 wait complete. 2024-03-06T07:24:43.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.945+0000 7f3f2804e640 1 Processor -- start 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.946+0000 7f3f2804e640 1 -- start start 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.946+0000 7f3f2804e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f200d07f0 0x7f3f200cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.946+0000 7f3f2804e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2006dba0 con 0x7f3f2006d3a0 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.946+0000 7f3f2804e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2006de80 con 0x7f3f2015e170 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.946+0000 7f3f2804e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f200cef00 con 0x7f3f200d07f0 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.947+0000 7f3f255e9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f200d07f0 0x7f3f200cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.947+0000 7f3f265eb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3f2006d3a0 0x7f3f200b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56152/0 (socket says 172.21.15.27:56152) 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.947+0000 7f3f255e9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f200d07f0 0x7f3f200cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46062/0 (socket says 172.21.15.27:46062) 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.947+0000 7f3f265eb640 1 -- 172.21.15.27:0/4028505342 learned_addr learned my addr 172.21.15.27:0/4028505342 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.947+0000 7f3f255e9640 1 -- 172.21.15.27:0/4028505342 >> v1:172.21.15.69:6789/0 conn(0x7f3f2015e170 legacy=0x7f3f200d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.947+0000 7f3f255e9640 1 -- 172.21.15.27:0/4028505342 >> v1:172.21.15.27:6789/0 conn(0x7f3f2006d3a0 legacy=0x7f3f200b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.947+0000 7f3f255e9640 1 -- 172.21.15.27:0/4028505342 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f200ceb90 con 0x7f3f200d07f0 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.948+0000 7f3f255e9640 1 --2- 172.21.15.27:0/4028505342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 0x7f3f200cfa10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3f100097a0 tx=0x7f3f1002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd506e2b81c40f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.948+0000 7f3f24de8640 1 -- 172.21.15.27:0/4028505342 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100315a0 con 0x7f3f200d07f0 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.948+0000 7f3f24de8640 1 -- 172.21.15.27:0/4028505342 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f10037070 con 0x7f3f200d07f0 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.948+0000 7f3f24de8640 1 -- 172.21.15.27:0/4028505342 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10031960 con 0x7f3f200d07f0 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.948+0000 7f3f2804e640 1 -- 172.21.15.27:0/4028505342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 msgr2=0x7f3f200cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.948+0000 7f3f2804e640 1 --2- 172.21.15.27:0/4028505342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 0x7f3f200cfa10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3f100097a0 tx=0x7f3f1002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:43.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.948+0000 7f3f2804e640 1 -- 172.21.15.27:0/4028505342 shutdown_connections 2024-03-06T07:24:43.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.948+0000 7f3f2804e640 1 --2- 172.21.15.27:0/4028505342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 0x7f3f200cfa10 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.948+0000 7f3f2804e640 1 -- 172.21.15.27:0/4028505342 >> 172.21.15.27:0/4028505342 conn(0x7f3f2006af20 msgr2=0x7f3f2006b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.949+0000 7f3f2804e640 1 -- 172.21.15.27:0/4028505342 shutdown_connections 2024-03-06T07:24:43.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.949+0000 7f3f2804e640 1 -- 172.21.15.27:0/4028505342 wait complete. 2024-03-06T07:24:43.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.950+0000 7f3f2804e640 1 Processor -- start 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.950+0000 7f3f2804e640 1 -- start start 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.950+0000 7f3f2804e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f2006d3a0 0x7f3f2009d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.950+0000 7f3f2804e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 0x7f3f2009ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.950+0000 7f3f2804e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f2015e170 0x7f3f200ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.950+0000 7f3f2804e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f200b3c90 con 0x7f3f200d07f0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.950+0000 7f3f2804e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2006e160 con 0x7f3f2006d3a0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.950+0000 7f3f2804e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f200b3e10 con 0x7f3f2015e170 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.950+0000 7f3f255e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 0x7f3f2009ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f255e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 0x7f3f2009ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46066/0 (socket says 172.21.15.27:46066) 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f255e9640 1 -- 172.21.15.27:0/930354703 learned_addr learned my addr 172.21.15.27:0/930354703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f255e9640 1 -- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f2015e170 msgr2=0x7f3f200ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f255e9640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f2015e170 0x7f3f200ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f25dea640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f2006d3a0 0x7f3f2009d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f255e9640 1 -- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f2006d3a0 msgr2=0x7f3f2009d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f255e9640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f2006d3a0 0x7f3f2009d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f255e9640 1 -- 172.21.15.27:0/930354703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2009bfd0 con 0x7f3f200d07f0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f255e9640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 0x7f3f2009ba90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3f10000c00 tx=0x7f3f100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f167fc640 1 -- 172.21.15.27:0/930354703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10032840 con 0x7f3f200d07f0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f167fc640 1 -- 172.21.15.27:0/930354703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f100329e0 con 0x7f3f200d07f0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2014e670 con 0x7f3f200d07f0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f167fc640 1 -- 172.21.15.27:0/930354703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10032dd0 con 0x7f3f200d07f0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.951+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2009c530 con 0x7f3f200d07f0 2024-03-06T07:24:43.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.952+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f200d12a0 con 0x7f3f200d07f0 2024-03-06T07:24:43.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.953+0000 7f3f167fc640 1 -- 172.21.15.27:0/930354703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f1004d3f0 con 0x7f3f200d07f0 2024-03-06T07:24:43.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.953+0000 7f3f167fc640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f0406b4d0 0x7f3f0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.953+0000 7f3f167fc640 1 -- 172.21.15.27:0/930354703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f3f1002f030 con 0x7f3f200d07f0 2024-03-06T07:24:43.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.954+0000 7f3f25dea640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f0406b4d0 0x7f3f0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.954+0000 7f3f25dea640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f0406b4d0 0x7f3f0406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00a750 tx=0x7f3f1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:43.958+0000 7f3f167fc640 1 -- 172.21.15.27:0/930354703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f200d12a0 con 0x7f3f200d07f0 2024-03-06T07:24:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.140+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3f2009e320 con 0x7f3f200d07f0 2024-03-06T07:24:44.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.821+0000 7f3f167fc640 1 -- 172.21.15.27:0/930354703 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v44) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3f2009bfd0 con 0x7f3f200d07f0 2024-03-06T07:24:44.824 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f0406b4d0 msgr2=0x7f3f0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f0406b4d0 0x7f3f0406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00a750 tx=0x7f3f1c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 msgr2=0x7f3f2009ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 0x7f3f2009ba90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3f10000c00 tx=0x7f3f100046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 shutdown_connections 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f0406b4d0 0x7f3f0406d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f2015e170 0x7f3f200ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f200d07f0 0x7f3f2009ba90 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 --2- 172.21.15.27:0/930354703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f2006d3a0 0x7f3f2009d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 >> 172.21.15.27:0/930354703 conn(0x7f3f2006af20 msgr2=0x7f3f200b0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 shutdown_connections 2024-03-06T07:24:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.825+0000 7f3f2804e640 1 -- 172.21.15.27:0/930354703 wait complete. 2024-03-06T07:24:44.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:44.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.967+0000 7f5f39e65640 1 Processor -- start 2024-03-06T07:24:44.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.967+0000 7f5f39e65640 1 -- start start 2024-03-06T07:24:44.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.967+0000 7f5f39e65640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f340ae990 0x7f5f340b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.967+0000 7f5f39e65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3415ff20 0x7f5f341603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.967+0000 7f5f39e65640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3415ca70 con 0x7f5f340aed70 2024-03-06T07:24:44.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.967+0000 7f5f39e65640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3415cd50 con 0x7f5f340ae990 2024-03-06T07:24:44.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.967+0000 7f5f39e65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3415d030 con 0x7f5f3415ff20 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f337fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f340ae990 0x7f5f340b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f32ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5f340aed70 0x7f5f3415c290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56156/0 (socket says 172.21.15.27:56156) 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f337fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f340ae990 0x7f5f340b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46068/0 (socket says 172.21.15.27:46068) 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f32ffd640 1 -- 172.21.15.27:0/826630630 learned_addr learned my addr 172.21.15.27:0/826630630 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f33fff640 1 --2- 172.21.15.27:0/826630630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3415ff20 0x7f5f341603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f337fe640 1 -- 172.21.15.27:0/826630630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3415ff20 msgr2=0x7f5f341603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f337fe640 1 --2- 172.21.15.27:0/826630630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3415ff20 0x7f5f341603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f337fe640 1 -- 172.21.15.27:0/826630630 >> v1:172.21.15.27:6789/0 conn(0x7f5f340aed70 legacy=0x7f5f3415c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f327fc640 1 -- 172.21.15.27:0/826630630 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873459231 0 0) 0x7f5f3415ca70 con 0x7f5f340aed70 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.968+0000 7f5f337fe640 1 -- 172.21.15.27:0/826630630 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3415fa00 con 0x7f5f340ae990 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.969+0000 7f5f337fe640 1 --2- 172.21.15.27:0/826630630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f340ae990 0x7f5f340b4a90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009d90 tx=0x7f5f2402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523acf1c5cfcceaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.969+0000 7f5f327fc640 1 -- 172.21.15.27:0/826630630 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24039070 con 0x7f5f340ae990 2024-03-06T07:24:44.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.969+0000 7f5f327fc640 1 -- 172.21.15.27:0/826630630 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2402cbc0 con 0x7f5f340ae990 2024-03-06T07:24:44.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.969+0000 7f5f327fc640 1 -- 172.21.15.27:0/826630630 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24034440 con 0x7f5f340ae990 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.969+0000 7f5f39e65640 1 -- 172.21.15.27:0/826630630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f340ae990 msgr2=0x7f5f340b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.969+0000 7f5f39e65640 1 --2- 172.21.15.27:0/826630630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f340ae990 0x7f5f340b4a90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009d90 tx=0x7f5f2402be10 comp rx=0 tx=0).stop 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.970+0000 7f5f39e65640 1 -- 172.21.15.27:0/826630630 shutdown_connections 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.970+0000 7f5f39e65640 1 --2- 172.21.15.27:0/826630630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f3415ff20 0x7f5f341603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.970+0000 7f5f39e65640 1 --2- 172.21.15.27:0/826630630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f340ae990 0x7f5f340b4a90 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.970+0000 7f5f39e65640 1 -- 172.21.15.27:0/826630630 >> 172.21.15.27:0/826630630 conn(0x7f5f34066d00 msgr2=0x7f5f34067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.970+0000 7f5f39e65640 1 -- 172.21.15.27:0/826630630 shutdown_connections 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.970+0000 7f5f39e65640 1 -- 172.21.15.27:0/826630630 wait complete. 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.971+0000 7f5f39e65640 1 Processor -- start 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.971+0000 7f5f39e65640 1 -- start start 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f39e65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340ae990 0x7f5f340b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f39e65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f340aed70 0x7f5f340b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f39e65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f3415ff20 0x7f5f341f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f39e65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340cc6b0 con 0x7f5f3415ff20 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f39e65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340cc530 con 0x7f5f340ae990 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f39e65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340cc830 con 0x7f5f340aed70 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f33fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f3415ff20 0x7f5f341f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f33fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f3415ff20 0x7f5f341f5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46076/0 (socket says 172.21.15.27:46076) 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f33fff640 1 -- 172.21.15.27:0/4260181008 learned_addr learned my addr 172.21.15.27:0/4260181008 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f337fe640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340ae990 0x7f5f340b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f32ffd640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f340aed70 0x7f5f340b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f33fff640 1 -- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f340aed70 msgr2=0x7f5f340b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f33fff640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f340aed70 0x7f5f340b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f33fff640 1 -- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340ae990 msgr2=0x7f5f340b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f33fff640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340ae990 0x7f5f340b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f33fff640 1 -- 172.21.15.27:0/4260181008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3415a7e0 con 0x7f5f3415ff20 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f337fe640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340ae990 0x7f5f340b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:44.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f32ffd640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f340aed70 0x7f5f340b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:44.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f33fff640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f3415ff20 0x7f5f341f5450 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00e7f0 tx=0x7f5f2c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f1bfff640 1 -- 172.21.15.27:0/4260181008 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c017070 con 0x7f5f3415ff20 2024-03-06T07:24:44.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f1bfff640 1 -- 172.21.15.27:0/4260181008 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c00ee40 con 0x7f5f3415ff20 2024-03-06T07:24:44.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3415aac0 con 0x7f5f3415ff20 2024-03-06T07:24:44.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.972+0000 7f5f1bfff640 1 -- 172.21.15.27:0/4260181008 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c00cb80 con 0x7f5f3415ff20 2024-03-06T07:24:44.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.973+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3415b080 con 0x7f5f3415ff20 2024-03-06T07:24:44.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.973+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f34074980 con 0x7f5f3415ff20 2024-03-06T07:24:44.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.974+0000 7f5f1bfff640 1 -- 172.21.15.27:0/4260181008 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f2c01a070 con 0x7f5f3415ff20 2024-03-06T07:24:44.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.974+0000 7f5f1bfff640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f2006b4d0 0x7f5f2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.975+0000 7f5f1bfff640 1 -- 172.21.15.27:0/4260181008 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f5f2c08a000 con 0x7f5f3415ff20 2024-03-06T07:24:44.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.975+0000 7f5f337fe640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f2006b4d0 0x7f5f2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.975+0000 7f5f337fe640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f2006b4d0 0x7f5f2006d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5f24036000 tx=0x7f5f24030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:44.979+0000 7f5f1bfff640 1 -- 172.21.15.27:0/4260181008 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f34074980 con 0x7f5f3415ff20 2024-03-06T07:24:45.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.199+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5f340733c0 con 0x7f5f3415ff20 2024-03-06T07:24:45.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.826+0000 7f5f1bfff640 1 -- 172.21.15.27:0/4260181008 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) v1 ==== 172+0+0 (secure 0 0 0) 0x7f5f2c058ee0 con 0x7f5f3415ff20 2024-03-06T07:24:45.829 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:24:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.831+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f2006b4d0 msgr2=0x7f5f2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.831+0000 7f5f39e65640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f2006b4d0 0x7f5f2006d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5f24036000 tx=0x7f5f24030000 comp rx=0 tx=0).stop 2024-03-06T07:24:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.832+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f3415ff20 msgr2=0x7f5f341f5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.832+0000 7f5f39e65640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f3415ff20 0x7f5f341f5450 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00e7f0 tx=0x7f5f2c00ebf0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.835+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 shutdown_connections 2024-03-06T07:24:45.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.835+0000 7f5f39e65640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f2006b4d0 0x7f5f2006d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.835+0000 7f5f39e65640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f3415ff20 0x7f5f341f5450 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.835+0000 7f5f39e65640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f340aed70 0x7f5f340b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.836+0000 7f5f39e65640 1 --2- 172.21.15.27:0/4260181008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f340ae990 0x7f5f340b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.836+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 >> 172.21.15.27:0/4260181008 conn(0x7f5f34066d00 msgr2=0x7f5f340708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.839+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 shutdown_connections 2024-03-06T07:24:45.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.839+0000 7f5f39e65640 1 -- 172.21.15.27:0/4260181008 wait complete. 2024-03-06T07:24:45.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:45.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.970+0000 7f7495a4e640 1 Processor -- start 2024-03-06T07:24:45.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.971+0000 7f7495a4e640 1 -- start start 2024-03-06T07:24:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.971+0000 7f7495a4e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f74900cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.971+0000 7f7495a4e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74900b4260 0x7f74900b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.971+0000 7f7495a4e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900b0dd0 con 0x7f74900d06c0 2024-03-06T07:24:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.971+0000 7f7495a4e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74900b0b80 con 0x7f74900b4260 2024-03-06T07:24:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.971+0000 7f7495a4e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74900aeec0 con 0x7f74900d02e0 2024-03-06T07:24:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.971+0000 7f748e7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74900b4260 0x7f74900b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.971+0000 7f748f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f74900d06c0 0x7f74900b1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56168/0 (socket says 172.21.15.27:56168) 2024-03-06T07:24:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.971+0000 7f748f7fe640 1 -- 172.21.15.27:0/3942110860 learned_addr learned my addr 172.21.15.27:0/3942110860 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.972+0000 7f748effd640 1 --2- 172.21.15.27:0/3942110860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f74900cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.972+0000 7f748e7fc640 1 -- 172.21.15.27:0/3942110860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 msgr2=0x7f74900cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.972+0000 7f748e7fc640 1 --2- 172.21.15.27:0/3942110860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f74900cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.972+0000 7f748e7fc640 1 -- 172.21.15.27:0/3942110860 >> v1:172.21.15.27:6789/0 conn(0x7f74900d06c0 legacy=0x7f74900b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.972+0000 7f748e7fc640 1 -- 172.21.15.27:0/3942110860 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900b10b0 con 0x7f74900b4260 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.972+0000 7f748dffb640 1 -- 172.21.15.27:0/3942110860 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820987446 0 0) 0x7f74900b0dd0 con 0x7f74900d06c0 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.972+0000 7f748effd640 1 --2- 172.21.15.27:0/3942110860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f74900cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f748e7fc640 1 --2- 172.21.15.27:0/3942110860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 0x7f74900b1500 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f74780097a0 tx=0x7f747802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3e9b3eff8d7d27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f748dffb640 1 -- 172.21.15.27:0/3942110860 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478034070 con 0x7f74900b4260 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f748dffb640 1 -- 172.21.15.27:0/3942110860 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7478030800 con 0x7f74900b4260 2024-03-06T07:24:45.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f748dffb640 1 -- 172.21.15.27:0/3942110860 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478030bd0 con 0x7f74900b4260 2024-03-06T07:24:45.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f7495a4e640 1 -- 172.21.15.27:0/3942110860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 msgr2=0x7f74900b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f7495a4e640 1 --2- 172.21.15.27:0/3942110860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 0x7f74900b1500 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f74780097a0 tx=0x7f747802b940 comp rx=0 tx=0).stop 2024-03-06T07:24:45.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f7495a4e640 1 -- 172.21.15.27:0/3942110860 shutdown_connections 2024-03-06T07:24:45.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f7495a4e640 1 --2- 172.21.15.27:0/3942110860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 0x7f74900b1500 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f7495a4e640 1 --2- 172.21.15.27:0/3942110860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f74900cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f7495a4e640 1 -- 172.21.15.27:0/3942110860 >> 172.21.15.27:0/3942110860 conn(0x7f74900b36d0 msgr2=0x7f74900b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f7495a4e640 1 -- 172.21.15.27:0/3942110860 shutdown_connections 2024-03-06T07:24:45.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.973+0000 7f7495a4e640 1 -- 172.21.15.27:0/3942110860 wait complete. 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.974+0000 7f7495a4e640 1 Processor -- start 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f7495a4e640 1 -- start start 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f7495a4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 0x7f74901f7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f7495a4e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f749006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f7495a4e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74900d06c0 0x7f74901f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f7495a4e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74900b22d0 con 0x7f74900b4260 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f7495a4e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74900b2150 con 0x7f74900d06c0 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f7495a4e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74900af350 con 0x7f74900d02e0 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f748effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 0x7f74901f7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f748effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 0x7f74901f7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46098/0 (socket says 172.21.15.27:46098) 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.975+0000 7f748effd640 1 -- 172.21.15.27:0/3026135968 learned_addr learned my addr 172.21.15.27:0/3026135968 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f748e7fc640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f749006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f748f7fe640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74900d06c0 0x7f74901f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f748effd640 1 -- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 msgr2=0x7f749006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f748effd640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f749006c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f748effd640 1 -- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74900d06c0 msgr2=0x7f74901f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f748effd640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74900d06c0 0x7f74901f34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f748effd640 1 -- 172.21.15.27:0/3026135968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749006ce60 con 0x7f74900b4260 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f748e7fc640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f749006c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f748effd640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 0x7f74901f7d50 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f748800c7f0 tx=0x7f748800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f7487fff640 1 -- 172.21.15.27:0/3026135968 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488015070 con 0x7f74900b4260 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f7487fff640 1 -- 172.21.15.27:0/3026135968 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7488007a30 con 0x7f74900b4260 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74900ca860 con 0x7f74900b4260 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f7487fff640 1 -- 172.21.15.27:0/3026135968 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488007e00 con 0x7f74900b4260 2024-03-06T07:24:45.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.976+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74900cae20 con 0x7f74900b4260 2024-03-06T07:24:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.977+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7490068700 con 0x7f74900b4260 2024-03-06T07:24:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.978+0000 7f7487fff640 1 -- 172.21.15.27:0/3026135968 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7488018070 con 0x7f74900b4260 2024-03-06T07:24:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.978+0000 7f7487fff640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.978+0000 7f7487fff640 1 -- 172.21.15.27:0/3026135968 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f748808a0e0 con 0x7f74900b4260 2024-03-06T07:24:45.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.978+0000 7f748e7fc640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.979+0000 7f748e7fc640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7478036000 tx=0x7f747803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:45.982+0000 7f7487fff640 1 -- 172.21.15.27:0/3026135968 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7488058ff0 con 0x7f74900b4260 2024-03-06T07:24:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.165+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f749015bf20 con 0x7f74900b4260 2024-03-06T07:24:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.166+0000 7f7487fff640 1 -- 172.21.15.27:0/3026135968 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16554 (secure 0 0 0) 0x7f74880588f0 con 0x7f74900b4260 2024-03-06T07:24:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:46.169 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":45,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:24:45.201646+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:43.493554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:44.172410+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 msgr2=0x7f747006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7478036000 tx=0x7f747803b040 comp rx=0 tx=0).stop 2024-03-06T07:24:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 msgr2=0x7f74901f7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 0x7f74901f7d50 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f748800c7f0 tx=0x7f748800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 shutdown_connections 2024-03-06T07:24:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f747006b4d0 0x7f747006d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74900d06c0 0x7f74901f34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74900d02e0 0x7f749006c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 --2- 172.21.15.27:0/3026135968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74900b4260 0x7f74901f7d50 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 >> 172.21.15.27:0/3026135968 conn(0x7f74900b36d0 msgr2=0x7f74900cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 shutdown_connections 2024-03-06T07:24:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.168+0000 7f7495a4e640 1 -- 172.21.15.27:0/3026135968 wait complete. 2024-03-06T07:24:46.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:46.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.286+0000 7f5d9cf13640 1 Processor -- start 2024-03-06T07:24:46.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.287+0000 7f5d9cf13640 1 -- start start 2024-03-06T07:24:46.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.287+0000 7f5d9cf13640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d980b21b0 0x7f5d9806d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.287+0000 7f5d9cf13640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d9815e410 con 0x7f5d980b1d50 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.287+0000 7f5d9cf13640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d980b2660 con 0x7f5d9815e030 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.287+0000 7f5d9cf13640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98162700 con 0x7f5d980b21b0 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.287+0000 7f5d8ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5d980b1d50 0x7f5d9806d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56178/0 (socket says 172.21.15.27:56178) 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.287+0000 7f5d8ffff640 1 -- 172.21.15.27:0/1397101108 learned_addr learned my addr 172.21.15.27:0/1397101108 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.288+0000 7f5d96d9d640 1 --2- 172.21.15.27:0/1397101108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d980b21b0 0x7f5d9806d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.288+0000 7f5d95d9b640 1 -- 172.21.15.27:0/1397101108 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550411441 0 0) 0x7f5d80001110 con 0x7f5d980b1d50 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.288+0000 7f5d95d9b640 1 -- 172.21.15.27:0/1397101108 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d84003520 con 0x7f5d980b1d50 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.288+0000 7f5d95d9b640 1 -- 172.21.15.27:0/1397101108 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963771074 0 0) 0x7f5d980b2660 con 0x7f5d9815e030 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.288+0000 7f5d95d9b640 1 -- 172.21.15.27:0/1397101108 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d80001110 con 0x7f5d9815e030 2024-03-06T07:24:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.288+0000 7f5d96d9d640 1 -- 172.21.15.27:0/1397101108 >> v1:172.21.15.69:6789/0 conn(0x7f5d9815e030 legacy=0x7f5d9806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.288+0000 7f5d96d9d640 1 -- 172.21.15.27:0/1397101108 >> v1:172.21.15.27:6789/0 conn(0x7f5d980b1d50 legacy=0x7f5d9806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.288+0000 7f5d96d9d640 1 -- 172.21.15.27:0/1397101108 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9806f140 con 0x7f5d980b21b0 2024-03-06T07:24:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d96d9d640 1 --2- 172.21.15.27:0/1397101108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d980b21b0 0x7f5d9806d920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009dc0 tx=0x7f5d9002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44d644668af18656 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d95d9b640 1 -- 172.21.15.27:0/1397101108 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d90039070 con 0x7f5d980b21b0 2024-03-06T07:24:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d95d9b640 1 -- 172.21.15.27:0/1397101108 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9002ccd0 con 0x7f5d980b21b0 2024-03-06T07:24:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d95d9b640 1 -- 172.21.15.27:0/1397101108 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d90034640 con 0x7f5d980b21b0 2024-03-06T07:24:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d9cf13640 1 -- 172.21.15.27:0/1397101108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d980b21b0 msgr2=0x7f5d9806d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d9cf13640 1 --2- 172.21.15.27:0/1397101108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d980b21b0 0x7f5d9806d920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009dc0 tx=0x7f5d9002bf20 comp rx=0 tx=0).stop 2024-03-06T07:24:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d9cf13640 1 -- 172.21.15.27:0/1397101108 shutdown_connections 2024-03-06T07:24:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d9cf13640 1 --2- 172.21.15.27:0/1397101108 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d980b21b0 0x7f5d9806d920 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d9cf13640 1 -- 172.21.15.27:0/1397101108 >> 172.21.15.27:0/1397101108 conn(0x7f5d980b55e0 msgr2=0x7f5d980b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.289+0000 7f5d9cf13640 1 -- 172.21.15.27:0/1397101108 shutdown_connections 2024-03-06T07:24:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.290+0000 7f5d9cf13640 1 -- 172.21.15.27:0/1397101108 wait complete. 2024-03-06T07:24:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.291+0000 7f5d9cf13640 1 Processor -- start 2024-03-06T07:24:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.291+0000 7f5d9cf13640 1 -- start start 2024-03-06T07:24:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.291+0000 7f5d9cf13640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1d50 0x7f5d981f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.291+0000 7f5d9cf13640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d980b21b0 0x7f5d98151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.291+0000 7f5d9cf13640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d9815e030 0x7f5d98152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9cf13640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d981620b0 con 0x7f5d980b1d50 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9cf13640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98161dd0 con 0x7f5d980b21b0 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9cf13640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98162390 con 0x7f5d9815e030 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1d50 0x7f5d981f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1d50 0x7f5d981f3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46100/0 (socket says 172.21.15.27:46100) 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9659c640 1 -- 172.21.15.27:0/2729244281 learned_addr learned my addr 172.21.15.27:0/2729244281 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d8ffff640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d980b21b0 0x7f5d98151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d96d9d640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d9815e030 0x7f5d98152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9659c640 1 -- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d9815e030 msgr2=0x7f5d98152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9659c640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d9815e030 0x7f5d98152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9659c640 1 -- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d980b21b0 msgr2=0x7f5d98151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9659c640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d980b21b0 0x7f5d98151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9659c640 1 -- 172.21.15.27:0/2729244281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9815c740 con 0x7f5d980b1d50 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d96d9d640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d9815e030 0x7f5d98152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9659c640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1d50 0x7f5d981f3c30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5d8800c430 tx=0x7f5d8800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d8effd640 1 -- 172.21.15.27:0/2729244281 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d88017070 con 0x7f5d980b1d50 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d8effd640 1 -- 172.21.15.27:0/2729244281 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d880052f0 con 0x7f5d980b1d50 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d98153030 con 0x7f5d980b1d50 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d8effd640 1 -- 172.21.15.27:0/2729244281 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d88003830 con 0x7f5d980b1d50 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.292+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d981535f0 con 0x7f5d980b1d50 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.293+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d980af640 con 0x7f5d980b1d50 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.294+0000 7f5d8effd640 1 -- 172.21.15.27:0/2729244281 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d8801a070 con 0x7f5d980b1d50 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.294+0000 7f5d8effd640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5d7806b4d0 0x7f5d7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.294+0000 7f5d8effd640 1 -- 172.21.15.27:0/2729244281 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d88089540 con 0x7f5d980b1d50 2024-03-06T07:24:46.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.294+0000 7f5d8ffff640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5d7806b4d0 0x7f5d7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.295+0000 7f5d8ffff640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5d7806b4d0 0x7f5d7806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5d98158360 tx=0x7f5d80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.299+0000 7f5d8effd640 1 -- 172.21.15.27:0/2729244281 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d980af640 con 0x7f5d980b1d50 2024-03-06T07:24:46.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.483+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d981537f0 con 0x7f5d980b1d50 2024-03-06T07:24:46.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.484+0000 7f5d8effd640 1 -- 172.21.15.27:0/2729244281 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16554 (secure 0 0 0) 0x7f5d981537f0 con 0x7f5d980b1d50 2024-03-06T07:24:46.486 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:46.487 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":45,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:24:45.201646+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:43.493554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:44.172410+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:46.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5d7806b4d0 msgr2=0x7f5d7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5d7806b4d0 0x7f5d7806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5d98158360 tx=0x7f5d80009040 comp rx=0 tx=0).stop 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1d50 msgr2=0x7f5d981f3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1d50 0x7f5d981f3c30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5d8800c430 tx=0x7f5d8800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 shutdown_connections 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5d7806b4d0 0x7f5d7806d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d9815e030 0x7f5d98152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d980b21b0 0x7f5d98151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 --2- 172.21.15.27:0/2729244281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1d50 0x7f5d981f3c30 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 >> 172.21.15.27:0/2729244281 conn(0x7f5d980b55e0 msgr2=0x7f5d980ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 shutdown_connections 2024-03-06T07:24:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.486+0000 7f5d9cf13640 1 -- 172.21.15.27:0/2729244281 wait complete. 2024-03-06T07:24:46.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:46.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.607+0000 7fcb4fd56640 1 Processor -- start 2024-03-06T07:24:46.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.607+0000 7fcb4fd56640 1 -- start start 2024-03-06T07:24:46.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4fd56640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcb48142170 0x7fcb48142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4fd56640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcb480b4830 0x7fcb48160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4fd56640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb48070980 con 0x7fcb480b29a0 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4fd56640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb481458e0 con 0x7fcb48142170 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4fd56640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb481607c0 con 0x7fcb480b4830 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4daf2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcb48142170 0x7fcb48142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4daf2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcb48142170 0x7fcb48142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46110/0 (socket says 172.21.15.27:46110) 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4daf2640 1 -- 172.21.15.27:0/1856306587 learned_addr learned my addr 172.21.15.27:0/1856306587 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4d2f1640 1 --2- 172.21.15.27:0/1856306587 >> v2:172.21.15.69:3300/0 conn(0x7fcb480b4830 0x7fcb48160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4daf2640 1 -- 172.21.15.27:0/1856306587 >> v2:172.21.15.69:3300/0 conn(0x7fcb480b4830 msgr2=0x7fcb48160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4daf2640 1 --2- 172.21.15.27:0/1856306587 >> v2:172.21.15.69:3300/0 conn(0x7fcb480b4830 0x7fcb48160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4daf2640 1 -- 172.21.15.27:0/1856306587 >> v1:172.21.15.69:6789/0 conn(0x7fcb480b29a0 legacy=0x7fcb480b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.608+0000 7fcb4daf2640 1 -- 172.21.15.27:0/1856306587 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48142f60 con 0x7fcb48142170 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.609+0000 7fcb4daf2640 1 --2- 172.21.15.27:0/1856306587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48142170 0x7fcb48142a20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcb480af240 tx=0x7fcb4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd96087f33ddb55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.609+0000 7fcb4caf0640 1 -- 172.21.15.27:0/1856306587 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44035020 con 0x7fcb48142170 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.609+0000 7fcb4caf0640 1 -- 172.21.15.27:0/1856306587 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb44031870 con 0x7fcb48142170 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.609+0000 7fcb4caf0640 1 -- 172.21.15.27:0/1856306587 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44031c40 con 0x7fcb48142170 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.610+0000 7fcb4fd56640 1 -- 172.21.15.27:0/1856306587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48142170 msgr2=0x7fcb48142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.610+0000 7fcb4fd56640 1 --2- 172.21.15.27:0/1856306587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48142170 0x7fcb48142a20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcb480af240 tx=0x7fcb4402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.610+0000 7fcb4fd56640 1 -- 172.21.15.27:0/1856306587 shutdown_connections 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.610+0000 7fcb4fd56640 1 --2- 172.21.15.27:0/1856306587 >> v2:172.21.15.69:3300/0 conn(0x7fcb480b4830 0x7fcb48160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.610+0000 7fcb4fd56640 1 --2- 172.21.15.27:0/1856306587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48142170 0x7fcb48142a20 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.610+0000 7fcb4fd56640 1 -- 172.21.15.27:0/1856306587 >> 172.21.15.27:0/1856306587 conn(0x7fcb48066d00 msgr2=0x7fcb48067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.610+0000 7fcb4fd56640 1 -- 172.21.15.27:0/1856306587 shutdown_connections 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.610+0000 7fcb4fd56640 1 -- 172.21.15.27:0/1856306587 wait complete. 2024-03-06T07:24:46.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.611+0000 7fcb4fd56640 1 Processor -- start 2024-03-06T07:24:46.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.611+0000 7fcb4fd56640 1 -- start start 2024-03-06T07:24:46.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4fd56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb480b29a0 0x7fcb4815abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4fd56640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b4830 0x7fcb4815e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4fd56640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb48142170 0x7fcb4815c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4fd56640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480711b0 con 0x7fcb480b4830 2024-03-06T07:24:46.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4fd56640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb48070ed0 con 0x7fcb48142170 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4fd56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480b4df0 con 0x7fcb480b29a0 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4d2f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b4830 0x7fcb4815e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4d2f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b4830 0x7fcb4815e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46112/0 (socket says 172.21.15.27:46112) 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4d2f1640 1 -- 172.21.15.27:0/2901459629 learned_addr learned my addr 172.21.15.27:0/2901459629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4d2f1640 1 -- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb480b29a0 msgr2=0x7fcb4815abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4d2f1640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb480b29a0 0x7fcb4815abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4d2f1640 1 -- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb48142170 msgr2=0x7fcb4815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4d2f1640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb48142170 0x7fcb4815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4d2f1640 1 -- 172.21.15.27:0/2901459629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4815b6d0 con 0x7fcb480b4830 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4d2f1640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b4830 0x7fcb4815e620 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800d5f0 tx=0x7fcb3800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/2901459629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb380125f0 con 0x7fcb480b4830 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/2901459629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb38019070 con 0x7fcb480b4830 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/2901459629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb380129d0 con 0x7fcb480b4830 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4815d370 con 0x7fcb480b4830 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.612+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4815de20 con 0x7fcb480b4830 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.613+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb480733c0 con 0x7fcb480b4830 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.614+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/2901459629 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb38012b70 con 0x7fcb480b4830 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.614+0000 7fcb3e7fc640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b500 0x7fcb2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.614+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/2901459629 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb3808a6b0 con 0x7fcb480b4830 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.614+0000 7fcb4daf2640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b500 0x7fcb2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.615+0000 7fcb4daf2640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b500 0x7fcb2c06d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcb44036000 tx=0x7fcb44030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.618+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/2901459629 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb38059490 con 0x7fcb480b4830 2024-03-06T07:24:46.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:46.842+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcb48143ee0 con 0x7fcb480b4830 2024-03-06T07:24:47.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:47.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.839+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/2901459629 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcb38058d90 con 0x7fcb480b4830 2024-03-06T07:24:47.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:47.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.841+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b500 msgr2=0x7fcb2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.841+0000 7fcb4fd56640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b500 0x7fcb2c06d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcb44036000 tx=0x7fcb44030000 comp rx=0 tx=0).stop 2024-03-06T07:24:47.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b4830 msgr2=0x7fcb4815e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b4830 0x7fcb4815e620 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800d5f0 tx=0x7fcb3800dab0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 shutdown_connections 2024-03-06T07:24:47.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b500 0x7fcb2c06d9c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb48142170 0x7fcb4815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b4830 0x7fcb4815e620 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 --2- 172.21.15.27:0/2901459629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb480b29a0 0x7fcb4815abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 >> 172.21.15.27:0/2901459629 conn(0x7fcb48066d00 msgr2=0x7fcb48142670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 shutdown_connections 2024-03-06T07:24:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:47.842+0000 7fcb4fd56640 1 -- 172.21.15.27:0/2901459629 wait complete. 2024-03-06T07:24:47.855 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:47.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.054+0000 7fa7ff676640 1 Processor -- start 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.054+0000 7fa7ff676640 1 -- start start 2024-03-06T07:24:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7ff676640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80b79f0 0x7fa7f80b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7ff676640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f80b83b0 con 0x7fa7f80b2d30 2024-03-06T07:24:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7ff676640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f80d4960 con 0x7fa7f8071540 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7ff676640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80d3230 con 0x7fa7f80b79f0 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7fcc11640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa7f80b2d30 0x7fa7f80b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56182/0 (socket says 172.21.15.27:56182) 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7fcc11640 1 -- 172.21.15.27:0/2765509759 learned_addr learned my addr 172.21.15.27:0/2765509759 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7fdc13640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80b79f0 0x7fa7f80b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259045876 0 0) 0x7fa7f80b83b0 con 0x7fa7f80b2d30 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e4003520 con 0x7fa7f80b2d30 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2455527032 0 0) 0x7fa7e4003520 con 0x7fa7f80b2d30 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80b79f0 msgr2=0x7fa7f80b7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7effff640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80b79f0 0x7fa7f80b7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 >> v1:172.21.15.69:6789/0 conn(0x7fa7f8071540 legacy=0x7fa7f8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f80d36c0 con 0x7fa7f80b2d30 2024-03-06T07:24:48.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.055+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fa7e8001670 con 0x7fa7f80b2d30 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 >> v1:172.21.15.27:6789/0 conn(0x7fa7f80b2d30 legacy=0x7fa7f80b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7effff640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80b79f0 0x7fa7e40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7effff640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7effff640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7e4004d40 0x7fa7e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80d3230 con 0x7fa7e4003f20 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e400ba20 con 0x7fa7e4004d40 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e400bc20 con 0x7fa7f80b79f0 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7fcc11640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7fd412640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80b79f0 0x7fa7e40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7fdc13640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7e4004d40 0x7fa7e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7fcc11640 1 -- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80b79f0 msgr2=0x7fa7e40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7fcc11640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80b79f0 0x7fa7e40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7fcc11640 1 -- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7e4004d40 msgr2=0x7fa7e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7fcc11640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7e4004d40 0x7fa7e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.056+0000 7fa7fcc11640 1 -- 172.21.15.27:0/2765509759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e4003520 con 0x7fa7e4003f20 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.057+0000 7fa7fcc11640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7e4004800 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8007fd0 tx=0x7fa7e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25e28539dde1cf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.057+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8007d50 con 0x7fa7e4003f20 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.057+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e8030070 con 0x7fa7e4003f20 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.057+0000 7fa7effff640 1 -- 172.21.15.27:0/2765509759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8005d70 con 0x7fa7e4003f20 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.057+0000 7fa7ff676640 1 -- 172.21.15.27:0/2765509759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 msgr2=0x7fa7e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.057+0000 7fa7ff676640 1 --2- 172.21.15.27:0/2765509759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7e4004800 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8007fd0 tx=0x7fa7e4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.057+0000 7fa7fdc13640 1 -- 172.21.15.27:0/2765509759 reap_dead start 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.057+0000 7fa7ff676640 1 -- 172.21.15.27:0/2765509759 shutdown_connections 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.057+0000 7fa7ff676640 1 -- 172.21.15.27:0/2765509759 >> 172.21.15.27:0/2765509759 conn(0x7fa7f80d3d60 msgr2=0x7fa7f80d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.058+0000 7fa7ff676640 1 -- 172.21.15.27:0/2765509759 shutdown_connections 2024-03-06T07:24:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.058+0000 7fa7ff676640 1 -- 172.21.15.27:0/2765509759 wait complete. 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.059+0000 7fa7ff676640 1 Processor -- start 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.059+0000 7fa7ff676640 1 -- start start 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7ff676640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7f80c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7ff676640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80c95c0 0x7fa7f80c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7ff676640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7f81e7190 0x7fa7f81e7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7ff676640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80d2be0 con 0x7fa7e4003f20 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7ff676640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80d2900 con 0x7fa7f81e7190 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7ff676640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80d2ec0 con 0x7fa7f80c95c0 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fd412640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7f80c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fd412640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7f80c6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46130/0 (socket says 172.21.15.27:46130) 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fd412640 1 -- 172.21.15.27:0/463615363 learned_addr learned my addr 172.21.15.27:0/463615363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fcc11640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80c95c0 0x7fa7f80c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fdc13640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7f81e7190 0x7fa7f81e7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fd412640 1 -- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80c95c0 msgr2=0x7fa7f80c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fd412640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80c95c0 0x7fa7f80c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fd412640 1 -- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7f81e7190 msgr2=0x7fa7f81e7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fd412640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7f81e7190 0x7fa7f81e7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fd412640 1 -- 172.21.15.27:0/463615363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f81e7ec0 con 0x7fa7e4003f20 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fcc11640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80c95c0 0x7fa7f80c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fdc13640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7f81e7190 0x7fa7f81e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7fd412640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7f80c6a30 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400f7c0 tx=0x7fa7f400fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7edffb640 1 -- 172.21.15.27:0/463615363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40052e0 con 0x7fa7e4003f20 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7edffb640 1 -- 172.21.15.27:0/463615363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f4018070 con 0x7fa7e4003f20 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7edffb640 1 -- 172.21.15.27:0/463615363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f400d970 con 0x7fa7e4003f20 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.060+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f81e81a0 con 0x7fa7e4003f20 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.061+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f80cb080 con 0x7fa7e4003f20 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.062+0000 7fa7edffb640 1 -- 172.21.15.27:0/463615363 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7f400db10 con 0x7fa7e4003f20 2024-03-06T07:24:48.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.062+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f80cb250 con 0x7fa7e4003f20 2024-03-06T07:24:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.062+0000 7fa7edffb640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.062+0000 7fa7edffb640 1 -- 172.21.15.27:0/463615363 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7f408a200 con 0x7fa7e4003f20 2024-03-06T07:24:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.063+0000 7fa7fcc11640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.063+0000 7fa7fcc11640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8034000 tx=0x7fa7e8039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.068+0000 7fa7edffb640 1 -- 172.21.15.27:0/463615363 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7f401a310 con 0x7fa7e4003f20 2024-03-06T07:24:48.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.288+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa7f8072070 con 0x7fa7e4003f20 2024-03-06T07:24:48.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.843+0000 7fa7edffb640 1 -- 172.21.15.27:0/463615363 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa7f4059080 con 0x7fa7e4003f20 2024-03-06T07:24:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa7dc06b4d0 msgr2=0x7fa7dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8034000 tx=0x7fa7e8039040 comp rx=0 tx=0).stop 2024-03-06T07:24:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 msgr2=0x7fa7f80c6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7f80c6a30 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400f7c0 tx=0x7fa7f400fc80 comp rx=0 tx=0).stop 2024-03-06T07:24:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 shutdown_connections 2024-03-06T07:24:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa7dc06b4d0 0x7fa7dc06d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7f81e7190 0x7fa7f81e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7f80c95c0 0x7fa7f80c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 --2- 172.21.15.27:0/463615363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e4003f20 0x7fa7f80c6a30 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 >> 172.21.15.27:0/463615363 conn(0x7fa7f80d3d60 msgr2=0x7fa7f815d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 shutdown_connections 2024-03-06T07:24:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.846+0000 7fa7ff676640 1 -- 172.21.15.27:0/463615363 wait complete. 2024-03-06T07:24:48.864 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:48.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:48.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.969+0000 7f5f9ea9d640 1 Processor -- start 2024-03-06T07:24:48.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.969+0000 7f5f9ea9d640 1 -- start start 2024-03-06T07:24:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f9ea9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f98071540 0x7f5f980b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f9ea9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980d3400 0x7f5f98071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f9ea9d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98071e60 con 0x7f5f980b2b50 2024-03-06T07:24:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f9ea9d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f980d2f70 con 0x7f5f98071540 2024-03-06T07:24:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f9ea9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980d4be0 con 0x7f5f980d3400 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f9d03a640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5f980b2b50 0x7f5f980d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56196/0 (socket says 172.21.15.27:56196) 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f9d03a640 1 -- 172.21.15.27:0/2487693098 learned_addr learned my addr 172.21.15.27:0/2487693098 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063584807 0 0) 0x7f5f98071e60 con 0x7f5f980b2b50 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f9c839640 1 --2- 172.21.15.27:0/2487693098 >> v2:172.21.15.69:3300/0 conn(0x7f5f98071540 0x7f5f980b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f88003520 con 0x7f5f980b2b50 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f97fff640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980d3400 0x7f5f98071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930320911 0 0) 0x7f5f88003520 con 0x7f5f980b2b50 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980d3400 msgr2=0x7f5f98071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980d3400 0x7f5f98071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 >> v2:172.21.15.69:3300/0 conn(0x7f5f98071540 msgr2=0x7f5f980b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 --2- 172.21.15.27:0/2487693098 >> v2:172.21.15.69:3300/0 conn(0x7f5f98071540 0x7f5f980b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980ce610 con 0x7f5f980b2b50 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f5f900019f0 con 0x7f5f980b2b50 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 >> v1:172.21.15.27:6789/0 conn(0x7f5f980b2b50 legacy=0x7f5f980d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880039b0 0x7f5f88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.970+0000 7f5f977fe640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f880043a0 0x7f5f88004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f977fe640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88004fa0 0x7f5f8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980d2f70 con 0x7f5f88004fa0 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980d4be0 con 0x7f5f880043a0 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8800bf30 con 0x7f5f880039b0 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f97fff640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980d3400 0x7f5f98071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f9d03a640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88004fa0 0x7f5f8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f9c839640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880039b0 0x7f5f88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f97fff640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f880043a0 0x7f5f88004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f9d03a640 1 -- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880039b0 msgr2=0x7f5f88003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f9d03a640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880039b0 0x7f5f88003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f9d03a640 1 -- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f880043a0 msgr2=0x7f5f88004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f9d03a640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f880043a0 0x7f5f88004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.971+0000 7f5f9d03a640 1 -- 172.21.15.27:0/2487693098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f88003520 con 0x7f5f88004fa0 2024-03-06T07:24:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.972+0000 7f5f9d03a640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88004fa0 0x7f5f8800b9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000a8a0 tx=0x7f5f88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdae5c4f0da62d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.972+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000a4c0 con 0x7f5f88004fa0 2024-03-06T07:24:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.972+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90034650 con 0x7f5f88004fa0 2024-03-06T07:24:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.972+0000 7f5f977fe640 1 -- 172.21.15.27:0/2487693098 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90034a80 con 0x7f5f88004fa0 2024-03-06T07:24:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.973+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/2487693098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88004fa0 msgr2=0x7f5f8800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.973+0000 7f5f9ea9d640 1 --2- 172.21.15.27:0/2487693098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88004fa0 0x7f5f8800b9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000a8a0 tx=0x7f5f88005830 comp rx=0 tx=0).stop 2024-03-06T07:24:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.973+0000 7f5f9d03a640 1 -- 172.21.15.27:0/2487693098 reap_dead start 2024-03-06T07:24:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.973+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/2487693098 shutdown_connections 2024-03-06T07:24:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.973+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/2487693098 >> 172.21.15.27:0/2487693098 conn(0x7f5f980d3ca0 msgr2=0x7f5f980d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.973+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/2487693098 shutdown_connections 2024-03-06T07:24:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.973+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/2487693098 wait complete. 2024-03-06T07:24:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.975+0000 7f5f9ea9d640 1 Processor -- start 2024-03-06T07:24:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.975+0000 7f5f9ea9d640 1 -- start start 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.976+0000 7f5f9ea9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980b2440 0x7f5f980ab330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.976+0000 7f5f9ea9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f980abb00 0x7f5f980afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.976+0000 7f5f9ea9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f980b00c0 0x7f5f980cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.976+0000 7f5f9ea9d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980ce8f0 con 0x7f5f980abb00 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.976+0000 7f5f9ea9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980d2c00 con 0x7f5f980b00c0 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.976+0000 7f5f9ea9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9815e370 con 0x7f5f980b2440 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.976+0000 7f5f97fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f980abb00 0x7f5f980afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.976+0000 7f5f97fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f980abb00 0x7f5f980afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46142/0 (socket says 172.21.15.27:46142) 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f97fff640 1 -- 172.21.15.27:0/1079262019 learned_addr learned my addr 172.21.15.27:0/1079262019 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f9d03a640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f980b00c0 0x7f5f980cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f9c839640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980b2440 0x7f5f980ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f97fff640 1 -- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980b2440 msgr2=0x7f5f980ab330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f97fff640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980b2440 0x7f5f980ab330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f97fff640 1 -- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f980b00c0 msgr2=0x7f5f980cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f97fff640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f980b00c0 0x7f5f980cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f97fff640 1 -- 172.21.15.27:0/1079262019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980ac2c0 con 0x7f5f980abb00 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f9d03a640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f980b00c0 0x7f5f980cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f97fff640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f980abb00 0x7f5f980afb80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5f84011540 tx=0x7f5f84011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f957fa640 1 -- 172.21.15.27:0/1079262019 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84019070 con 0x7f5f980abb00 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f957fa640 1 -- 172.21.15.27:0/1079262019 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84015930 con 0x7f5f980abb00 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f980ac5a0 con 0x7f5f980abb00 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f957fa640 1 -- 172.21.15.27:0/1079262019 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8400f040 con 0x7f5f980abb00 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.977+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f980acb60 con 0x7f5f980abb00 2024-03-06T07:24:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.978+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f98068700 con 0x7f5f980abb00 2024-03-06T07:24:48.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.979+0000 7f5f957fa640 1 -- 172.21.15.27:0/1079262019 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f8401c070 con 0x7f5f980abb00 2024-03-06T07:24:48.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.979+0000 7f5f957fa640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f7c06b500 0x7f5f7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.979+0000 7f5f957fa640 1 -- 172.21.15.27:0/1079262019 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f8408a840 con 0x7f5f980abb00 2024-03-06T07:24:48.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.980+0000 7f5f9c839640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f7c06b500 0x7f5f7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.980+0000 7f5f9c839640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f7c06b500 0x7f5f7c06d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5f9815d1c0 tx=0x7f5f8c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:48.983+0000 7f5f957fa640 1 -- 172.21.15.27:0/1079262019 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f840597a0 con 0x7f5f980abb00 2024-03-06T07:24:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.288+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f98072030 con 0x7f5f980abb00 2024-03-06T07:24:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.289+0000 7f5f957fa640 1 -- 172.21.15.27:0/1079262019 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f5f840590a0 con 0x7f5f980abb00 2024-03-06T07:24:49.292 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:49.292 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5126,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2827284911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2827284911},{"type":"v1","addr":"172.21.15.69:6836","nonce":2827284911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5419,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1663054032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1663054032},{"type":"v1","addr":"172.21.15.69:6837","nonce":1663054032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2744957333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2744957333},{"type":"v1","addr":"172.21.15.27:6834","nonce":2744957333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:45.201958+0000","modified":"2024-03-06T07:24:48.290611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5754":{"gid":5754,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2008146374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2008146374},{"type":"v1","addr":"172.21.15.27:6835","nonce":2008146374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5754,"qdb_cluster":[5754]},"id":3}]} 2024-03-06T07:24:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-03-06T07:24:49.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f7c06b500 msgr2=0x7f5f7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f7c06b500 0x7f5f7c06d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5f9815d1c0 tx=0x7f5f8c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f980abb00 msgr2=0x7f5f980afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f980abb00 0x7f5f980afb80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5f84011540 tx=0x7f5f84011a00 comp rx=0 tx=0).stop 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 shutdown_connections 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f7c06b500 0x7f5f7c06d9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f980b00c0 0x7f5f980cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f980abb00 0x7f5f980afb80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 --2- 172.21.15.27:0/1079262019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f980b2440 0x7f5f980ab330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 >> 172.21.15.27:0/1079262019 conn(0x7f5f980d3ca0 msgr2=0x7f5f9815fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 shutdown_connections 2024-03-06T07:24:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.291+0000 7f5f9ea9d640 1 -- 172.21.15.27:0/1079262019 wait complete. 2024-03-06T07:24:49.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.408+0000 7f6a19b8a640 1 Processor -- start 2024-03-06T07:24:49.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.408+0000 7f6a19b8a640 1 -- start start 2024-03-06T07:24:49.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a19b8a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a140b95a0 0x7f6a140b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a19b8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 0x7f6a140b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a19b8a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a140cea30 con 0x7f6a1415df00 2024-03-06T07:24:49.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a19b8a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a140cf3c0 con 0x7f6a140b95a0 2024-03-06T07:24:49.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a19b8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140cf0e0 con 0x7f6a140b9a10 2024-03-06T07:24:49.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a13fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 0x7f6a140b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a12ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a140b95a0 0x7f6a140b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a137fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6a1415df00 0x7f6a140d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:36808/0 (socket says 172.21.15.27:36808) 2024-03-06T07:24:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a12ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a140b95a0 0x7f6a140b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:44436/0 (socket says 172.21.15.27:44436) 2024-03-06T07:24:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.409+0000 7f6a137fe640 1 -- 172.21.15.27:0/2531522730 learned_addr learned my addr 172.21.15.27:0/2531522730 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.410+0000 7f6a13fff640 1 -- 172.21.15.27:0/2531522730 >> v2:172.21.15.69:3300/0 conn(0x7f6a140b95a0 msgr2=0x7f6a140b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.410+0000 7f6a13fff640 1 --2- 172.21.15.27:0/2531522730 >> v2:172.21.15.69:3300/0 conn(0x7f6a140b95a0 0x7f6a140b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.410+0000 7f6a13fff640 1 -- 172.21.15.27:0/2531522730 >> v1:172.21.15.69:6789/0 conn(0x7f6a1415df00 legacy=0x7f6a140d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.410+0000 7f6a13fff640 1 -- 172.21.15.27:0/2531522730 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140ce470 con 0x7f6a140b9a10 2024-03-06T07:24:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.410+0000 7f6a13fff640 1 --2- 172.21.15.27:0/2531522730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 0x7f6a140b5c40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009d90 tx=0x7f6a0c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=225d165d915af613 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a127fc640 1 -- 172.21.15.27:0/2531522730 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c039070 con 0x7f6a140b9a10 2024-03-06T07:24:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a127fc640 1 -- 172.21.15.27:0/2531522730 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c02cc50 con 0x7f6a140b9a10 2024-03-06T07:24:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a127fc640 1 -- 172.21.15.27:0/2531522730 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c034440 con 0x7f6a140b9a10 2024-03-06T07:24:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a19b8a640 1 -- 172.21.15.27:0/2531522730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 msgr2=0x7f6a140b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a19b8a640 1 --2- 172.21.15.27:0/2531522730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 0x7f6a140b5c40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009d90 tx=0x7f6a0c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a19b8a640 1 -- 172.21.15.27:0/2531522730 shutdown_connections 2024-03-06T07:24:49.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a19b8a640 1 --2- 172.21.15.27:0/2531522730 >> v2:172.21.15.69:3300/0 conn(0x7f6a140b95a0 0x7f6a140b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a19b8a640 1 --2- 172.21.15.27:0/2531522730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 0x7f6a140b5c40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a19b8a640 1 -- 172.21.15.27:0/2531522730 >> 172.21.15.27:0/2531522730 conn(0x7f6a14071960 msgr2=0x7f6a14071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a19b8a640 1 -- 172.21.15.27:0/2531522730 shutdown_connections 2024-03-06T07:24:49.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.411+0000 7f6a19b8a640 1 -- 172.21.15.27:0/2531522730 wait complete. 2024-03-06T07:24:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.413+0000 7f6a19b8a640 1 Processor -- start 2024-03-06T07:24:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.413+0000 7f6a19b8a640 1 -- start start 2024-03-06T07:24:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.413+0000 7f6a19b8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a140b95a0 0x7f6a1415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.413+0000 7f6a19b8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 0x7f6a14154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.413+0000 7f6a19b8a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a1415df00 0x7f6a140d3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.413+0000 7f6a19b8a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140cedf0 con 0x7f6a140b95a0 2024-03-06T07:24:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.413+0000 7f6a19b8a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140cec70 con 0x7f6a1415df00 2024-03-06T07:24:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.413+0000 7f6a19b8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140d3220 con 0x7f6a140b9a10 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a137fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a140b95a0 0x7f6a1415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a137fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a140b95a0 0x7f6a1415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46150/0 (socket says 172.21.15.27:46150) 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a137fe640 1 -- 172.21.15.27:0/3771975256 learned_addr learned my addr 172.21.15.27:0/3771975256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a12ffd640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 0x7f6a14154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a13fff640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a1415df00 0x7f6a140d3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a137fe640 1 -- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 msgr2=0x7f6a14154e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a137fe640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 0x7f6a14154e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a137fe640 1 -- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a1415df00 msgr2=0x7f6a140d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a137fe640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a1415df00 0x7f6a140d3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a137fe640 1 -- 172.21.15.27:0/3771975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140cd910 con 0x7f6a140b95a0 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a13fff640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a1415df00 0x7f6a140d3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a137fe640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a140b95a0 0x7f6a1415ef40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c480 tx=0x7f69fc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a0bfff640 1 -- 172.21.15.27:0/3771975256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc017070 con 0x7f6a140b95a0 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a0bfff640 1 -- 172.21.15.27:0/3771975256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc005ce0 con 0x7f6a140b95a0 2024-03-06T07:24:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1415e460 con 0x7f6a140b95a0 2024-03-06T07:24:49.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a0bfff640 1 -- 172.21.15.27:0/3771975256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc0072f0 con 0x7f6a140b95a0 2024-03-06T07:24:49.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.414+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1415ea20 con 0x7f6a140b95a0 2024-03-06T07:24:49.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.415+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a140b86f0 con 0x7f6a140b95a0 2024-03-06T07:24:49.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.416+0000 7f6a0bfff640 1 -- 172.21.15.27:0/3771975256 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69fc01a070 con 0x7f6a140b95a0 2024-03-06T07:24:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.417+0000 7f6a0bfff640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f69f406b4d0 0x7f69f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.417+0000 7f6a0bfff640 1 -- 172.21.15.27:0/3771975256 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69fc089420 con 0x7f6a140b95a0 2024-03-06T07:24:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.417+0000 7f6a12ffd640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f69f406b4d0 0x7f69f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.418+0000 7f6a12ffd640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f69f406b4d0 0x7f69f406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a141589d0 tx=0x7f6a04008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.421+0000 7f6a0bfff640 1 -- 172.21.15.27:0/3771975256 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a140b86f0 con 0x7f6a140b95a0 2024-03-06T07:24:49.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.606+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a14067ff0 con 0x7f6a140b95a0 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.607+0000 7f6a0bfff640 1 -- 172.21.15.27:0/3771975256 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f69fc057ce0 con 0x7f6a140b95a0 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:24:46.825646+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:43.493554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:44.172410+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f69f406b4d0 msgr2=0x7f69f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f69f406b4d0 0x7f69f406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a141589d0 tx=0x7f6a04008040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a140b95a0 msgr2=0x7f6a1415ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a140b95a0 0x7f6a1415ef40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c480 tx=0x7f69fc00c940 comp rx=0 tx=0).stop 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 shutdown_connections 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f69f406b4d0 0x7f69f406d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a1415df00 0x7f6a140d3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a140b9a10 0x7f6a14154e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 --2- 172.21.15.27:0/3771975256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a140b95a0 0x7f6a1415ef40 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 >> 172.21.15.27:0/3771975256 conn(0x7f6a14071960 msgr2=0x7f6a14071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 shutdown_connections 2024-03-06T07:24:49.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.610+0000 7f6a19b8a640 1 -- 172.21.15.27:0/3771975256 wait complete. 2024-03-06T07:24:49.629 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:49.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:49.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.734+0000 7f7d7855e640 1 Processor -- start 2024-03-06T07:24:49.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.734+0000 7f7d7855e640 1 -- start start 2024-03-06T07:24:49.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d7855e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d700b9a10 0x7f7d700d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d7855e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d700cea30 con 0x7f7d7015df00 2024-03-06T07:24:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d7855e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d700cf3c0 con 0x7f7d700b95a0 2024-03-06T07:24:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d7855e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d700cf0e0 con 0x7f7d700b9a10 2024-03-06T07:24:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d76afb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d700b9a10 0x7f7d700d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d762fa640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7d7015df00 0x7f7d700d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56202/0 (socket says 172.21.15.27:56202) 2024-03-06T07:24:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d762fa640 1 -- 172.21.15.27:0/322262937 learned_addr learned my addr 172.21.15.27:0/322262937 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d76afb640 1 -- 172.21.15.27:0/322262937 >> v1:172.21.15.69:6789/0 conn(0x7f7d700b95a0 legacy=0x7f7d700b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d76afb640 1 -- 172.21.15.27:0/322262937 >> v1:172.21.15.27:6789/0 conn(0x7f7d7015df00 legacy=0x7f7d700d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d752f8640 1 -- 172.21.15.27:0/322262937 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767930924 0 0) 0x7f7d700cea30 con 0x7f7d7015df00 2024-03-06T07:24:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.735+0000 7f7d76afb640 1 -- 172.21.15.27:0/322262937 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d700ce470 con 0x7f7d700b9a10 2024-03-06T07:24:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.736+0000 7f7d76afb640 1 --2- 172.21.15.27:0/322262937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b9a10 0x7f7d700d3220 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7d700cf590 tx=0x7f7d6802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa04278a65732020 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.736+0000 7f7d752f8640 1 -- 172.21.15.27:0/322262937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68039070 con 0x7f7d700b9a10 2024-03-06T07:24:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.736+0000 7f7d752f8640 1 -- 172.21.15.27:0/322262937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6802cce0 con 0x7f7d700b9a10 2024-03-06T07:24:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.736+0000 7f7d752f8640 1 -- 172.21.15.27:0/322262937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68034660 con 0x7f7d700b9a10 2024-03-06T07:24:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.737+0000 7f7d7855e640 1 -- 172.21.15.27:0/322262937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b9a10 msgr2=0x7f7d700d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.737+0000 7f7d7855e640 1 --2- 172.21.15.27:0/322262937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b9a10 0x7f7d700d3220 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7d700cf590 tx=0x7f7d6802bf30 comp rx=0 tx=0).stop 2024-03-06T07:24:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.737+0000 7f7d7855e640 1 -- 172.21.15.27:0/322262937 shutdown_connections 2024-03-06T07:24:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.737+0000 7f7d7855e640 1 --2- 172.21.15.27:0/322262937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b9a10 0x7f7d700d3220 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.737+0000 7f7d7855e640 1 -- 172.21.15.27:0/322262937 >> 172.21.15.27:0/322262937 conn(0x7f7d70071960 msgr2=0x7f7d70071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.737+0000 7f7d7855e640 1 -- 172.21.15.27:0/322262937 shutdown_connections 2024-03-06T07:24:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.737+0000 7f7d7855e640 1 -- 172.21.15.27:0/322262937 wait complete. 2024-03-06T07:24:49.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.738+0000 7f7d7855e640 1 Processor -- start 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.738+0000 7f7d7855e640 1 -- start start 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d7855e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b95a0 0x7f7d70158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d7855e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d700b9a10 0x7f7d70160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d7855e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d7015df00 0x7f7d701f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d7855e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d700cedf0 con 0x7f7d700b95a0 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d7855e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d700cec70 con 0x7f7d7015df00 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d7855e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d700b5e10 con 0x7f7d700b9a10 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d762fa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b95a0 0x7f7d70158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d762fa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b95a0 0x7f7d70158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46168/0 (socket says 172.21.15.27:46168) 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d762fa640 1 -- 172.21.15.27:0/3361619551 learned_addr learned my addr 172.21.15.27:0/3361619551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d76afb640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d7015df00 0x7f7d701f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d75af9640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d700b9a10 0x7f7d70160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d762fa640 1 -- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d700b9a10 msgr2=0x7f7d70160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d762fa640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d700b9a10 0x7f7d70160ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d762fa640 1 -- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d7015df00 msgr2=0x7f7d701f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d762fa640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d7015df00 0x7f7d701f6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d762fa640 1 -- 172.21.15.27:0/3361619551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d701f7370 con 0x7f7d700b95a0 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d75af9640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d700b9a10 0x7f7d70160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d76afb640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d7015df00 0x7f7d701f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d762fa640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b95a0 0x7f7d70158620 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00c420 tx=0x7f7d6c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.739+0000 7f7d66ffd640 1 -- 172.21.15.27:0/3361619551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0074e0 con 0x7f7d700b95a0 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.740+0000 7f7d66ffd640 1 -- 172.21.15.27:0/3361619551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6c0047e0 con 0x7f7d700b95a0 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.740+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d701f7610 con 0x7f7d700b95a0 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.740+0000 7f7d66ffd640 1 -- 172.21.15.27:0/3361619551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c004c10 con 0x7f7d700b95a0 2024-03-06T07:24:49.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.740+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d701f7bd0 con 0x7f7d700b95a0 2024-03-06T07:24:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.741+0000 7f7d66ffd640 1 -- 172.21.15.27:0/3361619551 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d6c011070 con 0x7f7d700b95a0 2024-03-06T07:24:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.741+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d70067ff0 con 0x7f7d700b95a0 2024-03-06T07:24:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.741+0000 7f7d66ffd640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d5406b4d0 0x7f7d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.741+0000 7f7d66ffd640 1 -- 172.21.15.27:0/3361619551 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d6c089dc0 con 0x7f7d700b95a0 2024-03-06T07:24:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.742+0000 7f7d75af9640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d5406b4d0 0x7f7d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.742+0000 7f7d75af9640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d5406b4d0 0x7f7d5406d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7d700d4680 tx=0x7f7d60009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.746+0000 7f7d66ffd640 1 -- 172.21.15.27:0/3361619551 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d6c00e280 con 0x7f7d700b95a0 2024-03-06T07:24:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.966+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d700b8f60 con 0x7f7d5406b4d0 2024-03-06T07:24:49.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.973+0000 7f7d66ffd640 1 -- 172.21.15.27:0/3361619551 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246637 (secure 0 0 0) 0x7f7d700b8f60 con 0x7f7d5406b4d0 2024-03-06T07:24:49.978 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:49.980 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:24:49.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d5406b4d0 msgr2=0x7f7d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d5406b4d0 0x7f7d5406d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7d700d4680 tx=0x7f7d60009040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b95a0 msgr2=0x7f7d70158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b95a0 0x7f7d70158620 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00c420 tx=0x7f7d6c00c8e0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 shutdown_connections 2024-03-06T07:24:49.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d5406b4d0 0x7f7d5406d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d7015df00 0x7f7d701f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d700b9a10 0x7f7d70160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 --2- 172.21.15.27:0/3361619551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d700b95a0 0x7f7d70158620 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 >> 172.21.15.27:0/3361619551 conn(0x7f7d70071960 msgr2=0x7f7d700b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 shutdown_connections 2024-03-06T07:24:49.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:49.976+0000 7f7d7855e640 1 -- 172.21.15.27:0/3361619551 wait complete. 2024-03-06T07:24:49.989 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-03-06T07:24:48.014691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254572,"kb_used_data":5120,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720148,"statfs":{"total":773094113280,"available":772833431552,"internally_reserved":0,"allocated":5242880,"data_stored":2796886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.194724"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.829080+0000","last_change":"2024-03-06T07:24:44.829080+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.829080+0000","last_undegraded":"2024-03-06T07:24:44.829080+0000","last_fullsized":"2024-03-06T07:24:44.829080+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.832490+0000","last_change":"2024-03-06T07:24:44.832490+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.832490+0000","last_undegraded":"2024-03-06T07:24:44.832490+0000","last_fullsized":"2024-03-06T07:24:44.832490+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828899+0000","last_change":"2024-03-06T07:24:44.828899+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828899+0000","last_undegraded":"2024-03-06T07:24:44.828899+0000","last_fullsized":"2024-03-06T07:24:44.828899+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.833390+0000","last_change":"2024-03-06T07:24:44.833390+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.833390+0000","last_undegraded":"2024-03-06T07:24:44.833390+0000","last_fullsized":"2024-03-06T07:24:44.833390+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.832258+0000","last_change":"2024-03-06T07:24:44.832258+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.832258+0000","last_undegraded":"2024-03-06T07:24:44.832258+0000","last_fullsized":"2024-03-06T07:24:44.832258+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.834990+0000","last_change":"2024-03-06T07:24:44.834990+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.834990+0000","last_undegraded":"2024-03-06T07:24:44.834990+0000","last_fullsized":"2024-03-06T07:24:44.834990+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828673+0000","last_change":"2024-03-06T07:24:44.828673+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828673+0000","last_undegraded":"2024-03-06T07:24:44.828673+0000","last_fullsized":"2024-03-06T07:24:44.828673+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.834257+0000","last_change":"2024-03-06T07:24:44.834257+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.834257+0000","last_undegraded":"2024-03-06T07:24:44.834257+0000","last_fullsized":"2024-03-06T07:24:44.834257+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.832828+0000","last_change":"2024-03-06T07:24:44.832828+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.832828+0000","last_undegraded":"2024-03-06T07:24:44.832828+0000","last_fullsized":"2024-03-06T07:24:44.832828+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.832691+0000","last_change":"2024-03-06T07:24:44.832691+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.832691+0000","last_undegraded":"2024-03-06T07:24:44.832691+0000","last_fullsized":"2024-03-06T07:24:44.832691+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.830661+0000","last_change":"2024-03-06T07:24:44.830661+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.830661+0000","last_undegraded":"2024-03-06T07:24:44.830661+0000","last_fullsized":"2024-03-06T07:24:44.830661+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.836103+0000","last_change":"2024-03-06T07:24:44.836103+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.836103+0000","last_undegraded":"2024-03-06T07:24:44.836103+0000","last_fullsized":"2024-03-06T07:24:44.836103+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.829203+0000","last_change":"2024-03-06T07:24:44.829203+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.829203+0000","last_undegraded":"2024-03-06T07:24:44.829203+0000","last_fullsized":"2024-03-06T07:24:44.829203+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.830995+0000","last_change":"2024-03-06T07:24:44.830995+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.830995+0000","last_undegraded":"2024-03-06T07:24:44.830995+0000","last_fullsized":"2024-03-06T07:24:44.830995+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.830620+0000","last_change":"2024-03-06T07:24:44.830620+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.830620+0000","last_undegraded":"2024-03-06T07:24:44.830620+0000","last_fullsized":"2024-03-06T07:24:44.830620+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.833104+0000","last_change":"2024-03-06T07:24:44.833104+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.833104+0000","last_undegraded":"2024-03-06T07:24:44.833104+0000","last_fullsized":"2024-03-06T07:24:44.833104+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.829502+0000","last_change":"2024-03-06T07:24:44.829502+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.829502+0000","last_undegraded":"2024-03-06T07:24:44.829502+0000","last_fullsized":"2024-03-06T07:24:44.829502+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.829362+0000","last_change":"2024-03-06T07:24:44.829362+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.829362+0000","last_undegraded":"2024-03-06T07:24:44.829362+0000","last_fullsized":"2024-03-06T07:24:44.829362+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.831226+0000","last_change":"2024-03-06T07:24:44.831226+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.831226+0000","last_undegraded":"2024-03-06T07:24:44.831226+0000","last_fullsized":"2024-03-06T07:24:44.831226+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.829567+0000","last_change":"2024-03-06T07:24:44.829567+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.829567+0000","last_undegraded":"2024-03-06T07:24:44.829567+0000","last_fullsized":"2024-03-06T07:24:44.829567+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.835238+0000","last_change":"2024-03-06T07:24:44.835238+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.835238+0000","last_undegraded":"2024-03-06T07:24:44.835238+0000","last_fullsized":"2024-03-06T07:24:44.835238+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.827853+0000","last_change":"2024-03-06T07:24:44.827853+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.827853+0000","last_undegraded":"2024-03-06T07:24:44.827853+0000","last_fullsized":"2024-03-06T07:24:44.827853+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.833457+0000","last_change":"2024-03-06T07:24:44.833457+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.833457+0000","last_undegraded":"2024-03-06T07:24:44.833457+0000","last_fullsized":"2024-03-06T07:24:44.833457+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.832669+0000","last_change":"2024-03-06T07:24:44.832669+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.832669+0000","last_undegraded":"2024-03-06T07:24:44.832669+0000","last_fullsized":"2024-03-06T07:24:44.832669+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.835068+0000","last_change":"2024-03-06T07:24:44.835068+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.835068+0000","last_undegraded":"2024-03-06T07:24:44.835068+0000","last_fullsized":"2024-03-06T07:24:44.835068+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.829539+0000","last_change":"2024-03-06T07:24:44.829539+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.829539+0000","last_undegraded":"2024-03-06T07:24:44.829539+0000","last_fullsized":"2024-03-06T07:24:44.829539+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828462+0000","last_change":"2024-03-06T07:24:44.828462+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828462+0000","last_undegraded":"2024-03-06T07:24:44.828462+0000","last_fullsized":"2024-03-06T07:24:44.828462+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.835697+0000","last_change":"2024-03-06T07:24:44.835697+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.835697+0000","last_undegraded":"2024-03-06T07:24:44.835697+0000","last_fullsized":"2024-03-06T07:24:44.835697+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.831680+0000","last_change":"2024-03-06T07:24:44.831680+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.831680+0000","last_undegraded":"2024-03-06T07:24:44.831680+0000","last_fullsized":"2024-03-06T07:24:44.831680+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828664+0000","last_change":"2024-03-06T07:24:44.828664+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828664+0000","last_undegraded":"2024-03-06T07:24:44.828664+0000","last_fullsized":"2024-03-06T07:24:44.828664+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.829906+0000","last_change":"2024-03-06T07:24:44.829906+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.829906+0000","last_undegraded":"2024-03-06T07:24:44.829906+0000","last_fullsized":"2024-03-06T07:24:44.829906+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828344+0000","last_change":"2024-03-06T07:24:44.828344+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828344+0000","last_undegraded":"2024-03-06T07:24:44.828344+0000","last_fullsized":"2024-03-06T07:24:44.828344+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828201+0000","last_change":"2024-03-06T07:24:44.828201+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828201+0000","last_undegraded":"2024-03-06T07:24:44.828201+0000","last_fullsized":"2024-03-06T07:24:44.828201+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.844009+0000","last_change":"2024-03-06T07:24:44.839049+0000","last_active":"2024-03-06T07:24:44.844009+0000","last_peered":"2024-03-06T07:24:44.844009+0000","last_clean":"2024-03-06T07:24:44.844009+0000","last_became_active":"2024-03-06T07:24:44.838225+0000","last_became_peered":"2024-03-06T07:24:44.838225+0000","last_unstale":"2024-03-06T07:24:44.844009+0000","last_undegraded":"2024-03-06T07:24:44.844009+0000","last_fullsized":"2024-03-06T07:24:44.844009+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:52.834947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.834460+0000","last_change":"2024-03-06T07:24:44.834460+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.834460+0000","last_undegraded":"2024-03-06T07:24:44.834460+0000","last_fullsized":"2024-03-06T07:24:44.834460+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.841562+0000","last_change":"2024-03-06T07:24:44.837067+0000","last_active":"2024-03-06T07:24:44.841562+0000","last_peered":"2024-03-06T07:24:44.841562+0000","last_clean":"2024-03-06T07:24:44.841562+0000","last_became_active":"2024-03-06T07:24:44.836744+0000","last_became_peered":"2024-03-06T07:24:44.836744+0000","last_unstale":"2024-03-06T07:24:44.841562+0000","last_undegraded":"2024-03-06T07:24:44.841562+0000","last_fullsized":"2024-03-06T07:24:44.841562+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:35.207003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.835277+0000","last_change":"2024-03-06T07:24:44.835277+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.835277+0000","last_undegraded":"2024-03-06T07:24:44.835277+0000","last_fullsized":"2024-03-06T07:24:44.835277+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.844624+0000","last_change":"2024-03-06T07:24:44.840722+0000","last_active":"2024-03-06T07:24:44.844624+0000","last_peered":"2024-03-06T07:24:44.844624+0000","last_clean":"2024-03-06T07:24:44.844624+0000","last_became_active":"2024-03-06T07:24:44.840432+0000","last_became_peered":"2024-03-06T07:24:44.840432+0000","last_unstale":"2024-03-06T07:24:44.844624+0000","last_undegraded":"2024-03-06T07:24:44.844624+0000","last_fullsized":"2024-03-06T07:24:44.844624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:31.940412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.829150+0000","last_change":"2024-03-06T07:24:44.829150+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.829150+0000","last_undegraded":"2024-03-06T07:24:44.829150+0000","last_fullsized":"2024-03-06T07:24:44.829150+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.841490+0000","last_change":"2024-03-06T07:24:44.840470+0000","last_active":"2024-03-06T07:24:44.841490+0000","last_peered":"2024-03-06T07:24:44.841490+0000","last_clean":"2024-03-06T07:24:44.841490+0000","last_became_active":"2024-03-06T07:24:44.840199+0000","last_became_peered":"2024-03-06T07:24:44.840199+0000","last_unstale":"2024-03-06T07:24:44.841490+0000","last_undegraded":"2024-03-06T07:24:44.841490+0000","last_fullsized":"2024-03-06T07:24:44.841490+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:14.798796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828055+0000","last_change":"2024-03-06T07:24:44.828055+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828055+0000","last_undegraded":"2024-03-06T07:24:44.828055+0000","last_fullsized":"2024-03-06T07:24:44.828055+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.846530+0000","last_change":"2024-03-06T07:24:44.841918+0000","last_active":"2024-03-06T07:24:44.846530+0000","last_peered":"2024-03-06T07:24:44.846530+0000","last_clean":"2024-03-06T07:24:44.846530+0000","last_became_active":"2024-03-06T07:24:44.841583+0000","last_became_peered":"2024-03-06T07:24:44.841583+0000","last_unstale":"2024-03-06T07:24:44.846530+0000","last_undegraded":"2024-03-06T07:24:44.846530+0000","last_fullsized":"2024-03-06T07:24:44.846530+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:10.238622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.831425+0000","last_change":"2024-03-06T07:24:44.831425+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.831425+0000","last_undegraded":"2024-03-06T07:24:44.831425+0000","last_fullsized":"2024-03-06T07:24:44.831425+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.841312+0000","last_change":"2024-03-06T07:24:44.837214+0000","last_active":"2024-03-06T07:24:44.841312+0000","last_peered":"2024-03-06T07:24:44.841312+0000","last_clean":"2024-03-06T07:24:44.841312+0000","last_became_active":"2024-03-06T07:24:44.836884+0000","last_became_peered":"2024-03-06T07:24:44.836884+0000","last_unstale":"2024-03-06T07:24:44.841312+0000","last_undegraded":"2024-03-06T07:24:44.841312+0000","last_fullsized":"2024-03-06T07:24:44.841312+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:16.805059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828309+0000","last_change":"2024-03-06T07:24:44.828309+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828309+0000","last_undegraded":"2024-03-06T07:24:44.828309+0000","last_fullsized":"2024-03-06T07:24:44.828309+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.844149+0000","last_change":"2024-03-06T07:24:44.842872+0000","last_active":"2024-03-06T07:24:44.844149+0000","last_peered":"2024-03-06T07:24:44.844149+0000","last_clean":"2024-03-06T07:24:44.844149+0000","last_became_active":"2024-03-06T07:24:44.842566+0000","last_became_peered":"2024-03-06T07:24:44.842566+0000","last_unstale":"2024-03-06T07:24:44.844149+0000","last_undegraded":"2024-03-06T07:24:44.844149+0000","last_fullsized":"2024-03-06T07:24:44.844149+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:11.549332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.833877+0000","last_change":"2024-03-06T07:24:44.833877+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.833877+0000","last_undegraded":"2024-03-06T07:24:44.833877+0000","last_fullsized":"2024-03-06T07:24:44.833877+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.842442+0000","last_change":"2024-03-06T07:24:44.836677+0000","last_active":"2024-03-06T07:24:44.842442+0000","last_peered":"2024-03-06T07:24:44.842442+0000","last_clean":"2024-03-06T07:24:44.842442+0000","last_became_active":"2024-03-06T07:24:44.836387+0000","last_became_peered":"2024-03-06T07:24:44.836387+0000","last_unstale":"2024-03-06T07:24:44.842442+0000","last_undegraded":"2024-03-06T07:24:44.842442+0000","last_fullsized":"2024-03-06T07:24:44.842442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:31.450535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828915+0000","last_change":"2024-03-06T07:24:44.828915+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828915+0000","last_undegraded":"2024-03-06T07:24:44.828915+0000","last_fullsized":"2024-03-06T07:24:44.828915+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.842311+0000","last_change":"2024-03-06T07:24:44.840215+0000","last_active":"2024-03-06T07:24:44.842311+0000","last_peered":"2024-03-06T07:24:44.842311+0000","last_clean":"2024-03-06T07:24:44.842311+0000","last_became_active":"2024-03-06T07:24:44.839701+0000","last_became_peered":"2024-03-06T07:24:44.839701+0000","last_unstale":"2024-03-06T07:24:44.842311+0000","last_undegraded":"2024-03-06T07:24:44.842311+0000","last_fullsized":"2024-03-06T07:24:44.842311+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:31.940412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.827842+0000","last_change":"2024-03-06T07:24:44.827842+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.827842+0000","last_undegraded":"2024-03-06T07:24:44.827842+0000","last_fullsized":"2024-03-06T07:24:44.827842+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.842557+0000","last_change":"2024-03-06T07:24:44.838398+0000","last_active":"2024-03-06T07:24:44.842557+0000","last_peered":"2024-03-06T07:24:44.842557+0000","last_clean":"2024-03-06T07:24:44.842557+0000","last_became_active":"2024-03-06T07:24:44.838122+0000","last_became_peered":"2024-03-06T07:24:44.838122+0000","last_unstale":"2024-03-06T07:24:44.842557+0000","last_undegraded":"2024-03-06T07:24:44.842557+0000","last_fullsized":"2024-03-06T07:24:44.842557+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:31.579386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.831150+0000","last_change":"2024-03-06T07:24:44.831150+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.831150+0000","last_undegraded":"2024-03-06T07:24:44.831150+0000","last_fullsized":"2024-03-06T07:24:44.831150+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.843052+0000","last_change":"2024-03-06T07:24:44.841428+0000","last_active":"2024-03-06T07:24:44.843052+0000","last_peered":"2024-03-06T07:24:44.843052+0000","last_clean":"2024-03-06T07:24:44.843052+0000","last_became_active":"2024-03-06T07:24:44.841148+0000","last_became_peered":"2024-03-06T07:24:44.841148+0000","last_unstale":"2024-03-06T07:24:44.843052+0000","last_undegraded":"2024-03-06T07:24:44.843052+0000","last_fullsized":"2024-03-06T07:24:44.843052+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:48.042962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.830291+0000","last_change":"2024-03-06T07:24:44.830291+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.830291+0000","last_undegraded":"2024-03-06T07:24:44.830291+0000","last_fullsized":"2024-03-06T07:24:44.830291+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.844477+0000","last_change":"2024-03-06T07:24:44.841064+0000","last_active":"2024-03-06T07:24:44.844477+0000","last_peered":"2024-03-06T07:24:44.844477+0000","last_clean":"2024-03-06T07:24:44.844477+0000","last_became_active":"2024-03-06T07:24:44.840735+0000","last_became_peered":"2024-03-06T07:24:44.840735+0000","last_unstale":"2024-03-06T07:24:44.844477+0000","last_undegraded":"2024-03-06T07:24:44.844477+0000","last_fullsized":"2024-03-06T07:24:44.844477+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:33.706628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.831755+0000","last_change":"2024-03-06T07:24:44.831755+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.831755+0000","last_undegraded":"2024-03-06T07:24:44.831755+0000","last_fullsized":"2024-03-06T07:24:44.831755+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.846802+0000","last_change":"2024-03-06T07:24:44.843551+0000","last_active":"2024-03-06T07:24:44.846802+0000","last_peered":"2024-03-06T07:24:44.846802+0000","last_clean":"2024-03-06T07:24:44.846802+0000","last_became_active":"2024-03-06T07:24:44.843257+0000","last_became_peered":"2024-03-06T07:24:44.843257+0000","last_unstale":"2024-03-06T07:24:44.846802+0000","last_undegraded":"2024-03-06T07:24:44.846802+0000","last_fullsized":"2024-03-06T07:24:44.846802+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:48.042962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.830851+0000","last_change":"2024-03-06T07:24:44.830851+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.830851+0000","last_undegraded":"2024-03-06T07:24:44.830851+0000","last_fullsized":"2024-03-06T07:24:44.830851+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.840208+0000","last_change":"2024-03-06T07:24:44.837407+0000","last_active":"2024-03-06T07:24:44.840208+0000","last_peered":"2024-03-06T07:24:44.840208+0000","last_clean":"2024-03-06T07:24:44.840208+0000","last_became_active":"2024-03-06T07:24:44.836928+0000","last_became_peered":"2024-03-06T07:24:44.836928+0000","last_unstale":"2024-03-06T07:24:44.840208+0000","last_undegraded":"2024-03-06T07:24:44.840208+0000","last_fullsized":"2024-03-06T07:24:44.840208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:52.834947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.834786+0000","last_change":"2024-03-06T07:24:44.834786+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.834786+0000","last_undegraded":"2024-03-06T07:24:44.834786+0000","last_fullsized":"2024-03-06T07:24:44.834786+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.846566+0000","last_change":"2024-03-06T07:24:44.843409+0000","last_active":"2024-03-06T07:24:44.846566+0000","last_peered":"2024-03-06T07:24:44.846566+0000","last_clean":"2024-03-06T07:24:44.846566+0000","last_became_active":"2024-03-06T07:24:44.843072+0000","last_became_peered":"2024-03-06T07:24:44.843072+0000","last_unstale":"2024-03-06T07:24:44.846566+0000","last_undegraded":"2024-03-06T07:24:44.846566+0000","last_fullsized":"2024-03-06T07:24:44.846566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:14.798796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828275+0000","last_change":"2024-03-06T07:24:44.828275+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828275+0000","last_undegraded":"2024-03-06T07:24:44.828275+0000","last_fullsized":"2024-03-06T07:24:44.828275+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.842504+0000","last_change":"2024-03-06T07:24:44.838168+0000","last_active":"2024-03-06T07:24:44.842504+0000","last_peered":"2024-03-06T07:24:44.842504+0000","last_clean":"2024-03-06T07:24:44.842504+0000","last_became_active":"2024-03-06T07:24:44.837640+0000","last_became_peered":"2024-03-06T07:24:44.837640+0000","last_unstale":"2024-03-06T07:24:44.842504+0000","last_undegraded":"2024-03-06T07:24:44.842504+0000","last_fullsized":"2024-03-06T07:24:44.842504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:16.805059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828967+0000","last_change":"2024-03-06T07:24:44.828967+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828967+0000","last_undegraded":"2024-03-06T07:24:44.828967+0000","last_fullsized":"2024-03-06T07:24:44.828967+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.846662+0000","last_change":"2024-03-06T07:24:44.842676+0000","last_active":"2024-03-06T07:24:44.846662+0000","last_peered":"2024-03-06T07:24:44.846662+0000","last_clean":"2024-03-06T07:24:44.846662+0000","last_became_active":"2024-03-06T07:24:44.842380+0000","last_became_peered":"2024-03-06T07:24:44.842380+0000","last_unstale":"2024-03-06T07:24:44.846662+0000","last_undegraded":"2024-03-06T07:24:44.846662+0000","last_fullsized":"2024-03-06T07:24:44.846662+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:03.504170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828079+0000","last_change":"2024-03-06T07:24:44.828079+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828079+0000","last_undegraded":"2024-03-06T07:24:44.828079+0000","last_fullsized":"2024-03-06T07:24:44.828079+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.842402+0000","last_change":"2024-03-06T07:24:44.840199+0000","last_active":"2024-03-06T07:24:44.842402+0000","last_peered":"2024-03-06T07:24:44.842402+0000","last_clean":"2024-03-06T07:24:44.842402+0000","last_became_active":"2024-03-06T07:24:44.839854+0000","last_became_peered":"2024-03-06T07:24:44.839854+0000","last_unstale":"2024-03-06T07:24:44.842402+0000","last_undegraded":"2024-03-06T07:24:44.842402+0000","last_fullsized":"2024-03-06T07:24:44.842402+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:55.061981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.831954+0000","last_change":"2024-03-06T07:24:44.831954+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.831954+0000","last_undegraded":"2024-03-06T07:24:44.831954+0000","last_fullsized":"2024-03-06T07:24:44.831954+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.845049+0000","last_change":"2024-03-06T07:24:44.843436+0000","last_active":"2024-03-06T07:24:44.845049+0000","last_peered":"2024-03-06T07:24:44.845049+0000","last_clean":"2024-03-06T07:24:44.845049+0000","last_became_active":"2024-03-06T07:24:44.843137+0000","last_became_peered":"2024-03-06T07:24:44.843137+0000","last_unstale":"2024-03-06T07:24:44.845049+0000","last_undegraded":"2024-03-06T07:24:44.845049+0000","last_fullsized":"2024-03-06T07:24:44.845049+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:42.430986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.831749+0000","last_change":"2024-03-06T07:24:44.831749+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.831749+0000","last_undegraded":"2024-03-06T07:24:44.831749+0000","last_fullsized":"2024-03-06T07:24:44.831749+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.844117+0000","last_change":"2024-03-06T07:24:44.836475+0000","last_active":"2024-03-06T07:24:44.844117+0000","last_peered":"2024-03-06T07:24:44.844117+0000","last_clean":"2024-03-06T07:24:44.844117+0000","last_became_active":"2024-03-06T07:24:44.836125+0000","last_became_peered":"2024-03-06T07:24:44.836125+0000","last_unstale":"2024-03-06T07:24:44.844117+0000","last_undegraded":"2024-03-06T07:24:44.844117+0000","last_fullsized":"2024-03-06T07:24:44.844117+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:11.466919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.834610+0000","last_change":"2024-03-06T07:24:44.834610+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.834610+0000","last_undegraded":"2024-03-06T07:24:44.834610+0000","last_fullsized":"2024-03-06T07:24:44.834610+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.846466+0000","last_change":"2024-03-06T07:24:44.841835+0000","last_active":"2024-03-06T07:24:44.846466+0000","last_peered":"2024-03-06T07:24:44.846466+0000","last_clean":"2024-03-06T07:24:44.846466+0000","last_became_active":"2024-03-06T07:24:44.841270+0000","last_became_peered":"2024-03-06T07:24:44.841270+0000","last_unstale":"2024-03-06T07:24:44.846466+0000","last_undegraded":"2024-03-06T07:24:44.846466+0000","last_fullsized":"2024-03-06T07:24:44.846466+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:58.149525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828625+0000","last_change":"2024-03-06T07:24:44.828625+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828625+0000","last_undegraded":"2024-03-06T07:24:44.828625+0000","last_fullsized":"2024-03-06T07:24:44.828625+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.842983+0000","last_change":"2024-03-06T07:24:44.841015+0000","last_active":"2024-03-06T07:24:44.842983+0000","last_peered":"2024-03-06T07:24:44.842983+0000","last_clean":"2024-03-06T07:24:44.842983+0000","last_became_active":"2024-03-06T07:24:44.840753+0000","last_became_peered":"2024-03-06T07:24:44.840753+0000","last_unstale":"2024-03-06T07:24:44.842983+0000","last_undegraded":"2024-03-06T07:24:44.842983+0000","last_fullsized":"2024-03-06T07:24:44.842983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:29.950110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.827852+0000","last_change":"2024-03-06T07:24:44.827852+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.827852+0000","last_undegraded":"2024-03-06T07:24:44.827852+0000","last_fullsized":"2024-03-06T07:24:44.827852+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.842503+0000","last_change":"2024-03-06T07:24:44.837932+0000","last_active":"2024-03-06T07:24:44.842503+0000","last_peered":"2024-03-06T07:24:44.842503+0000","last_clean":"2024-03-06T07:24:44.842503+0000","last_became_active":"2024-03-06T07:24:44.837625+0000","last_became_peered":"2024-03-06T07:24:44.837625+0000","last_unstale":"2024-03-06T07:24:44.842503+0000","last_undegraded":"2024-03-06T07:24:44.842503+0000","last_fullsized":"2024-03-06T07:24:44.842503+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:35.207003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.827869+0000","last_change":"2024-03-06T07:24:44.827869+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.827869+0000","last_undegraded":"2024-03-06T07:24:44.827869+0000","last_fullsized":"2024-03-06T07:24:44.827869+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.841164+0000","last_change":"2024-03-06T07:24:44.838096+0000","last_active":"2024-03-06T07:24:44.841164+0000","last_peered":"2024-03-06T07:24:44.841164+0000","last_clean":"2024-03-06T07:24:44.841164+0000","last_became_active":"2024-03-06T07:24:44.837560+0000","last_became_peered":"2024-03-06T07:24:44.837560+0000","last_unstale":"2024-03-06T07:24:44.841164+0000","last_undegraded":"2024-03-06T07:24:44.841164+0000","last_fullsized":"2024-03-06T07:24:44.841164+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:10.663190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.834047+0000","last_change":"2024-03-06T07:24:44.834047+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.834047+0000","last_undegraded":"2024-03-06T07:24:44.834047+0000","last_fullsized":"2024-03-06T07:24:44.834047+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.846538+0000","last_change":"2024-03-06T07:24:44.841831+0000","last_active":"2024-03-06T07:24:44.846538+0000","last_peered":"2024-03-06T07:24:44.846538+0000","last_clean":"2024-03-06T07:24:44.846538+0000","last_became_active":"2024-03-06T07:24:44.841166+0000","last_became_peered":"2024-03-06T07:24:44.841166+0000","last_unstale":"2024-03-06T07:24:44.846538+0000","last_undegraded":"2024-03-06T07:24:44.846538+0000","last_fullsized":"2024-03-06T07:24:44.846538+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:10.663190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.827544+0000","last_change":"2024-03-06T07:24:44.827544+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.827544+0000","last_undegraded":"2024-03-06T07:24:44.827544+0000","last_fullsized":"2024-03-06T07:24:44.827544+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.841553+0000","last_change":"2024-03-06T07:24:44.839000+0000","last_active":"2024-03-06T07:24:44.841553+0000","last_peered":"2024-03-06T07:24:44.841553+0000","last_clean":"2024-03-06T07:24:44.841553+0000","last_became_active":"2024-03-06T07:24:44.838728+0000","last_became_peered":"2024-03-06T07:24:44.838728+0000","last_unstale":"2024-03-06T07:24:44.841553+0000","last_undegraded":"2024-03-06T07:24:44.841553+0000","last_fullsized":"2024-03-06T07:24:44.841553+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:10.238622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.829175+0000","last_change":"2024-03-06T07:24:44.829175+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.829175+0000","last_undegraded":"2024-03-06T07:24:44.829175+0000","last_fullsized":"2024-03-06T07:24:44.829175+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.844656+0000","last_change":"2024-03-06T07:24:44.840821+0000","last_active":"2024-03-06T07:24:44.844656+0000","last_peered":"2024-03-06T07:24:44.844656+0000","last_clean":"2024-03-06T07:24:44.844656+0000","last_became_active":"2024-03-06T07:24:44.840543+0000","last_became_peered":"2024-03-06T07:24:44.840543+0000","last_unstale":"2024-03-06T07:24:44.844656+0000","last_undegraded":"2024-03-06T07:24:44.844656+0000","last_fullsized":"2024-03-06T07:24:44.844656+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:21.834862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.834126+0000","last_change":"2024-03-06T07:24:44.834126+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.834126+0000","last_undegraded":"2024-03-06T07:24:44.834126+0000","last_fullsized":"2024-03-06T07:24:44.834126+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.841166+0000","last_change":"2024-03-06T07:24:44.838578+0000","last_active":"2024-03-06T07:24:44.841166+0000","last_peered":"2024-03-06T07:24:44.841166+0000","last_clean":"2024-03-06T07:24:44.841166+0000","last_became_active":"2024-03-06T07:24:44.838264+0000","last_became_peered":"2024-03-06T07:24:44.838264+0000","last_unstale":"2024-03-06T07:24:44.841166+0000","last_undegraded":"2024-03-06T07:24:44.841166+0000","last_fullsized":"2024-03-06T07:24:44.841166+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:58.149525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.834714+0000","last_change":"2024-03-06T07:24:44.834714+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.834714+0000","last_undegraded":"2024-03-06T07:24:44.834714+0000","last_fullsized":"2024-03-06T07:24:44.834714+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.842127+0000","last_change":"2024-03-06T07:24:44.839270+0000","last_active":"2024-03-06T07:24:44.842127+0000","last_peered":"2024-03-06T07:24:44.842127+0000","last_clean":"2024-03-06T07:24:44.842127+0000","last_became_active":"2024-03-06T07:24:44.838971+0000","last_became_peered":"2024-03-06T07:24:44.838971+0000","last_unstale":"2024-03-06T07:24:44.842127+0000","last_undegraded":"2024-03-06T07:24:44.842127+0000","last_fullsized":"2024-03-06T07:24:44.842127+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:31.579386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.832593+0000","last_change":"2024-03-06T07:24:44.832593+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.832593+0000","last_undegraded":"2024-03-06T07:24:44.832593+0000","last_fullsized":"2024-03-06T07:24:44.832593+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.841198+0000","last_change":"2024-03-06T07:24:44.837121+0000","last_active":"2024-03-06T07:24:44.841198+0000","last_peered":"2024-03-06T07:24:44.841198+0000","last_clean":"2024-03-06T07:24:44.841198+0000","last_became_active":"2024-03-06T07:24:44.836330+0000","last_became_peered":"2024-03-06T07:24:44.836330+0000","last_unstale":"2024-03-06T07:24:44.841198+0000","last_undegraded":"2024-03-06T07:24:44.841198+0000","last_fullsized":"2024-03-06T07:24:44.841198+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:55.061981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.827671+0000","last_change":"2024-03-06T07:24:44.827671+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.827671+0000","last_undegraded":"2024-03-06T07:24:44.827671+0000","last_fullsized":"2024-03-06T07:24:44.827671+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.842481+0000","last_change":"2024-03-06T07:24:44.837610+0000","last_active":"2024-03-06T07:24:44.842481+0000","last_peered":"2024-03-06T07:24:44.842481+0000","last_clean":"2024-03-06T07:24:44.842481+0000","last_became_active":"2024-03-06T07:24:44.837262+0000","last_became_peered":"2024-03-06T07:24:44.837262+0000","last_unstale":"2024-03-06T07:24:44.842481+0000","last_undegraded":"2024-03-06T07:24:44.842481+0000","last_fullsized":"2024-03-06T07:24:44.842481+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:19.339152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:24:44.828378+0000","last_change":"2024-03-06T07:24:44.828378+0000","last_active":"2024-03-06T07:24:44.815470+0000","last_peered":"2024-03-06T07:24:44.815470+0000","last_clean":"2024-03-06T07:24:44.815470+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:44.828378+0000","last_undegraded":"2024-03-06T07:24:44.828378+0000","last_fullsized":"2024-03-06T07:24:44.828378+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.846319+0000","last_change":"2024-03-06T07:24:44.844868+0000","last_active":"2024-03-06T07:24:44.846319+0000","last_peered":"2024-03-06T07:24:44.846319+0000","last_clean":"2024-03-06T07:24:44.846319+0000","last_became_active":"2024-03-06T07:24:44.844594+0000","last_became_peered":"2024-03-06T07:24:44.844594+0000","last_unstale":"2024-03-06T07:24:44.846319+0000","last_undegraded":"2024-03-06T07:24:44.846319+0000","last_fullsized":"2024-03-06T07:24:44.846319+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:11.466919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":215,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:24:44.829630+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:24:44.829630+0000","last_peered":"2024-03-06T07:24:44.829630+0000","last_clean":"2024-03-06T07:24:44.829630+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:24:44.829630+0000","last_undegraded":"2024-03-06T07:24:44.829630+0000","last_fullsized":"2024-03-06T07:24:44.829630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":797087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":516096,"data_stored":206719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":790820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:49.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:50.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.133+0000 7ffaf086b640 1 Processor -- start 2024-03-06T07:24:50.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.133+0000 7ffaf086b640 1 -- start start 2024-03-06T07:24:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.134+0000 7ffaf086b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffaec0d3400 0x7ffaec0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.134+0000 7ffaf086b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0ce610 con 0x7ffaec0b2b50 2024-03-06T07:24:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.134+0000 7ffaf086b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0d2920 con 0x7ffaec071540 2024-03-06T07:24:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.134+0000 7ffaf086b640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec0d2f70 con 0x7ffaec0d3400 2024-03-06T07:24:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.134+0000 7ffae9d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffaec0d3400 0x7ffaec0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.134+0000 7ffaead9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffaec0b2b50 0x7ffaec0b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56210/0 (socket says 172.21.15.27:56210) 2024-03-06T07:24:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.134+0000 7ffaead9d640 1 -- 172.21.15.27:0/2545064797 learned_addr learned my addr 172.21.15.27:0/2545064797 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.135+0000 7ffae9d9b640 1 -- 172.21.15.27:0/2545064797 >> v1:172.21.15.69:6789/0 conn(0x7ffaec071540 legacy=0x7ffaec071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.135+0000 7ffae9d9b640 1 -- 172.21.15.27:0/2545064797 >> v1:172.21.15.27:6789/0 conn(0x7ffaec0b2b50 legacy=0x7ffaec0b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.135+0000 7ffae9d9b640 1 -- 172.21.15.27:0/2545064797 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0d2c00 con 0x7ffaec0d3400 2024-03-06T07:24:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.135+0000 7ffae959a640 1 -- 172.21.15.27:0/2545064797 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971730223 0 0) 0x7ffaec0ce610 con 0x7ffaec0b2b50 2024-03-06T07:24:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.135+0000 7ffae9d9b640 1 --2- 172.21.15.27:0/2545064797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0d3400 0x7ffaec0b30a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ffad40097a0 tx=0x7ffad402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846ffbfb16529af2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffae959a640 1 -- 172.21.15.27:0/2545064797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4034070 con 0x7ffaec0d3400 2024-03-06T07:24:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffae959a640 1 -- 172.21.15.27:0/2545064797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad4030610 con 0x7ffaec0d3400 2024-03-06T07:24:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffae959a640 1 -- 172.21.15.27:0/2545064797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad40309e0 con 0x7ffaec0d3400 2024-03-06T07:24:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffaf086b640 1 -- 172.21.15.27:0/2545064797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0d3400 msgr2=0x7ffaec0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffaf086b640 1 --2- 172.21.15.27:0/2545064797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0d3400 0x7ffaec0b30a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ffad40097a0 tx=0x7ffad402b960 comp rx=0 tx=0).stop 2024-03-06T07:24:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffaf086b640 1 -- 172.21.15.27:0/2545064797 shutdown_connections 2024-03-06T07:24:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffaf086b640 1 --2- 172.21.15.27:0/2545064797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0d3400 0x7ffaec0b30a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffaf086b640 1 -- 172.21.15.27:0/2545064797 >> 172.21.15.27:0/2545064797 conn(0x7ffaec0d3ca0 msgr2=0x7ffaec0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffaf086b640 1 -- 172.21.15.27:0/2545064797 shutdown_connections 2024-03-06T07:24:50.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.136+0000 7ffaf086b640 1 -- 172.21.15.27:0/2545064797 wait complete. 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffaf086b640 1 Processor -- start 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffaf086b640 1 -- start start 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffaf086b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffaec071540 0x7ffaec0b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffaf086b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0b2b50 0x7ffaec0c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffaf086b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaec0d3400 0x7ffaec0b59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffaf086b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec15e370 con 0x7ffaec0b2b50 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffaf086b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0d4be0 con 0x7ffaec0d3400 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffaf086b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0b6280 con 0x7ffaec071540 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffae9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0b2b50 0x7ffaec0c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.138+0000 7ffae9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0b2b50 0x7ffaec0c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46176/0 (socket says 172.21.15.27:46176) 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae9d9b640 1 -- 172.21.15.27:0/1406998177 learned_addr learned my addr 172.21.15.27:0/1406998177 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffaea59c640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffaec071540 0x7ffaec0b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffaead9d640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaec0d3400 0x7ffaec0b59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae9d9b640 1 -- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffaec071540 msgr2=0x7ffaec0b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae9d9b640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffaec071540 0x7ffaec0b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae9d9b640 1 -- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaec0d3400 msgr2=0x7ffaec0b59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae9d9b640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaec0d3400 0x7ffaec0b59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae9d9b640 1 -- 172.21.15.27:0/1406998177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0b6560 con 0x7ffaec0b2b50 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae9d9b640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0b2b50 0x7ffaec0c7fc0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ffad4000c00 tx=0x7ffad40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae2ffd640 1 -- 172.21.15.27:0/1406998177 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4047070 con 0x7ffaec0b2b50 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae2ffd640 1 -- 172.21.15.27:0/1406998177 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad4004030 con 0x7ffaec0b2b50 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffae2ffd640 1 -- 172.21.15.27:0/1406998177 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4034040 con 0x7ffaec0b2b50 2024-03-06T07:24:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec0c74e0 con 0x7ffaec0b2b50 2024-03-06T07:24:50.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.139+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec0c7aa0 con 0x7ffaec0b2b50 2024-03-06T07:24:50.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.140+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec0c7ca0 con 0x7ffaec0b2b50 2024-03-06T07:24:50.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.141+0000 7ffae2ffd640 1 -- 172.21.15.27:0/1406998177 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffad404a070 con 0x7ffaec0b2b50 2024-03-06T07:24:50.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.141+0000 7ffae2ffd640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffacc06b4d0 0x7ffacc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.141+0000 7ffae2ffd640 1 -- 172.21.15.27:0/1406998177 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffad40abae0 con 0x7ffaec0b2b50 2024-03-06T07:24:50.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.142+0000 7ffaea59c640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffacc06b4d0 0x7ffacc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.142+0000 7ffaea59c640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffacc06b4d0 0x7ffacc06d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ffae4000f60 tx=0x7ffae4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.146+0000 7ffae2ffd640 1 -- 172.21.15.27:0/1406998177 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffaec0c7ca0 con 0x7ffaec0b2b50 2024-03-06T07:24:50.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.325+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffaec068700 con 0x7ffacc06b4d0 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.332+0000 7ffae2ffd640 1 -- 172.21.15.27:0/1406998177 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251960 (secure 0 0 0) 0x7ffaec068700 con 0x7ffacc06b4d0 2024-03-06T07:24:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:24:50.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffacc06b4d0 msgr2=0x7ffacc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffacc06b4d0 0x7ffacc06d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ffae4000f60 tx=0x7ffae4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0b2b50 msgr2=0x7ffaec0c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0b2b50 0x7ffaec0c7fc0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ffad4000c00 tx=0x7ffad40046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 shutdown_connections 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ffacc06b4d0 0x7ffacc06d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaec0d3400 0x7ffaec0b59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaec0b2b50 0x7ffaec0c7fc0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 --2- 172.21.15.27:0/1406998177 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ffaec071540 0x7ffaec0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.334+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 >> 172.21.15.27:0/1406998177 conn(0x7ffaec0d3ca0 msgr2=0x7ffaec0ceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.335+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 shutdown_connections 2024-03-06T07:24:50.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.335+0000 7ffaf086b640 1 -- 172.21.15.27:0/1406998177 wait complete. 2024-03-06T07:24:50.348 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-03-06T07:24:50.015195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254916,"kb_used_data":5448,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719804,"statfs":{"total":773094113280,"available":772833079296,"internally_reserved":0,"allocated":5578752,"data_stored":2943908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001542"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837327+0000","last_change":"2024-03-06T07:24:45.835003+0000","last_active":"2024-03-06T07:24:46.837327+0000","last_peered":"2024-03-06T07:24:46.837327+0000","last_clean":"2024-03-06T07:24:46.837327+0000","last_became_active":"2024-03-06T07:24:45.834676+0000","last_became_peered":"2024-03-06T07:24:45.834676+0000","last_unstale":"2024-03-06T07:24:46.837327+0000","last_undegraded":"2024-03-06T07:24:46.837327+0000","last_fullsized":"2024-03-06T07:24:46.837327+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:50.725061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196906+0000","last_change":"2024-03-06T07:24:45.830762+0000","last_active":"2024-03-06T07:24:48.196906+0000","last_peered":"2024-03-06T07:24:48.196906+0000","last_clean":"2024-03-06T07:24:48.196906+0000","last_became_active":"2024-03-06T07:24:45.830477+0000","last_became_peered":"2024-03-06T07:24:45.830477+0000","last_unstale":"2024-03-06T07:24:48.196906+0000","last_undegraded":"2024-03-06T07:24:48.196906+0000","last_fullsized":"2024-03-06T07:24:48.196906+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:08.598965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838071+0000","last_change":"2024-03-06T07:24:45.845515+0000","last_active":"2024-03-06T07:24:46.838071+0000","last_peered":"2024-03-06T07:24:46.838071+0000","last_clean":"2024-03-06T07:24:46.838071+0000","last_became_active":"2024-03-06T07:24:45.845181+0000","last_became_peered":"2024-03-06T07:24:45.845181+0000","last_unstale":"2024-03-06T07:24:46.838071+0000","last_undegraded":"2024-03-06T07:24:46.838071+0000","last_fullsized":"2024-03-06T07:24:46.838071+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:11.240901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196484+0000","last_change":"2024-03-06T07:24:45.841947+0000","last_active":"2024-03-06T07:24:48.196484+0000","last_peered":"2024-03-06T07:24:48.196484+0000","last_clean":"2024-03-06T07:24:48.196484+0000","last_became_active":"2024-03-06T07:24:45.841337+0000","last_became_peered":"2024-03-06T07:24:45.841337+0000","last_unstale":"2024-03-06T07:24:48.196484+0000","last_undegraded":"2024-03-06T07:24:48.196484+0000","last_fullsized":"2024-03-06T07:24:48.196484+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:04.763038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196778+0000","last_change":"2024-03-06T07:24:45.831223+0000","last_active":"2024-03-06T07:24:48.196778+0000","last_peered":"2024-03-06T07:24:48.196778+0000","last_clean":"2024-03-06T07:24:48.196778+0000","last_became_active":"2024-03-06T07:24:45.830925+0000","last_became_peered":"2024-03-06T07:24:45.830925+0000","last_unstale":"2024-03-06T07:24:48.196778+0000","last_undegraded":"2024-03-06T07:24:48.196778+0000","last_fullsized":"2024-03-06T07:24:48.196778+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:28.055174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838694+0000","last_change":"2024-03-06T07:24:45.833374+0000","last_active":"2024-03-06T07:24:46.838694+0000","last_peered":"2024-03-06T07:24:46.838694+0000","last_clean":"2024-03-06T07:24:46.838694+0000","last_became_active":"2024-03-06T07:24:45.833120+0000","last_became_peered":"2024-03-06T07:24:45.833120+0000","last_unstale":"2024-03-06T07:24:46.838694+0000","last_undegraded":"2024-03-06T07:24:46.838694+0000","last_fullsized":"2024-03-06T07:24:46.838694+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:10.244273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.836891+0000","last_change":"2024-03-06T07:24:45.842234+0000","last_active":"2024-03-06T07:24:46.836891+0000","last_peered":"2024-03-06T07:24:46.836891+0000","last_clean":"2024-03-06T07:24:46.836891+0000","last_became_active":"2024-03-06T07:24:45.841575+0000","last_became_peered":"2024-03-06T07:24:45.841575+0000","last_unstale":"2024-03-06T07:24:46.836891+0000","last_undegraded":"2024-03-06T07:24:46.836891+0000","last_fullsized":"2024-03-06T07:24:46.836891+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:11.318349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196104+0000","last_change":"2024-03-06T07:24:45.846279+0000","last_active":"2024-03-06T07:24:48.196104+0000","last_peered":"2024-03-06T07:24:48.196104+0000","last_clean":"2024-03-06T07:24:48.196104+0000","last_became_active":"2024-03-06T07:24:45.845930+0000","last_became_peered":"2024-03-06T07:24:45.845930+0000","last_unstale":"2024-03-06T07:24:48.196104+0000","last_undegraded":"2024-03-06T07:24:48.196104+0000","last_fullsized":"2024-03-06T07:24:48.196104+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:11.240901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196619+0000","last_change":"2024-03-06T07:24:45.833220+0000","last_active":"2024-03-06T07:24:48.196619+0000","last_peered":"2024-03-06T07:24:48.196619+0000","last_clean":"2024-03-06T07:24:48.196619+0000","last_became_active":"2024-03-06T07:24:45.832944+0000","last_became_peered":"2024-03-06T07:24:45.832944+0000","last_unstale":"2024-03-06T07:24:48.196619+0000","last_undegraded":"2024-03-06T07:24:48.196619+0000","last_fullsized":"2024-03-06T07:24:48.196619+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:17.205173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196623+0000","last_change":"2024-03-06T07:24:45.841143+0000","last_active":"2024-03-06T07:24:48.196623+0000","last_peered":"2024-03-06T07:24:48.196623+0000","last_clean":"2024-03-06T07:24:48.196623+0000","last_became_active":"2024-03-06T07:24:45.840880+0000","last_became_peered":"2024-03-06T07:24:45.840880+0000","last_unstale":"2024-03-06T07:24:48.196623+0000","last_undegraded":"2024-03-06T07:24:48.196623+0000","last_fullsized":"2024-03-06T07:24:48.196623+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:27.202856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196367+0000","last_change":"2024-03-06T07:24:45.834861+0000","last_active":"2024-03-06T07:24:48.196367+0000","last_peered":"2024-03-06T07:24:48.196367+0000","last_clean":"2024-03-06T07:24:48.196367+0000","last_became_active":"2024-03-06T07:24:45.832753+0000","last_became_peered":"2024-03-06T07:24:45.832753+0000","last_unstale":"2024-03-06T07:24:48.196367+0000","last_undegraded":"2024-03-06T07:24:48.196367+0000","last_fullsized":"2024-03-06T07:24:48.196367+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:30.931860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196969+0000","last_change":"2024-03-06T07:24:45.837841+0000","last_active":"2024-03-06T07:24:48.196969+0000","last_peered":"2024-03-06T07:24:48.196969+0000","last_clean":"2024-03-06T07:24:48.196969+0000","last_became_active":"2024-03-06T07:24:45.836777+0000","last_became_peered":"2024-03-06T07:24:45.836777+0000","last_unstale":"2024-03-06T07:24:48.196969+0000","last_undegraded":"2024-03-06T07:24:48.196969+0000","last_fullsized":"2024-03-06T07:24:48.196969+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:00.194744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.835877+0000","last_change":"2024-03-06T07:24:45.842978+0000","last_active":"2024-03-06T07:24:46.835877+0000","last_peered":"2024-03-06T07:24:46.835877+0000","last_clean":"2024-03-06T07:24:46.835877+0000","last_became_active":"2024-03-06T07:24:45.842652+0000","last_became_peered":"2024-03-06T07:24:45.842652+0000","last_unstale":"2024-03-06T07:24:46.835877+0000","last_undegraded":"2024-03-06T07:24:46.835877+0000","last_fullsized":"2024-03-06T07:24:46.835877+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:26.801115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196466+0000","last_change":"2024-03-06T07:24:45.829493+0000","last_active":"2024-03-06T07:24:48.196466+0000","last_peered":"2024-03-06T07:24:48.196466+0000","last_clean":"2024-03-06T07:24:48.196466+0000","last_became_active":"2024-03-06T07:24:45.829125+0000","last_became_peered":"2024-03-06T07:24:45.829125+0000","last_unstale":"2024-03-06T07:24:48.196466+0000","last_undegraded":"2024-03-06T07:24:48.196466+0000","last_fullsized":"2024-03-06T07:24:48.196466+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:39.565532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838643+0000","last_change":"2024-03-06T07:24:45.830582+0000","last_active":"2024-03-06T07:24:46.838643+0000","last_peered":"2024-03-06T07:24:46.838643+0000","last_clean":"2024-03-06T07:24:46.838643+0000","last_became_active":"2024-03-06T07:24:45.829747+0000","last_became_peered":"2024-03-06T07:24:45.829747+0000","last_unstale":"2024-03-06T07:24:46.838643+0000","last_undegraded":"2024-03-06T07:24:46.838643+0000","last_fullsized":"2024-03-06T07:24:46.838643+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:28.935085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196937+0000","last_change":"2024-03-06T07:24:45.848421+0000","last_active":"2024-03-06T07:24:48.196937+0000","last_peered":"2024-03-06T07:24:48.196937+0000","last_clean":"2024-03-06T07:24:48.196937+0000","last_became_active":"2024-03-06T07:24:45.847813+0000","last_became_peered":"2024-03-06T07:24:45.847813+0000","last_unstale":"2024-03-06T07:24:48.196937+0000","last_undegraded":"2024-03-06T07:24:48.196937+0000","last_fullsized":"2024-03-06T07:24:48.196937+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:24.222183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838214+0000","last_change":"2024-03-06T07:24:45.834126+0000","last_active":"2024-03-06T07:24:46.838214+0000","last_peered":"2024-03-06T07:24:46.838214+0000","last_clean":"2024-03-06T07:24:46.838214+0000","last_became_active":"2024-03-06T07:24:45.833787+0000","last_became_peered":"2024-03-06T07:24:45.833787+0000","last_unstale":"2024-03-06T07:24:46.838214+0000","last_undegraded":"2024-03-06T07:24:46.838214+0000","last_fullsized":"2024-03-06T07:24:46.838214+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:49.045241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838239+0000","last_change":"2024-03-06T07:24:45.843875+0000","last_active":"2024-03-06T07:24:46.838239+0000","last_peered":"2024-03-06T07:24:46.838239+0000","last_clean":"2024-03-06T07:24:46.838239+0000","last_became_active":"2024-03-06T07:24:45.843591+0000","last_became_peered":"2024-03-06T07:24:45.843591+0000","last_unstale":"2024-03-06T07:24:46.838239+0000","last_undegraded":"2024-03-06T07:24:46.838239+0000","last_fullsized":"2024-03-06T07:24:46.838239+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:39.565532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197713+0000","last_change":"2024-03-06T07:24:45.847904+0000","last_active":"2024-03-06T07:24:48.197713+0000","last_peered":"2024-03-06T07:24:48.197713+0000","last_clean":"2024-03-06T07:24:48.197713+0000","last_became_active":"2024-03-06T07:24:45.847561+0000","last_became_peered":"2024-03-06T07:24:45.847561+0000","last_unstale":"2024-03-06T07:24:48.197713+0000","last_undegraded":"2024-03-06T07:24:48.197713+0000","last_fullsized":"2024-03-06T07:24:48.197713+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:12.469198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.836106+0000","last_change":"2024-03-06T07:24:45.844315+0000","last_active":"2024-03-06T07:24:46.836106+0000","last_peered":"2024-03-06T07:24:46.836106+0000","last_clean":"2024-03-06T07:24:46.836106+0000","last_became_active":"2024-03-06T07:24:45.844003+0000","last_became_peered":"2024-03-06T07:24:45.844003+0000","last_unstale":"2024-03-06T07:24:46.836106+0000","last_undegraded":"2024-03-06T07:24:46.836106+0000","last_fullsized":"2024-03-06T07:24:46.836106+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:53.059921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838888+0000","last_change":"2024-03-06T07:24:45.839721+0000","last_active":"2024-03-06T07:24:46.838888+0000","last_peered":"2024-03-06T07:24:46.838888+0000","last_clean":"2024-03-06T07:24:46.838888+0000","last_became_active":"2024-03-06T07:24:45.839422+0000","last_became_peered":"2024-03-06T07:24:45.839422+0000","last_unstale":"2024-03-06T07:24:46.838888+0000","last_undegraded":"2024-03-06T07:24:46.838888+0000","last_fullsized":"2024-03-06T07:24:46.838888+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:42.725731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196738+0000","last_change":"2024-03-06T07:24:45.842154+0000","last_active":"2024-03-06T07:24:48.196738+0000","last_peered":"2024-03-06T07:24:48.196738+0000","last_clean":"2024-03-06T07:24:48.196738+0000","last_became_active":"2024-03-06T07:24:45.841898+0000","last_became_peered":"2024-03-06T07:24:45.841898+0000","last_unstale":"2024-03-06T07:24:48.196738+0000","last_undegraded":"2024-03-06T07:24:48.196738+0000","last_fullsized":"2024-03-06T07:24:48.196738+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:08.598965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197569+0000","last_change":"2024-03-06T07:24:45.841290+0000","last_active":"2024-03-06T07:24:48.197569+0000","last_peered":"2024-03-06T07:24:48.197569+0000","last_clean":"2024-03-06T07:24:48.197569+0000","last_became_active":"2024-03-06T07:24:45.841013+0000","last_became_peered":"2024-03-06T07:24:45.841013+0000","last_unstale":"2024-03-06T07:24:48.197569+0000","last_undegraded":"2024-03-06T07:24:48.197569+0000","last_fullsized":"2024-03-06T07:24:48.197569+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:58.685980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197242+0000","last_change":"2024-03-06T07:24:45.840332+0000","last_active":"2024-03-06T07:24:48.197242+0000","last_peered":"2024-03-06T07:24:48.197242+0000","last_clean":"2024-03-06T07:24:48.197242+0000","last_became_active":"2024-03-06T07:24:45.840008+0000","last_became_peered":"2024-03-06T07:24:45.840008+0000","last_unstale":"2024-03-06T07:24:48.197242+0000","last_undegraded":"2024-03-06T07:24:48.197242+0000","last_fullsized":"2024-03-06T07:24:48.197242+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:59.011592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197783+0000","last_change":"2024-03-06T07:24:45.841945+0000","last_active":"2024-03-06T07:24:48.197783+0000","last_peered":"2024-03-06T07:24:48.197783+0000","last_clean":"2024-03-06T07:24:48.197783+0000","last_became_active":"2024-03-06T07:24:45.841173+0000","last_became_peered":"2024-03-06T07:24:45.841173+0000","last_unstale":"2024-03-06T07:24:48.197783+0000","last_undegraded":"2024-03-06T07:24:48.197783+0000","last_fullsized":"2024-03-06T07:24:48.197783+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:53.837226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.835970+0000","last_change":"2024-03-06T07:24:45.841797+0000","last_active":"2024-03-06T07:24:46.835970+0000","last_peered":"2024-03-06T07:24:46.835970+0000","last_clean":"2024-03-06T07:24:46.835970+0000","last_became_active":"2024-03-06T07:24:45.841450+0000","last_became_peered":"2024-03-06T07:24:45.841450+0000","last_unstale":"2024-03-06T07:24:46.835970+0000","last_undegraded":"2024-03-06T07:24:46.835970+0000","last_fullsized":"2024-03-06T07:24:46.835970+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:43.333534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838106+0000","last_change":"2024-03-06T07:24:45.834266+0000","last_active":"2024-03-06T07:24:46.838106+0000","last_peered":"2024-03-06T07:24:46.838106+0000","last_clean":"2024-03-06T07:24:46.838106+0000","last_became_active":"2024-03-06T07:24:45.833971+0000","last_became_peered":"2024-03-06T07:24:45.833971+0000","last_unstale":"2024-03-06T07:24:46.838106+0000","last_undegraded":"2024-03-06T07:24:46.838106+0000","last_fullsized":"2024-03-06T07:24:46.838106+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:15.801075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196872+0000","last_change":"2024-03-06T07:24:45.836820+0000","last_active":"2024-03-06T07:24:48.196872+0000","last_peered":"2024-03-06T07:24:48.196872+0000","last_clean":"2024-03-06T07:24:48.196872+0000","last_became_active":"2024-03-06T07:24:45.836529+0000","last_became_peered":"2024-03-06T07:24:45.836529+0000","last_unstale":"2024-03-06T07:24:48.196872+0000","last_undegraded":"2024-03-06T07:24:48.196872+0000","last_fullsized":"2024-03-06T07:24:48.196872+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:19.847774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197408+0000","last_change":"2024-03-06T07:24:45.843366+0000","last_active":"2024-03-06T07:24:48.197408+0000","last_peered":"2024-03-06T07:24:48.197408+0000","last_clean":"2024-03-06T07:24:48.197408+0000","last_became_active":"2024-03-06T07:24:45.843101+0000","last_became_peered":"2024-03-06T07:24:45.843101+0000","last_unstale":"2024-03-06T07:24:48.197408+0000","last_undegraded":"2024-03-06T07:24:48.197408+0000","last_fullsized":"2024-03-06T07:24:48.197408+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:03.672707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837899+0000","last_change":"2024-03-06T07:24:45.836856+0000","last_active":"2024-03-06T07:24:46.837899+0000","last_peered":"2024-03-06T07:24:46.837899+0000","last_clean":"2024-03-06T07:24:46.837899+0000","last_became_active":"2024-03-06T07:24:45.836576+0000","last_became_peered":"2024-03-06T07:24:45.836576+0000","last_unstale":"2024-03-06T07:24:46.837899+0000","last_undegraded":"2024-03-06T07:24:46.837899+0000","last_fullsized":"2024-03-06T07:24:46.837899+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:11.665469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838715+0000","last_change":"2024-03-06T07:24:45.832798+0000","last_active":"2024-03-06T07:24:46.838715+0000","last_peered":"2024-03-06T07:24:46.838715+0000","last_clean":"2024-03-06T07:24:46.838715+0000","last_became_active":"2024-03-06T07:24:45.832525+0000","last_became_peered":"2024-03-06T07:24:45.832525+0000","last_unstale":"2024-03-06T07:24:46.838715+0000","last_undegraded":"2024-03-06T07:24:46.838715+0000","last_fullsized":"2024-03-06T07:24:46.838715+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:32.661499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.835738+0000","last_change":"2024-03-06T07:24:45.843993+0000","last_active":"2024-03-06T07:24:46.835738+0000","last_peered":"2024-03-06T07:24:46.835738+0000","last_clean":"2024-03-06T07:24:46.835738+0000","last_became_active":"2024-03-06T07:24:45.843651+0000","last_became_peered":"2024-03-06T07:24:45.843651+0000","last_unstale":"2024-03-06T07:24:46.835738+0000","last_undegraded":"2024-03-06T07:24:46.835738+0000","last_fullsized":"2024-03-06T07:24:46.835738+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:32.942690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:47.393160+0000","last_change":"2024-03-06T07:24:45.839202+0000","last_active":"2024-03-06T07:24:47.393160+0000","last_peered":"2024-03-06T07:24:47.393160+0000","last_clean":"2024-03-06T07:24:47.393160+0000","last_became_active":"2024-03-06T07:24:45.838890+0000","last_became_peered":"2024-03-06T07:24:45.838890+0000","last_unstale":"2024-03-06T07:24:47.393160+0000","last_undegraded":"2024-03-06T07:24:47.393160+0000","last_fullsized":"2024-03-06T07:24:47.393160+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:58.685980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":18,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.839208+0000","last_change":"2024-03-06T07:24:44.839049+0000","last_active":"2024-03-06T07:24:46.839208+0000","last_peered":"2024-03-06T07:24:46.839208+0000","last_clean":"2024-03-06T07:24:46.839208+0000","last_became_active":"2024-03-06T07:24:44.838225+0000","last_became_peered":"2024-03-06T07:24:44.838225+0000","last_unstale":"2024-03-06T07:24:46.839208+0000","last_undegraded":"2024-03-06T07:24:46.839208+0000","last_fullsized":"2024-03-06T07:24:46.839208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:38.563253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197810+0000","last_change":"2024-03-06T07:24:45.840454+0000","last_active":"2024-03-06T07:24:48.197810+0000","last_peered":"2024-03-06T07:24:48.197810+0000","last_clean":"2024-03-06T07:24:48.197810+0000","last_became_active":"2024-03-06T07:24:45.839939+0000","last_became_peered":"2024-03-06T07:24:45.839939+0000","last_unstale":"2024-03-06T07:24:48.197810+0000","last_undegraded":"2024-03-06T07:24:48.197810+0000","last_fullsized":"2024-03-06T07:24:48.197810+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:11.665469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838352+0000","last_change":"2024-03-06T07:24:44.837067+0000","last_active":"2024-03-06T07:24:46.838352+0000","last_peered":"2024-03-06T07:24:46.838352+0000","last_clean":"2024-03-06T07:24:46.838352+0000","last_became_active":"2024-03-06T07:24:44.836744+0000","last_became_peered":"2024-03-06T07:24:44.836744+0000","last_unstale":"2024-03-06T07:24:46.838352+0000","last_undegraded":"2024-03-06T07:24:46.838352+0000","last_fullsized":"2024-03-06T07:24:46.838352+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:25.798836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197170+0000","last_change":"2024-03-06T07:24:45.826375+0000","last_active":"2024-03-06T07:24:48.197170+0000","last_peered":"2024-03-06T07:24:48.197170+0000","last_clean":"2024-03-06T07:24:48.197170+0000","last_became_active":"2024-03-06T07:24:45.826102+0000","last_became_peered":"2024-03-06T07:24:45.826102+0000","last_unstale":"2024-03-06T07:24:48.197170+0000","last_undegraded":"2024-03-06T07:24:48.197170+0000","last_fullsized":"2024-03-06T07:24:48.197170+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:48.321964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:47.393217+0000","last_change":"2024-03-06T07:24:44.840722+0000","last_active":"2024-03-06T07:24:47.393217+0000","last_peered":"2024-03-06T07:24:47.393217+0000","last_clean":"2024-03-06T07:24:47.393217+0000","last_became_active":"2024-03-06T07:24:44.840432+0000","last_became_peered":"2024-03-06T07:24:44.840432+0000","last_unstale":"2024-03-06T07:24:47.393217+0000","last_undegraded":"2024-03-06T07:24:47.393217+0000","last_fullsized":"2024-03-06T07:24:47.393217+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:06.809576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:47.393211+0000","last_change":"2024-03-06T07:24:45.840778+0000","last_active":"2024-03-06T07:24:47.393211+0000","last_peered":"2024-03-06T07:24:47.393211+0000","last_clean":"2024-03-06T07:24:47.393211+0000","last_became_active":"2024-03-06T07:24:45.840462+0000","last_became_peered":"2024-03-06T07:24:45.840462+0000","last_unstale":"2024-03-06T07:24:47.393211+0000","last_undegraded":"2024-03-06T07:24:47.393211+0000","last_fullsized":"2024-03-06T07:24:47.393211+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:30.952388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837060+0000","last_change":"2024-03-06T07:24:44.840470+0000","last_active":"2024-03-06T07:24:46.837060+0000","last_peered":"2024-03-06T07:24:46.837060+0000","last_clean":"2024-03-06T07:24:46.837060+0000","last_became_active":"2024-03-06T07:24:44.840199+0000","last_became_peered":"2024-03-06T07:24:44.840199+0000","last_unstale":"2024-03-06T07:24:46.837060+0000","last_undegraded":"2024-03-06T07:24:46.837060+0000","last_fullsized":"2024-03-06T07:24:46.837060+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:16.202894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837931+0000","last_change":"2024-03-06T07:24:45.840679+0000","last_active":"2024-03-06T07:24:46.837931+0000","last_peered":"2024-03-06T07:24:46.837931+0000","last_clean":"2024-03-06T07:24:46.837931+0000","last_became_active":"2024-03-06T07:24:45.840360+0000","last_became_peered":"2024-03-06T07:24:45.840360+0000","last_unstale":"2024-03-06T07:24:46.837931+0000","last_undegraded":"2024-03-06T07:24:46.837931+0000","last_fullsized":"2024-03-06T07:24:46.837931+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:59.151804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838878+0000","last_change":"2024-03-06T07:24:44.841918+0000","last_active":"2024-03-06T07:24:46.838878+0000","last_peered":"2024-03-06T07:24:46.838878+0000","last_clean":"2024-03-06T07:24:46.838878+0000","last_became_active":"2024-03-06T07:24:44.841583+0000","last_became_peered":"2024-03-06T07:24:44.841583+0000","last_unstale":"2024-03-06T07:24:46.838878+0000","last_undegraded":"2024-03-06T07:24:46.838878+0000","last_fullsized":"2024-03-06T07:24:46.838878+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:39.286721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196810+0000","last_change":"2024-03-06T07:24:45.837696+0000","last_active":"2024-03-06T07:24:48.196810+0000","last_peered":"2024-03-06T07:24:48.196810+0000","last_clean":"2024-03-06T07:24:48.196810+0000","last_became_active":"2024-03-06T07:24:45.837252+0000","last_became_peered":"2024-03-06T07:24:45.837252+0000","last_unstale":"2024-03-06T07:24:48.196810+0000","last_undegraded":"2024-03-06T07:24:48.196810+0000","last_fullsized":"2024-03-06T07:24:48.196810+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:28.055174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838037+0000","last_change":"2024-03-06T07:24:44.837214+0000","last_active":"2024-03-06T07:24:46.838037+0000","last_peered":"2024-03-06T07:24:46.838037+0000","last_clean":"2024-03-06T07:24:46.838037+0000","last_became_active":"2024-03-06T07:24:44.836884+0000","last_became_peered":"2024-03-06T07:24:44.836884+0000","last_unstale":"2024-03-06T07:24:46.838037+0000","last_undegraded":"2024-03-06T07:24:46.838037+0000","last_fullsized":"2024-03-06T07:24:46.838037+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:42.331255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196210+0000","last_change":"2024-03-06T07:24:45.844997+0000","last_active":"2024-03-06T07:24:48.196210+0000","last_peered":"2024-03-06T07:24:48.196210+0000","last_clean":"2024-03-06T07:24:48.196210+0000","last_became_active":"2024-03-06T07:24:45.844677+0000","last_became_peered":"2024-03-06T07:24:45.844677+0000","last_unstale":"2024-03-06T07:24:48.196210+0000","last_undegraded":"2024-03-06T07:24:48.196210+0000","last_fullsized":"2024-03-06T07:24:48.196210+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:49.045241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196596+0000","last_change":"2024-03-06T07:24:44.842872+0000","last_active":"2024-03-06T07:24:48.196596+0000","last_peered":"2024-03-06T07:24:48.196596+0000","last_clean":"2024-03-06T07:24:48.196596+0000","last_became_active":"2024-03-06T07:24:44.842566+0000","last_became_peered":"2024-03-06T07:24:44.842566+0000","last_unstale":"2024-03-06T07:24:48.196596+0000","last_undegraded":"2024-03-06T07:24:48.196596+0000","last_fullsized":"2024-03-06T07:24:48.196596+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:52.834947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197581+0000","last_change":"2024-03-06T07:24:45.840252+0000","last_active":"2024-03-06T07:24:48.197581+0000","last_peered":"2024-03-06T07:24:48.197581+0000","last_clean":"2024-03-06T07:24:48.197581+0000","last_became_active":"2024-03-06T07:24:45.839915+0000","last_became_peered":"2024-03-06T07:24:45.839915+0000","last_unstale":"2024-03-06T07:24:48.197581+0000","last_undegraded":"2024-03-06T07:24:48.197581+0000","last_fullsized":"2024-03-06T07:24:48.197581+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:30.952388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196212+0000","last_change":"2024-03-06T07:24:44.836677+0000","last_active":"2024-03-06T07:24:48.196212+0000","last_peered":"2024-03-06T07:24:48.196212+0000","last_clean":"2024-03-06T07:24:48.196212+0000","last_became_active":"2024-03-06T07:24:44.836387+0000","last_became_peered":"2024-03-06T07:24:44.836387+0000","last_unstale":"2024-03-06T07:24:48.196212+0000","last_undegraded":"2024-03-06T07:24:48.196212+0000","last_fullsized":"2024-03-06T07:24:48.196212+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:21.834862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838225+0000","last_change":"2024-03-06T07:24:45.844027+0000","last_active":"2024-03-06T07:24:46.838225+0000","last_peered":"2024-03-06T07:24:46.838225+0000","last_clean":"2024-03-06T07:24:46.838225+0000","last_became_active":"2024-03-06T07:24:45.843723+0000","last_became_peered":"2024-03-06T07:24:45.843723+0000","last_unstale":"2024-03-06T07:24:46.838225+0000","last_undegraded":"2024-03-06T07:24:46.838225+0000","last_fullsized":"2024-03-06T07:24:46.838225+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:30.931860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838323+0000","last_change":"2024-03-06T07:24:44.840215+0000","last_active":"2024-03-06T07:24:46.838323+0000","last_peered":"2024-03-06T07:24:46.838323+0000","last_clean":"2024-03-06T07:24:46.838323+0000","last_became_active":"2024-03-06T07:24:44.839701+0000","last_became_peered":"2024-03-06T07:24:44.839701+0000","last_unstale":"2024-03-06T07:24:46.838323+0000","last_undegraded":"2024-03-06T07:24:46.838323+0000","last_fullsized":"2024-03-06T07:24:46.838323+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:06.809576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837498+0000","last_change":"2024-03-06T07:24:45.834946+0000","last_active":"2024-03-06T07:24:46.837498+0000","last_peered":"2024-03-06T07:24:46.837498+0000","last_clean":"2024-03-06T07:24:46.837498+0000","last_became_active":"2024-03-06T07:24:45.832763+0000","last_became_peered":"2024-03-06T07:24:45.832763+0000","last_unstale":"2024-03-06T07:24:46.837498+0000","last_undegraded":"2024-03-06T07:24:46.837498+0000","last_fullsized":"2024-03-06T07:24:46.837498+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:28.935085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196735+0000","last_change":"2024-03-06T07:24:44.838398+0000","last_active":"2024-03-06T07:24:48.196735+0000","last_peered":"2024-03-06T07:24:48.196735+0000","last_clean":"2024-03-06T07:24:48.196735+0000","last_became_active":"2024-03-06T07:24:44.838122+0000","last_became_peered":"2024-03-06T07:24:44.838122+0000","last_unstale":"2024-03-06T07:24:48.196735+0000","last_undegraded":"2024-03-06T07:24:48.196735+0000","last_fullsized":"2024-03-06T07:24:48.196735+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:06.809576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.839133+0000","last_change":"2024-03-06T07:24:45.838408+0000","last_active":"2024-03-06T07:24:46.839133+0000","last_peered":"2024-03-06T07:24:46.839133+0000","last_clean":"2024-03-06T07:24:46.839133+0000","last_became_active":"2024-03-06T07:24:45.838102+0000","last_became_peered":"2024-03-06T07:24:45.838102+0000","last_unstale":"2024-03-06T07:24:46.839133+0000","last_undegraded":"2024-03-06T07:24:46.839133+0000","last_fullsized":"2024-03-06T07:24:46.839133+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:47.002170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":17,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837448+0000","last_change":"2024-03-06T07:24:44.841428+0000","last_active":"2024-03-06T07:24:46.837448+0000","last_peered":"2024-03-06T07:24:46.837448+0000","last_clean":"2024-03-06T07:24:46.837448+0000","last_became_active":"2024-03-06T07:24:44.841148+0000","last_became_peered":"2024-03-06T07:24:44.841148+0000","last_unstale":"2024-03-06T07:24:46.837448+0000","last_undegraded":"2024-03-06T07:24:46.837448+0000","last_fullsized":"2024-03-06T07:24:46.837448+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:26.200577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837172+0000","last_change":"2024-03-06T07:24:45.850196+0000","last_active":"2024-03-06T07:24:46.837172+0000","last_peered":"2024-03-06T07:24:46.837172+0000","last_clean":"2024-03-06T07:24:46.837172+0000","last_became_active":"2024-03-06T07:24:45.849806+0000","last_became_peered":"2024-03-06T07:24:45.849806+0000","last_unstale":"2024-03-06T07:24:46.837172+0000","last_undegraded":"2024-03-06T07:24:46.837172+0000","last_fullsized":"2024-03-06T07:24:46.837172+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:10.077849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":20,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.836127+0000","last_change":"2024-03-06T07:24:44.841064+0000","last_active":"2024-03-06T07:24:46.836127+0000","last_peered":"2024-03-06T07:24:46.836127+0000","last_clean":"2024-03-06T07:24:46.836127+0000","last_became_active":"2024-03-06T07:24:44.840735+0000","last_became_peered":"2024-03-06T07:24:44.840735+0000","last_unstale":"2024-03-06T07:24:46.836127+0000","last_undegraded":"2024-03-06T07:24:46.836127+0000","last_fullsized":"2024-03-06T07:24:46.836127+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:59.530989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197506+0000","last_change":"2024-03-06T07:24:45.848190+0000","last_active":"2024-03-06T07:24:48.197506+0000","last_peered":"2024-03-06T07:24:48.197506+0000","last_clean":"2024-03-06T07:24:48.197506+0000","last_became_active":"2024-03-06T07:24:45.847451+0000","last_became_peered":"2024-03-06T07:24:45.847451+0000","last_unstale":"2024-03-06T07:24:48.197506+0000","last_undegraded":"2024-03-06T07:24:48.197506+0000","last_fullsized":"2024-03-06T07:24:48.197506+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:04.334703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838951+0000","last_change":"2024-03-06T07:24:44.843551+0000","last_active":"2024-03-06T07:24:46.838951+0000","last_peered":"2024-03-06T07:24:46.838951+0000","last_clean":"2024-03-06T07:24:46.838951+0000","last_became_active":"2024-03-06T07:24:44.843257+0000","last_became_peered":"2024-03-06T07:24:44.843257+0000","last_unstale":"2024-03-06T07:24:46.838951+0000","last_undegraded":"2024-03-06T07:24:46.838951+0000","last_fullsized":"2024-03-06T07:24:46.838951+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:07.596686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196827+0000","last_change":"2024-03-06T07:24:45.836875+0000","last_active":"2024-03-06T07:24:48.196827+0000","last_peered":"2024-03-06T07:24:48.196827+0000","last_clean":"2024-03-06T07:24:48.196827+0000","last_became_active":"2024-03-06T07:24:45.836613+0000","last_became_peered":"2024-03-06T07:24:45.836613+0000","last_unstale":"2024-03-06T07:24:48.196827+0000","last_undegraded":"2024-03-06T07:24:48.196827+0000","last_fullsized":"2024-03-06T07:24:48.196827+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:19.847774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.836952+0000","last_change":"2024-03-06T07:24:44.837407+0000","last_active":"2024-03-06T07:24:46.836952+0000","last_peered":"2024-03-06T07:24:46.836952+0000","last_clean":"2024-03-06T07:24:46.836952+0000","last_became_active":"2024-03-06T07:24:44.836928+0000","last_became_peered":"2024-03-06T07:24:44.836928+0000","last_unstale":"2024-03-06T07:24:46.836952+0000","last_undegraded":"2024-03-06T07:24:46.836952+0000","last_fullsized":"2024-03-06T07:24:46.836952+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:29.929581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838637+0000","last_change":"2024-03-06T07:24:45.838278+0000","last_active":"2024-03-06T07:24:46.838637+0000","last_peered":"2024-03-06T07:24:46.838637+0000","last_clean":"2024-03-06T07:24:46.838637+0000","last_became_active":"2024-03-06T07:24:45.837901+0000","last_became_peered":"2024-03-06T07:24:45.837901+0000","last_unstale":"2024-03-06T07:24:46.838637+0000","last_undegraded":"2024-03-06T07:24:46.838637+0000","last_fullsized":"2024-03-06T07:24:46.838637+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:18.790990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838748+0000","last_change":"2024-03-06T07:24:44.843409+0000","last_active":"2024-03-06T07:24:46.838748+0000","last_peered":"2024-03-06T07:24:46.838748+0000","last_clean":"2024-03-06T07:24:46.838748+0000","last_became_active":"2024-03-06T07:24:44.843072+0000","last_became_peered":"2024-03-06T07:24:44.843072+0000","last_unstale":"2024-03-06T07:24:46.838748+0000","last_undegraded":"2024-03-06T07:24:46.838748+0000","last_fullsized":"2024-03-06T07:24:46.838748+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:16.202894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.836911+0000","last_change":"2024-03-06T07:24:45.843163+0000","last_active":"2024-03-06T07:24:46.836911+0000","last_peered":"2024-03-06T07:24:46.836911+0000","last_clean":"2024-03-06T07:24:46.836911+0000","last_became_active":"2024-03-06T07:24:45.842797+0000","last_became_peered":"2024-03-06T07:24:45.842797+0000","last_unstale":"2024-03-06T07:24:46.836911+0000","last_undegraded":"2024-03-06T07:24:46.836911+0000","last_fullsized":"2024-03-06T07:24:46.836911+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:04.594889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196319+0000","last_change":"2024-03-06T07:24:44.838168+0000","last_active":"2024-03-06T07:24:48.196319+0000","last_peered":"2024-03-06T07:24:48.196319+0000","last_clean":"2024-03-06T07:24:48.196319+0000","last_became_active":"2024-03-06T07:24:44.837640+0000","last_became_peered":"2024-03-06T07:24:44.837640+0000","last_unstale":"2024-03-06T07:24:48.196319+0000","last_undegraded":"2024-03-06T07:24:48.196319+0000","last_fullsized":"2024-03-06T07:24:48.196319+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:55.061981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.835920+0000","last_change":"2024-03-06T07:24:45.835631+0000","last_active":"2024-03-06T07:24:46.835920+0000","last_peered":"2024-03-06T07:24:46.835920+0000","last_clean":"2024-03-06T07:24:46.835920+0000","last_became_active":"2024-03-06T07:24:45.835322+0000","last_became_peered":"2024-03-06T07:24:45.835322+0000","last_unstale":"2024-03-06T07:24:46.835920+0000","last_undegraded":"2024-03-06T07:24:46.835920+0000","last_fullsized":"2024-03-06T07:24:46.835920+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:07.811855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.835977+0000","last_change":"2024-03-06T07:24:44.842676+0000","last_active":"2024-03-06T07:24:46.835977+0000","last_peered":"2024-03-06T07:24:46.835977+0000","last_clean":"2024-03-06T07:24:46.835977+0000","last_became_active":"2024-03-06T07:24:44.842380+0000","last_became_peered":"2024-03-06T07:24:44.842380+0000","last_unstale":"2024-03-06T07:24:46.835977+0000","last_undegraded":"2024-03-06T07:24:46.835977+0000","last_fullsized":"2024-03-06T07:24:46.835977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:30.086505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196766+0000","last_change":"2024-03-06T07:24:45.826318+0000","last_active":"2024-03-06T07:24:48.196766+0000","last_peered":"2024-03-06T07:24:48.196766+0000","last_clean":"2024-03-06T07:24:48.196766+0000","last_became_active":"2024-03-06T07:24:45.826023+0000","last_became_peered":"2024-03-06T07:24:45.826023+0000","last_unstale":"2024-03-06T07:24:48.196766+0000","last_undegraded":"2024-03-06T07:24:48.196766+0000","last_fullsized":"2024-03-06T07:24:48.196766+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:00.194744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838325+0000","last_change":"2024-03-06T07:24:44.840199+0000","last_active":"2024-03-06T07:24:46.838325+0000","last_peered":"2024-03-06T07:24:46.838325+0000","last_clean":"2024-03-06T07:24:46.838325+0000","last_became_active":"2024-03-06T07:24:44.839854+0000","last_became_peered":"2024-03-06T07:24:44.839854+0000","last_unstale":"2024-03-06T07:24:46.838325+0000","last_undegraded":"2024-03-06T07:24:46.838325+0000","last_fullsized":"2024-03-06T07:24:46.838325+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:21.834862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197551+0000","last_change":"2024-03-06T07:24:45.849318+0000","last_active":"2024-03-06T07:24:48.197551+0000","last_peered":"2024-03-06T07:24:48.197551+0000","last_clean":"2024-03-06T07:24:48.197551+0000","last_became_active":"2024-03-06T07:24:45.849032+0000","last_became_peered":"2024-03-06T07:24:45.849032+0000","last_unstale":"2024-03-06T07:24:48.197551+0000","last_undegraded":"2024-03-06T07:24:48.197551+0000","last_fullsized":"2024-03-06T07:24:48.197551+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:12.551611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196992+0000","last_change":"2024-03-06T07:24:44.843436+0000","last_active":"2024-03-06T07:24:48.196992+0000","last_peered":"2024-03-06T07:24:48.196992+0000","last_clean":"2024-03-06T07:24:48.196992+0000","last_became_active":"2024-03-06T07:24:44.843137+0000","last_became_peered":"2024-03-06T07:24:44.843137+0000","last_unstale":"2024-03-06T07:24:48.196992+0000","last_undegraded":"2024-03-06T07:24:48.196992+0000","last_fullsized":"2024-03-06T07:24:48.196992+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:10.238622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197707+0000","last_change":"2024-03-06T07:24:45.850448+0000","last_active":"2024-03-06T07:24:48.197707+0000","last_peered":"2024-03-06T07:24:48.197707+0000","last_clean":"2024-03-06T07:24:48.197707+0000","last_became_active":"2024-03-06T07:24:45.850134+0000","last_became_peered":"2024-03-06T07:24:45.850134+0000","last_unstale":"2024-03-06T07:24:48.197707+0000","last_undegraded":"2024-03-06T07:24:48.197707+0000","last_fullsized":"2024-03-06T07:24:48.197707+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:43.433265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.839125+0000","last_change":"2024-03-06T07:24:44.836475+0000","last_active":"2024-03-06T07:24:46.839125+0000","last_peered":"2024-03-06T07:24:46.839125+0000","last_clean":"2024-03-06T07:24:46.839125+0000","last_became_active":"2024-03-06T07:24:44.836125+0000","last_became_peered":"2024-03-06T07:24:44.836125+0000","last_unstale":"2024-03-06T07:24:46.839125+0000","last_undegraded":"2024-03-06T07:24:46.839125+0000","last_fullsized":"2024-03-06T07:24:46.839125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:26.200577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838830+0000","last_change":"2024-03-06T07:24:45.838532+0000","last_active":"2024-03-06T07:24:46.838830+0000","last_peered":"2024-03-06T07:24:46.838830+0000","last_clean":"2024-03-06T07:24:46.838830+0000","last_became_active":"2024-03-06T07:24:45.837726+0000","last_became_peered":"2024-03-06T07:24:45.837726+0000","last_unstale":"2024-03-06T07:24:46.838830+0000","last_undegraded":"2024-03-06T07:24:46.838830+0000","last_fullsized":"2024-03-06T07:24:46.838830+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:37.980394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.839006+0000","last_change":"2024-03-06T07:24:44.841835+0000","last_active":"2024-03-06T07:24:46.839006+0000","last_peered":"2024-03-06T07:24:46.839006+0000","last_clean":"2024-03-06T07:24:46.839006+0000","last_became_active":"2024-03-06T07:24:44.841270+0000","last_became_peered":"2024-03-06T07:24:44.841270+0000","last_unstale":"2024-03-06T07:24:46.839006+0000","last_undegraded":"2024-03-06T07:24:46.839006+0000","last_fullsized":"2024-03-06T07:24:46.839006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:59.192465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:47.393026+0000","last_change":"2024-03-06T07:24:45.834151+0000","last_active":"2024-03-06T07:24:47.393026+0000","last_peered":"2024-03-06T07:24:47.393026+0000","last_clean":"2024-03-06T07:24:47.393026+0000","last_became_active":"2024-03-06T07:24:45.833870+0000","last_became_peered":"2024-03-06T07:24:45.833870+0000","last_unstale":"2024-03-06T07:24:47.393026+0000","last_undegraded":"2024-03-06T07:24:47.393026+0000","last_fullsized":"2024-03-06T07:24:47.393026+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:04.334703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196651+0000","last_change":"2024-03-06T07:24:44.841015+0000","last_active":"2024-03-06T07:24:48.196651+0000","last_peered":"2024-03-06T07:24:48.196651+0000","last_clean":"2024-03-06T07:24:48.196651+0000","last_became_active":"2024-03-06T07:24:44.840753+0000","last_became_peered":"2024-03-06T07:24:44.840753+0000","last_unstale":"2024-03-06T07:24:48.196651+0000","last_undegraded":"2024-03-06T07:24:48.196651+0000","last_fullsized":"2024-03-06T07:24:48.196651+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:10.663190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.835756+0000","last_change":"2024-03-06T07:24:45.842040+0000","last_active":"2024-03-06T07:24:46.835756+0000","last_peered":"2024-03-06T07:24:46.835756+0000","last_clean":"2024-03-06T07:24:46.835756+0000","last_became_active":"2024-03-06T07:24:45.841760+0000","last_became_peered":"2024-03-06T07:24:45.841760+0000","last_unstale":"2024-03-06T07:24:46.835756+0000","last_undegraded":"2024-03-06T07:24:46.835756+0000","last_fullsized":"2024-03-06T07:24:46.835756+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:22.837141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196171+0000","last_change":"2024-03-06T07:24:44.837932+0000","last_active":"2024-03-06T07:24:48.196171+0000","last_peered":"2024-03-06T07:24:48.196171+0000","last_clean":"2024-03-06T07:24:48.196171+0000","last_became_active":"2024-03-06T07:24:44.837625+0000","last_became_peered":"2024-03-06T07:24:44.837625+0000","last_unstale":"2024-03-06T07:24:48.196171+0000","last_undegraded":"2024-03-06T07:24:48.196171+0000","last_fullsized":"2024-03-06T07:24:48.196171+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:25.798836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196181+0000","last_change":"2024-03-06T07:24:45.832217+0000","last_active":"2024-03-06T07:24:48.196181+0000","last_peered":"2024-03-06T07:24:48.196181+0000","last_clean":"2024-03-06T07:24:48.196181+0000","last_became_active":"2024-03-06T07:24:45.831929+0000","last_became_peered":"2024-03-06T07:24:45.831929+0000","last_unstale":"2024-03-06T07:24:48.196181+0000","last_undegraded":"2024-03-06T07:24:48.196181+0000","last_fullsized":"2024-03-06T07:24:48.196181+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:15.801075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.836728+0000","last_change":"2024-03-06T07:24:44.838096+0000","last_active":"2024-03-06T07:24:46.836728+0000","last_peered":"2024-03-06T07:24:46.836728+0000","last_clean":"2024-03-06T07:24:46.836728+0000","last_became_active":"2024-03-06T07:24:44.837560+0000","last_became_peered":"2024-03-06T07:24:44.837560+0000","last_unstale":"2024-03-06T07:24:46.836728+0000","last_undegraded":"2024-03-06T07:24:46.836728+0000","last_fullsized":"2024-03-06T07:24:46.836728+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:03.760760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.197902+0000","last_change":"2024-03-06T07:24:45.848511+0000","last_active":"2024-03-06T07:24:48.197902+0000","last_peered":"2024-03-06T07:24:48.197902+0000","last_clean":"2024-03-06T07:24:48.197902+0000","last_became_active":"2024-03-06T07:24:45.848216+0000","last_became_peered":"2024-03-06T07:24:45.848216+0000","last_unstale":"2024-03-06T07:24:48.197902+0000","last_undegraded":"2024-03-06T07:24:48.197902+0000","last_fullsized":"2024-03-06T07:24:48.197902+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:59.151804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"46'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.839447+0000","last_change":"2024-03-06T07:24:44.841831+0000","last_active":"2024-03-06T07:24:46.839447+0000","last_peered":"2024-03-06T07:24:46.839447+0000","last_clean":"2024-03-06T07:24:46.839447+0000","last_became_active":"2024-03-06T07:24:44.841166+0000","last_became_peered":"2024-03-06T07:24:44.841166+0000","last_unstale":"2024-03-06T07:24:46.839447+0000","last_undegraded":"2024-03-06T07:24:46.839447+0000","last_fullsized":"2024-03-06T07:24:46.839447+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:03.760760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196648+0000","last_change":"2024-03-06T07:24:45.829797+0000","last_active":"2024-03-06T07:24:48.196648+0000","last_peered":"2024-03-06T07:24:48.196648+0000","last_clean":"2024-03-06T07:24:48.196648+0000","last_became_active":"2024-03-06T07:24:45.829513+0000","last_became_peered":"2024-03-06T07:24:45.829513+0000","last_unstale":"2024-03-06T07:24:48.196648+0000","last_undegraded":"2024-03-06T07:24:48.196648+0000","last_fullsized":"2024-03-06T07:24:48.196648+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:40.289000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837442+0000","last_change":"2024-03-06T07:24:44.839000+0000","last_active":"2024-03-06T07:24:46.837442+0000","last_peered":"2024-03-06T07:24:46.837442+0000","last_clean":"2024-03-06T07:24:46.837442+0000","last_became_active":"2024-03-06T07:24:44.838728+0000","last_became_peered":"2024-03-06T07:24:44.838728+0000","last_unstale":"2024-03-06T07:24:46.837442+0000","last_undegraded":"2024-03-06T07:24:46.837442+0000","last_fullsized":"2024-03-06T07:24:46.837442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:39.286721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837342+0000","last_change":"2024-03-06T07:24:45.845786+0000","last_active":"2024-03-06T07:24:46.837342+0000","last_peered":"2024-03-06T07:24:46.837342+0000","last_clean":"2024-03-06T07:24:46.837342+0000","last_became_active":"2024-03-06T07:24:45.845512+0000","last_became_peered":"2024-03-06T07:24:45.845512+0000","last_unstale":"2024-03-06T07:24:46.837342+0000","last_undegraded":"2024-03-06T07:24:46.837342+0000","last_fullsized":"2024-03-06T07:24:46.837342+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:18.790990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:47.392976+0000","last_change":"2024-03-06T07:24:44.840821+0000","last_active":"2024-03-06T07:24:47.392976+0000","last_peered":"2024-03-06T07:24:47.392976+0000","last_clean":"2024-03-06T07:24:47.392976+0000","last_became_active":"2024-03-06T07:24:44.840543+0000","last_became_peered":"2024-03-06T07:24:44.840543+0000","last_unstale":"2024-03-06T07:24:47.392976+0000","last_undegraded":"2024-03-06T07:24:47.392976+0000","last_fullsized":"2024-03-06T07:24:47.392976+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:25.798836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838848+0000","last_change":"2024-03-06T07:24:45.834885+0000","last_active":"2024-03-06T07:24:46.838848+0000","last_peered":"2024-03-06T07:24:46.838848+0000","last_clean":"2024-03-06T07:24:46.838848+0000","last_became_active":"2024-03-06T07:24:45.834609+0000","last_became_peered":"2024-03-06T07:24:45.834609+0000","last_unstale":"2024-03-06T07:24:46.838848+0000","last_undegraded":"2024-03-06T07:24:46.838848+0000","last_fullsized":"2024-03-06T07:24:46.838848+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:31.279294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837136+0000","last_change":"2024-03-06T07:24:44.838578+0000","last_active":"2024-03-06T07:24:46.837136+0000","last_peered":"2024-03-06T07:24:46.837136+0000","last_clean":"2024-03-06T07:24:46.837136+0000","last_became_active":"2024-03-06T07:24:44.838264+0000","last_became_peered":"2024-03-06T07:24:44.838264+0000","last_unstale":"2024-03-06T07:24:46.837136+0000","last_undegraded":"2024-03-06T07:24:46.837136+0000","last_fullsized":"2024-03-06T07:24:46.837136+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:38.563253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838791+0000","last_change":"2024-03-06T07:24:45.838769+0000","last_active":"2024-03-06T07:24:46.838791+0000","last_peered":"2024-03-06T07:24:46.838791+0000","last_clean":"2024-03-06T07:24:46.838791+0000","last_became_active":"2024-03-06T07:24:45.838440+0000","last_became_peered":"2024-03-06T07:24:45.838440+0000","last_unstale":"2024-03-06T07:24:46.838791+0000","last_undegraded":"2024-03-06T07:24:46.838791+0000","last_fullsized":"2024-03-06T07:24:46.838791+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:15.071984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.838039+0000","last_change":"2024-03-06T07:24:44.839270+0000","last_active":"2024-03-06T07:24:46.838039+0000","last_peered":"2024-03-06T07:24:46.838039+0000","last_clean":"2024-03-06T07:24:46.838039+0000","last_became_active":"2024-03-06T07:24:44.838971+0000","last_became_peered":"2024-03-06T07:24:44.838971+0000","last_unstale":"2024-03-06T07:24:46.838039+0000","last_undegraded":"2024-03-06T07:24:46.838039+0000","last_fullsized":"2024-03-06T07:24:46.838039+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:52.057642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196632+0000","last_change":"2024-03-06T07:24:45.831169+0000","last_active":"2024-03-06T07:24:48.196632+0000","last_peered":"2024-03-06T07:24:48.196632+0000","last_clean":"2024-03-06T07:24:48.196632+0000","last_became_active":"2024-03-06T07:24:45.830874+0000","last_became_peered":"2024-03-06T07:24:45.830874+0000","last_unstale":"2024-03-06T07:24:48.196632+0000","last_undegraded":"2024-03-06T07:24:48.196632+0000","last_fullsized":"2024-03-06T07:24:48.196632+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:40.289000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:47.392908+0000","last_change":"2024-03-06T07:24:44.837121+0000","last_active":"2024-03-06T07:24:47.392908+0000","last_peered":"2024-03-06T07:24:47.392908+0000","last_clean":"2024-03-06T07:24:47.392908+0000","last_became_active":"2024-03-06T07:24:44.836330+0000","last_became_peered":"2024-03-06T07:24:44.836330+0000","last_unstale":"2024-03-06T07:24:47.392908+0000","last_undegraded":"2024-03-06T07:24:47.392908+0000","last_fullsized":"2024-03-06T07:24:47.392908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:42.331255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.837896+0000","last_change":"2024-03-06T07:24:45.841051+0000","last_active":"2024-03-06T07:24:46.837896+0000","last_peered":"2024-03-06T07:24:46.837896+0000","last_clean":"2024-03-06T07:24:46.837896+0000","last_became_active":"2024-03-06T07:24:45.840765+0000","last_became_peered":"2024-03-06T07:24:45.840765+0000","last_unstale":"2024-03-06T07:24:46.837896+0000","last_undegraded":"2024-03-06T07:24:46.837896+0000","last_fullsized":"2024-03-06T07:24:46.837896+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:53.837226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196427+0000","last_change":"2024-03-06T07:24:44.837610+0000","last_active":"2024-03-06T07:24:48.196427+0000","last_peered":"2024-03-06T07:24:48.196427+0000","last_clean":"2024-03-06T07:24:48.196427+0000","last_became_active":"2024-03-06T07:24:44.837262+0000","last_became_peered":"2024-03-06T07:24:44.837262+0000","last_unstale":"2024-03-06T07:24:48.196427+0000","last_undegraded":"2024-03-06T07:24:48.196427+0000","last_fullsized":"2024-03-06T07:24:48.196427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:31.940412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196880+0000","last_change":"2024-03-06T07:24:45.833406+0000","last_active":"2024-03-06T07:24:48.196880+0000","last_peered":"2024-03-06T07:24:48.196880+0000","last_clean":"2024-03-06T07:24:48.196880+0000","last_became_active":"2024-03-06T07:24:45.833099+0000","last_became_peered":"2024-03-06T07:24:45.833099+0000","last_unstale":"2024-03-06T07:24:48.196880+0000","last_undegraded":"2024-03-06T07:24:48.196880+0000","last_fullsized":"2024-03-06T07:24:48.196880+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:44.815470+0000","last_clean_scrub_stamp":"2024-03-06T07:24:44.815470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:48.321964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:48.196921+0000","last_change":"2024-03-06T07:24:44.844868+0000","last_active":"2024-03-06T07:24:48.196921+0000","last_peered":"2024-03-06T07:24:48.196921+0000","last_clean":"2024-03-06T07:24:48.196921+0000","last_became_active":"2024-03-06T07:24:44.844594+0000","last_became_peered":"2024-03-06T07:24:44.844594+0000","last_unstale":"2024-03-06T07:24:48.196921+0000","last_undegraded":"2024-03-06T07:24:48.196921+0000","last_fullsized":"2024-03-06T07:24:48.196921+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:43.813191+0000","last_clean_scrub_stamp":"2024-03-06T07:24:43.813191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:58.149525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":219,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:24:46.836754+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:24:46.836754+0000","last_peered":"2024-03-06T07:24:46.836754+0000","last_clean":"2024-03-06T07:24:46.836754+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:24:46.836754+0000","last_undegraded":"2024-03-06T07:24:46.836754+0000","last_fullsized":"2024-03-06T07:24:46.836754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":810252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":548864,"data_stored":220466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":219864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574878,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":552960,"data_stored":221406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":552960,"data_stored":220432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31728,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340112,"statfs":{"total":96636764160,"available":96604274688,"internally_reserved":0,"allocated":557056,"data_stored":221428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":219864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":810196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:50.352 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:50.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:50.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.501+0000 7fe53d7bc640 1 Processor -- start 2024-03-06T07:24:50.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.501+0000 7fe53d7bc640 1 -- start start 2024-03-06T07:24:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe53d7bc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe5380717b0 0x7fe538071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe53d7bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 0x7fe5380d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe53d7bc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53815e4b0 con 0x7fe5380b2b10 2024-03-06T07:24:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe53d7bc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5380d2e10 con 0x7fe5380717b0 2024-03-06T07:24:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe53d7bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380b97e0 con 0x7fe5380b3060 2024-03-06T07:24:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe536ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe5380717b0 0x7fe538071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe536ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe5380717b0 0x7fe538071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46190/0 (socket says 172.21.15.27:46190) 2024-03-06T07:24:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe536ffd640 1 -- 172.21.15.27:0/559440133 learned_addr learned my addr 172.21.15.27:0/559440133 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe5367fc640 1 --2- 172.21.15.27:0/559440133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 0x7fe5380d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe536ffd640 1 -- 172.21.15.27:0/559440133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 msgr2=0x7fe5380d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe536ffd640 1 --2- 172.21.15.27:0/559440133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 0x7fe5380d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe536ffd640 1 -- 172.21.15.27:0/559440133 >> v1:172.21.15.69:6789/0 conn(0x7fe5380b2b10 legacy=0x7fe5380b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe536ffd640 1 -- 172.21.15.27:0/559440133 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5380b9a80 con 0x7fe5380717b0 2024-03-06T07:24:50.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.502+0000 7fe5367fc640 1 --2- 172.21.15.27:0/559440133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 0x7fe5380d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:50.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.503+0000 7fe536ffd640 1 --2- 172.21.15.27:0/559440133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 0x7fe538071b90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009870 tx=0x7fe52c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59cd2aeff27df30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.503+0000 7fe535ffb640 1 -- 172.21.15.27:0/559440133 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c034070 con 0x7fe5380717b0 2024-03-06T07:24:50.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.503+0000 7fe535ffb640 1 -- 172.21.15.27:0/559440133 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c030840 con 0x7fe5380717b0 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.503+0000 7fe535ffb640 1 -- 172.21.15.27:0/559440133 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c030c10 con 0x7fe5380717b0 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.504+0000 7fe53d7bc640 1 -- 172.21.15.27:0/559440133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 msgr2=0x7fe538071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.504+0000 7fe53d7bc640 1 --2- 172.21.15.27:0/559440133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 0x7fe538071b90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009870 tx=0x7fe52c02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.504+0000 7fe53d7bc640 1 -- 172.21.15.27:0/559440133 shutdown_connections 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.504+0000 7fe53d7bc640 1 --2- 172.21.15.27:0/559440133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 0x7fe5380d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.504+0000 7fe53d7bc640 1 --2- 172.21.15.27:0/559440133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 0x7fe538071b90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.504+0000 7fe53d7bc640 1 -- 172.21.15.27:0/559440133 >> 172.21.15.27:0/559440133 conn(0x7fe5380d3d20 msgr2=0x7fe5380d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.504+0000 7fe53d7bc640 1 -- 172.21.15.27:0/559440133 shutdown_connections 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.504+0000 7fe53d7bc640 1 -- 172.21.15.27:0/559440133 wait complete. 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.505+0000 7fe53d7bc640 1 Processor -- start 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.505+0000 7fe53d7bc640 1 -- start start 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe53d7bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 0x7fe5381f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe53d7bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5380b2b10 0x7fe538156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe53d7bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 0x7fe538159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe53d7bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380b7560 con 0x7fe5380717b0 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe53d7bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380d4bf0 con 0x7fe5380b2b10 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe53d7bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5380b76e0 con 0x7fe5380b3060 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe536ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 0x7fe5381f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe536ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 0x7fe5381f4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46202/0 (socket says 172.21.15.27:46202) 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe536ffd640 1 -- 172.21.15.27:0/3926102956 learned_addr learned my addr 172.21.15.27:0/3926102956 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe5367fc640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5380b2b10 0x7fe538156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe5377fe640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 0x7fe538159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe536ffd640 1 -- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 msgr2=0x7fe538159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe536ffd640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 0x7fe538159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe536ffd640 1 -- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5380b2b10 msgr2=0x7fe538156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe536ffd640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5380b2b10 0x7fe538156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.506+0000 7fe536ffd640 1 -- 172.21.15.27:0/3926102956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5381e6d80 con 0x7fe5380717b0 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.507+0000 7fe536ffd640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 0x7fe5381f4e50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe52c000c00 tx=0x7fe52c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.507+0000 7fe51f7fe640 1 -- 172.21.15.27:0/3926102956 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c048070 con 0x7fe5380717b0 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.507+0000 7fe51f7fe640 1 -- 172.21.15.27:0/3926102956 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c030640 con 0x7fe5380717b0 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.507+0000 7fe51f7fe640 1 -- 172.21.15.27:0/3926102956 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c034040 con 0x7fe5380717b0 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.507+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5381e7040 con 0x7fe5380717b0 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.507+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5381e7320 con 0x7fe5380717b0 2024-03-06T07:24:50.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.508+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe538157a50 con 0x7fe5380717b0 2024-03-06T07:24:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.509+0000 7fe51f7fe640 1 -- 172.21.15.27:0/3926102956 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe52c04b070 con 0x7fe5380717b0 2024-03-06T07:24:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.509+0000 7fe51f7fe640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe52406b500 0x7fe52406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.509+0000 7fe51f7fe640 1 -- 172.21.15.27:0/3926102956 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe52c0ac120 con 0x7fe5380717b0 2024-03-06T07:24:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.510+0000 7fe5367fc640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe52406b500 0x7fe52406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.510+0000 7fe5367fc640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe52406b500 0x7fe52406d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5381f1ff0 tx=0x7fe528008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.514+0000 7fe51f7fe640 1 -- 172.21.15.27:0/3926102956 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe52c07afa0 con 0x7fe5380717b0 2024-03-06T07:24:50.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.765+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe53806ce40 con 0x7fe5380717b0 2024-03-06T07:24:50.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.767+0000 7fe51f7fe640 1 -- 172.21.15.27:0/3926102956 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe52c030810 con 0x7fe5380717b0 2024-03-06T07:24:50.769 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:24:50.769 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:24:50.769 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-03-06T07:24:50.769 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-03-06T07:24:50.769 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:50.769 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-03-06T07:24:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe52406b500 msgr2=0x7fe52406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe52406b500 0x7fe52406d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5381f1ff0 tx=0x7fe528008040 comp rx=0 tx=0).stop 2024-03-06T07:24:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 msgr2=0x7fe5381f4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 0x7fe5381f4e50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe52c000c00 tx=0x7fe52c004730 comp rx=0 tx=0).stop 2024-03-06T07:24:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 shutdown_connections 2024-03-06T07:24:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe52406b500 0x7fe52406d9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5380b3060 0x7fe538159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5380b2b10 0x7fe538156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 --2- 172.21.15.27:0/3926102956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5380717b0 0x7fe5381f4e50 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 >> 172.21.15.27:0/3926102956 conn(0x7fe5380d3d20 msgr2=0x7fe53815ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 shutdown_connections 2024-03-06T07:24:50.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.769+0000 7fe53d7bc640 1 -- 172.21.15.27:0/3926102956 wait complete. 2024-03-06T07:24:50.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:50.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.894+0000 7f8b78b55640 1 Processor -- start 2024-03-06T07:24:50.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.894+0000 7f8b78b55640 1 -- start start 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b78b55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b7415df00 0x7f8b740d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b78b55640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b740b95a0 0x7f8b740b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b78b55640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b740cea30 con 0x7f8b740b9a30 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b78b55640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b740d2af0 con 0x7f8b740b95a0 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b78b55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b740cf3c0 con 0x7f8b7415df00 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b71d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b740b95a0 0x7f8b740b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b71d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b740b95a0 0x7f8b740b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46210/0 (socket says 172.21.15.27:46210) 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b71d9b640 1 -- 172.21.15.27:0/2427688269 learned_addr learned my addr 172.21.15.27:0/2427688269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b7259c640 1 --2- 172.21.15.27:0/2427688269 >> v2:172.21.15.69:3300/0 conn(0x7f8b7415df00 0x7f8b740d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b71d9b640 1 -- 172.21.15.27:0/2427688269 >> v2:172.21.15.69:3300/0 conn(0x7f8b7415df00 msgr2=0x7f8b740d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b71d9b640 1 --2- 172.21.15.27:0/2427688269 >> v2:172.21.15.69:3300/0 conn(0x7f8b7415df00 0x7f8b740d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b71d9b640 1 -- 172.21.15.27:0/2427688269 >> v1:172.21.15.69:6789/0 conn(0x7f8b740b9a30 legacy=0x7f8b740d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.895+0000 7f8b71d9b640 1 -- 172.21.15.27:0/2427688269 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b740cf0e0 con 0x7f8b740b95a0 2024-03-06T07:24:50.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.896+0000 7f8b71d9b640 1 --2- 172.21.15.27:0/2427688269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 0x7f8b740b5700 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8b740d11b0 tx=0x7f8b5c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74e623d78f3ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.896+0000 7f8b7159a640 1 -- 172.21.15.27:0/2427688269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c034070 con 0x7f8b740b95a0 2024-03-06T07:24:50.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.896+0000 7f8b7159a640 1 -- 172.21.15.27:0/2427688269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c030860 con 0x7f8b740b95a0 2024-03-06T07:24:50.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.896+0000 7f8b7159a640 1 -- 172.21.15.27:0/2427688269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c030c30 con 0x7f8b740b95a0 2024-03-06T07:24:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.897+0000 7f8b78b55640 1 -- 172.21.15.27:0/2427688269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 msgr2=0x7f8b740b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.897+0000 7f8b78b55640 1 --2- 172.21.15.27:0/2427688269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 0x7f8b740b5700 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8b740d11b0 tx=0x7f8b5c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.897+0000 7f8b78b55640 1 -- 172.21.15.27:0/2427688269 shutdown_connections 2024-03-06T07:24:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.897+0000 7f8b78b55640 1 --2- 172.21.15.27:0/2427688269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 0x7f8b740b5700 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.897+0000 7f8b78b55640 1 --2- 172.21.15.27:0/2427688269 >> v2:172.21.15.69:3300/0 conn(0x7f8b7415df00 0x7f8b740d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.897+0000 7f8b78b55640 1 -- 172.21.15.27:0/2427688269 >> 172.21.15.27:0/2427688269 conn(0x7f8b74071960 msgr2=0x7f8b74071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.897+0000 7f8b78b55640 1 -- 172.21.15.27:0/2427688269 shutdown_connections 2024-03-06T07:24:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.897+0000 7f8b78b55640 1 -- 172.21.15.27:0/2427688269 wait complete. 2024-03-06T07:24:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.898+0000 7f8b78b55640 1 Processor -- start 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.898+0000 7f8b78b55640 1 -- start start 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b78b55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 0x7f8b741550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b78b55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b7415df00 0x7f8b740cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b78b55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b740cd790 0x7f8b7415f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b78b55640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740cec70 con 0x7f8b740b95a0 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b78b55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740ce750 con 0x7f8b7415df00 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b78b55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740cedf0 con 0x7f8b740cd790 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b7259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 0x7f8b741550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b7259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 0x7f8b741550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46224/0 (socket says 172.21.15.27:46224) 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b7259c640 1 -- 172.21.15.27:0/3574328067 learned_addr learned my addr 172.21.15.27:0/3574328067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b71d9b640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b7415df00 0x7f8b740cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.899+0000 7f8b72d9d640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b740cd790 0x7f8b7415f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b7259c640 1 -- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b740cd790 msgr2=0x7f8b7415f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b7259c640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b740cd790 0x7f8b7415f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b7259c640 1 -- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b7415df00 msgr2=0x7f8b740cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b7259c640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b7415df00 0x7f8b740cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b7259c640 1 -- 172.21.15.27:0/3574328067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b74160010 con 0x7f8b740b95a0 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b7259c640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 0x7f8b741550a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8b64007e90 tx=0x7f8b6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b6affd640 1 -- 172.21.15.27:0/3574328067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b64014620 con 0x7f8b740b95a0 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b6affd640 1 -- 172.21.15.27:0/3574328067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b64019070 con 0x7f8b740b95a0 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b6affd640 1 -- 172.21.15.27:0/3574328067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b64014a20 con 0x7f8b740b95a0 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b741555e0 con 0x7f8b740b95a0 2024-03-06T07:24:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.900+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b740cb0d0 con 0x7f8b740b95a0 2024-03-06T07:24:50.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.901+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b740cdba0 con 0x7f8b740b95a0 2024-03-06T07:24:50.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.902+0000 7f8b6affd640 1 -- 172.21.15.27:0/3574328067 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b64014bc0 con 0x7f8b740b95a0 2024-03-06T07:24:50.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.902+0000 7f8b6affd640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b5406b500 0x7f8b5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.902+0000 7f8b6affd640 1 -- 172.21.15.27:0/3574328067 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b6408a950 con 0x7f8b740b95a0 2024-03-06T07:24:50.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.902+0000 7f8b71d9b640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b5406b500 0x7f8b5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.903+0000 7f8b71d9b640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b5406b500 0x7f8b5406d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c036000 tx=0x7f8b5c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:50.906+0000 7f8b6affd640 1 -- 172.21.15.27:0/3574328067 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b740cdba0 con 0x7f8b740b95a0 2024-03-06T07:24:51.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.125+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b74067ff0 con 0x7f8b740b95a0 2024-03-06T07:24:51.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.125+0000 7f8b6affd640 1 -- 172.21.15.27:0/3574328067 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f8b64059180 con 0x7f8b740b95a0 2024-03-06T07:24:51.128 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:24:51.128 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5126,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2827284911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2827284911},{"type":"v1","addr":"172.21.15.69:6836","nonce":2827284911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5419,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1663054032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1663054032},{"type":"v1","addr":"172.21.15.69:6837","nonce":1663054032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2744957333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2744957333},{"type":"v1","addr":"172.21.15.27:6834","nonce":2744957333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:45.201958+0000","modified":"2024-03-06T07:24:48.290611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5754":{"gid":5754,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2008146374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2008146374},{"type":"v1","addr":"172.21.15.27:6835","nonce":2008146374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5754,"qdb_cluster":[5754]},"id":3}]} 2024-03-06T07:24:51.128 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-03-06T07:24:51.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b5406b500 msgr2=0x7f8b5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b5406b500 0x7f8b5406d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c036000 tx=0x7f8b5c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 msgr2=0x7f8b741550a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 0x7f8b741550a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8b64007e90 tx=0x7f8b6400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 shutdown_connections 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b5406b500 0x7f8b5406d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b740cd790 0x7f8b7415f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b7415df00 0x7f8b740cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 --2- 172.21.15.27:0/3574328067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b740b95a0 0x7f8b741550a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 >> 172.21.15.27:0/3574328067 conn(0x7f8b74071960 msgr2=0x7f8b74071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 shutdown_connections 2024-03-06T07:24:51.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.128+0000 7f8b78b55640 1 -- 172.21.15.27:0/3574328067 wait complete. 2024-03-06T07:24:51.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:45.201958+0000', 'modified': '2024-03-06T07:24:48.290611+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5754}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5754': {'gid': 5754, 'name': 'c', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/2008146374', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 2008146374}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 2008146374}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5754, 'qdb_cluster': [5754]} 2024-03-06T07:24:51.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:51.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:24:51.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.649+0000 7f0db4415640 1 Processor -- start 2024-03-06T07:24:51.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.649+0000 7f0db4415640 1 -- start start 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.650+0000 7f0db4415640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0dac15bac0 0x7f0dac15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.650+0000 7f0db4415640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dac0729b0 con 0x7f0dac0aee10 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.650+0000 7f0db4415640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dac06c920 con 0x7f0dac06d400 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.650+0000 7f0db4415640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dac06cf70 con 0x7f0dac15bac0 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.650+0000 7f0db29b2640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0dac0aee10 0x7f0dac070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56220/0 (socket says 172.21.15.27:56220) 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.650+0000 7f0db29b2640 1 -- 172.21.15.27:0/869177670 learned_addr learned my addr 172.21.15.27:0/869177670 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db21b1640 1 --2- 172.21.15.27:0/869177670 >> v2:172.21.15.69:3300/0 conn(0x7f0dac15bac0 0x7f0dac15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246614785 0 0) 0x7f0dac0729b0 con 0x7f0dac0aee10 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d98003520 con 0x7f0dac0aee10 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1911613182 0 0) 0x7f0d98003520 con 0x7f0dac0aee10 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 >> v2:172.21.15.69:3300/0 conn(0x7f0dac15bac0 msgr2=0x7f0dac15cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 --2- 172.21.15.27:0/869177670 >> v2:172.21.15.69:3300/0 conn(0x7f0dac15bac0 0x7f0dac15cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 >> v1:172.21.15.69:6789/0 conn(0x7f0dac06d400 legacy=0x7f0dac0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac06cc00 con 0x7f0dac0aee10 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f0da40019f0 con 0x7f0dac0aee10 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db21b1640 1 --2- 172.21.15.27:0/869177670 >> v2:172.21.15.69:3300/0 conn(0x7f0dac15bac0 0x7f0dac15cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 >> v1:172.21.15.27:6789/0 conn(0x7f0dac0aee10 legacy=0x7f0dac070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d980039b0 0x7f0d98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d98004800 0x7f0d98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d980051c0 0x7f0d9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac06cf70 con 0x7f0d98004800 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9800c060 con 0x7f0d980039b0 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9800c260 con 0x7f0d980051c0 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db19b0640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d98004800 0x7f0d98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db21b1640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d980039b0 0x7f0d98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db29b2640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d980051c0 0x7f0d9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db19b0640 1 -- 172.21.15.27:0/869177670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d980051c0 msgr2=0x7f0d9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db19b0640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d980051c0 0x7f0d9800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db19b0640 1 -- 172.21.15.27:0/869177670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d980039b0 msgr2=0x7f0d98003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db19b0640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d980039b0 0x7f0d98003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db19b0640 1 -- 172.21.15.27:0/869177670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d98003520 con 0x7f0d98004800 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.651+0000 7f0db29b2640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d980051c0 0x7f0d9800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.652+0000 7f0db29b2640 1 -- 172.21.15.27:0/869177670 reap_dead start 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.652+0000 7f0db19b0640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d98004800 0x7f0d98004c80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c001750 tx=0x7f0d98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b551dc634d736671 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.652+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c001370 con 0x7f0d98004800 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.652+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9c0052d0 con 0x7f0d98004800 2024-03-06T07:24:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.652+0000 7f0db11af640 1 -- 172.21.15.27:0/869177670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c003830 con 0x7f0d98004800 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.653+0000 7f0db4415640 1 -- 172.21.15.27:0/869177670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d98004800 msgr2=0x7f0d98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.653+0000 7f0db4415640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d98004800 0x7f0d98004c80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c001750 tx=0x7f0d98005830 comp rx=0 tx=0).stop 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.653+0000 7f0db4415640 1 -- 172.21.15.27:0/869177670 shutdown_connections 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.653+0000 7f0db4415640 1 --2- 172.21.15.27:0/869177670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d98004800 0x7f0d98004c80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.653+0000 7f0db4415640 1 -- 172.21.15.27:0/869177670 >> 172.21.15.27:0/869177670 conn(0x7f0dac066d00 msgr2=0x7f0dac0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.653+0000 7f0db4415640 1 -- 172.21.15.27:0/869177670 shutdown_connections 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.653+0000 7f0db4415640 1 -- 172.21.15.27:0/869177670 wait complete. 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.654+0000 7f0db4415640 1 Processor -- start 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.654+0000 7f0db4415640 1 -- start start 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db4415640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d98004800 0x7f0dac152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db4415640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac147f90 0x7f0dac159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db4415640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dac148410 0x7f0dac159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db4415640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac15c020 con 0x7f0dac147f90 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db4415640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac15bea0 con 0x7f0d98004800 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db4415640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac072580 con 0x7f0dac148410 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db19b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac147f90 0x7f0dac159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db19b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac147f90 0x7f0dac159820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46230/0 (socket says 172.21.15.27:46230) 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db19b0640 1 -- 172.21.15.27:0/4024100085 learned_addr learned my addr 172.21.15.27:0/4024100085 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db21b1640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d98004800 0x7f0dac152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db29b2640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dac148410 0x7f0dac159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db19b0640 1 -- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dac148410 msgr2=0x7f0dac159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db19b0640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dac148410 0x7f0dac159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db19b0640 1 -- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d98004800 msgr2=0x7f0dac152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db19b0640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d98004800 0x7f0dac152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.655+0000 7f0db19b0640 1 -- 172.21.15.27:0/4024100085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac151c10 con 0x7f0dac147f90 2024-03-06T07:24:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.656+0000 7f0db19b0640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac147f90 0x7f0dac159820 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c000c00 tx=0x7f0d9c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.656+0000 7f0da2ffd640 1 -- 172.21.15.27:0/4024100085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c041020 con 0x7f0dac147f90 2024-03-06T07:24:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.656+0000 7f0da2ffd640 1 -- 172.21.15.27:0/4024100085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9c005480 con 0x7f0dac147f90 2024-03-06T07:24:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.656+0000 7f0da2ffd640 1 -- 172.21.15.27:0/4024100085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c007a30 con 0x7f0dac147f90 2024-03-06T07:24:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.656+0000 7f0db4415640 1 -- 172.21.15.27:0/4024100085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dac151ef0 con 0x7f0dac147f90 2024-03-06T07:24:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.656+0000 7f0db4415640 1 -- 172.21.15.27:0/4024100085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dac1524b0 con 0x7f0dac147f90 2024-03-06T07:24:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.658+0000 7f0da2ffd640 1 -- 172.21.15.27:0/4024100085 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d9c044020 con 0x7f0dac147f90 2024-03-06T07:24:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.658+0000 7f0da2ffd640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d9006b4d0 0x7f0d9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.658+0000 7f0da2ffd640 1 -- 172.21.15.27:0/4024100085 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d9c0abb20 con 0x7f0dac147f90 2024-03-06T07:24:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.658+0000 7f0db21b1640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d9006b4d0 0x7f0d9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.659+0000 7f0db21b1640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d9006b4d0 0x7f0d9006d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009dc0 tx=0x7f0da8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.659+0000 7f0db7004740 1 Processor -- start 2024-03-06T07:24:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.659+0000 7f0db7004740 1 -- start start 2024-03-06T07:24:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db7004740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006217870 0x55c006217c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db7004740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00621b8c0 0x55c00621d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db7004740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006247600 0x55c006247a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db7004740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0061816b0 con 0x55c006247600 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db7004740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c00612aa60 con 0x55c006217870 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db7004740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c00617d750 con 0x55c00621b8c0 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db29b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006247600 0x55c006247a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db29b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006247600 0x55c006247a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46232/0 (socket says 172.21.15.27:46232) 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db29b2640 1 -- 172.21.15.27:0/335107285 learned_addr learned my addr 172.21.15.27:0/335107285 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db21b1640 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006217870 0x55c006217c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db19b0640 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00621b8c0 0x55c00621d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db29b2640 1 -- 172.21.15.27:0/335107285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00621b8c0 msgr2=0x55c00621d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db29b2640 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00621b8c0 0x55c00621d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db29b2640 1 -- 172.21.15.27:0/335107285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006217870 msgr2=0x55c006217c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db29b2640 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006217870 0x55c006217c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db29b2640 1 -- 172.21.15.27:0/335107285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c006148f30 con 0x55c006247600 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.660+0000 7f0db19b0640 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00621b8c0 0x55c00621d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.661+0000 7f0db29b2640 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006247600 0x55c006247a80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0dac06ea70 tx=0x7f0da4005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45205d448b1650ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.661+0000 7f0d8ffff640 1 -- 172.21.15.27:0/335107285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da401a070 con 0x55c006247600 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.661+0000 7f0d8ffff640 1 -- 172.21.15.27:0/335107285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da4007970 con 0x55c006247600 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.661+0000 7f0d8ffff640 1 -- 172.21.15.27:0/335107285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da40156b0 con 0x55c006247600 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.661+0000 7f0db7004740 1 -- 172.21.15.27:0/335107285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006247600 msgr2=0x55c006247a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.661+0000 7f0db7004740 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006247600 0x55c006247a80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0dac06ea70 tx=0x7f0da4005f40 comp rx=0 tx=0).stop 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.662+0000 7f0db7004740 1 -- 172.21.15.27:0/335107285 shutdown_connections 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.662+0000 7f0db7004740 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006247600 0x55c006247a80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.662+0000 7f0db7004740 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00621b8c0 0x55c00621d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.662+0000 7f0db7004740 1 --2- 172.21.15.27:0/335107285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006217870 0x55c006217c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.662+0000 7f0db7004740 1 -- 172.21.15.27:0/335107285 >> 172.21.15.27:0/335107285 conn(0x55c0062198f0 msgr2=0x55c00621b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.662+0000 7f0db7004740 1 -- 172.21.15.27:0/335107285 shutdown_connections 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.662+0000 7f0db7004740 1 -- 172.21.15.27:0/335107285 wait complete. 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.662+0000 7f0db7004740 1 -- start start 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.662+0000 7f0db7004740 1 Processor -- start 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db7004740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 0x55c006272ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db7004740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00625c590 0x55c006242a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db7004740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006249fe0 0x55c006260820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db7004740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c00621bd50 con 0x55c006249fe0 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db7004740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c00613dea0 con 0x55c006247600 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db7004740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c00621c030 con 0x55c00625c590 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db7004740 10 client.0 _ll_register_callbacks cb 0x55c00613d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db29b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006249fe0 0x55c006260820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db29b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006249fe0 0x55c006260820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46244/0 (socket says 172.21.15.27:46244) 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db29b2640 1 -- 172.21.15.27:0/1017592246 learned_addr learned my addr 172.21.15.27:0/1017592246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db21b1640 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 0x55c006272ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.663+0000 7f0db19b0640 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00625c590 0x55c006242a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0db29b2640 1 -- 172.21.15.27:0/1017592246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00625c590 msgr2=0x55c006242a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0db29b2640 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00625c590 0x55c006242a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0db29b2640 1 -- 172.21.15.27:0/1017592246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 msgr2=0x55c006272ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0db29b2640 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 0x55c006272ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0db29b2640 1 -- 172.21.15.27:0/1017592246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c00613e270 con 0x55c006249fe0 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0db29b2640 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006249fe0 0x55c006260820 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0da4002230 tx=0x7f0dac0cc1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761f66ac36a0382 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/1017592246 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4025060 con 0x55c006249fe0 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/1017592246 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0da4015d90 con 0x55c006249fe0 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0db7004740 1 -- 172.21.15.27:0/1017592246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0da4007970 con 0x55c006249fe0 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/1017592246 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0da402b050 con 0x55c006249fe0 2024-03-06T07:24:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0db7004740 10 client.5904 fetch_fsmap learned FSMap version 26 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0db7004740 1 -- 172.21.15.27:0/1017592246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55c005e962d0 con 0x55c006249fe0 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.664+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/1017592246 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da4007970 con 0x55c006249fe0 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/1017592246 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0da402e020 con 0x55c006249fe0 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0db7004740 10 client.5904 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0db7004740 10 client.5904 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0db7004740 20 client.5904 populate_metadata read hostname 'smithi027' 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0db7004740 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] conn(0x55c006276960 0x55c006281540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0db7004740 4 client.5904 mds_command: new command op to 5754 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0db7004740 1 -- 172.21.15.27:0/1017592246 --> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c00618ebf0 con 0x55c006276960 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0db21b1640 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] conn(0x55c006276960 0x55c006281540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0d8d7fa640 10 client.5904 ms_handle_connect on v2:172.21.15.27:6833/2008146374 2024-03-06T07:24:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.665+0000 7f0db21b1640 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] conn(0x55c006276960 0x55c006281540 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c27148202cbc84a server_cookie=b27057f7917fd844 in_seq=0 out_seq=0 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.666+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/1017592246 <== mds.0 v2:172.21.15.27:6833/2008146374 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c00618ebf0 con 0x55c006276960 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.666+0000 7f0d8d7fa640 10 client.5904 handle_command_reply: tid=0 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.666+0000 7f0db7004740 1 client.5904 shutdown 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.666+0000 7f0db7004740 1 -- 172.21.15.27:0/1017592246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006249fe0 msgr2=0x55c006260820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.666+0000 7f0db7004740 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006249fe0 0x55c006260820 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0da4002230 tx=0x7f0dac0cc1f0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0db7004740 1 -- 172.21.15.27:0/1017592246 shutdown_connections 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0db7004740 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] conn(0x55c006276960 0x55c006281540 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0db7004740 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c006249fe0 0x55c006260820 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0db7004740 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c00625c590 0x55c006242a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0d8d7fa640 0 client.5904 ms_handle_reset on v2:172.21.15.27:6833/2008146374 2024-03-06T07:24:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0db7004740 1 --2- 172.21.15.27:0/1017592246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 0x55c006272ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0db7004740 1 -- 172.21.15.27:0/1017592246 >> 172.21.15.27:0/1017592246 conn(0x55c006217870 msgr2=0x55c0062455c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0db7004740 1 -- 172.21.15.27:0/1017592246 shutdown_connections 2024-03-06T07:24:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0db7004740 1 -- 172.21.15.27:0/1017592246 wait complete. 2024-03-06T07:24:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.667+0000 7f0db7004740 20 client.5904 trim_cache size 0 max 16384 2024-03-06T07:24:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.684+0000 7f0db7004740 1 Processor -- start 2024-03-06T07:24:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.684+0000 7f0db7004740 1 -- start start 2024-03-06T07:24:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.684+0000 7f0db7004740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 0x55c006216700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db7004740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006249fe0 0x55c006247c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db7004740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db7004740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac1fd700 con 0x55c00625c590 2024-03-06T07:24:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db7004740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac1589e0 con 0x55c006247600 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db7004740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac158fe0 con 0x55c006249fe0 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db29b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db29b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46256/0 (socket says 172.21.15.27:46256) 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db29b2640 1 -- 172.21.15.27:0/839208418 learned_addr learned my addr 172.21.15.27:0/839208418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db19b0640 1 --2- 172.21.15.27:0/839208418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006249fe0 0x55c006247c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db21b1640 1 --2- 172.21.15.27:0/839208418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 0x55c006216700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db29b2640 1 -- 172.21.15.27:0/839208418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006249fe0 msgr2=0x55c006247c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db29b2640 1 --2- 172.21.15.27:0/839208418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006249fe0 0x55c006247c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db29b2640 1 -- 172.21.15.27:0/839208418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 msgr2=0x55c006216700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db29b2640 1 --2- 172.21.15.27:0/839208418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 0x55c006216700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db29b2640 1 -- 172.21.15.27:0/839208418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c005f606b0 con 0x55c00625c590 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0db29b2640 1 --2- 172.21.15.27:0/839208418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0da4020540 tx=0x7f0da401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7f74b015908901 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0d8f7fe640 1 -- 172.21.15.27:0/839208418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da401a040 con 0x55c00625c590 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0d8f7fe640 1 -- 172.21.15.27:0/839208418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da4020660 con 0x55c00625c590 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.685+0000 7f0d8f7fe640 1 -- 172.21.15.27:0/839208418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4015c30 con 0x55c00625c590 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.686+0000 7f0db7004740 1 -- 172.21.15.27:0/839208418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 msgr2=0x55c006248150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.686+0000 7f0db7004740 1 --2- 172.21.15.27:0/839208418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0da4020540 tx=0x7f0da401e000 comp rx=0 tx=0).stop 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.686+0000 7f0db7004740 1 -- 172.21.15.27:0/839208418 shutdown_connections 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.686+0000 7f0db7004740 1 --2- 172.21.15.27:0/839208418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.686+0000 7f0db7004740 1 --2- 172.21.15.27:0/839208418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006249fe0 0x55c006247c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.686+0000 7f0db7004740 1 --2- 172.21.15.27:0/839208418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006247600 0x55c006216700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.686+0000 7f0db7004740 1 -- 172.21.15.27:0/839208418 >> 172.21.15.27:0/839208418 conn(0x55c006217870 msgr2=0x55c006275b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.686+0000 7f0db7004740 1 -- 172.21.15.27:0/839208418 shutdown_connections 2024-03-06T07:24:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.686+0000 7f0db7004740 1 -- 172.21.15.27:0/839208418 wait complete. 2024-03-06T07:24:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.687+0000 7f0db7004740 1 -- start start 2024-03-06T07:24:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.687+0000 7f0db7004740 1 Processor -- start 2024-03-06T07:24:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.687+0000 7f0db7004740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006247600 0x55c006243190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db7004740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006249fe0 0x55c006247c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db7004740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db7004740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c00624a620 con 0x55c00625c590 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db7004740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c006273860 con 0x55c006249fe0 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db7004740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c00625e9a0 con 0x55c006247600 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db7004740 10 client.0 _ll_register_callbacks cb 0x7f0d9c076560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db29b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db29b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46272/0 (socket says 172.21.15.27:46272) 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db29b2640 1 -- 172.21.15.27:0/3779604676 learned_addr learned my addr 172.21.15.27:0/3779604676 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db19b0640 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006249fe0 0x55c006247c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db21b1640 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006247600 0x55c006243190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db29b2640 1 -- 172.21.15.27:0/3779604676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006247600 msgr2=0x55c006243190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db29b2640 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006247600 0x55c006243190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db29b2640 1 -- 172.21.15.27:0/3779604676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006249fe0 msgr2=0x55c006247c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db29b2640 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006249fe0 0x55c006247c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db29b2640 1 -- 172.21.15.27:0/3779604676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c006244240 con 0x55c00625c590 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db29b2640 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0da4001950 tx=0x7f0da4001490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ae6a7eb0ea3c60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/3779604676 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da403a060 con 0x55c00625c590 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/3779604676 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0da4020c20 con 0x55c00625c590 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.688+0000 7f0db7004740 1 -- 172.21.15.27:0/3779604676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0da4020660 con 0x55c00625c590 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/3779604676 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0da4040050 con 0x55c00625c590 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db7004740 10 client.5910 fetch_fsmap learned FSMap version 26 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db7004740 1 -- 172.21.15.27:0/3779604676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55c006243b40 con 0x55c00625c590 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/3779604676 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da4020660 con 0x55c00625c590 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/3779604676 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0da4043020 con 0x55c00625c590 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db7004740 10 client.5910 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:24:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db7004740 10 client.5910 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db7004740 20 client.5910 populate_metadata read hostname 'smithi027' 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db7004740 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] conn(0x55c006212fc0 0x55c006285560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db7004740 4 client.5910 mds_command: new command op to 5754 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db7004740 1 -- 172.21.15.27:0/3779604676 --> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c0062176c0 con 0x55c006212fc0 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db21b1640 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] conn(0x55c006212fc0 0x55c006285560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0d8d7fa640 10 client.5910 ms_handle_connect on v2:172.21.15.27:6833/2008146374 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.689+0000 7f0db21b1640 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] conn(0x55c006212fc0 0x55c006285560 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4458496eef502e0 server_cookie=37f45dbeb89c92e8 in_seq=0 out_seq=0 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0d8d7fa640 1 -- 172.21.15.27:0/3779604676 <== mds.0 v2:172.21.15.27:6833/2008146374 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55c0062176c0 con 0x55c006212fc0 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0d8d7fa640 10 client.5910 handle_command_reply: tid=0 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 client.5910 shutdown 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 -- 172.21.15.27:0/3779604676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 msgr2=0x55c006248150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0da4001950 tx=0x7f0da4001490 comp rx=0 tx=0).stop 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 -- 172.21.15.27:0/3779604676 shutdown_connections 2024-03-06T07:24:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.27:6833/2008146374,v1:172.21.15.27:6835/2008146374] conn(0x55c006212fc0 0x55c006285560 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c00625c590 0x55c006248150 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c006249fe0 0x55c006247c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 --2- 172.21.15.27:0/3779604676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c006247600 0x55c006243190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0d8d7fa640 0 client.5910 ms_handle_reset on v2:172.21.15.27:6833/2008146374 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 -- 172.21.15.27:0/3779604676 >> 172.21.15.27:0/3779604676 conn(0x55c006217870 msgr2=0x55c006266210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 -- 172.21.15.27:0/3779604676 shutdown_connections 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 1 -- 172.21.15.27:0/3779604676 wait complete. 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.690+0000 7f0db7004740 20 client.5910 trim_cache size 0 max 16384 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:24:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5754, 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 5.8645184810000002, 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 26, 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 46, 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 46, 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 10.87366222 2024-03-06T07:24:51.700 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:24:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 -- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d9006b4d0 msgr2=0x7f0d9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d9006b4d0 0x7f0d9006d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009dc0 tx=0x7f0da8009190 comp rx=0 tx=0).stop 2024-03-06T07:24:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 -- 172.21.15.27:0/4024100085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac147f90 msgr2=0x7f0dac159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac147f90 0x7f0dac159820 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c000c00 tx=0x7f0d9c001120 comp rx=0 tx=0).stop 2024-03-06T07:24:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 -- 172.21.15.27:0/4024100085 shutdown_connections 2024-03-06T07:24:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d9006b4d0 0x7f0d9006d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0dac148410 0x7f0dac159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac147f90 0x7f0dac159820 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 --2- 172.21.15.27:0/4024100085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d98004800 0x7f0dac152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 -- 172.21.15.27:0/4024100085 >> 172.21.15.27:0/4024100085 conn(0x7f0dac066d00 msgr2=0x7f0dac0d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 -- 172.21.15.27:0/4024100085 shutdown_connections 2024-03-06T07:24:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.691+0000 7f0db4415640 1 -- 172.21.15.27:0/4024100085 wait complete. 2024-03-06T07:24:51.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:51.801 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-06T07:24:51.799+0000 7f0bca61b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:51.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.841+0000 7f02a39bc640 1 Processor -- start 2024-03-06T07:24:51.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.842+0000 7f02a39bc640 1 -- start start 2024-03-06T07:24:51.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.842+0000 7f02a39bc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f029c06b5c0 0x7f029c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.842+0000 7f02a39bc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f029c0ae9b0 0x7f029c0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.842+0000 7f02a39bc640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c0b3340 con 0x7f029c0b0e90 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.842+0000 7f02a39bc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029c0cfa00 con 0x7f029c06b5c0 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.842+0000 7f02a39bc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029c0cee90 con 0x7f029c0ae9b0 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.843+0000 7f02a1758640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f029c06b5c0 0x7f029c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.843+0000 7f02a0f57640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f029c0b0e90 0x7f029c06b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56226/0 (socket says 172.21.15.27:56226) 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.843+0000 7f02a0f57640 1 -- 172.21.15.27:0/3541114930 learned_addr learned my addr 172.21.15.27:0/3541114930 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.843+0000 7f02a1f59640 1 --2- 172.21.15.27:0/3541114930 >> v2:172.21.15.69:3300/0 conn(0x7f029c0ae9b0 0x7f029c0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.843+0000 7f02a1758640 1 -- 172.21.15.27:0/3541114930 >> v2:172.21.15.69:3300/0 conn(0x7f029c0ae9b0 msgr2=0x7f029c0d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.843+0000 7f02a1758640 1 --2- 172.21.15.27:0/3541114930 >> v2:172.21.15.69:3300/0 conn(0x7f029c0ae9b0 0x7f029c0d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.843+0000 7f02a1758640 1 -- 172.21.15.27:0/3541114930 >> v1:172.21.15.27:6789/0 conn(0x7f029c0b0e90 legacy=0x7f029c06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.843+0000 7f02a1758640 1 -- 172.21.15.27:0/3541114930 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0cfc50 con 0x7f029c06b5c0 2024-03-06T07:24:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.843+0000 7f0293fff640 1 -- 172.21.15.27:0/3541114930 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467530016 0 0) 0x7f029c0b3340 con 0x7f029c0b0e90 2024-03-06T07:24:51.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f02a1758640 1 --2- 172.21.15.27:0/3541114930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c06b5c0 0x7f029c0b2e00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f029c072170 tx=0x7f028c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ebd0116da36fa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f0293fff640 1 -- 172.21.15.27:0/3541114930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c0315a0 con 0x7f029c06b5c0 2024-03-06T07:24:51.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f0293fff640 1 -- 172.21.15.27:0/3541114930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c037070 con 0x7f029c06b5c0 2024-03-06T07:24:51.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f0293fff640 1 -- 172.21.15.27:0/3541114930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c031960 con 0x7f029c06b5c0 2024-03-06T07:24:51.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f02a39bc640 1 -- 172.21.15.27:0/3541114930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c06b5c0 msgr2=0x7f029c0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f02a39bc640 1 --2- 172.21.15.27:0/3541114930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c06b5c0 0x7f029c0b2e00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f029c072170 tx=0x7f028c02b840 comp rx=0 tx=0).stop 2024-03-06T07:24:51.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f02a39bc640 1 -- 172.21.15.27:0/3541114930 shutdown_connections 2024-03-06T07:24:51.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f02a39bc640 1 --2- 172.21.15.27:0/3541114930 >> v2:172.21.15.69:3300/0 conn(0x7f029c0ae9b0 0x7f029c0d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f02a39bc640 1 --2- 172.21.15.27:0/3541114930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c06b5c0 0x7f029c0b2e00 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f02a39bc640 1 -- 172.21.15.27:0/3541114930 >> 172.21.15.27:0/3541114930 conn(0x7f029c0b36d0 msgr2=0x7f029c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.844+0000 7f02a39bc640 1 -- 172.21.15.27:0/3541114930 shutdown_connections 2024-03-06T07:24:51.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.845+0000 7f02a39bc640 1 -- 172.21.15.27:0/3541114930 wait complete. 2024-03-06T07:24:51.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.846+0000 7f02a39bc640 1 Processor -- start 2024-03-06T07:24:51.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.846+0000 7f02a39bc640 1 -- start start 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.846+0000 7f02a39bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f029c06b5c0 0x7f029c1d5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.846+0000 7f02a39bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f029c0ae9b0 0x7f029c0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.846+0000 7f02a39bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c0b0e90 0x7f029c1e29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.846+0000 7f02a39bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c06c910 con 0x7f029c0b0e90 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.846+0000 7f02a39bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c06c790 con 0x7f029c06b5c0 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.846+0000 7f02a39bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0aedc0 con 0x7f029c0ae9b0 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1f59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c0b0e90 0x7f029c1e29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1f59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c0b0e90 0x7f029c1e29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46280/0 (socket says 172.21.15.27:46280) 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1f59640 1 -- 172.21.15.27:0/693755928 learned_addr learned my addr 172.21.15.27:0/693755928 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1758640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f029c06b5c0 0x7f029c1d5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a0f57640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f029c0ae9b0 0x7f029c0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1f59640 1 -- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f029c0ae9b0 msgr2=0x7f029c0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1f59640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f029c0ae9b0 0x7f029c0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1f59640 1 -- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f029c06b5c0 msgr2=0x7f029c1d5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1f59640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f029c06b5c0 0x7f029c1d5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1f59640 1 -- 172.21.15.27:0/693755928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0c5ac0 con 0x7f029c0b0e90 2024-03-06T07:24:51.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1758640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f029c06b5c0 0x7f029c1d5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a1f59640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c0b0e90 0x7f029c1e29d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0294007e70 tx=0x7f029400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f0291ffb640 1 -- 172.21.15.27:0/693755928 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294014620 con 0x7f029c0b0e90 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f0291ffb640 1 -- 172.21.15.27:0/693755928 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0294019070 con 0x7f029c0b0e90 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f0291ffb640 1 -- 172.21.15.27:0/693755928 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294014a20 con 0x7f029c0b0e90 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.847+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029c1e6b80 con 0x7f029c0b0e90 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.848+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029c1e7140 con 0x7f029c0b0e90 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.849+0000 7f0291ffb640 1 -- 172.21.15.27:0/693755928 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0294014bc0 con 0x7f029c0b0e90 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.849+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029c068700 con 0x7f029c0b0e90 2024-03-06T07:24:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.849+0000 7f0291ffb640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f028006b4d0 0x7f028006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.849+0000 7f0291ffb640 1 -- 172.21.15.27:0/693755928 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f029408aab0 con 0x7f029c0b0e90 2024-03-06T07:24:51.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.849+0000 7f02a1758640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f028006b4d0 0x7f028006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.850+0000 7f02a1758640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f028006b4d0 0x7f028006d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f028c036000 tx=0x7f028c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:51.854+0000 7f0291ffb640 1 -- 172.21.15.27:0/693755928 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0294059930 con 0x7f029c0b0e90 2024-03-06T07:24:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.071+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f029c1e7310 con 0x7f029c0b0e90 2024-03-06T07:24:52.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.071+0000 7f0291ffb640 1 -- 172.21.15.27:0/693755928 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0294059230 con 0x7f029c0b0e90 2024-03-06T07:24:52.074 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f028006b4d0 msgr2=0x7f028006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f028006b4d0 0x7f028006d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f028c036000 tx=0x7f028c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c0b0e90 msgr2=0x7f029c1e29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c0b0e90 0x7f029c1e29d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0294007e70 tx=0x7f029400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 shutdown_connections 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f028006b4d0 0x7f028006d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f029c0b0e90 0x7f029c1e29d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f029c0ae9b0 0x7f029c0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 --2- 172.21.15.27:0/693755928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f029c06b5c0 0x7f029c1d5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 >> 172.21.15.27:0/693755928 conn(0x7f029c0b36d0 msgr2=0x7f029c1e5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 shutdown_connections 2024-03-06T07:24:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:52.074+0000 7f02a39bc640 1 -- 172.21.15.27:0/693755928 wait complete. 2024-03-06T07:24:52.092 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:52.092 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:52.092 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:24:52.092 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:52.092 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:52.093 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:52.093 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:52.093 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:52.093 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:52.093 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42598sec preferred_lft 42598sec 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:24:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:52.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:52.112 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:52.178 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:24:52.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:52.180 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:24:52.235 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:24:52.236 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:52.236 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:24:52.236 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:24:52.236 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:52.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:52.236 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:24:52.236 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:24:52.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:24:52.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:24:52.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:24:52.237 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:24:52.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:52.344 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:24:52.358 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:52.358 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:52.359 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:52.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:52.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:24:52.430 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:52.430 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:52.431 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:52.485 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:52.485 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:52.485 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:52.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:24:52.613 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:52.668 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:24:52.668 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:52.669 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:24:52.670 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:52.671 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:52.671 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:52.671 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:52.671 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:24:52.671 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:52.671 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:52.671 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:52.671 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:52.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:52.672 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:24:52.727 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:52.727 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:52.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:24:52.800 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:52.859 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:24:52.860 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:52.862 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:52.862 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:52.862 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:24:52.862 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:52.862 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:52.862 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:52.862 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:52.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:52.862 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:24:52.869 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:24:52.866+0000 7f90b9445480 -1 init, newargv = 0x55def828e5c0 newargc=15 2024-03-06T07:24:52.869 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[56033]: starting ceph client 2024-03-06T07:24:52.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[56033]: starting fuse 2024-03-06T07:24:52.885 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:24:52.885 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:24:52.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:52.885 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:24:52.886 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:52.887 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:24:53.010 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.56033.asok 2024-03-06T07:24:53.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:53.021 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56033.asok 2024-03-06T07:24:53.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56033.asok status 2024-03-06T07:24:53.163 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:24:53.163 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:24:53.163 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "56033", 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5922, 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:24:53.164 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 5922 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1027561497 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1027561497 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.5922 192.168.0.1:0/1027561497", 2024-03-06T07:24:53.165 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/1027561497", 2024-03-06T07:24:53.166 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:24:53.166 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 26, 2024-03-06T07:24:53.166 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 46, 2024-03-06T07:24:53.166 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:53.166 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:24:53.166 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:24:53.166 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:24:53.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.172 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:24:53.172 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:53.187 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:24:53.187 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:53.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:53.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:53.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:24:53.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.406+0000 7f32d960e640 1 Processor -- start 2024-03-06T07:24:53.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.406+0000 7f32d960e640 1 -- start start 2024-03-06T07:24:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.407+0000 7f32d960e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32d40b2200 0x7f32d40b2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.407+0000 7f32d960e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f32d40c5250 0x7f32d416bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.407+0000 7f32d960e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40cf490 con 0x7f32d40b3190 2024-03-06T07:24:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.407+0000 7f32d960e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d40cef60 con 0x7f32d40b2200 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.407+0000 7f32d960e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d406b610 con 0x7f32d40c5250 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.407+0000 7f32d2ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32d40b2200 0x7f32d40b2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.407+0000 7f32d2ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32d40b2200 0x7f32d40b2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46284/0 (socket says 172.21.15.27:46284) 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.407+0000 7f32d2ffd640 1 -- 172.21.15.27:0/1120608402 learned_addr learned my addr 172.21.15.27:0/1120608402 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.407+0000 7f32d37fe640 1 --2- 172.21.15.27:0/1120608402 >> v2:172.21.15.69:3300/0 conn(0x7f32d40c5250 0x7f32d416bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.408+0000 7f32d2ffd640 1 -- 172.21.15.27:0/1120608402 >> v2:172.21.15.69:3300/0 conn(0x7f32d40c5250 msgr2=0x7f32d416bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.408+0000 7f32d2ffd640 1 --2- 172.21.15.27:0/1120608402 >> v2:172.21.15.69:3300/0 conn(0x7f32d40c5250 0x7f32d416bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.408+0000 7f32d2ffd640 1 -- 172.21.15.27:0/1120608402 >> v1:172.21.15.69:6789/0 conn(0x7f32d40b3190 legacy=0x7f32d40c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.408+0000 7f32d2ffd640 1 -- 172.21.15.27:0/1120608402 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d40cf1b0 con 0x7f32d40b2200 2024-03-06T07:24:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.408+0000 7f32d37fe640 1 --2- 172.21.15.27:0/1120608402 >> v2:172.21.15.69:3300/0 conn(0x7f32d40c5250 0x7f32d416bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.409+0000 7f32d2ffd640 1 --2- 172.21.15.27:0/1120608402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40b2200 0x7f32d40b2b40 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f32d40b47a0 tx=0x7f32c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ac765efc6d7a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.409+0000 7f32d1ffb640 1 -- 172.21.15.27:0/1120608402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4035020 con 0x7f32d40b2200 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.409+0000 7f32d1ffb640 1 -- 172.21.15.27:0/1120608402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c4031870 con 0x7f32d40b2200 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.409+0000 7f32d1ffb640 1 -- 172.21.15.27:0/1120608402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4031c40 con 0x7f32d40b2200 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.410+0000 7f32d960e640 1 -- 172.21.15.27:0/1120608402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40b2200 msgr2=0x7f32d40b2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.410+0000 7f32d960e640 1 --2- 172.21.15.27:0/1120608402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40b2200 0x7f32d40b2b40 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f32d40b47a0 tx=0x7f32c402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.410+0000 7f32d960e640 1 -- 172.21.15.27:0/1120608402 shutdown_connections 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.410+0000 7f32d960e640 1 --2- 172.21.15.27:0/1120608402 >> v2:172.21.15.69:3300/0 conn(0x7f32d40c5250 0x7f32d416bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.410+0000 7f32d960e640 1 --2- 172.21.15.27:0/1120608402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40b2200 0x7f32d40b2b40 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.410+0000 7f32d960e640 1 -- 172.21.15.27:0/1120608402 >> 172.21.15.27:0/1120608402 conn(0x7f32d40b3700 msgr2=0x7f32d40b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.410+0000 7f32d960e640 1 -- 172.21.15.27:0/1120608402 shutdown_connections 2024-03-06T07:24:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.410+0000 7f32d960e640 1 -- 172.21.15.27:0/1120608402 wait complete. 2024-03-06T07:24:53.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.412+0000 7f32d960e640 1 Processor -- start 2024-03-06T07:24:53.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.412+0000 7f32d960e640 1 -- start start 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.413+0000 7f32d960e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d40b2200 0x7f32d41530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.413+0000 7f32d960e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d40b3190 0x7f32d40b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.413+0000 7f32d960e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c5250 0x7f32d40af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.413+0000 7f32d960e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d40d0990 con 0x7f32d40c5250 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.413+0000 7f32d960e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d406b2a0 con 0x7f32d40b3190 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.413+0000 7f32d960e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d40d0b10 con 0x7f32d40b2200 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c5250 0x7f32d40af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c5250 0x7f32d40af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46290/0 (socket says 172.21.15.27:46290) 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d37fe640 1 -- 172.21.15.27:0/1373237105 learned_addr learned my addr 172.21.15.27:0/1373237105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d37fe640 1 -- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d40b2200 msgr2=0x7f32d41530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d37fe640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d40b2200 0x7f32d41530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d37fe640 1 -- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d40b3190 msgr2=0x7f32d40b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d37fe640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d40b3190 0x7f32d40b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d37fe640 1 -- 172.21.15.27:0/1373237105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d40b1720 con 0x7f32d40c5250 2024-03-06T07:24:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d37fe640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c5250 0x7f32d40af410 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f32cc007ec0 tx=0x7f32cc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32cb7fe640 1 -- 172.21.15.27:0/1373237105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc01a070 con 0x7f32d40c5250 2024-03-06T07:24:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32cb7fe640 1 -- 172.21.15.27:0/1373237105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32cc003b90 con 0x7f32d40c5250 2024-03-06T07:24:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32cb7fe640 1 -- 172.21.15.27:0/1373237105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc004030 con 0x7f32d40c5250 2024-03-06T07:24:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d40b0140 con 0x7f32d40c5250 2024-03-06T07:24:53.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.414+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d40b0700 con 0x7f32d40c5250 2024-03-06T07:24:53.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.415+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d409fcf0 con 0x7f32d40c5250 2024-03-06T07:24:53.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.416+0000 7f32cb7fe640 1 -- 172.21.15.27:0/1373237105 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32cc01d070 con 0x7f32d40c5250 2024-03-06T07:24:53.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.416+0000 7f32cb7fe640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32b406b500 0x7f32b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.416+0000 7f32cb7fe640 1 -- 172.21.15.27:0/1373237105 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32cc08a420 con 0x7f32d40c5250 2024-03-06T07:24:53.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.417+0000 7f32d2ffd640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32b406b500 0x7f32b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.417+0000 7f32d2ffd640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32b406b500 0x7f32b406d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f32c4036000 tx=0x7f32c4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.421+0000 7f32cb7fe640 1 -- 172.21.15.27:0/1373237105 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32cc010070 con 0x7f32d40c5250 2024-03-06T07:24:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.602+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f32d40cbc70 con 0x7f32d40c5250 2024-03-06T07:24:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.605+0000 7f32cb7fe640 1 -- 172.21.15.27:0/1373237105 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7f32cc059330 con 0x7f32d40c5250 2024-03-06T07:24:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32b406b500 msgr2=0x7f32b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32b406b500 0x7f32b406d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f32c4036000 tx=0x7f32c4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c5250 msgr2=0x7f32d40af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c5250 0x7f32d40af410 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f32cc007ec0 tx=0x7f32cc00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 shutdown_connections 2024-03-06T07:24:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32b406b500 0x7f32b406d9c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c5250 0x7f32d40af410 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d40b3190 0x7f32d40b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 --2- 172.21.15.27:0/1373237105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d40b2200 0x7f32d41530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 >> 172.21.15.27:0/1373237105 conn(0x7f32d40b3700 msgr2=0x7f32d40cbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 shutdown_connections 2024-03-06T07:24:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:24:53.608+0000 7f32d960e640 1 -- 172.21.15.27:0/1373237105 wait complete. 2024-03-06T07:25:03.621 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:03.621 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:03.637 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:25:03.637 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:25:03.637 DEBUG:teuthology.orchestra.run.smithi027:> import uuid 2024-03-06T07:25:03.637 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:03.637 DEBUG:teuthology.orchestra.run.smithi027:> n = 51 2024-03-06T07:25:03.637 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = False 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> if 0 > 0: 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = True 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> dpath = os.path.dirname(path) 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> try: 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> for i in range(n): 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> f.write(f"{i}") 2024-03-06T07:25:03.638 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> f.flush() 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(f.fileno()) 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> os.unlink(fpath) 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> if create_hard_links: 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> for j in range(0): 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> if True: 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> finally: 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> os.close(dirfd) 2024-03-06T07:25:03.639 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:25:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:Traceback (most recent call last): 2024-03-06T07:25:03.844 INFO:teuthology.orchestra.run.smithi027.stderr: File "", line 20, in 2024-03-06T07:25:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_50' 2024-03-06T07:25:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:03 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:25:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:03.854 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:03.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:03.942 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:03.993 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:03.993 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:03.993 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:04.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:04.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.153+0000 7fe89f535640 1 Processor -- start 2024-03-06T07:25:04.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.154+0000 7fe89f535640 1 -- start start 2024-03-06T07:25:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.154+0000 7fe89f535640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe89815df00 0x7fe8980d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.154+0000 7fe89f535640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8980b9a30 0x7fe8980d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.154+0000 7fe89f535640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8980cea30 con 0x7fe8980b95a0 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.154+0000 7fe89f535640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8980d2af0 con 0x7fe89815df00 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.154+0000 7fe89f535640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980cf3c0 con 0x7fe8980b9a30 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89d2d1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe89815df00 0x7fe8980d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89dad2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8980b9a30 0x7fe8980d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89d2d1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe89815df00 0x7fe8980d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:34028/0 (socket says 172.21.15.27:34028) 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89dad2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8980b9a30 0x7fe8980d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:60426/0 (socket says 172.21.15.27:60426) 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89d2d1640 1 -- 172.21.15.27:0/1828996684 learned_addr learned my addr 172.21.15.27:0/1828996684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89cad0640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe8980b95a0 0x7fe8980b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:52560/0 (socket says 172.21.15.27:52560) 2024-03-06T07:25:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89dad2640 1 -- 172.21.15.27:0/1828996684 >> v2:172.21.15.69:3300/0 conn(0x7fe89815df00 msgr2=0x7fe8980d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89dad2640 1 --2- 172.21.15.27:0/1828996684 >> v2:172.21.15.69:3300/0 conn(0x7fe89815df00 0x7fe8980d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89dad2640 1 -- 172.21.15.27:0/1828996684 >> v1:172.21.15.69:6789/0 conn(0x7fe8980b95a0 legacy=0x7fe8980b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89dad2640 1 -- 172.21.15.27:0/1828996684 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980cf0e0 con 0x7fe8980b9a30 2024-03-06T07:25:04.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.155+0000 7fe89d2d1640 1 --2- 172.21.15.27:0/1828996684 >> v2:172.21.15.69:3300/0 conn(0x7fe89815df00 0x7fe8980d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:04.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.156+0000 7fe89dad2640 1 --2- 172.21.15.27:0/1828996684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8980b9a30 0x7fe8980d3220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe890009d90 tx=0x7fe89002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b0854ffb3e7ffcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.156+0000 7fe88ffff640 1 -- 172.21.15.27:0/1828996684 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890039070 con 0x7fe8980b9a30 2024-03-06T07:25:04.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.156+0000 7fe88ffff640 1 -- 172.21.15.27:0/1828996684 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89002cc50 con 0x7fe8980b9a30 2024-03-06T07:25:04.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.156+0000 7fe88ffff640 1 -- 172.21.15.27:0/1828996684 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890034440 con 0x7fe8980b9a30 2024-03-06T07:25:04.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.157+0000 7fe89f535640 1 -- 172.21.15.27:0/1828996684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8980b9a30 msgr2=0x7fe8980d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.157+0000 7fe89f535640 1 --2- 172.21.15.27:0/1828996684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8980b9a30 0x7fe8980d3220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe890009d90 tx=0x7fe89002bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.157+0000 7fe89f535640 1 -- 172.21.15.27:0/1828996684 shutdown_connections 2024-03-06T07:25:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.157+0000 7fe89f535640 1 --2- 172.21.15.27:0/1828996684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8980b9a30 0x7fe8980d3220 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.157+0000 7fe89f535640 1 --2- 172.21.15.27:0/1828996684 >> v2:172.21.15.69:3300/0 conn(0x7fe89815df00 0x7fe8980d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.157+0000 7fe89f535640 1 -- 172.21.15.27:0/1828996684 >> 172.21.15.27:0/1828996684 conn(0x7fe898071960 msgr2=0x7fe898071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.157+0000 7fe89f535640 1 -- 172.21.15.27:0/1828996684 shutdown_connections 2024-03-06T07:25:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.157+0000 7fe89f535640 1 -- 172.21.15.27:0/1828996684 wait complete. 2024-03-06T07:25:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.158+0000 7fe89f535640 1 Processor -- start 2024-03-06T07:25:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.158+0000 7fe89f535640 1 -- start start 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89f535640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8980b95a0 0x7fe89815ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89f535640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe89815df00 0x7fe89815f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89f535640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8981f7070 0x7fe8981f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89f535640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980cec70 con 0x7fe89815df00 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89f535640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980ce750 con 0x7fe8980b95a0 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89f535640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980cedf0 con 0x7fe8981f7070 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89cad0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe89815df00 0x7fe89815f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89cad0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe89815df00 0x7fe89815f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33216/0 (socket says 172.21.15.27:33216) 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89cad0640 1 -- 172.21.15.27:0/946291692 learned_addr learned my addr 172.21.15.27:0/946291692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89dad2640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8981f7070 0x7fe8981f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89d2d1640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8980b95a0 0x7fe89815ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89cad0640 1 -- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8981f7070 msgr2=0x7fe8981f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89cad0640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8981f7070 0x7fe8981f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89cad0640 1 -- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8980b95a0 msgr2=0x7fe89815ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89cad0640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8980b95a0 0x7fe89815ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89cad0640 1 -- 172.21.15.27:0/946291692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980ca510 con 0x7fe89815df00 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe89cad0640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe89815df00 0x7fe89815f3a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c400 tx=0x7fe89400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.159+0000 7fe88dffb640 1 -- 172.21.15.27:0/946291692 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8940074e0 con 0x7fe89815df00 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.160+0000 7fe88dffb640 1 -- 172.21.15.27:0/946291692 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe894007680 con 0x7fe89815df00 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.160+0000 7fe88dffb640 1 -- 172.21.15.27:0/946291692 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894004a00 con 0x7fe89815df00 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.160+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8981f7550 con 0x7fe89815df00 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.160+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8981f7b10 con 0x7fe89815df00 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.161+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe898067ff0 con 0x7fe89815df00 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.161+0000 7fe88dffb640 1 -- 172.21.15.27:0/946291692 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe89401a020 con 0x7fe89815df00 2024-03-06T07:25:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.161+0000 7fe88dffb640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe87c06b4d0 0x7fe87c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.161+0000 7fe88dffb640 1 -- 172.21.15.27:0/946291692 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe894089910 con 0x7fe89815df00 2024-03-06T07:25:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.162+0000 7fe89d2d1640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe87c06b4d0 0x7fe87c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.162+0000 7fe89d2d1640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe87c06b4d0 0x7fe87c06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe888009630 tx=0x7fe888009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.166+0000 7fe88dffb640 1 -- 172.21.15.27:0/946291692 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe894058120 con 0x7fe89815df00 2024-03-06T07:25:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.347+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8980687a0 con 0x7fe89815df00 2024-03-06T07:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.348+0000 7fe88dffb640 1 -- 172.21.15.27:0/946291692 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fe89405c1c0 con 0x7fe89815df00 2024-03-06T07:25:04.350 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:04.350 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:24:46.825646+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:43.493554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:44.172410+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe87c06b4d0 msgr2=0x7fe87c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe87c06b4d0 0x7fe87c06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe888009630 tx=0x7fe888009240 comp rx=0 tx=0).stop 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe89815df00 msgr2=0x7fe89815f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe89815df00 0x7fe89815f3a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c400 tx=0x7fe89400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 shutdown_connections 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe87c06b4d0 0x7fe87c06d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8981f7070 0x7fe8981f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe89815df00 0x7fe89815f3a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 --2- 172.21.15.27:0/946291692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8980b95a0 0x7fe89815ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 >> 172.21.15.27:0/946291692 conn(0x7fe898071960 msgr2=0x7fe8980b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 shutdown_connections 2024-03-06T07:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.350+0000 7fe89f535640 1 -- 172.21.15.27:0/946291692 wait complete. 2024-03-06T07:25:04.362 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:43.493554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:44.172410+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:04.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.471+0000 7f12189e2640 1 Processor -- start 2024-03-06T07:25:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.471+0000 7f12189e2640 1 -- start start 2024-03-06T07:25:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.472+0000 7f12189e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12140b2d30 0x7f12140b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.472+0000 7f12189e2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12140b83b0 con 0x7f1214071540 2024-03-06T07:25:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.472+0000 7f12189e2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12140d4960 con 0x7f12140b7820 2024-03-06T07:25:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.472+0000 7f12189e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12140d3230 con 0x7f12140b2d30 2024-03-06T07:25:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.472+0000 7f121259c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1214071540 0x7f1214071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41928/0 (socket says 172.21.15.27:41928) 2024-03-06T07:25:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.472+0000 7f121259c640 1 -- 172.21.15.27:0/82378898 learned_addr learned my addr 172.21.15.27:0/82378898 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f1211d9b640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12140b2d30 0x7f12140b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916880293 0 0) 0x7f12140b83b0 con 0x7f1214071540 2024-03-06T07:25:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11fc003520 con 0x7f1214071540 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168798938 0 0) 0x7f12140d4960 con 0x7f12140b7820 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12140b83b0 con 0x7f12140b7820 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3122987243 0 0) 0x7f11fc003520 con 0x7f1214071540 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12140b2d30 msgr2=0x7f12140b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12140b2d30 0x7f12140b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 >> v1:172.21.15.69:6789/0 conn(0x7f12140b7820 legacy=0x7f12140b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12140d36c0 con 0x7f1214071540 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f1208001670 con 0x7f1214071540 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 >> v1:172.21.15.27:6789/0 conn(0x7f1214071540 legacy=0x7f1214071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc003b70 0x7f11fc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc004800 0x7f11fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc0051c0 0x7f11fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12140d3230 con 0x7f11fc004800 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc00bfb0 con 0x7f11fc003b70 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc00c1b0 con 0x7f11fc0051c0 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f1211d9b640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12140b2d30 0x7f12140b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f121259c640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc003b70 0x7f11fc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f1212d9d640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc0051c0 0x7f11fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.473+0000 7f1211d9b640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc004800 0x7f11fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.474+0000 7f121259c640 1 -- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc0051c0 msgr2=0x7f11fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.474+0000 7f121259c640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc0051c0 0x7f11fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.474+0000 7f121259c640 1 -- 172.21.15.27:0/82378898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc004800 msgr2=0x7f11fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.474+0000 7f121259c640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc004800 0x7f11fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.474+0000 7f121259c640 1 -- 172.21.15.27:0/82378898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc003520 con 0x7f11fc003b70 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f121259c640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc003b70 0x7f11fc003f70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f120800c1c0 tx=0x7f11fc005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab179cf70a288116 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1208007cc0 con 0x7f11fc003b70 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1208030070 con 0x7f11fc003b70 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f121159a640 1 -- 172.21.15.27:0/82378898 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1208005d10 con 0x7f11fc003b70 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f12189e2640 1 -- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc003b70 msgr2=0x7f11fc003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f12189e2640 1 --2- 172.21.15.27:0/82378898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc003b70 0x7f11fc003f70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f120800c1c0 tx=0x7f11fc005830 comp rx=0 tx=0).stop 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f1212d9d640 1 -- 172.21.15.27:0/82378898 reap_dead start 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f12189e2640 1 -- 172.21.15.27:0/82378898 shutdown_connections 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f12189e2640 1 -- 172.21.15.27:0/82378898 >> 172.21.15.27:0/82378898 conn(0x7f12140d3d60 msgr2=0x7f12140d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f12189e2640 1 -- 172.21.15.27:0/82378898 shutdown_connections 2024-03-06T07:25:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.475+0000 7f12189e2640 1 -- 172.21.15.27:0/82378898 wait complete. 2024-03-06T07:25:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.477+0000 7f12189e2640 1 Processor -- start 2024-03-06T07:25:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.477+0000 7f12189e2640 1 -- start start 2024-03-06T07:25:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.477+0000 7f12189e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc003b70 0x7f12140cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.477+0000 7f12189e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12141503e0 0x7f12141e5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.477+0000 7f12189e2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12140cbd80 0x7f12141e59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.477+0000 7f12189e2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12140d2be0 con 0x7f11fc003b70 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.477+0000 7f12189e2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12140d2900 con 0x7f12140cbd80 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.477+0000 7f12189e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12140d2ec0 con 0x7f12141503e0 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f121259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc003b70 0x7f12140cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f121259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc003b70 0x7f12140cb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33224/0 (socket says 172.21.15.27:33224) 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f121259c640 1 -- 172.21.15.27:0/2168435491 learned_addr learned my addr 172.21.15.27:0/2168435491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f1212d9d640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12140cbd80 0x7f12141e59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f1211d9b640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12141503e0 0x7f12141e5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f121259c640 1 -- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12141503e0 msgr2=0x7f12141e5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f121259c640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12141503e0 0x7f12141e5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f121259c640 1 -- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12140cbd80 msgr2=0x7f12141e59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f121259c640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12140cbd80 0x7f12141e59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f121259c640 1 -- 172.21.15.27:0/2168435491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12140cc2e0 con 0x7f11fc003b70 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f121259c640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc003b70 0x7f12140cb840 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f120800c2e0 tx=0x7f12080019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f1202ffd640 1 -- 172.21.15.27:0/2168435491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1208045070 con 0x7f11fc003b70 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f1202ffd640 1 -- 172.21.15.27:0/2168435491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1208004bf0 con 0x7f11fc003b70 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12140cc5c0 con 0x7f11fc003b70 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f1202ffd640 1 -- 172.21.15.27:0/2168435491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1208001c10 con 0x7f11fc003b70 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.478+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12141f3ed0 con 0x7f11fc003b70 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.479+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121404f410 con 0x7f11fc003b70 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.480+0000 7f1202ffd640 1 -- 172.21.15.27:0/2168435491 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1208007070 con 0x7f11fc003b70 2024-03-06T07:25:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.480+0000 7f1202ffd640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11f406b500 0x7f11f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.481+0000 7f1202ffd640 1 -- 172.21.15.27:0/2168435491 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12080ab180 con 0x7f11fc003b70 2024-03-06T07:25:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.481+0000 7f1211d9b640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11f406b500 0x7f11f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.481+0000 7f1211d9b640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11f406b500 0x7f11f406d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f121415ba60 tx=0x7f1204008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.485+0000 7f1202ffd640 1 -- 172.21.15.27:0/2168435491 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1208079b20 con 0x7f11fc003b70 2024-03-06T07:25:04.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.666+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12140b6b20 con 0x7f11fc003b70 2024-03-06T07:25:04.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.666+0000 7f1202ffd640 1 -- 172.21.15.27:0/2168435491 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f120807dbc0 con 0x7f11fc003b70 2024-03-06T07:25:04.668 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11f406b500 msgr2=0x7f11f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11f406b500 0x7f11f406d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f121415ba60 tx=0x7f1204008040 comp rx=0 tx=0).stop 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc003b70 msgr2=0x7f12140cb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc003b70 0x7f12140cb840 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f120800c2e0 tx=0x7f12080019c0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 shutdown_connections 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11f406b500 0x7f11f406d9c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12140cbd80 0x7f12141e59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12141503e0 0x7f12141e5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 --2- 172.21.15.27:0/2168435491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc003b70 0x7f12140cb840 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 >> 172.21.15.27:0/2168435491 conn(0x7f12140d3d60 msgr2=0x7f12140b85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 shutdown_connections 2024-03-06T07:25:04.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.669+0000 7f12189e2640 1 -- 172.21.15.27:0/2168435491 wait complete. 2024-03-06T07:25:04.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.794+0000 7f151f902640 1 Processor -- start 2024-03-06T07:25:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.794+0000 7f151f902640 1 -- start start 2024-03-06T07:25:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.795+0000 7f151f902640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180ceea0 0x7f15180d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.795+0000 7f151f902640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15180d3740 con 0x7f151815dd90 2024-03-06T07:25:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.795+0000 7f151f902640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15180d3120 con 0x7f15180b6d10 2024-03-06T07:25:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.795+0000 7f151f902640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15180d3400 con 0x7f15180ceea0 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.795+0000 7f151d69e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f151815dd90 0x7f15180d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41936/0 (socket says 172.21.15.27:41936) 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.795+0000 7f151d69e640 1 -- 172.21.15.27:0/1579766366 learned_addr learned my addr 172.21.15.27:0/1579766366 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.795+0000 7f151ce9d640 1 --2- 172.21.15.27:0/1579766366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180ceea0 0x7f15180d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.795+0000 7f150ffff640 1 -- 172.21.15.27:0/1579766366 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478254923 0 0) 0x7f15180d3740 con 0x7f151815dd90 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.795+0000 7f150ffff640 1 -- 172.21.15.27:0/1579766366 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1504003520 con 0x7f151815dd90 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.796+0000 7f150ffff640 1 -- 172.21.15.27:0/1579766366 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351871059 0 0) 0x7f15180d3120 con 0x7f15180b6d10 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.796+0000 7f150ffff640 1 -- 172.21.15.27:0/1579766366 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15180d3740 con 0x7f15180b6d10 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.796+0000 7f151ce9d640 1 -- 172.21.15.27:0/1579766366 >> v1:172.21.15.69:6789/0 conn(0x7f15180b6d10 legacy=0x7f15180d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.796+0000 7f150ffff640 1 -- 172.21.15.27:0/1579766366 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218567404 0 0) 0x7f1504003520 con 0x7f151815dd90 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.796+0000 7f151ce9d640 1 -- 172.21.15.27:0/1579766366 >> v1:172.21.15.27:6789/0 conn(0x7f151815dd90 legacy=0x7f15180d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.796+0000 7f151ce9d640 1 -- 172.21.15.27:0/1579766366 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15180d39e0 con 0x7f15180ceea0 2024-03-06T07:25:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f151ce9d640 1 --2- 172.21.15.27:0/1579766366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180ceea0 0x7f15180d29d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1514009800 tx=0x7f151402b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5759862c8f8aee8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f150ffff640 1 -- 172.21.15.27:0/1579766366 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508001670 con 0x7f15180ceea0 2024-03-06T07:25:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f150ffff640 1 -- 172.21.15.27:0/1579766366 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514037070 con 0x7f15180ceea0 2024-03-06T07:25:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f150ffff640 1 -- 172.21.15.27:0/1579766366 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514031760 con 0x7f15180ceea0 2024-03-06T07:25:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f151f902640 1 -- 172.21.15.27:0/1579766366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180ceea0 msgr2=0x7f15180d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f151f902640 1 --2- 172.21.15.27:0/1579766366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180ceea0 0x7f15180d29d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1514009800 tx=0x7f151402b880 comp rx=0 tx=0).stop 2024-03-06T07:25:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f151f902640 1 -- 172.21.15.27:0/1579766366 shutdown_connections 2024-03-06T07:25:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f151f902640 1 --2- 172.21.15.27:0/1579766366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180ceea0 0x7f15180d29d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f151f902640 1 -- 172.21.15.27:0/1579766366 >> 172.21.15.27:0/1579766366 conn(0x7f1518071960 msgr2=0x7f1518071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f151f902640 1 -- 172.21.15.27:0/1579766366 shutdown_connections 2024-03-06T07:25:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.797+0000 7f151f902640 1 -- 172.21.15.27:0/1579766366 wait complete. 2024-03-06T07:25:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151f902640 1 Processor -- start 2024-03-06T07:25:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151f902640 1 -- start start 2024-03-06T07:25:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151f902640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180b6d10 0x7f1518145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151f902640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15180ceea0 0x7f15181464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151f902640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815dd90 0x7f1518160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151f902640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15180d48f0 con 0x7f151815dd90 2024-03-06T07:25:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151f902640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151815e170 con 0x7f15180ceea0 2024-03-06T07:25:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151f902640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15180d4a70 con 0x7f15180b6d10 2024-03-06T07:25:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151de9f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815dd90 0x7f1518160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151de9f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815dd90 0x7f1518160840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33236/0 (socket says 172.21.15.27:33236) 2024-03-06T07:25:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.799+0000 7f151de9f640 1 -- 172.21.15.27:0/1471175614 learned_addr learned my addr 172.21.15.27:0/1471175614 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151d69e640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180b6d10 0x7f1518145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151ce9d640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15180ceea0 0x7f15181464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151de9f640 1 -- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180b6d10 msgr2=0x7f1518145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151de9f640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180b6d10 0x7f1518145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151de9f640 1 -- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15180ceea0 msgr2=0x7f15181464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151de9f640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15180ceea0 0x7f15181464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151de9f640 1 -- 172.21.15.27:0/1471175614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15181f4b40 con 0x7f151815dd90 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151d69e640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180b6d10 0x7f1518145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151ce9d640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15180ceea0 0x7f15181464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151de9f640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815dd90 0x7f1518160840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f151000c790 tx=0x7f151000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f150dffb640 1 -- 172.21.15.27:0/1471175614 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510003b70 con 0x7f151815dd90 2024-03-06T07:25:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f150dffb640 1 -- 172.21.15.27:0/1471175614 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1510003d10 con 0x7f151815dd90 2024-03-06T07:25:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1518157120 con 0x7f151815dd90 2024-03-06T07:25:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f150dffb640 1 -- 172.21.15.27:0/1471175614 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151000aa50 con 0x7f151815dd90 2024-03-06T07:25:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.800+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15181576e0 con 0x7f151815dd90 2024-03-06T07:25:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.802+0000 7f150dffb640 1 -- 172.21.15.27:0/1471175614 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f151000abf0 con 0x7f151815dd90 2024-03-06T07:25:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.802+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1518072a30 con 0x7f151815dd90 2024-03-06T07:25:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.802+0000 7f150dffb640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14fc06b4d0 0x7f14fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.802+0000 7f150dffb640 1 -- 172.21.15.27:0/1471175614 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f151008a110 con 0x7f151815dd90 2024-03-06T07:25:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.802+0000 7f151d69e640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14fc06b4d0 0x7f14fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.803+0000 7f151d69e640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14fc06b4d0 0x7f14fc06d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f15080010e0 tx=0x7f1508009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.810+0000 7f150dffb640 1 -- 172.21.15.27:0/1471175614 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1510059020 con 0x7f151815dd90 2024-03-06T07:25:04.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.992+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f151804f410 con 0x7f151815dd90 2024-03-06T07:25:04.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.992+0000 7f150dffb640 1 -- 172.21.15.27:0/1471175614 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1510058920 con 0x7f151815dd90 2024-03-06T07:25:04.994 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:25:04.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14fc06b4d0 msgr2=0x7f14fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14fc06b4d0 0x7f14fc06d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f15080010e0 tx=0x7f1508009040 comp rx=0 tx=0).stop 2024-03-06T07:25:04.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815dd90 msgr2=0x7f1518160840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815dd90 0x7f1518160840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f151000c790 tx=0x7f151000cc50 comp rx=0 tx=0).stop 2024-03-06T07:25:04.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 shutdown_connections 2024-03-06T07:25:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f14fc06b4d0 0x7f14fc06d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815dd90 0x7f1518160840 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15180ceea0 0x7f15181464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 --2- 172.21.15.27:0/1471175614 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15180b6d10 0x7f1518145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 >> 172.21.15.27:0/1471175614 conn(0x7f1518071960 msgr2=0x7f15180b4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 shutdown_connections 2024-03-06T07:25:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:04.994+0000 7f151f902640 1 -- 172.21.15.27:0/1471175614 wait complete. 2024-03-06T07:25:05.006 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.115+0000 7fd765650640 1 Processor -- start 2024-03-06T07:25:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.115+0000 7fd765650640 1 -- start start 2024-03-06T07:25:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd765650640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7600717b0 0x7fd760071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd765650640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd7600b3060 0x7fd7600d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd765650640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76015e4b0 con 0x7fd7600b2b10 2024-03-06T07:25:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd765650640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7600d2e10 con 0x7fd7600717b0 2024-03-06T07:25:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd765650640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7600b97e0 con 0x7fd7600b3060 2024-03-06T07:25:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd75effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7600717b0 0x7fd760071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd75effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7600717b0 0x7fd760071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33252/0 (socket says 172.21.15.27:33252) 2024-03-06T07:25:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd75effd640 1 -- 172.21.15.27:0/3561086095 learned_addr learned my addr 172.21.15.27:0/3561086095 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd75e7fc640 1 --2- 172.21.15.27:0/3561086095 >> v2:172.21.15.69:3300/0 conn(0x7fd7600b3060 0x7fd7600d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd75effd640 1 -- 172.21.15.27:0/3561086095 >> v2:172.21.15.69:3300/0 conn(0x7fd7600b3060 msgr2=0x7fd7600d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd75effd640 1 --2- 172.21.15.27:0/3561086095 >> v2:172.21.15.69:3300/0 conn(0x7fd7600b3060 0x7fd7600d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd75effd640 1 -- 172.21.15.27:0/3561086095 >> v1:172.21.15.69:6789/0 conn(0x7fd7600b2b10 legacy=0x7fd7600b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd75effd640 1 -- 172.21.15.27:0/3561086095 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600b9a80 con 0x7fd7600717b0 2024-03-06T07:25:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.116+0000 7fd75e7fc640 1 --2- 172.21.15.27:0/3561086095 >> v2:172.21.15.69:3300/0 conn(0x7fd7600b3060 0x7fd7600d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.117+0000 7fd75effd640 1 --2- 172.21.15.27:0/3561086095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 0x7fd760071b90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd76006ce10 tx=0x7fd75002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654bf0f92444c320 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd75dffb640 1 -- 172.21.15.27:0/3561086095 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750034070 con 0x7fd7600717b0 2024-03-06T07:25:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd75dffb640 1 -- 172.21.15.27:0/3561086095 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd750030860 con 0x7fd7600717b0 2024-03-06T07:25:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd75dffb640 1 -- 172.21.15.27:0/3561086095 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750030c30 con 0x7fd7600717b0 2024-03-06T07:25:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd765650640 1 -- 172.21.15.27:0/3561086095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 msgr2=0x7fd760071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd765650640 1 --2- 172.21.15.27:0/3561086095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 0x7fd760071b90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd76006ce10 tx=0x7fd75002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd765650640 1 -- 172.21.15.27:0/3561086095 shutdown_connections 2024-03-06T07:25:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd765650640 1 --2- 172.21.15.27:0/3561086095 >> v2:172.21.15.69:3300/0 conn(0x7fd7600b3060 0x7fd7600d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd765650640 1 --2- 172.21.15.27:0/3561086095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 0x7fd760071b90 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd765650640 1 -- 172.21.15.27:0/3561086095 >> 172.21.15.27:0/3561086095 conn(0x7fd7600d3d20 msgr2=0x7fd7600d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd765650640 1 -- 172.21.15.27:0/3561086095 shutdown_connections 2024-03-06T07:25:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.118+0000 7fd765650640 1 -- 172.21.15.27:0/3561086095 wait complete. 2024-03-06T07:25:05.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd765650640 1 Processor -- start 2024-03-06T07:25:05.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd765650640 1 -- start start 2024-03-06T07:25:05.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd765650640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 0x7fd76014dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd765650640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7600b2b10 0x7fd7600b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd765650640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7600b3060 0x7fd7600b9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd765650640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600b7560 con 0x7fd7600717b0 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd765650640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600d4bf0 con 0x7fd7600b2b10 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd765650640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600b76e0 con 0x7fd7600b3060 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd75effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 0x7fd76014dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd75effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 0x7fd76014dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33254/0 (socket says 172.21.15.27:33254) 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.120+0000 7fd75effd640 1 -- 172.21.15.27:0/2019383765 learned_addr learned my addr 172.21.15.27:0/2019383765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd75e7fc640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7600b2b10 0x7fd7600b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd75f7fe640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7600b3060 0x7fd7600b9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd75effd640 1 -- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7600b3060 msgr2=0x7fd7600b9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd75effd640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7600b3060 0x7fd7600b9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd75effd640 1 -- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7600b2b10 msgr2=0x7fd7600b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd75effd640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7600b2b10 0x7fd7600b8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd75effd640 1 -- 172.21.15.27:0/2019383765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7601f4150 con 0x7fd7600717b0 2024-03-06T07:25:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd75e7fc640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7600b2b10 0x7fd7600b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd75effd640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 0x7fd76014dfa0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd750000c00 tx=0x7fd750004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd7477fe640 1 -- 172.21.15.27:0/2019383765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750048070 con 0x7fd7600717b0 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd7477fe640 1 -- 172.21.15.27:0/2019383765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd750030660 con 0x7fd7600717b0 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7601f4430 con 0x7fd7600717b0 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd7477fe640 1 -- 172.21.15.27:0/2019383765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750034040 con 0x7fd7600717b0 2024-03-06T07:25:05.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.121+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7601f49f0 con 0x7fd7600717b0 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.122+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7600cabc0 con 0x7fd7600717b0 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.123+0000 7fd7477fe640 1 -- 172.21.15.27:0/2019383765 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd75004b070 con 0x7fd7600717b0 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.123+0000 7fd7477fe640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd74c06b4d0 0x7fd74c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.123+0000 7fd7477fe640 1 -- 172.21.15.27:0/2019383765 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7500ac350 con 0x7fd7600717b0 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.123+0000 7fd75e7fc640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd74c06b4d0 0x7fd74c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.124+0000 7fd75e7fc640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd74c06b4d0 0x7fd74c06d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd7600b9690 tx=0x7fd754008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.128+0000 7fd7477fe640 1 -- 172.21.15.27:0/2019383765 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7600cabc0 con 0x7fd7600717b0 2024-03-06T07:25:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.307+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd76006ce40 con 0x7fd7600717b0 2024-03-06T07:25:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.307+0000 7fd7477fe640 1 -- 172.21.15.27:0/2019383765 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd75007b260 con 0x7fd7600717b0 2024-03-06T07:25:05.309 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd74c06b4d0 msgr2=0x7fd74c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd74c06b4d0 0x7fd74c06d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd7600b9690 tx=0x7fd754008040 comp rx=0 tx=0).stop 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 msgr2=0x7fd76014dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 0x7fd76014dfa0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd750000c00 tx=0x7fd750004730 comp rx=0 tx=0).stop 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 shutdown_connections 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd74c06b4d0 0x7fd74c06d990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7600b3060 0x7fd7600b9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7600b2b10 0x7fd7600b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 --2- 172.21.15.27:0/2019383765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600717b0 0x7fd76014dfa0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 >> 172.21.15.27:0/2019383765 conn(0x7fd7600d3d20 msgr2=0x7fd7600b5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 shutdown_connections 2024-03-06T07:25:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.309+0000 7fd765650640 1 -- 172.21.15.27:0/2019383765 wait complete. 2024-03-06T07:25:05.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:05.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.430+0000 7fbfe14b5640 1 Processor -- start 2024-03-06T07:25:05.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.430+0000 7fbfe14b5640 1 -- start start 2024-03-06T07:25:05.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.431+0000 7fbfe14b5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfdc0b00c0 0x7fbfdc168b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.431+0000 7fbfe14b5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc161c20 con 0x7fbfdc15e080 2024-03-06T07:25:05.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.431+0000 7fbfe14b5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc161f00 con 0x7fbfdc15dca0 2024-03-06T07:25:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.431+0000 7fbfe14b5640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc1619d0 con 0x7fbfdc0b00c0 2024-03-06T07:25:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.431+0000 7fbfdb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfdc0b00c0 0x7fbfdc168b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.431+0000 7fbfda7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbfdc15e080 0x7fbfdc066d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41938/0 (socket says 172.21.15.27:41938) 2024-03-06T07:25:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.431+0000 7fbfda7fc640 1 -- 172.21.15.27:0/2346057331 learned_addr learned my addr 172.21.15.27:0/2346057331 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.431+0000 7fbfdb7fe640 1 -- 172.21.15.27:0/2346057331 >> v1:172.21.15.69:6789/0 conn(0x7fbfdc15dca0 legacy=0x7fbfdc06fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.432+0000 7fbfdb7fe640 1 -- 172.21.15.27:0/2346057331 >> v1:172.21.15.27:6789/0 conn(0x7fbfdc15e080 legacy=0x7fbfdc066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.432+0000 7fbfd9ffb640 1 -- 172.21.15.27:0/2346057331 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080580495 0 0) 0x7fbfdc161c20 con 0x7fbfdc15e080 2024-03-06T07:25:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.432+0000 7fbfdb7fe640 1 -- 172.21.15.27:0/2346057331 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc1621e0 con 0x7fbfdc0b00c0 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.432+0000 7fbfdb7fe640 1 --2- 172.21.15.27:0/2346057331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc0b00c0 0x7fbfdc168b50 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009d90 tx=0x7fbfd402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704f72721e892666 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.432+0000 7fbfd9ffb640 1 -- 172.21.15.27:0/2346057331 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4039070 con 0x7fbfdc0b00c0 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.433+0000 7fbfd9ffb640 1 -- 172.21.15.27:0/2346057331 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd402cc60 con 0x7fbfdc0b00c0 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.433+0000 7fbfd9ffb640 1 -- 172.21.15.27:0/2346057331 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4034660 con 0x7fbfdc0b00c0 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.433+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2346057331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc0b00c0 msgr2=0x7fbfdc168b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.433+0000 7fbfe14b5640 1 --2- 172.21.15.27:0/2346057331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc0b00c0 0x7fbfdc168b50 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009d90 tx=0x7fbfd402beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.433+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2346057331 shutdown_connections 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.433+0000 7fbfe14b5640 1 --2- 172.21.15.27:0/2346057331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc0b00c0 0x7fbfdc168b50 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.433+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2346057331 >> 172.21.15.27:0/2346057331 conn(0x7fbfdc0d0990 msgr2=0x7fbfdc0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.433+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2346057331 shutdown_connections 2024-03-06T07:25:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.433+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2346057331 wait complete. 2024-03-06T07:25:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.435+0000 7fbfe14b5640 1 Processor -- start 2024-03-06T07:25:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.435+0000 7fbfe14b5640 1 -- start start 2024-03-06T07:25:05.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfe14b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0b00c0 0x7fbfdc149520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfe14b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc15dca0 0x7fbfdc160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfe14b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc160990 0x7fbfdc157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfe14b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc067590 con 0x7fbfdc160990 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfe14b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc067410 con 0x7fbfdc0b00c0 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfe14b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0b4cb0 con 0x7fbfdc15dca0 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc160990 0x7fbfdc157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc160990 0x7fbfdc157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33268/0 (socket says 172.21.15.27:33268) 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdb7fe640 1 -- 172.21.15.27:0/2549057638 learned_addr learned my addr 172.21.15.27:0/2549057638 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfda7fc640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc15dca0 0x7fbfdc160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdaffd640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0b00c0 0x7fbfdc149520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdb7fe640 1 -- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc15dca0 msgr2=0x7fbfdc160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdb7fe640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc15dca0 0x7fbfdc160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdb7fe640 1 -- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0b00c0 msgr2=0x7fbfdc149520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdb7fe640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0b00c0 0x7fbfdc149520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdb7fe640 1 -- 172.21.15.27:0/2549057638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc156fc0 con 0x7fbfdc160990 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.437+0000 7fbfdb7fe640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc160990 0x7fbfdc157c20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009d90 tx=0x7fbfd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.438+0000 7fbfd37fe640 1 -- 172.21.15.27:0/2549057638 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4044020 con 0x7fbfdc160990 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.438+0000 7fbfd37fe640 1 -- 172.21.15.27:0/2549057638 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd403a040 con 0x7fbfdc160990 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.438+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc1572a0 con 0x7fbfdc160990 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.438+0000 7fbfd37fe640 1 -- 172.21.15.27:0/2549057638 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4039050 con 0x7fbfdc160990 2024-03-06T07:25:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.438+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc157860 con 0x7fbfdc160990 2024-03-06T07:25:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.439+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc068700 con 0x7fbfdc160990 2024-03-06T07:25:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.439+0000 7fbfd37fe640 1 -- 172.21.15.27:0/2549057638 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfd402cae0 con 0x7fbfdc160990 2024-03-06T07:25:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.440+0000 7fbfd37fe640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.440+0000 7fbfd37fe640 1 -- 172.21.15.27:0/2549057638 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfd40ac7b0 con 0x7fbfdc160990 2024-03-06T07:25:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.440+0000 7fbfdaffd640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.440+0000 7fbfdaffd640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc070020 tx=0x7fbfc4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.444+0000 7fbfd37fe640 1 -- 172.21.15.27:0/2549057638 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfd404e060 con 0x7fbfdc160990 2024-03-06T07:25:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.640+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbfdc0008d0 con 0x7fbfdc160990 2024-03-06T07:25:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.641+0000 7fbfd37fe640 1 -- 172.21.15.27:0/2549057638 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbfd407b070 con 0x7fbfdc160990 2024-03-06T07:25:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:25:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbfbc06b4d0 msgr2=0x7fbfbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc070020 tx=0x7fbfc4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc160990 msgr2=0x7fbfdc157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc160990 0x7fbfdc157c20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009d90 tx=0x7fbfd4004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 shutdown_connections 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfdc160990 0x7fbfdc157c20 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfdc15dca0 0x7fbfdc160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 --2- 172.21.15.27:0/2549057638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfdc0b00c0 0x7fbfdc149520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 >> 172.21.15.27:0/2549057638 conn(0x7fbfdc0d0990 msgr2=0x7fbfdc15d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 shutdown_connections 2024-03-06T07:25:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:05.644+0000 7fbfe14b5640 1 -- 172.21.15.27:0/2549057638 wait complete. 2024-03-06T07:25:05.656 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:25:05.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:05.657 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:05.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[56033]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:05.688 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:25:05.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:05.688 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:05.869 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:25:05.870 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:05.887 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:05.887 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:25:05.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:05.888 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:25:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:05.944 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:25:05.945 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:05.945 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:05.960 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:05.960 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:25:05.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:05.961 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:25:06.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:25:06.015 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:25:06.015 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:06.032 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:06.032 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:25:06.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:06.032 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:25:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:06.088 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:25:06.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:06.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.193+0000 7ff647332640 1 Processor -- start 2024-03-06T07:25:06.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.193+0000 7ff647332640 1 -- start start 2024-03-06T07:25:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.194+0000 7ff647332640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff6400b9a10 0x7ff6400d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.194+0000 7ff647332640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400cea30 con 0x7ff64015df00 2024-03-06T07:25:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.194+0000 7ff647332640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400cf3c0 con 0x7ff6400b95a0 2024-03-06T07:25:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.194+0000 7ff647332640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6400cf0e0 con 0x7ff6400b9a10 2024-03-06T07:25:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.194+0000 7ff6458cf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff6400b9a10 0x7ff6400d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.194+0000 7ff6450ce640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff64015df00 0x7ff6400d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41944/0 (socket says 172.21.15.27:41944) 2024-03-06T07:25:06.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.194+0000 7ff6450ce640 1 -- 172.21.15.27:0/3833636822 learned_addr learned my addr 172.21.15.27:0/3833636822 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:06.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.195+0000 7ff6458cf640 1 -- 172.21.15.27:0/3833636822 >> v1:172.21.15.69:6789/0 conn(0x7ff6400b95a0 legacy=0x7ff6400b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.195+0000 7ff6458cf640 1 -- 172.21.15.27:0/3833636822 >> v1:172.21.15.27:6789/0 conn(0x7ff64015df00 legacy=0x7ff6400d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.195+0000 7ff637fff640 1 -- 172.21.15.27:0/3833636822 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369137281 0 0) 0x7ff6400cea30 con 0x7ff64015df00 2024-03-06T07:25:06.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.195+0000 7ff6458cf640 1 -- 172.21.15.27:0/3833636822 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6400ce470 con 0x7ff6400b9a10 2024-03-06T07:25:06.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.196+0000 7ff6458cf640 1 --2- 172.21.15.27:0/3833636822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 0x7ff6400d3220 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff6400cf590 tx=0x7ff63802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40abaf448c52003c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.196+0000 7ff637fff640 1 -- 172.21.15.27:0/3833636822 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff638039070 con 0x7ff6400b9a10 2024-03-06T07:25:06.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.196+0000 7ff637fff640 1 -- 172.21.15.27:0/3833636822 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63802cc50 con 0x7ff6400b9a10 2024-03-06T07:25:06.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.196+0000 7ff637fff640 1 -- 172.21.15.27:0/3833636822 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff638034440 con 0x7ff6400b9a10 2024-03-06T07:25:06.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.197+0000 7ff647332640 1 -- 172.21.15.27:0/3833636822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 msgr2=0x7ff6400d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.197+0000 7ff647332640 1 --2- 172.21.15.27:0/3833636822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 0x7ff6400d3220 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff6400cf590 tx=0x7ff63802bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.197+0000 7ff647332640 1 -- 172.21.15.27:0/3833636822 shutdown_connections 2024-03-06T07:25:06.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.197+0000 7ff647332640 1 --2- 172.21.15.27:0/3833636822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 0x7ff6400d3220 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.197+0000 7ff647332640 1 -- 172.21.15.27:0/3833636822 >> 172.21.15.27:0/3833636822 conn(0x7ff640071960 msgr2=0x7ff640071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.197+0000 7ff647332640 1 -- 172.21.15.27:0/3833636822 shutdown_connections 2024-03-06T07:25:06.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.197+0000 7ff647332640 1 -- 172.21.15.27:0/3833636822 wait complete. 2024-03-06T07:25:06.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.198+0000 7ff647332640 1 Processor -- start 2024-03-06T07:25:06.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.198+0000 7ff647332640 1 -- start start 2024-03-06T07:25:06.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff647332640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6400b95a0 0x7ff6401554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff647332640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 0x7ff6400cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff647332640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff64015df00 0x7ff6401e8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff647332640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400cedf0 con 0x7ff6400b9a10 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff647332640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400cec70 con 0x7ff6400b95a0 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff647332640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400b5e10 con 0x7ff64015df00 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff6448cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 0x7ff6400cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff6448cd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 0x7ff6400cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33278/0 (socket says 172.21.15.27:33278) 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff6448cd640 1 -- 172.21.15.27:0/4244218169 learned_addr learned my addr 172.21.15.27:0/4244218169 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff6450ce640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6400b95a0 0x7ff6401554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.199+0000 7ff6458cf640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff64015df00 0x7ff6401e8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff6448cd640 1 -- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff64015df00 msgr2=0x7ff6401e8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff6448cd640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff64015df00 0x7ff6401e8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff6448cd640 1 -- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6400b95a0 msgr2=0x7ff6401554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff6448cd640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6400b95a0 0x7ff6401554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff6448cd640 1 -- 172.21.15.27:0/4244218169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6400c8af0 con 0x7ff6400b9a10 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff6450ce640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6400b95a0 0x7ff6401554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff6458cf640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff64015df00 0x7ff6401e8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff6448cd640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 0x7ff6400cad50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00c460 tx=0x7ff63c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff635ffb640 1 -- 172.21.15.27:0/4244218169 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c016020 con 0x7ff6400b9a10 2024-03-06T07:25:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff635ffb640 1 -- 172.21.15.27:0/4244218169 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c012870 con 0x7ff6400b9a10 2024-03-06T07:25:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6400c8dd0 con 0x7ff6400b9a10 2024-03-06T07:25:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff635ffb640 1 -- 172.21.15.27:0/4244218169 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c012c60 con 0x7ff6400b9a10 2024-03-06T07:25:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.200+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6400c9390 con 0x7ff6400b9a10 2024-03-06T07:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.202+0000 7ff635ffb640 1 -- 172.21.15.27:0/4244218169 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff63c019020 con 0x7ff6400b9a10 2024-03-06T07:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.202+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff640067ff0 con 0x7ff6400b9a10 2024-03-06T07:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.202+0000 7ff635ffb640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff62406b4d0 0x7ff62406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.202+0000 7ff635ffb640 1 -- 172.21.15.27:0/4244218169 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff63c089e90 con 0x7ff6400b9a10 2024-03-06T07:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.202+0000 7ff6450ce640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff62406b4d0 0x7ff62406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.203+0000 7ff6450ce640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff62406b4d0 0x7ff62406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff630000f60 tx=0x7ff630007cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.206+0000 7ff635ffb640 1 -- 172.21.15.27:0/4244218169 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff63c020030 con 0x7ff6400b9a10 2024-03-06T07:25:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.487+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6400d1b90 con 0x7ff6400b9a10 2024-03-06T07:25:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.488+0000 7ff635ffb640 1 -- 172.21.15.27:0/4244218169 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7ff63c058da0 con 0x7ff6400b9a10 2024-03-06T07:25:06.489 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:06.490 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5126,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2827284911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2827284911},{"type":"v1","addr":"172.21.15.69:6836","nonce":2827284911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5419,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1663054032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1663054032},{"type":"v1","addr":"172.21.15.69:6837","nonce":1663054032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2744957333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2744957333},{"type":"v1","addr":"172.21.15.27:6834","nonce":2744957333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:45.201958+0000","modified":"2024-03-06T07:24:48.290611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5754":{"gid":5754,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2008146374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2008146374},{"type":"v1","addr":"172.21.15.27:6835","nonce":2008146374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5754,"qdb_cluster":[5754]},"id":3}]} 2024-03-06T07:25:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-03-06T07:25:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff62406b4d0 msgr2=0x7ff62406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff62406b4d0 0x7ff62406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff630000f60 tx=0x7ff630007cc0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 msgr2=0x7ff6400cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 0x7ff6400cad50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00c460 tx=0x7ff63c00c920 comp rx=0 tx=0).stop 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 shutdown_connections 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff62406b4d0 0x7ff62406d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff64015df00 0x7ff6401e8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6400b9a10 0x7ff6400cad50 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 --2- 172.21.15.27:0/4244218169 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6400b95a0 0x7ff6401554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 >> 172.21.15.27:0/4244218169 conn(0x7ff640071960 msgr2=0x7ff640071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 shutdown_connections 2024-03-06T07:25:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.490+0000 7ff647332640 1 -- 172.21.15.27:0/4244218169 wait complete. 2024-03-06T07:25:06.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.607+0000 7fa5f1878640 1 Processor -- start 2024-03-06T07:25:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.607+0000 7fa5f1878640 1 -- start start 2024-03-06T07:25:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5f1878640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5ec0d3400 0x7fa5ec0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5f1878640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec0b2b50 0x7fa5ec0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5f1878640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec0ce610 con 0x7fa5ec071540 2024-03-06T07:25:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5f1878640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec0d2f70 con 0x7fa5ec0d3400 2024-03-06T07:25:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5f1878640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0d4be0 con 0x7fa5ec0b2b50 2024-03-06T07:25:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5ea7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5ec0d3400 0x7fa5ec0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5eb7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec0b2b50 0x7fa5ec0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5eaffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa5ec071540 0x7fa5ec071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:52588/0 (socket says 172.21.15.27:52588) 2024-03-06T07:25:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5ea7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5ec0d3400 0x7fa5ec0b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:34106/0 (socket says 172.21.15.27:34106) 2024-03-06T07:25:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.608+0000 7fa5eaffd640 1 -- 172.21.15.27:0/1314733633 learned_addr learned my addr 172.21.15.27:0/1314733633 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.609+0000 7fa5ea7fc640 1 -- 172.21.15.27:0/1314733633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec0b2b50 msgr2=0x7fa5ec0b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.609+0000 7fa5ea7fc640 1 --2- 172.21.15.27:0/1314733633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec0b2b50 0x7fa5ec0b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.609+0000 7fa5ea7fc640 1 -- 172.21.15.27:0/1314733633 >> v1:172.21.15.69:6789/0 conn(0x7fa5ec071540 legacy=0x7fa5ec071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.609+0000 7fa5ea7fc640 1 -- 172.21.15.27:0/1314733633 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec0d2920 con 0x7fa5ec0d3400 2024-03-06T07:25:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.609+0000 7fa5e9ffb640 1 -- 172.21.15.27:0/1314733633 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889027833 0 0) 0x7fa5ec0ce610 con 0x7fa5ec071540 2024-03-06T07:25:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.609+0000 7fa5eb7fe640 1 --2- 172.21.15.27:0/1314733633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec0b2b50 0x7fa5ec0b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.610+0000 7fa5ea7fc640 1 --2- 172.21.15.27:0/1314733633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5ec0d3400 0x7fa5ec0b30a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0097a0 tx=0x7fa5dc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0fea6edd90b8213 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.610+0000 7fa5e9ffb640 1 -- 172.21.15.27:0/1314733633 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc035020 con 0x7fa5ec0d3400 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.610+0000 7fa5e9ffb640 1 -- 172.21.15.27:0/1314733633 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc031810 con 0x7fa5ec0d3400 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.610+0000 7fa5e9ffb640 1 -- 172.21.15.27:0/1314733633 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc031be0 con 0x7fa5ec0d3400 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.610+0000 7fa5f1878640 1 -- 172.21.15.27:0/1314733633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5ec0d3400 msgr2=0x7fa5ec0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.611+0000 7fa5f1878640 1 --2- 172.21.15.27:0/1314733633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5ec0d3400 0x7fa5ec0b30a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0097a0 tx=0x7fa5dc02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.611+0000 7fa5f1878640 1 -- 172.21.15.27:0/1314733633 shutdown_connections 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.611+0000 7fa5f1878640 1 --2- 172.21.15.27:0/1314733633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5ec0d3400 0x7fa5ec0b30a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.611+0000 7fa5f1878640 1 --2- 172.21.15.27:0/1314733633 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec0b2b50 0x7fa5ec0b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.611+0000 7fa5f1878640 1 -- 172.21.15.27:0/1314733633 >> 172.21.15.27:0/1314733633 conn(0x7fa5ec0d3ca0 msgr2=0x7fa5ec0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.611+0000 7fa5f1878640 1 -- 172.21.15.27:0/1314733633 shutdown_connections 2024-03-06T07:25:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.611+0000 7fa5f1878640 1 -- 172.21.15.27:0/1314733633 wait complete. 2024-03-06T07:25:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.613+0000 7fa5f1878640 1 Processor -- start 2024-03-06T07:25:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.613+0000 7fa5f1878640 1 -- start start 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5f1878640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec071540 0x7fa5ec0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5f1878640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5ec0b2b50 0x7fa5ec0b8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5f1878640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec0d3400 0x7fa5ec0b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5f1878640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec15e370 con 0x7fa5ec0d3400 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5f1878640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0ce8f0 con 0x7fa5ec0b2b50 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5f1878640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec14d980 con 0x7fa5ec071540 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec0d3400 0x7fa5ec0b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec0d3400 0x7fa5ec0b87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33282/0 (socket says 172.21.15.27:33282) 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5eb7fe640 1 -- 172.21.15.27:0/2210902583 learned_addr learned my addr 172.21.15.27:0/2210902583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5eaffd640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec071540 0x7fa5ec0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.614+0000 7fa5ea7fc640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5ec0b2b50 0x7fa5ec0b8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5eb7fe640 1 -- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec071540 msgr2=0x7fa5ec0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5eb7fe640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec071540 0x7fa5ec0cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5eb7fe640 1 -- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5ec0b2b50 msgr2=0x7fa5ec0b8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5eb7fe640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5ec0b2b50 0x7fa5ec0b8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5eb7fe640 1 -- 172.21.15.27:0/2210902583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec1f46b0 con 0x7fa5ec0d3400 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5eb7fe640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec0d3400 0x7fa5ec0b87c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400efc0 tx=0x7fa5e4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5d3fff640 1 -- 172.21.15.27:0/2210902583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4018030 con 0x7fa5ec0d3400 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5d3fff640 1 -- 172.21.15.27:0/2210902583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e400cde0 con 0x7fa5ec0d3400 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5d3fff640 1 -- 172.21.15.27:0/2210902583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4003bb0 con 0x7fa5ec0d3400 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec1f38f0 con 0x7fa5ec0d3400 2024-03-06T07:25:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.615+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec1f3eb0 con 0x7fa5ec0d3400 2024-03-06T07:25:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.616+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ec068700 con 0x7fa5ec0d3400 2024-03-06T07:25:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.617+0000 7fa5d3fff640 1 -- 172.21.15.27:0/2210902583 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5e401b070 con 0x7fa5ec0d3400 2024-03-06T07:25:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.617+0000 7fa5d3fff640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa5d806b4d0 0x7fa5d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.617+0000 7fa5d3fff640 1 -- 172.21.15.27:0/2210902583 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5e408aab0 con 0x7fa5ec0d3400 2024-03-06T07:25:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.617+0000 7fa5eaffd640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa5d806b4d0 0x7fa5d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.618+0000 7fa5eaffd640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa5d806b4d0 0x7fa5d806d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0000f60 tx=0x7fa5e0007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.621+0000 7fa5d3fff640 1 -- 172.21.15.27:0/2210902583 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5e4010040 con 0x7fa5ec0d3400 2024-03-06T07:25:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.804+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5ec15e890 con 0x7fa5ec0d3400 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.805+0000 7fa5d3fff640 1 -- 172.21.15.27:0/2210902583 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fa5e4059370 con 0x7fa5ec0d3400 2024-03-06T07:25:06.806 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:06.807 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:24:46.825646+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:43.493554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:44.172410+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa5d806b4d0 msgr2=0x7fa5d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa5d806b4d0 0x7fa5d806d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0000f60 tx=0x7fa5e0007d50 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec0d3400 msgr2=0x7fa5ec0b87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec0d3400 0x7fa5ec0b87c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400efc0 tx=0x7fa5e4009f90 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 shutdown_connections 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa5d806b4d0 0x7fa5d806d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec0d3400 0x7fa5ec0b87c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5ec0b2b50 0x7fa5ec0b8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 --2- 172.21.15.27:0/2210902583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5ec071540 0x7fa5ec0cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 >> 172.21.15.27:0/2210902583 conn(0x7fa5ec0d3ca0 msgr2=0x7fa5ec0cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 shutdown_connections 2024-03-06T07:25:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.807+0000 7fa5f1878640 1 -- 172.21.15.27:0/2210902583 wait complete. 2024-03-06T07:25:06.820 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:43.493554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:44.172410+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:06.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:06.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.928+0000 7efd1393a640 1 Processor -- start 2024-03-06T07:25:06.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.928+0000 7efd1393a640 1 -- start start 2024-03-06T07:25:06.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.929+0000 7efd1393a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c1624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.929+0000 7efd1393a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efd0c0cf9f0 0x7efd0c0d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.929+0000 7efd1393a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c06d4d0 con 0x7efd0c0d05f0 2024-03-06T07:25:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.929+0000 7efd1393a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd0c06dba0 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.929+0000 7efd1393a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c0d0a60 con 0x7efd0c072a30 2024-03-06T07:25:06.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.930+0000 7efd116d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c1624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.930+0000 7efd10ed5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efd0c0cf9f0 0x7efd0c0d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.930+0000 7efd116d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c1624c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:60572/0 (socket says 172.21.15.27:60572) 2024-03-06T07:25:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.930+0000 7efd116d6640 1 -- 172.21.15.27:0/1257744204 learned_addr learned my addr 172.21.15.27:0/1257744204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.930+0000 7efd10ed5640 1 -- 172.21.15.27:0/1257744204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 msgr2=0x7efd0c1624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.930+0000 7efd10ed5640 1 --2- 172.21.15.27:0/1257744204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c1624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.930+0000 7efd10ed5640 1 -- 172.21.15.27:0/1257744204 >> v1:172.21.15.69:6789/0 conn(0x7efd0c0d05f0 legacy=0x7efd0c15d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.930+0000 7efd10ed5640 1 -- 172.21.15.27:0/1257744204 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c0b4950 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.930+0000 7efd116d6640 1 --2- 172.21.15.27:0/1257744204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c1624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:06.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.931+0000 7efd10ed5640 1 --2- 172.21.15.27:0/1257744204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0c0cf9f0 0x7efd0c0d00b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efd080097a0 tx=0x7efd0802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4341f93c33ca2d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.931+0000 7efd03fff640 1 -- 172.21.15.27:0/1257744204 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08039070 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.931+0000 7efd03fff640 1 -- 172.21.15.27:0/1257744204 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd080316f0 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.931+0000 7efd03fff640 1 -- 172.21.15.27:0/1257744204 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08031ac0 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.932+0000 7efd1393a640 1 -- 172.21.15.27:0/1257744204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0c0cf9f0 msgr2=0x7efd0c0d00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.932+0000 7efd1393a640 1 --2- 172.21.15.27:0/1257744204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0c0cf9f0 0x7efd0c0d00b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efd080097a0 tx=0x7efd0802b790 comp rx=0 tx=0).stop 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.932+0000 7efd1393a640 1 -- 172.21.15.27:0/1257744204 shutdown_connections 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.932+0000 7efd1393a640 1 --2- 172.21.15.27:0/1257744204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0c0cf9f0 0x7efd0c0d00b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.932+0000 7efd1393a640 1 --2- 172.21.15.27:0/1257744204 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c1624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.932+0000 7efd1393a640 1 -- 172.21.15.27:0/1257744204 >> 172.21.15.27:0/1257744204 conn(0x7efd0c06af20 msgr2=0x7efd0c06b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.932+0000 7efd1393a640 1 -- 172.21.15.27:0/1257744204 shutdown_connections 2024-03-06T07:25:06.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.932+0000 7efd1393a640 1 -- 172.21.15.27:0/1257744204 wait complete. 2024-03-06T07:25:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.934+0000 7efd1393a640 1 Processor -- start 2024-03-06T07:25:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.934+0000 7efd1393a640 1 -- start start 2024-03-06T07:25:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.935+0000 7efd1393a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.935+0000 7efd1393a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd0c0cf9f0 0x7efd0c0717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.935+0000 7efd1393a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0c0d05f0 0x7efd0c15f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.935+0000 7efd1393a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c071130 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.935+0000 7efd1393a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c06ddf0 con 0x7efd0c0d05f0 2024-03-06T07:25:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.935+0000 7efd1393a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c072e10 con 0x7efd0c072a30 2024-03-06T07:25:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.935+0000 7efd10ed5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd0c0cf9f0 0x7efd0c0717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.935+0000 7efd10ed5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd0c0cf9f0 0x7efd0c0717c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33294/0 (socket says 172.21.15.27:33294) 2024-03-06T07:25:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.935+0000 7efd10ed5640 1 -- 172.21.15.27:0/2557002469 learned_addr learned my addr 172.21.15.27:0/2557002469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd116d6640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd11ed7640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0c0d05f0 0x7efd0c15f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd10ed5640 1 -- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 msgr2=0x7efd0c152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd10ed5640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd10ed5640 1 -- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0c0d05f0 msgr2=0x7efd0c15f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd10ed5640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0c0d05f0 0x7efd0c15f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd10ed5640 1 -- 172.21.15.27:0/2557002469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c15e120 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd10ed5640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd0c0cf9f0 0x7efd0c0717c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7efd08037040 tx=0x7efd080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd01ffb640 1 -- 172.21.15.27:0/2557002469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08043070 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd01ffb640 1 -- 172.21.15.27:0/2557002469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd080346a0 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c0c9500 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd01ffb640 1 -- 172.21.15.27:0/2557002469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08039040 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.936+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c0c9ac0 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.937+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0c09e320 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.938+0000 7efd01ffb640 1 -- 172.21.15.27:0/2557002469 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd08046070 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.938+0000 7efd01ffb640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcf006b4d0 0x7efcf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.939+0000 7efd01ffb640 1 -- 172.21.15.27:0/2557002469 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd080abcd0 con 0x7efd0c0cf9f0 2024-03-06T07:25:06.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.939+0000 7efd116d6640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcf006b4d0 0x7efcf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.939+0000 7efd116d6640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcf006b4d0 0x7efcf006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efcfc006ca0 tx=0x7efcfc009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:06.943+0000 7efd01ffb640 1 -- 172.21.15.27:0/2557002469 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd0807a590 con 0x7efd0c0cf9f0 2024-03-06T07:25:07.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.122+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd0c04f410 con 0x7efd0c0cf9f0 2024-03-06T07:25:07.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.122+0000 7efd01ffb640 1 -- 172.21.15.27:0/2557002469 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd0807e630 con 0x7efd0c0cf9f0 2024-03-06T07:25:07.124 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:25:07.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcf006b4d0 msgr2=0x7efcf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcf006b4d0 0x7efcf006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efcfc006ca0 tx=0x7efcfc009240 comp rx=0 tx=0).stop 2024-03-06T07:25:07.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd0c0cf9f0 msgr2=0x7efd0c0717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd0c0cf9f0 0x7efd0c0717c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7efd08037040 tx=0x7efd080046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 shutdown_connections 2024-03-06T07:25:07.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcf006b4d0 0x7efcf006d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd0c0d05f0 0x7efd0c15f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd0c0cf9f0 0x7efd0c0717c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 --2- 172.21.15.27:0/2557002469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efd0c072a30 0x7efd0c152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.124+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 >> 172.21.15.27:0/2557002469 conn(0x7efd0c06af20 msgr2=0x7efd0c161980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.125+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 shutdown_connections 2024-03-06T07:25:07.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.125+0000 7efd1393a640 1 -- 172.21.15.27:0/2557002469 wait complete. 2024-03-06T07:25:07.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:07.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.245+0000 7ff3be064640 1 Processor -- start 2024-03-06T07:25:07.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.245+0000 7ff3be064640 1 -- start start 2024-03-06T07:25:07.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3be064640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b8071540 0x7ff3b8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3be064640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3b80b2d30 0x7ff3b80ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3be064640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff3b80cecb0 0x7ff3b80cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3be064640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b80d3230 con 0x7ff3b80b2d30 2024-03-06T07:25:07.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3be064640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b8071e60 con 0x7ff3b80cecb0 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3be064640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b80d4960 con 0x7ff3b8071540 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3aeffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3b80b2d30 0x7ff3b80ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3aeffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3b80b2d30 0x7ff3b80ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33310/0 (socket says 172.21.15.27:33310) 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3aeffd640 1 -- 172.21.15.27:0/1299452413 learned_addr learned my addr 172.21.15.27:0/1299452413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3b7fff640 1 --2- 172.21.15.27:0/1299452413 >> v2:172.21.15.69:3300/0 conn(0x7ff3b80cecb0 0x7ff3b80cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3b77fe640 1 --2- 172.21.15.27:0/1299452413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b8071540 0x7ff3b8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3aeffd640 1 -- 172.21.15.27:0/1299452413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b8071540 msgr2=0x7ff3b8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3aeffd640 1 --2- 172.21.15.27:0/1299452413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b8071540 0x7ff3b8071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3aeffd640 1 -- 172.21.15.27:0/1299452413 >> v2:172.21.15.69:3300/0 conn(0x7ff3b80cecb0 msgr2=0x7ff3b80cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3aeffd640 1 --2- 172.21.15.27:0/1299452413 >> v2:172.21.15.69:3300/0 conn(0x7ff3b80cecb0 0x7ff3b80cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.246+0000 7ff3aeffd640 1 -- 172.21.15.27:0/1299452413 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b80d36c0 con 0x7ff3b80b2d30 2024-03-06T07:25:07.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.247+0000 7ff3aeffd640 1 --2- 172.21.15.27:0/1299452413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b80b2d30 0x7ff3b80ce770 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8068380 tx=0x7ff3a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abed08e3fe24650 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.247+0000 7ff3b6ffd640 1 -- 172.21.15.27:0/1299452413 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0035020 con 0x7ff3b80b2d30 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.247+0000 7ff3b6ffd640 1 -- 172.21.15.27:0/1299452413 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a0031870 con 0x7ff3b80b2d30 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.247+0000 7ff3b6ffd640 1 -- 172.21.15.27:0/1299452413 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0031c40 con 0x7ff3b80b2d30 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.248+0000 7ff3be064640 1 -- 172.21.15.27:0/1299452413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b80b2d30 msgr2=0x7ff3b80ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.248+0000 7ff3be064640 1 --2- 172.21.15.27:0/1299452413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b80b2d30 0x7ff3b80ce770 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8068380 tx=0x7ff3a002b910 comp rx=0 tx=0).stop 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.248+0000 7ff3be064640 1 -- 172.21.15.27:0/1299452413 shutdown_connections 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.248+0000 7ff3be064640 1 --2- 172.21.15.27:0/1299452413 >> v2:172.21.15.69:3300/0 conn(0x7ff3b80cecb0 0x7ff3b80cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.248+0000 7ff3be064640 1 --2- 172.21.15.27:0/1299452413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b80b2d30 0x7ff3b80ce770 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.248+0000 7ff3be064640 1 --2- 172.21.15.27:0/1299452413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b8071540 0x7ff3b8071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.248+0000 7ff3be064640 1 -- 172.21.15.27:0/1299452413 >> 172.21.15.27:0/1299452413 conn(0x7ff3b80d3d60 msgr2=0x7ff3b80d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.248+0000 7ff3be064640 1 -- 172.21.15.27:0/1299452413 shutdown_connections 2024-03-06T07:25:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.248+0000 7ff3be064640 1 -- 172.21.15.27:0/1299452413 wait complete. 2024-03-06T07:25:07.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.249+0000 7ff3be064640 1 Processor -- start 2024-03-06T07:25:07.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.249+0000 7ff3be064640 1 -- start start 2024-03-06T07:25:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3be064640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b8071540 0x7ff3b81f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3be064640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3b80b2d30 0x7ff3b80caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3be064640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b80cecb0 0x7ff3b80d1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3be064640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b80d2be0 con 0x7ff3b8071540 2024-03-06T07:25:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3be064640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b80d2900 con 0x7ff3b80b2d30 2024-03-06T07:25:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3be064640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b80d2ec0 con 0x7ff3b80cecb0 2024-03-06T07:25:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b8071540 0x7ff3b81f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b8071540 0x7ff3b81f38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33322/0 (socket says 172.21.15.27:33322) 2024-03-06T07:25:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3b77fe640 1 -- 172.21.15.27:0/1059174391 learned_addr learned my addr 172.21.15.27:0/1059174391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3b7fff640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b80cecb0 0x7ff3b80d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.250+0000 7ff3aeffd640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3b80b2d30 0x7ff3b80caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b77fe640 1 -- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b80cecb0 msgr2=0x7ff3b80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b77fe640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b80cecb0 0x7ff3b80d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b77fe640 1 -- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3b80b2d30 msgr2=0x7ff3b80caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b77fe640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3b80b2d30 0x7ff3b80caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b77fe640 1 -- 172.21.15.27:0/1059174391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b80cb910 con 0x7ff3b8071540 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b7fff640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b80cecb0 0x7ff3b80d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3aeffd640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3b80b2d30 0x7ff3b80caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b77fe640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b8071540 0x7ff3b81f38e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800e870 tx=0x7ff3a800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b4ff9640 1 -- 172.21.15.27:0/1059174391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8017070 con 0x7ff3b8071540 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b4ff9640 1 -- 172.21.15.27:0/1059174391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a800ee60 con 0x7ff3b8071540 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b80cb420 con 0x7ff3b8071540 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3b4ff9640 1 -- 172.21.15.27:0/1059174391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a800cb80 con 0x7ff3b8071540 2024-03-06T07:25:07.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.251+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b81569a0 con 0x7ff3b8071540 2024-03-06T07:25:07.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.252+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b8072070 con 0x7ff3b8071540 2024-03-06T07:25:07.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.253+0000 7ff3b4ff9640 1 -- 172.21.15.27:0/1059174391 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3a801a070 con 0x7ff3b8071540 2024-03-06T07:25:07.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.254+0000 7ff3b4ff9640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff39806b4d0 0x7ff39806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.254+0000 7ff3b4ff9640 1 -- 172.21.15.27:0/1059174391 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3a808a0d0 con 0x7ff3b8071540 2024-03-06T07:25:07.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.254+0000 7ff3aeffd640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff39806b4d0 0x7ff39806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.255+0000 7ff3aeffd640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff39806b4d0 0x7ff39806d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0036000 tx=0x7ff3a003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.263+0000 7ff3b4ff9640 1 -- 172.21.15.27:0/1059174391 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3a8058f50 con 0x7ff3b8071540 2024-03-06T07:25:07.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.443+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff3b804f410 con 0x7ff3b8071540 2024-03-06T07:25:07.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.444+0000 7ff3b4ff9640 1 -- 172.21.15.27:0/1059174391 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff3a8058850 con 0x7ff3b8071540 2024-03-06T07:25:07.445 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:25:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.445+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff39806b4d0 msgr2=0x7ff39806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.445+0000 7ff3be064640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff39806b4d0 0x7ff39806d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0036000 tx=0x7ff3a003c040 comp rx=0 tx=0).stop 2024-03-06T07:25:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.445+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b8071540 msgr2=0x7ff3b81f38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.445+0000 7ff3be064640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b8071540 0x7ff3b81f38e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800e870 tx=0x7ff3a800ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.446+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 shutdown_connections 2024-03-06T07:25:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.446+0000 7ff3be064640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff39806b4d0 0x7ff39806d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.446+0000 7ff3be064640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3b80cecb0 0x7ff3b80d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.446+0000 7ff3be064640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3b80b2d30 0x7ff3b80caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.446+0000 7ff3be064640 1 --2- 172.21.15.27:0/1059174391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3b8071540 0x7ff3b81f38e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.446+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 >> 172.21.15.27:0/1059174391 conn(0x7ff3b80d3d60 msgr2=0x7ff3b815b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.446+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 shutdown_connections 2024-03-06T07:25:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.446+0000 7ff3be064640 1 -- 172.21.15.27:0/1059174391 wait complete. 2024-03-06T07:25:07.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:07.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.567+0000 7f46c773a640 1 Processor -- start 2024-03-06T07:25:07.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.567+0000 7f46c773a640 1 -- start start 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.567+0000 7f46c773a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f46c015dd90 0x7f46c00ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46c773a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c00d3740 con 0x7f46c00b6d10 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46c773a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c00d3120 con 0x7f46c00d29d0 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46c773a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c00d3400 con 0x7f46c015dd90 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46c5cd7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f46c00b6d10 0x7f46c00d0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41946/0 (socket says 172.21.15.27:41946) 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46c5cd7640 1 -- 172.21.15.27:0/561703658 learned_addr learned my addr 172.21.15.27:0/561703658 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46c54d6640 1 --2- 172.21.15.27:0/561703658 >> v2:172.21.15.69:3300/0 conn(0x7f46c015dd90 0x7f46c00ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46b7fff640 1 -- 172.21.15.27:0/561703658 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305338644 0 0) 0x7f46c00d3740 con 0x7f46c00b6d10 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46b7fff640 1 -- 172.21.15.27:0/561703658 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ac003520 con 0x7f46c00b6d10 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46c54d6640 1 -- 172.21.15.27:0/561703658 >> v1:172.21.15.69:6789/0 conn(0x7f46c00d29d0 legacy=0x7f46c00d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46c54d6640 1 -- 172.21.15.27:0/561703658 >> v1:172.21.15.27:6789/0 conn(0x7f46c00b6d10 legacy=0x7f46c00d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.568+0000 7f46c54d6640 1 -- 172.21.15.27:0/561703658 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c00d39e0 con 0x7f46c015dd90 2024-03-06T07:25:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.569+0000 7f46c54d6640 1 --2- 172.21.15.27:0/561703658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c015dd90 0x7f46c00ceea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0097a0 tx=0x7f46bc02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d0a30975bd15acc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.569+0000 7f46b7fff640 1 -- 172.21.15.27:0/561703658 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc02f810 con 0x7f46c015dd90 2024-03-06T07:25:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.569+0000 7f46b7fff640 1 -- 172.21.15.27:0/561703658 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc02f9b0 con 0x7f46c015dd90 2024-03-06T07:25:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.569+0000 7f46b7fff640 1 -- 172.21.15.27:0/561703658 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc02fd80 con 0x7f46c015dd90 2024-03-06T07:25:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.570+0000 7f46c773a640 1 -- 172.21.15.27:0/561703658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c015dd90 msgr2=0x7f46c00ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.570+0000 7f46c773a640 1 --2- 172.21.15.27:0/561703658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c015dd90 0x7f46c00ceea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0097a0 tx=0x7f46bc02b9f0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.570+0000 7f46c773a640 1 -- 172.21.15.27:0/561703658 shutdown_connections 2024-03-06T07:25:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.570+0000 7f46c773a640 1 --2- 172.21.15.27:0/561703658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c015dd90 0x7f46c00ceea0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.570+0000 7f46c773a640 1 -- 172.21.15.27:0/561703658 >> 172.21.15.27:0/561703658 conn(0x7f46c0071960 msgr2=0x7f46c0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.570+0000 7f46c773a640 1 -- 172.21.15.27:0/561703658 shutdown_connections 2024-03-06T07:25:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.570+0000 7f46c773a640 1 -- 172.21.15.27:0/561703658 wait complete. 2024-03-06T07:25:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.571+0000 7f46c773a640 1 Processor -- start 2024-03-06T07:25:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.571+0000 7f46c773a640 1 -- start start 2024-03-06T07:25:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c773a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b6d10 0x7f46c0147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c773a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c00d29d0 0x7f46c0154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c773a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46c015dd90 0x7f46c00b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c773a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c00d2df0 con 0x7f46c015dd90 2024-03-06T07:25:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c773a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c015e170 con 0x7f46c00b6d10 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c773a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c00d48f0 con 0x7f46c00d29d0 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c5cd7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46c015dd90 0x7f46c00b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c5cd7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46c015dd90 0x7f46c00b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33334/0 (socket says 172.21.15.27:33334) 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c5cd7640 1 -- 172.21.15.27:0/2839112484 learned_addr learned my addr 172.21.15.27:0/2839112484 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c54d6640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b6d10 0x7f46c0147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c4cd5640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c00d29d0 0x7f46c0154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c5cd7640 1 -- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c00d29d0 msgr2=0x7f46c0154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c5cd7640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c00d29d0 0x7f46c0154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c5cd7640 1 -- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b6d10 msgr2=0x7f46c0147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c5cd7640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b6d10 0x7f46c0147ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c5cd7640 1 -- 172.21.15.27:0/2839112484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c00c8730 con 0x7f46c015dd90 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c54d6640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b6d10 0x7f46c0147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.572+0000 7f46c5cd7640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46c015dd90 0x7f46c00b3700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f46b800cb40 tx=0x7f46b800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.573+0000 7f46b5ffb640 1 -- 172.21.15.27:0/2839112484 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8017070 con 0x7f46c015dd90 2024-03-06T07:25:07.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.573+0000 7f46b5ffb640 1 -- 172.21.15.27:0/2839112484 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b800aca0 con 0x7f46c015dd90 2024-03-06T07:25:07.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.573+0000 7f46b5ffb640 1 -- 172.21.15.27:0/2839112484 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8012620 con 0x7f46c015dd90 2024-03-06T07:25:07.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.573+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c00c8a70 con 0x7f46c015dd90 2024-03-06T07:25:07.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.573+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c00c92b0 con 0x7f46c015dd90 2024-03-06T07:25:07.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.574+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c0072a30 con 0x7f46c015dd90 2024-03-06T07:25:07.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.574+0000 7f46b5ffb640 1 -- 172.21.15.27:0/2839112484 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46b801a070 con 0x7f46c015dd90 2024-03-06T07:25:07.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.575+0000 7f46b5ffb640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f46a406b4d0 0x7f46a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.575+0000 7f46b5ffb640 1 -- 172.21.15.27:0/2839112484 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46b808a090 con 0x7f46c015dd90 2024-03-06T07:25:07.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.575+0000 7f46c54d6640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f46a406b4d0 0x7f46a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.575+0000 7f46c54d6640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f46a406b4d0 0x7f46a406d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f46bc036000 tx=0x7f46bc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.584+0000 7f46b5ffb640 1 -- 172.21.15.27:0/2839112484 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46c00c8a70 con 0x7f46c015dd90 2024-03-06T07:25:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.814+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46c00c9480 con 0x7f46c015dd90 2024-03-06T07:25:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.814+0000 7f46b5ffb640 1 -- 172.21.15.27:0/2839112484 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7f46b8058e90 con 0x7f46c015dd90 2024-03-06T07:25:07.816 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f46a406b4d0 msgr2=0x7f46a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f46a406b4d0 0x7f46a406d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f46bc036000 tx=0x7f46bc008030 comp rx=0 tx=0).stop 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46c015dd90 msgr2=0x7f46c00b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46c015dd90 0x7f46c00b3700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f46b800cb40 tx=0x7f46b800a060 comp rx=0 tx=0).stop 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 shutdown_connections 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f46a406b4d0 0x7f46a406d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46c015dd90 0x7f46c00b3700 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46c00d29d0 0x7f46c0154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 --2- 172.21.15.27:0/2839112484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46c00b6d10 0x7f46c0147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 >> 172.21.15.27:0/2839112484 conn(0x7f46c0071960 msgr2=0x7f46c015c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 shutdown_connections 2024-03-06T07:25:07.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.816+0000 7f46c773a640 1 -- 172.21.15.27:0/2839112484 wait complete. 2024-03-06T07:25:07.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:07.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.938+0000 7f7664977640 1 Processor -- start 2024-03-06T07:25:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.938+0000 7f7664977640 1 -- start start 2024-03-06T07:25:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.939+0000 7f7664977640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f766015df00 0x7f76600d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.939+0000 7f7664977640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600b9a30 0x7f76600d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.939+0000 7f7664977640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76600cea30 con 0x7f76600b95a0 2024-03-06T07:25:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.939+0000 7f7664977640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76600d2af0 con 0x7f766015df00 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.939+0000 7f7664977640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76600cf3c0 con 0x7f76600b9a30 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.939+0000 7f765dd9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f76600b95a0 0x7f76600b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41950/0 (socket says 172.21.15.27:41950) 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.939+0000 7f765dd9b640 1 -- 172.21.15.27:0/644307611 learned_addr learned my addr 172.21.15.27:0/644307611 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.939+0000 7f765e59c640 1 --2- 172.21.15.27:0/644307611 >> v2:172.21.15.69:3300/0 conn(0x7f766015df00 0x7f76600d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.939+0000 7f765ed9d640 1 --2- 172.21.15.27:0/644307611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600b9a30 0x7f76600d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.940+0000 7f765d59a640 1 -- 172.21.15.27:0/644307611 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086420708 0 0) 0x7f76600cea30 con 0x7f76600b95a0 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.940+0000 7f765d59a640 1 -- 172.21.15.27:0/644307611 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7648003520 con 0x7f76600b95a0 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.940+0000 7f765e59c640 1 -- 172.21.15.27:0/644307611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600b9a30 msgr2=0x7f76600d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.940+0000 7f765d59a640 1 -- 172.21.15.27:0/644307611 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 562924496 0 0) 0x7f7648003520 con 0x7f76600b95a0 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.940+0000 7f765e59c640 1 --2- 172.21.15.27:0/644307611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600b9a30 0x7f76600d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.940+0000 7f765e59c640 1 -- 172.21.15.27:0/644307611 >> v1:172.21.15.27:6789/0 conn(0x7f76600b95a0 legacy=0x7f76600b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.940+0000 7f765e59c640 1 -- 172.21.15.27:0/644307611 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76600cf0e0 con 0x7f766015df00 2024-03-06T07:25:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.940+0000 7f765ed9d640 1 --2- 172.21.15.27:0/644307611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600b9a30 0x7f76600d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:07.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.941+0000 7f765e59c640 1 --2- 172.21.15.27:0/644307611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f766015df00 0x7f76600d25b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f76500097a0 tx=0x7f765002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e050815a098ad57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.941+0000 7f765d59a640 1 -- 172.21.15.27:0/644307611 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650037070 con 0x7f766015df00 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.941+0000 7f765d59a640 1 -- 172.21.15.27:0/644307611 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7650030510 con 0x7f766015df00 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.941+0000 7f765d59a640 1 -- 172.21.15.27:0/644307611 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654001670 con 0x7f766015df00 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.942+0000 7f7664977640 1 -- 172.21.15.27:0/644307611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f766015df00 msgr2=0x7f76600d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.942+0000 7f7664977640 1 --2- 172.21.15.27:0/644307611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f766015df00 0x7f76600d25b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f76500097a0 tx=0x7f765002b790 comp rx=0 tx=0).stop 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.942+0000 7f7664977640 1 -- 172.21.15.27:0/644307611 shutdown_connections 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.942+0000 7f7664977640 1 --2- 172.21.15.27:0/644307611 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600b9a30 0x7f76600d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.942+0000 7f7664977640 1 --2- 172.21.15.27:0/644307611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f766015df00 0x7f76600d25b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.942+0000 7f7664977640 1 -- 172.21.15.27:0/644307611 >> 172.21.15.27:0/644307611 conn(0x7f7660071960 msgr2=0x7f7660071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.942+0000 7f7664977640 1 -- 172.21.15.27:0/644307611 shutdown_connections 2024-03-06T07:25:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.942+0000 7f7664977640 1 -- 172.21.15.27:0/644307611 wait complete. 2024-03-06T07:25:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.943+0000 7f7664977640 1 Processor -- start 2024-03-06T07:25:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f7664977640 1 -- start start 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f7664977640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76600b95a0 0x7f7660161b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f7664977640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766015df00 0x7f766015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f7664977640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600c8bc0 0x7f7660150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f7664977640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76600cec70 con 0x7f766015df00 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f7664977640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76600ce750 con 0x7f76600b95a0 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f7664977640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76600cedf0 con 0x7f76600c8bc0 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f765dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766015df00 0x7f766015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f765dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766015df00 0x7f766015ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33338/0 (socket says 172.21.15.27:33338) 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f765dd9b640 1 -- 172.21.15.27:0/481767620 learned_addr learned my addr 172.21.15.27:0/481767620 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f765ed9d640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600c8bc0 0x7f7660150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.944+0000 7f765e59c640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76600b95a0 0x7f7660161b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f765dd9b640 1 -- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600c8bc0 msgr2=0x7f7660150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f765dd9b640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600c8bc0 0x7f7660150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f765dd9b640 1 -- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76600b95a0 msgr2=0x7f7660161b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f765dd9b640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76600b95a0 0x7f7660161b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f765dd9b640 1 -- 172.21.15.27:0/481767620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76600c7b20 con 0x7f766015df00 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f765dd9b640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766015df00 0x7f766015ecf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f76540017e0 tx=0x7f765400cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f764effd640 1 -- 172.21.15.27:0/481767620 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765400ce60 con 0x7f766015df00 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f764effd640 1 -- 172.21.15.27:0/481767620 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7654005af0 con 0x7f766015df00 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f764effd640 1 -- 172.21.15.27:0/481767620 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654007070 con 0x7f766015df00 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76600c7e00 con 0x7f766015df00 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.945+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76600c83c0 con 0x7f766015df00 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.946+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7660067ff0 con 0x7f766015df00 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.947+0000 7f764effd640 1 -- 172.21.15.27:0/481767620 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7654007210 con 0x7f766015df00 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.948+0000 7f764effd640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f764006b4d0 0x7f764006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.948+0000 7f764effd640 1 -- 172.21.15.27:0/481767620 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7654089f10 con 0x7f766015df00 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.948+0000 7f765e59c640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f764006b4d0 0x7f764006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.948+0000 7f765e59c640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f764006b4d0 0x7f764006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7650034000 tx=0x7f765002f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:07.952+0000 7f764effd640 1 -- 172.21.15.27:0/481767620 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7654058d00 con 0x7f766015df00 2024-03-06T07:25:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.171+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76600721e0 con 0x7f766015df00 2024-03-06T07:25:08.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.172+0000 7f764effd640 1 -- 172.21.15.27:0/481767620 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f7654058600 con 0x7f766015df00 2024-03-06T07:25:08.173 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:08.173 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5126,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2827284911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2827284911},{"type":"v1","addr":"172.21.15.69:6836","nonce":2827284911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5419,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1663054032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1663054032},{"type":"v1","addr":"172.21.15.69:6837","nonce":1663054032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2744957333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2744957333},{"type":"v1","addr":"172.21.15.27:6834","nonce":2744957333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:45.201958+0000","modified":"2024-03-06T07:24:48.290611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5754":{"gid":5754,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2008146374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2008146374},{"type":"v1","addr":"172.21.15.27:6835","nonce":2008146374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5754,"qdb_cluster":[5754]},"id":3}]} 2024-03-06T07:25:08.174 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-03-06T07:25:08.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f764006b4d0 msgr2=0x7f764006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f764006b4d0 0x7f764006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7650034000 tx=0x7f765002f000 comp rx=0 tx=0).stop 2024-03-06T07:25:08.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766015df00 msgr2=0x7f766015ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766015df00 0x7f766015ecf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f76540017e0 tx=0x7f765400cb20 comp rx=0 tx=0).stop 2024-03-06T07:25:08.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 shutdown_connections 2024-03-06T07:25:08.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f764006b4d0 0x7f764006d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76600c8bc0 0x7f7660150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766015df00 0x7f766015ecf0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 --2- 172.21.15.27:0/481767620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76600b95a0 0x7f7660161b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 >> 172.21.15.27:0/481767620 conn(0x7f7660071960 msgr2=0x7f76600b82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 shutdown_connections 2024-03-06T07:25:08.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.174+0000 7f7664977640 1 -- 172.21.15.27:0/481767620 wait complete. 2024-03-06T07:25:08.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.291+0000 7fd011c83640 1 Processor -- start 2024-03-06T07:25:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.291+0000 7fd011c83640 1 -- start start 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd011c83640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd00c071540 0x7fd00c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd011c83640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd00c0b2b50 0x7fd00c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd011c83640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd00c0ce610 con 0x7fd00c0d3400 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd011c83640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00c0d2f70 con 0x7fd00c0b2b50 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd011c83640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00c0d4be0 con 0x7fd00c071540 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd00bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd00c0b2b50 0x7fd00c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd00bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd00c0b2b50 0x7fd00c0b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33348/0 (socket says 172.21.15.27:33348) 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd00bfff640 1 -- 172.21.15.27:0/2780811159 learned_addr learned my addr 172.21.15.27:0/2780811159 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd00b7fe640 1 --2- 172.21.15.27:0/2780811159 >> v2:172.21.15.69:3300/0 conn(0x7fd00c071540 0x7fd00c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd00bfff640 1 -- 172.21.15.27:0/2780811159 >> v2:172.21.15.69:3300/0 conn(0x7fd00c071540 msgr2=0x7fd00c0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd00bfff640 1 --2- 172.21.15.27:0/2780811159 >> v2:172.21.15.69:3300/0 conn(0x7fd00c071540 0x7fd00c0b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd00bfff640 1 -- 172.21.15.27:0/2780811159 >> v1:172.21.15.69:6789/0 conn(0x7fd00c0d3400 legacy=0x7fd00c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.293+0000 7fd00bfff640 1 -- 172.21.15.27:0/2780811159 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c0d2920 con 0x7fd00c0b2b50 2024-03-06T07:25:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.294+0000 7fd00bfff640 1 --2- 172.21.15.27:0/2780811159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 0x7fd00c0b6e20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0d4da0 tx=0x7fd00402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3becdeff88c0c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.294+0000 7fd00a7fc640 1 -- 172.21.15.27:0/2780811159 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00402ca80 con 0x7fd00c0b2b50 2024-03-06T07:25:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.294+0000 7fd00a7fc640 1 -- 172.21.15.27:0/2780811159 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00402cc20 con 0x7fd00c0b2b50 2024-03-06T07:25:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.294+0000 7fd00a7fc640 1 -- 172.21.15.27:0/2780811159 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0040343f0 con 0x7fd00c0b2b50 2024-03-06T07:25:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.295+0000 7fd011c83640 1 -- 172.21.15.27:0/2780811159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 msgr2=0x7fd00c0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.295+0000 7fd011c83640 1 --2- 172.21.15.27:0/2780811159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 0x7fd00c0b6e20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0d4da0 tx=0x7fd00402bee0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.295+0000 7fd011c83640 1 -- 172.21.15.27:0/2780811159 shutdown_connections 2024-03-06T07:25:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.295+0000 7fd011c83640 1 --2- 172.21.15.27:0/2780811159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 0x7fd00c0b6e20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.295+0000 7fd011c83640 1 --2- 172.21.15.27:0/2780811159 >> v2:172.21.15.69:3300/0 conn(0x7fd00c071540 0x7fd00c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.295+0000 7fd011c83640 1 -- 172.21.15.27:0/2780811159 >> 172.21.15.27:0/2780811159 conn(0x7fd00c0d3ca0 msgr2=0x7fd00c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.295+0000 7fd011c83640 1 -- 172.21.15.27:0/2780811159 shutdown_connections 2024-03-06T07:25:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.295+0000 7fd011c83640 1 -- 172.21.15.27:0/2780811159 wait complete. 2024-03-06T07:25:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.297+0000 7fd011c83640 1 Processor -- start 2024-03-06T07:25:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.297+0000 7fd011c83640 1 -- start start 2024-03-06T07:25:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.298+0000 7fd011c83640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00c071540 0x7fd00c1533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.298+0000 7fd011c83640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 0x7fd00c0b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd011c83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd00c0d3400 0x7fd00c0cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd011c83640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c15e370 con 0x7fd00c0b2b50 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd011c83640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c0ce8f0 con 0x7fd00c071540 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd011c83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c155940 con 0x7fd00c0d3400 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 0x7fd00c0b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 0x7fd00c0b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33350/0 (socket says 172.21.15.27:33350) 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00affd640 1 -- 172.21.15.27:0/437411400 learned_addr learned my addr 172.21.15.27:0/437411400 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00b7fe640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00c071540 0x7fd00c1533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00affd640 1 -- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd00c0d3400 msgr2=0x7fd00c0cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00bfff640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd00c0d3400 0x7fd00c0cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00affd640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd00c0d3400 0x7fd00c0cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00affd640 1 -- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00c071540 msgr2=0x7fd00c1533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00affd640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00c071540 0x7fd00c1533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00affd640 1 -- 172.21.15.27:0/437411400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c1580c0 con 0x7fd00c0b2b50 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00b7fe640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00c071540 0x7fd00c1533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd00affd640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 0x7fd00c0b5cd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00b3c0 tx=0x7fcffc00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd003fff640 1 -- 172.21.15.27:0/437411400 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc011830 con 0x7fd00c0b2b50 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd003fff640 1 -- 172.21.15.27:0/437411400 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc016070 con 0x7fd00c0b2b50 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd003fff640 1 -- 172.21.15.27:0/437411400 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc011c50 con 0x7fd00c0b2b50 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00c1583a0 con 0x7fd00c0b2b50 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.299+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00c158960 con 0x7fd00c0b2b50 2024-03-06T07:25:08.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.301+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00c068700 con 0x7fd00c0b2b50 2024-03-06T07:25:08.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.301+0000 7fd003fff640 1 -- 172.21.15.27:0/437411400 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcffc01b460 con 0x7fd00c0b2b50 2024-03-06T07:25:08.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.301+0000 7fd003fff640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.301+0000 7fd003fff640 1 -- 172.21.15.27:0/437411400 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcffc089d50 con 0x7fd00c0b2b50 2024-03-06T07:25:08.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.301+0000 7fd00b7fe640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.302+0000 7fd00b7fe640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcff4008890 tx=0x7fcff4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.306+0000 7fd003fff640 1 -- 172.21.15.27:0/437411400 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcffc08d030 con 0x7fd00c0b2b50 2024-03-06T07:25:08.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.599+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd00c0b7b20 con 0x7fd00c0b2b50 2024-03-06T07:25:08.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.600+0000 7fd003fff640 1 -- 172.21.15.27:0/437411400 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fcffc020020 con 0x7fd00c0b2b50 2024-03-06T07:25:08.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:08.602 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:24:46.825646+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:43.493554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:44.172410+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:08.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 msgr2=0x7fcfec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcff4008890 tx=0x7fcff4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 msgr2=0x7fd00c0b5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 0x7fd00c0b5cd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00b3c0 tx=0x7fcffc00b880 comp rx=0 tx=0).stop 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 shutdown_connections 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd00c0d3400 0x7fd00c0cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0b2b50 0x7fd00c0b5cd0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 --2- 172.21.15.27:0/437411400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00c071540 0x7fd00c1533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 >> 172.21.15.27:0/437411400 conn(0x7fd00c0d3ca0 msgr2=0x7fd00c0d4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 shutdown_connections 2024-03-06T07:25:08.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.602+0000 7fd011c83640 1 -- 172.21.15.27:0/437411400 wait complete. 2024-03-06T07:25:08.615 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:08.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:08.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.721+0000 7f002a629640 1 Processor -- start 2024-03-06T07:25:08.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.721+0000 7f002a629640 1 -- start start 2024-03-06T07:25:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f002a629640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f002406b5c0 0x7f00240b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f002a629640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 0x7f002406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f002a629640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00240b3340 con 0x7f00240ae9b0 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f002a629640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00240cfa00 con 0x7f002406b5c0 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f002a629640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00240cee90 con 0x7f00240b0e90 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f0023fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f002406b5c0 0x7f00240b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f0023fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f002406b5c0 0x7f00240b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33354/0 (socket says 172.21.15.27:33354) 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f0023fff640 1 -- 172.21.15.27:0/3097187390 learned_addr learned my addr 172.21.15.27:0/3097187390 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f0028bc6640 1 --1- 172.21.15.27:0/3097187390 >> v1:172.21.15.27:6789/0 conn(0x7f00240ae9b0 0x7f00240d03a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41956/0 (socket says 172.21.15.27:41956) 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f00237fe640 1 --2- 172.21.15.27:0/3097187390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 0x7f002406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f0023fff640 1 -- 172.21.15.27:0/3097187390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 msgr2=0x7f002406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f0023fff640 1 --2- 172.21.15.27:0/3097187390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 0x7f002406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f0023fff640 1 -- 172.21.15.27:0/3097187390 >> v1:172.21.15.27:6789/0 conn(0x7f00240ae9b0 legacy=0x7f00240d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.722+0000 7f0023fff640 1 -- 172.21.15.27:0/3097187390 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00240cfc50 con 0x7f002406b5c0 2024-03-06T07:25:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.723+0000 7f0023fff640 1 --2- 172.21.15.27:0/3097187390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 0x7f00240b2e00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0014009870 tx=0x7f001402b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655f91f935544d34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.723+0000 7f0022ffd640 1 -- 172.21.15.27:0/3097187390 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014037070 con 0x7f002406b5c0 2024-03-06T07:25:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.723+0000 7f0022ffd640 1 -- 172.21.15.27:0/3097187390 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0014030710 con 0x7f002406b5c0 2024-03-06T07:25:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.723+0000 7f0022ffd640 1 -- 172.21.15.27:0/3097187390 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014030ae0 con 0x7f002406b5c0 2024-03-06T07:25:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.724+0000 7f002a629640 1 -- 172.21.15.27:0/3097187390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 msgr2=0x7f00240b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.724+0000 7f002a629640 1 --2- 172.21.15.27:0/3097187390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 0x7f00240b2e00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0014009870 tx=0x7f001402b740 comp rx=0 tx=0).stop 2024-03-06T07:25:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.724+0000 7f002a629640 1 -- 172.21.15.27:0/3097187390 shutdown_connections 2024-03-06T07:25:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.724+0000 7f002a629640 1 --2- 172.21.15.27:0/3097187390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 0x7f002406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.724+0000 7f002a629640 1 --2- 172.21.15.27:0/3097187390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 0x7f00240b2e00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.724+0000 7f002a629640 1 -- 172.21.15.27:0/3097187390 >> 172.21.15.27:0/3097187390 conn(0x7f00240b36d0 msgr2=0x7f00240b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.724+0000 7f002a629640 1 -- 172.21.15.27:0/3097187390 shutdown_connections 2024-03-06T07:25:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.724+0000 7f002a629640 1 -- 172.21.15.27:0/3097187390 wait complete. 2024-03-06T07:25:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.725+0000 7f002a629640 1 Processor -- start 2024-03-06T07:25:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.725+0000 7f002a629640 1 -- start start 2024-03-06T07:25:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f002a629640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 0x7f00240cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f002a629640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00240ae9b0 0x7f00240cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f002a629640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 0x7f00240c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f002a629640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002406c790 con 0x7f002406b5c0 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f002a629640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002406bee0 con 0x7f00240ae9b0 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f002a629640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002406c910 con 0x7f00240b0e90 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0023fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 0x7f00240cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0023fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 0x7f00240cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33364/0 (socket says 172.21.15.27:33364) 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0023fff640 1 -- 172.21.15.27:0/971796627 learned_addr learned my addr 172.21.15.27:0/971796627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0028bc6640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 0x7f00240c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0023fff640 1 -- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 msgr2=0x7f00240c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f00237fe640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00240ae9b0 0x7f00240cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0023fff640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 0x7f00240c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0023fff640 1 -- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00240ae9b0 msgr2=0x7f00240cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0023fff640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00240ae9b0 0x7f00240cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0023fff640 1 -- 172.21.15.27:0/971796627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00240b6630 con 0x7f002406b5c0 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0028bc6640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 0x7f00240c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f00237fe640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00240ae9b0 0x7f00240cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0023fff640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 0x7f00240cad50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0014000c00 tx=0x7f00140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.726+0000 7f0020ff9640 1 -- 172.21.15.27:0/971796627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014041020 con 0x7f002406b5c0 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.727+0000 7f0020ff9640 1 -- 172.21.15.27:0/971796627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001403c870 con 0x7f002406b5c0 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.727+0000 7f0020ff9640 1 -- 172.21.15.27:0/971796627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014037040 con 0x7f002406b5c0 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.727+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00240b6910 con 0x7f002406b5c0 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.727+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00240c92b0 con 0x7f002406b5c0 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.728+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0024068700 con 0x7f002406b5c0 2024-03-06T07:25:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.728+0000 7f0020ff9640 1 -- 172.21.15.27:0/971796627 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0014044020 con 0x7f002406b5c0 2024-03-06T07:25:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.728+0000 7f0020ff9640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f001006b4d0 0x7f001006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.729+0000 7f0020ff9640 1 -- 172.21.15.27:0/971796627 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00140abb90 con 0x7f002406b5c0 2024-03-06T07:25:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.729+0000 7f00237fe640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f001006b4d0 0x7f001006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.729+0000 7f00237fe640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f001006b4d0 0x7f001006d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f00240b05e0 tx=0x7f0018008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.733+0000 7f0020ff9640 1 -- 172.21.15.27:0/971796627 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f001404b030 con 0x7f002406b5c0 2024-03-06T07:25:08.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.953+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f002406ed30 con 0x7f002406b5c0 2024-03-06T07:25:08.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.954+0000 7f0020ff9640 1 -- 172.21.15.27:0/971796627 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f001407aa10 con 0x7f002406b5c0 2024-03-06T07:25:08.955 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:08.955 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5126,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2827284911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2827284911},{"type":"v1","addr":"172.21.15.69:6836","nonce":2827284911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5419,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1663054032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1663054032},{"type":"v1","addr":"172.21.15.69:6837","nonce":1663054032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2744957333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2744957333},{"type":"v1","addr":"172.21.15.27:6834","nonce":2744957333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:45.201958+0000","modified":"2024-03-06T07:24:48.290611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5754":{"gid":5754,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2008146374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2008146374},{"type":"v1","addr":"172.21.15.27:6835","nonce":2008146374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5754,"qdb_cluster":[5754]},"id":3}]} 2024-03-06T07:25:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f001006b4d0 msgr2=0x7f001006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f001006b4d0 0x7f001006d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f00240b05e0 tx=0x7f0018008040 comp rx=0 tx=0).stop 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 msgr2=0x7f00240cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 0x7f00240cad50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0014000c00 tx=0x7f00140046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 shutdown_connections 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f001006b4d0 0x7f001006d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00240b0e90 0x7f00240c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00240ae9b0 0x7f00240cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 --2- 172.21.15.27:0/971796627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f002406b5c0 0x7f00240cad50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 >> 172.21.15.27:0/971796627 conn(0x7f00240b36d0 msgr2=0x7f002406f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 shutdown_connections 2024-03-06T07:25:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:08.956+0000 7f002a629640 1 -- 172.21.15.27:0/971796627 wait complete. 2024-03-06T07:25:08.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:09.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.074+0000 7fa6703ae640 1 Processor -- start 2024-03-06T07:25:09.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.074+0000 7fa6703ae640 1 -- start start 2024-03-06T07:25:09.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa6703ae640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa668071540 0x7fa6680b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa6703ae640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6680b2b50 0x7fa6680b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa6703ae640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6680ce610 con 0x7fa6680d3400 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa6703ae640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6680d2f70 con 0x7fa668071540 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa6703ae640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6680d4be0 con 0x7fa6680b2b50 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa66e94b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6680b2b50 0x7fa6680b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa66e14a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa668071540 0x7fa6680b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa66d949640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa6680d3400 0x7fa668071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:52646/0 (socket says 172.21.15.27:52646) 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa66e94b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6680b2b50 0x7fa6680b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:60636/0 (socket says 172.21.15.27:60636) 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa66d949640 1 -- 172.21.15.27:0/4001851075 learned_addr learned my addr 172.21.15.27:0/4001851075 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.075+0000 7fa66e14a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa668071540 0x7fa6680b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:34228/0 (socket says 172.21.15.27:34228) 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.076+0000 7fa66e14a640 1 -- 172.21.15.27:0/4001851075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6680b2b50 msgr2=0x7fa6680b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.076+0000 7fa66e14a640 1 --2- 172.21.15.27:0/4001851075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6680b2b50 0x7fa6680b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.076+0000 7fa66e14a640 1 -- 172.21.15.27:0/4001851075 >> v1:172.21.15.69:6789/0 conn(0x7fa6680d3400 legacy=0x7fa668071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.076+0000 7fa66e14a640 1 -- 172.21.15.27:0/4001851075 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6680d2920 con 0x7fa668071540 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.076+0000 7fa66e14a640 1 --2- 172.21.15.27:0/4001851075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa668071540 0x7fa6680b30a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa6640097a0 tx=0x7fa66402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4923adc2663ddef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa66d148640 1 -- 172.21.15.27:0/4001851075 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664035020 con 0x7fa668071540 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa66d148640 1 -- 172.21.15.27:0/4001851075 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa664031830 con 0x7fa668071540 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa66d148640 1 -- 172.21.15.27:0/4001851075 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664031c00 con 0x7fa668071540 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa6703ae640 1 -- 172.21.15.27:0/4001851075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa668071540 msgr2=0x7fa6680b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa6703ae640 1 --2- 172.21.15.27:0/4001851075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa668071540 0x7fa6680b30a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa6640097a0 tx=0x7fa66402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa6703ae640 1 -- 172.21.15.27:0/4001851075 shutdown_connections 2024-03-06T07:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa6703ae640 1 --2- 172.21.15.27:0/4001851075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6680b2b50 0x7fa6680b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa6703ae640 1 --2- 172.21.15.27:0/4001851075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa668071540 0x7fa6680b30a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa6703ae640 1 -- 172.21.15.27:0/4001851075 >> 172.21.15.27:0/4001851075 conn(0x7fa6680d3ca0 msgr2=0x7fa6680d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa6703ae640 1 -- 172.21.15.27:0/4001851075 shutdown_connections 2024-03-06T07:25:09.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.077+0000 7fa6703ae640 1 -- 172.21.15.27:0/4001851075 wait complete. 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.078+0000 7fa6703ae640 1 Processor -- start 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.078+0000 7fa6703ae640 1 -- start start 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa6703ae640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa668071540 0x7fa66814d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa6703ae640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680b2b50 0x7fa66815b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa6703ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680d3400 0x7fa66815f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa6703ae640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66815e370 con 0x7fa6680d3400 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa6703ae640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6680ce8f0 con 0x7fa6680b2b50 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa6703ae640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6680cad70 con 0x7fa668071540 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa66e94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680d3400 0x7fa66815f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa66e94b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680d3400 0x7fa66815f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33372/0 (socket says 172.21.15.27:33372) 2024-03-06T07:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa66e94b640 1 -- 172.21.15.27:0/3891734140 learned_addr learned my addr 172.21.15.27:0/3891734140 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa66e14a640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa668071540 0x7fa66814d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.079+0000 7fa66d949640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680b2b50 0x7fa66815b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa66e94b640 1 -- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa668071540 msgr2=0x7fa66814d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa66e94b640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa668071540 0x7fa66814d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa66e94b640 1 -- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680b2b50 msgr2=0x7fa66815b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa66e94b640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680b2b50 0x7fa66815b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa66e94b640 1 -- 172.21.15.27:0/3891734140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6680cafc0 con 0x7fa6680d3400 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa66e94b640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680d3400 0x7fa66815f450 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa66000efc0 tx=0x7fa66000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa65effd640 1 -- 172.21.15.27:0/3891734140 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660018020 con 0x7fa6680d3400 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa65effd640 1 -- 172.21.15.27:0/3891734140 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa660014480 con 0x7fa6680d3400 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6680cb2a0 con 0x7fa6680d3400 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa65effd640 1 -- 172.21.15.27:0/3891734140 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660014870 con 0x7fa6680d3400 2024-03-06T07:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.080+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6680cb860 con 0x7fa6680d3400 2024-03-06T07:25:09.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.081+0000 7fa65effd640 1 -- 172.21.15.27:0/3891734140 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa660005d10 con 0x7fa6680d3400 2024-03-06T07:25:09.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.081+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa668068700 con 0x7fa6680d3400 2024-03-06T07:25:09.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.082+0000 7fa65effd640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.082+0000 7fa65effd640 1 -- 172.21.15.27:0/3891734140 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa66008ac90 con 0x7fa6680d3400 2024-03-06T07:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.082+0000 7fa66e14a640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.083+0000 7fa66e14a640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa664036000 tx=0x7fa664008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.086+0000 7fa65effd640 1 -- 172.21.15.27:0/3891734140 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa660022070 con 0x7fa6680d3400 2024-03-06T07:25:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.314+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6680b8850 con 0x7fa6680d3400 2024-03-06T07:25:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.315+0000 7fa65effd640 1 -- 172.21.15.27:0/3891734140 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fa660010040 con 0x7fa6680d3400 2024-03-06T07:25:09.316 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:09.316 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5126,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2827284911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2827284911},{"type":"v1","addr":"172.21.15.69:6836","nonce":2827284911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5419,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1663054032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1663054032},{"type":"v1","addr":"172.21.15.69:6837","nonce":1663054032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2744957333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2744957333},{"type":"v1","addr":"172.21.15.27:6834","nonce":2744957333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:45.201958+0000","modified":"2024-03-06T07:24:48.290611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5754},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5754":{"gid":5754,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2008146374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2008146374},{"type":"v1","addr":"172.21.15.27:6835","nonce":2008146374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5754,"qdb_cluster":[5754]},"id":3}]} 2024-03-06T07:25:09.317 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-03-06T07:25:09.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 msgr2=0x7fa64c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa664036000 tx=0x7fa664008030 comp rx=0 tx=0).stop 2024-03-06T07:25:09.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680d3400 msgr2=0x7fa66815f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680d3400 0x7fa66815f450 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa66000efc0 tx=0x7fa66000c490 comp rx=0 tx=0).stop 2024-03-06T07:25:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 shutdown_connections 2024-03-06T07:25:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6680d3400 0x7fa66815f450 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6680b2b50 0x7fa66815b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 --2- 172.21.15.27:0/3891734140 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa668071540 0x7fa66814d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.316+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 >> 172.21.15.27:0/3891734140 conn(0x7fa6680d3ca0 msgr2=0x7fa668160590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.317+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 shutdown_connections 2024-03-06T07:25:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.317+0000 7fa6703ae640 1 -- 172.21.15.27:0/3891734140 wait complete. 2024-03-06T07:25:09.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:09.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.434+0000 7fa978c88640 1 Processor -- start 2024-03-06T07:25:09.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.434+0000 7fa978c88640 1 -- start start 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa978c88640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa97415df00 0x7fa9740d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa978c88640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9740b95a0 0x7fa9740b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa978c88640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9740cea30 con 0x7fa9740b9a30 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa978c88640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9740d2af0 con 0x7fa9740b95a0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa978c88640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9740cf3c0 con 0x7fa97415df00 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa969d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9740b95a0 0x7fa9740b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa969d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9740b95a0 0x7fa9740b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33376/0 (socket says 172.21.15.27:33376) 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa969d9b640 1 -- 172.21.15.27:0/3570508386 learned_addr learned my addr 172.21.15.27:0/3570508386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa969d9b640 1 -- 172.21.15.27:0/3570508386 >> v2:172.21.15.69:3300/0 conn(0x7fa97415df00 msgr2=0x7fa9740d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa969d9b640 1 --2- 172.21.15.27:0/3570508386 >> v2:172.21.15.69:3300/0 conn(0x7fa97415df00 0x7fa9740d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa969d9b640 1 -- 172.21.15.27:0/3570508386 >> v1:172.21.15.27:6789/0 conn(0x7fa9740b9a30 legacy=0x7fa9740d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa969d9b640 1 -- 172.21.15.27:0/3570508386 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9740cf0e0 con 0x7fa9740b95a0 2024-03-06T07:25:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.435+0000 7fa971d9b640 1 -- 172.21.15.27:0/3570508386 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514616441 0 0) 0x7fa9740cea30 con 0x7fa9740b9a30 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.436+0000 7fa969d9b640 1 --2- 172.21.15.27:0/3570508386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9740b95a0 0x7fa9740b5700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa9740d11b0 tx=0x7fa95c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96e3c0e029c5809 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.436+0000 7fa971d9b640 1 -- 172.21.15.27:0/3570508386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c034070 con 0x7fa9740b95a0 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.436+0000 7fa971d9b640 1 -- 172.21.15.27:0/3570508386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95c030860 con 0x7fa9740b95a0 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.436+0000 7fa971d9b640 1 -- 172.21.15.27:0/3570508386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c030c30 con 0x7fa9740b95a0 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.437+0000 7fa978c88640 1 -- 172.21.15.27:0/3570508386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9740b95a0 msgr2=0x7fa9740b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.437+0000 7fa978c88640 1 --2- 172.21.15.27:0/3570508386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9740b95a0 0x7fa9740b5700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa9740d11b0 tx=0x7fa95c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.437+0000 7fa978c88640 1 -- 172.21.15.27:0/3570508386 shutdown_connections 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.437+0000 7fa978c88640 1 --2- 172.21.15.27:0/3570508386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9740b95a0 0x7fa9740b5700 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.437+0000 7fa978c88640 1 --2- 172.21.15.27:0/3570508386 >> v2:172.21.15.69:3300/0 conn(0x7fa97415df00 0x7fa9740d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.437+0000 7fa978c88640 1 -- 172.21.15.27:0/3570508386 >> 172.21.15.27:0/3570508386 conn(0x7fa974071960 msgr2=0x7fa974071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.437+0000 7fa978c88640 1 -- 172.21.15.27:0/3570508386 shutdown_connections 2024-03-06T07:25:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.437+0000 7fa978c88640 1 -- 172.21.15.27:0/3570508386 wait complete. 2024-03-06T07:25:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.439+0000 7fa978c88640 1 Processor -- start 2024-03-06T07:25:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.439+0000 7fa978c88640 1 -- start start 2024-03-06T07:25:09.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.440+0000 7fa978c88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9740b95a0 0x7fa97415fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.440+0000 7fa978c88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97415df00 0x7fa9740b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.440+0000 7fa978c88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa97415e2e0 0x7fa974158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.440+0000 7fa978c88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740cec70 con 0x7fa97415df00 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa978c88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740ce750 con 0x7fa9740b95a0 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa978c88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740cedf0 con 0x7fa97415e2e0 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa969d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97415df00 0x7fa9740b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa969d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97415df00 0x7fa9740b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33380/0 (socket says 172.21.15.27:33380) 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa969d9b640 1 -- 172.21.15.27:0/2296772928 learned_addr learned my addr 172.21.15.27:0/2296772928 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa97259c640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9740b95a0 0x7fa97415fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa972d9d640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa97415e2e0 0x7fa974158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa969d9b640 1 -- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa97415e2e0 msgr2=0x7fa974158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa969d9b640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa97415e2e0 0x7fa974158410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa969d9b640 1 -- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9740b95a0 msgr2=0x7fa97415fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa969d9b640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9740b95a0 0x7fa97415fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa969d9b640 1 -- 172.21.15.27:0/2296772928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97415e760 con 0x7fa97415df00 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa97259c640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9740b95a0 0x7fa97415fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa969d9b640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97415df00 0x7fa9740b49a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa95c000c00 tx=0x7fa95c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa96b7fe640 1 -- 172.21.15.27:0/2296772928 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c048070 con 0x7fa97415df00 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa96b7fe640 1 -- 172.21.15.27:0/2296772928 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95c030660 con 0x7fa97415df00 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa974158950 con 0x7fa97415df00 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa96b7fe640 1 -- 172.21.15.27:0/2296772928 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c034040 con 0x7fa97415df00 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.441+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97415ed10 con 0x7fa97415df00 2024-03-06T07:25:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.442+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa974067ff0 con 0x7fa97415df00 2024-03-06T07:25:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.443+0000 7fa96b7fe640 1 -- 172.21.15.27:0/2296772928 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa95c04b070 con 0x7fa97415df00 2024-03-06T07:25:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.443+0000 7fa96b7fe640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa95406b500 0x7fa95406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.444+0000 7fa96b7fe640 1 -- 172.21.15.27:0/2296772928 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa95c0ac2d0 con 0x7fa97415df00 2024-03-06T07:25:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.444+0000 7fa97259c640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa95406b500 0x7fa95406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.444+0000 7fa97259c640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa95406b500 0x7fa95406d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa96400a870 tx=0x7fa964008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.448+0000 7fa96b7fe640 1 -- 172.21.15.27:0/2296772928 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa95c07b150 con 0x7fa97415df00 2024-03-06T07:25:09.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.630+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa974071ec0 con 0x7fa97415df00 2024-03-06T07:25:09.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.631+0000 7fa96b7fe640 1 -- 172.21.15.27:0/2296772928 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fa95c07aa50 con 0x7fa97415df00 2024-03-06T07:25:09.633 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:09.633 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:24:46.825646+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:43.493554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:44.172410+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa95406b500 msgr2=0x7fa95406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa95406b500 0x7fa95406d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa96400a870 tx=0x7fa964008040 comp rx=0 tx=0).stop 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97415df00 msgr2=0x7fa9740b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97415df00 0x7fa9740b49a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa95c000c00 tx=0x7fa95c004730 comp rx=0 tx=0).stop 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 shutdown_connections 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa95406b500 0x7fa95406d9c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa97415e2e0 0x7fa974158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa97415df00 0x7fa9740b49a0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 --2- 172.21.15.27:0/2296772928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9740b95a0 0x7fa97415fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 >> 172.21.15.27:0/2296772928 conn(0x7fa974071960 msgr2=0x7fa9740b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 shutdown_connections 2024-03-06T07:25:09.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.633+0000 7fa978c88640 1 -- 172.21.15.27:0/2296772928 wait complete. 2024-03-06T07:25:09.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.782+0000 7fbe6af3e640 1 Processor -- start 2024-03-06T07:25:09.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.782+0000 7fbe6af3e640 1 -- start start 2024-03-06T07:25:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.783+0000 7fbe6af3e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe64071540 0x7fbe640b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.783+0000 7fbe6af3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe640d3400 0x7fbe64071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.783+0000 7fbe6af3e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe64071e60 con 0x7fbe640b2b50 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.783+0000 7fbe6af3e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe640d2f70 con 0x7fbe64071540 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.783+0000 7fbe6af3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640d4be0 con 0x7fbe640d3400 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.783+0000 7fbe68cda640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe64071540 0x7fbe640b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.783+0000 7fbe694db640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbe640b2b50 0x7fbe640d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41968/0 (socket says 172.21.15.27:41968) 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.783+0000 7fbe694db640 1 -- 172.21.15.27:0/2736306702 learned_addr learned my addr 172.21.15.27:0/2736306702 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.783+0000 7fbe63fff640 1 --2- 172.21.15.27:0/2736306702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe640d3400 0x7fbe64071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.784+0000 7fbe68cda640 1 -- 172.21.15.27:0/2736306702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe640d3400 msgr2=0x7fbe64071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.784+0000 7fbe68cda640 1 --2- 172.21.15.27:0/2736306702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe640d3400 0x7fbe64071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.784+0000 7fbe68cda640 1 -- 172.21.15.27:0/2736306702 >> v1:172.21.15.27:6789/0 conn(0x7fbe640b2b50 legacy=0x7fbe640d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.784+0000 7fbe68cda640 1 -- 172.21.15.27:0/2736306702 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe640ce610 con 0x7fbe64071540 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.784+0000 7fbe637fe640 1 -- 172.21.15.27:0/2736306702 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243481696 0 0) 0x7fbe64071e60 con 0x7fbe640b2b50 2024-03-06T07:25:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.784+0000 7fbe68cda640 1 --2- 172.21.15.27:0/2736306702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64071540 0x7fbe640b30a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbe580097a0 tx=0x7fbe5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd163805074006e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.784+0000 7fbe637fe640 1 -- 172.21.15.27:0/2736306702 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58035020 con 0x7fbe64071540 2024-03-06T07:25:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe637fe640 1 -- 172.21.15.27:0/2736306702 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe58031830 con 0x7fbe64071540 2024-03-06T07:25:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe637fe640 1 -- 172.21.15.27:0/2736306702 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58031c00 con 0x7fbe64071540 2024-03-06T07:25:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe6af3e640 1 -- 172.21.15.27:0/2736306702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64071540 msgr2=0x7fbe640b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe6af3e640 1 --2- 172.21.15.27:0/2736306702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64071540 0x7fbe640b30a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbe580097a0 tx=0x7fbe5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe6af3e640 1 -- 172.21.15.27:0/2736306702 shutdown_connections 2024-03-06T07:25:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe6af3e640 1 --2- 172.21.15.27:0/2736306702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe640d3400 0x7fbe64071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe6af3e640 1 --2- 172.21.15.27:0/2736306702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64071540 0x7fbe640b30a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe6af3e640 1 -- 172.21.15.27:0/2736306702 >> 172.21.15.27:0/2736306702 conn(0x7fbe640d3ca0 msgr2=0x7fbe640d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe6af3e640 1 -- 172.21.15.27:0/2736306702 shutdown_connections 2024-03-06T07:25:09.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.785+0000 7fbe6af3e640 1 -- 172.21.15.27:0/2736306702 wait complete. 2024-03-06T07:25:09.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.787+0000 7fbe6af3e640 1 Processor -- start 2024-03-06T07:25:09.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.787+0000 7fbe6af3e640 1 -- start start 2024-03-06T07:25:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe6af3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe64071540 0x7fbe640ab770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe6af3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe640b2b50 0x7fbe6415d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe6af3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe640d3400 0x7fbe640ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe6af3e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640ce8f0 con 0x7fbe640b2b50 2024-03-06T07:25:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe6af3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640d2c00 con 0x7fbe640d3400 2024-03-06T07:25:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe6af3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6415e370 con 0x7fbe64071540 2024-03-06T07:25:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe640b2b50 0x7fbe6415d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe640b2b50 0x7fbe6415d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33392/0 (socket says 172.21.15.27:33392) 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe63fff640 1 -- 172.21.15.27:0/3555173872 learned_addr learned my addr 172.21.15.27:0/3555173872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe63fff640 1 -- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe64071540 msgr2=0x7fbe640ab770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe63fff640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe64071540 0x7fbe640ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe694db640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe640d3400 0x7fbe640ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe63fff640 1 -- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe640d3400 msgr2=0x7fbe640ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe63fff640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe640d3400 0x7fbe640ac950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe63fff640 1 -- 172.21.15.27:0/3555173872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe640ace90 con 0x7fbe640b2b50 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe63fff640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe640b2b50 0x7fbe6415d080 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbe5000d600 tx=0x7fbe5000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe617fa640 1 -- 172.21.15.27:0/3555173872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe50004240 con 0x7fbe640b2b50 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe617fa640 1 -- 172.21.15.27:0/3555173872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe500043e0 con 0x7fbe640b2b50 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe640cc5c0 con 0x7fbe640b2b50 2024-03-06T07:25:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe617fa640 1 -- 172.21.15.27:0/3555173872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe50003830 con 0x7fbe640b2b50 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.788+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe640ccb80 con 0x7fbe640b2b50 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.790+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe64068700 con 0x7fbe640b2b50 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.790+0000 7fbe617fa640 1 -- 172.21.15.27:0/3555173872 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe500039d0 con 0x7fbe640b2b50 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.790+0000 7fbe617fa640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe4806b4d0 0x7fbe4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.791+0000 7fbe617fa640 1 -- 172.21.15.27:0/3555173872 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe5008a460 con 0x7fbe640b2b50 2024-03-06T07:25:09.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.791+0000 7fbe68cda640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe4806b4d0 0x7fbe4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.791+0000 7fbe68cda640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe4806b4d0 0x7fbe4806d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbe58036000 tx=0x7fbe58030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:09.795+0000 7fbe617fa640 1 -- 172.21.15.27:0/3555173872 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe5001c360 con 0x7fbe640b2b50 2024-03-06T07:25:10.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.019+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbe640b95f0 con 0x7fbe640b2b50 2024-03-06T07:25:10.021 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:25:10.020+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:10.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.021+0000 7fbe617fa640 1 -- 172.21.15.27:0/3555173872 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbe50059370 con 0x7fbe640b2b50 2024-03-06T07:25:10.022 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe4806b4d0 msgr2=0x7fbe4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe4806b4d0 0x7fbe4806d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbe58036000 tx=0x7fbe58030000 comp rx=0 tx=0).stop 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe640b2b50 msgr2=0x7fbe6415d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe640b2b50 0x7fbe6415d080 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbe5000d600 tx=0x7fbe5000dac0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 shutdown_connections 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe4806b4d0 0x7fbe4806d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe640d3400 0x7fbe640ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe640b2b50 0x7fbe6415d080 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 --2- 172.21.15.27:0/3555173872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbe64071540 0x7fbe640ab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 >> 172.21.15.27:0/3555173872 conn(0x7fbe640d3ca0 msgr2=0x7fbe6415f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 shutdown_connections 2024-03-06T07:25:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.024+0000 7fbe6af3e640 1 -- 172.21.15.27:0/3555173872 wait complete. 2024-03-06T07:25:10.037 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:10.063 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:25:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.146+0000 7f3611e81640 1 Processor -- start 2024-03-06T07:25:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.146+0000 7f3611e81640 1 -- start start 2024-03-06T07:25:10.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f3611e81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c15dbd0 0x7f360c15ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f3611e81640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f360c0b5840 0x7f360c0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f3611e81640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360c15f3b0 con 0x7f360c15dfb0 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f3611e81640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f360c06f070 con 0x7f360c0b5840 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f3611e81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f360c071940 con 0x7f360c15dbd0 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f360affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f360c0b5840 0x7f360c0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f360affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f360c0b5840 0x7f360c0b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33414/0 (socket says 172.21.15.27:33414) 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f360affd640 1 -- 172.21.15.27:0/938038432 learned_addr learned my addr 172.21.15.27:0/938038432 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f360b7fe640 1 --2- 172.21.15.27:0/938038432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c15dbd0 0x7f360c15ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f360affd640 1 -- 172.21.15.27:0/938038432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c15dbd0 msgr2=0x7f360c15ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f360affd640 1 --2- 172.21.15.27:0/938038432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c15dbd0 0x7f360c15ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f360affd640 1 -- 172.21.15.27:0/938038432 >> v1:172.21.15.69:6789/0 conn(0x7f360c15dfb0 legacy=0x7f360c161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.147+0000 7f360affd640 1 -- 172.21.15.27:0/938038432 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c0b5db0 con 0x7f360c0b5840 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.148+0000 7f360affd640 1 --2- 172.21.15.27:0/938038432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c0b5840 0x7f360c0b39b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3600009870 tx=0x7f360002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ff7f9e382bca45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.148+0000 7f360a7fc640 1 -- 172.21.15.27:0/938038432 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600035020 con 0x7f360c0b5840 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.148+0000 7f360a7fc640 1 -- 172.21.15.27:0/938038432 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600031640 con 0x7f360c0b5840 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.148+0000 7f360a7fc640 1 -- 172.21.15.27:0/938038432 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360002d040 con 0x7f360c0b5840 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.148+0000 7f3611e81640 1 -- 172.21.15.27:0/938038432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c0b5840 msgr2=0x7f360c0b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.149+0000 7f3611e81640 1 --2- 172.21.15.27:0/938038432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c0b5840 0x7f360c0b39b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3600009870 tx=0x7f360002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.149+0000 7f3611e81640 1 -- 172.21.15.27:0/938038432 shutdown_connections 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.149+0000 7f3611e81640 1 --2- 172.21.15.27:0/938038432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c0b5840 0x7f360c0b39b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.149+0000 7f3611e81640 1 --2- 172.21.15.27:0/938038432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c15dbd0 0x7f360c15ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.149+0000 7f3611e81640 1 -- 172.21.15.27:0/938038432 >> 172.21.15.27:0/938038432 conn(0x7f360c0b26b0 msgr2=0x7f360c0b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.149+0000 7f3611e81640 1 -- 172.21.15.27:0/938038432 shutdown_connections 2024-03-06T07:25:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.149+0000 7f3611e81640 1 -- 172.21.15.27:0/938038432 wait complete. 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.150+0000 7f3611e81640 1 Processor -- start 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.150+0000 7f3611e81640 1 -- start start 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f3611e81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c0b5840 0x7f360c1f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f3611e81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c15dbd0 0x7f360c14f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f3611e81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f360c15dfb0 0x7f360c1525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f3611e81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c0721c0 con 0x7f360c15dbd0 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f3611e81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c071ee0 con 0x7f360c15dfb0 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f3611e81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f360c0726e0 con 0x7f360c0b5840 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c15dbd0 0x7f360c14f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c15dbd0 0x7f360c14f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33416/0 (socket says 172.21.15.27:33416) 2024-03-06T07:25:10.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360affd640 1 -- 172.21.15.27:0/2381670682 learned_addr learned my addr 172.21.15.27:0/2381670682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360b7fe640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c0b5840 0x7f360c1f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360bfff640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f360c15dfb0 0x7f360c1525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360affd640 1 -- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c0b5840 msgr2=0x7f360c1f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360affd640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c0b5840 0x7f360c1f4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360affd640 1 -- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f360c15dfb0 msgr2=0x7f360c1525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360affd640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f360c15dfb0 0x7f360c1525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360affd640 1 -- 172.21.15.27:0/2381670682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c14fb00 con 0x7f360c15dbd0 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f360affd640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c15dbd0 0x7f360c14f5c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3600000c00 tx=0x7f36000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.151+0000 7f35f3fff640 1 -- 172.21.15.27:0/2381670682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600040070 con 0x7f360c15dbd0 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.152+0000 7f35f3fff640 1 -- 172.21.15.27:0/2381670682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600039c10 con 0x7f360c15dbd0 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.152+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360c150560 con 0x7f360c15dbd0 2024-03-06T07:25:10.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.152+0000 7f35f3fff640 1 -- 172.21.15.27:0/2381670682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600035040 con 0x7f360c15dbd0 2024-03-06T07:25:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.152+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360c150d20 con 0x7f360c15dbd0 2024-03-06T07:25:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.152+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f360c073380 con 0x7f360c15dbd0 2024-03-06T07:25:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.153+0000 7f35f3fff640 1 -- 172.21.15.27:0/2381670682 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3600038020 con 0x7f360c15dbd0 2024-03-06T07:25:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.153+0000 7f35f3fff640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35f806b4d0 0x7f35f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.153+0000 7f35f3fff640 1 -- 172.21.15.27:0/2381670682 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f36000abe30 con 0x7f360c15dbd0 2024-03-06T07:25:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.154+0000 7f360b7fe640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35f806b4d0 0x7f35f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.154+0000 7f360b7fe640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35f806b4d0 0x7f35f806d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f360c1581b0 tx=0x7f35fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.158+0000 7f35f3fff640 1 -- 172.21.15.27:0/2381670682 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f360007ac50 con 0x7f360c15dbd0 2024-03-06T07:25:10.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:10.377+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f360c04f410 con 0x7f360c15dbd0 2024-03-06T07:25:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.023+0000 7f35f3fff640 1 -- 172.21.15.27:0/2381670682 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7f360007a550 con 0x7f360c15dbd0 2024-03-06T07:25:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35f806b4d0 msgr2=0x7f35f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35f806b4d0 0x7f35f806d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f360c1581b0 tx=0x7f35fc008040 comp rx=0 tx=0).stop 2024-03-06T07:25:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c15dbd0 msgr2=0x7f360c14f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c15dbd0 0x7f360c14f5c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3600000c00 tx=0x7f36000046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 shutdown_connections 2024-03-06T07:25:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f35f806b4d0 0x7f35f806d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f360c15dfb0 0x7f360c1525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f360c15dbd0 0x7f360c14f5c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 --2- 172.21.15.27:0/2381670682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f360c0b5840 0x7f360c1f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 >> 172.21.15.27:0/2381670682 conn(0x7f360c0b26b0 msgr2=0x7f360c15e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 shutdown_connections 2024-03-06T07:25:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.025+0000 7f3611e81640 1 -- 172.21.15.27:0/2381670682 wait complete. 2024-03-06T07:25:11.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:11.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.196+0000 7f78d9a81640 1 Processor -- start 2024-03-06T07:25:11.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.196+0000 7f78d9a81640 1 -- start start 2024-03-06T07:25:11.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.197+0000 7f78d9a81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78d40ceea0 0x7f78d40d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.197+0000 7f78d9a81640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d40d3740 con 0x7f78d415dd90 2024-03-06T07:25:11.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.197+0000 7f78d9a81640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d40d3120 con 0x7f78d40b6d10 2024-03-06T07:25:11.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.197+0000 7f78d9a81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d40d3400 con 0x7f78d40ceea0 2024-03-06T07:25:11.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.197+0000 7f78d37fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f78d415dd90 0x7f78d40d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41984/0 (socket says 172.21.15.27:41984) 2024-03-06T07:25:11.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.197+0000 7f78d37fe640 1 -- 172.21.15.27:0/4013515370 learned_addr learned my addr 172.21.15.27:0/4013515370 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:11.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d2ffd640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78d40ceea0 0x7f78d40d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30985549 0 0) 0x7f78d40d3740 con 0x7f78d415dd90 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78bc003520 con 0x7f78d415dd90 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152574948 0 0) 0x7f78d40d3120 con 0x7f78d40b6d10 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d40d3740 con 0x7f78d40b6d10 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415889923 0 0) 0x7f78bc003520 con 0x7f78d415dd90 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78d40ceea0 msgr2=0x7f78d40d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78d40ceea0 0x7f78d40d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 >> v1:172.21.15.69:6789/0 conn(0x7f78d40b6d10 legacy=0x7f78d40d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d40d39e0 con 0x7f78d415dd90 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f78c40019f0 con 0x7f78d415dd90 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 >> v1:172.21.15.27:6789/0 conn(0x7f78d415dd90 legacy=0x7f78d40d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78bc003b70 0x7f78bc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78d40ceea0 0x7f78bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78bc004d40 0x7f78bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d40d3400 con 0x7f78bc003b70 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc00ba70 con 0x7f78bc004d40 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc00bc70 con 0x7f78d40ceea0 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d37fe640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78bc003b70 0x7f78bc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d2ffd640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78d40ceea0 0x7f78bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d3fff640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78bc004d40 0x7f78bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d37fe640 1 -- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78d40ceea0 msgr2=0x7f78bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d37fe640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78d40ceea0 0x7f78bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d37fe640 1 -- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78bc004d40 msgr2=0x7f78bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d37fe640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78bc004d40 0x7f78bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.198+0000 7f78d37fe640 1 -- 172.21.15.27:0/4013515370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc003520 con 0x7f78bc003b70 2024-03-06T07:25:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.199+0000 7f78d37fe640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78bc003b70 0x7f78bc003f70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f78c400a730 tx=0x7f78bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d04d5987cd8a82 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.199+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c400a350 con 0x7f78bc003b70 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.199+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c4034630 con 0x7f78bc003b70 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.199+0000 7f78d27fc640 1 -- 172.21.15.27:0/4013515370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4034a40 con 0x7f78bc003b70 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.200+0000 7f78d9a81640 1 -- 172.21.15.27:0/4013515370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78bc003b70 msgr2=0x7f78bc003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.200+0000 7f78d9a81640 1 --2- 172.21.15.27:0/4013515370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78bc003b70 0x7f78bc003f70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f78c400a730 tx=0x7f78bc005830 comp rx=0 tx=0).stop 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.200+0000 7f78d3fff640 1 -- 172.21.15.27:0/4013515370 reap_dead start 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.200+0000 7f78d9a81640 1 -- 172.21.15.27:0/4013515370 shutdown_connections 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.200+0000 7f78d9a81640 1 -- 172.21.15.27:0/4013515370 >> 172.21.15.27:0/4013515370 conn(0x7f78d4071960 msgr2=0x7f78d4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.200+0000 7f78d9a81640 1 -- 172.21.15.27:0/4013515370 shutdown_connections 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.200+0000 7f78d9a81640 1 -- 172.21.15.27:0/4013515370 wait complete. 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.201+0000 7f78d9a81640 1 Processor -- start 2024-03-06T07:25:11.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.201+0000 7f78d9a81640 1 -- start start 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d9a81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78bc003b70 0x7f78d40c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d9a81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78d40c7bf0 0x7f78d4156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d9a81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d41573a0 0x7f78d40b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d9a81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d40d48f0 con 0x7f78d41573a0 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d9a81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d415e170 con 0x7f78d40c7bf0 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d9a81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d40d4a70 con 0x7f78bc003b70 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d41573a0 0x7f78d40b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d41573a0 0x7f78d40b4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33434/0 (socket says 172.21.15.27:33434) 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d3fff640 1 -- 172.21.15.27:0/2660142580 learned_addr learned my addr 172.21.15.27:0/2660142580 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d37fe640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78bc003b70 0x7f78d40c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.202+0000 7f78d2ffd640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78d40c7bf0 0x7f78d4156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78d3fff640 1 -- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78bc003b70 msgr2=0x7f78d40c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78d3fff640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78bc003b70 0x7f78d40c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78d3fff640 1 -- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78d40c7bf0 msgr2=0x7f78d4156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78d3fff640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78d40c7bf0 0x7f78d4156e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78d3fff640 1 -- 172.21.15.27:0/2660142580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d40c8110 con 0x7f78d41573a0 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78d3fff640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d41573a0 0x7f78d40b4750 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f78d40b9df0 tx=0x7f78cc00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78bbfff640 1 -- 172.21.15.27:0/2660142580 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc019070 con 0x7f78d41573a0 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78bbfff640 1 -- 172.21.15.27:0/2660142580 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78cc005030 con 0x7f78d41573a0 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d40c83f0 con 0x7f78d41573a0 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78bbfff640 1 -- 172.21.15.27:0/2660142580 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc005420 con 0x7f78d41573a0 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.203+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d40c89b0 con 0x7f78d41573a0 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.204+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d4072a30 con 0x7f78d41573a0 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.204+0000 7f78bbfff640 1 -- 172.21.15.27:0/2660142580 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78cc0067d0 con 0x7f78d41573a0 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.205+0000 7f78bbfff640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78c006b500 0x7f78c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.205+0000 7f78bbfff640 1 -- 172.21.15.27:0/2660142580 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f78cc0890e0 con 0x7f78d41573a0 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.205+0000 7f78d37fe640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78c006b500 0x7f78c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.205+0000 7f78d37fe640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78c006b500 0x7f78c006d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f78c4034000 tx=0x7f78c402f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.209+0000 7f78bbfff640 1 -- 172.21.15.27:0/2660142580 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78cc057f00 con 0x7f78d41573a0 2024-03-06T07:25:11.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:11.389+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f78d415ca60 con 0x7f78d41573a0 2024-03-06T07:25:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.024+0000 7f78bbfff640 1 -- 172.21.15.27:0/2660142580 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v48) v1 ==== 181+0+0 (secure 0 0 0) 0x7f78cc057800 con 0x7f78d41573a0 2024-03-06T07:25:12.026 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78c006b500 msgr2=0x7f78c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78c006b500 0x7f78c006d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f78c4034000 tx=0x7f78c402f000 comp rx=0 tx=0).stop 2024-03-06T07:25:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d41573a0 msgr2=0x7f78d40b4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d41573a0 0x7f78d40b4750 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f78d40b9df0 tx=0x7f78cc00fc00 comp rx=0 tx=0).stop 2024-03-06T07:25:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 shutdown_connections 2024-03-06T07:25:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f78c006b500 0x7f78c006d9c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d41573a0 0x7f78d40b4750 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78d40c7bf0 0x7f78d4156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 --2- 172.21.15.27:0/2660142580 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78bc003b70 0x7f78d40c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.028+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 >> 172.21.15.27:0/2660142580 conn(0x7f78d4071960 msgr2=0x7f78d40d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.029+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 shutdown_connections 2024-03-06T07:25:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.030+0000 7f78d9a81640 1 -- 172.21.15.27:0/2660142580 wait complete. 2024-03-06T07:25:12.046 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:12.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.161+0000 7f0e24647640 1 Processor -- start 2024-03-06T07:25:12.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.162+0000 7f0e24647640 1 -- start start 2024-03-06T07:25:12.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.162+0000 7f0e24647640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e1c144640 0x7f0e1c06e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e24647640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c06e940 0x7f0e1c15b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e24647640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e1c15b670 0x7f0e1c0ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e24647640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1c06df70 con 0x7f0e1c15b670 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e24647640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1c15bc40 con 0x7f0e1c144640 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e24647640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c15bf20 con 0x7f0e1c06e940 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e22be4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e1c15b670 0x7f0e1c0ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e22be4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e1c15b670 0x7f0e1c0ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33438/0 (socket says 172.21.15.27:33438) 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e22be4640 1 -- 172.21.15.27:0/183265427 learned_addr learned my addr 172.21.15.27:0/183265427 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e223e3640 1 --2- 172.21.15.27:0/183265427 >> v2:172.21.15.69:3300/0 conn(0x7f0e1c144640 0x7f0e1c06e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e21be2640 1 --2- 172.21.15.27:0/183265427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c06e940 0x7f0e1c15b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e22be4640 1 -- 172.21.15.27:0/183265427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c06e940 msgr2=0x7f0e1c15b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e22be4640 1 --2- 172.21.15.27:0/183265427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c06e940 0x7f0e1c15b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e22be4640 1 -- 172.21.15.27:0/183265427 >> v2:172.21.15.69:3300/0 conn(0x7f0e1c144640 msgr2=0x7f0e1c06e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:12.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e22be4640 1 --2- 172.21.15.27:0/183265427 >> v2:172.21.15.69:3300/0 conn(0x7f0e1c144640 0x7f0e1c06e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e22be4640 1 -- 172.21.15.27:0/183265427 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c06dc00 con 0x7f0e1c15b670 2024-03-06T07:25:12.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.163+0000 7f0e223e3640 1 --2- 172.21.15.27:0/183265427 >> v2:172.21.15.69:3300/0 conn(0x7f0e1c144640 0x7f0e1c06e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:12.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.164+0000 7f0e22be4640 1 --2- 172.21.15.27:0/183265427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c15b670 0x7f0e1c0ce900 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c06d360 tx=0x7f0e1402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96c08b341b0d7b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.164+0000 7f0e213e1640 1 -- 172.21.15.27:0/183265427 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1402ea80 con 0x7f0e1c15b670 2024-03-06T07:25:12.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.164+0000 7f0e213e1640 1 -- 172.21.15.27:0/183265427 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1402ec20 con 0x7f0e1c15b670 2024-03-06T07:25:12.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.164+0000 7f0e213e1640 1 -- 172.21.15.27:0/183265427 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14009600 con 0x7f0e1c15b670 2024-03-06T07:25:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.165+0000 7f0e24647640 1 -- 172.21.15.27:0/183265427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c15b670 msgr2=0x7f0e1c0ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.165+0000 7f0e24647640 1 --2- 172.21.15.27:0/183265427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c15b670 0x7f0e1c0ce900 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c06d360 tx=0x7f0e1402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.165+0000 7f0e24647640 1 -- 172.21.15.27:0/183265427 shutdown_connections 2024-03-06T07:25:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.165+0000 7f0e24647640 1 --2- 172.21.15.27:0/183265427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c15b670 0x7f0e1c0ce900 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.165+0000 7f0e24647640 1 --2- 172.21.15.27:0/183265427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c06e940 0x7f0e1c15b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.165+0000 7f0e24647640 1 --2- 172.21.15.27:0/183265427 >> v2:172.21.15.69:3300/0 conn(0x7f0e1c144640 0x7f0e1c06e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.165+0000 7f0e24647640 1 -- 172.21.15.27:0/183265427 >> 172.21.15.27:0/183265427 conn(0x7f0e1c066d00 msgr2=0x7f0e1c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.165+0000 7f0e24647640 1 -- 172.21.15.27:0/183265427 shutdown_connections 2024-03-06T07:25:12.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.165+0000 7f0e24647640 1 -- 172.21.15.27:0/183265427 wait complete. 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.166+0000 7f0e24647640 1 Processor -- start 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.166+0000 7f0e24647640 1 -- start start 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e24647640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c06e940 0x7f0e1c1e46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e24647640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c144640 0x7f0e1c0af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e24647640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c15b670 0x7f0e1c15a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e24647640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0cefc0 con 0x7f0e1c06e940 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e24647640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0cee40 con 0x7f0e1c15b670 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e24647640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0cf140 con 0x7f0e1c144640 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e223e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c06e940 0x7f0e1c1e46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e223e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c06e940 0x7f0e1c1e46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33440/0 (socket says 172.21.15.27:33440) 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e223e3640 1 -- 172.21.15.27:0/938602464 learned_addr learned my addr 172.21.15.27:0/938602464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e21be2640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c144640 0x7f0e1c0af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e22be4640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c15b670 0x7f0e1c15a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e223e3640 1 -- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c144640 msgr2=0x7f0e1c0af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e223e3640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c144640 0x7f0e1c0af0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e223e3640 1 -- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c15b670 msgr2=0x7f0e1c15a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e223e3640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c15b670 0x7f0e1c15a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e223e3640 1 -- 172.21.15.27:0/938602464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0ae020 con 0x7f0e1c06e940 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e22be4640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c15b670 0x7f0e1c15a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e223e3640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c06e940 0x7f0e1c1e46f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800e730 tx=0x7f0e1800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e12ffd640 1 -- 172.21.15.27:0/938602464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1800f040 con 0x7f0e1c06e940 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e12ffd640 1 -- 172.21.15.27:0/938602464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e18004230 con 0x7f0e1c06e940 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.167+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1c0ae300 con 0x7f0e1c06e940 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.168+0000 7f0e12ffd640 1 -- 172.21.15.27:0/938602464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1800ee40 con 0x7f0e1c06e940 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.168+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1c0ae8c0 con 0x7f0e1c06e940 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.169+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1c0733c0 con 0x7f0e1c06e940 2024-03-06T07:25:12.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.169+0000 7f0e12ffd640 1 -- 172.21.15.27:0/938602464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e1800c6d0 con 0x7f0e1c06e940 2024-03-06T07:25:12.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.169+0000 7f0e12ffd640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0006b4d0 0x7f0e0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.169+0000 7f0e12ffd640 1 -- 172.21.15.27:0/938602464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f0e18089c80 con 0x7f0e1c06e940 2024-03-06T07:25:12.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.170+0000 7f0e21be2640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0006b4d0 0x7f0e0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.170+0000 7f0e21be2640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0006b4d0 0x7f0e0006d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c006fd0 tx=0x7f0e0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.174+0000 7f0e12ffd640 1 -- 172.21.15.27:0/938602464 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e18017020 con 0x7f0e1c06e940 2024-03-06T07:25:12.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:12.355+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0e1c143e60 con 0x7f0e1c06e940 2024-03-06T07:25:13.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.031+0000 7f0e12ffd640 1 -- 172.21.15.27:0/938602464 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v49) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0e18058cf0 con 0x7f0e1c06e940 2024-03-06T07:25:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0006b4d0 msgr2=0x7f0e0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0006b4d0 0x7f0e0006d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c006fd0 tx=0x7f0e0c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c06e940 msgr2=0x7f0e1c1e46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c06e940 0x7f0e1c1e46f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800e730 tx=0x7f0e1800ebf0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 shutdown_connections 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e0006b4d0 0x7f0e0006d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e1c15b670 0x7f0e1c15a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e1c144640 0x7f0e1c0af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 --2- 172.21.15.27:0/938602464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c06e940 0x7f0e1c1e46f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 >> 172.21.15.27:0/938602464 conn(0x7f0e1c066d00 msgr2=0x7f0e1c0b4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 shutdown_connections 2024-03-06T07:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.033+0000 7f0e24647640 1 -- 172.21.15.27:0/938602464 wait complete. 2024-03-06T07:25:13.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:13.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.162+0000 7fdb9eadf640 1 Processor -- start 2024-03-06T07:25:13.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.162+0000 7fdb9eadf640 1 -- start start 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.162+0000 7fdb9eadf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdb980b7420 0x7fdb980d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.162+0000 7fdb9eadf640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb980d3740 con 0x7fdb9815dd90 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.162+0000 7fdb9eadf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb980d3120 con 0x7fdb980ceea0 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.162+0000 7fdb9eadf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb980d3400 con 0x7fdb980b7420 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb9c87b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb9815dd90 0x7fdb980d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41996/0 (socket says 172.21.15.27:41996) 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb9c87b640 1 -- 172.21.15.27:0/3890499846 learned_addr learned my addr 172.21.15.27:0/3890499846 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb9d07c640 1 --2- 172.21.15.27:0/3890499846 >> v2:172.21.15.69:3300/0 conn(0x7fdb980b7420 0x7fdb980d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010769696 0 0) 0x7fdb980d3740 con 0x7fdb9815dd90 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb80003520 con 0x7fdb9815dd90 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3358411222 0 0) 0x7fdb80003520 con 0x7fdb9815dd90 2024-03-06T07:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 >> v2:172.21.15.69:3300/0 conn(0x7fdb980b7420 msgr2=0x7fdb980d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 --2- 172.21.15.27:0/3890499846 >> v2:172.21.15.69:3300/0 conn(0x7fdb980b7420 0x7fdb980d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 >> v1:172.21.15.69:6789/0 conn(0x7fdb980ceea0 legacy=0x7fdb980b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb980d39e0 con 0x7fdb9815dd90 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fdb94001690 con 0x7fdb9815dd90 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 >> v1:172.21.15.27:6789/0 conn(0x7fdb9815dd90 legacy=0x7fdb980d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb80003a00 0x7fdb80003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80004800 0x7fdb80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb800051c0 0x7fdb8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980d3400 con 0x7fdb80004800 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8000c0c0 con 0x7fdb800051c0 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8000c2c0 con 0x7fdb80003a00 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb8ffff640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80004800 0x7fdb80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb9d07c640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb800051c0 0x7fdb8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.163+0000 7fdb9c87b640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb80003a00 0x7fdb80003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.164+0000 7fdb8ffff640 1 -- 172.21.15.27:0/3890499846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb80003a00 msgr2=0x7fdb80003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.164+0000 7fdb8ffff640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb80003a00 0x7fdb80003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.164+0000 7fdb8ffff640 1 -- 172.21.15.27:0/3890499846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb800051c0 msgr2=0x7fdb8000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.164+0000 7fdb8ffff640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb800051c0 0x7fdb8000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.164+0000 7fdb8ffff640 1 -- 172.21.15.27:0/3890499846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb80003520 con 0x7fdb80004800 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.164+0000 7fdb9d07c640 1 -- 172.21.15.27:0/3890499846 reap_dead start 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.164+0000 7fdb8ffff640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80004800 0x7fdb80004c80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdb88001750 tx=0x7fdb80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8376ec59db8e655 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88001370 con 0x7fdb80004800 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb880052d0 con 0x7fdb80004800 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb8f7fe640 1 -- 172.21.15.27:0/3890499846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88003830 con 0x7fdb80004800 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb9eadf640 1 -- 172.21.15.27:0/3890499846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80004800 msgr2=0x7fdb80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb9eadf640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80004800 0x7fdb80004c80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdb88001750 tx=0x7fdb80005830 comp rx=0 tx=0).stop 2024-03-06T07:25:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb9eadf640 1 -- 172.21.15.27:0/3890499846 shutdown_connections 2024-03-06T07:25:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb9eadf640 1 --2- 172.21.15.27:0/3890499846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb80004800 0x7fdb80004c80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb9eadf640 1 -- 172.21.15.27:0/3890499846 >> 172.21.15.27:0/3890499846 conn(0x7fdb98071960 msgr2=0x7fdb98071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb9eadf640 1 -- 172.21.15.27:0/3890499846 shutdown_connections 2024-03-06T07:25:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.165+0000 7fdb9eadf640 1 -- 172.21.15.27:0/3890499846 wait complete. 2024-03-06T07:25:13.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.166+0000 7fdb9eadf640 1 Processor -- start 2024-03-06T07:25:13.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.166+0000 7fdb9eadf640 1 -- start start 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.167+0000 7fdb9eadf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb80004800 0x7fdb9815c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.167+0000 7fdb9eadf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9815c430 0x7fdb98160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.167+0000 7fdb9eadf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb981e7760 0x7fdb980ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.167+0000 7fdb9eadf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980d29d0 con 0x7fdb981e7760 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.167+0000 7fdb9eadf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9815e170 con 0x7fdb80004800 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.167+0000 7fdb9eadf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980d2b50 con 0x7fdb9815c430 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.167+0000 7fdb9d07c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb981e7760 0x7fdb980ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.167+0000 7fdb9d07c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb981e7760 0x7fdb980ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33444/0 (socket says 172.21.15.27:33444) 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.167+0000 7fdb9d07c640 1 -- 172.21.15.27:0/2941496804 learned_addr learned my addr 172.21.15.27:0/2941496804 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9c87b640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb80004800 0x7fdb9815c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb8ffff640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9815c430 0x7fdb98160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9d07c640 1 -- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9815c430 msgr2=0x7fdb98160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9d07c640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9815c430 0x7fdb98160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9d07c640 1 -- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb80004800 msgr2=0x7fdb9815c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9d07c640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb80004800 0x7fdb9815c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9d07c640 1 -- 172.21.15.27:0/2941496804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb980c9b00 con 0x7fdb981e7760 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9c87b640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb80004800 0x7fdb9815c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9d07c640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb981e7760 0x7fdb980ccd80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdb980b9df0 tx=0x7fdb9000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb8d7fa640 1 -- 172.21.15.27:0/2941496804 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9000fd80 con 0x7fdb981e7760 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb8d7fa640 1 -- 172.21.15.27:0/2941496804 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9001d070 con 0x7fdb981e7760 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb980c9de0 con 0x7fdb981e7760 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb8d7fa640 1 -- 172.21.15.27:0/2941496804 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb90015670 con 0x7fdb981e7760 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.168+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9815f3f0 con 0x7fdb981e7760 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.169+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb98072a30 con 0x7fdb981e7760 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.170+0000 7fdb8d7fa640 1 -- 172.21.15.27:0/2941496804 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb90015810 con 0x7fdb981e7760 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.170+0000 7fdb8d7fa640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb8406b500 0x7fdb8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.170+0000 7fdb8d7fa640 1 -- 172.21.15.27:0/2941496804 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdb9008a950 con 0x7fdb981e7760 2024-03-06T07:25:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.170+0000 7fdb9c87b640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb8406b500 0x7fdb8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.171+0000 7fdb9c87b640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb8406b500 0x7fdb8406d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdb94000c00 tx=0x7fdb94009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.174+0000 7fdb8d7fa640 1 -- 172.21.15.27:0/2941496804 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb90010040 con 0x7fdb981e7760 2024-03-06T07:25:13.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.360+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdb9804f410 con 0x7fdb981e7760 2024-03-06T07:25:13.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.363+0000 7fdb8d7fa640 1 -- 172.21.15.27:0/2941496804 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdb90059ad0 con 0x7fdb981e7760 2024-03-06T07:25:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.365+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb8406b500 msgr2=0x7fdb8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb8406b500 0x7fdb8406d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdb94000c00 tx=0x7fdb94009040 comp rx=0 tx=0).stop 2024-03-06T07:25:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb981e7760 msgr2=0x7fdb980ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb981e7760 0x7fdb980ccd80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdb980b9df0 tx=0x7fdb9000f580 comp rx=0 tx=0).stop 2024-03-06T07:25:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 shutdown_connections 2024-03-06T07:25:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb8406b500 0x7fdb8406d9c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb981e7760 0x7fdb980ccd80 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb9815c430 0x7fdb98160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 --2- 172.21.15.27:0/2941496804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb80004800 0x7fdb9815c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 >> 172.21.15.27:0/2941496804 conn(0x7fdb98071960 msgr2=0x7fdb980b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 shutdown_connections 2024-03-06T07:25:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.366+0000 7fdb9eadf640 1 -- 172.21.15.27:0/2941496804 wait complete. 2024-03-06T07:25:13.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:25:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.486+0000 7fe5abefb640 1 Processor -- start 2024-03-06T07:25:13.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.486+0000 7fe5abefb640 1 -- start start 2024-03-06T07:25:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5abefb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe5a415e2f0 0x7fe5a40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5abefb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe5a406c050 0x7fe5a40cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5abefb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a415e6d0 con 0x7fe5a406b4e0 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5abefb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a40d0a10 con 0x7fe5a415e2f0 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5abefb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a40ccd00 con 0x7fe5a406c050 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5a9c97640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe5a415e2f0 0x7fe5a40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5a9496640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe5a406b4e0 0x7fe5a406b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42010/0 (socket says 172.21.15.27:42010) 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5a9c97640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe5a415e2f0 0x7fe5a40b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33452/0 (socket says 172.21.15.27:33452) 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5a9496640 1 -- 172.21.15.27:0/1577146344 learned_addr learned my addr 172.21.15.27:0/1577146344 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5aa498640 1 --2- 172.21.15.27:0/1577146344 >> v2:172.21.15.69:3300/0 conn(0x7fe5a406c050 0x7fe5a40cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5a9c97640 1 -- 172.21.15.27:0/1577146344 >> v2:172.21.15.69:3300/0 conn(0x7fe5a406c050 msgr2=0x7fe5a40cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5a9c97640 1 --2- 172.21.15.27:0/1577146344 >> v2:172.21.15.69:3300/0 conn(0x7fe5a406c050 0x7fe5a40cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5a9c97640 1 -- 172.21.15.27:0/1577146344 >> v1:172.21.15.27:6789/0 conn(0x7fe5a406b4e0 legacy=0x7fe5a406b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5a9c97640 1 -- 172.21.15.27:0/1577146344 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40ccfe0 con 0x7fe5a415e2f0 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5a8c95640 1 -- 172.21.15.27:0/1577146344 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216233558 0 0) 0x7fe5a415e6d0 con 0x7fe5a406b4e0 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.487+0000 7fe5aa498640 1 --2- 172.21.15.27:0/1577146344 >> v2:172.21.15.69:3300/0 conn(0x7fe5a406c050 0x7fe5a40cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.488+0000 7fe5a9c97640 1 --2- 172.21.15.27:0/1577146344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 0x7fe5a40b41f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4072430 tx=0x7fe5a002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8241c5c74f3a51ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.488+0000 7fe5a8c95640 1 -- 172.21.15.27:0/1577146344 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0038070 con 0x7fe5a415e2f0 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.488+0000 7fe5a8c95640 1 -- 172.21.15.27:0/1577146344 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a0031710 con 0x7fe5a415e2f0 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.488+0000 7fe5a8c95640 1 -- 172.21.15.27:0/1577146344 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0031ae0 con 0x7fe5a415e2f0 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.488+0000 7fe5abefb640 1 -- 172.21.15.27:0/1577146344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 msgr2=0x7fe5a40b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.488+0000 7fe5abefb640 1 --2- 172.21.15.27:0/1577146344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 0x7fe5a40b41f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4072430 tx=0x7fe5a002b7b0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.489+0000 7fe5abefb640 1 -- 172.21.15.27:0/1577146344 shutdown_connections 2024-03-06T07:25:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.489+0000 7fe5abefb640 1 --2- 172.21.15.27:0/1577146344 >> v2:172.21.15.69:3300/0 conn(0x7fe5a406c050 0x7fe5a40cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.489+0000 7fe5abefb640 1 --2- 172.21.15.27:0/1577146344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 0x7fe5a40b41f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.489+0000 7fe5abefb640 1 -- 172.21.15.27:0/1577146344 >> 172.21.15.27:0/1577146344 conn(0x7fe5a40b36d0 msgr2=0x7fe5a40b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.489+0000 7fe5abefb640 1 -- 172.21.15.27:0/1577146344 shutdown_connections 2024-03-06T07:25:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.489+0000 7fe5abefb640 1 -- 172.21.15.27:0/1577146344 wait complete. 2024-03-06T07:25:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.490+0000 7fe5abefb640 1 Processor -- start 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.490+0000 7fe5abefb640 1 -- start start 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.490+0000 7fe5abefb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5a406b4e0 0x7fe5a415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.490+0000 7fe5abefb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a406c050 0x7fe5a40b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.490+0000 7fe5abefb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 0x7fe5a414bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.490+0000 7fe5abefb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40ce000 con 0x7fe5a415e2f0 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.490+0000 7fe5abefb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40cd5a0 con 0x7fe5a406c050 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.490+0000 7fe5abefb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40ce180 con 0x7fe5a406b4e0 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5aa498640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 0x7fe5a414bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5aa498640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 0x7fe5a414bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33458/0 (socket says 172.21.15.27:33458) 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5aa498640 1 -- 172.21.15.27:0/1827047936 learned_addr learned my addr 172.21.15.27:0/1827047936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5a9c97640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5a406b4e0 0x7fe5a415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5a9496640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a406c050 0x7fe5a40b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5aa498640 1 -- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5a406b4e0 msgr2=0x7fe5a415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5aa498640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5a406b4e0 0x7fe5a415eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5aa498640 1 -- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a406c050 msgr2=0x7fe5a40b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5aa498640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a406c050 0x7fe5a40b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5aa498640 1 -- 172.21.15.27:0/1827047936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40b62d0 con 0x7fe5a415e2f0 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5a9496640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a406c050 0x7fe5a40b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5aa498640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 0x7fe5a414bb10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00cf20 tx=0x7fe59c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe59a7fc640 1 -- 172.21.15.27:0/1827047936 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c01a070 con 0x7fe5a415e2f0 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe59a7fc640 1 -- 172.21.15.27:0/1827047936 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59c00d840 con 0x7fe5a415e2f0 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a40b65b0 con 0x7fe5a415e2f0 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe59a7fc640 1 -- 172.21.15.27:0/1827047936 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c00dc30 con 0x7fe5a415e2f0 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.491+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a40b6b70 con 0x7fe5a415e2f0 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.492+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a409fd30 con 0x7fe5a415e2f0 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.493+0000 7fe59a7fc640 1 -- 172.21.15.27:0/1827047936 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe59c01d070 con 0x7fe5a415e2f0 2024-03-06T07:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.493+0000 7fe59a7fc640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe58806b4d0 0x7fe58806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.493+0000 7fe59a7fc640 1 -- 172.21.15.27:0/1827047936 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe59c08a770 con 0x7fe5a415e2f0 2024-03-06T07:25:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.494+0000 7fe5a9c97640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe58806b4d0 0x7fe58806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.494+0000 7fe5a9c97640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe58806b4d0 0x7fe58806d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0035000 tx=0x7fe5a0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.498+0000 7fe59a7fc640 1 -- 172.21.15.27:0/1827047936 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe59c0592a0 con 0x7fe5a415e2f0 2024-03-06T07:25:13.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.684+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fe5a404f410 con 0x7fe5a415e2f0 2024-03-06T07:25:13.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.687+0000 7fe59a7fc640 1 -- 172.21.15.27:0/1827047936 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v15) v1 ==== 108+0+0 (secure 0 0 0) 0x7fe59c05d340 con 0x7fe5a415e2f0 2024-03-06T07:25:13.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe58806b4d0 msgr2=0x7fe58806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe58806b4d0 0x7fe58806d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0035000 tx=0x7fe5a0008030 comp rx=0 tx=0).stop 2024-03-06T07:25:13.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 msgr2=0x7fe5a414bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 0x7fe5a414bb10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00cf20 tx=0x7fe59c00a7d0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 shutdown_connections 2024-03-06T07:25:13.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe58806b4d0 0x7fe58806d990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5a415e2f0 0x7fe5a414bb10 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a406c050 0x7fe5a40b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 --2- 172.21.15.27:0/1827047936 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe5a406b4e0 0x7fe5a415eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 >> 172.21.15.27:0/1827047936 conn(0x7fe5a40b36d0 msgr2=0x7fe5a415af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.689+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 shutdown_connections 2024-03-06T07:25:13.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.690+0000 7fe5abefb640 1 -- 172.21.15.27:0/1827047936 wait complete. 2024-03-06T07:25:13.702 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:25:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.807+0000 7f21eb078640 1 Processor -- start 2024-03-06T07:25:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.807+0000 7f21eb078640 1 -- start start 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.808+0000 7f21eb078640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 0x7f21e40d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.808+0000 7f21eb078640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21e40d18c0 0x7f21e4071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.808+0000 7f21eb078640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e40b6f20 con 0x7f21e4071a80 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.808+0000 7f21eb078640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e40d3030 con 0x7f21e40d18c0 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.808+0000 7f21eb078640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e40d3310 con 0x7f21e415df70 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.808+0000 7f21e3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21e40d18c0 0x7f21e4071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.808+0000 7f21e3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21e40d18c0 0x7f21e4071540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37682/0 (socket says 172.21.15.27:37682) 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.808+0000 7f21e3fff640 1 -- 172.21.15.27:0/3364152176 learned_addr learned my addr 172.21.15.27:0/3364152176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.809+0000 7f21e3fff640 1 -- 172.21.15.27:0/3364152176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 msgr2=0x7f21e40d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.809+0000 7f21e8e14640 1 --2- 172.21.15.27:0/3364152176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 0x7f21e40d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.809+0000 7f21e3fff640 1 --2- 172.21.15.27:0/3364152176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 0x7f21e40d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.809+0000 7f21e3fff640 1 -- 172.21.15.27:0/3364152176 >> v1:172.21.15.69:6789/0 conn(0x7f21e4071a80 legacy=0x7f21e40b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.809+0000 7f21e3fff640 1 -- 172.21.15.27:0/3364152176 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e40d35f0 con 0x7f21e40d18c0 2024-03-06T07:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.809+0000 7f21e8e14640 1 --2- 172.21.15.27:0/3364152176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 0x7f21e40d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.809+0000 7f21e3fff640 1 --2- 172.21.15.27:0/3364152176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40d18c0 0x7f21e4071540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f21d8009870 tx=0x7f21d802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4827fdef543fb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21e37fe640 1 -- 172.21.15.27:0/3364152176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8034070 con 0x7f21e40d18c0 2024-03-06T07:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21e37fe640 1 -- 172.21.15.27:0/3364152176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d8030840 con 0x7f21e40d18c0 2024-03-06T07:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21e37fe640 1 -- 172.21.15.27:0/3364152176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8030c10 con 0x7f21e40d18c0 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21eb078640 1 -- 172.21.15.27:0/3364152176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40d18c0 msgr2=0x7f21e4071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21eb078640 1 --2- 172.21.15.27:0/3364152176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40d18c0 0x7f21e4071540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f21d8009870 tx=0x7f21d802b980 comp rx=0 tx=0).stop 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21eb078640 1 -- 172.21.15.27:0/3364152176 shutdown_connections 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21eb078640 1 --2- 172.21.15.27:0/3364152176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40d18c0 0x7f21e4071540 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21eb078640 1 --2- 172.21.15.27:0/3364152176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 0x7f21e40d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21eb078640 1 -- 172.21.15.27:0/3364152176 >> 172.21.15.27:0/3364152176 conn(0x7f21e40d3cf0 msgr2=0x7f21e40d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21eb078640 1 -- 172.21.15.27:0/3364152176 shutdown_connections 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.810+0000 7f21eb078640 1 -- 172.21.15.27:0/3364152176 wait complete. 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.811+0000 7f21eb078640 1 Processor -- start 2024-03-06T07:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.811+0000 7f21eb078640 1 -- start start 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.812+0000 7f21eb078640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e4071a80 0x7f21e40cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.812+0000 7f21eb078640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21e40d18c0 0x7f21e40ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.812+0000 7f21eb078640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 0x7f21e40ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.812+0000 7f21eb078640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e415e350 con 0x7f21e4071a80 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.812+0000 7f21eb078640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e40d2cf0 con 0x7f21e40d18c0 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.812+0000 7f21eb078640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e415e4d0 con 0x7f21e415df70 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.812+0000 7f21e8e14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e4071a80 0x7f21e40cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.812+0000 7f21e8e14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e4071a80 0x7f21e40cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37684/0 (socket says 172.21.15.27:37684) 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.812+0000 7f21e8e14640 1 -- 172.21.15.27:0/2895648675 learned_addr learned my addr 172.21.15.27:0/2895648675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e3fff640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21e40d18c0 0x7f21e40ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e9615640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 0x7f21e40ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e8e14640 1 -- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 msgr2=0x7f21e40ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e8e14640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 0x7f21e40ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e8e14640 1 -- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21e40d18c0 msgr2=0x7f21e40ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e8e14640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21e40d18c0 0x7f21e40ce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e8e14640 1 -- 172.21.15.27:0/2895648675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e40cbd20 con 0x7f21e4071a80 2024-03-06T07:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e8e14640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e4071a80 0x7f21e40cb7e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c7f0 tx=0x7f21d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e17fa640 1 -- 172.21.15.27:0/2895648675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4015070 con 0x7f21e4071a80 2024-03-06T07:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e17fa640 1 -- 172.21.15.27:0/2895648675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d4007820 con 0x7f21e4071a80 2024-03-06T07:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21e17fa640 1 -- 172.21.15.27:0/2895648675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4007bd0 con 0x7f21e4071a80 2024-03-06T07:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e40ccdd0 con 0x7f21e4071a80 2024-03-06T07:25:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.813+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e40cd390 con 0x7f21e4071a80 2024-03-06T07:25:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.814+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e40c8a50 con 0x7f21e4071a80 2024-03-06T07:25:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.814+0000 7f21e17fa640 1 -- 172.21.15.27:0/2895648675 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21d4018070 con 0x7f21e4071a80 2024-03-06T07:25:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.815+0000 7f21e17fa640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f21d006b4d0 0x7f21d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.815+0000 7f21e17fa640 1 -- 172.21.15.27:0/2895648675 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f21d408a030 con 0x7f21e4071a80 2024-03-06T07:25:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.815+0000 7f21e3fff640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f21d006b4d0 0x7f21d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.816+0000 7f21e3fff640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f21d006b4d0 0x7f21d006d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f21d8036000 tx=0x7f21d803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:13.819+0000 7f21e17fa640 1 -- 172.21.15.27:0/2895648675 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21d40591b0 con 0x7f21e4071a80 2024-03-06T07:25:14.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:14.074+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f21e4067ff0 con 0x7f21e4071a80 2024-03-06T07:25:15.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.036+0000 7f21e17fa640 1 -- 172.21.15.27:0/2895648675 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v157) v1 ==== 133+0+0 (secure 0 0 0) 0x7f21d4058ab0 con 0x7f21e4071a80 2024-03-06T07:25:15.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.038+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f21d006b4d0 msgr2=0x7f21d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.038+0000 7f21eb078640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f21d006b4d0 0x7f21d006d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f21d8036000 tx=0x7f21d803b040 comp rx=0 tx=0).stop 2024-03-06T07:25:15.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.038+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e4071a80 msgr2=0x7f21e40cb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.038+0000 7f21eb078640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e4071a80 0x7f21e40cb7e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c7f0 tx=0x7f21d400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.038+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 shutdown_connections 2024-03-06T07:25:15.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.038+0000 7f21eb078640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f21d006b4d0 0x7f21d006d990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.038+0000 7f21eb078640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21e415df70 0x7f21e40ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.038+0000 7f21eb078640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21e40d18c0 0x7f21e40ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.039+0000 7f21eb078640 1 --2- 172.21.15.27:0/2895648675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e4071a80 0x7f21e40cb7e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.039+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 >> 172.21.15.27:0/2895648675 conn(0x7f21e40d3cf0 msgr2=0x7f21e40b60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.039+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 shutdown_connections 2024-03-06T07:25:15.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.039+0000 7f21eb078640 1 -- 172.21.15.27:0/2895648675 wait complete. 2024-03-06T07:25:15.053 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:25:15.054 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:25:15.054 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:25:15.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.166+0000 7f63cb5d0640 1 Processor -- start 2024-03-06T07:25:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.166+0000 7f63cb5d0640 1 -- start start 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63cb5d0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63c406e700 0x7f63c415ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63cb5d0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63c40b55d0 0x7f63c4071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63cb5d0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40b4160 con 0x7f63c4072660 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63cb5d0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c40b5b40 con 0x7f63c40b55d0 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63cb5d0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c40b3ba0 con 0x7f63c406e700 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63c9b6d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63c40b55d0 0x7f63c4071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63c9b6d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63c40b55d0 0x7f63c4071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37688/0 (socket says 172.21.15.27:37688) 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63c9b6d640 1 -- 172.21.15.27:0/1870218881 learned_addr learned my addr 172.21.15.27:0/1870218881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63c936c640 1 --2- 172.21.15.27:0/1870218881 >> v2:172.21.15.69:3300/0 conn(0x7f63c406e700 0x7f63c415ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63c9b6d640 1 -- 172.21.15.27:0/1870218881 >> v2:172.21.15.69:3300/0 conn(0x7f63c406e700 msgr2=0x7f63c415ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63c9b6d640 1 --2- 172.21.15.27:0/1870218881 >> v2:172.21.15.69:3300/0 conn(0x7f63c406e700 0x7f63c415ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63c9b6d640 1 -- 172.21.15.27:0/1870218881 >> v1:172.21.15.69:6789/0 conn(0x7f63c4072660 legacy=0x7f63c406d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.167+0000 7f63c9b6d640 1 -- 172.21.15.27:0/1870218881 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40b3e80 con 0x7f63c40b55d0 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.168+0000 7f63c9b6d640 1 --2- 172.21.15.27:0/1870218881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 0x7f63c4071980 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f63c406bf60 tx=0x7f63bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5efb5a2613cae91 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.168+0000 7f63bbfff640 1 -- 172.21.15.27:0/1870218881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc02ca80 con 0x7f63c40b55d0 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.168+0000 7f63bbfff640 1 -- 172.21.15.27:0/1870218881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc02cc20 con 0x7f63c40b55d0 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.168+0000 7f63bbfff640 1 -- 172.21.15.27:0/1870218881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0343f0 con 0x7f63c40b55d0 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.169+0000 7f63cb5d0640 1 -- 172.21.15.27:0/1870218881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 msgr2=0x7f63c4071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.169+0000 7f63cb5d0640 1 --2- 172.21.15.27:0/1870218881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 0x7f63c4071980 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f63c406bf60 tx=0x7f63bc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.169+0000 7f63cb5d0640 1 -- 172.21.15.27:0/1870218881 shutdown_connections 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.169+0000 7f63cb5d0640 1 --2- 172.21.15.27:0/1870218881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 0x7f63c4071980 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.169+0000 7f63cb5d0640 1 --2- 172.21.15.27:0/1870218881 >> v2:172.21.15.69:3300/0 conn(0x7f63c406e700 0x7f63c415ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.169+0000 7f63cb5d0640 1 -- 172.21.15.27:0/1870218881 >> 172.21.15.27:0/1870218881 conn(0x7f63c40b2720 msgr2=0x7f63c40b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.169+0000 7f63cb5d0640 1 -- 172.21.15.27:0/1870218881 shutdown_connections 2024-03-06T07:25:15.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.169+0000 7f63cb5d0640 1 -- 172.21.15.27:0/1870218881 wait complete. 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.170+0000 7f63cb5d0640 1 Processor -- start 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.170+0000 7f63cb5d0640 1 -- start start 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63cb5d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c406e700 0x7f63c413e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63cb5d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c4072660 0x7f63c40cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63cb5d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 0x7f63c40cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63cb5d0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4072040 con 0x7f63c40b55d0 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63cb5d0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4071ec0 con 0x7f63c406e700 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63cb5d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40721c0 con 0x7f63c4072660 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c9b6d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 0x7f63c40cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c9b6d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 0x7f63c40cff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37700/0 (socket says 172.21.15.27:37700) 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c9b6d640 1 -- 172.21.15.27:0/274248983 learned_addr learned my addr 172.21.15.27:0/274248983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c936c640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c406e700 0x7f63c413e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c8b6b640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c4072660 0x7f63c40cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c9b6d640 1 -- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c4072660 msgr2=0x7f63c40cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c9b6d640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c4072660 0x7f63c40cfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c9b6d640 1 -- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c406e700 msgr2=0x7f63c413e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c9b6d640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c406e700 0x7f63c413e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.171+0000 7f63c9b6d640 1 -- 172.21.15.27:0/274248983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c41e7ad0 con 0x7f63c40b55d0 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.172+0000 7f63c9b6d640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 0x7f63c40cff90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f63bc036040 tx=0x7f63bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.172+0000 7f63b9ffb640 1 -- 172.21.15.27:0/274248983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc043020 con 0x7f63c40b55d0 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.172+0000 7f63b9ffb640 1 -- 172.21.15.27:0/274248983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc03e450 con 0x7f63c40b55d0 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.172+0000 7f63b9ffb640 1 -- 172.21.15.27:0/274248983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc009510 con 0x7f63c40b55d0 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.172+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c41e7db0 con 0x7f63c40b55d0 2024-03-06T07:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.172+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c41e8370 con 0x7f63c40b55d0 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.173+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4067ff0 con 0x7f63c40b55d0 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.174+0000 7f63b9ffb640 1 -- 172.21.15.27:0/274248983 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63bc046020 con 0x7f63c40b55d0 2024-03-06T07:25:15.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.174+0000 7f63b9ffb640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63a806b4d0 0x7f63a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.174+0000 7f63b9ffb640 1 -- 172.21.15.27:0/274248983 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f63bc0ac6b0 con 0x7f63c40b55d0 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.174+0000 7f63c936c640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63a806b4d0 0x7f63a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.175+0000 7f63c936c640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63a806b4d0 0x7f63a806d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63c0006fd0 tx=0x7f63c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.178+0000 7f63b9ffb640 1 -- 172.21.15.27:0/274248983 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63bc04d030 con 0x7f63c40b55d0 2024-03-06T07:25:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.362+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63c4157ea0 con 0x7f63c40b55d0 2024-03-06T07:25:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.362+0000 7f63b9ffb640 1 -- 172.21.15.27:0/274248983 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+12948 (secure 0 0 0) 0x7f63bc07b830 con 0x7f63c40b55d0 2024-03-06T07:25:15.364 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:15.364 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:13.025213+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/2008146374":"2024-03-07T07:25:10.020423+0000","172.21.15.27:6833/2008146374":"2024-03-07T07:25:10.020423+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.364+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63a806b4d0 msgr2=0x7f63a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.364+0000 7f63cb5d0640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63a806b4d0 0x7f63a806d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63c0006fd0 tx=0x7f63c0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.364+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 msgr2=0x7f63c40cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.364+0000 7f63cb5d0640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 0x7f63c40cff90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f63bc036040 tx=0x7f63bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.365+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 shutdown_connections 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.365+0000 7f63cb5d0640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63a806b4d0 0x7f63a806d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.365+0000 7f63cb5d0640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c40b55d0 0x7f63c40cff90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.365+0000 7f63cb5d0640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63c4072660 0x7f63c40cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.365+0000 7f63cb5d0640 1 --2- 172.21.15.27:0/274248983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63c406e700 0x7f63c413e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.365+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 >> 172.21.15.27:0/274248983 conn(0x7f63c40b2720 msgr2=0x7f63c40b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.365+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 shutdown_connections 2024-03-06T07:25:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.365+0000 7f63cb5d0640 1 -- 172.21.15.27:0/274248983 wait complete. 2024-03-06T07:25:15.377 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:15.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.486+0000 7f458544e640 1 Processor -- start 2024-03-06T07:25:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.486+0000 7f458544e640 1 -- start start 2024-03-06T07:25:15.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.486+0000 7f458544e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f458015d2c0 0x7f458015d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.486+0000 7f458544e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f458015fb60 0x7f458015ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.486+0000 7f458544e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45801604b0 con 0x7f45800b4340 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.486+0000 7f458544e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45800d0a30 con 0x7f458015fb60 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.486+0000 7f458544e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45800d01d0 con 0x7f458015d2c0 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.487+0000 7f457f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f458015fb60 0x7f458015ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.487+0000 7f457f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f458015fb60 0x7f458015ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37710/0 (socket says 172.21.15.27:37710) 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.487+0000 7f457f7fe640 1 -- 172.21.15.27:0/1021165970 learned_addr learned my addr 172.21.15.27:0/1021165970 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.487+0000 7f457effd640 1 --2- 172.21.15.27:0/1021165970 >> v2:172.21.15.69:3300/0 conn(0x7f458015d2c0 0x7f458015d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.487+0000 7f457f7fe640 1 -- 172.21.15.27:0/1021165970 >> v2:172.21.15.69:3300/0 conn(0x7f458015d2c0 msgr2=0x7f458015d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.487+0000 7f457f7fe640 1 --2- 172.21.15.27:0/1021165970 >> v2:172.21.15.69:3300/0 conn(0x7f458015d2c0 0x7f458015d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.487+0000 7f457f7fe640 1 -- 172.21.15.27:0/1021165970 >> v1:172.21.15.69:6789/0 conn(0x7f45800b4340 legacy=0x7f45800b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.487+0000 7f457f7fe640 1 -- 172.21.15.27:0/1021165970 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45800cc5c0 con 0x7f458015fb60 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f457f7fe640 1 --2- 172.21.15.27:0/1021165970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015fb60 0x7f458015ff70 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4580067ce0 tx=0x7f457802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06afe087eef2e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f457dffb640 1 -- 172.21.15.27:0/1021165970 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578035020 con 0x7f458015fb60 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f457dffb640 1 -- 172.21.15.27:0/1021165970 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4578031870 con 0x7f458015fb60 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f457dffb640 1 -- 172.21.15.27:0/1021165970 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578031c40 con 0x7f458015fb60 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f458544e640 1 -- 172.21.15.27:0/1021165970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015fb60 msgr2=0x7f458015ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f458544e640 1 --2- 172.21.15.27:0/1021165970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015fb60 0x7f458015ff70 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4580067ce0 tx=0x7f457802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f458544e640 1 -- 172.21.15.27:0/1021165970 shutdown_connections 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f458544e640 1 --2- 172.21.15.27:0/1021165970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015fb60 0x7f458015ff70 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f458544e640 1 --2- 172.21.15.27:0/1021165970 >> v2:172.21.15.69:3300/0 conn(0x7f458015d2c0 0x7f458015d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f458544e640 1 -- 172.21.15.27:0/1021165970 >> 172.21.15.27:0/1021165970 conn(0x7f45800b3660 msgr2=0x7f45800b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f458544e640 1 -- 172.21.15.27:0/1021165970 shutdown_connections 2024-03-06T07:25:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.488+0000 7f458544e640 1 -- 172.21.15.27:0/1021165970 wait complete. 2024-03-06T07:25:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.490+0000 7f458544e640 1 Processor -- start 2024-03-06T07:25:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.490+0000 7f458544e640 1 -- start start 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.490+0000 7f458544e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45800b4340 0x7f458014f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.490+0000 7f458544e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015d2c0 0x7f45800c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.490+0000 7f458544e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f458015fb60 0x7f4580150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.490+0000 7f458544e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45800cc860 con 0x7f458015d2c0 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.490+0000 7f458544e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45800d0750 con 0x7f45800b4340 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.490+0000 7f458544e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45800b31d0 con 0x7f458015fb60 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015d2c0 0x7f45800c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015d2c0 0x7f45800c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37724/0 (socket says 172.21.15.27:37724) 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457e7fc640 1 -- 172.21.15.27:0/473306292 learned_addr learned my addr 172.21.15.27:0/473306292 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:15.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457effd640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45800b4340 0x7f458014f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457f7fe640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f458015fb60 0x7f4580150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457e7fc640 1 -- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f458015fb60 msgr2=0x7f4580150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457e7fc640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f458015fb60 0x7f4580150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457e7fc640 1 -- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45800b4340 msgr2=0x7f458014f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457e7fc640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45800b4340 0x7f458014f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457e7fc640 1 -- 172.21.15.27:0/473306292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45800c83e0 con 0x7f458015d2c0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f457e7fc640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015d2c0 0x7f45800c78e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f457400b3e0 tx=0x7f457400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f45677fe640 1 -- 172.21.15.27:0/473306292 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574015020 con 0x7f458015d2c0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f45677fe640 1 -- 172.21.15.27:0/473306292 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574011850 con 0x7f458015d2c0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f45677fe640 1 -- 172.21.15.27:0/473306292 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574011c40 con 0x7f458015d2c0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45800c86c0 con 0x7f458015d2c0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.491+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4580159680 con 0x7f458015d2c0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.492+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458009fc50 con 0x7f458015d2c0 2024-03-06T07:25:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.493+0000 7f45677fe640 1 -- 172.21.15.27:0/473306292 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4574018020 con 0x7f458015d2c0 2024-03-06T07:25:15.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.493+0000 7f45677fe640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456c06b4d0 0x7f456c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.493+0000 7f45677fe640 1 -- 172.21.15.27:0/473306292 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4574089ec0 con 0x7f458015d2c0 2024-03-06T07:25:15.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.493+0000 7f457effd640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456c06b4d0 0x7f456c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.494+0000 7f457effd640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456c06b4d0 0x7f456c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4570008ef0 tx=0x7f4570008440 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.497+0000 7f45677fe640 1 -- 172.21.15.27:0/473306292 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f457401f020 con 0x7f458015d2c0 2024-03-06T07:25:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.681+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f458015ef00 con 0x7f458015d2c0 2024-03-06T07:25:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.681+0000 7f45677fe640 1 -- 172.21.15.27:0/473306292 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7f45740172c0 con 0x7f458015d2c0 2024-03-06T07:25:15.683 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:25:15.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456c06b4d0 msgr2=0x7f456c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456c06b4d0 0x7f456c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4570008ef0 tx=0x7f4570008440 comp rx=0 tx=0).stop 2024-03-06T07:25:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015d2c0 msgr2=0x7f45800c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015d2c0 0x7f45800c78e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f457400b3e0 tx=0x7f457400b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 shutdown_connections 2024-03-06T07:25:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456c06b4d0 0x7f456c06d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f458015fb60 0x7f4580150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458015d2c0 0x7f45800c78e0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 --2- 172.21.15.27:0/473306292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45800b4340 0x7f458014f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.683+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 >> 172.21.15.27:0/473306292 conn(0x7f45800b3660 msgr2=0x7f4580159d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.684+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 shutdown_connections 2024-03-06T07:25:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.684+0000 7f458544e640 1 -- 172.21.15.27:0/473306292 wait complete. 2024-03-06T07:25:15.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:25:15.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.803+0000 7f732100e640 1 Processor -- start 2024-03-06T07:25:15.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.803+0000 7f732100e640 1 -- start start 2024-03-06T07:25:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f732100e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f731c0aee10 0x7f731c15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f732100e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 0x7f731c070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f732100e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c0729b0 con 0x7f731c15bac0 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f732100e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731c06cf70 con 0x7f731c0aee10 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f732100e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c06c920 con 0x7f731c06d400 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f731ad9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f731c15bac0 0x7f731c0ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35904/0 (socket says 172.21.15.27:35904) 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f731ad9d640 1 -- 172.21.15.27:0/45051590 learned_addr learned my addr 172.21.15.27:0/45051590 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f731a59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f731c0aee10 0x7f731c15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f731b59e640 1 --2- 172.21.15.27:0/45051590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 0x7f731c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f731a59c640 1 -- 172.21.15.27:0/45051590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 msgr2=0x7f731c070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f731a59c640 1 --2- 172.21.15.27:0/45051590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 0x7f731c070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f731a59c640 1 -- 172.21.15.27:0/45051590 >> v1:172.21.15.27:6789/0 conn(0x7f731c15bac0 legacy=0x7f731c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f7319d9b640 1 -- 172.21.15.27:0/45051590 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982902580 0 0) 0x7f731c0729b0 con 0x7f731c15bac0 2024-03-06T07:25:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.804+0000 7f731a59c640 1 -- 172.21.15.27:0/45051590 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c06cc00 con 0x7f731c0aee10 2024-03-06T07:25:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.805+0000 7f731a59c640 1 --2- 172.21.15.27:0/45051590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c0aee10 0x7f731c15cbe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f73100097a0 tx=0x7f731002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449adbe3134a0596 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.805+0000 7f7319d9b640 1 -- 172.21.15.27:0/45051590 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310035020 con 0x7f731c0aee10 2024-03-06T07:25:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.805+0000 7f7319d9b640 1 -- 172.21.15.27:0/45051590 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7310031830 con 0x7f731c0aee10 2024-03-06T07:25:15.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.805+0000 7f7319d9b640 1 -- 172.21.15.27:0/45051590 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310031c00 con 0x7f731c0aee10 2024-03-06T07:25:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.806+0000 7f732100e640 1 -- 172.21.15.27:0/45051590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c0aee10 msgr2=0x7f731c15cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.806+0000 7f732100e640 1 --2- 172.21.15.27:0/45051590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c0aee10 0x7f731c15cbe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f73100097a0 tx=0x7f731002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.806+0000 7f732100e640 1 -- 172.21.15.27:0/45051590 shutdown_connections 2024-03-06T07:25:15.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.806+0000 7f732100e640 1 --2- 172.21.15.27:0/45051590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 0x7f731c070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.806+0000 7f732100e640 1 --2- 172.21.15.27:0/45051590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c0aee10 0x7f731c15cbe0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.806+0000 7f732100e640 1 -- 172.21.15.27:0/45051590 >> 172.21.15.27:0/45051590 conn(0x7f731c066d00 msgr2=0x7f731c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.806+0000 7f732100e640 1 -- 172.21.15.27:0/45051590 shutdown_connections 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.806+0000 7f732100e640 1 -- 172.21.15.27:0/45051590 wait complete. 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.807+0000 7f732100e640 1 Processor -- start 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.807+0000 7f732100e640 1 -- start start 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f732100e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 0x7f731c158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f732100e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f731c0aee10 0x7f731c142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f732100e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c15bac0 0x7f731c142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f732100e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c15c020 con 0x7f731c15bac0 2024-03-06T07:25:15.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f732100e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c15bea0 con 0x7f731c0aee10 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f732100e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c072580 con 0x7f731c06d400 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f731b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c15bac0 0x7f731c142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f731b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c15bac0 0x7f731c142f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37754/0 (socket says 172.21.15.27:37754) 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f731b59e640 1 -- 172.21.15.27:0/1430520240 learned_addr learned my addr 172.21.15.27:0/1430520240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f731ad9d640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 0x7f731c158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.808+0000 7f731a59c640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f731c0aee10 0x7f731c142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f731b59e640 1 -- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 msgr2=0x7f731c158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f731b59e640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 0x7f731c158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f731b59e640 1 -- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f731c0aee10 msgr2=0x7f731c142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f731b59e640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f731c0aee10 0x7f731c142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f731b59e640 1 -- 172.21.15.27:0/1430520240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c159380 con 0x7f731c15bac0 2024-03-06T07:25:15.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f731b59e640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c15bac0 0x7f731c142f40 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f730c007e60 tx=0x7f730c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f73037fe640 1 -- 172.21.15.27:0/1430520240 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c014420 con 0x7f731c15bac0 2024-03-06T07:25:15.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f73037fe640 1 -- 172.21.15.27:0/1430520240 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c019070 con 0x7f731c15bac0 2024-03-06T07:25:15.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c159660 con 0x7f731c15bac0 2024-03-06T07:25:15.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f73037fe640 1 -- 172.21.15.27:0/1430520240 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c014820 con 0x7f731c15bac0 2024-03-06T07:25:15.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.809+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c159c20 con 0x7f731c15bac0 2024-03-06T07:25:15.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.810+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c067ff0 con 0x7f731c15bac0 2024-03-06T07:25:15.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.811+0000 7f73037fe640 1 -- 172.21.15.27:0/1430520240 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f730c0149c0 con 0x7f731c15bac0 2024-03-06T07:25:15.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.811+0000 7f73037fe640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f730806b4d0 0x7f730806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:15.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.811+0000 7f73037fe640 1 -- 172.21.15.27:0/1430520240 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f730c08a580 con 0x7f731c15bac0 2024-03-06T07:25:15.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.811+0000 7f731ad9d640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f730806b4d0 0x7f730806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:15.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.812+0000 7f731ad9d640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f730806b4d0 0x7f730806d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7314000f60 tx=0x7f7314009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:15.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:15.815+0000 7f73037fe640 1 -- 172.21.15.27:0/1430520240 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f730c059140 con 0x7f731c15bac0 2024-03-06T07:25:16.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:16.065+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f731c06de50 con 0x7f731c15bac0 2024-03-06T07:25:17.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.044+0000 7f73037fe640 1 -- 172.21.15.27:0/1430520240 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v159) v1 ==== 147+0+0 (secure 0 0 0) 0x7f730c05d1e0 con 0x7f731c15bac0 2024-03-06T07:25:17.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f730806b4d0 msgr2=0x7f730806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f730806b4d0 0x7f730806d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7314000f60 tx=0x7f7314009040 comp rx=0 tx=0).stop 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c15bac0 msgr2=0x7f731c142f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c15bac0 0x7f731c142f40 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f730c007e60 tx=0x7f730c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 shutdown_connections 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f730806b4d0 0x7f730806d990 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f731c15bac0 0x7f731c142f40 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f731c0aee10 0x7f731c142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 --2- 172.21.15.27:0/1430520240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f731c06d400 0x7f731c158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 >> 172.21.15.27:0/1430520240 conn(0x7f731c066d00 msgr2=0x7f731c0b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 shutdown_connections 2024-03-06T07:25:17.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.047+0000 7f732100e640 1 -- 172.21.15.27:0/1430520240 wait complete. 2024-03-06T07:25:17.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:25:17.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.177+0000 7fcaf8008640 1 Processor -- start 2024-03-06T07:25:17.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.177+0000 7fcaf8008640 1 -- start start 2024-03-06T07:25:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.178+0000 7fcaf8008640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcaf00cdcd0 0x7fcaf00b1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.178+0000 7fcaf8008640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf0161a40 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.178+0000 7fcaf8008640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf00b2a20 con 0x7fcaf006b2c0 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.178+0000 7fcaf8008640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf00d11e0 con 0x7fcaf00cdcd0 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.178+0000 7fcaf55a3640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcaf00b2cd0 0x7fcaf00b1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35912/0 (socket says 172.21.15.27:35912) 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.178+0000 7fcaf55a3640 1 -- 172.21.15.27:0/3635986755 learned_addr learned my addr 172.21.15.27:0/3635986755 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.179+0000 7fcaf5da4640 1 --2- 172.21.15.27:0/3635986755 >> v2:172.21.15.69:3300/0 conn(0x7fcaf00cdcd0 0x7fcaf00b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.179+0000 7fcaf4da2640 1 -- 172.21.15.27:0/3635986755 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830547197 0 0) 0x7fcaf0161a40 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.179+0000 7fcaf4da2640 1 -- 172.21.15.27:0/3635986755 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcad8003520 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.179+0000 7fcaf5da4640 1 -- 172.21.15.27:0/3635986755 >> v1:172.21.15.69:6789/0 conn(0x7fcaf006b2c0 legacy=0x7fcaf006b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.179+0000 7fcaf4da2640 1 -- 172.21.15.27:0/3635986755 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3438315335 0 0) 0x7fcad8003520 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.179+0000 7fcaf5da4640 1 -- 172.21.15.27:0/3635986755 >> v1:172.21.15.27:6789/0 conn(0x7fcaf00b2cd0 legacy=0x7fcaf00b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.179+0000 7fcaf5da4640 1 -- 172.21.15.27:0/3635986755 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf00d0940 con 0x7fcaf00cdcd0 2024-03-06T07:25:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.180+0000 7fcaf5da4640 1 --2- 172.21.15.27:0/3635986755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 0x7fcaf00b1550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcae80097a0 tx=0x7fcae802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f46f0846795844be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.180+0000 7fcaf4da2640 1 -- 172.21.15.27:0/3635986755 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae8037070 con 0x7fcaf00cdcd0 2024-03-06T07:25:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.180+0000 7fcaf4da2640 1 -- 172.21.15.27:0/3635986755 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae8030510 con 0x7fcaf00cdcd0 2024-03-06T07:25:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.180+0000 7fcaf4da2640 1 -- 172.21.15.27:0/3635986755 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4001670 con 0x7fcaf00cdcd0 2024-03-06T07:25:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.180+0000 7fcaf8008640 1 -- 172.21.15.27:0/3635986755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 msgr2=0x7fcaf00b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.180+0000 7fcaf8008640 1 --2- 172.21.15.27:0/3635986755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 0x7fcaf00b1550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcae80097a0 tx=0x7fcae802b790 comp rx=0 tx=0).stop 2024-03-06T07:25:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.181+0000 7fcaf8008640 1 -- 172.21.15.27:0/3635986755 shutdown_connections 2024-03-06T07:25:17.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.181+0000 7fcaf8008640 1 --2- 172.21.15.27:0/3635986755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 0x7fcaf00b1550 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.181+0000 7fcaf8008640 1 -- 172.21.15.27:0/3635986755 >> 172.21.15.27:0/3635986755 conn(0x7fcaf00b3700 msgr2=0x7fcaf00b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.181+0000 7fcaf8008640 1 -- 172.21.15.27:0/3635986755 shutdown_connections 2024-03-06T07:25:17.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.181+0000 7fcaf8008640 1 -- 172.21.15.27:0/3635986755 wait complete. 2024-03-06T07:25:17.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.182+0000 7fcaf8008640 1 Processor -- start 2024-03-06T07:25:17.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.182+0000 7fcaf8008640 1 -- start start 2024-03-06T07:25:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.182+0000 7fcaf8008640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcaf006b2c0 0x7fcaf00c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf8008640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00b2cd0 0x7fcaf0155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf8008640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 0x7fcaf00af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf8008640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00cc1c0 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf8008640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00d0f00 con 0x7fcaf00cdcd0 2024-03-06T07:25:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf8008640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00cc340 con 0x7fcaf006b2c0 2024-03-06T07:25:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf55a3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00b2cd0 0x7fcaf0155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf55a3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00b2cd0 0x7fcaf0155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37756/0 (socket says 172.21.15.27:37756) 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf55a3640 1 -- 172.21.15.27:0/2570516016 learned_addr learned my addr 172.21.15.27:0/2570516016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf65a5640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 0x7fcaf00af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf5da4640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcaf006b2c0 0x7fcaf00c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf55a3640 1 -- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcaf006b2c0 msgr2=0x7fcaf00c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf55a3640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcaf006b2c0 0x7fcaf00c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf55a3640 1 -- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 msgr2=0x7fcaf00af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf55a3640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 0x7fcaf00af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf55a3640 1 -- 172.21.15.27:0/2570516016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf00c68c0 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf65a5640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 0x7fcaf00af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf55a3640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00b2cd0 0x7fcaf0155fc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcae40017e0 tx=0x7fcae400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaee7fc640 1 -- 172.21.15.27:0/2570516016 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4005ad0 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaee7fc640 1 -- 172.21.15.27:0/2570516016 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae4005c70 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaee7fc640 1 -- 172.21.15.27:0/2570516016 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4007290 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf00c6ba0 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.183+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf00c7160 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.184+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf009fcf0 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.185+0000 7fcaee7fc640 1 -- 172.21.15.27:0/2570516016 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcae4007430 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.185+0000 7fcaee7fc640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcadc06b4d0 0x7fcadc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.185+0000 7fcaee7fc640 1 -- 172.21.15.27:0/2570516016 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcae4089db0 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.185+0000 7fcaf5da4640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcadc06b4d0 0x7fcadc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.186+0000 7fcaf5da4640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcadc06b4d0 0x7fcadc06d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcae8034000 tx=0x7fcae802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.190+0000 7fcaee7fc640 1 -- 172.21.15.27:0/2570516016 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcae4058f30 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.376+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcaf00c7330 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.379+0000 7fcaee7fc640 1 -- 172.21.15.27:0/2570516016 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcae4058830 con 0x7fcaf00b2cd0 2024-03-06T07:25:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcadc06b4d0 msgr2=0x7fcadc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcadc06b4d0 0x7fcadc06d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcae8034000 tx=0x7fcae802f000 comp rx=0 tx=0).stop 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00b2cd0 msgr2=0x7fcaf0155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00b2cd0 0x7fcaf0155fc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcae40017e0 tx=0x7fcae400cbe0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 shutdown_connections 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcadc06b4d0 0x7fcadc06d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaf00cdcd0 0x7fcaf00af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00b2cd0 0x7fcaf0155fc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 --2- 172.21.15.27:0/2570516016 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcaf006b2c0 0x7fcaf00c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 >> 172.21.15.27:0/2570516016 conn(0x7fcaf00b3700 msgr2=0x7fcaf006b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 shutdown_connections 2024-03-06T07:25:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.381+0000 7fcaf8008640 1 -- 172.21.15.27:0/2570516016 wait complete. 2024-03-06T07:25:17.393 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:17.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:17.436 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:17.481 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:17.482 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:17.482 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:17.497 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:17.497 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:25:17.497 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:17.515 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:17.515 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:17.515 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:17.553 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:17.553 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:25:17.553 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:17.571 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:17.571 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:25:17.572 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:17.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.698+0000 7f2adfb1e640 1 Processor -- start 2024-03-06T07:25:17.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.698+0000 7f2adfb1e640 1 -- start start 2024-03-06T07:25:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2adfb1e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ad815e030 0x7f2ad806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2adfb1e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ad80b2460 0x7f2ad806d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2adfb1e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad815e410 con 0x7f2ad806ef70 2024-03-06T07:25:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2adfb1e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad8162700 con 0x7f2ad80b2460 2024-03-06T07:25:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2adfb1e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad8161af0 con 0x7f2ad815e030 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2ade0bb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ad80b2460 0x7f2ad806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2add0b9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2ad806ef70 0x7f2ad80b1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35918/0 (socket says 172.21.15.27:35918) 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2add0b9640 1 -- 172.21.15.27:0/2903229689 learned_addr learned my addr 172.21.15.27:0/2903229689 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2add8ba640 1 --2- 172.21.15.27:0/2903229689 >> v2:172.21.15.69:3300/0 conn(0x7f2ad815e030 0x7f2ad806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2ade0bb640 1 -- 172.21.15.27:0/2903229689 >> v2:172.21.15.69:3300/0 conn(0x7f2ad815e030 msgr2=0x7f2ad806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2ade0bb640 1 --2- 172.21.15.27:0/2903229689 >> v2:172.21.15.69:3300/0 conn(0x7f2ad815e030 0x7f2ad806ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2ade0bb640 1 -- 172.21.15.27:0/2903229689 >> v1:172.21.15.27:6789/0 conn(0x7f2ad806ef70 legacy=0x7f2ad80b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2adc8b8640 1 -- 172.21.15.27:0/2903229689 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749132053 0 0) 0x7f2acc001110 con 0x7f2ad806ef70 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2ade0bb640 1 -- 172.21.15.27:0/2903229689 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad8161dd0 con 0x7f2ad80b2460 2024-03-06T07:25:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.699+0000 7f2add8ba640 1 --2- 172.21.15.27:0/2903229689 >> v2:172.21.15.69:3300/0 conn(0x7f2ad815e030 0x7f2ad806ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.700+0000 7f2ade0bb640 1 --2- 172.21.15.27:0/2903229689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 0x7f2ad806d210 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8161a50 tx=0x7f2ad002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84442f4b2018406 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.700+0000 7f2adc8b8640 1 -- 172.21.15.27:0/2903229689 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0039070 con 0x7f2ad80b2460 2024-03-06T07:25:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.700+0000 7f2adc8b8640 1 -- 172.21.15.27:0/2903229689 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad002cc80 con 0x7f2ad80b2460 2024-03-06T07:25:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.700+0000 7f2adc8b8640 1 -- 172.21.15.27:0/2903229689 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0034660 con 0x7f2ad80b2460 2024-03-06T07:25:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.701+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2903229689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 msgr2=0x7f2ad806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.701+0000 7f2adfb1e640 1 --2- 172.21.15.27:0/2903229689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 0x7f2ad806d210 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8161a50 tx=0x7f2ad002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.701+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2903229689 shutdown_connections 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.701+0000 7f2adfb1e640 1 --2- 172.21.15.27:0/2903229689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 0x7f2ad806d210 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.701+0000 7f2adfb1e640 1 --2- 172.21.15.27:0/2903229689 >> v2:172.21.15.69:3300/0 conn(0x7f2ad815e030 0x7f2ad806ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.701+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2903229689 >> 172.21.15.27:0/2903229689 conn(0x7f2ad80b55e0 msgr2=0x7f2ad80b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.701+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2903229689 shutdown_connections 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.701+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2903229689 wait complete. 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.702+0000 7f2adfb1e640 1 Processor -- start 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.702+0000 7f2adfb1e640 1 -- start start 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2adfb1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad806ef70 0x7f2ad815f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2adfb1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 0x7f2ad8159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2adfb1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad815e030 0x7f2ad8158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2adfb1e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad80b31e0 con 0x7f2ad80b2460 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2adfb1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad8162390 con 0x7f2ad815e030 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2adfb1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad80b3360 con 0x7f2ad806ef70 2024-03-06T07:25:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add0b9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 0x7f2ad8159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add0b9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 0x7f2ad8159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37770/0 (socket says 172.21.15.27:37770) 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add0b9640 1 -- 172.21.15.27:0/2162468562 learned_addr learned my addr 172.21.15.27:0/2162468562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2ade0bb640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad815e030 0x7f2ad8158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add8ba640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad806ef70 0x7f2ad815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add0b9640 1 -- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad806ef70 msgr2=0x7f2ad815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add0b9640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad806ef70 0x7f2ad815f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add0b9640 1 -- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad815e030 msgr2=0x7f2ad8158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add0b9640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad815e030 0x7f2ad8158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add0b9640 1 -- 172.21.15.27:0/2162468562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad8159310 con 0x7f2ad80b2460 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2ade0bb640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad815e030 0x7f2ad8158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2add0b9640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 0x7f2ad8159a70 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c7b0 tx=0x7f2acc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2aca7fc640 1 -- 172.21.15.27:0/2162468562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc018070 con 0x7f2ad80b2460 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2aca7fc640 1 -- 172.21.15.27:0/2162468562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2acc0072d0 con 0x7f2ad80b2460 2024-03-06T07:25:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.703+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad809d690 con 0x7f2ad80b2460 2024-03-06T07:25:17.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.704+0000 7f2aca7fc640 1 -- 172.21.15.27:0/2162468562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0047e0 con 0x7f2ad80b2460 2024-03-06T07:25:17.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.704+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad809dc50 con 0x7f2ad80b2460 2024-03-06T07:25:17.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.705+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad8067ff0 con 0x7f2ad80b2460 2024-03-06T07:25:17.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.705+0000 7f2aca7fc640 1 -- 172.21.15.27:0/2162468562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2acc01b070 con 0x7f2ad80b2460 2024-03-06T07:25:17.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.705+0000 7f2aca7fc640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2abc06b4d0 0x7f2abc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:17.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.706+0000 7f2aca7fc640 1 -- 172.21.15.27:0/2162468562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2acc0891e0 con 0x7f2ad80b2460 2024-03-06T07:25:17.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.706+0000 7f2add8ba640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2abc06b4d0 0x7f2abc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:17.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.706+0000 7f2add8ba640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2abc06b4d0 0x7f2abc06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80b0770 tx=0x7f2ad4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:17.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.710+0000 7f2aca7fc640 1 -- 172.21.15.27:0/2162468562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ad815e410 con 0x7f2ad80b2460 2024-03-06T07:25:17.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.927+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ad806e430 con 0x7f2ad80b2460 2024-03-06T07:25:17.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.928+0000 7f2aca7fc640 1 -- 172.21.15.27:0/2162468562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f2acc058360 con 0x7f2ad80b2460 2024-03-06T07:25:17.929 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:17.930 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5126,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/2827284911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2827284911},{"type":"v1","addr":"172.21.15.69:6836","nonce":2827284911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5419,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1663054032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1663054032},{"type":"v1","addr":"172.21.15.69:6837","nonce":1663054032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2744957333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2744957333},{"type":"v1","addr":"172.21.15.27:6834","nonce":2744957333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":6033,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3879301994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3879301994},{"type":"v1","addr":"172.21.15.27:6835","nonce":3879301994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[]} 2024-03-06T07:25:17.930 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-03-06T07:25:17.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2abc06b4d0 msgr2=0x7f2abc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2abc06b4d0 0x7f2abc06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80b0770 tx=0x7f2ad4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:17.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 msgr2=0x7f2ad8159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:17.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 0x7f2ad8159a70 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c7b0 tx=0x7f2acc00cc70 comp rx=0 tx=0).stop 2024-03-06T07:25:17.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 shutdown_connections 2024-03-06T07:25:17.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2abc06b4d0 0x7f2abc06d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ad815e030 0x7f2ad8158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ad80b2460 0x7f2ad8159a70 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 --2- 172.21.15.27:0/2162468562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ad806ef70 0x7f2ad815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:17.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 >> 172.21.15.27:0/2162468562 conn(0x7f2ad80b55e0 msgr2=0x7f2ad80ad4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:17.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 shutdown_connections 2024-03-06T07:25:17.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:17.930+0000 7f2adfb1e640 1 -- 172.21.15.27:0/2162468562 wait complete. 2024-03-06T07:25:17.942 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:17.942 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:17.943 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:17.943 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:17.943 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:17.943 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:17.943 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:17.943 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:17.944 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:17.944 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:17.944 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:17.944 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:17.944 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:17.944 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:17.944 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:17.945 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:17.953 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:17.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:17.954 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:17.954 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:17.956 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:17.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:17.957 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:17.957 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:17.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:17.995 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:17.995 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:17.997 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:17.997 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:17.997 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:18.000 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:18.000 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:18.001 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:18.002 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:18.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:18.079 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:25:18.083 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:25:18.083 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:25:18.083 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:25:18.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.148+0000 7f2d28e50640 1 Processor -- start 2024-03-06T07:25:18.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.148+0000 7f2d28e50640 1 -- start start 2024-03-06T07:25:18.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.149+0000 7f2d28e50640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d2415df00 0x7f2d240d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.149+0000 7f2d28e50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d240b95a0 0x7f2d240b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.149+0000 7f2d28e50640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d240b9a30 0x7f2d240b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.149+0000 7f2d28e50640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d240cea30 con 0x7f2d2415df00 2024-03-06T07:25:18.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.149+0000 7f2d28e50640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d240d2af0 con 0x7f2d240b9a30 2024-03-06T07:25:18.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.149+0000 7f2d28e50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d240cf3c0 con 0x7f2d240b95a0 2024-03-06T07:25:18.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.149+0000 7f2d2259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d2415df00 0x7f2d240d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.149+0000 7f2d2259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d2415df00 0x7f2d240d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37804/0 (socket says 172.21.15.27:37804) 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.149+0000 7f2d2259c640 1 -- 172.21.15.27:0/2345745814 learned_addr learned my addr 172.21.15.27:0/2345745814 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.150+0000 7f2d21d9b640 1 --2- 172.21.15.27:0/2345745814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d240b95a0 0x7f2d240b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.150+0000 7f2d22d9d640 1 --2- 172.21.15.27:0/2345745814 >> v2:172.21.15.69:3300/0 conn(0x7f2d240b9a30 0x7f2d240b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.150+0000 7f2d2259c640 1 -- 172.21.15.27:0/2345745814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d240b95a0 msgr2=0x7f2d240b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.150+0000 7f2d2259c640 1 --2- 172.21.15.27:0/2345745814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d240b95a0 0x7f2d240b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.150+0000 7f2d2259c640 1 -- 172.21.15.27:0/2345745814 >> v2:172.21.15.69:3300/0 conn(0x7f2d240b9a30 msgr2=0x7f2d240b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.150+0000 7f2d2259c640 1 --2- 172.21.15.27:0/2345745814 >> v2:172.21.15.69:3300/0 conn(0x7f2d240b9a30 0x7f2d240b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.150+0000 7f2d2259c640 1 -- 172.21.15.27:0/2345745814 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d240cf0e0 con 0x7f2d2415df00 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.150+0000 7f2d22d9d640 1 --2- 172.21.15.27:0/2345745814 >> v2:172.21.15.69:3300/0 conn(0x7f2d240b9a30 0x7f2d240b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d2259c640 1 --2- 172.21.15.27:0/2345745814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d2415df00 0x7f2d240d25b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2d240d1160 tx=0x7f2d1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e989ab5ef6e715 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d2159a640 1 -- 172.21.15.27:0/2345745814 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14035020 con 0x7f2d2415df00 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d2159a640 1 -- 172.21.15.27:0/2345745814 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d14031870 con 0x7f2d2415df00 2024-03-06T07:25:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d2159a640 1 -- 172.21.15.27:0/2345745814 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14031c40 con 0x7f2d2415df00 2024-03-06T07:25:18.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d28e50640 1 -- 172.21.15.27:0/2345745814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d2415df00 msgr2=0x7f2d240d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d28e50640 1 --2- 172.21.15.27:0/2345745814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d2415df00 0x7f2d240d25b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2d240d1160 tx=0x7f2d1402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:18.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d28e50640 1 -- 172.21.15.27:0/2345745814 shutdown_connections 2024-03-06T07:25:18.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d28e50640 1 --2- 172.21.15.27:0/2345745814 >> v2:172.21.15.69:3300/0 conn(0x7f2d240b9a30 0x7f2d240b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d28e50640 1 --2- 172.21.15.27:0/2345745814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d240b95a0 0x7f2d240b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d28e50640 1 --2- 172.21.15.27:0/2345745814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d2415df00 0x7f2d240d25b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d28e50640 1 -- 172.21.15.27:0/2345745814 >> 172.21.15.27:0/2345745814 conn(0x7f2d24071960 msgr2=0x7f2d24071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d28e50640 1 -- 172.21.15.27:0/2345745814 shutdown_connections 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.151+0000 7f2d28e50640 1 -- 172.21.15.27:0/2345745814 wait complete. 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.153+0000 7f2d28e50640 1 Processor -- start 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.153+0000 7f2d28e50640 1 -- start start 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.153+0000 7f2d28e50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d240b95a0 0x7f2d24151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.153+0000 7f2d28e50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2415df00 0x7f2d2415fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.153+0000 7f2d28e50640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d241522b0 0x7f2d240ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.153+0000 7f2d28e50640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d240cec70 con 0x7f2d241522b0 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.153+0000 7f2d28e50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d240ce750 con 0x7f2d240b95a0 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.153+0000 7f2d28e50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d240cedf0 con 0x7f2d2415df00 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d22d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d241522b0 0x7f2d240ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d22d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d241522b0 0x7f2d240ccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37806/0 (socket says 172.21.15.27:37806) 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d22d9d640 1 -- 172.21.15.27:0/565780215 learned_addr learned my addr 172.21.15.27:0/565780215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d2259c640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d240b95a0 0x7f2d24151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d21d9b640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2415df00 0x7f2d2415fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d22d9d640 1 -- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2415df00 msgr2=0x7f2d2415fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d22d9d640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2415df00 0x7f2d2415fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d22d9d640 1 -- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d240b95a0 msgr2=0x7f2d24151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d22d9d640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d240b95a0 0x7f2d24151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d22d9d640 1 -- 172.21.15.27:0/565780215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d24160860 con 0x7f2d241522b0 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d2259c640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d240b95a0 0x7f2d24151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d22d9d640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d241522b0 0x7f2d240ccb20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00ced0 tx=0x7f2d1c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d1affd640 1 -- 172.21.15.27:0/565780215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c003b90 con 0x7f2d241522b0 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d1affd640 1 -- 172.21.15.27:0/565780215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c003d50 con 0x7f2d241522b0 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d240c7930 con 0x7f2d241522b0 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d1affd640 1 -- 172.21.15.27:0/565780215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c004260 con 0x7f2d241522b0 2024-03-06T07:25:18.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.154+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d240b2940 con 0x7f2d241522b0 2024-03-06T07:25:18.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.155+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d24067ff0 con 0x7f2d241522b0 2024-03-06T07:25:18.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.156+0000 7f2d1affd640 1 -- 172.21.15.27:0/565780215 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d1c004400 con 0x7f2d241522b0 2024-03-06T07:25:18.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.156+0000 7f2d1affd640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2d0406b4d0 0x7f2d0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.156+0000 7f2d1affd640 1 -- 172.21.15.27:0/565780215 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2d1c08a820 con 0x7f2d241522b0 2024-03-06T07:25:18.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.157+0000 7f2d2259c640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2d0406b4d0 0x7f2d0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.157+0000 7f2d2259c640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2d0406b4d0 0x7f2d0406d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2d14036000 tx=0x7f2d1403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.161+0000 7f2d1affd640 1 -- 172.21.15.27:0/565780215 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d1c0599a0 con 0x7f2d241522b0 2024-03-06T07:25:18.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:18.353+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2d240b6e40 con 0x7f2d241522b0 2024-03-06T07:25:19.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.393+0000 7f2d1affd640 1 -- 172.21.15.27:0/565780215 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v54) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2d1c0592a0 con 0x7f2d241522b0 2024-03-06T07:25:19.395 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-03-06T07:25:19.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.395+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2d0406b4d0 msgr2=0x7f2d0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.395+0000 7f2d28e50640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2d0406b4d0 0x7f2d0406d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2d14036000 tx=0x7f2d1403c040 comp rx=0 tx=0).stop 2024-03-06T07:25:19.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.395+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d241522b0 msgr2=0x7f2d240ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.395+0000 7f2d28e50640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d241522b0 0x7f2d240ccb20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00ced0 tx=0x7f2d1c00a710 comp rx=0 tx=0).stop 2024-03-06T07:25:19.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.396+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 shutdown_connections 2024-03-06T07:25:19.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.396+0000 7f2d28e50640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2d0406b4d0 0x7f2d0406d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.396+0000 7f2d28e50640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d241522b0 0x7f2d240ccb20 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.396+0000 7f2d28e50640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2d2415df00 0x7f2d2415fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.396+0000 7f2d28e50640 1 --2- 172.21.15.27:0/565780215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d240b95a0 0x7f2d24151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.396+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 >> 172.21.15.27:0/565780215 conn(0x7f2d24071960 msgr2=0x7f2d240b8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.396+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 shutdown_connections 2024-03-06T07:25:19.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.396+0000 7f2d28e50640 1 -- 172.21.15.27:0/565780215 wait complete. 2024-03-06T07:25:19.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:19.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.565+0000 7f9bde1bc640 1 Processor -- start 2024-03-06T07:25:19.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.565+0000 7f9bde1bc640 1 -- start start 2024-03-06T07:25:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.566+0000 7f9bde1bc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9bd80cea70 0x7f9bd80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.566+0000 7f9bde1bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80b0ed0 0x7f9bd806f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.566+0000 7f9bde1bc640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd80cd350 con 0x7f9bd815bda0 2024-03-06T07:25:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.566+0000 7f9bde1bc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd80b1360 con 0x7f9bd80cea70 2024-03-06T07:25:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.566+0000 7f9bde1bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd806eb20 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.566+0000 7f9bd77fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9bd815bda0 0x7f9bd80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35948/0 (socket says 172.21.15.27:35948) 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.566+0000 7f9bd77fe640 1 -- 172.21.15.27:0/2585360516 learned_addr learned my addr 172.21.15.27:0/2585360516 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.566+0000 7f9bd6ffd640 1 --2- 172.21.15.27:0/2585360516 >> v2:172.21.15.69:3300/0 conn(0x7f9bd80cea70 0x7f9bd80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.566+0000 7f9bd7fff640 1 --2- 172.21.15.27:0/2585360516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80b0ed0 0x7f9bd806f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.567+0000 7f9bd67fc640 1 -- 172.21.15.27:0/2585360516 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266181400 0 0) 0x7f9bd80cd350 con 0x7f9bd815bda0 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.567+0000 7f9bd67fc640 1 -- 172.21.15.27:0/2585360516 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc0003520 con 0x7f9bd815bda0 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.567+0000 7f9bd6ffd640 1 -- 172.21.15.27:0/2585360516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80b0ed0 msgr2=0x7f9bd806f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.567+0000 7f9bd6ffd640 1 --2- 172.21.15.27:0/2585360516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80b0ed0 0x7f9bd806f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.567+0000 7f9bd6ffd640 1 -- 172.21.15.27:0/2585360516 >> v1:172.21.15.27:6789/0 conn(0x7f9bd815bda0 legacy=0x7f9bd80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.567+0000 7f9bd6ffd640 1 -- 172.21.15.27:0/2585360516 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd80cd630 con 0x7f9bd80cea70 2024-03-06T07:25:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.567+0000 7f9bd7fff640 1 --2- 172.21.15.27:0/2585360516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80b0ed0 0x7f9bd806f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.568+0000 7f9bd6ffd640 1 --2- 172.21.15.27:0/2585360516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd80cea70 0x7f9bd80d25b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0097a0 tx=0x7f9bcc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64350ce79cf1e073 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.568+0000 7f9bd67fc640 1 -- 172.21.15.27:0/2585360516 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc035020 con 0x7f9bd80cea70 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.568+0000 7f9bd67fc640 1 -- 172.21.15.27:0/2585360516 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc031830 con 0x7f9bd80cea70 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.568+0000 7f9bd67fc640 1 -- 172.21.15.27:0/2585360516 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc031be0 con 0x7f9bd80cea70 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.568+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2585360516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd80cea70 msgr2=0x7f9bd80d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.568+0000 7f9bde1bc640 1 --2- 172.21.15.27:0/2585360516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd80cea70 0x7f9bd80d25b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0097a0 tx=0x7f9bcc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.569+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2585360516 shutdown_connections 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.569+0000 7f9bde1bc640 1 --2- 172.21.15.27:0/2585360516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80b0ed0 0x7f9bd806f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.569+0000 7f9bde1bc640 1 --2- 172.21.15.27:0/2585360516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd80cea70 0x7f9bd80d25b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.569+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2585360516 >> 172.21.15.27:0/2585360516 conn(0x7f9bd80d1a90 msgr2=0x7f9bd80d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.569+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2585360516 shutdown_connections 2024-03-06T07:25:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.569+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2585360516 wait complete. 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.570+0000 7f9bde1bc640 1 Processor -- start 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.570+0000 7f9bde1bc640 1 -- start start 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bde1bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd80b0ed0 0x7f9bd814be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bde1bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80cea70 0x7f9bd80c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bde1bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd815bda0 0x7f9bd80c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bde1bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd815c180 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bde1bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd806ed70 con 0x7f9bd815bda0 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bde1bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd815c300 con 0x7f9bd80cea70 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd80b0ed0 0x7f9bd814be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd80b0ed0 0x7f9bd814be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37808/0 (socket says 172.21.15.27:37808) 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd77fe640 1 -- 172.21.15.27:0/2467169724 learned_addr learned my addr 172.21.15.27:0/2467169724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd6ffd640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80cea70 0x7f9bd80c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd7fff640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd815bda0 0x7f9bd80c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd77fe640 1 -- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80cea70 msgr2=0x7f9bd80c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd77fe640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80cea70 0x7f9bd80c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd77fe640 1 -- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd815bda0 msgr2=0x7f9bd80c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd77fe640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd815bda0 0x7f9bd80c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd77fe640 1 -- 172.21.15.27:0/2467169724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd815de50 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd7fff640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd815bda0 0x7f9bd80c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bd77fe640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd80b0ed0 0x7f9bd814be60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800c430 tx=0x7f9bc800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.571+0000 7f9bbffff640 1 -- 172.21.15.27:0/2467169724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8017070 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.572+0000 7f9bbffff640 1 -- 172.21.15.27:0/2467169724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc8005ce0 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.572+0000 7f9bbffff640 1 -- 172.21.15.27:0/2467169724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc80072f0 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.572+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd815d2d0 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.572+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd815d890 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.573+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd8074980 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.573+0000 7f9bbffff640 1 -- 172.21.15.27:0/2467169724 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bc801a070 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.573+0000 7f9bbffff640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bc406b4d0 0x7f9bc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.574+0000 7f9bbffff640 1 -- 172.21.15.27:0/2467169724 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9bc8089390 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.574+0000 7f9bd6ffd640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bc406b4d0 0x7f9bc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.574+0000 7f9bd6ffd640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bc406b4d0 0x7f9bc406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc036000 tx=0x7f9bcc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.578+0000 7f9bbffff640 1 -- 172.21.15.27:0/2467169724 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bd8074980 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.827+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9bd8067ff0 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.827+0000 7f9bbffff640 1 -- 172.21.15.27:0/2467169724 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9bc8058560 con 0x7f9bd80b0ed0 2024-03-06T07:25:19.830 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:19.830 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:25:19.830 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-03-06T07:25:19.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.830 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-03-06T07:25:19.830 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g==", 2024-03-06T07:25:19.830 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.830 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ==", 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:25:19.831 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg==", 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ==", 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw==", 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.833 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA==", 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA==", 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg==", 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.835 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A==", 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw==", 2024-03-06T07:25:19.836 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA==", 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.837 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A==", 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA==", 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ==", 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ==", 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:25:19.839 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg==", 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:25:19.840 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlkc2iEhAA6BuYba0FzXeVjGPOP8sgyQ==", 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-03-06T07:25:19.841 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw==", 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg==", 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.842 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:19.843 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:25:19.843 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:25:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bc406b4d0 msgr2=0x7f9bc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bc406b4d0 0x7f9bc406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc036000 tx=0x7f9bcc008030 comp rx=0 tx=0).stop 2024-03-06T07:25:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd80b0ed0 msgr2=0x7f9bd814be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd80b0ed0 0x7f9bd814be60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800c430 tx=0x7f9bc800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 shutdown_connections 2024-03-06T07:25:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9bc406b4d0 0x7f9bc406d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9bd815bda0 0x7f9bd80c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9bd80cea70 0x7f9bd80c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 --2- 172.21.15.27:0/2467169724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bd80b0ed0 0x7f9bd814be60 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 >> 172.21.15.27:0/2467169724 conn(0x7f9bd80d1a90 msgr2=0x7f9bd80b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 shutdown_connections 2024-03-06T07:25:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.829+0000 7f9bde1bc640 1 -- 172.21.15.27:0/2467169724 wait complete. 2024-03-06T07:25:19.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:19.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.950+0000 7f74b8aa4640 1 Processor -- start 2024-03-06T07:25:19.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.950+0000 7f74b8aa4640 1 -- start start 2024-03-06T07:25:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b8aa4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 0x7f74b4071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b8aa4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74b4071a80 0x7f74b40b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b8aa4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40b6f20 con 0x7f74b415df70 2024-03-06T07:25:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b8aa4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b40d3030 con 0x7f74b4071a80 2024-03-06T07:25:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b8aa4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40d3310 con 0x7f74b40d18c0 2024-03-06T07:25:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b2d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74b4071a80 0x7f74b40b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b2d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74b4071a80 0x7f74b40b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37810/0 (socket says 172.21.15.27:37810) 2024-03-06T07:25:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b2d9d640 1 -- 172.21.15.27:0/996190862 learned_addr learned my addr 172.21.15.27:0/996190862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b1d9b640 1 --2- 172.21.15.27:0/996190862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 0x7f74b4071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b2d9d640 1 -- 172.21.15.27:0/996190862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 msgr2=0x7f74b4071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b2d9d640 1 --2- 172.21.15.27:0/996190862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 0x7f74b4071540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b2d9d640 1 -- 172.21.15.27:0/996190862 >> v1:172.21.15.69:6789/0 conn(0x7f74b415df70 legacy=0x7f74b40d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.951+0000 7f74b2d9d640 1 -- 172.21.15.27:0/996190862 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b40d35f0 con 0x7f74b4071a80 2024-03-06T07:25:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.952+0000 7f74b2d9d640 1 --2- 172.21.15.27:0/996190862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b4071a80 0x7f74b40b2700 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009e50 tx=0x7f74ac02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186e370776051961 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.952+0000 7f74b159a640 1 -- 172.21.15.27:0/996190862 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac02cb90 con 0x7f74b4071a80 2024-03-06T07:25:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.952+0000 7f74b159a640 1 -- 172.21.15.27:0/996190862 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac02cd30 con 0x7f74b4071a80 2024-03-06T07:25:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.952+0000 7f74b159a640 1 -- 172.21.15.27:0/996190862 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac034610 con 0x7f74b4071a80 2024-03-06T07:25:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.953+0000 7f74b8aa4640 1 -- 172.21.15.27:0/996190862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b4071a80 msgr2=0x7f74b40b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.953+0000 7f74b8aa4640 1 --2- 172.21.15.27:0/996190862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b4071a80 0x7f74b40b2700 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009e50 tx=0x7f74ac02bff0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.953+0000 7f74b8aa4640 1 -- 172.21.15.27:0/996190862 shutdown_connections 2024-03-06T07:25:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.953+0000 7f74b8aa4640 1 --2- 172.21.15.27:0/996190862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b4071a80 0x7f74b40b2700 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.953+0000 7f74b8aa4640 1 --2- 172.21.15.27:0/996190862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 0x7f74b4071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.953+0000 7f74b8aa4640 1 -- 172.21.15.27:0/996190862 >> 172.21.15.27:0/996190862 conn(0x7f74b40d3cf0 msgr2=0x7f74b40d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.953+0000 7f74b8aa4640 1 -- 172.21.15.27:0/996190862 shutdown_connections 2024-03-06T07:25:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.953+0000 7f74b8aa4640 1 -- 172.21.15.27:0/996190862 wait complete. 2024-03-06T07:25:19.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.954+0000 7f74b8aa4640 1 Processor -- start 2024-03-06T07:25:19.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.954+0000 7f74b8aa4640 1 -- start start 2024-03-06T07:25:19.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b8aa4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74b4071a80 0x7f74b40cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b8aa4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 0x7f74b415eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b8aa4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b415df70 0x7f74b415f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b8aa4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b415e4d0 con 0x7f74b415df70 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b8aa4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b415e350 con 0x7f74b4071a80 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b8aa4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40b2c40 con 0x7f74b40d18c0 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b415df70 0x7f74b415f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b415df70 0x7f74b415f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37824/0 (socket says 172.21.15.27:37824) 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b2d9d640 1 -- 172.21.15.27:0/1584944377 learned_addr learned my addr 172.21.15.27:0/1584944377 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b1d9b640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 0x7f74b415eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b259c640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74b4071a80 0x7f74b40cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b2d9d640 1 -- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 msgr2=0x7f74b415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b2d9d640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 0x7f74b415eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b2d9d640 1 -- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74b4071a80 msgr2=0x7f74b40cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b2d9d640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74b4071a80 0x7f74b40cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b2d9d640 1 -- 172.21.15.27:0/1584944377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b40cd240 con 0x7f74b415df70 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.955+0000 7f74b259c640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74b4071a80 0x7f74b40cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.956+0000 7f74b2d9d640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b415df70 0x7f74b415f420 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f74ac036040 tx=0x7f74ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.956+0000 7f74aaffd640 1 -- 172.21.15.27:0/1584944377 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac043020 con 0x7f74b415df70 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.956+0000 7f74aaffd640 1 -- 172.21.15.27:0/1584944377 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac03e450 con 0x7f74b415df70 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.956+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b40cd520 con 0x7f74b415df70 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.956+0000 7f74aaffd640 1 -- 172.21.15.27:0/1584944377 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac009510 con 0x7f74b415df70 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.956+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b40ccf20 con 0x7f74b415df70 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.957+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b41f48e0 con 0x7f74b415df70 2024-03-06T07:25:19.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.957+0000 7f74aaffd640 1 -- 172.21.15.27:0/1584944377 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74ac046020 con 0x7f74b415df70 2024-03-06T07:25:19.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.958+0000 7f74aaffd640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f749406b4d0 0x7f749406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.958+0000 7f74aaffd640 1 -- 172.21.15.27:0/1584944377 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f74ac0ac480 con 0x7f74b415df70 2024-03-06T07:25:19.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.958+0000 7f74b259c640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f749406b4d0 0x7f749406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.958+0000 7f74b259c640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f749406b4d0 0x7f749406d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f74a40015e0 tx=0x7f74a4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:19.962+0000 7f74aaffd640 1 -- 172.21.15.27:0/1584944377 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74ac04d030 con 0x7f74b415df70 2024-03-06T07:25:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.144+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74b4067ff0 con 0x7f74b415df70 2024-03-06T07:25:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.145+0000 7f74aaffd640 1 -- 172.21.15.27:0/1584944377 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+12819 (secure 0 0 0) 0x7f74ac07b650 con 0x7f74b415df70 2024-03-06T07:25:20.148 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:20.148 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":54,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:19.389984+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f749406b4d0 msgr2=0x7f749406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f749406b4d0 0x7f749406d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f74a40015e0 tx=0x7f74a4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b415df70 msgr2=0x7f74b415f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b415df70 0x7f74b415f420 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f74ac036040 tx=0x7f74ac004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 shutdown_connections 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f749406b4d0 0x7f749406d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b415df70 0x7f74b415f420 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74b40d18c0 0x7f74b415eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 --2- 172.21.15.27:0/1584944377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74b4071a80 0x7f74b40cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 >> 172.21.15.27:0/1584944377 conn(0x7f74b40d3cf0 msgr2=0x7f74b41602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 shutdown_connections 2024-03-06T07:25:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.147+0000 7f74b8aa4640 1 -- 172.21.15.27:0/1584944377 wait complete. 2024-03-06T07:25:20.161 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:20.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.268+0000 7fba868a3640 1 Processor -- start 2024-03-06T07:25:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.268+0000 7fba868a3640 1 -- start start 2024-03-06T07:25:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba868a3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba80072a30 0x7fba801624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba868a3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba800cf9f0 0x7fba800d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba868a3640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8006d4d0 con 0x7fba800d05f0 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba868a3640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8006dba0 con 0x7fba80072a30 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba868a3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba800d0a60 con 0x7fba800cf9f0 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba7ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba80072a30 0x7fba801624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba84e40640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fba800d05f0 0x7fba8015d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35964/0 (socket says 172.21.15.27:35964) 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba84e40640 1 -- 172.21.15.27:0/1266873823 learned_addr learned my addr 172.21.15.27:0/1266873823 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba7f7fe640 1 --2- 172.21.15.27:0/1266873823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba800cf9f0 0x7fba800d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba7ffff640 1 -- 172.21.15.27:0/1266873823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba800cf9f0 msgr2=0x7fba800d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba7ffff640 1 --2- 172.21.15.27:0/1266873823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba800cf9f0 0x7fba800d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba7ffff640 1 -- 172.21.15.27:0/1266873823 >> v1:172.21.15.27:6789/0 conn(0x7fba800d05f0 legacy=0x7fba8015d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba7effd640 1 -- 172.21.15.27:0/1266873823 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18707178 0 0) 0x7fba8006d4d0 con 0x7fba800d05f0 2024-03-06T07:25:20.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba7ffff640 1 -- 172.21.15.27:0/1266873823 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba800b4950 con 0x7fba80072a30 2024-03-06T07:25:20.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.269+0000 7fba7f7fe640 1 --2- 172.21.15.27:0/1266873823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba800cf9f0 0x7fba800d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:20.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.270+0000 7fba7ffff640 1 --2- 172.21.15.27:0/1266873823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba80072a30 0x7fba801624c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fba680097a0 tx=0x7fba6802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a744ed90a2dfc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.270+0000 7fba7effd640 1 -- 172.21.15.27:0/1266873823 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68038070 con 0x7fba80072a30 2024-03-06T07:25:20.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.270+0000 7fba7effd640 1 -- 172.21.15.27:0/1266873823 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba68031700 con 0x7fba80072a30 2024-03-06T07:25:20.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.270+0000 7fba7effd640 1 -- 172.21.15.27:0/1266873823 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68031ad0 con 0x7fba80072a30 2024-03-06T07:25:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.271+0000 7fba868a3640 1 -- 172.21.15.27:0/1266873823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba80072a30 msgr2=0x7fba801624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.271+0000 7fba868a3640 1 --2- 172.21.15.27:0/1266873823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba80072a30 0x7fba801624c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fba680097a0 tx=0x7fba6802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.271+0000 7fba868a3640 1 -- 172.21.15.27:0/1266873823 shutdown_connections 2024-03-06T07:25:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.271+0000 7fba868a3640 1 --2- 172.21.15.27:0/1266873823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba800cf9f0 0x7fba800d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.271+0000 7fba868a3640 1 --2- 172.21.15.27:0/1266873823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba80072a30 0x7fba801624c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.271+0000 7fba868a3640 1 -- 172.21.15.27:0/1266873823 >> 172.21.15.27:0/1266873823 conn(0x7fba8006af20 msgr2=0x7fba8006b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.271+0000 7fba868a3640 1 -- 172.21.15.27:0/1266873823 shutdown_connections 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.271+0000 7fba868a3640 1 -- 172.21.15.27:0/1266873823 wait complete. 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.272+0000 7fba868a3640 1 Processor -- start 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.272+0000 7fba868a3640 1 -- start start 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba868a3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba80072a30 0x7fba800ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba868a3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba800cf9f0 0x7fba801f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba868a3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba800d05f0 0x7fba801f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba868a3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba80071130 con 0x7fba800d05f0 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba868a3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8006ddf0 con 0x7fba800cf9f0 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba868a3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba80072e10 con 0x7fba80072a30 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba84e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba800d05f0 0x7fba801f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba84e40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba800d05f0 0x7fba801f7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37844/0 (socket says 172.21.15.27:37844) 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba84e40640 1 -- 172.21.15.27:0/1973819319 learned_addr learned my addr 172.21.15.27:0/1973819319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba7ffff640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba80072a30 0x7fba800ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba7f7fe640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba800cf9f0 0x7fba801f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba84e40640 1 -- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba80072a30 msgr2=0x7fba800ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba84e40640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba80072a30 0x7fba800ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba84e40640 1 -- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba800cf9f0 msgr2=0x7fba801f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba84e40640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba800cf9f0 0x7fba801f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.273+0000 7fba84e40640 1 -- 172.21.15.27:0/1973819319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba800cd2b0 con 0x7fba800d05f0 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.274+0000 7fba84e40640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba800d05f0 0x7fba801f7a80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fba7800c700 tx=0x7fba7800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.274+0000 7fba7cff9640 1 -- 172.21.15.27:0/1973819319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78018070 con 0x7fba800d05f0 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.274+0000 7fba7cff9640 1 -- 172.21.15.27:0/1973819319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7800ce10 con 0x7fba800d05f0 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.274+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba800ca4c0 con 0x7fba800d05f0 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.274+0000 7fba7cff9640 1 -- 172.21.15.27:0/1973819319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7800a860 con 0x7fba800d05f0 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.274+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba800ad140 con 0x7fba800d05f0 2024-03-06T07:25:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.275+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8009e320 con 0x7fba800d05f0 2024-03-06T07:25:20.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.275+0000 7fba7cff9640 1 -- 172.21.15.27:0/1973819319 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba7801b070 con 0x7fba800d05f0 2024-03-06T07:25:20.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.276+0000 7fba7cff9640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba6006b4d0 0x7fba6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.276+0000 7fba7cff9640 1 -- 172.21.15.27:0/1973819319 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba78089420 con 0x7fba800d05f0 2024-03-06T07:25:20.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.276+0000 7fba7ffff640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba6006b4d0 0x7fba6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.276+0000 7fba7ffff640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba6006b4d0 0x7fba6006d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fba68035000 tx=0x7fba68008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.280+0000 7fba7cff9640 1 -- 172.21.15.27:0/1973819319 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba78058680 con 0x7fba800d05f0 2024-03-06T07:25:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.463+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba8004f410 con 0x7fba800d05f0 2024-03-06T07:25:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.463+0000 7fba7cff9640 1 -- 172.21.15.27:0/1973819319 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba78057f80 con 0x7fba800d05f0 2024-03-06T07:25:20.465 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:25:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba6006b4d0 msgr2=0x7fba6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba6006b4d0 0x7fba6006d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fba68035000 tx=0x7fba68008030 comp rx=0 tx=0).stop 2024-03-06T07:25:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba800d05f0 msgr2=0x7fba801f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba800d05f0 0x7fba801f7a80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fba7800c700 tx=0x7fba7800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 shutdown_connections 2024-03-06T07:25:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba6006b4d0 0x7fba6006d990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba800d05f0 0x7fba801f7a80 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba800cf9f0 0x7fba801f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 --2- 172.21.15.27:0/1973819319 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba80072a30 0x7fba800ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 >> 172.21.15.27:0/1973819319 conn(0x7fba8006af20 msgr2=0x7fba801617f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 shutdown_connections 2024-03-06T07:25:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.465+0000 7fba868a3640 1 -- 172.21.15.27:0/1973819319 wait complete. 2024-03-06T07:25:20.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:20.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.583+0000 7fdfb5956640 1 Processor -- start 2024-03-06T07:25:20.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.583+0000 7fdfb5956640 1 -- start start 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.584+0000 7fdfb5956640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00b2b50 0x7fdfb00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.584+0000 7fdfb5956640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00ce610 con 0x7fdfb00d3400 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.584+0000 7fdfb5956640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00d2920 con 0x7fdfb0071540 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.584+0000 7fdfb5956640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00d2f70 con 0x7fdfb00b2b50 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.584+0000 7fdfae7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdfb00d3400 0x7fdfb00b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35976/0 (socket says 172.21.15.27:35976) 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.584+0000 7fdfae7fc640 1 -- 172.21.15.27:0/3605119650 learned_addr learned my addr 172.21.15.27:0/3605119650 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.584+0000 7fdfaf7fe640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00b2b50 0x7fdfb00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651257020 0 0) 0x7fdfb00ce610 con 0x7fdfb00d3400 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf9c003520 con 0x7fdfb00d3400 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201914032 0 0) 0x7fdfb00d2920 con 0x7fdfb0071540 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb00ce610 con 0x7fdfb0071540 2024-03-06T07:25:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 233429490 0 0) 0x7fdf9c003520 con 0x7fdfb00d3400 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00b2b50 msgr2=0x7fdfb00b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00b2b50 0x7fdfb00b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 >> v1:172.21.15.69:6789/0 conn(0x7fdfb0071540 legacy=0x7fdfb0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb00d2c00 con 0x7fdfb00d3400 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fdf98001670 con 0x7fdfb00d3400 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfaf7fe640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00b2b50 0x7fdfb00b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 >> v1:172.21.15.27:6789/0 conn(0x7fdfb00d3400 legacy=0x7fdfb00b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00b2b50 0x7fdf9c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf9c004100 0x7fdf9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9c004d40 0x7fdf9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00d2f70 con 0x7fdf9c004d40 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c00bad0 con 0x7fdf9c004100 2024-03-06T07:25:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c00bcd0 con 0x7fdfb00b2b50 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfaf7fe640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9c004d40 0x7fdf9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfaeffd640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00b2b50 0x7fdf9c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfae7fc640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf9c004100 0x7fdf9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfaf7fe640 1 -- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00b2b50 msgr2=0x7fdf9c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfaf7fe640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00b2b50 0x7fdf9c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfaf7fe640 1 -- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf9c004100 msgr2=0x7fdf9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfaf7fe640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf9c004100 0x7fdf9c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.585+0000 7fdfaf7fe640 1 -- 172.21.15.27:0/3605119650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c003520 con 0x7fdf9c004d40 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.586+0000 7fdfaf7fe640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9c004d40 0x7fdf9c0051c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000e320 tx=0x7fdf9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547293af8ab2dc73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.586+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0009dc0 con 0x7fdf9c004d40 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.586+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa002cd60 con 0x7fdf9c004d40 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.586+0000 7fdfadffb640 1 -- 172.21.15.27:0/3605119650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0003b90 con 0x7fdf9c004d40 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.587+0000 7fdfb5956640 1 -- 172.21.15.27:0/3605119650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9c004d40 msgr2=0x7fdf9c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.587+0000 7fdfb5956640 1 --2- 172.21.15.27:0/3605119650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9c004d40 0x7fdf9c0051c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000e320 tx=0x7fdf9c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.587+0000 7fdfaf7fe640 1 -- 172.21.15.27:0/3605119650 reap_dead start 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.587+0000 7fdfb5956640 1 -- 172.21.15.27:0/3605119650 shutdown_connections 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.587+0000 7fdfb5956640 1 -- 172.21.15.27:0/3605119650 >> 172.21.15.27:0/3605119650 conn(0x7fdfb00d3ca0 msgr2=0x7fdfb00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.587+0000 7fdfb5956640 1 -- 172.21.15.27:0/3605119650 shutdown_connections 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.587+0000 7fdfb5956640 1 -- 172.21.15.27:0/3605119650 wait complete. 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.588+0000 7fdfb5956640 1 Processor -- start 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.588+0000 7fdfb5956640 1 -- start start 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfb5956640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb00c9bf0 0x7fdfb01e75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfb5956640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00c9fd0 0x7fdfb014b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfb5956640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfb01e7b00 0x7fdfb014b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfb5956640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb015e370 con 0x7fdfb01e7b00 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfb5956640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00d4be0 con 0x7fdfb00c9bf0 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfb5956640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb014a0c0 con 0x7fdfb00c9fd0 2024-03-06T07:25:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfb01e7b00 0x7fdfb014b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfb01e7b00 0x7fdfb014b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37870/0 (socket says 172.21.15.27:37870) 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaf7fe640 1 -- 172.21.15.27:0/3580873439 learned_addr learned my addr 172.21.15.27:0/3580873439 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfae7fc640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00c9fd0 0x7fdfb014b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaeffd640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb00c9bf0 0x7fdfb01e75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaf7fe640 1 -- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00c9fd0 msgr2=0x7fdfb014b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaf7fe640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00c9fd0 0x7fdfb014b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaf7fe640 1 -- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb00c9bf0 msgr2=0x7fdfb01e75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaf7fe640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb00c9bf0 0x7fdfb01e75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaf7fe640 1 -- 172.21.15.27:0/3580873439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb014a3a0 con 0x7fdfb01e7b00 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.589+0000 7fdfaf7fe640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfb01e7b00 0x7fdfb014b6a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000e410 tx=0x7fdfa0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.590+0000 7fdfa7fff640 1 -- 172.21.15.27:0/3580873439 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0009dc0 con 0x7fdfb01e7b00 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.590+0000 7fdfa7fff640 1 -- 172.21.15.27:0/3580873439 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa0045070 con 0x7fdfb01e7b00 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.590+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb014a680 con 0x7fdfb01e7b00 2024-03-06T07:25:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.590+0000 7fdfa7fff640 1 -- 172.21.15.27:0/3580873439 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0009a50 con 0x7fdfb01e7b00 2024-03-06T07:25:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.590+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb0149d00 con 0x7fdfb01e7b00 2024-03-06T07:25:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.591+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb00d39e0 con 0x7fdfb01e7b00 2024-03-06T07:25:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.591+0000 7fdfa7fff640 1 -- 172.21.15.27:0/3580873439 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfa003d040 con 0x7fdfb01e7b00 2024-03-06T07:25:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.592+0000 7fdfa7fff640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf9006b670 0x7fdf9006db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.592+0000 7fdfa7fff640 1 -- 172.21.15.27:0/3580873439 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdfa00ac520 con 0x7fdfb01e7b00 2024-03-06T07:25:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.592+0000 7fdfaeffd640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf9006b670 0x7fdf9006db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.592+0000 7fdfaeffd640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf9006b670 0x7fdf9006db30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000c00 tx=0x7fdfa8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.596+0000 7fdfa7fff640 1 -- 172.21.15.27:0/3580873439 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfb00d39e0 con 0x7fdfb01e7b00 2024-03-06T07:25:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.781+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdfb0068700 con 0x7fdfb01e7b00 2024-03-06T07:25:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.782+0000 7fdfa7fff640 1 -- 172.21.15.27:0/3580873439 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) v1 ==== 84+0+65 (secure 0 0 0) 0x7fdfa007b130 con 0x7fdfb01e7b00 2024-03-06T07:25:20.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:20.784 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:25:20.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:25:20.784 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:25:20.785 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:25:20.785 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:20.785 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:25:20.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf9006b670 msgr2=0x7fdf9006db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf9006b670 0x7fdf9006db30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000c00 tx=0x7fdfa8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfb01e7b00 msgr2=0x7fdfb014b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfb01e7b00 0x7fdfb014b6a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000e410 tx=0x7fdfa0009750 comp rx=0 tx=0).stop 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 shutdown_connections 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf9006b670 0x7fdf9006db30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfb01e7b00 0x7fdfb014b6a0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdfb00c9fd0 0x7fdfb014b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 --2- 172.21.15.27:0/3580873439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb00c9bf0 0x7fdfb01e75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 >> 172.21.15.27:0/3580873439 conn(0x7fdfb00d3ca0 msgr2=0x7fdfb0071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 shutdown_connections 2024-03-06T07:25:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.784+0000 7fdfb5956640 1 -- 172.21.15.27:0/3580873439 wait complete. 2024-03-06T07:25:20.797 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:20.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:20.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.906+0000 7fdb869c7640 1 Processor -- start 2024-03-06T07:25:20.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.906+0000 7fdb869c7640 1 -- start start 2024-03-06T07:25:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.907+0000 7fdb869c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8015e2f0 0x7fdb800b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.907+0000 7fdb869c7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8015e6d0 con 0x7fdb8006c050 2024-03-06T07:25:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.907+0000 7fdb869c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb800ccd00 con 0x7fdb8006b4e0 2024-03-06T07:25:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.907+0000 7fdb869c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800d0a10 con 0x7fdb8015e2f0 2024-03-06T07:25:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.907+0000 7fdb84f64640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb8006c050 0x7fdb800cdac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35978/0 (socket says 172.21.15.27:35978) 2024-03-06T07:25:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.907+0000 7fdb84f64640 1 -- 172.21.15.27:0/443740245 learned_addr learned my addr 172.21.15.27:0/443740245 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.907+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644754503 0 0) 0x7fdb8015e6d0 con 0x7fdb8006c050 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.907+0000 7fdb7ffff640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8015e2f0 0x7fdb800b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.907+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb68003520 con 0x7fdb8006c050 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 562155054 0 0) 0x7fdb68003520 con 0x7fdb8006c050 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8015e2f0 msgr2=0x7fdb800b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8015e2f0 0x7fdb800b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 >> v1:172.21.15.69:6789/0 conn(0x7fdb8006b4e0 legacy=0x7fdb8006b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb800ccfe0 con 0x7fdb8006c050 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fdb780019f0 con 0x7fdb8006c050 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 >> v1:172.21.15.27:6789/0 conn(0x7fdb8006c050 legacy=0x7fdb800cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb68003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb68004800 0x7fdb68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8015e2f0 0x7fdb68005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800d0a10 con 0x7fdb680039e0 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6800bb80 con 0x7fdb68004800 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6800bd80 con 0x7fdb8015e2f0 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7ffff640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb68003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7f7fe640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb68004800 0x7fdb68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb84f64640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8015e2f0 0x7fdb68005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7ffff640 1 -- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8015e2f0 msgr2=0x7fdb68005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7ffff640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb8015e2f0 0x7fdb68005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7ffff640 1 -- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb68004800 msgr2=0x7fdb68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7ffff640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb68004800 0x7fdb68004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7ffff640 1 -- 172.21.15.27:0/443740245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb68003520 con 0x7fdb680039e0 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.908+0000 7fdb7f7fe640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb68004800 0x7fdb68004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.909+0000 7fdb7ffff640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb68003e00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009ac0 tx=0x7fdb68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a03d0cad915a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.909+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74039070 con 0x7fdb680039e0 2024-03-06T07:25:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.909+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7402c6b0 con 0x7fdb680039e0 2024-03-06T07:25:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.909+0000 7fdb7effd640 1 -- 172.21.15.27:0/443740245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74009690 con 0x7fdb680039e0 2024-03-06T07:25:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.910+0000 7fdb869c7640 1 -- 172.21.15.27:0/443740245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 msgr2=0x7fdb68003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.910+0000 7fdb869c7640 1 --2- 172.21.15.27:0/443740245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb68003e00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009ac0 tx=0x7fdb68005830 comp rx=0 tx=0).stop 2024-03-06T07:25:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.910+0000 7fdb84f64640 1 -- 172.21.15.27:0/443740245 reap_dead start 2024-03-06T07:25:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.910+0000 7fdb869c7640 1 -- 172.21.15.27:0/443740245 shutdown_connections 2024-03-06T07:25:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.910+0000 7fdb869c7640 1 -- 172.21.15.27:0/443740245 >> 172.21.15.27:0/443740245 conn(0x7fdb800b36d0 msgr2=0x7fdb800b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.910+0000 7fdb869c7640 1 -- 172.21.15.27:0/443740245 shutdown_connections 2024-03-06T07:25:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.910+0000 7fdb869c7640 1 -- 172.21.15.27:0/443740245 wait complete. 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.911+0000 7fdb869c7640 1 Processor -- start 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.911+0000 7fdb869c7640 1 -- start start 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb869c7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb800b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb869c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb800cb9c0 0x7fdb801f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb869c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb801f41a0 0x7fdb801f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb869c7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800b41f0 con 0x7fdb680039e0 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb869c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800cd5a0 con 0x7fdb801f41a0 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb869c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800b4370 con 0x7fdb800cb9c0 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb800b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb800b6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37890/0 (socket says 172.21.15.27:37890) 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7ffff640 1 -- 172.21.15.27:0/2264523418 learned_addr learned my addr 172.21.15.27:0/2264523418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7f7fe640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb800cb9c0 0x7fdb801f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7ffff640 1 -- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb800cb9c0 msgr2=0x7fdb801f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb84f64640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb801f41a0 0x7fdb801f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7ffff640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb800cb9c0 0x7fdb801f3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7ffff640 1 -- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb801f41a0 msgr2=0x7fdb801f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7ffff640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb801f41a0 0x7fdb801f4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7ffff640 1 -- 172.21.15.27:0/2264523418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb800b73d0 con 0x7fdb680039e0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.912+0000 7fdb7ffff640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb800b6ba0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009ac0 tx=0x7fdb74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.913+0000 7fdb7cff9640 1 -- 172.21.15.27:0/2264523418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74039050 con 0x7fdb680039e0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.913+0000 7fdb7cff9640 1 -- 172.21.15.27:0/2264523418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb74045070 con 0x7fdb680039e0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.913+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb800b76b0 con 0x7fdb680039e0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.913+0000 7fdb7cff9640 1 -- 172.21.15.27:0/2264523418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74003a50 con 0x7fdb680039e0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.913+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8015ed70 con 0x7fdb680039e0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.914+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8009fd30 con 0x7fdb680039e0 2024-03-06T07:25:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.914+0000 7fdb7cff9640 1 -- 172.21.15.27:0/2264523418 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb74003bf0 con 0x7fdb680039e0 2024-03-06T07:25:20.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.915+0000 7fdb7cff9640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb6c06b500 0x7fdb6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.915+0000 7fdb7cff9640 1 -- 172.21.15.27:0/2264523418 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdb740ab950 con 0x7fdb680039e0 2024-03-06T07:25:20.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.915+0000 7fdb7f7fe640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb6c06b500 0x7fdb6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.916+0000 7fdb7f7fe640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb6c06b500 0x7fdb6c06d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdb800cb1f0 tx=0x7fdb70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:20.919+0000 7fdb7cff9640 1 -- 172.21.15.27:0/2264523418 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb7404e020 con 0x7fdb680039e0 2024-03-06T07:25:21.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:21.102+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdb8004f410 con 0x7fdb680039e0 2024-03-06T07:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.069+0000 7fdb7cff9640 1 -- 172.21.15.27:0/2264523418 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdb7407ad90 con 0x7fdb680039e0 2024-03-06T07:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb6c06b500 msgr2=0x7fdb6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb6c06b500 0x7fdb6c06d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdb800cb1f0 tx=0x7fdb70009040 comp rx=0 tx=0).stop 2024-03-06T07:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 msgr2=0x7fdb800b6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb800b6ba0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009ac0 tx=0x7fdb74004730 comp rx=0 tx=0).stop 2024-03-06T07:25:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 shutdown_connections 2024-03-06T07:25:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb6c06b500 0x7fdb6c06d9c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb801f41a0 0x7fdb801f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb800cb9c0 0x7fdb801f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 --2- 172.21.15.27:0/2264523418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb680039e0 0x7fdb800b6ba0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 >> 172.21.15.27:0/2264523418 conn(0x7fdb800b36d0 msgr2=0x7fdb800b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 shutdown_connections 2024-03-06T07:25:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.072+0000 7fdb869c7640 1 -- 172.21.15.27:0/2264523418 wait complete. 2024-03-06T07:25:22.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.208+0000 7f7ba593a640 1 Processor -- start 2024-03-06T07:25:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.209+0000 7f7ba593a640 1 -- start start 2024-03-06T07:25:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.209+0000 7f7ba593a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ba015b130 0x7f7ba015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.209+0000 7f7ba593a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7ba00ce900 0x7f7ba00ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.209+0000 7f7ba593a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba015bc40 con 0x7f7ba0144640 2024-03-06T07:25:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.209+0000 7f7ba593a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba006df70 con 0x7f7ba015b130 2024-03-06T07:25:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.209+0000 7f7ba593a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba00cf270 con 0x7f7ba00ce900 2024-03-06T07:25:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.210+0000 7f7b9e7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ba015b130 0x7f7ba015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.210+0000 7f7b9e7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ba015b130 0x7f7ba015b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37902/0 (socket says 172.21.15.27:37902) 2024-03-06T07:25:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.210+0000 7f7b9e7fc640 1 -- 172.21.15.27:0/2045668165 learned_addr learned my addr 172.21.15.27:0/2045668165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.210+0000 7f7b9f7fe640 1 --2- 172.21.15.27:0/2045668165 >> v2:172.21.15.69:3300/0 conn(0x7f7ba00ce900 0x7f7ba00ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.210+0000 7f7b9e7fc640 1 -- 172.21.15.27:0/2045668165 >> v2:172.21.15.69:3300/0 conn(0x7f7ba00ce900 msgr2=0x7f7ba00ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.210+0000 7f7b9e7fc640 1 --2- 172.21.15.27:0/2045668165 >> v2:172.21.15.69:3300/0 conn(0x7f7ba00ce900 0x7f7ba00ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.210+0000 7f7b9e7fc640 1 -- 172.21.15.27:0/2045668165 >> v1:172.21.15.69:6789/0 conn(0x7f7ba0144640 legacy=0x7f7ba006e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.210+0000 7f7b9e7fc640 1 -- 172.21.15.27:0/2045668165 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba015bf20 con 0x7f7ba015b130 2024-03-06T07:25:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.210+0000 7f7b9f7fe640 1 --2- 172.21.15.27:0/2045668165 >> v2:172.21.15.69:3300/0 conn(0x7f7ba00ce900 0x7f7ba00ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7b9e7fc640 1 --2- 172.21.15.27:0/2045668165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba015b130 0x7f7ba015b5c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00b4f40 tx=0x7f7b9402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895d1566d6ddfb92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7b9dffb640 1 -- 172.21.15.27:0/2045668165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9403b070 con 0x7f7ba015b130 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7b9dffb640 1 -- 172.21.15.27:0/2045668165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b94036440 con 0x7f7ba015b130 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7b9dffb640 1 -- 172.21.15.27:0/2045668165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b94036810 con 0x7f7ba015b130 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7ba593a640 1 -- 172.21.15.27:0/2045668165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba015b130 msgr2=0x7f7ba015b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7ba593a640 1 --2- 172.21.15.27:0/2045668165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba015b130 0x7f7ba015b5c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00b4f40 tx=0x7f7b9402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7ba593a640 1 -- 172.21.15.27:0/2045668165 shutdown_connections 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7ba593a640 1 --2- 172.21.15.27:0/2045668165 >> v2:172.21.15.69:3300/0 conn(0x7f7ba00ce900 0x7f7ba00ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7ba593a640 1 --2- 172.21.15.27:0/2045668165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba015b130 0x7f7ba015b5c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7ba593a640 1 -- 172.21.15.27:0/2045668165 >> 172.21.15.27:0/2045668165 conn(0x7f7ba0066d00 msgr2=0x7f7ba0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7ba593a640 1 -- 172.21.15.27:0/2045668165 shutdown_connections 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.211+0000 7f7ba593a640 1 -- 172.21.15.27:0/2045668165 wait complete. 2024-03-06T07:25:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.213+0000 7f7ba593a640 1 Processor -- start 2024-03-06T07:25:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.213+0000 7f7ba593a640 1 -- start start 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.213+0000 7f7ba593a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ba00ce900 0x7f7ba0139bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.213+0000 7f7ba593a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba0144640 0x7f7ba0142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.213+0000 7f7ba593a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ba015b130 0x7f7ba0157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.213+0000 7f7ba593a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba0070aa0 con 0x7f7ba0144640 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7ba593a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba006eb10 con 0x7f7ba00ce900 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7ba593a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba0070c20 con 0x7f7ba015b130 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba0144640 0x7f7ba0142f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba0144640 0x7f7ba0142f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37904/0 (socket says 172.21.15.27:37904) 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9e7fc640 1 -- 172.21.15.27:0/545827894 learned_addr learned my addr 172.21.15.27:0/545827894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9f7fe640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ba015b130 0x7f7ba0157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9effd640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ba00ce900 0x7f7ba0139bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9e7fc640 1 -- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ba015b130 msgr2=0x7f7ba0157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9e7fc640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ba015b130 0x7f7ba0157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9e7fc640 1 -- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ba00ce900 msgr2=0x7f7ba0139bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9e7fc640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ba00ce900 0x7f7ba0139bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9e7fc640 1 -- 172.21.15.27:0/545827894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba01434a0 con 0x7f7ba0144640 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b9e7fc640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba0144640 0x7f7ba0142f60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7b9402e120 tx=0x7f7b940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b87fff640 1 -- 172.21.15.27:0/545827894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9403b040 con 0x7f7ba0144640 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b87fff640 1 -- 172.21.15.27:0/545827894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b94046070 con 0x7f7ba0144640 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba0143740 con 0x7f7ba0144640 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7b87fff640 1 -- 172.21.15.27:0/545827894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b94034bd0 con 0x7f7ba0144640 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.214+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba013a3d0 con 0x7f7ba0144640 2024-03-06T07:25:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.215+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba00733c0 con 0x7f7ba0144640 2024-03-06T07:25:22.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.216+0000 7f7b87fff640 1 -- 172.21.15.27:0/545827894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b9404b460 con 0x7f7ba0144640 2024-03-06T07:25:22.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.216+0000 7f7b87fff640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.217+0000 7f7b87fff640 1 -- 172.21.15.27:0/545827894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7b940aefa0 con 0x7f7ba0144640 2024-03-06T07:25:22.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.217+0000 7f7b9effd640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.217+0000 7f7b9effd640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7b90003d90 tx=0x7f7b90009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.221+0000 7f7b87fff640 1 -- 172.21.15.27:0/545827894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b94049360 con 0x7f7ba0144640 2024-03-06T07:25:22.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:22.403+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7ba004f410 con 0x7f7ba0144640 2024-03-06T07:25:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.071+0000 7f7b87fff640 1 -- 172.21.15.27:0/545827894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7b9407dff0 con 0x7f7ba0144640 2024-03-06T07:25:23.074 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.073+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b8c06b4d0 msgr2=0x7f7b8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7b90003d90 tx=0x7f7b90009040 comp rx=0 tx=0).stop 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba0144640 msgr2=0x7f7ba0142f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba0144640 0x7f7ba0142f60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7b9402e120 tx=0x7f7b940046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 shutdown_connections 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ba015b130 0x7f7ba0157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ba0144640 0x7f7ba0142f60 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 --2- 172.21.15.27:0/545827894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ba00ce900 0x7f7ba0139bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 >> 172.21.15.27:0/545827894 conn(0x7f7ba0066d00 msgr2=0x7f7ba00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 shutdown_connections 2024-03-06T07:25:23.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.074+0000 7f7ba593a640 1 -- 172.21.15.27:0/545827894 wait complete. 2024-03-06T07:25:23.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:23.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.211+0000 7f43ce9a0640 1 Processor -- start 2024-03-06T07:25:23.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.212+0000 7f43ce9a0640 1 -- start start 2024-03-06T07:25:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.212+0000 7f43ce9a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c80d1140 0x7f43c80d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.212+0000 7f43ce9a0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c80d23f0 con 0x7f43c80b9430 2024-03-06T07:25:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.212+0000 7f43ce9a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c80d26d0 con 0x7f43c815dcb0 2024-03-06T07:25:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.212+0000 7f43ce9a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80d29b0 con 0x7f43c80d1140 2024-03-06T07:25:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cd19d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f43c80b9430 0x7f43c80d0a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35982/0 (socket says 172.21.15.27:35982) 2024-03-06T07:25:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cd19d640 1 -- 172.21.15.27:0/1631779508 learned_addr learned my addr 172.21.15.27:0/1631779508 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43ce19f640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c80d1140 0x7f43c80d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744839446 0 0) 0x7f43c80d23f0 con 0x7f43c80b9430 2024-03-06T07:25:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43b0003520 con 0x7f43c80b9430 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2955433813 0 0) 0x7f43b0003520 con 0x7f43c80b9430 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c80d1140 msgr2=0x7f43c80d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c80d1140 0x7f43c80d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 >> v1:172.21.15.69:6789/0 conn(0x7f43c815dcb0 legacy=0x7f43c80d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c80d2c90 con 0x7f43c80b9430 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f43bc001670 con 0x7f43c80b9430 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 >> v1:172.21.15.27:6789/0 conn(0x7f43c80b9430 legacy=0x7f43c80d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43b0004800 0x7f43b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c80d1140 0x7f43b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80d29b0 con 0x7f43b00039e0 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b000ba70 con 0x7f43b0004800 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b000bc70 con 0x7f43c80d1140 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cd99e640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43cd19d640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43b0004800 0x7f43b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.213+0000 7f43ce19f640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c80d1140 0x7f43b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.214+0000 7f43cd99e640 1 -- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c80d1140 msgr2=0x7f43b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.214+0000 7f43cd99e640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c80d1140 0x7f43b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.214+0000 7f43cd99e640 1 -- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43b0004800 msgr2=0x7f43b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.214+0000 7f43cd99e640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43b0004800 0x7f43b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.214+0000 7f43cd99e640 1 -- 172.21.15.27:0/1631779508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0003520 con 0x7f43b00039e0 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.214+0000 7f43cd99e640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43b0003e00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f43c0001720 tx=0x7f43b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91d550da503dd2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0001370 con 0x7f43b00039e0 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c00052d0 con 0x7f43b00039e0 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43cc99c640 1 -- 172.21.15.27:0/1631779508 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0003830 con 0x7f43b00039e0 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43ce9a0640 1 -- 172.21.15.27:0/1631779508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 msgr2=0x7f43b0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43ce9a0640 1 --2- 172.21.15.27:0/1631779508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43b0003e00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f43c0001720 tx=0x7f43b0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43ce19f640 1 -- 172.21.15.27:0/1631779508 reap_dead start 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43ce9a0640 1 -- 172.21.15.27:0/1631779508 shutdown_connections 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43ce9a0640 1 -- 172.21.15.27:0/1631779508 >> 172.21.15.27:0/1631779508 conn(0x7f43c8071960 msgr2=0x7f43c8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43ce9a0640 1 -- 172.21.15.27:0/1631779508 shutdown_connections 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.215+0000 7f43ce9a0640 1 -- 172.21.15.27:0/1631779508 wait complete. 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.216+0000 7f43ce9a0640 1 Processor -- start 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43ce9a0640 1 -- start start 2024-03-06T07:25:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43ce9a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43c815adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43ce9a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c815bad0 0x7f43c80c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43ce9a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c815b580 0x7f43c80c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43ce9a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80d3490 con 0x7f43b00039e0 2024-03-06T07:25:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43ce9a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c815e090 con 0x7f43c815b580 2024-03-06T07:25:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43ce9a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80d3610 con 0x7f43c815bad0 2024-03-06T07:25:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43cd99e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43c815adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43cd99e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43c815adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37926/0 (socket says 172.21.15.27:37926) 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.217+0000 7f43cd99e640 1 -- 172.21.15.27:0/494086246 learned_addr learned my addr 172.21.15.27:0/494086246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43cd19d640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c815bad0 0x7f43c80c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43ce19f640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c815b580 0x7f43c80c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43cd99e640 1 -- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c815bad0 msgr2=0x7f43c80c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43cd99e640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c815bad0 0x7f43c80c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43cd99e640 1 -- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c815b580 msgr2=0x7f43c80c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43cd99e640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c815b580 0x7f43c80c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43cd99e640 1 -- 172.21.15.27:0/494086246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c815b330 con 0x7f43b00039e0 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43cd99e640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43c815adf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f43c0001840 tx=0x7f43c0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43c67fc640 1 -- 172.21.15.27:0/494086246 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0041070 con 0x7f43b00039e0 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43c67fc640 1 -- 172.21.15.27:0/494086246 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c0005480 con 0x7f43b00039e0 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43c67fc640 1 -- 172.21.15.27:0/494086246 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0007a30 con 0x7f43b00039e0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c80cd500 con 0x7f43b00039e0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.218+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c80c83d0 con 0x7f43b00039e0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.219+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c804f330 con 0x7f43b00039e0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.219+0000 7f43c67fc640 1 -- 172.21.15.27:0/494086246 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43c0044070 con 0x7f43b00039e0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.220+0000 7f43c67fc640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.220+0000 7f43c67fc640 1 -- 172.21.15.27:0/494086246 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f43c00abca0 con 0x7f43b00039e0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.220+0000 7f43cd19d640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.221+0000 7f43cd19d640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f43c80cc960 tx=0x7f43bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.225+0000 7f43c67fc640 1 -- 172.21.15.27:0/494086246 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43c007ad30 con 0x7f43b00039e0 2024-03-06T07:25:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:23.443+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f43c80b2d20 con 0x7f43b00039e0 2024-03-06T07:25:24.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.095+0000 7f43c67fc640 1 -- 172.21.15.27:0/494086246 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) v1 ==== 172+0+0 (secure 0 0 0) 0x7f43c007a630 con 0x7f43b00039e0 2024-03-06T07:25:24.099 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:25:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.104+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 msgr2=0x7f43b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.104+0000 7f43ce9a0640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f43c80cc960 tx=0x7f43bc009040 comp rx=0 tx=0).stop 2024-03-06T07:25:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 msgr2=0x7f43c815adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43c815adf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f43c0001840 tx=0x7f43c0001120 comp rx=0 tx=0).stop 2024-03-06T07:25:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 shutdown_connections 2024-03-06T07:25:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43b406b4d0 0x7f43b406d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c815b580 0x7f43c80c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43c815bad0 0x7f43c80c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 --2- 172.21.15.27:0/494086246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43b00039e0 0x7f43c815adf0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 >> 172.21.15.27:0/494086246 conn(0x7f43c8071960 msgr2=0x7f43c80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 shutdown_connections 2024-03-06T07:25:24.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.105+0000 7f43ce9a0640 1 -- 172.21.15.27:0/494086246 wait complete. 2024-03-06T07:25:24.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:24.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.232+0000 7fe7ad956640 1 Processor -- start 2024-03-06T07:25:24.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.232+0000 7fe7ad956640 1 -- start start 2024-03-06T07:25:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7ad956640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7a80d3400 0x7fe7a80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7ad956640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80b2b50 0x7fe7a80b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7ad956640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a80ce610 con 0x7fe7a8071540 2024-03-06T07:25:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7ad956640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a80d2f70 con 0x7fe7a80d3400 2024-03-06T07:25:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7ad956640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a80d4be0 con 0x7fe7a80b2b50 2024-03-06T07:25:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7a6ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe7a8071540 0x7fe7a8071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60210/0 (socket says 172.21.15.27:60210) 2024-03-06T07:25:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7a6ffd640 1 -- 172.21.15.27:0/4082859987 learned_addr learned my addr 172.21.15.27:0/4082859987 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7a67fc640 1 --2- 172.21.15.27:0/4082859987 >> v2:172.21.15.27:3300/0 conn(0x7fe7a80d3400 0x7fe7a80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7a77fe640 1 --2- 172.21.15.27:0/4082859987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80b2b50 0x7fe7a80b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7a67fc640 1 -- 172.21.15.27:0/4082859987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80b2b50 msgr2=0x7fe7a80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.233+0000 7fe7a67fc640 1 --2- 172.21.15.27:0/4082859987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80b2b50 0x7fe7a80b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.234+0000 7fe7a67fc640 1 -- 172.21.15.27:0/4082859987 >> v1:172.21.15.27:6789/0 conn(0x7fe7a8071540 legacy=0x7fe7a8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.234+0000 7fe7a67fc640 1 -- 172.21.15.27:0/4082859987 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a80d2920 con 0x7fe7a80d3400 2024-03-06T07:25:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.234+0000 7fe7a67fc640 1 --2- 172.21.15.27:0/4082859987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80d3400 0x7fe7a80b30a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe79c0097a0 tx=0x7fe79c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f718a4dd0999f1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.234+0000 7fe7a5ffb640 1 -- 172.21.15.27:0/4082859987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c035020 con 0x7fe7a80d3400 2024-03-06T07:25:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.234+0000 7fe7a5ffb640 1 -- 172.21.15.27:0/4082859987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79c031830 con 0x7fe7a80d3400 2024-03-06T07:25:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.234+0000 7fe7a5ffb640 1 -- 172.21.15.27:0/4082859987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c031c00 con 0x7fe7a80d3400 2024-03-06T07:25:24.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.235+0000 7fe7ad956640 1 -- 172.21.15.27:0/4082859987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80d3400 msgr2=0x7fe7a80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.235+0000 7fe7ad956640 1 --2- 172.21.15.27:0/4082859987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80d3400 0x7fe7a80b30a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe79c0097a0 tx=0x7fe79c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.235+0000 7fe7ad956640 1 -- 172.21.15.27:0/4082859987 shutdown_connections 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.235+0000 7fe7ad956640 1 --2- 172.21.15.27:0/4082859987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80b2b50 0x7fe7a80b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.235+0000 7fe7ad956640 1 --2- 172.21.15.27:0/4082859987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80d3400 0x7fe7a80b30a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.235+0000 7fe7ad956640 1 -- 172.21.15.27:0/4082859987 >> 172.21.15.27:0/4082859987 conn(0x7fe7a80d3ca0 msgr2=0x7fe7a80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.235+0000 7fe7ad956640 1 -- 172.21.15.27:0/4082859987 shutdown_connections 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.235+0000 7fe7ad956640 1 -- 172.21.15.27:0/4082859987 wait complete. 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.236+0000 7fe7ad956640 1 Processor -- start 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.236+0000 7fe7ad956640 1 -- start start 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7ad956640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7a8071540 0x7fe7a8159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7ad956640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80b2b50 0x7fe7a814f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7ad956640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80d3400 0x7fe7a81606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7ad956640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a815e370 con 0x7fe7a80b2b50 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7ad956640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a80ce8f0 con 0x7fe7a8071540 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7ad956640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a814f6a0 con 0x7fe7a80d3400 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7a67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80b2b50 0x7fe7a814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7a67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80b2b50 0x7fe7a814f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42008/0 (socket says 172.21.15.27:42008) 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7a67fc640 1 -- 172.21.15.27:0/3623939240 learned_addr learned my addr 172.21.15.27:0/3623939240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7a77fe640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80d3400 0x7fe7a81606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7a6ffd640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7a8071540 0x7fe7a8159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7a67fc640 1 -- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80d3400 msgr2=0x7fe7a81606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7a67fc640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80d3400 0x7fe7a81606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7a67fc640 1 -- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7a8071540 msgr2=0x7fe7a8159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.237+0000 7fe7a67fc640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7a8071540 0x7fe7a8159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.238+0000 7fe7a67fc640 1 -- 172.21.15.27:0/3623939240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a8158f00 con 0x7fe7a80b2b50 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.238+0000 7fe7a67fc640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80b2b50 0x7fe7a814f160 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe79c0096f0 tx=0x7fe79c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.238+0000 7fe78ffff640 1 -- 172.21.15.27:0/3623939240 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c040070 con 0x7fe7a80b2b50 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.238+0000 7fe78ffff640 1 -- 172.21.15.27:0/3623939240 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79c03a910 con 0x7fe7a80b2b50 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.238+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a81591e0 con 0x7fe7a80b2b50 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.238+0000 7fe78ffff640 1 -- 172.21.15.27:0/3623939240 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c035040 con 0x7fe7a80b2b50 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.238+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a81597a0 con 0x7fe7a80b2b50 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.239+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a8068700 con 0x7fe7a80b2b50 2024-03-06T07:25:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.239+0000 7fe78ffff640 1 -- 172.21.15.27:0/3623939240 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe79c04b060 con 0x7fe7a80b2b50 2024-03-06T07:25:24.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.240+0000 7fe78ffff640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe79406b4d0 0x7fe79406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.240+0000 7fe78ffff640 1 -- 172.21.15.27:0/3623939240 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe79c0ab880 con 0x7fe7a80b2b50 2024-03-06T07:25:24.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.240+0000 7fe7a6ffd640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe79406b4d0 0x7fe79406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.241+0000 7fe7a6ffd640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe79406b4d0 0x7fe79406d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0007fd0 tx=0x7fe7a0007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.244+0000 7fe78ffff640 1 -- 172.21.15.27:0/3623939240 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe79c031800 con 0x7fe7a80b2b50 2024-03-06T07:25:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.426+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7a806b8e0 con 0x7fe7a80b2b50 2024-03-06T07:25:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.427+0000 7fe78ffff640 1 -- 172.21.15.27:0/3623939240 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16606 (secure 0 0 0) 0x7fe79c07a140 con 0x7fe7a80b2b50 2024-03-06T07:25:24.430 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:24.430 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":57,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:23.445391+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:21.134461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:22.434887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe79406b4d0 msgr2=0x7fe79406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe79406b4d0 0x7fe79406d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0007fd0 tx=0x7fe7a0007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80b2b50 msgr2=0x7fe7a814f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80b2b50 0x7fe7a814f160 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe79c0096f0 tx=0x7fe79c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 shutdown_connections 2024-03-06T07:25:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe79406b4d0 0x7fe79406d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7a80d3400 0x7fe7a81606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7a80b2b50 0x7fe7a814f160 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 --2- 172.21.15.27:0/3623939240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7a8071540 0x7fe7a8159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 >> 172.21.15.27:0/3623939240 conn(0x7fe7a80d3ca0 msgr2=0x7fe7a80b86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.429+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 shutdown_connections 2024-03-06T07:25:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.430+0000 7fe7ad956640 1 -- 172.21.15.27:0/3623939240 wait complete. 2024-03-06T07:25:24.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.547+0000 7f372b31e640 1 Processor -- start 2024-03-06T07:25:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.548+0000 7f372b31e640 1 -- start start 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.548+0000 7f372b31e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f372415df00 0x7f37240d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.548+0000 7f372b31e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f37240b9a30 0x7f37240d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.548+0000 7f372b31e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37240cea30 con 0x7f37240b95a0 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.548+0000 7f372b31e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37240d2af0 con 0x7f372415df00 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.548+0000 7f372b31e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37240cf3c0 con 0x7f37240b9a30 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37290ba640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f372415df00 0x7f37240d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37288b9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f37240b95a0 0x7f37240b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60216/0 (socket says 172.21.15.27:60216) 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37290ba640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f372415df00 0x7f37240d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42024/0 (socket says 172.21.15.27:42024) 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37288b9640 1 -- 172.21.15.27:0/424108271 learned_addr learned my addr 172.21.15.27:0/424108271 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37298bb640 1 --2- 172.21.15.27:0/424108271 >> v2:172.21.15.69:3300/0 conn(0x7f37240b9a30 0x7f37240d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37290ba640 1 -- 172.21.15.27:0/424108271 >> v2:172.21.15.69:3300/0 conn(0x7f37240b9a30 msgr2=0x7f37240d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37290ba640 1 --2- 172.21.15.27:0/424108271 >> v2:172.21.15.69:3300/0 conn(0x7f37240b9a30 0x7f37240d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37290ba640 1 -- 172.21.15.27:0/424108271 >> v1:172.21.15.27:6789/0 conn(0x7f37240b95a0 legacy=0x7f37240b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f3723fff640 1 -- 172.21.15.27:0/424108271 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418889681 0 0) 0x7f37240cea30 con 0x7f37240b95a0 2024-03-06T07:25:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37290ba640 1 -- 172.21.15.27:0/424108271 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37240cf0e0 con 0x7f372415df00 2024-03-06T07:25:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.549+0000 7f37298bb640 1 --2- 172.21.15.27:0/424108271 >> v2:172.21.15.69:3300/0 conn(0x7f37240b9a30 0x7f37240d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f37290ba640 1 --2- 172.21.15.27:0/424108271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372415df00 0x7f37240d25b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f37240d4d20 tx=0x7f371402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24788f63bc3a05f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f3723fff640 1 -- 172.21.15.27:0/424108271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714039070 con 0x7f372415df00 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f3723fff640 1 -- 172.21.15.27:0/424108271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371402cc80 con 0x7f372415df00 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f3723fff640 1 -- 172.21.15.27:0/424108271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714034660 con 0x7f372415df00 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f372b31e640 1 -- 172.21.15.27:0/424108271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372415df00 msgr2=0x7f37240d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f372b31e640 1 --2- 172.21.15.27:0/424108271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372415df00 0x7f37240d25b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f37240d4d20 tx=0x7f371402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f372b31e640 1 -- 172.21.15.27:0/424108271 shutdown_connections 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f372b31e640 1 --2- 172.21.15.27:0/424108271 >> v2:172.21.15.69:3300/0 conn(0x7f37240b9a30 0x7f37240d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f372b31e640 1 --2- 172.21.15.27:0/424108271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f372415df00 0x7f37240d25b0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.550+0000 7f372b31e640 1 -- 172.21.15.27:0/424108271 >> 172.21.15.27:0/424108271 conn(0x7f3724071960 msgr2=0x7f3724071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.551+0000 7f372b31e640 1 -- 172.21.15.27:0/424108271 shutdown_connections 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.551+0000 7f372b31e640 1 -- 172.21.15.27:0/424108271 wait complete. 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.552+0000 7f372b31e640 1 Processor -- start 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.552+0000 7f372b31e640 1 -- start start 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.552+0000 7f372b31e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37240b95a0 0x7f3724156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.552+0000 7f372b31e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f372415df00 0x7f37240ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.552+0000 7f372b31e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37240cd330 0x7f37240caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.552+0000 7f372b31e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37240cec70 con 0x7f37240b95a0 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f372b31e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37240ce750 con 0x7f37240cd330 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f372b31e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37240cedf0 con 0x7f372415df00 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37290ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37240b95a0 0x7f3724156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37290ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37240b95a0 0x7f3724156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42032/0 (socket says 172.21.15.27:42032) 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37290ba640 1 -- 172.21.15.27:0/1047365366 learned_addr learned my addr 172.21.15.27:0/1047365366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37288b9640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f372415df00 0x7f37240ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37290ba640 1 -- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f372415df00 msgr2=0x7f37240ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37290ba640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f372415df00 0x7f37240ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37290ba640 1 -- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37240cd330 msgr2=0x7f37240caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37298bb640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37240cd330 0x7f37240caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37290ba640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37240cd330 0x7f37240caea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37290ba640 1 -- 172.21.15.27:0/1047365366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37240cbc10 con 0x7f37240b95a0 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f37290ba640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37240b95a0 0x7f3724156260 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3714037040 tx=0x7f3714004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f3721ffb640 1 -- 172.21.15.27:0/1047365366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714039050 con 0x7f37240b95a0 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f3721ffb640 1 -- 172.21.15.27:0/1047365366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3714045070 con 0x7f37240b95a0 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f3721ffb640 1 -- 172.21.15.27:0/1047365366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371403ea50 con 0x7f37240b95a0 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.553+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37240cbef0 con 0x7f37240b95a0 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.554+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37240cc4b0 con 0x7f37240b95a0 2024-03-06T07:25:24.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.555+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37240cd740 con 0x7f37240b95a0 2024-03-06T07:25:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.555+0000 7f3721ffb640 1 -- 172.21.15.27:0/1047365366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f371402cb00 con 0x7f37240b95a0 2024-03-06T07:25:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.555+0000 7f3721ffb640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f371006b4d0 0x7f371006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.555+0000 7f3721ffb640 1 -- 172.21.15.27:0/1047365366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37140ad1b0 con 0x7f37240b95a0 2024-03-06T07:25:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.556+0000 7f37288b9640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f371006b4d0 0x7f371006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.556+0000 7f37288b9640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f371006b4d0 0x7f371006d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3718007fd0 tx=0x7f3718009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.560+0000 7f3721ffb640 1 -- 172.21.15.27:0/1047365366 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37240cd740 con 0x7f37240b95a0 2024-03-06T07:25:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.740+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3724067ff0 con 0x7f37240b95a0 2024-03-06T07:25:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.741+0000 7f3721ffb640 1 -- 172.21.15.27:0/1047365366 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16606 (secure 0 0 0) 0x7f371407c150 con 0x7f37240b95a0 2024-03-06T07:25:24.744 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:24.744 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":57,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:23.445391+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:21.134461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:22.434887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:24.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.743+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f371006b4d0 msgr2=0x7f371006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.743+0000 7f372b31e640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f371006b4d0 0x7f371006d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3718007fd0 tx=0x7f3718009040 comp rx=0 tx=0).stop 2024-03-06T07:25:24.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.743+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37240b95a0 msgr2=0x7f3724156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.743+0000 7f372b31e640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37240b95a0 0x7f3724156260 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3714037040 tx=0x7f3714004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.744+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 shutdown_connections 2024-03-06T07:25:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.744+0000 7f372b31e640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f371006b4d0 0x7f371006d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.744+0000 7f372b31e640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37240cd330 0x7f37240caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.744+0000 7f372b31e640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f372415df00 0x7f37240ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.744+0000 7f372b31e640 1 --2- 172.21.15.27:0/1047365366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37240b95a0 0x7f3724156260 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.744+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 >> 172.21.15.27:0/1047365366 conn(0x7f3724071960 msgr2=0x7f3724162250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.744+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 shutdown_connections 2024-03-06T07:25:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.744+0000 7f372b31e640 1 -- 172.21.15.27:0/1047365366 wait complete. 2024-03-06T07:25:24.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:24.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.864+0000 7f282629d640 1 Processor -- start 2024-03-06T07:25:24.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.864+0000 7f282629d640 1 -- start start 2024-03-06T07:25:24.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.865+0000 7f282629d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282015c180 0x7f28200670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.865+0000 7f282629d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f282006def0 0x7f2820071e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.865+0000 7f282629d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2820067620 con 0x7f2820066d00 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.865+0000 7f282629d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2820145910 con 0x7f282006def0 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.865+0000 7f282629d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282015c6f0 con 0x7f282015c180 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.865+0000 7f281ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2820066d00 0x7f2820160180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60218/0 (socket says 172.21.15.27:60218) 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.865+0000 7f281ffff640 1 -- 172.21.15.27:0/912903469 learned_addr learned my addr 172.21.15.27:0/912903469 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.865+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694240404 0 0) 0x7f2820067620 con 0x7f2820066d00 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.865+0000 7f281f7fe640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282015c180 0x7f28200670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2808003520 con 0x7f2820066d00 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f282483a640 1 --2- 172.21.15.27:0/912903469 >> v2:172.21.15.69:3300/0 conn(0x7f282006def0 0x7f2820071e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1297038604 0 0) 0x7f2808003520 con 0x7f2820066d00 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282015c180 msgr2=0x7f28200670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282015c180 0x7f28200670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 >> v2:172.21.15.69:3300/0 conn(0x7f282006def0 msgr2=0x7f2820071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 --2- 172.21.15.27:0/912903469 >> v2:172.21.15.69:3300/0 conn(0x7f282006def0 0x7f2820071e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282015c9d0 con 0x7f2820066d00 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f2814001670 con 0x7f2820066d00 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 >> v1:172.21.15.27:6789/0 conn(0x7f2820066d00 legacy=0x7f2820160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282015c180 0x7f28080039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2808003f30 0x7f2808004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808004d40 0x7f28080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2820145910 con 0x7f2808004d40 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282015c6f0 con 0x7f2808003f30 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280800b9c0 con 0x7f282015c180 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281ffff640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282015c180 0x7f28080039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f282483a640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808004d40 0x7f28080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281f7fe640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2808003f30 0x7f2808004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281f7fe640 1 -- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282015c180 msgr2=0x7f28080039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281f7fe640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282015c180 0x7f28080039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281f7fe640 1 -- 172.21.15.27:0/912903469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808004d40 msgr2=0x7f28080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281f7fe640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808004d40 0x7f28080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f281f7fe640 1 -- 172.21.15.27:0/912903469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2808003520 con 0x7f2808003f30 2024-03-06T07:25:24.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.866+0000 7f282483a640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808004d40 0x7f28080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:24.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.867+0000 7f281f7fe640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2808003f30 0x7f2808004800 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2810009ac0 tx=0x7f2808005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb9d1b2a80545d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.867+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810039070 con 0x7f2808003f30 2024-03-06T07:25:24.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.867+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281002c6b0 con 0x7f2808003f30 2024-03-06T07:25:24.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.867+0000 7f281effd640 1 -- 172.21.15.27:0/912903469 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810009690 con 0x7f2808003f30 2024-03-06T07:25:24.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.868+0000 7f282629d640 1 -- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2808003f30 msgr2=0x7f2808004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.868+0000 7f282629d640 1 --2- 172.21.15.27:0/912903469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2808003f30 0x7f2808004800 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2810009ac0 tx=0x7f2808005830 comp rx=0 tx=0).stop 2024-03-06T07:25:24.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.868+0000 7f282483a640 1 -- 172.21.15.27:0/912903469 reap_dead start 2024-03-06T07:25:24.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.868+0000 7f282629d640 1 -- 172.21.15.27:0/912903469 shutdown_connections 2024-03-06T07:25:24.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.868+0000 7f282629d640 1 -- 172.21.15.27:0/912903469 >> 172.21.15.27:0/912903469 conn(0x7f28200d09c0 msgr2=0x7f28200d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.868+0000 7f282629d640 1 -- 172.21.15.27:0/912903469 shutdown_connections 2024-03-06T07:25:24.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.868+0000 7f282629d640 1 -- 172.21.15.27:0/912903469 wait complete. 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.869+0000 7f282629d640 1 Processor -- start 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.869+0000 7f282629d640 1 -- start start 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f282629d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808003f30 0x7f2820137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f282629d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28201e6620 0x7f28201e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f282629d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282013bd20 0x7f282013c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f282629d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282006e380 con 0x7f2808003f30 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f282629d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282015cf90 con 0x7f28201e6620 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f282629d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282006e500 con 0x7f282013bd20 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f281ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808003f30 0x7f2820137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f281ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808003f30 0x7f2820137ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42050/0 (socket says 172.21.15.27:42050) 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f281ffff640 1 -- 172.21.15.27:0/533078494 learned_addr learned my addr 172.21.15.27:0/533078494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f281f7fe640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28201e6620 0x7f28201e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f282483a640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282013bd20 0x7f282013c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f281ffff640 1 -- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282013bd20 msgr2=0x7f282013c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f281ffff640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282013bd20 0x7f282013c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f281ffff640 1 -- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28201e6620 msgr2=0x7f28201e6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f281ffff640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28201e6620 0x7f28201e6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.870+0000 7f281ffff640 1 -- 172.21.15.27:0/533078494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28201e70a0 con 0x7f2808003f30 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.871+0000 7f281ffff640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808003f30 0x7f2820137ce0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2814007340 tx=0x7f2814007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.871+0000 7f281cff9640 1 -- 172.21.15.27:0/533078494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281400f940 con 0x7f2808003f30 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.871+0000 7f281cff9640 1 -- 172.21.15.27:0/533078494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2814018070 con 0x7f2808003f30 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.871+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28201e7380 con 0x7f2808003f30 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.871+0000 7f281cff9640 1 -- 172.21.15.27:0/533078494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281400fd20 con 0x7f2808003f30 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.871+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28201e7940 con 0x7f2808003f30 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.872+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28200733d0 con 0x7f2808003f30 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.872+0000 7f281cff9640 1 -- 172.21.15.27:0/533078494 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f281401d460 con 0x7f2808003f30 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.873+0000 7f281cff9640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f280c06b4d0 0x7f280c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.873+0000 7f281cff9640 1 -- 172.21.15.27:0/533078494 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f281408a1f0 con 0x7f2808003f30 2024-03-06T07:25:24.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.873+0000 7f281f7fe640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f280c06b4d0 0x7f280c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.873+0000 7f281f7fe640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f280c06b4d0 0x7f280c06d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2810036000 tx=0x7f2810030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:24.877+0000 7f281cff9640 1 -- 172.21.15.27:0/533078494 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2814058a20 con 0x7f2808003f30 2024-03-06T07:25:25.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:25.098+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f282004f410 con 0x7f2808003f30 2024-03-06T07:25:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.105+0000 7f281cff9640 1 -- 172.21.15.27:0/533078494 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f281405cac0 con 0x7f2808003f30 2024-03-06T07:25:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f280c06b4d0 msgr2=0x7f280c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f280c06b4d0 0x7f280c06d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2810036000 tx=0x7f2810030000 comp rx=0 tx=0).stop 2024-03-06T07:25:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808003f30 msgr2=0x7f2820137ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808003f30 0x7f2820137ce0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2814007340 tx=0x7f2814007800 comp rx=0 tx=0).stop 2024-03-06T07:25:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 shutdown_connections 2024-03-06T07:25:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f280c06b4d0 0x7f280c06d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f282013bd20 0x7f282013c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28201e6620 0x7f28201e6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 --2- 172.21.15.27:0/533078494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808003f30 0x7f2820137ce0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 >> 172.21.15.27:0/533078494 conn(0x7f28200d09c0 msgr2=0x7f2820071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 shutdown_connections 2024-03-06T07:25:26.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.107+0000 7f282629d640 1 -- 172.21.15.27:0/533078494 wait complete. 2024-03-06T07:25:26.124 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:26.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:26.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.231+0000 7f91dceed640 1 Processor -- start 2024-03-06T07:25:26.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.232+0000 7f91dceed640 1 -- start start 2024-03-06T07:25:26.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.232+0000 7f91dceed640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91d80cdcd0 0x7f91d80b1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.232+0000 7f91dceed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80b2cd0 0x7f91d80b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.232+0000 7f91dceed640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d8161a40 con 0x7f91d80b1fd0 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.232+0000 7f91dceed640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d80d11e0 con 0x7f91d80cdcd0 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.232+0000 7f91dceed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d806c2d0 con 0x7f91d80b2cd0 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d6d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f91d80b1fd0 0x7f91d80ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60234/0 (socket says 172.21.15.27:60234) 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d6d9d640 1 -- 172.21.15.27:0/1034138478 learned_addr learned my addr 172.21.15.27:0/1034138478 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91cffff640 1 --2- 172.21.15.27:0/1034138478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80b2cd0 0x7f91d80b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d659c640 1 --2- 172.21.15.27:0/1034138478 >> v2:172.21.15.69:3300/0 conn(0x7f91d80cdcd0 0x7f91d80b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d5d9b640 1 -- 172.21.15.27:0/1034138478 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521305071 0 0) 0x7f91d8161a40 con 0x7f91d80b1fd0 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d5d9b640 1 -- 172.21.15.27:0/1034138478 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4003520 con 0x7f91d80b1fd0 2024-03-06T07:25:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d5d9b640 1 -- 172.21.15.27:0/1034138478 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2499225776 0 0) 0x7f91c4003520 con 0x7f91d80b1fd0 2024-03-06T07:25:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d659c640 1 -- 172.21.15.27:0/1034138478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80b2cd0 msgr2=0x7f91d80b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d659c640 1 --2- 172.21.15.27:0/1034138478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80b2cd0 0x7f91d80b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d659c640 1 -- 172.21.15.27:0/1034138478 >> v1:172.21.15.27:6789/0 conn(0x7f91d80b1fd0 legacy=0x7f91d80ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91d659c640 1 -- 172.21.15.27:0/1034138478 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d80b2a20 con 0x7f91d80cdcd0 2024-03-06T07:25:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.233+0000 7f91cffff640 1 --2- 172.21.15.27:0/1034138478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80b2cd0 0x7f91d80b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.234+0000 7f91d659c640 1 --2- 172.21.15.27:0/1034138478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d80cdcd0 0x7f91d80b1550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f91c80097a0 tx=0x7f91c802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3540d5f8123c7b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.234+0000 7f91d5d9b640 1 -- 172.21.15.27:0/1034138478 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c8037070 con 0x7f91d80cdcd0 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.234+0000 7f91d5d9b640 1 -- 172.21.15.27:0/1034138478 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c8030510 con 0x7f91d80cdcd0 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.234+0000 7f91d5d9b640 1 -- 172.21.15.27:0/1034138478 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d00019f0 con 0x7f91d80cdcd0 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.235+0000 7f91dceed640 1 -- 172.21.15.27:0/1034138478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d80cdcd0 msgr2=0x7f91d80b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.235+0000 7f91dceed640 1 --2- 172.21.15.27:0/1034138478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d80cdcd0 0x7f91d80b1550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f91c80097a0 tx=0x7f91c802b710 comp rx=0 tx=0).stop 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.235+0000 7f91dceed640 1 -- 172.21.15.27:0/1034138478 shutdown_connections 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.235+0000 7f91dceed640 1 --2- 172.21.15.27:0/1034138478 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80b2cd0 0x7f91d80b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.235+0000 7f91dceed640 1 --2- 172.21.15.27:0/1034138478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d80cdcd0 0x7f91d80b1550 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.235+0000 7f91dceed640 1 -- 172.21.15.27:0/1034138478 >> 172.21.15.27:0/1034138478 conn(0x7f91d80b3700 msgr2=0x7f91d80b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.235+0000 7f91dceed640 1 -- 172.21.15.27:0/1034138478 shutdown_connections 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.235+0000 7f91dceed640 1 -- 172.21.15.27:0/1034138478 wait complete. 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.236+0000 7f91dceed640 1 Processor -- start 2024-03-06T07:25:26.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.236+0000 7f91dceed640 1 -- start start 2024-03-06T07:25:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91dceed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d80b1fd0 0x7f91d80c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91dceed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d80b2cd0 0x7f91d8155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91dceed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80cdcd0 0x7f91d80cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91dceed640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80d0f00 con 0x7f91d80b2cd0 2024-03-06T07:25:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91dceed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80d0c20 con 0x7f91d80b1fd0 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91dceed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80cc1c0 con 0x7f91d80cdcd0 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d80b2cd0 0x7f91d8155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d80b2cd0 0x7f91d8155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42064/0 (socket says 172.21.15.27:42064) 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91cffff640 1 -- 172.21.15.27:0/183491033 learned_addr learned my addr 172.21.15.27:0/183491033 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91d659c640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d80b1fd0 0x7f91d80c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91d6d9d640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80cdcd0 0x7f91d80cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91cffff640 1 -- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80cdcd0 msgr2=0x7f91d80cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91cffff640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80cdcd0 0x7f91d80cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91cffff640 1 -- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d80b1fd0 msgr2=0x7f91d80c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91cffff640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d80b1fd0 0x7f91d80c55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91cffff640 1 -- 172.21.15.27:0/183491033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d80c5b10 con 0x7f91d80b2cd0 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.237+0000 7f91cffff640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d80b2cd0 0x7f91d8155fe0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f91c000e9b0 tx=0x7f91c000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.238+0000 7f91ceffd640 1 -- 172.21.15.27:0/183491033 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0017070 con 0x7f91d80b2cd0 2024-03-06T07:25:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.238+0000 7f91ceffd640 1 -- 172.21.15.27:0/183491033 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c000ca00 con 0x7f91d80b2cd0 2024-03-06T07:25:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.238+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d80b7490 con 0x7f91d80b2cd0 2024-03-06T07:25:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.238+0000 7f91ceffd640 1 -- 172.21.15.27:0/183491033 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c000cdf0 con 0x7f91d80b2cd0 2024-03-06T07:25:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.238+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d80aeea0 con 0x7f91d80b2cd0 2024-03-06T07:25:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.239+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d80ae9d0 con 0x7f91d80b2cd0 2024-03-06T07:25:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.240+0000 7f91ceffd640 1 -- 172.21.15.27:0/183491033 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91c001a070 con 0x7f91d80b2cd0 2024-03-06T07:25:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.240+0000 7f91ceffd640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91b806b500 0x7f91b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.240+0000 7f91ceffd640 1 -- 172.21.15.27:0/183491033 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91c0089cb0 con 0x7f91d80b2cd0 2024-03-06T07:25:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.240+0000 7f91d659c640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91b806b500 0x7f91b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.241+0000 7f91d659c640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91b806b500 0x7f91b806d9c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f91c8034000 tx=0x7f91c8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.245+0000 7f91ceffd640 1 -- 172.21.15.27:0/183491033 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91c00584f0 con 0x7f91d80b2cd0 2024-03-06T07:25:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:26.466+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f91d80d1690 con 0x7f91d80b2cd0 2024-03-06T07:25:27.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.106+0000 7f91ceffd640 1 -- 172.21.15.27:0/183491033 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f91c005c590 con 0x7f91d80b2cd0 2024-03-06T07:25:27.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91b806b500 msgr2=0x7f91b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91b806b500 0x7f91b806d9c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f91c8034000 tx=0x7f91c8008030 comp rx=0 tx=0).stop 2024-03-06T07:25:27.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d80b2cd0 msgr2=0x7f91d8155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d80b2cd0 0x7f91d8155fe0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f91c000e9b0 tx=0x7f91c000ee70 comp rx=0 tx=0).stop 2024-03-06T07:25:27.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 shutdown_connections 2024-03-06T07:25:27.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f91b806b500 0x7f91b806d9c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91d80cdcd0 0x7f91d80cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d80b2cd0 0x7f91d8155fe0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 --2- 172.21.15.27:0/183491033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91d80b1fd0 0x7f91d80c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 >> 172.21.15.27:0/183491033 conn(0x7f91d80b3700 msgr2=0x7f91d80ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 shutdown_connections 2024-03-06T07:25:27.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.108+0000 7f91dceed640 1 -- 172.21.15.27:0/183491033 wait complete. 2024-03-06T07:25:27.121 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:27.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.225+0000 7f2e83d55640 1 Processor -- start 2024-03-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.226+0000 7f2e83d55640 1 -- start start 2024-03-06T07:25:27.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e83d55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2e7c06b5c0 0x7f2e7c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e83d55640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e7c0b0e90 0x7f2e7c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e83d55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e83d55640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c0cfa00 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e83d55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c0b3340 con 0x7f2e7c06b5c0 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e83d55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0cee90 con 0x7f2e7c0ae9b0 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e812f0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e7c0b0e90 0x7f2e7c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e812f0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e7c0b0e90 0x7f2e7c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42066/0 (socket says 172.21.15.27:42066) 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e812f0640 1 -- 172.21.15.27:0/3654783098 learned_addr learned my addr 172.21.15.27:0/3654783098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e822f2640 1 --2- 172.21.15.27:0/3654783098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.227+0000 7f2e81af1640 1 --2- 172.21.15.27:0/3654783098 >> v2:172.21.15.69:3300/0 conn(0x7f2e7c06b5c0 0x7f2e7c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.228+0000 7f2e812f0640 1 -- 172.21.15.27:0/3654783098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 msgr2=0x7f2e7c0d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.228+0000 7f2e812f0640 1 --2- 172.21.15.27:0/3654783098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c0d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.228+0000 7f2e812f0640 1 -- 172.21.15.27:0/3654783098 >> v2:172.21.15.69:3300/0 conn(0x7f2e7c06b5c0 msgr2=0x7f2e7c0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.228+0000 7f2e812f0640 1 --2- 172.21.15.27:0/3654783098 >> v2:172.21.15.69:3300/0 conn(0x7f2e7c06b5c0 0x7f2e7c0b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.228+0000 7f2e812f0640 1 -- 172.21.15.27:0/3654783098 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c0cfc50 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.228+0000 7f2e81af1640 1 --2- 172.21.15.27:0/3654783098 >> v2:172.21.15.69:3300/0 conn(0x7f2e7c06b5c0 0x7f2e7c0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.228+0000 7f2e822f2640 1 --2- 172.21.15.27:0/3654783098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c0d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.228+0000 7f2e812f0640 1 --2- 172.21.15.27:0/3654783098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 0x7f2e7c06b9a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c071b00 tx=0x7f2e7002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ac7eac4ab155c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.228+0000 7f2e80aef640 1 -- 172.21.15.27:0/3654783098 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7002e070 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e80aef640 1 -- 172.21.15.27:0/3654783098 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e70037070 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e80aef640 1 -- 172.21.15.27:0/3654783098 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70031a00 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e83d55640 1 -- 172.21.15.27:0/3654783098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 msgr2=0x7f2e7c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e83d55640 1 --2- 172.21.15.27:0/3654783098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 0x7f2e7c06b9a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c071b00 tx=0x7f2e7002b880 comp rx=0 tx=0).stop 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e83d55640 1 -- 172.21.15.27:0/3654783098 shutdown_connections 2024-03-06T07:25:27.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e83d55640 1 --2- 172.21.15.27:0/3654783098 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c0d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e83d55640 1 --2- 172.21.15.27:0/3654783098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 0x7f2e7c06b9a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e83d55640 1 --2- 172.21.15.27:0/3654783098 >> v2:172.21.15.69:3300/0 conn(0x7f2e7c06b5c0 0x7f2e7c0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e83d55640 1 -- 172.21.15.27:0/3654783098 >> 172.21.15.27:0/3654783098 conn(0x7f2e7c0b36d0 msgr2=0x7f2e7c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e83d55640 1 -- 172.21.15.27:0/3654783098 shutdown_connections 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.229+0000 7f2e83d55640 1 -- 172.21.15.27:0/3654783098 wait complete. 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.230+0000 7f2e83d55640 1 Processor -- start 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.230+0000 7f2e83d55640 1 -- start start 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e83d55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c06b5c0 0x7f2e7c1599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e83d55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c15b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e83d55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 0x7f2e7c0af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e83d55640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c06c790 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e83d55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c06bee0 con 0x7f2e7c06b5c0 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e83d55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c06c910 con 0x7f2e7c0ae9b0 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e822f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 0x7f2e7c0af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e822f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 0x7f2e7c0af3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42070/0 (socket says 172.21.15.27:42070) 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e822f2640 1 -- 172.21.15.27:0/60160559 learned_addr learned my addr 172.21.15.27:0/60160559 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e812f0640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c15b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.231+0000 7f2e81af1640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c06b5c0 0x7f2e7c1599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e822f2640 1 -- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 msgr2=0x7f2e7c15b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e822f2640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c15b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e822f2640 1 -- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c06b5c0 msgr2=0x7f2e7c1599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e822f2640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c06b5c0 0x7f2e7c1599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e822f2640 1 -- 172.21.15.27:0/60160559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c1f3e40 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e812f0640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c15b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e81af1640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c06b5c0 0x7f2e7c1599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e822f2640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 0x7f2e7c0af3a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c007ef0 tx=0x7f2e6c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e7a7fc640 1 -- 172.21.15.27:0/60160559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c01a070 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e7a7fc640 1 -- 172.21.15.27:0/60160559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c003b90 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c1f4120 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e7a7fc640 1 -- 172.21.15.27:0/60160559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c004030 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.232+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c1f46e0 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.233+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7c068700 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.234+0000 7f2e7a7fc640 1 -- 172.21.15.27:0/60160559 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e6c01d070 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.234+0000 7f2e7a7fc640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e6806b4d0 0x7f2e6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.234+0000 7f2e7a7fc640 1 -- 172.21.15.27:0/60160559 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e6c08a440 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.234+0000 7f2e81af1640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e6806b4d0 0x7f2e6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.235+0000 7f2e81af1640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e6806b4d0 0x7f2e6806d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400ba50 tx=0x7f2e74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.239+0000 7f2e7a7fc640 1 -- 172.21.15.27:0/60160559 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e6c010070 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.458+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e7c0706e0 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.459+0000 7f2e7a7fc640 1 -- 172.21.15.27:0/60160559 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f2e6c058c70 con 0x7f2e7c0b0e90 2024-03-06T07:25:27.462 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:27.462 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5324,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3885016831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3885016831},{"type":"v1","addr":"172.21.15.69:6836","nonce":3885016831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5327,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3510003111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3510003111},{"type":"v1","addr":"172.21.15.69:6837","nonce":3510003111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5692,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3053251134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3053251134},{"type":"v1","addr":"172.21.15.27:6835","nonce":3053251134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:23.445711+0000","modified":"2024-03-06T07:25:26.468573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6129":{"gid":6129,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6833/60877422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":60877422},{"type":"v1","addr":"172.21.15.27:6833","nonce":60877422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6129,"qdb_cluster":[6129]},"id":4}]} 2024-03-06T07:25:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e6806b4d0 msgr2=0x7f2e6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e6806b4d0 0x7f2e6806d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400ba50 tx=0x7f2e74008040 comp rx=0 tx=0).stop 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 msgr2=0x7f2e7c0af3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 0x7f2e7c0af3a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c007ef0 tx=0x7f2e6c00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 shutdown_connections 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e6806b4d0 0x7f2e6806d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c0b0e90 0x7f2e7c0af3a0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e7c0ae9b0 0x7f2e7c15b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 --2- 172.21.15.27:0/60160559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c06b5c0 0x7f2e7c1599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 >> 172.21.15.27:0/60160559 conn(0x7f2e7c0b36d0 msgr2=0x7f2e7c0cdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 shutdown_connections 2024-03-06T07:25:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.461+0000 7f2e83d55640 1 -- 172.21.15.27:0/60160559 wait complete. 2024-03-06T07:25:27.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.578+0000 7f9aa749d640 1 Processor -- start 2024-03-06T07:25:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.578+0000 7f9aa749d640 1 -- start start 2024-03-06T07:25:27.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.579+0000 7f9aa749d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9aa00d3400 0x7f9aa00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.579+0000 7f9aa749d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa00ce610 con 0x7f9aa0071540 2024-03-06T07:25:27.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.579+0000 7f9aa749d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa00d2920 con 0x7f9aa00b2b50 2024-03-06T07:25:27.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.579+0000 7f9aa749d640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa00d2f70 con 0x7f9aa00d3400 2024-03-06T07:25:27.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.579+0000 7f9aa4a38640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9aa00d3400 0x7f9aa00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.579+0000 7f9aa5239640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9aa0071540 0x7f9aa0071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60248/0 (socket says 172.21.15.27:60248) 2024-03-06T07:25:27.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.579+0000 7f9aa4a38640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9aa00d3400 0x7f9aa00b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42086/0 (socket says 172.21.15.27:42086) 2024-03-06T07:25:27.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.579+0000 7f9aa5239640 1 -- 172.21.15.27:0/2245408868 learned_addr learned my addr 172.21.15.27:0/2245408868 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:27.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.580+0000 7f9aa4a38640 1 -- 172.21.15.27:0/2245408868 >> v1:172.21.15.69:6789/0 conn(0x7f9aa00b2b50 legacy=0x7f9aa00b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.580+0000 7f9aa4a38640 1 -- 172.21.15.27:0/2245408868 >> v1:172.21.15.27:6789/0 conn(0x7f9aa0071540 legacy=0x7f9aa0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.580+0000 7f9aa4a38640 1 -- 172.21.15.27:0/2245408868 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa00d2c00 con 0x7f9aa00d3400 2024-03-06T07:25:27.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.580+0000 7f9aa4a38640 1 --2- 172.21.15.27:0/2245408868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 0x7f9aa00b30a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9a900097a0 tx=0x7f9a9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9cd0f2b8433de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.581+0000 7f9a97fff640 1 -- 172.21.15.27:0/2245408868 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90035020 con 0x7f9aa00d3400 2024-03-06T07:25:27.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.581+0000 7f9a97fff640 1 -- 172.21.15.27:0/2245408868 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a90031830 con 0x7f9aa00d3400 2024-03-06T07:25:27.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.581+0000 7f9a97fff640 1 -- 172.21.15.27:0/2245408868 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90031c00 con 0x7f9aa00d3400 2024-03-06T07:25:27.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.581+0000 7f9aa749d640 1 -- 172.21.15.27:0/2245408868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 msgr2=0x7f9aa00b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.581+0000 7f9aa749d640 1 --2- 172.21.15.27:0/2245408868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 0x7f9aa00b30a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9a900097a0 tx=0x7f9a9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.581+0000 7f9aa749d640 1 -- 172.21.15.27:0/2245408868 shutdown_connections 2024-03-06T07:25:27.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.581+0000 7f9aa749d640 1 --2- 172.21.15.27:0/2245408868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 0x7f9aa00b30a0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.581+0000 7f9aa749d640 1 -- 172.21.15.27:0/2245408868 >> 172.21.15.27:0/2245408868 conn(0x7f9aa00d3ca0 msgr2=0x7f9aa00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.582+0000 7f9aa749d640 1 -- 172.21.15.27:0/2245408868 shutdown_connections 2024-03-06T07:25:27.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.582+0000 7f9aa749d640 1 -- 172.21.15.27:0/2245408868 wait complete. 2024-03-06T07:25:27.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.583+0000 7f9aa749d640 1 Processor -- start 2024-03-06T07:25:27.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa749d640 1 -- start start 2024-03-06T07:25:27.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa749d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aa0071540 0x7f9aa00b7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa749d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aa00b2b50 0x7f9aa0149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa749d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 0x7f9aa0149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa749d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa015e370 con 0x7f9aa00d3400 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa749d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00d4be0 con 0x7f9aa00b2b50 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa749d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa014af40 con 0x7f9aa0071540 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa5a3a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 0x7f9aa0149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa5a3a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 0x7f9aa0149c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42092/0 (socket says 172.21.15.27:42092) 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.584+0000 7f9aa5a3a640 1 -- 172.21.15.27:0/3550894688 learned_addr learned my addr 172.21.15.27:0/3550894688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa5239640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aa0071540 0x7f9aa00b7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa4a38640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aa00b2b50 0x7f9aa0149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa5a3a640 1 -- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aa0071540 msgr2=0x7f9aa00b7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa5a3a640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aa0071540 0x7f9aa00b7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa5a3a640 1 -- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aa00b2b50 msgr2=0x7f9aa0149720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa5a3a640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aa00b2b50 0x7f9aa0149720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa5a3a640 1 -- 172.21.15.27:0/3550894688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa014a330 con 0x7f9aa00d3400 2024-03-06T07:25:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa5a3a640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 0x7f9aa0149c60 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800c780 tx=0x7f9a9800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9a95ffb640 1 -- 172.21.15.27:0/3550894688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98004b60 con 0x7f9aa00d3400 2024-03-06T07:25:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9a95ffb640 1 -- 172.21.15.27:0/3550894688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a98004d00 con 0x7f9aa00d3400 2024-03-06T07:25:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa014a610 con 0x7f9aa00d3400 2024-03-06T07:25:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9a95ffb640 1 -- 172.21.15.27:0/3550894688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9800aa90 con 0x7f9aa00d3400 2024-03-06T07:25:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.585+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa014abd0 con 0x7f9aa00d3400 2024-03-06T07:25:27.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.586+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa0068700 con 0x7f9aa00d3400 2024-03-06T07:25:27.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.587+0000 7f9a95ffb640 1 -- 172.21.15.27:0/3550894688 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a9800ac30 con 0x7f9aa00d3400 2024-03-06T07:25:27.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.587+0000 7f9a95ffb640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a8406b4d0 0x7f9a8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.588+0000 7f9a95ffb640 1 -- 172.21.15.27:0/3550894688 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a9808a4e0 con 0x7f9aa00d3400 2024-03-06T07:25:27.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.588+0000 7f9aa5239640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a8406b4d0 0x7f9a8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.588+0000 7f9aa5239640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a8406b4d0 0x7f9a8406d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c000f60 tx=0x7f9a9c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.597+0000 7f9a95ffb640 1 -- 172.21.15.27:0/3550894688 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a98020070 con 0x7f9aa00d3400 2024-03-06T07:25:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.780+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9aa015f510 con 0x7f9aa00d3400 2024-03-06T07:25:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.781+0000 7f9a95ffb640 1 -- 172.21.15.27:0/3550894688 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f9a98059360 con 0x7f9aa00d3400 2024-03-06T07:25:27.783 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:27.784 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:25.095713+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:21.134461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:22.434887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a8406b4d0 msgr2=0x7f9a8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a8406b4d0 0x7f9a8406d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c000f60 tx=0x7f9a9c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 msgr2=0x7f9aa0149c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 0x7f9aa0149c60 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800c780 tx=0x7f9a9800cc40 comp rx=0 tx=0).stop 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 shutdown_connections 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a8406b4d0 0x7f9a8406d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00d3400 0x7f9aa0149c60 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aa00b2b50 0x7f9aa0149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 --2- 172.21.15.27:0/3550894688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9aa0071540 0x7f9aa00b7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 >> 172.21.15.27:0/3550894688 conn(0x7f9aa00d3ca0 msgr2=0x7f9aa015f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 shutdown_connections 2024-03-06T07:25:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.783+0000 7f9aa749d640 1 -- 172.21.15.27:0/3550894688 wait complete. 2024-03-06T07:25:27.797 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:27.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:27.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.905+0000 7f2ae52f4640 1 Processor -- start 2024-03-06T07:25:27.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.905+0000 7f2ae52f4640 1 -- start start 2024-03-06T07:25:27.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.906+0000 7f2ae52f4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ae006ed10 0x7f2ae006af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.906+0000 7f2ae52f4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ae00cd890 0x7f2ae015d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.906+0000 7f2ae52f4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae00cc0e0 con 0x7f2ae00b1020 2024-03-06T07:25:27.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.906+0000 7f2ae52f4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae00d28b0 con 0x7f2ae006ed10 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.906+0000 7f2ae52f4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae00cf320 con 0x7f2ae00cd890 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.906+0000 7f2ade7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ae006ed10 0x7f2ae006af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.906+0000 7f2adeffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2ae00b1020 0x7f2ae00cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60250/0 (socket says 172.21.15.27:60250) 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.906+0000 7f2adeffd640 1 -- 172.21.15.27:0/4219960922 learned_addr learned my addr 172.21.15.27:0/4219960922 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.906+0000 7f2adf7fe640 1 --2- 172.21.15.27:0/4219960922 >> v2:172.21.15.69:3300/0 conn(0x7f2ae00cd890 0x7f2ae015d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.907+0000 7f2ade7fc640 1 -- 172.21.15.27:0/4219960922 >> v2:172.21.15.69:3300/0 conn(0x7f2ae00cd890 msgr2=0x7f2ae015d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.907+0000 7f2ade7fc640 1 --2- 172.21.15.27:0/4219960922 >> v2:172.21.15.69:3300/0 conn(0x7f2ae00cd890 0x7f2ae015d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.907+0000 7f2ade7fc640 1 -- 172.21.15.27:0/4219960922 >> v1:172.21.15.27:6789/0 conn(0x7f2ae00b1020 legacy=0x7f2ae00cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.907+0000 7f2addffb640 1 -- 172.21.15.27:0/4219960922 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978186982 0 0) 0x7f2ae00cc0e0 con 0x7f2ae00b1020 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.907+0000 7f2ade7fc640 1 -- 172.21.15.27:0/4219960922 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae00cc3c0 con 0x7f2ae006ed10 2024-03-06T07:25:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.907+0000 7f2adf7fe640 1 --2- 172.21.15.27:0/4219960922 >> v2:172.21.15.69:3300/0 conn(0x7f2ae00cd890 0x7f2ae015d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.907+0000 7f2ade7fc640 1 --2- 172.21.15.27:0/4219960922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae006ed10 0x7f2ae006af20 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00cc070 tx=0x7f2ad402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a2ad7829cbdbbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2addffb640 1 -- 172.21.15.27:0/4219960922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4035020 con 0x7f2ae006ed10 2024-03-06T07:25:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2addffb640 1 -- 172.21.15.27:0/4219960922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad4031830 con 0x7f2ae006ed10 2024-03-06T07:25:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2addffb640 1 -- 172.21.15.27:0/4219960922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4031c00 con 0x7f2ae006ed10 2024-03-06T07:25:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2ae52f4640 1 -- 172.21.15.27:0/4219960922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae006ed10 msgr2=0x7f2ae006af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2ae52f4640 1 --2- 172.21.15.27:0/4219960922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae006ed10 0x7f2ae006af20 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00cc070 tx=0x7f2ad402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2ae52f4640 1 -- 172.21.15.27:0/4219960922 shutdown_connections 2024-03-06T07:25:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2ae52f4640 1 --2- 172.21.15.27:0/4219960922 >> v2:172.21.15.69:3300/0 conn(0x7f2ae00cd890 0x7f2ae015d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2ae52f4640 1 --2- 172.21.15.27:0/4219960922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae006ed10 0x7f2ae006af20 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2ae52f4640 1 -- 172.21.15.27:0/4219960922 >> 172.21.15.27:0/4219960922 conn(0x7f2ae00d1a90 msgr2=0x7f2ae00d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2ae52f4640 1 -- 172.21.15.27:0/4219960922 shutdown_connections 2024-03-06T07:25:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.908+0000 7f2ae52f4640 1 -- 172.21.15.27:0/4219960922 wait complete. 2024-03-06T07:25:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.910+0000 7f2ae52f4640 1 Processor -- start 2024-03-06T07:25:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.910+0000 7f2ae52f4640 1 -- start start 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.910+0000 7f2ae52f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ae006ed10 0x7f2ae015fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.910+0000 7f2ae52f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ae00b1020 0x7f2ae00c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.910+0000 7f2ae52f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae00cd890 0x7f2ae00c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.910+0000 7f2ae52f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae00ccc60 con 0x7f2ae00cd890 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.910+0000 7f2ae52f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae00cc980 con 0x7f2ae006ed10 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.910+0000 7f2ae52f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae00cdca0 con 0x7f2ae00b1020 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae00cd890 0x7f2ae00c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae00cd890 0x7f2ae00c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42122/0 (socket says 172.21.15.27:42122) 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adf7fe640 1 -- 172.21.15.27:0/61362908 learned_addr learned my addr 172.21.15.27:0/61362908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2ade7fc640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ae00b1020 0x7f2ae00c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adf7fe640 1 -- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ae00b1020 msgr2=0x7f2ae00c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adeffd640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ae006ed10 0x7f2ae015fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adf7fe640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ae00b1020 0x7f2ae00c8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adf7fe640 1 -- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ae006ed10 msgr2=0x7f2ae015fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adf7fe640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ae006ed10 0x7f2ae015fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adf7fe640 1 -- 172.21.15.27:0/61362908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae00b5a70 con 0x7f2ae00cd890 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2adf7fe640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae00cd890 0x7f2ae00c90a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8007ea0 tx=0x7f2ad8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2ac77fe640 1 -- 172.21.15.27:0/61362908 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad801a070 con 0x7f2ae00cd890 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2ac77fe640 1 -- 172.21.15.27:0/61362908 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad8003bb0 con 0x7f2ae00cd890 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae00c9800 con 0x7f2ae00cd890 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2ac77fe640 1 -- 172.21.15.27:0/61362908 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad800de70 con 0x7f2ae00cd890 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.911+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae00b6010 con 0x7f2ae00cd890 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.912+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae00b58a0 con 0x7f2ae00cd890 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.913+0000 7f2ac77fe640 1 -- 172.21.15.27:0/61362908 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ad801d070 con 0x7f2ae00cd890 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.913+0000 7f2ac77fe640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2acc06b500 0x7f2acc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.913+0000 7f2ac77fe640 1 -- 172.21.15.27:0/61362908 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ad808a410 con 0x7f2ae00cd890 2024-03-06T07:25:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.913+0000 7f2adeffd640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2acc06b500 0x7f2acc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.914+0000 7f2adeffd640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2acc06b500 0x7f2acc06d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0007fd0 tx=0x7f2ad0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:27.922+0000 7f2ac77fe640 1 -- 172.21.15.27:0/61362908 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ad8010070 con 0x7f2ae00cd890 2024-03-06T07:25:28.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.105+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ae00c9ae0 con 0x7f2acc06b500 2024-03-06T07:25:28.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.113+0000 7f2ac77fe640 1 -- 172.21.15.27:0/61362908 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251884 (secure 0 0 0) 0x7f2ad0001c70 con 0x7f2acc06b500 2024-03-06T07:25:28.118 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:28.119 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2acc06b500 msgr2=0x7f2acc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2acc06b500 0x7f2acc06d9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0007fd0 tx=0x7f2ad0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae00cd890 msgr2=0x7f2ae00c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae00cd890 0x7f2ae00c90a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8007ea0 tx=0x7f2ad8007f90 comp rx=0 tx=0).stop 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 shutdown_connections 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2acc06b500 0x7f2acc06d9c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ae00cd890 0x7f2ae00c90a0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ae00b1020 0x7f2ae00c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 --2- 172.21.15.27:0/61362908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ae006ed10 0x7f2ae015fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 >> 172.21.15.27:0/61362908 conn(0x7f2ae00d1a90 msgr2=0x7f2ae0068e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 shutdown_connections 2024-03-06T07:25:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.117+0000 7f2ae52f4640 1 -- 172.21.15.27:0/61362908 wait complete. 2024-03-06T07:25:28.132 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-03-06T07:25:28.024235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256044,"kb_used_data":6548,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718676,"statfs":{"total":773094113280,"available":772831924224,"internally_reserved":0,"allocated":6705152,"data_stored":3434733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.952713"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.115564+0000","last_change":"2024-03-06T07:25:24.110516+0000","last_active":"2024-03-06T07:25:24.115564+0000","last_peered":"2024-03-06T07:25:24.115564+0000","last_clean":"2024-03-06T07:25:24.115564+0000","last_became_active":"2024-03-06T07:25:24.110246+0000","last_became_peered":"2024-03-06T07:25:24.110246+0000","last_unstale":"2024-03-06T07:25:24.115564+0000","last_undegraded":"2024-03-06T07:25:24.115564+0000","last_fullsized":"2024-03-06T07:25:24.115564+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:17.820565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109752+0000","last_change":"2024-03-06T07:25:24.105382+0000","last_active":"2024-03-06T07:25:24.109752+0000","last_peered":"2024-03-06T07:25:24.109752+0000","last_clean":"2024-03-06T07:25:24.109752+0000","last_became_active":"2024-03-06T07:25:24.105111+0000","last_became_peered":"2024-03-06T07:25:24.105111+0000","last_unstale":"2024-03-06T07:25:24.109752+0000","last_undegraded":"2024-03-06T07:25:24.109752+0000","last_fullsized":"2024-03-06T07:25:24.109752+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:32.294062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108365+0000","last_change":"2024-03-06T07:25:24.105622+0000","last_active":"2024-03-06T07:25:24.108365+0000","last_peered":"2024-03-06T07:25:24.108365+0000","last_clean":"2024-03-06T07:25:24.108365+0000","last_became_active":"2024-03-06T07:25:24.105341+0000","last_became_peered":"2024-03-06T07:25:24.105341+0000","last_unstale":"2024-03-06T07:25:24.108365+0000","last_undegraded":"2024-03-06T07:25:24.108365+0000","last_fullsized":"2024-03-06T07:25:24.108365+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:38.207630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105969+0000","last_change":"2024-03-06T07:25:24.099774+0000","last_active":"2024-03-06T07:25:24.105969+0000","last_peered":"2024-03-06T07:25:24.105969+0000","last_clean":"2024-03-06T07:25:24.105969+0000","last_became_active":"2024-03-06T07:25:24.099458+0000","last_became_peered":"2024-03-06T07:25:24.099458+0000","last_unstale":"2024-03-06T07:25:24.105969+0000","last_undegraded":"2024-03-06T07:25:24.105969+0000","last_fullsized":"2024-03-06T07:25:24.105969+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:44.507358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108098+0000","last_change":"2024-03-06T07:25:24.104105+0000","last_active":"2024-03-06T07:25:24.108098+0000","last_peered":"2024-03-06T07:25:24.108098+0000","last_clean":"2024-03-06T07:25:24.108098+0000","last_became_active":"2024-03-06T07:25:24.103807+0000","last_became_peered":"2024-03-06T07:25:24.103807+0000","last_unstale":"2024-03-06T07:25:24.108098+0000","last_undegraded":"2024-03-06T07:25:24.108098+0000","last_fullsized":"2024-03-06T07:25:24.108098+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:37.406837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.098354+0000","last_change":"2024-03-06T07:25:24.097152+0000","last_active":"2024-03-06T07:25:24.098354+0000","last_peered":"2024-03-06T07:25:24.098354+0000","last_clean":"2024-03-06T07:25:24.098354+0000","last_became_active":"2024-03-06T07:25:24.096878+0000","last_became_peered":"2024-03-06T07:25:24.096878+0000","last_unstale":"2024-03-06T07:25:24.098354+0000","last_undegraded":"2024-03-06T07:25:24.098354+0000","last_fullsized":"2024-03-06T07:25:24.098354+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:44.507358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105056+0000","last_change":"2024-03-06T07:25:24.100455+0000","last_active":"2024-03-06T07:25:24.105056+0000","last_peered":"2024-03-06T07:25:24.105056+0000","last_clean":"2024-03-06T07:25:24.105056+0000","last_became_active":"2024-03-06T07:25:24.100185+0000","last_became_peered":"2024-03-06T07:25:24.100185+0000","last_unstale":"2024-03-06T07:25:24.105056+0000","last_undegraded":"2024-03-06T07:25:24.105056+0000","last_fullsized":"2024-03-06T07:25:24.105056+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:09.534327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.103014+0000","last_change":"2024-03-06T07:25:24.096626+0000","last_active":"2024-03-06T07:25:24.103014+0000","last_peered":"2024-03-06T07:25:24.103014+0000","last_clean":"2024-03-06T07:25:24.103014+0000","last_became_active":"2024-03-06T07:25:24.096271+0000","last_became_peered":"2024-03-06T07:25:24.096271+0000","last_unstale":"2024-03-06T07:25:24.103014+0000","last_undegraded":"2024-03-06T07:25:24.103014+0000","last_fullsized":"2024-03-06T07:25:24.103014+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:01.234337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.104932+0000","last_change":"2024-03-06T07:25:24.099697+0000","last_active":"2024-03-06T07:25:24.104932+0000","last_peered":"2024-03-06T07:25:24.104932+0000","last_clean":"2024-03-06T07:25:24.104932+0000","last_became_active":"2024-03-06T07:25:24.099031+0000","last_became_peered":"2024-03-06T07:25:24.099031+0000","last_unstale":"2024-03-06T07:25:24.104932+0000","last_undegraded":"2024-03-06T07:25:24.104932+0000","last_fullsized":"2024-03-06T07:25:24.104932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:16.235427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108301+0000","last_change":"2024-03-06T07:25:24.096687+0000","last_active":"2024-03-06T07:25:24.108301+0000","last_peered":"2024-03-06T07:25:24.108301+0000","last_clean":"2024-03-06T07:25:24.108301+0000","last_became_active":"2024-03-06T07:25:24.096384+0000","last_became_peered":"2024-03-06T07:25:24.096384+0000","last_unstale":"2024-03-06T07:25:24.108301+0000","last_undegraded":"2024-03-06T07:25:24.108301+0000","last_fullsized":"2024-03-06T07:25:24.108301+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:49.573382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108435+0000","last_change":"2024-03-06T07:25:24.101960+0000","last_active":"2024-03-06T07:25:24.108435+0000","last_peered":"2024-03-06T07:25:24.108435+0000","last_clean":"2024-03-06T07:25:24.108435+0000","last_became_active":"2024-03-06T07:25:24.101702+0000","last_became_peered":"2024-03-06T07:25:24.101702+0000","last_unstale":"2024-03-06T07:25:24.108435+0000","last_undegraded":"2024-03-06T07:25:24.108435+0000","last_fullsized":"2024-03-06T07:25:24.108435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.849922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.106251+0000","last_change":"2024-03-06T07:25:24.102111+0000","last_active":"2024-03-06T07:25:24.106251+0000","last_peered":"2024-03-06T07:25:24.106251+0000","last_clean":"2024-03-06T07:25:24.106251+0000","last_became_active":"2024-03-06T07:25:24.101795+0000","last_became_peered":"2024-03-06T07:25:24.101795+0000","last_unstale":"2024-03-06T07:25:24.106251+0000","last_undegraded":"2024-03-06T07:25:24.106251+0000","last_fullsized":"2024-03-06T07:25:24.106251+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:38.207630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105986+0000","last_change":"2024-03-06T07:25:24.101492+0000","last_active":"2024-03-06T07:25:24.105986+0000","last_peered":"2024-03-06T07:25:24.105986+0000","last_clean":"2024-03-06T07:25:24.105986+0000","last_became_active":"2024-03-06T07:25:24.101229+0000","last_became_peered":"2024-03-06T07:25:24.101229+0000","last_unstale":"2024-03-06T07:25:24.105986+0000","last_undegraded":"2024-03-06T07:25:24.105986+0000","last_fullsized":"2024-03-06T07:25:24.105986+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.849922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.100940+0000","last_change":"2024-03-06T07:25:24.097551+0000","last_active":"2024-03-06T07:25:24.100940+0000","last_peered":"2024-03-06T07:25:24.100940+0000","last_clean":"2024-03-06T07:25:24.100940+0000","last_became_active":"2024-03-06T07:25:24.097292+0000","last_became_peered":"2024-03-06T07:25:24.097292+0000","last_unstale":"2024-03-06T07:25:24.100940+0000","last_undegraded":"2024-03-06T07:25:24.100940+0000","last_fullsized":"2024-03-06T07:25:24.100940+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:10.916532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108264+0000","last_change":"2024-03-06T07:25:24.105621+0000","last_active":"2024-03-06T07:25:24.108264+0000","last_peered":"2024-03-06T07:25:24.108264+0000","last_clean":"2024-03-06T07:25:24.108264+0000","last_became_active":"2024-03-06T07:25:24.105314+0000","last_became_peered":"2024-03-06T07:25:24.105314+0000","last_unstale":"2024-03-06T07:25:24.108264+0000","last_undegraded":"2024-03-06T07:25:24.108264+0000","last_fullsized":"2024-03-06T07:25:24.108264+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:25.257203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102004+0000","last_change":"2024-03-06T07:25:24.098643+0000","last_active":"2024-03-06T07:25:24.102004+0000","last_peered":"2024-03-06T07:25:24.102004+0000","last_clean":"2024-03-06T07:25:24.102004+0000","last_became_active":"2024-03-06T07:25:24.098367+0000","last_became_peered":"2024-03-06T07:25:24.098367+0000","last_unstale":"2024-03-06T07:25:24.102004+0000","last_undegraded":"2024-03-06T07:25:24.102004+0000","last_fullsized":"2024-03-06T07:25:24.102004+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:54.967926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109654+0000","last_change":"2024-03-06T07:25:24.098710+0000","last_active":"2024-03-06T07:25:24.109654+0000","last_peered":"2024-03-06T07:25:24.109654+0000","last_clean":"2024-03-06T07:25:24.109654+0000","last_became_active":"2024-03-06T07:25:24.096601+0000","last_became_peered":"2024-03-06T07:25:24.096601+0000","last_unstale":"2024-03-06T07:25:24.109654+0000","last_undegraded":"2024-03-06T07:25:24.109654+0000","last_fullsized":"2024-03-06T07:25:24.109654+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:05.660880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109866+0000","last_change":"2024-03-06T07:25:24.107802+0000","last_active":"2024-03-06T07:25:24.109866+0000","last_peered":"2024-03-06T07:25:24.109866+0000","last_clean":"2024-03-06T07:25:24.109866+0000","last_became_active":"2024-03-06T07:25:24.107528+0000","last_became_peered":"2024-03-06T07:25:24.107528+0000","last_unstale":"2024-03-06T07:25:24.109866+0000","last_undegraded":"2024-03-06T07:25:24.109866+0000","last_fullsized":"2024-03-06T07:25:24.109866+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:59.468984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108044+0000","last_change":"2024-03-06T07:25:24.104229+0000","last_active":"2024-03-06T07:25:24.108044+0000","last_peered":"2024-03-06T07:25:24.108044+0000","last_clean":"2024-03-06T07:25:24.108044+0000","last_became_active":"2024-03-06T07:25:24.103948+0000","last_became_peered":"2024-03-06T07:25:24.103948+0000","last_unstale":"2024-03-06T07:25:24.108044+0000","last_undegraded":"2024-03-06T07:25:24.108044+0000","last_fullsized":"2024-03-06T07:25:24.108044+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:49.495934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109622+0000","last_change":"2024-03-06T07:25:24.099289+0000","last_active":"2024-03-06T07:25:24.109622+0000","last_peered":"2024-03-06T07:25:24.109622+0000","last_clean":"2024-03-06T07:25:24.109622+0000","last_became_active":"2024-03-06T07:25:24.099046+0000","last_became_peered":"2024-03-06T07:25:24.099046+0000","last_unstale":"2024-03-06T07:25:24.109622+0000","last_undegraded":"2024-03-06T07:25:24.109622+0000","last_fullsized":"2024-03-06T07:25:24.109622+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:54.399281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108336+0000","last_change":"2024-03-06T07:25:24.104074+0000","last_active":"2024-03-06T07:25:24.108336+0000","last_peered":"2024-03-06T07:25:24.108336+0000","last_clean":"2024-03-06T07:25:24.108336+0000","last_became_active":"2024-03-06T07:25:24.103786+0000","last_became_peered":"2024-03-06T07:25:24.103786+0000","last_unstale":"2024-03-06T07:25:24.108336+0000","last_undegraded":"2024-03-06T07:25:24.108336+0000","last_fullsized":"2024-03-06T07:25:24.108336+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:33.972181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109690+0000","last_change":"2024-03-06T07:25:24.101550+0000","last_active":"2024-03-06T07:25:24.109690+0000","last_peered":"2024-03-06T07:25:24.109690+0000","last_clean":"2024-03-06T07:25:24.109690+0000","last_became_active":"2024-03-06T07:25:24.101274+0000","last_became_peered":"2024-03-06T07:25:24.101274+0000","last_unstale":"2024-03-06T07:25:24.109690+0000","last_undegraded":"2024-03-06T07:25:24.109690+0000","last_fullsized":"2024-03-06T07:25:24.109690+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:40.847577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112554+0000","last_change":"2024-03-06T07:25:24.110665+0000","last_active":"2024-03-06T07:25:24.112554+0000","last_peered":"2024-03-06T07:25:24.112554+0000","last_clean":"2024-03-06T07:25:24.112554+0000","last_became_active":"2024-03-06T07:25:24.110411+0000","last_became_peered":"2024-03-06T07:25:24.110411+0000","last_unstale":"2024-03-06T07:25:24.112554+0000","last_undegraded":"2024-03-06T07:25:24.112554+0000","last_fullsized":"2024-03-06T07:25:24.112554+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:51.284772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.103124+0000","last_change":"2024-03-06T07:25:24.098256+0000","last_active":"2024-03-06T07:25:24.103124+0000","last_peered":"2024-03-06T07:25:24.103124+0000","last_clean":"2024-03-06T07:25:24.103124+0000","last_became_active":"2024-03-06T07:25:24.097860+0000","last_became_peered":"2024-03-06T07:25:24.097860+0000","last_unstale":"2024-03-06T07:25:24.103124+0000","last_undegraded":"2024-03-06T07:25:24.103124+0000","last_fullsized":"2024-03-06T07:25:24.103124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:49.573382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.113207+0000","last_change":"2024-03-06T07:25:24.109408+0000","last_active":"2024-03-06T07:25:24.113207+0000","last_peered":"2024-03-06T07:25:24.113207+0000","last_clean":"2024-03-06T07:25:24.113207+0000","last_became_active":"2024-03-06T07:25:24.109123+0000","last_became_peered":"2024-03-06T07:25:24.109123+0000","last_unstale":"2024-03-06T07:25:24.113207+0000","last_undegraded":"2024-03-06T07:25:24.113207+0000","last_fullsized":"2024-03-06T07:25:24.113207+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:09.186893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105988+0000","last_change":"2024-03-06T07:25:24.099799+0000","last_active":"2024-03-06T07:25:24.105988+0000","last_peered":"2024-03-06T07:25:24.105988+0000","last_clean":"2024-03-06T07:25:24.105988+0000","last_became_active":"2024-03-06T07:25:24.099492+0000","last_became_peered":"2024-03-06T07:25:24.099492+0000","last_unstale":"2024-03-06T07:25:24.105988+0000","last_undegraded":"2024-03-06T07:25:24.105988+0000","last_fullsized":"2024-03-06T07:25:24.105988+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:54.967926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102885+0000","last_change":"2024-03-06T07:25:24.101674+0000","last_active":"2024-03-06T07:25:24.102885+0000","last_peered":"2024-03-06T07:25:24.102885+0000","last_clean":"2024-03-06T07:25:24.102885+0000","last_became_active":"2024-03-06T07:25:24.101369+0000","last_became_peered":"2024-03-06T07:25:24.101369+0000","last_unstale":"2024-03-06T07:25:24.102885+0000","last_undegraded":"2024-03-06T07:25:24.102885+0000","last_fullsized":"2024-03-06T07:25:24.102885+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:59.886631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108046+0000","last_change":"2024-03-06T07:25:24.104390+0000","last_active":"2024-03-06T07:25:24.108046+0000","last_peered":"2024-03-06T07:25:24.108046+0000","last_clean":"2024-03-06T07:25:24.108046+0000","last_became_active":"2024-03-06T07:25:24.104106+0000","last_became_peered":"2024-03-06T07:25:24.104106+0000","last_unstale":"2024-03-06T07:25:24.108046+0000","last_undegraded":"2024-03-06T07:25:24.108046+0000","last_fullsized":"2024-03-06T07:25:24.108046+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:54.056108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109696+0000","last_change":"2024-03-06T07:25:24.100977+0000","last_active":"2024-03-06T07:25:24.109696+0000","last_peered":"2024-03-06T07:25:24.109696+0000","last_clean":"2024-03-06T07:25:24.109696+0000","last_became_active":"2024-03-06T07:25:24.100193+0000","last_became_peered":"2024-03-06T07:25:24.100193+0000","last_unstale":"2024-03-06T07:25:24.109696+0000","last_undegraded":"2024-03-06T07:25:24.109696+0000","last_fullsized":"2024-03-06T07:25:24.109696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:56.585961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109742+0000","last_change":"2024-03-06T07:25:24.106056+0000","last_active":"2024-03-06T07:25:24.109742+0000","last_peered":"2024-03-06T07:25:24.109742+0000","last_clean":"2024-03-06T07:25:24.109742+0000","last_became_active":"2024-03-06T07:25:24.105318+0000","last_became_peered":"2024-03-06T07:25:24.105318+0000","last_unstale":"2024-03-06T07:25:24.109742+0000","last_undegraded":"2024-03-06T07:25:24.109742+0000","last_fullsized":"2024-03-06T07:25:24.109742+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:09.004465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.113234+0000","last_change":"2024-03-06T07:25:24.109288+0000","last_active":"2024-03-06T07:25:24.113234+0000","last_peered":"2024-03-06T07:25:24.113234+0000","last_clean":"2024-03-06T07:25:24.113234+0000","last_became_active":"2024-03-06T07:25:24.108947+0000","last_became_peered":"2024-03-06T07:25:24.108947+0000","last_unstale":"2024-03-06T07:25:24.113234+0000","last_undegraded":"2024-03-06T07:25:24.113234+0000","last_fullsized":"2024-03-06T07:25:24.113234+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:27.300274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111036+0000","last_change":"2024-03-06T07:25:24.106381+0000","last_active":"2024-03-06T07:25:24.111036+0000","last_peered":"2024-03-06T07:25:24.111036+0000","last_clean":"2024-03-06T07:25:24.111036+0000","last_became_active":"2024-03-06T07:25:24.106001+0000","last_became_peered":"2024-03-06T07:25:24.106001+0000","last_unstale":"2024-03-06T07:25:24.111036+0000","last_undegraded":"2024-03-06T07:25:24.111036+0000","last_fullsized":"2024-03-06T07:25:24.111036+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:59.468984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108285+0000","last_change":"2024-03-06T07:25:24.102962+0000","last_active":"2024-03-06T07:25:24.108285+0000","last_peered":"2024-03-06T07:25:24.108285+0000","last_clean":"2024-03-06T07:25:24.108285+0000","last_became_active":"2024-03-06T07:25:24.102311+0000","last_became_peered":"2024-03-06T07:25:24.102311+0000","last_unstale":"2024-03-06T07:25:24.108285+0000","last_undegraded":"2024-03-06T07:25:24.108285+0000","last_fullsized":"2024-03-06T07:25:24.108285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:09.004465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.121726+0000","last_change":"2024-03-06T07:25:23.092130+0000","last_active":"2024-03-06T07:25:24.121726+0000","last_peered":"2024-03-06T07:25:24.121726+0000","last_clean":"2024-03-06T07:25:24.121726+0000","last_became_active":"2024-03-06T07:25:23.091809+0000","last_became_peered":"2024-03-06T07:25:23.091809+0000","last_unstale":"2024-03-06T07:25:24.121726+0000","last_undegraded":"2024-03-06T07:25:24.121726+0000","last_fullsized":"2024-03-06T07:25:24.121726+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:01.112680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.110307+0000","last_change":"2024-03-06T07:25:24.105491+0000","last_active":"2024-03-06T07:25:24.110307+0000","last_peered":"2024-03-06T07:25:24.110307+0000","last_clean":"2024-03-06T07:25:24.110307+0000","last_became_active":"2024-03-06T07:25:24.105174+0000","last_became_peered":"2024-03-06T07:25:24.105174+0000","last_unstale":"2024-03-06T07:25:24.110307+0000","last_undegraded":"2024-03-06T07:25:24.110307+0000","last_fullsized":"2024-03-06T07:25:24.110307+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:48.332882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.121483+0000","last_change":"2024-03-06T07:25:23.092085+0000","last_active":"2024-03-06T07:25:24.121483+0000","last_peered":"2024-03-06T07:25:24.121483+0000","last_clean":"2024-03-06T07:25:24.121483+0000","last_became_active":"2024-03-06T07:25:23.091792+0000","last_became_peered":"2024-03-06T07:25:23.091792+0000","last_unstale":"2024-03-06T07:25:24.121483+0000","last_undegraded":"2024-03-06T07:25:24.121483+0000","last_fullsized":"2024-03-06T07:25:24.121483+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:28.888672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112841+0000","last_change":"2024-03-06T07:25:24.109440+0000","last_active":"2024-03-06T07:25:24.112841+0000","last_peered":"2024-03-06T07:25:24.112841+0000","last_clean":"2024-03-06T07:25:24.112841+0000","last_became_active":"2024-03-06T07:25:24.109069+0000","last_became_peered":"2024-03-06T07:25:24.109069+0000","last_unstale":"2024-03-06T07:25:24.112841+0000","last_undegraded":"2024-03-06T07:25:24.112841+0000","last_fullsized":"2024-03-06T07:25:24.112841+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:29.895963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.118552+0000","last_change":"2024-03-06T07:25:23.091418+0000","last_active":"2024-03-06T07:25:24.118552+0000","last_peered":"2024-03-06T07:25:24.118552+0000","last_clean":"2024-03-06T07:25:24.118552+0000","last_became_active":"2024-03-06T07:25:23.091106+0000","last_became_peered":"2024-03-06T07:25:23.091106+0000","last_unstale":"2024-03-06T07:25:24.118552+0000","last_undegraded":"2024-03-06T07:25:24.118552+0000","last_fullsized":"2024-03-06T07:25:24.118552+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:36.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.110841+0000","last_change":"2024-03-06T07:25:24.106146+0000","last_active":"2024-03-06T07:25:24.110841+0000","last_peered":"2024-03-06T07:25:24.110841+0000","last_clean":"2024-03-06T07:25:24.110841+0000","last_became_active":"2024-03-06T07:25:24.105840+0000","last_became_peered":"2024-03-06T07:25:24.105840+0000","last_unstale":"2024-03-06T07:25:24.110841+0000","last_undegraded":"2024-03-06T07:25:24.110841+0000","last_fullsized":"2024-03-06T07:25:24.110841+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:03.393105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.119328+0000","last_change":"2024-03-06T07:25:23.092990+0000","last_active":"2024-03-06T07:25:24.119328+0000","last_peered":"2024-03-06T07:25:24.119328+0000","last_clean":"2024-03-06T07:25:24.119328+0000","last_became_active":"2024-03-06T07:25:23.092560+0000","last_became_peered":"2024-03-06T07:25:23.092560+0000","last_unstale":"2024-03-06T07:25:24.119328+0000","last_undegraded":"2024-03-06T07:25:24.119328+0000","last_fullsized":"2024-03-06T07:25:24.119328+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:01.112680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109655+0000","last_change":"2024-03-06T07:25:24.100892+0000","last_active":"2024-03-06T07:25:24.109655+0000","last_peered":"2024-03-06T07:25:24.109655+0000","last_clean":"2024-03-06T07:25:24.109655+0000","last_became_active":"2024-03-06T07:25:24.100291+0000","last_became_peered":"2024-03-06T07:25:24.100291+0000","last_unstale":"2024-03-06T07:25:24.109655+0000","last_undegraded":"2024-03-06T07:25:24.109655+0000","last_fullsized":"2024-03-06T07:25:24.109655+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:03.571121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.116951+0000","last_change":"2024-03-06T07:25:23.089496+0000","last_active":"2024-03-06T07:25:24.116951+0000","last_peered":"2024-03-06T07:25:24.116951+0000","last_clean":"2024-03-06T07:25:24.116951+0000","last_became_active":"2024-03-06T07:25:23.089232+0000","last_became_peered":"2024-03-06T07:25:23.089232+0000","last_unstale":"2024-03-06T07:25:24.116951+0000","last_undegraded":"2024-03-06T07:25:24.116951+0000","last_fullsized":"2024-03-06T07:25:24.116951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:06.182827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109718+0000","last_change":"2024-03-06T07:25:24.103749+0000","last_active":"2024-03-06T07:25:24.109718+0000","last_peered":"2024-03-06T07:25:24.109718+0000","last_clean":"2024-03-06T07:25:24.109718+0000","last_became_active":"2024-03-06T07:25:24.103478+0000","last_became_peered":"2024-03-06T07:25:24.103478+0000","last_unstale":"2024-03-06T07:25:24.109718+0000","last_undegraded":"2024-03-06T07:25:24.109718+0000","last_fullsized":"2024-03-06T07:25:24.109718+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:25.146984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.119128+0000","last_change":"2024-03-06T07:25:23.092760+0000","last_active":"2024-03-06T07:25:24.119128+0000","last_peered":"2024-03-06T07:25:24.119128+0000","last_clean":"2024-03-06T07:25:24.119128+0000","last_became_active":"2024-03-06T07:25:23.092498+0000","last_became_peered":"2024-03-06T07:25:23.092498+0000","last_unstale":"2024-03-06T07:25:24.119128+0000","last_undegraded":"2024-03-06T07:25:24.119128+0000","last_fullsized":"2024-03-06T07:25:24.119128+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:06.362639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.106025+0000","last_change":"2024-03-06T07:25:24.101599+0000","last_active":"2024-03-06T07:25:24.106025+0000","last_peered":"2024-03-06T07:25:24.106025+0000","last_clean":"2024-03-06T07:25:24.106025+0000","last_became_active":"2024-03-06T07:25:24.101340+0000","last_became_peered":"2024-03-06T07:25:24.101340+0000","last_unstale":"2024-03-06T07:25:24.106025+0000","last_undegraded":"2024-03-06T07:25:24.106025+0000","last_fullsized":"2024-03-06T07:25:24.106025+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:48.332882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.094502+0000","last_change":"2024-03-06T07:25:23.089745+0000","last_active":"2024-03-06T07:25:24.094502+0000","last_peered":"2024-03-06T07:25:24.094502+0000","last_clean":"2024-03-06T07:25:24.094502+0000","last_became_active":"2024-03-06T07:25:23.089434+0000","last_became_peered":"2024-03-06T07:25:23.089434+0000","last_unstale":"2024-03-06T07:25:24.094502+0000","last_undegraded":"2024-03-06T07:25:24.094502+0000","last_fullsized":"2024-03-06T07:25:24.094502+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:38.539557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109670+0000","last_change":"2024-03-06T07:25:24.102358+0000","last_active":"2024-03-06T07:25:24.109670+0000","last_peered":"2024-03-06T07:25:24.109670+0000","last_clean":"2024-03-06T07:25:24.109670+0000","last_became_active":"2024-03-06T07:25:24.102096+0000","last_became_peered":"2024-03-06T07:25:24.102096+0000","last_unstale":"2024-03-06T07:25:24.109670+0000","last_undegraded":"2024-03-06T07:25:24.109670+0000","last_fullsized":"2024-03-06T07:25:24.109670+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:25.198052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.090868+0000","last_change":"2024-03-06T07:25:23.090204+0000","last_active":"2024-03-06T07:25:24.090868+0000","last_peered":"2024-03-06T07:25:24.090868+0000","last_clean":"2024-03-06T07:25:24.090868+0000","last_became_active":"2024-03-06T07:25:23.089920+0000","last_became_peered":"2024-03-06T07:25:23.089920+0000","last_unstale":"2024-03-06T07:25:24.090868+0000","last_undegraded":"2024-03-06T07:25:24.090868+0000","last_fullsized":"2024-03-06T07:25:24.090868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:47.492015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.100975+0000","last_change":"2024-03-06T07:25:24.097780+0000","last_active":"2024-03-06T07:25:24.100975+0000","last_peered":"2024-03-06T07:25:24.100975+0000","last_clean":"2024-03-06T07:25:24.100975+0000","last_became_active":"2024-03-06T07:25:24.097512+0000","last_became_peered":"2024-03-06T07:25:24.097512+0000","last_unstale":"2024-03-06T07:25:24.100975+0000","last_undegraded":"2024-03-06T07:25:24.100975+0000","last_fullsized":"2024-03-06T07:25:24.100975+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:53.327017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.092982+0000","last_change":"2024-03-06T07:25:23.095261+0000","last_active":"2024-03-06T07:25:24.092982+0000","last_peered":"2024-03-06T07:25:24.092982+0000","last_clean":"2024-03-06T07:25:24.092982+0000","last_became_active":"2024-03-06T07:25:23.094991+0000","last_became_peered":"2024-03-06T07:25:23.094991+0000","last_unstale":"2024-03-06T07:25:24.092982+0000","last_undegraded":"2024-03-06T07:25:24.092982+0000","last_fullsized":"2024-03-06T07:25:24.092982+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:09.181395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112526+0000","last_change":"2024-03-06T07:25:24.109459+0000","last_active":"2024-03-06T07:25:24.112526+0000","last_peered":"2024-03-06T07:25:24.112526+0000","last_clean":"2024-03-06T07:25:24.112526+0000","last_became_active":"2024-03-06T07:25:24.109035+0000","last_became_peered":"2024-03-06T07:25:24.109035+0000","last_unstale":"2024-03-06T07:25:24.112526+0000","last_undegraded":"2024-03-06T07:25:24.112526+0000","last_fullsized":"2024-03-06T07:25:24.112526+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:29.895963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.090973+0000","last_change":"2024-03-06T07:25:23.080697+0000","last_active":"2024-03-06T07:25:24.090973+0000","last_peered":"2024-03-06T07:25:24.090973+0000","last_clean":"2024-03-06T07:25:24.090973+0000","last_became_active":"2024-03-06T07:25:23.080406+0000","last_became_peered":"2024-03-06T07:25:23.080406+0000","last_unstale":"2024-03-06T07:25:24.090973+0000","last_undegraded":"2024-03-06T07:25:24.090973+0000","last_fullsized":"2024-03-06T07:25:24.090973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:50.277480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108132+0000","last_change":"2024-03-06T07:25:24.093749+0000","last_active":"2024-03-06T07:25:24.108132+0000","last_peered":"2024-03-06T07:25:24.108132+0000","last_clean":"2024-03-06T07:25:24.108132+0000","last_became_active":"2024-03-06T07:25:24.093440+0000","last_became_peered":"2024-03-06T07:25:24.093440+0000","last_unstale":"2024-03-06T07:25:24.108132+0000","last_undegraded":"2024-03-06T07:25:24.108132+0000","last_fullsized":"2024-03-06T07:25:24.108132+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:44.331858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.119150+0000","last_change":"2024-03-06T07:25:23.090765+0000","last_active":"2024-03-06T07:25:24.119150+0000","last_peered":"2024-03-06T07:25:24.119150+0000","last_clean":"2024-03-06T07:25:24.119150+0000","last_became_active":"2024-03-06T07:25:23.090491+0000","last_became_peered":"2024-03-06T07:25:23.090491+0000","last_unstale":"2024-03-06T07:25:24.119150+0000","last_undegraded":"2024-03-06T07:25:24.119150+0000","last_fullsized":"2024-03-06T07:25:24.119150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:53.960634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112490+0000","last_change":"2024-03-06T07:25:24.110068+0000","last_active":"2024-03-06T07:25:24.112490+0000","last_peered":"2024-03-06T07:25:24.112490+0000","last_clean":"2024-03-06T07:25:24.112490+0000","last_became_active":"2024-03-06T07:25:24.109756+0000","last_became_peered":"2024-03-06T07:25:24.109756+0000","last_unstale":"2024-03-06T07:25:24.112490+0000","last_undegraded":"2024-03-06T07:25:24.112490+0000","last_fullsized":"2024-03-06T07:25:24.112490+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:19.248382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.123984+0000","last_change":"2024-03-06T07:25:23.097336+0000","last_active":"2024-03-06T07:25:24.123984+0000","last_peered":"2024-03-06T07:25:24.123984+0000","last_clean":"2024-03-06T07:25:24.123984+0000","last_became_active":"2024-03-06T07:25:23.097084+0000","last_became_peered":"2024-03-06T07:25:23.097084+0000","last_unstale":"2024-03-06T07:25:24.123984+0000","last_undegraded":"2024-03-06T07:25:24.123984+0000","last_fullsized":"2024-03-06T07:25:24.123984+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:27.972802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111304+0000","last_change":"2024-03-06T07:25:24.110168+0000","last_active":"2024-03-06T07:25:24.111304+0000","last_peered":"2024-03-06T07:25:24.111304+0000","last_clean":"2024-03-06T07:25:24.111304+0000","last_became_active":"2024-03-06T07:25:24.109768+0000","last_became_peered":"2024-03-06T07:25:24.109768+0000","last_unstale":"2024-03-06T07:25:24.111304+0000","last_undegraded":"2024-03-06T07:25:24.111304+0000","last_fullsized":"2024-03-06T07:25:24.111304+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:59.468984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.118223+0000","last_change":"2024-03-06T07:25:23.083712+0000","last_active":"2024-03-06T07:25:24.118223+0000","last_peered":"2024-03-06T07:25:24.118223+0000","last_clean":"2024-03-06T07:25:24.118223+0000","last_became_active":"2024-03-06T07:25:23.083477+0000","last_became_peered":"2024-03-06T07:25:23.083477+0000","last_unstale":"2024-03-06T07:25:24.118223+0000","last_undegraded":"2024-03-06T07:25:24.118223+0000","last_fullsized":"2024-03-06T07:25:24.118223+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:34.798242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108358+0000","last_change":"2024-03-06T07:25:24.104682+0000","last_active":"2024-03-06T07:25:24.108358+0000","last_peered":"2024-03-06T07:25:24.108358+0000","last_clean":"2024-03-06T07:25:24.108358+0000","last_became_active":"2024-03-06T07:25:24.104390+0000","last_became_peered":"2024-03-06T07:25:24.104390+0000","last_unstale":"2024-03-06T07:25:24.108358+0000","last_undegraded":"2024-03-06T07:25:24.108358+0000","last_fullsized":"2024-03-06T07:25:24.108358+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:59.886631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.093325+0000","last_change":"2024-03-06T07:25:23.087632+0000","last_active":"2024-03-06T07:25:24.093325+0000","last_peered":"2024-03-06T07:25:24.093325+0000","last_clean":"2024-03-06T07:25:24.093325+0000","last_became_active":"2024-03-06T07:25:23.087259+0000","last_became_peered":"2024-03-06T07:25:23.087259+0000","last_unstale":"2024-03-06T07:25:24.093325+0000","last_undegraded":"2024-03-06T07:25:24.093325+0000","last_fullsized":"2024-03-06T07:25:24.093325+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:25.569705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105773+0000","last_change":"2024-03-06T07:25:24.101290+0000","last_active":"2024-03-06T07:25:24.105773+0000","last_peered":"2024-03-06T07:25:24.105773+0000","last_clean":"2024-03-06T07:25:24.105773+0000","last_became_active":"2024-03-06T07:25:24.101034+0000","last_became_peered":"2024-03-06T07:25:24.101034+0000","last_unstale":"2024-03-06T07:25:24.105773+0000","last_undegraded":"2024-03-06T07:25:24.105773+0000","last_fullsized":"2024-03-06T07:25:24.105773+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:20.980764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.094692+0000","last_change":"2024-03-06T07:25:23.092145+0000","last_active":"2024-03-06T07:25:24.094692+0000","last_peered":"2024-03-06T07:25:24.094692+0000","last_clean":"2024-03-06T07:25:24.094692+0000","last_became_active":"2024-03-06T07:25:23.091895+0000","last_became_peered":"2024-03-06T07:25:23.091895+0000","last_unstale":"2024-03-06T07:25:24.094692+0000","last_undegraded":"2024-03-06T07:25:24.094692+0000","last_fullsized":"2024-03-06T07:25:24.094692+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:36.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112573+0000","last_change":"2024-03-06T07:25:24.110797+0000","last_active":"2024-03-06T07:25:24.112573+0000","last_peered":"2024-03-06T07:25:24.112573+0000","last_clean":"2024-03-06T07:25:24.112573+0000","last_became_active":"2024-03-06T07:25:24.110499+0000","last_became_peered":"2024-03-06T07:25:24.110499+0000","last_unstale":"2024-03-06T07:25:24.112573+0000","last_undegraded":"2024-03-06T07:25:24.112573+0000","last_fullsized":"2024-03-06T07:25:24.112573+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:35.805534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.121646+0000","last_change":"2024-03-06T07:25:23.092882+0000","last_active":"2024-03-06T07:25:24.121646+0000","last_peered":"2024-03-06T07:25:24.121646+0000","last_clean":"2024-03-06T07:25:24.121646+0000","last_became_active":"2024-03-06T07:25:23.092629+0000","last_became_peered":"2024-03-06T07:25:23.092629+0000","last_unstale":"2024-03-06T07:25:24.121646+0000","last_undegraded":"2024-03-06T07:25:24.121646+0000","last_fullsized":"2024-03-06T07:25:24.121646+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:43.500066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109845+0000","last_change":"2024-03-06T07:25:24.106360+0000","last_active":"2024-03-06T07:25:24.109845+0000","last_peered":"2024-03-06T07:25:24.109845+0000","last_clean":"2024-03-06T07:25:24.109845+0000","last_became_active":"2024-03-06T07:25:24.106060+0000","last_became_peered":"2024-03-06T07:25:24.106060+0000","last_unstale":"2024-03-06T07:25:24.109845+0000","last_undegraded":"2024-03-06T07:25:24.109845+0000","last_fullsized":"2024-03-06T07:25:24.109845+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:03.393105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.117514+0000","last_change":"2024-03-06T07:25:23.083039+0000","last_active":"2024-03-06T07:25:24.117514+0000","last_peered":"2024-03-06T07:25:24.117514+0000","last_clean":"2024-03-06T07:25:24.117514+0000","last_became_active":"2024-03-06T07:25:23.082770+0000","last_became_peered":"2024-03-06T07:25:23.082770+0000","last_unstale":"2024-03-06T07:25:24.117514+0000","last_undegraded":"2024-03-06T07:25:24.117514+0000","last_fullsized":"2024-03-06T07:25:24.117514+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:18.241091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111300+0000","last_change":"2024-03-06T07:25:24.109619+0000","last_active":"2024-03-06T07:25:24.111300+0000","last_peered":"2024-03-06T07:25:24.111300+0000","last_clean":"2024-03-06T07:25:24.111300+0000","last_became_active":"2024-03-06T07:25:24.108824+0000","last_became_peered":"2024-03-06T07:25:24.108824+0000","last_unstale":"2024-03-06T07:25:24.111300+0000","last_undegraded":"2024-03-06T07:25:24.111300+0000","last_fullsized":"2024-03-06T07:25:24.111300+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:03.393105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.118271+0000","last_change":"2024-03-06T07:25:23.089357+0000","last_active":"2024-03-06T07:25:24.118271+0000","last_peered":"2024-03-06T07:25:24.118271+0000","last_clean":"2024-03-06T07:25:24.118271+0000","last_became_active":"2024-03-06T07:25:23.089045+0000","last_became_peered":"2024-03-06T07:25:23.089045+0000","last_unstale":"2024-03-06T07:25:24.118271+0000","last_undegraded":"2024-03-06T07:25:24.118271+0000","last_fullsized":"2024-03-06T07:25:24.118271+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:58.487125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109598+0000","last_change":"2024-03-06T07:25:24.097976+0000","last_active":"2024-03-06T07:25:24.109598+0000","last_peered":"2024-03-06T07:25:24.109598+0000","last_clean":"2024-03-06T07:25:24.109598+0000","last_became_active":"2024-03-06T07:25:24.097074+0000","last_became_peered":"2024-03-06T07:25:24.097074+0000","last_unstale":"2024-03-06T07:25:24.109598+0000","last_undegraded":"2024-03-06T07:25:24.109598+0000","last_fullsized":"2024-03-06T07:25:24.109598+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:41.069951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.116703+0000","last_change":"2024-03-06T07:25:23.092189+0000","last_active":"2024-03-06T07:25:24.116703+0000","last_peered":"2024-03-06T07:25:24.116703+0000","last_clean":"2024-03-06T07:25:24.116703+0000","last_became_active":"2024-03-06T07:25:23.091901+0000","last_became_peered":"2024-03-06T07:25:23.091901+0000","last_unstale":"2024-03-06T07:25:24.116703+0000","last_undegraded":"2024-03-06T07:25:24.116703+0000","last_fullsized":"2024-03-06T07:25:24.116703+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:58.487125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112458+0000","last_change":"2024-03-06T07:25:24.109266+0000","last_active":"2024-03-06T07:25:24.112458+0000","last_peered":"2024-03-06T07:25:24.112458+0000","last_clean":"2024-03-06T07:25:24.112458+0000","last_became_active":"2024-03-06T07:25:24.108995+0000","last_became_peered":"2024-03-06T07:25:24.108995+0000","last_unstale":"2024-03-06T07:25:24.112458+0000","last_undegraded":"2024-03-06T07:25:24.112458+0000","last_fullsized":"2024-03-06T07:25:24.112458+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:26.022393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.091506+0000","last_change":"2024-03-06T07:25:23.095670+0000","last_active":"2024-03-06T07:25:24.091506+0000","last_peered":"2024-03-06T07:25:24.091506+0000","last_clean":"2024-03-06T07:25:24.091506+0000","last_became_active":"2024-03-06T07:25:23.094714+0000","last_became_peered":"2024-03-06T07:25:23.094714+0000","last_unstale":"2024-03-06T07:25:24.091506+0000","last_undegraded":"2024-03-06T07:25:24.091506+0000","last_fullsized":"2024-03-06T07:25:24.091506+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:38.539557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111259+0000","last_change":"2024-03-06T07:25:24.109297+0000","last_active":"2024-03-06T07:25:24.111259+0000","last_peered":"2024-03-06T07:25:24.111259+0000","last_clean":"2024-03-06T07:25:24.111259+0000","last_became_active":"2024-03-06T07:25:24.109042+0000","last_became_peered":"2024-03-06T07:25:24.109042+0000","last_unstale":"2024-03-06T07:25:24.111259+0000","last_undegraded":"2024-03-06T07:25:24.111259+0000","last_fullsized":"2024-03-06T07:25:24.111259+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:09.004465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.121645+0000","last_change":"2024-03-06T07:25:23.086139+0000","last_active":"2024-03-06T07:25:24.121645+0000","last_peered":"2024-03-06T07:25:24.121645+0000","last_clean":"2024-03-06T07:25:24.121645+0000","last_became_active":"2024-03-06T07:25:23.085864+0000","last_became_peered":"2024-03-06T07:25:23.085864+0000","last_unstale":"2024-03-06T07:25:24.121645+0000","last_undegraded":"2024-03-06T07:25:24.121645+0000","last_fullsized":"2024-03-06T07:25:24.121645+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:52.216630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102684+0000","last_change":"2024-03-06T07:25:24.099480+0000","last_active":"2024-03-06T07:25:24.102684+0000","last_peered":"2024-03-06T07:25:24.102684+0000","last_clean":"2024-03-06T07:25:24.102684+0000","last_became_active":"2024-03-06T07:25:24.099188+0000","last_became_peered":"2024-03-06T07:25:24.099188+0000","last_unstale":"2024-03-06T07:25:24.102684+0000","last_undegraded":"2024-03-06T07:25:24.102684+0000","last_fullsized":"2024-03-06T07:25:24.102684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:33.972181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.087420+0000","last_change":"2024-03-06T07:25:23.090129+0000","last_active":"2024-03-06T07:25:24.087420+0000","last_peered":"2024-03-06T07:25:24.087420+0000","last_clean":"2024-03-06T07:25:24.087420+0000","last_became_active":"2024-03-06T07:25:23.089825+0000","last_became_peered":"2024-03-06T07:25:23.089825+0000","last_unstale":"2024-03-06T07:25:24.087420+0000","last_undegraded":"2024-03-06T07:25:24.087420+0000","last_fullsized":"2024-03-06T07:25:24.087420+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:45.846706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102741+0000","last_change":"2024-03-06T07:25:24.101847+0000","last_active":"2024-03-06T07:25:24.102741+0000","last_peered":"2024-03-06T07:25:24.102741+0000","last_clean":"2024-03-06T07:25:24.102741+0000","last_became_active":"2024-03-06T07:25:24.101554+0000","last_became_peered":"2024-03-06T07:25:24.101554+0000","last_unstale":"2024-03-06T07:25:24.102741+0000","last_undegraded":"2024-03-06T07:25:24.102741+0000","last_fullsized":"2024-03-06T07:25:24.102741+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:38.207630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.093659+0000","last_change":"2024-03-06T07:25:23.093613+0000","last_active":"2024-03-06T07:25:24.093659+0000","last_peered":"2024-03-06T07:25:24.093659+0000","last_clean":"2024-03-06T07:25:24.093659+0000","last_became_active":"2024-03-06T07:25:23.093098+0000","last_became_peered":"2024-03-06T07:25:23.093098+0000","last_unstale":"2024-03-06T07:25:24.093659+0000","last_undegraded":"2024-03-06T07:25:24.093659+0000","last_fullsized":"2024-03-06T07:25:24.093659+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:06.362639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.106018+0000","last_change":"2024-03-06T07:25:24.100746+0000","last_active":"2024-03-06T07:25:24.106018+0000","last_peered":"2024-03-06T07:25:24.106018+0000","last_clean":"2024-03-06T07:25:24.106018+0000","last_became_active":"2024-03-06T07:25:24.100485+0000","last_became_peered":"2024-03-06T07:25:24.100485+0000","last_unstale":"2024-03-06T07:25:24.106018+0000","last_undegraded":"2024-03-06T07:25:24.106018+0000","last_fullsized":"2024-03-06T07:25:24.106018+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:33.972181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.118617+0000","last_change":"2024-03-06T07:25:23.092359+0000","last_active":"2024-03-06T07:25:24.118617+0000","last_peered":"2024-03-06T07:25:24.118617+0000","last_clean":"2024-03-06T07:25:24.118617+0000","last_became_active":"2024-03-06T07:25:23.092081+0000","last_became_peered":"2024-03-06T07:25:23.092081+0000","last_unstale":"2024-03-06T07:25:24.118617+0000","last_undegraded":"2024-03-06T07:25:24.118617+0000","last_fullsized":"2024-03-06T07:25:24.118617+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:57.095516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105927+0000","last_change":"2024-03-06T07:25:24.101437+0000","last_active":"2024-03-06T07:25:24.105927+0000","last_peered":"2024-03-06T07:25:24.105927+0000","last_clean":"2024-03-06T07:25:24.105927+0000","last_became_active":"2024-03-06T07:25:24.101184+0000","last_became_peered":"2024-03-06T07:25:24.101184+0000","last_unstale":"2024-03-06T07:25:24.105927+0000","last_undegraded":"2024-03-06T07:25:24.105927+0000","last_fullsized":"2024-03-06T07:25:24.105927+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:59.886631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.095076+0000","last_change":"2024-03-06T07:25:23.091258+0000","last_active":"2024-03-06T07:25:24.095076+0000","last_peered":"2024-03-06T07:25:24.095076+0000","last_clean":"2024-03-06T07:25:24.095076+0000","last_became_active":"2024-03-06T07:25:23.090379+0000","last_became_peered":"2024-03-06T07:25:23.090379+0000","last_unstale":"2024-03-06T07:25:24.095076+0000","last_undegraded":"2024-03-06T07:25:24.095076+0000","last_fullsized":"2024-03-06T07:25:24.095076+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:09.181395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108109+0000","last_change":"2024-03-06T07:25:24.092988+0000","last_active":"2024-03-06T07:25:24.108109+0000","last_peered":"2024-03-06T07:25:24.108109+0000","last_clean":"2024-03-06T07:25:24.108109+0000","last_became_active":"2024-03-06T07:25:24.092717+0000","last_became_peered":"2024-03-06T07:25:24.092717+0000","last_unstale":"2024-03-06T07:25:24.108109+0000","last_undegraded":"2024-03-06T07:25:24.108109+0000","last_fullsized":"2024-03-06T07:25:24.108109+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:26.576997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.093493+0000","last_change":"2024-03-06T07:25:23.085559+0000","last_active":"2024-03-06T07:25:24.093493+0000","last_peered":"2024-03-06T07:25:24.093493+0000","last_clean":"2024-03-06T07:25:24.093493+0000","last_became_active":"2024-03-06T07:25:23.085283+0000","last_became_peered":"2024-03-06T07:25:23.085283+0000","last_unstale":"2024-03-06T07:25:24.093493+0000","last_undegraded":"2024-03-06T07:25:24.093493+0000","last_fullsized":"2024-03-06T07:25:24.093493+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:43.324566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111069+0000","last_change":"2024-03-06T07:25:24.106961+0000","last_active":"2024-03-06T07:25:24.111069+0000","last_peered":"2024-03-06T07:25:24.111069+0000","last_clean":"2024-03-06T07:25:24.111069+0000","last_became_active":"2024-03-06T07:25:24.106668+0000","last_became_peered":"2024-03-06T07:25:24.106668+0000","last_unstale":"2024-03-06T07:25:24.111069+0000","last_undegraded":"2024-03-06T07:25:24.111069+0000","last_fullsized":"2024-03-06T07:25:24.111069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:26.022393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.095064+0000","last_change":"2024-03-06T07:25:23.091449+0000","last_active":"2024-03-06T07:25:24.095064+0000","last_peered":"2024-03-06T07:25:24.095064+0000","last_clean":"2024-03-06T07:25:24.095064+0000","last_became_active":"2024-03-06T07:25:23.091167+0000","last_became_peered":"2024-03-06T07:25:23.091167+0000","last_unstale":"2024-03-06T07:25:24.095064+0000","last_undegraded":"2024-03-06T07:25:24.095064+0000","last_fullsized":"2024-03-06T07:25:24.095064+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:09.181395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102972+0000","last_change":"2024-03-06T07:25:24.096010+0000","last_active":"2024-03-06T07:25:24.102972+0000","last_peered":"2024-03-06T07:25:24.102972+0000","last_clean":"2024-03-06T07:25:24.102972+0000","last_became_active":"2024-03-06T07:25:24.095639+0000","last_became_peered":"2024-03-06T07:25:24.095639+0000","last_unstale":"2024-03-06T07:25:24.102972+0000","last_undegraded":"2024-03-06T07:25:24.102972+0000","last_fullsized":"2024-03-06T07:25:24.102972+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:35.404387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.087471+0000","last_change":"2024-03-06T07:25:23.089829+0000","last_active":"2024-03-06T07:25:24.087471+0000","last_peered":"2024-03-06T07:25:24.087471+0000","last_clean":"2024-03-06T07:25:24.087471+0000","last_became_active":"2024-03-06T07:25:23.089568+0000","last_became_peered":"2024-03-06T07:25:23.089568+0000","last_unstale":"2024-03-06T07:25:24.087471+0000","last_undegraded":"2024-03-06T07:25:24.087471+0000","last_fullsized":"2024-03-06T07:25:24.087471+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:37.442485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112571+0000","last_change":"2024-03-06T07:25:24.110889+0000","last_active":"2024-03-06T07:25:24.112571+0000","last_peered":"2024-03-06T07:25:24.112571+0000","last_clean":"2024-03-06T07:25:24.112571+0000","last_became_active":"2024-03-06T07:25:24.110465+0000","last_became_peered":"2024-03-06T07:25:24.110465+0000","last_unstale":"2024-03-06T07:25:24.112571+0000","last_undegraded":"2024-03-06T07:25:24.112571+0000","last_fullsized":"2024-03-06T07:25:24.112571+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:53.223922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.090718+0000","last_change":"2024-03-06T07:25:23.100020+0000","last_active":"2024-03-06T07:25:24.090718+0000","last_peered":"2024-03-06T07:25:24.090718+0000","last_clean":"2024-03-06T07:25:24.090718+0000","last_became_active":"2024-03-06T07:25:23.099744+0000","last_became_peered":"2024-03-06T07:25:23.099744+0000","last_unstale":"2024-03-06T07:25:24.090718+0000","last_undegraded":"2024-03-06T07:25:24.090718+0000","last_fullsized":"2024-03-06T07:25:24.090718+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:56.038731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108239+0000","last_change":"2024-03-06T07:25:24.095978+0000","last_active":"2024-03-06T07:25:24.108239+0000","last_peered":"2024-03-06T07:25:24.108239+0000","last_clean":"2024-03-06T07:25:24.108239+0000","last_became_active":"2024-03-06T07:25:24.095676+0000","last_became_peered":"2024-03-06T07:25:24.095676+0000","last_unstale":"2024-03-06T07:25:24.108239+0000","last_undegraded":"2024-03-06T07:25:24.108239+0000","last_fullsized":"2024-03-06T07:25:24.108239+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:40.088432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.087595+0000","last_change":"2024-03-06T07:25:23.092771+0000","last_active":"2024-03-06T07:25:24.087595+0000","last_peered":"2024-03-06T07:25:24.087595+0000","last_clean":"2024-03-06T07:25:24.087595+0000","last_became_active":"2024-03-06T07:25:23.091105+0000","last_became_peered":"2024-03-06T07:25:23.091105+0000","last_unstale":"2024-03-06T07:25:24.087595+0000","last_undegraded":"2024-03-06T07:25:24.087595+0000","last_fullsized":"2024-03-06T07:25:24.087595+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:05.302915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109647+0000","last_change":"2024-03-06T07:25:24.101852+0000","last_active":"2024-03-06T07:25:24.109647+0000","last_peered":"2024-03-06T07:25:24.109647+0000","last_clean":"2024-03-06T07:25:24.109647+0000","last_became_active":"2024-03-06T07:25:24.101461+0000","last_became_peered":"2024-03-06T07:25:24.101461+0000","last_unstale":"2024-03-06T07:25:24.109647+0000","last_undegraded":"2024-03-06T07:25:24.109647+0000","last_fullsized":"2024-03-06T07:25:24.109647+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:15.817717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.090640+0000","last_change":"2024-03-06T07:25:23.092253+0000","last_active":"2024-03-06T07:25:24.090640+0000","last_peered":"2024-03-06T07:25:24.090640+0000","last_clean":"2024-03-06T07:25:24.090640+0000","last_became_active":"2024-03-06T07:25:23.091929+0000","last_became_peered":"2024-03-06T07:25:23.091929+0000","last_unstale":"2024-03-06T07:25:24.090640+0000","last_undegraded":"2024-03-06T07:25:24.090640+0000","last_fullsized":"2024-03-06T07:25:24.090640+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:25.015102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111171+0000","last_change":"2024-03-06T07:25:24.099930+0000","last_active":"2024-03-06T07:25:24.111171+0000","last_peered":"2024-03-06T07:25:24.111171+0000","last_clean":"2024-03-06T07:25:24.111171+0000","last_became_active":"2024-03-06T07:25:24.099666+0000","last_became_peered":"2024-03-06T07:25:24.099666+0000","last_unstale":"2024-03-06T07:25:24.111171+0000","last_undegraded":"2024-03-06T07:25:24.111171+0000","last_fullsized":"2024-03-06T07:25:24.111171+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:27.533755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.094638+0000","last_change":"2024-03-06T07:25:23.089208+0000","last_active":"2024-03-06T07:25:24.094638+0000","last_peered":"2024-03-06T07:25:24.094638+0000","last_clean":"2024-03-06T07:25:24.094638+0000","last_became_active":"2024-03-06T07:25:23.088914+0000","last_became_peered":"2024-03-06T07:25:23.088914+0000","last_unstale":"2024-03-06T07:25:24.094638+0000","last_undegraded":"2024-03-06T07:25:24.094638+0000","last_fullsized":"2024-03-06T07:25:24.094638+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:39.081140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.095182+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:25:24.095182+0000","last_peered":"2024-03-06T07:25:24.095182+0000","last_clean":"2024-03-06T07:25:24.095182+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:25:24.095182+0000","last_undegraded":"2024-03-06T07:25:24.095182+0000","last_fullsized":"2024-03-06T07:25:24.095182+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":872529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":684032,"data_stored":282001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":282653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":283613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":282057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":671744,"data_stored":275226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":688128,"data_stored":282091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1294336,"data_stored":874563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:28.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.283+0000 7f23dcd5e640 1 Processor -- start 2024-03-06T07:25:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.283+0000 7f23dcd5e640 1 -- start start 2024-03-06T07:25:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.284+0000 7f23dcd5e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 0x7f23d80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.284+0000 7f23dcd5e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f23d80b9a10 0x7f23d80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.284+0000 7f23dcd5e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d80cea30 con 0x7f23d815df00 2024-03-06T07:25:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.284+0000 7f23dcd5e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d80cf3c0 con 0x7f23d80b9a10 2024-03-06T07:25:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.284+0000 7f23dcd5e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d80cf0e0 con 0x7f23d80b95a0 2024-03-06T07:25:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.284+0000 7f23d5d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 0x7f23d80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.284+0000 7f23d6d9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f23d80b9a10 0x7f23d80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.284+0000 7f23d659c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f23d815df00 0x7f23d80d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:39892/0 (socket says 172.21.15.27:39892) 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.284+0000 7f23d659c640 1 -- 172.21.15.27:0/362307136 learned_addr learned my addr 172.21.15.27:0/362307136 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.285+0000 7f23d6d9d640 1 -- 172.21.15.27:0/362307136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 msgr2=0x7f23d80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.285+0000 7f23d6d9d640 1 --2- 172.21.15.27:0/362307136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 0x7f23d80b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.285+0000 7f23d6d9d640 1 -- 172.21.15.27:0/362307136 >> v1:172.21.15.69:6789/0 conn(0x7f23d815df00 legacy=0x7f23d80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.285+0000 7f23d6d9d640 1 -- 172.21.15.27:0/362307136 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d80ce470 con 0x7f23d80b9a10 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.285+0000 7f23d5d9b640 1 --2- 172.21.15.27:0/362307136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 0x7f23d80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23d6d9d640 1 --2- 172.21.15.27:0/362307136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 0x7f23d80b5c40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009d90 tx=0x7f23d002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=786120f3fff22641 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23d559a640 1 -- 172.21.15.27:0/362307136 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0039070 con 0x7f23d80b9a10 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23d559a640 1 -- 172.21.15.27:0/362307136 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d002cc50 con 0x7f23d80b9a10 2024-03-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23d559a640 1 -- 172.21.15.27:0/362307136 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0034440 con 0x7f23d80b9a10 2024-03-06T07:25:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23dcd5e640 1 -- 172.21.15.27:0/362307136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 msgr2=0x7f23d80b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23dcd5e640 1 --2- 172.21.15.27:0/362307136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 0x7f23d80b5c40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009d90 tx=0x7f23d002bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23dcd5e640 1 -- 172.21.15.27:0/362307136 shutdown_connections 2024-03-06T07:25:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23dcd5e640 1 --2- 172.21.15.27:0/362307136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 0x7f23d80b5c40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23dcd5e640 1 --2- 172.21.15.27:0/362307136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 0x7f23d80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23dcd5e640 1 -- 172.21.15.27:0/362307136 >> 172.21.15.27:0/362307136 conn(0x7f23d8071960 msgr2=0x7f23d8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23dcd5e640 1 -- 172.21.15.27:0/362307136 shutdown_connections 2024-03-06T07:25:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.286+0000 7f23dcd5e640 1 -- 172.21.15.27:0/362307136 wait complete. 2024-03-06T07:25:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.288+0000 7f23dcd5e640 1 Processor -- start 2024-03-06T07:25:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.288+0000 7f23dcd5e640 1 -- start start 2024-03-06T07:25:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.288+0000 7f23dcd5e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 0x7f23d80c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.288+0000 7f23dcd5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 0x7f23d80cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.288+0000 7f23dcd5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23d815df00 0x7f23d815f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.288+0000 7f23dcd5e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d80cedf0 con 0x7f23d815df00 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.288+0000 7f23dcd5e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d80cec70 con 0x7f23d80b9a10 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.288+0000 7f23dcd5e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d80d3220 con 0x7f23d80b95a0 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23d815df00 0x7f23d815f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23d815df00 0x7f23d815f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42132/0 (socket says 172.21.15.27:42132) 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d6d9d640 1 -- 172.21.15.27:0/2743622740 learned_addr learned my addr 172.21.15.27:0/2743622740 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d659c640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 0x7f23d80c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d5d9b640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 0x7f23d80cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d6d9d640 1 -- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 msgr2=0x7f23d80c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d6d9d640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 0x7f23d80c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d6d9d640 1 -- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 msgr2=0x7f23d80cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d6d9d640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 0x7f23d80cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d6d9d640 1 -- 172.21.15.27:0/2743622740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d80cc460 con 0x7f23d815df00 2024-03-06T07:25:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d5d9b640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 0x7f23d80cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23d6d9d640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23d815df00 0x7f23d815f8d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009d90 tx=0x7f23d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23c6ffd640 1 -- 172.21.15.27:0/2743622740 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0039050 con 0x7f23d815df00 2024-03-06T07:25:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23c6ffd640 1 -- 172.21.15.27:0/2743622740 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d0045070 con 0x7f23d815df00 2024-03-06T07:25:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d80cc740 con 0x7f23d815df00 2024-03-06T07:25:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23c6ffd640 1 -- 172.21.15.27:0/2743622740 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0009450 con 0x7f23d815df00 2024-03-06T07:25:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.289+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d80ccd00 con 0x7f23d815df00 2024-03-06T07:25:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.291+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d8067ff0 con 0x7f23d815df00 2024-03-06T07:25:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.292+0000 7f23c6ffd640 1 -- 172.21.15.27:0/2743622740 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23d003e610 con 0x7f23d815df00 2024-03-06T07:25:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.292+0000 7f23c6ffd640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23b806b4d0 0x7f23b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.292+0000 7f23c6ffd640 1 -- 172.21.15.27:0/2743622740 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23d00acb90 con 0x7f23d815df00 2024-03-06T07:25:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.292+0000 7f23d659c640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23b806b4d0 0x7f23b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.293+0000 7f23d659c640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23b806b4d0 0x7f23b806d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f23d8160590 tx=0x7f23cc007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.297+0000 7f23c6ffd640 1 -- 172.21.15.27:0/2743622740 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23d007b340 con 0x7f23d815df00 2024-03-06T07:25:28.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.474+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23d815b800 con 0x7f23b806b4d0 2024-03-06T07:25:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.480+0000 7f23c6ffd640 1 -- 172.21.15.27:0/2743622740 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251884 (secure 0 0 0) 0x7f23d815b800 con 0x7f23b806b4d0 2024-03-06T07:25:28.485 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:25:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23b806b4d0 msgr2=0x7f23b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23b806b4d0 0x7f23b806d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f23d8160590 tx=0x7f23cc007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23d815df00 msgr2=0x7f23d815f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23d815df00 0x7f23d815f8d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009d90 tx=0x7f23d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 shutdown_connections 2024-03-06T07:25:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23b806b4d0 0x7f23b806d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23d815df00 0x7f23d815f8d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23d80b9a10 0x7f23d80cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 --2- 172.21.15.27:0/2743622740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23d80b95a0 0x7f23d80c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 >> 172.21.15.27:0/2743622740 conn(0x7f23d8071960 msgr2=0x7f23d815c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 shutdown_connections 2024-03-06T07:25:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.483+0000 7f23dcd5e640 1 -- 172.21.15.27:0/2743622740 wait complete. 2024-03-06T07:25:28.496 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-03-06T07:25:28.024235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256044,"kb_used_data":6548,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718676,"statfs":{"total":773094113280,"available":772831924224,"internally_reserved":0,"allocated":6705152,"data_stored":3434733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.952713"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.115564+0000","last_change":"2024-03-06T07:25:24.110516+0000","last_active":"2024-03-06T07:25:24.115564+0000","last_peered":"2024-03-06T07:25:24.115564+0000","last_clean":"2024-03-06T07:25:24.115564+0000","last_became_active":"2024-03-06T07:25:24.110246+0000","last_became_peered":"2024-03-06T07:25:24.110246+0000","last_unstale":"2024-03-06T07:25:24.115564+0000","last_undegraded":"2024-03-06T07:25:24.115564+0000","last_fullsized":"2024-03-06T07:25:24.115564+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:17.820565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109752+0000","last_change":"2024-03-06T07:25:24.105382+0000","last_active":"2024-03-06T07:25:24.109752+0000","last_peered":"2024-03-06T07:25:24.109752+0000","last_clean":"2024-03-06T07:25:24.109752+0000","last_became_active":"2024-03-06T07:25:24.105111+0000","last_became_peered":"2024-03-06T07:25:24.105111+0000","last_unstale":"2024-03-06T07:25:24.109752+0000","last_undegraded":"2024-03-06T07:25:24.109752+0000","last_fullsized":"2024-03-06T07:25:24.109752+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:32.294062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108365+0000","last_change":"2024-03-06T07:25:24.105622+0000","last_active":"2024-03-06T07:25:24.108365+0000","last_peered":"2024-03-06T07:25:24.108365+0000","last_clean":"2024-03-06T07:25:24.108365+0000","last_became_active":"2024-03-06T07:25:24.105341+0000","last_became_peered":"2024-03-06T07:25:24.105341+0000","last_unstale":"2024-03-06T07:25:24.108365+0000","last_undegraded":"2024-03-06T07:25:24.108365+0000","last_fullsized":"2024-03-06T07:25:24.108365+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:38.207630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105969+0000","last_change":"2024-03-06T07:25:24.099774+0000","last_active":"2024-03-06T07:25:24.105969+0000","last_peered":"2024-03-06T07:25:24.105969+0000","last_clean":"2024-03-06T07:25:24.105969+0000","last_became_active":"2024-03-06T07:25:24.099458+0000","last_became_peered":"2024-03-06T07:25:24.099458+0000","last_unstale":"2024-03-06T07:25:24.105969+0000","last_undegraded":"2024-03-06T07:25:24.105969+0000","last_fullsized":"2024-03-06T07:25:24.105969+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:44.507358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108098+0000","last_change":"2024-03-06T07:25:24.104105+0000","last_active":"2024-03-06T07:25:24.108098+0000","last_peered":"2024-03-06T07:25:24.108098+0000","last_clean":"2024-03-06T07:25:24.108098+0000","last_became_active":"2024-03-06T07:25:24.103807+0000","last_became_peered":"2024-03-06T07:25:24.103807+0000","last_unstale":"2024-03-06T07:25:24.108098+0000","last_undegraded":"2024-03-06T07:25:24.108098+0000","last_fullsized":"2024-03-06T07:25:24.108098+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:37.406837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.098354+0000","last_change":"2024-03-06T07:25:24.097152+0000","last_active":"2024-03-06T07:25:24.098354+0000","last_peered":"2024-03-06T07:25:24.098354+0000","last_clean":"2024-03-06T07:25:24.098354+0000","last_became_active":"2024-03-06T07:25:24.096878+0000","last_became_peered":"2024-03-06T07:25:24.096878+0000","last_unstale":"2024-03-06T07:25:24.098354+0000","last_undegraded":"2024-03-06T07:25:24.098354+0000","last_fullsized":"2024-03-06T07:25:24.098354+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:44.507358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105056+0000","last_change":"2024-03-06T07:25:24.100455+0000","last_active":"2024-03-06T07:25:24.105056+0000","last_peered":"2024-03-06T07:25:24.105056+0000","last_clean":"2024-03-06T07:25:24.105056+0000","last_became_active":"2024-03-06T07:25:24.100185+0000","last_became_peered":"2024-03-06T07:25:24.100185+0000","last_unstale":"2024-03-06T07:25:24.105056+0000","last_undegraded":"2024-03-06T07:25:24.105056+0000","last_fullsized":"2024-03-06T07:25:24.105056+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:09.534327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.103014+0000","last_change":"2024-03-06T07:25:24.096626+0000","last_active":"2024-03-06T07:25:24.103014+0000","last_peered":"2024-03-06T07:25:24.103014+0000","last_clean":"2024-03-06T07:25:24.103014+0000","last_became_active":"2024-03-06T07:25:24.096271+0000","last_became_peered":"2024-03-06T07:25:24.096271+0000","last_unstale":"2024-03-06T07:25:24.103014+0000","last_undegraded":"2024-03-06T07:25:24.103014+0000","last_fullsized":"2024-03-06T07:25:24.103014+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:01.234337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.104932+0000","last_change":"2024-03-06T07:25:24.099697+0000","last_active":"2024-03-06T07:25:24.104932+0000","last_peered":"2024-03-06T07:25:24.104932+0000","last_clean":"2024-03-06T07:25:24.104932+0000","last_became_active":"2024-03-06T07:25:24.099031+0000","last_became_peered":"2024-03-06T07:25:24.099031+0000","last_unstale":"2024-03-06T07:25:24.104932+0000","last_undegraded":"2024-03-06T07:25:24.104932+0000","last_fullsized":"2024-03-06T07:25:24.104932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:16.235427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108301+0000","last_change":"2024-03-06T07:25:24.096687+0000","last_active":"2024-03-06T07:25:24.108301+0000","last_peered":"2024-03-06T07:25:24.108301+0000","last_clean":"2024-03-06T07:25:24.108301+0000","last_became_active":"2024-03-06T07:25:24.096384+0000","last_became_peered":"2024-03-06T07:25:24.096384+0000","last_unstale":"2024-03-06T07:25:24.108301+0000","last_undegraded":"2024-03-06T07:25:24.108301+0000","last_fullsized":"2024-03-06T07:25:24.108301+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:49.573382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108435+0000","last_change":"2024-03-06T07:25:24.101960+0000","last_active":"2024-03-06T07:25:24.108435+0000","last_peered":"2024-03-06T07:25:24.108435+0000","last_clean":"2024-03-06T07:25:24.108435+0000","last_became_active":"2024-03-06T07:25:24.101702+0000","last_became_peered":"2024-03-06T07:25:24.101702+0000","last_unstale":"2024-03-06T07:25:24.108435+0000","last_undegraded":"2024-03-06T07:25:24.108435+0000","last_fullsized":"2024-03-06T07:25:24.108435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.849922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.106251+0000","last_change":"2024-03-06T07:25:24.102111+0000","last_active":"2024-03-06T07:25:24.106251+0000","last_peered":"2024-03-06T07:25:24.106251+0000","last_clean":"2024-03-06T07:25:24.106251+0000","last_became_active":"2024-03-06T07:25:24.101795+0000","last_became_peered":"2024-03-06T07:25:24.101795+0000","last_unstale":"2024-03-06T07:25:24.106251+0000","last_undegraded":"2024-03-06T07:25:24.106251+0000","last_fullsized":"2024-03-06T07:25:24.106251+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:38.207630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105986+0000","last_change":"2024-03-06T07:25:24.101492+0000","last_active":"2024-03-06T07:25:24.105986+0000","last_peered":"2024-03-06T07:25:24.105986+0000","last_clean":"2024-03-06T07:25:24.105986+0000","last_became_active":"2024-03-06T07:25:24.101229+0000","last_became_peered":"2024-03-06T07:25:24.101229+0000","last_unstale":"2024-03-06T07:25:24.105986+0000","last_undegraded":"2024-03-06T07:25:24.105986+0000","last_fullsized":"2024-03-06T07:25:24.105986+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.849922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.100940+0000","last_change":"2024-03-06T07:25:24.097551+0000","last_active":"2024-03-06T07:25:24.100940+0000","last_peered":"2024-03-06T07:25:24.100940+0000","last_clean":"2024-03-06T07:25:24.100940+0000","last_became_active":"2024-03-06T07:25:24.097292+0000","last_became_peered":"2024-03-06T07:25:24.097292+0000","last_unstale":"2024-03-06T07:25:24.100940+0000","last_undegraded":"2024-03-06T07:25:24.100940+0000","last_fullsized":"2024-03-06T07:25:24.100940+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:10.916532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108264+0000","last_change":"2024-03-06T07:25:24.105621+0000","last_active":"2024-03-06T07:25:24.108264+0000","last_peered":"2024-03-06T07:25:24.108264+0000","last_clean":"2024-03-06T07:25:24.108264+0000","last_became_active":"2024-03-06T07:25:24.105314+0000","last_became_peered":"2024-03-06T07:25:24.105314+0000","last_unstale":"2024-03-06T07:25:24.108264+0000","last_undegraded":"2024-03-06T07:25:24.108264+0000","last_fullsized":"2024-03-06T07:25:24.108264+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:25.257203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102004+0000","last_change":"2024-03-06T07:25:24.098643+0000","last_active":"2024-03-06T07:25:24.102004+0000","last_peered":"2024-03-06T07:25:24.102004+0000","last_clean":"2024-03-06T07:25:24.102004+0000","last_became_active":"2024-03-06T07:25:24.098367+0000","last_became_peered":"2024-03-06T07:25:24.098367+0000","last_unstale":"2024-03-06T07:25:24.102004+0000","last_undegraded":"2024-03-06T07:25:24.102004+0000","last_fullsized":"2024-03-06T07:25:24.102004+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:54.967926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109654+0000","last_change":"2024-03-06T07:25:24.098710+0000","last_active":"2024-03-06T07:25:24.109654+0000","last_peered":"2024-03-06T07:25:24.109654+0000","last_clean":"2024-03-06T07:25:24.109654+0000","last_became_active":"2024-03-06T07:25:24.096601+0000","last_became_peered":"2024-03-06T07:25:24.096601+0000","last_unstale":"2024-03-06T07:25:24.109654+0000","last_undegraded":"2024-03-06T07:25:24.109654+0000","last_fullsized":"2024-03-06T07:25:24.109654+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:05.660880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109866+0000","last_change":"2024-03-06T07:25:24.107802+0000","last_active":"2024-03-06T07:25:24.109866+0000","last_peered":"2024-03-06T07:25:24.109866+0000","last_clean":"2024-03-06T07:25:24.109866+0000","last_became_active":"2024-03-06T07:25:24.107528+0000","last_became_peered":"2024-03-06T07:25:24.107528+0000","last_unstale":"2024-03-06T07:25:24.109866+0000","last_undegraded":"2024-03-06T07:25:24.109866+0000","last_fullsized":"2024-03-06T07:25:24.109866+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:59.468984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108044+0000","last_change":"2024-03-06T07:25:24.104229+0000","last_active":"2024-03-06T07:25:24.108044+0000","last_peered":"2024-03-06T07:25:24.108044+0000","last_clean":"2024-03-06T07:25:24.108044+0000","last_became_active":"2024-03-06T07:25:24.103948+0000","last_became_peered":"2024-03-06T07:25:24.103948+0000","last_unstale":"2024-03-06T07:25:24.108044+0000","last_undegraded":"2024-03-06T07:25:24.108044+0000","last_fullsized":"2024-03-06T07:25:24.108044+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:49.495934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109622+0000","last_change":"2024-03-06T07:25:24.099289+0000","last_active":"2024-03-06T07:25:24.109622+0000","last_peered":"2024-03-06T07:25:24.109622+0000","last_clean":"2024-03-06T07:25:24.109622+0000","last_became_active":"2024-03-06T07:25:24.099046+0000","last_became_peered":"2024-03-06T07:25:24.099046+0000","last_unstale":"2024-03-06T07:25:24.109622+0000","last_undegraded":"2024-03-06T07:25:24.109622+0000","last_fullsized":"2024-03-06T07:25:24.109622+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:54.399281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108336+0000","last_change":"2024-03-06T07:25:24.104074+0000","last_active":"2024-03-06T07:25:24.108336+0000","last_peered":"2024-03-06T07:25:24.108336+0000","last_clean":"2024-03-06T07:25:24.108336+0000","last_became_active":"2024-03-06T07:25:24.103786+0000","last_became_peered":"2024-03-06T07:25:24.103786+0000","last_unstale":"2024-03-06T07:25:24.108336+0000","last_undegraded":"2024-03-06T07:25:24.108336+0000","last_fullsized":"2024-03-06T07:25:24.108336+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:33.972181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109690+0000","last_change":"2024-03-06T07:25:24.101550+0000","last_active":"2024-03-06T07:25:24.109690+0000","last_peered":"2024-03-06T07:25:24.109690+0000","last_clean":"2024-03-06T07:25:24.109690+0000","last_became_active":"2024-03-06T07:25:24.101274+0000","last_became_peered":"2024-03-06T07:25:24.101274+0000","last_unstale":"2024-03-06T07:25:24.109690+0000","last_undegraded":"2024-03-06T07:25:24.109690+0000","last_fullsized":"2024-03-06T07:25:24.109690+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:40.847577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112554+0000","last_change":"2024-03-06T07:25:24.110665+0000","last_active":"2024-03-06T07:25:24.112554+0000","last_peered":"2024-03-06T07:25:24.112554+0000","last_clean":"2024-03-06T07:25:24.112554+0000","last_became_active":"2024-03-06T07:25:24.110411+0000","last_became_peered":"2024-03-06T07:25:24.110411+0000","last_unstale":"2024-03-06T07:25:24.112554+0000","last_undegraded":"2024-03-06T07:25:24.112554+0000","last_fullsized":"2024-03-06T07:25:24.112554+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:51.284772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.103124+0000","last_change":"2024-03-06T07:25:24.098256+0000","last_active":"2024-03-06T07:25:24.103124+0000","last_peered":"2024-03-06T07:25:24.103124+0000","last_clean":"2024-03-06T07:25:24.103124+0000","last_became_active":"2024-03-06T07:25:24.097860+0000","last_became_peered":"2024-03-06T07:25:24.097860+0000","last_unstale":"2024-03-06T07:25:24.103124+0000","last_undegraded":"2024-03-06T07:25:24.103124+0000","last_fullsized":"2024-03-06T07:25:24.103124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:49.573382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.113207+0000","last_change":"2024-03-06T07:25:24.109408+0000","last_active":"2024-03-06T07:25:24.113207+0000","last_peered":"2024-03-06T07:25:24.113207+0000","last_clean":"2024-03-06T07:25:24.113207+0000","last_became_active":"2024-03-06T07:25:24.109123+0000","last_became_peered":"2024-03-06T07:25:24.109123+0000","last_unstale":"2024-03-06T07:25:24.113207+0000","last_undegraded":"2024-03-06T07:25:24.113207+0000","last_fullsized":"2024-03-06T07:25:24.113207+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:09.186893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105988+0000","last_change":"2024-03-06T07:25:24.099799+0000","last_active":"2024-03-06T07:25:24.105988+0000","last_peered":"2024-03-06T07:25:24.105988+0000","last_clean":"2024-03-06T07:25:24.105988+0000","last_became_active":"2024-03-06T07:25:24.099492+0000","last_became_peered":"2024-03-06T07:25:24.099492+0000","last_unstale":"2024-03-06T07:25:24.105988+0000","last_undegraded":"2024-03-06T07:25:24.105988+0000","last_fullsized":"2024-03-06T07:25:24.105988+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:54.967926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102885+0000","last_change":"2024-03-06T07:25:24.101674+0000","last_active":"2024-03-06T07:25:24.102885+0000","last_peered":"2024-03-06T07:25:24.102885+0000","last_clean":"2024-03-06T07:25:24.102885+0000","last_became_active":"2024-03-06T07:25:24.101369+0000","last_became_peered":"2024-03-06T07:25:24.101369+0000","last_unstale":"2024-03-06T07:25:24.102885+0000","last_undegraded":"2024-03-06T07:25:24.102885+0000","last_fullsized":"2024-03-06T07:25:24.102885+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:59.886631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108046+0000","last_change":"2024-03-06T07:25:24.104390+0000","last_active":"2024-03-06T07:25:24.108046+0000","last_peered":"2024-03-06T07:25:24.108046+0000","last_clean":"2024-03-06T07:25:24.108046+0000","last_became_active":"2024-03-06T07:25:24.104106+0000","last_became_peered":"2024-03-06T07:25:24.104106+0000","last_unstale":"2024-03-06T07:25:24.108046+0000","last_undegraded":"2024-03-06T07:25:24.108046+0000","last_fullsized":"2024-03-06T07:25:24.108046+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:54.056108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109696+0000","last_change":"2024-03-06T07:25:24.100977+0000","last_active":"2024-03-06T07:25:24.109696+0000","last_peered":"2024-03-06T07:25:24.109696+0000","last_clean":"2024-03-06T07:25:24.109696+0000","last_became_active":"2024-03-06T07:25:24.100193+0000","last_became_peered":"2024-03-06T07:25:24.100193+0000","last_unstale":"2024-03-06T07:25:24.109696+0000","last_undegraded":"2024-03-06T07:25:24.109696+0000","last_fullsized":"2024-03-06T07:25:24.109696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:56.585961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109742+0000","last_change":"2024-03-06T07:25:24.106056+0000","last_active":"2024-03-06T07:25:24.109742+0000","last_peered":"2024-03-06T07:25:24.109742+0000","last_clean":"2024-03-06T07:25:24.109742+0000","last_became_active":"2024-03-06T07:25:24.105318+0000","last_became_peered":"2024-03-06T07:25:24.105318+0000","last_unstale":"2024-03-06T07:25:24.109742+0000","last_undegraded":"2024-03-06T07:25:24.109742+0000","last_fullsized":"2024-03-06T07:25:24.109742+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:09.004465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.113234+0000","last_change":"2024-03-06T07:25:24.109288+0000","last_active":"2024-03-06T07:25:24.113234+0000","last_peered":"2024-03-06T07:25:24.113234+0000","last_clean":"2024-03-06T07:25:24.113234+0000","last_became_active":"2024-03-06T07:25:24.108947+0000","last_became_peered":"2024-03-06T07:25:24.108947+0000","last_unstale":"2024-03-06T07:25:24.113234+0000","last_undegraded":"2024-03-06T07:25:24.113234+0000","last_fullsized":"2024-03-06T07:25:24.113234+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:27.300274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111036+0000","last_change":"2024-03-06T07:25:24.106381+0000","last_active":"2024-03-06T07:25:24.111036+0000","last_peered":"2024-03-06T07:25:24.111036+0000","last_clean":"2024-03-06T07:25:24.111036+0000","last_became_active":"2024-03-06T07:25:24.106001+0000","last_became_peered":"2024-03-06T07:25:24.106001+0000","last_unstale":"2024-03-06T07:25:24.111036+0000","last_undegraded":"2024-03-06T07:25:24.111036+0000","last_fullsized":"2024-03-06T07:25:24.111036+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:59.468984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108285+0000","last_change":"2024-03-06T07:25:24.102962+0000","last_active":"2024-03-06T07:25:24.108285+0000","last_peered":"2024-03-06T07:25:24.108285+0000","last_clean":"2024-03-06T07:25:24.108285+0000","last_became_active":"2024-03-06T07:25:24.102311+0000","last_became_peered":"2024-03-06T07:25:24.102311+0000","last_unstale":"2024-03-06T07:25:24.108285+0000","last_undegraded":"2024-03-06T07:25:24.108285+0000","last_fullsized":"2024-03-06T07:25:24.108285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:09.004465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.121726+0000","last_change":"2024-03-06T07:25:23.092130+0000","last_active":"2024-03-06T07:25:24.121726+0000","last_peered":"2024-03-06T07:25:24.121726+0000","last_clean":"2024-03-06T07:25:24.121726+0000","last_became_active":"2024-03-06T07:25:23.091809+0000","last_became_peered":"2024-03-06T07:25:23.091809+0000","last_unstale":"2024-03-06T07:25:24.121726+0000","last_undegraded":"2024-03-06T07:25:24.121726+0000","last_fullsized":"2024-03-06T07:25:24.121726+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:01.112680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.110307+0000","last_change":"2024-03-06T07:25:24.105491+0000","last_active":"2024-03-06T07:25:24.110307+0000","last_peered":"2024-03-06T07:25:24.110307+0000","last_clean":"2024-03-06T07:25:24.110307+0000","last_became_active":"2024-03-06T07:25:24.105174+0000","last_became_peered":"2024-03-06T07:25:24.105174+0000","last_unstale":"2024-03-06T07:25:24.110307+0000","last_undegraded":"2024-03-06T07:25:24.110307+0000","last_fullsized":"2024-03-06T07:25:24.110307+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:48.332882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.121483+0000","last_change":"2024-03-06T07:25:23.092085+0000","last_active":"2024-03-06T07:25:24.121483+0000","last_peered":"2024-03-06T07:25:24.121483+0000","last_clean":"2024-03-06T07:25:24.121483+0000","last_became_active":"2024-03-06T07:25:23.091792+0000","last_became_peered":"2024-03-06T07:25:23.091792+0000","last_unstale":"2024-03-06T07:25:24.121483+0000","last_undegraded":"2024-03-06T07:25:24.121483+0000","last_fullsized":"2024-03-06T07:25:24.121483+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:28.888672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112841+0000","last_change":"2024-03-06T07:25:24.109440+0000","last_active":"2024-03-06T07:25:24.112841+0000","last_peered":"2024-03-06T07:25:24.112841+0000","last_clean":"2024-03-06T07:25:24.112841+0000","last_became_active":"2024-03-06T07:25:24.109069+0000","last_became_peered":"2024-03-06T07:25:24.109069+0000","last_unstale":"2024-03-06T07:25:24.112841+0000","last_undegraded":"2024-03-06T07:25:24.112841+0000","last_fullsized":"2024-03-06T07:25:24.112841+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:29.895963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.118552+0000","last_change":"2024-03-06T07:25:23.091418+0000","last_active":"2024-03-06T07:25:24.118552+0000","last_peered":"2024-03-06T07:25:24.118552+0000","last_clean":"2024-03-06T07:25:24.118552+0000","last_became_active":"2024-03-06T07:25:23.091106+0000","last_became_peered":"2024-03-06T07:25:23.091106+0000","last_unstale":"2024-03-06T07:25:24.118552+0000","last_undegraded":"2024-03-06T07:25:24.118552+0000","last_fullsized":"2024-03-06T07:25:24.118552+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:36.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.110841+0000","last_change":"2024-03-06T07:25:24.106146+0000","last_active":"2024-03-06T07:25:24.110841+0000","last_peered":"2024-03-06T07:25:24.110841+0000","last_clean":"2024-03-06T07:25:24.110841+0000","last_became_active":"2024-03-06T07:25:24.105840+0000","last_became_peered":"2024-03-06T07:25:24.105840+0000","last_unstale":"2024-03-06T07:25:24.110841+0000","last_undegraded":"2024-03-06T07:25:24.110841+0000","last_fullsized":"2024-03-06T07:25:24.110841+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:03.393105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.119328+0000","last_change":"2024-03-06T07:25:23.092990+0000","last_active":"2024-03-06T07:25:24.119328+0000","last_peered":"2024-03-06T07:25:24.119328+0000","last_clean":"2024-03-06T07:25:24.119328+0000","last_became_active":"2024-03-06T07:25:23.092560+0000","last_became_peered":"2024-03-06T07:25:23.092560+0000","last_unstale":"2024-03-06T07:25:24.119328+0000","last_undegraded":"2024-03-06T07:25:24.119328+0000","last_fullsized":"2024-03-06T07:25:24.119328+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:01.112680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109655+0000","last_change":"2024-03-06T07:25:24.100892+0000","last_active":"2024-03-06T07:25:24.109655+0000","last_peered":"2024-03-06T07:25:24.109655+0000","last_clean":"2024-03-06T07:25:24.109655+0000","last_became_active":"2024-03-06T07:25:24.100291+0000","last_became_peered":"2024-03-06T07:25:24.100291+0000","last_unstale":"2024-03-06T07:25:24.109655+0000","last_undegraded":"2024-03-06T07:25:24.109655+0000","last_fullsized":"2024-03-06T07:25:24.109655+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:03.571121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.116951+0000","last_change":"2024-03-06T07:25:23.089496+0000","last_active":"2024-03-06T07:25:24.116951+0000","last_peered":"2024-03-06T07:25:24.116951+0000","last_clean":"2024-03-06T07:25:24.116951+0000","last_became_active":"2024-03-06T07:25:23.089232+0000","last_became_peered":"2024-03-06T07:25:23.089232+0000","last_unstale":"2024-03-06T07:25:24.116951+0000","last_undegraded":"2024-03-06T07:25:24.116951+0000","last_fullsized":"2024-03-06T07:25:24.116951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:06.182827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109718+0000","last_change":"2024-03-06T07:25:24.103749+0000","last_active":"2024-03-06T07:25:24.109718+0000","last_peered":"2024-03-06T07:25:24.109718+0000","last_clean":"2024-03-06T07:25:24.109718+0000","last_became_active":"2024-03-06T07:25:24.103478+0000","last_became_peered":"2024-03-06T07:25:24.103478+0000","last_unstale":"2024-03-06T07:25:24.109718+0000","last_undegraded":"2024-03-06T07:25:24.109718+0000","last_fullsized":"2024-03-06T07:25:24.109718+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:25.146984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.119128+0000","last_change":"2024-03-06T07:25:23.092760+0000","last_active":"2024-03-06T07:25:24.119128+0000","last_peered":"2024-03-06T07:25:24.119128+0000","last_clean":"2024-03-06T07:25:24.119128+0000","last_became_active":"2024-03-06T07:25:23.092498+0000","last_became_peered":"2024-03-06T07:25:23.092498+0000","last_unstale":"2024-03-06T07:25:24.119128+0000","last_undegraded":"2024-03-06T07:25:24.119128+0000","last_fullsized":"2024-03-06T07:25:24.119128+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:06.362639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.106025+0000","last_change":"2024-03-06T07:25:24.101599+0000","last_active":"2024-03-06T07:25:24.106025+0000","last_peered":"2024-03-06T07:25:24.106025+0000","last_clean":"2024-03-06T07:25:24.106025+0000","last_became_active":"2024-03-06T07:25:24.101340+0000","last_became_peered":"2024-03-06T07:25:24.101340+0000","last_unstale":"2024-03-06T07:25:24.106025+0000","last_undegraded":"2024-03-06T07:25:24.106025+0000","last_fullsized":"2024-03-06T07:25:24.106025+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:48.332882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.094502+0000","last_change":"2024-03-06T07:25:23.089745+0000","last_active":"2024-03-06T07:25:24.094502+0000","last_peered":"2024-03-06T07:25:24.094502+0000","last_clean":"2024-03-06T07:25:24.094502+0000","last_became_active":"2024-03-06T07:25:23.089434+0000","last_became_peered":"2024-03-06T07:25:23.089434+0000","last_unstale":"2024-03-06T07:25:24.094502+0000","last_undegraded":"2024-03-06T07:25:24.094502+0000","last_fullsized":"2024-03-06T07:25:24.094502+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:38.539557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109670+0000","last_change":"2024-03-06T07:25:24.102358+0000","last_active":"2024-03-06T07:25:24.109670+0000","last_peered":"2024-03-06T07:25:24.109670+0000","last_clean":"2024-03-06T07:25:24.109670+0000","last_became_active":"2024-03-06T07:25:24.102096+0000","last_became_peered":"2024-03-06T07:25:24.102096+0000","last_unstale":"2024-03-06T07:25:24.109670+0000","last_undegraded":"2024-03-06T07:25:24.109670+0000","last_fullsized":"2024-03-06T07:25:24.109670+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:25.198052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.090868+0000","last_change":"2024-03-06T07:25:23.090204+0000","last_active":"2024-03-06T07:25:24.090868+0000","last_peered":"2024-03-06T07:25:24.090868+0000","last_clean":"2024-03-06T07:25:24.090868+0000","last_became_active":"2024-03-06T07:25:23.089920+0000","last_became_peered":"2024-03-06T07:25:23.089920+0000","last_unstale":"2024-03-06T07:25:24.090868+0000","last_undegraded":"2024-03-06T07:25:24.090868+0000","last_fullsized":"2024-03-06T07:25:24.090868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:47.492015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.100975+0000","last_change":"2024-03-06T07:25:24.097780+0000","last_active":"2024-03-06T07:25:24.100975+0000","last_peered":"2024-03-06T07:25:24.100975+0000","last_clean":"2024-03-06T07:25:24.100975+0000","last_became_active":"2024-03-06T07:25:24.097512+0000","last_became_peered":"2024-03-06T07:25:24.097512+0000","last_unstale":"2024-03-06T07:25:24.100975+0000","last_undegraded":"2024-03-06T07:25:24.100975+0000","last_fullsized":"2024-03-06T07:25:24.100975+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:53.327017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.092982+0000","last_change":"2024-03-06T07:25:23.095261+0000","last_active":"2024-03-06T07:25:24.092982+0000","last_peered":"2024-03-06T07:25:24.092982+0000","last_clean":"2024-03-06T07:25:24.092982+0000","last_became_active":"2024-03-06T07:25:23.094991+0000","last_became_peered":"2024-03-06T07:25:23.094991+0000","last_unstale":"2024-03-06T07:25:24.092982+0000","last_undegraded":"2024-03-06T07:25:24.092982+0000","last_fullsized":"2024-03-06T07:25:24.092982+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:09.181395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112526+0000","last_change":"2024-03-06T07:25:24.109459+0000","last_active":"2024-03-06T07:25:24.112526+0000","last_peered":"2024-03-06T07:25:24.112526+0000","last_clean":"2024-03-06T07:25:24.112526+0000","last_became_active":"2024-03-06T07:25:24.109035+0000","last_became_peered":"2024-03-06T07:25:24.109035+0000","last_unstale":"2024-03-06T07:25:24.112526+0000","last_undegraded":"2024-03-06T07:25:24.112526+0000","last_fullsized":"2024-03-06T07:25:24.112526+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:29.895963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.090973+0000","last_change":"2024-03-06T07:25:23.080697+0000","last_active":"2024-03-06T07:25:24.090973+0000","last_peered":"2024-03-06T07:25:24.090973+0000","last_clean":"2024-03-06T07:25:24.090973+0000","last_became_active":"2024-03-06T07:25:23.080406+0000","last_became_peered":"2024-03-06T07:25:23.080406+0000","last_unstale":"2024-03-06T07:25:24.090973+0000","last_undegraded":"2024-03-06T07:25:24.090973+0000","last_fullsized":"2024-03-06T07:25:24.090973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:50.277480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108132+0000","last_change":"2024-03-06T07:25:24.093749+0000","last_active":"2024-03-06T07:25:24.108132+0000","last_peered":"2024-03-06T07:25:24.108132+0000","last_clean":"2024-03-06T07:25:24.108132+0000","last_became_active":"2024-03-06T07:25:24.093440+0000","last_became_peered":"2024-03-06T07:25:24.093440+0000","last_unstale":"2024-03-06T07:25:24.108132+0000","last_undegraded":"2024-03-06T07:25:24.108132+0000","last_fullsized":"2024-03-06T07:25:24.108132+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:44.331858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.119150+0000","last_change":"2024-03-06T07:25:23.090765+0000","last_active":"2024-03-06T07:25:24.119150+0000","last_peered":"2024-03-06T07:25:24.119150+0000","last_clean":"2024-03-06T07:25:24.119150+0000","last_became_active":"2024-03-06T07:25:23.090491+0000","last_became_peered":"2024-03-06T07:25:23.090491+0000","last_unstale":"2024-03-06T07:25:24.119150+0000","last_undegraded":"2024-03-06T07:25:24.119150+0000","last_fullsized":"2024-03-06T07:25:24.119150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:53.960634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112490+0000","last_change":"2024-03-06T07:25:24.110068+0000","last_active":"2024-03-06T07:25:24.112490+0000","last_peered":"2024-03-06T07:25:24.112490+0000","last_clean":"2024-03-06T07:25:24.112490+0000","last_became_active":"2024-03-06T07:25:24.109756+0000","last_became_peered":"2024-03-06T07:25:24.109756+0000","last_unstale":"2024-03-06T07:25:24.112490+0000","last_undegraded":"2024-03-06T07:25:24.112490+0000","last_fullsized":"2024-03-06T07:25:24.112490+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:19.248382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.123984+0000","last_change":"2024-03-06T07:25:23.097336+0000","last_active":"2024-03-06T07:25:24.123984+0000","last_peered":"2024-03-06T07:25:24.123984+0000","last_clean":"2024-03-06T07:25:24.123984+0000","last_became_active":"2024-03-06T07:25:23.097084+0000","last_became_peered":"2024-03-06T07:25:23.097084+0000","last_unstale":"2024-03-06T07:25:24.123984+0000","last_undegraded":"2024-03-06T07:25:24.123984+0000","last_fullsized":"2024-03-06T07:25:24.123984+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:27.972802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111304+0000","last_change":"2024-03-06T07:25:24.110168+0000","last_active":"2024-03-06T07:25:24.111304+0000","last_peered":"2024-03-06T07:25:24.111304+0000","last_clean":"2024-03-06T07:25:24.111304+0000","last_became_active":"2024-03-06T07:25:24.109768+0000","last_became_peered":"2024-03-06T07:25:24.109768+0000","last_unstale":"2024-03-06T07:25:24.111304+0000","last_undegraded":"2024-03-06T07:25:24.111304+0000","last_fullsized":"2024-03-06T07:25:24.111304+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:59.468984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.118223+0000","last_change":"2024-03-06T07:25:23.083712+0000","last_active":"2024-03-06T07:25:24.118223+0000","last_peered":"2024-03-06T07:25:24.118223+0000","last_clean":"2024-03-06T07:25:24.118223+0000","last_became_active":"2024-03-06T07:25:23.083477+0000","last_became_peered":"2024-03-06T07:25:23.083477+0000","last_unstale":"2024-03-06T07:25:24.118223+0000","last_undegraded":"2024-03-06T07:25:24.118223+0000","last_fullsized":"2024-03-06T07:25:24.118223+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:34.798242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108358+0000","last_change":"2024-03-06T07:25:24.104682+0000","last_active":"2024-03-06T07:25:24.108358+0000","last_peered":"2024-03-06T07:25:24.108358+0000","last_clean":"2024-03-06T07:25:24.108358+0000","last_became_active":"2024-03-06T07:25:24.104390+0000","last_became_peered":"2024-03-06T07:25:24.104390+0000","last_unstale":"2024-03-06T07:25:24.108358+0000","last_undegraded":"2024-03-06T07:25:24.108358+0000","last_fullsized":"2024-03-06T07:25:24.108358+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:59.886631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.093325+0000","last_change":"2024-03-06T07:25:23.087632+0000","last_active":"2024-03-06T07:25:24.093325+0000","last_peered":"2024-03-06T07:25:24.093325+0000","last_clean":"2024-03-06T07:25:24.093325+0000","last_became_active":"2024-03-06T07:25:23.087259+0000","last_became_peered":"2024-03-06T07:25:23.087259+0000","last_unstale":"2024-03-06T07:25:24.093325+0000","last_undegraded":"2024-03-06T07:25:24.093325+0000","last_fullsized":"2024-03-06T07:25:24.093325+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:25.569705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105773+0000","last_change":"2024-03-06T07:25:24.101290+0000","last_active":"2024-03-06T07:25:24.105773+0000","last_peered":"2024-03-06T07:25:24.105773+0000","last_clean":"2024-03-06T07:25:24.105773+0000","last_became_active":"2024-03-06T07:25:24.101034+0000","last_became_peered":"2024-03-06T07:25:24.101034+0000","last_unstale":"2024-03-06T07:25:24.105773+0000","last_undegraded":"2024-03-06T07:25:24.105773+0000","last_fullsized":"2024-03-06T07:25:24.105773+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:20.980764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.094692+0000","last_change":"2024-03-06T07:25:23.092145+0000","last_active":"2024-03-06T07:25:24.094692+0000","last_peered":"2024-03-06T07:25:24.094692+0000","last_clean":"2024-03-06T07:25:24.094692+0000","last_became_active":"2024-03-06T07:25:23.091895+0000","last_became_peered":"2024-03-06T07:25:23.091895+0000","last_unstale":"2024-03-06T07:25:24.094692+0000","last_undegraded":"2024-03-06T07:25:24.094692+0000","last_fullsized":"2024-03-06T07:25:24.094692+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:36.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112573+0000","last_change":"2024-03-06T07:25:24.110797+0000","last_active":"2024-03-06T07:25:24.112573+0000","last_peered":"2024-03-06T07:25:24.112573+0000","last_clean":"2024-03-06T07:25:24.112573+0000","last_became_active":"2024-03-06T07:25:24.110499+0000","last_became_peered":"2024-03-06T07:25:24.110499+0000","last_unstale":"2024-03-06T07:25:24.112573+0000","last_undegraded":"2024-03-06T07:25:24.112573+0000","last_fullsized":"2024-03-06T07:25:24.112573+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:35.805534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.121646+0000","last_change":"2024-03-06T07:25:23.092882+0000","last_active":"2024-03-06T07:25:24.121646+0000","last_peered":"2024-03-06T07:25:24.121646+0000","last_clean":"2024-03-06T07:25:24.121646+0000","last_became_active":"2024-03-06T07:25:23.092629+0000","last_became_peered":"2024-03-06T07:25:23.092629+0000","last_unstale":"2024-03-06T07:25:24.121646+0000","last_undegraded":"2024-03-06T07:25:24.121646+0000","last_fullsized":"2024-03-06T07:25:24.121646+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:43.500066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109845+0000","last_change":"2024-03-06T07:25:24.106360+0000","last_active":"2024-03-06T07:25:24.109845+0000","last_peered":"2024-03-06T07:25:24.109845+0000","last_clean":"2024-03-06T07:25:24.109845+0000","last_became_active":"2024-03-06T07:25:24.106060+0000","last_became_peered":"2024-03-06T07:25:24.106060+0000","last_unstale":"2024-03-06T07:25:24.109845+0000","last_undegraded":"2024-03-06T07:25:24.109845+0000","last_fullsized":"2024-03-06T07:25:24.109845+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:03.393105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.117514+0000","last_change":"2024-03-06T07:25:23.083039+0000","last_active":"2024-03-06T07:25:24.117514+0000","last_peered":"2024-03-06T07:25:24.117514+0000","last_clean":"2024-03-06T07:25:24.117514+0000","last_became_active":"2024-03-06T07:25:23.082770+0000","last_became_peered":"2024-03-06T07:25:23.082770+0000","last_unstale":"2024-03-06T07:25:24.117514+0000","last_undegraded":"2024-03-06T07:25:24.117514+0000","last_fullsized":"2024-03-06T07:25:24.117514+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:18.241091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111300+0000","last_change":"2024-03-06T07:25:24.109619+0000","last_active":"2024-03-06T07:25:24.111300+0000","last_peered":"2024-03-06T07:25:24.111300+0000","last_clean":"2024-03-06T07:25:24.111300+0000","last_became_active":"2024-03-06T07:25:24.108824+0000","last_became_peered":"2024-03-06T07:25:24.108824+0000","last_unstale":"2024-03-06T07:25:24.111300+0000","last_undegraded":"2024-03-06T07:25:24.111300+0000","last_fullsized":"2024-03-06T07:25:24.111300+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:03.393105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.118271+0000","last_change":"2024-03-06T07:25:23.089357+0000","last_active":"2024-03-06T07:25:24.118271+0000","last_peered":"2024-03-06T07:25:24.118271+0000","last_clean":"2024-03-06T07:25:24.118271+0000","last_became_active":"2024-03-06T07:25:23.089045+0000","last_became_peered":"2024-03-06T07:25:23.089045+0000","last_unstale":"2024-03-06T07:25:24.118271+0000","last_undegraded":"2024-03-06T07:25:24.118271+0000","last_fullsized":"2024-03-06T07:25:24.118271+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:58.487125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109598+0000","last_change":"2024-03-06T07:25:24.097976+0000","last_active":"2024-03-06T07:25:24.109598+0000","last_peered":"2024-03-06T07:25:24.109598+0000","last_clean":"2024-03-06T07:25:24.109598+0000","last_became_active":"2024-03-06T07:25:24.097074+0000","last_became_peered":"2024-03-06T07:25:24.097074+0000","last_unstale":"2024-03-06T07:25:24.109598+0000","last_undegraded":"2024-03-06T07:25:24.109598+0000","last_fullsized":"2024-03-06T07:25:24.109598+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:41.069951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.116703+0000","last_change":"2024-03-06T07:25:23.092189+0000","last_active":"2024-03-06T07:25:24.116703+0000","last_peered":"2024-03-06T07:25:24.116703+0000","last_clean":"2024-03-06T07:25:24.116703+0000","last_became_active":"2024-03-06T07:25:23.091901+0000","last_became_peered":"2024-03-06T07:25:23.091901+0000","last_unstale":"2024-03-06T07:25:24.116703+0000","last_undegraded":"2024-03-06T07:25:24.116703+0000","last_fullsized":"2024-03-06T07:25:24.116703+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:58.487125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112458+0000","last_change":"2024-03-06T07:25:24.109266+0000","last_active":"2024-03-06T07:25:24.112458+0000","last_peered":"2024-03-06T07:25:24.112458+0000","last_clean":"2024-03-06T07:25:24.112458+0000","last_became_active":"2024-03-06T07:25:24.108995+0000","last_became_peered":"2024-03-06T07:25:24.108995+0000","last_unstale":"2024-03-06T07:25:24.112458+0000","last_undegraded":"2024-03-06T07:25:24.112458+0000","last_fullsized":"2024-03-06T07:25:24.112458+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:26.022393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.091506+0000","last_change":"2024-03-06T07:25:23.095670+0000","last_active":"2024-03-06T07:25:24.091506+0000","last_peered":"2024-03-06T07:25:24.091506+0000","last_clean":"2024-03-06T07:25:24.091506+0000","last_became_active":"2024-03-06T07:25:23.094714+0000","last_became_peered":"2024-03-06T07:25:23.094714+0000","last_unstale":"2024-03-06T07:25:24.091506+0000","last_undegraded":"2024-03-06T07:25:24.091506+0000","last_fullsized":"2024-03-06T07:25:24.091506+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:38.539557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111259+0000","last_change":"2024-03-06T07:25:24.109297+0000","last_active":"2024-03-06T07:25:24.111259+0000","last_peered":"2024-03-06T07:25:24.111259+0000","last_clean":"2024-03-06T07:25:24.111259+0000","last_became_active":"2024-03-06T07:25:24.109042+0000","last_became_peered":"2024-03-06T07:25:24.109042+0000","last_unstale":"2024-03-06T07:25:24.111259+0000","last_undegraded":"2024-03-06T07:25:24.111259+0000","last_fullsized":"2024-03-06T07:25:24.111259+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:09.004465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.121645+0000","last_change":"2024-03-06T07:25:23.086139+0000","last_active":"2024-03-06T07:25:24.121645+0000","last_peered":"2024-03-06T07:25:24.121645+0000","last_clean":"2024-03-06T07:25:24.121645+0000","last_became_active":"2024-03-06T07:25:23.085864+0000","last_became_peered":"2024-03-06T07:25:23.085864+0000","last_unstale":"2024-03-06T07:25:24.121645+0000","last_undegraded":"2024-03-06T07:25:24.121645+0000","last_fullsized":"2024-03-06T07:25:24.121645+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:52.216630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102684+0000","last_change":"2024-03-06T07:25:24.099480+0000","last_active":"2024-03-06T07:25:24.102684+0000","last_peered":"2024-03-06T07:25:24.102684+0000","last_clean":"2024-03-06T07:25:24.102684+0000","last_became_active":"2024-03-06T07:25:24.099188+0000","last_became_peered":"2024-03-06T07:25:24.099188+0000","last_unstale":"2024-03-06T07:25:24.102684+0000","last_undegraded":"2024-03-06T07:25:24.102684+0000","last_fullsized":"2024-03-06T07:25:24.102684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:33.972181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.087420+0000","last_change":"2024-03-06T07:25:23.090129+0000","last_active":"2024-03-06T07:25:24.087420+0000","last_peered":"2024-03-06T07:25:24.087420+0000","last_clean":"2024-03-06T07:25:24.087420+0000","last_became_active":"2024-03-06T07:25:23.089825+0000","last_became_peered":"2024-03-06T07:25:23.089825+0000","last_unstale":"2024-03-06T07:25:24.087420+0000","last_undegraded":"2024-03-06T07:25:24.087420+0000","last_fullsized":"2024-03-06T07:25:24.087420+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:45.846706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102741+0000","last_change":"2024-03-06T07:25:24.101847+0000","last_active":"2024-03-06T07:25:24.102741+0000","last_peered":"2024-03-06T07:25:24.102741+0000","last_clean":"2024-03-06T07:25:24.102741+0000","last_became_active":"2024-03-06T07:25:24.101554+0000","last_became_peered":"2024-03-06T07:25:24.101554+0000","last_unstale":"2024-03-06T07:25:24.102741+0000","last_undegraded":"2024-03-06T07:25:24.102741+0000","last_fullsized":"2024-03-06T07:25:24.102741+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:38.207630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.093659+0000","last_change":"2024-03-06T07:25:23.093613+0000","last_active":"2024-03-06T07:25:24.093659+0000","last_peered":"2024-03-06T07:25:24.093659+0000","last_clean":"2024-03-06T07:25:24.093659+0000","last_became_active":"2024-03-06T07:25:23.093098+0000","last_became_peered":"2024-03-06T07:25:23.093098+0000","last_unstale":"2024-03-06T07:25:24.093659+0000","last_undegraded":"2024-03-06T07:25:24.093659+0000","last_fullsized":"2024-03-06T07:25:24.093659+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:06.362639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.106018+0000","last_change":"2024-03-06T07:25:24.100746+0000","last_active":"2024-03-06T07:25:24.106018+0000","last_peered":"2024-03-06T07:25:24.106018+0000","last_clean":"2024-03-06T07:25:24.106018+0000","last_became_active":"2024-03-06T07:25:24.100485+0000","last_became_peered":"2024-03-06T07:25:24.100485+0000","last_unstale":"2024-03-06T07:25:24.106018+0000","last_undegraded":"2024-03-06T07:25:24.106018+0000","last_fullsized":"2024-03-06T07:25:24.106018+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:33.972181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.118617+0000","last_change":"2024-03-06T07:25:23.092359+0000","last_active":"2024-03-06T07:25:24.118617+0000","last_peered":"2024-03-06T07:25:24.118617+0000","last_clean":"2024-03-06T07:25:24.118617+0000","last_became_active":"2024-03-06T07:25:23.092081+0000","last_became_peered":"2024-03-06T07:25:23.092081+0000","last_unstale":"2024-03-06T07:25:24.118617+0000","last_undegraded":"2024-03-06T07:25:24.118617+0000","last_fullsized":"2024-03-06T07:25:24.118617+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:57.095516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.105927+0000","last_change":"2024-03-06T07:25:24.101437+0000","last_active":"2024-03-06T07:25:24.105927+0000","last_peered":"2024-03-06T07:25:24.105927+0000","last_clean":"2024-03-06T07:25:24.105927+0000","last_became_active":"2024-03-06T07:25:24.101184+0000","last_became_peered":"2024-03-06T07:25:24.101184+0000","last_unstale":"2024-03-06T07:25:24.105927+0000","last_undegraded":"2024-03-06T07:25:24.105927+0000","last_fullsized":"2024-03-06T07:25:24.105927+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:59.886631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.095076+0000","last_change":"2024-03-06T07:25:23.091258+0000","last_active":"2024-03-06T07:25:24.095076+0000","last_peered":"2024-03-06T07:25:24.095076+0000","last_clean":"2024-03-06T07:25:24.095076+0000","last_became_active":"2024-03-06T07:25:23.090379+0000","last_became_peered":"2024-03-06T07:25:23.090379+0000","last_unstale":"2024-03-06T07:25:24.095076+0000","last_undegraded":"2024-03-06T07:25:24.095076+0000","last_fullsized":"2024-03-06T07:25:24.095076+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:09.181395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108109+0000","last_change":"2024-03-06T07:25:24.092988+0000","last_active":"2024-03-06T07:25:24.108109+0000","last_peered":"2024-03-06T07:25:24.108109+0000","last_clean":"2024-03-06T07:25:24.108109+0000","last_became_active":"2024-03-06T07:25:24.092717+0000","last_became_peered":"2024-03-06T07:25:24.092717+0000","last_unstale":"2024-03-06T07:25:24.108109+0000","last_undegraded":"2024-03-06T07:25:24.108109+0000","last_fullsized":"2024-03-06T07:25:24.108109+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:26.576997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.093493+0000","last_change":"2024-03-06T07:25:23.085559+0000","last_active":"2024-03-06T07:25:24.093493+0000","last_peered":"2024-03-06T07:25:24.093493+0000","last_clean":"2024-03-06T07:25:24.093493+0000","last_became_active":"2024-03-06T07:25:23.085283+0000","last_became_peered":"2024-03-06T07:25:23.085283+0000","last_unstale":"2024-03-06T07:25:24.093493+0000","last_undegraded":"2024-03-06T07:25:24.093493+0000","last_fullsized":"2024-03-06T07:25:24.093493+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:43.324566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111069+0000","last_change":"2024-03-06T07:25:24.106961+0000","last_active":"2024-03-06T07:25:24.111069+0000","last_peered":"2024-03-06T07:25:24.111069+0000","last_clean":"2024-03-06T07:25:24.111069+0000","last_became_active":"2024-03-06T07:25:24.106668+0000","last_became_peered":"2024-03-06T07:25:24.106668+0000","last_unstale":"2024-03-06T07:25:24.111069+0000","last_undegraded":"2024-03-06T07:25:24.111069+0000","last_fullsized":"2024-03-06T07:25:24.111069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:26.022393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.095064+0000","last_change":"2024-03-06T07:25:23.091449+0000","last_active":"2024-03-06T07:25:24.095064+0000","last_peered":"2024-03-06T07:25:24.095064+0000","last_clean":"2024-03-06T07:25:24.095064+0000","last_became_active":"2024-03-06T07:25:23.091167+0000","last_became_peered":"2024-03-06T07:25:23.091167+0000","last_unstale":"2024-03-06T07:25:24.095064+0000","last_undegraded":"2024-03-06T07:25:24.095064+0000","last_fullsized":"2024-03-06T07:25:24.095064+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:09.181395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.102972+0000","last_change":"2024-03-06T07:25:24.096010+0000","last_active":"2024-03-06T07:25:24.102972+0000","last_peered":"2024-03-06T07:25:24.102972+0000","last_clean":"2024-03-06T07:25:24.102972+0000","last_became_active":"2024-03-06T07:25:24.095639+0000","last_became_peered":"2024-03-06T07:25:24.095639+0000","last_unstale":"2024-03-06T07:25:24.102972+0000","last_undegraded":"2024-03-06T07:25:24.102972+0000","last_fullsized":"2024-03-06T07:25:24.102972+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:35.404387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.087471+0000","last_change":"2024-03-06T07:25:23.089829+0000","last_active":"2024-03-06T07:25:24.087471+0000","last_peered":"2024-03-06T07:25:24.087471+0000","last_clean":"2024-03-06T07:25:24.087471+0000","last_became_active":"2024-03-06T07:25:23.089568+0000","last_became_peered":"2024-03-06T07:25:23.089568+0000","last_unstale":"2024-03-06T07:25:24.087471+0000","last_undegraded":"2024-03-06T07:25:24.087471+0000","last_fullsized":"2024-03-06T07:25:24.087471+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:37.442485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.112571+0000","last_change":"2024-03-06T07:25:24.110889+0000","last_active":"2024-03-06T07:25:24.112571+0000","last_peered":"2024-03-06T07:25:24.112571+0000","last_clean":"2024-03-06T07:25:24.112571+0000","last_became_active":"2024-03-06T07:25:24.110465+0000","last_became_peered":"2024-03-06T07:25:24.110465+0000","last_unstale":"2024-03-06T07:25:24.112571+0000","last_undegraded":"2024-03-06T07:25:24.112571+0000","last_fullsized":"2024-03-06T07:25:24.112571+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:53.223922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.090718+0000","last_change":"2024-03-06T07:25:23.100020+0000","last_active":"2024-03-06T07:25:24.090718+0000","last_peered":"2024-03-06T07:25:24.090718+0000","last_clean":"2024-03-06T07:25:24.090718+0000","last_became_active":"2024-03-06T07:25:23.099744+0000","last_became_peered":"2024-03-06T07:25:23.099744+0000","last_unstale":"2024-03-06T07:25:24.090718+0000","last_undegraded":"2024-03-06T07:25:24.090718+0000","last_fullsized":"2024-03-06T07:25:24.090718+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:56.038731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.108239+0000","last_change":"2024-03-06T07:25:24.095978+0000","last_active":"2024-03-06T07:25:24.108239+0000","last_peered":"2024-03-06T07:25:24.108239+0000","last_clean":"2024-03-06T07:25:24.108239+0000","last_became_active":"2024-03-06T07:25:24.095676+0000","last_became_peered":"2024-03-06T07:25:24.095676+0000","last_unstale":"2024-03-06T07:25:24.108239+0000","last_undegraded":"2024-03-06T07:25:24.108239+0000","last_fullsized":"2024-03-06T07:25:24.108239+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:40.088432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.087595+0000","last_change":"2024-03-06T07:25:23.092771+0000","last_active":"2024-03-06T07:25:24.087595+0000","last_peered":"2024-03-06T07:25:24.087595+0000","last_clean":"2024-03-06T07:25:24.087595+0000","last_became_active":"2024-03-06T07:25:23.091105+0000","last_became_peered":"2024-03-06T07:25:23.091105+0000","last_unstale":"2024-03-06T07:25:24.087595+0000","last_undegraded":"2024-03-06T07:25:24.087595+0000","last_fullsized":"2024-03-06T07:25:24.087595+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:05.302915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.109647+0000","last_change":"2024-03-06T07:25:24.101852+0000","last_active":"2024-03-06T07:25:24.109647+0000","last_peered":"2024-03-06T07:25:24.109647+0000","last_clean":"2024-03-06T07:25:24.109647+0000","last_became_active":"2024-03-06T07:25:24.101461+0000","last_became_peered":"2024-03-06T07:25:24.101461+0000","last_unstale":"2024-03-06T07:25:24.109647+0000","last_undegraded":"2024-03-06T07:25:24.109647+0000","last_fullsized":"2024-03-06T07:25:24.109647+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:15.817717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.090640+0000","last_change":"2024-03-06T07:25:23.092253+0000","last_active":"2024-03-06T07:25:24.090640+0000","last_peered":"2024-03-06T07:25:24.090640+0000","last_clean":"2024-03-06T07:25:24.090640+0000","last_became_active":"2024-03-06T07:25:23.091929+0000","last_became_peered":"2024-03-06T07:25:23.091929+0000","last_unstale":"2024-03-06T07:25:24.090640+0000","last_undegraded":"2024-03-06T07:25:24.090640+0000","last_fullsized":"2024-03-06T07:25:24.090640+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:25.015102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.111171+0000","last_change":"2024-03-06T07:25:24.099930+0000","last_active":"2024-03-06T07:25:24.111171+0000","last_peered":"2024-03-06T07:25:24.111171+0000","last_clean":"2024-03-06T07:25:24.111171+0000","last_became_active":"2024-03-06T07:25:24.099666+0000","last_became_peered":"2024-03-06T07:25:24.099666+0000","last_unstale":"2024-03-06T07:25:24.111171+0000","last_undegraded":"2024-03-06T07:25:24.111171+0000","last_fullsized":"2024-03-06T07:25:24.111171+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:23.070503+0000","last_clean_scrub_stamp":"2024-03-06T07:25:23.070503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:27.533755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.094638+0000","last_change":"2024-03-06T07:25:23.089208+0000","last_active":"2024-03-06T07:25:24.094638+0000","last_peered":"2024-03-06T07:25:24.094638+0000","last_clean":"2024-03-06T07:25:24.094638+0000","last_became_active":"2024-03-06T07:25:23.088914+0000","last_became_peered":"2024-03-06T07:25:23.088914+0000","last_unstale":"2024-03-06T07:25:24.094638+0000","last_undegraded":"2024-03-06T07:25:24.094638+0000","last_fullsized":"2024-03-06T07:25:24.094638+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:22.063212+0000","last_clean_scrub_stamp":"2024-03-06T07:25:22.063212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:39.081140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:25:24.095182+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:25:24.095182+0000","last_peered":"2024-03-06T07:25:24.095182+0000","last_clean":"2024-03-06T07:25:24.095182+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:25:24.095182+0000","last_undegraded":"2024-03-06T07:25:24.095182+0000","last_fullsized":"2024-03-06T07:25:24.095182+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":872529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":684032,"data_stored":282001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":282653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":283613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":282057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":671744,"data_stored":275226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":688128,"data_stored":282091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1294336,"data_stored":874563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:28.500 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:28.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:28.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.681+0000 7fe6a2484640 1 Processor -- start 2024-03-06T07:25:28.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.681+0000 7fe6a2484640 1 -- start start 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe6a2484640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe69c0cf010 0x7fe69c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe6a2484640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c15bc40 con 0x7fe69c144640 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe6a2484640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c15bf20 con 0x7fe69c15b130 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe6a2484640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69c06df70 con 0x7fe69c0cf010 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe6a0a21640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe69c0cf010 0x7fe69c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe69bfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe69c144640 0x7fe69c06e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60256/0 (socket says 172.21.15.27:60256) 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe69bfff640 1 -- 172.21.15.27:0/3838936203 learned_addr learned my addr 172.21.15.27:0/3838936203 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe6a0a21640 1 -- 172.21.15.27:0/3838936203 >> v1:172.21.15.69:6789/0 conn(0x7fe69c15b130 legacy=0x7fe69c0ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe6a0a21640 1 -- 172.21.15.27:0/3838936203 >> v1:172.21.15.27:6789/0 conn(0x7fe69c144640 legacy=0x7fe69c06e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.682+0000 7fe6a0a21640 1 -- 172.21.15.27:0/3838936203 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c06dc00 con 0x7fe69c0cf010 2024-03-06T07:25:28.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.683+0000 7fe6a0a21640 1 --2- 172.21.15.27:0/3838936203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c0cf010 0x7fe69c15b5c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe694009d90 tx=0x7fe69402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8b69c9ad710b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.683+0000 7fe69affd640 1 -- 172.21.15.27:0/3838936203 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe694039070 con 0x7fe69c0cf010 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.683+0000 7fe69affd640 1 -- 172.21.15.27:0/3838936203 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69402cc60 con 0x7fe69c0cf010 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.683+0000 7fe69affd640 1 -- 172.21.15.27:0/3838936203 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe694034660 con 0x7fe69c0cf010 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.684+0000 7fe6a2484640 1 -- 172.21.15.27:0/3838936203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c0cf010 msgr2=0x7fe69c15b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.684+0000 7fe6a2484640 1 --2- 172.21.15.27:0/3838936203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c0cf010 0x7fe69c15b5c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe694009d90 tx=0x7fe69402beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.684+0000 7fe6a2484640 1 -- 172.21.15.27:0/3838936203 shutdown_connections 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.684+0000 7fe6a2484640 1 --2- 172.21.15.27:0/3838936203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c0cf010 0x7fe69c15b5c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.684+0000 7fe6a2484640 1 -- 172.21.15.27:0/3838936203 >> 172.21.15.27:0/3838936203 conn(0x7fe69c066d00 msgr2=0x7fe69c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.684+0000 7fe6a2484640 1 -- 172.21.15.27:0/3838936203 shutdown_connections 2024-03-06T07:25:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.684+0000 7fe6a2484640 1 -- 172.21.15.27:0/3838936203 wait complete. 2024-03-06T07:25:28.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.685+0000 7fe6a2484640 1 Processor -- start 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.685+0000 7fe6a2484640 1 -- start start 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe6a2484640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69c0cf010 0x7fe69c159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe6a2484640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c144640 0x7fe69c13d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe6a2484640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe69c15b130 0x7fe69c13f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe6a2484640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c070c20 con 0x7fe69c144640 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe6a2484640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c070aa0 con 0x7fe69c0cf010 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe6a2484640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c0b36d0 con 0x7fe69c15b130 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe69b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c144640 0x7fe69c13d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe69b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c144640 0x7fe69c13d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42142/0 (socket says 172.21.15.27:42142) 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe69b7fe640 1 -- 172.21.15.27:0/997092396 learned_addr learned my addr 172.21.15.27:0/997092396 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe69bfff640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69c0cf010 0x7fe69c159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe6a0a21640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe69c15b130 0x7fe69c13f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe69b7fe640 1 -- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe69c15b130 msgr2=0x7fe69c13f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe69b7fe640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe69c15b130 0x7fe69c13f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe69b7fe640 1 -- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69c0cf010 msgr2=0x7fe69c159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe69b7fe640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69c0cf010 0x7fe69c159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.686+0000 7fe69b7fe640 1 -- 172.21.15.27:0/997092396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c13c940 con 0x7fe69c144640 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.687+0000 7fe69b7fe640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c144640 0x7fe69c13d9e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00b3e0 tx=0x7fe68c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.687+0000 7fe698ff9640 1 -- 172.21.15.27:0/997092396 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c015020 con 0x7fe69c144640 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.687+0000 7fe698ff9640 1 -- 172.21.15.27:0/997092396 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68c011850 con 0x7fe69c144640 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.687+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69c13cc20 con 0x7fe69c144640 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.687+0000 7fe698ff9640 1 -- 172.21.15.27:0/997092396 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c011c40 con 0x7fe69c144640 2024-03-06T07:25:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.687+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69c13d1e0 con 0x7fe69c144640 2024-03-06T07:25:28.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.688+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69c0733c0 con 0x7fe69c144640 2024-03-06T07:25:28.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.688+0000 7fe698ff9640 1 -- 172.21.15.27:0/997092396 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe68c018020 con 0x7fe69c144640 2024-03-06T07:25:28.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.689+0000 7fe698ff9640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe68806b500 0x7fe68806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.689+0000 7fe698ff9640 1 -- 172.21.15.27:0/997092396 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe68c08a040 con 0x7fe69c144640 2024-03-06T07:25:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.689+0000 7fe69bfff640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe68806b500 0x7fe68806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.689+0000 7fe69bfff640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe68806b500 0x7fe68806d9c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe690007fd0 tx=0x7fe690007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.698+0000 7fe698ff9640 1 -- 172.21.15.27:0/997092396 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe68c01f020 con 0x7fe69c144640 2024-03-06T07:25:28.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.949+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe69c1406f0 con 0x7fe69c144640 2024-03-06T07:25:28.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.951+0000 7fe698ff9640 1 -- 172.21.15.27:0/997092396 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe68c058870 con 0x7fe69c144640 2024-03-06T07:25:28.954 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:25:28.954 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:25:28.954 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-03-06T07:25:28.954 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-03-06T07:25:28.954 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:28.955 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-03-06T07:25:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe68806b500 msgr2=0x7fe68806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe68806b500 0x7fe68806d9c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe690007fd0 tx=0x7fe690007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c144640 msgr2=0x7fe69c13d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c144640 0x7fe69c13d9e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00b3e0 tx=0x7fe68c00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 shutdown_connections 2024-03-06T07:25:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe68806b500 0x7fe68806d9c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe69c15b130 0x7fe69c13f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69c144640 0x7fe69c13d9e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 --2- 172.21.15.27:0/997092396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69c0cf010 0x7fe69c159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 >> 172.21.15.27:0/997092396 conn(0x7fe69c066d00 msgr2=0x7fe69c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 shutdown_connections 2024-03-06T07:25:28.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:28.954+0000 7fe6a2484640 1 -- 172.21.15.27:0/997092396 wait complete. 2024-03-06T07:25:28.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:29.066 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:25:29.064+0000 7f0536f26640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.072+0000 7f4587688640 1 Processor -- start 2024-03-06T07:25:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.072+0000 7f4587688640 1 -- start start 2024-03-06T07:25:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.073+0000 7f4587688640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45800d3400 0x7f45800b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.073+0000 7f4587688640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f45800b2b50 0x7f45800b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.073+0000 7f4587688640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45800ce610 con 0x7f4580071540 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.073+0000 7f4587688640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45800d2f70 con 0x7f45800d3400 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.073+0000 7f4587688640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45800d4be0 con 0x7f45800b2b50 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.073+0000 7f4584c23640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45800d3400 0x7f45800b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.073+0000 7f4585424640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4580071540 0x7f4580071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60258/0 (socket says 172.21.15.27:60258) 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.073+0000 7f4585424640 1 -- 172.21.15.27:0/929314493 learned_addr learned my addr 172.21.15.27:0/929314493 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.073+0000 7f4585c25640 1 --2- 172.21.15.27:0/929314493 >> v2:172.21.15.69:3300/0 conn(0x7f45800b2b50 0x7f45800b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.074+0000 7f4584c23640 1 -- 172.21.15.27:0/929314493 >> v2:172.21.15.69:3300/0 conn(0x7f45800b2b50 msgr2=0x7f45800b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.074+0000 7f4584c23640 1 --2- 172.21.15.27:0/929314493 >> v2:172.21.15.69:3300/0 conn(0x7f45800b2b50 0x7f45800b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.074+0000 7f4584c23640 1 -- 172.21.15.27:0/929314493 >> v1:172.21.15.27:6789/0 conn(0x7f4580071540 legacy=0x7f4580071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.074+0000 7f4584c23640 1 -- 172.21.15.27:0/929314493 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45800d2920 con 0x7f45800d3400 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.074+0000 7f4585c25640 1 --2- 172.21.15.27:0/929314493 >> v2:172.21.15.69:3300/0 conn(0x7f45800b2b50 0x7f45800b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.074+0000 7f4584c23640 1 --2- 172.21.15.27:0/929314493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 0x7f45800b30a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f45800683b0 tx=0x7f457002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d748f8d1926e8e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.074+0000 7f4577fff640 1 -- 172.21.15.27:0/929314493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570035020 con 0x7f45800d3400 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.074+0000 7f4577fff640 1 -- 172.21.15.27:0/929314493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4570031830 con 0x7f45800d3400 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.074+0000 7f4577fff640 1 -- 172.21.15.27:0/929314493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570031c00 con 0x7f45800d3400 2024-03-06T07:25:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.075+0000 7f4587688640 1 -- 172.21.15.27:0/929314493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 msgr2=0x7f45800b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.075+0000 7f4587688640 1 --2- 172.21.15.27:0/929314493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 0x7f45800b30a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f45800683b0 tx=0x7f457002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.075+0000 7f4587688640 1 -- 172.21.15.27:0/929314493 shutdown_connections 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.075+0000 7f4587688640 1 --2- 172.21.15.27:0/929314493 >> v2:172.21.15.69:3300/0 conn(0x7f45800b2b50 0x7f45800b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.075+0000 7f4587688640 1 --2- 172.21.15.27:0/929314493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 0x7f45800b30a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.075+0000 7f4587688640 1 -- 172.21.15.27:0/929314493 >> 172.21.15.27:0/929314493 conn(0x7f45800d3ca0 msgr2=0x7f45800d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.075+0000 7f4587688640 1 -- 172.21.15.27:0/929314493 shutdown_connections 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.075+0000 7f4587688640 1 -- 172.21.15.27:0/929314493 wait complete. 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.076+0000 7f4587688640 1 Processor -- start 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.076+0000 7f4587688640 1 -- start start 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4587688640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4580071540 0x7f45801522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4587688640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45800b2b50 0x7f45800b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4587688640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 0x7f45800b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4587688640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458015e370 con 0x7f45800d3400 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4587688640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45800ce8f0 con 0x7f4580071540 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4587688640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45800cad70 con 0x7f45800b2b50 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4585c25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 0x7f45800b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4585c25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 0x7f45800b4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42152/0 (socket says 172.21.15.27:42152) 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4585c25640 1 -- 172.21.15.27:0/2894218967 learned_addr learned my addr 172.21.15.27:0/2894218967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4585c25640 1 -- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45800b2b50 msgr2=0x7f45800b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4585c25640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45800b2b50 0x7f45800b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4585c25640 1 -- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4580071540 msgr2=0x7f45801522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4585c25640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4580071540 0x7f45801522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.077+0000 7f4585c25640 1 -- 172.21.15.27:0/2894218967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45800b40c0 con 0x7f45800d3400 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.078+0000 7f4585c25640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 0x7f45800b4e70 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4578007ef0 tx=0x7f457800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.078+0000 7f4575ffb640 1 -- 172.21.15.27:0/2894218967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457801a070 con 0x7f45800d3400 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.078+0000 7f4575ffb640 1 -- 172.21.15.27:0/2894218967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4578003b90 con 0x7f45800d3400 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.078+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45800b43a0 con 0x7f45800d3400 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.078+0000 7f4575ffb640 1 -- 172.21.15.27:0/2894218967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578004030 con 0x7f45800d3400 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.078+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45800cafc0 con 0x7f45800d3400 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.079+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4580068700 con 0x7f45800d3400 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.079+0000 7f4575ffb640 1 -- 172.21.15.27:0/2894218967 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f457801d070 con 0x7f45800d3400 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.080+0000 7f4575ffb640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456406b4d0 0x7f456406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.080+0000 7f4575ffb640 1 -- 172.21.15.27:0/2894218967 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f457808a440 con 0x7f45800d3400 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.080+0000 7f4585424640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456406b4d0 0x7f456406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.081+0000 7f4585424640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456406b4d0 0x7f456406d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f457c000f60 tx=0x7f457c007a80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.084+0000 7f4575ffb640 1 -- 172.21.15.27:0/2894218967 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4578010070 con 0x7f45800d3400 2024-03-06T07:25:29.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.308+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f458006b8e0 con 0x7f45800d3400 2024-03-06T07:25:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.308+0000 7f4575ffb640 1 -- 172.21.15.27:0/2894218967 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f4578058c70 con 0x7f45800d3400 2024-03-06T07:25:29.311 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:29.311 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5324,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3885016831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3885016831},{"type":"v1","addr":"172.21.15.69:6836","nonce":3885016831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5327,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3510003111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3510003111},{"type":"v1","addr":"172.21.15.69:6837","nonce":3510003111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5692,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3053251134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3053251134},{"type":"v1","addr":"172.21.15.27:6835","nonce":3053251134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:23.445711+0000","modified":"2024-03-06T07:25:26.468573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6129":{"gid":6129,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6833/60877422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":60877422},{"type":"v1","addr":"172.21.15.27:6833","nonce":60877422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6129,"qdb_cluster":[6129]},"id":4}]} 2024-03-06T07:25:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456406b4d0 msgr2=0x7f456406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456406b4d0 0x7f456406d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f457c000f60 tx=0x7f457c007a80 comp rx=0 tx=0).stop 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 msgr2=0x7f45800b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 0x7f45800b4e70 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4578007ef0 tx=0x7f457800d4a0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 shutdown_connections 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f456406b4d0 0x7f456406d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45800d3400 0x7f45800b4e70 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f45800b2b50 0x7f45800b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 --2- 172.21.15.27:0/2894218967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4580071540 0x7f45801522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 >> 172.21.15.27:0/2894218967 conn(0x7f45800d3ca0 msgr2=0x7f45800ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 shutdown_connections 2024-03-06T07:25:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.310+0000 7f4587688640 1 -- 172.21.15.27:0/2894218967 wait complete. 2024-03-06T07:25:29.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:23.445711+0000', 'modified': '2024-03-06T07:25:26.468573+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6129}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6129': {'gid': 6129, 'name': 'a', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6833/60877422', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 60877422}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 60877422}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6129, 'qdb_cluster': [6129]} 2024-03-06T07:25:29.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:29.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:25:29.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.831+0000 7f4ebb91e640 1 Processor -- start 2024-03-06T07:25:29.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.831+0000 7f4ebb91e640 1 -- start start 2024-03-06T07:25:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4ebb91e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4eb40b0e90 0x7f4eb406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4ebb91e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4eb40ae9b0 0x7f4eb40d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4ebb91e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb40cee90 con 0x7f4eb406b5c0 2024-03-06T07:25:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4ebb91e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb40cfa00 con 0x7f4eb40b0e90 2024-03-06T07:25:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4ebb91e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb40cfc50 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4eb8eb9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4eb40b0e90 0x7f4eb406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4eb8eb9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4eb40b0e90 0x7f4eb406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42168/0 (socket says 172.21.15.27:42168) 2024-03-06T07:25:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/3395951070 learned_addr learned my addr 172.21.15.27:0/3395951070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:29.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/3395951070 >> v2:172.21.15.69:3300/0 conn(0x7f4eb40ae9b0 0x7f4eb40d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/3395951070 >> v2:172.21.15.69:3300/0 conn(0x7f4eb40ae9b0 msgr2=0x7f4eb40d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/3395951070 >> v2:172.21.15.69:3300/0 conn(0x7f4eb40ae9b0 0x7f4eb40d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/3395951070 >> v1:172.21.15.69:6789/0 conn(0x7f4eb406b5c0 legacy=0x7f4eb40b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.832+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/3395951070 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb40cff30 con 0x7f4eb40b0e90 2024-03-06T07:25:29.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.833+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/3395951070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40b0e90 0x7f4eb406b9a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4072170 tx=0x7f4ea402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355d8095c5d4a766 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.833+0000 7f4eabfff640 1 -- 172.21.15.27:0/3395951070 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4035020 con 0x7f4eb40b0e90 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.833+0000 7f4eabfff640 1 -- 172.21.15.27:0/3395951070 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea4031870 con 0x7f4eb40b0e90 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.833+0000 7f4eabfff640 1 -- 172.21.15.27:0/3395951070 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4031c40 con 0x7f4eb40b0e90 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.834+0000 7f4ebb91e640 1 -- 172.21.15.27:0/3395951070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40b0e90 msgr2=0x7f4eb406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.834+0000 7f4ebb91e640 1 --2- 172.21.15.27:0/3395951070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40b0e90 0x7f4eb406b9a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4072170 tx=0x7f4ea402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.834+0000 7f4ebb91e640 1 -- 172.21.15.27:0/3395951070 shutdown_connections 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.834+0000 7f4ebb91e640 1 --2- 172.21.15.27:0/3395951070 >> v2:172.21.15.69:3300/0 conn(0x7f4eb40ae9b0 0x7f4eb40d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.834+0000 7f4ebb91e640 1 --2- 172.21.15.27:0/3395951070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40b0e90 0x7f4eb406b9a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.834+0000 7f4ebb91e640 1 -- 172.21.15.27:0/3395951070 >> 172.21.15.27:0/3395951070 conn(0x7f4eb40b36d0 msgr2=0x7f4eb40b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.834+0000 7f4ebb91e640 1 -- 172.21.15.27:0/3395951070 shutdown_connections 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.834+0000 7f4ebb91e640 1 -- 172.21.15.27:0/3395951070 wait complete. 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.835+0000 7f4ebb91e640 1 Processor -- start 2024-03-06T07:25:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.835+0000 7f4ebb91e640 1 -- start start 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4ebb91e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4eb406b5c0 0x7f4eb41dd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4ebb91e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40ae9b0 0x7f4eb40cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4ebb91e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb40b0e90 0x7f4eb406f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4ebb91e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb406c910 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4ebb91e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb406c790 con 0x7f4eb40b0e90 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4ebb91e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb40aedc0 con 0x7f4eb406b5c0 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4eb8eb9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40ae9b0 0x7f4eb40cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4eb8eb9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40ae9b0 0x7f4eb40cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42170/0 (socket says 172.21.15.27:42170) 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/2301754595 learned_addr learned my addr 172.21.15.27:0/2301754595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb40b0e90 0x7f4eb406f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4eb406b5c0 0x7f4eb41dd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4eb406b5c0 msgr2=0x7f4eb41dd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4eb406b5c0 0x7f4eb41dd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb40b0e90 msgr2=0x7f4eb406f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.836+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb40b0e90 0x7f4eb406f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.837+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/2301754595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb40c5b30 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.837+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40ae9b0 0x7f4eb40cd700 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4ea402b950 tx=0x7f4ea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.837+0000 7f4ea9ffb640 1 -- 172.21.15.27:0/2301754595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4040070 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.837+0000 7f4ea9ffb640 1 -- 172.21.15.27:0/2301754595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea4030d00 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.837+0000 7f4ebb91e640 1 -- 172.21.15.27:0/2301754595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb40c5e10 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.837+0000 7f4ea9ffb640 1 -- 172.21.15.27:0/2301754595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4035040 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.837+0000 7f4ebb91e640 1 -- 172.21.15.27:0/2301754595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb40c63d0 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.838+0000 7f4ea9ffb640 1 -- 172.21.15.27:0/2301754595 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ea4038020 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.839+0000 7f4ea9ffb640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e9806b4d0 0x7f4e9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.839+0000 7f4ea9ffb640 1 -- 172.21.15.27:0/2301754595 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ea40abe70 con 0x7f4eb40ae9b0 2024-03-06T07:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.839+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e9806b4d0 0x7f4e9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.840+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e9806b4d0 0x7f4e9806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0001780 tx=0x7f4eb0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.841+0000 7f4ebe5e5740 1 Processor -- start 2024-03-06T07:25:29.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.842+0000 7f4ebe5e5740 1 -- start start 2024-03-06T07:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d60870 0x561124d60c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d648c0 0x561124d66360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 0x561124d90a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561124cca6b0 con 0x561124d60870 2024-03-06T07:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x561124c73a60 con 0x561124d90600 2024-03-06T07:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x561124cc6750 con 0x561124d648c0 2024-03-06T07:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb96ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d60870 0x561124d60c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb96ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d60870 0x561124d60c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42176/0 (socket says 172.21.15.27:42176) 2024-03-06T07:25:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb96ba640 1 -- 172.21.15.27:0/1048251220 learned_addr learned my addr 172.21.15.27:0/1048251220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d648c0 0x561124d66360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 0x561124d90a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb96ba640 1 -- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d648c0 msgr2=0x561124d66360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d648c0 0x561124d66360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb96ba640 1 -- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 msgr2=0x561124d90a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 0x561124d90a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb96ba640 1 -- 172.21.15.27:0/1048251220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561124c91f30 con 0x561124d60870 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d648c0 0x561124d66360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 0x561124d90a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.843+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d60870 0x561124d60c50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000fbd0 tx=0x7f4eb0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba492bf74ad3785 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4e9effd640 1 -- 172.21.15.27:0/1048251220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0026070 con 0x561124d60870 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4e9effd640 1 -- 172.21.15.27:0/1048251220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb0006e10 con 0x561124d60870 2024-03-06T07:25:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4e9effd640 1 -- 172.21.15.27:0/1048251220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0007440 con 0x561124d60870 2024-03-06T07:25:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/1048251220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d60870 msgr2=0x561124d60c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d60870 0x561124d60c50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000fbd0 tx=0x7f4eb0012000 comp rx=0 tx=0).stop 2024-03-06T07:25:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/1048251220 shutdown_connections 2024-03-06T07:25:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 0x561124d90a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d648c0 0x561124d66360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/1048251220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d60870 0x561124d60c50 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/1048251220 >> 172.21.15.27:0/1048251220 conn(0x561124d628f0 msgr2=0x561124d641b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/1048251220 shutdown_connections 2024-03-06T07:25:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.844+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/1048251220 wait complete. 2024-03-06T07:25:29.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.845+0000 7f4ebe5e5740 1 -- start start 2024-03-06T07:25:29.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.845+0000 7f4ebe5e5740 1 Processor -- start 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d90600 0x561124d82bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 0x561124d866e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d836a0 0x561124d86c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561124d64d50 con 0x561124d836a0 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x561124c86ea0 con 0x561124d851e0 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x561124d65030 con 0x561124d90600 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4ebe5e5740 10 client.0 _ll_register_callbacks cb 0x561124c86870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb9ebb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d836a0 0x561124d86c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb9ebb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d836a0 0x561124d86c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42188/0 (socket says 172.21.15.27:42188) 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb9ebb640 1 -- 172.21.15.27:0/538937109 learned_addr learned my addr 172.21.15.27:0/538937109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 0x561124d866e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d90600 0x561124d82bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb9ebb640 1 -- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d90600 msgr2=0x561124d82bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d90600 0x561124d82bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb9ebb640 1 -- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 msgr2=0x561124d866e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 0x561124d866e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb9ebb640 1 -- 172.21.15.27:0/538937109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561124c87270 con 0x561124d836a0 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d90600 0x561124d82bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.847+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 0x561124d866e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d836a0 0x561124d86c20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f4eac004630 tx=0x7f4eac007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2302204435538d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4e93fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4e93fff640 1 -- 172.21.15.27:0/538937109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac019020 con 0x561124d836a0 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4e93fff640 1 -- 172.21.15.27:0/538937109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4eac004720 con 0x561124d836a0 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/538937109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4eb0006e10 con 0x561124d836a0 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4e93fff640 1 -- 172.21.15.27:0/538937109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4eac023050 con 0x561124d836a0 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4ebe5e5740 10 client.6276 fetch_fsmap learned FSMap version 34 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/538937109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x5611249df2d0 con 0x561124d836a0 2024-03-06T07:25:29.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4e93fff640 1 -- 172.21.15.27:0/538937109 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb0006e10 con 0x561124d836a0 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4e93fff640 1 -- 172.21.15.27:0/538937109 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4eac026020 con 0x561124d836a0 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4ebe5e5740 10 client.6276 fetch_fsmap finished waiting for FSMap version 34 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4ebe5e5740 10 client.6276 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4ebe5e5740 20 client.6276 populate_metadata read hostname 'smithi027' 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] conn(0x561124d8b6d0 0x561124dca600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4ebe5e5740 4 client.6276 mds_command: new command op to 6129 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/538937109 --> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561124cd7bf0 con 0x561124d8b6d0 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.848+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] conn(0x561124d8b6d0 0x561124dca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.849+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] conn(0x561124d8b6d0 0x561124dca600 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8a4f27127c1a86f server_cookie=34895038f3bf14aa in_seq=0 out_seq=0 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.849+0000 7f4e93fff640 10 client.6276 ms_handle_connect on v2:172.21.15.27:6832/60877422 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.849+0000 7f4e93fff640 1 -- 172.21.15.27:0/538937109 <== mds.0 v2:172.21.15.27:6832/60877422 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561124cd7bf0 con 0x561124d8b6d0 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.849+0000 7f4e93fff640 10 client.6276 handle_command_reply: tid=0 2024-03-06T07:25:29.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.849+0000 7f4ebe5e5740 1 client.6276 shutdown 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.849+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/538937109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d836a0 msgr2=0x561124d86c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.849+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d836a0 0x561124d86c20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f4eac004630 tx=0x7f4eac007700 comp rx=0 tx=0).stop 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/538937109 shutdown_connections 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d836a0 0x561124d86c20 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 0x561124d866e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] conn(0x561124d8b6d0 0x561124dca600 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/538937109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d90600 0x561124d82bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4e93fff640 0 client.6276 ms_handle_reset on v2:172.21.15.27:6832/60877422 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/538937109 >> 172.21.15.27:0/538937109 conn(0x561124d60870 msgr2=0x561124d8e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/538937109 shutdown_connections 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/538937109 wait complete. 2024-03-06T07:25:29.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.850+0000 7f4ebe5e5740 20 client.6276 trim_cache size 0 max 16384 2024-03-06T07:25:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.866+0000 7f4ebe5e5740 1 Processor -- start 2024-03-06T07:25:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4ebe5e5740 1 -- start start 2024-03-06T07:25:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 0x561124d64080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d851e0 0x561124dacac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 0x561124d86480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb40c79c0 con 0x561124d851e0 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb40cc9e0 con 0x561124d90600 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb40c73c0 con 0x561124d836a0 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4eb8eb9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d851e0 0x561124dacac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4eb8eb9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d851e0 0x561124dacac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42190/0 (socket says 172.21.15.27:42190) 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.867+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/2433527229 learned_addr learned my addr 172.21.15.27:0/2433527229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/2433527229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 0x561124d64080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/2433527229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 0x561124d86480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/2433527229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 msgr2=0x561124d64080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/2433527229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 0x561124d64080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/2433527229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 msgr2=0x561124d86480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/2433527229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 0x561124d86480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4eb8eb9640 1 -- 172.21.15.27:0/2433527229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561124aa96b0 con 0x561124d851e0 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/2433527229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d851e0 0x561124dacac0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4006b10 tx=0x7f4ea403c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa9961c4ed858a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4e9e7fc640 1 -- 172.21.15.27:0/2433527229 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea407a730 con 0x561124d851e0 2024-03-06T07:25:29.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4e9e7fc640 1 -- 172.21.15.27:0/2433527229 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea407e7d0 con 0x561124d851e0 2024-03-06T07:25:29.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4e9e7fc640 1 -- 172.21.15.27:0/2433527229 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4082d90 con 0x561124d851e0 2024-03-06T07:25:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/2433527229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d851e0 msgr2=0x561124dacac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/2433527229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d851e0 0x561124dacac0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4006b10 tx=0x7f4ea403c040 comp rx=0 tx=0).stop 2024-03-06T07:25:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/2433527229 shutdown_connections 2024-03-06T07:25:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/2433527229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d90600 0x561124d86480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/2433527229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d851e0 0x561124dacac0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/2433527229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 0x561124d64080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.868+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/2433527229 >> 172.21.15.27:0/2433527229 conn(0x561124d60870 msgr2=0x561124d8e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.869+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/2433527229 shutdown_connections 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.869+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/2433527229 wait complete. 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.869+0000 7f4ebe5e5740 1 -- start start 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.869+0000 7f4ebe5e5740 1 Processor -- start 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 0x561124d86450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 0x561124d86990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4ebe5e5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d90600 0x561124d83e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561124d83470 con 0x561124d90600 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x561124d91ee0 con 0x561124d851e0 2024-03-06T07:25:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4ebe5e5740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x561124d885f0 con 0x561124d836a0 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4ebe5e5740 10 client.0 _ll_register_callbacks cb 0x7f4eac004270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb9ebb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d90600 0x561124d83e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb9ebb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d90600 0x561124d83e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42204/0 (socket says 172.21.15.27:42204) 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb9ebb640 1 -- 172.21.15.27:0/3542691228 learned_addr learned my addr 172.21.15.27:0/3542691228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb8eb9640 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 0x561124d86990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 0x561124d86450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb9ebb640 1 -- 172.21.15.27:0/3542691228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 msgr2=0x561124d86450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 0x561124d86450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb9ebb640 1 -- 172.21.15.27:0/3542691228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 msgr2=0x561124d86990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 0x561124d86990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.870+0000 7f4eb9ebb640 1 -- 172.21.15.27:0/3542691228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561124d5f640 con 0x561124d90600 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4eb9ebb640 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d90600 0x561124d83e00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4eac01ed90 tx=0x7f4eac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65512ea2545610ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4e93fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:25:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4e93fff640 1 -- 172.21.15.27:0/3542691228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac019070 con 0x561124d90600 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4e93fff640 1 -- 172.21.15.27:0/3542691228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4eac004720 con 0x561124d90600 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/3542691228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ea407e7d0 con 0x561124d90600 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4e93fff640 1 -- 172.21.15.27:0/3542691228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4eac036050 con 0x561124d90600 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4ebe5e5740 10 client.6282 fetch_fsmap learned FSMap version 34 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/3542691228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x561124d5ff40 con 0x561124d90600 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4e93fff640 1 -- 172.21.15.27:0/3542691228 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea407e7d0 con 0x561124d90600 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4e93fff640 1 -- 172.21.15.27:0/3542691228 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4eac039020 con 0x561124d90600 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4ebe5e5740 10 client.6282 fetch_fsmap finished waiting for FSMap version 34 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4ebe5e5740 10 client.6282 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4ebe5e5740 20 client.6282 populate_metadata read hostname 'smithi027' 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] conn(0x561124d849b0 0x561124d64f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4ebe5e5740 4 client.6282 mds_command: new command op to 6129 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/3542691228 --> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561124cd1f40 con 0x561124d849b0 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.871+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] conn(0x561124d849b0 0x561124d64f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4e93fff640 10 client.6282 ms_handle_connect on v2:172.21.15.27:6832/60877422 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4eb96ba640 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] conn(0x561124d849b0 0x561124d64f60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf0ddff773a1c8eb server_cookie=b74589758f76fb2 in_seq=0 out_seq=0 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4e93fff640 1 -- 172.21.15.27:0/3542691228 <== mds.0 v2:172.21.15.27:6832/60877422 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x561124cd1f40 con 0x561124d849b0 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4e93fff640 10 client.6282 handle_command_reply: tid=0 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 client.6282 shutdown 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/3542691228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d90600 msgr2=0x561124d83e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d90600 0x561124d83e00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4eac01ed90 tx=0x7f4eac004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/3542691228 shutdown_connections 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561124d90600 0x561124d83e00 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561124d851e0 0x561124d86990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.27:6832/60877422,v1:172.21.15.27:6833/60877422] conn(0x561124d849b0 0x561124d64f60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 --2- 172.21.15.27:0/3542691228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561124d836a0 0x561124d86450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4e93fff640 0 client.6282 ms_handle_reset on v2:172.21.15.27:6832/60877422 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/3542691228 >> 172.21.15.27:0/3542691228 conn(0x561124d60870 msgr2=0x561124d89400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/3542691228 shutdown_connections 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 1 -- 172.21.15.27:0/3542691228 wait complete. 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.872+0000 7f4ebe5e5740 20 client.6282 trim_cache size 0 max 16384 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 6129, 2024-03-06T07:25:29.881 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:25:29.882 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:25:29.882 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:25:29.882 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 5.7771879520000002, 2024-03-06T07:25:29.882 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 34, 2024-03-06T07:25:29.882 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 58, 2024-03-06T07:25:29.882 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 58, 2024-03-06T07:25:29.882 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 11.793325534999999 2024-03-06T07:25:29.882 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:25:29.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 -- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e9806b4d0 msgr2=0x7f4e9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e9806b4d0 0x7f4e9806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0001780 tx=0x7f4eb0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:29.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 -- 172.21.15.27:0/2301754595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40ae9b0 msgr2=0x7f4eb40cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40ae9b0 0x7f4eb40cd700 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4ea402b950 tx=0x7f4ea40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 -- 172.21.15.27:0/2301754595 shutdown_connections 2024-03-06T07:25:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e9806b4d0 0x7f4e9806d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb40b0e90 0x7f4eb406f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4eb40ae9b0 0x7f4eb40cd700 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 --2- 172.21.15.27:0/2301754595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4eb406b5c0 0x7f4eb41dd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 -- 172.21.15.27:0/2301754595 >> 172.21.15.27:0/2301754595 conn(0x7f4eb40b36d0 msgr2=0x7f4eb40b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 -- 172.21.15.27:0/2301754595 shutdown_connections 2024-03-06T07:25:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.873+0000 7f4ebb91e640 1 -- 172.21.15.27:0/2301754595 wait complete. 2024-03-06T07:25:29.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.990+0000 7fcd53af4640 1 Processor -- start 2024-03-06T07:25:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.991+0000 7fcd53af4640 1 -- start start 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.991+0000 7fcd53af4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c15df00 0x7fcd4c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.991+0000 7fcd53af4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4c0cea30 con 0x7fcd4c0b95a0 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.991+0000 7fcd53af4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4c0d2af0 con 0x7fcd4c0b9a30 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.991+0000 7fcd53af4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c0cf3c0 con 0x7fcd4c15df00 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd5108f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcd4c0b95a0 0x7fcd4c0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60268/0 (socket says 172.21.15.27:60268) 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd5108f640 1 -- 172.21.15.27:0/489504551 learned_addr learned my addr 172.21.15.27:0/489504551 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd51890640 1 --2- 172.21.15.27:0/489504551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c15df00 0x7fcd4c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd5088e640 1 -- 172.21.15.27:0/489504551 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855405738 0 0) 0x7fcd4c0cea30 con 0x7fcd4c0b95a0 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd5088e640 1 -- 172.21.15.27:0/489504551 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd34003520 con 0x7fcd4c0b95a0 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd5088e640 1 -- 172.21.15.27:0/489504551 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870525988 0 0) 0x7fcd4c0d2af0 con 0x7fcd4c0b9a30 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd5088e640 1 -- 172.21.15.27:0/489504551 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd4c0cea30 con 0x7fcd4c0b9a30 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd51890640 1 -- 172.21.15.27:0/489504551 >> v1:172.21.15.69:6789/0 conn(0x7fcd4c0b9a30 legacy=0x7fcd4c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd51890640 1 -- 172.21.15.27:0/489504551 >> v1:172.21.15.27:6789/0 conn(0x7fcd4c0b95a0 legacy=0x7fcd4c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.992+0000 7fcd51890640 1 -- 172.21.15.27:0/489504551 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4c0cf0e0 con 0x7fcd4c15df00 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.993+0000 7fcd51890640 1 --2- 172.21.15.27:0/489504551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c15df00 0x7fcd4c0d25b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcd440097d0 tx=0x7fcd4402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=738ca187dd510815 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.993+0000 7fcd5088e640 1 -- 172.21.15.27:0/489504551 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44035020 con 0x7fcd4c15df00 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.993+0000 7fcd5088e640 1 -- 172.21.15.27:0/489504551 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd44031890 con 0x7fcd4c15df00 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.993+0000 7fcd5088e640 1 -- 172.21.15.27:0/489504551 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44031c60 con 0x7fcd4c15df00 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.994+0000 7fcd53af4640 1 -- 172.21.15.27:0/489504551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c15df00 msgr2=0x7fcd4c0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.994+0000 7fcd53af4640 1 --2- 172.21.15.27:0/489504551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c15df00 0x7fcd4c0d25b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcd440097d0 tx=0x7fcd4402b930 comp rx=0 tx=0).stop 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.994+0000 7fcd53af4640 1 -- 172.21.15.27:0/489504551 shutdown_connections 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.994+0000 7fcd53af4640 1 --2- 172.21.15.27:0/489504551 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c15df00 0x7fcd4c0d25b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.994+0000 7fcd53af4640 1 -- 172.21.15.27:0/489504551 >> 172.21.15.27:0/489504551 conn(0x7fcd4c071960 msgr2=0x7fcd4c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.994+0000 7fcd53af4640 1 -- 172.21.15.27:0/489504551 shutdown_connections 2024-03-06T07:25:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.994+0000 7fcd53af4640 1 -- 172.21.15.27:0/489504551 wait complete. 2024-03-06T07:25:30.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.995+0000 7fcd53af4640 1 Processor -- start 2024-03-06T07:25:30.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.995+0000 7fcd53af4640 1 -- start start 2024-03-06T07:25:30.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd53af4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c0b95a0 0x7fcd4c160e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd53af4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4c15df00 0x7fcd4c0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd53af4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd4c0cc880 0x7fcd4c0c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd53af4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c0cec70 con 0x7fcd4c0cc880 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd53af4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c0ce750 con 0x7fcd4c15df00 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd53af4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c0cedf0 con 0x7fcd4c0b95a0 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd52091640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd4c0cc880 0x7fcd4c0c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd52091640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd4c0cc880 0x7fcd4c0c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42210/0 (socket says 172.21.15.27:42210) 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd52091640 1 -- 172.21.15.27:0/898156811 learned_addr learned my addr 172.21.15.27:0/898156811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd51890640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c0b95a0 0x7fcd4c160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd5108f640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4c15df00 0x7fcd4c0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd52091640 1 -- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c0b95a0 msgr2=0x7fcd4c160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd52091640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c0b95a0 0x7fcd4c160e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd52091640 1 -- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4c15df00 msgr2=0x7fcd4c0cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd52091640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4c15df00 0x7fcd4c0cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd52091640 1 -- 172.21.15.27:0/898156811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4c15e410 con 0x7fcd4c0cc880 2024-03-06T07:25:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd51890640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c0b95a0 0x7fcd4c160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd5108f640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4c15df00 0x7fcd4c0cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.996+0000 7fcd52091640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd4c0cc880 0x7fcd4c0c9ea0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00c7d0 tx=0x7fcd3c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.997+0000 7fcd4a7fc640 1 -- 172.21.15.27:0/898156811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c003b70 con 0x7fcd4c0cc880 2024-03-06T07:25:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.997+0000 7fcd4a7fc640 1 -- 172.21.15.27:0/898156811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3c003d10 con 0x7fcd4c0cc880 2024-03-06T07:25:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.997+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4c15e6f0 con 0x7fcd4c0cc880 2024-03-06T07:25:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.997+0000 7fcd4a7fc640 1 -- 172.21.15.27:0/898156811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c00aa90 con 0x7fcd4c0cc880 2024-03-06T07:25:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.997+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4c15ecb0 con 0x7fcd4c0cc880 2024-03-06T07:25:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.997+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4c15eeb0 con 0x7fcd4c0cc880 2024-03-06T07:25:30.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.998+0000 7fcd4a7fc640 1 -- 172.21.15.27:0/898156811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd3c00ac30 con 0x7fcd4c0cc880 2024-03-06T07:25:30.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.999+0000 7fcd4a7fc640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcd3806b4d0 0x7fcd3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.999+0000 7fcd4a7fc640 1 -- 172.21.15.27:0/898156811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd3c08a180 con 0x7fcd4c0cc880 2024-03-06T07:25:30.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.999+0000 7fcd51890640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcd3806b4d0 0x7fcd3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:29.999+0000 7fcd51890640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcd3806b4d0 0x7fcd3806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fcd44036000 tx=0x7fcd44008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.003+0000 7fcd4a7fc640 1 -- 172.21.15.27:0/898156811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd4c15eeb0 con 0x7fcd4c0cc880 2024-03-06T07:25:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.224+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcd4c067ff0 con 0x7fcd4c0cc880 2024-03-06T07:25:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.224+0000 7fcd4a7fc640 1 -- 172.21.15.27:0/898156811 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcd3c059010 con 0x7fcd4c0cc880 2024-03-06T07:25:30.227 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcd3806b4d0 msgr2=0x7fcd3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcd3806b4d0 0x7fcd3806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fcd44036000 tx=0x7fcd44008030 comp rx=0 tx=0).stop 2024-03-06T07:25:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd4c0cc880 msgr2=0x7fcd4c0c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd4c0cc880 0x7fcd4c0c9ea0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00c7d0 tx=0x7fcd3c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:25:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 shutdown_connections 2024-03-06T07:25:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcd3806b4d0 0x7fcd3806d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd4c0cc880 0x7fcd4c0c9ea0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4c15df00 0x7fcd4c0cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 --2- 172.21.15.27:0/898156811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcd4c0b95a0 0x7fcd4c160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 >> 172.21.15.27:0/898156811 conn(0x7fcd4c071960 msgr2=0x7fcd4c0b8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 shutdown_connections 2024-03-06T07:25:30.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:30.226+0000 7fcd53af4640 1 -- 172.21.15.27:0/898156811 wait complete. 2024-03-06T07:25:30.239 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:30.239 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:30.240 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:25:30.240 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:30.240 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:30.240 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:30.240 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:30.240 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:30.240 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:30.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.240 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42559sec preferred_lft 42559sec 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:30.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.258 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:30.325 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:25:30.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.326 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:25:30.382 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:25:30.382 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:30.382 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:25:30.382 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:25:30.382 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:30.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.383 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:25:30.383 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:25:30.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:25:30.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:25:30.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:25:30.383 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:25:30.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.487 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:25:30.502 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:30.502 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:30.502 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:30.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:25:30.572 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:30.572 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:30.573 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:30.628 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:30.682 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:25:30.752 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:30.808 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:25:30.808 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:25:30.808 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:25:30.808 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:25:30.808 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:25:30.808 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:25:30.808 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:30.809 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:30.810 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:30.811 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:30.811 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:30.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.812 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:25:30.867 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:30.868 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:30.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:25:30.940 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:25:30.997 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:25:30.998 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.999 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:25:31.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:25:31.001+0000 7f2153c52480 -1 init, newargv = 0x5603c24e05e0 newargc=15 2024-03-06T07:25:31.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[58768]: starting ceph client 2024-03-06T07:25:31.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[58768]: starting fuse 2024-03-06T07:25:31.019 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:25:31.019 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:25:31.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:31.019 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:25:31.020 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:31.021 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:25:31.147 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.58768.asok 2024-03-06T07:25:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:31.157 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58768.asok 2024-03-06T07:25:31.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:31.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58768.asok status 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "58768", 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 6297, 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 6297 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 526122105 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 526122105 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.6297 192.168.0.1:0/526122105", 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/526122105", 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 34, 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 58, 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:25:31.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:31.344 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:25:31.344 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:31.359 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:25:31.359 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:31.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:31.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:31.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:25:31.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.582+0000 7fe94745c640 1 Processor -- start 2024-03-06T07:25:31.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.582+0000 7fe94745c640 1 -- start start 2024-03-06T07:25:31.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.583+0000 7fe94745c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe940072a30 0x7fe9401624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.583+0000 7fe94745c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9400d07c0 0x7fe94015d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.583+0000 7fe94745c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94006d4d0 con 0x7fe9400cf9f0 2024-03-06T07:25:31.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.583+0000 7fe94745c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe94006dba0 con 0x7fe9400d07c0 2024-03-06T07:25:31.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.583+0000 7fe94745c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9400b4950 con 0x7fe940072a30 2024-03-06T07:25:31.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.583+0000 7fe9459f9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9400d07c0 0x7fe94015d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.583+0000 7fe9459f9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9400d07c0 0x7fe94015d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42220/0 (socket says 172.21.15.27:42220) 2024-03-06T07:25:31.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.583+0000 7fe9459f9640 1 -- 172.21.15.27:0/1484995729 learned_addr learned my addr 172.21.15.27:0/1484995729 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:31.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.584+0000 7fe9451f8640 1 --2- 172.21.15.27:0/1484995729 >> v2:172.21.15.69:3300/0 conn(0x7fe940072a30 0x7fe9401624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.584+0000 7fe9459f9640 1 -- 172.21.15.27:0/1484995729 >> v2:172.21.15.69:3300/0 conn(0x7fe940072a30 msgr2=0x7fe9401624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.584+0000 7fe9459f9640 1 --2- 172.21.15.27:0/1484995729 >> v2:172.21.15.69:3300/0 conn(0x7fe940072a30 0x7fe9401624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.584+0000 7fe9459f9640 1 -- 172.21.15.27:0/1484995729 >> v1:172.21.15.69:6789/0 conn(0x7fe9400cf9f0 legacy=0x7fe9400d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.584+0000 7fe9459f9640 1 -- 172.21.15.27:0/1484995729 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94006d7b0 con 0x7fe9400d07c0 2024-03-06T07:25:31.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.584+0000 7fe9459f9640 1 --2- 172.21.15.27:0/1484995729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 0x7fe94015d660 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe940074550 tx=0x7fe93802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d04d576fccea81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.585+0000 7fe937fff640 1 -- 172.21.15.27:0/1484995729 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93802c960 con 0x7fe9400d07c0 2024-03-06T07:25:31.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.585+0000 7fe937fff640 1 -- 172.21.15.27:0/1484995729 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93802cb00 con 0x7fe9400d07c0 2024-03-06T07:25:31.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.585+0000 7fe937fff640 1 -- 172.21.15.27:0/1484995729 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9380343f0 con 0x7fe9400d07c0 2024-03-06T07:25:31.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.585+0000 7fe94745c640 1 -- 172.21.15.27:0/1484995729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 msgr2=0x7fe94015d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.585+0000 7fe94745c640 1 --2- 172.21.15.27:0/1484995729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 0x7fe94015d660 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe940074550 tx=0x7fe93802bdc0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.586+0000 7fe94745c640 1 -- 172.21.15.27:0/1484995729 shutdown_connections 2024-03-06T07:25:31.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.586+0000 7fe94745c640 1 --2- 172.21.15.27:0/1484995729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 0x7fe94015d660 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.586+0000 7fe94745c640 1 --2- 172.21.15.27:0/1484995729 >> v2:172.21.15.69:3300/0 conn(0x7fe940072a30 0x7fe9401624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.586+0000 7fe94745c640 1 -- 172.21.15.27:0/1484995729 >> 172.21.15.27:0/1484995729 conn(0x7fe94006af20 msgr2=0x7fe94006b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.586+0000 7fe94745c640 1 -- 172.21.15.27:0/1484995729 shutdown_connections 2024-03-06T07:25:31.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.586+0000 7fe94745c640 1 -- 172.21.15.27:0/1484995729 wait complete. 2024-03-06T07:25:31.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.588+0000 7fe94745c640 1 Processor -- start 2024-03-06T07:25:31.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.588+0000 7fe94745c640 1 -- start start 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe94745c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe940072a30 0x7fe94015e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe94745c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9400cf9f0 0x7fe94009daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe94745c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 0x7fe9400acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe94745c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940072e10 con 0x7fe9400d07c0 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe94745c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940071130 con 0x7fe940072a30 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe94745c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe940072f90 con 0x7fe9400cf9f0 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9459f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 0x7fe9400acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9459f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 0x7fe9400acfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42232/0 (socket says 172.21.15.27:42232) 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9459f9640 1 -- 172.21.15.27:0/1483521951 learned_addr learned my addr 172.21.15.27:0/1483521951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9451f8640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe940072a30 0x7fe94015e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9449f7640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9400cf9f0 0x7fe94009daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9459f9640 1 -- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9400cf9f0 msgr2=0x7fe94009daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9459f9640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9400cf9f0 0x7fe94009daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9459f9640 1 -- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe940072a30 msgr2=0x7fe94015e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9459f9640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe940072a30 0x7fe94015e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.589+0000 7fe9459f9640 1 -- 172.21.15.27:0/1483521951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94009dfe0 con 0x7fe9400d07c0 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.590+0000 7fe9459f9640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 0x7fe9400acfa0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe938009f40 tx=0x7fe938004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.590+0000 7fe935ffb640 1 -- 172.21.15.27:0/1483521951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe938043020 con 0x7fe9400d07c0 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.590+0000 7fe935ffb640 1 -- 172.21.15.27:0/1483521951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93803e450 con 0x7fe9400d07c0 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.590+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9401f7810 con 0x7fe9400d07c0 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.590+0000 7fe935ffb640 1 -- 172.21.15.27:0/1483521951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe938009510 con 0x7fe9400d07c0 2024-03-06T07:25:31.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.590+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9401f7dd0 con 0x7fe9400d07c0 2024-03-06T07:25:31.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.591+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94009e320 con 0x7fe9400d07c0 2024-03-06T07:25:31.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.592+0000 7fe935ffb640 1 -- 172.21.15.27:0/1483521951 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe938046020 con 0x7fe9400d07c0 2024-03-06T07:25:31.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.592+0000 7fe935ffb640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe92406b500 0x7fe92406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.592+0000 7fe935ffb640 1 -- 172.21.15.27:0/1483521951 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9380ac510 con 0x7fe9400d07c0 2024-03-06T07:25:31.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.593+0000 7fe9451f8640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe92406b500 0x7fe92406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.593+0000 7fe9451f8640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe92406b500 0x7fe92406d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0079a0 tx=0x7fe93c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.597+0000 7fe935ffb640 1 -- 172.21.15.27:0/1483521951 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe93804d070 con 0x7fe9400d07c0 2024-03-06T07:25:31.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.779+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fe94004f410 con 0x7fe9400d07c0 2024-03-06T07:25:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.782+0000 7fe935ffb640 1 -- 172.21.15.27:0/1483521951 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe93807add0 con 0x7fe9400d07c0 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe92406b500 msgr2=0x7fe92406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe92406b500 0x7fe92406d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0079a0 tx=0x7fe93c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 msgr2=0x7fe9400acfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 0x7fe9400acfa0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe938009f40 tx=0x7fe938004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 shutdown_connections 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe92406b500 0x7fe92406d9c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9400d07c0 0x7fe9400acfa0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9400cf9f0 0x7fe94009daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 --2- 172.21.15.27:0/1483521951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe940072a30 0x7fe94015e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 >> 172.21.15.27:0/1483521951 conn(0x7fe94006af20 msgr2=0x7fe94006b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 shutdown_connections 2024-03-06T07:25:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.784+0000 7fe94745c640 1 -- 172.21.15.27:0/1483521951 wait complete. 2024-03-06T07:25:31.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:25:31.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.905+0000 7fe2fa5ac640 1 Processor -- start 2024-03-06T07:25:31.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.905+0000 7fe2fa5ac640 1 -- start start 2024-03-06T07:25:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2fa5ac640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2f40ceea0 0x7fe2f40d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2fa5ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f40b6d10 0x7fe2f40b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2fa5ac640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f40d3740 con 0x7fe2f415dd90 2024-03-06T07:25:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2fa5ac640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f40d3400 con 0x7fe2f40ceea0 2024-03-06T07:25:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2fa5ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40d3120 con 0x7fe2f40b6d10 2024-03-06T07:25:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2f8b49640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f40b6d10 0x7fe2f40b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2f37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2f40ceea0 0x7fe2f40d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2f8b49640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f40b6d10 0x7fe2f40b71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:48912/0 (socket says 172.21.15.27:48912) 2024-03-06T07:25:31.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2f37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2f40ceea0 0x7fe2f40d29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:56674/0 (socket says 172.21.15.27:56674) 2024-03-06T07:25:31.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.906+0000 7fe2f8b49640 1 -- 172.21.15.27:0/3309985880 learned_addr learned my addr 172.21.15.27:0/3309985880 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:31.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.907+0000 7fe2f8b49640 1 -- 172.21.15.27:0/3309985880 >> v2:172.21.15.69:3300/0 conn(0x7fe2f40ceea0 msgr2=0x7fe2f40d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.907+0000 7fe2f8b49640 1 --2- 172.21.15.27:0/3309985880 >> v2:172.21.15.69:3300/0 conn(0x7fe2f40ceea0 0x7fe2f40d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.907+0000 7fe2f8b49640 1 -- 172.21.15.27:0/3309985880 >> v1:172.21.15.69:6789/0 conn(0x7fe2f415dd90 legacy=0x7fe2f40d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.907+0000 7fe2f8b49640 1 -- 172.21.15.27:0/3309985880 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f40d39e0 con 0x7fe2f40b6d10 2024-03-06T07:25:31.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.907+0000 7fe2f37fe640 1 --2- 172.21.15.27:0/3309985880 >> v2:172.21.15.69:3300/0 conn(0x7fe2f40ceea0 0x7fe2f40d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:31.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.907+0000 7fe2f8b49640 1 --2- 172.21.15.27:0/3309985880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f40b6d10 0x7fe2f40b71a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009e50 tx=0x7fe2ec02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88208f07029c4433 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2f2ffd640 1 -- 172.21.15.27:0/3309985880 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec02cb90 con 0x7fe2f40b6d10 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2f2ffd640 1 -- 172.21.15.27:0/3309985880 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec02cd30 con 0x7fe2f40b6d10 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2f2ffd640 1 -- 172.21.15.27:0/3309985880 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec034610 con 0x7fe2f40b6d10 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/3309985880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f40b6d10 msgr2=0x7fe2f40b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2fa5ac640 1 --2- 172.21.15.27:0/3309985880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f40b6d10 0x7fe2f40b71a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009e50 tx=0x7fe2ec02bff0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/3309985880 shutdown_connections 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2fa5ac640 1 --2- 172.21.15.27:0/3309985880 >> v2:172.21.15.69:3300/0 conn(0x7fe2f40ceea0 0x7fe2f40d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2fa5ac640 1 --2- 172.21.15.27:0/3309985880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f40b6d10 0x7fe2f40b71a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/3309985880 >> 172.21.15.27:0/3309985880 conn(0x7fe2f4071960 msgr2=0x7fe2f4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/3309985880 shutdown_connections 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.908+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/3309985880 wait complete. 2024-03-06T07:25:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.909+0000 7fe2fa5ac640 1 Processor -- start 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2fa5ac640 1 -- start start 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2fa5ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f40b6d10 0x7fe2f40cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2fa5ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f40ceea0 0x7fe2f40b9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2fa5ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f415dd90 0x7fe2f41532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2fa5ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40d48f0 con 0x7fe2f40b6d10 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2fa5ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f415e170 con 0x7fe2f40ceea0 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2fa5ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40d4a70 con 0x7fe2f415dd90 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2f3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f40b6d10 0x7fe2f40cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2f3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f40b6d10 0x7fe2f40cc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42234/0 (socket says 172.21.15.27:42234) 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.910+0000 7fe2f3fff640 1 -- 172.21.15.27:0/110798615 learned_addr learned my addr 172.21.15.27:0/110798615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f37fe640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f40ceea0 0x7fe2f40b9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f8b49640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f415dd90 0x7fe2f41532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f3fff640 1 -- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f415dd90 msgr2=0x7fe2f41532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f3fff640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f415dd90 0x7fe2f41532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f3fff640 1 -- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f40ceea0 msgr2=0x7fe2f40b9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f3fff640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f40ceea0 0x7fe2f40b9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f3fff640 1 -- 172.21.15.27:0/110798615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f40c7c40 con 0x7fe2f40b6d10 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f8b49640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f415dd90 0x7fe2f41532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f37fe640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f40ceea0 0x7fe2f40b9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f3fff640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f40b6d10 0x7fe2f40cc840 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c460 tx=0x7fe2e400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f0ff9640 1 -- 172.21.15.27:0/110798615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e40052d0 con 0x7fe2f40b6d10 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f0ff9640 1 -- 172.21.15.27:0/110798615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e4012630 con 0x7fe2f40b6d10 2024-03-06T07:25:31.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2f0ff9640 1 -- 172.21.15.27:0/110798615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e40054b0 con 0x7fe2f40b6d10 2024-03-06T07:25:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f40c7f20 con 0x7fe2f40b6d10 2024-03-06T07:25:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.911+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f40c84e0 con 0x7fe2f40b6d10 2024-03-06T07:25:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.912+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f4072a30 con 0x7fe2f40b6d10 2024-03-06T07:25:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.913+0000 7fe2f0ff9640 1 -- 172.21.15.27:0/110798615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2e4012a20 con 0x7fe2f40b6d10 2024-03-06T07:25:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.913+0000 7fe2f0ff9640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2e006b4d0 0x7fe2e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.913+0000 7fe2f0ff9640 1 -- 172.21.15.27:0/110798615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2e4089d60 con 0x7fe2f40b6d10 2024-03-06T07:25:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.913+0000 7fe2f37fe640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2e006b4d0 0x7fe2e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.914+0000 7fe2f37fe640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2e006b4d0 0x7fe2e006d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40723d0 tx=0x7fe2e8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:31.918+0000 7fe2f0ff9640 1 -- 172.21.15.27:0/110798615 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2e4058c70 con 0x7fe2f40b6d10 2024-03-06T07:25:32.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.101+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7fe2f404f410 con 0x7fe2f40b6d10 2024-03-06T07:25:32.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.104+0000 7fe2f0ff9640 1 -- 172.21.15.27:0/110798615 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7fe2e4058570 con 0x7fe2f40b6d10 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2e006b4d0 msgr2=0x7fe2e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2e006b4d0 0x7fe2e006d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40723d0 tx=0x7fe2e8008040 comp rx=0 tx=0).stop 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f40b6d10 msgr2=0x7fe2f40cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f40b6d10 0x7fe2f40cc840 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c460 tx=0x7fe2e400c920 comp rx=0 tx=0).stop 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 shutdown_connections 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2e006b4d0 0x7fe2e006d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2f415dd90 0x7fe2f41532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f40ceea0 0x7fe2f40b9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 --2- 172.21.15.27:0/110798615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f40b6d10 0x7fe2f40cc840 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 >> 172.21.15.27:0/110798615 conn(0x7fe2f4071960 msgr2=0x7fe2f40b4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 shutdown_connections 2024-03-06T07:25:32.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:32.106+0000 7fe2fa5ac640 1 -- 172.21.15.27:0/110798615 wait complete. 2024-03-06T07:25:42.121 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:42.121 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:42.137 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:25:42.137 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:25:42.137 DEBUG:teuthology.orchestra.run.smithi027:> import uuid 2024-03-06T07:25:42.137 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:42.137 DEBUG:teuthology.orchestra.run.smithi027:> n = 50 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = False 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> if 0 > 0: 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = True 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> dpath = os.path.dirname(path) 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> try: 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:42.138 DEBUG:teuthology.orchestra.run.smithi027:> for i in range(n): 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> f.write(f"{i}") 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> f.flush() 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(f.fileno()) 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> os.unlink(fpath) 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> if create_hard_links: 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> for j in range(0): 2024-03-06T07:25:42.139 DEBUG:teuthology.orchestra.run.smithi027:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:42.140 DEBUG:teuthology.orchestra.run.smithi027:> if True: 2024-03-06T07:25:42.140 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:25:42.140 DEBUG:teuthology.orchestra.run.smithi027:> finally: 2024-03-06T07:25:42.140 DEBUG:teuthology.orchestra.run.smithi027:> os.close(dirfd) 2024-03-06T07:25:42.140 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:25:42.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:42.359 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:42.359 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:42.415 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:42.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.560+0000 7f10fab3e640 1 Processor -- start 2024-03-06T07:25:42.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.560+0000 7f10fab3e640 1 -- start start 2024-03-06T07:25:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10fab3e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f10f415df00 0x7f10f40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10fab3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f40b95a0 0x7f10f40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10fab3e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10f40b9a30 0x7f10f40b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10fab3e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f40cea30 con 0x7f10f415df00 2024-03-06T07:25:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10fab3e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f40d2af0 con 0x7f10f40b9a30 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10fab3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f40cf3c0 con 0x7f10f40b95a0 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f88da640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f10f415df00 0x7f10f40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f88da640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f10f415df00 0x7f10f40d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47392/0 (socket says 172.21.15.27:47392) 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f88da640 1 -- 172.21.15.27:0/2687052069 learned_addr learned my addr 172.21.15.27:0/2687052069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f3fff640 1 --2- 172.21.15.27:0/2687052069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f40b95a0 0x7f10f40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f90db640 1 --2- 172.21.15.27:0/2687052069 >> v2:172.21.15.69:3300/0 conn(0x7f10f40b9a30 0x7f10f40b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f88da640 1 -- 172.21.15.27:0/2687052069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f40b95a0 msgr2=0x7f10f40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f88da640 1 --2- 172.21.15.27:0/2687052069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f40b95a0 0x7f10f40b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f88da640 1 -- 172.21.15.27:0/2687052069 >> v2:172.21.15.69:3300/0 conn(0x7f10f40b9a30 msgr2=0x7f10f40b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f88da640 1 --2- 172.21.15.27:0/2687052069 >> v2:172.21.15.69:3300/0 conn(0x7f10f40b9a30 0x7f10f40b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.561+0000 7f10f88da640 1 -- 172.21.15.27:0/2687052069 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f40cf0e0 con 0x7f10f415df00 2024-03-06T07:25:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.562+0000 7f10f88da640 1 --2- 172.21.15.27:0/2687052069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 0x7f10f40d25b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f10f40d1160 tx=0x7f10e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5cc270c853dfbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.562+0000 7f10f37fe640 1 -- 172.21.15.27:0/2687052069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e8035020 con 0x7f10f415df00 2024-03-06T07:25:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.562+0000 7f10f37fe640 1 -- 172.21.15.27:0/2687052069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e8031870 con 0x7f10f415df00 2024-03-06T07:25:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.562+0000 7f10f37fe640 1 -- 172.21.15.27:0/2687052069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e8031c40 con 0x7f10f415df00 2024-03-06T07:25:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.563+0000 7f10fab3e640 1 -- 172.21.15.27:0/2687052069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 msgr2=0x7f10f40d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.563+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2687052069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 0x7f10f40d25b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f10f40d1160 tx=0x7f10e802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.563+0000 7f10fab3e640 1 -- 172.21.15.27:0/2687052069 shutdown_connections 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.563+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2687052069 >> v2:172.21.15.69:3300/0 conn(0x7f10f40b9a30 0x7f10f40b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.563+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2687052069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f40b95a0 0x7f10f40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.563+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2687052069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 0x7f10f40d25b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.563+0000 7f10fab3e640 1 -- 172.21.15.27:0/2687052069 >> 172.21.15.27:0/2687052069 conn(0x7f10f4071960 msgr2=0x7f10f4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.563+0000 7f10fab3e640 1 -- 172.21.15.27:0/2687052069 shutdown_connections 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.563+0000 7f10fab3e640 1 -- 172.21.15.27:0/2687052069 wait complete. 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.564+0000 7f10fab3e640 1 Processor -- start 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.564+0000 7f10fab3e640 1 -- start start 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10fab3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f40b95a0 0x7f10f40ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10fab3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 0x7f10f40cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10fab3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f415e2e0 0x7f10f41f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10fab3e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f40cec70 con 0x7f10f415df00 2024-03-06T07:25:42.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10fab3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f40ce750 con 0x7f10f40b95a0 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10fab3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f40cedf0 con 0x7f10f415e2e0 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 0x7f10f40cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 0x7f10f40cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47400/0 (socket says 172.21.15.27:47400) 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f3fff640 1 -- 172.21.15.27:0/2556743925 learned_addr learned my addr 172.21.15.27:0/2556743925 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f88da640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f40b95a0 0x7f10f40ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f90db640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f415e2e0 0x7f10f41f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f3fff640 1 -- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f415e2e0 msgr2=0x7f10f41f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f3fff640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f415e2e0 0x7f10f41f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f3fff640 1 -- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f40b95a0 msgr2=0x7f10f40ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f3fff640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f40b95a0 0x7f10f40ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.565+0000 7f10f3fff640 1 -- 172.21.15.27:0/2556743925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f41600f0 con 0x7f10f415df00 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.566+0000 7f10f3fff640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 0x7f10f40cd1b0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f10e000e9b0 tx=0x7f10e000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.566+0000 7f10f17fa640 1 -- 172.21.15.27:0/2556743925 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0017070 con 0x7f10f415df00 2024-03-06T07:25:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.566+0000 7f10f17fa640 1 -- 172.21.15.27:0/2556743925 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e000c9a0 con 0x7f10f415df00 2024-03-06T07:25:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.566+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f415e7a0 con 0x7f10f415df00 2024-03-06T07:25:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.566+0000 7f10f17fa640 1 -- 172.21.15.27:0/2556743925 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e000cd90 con 0x7f10f415df00 2024-03-06T07:25:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.566+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f415ed60 con 0x7f10f415df00 2024-03-06T07:25:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.567+0000 7f10f17fa640 1 -- 172.21.15.27:0/2556743925 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10e001a070 con 0x7f10f415df00 2024-03-06T07:25:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.567+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f4067ff0 con 0x7f10f415df00 2024-03-06T07:25:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.568+0000 7f10f17fa640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 0x7f10d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.568+0000 7f10f17fa640 1 -- 172.21.15.27:0/2556743925 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10e008a150 con 0x7f10f415df00 2024-03-06T07:25:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.568+0000 7f10f88da640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 0x7f10d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.569+0000 7f10f88da640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 0x7f10d806d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f10e8036000 tx=0x7f10e803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.572+0000 7f10f17fa640 1 -- 172.21.15.27:0/2556743925 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10f41600f0 con 0x7f10f415df00 2024-03-06T07:25:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.754+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10f4160530 con 0x7f10f415df00 2024-03-06T07:25:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.755+0000 7f10f17fa640 1 -- 172.21.15.27:0/2556743925 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f10e0058980 con 0x7f10f415df00 2024-03-06T07:25:42.756 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:42.756 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:25.095713+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:21.134461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:22.434887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.756+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 msgr2=0x7f10d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.756+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 0x7f10d806d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f10e8036000 tx=0x7f10e803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.756+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 msgr2=0x7f10f40cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.756+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 0x7f10f40cd1b0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f10e000e9b0 tx=0x7f10e000ee70 comp rx=0 tx=0).stop 2024-03-06T07:25:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.757+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 shutdown_connections 2024-03-06T07:25:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.757+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 0x7f10d806d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.757+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f415e2e0 0x7f10f41f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.757+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f415df00 0x7f10f40cd1b0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.757+0000 7f10fab3e640 1 --2- 172.21.15.27:0/2556743925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f40b95a0 0x7f10f40ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.757+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 >> 172.21.15.27:0/2556743925 conn(0x7f10f4071960 msgr2=0x7f10f40cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.757+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 shutdown_connections 2024-03-06T07:25:42.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.757+0000 7f10fab3e640 1 -- 172.21.15.27:0/2556743925 wait complete. 2024-03-06T07:25:42.769 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:21.134461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:22.434887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:42.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:42.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.877+0000 7f717d084640 1 Processor -- start 2024-03-06T07:25:42.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.877+0000 7f717d084640 1 -- start start 2024-03-06T07:25:42.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717d084640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71780ceea0 0x7f71780d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717d084640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71780d3740 con 0x7f717815dd90 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717d084640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71780d3120 con 0x7f71780b6d10 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717d084640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71780d3400 con 0x7f71780ceea0 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71780ceea0 0x7f71780d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71780ceea0 0x7f71780d29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47402/0 (socket says 172.21.15.27:47402) 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f7176d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f717815dd90 0x7f71780d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48426/0 (socket says 172.21.15.27:48426) 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717659c640 1 -- 172.21.15.27:0/1323234649 learned_addr learned my addr 172.21.15.27:0/1323234649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717659c640 1 -- 172.21.15.27:0/1323234649 >> v1:172.21.15.69:6789/0 conn(0x7f71780b6d10 legacy=0x7f71780d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717659c640 1 -- 172.21.15.27:0/1323234649 >> v1:172.21.15.27:6789/0 conn(0x7f717815dd90 legacy=0x7f71780d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.879+0000 7f717659c640 1 -- 172.21.15.27:0/1323234649 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71780d39e0 con 0x7f71780ceea0 2024-03-06T07:25:42.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.880+0000 7f717659c640 1 --2- 172.21.15.27:0/1323234649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 0x7f71780d29d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7168009870 tx=0x7f716802bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a9c30ed9e417a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.880+0000 7f7175d9b640 1 -- 172.21.15.27:0/1323234649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716802d870 con 0x7f71780ceea0 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.880+0000 7f7175d9b640 1 -- 172.21.15.27:0/1323234649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716802da10 con 0x7f71780ceea0 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.880+0000 7f7175d9b640 1 -- 172.21.15.27:0/1323234649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716802dde0 con 0x7f71780ceea0 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.880+0000 7f717d084640 1 -- 172.21.15.27:0/1323234649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 msgr2=0x7f71780d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.880+0000 7f717d084640 1 --2- 172.21.15.27:0/1323234649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 0x7f71780d29d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7168009870 tx=0x7f716802bab0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.881+0000 7f717d084640 1 -- 172.21.15.27:0/1323234649 shutdown_connections 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.881+0000 7f717d084640 1 --2- 172.21.15.27:0/1323234649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 0x7f71780d29d0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.881+0000 7f717d084640 1 -- 172.21.15.27:0/1323234649 >> 172.21.15.27:0/1323234649 conn(0x7f7178071960 msgr2=0x7f7178071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.881+0000 7f717d084640 1 -- 172.21.15.27:0/1323234649 shutdown_connections 2024-03-06T07:25:42.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.881+0000 7f717d084640 1 -- 172.21.15.27:0/1323234649 wait complete. 2024-03-06T07:25:42.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.882+0000 7f717d084640 1 Processor -- start 2024-03-06T07:25:42.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.882+0000 7f717d084640 1 -- start start 2024-03-06T07:25:42.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717d084640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71780b6d10 0x7f71781f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717d084640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 0x7f71781e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717d084640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f717815dd90 0x7f71781e7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717d084640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780d48f0 con 0x7f71780ceea0 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717d084640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717815e170 con 0x7f717815dd90 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717d084640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71780d4a70 con 0x7f71780b6d10 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 0x7f71781e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 0x7f71781e7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47404/0 (socket says 172.21.15.27:47404) 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717659c640 1 -- 172.21.15.27:0/1066931738 learned_addr learned my addr 172.21.15.27:0/1066931738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717659c640 1 -- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71780b6d10 msgr2=0x7f71781f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f7176d9d640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71780b6d10 0x7f71781f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717759e640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f717815dd90 0x7f71781e7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717659c640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71780b6d10 0x7f71781f5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717659c640 1 -- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f717815dd90 msgr2=0x7f71781e7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717659c640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f717815dd90 0x7f71781e7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717659c640 1 -- 172.21.15.27:0/1066931738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71781f8350 con 0x7f71780ceea0 2024-03-06T07:25:42.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f717659c640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 0x7f71781e7760 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f716802c0c0 tx=0x7f7168004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.883+0000 7f715f7fe640 1 -- 172.21.15.27:0/1066931738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168046070 con 0x7f71780ceea0 2024-03-06T07:25:42.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.884+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71780c89a0 con 0x7f71780ceea0 2024-03-06T07:25:42.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.884+0000 7f715f7fe640 1 -- 172.21.15.27:0/1066931738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7168037a40 con 0x7f71780ceea0 2024-03-06T07:25:42.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.884+0000 7f715f7fe640 1 -- 172.21.15.27:0/1066931738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168037e10 con 0x7f71780ceea0 2024-03-06T07:25:42.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.884+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71781ea380 con 0x7f71780ceea0 2024-03-06T07:25:42.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.885+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71780d2e90 con 0x7f71780ceea0 2024-03-06T07:25:42.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.885+0000 7f715f7fe640 1 -- 172.21.15.27:0/1066931738 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71680403e0 con 0x7f71780ceea0 2024-03-06T07:25:42.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.885+0000 7f715f7fe640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716406b500 0x7f716406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.886+0000 7f715f7fe640 1 -- 172.21.15.27:0/1066931738 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71680abd90 con 0x7f71780ceea0 2024-03-06T07:25:42.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.886+0000 7f7176d9d640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716406b500 0x7f716406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.887+0000 7f7176d9d640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716406b500 0x7f716406d9c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f716c007fd0 tx=0x7f716c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:42.890+0000 7f715f7fe640 1 -- 172.21.15.27:0/1066931738 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f716807ab90 con 0x7f71780ceea0 2024-03-06T07:25:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.137+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7178072a30 con 0x7f71780ceea0 2024-03-06T07:25:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.138+0000 7f715f7fe640 1 -- 172.21.15.27:0/1066931738 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f716807a490 con 0x7f71780ceea0 2024-03-06T07:25:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:25:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716406b500 msgr2=0x7f716406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716406b500 0x7f716406d9c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f716c007fd0 tx=0x7f716c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 msgr2=0x7f71781e7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 0x7f71781e7760 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f716802c0c0 tx=0x7f7168004730 comp rx=0 tx=0).stop 2024-03-06T07:25:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 shutdown_connections 2024-03-06T07:25:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716406b500 0x7f716406d9c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f717815dd90 0x7f71781e7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71780ceea0 0x7f71781e7760 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 --2- 172.21.15.27:0/1066931738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71780b6d10 0x7f71781f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 >> 172.21.15.27:0/1066931738 conn(0x7f7178071960 msgr2=0x7f71780b71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 shutdown_connections 2024-03-06T07:25:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.140+0000 7f717d084640 1 -- 172.21.15.27:0/1066931738 wait complete. 2024-03-06T07:25:43.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:43.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.261+0000 7f6729bdd640 1 Processor -- start 2024-03-06T07:25:43.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.261+0000 7f6729bdd640 1 -- start start 2024-03-06T07:25:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f6729bdd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f672415dd90 0x7f67240ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f6729bdd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 0x7f67240b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f6729bdd640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67240d3740 con 0x7f67240d29d0 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f6729bdd640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67240d3400 con 0x7f672415dd90 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f6729bdd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67240d3120 con 0x7f67240b6d10 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f67237fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f672415dd90 0x7f67240ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f6722ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f67240d29d0 0x7f67240d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48436/0 (socket says 172.21.15.27:48436) 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f6722ffd640 1 -- 172.21.15.27:0/613915051 learned_addr learned my addr 172.21.15.27:0/613915051 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f6723fff640 1 --2- 172.21.15.27:0/613915051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 0x7f67240b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f67237fe640 1 -- 172.21.15.27:0/613915051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 msgr2=0x7f67240b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f67237fe640 1 --2- 172.21.15.27:0/613915051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 0x7f67240b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f67237fe640 1 -- 172.21.15.27:0/613915051 >> v1:172.21.15.27:6789/0 conn(0x7f67240d29d0 legacy=0x7f67240d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f67237fe640 1 -- 172.21.15.27:0/613915051 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67240d39e0 con 0x7f672415dd90 2024-03-06T07:25:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f67227fc640 1 -- 172.21.15.27:0/613915051 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943808789 0 0) 0x7f67240d3740 con 0x7f67240d29d0 2024-03-06T07:25:43.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.262+0000 7f6723fff640 1 --2- 172.21.15.27:0/613915051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 0x7f67240b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:43.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.263+0000 7f67237fe640 1 --2- 172.21.15.27:0/613915051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672415dd90 0x7f67240ceea0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f67140097a0 tx=0x7f671402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3906e485031bfd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.263+0000 7f67227fc640 1 -- 172.21.15.27:0/613915051 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714034070 con 0x7f672415dd90 2024-03-06T07:25:43.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.263+0000 7f67227fc640 1 -- 172.21.15.27:0/613915051 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6714030610 con 0x7f672415dd90 2024-03-06T07:25:43.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.263+0000 7f67227fc640 1 -- 172.21.15.27:0/613915051 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67140309e0 con 0x7f672415dd90 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.264+0000 7f6729bdd640 1 -- 172.21.15.27:0/613915051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672415dd90 msgr2=0x7f67240ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.264+0000 7f6729bdd640 1 --2- 172.21.15.27:0/613915051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672415dd90 0x7f67240ceea0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f67140097a0 tx=0x7f671402b960 comp rx=0 tx=0).stop 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.264+0000 7f6729bdd640 1 -- 172.21.15.27:0/613915051 shutdown_connections 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.264+0000 7f6729bdd640 1 --2- 172.21.15.27:0/613915051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 0x7f67240b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.264+0000 7f6729bdd640 1 --2- 172.21.15.27:0/613915051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672415dd90 0x7f67240ceea0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.264+0000 7f6729bdd640 1 -- 172.21.15.27:0/613915051 >> 172.21.15.27:0/613915051 conn(0x7f6724071960 msgr2=0x7f6724071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.264+0000 7f6729bdd640 1 -- 172.21.15.27:0/613915051 shutdown_connections 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.264+0000 7f6729bdd640 1 -- 172.21.15.27:0/613915051 wait complete. 2024-03-06T07:25:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.265+0000 7f6729bdd640 1 Processor -- start 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.265+0000 7f6729bdd640 1 -- start start 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6729bdd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 0x7f672415c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6729bdd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240d29d0 0x7f67240c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6729bdd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f672415dd90 0x7f67240ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6729bdd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67240d2df0 con 0x7f67240d29d0 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6729bdd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672415e170 con 0x7f672415dd90 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6729bdd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67240d48f0 con 0x7f67240b6d10 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6722ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240d29d0 0x7f67240c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6722ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240d29d0 0x7f67240c9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47434/0 (socket says 172.21.15.27:47434) 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6722ffd640 1 -- 172.21.15.27:0/1562012179 learned_addr learned my addr 172.21.15.27:0/1562012179 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f67237fe640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 0x7f672415c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6723fff640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f672415dd90 0x7f67240ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6722ffd640 1 -- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 msgr2=0x7f672415c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6722ffd640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 0x7f672415c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6722ffd640 1 -- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f672415dd90 msgr2=0x7f67240ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6722ffd640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f672415dd90 0x7f67240ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.266+0000 7f6722ffd640 1 -- 172.21.15.27:0/1562012179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67240c8950 con 0x7f67240d29d0 2024-03-06T07:25:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.267+0000 7f6722ffd640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240d29d0 0x7f67240c9f20 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f670c00c4a0 tx=0x7f670c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.267+0000 7f671bfff640 1 -- 172.21.15.27:0/1562012179 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c017070 con 0x7f67240d29d0 2024-03-06T07:25:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.267+0000 7f671bfff640 1 -- 172.21.15.27:0/1562012179 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670c005ce0 con 0x7f67240d29d0 2024-03-06T07:25:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.267+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67240c8c30 con 0x7f67240d29d0 2024-03-06T07:25:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.267+0000 7f671bfff640 1 -- 172.21.15.27:0/1562012179 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c0072b0 con 0x7f67240d29d0 2024-03-06T07:25:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.267+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67240c91f0 con 0x7f67240d29d0 2024-03-06T07:25:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.268+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6724072a30 con 0x7f67240d29d0 2024-03-06T07:25:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.268+0000 7f671bfff640 1 -- 172.21.15.27:0/1562012179 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f670c01a070 con 0x7f67240d29d0 2024-03-06T07:25:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.269+0000 7f671bfff640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670406b4d0 0x7f670406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.269+0000 7f671bfff640 1 -- 172.21.15.27:0/1562012179 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f670c089440 con 0x7f67240d29d0 2024-03-06T07:25:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.269+0000 7f67237fe640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670406b4d0 0x7f670406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.270+0000 7f67237fe640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670406b4d0 0x7f670406d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6714036000 tx=0x7f671403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.273+0000 7f671bfff640 1 -- 172.21.15.27:0/1562012179 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f670c057d00 con 0x7f67240d29d0 2024-03-06T07:25:43.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.454+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f67240c7220 con 0x7f67240d29d0 2024-03-06T07:25:43.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.454+0000 7f671bfff640 1 -- 172.21.15.27:0/1562012179 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f670c05bda0 con 0x7f67240d29d0 2024-03-06T07:25:43.455 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:25:43.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670406b4d0 msgr2=0x7f670406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670406b4d0 0x7f670406d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6714036000 tx=0x7f671403b040 comp rx=0 tx=0).stop 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240d29d0 msgr2=0x7f67240c9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240d29d0 0x7f67240c9f20 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f670c00c4a0 tx=0x7f670c00c960 comp rx=0 tx=0).stop 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 shutdown_connections 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670406b4d0 0x7f670406d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f672415dd90 0x7f67240ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240d29d0 0x7f67240c9f20 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 --2- 172.21.15.27:0/1562012179 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240b6d10 0x7f672415c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 >> 172.21.15.27:0/1562012179 conn(0x7f6724071960 msgr2=0x7f67240b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 shutdown_connections 2024-03-06T07:25:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.456+0000 7f6729bdd640 1 -- 172.21.15.27:0/1562012179 wait complete. 2024-03-06T07:25:43.468 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:43.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.576+0000 7fdc05ceb640 1 Processor -- start 2024-03-06T07:25:43.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.576+0000 7fdc05ceb640 1 -- start start 2024-03-06T07:25:43.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.577+0000 7fdc05ceb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc000b2d30 0x7fdc000b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.577+0000 7fdc05ceb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdc000b7820 0x7fdc000b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.577+0000 7fdc05ceb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc000d4960 con 0x7fdc00071540 2024-03-06T07:25:43.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.577+0000 7fdc05ceb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc000d3230 con 0x7fdc000b2d30 2024-03-06T07:25:43.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.577+0000 7fdc05ceb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc000d36c0 con 0x7fdc000b7820 2024-03-06T07:25:43.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.577+0000 7fdbfeffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc000b2d30 0x7fdc000b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.577+0000 7fdbff7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdc00071540 0x7fdc00071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38258/0 (socket says 172.21.15.27:38258) 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.577+0000 7fdbff7fe640 1 -- 172.21.15.27:0/3144665043 learned_addr learned my addr 172.21.15.27:0/3144665043 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.578+0000 7fdbfffff640 1 --2- 172.21.15.27:0/3144665043 >> v2:172.21.15.69:3300/0 conn(0x7fdc000b7820 0x7fdc000b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.578+0000 7fdbfeffd640 1 -- 172.21.15.27:0/3144665043 >> v2:172.21.15.69:3300/0 conn(0x7fdc000b7820 msgr2=0x7fdc000b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.578+0000 7fdbfeffd640 1 --2- 172.21.15.27:0/3144665043 >> v2:172.21.15.69:3300/0 conn(0x7fdc000b7820 0x7fdc000b7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.578+0000 7fdbfeffd640 1 -- 172.21.15.27:0/3144665043 >> v1:172.21.15.27:6789/0 conn(0x7fdc00071540 legacy=0x7fdc00071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.578+0000 7fdbfe7fc640 1 -- 172.21.15.27:0/3144665043 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031691414 0 0) 0x7fdc000d4960 con 0x7fdc00071540 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.578+0000 7fdbfeffd640 1 -- 172.21.15.27:0/3144665043 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc000d2620 con 0x7fdc000b2d30 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdbfeffd640 1 --2- 172.21.15.27:0/3144665043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b2d30 0x7fdc000b72e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdc000ce340 tx=0x7fdbf002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57afc244493895d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdbfe7fc640 1 -- 172.21.15.27:0/3144665043 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0035020 con 0x7fdc000b2d30 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdbfe7fc640 1 -- 172.21.15.27:0/3144665043 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf0031830 con 0x7fdc000b2d30 2024-03-06T07:25:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdbfe7fc640 1 -- 172.21.15.27:0/3144665043 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0031c00 con 0x7fdc000b2d30 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3144665043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b2d30 msgr2=0x7fdc000b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdc05ceb640 1 --2- 172.21.15.27:0/3144665043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b2d30 0x7fdc000b72e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdc000ce340 tx=0x7fdbf002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3144665043 shutdown_connections 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdc05ceb640 1 --2- 172.21.15.27:0/3144665043 >> v2:172.21.15.69:3300/0 conn(0x7fdc000b7820 0x7fdc000b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdc05ceb640 1 --2- 172.21.15.27:0/3144665043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b2d30 0x7fdc000b72e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3144665043 >> 172.21.15.27:0/3144665043 conn(0x7fdc000d3d60 msgr2=0x7fdc000d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3144665043 shutdown_connections 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.579+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3144665043 wait complete. 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.580+0000 7fdc05ceb640 1 Processor -- start 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.581+0000 7fdc05ceb640 1 -- start start 2024-03-06T07:25:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.581+0000 7fdc05ceb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc00071540 0x7fdc0014d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.581+0000 7fdc05ceb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc000b2d30 0x7fdc000ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.581+0000 7fdc05ceb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b7820 0x7fdc000ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.581+0000 7fdc05ceb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc000d2ec0 con 0x7fdc000b7820 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.581+0000 7fdc05ceb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc000d2be0 con 0x7fdc000b2d30 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.581+0000 7fdc05ceb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc000d22f0 con 0x7fdc00071540 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b7820 0x7fdc000ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b7820 0x7fdc000ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59506/0 (socket says 172.21.15.27:59506) 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfffff640 1 -- 172.21.15.27:0/3650323792 learned_addr learned my addr 172.21.15.27:0/3650323792 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbff7fe640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc00071540 0x7fdc0014d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfeffd640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc000b2d30 0x7fdc000ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfffff640 1 -- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc00071540 msgr2=0x7fdc0014d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfffff640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc00071540 0x7fdc0014d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfffff640 1 -- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc000b2d30 msgr2=0x7fdc000ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfffff640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc000b2d30 0x7fdc000ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfffff640 1 -- 172.21.15.27:0/3650323792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc001e4230 con 0x7fdc000b7820 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfeffd640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc000b2d30 0x7fdc000ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbfffff640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b7820 0x7fdc000ca580 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800efc0 tx=0x7fdbf800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbe7fff640 1 -- 172.21.15.27:0/3650323792 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8018020 con 0x7fdc000b7820 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbe7fff640 1 -- 172.21.15.27:0/3650323792 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf8014480 con 0x7fdc000b7820 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc001e4510 con 0x7fdc000b7820 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdbe7fff640 1 -- 172.21.15.27:0/3650323792 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8014870 con 0x7fdc000b7820 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.582+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc001e4ad0 con 0x7fdc000b7820 2024-03-06T07:25:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.583+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc00072070 con 0x7fdc000b7820 2024-03-06T07:25:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.584+0000 7fdbe7fff640 1 -- 172.21.15.27:0/3650323792 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbf8005d10 con 0x7fdc000b7820 2024-03-06T07:25:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.584+0000 7fdbe7fff640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbec06b500 0x7fdbec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.584+0000 7fdbe7fff640 1 -- 172.21.15.27:0/3650323792 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbf808ac50 con 0x7fdc000b7820 2024-03-06T07:25:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.585+0000 7fdbff7fe640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbec06b500 0x7fdbec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.585+0000 7fdbff7fe640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbec06b500 0x7fdbec06d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4007fd0 tx=0x7fdbf4007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.589+0000 7fdbe7fff640 1 -- 172.21.15.27:0/3650323792 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbf8090020 con 0x7fdc000b7820 2024-03-06T07:25:43.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.771+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdc0004f410 con 0x7fdc000b7820 2024-03-06T07:25:43.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.771+0000 7fdbe7fff640 1 -- 172.21.15.27:0/3650323792 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdbf8022070 con 0x7fdc000b7820 2024-03-06T07:25:43.773 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:25:43.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbec06b500 msgr2=0x7fdbec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbec06b500 0x7fdbec06d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4007fd0 tx=0x7fdbf4007de0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b7820 msgr2=0x7fdc000ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b7820 0x7fdc000ca580 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800efc0 tx=0x7fdbf800c490 comp rx=0 tx=0).stop 2024-03-06T07:25:43.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 shutdown_connections 2024-03-06T07:25:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdbec06b500 0x7fdbec06d9c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc000b7820 0x7fdc000ca580 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc000b2d30 0x7fdc000ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 --2- 172.21.15.27:0/3650323792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc00071540 0x7fdc0014d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.773+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 >> 172.21.15.27:0/3650323792 conn(0x7fdc000d3d60 msgr2=0x7fdc0015d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.774+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 shutdown_connections 2024-03-06T07:25:43.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.774+0000 7fdc05ceb640 1 -- 172.21.15.27:0/3650323792 wait complete. 2024-03-06T07:25:43.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:43.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.892+0000 7f32d85eb640 1 Processor -- start 2024-03-06T07:25:43.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.892+0000 7f32d85eb640 1 -- start start 2024-03-06T07:25:43.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d85eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32d00d3400 0x7f32d00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d85eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d00b2b50 0x7f32d00b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d85eb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d00ce610 con 0x7f32d0071540 2024-03-06T07:25:43.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d85eb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d00d2f70 con 0x7f32d00d3400 2024-03-06T07:25:43.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d85eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00d4be0 con 0x7f32d00b2b50 2024-03-06T07:25:43.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d5b86640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32d00d3400 0x7f32d00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d6387640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f32d0071540 0x7f32d0071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38260/0 (socket says 172.21.15.27:38260) 2024-03-06T07:25:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d6387640 1 -- 172.21.15.27:0/2885973829 learned_addr learned my addr 172.21.15.27:0/2885973829 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d6b88640 1 --2- 172.21.15.27:0/2885973829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d00b2b50 0x7f32d00b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d5b86640 1 -- 172.21.15.27:0/2885973829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d00b2b50 msgr2=0x7f32d00b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d5b86640 1 --2- 172.21.15.27:0/2885973829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d00b2b50 0x7f32d00b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d5b86640 1 -- 172.21.15.27:0/2885973829 >> v1:172.21.15.27:6789/0 conn(0x7f32d0071540 legacy=0x7f32d0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d5b86640 1 -- 172.21.15.27:0/2885973829 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d00d2920 con 0x7f32d00d3400 2024-03-06T07:25:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.893+0000 7f32d5385640 1 -- 172.21.15.27:0/2885973829 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147434836 0 0) 0x7f32d00ce610 con 0x7f32d0071540 2024-03-06T07:25:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.894+0000 7f32d5b86640 1 --2- 172.21.15.27:0/2885973829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 0x7f32d00b30a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f32c40097a0 tx=0x7f32c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4924f3b90da26b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.894+0000 7f32d5385640 1 -- 172.21.15.27:0/2885973829 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4035020 con 0x7f32d00d3400 2024-03-06T07:25:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.894+0000 7f32d5385640 1 -- 172.21.15.27:0/2885973829 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c4031830 con 0x7f32d00d3400 2024-03-06T07:25:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.894+0000 7f32d5385640 1 -- 172.21.15.27:0/2885973829 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4031c00 con 0x7f32d00d3400 2024-03-06T07:25:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.895+0000 7f32d85eb640 1 -- 172.21.15.27:0/2885973829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 msgr2=0x7f32d00b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.895+0000 7f32d85eb640 1 --2- 172.21.15.27:0/2885973829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 0x7f32d00b30a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f32c40097a0 tx=0x7f32c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.895+0000 7f32d85eb640 1 -- 172.21.15.27:0/2885973829 shutdown_connections 2024-03-06T07:25:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.895+0000 7f32d85eb640 1 --2- 172.21.15.27:0/2885973829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d00b2b50 0x7f32d00b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.895+0000 7f32d85eb640 1 --2- 172.21.15.27:0/2885973829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 0x7f32d00b30a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.895+0000 7f32d85eb640 1 -- 172.21.15.27:0/2885973829 >> 172.21.15.27:0/2885973829 conn(0x7f32d00d3ca0 msgr2=0x7f32d00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.895+0000 7f32d85eb640 1 -- 172.21.15.27:0/2885973829 shutdown_connections 2024-03-06T07:25:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.895+0000 7f32d85eb640 1 -- 172.21.15.27:0/2885973829 wait complete. 2024-03-06T07:25:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.896+0000 7f32d85eb640 1 Processor -- start 2024-03-06T07:25:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.896+0000 7f32d85eb640 1 -- start start 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d85eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0071540 0x7f32d01f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d85eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d00b2b50 0x7f32d00b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d85eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 0x7f32d00b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d85eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d015e370 con 0x7f32d00d3400 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d85eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00ce8f0 con 0x7f32d00b2b50 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d85eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00cbe70 con 0x7f32d0071540 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d6b88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 0x7f32d00b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d6b88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 0x7f32d00b4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59516/0 (socket says 172.21.15.27:59516) 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d6b88640 1 -- 172.21.15.27:0/3996560917 learned_addr learned my addr 172.21.15.27:0/3996560917 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d5b86640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d00b2b50 0x7f32d00b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d6387640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0071540 0x7f32d01f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d6b88640 1 -- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0071540 msgr2=0x7f32d01f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d6b88640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0071540 0x7f32d01f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d6b88640 1 -- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d00b2b50 msgr2=0x7f32d00b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d6b88640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d00b2b50 0x7f32d00b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d6b88640 1 -- 172.21.15.27:0/3996560917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d01f3320 con 0x7f32d00d3400 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.897+0000 7f32d5b86640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d00b2b50 0x7f32d00b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.898+0000 7f32d6b88640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 0x7f32d00b4760 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f32c800d830 tx=0x7f32c800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.898+0000 7f32ceffd640 1 -- 172.21.15.27:0/3996560917 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c800b6c0 con 0x7f32d00d3400 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.898+0000 7f32ceffd640 1 -- 172.21.15.27:0/3996560917 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c8019070 con 0x7f32d00d3400 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.898+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d00b8420 con 0x7f32d00d3400 2024-03-06T07:25:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.898+0000 7f32ceffd640 1 -- 172.21.15.27:0/3996560917 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c800bac0 con 0x7f32d00d3400 2024-03-06T07:25:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.898+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d00cb820 con 0x7f32d00d3400 2024-03-06T07:25:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.899+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d0068700 con 0x7f32d00d3400 2024-03-06T07:25:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.900+0000 7f32ceffd640 1 -- 172.21.15.27:0/3996560917 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32c800bc60 con 0x7f32d00d3400 2024-03-06T07:25:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.900+0000 7f32ceffd640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32bc06b4d0 0x7f32bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.901+0000 7f32ceffd640 1 -- 172.21.15.27:0/3996560917 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32c808a5d0 con 0x7f32d00d3400 2024-03-06T07:25:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.901+0000 7f32d6387640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32bc06b4d0 0x7f32bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.901+0000 7f32d6387640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32bc06b4d0 0x7f32bc06d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f32c00012e0 tx=0x7f32c0007ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:43.905+0000 7f32ceffd640 1 -- 172.21.15.27:0/3996560917 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32c80594e0 con 0x7f32d00d3400 2024-03-06T07:25:44.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.100+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f32d006b8e0 con 0x7f32d00d3400 2024-03-06T07:25:44.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.100+0000 7f32ceffd640 1 -- 172.21.15.27:0/3996560917 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7f32c8058de0 con 0x7f32d00d3400 2024-03-06T07:25:44.102 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:25:44.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32bc06b4d0 msgr2=0x7f32bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32bc06b4d0 0x7f32bc06d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f32c00012e0 tx=0x7f32c0007ed0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 msgr2=0x7f32d00b4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 0x7f32d00b4760 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f32c800d830 tx=0x7f32c800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 shutdown_connections 2024-03-06T07:25:44.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32bc06b4d0 0x7f32bc06d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d00d3400 0x7f32d00b4760 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32d00b2b50 0x7f32d00b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 --2- 172.21.15.27:0/3996560917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32d0071540 0x7f32d01f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 >> 172.21.15.27:0/3996560917 conn(0x7f32d00d3ca0 msgr2=0x7f32d00c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 shutdown_connections 2024-03-06T07:25:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.102+0000 7f32d85eb640 1 -- 172.21.15.27:0/3996560917 wait complete. 2024-03-06T07:25:44.114 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:44.114 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:44.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.275+0000 7f0f9f53e640 1 Processor -- start 2024-03-06T07:25:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.275+0000 7f0f9f53e640 1 -- start start 2024-03-06T07:25:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9f53e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0f98071540 0x7f0f980b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9f53e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980d3400 0x7f0f98071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9f53e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0f980b2b50 0x7f0f980d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9f53e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980d2f70 con 0x7f0f980b2b50 2024-03-06T07:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9f53e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980d4be0 con 0x7f0f98071540 2024-03-06T07:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9f53e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98071e60 con 0x7f0f980d3400 2024-03-06T07:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9dadb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0f980b2b50 0x7f0f980d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9dadb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0f980b2b50 0x7f0f980d1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59522/0 (socket says 172.21.15.27:59522) 2024-03-06T07:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9dadb640 1 -- 172.21.15.27:0/897460762 learned_addr learned my addr 172.21.15.27:0/897460762 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9cad9640 1 --2- 172.21.15.27:0/897460762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980d3400 0x7f0f98071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9d2da640 1 --2- 172.21.15.27:0/897460762 >> v2:172.21.15.69:3300/0 conn(0x7f0f98071540 0x7f0f980b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9dadb640 1 -- 172.21.15.27:0/897460762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980d3400 msgr2=0x7f0f98071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9dadb640 1 --2- 172.21.15.27:0/897460762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980d3400 0x7f0f98071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9dadb640 1 -- 172.21.15.27:0/897460762 >> v2:172.21.15.69:3300/0 conn(0x7f0f98071540 msgr2=0x7f0f980b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9dadb640 1 --2- 172.21.15.27:0/897460762 >> v2:172.21.15.69:3300/0 conn(0x7f0f98071540 0x7f0f980b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.276+0000 7f0f9dadb640 1 -- 172.21.15.27:0/897460762 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f980ce610 con 0x7f0f980b2b50 2024-03-06T07:25:44.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.277+0000 7f0f9dadb640 1 --2- 172.21.15.27:0/897460762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980b2b50 0x7f0f980d1870 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0f980d4da0 tx=0x7f0f9002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98d3bae8860eb2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.277+0000 7f0f8ffff640 1 -- 172.21.15.27:0/897460762 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9002ca70 con 0x7f0f980b2b50 2024-03-06T07:25:44.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.277+0000 7f0f8ffff640 1 -- 172.21.15.27:0/897460762 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9002cc10 con 0x7f0f980b2b50 2024-03-06T07:25:44.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.277+0000 7f0f8ffff640 1 -- 172.21.15.27:0/897460762 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f900343f0 con 0x7f0f980b2b50 2024-03-06T07:25:44.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.278+0000 7f0f9f53e640 1 -- 172.21.15.27:0/897460762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980b2b50 msgr2=0x7f0f980d1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.278+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/897460762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980b2b50 0x7f0f980d1870 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0f980d4da0 tx=0x7f0f9002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.278+0000 7f0f9f53e640 1 -- 172.21.15.27:0/897460762 shutdown_connections 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.278+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/897460762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980b2b50 0x7f0f980d1870 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.278+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/897460762 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980d3400 0x7f0f98071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.278+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/897460762 >> v2:172.21.15.69:3300/0 conn(0x7f0f98071540 0x7f0f980b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.278+0000 7f0f9f53e640 1 -- 172.21.15.27:0/897460762 >> 172.21.15.27:0/897460762 conn(0x7f0f980d3ca0 msgr2=0x7f0f980d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.278+0000 7f0f9f53e640 1 -- 172.21.15.27:0/897460762 shutdown_connections 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.278+0000 7f0f9f53e640 1 -- 172.21.15.27:0/897460762 wait complete. 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.279+0000 7f0f9f53e640 1 Processor -- start 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.279+0000 7f0f9f53e640 1 -- start start 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9f53e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f98071540 0x7f0f980b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9f53e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980b2b50 0x7f0f9815fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9f53e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980d3400 0x7f0f9814ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9f53e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980ce8f0 con 0x7f0f980d3400 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9f53e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980d2c00 con 0x7f0f98071540 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9f53e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9815e370 con 0x7f0f980b2b50 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9dadb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980d3400 0x7f0f9814ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9dadb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980d3400 0x7f0f9814ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59534/0 (socket says 172.21.15.27:59534) 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9dadb640 1 -- 172.21.15.27:0/3114674695 learned_addr learned my addr 172.21.15.27:0/3114674695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9cad9640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980b2b50 0x7f0f9815fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9d2da640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f98071540 0x7f0f980b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9dadb640 1 -- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980b2b50 msgr2=0x7f0f9815fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9dadb640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980b2b50 0x7f0f9815fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9dadb640 1 -- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f98071540 msgr2=0x7f0f980b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9dadb640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f98071540 0x7f0f980b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9dadb640 1 -- 172.21.15.27:0/3114674695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f980b5f30 con 0x7f0f980d3400 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9cad9640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980b2b50 0x7f0f9815fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.280+0000 7f0f9dadb640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980d3400 0x7f0f9814ecc0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0f90036040 tx=0x7f0f90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.281+0000 7f0f8dffb640 1 -- 172.21.15.27:0/3114674695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90043020 con 0x7f0f980d3400 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.281+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f98158d60 con 0x7f0f980d3400 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.281+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f98159320 con 0x7f0f980d3400 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.281+0000 7f0f8dffb640 1 -- 172.21.15.27:0/3114674695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9003e450 con 0x7f0f980d3400 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.281+0000 7f0f8dffb640 1 -- 172.21.15.27:0/3114674695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90009510 con 0x7f0f980d3400 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.282+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f78005290 con 0x7f0f980d3400 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.282+0000 7f0f8dffb640 1 -- 172.21.15.27:0/3114674695 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f90046020 con 0x7f0f980d3400 2024-03-06T07:25:44.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.283+0000 7f0f8dffb640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.283+0000 7f0f8dffb640 1 -- 172.21.15.27:0/3114674695 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f900ac6b0 con 0x7f0f980d3400 2024-03-06T07:25:44.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.283+0000 7f0f9d2da640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.284+0000 7f0f9d2da640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0f94006fd0 tx=0x7f0f94008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.288+0000 7f0f8dffb640 1 -- 172.21.15.27:0/3114674695 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f9004d030 con 0x7f0f980d3400 2024-03-06T07:25:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.469+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f78005850 con 0x7f0f980d3400 2024-03-06T07:25:44.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.470+0000 7f0f8dffb640 1 -- 172.21.15.27:0/3114674695 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f0f9007fe20 con 0x7f0f980d3400 2024-03-06T07:25:44.472 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:44.472 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:25.095713+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:21.134461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:22.434887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:44.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f7c06b4d0 msgr2=0x7f0f7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0f94006fd0 tx=0x7f0f94008040 comp rx=0 tx=0).stop 2024-03-06T07:25:44.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980d3400 msgr2=0x7f0f9814ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980d3400 0x7f0f9814ecc0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0f90036040 tx=0x7f0f90004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:44.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 shutdown_connections 2024-03-06T07:25:44.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f980d3400 0x7f0f9814ecc0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f980b2b50 0x7f0f9815fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 --2- 172.21.15.27:0/3114674695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f98071540 0x7f0f980b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 >> 172.21.15.27:0/3114674695 conn(0x7f0f980d3ca0 msgr2=0x7f0f980ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 shutdown_connections 2024-03-06T07:25:44.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.472+0000 7f0f9f53e640 1 -- 172.21.15.27:0/3114674695 wait complete. 2024-03-06T07:25:44.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:21.134461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:22.434887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:44.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:44.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.597+0000 7fb626956640 1 Processor -- start 2024-03-06T07:25:44.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.597+0000 7fb626956640 1 -- start start 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb626956640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 0x7fb6200b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb626956640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6200b83b0 con 0x7fb6200b2d30 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb626956640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6200d4960 con 0x7fb620071540 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb626956640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6200d3230 con 0x7fb6200b79f0 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb61f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb6200b2d30 0x7fb6200b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38266/0 (socket says 172.21.15.27:38266) 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb61f7fe640 1 -- 172.21.15.27:0/2713120869 learned_addr learned my addr 172.21.15.27:0/2713120869 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb624ef3640 1 --2- 172.21.15.27:0/2713120869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 0x7fb6200b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb61effd640 1 -- 172.21.15.27:0/2713120869 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248134348 0 0) 0x7fb6200b83b0 con 0x7fb6200b2d30 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb61effd640 1 -- 172.21.15.27:0/2713120869 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb608003520 con 0x7fb6200b2d30 2024-03-06T07:25:44.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb61effd640 1 -- 172.21.15.27:0/2713120869 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980893649 0 0) 0x7fb6200d4960 con 0x7fb620071540 2024-03-06T07:25:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb61effd640 1 -- 172.21.15.27:0/2713120869 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6200b83b0 con 0x7fb620071540 2024-03-06T07:25:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb624ef3640 1 -- 172.21.15.27:0/2713120869 >> v1:172.21.15.69:6789/0 conn(0x7fb620071540 legacy=0x7fb620071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb624ef3640 1 -- 172.21.15.27:0/2713120869 >> v1:172.21.15.27:6789/0 conn(0x7fb6200b2d30 legacy=0x7fb6200b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb61effd640 1 -- 172.21.15.27:0/2713120869 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129558711 0 0) 0x7fb608003520 con 0x7fb6200b2d30 2024-03-06T07:25:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.598+0000 7fb624ef3640 1 -- 172.21.15.27:0/2713120869 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6200d36c0 con 0x7fb6200b79f0 2024-03-06T07:25:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.599+0000 7fb624ef3640 1 --2- 172.21.15.27:0/2713120869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 0x7fb6200b7e70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb618009dc0 tx=0x7fb61802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cf710e54d12bf64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.599+0000 7fb61effd640 1 -- 172.21.15.27:0/2713120869 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb618039070 con 0x7fb6200b79f0 2024-03-06T07:25:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.599+0000 7fb61effd640 1 -- 172.21.15.27:0/2713120869 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61802c910 con 0x7fb6200b79f0 2024-03-06T07:25:44.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.599+0000 7fb61effd640 1 -- 172.21.15.27:0/2713120869 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614001670 con 0x7fb6200b79f0 2024-03-06T07:25:44.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.600+0000 7fb626956640 1 -- 172.21.15.27:0/2713120869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 msgr2=0x7fb6200b7e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.600+0000 7fb626956640 1 --2- 172.21.15.27:0/2713120869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 0x7fb6200b7e70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb618009dc0 tx=0x7fb61802bde0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.600+0000 7fb626956640 1 -- 172.21.15.27:0/2713120869 shutdown_connections 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.600+0000 7fb626956640 1 --2- 172.21.15.27:0/2713120869 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 0x7fb6200b7e70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.600+0000 7fb626956640 1 -- 172.21.15.27:0/2713120869 >> 172.21.15.27:0/2713120869 conn(0x7fb6200d3d60 msgr2=0x7fb6200d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.600+0000 7fb626956640 1 -- 172.21.15.27:0/2713120869 shutdown_connections 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.600+0000 7fb626956640 1 -- 172.21.15.27:0/2713120869 wait complete. 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.601+0000 7fb626956640 1 Processor -- start 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.601+0000 7fb626956640 1 -- start start 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb626956640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb620071540 0x7fb6201592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb626956640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6200b2d30 0x7fb6201566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb626956640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 0x7fb620154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb626956640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6200d2be0 con 0x7fb6200b2d30 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb626956640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6200d2900 con 0x7fb620071540 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb626956640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6200d2ec0 con 0x7fb6200b79f0 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb61f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6200b2d30 0x7fb6201566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb61f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6200b2d30 0x7fb6201566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59536/0 (socket says 172.21.15.27:59536) 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb61f7fe640 1 -- 172.21.15.27:0/2283107972 learned_addr learned my addr 172.21.15.27:0/2283107972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb61ffff640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb620071540 0x7fb6201592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb624ef3640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 0x7fb620154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb61f7fe640 1 -- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 msgr2=0x7fb620154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 0x7fb620154de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb61f7fe640 1 -- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb620071540 msgr2=0x7fb6201592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb620071540 0x7fb6201592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.602+0000 7fb61f7fe640 1 -- 172.21.15.27:0/2283107972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb620159b50 con 0x7fb6200b2d30 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.603+0000 7fb61f7fe640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6200b2d30 0x7fb6201566d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c750 tx=0x7fb61400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.603+0000 7fb61cff9640 1 -- 172.21.15.27:0/2283107972 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614017070 con 0x7fb6200b2d30 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.603+0000 7fb61cff9640 1 -- 172.21.15.27:0/2283107972 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb614005ad0 con 0x7fb6200b2d30 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.603+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6201f4850 con 0x7fb6200b2d30 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.603+0000 7fb61cff9640 1 -- 172.21.15.27:0/2283107972 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614007070 con 0x7fb6200b2d30 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.603+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb620159820 con 0x7fb6200b2d30 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.604+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb620071e50 con 0x7fb6200b2d30 2024-03-06T07:25:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.604+0000 7fb61cff9640 1 -- 172.21.15.27:0/2283107972 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb61401a070 con 0x7fb6200b2d30 2024-03-06T07:25:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.605+0000 7fb61cff9640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb60c06b500 0x7fb60c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.605+0000 7fb61cff9640 1 -- 172.21.15.27:0/2283107972 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb614089640 con 0x7fb6200b2d30 2024-03-06T07:25:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.605+0000 7fb61ffff640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb60c06b500 0x7fb60c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.606+0000 7fb61ffff640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb60c06b500 0x7fb60c06d9c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb6100011e0 tx=0x7fb610009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.614+0000 7fb61cff9640 1 -- 172.21.15.27:0/2283107972 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb620071e50 con 0x7fb6200b2d30 2024-03-06T07:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.794+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb620072070 con 0x7fb6200b2d30 2024-03-06T07:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.795+0000 7fb61cff9640 1 -- 172.21.15.27:0/2283107972 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb614057e70 con 0x7fb6200b2d30 2024-03-06T07:25:44.796 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:25:44.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb60c06b500 msgr2=0x7fb60c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb60c06b500 0x7fb60c06d9c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb6100011e0 tx=0x7fb610009040 comp rx=0 tx=0).stop 2024-03-06T07:25:44.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6200b2d30 msgr2=0x7fb6201566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6200b2d30 0x7fb6201566d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c750 tx=0x7fb61400cc10 comp rx=0 tx=0).stop 2024-03-06T07:25:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 shutdown_connections 2024-03-06T07:25:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb60c06b500 0x7fb60c06d9c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6200b79f0 0x7fb620154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6200b2d30 0x7fb6201566d0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 --2- 172.21.15.27:0/2283107972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb620071540 0x7fb6201592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 >> 172.21.15.27:0/2283107972 conn(0x7fb6200d3d60 msgr2=0x7fb62015adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 shutdown_connections 2024-03-06T07:25:44.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.797+0000 7fb626956640 1 -- 172.21.15.27:0/2283107972 wait complete. 2024-03-06T07:25:44.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:44.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.921+0000 7f17128af640 1 Processor -- start 2024-03-06T07:25:44.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.921+0000 7f17128af640 1 -- start start 2024-03-06T07:25:44.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f17128af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c15dd90 0x7f170c0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f17128af640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f170c0b6d10 0x7f170c0b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f17128af640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170c0d3740 con 0x7f170c0d29d0 2024-03-06T07:25:44.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f17128af640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170c0d3400 con 0x7f170c0b6d10 2024-03-06T07:25:44.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f17128af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0d3120 con 0x7f170c15dd90 2024-03-06T07:25:44.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f170b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f170c0d29d0 0x7f170c0d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38276/0 (socket says 172.21.15.27:38276) 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f170b7fe640 1 -- 172.21.15.27:0/3021823121 learned_addr learned my addr 172.21.15.27:0/3021823121 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f170bfff640 1 --2- 172.21.15.27:0/3021823121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c15dd90 0x7f170c0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f1710e4c640 1 --2- 172.21.15.27:0/3021823121 >> v2:172.21.15.69:3300/0 conn(0x7f170c0b6d10 0x7f170c0b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f170affd640 1 -- 172.21.15.27:0/3021823121 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926839314 0 0) 0x7f170c0d3740 con 0x7f170c0d29d0 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.923+0000 7f170affd640 1 -- 172.21.15.27:0/3021823121 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f4003520 con 0x7f170c0d29d0 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.924+0000 7f170bfff640 1 -- 172.21.15.27:0/3021823121 >> v2:172.21.15.69:3300/0 conn(0x7f170c0b6d10 msgr2=0x7f170c0b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.924+0000 7f170bfff640 1 --2- 172.21.15.27:0/3021823121 >> v2:172.21.15.69:3300/0 conn(0x7f170c0b6d10 0x7f170c0b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.924+0000 7f170affd640 1 -- 172.21.15.27:0/3021823121 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784715351 0 0) 0x7f16f4003520 con 0x7f170c0d29d0 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.924+0000 7f170bfff640 1 -- 172.21.15.27:0/3021823121 >> v1:172.21.15.27:6789/0 conn(0x7f170c0d29d0 legacy=0x7f170c0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.924+0000 7f170bfff640 1 -- 172.21.15.27:0/3021823121 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c0d39e0 con 0x7f170c15dd90 2024-03-06T07:25:44.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.924+0000 7f1710e4c640 1 --2- 172.21.15.27:0/3021823121 >> v2:172.21.15.69:3300/0 conn(0x7f170c0b6d10 0x7f170c0b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:44.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.924+0000 7f170bfff640 1 --2- 172.21.15.27:0/3021823121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c15dd90 0x7f170c0ceea0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0097a0 tx=0x7f16fc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc16c47aa543f17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f170affd640 1 -- 172.21.15.27:0/3021823121 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc038070 con 0x7f170c15dd90 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f170affd640 1 -- 172.21.15.27:0/3021823121 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc031500 con 0x7f170c15dd90 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f170affd640 1 -- 172.21.15.27:0/3021823121 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700001670 con 0x7f170c15dd90 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f17128af640 1 -- 172.21.15.27:0/3021823121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c15dd90 msgr2=0x7f170c0ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f17128af640 1 --2- 172.21.15.27:0/3021823121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c15dd90 0x7f170c0ceea0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0097a0 tx=0x7f16fc02b820 comp rx=0 tx=0).stop 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f17128af640 1 -- 172.21.15.27:0/3021823121 shutdown_connections 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f17128af640 1 --2- 172.21.15.27:0/3021823121 >> v2:172.21.15.69:3300/0 conn(0x7f170c0b6d10 0x7f170c0b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f17128af640 1 --2- 172.21.15.27:0/3021823121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c15dd90 0x7f170c0ceea0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f17128af640 1 -- 172.21.15.27:0/3021823121 >> 172.21.15.27:0/3021823121 conn(0x7f170c071960 msgr2=0x7f170c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f17128af640 1 -- 172.21.15.27:0/3021823121 shutdown_connections 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.925+0000 7f17128af640 1 -- 172.21.15.27:0/3021823121 wait complete. 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.926+0000 7f17128af640 1 Processor -- start 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.926+0000 7f17128af640 1 -- start start 2024-03-06T07:25:44.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f17128af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f170c0b6d10 0x7f170c0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f17128af640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c0d29d0 0x7f170c0c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f17128af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c15dd90 0x7f170c15a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f17128af640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0d2df0 con 0x7f170c15dd90 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f17128af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c15e170 con 0x7f170c0b6d10 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f17128af640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0d48f0 con 0x7f170c0d29d0 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f1710e4c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c15dd90 0x7f170c15a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f1710e4c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c15dd90 0x7f170c15a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59540/0 (socket says 172.21.15.27:59540) 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f1710e4c640 1 -- 172.21.15.27:0/4137740861 learned_addr learned my addr 172.21.15.27:0/4137740861 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f170bfff640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f170c0b6d10 0x7f170c0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.927+0000 7f170b7fe640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c0d29d0 0x7f170c0c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f1710e4c640 1 -- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c0d29d0 msgr2=0x7f170c0c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f1710e4c640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c0d29d0 0x7f170c0c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f1710e4c640 1 -- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f170c0b6d10 msgr2=0x7f170c0cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f1710e4c640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f170c0b6d10 0x7f170c0cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f1710e4c640 1 -- 172.21.15.27:0/4137740861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c0c71c0 con 0x7f170c15dd90 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f170b7fe640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c0d29d0 0x7f170c0c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f1710e4c640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c15dd90 0x7f170c15a370 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f170400efc0 tx=0x7f1704009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f1708ff9640 1 -- 172.21.15.27:0/4137740861 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704018030 con 0x7f170c15dd90 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f1708ff9640 1 -- 172.21.15.27:0/4137740861 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170400cde0 con 0x7f170c15dd90 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c0cafc0 con 0x7f170c15dd90 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f1708ff9640 1 -- 172.21.15.27:0/4137740861 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704003bb0 con 0x7f170c15dd90 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.928+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c0cb580 con 0x7f170c15dd90 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.929+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c072a30 con 0x7f170c15dd90 2024-03-06T07:25:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.929+0000 7f1708ff9640 1 -- 172.21.15.27:0/4137740861 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f170401b070 con 0x7f170c15dd90 2024-03-06T07:25:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.930+0000 7f1708ff9640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f16f806b500 0x7f16f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.930+0000 7f1708ff9640 1 -- 172.21.15.27:0/4137740861 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f170408ab20 con 0x7f170c15dd90 2024-03-06T07:25:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.930+0000 7f170bfff640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f16f806b500 0x7f16f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.931+0000 7f170bfff640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f16f806b500 0x7f16f806d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f16fc035000 tx=0x7f16fc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:44.934+0000 7f1708ff9640 1 -- 172.21.15.27:0/4137740861 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1704010040 con 0x7f170c15dd90 2024-03-06T07:25:45.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.114+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f170c04f410 con 0x7f170c15dd90 2024-03-06T07:25:45.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.115+0000 7f1708ff9640 1 -- 172.21.15.27:0/4137740861 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1704059a30 con 0x7f170c15dd90 2024-03-06T07:25:45.116 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:25:45.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f16f806b500 msgr2=0x7f16f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f16f806b500 0x7f16f806d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f16fc035000 tx=0x7f16fc008030 comp rx=0 tx=0).stop 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c15dd90 msgr2=0x7f170c15a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c15dd90 0x7f170c15a370 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f170400efc0 tx=0x7f1704009f90 comp rx=0 tx=0).stop 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 shutdown_connections 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f16f806b500 0x7f16f806d9c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c15dd90 0x7f170c15a370 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f170c0d29d0 0x7f170c0c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 --2- 172.21.15.27:0/4137740861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f170c0b6d10 0x7f170c0cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 >> 172.21.15.27:0/4137740861 conn(0x7f170c071960 msgr2=0x7f170c0d41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 shutdown_connections 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.117+0000 7f17128af640 1 -- 172.21.15.27:0/4137740861 wait complete. 2024-03-06T07:25:45.129 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:45.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.238+0000 7fc24c2c9640 1 Processor -- start 2024-03-06T07:25:45.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.238+0000 7fc24c2c9640 1 -- start start 2024-03-06T07:25:45.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.238+0000 7fc24c2c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 0x7fc244071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc24c2c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2440b2d30 0x7fc2440ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc24c2c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2440cecb0 0x7fc2440cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc24c2c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2440d3230 con 0x7fc2440b2d30 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc24c2c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc244071e60 con 0x7fc2440cecb0 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc24c2c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2440d4960 con 0x7fc244071540 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc249864640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2440b2d30 0x7fc2440ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc249864640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2440b2d30 0x7fc2440ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59550/0 (socket says 172.21.15.27:59550) 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc249864640 1 -- 172.21.15.27:0/1698990581 learned_addr learned my addr 172.21.15.27:0/1698990581 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc24a866640 1 --2- 172.21.15.27:0/1698990581 >> v2:172.21.15.69:3300/0 conn(0x7fc2440cecb0 0x7fc2440cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc24a065640 1 --2- 172.21.15.27:0/1698990581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 0x7fc244071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc249864640 1 -- 172.21.15.27:0/1698990581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 msgr2=0x7fc244071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc249864640 1 --2- 172.21.15.27:0/1698990581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 0x7fc244071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc249864640 1 -- 172.21.15.27:0/1698990581 >> v2:172.21.15.69:3300/0 conn(0x7fc2440cecb0 msgr2=0x7fc2440cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc249864640 1 --2- 172.21.15.27:0/1698990581 >> v2:172.21.15.69:3300/0 conn(0x7fc2440cecb0 0x7fc2440cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc249864640 1 -- 172.21.15.27:0/1698990581 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2440d36c0 con 0x7fc2440b2d30 2024-03-06T07:25:45.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc24a866640 1 --2- 172.21.15.27:0/1698990581 >> v2:172.21.15.69:3300/0 conn(0x7fc2440cecb0 0x7fc2440cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:45.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.239+0000 7fc24a065640 1 --2- 172.21.15.27:0/1698990581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 0x7fc244071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.240+0000 7fc249864640 1 --2- 172.21.15.27:0/1698990581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 0x7fc2440ce770 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc244068380 tx=0x7fc23402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6c8075b31b4af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.240+0000 7fc249063640 1 -- 172.21.15.27:0/1698990581 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234035020 con 0x7fc2440b2d30 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.240+0000 7fc249063640 1 -- 172.21.15.27:0/1698990581 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc234031870 con 0x7fc2440b2d30 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.240+0000 7fc249063640 1 -- 172.21.15.27:0/1698990581 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234031c40 con 0x7fc2440b2d30 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.240+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1698990581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 msgr2=0x7fc2440ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.240+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1698990581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 0x7fc2440ce770 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc244068380 tx=0x7fc23402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.241+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1698990581 shutdown_connections 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.241+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1698990581 >> v2:172.21.15.69:3300/0 conn(0x7fc2440cecb0 0x7fc2440cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.241+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1698990581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 0x7fc2440ce770 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.241+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1698990581 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 0x7fc244071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.241+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1698990581 >> 172.21.15.27:0/1698990581 conn(0x7fc2440d3d60 msgr2=0x7fc2440d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.241+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1698990581 shutdown_connections 2024-03-06T07:25:45.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.241+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1698990581 wait complete. 2024-03-06T07:25:45.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.242+0000 7fc24c2c9640 1 Processor -- start 2024-03-06T07:25:45.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.242+0000 7fc24c2c9640 1 -- start start 2024-03-06T07:25:45.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.242+0000 7fc24c2c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 0x7fc2441f50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.242+0000 7fc24c2c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 0x7fc2440ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.242+0000 7fc24c2c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2440cecb0 0x7fc2440ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.242+0000 7fc24c2c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2440d2be0 con 0x7fc2440b2d30 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.242+0000 7fc24c2c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2440d2900 con 0x7fc2440cecb0 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.242+0000 7fc24c2c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2440d2ec0 con 0x7fc244071540 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc249864640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 0x7fc2440ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc249864640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 0x7fc2440ae130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59554/0 (socket says 172.21.15.27:59554) 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc249864640 1 -- 172.21.15.27:0/1496092138 learned_addr learned my addr 172.21.15.27:0/1496092138 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc24a065640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 0x7fc2441f50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc24a866640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2440cecb0 0x7fc2440ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc249864640 1 -- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 msgr2=0x7fc2441f50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc249864640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 0x7fc2441f50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc249864640 1 -- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2440cecb0 msgr2=0x7fc2440ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc249864640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2440cecb0 0x7fc2440ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc249864640 1 -- 172.21.15.27:0/1496092138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2440b4440 con 0x7fc2440b2d30 2024-03-06T07:25:45.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc249864640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 0x7fc2440ae130 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc23402b950 tx=0x7fc2340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc23affd640 1 -- 172.21.15.27:0/1496092138 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234040070 con 0x7fc2440b2d30 2024-03-06T07:25:45.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc23affd640 1 -- 172.21.15.27:0/1496092138 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23403a420 con 0x7fc2440b2d30 2024-03-06T07:25:45.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2440b4720 con 0x7fc2440b2d30 2024-03-06T07:25:45.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc23affd640 1 -- 172.21.15.27:0/1496092138 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234035040 con 0x7fc2440b2d30 2024-03-06T07:25:45.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.243+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2440b4ce0 con 0x7fc2440b2d30 2024-03-06T07:25:45.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.245+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2440c6fc0 con 0x7fc2440b2d30 2024-03-06T07:25:45.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.245+0000 7fc23affd640 1 -- 172.21.15.27:0/1496092138 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc23404b060 con 0x7fc2440b2d30 2024-03-06T07:25:45.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.245+0000 7fc23affd640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc22806b4d0 0x7fc22806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.245+0000 7fc23affd640 1 -- 172.21.15.27:0/1496092138 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2340ac070 con 0x7fc2440b2d30 2024-03-06T07:25:45.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.246+0000 7fc24a065640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc22806b4d0 0x7fc22806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.246+0000 7fc24a065640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc22806b4d0 0x7fc22806d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc2440b2820 tx=0x7fc240008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.250+0000 7fc23affd640 1 -- 172.21.15.27:0/1496092138 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc23407a8a0 con 0x7fc2440b2d30 2024-03-06T07:25:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.430+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc244072070 con 0x7fc2440b2d30 2024-03-06T07:25:45.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.431+0000 7fc23affd640 1 -- 172.21.15.27:0/1496092138 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc23407e940 con 0x7fc2440b2d30 2024-03-06T07:25:45.432 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:25:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc22806b4d0 msgr2=0x7fc22806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc22806b4d0 0x7fc22806d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc2440b2820 tx=0x7fc240008040 comp rx=0 tx=0).stop 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 msgr2=0x7fc2440ae130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 0x7fc2440ae130 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc23402b950 tx=0x7fc2340046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 shutdown_connections 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc22806b4d0 0x7fc22806d990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2440cecb0 0x7fc2440ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2440b2d30 0x7fc2440ae130 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 --2- 172.21.15.27:0/1496092138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc244071540 0x7fc2441f50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 >> 172.21.15.27:0/1496092138 conn(0x7fc2440d3d60 msgr2=0x7fc2440c76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 shutdown_connections 2024-03-06T07:25:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.433+0000 7fc24c2c9640 1 -- 172.21.15.27:0/1496092138 wait complete. 2024-03-06T07:25:45.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.556+0000 7fd3a3fe2640 1 Processor -- start 2024-03-06T07:25:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.556+0000 7fd3a3fe2640 1 -- start start 2024-03-06T07:25:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a3fe2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd39c06d400 0x7fd39c15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a3fe2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39c0729b0 con 0x7fd39c15bac0 2024-03-06T07:25:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a3fe2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39c06c920 con 0x7fd39c0aee10 2024-03-06T07:25:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a3fe2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39c06cf70 con 0x7fd39c06d400 2024-03-06T07:25:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a257f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd39c06d400 0x7fd39c15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a1d7e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd39c15bac0 0x7fd39c0ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38278/0 (socket says 172.21.15.27:38278) 2024-03-06T07:25:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a1d7e640 1 -- 172.21.15.27:0/1350926294 learned_addr learned my addr 172.21.15.27:0/1350926294 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a257f640 1 -- 172.21.15.27:0/1350926294 >> v1:172.21.15.69:6789/0 conn(0x7fd39c0aee10 legacy=0x7fd39c070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a257f640 1 -- 172.21.15.27:0/1350926294 >> v1:172.21.15.27:6789/0 conn(0x7fd39c15bac0 legacy=0x7fd39c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.557+0000 7fd3a257f640 1 -- 172.21.15.27:0/1350926294 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c06cc00 con 0x7fd39c06d400 2024-03-06T07:25:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.558+0000 7fd3a257f640 1 --2- 172.21.15.27:0/1350926294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c06d400 0x7fd39c15cbe0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd394009d90 tx=0x7fd39402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb4ef5ea0ccebc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.558+0000 7fd3a0d7c640 1 -- 172.21.15.27:0/1350926294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394039070 con 0x7fd39c06d400 2024-03-06T07:25:45.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.559+0000 7fd3a0d7c640 1 -- 172.21.15.27:0/1350926294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39402cc60 con 0x7fd39c06d400 2024-03-06T07:25:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.559+0000 7fd3a0d7c640 1 -- 172.21.15.27:0/1350926294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394034660 con 0x7fd39c06d400 2024-03-06T07:25:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.559+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/1350926294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c06d400 msgr2=0x7fd39c15cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.559+0000 7fd3a3fe2640 1 --2- 172.21.15.27:0/1350926294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c06d400 0x7fd39c15cbe0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd394009d90 tx=0x7fd39402beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.559+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/1350926294 shutdown_connections 2024-03-06T07:25:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.559+0000 7fd3a3fe2640 1 --2- 172.21.15.27:0/1350926294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c06d400 0x7fd39c15cbe0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.559+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/1350926294 >> 172.21.15.27:0/1350926294 conn(0x7fd39c066d00 msgr2=0x7fd39c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.559+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/1350926294 shutdown_connections 2024-03-06T07:25:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.559+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/1350926294 wait complete. 2024-03-06T07:25:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.560+0000 7fd3a3fe2640 1 Processor -- start 2024-03-06T07:25:45.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.560+0000 7fd3a3fe2640 1 -- start start 2024-03-06T07:25:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a3fe2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd39c06d400 0x7fd39c15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a3fe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd39c0aee10 0x7fd39c159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a3fe2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c15bac0 0x7fd39c143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a3fe2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c15c020 con 0x7fd39c15bac0 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a3fe2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c15bea0 con 0x7fd39c06d400 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a3fe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c072580 con 0x7fd39c0aee10 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a257f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c15bac0 0x7fd39c143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a257f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c15bac0 0x7fd39c143360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59568/0 (socket says 172.21.15.27:59568) 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a257f640 1 -- 172.21.15.27:0/179441512 learned_addr learned my addr 172.21.15.27:0/179441512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a1d7e640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd39c06d400 0x7fd39c15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a157d640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd39c0aee10 0x7fd39c159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a257f640 1 -- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd39c0aee10 msgr2=0x7fd39c159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a257f640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd39c0aee10 0x7fd39c159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a257f640 1 -- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd39c06d400 msgr2=0x7fd39c15aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a257f640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd39c06d400 0x7fd39c15aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.561+0000 7fd3a257f640 1 -- 172.21.15.27:0/179441512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c1438a0 con 0x7fd39c15bac0 2024-03-06T07:25:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.562+0000 7fd3a257f640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c15bac0 0x7fd39c143360 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd394037040 tx=0x7fd394004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.562+0000 7fd3927fc640 1 -- 172.21.15.27:0/179441512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394039050 con 0x7fd39c15bac0 2024-03-06T07:25:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.562+0000 7fd3927fc640 1 -- 172.21.15.27:0/179441512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39403a040 con 0x7fd39c15bac0 2024-03-06T07:25:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.562+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39c1597b0 con 0x7fd39c15bac0 2024-03-06T07:25:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.562+0000 7fd3927fc640 1 -- 172.21.15.27:0/179441512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39403e610 con 0x7fd39c15bac0 2024-03-06T07:25:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.562+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39c159d70 con 0x7fd39c15bac0 2024-03-06T07:25:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.563+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39c067ff0 con 0x7fd39c15bac0 2024-03-06T07:25:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.564+0000 7fd3927fc640 1 -- 172.21.15.27:0/179441512 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd394047020 con 0x7fd39c15bac0 2024-03-06T07:25:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.564+0000 7fd3927fc640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd38006b4d0 0x7fd38006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.564+0000 7fd3927fc640 1 -- 172.21.15.27:0/179441512 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3940ac3b0 con 0x7fd39c15bac0 2024-03-06T07:25:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.565+0000 7fd3a1d7e640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd38006b4d0 0x7fd38006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.565+0000 7fd3a1d7e640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd38006b4d0 0x7fd38006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd398007fd0 tx=0x7fd398007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.569+0000 7fd3927fc640 1 -- 172.21.15.27:0/179441512 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd39404e020 con 0x7fd39c15bac0 2024-03-06T07:25:45.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.763+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd39c06de50 con 0x7fd39c15bac0 2024-03-06T07:25:45.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.764+0000 7fd3927fc640 1 -- 172.21.15.27:0/179441512 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd39407ad30 con 0x7fd39c15bac0 2024-03-06T07:25:45.765 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd38006b4d0 msgr2=0x7fd38006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd38006b4d0 0x7fd38006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd398007fd0 tx=0x7fd398007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c15bac0 msgr2=0x7fd39c143360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c15bac0 0x7fd39c143360 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd394037040 tx=0x7fd394004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 shutdown_connections 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd38006b4d0 0x7fd38006d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd39c15bac0 0x7fd39c143360 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd39c0aee10 0x7fd39c159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 --2- 172.21.15.27:0/179441512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd39c06d400 0x7fd39c15aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 >> 172.21.15.27:0/179441512 conn(0x7fd39c066d00 msgr2=0x7fd39c15e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 shutdown_connections 2024-03-06T07:25:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:45.766+0000 7fd3a3fe2640 1 -- 172.21.15.27:0/179441512 wait complete. 2024-03-06T07:25:45.778 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:25:45.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:45.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:45.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[58768]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:45.811 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:48.059 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:48.062 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:51.913 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:25:51.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.913 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:51.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:52.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.091+0000 7fe086423640 1 Processor -- start 2024-03-06T07:25:52.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.091+0000 7fe086423640 1 -- start start 2024-03-06T07:25:52.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe086423640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08015df00 0x7fe0800d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe086423640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0800b95a0 0x7fe0800b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe086423640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe0800b9a30 0x7fe0800b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe086423640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0800cea30 con 0x7fe0800b95a0 2024-03-06T07:25:52.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe086423640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0800d2af0 con 0x7fe0800b9a30 2024-03-06T07:25:52.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe086423640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800cf3c0 con 0x7fe08015df00 2024-03-06T07:25:52.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe07f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0800b95a0 0x7fe0800b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe07f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0800b95a0 0x7fe0800b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59570/0 (socket says 172.21.15.27:59570) 2024-03-06T07:25:52.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe07f7fe640 1 -- 172.21.15.27:0/2305488214 learned_addr learned my addr 172.21.15.27:0/2305488214 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:52.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe0849c0640 1 --2- 172.21.15.27:0/2305488214 >> v2:172.21.15.69:3300/0 conn(0x7fe0800b9a30 0x7fe0800b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe07ffff640 1 --2- 172.21.15.27:0/2305488214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08015df00 0x7fe0800d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe07f7fe640 1 -- 172.21.15.27:0/2305488214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08015df00 msgr2=0x7fe0800d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe07f7fe640 1 --2- 172.21.15.27:0/2305488214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08015df00 0x7fe0800d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe07f7fe640 1 -- 172.21.15.27:0/2305488214 >> v2:172.21.15.69:3300/0 conn(0x7fe0800b9a30 msgr2=0x7fe0800b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe07f7fe640 1 --2- 172.21.15.27:0/2305488214 >> v2:172.21.15.69:3300/0 conn(0x7fe0800b9a30 0x7fe0800b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.092+0000 7fe07f7fe640 1 -- 172.21.15.27:0/2305488214 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0800cf0e0 con 0x7fe0800b95a0 2024-03-06T07:25:52.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.093+0000 7fe07f7fe640 1 --2- 172.21.15.27:0/2305488214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0800b95a0 0x7fe0800b5700 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe0800d1160 tx=0x7fe07002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b78d89f07c8bb43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.093+0000 7fe07effd640 1 -- 172.21.15.27:0/2305488214 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070035020 con 0x7fe0800b95a0 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.093+0000 7fe07effd640 1 -- 172.21.15.27:0/2305488214 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe070031870 con 0x7fe0800b95a0 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.093+0000 7fe07effd640 1 -- 172.21.15.27:0/2305488214 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070031c40 con 0x7fe0800b95a0 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.094+0000 7fe086423640 1 -- 172.21.15.27:0/2305488214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0800b95a0 msgr2=0x7fe0800b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.094+0000 7fe086423640 1 --2- 172.21.15.27:0/2305488214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0800b95a0 0x7fe0800b5700 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe0800d1160 tx=0x7fe07002b910 comp rx=0 tx=0).stop 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.094+0000 7fe086423640 1 -- 172.21.15.27:0/2305488214 shutdown_connections 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.094+0000 7fe086423640 1 --2- 172.21.15.27:0/2305488214 >> v2:172.21.15.69:3300/0 conn(0x7fe0800b9a30 0x7fe0800b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.094+0000 7fe086423640 1 --2- 172.21.15.27:0/2305488214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0800b95a0 0x7fe0800b5700 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.094+0000 7fe086423640 1 --2- 172.21.15.27:0/2305488214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08015df00 0x7fe0800d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.094+0000 7fe086423640 1 -- 172.21.15.27:0/2305488214 >> 172.21.15.27:0/2305488214 conn(0x7fe080071960 msgr2=0x7fe080071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.094+0000 7fe086423640 1 -- 172.21.15.27:0/2305488214 shutdown_connections 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.094+0000 7fe086423640 1 -- 172.21.15.27:0/2305488214 wait complete. 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.095+0000 7fe086423640 1 Processor -- start 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.095+0000 7fe086423640 1 -- start start 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe086423640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0800b95a0 0x7fe080160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe086423640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe08015df00 0x7fe0800b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe086423640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08014d8d0 0x7fe08014dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe086423640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800cec70 con 0x7fe08015df00 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe086423640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800ce750 con 0x7fe0800b95a0 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe086423640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800cedf0 con 0x7fe08014d8d0 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe07f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe08015df00 0x7fe0800b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe07f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe08015df00 0x7fe0800b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59584/0 (socket says 172.21.15.27:59584) 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe07f7fe640 1 -- 172.21.15.27:0/954906890 learned_addr learned my addr 172.21.15.27:0/954906890 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe07ffff640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0800b95a0 0x7fe080160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe0849c0640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08014d8d0 0x7fe08014dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe07f7fe640 1 -- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08014d8d0 msgr2=0x7fe08014dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe07f7fe640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08014d8d0 0x7fe08014dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe07f7fe640 1 -- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0800b95a0 msgr2=0x7fe080160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe07f7fe640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0800b95a0 0x7fe080160b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.096+0000 7fe07f7fe640 1 -- 172.21.15.27:0/954906890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0800c8cf0 con 0x7fe08015df00 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.097+0000 7fe07f7fe640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe08015df00 0x7fe0800b5230 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe07002b950 tx=0x7fe0700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.097+0000 7fe07cff9640 1 -- 172.21.15.27:0/954906890 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070040070 con 0x7fe08015df00 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.097+0000 7fe07cff9640 1 -- 172.21.15.27:0/954906890 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe070030d00 con 0x7fe08015df00 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.097+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0800c8fd0 con 0x7fe08015df00 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.097+0000 7fe07cff9640 1 -- 172.21.15.27:0/954906890 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070035040 con 0x7fe08015df00 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.097+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0800c9870 con 0x7fe08015df00 2024-03-06T07:25:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.098+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe080067ff0 con 0x7fe08015df00 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.099+0000 7fe07cff9640 1 -- 172.21.15.27:0/954906890 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe07004b060 con 0x7fe08015df00 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.099+0000 7fe07cff9640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe06c06b4d0 0x7fe06c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.099+0000 7fe07cff9640 1 -- 172.21.15.27:0/954906890 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0700abf30 con 0x7fe08015df00 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.099+0000 7fe07ffff640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe06c06b4d0 0x7fe06c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.100+0000 7fe07ffff640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe06c06b4d0 0x7fe06c06d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe074007920 tx=0x7fe074008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.104+0000 7fe07cff9640 1 -- 172.21.15.27:0/954906890 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe07007a6d0 con 0x7fe08015df00 2024-03-06T07:25:52.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.322+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0800c9360 con 0x7fe08015df00 2024-03-06T07:25:52.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.323+0000 7fe07cff9640 1 -- 172.21.15.27:0/954906890 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fe07007e770 con 0x7fe08015df00 2024-03-06T07:25:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:52.325 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5324,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3885016831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3885016831},{"type":"v1","addr":"172.21.15.69:6836","nonce":3885016831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5327,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3510003111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3510003111},{"type":"v1","addr":"172.21.15.69:6837","nonce":3510003111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5692,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3053251134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3053251134},{"type":"v1","addr":"172.21.15.27:6835","nonce":3053251134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:23.445711+0000","modified":"2024-03-06T07:25:26.468573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6129":{"gid":6129,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6833/60877422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":60877422},{"type":"v1","addr":"172.21.15.27:6833","nonce":60877422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6129,"qdb_cluster":[6129]},"id":4}]} 2024-03-06T07:25:52.325 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-03-06T07:25:52.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe06c06b4d0 msgr2=0x7fe06c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe06c06b4d0 0x7fe06c06d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe074007920 tx=0x7fe074008040 comp rx=0 tx=0).stop 2024-03-06T07:25:52.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe08015df00 msgr2=0x7fe0800b5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe08015df00 0x7fe0800b5230 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe07002b950 tx=0x7fe0700046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 shutdown_connections 2024-03-06T07:25:52.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe06c06b4d0 0x7fe06c06d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe08014d8d0 0x7fe08014dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe08015df00 0x7fe0800b5230 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 --2- 172.21.15.27:0/954906890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0800b95a0 0x7fe080160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 >> 172.21.15.27:0/954906890 conn(0x7fe080071960 msgr2=0x7fe0800cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 shutdown_connections 2024-03-06T07:25:52.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.325+0000 7fe086423640 1 -- 172.21.15.27:0/954906890 wait complete. 2024-03-06T07:25:52.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:25:52.460 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:25:52.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.565+0000 7f59581d7640 1 Processor -- start 2024-03-06T07:25:52.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.566+0000 7f59581d7640 1 -- start start 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.566+0000 7f59581d7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5950158520 0x7f595006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.566+0000 7f59581d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 0x7f59500b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f59581d7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5950071240 con 0x7f595006dd60 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f59581d7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5950071b70 con 0x7f5950158520 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f59581d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5950071520 con 0x7f59500aee10 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f5955f73640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5950158520 0x7f595006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f5955f73640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5950158520 0x7f595006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59596/0 (socket says 172.21.15.27:59596) 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f5955f73640 1 -- 172.21.15.27:0/1237908324 learned_addr learned my addr 172.21.15.27:0/1237908324 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f5956774640 1 --2- 172.21.15.27:0/1237908324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 0x7f59500b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f5955f73640 1 -- 172.21.15.27:0/1237908324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 msgr2=0x7f59500b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f5955f73640 1 --2- 172.21.15.27:0/1237908324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 0x7f59500b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f5955f73640 1 -- 172.21.15.27:0/1237908324 >> v1:172.21.15.69:6789/0 conn(0x7f595006dd60 legacy=0x7f59500ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.567+0000 7f5955f73640 1 -- 172.21.15.27:0/1237908324 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5950071800 con 0x7f5950158520 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.568+0000 7f5955f73640 1 --2- 172.21.15.27:0/1237908324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 0x7f595006d820 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f594c009870 tx=0x7f594c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af06d721184494 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.568+0000 7f5954f71640 1 -- 172.21.15.27:0/1237908324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c034070 con 0x7f5950158520 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.568+0000 7f5954f71640 1 -- 172.21.15.27:0/1237908324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594c030840 con 0x7f5950158520 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.568+0000 7f5954f71640 1 -- 172.21.15.27:0/1237908324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c030c10 con 0x7f5950158520 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.568+0000 7f59581d7640 1 -- 172.21.15.27:0/1237908324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 msgr2=0x7f595006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.568+0000 7f59581d7640 1 --2- 172.21.15.27:0/1237908324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 0x7f595006d820 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f594c009870 tx=0x7f594c02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.569+0000 7f59581d7640 1 -- 172.21.15.27:0/1237908324 shutdown_connections 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.569+0000 7f59581d7640 1 --2- 172.21.15.27:0/1237908324 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 0x7f59500b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.569+0000 7f59581d7640 1 --2- 172.21.15.27:0/1237908324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 0x7f595006d820 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.569+0000 7f59581d7640 1 -- 172.21.15.27:0/1237908324 >> 172.21.15.27:0/1237908324 conn(0x7f5950066d00 msgr2=0x7f59500670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.569+0000 7f59581d7640 1 -- 172.21.15.27:0/1237908324 shutdown_connections 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.569+0000 7f59581d7640 1 -- 172.21.15.27:0/1237908324 wait complete. 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.570+0000 7f59581d7640 1 Processor -- start 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.570+0000 7f59581d7640 1 -- start start 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f59581d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f595006dd60 0x7f59500d0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f59581d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 0x7f5950145f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f59581d7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 0x7f59500ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f59581d7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5950158a80 con 0x7f5950158520 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f59581d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5950158900 con 0x7f595006dd60 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f59581d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59500b3b80 con 0x7f59500aee10 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5956774640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 0x7f59500ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5956774640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 0x7f59500ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59602/0 (socket says 172.21.15.27:59602) 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5956774640 1 -- 172.21.15.27:0/3157448758 learned_addr learned my addr 172.21.15.27:0/3157448758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5955f73640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f595006dd60 0x7f59500d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5955772640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 0x7f5950145f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5956774640 1 -- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 msgr2=0x7f5950145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5956774640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 0x7f5950145f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5956774640 1 -- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f595006dd60 msgr2=0x7f59500d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5956774640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f595006dd60 0x7f59500d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5956774640 1 -- 172.21.15.27:0/3157448758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59501569a0 con 0x7f5950158520 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5955772640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 0x7f5950145f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5955f73640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f595006dd60 0x7f59500d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f5956774640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 0x7f59500ad8b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5948007e40 tx=0x7f594800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.571+0000 7f59427fc640 1 -- 172.21.15.27:0/3157448758 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948014620 con 0x7f5950158520 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.572+0000 7f59427fc640 1 -- 172.21.15.27:0/3157448758 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5948019070 con 0x7f5950158520 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.572+0000 7f59427fc640 1 -- 172.21.15.27:0/3157448758 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948014a20 con 0x7f5950158520 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.572+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5950156c80 con 0x7f5950158520 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.572+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5950157240 con 0x7f5950158520 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.573+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5950067ff0 con 0x7f5950158520 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.573+0000 7f59427fc640 1 -- 172.21.15.27:0/3157448758 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5948014bc0 con 0x7f5950158520 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.574+0000 7f59427fc640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593406b500 0x7f593406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.574+0000 7f59427fc640 1 -- 172.21.15.27:0/3157448758 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f594808a9c0 con 0x7f5950158520 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.574+0000 7f5955f73640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593406b500 0x7f593406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.575+0000 7f5955f73640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593406b500 0x7f593406d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f594c036000 tx=0x7f594c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.578+0000 7f59427fc640 1 -- 172.21.15.27:0/3157448758 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f594808ce20 con 0x7f5950158520 2024-03-06T07:25:52.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.795+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f595015d5b0 con 0x7f5950158520 2024-03-06T07:25:52.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.796+0000 7f59427fc640 1 -- 172.21.15.27:0/3157448758 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f59480598d0 con 0x7f5950158520 2024-03-06T07:25:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:52.798 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5324,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3885016831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3885016831},{"type":"v1","addr":"172.21.15.69:6836","nonce":3885016831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5327,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3510003111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3510003111},{"type":"v1","addr":"172.21.15.69:6837","nonce":3510003111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5692,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3053251134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3053251134},{"type":"v1","addr":"172.21.15.27:6835","nonce":3053251134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:23.445711+0000","modified":"2024-03-06T07:25:26.468573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6129":{"gid":6129,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6833/60877422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":60877422},{"type":"v1","addr":"172.21.15.27:6833","nonce":60877422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6129,"qdb_cluster":[6129]},"id":4}]} 2024-03-06T07:25:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-03-06T07:25:52.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593406b500 msgr2=0x7f593406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593406b500 0x7f593406d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f594c036000 tx=0x7f594c03b040 comp rx=0 tx=0).stop 2024-03-06T07:25:52.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 msgr2=0x7f59500ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 0x7f59500ad8b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5948007e40 tx=0x7f594800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 shutdown_connections 2024-03-06T07:25:52.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593406b500 0x7f593406d9c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5950158520 0x7f59500ad8b0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59500aee10 0x7f5950145f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 --2- 172.21.15.27:0/3157448758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f595006dd60 0x7f59500d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 >> 172.21.15.27:0/3157448758 conn(0x7f5950066d00 msgr2=0x7f59500af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 shutdown_connections 2024-03-06T07:25:52.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:52.798+0000 7f59581d7640 1 -- 172.21.15.27:0/3157448758 wait complete. 2024-03-06T07:25:52.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:25:52.952 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:25:52.952 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:25:52.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:53.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.058+0000 7f186a193640 1 Processor -- start 2024-03-06T07:25:53.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.058+0000 7f186a193640 1 -- start start 2024-03-06T07:25:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f186a193640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f186415df00 0x7f18640d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f186a193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18640b95a0 0x7f18640b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f186a193640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18640cea30 con 0x7f18640b9a30 2024-03-06T07:25:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f186a193640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18640d2af0 con 0x7f186415df00 2024-03-06T07:25:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f186a193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18640cf3c0 con 0x7f18640b95a0 2024-03-06T07:25:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f1863fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f18640b9a30 0x7f18640d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38284/0 (socket says 172.21.15.27:38284) 2024-03-06T07:25:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f1863fff640 1 -- 172.21.15.27:0/1088566681 learned_addr learned my addr 172.21.15.27:0/1088566681 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f1862ffd640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18640b95a0 0x7f18640b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f18637fe640 1 --2- 172.21.15.27:0/1088566681 >> v2:172.21.15.69:3300/0 conn(0x7f186415df00 0x7f18640d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966090151 0 0) 0x7f18640cea30 con 0x7f18640b9a30 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c003520 con 0x7f18640b9a30 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2456353571 0 0) 0x7f184c003520 con 0x7f18640b9a30 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18640b95a0 msgr2=0x7f18640b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f18627fc640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18640b95a0 0x7f18640b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.059+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 >> v2:172.21.15.69:3300/0 conn(0x7f186415df00 msgr2=0x7f18640d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 --2- 172.21.15.27:0/1088566681 >> v2:172.21.15.69:3300/0 conn(0x7f186415df00 0x7f18640d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18640cf0e0 con 0x7f18640b9a30 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f185c0019f0 con 0x7f18640b9a30 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 >> v1:172.21.15.27:6789/0 conn(0x7f18640b9a30 legacy=0x7f18640d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f184c0039b0 0x7f184c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f184c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18640b95a0 0x7f184c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18640d2af0 con 0x7f184c004800 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18640cf3c0 con 0x7f184c0039b0 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c00b9c0 con 0x7f18640b95a0 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f1862ffd640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f184c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18637fe640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f184c0039b0 0x7f184c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f1863fff640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18640b95a0 0x7f184c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f1862ffd640 1 -- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18640b95a0 msgr2=0x7f184c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f1862ffd640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18640b95a0 0x7f184c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f1862ffd640 1 -- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f184c0039b0 msgr2=0x7f184c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f1862ffd640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f184c0039b0 0x7f184c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f1862ffd640 1 -- 172.21.15.27:0/1088566681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c003520 con 0x7f184c004800 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f18637fe640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f184c0039b0 0x7f184c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:53.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.060+0000 7f1863fff640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18640b95a0 0x7f184c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.061+0000 7f1862ffd640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f184c004c80 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1858009ac0 tx=0x7f184c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6516f75cf939c7d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858039070 con 0x7f184c004800 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185802c6b0 con 0x7f184c004800 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f18627fc640 1 -- 172.21.15.27:0/1088566681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858009690 con 0x7f184c004800 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f186a193640 1 -- 172.21.15.27:0/1088566681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 msgr2=0x7f184c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f186a193640 1 --2- 172.21.15.27:0/1088566681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f184c004c80 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1858009ac0 tx=0x7f184c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f1863fff640 1 -- 172.21.15.27:0/1088566681 reap_dead start 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f186a193640 1 -- 172.21.15.27:0/1088566681 shutdown_connections 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f186a193640 1 -- 172.21.15.27:0/1088566681 >> 172.21.15.27:0/1088566681 conn(0x7f1864071960 msgr2=0x7f1864071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f186a193640 1 -- 172.21.15.27:0/1088566681 shutdown_connections 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.062+0000 7f186a193640 1 -- 172.21.15.27:0/1088566681 wait complete. 2024-03-06T07:25:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.063+0000 7f186a193640 1 Processor -- start 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f186a193640 1 -- start start 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f186a193640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f1864154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f186a193640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1864160c50 0x7f1864151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f186a193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1864155870 0x7f18640c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f186a193640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18640cec70 con 0x7f184c004800 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f186a193640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18640ce750 con 0x7f1864160c50 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f186a193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18640cedf0 con 0x7f1864155870 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f18637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f1864154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f18637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f1864154140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59616/0 (socket says 172.21.15.27:59616) 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.064+0000 7f18637fe640 1 -- 172.21.15.27:0/2516270450 learned_addr learned my addr 172.21.15.27:0/2516270450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f1863fff640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1864155870 0x7f18640c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f1862ffd640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1864160c50 0x7f1864151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f18637fe640 1 -- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1864155870 msgr2=0x7f18640c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f18637fe640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1864155870 0x7f18640c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f18637fe640 1 -- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1864160c50 msgr2=0x7f1864151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f18637fe640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1864160c50 0x7f1864151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f18637fe640 1 -- 172.21.15.27:0/2516270450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1864151fc0 con 0x7f184c004800 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f1862ffd640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1864160c50 0x7f1864151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f18637fe640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f1864154140 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f1854011d00 tx=0x7f185400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f184bfff640 1 -- 172.21.15.27:0/2516270450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185400fcc0 con 0x7f184c004800 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f184bfff640 1 -- 172.21.15.27:0/2516270450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185400fe60 con 0x7f184c004800 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18641547d0 con 0x7f184c004800 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f184bfff640 1 -- 172.21.15.27:0/2516270450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854014680 con 0x7f184c004800 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.065+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1864154d90 con 0x7f184c004800 2024-03-06T07:25:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.066+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1864067ff0 con 0x7f184c004800 2024-03-06T07:25:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.067+0000 7f184bfff640 1 -- 172.21.15.27:0/2516270450 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1854012040 con 0x7f184c004800 2024-03-06T07:25:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.068+0000 7f184bfff640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b500 0x7f185006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.068+0000 7f184bfff640 1 -- 172.21.15.27:0/2516270450 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f185408aad0 con 0x7f184c004800 2024-03-06T07:25:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.068+0000 7f1862ffd640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b500 0x7f185006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.069+0000 7f1862ffd640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b500 0x7f185006d9c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1858036000 tx=0x7f1858030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.073+0000 7f184bfff640 1 -- 172.21.15.27:0/2516270450 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1854019030 con 0x7f184c004800 2024-03-06T07:25:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.296+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1864072240 con 0x7f184c004800 2024-03-06T07:25:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.296+0000 7f184bfff640 1 -- 172.21.15.27:0/2516270450 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1854059950 con 0x7f184c004800 2024-03-06T07:25:53.298 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:53.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.298+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b500 msgr2=0x7f185006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.298+0000 7f186a193640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b500 0x7f185006d9c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1858036000 tx=0x7f1858030000 comp rx=0 tx=0).stop 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 msgr2=0x7f1864154140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f1864154140 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f1854011d00 tx=0x7f185400f610 comp rx=0 tx=0).stop 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 shutdown_connections 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f185006b500 0x7f185006d9c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1864155870 0x7f18640c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1864160c50 0x7f1864151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 --2- 172.21.15.27:0/2516270450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f184c004800 0x7f1864154140 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 >> 172.21.15.27:0/2516270450 conn(0x7f1864071960 msgr2=0x7f18640cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 shutdown_connections 2024-03-06T07:25:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:53.299+0000 7f186a193640 1 -- 172.21.15.27:0/2516270450 wait complete. 2024-03-06T07:25:53.311 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:53.311 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:53.311 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:25:53.311 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:53.311 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:53.311 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:53.311 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:53.311 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:53.311 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:53.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:53.312 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:25:53.327 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:53.327 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:25:53.328 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42536sec preferred_lft 42536sec 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:53.329 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:25:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:53.330 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:25:53.330 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:53.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:53.330 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:53.397 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:25:53.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:53.399 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:25:53.454 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:25:53.455 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:53.455 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:25:53.455 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:25:53.455 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:53.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:53.455 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:25:53.455 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:25:53.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:25:53.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:25:53.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:25:53.456 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:25:53.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:53.561 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:25:53.576 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:53.576 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:53.577 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:53.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:53.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:25:53.646 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:53.647 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:53.647 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:53.703 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:53.703 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:53.703 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:53.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:25:53.829 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:53.884 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:25:53.884 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:25:53.884 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:25:53.884 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:25:53.884 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:25:53.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:53.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:53.887 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:53.887 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:53.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:53.887 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:25:53.942 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:53.943 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:53.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:25:54.016 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:54.073 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:25:54.073 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:25:54.073 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:25:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:54.076 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:54.076 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:54.076 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:25:54.076 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:54.076 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:54.076 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:54.076 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:54.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:54.076 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:25:54.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:25:54.078+0000 7f16c180b480 -1 init, newargv = 0x564f14f13b80 newargc=15 2024-03-06T07:25:54.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[59932]: starting ceph client 2024-03-06T07:25:54.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[59932]: starting fuse 2024-03-06T07:25:54.095 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:25:54.096 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:25:54.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:54.096 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:25:54.096 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:25:54.096 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:25:54.096 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:25:54.096 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:54.096 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:25:54.096 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:25:54.098 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:54.098 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:54.098 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:54.098 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:25:54.226 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.59932.asok 2024-03-06T07:25:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:54.239 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59932.asok 2024-03-06T07:25:54.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:54.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59932.asok status 2024-03-06T07:25:54.381 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "59932", 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 6405, 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 6405 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3707046705 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3707046705 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.6405 192.168.0.1:0/3707046705", 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/3707046705", 2024-03-06T07:25:54.383 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 34, 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 58, 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:25:54.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:54.393 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:25:54.393 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:54.410 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:25:54.410 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:54.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:54.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:54.480 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:54.480 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:54.535 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:25:54.535 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:25:54.535 DEBUG:teuthology.orchestra.run.smithi027:> import uuid 2024-03-06T07:25:54.535 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:54.535 DEBUG:teuthology.orchestra.run.smithi027:> n = 75 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = False 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> if 0 > 0: 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = True 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> dpath = os.path.dirname(path) 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> try: 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> for i in range(n): 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:54.536 DEBUG:teuthology.orchestra.run.smithi027:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> f.write(f"{i}") 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> f.flush() 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(f.fileno()) 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> os.unlink(fpath) 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> if create_hard_links: 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> for j in range(0): 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> if True: 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> finally: 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> os.close(dirfd) 2024-03-06T07:25:54.537 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:25:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:54.993 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:54.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:55.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:55.122 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:55.122 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:55.122 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:55.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.285+0000 7fa6374b5640 1 Processor -- start 2024-03-06T07:25:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.285+0000 7fa6374b5640 1 -- start start 2024-03-06T07:25:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.286+0000 7fa6374b5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa63015df00 0x7fa6300d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.286+0000 7fa6374b5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300cea30 con 0x7fa6300b95a0 2024-03-06T07:25:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.286+0000 7fa6374b5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300d2af0 con 0x7fa6300b9a30 2024-03-06T07:25:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.286+0000 7fa6374b5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6300cf3c0 con 0x7fa63015df00 2024-03-06T07:25:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.286+0000 7fa634a50640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6300b95a0 0x7fa6300b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58194/0 (socket says 172.21.15.27:58194) 2024-03-06T07:25:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.286+0000 7fa634a50640 1 -- 172.21.15.27:0/3375997207 learned_addr learned my addr 172.21.15.27:0/3375997207 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.286+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032043525 0 0) 0x7fa6300cea30 con 0x7fa6300b95a0 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.286+0000 7fa635251640 1 --2- 172.21.15.27:0/3375997207 >> v2:172.21.15.69:3300/0 conn(0x7fa63015df00 0x7fa6300d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.286+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa61c003520 con 0x7fa6300b95a0 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157487905 0 0) 0x7fa61c003520 con 0x7fa6300b95a0 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 >> v2:172.21.15.69:3300/0 conn(0x7fa63015df00 msgr2=0x7fa6300d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 --2- 172.21.15.27:0/3375997207 >> v2:172.21.15.69:3300/0 conn(0x7fa63015df00 0x7fa6300d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 >> v1:172.21.15.69:6789/0 conn(0x7fa6300b9a30 legacy=0x7fa6300d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6300cf0e0 con 0x7fa6300b95a0 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fa62c001670 con 0x7fa6300b95a0 2024-03-06T07:25:55.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 >> v1:172.21.15.27:6789/0 conn(0x7fa6300b95a0 legacy=0x7fa6300b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c0039b0 0x7fa61c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa61c004800 0x7fa61c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa61c0051c0 0x7fa61c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300cf3c0 con 0x7fa61c004800 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c00c060 con 0x7fa61c0051c0 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c00c260 con 0x7fa61c0039b0 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa634a50640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa61c004800 0x7fa61c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa635251640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c0039b0 0x7fa61c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa635a52640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa61c0051c0 0x7fa61c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa634a50640 1 -- 172.21.15.27:0/3375997207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c0039b0 msgr2=0x7fa61c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa634a50640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c0039b0 0x7fa61c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa634a50640 1 -- 172.21.15.27:0/3375997207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa61c0051c0 msgr2=0x7fa61c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa634a50640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa61c0051c0 0x7fa61c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa634a50640 1 -- 172.21.15.27:0/3375997207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c003520 con 0x7fa61c004800 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa635a52640 1 -- 172.21.15.27:0/3375997207 reap_dead start 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.287+0000 7fa635251640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c0039b0 0x7fa61c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.288+0000 7fa634a50640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa61c004800 0x7fa61c004c80 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00c240 tx=0x7fa61c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad5643573f8c254 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.288+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c007cf0 con 0x7fa61c004800 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.288+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62c030070 con 0x7fa61c004800 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.288+0000 7fa627fff640 1 -- 172.21.15.27:0/3375997207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c005d70 con 0x7fa61c004800 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.289+0000 7fa6374b5640 1 -- 172.21.15.27:0/3375997207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa61c004800 msgr2=0x7fa61c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.289+0000 7fa6374b5640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa61c004800 0x7fa61c004c80 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00c240 tx=0x7fa61c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.289+0000 7fa6374b5640 1 -- 172.21.15.27:0/3375997207 shutdown_connections 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.289+0000 7fa6374b5640 1 --2- 172.21.15.27:0/3375997207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa61c004800 0x7fa61c004c80 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.289+0000 7fa6374b5640 1 -- 172.21.15.27:0/3375997207 >> 172.21.15.27:0/3375997207 conn(0x7fa630071960 msgr2=0x7fa630071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.289+0000 7fa6374b5640 1 -- 172.21.15.27:0/3375997207 shutdown_connections 2024-03-06T07:25:55.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.289+0000 7fa6374b5640 1 -- 172.21.15.27:0/3375997207 wait complete. 2024-03-06T07:25:55.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.291+0000 7fa6374b5640 1 Processor -- start 2024-03-06T07:25:55.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.291+0000 7fa6374b5640 1 -- start start 2024-03-06T07:25:55.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa6374b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c004800 0x7fa6300cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa6374b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6300ca300 0x7fa630158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa6374b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63015e2e0 0x7fa63015ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa6374b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300cec70 con 0x7fa63015e2e0 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa6374b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300ce750 con 0x7fa6300ca300 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa6374b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300cedf0 con 0x7fa61c004800 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa635a52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63015e2e0 0x7fa63015ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa635a52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63015e2e0 0x7fa63015ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48604/0 (socket says 172.21.15.27:48604) 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa635a52640 1 -- 172.21.15.27:0/76300158 learned_addr learned my addr 172.21.15.27:0/76300158 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa635251640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c004800 0x7fa6300cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.292+0000 7fa634a50640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6300ca300 0x7fa630158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa635a52640 1 -- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c004800 msgr2=0x7fa6300cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa635a52640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c004800 0x7fa6300cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa635a52640 1 -- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6300ca300 msgr2=0x7fa630158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa635a52640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6300ca300 0x7fa630158580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa635a52640 1 -- 172.21.15.27:0/76300158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa630161fd0 con 0x7fa63015e2e0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa634a50640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6300ca300 0x7fa630158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa635a52640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63015e2e0 0x7fa63015ed80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa6300b6b40 tx=0x7fa62800fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa625ffb640 1 -- 172.21.15.27:0/76300158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62801a070 con 0x7fa63015e2e0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa625ffb640 1 -- 172.21.15.27:0/76300158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa628006b50 con 0x7fa63015e2e0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa625ffb640 1 -- 172.21.15.27:0/76300158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628003970 con 0x7fa63015e2e0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6301622b0 con 0x7fa63015e2e0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.293+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6300c9e30 con 0x7fa63015e2e0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.294+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa630067ff0 con 0x7fa63015e2e0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.295+0000 7fa625ffb640 1 -- 172.21.15.27:0/76300158 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa62801d070 con 0x7fa63015e2e0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.295+0000 7fa625ffb640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa61406b4d0 0x7fa61406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.295+0000 7fa625ffb640 1 -- 172.21.15.27:0/76300158 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6280895b0 con 0x7fa63015e2e0 2024-03-06T07:25:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.295+0000 7fa635251640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa61406b4d0 0x7fa61406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.296+0000 7fa635251640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa61406b4d0 0x7fa61406d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa630069d20 tx=0x7fa620009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.300+0000 7fa625ffb640 1 -- 172.21.15.27:0/76300158 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa628057de0 con 0x7fa63015e2e0 2024-03-06T07:25:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.508+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6300b2d20 con 0x7fa63015e2e0 2024-03-06T07:25:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.509+0000 7fa625ffb640 1 -- 172.21.15.27:0/76300158 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fa62805be80 con 0x7fa63015e2e0 2024-03-06T07:25:55.511 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:55.511 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:25.095713+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:21.134461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:22.434887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa61406b4d0 msgr2=0x7fa61406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa61406b4d0 0x7fa61406d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa630069d20 tx=0x7fa620009190 comp rx=0 tx=0).stop 2024-03-06T07:25:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63015e2e0 msgr2=0x7fa63015ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63015e2e0 0x7fa63015ed80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa6300b6b40 tx=0x7fa62800fb40 comp rx=0 tx=0).stop 2024-03-06T07:25:55.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 shutdown_connections 2024-03-06T07:25:55.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa61406b4d0 0x7fa61406d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63015e2e0 0x7fa63015ed80 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6300ca300 0x7fa630158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 --2- 172.21.15.27:0/76300158 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa61c004800 0x7fa6300cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 >> 172.21.15.27:0/76300158 conn(0x7fa630071960 msgr2=0x7fa63015d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 shutdown_connections 2024-03-06T07:25:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.511+0000 7fa6374b5640 1 -- 172.21.15.27:0/76300158 wait complete. 2024-03-06T07:25:55.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:21.134461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:22.434887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:55.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.632+0000 7f7c7b8b8640 1 Processor -- start 2024-03-06T07:25:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.632+0000 7f7c7b8b8640 1 -- start start 2024-03-06T07:25:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c7b8b8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7c740ceea0 0x7f7c740d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c7b8b8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 0x7f7c740b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c7b8b8640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c740d3740 con 0x7f7c7415dd90 2024-03-06T07:25:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c7b8b8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c740d3400 con 0x7f7c740ceea0 2024-03-06T07:25:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c7b8b8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740d3120 con 0x7f7c740b6d10 2024-03-06T07:25:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c79654640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7c7415dd90 0x7f7c740d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58196/0 (socket says 172.21.15.27:58196) 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c79654640 1 -- 172.21.15.27:0/933069946 learned_addr learned my addr 172.21.15.27:0/933069946 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c78e53640 1 --2- 172.21.15.27:0/933069946 >> v2:172.21.15.69:3300/0 conn(0x7f7c740ceea0 0x7f7c740d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c79e55640 1 --2- 172.21.15.27:0/933069946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 0x7f7c740b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c6bfff640 1 -- 172.21.15.27:0/933069946 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672233824 0 0) 0x7f7c740d3740 con 0x7f7c7415dd90 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.633+0000 7f7c6bfff640 1 -- 172.21.15.27:0/933069946 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c60003520 con 0x7f7c7415dd90 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.634+0000 7f7c78e53640 1 -- 172.21.15.27:0/933069946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 msgr2=0x7f7c740b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.634+0000 7f7c78e53640 1 --2- 172.21.15.27:0/933069946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 0x7f7c740b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.634+0000 7f7c78e53640 1 -- 172.21.15.27:0/933069946 >> v1:172.21.15.27:6789/0 conn(0x7f7c7415dd90 legacy=0x7f7c740d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.634+0000 7f7c6bfff640 1 -- 172.21.15.27:0/933069946 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1320078373 0 0) 0x7f7c60003520 con 0x7f7c7415dd90 2024-03-06T07:25:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.634+0000 7f7c78e53640 1 -- 172.21.15.27:0/933069946 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c740d39e0 con 0x7f7c740ceea0 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c78e53640 1 --2- 172.21.15.27:0/933069946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c740ceea0 0x7f7c740d29d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7c700097a0 tx=0x7f7c7002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53fd84f08250dbc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c6bfff640 1 -- 172.21.15.27:0/933069946 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70038070 con 0x7f7c740ceea0 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c6bfff640 1 -- 172.21.15.27:0/933069946 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c70031500 con 0x7f7c740ceea0 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c6bfff640 1 -- 172.21.15.27:0/933069946 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64001690 con 0x7f7c740ceea0 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/933069946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c740ceea0 msgr2=0x7f7c740d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c7b8b8640 1 --2- 172.21.15.27:0/933069946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c740ceea0 0x7f7c740d29d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7c700097a0 tx=0x7f7c7002b820 comp rx=0 tx=0).stop 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/933069946 shutdown_connections 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c7b8b8640 1 --2- 172.21.15.27:0/933069946 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 0x7f7c740b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c7b8b8640 1 --2- 172.21.15.27:0/933069946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c740ceea0 0x7f7c740d29d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/933069946 >> 172.21.15.27:0/933069946 conn(0x7f7c74071960 msgr2=0x7f7c74071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/933069946 shutdown_connections 2024-03-06T07:25:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.635+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/933069946 wait complete. 2024-03-06T07:25:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c7b8b8640 1 Processor -- start 2024-03-06T07:25:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c7b8b8640 1 -- start start 2024-03-06T07:25:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c7b8b8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 0x7f7c740cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c7b8b8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c740ceea0 0x7f7c740b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c7b8b8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c7415dd90 0x7f7c7414cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c7b8b8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740d48f0 con 0x7f7c7415dd90 2024-03-06T07:25:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c7b8b8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7415e170 con 0x7f7c740ceea0 2024-03-06T07:25:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c7b8b8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740d4a70 con 0x7f7c740b6d10 2024-03-06T07:25:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c79e55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c7415dd90 0x7f7c7414cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c79e55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c7415dd90 0x7f7c7414cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48606/0 (socket says 172.21.15.27:48606) 2024-03-06T07:25:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c79e55640 1 -- 172.21.15.27:0/3482743958 learned_addr learned my addr 172.21.15.27:0/3482743958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c79654640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 0x7f7c740cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.637+0000 7f7c78e53640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c740ceea0 0x7f7c740b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c79e55640 1 -- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 msgr2=0x7f7c740cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c79e55640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 0x7f7c740cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c79e55640 1 -- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c740ceea0 msgr2=0x7f7c740b8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c79e55640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c740ceea0 0x7f7c740b8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c79e55640 1 -- 172.21.15.27:0/3482743958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c740cddb0 con 0x7f7c7415dd90 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c79e55640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c7415dd90 0x7f7c7414cb30 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00efc0 tx=0x7f7c6c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c69ffb640 1 -- 172.21.15.27:0/3482743958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c018020 con 0x7f7c7415dd90 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c69ffb640 1 -- 172.21.15.27:0/3482743958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c014480 con 0x7f7c7415dd90 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c740ccd10 con 0x7f7c7415dd90 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c69ffb640 1 -- 172.21.15.27:0/3482743958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c014870 con 0x7f7c7415dd90 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.638+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c740cd2d0 con 0x7f7c7415dd90 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.640+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c74072a30 con 0x7f7c7415dd90 2024-03-06T07:25:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.640+0000 7f7c69ffb640 1 -- 172.21.15.27:0/3482743958 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c6c005d10 con 0x7f7c7415dd90 2024-03-06T07:25:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.640+0000 7f7c69ffb640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c5806b4d0 0x7f7c5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.640+0000 7f7c69ffb640 1 -- 172.21.15.27:0/3482743958 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c6c08ab10 con 0x7f7c7415dd90 2024-03-06T07:25:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.641+0000 7f7c79654640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c5806b4d0 0x7f7c5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.641+0000 7f7c79654640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c5806b4d0 0x7f7c5806d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7c64001eb0 tx=0x7f7c64001e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.645+0000 7f7c69ffb640 1 -- 172.21.15.27:0/3482743958 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c6c022070 con 0x7f7c7415dd90 2024-03-06T07:25:55.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.824+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c7404f410 con 0x7f7c7415dd90 2024-03-06T07:25:55.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.825+0000 7f7c69ffb640 1 -- 172.21.15.27:0/3482743958 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c6c010040 con 0x7f7c7415dd90 2024-03-06T07:25:55.826 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:25:55.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c5806b4d0 msgr2=0x7f7c5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c5806b4d0 0x7f7c5806d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7c64001eb0 tx=0x7f7c64001e40 comp rx=0 tx=0).stop 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c7415dd90 msgr2=0x7f7c7414cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c7415dd90 0x7f7c7414cb30 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00efc0 tx=0x7f7c6c00c490 comp rx=0 tx=0).stop 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 shutdown_connections 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c5806b4d0 0x7f7c5806d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c7415dd90 0x7f7c7414cb30 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c740ceea0 0x7f7c740b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 --2- 172.21.15.27:0/3482743958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c740b6d10 0x7f7c740cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 >> 172.21.15.27:0/3482743958 conn(0x7f7c74071960 msgr2=0x7f7c740b71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 shutdown_connections 2024-03-06T07:25:55.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.827+0000 7f7c7b8b8640 1 -- 172.21.15.27:0/3482743958 wait complete. 2024-03-06T07:25:55.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:55.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.961+0000 7efccf6a4640 1 Processor -- start 2024-03-06T07:25:55.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.962+0000 7efccf6a4640 1 -- start start 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.962+0000 7efccf6a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efcc815b560 0x7efcc80715f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.962+0000 7efccf6a4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80b2e30 con 0x7efcc80708e0 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.962+0000 7efccf6a4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80b30b0 con 0x7efcc8071160 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.962+0000 7efccf6a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc80ccad0 con 0x7efcc815b560 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efccd440640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efcc80708e0 0x7efcc80cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58200/0 (socket says 172.21.15.27:58200) 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efccd440640 1 -- 172.21.15.27:0/126283124 learned_addr learned my addr 172.21.15.27:0/126283124 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132983057 0 0) 0x7efcb8001110 con 0x7efcc80708e0 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb4003520 con 0x7efcc80708e0 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efccdc41640 1 --2- 172.21.15.27:0/126283124 >> v2:172.21.15.69:3300/0 conn(0x7efcc815b560 0x7efcc80715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 609183231 0 0) 0x7efcb4003520 con 0x7efcc80708e0 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 >> v2:172.21.15.69:3300/0 conn(0x7efcc815b560 msgr2=0x7efcc80715f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 --2- 172.21.15.27:0/126283124 >> v2:172.21.15.69:3300/0 conn(0x7efcc815b560 0x7efcc80715f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 >> v1:172.21.15.69:6789/0 conn(0x7efcc8071160 legacy=0x7efcc815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc80b3390 con 0x7efcc80708e0 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7efcb8001840 con 0x7efcc80708e0 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 >> v1:172.21.15.27:6789/0 conn(0x7efcc80708e0 legacy=0x7efcc80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb40039b0 0x7efcb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb4004800 0x7efcb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcb40051c0 0x7efcb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80ccad0 con 0x7efcb40039b0 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb400c160 con 0x7efcb4004800 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb400c360 con 0x7efcb40051c0 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.963+0000 7efccd440640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb40039b0 0x7efcb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efcccc3f640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb4004800 0x7efcb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efccdc41640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcb40051c0 0x7efcb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efccd440640 1 -- 172.21.15.27:0/126283124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcb40051c0 msgr2=0x7efcb400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efccd440640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcb40051c0 0x7efcb400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efccd440640 1 -- 172.21.15.27:0/126283124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb4004800 msgr2=0x7efcb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efccd440640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb4004800 0x7efcb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efccd440640 1 -- 172.21.15.27:0/126283124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb4003520 con 0x7efcb40039b0 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efccdc41640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcb40051c0 0x7efcb400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efccdc41640 1 -- 172.21.15.27:0/126283124 reap_dead start 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.964+0000 7efccd440640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb40039b0 0x7efcb4003dd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efcb800c4c0 tx=0x7efcb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34961c5d83e8c4c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb800c0e0 con 0x7efcb40039b0 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb8030070 con 0x7efcb40039b0 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efcbffff640 1 -- 172.21.15.27:0/126283124 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb80058d0 con 0x7efcb40039b0 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efccf6a4640 1 -- 172.21.15.27:0/126283124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb40039b0 msgr2=0x7efcb4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efccf6a4640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb40039b0 0x7efcb4003dd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efcb800c4c0 tx=0x7efcb4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efccf6a4640 1 -- 172.21.15.27:0/126283124 shutdown_connections 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efccf6a4640 1 --2- 172.21.15.27:0/126283124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb40039b0 0x7efcb4003dd0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efccf6a4640 1 -- 172.21.15.27:0/126283124 >> 172.21.15.27:0/126283124 conn(0x7efcc8066d00 msgr2=0x7efcc8067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efccf6a4640 1 -- 172.21.15.27:0/126283124 shutdown_connections 2024-03-06T07:25:55.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.965+0000 7efccf6a4640 1 -- 172.21.15.27:0/126283124 wait complete. 2024-03-06T07:25:55.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.966+0000 7efccf6a4640 1 Processor -- start 2024-03-06T07:25:55.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.966+0000 7efccf6a4640 1 -- start start 2024-03-06T07:25:55.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccf6a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb40039b0 0x7efcc813e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccf6a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc813c3c0 0x7efcc80aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccf6a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc8133c80 0x7efcc8134100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccf6a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80b4150 con 0x7efcc8133c80 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccf6a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80b3950 con 0x7efcb40039b0 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccf6a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc8070cc0 con 0x7efcc813c3c0 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccdc41640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc8133c80 0x7efcc8134100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccdc41640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc8133c80 0x7efcc8134100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48626/0 (socket says 172.21.15.27:48626) 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccdc41640 1 -- 172.21.15.27:0/1669853299 learned_addr learned my addr 172.21.15.27:0/1669853299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efcccc3f640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc813c3c0 0x7efcc80aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.967+0000 7efccd440640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb40039b0 0x7efcc813e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efccdc41640 1 -- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc813c3c0 msgr2=0x7efcc80aff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efccdc41640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc813c3c0 0x7efcc80aff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efccdc41640 1 -- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb40039b0 msgr2=0x7efcc813e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efccdc41640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb40039b0 0x7efcc813e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efccdc41640 1 -- 172.21.15.27:0/1669853299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc80b08b0 con 0x7efcc8133c80 2024-03-06T07:25:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efccd440640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb40039b0 0x7efcc813e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efccdc41640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc8133c80 0x7efcc8134100 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7efcc80cf3c0 tx=0x7efcc0010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efcbdffb640 1 -- 172.21.15.27:0/1669853299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0010da0 con 0x7efcc8133c80 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efcbdffb640 1 -- 172.21.15.27:0/1669853299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc0003990 con 0x7efcc8133c80 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc80b0b90 con 0x7efcc8133c80 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efcbdffb640 1 -- 172.21.15.27:0/1669853299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0003dc0 con 0x7efcc8133c80 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.968+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc80b1150 con 0x7efcc8133c80 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.969+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc80733c0 con 0x7efcc8133c80 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.970+0000 7efcbdffb640 1 -- 172.21.15.27:0/1669853299 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcc001e020 con 0x7efcc8133c80 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.970+0000 7efcbdffb640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcac06b4d0 0x7efcac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.970+0000 7efcbdffb640 1 -- 172.21.15.27:0/1669853299 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcc008a7a0 con 0x7efcc8133c80 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.971+0000 7efccd440640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcac06b4d0 0x7efcac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.971+0000 7efccd440640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcac06b4d0 0x7efcac06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efcb8034000 tx=0x7efcb8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:55.975+0000 7efcbdffb640 1 -- 172.21.15.27:0/1669853299 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcc0059620 con 0x7efcc8133c80 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.160+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efcc80b4d60 con 0x7efcc8133c80 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.161+0000 7efcbdffb640 1 -- 172.21.15.27:0/1669853299 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7efcc0058f20 con 0x7efcc8133c80 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcac06b4d0 msgr2=0x7efcac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcac06b4d0 0x7efcac06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efcb8034000 tx=0x7efcb8009030 comp rx=0 tx=0).stop 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc8133c80 msgr2=0x7efcc8134100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc8133c80 0x7efcc8134100 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7efcc80cf3c0 tx=0x7efcc0010590 comp rx=0 tx=0).stop 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 shutdown_connections 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcac06b4d0 0x7efcac06d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcc8133c80 0x7efcc8134100 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcc813c3c0 0x7efcc80aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 --2- 172.21.15.27:0/1669853299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcb40039b0 0x7efcc813e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 >> 172.21.15.27:0/1669853299 conn(0x7efcc8066d00 msgr2=0x7efcc814c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 shutdown_connections 2024-03-06T07:25:56.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.163+0000 7efccf6a4640 1 -- 172.21.15.27:0/1669853299 wait complete. 2024-03-06T07:25:56.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.283+0000 7f3fc3aaa640 1 Processor -- start 2024-03-06T07:25:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.283+0000 7f3fc3aaa640 1 -- start start 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc3aaa640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3fbc0b2d30 0x7f3fbc0b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc3aaa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fbc0b7820 0x7f3fbc0b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc3aaa640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0d4960 con 0x7f3fbc071540 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc3aaa640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc0d3230 con 0x7f3fbc0b7820 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc3aaa640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc0d36c0 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc2047640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fbc0b7820 0x7f3fbc0b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc2047640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fbc0b7820 0x7f3fbc0b7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48638/0 (socket says 172.21.15.27:48638) 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc2047640 1 -- 172.21.15.27:0/1247720538 learned_addr learned my addr 172.21.15.27:0/1247720538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc1045640 1 --2- 172.21.15.27:0/1247720538 >> v2:172.21.15.69:3300/0 conn(0x7f3fbc0b2d30 0x7f3fbc0b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc2047640 1 -- 172.21.15.27:0/1247720538 >> v2:172.21.15.69:3300/0 conn(0x7f3fbc0b2d30 msgr2=0x7f3fbc0b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc2047640 1 --2- 172.21.15.27:0/1247720538 >> v2:172.21.15.69:3300/0 conn(0x7f3fbc0b2d30 0x7f3fbc0b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.284+0000 7f3fc2047640 1 -- 172.21.15.27:0/1247720538 >> v1:172.21.15.69:6789/0 conn(0x7f3fbc071540 legacy=0x7f3fbc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.285+0000 7f3fc2047640 1 -- 172.21.15.27:0/1247720538 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0d2620 con 0x7f3fbc0b7820 2024-03-06T07:25:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.285+0000 7f3fc2047640 1 --2- 172.21.15.27:0/1247720538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b7820 0x7f3fbc0b7cd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0c7690 tx=0x7f3fb402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e087e36e70bf78a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.285+0000 7f3fc0844640 1 -- 172.21.15.27:0/1247720538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb402ca80 con 0x7f3fbc0b7820 2024-03-06T07:25:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc0844640 1 -- 172.21.15.27:0/1247720538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb402cc20 con 0x7f3fbc0b7820 2024-03-06T07:25:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc0844640 1 -- 172.21.15.27:0/1247720538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb40343f0 con 0x7f3fbc0b7820 2024-03-06T07:25:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/1247720538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b7820 msgr2=0x7f3fbc0b7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc3aaa640 1 --2- 172.21.15.27:0/1247720538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b7820 0x7f3fbc0b7cd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0c7690 tx=0x7f3fb402bee0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/1247720538 shutdown_connections 2024-03-06T07:25:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc3aaa640 1 --2- 172.21.15.27:0/1247720538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b7820 0x7f3fbc0b7cd0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc3aaa640 1 --2- 172.21.15.27:0/1247720538 >> v2:172.21.15.69:3300/0 conn(0x7f3fbc0b2d30 0x7f3fbc0b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/1247720538 >> 172.21.15.27:0/1247720538 conn(0x7f3fbc0d3d60 msgr2=0x7f3fbc0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/1247720538 shutdown_connections 2024-03-06T07:25:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.286+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/1247720538 wait complete. 2024-03-06T07:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.287+0000 7f3fc3aaa640 1 Processor -- start 2024-03-06T07:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.287+0000 7f3fc3aaa640 1 -- start start 2024-03-06T07:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc3aaa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc071540 0x7f3fbc1f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc3aaa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc3aaa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc0b7820 0x7f3fbc0c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc3aaa640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0d2ec0 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc3aaa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0d2be0 con 0x7f3fbc071540 2024-03-06T07:25:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc3aaa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0d22f0 con 0x7f3fbc0b7820 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc1045640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc1045640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0aa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48654/0 (socket says 172.21.15.27:48654) 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc1846640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc071540 0x7f3fbc1f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc1045640 1 -- 172.21.15.27:0/3845650084 learned_addr learned my addr 172.21.15.27:0/3845650084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc2047640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc0b7820 0x7f3fbc0c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc1045640 1 -- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc0b7820 msgr2=0x7f3fbc0c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc1045640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc0b7820 0x7f3fbc0c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc1045640 1 -- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc071540 msgr2=0x7f3fbc1f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc1045640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc071540 0x7f3fbc1f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.288+0000 7f3fc1045640 1 -- 172.21.15.27:0/3845650084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0ab120 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.289+0000 7f3fc1045640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0aa6d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c820 tx=0x7f3fb800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.289+0000 7f3fae7fc640 1 -- 172.21.15.27:0/3845650084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8015070 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.289+0000 7f3fae7fc640 1 -- 172.21.15.27:0/3845650084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb8004220 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.289+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc0ab400 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.289+0000 7f3fae7fc640 1 -- 172.21.15.27:0/3845650084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb800ce10 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.289+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc0b0c60 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.290+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc0b58e0 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.291+0000 7f3fae7fc640 1 -- 172.21.15.27:0/3845650084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fb8007830 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.291+0000 7f3fae7fc640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa006b4d0 0x7f3fa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.291+0000 7f3fae7fc640 1 -- 172.21.15.27:0/3845650084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fb808a080 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.292+0000 7f3fc1846640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa006b4d0 0x7f3fa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.292+0000 7f3fc1846640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa006b4d0 0x7f3fa006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb0009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.296+0000 7f3fae7fc640 1 -- 172.21.15.27:0/3845650084 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fbc0b58e0 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.503+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3fbc072070 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.503+0000 7f3fae7fc640 1 -- 172.21.15.27:0/3845650084 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3fb8058830 con 0x7f3fbc0b2d30 2024-03-06T07:25:56.505 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa006b4d0 msgr2=0x7f3fa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa006b4d0 0x7f3fa006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb0009060 comp rx=0 tx=0).stop 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b2d30 msgr2=0x7f3fbc0aa6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0aa6d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c820 tx=0x7f3fb800cce0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 shutdown_connections 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3fa006b4d0 0x7f3fa006d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fbc0b7820 0x7f3fbc0c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0b2d30 0x7f3fbc0aa6d0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 --2- 172.21.15.27:0/3845650084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fbc071540 0x7f3fbc1f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 >> 172.21.15.27:0/3845650084 conn(0x7f3fbc0d3d60 msgr2=0x7f3fbc0b4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 shutdown_connections 2024-03-06T07:25:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.505+0000 7f3fc3aaa640 1 -- 172.21.15.27:0/3845650084 wait complete. 2024-03-06T07:25:56.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.623+0000 7f17ef9f7640 1 Processor -- start 2024-03-06T07:25:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.623+0000 7f17ef9f7640 1 -- start start 2024-03-06T07:25:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ef9f7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17e80b0e90 0x7f17e806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ef9f7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e80cee90 con 0x7f17e806b5c0 2024-03-06T07:25:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ef9f7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e80cfc50 con 0x7f17e80ae9b0 2024-03-06T07:25:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ef9f7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e80cfa00 con 0x7f17e80b0e90 2024-03-06T07:25:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ecf92640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17e80b0e90 0x7f17e806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ed793640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f17e806b5c0 0x7f17e80b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58210/0 (socket says 172.21.15.27:58210) 2024-03-06T07:25:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ed793640 1 -- 172.21.15.27:0/3310270066 learned_addr learned my addr 172.21.15.27:0/3310270066 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ecf92640 1 -- 172.21.15.27:0/3310270066 >> v1:172.21.15.69:6789/0 conn(0x7f17e80ae9b0 legacy=0x7f17e80d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ecf92640 1 -- 172.21.15.27:0/3310270066 >> v1:172.21.15.27:6789/0 conn(0x7f17e806b5c0 legacy=0x7f17e80b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.624+0000 7f17ecf92640 1 -- 172.21.15.27:0/3310270066 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80cff30 con 0x7f17e80b0e90 2024-03-06T07:25:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.625+0000 7f17ecf92640 1 --2- 172.21.15.27:0/3310270066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80b0e90 0x7f17e806b9a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0097a0 tx=0x7f17dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477609278ecf2bd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.625+0000 7f17dbfff640 1 -- 172.21.15.27:0/3310270066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc035020 con 0x7f17e80b0e90 2024-03-06T07:25:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.625+0000 7f17dbfff640 1 -- 172.21.15.27:0/3310270066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc031830 con 0x7f17e80b0e90 2024-03-06T07:25:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.625+0000 7f17dbfff640 1 -- 172.21.15.27:0/3310270066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc031c00 con 0x7f17e80b0e90 2024-03-06T07:25:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.626+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3310270066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80b0e90 msgr2=0x7f17e806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.626+0000 7f17ef9f7640 1 --2- 172.21.15.27:0/3310270066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80b0e90 0x7f17e806b9a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0097a0 tx=0x7f17dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.626+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3310270066 shutdown_connections 2024-03-06T07:25:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.626+0000 7f17ef9f7640 1 --2- 172.21.15.27:0/3310270066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80b0e90 0x7f17e806b9a0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.626+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3310270066 >> 172.21.15.27:0/3310270066 conn(0x7f17e80b36d0 msgr2=0x7f17e80b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.626+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3310270066 shutdown_connections 2024-03-06T07:25:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.626+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3310270066 wait complete. 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.627+0000 7f17ef9f7640 1 Processor -- start 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.627+0000 7f17ef9f7640 1 -- start start 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ef9f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e806b5c0 0x7f17e80b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ef9f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80ae9b0 0x7f17e80b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ef9f7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80b0e90 0x7f17e8157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ef9f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e806c910 con 0x7f17e80ae9b0 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ef9f7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e806c790 con 0x7f17e80b0e90 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ef9f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80d0ab0 con 0x7f17e806b5c0 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ecf92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80ae9b0 0x7f17e80b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ecf92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80ae9b0 0x7f17e80b4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48664/0 (socket says 172.21.15.27:48664) 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ecf92640 1 -- 172.21.15.27:0/3233440234 learned_addr learned my addr 172.21.15.27:0/3233440234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ed793640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e806b5c0 0x7f17e80b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17edf94640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80b0e90 0x7f17e8157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ecf92640 1 -- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e806b5c0 msgr2=0x7f17e80b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ecf92640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e806b5c0 0x7f17e80b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ecf92640 1 -- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80b0e90 msgr2=0x7f17e8157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ecf92640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80b0e90 0x7f17e8157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ecf92640 1 -- 172.21.15.27:0/3233440234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80b79d0 con 0x7f17e80ae9b0 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ed793640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e806b5c0 0x7f17e80b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:56.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17edf94640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80b0e90 0x7f17e8157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:56.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.628+0000 7f17ecf92640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80ae9b0 0x7f17e80b4930 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f17dc02bee0 tx=0x7f17dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.629+0000 7f17d9ffb640 1 -- 172.21.15.27:0/3233440234 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc041070 con 0x7f17e80ae9b0 2024-03-06T07:25:56.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.629+0000 7f17d9ffb640 1 -- 172.21.15.27:0/3233440234 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc03a950 con 0x7f17e80ae9b0 2024-03-06T07:25:56.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.629+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e80b7130 con 0x7f17e80ae9b0 2024-03-06T07:25:56.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.629+0000 7f17d9ffb640 1 -- 172.21.15.27:0/3233440234 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc035040 con 0x7f17e80ae9b0 2024-03-06T07:25:56.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.629+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e80b76f0 con 0x7f17e80ae9b0 2024-03-06T07:25:56.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.630+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e8068700 con 0x7f17e80ae9b0 2024-03-06T07:25:56.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.630+0000 7f17d9ffb640 1 -- 172.21.15.27:0/3233440234 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17dc04c060 con 0x7f17e80ae9b0 2024-03-06T07:25:56.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.631+0000 7f17d9ffb640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f17cc06b4d0 0x7f17cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.631+0000 7f17d9ffb640 1 -- 172.21.15.27:0/3233440234 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17dc0ac060 con 0x7f17e80ae9b0 2024-03-06T07:25:56.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.631+0000 7f17ed793640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f17cc06b4d0 0x7f17cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.632+0000 7f17ed793640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f17cc06b4d0 0x7f17cc06d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f17e4007fd0 tx=0x7f17e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.639+0000 7f17d9ffb640 1 -- 172.21.15.27:0/3233440234 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17dc07afc0 con 0x7f17e80ae9b0 2024-03-06T07:25:56.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.832+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f17e806daf0 con 0x7f17e80ae9b0 2024-03-06T07:25:56.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.833+0000 7f17d9ffb640 1 -- 172.21.15.27:0/3233440234 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7f17dc07a8c0 con 0x7f17e80ae9b0 2024-03-06T07:25:56.834 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:25:56.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f17cc06b4d0 msgr2=0x7f17cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f17cc06b4d0 0x7f17cc06d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f17e4007fd0 tx=0x7f17e4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80ae9b0 msgr2=0x7f17e80b4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80ae9b0 0x7f17e80b4930 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f17dc02bee0 tx=0x7f17dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 shutdown_connections 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f17cc06b4d0 0x7f17cc06d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17e80b0e90 0x7f17e8157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e80ae9b0 0x7f17e80b4930 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 --2- 172.21.15.27:0/3233440234 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17e806b5c0 0x7f17e80b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 >> 172.21.15.27:0/3233440234 conn(0x7f17e80b36d0 msgr2=0x7f17e80cdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 shutdown_connections 2024-03-06T07:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:56.836+0000 7f17ef9f7640 1 -- 172.21.15.27:0/3233440234 wait complete. 2024-03-06T07:25:56.853 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:25:56.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:56.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:56.882 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[59932]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:56.884 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:25:56.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:56.884 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:57.050 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:25:57.050 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:57.067 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:57.067 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:25:57.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.067 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:25:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:57.124 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:25:57.124 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:25:57.124 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:57.139 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:57.139 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:25:57.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.139 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:25:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:57.195 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:25:57.195 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:25:57.195 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:57.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:57.212 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:25:57.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.212 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:25:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:25:57.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.373+0000 7fb67730c640 1 Processor -- start 2024-03-06T07:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.373+0000 7fb67730c640 1 -- start start 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.374+0000 7fb67730c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700b9a10 0x7fb6700d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.374+0000 7fb67730c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6700cea30 con 0x7fb6700b95a0 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.374+0000 7fb67730c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6700cf3c0 con 0x7fb67015df00 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.374+0000 7fb67730c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700cf0e0 con 0x7fb6700b9a10 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.374+0000 7fb6748a7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb6700b95a0 0x7fb6700b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58222/0 (socket says 172.21.15.27:58222) 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.374+0000 7fb6748a7640 1 -- 172.21.15.27:0/4115150984 learned_addr learned my addr 172.21.15.27:0/4115150984 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.374+0000 7fb6758a9640 1 --2- 172.21.15.27:0/4115150984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700b9a10 0x7fb6700d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.374+0000 7fb667fff640 1 -- 172.21.15.27:0/4115150984 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887752961 0 0) 0x7fb6700cea30 con 0x7fb6700b95a0 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.375+0000 7fb667fff640 1 -- 172.21.15.27:0/4115150984 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb65c003520 con 0x7fb6700b95a0 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.375+0000 7fb667fff640 1 -- 172.21.15.27:0/4115150984 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754915151 0 0) 0x7fb6700cf3c0 con 0x7fb67015df00 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.375+0000 7fb667fff640 1 -- 172.21.15.27:0/4115150984 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6700cea30 con 0x7fb67015df00 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.375+0000 7fb6758a9640 1 -- 172.21.15.27:0/4115150984 >> v1:172.21.15.69:6789/0 conn(0x7fb67015df00 legacy=0x7fb6700d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.375+0000 7fb6758a9640 1 -- 172.21.15.27:0/4115150984 >> v1:172.21.15.27:6789/0 conn(0x7fb6700b95a0 legacy=0x7fb6700b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.375+0000 7fb6758a9640 1 -- 172.21.15.27:0/4115150984 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6700ce470 con 0x7fb6700b9a10 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.376+0000 7fb6758a9640 1 --2- 172.21.15.27:0/4115150984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700b9a10 0x7fb6700d3220 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb668009dc0 tx=0x7fb66802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bd3b205df4bec86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.376+0000 7fb667fff640 1 -- 172.21.15.27:0/4115150984 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668039070 con 0x7fb6700b9a10 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.376+0000 7fb667fff640 1 -- 172.21.15.27:0/4115150984 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66802ccd0 con 0x7fb6700b9a10 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.376+0000 7fb667fff640 1 -- 172.21.15.27:0/4115150984 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668034640 con 0x7fb6700b9a10 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.376+0000 7fb67730c640 1 -- 172.21.15.27:0/4115150984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700b9a10 msgr2=0x7fb6700d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.376+0000 7fb67730c640 1 --2- 172.21.15.27:0/4115150984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700b9a10 0x7fb6700d3220 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb668009dc0 tx=0x7fb66802bf20 comp rx=0 tx=0).stop 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.376+0000 7fb67730c640 1 -- 172.21.15.27:0/4115150984 shutdown_connections 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.376+0000 7fb67730c640 1 --2- 172.21.15.27:0/4115150984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6700b9a10 0x7fb6700d3220 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.376+0000 7fb67730c640 1 -- 172.21.15.27:0/4115150984 >> 172.21.15.27:0/4115150984 conn(0x7fb670071960 msgr2=0x7fb670071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.377+0000 7fb67730c640 1 -- 172.21.15.27:0/4115150984 shutdown_connections 2024-03-06T07:25:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.377+0000 7fb67730c640 1 -- 172.21.15.27:0/4115150984 wait complete. 2024-03-06T07:25:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.378+0000 7fb67730c640 1 Processor -- start 2024-03-06T07:25:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.378+0000 7fb67730c640 1 -- start start 2024-03-06T07:25:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.378+0000 7fb67730c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6700b95a0 0x7fb67015f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.378+0000 7fb67730c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6700b9a10 0x7fb6701553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.378+0000 7fb67730c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67015df00 0x7fb6700b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.378+0000 7fb67730c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700cedf0 con 0x7fb6700b9a10 2024-03-06T07:25:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.378+0000 7fb67730c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700cec70 con 0x7fb6700b95a0 2024-03-06T07:25:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.378+0000 7fb67730c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700b5e10 con 0x7fb67015df00 2024-03-06T07:25:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6748a7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6700b9a10 0x7fb6701553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6748a7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6700b9a10 0x7fb6701553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48676/0 (socket says 172.21.15.27:48676) 2024-03-06T07:25:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6748a7640 1 -- 172.21.15.27:0/1792868570 learned_addr learned my addr 172.21.15.27:0/1792868570 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6758a9640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67015df00 0x7fb6700b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6750a8640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6700b95a0 0x7fb67015f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6748a7640 1 -- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67015df00 msgr2=0x7fb6700b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6748a7640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67015df00 0x7fb6700b7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6748a7640 1 -- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6700b95a0 msgr2=0x7fb67015f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6748a7640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6700b95a0 0x7fb67015f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6748a7640 1 -- 172.21.15.27:0/1792868570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6701f3c00 con 0x7fb6700b9a10 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6758a9640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67015df00 0x7fb6700b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb6748a7640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6700b9a10 0x7fb6701553c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00c500 tx=0x7fb66c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb665ffb640 1 -- 172.21.15.27:0/1792868570 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c017070 con 0x7fb6700b9a10 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb665ffb640 1 -- 172.21.15.27:0/1792868570 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c005ce0 con 0x7fb6700b9a10 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6701f3ee0 con 0x7fb6700b9a10 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.379+0000 7fb665ffb640 1 -- 172.21.15.27:0/1792868570 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c0072f0 con 0x7fb6700b9a10 2024-03-06T07:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.380+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6701f44a0 con 0x7fb6700b9a10 2024-03-06T07:25:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.380+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb670067ff0 con 0x7fb6700b9a10 2024-03-06T07:25:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.381+0000 7fb665ffb640 1 -- 172.21.15.27:0/1792868570 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb66c01a070 con 0x7fb6700b9a10 2024-03-06T07:25:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.381+0000 7fb665ffb640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb65406b4d0 0x7fb65406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.382+0000 7fb665ffb640 1 -- 172.21.15.27:0/1792868570 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb66c089490 con 0x7fb6700b9a10 2024-03-06T07:25:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.382+0000 7fb6750a8640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb65406b4d0 0x7fb65406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.382+0000 7fb6750a8640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb65406b4d0 0x7fb65406d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb6700d4150 tx=0x7fb660009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.386+0000 7fb665ffb640 1 -- 172.21.15.27:0/1792868570 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb66c057d50 con 0x7fb6700b9a10 2024-03-06T07:25:57.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.697+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb670071de0 con 0x7fb6700b9a10 2024-03-06T07:25:57.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.697+0000 7fb665ffb640 1 -- 172.21.15.27:0/1792868570 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fb66c05bdf0 con 0x7fb6700b9a10 2024-03-06T07:25:57.699 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:57.699 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5324,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3885016831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3885016831},{"type":"v1","addr":"172.21.15.69:6836","nonce":3885016831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5327,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3510003111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3510003111},{"type":"v1","addr":"172.21.15.69:6837","nonce":3510003111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5692,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3053251134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3053251134},{"type":"v1","addr":"172.21.15.27:6835","nonce":3053251134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:23.445711+0000","modified":"2024-03-06T07:25:26.468573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6129":{"gid":6129,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6833/60877422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":60877422},{"type":"v1","addr":"172.21.15.27:6833","nonce":60877422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6129,"qdb_cluster":[6129]},"id":4}]} 2024-03-06T07:25:57.699 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-03-06T07:25:57.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.699+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb65406b4d0 msgr2=0x7fb65406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.699+0000 7fb67730c640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb65406b4d0 0x7fb65406d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb6700d4150 tx=0x7fb660009040 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6700b9a10 msgr2=0x7fb6701553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6700b9a10 0x7fb6701553c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00c500 tx=0x7fb66c00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 shutdown_connections 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb65406b4d0 0x7fb65406d990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb67015df00 0x7fb6700b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6700b9a10 0x7fb6701553c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 --2- 172.21.15.27:0/1792868570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6700b95a0 0x7fb67015f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 >> 172.21.15.27:0/1792868570 conn(0x7fb670071960 msgr2=0x7fb6700d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 shutdown_connections 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.700+0000 7fb67730c640 1 -- 172.21.15.27:0/1792868570 wait complete. 2024-03-06T07:25:57.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.844+0000 7fa9a97a0640 1 Processor -- start 2024-03-06T07:25:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.844+0000 7fa9a97a0640 1 -- start start 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a97a0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa9a415e030 0x7fa9a406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a97a0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a415e410 con 0x7fa9a406ef70 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a97a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a4161af0 con 0x7fa9a40b2460 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a97a0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a4162700 con 0x7fa9a415e030 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a27fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa9a406ef70 0x7fa9a40b1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58238/0 (socket says 172.21.15.27:58238) 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a27fc640 1 -- 172.21.15.27:0/871167713 learned_addr learned my addr 172.21.15.27:0/871167713 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a2ffd640 1 --2- 172.21.15.27:0/871167713 >> v2:172.21.15.69:3300/0 conn(0x7fa9a415e030 0x7fa9a406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a1ffb640 1 -- 172.21.15.27:0/871167713 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549554361 0 0) 0x7fa994001110 con 0x7fa9a406ef70 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a1ffb640 1 -- 172.21.15.27:0/871167713 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa990003520 con 0x7fa9a406ef70 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a2ffd640 1 -- 172.21.15.27:0/871167713 >> v1:172.21.15.69:6789/0 conn(0x7fa9a40b2460 legacy=0x7fa9a406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a2ffd640 1 -- 172.21.15.27:0/871167713 >> v1:172.21.15.27:6789/0 conn(0x7fa9a406ef70 legacy=0x7fa9a40b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a1ffb640 1 -- 172.21.15.27:0/871167713 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019644118 0 0) 0x7fa9a4161af0 con 0x7fa9a40b2460 2024-03-06T07:25:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.845+0000 7fa9a2ffd640 1 -- 172.21.15.27:0/871167713 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a4161dd0 con 0x7fa9a415e030 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.846+0000 7fa9a2ffd640 1 --2- 172.21.15.27:0/871167713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a415e030 0x7fa9a406ea30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0097a0 tx=0x7fa98c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d27808f8c34d6b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.846+0000 7fa9a1ffb640 1 -- 172.21.15.27:0/871167713 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c034070 con 0x7fa9a415e030 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.846+0000 7fa9a1ffb640 1 -- 172.21.15.27:0/871167713 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c030610 con 0x7fa9a415e030 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.846+0000 7fa9a1ffb640 1 -- 172.21.15.27:0/871167713 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0309c0 con 0x7fa9a415e030 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.847+0000 7fa9a97a0640 1 -- 172.21.15.27:0/871167713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a415e030 msgr2=0x7fa9a406ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.847+0000 7fa9a97a0640 1 --2- 172.21.15.27:0/871167713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a415e030 0x7fa9a406ea30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0097a0 tx=0x7fa98c02b960 comp rx=0 tx=0).stop 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.847+0000 7fa9a97a0640 1 -- 172.21.15.27:0/871167713 shutdown_connections 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.847+0000 7fa9a97a0640 1 --2- 172.21.15.27:0/871167713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a415e030 0x7fa9a406ea30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.847+0000 7fa9a97a0640 1 -- 172.21.15.27:0/871167713 >> 172.21.15.27:0/871167713 conn(0x7fa9a40b55e0 msgr2=0x7fa9a40b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.847+0000 7fa9a97a0640 1 -- 172.21.15.27:0/871167713 shutdown_connections 2024-03-06T07:25:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.847+0000 7fa9a97a0640 1 -- 172.21.15.27:0/871167713 wait complete. 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.848+0000 7fa9a97a0640 1 Processor -- start 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.848+0000 7fa9a97a0640 1 -- start start 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a97a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9a406ef70 0x7fa9a4099b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a97a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a40b2460 0x7fa9a406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a97a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9a415e030 0x7fa9a406d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a97a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40b31e0 con 0x7fa9a406ef70 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a97a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a4162390 con 0x7fa9a40b2460 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a97a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40b3360 con 0x7fa9a415e030 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9a406ef70 0x7fa9a4099b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9a406ef70 0x7fa9a4099b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48692/0 (socket says 172.21.15.27:48692) 2024-03-06T07:25:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a2ffd640 1 -- 172.21.15.27:0/4104318024 learned_addr learned my addr 172.21.15.27:0/4104318024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a27fc640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a40b2460 0x7fa9a406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a37fe640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9a415e030 0x7fa9a406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a2ffd640 1 -- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9a415e030 msgr2=0x7fa9a406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.849+0000 7fa9a2ffd640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9a415e030 0x7fa9a406d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa9a2ffd640 1 -- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a40b2460 msgr2=0x7fa9a406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa9a2ffd640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a40b2460 0x7fa9a406d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa9a2ffd640 1 -- 172.21.15.27:0/4104318024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a415c040 con 0x7fa9a406ef70 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa9a27fc640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a40b2460 0x7fa9a406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa9a2ffd640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9a406ef70 0x7fa9a4099b20 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000c00 tx=0x7fa98c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa99b7fe640 1 -- 172.21.15.27:0/4104318024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c034040 con 0x7fa9a406ef70 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa99b7fe640 1 -- 172.21.15.27:0/4104318024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c03aaa0 con 0x7fa9a406ef70 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a41f7600 con 0x7fa9a406ef70 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa99b7fe640 1 -- 172.21.15.27:0/4104318024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0443f0 con 0x7fa9a406ef70 2024-03-06T07:25:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.850+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a415bbf0 con 0x7fa9a406ef70 2024-03-06T07:25:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.851+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a4067ff0 con 0x7fa9a406ef70 2024-03-06T07:25:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.851+0000 7fa99b7fe640 1 -- 172.21.15.27:0/4104318024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa98c04b060 con 0x7fa9a406ef70 2024-03-06T07:25:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.852+0000 7fa99b7fe640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa98406b4d0 0x7fa98406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.852+0000 7fa99b7fe640 1 -- 172.21.15.27:0/4104318024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa98c0abfa0 con 0x7fa9a406ef70 2024-03-06T07:25:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.852+0000 7fa9a27fc640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa98406b4d0 0x7fa98406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.853+0000 7fa9a27fc640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa98406b4d0 0x7fa98406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa994007fd0 tx=0x7fa994009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:57.856+0000 7fa99b7fe640 1 -- 172.21.15.27:0/4104318024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa98c03f070 con 0x7fa9a406ef70 2024-03-06T07:25:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.038+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9a40d06e0 con 0x7fa9a406ef70 2024-03-06T07:25:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.039+0000 7fa99b7fe640 1 -- 172.21.15.27:0/4104318024 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fa98c04b350 con 0x7fa9a406ef70 2024-03-06T07:25:58.040 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:58.041 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:25.095713+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:21.134461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:22.434887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa98406b4d0 msgr2=0x7fa98406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa98406b4d0 0x7fa98406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa994007fd0 tx=0x7fa994009040 comp rx=0 tx=0).stop 2024-03-06T07:25:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9a406ef70 msgr2=0x7fa9a4099b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9a406ef70 0x7fa9a4099b20 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000c00 tx=0x7fa98c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 shutdown_connections 2024-03-06T07:25:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa98406b4d0 0x7fa98406d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9a415e030 0x7fa9a406d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9a40b2460 0x7fa9a406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 --2- 172.21.15.27:0/4104318024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9a406ef70 0x7fa9a4099b20 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 >> 172.21.15.27:0/4104318024 conn(0x7fa9a40b55e0 msgr2=0x7fa9a40af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 shutdown_connections 2024-03-06T07:25:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.041+0000 7fa9a97a0640 1 -- 172.21.15.27:0/4104318024 wait complete. 2024-03-06T07:25:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:21.134461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:22.434887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:58.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:58.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.169+0000 7febdf66b640 1 Processor -- start 2024-03-06T07:25:58.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.169+0000 7febdf66b640 1 -- start start 2024-03-06T07:25:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdf66b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdf66b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febd80b6d10 0x7febd80b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdf66b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd80d3740 con 0x7febd815dd90 2024-03-06T07:25:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdf66b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd80d3400 con 0x7febd80b6d10 2024-03-06T07:25:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdf66b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd80d3120 con 0x7febd80ceea0 2024-03-06T07:25:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdd407640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7febd815dd90 0x7febd80d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58254/0 (socket says 172.21.15.27:58254) 2024-03-06T07:25:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdd407640 1 -- 172.21.15.27:0/1567195268 learned_addr learned my addr 172.21.15.27:0/1567195268 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdcc06640 1 --2- 172.21.15.27:0/1567195268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febddc08640 1 --2- 172.21.15.27:0/1567195268 >> v2:172.21.15.69:3300/0 conn(0x7febd80b6d10 0x7febd80b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febcffff640 1 -- 172.21.15.27:0/1567195268 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318659970 0 0) 0x7febd80d3740 con 0x7febd815dd90 2024-03-06T07:25:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febcffff640 1 -- 172.21.15.27:0/1567195268 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc4003520 con 0x7febd815dd90 2024-03-06T07:25:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdcc06640 1 -- 172.21.15.27:0/1567195268 >> v2:172.21.15.69:3300/0 conn(0x7febd80b6d10 msgr2=0x7febd80b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdcc06640 1 --2- 172.21.15.27:0/1567195268 >> v2:172.21.15.69:3300/0 conn(0x7febd80b6d10 0x7febd80b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdcc06640 1 -- 172.21.15.27:0/1567195268 >> v1:172.21.15.27:6789/0 conn(0x7febd815dd90 legacy=0x7febd80d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febdcc06640 1 -- 172.21.15.27:0/1567195268 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd80d39e0 con 0x7febd80ceea0 2024-03-06T07:25:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.170+0000 7febddc08640 1 --2- 172.21.15.27:0/1567195268 >> v2:172.21.15.69:3300/0 conn(0x7febd80b6d10 0x7febd80b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.171+0000 7febdcc06640 1 --2- 172.21.15.27:0/1567195268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80d29d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7febc80097d0 tx=0x7febc802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4c22fce1e9e57a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febcffff640 1 -- 172.21.15.27:0/1567195268 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8035020 con 0x7febd80ceea0 2024-03-06T07:25:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febcffff640 1 -- 172.21.15.27:0/1567195268 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc8031890 con 0x7febd80ceea0 2024-03-06T07:25:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febcffff640 1 -- 172.21.15.27:0/1567195268 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8031c60 con 0x7febd80ceea0 2024-03-06T07:25:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febdf66b640 1 -- 172.21.15.27:0/1567195268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 msgr2=0x7febd80d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febdf66b640 1 --2- 172.21.15.27:0/1567195268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80d29d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7febc80097d0 tx=0x7febc802b930 comp rx=0 tx=0).stop 2024-03-06T07:25:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febdf66b640 1 -- 172.21.15.27:0/1567195268 shutdown_connections 2024-03-06T07:25:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febdf66b640 1 --2- 172.21.15.27:0/1567195268 >> v2:172.21.15.69:3300/0 conn(0x7febd80b6d10 0x7febd80b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febdf66b640 1 --2- 172.21.15.27:0/1567195268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80d29d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febdf66b640 1 -- 172.21.15.27:0/1567195268 >> 172.21.15.27:0/1567195268 conn(0x7febd8071960 msgr2=0x7febd8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febdf66b640 1 -- 172.21.15.27:0/1567195268 shutdown_connections 2024-03-06T07:25:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.172+0000 7febdf66b640 1 -- 172.21.15.27:0/1567195268 wait complete. 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.173+0000 7febdf66b640 1 Processor -- start 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.173+0000 7febdf66b640 1 -- start start 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febdf66b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd80b6d10 0x7febd80cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febdf66b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febdf66b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febd815dd90 0x7febd80cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febdf66b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd80d48f0 con 0x7febd815dd90 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febdf66b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd815e170 con 0x7febd80b6d10 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febdf66b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd80d4a70 con 0x7febd80ceea0 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febddc08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febd815dd90 0x7febd80cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febddc08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febd815dd90 0x7febd80cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48704/0 (socket says 172.21.15.27:48704) 2024-03-06T07:25:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febddc08640 1 -- 172.21.15.27:0/1806456671 learned_addr learned my addr 172.21.15.27:0/1806456671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febdd407640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd80b6d10 0x7febd80cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.174+0000 7febdcc06640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febddc08640 1 -- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 msgr2=0x7febd80cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febddc08640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febddc08640 1 -- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd80b6d10 msgr2=0x7febd80cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febddc08640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd80b6d10 0x7febd80cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febddc08640 1 -- 172.21.15.27:0/1806456671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd81e7b00 con 0x7febd815dd90 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febdcc06640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febdd407640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd80b6d10 0x7febd80cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febddc08640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febd815dd90 0x7febd80cb6f0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7febd000efc0 tx=0x7febd0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febcdffb640 1 -- 172.21.15.27:0/1806456671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0018030 con 0x7febd815dd90 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febcdffb640 1 -- 172.21.15.27:0/1806456671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd000cde0 con 0x7febd815dd90 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd8140bf0 con 0x7febd815dd90 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febcdffb640 1 -- 172.21.15.27:0/1806456671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0003bb0 con 0x7febd815dd90 2024-03-06T07:25:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.175+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd80ccba0 con 0x7febd815dd90 2024-03-06T07:25:58.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.176+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd8072a30 con 0x7febd815dd90 2024-03-06T07:25:58.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.177+0000 7febcdffb640 1 -- 172.21.15.27:0/1806456671 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febd001b070 con 0x7febd815dd90 2024-03-06T07:25:58.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.177+0000 7febcdffb640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febbc06b4d0 0x7febbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.177+0000 7febcdffb640 1 -- 172.21.15.27:0/1806456671 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febd008ac00 con 0x7febd815dd90 2024-03-06T07:25:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.177+0000 7febdd407640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febbc06b4d0 0x7febbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.178+0000 7febdd407640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febbc06b4d0 0x7febbc06d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7febd4007fd0 tx=0x7febd4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.182+0000 7febcdffb640 1 -- 172.21.15.27:0/1806456671 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febd0091080 con 0x7febd815dd90 2024-03-06T07:25:58.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.363+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7febd80b8d00 con 0x7febd815dd90 2024-03-06T07:25:58.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.364+0000 7febcdffb640 1 -- 172.21.15.27:0/1806456671 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7febd0010040 con 0x7febd815dd90 2024-03-06T07:25:58.365 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:25:58.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febbc06b4d0 msgr2=0x7febbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febbc06b4d0 0x7febbc06d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7febd4007fd0 tx=0x7febd4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:58.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febd815dd90 msgr2=0x7febd80cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febd815dd90 0x7febd80cb6f0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7febd000efc0 tx=0x7febd0009f90 comp rx=0 tx=0).stop 2024-03-06T07:25:58.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 shutdown_connections 2024-03-06T07:25:58.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febbc06b4d0 0x7febbc06d990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febd815dd90 0x7febd80cb6f0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febd80ceea0 0x7febd80cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 --2- 172.21.15.27:0/1806456671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd80b6d10 0x7febd80cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 >> 172.21.15.27:0/1806456671 conn(0x7febd8071960 msgr2=0x7febd80b71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 shutdown_connections 2024-03-06T07:25:58.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.366+0000 7febdf66b640 1 -- 172.21.15.27:0/1806456671 wait complete. 2024-03-06T07:25:58.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.486+0000 7fcb4fea4640 1 Processor -- start 2024-03-06T07:25:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.487+0000 7fcb4fea4640 1 -- start start 2024-03-06T07:25:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.487+0000 7fcb4fea4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcb480ceea0 0x7fcb480d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.487+0000 7fcb4fea4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480d3740 con 0x7fcb4815dd90 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.487+0000 7fcb4fea4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480d3120 con 0x7fcb480b6d10 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.487+0000 7fcb4fea4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb480d3400 con 0x7fcb480ceea0 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.487+0000 7fcb4dc40640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcb4815dd90 0x7fcb480d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58266/0 (socket says 172.21.15.27:58266) 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.487+0000 7fcb4dc40640 1 -- 172.21.15.27:0/2263759016 learned_addr learned my addr 172.21.15.27:0/2263759016 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.488+0000 7fcb4d43f640 1 --2- 172.21.15.27:0/2263759016 >> v2:172.21.15.69:3300/0 conn(0x7fcb480ceea0 0x7fcb480d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.488+0000 7fcb4cc3e640 1 -- 172.21.15.27:0/2263759016 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572524954 0 0) 0x7fcb480d3740 con 0x7fcb4815dd90 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.488+0000 7fcb4cc3e640 1 -- 172.21.15.27:0/2263759016 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb34003520 con 0x7fcb4815dd90 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.488+0000 7fcb4cc3e640 1 -- 172.21.15.27:0/2263759016 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856069209 0 0) 0x7fcb480d3120 con 0x7fcb480b6d10 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.488+0000 7fcb4cc3e640 1 -- 172.21.15.27:0/2263759016 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb480d3740 con 0x7fcb480b6d10 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.488+0000 7fcb4cc3e640 1 -- 172.21.15.27:0/2263759016 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2037903752 0 0) 0x7fcb34003520 con 0x7fcb4815dd90 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.488+0000 7fcb4d43f640 1 -- 172.21.15.27:0/2263759016 >> v1:172.21.15.69:6789/0 conn(0x7fcb480b6d10 legacy=0x7fcb480d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.488+0000 7fcb4d43f640 1 -- 172.21.15.27:0/2263759016 >> v1:172.21.15.27:6789/0 conn(0x7fcb4815dd90 legacy=0x7fcb480d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.488+0000 7fcb4d43f640 1 -- 172.21.15.27:0/2263759016 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb480d39e0 con 0x7fcb480ceea0 2024-03-06T07:25:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.489+0000 7fcb4d43f640 1 --2- 172.21.15.27:0/2263759016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 0x7fcb480d29d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcb440097a0 tx=0x7fcb4402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31f34812ee354507 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.489+0000 7fcb4cc3e640 1 -- 172.21.15.27:0/2263759016 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38001670 con 0x7fcb480ceea0 2024-03-06T07:25:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.489+0000 7fcb4cc3e640 1 -- 172.21.15.27:0/2263759016 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb44037070 con 0x7fcb480ceea0 2024-03-06T07:25:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.489+0000 7fcb4cc3e640 1 -- 172.21.15.27:0/2263759016 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44031790 con 0x7fcb480ceea0 2024-03-06T07:25:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.490+0000 7fcb4fea4640 1 -- 172.21.15.27:0/2263759016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 msgr2=0x7fcb480d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.490+0000 7fcb4fea4640 1 --2- 172.21.15.27:0/2263759016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 0x7fcb480d29d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcb440097a0 tx=0x7fcb4402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.490+0000 7fcb4fea4640 1 -- 172.21.15.27:0/2263759016 shutdown_connections 2024-03-06T07:25:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.490+0000 7fcb4fea4640 1 --2- 172.21.15.27:0/2263759016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 0x7fcb480d29d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.490+0000 7fcb4fea4640 1 -- 172.21.15.27:0/2263759016 >> 172.21.15.27:0/2263759016 conn(0x7fcb48071960 msgr2=0x7fcb48071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.490+0000 7fcb4fea4640 1 -- 172.21.15.27:0/2263759016 shutdown_connections 2024-03-06T07:25:58.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.490+0000 7fcb4fea4640 1 -- 172.21.15.27:0/2263759016 wait complete. 2024-03-06T07:25:58.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.492+0000 7fcb4fea4640 1 Processor -- start 2024-03-06T07:25:58.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.492+0000 7fcb4fea4640 1 -- start start 2024-03-06T07:25:58.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.493+0000 7fcb4fea4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b6d10 0x7fcb480c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.493+0000 7fcb4fea4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 0x7fcb481f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.493+0000 7fcb4fea4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb4815dd90 0x7fcb481f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.493+0000 7fcb4fea4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480d48f0 con 0x7fcb480b6d10 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.493+0000 7fcb4fea4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4815e170 con 0x7fcb480ceea0 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.493+0000 7fcb4fea4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480d4a70 con 0x7fcb4815dd90 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.493+0000 7fcb4dc40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b6d10 0x7fcb480c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4dc40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b6d10 0x7fcb480c9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48720/0 (socket says 172.21.15.27:48720) 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4dc40640 1 -- 172.21.15.27:0/167754705 learned_addr learned my addr 172.21.15.27:0/167754705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4e441640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb4815dd90 0x7fcb481f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4d43f640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 0x7fcb481f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4dc40640 1 -- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb4815dd90 msgr2=0x7fcb481f79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4dc40640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb4815dd90 0x7fcb481f79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4dc40640 1 -- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 msgr2=0x7fcb481f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4dc40640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 0x7fcb481f74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4dc40640 1 -- 172.21.15.27:0/167754705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb480c8530 con 0x7fcb480b6d10 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4e441640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb4815dd90 0x7fcb481f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4d43f640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 0x7fcb481f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4dc40640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b6d10 0x7fcb480c9e40 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c750 tx=0x7fcb3800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/167754705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38005ad0 con 0x7fcb480b6d10 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/167754705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb38005c70 con 0x7fcb480b6d10 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb480c8810 con 0x7fcb480b6d10 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/167754705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38005e60 con 0x7fcb480b6d10 2024-03-06T07:25:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.494+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb480c90b0 con 0x7fcb480b6d10 2024-03-06T07:25:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.495+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb480c89e0 con 0x7fcb480b6d10 2024-03-06T07:25:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.496+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/167754705 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb380072b0 con 0x7fcb480b6d10 2024-03-06T07:25:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.496+0000 7fcb3e7fc640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.497+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/167754705 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb3808a050 con 0x7fcb480b6d10 2024-03-06T07:25:58.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.497+0000 7fcb4d43f640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.497+0000 7fcb4d43f640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcb44036000 tx=0x7fcb44008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.501+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/167754705 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb38019300 con 0x7fcb480b6d10 2024-03-06T07:25:58.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.681+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcb48072a30 con 0x7fcb480b6d10 2024-03-06T07:25:58.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.681+0000 7fcb3e7fc640 1 -- 172.21.15.27:0/167754705 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcb38058f90 con 0x7fcb480b6d10 2024-03-06T07:25:58.684 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b4d0 msgr2=0x7fcb2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcb44036000 tx=0x7fcb44008030 comp rx=0 tx=0).stop 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b6d10 msgr2=0x7fcb480c9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b6d10 0x7fcb480c9e40 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c750 tx=0x7fcb3800cc10 comp rx=0 tx=0).stop 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 shutdown_connections 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcb2c06b4d0 0x7fcb2c06d990 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcb4815dd90 0x7fcb481f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb480ceea0 0x7fcb481f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 --2- 172.21.15.27:0/167754705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb480b6d10 0x7fcb480c9e40 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 >> 172.21.15.27:0/167754705 conn(0x7fcb48071960 msgr2=0x7fcb480b3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 shutdown_connections 2024-03-06T07:25:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.683+0000 7fcb4fea4640 1 -- 172.21.15.27:0/167754705 wait complete. 2024-03-06T07:25:58.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:58.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.803+0000 7fa93fad4640 1 Processor -- start 2024-03-06T07:25:58.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.803+0000 7fa93fad4640 1 -- start start 2024-03-06T07:25:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.804+0000 7fa93fad4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9380cdcd0 0x7fa9380b1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.804+0000 7fa93fad4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa93806b2c0 0x7fa93806b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.804+0000 7fa93fad4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa938161a40 con 0x7fa9380b2cd0 2024-03-06T07:25:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.804+0000 7fa93fad4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9380d11e0 con 0x7fa9380cdcd0 2024-03-06T07:25:58.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.804+0000 7fa93fad4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380b2a20 con 0x7fa93806b2c0 2024-03-06T07:25:58.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.804+0000 7fa93d870640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9380cdcd0 0x7fa9380b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.804+0000 7fa93d06f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa9380b2cd0 0x7fa9380b1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58278/0 (socket says 172.21.15.27:58278) 2024-03-06T07:25:58.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.804+0000 7fa93d870640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9380cdcd0 0x7fa9380b1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48728/0 (socket says 172.21.15.27:48728) 2024-03-06T07:25:58.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.804+0000 7fa93d06f640 1 -- 172.21.15.27:0/1028670865 learned_addr learned my addr 172.21.15.27:0/1028670865 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:58.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.805+0000 7fa93e071640 1 --2- 172.21.15.27:0/1028670865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa93806b2c0 0x7fa93806b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.805+0000 7fa93d870640 1 -- 172.21.15.27:0/1028670865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa93806b2c0 msgr2=0x7fa93806b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.805+0000 7fa93d870640 1 --2- 172.21.15.27:0/1028670865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa93806b2c0 0x7fa93806b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.805+0000 7fa93d870640 1 -- 172.21.15.27:0/1028670865 >> v1:172.21.15.27:6789/0 conn(0x7fa9380b2cd0 legacy=0x7fa9380b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.805+0000 7fa93d870640 1 -- 172.21.15.27:0/1028670865 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9380d0940 con 0x7fa9380cdcd0 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.805+0000 7fa93c86e640 1 -- 172.21.15.27:0/1028670865 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525449840 0 0) 0x7fa938161a40 con 0x7fa9380b2cd0 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93d870640 1 --2- 172.21.15.27:0/1028670865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9380cdcd0 0x7fa9380b1550 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa9340097a0 tx=0x7fa93402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c87754e3050d9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93c86e640 1 -- 172.21.15.27:0/1028670865 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9340315a0 con 0x7fa9380cdcd0 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93c86e640 1 -- 172.21.15.27:0/1028670865 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa934037070 con 0x7fa9380cdcd0 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93c86e640 1 -- 172.21.15.27:0/1028670865 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa934031960 con 0x7fa9380cdcd0 2024-03-06T07:25:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93fad4640 1 -- 172.21.15.27:0/1028670865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9380cdcd0 msgr2=0x7fa9380b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93fad4640 1 --2- 172.21.15.27:0/1028670865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9380cdcd0 0x7fa9380b1550 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa9340097a0 tx=0x7fa93402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93fad4640 1 -- 172.21.15.27:0/1028670865 shutdown_connections 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93fad4640 1 --2- 172.21.15.27:0/1028670865 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa93806b2c0 0x7fa93806b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93fad4640 1 --2- 172.21.15.27:0/1028670865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9380cdcd0 0x7fa9380b1550 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93fad4640 1 -- 172.21.15.27:0/1028670865 >> 172.21.15.27:0/1028670865 conn(0x7fa9380b3700 msgr2=0x7fa9380b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93fad4640 1 -- 172.21.15.27:0/1028670865 shutdown_connections 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.806+0000 7fa93fad4640 1 -- 172.21.15.27:0/1028670865 wait complete. 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93fad4640 1 Processor -- start 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93fad4640 1 -- start start 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93fad4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa93806b2c0 0x7fa938157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93fad4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9380b2cd0 0x7fa9380b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93fad4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9380cdcd0 0x7fa9380b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93fad4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380cc1c0 con 0x7fa93806b2c0 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93fad4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380d0f00 con 0x7fa9380b2cd0 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93fad4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380cc340 con 0x7fa9380cdcd0 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93d870640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa93806b2c0 0x7fa938157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93d870640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa93806b2c0 0x7fa938157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48732/0 (socket says 172.21.15.27:48732) 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.808+0000 7fa93d870640 1 -- 172.21.15.27:0/1294807911 learned_addr learned my addr 172.21.15.27:0/1294807911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93e071640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9380cdcd0 0x7fa9380b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93d06f640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9380b2cd0 0x7fa9380b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93d870640 1 -- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9380cdcd0 msgr2=0x7fa9380b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93d870640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9380cdcd0 0x7fa9380b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93d870640 1 -- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9380b2cd0 msgr2=0x7fa9380b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93d870640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9380b2cd0 0x7fa9380b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93d870640 1 -- 172.21.15.27:0/1294807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9381f4dd0 con 0x7fa93806b2c0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93d870640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa93806b2c0 0x7fa938157280 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa9340098c0 tx=0x7fa9340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa92e7fc640 1 -- 172.21.15.27:0/1294807911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa934032840 con 0x7fa93806b2c0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa92e7fc640 1 -- 172.21.15.27:0/1294807911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9340329e0 con 0x7fa93806b2c0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9380b51b0 con 0x7fa93806b2c0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa92e7fc640 1 -- 172.21.15.27:0/1294807911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa934032dd0 con 0x7fa93806b2c0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.809+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9380b5770 con 0x7fa93806b2c0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.810+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93809fcf0 con 0x7fa93806b2c0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.811+0000 7fa92e7fc640 1 -- 172.21.15.27:0/1294807911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa93404d3f0 con 0x7fa93806b2c0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.811+0000 7fa92e7fc640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa91c06b4d0 0x7fa91c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.811+0000 7fa92e7fc640 1 -- 172.21.15.27:0/1294807911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa93402f030 con 0x7fa93806b2c0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.811+0000 7fa93d06f640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa91c06b4d0 0x7fa91c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.812+0000 7fa93d06f640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa91c06b4d0 0x7fa91c06d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa93806fdc0 tx=0x7fa928007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:58.821+0000 7fa92e7fc640 1 -- 172.21.15.27:0/1294807911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa934040070 con 0x7fa93806b2c0 2024-03-06T07:25:59.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.004+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa9380d1690 con 0x7fa93806b2c0 2024-03-06T07:25:59.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.005+0000 7fa92e7fc640 1 -- 172.21.15.27:0/1294807911 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa93407aa60 con 0x7fa93806b2c0 2024-03-06T07:25:59.007 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:25:59.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa91c06b4d0 msgr2=0x7fa91c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa91c06b4d0 0x7fa91c06d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa93806fdc0 tx=0x7fa928007b50 comp rx=0 tx=0).stop 2024-03-06T07:25:59.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa93806b2c0 msgr2=0x7fa938157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa93806b2c0 0x7fa938157280 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa9340098c0 tx=0x7fa9340046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 shutdown_connections 2024-03-06T07:25:59.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa91c06b4d0 0x7fa91c06d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa9380cdcd0 0x7fa9380b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9380b2cd0 0x7fa9380b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 --2- 172.21.15.27:0/1294807911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa93806b2c0 0x7fa938157280 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 >> 172.21.15.27:0/1294807911 conn(0x7fa9380b3700 msgr2=0x7fa938070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 shutdown_connections 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.007+0000 7fa93fad4640 1 -- 172.21.15.27:0/1294807911 wait complete. 2024-03-06T07:25:59.020 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:59.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.125+0000 7f714c6b1640 1 Processor -- start 2024-03-06T07:25:59.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.125+0000 7f714c6b1640 1 -- start start 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f714c6b1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71440d3400 0x7f71440b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f714c6b1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f71440b2b50 0x7f71440b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f714c6b1640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71440ce610 con 0x7f7144071540 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f714c6b1640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71440d2f70 con 0x7f71440d3400 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f714c6b1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71440d4be0 con 0x7f71440b2b50 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f7149c4c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71440d3400 0x7f71440b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f7149c4c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71440d3400 0x7f71440b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48740/0 (socket says 172.21.15.27:48740) 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f7149c4c640 1 -- 172.21.15.27:0/3390076949 learned_addr learned my addr 172.21.15.27:0/3390076949 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f714ac4e640 1 --2- 172.21.15.27:0/3390076949 >> v2:172.21.15.69:3300/0 conn(0x7f71440b2b50 0x7f71440b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f7149c4c640 1 -- 172.21.15.27:0/3390076949 >> v2:172.21.15.69:3300/0 conn(0x7f71440b2b50 msgr2=0x7f71440b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f7149c4c640 1 --2- 172.21.15.27:0/3390076949 >> v2:172.21.15.69:3300/0 conn(0x7f71440b2b50 0x7f71440b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f7149c4c640 1 -- 172.21.15.27:0/3390076949 >> v1:172.21.15.69:6789/0 conn(0x7f7144071540 legacy=0x7f7144071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f7149c4c640 1 -- 172.21.15.27:0/3390076949 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71440d2920 con 0x7f71440d3400 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.126+0000 7f714ac4e640 1 --2- 172.21.15.27:0/3390076949 >> v2:172.21.15.69:3300/0 conn(0x7f71440b2b50 0x7f71440b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.127+0000 7f7149c4c640 1 --2- 172.21.15.27:0/3390076949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440d3400 0x7f71440b30a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f71440683b0 tx=0x7f713402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c3853acdb58142 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.129+0000 7f714944b640 1 -- 172.21.15.27:0/3390076949 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134035020 con 0x7f71440d3400 2024-03-06T07:25:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.129+0000 7f714944b640 1 -- 172.21.15.27:0/3390076949 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7134031870 con 0x7f71440d3400 2024-03-06T07:25:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.129+0000 7f714944b640 1 -- 172.21.15.27:0/3390076949 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134031c40 con 0x7f71440d3400 2024-03-06T07:25:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.129+0000 7f714c6b1640 1 -- 172.21.15.27:0/3390076949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440d3400 msgr2=0x7f71440b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.129+0000 7f714c6b1640 1 --2- 172.21.15.27:0/3390076949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440d3400 0x7f71440b30a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f71440683b0 tx=0x7f713402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.129+0000 7f714c6b1640 1 -- 172.21.15.27:0/3390076949 shutdown_connections 2024-03-06T07:25:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.129+0000 7f714c6b1640 1 --2- 172.21.15.27:0/3390076949 >> v2:172.21.15.69:3300/0 conn(0x7f71440b2b50 0x7f71440b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.129+0000 7f714c6b1640 1 --2- 172.21.15.27:0/3390076949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440d3400 0x7f71440b30a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.130+0000 7f714c6b1640 1 -- 172.21.15.27:0/3390076949 >> 172.21.15.27:0/3390076949 conn(0x7f71440d3ca0 msgr2=0x7f71440d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.130+0000 7f714c6b1640 1 -- 172.21.15.27:0/3390076949 shutdown_connections 2024-03-06T07:25:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.130+0000 7f714c6b1640 1 -- 172.21.15.27:0/3390076949 wait complete. 2024-03-06T07:25:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.131+0000 7f714c6b1640 1 Processor -- start 2024-03-06T07:25:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.131+0000 7f714c6b1640 1 -- start start 2024-03-06T07:25:59.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.131+0000 7f714c6b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7144071540 0x7f71440b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f714c6b1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b2b50 0x7f71440b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f714c6b1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d3400 0x7f7144158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f714c6b1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714415e370 con 0x7f71440b2b50 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f714c6b1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71440ce8f0 con 0x7f71440d3400 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f714c6b1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71440cb4c0 con 0x7f7144071540 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f7149c4c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b2b50 0x7f71440b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f7149c4c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b2b50 0x7f71440b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48742/0 (socket says 172.21.15.27:48742) 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f7149c4c640 1 -- 172.21.15.27:0/2203023967 learned_addr learned my addr 172.21.15.27:0/2203023967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f714a44d640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7144071540 0x7f71440b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f714ac4e640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d3400 0x7f7144158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f7149c4c640 1 -- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7144071540 msgr2=0x7f71440b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f7149c4c640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7144071540 0x7f71440b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f7149c4c640 1 -- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d3400 msgr2=0x7f7144158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f7149c4c640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d3400 0x7f7144158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f7149c4c640 1 -- 172.21.15.27:0/2203023967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144152800 con 0x7f71440b2b50 2024-03-06T07:25:59.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f7149c4c640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b2b50 0x7f71440b45c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7134000c00 tx=0x7f71340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f713affd640 1 -- 172.21.15.27:0/2203023967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134040070 con 0x7f71440b2b50 2024-03-06T07:25:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f713affd640 1 -- 172.21.15.27:0/2203023967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713403a420 con 0x7f71440b2b50 2024-03-06T07:25:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714415a110 con 0x7f71440b2b50 2024-03-06T07:25:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.132+0000 7f713affd640 1 -- 172.21.15.27:0/2203023967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134035040 con 0x7f71440b2b50 2024-03-06T07:25:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.133+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71440b4ed0 con 0x7f71440b2b50 2024-03-06T07:25:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.133+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7144068700 con 0x7f71440b2b50 2024-03-06T07:25:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.134+0000 7f713affd640 1 -- 172.21.15.27:0/2203023967 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f713404b060 con 0x7f71440b2b50 2024-03-06T07:25:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.134+0000 7f713affd640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f712806b500 0x7f712806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.134+0000 7f713affd640 1 -- 172.21.15.27:0/2203023967 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71340abfb0 con 0x7f71440b2b50 2024-03-06T07:25:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.135+0000 7f714a44d640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f712806b500 0x7f712806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.135+0000 7f714a44d640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f712806b500 0x7f712806d9c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7140004f60 tx=0x7f7140007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.139+0000 7f713affd640 1 -- 172.21.15.27:0/2203023967 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f713407a7e0 con 0x7f71440b2b50 2024-03-06T07:25:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.394+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7144161210 con 0x7f71440b2b50 2024-03-06T07:25:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.395+0000 7f713affd640 1 -- 172.21.15.27:0/2203023967 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f713407e880 con 0x7f71440b2b50 2024-03-06T07:25:59.397 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:59.398 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5324,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3885016831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3885016831},{"type":"v1","addr":"172.21.15.69:6836","nonce":3885016831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5327,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3510003111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3510003111},{"type":"v1","addr":"172.21.15.69:6837","nonce":3510003111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5692,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3053251134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3053251134},{"type":"v1","addr":"172.21.15.27:6835","nonce":3053251134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:23.445711+0000","modified":"2024-03-06T07:25:26.468573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6129":{"gid":6129,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6833/60877422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":60877422},{"type":"v1","addr":"172.21.15.27:6833","nonce":60877422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6129,"qdb_cluster":[6129]},"id":4}]} 2024-03-06T07:25:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-03-06T07:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f712806b500 msgr2=0x7f712806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f712806b500 0x7f712806d9c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7140004f60 tx=0x7f7140007e30 comp rx=0 tx=0).stop 2024-03-06T07:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b2b50 msgr2=0x7f71440b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b2b50 0x7f71440b45c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7134000c00 tx=0x7f71340046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 shutdown_connections 2024-03-06T07:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f712806b500 0x7f712806d9c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d3400 0x7f7144158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b2b50 0x7f71440b45c0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 --2- 172.21.15.27:0/2203023967 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7144071540 0x7f71440b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 >> 172.21.15.27:0/2203023967 conn(0x7f71440d3ca0 msgr2=0x7f714415f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 shutdown_connections 2024-03-06T07:25:59.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.397+0000 7f714c6b1640 1 -- 172.21.15.27:0/2203023967 wait complete. 2024-03-06T07:25:59.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.522+0000 7fdde9b9a640 1 Processor -- start 2024-03-06T07:25:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.522+0000 7fdde9b9a640 1 -- start start 2024-03-06T07:25:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.523+0000 7fdde9b9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde41511a0 0x7fdde4070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.523+0000 7fdde9b9a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdde406c710 0x7fdde40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.523+0000 7fdde9b9a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde406ccd0 con 0x7fdde406dd70 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.523+0000 7fdde9b9a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde40aeee0 con 0x7fdde406c710 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.523+0000 7fdde9b9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde406cfb0 con 0x7fdde41511a0 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.524+0000 7fdde37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde41511a0 0x7fdde4070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.524+0000 7fdde3fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdde406c710 0x7fdde40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.524+0000 7fdde2ffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdde406dd70 0x7fdde415dd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:43250/0 (socket says 172.21.15.27:43250) 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.524+0000 7fdde2ffd640 1 -- 172.21.15.27:0/3445189563 learned_addr learned my addr 172.21.15.27:0/3445189563 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.524+0000 7fdde3fff640 1 -- 172.21.15.27:0/3445189563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde41511a0 msgr2=0x7fdde4070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.524+0000 7fdde3fff640 1 --2- 172.21.15.27:0/3445189563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde41511a0 0x7fdde4070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.524+0000 7fdde3fff640 1 -- 172.21.15.27:0/3445189563 >> v1:172.21.15.69:6789/0 conn(0x7fdde406dd70 legacy=0x7fdde415dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.524+0000 7fdde3fff640 1 -- 172.21.15.27:0/3445189563 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde406d290 con 0x7fdde406c710 2024-03-06T07:25:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.524+0000 7fdde37fe640 1 --2- 172.21.15.27:0/3445189563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde41511a0 0x7fdde4070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.525+0000 7fdde3fff640 1 --2- 172.21.15.27:0/3445189563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde406c710 0x7fdde40710c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009d90 tx=0x7fdddc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60b652a0c0fdaf57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.525+0000 7fdde27fc640 1 -- 172.21.15.27:0/3445189563 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc039070 con 0x7fdde406c710 2024-03-06T07:25:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.525+0000 7fdde27fc640 1 -- 172.21.15.27:0/3445189563 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc02cc50 con 0x7fdde406c710 2024-03-06T07:25:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.525+0000 7fdde27fc640 1 -- 172.21.15.27:0/3445189563 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc034440 con 0x7fdde406c710 2024-03-06T07:25:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.525+0000 7fdde9b9a640 1 -- 172.21.15.27:0/3445189563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde406c710 msgr2=0x7fdde40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.525+0000 7fdde9b9a640 1 --2- 172.21.15.27:0/3445189563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde406c710 0x7fdde40710c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009d90 tx=0x7fdddc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.526+0000 7fdde9b9a640 1 -- 172.21.15.27:0/3445189563 shutdown_connections 2024-03-06T07:25:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.526+0000 7fdde9b9a640 1 --2- 172.21.15.27:0/3445189563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde406c710 0x7fdde40710c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.526+0000 7fdde9b9a640 1 --2- 172.21.15.27:0/3445189563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde41511a0 0x7fdde4070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.526+0000 7fdde9b9a640 1 -- 172.21.15.27:0/3445189563 >> 172.21.15.27:0/3445189563 conn(0x7fdde4066d00 msgr2=0x7fdde40670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.526+0000 7fdde9b9a640 1 -- 172.21.15.27:0/3445189563 shutdown_connections 2024-03-06T07:25:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.526+0000 7fdde9b9a640 1 -- 172.21.15.27:0/3445189563 wait complete. 2024-03-06T07:25:59.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.527+0000 7fdde9b9a640 1 Processor -- start 2024-03-06T07:25:59.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.527+0000 7fdde9b9a640 1 -- start start 2024-03-06T07:25:59.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde9b9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde406c710 0x7fdde4159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde9b9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde406dd70 0x7fdde414d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde9b9a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde41511a0 0x7fdde41f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde9b9a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde4151580 con 0x7fdde41511a0 2024-03-06T07:25:59.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde9b9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde406d850 con 0x7fdde406dd70 2024-03-06T07:25:59.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde9b9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde415c1d0 con 0x7fdde406c710 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde41511a0 0x7fdde41f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde406c710 0x7fdde4159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde41511a0 0x7fdde41f3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48744/0 (socket says 172.21.15.27:48744) 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde3fff640 1 -- 172.21.15.27:0/1468654558 learned_addr learned my addr 172.21.15.27:0/1468654558 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde2ffd640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde406dd70 0x7fdde414d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde37fe640 1 -- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde406dd70 msgr2=0x7fdde414d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde37fe640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde406dd70 0x7fdde414d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde37fe640 1 -- 172.21.15.27:0/1468654558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde41511a0 msgr2=0x7fdde41f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde37fe640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde41511a0 0x7fdde41f3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde37fe640 1 -- 172.21.15.27:0/1468654558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde414e8b0 con 0x7fdde406c710 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde3fff640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde41511a0 0x7fdde41f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.528+0000 7fdde37fe640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde406c710 0x7fdde4159440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fddd400e8e0 tx=0x7fddd400eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.529+0000 7fddcbfff640 1 -- 172.21.15.27:0/1468654558 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd400b040 con 0x7fdde406c710 2024-03-06T07:25:59.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.529+0000 7fddcbfff640 1 -- 172.21.15.27:0/1468654558 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd400cc40 con 0x7fdde406c710 2024-03-06T07:25:59.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.529+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde414cb30 con 0x7fdde406c710 2024-03-06T07:25:59.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.529+0000 7fddcbfff640 1 -- 172.21.15.27:0/1468654558 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4004030 con 0x7fdde406c710 2024-03-06T07:25:59.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.529+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde41f4bd0 con 0x7fdde406c710 2024-03-06T07:25:59.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.530+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde414d220 con 0x7fdde406c710 2024-03-06T07:25:59.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.530+0000 7fddcbfff640 1 -- 172.21.15.27:0/1468654558 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddd40041d0 con 0x7fdde406c710 2024-03-06T07:25:59.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.531+0000 7fddcbfff640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddd006b4d0 0x7fddd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.531+0000 7fddcbfff640 1 -- 172.21.15.27:0/1468654558 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddd4089d10 con 0x7fdde406c710 2024-03-06T07:25:59.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.531+0000 7fdde2ffd640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddd006b4d0 0x7fddd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.531+0000 7fdde2ffd640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddd006b4d0 0x7fddd006d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdde40cea10 tx=0x7fddd8007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.535+0000 7fddcbfff640 1 -- 172.21.15.27:0/1468654558 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddd4016020 con 0x7fdde406c710 2024-03-06T07:25:59.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.716+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdde4067ff0 con 0x7fdde406c710 2024-03-06T07:25:59.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.718+0000 7fddcbfff640 1 -- 172.21.15.27:0/1468654558 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fddd400a550 con 0x7fdde406c710 2024-03-06T07:25:59.720 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:25:59.721 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:25.095713+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:21.134461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:22.434887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddd006b4d0 msgr2=0x7fddd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddd006b4d0 0x7fddd006d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdde40cea10 tx=0x7fddd8007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde406c710 msgr2=0x7fdde4159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde406c710 0x7fdde4159440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fddd400e8e0 tx=0x7fddd400eda0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 shutdown_connections 2024-03-06T07:25:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddd006b4d0 0x7fddd006d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde41511a0 0x7fdde41f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdde406dd70 0x7fdde414d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 --2- 172.21.15.27:0/1468654558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdde406c710 0x7fdde4159440 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 >> 172.21.15.27:0/1468654558 conn(0x7fdde4066d00 msgr2=0x7fdde414f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 shutdown_connections 2024-03-06T07:25:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.720+0000 7fdde9b9a640 1 -- 172.21.15.27:0/1468654558 wait complete. 2024-03-06T07:25:59.734 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:59.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:59.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.838+0000 7f4a2b64e640 1 Processor -- start 2024-03-06T07:25:59.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.839+0000 7f4a2b64e640 1 -- start start 2024-03-06T07:25:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.839+0000 7f4a2b64e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2406b5c0 0x7f4a240b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.839+0000 7f4a2b64e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a240b3340 con 0x7f4a240b0e90 2024-03-06T07:25:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.839+0000 7f4a2b64e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a240cee90 con 0x7f4a240ae9b0 2024-03-06T07:25:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.839+0000 7f4a2b64e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a240cfa00 con 0x7f4a2406b5c0 2024-03-06T07:25:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a28be9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a240b0e90 0x7f4a2406b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58292/0 (socket says 172.21.15.27:58292) 2024-03-06T07:25:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a28be9640 1 -- 172.21.15.27:0/948463590 learned_addr learned my addr 172.21.15.27:0/948463590 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:25:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a293ea640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2406b5c0 0x7f4a240b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249747712 0 0) 0x7f4a240b3340 con 0x7f4a240b0e90 2024-03-06T07:25:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a10003520 con 0x7f4a240b0e90 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4221398902 0 0) 0x7f4a10003520 con 0x7f4a240b0e90 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2406b5c0 msgr2=0x7f4a240b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2406b5c0 0x7f4a240b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 >> v1:172.21.15.69:6789/0 conn(0x7f4a240ae9b0 legacy=0x7f4a240d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a240cfc50 con 0x7f4a240b0e90 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f4a14001670 con 0x7f4a240b0e90 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 >> v1:172.21.15.27:6789/0 conn(0x7f4a240b0e90 legacy=0x7f4a2406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a100039e0 0x7f4a10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a10004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2406b5c0 0x7f4a10005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a240cfa00 con 0x7f4a10004390 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1000ba70 con 0x7f4a100039e0 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1000bc70 con 0x7f4a2406b5c0 2024-03-06T07:25:59.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a28be9640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a10004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a293ea640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a100039e0 0x7f4a10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.840+0000 7f4a29beb640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2406b5c0 0x7f4a10005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.841+0000 7f4a28be9640 1 -- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2406b5c0 msgr2=0x7f4a10005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.841+0000 7f4a28be9640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2406b5c0 0x7f4a10005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.841+0000 7f4a28be9640 1 -- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a100039e0 msgr2=0x7f4a10003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.841+0000 7f4a28be9640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a100039e0 0x7f4a10003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.841+0000 7f4a28be9640 1 -- 172.21.15.27:0/948463590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a10003520 con 0x7f4a10004390 2024-03-06T07:25:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.841+0000 7f4a293ea640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a100039e0 0x7f4a10003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.841+0000 7f4a28be9640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a10004860 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4a1400c290 tx=0x7f4a10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1335077eb02a3b6b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14007d50 con 0x7f4a10004390 2024-03-06T07:25:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a14030070 con 0x7f4a10004390 2024-03-06T07:25:59.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a1bfff640 1 -- 172.21.15.27:0/948463590 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1402cd60 con 0x7f4a10004390 2024-03-06T07:25:59.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a2b64e640 1 -- 172.21.15.27:0/948463590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 msgr2=0x7f4a10004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a2b64e640 1 --2- 172.21.15.27:0/948463590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a10004860 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4a1400c290 tx=0x7f4a10005830 comp rx=0 tx=0).stop 2024-03-06T07:25:59.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a29beb640 1 -- 172.21.15.27:0/948463590 reap_dead start 2024-03-06T07:25:59.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a2b64e640 1 -- 172.21.15.27:0/948463590 shutdown_connections 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a2b64e640 1 -- 172.21.15.27:0/948463590 >> 172.21.15.27:0/948463590 conn(0x7f4a240b36d0 msgr2=0x7f4a240b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a2b64e640 1 -- 172.21.15.27:0/948463590 shutdown_connections 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.842+0000 7f4a2b64e640 1 -- 172.21.15.27:0/948463590 wait complete. 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.844+0000 7f4a2b64e640 1 Processor -- start 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.844+0000 7f4a2b64e640 1 -- start start 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a2b64e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a240c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a2b64e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2415c120 0x7f4a240cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a2b64e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a240c9330 0x7f4a240c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a2b64e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2406c910 con 0x7f4a10004390 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a2b64e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2406c790 con 0x7f4a240c9330 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a2b64e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a240d0ab0 con 0x7f4a2415c120 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a293ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a240c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a293ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a240c5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48754/0 (socket says 172.21.15.27:48754) 2024-03-06T07:25:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a293ea640 1 -- 172.21.15.27:0/3675976608 learned_addr learned my addr 172.21.15.27:0/3675976608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a28be9640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2415c120 0x7f4a240cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a29beb640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a240c9330 0x7f4a240c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a293ea640 1 -- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2415c120 msgr2=0x7f4a240cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a293ea640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2415c120 0x7f4a240cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a293ea640 1 -- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a240c9330 msgr2=0x7f4a240c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a293ea640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a240c9330 0x7f4a240c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a293ea640 1 -- 172.21.15.27:0/3675976608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a240c5c10 con 0x7f4a10004390 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a293ea640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a240c5630 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4a20011aa0 tx=0x7f4a20011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.846+0000 7f4a19ffb640 1 -- 172.21.15.27:0/3675976608 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000fb80 con 0x7f4a10004390 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.847+0000 7f4a19ffb640 1 -- 172.21.15.27:0/3675976608 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2000fd20 con 0x7f4a10004390 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.847+0000 7f4a19ffb640 1 -- 172.21.15.27:0/3675976608 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20014640 con 0x7f4a10004390 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.847+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a240c5ef0 con 0x7f4a10004390 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.847+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a240c64b0 con 0x7f4a10004390 2024-03-06T07:25:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.848+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a24068700 con 0x7f4a10004390 2024-03-06T07:25:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.848+0000 7f4a19ffb640 1 -- 172.21.15.27:0/3675976608 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a20012040 con 0x7f4a10004390 2024-03-06T07:25:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.849+0000 7f4a19ffb640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a0806b500 0x7f4a0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.849+0000 7f4a19ffb640 1 -- 172.21.15.27:0/3675976608 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a2008a8f0 con 0x7f4a10004390 2024-03-06T07:25:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.849+0000 7f4a28be9640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a0806b500 0x7f4a0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.849+0000 7f4a28be9640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a0806b500 0x7f4a0806d9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4a14034000 tx=0x7f4a14009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:25:59.853+0000 7f4a19ffb640 1 -- 172.21.15.27:0/3675976608 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a20019030 con 0x7f4a10004390 2024-03-06T07:26:00.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.072+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a240cbab0 con 0x7f4a10004390 2024-03-06T07:26:00.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.073+0000 7f4a19ffb640 1 -- 172.21.15.27:0/3675976608 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f4a20059800 con 0x7f4a10004390 2024-03-06T07:26:00.076 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:00.076 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5324,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3885016831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3885016831},{"type":"v1","addr":"172.21.15.69:6836","nonce":3885016831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5327,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3510003111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3510003111},{"type":"v1","addr":"172.21.15.69:6837","nonce":3510003111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5692,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3053251134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3053251134},{"type":"v1","addr":"172.21.15.27:6835","nonce":3053251134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:23.445711+0000","modified":"2024-03-06T07:25:26.468573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6129":{"gid":6129,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6833/60877422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":60877422},{"type":"v1","addr":"172.21.15.27:6833","nonce":60877422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6129,"qdb_cluster":[6129]},"id":4}]} 2024-03-06T07:26:00.076 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-03-06T07:26:00.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a0806b500 msgr2=0x7f4a0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a0806b500 0x7f4a0806d9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4a14034000 tx=0x7f4a14009030 comp rx=0 tx=0).stop 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 msgr2=0x7f4a240c5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a240c5630 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4a20011aa0 tx=0x7f4a20011f60 comp rx=0 tx=0).stop 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 shutdown_connections 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a0806b500 0x7f4a0806d9c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a240c9330 0x7f4a240c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a2415c120 0x7f4a240cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 --2- 172.21.15.27:0/3675976608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a10004390 0x7f4a240c5630 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 >> 172.21.15.27:0/3675976608 conn(0x7f4a240b36d0 msgr2=0x7f4a2415dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 shutdown_connections 2024-03-06T07:26:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.075+0000 7f4a2b64e640 1 -- 172.21.15.27:0/3675976608 wait complete. 2024-03-06T07:26:00.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.194+0000 7f8a5f4a4640 1 Processor -- start 2024-03-06T07:26:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.194+0000 7f8a5f4a4640 1 -- start start 2024-03-06T07:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.195+0000 7f8a5f4a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a580b0e90 0x7f8a5806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.195+0000 7f8a5f4a4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a580ae9b0 0x7f8a580d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.195+0000 7f8a5f4a4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a580cee90 con 0x7f8a5806b5c0 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.195+0000 7f8a5f4a4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a580cfa00 con 0x7f8a580ae9b0 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.195+0000 7f8a5f4a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a580cfc50 con 0x7f8a580b0e90 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.195+0000 7f8a5da41640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a580ae9b0 0x7f8a580d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.195+0000 7f8a5da41640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a580ae9b0 0x7f8a580d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48768/0 (socket says 172.21.15.27:48768) 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.195+0000 7f8a5da41640 1 -- 172.21.15.27:0/2019999020 learned_addr learned my addr 172.21.15.27:0/2019999020 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.196+0000 7f8a5ca3f640 1 --2- 172.21.15.27:0/2019999020 >> v2:172.21.15.69:3300/0 conn(0x7f8a580b0e90 0x7f8a5806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.196+0000 7f8a5da41640 1 -- 172.21.15.27:0/2019999020 >> v2:172.21.15.69:3300/0 conn(0x7f8a580b0e90 msgr2=0x7f8a5806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.196+0000 7f8a5da41640 1 --2- 172.21.15.27:0/2019999020 >> v2:172.21.15.69:3300/0 conn(0x7f8a580b0e90 0x7f8a5806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.196+0000 7f8a5da41640 1 -- 172.21.15.27:0/2019999020 >> v1:172.21.15.69:6789/0 conn(0x7f8a5806b5c0 legacy=0x7f8a580b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.196+0000 7f8a5da41640 1 -- 172.21.15.27:0/2019999020 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a580cff30 con 0x7f8a580ae9b0 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.196+0000 7f8a5da41640 1 --2- 172.21.15.27:0/2019999020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 0x7f8a580d03a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f8a580b0e40 tx=0x7f8a5002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff7975f7008028c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a4ffff640 1 -- 172.21.15.27:0/2019999020 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5002ca80 con 0x7f8a580ae9b0 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a4ffff640 1 -- 172.21.15.27:0/2019999020 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5002cc20 con 0x7f8a580ae9b0 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a4ffff640 1 -- 172.21.15.27:0/2019999020 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a500343f0 con 0x7f8a580ae9b0 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/2019999020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 msgr2=0x7f8a580d03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a5f4a4640 1 --2- 172.21.15.27:0/2019999020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 0x7f8a580d03a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f8a580b0e40 tx=0x7f8a5002bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/2019999020 shutdown_connections 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a5f4a4640 1 --2- 172.21.15.27:0/2019999020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 0x7f8a580d03a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a5f4a4640 1 --2- 172.21.15.27:0/2019999020 >> v2:172.21.15.69:3300/0 conn(0x7f8a580b0e90 0x7f8a5806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/2019999020 >> 172.21.15.27:0/2019999020 conn(0x7f8a580b36d0 msgr2=0x7f8a580b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/2019999020 shutdown_connections 2024-03-06T07:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.197+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/2019999020 wait complete. 2024-03-06T07:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.199+0000 7f8a5f4a4640 1 Processor -- start 2024-03-06T07:26:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.199+0000 7f8a5f4a4640 1 -- start start 2024-03-06T07:26:00.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5f4a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a5806b5c0 0x7f8a581dfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5f4a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 0x7f8a581e1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5f4a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a580b0e90 0x7f8a581e1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5f4a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5806c910 con 0x7f8a580ae9b0 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5f4a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5806c790 con 0x7f8a580b0e90 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5f4a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a580aedc0 con 0x7f8a5806b5c0 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5ca3f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 0x7f8a581e1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5ca3f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 0x7f8a581e1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48784/0 (socket says 172.21.15.27:48784) 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5ca3f640 1 -- 172.21.15.27:0/1418027205 learned_addr learned my addr 172.21.15.27:0/1418027205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5da41640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a580b0e90 0x7f8a581e1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5d240640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a5806b5c0 0x7f8a581dfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5ca3f640 1 -- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a5806b5c0 msgr2=0x7f8a581dfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5ca3f640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a5806b5c0 0x7f8a581dfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5ca3f640 1 -- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a580b0e90 msgr2=0x7f8a581e1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5ca3f640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a580b0e90 0x7f8a581e1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5ca3f640 1 -- 172.21.15.27:0/1418027205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a581e7300 con 0x7f8a580ae9b0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.201+0000 7f8a5ca3f640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 0x7f8a581e1700 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400e8a0 tx=0x7f8a5400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.202+0000 7f8a4dffb640 1 -- 172.21.15.27:0/1418027205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54017070 con 0x7f8a580ae9b0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.202+0000 7f8a4dffb640 1 -- 172.21.15.27:0/1418027205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5400c940 con 0x7f8a580ae9b0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.202+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a581e03f0 con 0x7f8a580ae9b0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.202+0000 7f8a4dffb640 1 -- 172.21.15.27:0/1418027205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5400cd30 con 0x7f8a580ae9b0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.202+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a581f8060 con 0x7f8a580ae9b0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.203+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a58068700 con 0x7f8a580ae9b0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.203+0000 7f8a4dffb640 1 -- 172.21.15.27:0/1418027205 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a5401a070 con 0x7f8a580ae9b0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.204+0000 7f8a4dffb640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a3c06b4d0 0x7f8a3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.204+0000 7f8a4dffb640 1 -- 172.21.15.27:0/1418027205 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a54089f50 con 0x7f8a580ae9b0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.204+0000 7f8a5d240640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a3c06b4d0 0x7f8a3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.205+0000 7f8a5d240640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a3c06b4d0 0x7f8a3c06d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8a48001780 tx=0x7f8a48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.213+0000 7f8a4dffb640 1 -- 172.21.15.27:0/1418027205 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a581e03f0 con 0x7f8a580ae9b0 2024-03-06T07:26:00.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.432+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a581e6380 con 0x7f8a580ae9b0 2024-03-06T07:26:00.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.433+0000 7f8a4dffb640 1 -- 172.21.15.27:0/1418027205 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f8a581e7300 con 0x7f8a580ae9b0 2024-03-06T07:26:00.436 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:00.436 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5324,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3885016831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3885016831},{"type":"v1","addr":"172.21.15.69:6836","nonce":3885016831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5327,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3510003111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3510003111},{"type":"v1","addr":"172.21.15.69:6837","nonce":3510003111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5692,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3053251134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3053251134},{"type":"v1","addr":"172.21.15.27:6835","nonce":3053251134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:23.445711+0000","modified":"2024-03-06T07:25:26.468573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6129":{"gid":6129,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6833/60877422","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":60877422},{"type":"v1","addr":"172.21.15.27:6833","nonce":60877422}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6129,"qdb_cluster":[6129]},"id":4}]} 2024-03-06T07:26:00.436 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-03-06T07:26:00.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a3c06b4d0 msgr2=0x7f8a3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a3c06b4d0 0x7f8a3c06d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8a48001780 tx=0x7f8a48009040 comp rx=0 tx=0).stop 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 msgr2=0x7f8a581e1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 0x7f8a581e1700 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400e8a0 tx=0x7f8a5400ed60 comp rx=0 tx=0).stop 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 shutdown_connections 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a3c06b4d0 0x7f8a3c06d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a580b0e90 0x7f8a581e1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a580ae9b0 0x7f8a581e1700 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 --2- 172.21.15.27:0/1418027205 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a5806b5c0 0x7f8a581dfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 >> 172.21.15.27:0/1418027205 conn(0x7f8a580b36d0 msgr2=0x7f8a580d12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 shutdown_connections 2024-03-06T07:26:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.435+0000 7f8a5f4a4640 1 -- 172.21.15.27:0/1418027205 wait complete. 2024-03-06T07:26:00.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.554+0000 7f8b1d6ed640 1 Processor -- start 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.554+0000 7f8b1d6ed640 1 -- start start 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.554+0000 7f8b1d6ed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1815e030 0x7f8b1806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.554+0000 7f8b1d6ed640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b1806ef70 0x7f8b180b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.554+0000 7f8b1d6ed640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1815e410 con 0x7f8b180b2290 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.554+0000 7f8b1d6ed640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b18162700 con 0x7f8b1806ef70 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.554+0000 7f8b1d6ed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b18161af0 con 0x7f8b1815e030 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b177fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8b180b2290 0x7f8b180d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58300/0 (socket says 172.21.15.27:58300) 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b177fe640 1 -- 172.21.15.27:0/1416431000 learned_addr learned my addr 172.21.15.27:0/1416431000 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b16ffd640 1 --2- 172.21.15.27:0/1416431000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1815e030 0x7f8b1806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b15ffb640 1 -- 172.21.15.27:0/1416431000 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917306539 0 0) 0x7f8b10001490 con 0x7f8b180b2290 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b15ffb640 1 -- 172.21.15.27:0/1416431000 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b04003520 con 0x7f8b180b2290 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b167fc640 1 --2- 172.21.15.27:0/1416431000 >> v2:172.21.15.69:3300/0 conn(0x7f8b1806ef70 0x7f8b180b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b15ffb640 1 -- 172.21.15.27:0/1416431000 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2234339595 0 0) 0x7f8b04003520 con 0x7f8b180b2290 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b16ffd640 1 -- 172.21.15.27:0/1416431000 >> v2:172.21.15.69:3300/0 conn(0x7f8b1806ef70 msgr2=0x7f8b180b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b16ffd640 1 --2- 172.21.15.27:0/1416431000 >> v2:172.21.15.69:3300/0 conn(0x7f8b1806ef70 0x7f8b180b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b16ffd640 1 -- 172.21.15.27:0/1416431000 >> v1:172.21.15.27:6789/0 conn(0x7f8b180b2290 legacy=0x7f8b180d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.555+0000 7f8b16ffd640 1 -- 172.21.15.27:0/1416431000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18161dd0 con 0x7f8b1815e030 2024-03-06T07:26:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.556+0000 7f8b16ffd640 1 --2- 172.21.15.27:0/1416431000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1815e030 0x7f8b1806ea30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8b080097a0 tx=0x7f8b0802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4177d5373ec010a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.556+0000 7f8b15ffb640 1 -- 172.21.15.27:0/1416431000 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08037070 con 0x7f8b1815e030 2024-03-06T07:26:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.556+0000 7f8b15ffb640 1 -- 172.21.15.27:0/1416431000 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b08030510 con 0x7f8b1815e030 2024-03-06T07:26:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.556+0000 7f8b15ffb640 1 -- 172.21.15.27:0/1416431000 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b10001bc0 con 0x7f8b1815e030 2024-03-06T07:26:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.557+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1416431000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1815e030 msgr2=0x7f8b1806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.557+0000 7f8b1d6ed640 1 --2- 172.21.15.27:0/1416431000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1815e030 0x7f8b1806ea30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8b080097a0 tx=0x7f8b0802b790 comp rx=0 tx=0).stop 2024-03-06T07:26:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.557+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1416431000 shutdown_connections 2024-03-06T07:26:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.557+0000 7f8b1d6ed640 1 --2- 172.21.15.27:0/1416431000 >> v2:172.21.15.69:3300/0 conn(0x7f8b1806ef70 0x7f8b180b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.557+0000 7f8b1d6ed640 1 --2- 172.21.15.27:0/1416431000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1815e030 0x7f8b1806ea30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.557+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1416431000 >> 172.21.15.27:0/1416431000 conn(0x7f8b180b55e0 msgr2=0x7f8b180b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.557+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1416431000 shutdown_connections 2024-03-06T07:26:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.557+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1416431000 wait complete. 2024-03-06T07:26:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.558+0000 7f8b1d6ed640 1 Processor -- start 2024-03-06T07:26:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.558+0000 7f8b1d6ed640 1 -- start start 2024-03-06T07:26:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b1d6ed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1806ef70 0x7f8b181e87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b1d6ed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180b2290 0x7f8b1814e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b1d6ed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b1815e030 0x7f8b1813ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b1d6ed640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1806e220 con 0x7f8b180b2290 2024-03-06T07:26:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b1d6ed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b18162390 con 0x7f8b1815e030 2024-03-06T07:26:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b1d6ed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1806e3a0 con 0x7f8b1806ef70 2024-03-06T07:26:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b167fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180b2290 0x7f8b1814e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b167fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180b2290 0x7f8b1814e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48786/0 (socket says 172.21.15.27:48786) 2024-03-06T07:26:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b167fc640 1 -- 172.21.15.27:0/1263581912 learned_addr learned my addr 172.21.15.27:0/1263581912 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b177fe640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b1815e030 0x7f8b1813ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b16ffd640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1806ef70 0x7f8b181e87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b167fc640 1 -- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1806ef70 msgr2=0x7f8b181e87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b167fc640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1806ef70 0x7f8b181e87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b167fc640 1 -- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b1815e030 msgr2=0x7f8b1813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b167fc640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b1815e030 0x7f8b1813ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b167fc640 1 -- 172.21.15.27:0/1263581912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b181e7a00 con 0x7f8b180b2290 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.559+0000 7f8b167fc640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180b2290 0x7f8b1814e100 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000e9b0 tx=0x7f8b0000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.560+0000 7f8b0f7fe640 1 -- 172.21.15.27:0/1263581912 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00017070 con 0x7f8b180b2290 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.560+0000 7f8b0f7fe640 1 -- 172.21.15.27:0/1263581912 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0000c9a0 con 0x7f8b180b2290 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.560+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b181e7ce0 con 0x7f8b180b2290 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.560+0000 7f8b0f7fe640 1 -- 172.21.15.27:0/1263581912 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0000cd90 con 0x7f8b180b2290 2024-03-06T07:26:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.560+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b181ea0b0 con 0x7f8b180b2290 2024-03-06T07:26:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.561+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b18067ff0 con 0x7f8b180b2290 2024-03-06T07:26:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.562+0000 7f8b0f7fe640 1 -- 172.21.15.27:0/1263581912 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b0001a070 con 0x7f8b180b2290 2024-03-06T07:26:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.562+0000 7f8b0f7fe640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8af806b500 0x7f8af806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.562+0000 7f8b0f7fe640 1 -- 172.21.15.27:0/1263581912 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b0008a4e0 con 0x7f8b180b2290 2024-03-06T07:26:00.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.562+0000 7f8b16ffd640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8af806b500 0x7f8af806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.563+0000 7f8b16ffd640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8af806b500 0x7f8af806d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8b08034000 tx=0x7f8b0802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.567+0000 7f8b0f7fe640 1 -- 172.21.15.27:0/1263581912 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b00058d10 con 0x7f8b180b2290 2024-03-06T07:26:00.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.748+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b18158e90 con 0x7f8b180b2290 2024-03-06T07:26:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.749+0000 7f8b0f7fe640 1 -- 172.21.15.27:0/1263581912 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f8b0005cdb0 con 0x7f8b180b2290 2024-03-06T07:26:00.752 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:00.752 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:25:25.095713+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:21.134461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:22.434887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:00.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8af806b500 msgr2=0x7f8af806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8af806b500 0x7f8af806d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8b08034000 tx=0x7f8b0802f000 comp rx=0 tx=0).stop 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180b2290 msgr2=0x7f8b1814e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180b2290 0x7f8b1814e100 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000e9b0 tx=0x7f8b0000ee70 comp rx=0 tx=0).stop 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 shutdown_connections 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8af806b500 0x7f8af806d9c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b1815e030 0x7f8b1813ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180b2290 0x7f8b1814e100 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 --2- 172.21.15.27:0/1263581912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b1806ef70 0x7f8b181e87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 >> 172.21.15.27:0/1263581912 conn(0x7f8b180b55e0 msgr2=0x7f8b18159db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 shutdown_connections 2024-03-06T07:26:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.752+0000 7f8b1d6ed640 1 -- 172.21.15.27:0/1263581912 wait complete. 2024-03-06T07:26:00.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.877+0000 7f7d315b5640 1 Processor -- start 2024-03-06T07:26:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.877+0000 7f7d315b5640 1 -- start start 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d315b5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d2c0d18c0 0x7f7d2c071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d315b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c071a80 0x7f7d2c0b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d315b5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c0b6f20 con 0x7f7d2c15df70 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d315b5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2c071ef0 con 0x7f7d2c0d18c0 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d315b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0d3030 con 0x7f7d2c071a80 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d2b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c071a80 0x7f7d2c0b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d2a7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d2c0d18c0 0x7f7d2c071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d2b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c071a80 0x7f7d2c0b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:34282/0 (socket says 172.21.15.27:34282) 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d2affd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7d2c15df70 0x7f7d2c0d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:43270/0 (socket says 172.21.15.27:43270) 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d2a7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d2c0d18c0 0x7f7d2c071540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:57394/0 (socket says 172.21.15.27:57394) 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.878+0000 7f7d2b7fe640 1 -- 172.21.15.27:0/1738316561 learned_addr learned my addr 172.21.15.27:0/1738316561 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.879+0000 7f7d2b7fe640 1 -- 172.21.15.27:0/1738316561 >> v2:172.21.15.69:3300/0 conn(0x7f7d2c0d18c0 msgr2=0x7f7d2c071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.879+0000 7f7d29ffb640 1 -- 172.21.15.27:0/1738316561 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559910323 0 0) 0x7f7d2c0b6f20 con 0x7f7d2c15df70 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.879+0000 7f7d2b7fe640 1 --2- 172.21.15.27:0/1738316561 >> v2:172.21.15.69:3300/0 conn(0x7f7d2c0d18c0 0x7f7d2c071540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.879+0000 7f7d2b7fe640 1 -- 172.21.15.27:0/1738316561 >> v1:172.21.15.69:6789/0 conn(0x7f7d2c15df70 legacy=0x7f7d2c0d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.879+0000 7f7d2b7fe640 1 -- 172.21.15.27:0/1738316561 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0d3310 con 0x7f7d2c071a80 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.879+0000 7f7d2a7fc640 1 --2- 172.21.15.27:0/1738316561 >> v2:172.21.15.69:3300/0 conn(0x7f7d2c0d18c0 0x7f7d2c071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d2b7fe640 1 --2- 172.21.15.27:0/1738316561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c071a80 0x7f7d2c0b2700 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009e50 tx=0x7f7d1c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca8ed349a36bd50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d29ffb640 1 -- 172.21.15.27:0/1738316561 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c02cb90 con 0x7f7d2c071a80 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d29ffb640 1 -- 172.21.15.27:0/1738316561 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c02cd30 con 0x7f7d2c071a80 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d29ffb640 1 -- 172.21.15.27:0/1738316561 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c034610 con 0x7f7d2c071a80 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d315b5640 1 -- 172.21.15.27:0/1738316561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c071a80 msgr2=0x7f7d2c0b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d315b5640 1 --2- 172.21.15.27:0/1738316561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c071a80 0x7f7d2c0b2700 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009e50 tx=0x7f7d1c02bff0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d315b5640 1 -- 172.21.15.27:0/1738316561 shutdown_connections 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d315b5640 1 --2- 172.21.15.27:0/1738316561 >> v2:172.21.15.69:3300/0 conn(0x7f7d2c0d18c0 0x7f7d2c071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d315b5640 1 --2- 172.21.15.27:0/1738316561 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c071a80 0x7f7d2c0b2700 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d315b5640 1 -- 172.21.15.27:0/1738316561 >> 172.21.15.27:0/1738316561 conn(0x7f7d2c0d3cf0 msgr2=0x7f7d2c0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d315b5640 1 -- 172.21.15.27:0/1738316561 shutdown_connections 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.880+0000 7f7d315b5640 1 -- 172.21.15.27:0/1738316561 wait complete. 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.882+0000 7f7d315b5640 1 Processor -- start 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.882+0000 7f7d315b5640 1 -- start start 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.882+0000 7f7d315b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d2c071a80 0x7f7d2c1f4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.882+0000 7f7d315b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c0d18c0 0x7f7d2c0ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.882+0000 7f7d315b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d2c15df70 0x7f7d2c1f0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.882+0000 7f7d315b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c15e350 con 0x7f7d2c071a80 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.882+0000 7f7d315b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0d38d0 con 0x7f7d2c15df70 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.882+0000 7f7d315b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c15e4d0 con 0x7f7d2c0d18c0 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d2c071a80 0x7f7d2c1f4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d2c071a80 0x7f7d2c1f4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48794/0 (socket says 172.21.15.27:48794) 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2affd640 1 -- 172.21.15.27:0/1903224714 learned_addr learned my addr 172.21.15.27:0/1903224714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2a7fc640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c0d18c0 0x7f7d2c0ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2b7fe640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d2c15df70 0x7f7d2c1f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2affd640 1 -- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c0d18c0 msgr2=0x7f7d2c0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2affd640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c0d18c0 0x7f7d2c0ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2affd640 1 -- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d2c15df70 msgr2=0x7f7d2c1f0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2affd640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d2c15df70 0x7f7d2c1f0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2affd640 1 -- 172.21.15.27:0/1903224714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0b25c0 con 0x7f7d2c071a80 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d2affd640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d2c071a80 0x7f7d2c1f4a80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400c430 tx=0x7f7d2400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d237fe640 1 -- 172.21.15.27:0/1903224714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d240074e0 con 0x7f7d2c071a80 2024-03-06T07:26:00.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d237fe640 1 -- 172.21.15.27:0/1903224714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d240047e0 con 0x7f7d2c071a80 2024-03-06T07:26:00.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c0c8cf0 con 0x7f7d2c071a80 2024-03-06T07:26:00.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d237fe640 1 -- 172.21.15.27:0/1903224714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24004bd0 con 0x7f7d2c071a80 2024-03-06T07:26:00.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.883+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c0c92b0 con 0x7f7d2c071a80 2024-03-06T07:26:00.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.885+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c067ff0 con 0x7f7d2c071a80 2024-03-06T07:26:00.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.885+0000 7f7d237fe640 1 -- 172.21.15.27:0/1903224714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d24011070 con 0x7f7d2c071a80 2024-03-06T07:26:00.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.886+0000 7f7d237fe640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d0c06b4d0 0x7f7d0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.886+0000 7f7d237fe640 1 -- 172.21.15.27:0/1903224714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d24089d60 con 0x7f7d2c071a80 2024-03-06T07:26:00.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.886+0000 7f7d2a7fc640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d0c06b4d0 0x7f7d0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.887+0000 7f7d2a7fc640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d0c06b4d0 0x7f7d0c06d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009560 tx=0x7f7d14009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:00.891+0000 7f7d237fe640 1 -- 172.21.15.27:0/1903224714 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d24058510 con 0x7f7d2c071a80 2024-03-06T07:26:01.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.113+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7d2c0b9a40 con 0x7f7d2c071a80 2024-03-06T07:26:01.116 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:26:01.114+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:01.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.115+0000 7f7d237fe640 1 -- 172.21.15.27:0/1903224714 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7d2405c5b0 con 0x7f7d2c071a80 2024-03-06T07:26:01.118 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:01.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d0c06b4d0 msgr2=0x7f7d0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d0c06b4d0 0x7f7d0c06d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009560 tx=0x7f7d14009190 comp rx=0 tx=0).stop 2024-03-06T07:26:01.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d2c071a80 msgr2=0x7f7d2c1f4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d2c071a80 0x7f7d2c1f4a80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400c430 tx=0x7f7d2400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 shutdown_connections 2024-03-06T07:26:01.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d0c06b4d0 0x7f7d0c06d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d2c15df70 0x7f7d2c1f0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d2c0d18c0 0x7f7d2c0ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 --2- 172.21.15.27:0/1903224714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d2c071a80 0x7f7d2c1f4a80 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.118+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 >> 172.21.15.27:0/1903224714 conn(0x7f7d2c0d3cf0 msgr2=0x7f7d2c15d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.119+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 shutdown_connections 2024-03-06T07:26:01.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.119+0000 7f7d315b5640 1 -- 172.21.15.27:0/1903224714 wait complete. 2024-03-06T07:26:01.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:01.167 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:26:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.244+0000 7ff119e0e640 1 Processor -- start 2024-03-06T07:26:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.245+0000 7ff119e0e640 1 -- start start 2024-03-06T07:26:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.245+0000 7ff119e0e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff11415dd90 0x7ff1140ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.245+0000 7ff119e0e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1140d29d0 0x7ff1140d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.245+0000 7ff119e0e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1140d3740 con 0x7ff1140d4720 2024-03-06T07:26:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.245+0000 7ff119e0e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1140d3400 con 0x7ff1140d29d0 2024-03-06T07:26:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.245+0000 7ff119e0e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1140d3120 con 0x7ff11415dd90 2024-03-06T07:26:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.245+0000 7ff112ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1140d29d0 0x7ff1140d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.245+0000 7ff112ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1140d29d0 0x7ff1140d41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48814/0 (socket says 172.21.15.27:48814) 2024-03-06T07:26:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.245+0000 7ff112ffd640 1 -- 172.21.15.27:0/901912179 learned_addr learned my addr 172.21.15.27:0/901912179 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.246+0000 7ff1137fe640 1 --2- 172.21.15.27:0/901912179 >> v2:172.21.15.69:3300/0 conn(0x7ff11415dd90 0x7ff1140ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.246+0000 7ff112ffd640 1 -- 172.21.15.27:0/901912179 >> v2:172.21.15.69:3300/0 conn(0x7ff11415dd90 msgr2=0x7ff1140ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.246+0000 7ff112ffd640 1 --2- 172.21.15.27:0/901912179 >> v2:172.21.15.69:3300/0 conn(0x7ff11415dd90 0x7ff1140ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.246+0000 7ff112ffd640 1 -- 172.21.15.27:0/901912179 >> v1:172.21.15.69:6789/0 conn(0x7ff1140d4720 legacy=0x7ff1140b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.246+0000 7ff112ffd640 1 -- 172.21.15.27:0/901912179 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1140d39e0 con 0x7ff1140d29d0 2024-03-06T07:26:01.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.246+0000 7ff112ffd640 1 --2- 172.21.15.27:0/901912179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d29d0 0x7ff1140d41e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff1140d1f90 tx=0x7ff10802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b36cacd279f8e89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff1127fc640 1 -- 172.21.15.27:0/901912179 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10802f850 con 0x7ff1140d29d0 2024-03-06T07:26:01.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff1127fc640 1 -- 172.21.15.27:0/901912179 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10802f9f0 con 0x7ff1140d29d0 2024-03-06T07:26:01.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff1127fc640 1 -- 172.21.15.27:0/901912179 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10802fdc0 con 0x7ff1140d29d0 2024-03-06T07:26:01.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff119e0e640 1 -- 172.21.15.27:0/901912179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d29d0 msgr2=0x7ff1140d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff119e0e640 1 --2- 172.21.15.27:0/901912179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d29d0 0x7ff1140d41e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff1140d1f90 tx=0x7ff10802ba30 comp rx=0 tx=0).stop 2024-03-06T07:26:01.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff119e0e640 1 -- 172.21.15.27:0/901912179 shutdown_connections 2024-03-06T07:26:01.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff119e0e640 1 --2- 172.21.15.27:0/901912179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d29d0 0x7ff1140d41e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff119e0e640 1 --2- 172.21.15.27:0/901912179 >> v2:172.21.15.69:3300/0 conn(0x7ff11415dd90 0x7ff1140ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff119e0e640 1 -- 172.21.15.27:0/901912179 >> 172.21.15.27:0/901912179 conn(0x7ff114071960 msgr2=0x7ff114071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff119e0e640 1 -- 172.21.15.27:0/901912179 shutdown_connections 2024-03-06T07:26:01.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.247+0000 7ff119e0e640 1 -- 172.21.15.27:0/901912179 wait complete. 2024-03-06T07:26:01.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.248+0000 7ff119e0e640 1 Processor -- start 2024-03-06T07:26:01.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.249+0000 7ff119e0e640 1 -- start start 2024-03-06T07:26:01.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.249+0000 7ff119e0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1140d29d0 0x7ff114146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.249+0000 7ff119e0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d4720 0x7ff11415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.249+0000 7ff119e0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff11415dd90 0x7ff11415e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.249+0000 7ff119e0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1140d2df0 con 0x7ff1140d4720 2024-03-06T07:26:01.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.249+0000 7ff119e0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11415e170 con 0x7ff1140d29d0 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.249+0000 7ff119e0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1140b7420 con 0x7ff11415dd90 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.249+0000 7ff112ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d4720 0x7ff11415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff112ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d4720 0x7ff11415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48820/0 (socket says 172.21.15.27:48820) 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff112ffd640 1 -- 172.21.15.27:0/56790143 learned_addr learned my addr 172.21.15.27:0/56790143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff1137fe640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1140d29d0 0x7ff114146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff113fff640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff11415dd90 0x7ff11415e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff112ffd640 1 -- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff11415dd90 msgr2=0x7ff11415e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff112ffd640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff11415dd90 0x7ff11415e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff112ffd640 1 -- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1140d29d0 msgr2=0x7ff114146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff112ffd640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1140d29d0 0x7ff114146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff112ffd640 1 -- 172.21.15.27:0/56790143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1140ccc80 con 0x7ff1140d4720 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff112ffd640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d4720 0x7ff11415e2f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff10802ba70 tx=0x7ff108004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff0fbfff640 1 -- 172.21.15.27:0/56790143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108037d10 con 0x7ff1140d4720 2024-03-06T07:26:01.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff0fbfff640 1 -- 172.21.15.27:0/56790143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff108046070 con 0x7ff1140d4720 2024-03-06T07:26:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1140ccf60 con 0x7ff1140d4720 2024-03-06T07:26:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff0fbfff640 1 -- 172.21.15.27:0/56790143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108041640 con 0x7ff1140d4720 2024-03-06T07:26:01.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.250+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1140cd520 con 0x7ff1140d4720 2024-03-06T07:26:01.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.251+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff114072a30 con 0x7ff1140d4720 2024-03-06T07:26:01.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.252+0000 7ff0fbfff640 1 -- 172.21.15.27:0/56790143 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff108040070 con 0x7ff1140d4720 2024-03-06T07:26:01.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.252+0000 7ff0fbfff640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff10006b4d0 0x7ff10006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.252+0000 7ff0fbfff640 1 -- 172.21.15.27:0/56790143 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff1080ac950 con 0x7ff1140d4720 2024-03-06T07:26:01.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.252+0000 7ff1137fe640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff10006b4d0 0x7ff10006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.253+0000 7ff1137fe640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff10006b4d0 0x7ff10006d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff104009a30 tx=0x7ff104008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.257+0000 7ff0fbfff640 1 -- 172.21.15.27:0/56790143 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff108033030 con 0x7ff1140d4720 2024-03-06T07:26:01.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:01.478+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff1140b4ac0 con 0x7ff1140d4720 2024-03-06T07:26:02.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.140+0000 7ff0fbfff640 1 -- 172.21.15.27:0/56790143 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff10807b770 con 0x7ff1140d4720 2024-03-06T07:26:02.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff10006b4d0 msgr2=0x7ff10006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff10006b4d0 0x7ff10006d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff104009a30 tx=0x7ff104008040 comp rx=0 tx=0).stop 2024-03-06T07:26:02.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d4720 msgr2=0x7ff11415e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d4720 0x7ff11415e2f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff10802ba70 tx=0x7ff108004730 comp rx=0 tx=0).stop 2024-03-06T07:26:02.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 shutdown_connections 2024-03-06T07:26:02.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff10006b4d0 0x7ff10006d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff11415dd90 0x7ff11415e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1140d4720 0x7ff11415e2f0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 --2- 172.21.15.27:0/56790143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1140d29d0 0x7ff114146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.142+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 >> 172.21.15.27:0/56790143 conn(0x7ff114071960 msgr2=0x7ff1140b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.143+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 shutdown_connections 2024-03-06T07:26:02.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.143+0000 7ff119e0e640 1 -- 172.21.15.27:0/56790143 wait complete. 2024-03-06T07:26:02.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:02.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.313+0000 7f227cd8a640 1 Processor -- start 2024-03-06T07:26:02.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.314+0000 7f227cd8a640 1 -- start start 2024-03-06T07:26:02.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.314+0000 7f227cd8a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22780ceea0 0x7f22780d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.314+0000 7f227cd8a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22780d3740 con 0x7f227815dd90 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.314+0000 7f227cd8a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22780d3120 con 0x7f22780b6d10 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.314+0000 7f227cd8a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22780d3400 con 0x7f22780ceea0 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.315+0000 7f226ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22780ceea0 0x7f22780d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.315+0000 7f227659c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f227815dd90 0x7f22780d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58304/0 (socket says 172.21.15.27:58304) 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.315+0000 7f227659c640 1 -- 172.21.15.27:0/3019121761 learned_addr learned my addr 172.21.15.27:0/3019121761 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.315+0000 7f226ffff640 1 -- 172.21.15.27:0/3019121761 >> v1:172.21.15.69:6789/0 conn(0x7f22780b6d10 legacy=0x7f22780d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.315+0000 7f226ffff640 1 -- 172.21.15.27:0/3019121761 >> v1:172.21.15.27:6789/0 conn(0x7f227815dd90 legacy=0x7f22780d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.315+0000 7f2275d9b640 1 -- 172.21.15.27:0/3019121761 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468925229 0 0) 0x7f22780d3740 con 0x7f227815dd90 2024-03-06T07:26:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.315+0000 7f226ffff640 1 -- 172.21.15.27:0/3019121761 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22780d39e0 con 0x7f22780ceea0 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f226ffff640 1 --2- 172.21.15.27:0/3019121761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22780ceea0 0x7f22780d29d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f22600097a0 tx=0x7f226002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad086b58d3605168 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f2275d9b640 1 -- 172.21.15.27:0/3019121761 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226002f810 con 0x7f22780ceea0 2024-03-06T07:26:02.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f2275d9b640 1 -- 172.21.15.27:0/3019121761 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226002f9b0 con 0x7f22780ceea0 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f2275d9b640 1 -- 172.21.15.27:0/3019121761 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226002fd80 con 0x7f22780ceea0 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f227cd8a640 1 -- 172.21.15.27:0/3019121761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22780ceea0 msgr2=0x7f22780d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f227cd8a640 1 --2- 172.21.15.27:0/3019121761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22780ceea0 0x7f22780d29d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f22600097a0 tx=0x7f226002b9f0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f227cd8a640 1 -- 172.21.15.27:0/3019121761 shutdown_connections 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f227cd8a640 1 --2- 172.21.15.27:0/3019121761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22780ceea0 0x7f22780d29d0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f227cd8a640 1 -- 172.21.15.27:0/3019121761 >> 172.21.15.27:0/3019121761 conn(0x7f2278071960 msgr2=0x7f2278071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f227cd8a640 1 -- 172.21.15.27:0/3019121761 shutdown_connections 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.316+0000 7f227cd8a640 1 -- 172.21.15.27:0/3019121761 wait complete. 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.318+0000 7f227cd8a640 1 Processor -- start 2024-03-06T07:26:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.318+0000 7f227cd8a640 1 -- start start 2024-03-06T07:26:02.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.318+0000 7f227cd8a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22780b6d10 0x7f2278143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.318+0000 7f227cd8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22780ceea0 0x7f22780c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.318+0000 7f227cd8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f227815dd90 0x7f22780c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.318+0000 7f227cd8a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22780d48f0 con 0x7f227815dd90 2024-03-06T07:26:02.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.318+0000 7f227cd8a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227815e170 con 0x7f22780b6d10 2024-03-06T07:26:02.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.318+0000 7f227cd8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22780d4a70 con 0x7f22780ceea0 2024-03-06T07:26:02.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f2276d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f227815dd90 0x7f22780c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f2276d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f227815dd90 0x7f22780c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48838/0 (socket says 172.21.15.27:48838) 2024-03-06T07:26:02.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f2276d9d640 1 -- 172.21.15.27:0/1357985500 learned_addr learned my addr 172.21.15.27:0/1357985500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:02.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f226ffff640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22780ceea0 0x7f22780c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f227659c640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22780b6d10 0x7f2278143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f2276d9d640 1 -- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22780ceea0 msgr2=0x7f22780c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f2276d9d640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22780ceea0 0x7f22780c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f2276d9d640 1 -- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22780b6d10 msgr2=0x7f2278143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f2276d9d640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22780b6d10 0x7f2278143b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f2276d9d640 1 -- 172.21.15.27:0/1357985500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22781e9260 con 0x7f227815dd90 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f227659c640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22780b6d10 0x7f2278143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f2276d9d640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f227815dd90 0x7f22780c96c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f227000c7a0 tx=0x7f227000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f226effd640 1 -- 172.21.15.27:0/1357985500 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2270003b70 con 0x7f227815dd90 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f226effd640 1 -- 172.21.15.27:0/1357985500 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2270003d30 con 0x7f227815dd90 2024-03-06T07:26:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22781e9500 con 0x7f227815dd90 2024-03-06T07:26:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f226effd640 1 -- 172.21.15.27:0/1357985500 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227000aa90 con 0x7f227815dd90 2024-03-06T07:26:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.319+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22780c7c00 con 0x7f227815dd90 2024-03-06T07:26:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.320+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2278072a30 con 0x7f227815dd90 2024-03-06T07:26:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.321+0000 7f226effd640 1 -- 172.21.15.27:0/1357985500 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f227000ac30 con 0x7f227815dd90 2024-03-06T07:26:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.321+0000 7f226effd640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f225806b500 0x7f225806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.321+0000 7f226effd640 1 -- 172.21.15.27:0/1357985500 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f227008a020 con 0x7f227815dd90 2024-03-06T07:26:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.322+0000 7f227659c640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f225806b500 0x7f225806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.322+0000 7f227659c640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f225806b500 0x7f225806d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2268007fd0 tx=0x7f2268007dc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.326+0000 7f226effd640 1 -- 172.21.15.27:0/1357985500 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2270058ed0 con 0x7f227815dd90 2024-03-06T07:26:02.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:02.506+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f22780b4c80 con 0x7f227815dd90 2024-03-06T07:26:03.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.146+0000 7f226effd640 1 -- 172.21.15.27:0/1357985500 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v60) v1 ==== 181+0+0 (secure 0 0 0) 0x7f22700587d0 con 0x7f227815dd90 2024-03-06T07:26:03.148 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f225806b500 msgr2=0x7f225806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f225806b500 0x7f225806d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2268007fd0 tx=0x7f2268007dc0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f227815dd90 msgr2=0x7f22780c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f227815dd90 0x7f22780c96c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f227000c7a0 tx=0x7f227000cc60 comp rx=0 tx=0).stop 2024-03-06T07:26:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 shutdown_connections 2024-03-06T07:26:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f225806b500 0x7f225806d9c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f227815dd90 0x7f22780c96c0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22780ceea0 0x7f22780c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 --2- 172.21.15.27:0/1357985500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22780b6d10 0x7f2278143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.149+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 >> 172.21.15.27:0/1357985500 conn(0x7f2278071960 msgr2=0x7f22780b71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.151+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 shutdown_connections 2024-03-06T07:26:03.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.151+0000 7f227cd8a640 1 -- 172.21.15.27:0/1357985500 wait complete. 2024-03-06T07:26:03.167 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:03.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.279+0000 7fda25db6640 1 Processor -- start 2024-03-06T07:26:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.279+0000 7fda25db6640 1 -- start start 2024-03-06T07:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda25db6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda2015bb50 0x7fda200cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda25db6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fda200d00e0 0x7fda200d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda25db6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200d0a30 con 0x7fda2015bf30 2024-03-06T07:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda25db6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2006df80 con 0x7fda2015bb50 2024-03-06T07:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda25db6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda200b32a0 con 0x7fda200d00e0 2024-03-06T07:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda1f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda2015bb50 0x7fda200cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda1effd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fda2015bf30 0x7fda2006d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58320/0 (socket says 172.21.15.27:58320) 2024-03-06T07:26:03.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda1effd640 1 -- 172.21.15.27:0/1373619176 learned_addr learned my addr 172.21.15.27:0/1373619176 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda1ffff640 1 --2- 172.21.15.27:0/1373619176 >> v2:172.21.15.69:3300/0 conn(0x7fda200d00e0 0x7fda200d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda1f7fe640 1 -- 172.21.15.27:0/1373619176 >> v2:172.21.15.69:3300/0 conn(0x7fda200d00e0 msgr2=0x7fda200d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda1f7fe640 1 --2- 172.21.15.27:0/1373619176 >> v2:172.21.15.69:3300/0 conn(0x7fda200d00e0 0x7fda200d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda1f7fe640 1 -- 172.21.15.27:0/1373619176 >> v1:172.21.15.27:6789/0 conn(0x7fda2015bf30 legacy=0x7fda2006d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda1e7fc640 1 -- 172.21.15.27:0/1373619176 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896886766 0 0) 0x7fda14001110 con 0x7fda2015bf30 2024-03-06T07:26:03.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.280+0000 7fda1f7fe640 1 -- 172.21.15.27:0/1373619176 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200b4d30 con 0x7fda2015bb50 2024-03-06T07:26:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.281+0000 7fda1f7fe640 1 --2- 172.21.15.27:0/1373619176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bb50 0x7fda200cf2f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fda2006d8d0 tx=0x7fda1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8711d7c8ac964f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.281+0000 7fda1e7fc640 1 -- 172.21.15.27:0/1373619176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10035020 con 0x7fda2015bb50 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.281+0000 7fda1e7fc640 1 -- 172.21.15.27:0/1373619176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda10031830 con 0x7fda2015bb50 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.281+0000 7fda1e7fc640 1 -- 172.21.15.27:0/1373619176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10031c00 con 0x7fda2015bb50 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.282+0000 7fda25db6640 1 -- 172.21.15.27:0/1373619176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bb50 msgr2=0x7fda200cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.282+0000 7fda25db6640 1 --2- 172.21.15.27:0/1373619176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bb50 0x7fda200cf2f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fda2006d8d0 tx=0x7fda1002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.282+0000 7fda25db6640 1 -- 172.21.15.27:0/1373619176 shutdown_connections 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.282+0000 7fda25db6640 1 --2- 172.21.15.27:0/1373619176 >> v2:172.21.15.69:3300/0 conn(0x7fda200d00e0 0x7fda200d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.282+0000 7fda25db6640 1 --2- 172.21.15.27:0/1373619176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bb50 0x7fda200cf2f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.282+0000 7fda25db6640 1 -- 172.21.15.27:0/1373619176 >> 172.21.15.27:0/1373619176 conn(0x7fda2006af20 msgr2=0x7fda2006b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.282+0000 7fda25db6640 1 -- 172.21.15.27:0/1373619176 shutdown_connections 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.282+0000 7fda25db6640 1 -- 172.21.15.27:0/1373619176 wait complete. 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.283+0000 7fda25db6640 1 Processor -- start 2024-03-06T07:26:03.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.283+0000 7fda25db6640 1 -- start start 2024-03-06T07:26:03.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda25db6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda200d00e0 0x7fda201515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda25db6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2015bb50 0x7fda20152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda25db6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bf30 0x7fda20154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda25db6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200b3820 con 0x7fda2015bf30 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda25db6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200b3540 con 0x7fda2015bb50 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda25db6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200acfa0 con 0x7fda200d00e0 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bf30 0x7fda20154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bf30 0x7fda20154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48862/0 (socket says 172.21.15.27:48862) 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1ffff640 1 -- 172.21.15.27:0/2741755612 learned_addr learned my addr 172.21.15.27:0/2741755612 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1effd640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2015bb50 0x7fda20152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1f7fe640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda200d00e0 0x7fda201515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1ffff640 1 -- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda200d00e0 msgr2=0x7fda201515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1ffff640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda200d00e0 0x7fda201515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1ffff640 1 -- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2015bb50 msgr2=0x7fda20152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1ffff640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2015bb50 0x7fda20152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.284+0000 7fda1ffff640 1 -- 172.21.15.27:0/2741755612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20152b40 con 0x7fda2015bf30 2024-03-06T07:26:03.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.285+0000 7fda1ffff640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bf30 0x7fda20154830 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fda18007e40 tx=0x7fda1800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.285+0000 7fda07fff640 1 -- 172.21.15.27:0/2741755612 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18014620 con 0x7fda2015bf30 2024-03-06T07:26:03.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.285+0000 7fda07fff640 1 -- 172.21.15.27:0/2741755612 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda18019070 con 0x7fda2015bf30 2024-03-06T07:26:03.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.285+0000 7fda07fff640 1 -- 172.21.15.27:0/2741755612 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18014a20 con 0x7fda2015bf30 2024-03-06T07:26:03.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.285+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda201533a0 con 0x7fda2015bf30 2024-03-06T07:26:03.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.285+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda201530c0 con 0x7fda2015bf30 2024-03-06T07:26:03.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.286+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2004f410 con 0x7fda2015bf30 2024-03-06T07:26:03.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.286+0000 7fda07fff640 1 -- 172.21.15.27:0/2741755612 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda18014bc0 con 0x7fda2015bf30 2024-03-06T07:26:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.287+0000 7fda07fff640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0c06b4d0 0x7fda0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.287+0000 7fda07fff640 1 -- 172.21.15.27:0/2741755612 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fda1808a880 con 0x7fda2015bf30 2024-03-06T07:26:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.287+0000 7fda1f7fe640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0c06b4d0 0x7fda0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.288+0000 7fda1f7fe640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0c06b4d0 0x7fda0c06d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fda10036000 tx=0x7fda10008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.291+0000 7fda07fff640 1 -- 172.21.15.27:0/2741755612 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda180598f0 con 0x7fda2015bf30 2024-03-06T07:26:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:03.473+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fda200af540 con 0x7fda2015bf30 2024-03-06T07:26:04.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.151+0000 7fda07fff640 1 -- 172.21.15.27:0/2741755612 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v61) v1 ==== 169+0+0 (secure 0 0 0) 0x7fda180591f0 con 0x7fda2015bf30 2024-03-06T07:26:04.154 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.158+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0c06b4d0 msgr2=0x7fda0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.158+0000 7fda25db6640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0c06b4d0 0x7fda0c06d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fda10036000 tx=0x7fda10008030 comp rx=0 tx=0).stop 2024-03-06T07:26:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.158+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bf30 msgr2=0x7fda20154830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.158+0000 7fda25db6640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bf30 0x7fda20154830 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fda18007e40 tx=0x7fda1800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.159+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 shutdown_connections 2024-03-06T07:26:04.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.159+0000 7fda25db6640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0c06b4d0 0x7fda0c06d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.159+0000 7fda25db6640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda2015bf30 0x7fda20154830 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.159+0000 7fda25db6640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2015bb50 0x7fda20152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.159+0000 7fda25db6640 1 --2- 172.21.15.27:0/2741755612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda200d00e0 0x7fda201515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.159+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 >> 172.21.15.27:0/2741755612 conn(0x7fda2006af20 msgr2=0x7fda2006b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.160+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 shutdown_connections 2024-03-06T07:26:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.160+0000 7fda25db6640 1 -- 172.21.15.27:0/2741755612 wait complete. 2024-03-06T07:26:04.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.298+0000 7f0d518eb640 1 Processor -- start 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.298+0000 7f0d518eb640 1 -- start start 2024-03-06T07:26:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d518eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d4c0b2cd0 0x7f0d4c0b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d518eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d4c06b2c0 0x7f0d4c06b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d518eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c161a40 con 0x7f0d4c0cdcd0 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d518eb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4c0d11e0 con 0x7f0d4c0b2cd0 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d518eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4c0b2a20 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d43fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d4c0b2cd0 0x7f0d4c0b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d43fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d4c0b2cd0 0x7f0d4c0b1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43314/0 (socket says 172.21.15.27:43314) 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d43fff640 1 -- 172.21.15.27:0/1389345048 learned_addr learned my addr 172.21.15.27:0/1389345048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d4b7fe640 1 --2- 172.21.15.27:0/1389345048 >> v2:172.21.15.69:3300/0 conn(0x7f0d4c06b2c0 0x7f0d4c06b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d43fff640 1 -- 172.21.15.27:0/1389345048 >> v2:172.21.15.69:3300/0 conn(0x7f0d4c06b2c0 msgr2=0x7f0d4c06b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d43fff640 1 --2- 172.21.15.27:0/1389345048 >> v2:172.21.15.69:3300/0 conn(0x7f0d4c06b2c0 0x7f0d4c06b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d43fff640 1 -- 172.21.15.27:0/1389345048 >> v1:172.21.15.69:6789/0 conn(0x7f0d4c0cdcd0 legacy=0x7f0d4c0b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.299+0000 7f0d43fff640 1 -- 172.21.15.27:0/1389345048 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c0d0940 con 0x7f0d4c0b2cd0 2024-03-06T07:26:04.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.300+0000 7f0d43fff640 1 --2- 172.21.15.27:0/1389345048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c0b2cd0 0x7f0d4c0b1c60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0b3570 tx=0x7f0d3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecda1f94155fd78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.300+0000 7f0d4a7fc640 1 -- 172.21.15.27:0/1389345048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d34035020 con 0x7f0d4c0b2cd0 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.300+0000 7f0d4a7fc640 1 -- 172.21.15.27:0/1389345048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d34031870 con 0x7f0d4c0b2cd0 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.300+0000 7f0d4a7fc640 1 -- 172.21.15.27:0/1389345048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d34031c40 con 0x7f0d4c0b2cd0 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.301+0000 7f0d518eb640 1 -- 172.21.15.27:0/1389345048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c0b2cd0 msgr2=0x7f0d4c0b1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.301+0000 7f0d518eb640 1 --2- 172.21.15.27:0/1389345048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c0b2cd0 0x7f0d4c0b1c60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0b3570 tx=0x7f0d3402b910 comp rx=0 tx=0).stop 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.301+0000 7f0d518eb640 1 -- 172.21.15.27:0/1389345048 shutdown_connections 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.301+0000 7f0d518eb640 1 --2- 172.21.15.27:0/1389345048 >> v2:172.21.15.69:3300/0 conn(0x7f0d4c06b2c0 0x7f0d4c06b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.301+0000 7f0d518eb640 1 --2- 172.21.15.27:0/1389345048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c0b2cd0 0x7f0d4c0b1c60 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.301+0000 7f0d518eb640 1 -- 172.21.15.27:0/1389345048 >> 172.21.15.27:0/1389345048 conn(0x7f0d4c0b3700 msgr2=0x7f0d4c0b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.301+0000 7f0d518eb640 1 -- 172.21.15.27:0/1389345048 shutdown_connections 2024-03-06T07:26:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.301+0000 7f0d518eb640 1 -- 172.21.15.27:0/1389345048 wait complete. 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d518eb640 1 Processor -- start 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d518eb640 1 -- start start 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d518eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c06b2c0 0x7f0d4c157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d518eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d4c0b2cd0 0x7f0d4c0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d518eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4c0cdcd0 0x7f0d4c0ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d518eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0cc1c0 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d518eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0d0f00 con 0x7f0d4c0cdcd0 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d518eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0cc340 con 0x7f0d4c0b2cd0 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d4affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c06b2c0 0x7f0d4c157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d4affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c06b2c0 0x7f0d4c157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43326/0 (socket says 172.21.15.27:43326) 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.303+0000 7f0d4affd640 1 -- 172.21.15.27:0/3985917114 learned_addr learned my addr 172.21.15.27:0/3985917114 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d43fff640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d4c0b2cd0 0x7f0d4c0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d4b7fe640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4c0cdcd0 0x7f0d4c0ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d4affd640 1 -- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d4c0b2cd0 msgr2=0x7f0d4c0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d4affd640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d4c0b2cd0 0x7f0d4c0ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d4affd640 1 -- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4c0cdcd0 msgr2=0x7f0d4c0ca980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d4affd640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4c0cdcd0 0x7f0d4c0ca980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d4affd640 1 -- 172.21.15.27:0/3985917114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c0c5bd0 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d4affd640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c06b2c0 0x7f0d4c157380 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c3c0 tx=0x7f0d3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d48ff9640 1 -- 172.21.15.27:0/3985917114 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c0052d0 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d48ff9640 1 -- 172.21.15.27:0/3985917114 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c005490 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c0b5ba0 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d48ff9640 1 -- 172.21.15.27:0/3985917114 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c007260 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.304+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c0b6160 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.305+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c0b63a0 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.306+0000 7f0d48ff9640 1 -- 172.21.15.27:0/3985917114 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d3c007400 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.306+0000 7f0d48ff9640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d2c06b500 0x7f0d2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.306+0000 7f0d48ff9640 1 -- 172.21.15.27:0/3985917114 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0d3c089400 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.306+0000 7f0d43fff640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d2c06b500 0x7f0d2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.307+0000 7f0d43fff640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d2c06b500 0x7f0d2c06d9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0d34036000 tx=0x7f0d3403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.310+0000 7f0d48ff9640 1 -- 172.21.15.27:0/3985917114 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d4c0c5bd0 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.491+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0d4c09fcf0 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.494+0000 7f0d48ff9640 1 -- 172.21.15.27:0/3985917114 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0d3c058580 con 0x7f0d4c06b2c0 2024-03-06T07:26:04.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.496+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d2c06b500 msgr2=0x7f0d2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.496+0000 7f0d518eb640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d2c06b500 0x7f0d2c06d9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0d34036000 tx=0x7f0d3403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.496+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c06b2c0 msgr2=0x7f0d4c157380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.496+0000 7f0d518eb640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c06b2c0 0x7f0d4c157380 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c3c0 tx=0x7f0d3c00c880 comp rx=0 tx=0).stop 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.497+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 shutdown_connections 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.497+0000 7f0d518eb640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d2c06b500 0x7f0d2c06d9c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.497+0000 7f0d518eb640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4c0cdcd0 0x7f0d4c0ca980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.497+0000 7f0d518eb640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d4c0b2cd0 0x7f0d4c0ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.497+0000 7f0d518eb640 1 --2- 172.21.15.27:0/3985917114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4c06b2c0 0x7f0d4c157380 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.497+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 >> 172.21.15.27:0/3985917114 conn(0x7f0d4c0b3700 msgr2=0x7f0d4c06b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.497+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 shutdown_connections 2024-03-06T07:26:04.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.497+0000 7f0d518eb640 1 -- 172.21.15.27:0/3985917114 wait complete. 2024-03-06T07:26:04.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:26:04.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.617+0000 7f25c8b25640 1 Processor -- start 2024-03-06T07:26:04.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.617+0000 7f25c8b25640 1 -- start start 2024-03-06T07:26:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c8b25640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c415e2f0 0x7f25c40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c8b25640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25c406b4e0 0x7f25c406b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c8b25640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c415e6d0 con 0x7f25c406be80 2024-03-06T07:26:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c8b25640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c40ccd00 con 0x7f25c406b4e0 2024-03-06T07:26:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c8b25640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40ccfe0 con 0x7f25c415e2f0 2024-03-06T07:26:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c1d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25c406b4e0 0x7f25c406b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c1d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25c406b4e0 0x7f25c406b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43334/0 (socket says 172.21.15.27:43334) 2024-03-06T07:26:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c1d9b640 1 -- 172.21.15.27:0/1878227982 learned_addr learned my addr 172.21.15.27:0/1878227982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c259c640 1 --2- 172.21.15.27:0/1878227982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c415e2f0 0x7f25c40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c1d9b640 1 -- 172.21.15.27:0/1878227982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c415e2f0 msgr2=0x7f25c40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c1d9b640 1 --2- 172.21.15.27:0/1878227982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c415e2f0 0x7f25c40b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c1d9b640 1 -- 172.21.15.27:0/1878227982 >> v1:172.21.15.69:6789/0 conn(0x7f25c406be80 legacy=0x7f25c40cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.618+0000 7f25c1d9b640 1 -- 172.21.15.27:0/1878227982 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c40cd2c0 con 0x7f25c406b4e0 2024-03-06T07:26:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.619+0000 7f25c1d9b640 1 --2- 172.21.15.27:0/1878227982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406b4e0 0x7f25c406b940 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f25b8009870 tx=0x7f25b802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d748a860694f960 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.619+0000 7f25c159a640 1 -- 172.21.15.27:0/1878227982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80317c0 con 0x7f25c406b4e0 2024-03-06T07:26:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.619+0000 7f25c159a640 1 -- 172.21.15.27:0/1878227982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b8037070 con 0x7f25c406b4e0 2024-03-06T07:26:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.619+0000 7f25c159a640 1 -- 172.21.15.27:0/1878227982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8031b80 con 0x7f25c406b4e0 2024-03-06T07:26:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.620+0000 7f25c8b25640 1 -- 172.21.15.27:0/1878227982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406b4e0 msgr2=0x7f25c406b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.620+0000 7f25c8b25640 1 --2- 172.21.15.27:0/1878227982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406b4e0 0x7f25c406b940 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f25b8009870 tx=0x7f25b802b860 comp rx=0 tx=0).stop 2024-03-06T07:26:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.620+0000 7f25c8b25640 1 -- 172.21.15.27:0/1878227982 shutdown_connections 2024-03-06T07:26:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.620+0000 7f25c8b25640 1 --2- 172.21.15.27:0/1878227982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406b4e0 0x7f25c406b940 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.620+0000 7f25c8b25640 1 --2- 172.21.15.27:0/1878227982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c415e2f0 0x7f25c40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.620+0000 7f25c8b25640 1 -- 172.21.15.27:0/1878227982 >> 172.21.15.27:0/1878227982 conn(0x7f25c40b36d0 msgr2=0x7f25c40b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.620+0000 7f25c8b25640 1 -- 172.21.15.27:0/1878227982 shutdown_connections 2024-03-06T07:26:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.620+0000 7f25c8b25640 1 -- 172.21.15.27:0/1878227982 wait complete. 2024-03-06T07:26:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.621+0000 7f25c8b25640 1 Processor -- start 2024-03-06T07:26:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.621+0000 7f25c8b25640 1 -- start start 2024-03-06T07:26:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c8b25640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c406b4e0 0x7f25c4160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c8b25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406be80 0x7f25c4152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c8b25640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25c415e2f0 0x7f25c41479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c8b25640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40cead0 con 0x7f25c406be80 2024-03-06T07:26:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c8b25640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c406c310 con 0x7f25c415e2f0 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c8b25640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40cec50 con 0x7f25c406b4e0 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406be80 0x7f25c4152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406be80 0x7f25c4152000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43338/0 (socket says 172.21.15.27:43338) 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c1d9b640 1 -- 172.21.15.27:0/2437048254 learned_addr learned my addr 172.21.15.27:0/2437048254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c2d9d640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25c415e2f0 0x7f25c41479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c259c640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c406b4e0 0x7f25c4160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c1d9b640 1 -- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c406b4e0 msgr2=0x7f25c4160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c1d9b640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c406b4e0 0x7f25c4160830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c1d9b640 1 -- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25c415e2f0 msgr2=0x7f25c41479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c1d9b640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25c415e2f0 0x7f25c41479d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.622+0000 7f25c1d9b640 1 -- 172.21.15.27:0/2437048254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c4160d70 con 0x7f25c406be80 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.623+0000 7f25c1d9b640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406be80 0x7f25c4152000 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f25b8000c00 tx=0x7f25b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.623+0000 7f25b2ffd640 1 -- 172.21.15.27:0/2437048254 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8032650 con 0x7f25c406be80 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.623+0000 7f25b2ffd640 1 -- 172.21.15.27:0/2437048254 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b8032810 con 0x7f25c406be80 2024-03-06T07:26:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.623+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c4161040 con 0x7f25c406be80 2024-03-06T07:26:04.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.623+0000 7f25b2ffd640 1 -- 172.21.15.27:0/2437048254 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8032c00 con 0x7f25c406be80 2024-03-06T07:26:04.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.623+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c40c64c0 con 0x7f25c406be80 2024-03-06T07:26:04.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.624+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c409fd30 con 0x7f25c406be80 2024-03-06T07:26:04.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.624+0000 7f25b2ffd640 1 -- 172.21.15.27:0/2437048254 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25b804c060 con 0x7f25c406be80 2024-03-06T07:26:04.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.624+0000 7f25b2ffd640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f25a406b4d0 0x7f25a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.625+0000 7f25b2ffd640 1 -- 172.21.15.27:0/2437048254 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f25b80abb20 con 0x7f25c406be80 2024-03-06T07:26:04.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.625+0000 7f25c259c640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f25a406b4d0 0x7f25a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.625+0000 7f25c259c640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f25a406b4d0 0x7f25a406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f25b4006fd0 tx=0x7f25b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.629+0000 7f25b2ffd640 1 -- 172.21.15.27:0/2437048254 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25b8040070 con 0x7f25c406be80 2024-03-06T07:26:04.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.809+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f25c404f410 con 0x7f25c406be80 2024-03-06T07:26:04.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.812+0000 7f25b2ffd640 1 -- 172.21.15.27:0/2437048254 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v20) v1 ==== 108+0+0 (secure 0 0 0) 0x7f25b807ac90 con 0x7f25c406be80 2024-03-06T07:26:04.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f25a406b4d0 msgr2=0x7f25a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f25a406b4d0 0x7f25a406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f25b4006fd0 tx=0x7f25b4008040 comp rx=0 tx=0).stop 2024-03-06T07:26:04.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406be80 msgr2=0x7f25c4152000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406be80 0x7f25c4152000 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f25b8000c00 tx=0x7f25b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 shutdown_connections 2024-03-06T07:26:04.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f25a406b4d0 0x7f25a406d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25c415e2f0 0x7f25c41479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25c406be80 0x7f25c4152000 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 --2- 172.21.15.27:0/2437048254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25c406b4e0 0x7f25c4160830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 >> 172.21.15.27:0/2437048254 conn(0x7f25c40b36d0 msgr2=0x7f25c40d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 shutdown_connections 2024-03-06T07:26:04.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.814+0000 7f25c8b25640 1 -- 172.21.15.27:0/2437048254 wait complete. 2024-03-06T07:26:04.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:26:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.934+0000 7f7654aa4640 1 Processor -- start 2024-03-06T07:26:04.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.934+0000 7f7654aa4640 1 -- start start 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.935+0000 7f7654aa4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76500cead0 0x7f7650168e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.935+0000 7f7654aa4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765015e6d0 con 0x7f76500b41f0 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.935+0000 7f7654aa4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76500ccd00 con 0x7f765015e2f0 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.935+0000 7f7654aa4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76500d0a10 con 0x7f76500cead0 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.935+0000 7f764ed9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76500cead0 0x7f7650168e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.935+0000 7f764dd9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f76500b41f0 0x7f765006bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35106/0 (socket says 172.21.15.27:35106) 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.935+0000 7f764dd9b640 1 -- 172.21.15.27:0/2328760068 learned_addr learned my addr 172.21.15.27:0/2328760068 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.936+0000 7f764ed9d640 1 -- 172.21.15.27:0/2328760068 >> v1:172.21.15.69:6789/0 conn(0x7f765015e2f0 legacy=0x7f765006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.936+0000 7f764ed9d640 1 -- 172.21.15.27:0/2328760068 >> v1:172.21.15.27:6789/0 conn(0x7f76500b41f0 legacy=0x7f765006bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.936+0000 7f764ed9d640 1 -- 172.21.15.27:0/2328760068 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76500ccfe0 con 0x7f76500cead0 2024-03-06T07:26:04.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.936+0000 7f764ed9d640 1 --2- 172.21.15.27:0/2328760068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 0x7f7650168e30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7648009d90 tx=0x7f764802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a38243ccec7972 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.936+0000 7f764d59a640 1 -- 172.21.15.27:0/2328760068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648039070 con 0x7f76500cead0 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.937+0000 7f764d59a640 1 -- 172.21.15.27:0/2328760068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764802cbd0 con 0x7f76500cead0 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.937+0000 7f764d59a640 1 -- 172.21.15.27:0/2328760068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648034440 con 0x7f76500cead0 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.937+0000 7f7654aa4640 1 -- 172.21.15.27:0/2328760068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 msgr2=0x7f7650168e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.937+0000 7f7654aa4640 1 --2- 172.21.15.27:0/2328760068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 0x7f7650168e30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7648009d90 tx=0x7f764802be20 comp rx=0 tx=0).stop 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.937+0000 7f7654aa4640 1 -- 172.21.15.27:0/2328760068 shutdown_connections 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.937+0000 7f7654aa4640 1 --2- 172.21.15.27:0/2328760068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 0x7f7650168e30 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.937+0000 7f7654aa4640 1 -- 172.21.15.27:0/2328760068 >> 172.21.15.27:0/2328760068 conn(0x7f76500b36d0 msgr2=0x7f76500b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.937+0000 7f7654aa4640 1 -- 172.21.15.27:0/2328760068 shutdown_connections 2024-03-06T07:26:04.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.937+0000 7f7654aa4640 1 -- 172.21.15.27:0/2328760068 wait complete. 2024-03-06T07:26:04.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.938+0000 7f7654aa4640 1 Processor -- start 2024-03-06T07:26:04.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.938+0000 7f7654aa4640 1 -- start start 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f7654aa4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76500b41f0 0x7f765014d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f7654aa4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 0x7f76500c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f7654aa4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765015e2f0 0x7f76500c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f7654aa4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765006c300 con 0x7f76500cead0 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f7654aa4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76500cd5a0 con 0x7f76500b41f0 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f7654aa4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7650169370 con 0x7f765015e2f0 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f764dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 0x7f76500c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f764dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 0x7f76500c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43354/0 (socket says 172.21.15.27:43354) 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f764dd9b640 1 -- 172.21.15.27:0/1970515923 learned_addr learned my addr 172.21.15.27:0/1970515923 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f764ed9d640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765015e2f0 0x7f76500c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.939+0000 7f764e59c640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76500b41f0 0x7f765014d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f764dd9b640 1 -- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765015e2f0 msgr2=0x7f76500c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f764dd9b640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765015e2f0 0x7f76500c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f764dd9b640 1 -- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76500b41f0 msgr2=0x7f765014d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f764dd9b640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76500b41f0 0x7f765014d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f764dd9b640 1 -- 172.21.15.27:0/1970515923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76500c7850 con 0x7f76500cead0 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f764dd9b640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 0x7f76500c68b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f764400c430 tx=0x7f764400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f763effd640 1 -- 172.21.15.27:0/1970515923 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644017070 con 0x7f76500cead0 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f763effd640 1 -- 172.21.15.27:0/1970515923 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644005ce0 con 0x7f76500cead0 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76500c7b30 con 0x7f76500cead0 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f763effd640 1 -- 172.21.15.27:0/1970515923 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76440072f0 con 0x7f76500cead0 2024-03-06T07:26:04.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.940+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76501f58a0 con 0x7f76500cead0 2024-03-06T07:26:04.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.941+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765009fd30 con 0x7f76500cead0 2024-03-06T07:26:04.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.941+0000 7f763effd640 1 -- 172.21.15.27:0/1970515923 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f764401a070 con 0x7f76500cead0 2024-03-06T07:26:04.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.942+0000 7f763effd640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f763006b4d0 0x7f763006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.942+0000 7f763effd640 1 -- 172.21.15.27:0/1970515923 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7644089320 con 0x7f76500cead0 2024-03-06T07:26:04.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.942+0000 7f764e59c640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f763006b4d0 0x7f763006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.943+0000 7f764e59c640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f763006b4d0 0x7f763006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7640006fd0 tx=0x7f7640008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:04.947+0000 7f763effd640 1 -- 172.21.15.27:0/1970515923 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76440584a0 con 0x7f76500cead0 2024-03-06T07:26:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.127+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7f765004f410 con 0x7f76500cead0 2024-03-06T07:26:05.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.130+0000 7f763effd640 1 -- 172.21.15.27:0/1970515923 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v21) v1 ==== 101+0+0 (secure 0 0 0) 0x7f7644057da0 con 0x7f76500cead0 2024-03-06T07:26:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f763006b4d0 msgr2=0x7f763006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f763006b4d0 0x7f763006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7640006fd0 tx=0x7f7640008040 comp rx=0 tx=0).stop 2024-03-06T07:26:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 msgr2=0x7f76500c68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 0x7f76500c68b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f764400c430 tx=0x7f764400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 shutdown_connections 2024-03-06T07:26:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f763006b4d0 0x7f763006d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f765015e2f0 0x7f76500c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76500cead0 0x7f76500c68b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 --2- 172.21.15.27:0/1970515923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76500b41f0 0x7f765014d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 >> 172.21.15.27:0/1970515923 conn(0x7f76500b36d0 msgr2=0x7f76500b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 shutdown_connections 2024-03-06T07:26:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.132+0000 7f7654aa4640 1 -- 172.21.15.27:0/1970515923 wait complete. 2024-03-06T07:26:05.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:26:05.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.250+0000 7f1af379a640 1 Processor -- start 2024-03-06T07:26:05.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.251+0000 7f1af379a640 1 -- start start 2024-03-06T07:26:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.251+0000 7f1af379a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1aec0b95a0 0x7f1aec0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.251+0000 7f1af379a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec0cea30 con 0x7f1aec0b9a10 2024-03-06T07:26:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.251+0000 7f1af379a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec0cf3c0 con 0x7f1aec15df00 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.251+0000 7f1af379a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec0cf0e0 con 0x7f1aec0b95a0 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.251+0000 7f1af1d37640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1aec0b9a10 0x7f1aec0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35112/0 (socket says 172.21.15.27:35112) 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.251+0000 7f1af1d37640 1 -- 172.21.15.27:0/278254519 learned_addr learned my addr 172.21.15.27:0/278254519 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af0d35640 1 --2- 172.21.15.27:0/278254519 >> v2:172.21.15.69:3300/0 conn(0x7f1aec0b95a0 0x7f1aec0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643126930 0 0) 0x7f1aec0cea30 con 0x7f1aec0b9a10 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad4003520 con 0x7f1aec0b9a10 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679042549 0 0) 0x7f1ad4003520 con 0x7f1aec0b9a10 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 >> v2:172.21.15.69:3300/0 conn(0x7f1aec0b95a0 msgr2=0x7f1aec0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 --2- 172.21.15.27:0/278254519 >> v2:172.21.15.69:3300/0 conn(0x7f1aec0b95a0 0x7f1aec0b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 >> v1:172.21.15.69:6789/0 conn(0x7f1aec15df00 legacy=0x7f1aec0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec0ce470 con 0x7f1aec0b9a10 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f1adc0019f0 con 0x7f1aec0b9a10 2024-03-06T07:26:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 >> v1:172.21.15.27:6789/0 conn(0x7f1aec0b9a10 legacy=0x7f1aec0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad40039b0 0x7f1ad4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ad4004800 0x7f1ad4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40051c0 0x7f1ad400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0cf0e0 con 0x7f1ad40039b0 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad400c090 con 0x7f1ad4004800 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad400c290 con 0x7f1ad40051c0 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af1536640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad40039b0 0x7f1ad4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af0d35640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ad4004800 0x7f1ad4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af1d37640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40051c0 0x7f1ad400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af1536640 1 -- 172.21.15.27:0/278254519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40051c0 msgr2=0x7f1ad400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af1536640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40051c0 0x7f1ad400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af1536640 1 -- 172.21.15.27:0/278254519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ad4004800 msgr2=0x7f1ad4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af1536640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ad4004800 0x7f1ad4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af1536640 1 -- 172.21.15.27:0/278254519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad4003520 con 0x7f1ad40039b0 2024-03-06T07:26:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af1d37640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40051c0 0x7f1ad400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.252+0000 7f1af1d37640 1 -- 172.21.15.27:0/278254519 reap_dead start 2024-03-06T07:26:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.253+0000 7f1af1536640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad40039b0 0x7f1ad4003df0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4001750 tx=0x7f1ad4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f05e670fca1d7bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.253+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4001370 con 0x7f1ad40039b0 2024-03-06T07:26:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.253+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae40052d0 con 0x7f1ad40039b0 2024-03-06T07:26:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.253+0000 7f1aebfff640 1 -- 172.21.15.27:0/278254519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4003830 con 0x7f1ad40039b0 2024-03-06T07:26:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.254+0000 7f1af379a640 1 -- 172.21.15.27:0/278254519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad40039b0 msgr2=0x7f1ad4003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.254+0000 7f1af379a640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad40039b0 0x7f1ad4003df0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4001750 tx=0x7f1ad4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.254+0000 7f1af379a640 1 -- 172.21.15.27:0/278254519 shutdown_connections 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.254+0000 7f1af379a640 1 --2- 172.21.15.27:0/278254519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ad40039b0 0x7f1ad4003df0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.254+0000 7f1af379a640 1 -- 172.21.15.27:0/278254519 >> 172.21.15.27:0/278254519 conn(0x7f1aec071960 msgr2=0x7f1aec071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.254+0000 7f1af379a640 1 -- 172.21.15.27:0/278254519 shutdown_connections 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.254+0000 7f1af379a640 1 -- 172.21.15.27:0/278254519 wait complete. 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.255+0000 7f1af379a640 1 Processor -- start 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.255+0000 7f1af379a640 1 -- start start 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af379a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40039b0 0x7f1aec160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af379a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aec15ab10 0x7f1aec15a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af379a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aec0c7b30 0x7f1aec0c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af379a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0cedf0 con 0x7f1aec15ab10 2024-03-06T07:26:05.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af379a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0cec70 con 0x7f1aec0c7b30 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af379a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0d25b0 con 0x7f1ad40039b0 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af0d35640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aec15ab10 0x7f1aec15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af0d35640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aec15ab10 0x7f1aec15a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43372/0 (socket says 172.21.15.27:43372) 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af0d35640 1 -- 172.21.15.27:0/1558243502 learned_addr learned my addr 172.21.15.27:0/1558243502 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af1536640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40039b0 0x7f1aec160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af1d37640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aec0c7b30 0x7f1aec0c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af0d35640 1 -- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40039b0 msgr2=0x7f1aec160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af0d35640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40039b0 0x7f1aec160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af0d35640 1 -- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aec0c7b30 msgr2=0x7f1aec0c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af0d35640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aec0c7b30 0x7f1aec0c7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af0d35640 1 -- 172.21.15.27:0/1558243502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec15fea0 con 0x7f1aec15ab10 2024-03-06T07:26:05.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.256+0000 7f1af0d35640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aec15ab10 0x7f1aec15a210 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0011aa0 tx=0x7f1ae0011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.257+0000 7f1ae9ffb640 1 -- 172.21.15.27:0/1558243502 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae000fb80 con 0x7f1aec15ab10 2024-03-06T07:26:05.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.257+0000 7f1ae9ffb640 1 -- 172.21.15.27:0/1558243502 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae000fd20 con 0x7f1aec15ab10 2024-03-06T07:26:05.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.257+0000 7f1ae9ffb640 1 -- 172.21.15.27:0/1558243502 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0014660 con 0x7f1aec15ab10 2024-03-06T07:26:05.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.257+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec160180 con 0x7f1aec15ab10 2024-03-06T07:26:05.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.257+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec0cc1b0 con 0x7f1aec15ab10 2024-03-06T07:26:05.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.258+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aec067ff0 con 0x7f1aec15ab10 2024-03-06T07:26:05.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.258+0000 7f1ae9ffb640 1 -- 172.21.15.27:0/1558243502 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ae0012040 con 0x7f1aec15ab10 2024-03-06T07:26:05.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.259+0000 7f1ae9ffb640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b4d0 0x7f1ad806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.259+0000 7f1ae9ffb640 1 -- 172.21.15.27:0/1558243502 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1ae008a520 con 0x7f1aec15ab10 2024-03-06T07:26:05.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.259+0000 7f1af1536640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b4d0 0x7f1ad806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.259+0000 7f1af1536640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b4d0 0x7f1ad806d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4001750 tx=0x7f1ae4009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.263+0000 7f1ae9ffb640 1 -- 172.21.15.27:0/1558243502 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ae0019030 con 0x7f1aec15ab10 2024-03-06T07:26:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:05.514+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f1aec0b2920 con 0x7f1aec15ab10 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.158+0000 7f1ae9ffb640 1 -- 172.21.15.27:0/1558243502 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v200) v1 ==== 144+0+0 (secure 0 0 0) 0x7f1ae0059730 con 0x7f1aec15ab10 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b4d0 msgr2=0x7f1ad806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b4d0 0x7f1ad806d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4001750 tx=0x7f1ae4009030 comp rx=0 tx=0).stop 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aec15ab10 msgr2=0x7f1aec15a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aec15ab10 0x7f1aec15a210 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0011aa0 tx=0x7f1ae0011f60 comp rx=0 tx=0).stop 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 shutdown_connections 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ad806b4d0 0x7f1ad806d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aec0c7b30 0x7f1aec0c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aec15ab10 0x7f1aec15a210 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 --2- 172.21.15.27:0/1558243502 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1ad40039b0 0x7f1aec160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 >> 172.21.15.27:0/1558243502 conn(0x7f1aec071960 msgr2=0x7f1aec0b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 shutdown_connections 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.160+0000 7f1af379a640 1 -- 172.21.15.27:0/1558243502 wait complete. 2024-03-06T07:26:06.175 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:26:06.175 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:26:06.175 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:26:06.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.283+0000 7fcc0c37a640 1 Processor -- start 2024-03-06T07:26:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.283+0000 7fcc0c37a640 1 -- start start 2024-03-06T07:26:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc0c37a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 0x7fcc040ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc0c37a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc0406fe20 0x7fcc040ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc0c37a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcc040d0880 0x7fcc040ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc0c37a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc040d0dd0 con 0x7fcc0406fe20 2024-03-06T07:26:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc0c37a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc040d10b0 con 0x7fcc040d0880 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc0c37a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040cc0e0 con 0x7fcc0415e060 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc09915640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc0406fe20 0x7fcc040ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc09915640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc0406fe20 0x7fcc040ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43382/0 (socket says 172.21.15.27:43382) 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc09915640 1 -- 172.21.15.27:0/952150307 learned_addr learned my addr 172.21.15.27:0/952150307 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc0a116640 1 --2- 172.21.15.27:0/952150307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 0x7fcc040ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc0a917640 1 --2- 172.21.15.27:0/952150307 >> v2:172.21.15.69:3300/0 conn(0x7fcc040d0880 0x7fcc040ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc09915640 1 -- 172.21.15.27:0/952150307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 msgr2=0x7fcc040ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc09915640 1 --2- 172.21.15.27:0/952150307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 0x7fcc040ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc09915640 1 -- 172.21.15.27:0/952150307 >> v2:172.21.15.69:3300/0 conn(0x7fcc040d0880 msgr2=0x7fcc040ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc09915640 1 --2- 172.21.15.27:0/952150307 >> v2:172.21.15.69:3300/0 conn(0x7fcc040d0880 0x7fcc040ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.284+0000 7fcc09915640 1 -- 172.21.15.27:0/952150307 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc040cc3c0 con 0x7fcc0406fe20 2024-03-06T07:26:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.285+0000 7fcc09915640 1 --2- 172.21.15.27:0/952150307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 0x7fcc040ce850 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fcc04072a10 tx=0x7fcbf402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7198232c2ffb6b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.285+0000 7fcc09114640 1 -- 172.21.15.27:0/952150307 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4035020 con 0x7fcc0406fe20 2024-03-06T07:26:06.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.285+0000 7fcc09114640 1 -- 172.21.15.27:0/952150307 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf4031870 con 0x7fcc0406fe20 2024-03-06T07:26:06.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.285+0000 7fcc09114640 1 -- 172.21.15.27:0/952150307 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4031c40 con 0x7fcc0406fe20 2024-03-06T07:26:06.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.286+0000 7fcc0c37a640 1 -- 172.21.15.27:0/952150307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 msgr2=0x7fcc040ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.286+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/952150307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 0x7fcc040ce850 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fcc04072a10 tx=0x7fcbf402b910 comp rx=0 tx=0).stop 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.286+0000 7fcc0c37a640 1 -- 172.21.15.27:0/952150307 shutdown_connections 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.286+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/952150307 >> v2:172.21.15.69:3300/0 conn(0x7fcc040d0880 0x7fcc040ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.286+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/952150307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 0x7fcc040ce850 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.286+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/952150307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 0x7fcc040ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.286+0000 7fcc0c37a640 1 -- 172.21.15.27:0/952150307 >> 172.21.15.27:0/952150307 conn(0x7fcc040d1a90 msgr2=0x7fcc040d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.286+0000 7fcc0c37a640 1 -- 172.21.15.27:0/952150307 shutdown_connections 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.286+0000 7fcc0c37a640 1 -- 172.21.15.27:0/952150307 wait complete. 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.287+0000 7fcc0c37a640 1 Processor -- start 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.287+0000 7fcc0c37a640 1 -- start start 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0c37a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 0x7fcc04160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0c37a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc040d0880 0x7fcc0406bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0c37a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 0x7fcc0406c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0c37a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0406ed30 con 0x7fcc0406fe20 2024-03-06T07:26:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0c37a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0415e440 con 0x7fcc040d0880 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0c37a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0406eeb0 con 0x7fcc0415e060 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a116640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 0x7fcc04160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a116640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 0x7fcc04160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43388/0 (socket says 172.21.15.27:43388) 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a116640 1 -- 172.21.15.27:0/2072757378 learned_addr learned my addr 172.21.15.27:0/2072757378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a917640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 0x7fcc0406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc09915640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc040d0880 0x7fcc0406bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a116640 1 -- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 msgr2=0x7fcc0406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a116640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 0x7fcc0406c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a116640 1 -- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc040d0880 msgr2=0x7fcc0406bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a116640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc040d0880 0x7fcc0406bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a116640 1 -- 172.21.15.27:0/2072757378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc040c6b00 con 0x7fcc0406fe20 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc09915640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc040d0880 0x7fcc0406bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.288+0000 7fcc0a116640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 0x7fcc04160020 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000d5d0 tx=0x7fcc0000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.289+0000 7fcbfaffd640 1 -- 172.21.15.27:0/2072757378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc000125f0 con 0x7fcc0406fe20 2024-03-06T07:26:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.289+0000 7fcbfaffd640 1 -- 172.21.15.27:0/2072757378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc00019070 con 0x7fcc0406fe20 2024-03-06T07:26:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.289+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc040c5f80 con 0x7fcc0406fe20 2024-03-06T07:26:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.289+0000 7fcbfaffd640 1 -- 172.21.15.27:0/2072757378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc000129f0 con 0x7fcc0406fe20 2024-03-06T07:26:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.289+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc040c6540 con 0x7fcc0406fe20 2024-03-06T07:26:06.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.290+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc04067ff0 con 0x7fcc0406fe20 2024-03-06T07:26:06.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.290+0000 7fcbfaffd640 1 -- 172.21.15.27:0/2072757378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc00012b90 con 0x7fcc0406fe20 2024-03-06T07:26:06.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.291+0000 7fcbfaffd640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcbe806b4d0 0x7fcbe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.291+0000 7fcbfaffd640 1 -- 172.21.15.27:0/2072757378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcc0008a260 con 0x7fcc0406fe20 2024-03-06T07:26:06.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.291+0000 7fcc09915640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcbe806b4d0 0x7fcbe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.292+0000 7fcc09915640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcbe806b4d0 0x7fcbe806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4036000 tx=0x7fcbf403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.295+0000 7fcbfaffd640 1 -- 172.21.15.27:0/2072757378 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc00059470 con 0x7fcc0406fe20 2024-03-06T07:26:06.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.478+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc041613d0 con 0x7fcc0406fe20 2024-03-06T07:26:06.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.479+0000 7fcbfaffd640 1 -- 172.21.15.27:0/2072757378 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+12944 (secure 0 0 0) 0x7fcc00058d70 con 0x7fcc0406fe20 2024-03-06T07:26:06.481 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:06.482 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":61,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:04.147968+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6833/60877422":"2024-03-07T07:26:01.115291+0000","172.21.15.27:6832/60877422":"2024-03-07T07:26:01.115291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcbe806b4d0 msgr2=0x7fcbe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcbe806b4d0 0x7fcbe806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4036000 tx=0x7fcbf403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 msgr2=0x7fcc04160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 0x7fcc04160020 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000d5d0 tx=0x7fcc0000da90 comp rx=0 tx=0).stop 2024-03-06T07:26:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 shutdown_connections 2024-03-06T07:26:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcbe806b4d0 0x7fcbe806d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc0415e060 0x7fcc0406c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc040d0880 0x7fcc0406bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 --2- 172.21.15.27:0/2072757378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0406fe20 0x7fcc04160020 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 >> 172.21.15.27:0/2072757378 conn(0x7fcc040d1a90 msgr2=0x7fcc040b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 shutdown_connections 2024-03-06T07:26:06.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.481+0000 7fcc0c37a640 1 -- 172.21.15.27:0/2072757378 wait complete. 2024-03-06T07:26:06.495 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:06.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:06.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.602+0000 7f31b5913640 1 Processor -- start 2024-03-06T07:26:06.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.603+0000 7f31b5913640 1 -- start start 2024-03-06T07:26:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.603+0000 7f31b5913640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f31b01618e0 0x7f31b006b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.603+0000 7f31b5913640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce960 0x7f31b0162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.603+0000 7f31b5913640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b015f730 con 0x7f31b00ce4f0 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.603+0000 7f31b5913640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b00cfce0 con 0x7f31b01618e0 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.603+0000 7f31b5913640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b015ebb0 con 0x7f31b00ce960 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31aeffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f31b01618e0 0x7f31b006b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31af7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce960 0x7f31b0162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31ae7fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f31b00ce4f0 0x7f31b0161d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:48322/0 (socket says 172.21.15.27:48322) 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31ae7fc640 1 -- 172.21.15.27:0/2588077665 learned_addr learned my addr 172.21.15.27:0/2588077665 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31adffb640 1 -- 172.21.15.27:0/2588077665 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716261189 0 0) 0x7f31b015f730 con 0x7f31b00ce4f0 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31adffb640 1 -- 172.21.15.27:0/2588077665 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f319c003520 con 0x7f31b00ce4f0 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31aeffd640 1 -- 172.21.15.27:0/2588077665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce960 msgr2=0x7f31b0162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31aeffd640 1 --2- 172.21.15.27:0/2588077665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce960 0x7f31b0162450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31aeffd640 1 -- 172.21.15.27:0/2588077665 >> v1:172.21.15.69:6789/0 conn(0x7f31b00ce4f0 legacy=0x7f31b0161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31aeffd640 1 -- 172.21.15.27:0/2588077665 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b015ee90 con 0x7f31b01618e0 2024-03-06T07:26:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.604+0000 7f31af7fe640 1 --2- 172.21.15.27:0/2588077665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce960 0x7f31b0162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.605+0000 7f31aeffd640 1 --2- 172.21.15.27:0/2588077665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b01618e0 0x7f31b006b130 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f31a80097a0 tx=0x7f31a802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dabe37d091a1fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.605+0000 7f31adffb640 1 -- 172.21.15.27:0/2588077665 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8034070 con 0x7f31b01618e0 2024-03-06T07:26:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.605+0000 7f31adffb640 1 -- 172.21.15.27:0/2588077665 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a8030610 con 0x7f31b01618e0 2024-03-06T07:26:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.605+0000 7f31adffb640 1 -- 172.21.15.27:0/2588077665 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a80309e0 con 0x7f31b01618e0 2024-03-06T07:26:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.606+0000 7f31b5913640 1 -- 172.21.15.27:0/2588077665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b01618e0 msgr2=0x7f31b006b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.606+0000 7f31b5913640 1 --2- 172.21.15.27:0/2588077665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b01618e0 0x7f31b006b130 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f31a80097a0 tx=0x7f31a802b960 comp rx=0 tx=0).stop 2024-03-06T07:26:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.606+0000 7f31b5913640 1 -- 172.21.15.27:0/2588077665 shutdown_connections 2024-03-06T07:26:06.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.606+0000 7f31b5913640 1 --2- 172.21.15.27:0/2588077665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce960 0x7f31b0162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.606+0000 7f31b5913640 1 --2- 172.21.15.27:0/2588077665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b01618e0 0x7f31b006b130 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.606+0000 7f31b5913640 1 -- 172.21.15.27:0/2588077665 >> 172.21.15.27:0/2588077665 conn(0x7f31b00b3660 msgr2=0x7f31b00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.606+0000 7f31b5913640 1 -- 172.21.15.27:0/2588077665 shutdown_connections 2024-03-06T07:26:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.606+0000 7f31b5913640 1 -- 172.21.15.27:0/2588077665 wait complete. 2024-03-06T07:26:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.607+0000 7f31b5913640 1 Processor -- start 2024-03-06T07:26:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.607+0000 7f31b5913640 1 -- start start 2024-03-06T07:26:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31b5913640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce4f0 0x7f31b00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31b5913640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31b00ce960 0x7f31b00c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31b5913640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b015fee0 0x7f31b0160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31b5913640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b006b670 con 0x7f31b00ce960 2024-03-06T07:26:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31b5913640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b015f450 con 0x7f31b015fee0 2024-03-06T07:26:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31b5913640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b00cedf0 con 0x7f31b00ce4f0 2024-03-06T07:26:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31ae7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31b00ce960 0x7f31b00c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31ae7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31b00ce960 0x7f31b00c9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43400/0 (socket says 172.21.15.27:43400) 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31ae7fc640 1 -- 172.21.15.27:0/1275636301 learned_addr learned my addr 172.21.15.27:0/1275636301 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31aeffd640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce4f0 0x7f31b00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31af7fe640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b015fee0 0x7f31b0160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31ae7fc640 1 -- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce4f0 msgr2=0x7f31b00b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31ae7fc640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce4f0 0x7f31b00b6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31ae7fc640 1 -- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b015fee0 msgr2=0x7f31b0160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31ae7fc640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b015fee0 0x7f31b0160370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31ae7fc640 1 -- 172.21.15.27:0/1275636301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b00b66a0 con 0x7f31b00ce960 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.608+0000 7f31ae7fc640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31b00ce960 0x7f31b00c9630 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f319800c460 tx=0x7f319800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.609+0000 7f31a7fff640 1 -- 172.21.15.27:0/1275636301 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198017070 con 0x7f31b00ce960 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.609+0000 7f31a7fff640 1 -- 172.21.15.27:0/1275636301 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3198005ce0 con 0x7f31b00ce960 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.609+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b00ca950 con 0x7f31b00ce960 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.609+0000 7f31a7fff640 1 -- 172.21.15.27:0/1275636301 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31980072f0 con 0x7f31b00ce960 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.609+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b00caf10 con 0x7f31b00ce960 2024-03-06T07:26:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.610+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b009fc50 con 0x7f31b00ce960 2024-03-06T07:26:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.610+0000 7f31a7fff640 1 -- 172.21.15.27:0/1275636301 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f319801a070 con 0x7f31b00ce960 2024-03-06T07:26:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.611+0000 7f31a7fff640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f319006b4d0 0x7f319006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.611+0000 7f31a7fff640 1 -- 172.21.15.27:0/1275636301 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f31980891b0 con 0x7f31b00ce960 2024-03-06T07:26:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.611+0000 7f31aeffd640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f319006b4d0 0x7f319006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.612+0000 7f31aeffd640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f319006b4d0 0x7f319006d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f31a8036000 tx=0x7f31a803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.615+0000 7f31a7fff640 1 -- 172.21.15.27:0/1275636301 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3198057d70 con 0x7f31b00ce960 2024-03-06T07:26:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.795+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31b004f410 con 0x7f31b00ce960 2024-03-06T07:26:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.796+0000 7f31a7fff640 1 -- 172.21.15.27:0/1275636301 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) v1 ==== 93+0+10 (secure 0 0 0) 0x7f319805be10 con 0x7f31b00ce960 2024-03-06T07:26:06.799 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:26:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f319006b4d0 msgr2=0x7f319006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f319006b4d0 0x7f319006d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f31a8036000 tx=0x7f31a803b040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31b00ce960 msgr2=0x7f31b00c9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31b00ce960 0x7f31b00c9630 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f319800c460 tx=0x7f319800c860 comp rx=0 tx=0).stop 2024-03-06T07:26:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 shutdown_connections 2024-03-06T07:26:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f319006b4d0 0x7f319006d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b015fee0 0x7f31b0160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31b00ce960 0x7f31b00c9630 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 --2- 172.21.15.27:0/1275636301 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f31b00ce4f0 0x7f31b00b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 >> 172.21.15.27:0/1275636301 conn(0x7f31b00b3660 msgr2=0x7f31b00cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 shutdown_connections 2024-03-06T07:26:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.798+0000 7f31b5913640 1 -- 172.21.15.27:0/1275636301 wait complete. 2024-03-06T07:26:06.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:26:06.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.915+0000 7f3971a88640 1 Processor -- start 2024-03-06T07:26:06.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.915+0000 7f3971a88640 1 -- start start 2024-03-06T07:26:06.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.916+0000 7f3971a88640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f396c0d3400 0x7f396c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.916+0000 7f3971a88640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f396c0b2b50 0x7f396c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.916+0000 7f3971a88640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c0ce610 con 0x7f396c071540 2024-03-06T07:26:06.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.916+0000 7f3971a88640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c0d2f70 con 0x7f396c0d3400 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.916+0000 7f3971a88640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c0d4be0 con 0x7f396c0b2b50 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.916+0000 7f396affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f396c0d3400 0x7f396c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.916+0000 7f396b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f396c071540 0x7f396c071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35118/0 (socket says 172.21.15.27:35118) 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.916+0000 7f396b7fe640 1 -- 172.21.15.27:0/381583950 learned_addr learned my addr 172.21.15.27:0/381583950 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.917+0000 7f396bfff640 1 --2- 172.21.15.27:0/381583950 >> v2:172.21.15.69:3300/0 conn(0x7f396c0b2b50 0x7f396c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.917+0000 7f396affd640 1 -- 172.21.15.27:0/381583950 >> v2:172.21.15.69:3300/0 conn(0x7f396c0b2b50 msgr2=0x7f396c0b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.917+0000 7f396affd640 1 --2- 172.21.15.27:0/381583950 >> v2:172.21.15.69:3300/0 conn(0x7f396c0b2b50 0x7f396c0b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.917+0000 7f396affd640 1 -- 172.21.15.27:0/381583950 >> v1:172.21.15.27:6789/0 conn(0x7f396c071540 legacy=0x7f396c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.917+0000 7f396affd640 1 -- 172.21.15.27:0/381583950 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c0d2920 con 0x7f396c0d3400 2024-03-06T07:26:06.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.917+0000 7f396a7fc640 1 -- 172.21.15.27:0/381583950 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642772499 0 0) 0x7f396c0ce610 con 0x7f396c071540 2024-03-06T07:26:06.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.917+0000 7f396affd640 1 --2- 172.21.15.27:0/381583950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 0x7f396c0b30a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f396c0683b0 tx=0x7f396002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29a83e9322d20be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f396a7fc640 1 -- 172.21.15.27:0/381583950 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960035020 con 0x7f396c0d3400 2024-03-06T07:26:06.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f396a7fc640 1 -- 172.21.15.27:0/381583950 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3960031830 con 0x7f396c0d3400 2024-03-06T07:26:06.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f396a7fc640 1 -- 172.21.15.27:0/381583950 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960031c00 con 0x7f396c0d3400 2024-03-06T07:26:06.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f3971a88640 1 -- 172.21.15.27:0/381583950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 msgr2=0x7f396c0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f3971a88640 1 --2- 172.21.15.27:0/381583950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 0x7f396c0b30a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f396c0683b0 tx=0x7f396002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f3971a88640 1 -- 172.21.15.27:0/381583950 shutdown_connections 2024-03-06T07:26:06.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f3971a88640 1 --2- 172.21.15.27:0/381583950 >> v2:172.21.15.69:3300/0 conn(0x7f396c0b2b50 0x7f396c0b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f3971a88640 1 --2- 172.21.15.27:0/381583950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 0x7f396c0b30a0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f3971a88640 1 -- 172.21.15.27:0/381583950 >> 172.21.15.27:0/381583950 conn(0x7f396c0d3ca0 msgr2=0x7f396c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f3971a88640 1 -- 172.21.15.27:0/381583950 shutdown_connections 2024-03-06T07:26:06.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.918+0000 7f3971a88640 1 -- 172.21.15.27:0/381583950 wait complete. 2024-03-06T07:26:06.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.919+0000 7f3971a88640 1 Processor -- start 2024-03-06T07:26:06.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f3971a88640 1 -- start start 2024-03-06T07:26:06.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f3971a88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f396c071540 0x7f396c1551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f3971a88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f396c0b2b50 0x7f396c0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f3971a88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 0x7f396c0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f3971a88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c15e370 con 0x7f396c0d3400 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f3971a88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c0ce8f0 con 0x7f396c071540 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f3971a88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1558d0 con 0x7f396c0b2b50 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f396bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 0x7f396c0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f396bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 0x7f396c0cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43418/0 (socket says 172.21.15.27:43418) 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.920+0000 7f396bfff640 1 -- 172.21.15.27:0/1420204519 learned_addr learned my addr 172.21.15.27:0/1420204519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f396b7fe640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f396c071540 0x7f396c1551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f396bfff640 1 -- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f396c0b2b50 msgr2=0x7f396c0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f396bfff640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f396c0b2b50 0x7f396c0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f396bfff640 1 -- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f396c071540 msgr2=0x7f396c1551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f396bfff640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f396c071540 0x7f396c1551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f396bfff640 1 -- 172.21.15.27:0/1420204519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c155bb0 con 0x7f396c0d3400 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f396bfff640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 0x7f396c0cb2b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f396400d800 tx=0x7f396400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f3953fff640 1 -- 172.21.15.27:0/1420204519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396400b6c0 con 0x7f396c0d3400 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f3953fff640 1 -- 172.21.15.27:0/1420204519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3964019070 con 0x7f396c0d3400 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c155e90 con 0x7f396c0d3400 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f3953fff640 1 -- 172.21.15.27:0/1420204519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396400bac0 con 0x7f396c0d3400 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.921+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c156450 con 0x7f396c0d3400 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.922+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c068700 con 0x7f396c0d3400 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.923+0000 7f3953fff640 1 -- 172.21.15.27:0/1420204519 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f396400bc60 con 0x7f396c0d3400 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.923+0000 7f3953fff640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f395806b4d0 0x7f395806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.923+0000 7f3953fff640 1 -- 172.21.15.27:0/1420204519 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f396408a390 con 0x7f396c0d3400 2024-03-06T07:26:06.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.923+0000 7f396b7fe640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f395806b4d0 0x7f395806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.924+0000 7f396b7fe640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f395806b4d0 0x7f395806d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f395c0012e0 tx=0x7f395c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:06.927+0000 7f3953fff640 1 -- 172.21.15.27:0/1420204519 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3964059510 con 0x7f396c0d3400 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:07.179+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f396c06b8e0 con 0x7f396c0d3400 2024-03-06T07:26:08.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.164+0000 7f3953fff640 1 -- 172.21.15.27:0/1420204519 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v202) v1 ==== 143+0+0 (secure 0 0 0) 0x7f3964058e10 con 0x7f396c0d3400 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f395806b4d0 msgr2=0x7f395806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f395806b4d0 0x7f395806d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f395c0012e0 tx=0x7f395c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 msgr2=0x7f396c0cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 0x7f396c0cb2b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f396400d800 tx=0x7f396400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 shutdown_connections 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f395806b4d0 0x7f395806d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c0d3400 0x7f396c0cb2b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f396c0b2b50 0x7f396c0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 --2- 172.21.15.27:0/1420204519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f396c071540 0x7f396c1551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.166+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 >> 172.21.15.27:0/1420204519 conn(0x7f396c0d3ca0 msgr2=0x7f396c15ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.167+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 shutdown_connections 2024-03-06T07:26:08.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.167+0000 7f3971a88640 1 -- 172.21.15.27:0/1420204519 wait complete. 2024-03-06T07:26:08.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.296+0000 7f0e1082a640 1 Processor -- start 2024-03-06T07:26:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.296+0000 7f0e1082a640 1 -- start start 2024-03-06T07:26:08.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.297+0000 7f0e1082a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c15e2f0 0x7f0e0c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.297+0000 7f0e1082a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c15e6d0 con 0x7f0e0c06c050 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.297+0000 7f0e1082a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0ccd00 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.297+0000 7f0e1082a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0d0a10 con 0x7f0e0c15e2f0 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.297+0000 7f0e0ad9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0e0c06c050 0x7f0e0c0cdac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35126/0 (socket says 172.21.15.27:35126) 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.297+0000 7f0e0ad9d640 1 -- 172.21.15.27:0/3196771292 learned_addr learned my addr 172.21.15.27:0/3196771292 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.297+0000 7f0e0a59c640 1 --2- 172.21.15.27:0/3196771292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c15e2f0 0x7f0e0c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.298+0000 7f0e0959a640 1 -- 172.21.15.27:0/3196771292 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914500798 0 0) 0x7f0e0c15e6d0 con 0x7f0e0c06c050 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.298+0000 7f0e0959a640 1 -- 172.21.15.27:0/3196771292 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0df4003520 con 0x7f0e0c06c050 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.298+0000 7f0e0959a640 1 -- 172.21.15.27:0/3196771292 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850085195 0 0) 0x7f0e0c0ccd00 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.298+0000 7f0e0959a640 1 -- 172.21.15.27:0/3196771292 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e0c15e6d0 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.298+0000 7f0e0959a640 1 -- 172.21.15.27:0/3196771292 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 249561800 0 0) 0x7f0df4003520 con 0x7f0e0c06c050 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.298+0000 7f0e0a59c640 1 -- 172.21.15.27:0/3196771292 >> v1:172.21.15.69:6789/0 conn(0x7f0e0c06b4e0 legacy=0x7f0e0c06b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.298+0000 7f0e0a59c640 1 -- 172.21.15.27:0/3196771292 >> v1:172.21.15.27:6789/0 conn(0x7f0e0c06c050 legacy=0x7f0e0c0cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.298+0000 7f0e0a59c640 1 -- 172.21.15.27:0/3196771292 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c0ccfe0 con 0x7f0e0c15e2f0 2024-03-06T07:26:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.299+0000 7f0e0a59c640 1 --2- 172.21.15.27:0/3196771292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c15e2f0 0x7f0e0c0b41f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0097d0 tx=0x7f0dfc02b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66b8d92ec893448d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.299+0000 7f0e0959a640 1 -- 172.21.15.27:0/3196771292 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc037070 con 0x7f0e0c15e2f0 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.299+0000 7f0e0959a640 1 -- 172.21.15.27:0/3196771292 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc030510 con 0x7f0e0c15e2f0 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.299+0000 7f0e0959a640 1 -- 172.21.15.27:0/3196771292 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e040019f0 con 0x7f0e0c15e2f0 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.299+0000 7f0e1082a640 1 -- 172.21.15.27:0/3196771292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c15e2f0 msgr2=0x7f0e0c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.299+0000 7f0e1082a640 1 --2- 172.21.15.27:0/3196771292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c15e2f0 0x7f0e0c0b41f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0097d0 tx=0x7f0dfc02b6e0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.299+0000 7f0e1082a640 1 -- 172.21.15.27:0/3196771292 shutdown_connections 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.299+0000 7f0e1082a640 1 --2- 172.21.15.27:0/3196771292 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c15e2f0 0x7f0e0c0b41f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.299+0000 7f0e1082a640 1 -- 172.21.15.27:0/3196771292 >> 172.21.15.27:0/3196771292 conn(0x7f0e0c0b36d0 msgr2=0x7f0e0c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.300+0000 7f0e1082a640 1 -- 172.21.15.27:0/3196771292 shutdown_connections 2024-03-06T07:26:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.300+0000 7f0e1082a640 1 -- 172.21.15.27:0/3196771292 wait complete. 2024-03-06T07:26:08.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.301+0000 7f0e1082a640 1 Processor -- start 2024-03-06T07:26:08.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.301+0000 7f0e1082a640 1 -- start start 2024-03-06T07:26:08.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e1082a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c06b4e0 0x7f0e0c14be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e1082a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c06c050 0x7f0e0c0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e1082a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c15e2f0 0x7f0e0c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e1082a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0ce1d0 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e1082a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0cd5a0 con 0x7f0e0c15e2f0 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e1082a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0ce350 con 0x7f0e0c06c050 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c06b4e0 0x7f0e0c14be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c06b4e0 0x7f0e0c14be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43422/0 (socket says 172.21.15.27:43422) 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e0a59c640 1 -- 172.21.15.27:0/1211786381 learned_addr learned my addr 172.21.15.27:0/1211786381 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e0a59c640 1 -- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c06c050 msgr2=0x7f0e0c0c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e0a59c640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c06c050 0x7f0e0c0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e0a59c640 1 -- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c15e2f0 msgr2=0x7f0e0c0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e0a59c640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c15e2f0 0x7f0e0c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e0a59c640 1 -- 172.21.15.27:0/1211786381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c0b79e0 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e0a59c640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c06b4e0 0x7f0e0c14be60 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc02b720 tx=0x7f0dfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0dfaffd640 1 -- 172.21.15.27:0/1211786381 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc041020 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0dfaffd640 1 -- 172.21.15.27:0/1211786381 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc03c870 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c0b48f0 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0dfaffd640 1 -- 172.21.15.27:0/1211786381 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc037040 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.302+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c15b150 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.303+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c09fd30 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.304+0000 7f0dfaffd640 1 -- 172.21.15.27:0/1211786381 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0dfc044020 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.304+0000 7f0dfaffd640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dec06b4d0 0x7f0dec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.304+0000 7f0dfaffd640 1 -- 172.21.15.27:0/1211786381 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0dfc0ab780 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.305+0000 7f0e09d9b640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dec06b4d0 0x7f0dec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.305+0000 7f0e09d9b640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dec06b4d0 0x7f0dec06d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0e000010e0 tx=0x7f0e00007950 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.309+0000 7f0dfaffd640 1 -- 172.21.15.27:0/1211786381 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0dfc04b030 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.489+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0e0c04f410 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.493+0000 7f0dfaffd640 1 -- 172.21.15.27:0/1211786381 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0dfc03ae20 con 0x7f0e0c06b4e0 2024-03-06T07:26:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dec06b4d0 msgr2=0x7f0dec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dec06b4d0 0x7f0dec06d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0e000010e0 tx=0x7f0e00007950 comp rx=0 tx=0).stop 2024-03-06T07:26:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c06b4e0 msgr2=0x7f0e0c14be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c06b4e0 0x7f0e0c14be60 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc02b720 tx=0x7f0dfc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 shutdown_connections 2024-03-06T07:26:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0dec06b4d0 0x7f0dec06d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e0c15e2f0 0x7f0e0c0ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e0c06c050 0x7f0e0c0c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 --2- 172.21.15.27:0/1211786381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e0c06b4e0 0x7f0e0c14be60 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 >> 172.21.15.27:0/1211786381 conn(0x7f0e0c0b36d0 msgr2=0x7f0e0c070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 shutdown_connections 2024-03-06T07:26:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.495+0000 7f0e1082a640 1 -- 172.21.15.27:0/1211786381 wait complete. 2024-03-06T07:26:08.508 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:08.508 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:08.558 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:08.606 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:08.606 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:26:08.606 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:08.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:08.622 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:26:08.622 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:08.639 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:08.639 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:26:08.639 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:08.677 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:08.677 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:26:08.677 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:08.696 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:08.696 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:26:08.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:08.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.821+0000 7f8c2346f640 1 Processor -- start 2024-03-06T07:26:08.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.822+0000 7f8c2346f640 1 -- start start 2024-03-06T07:26:08.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.822+0000 7f8c2346f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c1c147930 0x7f8c1c072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.822+0000 7f8c2346f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c161ac0 0x7f8c1c161f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.822+0000 7f8c2346f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1c072a70 con 0x7f8c1c0ad8b0 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.822+0000 7f8c2346f640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1c162460 con 0x7f8c1c147930 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.822+0000 7f8c2346f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c0ae450 con 0x7f8c1c161ac0 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.823+0000 7f8c2120b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c1c147930 0x7f8c1c072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.823+0000 7f8c2120b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c1c147930 0x7f8c1c072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43424/0 (socket says 172.21.15.27:43424) 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.823+0000 7f8c2120b640 1 -- 172.21.15.27:0/1002154219 learned_addr learned my addr 172.21.15.27:0/1002154219 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.823+0000 7f8c20a0a640 1 --2- 172.21.15.27:0/1002154219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c161ac0 0x7f8c1c161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.823+0000 7f8c2120b640 1 -- 172.21.15.27:0/1002154219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c161ac0 msgr2=0x7f8c1c161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.823+0000 7f8c2120b640 1 --2- 172.21.15.27:0/1002154219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c161ac0 0x7f8c1c161f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.823+0000 7f8c2120b640 1 -- 172.21.15.27:0/1002154219 >> v1:172.21.15.69:6789/0 conn(0x7f8c1c0ad8b0 legacy=0x7f8c1c0add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.823+0000 7f8c2120b640 1 -- 172.21.15.27:0/1002154219 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c06dbe0 con 0x7f8c1c147930 2024-03-06T07:26:08.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c2120b640 1 --2- 172.21.15.27:0/1002154219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 0x7f8c1c072530 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009870 tx=0x7f8c1802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab71b7f2c01f77a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c0ffff640 1 -- 172.21.15.27:0/1002154219 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18034070 con 0x7f8c1c147930 2024-03-06T07:26:08.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c0ffff640 1 -- 172.21.15.27:0/1002154219 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c18030840 con 0x7f8c1c147930 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c0ffff640 1 -- 172.21.15.27:0/1002154219 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18030c10 con 0x7f8c1c147930 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c2346f640 1 -- 172.21.15.27:0/1002154219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 msgr2=0x7f8c1c072530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c2346f640 1 --2- 172.21.15.27:0/1002154219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 0x7f8c1c072530 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009870 tx=0x7f8c1802b980 comp rx=0 tx=0).stop 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c2346f640 1 -- 172.21.15.27:0/1002154219 shutdown_connections 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c2346f640 1 --2- 172.21.15.27:0/1002154219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c161ac0 0x7f8c1c161f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c2346f640 1 --2- 172.21.15.27:0/1002154219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 0x7f8c1c072530 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.824+0000 7f8c2346f640 1 -- 172.21.15.27:0/1002154219 >> 172.21.15.27:0/1002154219 conn(0x7f8c1c06cd20 msgr2=0x7f8c1c06d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.825+0000 7f8c2346f640 1 -- 172.21.15.27:0/1002154219 shutdown_connections 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.825+0000 7f8c2346f640 1 -- 172.21.15.27:0/1002154219 wait complete. 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.826+0000 7f8c2346f640 1 Processor -- start 2024-03-06T07:26:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.826+0000 7f8c2346f640 1 -- start start 2024-03-06T07:26:08.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.826+0000 7f8c2346f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0ad8b0 0x7f8c1c13d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.826+0000 7f8c2346f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 0x7f8c1c143d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.826+0000 7f8c2346f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c1c161ac0 0x7f8c1c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.826+0000 7f8c2346f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c072010 con 0x7f8c1c147930 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.826+0000 7f8c2346f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c071d30 con 0x7f8c1c161ac0 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.826+0000 7f8c2346f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c147d10 con 0x7f8c1c0ad8b0 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c20a0a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 0x7f8c1c143d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c20a0a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 0x7f8c1c143d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43428/0 (socket says 172.21.15.27:43428) 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c20a0a640 1 -- 172.21.15.27:0/2153872320 learned_addr learned my addr 172.21.15.27:0/2153872320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c2120b640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0ad8b0 0x7f8c1c13d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c21a0c640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c1c161ac0 0x7f8c1c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c20a0a640 1 -- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0ad8b0 msgr2=0x7f8c1c13d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c20a0a640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0ad8b0 0x7f8c1c13d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c20a0a640 1 -- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c1c161ac0 msgr2=0x7f8c1c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c20a0a640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c1c161ac0 0x7f8c1c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c20a0a640 1 -- 172.21.15.27:0/2153872320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c1442a0 con 0x7f8c1c147930 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c2120b640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0ad8b0 0x7f8c1c13d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c21a0c640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c1c161ac0 0x7f8c1c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c20a0a640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 0x7f8c1c143d60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000d5d0 tx=0x7f8c1000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c0dffb640 1 -- 172.21.15.27:0/2153872320 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c100125f0 con 0x7f8c1c147930 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c0dffb640 1 -- 172.21.15.27:0/2153872320 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c10019070 con 0x7f8c1c147930 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1c141380 con 0x7f8c1c147930 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.827+0000 7f8c0dffb640 1 -- 172.21.15.27:0/2153872320 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c100129f0 con 0x7f8c1c147930 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.828+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1c144830 con 0x7f8c1c147930 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.828+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1c067ff0 con 0x7f8c1c147930 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.829+0000 7f8c0dffb640 1 -- 172.21.15.27:0/2153872320 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c10012b90 con 0x7f8c1c147930 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.829+0000 7f8c0dffb640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c0006b500 0x7f8c0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.829+0000 7f8c0dffb640 1 -- 172.21.15.27:0/2153872320 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8c1008a270 con 0x7f8c1c147930 2024-03-06T07:26:08.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.829+0000 7f8c2120b640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c0006b500 0x7f8c0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.830+0000 7f8c2120b640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c0006b500 0x7f8c0006d9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8c18036000 tx=0x7f8c1803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:08.833+0000 7f8c0dffb640 1 -- 172.21.15.27:0/2153872320 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c10059480 con 0x7f8c1c147930 2024-03-06T07:26:09.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.053+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c1c0cc5d0 con 0x7f8c1c147930 2024-03-06T07:26:09.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.053+0000 7f8c0dffb640 1 -- 172.21.15.27:0/2153872320 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f8c10058d80 con 0x7f8c1c147930 2024-03-06T07:26:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:09.056 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5324,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/3885016831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3885016831},{"type":"v1","addr":"172.21.15.69:6836","nonce":3885016831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5327,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3510003111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":3510003111},{"type":"v1","addr":"172.21.15.69:6837","nonce":3510003111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5692,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3053251134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3053251134},{"type":"v1","addr":"172.21.15.27:6835","nonce":3053251134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1071027798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1071027798},{"type":"v1","addr":"172.21.15.27:6833","nonce":1071027798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[]} 2024-03-06T07:26:09.056 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c0006b500 msgr2=0x7f8c0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c0006b500 0x7f8c0006d9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8c18036000 tx=0x7f8c1803b040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 msgr2=0x7f8c1c143d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 0x7f8c1c143d60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000d5d0 tx=0x7f8c1000da90 comp rx=0 tx=0).stop 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 shutdown_connections 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c0006b500 0x7f8c0006d9c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c1c161ac0 0x7f8c1c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c1c147930 0x7f8c1c143d60 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 --2- 172.21.15.27:0/2153872320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0ad8b0 0x7f8c1c13d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 >> 172.21.15.27:0/2153872320 conn(0x7f8c1c06cd20 msgr2=0x7f8c1c0ae630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 shutdown_connections 2024-03-06T07:26:09.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:09.055+0000 7f8c2346f640 1 -- 172.21.15.27:0/2153872320 wait complete. 2024-03-06T07:26:09.068 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:09.069 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:26:09.069 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:09.069 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:09.069 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:09.069 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:26:09.069 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:09.070 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:09.070 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:09.070 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:26:09.070 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:09.070 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:09.070 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:09.070 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:26:09.071 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:09.071 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:09.078 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:09.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:09.079 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:09.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:09.083 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:09.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:09.122 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:09.122 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:09.122 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:09.123 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:09.126 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:09.126 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:09.202 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:26:09.202 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:26:09.208 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:26:09.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-06T07:26:15.083 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:15.084 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:15.087 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:15.087 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:15.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:26:15.169 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:26:15.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.207+0000 7fc8b2815640 1 Processor -- start 2024-03-06T07:26:15.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.207+0000 7fc8b2815640 1 -- start start 2024-03-06T07:26:15.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.207+0000 7fc8b2815640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8ac15bac0 0x7fc8ac15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.207+0000 7fc8b2815640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc8ac0aee10 0x7fc8ac070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.207+0000 7fc8b2815640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ac0729b0 con 0x7fc8ac06d400 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8b2815640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ac06cf70 con 0x7fc8ac15bac0 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8b2815640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ac06c920 con 0x7fc8ac0aee10 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8abfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8ac15bac0 0x7fc8ac15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8ab7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc8ac06d400 0x7fc8ac0ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38808/0 (socket says 172.21.15.27:38808) 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8ab7fe640 1 -- 172.21.15.27:0/3000068099 learned_addr learned my addr 172.21.15.27:0/3000068099 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8b0db2640 1 --2- 172.21.15.27:0/3000068099 >> v2:172.21.15.69:3300/0 conn(0x7fc8ac0aee10 0x7fc8ac070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8abfff640 1 -- 172.21.15.27:0/3000068099 >> v2:172.21.15.69:3300/0 conn(0x7fc8ac0aee10 msgr2=0x7fc8ac070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8abfff640 1 --2- 172.21.15.27:0/3000068099 >> v2:172.21.15.69:3300/0 conn(0x7fc8ac0aee10 0x7fc8ac070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8abfff640 1 -- 172.21.15.27:0/3000068099 >> v1:172.21.15.27:6789/0 conn(0x7fc8ac06d400 legacy=0x7fc8ac0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8abfff640 1 -- 172.21.15.27:0/3000068099 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ac06cc00 con 0x7fc8ac15bac0 2024-03-06T07:26:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.208+0000 7fc8aaffd640 1 -- 172.21.15.27:0/3000068099 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283811035 0 0) 0x7fc8ac0729b0 con 0x7fc8ac06d400 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.209+0000 7fc8abfff640 1 --2- 172.21.15.27:0/3000068099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac15bac0 0x7fc8ac15cbe0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac06d0f0 tx=0x7fc89c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d898e98cdbba186c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.209+0000 7fc8aaffd640 1 -- 172.21.15.27:0/3000068099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c035020 con 0x7fc8ac15bac0 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.209+0000 7fc8aaffd640 1 -- 172.21.15.27:0/3000068099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89c031830 con 0x7fc8ac15bac0 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.209+0000 7fc8aaffd640 1 -- 172.21.15.27:0/3000068099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c031c00 con 0x7fc8ac15bac0 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.210+0000 7fc8b2815640 1 -- 172.21.15.27:0/3000068099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac15bac0 msgr2=0x7fc8ac15cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.210+0000 7fc8b2815640 1 --2- 172.21.15.27:0/3000068099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac15bac0 0x7fc8ac15cbe0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac06d0f0 tx=0x7fc89c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.210+0000 7fc8b2815640 1 -- 172.21.15.27:0/3000068099 shutdown_connections 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.210+0000 7fc8b2815640 1 --2- 172.21.15.27:0/3000068099 >> v2:172.21.15.69:3300/0 conn(0x7fc8ac0aee10 0x7fc8ac070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.210+0000 7fc8b2815640 1 --2- 172.21.15.27:0/3000068099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac15bac0 0x7fc8ac15cbe0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.210+0000 7fc8b2815640 1 -- 172.21.15.27:0/3000068099 >> 172.21.15.27:0/3000068099 conn(0x7fc8ac066d00 msgr2=0x7fc8ac0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.210+0000 7fc8b2815640 1 -- 172.21.15.27:0/3000068099 shutdown_connections 2024-03-06T07:26:15.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.210+0000 7fc8b2815640 1 -- 172.21.15.27:0/3000068099 wait complete. 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.211+0000 7fc8b2815640 1 Processor -- start 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.211+0000 7fc8b2815640 1 -- start start 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8b2815640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8ac06d400 0x7fc8ac0b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8b2815640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac0aee10 0x7fc8ac1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8b2815640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8ac15bac0 0x7fc8ac0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8b2815640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac15c020 con 0x7fc8ac0aee10 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8b2815640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac15bea0 con 0x7fc8ac06d400 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8b2815640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac072580 con 0x7fc8ac15bac0 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8ab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac0aee10 0x7fc8ac1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8ab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac0aee10 0x7fc8ac1482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34630/0 (socket says 172.21.15.27:34630) 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8ab7fe640 1 -- 172.21.15.27:0/2186790992 learned_addr learned my addr 172.21.15.27:0/2186790992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8abfff640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8ac06d400 0x7fc8ac0b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8b0db2640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8ac15bac0 0x7fc8ac0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8ab7fe640 1 -- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8ac15bac0 msgr2=0x7fc8ac0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8ab7fe640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8ac15bac0 0x7fc8ac0ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8ab7fe640 1 -- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8ac06d400 msgr2=0x7fc8ac0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8ab7fe640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8ac06d400 0x7fc8ac0b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8ab7fe640 1 -- 172.21.15.27:0/2186790992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ac15ac70 con 0x7fc8ac0aee10 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8abfff640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8ac06d400 0x7fc8ac0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.212+0000 7fc8ab7fe640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac0aee10 0x7fc8ac1482d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000c430 tx=0x7fc8a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.213+0000 7fc8a8ff9640 1 -- 172.21.15.27:0/2186790992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00074e0 con 0x7fc8ac0aee10 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.213+0000 7fc8a8ff9640 1 -- 172.21.15.27:0/2186790992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a00047e0 con 0x7fc8ac0aee10 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.213+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ac15af50 con 0x7fc8ac0aee10 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.213+0000 7fc8a8ff9640 1 -- 172.21.15.27:0/2186790992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0004bd0 con 0x7fc8ac0aee10 2024-03-06T07:26:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.213+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ac148810 con 0x7fc8ac0aee10 2024-03-06T07:26:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.214+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ac067ff0 con 0x7fc8ac0aee10 2024-03-06T07:26:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.214+0000 7fc8a8ff9640 1 -- 172.21.15.27:0/2186790992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8a0011070 con 0x7fc8ac0aee10 2024-03-06T07:26:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.215+0000 7fc8a8ff9640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc89806b4d0 0x7fc89806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.215+0000 7fc8a8ff9640 1 -- 172.21.15.27:0/2186790992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc8a0089970 con 0x7fc8ac0aee10 2024-03-06T07:26:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.215+0000 7fc8abfff640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc89806b4d0 0x7fc89806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.215+0000 7fc8abfff640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc89806b4d0 0x7fc89806d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc89c036000 tx=0x7fc89c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.219+0000 7fc8a8ff9640 1 -- 172.21.15.27:0/2186790992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8a0058530 con 0x7fc8ac0aee10 2024-03-06T07:26:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:15.413+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc8ac15e680 con 0x7fc8ac0aee10 2024-03-06T07:26:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.202+0000 7fc8a8ff9640 1 -- 172.21.15.27:0/2186790992 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v66) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc8a005c5d0 con 0x7fc8ac0aee10 2024-03-06T07:26:16.205 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc89806b4d0 msgr2=0x7fc89806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc89806b4d0 0x7fc89806d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc89c036000 tx=0x7fc89c03d040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac0aee10 msgr2=0x7fc8ac1482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac0aee10 0x7fc8ac1482d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000c430 tx=0x7fc8a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 shutdown_connections 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc89806b4d0 0x7fc89806d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8ac15bac0 0x7fc8ac0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8ac0aee10 0x7fc8ac1482d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 --2- 172.21.15.27:0/2186790992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8ac06d400 0x7fc8ac0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.204+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 >> 172.21.15.27:0/2186790992 conn(0x7fc8ac066d00 msgr2=0x7fc8ac0cfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.205+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 shutdown_connections 2024-03-06T07:26:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.205+0000 7fc8b2815640 1 -- 172.21.15.27:0/2186790992 wait complete. 2024-03-06T07:26:16.221 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:26:16.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.331+0000 7f5b2a75e640 1 Processor -- start 2024-03-06T07:26:16.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.331+0000 7f5b2a75e640 1 -- start start 2024-03-06T07:26:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.331+0000 7f5b2a75e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5b2415df00 0x7f5b240d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.331+0000 7f5b2a75e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b9a30 0x7f5b240d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.331+0000 7f5b2a75e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240cea30 con 0x7f5b240b95a0 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.331+0000 7f5b2a75e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240d2af0 con 0x7f5b2415df00 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.331+0000 7f5b2a75e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240cf3c0 con 0x7f5b240b9a30 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b28cfb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b9a30 0x7f5b240d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b23fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5b2415df00 0x7f5b240d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b237fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5b240b95a0 0x7f5b240b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:46610/0 (socket says 172.21.15.27:46610) 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b28cfb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b9a30 0x7f5b240d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:47382/0 (socket says 172.21.15.27:47382) 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b237fe640 1 -- 172.21.15.27:0/2664912427 learned_addr learned my addr 172.21.15.27:0/2664912427 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b23fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5b2415df00 0x7f5b240d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:52896/0 (socket says 172.21.15.27:52896) 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b23fff640 1 -- 172.21.15.27:0/2664912427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b9a30 msgr2=0x7f5b240d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b23fff640 1 --2- 172.21.15.27:0/2664912427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b9a30 0x7f5b240d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b23fff640 1 -- 172.21.15.27:0/2664912427 >> v1:172.21.15.69:6789/0 conn(0x7f5b240b95a0 legacy=0x7f5b240b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b23fff640 1 -- 172.21.15.27:0/2664912427 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240cf0e0 con 0x7f5b2415df00 2024-03-06T07:26:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.332+0000 7f5b28cfb640 1 --2- 172.21.15.27:0/2664912427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b9a30 0x7f5b240d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:16.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.333+0000 7f5b23fff640 1 --2- 172.21.15.27:0/2664912427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b2415df00 0x7f5b240d25b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5b140097a0 tx=0x7f5b1402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f903c9306b0fcb15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.333+0000 7f5b22ffd640 1 -- 172.21.15.27:0/2664912427 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14035020 con 0x7f5b2415df00 2024-03-06T07:26:16.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.333+0000 7f5b22ffd640 1 -- 172.21.15.27:0/2664912427 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14031830 con 0x7f5b2415df00 2024-03-06T07:26:16.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.333+0000 7f5b22ffd640 1 -- 172.21.15.27:0/2664912427 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14031c00 con 0x7f5b2415df00 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.334+0000 7f5b2a75e640 1 -- 172.21.15.27:0/2664912427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b2415df00 msgr2=0x7f5b240d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.334+0000 7f5b2a75e640 1 --2- 172.21.15.27:0/2664912427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b2415df00 0x7f5b240d25b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5b140097a0 tx=0x7f5b1402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.334+0000 7f5b2a75e640 1 -- 172.21.15.27:0/2664912427 shutdown_connections 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.334+0000 7f5b2a75e640 1 --2- 172.21.15.27:0/2664912427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b9a30 0x7f5b240d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.334+0000 7f5b2a75e640 1 --2- 172.21.15.27:0/2664912427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b2415df00 0x7f5b240d25b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.334+0000 7f5b2a75e640 1 -- 172.21.15.27:0/2664912427 >> 172.21.15.27:0/2664912427 conn(0x7f5b24071960 msgr2=0x7f5b24071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.334+0000 7f5b2a75e640 1 -- 172.21.15.27:0/2664912427 shutdown_connections 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.334+0000 7f5b2a75e640 1 -- 172.21.15.27:0/2664912427 wait complete. 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.335+0000 7f5b2a75e640 1 Processor -- start 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.335+0000 7f5b2a75e640 1 -- start start 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b2a75e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b95a0 0x7f5b240c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b2a75e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2415df00 0x7f5b241f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b2a75e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b240d3ac0 0x7f5b241f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b2a75e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240cec70 con 0x7f5b2415df00 2024-03-06T07:26:16.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b2a75e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240ce750 con 0x7f5b240d3ac0 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b2a75e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240cedf0 con 0x7f5b240b95a0 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2415df00 0x7f5b241f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2415df00 0x7f5b241f6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34642/0 (socket says 172.21.15.27:34642) 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b237fe640 1 -- 172.21.15.27:0/1054953572 learned_addr learned my addr 172.21.15.27:0/1054953572 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b23fff640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b95a0 0x7f5b240c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b28cfb640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b240d3ac0 0x7f5b241f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b237fe640 1 -- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b95a0 msgr2=0x7f5b240c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b237fe640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b95a0 0x7f5b240c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b237fe640 1 -- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b240d3ac0 msgr2=0x7f5b241f2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b237fe640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b240d3ac0 0x7f5b241f2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.336+0000 7f5b237fe640 1 -- 172.21.15.27:0/1054953572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240ca850 con 0x7f5b2415df00 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.337+0000 7f5b237fe640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2415df00 0x7f5b241f6c00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00c430 tx=0x7f5b0c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.337+0000 7f5b20ff9640 1 -- 172.21.15.27:0/1054953572 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c017070 con 0x7f5b2415df00 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.337+0000 7f5b20ff9640 1 -- 172.21.15.27:0/1054953572 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0c005ce0 con 0x7f5b2415df00 2024-03-06T07:26:16.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.337+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b241f7290 con 0x7f5b2415df00 2024-03-06T07:26:16.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.337+0000 7f5b20ff9640 1 -- 172.21.15.27:0/1054953572 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c0072f0 con 0x7f5b2415df00 2024-03-06T07:26:16.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.337+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b241f7850 con 0x7f5b2415df00 2024-03-06T07:26:16.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.338+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b24067ff0 con 0x7f5b2415df00 2024-03-06T07:26:16.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.338+0000 7f5b20ff9640 1 -- 172.21.15.27:0/1054953572 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b0c01a070 con 0x7f5b2415df00 2024-03-06T07:26:16.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.339+0000 7f5b20ff9640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5b0406b4d0 0x7f5b0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.339+0000 7f5b20ff9640 1 -- 172.21.15.27:0/1054953572 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5b0c0890d0 con 0x7f5b2415df00 2024-03-06T07:26:16.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.339+0000 7f5b23fff640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5b0406b4d0 0x7f5b0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.339+0000 7f5b23fff640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5b0406b4d0 0x7f5b0406d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5b14036000 tx=0x7f5b1403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.343+0000 7f5b20ff9640 1 -- 172.21.15.27:0/1054953572 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b240ca850 con 0x7f5b2415df00 2024-03-06T07:26:16.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-06T07:26:16.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.590+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5b2406b780 con 0x7f5b2415df00 2024-03-06T07:26:16.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.591+0000 7f5b20ff9640 1 -- 172.21.15.27:0/1054953572 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5b0c057ce0 con 0x7f5b2415df00 2024-03-06T07:26:16.593 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g==", 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ==", 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg==", 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.595 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ==", 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.596 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw==", 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA==", 2024-03-06T07:26:16.597 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA==", 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.598 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg==", 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A==", 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.599 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw==", 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA==", 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A==", 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA==", 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ==", 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ==", 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:26:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg==", 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlkc2iEhAA6BuYba0FzXeVjGPOP8sgyQ==", 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw==", 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.605 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg==", 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:16.606 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:26:16.607 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:26:16.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5b0406b4d0 msgr2=0x7f5b0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5b0406b4d0 0x7f5b0406d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5b14036000 tx=0x7f5b1403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2415df00 msgr2=0x7f5b241f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2415df00 0x7f5b241f6c00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00c430 tx=0x7f5b0c00c830 comp rx=0 tx=0).stop 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 shutdown_connections 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5b0406b4d0 0x7f5b0406d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b240d3ac0 0x7f5b241f2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2415df00 0x7f5b241f6c00 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 --2- 172.21.15.27:0/1054953572 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5b240b95a0 0x7f5b240c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 >> 172.21.15.27:0/1054953572 conn(0x7f5b24071960 msgr2=0x7f5b240d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 shutdown_connections 2024-03-06T07:26:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.593+0000 7f5b2a75e640 1 -- 172.21.15.27:0/1054953572 wait complete. 2024-03-06T07:26:16.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.714+0000 7fc696ec9640 1 Processor -- start 2024-03-06T07:26:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.714+0000 7fc696ec9640 1 -- start start 2024-03-06T07:26:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc696ec9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc69006e700 0x7fc69015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc696ec9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 0x7fc69006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc696ec9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6900b4160 con 0x7fc6900b55d0 2024-03-06T07:26:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc696ec9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6900b5b40 con 0x7fc69006e700 2024-03-06T07:26:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc696ec9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6900b3ba0 con 0x7fc690072660 2024-03-06T07:26:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc694c65640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc69006e700 0x7fc69015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc694c65640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc69006e700 0x7fc69015ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34650/0 (socket says 172.21.15.27:34650) 2024-03-06T07:26:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc694c65640 1 -- 172.21.15.27:0/889805491 learned_addr learned my addr 172.21.15.27:0/889805491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc68ffff640 1 --2- 172.21.15.27:0/889805491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 0x7fc69006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc694c65640 1 -- 172.21.15.27:0/889805491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 msgr2=0x7fc69006d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc694c65640 1 --2- 172.21.15.27:0/889805491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 0x7fc69006d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc694c65640 1 -- 172.21.15.27:0/889805491 >> v1:172.21.15.69:6789/0 conn(0x7fc6900b55d0 legacy=0x7fc690071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.715+0000 7fc694c65640 1 -- 172.21.15.27:0/889805491 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6900b3e80 con 0x7fc69006e700 2024-03-06T07:26:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.716+0000 7fc694c65640 1 --2- 172.21.15.27:0/889805491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69006e700 0x7fc69015ef80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc688009870 tx=0x7fc68802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799eb71a5f954228 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.716+0000 7fc68f7fe640 1 -- 172.21.15.27:0/889805491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688035020 con 0x7fc69006e700 2024-03-06T07:26:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.716+0000 7fc68f7fe640 1 -- 172.21.15.27:0/889805491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc688031640 con 0x7fc69006e700 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.716+0000 7fc68f7fe640 1 -- 172.21.15.27:0/889805491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68802d040 con 0x7fc69006e700 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.717+0000 7fc696ec9640 1 -- 172.21.15.27:0/889805491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69006e700 msgr2=0x7fc69015ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.717+0000 7fc696ec9640 1 --2- 172.21.15.27:0/889805491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69006e700 0x7fc69015ef80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc688009870 tx=0x7fc68802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.717+0000 7fc696ec9640 1 -- 172.21.15.27:0/889805491 shutdown_connections 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.717+0000 7fc696ec9640 1 --2- 172.21.15.27:0/889805491 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 0x7fc69006d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.717+0000 7fc696ec9640 1 --2- 172.21.15.27:0/889805491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69006e700 0x7fc69015ef80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.717+0000 7fc696ec9640 1 -- 172.21.15.27:0/889805491 >> 172.21.15.27:0/889805491 conn(0x7fc6900b2720 msgr2=0x7fc6900b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.717+0000 7fc696ec9640 1 -- 172.21.15.27:0/889805491 shutdown_connections 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.717+0000 7fc696ec9640 1 -- 172.21.15.27:0/889805491 wait complete. 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.718+0000 7fc696ec9640 1 Processor -- start 2024-03-06T07:26:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.718+0000 7fc696ec9640 1 -- start start 2024-03-06T07:26:16.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc696ec9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc69006e700 0x7fc69015d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc696ec9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 0x7fc69015b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc696ec9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6900b55d0 0x7fc69015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc696ec9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69006ec60 con 0x7fc6900b55d0 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc696ec9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69006eae0 con 0x7fc69006e700 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc696ec9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc690072090 con 0x7fc690072660 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc695466640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6900b55d0 0x7fc69015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc695466640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6900b55d0 0x7fc69015b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34652/0 (socket says 172.21.15.27:34652) 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc695466640 1 -- 172.21.15.27:0/1751288985 learned_addr learned my addr 172.21.15.27:0/1751288985 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc694c65640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc69006e700 0x7fc69015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc68ffff640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 0x7fc69015b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc695466640 1 -- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 msgr2=0x7fc69015b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc695466640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 0x7fc69015b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc695466640 1 -- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc69006e700 msgr2=0x7fc69015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc695466640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc69006e700 0x7fc69015d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.719+0000 7fc695466640 1 -- 172.21.15.27:0/1751288985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6900cd4c0 con 0x7fc6900b55d0 2024-03-06T07:26:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.720+0000 7fc695466640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6900b55d0 0x7fc69015b6d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc67800b410 tx=0x7fc67800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.720+0000 7fc68d7fa640 1 -- 172.21.15.27:0/1751288985 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc678015030 con 0x7fc6900b55d0 2024-03-06T07:26:16.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.720+0000 7fc68d7fa640 1 -- 172.21.15.27:0/1751288985 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc678004020 con 0x7fc6900b55d0 2024-03-06T07:26:16.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.720+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6900cd760 con 0x7fc6900b55d0 2024-03-06T07:26:16.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.720+0000 7fc68d7fa640 1 -- 172.21.15.27:0/1751288985 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc678004410 con 0x7fc6900b55d0 2024-03-06T07:26:16.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.720+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6900cdd20 con 0x7fc6900b55d0 2024-03-06T07:26:16.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.721+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6900cdf20 con 0x7fc6900b55d0 2024-03-06T07:26:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.722+0000 7fc68d7fa640 1 -- 172.21.15.27:0/1751288985 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc678018070 con 0x7fc6900b55d0 2024-03-06T07:26:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.722+0000 7fc68d7fa640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67006b4d0 0x7fc67006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.722+0000 7fc68d7fa640 1 -- 172.21.15.27:0/1751288985 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc6780897c0 con 0x7fc6900b55d0 2024-03-06T07:26:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.722+0000 7fc694c65640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67006b4d0 0x7fc67006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.723+0000 7fc694c65640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67006b4d0 0x7fc67006d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc688036000 tx=0x7fc68803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.726+0000 7fc68d7fa640 1 -- 172.21.15.27:0/1751288985 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6900cdf20 con 0x7fc6900b55d0 2024-03-06T07:26:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.908+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc690067ff0 con 0x7fc6900b55d0 2024-03-06T07:26:16.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.909+0000 7fc68d7fa640 1 -- 172.21.15.27:0/1751288985 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+12819 (secure 0 0 0) 0x7fc678058340 con 0x7fc6900b55d0 2024-03-06T07:26:16.944 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:16.944 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:16.195424+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:16.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.910+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67006b4d0 msgr2=0x7fc67006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.910+0000 7fc696ec9640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67006b4d0 0x7fc67006d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc688036000 tx=0x7fc68803c040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.910+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6900b55d0 msgr2=0x7fc69015b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.910+0000 7fc696ec9640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6900b55d0 0x7fc69015b6d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc67800b410 tx=0x7fc67800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.911+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 shutdown_connections 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.911+0000 7fc696ec9640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc67006b4d0 0x7fc67006d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.911+0000 7fc696ec9640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6900b55d0 0x7fc69015b6d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.911+0000 7fc696ec9640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc690072660 0x7fc69015b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.911+0000 7fc696ec9640 1 --2- 172.21.15.27:0/1751288985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc69006e700 0x7fc69015d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.911+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 >> 172.21.15.27:0/1751288985 conn(0x7fc6900b2720 msgr2=0x7fc69015f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.911+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 shutdown_connections 2024-03-06T07:26:16.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:16.911+0000 7fc696ec9640 1 -- 172.21.15.27:0/1751288985 wait complete. 2024-03-06T07:26:16.947 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:16.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:17.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.055+0000 7f83cbe81640 1 Processor -- start 2024-03-06T07:26:17.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.055+0000 7f83cbe81640 1 -- start start 2024-03-06T07:26:17.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83cbe81640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83c415dbd0 0x7f83c415ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83cbe81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c40b5840 0x7f83c40b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83cbe81640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c415f3b0 con 0x7f83c415dfb0 2024-03-06T07:26:17.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83cbe81640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c406f070 con 0x7f83c415dbd0 2024-03-06T07:26:17.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83cbe81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c4071940 con 0x7f83c40b5840 2024-03-06T07:26:17.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83c9c1d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83c415dbd0 0x7f83c415ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83ca41e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f83c415dfb0 0x7f83c4161a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38812/0 (socket says 172.21.15.27:38812) 2024-03-06T07:26:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83ca41e640 1 -- 172.21.15.27:0/1047119026 learned_addr learned my addr 172.21.15.27:0/1047119026 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83c8c1b640 1 -- 172.21.15.27:0/1047119026 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458114296 0 0) 0x7f83c415f3b0 con 0x7f83c415dfb0 2024-03-06T07:26:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83c8c1b640 1 -- 172.21.15.27:0/1047119026 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b0003520 con 0x7f83c415dfb0 2024-03-06T07:26:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83c941c640 1 --2- 172.21.15.27:0/1047119026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c40b5840 0x7f83c40b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83c9c1d640 1 -- 172.21.15.27:0/1047119026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c40b5840 msgr2=0x7f83c40b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83c9c1d640 1 --2- 172.21.15.27:0/1047119026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c40b5840 0x7f83c40b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83c9c1d640 1 -- 172.21.15.27:0/1047119026 >> v1:172.21.15.27:6789/0 conn(0x7f83c415dfb0 legacy=0x7f83c4161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83c9c1d640 1 -- 172.21.15.27:0/1047119026 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c40b5db0 con 0x7f83c415dbd0 2024-03-06T07:26:17.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.056+0000 7f83c941c640 1 --2- 172.21.15.27:0/1047119026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c40b5840 0x7f83c40b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:17.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.057+0000 7f83c9c1d640 1 --2- 172.21.15.27:0/1047119026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 0x7f83c415ee70 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f83c00097a0 tx=0x7f83c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e846116a7f2e48ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.057+0000 7f83c8c1b640 1 -- 172.21.15.27:0/1047119026 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0035020 con 0x7f83c415dbd0 2024-03-06T07:26:17.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.057+0000 7f83c8c1b640 1 -- 172.21.15.27:0/1047119026 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c0031830 con 0x7f83c415dbd0 2024-03-06T07:26:17.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.057+0000 7f83c8c1b640 1 -- 172.21.15.27:0/1047119026 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0031c00 con 0x7f83c415dbd0 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.058+0000 7f83cbe81640 1 -- 172.21.15.27:0/1047119026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 msgr2=0x7f83c415ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.058+0000 7f83cbe81640 1 --2- 172.21.15.27:0/1047119026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 0x7f83c415ee70 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f83c00097a0 tx=0x7f83c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.058+0000 7f83cbe81640 1 -- 172.21.15.27:0/1047119026 shutdown_connections 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.058+0000 7f83cbe81640 1 --2- 172.21.15.27:0/1047119026 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c40b5840 0x7f83c40b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.058+0000 7f83cbe81640 1 --2- 172.21.15.27:0/1047119026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 0x7f83c415ee70 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.058+0000 7f83cbe81640 1 -- 172.21.15.27:0/1047119026 >> 172.21.15.27:0/1047119026 conn(0x7f83c40b26b0 msgr2=0x7f83c40b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.058+0000 7f83cbe81640 1 -- 172.21.15.27:0/1047119026 shutdown_connections 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.058+0000 7f83cbe81640 1 -- 172.21.15.27:0/1047119026 wait complete. 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.059+0000 7f83cbe81640 1 Processor -- start 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.059+0000 7f83cbe81640 1 -- start start 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83cbe81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83c40b5840 0x7f83c40af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83cbe81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 0x7f83c40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83cbe81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c415dfb0 0x7f83c406e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83cbe81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40721c0 con 0x7f83c415dbd0 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83cbe81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c4071ee0 con 0x7f83c40b5840 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83cbe81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40726e0 con 0x7f83c415dfb0 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83c941c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 0x7f83c40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83c941c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 0x7f83c40cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34670/0 (socket says 172.21.15.27:34670) 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83c941c640 1 -- 172.21.15.27:0/3662792857 learned_addr learned my addr 172.21.15.27:0/3662792857 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83c9c1d640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83c40b5840 0x7f83c40af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83ca41e640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c415dfb0 0x7f83c406e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83c941c640 1 -- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c415dfb0 msgr2=0x7f83c406e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83c941c640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c415dfb0 0x7f83c406e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83c941c640 1 -- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83c40b5840 msgr2=0x7f83c40af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83c941c640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83c40b5840 0x7f83c40af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.060+0000 7f83c941c640 1 -- 172.21.15.27:0/3662792857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c40c5ee0 con 0x7f83c415dbd0 2024-03-06T07:26:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.061+0000 7f83c941c640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 0x7f83c40cf0d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f83b400c7f0 tx=0x7f83b400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.061+0000 7f83ba7fc640 1 -- 172.21.15.27:0/3662792857 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4015070 con 0x7f83c415dbd0 2024-03-06T07:26:17.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.061+0000 7f83ba7fc640 1 -- 172.21.15.27:0/3662792857 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b4007820 con 0x7f83c415dbd0 2024-03-06T07:26:17.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.061+0000 7f83ba7fc640 1 -- 172.21.15.27:0/3662792857 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4007bd0 con 0x7f83c415dbd0 2024-03-06T07:26:17.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.061+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c40c61c0 con 0x7f83c415dbd0 2024-03-06T07:26:17.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.061+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c40c6780 con 0x7f83c415dbd0 2024-03-06T07:26:17.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.062+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c4073380 con 0x7f83c415dbd0 2024-03-06T07:26:17.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.062+0000 7f83ba7fc640 1 -- 172.21.15.27:0/3662792857 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83b4018070 con 0x7f83c415dbd0 2024-03-06T07:26:17.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.063+0000 7f83ba7fc640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a806b4d0 0x7f83a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.063+0000 7f83ba7fc640 1 -- 172.21.15.27:0/3662792857 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f83b4089ee0 con 0x7f83c415dbd0 2024-03-06T07:26:17.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.063+0000 7f83c9c1d640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a806b4d0 0x7f83a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.063+0000 7f83c9c1d640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a806b4d0 0x7f83a806d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f83c0036000 tx=0x7f83c0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.067+0000 7f83ba7fc640 1 -- 172.21.15.27:0/3662792857 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83b4058af0 con 0x7f83c415dbd0 2024-03-06T07:26:17.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.249+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83c41f3dc0 con 0x7f83c415dbd0 2024-03-06T07:26:17.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.250+0000 7f83ba7fc640 1 -- 172.21.15.27:0/3662792857 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83b405cb90 con 0x7f83c415dbd0 2024-03-06T07:26:17.252 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:26:17.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a806b4d0 msgr2=0x7f83a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a806b4d0 0x7f83a806d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f83c0036000 tx=0x7f83c0008030 comp rx=0 tx=0).stop 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 msgr2=0x7f83c40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 0x7f83c40cf0d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f83b400c7f0 tx=0x7f83b400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 shutdown_connections 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a806b4d0 0x7f83a806d990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83c415dfb0 0x7f83c406e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c415dbd0 0x7f83c40cf0d0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 --2- 172.21.15.27:0/3662792857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83c40b5840 0x7f83c40af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 >> 172.21.15.27:0/3662792857 conn(0x7f83c40b26b0 msgr2=0x7f83c406da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 shutdown_connections 2024-03-06T07:26:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.252+0000 7f83cbe81640 1 -- 172.21.15.27:0/3662792857 wait complete. 2024-03-06T07:26:17.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:26:17.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.369+0000 7f76bb347640 1 Processor -- start 2024-03-06T07:26:17.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.370+0000 7f76bb347640 1 -- start start 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.370+0000 7f76bb347640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76b40b1020 0x7f76b406af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.370+0000 7f76bb347640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b40cd890 0x7f76b415d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.370+0000 7f76bb347640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40cc0e0 con 0x7f76b406ed10 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76bb347640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b40d28b0 con 0x7f76b40b1020 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76bb347640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40cf320 con 0x7f76b40cd890 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76b90e3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76b40b1020 0x7f76b406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76b88e2640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f76b406ed10 0x7f76b40cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38824/0 (socket says 172.21.15.27:38824) 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76b90e3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76b40b1020 0x7f76b406af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34682/0 (socket says 172.21.15.27:34682) 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76b88e2640 1 -- 172.21.15.27:0/1326068028 learned_addr learned my addr 172.21.15.27:0/1326068028 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76b90e3640 1 -- 172.21.15.27:0/1326068028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b40cd890 msgr2=0x7f76b415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76b98e4640 1 --2- 172.21.15.27:0/1326068028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b40cd890 0x7f76b415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76b90e3640 1 --2- 172.21.15.27:0/1326068028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b40cd890 0x7f76b415d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76a7fff640 1 -- 172.21.15.27:0/1326068028 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044104854 0 0) 0x7f76b40cc0e0 con 0x7f76b406ed10 2024-03-06T07:26:17.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76b90e3640 1 -- 172.21.15.27:0/1326068028 >> v1:172.21.15.27:6789/0 conn(0x7f76b406ed10 legacy=0x7f76b40cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.371+0000 7f76b90e3640 1 -- 172.21.15.27:0/1326068028 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b40cc3c0 con 0x7f76b40b1020 2024-03-06T07:26:17.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.372+0000 7f76b90e3640 1 --2- 172.21.15.27:0/1326068028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 0x7f76b406af20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f76a80097a0 tx=0x7f76a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adeb65a43b110cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.372+0000 7f76a7fff640 1 -- 172.21.15.27:0/1326068028 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a8035020 con 0x7f76b40b1020 2024-03-06T07:26:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.372+0000 7f76a7fff640 1 -- 172.21.15.27:0/1326068028 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a8031830 con 0x7f76b40b1020 2024-03-06T07:26:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.372+0000 7f76a7fff640 1 -- 172.21.15.27:0/1326068028 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a8031c00 con 0x7f76b40b1020 2024-03-06T07:26:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.372+0000 7f76bb347640 1 -- 172.21.15.27:0/1326068028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 msgr2=0x7f76b406af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.372+0000 7f76bb347640 1 --2- 172.21.15.27:0/1326068028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 0x7f76b406af20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f76a80097a0 tx=0x7f76a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.372+0000 7f76bb347640 1 -- 172.21.15.27:0/1326068028 shutdown_connections 2024-03-06T07:26:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.372+0000 7f76bb347640 1 --2- 172.21.15.27:0/1326068028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b40cd890 0x7f76b415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.373+0000 7f76bb347640 1 --2- 172.21.15.27:0/1326068028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 0x7f76b406af20 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.373+0000 7f76bb347640 1 -- 172.21.15.27:0/1326068028 >> 172.21.15.27:0/1326068028 conn(0x7f76b40d1a90 msgr2=0x7f76b40d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.373+0000 7f76bb347640 1 -- 172.21.15.27:0/1326068028 shutdown_connections 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.373+0000 7f76bb347640 1 -- 172.21.15.27:0/1326068028 wait complete. 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.374+0000 7f76bb347640 1 Processor -- start 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.374+0000 7f76bb347640 1 -- start start 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.374+0000 7f76bb347640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b406ed10 0x7f76b40b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.374+0000 7f76bb347640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 0x7f76b40b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.374+0000 7f76bb347640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76b40cd890 0x7f76b40c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.374+0000 7f76bb347640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40ccc60 con 0x7f76b40b1020 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.374+0000 7f76bb347640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40cc980 con 0x7f76b40cd890 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.374+0000 7f76bb347640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40cdca0 con 0x7f76b406ed10 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b88e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 0x7f76b40b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b88e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 0x7f76b40b2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34690/0 (socket says 172.21.15.27:34690) 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b88e2640 1 -- 172.21.15.27:0/558850605 learned_addr learned my addr 172.21.15.27:0/558850605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b98e4640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76b40cd890 0x7f76b40c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b90e3640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b406ed10 0x7f76b40b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b88e2640 1 -- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b406ed10 msgr2=0x7f76b40b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b88e2640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b406ed10 0x7f76b40b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b88e2640 1 -- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76b40cd890 msgr2=0x7f76b40c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b88e2640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76b40cd890 0x7f76b40c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b88e2640 1 -- 172.21.15.27:0/558850605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b40c9a20 con 0x7f76b40b1020 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b98e4640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76b40cd890 0x7f76b40c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b90e3640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b406ed10 0x7f76b40b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76b88e2640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 0x7f76b40b2e10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f76b000c4d0 tx=0x7f76b000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76a5ffb640 1 -- 172.21.15.27:0/558850605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b0018070 con 0x7f76b40b1020 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76a5ffb640 1 -- 172.21.15.27:0/558850605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b0007500 con 0x7f76b40b1020 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76a5ffb640 1 -- 172.21.15.27:0/558850605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b0004a20 con 0x7f76b40b1020 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b40b48e0 con 0x7f76b40b1020 2024-03-06T07:26:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.375+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b40b4ea0 con 0x7f76b40b1020 2024-03-06T07:26:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.377+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b4067ff0 con 0x7f76b40b1020 2024-03-06T07:26:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.377+0000 7f76a5ffb640 1 -- 172.21.15.27:0/558850605 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76b001b070 con 0x7f76b40b1020 2024-03-06T07:26:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.377+0000 7f76a5ffb640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f769806b4d0 0x7f769806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.377+0000 7f76a5ffb640 1 -- 172.21.15.27:0/558850605 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f76b0089210 con 0x7f76b40b1020 2024-03-06T07:26:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.378+0000 7f76b90e3640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f769806b4d0 0x7f769806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.378+0000 7f76b90e3640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f769806b4d0 0x7f769806d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f76a8036000 tx=0x7f76a8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.382+0000 7f76a5ffb640 1 -- 172.21.15.27:0/558850605 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76b00583e0 con 0x7f76b40b1020 2024-03-06T07:26:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.566+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f76b406f2f0 con 0x7f76b40b1020 2024-03-06T07:26:17.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.567+0000 7f76a5ffb640 1 -- 172.21.15.27:0/558850605 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) v1 ==== 84+0+65 (secure 0 0 0) 0x7f76b0057ce0 con 0x7f76b40b1020 2024-03-06T07:26:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:17.570 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:26:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:26:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:26:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:26:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:17.570 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:26:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f769806b4d0 msgr2=0x7f769806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f769806b4d0 0x7f769806d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f76a8036000 tx=0x7f76a8008030 comp rx=0 tx=0).stop 2024-03-06T07:26:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 msgr2=0x7f76b40b2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 0x7f76b40b2e10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f76b000c4d0 tx=0x7f76b000c990 comp rx=0 tx=0).stop 2024-03-06T07:26:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 shutdown_connections 2024-03-06T07:26:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f769806b4d0 0x7f769806d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76b40cd890 0x7f76b40c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76b40b1020 0x7f76b40b2e10 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 --2- 172.21.15.27:0/558850605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76b406ed10 0x7f76b40b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 >> 172.21.15.27:0/558850605 conn(0x7f76b40d1a90 msgr2=0x7f76b40ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 shutdown_connections 2024-03-06T07:26:17.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.569+0000 7f76bb347640 1 -- 172.21.15.27:0/558850605 wait complete. 2024-03-06T07:26:17.582 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:26:17.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:26:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.690+0000 7f3370071640 1 Processor -- start 2024-03-06T07:26:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.691+0000 7f3370071640 1 -- start start 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.691+0000 7f3370071640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f336815dd90 0x7f33680ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.691+0000 7f3370071640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33680b6d10 0x7f33680b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.691+0000 7f3370071640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33680d3740 con 0x7f33680d29d0 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.691+0000 7f3370071640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33680d3400 con 0x7f336815dd90 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.691+0000 7f3370071640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33680d3120 con 0x7f33680b6d10 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.692+0000 7f336de0d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f336815dd90 0x7f33680ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.692+0000 7f336de0d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f336815dd90 0x7f33680ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34696/0 (socket says 172.21.15.27:34696) 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.692+0000 7f336de0d640 1 -- 172.21.15.27:0/3933876298 learned_addr learned my addr 172.21.15.27:0/3933876298 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.692+0000 7f336e60e640 1 --2- 172.21.15.27:0/3933876298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33680b6d10 0x7f33680b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.692+0000 7f336de0d640 1 -- 172.21.15.27:0/3933876298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33680b6d10 msgr2=0x7f33680b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.692+0000 7f336de0d640 1 --2- 172.21.15.27:0/3933876298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33680b6d10 0x7f33680b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.692+0000 7f336de0d640 1 -- 172.21.15.27:0/3933876298 >> v1:172.21.15.69:6789/0 conn(0x7f33680d29d0 legacy=0x7f33680d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.692+0000 7f336de0d640 1 -- 172.21.15.27:0/3933876298 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33680d39e0 con 0x7f336815dd90 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.692+0000 7f336e60e640 1 --2- 172.21.15.27:0/3933876298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33680b6d10 0x7f33680b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.693+0000 7f336de0d640 1 --2- 172.21.15.27:0/3933876298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336815dd90 0x7f33680ceea0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3358009e50 tx=0x7f335802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65130ad1f60c46a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.693+0000 7f336ce0b640 1 -- 172.21.15.27:0/3933876298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335802cb90 con 0x7f336815dd90 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.693+0000 7f336ce0b640 1 -- 172.21.15.27:0/3933876298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335802cd30 con 0x7f336815dd90 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.693+0000 7f336ce0b640 1 -- 172.21.15.27:0/3933876298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358034610 con 0x7f336815dd90 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.693+0000 7f3370071640 1 -- 172.21.15.27:0/3933876298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336815dd90 msgr2=0x7f33680ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.693+0000 7f3370071640 1 --2- 172.21.15.27:0/3933876298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336815dd90 0x7f33680ceea0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3358009e50 tx=0x7f335802bff0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.694+0000 7f3370071640 1 -- 172.21.15.27:0/3933876298 shutdown_connections 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.694+0000 7f3370071640 1 --2- 172.21.15.27:0/3933876298 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33680b6d10 0x7f33680b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.694+0000 7f3370071640 1 --2- 172.21.15.27:0/3933876298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336815dd90 0x7f33680ceea0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.694+0000 7f3370071640 1 -- 172.21.15.27:0/3933876298 >> 172.21.15.27:0/3933876298 conn(0x7f3368071960 msgr2=0x7f3368071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.694+0000 7f3370071640 1 -- 172.21.15.27:0/3933876298 shutdown_connections 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.694+0000 7f3370071640 1 -- 172.21.15.27:0/3933876298 wait complete. 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.695+0000 7f3370071640 1 Processor -- start 2024-03-06T07:26:17.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.695+0000 7f3370071640 1 -- start start 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f3370071640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33680b6d10 0x7f33681480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f3370071640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33680d29d0 0x7f33680c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f3370071640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f336815dd90 0x7f33681603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f3370071640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33680d2df0 con 0x7f33680b6d10 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f3370071640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336815e170 con 0x7f33680d29d0 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f3370071640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33680d48f0 con 0x7f336815dd90 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336de0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33680b6d10 0x7f33681480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336de0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33680b6d10 0x7f33681480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34700/0 (socket says 172.21.15.27:34700) 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336de0d640 1 -- 172.21.15.27:0/3443003112 learned_addr learned my addr 172.21.15.27:0/3443003112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336d60c640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33680d29d0 0x7f33680c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336e60e640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f336815dd90 0x7f33681603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336de0d640 1 -- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f336815dd90 msgr2=0x7f33681603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336de0d640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f336815dd90 0x7f33681603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336de0d640 1 -- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33680d29d0 msgr2=0x7f33680c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336de0d640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33680d29d0 0x7f33680c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336de0d640 1 -- 172.21.15.27:0/3443003112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33680c9360 con 0x7f33680b6d10 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336e60e640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f336815dd90 0x7f33681603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f336de0d640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33680b6d10 0x7f33681480f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3358036040 tx=0x7f3358004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f33667fc640 1 -- 172.21.15.27:0/3443003112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358043020 con 0x7f33680b6d10 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f33667fc640 1 -- 172.21.15.27:0/3443003112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335803e450 con 0x7f33680b6d10 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33680c8570 con 0x7f33680b6d10 2024-03-06T07:26:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f33667fc640 1 -- 172.21.15.27:0/3443003112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358009510 con 0x7f33680b6d10 2024-03-06T07:26:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.696+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33680c8b30 con 0x7f33680b6d10 2024-03-06T07:26:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.697+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3368072a30 con 0x7f33680b6d10 2024-03-06T07:26:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.698+0000 7f33667fc640 1 -- 172.21.15.27:0/3443003112 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3358046020 con 0x7f33680b6d10 2024-03-06T07:26:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.698+0000 7f33667fc640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f335406b4d0 0x7f335406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.698+0000 7f33667fc640 1 -- 172.21.15.27:0/3443003112 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f33580ac580 con 0x7f33680b6d10 2024-03-06T07:26:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.698+0000 7f336d60c640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f335406b4d0 0x7f335406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.699+0000 7f336d60c640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f335406b4d0 0x7f335406d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f33681565e0 tx=0x7f335c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.703+0000 7f33667fc640 1 -- 172.21.15.27:0/3443003112 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f335804d030 con 0x7f33680b6d10 2024-03-06T07:26:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:17.882+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f336804f410 con 0x7f33680b6d10 2024-03-06T07:26:18.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:18.143 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.210+0000 7f33667fc640 1 -- 172.21.15.27:0/3443003112 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) v1 ==== 138+0+0 (secure 0 0 0) 0x7f335807b7e0 con 0x7f33680b6d10 2024-03-06T07:26:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:26:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f335406b4d0 msgr2=0x7f335406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f335406b4d0 0x7f335406d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f33681565e0 tx=0x7f335c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33680b6d10 msgr2=0x7f33681480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33680b6d10 0x7f33681480f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3358036040 tx=0x7f3358004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 shutdown_connections 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f335406b4d0 0x7f335406d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f336815dd90 0x7f33681603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33680d29d0 0x7f33680c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 --2- 172.21.15.27:0/3443003112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33680b6d10 0x7f33681480f0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 >> 172.21.15.27:0/3443003112 conn(0x7f3368071960 msgr2=0x7f33680b7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 shutdown_connections 2024-03-06T07:26:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.212+0000 7f3370071640 1 -- 172.21.15.27:0/3443003112 wait complete. 2024-03-06T07:26:18.228 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:26:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.380+0000 7f8ff3464640 1 Processor -- start 2024-03-06T07:26:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.380+0000 7f8ff3464640 1 -- start start 2024-03-06T07:26:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff3464640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 0x7f8fec0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff3464640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fec0b6d10 0x7f8fec0b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff3464640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fec0d3740 con 0x7f8fec0d29d0 2024-03-06T07:26:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff3464640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec0d3400 con 0x7f8fec0b6d10 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff3464640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0d3120 con 0x7f8fec15dd90 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff1a01640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fec0b6d10 0x7f8fec0b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff09ff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8fec0d29d0 0x7f8fec0d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:46630/0 (socket says 172.21.15.27:46630) 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff1200640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 0x7f8fec0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff09ff640 1 -- 172.21.15.27:0/1831068112 learned_addr learned my addr 172.21.15.27:0/1831068112 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff1a01640 1 -- 172.21.15.27:0/1831068112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 msgr2=0x7f8fec0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff1a01640 1 --2- 172.21.15.27:0/1831068112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 0x7f8fec0ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff1a01640 1 -- 172.21.15.27:0/1831068112 >> v1:172.21.15.69:6789/0 conn(0x7f8fec0d29d0 legacy=0x7f8fec0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8febfff640 1 -- 172.21.15.27:0/1831068112 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169444457 0 0) 0x7f8fec0d3740 con 0x7f8fec0d29d0 2024-03-06T07:26:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.381+0000 7f8ff1a01640 1 -- 172.21.15.27:0/1831068112 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec0d39e0 con 0x7f8fec0b6d10 2024-03-06T07:26:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.382+0000 7f8ff1a01640 1 --2- 172.21.15.27:0/1831068112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0b6d10 0x7f8fec0b71a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009d90 tx=0x7f8fe402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa83a5cac7e8a0c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.382+0000 7f8febfff640 1 -- 172.21.15.27:0/1831068112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4039070 con 0x7f8fec0b6d10 2024-03-06T07:26:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.382+0000 7f8febfff640 1 -- 172.21.15.27:0/1831068112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe402cce0 con 0x7f8fec0b6d10 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.382+0000 7f8febfff640 1 -- 172.21.15.27:0/1831068112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4034660 con 0x7f8fec0b6d10 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.383+0000 7f8ff3464640 1 -- 172.21.15.27:0/1831068112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0b6d10 msgr2=0x7f8fec0b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.383+0000 7f8ff3464640 1 --2- 172.21.15.27:0/1831068112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0b6d10 0x7f8fec0b71a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009d90 tx=0x7f8fe402bf30 comp rx=0 tx=0).stop 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.383+0000 7f8ff3464640 1 -- 172.21.15.27:0/1831068112 shutdown_connections 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.383+0000 7f8ff3464640 1 --2- 172.21.15.27:0/1831068112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0b6d10 0x7f8fec0b71a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.383+0000 7f8ff3464640 1 --2- 172.21.15.27:0/1831068112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 0x7f8fec0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.383+0000 7f8ff3464640 1 -- 172.21.15.27:0/1831068112 >> 172.21.15.27:0/1831068112 conn(0x7f8fec071960 msgr2=0x7f8fec071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.383+0000 7f8ff3464640 1 -- 172.21.15.27:0/1831068112 shutdown_connections 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.383+0000 7f8ff3464640 1 -- 172.21.15.27:0/1831068112 wait complete. 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.384+0000 7f8ff3464640 1 Processor -- start 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.384+0000 7f8ff3464640 1 -- start start 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff3464640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0b6d10 0x7f8fec15a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff3464640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0d29d0 0x7f8fec157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff3464640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 0x7f8fec156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff3464640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0d2df0 con 0x7f8fec0d29d0 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff3464640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec15e170 con 0x7f8fec0b6d10 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff3464640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0d48f0 con 0x7f8fec15dd90 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff09ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0d29d0 0x7f8fec157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff09ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0d29d0 0x7f8fec157cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34724/0 (socket says 172.21.15.27:34724) 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff09ff640 1 -- 172.21.15.27:0/2470329217 learned_addr learned my addr 172.21.15.27:0/2470329217 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff1200640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0b6d10 0x7f8fec15a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff1a01640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 0x7f8fec156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff09ff640 1 -- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 msgr2=0x7f8fec156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff09ff640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 0x7f8fec156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff09ff640 1 -- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0b6d10 msgr2=0x7f8fec15a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff09ff640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0b6d10 0x7f8fec15a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff09ff640 1 -- 172.21.15.27:0/2470329217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec1586c0 con 0x7f8fec0d29d0 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.385+0000 7f8ff09ff640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0d29d0 0x7f8fec157cc0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c4d0 tx=0x7f8fe000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.386+0000 7f8fe9ffb640 1 -- 172.21.15.27:0/2470329217 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0018070 con 0x7f8fec0d29d0 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.386+0000 7f8fe9ffb640 1 -- 172.21.15.27:0/2470329217 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe0007500 con 0x7f8fec0d29d0 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.386+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fec1589a0 con 0x7f8fec0d29d0 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.386+0000 7f8fe9ffb640 1 -- 172.21.15.27:0/2470329217 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0004a20 con 0x7f8fec0d29d0 2024-03-06T07:26:18.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.386+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fec158f60 con 0x7f8fec0d29d0 2024-03-06T07:26:18.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.387+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fec072a30 con 0x7f8fec0d29d0 2024-03-06T07:26:18.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.387+0000 7f8fe9ffb640 1 -- 172.21.15.27:0/2470329217 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fe001b070 con 0x7f8fec0d29d0 2024-03-06T07:26:18.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.388+0000 7f8fe9ffb640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fd806b4d0 0x7f8fd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.388+0000 7f8fe9ffb640 1 -- 172.21.15.27:0/2470329217 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8fe0089280 con 0x7f8fec0d29d0 2024-03-06T07:26:18.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.388+0000 7f8ff1200640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fd806b4d0 0x7f8fd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.389+0000 7f8ff1200640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fd806b4d0 0x7f8fd806d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8fec1f48b0 tx=0x7f8fdc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.392+0000 7f8fe9ffb640 1 -- 172.21.15.27:0/2470329217 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fe0058370 con 0x7f8fec0d29d0 2024-03-06T07:26:18.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:18.576+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8fec04f410 con 0x7f8fec0d29d0 2024-03-06T07:26:19.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.215+0000 7f8fe9ffb640 1 -- 172.21.15.27:0/2470329217 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v68) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8fe0057c70 con 0x7f8fec0d29d0 2024-03-06T07:26:19.218 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:26:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.218+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fd806b4d0 msgr2=0x7f8fd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.218+0000 7f8ff3464640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fd806b4d0 0x7f8fd806d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8fec1f48b0 tx=0x7f8fdc008040 comp rx=0 tx=0).stop 2024-03-06T07:26:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.218+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0d29d0 msgr2=0x7f8fec157cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.218+0000 7f8ff3464640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0d29d0 0x7f8fec157cc0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c4d0 tx=0x7f8fe000c990 comp rx=0 tx=0).stop 2024-03-06T07:26:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.218+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 shutdown_connections 2024-03-06T07:26:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.218+0000 7f8ff3464640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fd806b4d0 0x7f8fd806d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.218+0000 7f8ff3464640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fec15dd90 0x7f8fec156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.218+0000 7f8ff3464640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fec0d29d0 0x7f8fec157cc0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.218+0000 7f8ff3464640 1 --2- 172.21.15.27:0/2470329217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fec0b6d10 0x7f8fec15a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.219+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 >> 172.21.15.27:0/2470329217 conn(0x7f8fec071960 msgr2=0x7f8fec0c7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.219+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 shutdown_connections 2024-03-06T07:26:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.219+0000 7f8ff3464640 1 -- 172.21.15.27:0/2470329217 wait complete. 2024-03-06T07:26:19.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:26:19.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.355+0000 7fb18d3a2640 1 Processor -- start 2024-03-06T07:26:19.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.355+0000 7fb18d3a2640 1 -- start start 2024-03-06T07:26:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.356+0000 7fb18d3a2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1880d0a80 0x7fb1881623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.356+0000 7fb18d3a2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1880cfd00 con 0x7fb18806c020 2024-03-06T07:26:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.356+0000 7fb18d3a2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1880cffa0 con 0x7fb1880cea10 2024-03-06T07:26:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.356+0000 7fb18d3a2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1880d0f10 con 0x7fb1880d0a80 2024-03-06T07:26:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.356+0000 7fb18cba1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1880d0a80 0x7fb1881623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.356+0000 7fb1877fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb18806c020 0x7fb18815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38832/0 (socket says 172.21.15.27:38832) 2024-03-06T07:26:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.356+0000 7fb18cba1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1880d0a80 0x7fb1881623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34732/0 (socket says 172.21.15.27:34732) 2024-03-06T07:26:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.356+0000 7fb1877fe640 1 -- 172.21.15.27:0/3904234410 learned_addr learned my addr 172.21.15.27:0/3904234410 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.357+0000 7fb18cba1640 1 -- 172.21.15.27:0/3904234410 >> v1:172.21.15.69:6789/0 conn(0x7fb1880cea10 legacy=0x7fb18806b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.357+0000 7fb18cba1640 1 -- 172.21.15.27:0/3904234410 >> v1:172.21.15.27:6789/0 conn(0x7fb18806c020 legacy=0x7fb18815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.357+0000 7fb18cba1640 1 -- 172.21.15.27:0/3904234410 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1880d0280 con 0x7fb1880d0a80 2024-03-06T07:26:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.357+0000 7fb186ffd640 1 -- 172.21.15.27:0/3904234410 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773203469 0 0) 0x7fb1880cfd00 con 0x7fb18806c020 2024-03-06T07:26:19.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.357+0000 7fb18cba1640 1 --2- 172.21.15.27:0/3904234410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880d0a80 0x7fb1881623a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb180009d90 tx=0x7fb18002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae8460fc2ffe4e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb186ffd640 1 -- 172.21.15.27:0/3904234410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180039070 con 0x7fb1880d0a80 2024-03-06T07:26:19.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb186ffd640 1 -- 172.21.15.27:0/3904234410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18002cd00 con 0x7fb1880d0a80 2024-03-06T07:26:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb186ffd640 1 -- 172.21.15.27:0/3904234410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180034660 con 0x7fb1880d0a80 2024-03-06T07:26:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb18d3a2640 1 -- 172.21.15.27:0/3904234410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880d0a80 msgr2=0x7fb1881623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb18d3a2640 1 --2- 172.21.15.27:0/3904234410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880d0a80 0x7fb1881623a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb180009d90 tx=0x7fb18002bf50 comp rx=0 tx=0).stop 2024-03-06T07:26:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb18d3a2640 1 -- 172.21.15.27:0/3904234410 shutdown_connections 2024-03-06T07:26:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb18d3a2640 1 --2- 172.21.15.27:0/3904234410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880d0a80 0x7fb1881623a0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb18d3a2640 1 -- 172.21.15.27:0/3904234410 >> 172.21.15.27:0/3904234410 conn(0x7fb1880d1920 msgr2=0x7fb1880d1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb18d3a2640 1 -- 172.21.15.27:0/3904234410 shutdown_connections 2024-03-06T07:26:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.358+0000 7fb18d3a2640 1 -- 172.21.15.27:0/3904234410 wait complete. 2024-03-06T07:26:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.359+0000 7fb18d3a2640 1 Processor -- start 2024-03-06T07:26:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb18d3a2640 1 -- start start 2024-03-06T07:26:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb18d3a2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18806c020 0x7fb1881578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb18d3a2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1880cea10 0x7fb188157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb18d3a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1880d0a80 0x7fb1881f7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb18d3a2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18806af20 con 0x7fb18806c020 2024-03-06T07:26:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb18d3a2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880d05f0 con 0x7fb1880cea10 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb18d3a2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18806b0a0 con 0x7fb1880d0a80 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb187fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18806c020 0x7fb1881578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb187fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18806c020 0x7fb1881578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34748/0 (socket says 172.21.15.27:34748) 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.360+0000 7fb187fff640 1 -- 172.21.15.27:0/2203682209 learned_addr learned my addr 172.21.15.27:0/2203682209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb18cba1640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1880d0a80 0x7fb1881f7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb1877fe640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1880cea10 0x7fb188157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb187fff640 1 -- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1880d0a80 msgr2=0x7fb1881f7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb187fff640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1880d0a80 0x7fb1881f7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb187fff640 1 -- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1880cea10 msgr2=0x7fb188157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb187fff640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1880cea10 0x7fb188157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb187fff640 1 -- 172.21.15.27:0/2203682209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18815a600 con 0x7fb18806c020 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb187fff640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18806c020 0x7fb1881578f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c450 tx=0x7fb17800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb184ff9640 1 -- 172.21.15.27:0/2203682209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb178017070 con 0x7fb18806c020 2024-03-06T07:26:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb184ff9640 1 -- 172.21.15.27:0/2203682209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1780052d0 con 0x7fb18806c020 2024-03-06T07:26:19.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb184ff9640 1 -- 172.21.15.27:0/2203682209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb178003830 con 0x7fb18806c020 2024-03-06T07:26:19.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18815a8e0 con 0x7fb18806c020 2024-03-06T07:26:19.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.361+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18815aea0 con 0x7fb18806c020 2024-03-06T07:26:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.362+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18804f330 con 0x7fb18806c020 2024-03-06T07:26:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.363+0000 7fb184ff9640 1 -- 172.21.15.27:0/2203682209 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb17801a070 con 0x7fb18806c020 2024-03-06T07:26:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.363+0000 7fb184ff9640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb17406b500 0x7fb17406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.363+0000 7fb184ff9640 1 -- 172.21.15.27:0/2203682209 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fb178089400 con 0x7fb18806c020 2024-03-06T07:26:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.363+0000 7fb1877fe640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb17406b500 0x7fb17406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.364+0000 7fb1877fe640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb17406b500 0x7fb17406d9c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb17c007f70 tx=0x7fb17c007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.368+0000 7fb184ff9640 1 -- 172.21.15.27:0/2203682209 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb178057d20 con 0x7fb18806c020 2024-03-06T07:26:19.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:19.589+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb18806e610 con 0x7fb18806c020 2024-03-06T07:26:20.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.230+0000 7fb184ff9640 1 -- 172.21.15.27:0/2203682209 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v40) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb17805bdc0 con 0x7fb18806c020 2024-03-06T07:26:20.232 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:26:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.235+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb17406b500 msgr2=0x7fb17406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.235+0000 7fb18d3a2640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb17406b500 0x7fb17406d9c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb17c007f70 tx=0x7fb17c007f00 comp rx=0 tx=0).stop 2024-03-06T07:26:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.235+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18806c020 msgr2=0x7fb1881578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.235+0000 7fb18d3a2640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18806c020 0x7fb1881578f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c450 tx=0x7fb17800c910 comp rx=0 tx=0).stop 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.237+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 shutdown_connections 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.237+0000 7fb18d3a2640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb17406b500 0x7fb17406d9c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.237+0000 7fb18d3a2640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1880d0a80 0x7fb1881f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.237+0000 7fb18d3a2640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1880cea10 0x7fb188157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.237+0000 7fb18d3a2640 1 --2- 172.21.15.27:0/2203682209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18806c020 0x7fb1881578f0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.237+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 >> 172.21.15.27:0/2203682209 conn(0x7fb1880d1920 msgr2=0x7fb18815b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.240+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 shutdown_connections 2024-03-06T07:26:20.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.240+0000 7fb18d3a2640 1 -- 172.21.15.27:0/2203682209 wait complete. 2024-03-06T07:26:20.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.370+0000 7f794b037640 1 Processor -- start 2024-03-06T07:26:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.370+0000 7f794b037640 1 -- start start 2024-03-06T07:26:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.370+0000 7f794b037640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f794415e060 0x7f79440ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.370+0000 7f794b037640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79440d0dd0 con 0x7f79440d0880 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.370+0000 7f794b037640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79440d10b0 con 0x7f794406fe20 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.370+0000 7f794b037640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79440cc0e0 con 0x7f794415e060 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.371+0000 7f7948dd3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f794415e060 0x7f79440ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.371+0000 7f79495d4640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f79440d0880 0x7f79440ccbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38846/0 (socket says 172.21.15.27:38846) 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.371+0000 7f79495d4640 1 -- 172.21.15.27:0/910445967 learned_addr learned my addr 172.21.15.27:0/910445967 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.371+0000 7f7948dd3640 1 --2- 172.21.15.27:0/910445967 >> v2:172.21.15.27:3300/0 conn(0x7f794415e060 0x7f79440ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34764/0 (socket says 172.21.15.27:34764) 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.371+0000 7f7948dd3640 1 -- 172.21.15.27:0/910445967 >> v1:172.21.15.69:6789/0 conn(0x7f794406fe20 legacy=0x7f79440ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.371+0000 7f7948dd3640 1 -- 172.21.15.27:0/910445967 >> v1:172.21.15.27:6789/0 conn(0x7f79440d0880 legacy=0x7f79440ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.371+0000 7f7948dd3640 1 -- 172.21.15.27:0/910445967 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79440cc3c0 con 0x7f794415e060 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.371+0000 7f79437fe640 1 -- 172.21.15.27:0/910445967 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397503082 0 0) 0x7f79440d0dd0 con 0x7f79440d0880 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f7948dd3640 1 --2- 172.21.15.27:0/910445967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 0x7f79440ce310 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f793c009d90 tx=0x7f793c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7570041a44af465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f79437fe640 1 -- 172.21.15.27:0/910445967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c039070 con 0x7f794415e060 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f79437fe640 1 -- 172.21.15.27:0/910445967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c02cd00 con 0x7f794415e060 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f79437fe640 1 -- 172.21.15.27:0/910445967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c034660 con 0x7f794415e060 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f794b037640 1 -- 172.21.15.27:0/910445967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 msgr2=0x7f79440ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f794b037640 1 --2- 172.21.15.27:0/910445967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 0x7f79440ce310 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f793c009d90 tx=0x7f793c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f794b037640 1 -- 172.21.15.27:0/910445967 shutdown_connections 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f794b037640 1 --2- 172.21.15.27:0/910445967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 0x7f79440ce310 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f794b037640 1 -- 172.21.15.27:0/910445967 >> 172.21.15.27:0/910445967 conn(0x7f79440d1a90 msgr2=0x7f79440d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f794b037640 1 -- 172.21.15.27:0/910445967 shutdown_connections 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.372+0000 7f794b037640 1 -- 172.21.15.27:0/910445967 wait complete. 2024-03-06T07:26:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.374+0000 7f794b037640 1 Processor -- start 2024-03-06T07:26:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.374+0000 7f794b037640 1 -- start start 2024-03-06T07:26:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.374+0000 7f794b037640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f794406fe20 0x7f79441f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.374+0000 7f794b037640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79440d0880 0x7f79440b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.374+0000 7f794b037640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 0x7f79440cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f794b037640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79440cef60 con 0x7f794415e060 2024-03-06T07:26:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f794b037640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794415e440 con 0x7f794406fe20 2024-03-06T07:26:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f794b037640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79440cf0e0 con 0x7f79440d0880 2024-03-06T07:26:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79495d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 0x7f79440cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79495d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 0x7f79440cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34772/0 (socket says 172.21.15.27:34772) 2024-03-06T07:26:20.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79495d4640 1 -- 172.21.15.27:0/4084779665 learned_addr learned my addr 172.21.15.27:0/4084779665 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f7948dd3640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f794406fe20 0x7f79441f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f7943fff640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79440d0880 0x7f79440b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79495d4640 1 -- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79440d0880 msgr2=0x7f79440b73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79495d4640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79440d0880 0x7f79440b73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79495d4640 1 -- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f794406fe20 msgr2=0x7f79441f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79495d4640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f794406fe20 0x7f79441f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79495d4640 1 -- 172.21.15.27:0/4084779665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79440cb750 con 0x7f794415e060 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f7948dd3640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f794406fe20 0x7f79441f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79495d4640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 0x7f79440cb000 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f793400c450 tx=0x7f793400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79417fa640 1 -- 172.21.15.27:0/4084779665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934018070 con 0x7f794415e060 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79417fa640 1 -- 172.21.15.27:0/4084779665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7934007500 con 0x7f794415e060 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794415ea00 con 0x7f794415e060 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f79417fa640 1 -- 172.21.15.27:0/4084779665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934004a20 con 0x7f794415e060 2024-03-06T07:26:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.375+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794415efc0 con 0x7f794415e060 2024-03-06T07:26:20.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.376+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7944067ff0 con 0x7f794415e060 2024-03-06T07:26:20.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.377+0000 7f79417fa640 1 -- 172.21.15.27:0/4084779665 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f793401b070 con 0x7f794415e060 2024-03-06T07:26:20.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.377+0000 7f79417fa640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f792406b4d0 0x7f792406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.377+0000 7f79417fa640 1 -- 172.21.15.27:0/4084779665 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7934089410 con 0x7f794415e060 2024-03-06T07:26:20.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.377+0000 7f7948dd3640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f792406b4d0 0x7f792406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.378+0000 7f7948dd3640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f792406b4d0 0x7f792406d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f793c036000 tx=0x7f793c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.382+0000 7f79417fa640 1 -- 172.21.15.27:0/4084779665 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7934058290 con 0x7f794415e060 2024-03-06T07:26:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.565+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79440ce360 con 0x7f794415e060 2024-03-06T07:26:20.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.566+0000 7f79417fa640 1 -- 172.21.15.27:0/4084779665 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f7934057b90 con 0x7f794415e060 2024-03-06T07:26:20.567 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:20.568 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:19.591108+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.914214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.608105+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f792406b4d0 msgr2=0x7f792406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f792406b4d0 0x7f792406d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f793c036000 tx=0x7f793c030000 comp rx=0 tx=0).stop 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 msgr2=0x7f79440cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 0x7f79440cb000 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f793400c450 tx=0x7f793400c910 comp rx=0 tx=0).stop 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 shutdown_connections 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f792406b4d0 0x7f792406d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f794415e060 0x7f79440cb000 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79440d0880 0x7f79440b73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 --2- 172.21.15.27:0/4084779665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f794406fe20 0x7f79441f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 >> 172.21.15.27:0/4084779665 conn(0x7f79440d1a90 msgr2=0x7f79440d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 shutdown_connections 2024-03-06T07:26:20.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.568+0000 7f794b037640 1 -- 172.21.15.27:0/4084779665 wait complete. 2024-03-06T07:26:20.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.687+0000 7f485175e640 1 Processor -- start 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.687+0000 7f485175e640 1 -- start start 2024-03-06T07:26:20.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f485175e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c157b80 0x7f484c15b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f485175e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c06c7b0 con 0x7f484c0cbab0 2024-03-06T07:26:20.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f485175e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c06ca90 con 0x7f484c157f60 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f485175e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c06cd70 con 0x7f484c157b80 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f484c0cbab0 0x7f484c070b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38848/0 (socket says 172.21.15.27:38848) 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484b7fe640 1 -- 172.21.15.27:0/3943313736 learned_addr learned my addr 172.21.15.27:0/3943313736 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484affd640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c157b80 0x7f484c15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200249634 0 0) 0x7f484c06c7b0 con 0x7f484c0cbab0 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4838004630 con 0x7f484c0cbab0 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938336778 0 0) 0x7f484c06ca90 con 0x7f484c157f60 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f484c06c7b0 con 0x7f484c157f60 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467811498 0 0) 0x7f4838004630 con 0x7f484c0cbab0 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c157b80 msgr2=0x7f484c15b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c157b80 0x7f484c15b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 >> v1:172.21.15.69:6789/0 conn(0x7f484c157f60 legacy=0x7f484c15b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c06d050 con 0x7f484c0cbab0 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.688+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f48440019f0 con 0x7f484c0cbab0 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 >> v1:172.21.15.27:6789/0 conn(0x7f484c0cbab0 legacy=0x7f484c070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484a7fc640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c157b80 0x7f4838004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484a7fc640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48380051e0 0x7f4838005910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484a7fc640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4838005e50 0x7f48380062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c06cd70 con 0x7f4838005e50 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483800cbb0 con 0x7f48380051e0 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483800cdb0 con 0x7f484c157b80 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484b7fe640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4838005e50 0x7f48380062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484affd640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c157b80 0x7f4838004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f48427fc640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48380051e0 0x7f4838005910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484b7fe640 1 -- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c157b80 msgr2=0x7f4838004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484b7fe640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c157b80 0x7f4838004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484b7fe640 1 -- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48380051e0 msgr2=0x7f4838005910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484b7fe640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48380051e0 0x7f4838005910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.689+0000 7f484b7fe640 1 -- 172.21.15.27:0/3943313736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4838004630 con 0x7f4838005e50 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f484b7fe640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4838005e50 0x7f48380062d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f484400a7c0 tx=0x7f4838006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b416a86426ba9ed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484400a410 con 0x7f4838005e50 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4844034650 con 0x7f4838005e50 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f484a7fc640 1 -- 172.21.15.27:0/3943313736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844034a80 con 0x7f4838005e50 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f485175e640 1 -- 172.21.15.27:0/3943313736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4838005e50 msgr2=0x7f48380062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f485175e640 1 --2- 172.21.15.27:0/3943313736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4838005e50 0x7f48380062d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f484400a7c0 tx=0x7f4838006940 comp rx=0 tx=0).stop 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f484b7fe640 1 -- 172.21.15.27:0/3943313736 reap_dead start 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f485175e640 1 -- 172.21.15.27:0/3943313736 shutdown_connections 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f485175e640 1 -- 172.21.15.27:0/3943313736 >> 172.21.15.27:0/3943313736 conn(0x7f484c066d00 msgr2=0x7f484c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.690+0000 7f485175e640 1 -- 172.21.15.27:0/3943313736 shutdown_connections 2024-03-06T07:26:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.691+0000 7f485175e640 1 -- 172.21.15.27:0/3943313736 wait complete. 2024-03-06T07:26:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.692+0000 7f485175e640 1 Processor -- start 2024-03-06T07:26:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.692+0000 7f485175e640 1 -- start start 2024-03-06T07:26:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.692+0000 7f485175e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c155860 0x7f484c1e2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.692+0000 7f485175e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1e3500 0x7f484c155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.692+0000 7f485175e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c156180 0x7f484c156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f485175e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c071250 con 0x7f484c1e3500 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f485175e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0cc260 con 0x7f484c156180 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f485175e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0713d0 con 0x7f484c155860 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f48427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1e3500 0x7f484c155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f48427fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1e3500 0x7f484c155c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34790/0 (socket says 172.21.15.27:34790) 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f48427fc640 1 -- 172.21.15.27:0/4116071263 learned_addr learned my addr 172.21.15.27:0/4116071263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f484affd640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c155860 0x7f484c1e2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f484b7fe640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c156180 0x7f484c156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f48427fc640 1 -- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c155860 msgr2=0x7f484c1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f48427fc640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c155860 0x7f484c1e2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f48427fc640 1 -- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c156180 msgr2=0x7f484c156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f48427fc640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c156180 0x7f484c156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f48427fc640 1 -- 172.21.15.27:0/4116071263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0cd310 con 0x7f484c1e3500 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f484affd640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c155860 0x7f484c1e2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f48427fc640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1e3500 0x7f484c155c40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f4834011ac0 tx=0x7f4834011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f4843fff640 1 -- 172.21.15.27:0/4116071263 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483400fac0 con 0x7f484c1e3500 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f4843fff640 1 -- 172.21.15.27:0/4116071263 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4834016030 con 0x7f484c1e3500 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c0cd5f0 con 0x7f484c1e3500 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f4843fff640 1 -- 172.21.15.27:0/4116071263 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48340035f0 con 0x7f484c1e3500 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.693+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c0cdbb0 con 0x7f484c1e3500 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.694+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484c067ff0 con 0x7f484c1e3500 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.695+0000 7f4843fff640 1 -- 172.21.15.27:0/4116071263 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f483400c070 con 0x7f484c1e3500 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.695+0000 7f4843fff640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f482c06b500 0x7f482c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.695+0000 7f4843fff640 1 -- 172.21.15.27:0/4116071263 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f483408cae0 con 0x7f484c1e3500 2024-03-06T07:26:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.695+0000 7f484affd640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f482c06b500 0x7f482c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.696+0000 7f484affd640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f482c06b500 0x7f482c06d9c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f483c009bc0 tx=0x7f483c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.699+0000 7f4843fff640 1 -- 172.21.15.27:0/4116071263 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f483405b310 con 0x7f484c1e3500 2024-03-06T07:26:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.881+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f484c158b30 con 0x7f484c1e3500 2024-03-06T07:26:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.882+0000 7f4843fff640 1 -- 172.21.15.27:0/4116071263 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f483405f3b0 con 0x7f484c1e3500 2024-03-06T07:26:20.884 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:20.884 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:19.591108+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.914214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.608105+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f482c06b500 msgr2=0x7f482c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f482c06b500 0x7f482c06d9c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f483c009bc0 tx=0x7f483c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1e3500 msgr2=0x7f484c155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1e3500 0x7f484c155c40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f4834011ac0 tx=0x7f4834011f80 comp rx=0 tx=0).stop 2024-03-06T07:26:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 shutdown_connections 2024-03-06T07:26:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f482c06b500 0x7f482c06d9c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f484c156180 0x7f484c156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f484c1e3500 0x7f484c155c40 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 --2- 172.21.15.27:0/4116071263 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f484c155860 0x7f484c1e2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 >> 172.21.15.27:0/4116071263 conn(0x7f484c066d00 msgr2=0x7f484c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 shutdown_connections 2024-03-06T07:26:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:20.884+0000 7f485175e640 1 -- 172.21.15.27:0/4116071263 wait complete. 2024-03-06T07:26:20.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:26:21.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.006+0000 7f2aa6b81640 1 Processor -- start 2024-03-06T07:26:21.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.006+0000 7f2aa6b81640 1 -- start start 2024-03-06T07:26:21.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.006+0000 7f2aa6b81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 0x7f2aa00b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.006+0000 7f2aa6b81640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2aa00b7820 0x7f2aa00b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.006+0000 7f2aa6b81640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00d4960 con 0x7f2aa0071540 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.006+0000 7f2aa6b81640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00d3230 con 0x7f2aa00b7820 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.006+0000 7f2aa6b81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00d36c0 con 0x7f2aa00b2d30 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2aa491d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2aa0071540 0x7f2aa0071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38862/0 (socket says 172.21.15.27:38862) 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2aa491d640 1 -- 172.21.15.27:0/2087373860 learned_addr learned my addr 172.21.15.27:0/2087373860 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2aa511e640 1 --2- 172.21.15.27:0/2087373860 >> v2:172.21.15.69:3300/0 conn(0x7f2aa00b7820 0x7f2aa00b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2a9ffff640 1 --2- 172.21.15.27:0/2087373860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 0x7f2aa00b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2a9f7fe640 1 -- 172.21.15.27:0/2087373860 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106045172 0 0) 0x7f2aa00d4960 con 0x7f2aa0071540 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2a9f7fe640 1 -- 172.21.15.27:0/2087373860 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a88003520 con 0x7f2aa0071540 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2a9f7fe640 1 -- 172.21.15.27:0/2087373860 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 286579904 0 0) 0x7f2a88003520 con 0x7f2aa0071540 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2aa511e640 1 -- 172.21.15.27:0/2087373860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 msgr2=0x7f2aa00b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2aa511e640 1 --2- 172.21.15.27:0/2087373860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 0x7f2aa00b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2aa511e640 1 -- 172.21.15.27:0/2087373860 >> v1:172.21.15.27:6789/0 conn(0x7f2aa0071540 legacy=0x7f2aa0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.007+0000 7f2aa511e640 1 -- 172.21.15.27:0/2087373860 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa00d2620 con 0x7f2aa00b7820 2024-03-06T07:26:21.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.008+0000 7f2aa511e640 1 --2- 172.21.15.27:0/2087373860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa00b7820 0x7f2aa00b7ca0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2a98009d90 tx=0x7f2a9802bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c570445a85716cac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.008+0000 7f2a9f7fe640 1 -- 172.21.15.27:0/2087373860 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98039070 con 0x7f2aa00b7820 2024-03-06T07:26:21.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.008+0000 7f2a9f7fe640 1 -- 172.21.15.27:0/2087373860 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9802c8a0 con 0x7f2aa00b7820 2024-03-06T07:26:21.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.008+0000 7f2a9f7fe640 1 -- 172.21.15.27:0/2087373860 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90001670 con 0x7f2aa00b7820 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.009+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2087373860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa00b7820 msgr2=0x7f2aa00b7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.009+0000 7f2aa6b81640 1 --2- 172.21.15.27:0/2087373860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa00b7820 0x7f2aa00b7ca0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2a98009d90 tx=0x7f2a9802bd70 comp rx=0 tx=0).stop 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.009+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2087373860 shutdown_connections 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.009+0000 7f2aa6b81640 1 --2- 172.21.15.27:0/2087373860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa00b7820 0x7f2aa00b7ca0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.009+0000 7f2aa6b81640 1 --2- 172.21.15.27:0/2087373860 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 0x7f2aa00b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.009+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2087373860 >> 172.21.15.27:0/2087373860 conn(0x7f2aa00d3d60 msgr2=0x7f2aa00d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.009+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2087373860 shutdown_connections 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.009+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2087373860 wait complete. 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.010+0000 7f2aa6b81640 1 Processor -- start 2024-03-06T07:26:21.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.010+0000 7f2aa6b81640 1 -- start start 2024-03-06T07:26:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa6b81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa0071540 0x7f2aa01f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa6b81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 0x7f2aa00c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa6b81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aa00b7820 0x7f2aa014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa6b81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00d2ec0 con 0x7f2aa00b7820 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa6b81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00d2be0 con 0x7f2aa0071540 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa6b81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00d22f0 con 0x7f2aa00b2d30 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa511e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aa00b7820 0x7f2aa014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa511e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aa00b7820 0x7f2aa014d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34796/0 (socket says 172.21.15.27:34796) 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa511e640 1 -- 172.21.15.27:0/2663468927 learned_addr learned my addr 172.21.15.27:0/2663468927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa491d640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa0071540 0x7f2aa01f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2a9ffff640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 0x7f2aa00c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa511e640 1 -- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 msgr2=0x7f2aa00c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa511e640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 0x7f2aa00c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa511e640 1 -- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa0071540 msgr2=0x7f2aa01f7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa511e640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa0071540 0x7f2aa01f7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa511e640 1 -- 172.21.15.27:0/2663468927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa01f7b70 con 0x7f2aa00b7820 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.011+0000 7f2aa511e640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aa00b7820 0x7f2aa014d320 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2a98009eb0 tx=0x7f2a98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.012+0000 7f2a9d7fa640 1 -- 172.21.15.27:0/2663468927 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98039050 con 0x7f2aa00b7820 2024-03-06T07:26:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.012+0000 7f2a9d7fa640 1 -- 172.21.15.27:0/2663468927 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9803a040 con 0x7f2aa00b7820 2024-03-06T07:26:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.012+0000 7f2a9d7fa640 1 -- 172.21.15.27:0/2663468927 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98009450 con 0x7f2aa00b7820 2024-03-06T07:26:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.012+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa01f7e50 con 0x7f2aa00b7820 2024-03-06T07:26:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.012+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa01e36a0 con 0x7f2aa00b7820 2024-03-06T07:26:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.013+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa0072070 con 0x7f2aa00b7820 2024-03-06T07:26:21.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.013+0000 7f2a9d7fa640 1 -- 172.21.15.27:0/2663468927 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a98047020 con 0x7f2aa00b7820 2024-03-06T07:26:21.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.013+0000 7f2a9d7fa640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.014+0000 7f2a9d7fa640 1 -- 172.21.15.27:0/2663468927 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a980abf90 con 0x7f2aa00b7820 2024-03-06T07:26:21.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.014+0000 7f2aa491d640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.014+0000 7f2aa491d640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2a90001320 tx=0x7f2a900012b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.018+0000 7f2a9d7fa640 1 -- 172.21.15.27:0/2663468927 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a9804e020 con 0x7f2aa00b7820 2024-03-06T07:26:21.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:21.242+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2aa004f410 con 0x7f2aa00b7820 2024-03-06T07:26:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.242+0000 7f2a9d7fa640 1 -- 172.21.15.27:0/2663468927 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2a9807afc0 con 0x7f2aa00b7820 2024-03-06T07:26:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a8c06b4d0 msgr2=0x7f2a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2a90001320 tx=0x7f2a900012b0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aa00b7820 msgr2=0x7f2aa014d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aa00b7820 0x7f2aa014d320 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2a98009eb0 tx=0x7f2a98004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 shutdown_connections 2024-03-06T07:26:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aa00b7820 0x7f2aa014d320 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00b2d30 0x7f2aa00c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 --2- 172.21.15.27:0/2663468927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa0071540 0x7f2aa01f7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 >> 172.21.15.27:0/2663468927 conn(0x7f2aa00d3d60 msgr2=0x7f2aa015d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 shutdown_connections 2024-03-06T07:26:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.245+0000 7f2aa6b81640 1 -- 172.21.15.27:0/2663468927 wait complete. 2024-03-06T07:26:22.258 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:26:22.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:26:22.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.367+0000 7fd304181640 1 Processor -- start 2024-03-06T07:26:22.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.367+0000 7fd304181640 1 -- start start 2024-03-06T07:26:22.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.368+0000 7fd304181640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2fc15dd90 0x7fd2fc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.369+0000 7fd304181640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0d3740 con 0x7fd2fc0b6d10 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.369+0000 7fd304181640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0d3120 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.369+0000 7fd304181640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2fc0d3400 con 0x7fd2fc15dd90 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.369+0000 7fd301f1d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2fc15dd90 0x7fd2fc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.369+0000 7fd30271e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd2fc0b6d10 0x7fd2fc0d0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38866/0 (socket says 172.21.15.27:38866) 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.369+0000 7fd30271e640 1 -- 172.21.15.27:0/3812106470 learned_addr learned my addr 172.21.15.27:0/3812106470 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.369+0000 7fd301f1d640 1 -- 172.21.15.27:0/3812106470 >> v1:172.21.15.69:6789/0 conn(0x7fd2fc0d29d0 legacy=0x7fd2fc0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.369+0000 7fd301f1d640 1 -- 172.21.15.27:0/3812106470 >> v1:172.21.15.27:6789/0 conn(0x7fd2fc0b6d10 legacy=0x7fd2fc0d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.369+0000 7fd301f1d640 1 -- 172.21.15.27:0/3812106470 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0d39e0 con 0x7fd2fc15dd90 2024-03-06T07:26:22.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.370+0000 7fd301f1d640 1 --2- 172.21.15.27:0/3812106470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc15dd90 0x7fd2fc0ceea0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80097a0 tx=0x7fd2f802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b537020287f0a34b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.370+0000 7fd300f1b640 1 -- 172.21.15.27:0/3812106470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8034070 con 0x7fd2fc15dd90 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.370+0000 7fd300f1b640 1 -- 172.21.15.27:0/3812106470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f8030610 con 0x7fd2fc15dd90 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.370+0000 7fd300f1b640 1 -- 172.21.15.27:0/3812106470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80309e0 con 0x7fd2fc15dd90 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.371+0000 7fd304181640 1 -- 172.21.15.27:0/3812106470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc15dd90 msgr2=0x7fd2fc0ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.371+0000 7fd304181640 1 --2- 172.21.15.27:0/3812106470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc15dd90 0x7fd2fc0ceea0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80097a0 tx=0x7fd2f802b960 comp rx=0 tx=0).stop 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.371+0000 7fd304181640 1 -- 172.21.15.27:0/3812106470 shutdown_connections 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.371+0000 7fd304181640 1 --2- 172.21.15.27:0/3812106470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc15dd90 0x7fd2fc0ceea0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.371+0000 7fd304181640 1 -- 172.21.15.27:0/3812106470 >> 172.21.15.27:0/3812106470 conn(0x7fd2fc071960 msgr2=0x7fd2fc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.371+0000 7fd304181640 1 -- 172.21.15.27:0/3812106470 shutdown_connections 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.371+0000 7fd304181640 1 -- 172.21.15.27:0/3812106470 wait complete. 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.373+0000 7fd304181640 1 Processor -- start 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.373+0000 7fd304181640 1 -- start start 2024-03-06T07:26:22.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.374+0000 7fd304181640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2fc0b6d10 0x7fd2fc1605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.374+0000 7fd304181640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc0d29d0 0x7fd2fc15b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.374+0000 7fd304181640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2fc15dd90 0x7fd2fc15c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd304181640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0d2df0 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd304181640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc15e170 con 0x7fd2fc15dd90 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd304181640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0d48f0 con 0x7fd2fc0b6d10 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30171c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc0d29d0 0x7fd2fc15b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30171c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc0d29d0 0x7fd2fc15b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34810/0 (socket says 172.21.15.27:34810) 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30171c640 1 -- 172.21.15.27:0/3138071031 learned_addr learned my addr 172.21.15.27:0/3138071031 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30271e640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2fc15dd90 0x7fd2fc15c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd301f1d640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2fc0b6d10 0x7fd2fc1605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30171c640 1 -- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2fc0b6d10 msgr2=0x7fd2fc1605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30171c640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2fc0b6d10 0x7fd2fc1605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30171c640 1 -- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2fc15dd90 msgr2=0x7fd2fc15c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30171c640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2fc15dd90 0x7fd2fc15c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30171c640 1 -- 172.21.15.27:0/3138071031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc15f570 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd301f1d640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2fc0b6d10 0x7fd2fc1605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30271e640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2fc15dd90 0x7fd2fc15c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd30171c640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc0d29d0 0x7fd2fc15b760 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00c460 tx=0x7fd2ec00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd2f27fc640 1 -- 172.21.15.27:0/3138071031 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec016020 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd2f27fc640 1 -- 172.21.15.27:0/3138071031 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ec012870 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc15f7f0 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd2f27fc640 1 -- 172.21.15.27:0/3138071031 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec012c60 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.375+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc15fdb0 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.377+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc072a30 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.377+0000 7fd2f27fc640 1 -- 172.21.15.27:0/3138071031 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2ec019020 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.377+0000 7fd2f27fc640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.377+0000 7fd2f27fc640 1 -- 172.21.15.27:0/3138071031 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2ec089bf0 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.378+0000 7fd301f1d640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.378+0000 7fd301f1d640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2e006b4d0 0x7fd2e006d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8036000 tx=0x7fd2f803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.382+0000 7fd2f27fc640 1 -- 172.21.15.27:0/3138071031 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2ec020070 con 0x7fd2fc0d29d0 2024-03-06T07:26:22.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:22.766+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd2fc04f410 con 0x7fd2fc0d29d0 2024-03-06T07:26:23.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.243+0000 7fd2f27fc640 1 -- 172.21.15.27:0/3138071031 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd2fc15f570 con 0x7fd2fc0d29d0 2024-03-06T07:26:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2e006b4d0 msgr2=0x7fd2e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2e006b4d0 0x7fd2e006d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8036000 tx=0x7fd2f803b040 comp rx=0 tx=0).stop 2024-03-06T07:26:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc0d29d0 msgr2=0x7fd2fc15b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc0d29d0 0x7fd2fc15b760 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00c460 tx=0x7fd2ec00c920 comp rx=0 tx=0).stop 2024-03-06T07:26:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 shutdown_connections 2024-03-06T07:26:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2fc15dd90 0x7fd2fc15c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2fc0d29d0 0x7fd2fc15b760 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 --2- 172.21.15.27:0/3138071031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2fc0b6d10 0x7fd2fc1605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 >> 172.21.15.27:0/3138071031 conn(0x7fd2fc071960 msgr2=0x7fd2fc0b8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 shutdown_connections 2024-03-06T07:26:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.246+0000 7fd304181640 1 -- 172.21.15.27:0/3138071031 wait complete. 2024-03-06T07:26:23.258 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:26:23.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.364+0000 7fbec54eb640 1 Processor -- start 2024-03-06T07:26:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.364+0000 7fbec54eb640 1 -- start start 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.364+0000 7fbec54eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbec00b9a10 0x7fbec00d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.364+0000 7fbec54eb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec00cea30 con 0x7fbec015df00 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.364+0000 7fbec54eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec00cf3c0 con 0x7fbec00b95a0 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.364+0000 7fbec54eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec00cf0e0 con 0x7fbec00b9a10 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.365+0000 7fbebeffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbec015df00 0x7fbec00d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38874/0 (socket says 172.21.15.27:38874) 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.365+0000 7fbebeffd640 1 -- 172.21.15.27:0/2522023750 learned_addr learned my addr 172.21.15.27:0/2522023750 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.365+0000 7fbebf7fe640 1 --2- 172.21.15.27:0/2522023750 >> v2:172.21.15.69:3300/0 conn(0x7fbec00b9a10 0x7fbec00d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.365+0000 7fbebe7fc640 1 -- 172.21.15.27:0/2522023750 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957914129 0 0) 0x7fbec00cea30 con 0x7fbec015df00 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.365+0000 7fbebe7fc640 1 -- 172.21.15.27:0/2522023750 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeac003520 con 0x7fbec015df00 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.365+0000 7fbebf7fe640 1 -- 172.21.15.27:0/2522023750 >> v1:172.21.15.69:6789/0 conn(0x7fbec00b95a0 legacy=0x7fbec00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.365+0000 7fbebf7fe640 1 -- 172.21.15.27:0/2522023750 >> v1:172.21.15.27:6789/0 conn(0x7fbec015df00 legacy=0x7fbec00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.365+0000 7fbebf7fe640 1 -- 172.21.15.27:0/2522023750 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec00ce470 con 0x7fbec00b9a10 2024-03-06T07:26:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.366+0000 7fbebf7fe640 1 --2- 172.21.15.27:0/2522023750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 0x7fbec00d3220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbec00cf590 tx=0x7fbeb802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b31ec30e04b750f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.366+0000 7fbebe7fc640 1 -- 172.21.15.27:0/2522023750 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8039070 con 0x7fbec00b9a10 2024-03-06T07:26:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.366+0000 7fbebe7fc640 1 -- 172.21.15.27:0/2522023750 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb802cce0 con 0x7fbec00b9a10 2024-03-06T07:26:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.366+0000 7fbebe7fc640 1 -- 172.21.15.27:0/2522023750 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8034660 con 0x7fbec00b9a10 2024-03-06T07:26:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.367+0000 7fbec54eb640 1 -- 172.21.15.27:0/2522023750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 msgr2=0x7fbec00d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.367+0000 7fbec54eb640 1 --2- 172.21.15.27:0/2522023750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 0x7fbec00d3220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbec00cf590 tx=0x7fbeb802bf30 comp rx=0 tx=0).stop 2024-03-06T07:26:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.367+0000 7fbec54eb640 1 -- 172.21.15.27:0/2522023750 shutdown_connections 2024-03-06T07:26:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.367+0000 7fbec54eb640 1 --2- 172.21.15.27:0/2522023750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 0x7fbec00d3220 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.367+0000 7fbec54eb640 1 -- 172.21.15.27:0/2522023750 >> 172.21.15.27:0/2522023750 conn(0x7fbec0071960 msgr2=0x7fbec0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.367+0000 7fbec54eb640 1 -- 172.21.15.27:0/2522023750 shutdown_connections 2024-03-06T07:26:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.367+0000 7fbec54eb640 1 -- 172.21.15.27:0/2522023750 wait complete. 2024-03-06T07:26:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.368+0000 7fbec54eb640 1 Processor -- start 2024-03-06T07:26:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.368+0000 7fbec54eb640 1 -- start start 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbec54eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbec00b95a0 0x7fbec00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbec54eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 0x7fbec015e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbec54eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbec015df00 0x7fbec01581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbec54eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00cedf0 con 0x7fbec00b95a0 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbec54eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00cec70 con 0x7fbec00b9a10 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbec54eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00b5e10 con 0x7fbec015df00 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbec00b95a0 0x7fbec00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbec00b95a0 0x7fbec00cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34826/0 (socket says 172.21.15.27:34826) 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebeffd640 1 -- 172.21.15.27:0/1912586284 learned_addr learned my addr 172.21.15.27:0/1912586284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbeb7fff640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 0x7fbec015e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebf7fe640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbec015df00 0x7fbec01581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebeffd640 1 -- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbec015df00 msgr2=0x7fbec01581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebeffd640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbec015df00 0x7fbec01581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebeffd640 1 -- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 msgr2=0x7fbec015e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebeffd640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 0x7fbec015e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebeffd640 1 -- 172.21.15.27:0/1912586284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec015e820 con 0x7fbec00b95a0 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbeb7fff640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 0x7fbec015e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbebeffd640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbec00b95a0 0x7fbec00cca70 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000c410 tx=0x7fbeb000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbeb77fe640 1 -- 172.21.15.27:0/1912586284 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0017070 con 0x7fbec00b95a0 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbeb77fe640 1 -- 172.21.15.27:0/1912586284 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb0005ce0 con 0x7fbec00b95a0 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec015eac0 con 0x7fbec00b95a0 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.369+0000 7fbeb77fe640 1 -- 172.21.15.27:0/1912586284 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb00072f0 con 0x7fbec00b95a0 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.370+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec015f080 con 0x7fbec00b95a0 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.371+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec0067ff0 con 0x7fbec00b95a0 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.371+0000 7fbeb77fe640 1 -- 172.21.15.27:0/1912586284 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbeb001a070 con 0x7fbec00b95a0 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.371+0000 7fbeb77fe640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbea006b4d0 0x7fbea006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.372+0000 7fbeb77fe640 1 -- 172.21.15.27:0/1912586284 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbeb0089630 con 0x7fbec00b95a0 2024-03-06T07:26:23.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.372+0000 7fbeb7fff640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbea006b4d0 0x7fbea006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.372+0000 7fbeb7fff640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbea006b4d0 0x7fbea006d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbec00ca720 tx=0x7fbea8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.376+0000 7fbeb77fe640 1 -- 172.21.15.27:0/1912586284 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbeb0058600 con 0x7fbec00b95a0 2024-03-06T07:26:23.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.599+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbec00689a0 con 0x7fbec00b95a0 2024-03-06T07:26:23.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.600+0000 7fbeb77fe640 1 -- 172.21.15.27:0/1912586284 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fbeb0057f00 con 0x7fbec00b95a0 2024-03-06T07:26:23.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:23.602 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:23.602 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:23.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbea006b4d0 msgr2=0x7fbea006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbea006b4d0 0x7fbea006d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbec00ca720 tx=0x7fbea8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:23.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbec00b95a0 msgr2=0x7fbec00cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbec00b95a0 0x7fbec00cca70 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000c410 tx=0x7fbeb000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 shutdown_connections 2024-03-06T07:26:23.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbea006b4d0 0x7fbea006d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbec015df00 0x7fbec01581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbec00b9a10 0x7fbec015e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 --2- 172.21.15.27:0/1912586284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbec00b95a0 0x7fbec00cca70 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 >> 172.21.15.27:0/1912586284 conn(0x7fbec0071960 msgr2=0x7fbec015d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 shutdown_connections 2024-03-06T07:26:23.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.602+0000 7fbec54eb640 1 -- 172.21.15.27:0/1912586284 wait complete. 2024-03-06T07:26:23.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.720+0000 7fe64fadf640 1 Processor -- start 2024-03-06T07:26:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.720+0000 7fe64fadf640 1 -- start start 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64fadf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe64815df00 0x7fe6480d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64fadf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480b95a0 0x7fe6480b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64fadf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6480b9a30 0x7fe6480b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64fadf640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6480cea30 con 0x7fe6480b9a30 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64fadf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6480d2af0 con 0x7fe64815df00 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64fadf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480cf3c0 con 0x7fe6480b95a0 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64e07c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6480b9a30 0x7fe6480b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64e07c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6480b9a30 0x7fe6480b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47412/0 (socket says 172.21.15.27:47412) 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64e07c640 1 -- 172.21.15.27:0/1523313592 learned_addr learned my addr 172.21.15.27:0/1523313592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64d87b640 1 --2- 172.21.15.27:0/1523313592 >> v2:172.21.15.69:3300/0 conn(0x7fe64815df00 0x7fe6480d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64d07a640 1 --2- 172.21.15.27:0/1523313592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480b95a0 0x7fe6480b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64e07c640 1 -- 172.21.15.27:0/1523313592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480b95a0 msgr2=0x7fe6480b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64e07c640 1 --2- 172.21.15.27:0/1523313592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480b95a0 0x7fe6480b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64e07c640 1 -- 172.21.15.27:0/1523313592 >> v2:172.21.15.69:3300/0 conn(0x7fe64815df00 msgr2=0x7fe6480d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64e07c640 1 --2- 172.21.15.27:0/1523313592 >> v2:172.21.15.69:3300/0 conn(0x7fe64815df00 0x7fe6480d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.721+0000 7fe64e07c640 1 -- 172.21.15.27:0/1523313592 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6480cf0e0 con 0x7fe6480b9a30 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.722+0000 7fe64e07c640 1 --2- 172.21.15.27:0/1523313592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6480b9a30 0x7fe6480b5c40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe6480d11b0 tx=0x7fe64002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f8ef83b08860e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.722+0000 7fe64c879640 1 -- 172.21.15.27:0/1523313592 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64002ca70 con 0x7fe6480b9a30 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.722+0000 7fe64c879640 1 -- 172.21.15.27:0/1523313592 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64002cc10 con 0x7fe6480b9a30 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.722+0000 7fe64c879640 1 -- 172.21.15.27:0/1523313592 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6400343f0 con 0x7fe6480b9a30 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.723+0000 7fe64fadf640 1 -- 172.21.15.27:0/1523313592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6480b9a30 msgr2=0x7fe6480b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.723+0000 7fe64fadf640 1 --2- 172.21.15.27:0/1523313592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6480b9a30 0x7fe6480b5c40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe6480d11b0 tx=0x7fe64002bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.723+0000 7fe64fadf640 1 -- 172.21.15.27:0/1523313592 shutdown_connections 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.723+0000 7fe64fadf640 1 --2- 172.21.15.27:0/1523313592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6480b9a30 0x7fe6480b5c40 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.723+0000 7fe64fadf640 1 --2- 172.21.15.27:0/1523313592 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6480b95a0 0x7fe6480b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.723+0000 7fe64fadf640 1 --2- 172.21.15.27:0/1523313592 >> v2:172.21.15.69:3300/0 conn(0x7fe64815df00 0x7fe6480d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.723+0000 7fe64fadf640 1 -- 172.21.15.27:0/1523313592 >> 172.21.15.27:0/1523313592 conn(0x7fe648071960 msgr2=0x7fe648071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.723+0000 7fe64fadf640 1 -- 172.21.15.27:0/1523313592 shutdown_connections 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.723+0000 7fe64fadf640 1 -- 172.21.15.27:0/1523313592 wait complete. 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.724+0000 7fe64fadf640 1 Processor -- start 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.724+0000 7fe64fadf640 1 -- start start 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64fadf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480b95a0 0x7fe6481582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64fadf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe64815df00 0x7fe648160be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64fadf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64815fb50 0x7fe6480b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64fadf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480cec70 con 0x7fe64815df00 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64fadf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480ce750 con 0x7fe6480b95a0 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64fadf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480cedf0 con 0x7fe64815fb50 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64d07a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe64815df00 0x7fe648160be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64d07a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe64815df00 0x7fe648160be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47414/0 (socket says 172.21.15.27:47414) 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64d07a640 1 -- 172.21.15.27:0/4099028483 learned_addr learned my addr 172.21.15.27:0/4099028483 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64d87b640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480b95a0 0x7fe6481582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64e07c640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64815fb50 0x7fe6480b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64d07a640 1 -- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64815fb50 msgr2=0x7fe6480b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64d07a640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64815fb50 0x7fe6480b4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64d07a640 1 -- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480b95a0 msgr2=0x7fe6481582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64d07a640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480b95a0 0x7fe6481582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.725+0000 7fe64d07a640 1 -- 172.21.15.27:0/4099028483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe648161120 con 0x7fe64815df00 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.726+0000 7fe64d07a640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe64815df00 0x7fe648160be0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe63800e870 tx=0x7fe63800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.726+0000 7fe63e7fc640 1 -- 172.21.15.27:0/4099028483 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638017070 con 0x7fe64815df00 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.726+0000 7fe63e7fc640 1 -- 172.21.15.27:0/4099028483 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63800c8e0 con 0x7fe64815df00 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.726+0000 7fe63e7fc640 1 -- 172.21.15.27:0/4099028483 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63800ccd0 con 0x7fe64815df00 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.726+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6480b4550 con 0x7fe64815df00 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.726+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64815f070 con 0x7fe64815df00 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.727+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe648067ff0 con 0x7fe64815df00 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.728+0000 7fe63e7fc640 1 -- 172.21.15.27:0/4099028483 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe63801a070 con 0x7fe64815df00 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.728+0000 7fe63e7fc640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe62c06b4d0 0x7fe62c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.728+0000 7fe63e7fc640 1 -- 172.21.15.27:0/4099028483 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe638089f90 con 0x7fe64815df00 2024-03-06T07:26:23.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.729+0000 7fe64d87b640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe62c06b4d0 0x7fe62c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.729+0000 7fe64d87b640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe62c06b4d0 0x7fe62c06d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe644009ac0 tx=0x7fe644009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.738+0000 7fe63e7fc640 1 -- 172.21.15.27:0/4099028483 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe638058850 con 0x7fe64815df00 2024-03-06T07:26:23.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.919+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6480d1390 con 0x7fe64815df00 2024-03-06T07:26:23.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.921+0000 7fe63e7fc640 1 -- 172.21.15.27:0/4099028483 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16596 (secure 0 0 0) 0x7fe63805c8f0 con 0x7fe64815df00 2024-03-06T07:26:23.922 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:23.923 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":70,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:21.218630+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.914214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.608105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:23.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe62c06b4d0 msgr2=0x7fe62c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe62c06b4d0 0x7fe62c06d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe644009ac0 tx=0x7fe644009190 comp rx=0 tx=0).stop 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe64815df00 msgr2=0x7fe648160be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe64815df00 0x7fe648160be0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe63800e870 tx=0x7fe63800ed30 comp rx=0 tx=0).stop 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 shutdown_connections 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe62c06b4d0 0x7fe62c06d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe64815fb50 0x7fe6480b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe64815df00 0x7fe648160be0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 --2- 172.21.15.27:0/4099028483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6480b95a0 0x7fe6481582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 >> 172.21.15.27:0/4099028483 conn(0x7fe648071960 msgr2=0x7fe64815bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 shutdown_connections 2024-03-06T07:26:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:23.923+0000 7fe64fadf640 1 -- 172.21.15.27:0/4099028483 wait complete. 2024-03-06T07:26:23.936 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:26:23.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.042+0000 7fe7fe6c9640 1 Processor -- start 2024-03-06T07:26:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.042+0000 7fe7fe6c9640 1 -- start start 2024-03-06T07:26:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.043+0000 7fe7fe6c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 0x7fe7f80cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.043+0000 7fe7fe6c9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f80b2320 con 0x7fe7f80d06c0 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.043+0000 7fe7fe6c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f80b0dd0 con 0x7fe7f80b4260 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.043+0000 7fe7fe6c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80b0b80 con 0x7fe7f80d02e0 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.043+0000 7fe7fcc66640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe7f80d06c0 0x7fe7f80b1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47012/0 (socket says 172.21.15.27:47012) 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.043+0000 7fe7fcc66640 1 -- 172.21.15.27:0/1419651338 learned_addr learned my addr 172.21.15.27:0/1419651338 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645057161 0 0) 0x7fe7f80b2320 con 0x7fe7f80d06c0 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e0003520 con 0x7fe7f80d06c0 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f7fff640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 0x7fe7f80cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1290681703 0 0) 0x7fe7e0003520 con 0x7fe7f80d06c0 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 msgr2=0x7fe7f80cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 0x7fe7f80cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 >> v1:172.21.15.69:6789/0 conn(0x7fe7f80b4260 legacy=0x7fe7f80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f80b10b0 con 0x7fe7f80d06c0 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fe7f00019f0 con 0x7fe7f80d06c0 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f7fff640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 0x7fe7f80cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 >> v1:172.21.15.27:6789/0 conn(0x7fe7f80d06c0 legacy=0x7fe7f80b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 0x7fe7e00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0004d40 0x7fe7e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80b0b80 con 0x7fe7e0003fd0 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e000bad0 con 0x7fe7e0004d40 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e000bcd0 con 0x7fe7f80d02e0 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f77fe640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f7fff640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 0x7fe7e00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7fcc66640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0004d40 0x7fe7e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f77fe640 1 -- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 msgr2=0x7fe7e00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f77fe640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 0x7fe7e00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f77fe640 1 -- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0004d40 msgr2=0x7fe7e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f77fe640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0004d40 0x7fe7e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f77fe640 1 -- 172.21.15.27:0/1419651338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0003520 con 0x7fe7e0003fd0 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.044+0000 7fe7f7fff640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80d02e0 0x7fe7e00039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.045+0000 7fe7f77fe640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7e0004800 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80b46a0 tx=0x7fe7e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39779a1ed0712835 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.045+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec038070 con 0x7fe7e0003fd0 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.045+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec033630 con 0x7fe7e0003fd0 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.045+0000 7fe7f6ffd640 1 -- 172.21.15.27:0/1419651338 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec02f050 con 0x7fe7e0003fd0 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.046+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/1419651338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 msgr2=0x7fe7e0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.046+0000 7fe7fe6c9640 1 --2- 172.21.15.27:0/1419651338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7e0004800 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80b46a0 tx=0x7fe7e0005830 comp rx=0 tx=0).stop 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.046+0000 7fe7fcc66640 1 -- 172.21.15.27:0/1419651338 reap_dead start 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.046+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/1419651338 shutdown_connections 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.046+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/1419651338 >> 172.21.15.27:0/1419651338 conn(0x7fe7f80b36d0 msgr2=0x7fe7f80b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.046+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/1419651338 shutdown_connections 2024-03-06T07:26:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.046+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/1419651338 wait complete. 2024-03-06T07:26:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.047+0000 7fe7fe6c9640 1 Processor -- start 2024-03-06T07:26:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7fe6c9640 1 -- start start 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7fe6c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7f8150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7fe6c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f81535b0 0x7fe7f80cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7fe6c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80cb840 0x7fe7f815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7fe6c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80b0360 con 0x7fe7e0003fd0 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7fe6c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80aeec0 con 0x7fe7f81535b0 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7fe6c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80b04e0 con 0x7fe7f80cb840 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7f8150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7f8150d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47428/0 (socket says 172.21.15.27:47428) 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.048+0000 7fe7f7fff640 1 -- 172.21.15.27:0/2454336251 learned_addr learned my addr 172.21.15.27:0/2454336251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f7fff640 1 -- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80cb840 msgr2=0x7fe7f815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7fcc66640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80cb840 0x7fe7f815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f77fe640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f81535b0 0x7fe7f80cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f7fff640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80cb840 0x7fe7f815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f7fff640 1 -- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f81535b0 msgr2=0x7fe7f80cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f7fff640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f81535b0 0x7fe7f80cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f7fff640 1 -- 172.21.15.27:0/2454336251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f81f5130 con 0x7fe7e0003fd0 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f7fff640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7f8150d40 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8010860 tx=0x7fe7e8010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f4ff9640 1 -- 172.21.15.27:0/2454336251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e801a020 con 0x7fe7e0003fd0 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f4ff9640 1 -- 172.21.15.27:0/2454336251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e800eb60 con 0x7fe7e0003fd0 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f81f5410 con 0x7fe7e0003fd0 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7f4ff9640 1 -- 172.21.15.27:0/2454336251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800f040 con 0x7fe7e0003fd0 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.049+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f81f59d0 con 0x7fe7e0003fd0 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.050+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f8068700 con 0x7fe7e0003fd0 2024-03-06T07:26:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.051+0000 7fe7f4ff9640 1 -- 172.21.15.27:0/2454336251 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7e801d020 con 0x7fe7e0003fd0 2024-03-06T07:26:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.052+0000 7fe7f4ff9640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7e406b4d0 0x7fe7e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.052+0000 7fe7f4ff9640 1 -- 172.21.15.27:0/2454336251 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7e808ab50 con 0x7fe7e0003fd0 2024-03-06T07:26:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.052+0000 7fe7f77fe640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7e406b4d0 0x7fe7e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.053+0000 7fe7f77fe640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7e406b4d0 0x7fe7e406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec035000 tx=0x7fe7ec009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.056+0000 7fe7f4ff9640 1 -- 172.21.15.27:0/2454336251 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7e8024020 con 0x7fe7e0003fd0 2024-03-06T07:26:24.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.234+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7f806f210 con 0x7fe7e406b4d0 2024-03-06T07:26:24.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.240+0000 7fe7f4ff9640 1 -- 172.21.15.27:0/2454336251 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246754 (secure 0 0 0) 0x7fe7f806f210 con 0x7fe7e406b4d0 2024-03-06T07:26:24.244 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:24.246 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:26:24.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.243+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7e406b4d0 msgr2=0x7fe7e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.243+0000 7fe7fe6c9640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7e406b4d0 0x7fe7e406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec035000 tx=0x7fe7ec009030 comp rx=0 tx=0).stop 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 msgr2=0x7fe7f8150d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7f8150d40 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8010860 tx=0x7fe7e8010d20 comp rx=0 tx=0).stop 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 shutdown_connections 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7e406b4d0 0x7fe7e406d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7f80cb840 0x7fe7f815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7f81535b0 0x7fe7f80cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 --2- 172.21.15.27:0/2454336251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e0003fd0 0x7fe7f8150d40 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 >> 172.21.15.27:0/2454336251 conn(0x7fe7f80b36d0 msgr2=0x7fe7f8151f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 shutdown_connections 2024-03-06T07:26:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.244+0000 7fe7fe6c9640 1 -- 172.21.15.27:0/2454336251 wait complete. 2024-03-06T07:26:24.257 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-03-06T07:26:24.038057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257056,"kb_used_data":7600,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717664,"statfs":{"total":773094113280,"available":772830887936,"internally_reserved":0,"allocated":7782400,"data_stored":3918114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.825797"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227981+0000","last_change":"2024-03-06T07:26:19.227981+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227981+0000","last_undegraded":"2024-03-06T07:26:19.227981+0000","last_fullsized":"2024-03-06T07:26:19.227981+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228347+0000","last_change":"2024-03-06T07:26:19.228347+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228347+0000","last_undegraded":"2024-03-06T07:26:19.228347+0000","last_fullsized":"2024-03-06T07:26:19.228347+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228671+0000","last_change":"2024-03-06T07:26:19.228671+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228671+0000","last_undegraded":"2024-03-06T07:26:19.228671+0000","last_fullsized":"2024-03-06T07:26:19.228671+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224081+0000","last_change":"2024-03-06T07:26:19.224081+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224081+0000","last_undegraded":"2024-03-06T07:26:19.224081+0000","last_fullsized":"2024-03-06T07:26:19.224081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227451+0000","last_change":"2024-03-06T07:26:19.227451+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227451+0000","last_undegraded":"2024-03-06T07:26:19.227451+0000","last_fullsized":"2024-03-06T07:26:19.227451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228991+0000","last_change":"2024-03-06T07:26:19.228991+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228991+0000","last_undegraded":"2024-03-06T07:26:19.228991+0000","last_fullsized":"2024-03-06T07:26:19.228991+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231923+0000","last_change":"2024-03-06T07:26:19.231923+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231923+0000","last_undegraded":"2024-03-06T07:26:19.231923+0000","last_fullsized":"2024-03-06T07:26:19.231923+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228094+0000","last_change":"2024-03-06T07:26:19.228094+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228094+0000","last_undegraded":"2024-03-06T07:26:19.228094+0000","last_fullsized":"2024-03-06T07:26:19.228094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226958+0000","last_change":"2024-03-06T07:26:19.226958+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226958+0000","last_undegraded":"2024-03-06T07:26:19.226958+0000","last_fullsized":"2024-03-06T07:26:19.226958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224802+0000","last_change":"2024-03-06T07:26:19.224802+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224802+0000","last_undegraded":"2024-03-06T07:26:19.224802+0000","last_fullsized":"2024-03-06T07:26:19.224802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231680+0000","last_change":"2024-03-06T07:26:19.231680+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231680+0000","last_undegraded":"2024-03-06T07:26:19.231680+0000","last_fullsized":"2024-03-06T07:26:19.231680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223377+0000","last_change":"2024-03-06T07:26:19.223377+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223377+0000","last_undegraded":"2024-03-06T07:26:19.223377+0000","last_fullsized":"2024-03-06T07:26:19.223377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221736+0000","last_change":"2024-03-06T07:26:19.221736+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221736+0000","last_undegraded":"2024-03-06T07:26:19.221736+0000","last_fullsized":"2024-03-06T07:26:19.221736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226495+0000","last_change":"2024-03-06T07:26:19.226495+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226495+0000","last_undegraded":"2024-03-06T07:26:19.226495+0000","last_fullsized":"2024-03-06T07:26:19.226495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222226+0000","last_change":"2024-03-06T07:26:19.222226+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222226+0000","last_undegraded":"2024-03-06T07:26:19.222226+0000","last_fullsized":"2024-03-06T07:26:19.222226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224253+0000","last_change":"2024-03-06T07:26:19.224253+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224253+0000","last_undegraded":"2024-03-06T07:26:19.224253+0000","last_fullsized":"2024-03-06T07:26:19.224253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225964+0000","last_change":"2024-03-06T07:26:19.225964+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225964+0000","last_undegraded":"2024-03-06T07:26:19.225964+0000","last_fullsized":"2024-03-06T07:26:19.225964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221558+0000","last_change":"2024-03-06T07:26:19.221558+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221558+0000","last_undegraded":"2024-03-06T07:26:19.221558+0000","last_fullsized":"2024-03-06T07:26:19.221558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228445+0000","last_change":"2024-03-06T07:26:19.228445+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228445+0000","last_undegraded":"2024-03-06T07:26:19.228445+0000","last_fullsized":"2024-03-06T07:26:19.228445+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223388+0000","last_change":"2024-03-06T07:26:19.223388+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223388+0000","last_undegraded":"2024-03-06T07:26:19.223388+0000","last_fullsized":"2024-03-06T07:26:19.223388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223534+0000","last_change":"2024-03-06T07:26:19.223534+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223534+0000","last_undegraded":"2024-03-06T07:26:19.223534+0000","last_fullsized":"2024-03-06T07:26:19.223534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223910+0000","last_change":"2024-03-06T07:26:19.223910+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223910+0000","last_undegraded":"2024-03-06T07:26:19.223910+0000","last_fullsized":"2024-03-06T07:26:19.223910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222903+0000","last_change":"2024-03-06T07:26:19.222903+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222903+0000","last_undegraded":"2024-03-06T07:26:19.222903+0000","last_fullsized":"2024-03-06T07:26:19.222903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224000+0000","last_change":"2024-03-06T07:26:19.224000+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224000+0000","last_undegraded":"2024-03-06T07:26:19.224000+0000","last_fullsized":"2024-03-06T07:26:19.224000+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222350+0000","last_change":"2024-03-06T07:26:19.222350+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222350+0000","last_undegraded":"2024-03-06T07:26:19.222350+0000","last_fullsized":"2024-03-06T07:26:19.222350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229681+0000","last_change":"2024-03-06T07:26:19.229681+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229681+0000","last_undegraded":"2024-03-06T07:26:19.229681+0000","last_fullsized":"2024-03-06T07:26:19.229681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228249+0000","last_change":"2024-03-06T07:26:19.228249+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228249+0000","last_undegraded":"2024-03-06T07:26:19.228249+0000","last_fullsized":"2024-03-06T07:26:19.228249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224326+0000","last_change":"2024-03-06T07:26:19.224326+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224326+0000","last_undegraded":"2024-03-06T07:26:19.224326+0000","last_fullsized":"2024-03-06T07:26:19.224326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228898+0000","last_change":"2024-03-06T07:26:19.228898+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228898+0000","last_undegraded":"2024-03-06T07:26:19.228898+0000","last_fullsized":"2024-03-06T07:26:19.228898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231473+0000","last_change":"2024-03-06T07:26:19.231473+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231473+0000","last_undegraded":"2024-03-06T07:26:19.231473+0000","last_fullsized":"2024-03-06T07:26:19.231473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227762+0000","last_change":"2024-03-06T07:26:19.227762+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227762+0000","last_undegraded":"2024-03-06T07:26:19.227762+0000","last_fullsized":"2024-03-06T07:26:19.227762+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227670+0000","last_change":"2024-03-06T07:26:19.227670+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227670+0000","last_undegraded":"2024-03-06T07:26:19.227670+0000","last_fullsized":"2024-03-06T07:26:19.227670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225593+0000","last_change":"2024-03-06T07:26:19.225593+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225593+0000","last_undegraded":"2024-03-06T07:26:19.225593+0000","last_fullsized":"2024-03-06T07:26:19.225593+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239388+0000","last_change":"2024-03-06T07:26:19.235249+0000","last_active":"2024-03-06T07:26:19.239388+0000","last_peered":"2024-03-06T07:26:19.239388+0000","last_clean":"2024-03-06T07:26:19.239388+0000","last_became_active":"2024-03-06T07:26:19.234904+0000","last_became_peered":"2024-03-06T07:26:19.234904+0000","last_unstale":"2024-03-06T07:26:19.239388+0000","last_undegraded":"2024-03-06T07:26:19.239388+0000","last_fullsized":"2024-03-06T07:26:19.239388+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:23.463441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225476+0000","last_change":"2024-03-06T07:26:19.225476+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225476+0000","last_undegraded":"2024-03-06T07:26:19.225476+0000","last_fullsized":"2024-03-06T07:26:19.225476+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.233898+0000","last_change":"2024-03-06T07:26:19.225962+0000","last_active":"2024-03-06T07:26:19.233898+0000","last_peered":"2024-03-06T07:26:19.233898+0000","last_clean":"2024-03-06T07:26:19.233898+0000","last_became_active":"2024-03-06T07:26:19.225707+0000","last_became_peered":"2024-03-06T07:26:19.225707+0000","last_unstale":"2024-03-06T07:26:19.233898+0000","last_undegraded":"2024-03-06T07:26:19.233898+0000","last_fullsized":"2024-03-06T07:26:19.233898+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:50.887101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228771+0000","last_change":"2024-03-06T07:26:19.228771+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228771+0000","last_undegraded":"2024-03-06T07:26:19.228771+0000","last_fullsized":"2024-03-06T07:26:19.228771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235050+0000","last_change":"2024-03-06T07:26:19.232800+0000","last_active":"2024-03-06T07:26:19.235050+0000","last_peered":"2024-03-06T07:26:19.235050+0000","last_clean":"2024-03-06T07:26:19.235050+0000","last_became_active":"2024-03-06T07:26:19.232541+0000","last_became_peered":"2024-03-06T07:26:19.232541+0000","last_unstale":"2024-03-06T07:26:19.235050+0000","last_undegraded":"2024-03-06T07:26:19.235050+0000","last_fullsized":"2024-03-06T07:26:19.235050+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:49.533898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.230992+0000","last_change":"2024-03-06T07:26:19.230992+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.230992+0000","last_undegraded":"2024-03-06T07:26:19.230992+0000","last_fullsized":"2024-03-06T07:26:19.230992+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239827+0000","last_change":"2024-03-06T07:26:19.237329+0000","last_active":"2024-03-06T07:26:19.239827+0000","last_peered":"2024-03-06T07:26:19.239827+0000","last_clean":"2024-03-06T07:26:19.239827+0000","last_became_active":"2024-03-06T07:26:19.237096+0000","last_became_peered":"2024-03-06T07:26:19.237096+0000","last_unstale":"2024-03-06T07:26:19.239827+0000","last_undegraded":"2024-03-06T07:26:19.239827+0000","last_fullsized":"2024-03-06T07:26:19.239827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:54.603601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231064+0000","last_change":"2024-03-06T07:26:19.231064+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231064+0000","last_undegraded":"2024-03-06T07:26:19.231064+0000","last_fullsized":"2024-03-06T07:26:19.231064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.237618+0000","last_change":"2024-03-06T07:26:19.236188+0000","last_active":"2024-03-06T07:26:19.237618+0000","last_peered":"2024-03-06T07:26:19.237618+0000","last_clean":"2024-03-06T07:26:19.237618+0000","last_became_active":"2024-03-06T07:26:19.235921+0000","last_became_peered":"2024-03-06T07:26:19.235921+0000","last_unstale":"2024-03-06T07:26:19.237618+0000","last_undegraded":"2024-03-06T07:26:19.237618+0000","last_fullsized":"2024-03-06T07:26:19.237618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:13.224175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222902+0000","last_change":"2024-03-06T07:26:19.222902+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222902+0000","last_undegraded":"2024-03-06T07:26:19.222902+0000","last_fullsized":"2024-03-06T07:26:19.222902+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242242+0000","last_change":"2024-03-06T07:26:19.237984+0000","last_active":"2024-03-06T07:26:19.242242+0000","last_peered":"2024-03-06T07:26:19.242242+0000","last_clean":"2024-03-06T07:26:19.242242+0000","last_became_active":"2024-03-06T07:26:19.237697+0000","last_became_peered":"2024-03-06T07:26:19.237697+0000","last_unstale":"2024-03-06T07:26:19.242242+0000","last_undegraded":"2024-03-06T07:26:19.242242+0000","last_fullsized":"2024-03-06T07:26:19.242242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:21.157010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225590+0000","last_change":"2024-03-06T07:26:19.225590+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225590+0000","last_undegraded":"2024-03-06T07:26:19.225590+0000","last_fullsized":"2024-03-06T07:26:19.225590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.232619+0000","last_change":"2024-03-06T07:26:19.228888+0000","last_active":"2024-03-06T07:26:19.232619+0000","last_peered":"2024-03-06T07:26:19.232619+0000","last_clean":"2024-03-06T07:26:19.232619+0000","last_became_active":"2024-03-06T07:26:19.228602+0000","last_became_peered":"2024-03-06T07:26:19.228602+0000","last_unstale":"2024-03-06T07:26:19.232619+0000","last_undegraded":"2024-03-06T07:26:19.232619+0000","last_fullsized":"2024-03-06T07:26:19.232619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:55.021248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221713+0000","last_change":"2024-03-06T07:26:19.221713+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221713+0000","last_undegraded":"2024-03-06T07:26:19.221713+0000","last_fullsized":"2024-03-06T07:26:19.221713+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.234079+0000","last_change":"2024-03-06T07:26:19.231771+0000","last_active":"2024-03-06T07:26:19.234079+0000","last_peered":"2024-03-06T07:26:19.234079+0000","last_clean":"2024-03-06T07:26:19.234079+0000","last_became_active":"2024-03-06T07:26:19.231485+0000","last_became_peered":"2024-03-06T07:26:19.231485+0000","last_unstale":"2024-03-06T07:26:19.234079+0000","last_undegraded":"2024-03-06T07:26:19.234079+0000","last_fullsized":"2024-03-06T07:26:19.234079+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:30.451695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229259+0000","last_change":"2024-03-06T07:26:19.229259+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229259+0000","last_undegraded":"2024-03-06T07:26:19.229259+0000","last_fullsized":"2024-03-06T07:26:19.229259+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.230928+0000","last_change":"2024-03-06T07:26:19.229136+0000","last_active":"2024-03-06T07:26:19.230928+0000","last_peered":"2024-03-06T07:26:19.230928+0000","last_clean":"2024-03-06T07:26:19.230928+0000","last_became_active":"2024-03-06T07:26:19.228661+0000","last_became_peered":"2024-03-06T07:26:19.228661+0000","last_unstale":"2024-03-06T07:26:19.230928+0000","last_undegraded":"2024-03-06T07:26:19.230928+0000","last_fullsized":"2024-03-06T07:26:19.230928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:56.368954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225377+0000","last_change":"2024-03-06T07:26:19.225377+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225377+0000","last_undegraded":"2024-03-06T07:26:19.225377+0000","last_fullsized":"2024-03-06T07:26:19.225377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239800+0000","last_change":"2024-03-06T07:26:19.236797+0000","last_active":"2024-03-06T07:26:19.239800+0000","last_peered":"2024-03-06T07:26:19.239800+0000","last_clean":"2024-03-06T07:26:19.239800+0000","last_became_active":"2024-03-06T07:26:19.236118+0000","last_became_peered":"2024-03-06T07:26:19.236118+0000","last_unstale":"2024-03-06T07:26:19.239800+0000","last_undegraded":"2024-03-06T07:26:19.239800+0000","last_fullsized":"2024-03-06T07:26:19.239800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:04.139082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228026+0000","last_change":"2024-03-06T07:26:19.228026+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228026+0000","last_undegraded":"2024-03-06T07:26:19.228026+0000","last_fullsized":"2024-03-06T07:26:19.228026+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.236116+0000","last_change":"2024-03-06T07:26:19.235158+0000","last_active":"2024-03-06T07:26:19.236116+0000","last_peered":"2024-03-06T07:26:19.236116+0000","last_clean":"2024-03-06T07:26:19.236116+0000","last_became_active":"2024-03-06T07:26:19.234883+0000","last_became_peered":"2024-03-06T07:26:19.234883+0000","last_unstale":"2024-03-06T07:26:19.236116+0000","last_undegraded":"2024-03-06T07:26:19.236116+0000","last_fullsized":"2024-03-06T07:26:19.236116+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:17.059043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225110+0000","last_change":"2024-03-06T07:26:19.225110+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225110+0000","last_undegraded":"2024-03-06T07:26:19.225110+0000","last_fullsized":"2024-03-06T07:26:19.225110+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.238717+0000","last_change":"2024-03-06T07:26:19.236129+0000","last_active":"2024-03-06T07:26:19.238717+0000","last_peered":"2024-03-06T07:26:19.238717+0000","last_clean":"2024-03-06T07:26:19.238717+0000","last_became_active":"2024-03-06T07:26:19.235844+0000","last_became_peered":"2024-03-06T07:26:19.235844+0000","last_unstale":"2024-03-06T07:26:19.238717+0000","last_undegraded":"2024-03-06T07:26:19.238717+0000","last_fullsized":"2024-03-06T07:26:19.238717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:04.139082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223551+0000","last_change":"2024-03-06T07:26:19.223551+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223551+0000","last_undegraded":"2024-03-06T07:26:19.223551+0000","last_fullsized":"2024-03-06T07:26:19.223551+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.228861+0000","last_change":"2024-03-06T07:26:19.226330+0000","last_active":"2024-03-06T07:26:19.228861+0000","last_peered":"2024-03-06T07:26:19.228861+0000","last_clean":"2024-03-06T07:26:19.228861+0000","last_became_active":"2024-03-06T07:26:19.226062+0000","last_became_peered":"2024-03-06T07:26:19.226062+0000","last_unstale":"2024-03-06T07:26:19.228861+0000","last_undegraded":"2024-03-06T07:26:19.228861+0000","last_fullsized":"2024-03-06T07:26:19.228861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:29.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.230238+0000","last_change":"2024-03-06T07:26:19.230238+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.230238+0000","last_undegraded":"2024-03-06T07:26:19.230238+0000","last_fullsized":"2024-03-06T07:26:19.230238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242264+0000","last_change":"2024-03-06T07:26:19.238616+0000","last_active":"2024-03-06T07:26:19.242264+0000","last_peered":"2024-03-06T07:26:19.242264+0000","last_clean":"2024-03-06T07:26:19.242264+0000","last_became_active":"2024-03-06T07:26:19.238337+0000","last_became_peered":"2024-03-06T07:26:19.238337+0000","last_unstale":"2024-03-06T07:26:19.242264+0000","last_undegraded":"2024-03-06T07:26:19.242264+0000","last_fullsized":"2024-03-06T07:26:19.242264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:14.382999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222417+0000","last_change":"2024-03-06T07:26:19.222417+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222417+0000","last_undegraded":"2024-03-06T07:26:19.222417+0000","last_fullsized":"2024-03-06T07:26:19.222417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239417+0000","last_change":"2024-03-06T07:26:19.236852+0000","last_active":"2024-03-06T07:26:19.239417+0000","last_peered":"2024-03-06T07:26:19.239417+0000","last_clean":"2024-03-06T07:26:19.239417+0000","last_became_active":"2024-03-06T07:26:19.236469+0000","last_became_peered":"2024-03-06T07:26:19.236469+0000","last_unstale":"2024-03-06T07:26:19.239417+0000","last_undegraded":"2024-03-06T07:26:19.239417+0000","last_fullsized":"2024-03-06T07:26:19.239417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:20.767924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224910+0000","last_change":"2024-03-06T07:26:19.224910+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224910+0000","last_undegraded":"2024-03-06T07:26:19.224910+0000","last_fullsized":"2024-03-06T07:26:19.224910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235094+0000","last_change":"2024-03-06T07:26:19.231717+0000","last_active":"2024-03-06T07:26:19.235094+0000","last_peered":"2024-03-06T07:26:19.235094+0000","last_clean":"2024-03-06T07:26:19.235094+0000","last_became_active":"2024-03-06T07:26:19.231462+0000","last_became_peered":"2024-03-06T07:26:19.231462+0000","last_unstale":"2024-03-06T07:26:19.235094+0000","last_undegraded":"2024-03-06T07:26:19.235094+0000","last_fullsized":"2024-03-06T07:26:19.235094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.795497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223271+0000","last_change":"2024-03-06T07:26:19.223271+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223271+0000","last_undegraded":"2024-03-06T07:26:19.223271+0000","last_fullsized":"2024-03-06T07:26:19.223271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239822+0000","last_change":"2024-03-06T07:26:19.237309+0000","last_active":"2024-03-06T07:26:19.239822+0000","last_peered":"2024-03-06T07:26:19.239822+0000","last_clean":"2024-03-06T07:26:19.239822+0000","last_became_active":"2024-03-06T07:26:19.237011+0000","last_became_peered":"2024-03-06T07:26:19.237011+0000","last_unstale":"2024-03-06T07:26:19.239822+0000","last_undegraded":"2024-03-06T07:26:19.239822+0000","last_fullsized":"2024-03-06T07:26:19.239822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:58.527722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224977+0000","last_change":"2024-03-06T07:26:19.224977+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224977+0000","last_undegraded":"2024-03-06T07:26:19.224977+0000","last_fullsized":"2024-03-06T07:26:19.224977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235073+0000","last_change":"2024-03-06T07:26:19.233492+0000","last_active":"2024-03-06T07:26:19.235073+0000","last_peered":"2024-03-06T07:26:19.235073+0000","last_clean":"2024-03-06T07:26:19.235073+0000","last_became_active":"2024-03-06T07:26:19.233199+0000","last_became_peered":"2024-03-06T07:26:19.233199+0000","last_unstale":"2024-03-06T07:26:19.235073+0000","last_undegraded":"2024-03-06T07:26:19.235073+0000","last_fullsized":"2024-03-06T07:26:19.235073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:22.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222478+0000","last_change":"2024-03-06T07:26:19.222478+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222478+0000","last_undegraded":"2024-03-06T07:26:19.222478+0000","last_fullsized":"2024-03-06T07:26:19.222478+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239765+0000","last_change":"2024-03-06T07:26:19.236432+0000","last_active":"2024-03-06T07:26:19.239765+0000","last_peered":"2024-03-06T07:26:19.239765+0000","last_clean":"2024-03-06T07:26:19.239765+0000","last_became_active":"2024-03-06T07:26:19.235975+0000","last_became_peered":"2024-03-06T07:26:19.235975+0000","last_unstale":"2024-03-06T07:26:19.239765+0000","last_undegraded":"2024-03-06T07:26:19.239765+0000","last_fullsized":"2024-03-06T07:26:19.239765+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:55.021248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224496+0000","last_change":"2024-03-06T07:26:19.224496+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224496+0000","last_undegraded":"2024-03-06T07:26:19.224496+0000","last_fullsized":"2024-03-06T07:26:19.224496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.234690+0000","last_change":"2024-03-06T07:26:19.233564+0000","last_active":"2024-03-06T07:26:19.234690+0000","last_peered":"2024-03-06T07:26:19.234690+0000","last_clean":"2024-03-06T07:26:19.234690+0000","last_became_active":"2024-03-06T07:26:19.233235+0000","last_became_peered":"2024-03-06T07:26:19.233235+0000","last_unstale":"2024-03-06T07:26:19.234690+0000","last_undegraded":"2024-03-06T07:26:19.234690+0000","last_fullsized":"2024-03-06T07:26:19.234690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:44.732093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228500+0000","last_change":"2024-03-06T07:26:19.228500+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228500+0000","last_undegraded":"2024-03-06T07:26:19.228500+0000","last_fullsized":"2024-03-06T07:26:19.228500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239745+0000","last_change":"2024-03-06T07:26:19.236405+0000","last_active":"2024-03-06T07:26:19.239745+0000","last_peered":"2024-03-06T07:26:19.239745+0000","last_clean":"2024-03-06T07:26:19.239745+0000","last_became_active":"2024-03-06T07:26:19.235876+0000","last_became_peered":"2024-03-06T07:26:19.235876+0000","last_unstale":"2024-03-06T07:26:19.239745+0000","last_undegraded":"2024-03-06T07:26:19.239745+0000","last_fullsized":"2024-03-06T07:26:19.239745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:29.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225680+0000","last_change":"2024-03-06T07:26:19.225680+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225680+0000","last_undegraded":"2024-03-06T07:26:19.225680+0000","last_fullsized":"2024-03-06T07:26:19.225680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242212+0000","last_change":"2024-03-06T07:26:19.238646+0000","last_active":"2024-03-06T07:26:19.242212+0000","last_peered":"2024-03-06T07:26:19.242212+0000","last_clean":"2024-03-06T07:26:19.242212+0000","last_became_active":"2024-03-06T07:26:19.238376+0000","last_became_peered":"2024-03-06T07:26:19.238376+0000","last_unstale":"2024-03-06T07:26:19.242212+0000","last_undegraded":"2024-03-06T07:26:19.242212+0000","last_fullsized":"2024-03-06T07:26:19.242212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:33.080991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225633+0000","last_change":"2024-03-06T07:26:19.225633+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225633+0000","last_undegraded":"2024-03-06T07:26:19.225633+0000","last_fullsized":"2024-03-06T07:26:19.225633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.233488+0000","last_change":"2024-03-06T07:26:19.227589+0000","last_active":"2024-03-06T07:26:19.233488+0000","last_peered":"2024-03-06T07:26:19.233488+0000","last_clean":"2024-03-06T07:26:19.233488+0000","last_became_active":"2024-03-06T07:26:19.227308+0000","last_became_peered":"2024-03-06T07:26:19.227308+0000","last_unstale":"2024-03-06T07:26:19.233488+0000","last_undegraded":"2024-03-06T07:26:19.233488+0000","last_fullsized":"2024-03-06T07:26:19.233488+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:19.014952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222420+0000","last_change":"2024-03-06T07:26:19.222420+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222420+0000","last_undegraded":"2024-03-06T07:26:19.222420+0000","last_fullsized":"2024-03-06T07:26:19.222420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239645+0000","last_change":"2024-03-06T07:26:19.234268+0000","last_active":"2024-03-06T07:26:19.239645+0000","last_peered":"2024-03-06T07:26:19.239645+0000","last_clean":"2024-03-06T07:26:19.239645+0000","last_became_active":"2024-03-06T07:26:19.234000+0000","last_became_peered":"2024-03-06T07:26:19.234000+0000","last_unstale":"2024-03-06T07:26:19.239645+0000","last_undegraded":"2024-03-06T07:26:19.239645+0000","last_fullsized":"2024-03-06T07:26:19.239645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:50.102543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229363+0000","last_change":"2024-03-06T07:26:19.229363+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229363+0000","last_undegraded":"2024-03-06T07:26:19.229363+0000","last_fullsized":"2024-03-06T07:26:19.229363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242098+0000","last_change":"2024-03-06T07:26:19.237743+0000","last_active":"2024-03-06T07:26:19.242098+0000","last_peered":"2024-03-06T07:26:19.242098+0000","last_clean":"2024-03-06T07:26:19.242098+0000","last_became_active":"2024-03-06T07:26:19.237394+0000","last_became_peered":"2024-03-06T07:26:19.237394+0000","last_unstale":"2024-03-06T07:26:19.242098+0000","last_undegraded":"2024-03-06T07:26:19.242098+0000","last_fullsized":"2024-03-06T07:26:19.242098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:35.749995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225400+0000","last_change":"2024-03-06T07:26:19.225400+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225400+0000","last_undegraded":"2024-03-06T07:26:19.225400+0000","last_fullsized":"2024-03-06T07:26:19.225400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.231970+0000","last_change":"2024-03-06T07:26:19.230603+0000","last_active":"2024-03-06T07:26:19.231970+0000","last_peered":"2024-03-06T07:26:19.231970+0000","last_clean":"2024-03-06T07:26:19.231970+0000","last_became_active":"2024-03-06T07:26:19.230313+0000","last_became_peered":"2024-03-06T07:26:19.230313+0000","last_unstale":"2024-03-06T07:26:19.231970+0000","last_undegraded":"2024-03-06T07:26:19.231970+0000","last_fullsized":"2024-03-06T07:26:19.231970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:31.518943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226436+0000","last_change":"2024-03-06T07:26:19.226436+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226436+0000","last_undegraded":"2024-03-06T07:26:19.226436+0000","last_fullsized":"2024-03-06T07:26:19.226436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.236928+0000","last_change":"2024-03-06T07:26:19.234353+0000","last_active":"2024-03-06T07:26:19.236928+0000","last_peered":"2024-03-06T07:26:19.236928+0000","last_clean":"2024-03-06T07:26:19.236928+0000","last_became_active":"2024-03-06T07:26:19.234077+0000","last_became_peered":"2024-03-06T07:26:19.234077+0000","last_unstale":"2024-03-06T07:26:19.236928+0000","last_undegraded":"2024-03-06T07:26:19.236928+0000","last_fullsized":"2024-03-06T07:26:19.236928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:37.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221555+0000","last_change":"2024-03-06T07:26:19.221555+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221555+0000","last_undegraded":"2024-03-06T07:26:19.221555+0000","last_fullsized":"2024-03-06T07:26:19.221555+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.238279+0000","last_change":"2024-03-06T07:26:19.237287+0000","last_active":"2024-03-06T07:26:19.238279+0000","last_peered":"2024-03-06T07:26:19.238279+0000","last_clean":"2024-03-06T07:26:19.238279+0000","last_became_active":"2024-03-06T07:26:19.237017+0000","last_became_peered":"2024-03-06T07:26:19.237017+0000","last_unstale":"2024-03-06T07:26:19.238279+0000","last_undegraded":"2024-03-06T07:26:19.238279+0000","last_fullsized":"2024-03-06T07:26:19.238279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:46.817231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227191+0000","last_change":"2024-03-06T07:26:19.227191+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227191+0000","last_undegraded":"2024-03-06T07:26:19.227191+0000","last_fullsized":"2024-03-06T07:26:19.227191+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.232804+0000","last_change":"2024-03-06T07:26:19.229645+0000","last_active":"2024-03-06T07:26:19.232804+0000","last_peered":"2024-03-06T07:26:19.232804+0000","last_clean":"2024-03-06T07:26:19.232804+0000","last_became_active":"2024-03-06T07:26:19.229309+0000","last_became_peered":"2024-03-06T07:26:19.229309+0000","last_unstale":"2024-03-06T07:26:19.232804+0000","last_undegraded":"2024-03-06T07:26:19.232804+0000","last_fullsized":"2024-03-06T07:26:19.232804+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.342246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222416+0000","last_change":"2024-03-06T07:26:19.222416+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222416+0000","last_undegraded":"2024-03-06T07:26:19.222416+0000","last_fullsized":"2024-03-06T07:26:19.222416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235092+0000","last_change":"2024-03-06T07:26:19.231241+0000","last_active":"2024-03-06T07:26:19.235092+0000","last_peered":"2024-03-06T07:26:19.235092+0000","last_clean":"2024-03-06T07:26:19.235092+0000","last_became_active":"2024-03-06T07:26:19.230977+0000","last_became_peered":"2024-03-06T07:26:19.230977+0000","last_unstale":"2024-03-06T07:26:19.235092+0000","last_undegraded":"2024-03-06T07:26:19.235092+0000","last_fullsized":"2024-03-06T07:26:19.235092+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:36.204568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227602+0000","last_change":"2024-03-06T07:26:19.227602+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227602+0000","last_undegraded":"2024-03-06T07:26:19.227602+0000","last_fullsized":"2024-03-06T07:26:19.227602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242260+0000","last_change":"2024-03-06T07:26:19.238002+0000","last_active":"2024-03-06T07:26:19.242260+0000","last_peered":"2024-03-06T07:26:19.242260+0000","last_clean":"2024-03-06T07:26:19.242260+0000","last_became_active":"2024-03-06T07:26:19.237710+0000","last_became_peered":"2024-03-06T07:26:19.237710+0000","last_unstale":"2024-03-06T07:26:19.242260+0000","last_undegraded":"2024-03-06T07:26:19.242260+0000","last_fullsized":"2024-03-06T07:26:19.242260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:25.030580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226570+0000","last_change":"2024-03-06T07:26:19.226570+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226570+0000","last_undegraded":"2024-03-06T07:26:19.226570+0000","last_fullsized":"2024-03-06T07:26:19.226570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239767+0000","last_change":"2024-03-06T07:26:19.236691+0000","last_active":"2024-03-06T07:26:19.239767+0000","last_peered":"2024-03-06T07:26:19.239767+0000","last_clean":"2024-03-06T07:26:19.239767+0000","last_became_active":"2024-03-06T07:26:19.236432+0000","last_became_peered":"2024-03-06T07:26:19.236432+0000","last_unstale":"2024-03-06T07:26:19.239767+0000","last_undegraded":"2024-03-06T07:26:19.239767+0000","last_fullsized":"2024-03-06T07:26:19.239767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.342246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":261,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.224547+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:26:19.224547+0000","last_peered":"2024-03-06T07:26:19.224547+0000","last_clean":"2024-03-06T07:26:19.224547+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:26:19.224547+0000","last_undegraded":"2024-03-06T07:26:19.224547+0000","last_fullsized":"2024-03-06T07:26:19.224547+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":815104,"data_stored":337466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":337466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:24.262 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:24.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.402+0000 7f80367bf640 1 Processor -- start 2024-03-06T07:26:24.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.402+0000 7f80367bf640 1 -- start start 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f80367bf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f80300d3400 0x7f80300b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f80367bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f80300b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f80367bf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80300ce610 con 0x7f8030071540 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f80367bf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80300d2f70 con 0x7f80300d3400 2024-03-06T07:26:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f80367bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80300d4be0 con 0x7f80300b2b50 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f8034d5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f80300b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f8027fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f80300d3400 0x7f80300b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f8034d5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f80300b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:36678/0 (socket says 172.21.15.27:36678) 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f8034d5c640 1 -- 172.21.15.27:0/1590132055 learned_addr learned my addr 172.21.15.27:0/1590132055 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.403+0000 7f802ffff640 1 --1- 172.21.15.27:0/1590132055 >> v1:172.21.15.69:6789/0 conn(0x7f8030071540 0x7f8030071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:55188/0 (socket says 172.21.15.27:55188) 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.404+0000 7f802f7fe640 1 -- 172.21.15.27:0/1590132055 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429454306 0 0) 0x7f80300ce610 con 0x7f8030071540 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.404+0000 7f802f7fe640 1 -- 172.21.15.27:0/1590132055 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801c003520 con 0x7f8030071540 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.404+0000 7f8034d5c640 1 -- 172.21.15.27:0/1590132055 >> v2:172.21.15.69:3300/0 conn(0x7f80300d3400 msgr2=0x7f80300b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.404+0000 7f8034d5c640 1 --2- 172.21.15.27:0/1590132055 >> v2:172.21.15.69:3300/0 conn(0x7f80300d3400 0x7f80300b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.404+0000 7f8034d5c640 1 -- 172.21.15.27:0/1590132055 >> v1:172.21.15.69:6789/0 conn(0x7f8030071540 legacy=0x7f8030071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.404+0000 7f8034d5c640 1 -- 172.21.15.27:0/1590132055 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80300d2920 con 0x7f80300b2b50 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.404+0000 7f8027fff640 1 --2- 172.21.15.27:0/1590132055 >> v2:172.21.15.69:3300/0 conn(0x7f80300d3400 0x7f80300b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f8034d5c640 1 --2- 172.21.15.27:0/1590132055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f80300b6e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8028009e50 tx=0x7f802802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f88d16690b440670 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f802f7fe640 1 -- 172.21.15.27:0/1590132055 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802802cb00 con 0x7f80300b2b50 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f802f7fe640 1 -- 172.21.15.27:0/1590132055 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802802cca0 con 0x7f80300b2b50 2024-03-06T07:26:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f802f7fe640 1 -- 172.21.15.27:0/1590132055 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028034610 con 0x7f80300b2b50 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f80367bf640 1 -- 172.21.15.27:0/1590132055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 msgr2=0x7f80300b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f80367bf640 1 --2- 172.21.15.27:0/1590132055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f80300b6e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8028009e50 tx=0x7f802802bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f80367bf640 1 -- 172.21.15.27:0/1590132055 shutdown_connections 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f80367bf640 1 --2- 172.21.15.27:0/1590132055 >> v2:172.21.15.69:3300/0 conn(0x7f80300d3400 0x7f80300b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f80367bf640 1 --2- 172.21.15.27:0/1590132055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f80300b6e20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.405+0000 7f80367bf640 1 -- 172.21.15.27:0/1590132055 >> 172.21.15.27:0/1590132055 conn(0x7f80300d3ca0 msgr2=0x7f80300d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.406+0000 7f80367bf640 1 -- 172.21.15.27:0/1590132055 shutdown_connections 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.406+0000 7f80367bf640 1 -- 172.21.15.27:0/1590132055 wait complete. 2024-03-06T07:26:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.407+0000 7f80367bf640 1 Processor -- start 2024-03-06T07:26:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.407+0000 7f80367bf640 1 -- start start 2024-03-06T07:26:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f80367bf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8030071540 0x7f803014f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f80367bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f803014fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f80367bf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80300d3400 0x7f80300abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f80367bf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803015e370 con 0x7f80300d3400 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f80367bf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80300ce8f0 con 0x7f8030071540 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f80367bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80300b3be0 con 0x7f80300b2b50 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f8034d5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80300d3400 0x7f80300abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f8034d5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80300d3400 0x7f80300abe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47440/0 (socket says 172.21.15.27:47440) 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f8034d5c640 1 -- 172.21.15.27:0/895168830 learned_addr learned my addr 172.21.15.27:0/895168830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f802ffff640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8030071540 0x7f803014f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.408+0000 7f8027fff640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f803014fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f8034d5c640 1 -- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 msgr2=0x7f803014fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f8034d5c640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f803014fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f8034d5c640 1 -- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8030071540 msgr2=0x7f803014f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f8034d5c640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8030071540 0x7f803014f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f8034d5c640 1 -- 172.21.15.27:0/895168830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803014f630 con 0x7f80300d3400 2024-03-06T07:26:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f8034d5c640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80300d3400 0x7f80300abe00 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f8028036040 tx=0x7f8028004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f802d7fa640 1 -- 172.21.15.27:0/895168830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028043020 con 0x7f80300d3400 2024-03-06T07:26:24.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80300b2fd0 con 0x7f80300d3400 2024-03-06T07:26:24.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f802d7fa640 1 -- 172.21.15.27:0/895168830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802802ce50 con 0x7f80300d3400 2024-03-06T07:26:24.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f802d7fa640 1 -- 172.21.15.27:0/895168830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028009510 con 0x7f80300d3400 2024-03-06T07:26:24.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.409+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80300b3590 con 0x7f80300d3400 2024-03-06T07:26:24.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.410+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80300ce610 con 0x7f80300d3400 2024-03-06T07:26:24.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.411+0000 7f802d7fa640 1 -- 172.21.15.27:0/895168830 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8028005d10 con 0x7f80300d3400 2024-03-06T07:26:24.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.411+0000 7f802d7fa640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f801006b4d0 0x7f801006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.411+0000 7f802d7fa640 1 -- 172.21.15.27:0/895168830 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80280ac880 con 0x7f80300d3400 2024-03-06T07:26:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.411+0000 7f802ffff640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f801006b4d0 0x7f801006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.412+0000 7f802ffff640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f801006b4d0 0x7f801006d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8020000f60 tx=0x7f8020007cf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.415+0000 7f802d7fa640 1 -- 172.21.15.27:0/895168830 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80300ce610 con 0x7f80300d3400 2024-03-06T07:26:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.592+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8030068700 con 0x7f801006b4d0 2024-03-06T07:26:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.598+0000 7f802d7fa640 1 -- 172.21.15.27:0/895168830 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246754 (secure 0 0 0) 0x7f8030068700 con 0x7f801006b4d0 2024-03-06T07:26:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:24.603 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.601+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f801006b4d0 msgr2=0x7f801006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.601+0000 7f80367bf640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f801006b4d0 0x7f801006d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8020000f60 tx=0x7f8020007cf0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.601+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80300d3400 msgr2=0x7f80300abe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.601+0000 7f80367bf640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80300d3400 0x7f80300abe00 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f8028036040 tx=0x7f8028004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.602+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 shutdown_connections 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.602+0000 7f80367bf640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f801006b4d0 0x7f801006d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.602+0000 7f80367bf640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80300d3400 0x7f80300abe00 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.602+0000 7f80367bf640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80300b2b50 0x7f803014fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.602+0000 7f80367bf640 1 --2- 172.21.15.27:0/895168830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8030071540 0x7f803014f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.602+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 >> 172.21.15.27:0/895168830 conn(0x7f80300d3ca0 msgr2=0x7f803015e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.602+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 shutdown_connections 2024-03-06T07:26:24.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.602+0000 7f80367bf640 1 -- 172.21.15.27:0/895168830 wait complete. 2024-03-06T07:26:24.614 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-03-06T07:26:24.038057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257056,"kb_used_data":7600,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717664,"statfs":{"total":773094113280,"available":772830887936,"internally_reserved":0,"allocated":7782400,"data_stored":3918114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.825797"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227981+0000","last_change":"2024-03-06T07:26:19.227981+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227981+0000","last_undegraded":"2024-03-06T07:26:19.227981+0000","last_fullsized":"2024-03-06T07:26:19.227981+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228347+0000","last_change":"2024-03-06T07:26:19.228347+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228347+0000","last_undegraded":"2024-03-06T07:26:19.228347+0000","last_fullsized":"2024-03-06T07:26:19.228347+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228671+0000","last_change":"2024-03-06T07:26:19.228671+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228671+0000","last_undegraded":"2024-03-06T07:26:19.228671+0000","last_fullsized":"2024-03-06T07:26:19.228671+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224081+0000","last_change":"2024-03-06T07:26:19.224081+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224081+0000","last_undegraded":"2024-03-06T07:26:19.224081+0000","last_fullsized":"2024-03-06T07:26:19.224081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227451+0000","last_change":"2024-03-06T07:26:19.227451+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227451+0000","last_undegraded":"2024-03-06T07:26:19.227451+0000","last_fullsized":"2024-03-06T07:26:19.227451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228991+0000","last_change":"2024-03-06T07:26:19.228991+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228991+0000","last_undegraded":"2024-03-06T07:26:19.228991+0000","last_fullsized":"2024-03-06T07:26:19.228991+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231923+0000","last_change":"2024-03-06T07:26:19.231923+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231923+0000","last_undegraded":"2024-03-06T07:26:19.231923+0000","last_fullsized":"2024-03-06T07:26:19.231923+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228094+0000","last_change":"2024-03-06T07:26:19.228094+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228094+0000","last_undegraded":"2024-03-06T07:26:19.228094+0000","last_fullsized":"2024-03-06T07:26:19.228094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226958+0000","last_change":"2024-03-06T07:26:19.226958+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226958+0000","last_undegraded":"2024-03-06T07:26:19.226958+0000","last_fullsized":"2024-03-06T07:26:19.226958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224802+0000","last_change":"2024-03-06T07:26:19.224802+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224802+0000","last_undegraded":"2024-03-06T07:26:19.224802+0000","last_fullsized":"2024-03-06T07:26:19.224802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231680+0000","last_change":"2024-03-06T07:26:19.231680+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231680+0000","last_undegraded":"2024-03-06T07:26:19.231680+0000","last_fullsized":"2024-03-06T07:26:19.231680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223377+0000","last_change":"2024-03-06T07:26:19.223377+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223377+0000","last_undegraded":"2024-03-06T07:26:19.223377+0000","last_fullsized":"2024-03-06T07:26:19.223377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221736+0000","last_change":"2024-03-06T07:26:19.221736+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221736+0000","last_undegraded":"2024-03-06T07:26:19.221736+0000","last_fullsized":"2024-03-06T07:26:19.221736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226495+0000","last_change":"2024-03-06T07:26:19.226495+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226495+0000","last_undegraded":"2024-03-06T07:26:19.226495+0000","last_fullsized":"2024-03-06T07:26:19.226495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222226+0000","last_change":"2024-03-06T07:26:19.222226+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222226+0000","last_undegraded":"2024-03-06T07:26:19.222226+0000","last_fullsized":"2024-03-06T07:26:19.222226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224253+0000","last_change":"2024-03-06T07:26:19.224253+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224253+0000","last_undegraded":"2024-03-06T07:26:19.224253+0000","last_fullsized":"2024-03-06T07:26:19.224253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225964+0000","last_change":"2024-03-06T07:26:19.225964+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225964+0000","last_undegraded":"2024-03-06T07:26:19.225964+0000","last_fullsized":"2024-03-06T07:26:19.225964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221558+0000","last_change":"2024-03-06T07:26:19.221558+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221558+0000","last_undegraded":"2024-03-06T07:26:19.221558+0000","last_fullsized":"2024-03-06T07:26:19.221558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228445+0000","last_change":"2024-03-06T07:26:19.228445+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228445+0000","last_undegraded":"2024-03-06T07:26:19.228445+0000","last_fullsized":"2024-03-06T07:26:19.228445+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223388+0000","last_change":"2024-03-06T07:26:19.223388+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223388+0000","last_undegraded":"2024-03-06T07:26:19.223388+0000","last_fullsized":"2024-03-06T07:26:19.223388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223534+0000","last_change":"2024-03-06T07:26:19.223534+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223534+0000","last_undegraded":"2024-03-06T07:26:19.223534+0000","last_fullsized":"2024-03-06T07:26:19.223534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223910+0000","last_change":"2024-03-06T07:26:19.223910+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223910+0000","last_undegraded":"2024-03-06T07:26:19.223910+0000","last_fullsized":"2024-03-06T07:26:19.223910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222903+0000","last_change":"2024-03-06T07:26:19.222903+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222903+0000","last_undegraded":"2024-03-06T07:26:19.222903+0000","last_fullsized":"2024-03-06T07:26:19.222903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224000+0000","last_change":"2024-03-06T07:26:19.224000+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224000+0000","last_undegraded":"2024-03-06T07:26:19.224000+0000","last_fullsized":"2024-03-06T07:26:19.224000+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222350+0000","last_change":"2024-03-06T07:26:19.222350+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222350+0000","last_undegraded":"2024-03-06T07:26:19.222350+0000","last_fullsized":"2024-03-06T07:26:19.222350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229681+0000","last_change":"2024-03-06T07:26:19.229681+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229681+0000","last_undegraded":"2024-03-06T07:26:19.229681+0000","last_fullsized":"2024-03-06T07:26:19.229681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228249+0000","last_change":"2024-03-06T07:26:19.228249+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228249+0000","last_undegraded":"2024-03-06T07:26:19.228249+0000","last_fullsized":"2024-03-06T07:26:19.228249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224326+0000","last_change":"2024-03-06T07:26:19.224326+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224326+0000","last_undegraded":"2024-03-06T07:26:19.224326+0000","last_fullsized":"2024-03-06T07:26:19.224326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228898+0000","last_change":"2024-03-06T07:26:19.228898+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228898+0000","last_undegraded":"2024-03-06T07:26:19.228898+0000","last_fullsized":"2024-03-06T07:26:19.228898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231473+0000","last_change":"2024-03-06T07:26:19.231473+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231473+0000","last_undegraded":"2024-03-06T07:26:19.231473+0000","last_fullsized":"2024-03-06T07:26:19.231473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227762+0000","last_change":"2024-03-06T07:26:19.227762+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227762+0000","last_undegraded":"2024-03-06T07:26:19.227762+0000","last_fullsized":"2024-03-06T07:26:19.227762+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227670+0000","last_change":"2024-03-06T07:26:19.227670+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227670+0000","last_undegraded":"2024-03-06T07:26:19.227670+0000","last_fullsized":"2024-03-06T07:26:19.227670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225593+0000","last_change":"2024-03-06T07:26:19.225593+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225593+0000","last_undegraded":"2024-03-06T07:26:19.225593+0000","last_fullsized":"2024-03-06T07:26:19.225593+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239388+0000","last_change":"2024-03-06T07:26:19.235249+0000","last_active":"2024-03-06T07:26:19.239388+0000","last_peered":"2024-03-06T07:26:19.239388+0000","last_clean":"2024-03-06T07:26:19.239388+0000","last_became_active":"2024-03-06T07:26:19.234904+0000","last_became_peered":"2024-03-06T07:26:19.234904+0000","last_unstale":"2024-03-06T07:26:19.239388+0000","last_undegraded":"2024-03-06T07:26:19.239388+0000","last_fullsized":"2024-03-06T07:26:19.239388+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:23.463441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225476+0000","last_change":"2024-03-06T07:26:19.225476+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225476+0000","last_undegraded":"2024-03-06T07:26:19.225476+0000","last_fullsized":"2024-03-06T07:26:19.225476+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.233898+0000","last_change":"2024-03-06T07:26:19.225962+0000","last_active":"2024-03-06T07:26:19.233898+0000","last_peered":"2024-03-06T07:26:19.233898+0000","last_clean":"2024-03-06T07:26:19.233898+0000","last_became_active":"2024-03-06T07:26:19.225707+0000","last_became_peered":"2024-03-06T07:26:19.225707+0000","last_unstale":"2024-03-06T07:26:19.233898+0000","last_undegraded":"2024-03-06T07:26:19.233898+0000","last_fullsized":"2024-03-06T07:26:19.233898+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:50.887101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228771+0000","last_change":"2024-03-06T07:26:19.228771+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228771+0000","last_undegraded":"2024-03-06T07:26:19.228771+0000","last_fullsized":"2024-03-06T07:26:19.228771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235050+0000","last_change":"2024-03-06T07:26:19.232800+0000","last_active":"2024-03-06T07:26:19.235050+0000","last_peered":"2024-03-06T07:26:19.235050+0000","last_clean":"2024-03-06T07:26:19.235050+0000","last_became_active":"2024-03-06T07:26:19.232541+0000","last_became_peered":"2024-03-06T07:26:19.232541+0000","last_unstale":"2024-03-06T07:26:19.235050+0000","last_undegraded":"2024-03-06T07:26:19.235050+0000","last_fullsized":"2024-03-06T07:26:19.235050+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:49.533898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.230992+0000","last_change":"2024-03-06T07:26:19.230992+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.230992+0000","last_undegraded":"2024-03-06T07:26:19.230992+0000","last_fullsized":"2024-03-06T07:26:19.230992+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239827+0000","last_change":"2024-03-06T07:26:19.237329+0000","last_active":"2024-03-06T07:26:19.239827+0000","last_peered":"2024-03-06T07:26:19.239827+0000","last_clean":"2024-03-06T07:26:19.239827+0000","last_became_active":"2024-03-06T07:26:19.237096+0000","last_became_peered":"2024-03-06T07:26:19.237096+0000","last_unstale":"2024-03-06T07:26:19.239827+0000","last_undegraded":"2024-03-06T07:26:19.239827+0000","last_fullsized":"2024-03-06T07:26:19.239827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:54.603601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231064+0000","last_change":"2024-03-06T07:26:19.231064+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231064+0000","last_undegraded":"2024-03-06T07:26:19.231064+0000","last_fullsized":"2024-03-06T07:26:19.231064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.237618+0000","last_change":"2024-03-06T07:26:19.236188+0000","last_active":"2024-03-06T07:26:19.237618+0000","last_peered":"2024-03-06T07:26:19.237618+0000","last_clean":"2024-03-06T07:26:19.237618+0000","last_became_active":"2024-03-06T07:26:19.235921+0000","last_became_peered":"2024-03-06T07:26:19.235921+0000","last_unstale":"2024-03-06T07:26:19.237618+0000","last_undegraded":"2024-03-06T07:26:19.237618+0000","last_fullsized":"2024-03-06T07:26:19.237618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:13.224175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222902+0000","last_change":"2024-03-06T07:26:19.222902+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222902+0000","last_undegraded":"2024-03-06T07:26:19.222902+0000","last_fullsized":"2024-03-06T07:26:19.222902+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242242+0000","last_change":"2024-03-06T07:26:19.237984+0000","last_active":"2024-03-06T07:26:19.242242+0000","last_peered":"2024-03-06T07:26:19.242242+0000","last_clean":"2024-03-06T07:26:19.242242+0000","last_became_active":"2024-03-06T07:26:19.237697+0000","last_became_peered":"2024-03-06T07:26:19.237697+0000","last_unstale":"2024-03-06T07:26:19.242242+0000","last_undegraded":"2024-03-06T07:26:19.242242+0000","last_fullsized":"2024-03-06T07:26:19.242242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:21.157010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225590+0000","last_change":"2024-03-06T07:26:19.225590+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225590+0000","last_undegraded":"2024-03-06T07:26:19.225590+0000","last_fullsized":"2024-03-06T07:26:19.225590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.232619+0000","last_change":"2024-03-06T07:26:19.228888+0000","last_active":"2024-03-06T07:26:19.232619+0000","last_peered":"2024-03-06T07:26:19.232619+0000","last_clean":"2024-03-06T07:26:19.232619+0000","last_became_active":"2024-03-06T07:26:19.228602+0000","last_became_peered":"2024-03-06T07:26:19.228602+0000","last_unstale":"2024-03-06T07:26:19.232619+0000","last_undegraded":"2024-03-06T07:26:19.232619+0000","last_fullsized":"2024-03-06T07:26:19.232619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:55.021248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221713+0000","last_change":"2024-03-06T07:26:19.221713+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221713+0000","last_undegraded":"2024-03-06T07:26:19.221713+0000","last_fullsized":"2024-03-06T07:26:19.221713+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.234079+0000","last_change":"2024-03-06T07:26:19.231771+0000","last_active":"2024-03-06T07:26:19.234079+0000","last_peered":"2024-03-06T07:26:19.234079+0000","last_clean":"2024-03-06T07:26:19.234079+0000","last_became_active":"2024-03-06T07:26:19.231485+0000","last_became_peered":"2024-03-06T07:26:19.231485+0000","last_unstale":"2024-03-06T07:26:19.234079+0000","last_undegraded":"2024-03-06T07:26:19.234079+0000","last_fullsized":"2024-03-06T07:26:19.234079+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:30.451695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229259+0000","last_change":"2024-03-06T07:26:19.229259+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229259+0000","last_undegraded":"2024-03-06T07:26:19.229259+0000","last_fullsized":"2024-03-06T07:26:19.229259+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.230928+0000","last_change":"2024-03-06T07:26:19.229136+0000","last_active":"2024-03-06T07:26:19.230928+0000","last_peered":"2024-03-06T07:26:19.230928+0000","last_clean":"2024-03-06T07:26:19.230928+0000","last_became_active":"2024-03-06T07:26:19.228661+0000","last_became_peered":"2024-03-06T07:26:19.228661+0000","last_unstale":"2024-03-06T07:26:19.230928+0000","last_undegraded":"2024-03-06T07:26:19.230928+0000","last_fullsized":"2024-03-06T07:26:19.230928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:56.368954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225377+0000","last_change":"2024-03-06T07:26:19.225377+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225377+0000","last_undegraded":"2024-03-06T07:26:19.225377+0000","last_fullsized":"2024-03-06T07:26:19.225377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239800+0000","last_change":"2024-03-06T07:26:19.236797+0000","last_active":"2024-03-06T07:26:19.239800+0000","last_peered":"2024-03-06T07:26:19.239800+0000","last_clean":"2024-03-06T07:26:19.239800+0000","last_became_active":"2024-03-06T07:26:19.236118+0000","last_became_peered":"2024-03-06T07:26:19.236118+0000","last_unstale":"2024-03-06T07:26:19.239800+0000","last_undegraded":"2024-03-06T07:26:19.239800+0000","last_fullsized":"2024-03-06T07:26:19.239800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:04.139082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228026+0000","last_change":"2024-03-06T07:26:19.228026+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228026+0000","last_undegraded":"2024-03-06T07:26:19.228026+0000","last_fullsized":"2024-03-06T07:26:19.228026+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.236116+0000","last_change":"2024-03-06T07:26:19.235158+0000","last_active":"2024-03-06T07:26:19.236116+0000","last_peered":"2024-03-06T07:26:19.236116+0000","last_clean":"2024-03-06T07:26:19.236116+0000","last_became_active":"2024-03-06T07:26:19.234883+0000","last_became_peered":"2024-03-06T07:26:19.234883+0000","last_unstale":"2024-03-06T07:26:19.236116+0000","last_undegraded":"2024-03-06T07:26:19.236116+0000","last_fullsized":"2024-03-06T07:26:19.236116+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:17.059043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225110+0000","last_change":"2024-03-06T07:26:19.225110+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225110+0000","last_undegraded":"2024-03-06T07:26:19.225110+0000","last_fullsized":"2024-03-06T07:26:19.225110+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.238717+0000","last_change":"2024-03-06T07:26:19.236129+0000","last_active":"2024-03-06T07:26:19.238717+0000","last_peered":"2024-03-06T07:26:19.238717+0000","last_clean":"2024-03-06T07:26:19.238717+0000","last_became_active":"2024-03-06T07:26:19.235844+0000","last_became_peered":"2024-03-06T07:26:19.235844+0000","last_unstale":"2024-03-06T07:26:19.238717+0000","last_undegraded":"2024-03-06T07:26:19.238717+0000","last_fullsized":"2024-03-06T07:26:19.238717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:04.139082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223551+0000","last_change":"2024-03-06T07:26:19.223551+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223551+0000","last_undegraded":"2024-03-06T07:26:19.223551+0000","last_fullsized":"2024-03-06T07:26:19.223551+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.228861+0000","last_change":"2024-03-06T07:26:19.226330+0000","last_active":"2024-03-06T07:26:19.228861+0000","last_peered":"2024-03-06T07:26:19.228861+0000","last_clean":"2024-03-06T07:26:19.228861+0000","last_became_active":"2024-03-06T07:26:19.226062+0000","last_became_peered":"2024-03-06T07:26:19.226062+0000","last_unstale":"2024-03-06T07:26:19.228861+0000","last_undegraded":"2024-03-06T07:26:19.228861+0000","last_fullsized":"2024-03-06T07:26:19.228861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:29.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.230238+0000","last_change":"2024-03-06T07:26:19.230238+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.230238+0000","last_undegraded":"2024-03-06T07:26:19.230238+0000","last_fullsized":"2024-03-06T07:26:19.230238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242264+0000","last_change":"2024-03-06T07:26:19.238616+0000","last_active":"2024-03-06T07:26:19.242264+0000","last_peered":"2024-03-06T07:26:19.242264+0000","last_clean":"2024-03-06T07:26:19.242264+0000","last_became_active":"2024-03-06T07:26:19.238337+0000","last_became_peered":"2024-03-06T07:26:19.238337+0000","last_unstale":"2024-03-06T07:26:19.242264+0000","last_undegraded":"2024-03-06T07:26:19.242264+0000","last_fullsized":"2024-03-06T07:26:19.242264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:14.382999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222417+0000","last_change":"2024-03-06T07:26:19.222417+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222417+0000","last_undegraded":"2024-03-06T07:26:19.222417+0000","last_fullsized":"2024-03-06T07:26:19.222417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239417+0000","last_change":"2024-03-06T07:26:19.236852+0000","last_active":"2024-03-06T07:26:19.239417+0000","last_peered":"2024-03-06T07:26:19.239417+0000","last_clean":"2024-03-06T07:26:19.239417+0000","last_became_active":"2024-03-06T07:26:19.236469+0000","last_became_peered":"2024-03-06T07:26:19.236469+0000","last_unstale":"2024-03-06T07:26:19.239417+0000","last_undegraded":"2024-03-06T07:26:19.239417+0000","last_fullsized":"2024-03-06T07:26:19.239417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:20.767924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224910+0000","last_change":"2024-03-06T07:26:19.224910+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224910+0000","last_undegraded":"2024-03-06T07:26:19.224910+0000","last_fullsized":"2024-03-06T07:26:19.224910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235094+0000","last_change":"2024-03-06T07:26:19.231717+0000","last_active":"2024-03-06T07:26:19.235094+0000","last_peered":"2024-03-06T07:26:19.235094+0000","last_clean":"2024-03-06T07:26:19.235094+0000","last_became_active":"2024-03-06T07:26:19.231462+0000","last_became_peered":"2024-03-06T07:26:19.231462+0000","last_unstale":"2024-03-06T07:26:19.235094+0000","last_undegraded":"2024-03-06T07:26:19.235094+0000","last_fullsized":"2024-03-06T07:26:19.235094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.795497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223271+0000","last_change":"2024-03-06T07:26:19.223271+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223271+0000","last_undegraded":"2024-03-06T07:26:19.223271+0000","last_fullsized":"2024-03-06T07:26:19.223271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239822+0000","last_change":"2024-03-06T07:26:19.237309+0000","last_active":"2024-03-06T07:26:19.239822+0000","last_peered":"2024-03-06T07:26:19.239822+0000","last_clean":"2024-03-06T07:26:19.239822+0000","last_became_active":"2024-03-06T07:26:19.237011+0000","last_became_peered":"2024-03-06T07:26:19.237011+0000","last_unstale":"2024-03-06T07:26:19.239822+0000","last_undegraded":"2024-03-06T07:26:19.239822+0000","last_fullsized":"2024-03-06T07:26:19.239822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:58.527722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224977+0000","last_change":"2024-03-06T07:26:19.224977+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224977+0000","last_undegraded":"2024-03-06T07:26:19.224977+0000","last_fullsized":"2024-03-06T07:26:19.224977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235073+0000","last_change":"2024-03-06T07:26:19.233492+0000","last_active":"2024-03-06T07:26:19.235073+0000","last_peered":"2024-03-06T07:26:19.235073+0000","last_clean":"2024-03-06T07:26:19.235073+0000","last_became_active":"2024-03-06T07:26:19.233199+0000","last_became_peered":"2024-03-06T07:26:19.233199+0000","last_unstale":"2024-03-06T07:26:19.235073+0000","last_undegraded":"2024-03-06T07:26:19.235073+0000","last_fullsized":"2024-03-06T07:26:19.235073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:22.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222478+0000","last_change":"2024-03-06T07:26:19.222478+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222478+0000","last_undegraded":"2024-03-06T07:26:19.222478+0000","last_fullsized":"2024-03-06T07:26:19.222478+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239765+0000","last_change":"2024-03-06T07:26:19.236432+0000","last_active":"2024-03-06T07:26:19.239765+0000","last_peered":"2024-03-06T07:26:19.239765+0000","last_clean":"2024-03-06T07:26:19.239765+0000","last_became_active":"2024-03-06T07:26:19.235975+0000","last_became_peered":"2024-03-06T07:26:19.235975+0000","last_unstale":"2024-03-06T07:26:19.239765+0000","last_undegraded":"2024-03-06T07:26:19.239765+0000","last_fullsized":"2024-03-06T07:26:19.239765+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:55.021248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224496+0000","last_change":"2024-03-06T07:26:19.224496+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224496+0000","last_undegraded":"2024-03-06T07:26:19.224496+0000","last_fullsized":"2024-03-06T07:26:19.224496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.234690+0000","last_change":"2024-03-06T07:26:19.233564+0000","last_active":"2024-03-06T07:26:19.234690+0000","last_peered":"2024-03-06T07:26:19.234690+0000","last_clean":"2024-03-06T07:26:19.234690+0000","last_became_active":"2024-03-06T07:26:19.233235+0000","last_became_peered":"2024-03-06T07:26:19.233235+0000","last_unstale":"2024-03-06T07:26:19.234690+0000","last_undegraded":"2024-03-06T07:26:19.234690+0000","last_fullsized":"2024-03-06T07:26:19.234690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:44.732093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228500+0000","last_change":"2024-03-06T07:26:19.228500+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228500+0000","last_undegraded":"2024-03-06T07:26:19.228500+0000","last_fullsized":"2024-03-06T07:26:19.228500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239745+0000","last_change":"2024-03-06T07:26:19.236405+0000","last_active":"2024-03-06T07:26:19.239745+0000","last_peered":"2024-03-06T07:26:19.239745+0000","last_clean":"2024-03-06T07:26:19.239745+0000","last_became_active":"2024-03-06T07:26:19.235876+0000","last_became_peered":"2024-03-06T07:26:19.235876+0000","last_unstale":"2024-03-06T07:26:19.239745+0000","last_undegraded":"2024-03-06T07:26:19.239745+0000","last_fullsized":"2024-03-06T07:26:19.239745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:29.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225680+0000","last_change":"2024-03-06T07:26:19.225680+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225680+0000","last_undegraded":"2024-03-06T07:26:19.225680+0000","last_fullsized":"2024-03-06T07:26:19.225680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242212+0000","last_change":"2024-03-06T07:26:19.238646+0000","last_active":"2024-03-06T07:26:19.242212+0000","last_peered":"2024-03-06T07:26:19.242212+0000","last_clean":"2024-03-06T07:26:19.242212+0000","last_became_active":"2024-03-06T07:26:19.238376+0000","last_became_peered":"2024-03-06T07:26:19.238376+0000","last_unstale":"2024-03-06T07:26:19.242212+0000","last_undegraded":"2024-03-06T07:26:19.242212+0000","last_fullsized":"2024-03-06T07:26:19.242212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:33.080991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225633+0000","last_change":"2024-03-06T07:26:19.225633+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225633+0000","last_undegraded":"2024-03-06T07:26:19.225633+0000","last_fullsized":"2024-03-06T07:26:19.225633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.233488+0000","last_change":"2024-03-06T07:26:19.227589+0000","last_active":"2024-03-06T07:26:19.233488+0000","last_peered":"2024-03-06T07:26:19.233488+0000","last_clean":"2024-03-06T07:26:19.233488+0000","last_became_active":"2024-03-06T07:26:19.227308+0000","last_became_peered":"2024-03-06T07:26:19.227308+0000","last_unstale":"2024-03-06T07:26:19.233488+0000","last_undegraded":"2024-03-06T07:26:19.233488+0000","last_fullsized":"2024-03-06T07:26:19.233488+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:19.014952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222420+0000","last_change":"2024-03-06T07:26:19.222420+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222420+0000","last_undegraded":"2024-03-06T07:26:19.222420+0000","last_fullsized":"2024-03-06T07:26:19.222420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239645+0000","last_change":"2024-03-06T07:26:19.234268+0000","last_active":"2024-03-06T07:26:19.239645+0000","last_peered":"2024-03-06T07:26:19.239645+0000","last_clean":"2024-03-06T07:26:19.239645+0000","last_became_active":"2024-03-06T07:26:19.234000+0000","last_became_peered":"2024-03-06T07:26:19.234000+0000","last_unstale":"2024-03-06T07:26:19.239645+0000","last_undegraded":"2024-03-06T07:26:19.239645+0000","last_fullsized":"2024-03-06T07:26:19.239645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:50.102543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229363+0000","last_change":"2024-03-06T07:26:19.229363+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229363+0000","last_undegraded":"2024-03-06T07:26:19.229363+0000","last_fullsized":"2024-03-06T07:26:19.229363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242098+0000","last_change":"2024-03-06T07:26:19.237743+0000","last_active":"2024-03-06T07:26:19.242098+0000","last_peered":"2024-03-06T07:26:19.242098+0000","last_clean":"2024-03-06T07:26:19.242098+0000","last_became_active":"2024-03-06T07:26:19.237394+0000","last_became_peered":"2024-03-06T07:26:19.237394+0000","last_unstale":"2024-03-06T07:26:19.242098+0000","last_undegraded":"2024-03-06T07:26:19.242098+0000","last_fullsized":"2024-03-06T07:26:19.242098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:35.749995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225400+0000","last_change":"2024-03-06T07:26:19.225400+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225400+0000","last_undegraded":"2024-03-06T07:26:19.225400+0000","last_fullsized":"2024-03-06T07:26:19.225400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.231970+0000","last_change":"2024-03-06T07:26:19.230603+0000","last_active":"2024-03-06T07:26:19.231970+0000","last_peered":"2024-03-06T07:26:19.231970+0000","last_clean":"2024-03-06T07:26:19.231970+0000","last_became_active":"2024-03-06T07:26:19.230313+0000","last_became_peered":"2024-03-06T07:26:19.230313+0000","last_unstale":"2024-03-06T07:26:19.231970+0000","last_undegraded":"2024-03-06T07:26:19.231970+0000","last_fullsized":"2024-03-06T07:26:19.231970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:31.518943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226436+0000","last_change":"2024-03-06T07:26:19.226436+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226436+0000","last_undegraded":"2024-03-06T07:26:19.226436+0000","last_fullsized":"2024-03-06T07:26:19.226436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.236928+0000","last_change":"2024-03-06T07:26:19.234353+0000","last_active":"2024-03-06T07:26:19.236928+0000","last_peered":"2024-03-06T07:26:19.236928+0000","last_clean":"2024-03-06T07:26:19.236928+0000","last_became_active":"2024-03-06T07:26:19.234077+0000","last_became_peered":"2024-03-06T07:26:19.234077+0000","last_unstale":"2024-03-06T07:26:19.236928+0000","last_undegraded":"2024-03-06T07:26:19.236928+0000","last_fullsized":"2024-03-06T07:26:19.236928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:37.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221555+0000","last_change":"2024-03-06T07:26:19.221555+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221555+0000","last_undegraded":"2024-03-06T07:26:19.221555+0000","last_fullsized":"2024-03-06T07:26:19.221555+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.238279+0000","last_change":"2024-03-06T07:26:19.237287+0000","last_active":"2024-03-06T07:26:19.238279+0000","last_peered":"2024-03-06T07:26:19.238279+0000","last_clean":"2024-03-06T07:26:19.238279+0000","last_became_active":"2024-03-06T07:26:19.237017+0000","last_became_peered":"2024-03-06T07:26:19.237017+0000","last_unstale":"2024-03-06T07:26:19.238279+0000","last_undegraded":"2024-03-06T07:26:19.238279+0000","last_fullsized":"2024-03-06T07:26:19.238279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:46.817231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227191+0000","last_change":"2024-03-06T07:26:19.227191+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227191+0000","last_undegraded":"2024-03-06T07:26:19.227191+0000","last_fullsized":"2024-03-06T07:26:19.227191+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.232804+0000","last_change":"2024-03-06T07:26:19.229645+0000","last_active":"2024-03-06T07:26:19.232804+0000","last_peered":"2024-03-06T07:26:19.232804+0000","last_clean":"2024-03-06T07:26:19.232804+0000","last_became_active":"2024-03-06T07:26:19.229309+0000","last_became_peered":"2024-03-06T07:26:19.229309+0000","last_unstale":"2024-03-06T07:26:19.232804+0000","last_undegraded":"2024-03-06T07:26:19.232804+0000","last_fullsized":"2024-03-06T07:26:19.232804+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.342246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222416+0000","last_change":"2024-03-06T07:26:19.222416+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222416+0000","last_undegraded":"2024-03-06T07:26:19.222416+0000","last_fullsized":"2024-03-06T07:26:19.222416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235092+0000","last_change":"2024-03-06T07:26:19.231241+0000","last_active":"2024-03-06T07:26:19.235092+0000","last_peered":"2024-03-06T07:26:19.235092+0000","last_clean":"2024-03-06T07:26:19.235092+0000","last_became_active":"2024-03-06T07:26:19.230977+0000","last_became_peered":"2024-03-06T07:26:19.230977+0000","last_unstale":"2024-03-06T07:26:19.235092+0000","last_undegraded":"2024-03-06T07:26:19.235092+0000","last_fullsized":"2024-03-06T07:26:19.235092+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:36.204568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227602+0000","last_change":"2024-03-06T07:26:19.227602+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227602+0000","last_undegraded":"2024-03-06T07:26:19.227602+0000","last_fullsized":"2024-03-06T07:26:19.227602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242260+0000","last_change":"2024-03-06T07:26:19.238002+0000","last_active":"2024-03-06T07:26:19.242260+0000","last_peered":"2024-03-06T07:26:19.242260+0000","last_clean":"2024-03-06T07:26:19.242260+0000","last_became_active":"2024-03-06T07:26:19.237710+0000","last_became_peered":"2024-03-06T07:26:19.237710+0000","last_unstale":"2024-03-06T07:26:19.242260+0000","last_undegraded":"2024-03-06T07:26:19.242260+0000","last_fullsized":"2024-03-06T07:26:19.242260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:25.030580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226570+0000","last_change":"2024-03-06T07:26:19.226570+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226570+0000","last_undegraded":"2024-03-06T07:26:19.226570+0000","last_fullsized":"2024-03-06T07:26:19.226570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239767+0000","last_change":"2024-03-06T07:26:19.236691+0000","last_active":"2024-03-06T07:26:19.239767+0000","last_peered":"2024-03-06T07:26:19.239767+0000","last_clean":"2024-03-06T07:26:19.239767+0000","last_became_active":"2024-03-06T07:26:19.236432+0000","last_became_peered":"2024-03-06T07:26:19.236432+0000","last_unstale":"2024-03-06T07:26:19.239767+0000","last_undegraded":"2024-03-06T07:26:19.239767+0000","last_fullsized":"2024-03-06T07:26:19.239767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.342246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":261,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.224547+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:26:19.224547+0000","last_peered":"2024-03-06T07:26:19.224547+0000","last_clean":"2024-03-06T07:26:19.224547+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:26:19.224547+0000","last_undegraded":"2024-03-06T07:26:19.224547+0000","last_fullsized":"2024-03-06T07:26:19.224547+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":815104,"data_stored":337466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":337466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:24.618 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:24.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.759+0000 7efcea16c640 1 Processor -- start 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.759+0000 7efcea16c640 1 -- start start 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.760+0000 7efcea16c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efce40d3400 0x7efce40b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.760+0000 7efcea16c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efce40b2b50 0x7efce40b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.760+0000 7efcea16c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce40ce610 con 0x7efce4071540 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.760+0000 7efcea16c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce40d2f70 con 0x7efce40b2b50 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.760+0000 7efcea16c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce40d4be0 con 0x7efce40d3400 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.760+0000 7efce3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efce40b2b50 0x7efce40b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.760+0000 7efce3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efce40b2b50 0x7efce40b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47450/0 (socket says 172.21.15.27:47450) 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.760+0000 7efce3fff640 1 -- 172.21.15.27:0/492274589 learned_addr learned my addr 172.21.15.27:0/492274589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.761+0000 7efce2ffd640 1 --2- 172.21.15.27:0/492274589 >> v2:172.21.15.69:3300/0 conn(0x7efce40d3400 0x7efce40b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.761+0000 7efce3fff640 1 -- 172.21.15.27:0/492274589 >> v2:172.21.15.69:3300/0 conn(0x7efce40d3400 msgr2=0x7efce40b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.761+0000 7efce3fff640 1 --2- 172.21.15.27:0/492274589 >> v2:172.21.15.69:3300/0 conn(0x7efce40d3400 0x7efce40b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.761+0000 7efce3fff640 1 -- 172.21.15.27:0/492274589 >> v1:172.21.15.69:6789/0 conn(0x7efce4071540 legacy=0x7efce4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.761+0000 7efce3fff640 1 -- 172.21.15.27:0/492274589 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40d2920 con 0x7efce40b2b50 2024-03-06T07:26:24.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.761+0000 7efce2ffd640 1 --2- 172.21.15.27:0/492274589 >> v2:172.21.15.69:3300/0 conn(0x7efce40d3400 0x7efce40b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efce3fff640 1 --2- 172.21.15.27:0/492274589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40b2b50 0x7efce40b6e20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7efce40b6860 tx=0x7efcdc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa12366d9b5c91f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efce27fc640 1 -- 172.21.15.27:0/492274589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc02cb20 con 0x7efce40b2b50 2024-03-06T07:26:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efce27fc640 1 -- 172.21.15.27:0/492274589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc02ccc0 con 0x7efce40b2b50 2024-03-06T07:26:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efce27fc640 1 -- 172.21.15.27:0/492274589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc034610 con 0x7efce40b2b50 2024-03-06T07:26:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efcea16c640 1 -- 172.21.15.27:0/492274589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40b2b50 msgr2=0x7efce40b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efcea16c640 1 --2- 172.21.15.27:0/492274589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40b2b50 0x7efce40b6e20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7efce40b6860 tx=0x7efcdc02bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efcea16c640 1 -- 172.21.15.27:0/492274589 shutdown_connections 2024-03-06T07:26:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efcea16c640 1 --2- 172.21.15.27:0/492274589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40b2b50 0x7efce40b6e20 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efcea16c640 1 --2- 172.21.15.27:0/492274589 >> v2:172.21.15.69:3300/0 conn(0x7efce40d3400 0x7efce40b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efcea16c640 1 -- 172.21.15.27:0/492274589 >> 172.21.15.27:0/492274589 conn(0x7efce40d3ca0 msgr2=0x7efce40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efcea16c640 1 -- 172.21.15.27:0/492274589 shutdown_connections 2024-03-06T07:26:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.762+0000 7efcea16c640 1 -- 172.21.15.27:0/492274589 wait complete. 2024-03-06T07:26:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.763+0000 7efcea16c640 1 Processor -- start 2024-03-06T07:26:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.764+0000 7efcea16c640 1 -- start start 2024-03-06T07:26:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.764+0000 7efcea16c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce4071540 0x7efce41540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.764+0000 7efcea16c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce40b2b50 0x7efce4151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.764+0000 7efcea16c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40d3400 0x7efce415bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.764+0000 7efcea16c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce415e370 con 0x7efce40d3400 2024-03-06T07:26:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.764+0000 7efcea16c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce40ce8f0 con 0x7efce40b2b50 2024-03-06T07:26:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.764+0000 7efcea16c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce41545e0 con 0x7efce4071540 2024-03-06T07:26:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40d3400 0x7efce415bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40d3400 0x7efce415bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47456/0 (socket says 172.21.15.27:47456) 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce3fff640 1 -- 172.21.15.27:0/1972596702 learned_addr learned my addr 172.21.15.27:0/1972596702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce2ffd640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce40b2b50 0x7efce4151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce37fe640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce4071540 0x7efce41540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce3fff640 1 -- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce4071540 msgr2=0x7efce41540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce3fff640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce4071540 0x7efce41540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce3fff640 1 -- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce40b2b50 msgr2=0x7efce4151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce3fff640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce40b2b50 0x7efce4151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce3fff640 1 -- 172.21.15.27:0/1972596702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce4154e80 con 0x7efce40d3400 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce2ffd640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce40b2b50 0x7efce4151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efce3fff640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40d3400 0x7efce415bbf0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009f40 tx=0x7efcdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efccbfff640 1 -- 172.21.15.27:0/1972596702 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc043020 con 0x7efce40d3400 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efccbfff640 1 -- 172.21.15.27:0/1972596702 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc03e450 con 0x7efce40d3400 2024-03-06T07:26:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce4155160 con 0x7efce40d3400 2024-03-06T07:26:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efccbfff640 1 -- 172.21.15.27:0/1972596702 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc009510 con 0x7efce40d3400 2024-03-06T07:26:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.765+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce4155720 con 0x7efce40d3400 2024-03-06T07:26:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.766+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce4068700 con 0x7efce40d3400 2024-03-06T07:26:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.767+0000 7efccbfff640 1 -- 172.21.15.27:0/1972596702 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcdc046020 con 0x7efce40d3400 2024-03-06T07:26:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.767+0000 7efccbfff640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcd006b4d0 0x7efcd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.768+0000 7efccbfff640 1 -- 172.21.15.27:0/1972596702 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcdc0aca10 con 0x7efce40d3400 2024-03-06T07:26:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.768+0000 7efce37fe640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcd006b4d0 0x7efcd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.768+0000 7efce37fe640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcd006b4d0 0x7efcd006d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efcd4001640 tx=0x7efcd4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.772+0000 7efccbfff640 1 -- 172.21.15.27:0/1972596702 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcdc04d030 con 0x7efce40d3400 2024-03-06T07:26:24.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.949+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efce40b92c0 con 0x7efcd006b4d0 2024-03-06T07:26:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.956+0000 7efccbfff640 1 -- 172.21.15.27:0/1972596702 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246754 (secure 0 0 0) 0x7efce40b92c0 con 0x7efcd006b4d0 2024-03-06T07:26:24.959 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:24.961 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:26:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.958+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcd006b4d0 msgr2=0x7efcd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.958+0000 7efcea16c640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcd006b4d0 0x7efcd006d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efcd4001640 tx=0x7efcd4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.958+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40d3400 msgr2=0x7efce415bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.958+0000 7efcea16c640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40d3400 0x7efce415bbf0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009f40 tx=0x7efcdc004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.959+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 shutdown_connections 2024-03-06T07:26:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.959+0000 7efcea16c640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efcd006b4d0 0x7efcd006d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.959+0000 7efcea16c640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce40d3400 0x7efce415bbf0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.959+0000 7efcea16c640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efce40b2b50 0x7efce4151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.959+0000 7efcea16c640 1 --2- 172.21.15.27:0/1972596702 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efce4071540 0x7efce41540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.959+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 >> 172.21.15.27:0/1972596702 conn(0x7efce40d3ca0 msgr2=0x7efce40b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.959+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 shutdown_connections 2024-03-06T07:26:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:24.959+0000 7efcea16c640 1 -- 172.21.15.27:0/1972596702 wait complete. 2024-03-06T07:26:24.971 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-03-06T07:26:24.038057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257056,"kb_used_data":7600,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717664,"statfs":{"total":773094113280,"available":772830887936,"internally_reserved":0,"allocated":7782400,"data_stored":3918114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.825797"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227981+0000","last_change":"2024-03-06T07:26:19.227981+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227981+0000","last_undegraded":"2024-03-06T07:26:19.227981+0000","last_fullsized":"2024-03-06T07:26:19.227981+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228347+0000","last_change":"2024-03-06T07:26:19.228347+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228347+0000","last_undegraded":"2024-03-06T07:26:19.228347+0000","last_fullsized":"2024-03-06T07:26:19.228347+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228671+0000","last_change":"2024-03-06T07:26:19.228671+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228671+0000","last_undegraded":"2024-03-06T07:26:19.228671+0000","last_fullsized":"2024-03-06T07:26:19.228671+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224081+0000","last_change":"2024-03-06T07:26:19.224081+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224081+0000","last_undegraded":"2024-03-06T07:26:19.224081+0000","last_fullsized":"2024-03-06T07:26:19.224081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227451+0000","last_change":"2024-03-06T07:26:19.227451+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227451+0000","last_undegraded":"2024-03-06T07:26:19.227451+0000","last_fullsized":"2024-03-06T07:26:19.227451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228991+0000","last_change":"2024-03-06T07:26:19.228991+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228991+0000","last_undegraded":"2024-03-06T07:26:19.228991+0000","last_fullsized":"2024-03-06T07:26:19.228991+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231923+0000","last_change":"2024-03-06T07:26:19.231923+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231923+0000","last_undegraded":"2024-03-06T07:26:19.231923+0000","last_fullsized":"2024-03-06T07:26:19.231923+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228094+0000","last_change":"2024-03-06T07:26:19.228094+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228094+0000","last_undegraded":"2024-03-06T07:26:19.228094+0000","last_fullsized":"2024-03-06T07:26:19.228094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226958+0000","last_change":"2024-03-06T07:26:19.226958+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226958+0000","last_undegraded":"2024-03-06T07:26:19.226958+0000","last_fullsized":"2024-03-06T07:26:19.226958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224802+0000","last_change":"2024-03-06T07:26:19.224802+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224802+0000","last_undegraded":"2024-03-06T07:26:19.224802+0000","last_fullsized":"2024-03-06T07:26:19.224802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231680+0000","last_change":"2024-03-06T07:26:19.231680+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231680+0000","last_undegraded":"2024-03-06T07:26:19.231680+0000","last_fullsized":"2024-03-06T07:26:19.231680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223377+0000","last_change":"2024-03-06T07:26:19.223377+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223377+0000","last_undegraded":"2024-03-06T07:26:19.223377+0000","last_fullsized":"2024-03-06T07:26:19.223377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221736+0000","last_change":"2024-03-06T07:26:19.221736+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221736+0000","last_undegraded":"2024-03-06T07:26:19.221736+0000","last_fullsized":"2024-03-06T07:26:19.221736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226495+0000","last_change":"2024-03-06T07:26:19.226495+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226495+0000","last_undegraded":"2024-03-06T07:26:19.226495+0000","last_fullsized":"2024-03-06T07:26:19.226495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222226+0000","last_change":"2024-03-06T07:26:19.222226+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222226+0000","last_undegraded":"2024-03-06T07:26:19.222226+0000","last_fullsized":"2024-03-06T07:26:19.222226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224253+0000","last_change":"2024-03-06T07:26:19.224253+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224253+0000","last_undegraded":"2024-03-06T07:26:19.224253+0000","last_fullsized":"2024-03-06T07:26:19.224253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225964+0000","last_change":"2024-03-06T07:26:19.225964+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225964+0000","last_undegraded":"2024-03-06T07:26:19.225964+0000","last_fullsized":"2024-03-06T07:26:19.225964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221558+0000","last_change":"2024-03-06T07:26:19.221558+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221558+0000","last_undegraded":"2024-03-06T07:26:19.221558+0000","last_fullsized":"2024-03-06T07:26:19.221558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228445+0000","last_change":"2024-03-06T07:26:19.228445+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228445+0000","last_undegraded":"2024-03-06T07:26:19.228445+0000","last_fullsized":"2024-03-06T07:26:19.228445+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223388+0000","last_change":"2024-03-06T07:26:19.223388+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223388+0000","last_undegraded":"2024-03-06T07:26:19.223388+0000","last_fullsized":"2024-03-06T07:26:19.223388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223534+0000","last_change":"2024-03-06T07:26:19.223534+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223534+0000","last_undegraded":"2024-03-06T07:26:19.223534+0000","last_fullsized":"2024-03-06T07:26:19.223534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223910+0000","last_change":"2024-03-06T07:26:19.223910+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223910+0000","last_undegraded":"2024-03-06T07:26:19.223910+0000","last_fullsized":"2024-03-06T07:26:19.223910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222903+0000","last_change":"2024-03-06T07:26:19.222903+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222903+0000","last_undegraded":"2024-03-06T07:26:19.222903+0000","last_fullsized":"2024-03-06T07:26:19.222903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224000+0000","last_change":"2024-03-06T07:26:19.224000+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224000+0000","last_undegraded":"2024-03-06T07:26:19.224000+0000","last_fullsized":"2024-03-06T07:26:19.224000+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222350+0000","last_change":"2024-03-06T07:26:19.222350+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222350+0000","last_undegraded":"2024-03-06T07:26:19.222350+0000","last_fullsized":"2024-03-06T07:26:19.222350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229681+0000","last_change":"2024-03-06T07:26:19.229681+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229681+0000","last_undegraded":"2024-03-06T07:26:19.229681+0000","last_fullsized":"2024-03-06T07:26:19.229681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228249+0000","last_change":"2024-03-06T07:26:19.228249+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228249+0000","last_undegraded":"2024-03-06T07:26:19.228249+0000","last_fullsized":"2024-03-06T07:26:19.228249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224326+0000","last_change":"2024-03-06T07:26:19.224326+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224326+0000","last_undegraded":"2024-03-06T07:26:19.224326+0000","last_fullsized":"2024-03-06T07:26:19.224326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228898+0000","last_change":"2024-03-06T07:26:19.228898+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228898+0000","last_undegraded":"2024-03-06T07:26:19.228898+0000","last_fullsized":"2024-03-06T07:26:19.228898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231473+0000","last_change":"2024-03-06T07:26:19.231473+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231473+0000","last_undegraded":"2024-03-06T07:26:19.231473+0000","last_fullsized":"2024-03-06T07:26:19.231473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227762+0000","last_change":"2024-03-06T07:26:19.227762+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227762+0000","last_undegraded":"2024-03-06T07:26:19.227762+0000","last_fullsized":"2024-03-06T07:26:19.227762+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227670+0000","last_change":"2024-03-06T07:26:19.227670+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227670+0000","last_undegraded":"2024-03-06T07:26:19.227670+0000","last_fullsized":"2024-03-06T07:26:19.227670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225593+0000","last_change":"2024-03-06T07:26:19.225593+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225593+0000","last_undegraded":"2024-03-06T07:26:19.225593+0000","last_fullsized":"2024-03-06T07:26:19.225593+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239388+0000","last_change":"2024-03-06T07:26:19.235249+0000","last_active":"2024-03-06T07:26:19.239388+0000","last_peered":"2024-03-06T07:26:19.239388+0000","last_clean":"2024-03-06T07:26:19.239388+0000","last_became_active":"2024-03-06T07:26:19.234904+0000","last_became_peered":"2024-03-06T07:26:19.234904+0000","last_unstale":"2024-03-06T07:26:19.239388+0000","last_undegraded":"2024-03-06T07:26:19.239388+0000","last_fullsized":"2024-03-06T07:26:19.239388+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:23.463441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225476+0000","last_change":"2024-03-06T07:26:19.225476+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225476+0000","last_undegraded":"2024-03-06T07:26:19.225476+0000","last_fullsized":"2024-03-06T07:26:19.225476+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.233898+0000","last_change":"2024-03-06T07:26:19.225962+0000","last_active":"2024-03-06T07:26:19.233898+0000","last_peered":"2024-03-06T07:26:19.233898+0000","last_clean":"2024-03-06T07:26:19.233898+0000","last_became_active":"2024-03-06T07:26:19.225707+0000","last_became_peered":"2024-03-06T07:26:19.225707+0000","last_unstale":"2024-03-06T07:26:19.233898+0000","last_undegraded":"2024-03-06T07:26:19.233898+0000","last_fullsized":"2024-03-06T07:26:19.233898+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:50.887101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228771+0000","last_change":"2024-03-06T07:26:19.228771+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228771+0000","last_undegraded":"2024-03-06T07:26:19.228771+0000","last_fullsized":"2024-03-06T07:26:19.228771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235050+0000","last_change":"2024-03-06T07:26:19.232800+0000","last_active":"2024-03-06T07:26:19.235050+0000","last_peered":"2024-03-06T07:26:19.235050+0000","last_clean":"2024-03-06T07:26:19.235050+0000","last_became_active":"2024-03-06T07:26:19.232541+0000","last_became_peered":"2024-03-06T07:26:19.232541+0000","last_unstale":"2024-03-06T07:26:19.235050+0000","last_undegraded":"2024-03-06T07:26:19.235050+0000","last_fullsized":"2024-03-06T07:26:19.235050+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:49.533898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.230992+0000","last_change":"2024-03-06T07:26:19.230992+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.230992+0000","last_undegraded":"2024-03-06T07:26:19.230992+0000","last_fullsized":"2024-03-06T07:26:19.230992+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239827+0000","last_change":"2024-03-06T07:26:19.237329+0000","last_active":"2024-03-06T07:26:19.239827+0000","last_peered":"2024-03-06T07:26:19.239827+0000","last_clean":"2024-03-06T07:26:19.239827+0000","last_became_active":"2024-03-06T07:26:19.237096+0000","last_became_peered":"2024-03-06T07:26:19.237096+0000","last_unstale":"2024-03-06T07:26:19.239827+0000","last_undegraded":"2024-03-06T07:26:19.239827+0000","last_fullsized":"2024-03-06T07:26:19.239827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:54.603601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231064+0000","last_change":"2024-03-06T07:26:19.231064+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231064+0000","last_undegraded":"2024-03-06T07:26:19.231064+0000","last_fullsized":"2024-03-06T07:26:19.231064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.237618+0000","last_change":"2024-03-06T07:26:19.236188+0000","last_active":"2024-03-06T07:26:19.237618+0000","last_peered":"2024-03-06T07:26:19.237618+0000","last_clean":"2024-03-06T07:26:19.237618+0000","last_became_active":"2024-03-06T07:26:19.235921+0000","last_became_peered":"2024-03-06T07:26:19.235921+0000","last_unstale":"2024-03-06T07:26:19.237618+0000","last_undegraded":"2024-03-06T07:26:19.237618+0000","last_fullsized":"2024-03-06T07:26:19.237618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:13.224175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222902+0000","last_change":"2024-03-06T07:26:19.222902+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222902+0000","last_undegraded":"2024-03-06T07:26:19.222902+0000","last_fullsized":"2024-03-06T07:26:19.222902+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242242+0000","last_change":"2024-03-06T07:26:19.237984+0000","last_active":"2024-03-06T07:26:19.242242+0000","last_peered":"2024-03-06T07:26:19.242242+0000","last_clean":"2024-03-06T07:26:19.242242+0000","last_became_active":"2024-03-06T07:26:19.237697+0000","last_became_peered":"2024-03-06T07:26:19.237697+0000","last_unstale":"2024-03-06T07:26:19.242242+0000","last_undegraded":"2024-03-06T07:26:19.242242+0000","last_fullsized":"2024-03-06T07:26:19.242242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:21.157010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225590+0000","last_change":"2024-03-06T07:26:19.225590+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225590+0000","last_undegraded":"2024-03-06T07:26:19.225590+0000","last_fullsized":"2024-03-06T07:26:19.225590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.232619+0000","last_change":"2024-03-06T07:26:19.228888+0000","last_active":"2024-03-06T07:26:19.232619+0000","last_peered":"2024-03-06T07:26:19.232619+0000","last_clean":"2024-03-06T07:26:19.232619+0000","last_became_active":"2024-03-06T07:26:19.228602+0000","last_became_peered":"2024-03-06T07:26:19.228602+0000","last_unstale":"2024-03-06T07:26:19.232619+0000","last_undegraded":"2024-03-06T07:26:19.232619+0000","last_fullsized":"2024-03-06T07:26:19.232619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:55.021248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221713+0000","last_change":"2024-03-06T07:26:19.221713+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221713+0000","last_undegraded":"2024-03-06T07:26:19.221713+0000","last_fullsized":"2024-03-06T07:26:19.221713+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.234079+0000","last_change":"2024-03-06T07:26:19.231771+0000","last_active":"2024-03-06T07:26:19.234079+0000","last_peered":"2024-03-06T07:26:19.234079+0000","last_clean":"2024-03-06T07:26:19.234079+0000","last_became_active":"2024-03-06T07:26:19.231485+0000","last_became_peered":"2024-03-06T07:26:19.231485+0000","last_unstale":"2024-03-06T07:26:19.234079+0000","last_undegraded":"2024-03-06T07:26:19.234079+0000","last_fullsized":"2024-03-06T07:26:19.234079+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:30.451695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229259+0000","last_change":"2024-03-06T07:26:19.229259+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229259+0000","last_undegraded":"2024-03-06T07:26:19.229259+0000","last_fullsized":"2024-03-06T07:26:19.229259+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.230928+0000","last_change":"2024-03-06T07:26:19.229136+0000","last_active":"2024-03-06T07:26:19.230928+0000","last_peered":"2024-03-06T07:26:19.230928+0000","last_clean":"2024-03-06T07:26:19.230928+0000","last_became_active":"2024-03-06T07:26:19.228661+0000","last_became_peered":"2024-03-06T07:26:19.228661+0000","last_unstale":"2024-03-06T07:26:19.230928+0000","last_undegraded":"2024-03-06T07:26:19.230928+0000","last_fullsized":"2024-03-06T07:26:19.230928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:56.368954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225377+0000","last_change":"2024-03-06T07:26:19.225377+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225377+0000","last_undegraded":"2024-03-06T07:26:19.225377+0000","last_fullsized":"2024-03-06T07:26:19.225377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239800+0000","last_change":"2024-03-06T07:26:19.236797+0000","last_active":"2024-03-06T07:26:19.239800+0000","last_peered":"2024-03-06T07:26:19.239800+0000","last_clean":"2024-03-06T07:26:19.239800+0000","last_became_active":"2024-03-06T07:26:19.236118+0000","last_became_peered":"2024-03-06T07:26:19.236118+0000","last_unstale":"2024-03-06T07:26:19.239800+0000","last_undegraded":"2024-03-06T07:26:19.239800+0000","last_fullsized":"2024-03-06T07:26:19.239800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:04.139082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228026+0000","last_change":"2024-03-06T07:26:19.228026+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228026+0000","last_undegraded":"2024-03-06T07:26:19.228026+0000","last_fullsized":"2024-03-06T07:26:19.228026+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.236116+0000","last_change":"2024-03-06T07:26:19.235158+0000","last_active":"2024-03-06T07:26:19.236116+0000","last_peered":"2024-03-06T07:26:19.236116+0000","last_clean":"2024-03-06T07:26:19.236116+0000","last_became_active":"2024-03-06T07:26:19.234883+0000","last_became_peered":"2024-03-06T07:26:19.234883+0000","last_unstale":"2024-03-06T07:26:19.236116+0000","last_undegraded":"2024-03-06T07:26:19.236116+0000","last_fullsized":"2024-03-06T07:26:19.236116+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:17.059043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225110+0000","last_change":"2024-03-06T07:26:19.225110+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225110+0000","last_undegraded":"2024-03-06T07:26:19.225110+0000","last_fullsized":"2024-03-06T07:26:19.225110+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.238717+0000","last_change":"2024-03-06T07:26:19.236129+0000","last_active":"2024-03-06T07:26:19.238717+0000","last_peered":"2024-03-06T07:26:19.238717+0000","last_clean":"2024-03-06T07:26:19.238717+0000","last_became_active":"2024-03-06T07:26:19.235844+0000","last_became_peered":"2024-03-06T07:26:19.235844+0000","last_unstale":"2024-03-06T07:26:19.238717+0000","last_undegraded":"2024-03-06T07:26:19.238717+0000","last_fullsized":"2024-03-06T07:26:19.238717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:04.139082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223551+0000","last_change":"2024-03-06T07:26:19.223551+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223551+0000","last_undegraded":"2024-03-06T07:26:19.223551+0000","last_fullsized":"2024-03-06T07:26:19.223551+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.228861+0000","last_change":"2024-03-06T07:26:19.226330+0000","last_active":"2024-03-06T07:26:19.228861+0000","last_peered":"2024-03-06T07:26:19.228861+0000","last_clean":"2024-03-06T07:26:19.228861+0000","last_became_active":"2024-03-06T07:26:19.226062+0000","last_became_peered":"2024-03-06T07:26:19.226062+0000","last_unstale":"2024-03-06T07:26:19.228861+0000","last_undegraded":"2024-03-06T07:26:19.228861+0000","last_fullsized":"2024-03-06T07:26:19.228861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:29.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.230238+0000","last_change":"2024-03-06T07:26:19.230238+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.230238+0000","last_undegraded":"2024-03-06T07:26:19.230238+0000","last_fullsized":"2024-03-06T07:26:19.230238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242264+0000","last_change":"2024-03-06T07:26:19.238616+0000","last_active":"2024-03-06T07:26:19.242264+0000","last_peered":"2024-03-06T07:26:19.242264+0000","last_clean":"2024-03-06T07:26:19.242264+0000","last_became_active":"2024-03-06T07:26:19.238337+0000","last_became_peered":"2024-03-06T07:26:19.238337+0000","last_unstale":"2024-03-06T07:26:19.242264+0000","last_undegraded":"2024-03-06T07:26:19.242264+0000","last_fullsized":"2024-03-06T07:26:19.242264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:14.382999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222417+0000","last_change":"2024-03-06T07:26:19.222417+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222417+0000","last_undegraded":"2024-03-06T07:26:19.222417+0000","last_fullsized":"2024-03-06T07:26:19.222417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239417+0000","last_change":"2024-03-06T07:26:19.236852+0000","last_active":"2024-03-06T07:26:19.239417+0000","last_peered":"2024-03-06T07:26:19.239417+0000","last_clean":"2024-03-06T07:26:19.239417+0000","last_became_active":"2024-03-06T07:26:19.236469+0000","last_became_peered":"2024-03-06T07:26:19.236469+0000","last_unstale":"2024-03-06T07:26:19.239417+0000","last_undegraded":"2024-03-06T07:26:19.239417+0000","last_fullsized":"2024-03-06T07:26:19.239417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:20.767924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224910+0000","last_change":"2024-03-06T07:26:19.224910+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224910+0000","last_undegraded":"2024-03-06T07:26:19.224910+0000","last_fullsized":"2024-03-06T07:26:19.224910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235094+0000","last_change":"2024-03-06T07:26:19.231717+0000","last_active":"2024-03-06T07:26:19.235094+0000","last_peered":"2024-03-06T07:26:19.235094+0000","last_clean":"2024-03-06T07:26:19.235094+0000","last_became_active":"2024-03-06T07:26:19.231462+0000","last_became_peered":"2024-03-06T07:26:19.231462+0000","last_unstale":"2024-03-06T07:26:19.235094+0000","last_undegraded":"2024-03-06T07:26:19.235094+0000","last_fullsized":"2024-03-06T07:26:19.235094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.795497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223271+0000","last_change":"2024-03-06T07:26:19.223271+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223271+0000","last_undegraded":"2024-03-06T07:26:19.223271+0000","last_fullsized":"2024-03-06T07:26:19.223271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239822+0000","last_change":"2024-03-06T07:26:19.237309+0000","last_active":"2024-03-06T07:26:19.239822+0000","last_peered":"2024-03-06T07:26:19.239822+0000","last_clean":"2024-03-06T07:26:19.239822+0000","last_became_active":"2024-03-06T07:26:19.237011+0000","last_became_peered":"2024-03-06T07:26:19.237011+0000","last_unstale":"2024-03-06T07:26:19.239822+0000","last_undegraded":"2024-03-06T07:26:19.239822+0000","last_fullsized":"2024-03-06T07:26:19.239822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:58.527722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224977+0000","last_change":"2024-03-06T07:26:19.224977+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224977+0000","last_undegraded":"2024-03-06T07:26:19.224977+0000","last_fullsized":"2024-03-06T07:26:19.224977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235073+0000","last_change":"2024-03-06T07:26:19.233492+0000","last_active":"2024-03-06T07:26:19.235073+0000","last_peered":"2024-03-06T07:26:19.235073+0000","last_clean":"2024-03-06T07:26:19.235073+0000","last_became_active":"2024-03-06T07:26:19.233199+0000","last_became_peered":"2024-03-06T07:26:19.233199+0000","last_unstale":"2024-03-06T07:26:19.235073+0000","last_undegraded":"2024-03-06T07:26:19.235073+0000","last_fullsized":"2024-03-06T07:26:19.235073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:22.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222478+0000","last_change":"2024-03-06T07:26:19.222478+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222478+0000","last_undegraded":"2024-03-06T07:26:19.222478+0000","last_fullsized":"2024-03-06T07:26:19.222478+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239765+0000","last_change":"2024-03-06T07:26:19.236432+0000","last_active":"2024-03-06T07:26:19.239765+0000","last_peered":"2024-03-06T07:26:19.239765+0000","last_clean":"2024-03-06T07:26:19.239765+0000","last_became_active":"2024-03-06T07:26:19.235975+0000","last_became_peered":"2024-03-06T07:26:19.235975+0000","last_unstale":"2024-03-06T07:26:19.239765+0000","last_undegraded":"2024-03-06T07:26:19.239765+0000","last_fullsized":"2024-03-06T07:26:19.239765+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:55.021248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224496+0000","last_change":"2024-03-06T07:26:19.224496+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224496+0000","last_undegraded":"2024-03-06T07:26:19.224496+0000","last_fullsized":"2024-03-06T07:26:19.224496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.234690+0000","last_change":"2024-03-06T07:26:19.233564+0000","last_active":"2024-03-06T07:26:19.234690+0000","last_peered":"2024-03-06T07:26:19.234690+0000","last_clean":"2024-03-06T07:26:19.234690+0000","last_became_active":"2024-03-06T07:26:19.233235+0000","last_became_peered":"2024-03-06T07:26:19.233235+0000","last_unstale":"2024-03-06T07:26:19.234690+0000","last_undegraded":"2024-03-06T07:26:19.234690+0000","last_fullsized":"2024-03-06T07:26:19.234690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:44.732093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228500+0000","last_change":"2024-03-06T07:26:19.228500+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228500+0000","last_undegraded":"2024-03-06T07:26:19.228500+0000","last_fullsized":"2024-03-06T07:26:19.228500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239745+0000","last_change":"2024-03-06T07:26:19.236405+0000","last_active":"2024-03-06T07:26:19.239745+0000","last_peered":"2024-03-06T07:26:19.239745+0000","last_clean":"2024-03-06T07:26:19.239745+0000","last_became_active":"2024-03-06T07:26:19.235876+0000","last_became_peered":"2024-03-06T07:26:19.235876+0000","last_unstale":"2024-03-06T07:26:19.239745+0000","last_undegraded":"2024-03-06T07:26:19.239745+0000","last_fullsized":"2024-03-06T07:26:19.239745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:29.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225680+0000","last_change":"2024-03-06T07:26:19.225680+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225680+0000","last_undegraded":"2024-03-06T07:26:19.225680+0000","last_fullsized":"2024-03-06T07:26:19.225680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242212+0000","last_change":"2024-03-06T07:26:19.238646+0000","last_active":"2024-03-06T07:26:19.242212+0000","last_peered":"2024-03-06T07:26:19.242212+0000","last_clean":"2024-03-06T07:26:19.242212+0000","last_became_active":"2024-03-06T07:26:19.238376+0000","last_became_peered":"2024-03-06T07:26:19.238376+0000","last_unstale":"2024-03-06T07:26:19.242212+0000","last_undegraded":"2024-03-06T07:26:19.242212+0000","last_fullsized":"2024-03-06T07:26:19.242212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:33.080991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225633+0000","last_change":"2024-03-06T07:26:19.225633+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225633+0000","last_undegraded":"2024-03-06T07:26:19.225633+0000","last_fullsized":"2024-03-06T07:26:19.225633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.233488+0000","last_change":"2024-03-06T07:26:19.227589+0000","last_active":"2024-03-06T07:26:19.233488+0000","last_peered":"2024-03-06T07:26:19.233488+0000","last_clean":"2024-03-06T07:26:19.233488+0000","last_became_active":"2024-03-06T07:26:19.227308+0000","last_became_peered":"2024-03-06T07:26:19.227308+0000","last_unstale":"2024-03-06T07:26:19.233488+0000","last_undegraded":"2024-03-06T07:26:19.233488+0000","last_fullsized":"2024-03-06T07:26:19.233488+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:19.014952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222420+0000","last_change":"2024-03-06T07:26:19.222420+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222420+0000","last_undegraded":"2024-03-06T07:26:19.222420+0000","last_fullsized":"2024-03-06T07:26:19.222420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239645+0000","last_change":"2024-03-06T07:26:19.234268+0000","last_active":"2024-03-06T07:26:19.239645+0000","last_peered":"2024-03-06T07:26:19.239645+0000","last_clean":"2024-03-06T07:26:19.239645+0000","last_became_active":"2024-03-06T07:26:19.234000+0000","last_became_peered":"2024-03-06T07:26:19.234000+0000","last_unstale":"2024-03-06T07:26:19.239645+0000","last_undegraded":"2024-03-06T07:26:19.239645+0000","last_fullsized":"2024-03-06T07:26:19.239645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:50.102543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229363+0000","last_change":"2024-03-06T07:26:19.229363+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229363+0000","last_undegraded":"2024-03-06T07:26:19.229363+0000","last_fullsized":"2024-03-06T07:26:19.229363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242098+0000","last_change":"2024-03-06T07:26:19.237743+0000","last_active":"2024-03-06T07:26:19.242098+0000","last_peered":"2024-03-06T07:26:19.242098+0000","last_clean":"2024-03-06T07:26:19.242098+0000","last_became_active":"2024-03-06T07:26:19.237394+0000","last_became_peered":"2024-03-06T07:26:19.237394+0000","last_unstale":"2024-03-06T07:26:19.242098+0000","last_undegraded":"2024-03-06T07:26:19.242098+0000","last_fullsized":"2024-03-06T07:26:19.242098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:35.749995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225400+0000","last_change":"2024-03-06T07:26:19.225400+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225400+0000","last_undegraded":"2024-03-06T07:26:19.225400+0000","last_fullsized":"2024-03-06T07:26:19.225400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.231970+0000","last_change":"2024-03-06T07:26:19.230603+0000","last_active":"2024-03-06T07:26:19.231970+0000","last_peered":"2024-03-06T07:26:19.231970+0000","last_clean":"2024-03-06T07:26:19.231970+0000","last_became_active":"2024-03-06T07:26:19.230313+0000","last_became_peered":"2024-03-06T07:26:19.230313+0000","last_unstale":"2024-03-06T07:26:19.231970+0000","last_undegraded":"2024-03-06T07:26:19.231970+0000","last_fullsized":"2024-03-06T07:26:19.231970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:31.518943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226436+0000","last_change":"2024-03-06T07:26:19.226436+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226436+0000","last_undegraded":"2024-03-06T07:26:19.226436+0000","last_fullsized":"2024-03-06T07:26:19.226436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.236928+0000","last_change":"2024-03-06T07:26:19.234353+0000","last_active":"2024-03-06T07:26:19.236928+0000","last_peered":"2024-03-06T07:26:19.236928+0000","last_clean":"2024-03-06T07:26:19.236928+0000","last_became_active":"2024-03-06T07:26:19.234077+0000","last_became_peered":"2024-03-06T07:26:19.234077+0000","last_unstale":"2024-03-06T07:26:19.236928+0000","last_undegraded":"2024-03-06T07:26:19.236928+0000","last_fullsized":"2024-03-06T07:26:19.236928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:37.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221555+0000","last_change":"2024-03-06T07:26:19.221555+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221555+0000","last_undegraded":"2024-03-06T07:26:19.221555+0000","last_fullsized":"2024-03-06T07:26:19.221555+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.238279+0000","last_change":"2024-03-06T07:26:19.237287+0000","last_active":"2024-03-06T07:26:19.238279+0000","last_peered":"2024-03-06T07:26:19.238279+0000","last_clean":"2024-03-06T07:26:19.238279+0000","last_became_active":"2024-03-06T07:26:19.237017+0000","last_became_peered":"2024-03-06T07:26:19.237017+0000","last_unstale":"2024-03-06T07:26:19.238279+0000","last_undegraded":"2024-03-06T07:26:19.238279+0000","last_fullsized":"2024-03-06T07:26:19.238279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:46.817231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227191+0000","last_change":"2024-03-06T07:26:19.227191+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227191+0000","last_undegraded":"2024-03-06T07:26:19.227191+0000","last_fullsized":"2024-03-06T07:26:19.227191+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.232804+0000","last_change":"2024-03-06T07:26:19.229645+0000","last_active":"2024-03-06T07:26:19.232804+0000","last_peered":"2024-03-06T07:26:19.232804+0000","last_clean":"2024-03-06T07:26:19.232804+0000","last_became_active":"2024-03-06T07:26:19.229309+0000","last_became_peered":"2024-03-06T07:26:19.229309+0000","last_unstale":"2024-03-06T07:26:19.232804+0000","last_undegraded":"2024-03-06T07:26:19.232804+0000","last_fullsized":"2024-03-06T07:26:19.232804+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.342246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222416+0000","last_change":"2024-03-06T07:26:19.222416+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222416+0000","last_undegraded":"2024-03-06T07:26:19.222416+0000","last_fullsized":"2024-03-06T07:26:19.222416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235092+0000","last_change":"2024-03-06T07:26:19.231241+0000","last_active":"2024-03-06T07:26:19.235092+0000","last_peered":"2024-03-06T07:26:19.235092+0000","last_clean":"2024-03-06T07:26:19.235092+0000","last_became_active":"2024-03-06T07:26:19.230977+0000","last_became_peered":"2024-03-06T07:26:19.230977+0000","last_unstale":"2024-03-06T07:26:19.235092+0000","last_undegraded":"2024-03-06T07:26:19.235092+0000","last_fullsized":"2024-03-06T07:26:19.235092+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:36.204568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227602+0000","last_change":"2024-03-06T07:26:19.227602+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227602+0000","last_undegraded":"2024-03-06T07:26:19.227602+0000","last_fullsized":"2024-03-06T07:26:19.227602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242260+0000","last_change":"2024-03-06T07:26:19.238002+0000","last_active":"2024-03-06T07:26:19.242260+0000","last_peered":"2024-03-06T07:26:19.242260+0000","last_clean":"2024-03-06T07:26:19.242260+0000","last_became_active":"2024-03-06T07:26:19.237710+0000","last_became_peered":"2024-03-06T07:26:19.237710+0000","last_unstale":"2024-03-06T07:26:19.242260+0000","last_undegraded":"2024-03-06T07:26:19.242260+0000","last_fullsized":"2024-03-06T07:26:19.242260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:25.030580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226570+0000","last_change":"2024-03-06T07:26:19.226570+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226570+0000","last_undegraded":"2024-03-06T07:26:19.226570+0000","last_fullsized":"2024-03-06T07:26:19.226570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239767+0000","last_change":"2024-03-06T07:26:19.236691+0000","last_active":"2024-03-06T07:26:19.239767+0000","last_peered":"2024-03-06T07:26:19.239767+0000","last_clean":"2024-03-06T07:26:19.239767+0000","last_became_active":"2024-03-06T07:26:19.236432+0000","last_became_peered":"2024-03-06T07:26:19.236432+0000","last_unstale":"2024-03-06T07:26:19.239767+0000","last_undegraded":"2024-03-06T07:26:19.239767+0000","last_fullsized":"2024-03-06T07:26:19.239767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.342246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":261,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.224547+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:26:19.224547+0000","last_peered":"2024-03-06T07:26:19.224547+0000","last_clean":"2024-03-06T07:26:19.224547+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:26:19.224547+0000","last_undegraded":"2024-03-06T07:26:19.224547+0000","last_fullsized":"2024-03-06T07:26:19.224547+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":815104,"data_stored":337466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":337466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3f is not active+clean 2024-03-06T07:26:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.227981+0000', 'last_change': '2024-03-06T07:26:19.227981+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.227981+0000', 'last_undegraded': '2024-03-06T07:26:19.227981+0000', 'last_fullsized': '2024-03-06T07:26:19.227981+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3e is not active+clean 2024-03-06T07:26:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228347+0000', 'last_change': '2024-03-06T07:26:19.228347+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228347+0000', 'last_undegraded': '2024-03-06T07:26:19.228347+0000', 'last_fullsized': '2024-03-06T07:26:19.228347+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3d is not active+clean 2024-03-06T07:26:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228671+0000', 'last_change': '2024-03-06T07:26:19.228671+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228671+0000', 'last_undegraded': '2024-03-06T07:26:19.228671+0000', 'last_fullsized': '2024-03-06T07:26:19.228671+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3c is not active+clean 2024-03-06T07:26:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.224081+0000', 'last_change': '2024-03-06T07:26:19.224081+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.224081+0000', 'last_undegraded': '2024-03-06T07:26:19.224081+0000', 'last_fullsized': '2024-03-06T07:26:19.224081+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3b is not active+clean 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.227451+0000', 'last_change': '2024-03-06T07:26:19.227451+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.227451+0000', 'last_undegraded': '2024-03-06T07:26:19.227451+0000', 'last_fullsized': '2024-03-06T07:26:19.227451+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3a is not active+clean 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228991+0000', 'last_change': '2024-03-06T07:26:19.228991+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228991+0000', 'last_undegraded': '2024-03-06T07:26:19.228991+0000', 'last_fullsized': '2024-03-06T07:26:19.228991+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.39 is not active+clean 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.231923+0000', 'last_change': '2024-03-06T07:26:19.231923+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.231923+0000', 'last_undegraded': '2024-03-06T07:26:19.231923+0000', 'last_fullsized': '2024-03-06T07:26:19.231923+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.38 is not active+clean 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228094+0000', 'last_change': '2024-03-06T07:26:19.228094+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228094+0000', 'last_undegraded': '2024-03-06T07:26:19.228094+0000', 'last_fullsized': '2024-03-06T07:26:19.228094+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.37 is not active+clean 2024-03-06T07:26:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.226958+0000', 'last_change': '2024-03-06T07:26:19.226958+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.226958+0000', 'last_undegraded': '2024-03-06T07:26:19.226958+0000', 'last_fullsized': '2024-03-06T07:26:19.226958+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.36 is not active+clean 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.224802+0000', 'last_change': '2024-03-06T07:26:19.224802+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.224802+0000', 'last_undegraded': '2024-03-06T07:26:19.224802+0000', 'last_fullsized': '2024-03-06T07:26:19.224802+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.35 is not active+clean 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.231680+0000', 'last_change': '2024-03-06T07:26:19.231680+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.231680+0000', 'last_undegraded': '2024-03-06T07:26:19.231680+0000', 'last_fullsized': '2024-03-06T07:26:19.231680+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.34 is not active+clean 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.223377+0000', 'last_change': '2024-03-06T07:26:19.223377+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.223377+0000', 'last_undegraded': '2024-03-06T07:26:19.223377+0000', 'last_fullsized': '2024-03-06T07:26:19.223377+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.33 is not active+clean 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.221736+0000', 'last_change': '2024-03-06T07:26:19.221736+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.221736+0000', 'last_undegraded': '2024-03-06T07:26:19.221736+0000', 'last_fullsized': '2024-03-06T07:26:19.221736+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.32 is not active+clean 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.226495+0000', 'last_change': '2024-03-06T07:26:19.226495+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.226495+0000', 'last_undegraded': '2024-03-06T07:26:19.226495+0000', 'last_fullsized': '2024-03-06T07:26:19.226495+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.31 is not active+clean 2024-03-06T07:26:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.222226+0000', 'last_change': '2024-03-06T07:26:19.222226+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.222226+0000', 'last_undegraded': '2024-03-06T07:26:19.222226+0000', 'last_fullsized': '2024-03-06T07:26:19.222226+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.30 is not active+clean 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.224253+0000', 'last_change': '2024-03-06T07:26:19.224253+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.224253+0000', 'last_undegraded': '2024-03-06T07:26:19.224253+0000', 'last_fullsized': '2024-03-06T07:26:19.224253+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2f is not active+clean 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.225964+0000', 'last_change': '2024-03-06T07:26:19.225964+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.225964+0000', 'last_undegraded': '2024-03-06T07:26:19.225964+0000', 'last_fullsized': '2024-03-06T07:26:19.225964+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2e is not active+clean 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.221558+0000', 'last_change': '2024-03-06T07:26:19.221558+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.221558+0000', 'last_undegraded': '2024-03-06T07:26:19.221558+0000', 'last_fullsized': '2024-03-06T07:26:19.221558+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2d is not active+clean 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228445+0000', 'last_change': '2024-03-06T07:26:19.228445+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228445+0000', 'last_undegraded': '2024-03-06T07:26:19.228445+0000', 'last_fullsized': '2024-03-06T07:26:19.228445+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2c is not active+clean 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.223388+0000', 'last_change': '2024-03-06T07:26:19.223388+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.223388+0000', 'last_undegraded': '2024-03-06T07:26:19.223388+0000', 'last_fullsized': '2024-03-06T07:26:19.223388+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2b is not active+clean 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.223534+0000', 'last_change': '2024-03-06T07:26:19.223534+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.223534+0000', 'last_undegraded': '2024-03-06T07:26:19.223534+0000', 'last_fullsized': '2024-03-06T07:26:19.223534+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2a is not active+clean 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.223910+0000', 'last_change': '2024-03-06T07:26:19.223910+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.223910+0000', 'last_undegraded': '2024-03-06T07:26:19.223910+0000', 'last_fullsized': '2024-03-06T07:26:19.223910+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.29 is not active+clean 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.222903+0000', 'last_change': '2024-03-06T07:26:19.222903+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.222903+0000', 'last_undegraded': '2024-03-06T07:26:19.222903+0000', 'last_fullsized': '2024-03-06T07:26:19.222903+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.28 is not active+clean 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.224000+0000', 'last_change': '2024-03-06T07:26:19.224000+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.224000+0000', 'last_undegraded': '2024-03-06T07:26:19.224000+0000', 'last_fullsized': '2024-03-06T07:26:19.224000+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.27 is not active+clean 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.222350+0000', 'last_change': '2024-03-06T07:26:19.222350+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.222350+0000', 'last_undegraded': '2024-03-06T07:26:19.222350+0000', 'last_fullsized': '2024-03-06T07:26:19.222350+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.26 is not active+clean 2024-03-06T07:26:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.229681+0000', 'last_change': '2024-03-06T07:26:19.229681+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.229681+0000', 'last_undegraded': '2024-03-06T07:26:19.229681+0000', 'last_fullsized': '2024-03-06T07:26:19.229681+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.25 is not active+clean 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228249+0000', 'last_change': '2024-03-06T07:26:19.228249+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228249+0000', 'last_undegraded': '2024-03-06T07:26:19.228249+0000', 'last_fullsized': '2024-03-06T07:26:19.228249+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.24 is not active+clean 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.224326+0000', 'last_change': '2024-03-06T07:26:19.224326+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.224326+0000', 'last_undegraded': '2024-03-06T07:26:19.224326+0000', 'last_fullsized': '2024-03-06T07:26:19.224326+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.23 is not active+clean 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228898+0000', 'last_change': '2024-03-06T07:26:19.228898+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228898+0000', 'last_undegraded': '2024-03-06T07:26:19.228898+0000', 'last_fullsized': '2024-03-06T07:26:19.228898+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.22 is not active+clean 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.231473+0000', 'last_change': '2024-03-06T07:26:19.231473+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.231473+0000', 'last_undegraded': '2024-03-06T07:26:19.231473+0000', 'last_fullsized': '2024-03-06T07:26:19.231473+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.21 is not active+clean 2024-03-06T07:26:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.227762+0000', 'last_change': '2024-03-06T07:26:19.227762+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.227762+0000', 'last_undegraded': '2024-03-06T07:26:19.227762+0000', 'last_fullsized': '2024-03-06T07:26:19.227762+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.20 is not active+clean 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.227670+0000', 'last_change': '2024-03-06T07:26:19.227670+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.227670+0000', 'last_undegraded': '2024-03-06T07:26:19.227670+0000', 'last_fullsized': '2024-03-06T07:26:19.227670+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.225593+0000', 'last_change': '2024-03-06T07:26:19.225593+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.225593+0000', 'last_undegraded': '2024-03-06T07:26:19.225593+0000', 'last_fullsized': '2024-03-06T07:26:19.225593+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.225476+0000', 'last_change': '2024-03-06T07:26:19.225476+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.225476+0000', 'last_undegraded': '2024-03-06T07:26:19.225476+0000', 'last_fullsized': '2024-03-06T07:26:19.225476+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228771+0000', 'last_change': '2024-03-06T07:26:19.228771+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228771+0000', 'last_undegraded': '2024-03-06T07:26:19.228771+0000', 'last_fullsized': '2024-03-06T07:26:19.228771+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.230992+0000', 'last_change': '2024-03-06T07:26:19.230992+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.230992+0000', 'last_undegraded': '2024-03-06T07:26:19.230992+0000', 'last_fullsized': '2024-03-06T07:26:19.230992+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.231064+0000', 'last_change': '2024-03-06T07:26:19.231064+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.231064+0000', 'last_undegraded': '2024-03-06T07:26:19.231064+0000', 'last_fullsized': '2024-03-06T07:26:19.231064+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.222902+0000', 'last_change': '2024-03-06T07:26:19.222902+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.222902+0000', 'last_undegraded': '2024-03-06T07:26:19.222902+0000', 'last_fullsized': '2024-03-06T07:26:19.222902+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.225590+0000', 'last_change': '2024-03-06T07:26:19.225590+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.225590+0000', 'last_undegraded': '2024-03-06T07:26:19.225590+0000', 'last_fullsized': '2024-03-06T07:26:19.225590+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.221713+0000', 'last_change': '2024-03-06T07:26:19.221713+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.221713+0000', 'last_undegraded': '2024-03-06T07:26:19.221713+0000', 'last_fullsized': '2024-03-06T07:26:19.221713+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.229259+0000', 'last_change': '2024-03-06T07:26:19.229259+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.229259+0000', 'last_undegraded': '2024-03-06T07:26:19.229259+0000', 'last_fullsized': '2024-03-06T07:26:19.229259+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-03-06T07:26:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.225377+0000', 'last_change': '2024-03-06T07:26:19.225377+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.225377+0000', 'last_undegraded': '2024-03-06T07:26:19.225377+0000', 'last_fullsized': '2024-03-06T07:26:19.225377+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228026+0000', 'last_change': '2024-03-06T07:26:19.228026+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228026+0000', 'last_undegraded': '2024-03-06T07:26:19.228026+0000', 'last_fullsized': '2024-03-06T07:26:19.228026+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.225110+0000', 'last_change': '2024-03-06T07:26:19.225110+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.225110+0000', 'last_undegraded': '2024-03-06T07:26:19.225110+0000', 'last_fullsized': '2024-03-06T07:26:19.225110+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.223551+0000', 'last_change': '2024-03-06T07:26:19.223551+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.223551+0000', 'last_undegraded': '2024-03-06T07:26:19.223551+0000', 'last_fullsized': '2024-03-06T07:26:19.223551+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.230238+0000', 'last_change': '2024-03-06T07:26:19.230238+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.230238+0000', 'last_undegraded': '2024-03-06T07:26:19.230238+0000', 'last_fullsized': '2024-03-06T07:26:19.230238+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.222417+0000', 'last_change': '2024-03-06T07:26:19.222417+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.222417+0000', 'last_undegraded': '2024-03-06T07:26:19.222417+0000', 'last_fullsized': '2024-03-06T07:26:19.222417+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.224910+0000', 'last_change': '2024-03-06T07:26:19.224910+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.224910+0000', 'last_undegraded': '2024-03-06T07:26:19.224910+0000', 'last_fullsized': '2024-03-06T07:26:19.224910+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.223271+0000', 'last_change': '2024-03-06T07:26:19.223271+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.223271+0000', 'last_undegraded': '2024-03-06T07:26:19.223271+0000', 'last_fullsized': '2024-03-06T07:26:19.223271+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.224977+0000', 'last_change': '2024-03-06T07:26:19.224977+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.224977+0000', 'last_undegraded': '2024-03-06T07:26:19.224977+0000', 'last_fullsized': '2024-03-06T07:26:19.224977+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.222478+0000', 'last_change': '2024-03-06T07:26:19.222478+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.222478+0000', 'last_undegraded': '2024-03-06T07:26:19.222478+0000', 'last_fullsized': '2024-03-06T07:26:19.222478+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.224496+0000', 'last_change': '2024-03-06T07:26:19.224496+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.224496+0000', 'last_undegraded': '2024-03-06T07:26:19.224496+0000', 'last_fullsized': '2024-03-06T07:26:19.224496+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-03-06T07:26:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.228500+0000', 'last_change': '2024-03-06T07:26:19.228500+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.228500+0000', 'last_undegraded': '2024-03-06T07:26:19.228500+0000', 'last_fullsized': '2024-03-06T07:26:19.228500+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.225680+0000', 'last_change': '2024-03-06T07:26:19.225680+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.225680+0000', 'last_undegraded': '2024-03-06T07:26:19.225680+0000', 'last_fullsized': '2024-03-06T07:26:19.225680+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.225633+0000', 'last_change': '2024-03-06T07:26:19.225633+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.225633+0000', 'last_undegraded': '2024-03-06T07:26:19.225633+0000', 'last_fullsized': '2024-03-06T07:26:19.225633+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.222420+0000', 'last_change': '2024-03-06T07:26:19.222420+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.222420+0000', 'last_undegraded': '2024-03-06T07:26:19.222420+0000', 'last_fullsized': '2024-03-06T07:26:19.222420+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.229363+0000', 'last_change': '2024-03-06T07:26:19.229363+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.229363+0000', 'last_undegraded': '2024-03-06T07:26:19.229363+0000', 'last_fullsized': '2024-03-06T07:26:19.229363+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.225400+0000', 'last_change': '2024-03-06T07:26:19.225400+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.225400+0000', 'last_undegraded': '2024-03-06T07:26:19.225400+0000', 'last_fullsized': '2024-03-06T07:26:19.225400+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-03-06T07:26:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.226436+0000', 'last_change': '2024-03-06T07:26:19.226436+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.226436+0000', 'last_undegraded': '2024-03-06T07:26:19.226436+0000', 'last_fullsized': '2024-03-06T07:26:19.226436+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.221555+0000', 'last_change': '2024-03-06T07:26:19.221555+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.221555+0000', 'last_undegraded': '2024-03-06T07:26:19.221555+0000', 'last_fullsized': '2024-03-06T07:26:19.221555+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.227191+0000', 'last_change': '2024-03-06T07:26:19.227191+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.227191+0000', 'last_undegraded': '2024-03-06T07:26:19.227191+0000', 'last_fullsized': '2024-03-06T07:26:19.227191+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.222416+0000', 'last_change': '2024-03-06T07:26:19.222416+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.222416+0000', 'last_undegraded': '2024-03-06T07:26:19.222416+0000', 'last_fullsized': '2024-03-06T07:26:19.222416+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.227602+0000', 'last_change': '2024-03-06T07:26:19.227602+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.227602+0000', 'last_undegraded': '2024-03-06T07:26:19.227602+0000', 'last_fullsized': '2024-03-06T07:26:19.227602+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-03-06T07:26:24.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:19.226570+0000', 'last_change': '2024-03-06T07:26:19.226570+0000', 'last_active': '2024-03-06T07:26:19.211880+0000', 'last_peered': '2024-03-06T07:26:19.211880+0000', 'last_clean': '2024-03-06T07:26:19.211880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:19.226570+0000', 'last_undegraded': '2024-03-06T07:26:19.226570+0000', 'last_fullsized': '2024-03-06T07:26:19.226570+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:19.211880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:26:24.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:26:25.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.117+0000 7fe1d9ae1640 1 Processor -- start 2024-03-06T07:26:25.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.117+0000 7fe1d9ae1640 1 -- start start 2024-03-06T07:26:25.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d9ae1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1d4071540 0x7fe1d40b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d9ae1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe1d40b2b50 0x7fe1d40b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d9ae1640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d40ce610 con 0x7fe1d40d3400 2024-03-06T07:26:25.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d9ae1640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d40d2f70 con 0x7fe1d4071540 2024-03-06T07:26:25.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d9ae1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d40d4be0 con 0x7fe1d40b2b50 2024-03-06T07:26:25.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d37fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1d4071540 0x7fe1d40b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d37fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1d4071540 0x7fe1d40b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47462/0 (socket says 172.21.15.27:47462) 2024-03-06T07:26:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d2ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe1d40d3400 0x7fe1d4071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47028/0 (socket says 172.21.15.27:47028) 2024-03-06T07:26:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d37fe640 1 -- 172.21.15.27:0/2876668725 learned_addr learned my addr 172.21.15.27:0/2876668725 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d3fff640 1 --2- 172.21.15.27:0/2876668725 >> v2:172.21.15.69:3300/0 conn(0x7fe1d40b2b50 0x7fe1d40b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d37fe640 1 -- 172.21.15.27:0/2876668725 >> v2:172.21.15.69:3300/0 conn(0x7fe1d40b2b50 msgr2=0x7fe1d40b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d37fe640 1 --2- 172.21.15.27:0/2876668725 >> v2:172.21.15.69:3300/0 conn(0x7fe1d40b2b50 0x7fe1d40b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d37fe640 1 -- 172.21.15.27:0/2876668725 >> v1:172.21.15.27:6789/0 conn(0x7fe1d40d3400 legacy=0x7fe1d4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d37fe640 1 -- 172.21.15.27:0/2876668725 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d40d2920 con 0x7fe1d4071540 2024-03-06T07:26:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d27fc640 1 -- 172.21.15.27:0/2876668725 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156971312 0 0) 0x7fe1d40ce610 con 0x7fe1d40d3400 2024-03-06T07:26:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.118+0000 7fe1d3fff640 1 --2- 172.21.15.27:0/2876668725 >> v2:172.21.15.69:3300/0 conn(0x7fe1d40b2b50 0x7fe1d40b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:25.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.119+0000 7fe1d37fe640 1 --2- 172.21.15.27:0/2876668725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 0x7fe1d40b30a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40683b0 tx=0x7fe1bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168576a94e4d6240 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d27fc640 1 -- 172.21.15.27:0/2876668725 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc034070 con 0x7fe1d4071540 2024-03-06T07:26:25.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d27fc640 1 -- 172.21.15.27:0/2876668725 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc030860 con 0x7fe1d4071540 2024-03-06T07:26:25.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d27fc640 1 -- 172.21.15.27:0/2876668725 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc030c30 con 0x7fe1d4071540 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/2876668725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 msgr2=0x7fe1d40b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d9ae1640 1 --2- 172.21.15.27:0/2876668725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 0x7fe1d40b30a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40683b0 tx=0x7fe1bc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/2876668725 shutdown_connections 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d9ae1640 1 --2- 172.21.15.27:0/2876668725 >> v2:172.21.15.69:3300/0 conn(0x7fe1d40b2b50 0x7fe1d40b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d9ae1640 1 --2- 172.21.15.27:0/2876668725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 0x7fe1d40b30a0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/2876668725 >> 172.21.15.27:0/2876668725 conn(0x7fe1d40d3ca0 msgr2=0x7fe1d40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/2876668725 shutdown_connections 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.120+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/2876668725 wait complete. 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.121+0000 7fe1d9ae1640 1 Processor -- start 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d9ae1640 1 -- start start 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d9ae1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 0x7fe1d40b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d9ae1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1d40b2b50 0x7fe1d41f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d9ae1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1d40d3400 0x7fe1d41f7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d9ae1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d415e370 con 0x7fe1d4071540 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d9ae1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d40ce8f0 con 0x7fe1d40d3400 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d9ae1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d40ab840 con 0x7fe1d40b2b50 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 0x7fe1d40b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 0x7fe1d40b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47466/0 (socket says 172.21.15.27:47466) 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.122+0000 7fe1d37fe640 1 -- 172.21.15.27:0/3821617127 learned_addr learned my addr 172.21.15.27:0/3821617127 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d3fff640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1d40d3400 0x7fe1d41f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d2ffd640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1d40b2b50 0x7fe1d41f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d37fe640 1 -- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1d40b2b50 msgr2=0x7fe1d41f72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d37fe640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1d40b2b50 0x7fe1d41f72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d37fe640 1 -- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1d40d3400 msgr2=0x7fe1d41f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d37fe640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1d40d3400 0x7fe1d41f7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d37fe640 1 -- 172.21.15.27:0/3821617127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d40abae0 con 0x7fe1d4071540 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d37fe640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 0x7fe1d40b1f30 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0090b0 tx=0x7fe1bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1cbfff640 1 -- 172.21.15.27:0/3821617127 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc034040 con 0x7fe1d4071540 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1cbfff640 1 -- 172.21.15.27:0/3821617127 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc030660 con 0x7fe1d4071540 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d40abdc0 con 0x7fe1d4071540 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1cbfff640 1 -- 172.21.15.27:0/3821617127 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc039d20 con 0x7fe1d4071540 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.123+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d40ac380 con 0x7fe1d4071540 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.124+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d415fcd0 con 0x7fe1d4071540 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.125+0000 7fe1cbfff640 1 -- 172.21.15.27:0/3821617127 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1bc04b060 con 0x7fe1d4071540 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.125+0000 7fe1cbfff640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1b406b4d0 0x7fe1b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.125+0000 7fe1cbfff640 1 -- 172.21.15.27:0/3821617127 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1bc0abf40 con 0x7fe1d4071540 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.125+0000 7fe1d2ffd640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1b406b4d0 0x7fe1b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.126+0000 7fe1d2ffd640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1b406b4d0 0x7fe1b406d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40ca740 tx=0x7fe1c4007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.134+0000 7fe1cbfff640 1 -- 172.21.15.27:0/3821617127 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1d415fcd0 con 0x7fe1d4071540 2024-03-06T07:26:25.184 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:26:25.183+0000 7f165a49d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:25.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.384+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fe1d4068700 con 0x7fe1d4071540 2024-03-06T07:26:25.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.385+0000 7fe1cbfff640 1 -- 172.21.15.27:0/3821617127 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7fe1bc07adc0 con 0x7fe1d4071540 2024-03-06T07:26:25.386 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:25.386 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":241,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":70,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709754,"num_in_osds":8,"osd_in_since":1709709745,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":263225344,"bytes_avail":772830887936,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":44,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":6606}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-06T07:26:02.032664+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:26:25.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1b406b4d0 msgr2=0x7fe1b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1b406b4d0 0x7fe1b406d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40ca740 tx=0x7fe1c4007c00 comp rx=0 tx=0).stop 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 msgr2=0x7fe1d40b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 0x7fe1d40b1f30 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0090b0 tx=0x7fe1bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 shutdown_connections 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1b406b4d0 0x7fe1b406d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1d40d3400 0x7fe1d41f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1d40b2b50 0x7fe1d41f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 --2- 172.21.15.27:0/3821617127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1d4071540 0x7fe1d40b1f30 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 >> 172.21.15.27:0/3821617127 conn(0x7fe1d40d3ca0 msgr2=0x7fe1d40d4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 shutdown_connections 2024-03-06T07:26:25.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.387+0000 7fe1d9ae1640 1 -- 172.21.15.27:0/3821617127 wait complete. 2024-03-06T07:26:25.399 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:26:25.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.504+0000 7f8c84181640 1 Processor -- start 2024-03-06T07:26:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.505+0000 7f8c84181640 1 -- start start 2024-03-06T07:26:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.505+0000 7f8c84181640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8c7c158520 0x7f8c7c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.505+0000 7f8c84181640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c7c0aee10 0x7f8c7c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.505+0000 7f8c84181640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c071240 con 0x7f8c7c06dd60 2024-03-06T07:26:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c84181640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c071b70 con 0x7f8c7c0aee10 2024-03-06T07:26:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c84181640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c071520 con 0x7f8c7c158520 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c8271e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c7c0aee10 0x7f8c7c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c8271e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c7c0aee10 0x7f8c7c0b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47472/0 (socket says 172.21.15.27:47472) 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c8271e640 1 -- 172.21.15.27:0/3055389164 learned_addr learned my addr 172.21.15.27:0/3055389164 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c81f1d640 1 --2- 172.21.15.27:0/3055389164 >> v2:172.21.15.69:3300/0 conn(0x7f8c7c158520 0x7f8c7c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c8271e640 1 -- 172.21.15.27:0/3055389164 >> v2:172.21.15.69:3300/0 conn(0x7f8c7c158520 msgr2=0x7f8c7c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c8271e640 1 --2- 172.21.15.27:0/3055389164 >> v2:172.21.15.69:3300/0 conn(0x7f8c7c158520 0x7f8c7c06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c8271e640 1 -- 172.21.15.27:0/3055389164 >> v1:172.21.15.69:6789/0 conn(0x7f8c7c06dd60 legacy=0x7f8c7c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.506+0000 7f8c8271e640 1 -- 172.21.15.27:0/3055389164 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c071800 con 0x7f8c7c0aee10 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.507+0000 7f8c8271e640 1 --2- 172.21.15.27:0/3055389164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0aee10 0x7f8c7c0b3640 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c06e900 tx=0x7f8c6c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64ca485ca5a96dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.507+0000 7f8c80f1b640 1 -- 172.21.15.27:0/3055389164 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c02ca80 con 0x7f8c7c0aee10 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.507+0000 7f8c80f1b640 1 -- 172.21.15.27:0/3055389164 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c02cc20 con 0x7f8c7c0aee10 2024-03-06T07:26:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.507+0000 7f8c80f1b640 1 -- 172.21.15.27:0/3055389164 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c0343f0 con 0x7f8c7c0aee10 2024-03-06T07:26:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.507+0000 7f8c84181640 1 -- 172.21.15.27:0/3055389164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0aee10 msgr2=0x7f8c7c0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.507+0000 7f8c84181640 1 --2- 172.21.15.27:0/3055389164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0aee10 0x7f8c7c0b3640 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c06e900 tx=0x7f8c6c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.507+0000 7f8c84181640 1 -- 172.21.15.27:0/3055389164 shutdown_connections 2024-03-06T07:26:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.507+0000 7f8c84181640 1 --2- 172.21.15.27:0/3055389164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0aee10 0x7f8c7c0b3640 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.508+0000 7f8c84181640 1 --2- 172.21.15.27:0/3055389164 >> v2:172.21.15.69:3300/0 conn(0x7f8c7c158520 0x7f8c7c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.508+0000 7f8c84181640 1 -- 172.21.15.27:0/3055389164 >> 172.21.15.27:0/3055389164 conn(0x7f8c7c066d00 msgr2=0x7f8c7c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.508+0000 7f8c84181640 1 -- 172.21.15.27:0/3055389164 shutdown_connections 2024-03-06T07:26:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.508+0000 7f8c84181640 1 -- 172.21.15.27:0/3055389164 wait complete. 2024-03-06T07:26:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.509+0000 7f8c84181640 1 Processor -- start 2024-03-06T07:26:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.509+0000 7f8c84181640 1 -- start start 2024-03-06T07:26:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.509+0000 7f8c84181640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c06dd60 0x7f8c7c0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.509+0000 7f8c84181640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c7c0aee10 0x7f8c7c15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.509+0000 7f8c84181640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c7c158520 0x7f8c7c15d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c84181640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c158a80 con 0x7f8c7c06dd60 2024-03-06T07:26:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c84181640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c158900 con 0x7f8c7c158520 2024-03-06T07:26:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c84181640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0b3b80 con 0x7f8c7c0aee10 2024-03-06T07:26:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c81f1d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c06dd60 0x7f8c7c0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c81f1d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c06dd60 0x7f8c7c0ccb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47478/0 (socket says 172.21.15.27:47478) 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c81f1d640 1 -- 172.21.15.27:0/1849454453 learned_addr learned my addr 172.21.15.27:0/1849454453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c8271e640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c7c158520 0x7f8c7c15d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c8171c640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c7c0aee10 0x7f8c7c15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c81f1d640 1 -- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c7c0aee10 msgr2=0x7f8c7c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c81f1d640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c7c0aee10 0x7f8c7c15d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c81f1d640 1 -- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c7c158520 msgr2=0x7f8c7c15d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c81f1d640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c7c158520 0x7f8c7c15d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c81f1d640 1 -- 172.21.15.27:0/1849454453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0cd1d0 con 0x7f8c7c06dd60 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c81f1d640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c06dd60 0x7f8c7c0ccb40 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400c820 tx=0x7f8c7400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c7a7fc640 1 -- 172.21.15.27:0/1849454453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c74015070 con 0x7f8c7c06dd60 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c7a7fc640 1 -- 172.21.15.27:0/1849454453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c74004020 con 0x7f8c7c06dd60 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c0cec00 con 0x7f8c7c06dd60 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c7a7fc640 1 -- 172.21.15.27:0/1849454453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c740043f0 con 0x7f8c7c06dd60 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.510+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c0cf1c0 con 0x7f8c7c06dd60 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.511+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c0cf3c0 con 0x7f8c7c06dd60 2024-03-06T07:26:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.512+0000 7f8c7a7fc640 1 -- 172.21.15.27:0/1849454453 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c74007830 con 0x7f8c7c06dd60 2024-03-06T07:26:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.512+0000 7f8c7a7fc640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6806b4d0 0x7f8c6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.512+0000 7f8c7a7fc640 1 -- 172.21.15.27:0/1849454453 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c74089fe0 con 0x7f8c7c06dd60 2024-03-06T07:26:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.513+0000 7f8c8171c640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6806b4d0 0x7f8c6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.513+0000 7f8c8171c640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6806b4d0 0x7f8c6806d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c159460 tx=0x7f8c70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.517+0000 7f8c7a7fc640 1 -- 172.21.15.27:0/1849454453 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c7c0cf3c0 con 0x7f8c7c06dd60 2024-03-06T07:26:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.695+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c7c067ff0 con 0x7f8c6806b4d0 2024-03-06T07:26:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.702+0000 7f8c7a7fc640 1 -- 172.21.15.27:0/1849454453 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246754 (secure 0 0 0) 0x7f8c7c067ff0 con 0x7f8c6806b4d0 2024-03-06T07:26:25.705 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:25.706 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:26:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6806b4d0 msgr2=0x7f8c6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6806b4d0 0x7f8c6806d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c159460 tx=0x7f8c70009040 comp rx=0 tx=0).stop 2024-03-06T07:26:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c06dd60 msgr2=0x7f8c7c0ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c06dd60 0x7f8c7c0ccb40 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400c820 tx=0x7f8c7400cce0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 shutdown_connections 2024-03-06T07:26:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8c6806b4d0 0x7f8c6806d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c7c158520 0x7f8c7c15d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c7c0aee10 0x7f8c7c15d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 --2- 172.21.15.27:0/1849454453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c06dd60 0x7f8c7c0ccb40 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 >> 172.21.15.27:0/1849454453 conn(0x7f8c7c066d00 msgr2=0x7f8c7c15a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 shutdown_connections 2024-03-06T07:26:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:25.704+0000 7f8c84181640 1 -- 172.21.15.27:0/1849454453 wait complete. 2024-03-06T07:26:25.717 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-03-06T07:26:24.038057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257056,"kb_used_data":7600,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717664,"statfs":{"total":773094113280,"available":772830887936,"internally_reserved":0,"allocated":7782400,"data_stored":3918114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.825797"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227981+0000","last_change":"2024-03-06T07:26:19.227981+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227981+0000","last_undegraded":"2024-03-06T07:26:19.227981+0000","last_fullsized":"2024-03-06T07:26:19.227981+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228347+0000","last_change":"2024-03-06T07:26:19.228347+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228347+0000","last_undegraded":"2024-03-06T07:26:19.228347+0000","last_fullsized":"2024-03-06T07:26:19.228347+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228671+0000","last_change":"2024-03-06T07:26:19.228671+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228671+0000","last_undegraded":"2024-03-06T07:26:19.228671+0000","last_fullsized":"2024-03-06T07:26:19.228671+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224081+0000","last_change":"2024-03-06T07:26:19.224081+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224081+0000","last_undegraded":"2024-03-06T07:26:19.224081+0000","last_fullsized":"2024-03-06T07:26:19.224081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227451+0000","last_change":"2024-03-06T07:26:19.227451+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227451+0000","last_undegraded":"2024-03-06T07:26:19.227451+0000","last_fullsized":"2024-03-06T07:26:19.227451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228991+0000","last_change":"2024-03-06T07:26:19.228991+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228991+0000","last_undegraded":"2024-03-06T07:26:19.228991+0000","last_fullsized":"2024-03-06T07:26:19.228991+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231923+0000","last_change":"2024-03-06T07:26:19.231923+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231923+0000","last_undegraded":"2024-03-06T07:26:19.231923+0000","last_fullsized":"2024-03-06T07:26:19.231923+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228094+0000","last_change":"2024-03-06T07:26:19.228094+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228094+0000","last_undegraded":"2024-03-06T07:26:19.228094+0000","last_fullsized":"2024-03-06T07:26:19.228094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226958+0000","last_change":"2024-03-06T07:26:19.226958+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226958+0000","last_undegraded":"2024-03-06T07:26:19.226958+0000","last_fullsized":"2024-03-06T07:26:19.226958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224802+0000","last_change":"2024-03-06T07:26:19.224802+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224802+0000","last_undegraded":"2024-03-06T07:26:19.224802+0000","last_fullsized":"2024-03-06T07:26:19.224802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231680+0000","last_change":"2024-03-06T07:26:19.231680+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231680+0000","last_undegraded":"2024-03-06T07:26:19.231680+0000","last_fullsized":"2024-03-06T07:26:19.231680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223377+0000","last_change":"2024-03-06T07:26:19.223377+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223377+0000","last_undegraded":"2024-03-06T07:26:19.223377+0000","last_fullsized":"2024-03-06T07:26:19.223377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221736+0000","last_change":"2024-03-06T07:26:19.221736+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221736+0000","last_undegraded":"2024-03-06T07:26:19.221736+0000","last_fullsized":"2024-03-06T07:26:19.221736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226495+0000","last_change":"2024-03-06T07:26:19.226495+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226495+0000","last_undegraded":"2024-03-06T07:26:19.226495+0000","last_fullsized":"2024-03-06T07:26:19.226495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222226+0000","last_change":"2024-03-06T07:26:19.222226+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222226+0000","last_undegraded":"2024-03-06T07:26:19.222226+0000","last_fullsized":"2024-03-06T07:26:19.222226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224253+0000","last_change":"2024-03-06T07:26:19.224253+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224253+0000","last_undegraded":"2024-03-06T07:26:19.224253+0000","last_fullsized":"2024-03-06T07:26:19.224253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225964+0000","last_change":"2024-03-06T07:26:19.225964+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225964+0000","last_undegraded":"2024-03-06T07:26:19.225964+0000","last_fullsized":"2024-03-06T07:26:19.225964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221558+0000","last_change":"2024-03-06T07:26:19.221558+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221558+0000","last_undegraded":"2024-03-06T07:26:19.221558+0000","last_fullsized":"2024-03-06T07:26:19.221558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228445+0000","last_change":"2024-03-06T07:26:19.228445+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228445+0000","last_undegraded":"2024-03-06T07:26:19.228445+0000","last_fullsized":"2024-03-06T07:26:19.228445+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223388+0000","last_change":"2024-03-06T07:26:19.223388+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223388+0000","last_undegraded":"2024-03-06T07:26:19.223388+0000","last_fullsized":"2024-03-06T07:26:19.223388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223534+0000","last_change":"2024-03-06T07:26:19.223534+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223534+0000","last_undegraded":"2024-03-06T07:26:19.223534+0000","last_fullsized":"2024-03-06T07:26:19.223534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223910+0000","last_change":"2024-03-06T07:26:19.223910+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223910+0000","last_undegraded":"2024-03-06T07:26:19.223910+0000","last_fullsized":"2024-03-06T07:26:19.223910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222903+0000","last_change":"2024-03-06T07:26:19.222903+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222903+0000","last_undegraded":"2024-03-06T07:26:19.222903+0000","last_fullsized":"2024-03-06T07:26:19.222903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224000+0000","last_change":"2024-03-06T07:26:19.224000+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224000+0000","last_undegraded":"2024-03-06T07:26:19.224000+0000","last_fullsized":"2024-03-06T07:26:19.224000+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222350+0000","last_change":"2024-03-06T07:26:19.222350+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222350+0000","last_undegraded":"2024-03-06T07:26:19.222350+0000","last_fullsized":"2024-03-06T07:26:19.222350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229681+0000","last_change":"2024-03-06T07:26:19.229681+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229681+0000","last_undegraded":"2024-03-06T07:26:19.229681+0000","last_fullsized":"2024-03-06T07:26:19.229681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228249+0000","last_change":"2024-03-06T07:26:19.228249+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228249+0000","last_undegraded":"2024-03-06T07:26:19.228249+0000","last_fullsized":"2024-03-06T07:26:19.228249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224326+0000","last_change":"2024-03-06T07:26:19.224326+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224326+0000","last_undegraded":"2024-03-06T07:26:19.224326+0000","last_fullsized":"2024-03-06T07:26:19.224326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228898+0000","last_change":"2024-03-06T07:26:19.228898+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228898+0000","last_undegraded":"2024-03-06T07:26:19.228898+0000","last_fullsized":"2024-03-06T07:26:19.228898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231473+0000","last_change":"2024-03-06T07:26:19.231473+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231473+0000","last_undegraded":"2024-03-06T07:26:19.231473+0000","last_fullsized":"2024-03-06T07:26:19.231473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227762+0000","last_change":"2024-03-06T07:26:19.227762+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227762+0000","last_undegraded":"2024-03-06T07:26:19.227762+0000","last_fullsized":"2024-03-06T07:26:19.227762+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227670+0000","last_change":"2024-03-06T07:26:19.227670+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227670+0000","last_undegraded":"2024-03-06T07:26:19.227670+0000","last_fullsized":"2024-03-06T07:26:19.227670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225593+0000","last_change":"2024-03-06T07:26:19.225593+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225593+0000","last_undegraded":"2024-03-06T07:26:19.225593+0000","last_fullsized":"2024-03-06T07:26:19.225593+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239388+0000","last_change":"2024-03-06T07:26:19.235249+0000","last_active":"2024-03-06T07:26:19.239388+0000","last_peered":"2024-03-06T07:26:19.239388+0000","last_clean":"2024-03-06T07:26:19.239388+0000","last_became_active":"2024-03-06T07:26:19.234904+0000","last_became_peered":"2024-03-06T07:26:19.234904+0000","last_unstale":"2024-03-06T07:26:19.239388+0000","last_undegraded":"2024-03-06T07:26:19.239388+0000","last_fullsized":"2024-03-06T07:26:19.239388+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:23.463441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225476+0000","last_change":"2024-03-06T07:26:19.225476+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225476+0000","last_undegraded":"2024-03-06T07:26:19.225476+0000","last_fullsized":"2024-03-06T07:26:19.225476+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.233898+0000","last_change":"2024-03-06T07:26:19.225962+0000","last_active":"2024-03-06T07:26:19.233898+0000","last_peered":"2024-03-06T07:26:19.233898+0000","last_clean":"2024-03-06T07:26:19.233898+0000","last_became_active":"2024-03-06T07:26:19.225707+0000","last_became_peered":"2024-03-06T07:26:19.225707+0000","last_unstale":"2024-03-06T07:26:19.233898+0000","last_undegraded":"2024-03-06T07:26:19.233898+0000","last_fullsized":"2024-03-06T07:26:19.233898+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:50.887101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228771+0000","last_change":"2024-03-06T07:26:19.228771+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228771+0000","last_undegraded":"2024-03-06T07:26:19.228771+0000","last_fullsized":"2024-03-06T07:26:19.228771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235050+0000","last_change":"2024-03-06T07:26:19.232800+0000","last_active":"2024-03-06T07:26:19.235050+0000","last_peered":"2024-03-06T07:26:19.235050+0000","last_clean":"2024-03-06T07:26:19.235050+0000","last_became_active":"2024-03-06T07:26:19.232541+0000","last_became_peered":"2024-03-06T07:26:19.232541+0000","last_unstale":"2024-03-06T07:26:19.235050+0000","last_undegraded":"2024-03-06T07:26:19.235050+0000","last_fullsized":"2024-03-06T07:26:19.235050+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:49.533898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.230992+0000","last_change":"2024-03-06T07:26:19.230992+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.230992+0000","last_undegraded":"2024-03-06T07:26:19.230992+0000","last_fullsized":"2024-03-06T07:26:19.230992+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239827+0000","last_change":"2024-03-06T07:26:19.237329+0000","last_active":"2024-03-06T07:26:19.239827+0000","last_peered":"2024-03-06T07:26:19.239827+0000","last_clean":"2024-03-06T07:26:19.239827+0000","last_became_active":"2024-03-06T07:26:19.237096+0000","last_became_peered":"2024-03-06T07:26:19.237096+0000","last_unstale":"2024-03-06T07:26:19.239827+0000","last_undegraded":"2024-03-06T07:26:19.239827+0000","last_fullsized":"2024-03-06T07:26:19.239827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:54.603601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.231064+0000","last_change":"2024-03-06T07:26:19.231064+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.231064+0000","last_undegraded":"2024-03-06T07:26:19.231064+0000","last_fullsized":"2024-03-06T07:26:19.231064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.237618+0000","last_change":"2024-03-06T07:26:19.236188+0000","last_active":"2024-03-06T07:26:19.237618+0000","last_peered":"2024-03-06T07:26:19.237618+0000","last_clean":"2024-03-06T07:26:19.237618+0000","last_became_active":"2024-03-06T07:26:19.235921+0000","last_became_peered":"2024-03-06T07:26:19.235921+0000","last_unstale":"2024-03-06T07:26:19.237618+0000","last_undegraded":"2024-03-06T07:26:19.237618+0000","last_fullsized":"2024-03-06T07:26:19.237618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:13.224175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222902+0000","last_change":"2024-03-06T07:26:19.222902+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222902+0000","last_undegraded":"2024-03-06T07:26:19.222902+0000","last_fullsized":"2024-03-06T07:26:19.222902+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242242+0000","last_change":"2024-03-06T07:26:19.237984+0000","last_active":"2024-03-06T07:26:19.242242+0000","last_peered":"2024-03-06T07:26:19.242242+0000","last_clean":"2024-03-06T07:26:19.242242+0000","last_became_active":"2024-03-06T07:26:19.237697+0000","last_became_peered":"2024-03-06T07:26:19.237697+0000","last_unstale":"2024-03-06T07:26:19.242242+0000","last_undegraded":"2024-03-06T07:26:19.242242+0000","last_fullsized":"2024-03-06T07:26:19.242242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:21.157010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225590+0000","last_change":"2024-03-06T07:26:19.225590+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225590+0000","last_undegraded":"2024-03-06T07:26:19.225590+0000","last_fullsized":"2024-03-06T07:26:19.225590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.232619+0000","last_change":"2024-03-06T07:26:19.228888+0000","last_active":"2024-03-06T07:26:19.232619+0000","last_peered":"2024-03-06T07:26:19.232619+0000","last_clean":"2024-03-06T07:26:19.232619+0000","last_became_active":"2024-03-06T07:26:19.228602+0000","last_became_peered":"2024-03-06T07:26:19.228602+0000","last_unstale":"2024-03-06T07:26:19.232619+0000","last_undegraded":"2024-03-06T07:26:19.232619+0000","last_fullsized":"2024-03-06T07:26:19.232619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:55.021248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221713+0000","last_change":"2024-03-06T07:26:19.221713+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221713+0000","last_undegraded":"2024-03-06T07:26:19.221713+0000","last_fullsized":"2024-03-06T07:26:19.221713+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.234079+0000","last_change":"2024-03-06T07:26:19.231771+0000","last_active":"2024-03-06T07:26:19.234079+0000","last_peered":"2024-03-06T07:26:19.234079+0000","last_clean":"2024-03-06T07:26:19.234079+0000","last_became_active":"2024-03-06T07:26:19.231485+0000","last_became_peered":"2024-03-06T07:26:19.231485+0000","last_unstale":"2024-03-06T07:26:19.234079+0000","last_undegraded":"2024-03-06T07:26:19.234079+0000","last_fullsized":"2024-03-06T07:26:19.234079+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:30.451695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229259+0000","last_change":"2024-03-06T07:26:19.229259+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229259+0000","last_undegraded":"2024-03-06T07:26:19.229259+0000","last_fullsized":"2024-03-06T07:26:19.229259+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.230928+0000","last_change":"2024-03-06T07:26:19.229136+0000","last_active":"2024-03-06T07:26:19.230928+0000","last_peered":"2024-03-06T07:26:19.230928+0000","last_clean":"2024-03-06T07:26:19.230928+0000","last_became_active":"2024-03-06T07:26:19.228661+0000","last_became_peered":"2024-03-06T07:26:19.228661+0000","last_unstale":"2024-03-06T07:26:19.230928+0000","last_undegraded":"2024-03-06T07:26:19.230928+0000","last_fullsized":"2024-03-06T07:26:19.230928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:56.368954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225377+0000","last_change":"2024-03-06T07:26:19.225377+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225377+0000","last_undegraded":"2024-03-06T07:26:19.225377+0000","last_fullsized":"2024-03-06T07:26:19.225377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239800+0000","last_change":"2024-03-06T07:26:19.236797+0000","last_active":"2024-03-06T07:26:19.239800+0000","last_peered":"2024-03-06T07:26:19.239800+0000","last_clean":"2024-03-06T07:26:19.239800+0000","last_became_active":"2024-03-06T07:26:19.236118+0000","last_became_peered":"2024-03-06T07:26:19.236118+0000","last_unstale":"2024-03-06T07:26:19.239800+0000","last_undegraded":"2024-03-06T07:26:19.239800+0000","last_fullsized":"2024-03-06T07:26:19.239800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:04.139082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228026+0000","last_change":"2024-03-06T07:26:19.228026+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228026+0000","last_undegraded":"2024-03-06T07:26:19.228026+0000","last_fullsized":"2024-03-06T07:26:19.228026+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.236116+0000","last_change":"2024-03-06T07:26:19.235158+0000","last_active":"2024-03-06T07:26:19.236116+0000","last_peered":"2024-03-06T07:26:19.236116+0000","last_clean":"2024-03-06T07:26:19.236116+0000","last_became_active":"2024-03-06T07:26:19.234883+0000","last_became_peered":"2024-03-06T07:26:19.234883+0000","last_unstale":"2024-03-06T07:26:19.236116+0000","last_undegraded":"2024-03-06T07:26:19.236116+0000","last_fullsized":"2024-03-06T07:26:19.236116+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:17.059043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225110+0000","last_change":"2024-03-06T07:26:19.225110+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225110+0000","last_undegraded":"2024-03-06T07:26:19.225110+0000","last_fullsized":"2024-03-06T07:26:19.225110+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.238717+0000","last_change":"2024-03-06T07:26:19.236129+0000","last_active":"2024-03-06T07:26:19.238717+0000","last_peered":"2024-03-06T07:26:19.238717+0000","last_clean":"2024-03-06T07:26:19.238717+0000","last_became_active":"2024-03-06T07:26:19.235844+0000","last_became_peered":"2024-03-06T07:26:19.235844+0000","last_unstale":"2024-03-06T07:26:19.238717+0000","last_undegraded":"2024-03-06T07:26:19.238717+0000","last_fullsized":"2024-03-06T07:26:19.238717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:04.139082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223551+0000","last_change":"2024-03-06T07:26:19.223551+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223551+0000","last_undegraded":"2024-03-06T07:26:19.223551+0000","last_fullsized":"2024-03-06T07:26:19.223551+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.228861+0000","last_change":"2024-03-06T07:26:19.226330+0000","last_active":"2024-03-06T07:26:19.228861+0000","last_peered":"2024-03-06T07:26:19.228861+0000","last_clean":"2024-03-06T07:26:19.228861+0000","last_became_active":"2024-03-06T07:26:19.226062+0000","last_became_peered":"2024-03-06T07:26:19.226062+0000","last_unstale":"2024-03-06T07:26:19.228861+0000","last_undegraded":"2024-03-06T07:26:19.228861+0000","last_fullsized":"2024-03-06T07:26:19.228861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:29.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.230238+0000","last_change":"2024-03-06T07:26:19.230238+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.230238+0000","last_undegraded":"2024-03-06T07:26:19.230238+0000","last_fullsized":"2024-03-06T07:26:19.230238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242264+0000","last_change":"2024-03-06T07:26:19.238616+0000","last_active":"2024-03-06T07:26:19.242264+0000","last_peered":"2024-03-06T07:26:19.242264+0000","last_clean":"2024-03-06T07:26:19.242264+0000","last_became_active":"2024-03-06T07:26:19.238337+0000","last_became_peered":"2024-03-06T07:26:19.238337+0000","last_unstale":"2024-03-06T07:26:19.242264+0000","last_undegraded":"2024-03-06T07:26:19.242264+0000","last_fullsized":"2024-03-06T07:26:19.242264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:14.382999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222417+0000","last_change":"2024-03-06T07:26:19.222417+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222417+0000","last_undegraded":"2024-03-06T07:26:19.222417+0000","last_fullsized":"2024-03-06T07:26:19.222417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239417+0000","last_change":"2024-03-06T07:26:19.236852+0000","last_active":"2024-03-06T07:26:19.239417+0000","last_peered":"2024-03-06T07:26:19.239417+0000","last_clean":"2024-03-06T07:26:19.239417+0000","last_became_active":"2024-03-06T07:26:19.236469+0000","last_became_peered":"2024-03-06T07:26:19.236469+0000","last_unstale":"2024-03-06T07:26:19.239417+0000","last_undegraded":"2024-03-06T07:26:19.239417+0000","last_fullsized":"2024-03-06T07:26:19.239417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:20.767924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224910+0000","last_change":"2024-03-06T07:26:19.224910+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224910+0000","last_undegraded":"2024-03-06T07:26:19.224910+0000","last_fullsized":"2024-03-06T07:26:19.224910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235094+0000","last_change":"2024-03-06T07:26:19.231717+0000","last_active":"2024-03-06T07:26:19.235094+0000","last_peered":"2024-03-06T07:26:19.235094+0000","last_clean":"2024-03-06T07:26:19.235094+0000","last_became_active":"2024-03-06T07:26:19.231462+0000","last_became_peered":"2024-03-06T07:26:19.231462+0000","last_unstale":"2024-03-06T07:26:19.235094+0000","last_undegraded":"2024-03-06T07:26:19.235094+0000","last_fullsized":"2024-03-06T07:26:19.235094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.795497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.223271+0000","last_change":"2024-03-06T07:26:19.223271+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.223271+0000","last_undegraded":"2024-03-06T07:26:19.223271+0000","last_fullsized":"2024-03-06T07:26:19.223271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239822+0000","last_change":"2024-03-06T07:26:19.237309+0000","last_active":"2024-03-06T07:26:19.239822+0000","last_peered":"2024-03-06T07:26:19.239822+0000","last_clean":"2024-03-06T07:26:19.239822+0000","last_became_active":"2024-03-06T07:26:19.237011+0000","last_became_peered":"2024-03-06T07:26:19.237011+0000","last_unstale":"2024-03-06T07:26:19.239822+0000","last_undegraded":"2024-03-06T07:26:19.239822+0000","last_fullsized":"2024-03-06T07:26:19.239822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:58.527722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224977+0000","last_change":"2024-03-06T07:26:19.224977+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224977+0000","last_undegraded":"2024-03-06T07:26:19.224977+0000","last_fullsized":"2024-03-06T07:26:19.224977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235073+0000","last_change":"2024-03-06T07:26:19.233492+0000","last_active":"2024-03-06T07:26:19.235073+0000","last_peered":"2024-03-06T07:26:19.235073+0000","last_clean":"2024-03-06T07:26:19.235073+0000","last_became_active":"2024-03-06T07:26:19.233199+0000","last_became_peered":"2024-03-06T07:26:19.233199+0000","last_unstale":"2024-03-06T07:26:19.235073+0000","last_undegraded":"2024-03-06T07:26:19.235073+0000","last_fullsized":"2024-03-06T07:26:19.235073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:22.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222478+0000","last_change":"2024-03-06T07:26:19.222478+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222478+0000","last_undegraded":"2024-03-06T07:26:19.222478+0000","last_fullsized":"2024-03-06T07:26:19.222478+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239765+0000","last_change":"2024-03-06T07:26:19.236432+0000","last_active":"2024-03-06T07:26:19.239765+0000","last_peered":"2024-03-06T07:26:19.239765+0000","last_clean":"2024-03-06T07:26:19.239765+0000","last_became_active":"2024-03-06T07:26:19.235975+0000","last_became_peered":"2024-03-06T07:26:19.235975+0000","last_unstale":"2024-03-06T07:26:19.239765+0000","last_undegraded":"2024-03-06T07:26:19.239765+0000","last_fullsized":"2024-03-06T07:26:19.239765+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:55.021248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.224496+0000","last_change":"2024-03-06T07:26:19.224496+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.224496+0000","last_undegraded":"2024-03-06T07:26:19.224496+0000","last_fullsized":"2024-03-06T07:26:19.224496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.234690+0000","last_change":"2024-03-06T07:26:19.233564+0000","last_active":"2024-03-06T07:26:19.234690+0000","last_peered":"2024-03-06T07:26:19.234690+0000","last_clean":"2024-03-06T07:26:19.234690+0000","last_became_active":"2024-03-06T07:26:19.233235+0000","last_became_peered":"2024-03-06T07:26:19.233235+0000","last_unstale":"2024-03-06T07:26:19.234690+0000","last_undegraded":"2024-03-06T07:26:19.234690+0000","last_fullsized":"2024-03-06T07:26:19.234690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:44.732093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.228500+0000","last_change":"2024-03-06T07:26:19.228500+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.228500+0000","last_undegraded":"2024-03-06T07:26:19.228500+0000","last_fullsized":"2024-03-06T07:26:19.228500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239745+0000","last_change":"2024-03-06T07:26:19.236405+0000","last_active":"2024-03-06T07:26:19.239745+0000","last_peered":"2024-03-06T07:26:19.239745+0000","last_clean":"2024-03-06T07:26:19.239745+0000","last_became_active":"2024-03-06T07:26:19.235876+0000","last_became_peered":"2024-03-06T07:26:19.235876+0000","last_unstale":"2024-03-06T07:26:19.239745+0000","last_undegraded":"2024-03-06T07:26:19.239745+0000","last_fullsized":"2024-03-06T07:26:19.239745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:29.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225680+0000","last_change":"2024-03-06T07:26:19.225680+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225680+0000","last_undegraded":"2024-03-06T07:26:19.225680+0000","last_fullsized":"2024-03-06T07:26:19.225680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242212+0000","last_change":"2024-03-06T07:26:19.238646+0000","last_active":"2024-03-06T07:26:19.242212+0000","last_peered":"2024-03-06T07:26:19.242212+0000","last_clean":"2024-03-06T07:26:19.242212+0000","last_became_active":"2024-03-06T07:26:19.238376+0000","last_became_peered":"2024-03-06T07:26:19.238376+0000","last_unstale":"2024-03-06T07:26:19.242212+0000","last_undegraded":"2024-03-06T07:26:19.242212+0000","last_fullsized":"2024-03-06T07:26:19.242212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:33.080991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225633+0000","last_change":"2024-03-06T07:26:19.225633+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225633+0000","last_undegraded":"2024-03-06T07:26:19.225633+0000","last_fullsized":"2024-03-06T07:26:19.225633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.233488+0000","last_change":"2024-03-06T07:26:19.227589+0000","last_active":"2024-03-06T07:26:19.233488+0000","last_peered":"2024-03-06T07:26:19.233488+0000","last_clean":"2024-03-06T07:26:19.233488+0000","last_became_active":"2024-03-06T07:26:19.227308+0000","last_became_peered":"2024-03-06T07:26:19.227308+0000","last_unstale":"2024-03-06T07:26:19.233488+0000","last_undegraded":"2024-03-06T07:26:19.233488+0000","last_fullsized":"2024-03-06T07:26:19.233488+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:19.014952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222420+0000","last_change":"2024-03-06T07:26:19.222420+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222420+0000","last_undegraded":"2024-03-06T07:26:19.222420+0000","last_fullsized":"2024-03-06T07:26:19.222420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239645+0000","last_change":"2024-03-06T07:26:19.234268+0000","last_active":"2024-03-06T07:26:19.239645+0000","last_peered":"2024-03-06T07:26:19.239645+0000","last_clean":"2024-03-06T07:26:19.239645+0000","last_became_active":"2024-03-06T07:26:19.234000+0000","last_became_peered":"2024-03-06T07:26:19.234000+0000","last_unstale":"2024-03-06T07:26:19.239645+0000","last_undegraded":"2024-03-06T07:26:19.239645+0000","last_fullsized":"2024-03-06T07:26:19.239645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:50.102543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.229363+0000","last_change":"2024-03-06T07:26:19.229363+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.229363+0000","last_undegraded":"2024-03-06T07:26:19.229363+0000","last_fullsized":"2024-03-06T07:26:19.229363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242098+0000","last_change":"2024-03-06T07:26:19.237743+0000","last_active":"2024-03-06T07:26:19.242098+0000","last_peered":"2024-03-06T07:26:19.242098+0000","last_clean":"2024-03-06T07:26:19.242098+0000","last_became_active":"2024-03-06T07:26:19.237394+0000","last_became_peered":"2024-03-06T07:26:19.237394+0000","last_unstale":"2024-03-06T07:26:19.242098+0000","last_undegraded":"2024-03-06T07:26:19.242098+0000","last_fullsized":"2024-03-06T07:26:19.242098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:35.749995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.225400+0000","last_change":"2024-03-06T07:26:19.225400+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.225400+0000","last_undegraded":"2024-03-06T07:26:19.225400+0000","last_fullsized":"2024-03-06T07:26:19.225400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.231970+0000","last_change":"2024-03-06T07:26:19.230603+0000","last_active":"2024-03-06T07:26:19.231970+0000","last_peered":"2024-03-06T07:26:19.231970+0000","last_clean":"2024-03-06T07:26:19.231970+0000","last_became_active":"2024-03-06T07:26:19.230313+0000","last_became_peered":"2024-03-06T07:26:19.230313+0000","last_unstale":"2024-03-06T07:26:19.231970+0000","last_undegraded":"2024-03-06T07:26:19.231970+0000","last_fullsized":"2024-03-06T07:26:19.231970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:31.518943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226436+0000","last_change":"2024-03-06T07:26:19.226436+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226436+0000","last_undegraded":"2024-03-06T07:26:19.226436+0000","last_fullsized":"2024-03-06T07:26:19.226436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.236928+0000","last_change":"2024-03-06T07:26:19.234353+0000","last_active":"2024-03-06T07:26:19.236928+0000","last_peered":"2024-03-06T07:26:19.236928+0000","last_clean":"2024-03-06T07:26:19.236928+0000","last_became_active":"2024-03-06T07:26:19.234077+0000","last_became_peered":"2024-03-06T07:26:19.234077+0000","last_unstale":"2024-03-06T07:26:19.236928+0000","last_undegraded":"2024-03-06T07:26:19.236928+0000","last_fullsized":"2024-03-06T07:26:19.236928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:37.906633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.221555+0000","last_change":"2024-03-06T07:26:19.221555+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.221555+0000","last_undegraded":"2024-03-06T07:26:19.221555+0000","last_fullsized":"2024-03-06T07:26:19.221555+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.238279+0000","last_change":"2024-03-06T07:26:19.237287+0000","last_active":"2024-03-06T07:26:19.238279+0000","last_peered":"2024-03-06T07:26:19.238279+0000","last_clean":"2024-03-06T07:26:19.238279+0000","last_became_active":"2024-03-06T07:26:19.237017+0000","last_became_peered":"2024-03-06T07:26:19.237017+0000","last_unstale":"2024-03-06T07:26:19.238279+0000","last_undegraded":"2024-03-06T07:26:19.238279+0000","last_fullsized":"2024-03-06T07:26:19.238279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:46.817231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227191+0000","last_change":"2024-03-06T07:26:19.227191+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227191+0000","last_undegraded":"2024-03-06T07:26:19.227191+0000","last_fullsized":"2024-03-06T07:26:19.227191+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.232804+0000","last_change":"2024-03-06T07:26:19.229645+0000","last_active":"2024-03-06T07:26:19.232804+0000","last_peered":"2024-03-06T07:26:19.232804+0000","last_clean":"2024-03-06T07:26:19.232804+0000","last_became_active":"2024-03-06T07:26:19.229309+0000","last_became_peered":"2024-03-06T07:26:19.229309+0000","last_unstale":"2024-03-06T07:26:19.232804+0000","last_undegraded":"2024-03-06T07:26:19.232804+0000","last_fullsized":"2024-03-06T07:26:19.232804+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.342246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.222416+0000","last_change":"2024-03-06T07:26:19.222416+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.222416+0000","last_undegraded":"2024-03-06T07:26:19.222416+0000","last_fullsized":"2024-03-06T07:26:19.222416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.235092+0000","last_change":"2024-03-06T07:26:19.231241+0000","last_active":"2024-03-06T07:26:19.235092+0000","last_peered":"2024-03-06T07:26:19.235092+0000","last_clean":"2024-03-06T07:26:19.235092+0000","last_became_active":"2024-03-06T07:26:19.230977+0000","last_became_peered":"2024-03-06T07:26:19.230977+0000","last_unstale":"2024-03-06T07:26:19.235092+0000","last_undegraded":"2024-03-06T07:26:19.235092+0000","last_fullsized":"2024-03-06T07:26:19.235092+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:36.204568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.227602+0000","last_change":"2024-03-06T07:26:19.227602+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.227602+0000","last_undegraded":"2024-03-06T07:26:19.227602+0000","last_fullsized":"2024-03-06T07:26:19.227602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.242260+0000","last_change":"2024-03-06T07:26:19.238002+0000","last_active":"2024-03-06T07:26:19.242260+0000","last_peered":"2024-03-06T07:26:19.242260+0000","last_clean":"2024-03-06T07:26:19.242260+0000","last_became_active":"2024-03-06T07:26:19.237710+0000","last_became_peered":"2024-03-06T07:26:19.237710+0000","last_unstale":"2024-03-06T07:26:19.242260+0000","last_undegraded":"2024-03-06T07:26:19.242260+0000","last_fullsized":"2024-03-06T07:26:19.242260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:25.030580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-03-06T07:26:19.226570+0000","last_change":"2024-03-06T07:26:19.226570+0000","last_active":"2024-03-06T07:26:19.211880+0000","last_peered":"2024-03-06T07:26:19.211880+0000","last_clean":"2024-03-06T07:26:19.211880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:19.226570+0000","last_undegraded":"2024-03-06T07:26:19.226570+0000","last_fullsized":"2024-03-06T07:26:19.226570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.239767+0000","last_change":"2024-03-06T07:26:19.236691+0000","last_active":"2024-03-06T07:26:19.239767+0000","last_peered":"2024-03-06T07:26:19.239767+0000","last_clean":"2024-03-06T07:26:19.239767+0000","last_became_active":"2024-03-06T07:26:19.236432+0000","last_became_peered":"2024-03-06T07:26:19.236432+0000","last_unstale":"2024-03-06T07:26:19.239767+0000","last_undegraded":"2024-03-06T07:26:19.239767+0000","last_fullsized":"2024-03-06T07:26:19.239767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:33.342246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":261,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.224547+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:26:19.224547+0000","last_peered":"2024-03-06T07:26:19.224547+0000","last_clean":"2024-03-06T07:26:19.224547+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:26:19.224547+0000","last_undegraded":"2024-03-06T07:26:19.224547+0000","last_fullsized":"2024-03-06T07:26:19.224547+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":815104,"data_stored":337466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":343741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":337466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:28.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:28.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.829+0000 7fdc628a4640 1 Processor -- start 2024-03-06T07:26:28.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.829+0000 7fdc628a4640 1 -- start start 2024-03-06T07:26:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.830+0000 7fdc628a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdc5c15df00 0x7fdc5c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.830+0000 7fdc628a4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc5c0b95a0 0x7fdc5c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.830+0000 7fdc628a4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c0cea30 con 0x7fdc5c0b9a30 2024-03-06T07:26:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.830+0000 7fdc628a4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5c0d2af0 con 0x7fdc5c0b95a0 2024-03-06T07:26:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.830+0000 7fdc628a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5c0cf3c0 con 0x7fdc5c15df00 2024-03-06T07:26:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.830+0000 7fdc5b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc5c0b95a0 0x7fdc5c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.830+0000 7fdc60e41640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdc5c0b9a30 0x7fdc5c0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47038/0 (socket says 172.21.15.27:47038) 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.830+0000 7fdc60e41640 1 -- 172.21.15.27:0/517521129 learned_addr learned my addr 172.21.15.27:0/517521129 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.831+0000 7fdc5bfff640 1 --2- 172.21.15.27:0/517521129 >> v2:172.21.15.69:3300/0 conn(0x7fdc5c15df00 0x7fdc5c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.831+0000 7fdc5b7fe640 1 -- 172.21.15.27:0/517521129 >> v2:172.21.15.69:3300/0 conn(0x7fdc5c15df00 msgr2=0x7fdc5c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.831+0000 7fdc5b7fe640 1 --2- 172.21.15.27:0/517521129 >> v2:172.21.15.69:3300/0 conn(0x7fdc5c15df00 0x7fdc5c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.831+0000 7fdc5b7fe640 1 -- 172.21.15.27:0/517521129 >> v1:172.21.15.27:6789/0 conn(0x7fdc5c0b9a30 legacy=0x7fdc5c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.831+0000 7fdc5b7fe640 1 -- 172.21.15.27:0/517521129 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c0cf0e0 con 0x7fdc5c0b95a0 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.831+0000 7fdc5affd640 1 -- 172.21.15.27:0/517521129 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557006010 0 0) 0x7fdc5c0cea30 con 0x7fdc5c0b9a30 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.831+0000 7fdc5bfff640 1 --2- 172.21.15.27:0/517521129 >> v2:172.21.15.69:3300/0 conn(0x7fdc5c15df00 0x7fdc5c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc5b7fe640 1 --2- 172.21.15.27:0/517521129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0b95a0 0x7fdc5c0b5700 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0d11b0 tx=0x7fdc5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555dee2abe083798 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc5affd640 1 -- 172.21.15.27:0/517521129 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50035020 con 0x7fdc5c0b95a0 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc5affd640 1 -- 172.21.15.27:0/517521129 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc50031830 con 0x7fdc5c0b95a0 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc5affd640 1 -- 172.21.15.27:0/517521129 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50031c00 con 0x7fdc5c0b95a0 2024-03-06T07:26:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc628a4640 1 -- 172.21.15.27:0/517521129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0b95a0 msgr2=0x7fdc5c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc628a4640 1 --2- 172.21.15.27:0/517521129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0b95a0 0x7fdc5c0b5700 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0d11b0 tx=0x7fdc5002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc628a4640 1 -- 172.21.15.27:0/517521129 shutdown_connections 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc628a4640 1 --2- 172.21.15.27:0/517521129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0b95a0 0x7fdc5c0b5700 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc628a4640 1 --2- 172.21.15.27:0/517521129 >> v2:172.21.15.69:3300/0 conn(0x7fdc5c15df00 0x7fdc5c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc628a4640 1 -- 172.21.15.27:0/517521129 >> 172.21.15.27:0/517521129 conn(0x7fdc5c071960 msgr2=0x7fdc5c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc628a4640 1 -- 172.21.15.27:0/517521129 shutdown_connections 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.832+0000 7fdc628a4640 1 -- 172.21.15.27:0/517521129 wait complete. 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.833+0000 7fdc628a4640 1 Processor -- start 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.834+0000 7fdc628a4640 1 -- start start 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.834+0000 7fdc628a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0b95a0 0x7fdc5c0b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.834+0000 7fdc628a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5c15df00 0x7fdc5c0ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.834+0000 7fdc628a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0ca940 0x7fdc5c0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.834+0000 7fdc628a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c0cec70 con 0x7fdc5c0ca940 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.834+0000 7fdc628a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c0ce750 con 0x7fdc5c15df00 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.834+0000 7fdc628a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c0cedf0 con 0x7fdc5c0b95a0 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.834+0000 7fdc60e41640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0ca940 0x7fdc5c0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc60e41640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0ca940 0x7fdc5c0c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47508/0 (socket says 172.21.15.27:47508) 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc60e41640 1 -- 172.21.15.27:0/1953290379 learned_addr learned my addr 172.21.15.27:0/1953290379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc5b7fe640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5c15df00 0x7fdc5c0ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc5bfff640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0b95a0 0x7fdc5c0b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc60e41640 1 -- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0b95a0 msgr2=0x7fdc5c0b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc60e41640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0b95a0 0x7fdc5c0b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc60e41640 1 -- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5c15df00 msgr2=0x7fdc5c0ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc60e41640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5c15df00 0x7fdc5c0ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc60e41640 1 -- 172.21.15.27:0/1953290379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c14a110 con 0x7fdc5c0ca940 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc60e41640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0ca940 0x7fdc5c0c77b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400c7d0 tx=0x7fdc5400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc58ff9640 1 -- 172.21.15.27:0/1953290379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc54018070 con 0x7fdc5c0ca940 2024-03-06T07:26:28.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc58ff9640 1 -- 172.21.15.27:0/1953290379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5400a640 con 0x7fdc5c0ca940 2024-03-06T07:26:28.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc5c0b4390 con 0x7fdc5c0ca940 2024-03-06T07:26:28.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc58ff9640 1 -- 172.21.15.27:0/1953290379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5400aa70 con 0x7fdc5c0ca940 2024-03-06T07:26:28.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.835+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5c0b4950 con 0x7fdc5c0ca940 2024-03-06T07:26:28.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.836+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5c0b5970 con 0x7fdc5c0ca940 2024-03-06T07:26:28.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.837+0000 7fdc58ff9640 1 -- 172.21.15.27:0/1953290379 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc5401b070 con 0x7fdc5c0ca940 2024-03-06T07:26:28.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.837+0000 7fdc58ff9640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc4806b4d0 0x7fdc4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.837+0000 7fdc58ff9640 1 -- 172.21.15.27:0/1953290379 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdc54089820 con 0x7fdc5c0ca940 2024-03-06T07:26:28.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.837+0000 7fdc5bfff640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc4806b4d0 0x7fdc4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.838+0000 7fdc5bfff640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc4806b4d0 0x7fdc4806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c006fd0 tx=0x7fdc4c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:28.846+0000 7fdc58ff9640 1 -- 172.21.15.27:0/1953290379 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc54058660 con 0x7fdc5c0ca940 2024-03-06T07:26:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.024+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdc5c15d3b0 con 0x7fdc4806b4d0 2024-03-06T07:26:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.031+0000 7fdc58ff9640 1 -- 172.21.15.27:0/1953290379 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252069 (secure 0 0 0) 0x7fdc5c15d3b0 con 0x7fdc4806b4d0 2024-03-06T07:26:29.035 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:29.036 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc4806b4d0 msgr2=0x7fdc4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc4806b4d0 0x7fdc4806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c006fd0 tx=0x7fdc4c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0ca940 msgr2=0x7fdc5c0c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0ca940 0x7fdc5c0c77b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400c7d0 tx=0x7fdc5400cc90 comp rx=0 tx=0).stop 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 shutdown_connections 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc4806b4d0 0x7fdc4806d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc5c0ca940 0x7fdc5c0c77b0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5c15df00 0x7fdc5c0ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 --2- 172.21.15.27:0/1953290379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5c0b95a0 0x7fdc5c0b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 >> 172.21.15.27:0/1953290379 conn(0x7fdc5c071960 msgr2=0x7fdc5c15c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 shutdown_connections 2024-03-06T07:26:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.034+0000 7fdc628a4640 1 -- 172.21.15.27:0/1953290379 wait complete. 2024-03-06T07:26:29.046 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-03-06T07:26:28.259022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257372,"kb_used_data":7880,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717348,"statfs":{"total":773094113280,"available":772830564352,"internally_reserved":0,"allocated":8069120,"data_stored":4040084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.033926"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223880+0000","last_change":"2024-03-06T07:26:20.246753+0000","last_active":"2024-03-06T07:26:21.223880+0000","last_peered":"2024-03-06T07:26:21.223880+0000","last_clean":"2024-03-06T07:26:21.223880+0000","last_became_active":"2024-03-06T07:26:20.246478+0000","last_became_peered":"2024-03-06T07:26:20.246478+0000","last_unstale":"2024-03-06T07:26:21.223880+0000","last_undegraded":"2024-03-06T07:26:21.223880+0000","last_fullsized":"2024-03-06T07:26:21.223880+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:19.315535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303721+0000","last_change":"2024-03-06T07:26:20.250481+0000","last_active":"2024-03-06T07:26:21.303721+0000","last_peered":"2024-03-06T07:26:21.303721+0000","last_clean":"2024-03-06T07:26:21.303721+0000","last_became_active":"2024-03-06T07:26:20.250210+0000","last_became_peered":"2024-03-06T07:26:20.250210+0000","last_unstale":"2024-03-06T07:26:21.303721+0000","last_undegraded":"2024-03-06T07:26:21.303721+0000","last_fullsized":"2024-03-06T07:26:21.303721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:28.292322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302656+0000","last_change":"2024-03-06T07:26:20.235201+0000","last_active":"2024-03-06T07:26:21.302656+0000","last_peered":"2024-03-06T07:26:21.302656+0000","last_clean":"2024-03-06T07:26:21.302656+0000","last_became_active":"2024-03-06T07:26:20.234981+0000","last_became_peered":"2024-03-06T07:26:20.234981+0000","last_unstale":"2024-03-06T07:26:21.302656+0000","last_undegraded":"2024-03-06T07:26:21.302656+0000","last_fullsized":"2024-03-06T07:26:21.302656+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:40.172876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.221898+0000","last_change":"2024-03-06T07:26:20.242496+0000","last_active":"2024-03-06T07:26:21.221898+0000","last_peered":"2024-03-06T07:26:21.221898+0000","last_clean":"2024-03-06T07:26:21.221898+0000","last_became_active":"2024-03-06T07:26:20.242238+0000","last_became_peered":"2024-03-06T07:26:20.242238+0000","last_unstale":"2024-03-06T07:26:21.221898+0000","last_undegraded":"2024-03-06T07:26:21.221898+0000","last_fullsized":"2024-03-06T07:26:21.221898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:01.802258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302830+0000","last_change":"2024-03-06T07:26:20.241934+0000","last_active":"2024-03-06T07:26:21.302830+0000","last_peered":"2024-03-06T07:26:21.302830+0000","last_clean":"2024-03-06T07:26:21.302830+0000","last_became_active":"2024-03-06T07:26:20.241647+0000","last_became_peered":"2024-03-06T07:26:20.241647+0000","last_unstale":"2024-03-06T07:26:21.302830+0000","last_undegraded":"2024-03-06T07:26:21.302830+0000","last_fullsized":"2024-03-06T07:26:21.302830+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:47.426149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303191+0000","last_change":"2024-03-06T07:26:20.242656+0000","last_active":"2024-03-06T07:26:21.303191+0000","last_peered":"2024-03-06T07:26:21.303191+0000","last_clean":"2024-03-06T07:26:21.303191+0000","last_became_active":"2024-03-06T07:26:20.241397+0000","last_became_peered":"2024-03-06T07:26:20.241397+0000","last_unstale":"2024-03-06T07:26:21.303191+0000","last_undegraded":"2024-03-06T07:26:21.303191+0000","last_fullsized":"2024-03-06T07:26:21.303191+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:21.774684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.239688+0000","last_change":"2024-03-06T07:26:20.242723+0000","last_active":"2024-03-06T07:26:21.239688+0000","last_peered":"2024-03-06T07:26:21.239688+0000","last_clean":"2024-03-06T07:26:21.239688+0000","last_became_active":"2024-03-06T07:26:20.242470+0000","last_became_peered":"2024-03-06T07:26:20.242470+0000","last_unstale":"2024-03-06T07:26:21.239688+0000","last_undegraded":"2024-03-06T07:26:21.239688+0000","last_fullsized":"2024-03-06T07:26:21.239688+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:03.521715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303444+0000","last_change":"2024-03-06T07:26:20.242831+0000","last_active":"2024-03-06T07:26:21.303444+0000","last_peered":"2024-03-06T07:26:21.303444+0000","last_clean":"2024-03-06T07:26:21.303444+0000","last_became_active":"2024-03-06T07:26:20.242502+0000","last_became_peered":"2024-03-06T07:26:20.242502+0000","last_unstale":"2024-03-06T07:26:21.303444+0000","last_undegraded":"2024-03-06T07:26:21.303444+0000","last_fullsized":"2024-03-06T07:26:21.303444+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:20.021713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303038+0000","last_change":"2024-03-06T07:26:20.240430+0000","last_active":"2024-03-06T07:26:21.303038+0000","last_peered":"2024-03-06T07:26:21.303038+0000","last_clean":"2024-03-06T07:26:21.303038+0000","last_became_active":"2024-03-06T07:26:20.240011+0000","last_became_peered":"2024-03-06T07:26:20.240011+0000","last_unstale":"2024-03-06T07:26:21.303038+0000","last_undegraded":"2024-03-06T07:26:21.303038+0000","last_fullsized":"2024-03-06T07:26:21.303038+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:37.211328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223899+0000","last_change":"2024-03-06T07:26:20.239027+0000","last_active":"2024-03-06T07:26:21.223899+0000","last_peered":"2024-03-06T07:26:21.223899+0000","last_clean":"2024-03-06T07:26:21.223899+0000","last_became_active":"2024-03-06T07:26:20.238649+0000","last_became_peered":"2024-03-06T07:26:20.238649+0000","last_unstale":"2024-03-06T07:26:21.223899+0000","last_undegraded":"2024-03-06T07:26:21.223899+0000","last_fullsized":"2024-03-06T07:26:21.223899+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:28.292322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223669+0000","last_change":"2024-03-06T07:26:20.243212+0000","last_active":"2024-03-06T07:26:21.223669+0000","last_peered":"2024-03-06T07:26:21.223669+0000","last_clean":"2024-03-06T07:26:21.223669+0000","last_became_active":"2024-03-06T07:26:20.242917+0000","last_became_peered":"2024-03-06T07:26:20.242917+0000","last_unstale":"2024-03-06T07:26:21.223669+0000","last_undegraded":"2024-03-06T07:26:21.223669+0000","last_fullsized":"2024-03-06T07:26:21.223669+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:04.439664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.221965+0000","last_change":"2024-03-06T07:26:20.226627+0000","last_active":"2024-03-06T07:26:21.221965+0000","last_peered":"2024-03-06T07:26:21.221965+0000","last_clean":"2024-03-06T07:26:21.221965+0000","last_became_active":"2024-03-06T07:26:20.226302+0000","last_became_peered":"2024-03-06T07:26:20.226302+0000","last_unstale":"2024-03-06T07:26:21.221965+0000","last_undegraded":"2024-03-06T07:26:21.221965+0000","last_fullsized":"2024-03-06T07:26:21.221965+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:50.540658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.242109+0000","last_change":"2024-03-06T07:26:20.245045+0000","last_active":"2024-03-06T07:26:21.242109+0000","last_peered":"2024-03-06T07:26:21.242109+0000","last_clean":"2024-03-06T07:26:21.242109+0000","last_became_active":"2024-03-06T07:26:20.244749+0000","last_became_peered":"2024-03-06T07:26:20.244749+0000","last_unstale":"2024-03-06T07:26:21.242109+0000","last_undegraded":"2024-03-06T07:26:21.242109+0000","last_fullsized":"2024-03-06T07:26:21.242109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:17.086279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302865+0000","last_change":"2024-03-06T07:26:20.241118+0000","last_active":"2024-03-06T07:26:21.302865+0000","last_peered":"2024-03-06T07:26:21.302865+0000","last_clean":"2024-03-06T07:26:21.302865+0000","last_became_active":"2024-03-06T07:26:20.240847+0000","last_became_peered":"2024-03-06T07:26:20.240847+0000","last_unstale":"2024-03-06T07:26:21.302865+0000","last_undegraded":"2024-03-06T07:26:21.302865+0000","last_fullsized":"2024-03-06T07:26:21.302865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:31.946911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.241888+0000","last_change":"2024-03-06T07:26:20.229898+0000","last_active":"2024-03-06T07:26:21.241888+0000","last_peered":"2024-03-06T07:26:21.241888+0000","last_clean":"2024-03-06T07:26:21.241888+0000","last_became_active":"2024-03-06T07:26:20.229620+0000","last_became_peered":"2024-03-06T07:26:20.229620+0000","last_unstale":"2024-03-06T07:26:21.241888+0000","last_undegraded":"2024-03-06T07:26:21.241888+0000","last_fullsized":"2024-03-06T07:26:21.241888+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:46.112113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223191+0000","last_change":"2024-03-06T07:26:20.247612+0000","last_active":"2024-03-06T07:26:21.223191+0000","last_peered":"2024-03-06T07:26:21.223191+0000","last_clean":"2024-03-06T07:26:21.223191+0000","last_became_active":"2024-03-06T07:26:20.247335+0000","last_became_peered":"2024-03-06T07:26:20.247335+0000","last_unstale":"2024-03-06T07:26:21.223191+0000","last_undegraded":"2024-03-06T07:26:21.223191+0000","last_fullsized":"2024-03-06T07:26:21.223191+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:58.590792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303068+0000","last_change":"2024-03-06T07:26:20.240278+0000","last_active":"2024-03-06T07:26:21.303068+0000","last_peered":"2024-03-06T07:26:21.303068+0000","last_clean":"2024-03-06T07:26:21.303068+0000","last_became_active":"2024-03-06T07:26:20.239933+0000","last_became_peered":"2024-03-06T07:26:20.239933+0000","last_unstale":"2024-03-06T07:26:21.303068+0000","last_undegraded":"2024-03-06T07:26:21.303068+0000","last_fullsized":"2024-03-06T07:26:21.303068+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:57.375714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.221789+0000","last_change":"2024-03-06T07:26:20.241976+0000","last_active":"2024-03-06T07:26:21.221789+0000","last_peered":"2024-03-06T07:26:21.221789+0000","last_clean":"2024-03-06T07:26:21.221789+0000","last_became_active":"2024-03-06T07:26:20.241606+0000","last_became_peered":"2024-03-06T07:26:20.241606+0000","last_unstale":"2024-03-06T07:26:21.221789+0000","last_undegraded":"2024-03-06T07:26:21.221789+0000","last_fullsized":"2024-03-06T07:26:21.221789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:37.211328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303412+0000","last_change":"2024-03-06T07:26:20.244968+0000","last_active":"2024-03-06T07:26:21.303412+0000","last_peered":"2024-03-06T07:26:21.303412+0000","last_clean":"2024-03-06T07:26:21.303412+0000","last_became_active":"2024-03-06T07:26:20.244701+0000","last_became_peered":"2024-03-06T07:26:20.244701+0000","last_unstale":"2024-03-06T07:26:21.303412+0000","last_undegraded":"2024-03-06T07:26:21.303412+0000","last_fullsized":"2024-03-06T07:26:21.303412+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:58.590792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.242189+0000","last_change":"2024-03-06T07:26:20.240388+0000","last_active":"2024-03-06T07:26:21.242189+0000","last_peered":"2024-03-06T07:26:21.242189+0000","last_clean":"2024-03-06T07:26:21.242189+0000","last_became_active":"2024-03-06T07:26:20.240099+0000","last_became_peered":"2024-03-06T07:26:20.240099+0000","last_unstale":"2024-03-06T07:26:21.242189+0000","last_undegraded":"2024-03-06T07:26:21.242189+0000","last_fullsized":"2024-03-06T07:26:21.242189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:26.612731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223350+0000","last_change":"2024-03-06T07:26:20.230869+0000","last_active":"2024-03-06T07:26:21.223350+0000","last_peered":"2024-03-06T07:26:21.223350+0000","last_clean":"2024-03-06T07:26:21.223350+0000","last_became_active":"2024-03-06T07:26:20.230595+0000","last_became_peered":"2024-03-06T07:26:20.230595+0000","last_unstale":"2024-03-06T07:26:21.223350+0000","last_undegraded":"2024-03-06T07:26:21.223350+0000","last_fullsized":"2024-03-06T07:26:21.223350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:51.893862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223062+0000","last_change":"2024-03-06T07:26:20.234654+0000","last_active":"2024-03-06T07:26:21.223062+0000","last_peered":"2024-03-06T07:26:21.223062+0000","last_clean":"2024-03-06T07:26:21.223062+0000","last_became_active":"2024-03-06T07:26:20.234380+0000","last_became_peered":"2024-03-06T07:26:20.234380+0000","last_unstale":"2024-03-06T07:26:21.223062+0000","last_undegraded":"2024-03-06T07:26:21.223062+0000","last_fullsized":"2024-03-06T07:26:21.223062+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:34.087751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223714+0000","last_change":"2024-03-06T07:26:20.242274+0000","last_active":"2024-03-06T07:26:21.223714+0000","last_peered":"2024-03-06T07:26:21.223714+0000","last_clean":"2024-03-06T07:26:21.223714+0000","last_became_active":"2024-03-06T07:26:20.241349+0000","last_became_peered":"2024-03-06T07:26:20.241349+0000","last_unstale":"2024-03-06T07:26:21.223714+0000","last_undegraded":"2024-03-06T07:26:21.223714+0000","last_fullsized":"2024-03-06T07:26:21.223714+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:20.628607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.222631+0000","last_change":"2024-03-06T07:26:20.248313+0000","last_active":"2024-03-06T07:26:21.222631+0000","last_peered":"2024-03-06T07:26:21.222631+0000","last_clean":"2024-03-06T07:26:21.222631+0000","last_became_active":"2024-03-06T07:26:20.248032+0000","last_became_peered":"2024-03-06T07:26:20.248032+0000","last_unstale":"2024-03-06T07:26:21.222631+0000","last_undegraded":"2024-03-06T07:26:21.222631+0000","last_fullsized":"2024-03-06T07:26:21.222631+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:28.583503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223974+0000","last_change":"2024-03-06T07:26:20.240240+0000","last_active":"2024-03-06T07:26:21.223974+0000","last_peered":"2024-03-06T07:26:21.223974+0000","last_clean":"2024-03-06T07:26:21.223974+0000","last_became_active":"2024-03-06T07:26:20.239971+0000","last_became_peered":"2024-03-06T07:26:20.239971+0000","last_unstale":"2024-03-06T07:26:21.223974+0000","last_undegraded":"2024-03-06T07:26:21.223974+0000","last_fullsized":"2024-03-06T07:26:21.223974+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:22.973583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.240136+0000","last_change":"2024-03-06T07:26:20.247527+0000","last_active":"2024-03-06T07:26:21.240136+0000","last_peered":"2024-03-06T07:26:21.240136+0000","last_clean":"2024-03-06T07:26:21.240136+0000","last_became_active":"2024-03-06T07:26:20.245403+0000","last_became_peered":"2024-03-06T07:26:20.245403+0000","last_unstale":"2024-03-06T07:26:21.240136+0000","last_undegraded":"2024-03-06T07:26:21.240136+0000","last_fullsized":"2024-03-06T07:26:21.240136+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:26.612731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302764+0000","last_change":"2024-03-06T07:26:20.238652+0000","last_active":"2024-03-06T07:26:21.302764+0000","last_peered":"2024-03-06T07:26:21.302764+0000","last_clean":"2024-03-06T07:26:21.302764+0000","last_became_active":"2024-03-06T07:26:20.238293+0000","last_became_peered":"2024-03-06T07:26:20.238293+0000","last_unstale":"2024-03-06T07:26:21.302764+0000","last_undegraded":"2024-03-06T07:26:21.302764+0000","last_fullsized":"2024-03-06T07:26:21.302764+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:42.923787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223520+0000","last_change":"2024-03-06T07:26:20.240367+0000","last_active":"2024-03-06T07:26:21.223520+0000","last_peered":"2024-03-06T07:26:21.223520+0000","last_clean":"2024-03-06T07:26:21.223520+0000","last_became_active":"2024-03-06T07:26:20.240063+0000","last_became_peered":"2024-03-06T07:26:20.240063+0000","last_unstale":"2024-03-06T07:26:21.223520+0000","last_undegraded":"2024-03-06T07:26:21.223520+0000","last_fullsized":"2024-03-06T07:26:21.223520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:38.913394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302589+0000","last_change":"2024-03-06T07:26:20.240010+0000","last_active":"2024-03-06T07:26:21.302589+0000","last_peered":"2024-03-06T07:26:21.302589+0000","last_clean":"2024-03-06T07:26:21.302589+0000","last_became_active":"2024-03-06T07:26:20.239727+0000","last_became_peered":"2024-03-06T07:26:20.239727+0000","last_unstale":"2024-03-06T07:26:21.302589+0000","last_undegraded":"2024-03-06T07:26:21.302589+0000","last_fullsized":"2024-03-06T07:26:21.302589+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:54.208273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.239740+0000","last_change":"2024-03-06T07:26:20.252985+0000","last_active":"2024-03-06T07:26:21.239740+0000","last_peered":"2024-03-06T07:26:21.239740+0000","last_clean":"2024-03-06T07:26:21.239740+0000","last_became_active":"2024-03-06T07:26:20.252705+0000","last_became_peered":"2024-03-06T07:26:20.252705+0000","last_unstale":"2024-03-06T07:26:21.239740+0000","last_undegraded":"2024-03-06T07:26:21.239740+0000","last_fullsized":"2024-03-06T07:26:21.239740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:36.637698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.239949+0000","last_change":"2024-03-06T07:26:20.244998+0000","last_active":"2024-03-06T07:26:21.239949+0000","last_peered":"2024-03-06T07:26:21.239949+0000","last_clean":"2024-03-06T07:26:21.239949+0000","last_became_active":"2024-03-06T07:26:20.244736+0000","last_became_peered":"2024-03-06T07:26:20.244736+0000","last_unstale":"2024-03-06T07:26:21.239949+0000","last_undegraded":"2024-03-06T07:26:21.239949+0000","last_fullsized":"2024-03-06T07:26:21.239949+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:46.112113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303603+0000","last_change":"2024-03-06T07:26:20.251866+0000","last_active":"2024-03-06T07:26:21.303603+0000","last_peered":"2024-03-06T07:26:21.303603+0000","last_clean":"2024-03-06T07:26:21.303603+0000","last_became_active":"2024-03-06T07:26:20.251426+0000","last_became_peered":"2024-03-06T07:26:20.251426+0000","last_unstale":"2024-03-06T07:26:21.303603+0000","last_undegraded":"2024-03-06T07:26:21.303603+0000","last_fullsized":"2024-03-06T07:26:21.303603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:38.913394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.222946+0000","last_change":"2024-03-06T07:26:20.248547+0000","last_active":"2024-03-06T07:26:21.222946+0000","last_peered":"2024-03-06T07:26:21.222946+0000","last_clean":"2024-03-06T07:26:21.222946+0000","last_became_active":"2024-03-06T07:26:20.248286+0000","last_became_peered":"2024-03-06T07:26:20.248286+0000","last_unstale":"2024-03-06T07:26:21.222946+0000","last_undegraded":"2024-03-06T07:26:21.222946+0000","last_fullsized":"2024-03-06T07:26:21.222946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:18.065803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.224309+0000","last_change":"2024-03-06T07:26:19.235249+0000","last_active":"2024-03-06T07:26:21.224309+0000","last_peered":"2024-03-06T07:26:21.224309+0000","last_clean":"2024-03-06T07:26:21.224309+0000","last_became_active":"2024-03-06T07:26:19.234904+0000","last_became_peered":"2024-03-06T07:26:19.234904+0000","last_unstale":"2024-03-06T07:26:21.224309+0000","last_undegraded":"2024-03-06T07:26:21.224309+0000","last_fullsized":"2024-03-06T07:26:21.224309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:57.467124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303180+0000","last_change":"2024-03-06T07:26:20.240196+0000","last_active":"2024-03-06T07:26:21.303180+0000","last_peered":"2024-03-06T07:26:21.303180+0000","last_clean":"2024-03-06T07:26:21.303180+0000","last_became_active":"2024-03-06T07:26:20.239397+0000","last_became_peered":"2024-03-06T07:26:20.239397+0000","last_unstale":"2024-03-06T07:26:21.303180+0000","last_undegraded":"2024-03-06T07:26:21.303180+0000","last_fullsized":"2024-03-06T07:26:21.303180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:01.802258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.242237+0000","last_change":"2024-03-06T07:26:19.225962+0000","last_active":"2024-03-06T07:26:21.242237+0000","last_peered":"2024-03-06T07:26:21.242237+0000","last_clean":"2024-03-06T07:26:21.242237+0000","last_became_active":"2024-03-06T07:26:19.225707+0000","last_became_peered":"2024-03-06T07:26:19.225707+0000","last_unstale":"2024-03-06T07:26:21.242237+0000","last_undegraded":"2024-03-06T07:26:21.242237+0000","last_fullsized":"2024-03-06T07:26:21.242237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:44.732093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303339+0000","last_change":"2024-03-06T07:26:20.254812+0000","last_active":"2024-03-06T07:26:21.303339+0000","last_peered":"2024-03-06T07:26:21.303339+0000","last_clean":"2024-03-06T07:26:21.303339+0000","last_became_active":"2024-03-06T07:26:20.254513+0000","last_became_peered":"2024-03-06T07:26:20.254513+0000","last_unstale":"2024-03-06T07:26:21.303339+0000","last_undegraded":"2024-03-06T07:26:21.303339+0000","last_fullsized":"2024-03-06T07:26:21.303339+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:36.756755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223536+0000","last_change":"2024-03-06T07:26:19.232800+0000","last_active":"2024-03-06T07:26:21.223536+0000","last_peered":"2024-03-06T07:26:21.223536+0000","last_clean":"2024-03-06T07:26:21.223536+0000","last_became_active":"2024-03-06T07:26:19.232541+0000","last_became_peered":"2024-03-06T07:26:19.232541+0000","last_unstale":"2024-03-06T07:26:21.223536+0000","last_undegraded":"2024-03-06T07:26:21.223536+0000","last_fullsized":"2024-03-06T07:26:21.223536+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:20.332669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223858+0000","last_change":"2024-03-06T07:26:20.246612+0000","last_active":"2024-03-06T07:26:21.223858+0000","last_peered":"2024-03-06T07:26:21.223858+0000","last_clean":"2024-03-06T07:26:21.223858+0000","last_became_active":"2024-03-06T07:26:20.245252+0000","last_became_peered":"2024-03-06T07:26:20.245252+0000","last_unstale":"2024-03-06T07:26:21.223858+0000","last_undegraded":"2024-03-06T07:26:21.223858+0000","last_fullsized":"2024-03-06T07:26:21.223858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:31.775810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303470+0000","last_change":"2024-03-06T07:26:19.237329+0000","last_active":"2024-03-06T07:26:21.303470+0000","last_peered":"2024-03-06T07:26:21.303470+0000","last_clean":"2024-03-06T07:26:21.303470+0000","last_became_active":"2024-03-06T07:26:19.237096+0000","last_became_peered":"2024-03-06T07:26:19.237096+0000","last_unstale":"2024-03-06T07:26:21.303470+0000","last_undegraded":"2024-03-06T07:26:21.303470+0000","last_fullsized":"2024-03-06T07:26:21.303470+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:36.204568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.239774+0000","last_change":"2024-03-06T07:26:20.243796+0000","last_active":"2024-03-06T07:26:21.239774+0000","last_peered":"2024-03-06T07:26:21.239774+0000","last_clean":"2024-03-06T07:26:21.239774+0000","last_became_active":"2024-03-06T07:26:20.243461+0000","last_became_peered":"2024-03-06T07:26:20.243461+0000","last_unstale":"2024-03-06T07:26:21.239774+0000","last_undegraded":"2024-03-06T07:26:21.239774+0000","last_fullsized":"2024-03-06T07:26:21.239774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:12.623048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302661+0000","last_change":"2024-03-06T07:26:19.236188+0000","last_active":"2024-03-06T07:26:21.302661+0000","last_peered":"2024-03-06T07:26:21.302661+0000","last_clean":"2024-03-06T07:26:21.302661+0000","last_became_active":"2024-03-06T07:26:19.235921+0000","last_became_peered":"2024-03-06T07:26:19.235921+0000","last_unstale":"2024-03-06T07:26:21.302661+0000","last_undegraded":"2024-03-06T07:26:21.302661+0000","last_fullsized":"2024-03-06T07:26:21.302661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:03.432904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223375+0000","last_change":"2024-03-06T07:26:20.240894+0000","last_active":"2024-03-06T07:26:21.223375+0000","last_peered":"2024-03-06T07:26:21.223375+0000","last_clean":"2024-03-06T07:26:21.223375+0000","last_became_active":"2024-03-06T07:26:20.240626+0000","last_became_peered":"2024-03-06T07:26:20.240626+0000","last_unstale":"2024-03-06T07:26:21.223375+0000","last_undegraded":"2024-03-06T07:26:21.223375+0000","last_fullsized":"2024-03-06T07:26:21.223375+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:20.021713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303156+0000","last_change":"2024-03-06T07:26:19.237984+0000","last_active":"2024-03-06T07:26:21.303156+0000","last_peered":"2024-03-06T07:26:21.303156+0000","last_clean":"2024-03-06T07:26:21.303156+0000","last_became_active":"2024-03-06T07:26:19.237697+0000","last_became_peered":"2024-03-06T07:26:19.237697+0000","last_unstale":"2024-03-06T07:26:21.303156+0000","last_undegraded":"2024-03-06T07:26:21.303156+0000","last_fullsized":"2024-03-06T07:26:21.303156+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.795497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302432+0000","last_change":"2024-03-06T07:26:20.229781+0000","last_active":"2024-03-06T07:26:21.302432+0000","last_peered":"2024-03-06T07:26:21.302432+0000","last_clean":"2024-03-06T07:26:21.302432+0000","last_became_active":"2024-03-06T07:26:20.229501+0000","last_became_peered":"2024-03-06T07:26:20.229501+0000","last_unstale":"2024-03-06T07:26:21.302432+0000","last_undegraded":"2024-03-06T07:26:21.302432+0000","last_fullsized":"2024-03-06T07:26:21.302432+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:26.037340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.221621+0000","last_change":"2024-03-06T07:26:19.228888+0000","last_active":"2024-03-06T07:26:21.221621+0000","last_peered":"2024-03-06T07:26:21.221621+0000","last_clean":"2024-03-06T07:26:21.221621+0000","last_became_active":"2024-03-06T07:26:19.228602+0000","last_became_peered":"2024-03-06T07:26:19.228602+0000","last_unstale":"2024-03-06T07:26:21.221621+0000","last_undegraded":"2024-03-06T07:26:21.221621+0000","last_fullsized":"2024-03-06T07:26:21.221621+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:58.527722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.241924+0000","last_change":"2024-03-06T07:26:20.243391+0000","last_active":"2024-03-06T07:26:21.241924+0000","last_peered":"2024-03-06T07:26:21.241924+0000","last_clean":"2024-03-06T07:26:21.241924+0000","last_became_active":"2024-03-06T07:26:20.243111+0000","last_became_peered":"2024-03-06T07:26:20.243111+0000","last_unstale":"2024-03-06T07:26:21.241924+0000","last_undegraded":"2024-03-06T07:26:21.241924+0000","last_fullsized":"2024-03-06T07:26:21.241924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:07.966456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302561+0000","last_change":"2024-03-06T07:26:19.231771+0000","last_active":"2024-03-06T07:26:21.302561+0000","last_peered":"2024-03-06T07:26:21.302561+0000","last_clean":"2024-03-06T07:26:21.302561+0000","last_became_active":"2024-03-06T07:26:19.231485+0000","last_became_peered":"2024-03-06T07:26:19.231485+0000","last_unstale":"2024-03-06T07:26:21.302561+0000","last_undegraded":"2024-03-06T07:26:21.302561+0000","last_fullsized":"2024-03-06T07:26:21.302561+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:34.681465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.240153+0000","last_change":"2024-03-06T07:26:20.243932+0000","last_active":"2024-03-06T07:26:21.240153+0000","last_peered":"2024-03-06T07:26:21.240153+0000","last_clean":"2024-03-06T07:26:21.240153+0000","last_became_active":"2024-03-06T07:26:20.243616+0000","last_became_peered":"2024-03-06T07:26:20.243616+0000","last_unstale":"2024-03-06T07:26:21.240153+0000","last_undegraded":"2024-03-06T07:26:21.240153+0000","last_fullsized":"2024-03-06T07:26:21.240153+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:41.481894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223925+0000","last_change":"2024-03-06T07:26:19.229136+0000","last_active":"2024-03-06T07:26:21.223925+0000","last_peered":"2024-03-06T07:26:21.223925+0000","last_clean":"2024-03-06T07:26:21.223925+0000","last_became_active":"2024-03-06T07:26:19.228661+0000","last_became_peered":"2024-03-06T07:26:19.228661+0000","last_unstale":"2024-03-06T07:26:21.223925+0000","last_undegraded":"2024-03-06T07:26:21.223925+0000","last_fullsized":"2024-03-06T07:26:21.223925+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:10.952333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.224050+0000","last_change":"2024-03-06T07:26:20.249242+0000","last_active":"2024-03-06T07:26:21.224050+0000","last_peered":"2024-03-06T07:26:21.224050+0000","last_clean":"2024-03-06T07:26:21.224050+0000","last_became_active":"2024-03-06T07:26:20.248967+0000","last_became_peered":"2024-03-06T07:26:20.248967+0000","last_unstale":"2024-03-06T07:26:21.224050+0000","last_undegraded":"2024-03-06T07:26:21.224050+0000","last_fullsized":"2024-03-06T07:26:21.224050+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:04.457771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303706+0000","last_change":"2024-03-06T07:26:19.236797+0000","last_active":"2024-03-06T07:26:21.303706+0000","last_peered":"2024-03-06T07:26:21.303706+0000","last_clean":"2024-03-06T07:26:21.303706+0000","last_became_active":"2024-03-06T07:26:19.236118+0000","last_became_peered":"2024-03-06T07:26:19.236118+0000","last_unstale":"2024-03-06T07:26:21.303706+0000","last_undegraded":"2024-03-06T07:26:21.303706+0000","last_fullsized":"2024-03-06T07:26:21.303706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:38:30.940151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303491+0000","last_change":"2024-03-06T07:26:20.243150+0000","last_active":"2024-03-06T07:26:21.303491+0000","last_peered":"2024-03-06T07:26:21.303491+0000","last_clean":"2024-03-06T07:26:21.303491+0000","last_became_active":"2024-03-06T07:26:20.242240+0000","last_became_peered":"2024-03-06T07:26:20.242240+0000","last_unstale":"2024-03-06T07:26:21.303491+0000","last_undegraded":"2024-03-06T07:26:21.303491+0000","last_fullsized":"2024-03-06T07:26:21.303491+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:51.893862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302751+0000","last_change":"2024-03-06T07:26:19.235158+0000","last_active":"2024-03-06T07:26:21.302751+0000","last_peered":"2024-03-06T07:26:21.302751+0000","last_clean":"2024-03-06T07:26:21.302751+0000","last_became_active":"2024-03-06T07:26:19.234883+0000","last_became_peered":"2024-03-06T07:26:19.234883+0000","last_unstale":"2024-03-06T07:26:21.302751+0000","last_undegraded":"2024-03-06T07:26:21.302751+0000","last_fullsized":"2024-03-06T07:26:21.302751+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:19.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303009+0000","last_change":"2024-03-06T07:26:20.226886+0000","last_active":"2024-03-06T07:26:21.303009+0000","last_peered":"2024-03-06T07:26:21.303009+0000","last_clean":"2024-03-06T07:26:21.303009+0000","last_became_active":"2024-03-06T07:26:20.226461+0000","last_became_peered":"2024-03-06T07:26:20.226461+0000","last_unstale":"2024-03-06T07:26:21.303009+0000","last_undegraded":"2024-03-06T07:26:21.303009+0000","last_fullsized":"2024-03-06T07:26:21.303009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:31.545764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":20,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.222056+0000","last_change":"2024-03-06T07:26:19.236129+0000","last_active":"2024-03-06T07:26:21.222056+0000","last_peered":"2024-03-06T07:26:21.222056+0000","last_clean":"2024-03-06T07:26:21.222056+0000","last_became_active":"2024-03-06T07:26:19.235844+0000","last_became_peered":"2024-03-06T07:26:19.235844+0000","last_unstale":"2024-03-06T07:26:21.222056+0000","last_undegraded":"2024-03-06T07:26:21.222056+0000","last_fullsized":"2024-03-06T07:26:21.222056+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:25.030580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.222826+0000","last_change":"2024-03-06T07:26:20.248111+0000","last_active":"2024-03-06T07:26:21.222826+0000","last_peered":"2024-03-06T07:26:21.222826+0000","last_clean":"2024-03-06T07:26:21.222826+0000","last_became_active":"2024-03-06T07:26:20.245939+0000","last_became_peered":"2024-03-06T07:26:20.245939+0000","last_unstale":"2024-03-06T07:26:21.222826+0000","last_undegraded":"2024-03-06T07:26:21.222826+0000","last_fullsized":"2024-03-06T07:26:21.222826+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:00.465472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.221714+0000","last_change":"2024-03-06T07:26:19.226330+0000","last_active":"2024-03-06T07:26:21.221714+0000","last_peered":"2024-03-06T07:26:21.221714+0000","last_clean":"2024-03-06T07:26:21.221714+0000","last_became_active":"2024-03-06T07:26:19.226062+0000","last_became_peered":"2024-03-06T07:26:19.226062+0000","last_unstale":"2024-03-06T07:26:21.221714+0000","last_undegraded":"2024-03-06T07:26:21.221714+0000","last_fullsized":"2024-03-06T07:26:21.221714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:21.157010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.239970+0000","last_change":"2024-03-06T07:26:20.247294+0000","last_active":"2024-03-06T07:26:21.239970+0000","last_peered":"2024-03-06T07:26:21.239970+0000","last_clean":"2024-03-06T07:26:21.239970+0000","last_became_active":"2024-03-06T07:26:20.247017+0000","last_became_peered":"2024-03-06T07:26:20.247017+0000","last_unstale":"2024-03-06T07:26:21.239970+0000","last_undegraded":"2024-03-06T07:26:21.239970+0000","last_fullsized":"2024-03-06T07:26:21.239970+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:07.966456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303346+0000","last_change":"2024-03-06T07:26:19.238616+0000","last_active":"2024-03-06T07:26:21.303346+0000","last_peered":"2024-03-06T07:26:21.303346+0000","last_clean":"2024-03-06T07:26:21.303346+0000","last_became_active":"2024-03-06T07:26:19.238337+0000","last_became_peered":"2024-03-06T07:26:19.238337+0000","last_unstale":"2024-03-06T07:26:21.303346+0000","last_undegraded":"2024-03-06T07:26:21.303346+0000","last_fullsized":"2024-03-06T07:26:21.303346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:49.533898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.241821+0000","last_change":"2024-03-06T07:26:20.244841+0000","last_active":"2024-03-06T07:26:21.241821+0000","last_peered":"2024-03-06T07:26:21.241821+0000","last_clean":"2024-03-06T07:26:21.241821+0000","last_became_active":"2024-03-06T07:26:20.244448+0000","last_became_peered":"2024-03-06T07:26:20.244448+0000","last_unstale":"2024-03-06T07:26:21.241821+0000","last_undegraded":"2024-03-06T07:26:21.241821+0000","last_fullsized":"2024-03-06T07:26:21.241821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:12.623048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.239638+0000","last_change":"2024-03-06T07:26:19.236852+0000","last_active":"2024-03-06T07:26:21.239638+0000","last_peered":"2024-03-06T07:26:21.239638+0000","last_clean":"2024-03-06T07:26:21.239638+0000","last_became_active":"2024-03-06T07:26:19.236469+0000","last_became_peered":"2024-03-06T07:26:19.236469+0000","last_unstale":"2024-03-06T07:26:21.239638+0000","last_undegraded":"2024-03-06T07:26:21.239638+0000","last_fullsized":"2024-03-06T07:26:21.239638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:57.584032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302828+0000","last_change":"2024-03-06T07:26:20.235512+0000","last_active":"2024-03-06T07:26:21.302828+0000","last_peered":"2024-03-06T07:26:21.302828+0000","last_clean":"2024-03-06T07:26:21.302828+0000","last_became_active":"2024-03-06T07:26:20.235185+0000","last_became_peered":"2024-03-06T07:26:20.235185+0000","last_unstale":"2024-03-06T07:26:21.302828+0000","last_undegraded":"2024-03-06T07:26:21.302828+0000","last_fullsized":"2024-03-06T07:26:21.302828+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:15.389760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":22,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223169+0000","last_change":"2024-03-06T07:26:19.231717+0000","last_active":"2024-03-06T07:26:21.223169+0000","last_peered":"2024-03-06T07:26:21.223169+0000","last_clean":"2024-03-06T07:26:21.223169+0000","last_became_active":"2024-03-06T07:26:19.231462+0000","last_became_peered":"2024-03-06T07:26:19.231462+0000","last_unstale":"2024-03-06T07:26:21.223169+0000","last_undegraded":"2024-03-06T07:26:21.223169+0000","last_fullsized":"2024-03-06T07:26:21.223169+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:51.720578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.224020+0000","last_change":"2024-03-06T07:26:20.239772+0000","last_active":"2024-03-06T07:26:21.224020+0000","last_peered":"2024-03-06T07:26:21.224020+0000","last_clean":"2024-03-06T07:26:21.224020+0000","last_became_active":"2024-03-06T07:26:20.239488+0000","last_became_peered":"2024-03-06T07:26:20.239488+0000","last_unstale":"2024-03-06T07:26:21.224020+0000","last_undegraded":"2024-03-06T07:26:21.224020+0000","last_fullsized":"2024-03-06T07:26:21.224020+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:45.738854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303627+0000","last_change":"2024-03-06T07:26:19.237309+0000","last_active":"2024-03-06T07:26:21.303627+0000","last_peered":"2024-03-06T07:26:21.303627+0000","last_clean":"2024-03-06T07:26:21.303627+0000","last_became_active":"2024-03-06T07:26:19.237011+0000","last_became_peered":"2024-03-06T07:26:19.237011+0000","last_unstale":"2024-03-06T07:26:21.303627+0000","last_undegraded":"2024-03-06T07:26:21.303627+0000","last_fullsized":"2024-03-06T07:26:21.303627+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:46.419389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.224203+0000","last_change":"2024-03-06T07:26:20.244286+0000","last_active":"2024-03-06T07:26:21.224203+0000","last_peered":"2024-03-06T07:26:21.224203+0000","last_clean":"2024-03-06T07:26:21.224203+0000","last_became_active":"2024-03-06T07:26:20.243548+0000","last_became_peered":"2024-03-06T07:26:20.243548+0000","last_unstale":"2024-03-06T07:26:21.224203+0000","last_undegraded":"2024-03-06T07:26:21.224203+0000","last_fullsized":"2024-03-06T07:26:21.224203+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:20.125367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.224100+0000","last_change":"2024-03-06T07:26:19.233492+0000","last_active":"2024-03-06T07:26:21.224100+0000","last_peered":"2024-03-06T07:26:21.224100+0000","last_clean":"2024-03-06T07:26:21.224100+0000","last_became_active":"2024-03-06T07:26:19.233199+0000","last_became_peered":"2024-03-06T07:26:19.233199+0000","last_unstale":"2024-03-06T07:26:21.224100+0000","last_undegraded":"2024-03-06T07:26:21.224100+0000","last_fullsized":"2024-03-06T07:26:21.224100+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:35.982194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.221990+0000","last_change":"2024-03-06T07:26:20.226494+0000","last_active":"2024-03-06T07:26:21.221990+0000","last_peered":"2024-03-06T07:26:21.221990+0000","last_clean":"2024-03-06T07:26:21.221990+0000","last_became_active":"2024-03-06T07:26:20.226189+0000","last_became_peered":"2024-03-06T07:26:20.226189+0000","last_unstale":"2024-03-06T07:26:21.221990+0000","last_undegraded":"2024-03-06T07:26:21.221990+0000","last_fullsized":"2024-03-06T07:26:21.221990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:23.675132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303547+0000","last_change":"2024-03-06T07:26:19.236432+0000","last_active":"2024-03-06T07:26:21.303547+0000","last_peered":"2024-03-06T07:26:21.303547+0000","last_clean":"2024-03-06T07:26:21.303547+0000","last_became_active":"2024-03-06T07:26:19.235975+0000","last_became_peered":"2024-03-06T07:26:19.235975+0000","last_unstale":"2024-03-06T07:26:21.303547+0000","last_undegraded":"2024-03-06T07:26:21.303547+0000","last_fullsized":"2024-03-06T07:26:21.303547+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:30.539004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303280+0000","last_change":"2024-03-06T07:26:20.232094+0000","last_active":"2024-03-06T07:26:21.303280+0000","last_peered":"2024-03-06T07:26:21.303280+0000","last_clean":"2024-03-06T07:26:21.303280+0000","last_became_active":"2024-03-06T07:26:20.231652+0000","last_became_peered":"2024-03-06T07:26:20.231652+0000","last_unstale":"2024-03-06T07:26:21.303280+0000","last_undegraded":"2024-03-06T07:26:21.303280+0000","last_fullsized":"2024-03-06T07:26:21.303280+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:50.540658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223219+0000","last_change":"2024-03-06T07:26:19.233564+0000","last_active":"2024-03-06T07:26:21.223219+0000","last_peered":"2024-03-06T07:26:21.223219+0000","last_clean":"2024-03-06T07:26:21.223219+0000","last_became_active":"2024-03-06T07:26:19.233235+0000","last_became_peered":"2024-03-06T07:26:19.233235+0000","last_unstale":"2024-03-06T07:26:21.223219+0000","last_undegraded":"2024-03-06T07:26:21.223219+0000","last_fullsized":"2024-03-06T07:26:21.223219+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:53.616839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302570+0000","last_change":"2024-03-06T07:26:20.234358+0000","last_active":"2024-03-06T07:26:21.302570+0000","last_peered":"2024-03-06T07:26:21.302570+0000","last_clean":"2024-03-06T07:26:21.302570+0000","last_became_active":"2024-03-06T07:26:20.234071+0000","last_became_peered":"2024-03-06T07:26:20.234071+0000","last_unstale":"2024-03-06T07:26:21.302570+0000","last_undegraded":"2024-03-06T07:26:21.302570+0000","last_fullsized":"2024-03-06T07:26:21.302570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:25.497989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303360+0000","last_change":"2024-03-06T07:26:19.236405+0000","last_active":"2024-03-06T07:26:21.303360+0000","last_peered":"2024-03-06T07:26:21.303360+0000","last_clean":"2024-03-06T07:26:21.303360+0000","last_became_active":"2024-03-06T07:26:19.235876+0000","last_became_peered":"2024-03-06T07:26:19.235876+0000","last_unstale":"2024-03-06T07:26:21.303360+0000","last_undegraded":"2024-03-06T07:26:21.303360+0000","last_fullsized":"2024-03-06T07:26:21.303360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:22.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302307+0000","last_change":"2024-03-06T07:26:20.246035+0000","last_active":"2024-03-06T07:26:21.302307+0000","last_peered":"2024-03-06T07:26:21.302307+0000","last_clean":"2024-03-06T07:26:21.302307+0000","last_became_active":"2024-03-06T07:26:20.245781+0000","last_became_peered":"2024-03-06T07:26:20.245781+0000","last_unstale":"2024-03-06T07:26:21.302307+0000","last_undegraded":"2024-03-06T07:26:21.302307+0000","last_fullsized":"2024-03-06T07:26:21.302307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:46.235986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.239939+0000","last_change":"2024-03-06T07:26:19.238646+0000","last_active":"2024-03-06T07:26:21.239939+0000","last_peered":"2024-03-06T07:26:21.239939+0000","last_clean":"2024-03-06T07:26:21.239939+0000","last_became_active":"2024-03-06T07:26:19.238376+0000","last_became_peered":"2024-03-06T07:26:19.238376+0000","last_unstale":"2024-03-06T07:26:21.239939+0000","last_undegraded":"2024-03-06T07:26:21.239939+0000","last_fullsized":"2024-03-06T07:26:21.239939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:50.887101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302338+0000","last_change":"2024-03-06T07:26:20.244446+0000","last_active":"2024-03-06T07:26:21.302338+0000","last_peered":"2024-03-06T07:26:21.302338+0000","last_clean":"2024-03-06T07:26:21.302338+0000","last_became_active":"2024-03-06T07:26:20.244184+0000","last_became_peered":"2024-03-06T07:26:20.244184+0000","last_unstale":"2024-03-06T07:26:21.302338+0000","last_undegraded":"2024-03-06T07:26:21.302338+0000","last_fullsized":"2024-03-06T07:26:21.302338+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:18.796554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.241743+0000","last_change":"2024-03-06T07:26:19.227589+0000","last_active":"2024-03-06T07:26:21.241743+0000","last_peered":"2024-03-06T07:26:21.241743+0000","last_clean":"2024-03-06T07:26:21.241743+0000","last_became_active":"2024-03-06T07:26:19.227308+0000","last_became_peered":"2024-03-06T07:26:19.227308+0000","last_unstale":"2024-03-06T07:26:21.241743+0000","last_undegraded":"2024-03-06T07:26:21.241743+0000","last_fullsized":"2024-03-06T07:26:21.241743+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:27.285562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.242255+0000","last_change":"2024-03-06T07:26:20.232889+0000","last_active":"2024-03-06T07:26:21.242255+0000","last_peered":"2024-03-06T07:26:21.242255+0000","last_clean":"2024-03-06T07:26:21.242255+0000","last_became_active":"2024-03-06T07:26:20.232605+0000","last_became_peered":"2024-03-06T07:26:20.232605+0000","last_unstale":"2024-03-06T07:26:21.242255+0000","last_undegraded":"2024-03-06T07:26:21.242255+0000","last_fullsized":"2024-03-06T07:26:21.242255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:41.481894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303742+0000","last_change":"2024-03-06T07:26:19.234268+0000","last_active":"2024-03-06T07:26:21.303742+0000","last_peered":"2024-03-06T07:26:21.303742+0000","last_clean":"2024-03-06T07:26:21.303742+0000","last_became_active":"2024-03-06T07:26:19.234000+0000","last_became_peered":"2024-03-06T07:26:19.234000+0000","last_unstale":"2024-03-06T07:26:21.303742+0000","last_undegraded":"2024-03-06T07:26:21.303742+0000","last_fullsized":"2024-03-06T07:26:21.303742+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:48.358539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.240112+0000","last_change":"2024-03-06T07:26:20.243426+0000","last_active":"2024-03-06T07:26:21.240112+0000","last_peered":"2024-03-06T07:26:21.240112+0000","last_clean":"2024-03-06T07:26:21.240112+0000","last_became_active":"2024-03-06T07:26:20.243163+0000","last_became_peered":"2024-03-06T07:26:20.243163+0000","last_unstale":"2024-03-06T07:26:21.240112+0000","last_undegraded":"2024-03-06T07:26:21.240112+0000","last_fullsized":"2024-03-06T07:26:21.240112+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:17.086279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"70'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.240245+0000","last_change":"2024-03-06T07:26:19.237743+0000","last_active":"2024-03-06T07:26:21.240245+0000","last_peered":"2024-03-06T07:26:21.240245+0000","last_clean":"2024-03-06T07:26:21.240245+0000","last_became_active":"2024-03-06T07:26:19.237394+0000","last_became_peered":"2024-03-06T07:26:19.237394+0000","last_unstale":"2024-03-06T07:26:21.240245+0000","last_undegraded":"2024-03-06T07:26:21.240245+0000","last_fullsized":"2024-03-06T07:26:21.240245+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:19.014952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.223116+0000","last_change":"2024-03-06T07:26:20.249280+0000","last_active":"2024-03-06T07:26:21.223116+0000","last_peered":"2024-03-06T07:26:21.223116+0000","last_clean":"2024-03-06T07:26:21.223116+0000","last_became_active":"2024-03-06T07:26:20.249001+0000","last_became_peered":"2024-03-06T07:26:20.249001+0000","last_unstale":"2024-03-06T07:26:21.223116+0000","last_undegraded":"2024-03-06T07:26:21.223116+0000","last_fullsized":"2024-03-06T07:26:21.223116+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:14.230935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303147+0000","last_change":"2024-03-06T07:26:19.230603+0000","last_active":"2024-03-06T07:26:21.303147+0000","last_peered":"2024-03-06T07:26:21.303147+0000","last_clean":"2024-03-06T07:26:21.303147+0000","last_became_active":"2024-03-06T07:26:19.230313+0000","last_became_peered":"2024-03-06T07:26:19.230313+0000","last_unstale":"2024-03-06T07:26:21.303147+0000","last_undegraded":"2024-03-06T07:26:21.303147+0000","last_fullsized":"2024-03-06T07:26:21.303147+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:32.933600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.224031+0000","last_change":"2024-03-06T07:26:20.241437+0000","last_active":"2024-03-06T07:26:21.224031+0000","last_peered":"2024-03-06T07:26:21.224031+0000","last_clean":"2024-03-06T07:26:21.224031+0000","last_became_active":"2024-03-06T07:26:20.241150+0000","last_became_peered":"2024-03-06T07:26:20.241150+0000","last_unstale":"2024-03-06T07:26:21.224031+0000","last_undegraded":"2024-03-06T07:26:21.224031+0000","last_fullsized":"2024-03-06T07:26:21.224031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:00.007499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.242076+0000","last_change":"2024-03-06T07:26:19.234353+0000","last_active":"2024-03-06T07:26:21.242076+0000","last_peered":"2024-03-06T07:26:21.242076+0000","last_clean":"2024-03-06T07:26:21.242076+0000","last_became_active":"2024-03-06T07:26:19.234077+0000","last_became_peered":"2024-03-06T07:26:19.234077+0000","last_unstale":"2024-03-06T07:26:21.242076+0000","last_undegraded":"2024-03-06T07:26:21.242076+0000","last_fullsized":"2024-03-06T07:26:21.242076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:21.966822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302899+0000","last_change":"2024-03-06T07:26:20.238064+0000","last_active":"2024-03-06T07:26:21.302899+0000","last_peered":"2024-03-06T07:26:21.302899+0000","last_clean":"2024-03-06T07:26:21.302899+0000","last_became_active":"2024-03-06T07:26:20.236930+0000","last_became_peered":"2024-03-06T07:26:20.236930+0000","last_unstale":"2024-03-06T07:26:21.302899+0000","last_undegraded":"2024-03-06T07:26:21.302899+0000","last_fullsized":"2024-03-06T07:26:21.302899+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:49.365299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302885+0000","last_change":"2024-03-06T07:26:19.237287+0000","last_active":"2024-03-06T07:26:21.302885+0000","last_peered":"2024-03-06T07:26:21.302885+0000","last_clean":"2024-03-06T07:26:21.302885+0000","last_became_active":"2024-03-06T07:26:19.237017+0000","last_became_peered":"2024-03-06T07:26:19.237017+0000","last_unstale":"2024-03-06T07:26:21.302885+0000","last_undegraded":"2024-03-06T07:26:21.302885+0000","last_fullsized":"2024-03-06T07:26:21.302885+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:57.254589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303291+0000","last_change":"2024-03-06T07:26:20.254760+0000","last_active":"2024-03-06T07:26:21.303291+0000","last_peered":"2024-03-06T07:26:21.303291+0000","last_clean":"2024-03-06T07:26:21.303291+0000","last_became_active":"2024-03-06T07:26:20.254484+0000","last_became_peered":"2024-03-06T07:26:20.254484+0000","last_unstale":"2024-03-06T07:26:21.303291+0000","last_undegraded":"2024-03-06T07:26:21.303291+0000","last_fullsized":"2024-03-06T07:26:21.303291+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:34.087751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.221541+0000","last_change":"2024-03-06T07:26:19.229645+0000","last_active":"2024-03-06T07:26:21.221541+0000","last_peered":"2024-03-06T07:26:21.221541+0000","last_clean":"2024-03-06T07:26:21.221541+0000","last_became_active":"2024-03-06T07:26:19.229309+0000","last_became_peered":"2024-03-06T07:26:19.229309+0000","last_unstale":"2024-03-06T07:26:21.221541+0000","last_undegraded":"2024-03-06T07:26:21.221541+0000","last_fullsized":"2024-03-06T07:26:21.221541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:54.603601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.221672+0000","last_change":"2024-03-06T07:26:20.239740+0000","last_active":"2024-03-06T07:26:21.221672+0000","last_peered":"2024-03-06T07:26:21.221672+0000","last_clean":"2024-03-06T07:26:21.221672+0000","last_became_active":"2024-03-06T07:26:20.239477+0000","last_became_peered":"2024-03-06T07:26:20.239477+0000","last_unstale":"2024-03-06T07:26:21.221672+0000","last_undegraded":"2024-03-06T07:26:21.221672+0000","last_fullsized":"2024-03-06T07:26:21.221672+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:57.375714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.222817+0000","last_change":"2024-03-06T07:26:19.231241+0000","last_active":"2024-03-06T07:26:21.222817+0000","last_peered":"2024-03-06T07:26:21.222817+0000","last_clean":"2024-03-06T07:26:21.222817+0000","last_became_active":"2024-03-06T07:26:19.230977+0000","last_became_peered":"2024-03-06T07:26:19.230977+0000","last_unstale":"2024-03-06T07:26:21.222817+0000","last_undegraded":"2024-03-06T07:26:21.222817+0000","last_fullsized":"2024-03-06T07:26:21.222817+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:58.705738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303100+0000","last_change":"2024-03-06T07:26:20.252240+0000","last_active":"2024-03-06T07:26:21.303100+0000","last_peered":"2024-03-06T07:26:21.303100+0000","last_clean":"2024-03-06T07:26:21.303100+0000","last_became_active":"2024-03-06T07:26:20.251236+0000","last_became_peered":"2024-03-06T07:26:20.251236+0000","last_unstale":"2024-03-06T07:26:21.303100+0000","last_undegraded":"2024-03-06T07:26:21.303100+0000","last_fullsized":"2024-03-06T07:26:21.303100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:23.186073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303678+0000","last_change":"2024-03-06T07:26:19.238002+0000","last_active":"2024-03-06T07:26:21.303678+0000","last_peered":"2024-03-06T07:26:21.303678+0000","last_clean":"2024-03-06T07:26:21.303678+0000","last_became_active":"2024-03-06T07:26:19.237710+0000","last_became_peered":"2024-03-06T07:26:19.237710+0000","last_unstale":"2024-03-06T07:26:21.303678+0000","last_undegraded":"2024-03-06T07:26:21.303678+0000","last_fullsized":"2024-03-06T07:26:21.303678+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:58.705738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.302505+0000","last_change":"2024-03-06T07:26:20.245183+0000","last_active":"2024-03-06T07:26:21.302505+0000","last_peered":"2024-03-06T07:26:21.302505+0000","last_clean":"2024-03-06T07:26:21.302505+0000","last_became_active":"2024-03-06T07:26:20.244921+0000","last_became_peered":"2024-03-06T07:26:20.244921+0000","last_unstale":"2024-03-06T07:26:21.302505+0000","last_undegraded":"2024-03-06T07:26:21.302505+0000","last_fullsized":"2024-03-06T07:26:21.302505+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:19.211880+0000","last_clean_scrub_stamp":"2024-03-06T07:26:19.211880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:55.863041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.303188+0000","last_change":"2024-03-06T07:26:19.236691+0000","last_active":"2024-03-06T07:26:21.303188+0000","last_peered":"2024-03-06T07:26:21.303188+0000","last_clean":"2024-03-06T07:26:21.303188+0000","last_became_active":"2024-03-06T07:26:19.236432+0000","last_became_peered":"2024-03-06T07:26:19.236432+0000","last_unstale":"2024-03-06T07:26:21.303188+0000","last_undegraded":"2024-03-06T07:26:21.303188+0000","last_fullsized":"2024-03-06T07:26:21.303188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:18.205120+0000","last_clean_scrub_stamp":"2024-03-06T07:26:18.205120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:56.368954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":265,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.222635+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:26:21.222635+0000","last_peered":"2024-03-06T07:26:21.222635+0000","last_clean":"2024-03-06T07:26:21.222635+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:26:21.222635+0000","last_undegraded":"2024-03-06T07:26:21.222635+0000","last_fullsized":"2024-03-06T07:26:21.222635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1454080,"data_stored":947800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":860160,"data_stored":356908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":856064,"data_stored":356906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":856064,"data_stored":356886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":860160,"data_stored":356928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":864256,"data_stored":358428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":359044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1445888,"data_stored":947184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:29.050 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:26:29.050 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:26:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.198+0000 7f60f3725640 1 Processor -- start 2024-03-06T07:26:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.199+0000 7f60f3725640 1 -- start start 2024-03-06T07:26:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.200+0000 7f60f3725640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f60ec0ceea0 0x7f60ec0d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.200+0000 7f60f3725640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec0b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.200+0000 7f60f3725640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ec0d3740 con 0x7f60ec15dd90 2024-03-06T07:26:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.200+0000 7f60f3725640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ec0d3400 con 0x7f60ec0ceea0 2024-03-06T07:26:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.200+0000 7f60f3725640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0d3120 con 0x7f60ec0b6d10 2024-03-06T07:26:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.200+0000 7f60f1cc2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec0b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.200+0000 7f60f1cc2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec0b71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:36734/0 (socket says 172.21.15.27:36734) 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.200+0000 7f60f0cc0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f60ec0ceea0 0x7f60ec0d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.200+0000 7f60f1cc2640 1 -- 172.21.15.27:0/4229032723 learned_addr learned my addr 172.21.15.27:0/4229032723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.201+0000 7f60e3fff640 1 -- 172.21.15.27:0/4229032723 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237507091 0 0) 0x7f60ec0d3740 con 0x7f60ec15dd90 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.201+0000 7f60e3fff640 1 -- 172.21.15.27:0/4229032723 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d8003520 con 0x7f60ec15dd90 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.201+0000 7f60f1cc2640 1 -- 172.21.15.27:0/4229032723 >> v2:172.21.15.69:3300/0 conn(0x7f60ec0ceea0 msgr2=0x7f60ec0d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.201+0000 7f60f1cc2640 1 --2- 172.21.15.27:0/4229032723 >> v2:172.21.15.69:3300/0 conn(0x7f60ec0ceea0 0x7f60ec0d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.201+0000 7f60f1cc2640 1 -- 172.21.15.27:0/4229032723 >> v1:172.21.15.69:6789/0 conn(0x7f60ec15dd90 legacy=0x7f60ec0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.201+0000 7f60f1cc2640 1 -- 172.21.15.27:0/4229032723 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec0d39e0 con 0x7f60ec0b6d10 2024-03-06T07:26:29.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.201+0000 7f60f0cc0640 1 --2- 172.21.15.27:0/4229032723 >> v2:172.21.15.69:3300/0 conn(0x7f60ec0ceea0 0x7f60ec0d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60f1cc2640 1 --2- 172.21.15.27:0/4229032723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec0b71a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009e50 tx=0x7f60e402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd4e38f43fc56a24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60e3fff640 1 -- 172.21.15.27:0/4229032723 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e402cb90 con 0x7f60ec0b6d10 2024-03-06T07:26:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60e3fff640 1 -- 172.21.15.27:0/4229032723 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e402cd30 con 0x7f60ec0b6d10 2024-03-06T07:26:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60e3fff640 1 -- 172.21.15.27:0/4229032723 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e4034610 con 0x7f60ec0b6d10 2024-03-06T07:26:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60f3725640 1 -- 172.21.15.27:0/4229032723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 msgr2=0x7f60ec0b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60f3725640 1 --2- 172.21.15.27:0/4229032723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec0b71a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009e50 tx=0x7f60e402bff0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60f3725640 1 -- 172.21.15.27:0/4229032723 shutdown_connections 2024-03-06T07:26:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60f3725640 1 --2- 172.21.15.27:0/4229032723 >> v2:172.21.15.69:3300/0 conn(0x7f60ec0ceea0 0x7f60ec0d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60f3725640 1 --2- 172.21.15.27:0/4229032723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec0b71a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60f3725640 1 -- 172.21.15.27:0/4229032723 >> 172.21.15.27:0/4229032723 conn(0x7f60ec071960 msgr2=0x7f60ec071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60f3725640 1 -- 172.21.15.27:0/4229032723 shutdown_connections 2024-03-06T07:26:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.202+0000 7f60f3725640 1 -- 172.21.15.27:0/4229032723 wait complete. 2024-03-06T07:26:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.203+0000 7f60f3725640 1 Processor -- start 2024-03-06T07:26:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.204+0000 7f60f3725640 1 -- start start 2024-03-06T07:26:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f3725640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec15e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f3725640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60ec0ceea0 0x7f60ec160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f3725640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ec15dd90 0x7f60ec15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f3725640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0d48f0 con 0x7f60ec0ceea0 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f3725640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec15e170 con 0x7f60ec15dd90 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f3725640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0d4a70 con 0x7f60ec0b6d10 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f0cc0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60ec0ceea0 0x7f60ec160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f0cc0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60ec0ceea0 0x7f60ec160c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47518/0 (socket says 172.21.15.27:47518) 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f0cc0640 1 -- 172.21.15.27:0/1984479262 learned_addr learned my addr 172.21.15.27:0/1984479262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f1cc2640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ec15dd90 0x7f60ec15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f14c1640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec15e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f0cc0640 1 -- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 msgr2=0x7f60ec15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f0cc0640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec15e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f0cc0640 1 -- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ec15dd90 msgr2=0x7f60ec15ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f0cc0640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ec15dd90 0x7f60ec15ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f0cc0640 1 -- 172.21.15.27:0/1984479262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec15eb50 con 0x7f60ec0ceea0 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.205+0000 7f60f1cc2640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ec15dd90 0x7f60ec15ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.206+0000 7f60f0cc0640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60ec0ceea0 0x7f60ec160c20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f60e800ebe0 tx=0x7f60e800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.206+0000 7f60e1ffb640 1 -- 172.21.15.27:0/1984479262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8017070 con 0x7f60ec0ceea0 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.206+0000 7f60e1ffb640 1 -- 172.21.15.27:0/1984479262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e800cc30 con 0x7f60ec0ceea0 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.206+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ec0cd350 con 0x7f60ec0ceea0 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.206+0000 7f60e1ffb640 1 -- 172.21.15.27:0/1984479262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8013470 con 0x7f60ec0ceea0 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.206+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ec0cdbf0 con 0x7f60ec0ceea0 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.207+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ec0cd520 con 0x7f60ec0ceea0 2024-03-06T07:26:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.208+0000 7f60e1ffb640 1 -- 172.21.15.27:0/1984479262 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60e801a070 con 0x7f60ec0ceea0 2024-03-06T07:26:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.209+0000 7f60e1ffb640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60d006b4d0 0x7f60d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.209+0000 7f60e1ffb640 1 -- 172.21.15.27:0/1984479262 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f60e808a4a0 con 0x7f60ec0ceea0 2024-03-06T07:26:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.209+0000 7f60f14c1640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60d006b4d0 0x7f60d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.210+0000 7f60f14c1640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60d006b4d0 0x7f60d006d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0d3910 tx=0x7f60dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.213+0000 7f60e1ffb640 1 -- 172.21.15.27:0/1984479262 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60e8059370 con 0x7f60ec0ceea0 2024-03-06T07:26:29.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.466+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f60ec072a30 con 0x7f60ec0ceea0 2024-03-06T07:26:29.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.468+0000 7f60e1ffb640 1 -- 172.21.15.27:0/1984479262 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f60e8058c70 con 0x7f60ec0ceea0 2024-03-06T07:26:29.470 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:26:29.470 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:26:29.470 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-03-06T07:26:29.470 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-03-06T07:26:29.470 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:29.471 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-03-06T07:26:29.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.470+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60d006b4d0 msgr2=0x7f60d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.470+0000 7f60f3725640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60d006b4d0 0x7f60d006d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0d3910 tx=0x7f60dc009040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.470+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60ec0ceea0 msgr2=0x7f60ec160c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.470+0000 7f60f3725640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60ec0ceea0 0x7f60ec160c20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f60e800ebe0 tx=0x7f60e800c520 comp rx=0 tx=0).stop 2024-03-06T07:26:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.471+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 shutdown_connections 2024-03-06T07:26:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.471+0000 7f60f3725640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60d006b4d0 0x7f60d006d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.471+0000 7f60f3725640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60ec15dd90 0x7f60ec15ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.471+0000 7f60f3725640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60ec0ceea0 0x7f60ec160c20 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.471+0000 7f60f3725640 1 --2- 172.21.15.27:0/1984479262 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60ec0b6d10 0x7f60ec15e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.471+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 >> 172.21.15.27:0/1984479262 conn(0x7f60ec071960 msgr2=0x7f60ec0b4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.471+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 shutdown_connections 2024-03-06T07:26:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.471+0000 7f60f3725640 1 -- 172.21.15.27:0/1984479262 wait complete. 2024-03-06T07:26:29.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.588+0000 7f0e92919640 1 Processor -- start 2024-03-06T07:26:29.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.588+0000 7f0e92919640 1 -- start start 2024-03-06T07:26:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e92919640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e8c15df00 0x7f0e8c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e92919640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e8c0b95a0 0x7f0e8c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e92919640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8c0cea30 con 0x7f0e8c0b9a30 2024-03-06T07:26:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e92919640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c0d2af0 con 0x7f0e8c0b95a0 2024-03-06T07:26:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e92919640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c0cf3c0 con 0x7f0e8c15df00 2024-03-06T07:26:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e8b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e8c0b95a0 0x7f0e8c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e8b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e8c0b95a0 0x7f0e8c0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47520/0 (socket says 172.21.15.27:47520) 2024-03-06T07:26:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e8b7fe640 1 -- 172.21.15.27:0/4103776923 learned_addr learned my addr 172.21.15.27:0/4103776923 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e8bfff640 1 --2- 172.21.15.27:0/4103776923 >> v2:172.21.15.69:3300/0 conn(0x7f0e8c15df00 0x7f0e8c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e8b7fe640 1 -- 172.21.15.27:0/4103776923 >> v2:172.21.15.69:3300/0 conn(0x7f0e8c15df00 msgr2=0x7f0e8c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e8b7fe640 1 --2- 172.21.15.27:0/4103776923 >> v2:172.21.15.69:3300/0 conn(0x7f0e8c15df00 0x7f0e8c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e8b7fe640 1 -- 172.21.15.27:0/4103776923 >> v1:172.21.15.69:6789/0 conn(0x7f0e8c0b9a30 legacy=0x7f0e8c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.589+0000 7f0e8b7fe640 1 -- 172.21.15.27:0/4103776923 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c0cf0e0 con 0x7f0e8c0b95a0 2024-03-06T07:26:29.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.590+0000 7f0e8b7fe640 1 --2- 172.21.15.27:0/4103776923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0b95a0 0x7f0e8c0b5700 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0d11b0 tx=0x7f0e8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c189fa098f3465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.590+0000 7f0e8affd640 1 -- 172.21.15.27:0/4103776923 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80035020 con 0x7f0e8c0b95a0 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.590+0000 7f0e8affd640 1 -- 172.21.15.27:0/4103776923 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e80031870 con 0x7f0e8c0b95a0 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.590+0000 7f0e8affd640 1 -- 172.21.15.27:0/4103776923 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80031c40 con 0x7f0e8c0b95a0 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.591+0000 7f0e92919640 1 -- 172.21.15.27:0/4103776923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0b95a0 msgr2=0x7f0e8c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.591+0000 7f0e92919640 1 --2- 172.21.15.27:0/4103776923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0b95a0 0x7f0e8c0b5700 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0d11b0 tx=0x7f0e8002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.591+0000 7f0e92919640 1 -- 172.21.15.27:0/4103776923 shutdown_connections 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.591+0000 7f0e92919640 1 --2- 172.21.15.27:0/4103776923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0b95a0 0x7f0e8c0b5700 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.591+0000 7f0e92919640 1 --2- 172.21.15.27:0/4103776923 >> v2:172.21.15.69:3300/0 conn(0x7f0e8c15df00 0x7f0e8c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.591+0000 7f0e92919640 1 -- 172.21.15.27:0/4103776923 >> 172.21.15.27:0/4103776923 conn(0x7f0e8c071960 msgr2=0x7f0e8c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.591+0000 7f0e92919640 1 -- 172.21.15.27:0/4103776923 shutdown_connections 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.591+0000 7f0e92919640 1 -- 172.21.15.27:0/4103776923 wait complete. 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.592+0000 7f0e92919640 1 Processor -- start 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.592+0000 7f0e92919640 1 -- start start 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e92919640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e8c0b95a0 0x7f0e8c160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e92919640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e8c15df00 0x7f0e8c0cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e92919640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0c8bf0 0x7f0e8c0c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e92919640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c0cec70 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e92919640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c0ce750 con 0x7f0e8c15df00 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e92919640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c0cedf0 con 0x7f0e8c0b95a0 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e90eb6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0c8bf0 0x7f0e8c0c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e90eb6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0c8bf0 0x7f0e8c0c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47532/0 (socket says 172.21.15.27:47532) 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e90eb6640 1 -- 172.21.15.27:0/1832106563 learned_addr learned my addr 172.21.15.27:0/1832106563 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e8b7fe640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e8c15df00 0x7f0e8c0cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e8bfff640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e8c0b95a0 0x7f0e8c160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e90eb6640 1 -- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e8c0b95a0 msgr2=0x7f0e8c160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e90eb6640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e8c0b95a0 0x7f0e8c160520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e90eb6640 1 -- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e8c15df00 msgr2=0x7f0e8c0cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e90eb6640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e8c15df00 0x7f0e8c0cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.593+0000 7f0e90eb6640 1 -- 172.21.15.27:0/1832106563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c0c7b50 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.594+0000 7f0e90eb6640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0c8bf0 0x7f0e8c0c9070 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400c780 tx=0x7f0e8400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.594+0000 7f0e88ff9640 1 -- 172.21.15.27:0/1832106563 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84004b60 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.594+0000 7f0e88ff9640 1 -- 172.21.15.27:0/1832106563 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e84004d00 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.594+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c0c7e30 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.594+0000 7f0e88ff9640 1 -- 172.21.15.27:0/1832106563 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8400aa90 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.594+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8c0c83f0 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.595+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8c067ff0 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.595+0000 7f0e88ff9640 1 -- 172.21.15.27:0/1832106563 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e8400ac30 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.596+0000 7f0e88ff9640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e7806b4d0 0x7f0e7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.596+0000 7f0e88ff9640 1 -- 172.21.15.27:0/1832106563 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0e8408a490 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.596+0000 7f0e8bfff640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e7806b4d0 0x7f0e7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.597+0000 7f0e8bfff640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e7806b4d0 0x7f0e7806d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c006fd0 tx=0x7f0e7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.600+0000 7f0e88ff9640 1 -- 172.21.15.27:0/1832106563 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e84020070 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.820+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e8c15d400 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.821+0000 7f0e88ff9640 1 -- 172.21.15.27:0/1832106563 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f0e840592d0 con 0x7f0e8c0c8bf0 2024-03-06T07:26:29.823 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:29.823 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:29.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e7806b4d0 msgr2=0x7f0e7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e7806b4d0 0x7f0e7806d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c006fd0 tx=0x7f0e7c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0c8bf0 msgr2=0x7f0e8c0c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0c8bf0 0x7f0e8c0c9070 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400c780 tx=0x7f0e8400cc40 comp rx=0 tx=0).stop 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 shutdown_connections 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e7806b4d0 0x7f0e7806d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e8c0c8bf0 0x7f0e8c0c9070 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e8c15df00 0x7f0e8c0cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 --2- 172.21.15.27:0/1832106563 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e8c0b95a0 0x7f0e8c160520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 >> 172.21.15.27:0/1832106563 conn(0x7f0e8c071960 msgr2=0x7f0e8c1618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 shutdown_connections 2024-03-06T07:26:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:29.823+0000 7f0e92919640 1 -- 172.21.15.27:0/1832106563 wait complete. 2024-03-06T07:26:30.238 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:19.591416+0000', 'modified': '2024-03-06T07:26:22.766901+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6606}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6606': {'gid': 6606, 'name': 'a', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.27:6835/2393218771', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 2393218771}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 2393218771}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6606, 'qdb_cluster': [6606]} 2024-03-06T07:26:30.238 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:30.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:26:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.345+0000 7f59416e1640 1 Processor -- start 2024-03-06T07:26:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.345+0000 7f59416e1640 1 -- start start 2024-03-06T07:26:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f59416e1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f593c15bac0 0x7f593c15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f59416e1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c06d400 0x7f593c0ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f59416e1640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c0729b0 con 0x7f593c0aec40 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f59416e1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593c0af0b0 con 0x7f593c15bac0 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f59416e1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c06cf70 con 0x7f593c06d400 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f593b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f593c0aec40 0x7f593c070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47044/0 (socket says 172.21.15.27:47044) 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f593b7fe640 1 -- 172.21.15.27:0/2859710458 learned_addr learned my addr 172.21.15.27:0/2859710458 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826144561 0 0) 0x7f593c0729b0 con 0x7f593c0aec40 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f593affd640 1 --2- 172.21.15.27:0/2859710458 >> v2:172.21.15.69:3300/0 conn(0x7f593c15bac0 0x7f593c15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f593a7fc640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c06d400 0x7f593c0ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5928003520 con 0x7f593c0aec40 2024-03-06T07:26:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2998018804 0 0) 0x7f5928003520 con 0x7f593c0aec40 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c06d400 msgr2=0x7f593c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c06d400 0x7f593c0ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 >> v2:172.21.15.69:3300/0 conn(0x7f593c15bac0 msgr2=0x7f593c15cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 --2- 172.21.15.27:0/2859710458 >> v2:172.21.15.69:3300/0 conn(0x7f593c15bac0 0x7f593c15cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c06c920 con 0x7f593c0aec40 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f5934001670 con 0x7f593c0aec40 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 >> v1:172.21.15.27:6789/0 conn(0x7f593c0aec40 legacy=0x7f593c070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f5928003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c06d400 0x7f5928004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5928004d40 0x7f59280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.346+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0af0b0 con 0x7f59280039b0 2024-03-06T07:26:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c06cf70 con 0x7f5928004d40 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592800b9c0 con 0x7f593c06d400 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f593affd640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f5928003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f593b7fe640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5928004d40 0x7f59280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f593a7fc640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c06d400 0x7f5928004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f593affd640 1 -- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c06d400 msgr2=0x7f5928004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f593affd640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c06d400 0x7f5928004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f593affd640 1 -- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5928004d40 msgr2=0x7f59280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f593affd640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5928004d40 0x7f59280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.347+0000 7f593affd640 1 -- 172.21.15.27:0/2859710458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5928003520 con 0x7f59280039b0 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f593affd640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f5928003dd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f592c00e170 tx=0x7f5928005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ab8ec4eba66ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c039070 con 0x7f59280039b0 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c02cc20 con 0x7f59280039b0 2024-03-06T07:26:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f5939ffb640 1 -- 172.21.15.27:0/2859710458 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c009c10 con 0x7f59280039b0 2024-03-06T07:26:30.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f59416e1640 1 -- 172.21.15.27:0/2859710458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 msgr2=0x7f5928003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f59416e1640 1 --2- 172.21.15.27:0/2859710458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f5928003dd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f592c00e170 tx=0x7f5928005830 comp rx=0 tx=0).stop 2024-03-06T07:26:30.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f593b7fe640 1 -- 172.21.15.27:0/2859710458 reap_dead start 2024-03-06T07:26:30.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f59416e1640 1 -- 172.21.15.27:0/2859710458 shutdown_connections 2024-03-06T07:26:30.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f59416e1640 1 -- 172.21.15.27:0/2859710458 >> 172.21.15.27:0/2859710458 conn(0x7f593c066d00 msgr2=0x7f593c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f59416e1640 1 -- 172.21.15.27:0/2859710458 shutdown_connections 2024-03-06T07:26:30.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.348+0000 7f59416e1640 1 -- 172.21.15.27:0/2859710458 wait complete. 2024-03-06T07:26:30.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.350+0000 7f59416e1640 1 Processor -- start 2024-03-06T07:26:30.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.350+0000 7f59416e1640 1 -- start start 2024-03-06T07:26:30.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.350+0000 7f59416e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f593c15a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.350+0000 7f59416e1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c1535a0 0x7f593c151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f59416e1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f593c1566d0 0x7f593c156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f59416e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c15bea0 con 0x7f59280039b0 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f59416e1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0cc490 con 0x7f593c1566d0 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f59416e1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c15c020 con 0x7f593c1535a0 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f593c15a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f593c15a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47554/0 (socket says 172.21.15.27:47554) 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593affd640 1 -- 172.21.15.27:0/3056130210 learned_addr learned my addr 172.21.15.27:0/3056130210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593a7fc640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c1535a0 0x7f593c151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593b7fe640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f593c1566d0 0x7f593c156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593affd640 1 -- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c1535a0 msgr2=0x7f593c151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593affd640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c1535a0 0x7f593c151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593affd640 1 -- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f593c1566d0 msgr2=0x7f593c156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593affd640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f593c1566d0 0x7f593c156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593affd640 1 -- 172.21.15.27:0/3056130210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c158de0 con 0x7f59280039b0 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f593affd640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f593c15a160 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f592c009f90 tx=0x7f592c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f59337fe640 1 -- 172.21.15.27:0/3056130210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c039050 con 0x7f59280039b0 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f59337fe640 1 -- 172.21.15.27:0/3056130210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c045070 con 0x7f59280039b0 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f59416e1640 1 -- 172.21.15.27:0/3056130210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c1590c0 con 0x7f59280039b0 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f59337fe640 1 -- 172.21.15.27:0/3056130210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c009c10 con 0x7f59280039b0 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.351+0000 7f59416e1640 1 -- 172.21.15.27:0/3056130210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c159680 con 0x7f59280039b0 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.353+0000 7f59337fe640 1 -- 172.21.15.27:0/3056130210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f592c005d10 con 0x7f59280039b0 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.354+0000 7f59337fe640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f591c06b500 0x7f591c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.354+0000 7f59337fe640 1 -- 172.21.15.27:0/3056130210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f592c0ac8b0 con 0x7f59280039b0 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.354+0000 7f593a7fc640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f591c06b500 0x7f591c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.354+0000 7f594439c740 1 Processor -- start 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.354+0000 7f594439c740 1 -- start start 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f593a7fc640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f591c06b500 0x7f591c06d9c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f593c1572e0 tx=0x7f5924008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f594439c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1935f870 0x55db1935fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f594439c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db193638c0 0x55db19365360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f594439c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db1938f600 0x55db1938fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f594439c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db192c96b0 con 0x55db1935f870 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f594439c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db19272a60 con 0x55db1938f600 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f594439c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db192c5750 con 0x55db193638c0 2024-03-06T07:26:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f593affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1935f870 0x55db1935fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f593affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1935f870 0x55db1935fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47562/0 (socket says 172.21.15.27:47562) 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f593affd640 1 -- 172.21.15.27:0/1660961109 learned_addr learned my addr 172.21.15.27:0/1660961109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f593b7fe640 1 --2- 172.21.15.27:0/1660961109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db1938f600 0x55db1938fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.355+0000 7f593a7fc640 1 --2- 172.21.15.27:0/1660961109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db193638c0 0x55db19365360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.356+0000 7f593affd640 1 -- 172.21.15.27:0/1660961109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db193638c0 msgr2=0x55db19365360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.356+0000 7f593affd640 1 --2- 172.21.15.27:0/1660961109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db193638c0 0x55db19365360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.356+0000 7f593affd640 1 -- 172.21.15.27:0/1660961109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db1938f600 msgr2=0x55db1938fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.356+0000 7f593affd640 1 --2- 172.21.15.27:0/1660961109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db1938f600 0x55db1938fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.356+0000 7f593affd640 1 -- 172.21.15.27:0/1660961109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db19290f30 con 0x55db1935f870 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.356+0000 7f593affd640 1 --2- 172.21.15.27:0/1660961109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1935f870 0x55db1935fc50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f592c0774e0 tx=0x7f592c07af80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35742e55e7739ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.356+0000 7f5930ff9640 1 -- 172.21.15.27:0/1660961109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c0a9070 con 0x55db1935f870 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.356+0000 7f5930ff9640 1 -- 172.21.15.27:0/1660961109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c092030 con 0x55db1935f870 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.356+0000 7f5930ff9640 1 -- 172.21.15.27:0/1660961109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c07f1d0 con 0x55db1935f870 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 -- 172.21.15.27:0/1660961109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1935f870 msgr2=0x55db1935fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 --2- 172.21.15.27:0/1660961109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1935f870 0x55db1935fc50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f592c0774e0 tx=0x7f592c07af80 comp rx=0 tx=0).stop 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 -- 172.21.15.27:0/1660961109 shutdown_connections 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 --2- 172.21.15.27:0/1660961109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db1938f600 0x55db1938fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 --2- 172.21.15.27:0/1660961109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db193638c0 0x55db19365360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 --2- 172.21.15.27:0/1660961109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1935f870 0x55db1935fc50 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 -- 172.21.15.27:0/1660961109 >> 172.21.15.27:0/1660961109 conn(0x55db193618f0 msgr2=0x55db193631b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 -- 172.21.15.27:0/1660961109 shutdown_connections 2024-03-06T07:26:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 -- 172.21.15.27:0/1660961109 wait complete. 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 -- start start 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.357+0000 7f594439c740 1 Processor -- start 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f594439c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db19392f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f594439c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 0x55db1938e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f594439c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db19386790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f594439c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db19272a60 con 0x55db1938f600 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f594439c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db19285ea0 con 0x55db19382cd0 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f594439c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db192c96b0 con 0x55db1938e8f0 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f594439c740 10 client.0 _ll_register_callbacks cb 0x55db19285870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f593affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db19392f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f593affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db19392f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47576/0 (socket says 172.21.15.27:47576) 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f593affd640 1 -- 172.21.15.27:0/876504181 learned_addr learned my addr 172.21.15.27:0/876504181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f593a7fc640 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 0x55db1938e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f593b7fe640 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db19386790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f593affd640 1 -- 172.21.15.27:0/876504181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 msgr2=0x55db19386790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f593affd640 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db19386790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f593affd640 1 -- 172.21.15.27:0/876504181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 msgr2=0x55db1938e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.358+0000 7f593affd640 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 0x55db1938e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f593affd640 1 -- 172.21.15.27:0/876504181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db19286270 con 0x55db1938f600 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f593affd640 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db19392f60 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f592c000f80 tx=0x7f592c04b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6085ed1871ab1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f5919ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f5919ffb640 1 -- 172.21.15.27:0/876504181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c0a9030 con 0x55db1938f600 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f5919ffb640 1 -- 172.21.15.27:0/876504181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f592c083300 con 0x55db1938f600 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f594439c740 1 -- 172.21.15.27:0/876504181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f592c092030 con 0x55db1938f600 2024-03-06T07:26:30.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f5919ffb640 1 -- 172.21.15.27:0/876504181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f592c07f1d0 con 0x55db1938f600 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f594439c740 10 client.6780 fetch_fsmap learned FSMap version 44 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f594439c740 1 -- 172.21.15.27:0/876504181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x55db18fde2d0 con 0x55db1938f600 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f5919ffb640 1 -- 172.21.15.27:0/876504181 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c092030 con 0x55db1938f600 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f5919ffb640 1 -- 172.21.15.27:0/876504181 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 44) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f592c07a4d0 con 0x55db1938f600 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f594439c740 10 client.6780 fetch_fsmap finished waiting for FSMap version 44 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f594439c740 10 client.6780 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f594439c740 20 client.6780 populate_metadata read hostname 'smithi027' 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f594439c740 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] conn(0x55db193833b0 0x55db19383790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f594439c740 4 client.6780 mds_command: new command op to 6606 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f594439c740 1 -- 172.21.15.27:0/876504181 --> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db192d6bf0 con 0x55db193833b0 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.359+0000 7f593a7fc640 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] conn(0x55db193833b0 0x55db19383790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.360+0000 7f5919ffb640 10 client.6780 ms_handle_connect on v2:172.21.15.27:6833/2393218771 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.360+0000 7f593a7fc640 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] conn(0x55db193833b0 0x55db19383790 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e503c16e1f14f0d8 server_cookie=607723e5d9d07c08 in_seq=0 out_seq=0 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.360+0000 7f5919ffb640 1 -- 172.21.15.27:0/876504181 <== mds.0 v2:172.21.15.27:6833/2393218771 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55db192d6bf0 con 0x55db193833b0 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.360+0000 7f5919ffb640 10 client.6780 handle_command_reply: tid=0 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.360+0000 7f594439c740 1 client.6780 shutdown 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 -- 172.21.15.27:0/876504181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 msgr2=0x55db19392f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db19392f60 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f592c000f80 tx=0x7f592c04b040 comp rx=0 tx=0).stop 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 -- 172.21.15.27:0/876504181 shutdown_connections 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] conn(0x55db193833b0 0x55db19383790 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db19386790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f5919ffb640 0 client.6780 ms_handle_reset on v2:172.21.15.27:6833/2393218771 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 0x55db1938e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 --2- 172.21.15.27:0/876504181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db19392f60 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 -- 172.21.15.27:0/876504181 >> 172.21.15.27:0/876504181 conn(0x55db1935f870 msgr2=0x55db1938d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 -- 172.21.15.27:0/876504181 shutdown_connections 2024-03-06T07:26:30.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 1 -- 172.21.15.27:0/876504181 wait complete. 2024-03-06T07:26:30.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.361+0000 7f594439c740 20 client.6780 trim_cache size 0 max 16384 2024-03-06T07:26:30.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f594439c740 1 Processor -- start 2024-03-06T07:26:30.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f594439c740 1 -- start start 2024-03-06T07:26:30.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f594439c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db19382cd0 0x55db19383b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f594439c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db1938b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f594439c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db1938f600 0x55db1938b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f594439c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c1fd8f0 con 0x55db19382cd0 2024-03-06T07:26:30.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f594439c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c158760 con 0x55db1938f600 2024-03-06T07:26:30.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f594439c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c1fd2f0 con 0x55db1938e8f0 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f593affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db19382cd0 0x55db19383b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f593affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db19382cd0 0x55db19383b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47590/0 (socket says 172.21.15.27:47590) 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.378+0000 7f593affd640 1 -- 172.21.15.27:0/3245544042 learned_addr learned my addr 172.21.15.27:0/3245544042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f593affd640 1 -- 172.21.15.27:0/3245544042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 msgr2=0x55db1938b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f593affd640 1 --2- 172.21.15.27:0/3245544042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db1938b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f593affd640 1 -- 172.21.15.27:0/3245544042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db1938f600 msgr2=0x55db1938b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f593affd640 1 --2- 172.21.15.27:0/3245544042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db1938f600 0x55db1938b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f593affd640 1 -- 172.21.15.27:0/3245544042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db190a86b0 con 0x55db19382cd0 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f593affd640 1 --2- 172.21.15.27:0/3245544042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db19382cd0 0x55db19383b90 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f592c083870 tx=0x7f592c07e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5b5b5e102c964b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f591bfff640 1 -- 172.21.15.27:0/3245544042 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c0a9030 con 0x55db19382cd0 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f591bfff640 1 -- 172.21.15.27:0/3245544042 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c092040 con 0x55db19382cd0 2024-03-06T07:26:30.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f591bfff640 1 -- 172.21.15.27:0/3245544042 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c077320 con 0x55db19382cd0 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f594439c740 1 -- 172.21.15.27:0/3245544042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db19382cd0 msgr2=0x55db19383b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f594439c740 1 --2- 172.21.15.27:0/3245544042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db19382cd0 0x55db19383b90 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f592c083870 tx=0x7f592c07e000 comp rx=0 tx=0).stop 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f594439c740 1 -- 172.21.15.27:0/3245544042 shutdown_connections 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.379+0000 7f594439c740 1 --2- 172.21.15.27:0/3245544042 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db1938f600 0x55db1938b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.380+0000 7f594439c740 1 --2- 172.21.15.27:0/3245544042 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db1938b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.380+0000 7f594439c740 1 --2- 172.21.15.27:0/3245544042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db19382cd0 0x55db19383b90 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.380+0000 7f594439c740 1 -- 172.21.15.27:0/3245544042 >> 172.21.15.27:0/3245544042 conn(0x55db1935f870 msgr2=0x55db19385a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.380+0000 7f594439c740 1 -- 172.21.15.27:0/3245544042 shutdown_connections 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.380+0000 7f594439c740 1 -- 172.21.15.27:0/3245544042 wait complete. 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.380+0000 7f594439c740 1 -- start start 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.380+0000 7f594439c740 1 Processor -- start 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f594439c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 0x55db193aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f594439c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db193b9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f594439c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db193bc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f594439c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db19272a60 con 0x55db1938f600 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f594439c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db19285ea0 con 0x55db19382cd0 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f594439c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db193824d0 con 0x55db1938e8f0 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f594439c740 10 client.0 _ll_register_callbacks cb 0x7f592c009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db193bc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db193bc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47600/0 (socket says 172.21.15.27:47600) 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593b7fe640 1 -- 172.21.15.27:0/2618857654 learned_addr learned my addr 172.21.15.27:0/2618857654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593affd640 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 0x55db193aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593a7fc640 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db193b9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593b7fe640 1 -- 172.21.15.27:0/2618857654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 msgr2=0x55db193b9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593b7fe640 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db193b9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593b7fe640 1 -- 172.21.15.27:0/2618857654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 msgr2=0x55db193aabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593b7fe640 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 0x55db193aabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593b7fe640 1 -- 172.21.15.27:0/2618857654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db193841d0 con 0x55db1938f600 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593a7fc640 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db193b9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f593b7fe640 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db193bc860 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5934001eb0 tx=0x7f5934007490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dceb500645fbe87a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.381+0000 7f5919ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f5919ffb640 1 -- 172.21.15.27:0/2618857654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593401a020 con 0x55db1938f600 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f5919ffb640 1 -- 172.21.15.27:0/2618857654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f59340075c0 con 0x55db1938f600 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f594439c740 1 -- 172.21.15.27:0/2618857654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f592c092040 con 0x55db1938f600 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f5919ffb640 1 -- 172.21.15.27:0/2618857654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5934024050 con 0x55db1938f600 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f594439c740 10 client.6786 fetch_fsmap learned FSMap version 44 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f594439c740 1 -- 172.21.15.27:0/2618857654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x55db193aa730 con 0x55db1938f600 2024-03-06T07:26:30.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f5919ffb640 1 -- 172.21.15.27:0/2618857654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c092040 con 0x55db1938f600 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f5919ffb640 1 -- 172.21.15.27:0/2618857654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 44) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5934027020 con 0x55db1938f600 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f594439c740 10 client.6786 fetch_fsmap finished waiting for FSMap version 44 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f594439c740 10 client.6786 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f594439c740 20 client.6786 populate_metadata read hostname 'smithi027' 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f594439c740 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] conn(0x55db19364720 0x55db19364b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f594439c740 4 client.6786 mds_command: new command op to 6606 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f594439c740 1 -- 172.21.15.27:0/2618857654 --> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55db19360630 con 0x55db19364720 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.382+0000 7f593affd640 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] conn(0x55db19364720 0x55db19364b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f5919ffb640 10 client.6786 ms_handle_connect on v2:172.21.15.27:6833/2393218771 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f593affd640 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] conn(0x55db19364720 0x55db19364b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2adf6c02b8db2f7f server_cookie=84d9172afd9e073e in_seq=0 out_seq=0 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f5919ffb640 1 -- 172.21.15.27:0/2618857654 <== mds.0 v2:172.21.15.27:6833/2393218771 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55db19360630 con 0x55db19364720 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f5919ffb640 10 client.6786 handle_command_reply: tid=0 2024-03-06T07:26:30.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 client.6786 shutdown 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 -- 172.21.15.27:0/2618857654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 msgr2=0x55db193bc860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db193bc860 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5934001eb0 tx=0x7f5934007490 comp rx=0 tx=0).stop 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 -- 172.21.15.27:0/2618857654 shutdown_connections 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.27:6833/2393218771,v1:172.21.15.27:6835/2393218771] conn(0x55db19364720 0x55db19364b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db1938f600 0x55db193bc860 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f5919ffb640 0 client.6786 ms_handle_reset on v2:172.21.15.27:6833/2393218771 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55db1938e8f0 0x55db193b9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 --2- 172.21.15.27:0/2618857654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db19382cd0 0x55db193aabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 -- 172.21.15.27:0/2618857654 >> 172.21.15.27:0/2618857654 conn(0x55db1935f870 msgr2=0x55db1938fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 -- 172.21.15.27:0/2618857654 shutdown_connections 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 1 -- 172.21.15.27:0/2618857654 wait complete. 2024-03-06T07:26:30.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.383+0000 7f594439c740 20 client.6786 trim_cache size 0 max 16384 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 6606, 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.151589602, 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 44, 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 72, 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 70, 2024-03-06T07:26:30.393 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 21.180527808000001 2024-03-06T07:26:30.394 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:26:30.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 -- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f591c06b500 msgr2=0x7f591c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f591c06b500 0x7f591c06d9c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f593c1572e0 tx=0x7f5924008040 comp rx=0 tx=0).stop 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 -- 172.21.15.27:0/3056130210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 msgr2=0x7f593c15a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f593c15a160 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f592c009f90 tx=0x7f592c009750 comp rx=0 tx=0).stop 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 -- 172.21.15.27:0/3056130210 shutdown_connections 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f591c06b500 0x7f591c06d9c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f593c1566d0 0x7f593c156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f593c1535a0 0x7f593c151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 --2- 172.21.15.27:0/3056130210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59280039b0 0x7f593c15a160 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 -- 172.21.15.27:0/3056130210 >> 172.21.15.27:0/3056130210 conn(0x7f593c066d00 msgr2=0x7f593c0b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 -- 172.21.15.27:0/3056130210 shutdown_connections 2024-03-06T07:26:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.384+0000 7f59416e1640 1 -- 172.21.15.27:0/3056130210 wait complete. 2024-03-06T07:26:30.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:26:30.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.502+0000 7fec7b0df640 1 Processor -- start 2024-03-06T07:26:30.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.502+0000 7fec7b0df640 1 -- start start 2024-03-06T07:26:30.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.502+0000 7fec7b0df640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fec7415df00 0x7fec740d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.502+0000 7fec7b0df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740b95a0 0x7fec740b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.502+0000 7fec7b0df640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec740b9a30 0x7fec740b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.502+0000 7fec7b0df640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec740cea30 con 0x7fec7415df00 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.502+0000 7fec7b0df640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec740d2af0 con 0x7fec740b9a30 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.502+0000 7fec7b0df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec740cf3c0 con 0x7fec740b95a0 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec78e7b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fec7415df00 0x7fec740d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec78e7b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fec7415df00 0x7fec740d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47616/0 (socket says 172.21.15.27:47616) 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec78e7b640 1 -- 172.21.15.27:0/575569111 learned_addr learned my addr 172.21.15.27:0/575569111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec7967c640 1 --2- 172.21.15.27:0/575569111 >> v2:172.21.15.69:3300/0 conn(0x7fec740b9a30 0x7fec740b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec73fff640 1 --2- 172.21.15.27:0/575569111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740b95a0 0x7fec740b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec78e7b640 1 -- 172.21.15.27:0/575569111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740b95a0 msgr2=0x7fec740b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec78e7b640 1 --2- 172.21.15.27:0/575569111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740b95a0 0x7fec740b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec78e7b640 1 -- 172.21.15.27:0/575569111 >> v2:172.21.15.69:3300/0 conn(0x7fec740b9a30 msgr2=0x7fec740b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec78e7b640 1 --2- 172.21.15.27:0/575569111 >> v2:172.21.15.69:3300/0 conn(0x7fec740b9a30 0x7fec740b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.503+0000 7fec78e7b640 1 -- 172.21.15.27:0/575569111 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740cf0e0 con 0x7fec7415df00 2024-03-06T07:26:30.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.504+0000 7fec78e7b640 1 --2- 172.21.15.27:0/575569111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7415df00 0x7fec740d25b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fec740d1160 tx=0x7fec6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25868b3632218966 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.504+0000 7fec737fe640 1 -- 172.21.15.27:0/575569111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68035020 con 0x7fec7415df00 2024-03-06T07:26:30.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.504+0000 7fec737fe640 1 -- 172.21.15.27:0/575569111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec68031870 con 0x7fec7415df00 2024-03-06T07:26:30.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.504+0000 7fec737fe640 1 -- 172.21.15.27:0/575569111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68031c40 con 0x7fec7415df00 2024-03-06T07:26:30.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.504+0000 7fec7b0df640 1 -- 172.21.15.27:0/575569111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7415df00 msgr2=0x7fec740d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.504+0000 7fec7b0df640 1 --2- 172.21.15.27:0/575569111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7415df00 0x7fec740d25b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fec740d1160 tx=0x7fec6802b910 comp rx=0 tx=0).stop 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.505+0000 7fec7b0df640 1 -- 172.21.15.27:0/575569111 shutdown_connections 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.505+0000 7fec7b0df640 1 --2- 172.21.15.27:0/575569111 >> v2:172.21.15.69:3300/0 conn(0x7fec740b9a30 0x7fec740b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.505+0000 7fec7b0df640 1 --2- 172.21.15.27:0/575569111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740b95a0 0x7fec740b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.505+0000 7fec7b0df640 1 --2- 172.21.15.27:0/575569111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7415df00 0x7fec740d25b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.505+0000 7fec7b0df640 1 -- 172.21.15.27:0/575569111 >> 172.21.15.27:0/575569111 conn(0x7fec74071960 msgr2=0x7fec74071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.505+0000 7fec7b0df640 1 -- 172.21.15.27:0/575569111 shutdown_connections 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.505+0000 7fec7b0df640 1 -- 172.21.15.27:0/575569111 wait complete. 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.506+0000 7fec7b0df640 1 Processor -- start 2024-03-06T07:26:30.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.506+0000 7fec7b0df640 1 -- start start 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec7b0df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec740b95a0 0x7fec7415fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec7b0df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7415df00 0x7fec740cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec7b0df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740cd200 0x7fec7415e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec7b0df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec740cec70 con 0x7fec740b95a0 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec7b0df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec740ce750 con 0x7fec7415df00 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec7b0df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec740cedf0 con 0x7fec740cd200 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec78e7b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec740b95a0 0x7fec7415fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec78e7b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec740b95a0 0x7fec7415fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47618/0 (socket says 172.21.15.27:47618) 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec78e7b640 1 -- 172.21.15.27:0/1684995632 learned_addr learned my addr 172.21.15.27:0/1684995632 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec73fff640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7415df00 0x7fec740cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec7967c640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740cd200 0x7fec7415e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec78e7b640 1 -- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740cd200 msgr2=0x7fec7415e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec78e7b640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740cd200 0x7fec7415e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec78e7b640 1 -- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7415df00 msgr2=0x7fec740cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec78e7b640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7415df00 0x7fec740cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec78e7b640 1 -- 172.21.15.27:0/1684995632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec741603e0 con 0x7fec740b95a0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec78e7b640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec740b95a0 0x7fec7415fea0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fec6802b950 tx=0x7fec680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec717fa640 1 -- 172.21.15.27:0/1684995632 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68040070 con 0x7fec740b95a0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.507+0000 7fec717fa640 1 -- 172.21.15.27:0/1684995632 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec68031e10 con 0x7fec740b95a0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.508+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec74160660 con 0x7fec740b95a0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.508+0000 7fec717fa640 1 -- 172.21.15.27:0/1684995632 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68035040 con 0x7fec740b95a0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.508+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec74160c20 con 0x7fec740b95a0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.509+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec74067ff0 con 0x7fec740b95a0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.509+0000 7fec717fa640 1 -- 172.21.15.27:0/1684995632 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec6804b060 con 0x7fec740b95a0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.509+0000 7fec717fa640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec5806b4d0 0x7fec5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.509+0000 7fec717fa640 1 -- 172.21.15.27:0/1684995632 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fec680abe90 con 0x7fec740b95a0 2024-03-06T07:26:30.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.510+0000 7fec73fff640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec5806b4d0 0x7fec5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.510+0000 7fec73fff640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec5806b4d0 0x7fec5806d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fec74144f90 tx=0x7fec60008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.514+0000 7fec717fa640 1 -- 172.21.15.27:0/1684995632 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec741603e0 con 0x7fec740b95a0 2024-03-06T07:26:30.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.739+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fec740b6f70 con 0x7fec740b95a0 2024-03-06T07:26:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.739+0000 7fec717fa640 1 -- 172.21.15.27:0/1684995632 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) v1 ==== 53+0+73 (secure 0 0 0) 0x7fec6807acd0 con 0x7fec740b95a0 2024-03-06T07:26:30.741 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:26:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec5806b4d0 msgr2=0x7fec5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec5806b4d0 0x7fec5806d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fec74144f90 tx=0x7fec60008040 comp rx=0 tx=0).stop 2024-03-06T07:26:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec740b95a0 msgr2=0x7fec7415fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec740b95a0 0x7fec7415fea0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fec6802b950 tx=0x7fec680046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 shutdown_connections 2024-03-06T07:26:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec5806b4d0 0x7fec5806d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec740cd200 0x7fec7415e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7415df00 0x7fec740cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 --2- 172.21.15.27:0/1684995632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec740b95a0 0x7fec7415fea0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 >> 172.21.15.27:0/1684995632 conn(0x7fec74071960 msgr2=0x7fec740cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 shutdown_connections 2024-03-06T07:26:30.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:30.741+0000 7fec7b0df640 1 -- 172.21.15.27:0/1684995632 wait complete. 2024-03-06T07:26:30.754 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:26:30.754 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:26:30.754 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:26:30.754 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:26:30.754 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:30.754 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:26:30.754 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:26:30.754 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:26:30.754 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:30.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:30.755 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:26:30.771 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:26:30.771 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:26:30.771 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:26:30.771 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:30.771 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:26:30.772 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:30.772 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:30.772 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:30.772 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:26:30.772 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:30.772 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:30.773 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:26:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:26:30.773 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:30.773 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:26:30.773 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:26:30.773 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42499sec preferred_lft 42499sec 2024-03-06T07:26:30.773 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:26:30.774 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:30.774 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:30.774 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:30.774 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:26:30.774 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:26:30.774 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:30.774 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:26:30.775 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:30.775 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:26:30.775 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:30.775 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:30.775 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:30.775 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:26:30.775 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:30.776 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:30.776 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:30.776 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:26:30.776 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:30.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:30.777 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:26:30.844 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:26:30.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:30.845 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:26:30.901 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:26:30.901 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:26:30.901 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:26:30.901 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:26:30.901 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:26:30.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:30.901 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:26:30.901 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:26:30.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:26:30.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:26:30.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:26:30.902 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:26:31.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:31.007 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:26:31.021 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:26:31.022 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:26:31.022 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:31.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:31.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:26:31.092 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:26:31.092 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:31.093 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:31.148 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:31.148 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:31.148 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:31.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:26:31.274 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:31.329 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:26:31.330 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:26:31.330 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:26:31.330 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:26:31.330 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:26:31.330 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:26:31.330 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:26:31.330 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:26:31.330 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:26:31.330 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:26:31.331 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:26:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:26:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:31.332 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:31.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:31.333 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:26:31.385 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:26:31.385 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:26:31.427 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:26:31.460 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:31.516 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:26:31.516 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:26:31.516 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:26:31.516 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:26:31.516 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:26:31.516 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:26:31.516 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:26:31.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:26:31.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:26:31.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:26:31.517 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:26:31.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:26:31.517 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:31.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:26:31.517 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:26:31.517 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:26:31.518 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:31.519 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:31.519 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:31.519 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:31.519 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:26:31.519 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:31.519 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:31.519 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:31.519 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:31.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:31.520 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:26:31.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:26:31.521+0000 7fe26f83b480 -1 init, newargv = 0x563e9dbac880 newargc=15 2024-03-06T07:26:31.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[62838]: starting ceph client 2024-03-06T07:26:31.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[62838]: starting fuse 2024-03-06T07:26:31.539 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:26:31.540 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:26:31.540 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:26:31.540 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:26:31.540 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:26:31.540 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:26:31.540 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:26:31.540 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:31.540 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:26:31.541 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:26:31.542 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:26:31.542 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:26:31.542 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:26:31.542 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:31.542 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:26:31.542 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:26:31.666 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.62838.asok 2024-03-06T07:26:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:31.677 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62838.asok 2024-03-06T07:26:31.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:31.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62838.asok status 2024-03-06T07:26:31.819 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:26:31.819 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:26:31.819 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:26:31.819 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:26:31.819 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "62838", 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 6798, 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 6798 2024-03-06T07:26:31.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:31.821 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:26:31.821 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:26:31.821 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:31.821 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1781023031 2024-03-06T07:26:31.821 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:26:31.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:31.821 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:26:31.821 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1781023031 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.6798 192.168.0.1:0/1781023031", 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/1781023031", 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 44, 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 72, 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:26:31.822 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:26:31.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:31.828 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:26:31.828 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:31.843 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:26:31.844 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:31.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:31.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:31.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:26:32.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.063+0000 7f3489b93640 1 Processor -- start 2024-03-06T07:26:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.063+0000 7f3489b93640 1 -- start start 2024-03-06T07:26:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.064+0000 7f3489b93640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34840b2d30 0x7f34840b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.064+0000 7f3489b93640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f34840b7820 0x7f34840b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.064+0000 7f3489b93640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34840d4960 con 0x7f3484071540 2024-03-06T07:26:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.064+0000 7f3489b93640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34840d3230 con 0x7f34840b2d30 2024-03-06T07:26:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.064+0000 7f3489b93640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34840d36c0 con 0x7f34840b7820 2024-03-06T07:26:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.064+0000 7f3482ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34840b2d30 0x7f34840b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.064+0000 7f34837fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3484071540 0x7f3484071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47058/0 (socket says 172.21.15.27:47058) 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.064+0000 7f34837fe640 1 -- 172.21.15.27:0/3216837703 learned_addr learned my addr 172.21.15.27:0/3216837703 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.065+0000 7f3483fff640 1 --2- 172.21.15.27:0/3216837703 >> v2:172.21.15.69:3300/0 conn(0x7f34840b7820 0x7f34840b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.065+0000 7f3482ffd640 1 -- 172.21.15.27:0/3216837703 >> v2:172.21.15.69:3300/0 conn(0x7f34840b7820 msgr2=0x7f34840b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.065+0000 7f3482ffd640 1 --2- 172.21.15.27:0/3216837703 >> v2:172.21.15.69:3300/0 conn(0x7f34840b7820 0x7f34840b7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.065+0000 7f3482ffd640 1 -- 172.21.15.27:0/3216837703 >> v1:172.21.15.27:6789/0 conn(0x7f3484071540 legacy=0x7f3484071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.065+0000 7f3482ffd640 1 -- 172.21.15.27:0/3216837703 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34840d2620 con 0x7f34840b2d30 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.065+0000 7f34827fc640 1 -- 172.21.15.27:0/3216837703 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296516479 0 0) 0x7f34840d4960 con 0x7f3484071540 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.065+0000 7f3482ffd640 1 --2- 172.21.15.27:0/3216837703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b2d30 0x7f34840b72e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f34840ce340 tx=0x7f347402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692fd244e06dcb16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f34827fc640 1 -- 172.21.15.27:0/3216837703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474035020 con 0x7f34840b2d30 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f34827fc640 1 -- 172.21.15.27:0/3216837703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3474031830 con 0x7f34840b2d30 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f34827fc640 1 -- 172.21.15.27:0/3216837703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474031c00 con 0x7f34840b2d30 2024-03-06T07:26:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f3489b93640 1 -- 172.21.15.27:0/3216837703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b2d30 msgr2=0x7f34840b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f3489b93640 1 --2- 172.21.15.27:0/3216837703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b2d30 0x7f34840b72e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f34840ce340 tx=0x7f347402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f3489b93640 1 -- 172.21.15.27:0/3216837703 shutdown_connections 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f3489b93640 1 --2- 172.21.15.27:0/3216837703 >> v2:172.21.15.69:3300/0 conn(0x7f34840b7820 0x7f34840b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f3489b93640 1 --2- 172.21.15.27:0/3216837703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b2d30 0x7f34840b72e0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f3489b93640 1 -- 172.21.15.27:0/3216837703 >> 172.21.15.27:0/3216837703 conn(0x7f34840d3d60 msgr2=0x7f34840d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f3489b93640 1 -- 172.21.15.27:0/3216837703 shutdown_connections 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.066+0000 7f3489b93640 1 -- 172.21.15.27:0/3216837703 wait complete. 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.067+0000 7f3489b93640 1 Processor -- start 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3489b93640 1 -- start start 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3489b93640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3484071540 0x7f34840ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3489b93640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34840b2d30 0x7f34841e3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3489b93640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b7820 0x7f34841e5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3489b93640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840d2ec0 con 0x7f34840b7820 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3489b93640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840d2be0 con 0x7f3484071540 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3489b93640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34840d22f0 con 0x7f34840b2d30 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3483fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b7820 0x7f34841e5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3483fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b7820 0x7f34841e5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47640/0 (socket says 172.21.15.27:47640) 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.068+0000 7f3483fff640 1 -- 172.21.15.27:0/3726331557 learned_addr learned my addr 172.21.15.27:0/3726331557 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f34837fe640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3484071540 0x7f34840ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3482ffd640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34840b2d30 0x7f34841e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3483fff640 1 -- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34840b2d30 msgr2=0x7f34841e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3483fff640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34840b2d30 0x7f34841e3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3483fff640 1 -- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3484071540 msgr2=0x7f34840ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3483fff640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3484071540 0x7f34840ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3483fff640 1 -- 172.21.15.27:0/3726331557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34841e3ff0 con 0x7f34840b7820 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3482ffd640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34840b2d30 0x7f34841e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f34837fe640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3484071540 0x7f34840ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3483fff640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b7820 0x7f34841e5460 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f347c00cda0 tx=0x7f347c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f346bfff640 1 -- 172.21.15.27:0/3726331557 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c003b90 con 0x7f34840b7820 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f346bfff640 1 -- 172.21.15.27:0/3726331557 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347c003d30 con 0x7f34840b7820 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34841e42d0 con 0x7f34840b7820 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f346bfff640 1 -- 172.21.15.27:0/3726331557 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c004240 con 0x7f34840b7820 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.069+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34841e4890 con 0x7f34840b7820 2024-03-06T07:26:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.070+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3484072070 con 0x7f34840b7820 2024-03-06T07:26:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.071+0000 7f346bfff640 1 -- 172.21.15.27:0/3726331557 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f347c0043e0 con 0x7f34840b7820 2024-03-06T07:26:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.071+0000 7f346bfff640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f347006b4d0 0x7f347006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.072+0000 7f346bfff640 1 -- 172.21.15.27:0/3726331557 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f347c08acd0 con 0x7f34840b7820 2024-03-06T07:26:32.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.072+0000 7f34837fe640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f347006b4d0 0x7f347006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.072+0000 7f34837fe640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f347006b4d0 0x7f347006d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3478000f60 tx=0x7f3478007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.076+0000 7f346bfff640 1 -- 172.21.15.27:0/3726331557 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f347c01d340 con 0x7f34840b7820 2024-03-06T07:26:32.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.257+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f348404f410 con 0x7f34840b7820 2024-03-06T07:26:32.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.260+0000 7f346bfff640 1 -- 172.21.15.27:0/3726331557 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f347c059ba0 con 0x7f34840b7820 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.262+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f347006b4d0 msgr2=0x7f347006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.262+0000 7f3489b93640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f347006b4d0 0x7f347006d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3478000f60 tx=0x7f3478007ca0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.262+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b7820 msgr2=0x7f34841e5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.262+0000 7f3489b93640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b7820 0x7f34841e5460 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f347c00cda0 tx=0x7f347c00a660 comp rx=0 tx=0).stop 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.263+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 shutdown_connections 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.263+0000 7f3489b93640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f347006b4d0 0x7f347006d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.263+0000 7f3489b93640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34840b7820 0x7f34841e5460 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.263+0000 7f3489b93640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f34840b2d30 0x7f34841e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.263+0000 7f3489b93640 1 --2- 172.21.15.27:0/3726331557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3484071540 0x7f34840ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.263+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 >> 172.21.15.27:0/3726331557 conn(0x7f34840d3d60 msgr2=0x7f348415d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:32.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.263+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 shutdown_connections 2024-03-06T07:26:32.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:32.263+0000 7f3489b93640 1 -- 172.21.15.27:0/3726331557 wait complete. 2024-03-06T07:26:42.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:42.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.380+0000 7fe4aca88640 1 Processor -- start 2024-03-06T07:26:42.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.380+0000 7fe4aca88640 1 -- start start 2024-03-06T07:26:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.381+0000 7fe4aca88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815df00 0x7fe4a80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.381+0000 7fe4aca88640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe4a80b95a0 0x7fe4a80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.381+0000 7fe4aca88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80cea30 con 0x7fe4a80b9a30 2024-03-06T07:26:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.381+0000 7fe4aca88640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a80d2af0 con 0x7fe4a80b95a0 2024-03-06T07:26:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.381+0000 7fe4aca88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80cf3c0 con 0x7fe4a815df00 2024-03-06T07:26:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.381+0000 7fe49ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe4a80b95a0 0x7fe4a80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.381+0000 7fe49ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe4a80b95a0 0x7fe4a80b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34174/0 (socket says 172.21.15.27:34174) 2024-03-06T07:26:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.381+0000 7fe49ffff640 1 -- 172.21.15.27:0/2655549948 learned_addr learned my addr 172.21.15.27:0/2655549948 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.382+0000 7fe4a659c640 1 --2- 172.21.15.27:0/2655549948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815df00 0x7fe4a80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.382+0000 7fe49ffff640 1 -- 172.21.15.27:0/2655549948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815df00 msgr2=0x7fe4a80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.382+0000 7fe49ffff640 1 --2- 172.21.15.27:0/2655549948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815df00 0x7fe4a80d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.382+0000 7fe49ffff640 1 -- 172.21.15.27:0/2655549948 >> v1:172.21.15.69:6789/0 conn(0x7fe4a80b9a30 legacy=0x7fe4a80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.382+0000 7fe49ffff640 1 -- 172.21.15.27:0/2655549948 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80cf0e0 con 0x7fe4a80b95a0 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.382+0000 7fe4a659c640 1 --2- 172.21.15.27:0/2655549948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815df00 0x7fe4a80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.382+0000 7fe49ffff640 1 --2- 172.21.15.27:0/2655549948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 0x7fe4a80b5700 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe490009870 tx=0x7fe49002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5b491532cb8dab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4a5d9b640 1 -- 172.21.15.27:0/2655549948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490035020 con 0x7fe4a80b95a0 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4a5d9b640 1 -- 172.21.15.27:0/2655549948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe490031640 con 0x7fe4a80b95a0 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4a5d9b640 1 -- 172.21.15.27:0/2655549948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49002d040 con 0x7fe4a80b95a0 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4aca88640 1 -- 172.21.15.27:0/2655549948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 msgr2=0x7fe4a80b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4aca88640 1 --2- 172.21.15.27:0/2655549948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 0x7fe4a80b5700 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe490009870 tx=0x7fe49002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4aca88640 1 -- 172.21.15.27:0/2655549948 shutdown_connections 2024-03-06T07:26:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4aca88640 1 --2- 172.21.15.27:0/2655549948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 0x7fe4a80b5700 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4aca88640 1 --2- 172.21.15.27:0/2655549948 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815df00 0x7fe4a80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4aca88640 1 -- 172.21.15.27:0/2655549948 >> 172.21.15.27:0/2655549948 conn(0x7fe4a8071960 msgr2=0x7fe4a8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4aca88640 1 -- 172.21.15.27:0/2655549948 shutdown_connections 2024-03-06T07:26:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.383+0000 7fe4aca88640 1 -- 172.21.15.27:0/2655549948 wait complete. 2024-03-06T07:26:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.384+0000 7fe4aca88640 1 Processor -- start 2024-03-06T07:26:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.384+0000 7fe4aca88640 1 -- start start 2024-03-06T07:26:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.385+0000 7fe4aca88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 0x7fe4a815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.385+0000 7fe4aca88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4a815df00 0x7fe4a8150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.385+0000 7fe4aca88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815b230 0x7fe4a81f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.385+0000 7fe4aca88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80cec70 con 0x7fe4a80b95a0 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.385+0000 7fe4aca88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80ce750 con 0x7fe4a815df00 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.385+0000 7fe4aca88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80cedf0 con 0x7fe4a815b230 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.385+0000 7fe4a659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 0x7fe4a815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.385+0000 7fe4a659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 0x7fe4a815b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34184/0 (socket says 172.21.15.27:34184) 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.385+0000 7fe4a659c640 1 -- 172.21.15.27:0/2463696976 learned_addr learned my addr 172.21.15.27:0/2463696976 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe49ffff640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4a815df00 0x7fe4a8150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe4a6d9d640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815b230 0x7fe4a81f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe4a659c640 1 -- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815b230 msgr2=0x7fe4a81f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe4a659c640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815b230 0x7fe4a81f3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe4a659c640 1 -- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4a815df00 msgr2=0x7fe4a8150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe4a659c640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4a815df00 0x7fe4a8150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe4a659c640 1 -- 172.21.15.27:0/2463696976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a815e570 con 0x7fe4a80b95a0 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe49ffff640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4a815df00 0x7fe4a8150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe4a659c640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 0x7fe4a815b780 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe49800c820 tx=0x7fe49800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe49effd640 1 -- 172.21.15.27:0/2463696976 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498015070 con 0x7fe4a80b95a0 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe49effd640 1 -- 172.21.15.27:0/2463696976 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe498007a30 con 0x7fe4a80b95a0 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a80c8b10 con 0x7fe4a80b95a0 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe49effd640 1 -- 172.21.15.27:0/2463696976 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498007de0 con 0x7fe4a80b95a0 2024-03-06T07:26:42.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.386+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a80c90d0 con 0x7fe4a80b95a0 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.387+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a8067ff0 con 0x7fe4a80b95a0 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.388+0000 7fe49effd640 1 -- 172.21.15.27:0/2463696976 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe498018070 con 0x7fe4a80b95a0 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.388+0000 7fe49effd640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe48806b4d0 0x7fe48806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.388+0000 7fe49effd640 1 -- 172.21.15.27:0/2463696976 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe49808a180 con 0x7fe4a80b95a0 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.388+0000 7fe49ffff640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe48806b4d0 0x7fe48806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.389+0000 7fe49ffff640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe48806b4d0 0x7fe48806d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe490036000 tx=0x7fe49003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.393+0000 7fe49effd640 1 -- 172.21.15.27:0/2463696976 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4a815e570 con 0x7fe4a80b95a0 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.615+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4a806b780 con 0x7fe4a80b95a0 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.616+0000 7fe49effd640 1 -- 172.21.15.27:0/2463696976 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fe498058fc0 con 0x7fe4a80b95a0 2024-03-06T07:26:42.619 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:42.619 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:42.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe48806b4d0 msgr2=0x7fe48806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe48806b4d0 0x7fe48806d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe490036000 tx=0x7fe49003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 msgr2=0x7fe4a815b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 0x7fe4a815b780 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe49800c820 tx=0x7fe49800cce0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 shutdown_connections 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe48806b4d0 0x7fe48806d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4a815b230 0x7fe4a81f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4a815df00 0x7fe4a8150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 --2- 172.21.15.27:0/2463696976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a80b95a0 0x7fe4a815b780 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 >> 172.21.15.27:0/2463696976 conn(0x7fe4a8071960 msgr2=0x7fe4a80b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 shutdown_connections 2024-03-06T07:26:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42.618+0000 7fe4aca88640 1 -- 172.21.15.27:0/2463696976 wait complete. 2024-03-06T07:26:42.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:26:42.753 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:26:42.754 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:26:42.754 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:26:42.754 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:42.770 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:26:42.770 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:26:42.770 DEBUG:teuthology.orchestra.run.smithi027:> import uuid 2024-03-06T07:26:42.770 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:42.770 DEBUG:teuthology.orchestra.run.smithi027:> n = 100 2024-03-06T07:26:42.770 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = False 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> if 0 > 0: 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = True 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> dpath = os.path.dirname(path) 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> fnameprefix = os.path.basename(path) 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> try: 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> for i in range(n): 2024-03-06T07:26:42.771 DEBUG:teuthology.orchestra.run.smithi027:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> f.write(f"{i}") 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> f.flush() 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(f.fileno()) 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> if True: 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> os.unlink(fpath) 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> if create_hard_links: 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> for j in range(0): 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> if True: 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:26:42.772 DEBUG:teuthology.orchestra.run.smithi027:> finally: 2024-03-06T07:26:42.773 DEBUG:teuthology.orchestra.run.smithi027:> os.close(dirfd) 2024-03-06T07:26:42.773 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:26:42.922 INFO:teuthology.orchestra.run.smithi027.stderr:Traceback (most recent call last): 2024-03-06T07:26:42.923 INFO:teuthology.orchestra.run.smithi027.stderr: File "", line 26, in 2024-03-06T07:26:42.923 INFO:teuthology.orchestra.run.smithi027.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:26:42.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:42 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:26:42.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:42.935 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:43.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.080+0000 7f71371d0640 1 Processor -- start 2024-03-06T07:26:43.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.080+0000 7f71371d0640 1 -- start start 2024-03-06T07:26:43.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.080+0000 7f71371d0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f713015df00 0x7f71300d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.080+0000 7f71371d0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71300cea30 con 0x7f71300b9a30 2024-03-06T07:26:43.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.080+0000 7f71371d0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71300d2af0 con 0x7f71300b95a0 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.080+0000 7f71371d0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71300cf3c0 con 0x7f713015df00 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f713576d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f71300b9a30 0x7f71300d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58186/0 (socket says 172.21.15.27:58186) 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f713576d640 1 -- 172.21.15.27:0/2798783082 learned_addr learned my addr 172.21.15.27:0/2798783082 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f7134f6c640 1 --2- 172.21.15.27:0/2798783082 >> v2:172.21.15.69:3300/0 conn(0x7f713015df00 0x7f71300d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927059967 0 0) 0x7f71300cea30 con 0x7f71300b9a30 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7120003520 con 0x7f71300b9a30 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 141814954 0 0) 0x7f7120003520 con 0x7f71300b9a30 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 >> v2:172.21.15.69:3300/0 conn(0x7f713015df00 msgr2=0x7f71300d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 --2- 172.21.15.27:0/2798783082 >> v2:172.21.15.69:3300/0 conn(0x7f713015df00 0x7f71300d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 >> v1:172.21.15.69:6789/0 conn(0x7f71300b95a0 legacy=0x7f71300b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71300cf0e0 con 0x7f71300b9a30 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f71280019f0 con 0x7f71300b9a30 2024-03-06T07:26:43.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 >> v1:172.21.15.27:6789/0 conn(0x7f71300b9a30 legacy=0x7f71300d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71200039b0 0x7f7120003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7120004800 0x7f7120004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200051c0 0x7f712000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71300cf3c0 con 0x7f7120004800 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712000c060 con 0x7f71200039b0 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712000c260 con 0x7f71200051c0 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f712ffff640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7120004800 0x7f7120004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.081+0000 7f7134f6c640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71200039b0 0x7f7120003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f713576d640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200051c0 0x7f712000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f7134f6c640 1 -- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200051c0 msgr2=0x7f712000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f7134f6c640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200051c0 0x7f712000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f7134f6c640 1 -- 172.21.15.27:0/2798783082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7120004800 msgr2=0x7f7120004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f7134f6c640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7120004800 0x7f7120004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f7134f6c640 1 -- 172.21.15.27:0/2798783082 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7120003520 con 0x7f71200039b0 2024-03-06T07:26:43.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f713576d640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200051c0 0x7f712000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f712ffff640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7120004800 0x7f7120004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f713576d640 1 -- 172.21.15.27:0/2798783082 reap_dead start 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.082+0000 7f7134f6c640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71200039b0 0x7f7120003dd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7124009ac0 tx=0x7f7120005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e22c003ce4dac11e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7124039070 con 0x7f71200039b0 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712402c6b0 con 0x7f71200039b0 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f712f7fe640 1 -- 172.21.15.27:0/2798783082 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7124009690 con 0x7f71200039b0 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f71371d0640 1 -- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71200039b0 msgr2=0x7f7120003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f71371d0640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71200039b0 0x7f7120003dd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7124009ac0 tx=0x7f7120005830 comp rx=0 tx=0).stop 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f71371d0640 1 -- 172.21.15.27:0/2798783082 shutdown_connections 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f71371d0640 1 --2- 172.21.15.27:0/2798783082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71200039b0 0x7f7120003dd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f71371d0640 1 -- 172.21.15.27:0/2798783082 >> 172.21.15.27:0/2798783082 conn(0x7f7130071960 msgr2=0x7f7130071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f71371d0640 1 -- 172.21.15.27:0/2798783082 shutdown_connections 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.083+0000 7f71371d0640 1 -- 172.21.15.27:0/2798783082 wait complete. 2024-03-06T07:26:43.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.084+0000 7f71371d0640 1 Processor -- start 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.085+0000 7f71371d0640 1 -- start start 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.085+0000 7f71371d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200039b0 0x7f71301552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.085+0000 7f71371d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71300cca90 0x7f71300cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.085+0000 7f71371d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71300c8ce0 0x7f71300c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.085+0000 7f71371d0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71300cec70 con 0x7f71300c8ce0 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.085+0000 7f71371d0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71300ce750 con 0x7f71300cca90 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.085+0000 7f71371d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71300cedf0 con 0x7f71200039b0 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f713576d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71300c8ce0 0x7f71300c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f713576d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71300c8ce0 0x7f71300c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34194/0 (socket says 172.21.15.27:34194) 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f713576d640 1 -- 172.21.15.27:0/1921968165 learned_addr learned my addr 172.21.15.27:0/1921968165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f712ffff640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71300cca90 0x7f71300cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f7134f6c640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200039b0 0x7f71301552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f713576d640 1 -- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200039b0 msgr2=0x7f71301552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f713576d640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200039b0 0x7f71301552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f713576d640 1 -- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71300cca90 msgr2=0x7f71300cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f713576d640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71300cca90 0x7f71300cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f713576d640 1 -- 172.21.15.27:0/1921968165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71300cb760 con 0x7f71300c8ce0 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f7134f6c640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200039b0 0x7f71301552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f713576d640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71300c8ce0 0x7f71300c9160 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7128001ee0 tx=0x7f7128007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f712d7fa640 1 -- 172.21.15.27:0/1921968165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712800eaa0 con 0x7f71300c8ce0 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f712d7fa640 1 -- 172.21.15.27:0/1921968165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7128018070 con 0x7f71300c8ce0 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f712d7fa640 1 -- 172.21.15.27:0/1921968165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712800ee40 con 0x7f71300c8ce0 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71300cd7a0 con 0x7f71300c8ce0 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.086+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71300cba20 con 0x7f71300c8ce0 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.088+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7130067ff0 con 0x7f71300c8ce0 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.088+0000 7f712d7fa640 1 -- 172.21.15.27:0/1921968165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f712801d460 con 0x7f71300c8ce0 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.089+0000 7f712d7fa640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f711406b500 0x7f711406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.089+0000 7f712d7fa640 1 -- 172.21.15.27:0/1921968165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f712808a340 con 0x7f71300c8ce0 2024-03-06T07:26:43.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.089+0000 7f7134f6c640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f711406b500 0x7f711406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.090+0000 7f7134f6c640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f711406b500 0x7f711406d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7124036000 tx=0x7f7124030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.094+0000 7f712d7fa640 1 -- 172.21.15.27:0/1921968165 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7128059190 con 0x7f71300c8ce0 2024-03-06T07:26:43.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.335+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71301627a0 con 0x7f71300c8ce0 2024-03-06T07:26:43.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.335+0000 7f712d7fa640 1 -- 172.21.15.27:0/1921968165 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f7128058a90 con 0x7f71300c8ce0 2024-03-06T07:26:43.338 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:43.338 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:43.338 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f711406b500 msgr2=0x7f711406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f711406b500 0x7f711406d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7124036000 tx=0x7f7124030000 comp rx=0 tx=0).stop 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71300c8ce0 msgr2=0x7f71300c9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71300c8ce0 0x7f71300c9160 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7128001ee0 tx=0x7f7128007bc0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 shutdown_connections 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f711406b500 0x7f711406d9c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71300c8ce0 0x7f71300c9160 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71300cca90 0x7f71300cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 --2- 172.21.15.27:0/1921968165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71200039b0 0x7f71301552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 >> 172.21.15.27:0/1921968165 conn(0x7f7130071960 msgr2=0x7f71300b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 shutdown_connections 2024-03-06T07:26:43.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.337+0000 7f71371d0640 1 -- 172.21.15.27:0/1921968165 wait complete. 2024-03-06T07:26:43.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:26:43.473 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:26:43.474 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:26:43.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:43.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.579+0000 7f4ab2bf7640 1 Processor -- start 2024-03-06T07:26:43.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.579+0000 7f4ab2bf7640 1 -- start start 2024-03-06T07:26:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4ab2bf7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4aac15df00 0x7f4aac0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4ab2bf7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0cea30 con 0x7f4aac0b95a0 2024-03-06T07:26:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4ab2bf7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0d2af0 con 0x7f4aac0b9a30 2024-03-06T07:26:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4ab2bf7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aac0cf3c0 con 0x7f4aac15df00 2024-03-06T07:26:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4ab0993640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4aac15df00 0x7f4aac0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4aabfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4aac0b95a0 0x7f4aac0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38768/0 (socket says 172.21.15.27:38768) 2024-03-06T07:26:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4aabfff640 1 -- 172.21.15.27:0/2717463076 learned_addr learned my addr 172.21.15.27:0/2717463076 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4ab0993640 1 -- 172.21.15.27:0/2717463076 >> v1:172.21.15.69:6789/0 conn(0x7f4aac0b9a30 legacy=0x7f4aac0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4ab0993640 1 -- 172.21.15.27:0/2717463076 >> v1:172.21.15.27:6789/0 conn(0x7f4aac0b95a0 legacy=0x7f4aac0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.580+0000 7f4ab0993640 1 -- 172.21.15.27:0/2717463076 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac0cf0e0 con 0x7f4aac15df00 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.581+0000 7f4ab0993640 1 --2- 172.21.15.27:0/2717463076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac15df00 0x7f4aac0d25b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097a0 tx=0x7f4aa002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117161138f735dae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.581+0000 7f4aab7fe640 1 -- 172.21.15.27:0/2717463076 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0035020 con 0x7f4aac15df00 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.581+0000 7f4aab7fe640 1 -- 172.21.15.27:0/2717463076 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0031830 con 0x7f4aac15df00 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.581+0000 7f4aab7fe640 1 -- 172.21.15.27:0/2717463076 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0031c00 con 0x7f4aac15df00 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.582+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/2717463076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac15df00 msgr2=0x7f4aac0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.582+0000 7f4ab2bf7640 1 --2- 172.21.15.27:0/2717463076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac15df00 0x7f4aac0d25b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097a0 tx=0x7f4aa002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.582+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/2717463076 shutdown_connections 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.582+0000 7f4ab2bf7640 1 --2- 172.21.15.27:0/2717463076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac15df00 0x7f4aac0d25b0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.582+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/2717463076 >> 172.21.15.27:0/2717463076 conn(0x7f4aac071960 msgr2=0x7f4aac071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.582+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/2717463076 shutdown_connections 2024-03-06T07:26:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.582+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/2717463076 wait complete. 2024-03-06T07:26:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.583+0000 7f4ab2bf7640 1 Processor -- start 2024-03-06T07:26:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.583+0000 7f4ab2bf7640 1 -- start start 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab2bf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0b95a0 0x7f4aac1e8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab2bf7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aac15df00 0x7f4aac14f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab2bf7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aac0ccf20 0x7f4aac0cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab2bf7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0cec70 con 0x7f4aac0b95a0 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab2bf7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0ce750 con 0x7f4aac0ccf20 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab2bf7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0cedf0 con 0x7f4aac15df00 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab0993640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0b95a0 0x7f4aac1e8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab0993640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0b95a0 0x7f4aac1e8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32876/0 (socket says 172.21.15.27:32876) 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab0993640 1 -- 172.21.15.27:0/3827733969 learned_addr learned my addr 172.21.15.27:0/3827733969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4aabfff640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aac15df00 0x7f4aac14f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab1194640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aac0ccf20 0x7f4aac0cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab0993640 1 -- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aac15df00 msgr2=0x7f4aac14f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab0993640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aac15df00 0x7f4aac14f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab0993640 1 -- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aac0ccf20 msgr2=0x7f4aac0cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.584+0000 7f4ab0993640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aac0ccf20 0x7f4aac0cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.585+0000 7f4ab0993640 1 -- 172.21.15.27:0/3827733969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac1e8670 con 0x7f4aac0b95a0 2024-03-06T07:26:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.585+0000 7f4ab0993640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0b95a0 0x7f4aac1e8130 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00098c0 tx=0x7f4aa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.585+0000 7f4aa97fa640 1 -- 172.21.15.27:0/3827733969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0041070 con 0x7f4aac0b95a0 2024-03-06T07:26:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.585+0000 7f4aa97fa640 1 -- 172.21.15.27:0/3827733969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa003a970 con 0x7f4aac0b95a0 2024-03-06T07:26:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.585+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac1e8f50 con 0x7f4aac0b95a0 2024-03-06T07:26:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.585+0000 7f4aa97fa640 1 -- 172.21.15.27:0/3827733969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0035040 con 0x7f4aac0b95a0 2024-03-06T07:26:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.585+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac15e380 con 0x7f4aac0b95a0 2024-03-06T07:26:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.586+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aac067ff0 con 0x7f4aac0b95a0 2024-03-06T07:26:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.586+0000 7f4aa97fa640 1 -- 172.21.15.27:0/3827733969 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4aa0038020 con 0x7f4aac0b95a0 2024-03-06T07:26:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.587+0000 7f4aa97fa640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a9006b500 0x7f4a9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.587+0000 7f4aa97fa640 1 -- 172.21.15.27:0/3827733969 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4aa00abca0 con 0x7f4aac0b95a0 2024-03-06T07:26:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.587+0000 7f4aabfff640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a9006b500 0x7f4a9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.588+0000 7f4aabfff640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a9006b500 0x7f4a9006d9c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4aac15af90 tx=0x7f4a98009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.591+0000 7f4aa97fa640 1 -- 172.21.15.27:0/3827733969 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4aa007aae0 con 0x7f4aac0b95a0 2024-03-06T07:26:43.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.813+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4aac15d500 con 0x7f4aac0b95a0 2024-03-06T07:26:43.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.814+0000 7f4aa97fa640 1 -- 172.21.15.27:0/3827733969 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f4aa007a3e0 con 0x7f4aac0b95a0 2024-03-06T07:26:43.816 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:43.817 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:43.817 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:43.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a9006b500 msgr2=0x7f4a9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a9006b500 0x7f4a9006d9c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4aac15af90 tx=0x7f4a98009040 comp rx=0 tx=0).stop 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0b95a0 msgr2=0x7f4aac1e8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0b95a0 0x7f4aac1e8130 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00098c0 tx=0x7f4aa00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 shutdown_connections 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a9006b500 0x7f4a9006d9c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4aac0ccf20 0x7f4aac0cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4aac15df00 0x7f4aac14f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 --2- 172.21.15.27:0/3827733969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0b95a0 0x7f4aac1e8130 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 >> 172.21.15.27:0/3827733969 conn(0x7f4aac071960 msgr2=0x7f4aac0d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 shutdown_connections 2024-03-06T07:26:43.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:43.816+0000 7f4ab2bf7640 1 -- 172.21.15.27:0/3827733969 wait complete. 2024-03-06T07:26:43.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:26:43.971 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:26:43.972 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:26:43.972 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:26:48.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:48.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:48.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:49.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.077+0000 7fe4ee89d640 1 Processor -- start 2024-03-06T07:26:49.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.077+0000 7fe4ee89d640 1 -- start start 2024-03-06T07:26:49.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4ee89d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e815df00 0x7fe4e80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4ee89d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e80cea30 con 0x7fe4e80b9a30 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4ee89d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e80d2af0 con 0x7fe4e80b95a0 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4ee89d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e80cf3c0 con 0x7fe4e815df00 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4ece3a640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe4e80b9a30 0x7fe4e80d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38778/0 (socket says 172.21.15.27:38778) 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4ece3a640 1 -- 172.21.15.27:0/3052405387 learned_addr learned my addr 172.21.15.27:0/3052405387 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059613685 0 0) 0x7fe4e80cea30 con 0x7fe4e80b9a30 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d4003520 con 0x7fe4e80b9a30 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e7fff640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e815df00 0x7fe4e80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2380286948 0 0) 0x7fe4d4003520 con 0x7fe4e80b9a30 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e815df00 msgr2=0x7fe4e80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e815df00 0x7fe4e80d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 >> v1:172.21.15.69:6789/0 conn(0x7fe4e80b95a0 legacy=0x7fe4e80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e80cf0e0 con 0x7fe4e80b9a30 2024-03-06T07:26:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fe4e00019f0 con 0x7fe4e80b9a30 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 >> v1:172.21.15.27:6789/0 conn(0x7fe4e80b9a30 legacy=0x7fe4e80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.078+0000 7fe4e6ffd640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4d4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e6ffd640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4d4004800 0x7fe4d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e6ffd640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e815df00 0x7fe4d4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e80cf3c0 con 0x7fe4d40039e0 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d400baf0 con 0x7fe4d4004800 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d400bcf0 con 0x7fe4e815df00 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e7fff640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4d4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e77fe640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4d4004800 0x7fe4d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4ece3a640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e815df00 0x7fe4d4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e7fff640 1 -- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e815df00 msgr2=0x7fe4d4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e7fff640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e815df00 0x7fe4d4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e7fff640 1 -- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4d4004800 msgr2=0x7fe4d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e7fff640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4d4004800 0x7fe4d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4e7fff640 1 -- 172.21.15.27:0/3052405387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d4003520 con 0x7fe4d40039e0 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.079+0000 7fe4ece3a640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e815df00 0x7fe4d4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4e7fff640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4d4003e00 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800ba30 tx=0x7fe4d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab78792136d1a39d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8036020 con 0x7fe4d40039e0 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d8032640 con 0x7fe4d40039e0 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4e6ffd640 1 -- 172.21.15.27:0/3052405387 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d800b600 con 0x7fe4d40039e0 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4ee89d640 1 -- 172.21.15.27:0/3052405387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 msgr2=0x7fe4d4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4ee89d640 1 --2- 172.21.15.27:0/3052405387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4d4003e00 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800ba30 tx=0x7fe4d4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4ece3a640 1 -- 172.21.15.27:0/3052405387 reap_dead start 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4ee89d640 1 -- 172.21.15.27:0/3052405387 shutdown_connections 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4ee89d640 1 -- 172.21.15.27:0/3052405387 >> 172.21.15.27:0/3052405387 conn(0x7fe4e8071960 msgr2=0x7fe4e8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.080+0000 7fe4ee89d640 1 -- 172.21.15.27:0/3052405387 shutdown_connections 2024-03-06T07:26:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.081+0000 7fe4ee89d640 1 -- 172.21.15.27:0/3052405387 wait complete. 2024-03-06T07:26:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.082+0000 7fe4ee89d640 1 Processor -- start 2024-03-06T07:26:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.082+0000 7fe4ee89d640 1 -- start start 2024-03-06T07:26:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4ee89d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4e80cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4ee89d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e80c8d90 0x7fe4e80b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4ee89d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e80c91f0 0x7fe4e80b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4ee89d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e80cec70 con 0x7fe4d40039e0 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4ee89d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e80ce750 con 0x7fe4e80c91f0 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4ee89d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e80cedf0 con 0x7fe4e80c8d90 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4e80cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4e80cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32892/0 (socket says 172.21.15.27:32892) 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e7fff640 1 -- 172.21.15.27:0/1079120656 learned_addr learned my addr 172.21.15.27:0/1079120656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e7fff640 1 -- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e80c8d90 msgr2=0x7fe4e80b4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e7fff640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e80c8d90 0x7fe4e80b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e7fff640 1 -- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e80c91f0 msgr2=0x7fe4e80b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e7fff640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e80c91f0 0x7fe4e80b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e7fff640 1 -- 172.21.15.27:0/1079120656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e81f80a0 con 0x7fe4d40039e0 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e7fff640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4e80cbb10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800bb20 tx=0x7fe4d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e4ff9640 1 -- 172.21.15.27:0/1079120656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8036040 con 0x7fe4d40039e0 2024-03-06T07:26:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e4ff9640 1 -- 172.21.15.27:0/1079120656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d8030a00 con 0x7fe4d40039e0 2024-03-06T07:26:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e80cad50 con 0x7fe4d40039e0 2024-03-06T07:26:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4e4ff9640 1 -- 172.21.15.27:0/1079120656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d800b600 con 0x7fe4d40039e0 2024-03-06T07:26:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.083+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e80cb310 con 0x7fe4d40039e0 2024-03-06T07:26:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.084+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e80722a0 con 0x7fe4d40039e0 2024-03-06T07:26:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.085+0000 7fe4e4ff9640 1 -- 172.21.15.27:0/1079120656 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4d8039020 con 0x7fe4d40039e0 2024-03-06T07:26:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.085+0000 7fe4e4ff9640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4c806b4d0 0x7fe4c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.086+0000 7fe4e4ff9640 1 -- 172.21.15.27:0/1079120656 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe4d80ac890 con 0x7fe4d40039e0 2024-03-06T07:26:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.086+0000 7fe4e77fe640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4c806b4d0 0x7fe4c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.086+0000 7fe4e77fe640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4c806b4d0 0x7fe4c806d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8161100 tx=0x7fe4d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.090+0000 7fe4e4ff9640 1 -- 172.21.15.27:0/1079120656 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4e80722a0 con 0x7fe4d40039e0 2024-03-06T07:26:49.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.369+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4e80b60e0 con 0x7fe4d40039e0 2024-03-06T07:26:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.369+0000 7fe4e4ff9640 1 -- 172.21.15.27:0/1079120656 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fe4e80b60e0 con 0x7fe4d40039e0 2024-03-06T07:26:49.372 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:49.372 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:49.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4c806b4d0 msgr2=0x7fe4c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4c806b4d0 0x7fe4c806d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8161100 tx=0x7fe4d0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:49.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 msgr2=0x7fe4e80cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4e80cbb10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800bb20 tx=0x7fe4d8004730 comp rx=0 tx=0).stop 2024-03-06T07:26:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 shutdown_connections 2024-03-06T07:26:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4c806b4d0 0x7fe4c806d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e80c91f0 0x7fe4e80b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4e80c8d90 0x7fe4e80b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 --2- 172.21.15.27:0/1079120656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4d40039e0 0x7fe4e80cbb10 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 >> 172.21.15.27:0/1079120656 conn(0x7fe4e8071960 msgr2=0x7fe4e80b87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 shutdown_connections 2024-03-06T07:26:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.371+0000 7fe4ee89d640 1 -- 172.21.15.27:0/1079120656 wait complete. 2024-03-06T07:26:49.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:26:49.507 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:26:49.507 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:26:49.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:49.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.631+0000 7f9a3fa88640 1 Processor -- start 2024-03-06T07:26:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.631+0000 7f9a3fa88640 1 -- start start 2024-03-06T07:26:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.632+0000 7f9a3fa88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a380d0880 0x7f9a38168c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.632+0000 7f9a3fa88640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380cf130 con 0x7f9a3815e060 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.632+0000 7f9a3fa88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380d0dd0 con 0x7f9a3806fe20 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.632+0000 7f9a3fa88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a380d10b0 con 0x7f9a380d0880 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.632+0000 7f9a3d824640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9a3815e060 0x7f9a380ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38790/0 (socket says 172.21.15.27:38790) 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.632+0000 7f9a3d824640 1 -- 172.21.15.27:0/3108722616 learned_addr learned my addr 172.21.15.27:0/3108722616 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.632+0000 7f9a3e025640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a380d0880 0x7f9a38168c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.632+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036901663 0 0) 0x7f9a380cf130 con 0x7f9a3815e060 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.632+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a24003520 con 0x7f9a3815e060 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 806022011 0 0) 0x7f9a24003520 con 0x7f9a3815e060 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a380d0880 msgr2=0x7f9a38168c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a380d0880 0x7f9a38168c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 >> v1:172.21.15.69:6789/0 conn(0x7f9a3806fe20 legacy=0x7f9a380cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a380cc0e0 con 0x7f9a3815e060 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f9a34001670 con 0x7f9a3815e060 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 >> v1:172.21.15.27:6789/0 conn(0x7f9a3815e060 legacy=0x7f9a380ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a240039e0 0x7f9a24003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a380d0880 0x7f9a24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a24004d40 0x7f9a240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a380d10b0 con 0x7f9a24004d40 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2400ba70 con 0x7f9a240039e0 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2400bc70 con 0x7f9a380d0880 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3d824640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a240039e0 0x7f9a24003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3e025640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a24004d40 0x7f9a240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3d023640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a380d0880 0x7f9a24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3e025640 1 -- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a380d0880 msgr2=0x7f9a24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3e025640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a380d0880 0x7f9a24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3e025640 1 -- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a240039e0 msgr2=0x7f9a24003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3e025640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a240039e0 0x7f9a24003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.633+0000 7f9a3e025640 1 -- 172.21.15.27:0/3108722616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a24003520 con 0x7f9a24004d40 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.634+0000 7f9a3e025640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a24004d40 0x7f9a240051c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009f90 tx=0x7f9a24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8047ff47ec73b04b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.634+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30039070 con 0x7f9a24004d40 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.634+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3002cbd0 con 0x7f9a24004d40 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.634+0000 7f9a3c822640 1 -- 172.21.15.27:0/3108722616 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30009c10 con 0x7f9a24004d40 2024-03-06T07:26:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.635+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3108722616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a24004d40 msgr2=0x7f9a240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.635+0000 7f9a3fa88640 1 --2- 172.21.15.27:0/3108722616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a24004d40 0x7f9a240051c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009f90 tx=0x7f9a24005830 comp rx=0 tx=0).stop 2024-03-06T07:26:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.635+0000 7f9a3e025640 1 -- 172.21.15.27:0/3108722616 reap_dead start 2024-03-06T07:26:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.635+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3108722616 shutdown_connections 2024-03-06T07:26:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.635+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3108722616 >> 172.21.15.27:0/3108722616 conn(0x7f9a380d1a90 msgr2=0x7f9a380d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.635+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3108722616 shutdown_connections 2024-03-06T07:26:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.635+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3108722616 wait complete. 2024-03-06T07:26:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.636+0000 7f9a3fa88640 1 Processor -- start 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.636+0000 7f9a3fa88640 1 -- start start 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3fa88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a381f53a0 0x7f9a381f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3fa88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a381f76f0 0x7f9a381f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3fa88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b26d0 0x7f9a380b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3fa88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3815e440 con 0x7f9a381f53a0 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3fa88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a380cc6a0 con 0x7f9a380b26d0 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3fa88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a380cdbd0 con 0x7f9a381f76f0 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d824640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a381f53a0 0x7f9a381f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d824640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a381f53a0 0x7f9a381f5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32906/0 (socket says 172.21.15.27:32906) 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d824640 1 -- 172.21.15.27:0/3314827261 learned_addr learned my addr 172.21.15.27:0/3314827261 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d023640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a381f76f0 0x7f9a381f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3e025640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b26d0 0x7f9a380b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d824640 1 -- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a381f76f0 msgr2=0x7f9a381f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d824640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a381f76f0 0x7f9a381f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d824640 1 -- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b26d0 msgr2=0x7f9a380b2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d824640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b26d0 0x7f9a380b2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d824640 1 -- 172.21.15.27:0/3314827261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a381f6650 con 0x7f9a381f53a0 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.637+0000 7f9a3d023640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a381f76f0 0x7f9a381f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.638+0000 7f9a3d824640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a381f53a0 0x7f9a381f5ae0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9a34007290 tx=0x7f9a34007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.638+0000 7f9a2a7fc640 1 -- 172.21.15.27:0/3314827261 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3400f9a0 con 0x7f9a381f53a0 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.638+0000 7f9a2a7fc640 1 -- 172.21.15.27:0/3314827261 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a34018070 con 0x7f9a381f53a0 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.638+0000 7f9a2a7fc640 1 -- 172.21.15.27:0/3314827261 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3400fd80 con 0x7f9a381f53a0 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.638+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a381f6930 con 0x7f9a381f53a0 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.638+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a381f6ef0 con 0x7f9a381f53a0 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.639+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a38072560 con 0x7f9a381f53a0 2024-03-06T07:26:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.639+0000 7f9a2a7fc640 1 -- 172.21.15.27:0/3314827261 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a3401d460 con 0x7f9a381f53a0 2024-03-06T07:26:49.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.640+0000 7f9a2a7fc640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1c06b500 0x7f9a1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.640+0000 7f9a2a7fc640 1 -- 172.21.15.27:0/3314827261 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9a3408a340 con 0x7f9a381f53a0 2024-03-06T07:26:49.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.640+0000 7f9a3d023640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1c06b500 0x7f9a1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.641+0000 7f9a3d023640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1c06b500 0x7f9a1c06d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9a380b0d80 tx=0x7f9a2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.644+0000 7f9a2a7fc640 1 -- 172.21.15.27:0/3314827261 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a38072560 con 0x7f9a381f53a0 2024-03-06T07:26:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.862+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a38067ff0 con 0x7f9a381f53a0 2024-03-06T07:26:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.862+0000 7f9a2a7fc640 1 -- 172.21.15.27:0/3314827261 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f9a34059180 con 0x7f9a381f53a0 2024-03-06T07:26:49.865 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:49.865 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:49.865 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:49.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1c06b500 msgr2=0x7f9a1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1c06b500 0x7f9a1c06d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9a380b0d80 tx=0x7f9a2c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:49.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a381f53a0 msgr2=0x7f9a381f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a381f53a0 0x7f9a381f5ae0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9a34007290 tx=0x7f9a34007750 comp rx=0 tx=0).stop 2024-03-06T07:26:49.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 shutdown_connections 2024-03-06T07:26:49.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1c06b500 0x7f9a1c06d9c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a380b26d0 0x7f9a380b2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a381f76f0 0x7f9a381f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 --2- 172.21.15.27:0/3314827261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a381f53a0 0x7f9a381f5ae0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 >> 172.21.15.27:0/3314827261 conn(0x7f9a380d1a90 msgr2=0x7f9a380ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 shutdown_connections 2024-03-06T07:26:49.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:49.864+0000 7f9a3fa88640 1 -- 172.21.15.27:0/3314827261 wait complete. 2024-03-06T07:26:49.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:26:50.004 INFO:teuthology.orchestra.run.smithi027.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:26:50.005 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:26:50.005 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:26:50.005 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> import uuid 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> n = 10 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = False 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> if 0 > 0: 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> create_hard_links = True 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> dpath = os.path.dirname(path) 2024-03-06T07:26:50.021 DEBUG:teuthology.orchestra.run.smithi027:> fnameprefix = os.path.basename(path) 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> try: 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> for i in range(n): 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> f.write(f"{i}") 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> f.flush() 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(f.fileno()) 2024-03-06T07:26:50.022 DEBUG:teuthology.orchestra.run.smithi027:> if True: 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> os.unlink(fpath) 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> if True: 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> if create_hard_links: 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> for j in range(0): 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> if False: 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> os.fsync(dirfd) 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> finally: 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> os.close(dirfd) 2024-03-06T07:26:50.023 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:26:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:50.196 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:50.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:50.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:50.326 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:50.326 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:26:50.326 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:50.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.486+0000 7f6c8406b640 1 Processor -- start 2024-03-06T07:26:50.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.486+0000 7f6c8406b640 1 -- start start 2024-03-06T07:26:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c8406b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6c7c15df00 0x7f6c7c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c8406b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6c7c0b9a30 0x7f6c7c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c8406b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c0cea30 con 0x7f6c7c0b95a0 2024-03-06T07:26:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c8406b640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7c0d2af0 con 0x7f6c7c0b9a30 2024-03-06T07:26:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c8406b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7c0cf3c0 con 0x7f6c7c15df00 2024-03-06T07:26:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c82608640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6c7c0b9a30 0x7f6c7c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c82608640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6c7c0b9a30 0x7f6c7c0d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32910/0 (socket says 172.21.15.27:32910) 2024-03-06T07:26:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c82608640 1 -- 172.21.15.27:0/1347151759 learned_addr learned my addr 172.21.15.27:0/1347151759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c81e07640 1 --2- 172.21.15.27:0/1347151759 >> v2:172.21.15.69:3300/0 conn(0x7f6c7c15df00 0x7f6c7c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c82608640 1 -- 172.21.15.27:0/1347151759 >> v2:172.21.15.69:3300/0 conn(0x7f6c7c15df00 msgr2=0x7f6c7c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c82608640 1 --2- 172.21.15.27:0/1347151759 >> v2:172.21.15.69:3300/0 conn(0x7f6c7c15df00 0x7f6c7c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c82608640 1 -- 172.21.15.27:0/1347151759 >> v1:172.21.15.69:6789/0 conn(0x7f6c7c0b95a0 legacy=0x7f6c7c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.487+0000 7f6c82608640 1 -- 172.21.15.27:0/1347151759 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c0cf0e0 con 0x7f6c7c0b9a30 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.488+0000 7f6c82608640 1 --2- 172.21.15.27:0/1347151759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c0b9a30 0x7f6c7c0d3220 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0d11b0 tx=0x7f6c7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ec08099457e8d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.488+0000 7f6c80e05640 1 -- 172.21.15.27:0/1347151759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7402ca80 con 0x7f6c7c0b9a30 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.488+0000 7f6c80e05640 1 -- 172.21.15.27:0/1347151759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7402cc20 con 0x7f6c7c0b9a30 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.488+0000 7f6c80e05640 1 -- 172.21.15.27:0/1347151759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c740343f0 con 0x7f6c7c0b9a30 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.489+0000 7f6c8406b640 1 -- 172.21.15.27:0/1347151759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c0b9a30 msgr2=0x7f6c7c0d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.489+0000 7f6c8406b640 1 --2- 172.21.15.27:0/1347151759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c0b9a30 0x7f6c7c0d3220 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0d11b0 tx=0x7f6c7402bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.489+0000 7f6c8406b640 1 -- 172.21.15.27:0/1347151759 shutdown_connections 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.489+0000 7f6c8406b640 1 --2- 172.21.15.27:0/1347151759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c0b9a30 0x7f6c7c0d3220 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.489+0000 7f6c8406b640 1 --2- 172.21.15.27:0/1347151759 >> v2:172.21.15.69:3300/0 conn(0x7f6c7c15df00 0x7f6c7c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.489+0000 7f6c8406b640 1 -- 172.21.15.27:0/1347151759 >> 172.21.15.27:0/1347151759 conn(0x7f6c7c071960 msgr2=0x7f6c7c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.489+0000 7f6c8406b640 1 -- 172.21.15.27:0/1347151759 shutdown_connections 2024-03-06T07:26:50.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.489+0000 7f6c8406b640 1 -- 172.21.15.27:0/1347151759 wait complete. 2024-03-06T07:26:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.490+0000 7f6c8406b640 1 Processor -- start 2024-03-06T07:26:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.490+0000 7f6c8406b640 1 -- start start 2024-03-06T07:26:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c8406b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c7c0b95a0 0x7f6c7c0ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c8406b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c7c15df00 0x7f6c7c15ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c8406b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c1552a0 0x7f6c7c15faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c8406b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0cec70 con 0x7f6c7c1552a0 2024-03-06T07:26:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c8406b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0ce750 con 0x7f6c7c0b95a0 2024-03-06T07:26:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c8406b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0cedf0 con 0x7f6c7c15df00 2024-03-06T07:26:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c82608640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c1552a0 0x7f6c7c15faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c82608640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c1552a0 0x7f6c7c15faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32912/0 (socket says 172.21.15.27:32912) 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c82608640 1 -- 172.21.15.27:0/2944562622 learned_addr learned my addr 172.21.15.27:0/2944562622 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c81e07640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c7c0b95a0 0x7f6c7c0ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c81606640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c7c15df00 0x7f6c7c15ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c82608640 1 -- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c7c15df00 msgr2=0x7f6c7c15ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c82608640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c7c15df00 0x7f6c7c15ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c82608640 1 -- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c7c0b95a0 msgr2=0x7f6c7c0ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c82608640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c7c0b95a0 0x7f6c7c0ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c82608640 1 -- 172.21.15.27:0/2944562622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c15f490 con 0x7f6c7c1552a0 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c82608640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c1552a0 0x7f6c7c15faf0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009f40 tx=0x7f6c74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.491+0000 7f6c727fc640 1 -- 172.21.15.27:0/2944562622 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74043020 con 0x7f6c7c1552a0 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.492+0000 7f6c727fc640 1 -- 172.21.15.27:0/2944562622 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7403e450 con 0x7f6c7c1552a0 2024-03-06T07:26:50.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.492+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c15f770 con 0x7f6c7c1552a0 2024-03-06T07:26:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.492+0000 7f6c727fc640 1 -- 172.21.15.27:0/2944562622 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74009510 con 0x7f6c7c1552a0 2024-03-06T07:26:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.492+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c0cb280 con 0x7f6c7c1552a0 2024-03-06T07:26:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.493+0000 7f6c727fc640 1 -- 172.21.15.27:0/2944562622 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c74046020 con 0x7f6c7c1552a0 2024-03-06T07:26:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.493+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c067ff0 con 0x7f6c7c1552a0 2024-03-06T07:26:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.493+0000 7f6c727fc640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c6006b4d0 0x7f6c6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.493+0000 7f6c727fc640 1 -- 172.21.15.27:0/2944562622 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6c740ac9f0 con 0x7f6c7c1552a0 2024-03-06T07:26:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.494+0000 7f6c81e07640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c6006b4d0 0x7f6c6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.494+0000 7f6c81e07640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c6006b4d0 0x7f6c6006d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6c780079a0 tx=0x7f6c78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.498+0000 7f6c727fc640 1 -- 172.21.15.27:0/2944562622 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c7404d030 con 0x7f6c7c1552a0 2024-03-06T07:26:50.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.774+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c7c0d27c0 con 0x7f6c7c1552a0 2024-03-06T07:26:50.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.775+0000 7f6c727fc640 1 -- 172.21.15.27:0/2944562622 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f6c7407b950 con 0x7f6c7c1552a0 2024-03-06T07:26:50.777 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:50.777 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:29.258129+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.914214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.608105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.2e","mappings":[{"from":6,"to":0}]},{"pgid":"11.3a","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:50.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c6006b4d0 msgr2=0x7f6c6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c6006b4d0 0x7f6c6006d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6c780079a0 tx=0x7f6c78008040 comp rx=0 tx=0).stop 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c1552a0 msgr2=0x7f6c7c15faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c1552a0 0x7f6c7c15faf0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009f40 tx=0x7f6c74004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 shutdown_connections 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c6006b4d0 0x7f6c6006d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c7c1552a0 0x7f6c7c15faf0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c7c15df00 0x7f6c7c15ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 --2- 172.21.15.27:0/2944562622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c7c0b95a0 0x7f6c7c0ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 >> 172.21.15.27:0/2944562622 conn(0x7f6c7c071960 msgr2=0x7f6c7c067550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 shutdown_connections 2024-03-06T07:26:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.777+0000 7f6c8406b640 1 -- 172.21.15.27:0/2944562622 wait complete. 2024-03-06T07:26:50.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:17.914214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:18.608105+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:50.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:50.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.898+0000 7ff8db037640 1 Processor -- start 2024-03-06T07:26:50.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.898+0000 7ff8db037640 1 -- start start 2024-03-06T07:26:50.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.899+0000 7ff8db037640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8d40717b0 0x7ff8d4071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.899+0000 7ff8db037640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8d40b2b10 0x7ff8d40d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.899+0000 7ff8db037640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d415e4b0 con 0x7ff8d40b3060 2024-03-06T07:26:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.899+0000 7ff8db037640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d40d2e10 con 0x7ff8d40717b0 2024-03-06T07:26:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.899+0000 7ff8db037640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d40b97e0 con 0x7ff8d40b2b10 2024-03-06T07:26:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.899+0000 7ff8d8dd3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8d40717b0 0x7ff8d4071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.899+0000 7ff8d3fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff8d40b3060 0x7ff8d40b6e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38794/0 (socket says 172.21.15.27:38794) 2024-03-06T07:26:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.899+0000 7ff8d3fff640 1 -- 172.21.15.27:0/416054315 learned_addr learned my addr 172.21.15.27:0/416054315 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.900+0000 7ff8d95d4640 1 --2- 172.21.15.27:0/416054315 >> v2:172.21.15.69:3300/0 conn(0x7ff8d40b2b10 0x7ff8d40d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.900+0000 7ff8d37fe640 1 -- 172.21.15.27:0/416054315 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033509236 0 0) 0x7ff8c8001110 con 0x7ff8d40b3060 2024-03-06T07:26:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.900+0000 7ff8d37fe640 1 -- 172.21.15.27:0/416054315 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8bc003520 con 0x7ff8d40b3060 2024-03-06T07:26:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.900+0000 7ff8d8dd3640 1 -- 172.21.15.27:0/416054315 >> v2:172.21.15.69:3300/0 conn(0x7ff8d40b2b10 msgr2=0x7ff8d40d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.900+0000 7ff8d8dd3640 1 --2- 172.21.15.27:0/416054315 >> v2:172.21.15.69:3300/0 conn(0x7ff8d40b2b10 0x7ff8d40d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.900+0000 7ff8d8dd3640 1 -- 172.21.15.27:0/416054315 >> v1:172.21.15.27:6789/0 conn(0x7ff8d40b3060 legacy=0x7ff8d40b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.900+0000 7ff8d8dd3640 1 -- 172.21.15.27:0/416054315 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d40b9a80 con 0x7ff8d40717b0 2024-03-06T07:26:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8d8dd3640 1 --2- 172.21.15.27:0/416054315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40717b0 0x7ff8d4071b90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40736d0 tx=0x7ff8c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a1133dbb696194 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8d37fe640 1 -- 172.21.15.27:0/416054315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4035020 con 0x7ff8d40717b0 2024-03-06T07:26:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8d37fe640 1 -- 172.21.15.27:0/416054315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c4031830 con 0x7ff8d40717b0 2024-03-06T07:26:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8d37fe640 1 -- 172.21.15.27:0/416054315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4031c00 con 0x7ff8d40717b0 2024-03-06T07:26:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8db037640 1 -- 172.21.15.27:0/416054315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40717b0 msgr2=0x7ff8d4071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8db037640 1 --2- 172.21.15.27:0/416054315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40717b0 0x7ff8d4071b90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40736d0 tx=0x7ff8c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8db037640 1 -- 172.21.15.27:0/416054315 shutdown_connections 2024-03-06T07:26:50.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8db037640 1 --2- 172.21.15.27:0/416054315 >> v2:172.21.15.69:3300/0 conn(0x7ff8d40b2b10 0x7ff8d40d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8db037640 1 --2- 172.21.15.27:0/416054315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40717b0 0x7ff8d4071b90 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8db037640 1 -- 172.21.15.27:0/416054315 >> 172.21.15.27:0/416054315 conn(0x7ff8d40d3d20 msgr2=0x7ff8d40d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8db037640 1 -- 172.21.15.27:0/416054315 shutdown_connections 2024-03-06T07:26:50.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.901+0000 7ff8db037640 1 -- 172.21.15.27:0/416054315 wait complete. 2024-03-06T07:26:50.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.902+0000 7ff8db037640 1 Processor -- start 2024-03-06T07:26:50.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.902+0000 7ff8db037640 1 -- start start 2024-03-06T07:26:50.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.903+0000 7ff8db037640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8d40717b0 0x7ff8d41e3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.903+0000 7ff8db037640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8d40b2b10 0x7ff8d414d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.903+0000 7ff8db037640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40b3060 0x7ff8d41e4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.903+0000 7ff8db037640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d40b7560 con 0x7ff8d40b3060 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.903+0000 7ff8db037640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d40d4bf0 con 0x7ff8d40717b0 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.903+0000 7ff8db037640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d40b76e0 con 0x7ff8d40b2b10 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.903+0000 7ff8d95d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40b3060 0x7ff8d41e4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d95d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40b3060 0x7ff8d41e4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32936/0 (socket says 172.21.15.27:32936) 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d95d4640 1 -- 172.21.15.27:0/338731912 learned_addr learned my addr 172.21.15.27:0/338731912 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d8dd3640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8d40717b0 0x7ff8d41e3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d3fff640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8d40b2b10 0x7ff8d414d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d95d4640 1 -- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8d40b2b10 msgr2=0x7ff8d414d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d95d4640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8d40b2b10 0x7ff8d414d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d95d4640 1 -- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8d40717b0 msgr2=0x7ff8d41e3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d95d4640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8d40717b0 0x7ff8d41e3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d95d4640 1 -- 172.21.15.27:0/338731912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d414d790 con 0x7ff8d40b3060 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d95d4640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40b3060 0x7ff8d41e4950 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00efc0 tx=0x7ff8cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d17fa640 1 -- 172.21.15.27:0/338731912 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc018020 con 0x7ff8d40b3060 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d17fa640 1 -- 172.21.15.27:0/338731912 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc014480 con 0x7ff8d40b3060 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d41e5210 con 0x7ff8d40b3060 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8d17fa640 1 -- 172.21.15.27:0/338731912 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc014870 con 0x7ff8d40b3060 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.904+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d40cd010 con 0x7ff8d40b3060 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.905+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d406ce40 con 0x7ff8d40b3060 2024-03-06T07:26:50.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.906+0000 7ff8d17fa640 1 -- 172.21.15.27:0/338731912 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8cc005d10 con 0x7ff8d40b3060 2024-03-06T07:26:50.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.906+0000 7ff8d17fa640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff8c006b4d0 0x7ff8c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.906+0000 7ff8d17fa640 1 -- 172.21.15.27:0/338731912 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff8cc08ad30 con 0x7ff8d40b3060 2024-03-06T07:26:50.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.906+0000 7ff8d8dd3640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff8c006b4d0 0x7ff8c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.907+0000 7ff8d8dd3640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff8c006b4d0 0x7ff8c006d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4036000 tx=0x7ff8c4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:50.915+0000 7ff8d17fa640 1 -- 172.21.15.27:0/338731912 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8cc022070 con 0x7ff8d40b3060 2024-03-06T07:26:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.172+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8d404f410 con 0x7ff8d40b3060 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.172+0000 7ff8d17fa640 1 -- 172.21.15.27:0/338731912 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff8d414d790 con 0x7ff8d40b3060 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff8c006b4d0 msgr2=0x7ff8c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff8c006b4d0 0x7ff8c006d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4036000 tx=0x7ff8c4008030 comp rx=0 tx=0).stop 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40b3060 msgr2=0x7ff8d41e4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40b3060 0x7ff8d41e4950 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00efc0 tx=0x7ff8cc00c490 comp rx=0 tx=0).stop 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 shutdown_connections 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff8c006b4d0 0x7ff8c006d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d40b3060 0x7ff8d41e4950 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8d40b2b10 0x7ff8d414d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 --2- 172.21.15.27:0/338731912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8d40717b0 0x7ff8d41e3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 >> 172.21.15.27:0/338731912 conn(0x7ff8d40d3d20 msgr2=0x7ff8d40b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 shutdown_connections 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.174+0000 7ff8db037640 1 -- 172.21.15.27:0/338731912 wait complete. 2024-03-06T07:26:51.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:51.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.296+0000 7f7b6273e640 1 Processor -- start 2024-03-06T07:26:51.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.296+0000 7f7b6273e640 1 -- start start 2024-03-06T07:26:51.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b6273e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b5c0ceea0 0x7f7b5c0d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b6273e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b5c0b6d10 0x7f7b5c0b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b6273e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c0d3740 con 0x7f7b5c15dd90 2024-03-06T07:26:51.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b6273e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b5c0d3400 con 0x7f7b5c0ceea0 2024-03-06T07:26:51.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b6273e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b5c0d3120 con 0x7f7b5c0b6d10 2024-03-06T07:26:51.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b5b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b5c0ceea0 0x7f7b5c0d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b5bfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7b5c15dd90 0x7f7b5c0d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38796/0 (socket says 172.21.15.27:38796) 2024-03-06T07:26:51.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b5bfff640 1 -- 172.21.15.27:0/2702325841 learned_addr learned my addr 172.21.15.27:0/2702325841 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:51.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b60cdb640 1 --2- 172.21.15.27:0/2702325841 >> v2:172.21.15.69:3300/0 conn(0x7f7b5c0b6d10 0x7f7b5c0b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b5b7fe640 1 -- 172.21.15.27:0/2702325841 >> v2:172.21.15.69:3300/0 conn(0x7f7b5c0b6d10 msgr2=0x7f7b5c0b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b5b7fe640 1 --2- 172.21.15.27:0/2702325841 >> v2:172.21.15.69:3300/0 conn(0x7f7b5c0b6d10 0x7f7b5c0b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b5b7fe640 1 -- 172.21.15.27:0/2702325841 >> v1:172.21.15.27:6789/0 conn(0x7f7b5c15dd90 legacy=0x7f7b5c0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b5affd640 1 -- 172.21.15.27:0/2702325841 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544064347 0 0) 0x7f7b5c0d3740 con 0x7f7b5c15dd90 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b5b7fe640 1 -- 172.21.15.27:0/2702325841 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c0d39e0 con 0x7f7b5c0ceea0 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.297+0000 7f7b60cdb640 1 --2- 172.21.15.27:0/2702325841 >> v2:172.21.15.69:3300/0 conn(0x7f7b5c0b6d10 0x7f7b5c0b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.298+0000 7f7b5b7fe640 1 --2- 172.21.15.27:0/2702325841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c0ceea0 0x7f7b5c0d29d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0b5850 tx=0x7f7b4c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f6c1db9288ac5d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b5affd640 1 -- 172.21.15.27:0/2702325841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c034070 con 0x7f7b5c0ceea0 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b5affd640 1 -- 172.21.15.27:0/2702325841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4c030610 con 0x7f7b5c0ceea0 2024-03-06T07:26:51.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b5affd640 1 -- 172.21.15.27:0/2702325841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c0309e0 con 0x7f7b5c0ceea0 2024-03-06T07:26:51.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b6273e640 1 -- 172.21.15.27:0/2702325841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c0ceea0 msgr2=0x7f7b5c0d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b6273e640 1 --2- 172.21.15.27:0/2702325841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c0ceea0 0x7f7b5c0d29d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0b5850 tx=0x7f7b4c02b960 comp rx=0 tx=0).stop 2024-03-06T07:26:51.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b6273e640 1 -- 172.21.15.27:0/2702325841 shutdown_connections 2024-03-06T07:26:51.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b6273e640 1 --2- 172.21.15.27:0/2702325841 >> v2:172.21.15.69:3300/0 conn(0x7f7b5c0b6d10 0x7f7b5c0b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b6273e640 1 --2- 172.21.15.27:0/2702325841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c0ceea0 0x7f7b5c0d29d0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b6273e640 1 -- 172.21.15.27:0/2702325841 >> 172.21.15.27:0/2702325841 conn(0x7f7b5c071960 msgr2=0x7f7b5c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b6273e640 1 -- 172.21.15.27:0/2702325841 shutdown_connections 2024-03-06T07:26:51.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.299+0000 7f7b6273e640 1 -- 172.21.15.27:0/2702325841 wait complete. 2024-03-06T07:26:51.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.301+0000 7f7b6273e640 1 Processor -- start 2024-03-06T07:26:51.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.301+0000 7f7b6273e640 1 -- start start 2024-03-06T07:26:51.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.301+0000 7f7b6273e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b5c0b6d10 0x7f7b5c15e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.301+0000 7f7b6273e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b5c0ceea0 0x7f7b5c14cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.301+0000 7f7b6273e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c15dd90 0x7f7b5c0b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.301+0000 7f7b6273e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c0d48f0 con 0x7f7b5c15dd90 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.301+0000 7f7b6273e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c15e170 con 0x7f7b5c0b6d10 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.301+0000 7f7b6273e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c0d4a70 con 0x7f7b5c0ceea0 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b60cdb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c15dd90 0x7f7b5c0b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b60cdb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c15dd90 0x7f7b5c0b4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32958/0 (socket says 172.21.15.27:32958) 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b60cdb640 1 -- 172.21.15.27:0/1449731811 learned_addr learned my addr 172.21.15.27:0/1449731811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b5bfff640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b5c0b6d10 0x7f7b5c15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b5b7fe640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b5c0ceea0 0x7f7b5c14cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b60cdb640 1 -- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b5c0ceea0 msgr2=0x7f7b5c14cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b60cdb640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b5c0ceea0 0x7f7b5c14cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b60cdb640 1 -- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b5c0b6d10 msgr2=0x7f7b5c15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b60cdb640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b5c0b6d10 0x7f7b5c15e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b60cdb640 1 -- 172.21.15.27:0/1449731811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c160da0 con 0x7f7b5c15dd90 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b5b7fe640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b5c0ceea0 0x7f7b5c14cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b60cdb640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c15dd90 0x7f7b5c0b4a70 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400cef0 tx=0x7f7b5400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b58ff9640 1 -- 172.21.15.27:0/1449731811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5401a070 con 0x7f7b5c15dd90 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b58ff9640 1 -- 172.21.15.27:0/1449731811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5400d860 con 0x7f7b5c15dd90 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b58ff9640 1 -- 172.21.15.27:0/1449731811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5400dc50 con 0x7f7b5c15dd90 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5c161080 con 0x7f7b5c15dd90 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.302+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5c0c7d50 con 0x7f7b5c15dd90 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.303+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b5c072a30 con 0x7f7b5c15dd90 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.304+0000 7f7b58ff9640 1 -- 172.21.15.27:0/1449731811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b5401d070 con 0x7f7b5c15dd90 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.304+0000 7f7b58ff9640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b4806b4d0 0x7f7b4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.304+0000 7f7b58ff9640 1 -- 172.21.15.27:0/1449731811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7b5408aa60 con 0x7f7b5c15dd90 2024-03-06T07:26:51.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.305+0000 7f7b5bfff640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b4806b4d0 0x7f7b4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.305+0000 7f7b5bfff640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b4806b4d0 0x7f7b4806d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7b50007fd0 tx=0x7f7b50007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.309+0000 7f7b58ff9640 1 -- 172.21.15.27:0/1449731811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b54059310 con 0x7f7b5c15dd90 2024-03-06T07:26:51.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.532+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7b5c04f410 con 0x7f7b5c15dd90 2024-03-06T07:26:51.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.532+0000 7f7b58ff9640 1 -- 172.21.15.27:0/1449731811 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7b5405d3b0 con 0x7f7b5c15dd90 2024-03-06T07:26:51.535 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:26:51.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b4806b4d0 msgr2=0x7f7b4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b4806b4d0 0x7f7b4806d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7b50007fd0 tx=0x7f7b50007de0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c15dd90 msgr2=0x7f7b5c0b4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c15dd90 0x7f7b5c0b4a70 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400cef0 tx=0x7f7b5400a710 comp rx=0 tx=0).stop 2024-03-06T07:26:51.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 shutdown_connections 2024-03-06T07:26:51.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b4806b4d0 0x7f7b4806d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b5c15dd90 0x7f7b5c0b4a70 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b5c0ceea0 0x7f7b5c14cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 --2- 172.21.15.27:0/1449731811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b5c0b6d10 0x7f7b5c15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 >> 172.21.15.27:0/1449731811 conn(0x7f7b5c071960 msgr2=0x7f7b5c0b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 shutdown_connections 2024-03-06T07:26:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.535+0000 7f7b6273e640 1 -- 172.21.15.27:0/1449731811 wait complete. 2024-03-06T07:26:51.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:51.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.657+0000 7f982b30c640 1 Processor -- start 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.657+0000 7f982b30c640 1 -- start start 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f982b30c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f982415dd90 0x7f98240ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f982b30c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98240d29d0 0x7f98240d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f982b30c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240d3740 con 0x7f98240d4720 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f982b30c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98240d3400 con 0x7f98240d29d0 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f982b30c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98240d3120 con 0x7f982415dd90 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f98288a7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98240d29d0 0x7f98240d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f98288a7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98240d29d0 0x7f98240d41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32962/0 (socket says 172.21.15.27:32962) 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f98288a7640 1 -- 172.21.15.27:0/2660871901 learned_addr learned my addr 172.21.15.27:0/2660871901 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f98290a8640 1 --2- 172.21.15.27:0/2660871901 >> v2:172.21.15.69:3300/0 conn(0x7f982415dd90 0x7f98240ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f98288a7640 1 -- 172.21.15.27:0/2660871901 >> v2:172.21.15.69:3300/0 conn(0x7f982415dd90 msgr2=0x7f98240ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f98288a7640 1 --2- 172.21.15.27:0/2660871901 >> v2:172.21.15.69:3300/0 conn(0x7f982415dd90 0x7f98240ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f98288a7640 1 -- 172.21.15.27:0/2660871901 >> v1:172.21.15.69:6789/0 conn(0x7f98240d4720 legacy=0x7f98240b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.658+0000 7f98288a7640 1 -- 172.21.15.27:0/2660871901 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98240d39e0 con 0x7f98240d29d0 2024-03-06T07:26:51.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.659+0000 7f98288a7640 1 --2- 172.21.15.27:0/2660871901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 0x7f98240d41e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f98240d1f90 tx=0x7f981402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e925757191b087 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.659+0000 7f981bfff640 1 -- 172.21.15.27:0/2660871901 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814034070 con 0x7f98240d29d0 2024-03-06T07:26:51.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.659+0000 7f981bfff640 1 -- 172.21.15.27:0/2660871901 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9814030860 con 0x7f98240d29d0 2024-03-06T07:26:51.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.659+0000 7f981bfff640 1 -- 172.21.15.27:0/2660871901 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814030c30 con 0x7f98240d29d0 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.660+0000 7f982b30c640 1 -- 172.21.15.27:0/2660871901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 msgr2=0x7f98240d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.660+0000 7f982b30c640 1 --2- 172.21.15.27:0/2660871901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 0x7f98240d41e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f98240d1f90 tx=0x7f981402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.660+0000 7f982b30c640 1 -- 172.21.15.27:0/2660871901 shutdown_connections 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.660+0000 7f982b30c640 1 --2- 172.21.15.27:0/2660871901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 0x7f98240d41e0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.660+0000 7f982b30c640 1 --2- 172.21.15.27:0/2660871901 >> v2:172.21.15.69:3300/0 conn(0x7f982415dd90 0x7f98240ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.660+0000 7f982b30c640 1 -- 172.21.15.27:0/2660871901 >> 172.21.15.27:0/2660871901 conn(0x7f9824071960 msgr2=0x7f9824071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.660+0000 7f982b30c640 1 -- 172.21.15.27:0/2660871901 shutdown_connections 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.660+0000 7f982b30c640 1 -- 172.21.15.27:0/2660871901 wait complete. 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.661+0000 7f982b30c640 1 Processor -- start 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.661+0000 7f982b30c640 1 -- start start 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f982b30c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 0x7f9824157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f982b30c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98240d4720 0x7f982415f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f982b30c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f982415dd90 0x7f98241f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f982b30c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98240d2df0 con 0x7f98240d29d0 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f982b30c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982415e170 con 0x7f982415dd90 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f982b30c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98240b7420 con 0x7f98240d4720 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98290a8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 0x7f9824157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98290a8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 0x7f9824157020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32978/0 (socket says 172.21.15.27:32978) 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98290a8640 1 -- 172.21.15.27:0/3213367941 learned_addr learned my addr 172.21.15.27:0/3213367941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98298a9640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f982415dd90 0x7f98241f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98288a7640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98240d4720 0x7f982415f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98290a8640 1 -- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98240d4720 msgr2=0x7f982415f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98290a8640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98240d4720 0x7f982415f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98290a8640 1 -- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f982415dd90 msgr2=0x7f98241f6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98290a8640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f982415dd90 0x7f98241f6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98290a8640 1 -- 172.21.15.27:0/3213367941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824160290 con 0x7f98240d29d0 2024-03-06T07:26:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f98290a8640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 0x7f9824157020 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f982000c7f0 tx=0x7f982000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f9819ffb640 1 -- 172.21.15.27:0/3213367941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820015070 con 0x7f98240d29d0 2024-03-06T07:26:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f9819ffb640 1 -- 172.21.15.27:0/3213367941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9820004220 con 0x7f98240d29d0 2024-03-06T07:26:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9824160570 con 0x7f98240d29d0 2024-03-06T07:26:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.662+0000 7f9819ffb640 1 -- 172.21.15.27:0/3213367941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820007620 con 0x7f98240d29d0 2024-03-06T07:26:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.663+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9824160b30 con 0x7f98240d29d0 2024-03-06T07:26:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.664+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9824072a30 con 0x7f98240d29d0 2024-03-06T07:26:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.664+0000 7f9819ffb640 1 -- 172.21.15.27:0/3213367941 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98200079d0 con 0x7f98240d29d0 2024-03-06T07:26:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.665+0000 7f9819ffb640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f980806b4d0 0x7f980806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.665+0000 7f9819ffb640 1 -- 172.21.15.27:0/3213367941 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f982008a600 con 0x7f98240d29d0 2024-03-06T07:26:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.665+0000 7f98288a7640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f980806b4d0 0x7f980806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.665+0000 7f98288a7640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f980806b4d0 0x7f980806d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9814036000 tx=0x7f981403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.669+0000 7f9819ffb640 1 -- 172.21.15.27:0/3213367941 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9820059440 con 0x7f98240d29d0 2024-03-06T07:26:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.850+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f982404f410 con 0x7f98240d29d0 2024-03-06T07:26:51.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.851+0000 7f9819ffb640 1 -- 172.21.15.27:0/3213367941 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9820058d40 con 0x7f98240d29d0 2024-03-06T07:26:51.853 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:26:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f980806b4d0 msgr2=0x7f980806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f980806b4d0 0x7f980806d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9814036000 tx=0x7f981403b040 comp rx=0 tx=0).stop 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 msgr2=0x7f9824157020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 0x7f9824157020 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f982000c7f0 tx=0x7f982000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 shutdown_connections 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f980806b4d0 0x7f980806d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f982415dd90 0x7f98241f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98240d4720 0x7f982415f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 --2- 172.21.15.27:0/3213367941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98240d29d0 0x7f9824157020 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 >> 172.21.15.27:0/3213367941 conn(0x7f9824071960 msgr2=0x7f98240b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 shutdown_connections 2024-03-06T07:26:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.853+0000 7f982b30c640 1 -- 172.21.15.27:0/3213367941 wait complete. 2024-03-06T07:26:51.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:26:51.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.972+0000 7fe69d04e640 1 Processor -- start 2024-03-06T07:26:51.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.972+0000 7fe69d04e640 1 -- start start 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69d04e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe69806df30 0x7fe6980ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69d04e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe6980aec40 0x7fe6980b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69d04e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe698071240 con 0x7fe698158520 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69d04e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6980af0f0 con 0x7fe69806df30 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69d04e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe698071b70 con 0x7fe6980aec40 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe69806df30 0x7fe6980ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe696d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe698158520 0x7fe69806d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38798/0 (socket says 172.21.15.27:38798) 2024-03-06T07:26:51.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe696d9d640 1 -- 172.21.15.27:0/1356881037 learned_addr learned my addr 172.21.15.27:0/1356881037 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69759e640 1 --2- 172.21.15.27:0/1356881037 >> v2:172.21.15.69:3300/0 conn(0x7fe6980aec40 0x7fe6980b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69659c640 1 -- 172.21.15.27:0/1356881037 >> v2:172.21.15.69:3300/0 conn(0x7fe6980aec40 msgr2=0x7fe6980b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69659c640 1 --2- 172.21.15.27:0/1356881037 >> v2:172.21.15.69:3300/0 conn(0x7fe6980aec40 0x7fe6980b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69659c640 1 -- 172.21.15.27:0/1356881037 >> v1:172.21.15.27:6789/0 conn(0x7fe698158520 legacy=0x7fe69806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe695d9b640 1 -- 172.21.15.27:0/1356881037 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984131541 0 0) 0x7fe698071240 con 0x7fe698158520 2024-03-06T07:26:51.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.973+0000 7fe69659c640 1 -- 172.21.15.27:0/1356881037 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe698071520 con 0x7fe69806df30 2024-03-06T07:26:51.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.974+0000 7fe69659c640 1 --2- 172.21.15.27:0/1356881037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69806df30 0x7fe6980ae700 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe6880097a0 tx=0x7fe68802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ce855ecff2e701 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.974+0000 7fe695d9b640 1 -- 172.21.15.27:0/1356881037 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688035020 con 0x7fe69806df30 2024-03-06T07:26:51.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.974+0000 7fe695d9b640 1 -- 172.21.15.27:0/1356881037 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe688031830 con 0x7fe69806df30 2024-03-06T07:26:51.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.974+0000 7fe695d9b640 1 -- 172.21.15.27:0/1356881037 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688031c00 con 0x7fe69806df30 2024-03-06T07:26:51.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.975+0000 7fe69d04e640 1 -- 172.21.15.27:0/1356881037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69806df30 msgr2=0x7fe6980ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.975+0000 7fe69d04e640 1 --2- 172.21.15.27:0/1356881037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69806df30 0x7fe6980ae700 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe6880097a0 tx=0x7fe68802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.975+0000 7fe69d04e640 1 -- 172.21.15.27:0/1356881037 shutdown_connections 2024-03-06T07:26:51.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.975+0000 7fe69d04e640 1 --2- 172.21.15.27:0/1356881037 >> v2:172.21.15.69:3300/0 conn(0x7fe6980aec40 0x7fe6980b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.975+0000 7fe69d04e640 1 --2- 172.21.15.27:0/1356881037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe69806df30 0x7fe6980ae700 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.975+0000 7fe69d04e640 1 -- 172.21.15.27:0/1356881037 >> 172.21.15.27:0/1356881037 conn(0x7fe698066d00 msgr2=0x7fe6980670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.975+0000 7fe69d04e640 1 -- 172.21.15.27:0/1356881037 shutdown_connections 2024-03-06T07:26:51.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.975+0000 7fe69d04e640 1 -- 172.21.15.27:0/1356881037 wait complete. 2024-03-06T07:26:51.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.977+0000 7fe69d04e640 1 Processor -- start 2024-03-06T07:26:51.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.977+0000 7fe69d04e640 1 -- start start 2024-03-06T07:26:51.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69d04e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69806df30 0x7fe698135520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69d04e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6980aec40 0x7fe69814a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69d04e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe698158520 0x7fe698159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69d04e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe698158900 con 0x7fe698158520 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69d04e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe698072000 con 0x7fe69806df30 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69d04e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe698158a80 con 0x7fe6980aec40 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe698158520 0x7fe698159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe698158520 0x7fe698159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33000/0 (socket says 172.21.15.27:33000) 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69759e640 1 -- 172.21.15.27:0/3359182595 learned_addr learned my addr 172.21.15.27:0/3359182595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe696d9d640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69806df30 0x7fe698135520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69659c640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6980aec40 0x7fe69814a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69759e640 1 -- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6980aec40 msgr2=0x7fe69814a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69759e640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6980aec40 0x7fe69814a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69759e640 1 -- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69806df30 msgr2=0x7fe698135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69759e640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69806df30 0x7fe698135520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69759e640 1 -- 172.21.15.27:0/3359182595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69814ac80 con 0x7fe698158520 2024-03-06T07:26:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe696d9d640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69806df30 0x7fe698135520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69759e640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe698158520 0x7fe698159ba0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe690007e70 tx=0x7fe69000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe68f7fe640 1 -- 172.21.15.27:0/3359182595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe690014620 con 0x7fe698158520 2024-03-06T07:26:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe68f7fe640 1 -- 172.21.15.27:0/3359182595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe690019070 con 0x7fe698158520 2024-03-06T07:26:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe68f7fe640 1 -- 172.21.15.27:0/3359182595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe690014a20 con 0x7fe698158520 2024-03-06T07:26:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.979+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69814af60 con 0x7fe698158520 2024-03-06T07:26:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.980+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6980ccd40 con 0x7fe698158520 2024-03-06T07:26:51.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.980+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe698067ff0 con 0x7fe698158520 2024-03-06T07:26:51.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.981+0000 7fe68f7fe640 1 -- 172.21.15.27:0/3359182595 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe690014bc0 con 0x7fe698158520 2024-03-06T07:26:51.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.981+0000 7fe68f7fe640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe67806b4d0 0x7fe67806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.981+0000 7fe68f7fe640 1 -- 172.21.15.27:0/3359182595 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe69008aab0 con 0x7fe698158520 2024-03-06T07:26:51.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.982+0000 7fe696d9d640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe67806b4d0 0x7fe67806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.982+0000 7fe696d9d640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe67806b4d0 0x7fe67806d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe680000f60 tx=0x7fe680009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:51.986+0000 7fe68f7fe640 1 -- 172.21.15.27:0/3359182595 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe690059980 con 0x7fe698158520 2024-03-06T07:26:52.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.191+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe6980b46f0 con 0x7fe698158520 2024-03-06T07:26:52.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.191+0000 7fe68f7fe640 1 -- 172.21.15.27:0/3359182595 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v72) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe690059280 con 0x7fe698158520 2024-03-06T07:26:52.194 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe67806b4d0 msgr2=0x7fe67806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe67806b4d0 0x7fe67806d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe680000f60 tx=0x7fe680009040 comp rx=0 tx=0).stop 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe698158520 msgr2=0x7fe698159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe698158520 0x7fe698159ba0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe690007e70 tx=0x7fe69000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 shutdown_connections 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe67806b4d0 0x7fe67806d990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe698158520 0x7fe698159ba0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe6980aec40 0x7fe69814a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 --2- 172.21.15.27:0/3359182595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe69806df30 0x7fe698135520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 >> 172.21.15.27:0/3359182595 conn(0x7fe698066d00 msgr2=0x7fe698070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 shutdown_connections 2024-03-06T07:26:52.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.193+0000 7fe69d04e640 1 -- 172.21.15.27:0/3359182595 wait complete. 2024-03-06T07:26:52.210 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:26:52.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:52.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:52.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[62838]: fuse finished with error 0 and tester_r 0 2024-03-06T07:26:52.242 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:26:52.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:52.242 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:26:52.333 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:26:52.333 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:52.350 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:52.350 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:26:52.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:52.351 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:26:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:52.407 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:26:52.407 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:26:52.407 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:52.422 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:52.422 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:26:52.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:52.422 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:26:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:52.477 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:26:52.477 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:26:52.477 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:52.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:52.494 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:26:52.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:52.494 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:26:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:52.550 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:26:52.550 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.654+0000 7f415e434640 1 Processor -- start 2024-03-06T07:26:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.654+0000 7f415e434640 1 -- start start 2024-03-06T07:26:52.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f415e434640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580d3400 0x7f41580b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f415e434640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580ce610 con 0x7f4158071540 2024-03-06T07:26:52.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f415e434640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580d2920 con 0x7f41580b2b50 2024-03-06T07:26:52.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f415e434640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41580d2f70 con 0x7f41580d3400 2024-03-06T07:26:52.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f4157fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4158071540 0x7f4158071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38802/0 (socket says 172.21.15.27:38802) 2024-03-06T07:26:52.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f4157fff640 1 -- 172.21.15.27:0/141001623 learned_addr learned my addr 172.21.15.27:0/141001623 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:52.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f41577fe640 1 --2- 172.21.15.27:0/141001623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580d3400 0x7f41580b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f4156ffd640 1 -- 172.21.15.27:0/141001623 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596478952 0 0) 0x7f41580ce610 con 0x7f4158071540 2024-03-06T07:26:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f4156ffd640 1 -- 172.21.15.27:0/141001623 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140003520 con 0x7f4158071540 2024-03-06T07:26:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f4156ffd640 1 -- 172.21.15.27:0/141001623 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018643199 0 0) 0x7f41580d2920 con 0x7f41580b2b50 2024-03-06T07:26:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f4156ffd640 1 -- 172.21.15.27:0/141001623 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41580ce610 con 0x7f41580b2b50 2024-03-06T07:26:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f4156ffd640 1 -- 172.21.15.27:0/141001623 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889131268 0 0) 0x7f4140003520 con 0x7f4158071540 2024-03-06T07:26:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f41577fe640 1 -- 172.21.15.27:0/141001623 >> v1:172.21.15.69:6789/0 conn(0x7f41580b2b50 legacy=0x7f41580b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f41577fe640 1 -- 172.21.15.27:0/141001623 >> v1:172.21.15.27:6789/0 conn(0x7f4158071540 legacy=0x7f4158071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.655+0000 7f41577fe640 1 -- 172.21.15.27:0/141001623 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580d2c00 con 0x7f41580d3400 2024-03-06T07:26:52.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.656+0000 7f41577fe640 1 --2- 172.21.15.27:0/141001623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580d3400 0x7f41580b30a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f414c0097d0 tx=0x7f414c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3664ac79bd5d045c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.656+0000 7f4156ffd640 1 -- 172.21.15.27:0/141001623 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c037070 con 0x7f41580d3400 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.656+0000 7f4156ffd640 1 -- 172.21.15.27:0/141001623 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414c030510 con 0x7f41580d3400 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.657+0000 7f4156ffd640 1 -- 172.21.15.27:0/141001623 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148001670 con 0x7f41580d3400 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.657+0000 7f415e434640 1 -- 172.21.15.27:0/141001623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580d3400 msgr2=0x7f41580b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.657+0000 7f415e434640 1 --2- 172.21.15.27:0/141001623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580d3400 0x7f41580b30a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f414c0097d0 tx=0x7f414c02b800 comp rx=0 tx=0).stop 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.657+0000 7f415e434640 1 -- 172.21.15.27:0/141001623 shutdown_connections 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.657+0000 7f415e434640 1 --2- 172.21.15.27:0/141001623 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41580d3400 0x7f41580b30a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.657+0000 7f415e434640 1 -- 172.21.15.27:0/141001623 >> 172.21.15.27:0/141001623 conn(0x7f41580d3ca0 msgr2=0x7f41580d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.657+0000 7f415e434640 1 -- 172.21.15.27:0/141001623 shutdown_connections 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.657+0000 7f415e434640 1 -- 172.21.15.27:0/141001623 wait complete. 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.658+0000 7f415e434640 1 Processor -- start 2024-03-06T07:26:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.658+0000 7f415e434640 1 -- start start 2024-03-06T07:26:52.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f415e434640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158071540 0x7f4158156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f415e434640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41580b2b50 0x7f4158154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f415e434640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580d3400 0x7f415815f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f415e434640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415815e370 con 0x7f41580d3400 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f415e434640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41580d4be0 con 0x7f41580b2b50 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f415e434640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41581572d0 con 0x7f4158071540 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f415c9d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580d3400 0x7f415815f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f415c9d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580d3400 0x7f415815f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33016/0 (socket says 172.21.15.27:33016) 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f415c9d1640 1 -- 172.21.15.27:0/522494963 learned_addr learned my addr 172.21.15.27:0/522494963 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f41577fe640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41580b2b50 0x7f4158154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.659+0000 7f4157fff640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158071540 0x7f4158156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f415c9d1640 1 -- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158071540 msgr2=0x7f4158156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f415c9d1640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158071540 0x7f4158156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f415c9d1640 1 -- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41580b2b50 msgr2=0x7f4158154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f415c9d1640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41580b2b50 0x7f4158154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f415c9d1640 1 -- 172.21.15.27:0/522494963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158155240 con 0x7f41580d3400 2024-03-06T07:26:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f415c9d1640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580d3400 0x7f415815f200 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f415000c790 tx=0x7f415000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f4154ff9640 1 -- 172.21.15.27:0/522494963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150003b70 con 0x7f41580d3400 2024-03-06T07:26:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f4154ff9640 1 -- 172.21.15.27:0/522494963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4150003d10 con 0x7f41580d3400 2024-03-06T07:26:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4158155cf0 con 0x7f41580d3400 2024-03-06T07:26:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f4154ff9640 1 -- 172.21.15.27:0/522494963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415000aa50 con 0x7f41580d3400 2024-03-06T07:26:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.660+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41581562b0 con 0x7f41580d3400 2024-03-06T07:26:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.661+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4158068700 con 0x7f41580d3400 2024-03-06T07:26:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.662+0000 7f4154ff9640 1 -- 172.21.15.27:0/522494963 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f415000abf0 con 0x7f41580d3400 2024-03-06T07:26:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.662+0000 7f4154ff9640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f414406b4d0 0x7f414406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.663+0000 7f4154ff9640 1 -- 172.21.15.27:0/522494963 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f415008a020 con 0x7f41580d3400 2024-03-06T07:26:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.663+0000 7f4157fff640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f414406b4d0 0x7f414406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.663+0000 7f4157fff640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f414406b4d0 0x7f414406d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f41480011e0 tx=0x7f4148001690 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.667+0000 7f4154ff9640 1 -- 172.21.15.27:0/522494963 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41500588a0 con 0x7f41580d3400 2024-03-06T07:26:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.890+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f415815d8d0 con 0x7f41580d3400 2024-03-06T07:26:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.890+0000 7f4154ff9640 1 -- 172.21.15.27:0/522494963 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f415005c940 con 0x7f41580d3400 2024-03-06T07:26:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:52.893 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f414406b4d0 msgr2=0x7f414406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f414406b4d0 0x7f414406d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f41480011e0 tx=0x7f4148001690 comp rx=0 tx=0).stop 2024-03-06T07:26:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580d3400 msgr2=0x7f415815f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580d3400 0x7f415815f200 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f415000c790 tx=0x7f415000cc50 comp rx=0 tx=0).stop 2024-03-06T07:26:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 shutdown_connections 2024-03-06T07:26:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f414406b4d0 0x7f414406d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41580d3400 0x7f415815f200 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41580b2b50 0x7f4158154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 --2- 172.21.15.27:0/522494963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4158071540 0x7f4158156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 >> 172.21.15.27:0/522494963 conn(0x7f41580d3ca0 msgr2=0x7f41580ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 shutdown_connections 2024-03-06T07:26:52.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:52.892+0000 7f415e434640 1 -- 172.21.15.27:0/522494963 wait complete. 2024-03-06T07:26:52.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:53.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.010+0000 7f3d76979640 1 Processor -- start 2024-03-06T07:26:53.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.010+0000 7f3d76979640 1 -- start start 2024-03-06T07:26:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.011+0000 7f3d76979640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d7015df00 0x7f3d700d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.011+0000 7f3d76979640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b9a30 0x7f3d700d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.011+0000 7f3d76979640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d700cea30 con 0x7f3d700b95a0 2024-03-06T07:26:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.011+0000 7f3d76979640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d700d2af0 con 0x7f3d7015df00 2024-03-06T07:26:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.011+0000 7f3d76979640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700cf3c0 con 0x7f3d700b9a30 2024-03-06T07:26:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.011+0000 7f3d6ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d7015df00 0x7f3d700d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.011+0000 7f3d6ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d7015df00 0x7f3d700d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33030/0 (socket says 172.21.15.27:33030) 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.011+0000 7f3d6f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d700b95a0 0x7f3d700b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38816/0 (socket says 172.21.15.27:38816) 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.011+0000 7f3d6ffff640 1 -- 172.21.15.27:0/1011770950 learned_addr learned my addr 172.21.15.27:0/1011770950 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.012+0000 7f3d74f16640 1 --2- 172.21.15.27:0/1011770950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b9a30 0x7f3d700d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.012+0000 7f3d6ffff640 1 -- 172.21.15.27:0/1011770950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b9a30 msgr2=0x7f3d700d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.012+0000 7f3d6ffff640 1 --2- 172.21.15.27:0/1011770950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b9a30 0x7f3d700d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.012+0000 7f3d6ffff640 1 -- 172.21.15.27:0/1011770950 >> v1:172.21.15.27:6789/0 conn(0x7f3d700b95a0 legacy=0x7f3d700b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.012+0000 7f3d6ffff640 1 -- 172.21.15.27:0/1011770950 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d700cf0e0 con 0x7f3d7015df00 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.012+0000 7f3d6effd640 1 -- 172.21.15.27:0/1011770950 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530326373 0 0) 0x7f3d700cea30 con 0x7f3d700b95a0 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d6ffff640 1 --2- 172.21.15.27:0/1011770950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015df00 0x7f3d700d25b0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009870 tx=0x7f3d6002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4529d91b6832e058 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d6effd640 1 -- 172.21.15.27:0/1011770950 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60034070 con 0x7f3d7015df00 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d6effd640 1 -- 172.21.15.27:0/1011770950 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d60030840 con 0x7f3d7015df00 2024-03-06T07:26:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d6effd640 1 -- 172.21.15.27:0/1011770950 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60030c10 con 0x7f3d7015df00 2024-03-06T07:26:53.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d76979640 1 -- 172.21.15.27:0/1011770950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015df00 msgr2=0x7f3d700d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d76979640 1 --2- 172.21.15.27:0/1011770950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015df00 0x7f3d700d25b0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009870 tx=0x7f3d6002b980 comp rx=0 tx=0).stop 2024-03-06T07:26:53.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d76979640 1 -- 172.21.15.27:0/1011770950 shutdown_connections 2024-03-06T07:26:53.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d76979640 1 --2- 172.21.15.27:0/1011770950 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b9a30 0x7f3d700d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d76979640 1 --2- 172.21.15.27:0/1011770950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015df00 0x7f3d700d25b0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d76979640 1 -- 172.21.15.27:0/1011770950 >> 172.21.15.27:0/1011770950 conn(0x7f3d70071960 msgr2=0x7f3d70071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d76979640 1 -- 172.21.15.27:0/1011770950 shutdown_connections 2024-03-06T07:26:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.013+0000 7f3d76979640 1 -- 172.21.15.27:0/1011770950 wait complete. 2024-03-06T07:26:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.015+0000 7f3d76979640 1 Processor -- start 2024-03-06T07:26:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.016+0000 7f3d76979640 1 -- start start 2024-03-06T07:26:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d76979640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b95a0 0x7f3d701570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d76979640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d7015df00 0x7f3d7015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d76979640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d701f49b0 0x7f3d70160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d76979640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700cec70 con 0x7f3d701f49b0 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d76979640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700ce750 con 0x7f3d7015df00 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d76979640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700cedf0 con 0x7f3d700b95a0 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d74f16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d701f49b0 0x7f3d70160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d74f16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d701f49b0 0x7f3d70160830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33036/0 (socket says 172.21.15.27:33036) 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d74f16640 1 -- 172.21.15.27:0/1565048651 learned_addr learned my addr 172.21.15.27:0/1565048651 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d6ffff640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b95a0 0x7f3d701570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d6f7fe640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d7015df00 0x7f3d7015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d74f16640 1 -- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b95a0 msgr2=0x7f3d701570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d74f16640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b95a0 0x7f3d701570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d74f16640 1 -- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d7015df00 msgr2=0x7f3d7015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d74f16640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d7015df00 0x7f3d7015eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d74f16640 1 -- 172.21.15.27:0/1565048651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7015e750 con 0x7f3d701f49b0 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.017+0000 7f3d6f7fe640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d7015df00 0x7f3d7015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.018+0000 7f3d74f16640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d701f49b0 0x7f3d70160830 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3d68007e40 tx=0x7f3d6800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.018+0000 7f3d6cff9640 1 -- 172.21.15.27:0/1565048651 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68014620 con 0x7f3d701f49b0 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.018+0000 7f3d6cff9640 1 -- 172.21.15.27:0/1565048651 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d68019070 con 0x7f3d701f49b0 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.018+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7015ea30 con 0x7f3d701f49b0 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.018+0000 7f3d6cff9640 1 -- 172.21.15.27:0/1565048651 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68014a20 con 0x7f3d701f49b0 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.018+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d700cc120 con 0x7f3d701f49b0 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.019+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d70067ff0 con 0x7f3d701f49b0 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.019+0000 7f3d6cff9640 1 -- 172.21.15.27:0/1565048651 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d68014bc0 con 0x7f3d701f49b0 2024-03-06T07:26:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.020+0000 7f3d6cff9640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d5c06b4d0 0x7f3d5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.020+0000 7f3d6cff9640 1 -- 172.21.15.27:0/1565048651 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3d6808ab80 con 0x7f3d701f49b0 2024-03-06T07:26:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.020+0000 7f3d6ffff640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d5c06b4d0 0x7f3d5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.021+0000 7f3d6ffff640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d5c06b4d0 0x7f3d5c06d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3d60036000 tx=0x7f3d60008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.024+0000 7f3d6cff9640 1 -- 172.21.15.27:0/1565048651 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d680599c0 con 0x7f3d701f49b0 2024-03-06T07:26:53.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.205+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d700d3720 con 0x7f3d701f49b0 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.206+0000 7f3d6cff9640 1 -- 172.21.15.27:0/1565048651 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f3d680592c0 con 0x7f3d701f49b0 2024-03-06T07:26:53.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:53.209 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:29.258129+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.914214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.608105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.2e","mappings":[{"from":6,"to":0}]},{"pgid":"11.3a","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.207+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d5c06b4d0 msgr2=0x7f3d5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.207+0000 7f3d76979640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d5c06b4d0 0x7f3d5c06d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3d60036000 tx=0x7f3d60008030 comp rx=0 tx=0).stop 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.207+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d701f49b0 msgr2=0x7f3d70160830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.207+0000 7f3d76979640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d701f49b0 0x7f3d70160830 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3d68007e40 tx=0x7f3d6800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.208+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 shutdown_connections 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.208+0000 7f3d76979640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d5c06b4d0 0x7f3d5c06d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.208+0000 7f3d76979640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d701f49b0 0x7f3d70160830 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.208+0000 7f3d76979640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d7015df00 0x7f3d7015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.208+0000 7f3d76979640 1 --2- 172.21.15.27:0/1565048651 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d700b95a0 0x7f3d701570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.208+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 >> 172.21.15.27:0/1565048651 conn(0x7f3d70071960 msgr2=0x7f3d700b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.208+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 shutdown_connections 2024-03-06T07:26:53.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.208+0000 7f3d76979640 1 -- 172.21.15.27:0/1565048651 wait complete. 2024-03-06T07:26:53.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:17.914214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:18.608105+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:53.222 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:53.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.331+0000 7f686ad3e640 1 Processor -- start 2024-03-06T07:26:53.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.331+0000 7f686ad3e640 1 -- start start 2024-03-06T07:26:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.332+0000 7f686ad3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640ceea0 0x7f68640d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.332+0000 7f686ad3e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68640b6d10 0x7f68640b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.332+0000 7f686ad3e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68640d3740 con 0x7f686415dd90 2024-03-06T07:26:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.332+0000 7f686ad3e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68640d3400 con 0x7f68640b6d10 2024-03-06T07:26:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.332+0000 7f686ad3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68640d3120 con 0x7f68640ceea0 2024-03-06T07:26:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.332+0000 7f68692db640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68640b6d10 0x7f68640b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.332+0000 7f68692db640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68640b6d10 0x7f68640b71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33040/0 (socket says 172.21.15.27:33040) 2024-03-06T07:26:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.332+0000 7f68692db640 1 -- 172.21.15.27:0/3881743328 learned_addr learned my addr 172.21.15.27:0/3881743328 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.332+0000 7f6863fff640 1 --2- 172.21.15.27:0/3881743328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640ceea0 0x7f68640d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.333+0000 7f68692db640 1 -- 172.21.15.27:0/3881743328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640ceea0 msgr2=0x7f68640d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.333+0000 7f68692db640 1 --2- 172.21.15.27:0/3881743328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640ceea0 0x7f68640d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.333+0000 7f68692db640 1 -- 172.21.15.27:0/3881743328 >> v1:172.21.15.69:6789/0 conn(0x7f686415dd90 legacy=0x7f68640d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.333+0000 7f68692db640 1 -- 172.21.15.27:0/3881743328 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68640d39e0 con 0x7f68640b6d10 2024-03-06T07:26:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.334+0000 7f68692db640 1 --2- 172.21.15.27:0/3881743328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640b6d10 0x7f68640b71a0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f685c009e80 tx=0x7f685c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba51ba9b2e204fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.334+0000 7f68637fe640 1 -- 172.21.15.27:0/3881743328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c02cb70 con 0x7f68640b6d10 2024-03-06T07:26:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.334+0000 7f68637fe640 1 -- 172.21.15.27:0/3881743328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c02cd10 con 0x7f68640b6d10 2024-03-06T07:26:53.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.334+0000 7f68637fe640 1 -- 172.21.15.27:0/3881743328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c034610 con 0x7f68640b6d10 2024-03-06T07:26:53.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.335+0000 7f686ad3e640 1 -- 172.21.15.27:0/3881743328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640b6d10 msgr2=0x7f68640b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.335+0000 7f686ad3e640 1 --2- 172.21.15.27:0/3881743328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640b6d10 0x7f68640b71a0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f685c009e80 tx=0x7f685c02bfd0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.335+0000 7f686ad3e640 1 -- 172.21.15.27:0/3881743328 shutdown_connections 2024-03-06T07:26:53.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.335+0000 7f686ad3e640 1 --2- 172.21.15.27:0/3881743328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640b6d10 0x7f68640b71a0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.335+0000 7f686ad3e640 1 --2- 172.21.15.27:0/3881743328 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640ceea0 0x7f68640d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.335+0000 7f686ad3e640 1 -- 172.21.15.27:0/3881743328 >> 172.21.15.27:0/3881743328 conn(0x7f6864071960 msgr2=0x7f6864071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.335+0000 7f686ad3e640 1 -- 172.21.15.27:0/3881743328 shutdown_connections 2024-03-06T07:26:53.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.335+0000 7f686ad3e640 1 -- 172.21.15.27:0/3881743328 wait complete. 2024-03-06T07:26:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.336+0000 7f686ad3e640 1 Processor -- start 2024-03-06T07:26:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.336+0000 7f686ad3e640 1 -- start start 2024-03-06T07:26:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f686ad3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640b6d10 0x7f6864160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f686ad3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ceea0 0x7f6864157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f686ad3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686415dd90 0x7f68641532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f686ad3e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68640d48f0 con 0x7f68640ceea0 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f686ad3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686415e170 con 0x7f686415dd90 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f686ad3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68640d4a70 con 0x7f68640b6d10 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6863fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ceea0 0x7f6864157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6863fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ceea0 0x7f6864157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33052/0 (socket says 172.21.15.27:33052) 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6863fff640 1 -- 172.21.15.27:0/2023306244 learned_addr learned my addr 172.21.15.27:0/2023306244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6868ada640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640b6d10 0x7f6864160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f68692db640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686415dd90 0x7f68641532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6863fff640 1 -- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640b6d10 msgr2=0x7f6864160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6863fff640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640b6d10 0x7f6864160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6863fff640 1 -- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686415dd90 msgr2=0x7f68641532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6863fff640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686415dd90 0x7f68641532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6863fff640 1 -- 172.21.15.27:0/2023306244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6864158250 con 0x7f68640ceea0 2024-03-06T07:26:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6868ada640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640b6d10 0x7f6864160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.337+0000 7f6863fff640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ceea0 0x7f6864157a00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f685000e9b0 tx=0x7f685000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.338+0000 7f68617fa640 1 -- 172.21.15.27:0/2023306244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850017070 con 0x7f68640ceea0 2024-03-06T07:26:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.338+0000 7f68617fa640 1 -- 172.21.15.27:0/2023306244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685000c9a0 con 0x7f68640ceea0 2024-03-06T07:26:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.338+0000 7f68617fa640 1 -- 172.21.15.27:0/2023306244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685000cd90 con 0x7f68640ceea0 2024-03-06T07:26:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.338+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6864158530 con 0x7f68640ceea0 2024-03-06T07:26:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.338+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68641f56f0 con 0x7f68640ceea0 2024-03-06T07:26:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.339+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6864158a50 con 0x7f68640ceea0 2024-03-06T07:26:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.339+0000 7f68617fa640 1 -- 172.21.15.27:0/2023306244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f685001a070 con 0x7f68640ceea0 2024-03-06T07:26:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.340+0000 7f68617fa640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f684806b4d0 0x7f684806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.340+0000 7f68617fa640 1 -- 172.21.15.27:0/2023306244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f685008a910 con 0x7f68640ceea0 2024-03-06T07:26:53.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.340+0000 7f6868ada640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f684806b4d0 0x7f684806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.340+0000 7f6868ada640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f684806b4d0 0x7f684806d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6858001660 tx=0x7f6858009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.343+0000 7f68617fa640 1 -- 172.21.15.27:0/2023306244 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6864158a50 con 0x7f68640ceea0 2024-03-06T07:26:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.523+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6864072a30 con 0x7f68640ceea0 2024-03-06T07:26:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.524+0000 7f68617fa640 1 -- 172.21.15.27:0/2023306244 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7f68500598c0 con 0x7f68640ceea0 2024-03-06T07:26:53.526 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:26:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f684806b4d0 msgr2=0x7f684806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f684806b4d0 0x7f684806d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6858001660 tx=0x7f6858009040 comp rx=0 tx=0).stop 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ceea0 msgr2=0x7f6864157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ceea0 0x7f6864157a00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f685000e9b0 tx=0x7f685000ee70 comp rx=0 tx=0).stop 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 shutdown_connections 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f684806b4d0 0x7f684806d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686415dd90 0x7f68641532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ceea0 0x7f6864157a00 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 --2- 172.21.15.27:0/2023306244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68640b6d10 0x7f6864160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 >> 172.21.15.27:0/2023306244 conn(0x7f6864071960 msgr2=0x7f68640b4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 shutdown_connections 2024-03-06T07:26:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.526+0000 7f686ad3e640 1 -- 172.21.15.27:0/2023306244 wait complete. 2024-03-06T07:26:53.540 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.648+0000 7fc93920e640 1 Processor -- start 2024-03-06T07:26:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.648+0000 7fc93920e640 1 -- start start 2024-03-06T07:26:53.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.648+0000 7fc93920e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9340b7420 0x7fc9340d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.648+0000 7fc93920e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9340d3740 con 0x7fc9340ceea0 2024-03-06T07:26:53.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.648+0000 7fc93920e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9340d3120 con 0x7fc93415dd90 2024-03-06T07:26:53.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.648+0000 7fc93920e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9340d3400 con 0x7fc9340b7420 2024-03-06T07:26:53.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.649+0000 7fc93359e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9340b7420 0x7fc9340d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.649+0000 7fc93259c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc9340ceea0 0x7fc9340b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44534/0 (socket says 172.21.15.27:44534) 2024-03-06T07:26:53.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.649+0000 7fc93359e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9340b7420 0x7fc9340d29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34452/0 (socket says 172.21.15.27:34452) 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.649+0000 7fc93259c640 1 -- 172.21.15.27:0/2661349453 learned_addr learned my addr 172.21.15.27:0/2661349453 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.649+0000 7fc93359e640 1 -- 172.21.15.27:0/2661349453 >> v1:172.21.15.69:6789/0 conn(0x7fc93415dd90 legacy=0x7fc9340d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.649+0000 7fc93359e640 1 -- 172.21.15.27:0/2661349453 >> v1:172.21.15.27:6789/0 conn(0x7fc9340ceea0 legacy=0x7fc9340b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.649+0000 7fc931d9b640 1 -- 172.21.15.27:0/2661349453 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795778707 0 0) 0x7fc9340d3740 con 0x7fc9340ceea0 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.649+0000 7fc93359e640 1 -- 172.21.15.27:0/2661349453 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9340d39e0 con 0x7fc9340b7420 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.650+0000 7fc93359e640 1 --2- 172.21.15.27:0/2661349453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9340b7420 0x7fc9340d29d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009d90 tx=0x7fc92c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be84a769f4067341 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.650+0000 7fc931d9b640 1 -- 172.21.15.27:0/2661349453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c039070 con 0x7fc9340b7420 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.650+0000 7fc931d9b640 1 -- 172.21.15.27:0/2661349453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92c02cc60 con 0x7fc9340b7420 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.650+0000 7fc931d9b640 1 -- 172.21.15.27:0/2661349453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c034660 con 0x7fc9340b7420 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.651+0000 7fc93920e640 1 -- 172.21.15.27:0/2661349453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9340b7420 msgr2=0x7fc9340d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.651+0000 7fc93920e640 1 --2- 172.21.15.27:0/2661349453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9340b7420 0x7fc9340d29d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009d90 tx=0x7fc92c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.651+0000 7fc93920e640 1 -- 172.21.15.27:0/2661349453 shutdown_connections 2024-03-06T07:26:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.651+0000 7fc93920e640 1 --2- 172.21.15.27:0/2661349453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9340b7420 0x7fc9340d29d0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.651+0000 7fc93920e640 1 -- 172.21.15.27:0/2661349453 >> 172.21.15.27:0/2661349453 conn(0x7fc934071960 msgr2=0x7fc934071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.651+0000 7fc93920e640 1 -- 172.21.15.27:0/2661349453 shutdown_connections 2024-03-06T07:26:53.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.651+0000 7fc93920e640 1 -- 172.21.15.27:0/2661349453 wait complete. 2024-03-06T07:26:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.653+0000 7fc93920e640 1 Processor -- start 2024-03-06T07:26:53.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.653+0000 7fc93920e640 1 -- start start 2024-03-06T07:26:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.654+0000 7fc93920e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9340ceea0 0x7fc9341f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.654+0000 7fc93920e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc93415dd90 0x7fc9340c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.654+0000 7fc93920e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc934158310 0x7fc93414cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.654+0000 7fc93920e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340d48f0 con 0x7fc93415dd90 2024-03-06T07:26:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.654+0000 7fc93920e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93415e170 con 0x7fc9340ceea0 2024-03-06T07:26:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.654+0000 7fc93920e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340d4a70 con 0x7fc934158310 2024-03-06T07:26:53.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.654+0000 7fc93259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc93415dd90 0x7fc9340c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc93415dd90 0x7fc9340c9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34460/0 (socket says 172.21.15.27:34460) 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93259c640 1 -- 172.21.15.27:0/3475494373 learned_addr learned my addr 172.21.15.27:0/3475494373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc932d9d640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9340ceea0 0x7fc9341f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93359e640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc934158310 0x7fc93414cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93259c640 1 -- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc934158310 msgr2=0x7fc93414cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93259c640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc934158310 0x7fc93414cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93259c640 1 -- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9340ceea0 msgr2=0x7fc9341f3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93259c640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9340ceea0 0x7fc9341f3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93259c640 1 -- 172.21.15.27:0/3475494373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93415f530 con 0x7fc93415dd90 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93259c640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc93415dd90 0x7fc9340c9bf0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c4d0 tx=0x7fc91c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc92b7fe640 1 -- 172.21.15.27:0/3475494373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c017070 con 0x7fc93415dd90 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc92b7fe640 1 -- 172.21.15.27:0/3475494373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c005d00 con 0x7fc93415dd90 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93415f810 con 0x7fc93415dd90 2024-03-06T07:26:53.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc92b7fe640 1 -- 172.21.15.27:0/3475494373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c0072b0 con 0x7fc93415dd90 2024-03-06T07:26:53.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.655+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93415fdd0 con 0x7fc93415dd90 2024-03-06T07:26:53.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.656+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc934072a30 con 0x7fc93415dd90 2024-03-06T07:26:53.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.657+0000 7fc92b7fe640 1 -- 172.21.15.27:0/3475494373 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc91c01a070 con 0x7fc93415dd90 2024-03-06T07:26:53.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.657+0000 7fc92b7fe640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc91406b4d0 0x7fc91406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.657+0000 7fc92b7fe640 1 -- 172.21.15.27:0/3475494373 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc91c0894d0 con 0x7fc93415dd90 2024-03-06T07:26:53.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.658+0000 7fc932d9d640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc91406b4d0 0x7fc91406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.658+0000 7fc932d9d640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc91406b4d0 0x7fc91406d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc924006fd0 tx=0x7fc924009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.662+0000 7fc92b7fe640 1 -- 172.21.15.27:0/3475494373 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc91c057d50 con 0x7fc93415dd90 2024-03-06T07:26:53.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.893+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc934069300 con 0x7fc93415dd90 2024-03-06T07:26:53.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.894+0000 7fc92b7fe640 1 -- 172.21.15.27:0/3475494373 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc91c05bdf0 con 0x7fc93415dd90 2024-03-06T07:26:53.896 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:26:53.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc91406b4d0 msgr2=0x7fc91406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc91406b4d0 0x7fc91406d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc924006fd0 tx=0x7fc924009060 comp rx=0 tx=0).stop 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc93415dd90 msgr2=0x7fc9340c9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc93415dd90 0x7fc9340c9bf0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c4d0 tx=0x7fc91c00c990 comp rx=0 tx=0).stop 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 shutdown_connections 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc91406b4d0 0x7fc91406d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc934158310 0x7fc93414cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc93415dd90 0x7fc9340c9bf0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 --2- 172.21.15.27:0/3475494373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9340ceea0 0x7fc9341f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 >> 172.21.15.27:0/3475494373 conn(0x7fc934071960 msgr2=0x7fc9340b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 shutdown_connections 2024-03-06T07:26:53.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:53.896+0000 7fc93920e640 1 -- 172.21.15.27:0/3475494373 wait complete. 2024-03-06T07:26:53.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.021+0000 7f39da2d4640 1 Processor -- start 2024-03-06T07:26:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.021+0000 7f39da2d4640 1 -- start start 2024-03-06T07:26:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.022+0000 7f39da2d4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39d415dd90 0x7f39d40ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.022+0000 7f39da2d4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39d40d29d0 0x7f39d40d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.022+0000 7f39da2d4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d40d3740 con 0x7f39d40d4720 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.022+0000 7f39da2d4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d40d3400 con 0x7f39d415dd90 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.022+0000 7f39da2d4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d40d3120 con 0x7f39d40d29d0 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.022+0000 7f39d3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39d415dd90 0x7f39d40ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.022+0000 7f39d3fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39d415dd90 0x7f39d40ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34472/0 (socket says 172.21.15.27:34472) 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.022+0000 7f39d3fff640 1 -- 172.21.15.27:0/3774962412 learned_addr learned my addr 172.21.15.27:0/3774962412 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.022+0000 7f39d37fe640 1 --2- 172.21.15.27:0/3774962412 >> v2:172.21.15.69:3300/0 conn(0x7f39d40d29d0 0x7f39d40d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.023+0000 7f39d3fff640 1 -- 172.21.15.27:0/3774962412 >> v2:172.21.15.69:3300/0 conn(0x7f39d40d29d0 msgr2=0x7f39d40d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.023+0000 7f39d3fff640 1 --2- 172.21.15.27:0/3774962412 >> v2:172.21.15.69:3300/0 conn(0x7f39d40d29d0 0x7f39d40d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.023+0000 7f39d3fff640 1 -- 172.21.15.27:0/3774962412 >> v1:172.21.15.69:6789/0 conn(0x7f39d40d4720 legacy=0x7f39d40b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.023+0000 7f39d3fff640 1 -- 172.21.15.27:0/3774962412 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d40d39e0 con 0x7f39d415dd90 2024-03-06T07:26:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.023+0000 7f39d37fe640 1 --2- 172.21.15.27:0/3774962412 >> v2:172.21.15.69:3300/0 conn(0x7f39d40d29d0 0x7f39d40d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.023+0000 7f39d3fff640 1 --2- 172.21.15.27:0/3774962412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d415dd90 0x7f39d40ceea0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f39d40724d0 tx=0x7f39bc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270e804ab5d7e065 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.024+0000 7f39d2ffd640 1 -- 172.21.15.27:0/3774962412 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc02f850 con 0x7f39d415dd90 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.024+0000 7f39d2ffd640 1 -- 172.21.15.27:0/3774962412 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc02f9f0 con 0x7f39d415dd90 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.024+0000 7f39d2ffd640 1 -- 172.21.15.27:0/3774962412 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc02fdc0 con 0x7f39d415dd90 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.024+0000 7f39da2d4640 1 -- 172.21.15.27:0/3774962412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d415dd90 msgr2=0x7f39d40ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.024+0000 7f39da2d4640 1 --2- 172.21.15.27:0/3774962412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d415dd90 0x7f39d40ceea0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f39d40724d0 tx=0x7f39bc02ba30 comp rx=0 tx=0).stop 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.024+0000 7f39da2d4640 1 -- 172.21.15.27:0/3774962412 shutdown_connections 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.024+0000 7f39da2d4640 1 --2- 172.21.15.27:0/3774962412 >> v2:172.21.15.69:3300/0 conn(0x7f39d40d29d0 0x7f39d40d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.024+0000 7f39da2d4640 1 --2- 172.21.15.27:0/3774962412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d415dd90 0x7f39d40ceea0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.024+0000 7f39da2d4640 1 -- 172.21.15.27:0/3774962412 >> 172.21.15.27:0/3774962412 conn(0x7f39d4071960 msgr2=0x7f39d4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.025+0000 7f39da2d4640 1 -- 172.21.15.27:0/3774962412 shutdown_connections 2024-03-06T07:26:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.025+0000 7f39da2d4640 1 -- 172.21.15.27:0/3774962412 wait complete. 2024-03-06T07:26:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.026+0000 7f39da2d4640 1 Processor -- start 2024-03-06T07:26:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.026+0000 7f39da2d4640 1 -- start start 2024-03-06T07:26:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.026+0000 7f39da2d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39d40d29d0 0x7f39d40c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.026+0000 7f39da2d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d40d4720 0x7f39d40cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39da2d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d415dd90 0x7f39d40cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39da2d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d40d2df0 con 0x7f39d40d4720 2024-03-06T07:26:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39da2d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d415e170 con 0x7f39d415dd90 2024-03-06T07:26:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39da2d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d40b7420 con 0x7f39d40d29d0 2024-03-06T07:26:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d40d4720 0x7f39d40cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d40d4720 0x7f39d40cccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34484/0 (socket says 172.21.15.27:34484) 2024-03-06T07:26:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d37fe640 1 -- 172.21.15.27:0/556885316 learned_addr learned my addr 172.21.15.27:0/556885316 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d8871640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d415dd90 0x7f39d40cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d3fff640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39d40d29d0 0x7f39d40c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d37fe640 1 -- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39d40d29d0 msgr2=0x7f39d40c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d37fe640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39d40d29d0 0x7f39d40c6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d37fe640 1 -- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d415dd90 msgr2=0x7f39d40cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d37fe640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d415dd90 0x7f39d40cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d37fe640 1 -- 172.21.15.27:0/556885316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d40cd790 con 0x7f39d40d4720 2024-03-06T07:26:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d37fe640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d40d4720 0x7f39d40cccb0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f39c400e7e0 tx=0x7f39c400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d0ff9640 1 -- 172.21.15.27:0/556885316 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c400f040 con 0x7f39d40d4720 2024-03-06T07:26:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.027+0000 7f39d0ff9640 1 -- 172.21.15.27:0/556885316 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c4004030 con 0x7f39d40d4720 2024-03-06T07:26:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.028+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d40cda70 con 0x7f39d40d4720 2024-03-06T07:26:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.028+0000 7f39d0ff9640 1 -- 172.21.15.27:0/556885316 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4004400 con 0x7f39d40d4720 2024-03-06T07:26:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.028+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d40c8a50 con 0x7f39d40d4720 2024-03-06T07:26:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.029+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d40c8860 con 0x7f39d40d4720 2024-03-06T07:26:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.029+0000 7f39d0ff9640 1 -- 172.21.15.27:0/556885316 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39c400c6d0 con 0x7f39d40d4720 2024-03-06T07:26:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.030+0000 7f39d0ff9640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b500 0x7f39b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.030+0000 7f39d0ff9640 1 -- 172.21.15.27:0/556885316 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f39c4089f60 con 0x7f39d40d4720 2024-03-06T07:26:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.030+0000 7f39d3fff640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b500 0x7f39b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.030+0000 7f39d3fff640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b500 0x7f39b406d9c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f39bc036000 tx=0x7f39bc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.034+0000 7f39d0ff9640 1 -- 172.21.15.27:0/556885316 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39c4017020 con 0x7f39d40d4720 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.213+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39d4072a30 con 0x7f39d40d4720 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.213+0000 7f39d0ff9640 1 -- 172.21.15.27:0/556885316 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) v1 ==== 100+0+11 (secure 0 0 0) 0x7f39c4058750 con 0x7f39d40d4720 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:26:54.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b500 msgr2=0x7f39b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b500 0x7f39b406d9c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f39bc036000 tx=0x7f39bc008030 comp rx=0 tx=0).stop 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d40d4720 msgr2=0x7f39d40cccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d40d4720 0x7f39d40cccb0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f39c400e7e0 tx=0x7f39c400eca0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 shutdown_connections 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f39b406b500 0x7f39b406d9c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d415dd90 0x7f39d40cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39d40d4720 0x7f39d40cccb0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 --2- 172.21.15.27:0/556885316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39d40d29d0 0x7f39d40c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 >> 172.21.15.27:0/556885316 conn(0x7f39d4071960 msgr2=0x7f39d40b77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 shutdown_connections 2024-03-06T07:26:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.216+0000 7f39da2d4640 1 -- 172.21.15.27:0/556885316 wait complete. 2024-03-06T07:26:54.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.335+0000 7f1ac26c3640 1 Processor -- start 2024-03-06T07:26:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.335+0000 7f1ac26c3640 1 -- start start 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1ac26c3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1abc071540 0x7f1abc0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1ac26c3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0b2b50 0x7f1abc0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1ac26c3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1abc0ce610 con 0x7f1abc0d3400 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1ac26c3640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc0d2f70 con 0x7f1abc071540 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1ac26c3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0d4be0 con 0x7f1abc0b2b50 2024-03-06T07:26:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1abbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1abc071540 0x7f1abc0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1abbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1abc071540 0x7f1abc0b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34488/0 (socket says 172.21.15.27:34488) 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1abbfff640 1 -- 172.21.15.27:0/1404573088 learned_addr learned my addr 172.21.15.27:0/1404573088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1ac0c60640 1 --2- 172.21.15.27:0/1404573088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0b2b50 0x7f1abc0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1abbfff640 1 -- 172.21.15.27:0/1404573088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0b2b50 msgr2=0x7f1abc0b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1abbfff640 1 --2- 172.21.15.27:0/1404573088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0b2b50 0x7f1abc0b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1abbfff640 1 -- 172.21.15.27:0/1404573088 >> v1:172.21.15.69:6789/0 conn(0x7f1abc0d3400 legacy=0x7f1abc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.336+0000 7f1abbfff640 1 -- 172.21.15.27:0/1404573088 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc0d2920 con 0x7f1abc071540 2024-03-06T07:26:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.337+0000 7f1abbfff640 1 --2- 172.21.15.27:0/1404573088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc071540 0x7f1abc0b30a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009870 tx=0x7f1aac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f07be7f4ede947 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.337+0000 7f1abaffd640 1 -- 172.21.15.27:0/1404573088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac034070 con 0x7f1abc071540 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.337+0000 7f1abaffd640 1 -- 172.21.15.27:0/1404573088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac030840 con 0x7f1abc071540 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.337+0000 7f1abaffd640 1 -- 172.21.15.27:0/1404573088 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac030c10 con 0x7f1abc071540 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.338+0000 7f1ac26c3640 1 -- 172.21.15.27:0/1404573088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc071540 msgr2=0x7f1abc0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.338+0000 7f1ac26c3640 1 --2- 172.21.15.27:0/1404573088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc071540 0x7f1abc0b30a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009870 tx=0x7f1aac02b980 comp rx=0 tx=0).stop 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.338+0000 7f1ac26c3640 1 -- 172.21.15.27:0/1404573088 shutdown_connections 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.338+0000 7f1ac26c3640 1 --2- 172.21.15.27:0/1404573088 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0b2b50 0x7f1abc0b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.338+0000 7f1ac26c3640 1 --2- 172.21.15.27:0/1404573088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc071540 0x7f1abc0b30a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.338+0000 7f1ac26c3640 1 -- 172.21.15.27:0/1404573088 >> 172.21.15.27:0/1404573088 conn(0x7f1abc0d3ca0 msgr2=0x7f1abc0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.338+0000 7f1ac26c3640 1 -- 172.21.15.27:0/1404573088 shutdown_connections 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.338+0000 7f1ac26c3640 1 -- 172.21.15.27:0/1404573088 wait complete. 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.339+0000 7f1ac26c3640 1 Processor -- start 2024-03-06T07:26:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.339+0000 7f1ac26c3640 1 -- start start 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1ac26c3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1abc071540 0x7f1abc14d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1ac26c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc0b2b50 0x7f1abc0b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1ac26c3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0d3400 0x7f1abc0b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1ac26c3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc15e370 con 0x7f1abc0b2b50 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1ac26c3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0ce8f0 con 0x7f1abc071540 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1ac26c3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0b3b50 con 0x7f1abc0d3400 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1abb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc0b2b50 0x7f1abc0b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1abb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc0b2b50 0x7f1abc0b4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34496/0 (socket says 172.21.15.27:34496) 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1abbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1abc071540 0x7f1abc14d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1abb7fe640 1 -- 172.21.15.27:0/2991561154 learned_addr learned my addr 172.21.15.27:0/2991561154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.340+0000 7f1ac0c60640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0d3400 0x7f1abc0b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1abb7fe640 1 -- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0d3400 msgr2=0x7f1abc0b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1abb7fe640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0d3400 0x7f1abc0b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1abb7fe640 1 -- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1abc071540 msgr2=0x7f1abc14d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1abb7fe640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1abc071540 0x7f1abc14d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1abb7fe640 1 -- 172.21.15.27:0/2991561154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc0b3e30 con 0x7f1abc0b2b50 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1abb7fe640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc0b2b50 0x7f1abc0b4bf0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c440 tx=0x7f1ab000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1ab8ff9640 1 -- 172.21.15.27:0/2991561154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00052d0 con 0x7f1abc0b2b50 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1ab8ff9640 1 -- 172.21.15.27:0/2991561154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab0005470 con 0x7f1abc0b2b50 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1abc0b4110 con 0x7f1abc0b2b50 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1ab8ff9640 1 -- 172.21.15.27:0/2991561154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0007260 con 0x7f1abc0b2b50 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.341+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1abc0b46d0 con 0x7f1abc0b2b50 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.342+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1abc0b48d0 con 0x7f1abc0b2b50 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.342+0000 7f1ab8ff9640 1 -- 172.21.15.27:0/2991561154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ab0007400 con 0x7f1abc0b2b50 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.343+0000 7f1ab8ff9640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1aa806b500 0x7f1aa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.343+0000 7f1ab8ff9640 1 -- 172.21.15.27:0/2991561154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1ab0089500 con 0x7f1abc0b2b50 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.343+0000 7f1abbfff640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1aa806b500 0x7f1aa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.344+0000 7f1abbfff640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1aa806b500 0x7f1aa806d9c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1aac036000 tx=0x7f1aac008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.347+0000 7f1ab8ff9640 1 -- 172.21.15.27:0/2991561154 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1abc0b48d0 con 0x7f1abc0b2b50 2024-03-06T07:26:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.566+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1abc068700 con 0x7f1abc0b2b50 2024-03-06T07:26:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.566+0000 7f1ab8ff9640 1 -- 172.21.15.27:0/2991561154 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f1ab005e0d0 con 0x7f1abc0b2b50 2024-03-06T07:26:54.569 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:54.569 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1aa806b500 msgr2=0x7f1aa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1aa806b500 0x7f1aa806d9c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1aac036000 tx=0x7f1aac008030 comp rx=0 tx=0).stop 2024-03-06T07:26:54.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc0b2b50 msgr2=0x7f1abc0b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc0b2b50 0x7f1abc0b4bf0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c440 tx=0x7f1ab000c900 comp rx=0 tx=0).stop 2024-03-06T07:26:54.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 shutdown_connections 2024-03-06T07:26:54.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1aa806b500 0x7f1aa806d9c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1abc0d3400 0x7f1abc0b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1abc0b2b50 0x7f1abc0b4bf0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 --2- 172.21.15.27:0/2991561154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1abc071540 0x7f1abc14d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.568+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 >> 172.21.15.27:0/2991561154 conn(0x7f1abc0d3ca0 msgr2=0x7f1abc15f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.569+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 shutdown_connections 2024-03-06T07:26:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.569+0000 7f1ac26c3640 1 -- 172.21.15.27:0/2991561154 wait complete. 2024-03-06T07:26:54.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.687+0000 7f4adf447640 1 Processor -- start 2024-03-06T07:26:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.687+0000 7f4adf447640 1 -- start start 2024-03-06T07:26:54.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4adf447640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4ad815df00 0x7f4ad80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4adf447640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ad80b95a0 0x7f4ad80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4adf447640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad80cea30 con 0x7f4ad80b9a30 2024-03-06T07:26:54.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4adf447640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad80d2af0 con 0x7f4ad80b95a0 2024-03-06T07:26:54.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4adf447640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad80cf3c0 con 0x7f4ad815df00 2024-03-06T07:26:54.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4adc9e2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ad80b95a0 0x7f4ad80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4add9e4640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4ad80b9a30 0x7f4ad80d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44538/0 (socket says 172.21.15.27:44538) 2024-03-06T07:26:54.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4adc9e2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ad80b95a0 0x7f4ad80b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34506/0 (socket says 172.21.15.27:34506) 2024-03-06T07:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4add9e4640 1 -- 172.21.15.27:0/122628693 learned_addr learned my addr 172.21.15.27:0/122628693 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.688+0000 7f4add1e3640 1 --2- 172.21.15.27:0/122628693 >> v2:172.21.15.69:3300/0 conn(0x7f4ad815df00 0x7f4ad80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.689+0000 7f4adc9e2640 1 -- 172.21.15.27:0/122628693 >> v2:172.21.15.69:3300/0 conn(0x7f4ad815df00 msgr2=0x7f4ad80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.689+0000 7f4adc9e2640 1 --2- 172.21.15.27:0/122628693 >> v2:172.21.15.69:3300/0 conn(0x7f4ad815df00 0x7f4ad80d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.689+0000 7f4adc9e2640 1 -- 172.21.15.27:0/122628693 >> v1:172.21.15.27:6789/0 conn(0x7f4ad80b9a30 legacy=0x7f4ad80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.689+0000 7f4adc9e2640 1 -- 172.21.15.27:0/122628693 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad80cf0e0 con 0x7f4ad80b95a0 2024-03-06T07:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.689+0000 7f4acffff640 1 -- 172.21.15.27:0/122628693 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668922293 0 0) 0x7f4ad80cea30 con 0x7f4ad80b9a30 2024-03-06T07:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.689+0000 7f4add1e3640 1 --2- 172.21.15.27:0/122628693 >> v2:172.21.15.69:3300/0 conn(0x7f4ad815df00 0x7f4ad80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.689+0000 7f4adc9e2640 1 --2- 172.21.15.27:0/122628693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80b95a0 0x7f4ad80b5700 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80d11b0 tx=0x7f4ad402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28063ff2c4883ff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.689+0000 7f4acffff640 1 -- 172.21.15.27:0/122628693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4035020 con 0x7f4ad80b95a0 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.689+0000 7f4acffff640 1 -- 172.21.15.27:0/122628693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad4031830 con 0x7f4ad80b95a0 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.690+0000 7f4acffff640 1 -- 172.21.15.27:0/122628693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4031c00 con 0x7f4ad80b95a0 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.690+0000 7f4adf447640 1 -- 172.21.15.27:0/122628693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80b95a0 msgr2=0x7f4ad80b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.690+0000 7f4adf447640 1 --2- 172.21.15.27:0/122628693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80b95a0 0x7f4ad80b5700 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80d11b0 tx=0x7f4ad402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.690+0000 7f4adf447640 1 -- 172.21.15.27:0/122628693 shutdown_connections 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.690+0000 7f4adf447640 1 --2- 172.21.15.27:0/122628693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80b95a0 0x7f4ad80b5700 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.690+0000 7f4adf447640 1 --2- 172.21.15.27:0/122628693 >> v2:172.21.15.69:3300/0 conn(0x7f4ad815df00 0x7f4ad80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.690+0000 7f4adf447640 1 -- 172.21.15.27:0/122628693 >> 172.21.15.27:0/122628693 conn(0x7f4ad8071960 msgr2=0x7f4ad8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.690+0000 7f4adf447640 1 -- 172.21.15.27:0/122628693 shutdown_connections 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.690+0000 7f4adf447640 1 -- 172.21.15.27:0/122628693 wait complete. 2024-03-06T07:26:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.691+0000 7f4adf447640 1 Processor -- start 2024-03-06T07:26:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.691+0000 7f4adf447640 1 -- start start 2024-03-06T07:26:54.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4adf447640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ad80b95a0 0x7f4ad80b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4adf447640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ad815df00 0x7f4ad80ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4adf447640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80ca940 0x7f4ad80c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4adf447640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80cec70 con 0x7f4ad80ca940 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4adf447640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80ce750 con 0x7f4ad80b95a0 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4adf447640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80cedf0 con 0x7f4ad815df00 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4add9e4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80ca940 0x7f4ad80c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4add9e4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80ca940 0x7f4ad80c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34514/0 (socket says 172.21.15.27:34514) 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4add9e4640 1 -- 172.21.15.27:0/1231388972 learned_addr learned my addr 172.21.15.27:0/1231388972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4add1e3640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ad80b95a0 0x7f4ad80b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4adc9e2640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ad815df00 0x7f4ad80ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4add9e4640 1 -- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ad815df00 msgr2=0x7f4ad80ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4add9e4640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ad815df00 0x7f4ad80ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4add9e4640 1 -- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ad80b95a0 msgr2=0x7f4ad80b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4add9e4640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ad80b95a0 0x7f4ad80b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.692+0000 7f4add9e4640 1 -- 172.21.15.27:0/1231388972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad814a110 con 0x7f4ad80ca940 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.693+0000 7f4add1e3640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ad80b95a0 0x7f4ad80b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.693+0000 7f4add9e4640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80ca940 0x7f4ad80c77b0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000c7d0 tx=0x7f4ad000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.693+0000 7f4acdffb640 1 -- 172.21.15.27:0/1231388972 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad0018070 con 0x7f4ad80ca940 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.693+0000 7f4acdffb640 1 -- 172.21.15.27:0/1231388972 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad000a640 con 0x7f4ad80ca940 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.693+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad80b4390 con 0x7f4ad80ca940 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.693+0000 7f4acdffb640 1 -- 172.21.15.27:0/1231388972 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad000aa70 con 0x7f4ad80ca940 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.693+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad80b4950 con 0x7f4ad80ca940 2024-03-06T07:26:54.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.694+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad80b5970 con 0x7f4ad80ca940 2024-03-06T07:26:54.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.695+0000 7f4acdffb640 1 -- 172.21.15.27:0/1231388972 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ad001b070 con 0x7f4ad80ca940 2024-03-06T07:26:54.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.696+0000 7f4acdffb640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4abc06b4d0 0x7f4abc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.696+0000 7f4acdffb640 1 -- 172.21.15.27:0/1231388972 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4ad0089820 con 0x7f4ad80ca940 2024-03-06T07:26:54.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.696+0000 7f4add1e3640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4abc06b4d0 0x7f4abc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.697+0000 7f4add1e3640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4abc06b4d0 0x7f4abc06d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006fd0 tx=0x7f4ac8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.700+0000 7f4acdffb640 1 -- 172.21.15.27:0/1231388972 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ad0058660 con 0x7f4ad80ca940 2024-03-06T07:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.883+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ad815d3b0 con 0x7f4ad80ca940 2024-03-06T07:26:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.884+0000 7f4acdffb640 1 -- 172.21.15.27:0/1231388972 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f4ad0057f60 con 0x7f4ad80ca940 2024-03-06T07:26:54.887 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:54.887 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:29.258129+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.914214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.608105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.2e","mappings":[{"from":6,"to":0}]},{"pgid":"11.3a","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4abc06b4d0 msgr2=0x7f4abc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4abc06b4d0 0x7f4abc06d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006fd0 tx=0x7f4ac8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80ca940 msgr2=0x7f4ad80c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80ca940 0x7f4ad80c77b0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000c7d0 tx=0x7f4ad000cc90 comp rx=0 tx=0).stop 2024-03-06T07:26:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 shutdown_connections 2024-03-06T07:26:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4abc06b4d0 0x7f4abc06d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad80ca940 0x7f4ad80c77b0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ad815df00 0x7f4ad80ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 --2- 172.21.15.27:0/1231388972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ad80b95a0 0x7f4ad80b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 >> 172.21.15.27:0/1231388972 conn(0x7f4ad8071960 msgr2=0x7f4ad815c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.886+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 shutdown_connections 2024-03-06T07:26:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:54.887+0000 7f4adf447640 1 -- 172.21.15.27:0/1231388972 wait complete. 2024-03-06T07:26:54.900 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:54.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:55.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.004+0000 7f0b172eb640 1 Processor -- start 2024-03-06T07:26:55.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.005+0000 7f0b172eb640 1 -- start start 2024-03-06T07:26:55.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.005+0000 7f0b172eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b10071540 0x7f0b100b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.005+0000 7f0b172eb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b100ce610 con 0x7f0b100d3400 2024-03-06T07:26:55.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.005+0000 7f0b172eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b100d2920 con 0x7f0b100b2b50 2024-03-06T07:26:55.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.005+0000 7f0b172eb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b100d2f70 con 0x7f0b10071540 2024-03-06T07:26:55.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.006+0000 7f0b15087640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b10071540 0x7f0b100b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.006+0000 7f0b15087640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b10071540 0x7f0b100b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34520/0 (socket says 172.21.15.27:34520) 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.006+0000 7f0b14886640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0b100d3400 0x7f0b10071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44552/0 (socket says 172.21.15.27:44552) 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.006+0000 7f0b15087640 1 -- 172.21.15.27:0/1508833289 learned_addr learned my addr 172.21.15.27:0/1508833289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.006+0000 7f0b15087640 1 -- 172.21.15.27:0/1508833289 >> v1:172.21.15.69:6789/0 conn(0x7f0b100b2b50 legacy=0x7f0b100b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.006+0000 7f0b15087640 1 -- 172.21.15.27:0/1508833289 >> v1:172.21.15.27:6789/0 conn(0x7f0b100d3400 legacy=0x7f0b10071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.006+0000 7f0b15087640 1 -- 172.21.15.27:0/1508833289 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b100d2c00 con 0x7f0b10071540 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.006+0000 7f0b07fff640 1 -- 172.21.15.27:0/1508833289 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090449246 0 0) 0x7f0b100ce610 con 0x7f0b100d3400 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.007+0000 7f0b15087640 1 --2- 172.21.15.27:0/1508833289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 0x7f0b100b30a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009870 tx=0x7f0b0c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a29e1be1ec7e90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.007+0000 7f0b07fff640 1 -- 172.21.15.27:0/1508833289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c02f840 con 0x7f0b10071540 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.007+0000 7f0b07fff640 1 -- 172.21.15.27:0/1508833289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c02f9e0 con 0x7f0b10071540 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.007+0000 7f0b07fff640 1 -- 172.21.15.27:0/1508833289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c02fdb0 con 0x7f0b10071540 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.008+0000 7f0b172eb640 1 -- 172.21.15.27:0/1508833289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 msgr2=0x7f0b100b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.008+0000 7f0b172eb640 1 --2- 172.21.15.27:0/1508833289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 0x7f0b100b30a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009870 tx=0x7f0b0c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:26:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.008+0000 7f0b172eb640 1 -- 172.21.15.27:0/1508833289 shutdown_connections 2024-03-06T07:26:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.008+0000 7f0b172eb640 1 --2- 172.21.15.27:0/1508833289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 0x7f0b100b30a0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.008+0000 7f0b172eb640 1 -- 172.21.15.27:0/1508833289 >> 172.21.15.27:0/1508833289 conn(0x7f0b100d3ca0 msgr2=0x7f0b100d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.008+0000 7f0b172eb640 1 -- 172.21.15.27:0/1508833289 shutdown_connections 2024-03-06T07:26:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.008+0000 7f0b172eb640 1 -- 172.21.15.27:0/1508833289 wait complete. 2024-03-06T07:26:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.009+0000 7f0b172eb640 1 Processor -- start 2024-03-06T07:26:55.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.009+0000 7f0b172eb640 1 -- start start 2024-03-06T07:26:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b172eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 0x7f0b100cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b172eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b100b2b50 0x7f0b1015aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b172eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b100d3400 0x7f0b1014cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b172eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1015e370 con 0x7f0b10071540 2024-03-06T07:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b172eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b100d4be0 con 0x7f0b100b2b50 2024-03-06T07:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b172eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b101f3ca0 con 0x7f0b100d3400 2024-03-06T07:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b15087640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 0x7f0b100cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b15087640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 0x7f0b100cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34526/0 (socket says 172.21.15.27:34526) 2024-03-06T07:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b15087640 1 -- 172.21.15.27:0/443714625 learned_addr learned my addr 172.21.15.27:0/443714625 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b15087640 1 -- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b100d3400 msgr2=0x7f0b1014cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b15087640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b100d3400 0x7f0b1014cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b15087640 1 -- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b100b2b50 msgr2=0x7f0b1015aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b14886640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b100b2b50 0x7f0b1015aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b15087640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b100b2b50 0x7f0b1015aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b15087640 1 -- 172.21.15.27:0/443714625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b10159a20 con 0x7f0b10071540 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b14886640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b100b2b50 0x7f0b1015aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b15087640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 0x7f0b100cd6a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0090b0 tx=0x7f0b0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b05ffb640 1 -- 172.21.15.27:0/443714625 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c047070 con 0x7f0b10071540 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b05ffb640 1 -- 172.21.15.27:0/443714625 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c037cc0 con 0x7f0b10071540 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b05ffb640 1 -- 172.21.15.27:0/443714625 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c004270 con 0x7f0b10071540 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.010+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b10159d00 con 0x7f0b10071540 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.011+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b100cad70 con 0x7f0b10071540 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.012+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1015deb0 con 0x7f0b10071540 2024-03-06T07:26:55.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.012+0000 7f0b05ffb640 1 -- 172.21.15.27:0/443714625 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b0c04a070 con 0x7f0b10071540 2024-03-06T07:26:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.013+0000 7f0b05ffb640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0af406b4d0 0x7f0af406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.013+0000 7f0b05ffb640 1 -- 172.21.15.27:0/443714625 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0b0c0ac370 con 0x7f0b10071540 2024-03-06T07:26:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.013+0000 7f0b14886640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0af406b4d0 0x7f0af406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.014+0000 7f0b14886640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0af406b4d0 0x7f0af406d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0b100b8180 tx=0x7f0b00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.017+0000 7f0b05ffb640 1 -- 172.21.15.27:0/443714625 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b0c033030 con 0x7f0b10071540 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.237+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b10068700 con 0x7f0b10071540 2024-03-06T07:26:55.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.237+0000 7f0b05ffb640 1 -- 172.21.15.27:0/443714625 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f0b0c07b240 con 0x7f0b10071540 2024-03-06T07:26:55.240 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:55.240 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0af406b4d0 msgr2=0x7f0af406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0af406b4d0 0x7f0af406d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0b100b8180 tx=0x7f0b00009040 comp rx=0 tx=0).stop 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 msgr2=0x7f0b100cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 0x7f0b100cd6a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0090b0 tx=0x7f0b0c004730 comp rx=0 tx=0).stop 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 shutdown_connections 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0af406b4d0 0x7f0af406d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b100d3400 0x7f0b1014cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b100b2b50 0x7f0b1015aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 --2- 172.21.15.27:0/443714625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b10071540 0x7f0b100cd6a0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 >> 172.21.15.27:0/443714625 conn(0x7f0b100d3ca0 msgr2=0x7f0b10160830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 shutdown_connections 2024-03-06T07:26:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.239+0000 7f0b172eb640 1 -- 172.21.15.27:0/443714625 wait complete. 2024-03-06T07:26:55.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.358+0000 7ff7d57b6640 1 Processor -- start 2024-03-06T07:26:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.358+0000 7ff7d57b6640 1 -- start start 2024-03-06T07:26:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.359+0000 7ff7d57b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7d00b2b50 0x7ff7d00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.359+0000 7ff7d57b6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d00ce610 con 0x7ff7d00d3400 2024-03-06T07:26:55.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.359+0000 7ff7d57b6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d00d2920 con 0x7ff7d0071540 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.359+0000 7ff7d57b6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00d2f70 con 0x7ff7d00b2b50 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.359+0000 7ff7ce7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff7d00d3400 0x7ff7d00b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44564/0 (socket says 172.21.15.27:44564) 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.359+0000 7ff7ce7fc640 1 -- 172.21.15.27:0/3958643000 learned_addr learned my addr 172.21.15.27:0/3958643000 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.359+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665122984 0 0) 0x7ff7d00ce610 con 0x7ff7d00d3400 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.359+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b8003520 con 0x7ff7d00d3400 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.359+0000 7ff7cf7fe640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7d00b2b50 0x7ff7d00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2710260576 0 0) 0x7ff7b8003520 con 0x7ff7d00d3400 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7d00b2b50 msgr2=0x7ff7d00b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7d00b2b50 0x7ff7d00b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 >> v1:172.21.15.69:6789/0 conn(0x7ff7d0071540 legacy=0x7ff7d0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d00d2c00 con 0x7ff7d00d3400 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7ff7c4001670 con 0x7ff7d00d3400 2024-03-06T07:26:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 >> v1:172.21.15.27:6789/0 conn(0x7ff7d00d3400 legacy=0x7ff7d00b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b80039e0 0x7ff7b8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b8004390 0x7ff7b8004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7d00b2b50 0x7ff7b80050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00d2f70 con 0x7ff7b80039e0 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b800b9f0 con 0x7ff7b8004390 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b800bbf0 con 0x7ff7d00b2b50 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7ceffd640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b80039e0 0x7ff7b8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7ce7fc640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b8004390 0x7ff7b8004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cf7fe640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7d00b2b50 0x7ff7b80050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7ceffd640 1 -- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7d00b2b50 msgr2=0x7ff7b80050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7ceffd640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7d00b2b50 0x7ff7b80050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7ceffd640 1 -- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b8004390 msgr2=0x7ff7b8004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7ceffd640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b8004390 0x7ff7b8004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7ceffd640 1 -- 172.21.15.27:0/3958643000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b8003520 con 0x7ff7b80039e0 2024-03-06T07:26:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7ce7fc640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7b8004390 0x7ff7b8004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.360+0000 7ff7cf7fe640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7d00b2b50 0x7ff7b80050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.361+0000 7ff7ceffd640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b80039e0 0x7ff7b8003dc0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0001860 tx=0x7ff7b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d4ced7155d1d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.361+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c002f050 con 0x7ff7b80039e0 2024-03-06T07:26:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.361+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c0033650 con 0x7ff7b80039e0 2024-03-06T07:26:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.361+0000 7ff7cdffb640 1 -- 172.21.15.27:0/3958643000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0001430 con 0x7ff7b80039e0 2024-03-06T07:26:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.362+0000 7ff7d57b6640 1 -- 172.21.15.27:0/3958643000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b80039e0 msgr2=0x7ff7b8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.362+0000 7ff7d57b6640 1 --2- 172.21.15.27:0/3958643000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b80039e0 0x7ff7b8003dc0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0001860 tx=0x7ff7b8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.362+0000 7ff7cf7fe640 1 -- 172.21.15.27:0/3958643000 reap_dead start 2024-03-06T07:26:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.362+0000 7ff7d57b6640 1 -- 172.21.15.27:0/3958643000 shutdown_connections 2024-03-06T07:26:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.362+0000 7ff7d57b6640 1 -- 172.21.15.27:0/3958643000 >> 172.21.15.27:0/3958643000 conn(0x7ff7d00d3ca0 msgr2=0x7ff7d00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.362+0000 7ff7d57b6640 1 -- 172.21.15.27:0/3958643000 shutdown_connections 2024-03-06T07:26:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.362+0000 7ff7d57b6640 1 -- 172.21.15.27:0/3958643000 wait complete. 2024-03-06T07:26:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.363+0000 7ff7d57b6640 1 Processor -- start 2024-03-06T07:26:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.363+0000 7ff7d57b6640 1 -- start start 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7d57b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b80039e0 0x7ff7d00ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7d57b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d01576c0 0x7ff7d00cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7d57b6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7d0157b20 0x7ff7d015f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7d57b6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d015e370 con 0x7ff7d01576c0 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7d57b6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00d4be0 con 0x7ff7d0157b20 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7d57b6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00cbe60 con 0x7ff7b80039e0 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d01576c0 0x7ff7d00cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ce7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d01576c0 0x7ff7d00cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34544/0 (socket says 172.21.15.27:34544) 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ce7fc640 1 -- 172.21.15.27:0/961410459 learned_addr learned my addr 172.21.15.27:0/961410459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ceffd640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b80039e0 0x7ff7d00ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7cf7fe640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7d0157b20 0x7ff7d015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ce7fc640 1 -- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b80039e0 msgr2=0x7ff7d00ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ce7fc640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b80039e0 0x7ff7d00ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ce7fc640 1 -- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7d0157b20 msgr2=0x7ff7d015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ce7fc640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7d0157b20 0x7ff7d015f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ce7fc640 1 -- 172.21.15.27:0/961410459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d00cc0d0 con 0x7ff7d01576c0 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ceffd640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b80039e0 0x7ff7d00ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7cf7fe640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7d0157b20 0x7ff7d015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.364+0000 7ff7ce7fc640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d01576c0 0x7ff7d00cd3f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4000fd0 tx=0x7ff7c400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.365+0000 7ff7b77fe640 1 -- 172.21.15.27:0/961410459 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400ee00 con 0x7ff7d01576c0 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.365+0000 7ff7b77fe640 1 -- 172.21.15.27:0/961410459 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c4018070 con 0x7ff7d01576c0 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.365+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d00cc3b0 con 0x7ff7d01576c0 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.365+0000 7ff7b77fe640 1 -- 172.21.15.27:0/961410459 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400ca70 con 0x7ff7d01576c0 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.365+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d01586b0 con 0x7ff7d01576c0 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.366+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d0068700 con 0x7ff7d01576c0 2024-03-06T07:26:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.367+0000 7ff7b77fe640 1 -- 172.21.15.27:0/961410459 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7c400cc10 con 0x7ff7d01576c0 2024-03-06T07:26:55.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.367+0000 7ff7b77fe640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff7bc06b4d0 0x7ff7bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.367+0000 7ff7b77fe640 1 -- 172.21.15.27:0/961410459 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff7c408a0c0 con 0x7ff7d01576c0 2024-03-06T07:26:55.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.367+0000 7ff7ceffd640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff7bc06b4d0 0x7ff7bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.368+0000 7ff7ceffd640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff7bc06b4d0 0x7ff7bc06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0036000 tx=0x7ff7c0031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.371+0000 7ff7b77fe640 1 -- 172.21.15.27:0/961410459 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7c4058f00 con 0x7ff7d01576c0 2024-03-06T07:26:55.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.592+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7d00cc8d0 con 0x7ff7d01576c0 2024-03-06T07:26:55.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.593+0000 7ff7b77fe640 1 -- 172.21.15.27:0/961410459 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4973 (secure 0 0 0) 0x7ff7c4058800 con 0x7ff7d01576c0 2024-03-06T07:26:55.596 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:55.596 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:19.591416+0000","modified":"2024-03-06T07:26:22.766901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6606":{"gid":6606,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/2393218771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2393218771},{"type":"v1","addr":"172.21.15.27:6835","nonce":2393218771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6606,"qdb_cluster":[6606]},"id":5}]} 2024-03-06T07:26:55.596 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff7bc06b4d0 msgr2=0x7ff7bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff7bc06b4d0 0x7ff7bc06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0036000 tx=0x7ff7c0031000 comp rx=0 tx=0).stop 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d01576c0 msgr2=0x7ff7d00cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d01576c0 0x7ff7d00cd3f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4000fd0 tx=0x7ff7c400ec10 comp rx=0 tx=0).stop 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 shutdown_connections 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff7bc06b4d0 0x7ff7bc06d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7d0157b20 0x7ff7d015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d01576c0 0x7ff7d00cd3f0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 --2- 172.21.15.27:0/961410459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7b80039e0 0x7ff7d00ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 >> 172.21.15.27:0/961410459 conn(0x7ff7d00d3ca0 msgr2=0x7ff7d00b6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 shutdown_connections 2024-03-06T07:26:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.596+0000 7ff7d57b6640 1 -- 172.21.15.27:0/961410459 wait complete. 2024-03-06T07:26:55.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:55.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.715+0000 7fed9a681640 1 Processor -- start 2024-03-06T07:26:55.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.715+0000 7fed9a681640 1 -- start start 2024-03-06T07:26:55.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.716+0000 7fed9a681640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fed9415df00 0x7fed940d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.716+0000 7fed9a681640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed940b95a0 0x7fed940b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.716+0000 7fed9a681640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed940cea30 con 0x7fed940b9a30 2024-03-06T07:26:55.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.716+0000 7fed9a681640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed940d2af0 con 0x7fed9415df00 2024-03-06T07:26:55.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.716+0000 7fed9a681640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed940cf3c0 con 0x7fed940b95a0 2024-03-06T07:26:55.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed8bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed940b95a0 0x7fed940b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed93fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fed9415df00 0x7fed940d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed98c1e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fed940b9a30 0x7fed940d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:53546/0 (socket says 172.21.15.27:53546) 2024-03-06T07:26:55.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed98c1e640 1 -- 172.21.15.27:0/1072595082 learned_addr learned my addr 172.21.15.27:0/1072595082 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed937fe640 1 -- 172.21.15.27:0/1072595082 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485040350 0 0) 0x7fed940cea30 con 0x7fed940b9a30 2024-03-06T07:26:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed937fe640 1 -- 172.21.15.27:0/1072595082 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed80003520 con 0x7fed940b9a30 2024-03-06T07:26:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed8bfff640 1 -- 172.21.15.27:0/1072595082 >> v2:172.21.15.69:3300/0 conn(0x7fed9415df00 msgr2=0x7fed940d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed8bfff640 1 --2- 172.21.15.27:0/1072595082 >> v2:172.21.15.69:3300/0 conn(0x7fed9415df00 0x7fed940d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed8bfff640 1 -- 172.21.15.27:0/1072595082 >> v1:172.21.15.69:6789/0 conn(0x7fed940b9a30 legacy=0x7fed940d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed8bfff640 1 -- 172.21.15.27:0/1072595082 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed940cf0e0 con 0x7fed940b95a0 2024-03-06T07:26:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.717+0000 7fed93fff640 1 --2- 172.21.15.27:0/1072595082 >> v2:172.21.15.69:3300/0 conn(0x7fed9415df00 0x7fed940d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.718+0000 7fed8bfff640 1 --2- 172.21.15.27:0/1072595082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed940b95a0 0x7fed940b5700 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0097a0 tx=0x7fed7c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a51dcd62fab9be2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.718+0000 7fed937fe640 1 -- 172.21.15.27:0/1072595082 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c035020 con 0x7fed940b95a0 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.718+0000 7fed937fe640 1 -- 172.21.15.27:0/1072595082 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c031830 con 0x7fed940b95a0 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.718+0000 7fed937fe640 1 -- 172.21.15.27:0/1072595082 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c031c00 con 0x7fed940b95a0 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.719+0000 7fed9a681640 1 -- 172.21.15.27:0/1072595082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed940b95a0 msgr2=0x7fed940b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.719+0000 7fed9a681640 1 --2- 172.21.15.27:0/1072595082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed940b95a0 0x7fed940b5700 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0097a0 tx=0x7fed7c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.719+0000 7fed9a681640 1 -- 172.21.15.27:0/1072595082 shutdown_connections 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.719+0000 7fed9a681640 1 --2- 172.21.15.27:0/1072595082 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed940b95a0 0x7fed940b5700 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.719+0000 7fed9a681640 1 --2- 172.21.15.27:0/1072595082 >> v2:172.21.15.69:3300/0 conn(0x7fed9415df00 0x7fed940d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.719+0000 7fed9a681640 1 -- 172.21.15.27:0/1072595082 >> 172.21.15.27:0/1072595082 conn(0x7fed94071960 msgr2=0x7fed94071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.719+0000 7fed9a681640 1 -- 172.21.15.27:0/1072595082 shutdown_connections 2024-03-06T07:26:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.719+0000 7fed9a681640 1 -- 172.21.15.27:0/1072595082 wait complete. 2024-03-06T07:26:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.720+0000 7fed9a681640 1 Processor -- start 2024-03-06T07:26:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.720+0000 7fed9a681640 1 -- start start 2024-03-06T07:26:55.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed9a681640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed940b95a0 0x7fed94160c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed9a681640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed9415df00 0x7fed940c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed9a681640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed940c9750 0x7fed9415ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed9a681640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed940cec70 con 0x7fed940b95a0 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed9a681640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed940ce750 con 0x7fed940c9750 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed9a681640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed940cedf0 con 0x7fed9415df00 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed93fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed940b95a0 0x7fed94160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed93fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed940b95a0 0x7fed94160c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34548/0 (socket says 172.21.15.27:34548) 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed93fff640 1 -- 172.21.15.27:0/2545494514 learned_addr learned my addr 172.21.15.27:0/2545494514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed8bfff640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed9415df00 0x7fed940c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed98c1e640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed940c9750 0x7fed9415ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed93fff640 1 -- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed9415df00 msgr2=0x7fed940c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed93fff640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed9415df00 0x7fed940c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed93fff640 1 -- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed940c9750 msgr2=0x7fed9415ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed93fff640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed940c9750 0x7fed9415ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.721+0000 7fed93fff640 1 -- 172.21.15.27:0/2545494514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed940c9cb0 con 0x7fed940b95a0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.722+0000 7fed93fff640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed940b95a0 0x7fed94160c40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fed8400e960 tx=0x7fed8400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.722+0000 7fed917fa640 1 -- 172.21.15.27:0/2545494514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84017070 con 0x7fed940b95a0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.722+0000 7fed917fa640 1 -- 172.21.15.27:0/2545494514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8400ca00 con 0x7fed940b95a0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.722+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed940c9f90 con 0x7fed940b95a0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.722+0000 7fed917fa640 1 -- 172.21.15.27:0/2545494514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8400cdd0 con 0x7fed940b95a0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.722+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9415edd0 con 0x7fed940b95a0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.723+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed940d0140 con 0x7fed940b95a0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.723+0000 7fed917fa640 1 -- 172.21.15.27:0/2545494514 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed8401a070 con 0x7fed940b95a0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.724+0000 7fed917fa640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fed7406b4d0 0x7fed7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.724+0000 7fed917fa640 1 -- 172.21.15.27:0/2545494514 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fed8408a370 con 0x7fed940b95a0 2024-03-06T07:26:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.724+0000 7fed8bfff640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fed7406b4d0 0x7fed7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.725+0000 7fed8bfff640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fed7406b4d0 0x7fed7406d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fed7c036000 tx=0x7fed7c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.728+0000 7fed917fa640 1 -- 172.21.15.27:0/2545494514 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed940d0140 con 0x7fed940b95a0 2024-03-06T07:26:55.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.911+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed94067ff0 con 0x7fed940b95a0 2024-03-06T07:26:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.912+0000 7fed917fa640 1 -- 172.21.15.27:0/2545494514 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16694 (secure 0 0 0) 0x7fed84058bf0 con 0x7fed940b95a0 2024-03-06T07:26:55.915 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:26:55.915 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:29.258129+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:17.914214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:18.608105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.2e","mappings":[{"from":6,"to":0}]},{"pgid":"11.3a","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fed7406b4d0 msgr2=0x7fed7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fed7406b4d0 0x7fed7406d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fed7c036000 tx=0x7fed7c008030 comp rx=0 tx=0).stop 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed940b95a0 msgr2=0x7fed94160c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed940b95a0 0x7fed94160c40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fed8400e960 tx=0x7fed8400ee20 comp rx=0 tx=0).stop 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 shutdown_connections 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fed7406b4d0 0x7fed7406d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed940c9750 0x7fed9415ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fed9415df00 0x7fed940c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 --2- 172.21.15.27:0/2545494514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed940b95a0 0x7fed94160c40 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 >> 172.21.15.27:0/2545494514 conn(0x7fed94071960 msgr2=0x7fed94162140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 shutdown_connections 2024-03-06T07:26:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:55.914+0000 7fed9a681640 1 -- 172.21.15.27:0/2545494514 wait complete. 2024-03-06T07:26:55.928 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.033+0000 7f8139e2a640 1 Processor -- start 2024-03-06T07:26:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.033+0000 7f8139e2a640 1 -- start start 2024-03-06T07:26:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.034+0000 7f8139e2a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f813415df00 0x7f81340d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.034+0000 7f8139e2a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81340b9a30 0x7f81340d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.034+0000 7f8139e2a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81340cea30 con 0x7f81340b95a0 2024-03-06T07:26:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.034+0000 7f8139e2a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81340d2af0 con 0x7f81340b9a30 2024-03-06T07:26:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.034+0000 7f8139e2a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81340cf3c0 con 0x7f813415df00 2024-03-06T07:26:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.034+0000 7f8133fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81340b9a30 0x7f81340d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.034+0000 7f8133fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81340b9a30 0x7f81340d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34558/0 (socket says 172.21.15.27:34558) 2024-03-06T07:26:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.034+0000 7f8133fff640 1 -- 172.21.15.27:0/2553999295 learned_addr learned my addr 172.21.15.27:0/2553999295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.034+0000 7f81337fe640 1 --2- 172.21.15.27:0/2553999295 >> v2:172.21.15.69:3300/0 conn(0x7f813415df00 0x7f81340d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.035+0000 7f8133fff640 1 -- 172.21.15.27:0/2553999295 >> v2:172.21.15.69:3300/0 conn(0x7f813415df00 msgr2=0x7f81340d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.035+0000 7f8133fff640 1 --2- 172.21.15.27:0/2553999295 >> v2:172.21.15.69:3300/0 conn(0x7f813415df00 0x7f81340d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.035+0000 7f8133fff640 1 -- 172.21.15.27:0/2553999295 >> v1:172.21.15.69:6789/0 conn(0x7f81340b95a0 legacy=0x7f81340b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.035+0000 7f8133fff640 1 -- 172.21.15.27:0/2553999295 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81340cf0e0 con 0x7f81340b9a30 2024-03-06T07:26:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.035+0000 7f8133fff640 1 --2- 172.21.15.27:0/2553999295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81340b9a30 0x7f81340d3220 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f81340d11b0 tx=0x7f812c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba01ed77c115d8ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f81327fc640 1 -- 172.21.15.27:0/2553999295 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c035020 con 0x7f81340b9a30 2024-03-06T07:26:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f81327fc640 1 -- 172.21.15.27:0/2553999295 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c031870 con 0x7f81340b9a30 2024-03-06T07:26:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f81327fc640 1 -- 172.21.15.27:0/2553999295 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c031c40 con 0x7f81340b9a30 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f8139e2a640 1 -- 172.21.15.27:0/2553999295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81340b9a30 msgr2=0x7f81340d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f8139e2a640 1 --2- 172.21.15.27:0/2553999295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81340b9a30 0x7f81340d3220 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f81340d11b0 tx=0x7f812c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f8139e2a640 1 -- 172.21.15.27:0/2553999295 shutdown_connections 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f8139e2a640 1 --2- 172.21.15.27:0/2553999295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81340b9a30 0x7f81340d3220 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f8139e2a640 1 --2- 172.21.15.27:0/2553999295 >> v2:172.21.15.69:3300/0 conn(0x7f813415df00 0x7f81340d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f8139e2a640 1 -- 172.21.15.27:0/2553999295 >> 172.21.15.27:0/2553999295 conn(0x7f8134071960 msgr2=0x7f8134071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f8139e2a640 1 -- 172.21.15.27:0/2553999295 shutdown_connections 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.036+0000 7f8139e2a640 1 -- 172.21.15.27:0/2553999295 wait complete. 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.037+0000 7f8139e2a640 1 Processor -- start 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.037+0000 7f8139e2a640 1 -- start start 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.038+0000 7f8139e2a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81340b95a0 0x7f81340c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.038+0000 7f8139e2a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813415df00 0x7f813415a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.038+0000 7f8139e2a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8134161020 0x7f81340cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.038+0000 7f8139e2a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340cec70 con 0x7f8134161020 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.038+0000 7f8139e2a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340ce750 con 0x7f813415df00 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.038+0000 7f8139e2a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81340cedf0 con 0x7f81340b95a0 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.038+0000 7f8133fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8134161020 0x7f81340cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.038+0000 7f8133fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8134161020 0x7f81340cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34566/0 (socket says 172.21.15.27:34566) 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.038+0000 7f8133fff640 1 -- 172.21.15.27:0/4117900286 learned_addr learned my addr 172.21.15.27:0/4117900286 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f81337fe640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81340b95a0 0x7f81340c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8132ffd640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813415df00 0x7f813415a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8133fff640 1 -- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81340b95a0 msgr2=0x7f81340c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8133fff640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81340b95a0 0x7f81340c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8133fff640 1 -- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813415df00 msgr2=0x7f813415a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8133fff640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813415df00 0x7f813415a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8133fff640 1 -- 172.21.15.27:0/4117900286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81340cbd80 con 0x7f8134161020 2024-03-06T07:26:56.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f81337fe640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81340b95a0 0x7f81340c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:56.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8132ffd640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813415df00 0x7f813415a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:56.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8133fff640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8134161020 0x7f81340cce20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f812c02b950 tx=0x7f812c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f811bfff640 1 -- 172.21.15.27:0/4117900286 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c040070 con 0x7f8134161020 2024-03-06T07:26:56.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f811bfff640 1 -- 172.21.15.27:0/4117900286 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c030d00 con 0x7f8134161020 2024-03-06T07:26:56.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81340cc060 con 0x7f8134161020 2024-03-06T07:26:56.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f811bfff640 1 -- 172.21.15.27:0/4117900286 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c035040 con 0x7f8134161020 2024-03-06T07:26:56.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.039+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81340cc620 con 0x7f8134161020 2024-03-06T07:26:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.040+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8134067ff0 con 0x7f8134161020 2024-03-06T07:26:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.041+0000 7f811bfff640 1 -- 172.21.15.27:0/4117900286 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f812c038020 con 0x7f8134161020 2024-03-06T07:26:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.041+0000 7f811bfff640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f812006b4d0 0x7f812006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.041+0000 7f811bfff640 1 -- 172.21.15.27:0/4117900286 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f812c0ac0d0 con 0x7f8134161020 2024-03-06T07:26:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.041+0000 7f81337fe640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f812006b4d0 0x7f812006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.042+0000 7f81337fe640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f812006b4d0 0x7f812006d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8124009a80 tx=0x7f81240039d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.045+0000 7f811bfff640 1 -- 172.21.15.27:0/4117900286 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f812c07ae90 con 0x7f8134161020 2024-03-06T07:26:56.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.265+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f81340688e0 con 0x7f8134161020 2024-03-06T07:26:56.268 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:26:56.266+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.267+0000 7f811bfff640 1 -- 172.21.15.27:0/4117900286 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) v1 ==== 161+0+0 (secure 0 0 0) 0x7f812c07a790 con 0x7f8134161020 2024-03-06T07:26:56.269 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f812006b4d0 msgr2=0x7f812006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f812006b4d0 0x7f812006d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8124009a80 tx=0x7f81240039d0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8134161020 msgr2=0x7f81340cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8134161020 0x7f81340cce20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f812c02b950 tx=0x7f812c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 shutdown_connections 2024-03-06T07:26:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f812006b4d0 0x7f812006d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8134161020 0x7f81340cce20 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f813415df00 0x7f813415a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 --2- 172.21.15.27:0/4117900286 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f81340b95a0 0x7f81340c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.270+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 >> 172.21.15.27:0/4117900286 conn(0x7f8134071960 msgr2=0x7f813415d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.271+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 shutdown_connections 2024-03-06T07:26:56.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.271+0000 7f8139e2a640 1 -- 172.21.15.27:0/4117900286 wait complete. 2024-03-06T07:26:56.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:56.313 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:26:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.394+0000 7f7179de9640 1 Processor -- start 2024-03-06T07:26:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.394+0000 7f7179de9640 1 -- start start 2024-03-06T07:26:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.395+0000 7f7179de9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7174071540 0x7f7174071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.395+0000 7f7179de9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71740b2d30 0x7f71740ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.395+0000 7f7179de9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7174071e60 con 0x7f71740cecb0 2024-03-06T07:26:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.395+0000 7f7179de9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71740d3230 con 0x7f7174071540 2024-03-06T07:26:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.395+0000 7f7179de9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71740d4960 con 0x7f71740b2d30 2024-03-06T07:26:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.396+0000 7f7172ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71740b2d30 0x7f71740ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.396+0000 7f71737fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7174071540 0x7f7174071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.396+0000 7f7173fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f71740cecb0 0x7f71740b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:53572/0 (socket says 172.21.15.27:53572) 2024-03-06T07:26:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.396+0000 7f7172ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71740b2d30 0x7f71740ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:50910/0 (socket says 172.21.15.27:50910) 2024-03-06T07:26:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.396+0000 7f7172ffd640 1 -- 172.21.15.27:0/1360054772 learned_addr learned my addr 172.21.15.27:0/1360054772 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.396+0000 7f7172ffd640 1 -- 172.21.15.27:0/1360054772 >> v2:172.21.15.69:3300/0 conn(0x7f7174071540 msgr2=0x7f7174071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.396+0000 7f7172ffd640 1 --2- 172.21.15.27:0/1360054772 >> v2:172.21.15.69:3300/0 conn(0x7f7174071540 0x7f7174071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.396+0000 7f7172ffd640 1 -- 172.21.15.27:0/1360054772 >> v1:172.21.15.69:6789/0 conn(0x7f71740cecb0 legacy=0x7f71740b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.396+0000 7f7172ffd640 1 -- 172.21.15.27:0/1360054772 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71740d36c0 con 0x7f71740b2d30 2024-03-06T07:26:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f7172ffd640 1 --2- 172.21.15.27:0/1360054772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71740b2d30 0x7f71740ce770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7168009870 tx=0x7f716802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f9542c882528b27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f71727fc640 1 -- 172.21.15.27:0/1360054772 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168034070 con 0x7f71740b2d30 2024-03-06T07:26:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f71727fc640 1 -- 172.21.15.27:0/1360054772 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7168030840 con 0x7f71740b2d30 2024-03-06T07:26:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f71727fc640 1 -- 172.21.15.27:0/1360054772 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168030c10 con 0x7f71740b2d30 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f7179de9640 1 -- 172.21.15.27:0/1360054772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71740b2d30 msgr2=0x7f71740ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f7179de9640 1 --2- 172.21.15.27:0/1360054772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71740b2d30 0x7f71740ce770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7168009870 tx=0x7f716802b980 comp rx=0 tx=0).stop 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f7179de9640 1 -- 172.21.15.27:0/1360054772 shutdown_connections 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f7179de9640 1 --2- 172.21.15.27:0/1360054772 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71740b2d30 0x7f71740ce770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f7179de9640 1 --2- 172.21.15.27:0/1360054772 >> v2:172.21.15.69:3300/0 conn(0x7f7174071540 0x7f7174071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f7179de9640 1 -- 172.21.15.27:0/1360054772 >> 172.21.15.27:0/1360054772 conn(0x7f71740d3d60 msgr2=0x7f71740d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f7179de9640 1 -- 172.21.15.27:0/1360054772 shutdown_connections 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.397+0000 7f7179de9640 1 -- 172.21.15.27:0/1360054772 wait complete. 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.399+0000 7f7179de9640 1 Processor -- start 2024-03-06T07:26:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.399+0000 7f7179de9640 1 -- start start 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.399+0000 7f7179de9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7174071540 0x7f71740cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.399+0000 7f7179de9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71740b2d30 0x7f71741f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.399+0000 7f7179de9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71740cecb0 0x7f71741f7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.399+0000 7f7179de9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71740d2be0 con 0x7f71740b2d30 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.399+0000 7f7179de9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71740d2900 con 0x7f71740cecb0 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.399+0000 7f7179de9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71740d2ec0 con 0x7f7174071540 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7172ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71740b2d30 0x7f71741f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7172ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71740b2d30 0x7f71741f4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34586/0 (socket says 172.21.15.27:34586) 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7172ffd640 1 -- 172.21.15.27:0/1479061778 learned_addr learned my addr 172.21.15.27:0/1479061778 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f71737fe640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7174071540 0x7f71740cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7173fff640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71740cecb0 0x7f71741f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7172ffd640 1 -- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7174071540 msgr2=0x7f71740cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7172ffd640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7174071540 0x7f71740cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7172ffd640 1 -- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71740cecb0 msgr2=0x7f71741f7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7172ffd640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71740cecb0 0x7f71741f7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7172ffd640 1 -- 172.21.15.27:0/1479061778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71740cc5b0 con 0x7f71740b2d30 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7172ffd640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71740b2d30 0x7f71741f4e50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f716802b9c0 tx=0x7f7168004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f715bfff640 1 -- 172.21.15.27:0/1479061778 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168048070 con 0x7f71740b2d30 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f715bfff640 1 -- 172.21.15.27:0/1479061778 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7168030640 con 0x7f71740b2d30 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71740cc890 con 0x7f71740b2d30 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f715bfff640 1 -- 172.21.15.27:0/1479061778 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168034040 con 0x7f71740b2d30 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.400+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71740cce50 con 0x7f71740b2d30 2024-03-06T07:26:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.401+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7174072070 con 0x7f71740b2d30 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.402+0000 7f715bfff640 1 -- 172.21.15.27:0/1479061778 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7168037070 con 0x7f71740b2d30 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.402+0000 7f715bfff640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716006b4d0 0x7f716006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.402+0000 7f715bfff640 1 -- 172.21.15.27:0/1479061778 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f71680abf80 con 0x7f71740b2d30 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.403+0000 7f71737fe640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716006b4d0 0x7f716006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.403+0000 7f71737fe640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716006b4d0 0x7f716006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f71740cda80 tx=0x7f716400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.407+0000 7f715bfff640 1 -- 172.21.15.27:0/1479061778 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f716807a7c0 con 0x7f71740b2d30 2024-03-06T07:26:56.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:56.626+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f717404f410 con 0x7f71740b2d30 2024-03-06T07:26:57.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.305+0000 7f715bfff640 1 -- 172.21.15.27:0/1479061778 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) v1 ==== 104+0+0 (secure 0 0 0) 0x7f716807e860 con 0x7f71740b2d30 2024-03-06T07:26:57.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716006b4d0 msgr2=0x7f716006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716006b4d0 0x7f716006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f71740cda80 tx=0x7f716400b040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71740b2d30 msgr2=0x7f71741f4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71740b2d30 0x7f71741f4e50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f716802b9c0 tx=0x7f7168004730 comp rx=0 tx=0).stop 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 shutdown_connections 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f716006b4d0 0x7f716006d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71740cecb0 0x7f71741f7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71740b2d30 0x7f71741f4e50 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 --2- 172.21.15.27:0/1479061778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7174071540 0x7f71740cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 >> 172.21.15.27:0/1479061778 conn(0x7f71740d3d60 msgr2=0x7f717415b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 shutdown_connections 2024-03-06T07:26:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.308+0000 7f7179de9640 1 -- 172.21.15.27:0/1479061778 wait complete. 2024-03-06T07:26:57.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:57.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.477+0000 7f5a8b850640 1 Processor -- start 2024-03-06T07:26:57.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.478+0000 7f5a8b850640 1 -- start start 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.478+0000 7f5a8b850640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5a840b7420 0x7f5a840d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.478+0000 7f5a8b850640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a840d3740 con 0x7f5a8415dd90 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.478+0000 7f5a8b850640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a840d3120 con 0x7f5a840ceea0 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.478+0000 7f5a8b850640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a840d3400 con 0x7f5a840b7420 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.478+0000 7f5a895ec640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5a8415dd90 0x7f5a840d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44580/0 (socket says 172.21.15.27:44580) 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a895ec640 1 -- 172.21.15.27:0/3717730916 learned_addr learned my addr 172.21.15.27:0/3717730916 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a89ded640 1 --2- 172.21.15.27:0/3717730916 >> v2:172.21.15.69:3300/0 conn(0x7f5a840b7420 0x7f5a840d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186853384 0 0) 0x7f5a840d3740 con 0x7f5a8415dd90 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a70003520 con 0x7f5a8415dd90 2024-03-06T07:26:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3989428049 0 0) 0x7f5a70003520 con 0x7f5a8415dd90 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 >> v2:172.21.15.69:3300/0 conn(0x7f5a840b7420 msgr2=0x7f5a840d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 --2- 172.21.15.27:0/3717730916 >> v2:172.21.15.69:3300/0 conn(0x7f5a840b7420 0x7f5a840d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 >> v1:172.21.15.69:6789/0 conn(0x7f5a840ceea0 legacy=0x7f5a840b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a840d39e0 con 0x7f5a8415dd90 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f5a80001690 con 0x7f5a8415dd90 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 >> v1:172.21.15.27:6789/0 conn(0x7f5a8415dd90 legacy=0x7f5a840d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a700039b0 0x7f5a70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700043b0 0x7f5a70004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a70005080 0x7f5a7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840d3400 con 0x7f5a70005080 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7000bfe0 con 0x7f5a700039b0 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7000c1e0 con 0x7f5a700043b0 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a89ded640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a70005080 0x7f5a7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a88deb640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700043b0 0x7f5a70004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.479+0000 7f5a895ec640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a700039b0 0x7f5a70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a89ded640 1 -- 172.21.15.27:0/3717730916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700043b0 msgr2=0x7f5a70004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a89ded640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a700043b0 0x7f5a70004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a89ded640 1 -- 172.21.15.27:0/3717730916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a700039b0 msgr2=0x7f5a70003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a89ded640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a700039b0 0x7f5a70003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a89ded640 1 -- 172.21.15.27:0/3717730916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a70003520 con 0x7f5a70005080 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a89ded640 1 -- 172.21.15.27:0/3717730916 reap_dead start 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a89ded640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a70005080 0x7f5a7000b9c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009fc0 tx=0x7f5a70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54abcaabe25ff72b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c009c10 con 0x7f5a70005080 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c02cca0 con 0x7f5a70005080 2024-03-06T07:26:57.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.480+0000 7f5a7bfff640 1 -- 172.21.15.27:0/3717730916 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c003bd0 con 0x7f5a70005080 2024-03-06T07:26:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.481+0000 7f5a8b850640 1 -- 172.21.15.27:0/3717730916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a70005080 msgr2=0x7f5a7000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.481+0000 7f5a8b850640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a70005080 0x7f5a7000b9c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009fc0 tx=0x7f5a70005830 comp rx=0 tx=0).stop 2024-03-06T07:26:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.481+0000 7f5a8b850640 1 -- 172.21.15.27:0/3717730916 shutdown_connections 2024-03-06T07:26:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.481+0000 7f5a8b850640 1 --2- 172.21.15.27:0/3717730916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a70005080 0x7f5a7000b9c0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.481+0000 7f5a8b850640 1 -- 172.21.15.27:0/3717730916 >> 172.21.15.27:0/3717730916 conn(0x7f5a84071960 msgr2=0x7f5a84071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.481+0000 7f5a8b850640 1 -- 172.21.15.27:0/3717730916 shutdown_connections 2024-03-06T07:26:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.481+0000 7f5a8b850640 1 -- 172.21.15.27:0/3717730916 wait complete. 2024-03-06T07:26:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.482+0000 7f5a8b850640 1 Processor -- start 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.482+0000 7f5a8b850640 1 -- start start 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a8b850640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a70005080 0x7f5a840cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a8b850640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a84149800 0x7f5a840cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a8b850640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a840c95f0 0x7f5a840c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a8b850640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840d29d0 con 0x7f5a84149800 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a8b850640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8415e170 con 0x7f5a840c95f0 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a8b850640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840d2b50 con 0x7f5a70005080 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a88deb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a84149800 0x7f5a840cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a88deb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a84149800 0x7f5a840cc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34616/0 (socket says 172.21.15.27:34616) 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a88deb640 1 -- 172.21.15.27:0/1280318422 learned_addr learned my addr 172.21.15.27:0/1280318422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a89ded640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a840c95f0 0x7f5a840c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a895ec640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a70005080 0x7f5a840cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a88deb640 1 -- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a70005080 msgr2=0x7f5a840cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a88deb640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a70005080 0x7f5a840cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a88deb640 1 -- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a840c95f0 msgr2=0x7f5a840c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.483+0000 7f5a88deb640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a840c95f0 0x7f5a840c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.484+0000 7f5a88deb640 1 -- 172.21.15.27:0/1280318422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8415e440 con 0x7f5a84149800 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.484+0000 7f5a88deb640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a84149800 0x7f5a840cc660 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400f7c0 tx=0x7f5a7400fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.484+0000 7f5a79ffb640 1 -- 172.21.15.27:0/1280318422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400fe70 con 0x7f5a84149800 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.484+0000 7f5a79ffb640 1 -- 172.21.15.27:0/1280318422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a74018070 con 0x7f5a84149800 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.484+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a84160f60 con 0x7f5a84149800 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.484+0000 7f5a79ffb640 1 -- 172.21.15.27:0/1280318422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400d6d0 con 0x7f5a84149800 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.484+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a84161240 con 0x7f5a84149800 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.485+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a84072a30 con 0x7f5a84149800 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.485+0000 7f5a79ffb640 1 -- 172.21.15.27:0/1280318422 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a7400d870 con 0x7f5a84149800 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.486+0000 7f5a79ffb640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5a6806b4d0 0x7f5a6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.486+0000 7f5a79ffb640 1 -- 172.21.15.27:0/1280318422 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5a7408a120 con 0x7f5a84149800 2024-03-06T07:26:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.486+0000 7f5a895ec640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5a6806b4d0 0x7f5a6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.486+0000 7f5a895ec640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5a6806b4d0 0x7f5a6806d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a840b36d0 tx=0x7f5a80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.490+0000 7f5a79ffb640 1 -- 172.21.15.27:0/1280318422 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a74058f10 con 0x7f5a84149800 2024-03-06T07:26:57.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:57.673+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5a8404f410 con 0x7f5a84149800 2024-03-06T07:26:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.314+0000 7f5a79ffb640 1 -- 172.21.15.27:0/1280318422 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v74) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5a74058810 con 0x7f5a84149800 2024-03-06T07:26:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5a6806b4d0 msgr2=0x7f5a6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5a6806b4d0 0x7f5a6806d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a840b36d0 tx=0x7f5a80009040 comp rx=0 tx=0).stop 2024-03-06T07:26:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a84149800 msgr2=0x7f5a840cc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a84149800 0x7f5a840cc660 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400f7c0 tx=0x7f5a7400fc80 comp rx=0 tx=0).stop 2024-03-06T07:26:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 shutdown_connections 2024-03-06T07:26:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5a6806b4d0 0x7f5a6806d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a840c95f0 0x7f5a840c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a84149800 0x7f5a840cc660 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 --2- 172.21.15.27:0/1280318422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5a70005080 0x7f5a840cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.318+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 >> 172.21.15.27:0/1280318422 conn(0x7f5a84071960 msgr2=0x7f5a840d4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.319+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 shutdown_connections 2024-03-06T07:26:58.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.319+0000 7f5a8b850640 1 -- 172.21.15.27:0/1280318422 wait complete. 2024-03-06T07:26:58.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.448+0000 7f0167d7a640 1 Processor -- start 2024-03-06T07:26:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.448+0000 7f0167d7a640 1 -- start start 2024-03-06T07:26:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.449+0000 7f0167d7a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0160144640 0x7f016006e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.449+0000 7f0167d7a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f01600ce900 0x7f01600ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.449+0000 7f0167d7a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016015bc40 con 0x7f016006e940 2024-03-06T07:26:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.449+0000 7f0167d7a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016006df70 con 0x7f0160144640 2024-03-06T07:26:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.449+0000 7f0167d7a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01600cf270 con 0x7f01600ce900 2024-03-06T07:26:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.449+0000 7f0165b16640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0160144640 0x7f016006e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.449+0000 7f0165b16640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0160144640 0x7f016006e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34626/0 (socket says 172.21.15.27:34626) 2024-03-06T07:26:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.449+0000 7f0165b16640 1 -- 172.21.15.27:0/3913638315 learned_addr learned my addr 172.21.15.27:0/3913638315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.450+0000 7f0165b16640 1 -- 172.21.15.27:0/3913638315 >> v2:172.21.15.69:3300/0 conn(0x7f01600ce900 msgr2=0x7f01600ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.450+0000 7f0166317640 1 --2- 172.21.15.27:0/3913638315 >> v2:172.21.15.69:3300/0 conn(0x7f01600ce900 0x7f01600ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.450+0000 7f0165b16640 1 --2- 172.21.15.27:0/3913638315 >> v2:172.21.15.69:3300/0 conn(0x7f01600ce900 0x7f01600ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.450+0000 7f0165b16640 1 -- 172.21.15.27:0/3913638315 >> v1:172.21.15.69:6789/0 conn(0x7f016006e940 legacy=0x7f016015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.450+0000 7f0165b16640 1 -- 172.21.15.27:0/3913638315 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016015bf20 con 0x7f0160144640 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.450+0000 7f0166317640 1 --2- 172.21.15.27:0/3913638315 >> v2:172.21.15.69:3300/0 conn(0x7f01600ce900 0x7f01600ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.450+0000 7f0165b16640 1 --2- 172.21.15.27:0/3913638315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 0x7f016006e400 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f01600b4f40 tx=0x7f015802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13811376b072384 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0164b14640 1 -- 172.21.15.27:0/3913638315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015803b070 con 0x7f0160144640 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0164b14640 1 -- 172.21.15.27:0/3913638315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0158036440 con 0x7f0160144640 2024-03-06T07:26:58.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0164b14640 1 -- 172.21.15.27:0/3913638315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158036810 con 0x7f0160144640 2024-03-06T07:26:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0167d7a640 1 -- 172.21.15.27:0/3913638315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 msgr2=0x7f016006e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0167d7a640 1 --2- 172.21.15.27:0/3913638315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 0x7f016006e400 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f01600b4f40 tx=0x7f015802e0e0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0167d7a640 1 -- 172.21.15.27:0/3913638315 shutdown_connections 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0167d7a640 1 --2- 172.21.15.27:0/3913638315 >> v2:172.21.15.69:3300/0 conn(0x7f01600ce900 0x7f01600ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0167d7a640 1 --2- 172.21.15.27:0/3913638315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 0x7f016006e400 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0167d7a640 1 -- 172.21.15.27:0/3913638315 >> 172.21.15.27:0/3913638315 conn(0x7f0160066d00 msgr2=0x7f0160067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0167d7a640 1 -- 172.21.15.27:0/3913638315 shutdown_connections 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.451+0000 7f0167d7a640 1 -- 172.21.15.27:0/3913638315 wait complete. 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.452+0000 7f0167d7a640 1 Processor -- start 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.452+0000 7f0167d7a640 1 -- start start 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0167d7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f016006e940 0x7f016015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0167d7a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01600ce900 0x7f01600b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0167d7a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 0x7f0160142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0167d7a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0160070c20 con 0x7f0160144640 2024-03-06T07:26:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0167d7a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0160070aa0 con 0x7f01600ce900 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0167d7a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016015b840 con 0x7f016006e940 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0166317640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 0x7f0160142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0166317640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 0x7f0160142f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34632/0 (socket says 172.21.15.27:34632) 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0166317640 1 -- 172.21.15.27:0/1534731829 learned_addr learned my addr 172.21.15.27:0/1534731829 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0165315640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01600ce900 0x7f01600b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.453+0000 7f0165b16640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f016006e940 0x7f016015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f0166317640 1 -- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f016006e940 msgr2=0x7f016015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f0166317640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f016006e940 0x7f016015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f0166317640 1 -- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01600ce900 msgr2=0x7f01600b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f0166317640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01600ce900 0x7f01600b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f0166317640 1 -- 172.21.15.27:0/1534731829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01600af8e0 con 0x7f0160144640 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f0166317640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 0x7f0160142f40 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f015000cef0 tx=0x7f015000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f015e7fc640 1 -- 172.21.15.27:0/1534731829 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015001a070 con 0x7f0160144640 2024-03-06T07:26:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f015e7fc640 1 -- 172.21.15.27:0/1534731829 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015000d860 con 0x7f0160144640 2024-03-06T07:26:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f015e7fc640 1 -- 172.21.15.27:0/1534731829 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015000dc50 con 0x7f0160144640 2024-03-06T07:26:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01600afbc0 con 0x7f0160144640 2024-03-06T07:26:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.454+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01600b0180 con 0x7f0160144640 2024-03-06T07:26:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.455+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01600733c0 con 0x7f0160144640 2024-03-06T07:26:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.455+0000 7f015e7fc640 1 -- 172.21.15.27:0/1534731829 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f015001d070 con 0x7f0160144640 2024-03-06T07:26:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.456+0000 7f015e7fc640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f014c06b4d0 0x7f014c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.456+0000 7f015e7fc640 1 -- 172.21.15.27:0/1534731829 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f015008a8b0 con 0x7f0160144640 2024-03-06T07:26:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.456+0000 7f0165b16640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f014c06b4d0 0x7f014c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.457+0000 7f0165b16640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f014c06b4d0 0x7f014c06d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0158038000 tx=0x7f0158008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.460+0000 7f015e7fc640 1 -- 172.21.15.27:0/1534731829 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01500592c0 con 0x7f0160144640 2024-03-06T07:26:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:58.643+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0160072150 con 0x7f0160144640 2024-03-06T07:26:59.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.316+0000 7f015e7fc640 1 -- 172.21.15.27:0/1534731829 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v75) v1 ==== 169+0+0 (secure 0 0 0) 0x7f015005d360 con 0x7f0160144640 2024-03-06T07:26:59.318 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f014c06b4d0 msgr2=0x7f014c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f014c06b4d0 0x7f014c06d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0158038000 tx=0x7f0158008040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 msgr2=0x7f0160142f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 0x7f0160142f40 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f015000cef0 tx=0x7f015000a710 comp rx=0 tx=0).stop 2024-03-06T07:26:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 shutdown_connections 2024-03-06T07:26:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f014c06b4d0 0x7f014c06d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0160144640 0x7f0160142f40 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01600ce900 0x7f01600b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 --2- 172.21.15.27:0/1534731829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f016006e940 0x7f016015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 >> 172.21.15.27:0/1534731829 conn(0x7f0160066d00 msgr2=0x7f0160071460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 shutdown_connections 2024-03-06T07:26:59.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.320+0000 7f0167d7a640 1 -- 172.21.15.27:0/1534731829 wait complete. 2024-03-06T07:26:59.338 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:59.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.452+0000 7f3c81bf7640 1 Processor -- start 2024-03-06T07:26:59.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.452+0000 7f3c81bf7640 1 -- start start 2024-03-06T07:26:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.453+0000 7f3c81bf7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 0x7f3c7c0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.453+0000 7f3c81bf7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7c06f070 con 0x7f3c7c15dbd0 2024-03-06T07:26:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.453+0000 7f3c81bf7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7c071940 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.453+0000 7f3c81bf7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0b5db0 con 0x7f3c7c0b5840 2024-03-06T07:26:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.453+0000 7f3c7b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3c7c15dbd0 0x7f3c7c15ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44594/0 (socket says 172.21.15.27:44594) 2024-03-06T07:26:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.453+0000 7f3c7b7fe640 1 -- 172.21.15.27:0/1843533093 learned_addr learned my addr 172.21.15.27:0/1843533093 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.453+0000 7f3c7affd640 1 --2- 172.21.15.27:0/1843533093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 0x7f3c7c0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.453+0000 7f3c7a7fc640 1 -- 172.21.15.27:0/1843533093 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55669029 0 0) 0x7f3c7c06f070 con 0x7f3c7c15dbd0 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.453+0000 7f3c7a7fc640 1 -- 172.21.15.27:0/1843533093 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c64003520 con 0x7f3c7c15dbd0 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.454+0000 7f3c7a7fc640 1 -- 172.21.15.27:0/1843533093 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213072778 0 0) 0x7f3c7c071940 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.454+0000 7f3c7a7fc640 1 -- 172.21.15.27:0/1843533093 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c7c06f070 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.454+0000 7f3c7a7fc640 1 -- 172.21.15.27:0/1843533093 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3416632731 0 0) 0x7f3c64003520 con 0x7f3c7c15dbd0 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.454+0000 7f3c7affd640 1 -- 172.21.15.27:0/1843533093 >> v1:172.21.15.69:6789/0 conn(0x7f3c7c15dfb0 legacy=0x7f3c7c066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.454+0000 7f3c7affd640 1 -- 172.21.15.27:0/1843533093 >> v1:172.21.15.27:6789/0 conn(0x7f3c7c15dbd0 legacy=0x7f3c7c15ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.454+0000 7f3c7affd640 1 -- 172.21.15.27:0/1843533093 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c071c00 con 0x7f3c7c0b5840 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c7affd640 1 --2- 172.21.15.27:0/1843533093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 0x7f3c7c0b39b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3c700097d0 tx=0x7f3c7002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e9cb1c348443721 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c7a7fc640 1 -- 172.21.15.27:0/1843533093 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70037070 con 0x7f3c7c0b5840 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c7a7fc640 1 -- 172.21.15.27:0/1843533093 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70030510 con 0x7f3c7c0b5840 2024-03-06T07:26:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c7a7fc640 1 -- 172.21.15.27:0/1843533093 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c001690 con 0x7f3c7c0b5840 2024-03-06T07:26:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c81bf7640 1 -- 172.21.15.27:0/1843533093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 msgr2=0x7f3c7c0b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c81bf7640 1 --2- 172.21.15.27:0/1843533093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 0x7f3c7c0b39b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3c700097d0 tx=0x7f3c7002b800 comp rx=0 tx=0).stop 2024-03-06T07:26:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c81bf7640 1 -- 172.21.15.27:0/1843533093 shutdown_connections 2024-03-06T07:26:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c81bf7640 1 --2- 172.21.15.27:0/1843533093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 0x7f3c7c0b39b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c81bf7640 1 -- 172.21.15.27:0/1843533093 >> 172.21.15.27:0/1843533093 conn(0x7f3c7c0b26b0 msgr2=0x7f3c7c0b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c81bf7640 1 -- 172.21.15.27:0/1843533093 shutdown_connections 2024-03-06T07:26:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.455+0000 7f3c81bf7640 1 -- 172.21.15.27:0/1843533093 wait complete. 2024-03-06T07:26:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.456+0000 7f3c81bf7640 1 Processor -- start 2024-03-06T07:26:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.457+0000 7f3c81bf7640 1 -- start start 2024-03-06T07:26:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.457+0000 7f3c81bf7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 0x7f3c7c157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.457+0000 7f3c81bf7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c7c15dbd0 0x7f3c7c14df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.457+0000 7f3c81bf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c15dfb0 0x7f3c7c14e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.457+0000 7f3c81bf7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0726e0 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.457+0000 7f3c81bf7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0721c0 con 0x7f3c7c15dbd0 2024-03-06T07:26:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.457+0000 7f3c81bf7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c072860 con 0x7f3c7c0b5840 2024-03-06T07:26:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.457+0000 7f3c7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c15dfb0 0x7f3c7c14e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c15dfb0 0x7f3c7c14e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34634/0 (socket says 172.21.15.27:34634) 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7bfff640 1 -- 172.21.15.27:0/911035366 learned_addr learned my addr 172.21.15.27:0/911035366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7affd640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c7c15dbd0 0x7f3c7c14df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7b7fe640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 0x7f3c7c157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7bfff640 1 -- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 msgr2=0x7f3c7c157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7bfff640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 0x7f3c7c157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7bfff640 1 -- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c7c15dbd0 msgr2=0x7f3c7c14df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7bfff640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c7c15dbd0 0x7f3c7c14df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7bfff640 1 -- 172.21.15.27:0/911035366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c15f8d0 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c7bfff640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c15dfb0 0x7f3c7c14e480 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c6e0 tx=0x7f3c7400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c63fff640 1 -- 172.21.15.27:0/911035366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74003b70 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c63fff640 1 -- 172.21.15.27:0/911035366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c74003d10 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c63fff640 1 -- 172.21.15.27:0/911035366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c740052d0 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c15fbb0 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.458+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c160170 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.459+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7c073380 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.460+0000 7f3c63fff640 1 -- 172.21.15.27:0/911035366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c74005470 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.460+0000 7f3c63fff640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c6806b4d0 0x7f3c6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.460+0000 7f3c63fff640 1 -- 172.21.15.27:0/911035366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3c74089d50 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.461+0000 7f3c7b7fe640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c6806b4d0 0x7f3c6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.461+0000 7f3c7b7fe640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c6806b4d0 0x7f3c6806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c001200 tx=0x7f3c6c001040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.465+0000 7f3c63fff640 1 -- 172.21.15.27:0/911035366 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c7408d030 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.650+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3c7c04f410 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.654+0000 7f3c63fff640 1 -- 172.21.15.27:0/911035366 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3c74058800 con 0x7f3c7c15dfb0 2024-03-06T07:26:59.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c6806b4d0 msgr2=0x7f3c6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c6806b4d0 0x7f3c6806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c001200 tx=0x7f3c6c001040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c15dfb0 msgr2=0x7f3c7c14e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c15dfb0 0x7f3c7c14e480 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c6e0 tx=0x7f3c7400cba0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 shutdown_connections 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c6806b4d0 0x7f3c6806d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7c15dfb0 0x7f3c7c14e480 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c7c15dbd0 0x7f3c7c14df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 --2- 172.21.15.27:0/911035366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c7c0b5840 0x7f3c7c157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 >> 172.21.15.27:0/911035366 conn(0x7f3c7c0b26b0 msgr2=0x7f3c7c161820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 shutdown_connections 2024-03-06T07:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.656+0000 7f3c81bf7640 1 -- 172.21.15.27:0/911035366 wait complete. 2024-03-06T07:26:59.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:26:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.782+0000 7f01f1a5c640 1 Processor -- start 2024-03-06T07:26:59.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.782+0000 7f01f1a5c640 1 -- start start 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01f1a5c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f01ec15d2c0 0x7f01ec15d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01f1a5c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec0d0a30 con 0x7f01ec0b4340 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01f1a5c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec0d01d0 con 0x7f01ec15fb60 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01f1a5c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec0cc5c0 con 0x7f01ec15d2c0 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01ea7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f01ec0b4340 0x7f01ec0b2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44608/0 (socket says 172.21.15.27:44608) 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01ea7fc640 1 -- 172.21.15.27:0/1523180906 learned_addr learned my addr 172.21.15.27:0/1523180906 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01eaffd640 1 --2- 172.21.15.27:0/1523180906 >> v2:172.21.15.69:3300/0 conn(0x7f01ec15d2c0 0x7f01ec15d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274362890 0 0) 0x7f01ec0d0a30 con 0x7f01ec0b4340 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01d4003520 con 0x7f01ec0b4340 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714613020 0 0) 0x7f01ec0d01d0 con 0x7f01ec15fb60 2024-03-06T07:26:59.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ec0d0a30 con 0x7f01ec15fb60 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1689837974 0 0) 0x7f01d4003520 con 0x7f01ec0b4340 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 >> v2:172.21.15.69:3300/0 conn(0x7f01ec15d2c0 msgr2=0x7f01ec15d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 --2- 172.21.15.27:0/1523180906 >> v2:172.21.15.69:3300/0 conn(0x7f01ec15d2c0 0x7f01ec15d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 >> v1:172.21.15.69:6789/0 conn(0x7f01ec15fb60 legacy=0x7f01ec15ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec0d0470 con 0x7f01ec0b4340 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f01e0001670 con 0x7f01ec0b4340 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 >> v1:172.21.15.27:6789/0 conn(0x7f01ec0b4340 legacy=0x7f01ec0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01d4003b40 0x7f01d4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01d40051c0 0x7f01d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0cc5c0 con 0x7f01d4004800 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d400c040 con 0x7f01d4003b40 2024-03-06T07:26:59.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.783+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d400c240 con 0x7f01d40051c0 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.784+0000 7f01ea7fc640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.784+0000 7f01eaffd640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01d4003b40 0x7f01d4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.784+0000 7f01eb7fe640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01d40051c0 0x7f01d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.784+0000 7f01ea7fc640 1 -- 172.21.15.27:0/1523180906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01d40051c0 msgr2=0x7f01d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.784+0000 7f01ea7fc640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01d40051c0 0x7f01d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.784+0000 7f01ea7fc640 1 -- 172.21.15.27:0/1523180906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01d4003b40 msgr2=0x7f01d4003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.784+0000 7f01ea7fc640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01d4003b40 0x7f01d4003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.784+0000 7f01ea7fc640 1 -- 172.21.15.27:0/1523180906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d4003520 con 0x7f01d4004800 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.784+0000 7f01eb7fe640 1 -- 172.21.15.27:0/1523180906 reap_dead start 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01ea7fc640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01d4004c80 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f01e000c190 tx=0x7f01d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce4091fca96c502 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0007ce0 con 0x7f01d4004800 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e0030070 con 0x7f01d4004800 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01e9ffb640 1 -- 172.21.15.27:0/1523180906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0005d90 con 0x7f01d4004800 2024-03-06T07:26:59.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1523180906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 msgr2=0x7f01d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01f1a5c640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01d4004c80 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f01e000c190 tx=0x7f01d4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:59.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1523180906 shutdown_connections 2024-03-06T07:26:59.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01f1a5c640 1 --2- 172.21.15.27:0/1523180906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01d4004c80 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1523180906 >> 172.21.15.27:0/1523180906 conn(0x7f01ec0b3660 msgr2=0x7f01ec0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1523180906 shutdown_connections 2024-03-06T07:26:59.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.785+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1523180906 wait complete. 2024-03-06T07:26:59.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.787+0000 7f01f1a5c640 1 Processor -- start 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.787+0000 7f01f1a5c640 1 -- start start 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.787+0000 7f01f1a5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01ec0b6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.787+0000 7f01f1a5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01ec156680 0x7f01ec0ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.787+0000 7f01f1a5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01ec156ae0 0x7f01ec06f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01f1a5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec15d9d0 con 0x7f01d4004800 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01f1a5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0cc860 con 0x7f01ec156680 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01f1a5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec15db50 con 0x7f01ec156ae0 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01ec0b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01ec0b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34654/0 (socket says 172.21.15.27:34654) 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eaffd640 1 -- 172.21.15.27:0/1992422616 learned_addr learned my addr 172.21.15.27:0/1992422616 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eb7fe640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01ec156ae0 0x7f01ec06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01ea7fc640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01ec156680 0x7f01ec0ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eaffd640 1 -- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01ec156ae0 msgr2=0x7f01ec06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eaffd640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01ec156ae0 0x7f01ec06f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eaffd640 1 -- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01ec156680 msgr2=0x7f01ec0ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eaffd640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01ec156680 0x7f01ec0ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eaffd640 1 -- 172.21.15.27:0/1992422616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec0b67d0 con 0x7f01d4004800 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01eaffd640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01ec0b6140 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0119c0 tx=0x7f01dc011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01d3fff640 1 -- 172.21.15.27:0/1992422616 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc00fb50 con 0x7f01d4004800 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01d3fff640 1 -- 172.21.15.27:0/1992422616 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01dc00fcf0 con 0x7f01d4004800 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec0c9760 con 0x7f01d4004800 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01d3fff640 1 -- 172.21.15.27:0/1992422616 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc00fea0 con 0x7f01d4004800 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.788+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec0c9a00 con 0x7f01d4004800 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.789+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec0b3440 con 0x7f01d4004800 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.790+0000 7f01d3fff640 1 -- 172.21.15.27:0/1992422616 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01dc012040 con 0x7f01d4004800 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.790+0000 7f01d3fff640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01d806b500 0x7f01d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.790+0000 7f01d3fff640 1 -- 172.21.15.27:0/1992422616 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f01dc08a400 con 0x7f01d4004800 2024-03-06T07:26:59.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.790+0000 7f01ea7fc640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01d806b500 0x7f01d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.791+0000 7f01ea7fc640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01d806b500 0x7f01d806d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f01e0034000 tx=0x7f01e0007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.795+0000 7f01d3fff640 1 -- 172.21.15.27:0/1992422616 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01dc019030 con 0x7f01d4004800 2024-03-06T07:26:59.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.975+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f01ec04f410 con 0x7f01d4004800 2024-03-06T07:26:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.978+0000 7f01d3fff640 1 -- 172.21.15.27:0/1992422616 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v25) v1 ==== 108+0+0 (secure 0 0 0) 0x7f01dc058fc0 con 0x7f01d4004800 2024-03-06T07:26:59.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01d806b500 msgr2=0x7f01d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01d806b500 0x7f01d806d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f01e0034000 tx=0x7f01e0007000 comp rx=0 tx=0).stop 2024-03-06T07:26:59.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 msgr2=0x7f01ec0b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01ec0b6140 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0119c0 tx=0x7f01dc011e80 comp rx=0 tx=0).stop 2024-03-06T07:26:59.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 shutdown_connections 2024-03-06T07:26:59.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01d806b500 0x7f01d806d9c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01ec156ae0 0x7f01ec06f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01ec156680 0x7f01ec0ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 --2- 172.21.15.27:0/1992422616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01d4004800 0x7f01ec0b6140 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 >> 172.21.15.27:0/1992422616 conn(0x7f01ec0b3660 msgr2=0x7f01ec15c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 shutdown_connections 2024-03-06T07:26:59.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:26:59.980+0000 7f01f1a5c640 1 -- 172.21.15.27:0/1992422616 wait complete. 2024-03-06T07:26:59.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:27:00.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.100+0000 7f1c4a3f6640 1 Processor -- start 2024-03-06T07:27:00.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.100+0000 7f1c4a3f6640 1 -- start start 2024-03-06T07:27:00.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c4a3f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4415e060 0x7f1c440ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c4a3f6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c4406fe20 0x7f1c440ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c4a3f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440d0dd0 con 0x7f1c440d0880 2024-03-06T07:27:00.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c4a3f6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c440d10b0 con 0x7f1c4406fe20 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c4a3f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440cc0e0 con 0x7f1c4415e060 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c437fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c4406fe20 0x7f1c440ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c437fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c4406fe20 0x7f1c440ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34672/0 (socket says 172.21.15.27:34672) 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c437fe640 1 -- 172.21.15.27:0/1685429411 learned_addr learned my addr 172.21.15.27:0/1685429411 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c43fff640 1 --2- 172.21.15.27:0/1685429411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4415e060 0x7f1c440ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c437fe640 1 -- 172.21.15.27:0/1685429411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4415e060 msgr2=0x7f1c440ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c437fe640 1 --2- 172.21.15.27:0/1685429411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4415e060 0x7f1c440ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c437fe640 1 -- 172.21.15.27:0/1685429411 >> v1:172.21.15.69:6789/0 conn(0x7f1c440d0880 legacy=0x7f1c4406ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.101+0000 7f1c437fe640 1 -- 172.21.15.27:0/1685429411 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c440cc3c0 con 0x7f1c4406fe20 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.102+0000 7f1c437fe640 1 --2- 172.21.15.27:0/1685429411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c4406fe20 0x7f1c440ce850 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009870 tx=0x7f1c3802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456edac94c0166eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.102+0000 7f1c42ffd640 1 -- 172.21.15.27:0/1685429411 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38035020 con 0x7f1c4406fe20 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.102+0000 7f1c42ffd640 1 -- 172.21.15.27:0/1685429411 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c38031640 con 0x7f1c4406fe20 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.102+0000 7f1c42ffd640 1 -- 172.21.15.27:0/1685429411 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3802d040 con 0x7f1c4406fe20 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.102+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/1685429411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c4406fe20 msgr2=0x7f1c440ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.102+0000 7f1c4a3f6640 1 --2- 172.21.15.27:0/1685429411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c4406fe20 0x7f1c440ce850 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009870 tx=0x7f1c3802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.103+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/1685429411 shutdown_connections 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.103+0000 7f1c4a3f6640 1 --2- 172.21.15.27:0/1685429411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c4406fe20 0x7f1c440ce850 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.103+0000 7f1c4a3f6640 1 --2- 172.21.15.27:0/1685429411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4415e060 0x7f1c440ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.103+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/1685429411 >> 172.21.15.27:0/1685429411 conn(0x7f1c440d1a90 msgr2=0x7f1c440d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.103+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/1685429411 shutdown_connections 2024-03-06T07:27:00.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.103+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/1685429411 wait complete. 2024-03-06T07:27:00.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.104+0000 7f1c4a3f6640 1 Processor -- start 2024-03-06T07:27:00.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.104+0000 7f1c4a3f6640 1 -- start start 2024-03-06T07:27:00.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c4a3f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4406fe20 0x7f1c440b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c4a3f6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c440d0880 0x7f1c4415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c4a3f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4415e060 0x7f1c440b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c4a3f6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440cdbd0 con 0x7f1c440d0880 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c4a3f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4415e440 con 0x7f1c4415e060 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c4a3f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440cdd50 con 0x7f1c4406fe20 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c440d0880 0x7f1c4415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c440d0880 0x7f1c4415e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34674/0 (socket says 172.21.15.27:34674) 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c437fe640 1 -- 172.21.15.27:0/3916970522 learned_addr learned my addr 172.21.15.27:0/3916970522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c48993640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4415e060 0x7f1c440b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c43fff640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4406fe20 0x7f1c440b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c437fe640 1 -- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4406fe20 msgr2=0x7f1c440b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c437fe640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4406fe20 0x7f1c440b2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c437fe640 1 -- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4415e060 msgr2=0x7f1c440b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c437fe640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4415e060 0x7f1c440b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c437fe640 1 -- 172.21.15.27:0/3916970522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c440b5010 con 0x7f1c440d0880 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c48993640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4415e060 0x7f1c440b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c437fe640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c440d0880 0x7f1c4415e5c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1c3802b930 tx=0x7f1c380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c40ff9640 1 -- 172.21.15.27:0/3916970522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38035040 con 0x7f1c440d0880 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c40ff9640 1 -- 172.21.15.27:0/3916970522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c38039bf0 con 0x7f1c440d0880 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c40ff9640 1 -- 172.21.15.27:0/3916970522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38045480 con 0x7f1c440d0880 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.105+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c440b52f0 con 0x7f1c440d0880 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.106+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c440b58b0 con 0x7f1c440d0880 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.107+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c44067ff0 con 0x7f1c440d0880 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.107+0000 7f1c40ff9640 1 -- 172.21.15.27:0/3916970522 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c38038020 con 0x7f1c440d0880 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.107+0000 7f1c40ff9640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c3006b4d0 0x7f1c3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.108+0000 7f1c40ff9640 1 -- 172.21.15.27:0/3916970522 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1c380abdb0 con 0x7f1c440d0880 2024-03-06T07:27:00.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.108+0000 7f1c43fff640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c3006b4d0 0x7f1c3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.108+0000 7f1c43fff640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c3006b4d0 0x7f1c3006d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1c34006fd0 tx=0x7f1c34008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.112+0000 7f1c40ff9640 1 -- 172.21.15.27:0/3916970522 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c38043380 con 0x7f1c440d0880 2024-03-06T07:27:00.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:00.365+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f1c440d2130 con 0x7f1c440d0880 2024-03-06T07:27:01.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.325+0000 7f1c40ff9640 1 -- 172.21.15.27:0/3916970522 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v246) v1 ==== 140+0+0 (secure 0 0 0) 0x7f1c3807afc0 con 0x7f1c440d0880 2024-03-06T07:27:01.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c3006b4d0 msgr2=0x7f1c3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c3006b4d0 0x7f1c3006d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1c34006fd0 tx=0x7f1c34008040 comp rx=0 tx=0).stop 2024-03-06T07:27:01.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c440d0880 msgr2=0x7f1c4415e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c440d0880 0x7f1c4415e5c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1c3802b930 tx=0x7f1c380046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 shutdown_connections 2024-03-06T07:27:01.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c3006b4d0 0x7f1c3006d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c4415e060 0x7f1c440b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c440d0880 0x7f1c4415e5c0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 --2- 172.21.15.27:0/3916970522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1c4406fe20 0x7f1c440b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 >> 172.21.15.27:0/3916970522 conn(0x7f1c440d1a90 msgr2=0x7f1c4415d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 shutdown_connections 2024-03-06T07:27:01.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.329+0000 7f1c4a3f6640 1 -- 172.21.15.27:0/3916970522 wait complete. 2024-03-06T07:27:01.344 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:27:01.344 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:27:01.345 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:27:01.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.456+0000 7fd834abb640 1 Processor -- start 2024-03-06T07:27:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.456+0000 7fd834abb640 1 -- start start 2024-03-06T07:27:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.457+0000 7fd834abb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd830157f60 0x7fd83015b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.457+0000 7fd834abb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8300cbab0 0x7fd830070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.457+0000 7fd834abb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83006c7b0 con 0x7fd830157b80 2024-03-06T07:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.457+0000 7fd834abb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83006ca90 con 0x7fd830157f60 2024-03-06T07:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.457+0000 7fd834abb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83006cd70 con 0x7fd8300cbab0 2024-03-06T07:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.457+0000 7fd82dd9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd830157f60 0x7fd83015b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.457+0000 7fd82ed9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8300cbab0 0x7fd830070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.457+0000 7fd82e59c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd830157b80 0x7fd83015b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:53610/0 (socket says 172.21.15.27:53610) 2024-03-06T07:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.457+0000 7fd82e59c640 1 -- 172.21.15.27:0/1658082355 learned_addr learned my addr 172.21.15.27:0/1658082355 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.458+0000 7fd82dd9b640 1 -- 172.21.15.27:0/1658082355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8300cbab0 msgr2=0x7fd830070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.458+0000 7fd82dd9b640 1 --2- 172.21.15.27:0/1658082355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8300cbab0 0x7fd830070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.458+0000 7fd82dd9b640 1 -- 172.21.15.27:0/1658082355 >> v1:172.21.15.69:6789/0 conn(0x7fd830157b80 legacy=0x7fd83015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.458+0000 7fd82dd9b640 1 -- 172.21.15.27:0/1658082355 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83006d050 con 0x7fd830157f60 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.458+0000 7fd82ed9d640 1 --2- 172.21.15.27:0/1658082355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8300cbab0 0x7fd830070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.458+0000 7fd82dd9b640 1 --2- 172.21.15.27:0/1658082355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 0x7fd83015b260 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd82400ba40 tx=0x7fd82402e080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e7a869fe2502159 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd82d59a640 1 -- 172.21.15.27:0/1658082355 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82403b070 con 0x7fd830157f60 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd82d59a640 1 -- 172.21.15.27:0/1658082355 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82402ee30 con 0x7fd830157f60 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd82d59a640 1 -- 172.21.15.27:0/1658082355 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824036640 con 0x7fd830157f60 2024-03-06T07:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd834abb640 1 -- 172.21.15.27:0/1658082355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 msgr2=0x7fd83015b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd834abb640 1 --2- 172.21.15.27:0/1658082355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 0x7fd83015b260 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd82400ba40 tx=0x7fd82402e080 comp rx=0 tx=0).stop 2024-03-06T07:27:01.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd834abb640 1 -- 172.21.15.27:0/1658082355 shutdown_connections 2024-03-06T07:27:01.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd834abb640 1 --2- 172.21.15.27:0/1658082355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 0x7fd83015b260 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd834abb640 1 --2- 172.21.15.27:0/1658082355 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd8300cbab0 0x7fd830070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd834abb640 1 -- 172.21.15.27:0/1658082355 >> 172.21.15.27:0/1658082355 conn(0x7fd830066d00 msgr2=0x7fd8300670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd834abb640 1 -- 172.21.15.27:0/1658082355 shutdown_connections 2024-03-06T07:27:01.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.459+0000 7fd834abb640 1 -- 172.21.15.27:0/1658082355 wait complete. 2024-03-06T07:27:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.461+0000 7fd834abb640 1 Processor -- start 2024-03-06T07:27:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.461+0000 7fd834abb640 1 -- start start 2024-03-06T07:27:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.461+0000 7fd834abb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8300cbab0 0x7fd830147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.461+0000 7fd834abb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd830157b80 0x7fd8300cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.461+0000 7fd834abb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 0x7fd8300ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.461+0000 7fd834abb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd830071200 con 0x7fd8300cbab0 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.461+0000 7fd834abb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd830071080 con 0x7fd830157f60 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.461+0000 7fd834abb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd830071380 con 0x7fd830157b80 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8300cbab0 0x7fd830147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8300cbab0 0x7fd830147330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34680/0 (socket says 172.21.15.27:34680) 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82e59c640 1 -- 172.21.15.27:0/3215212427 learned_addr learned my addr 172.21.15.27:0/3215212427 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82ed9d640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 0x7fd8300ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82dd9b640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd830157b80 0x7fd8300cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82e59c640 1 -- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd830157b80 msgr2=0x7fd8300cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82e59c640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd830157b80 0x7fd8300cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82e59c640 1 -- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 msgr2=0x7fd8300ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82e59c640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 0x7fd8300ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82e59c640 1 -- 172.21.15.27:0/3215212427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd830147870 con 0x7fd8300cbab0 2024-03-06T07:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82ed9d640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 0x7fd8300ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd82e59c640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8300cbab0 0x7fd830147330 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd82000c400 tx=0x7fd82000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd81effd640 1 -- 172.21.15.27:0/3215212427 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820017070 con 0x7fd8300cbab0 2024-03-06T07:27:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd81effd640 1 -- 172.21.15.27:0/3215212427 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd820005ce0 con 0x7fd8300cbab0 2024-03-06T07:27:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd81effd640 1 -- 172.21.15.27:0/3215212427 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8200072f0 con 0x7fd8300cbab0 2024-03-06T07:27:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8300cd190 con 0x7fd8300cbab0 2024-03-06T07:27:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.462+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8300cd750 con 0x7fd8300cbab0 2024-03-06T07:27:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.463+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd830067ff0 con 0x7fd8300cbab0 2024-03-06T07:27:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.464+0000 7fd81effd640 1 -- 172.21.15.27:0/3215212427 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd82001a070 con 0x7fd8300cbab0 2024-03-06T07:27:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.464+0000 7fd81effd640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b500 0x7fd81006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.465+0000 7fd81effd640 1 -- 172.21.15.27:0/3215212427 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd8200891c0 con 0x7fd8300cbab0 2024-03-06T07:27:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.465+0000 7fd82dd9b640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b500 0x7fd81006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.465+0000 7fd82dd9b640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b500 0x7fd81006d9c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd824038000 tx=0x7fd824008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.469+0000 7fd81effd640 1 -- 172.21.15.27:0/3215212427 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8200583d0 con 0x7fd8300cbab0 2024-03-06T07:27:01.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.651+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd830159aa0 con 0x7fd8300cbab0 2024-03-06T07:27:01.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.651+0000 7fd81effd640 1 -- 172.21.15.27:0/3215212427 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+12949 (secure 0 0 0) 0x7fd820057cd0 con 0x7fd8300cbab0 2024-03-06T07:27:01.653 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:01.653 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:26:59.316091+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/2393218771":"2024-03-07T07:26:56.267155+0000","172.21.15.27:6833/2393218771":"2024-03-07T07:26:56.267155+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:01.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b500 msgr2=0x7fd81006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b500 0x7fd81006d9c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd824038000 tx=0x7fd824008040 comp rx=0 tx=0).stop 2024-03-06T07:27:01.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8300cbab0 msgr2=0x7fd830147330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8300cbab0 0x7fd830147330 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd82000c400 tx=0x7fd82000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 shutdown_connections 2024-03-06T07:27:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b500 0x7fd81006d9c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd830157f60 0x7fd8300ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd830157b80 0x7fd8300cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 --2- 172.21.15.27:0/3215212427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8300cbab0 0x7fd830147330 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 >> 172.21.15.27:0/3215212427 conn(0x7fd830066d00 msgr2=0x7fd83015a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 shutdown_connections 2024-03-06T07:27:01.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.653+0000 7fd834abb640 1 -- 172.21.15.27:0/3215212427 wait complete. 2024-03-06T07:27:01.666 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:01.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:01.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.775+0000 7f97eef55640 1 Processor -- start 2024-03-06T07:27:01.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.775+0000 7f97eef55640 1 -- start start 2024-03-06T07:27:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97eef55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 0x7f97e815ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97eef55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f97e80b5840 0x7f97e80b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97eef55640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e815f3b0 con 0x7f97e815dfb0 2024-03-06T07:27:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97eef55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e806f070 con 0x7f97e80b5840 2024-03-06T07:27:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97eef55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e8071940 con 0x7f97e815dbd0 2024-03-06T07:27:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97ed4f2640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f97e815dfb0 0x7f97e8161a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44610/0 (socket says 172.21.15.27:44610) 2024-03-06T07:27:01.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97ed4f2640 1 -- 172.21.15.27:0/360075728 learned_addr learned my addr 172.21.15.27:0/360075728 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:01.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97e7fff640 1 --2- 172.21.15.27:0/360075728 >> v2:172.21.15.69:3300/0 conn(0x7f97e80b5840 0x7f97e80b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97eccf1640 1 --2- 172.21.15.27:0/360075728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 0x7f97e815ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97e77fe640 1 -- 172.21.15.27:0/360075728 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731626204 0 0) 0x7f97e815f3b0 con 0x7f97e815dfb0 2024-03-06T07:27:01.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.776+0000 7f97e77fe640 1 -- 172.21.15.27:0/360075728 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d4003520 con 0x7f97e815dfb0 2024-03-06T07:27:01.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.777+0000 7f97e7fff640 1 -- 172.21.15.27:0/360075728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 msgr2=0x7f97e815ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.777+0000 7f97e7fff640 1 --2- 172.21.15.27:0/360075728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 0x7f97e815ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.777+0000 7f97e7fff640 1 -- 172.21.15.27:0/360075728 >> v1:172.21.15.27:6789/0 conn(0x7f97e815dfb0 legacy=0x7f97e8161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.777+0000 7f97e7fff640 1 -- 172.21.15.27:0/360075728 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e80b5db0 con 0x7f97e80b5840 2024-03-06T07:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.777+0000 7f97e7fff640 1 --2- 172.21.15.27:0/360075728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97e80b5840 0x7f97e80b39b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009d90 tx=0x7f97d802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=877945623792d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97e77fe640 1 -- 172.21.15.27:0/360075728 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8039070 con 0x7f97e80b5840 2024-03-06T07:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97e77fe640 1 -- 172.21.15.27:0/360075728 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d802cc50 con 0x7f97e80b5840 2024-03-06T07:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97e77fe640 1 -- 172.21.15.27:0/360075728 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8034440 con 0x7f97e80b5840 2024-03-06T07:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97eef55640 1 -- 172.21.15.27:0/360075728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97e80b5840 msgr2=0x7f97e80b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97eef55640 1 --2- 172.21.15.27:0/360075728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97e80b5840 0x7f97e80b39b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009d90 tx=0x7f97d802bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97eef55640 1 -- 172.21.15.27:0/360075728 shutdown_connections 2024-03-06T07:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97eef55640 1 --2- 172.21.15.27:0/360075728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97e80b5840 0x7f97e80b39b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97eef55640 1 --2- 172.21.15.27:0/360075728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 0x7f97e815ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97eef55640 1 -- 172.21.15.27:0/360075728 >> 172.21.15.27:0/360075728 conn(0x7f97e80b26b0 msgr2=0x7f97e80b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97eef55640 1 -- 172.21.15.27:0/360075728 shutdown_connections 2024-03-06T07:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.778+0000 7f97eef55640 1 -- 172.21.15.27:0/360075728 wait complete. 2024-03-06T07:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.779+0000 7f97eef55640 1 Processor -- start 2024-03-06T07:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.779+0000 7f97eef55640 1 -- start start 2024-03-06T07:27:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.780+0000 7f97eef55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97e80b5840 0x7f97e8155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.780+0000 7f97eef55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 0x7f97e80ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.780+0000 7f97eef55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e815dfb0 0x7f97e8154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.780+0000 7f97eef55640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e80721c0 con 0x7f97e815dfb0 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.780+0000 7f97eef55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e8071ee0 con 0x7f97e80b5840 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.780+0000 7f97eef55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e80726e0 con 0x7f97e815dbd0 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.780+0000 7f97ed4f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e815dfb0 0x7f97e8154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97ed4f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e815dfb0 0x7f97e8154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34684/0 (socket says 172.21.15.27:34684) 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97ed4f2640 1 -- 172.21.15.27:0/4212215166 learned_addr learned my addr 172.21.15.27:0/4212215166 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97e7fff640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 0x7f97e80ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97eccf1640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97e80b5840 0x7f97e8155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97ed4f2640 1 -- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 msgr2=0x7f97e80ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97ed4f2640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 0x7f97e80ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97ed4f2640 1 -- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97e80b5840 msgr2=0x7f97e8155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97ed4f2640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97e80b5840 0x7f97e8155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97ed4f2640 1 -- 172.21.15.27:0/4212215166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e8156680 con 0x7f97e815dfb0 2024-03-06T07:27:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97ed4f2640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e815dfb0 0x7f97e8154b40 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c390 tx=0x7f97d000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97e57fa640 1 -- 172.21.15.27:0/4212215166 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0017070 con 0x7f97e815dfb0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97e57fa640 1 -- 172.21.15.27:0/4212215166 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d0005ce0 con 0x7f97e815dfb0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97e57fa640 1 -- 172.21.15.27:0/4212215166 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0007310 con 0x7f97e815dfb0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e8156960 con 0x7f97e815dfb0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.781+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e8156f20 con 0x7f97e815dfb0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.782+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e8073380 con 0x7f97e815dfb0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.783+0000 7f97e57fa640 1 -- 172.21.15.27:0/4212215166 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97d001a070 con 0x7f97e815dfb0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.783+0000 7f97e57fa640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f97c806b4d0 0x7f97c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.783+0000 7f97e57fa640 1 -- 172.21.15.27:0/4212215166 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f97d00891e0 con 0x7f97e815dfb0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.783+0000 7f97eccf1640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f97c806b4d0 0x7f97c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.784+0000 7f97eccf1640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f97c806b4d0 0x7f97c806d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009b40 tx=0x7f97e0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.787+0000 7f97e57fa640 1 -- 172.21.15.27:0/4212215166 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97d0057da0 con 0x7f97e815dfb0 2024-03-06T07:27:01.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.967+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f97e804f410 con 0x7f97e815dfb0 2024-03-06T07:27:01.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.968+0000 7f97e57fa640 1 -- 172.21.15.27:0/4212215166 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97d005e0f0 con 0x7f97e815dfb0 2024-03-06T07:27:01.969 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:27:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f97c806b4d0 msgr2=0x7f97c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f97c806b4d0 0x7f97c806d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009b40 tx=0x7f97e0009190 comp rx=0 tx=0).stop 2024-03-06T07:27:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e815dfb0 msgr2=0x7f97e8154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e815dfb0 0x7f97e8154b40 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c390 tx=0x7f97d000c850 comp rx=0 tx=0).stop 2024-03-06T07:27:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 shutdown_connections 2024-03-06T07:27:01.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f97c806b4d0 0x7f97c806d990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e815dfb0 0x7f97e8154b40 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f97e815dbd0 0x7f97e80ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 --2- 172.21.15.27:0/4212215166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97e80b5840 0x7f97e8155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.969+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 >> 172.21.15.27:0/4212215166 conn(0x7f97e80b26b0 msgr2=0x7f97e8160c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.970+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 shutdown_connections 2024-03-06T07:27:01.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:01.970+0000 7f97eef55640 1 -- 172.21.15.27:0/4212215166 wait complete. 2024-03-06T07:27:01.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:27:02.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.090+0000 7f8e8133a640 1 Processor -- start 2024-03-06T07:27:02.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.090+0000 7f8e8133a640 1 -- start start 2024-03-06T07:27:02.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.091+0000 7f8e8133a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0d02b0 0x7f8e7c166cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.091+0000 7f8e8133a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c0cc980 con 0x7f8e7c15bf30 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.091+0000 7f8e8133a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c0ccc60 con 0x7f8e7c0cd740 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.091+0000 7f8e8133a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c15c310 con 0x7f8e7c0d02b0 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.091+0000 7f8e7affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8e7c15bf30 0x7f8e7c0cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44616/0 (socket says 172.21.15.27:44616) 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.091+0000 7f8e7affd640 1 -- 172.21.15.27:0/145926503 learned_addr learned my addr 172.21.15.27:0/145926503 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531912488 0 0) 0x7f8e7c0cc980 con 0x7f8e7c15bf30 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7b7fe640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0d02b0 0x7f8e7c166cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e68003520 con 0x7f8e7c15bf30 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2606185617 0 0) 0x7f8e68003520 con 0x7f8e7c15bf30 2024-03-06T07:27:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0d02b0 msgr2=0x7f8e7c166cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0d02b0 0x7f8e7c166cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 >> v1:172.21.15.69:6789/0 conn(0x7f8e7c0cd740 legacy=0x7f8e7c0cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c0ccf40 con 0x7f8e7c15bf30 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f8e74001670 con 0x7f8e7c15bf30 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 >> v1:172.21.15.27:6789/0 conn(0x7f8e7c15bf30 legacy=0x7f8e7c0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e68003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0d02b0 0x7f8e68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e68004d40 0x7f8e680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c15c310 con 0x7f8e680039e0 2024-03-06T07:27:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6800b9f0 con 0x7f8e68004d40 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6800bbf0 con 0x7f8e7c0d02b0 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7affd640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e68003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7b7fe640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e68004d40 0x7f8e680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7a7fc640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0d02b0 0x7f8e68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7affd640 1 -- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0d02b0 msgr2=0x7f8e68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7affd640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0d02b0 0x7f8e68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7affd640 1 -- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e68004d40 msgr2=0x7f8e680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7affd640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e68004d40 0x7f8e680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7affd640 1 -- 172.21.15.27:0/145926503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e68003520 con 0x7f8e680039e0 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.092+0000 7f8e7a7fc640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0d02b0 0x7f8e68004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.093+0000 7f8e7affd640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e68003dc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400c190 tx=0x7f8e68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4d03e483416a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.093+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74007cc0 con 0x7f8e680039e0 2024-03-06T07:27:02.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.093+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e74030070 con 0x7f8e680039e0 2024-03-06T07:27:02.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.093+0000 7f8e79ffb640 1 -- 172.21.15.27:0/145926503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74005d10 con 0x7f8e680039e0 2024-03-06T07:27:02.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.094+0000 7f8e8133a640 1 -- 172.21.15.27:0/145926503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 msgr2=0x7f8e68003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.094+0000 7f8e8133a640 1 --2- 172.21.15.27:0/145926503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e68003dc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400c190 tx=0x7f8e68005830 comp rx=0 tx=0).stop 2024-03-06T07:27:02.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.094+0000 7f8e7b7fe640 1 -- 172.21.15.27:0/145926503 reap_dead start 2024-03-06T07:27:02.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.094+0000 7f8e8133a640 1 -- 172.21.15.27:0/145926503 shutdown_connections 2024-03-06T07:27:02.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.094+0000 7f8e8133a640 1 -- 172.21.15.27:0/145926503 >> 172.21.15.27:0/145926503 conn(0x7f8e7c0b3670 msgr2=0x7f8e7c0b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.094+0000 7f8e8133a640 1 -- 172.21.15.27:0/145926503 shutdown_connections 2024-03-06T07:27:02.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.094+0000 7f8e8133a640 1 -- 172.21.15.27:0/145926503 wait complete. 2024-03-06T07:27:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.095+0000 7f8e8133a640 1 Processor -- start 2024-03-06T07:27:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.095+0000 7f8e8133a640 1 -- start start 2024-03-06T07:27:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e8133a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e7c1501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e8133a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0afc70 0x7f8e7c0b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e8133a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7c0c7210 0x7f8e7c0c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e8133a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0cd2b0 con 0x7f8e680039e0 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e8133a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0cf3c0 con 0x7f8e7c0c7210 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e8133a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0b4290 con 0x7f8e7c0afc70 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e7c1501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e7c1501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34698/0 (socket says 172.21.15.27:34698) 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7affd640 1 -- 172.21.15.27:0/3731542698 learned_addr learned my addr 172.21.15.27:0/3731542698 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7a7fc640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0afc70 0x7f8e7c0b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7b7fe640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7c0c7210 0x7f8e7c0c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7affd640 1 -- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0afc70 msgr2=0x7f8e7c0b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7affd640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0afc70 0x7f8e7c0b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7affd640 1 -- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7c0c7210 msgr2=0x7f8e7c0c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7affd640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7c0c7210 0x7f8e7c0c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.096+0000 7f8e7affd640 1 -- 172.21.15.27:0/3731542698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c0c8540 con 0x7f8e680039e0 2024-03-06T07:27:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.097+0000 7f8e7affd640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e7c1501a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f8e740018f0 tx=0x7f8e740019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.097+0000 7f8e737fe640 1 -- 172.21.15.27:0/3731542698 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74007cc0 con 0x7f8e680039e0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.097+0000 7f8e737fe640 1 -- 172.21.15.27:0/3731542698 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e74007480 con 0x7f8e680039e0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.097+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7c0c7f80 con 0x7f8e680039e0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.097+0000 7f8e737fe640 1 -- 172.21.15.27:0/3731542698 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7402c720 con 0x7f8e680039e0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.097+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7c0b46c0 con 0x7f8e680039e0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.098+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7c068700 con 0x7f8e680039e0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.098+0000 7f8e737fe640 1 -- 172.21.15.27:0/3731542698 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e7402c8c0 con 0x7f8e680039e0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.099+0000 7f8e737fe640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e5c06b4d0 0x7f8e5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.099+0000 7f8e737fe640 1 -- 172.21.15.27:0/3731542698 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8e740ab300 con 0x7f8e680039e0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.099+0000 7f8e7a7fc640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e5c06b4d0 0x7f8e5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.099+0000 7f8e7a7fc640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e5c06b4d0 0x7f8e5c06d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8e64007fd0 tx=0x7f8e64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.103+0000 7f8e737fe640 1 -- 172.21.15.27:0/3731542698 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e7407a540 con 0x7f8e680039e0 2024-03-06T07:27:02.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:02.354+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f8e7c0b12a0 con 0x7f8e680039e0 2024-03-06T07:27:03.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.333+0000 7f8e737fe640 1 -- 172.21.15.27:0/3731542698 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v248) v1 ==== 135+0+0 (secure 0 0 0) 0x7f8e74079e40 con 0x7f8e680039e0 2024-03-06T07:27:03.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e5c06b4d0 msgr2=0x7f8e5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e5c06b4d0 0x7f8e5c06d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8e64007fd0 tx=0x7f8e64009040 comp rx=0 tx=0).stop 2024-03-06T07:27:03.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 msgr2=0x7f8e7c1501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e7c1501a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f8e740018f0 tx=0x7f8e740019c0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 shutdown_connections 2024-03-06T07:27:03.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8e5c06b4d0 0x7f8e5c06d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7c0c7210 0x7f8e7c0c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e7c0afc70 0x7f8e7c0b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 --2- 172.21.15.27:0/3731542698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e680039e0 0x7f8e7c1501a0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 >> 172.21.15.27:0/3731542698 conn(0x7f8e7c0b3670 msgr2=0x7f8e7c06f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 shutdown_connections 2024-03-06T07:27:03.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.336+0000 7f8e8133a640 1 -- 172.21.15.27:0/3731542698 wait complete. 2024-03-06T07:27:03.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:27:03.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.468+0000 7f0c6d4a4640 1 Processor -- start 2024-03-06T07:27:03.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.468+0000 7f0c6d4a4640 1 -- start start 2024-03-06T07:27:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c6d4a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0c6815dbd0 0x7f0c6815ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c6d4a4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c680b5840 0x7f0c680b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c6d4a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 0x7f0c680b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c6d4a4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6815f3b0 con 0x7f0c680b5840 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c6d4a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6806f070 con 0x7f0c6815dbd0 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c6d4a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c68071940 con 0x7f0c6815dfb0 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c5ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c680b5840 0x7f0c680b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c5ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c680b5840 0x7f0c680b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34708/0 (socket says 172.21.15.27:34708) 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c5ffff640 1 -- 172.21.15.27:0/823697783 learned_addr learned my addr 172.21.15.27:0/823697783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c66ffd640 1 --2- 172.21.15.27:0/823697783 >> v2:172.21.15.69:3300/0 conn(0x7f0c6815dbd0 0x7f0c6815ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c677fe640 1 --2- 172.21.15.27:0/823697783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 0x7f0c680b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c5ffff640 1 -- 172.21.15.27:0/823697783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 msgr2=0x7f0c680b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c5ffff640 1 --2- 172.21.15.27:0/823697783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 0x7f0c680b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c5ffff640 1 -- 172.21.15.27:0/823697783 >> v2:172.21.15.69:3300/0 conn(0x7f0c6815dbd0 msgr2=0x7f0c6815ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c5ffff640 1 --2- 172.21.15.27:0/823697783 >> v2:172.21.15.69:3300/0 conn(0x7f0c6815dbd0 0x7f0c6815ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c5ffff640 1 -- 172.21.15.27:0/823697783 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c680b5db0 con 0x7f0c680b5840 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.469+0000 7f0c677fe640 1 --2- 172.21.15.27:0/823697783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 0x7f0c680b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.470+0000 7f0c5ffff640 1 --2- 172.21.15.27:0/823697783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c680b5840 0x7f0c680b39b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0c680b36f0 tx=0x7f0c5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed793ab92e1b6a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.470+0000 7f0c667fc640 1 -- 172.21.15.27:0/823697783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50035020 con 0x7f0c680b5840 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.470+0000 7f0c667fc640 1 -- 172.21.15.27:0/823697783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c50031870 con 0x7f0c680b5840 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.470+0000 7f0c667fc640 1 -- 172.21.15.27:0/823697783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50031c40 con 0x7f0c680b5840 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.471+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/823697783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c680b5840 msgr2=0x7f0c680b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.471+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/823697783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c680b5840 0x7f0c680b39b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0c680b36f0 tx=0x7f0c5002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.471+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/823697783 shutdown_connections 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.471+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/823697783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 0x7f0c680b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.471+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/823697783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c680b5840 0x7f0c680b39b0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.471+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/823697783 >> v2:172.21.15.69:3300/0 conn(0x7f0c6815dbd0 0x7f0c6815ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.471+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/823697783 >> 172.21.15.27:0/823697783 conn(0x7f0c680b26b0 msgr2=0x7f0c680b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.471+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/823697783 shutdown_connections 2024-03-06T07:27:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.471+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/823697783 wait complete. 2024-03-06T07:27:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.472+0000 7f0c6d4a4640 1 Processor -- start 2024-03-06T07:27:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.472+0000 7f0c6d4a4640 1 -- start start 2024-03-06T07:27:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c6d4a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c680b5840 0x7f0c681535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c6d4a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6815dbd0 0x7f0c680b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c6d4a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 0x7f0c6814f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c6d4a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c680721c0 con 0x7f0c6815dbd0 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c6d4a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c68071ee0 con 0x7f0c680b5840 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c6d4a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c680726e0 con 0x7f0c6815dfb0 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6815dbd0 0x7f0c680b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6815dbd0 0x7f0c680b1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34718/0 (socket says 172.21.15.27:34718) 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c5ffff640 1 -- 172.21.15.27:0/3013759270 learned_addr learned my addr 172.21.15.27:0/3013759270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c66ffd640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c680b5840 0x7f0c681535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c677fe640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 0x7f0c6814f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c5ffff640 1 -- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 msgr2=0x7f0c6814f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c5ffff640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 0x7f0c6814f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c5ffff640 1 -- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c680b5840 msgr2=0x7f0c681535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c5ffff640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c680b5840 0x7f0c681535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c5ffff640 1 -- 172.21.15.27:0/3013759270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c680b1b00 con 0x7f0c6815dbd0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.473+0000 7f0c5ffff640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6815dbd0 0x7f0c680b1490 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0c5002b950 tx=0x7f0c500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.474+0000 7f0c5f7fe640 1 -- 172.21.15.27:0/3013759270 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50040070 con 0x7f0c6815dbd0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.474+0000 7f0c5f7fe640 1 -- 172.21.15.27:0/3013759270 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c50030d00 con 0x7f0c6815dbd0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.474+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6815c6a0 con 0x7f0c6815dbd0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.474+0000 7f0c5f7fe640 1 -- 172.21.15.27:0/3013759270 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50035040 con 0x7f0c6815dbd0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.474+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6815cc60 con 0x7f0c6815dbd0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.475+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c68074980 con 0x7f0c6815dbd0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.475+0000 7f0c5f7fe640 1 -- 172.21.15.27:0/3013759270 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c5004b060 con 0x7f0c6815dbd0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.475+0000 7f0c5f7fe640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c4806b500 0x7f0c4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.476+0000 7f0c5f7fe640 1 -- 172.21.15.27:0/3013759270 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0c500abaf0 con 0x7f0c6815dbd0 2024-03-06T07:27:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.476+0000 7f0c66ffd640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c4806b500 0x7f0c4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.476+0000 7f0c66ffd640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c4806b500 0x7f0c4806d9c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0c580079a0 tx=0x7f0c58008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.480+0000 7f0c5f7fe640 1 -- 172.21.15.27:0/3013759270 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c68074980 con 0x7f0c6815dbd0 2024-03-06T07:27:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.665+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0c6815cea0 con 0x7f0c6815dbd0 2024-03-06T07:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.670+0000 7f0c5f7fe640 1 -- 172.21.15.27:0/3013759270 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0c5007ad00 con 0x7f0c6815dbd0 2024-03-06T07:27:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c4806b500 msgr2=0x7f0c4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c4806b500 0x7f0c4806d9c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0c580079a0 tx=0x7f0c58008040 comp rx=0 tx=0).stop 2024-03-06T07:27:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6815dbd0 msgr2=0x7f0c680b1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6815dbd0 0x7f0c680b1490 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0c5002b950 tx=0x7f0c500046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 shutdown_connections 2024-03-06T07:27:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c4806b500 0x7f0c4806d9c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c6815dfb0 0x7f0c6814f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6815dbd0 0x7f0c680b1490 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 --2- 172.21.15.27:0/3013759270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c680b5840 0x7f0c681535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 >> 172.21.15.27:0/3013759270 conn(0x7f0c680b26b0 msgr2=0x7f0c6815e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 shutdown_connections 2024-03-06T07:27:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.673+0000 7f0c6d4a4640 1 -- 172.21.15.27:0/3013759270 wait complete. 2024-03-06T07:27:03.685 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:03.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:03.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:03.775 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:03.775 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:27:03.775 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:03.791 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:03.791 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:27:03.791 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:03.808 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:03.808 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:27:03.808 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:03.846 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:03.846 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:27:03.847 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:03.865 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:03.865 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:27:03.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:03.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.992+0000 7f4d12c5c640 1 Processor -- start 2024-03-06T07:27:03.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.992+0000 7f4d12c5c640 1 -- start start 2024-03-06T07:27:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d12c5c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4d0c06df30 0x7f4d0c0ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d12c5c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d0c0aec40 0x7f4d0c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d12c5c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0c071240 con 0x7f4d0c158520 2024-03-06T07:27:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d12c5c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d0c0af0f0 con 0x7f4d0c0aec40 2024-03-06T07:27:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d12c5c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d0c071b70 con 0x7f4d0c06df30 2024-03-06T07:27:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d111f9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d0c0aec40 0x7f4d0c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d109f8640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4d0c158520 0x7f4d0c06d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49720/0 (socket says 172.21.15.27:49720) 2024-03-06T07:27:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d111f9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d0c0aec40 0x7f4d0c0b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52452/0 (socket says 172.21.15.27:52452) 2024-03-06T07:27:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d109f8640 1 -- 172.21.15.27:0/1853249004 learned_addr learned my addr 172.21.15.27:0/1853249004 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:03.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d0bfff640 1 --2- 172.21.15.27:0/1853249004 >> v2:172.21.15.69:3300/0 conn(0x7f4d0c06df30 0x7f4d0c0ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d111f9640 1 -- 172.21.15.27:0/1853249004 >> v2:172.21.15.69:3300/0 conn(0x7f4d0c06df30 msgr2=0x7f4d0c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d111f9640 1 --2- 172.21.15.27:0/1853249004 >> v2:172.21.15.69:3300/0 conn(0x7f4d0c06df30 0x7f4d0c0ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d111f9640 1 -- 172.21.15.27:0/1853249004 >> v1:172.21.15.27:6789/0 conn(0x7f4d0c158520 legacy=0x7f4d0c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d111f9640 1 -- 172.21.15.27:0/1853249004 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0c071520 con 0x7f4d0c0aec40 2024-03-06T07:27:03.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.993+0000 7f4d0b7fe640 1 -- 172.21.15.27:0/1853249004 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450568477 0 0) 0x7f4d0c071240 con 0x7f4d0c158520 2024-03-06T07:27:03.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.994+0000 7f4d111f9640 1 --2- 172.21.15.27:0/1853249004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 0x7f4d0c0b3640 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4d04009d90 tx=0x7f4d0402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2105b33d77e476 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.994+0000 7f4d0b7fe640 1 -- 172.21.15.27:0/1853249004 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d04039070 con 0x7f4d0c0aec40 2024-03-06T07:27:03.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.994+0000 7f4d0b7fe640 1 -- 172.21.15.27:0/1853249004 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0402cc60 con 0x7f4d0c0aec40 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.994+0000 7f4d0b7fe640 1 -- 172.21.15.27:0/1853249004 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d04034660 con 0x7f4d0c0aec40 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.995+0000 7f4d12c5c640 1 -- 172.21.15.27:0/1853249004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 msgr2=0x7f4d0c0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.995+0000 7f4d12c5c640 1 --2- 172.21.15.27:0/1853249004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 0x7f4d0c0b3640 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4d04009d90 tx=0x7f4d0402beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.995+0000 7f4d12c5c640 1 -- 172.21.15.27:0/1853249004 shutdown_connections 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.995+0000 7f4d12c5c640 1 --2- 172.21.15.27:0/1853249004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 0x7f4d0c0b3640 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.995+0000 7f4d12c5c640 1 --2- 172.21.15.27:0/1853249004 >> v2:172.21.15.69:3300/0 conn(0x7f4d0c06df30 0x7f4d0c0ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.995+0000 7f4d12c5c640 1 -- 172.21.15.27:0/1853249004 >> 172.21.15.27:0/1853249004 conn(0x7f4d0c066d00 msgr2=0x7f4d0c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.995+0000 7f4d12c5c640 1 -- 172.21.15.27:0/1853249004 shutdown_connections 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.995+0000 7f4d12c5c640 1 -- 172.21.15.27:0/1853249004 wait complete. 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.996+0000 7f4d12c5c640 1 Processor -- start 2024-03-06T07:27:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.996+0000 7f4d12c5c640 1 -- start start 2024-03-06T07:27:04.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d12c5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d0c06df30 0x7f4d0c154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d12c5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 0x7f4d0c157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d12c5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d0c158520 0x7f4d0c14b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d12c5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c158900 con 0x7f4d0c0aec40 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d12c5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c072000 con 0x7f4d0c06df30 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d12c5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c158a80 con 0x7f4d0c158520 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 0x7f4d0c157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 0x7f4d0c157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52458/0 (socket says 172.21.15.27:52458) 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d0bfff640 1 -- 172.21.15.27:0/3142597253 learned_addr learned my addr 172.21.15.27:0/3142597253 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d109f8640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d0c06df30 0x7f4d0c154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d111f9640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d0c158520 0x7f4d0c14b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d0bfff640 1 -- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d0c158520 msgr2=0x7f4d0c14b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d0bfff640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d0c158520 0x7f4d0c14b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d0bfff640 1 -- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d0c06df30 msgr2=0x7f4d0c154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d0bfff640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d0c06df30 0x7f4d0c154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d0bfff640 1 -- 172.21.15.27:0/3142597253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0c157d70 con 0x7f4d0c0aec40 2024-03-06T07:27:04.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d109f8640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d0c06df30 0x7f4d0c154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:04.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.997+0000 7f4d0bfff640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 0x7f4d0c157830 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800d830 tx=0x7f4cf800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.998+0000 7f4d097fa640 1 -- 172.21.15.27:0/3142597253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf800b6c0 con 0x7f4d0c0aec40 2024-03-06T07:27:04.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.998+0000 7f4d097fa640 1 -- 172.21.15.27:0/3142597253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf8019070 con 0x7f4d0c0aec40 2024-03-06T07:27:04.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.998+0000 7f4d097fa640 1 -- 172.21.15.27:0/3142597253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf800ba80 con 0x7f4d0c0aec40 2024-03-06T07:27:04.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.998+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0c1555c0 con 0x7f4d0c0aec40 2024-03-06T07:27:04.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.998+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0c155b80 con 0x7f4d0c0aec40 2024-03-06T07:27:04.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.999+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0c067ff0 con 0x7f4d0c0aec40 2024-03-06T07:27:04.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:03.999+0000 7f4d097fa640 1 -- 172.21.15.27:0/3142597253 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cf801e460 con 0x7f4d0c0aec40 2024-03-06T07:27:04.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.000+0000 7f4d097fa640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4cf006b4d0 0x7f4cf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.000+0000 7f4d097fa640 1 -- 172.21.15.27:0/3142597253 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4cf808a630 con 0x7f4d0c0aec40 2024-03-06T07:27:04.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.000+0000 7f4d109f8640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4cf006b4d0 0x7f4cf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.000+0000 7f4d109f8640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4cf006b4d0 0x7f4cf006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4d00000f60 tx=0x7f4d00007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.004+0000 7f4d097fa640 1 -- 172.21.15.27:0/3142597253 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cf8059730 con 0x7f4d0c0aec40 2024-03-06T07:27:04.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.225+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d0c0b31f0 con 0x7f4d0c0aec40 2024-03-06T07:27:04.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.226+0000 7f4d097fa640 1 -- 172.21.15.27:0/3142597253 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f4cf8059030 con 0x7f4d0c0aec40 2024-03-06T07:27:04.227 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:04.228 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5609,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3385619229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3385619229},{"type":"v1","addr":"172.21.15.69:6835","nonce":3385619229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6055,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/582817533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":582817533},{"type":"v1","addr":"172.21.15.69:6837","nonce":582817533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/502405068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":502405068},{"type":"v1","addr":"172.21.15.27:6834","nonce":502405068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4219841834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4219841834},{"type":"v1","addr":"172.21.15.27:6835","nonce":4219841834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[]} 2024-03-06T07:27:04.228 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 46 2024-03-06T07:27:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4cf006b4d0 msgr2=0x7f4cf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4cf006b4d0 0x7f4cf006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4d00000f60 tx=0x7f4d00007ca0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 msgr2=0x7f4d0c157830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 0x7f4d0c157830 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800d830 tx=0x7f4cf800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 shutdown_connections 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4cf006b4d0 0x7f4cf006d990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d0c158520 0x7f4d0c14b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d0c0aec40 0x7f4d0c157830 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 --2- 172.21.15.27:0/3142597253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d0c06df30 0x7f4d0c154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 >> 172.21.15.27:0/3142597253 conn(0x7f4d0c066d00 msgr2=0x7f4d0c1594f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 shutdown_connections 2024-03-06T07:27:04.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.228+0000 7f4d12c5c640 1 -- 172.21.15.27:0/3142597253 wait complete. 2024-03-06T07:27:04.240 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:04.240 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:27:04.241 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:04.241 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:04.241 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:04.241 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:27:04.241 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:04.241 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:04.241 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:04.241 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:27:04.242 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:04.242 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:04.242 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:04.242 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:27:04.242 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:04.243 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:04.251 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:04.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:04.252 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:04.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:04.257 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:04.257 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:04.258 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:04.258 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:04.294 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:04.294 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:04.294 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:04.295 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:04.300 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:04.300 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:04.301 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:04.301 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:04.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:27:04.376 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:27:04.376 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:27:04.386 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:27:04.386 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:27:04.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.412+0000 7fd9b7db5640 1 Processor -- start 2024-03-06T07:27:04.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.413+0000 7fd9b7db5640 1 -- start start 2024-03-06T07:27:04.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.413+0000 7fd9b7db5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9b00b95a0 0x7fd9b00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.413+0000 7fd9b7db5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b00cea30 con 0x7fd9b00b9a10 2024-03-06T07:27:04.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.413+0000 7fd9b7db5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b00cf3c0 con 0x7fd9b015df00 2024-03-06T07:27:04.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.413+0000 7fd9b7db5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b00cf0e0 con 0x7fd9b00b95a0 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b6352640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd9b00b9a10 0x7fd9b00d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49744/0 (socket says 172.21.15.27:49744) 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b6352640 1 -- 172.21.15.27:0/28708516 learned_addr learned my addr 172.21.15.27:0/28708516 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b5350640 1 --2- 172.21.15.27:0/28708516 >> v2:172.21.15.69:3300/0 conn(0x7fd9b00b95a0 0x7fd9b00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937444376 0 0) 0x7fd9b00cea30 con 0x7fd9b00b9a10 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd99c003520 con 0x7fd9b00b9a10 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243324798 0 0) 0x7fd9b00cf3c0 con 0x7fd9b015df00 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b00cea30 con 0x7fd9b015df00 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 644560854 0 0) 0x7fd99c003520 con 0x7fd9b00b9a10 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 >> v2:172.21.15.69:3300/0 conn(0x7fd9b00b95a0 msgr2=0x7fd9b00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 --2- 172.21.15.27:0/28708516 >> v2:172.21.15.69:3300/0 conn(0x7fd9b00b95a0 0x7fd9b00b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 >> v1:172.21.15.69:6789/0 conn(0x7fd9b015df00 legacy=0x7fd9b00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b00ce470 con 0x7fd9b00b9a10 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fd9a80019f0 con 0x7fd9b00b9a10 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b5350640 1 --2- 172.21.15.27:0/28708516 >> v2:172.21.15.69:3300/0 conn(0x7fd9b00b95a0 0x7fd9b00b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 >> v1:172.21.15.27:6789/0 conn(0x7fd9b00b9a10 legacy=0x7fd9b00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 0x7fd99c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99c004800 0x7fd99c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd99c0051c0 0x7fd99c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b00cf0e0 con 0x7fd99c0051c0 2024-03-06T07:27:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c00c010 con 0x7fd99c004800 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c00c210 con 0x7fd99c003b70 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.414+0000 7fd9b5b51640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 0x7fd99c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.415+0000 7fd9b6352640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd99c0051c0 0x7fd99c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.415+0000 7fd9b5350640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99c004800 0x7fd99c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.415+0000 7fd9b5b51640 1 -- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99c004800 msgr2=0x7fd99c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.415+0000 7fd9b5b51640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd99c004800 0x7fd99c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.415+0000 7fd9b5b51640 1 -- 172.21.15.27:0/28708516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd99c0051c0 msgr2=0x7fd99c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.415+0000 7fd9b5b51640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd99c0051c0 0x7fd99c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.415+0000 7fd9b5b51640 1 -- 172.21.15.27:0/28708516 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c003520 con 0x7fd99c003b70 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.415+0000 7fd9b6352640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd99c0051c0 0x7fd99c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:04.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.415+0000 7fd9b6352640 1 -- 172.21.15.27:0/28708516 reap_dead start 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b5b51640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 0x7fd99c003f50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac007f00 tx=0x7fd99c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67465fe7dd6ce198 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac007ad0 con 0x7fd99c003b70 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ac030070 con 0x7fd99c003b70 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b4b4f640 1 -- 172.21.15.27:0/28708516 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac02c6e0 con 0x7fd99c003b70 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b7db5640 1 -- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 msgr2=0x7fd99c003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b7db5640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 0x7fd99c003f50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac007f00 tx=0x7fd99c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b7db5640 1 -- 172.21.15.27:0/28708516 shutdown_connections 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b7db5640 1 --2- 172.21.15.27:0/28708516 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 0x7fd99c003f50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b7db5640 1 -- 172.21.15.27:0/28708516 >> 172.21.15.27:0/28708516 conn(0x7fd9b0071960 msgr2=0x7fd9b0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b7db5640 1 -- 172.21.15.27:0/28708516 shutdown_connections 2024-03-06T07:27:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.416+0000 7fd9b7db5640 1 -- 172.21.15.27:0/28708516 wait complete. 2024-03-06T07:27:04.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b7db5640 1 Processor -- start 2024-03-06T07:27:04.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b7db5640 1 -- start start 2024-03-06T07:27:04.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b7db5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 0x7fd9b014cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b7db5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b0149fa0 0x7fd9b00ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b7db5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b01f2f10 0x7fd9b00c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b7db5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b00cedf0 con 0x7fd9b01f2f10 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b7db5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b00cec70 con 0x7fd9b0149fa0 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b7db5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b00d25b0 con 0x7fd99c003b70 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b6352640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b01f2f10 0x7fd9b00c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b6352640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b01f2f10 0x7fd9b00c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52502/0 (socket says 172.21.15.27:52502) 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.418+0000 7fd9b6352640 1 -- 172.21.15.27:0/3356344371 learned_addr learned my addr 172.21.15.27:0/3356344371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b5350640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b0149fa0 0x7fd9b00ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b5b51640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 0x7fd9b014cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b6352640 1 -- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 msgr2=0x7fd9b014cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b6352640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 0x7fd9b014cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b6352640 1 -- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b0149fa0 msgr2=0x7fd9b00ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b6352640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b0149fa0 0x7fd9b00ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b6352640 1 -- 172.21.15.27:0/3356344371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b014a420 con 0x7fd9b01f2f10 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b6352640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b01f2f10 0x7fd9b00c77b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd9b00c7750 tx=0x7fd9a8007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9a27fc640 1 -- 172.21.15.27:0/3356344371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a800a450 con 0x7fd9b01f2f10 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9a27fc640 1 -- 172.21.15.27:0/3356344371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a801c070 con 0x7fd9b01f2f10 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9a27fc640 1 -- 172.21.15.27:0/3356344371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a800ae60 con 0x7fd9b01f2f10 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b00ca670 con 0x7fd9b01f2f10 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.419+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b00cbf40 con 0x7fd9b01f2f10 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.420+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b0067ff0 con 0x7fd9b01f2f10 2024-03-06T07:27:04.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.421+0000 7fd9a27fc640 1 -- 172.21.15.27:0/3356344371 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9a8018c50 con 0x7fd9b01f2f10 2024-03-06T07:27:04.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.421+0000 7fd9a27fc640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99406b500 0x7fd99406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.421+0000 7fd9a27fc640 1 -- 172.21.15.27:0/3356344371 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd9a80926a0 con 0x7fd9b01f2f10 2024-03-06T07:27:04.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.421+0000 7fd9b5b51640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99406b500 0x7fd99406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.422+0000 7fd9b5b51640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99406b500 0x7fd99406d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac034000 tx=0x7fd9ac009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.425+0000 7fd9a27fc640 1 -- 172.21.15.27:0/3356344371 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9a8028030 con 0x7fd9b01f2f10 2024-03-06T07:27:04.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:04.617+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd9b00c7530 con 0x7fd9b01f2f10 2024-03-06T07:27:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.695+0000 7fd9a27fc640 1 -- 172.21.15.27:0/3356344371 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v80) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd9b014a420 con 0x7fd9b01f2f10 2024-03-06T07:27:05.696 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-03-06T07:27:05.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.698+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99406b500 msgr2=0x7fd99406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.698+0000 7fd9b7db5640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99406b500 0x7fd99406d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac034000 tx=0x7fd9ac009030 comp rx=0 tx=0).stop 2024-03-06T07:27:05.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.698+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b01f2f10 msgr2=0x7fd9b00c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.698+0000 7fd9b7db5640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b01f2f10 0x7fd9b00c77b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd9b00c7750 tx=0x7fd9a8007430 comp rx=0 tx=0).stop 2024-03-06T07:27:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.698+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 shutdown_connections 2024-03-06T07:27:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.699+0000 7fd9b7db5640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99406b500 0x7fd99406d9c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.699+0000 7fd9b7db5640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b01f2f10 0x7fd9b00c77b0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.699+0000 7fd9b7db5640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b0149fa0 0x7fd9b00ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.699+0000 7fd9b7db5640 1 --2- 172.21.15.27:0/3356344371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd99c003b70 0x7fd9b014cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.699+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 >> 172.21.15.27:0/3356344371 conn(0x7fd9b0071960 msgr2=0x7fd9b00b7420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.699+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 shutdown_connections 2024-03-06T07:27:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.699+0000 7fd9b7db5640 1 -- 172.21.15.27:0/3356344371 wait complete. 2024-03-06T07:27:05.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:27:05.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.826+0000 7fc3358c9640 1 Processor -- start 2024-03-06T07:27:05.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.826+0000 7fc3358c9640 1 -- start start 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.826+0000 7fc3358c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3300b1d50 0x7fc3300b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.826+0000 7fc3358c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc33006d210 0x7fc33006d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.826+0000 7fc3358c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33015e410 con 0x7fc33015e030 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.826+0000 7fc3358c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3300b26f0 con 0x7fc3300b1d50 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.826+0000 7fc3358c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc330162700 con 0x7fc33006d210 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.827+0000 7fc327fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3300b1d50 0x7fc3300b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.827+0000 7fc327fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3300b1d50 0x7fc3300b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52514/0 (socket says 172.21.15.27:52514) 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.827+0000 7fc327fff640 1 -- 172.21.15.27:0/2453732065 learned_addr learned my addr 172.21.15.27:0/2453732065 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.827+0000 7fc32f7fe640 1 --2- 172.21.15.27:0/2453732065 >> v2:172.21.15.69:3300/0 conn(0x7fc33006d210 0x7fc33006d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.827+0000 7fc327fff640 1 -- 172.21.15.27:0/2453732065 >> v2:172.21.15.69:3300/0 conn(0x7fc33006d210 msgr2=0x7fc33006d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.827+0000 7fc327fff640 1 --2- 172.21.15.27:0/2453732065 >> v2:172.21.15.69:3300/0 conn(0x7fc33006d210 0x7fc33006d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.827+0000 7fc327fff640 1 -- 172.21.15.27:0/2453732065 >> v1:172.21.15.69:6789/0 conn(0x7fc33015e030 legacy=0x7fc33006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.827+0000 7fc327fff640 1 -- 172.21.15.27:0/2453732065 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33006f140 con 0x7fc3300b1d50 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.828+0000 7fc327fff640 1 --2- 172.21.15.27:0/2453732065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3300b1d50 0x7fc3300b21b0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc330162b90 tx=0x7fc31802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f15b28276b8da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.828+0000 7fc32e7fc640 1 -- 172.21.15.27:0/2453732065 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318034070 con 0x7fc3300b1d50 2024-03-06T07:27:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.828+0000 7fc32e7fc640 1 -- 172.21.15.27:0/2453732065 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318030860 con 0x7fc3300b1d50 2024-03-06T07:27:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.828+0000 7fc32e7fc640 1 -- 172.21.15.27:0/2453732065 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318030c30 con 0x7fc3300b1d50 2024-03-06T07:27:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.828+0000 7fc3358c9640 1 -- 172.21.15.27:0/2453732065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3300b1d50 msgr2=0x7fc3300b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.828+0000 7fc3358c9640 1 --2- 172.21.15.27:0/2453732065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3300b1d50 0x7fc3300b21b0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc330162b90 tx=0x7fc31802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.829+0000 7fc3358c9640 1 -- 172.21.15.27:0/2453732065 shutdown_connections 2024-03-06T07:27:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.829+0000 7fc3358c9640 1 --2- 172.21.15.27:0/2453732065 >> v2:172.21.15.69:3300/0 conn(0x7fc33006d210 0x7fc33006d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.829+0000 7fc3358c9640 1 --2- 172.21.15.27:0/2453732065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3300b1d50 0x7fc3300b21b0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.829+0000 7fc3358c9640 1 -- 172.21.15.27:0/2453732065 >> 172.21.15.27:0/2453732065 conn(0x7fc3300b55e0 msgr2=0x7fc3300b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.829+0000 7fc3358c9640 1 -- 172.21.15.27:0/2453732065 shutdown_connections 2024-03-06T07:27:05.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.829+0000 7fc3358c9640 1 -- 172.21.15.27:0/2453732065 wait complete. 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.830+0000 7fc3358c9640 1 Processor -- start 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.830+0000 7fc3358c9640 1 -- start start 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.830+0000 7fc3358c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33006d210 0x7fc330156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.830+0000 7fc3358c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3300b1d50 0x7fc3301f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.830+0000 7fc3358c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33015e030 0x7fc33015fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc3358c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3301620b0 con 0x7fc33006d210 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc3358c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc330161dd0 con 0x7fc3300b1d50 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc3358c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc330162390 con 0x7fc33015e030 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33006d210 0x7fc330156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33006d210 0x7fc330156860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52518/0 (socket says 172.21.15.27:52518) 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32effd640 1 -- 172.21.15.27:0/3196826383 learned_addr learned my addr 172.21.15.27:0/3196826383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc327fff640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3300b1d50 0x7fc3301f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32f7fe640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33015e030 0x7fc33015fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32effd640 1 -- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33015e030 msgr2=0x7fc33015fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32effd640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33015e030 0x7fc33015fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32effd640 1 -- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3300b1d50 msgr2=0x7fc3301f4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32effd640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3300b1d50 0x7fc3301f4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32effd640 1 -- 172.21.15.27:0/3196826383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33013dc50 con 0x7fc33006d210 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32effd640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33006d210 0x7fc330156860 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c3c0 tx=0x7fc32000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32cff9640 1 -- 172.21.15.27:0/3196826383 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3200052d0 con 0x7fc33006d210 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32cff9640 1 -- 172.21.15.27:0/3196826383 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc320005490 con 0x7fc33006d210 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc32cff9640 1 -- 172.21.15.27:0/3196826383 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320007260 con 0x7fc33006d210 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc330156da0 con 0x7fc33006d210 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.831+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33009d5c0 con 0x7fc33006d210 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.832+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc330067ff0 con 0x7fc33006d210 2024-03-06T07:27:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.833+0000 7fc32cff9640 1 -- 172.21.15.27:0/3196826383 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc320007400 con 0x7fc33006d210 2024-03-06T07:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.833+0000 7fc32cff9640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc31006b4d0 0x7fc31006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.833+0000 7fc32cff9640 1 -- 172.21.15.27:0/3196826383 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc320089390 con 0x7fc33006d210 2024-03-06T07:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.833+0000 7fc327fff640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc31006b4d0 0x7fc31006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.834+0000 7fc327fff640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc31006b4d0 0x7fc31006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc318036000 tx=0x7fc31803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:05.837+0000 7fc32cff9640 1 -- 172.21.15.27:0/3196826383 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc330156da0 con 0x7fc33006d210 2024-03-06T07:27:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.087+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc3300b2910 con 0x7fc33006d210 2024-03-06T07:27:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.087+0000 7fc32cff9640 1 -- 172.21.15.27:0/3196826383 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc33013dc50 con 0x7fc33006d210 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g==", 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ==", 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg==", 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ==", 2024-03-06T07:27:06.091 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw==", 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.092 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA==", 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA==", 2024-03-06T07:27:06.093 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg==", 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.094 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A==", 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw==", 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA==", 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:06.096 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A==", 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.097 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA==", 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ==", 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:27:06.098 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ==", 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg==", 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlkc2iEhAA6BuYba0FzXeVjGPOP8sgyQ==", 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.100 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw==", 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg==", 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:27:06.102 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:27:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc31006b4d0 msgr2=0x7fc31006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc31006b4d0 0x7fc31006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc318036000 tx=0x7fc31803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33006d210 msgr2=0x7fc330156860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33006d210 0x7fc330156860 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c3c0 tx=0x7fc32000c880 comp rx=0 tx=0).stop 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 shutdown_connections 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc31006b4d0 0x7fc31006d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc33015e030 0x7fc33015fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3300b1d50 0x7fc3301f4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 --2- 172.21.15.27:0/3196826383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33006d210 0x7fc330156860 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 >> 172.21.15.27:0/3196826383 conn(0x7fc3300b55e0 msgr2=0x7fc3300ad610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 shutdown_connections 2024-03-06T07:27:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.089+0000 7fc3358c9640 1 -- 172.21.15.27:0/3196826383 wait complete. 2024-03-06T07:27:06.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:06.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.211+0000 7fb06f919640 1 Processor -- start 2024-03-06T07:27:06.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.211+0000 7fb06f919640 1 -- start start 2024-03-06T07:27:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06f919640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb06806df30 0x7fb0680ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06f919640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0680aec40 0x7fb0680b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06f919640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb068071240 con 0x7fb068158520 2024-03-06T07:27:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06f919640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0680af0f0 con 0x7fb0680aec40 2024-03-06T07:27:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06f919640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb068071b70 con 0x7fb06806df30 2024-03-06T07:27:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06d6b5640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb068158520 0x7fb06806d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49746/0 (socket says 172.21.15.27:49746) 2024-03-06T07:27:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06d6b5640 1 -- 172.21.15.27:0/3211413501 learned_addr learned my addr 172.21.15.27:0/3211413501 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06deb6640 1 --2- 172.21.15.27:0/3211413501 >> v2:172.21.15.27:3300/0 conn(0x7fb0680aec40 0x7fb0680b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb067fff640 1 -- 172.21.15.27:0/3211413501 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105693190 0 0) 0x7fb068071240 con 0x7fb068158520 2024-03-06T07:27:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb067fff640 1 -- 172.21.15.27:0/3211413501 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb050003520 con 0x7fb068158520 2024-03-06T07:27:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06ceb4640 1 --2- 172.21.15.27:0/3211413501 >> v2:172.21.15.69:3300/0 conn(0x7fb06806df30 0x7fb0680ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06deb6640 1 -- 172.21.15.27:0/3211413501 >> v2:172.21.15.69:3300/0 conn(0x7fb06806df30 msgr2=0x7fb0680ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06deb6640 1 --2- 172.21.15.27:0/3211413501 >> v2:172.21.15.69:3300/0 conn(0x7fb06806df30 0x7fb0680ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06deb6640 1 -- 172.21.15.27:0/3211413501 >> v1:172.21.15.27:6789/0 conn(0x7fb068158520 legacy=0x7fb06806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.212+0000 7fb06deb6640 1 -- 172.21.15.27:0/3211413501 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb068071520 con 0x7fb0680aec40 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.213+0000 7fb06deb6640 1 --2- 172.21.15.27:0/3211413501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0680aec40 0x7fb0680b3640 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb058009d90 tx=0x7fb05802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa6f6bfc48800e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.213+0000 7fb067fff640 1 -- 172.21.15.27:0/3211413501 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058039070 con 0x7fb0680aec40 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.213+0000 7fb067fff640 1 -- 172.21.15.27:0/3211413501 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05802cc60 con 0x7fb0680aec40 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.213+0000 7fb067fff640 1 -- 172.21.15.27:0/3211413501 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058034660 con 0x7fb0680aec40 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.214+0000 7fb06f919640 1 -- 172.21.15.27:0/3211413501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0680aec40 msgr2=0x7fb0680b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.214+0000 7fb06f919640 1 --2- 172.21.15.27:0/3211413501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0680aec40 0x7fb0680b3640 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb058009d90 tx=0x7fb05802beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.214+0000 7fb06f919640 1 -- 172.21.15.27:0/3211413501 shutdown_connections 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.214+0000 7fb06f919640 1 --2- 172.21.15.27:0/3211413501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0680aec40 0x7fb0680b3640 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.214+0000 7fb06f919640 1 --2- 172.21.15.27:0/3211413501 >> v2:172.21.15.69:3300/0 conn(0x7fb06806df30 0x7fb0680ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.214+0000 7fb06f919640 1 -- 172.21.15.27:0/3211413501 >> 172.21.15.27:0/3211413501 conn(0x7fb068066d00 msgr2=0x7fb0680670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.214+0000 7fb06f919640 1 -- 172.21.15.27:0/3211413501 shutdown_connections 2024-03-06T07:27:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.214+0000 7fb06f919640 1 -- 172.21.15.27:0/3211413501 wait complete. 2024-03-06T07:27:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.215+0000 7fb06f919640 1 Processor -- start 2024-03-06T07:27:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.215+0000 7fb06f919640 1 -- start start 2024-03-06T07:27:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06f919640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb06806df30 0x7fb06814bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06f919640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0680aec40 0x7fb0680cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06f919640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068158520 0x7fb068156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06f919640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb068158900 con 0x7fb068158520 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06f919640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb068072000 con 0x7fb0680aec40 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06f919640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb068158a80 con 0x7fb06806df30 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06deb6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068158520 0x7fb068156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06deb6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068158520 0x7fb068156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52540/0 (socket says 172.21.15.27:52540) 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06deb6640 1 -- 172.21.15.27:0/3989527898 learned_addr learned my addr 172.21.15.27:0/3989527898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06ceb4640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0680aec40 0x7fb0680cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06deb6640 1 -- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb06806df30 msgr2=0x7fb06814bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06d6b5640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb06806df30 0x7fb06814bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06deb6640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb06806df30 0x7fb06814bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06deb6640 1 -- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0680aec40 msgr2=0x7fb0680cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06deb6640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0680aec40 0x7fb0680cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06deb6640 1 -- 172.21.15.27:0/3989527898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0680ce390 con 0x7fb068158520 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06ceb4640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0680aec40 0x7fb0680cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.216+0000 7fb06d6b5640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb06806df30 0x7fb06814bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.217+0000 7fb06deb6640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068158520 0x7fb068156ff0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb058009d90 tx=0x7fb058004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.217+0000 7fb065ffb640 1 -- 172.21.15.27:0/3989527898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058039050 con 0x7fb068158520 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.217+0000 7fb065ffb640 1 -- 172.21.15.27:0/3989527898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb058045070 con 0x7fb068158520 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.217+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0680ce670 con 0x7fb068158520 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.217+0000 7fb065ffb640 1 -- 172.21.15.27:0/3989527898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058009450 con 0x7fb068158520 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.217+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0680cec30 con 0x7fb068158520 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.218+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb068067ff0 con 0x7fb068158520 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.218+0000 7fb065ffb640 1 -- 172.21.15.27:0/3989527898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb05803e850 con 0x7fb068158520 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.219+0000 7fb065ffb640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb05406b4d0 0x7fb05406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.219+0000 7fb065ffb640 1 -- 172.21.15.27:0/3989527898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb0580acb00 con 0x7fb068158520 2024-03-06T07:27:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.219+0000 7fb06d6b5640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb05406b4d0 0x7fb05406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.219+0000 7fb06d6b5640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb05406b4d0 0x7fb05406d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb060005880 tx=0x7fb060009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.223+0000 7fb065ffb640 1 -- 172.21.15.27:0/3989527898 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb058032e20 con 0x7fb068158520 2024-03-06T07:27:06.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.405+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb06806d7a0 con 0x7fb068158520 2024-03-06T07:27:06.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.405+0000 7fb065ffb640 1 -- 172.21.15.27:0/3989527898 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+12820 (secure 0 0 0) 0x7fb05807bd60 con 0x7fb068158520 2024-03-06T07:27:06.407 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:06.407 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":80,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:05.693850+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:06.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.407+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb05406b4d0 msgr2=0x7fb05406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.407+0000 7fb06f919640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb05406b4d0 0x7fb05406d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb060005880 tx=0x7fb060009040 comp rx=0 tx=0).stop 2024-03-06T07:27:06.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.407+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068158520 msgr2=0x7fb068156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.407+0000 7fb06f919640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068158520 0x7fb068156ff0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb058009d90 tx=0x7fb058004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:06.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.408+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 shutdown_connections 2024-03-06T07:27:06.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.408+0000 7fb06f919640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb05406b4d0 0x7fb05406d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.408+0000 7fb06f919640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068158520 0x7fb068156ff0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.408+0000 7fb06f919640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0680aec40 0x7fb0680cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.408+0000 7fb06f919640 1 --2- 172.21.15.27:0/3989527898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb06806df30 0x7fb06814bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.408+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 >> 172.21.15.27:0/3989527898 conn(0x7fb068066d00 msgr2=0x7fb068159fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.408+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 shutdown_connections 2024-03-06T07:27:06.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.408+0000 7fb06f919640 1 -- 172.21.15.27:0/3989527898 wait complete. 2024-03-06T07:27:06.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:06.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.529+0000 7f579dff4640 1 Processor -- start 2024-03-06T07:27:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.529+0000 7f579dff4640 1 -- start start 2024-03-06T07:27:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f579dff4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f579815b130 0x7f579815b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f579dff4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579815bc40 con 0x7f5798144640 2024-03-06T07:27:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f579dff4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579815bf20 con 0x7f57980ce900 2024-03-06T07:27:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f579dff4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f579806df70 con 0x7f579815b130 2024-03-06T07:27:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f57977fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5798144640 0x7f579806e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49760/0 (socket says 172.21.15.27:49760) 2024-03-06T07:27:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f57977fe640 1 -- 172.21.15.27:0/619793742 learned_addr learned my addr 172.21.15.27:0/619793742 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f5796ffd640 1 --2- 172.21.15.27:0/619793742 >> v2:172.21.15.27:3300/0 conn(0x7f579815b130 0x7f579815b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f57967fc640 1 -- 172.21.15.27:0/619793742 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409655961 0 0) 0x7f579815bc40 con 0x7f5798144640 2024-03-06T07:27:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f57967fc640 1 -- 172.21.15.27:0/619793742 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5780003520 con 0x7f5798144640 2024-03-06T07:27:06.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f5796ffd640 1 -- 172.21.15.27:0/619793742 >> v1:172.21.15.69:6789/0 conn(0x7f57980ce900 legacy=0x7f57980b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f5796ffd640 1 -- 172.21.15.27:0/619793742 >> v1:172.21.15.27:6789/0 conn(0x7f5798144640 legacy=0x7f579806e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.530+0000 7f5796ffd640 1 -- 172.21.15.27:0/619793742 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579806dc00 con 0x7f579815b130 2024-03-06T07:27:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.531+0000 7f5796ffd640 1 --2- 172.21.15.27:0/619793742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f579815b130 0x7f579815b5c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f578c0097a0 tx=0x7f578c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ffeb6ef3774402 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.531+0000 7f57967fc640 1 -- 172.21.15.27:0/619793742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c034070 con 0x7f579815b130 2024-03-06T07:27:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.531+0000 7f57967fc640 1 -- 172.21.15.27:0/619793742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c030610 con 0x7f579815b130 2024-03-06T07:27:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.531+0000 7f57967fc640 1 -- 172.21.15.27:0/619793742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c0309c0 con 0x7f579815b130 2024-03-06T07:27:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.532+0000 7f579dff4640 1 -- 172.21.15.27:0/619793742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f579815b130 msgr2=0x7f579815b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.532+0000 7f579dff4640 1 --2- 172.21.15.27:0/619793742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f579815b130 0x7f579815b5c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f578c0097a0 tx=0x7f578c02b960 comp rx=0 tx=0).stop 2024-03-06T07:27:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.532+0000 7f579dff4640 1 -- 172.21.15.27:0/619793742 shutdown_connections 2024-03-06T07:27:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.532+0000 7f579dff4640 1 --2- 172.21.15.27:0/619793742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f579815b130 0x7f579815b5c0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.532+0000 7f579dff4640 1 -- 172.21.15.27:0/619793742 >> 172.21.15.27:0/619793742 conn(0x7f5798066d00 msgr2=0x7f5798067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.532+0000 7f579dff4640 1 -- 172.21.15.27:0/619793742 shutdown_connections 2024-03-06T07:27:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.532+0000 7f579dff4640 1 -- 172.21.15.27:0/619793742 wait complete. 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.533+0000 7f579dff4640 1 Processor -- start 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.533+0000 7f579dff4640 1 -- start start 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f579dff4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57980ce900 0x7f579813ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f579dff4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5798144640 0x7f5798159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f579dff4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f579815b130 0x7f579815d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f579dff4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5798070c20 con 0x7f57980ce900 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f579dff4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5798070aa0 con 0x7f579815b130 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f579dff4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57980b3de0 con 0x7f5798144640 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f57977fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57980ce900 0x7f579813ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f57977fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57980ce900 0x7f579813ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52556/0 (socket says 172.21.15.27:52556) 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f57977fe640 1 -- 172.21.15.27:0/1255726827 learned_addr learned my addr 172.21.15.27:0/1255726827 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f5797fff640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f579815b130 0x7f579815d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.534+0000 7f5796ffd640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5798144640 0x7f5798159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f57977fe640 1 -- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5798144640 msgr2=0x7f5798159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f57977fe640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5798144640 0x7f5798159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f57977fe640 1 -- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f579815b130 msgr2=0x7f579815d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f57977fe640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f579815b130 0x7f579815d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f57977fe640 1 -- 172.21.15.27:0/1255726827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579813f4a0 con 0x7f57980ce900 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f57977fe640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57980ce900 0x7f579813ef60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f578800c500 tx=0x7f578800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f577ffff640 1 -- 172.21.15.27:0/1255726827 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5788017070 con 0x7f57980ce900 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f577ffff640 1 -- 172.21.15.27:0/1255726827 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5788005ce0 con 0x7f57980ce900 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579813d6c0 con 0x7f57980ce900 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f577ffff640 1 -- 172.21.15.27:0/1255726827 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5788007330 con 0x7f57980ce900 2024-03-06T07:27:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.535+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579815cce0 con 0x7f57980ce900 2024-03-06T07:27:06.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.536+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57980733c0 con 0x7f57980ce900 2024-03-06T07:27:06.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.536+0000 7f577ffff640 1 -- 172.21.15.27:0/1255726827 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f578801a070 con 0x7f57980ce900 2024-03-06T07:27:06.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.537+0000 7f577ffff640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f578406b4d0 0x7f578406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.537+0000 7f577ffff640 1 -- 172.21.15.27:0/1255726827 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5788089260 con 0x7f57980ce900 2024-03-06T07:27:06.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.537+0000 7f5796ffd640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f578406b4d0 0x7f578406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.538+0000 7f5796ffd640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f578406b4d0 0x7f578406d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f578c036000 tx=0x7f578c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.541+0000 7f577ffff640 1 -- 172.21.15.27:0/1255726827 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f579813f4a0 con 0x7f57980ce900 2024-03-06T07:27:06.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.722+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f579815ae10 con 0x7f57980ce900 2024-03-06T07:27:06.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.722+0000 7f577ffff640 1 -- 172.21.15.27:0/1255726827 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5788057f30 con 0x7f57980ce900 2024-03-06T07:27:06.724 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:27:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f578406b4d0 msgr2=0x7f578406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f578406b4d0 0x7f578406d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f578c036000 tx=0x7f578c008030 comp rx=0 tx=0).stop 2024-03-06T07:27:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57980ce900 msgr2=0x7f579813ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57980ce900 0x7f579813ef60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f578800c500 tx=0x7f578800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 shutdown_connections 2024-03-06T07:27:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f578406b4d0 0x7f578406d990 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f579815b130 0x7f579815d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5798144640 0x7f5798159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 --2- 172.21.15.27:0/1255726827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57980ce900 0x7f579813ef60 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.724+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 >> 172.21.15.27:0/1255726827 conn(0x7f5798066d00 msgr2=0x7f579806e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.725+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 shutdown_connections 2024-03-06T07:27:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.725+0000 7f579dff4640 1 -- 172.21.15.27:0/1255726827 wait complete. 2024-03-06T07:27:06.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.843+0000 7fb143393640 1 Processor -- start 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.843+0000 7fb143393640 1 -- start start 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb143393640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb13c157b80 0x7fb13c15b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb143393640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb13c0cbab0 0x7fb13c070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb143393640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c06c7b0 con 0x7fb13c157f60 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb143393640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c06ca90 con 0x7fb13c0cbab0 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb143393640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c06cd70 con 0x7fb13c157b80 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb141930640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb13c0cbab0 0x7fb13c070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb14092e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb13c157f60 0x7fb13c15b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49774/0 (socket says 172.21.15.27:49774) 2024-03-06T07:27:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb14092e640 1 -- 172.21.15.27:0/159968502 learned_addr learned my addr 172.21.15.27:0/159968502 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb14112f640 1 --2- 172.21.15.27:0/159968502 >> v2:172.21.15.69:3300/0 conn(0x7fb13c157b80 0x7fb13c15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb141930640 1 -- 172.21.15.27:0/159968502 >> v2:172.21.15.69:3300/0 conn(0x7fb13c157b80 msgr2=0x7fb13c15b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb141930640 1 --2- 172.21.15.27:0/159968502 >> v2:172.21.15.69:3300/0 conn(0x7fb13c157b80 0x7fb13c15b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb141930640 1 -- 172.21.15.27:0/159968502 >> v1:172.21.15.27:6789/0 conn(0x7fb13c157f60 legacy=0x7fb13c15b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb133fff640 1 -- 172.21.15.27:0/159968502 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108380628 0 0) 0x7fb13c06c7b0 con 0x7fb13c157f60 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb141930640 1 -- 172.21.15.27:0/159968502 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c06d050 con 0x7fb13c0cbab0 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.844+0000 7fb14112f640 1 --2- 172.21.15.27:0/159968502 >> v2:172.21.15.69:3300/0 conn(0x7fb13c157b80 0x7fb13c15b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.845+0000 7fb141930640 1 --2- 172.21.15.27:0/159968502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c0cbab0 0x7fb13c070b40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb13c06ed60 tx=0x7fb13402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c612166902b59103 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.845+0000 7fb133fff640 1 -- 172.21.15.27:0/159968502 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134039070 con 0x7fb13c0cbab0 2024-03-06T07:27:06.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.845+0000 7fb133fff640 1 -- 172.21.15.27:0/159968502 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13402cc80 con 0x7fb13c0cbab0 2024-03-06T07:27:06.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.845+0000 7fb133fff640 1 -- 172.21.15.27:0/159968502 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134034660 con 0x7fb13c0cbab0 2024-03-06T07:27:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.846+0000 7fb143393640 1 -- 172.21.15.27:0/159968502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c0cbab0 msgr2=0x7fb13c070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.846+0000 7fb143393640 1 --2- 172.21.15.27:0/159968502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c0cbab0 0x7fb13c070b40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb13c06ed60 tx=0x7fb13402bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.846+0000 7fb143393640 1 -- 172.21.15.27:0/159968502 shutdown_connections 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.846+0000 7fb143393640 1 --2- 172.21.15.27:0/159968502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c0cbab0 0x7fb13c070b40 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.846+0000 7fb143393640 1 --2- 172.21.15.27:0/159968502 >> v2:172.21.15.69:3300/0 conn(0x7fb13c157b80 0x7fb13c15b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.846+0000 7fb143393640 1 -- 172.21.15.27:0/159968502 >> 172.21.15.27:0/159968502 conn(0x7fb13c066d00 msgr2=0x7fb13c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.846+0000 7fb143393640 1 -- 172.21.15.27:0/159968502 shutdown_connections 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.846+0000 7fb143393640 1 -- 172.21.15.27:0/159968502 wait complete. 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.847+0000 7fb143393640 1 Processor -- start 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.847+0000 7fb143393640 1 -- start start 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb143393640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb13c0cbab0 0x7fb13c150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb143393640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c157b80 0x7fb13c1464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb143393640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13c157f60 0x7fb13c14e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb143393640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c071080 con 0x7fb13c157b80 2024-03-06T07:27:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb143393640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c0cc260 con 0x7fb13c157f60 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb143393640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c071200 con 0x7fb13c0cbab0 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14092e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c157b80 0x7fb13c1464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14092e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c157b80 0x7fb13c1464a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52576/0 (socket says 172.21.15.27:52576) 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14092e640 1 -- 172.21.15.27:0/1442889176 learned_addr learned my addr 172.21.15.27:0/1442889176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb141930640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13c157f60 0x7fb13c14e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14112f640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb13c0cbab0 0x7fb13c150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14092e640 1 -- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb13c0cbab0 msgr2=0x7fb13c150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14092e640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb13c0cbab0 0x7fb13c150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14092e640 1 -- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13c157f60 msgr2=0x7fb13c14e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14092e640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13c157f60 0x7fb13c14e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14092e640 1 -- 172.21.15.27:0/1442889176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c150810 con 0x7fb13c157b80 2024-03-06T07:27:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb141930640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13c157f60 0x7fb13c14e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb14092e640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c157b80 0x7fb13c1464a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00c410 tx=0x7fb12c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb131ffb640 1 -- 172.21.15.27:0/1442889176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0074e0 con 0x7fb13c157b80 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.848+0000 7fb131ffb640 1 -- 172.21.15.27:0/1442889176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c0047e0 con 0x7fb13c157b80 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.849+0000 7fb131ffb640 1 -- 172.21.15.27:0/1442889176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c004bd0 con 0x7fb13c157b80 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.849+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c150af0 con 0x7fb13c157b80 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.849+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c1510b0 con 0x7fb13c157b80 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.850+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c071da0 con 0x7fb13c157b80 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.850+0000 7fb131ffb640 1 -- 172.21.15.27:0/1442889176 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb12c011070 con 0x7fb13c157b80 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.850+0000 7fb131ffb640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb12006b4d0 0x7fb12006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.851+0000 7fb131ffb640 1 -- 172.21.15.27:0/1442889176 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb12c089b00 con 0x7fb13c157b80 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.851+0000 7fb14112f640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb12006b4d0 0x7fb12006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.851+0000 7fb14112f640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb12006b4d0 0x7fb12006d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb13800d8b0 tx=0x7fb138008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:06.855+0000 7fb131ffb640 1 -- 172.21.15.27:0/1442889176 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb12c00e280 con 0x7fb13c157b80 2024-03-06T07:27:07.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.036+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb13c15d590 con 0x7fb13c157b80 2024-03-06T07:27:07.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.036+0000 7fb131ffb640 1 -- 172.21.15.27:0/1442889176 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v80) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb12c058cd0 con 0x7fb13c157b80 2024-03-06T07:27:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:07.038 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:27:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:27:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:27:07.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:07.039 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:27:07.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.038+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb12006b4d0 msgr2=0x7fb12006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.038+0000 7fb143393640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb12006b4d0 0x7fb12006d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb13800d8b0 tx=0x7fb138008040 comp rx=0 tx=0).stop 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.038+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c157b80 msgr2=0x7fb13c1464a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.038+0000 7fb143393640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c157b80 0x7fb13c1464a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00c410 tx=0x7fb12c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.039+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 shutdown_connections 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.039+0000 7fb143393640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb12006b4d0 0x7fb12006d990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.039+0000 7fb143393640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13c157f60 0x7fb13c14e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.039+0000 7fb143393640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb13c157b80 0x7fb13c1464a0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.039+0000 7fb143393640 1 --2- 172.21.15.27:0/1442889176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb13c0cbab0 0x7fb13c150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.039+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 >> 172.21.15.27:0/1442889176 conn(0x7fb13c066d00 msgr2=0x7fb13c071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.039+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 shutdown_connections 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.039+0000 7fb143393640 1 -- 172.21.15.27:0/1442889176 wait complete. 2024-03-06T07:27:07.051 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:07.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:07.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.161+0000 7f535d35c640 1 Processor -- start 2024-03-06T07:27:07.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.161+0000 7f535d35c640 1 -- start start 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.161+0000 7f535d35c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5358144640 0x7f535806e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.161+0000 7f535d35c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f535806e940 0x7f535815b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.161+0000 7f535d35c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f535815b670 0x7f53580ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.161+0000 7f535d35c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535806df70 con 0x7f535815b670 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.161+0000 7f535d35c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535815bc40 con 0x7f535806e940 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.161+0000 7f535d35c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f535815bf20 con 0x7f5358144640 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f53577fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f535815b670 0x7f53580ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f53577fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f535815b670 0x7f53580ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52578/0 (socket says 172.21.15.27:52578) 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f53577fe640 1 -- 172.21.15.27:0/1729112937 learned_addr learned my addr 172.21.15.27:0/1729112937 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f5356ffd640 1 --2- 172.21.15.27:0/1729112937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5358144640 0x7f535806e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f53567fc640 1 --2- 172.21.15.27:0/1729112937 >> v2:172.21.15.69:3300/0 conn(0x7f535806e940 0x7f535815b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f53577fe640 1 -- 172.21.15.27:0/1729112937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5358144640 msgr2=0x7f535806e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f53577fe640 1 --2- 172.21.15.27:0/1729112937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5358144640 0x7f535806e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f53577fe640 1 -- 172.21.15.27:0/1729112937 >> v2:172.21.15.69:3300/0 conn(0x7f535806e940 msgr2=0x7f535815b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f53577fe640 1 --2- 172.21.15.27:0/1729112937 >> v2:172.21.15.69:3300/0 conn(0x7f535806e940 0x7f535815b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.162+0000 7f53577fe640 1 -- 172.21.15.27:0/1729112937 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535806dc00 con 0x7f535815b670 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f53577fe640 1 --2- 172.21.15.27:0/1729112937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 0x7f53580ce900 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f53580cd510 tx=0x7f534802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2781976fbbf4808d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f5355ffb640 1 -- 172.21.15.27:0/1729112937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534802ca70 con 0x7f535815b670 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f5355ffb640 1 -- 172.21.15.27:0/1729112937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534802cc10 con 0x7f535815b670 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f5355ffb640 1 -- 172.21.15.27:0/1729112937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53480343f0 con 0x7f535815b670 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f535d35c640 1 -- 172.21.15.27:0/1729112937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 msgr2=0x7f53580ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f535d35c640 1 --2- 172.21.15.27:0/1729112937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 0x7f53580ce900 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f53580cd510 tx=0x7f534802bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f535d35c640 1 -- 172.21.15.27:0/1729112937 shutdown_connections 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f535d35c640 1 --2- 172.21.15.27:0/1729112937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 0x7f53580ce900 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f535d35c640 1 --2- 172.21.15.27:0/1729112937 >> v2:172.21.15.69:3300/0 conn(0x7f535806e940 0x7f535815b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f535d35c640 1 --2- 172.21.15.27:0/1729112937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5358144640 0x7f535806e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f535d35c640 1 -- 172.21.15.27:0/1729112937 >> 172.21.15.27:0/1729112937 conn(0x7f5358066d00 msgr2=0x7f5358067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f535d35c640 1 -- 172.21.15.27:0/1729112937 shutdown_connections 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.163+0000 7f535d35c640 1 -- 172.21.15.27:0/1729112937 wait complete. 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.165+0000 7f535d35c640 1 Processor -- start 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.165+0000 7f535d35c640 1 -- start start 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.165+0000 7f535d35c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f535806e940 0x7f53581f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.165+0000 7f535d35c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5358144640 0x7f53581f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.165+0000 7f535d35c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 0x7f535813ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.165+0000 7f535d35c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53580cefc0 con 0x7f535815b670 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.165+0000 7f535d35c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53580cee40 con 0x7f5358144640 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.165+0000 7f535d35c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53580cf140 con 0x7f535806e940 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 0x7f535813ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 0x7f535813ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52592/0 (socket says 172.21.15.27:52592) 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53577fe640 1 -- 172.21.15.27:0/494780728 learned_addr learned my addr 172.21.15.27:0/494780728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:07.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53567fc640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5358144640 0x7f53581f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f5356ffd640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f535806e940 0x7f53581f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53577fe640 1 -- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f535806e940 msgr2=0x7f53581f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53577fe640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f535806e940 0x7f53581f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53577fe640 1 -- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5358144640 msgr2=0x7f53581f6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53577fe640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5358144640 0x7f53581f6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53577fe640 1 -- 172.21.15.27:0/494780728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535813b760 con 0x7f535815b670 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f53577fe640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 0x7f535813ada0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5348009f40 tx=0x7f5348004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f534f7fe640 1 -- 172.21.15.27:0/494780728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348043020 con 0x7f535815b670 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f534f7fe640 1 -- 172.21.15.27:0/494780728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534803e450 con 0x7f535815b670 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f534f7fe640 1 -- 172.21.15.27:0/494780728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348009510 con 0x7f535815b670 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535813ba40 con 0x7f535815b670 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.166+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5358138fc0 con 0x7f535815b670 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.167+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53581391c0 con 0x7f535815b670 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.168+0000 7f534f7fe640 1 -- 172.21.15.27:0/494780728 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5348046020 con 0x7f535815b670 2024-03-06T07:27:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.168+0000 7f534f7fe640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f533806b4d0 0x7f533806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.168+0000 7f534f7fe640 1 -- 172.21.15.27:0/494780728 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f53480ac3a0 con 0x7f535815b670 2024-03-06T07:27:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.168+0000 7f5356ffd640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f533806b4d0 0x7f533806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.169+0000 7f5356ffd640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f533806b4d0 0x7f533806d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5350009ac0 tx=0x7f5350009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.172+0000 7f534f7fe640 1 -- 172.21.15.27:0/494780728 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f534804d030 con 0x7f535815b670 2024-03-06T07:27:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:07.357+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f53580733c0 con 0x7f535815b670 2024-03-06T07:27:08.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.352+0000 7f534f7fe640 1 -- 172.21.15.27:0/494780728 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v81) v1 ==== 138+0+0 (secure 0 0 0) 0x7f534807b070 con 0x7f535815b670 2024-03-06T07:27:08.354 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:08.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f533806b4d0 msgr2=0x7f533806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f533806b4d0 0x7f533806d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5350009ac0 tx=0x7f5350009190 comp rx=0 tx=0).stop 2024-03-06T07:27:08.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 msgr2=0x7f535813ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 0x7f535813ada0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5348009f40 tx=0x7f5348004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:08.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 shutdown_connections 2024-03-06T07:27:08.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f533806b4d0 0x7f533806d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f535815b670 0x7f535813ada0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5358144640 0x7f53581f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 --2- 172.21.15.27:0/494780728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f535806e940 0x7f53581f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 >> 172.21.15.27:0/494780728 conn(0x7f5358066d00 msgr2=0x7f53580cd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 shutdown_connections 2024-03-06T07:27:08.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.355+0000 7f535d35c640 1 -- 172.21.15.27:0/494780728 wait complete. 2024-03-06T07:27:08.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.493+0000 7fb98f1c9640 1 Processor -- start 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.493+0000 7fb98f1c9640 1 -- start start 2024-03-06T07:27:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.493+0000 7fb98f1c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 0x7fb9881624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.493+0000 7fb98f1c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9880cf9f0 0x7fb98815d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.493+0000 7fb98f1c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98806d4d0 con 0x7fb988072a30 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.493+0000 7fb98f1c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb98806dba0 con 0x7fb9880cf9f0 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.493+0000 7fb98f1c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880b4950 con 0x7fb9880d07c0 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.494+0000 7fb98d766640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9880cf9f0 0x7fb98815d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.494+0000 7fb98d766640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9880cf9f0 0x7fb98815d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52594/0 (socket says 172.21.15.27:52594) 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.494+0000 7fb98d766640 1 -- 172.21.15.27:0/1341196421 learned_addr learned my addr 172.21.15.27:0/1341196421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.494+0000 7fb987fff640 1 --2- 172.21.15.27:0/1341196421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 0x7fb9881624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.494+0000 7fb98d766640 1 -- 172.21.15.27:0/1341196421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 msgr2=0x7fb9881624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.494+0000 7fb98d766640 1 --2- 172.21.15.27:0/1341196421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 0x7fb9881624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.494+0000 7fb98d766640 1 -- 172.21.15.27:0/1341196421 >> v1:172.21.15.69:6789/0 conn(0x7fb988072a30 legacy=0x7fb9880d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.494+0000 7fb98d766640 1 -- 172.21.15.27:0/1341196421 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98806d7b0 con 0x7fb9880cf9f0 2024-03-06T07:27:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.495+0000 7fb98d766640 1 --2- 172.21.15.27:0/1341196421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9880cf9f0 0x7fb98815d660 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fb980009e50 tx=0x7fb98002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dcfd6f7791630de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.495+0000 7fb9877fe640 1 -- 172.21.15.27:0/1341196421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98002c9e0 con 0x7fb9880cf9f0 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.495+0000 7fb9877fe640 1 -- 172.21.15.27:0/1341196421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98002cb80 con 0x7fb9880cf9f0 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.495+0000 7fb9877fe640 1 -- 172.21.15.27:0/1341196421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9800343f0 con 0x7fb9880cf9f0 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.495+0000 7fb98f1c9640 1 -- 172.21.15.27:0/1341196421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9880cf9f0 msgr2=0x7fb98815d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.495+0000 7fb98f1c9640 1 --2- 172.21.15.27:0/1341196421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9880cf9f0 0x7fb98815d660 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fb980009e50 tx=0x7fb98002be40 comp rx=0 tx=0).stop 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.496+0000 7fb98f1c9640 1 -- 172.21.15.27:0/1341196421 shutdown_connections 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.496+0000 7fb98f1c9640 1 --2- 172.21.15.27:0/1341196421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9880cf9f0 0x7fb98815d660 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.496+0000 7fb98f1c9640 1 --2- 172.21.15.27:0/1341196421 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 0x7fb9881624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.496+0000 7fb98f1c9640 1 -- 172.21.15.27:0/1341196421 >> 172.21.15.27:0/1341196421 conn(0x7fb98806af20 msgr2=0x7fb98806b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.496+0000 7fb98f1c9640 1 -- 172.21.15.27:0/1341196421 shutdown_connections 2024-03-06T07:27:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.496+0000 7fb98f1c9640 1 -- 172.21.15.27:0/1341196421 wait complete. 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.497+0000 7fb98f1c9640 1 Processor -- start 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.497+0000 7fb98f1c9640 1 -- start start 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98f1c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb988072a30 0x7fb98815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98f1c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9880cf9f0 0x7fb98814ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98f1c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 0x7fb98815b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98f1c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb988072e10 con 0x7fb988072a30 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98f1c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb988071130 con 0x7fb9880cf9f0 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98f1c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb988072f90 con 0x7fb9880d07c0 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98cf65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb988072a30 0x7fb98815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98cf65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb988072a30 0x7fb98815cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52596/0 (socket says 172.21.15.27:52596) 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98cf65640 1 -- 172.21.15.27:0/655669253 learned_addr learned my addr 172.21.15.27:0/655669253 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb987fff640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9880cf9f0 0x7fb98814ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98d766640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 0x7fb98815b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98cf65640 1 -- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 msgr2=0x7fb98815b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98cf65640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 0x7fb98815b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98cf65640 1 -- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9880cf9f0 msgr2=0x7fb98814ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98cf65640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9880cf9f0 0x7fb98814ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98cf65640 1 -- 172.21.15.27:0/655669253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98815bee0 con 0x7fb988072a30 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.498+0000 7fb98cf65640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb988072a30 0x7fb98815cf80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00c3c0 tx=0x7fb97c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.499+0000 7fb9857fa640 1 -- 172.21.15.27:0/655669253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c0052d0 con 0x7fb988072a30 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.499+0000 7fb9857fa640 1 -- 172.21.15.27:0/655669253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c005490 con 0x7fb988072a30 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.499+0000 7fb9857fa640 1 -- 172.21.15.27:0/655669253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c007260 con 0x7fb988072a30 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.499+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98815c1c0 con 0x7fb988072a30 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.499+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98815c780 con 0x7fb988072a30 2024-03-06T07:27:08.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.500+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98809e320 con 0x7fb988072a30 2024-03-06T07:27:08.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.500+0000 7fb9857fa640 1 -- 172.21.15.27:0/655669253 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb97c007400 con 0x7fb988072a30 2024-03-06T07:27:08.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.501+0000 7fb9857fa640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96c06b4d0 0x7fb96c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.501+0000 7fb9857fa640 1 -- 172.21.15.27:0/655669253 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb97c0894d0 con 0x7fb988072a30 2024-03-06T07:27:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.501+0000 7fb987fff640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96c06b4d0 0x7fb96c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.501+0000 7fb987fff640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96c06b4d0 0x7fb96c06d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb9880aee30 tx=0x7fb974009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.505+0000 7fb9857fa640 1 -- 172.21.15.27:0/655669253 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb97c057e60 con 0x7fb988072a30 2024-03-06T07:27:08.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:08.689+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb9880af0a0 con 0x7fb988072a30 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.354+0000 7fb9857fa640 1 -- 172.21.15.27:0/655669253 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v82) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb97c05ce20 con 0x7fb988072a30 2024-03-06T07:27:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:27:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96c06b4d0 msgr2=0x7fb96c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96c06b4d0 0x7fb96c06d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb9880aee30 tx=0x7fb974009190 comp rx=0 tx=0).stop 2024-03-06T07:27:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb988072a30 msgr2=0x7fb98815cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb988072a30 0x7fb98815cf80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00c3c0 tx=0x7fb97c00c880 comp rx=0 tx=0).stop 2024-03-06T07:27:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 shutdown_connections 2024-03-06T07:27:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb96c06b4d0 0x7fb96c06d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9880d07c0 0x7fb98815b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9880cf9f0 0x7fb98814ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 --2- 172.21.15.27:0/655669253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb988072a30 0x7fb98815cf80 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.357+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 >> 172.21.15.27:0/655669253 conn(0x7fb98806af20 msgr2=0x7fb9881617f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.358+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 shutdown_connections 2024-03-06T07:27:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.358+0000 7fb98f1c9640 1 -- 172.21.15.27:0/655669253 wait complete. 2024-03-06T07:27:09.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.497+0000 7fee81783640 1 Processor -- start 2024-03-06T07:27:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.497+0000 7fee81783640 1 -- start start 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee81783640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fee7c15b570 0x7fee7c15e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee81783640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0ccb60 0x7fee7c0ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee81783640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c0b4090 con 0x7fee7c0cd700 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee81783640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee7c0749b0 con 0x7fee7c15b570 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee81783640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0cd510 con 0x7fee7c0ccb60 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fee7c0cd700 0x7fee7c0742a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49780/0 (socket says 172.21.15.27:49780) 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7affd640 1 -- 172.21.15.27:0/1656418958 learned_addr learned my addr 172.21.15.27:0/1656418958 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7b7fe640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0ccb60 0x7fee7c0ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee727fc640 1 --2- 172.21.15.27:0/1656418958 >> v2:172.21.15.69:3300/0 conn(0x7fee7c15b570 0x7fee7c15e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589628970 0 0) 0x7fee7c0b4090 con 0x7fee7c0cd700 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee68003520 con 0x7fee7c0cd700 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3477953316 0 0) 0x7fee68003520 con 0x7fee7c0cd700 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0ccb60 msgr2=0x7fee7c0ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0ccb60 0x7fee7c0ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 >> v2:172.21.15.69:3300/0 conn(0x7fee7c15b570 msgr2=0x7fee7c15e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 --2- 172.21.15.27:0/1656418958 >> v2:172.21.15.69:3300/0 conn(0x7fee7c15b570 0x7fee7c15e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c0b4350 con 0x7fee7c0cd700 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fee6c001670 con 0x7fee7c0cd700 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 >> v1:172.21.15.27:6789/0 conn(0x7fee7c0cd700 legacy=0x7fee7c0742a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0ccb60 0x7fee680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee68003f30 0x7fee68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0749b0 con 0x7fee68004d40 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0cd510 con 0x7fee68003f30 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6800ba10 con 0x7fee7c0ccb60 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.498+0000 7fee7b7fe640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.499+0000 7fee7affd640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0ccb60 0x7fee680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.499+0000 7fee727fc640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee68003f30 0x7fee68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.499+0000 7fee7b7fe640 1 -- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0ccb60 msgr2=0x7fee680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.499+0000 7fee7b7fe640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0ccb60 0x7fee680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.499+0000 7fee7b7fe640 1 -- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee68003f30 msgr2=0x7fee68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.499+0000 7fee7b7fe640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee68003f30 0x7fee68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.499+0000 7fee7b7fe640 1 -- 172.21.15.27:0/1656418958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee68003520 con 0x7fee68004d40 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.499+0000 7fee7b7fe640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee680051c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fee7400e170 tx=0x7fee68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fa01e2b13d79fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee74039070 con 0x7fee68004d40 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7402cbf0 con 0x7fee68004d40 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee7a7fc640 1 -- 172.21.15.27:0/1656418958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee74009c10 con 0x7fee68004d40 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee81783640 1 -- 172.21.15.27:0/1656418958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 msgr2=0x7fee680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee81783640 1 --2- 172.21.15.27:0/1656418958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee680051c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fee7400e170 tx=0x7fee68005830 comp rx=0 tx=0).stop 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee7b7fe640 1 -- 172.21.15.27:0/1656418958 reap_dead start 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee81783640 1 -- 172.21.15.27:0/1656418958 shutdown_connections 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee81783640 1 -- 172.21.15.27:0/1656418958 >> 172.21.15.27:0/1656418958 conn(0x7fee7c06af20 msgr2=0x7fee7c06b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee81783640 1 -- 172.21.15.27:0/1656418958 shutdown_connections 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.500+0000 7fee81783640 1 -- 172.21.15.27:0/1656418958 wait complete. 2024-03-06T07:27:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.501+0000 7fee81783640 1 Processor -- start 2024-03-06T07:27:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.502+0000 7fee81783640 1 -- start start 2024-03-06T07:27:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.502+0000 7fee81783640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee7c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.502+0000 7fee81783640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee7c1eeec0 0x7fee7c1ef340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.502+0000 7fee81783640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0b0590 0x7fee7c0b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.502+0000 7fee81783640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0b3d80 con 0x7fee68004d40 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.502+0000 7fee81783640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0b4910 con 0x7fee7c1eeec0 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.502+0000 7fee81783640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0ccfd0 con 0x7fee7c0b0590 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.502+0000 7fee7affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee7c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.502+0000 7fee7affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee7c0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52604/0 (socket says 172.21.15.27:52604) 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee7affd640 1 -- 172.21.15.27:0/1345789409 learned_addr learned my addr 172.21.15.27:0/1345789409 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee7b7fe640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0b0590 0x7fee7c0b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee727fc640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee7c1eeec0 0x7fee7c1ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee7affd640 1 -- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0b0590 msgr2=0x7fee7c0b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee7affd640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0b0590 0x7fee7c0b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee7affd640 1 -- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee7c1eeec0 msgr2=0x7fee7c1ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee7affd640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee7c1eeec0 0x7fee7c1ef340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee7affd640 1 -- 172.21.15.27:0/1345789409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c09ac90 con 0x7fee68004d40 2024-03-06T07:27:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee727fc640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee7c1eeec0 0x7fee7c1ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee7affd640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee7c0cfa90 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fee6c007310 tx=0x7fee6c007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee73fff640 1 -- 172.21.15.27:0/1345789409 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c00f9c0 con 0x7fee68004d40 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee73fff640 1 -- 172.21.15.27:0/1345789409 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6c018070 con 0x7fee68004d40 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee73fff640 1 -- 172.21.15.27:0/1345789409 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c00fda0 con 0x7fee68004d40 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7c09af70 con 0x7fee68004d40 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.503+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7c09b530 con 0x7fee68004d40 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.504+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7c09b730 con 0x7fee68004d40 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.505+0000 7fee73fff640 1 -- 172.21.15.27:0/1345789409 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee6c01d460 con 0x7fee68004d40 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.505+0000 7fee73fff640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee5c06b500 0x7fee5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.505+0000 7fee73fff640 1 -- 172.21.15.27:0/1345789409 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fee6c08a290 con 0x7fee68004d40 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.505+0000 7fee727fc640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee5c06b500 0x7fee5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.506+0000 7fee727fc640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee5c06b500 0x7fee5c06d9c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fee7c09c040 tx=0x7fee64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.510+0000 7fee73fff640 1 -- 172.21.15.27:0/1345789409 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee7c09b730 con 0x7fee68004d40 2024-03-06T07:27:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:09.728+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fee7c04f410 con 0x7fee68004d40 2024-03-06T07:27:10.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.382+0000 7fee73fff640 1 -- 172.21.15.27:0/1345789409 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v48) v1 ==== 174+0+0 (secure 0 0 0) 0x7fee6c059200 con 0x7fee68004d40 2024-03-06T07:27:10.387 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:27:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.393+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee5c06b500 msgr2=0x7fee5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.393+0000 7fee81783640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee5c06b500 0x7fee5c06d9c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fee7c09c040 tx=0x7fee64008040 comp rx=0 tx=0).stop 2024-03-06T07:27:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.393+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 msgr2=0x7fee7c0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.393+0000 7fee81783640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee7c0cfa90 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fee6c007310 tx=0x7fee6c007710 comp rx=0 tx=0).stop 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.394+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 shutdown_connections 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.394+0000 7fee81783640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee5c06b500 0x7fee5c06d9c0 secure :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fee7c09c040 tx=0x7fee64008040 comp rx=0 tx=0).stop 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.394+0000 7fee81783640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee7c0b0590 0x7fee7c0b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.394+0000 7fee81783640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee7c1eeec0 0x7fee7c1ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.394+0000 7fee81783640 1 --2- 172.21.15.27:0/1345789409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee68004d40 0x7fee7c0cfa90 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.394+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 >> 172.21.15.27:0/1345789409 conn(0x7fee7c06af20 msgr2=0x7fee7c15c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.400+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 shutdown_connections 2024-03-06T07:27:10.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.401+0000 7fee81783640 1 -- 172.21.15.27:0/1345789409 wait complete. 2024-03-06T07:27:10.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:10.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.529+0000 7fe821a40640 1 Processor -- start 2024-03-06T07:27:10.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.529+0000 7fe821a40640 1 -- start start 2024-03-06T07:27:10.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.530+0000 7fe821a40640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe81c0d3400 0x7fe81c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.530+0000 7fe821a40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 0x7fe81c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.530+0000 7fe821a40640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81c0ce610 con 0x7fe81c071540 2024-03-06T07:27:10.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.530+0000 7fe821a40640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe81c0d2f70 con 0x7fe81c0d3400 2024-03-06T07:27:10.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.530+0000 7fe821a40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c0d4be0 con 0x7fe81c0b2b50 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.530+0000 7fe81affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe81c071540 0x7fe81c071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49792/0 (socket says 172.21.15.27:49792) 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.530+0000 7fe81affd640 1 -- 172.21.15.27:0/581665770 learned_addr learned my addr 172.21.15.27:0/581665770 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.530+0000 7fe81a7fc640 1 --2- 172.21.15.27:0/581665770 >> v2:172.21.15.69:3300/0 conn(0x7fe81c0d3400 0x7fe81c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.530+0000 7fe81b7fe640 1 --2- 172.21.15.27:0/581665770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 0x7fe81c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.531+0000 7fe819ffb640 1 -- 172.21.15.27:0/581665770 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686299755 0 0) 0x7fe81c0ce610 con 0x7fe81c071540 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.531+0000 7fe819ffb640 1 -- 172.21.15.27:0/581665770 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808003520 con 0x7fe81c071540 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.531+0000 7fe81b7fe640 1 -- 172.21.15.27:0/581665770 >> v2:172.21.15.69:3300/0 conn(0x7fe81c0d3400 msgr2=0x7fe81c0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.531+0000 7fe81b7fe640 1 --2- 172.21.15.27:0/581665770 >> v2:172.21.15.69:3300/0 conn(0x7fe81c0d3400 0x7fe81c0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.531+0000 7fe81b7fe640 1 -- 172.21.15.27:0/581665770 >> v1:172.21.15.27:6789/0 conn(0x7fe81c071540 legacy=0x7fe81c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.531+0000 7fe81b7fe640 1 -- 172.21.15.27:0/581665770 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c0d2920 con 0x7fe81c0b2b50 2024-03-06T07:27:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.531+0000 7fe81a7fc640 1 --2- 172.21.15.27:0/581665770 >> v2:172.21.15.69:3300/0 conn(0x7fe81c0d3400 0x7fe81c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:10.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe81b7fe640 1 --2- 172.21.15.27:0/581665770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 0x7fe81c0b6e20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009d90 tx=0x7fe80c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c36b1152c6d39249 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe819ffb640 1 -- 172.21.15.27:0/581665770 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c039070 con 0x7fe81c0b2b50 2024-03-06T07:27:10.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe819ffb640 1 -- 172.21.15.27:0/581665770 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80c02cc50 con 0x7fe81c0b2b50 2024-03-06T07:27:10.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe819ffb640 1 -- 172.21.15.27:0/581665770 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c034440 con 0x7fe81c0b2b50 2024-03-06T07:27:10.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe821a40640 1 -- 172.21.15.27:0/581665770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 msgr2=0x7fe81c0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe821a40640 1 --2- 172.21.15.27:0/581665770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 0x7fe81c0b6e20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009d90 tx=0x7fe80c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe821a40640 1 -- 172.21.15.27:0/581665770 shutdown_connections 2024-03-06T07:27:10.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe821a40640 1 --2- 172.21.15.27:0/581665770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 0x7fe81c0b6e20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe821a40640 1 --2- 172.21.15.27:0/581665770 >> v2:172.21.15.69:3300/0 conn(0x7fe81c0d3400 0x7fe81c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe821a40640 1 -- 172.21.15.27:0/581665770 >> 172.21.15.27:0/581665770 conn(0x7fe81c0d3ca0 msgr2=0x7fe81c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe821a40640 1 -- 172.21.15.27:0/581665770 shutdown_connections 2024-03-06T07:27:10.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.532+0000 7fe821a40640 1 -- 172.21.15.27:0/581665770 wait complete. 2024-03-06T07:27:10.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.534+0000 7fe821a40640 1 Processor -- start 2024-03-06T07:27:10.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.534+0000 7fe821a40640 1 -- start start 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.534+0000 7fe821a40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c071540 0x7fe81c0cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.534+0000 7fe821a40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 0x7fe81c0ab410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.534+0000 7fe821a40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81c0d3400 0x7fe81c0b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe821a40640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c15e370 con 0x7fe81c071540 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe821a40640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c0ce8f0 con 0x7fe81c0d3400 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe821a40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c0ccd40 con 0x7fe81c0b2b50 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe81affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c071540 0x7fe81c0cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe81affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c071540 0x7fe81c0cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52612/0 (socket says 172.21.15.27:52612) 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe81affd640 1 -- 172.21.15.27:0/2087778422 learned_addr learned my addr 172.21.15.27:0/2087778422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe81affd640 1 -- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 msgr2=0x7fe81c0ab410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe81affd640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 0x7fe81c0ab410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe81affd640 1 -- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81c0d3400 msgr2=0x7fe81c0b7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe81affd640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81c0d3400 0x7fe81c0b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe81affd640 1 -- 172.21.15.27:0/2087778422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c0cc780 con 0x7fe81c071540 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe81affd640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c071540 0x7fe81c0cbdc0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe81400c430 tx=0x7fe81400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe813fff640 1 -- 172.21.15.27:0/2087778422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe814017070 con 0x7fe81c071540 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe813fff640 1 -- 172.21.15.27:0/2087778422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe814005ce0 con 0x7fe81c071540 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81c0ccac0 con 0x7fe81c071540 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe813fff640 1 -- 172.21.15.27:0/2087778422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8140072f0 con 0x7fe81c071540 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.535+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81c0cd7d0 con 0x7fe81c071540 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.536+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c068700 con 0x7fe81c071540 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.537+0000 7fe813fff640 1 -- 172.21.15.27:0/2087778422 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe81401a070 con 0x7fe81c071540 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.537+0000 7fe813fff640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7fc06b500 0x7fe7fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.537+0000 7fe813fff640 1 -- 172.21.15.27:0/2087778422 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe814089610 con 0x7fe81c071540 2024-03-06T07:27:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.537+0000 7fe81a7fc640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7fc06b500 0x7fe7fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.538+0000 7fe81a7fc640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7fc06b500 0x7fe7fc06d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe804006e80 tx=0x7fe804006dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.542+0000 7fe813fff640 1 -- 172.21.15.27:0/2087778422 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe81c0ccac0 con 0x7fe81c071540 2024-03-06T07:27:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.723+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe81c0b9a20 con 0x7fe81c071540 2024-03-06T07:27:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.724+0000 7fe813fff640 1 -- 172.21.15.27:0/2087778422 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16601 (secure 0 0 0) 0x7fe814058520 con 0x7fe81c071540 2024-03-06T07:27:10.754 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:10.754 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:09.728835+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:07.388472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:08.720233+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7fc06b500 msgr2=0x7fe7fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7fc06b500 0x7fe7fc06d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe804006e80 tx=0x7fe804006dd0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c071540 msgr2=0x7fe81c0cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c071540 0x7fe81c0cbdc0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe81400c430 tx=0x7fe81400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 shutdown_connections 2024-03-06T07:27:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7fc06b500 0x7fe7fc06d9c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81c0d3400 0x7fe81c0b7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe81c0b2b50 0x7fe81c0ab410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 --2- 172.21.15.27:0/2087778422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c071540 0x7fe81c0cbdc0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 >> 172.21.15.27:0/2087778422 conn(0x7fe81c0d3ca0 msgr2=0x7fe81c0b5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 shutdown_connections 2024-03-06T07:27:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.726+0000 7fe821a40640 1 -- 172.21.15.27:0/2087778422 wait complete. 2024-03-06T07:27:10.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:10.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.862+0000 7f2fbcdf7640 1 Processor -- start 2024-03-06T07:27:10.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.862+0000 7f2fbcdf7640 1 -- start start 2024-03-06T07:27:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fbcdf7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2fb815bd20 0x7f2fb80cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fbcdf7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fb8072810 0x7f2fb806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fbcdf7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb8071a50 con 0x7f2fb815c100 2024-03-06T07:27:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fbcdf7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb806dd80 con 0x7f2fb8072810 2024-03-06T07:27:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fbcdf7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb8071d30 con 0x7f2fb815bd20 2024-03-06T07:27:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fb5d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fb8072810 0x7f2fb806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fb5d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fb8072810 0x7f2fb806d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52614/0 (socket says 172.21.15.27:52614) 2024-03-06T07:27:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fb5d9b640 1 -- 172.21.15.27:0/4002473435 learned_addr learned my addr 172.21.15.27:0/4002473435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fb659c640 1 --2- 172.21.15.27:0/4002473435 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815bd20 0x7f2fb80cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fb5d9b640 1 -- 172.21.15.27:0/4002473435 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815bd20 msgr2=0x7f2fb80cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fb5d9b640 1 --2- 172.21.15.27:0/4002473435 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815bd20 0x7f2fb80cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fb5d9b640 1 -- 172.21.15.27:0/4002473435 >> v1:172.21.15.69:6789/0 conn(0x7f2fb815c100 legacy=0x7f2fb815c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fb5d9b640 1 -- 172.21.15.27:0/4002473435 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb8072010 con 0x7f2fb8072810 2024-03-06T07:27:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.863+0000 7f2fb659c640 1 --2- 172.21.15.27:0/4002473435 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815bd20 0x7f2fb80cf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.864+0000 7f2fb5d9b640 1 --2- 172.21.15.27:0/4002473435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 0x7f2fb806d840 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80740f0 tx=0x7f2fac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bce1b21e17898c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.864+0000 7f2fb559a640 1 -- 172.21.15.27:0/4002473435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac035020 con 0x7f2fb8072810 2024-03-06T07:27:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.864+0000 7f2fb559a640 1 -- 172.21.15.27:0/4002473435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac031870 con 0x7f2fb8072810 2024-03-06T07:27:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.864+0000 7f2fb559a640 1 -- 172.21.15.27:0/4002473435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac031c40 con 0x7f2fb8072810 2024-03-06T07:27:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.865+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/4002473435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 msgr2=0x7f2fb806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.865+0000 7f2fbcdf7640 1 --2- 172.21.15.27:0/4002473435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 0x7f2fb806d840 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80740f0 tx=0x7f2fac02b910 comp rx=0 tx=0).stop 2024-03-06T07:27:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.865+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/4002473435 shutdown_connections 2024-03-06T07:27:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.865+0000 7f2fbcdf7640 1 --2- 172.21.15.27:0/4002473435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 0x7f2fb806d840 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.865+0000 7f2fbcdf7640 1 --2- 172.21.15.27:0/4002473435 >> v2:172.21.15.69:3300/0 conn(0x7f2fb815bd20 0x7f2fb80cf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.865+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/4002473435 >> 172.21.15.27:0/4002473435 conn(0x7f2fb806cd20 msgr2=0x7f2fb806d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.865+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/4002473435 shutdown_connections 2024-03-06T07:27:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.865+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/4002473435 wait complete. 2024-03-06T07:27:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.866+0000 7f2fbcdf7640 1 Processor -- start 2024-03-06T07:27:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.866+0000 7f2fbcdf7640 1 -- start start 2024-03-06T07:27:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fbcdf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 0x7f2fb81f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fbcdf7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb815bd20 0x7f2fb8151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fbcdf7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb815c100 0x7f2fb8152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fbcdf7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb815ce40 con 0x7f2fb8072810 2024-03-06T07:27:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fbcdf7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb815ccc0 con 0x7f2fb815bd20 2024-03-06T07:27:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fbcdf7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb815cfc0 con 0x7f2fb815c100 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fb659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 0x7f2fb81f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fb659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 0x7f2fb81f2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52618/0 (socket says 172.21.15.27:52618) 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fb659c640 1 -- 172.21.15.27:0/3023259898 learned_addr learned my addr 172.21.15.27:0/3023259898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fb5d9b640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb815bd20 0x7f2fb8151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.867+0000 7f2fb6d9d640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb815c100 0x7f2fb8152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fb659c640 1 -- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb815c100 msgr2=0x7f2fb8152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fb659c640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb815c100 0x7f2fb8152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fb659c640 1 -- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb815bd20 msgr2=0x7f2fb8151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fb659c640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb815bd20 0x7f2fb8151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fb659c640 1 -- 172.21.15.27:0/3023259898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb81f2670 con 0x7f2fb8072810 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fb6d9d640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb815c100 0x7f2fb8152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fb659c640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 0x7f2fb81f2130 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000e810 tx=0x7f2fb000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fa6ffd640 1 -- 172.21.15.27:0/3023259898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb000f040 con 0x7f2fb8072810 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fa6ffd640 1 -- 172.21.15.27:0/3023259898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb0004030 con 0x7f2fb8072810 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fa6ffd640 1 -- 172.21.15.27:0/3023259898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0004400 con 0x7f2fb8072810 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb8150640 con 0x7f2fb8072810 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.868+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb8150c00 con 0x7f2fb8072810 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.869+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb8067ff0 con 0x7f2fb8072810 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.869+0000 7f2fa6ffd640 1 -- 172.21.15.27:0/3023259898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fb000c6d0 con 0x7f2fb8072810 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.870+0000 7f2fa6ffd640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f9806b500 0x7f2f9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.870+0000 7f2fa6ffd640 1 -- 172.21.15.27:0/3023259898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fb0089ef0 con 0x7f2fb8072810 2024-03-06T07:27:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.870+0000 7f2fb5d9b640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f9806b500 0x7f2f9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.871+0000 7f2fb5d9b640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f9806b500 0x7f2f9806d9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2fac036000 tx=0x7f2fac03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:10.874+0000 7f2fa6ffd640 1 -- 172.21.15.27:0/3023259898 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fb81f2670 con 0x7f2fb8072810 2024-03-06T07:27:11.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.056+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fb8151250 con 0x7f2fb8072810 2024-03-06T07:27:11.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.057+0000 7f2fa6ffd640 1 -- 172.21.15.27:0/3023259898 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16601 (secure 0 0 0) 0x7f2fb0017020 con 0x7f2fb8072810 2024-03-06T07:27:11.059 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:11.059 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:09.728835+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:07.388472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:08.720233+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f9806b500 msgr2=0x7f2f9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f9806b500 0x7f2f9806d9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2fac036000 tx=0x7f2fac03c040 comp rx=0 tx=0).stop 2024-03-06T07:27:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 msgr2=0x7f2fb81f2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 0x7f2fb81f2130 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000e810 tx=0x7f2fb000ecd0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 shutdown_connections 2024-03-06T07:27:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f9806b500 0x7f2f9806d9c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fb815c100 0x7f2fb8152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fb815bd20 0x7f2fb8151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 --2- 172.21.15.27:0/3023259898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb8072810 0x7f2fb81f2130 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 >> 172.21.15.27:0/3023259898 conn(0x7f2fb806cd20 msgr2=0x7f2fb80cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 shutdown_connections 2024-03-06T07:27:11.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.059+0000 7f2fbcdf7640 1 -- 172.21.15.27:0/3023259898 wait complete. 2024-03-06T07:27:11.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:11.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.180+0000 7fa68ca15640 1 Processor -- start 2024-03-06T07:27:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.180+0000 7fa68ca15640 1 -- start start 2024-03-06T07:27:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.181+0000 7fa68ca15640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6881618e0 0x7fa68806b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.181+0000 7fa68ca15640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa688162b60 con 0x7fa6880ce4f0 2024-03-06T07:27:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.181+0000 7fa68ca15640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa68815f730 con 0x7fa6880ce960 2024-03-06T07:27:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.181+0000 7fa68ca15640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6880cfce0 con 0x7fa6881618e0 2024-03-06T07:27:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.181+0000 7fa685d9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6880ce4f0 0x7fa688161d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49804/0 (socket says 172.21.15.27:49804) 2024-03-06T07:27:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.181+0000 7fa685d9b640 1 -- 172.21.15.27:0/3511491846 learned_addr learned my addr 172.21.15.27:0/3511491846 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.181+0000 7fa68659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6881618e0 0x7fa68806b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.182+0000 7fa68659c640 1 -- 172.21.15.27:0/3511491846 >> v1:172.21.15.69:6789/0 conn(0x7fa6880ce960 legacy=0x7fa688162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.182+0000 7fa68659c640 1 -- 172.21.15.27:0/3511491846 >> v1:172.21.15.27:6789/0 conn(0x7fa6880ce4f0 legacy=0x7fa688161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.182+0000 7fa68659c640 1 -- 172.21.15.27:0/3511491846 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68815ebb0 con 0x7fa6881618e0 2024-03-06T07:27:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.182+0000 7fa68559a640 1 -- 172.21.15.27:0/3511491846 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222264539 0 0) 0x7fa688162b60 con 0x7fa6880ce4f0 2024-03-06T07:27:11.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.182+0000 7fa68659c640 1 --2- 172.21.15.27:0/3511491846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6881618e0 0x7fa68806b130 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fa678009d90 tx=0x7fa67802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2a790a5e381037 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68559a640 1 -- 172.21.15.27:0/3511491846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678039070 con 0x7fa6881618e0 2024-03-06T07:27:11.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68559a640 1 -- 172.21.15.27:0/3511491846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67802cd00 con 0x7fa6881618e0 2024-03-06T07:27:11.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68559a640 1 -- 172.21.15.27:0/3511491846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678034660 con 0x7fa6881618e0 2024-03-06T07:27:11.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68ca15640 1 -- 172.21.15.27:0/3511491846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6881618e0 msgr2=0x7fa68806b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68ca15640 1 --2- 172.21.15.27:0/3511491846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6881618e0 0x7fa68806b130 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fa678009d90 tx=0x7fa67802bf50 comp rx=0 tx=0).stop 2024-03-06T07:27:11.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68ca15640 1 -- 172.21.15.27:0/3511491846 shutdown_connections 2024-03-06T07:27:11.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68ca15640 1 --2- 172.21.15.27:0/3511491846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6881618e0 0x7fa68806b130 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68ca15640 1 -- 172.21.15.27:0/3511491846 >> 172.21.15.27:0/3511491846 conn(0x7fa6880b3660 msgr2=0x7fa6880b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68ca15640 1 -- 172.21.15.27:0/3511491846 shutdown_connections 2024-03-06T07:27:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.183+0000 7fa68ca15640 1 -- 172.21.15.27:0/3511491846 wait complete. 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.184+0000 7fa68ca15640 1 Processor -- start 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.184+0000 7fa68ca15640 1 -- start start 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.185+0000 7fa68ca15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6880ce4f0 0x7fa6880b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.185+0000 7fa68ca15640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6880ce960 0x7fa6880c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.185+0000 7fa68ca15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6880c79f0 0x7fa68806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.185+0000 7fa68ca15640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68815f450 con 0x7fa6880ce4f0 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.185+0000 7fa68ca15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68815f170 con 0x7fa6880c79f0 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.185+0000 7fa68ca15640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68806b670 con 0x7fa6880ce960 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.185+0000 7fa68659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6880ce4f0 0x7fa6880b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.185+0000 7fa68659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6880ce4f0 0x7fa6880b5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52634/0 (socket says 172.21.15.27:52634) 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.185+0000 7fa68659c640 1 -- 172.21.15.27:0/2942919024 learned_addr learned my addr 172.21.15.27:0/2942919024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa685d9b640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6880ce960 0x7fa6880c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa686d9d640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6880c79f0 0x7fa68806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa68659c640 1 -- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6880ce960 msgr2=0x7fa6880c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa68659c640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6880ce960 0x7fa6880c74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa68659c640 1 -- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6880c79f0 msgr2=0x7fa68806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa68659c640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6880c79f0 0x7fa68806fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa68659c640 1 -- 172.21.15.27:0/2942919024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6880c7f70 con 0x7fa6880ce4f0 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa686d9d640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6880c79f0 0x7fa68806fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa68659c640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6880ce4f0 0x7fa6880b5df0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa678009d90 tx=0x7fa678004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa67effd640 1 -- 172.21.15.27:0/2942919024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678044020 con 0x7fa6880ce4f0 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa67effd640 1 -- 172.21.15.27:0/2942919024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67803f670 con 0x7fa6880ce4f0 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6880c8250 con 0x7fa6880ce4f0 2024-03-06T07:27:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa67effd640 1 -- 172.21.15.27:0/2942919024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678039040 con 0x7fa6880ce4f0 2024-03-06T07:27:11.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.186+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6880c5ae0 con 0x7fa6880ce4f0 2024-03-06T07:27:11.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.187+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa688074980 con 0x7fa6880ce4f0 2024-03-06T07:27:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.187+0000 7fa67effd640 1 -- 172.21.15.27:0/2942919024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa678047020 con 0x7fa6880ce4f0 2024-03-06T07:27:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.188+0000 7fa67effd640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66806b4d0 0x7fa66806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.188+0000 7fa67effd640 1 -- 172.21.15.27:0/2942919024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6780ace60 con 0x7fa6880ce4f0 2024-03-06T07:27:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.188+0000 7fa685d9b640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66806b4d0 0x7fa66806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.189+0000 7fa685d9b640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66806b4d0 0x7fa66806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa6880b41f0 tx=0x7fa670009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.192+0000 7fa67effd640 1 -- 172.21.15.27:0/2942919024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa688074980 con 0x7fa6880ce4f0 2024-03-06T07:27:11.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:11.414+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa68809fc50 con 0x7fa6880ce4f0 2024-03-06T07:27:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.398+0000 7fa67effd640 1 -- 172.21.15.27:0/2942919024 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v51) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa67804e030 con 0x7fa6880ce4f0 2024-03-06T07:27:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66806b4d0 msgr2=0x7fa66806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66806b4d0 0x7fa66806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa6880b41f0 tx=0x7fa670009040 comp rx=0 tx=0).stop 2024-03-06T07:27:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6880ce4f0 msgr2=0x7fa6880b5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6880ce4f0 0x7fa6880b5df0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa678009d90 tx=0x7fa678004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 shutdown_connections 2024-03-06T07:27:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa66806b4d0 0x7fa66806d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6880c79f0 0x7fa68806fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6880ce960 0x7fa6880c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 --2- 172.21.15.27:0/2942919024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6880ce4f0 0x7fa6880b5df0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 >> 172.21.15.27:0/2942919024 conn(0x7fa6880b3660 msgr2=0x7fa6880ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 shutdown_connections 2024-03-06T07:27:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.400+0000 7fa68ca15640 1 -- 172.21.15.27:0/2942919024 wait complete. 2024-03-06T07:27:12.413 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:12.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.521+0000 7f7a25b3e640 1 Processor -- start 2024-03-06T07:27:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.521+0000 7f7a25b3e640 1 -- start start 2024-03-06T07:27:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.522+0000 7f7a25b3e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7a2015ae50 0x7f7a200708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.522+0000 7f7a25b3e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a20070e20 con 0x7f7a20071870 2024-03-06T07:27:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.522+0000 7f7a25b3e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200b2e30 con 0x7f7a2015b230 2024-03-06T07:27:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.522+0000 7f7a25b3e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2006dfe0 con 0x7f7a2015ae50 2024-03-06T07:27:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.523+0000 7f7a1f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7a2015ae50 0x7f7a200708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.523+0000 7f7a1ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7a20071870 0x7f7a20150a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49816/0 (socket says 172.21.15.27:49816) 2024-03-06T07:27:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.523+0000 7f7a1ffff640 1 -- 172.21.15.27:0/4045648127 learned_addr learned my addr 172.21.15.27:0/4045648127 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.523+0000 7f7a1f7fe640 1 -- 172.21.15.27:0/4045648127 >> v1:172.21.15.69:6789/0 conn(0x7f7a2015b230 legacy=0x7f7a20071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.523+0000 7f7a1f7fe640 1 -- 172.21.15.27:0/4045648127 >> v1:172.21.15.27:6789/0 conn(0x7f7a20071870 legacy=0x7f7a20150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.523+0000 7f7a1f7fe640 1 -- 172.21.15.27:0/4045648127 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a200b30b0 con 0x7f7a2015ae50 2024-03-06T07:27:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.523+0000 7f7a1e7fc640 1 -- 172.21.15.27:0/4045648127 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614116107 0 0) 0x7f7a20070e20 con 0x7f7a20071870 2024-03-06T07:27:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.524+0000 7f7a1f7fe640 1 --2- 172.21.15.27:0/4045648127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 0x7f7a200708e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7a100097a0 tx=0x7f7a1002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1165a7de0c80962 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.524+0000 7f7a1e7fc640 1 -- 172.21.15.27:0/4045648127 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100315a0 con 0x7f7a2015ae50 2024-03-06T07:27:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.524+0000 7f7a1e7fc640 1 -- 172.21.15.27:0/4045648127 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a10037070 con 0x7f7a2015ae50 2024-03-06T07:27:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.524+0000 7f7a1e7fc640 1 -- 172.21.15.27:0/4045648127 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10031960 con 0x7f7a2015ae50 2024-03-06T07:27:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.525+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4045648127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 msgr2=0x7f7a200708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.525+0000 7f7a25b3e640 1 --2- 172.21.15.27:0/4045648127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 0x7f7a200708e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7a100097a0 tx=0x7f7a1002b840 comp rx=0 tx=0).stop 2024-03-06T07:27:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.525+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4045648127 shutdown_connections 2024-03-06T07:27:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.525+0000 7f7a25b3e640 1 --2- 172.21.15.27:0/4045648127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 0x7f7a200708e0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.525+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4045648127 >> 172.21.15.27:0/4045648127 conn(0x7f7a20066d00 msgr2=0x7f7a20067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.525+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4045648127 shutdown_connections 2024-03-06T07:27:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.525+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4045648127 wait complete. 2024-03-06T07:27:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.526+0000 7f7a25b3e640 1 Processor -- start 2024-03-06T07:27:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.526+0000 7f7a25b3e640 1 -- start start 2024-03-06T07:27:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a25b3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a20071870 0x7f7a2014adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a25b3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 0x7f7a2014c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a25b3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a2015b230 0x7f7a2015cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a25b3e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200b3950 con 0x7f7a2015ae50 2024-03-06T07:27:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a25b3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200b3670 con 0x7f7a2015b230 2024-03-06T07:27:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a25b3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200b4150 con 0x7f7a20071870 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 0x7f7a2014c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 0x7f7a2014c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52646/0 (socket says 172.21.15.27:52646) 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1effd640 1 -- 172.21.15.27:0/4083158496 learned_addr learned my addr 172.21.15.27:0/4083158496 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1f7fe640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a20071870 0x7f7a2014adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1ffff640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a2015b230 0x7f7a2015cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1effd640 1 -- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a20071870 msgr2=0x7f7a2014adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1effd640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a20071870 0x7f7a2014adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1effd640 1 -- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a2015b230 msgr2=0x7f7a2015cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1effd640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a2015b230 0x7f7a2015cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1effd640 1 -- 172.21.15.27:0/4083158496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2014b2f0 con 0x7f7a2015ae50 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.527+0000 7f7a1ffff640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a2015b230 0x7f7a2015cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.528+0000 7f7a1effd640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 0x7f7a2014c660 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f7a1400c430 tx=0x7f7a1400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.528+0000 7f7a07fff640 1 -- 172.21.15.27:0/4083158496 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14011790 con 0x7f7a2015ae50 2024-03-06T07:27:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.528+0000 7f7a07fff640 1 -- 172.21.15.27:0/4083158496 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a14017070 con 0x7f7a2015ae50 2024-03-06T07:27:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.528+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2014b5c0 con 0x7f7a2015ae50 2024-03-06T07:27:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.528+0000 7f7a07fff640 1 -- 172.21.15.27:0/4083158496 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14011b70 con 0x7f7a2015ae50 2024-03-06T07:27:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.528+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2014be60 con 0x7f7a2015ae50 2024-03-06T07:27:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.529+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a200733c0 con 0x7f7a2015ae50 2024-03-06T07:27:12.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.530+0000 7f7a07fff640 1 -- 172.21.15.27:0/4083158496 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a14011d10 con 0x7f7a2015ae50 2024-03-06T07:27:12.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.530+0000 7f7a07fff640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.530+0000 7f7a07fff640 1 -- 172.21.15.27:0/4083158496 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a1408a0b0 con 0x7f7a2015ae50 2024-03-06T07:27:12.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.531+0000 7f7a1f7fe640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.531+0000 7f7a1f7fe640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7a10036000 tx=0x7f7a1003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.535+0000 7f7a07fff640 1 -- 172.21.15.27:0/4083158496 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a14058f30 con 0x7f7a2015ae50 2024-03-06T07:27:12.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:12.756+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7a2004f410 con 0x7f7a2015ae50 2024-03-06T07:27:13.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.399+0000 7f7a07fff640 1 -- 172.21.15.27:0/4083158496 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7a14058830 con 0x7f7a2015ae50 2024-03-06T07:27:13.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a0c06b4d0 msgr2=0x7f7a0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7a10036000 tx=0x7f7a1003d040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 msgr2=0x7f7a2014c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 0x7f7a2014c660 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f7a1400c430 tx=0x7f7a1400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 shutdown_connections 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7a0c06b4d0 0x7f7a0c06d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a2015b230 0x7f7a2015cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a2015ae50 0x7f7a2014c660 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 --2- 172.21.15.27:0/4083158496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7a20071870 0x7f7a2014adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 >> 172.21.15.27:0/4083158496 conn(0x7f7a20066d00 msgr2=0x7f7a2015e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 shutdown_connections 2024-03-06T07:27:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.401+0000 7f7a25b3e640 1 -- 172.21.15.27:0/4083158496 wait complete. 2024-03-06T07:27:13.413 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:13.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.519+0000 7f740760e640 1 Processor -- start 2024-03-06T07:27:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.519+0000 7f740760e640 1 -- start start 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f740760e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74000b9a10 0x7f74000d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f740760e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74000cea30 con 0x7f74000b95a0 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f740760e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74000cf3c0 con 0x7f740015df00 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f740760e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74000cf0e0 con 0x7f74000b9a10 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f7405bab640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74000b9a10 0x7f74000d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f7404ba9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f74000b95a0 0x7f74000b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49826/0 (socket says 172.21.15.27:49826) 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f7404ba9640 1 -- 172.21.15.27:0/946881705 learned_addr learned my addr 172.21.15.27:0/946881705 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f7405bab640 1 -- 172.21.15.27:0/946881705 >> v1:172.21.15.69:6789/0 conn(0x7f740015df00 legacy=0x7f74000d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f7405bab640 1 -- 172.21.15.27:0/946881705 >> v1:172.21.15.27:6789/0 conn(0x7f74000b95a0 legacy=0x7f74000b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.520+0000 7f7405bab640 1 -- 172.21.15.27:0/946881705 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74000ce470 con 0x7f74000b9a10 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.521+0000 7f7405bab640 1 --2- 172.21.15.27:0/946881705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74000b9a10 0x7f74000d3220 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f74000cf590 tx=0x7f73f802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43778336a8b21d0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.521+0000 7f73f7fff640 1 -- 172.21.15.27:0/946881705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8039070 con 0x7f74000b9a10 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.521+0000 7f73f7fff640 1 -- 172.21.15.27:0/946881705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f802cce0 con 0x7f74000b9a10 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.521+0000 7f73f7fff640 1 -- 172.21.15.27:0/946881705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8034660 con 0x7f74000b9a10 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.522+0000 7f740760e640 1 -- 172.21.15.27:0/946881705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74000b9a10 msgr2=0x7f74000d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.522+0000 7f740760e640 1 --2- 172.21.15.27:0/946881705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74000b9a10 0x7f74000d3220 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f74000cf590 tx=0x7f73f802bf30 comp rx=0 tx=0).stop 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.522+0000 7f740760e640 1 -- 172.21.15.27:0/946881705 shutdown_connections 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.522+0000 7f740760e640 1 --2- 172.21.15.27:0/946881705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74000b9a10 0x7f74000d3220 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.522+0000 7f740760e640 1 -- 172.21.15.27:0/946881705 >> 172.21.15.27:0/946881705 conn(0x7f7400071960 msgr2=0x7f7400071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.522+0000 7f740760e640 1 -- 172.21.15.27:0/946881705 shutdown_connections 2024-03-06T07:27:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.522+0000 7f740760e640 1 -- 172.21.15.27:0/946881705 wait complete. 2024-03-06T07:27:13.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.523+0000 7f740760e640 1 Processor -- start 2024-03-06T07:27:13.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.523+0000 7f740760e640 1 -- start start 2024-03-06T07:27:13.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f740760e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74000b95a0 0x7f74001f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f740760e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74000b9a10 0x7f740015ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f740760e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f740015df00 0x7f74000b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f740760e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74000cedf0 con 0x7f740015df00 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f740760e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74000cec70 con 0x7f74000b95a0 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f740760e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74000b5e10 con 0x7f74000b9a10 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f7405bab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f740015df00 0x7f74000b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f7405bab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f740015df00 0x7f74000b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52664/0 (socket says 172.21.15.27:52664) 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f7405bab640 1 -- 172.21.15.27:0/1280345522 learned_addr learned my addr 172.21.15.27:0/1280345522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f74053aa640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74000b95a0 0x7f74001f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f7404ba9640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74000b9a10 0x7f740015ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f7405bab640 1 -- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74000b9a10 msgr2=0x7f740015ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f7405bab640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74000b9a10 0x7f740015ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f7405bab640 1 -- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74000b95a0 msgr2=0x7f74001f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f7405bab640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74000b95a0 0x7f74001f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.524+0000 7f7405bab640 1 -- 172.21.15.27:0/1280345522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74000c7950 con 0x7f740015df00 2024-03-06T07:27:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.525+0000 7f7405bab640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f740015df00 0x7f74000b1ed0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f73f802c540 tx=0x7f73f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.525+0000 7f73f5ffb640 1 -- 172.21.15.27:0/1280345522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8039050 con 0x7f740015df00 2024-03-06T07:27:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.525+0000 7f73f5ffb640 1 -- 172.21.15.27:0/1280345522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f8045070 con 0x7f740015df00 2024-03-06T07:27:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.525+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74000c7c30 con 0x7f740015df00 2024-03-06T07:27:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.525+0000 7f73f5ffb640 1 -- 172.21.15.27:0/1280345522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f803e830 con 0x7f740015df00 2024-03-06T07:27:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.525+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74000c81f0 con 0x7f740015df00 2024-03-06T07:27:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.526+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74000d2030 con 0x7f740015df00 2024-03-06T07:27:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.527+0000 7f73f5ffb640 1 -- 172.21.15.27:0/1280345522 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73f803e9d0 con 0x7f740015df00 2024-03-06T07:27:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.527+0000 7f73f5ffb640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73e406b4d0 0x7f73e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.528+0000 7f73f5ffb640 1 -- 172.21.15.27:0/1280345522 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73f80ad1a0 con 0x7f740015df00 2024-03-06T07:27:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.528+0000 7f74053aa640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73e406b4d0 0x7f73e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.528+0000 7f74053aa640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73e406b4d0 0x7f73e406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f73f0000f60 tx=0x7f73f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.532+0000 7f73f5ffb640 1 -- 172.21.15.27:0/1280345522 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73f807c150 con 0x7f740015df00 2024-03-06T07:27:13.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.756+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7400067ff0 con 0x7f740015df00 2024-03-06T07:27:13.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.756+0000 7f73f5ffb640 1 -- 172.21.15.27:0/1280345522 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f73f807ba50 con 0x7f740015df00 2024-03-06T07:27:13.758 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:13.758 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/930650799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":930650799},{"type":"v1","addr":"172.21.15.69:6836","nonce":930650799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1050755919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1050755919},{"type":"v1","addr":"172.21.15.27:6835","nonce":1050755919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1173605349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1173605349},{"type":"v1","addr":"172.21.15.69:6837","nonce":1173605349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:09.729142+0000","modified":"2024-03-06T07:27:12.757645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7032},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7032":{"gid":7032,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3587849188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3587849188},{"type":"v1","addr":"172.21.15.27:6834","nonce":3587849188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7032,"qdb_cluster":[7032]},"id":6}]} 2024-03-06T07:27:13.758 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 52 2024-03-06T07:27:13.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.758+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73e406b4d0 msgr2=0x7f73e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.758+0000 7f740760e640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73e406b4d0 0x7f73e406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f73f0000f60 tx=0x7f73f0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.758+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f740015df00 msgr2=0x7f74000b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.758+0000 7f740760e640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f740015df00 0x7f74000b1ed0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f73f802c540 tx=0x7f73f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:13.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.759+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 shutdown_connections 2024-03-06T07:27:13.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.759+0000 7f740760e640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73e406b4d0 0x7f73e406d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.759+0000 7f740760e640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f740015df00 0x7f74000b1ed0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.759+0000 7f740760e640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74000b9a10 0x7f740015ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.759+0000 7f740760e640 1 --2- 172.21.15.27:0/1280345522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74000b95a0 0x7f74001f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.759+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 >> 172.21.15.27:0/1280345522 conn(0x7f7400071960 msgr2=0x7f74000d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.759+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 shutdown_connections 2024-03-06T07:27:13.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.759+0000 7f740760e640 1 -- 172.21.15.27:0/1280345522 wait complete. 2024-03-06T07:27:13.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:13.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.877+0000 7fe30a3c9640 1 Processor -- start 2024-03-06T07:27:13.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.877+0000 7fe30a3c9640 1 -- start start 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.877+0000 7fe30a3c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe30415df00 0x7fe3040d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.877+0000 7fe30a3c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe3040b9a30 0x7fe3040d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe30a3c9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3040cea30 con 0x7fe3040b95a0 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe30a3c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3040d2af0 con 0x7fe3040b9a30 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe30a3c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040cf3c0 con 0x7fe30415df00 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe3037fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe3040b95a0 0x7fe3040b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43304/0 (socket says 172.21.15.27:43304) 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe3037fe640 1 -- 172.21.15.27:0/1802587061 learned_addr learned my addr 172.21.15.27:0/1802587061 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232205205 0 0) 0x7fe3040cea30 con 0x7fe3040b95a0 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe308966640 1 --2- 172.21.15.27:0/1802587061 >> v2:172.21.15.69:3300/0 conn(0x7fe3040b9a30 0x7fe3040d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe303fff640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe30415df00 0x7fe3040d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2ec003520 con 0x7fe3040b95a0 2024-03-06T07:27:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2172832719 0 0) 0x7fe2ec003520 con 0x7fe3040b95a0 2024-03-06T07:27:13.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe30415df00 msgr2=0x7fe3040d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe30415df00 0x7fe3040d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 >> v2:172.21.15.69:3300/0 conn(0x7fe3040b9a30 msgr2=0x7fe3040d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 --2- 172.21.15.27:0/1802587061 >> v2:172.21.15.69:3300/0 conn(0x7fe3040b9a30 0x7fe3040d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3040cf0e0 con 0x7fe3040b95a0 2024-03-06T07:27:13.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fe2f8001670 con 0x7fe3040b95a0 2024-03-06T07:27:13.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 >> v1:172.21.15.27:6789/0 conn(0x7fe3040b95a0 legacy=0x7fe3040b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2ec0039b0 0x7fe2ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe30415df00 0x7fe2ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2ec004d40 0x7fe2ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040d2af0 con 0x7fe2ec004d40 2024-03-06T07:27:13.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040cf3c0 con 0x7fe2ec0039b0 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec00b9c0 con 0x7fe30415df00 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.878+0000 7fe308966640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2ec004d40 0x7fe2ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.879+0000 7fe303fff640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2ec0039b0 0x7fe2ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.879+0000 7fe3037fe640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe30415df00 0x7fe2ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.879+0000 7fe308966640 1 -- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe30415df00 msgr2=0x7fe2ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.879+0000 7fe308966640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe30415df00 0x7fe2ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.879+0000 7fe308966640 1 -- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2ec0039b0 msgr2=0x7fe2ec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.879+0000 7fe308966640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2ec0039b0 0x7fe2ec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.879+0000 7fe308966640 1 -- 172.21.15.27:0/1802587061 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec003520 con 0x7fe2ec004d40 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe308966640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2ec004d40 0x7fe2ec0051c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009f90 tx=0x7fe2ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7cdb1ec8384d14d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc039070 con 0x7fe2ec004d40 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc02cc20 con 0x7fe2ec004d40 2024-03-06T07:27:13.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe302ffd640 1 -- 172.21.15.27:0/1802587061 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc009c10 con 0x7fe2ec004d40 2024-03-06T07:27:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe30a3c9640 1 -- 172.21.15.27:0/1802587061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2ec004d40 msgr2=0x7fe2ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe30a3c9640 1 --2- 172.21.15.27:0/1802587061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2ec004d40 0x7fe2ec0051c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009f90 tx=0x7fe2ec005830 comp rx=0 tx=0).stop 2024-03-06T07:27:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe308966640 1 -- 172.21.15.27:0/1802587061 reap_dead start 2024-03-06T07:27:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe30a3c9640 1 -- 172.21.15.27:0/1802587061 shutdown_connections 2024-03-06T07:27:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe30a3c9640 1 -- 172.21.15.27:0/1802587061 >> 172.21.15.27:0/1802587061 conn(0x7fe304071960 msgr2=0x7fe304071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe30a3c9640 1 -- 172.21.15.27:0/1802587061 shutdown_connections 2024-03-06T07:27:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.880+0000 7fe30a3c9640 1 -- 172.21.15.27:0/1802587061 wait complete. 2024-03-06T07:27:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.882+0000 7fe30a3c9640 1 Processor -- start 2024-03-06T07:27:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.883+0000 7fe30a3c9640 1 -- start start 2024-03-06T07:27:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe30a3c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30415af60 0x7fe3041f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe30a3c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3041f4860 0x7fe3040d38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe30a3c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3040d3e10 0x7fe30415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe30a3c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040cec70 con 0x7fe30415af60 2024-03-06T07:27:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe30a3c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040ce750 con 0x7fe3040d3e10 2024-03-06T07:27:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe30a3c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040cedf0 con 0x7fe3041f4860 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe303fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30415af60 0x7fe3041f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe303fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30415af60 0x7fe3041f4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34260/0 (socket says 172.21.15.27:34260) 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe303fff640 1 -- 172.21.15.27:0/3125554641 learned_addr learned my addr 172.21.15.27:0/3125554641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe308966640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3040d3e10 0x7fe30415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe3037fe640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3041f4860 0x7fe3040d38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe303fff640 1 -- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3041f4860 msgr2=0x7fe3040d38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe303fff640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3041f4860 0x7fe3040d38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe303fff640 1 -- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3040d3e10 msgr2=0x7fe30415e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe303fff640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3040d3e10 0x7fe30415e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe303fff640 1 -- 172.21.15.27:0/3125554641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3041f4d70 con 0x7fe30415af60 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.884+0000 7fe308966640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3040d3e10 0x7fe30415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.885+0000 7fe303fff640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30415af60 0x7fe3041f4320 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40118e0 tx=0x7fe2f4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.885+0000 7fe300ff9640 1 -- 172.21.15.27:0/3125554641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f400faf0 con 0x7fe30415af60 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.885+0000 7fe300ff9640 1 -- 172.21.15.27:0/3125554641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f400fc90 con 0x7fe30415af60 2024-03-06T07:27:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.885+0000 7fe300ff9640 1 -- 172.21.15.27:0/3125554641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4014440 con 0x7fe30415af60 2024-03-06T07:27:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.885+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3041f5050 con 0x7fe30415af60 2024-03-06T07:27:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.885+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3040c7c30 con 0x7fe30415af60 2024-03-06T07:27:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.886+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3040d4220 con 0x7fe30415af60 2024-03-06T07:27:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.886+0000 7fe300ff9640 1 -- 172.21.15.27:0/3125554641 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2f4012040 con 0x7fe30415af60 2024-03-06T07:27:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.887+0000 7fe300ff9640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2f006b4d0 0x7fe2f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.887+0000 7fe300ff9640 1 -- 172.21.15.27:0/3125554641 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2f408a8c0 con 0x7fe30415af60 2024-03-06T07:27:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.887+0000 7fe3037fe640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2f006b4d0 0x7fe2f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.887+0000 7fe3037fe640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2f006b4d0 0x7fe2f006d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8004840 tx=0x7fe2f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:13.891+0000 7fe300ff9640 1 -- 172.21.15.27:0/3125554641 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3040d4220 con 0x7fe30415af60 2024-03-06T07:27:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.155+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe304067ff0 con 0x7fe30415af60 2024-03-06T07:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.156+0000 7fe300ff9640 1 -- 172.21.15.27:0/3125554641 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16588 (secure 0 0 0) 0x7fe2f4019030 con 0x7fe30415af60 2024-03-06T07:27:14.158 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:14.158 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":84,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:11.381712+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:07.388472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:08.720233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2f006b4d0 msgr2=0x7fe2f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2f006b4d0 0x7fe2f006d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8004840 tx=0x7fe2f8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30415af60 msgr2=0x7fe3041f4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30415af60 0x7fe3041f4320 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40118e0 tx=0x7fe2f4011da0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 shutdown_connections 2024-03-06T07:27:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2f006b4d0 0x7fe2f006d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3040d3e10 0x7fe30415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3041f4860 0x7fe3040d38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 --2- 172.21.15.27:0/3125554641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30415af60 0x7fe3041f4320 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 >> 172.21.15.27:0/3125554641 conn(0x7fe304071960 msgr2=0x7fe3040b8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 shutdown_connections 2024-03-06T07:27:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.158+0000 7fe30a3c9640 1 -- 172.21.15.27:0/3125554641 wait complete. 2024-03-06T07:27:14.171 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:14.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:14.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.276+0000 7f93247e9640 1 Processor -- start 2024-03-06T07:27:14.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.276+0000 7f93247e9640 1 -- start start 2024-03-06T07:27:14.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.277+0000 7f93247e9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f931c15df00 0x7f931c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.277+0000 7f93247e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f931c0b95a0 0x7f931c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.277+0000 7f93247e9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931c0cea30 con 0x7f931c0b9a30 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.277+0000 7f93247e9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931c0d2af0 con 0x7f931c15df00 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.277+0000 7f93247e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931c0cf3c0 con 0x7f931c0b95a0 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.277+0000 7f9322585640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f931c15df00 0x7f931c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.277+0000 7f9322585640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f931c15df00 0x7f931c0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34274/0 (socket says 172.21.15.27:34274) 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.277+0000 7f9322585640 1 -- 172.21.15.27:0/3652178649 learned_addr learned my addr 172.21.15.27:0/3652178649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.278+0000 7f9321d84640 1 --2- 172.21.15.27:0/3652178649 >> v2:172.21.15.69:3300/0 conn(0x7f931c0b95a0 0x7f931c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.278+0000 7f9322585640 1 -- 172.21.15.27:0/3652178649 >> v2:172.21.15.69:3300/0 conn(0x7f931c0b95a0 msgr2=0x7f931c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.278+0000 7f9322585640 1 --2- 172.21.15.27:0/3652178649 >> v2:172.21.15.69:3300/0 conn(0x7f931c0b95a0 0x7f931c0b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.278+0000 7f9322585640 1 -- 172.21.15.27:0/3652178649 >> v1:172.21.15.69:6789/0 conn(0x7f931c0b9a30 legacy=0x7f931c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.278+0000 7f9322585640 1 -- 172.21.15.27:0/3652178649 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c0cf0e0 con 0x7f931c15df00 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.278+0000 7f9322585640 1 --2- 172.21.15.27:0/3652178649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 0x7f931c0d25b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f931c0d1760 tx=0x7f931802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad516a21c2da9127 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f9321583640 1 -- 172.21.15.27:0/3652178649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318035020 con 0x7f931c15df00 2024-03-06T07:27:14.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f9321583640 1 -- 172.21.15.27:0/3652178649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9318031870 con 0x7f931c15df00 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f9321583640 1 -- 172.21.15.27:0/3652178649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318031c40 con 0x7f931c15df00 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f93247e9640 1 -- 172.21.15.27:0/3652178649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 msgr2=0x7f931c0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f93247e9640 1 --2- 172.21.15.27:0/3652178649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 0x7f931c0d25b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f931c0d1760 tx=0x7f931802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f93247e9640 1 -- 172.21.15.27:0/3652178649 shutdown_connections 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f93247e9640 1 --2- 172.21.15.27:0/3652178649 >> v2:172.21.15.69:3300/0 conn(0x7f931c0b95a0 0x7f931c0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f93247e9640 1 --2- 172.21.15.27:0/3652178649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 0x7f931c0d25b0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f93247e9640 1 -- 172.21.15.27:0/3652178649 >> 172.21.15.27:0/3652178649 conn(0x7f931c071960 msgr2=0x7f931c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f93247e9640 1 -- 172.21.15.27:0/3652178649 shutdown_connections 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.279+0000 7f93247e9640 1 -- 172.21.15.27:0/3652178649 wait complete. 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f93247e9640 1 Processor -- start 2024-03-06T07:27:14.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f93247e9640 1 -- start start 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f93247e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931c0b95a0 0x7f931c0cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f93247e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 0x7f931c0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f93247e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f931c155430 0x7f931c0b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f93247e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0cec70 con 0x7f931c15df00 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f93247e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0ce750 con 0x7f931c155430 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f93247e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0cedf0 con 0x7f931c0b95a0 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f9321d84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 0x7f931c0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.281+0000 7f9321d84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 0x7f931c0cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34276/0 (socket says 172.21.15.27:34276) 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9321d84640 1 -- 172.21.15.27:0/2365024912 learned_addr learned my addr 172.21.15.27:0/2365024912 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9322d86640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f931c155430 0x7f931c0b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9322585640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931c0b95a0 0x7f931c0cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9321d84640 1 -- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931c0b95a0 msgr2=0x7f931c0cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9321d84640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931c0b95a0 0x7f931c0cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9321d84640 1 -- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f931c155430 msgr2=0x7f931c0b54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9321d84640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f931c155430 0x7f931c0b54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9321d84640 1 -- 172.21.15.27:0/2365024912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c0cb730 con 0x7f931c15df00 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9322585640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931c0b95a0 0x7f931c0cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9321d84640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 0x7f931c0cad50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f930c00d830 tx=0x7f930c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9312ffd640 1 -- 172.21.15.27:0/2365024912 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c00b6c0 con 0x7f931c15df00 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9312ffd640 1 -- 172.21.15.27:0/2365024912 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c019070 con 0x7f931c15df00 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f9312ffd640 1 -- 172.21.15.27:0/2365024912 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c00ba80 con 0x7f931c15df00 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c0cba10 con 0x7f931c15df00 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.282+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c0cbcf0 con 0x7f931c15df00 2024-03-06T07:27:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.283+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c067ff0 con 0x7f931c15df00 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.284+0000 7f9312ffd640 1 -- 172.21.15.27:0/2365024912 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f930c01e460 con 0x7f931c15df00 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.284+0000 7f9312ffd640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f930006b4d0 0x7f930006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.285+0000 7f9312ffd640 1 -- 172.21.15.27:0/2365024912 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f930c08a7f0 con 0x7f931c15df00 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.285+0000 7f9322585640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f930006b4d0 0x7f930006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.285+0000 7f9322585640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f930006b4d0 0x7f930006d990 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9318036000 tx=0x7f931803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.289+0000 7f9312ffd640 1 -- 172.21.15.27:0/2365024912 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f930c059670 con 0x7f931c15df00 2024-03-06T07:27:14.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.467+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f931c0d30e0 con 0x7f930006b4d0 2024-03-06T07:27:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.474+0000 7f9312ffd640 1 -- 172.21.15.27:0/2365024912 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246760 (secure 0 0 0) 0x7f931c0d30e0 con 0x7f930006b4d0 2024-03-06T07:27:14.478 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:14.479 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f930006b4d0 msgr2=0x7f930006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f930006b4d0 0x7f930006d990 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9318036000 tx=0x7f931803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 msgr2=0x7f931c0cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 0x7f931c0cad50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f930c00d830 tx=0x7f930c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 shutdown_connections 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f930006b4d0 0x7f930006d990 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f931c155430 0x7f931c0b54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f931c15df00 0x7f931c0cad50 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 --2- 172.21.15.27:0/2365024912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f931c0b95a0 0x7f931c0cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.476+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 >> 172.21.15.27:0/2365024912 conn(0x7f931c071960 msgr2=0x7f931c1618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.477+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 shutdown_connections 2024-03-06T07:27:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.477+0000 7f93247e9640 1 -- 172.21.15.27:0/2365024912 wait complete. 2024-03-06T07:27:14.493 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-03-06T07:27:14.050694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258488,"kb_used_data":9020,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716232,"statfs":{"total":773094113280,"available":772829421568,"internally_reserved":0,"allocated":9236480,"data_stored":4569773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.694133"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366331+0000","last_change":"2024-03-06T07:27:09.366331+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366331+0000","last_undegraded":"2024-03-06T07:27:09.366331+0000","last_fullsized":"2024-03-06T07:27:09.366331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365065+0000","last_change":"2024-03-06T07:27:09.365065+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365065+0000","last_undegraded":"2024-03-06T07:27:09.365065+0000","last_fullsized":"2024-03-06T07:27:09.365065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361750+0000","last_change":"2024-03-06T07:27:09.361750+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361750+0000","last_undegraded":"2024-03-06T07:27:09.361750+0000","last_fullsized":"2024-03-06T07:27:09.361750+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368354+0000","last_change":"2024-03-06T07:27:09.368354+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368354+0000","last_undegraded":"2024-03-06T07:27:09.368354+0000","last_fullsized":"2024-03-06T07:27:09.368354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368139+0000","last_change":"2024-03-06T07:27:09.368139+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368139+0000","last_undegraded":"2024-03-06T07:27:09.368139+0000","last_fullsized":"2024-03-06T07:27:09.368139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360754+0000","last_change":"2024-03-06T07:27:09.360754+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360754+0000","last_undegraded":"2024-03-06T07:27:09.360754+0000","last_fullsized":"2024-03-06T07:27:09.360754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366900+0000","last_change":"2024-03-06T07:27:09.366900+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366900+0000","last_undegraded":"2024-03-06T07:27:09.366900+0000","last_fullsized":"2024-03-06T07:27:09.366900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360184+0000","last_change":"2024-03-06T07:27:09.360184+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360184+0000","last_undegraded":"2024-03-06T07:27:09.360184+0000","last_fullsized":"2024-03-06T07:27:09.360184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360603+0000","last_change":"2024-03-06T07:27:09.360603+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360603+0000","last_undegraded":"2024-03-06T07:27:09.360603+0000","last_fullsized":"2024-03-06T07:27:09.360603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364335+0000","last_change":"2024-03-06T07:27:09.364335+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364335+0000","last_undegraded":"2024-03-06T07:27:09.364335+0000","last_fullsized":"2024-03-06T07:27:09.364335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364708+0000","last_change":"2024-03-06T07:27:09.364708+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364708+0000","last_undegraded":"2024-03-06T07:27:09.364708+0000","last_fullsized":"2024-03-06T07:27:09.364708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366392+0000","last_change":"2024-03-06T07:27:09.366392+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366392+0000","last_undegraded":"2024-03-06T07:27:09.366392+0000","last_fullsized":"2024-03-06T07:27:09.366392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361471+0000","last_change":"2024-03-06T07:27:09.361471+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361471+0000","last_undegraded":"2024-03-06T07:27:09.361471+0000","last_fullsized":"2024-03-06T07:27:09.361471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368716+0000","last_change":"2024-03-06T07:27:09.368716+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368716+0000","last_undegraded":"2024-03-06T07:27:09.368716+0000","last_fullsized":"2024-03-06T07:27:09.368716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.362610+0000","last_change":"2024-03-06T07:27:09.362610+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.362610+0000","last_undegraded":"2024-03-06T07:27:09.362610+0000","last_fullsized":"2024-03-06T07:27:09.362610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365314+0000","last_change":"2024-03-06T07:27:09.365314+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365314+0000","last_undegraded":"2024-03-06T07:27:09.365314+0000","last_fullsized":"2024-03-06T07:27:09.365314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364605+0000","last_change":"2024-03-06T07:27:09.364605+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364605+0000","last_undegraded":"2024-03-06T07:27:09.364605+0000","last_fullsized":"2024-03-06T07:27:09.364605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366597+0000","last_change":"2024-03-06T07:27:09.366597+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366597+0000","last_undegraded":"2024-03-06T07:27:09.366597+0000","last_fullsized":"2024-03-06T07:27:09.366597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367523+0000","last_change":"2024-03-06T07:27:09.367523+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367523+0000","last_undegraded":"2024-03-06T07:27:09.367523+0000","last_fullsized":"2024-03-06T07:27:09.367523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367857+0000","last_change":"2024-03-06T07:27:09.367857+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367857+0000","last_undegraded":"2024-03-06T07:27:09.367857+0000","last_fullsized":"2024-03-06T07:27:09.367857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366174+0000","last_change":"2024-03-06T07:27:09.366174+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366174+0000","last_undegraded":"2024-03-06T07:27:09.366174+0000","last_fullsized":"2024-03-06T07:27:09.366174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369885+0000","last_change":"2024-03-06T07:27:09.369885+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369885+0000","last_undegraded":"2024-03-06T07:27:09.369885+0000","last_fullsized":"2024-03-06T07:27:09.369885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367067+0000","last_change":"2024-03-06T07:27:09.367067+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367067+0000","last_undegraded":"2024-03-06T07:27:09.367067+0000","last_fullsized":"2024-03-06T07:27:09.367067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364968+0000","last_change":"2024-03-06T07:27:09.364968+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364968+0000","last_undegraded":"2024-03-06T07:27:09.364968+0000","last_fullsized":"2024-03-06T07:27:09.364968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367295+0000","last_change":"2024-03-06T07:27:09.367295+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367295+0000","last_undegraded":"2024-03-06T07:27:09.367295+0000","last_fullsized":"2024-03-06T07:27:09.367295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360671+0000","last_change":"2024-03-06T07:27:09.360671+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360671+0000","last_undegraded":"2024-03-06T07:27:09.360671+0000","last_fullsized":"2024-03-06T07:27:09.360671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366535+0000","last_change":"2024-03-06T07:27:09.366535+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366535+0000","last_undegraded":"2024-03-06T07:27:09.366535+0000","last_fullsized":"2024-03-06T07:27:09.366535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365953+0000","last_change":"2024-03-06T07:27:09.365953+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365953+0000","last_undegraded":"2024-03-06T07:27:09.365953+0000","last_fullsized":"2024-03-06T07:27:09.365953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369681+0000","last_change":"2024-03-06T07:27:09.369681+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369681+0000","last_undegraded":"2024-03-06T07:27:09.369681+0000","last_fullsized":"2024-03-06T07:27:09.369681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369176+0000","last_change":"2024-03-06T07:27:09.369176+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369176+0000","last_undegraded":"2024-03-06T07:27:09.369176+0000","last_fullsized":"2024-03-06T07:27:09.369176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367187+0000","last_change":"2024-03-06T07:27:09.367187+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367187+0000","last_undegraded":"2024-03-06T07:27:09.367187+0000","last_fullsized":"2024-03-06T07:27:09.367187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.363528+0000","last_change":"2024-03-06T07:27:09.363528+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.363528+0000","last_undegraded":"2024-03-06T07:27:09.363528+0000","last_fullsized":"2024-03-06T07:27:09.363528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369136+0000","last_change":"2024-03-06T07:27:09.369136+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369136+0000","last_undegraded":"2024-03-06T07:27:09.369136+0000","last_fullsized":"2024-03-06T07:27:09.369136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383880+0000","last_change":"2024-03-06T07:27:09.378545+0000","last_active":"2024-03-06T07:27:09.383880+0000","last_peered":"2024-03-06T07:27:09.383880+0000","last_clean":"2024-03-06T07:27:09.383880+0000","last_became_active":"2024-03-06T07:27:09.378276+0000","last_became_peered":"2024-03-06T07:27:09.378276+0000","last_unstale":"2024-03-06T07:27:09.383880+0000","last_undegraded":"2024-03-06T07:27:09.383880+0000","last_fullsized":"2024-03-06T07:27:09.383880+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:09.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365340+0000","last_change":"2024-03-06T07:27:09.365340+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365340+0000","last_undegraded":"2024-03-06T07:27:09.365340+0000","last_fullsized":"2024-03-06T07:27:09.365340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.381029+0000","last_change":"2024-03-06T07:27:09.375904+0000","last_active":"2024-03-06T07:27:09.381029+0000","last_peered":"2024-03-06T07:27:09.381029+0000","last_clean":"2024-03-06T07:27:09.381029+0000","last_became_active":"2024-03-06T07:27:09.375566+0000","last_became_peered":"2024-03-06T07:27:09.375566+0000","last_unstale":"2024-03-06T07:27:09.381029+0000","last_undegraded":"2024-03-06T07:27:09.381029+0000","last_fullsized":"2024-03-06T07:27:09.381029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:07.197604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365542+0000","last_change":"2024-03-06T07:27:09.365542+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365542+0000","last_undegraded":"2024-03-06T07:27:09.365542+0000","last_fullsized":"2024-03-06T07:27:09.365542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374222+0000","last_change":"2024-03-06T07:27:09.371083+0000","last_active":"2024-03-06T07:27:09.374222+0000","last_peered":"2024-03-06T07:27:09.374222+0000","last_clean":"2024-03-06T07:27:09.374222+0000","last_became_active":"2024-03-06T07:27:09.370820+0000","last_became_peered":"2024-03-06T07:27:09.370820+0000","last_unstale":"2024-03-06T07:27:09.374222+0000","last_undegraded":"2024-03-06T07:27:09.374222+0000","last_fullsized":"2024-03-06T07:27:09.374222+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:29.304676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359572+0000","last_change":"2024-03-06T07:27:09.359572+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359572+0000","last_undegraded":"2024-03-06T07:27:09.359572+0000","last_fullsized":"2024-03-06T07:27:09.359572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.378612+0000","last_change":"2024-03-06T07:27:09.374749+0000","last_active":"2024-03-06T07:27:09.378612+0000","last_peered":"2024-03-06T07:27:09.378612+0000","last_clean":"2024-03-06T07:27:09.378612+0000","last_became_active":"2024-03-06T07:27:09.374421+0000","last_became_peered":"2024-03-06T07:27:09.374421+0000","last_unstale":"2024-03-06T07:27:09.378612+0000","last_undegraded":"2024-03-06T07:27:09.378612+0000","last_fullsized":"2024-03-06T07:27:09.378612+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:36.751221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368708+0000","last_change":"2024-03-06T07:27:09.368708+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368708+0000","last_undegraded":"2024-03-06T07:27:09.368708+0000","last_fullsized":"2024-03-06T07:27:09.368708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383926+0000","last_change":"2024-03-06T07:27:09.379796+0000","last_active":"2024-03-06T07:27:09.383926+0000","last_peered":"2024-03-06T07:27:09.383926+0000","last_clean":"2024-03-06T07:27:09.383926+0000","last_became_active":"2024-03-06T07:27:09.379609+0000","last_became_peered":"2024-03-06T07:27:09.379609+0000","last_unstale":"2024-03-06T07:27:09.383926+0000","last_undegraded":"2024-03-06T07:27:09.383926+0000","last_fullsized":"2024-03-06T07:27:09.383926+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:53.571465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367797+0000","last_change":"2024-03-06T07:27:09.367797+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367797+0000","last_undegraded":"2024-03-06T07:27:09.367797+0000","last_fullsized":"2024-03-06T07:27:09.367797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380992+0000","last_change":"2024-03-06T07:27:09.375952+0000","last_active":"2024-03-06T07:27:09.380992+0000","last_peered":"2024-03-06T07:27:09.380992+0000","last_clean":"2024-03-06T07:27:09.380992+0000","last_became_active":"2024-03-06T07:27:09.375584+0000","last_became_peered":"2024-03-06T07:27:09.375584+0000","last_unstale":"2024-03-06T07:27:09.380992+0000","last_undegraded":"2024-03-06T07:27:09.380992+0000","last_fullsized":"2024-03-06T07:27:09.380992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:03.362736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359735+0000","last_change":"2024-03-06T07:27:09.359735+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359735+0000","last_undegraded":"2024-03-06T07:27:09.359735+0000","last_fullsized":"2024-03-06T07:27:09.359735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.370532+0000","last_change":"2024-03-06T07:27:09.366681+0000","last_active":"2024-03-06T07:27:09.370532+0000","last_peered":"2024-03-06T07:27:09.370532+0000","last_clean":"2024-03-06T07:27:09.370532+0000","last_became_active":"2024-03-06T07:27:09.366329+0000","last_became_peered":"2024-03-06T07:27:09.366329+0000","last_unstale":"2024-03-06T07:27:09.370532+0000","last_undegraded":"2024-03-06T07:27:09.370532+0000","last_fullsized":"2024-03-06T07:27:09.370532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:07.928354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361649+0000","last_change":"2024-03-06T07:27:09.361649+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361649+0000","last_undegraded":"2024-03-06T07:27:09.361649+0000","last_fullsized":"2024-03-06T07:27:09.361649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380679+0000","last_change":"2024-03-06T07:27:09.378990+0000","last_active":"2024-03-06T07:27:09.380679+0000","last_peered":"2024-03-06T07:27:09.380679+0000","last_clean":"2024-03-06T07:27:09.380679+0000","last_became_active":"2024-03-06T07:27:09.378578+0000","last_became_peered":"2024-03-06T07:27:09.378578+0000","last_unstale":"2024-03-06T07:27:09.380679+0000","last_undegraded":"2024-03-06T07:27:09.380679+0000","last_fullsized":"2024-03-06T07:27:09.380679+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:41.025662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.362615+0000","last_change":"2024-03-06T07:27:09.362615+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.362615+0000","last_undegraded":"2024-03-06T07:27:09.362615+0000","last_fullsized":"2024-03-06T07:27:09.362615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.372532+0000","last_change":"2024-03-06T07:27:09.369411+0000","last_active":"2024-03-06T07:27:09.372532+0000","last_peered":"2024-03-06T07:27:09.372532+0000","last_clean":"2024-03-06T07:27:09.372532+0000","last_became_active":"2024-03-06T07:27:09.369100+0000","last_became_peered":"2024-03-06T07:27:09.369100+0000","last_unstale":"2024-03-06T07:27:09.372532+0000","last_undegraded":"2024-03-06T07:27:09.372532+0000","last_fullsized":"2024-03-06T07:27:09.372532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:07.197604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360078+0000","last_change":"2024-03-06T07:27:09.360078+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360078+0000","last_undegraded":"2024-03-06T07:27:09.360078+0000","last_fullsized":"2024-03-06T07:27:09.360078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.375998+0000","last_change":"2024-03-06T07:27:09.372377+0000","last_active":"2024-03-06T07:27:09.375998+0000","last_peered":"2024-03-06T07:27:09.375998+0000","last_clean":"2024-03-06T07:27:09.375998+0000","last_became_active":"2024-03-06T07:27:09.371991+0000","last_became_peered":"2024-03-06T07:27:09.371991+0000","last_unstale":"2024-03-06T07:27:09.375998+0000","last_undegraded":"2024-03-06T07:27:09.375998+0000","last_fullsized":"2024-03-06T07:27:09.375998+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:03.362736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367153+0000","last_change":"2024-03-06T07:27:09.367153+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367153+0000","last_undegraded":"2024-03-06T07:27:09.367153+0000","last_fullsized":"2024-03-06T07:27:09.367153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383917+0000","last_change":"2024-03-06T07:27:09.378597+0000","last_active":"2024-03-06T07:27:09.383917+0000","last_peered":"2024-03-06T07:27:09.383917+0000","last_clean":"2024-03-06T07:27:09.383917+0000","last_became_active":"2024-03-06T07:27:09.378334+0000","last_became_peered":"2024-03-06T07:27:09.378334+0000","last_unstale":"2024-03-06T07:27:09.383917+0000","last_undegraded":"2024-03-06T07:27:09.383917+0000","last_fullsized":"2024-03-06T07:27:09.383917+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:20.907611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366065+0000","last_change":"2024-03-06T07:27:09.366065+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366065+0000","last_undegraded":"2024-03-06T07:27:09.366065+0000","last_fullsized":"2024-03-06T07:27:09.366065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.368379+0000","last_change":"2024-03-06T07:27:09.365562+0000","last_active":"2024-03-06T07:27:09.368379+0000","last_peered":"2024-03-06T07:27:09.368379+0000","last_clean":"2024-03-06T07:27:09.368379+0000","last_became_active":"2024-03-06T07:27:09.365279+0000","last_became_peered":"2024-03-06T07:27:09.365279+0000","last_unstale":"2024-03-06T07:27:09.368379+0000","last_undegraded":"2024-03-06T07:27:09.368379+0000","last_fullsized":"2024-03-06T07:27:09.368379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:35.367786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366631+0000","last_change":"2024-03-06T07:27:09.366631+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366631+0000","last_undegraded":"2024-03-06T07:27:09.366631+0000","last_fullsized":"2024-03-06T07:27:09.366631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.379502+0000","last_change":"2024-03-06T07:27:09.378045+0000","last_active":"2024-03-06T07:27:09.379502+0000","last_peered":"2024-03-06T07:27:09.379502+0000","last_clean":"2024-03-06T07:27:09.379502+0000","last_became_active":"2024-03-06T07:27:09.377782+0000","last_became_peered":"2024-03-06T07:27:09.377782+0000","last_unstale":"2024-03-06T07:27:09.379502+0000","last_undegraded":"2024-03-06T07:27:09.379502+0000","last_fullsized":"2024-03-06T07:27:09.379502+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:47.722593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359370+0000","last_change":"2024-03-06T07:27:09.359370+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359370+0000","last_undegraded":"2024-03-06T07:27:09.359370+0000","last_fullsized":"2024-03-06T07:27:09.359370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380600+0000","last_change":"2024-03-06T07:27:09.378309+0000","last_active":"2024-03-06T07:27:09.380600+0000","last_peered":"2024-03-06T07:27:09.380600+0000","last_clean":"2024-03-06T07:27:09.380600+0000","last_became_active":"2024-03-06T07:27:09.378012+0000","last_became_peered":"2024-03-06T07:27:09.378012+0000","last_unstale":"2024-03-06T07:27:09.380600+0000","last_undegraded":"2024-03-06T07:27:09.380600+0000","last_fullsized":"2024-03-06T07:27:09.380600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:49.071313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365822+0000","last_change":"2024-03-06T07:27:09.365822+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365822+0000","last_undegraded":"2024-03-06T07:27:09.365822+0000","last_fullsized":"2024-03-06T07:27:09.365822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.372446+0000","last_change":"2024-03-06T07:27:09.370907+0000","last_active":"2024-03-06T07:27:09.372446+0000","last_peered":"2024-03-06T07:27:09.372446+0000","last_clean":"2024-03-06T07:27:09.372446+0000","last_became_active":"2024-03-06T07:27:09.370655+0000","last_became_peered":"2024-03-06T07:27:09.370655+0000","last_unstale":"2024-03-06T07:27:09.372446+0000","last_undegraded":"2024-03-06T07:27:09.372446+0000","last_fullsized":"2024-03-06T07:27:09.372446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:07.928354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368019+0000","last_change":"2024-03-06T07:27:09.368019+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368019+0000","last_undegraded":"2024-03-06T07:27:09.368019+0000","last_fullsized":"2024-03-06T07:27:09.368019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371431+0000","last_change":"2024-03-06T07:27:09.368550+0000","last_active":"2024-03-06T07:27:09.371431+0000","last_peered":"2024-03-06T07:27:09.371431+0000","last_clean":"2024-03-06T07:27:09.371431+0000","last_became_active":"2024-03-06T07:27:09.367237+0000","last_became_peered":"2024-03-06T07:27:09.367237+0000","last_unstale":"2024-03-06T07:27:09.371431+0000","last_undegraded":"2024-03-06T07:27:09.371431+0000","last_fullsized":"2024-03-06T07:27:09.371431+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:43.340074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359006+0000","last_change":"2024-03-06T07:27:09.359006+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359006+0000","last_undegraded":"2024-03-06T07:27:09.359006+0000","last_fullsized":"2024-03-06T07:27:09.359006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.369618+0000","last_change":"2024-03-06T07:27:09.367473+0000","last_active":"2024-03-06T07:27:09.369618+0000","last_peered":"2024-03-06T07:27:09.369618+0000","last_clean":"2024-03-06T07:27:09.369618+0000","last_became_active":"2024-03-06T07:27:09.367201+0000","last_became_peered":"2024-03-06T07:27:09.367201+0000","last_unstale":"2024-03-06T07:27:09.369618+0000","last_undegraded":"2024-03-06T07:27:09.369618+0000","last_fullsized":"2024-03-06T07:27:09.369618+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:26.071654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364913+0000","last_change":"2024-03-06T07:27:09.364913+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364913+0000","last_undegraded":"2024-03-06T07:27:09.364913+0000","last_fullsized":"2024-03-06T07:27:09.364913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374806+0000","last_change":"2024-03-06T07:27:09.373607+0000","last_active":"2024-03-06T07:27:09.374806+0000","last_peered":"2024-03-06T07:27:09.374806+0000","last_clean":"2024-03-06T07:27:09.374806+0000","last_became_active":"2024-03-06T07:27:09.373306+0000","last_became_peered":"2024-03-06T07:27:09.373306+0000","last_unstale":"2024-03-06T07:27:09.374806+0000","last_undegraded":"2024-03-06T07:27:09.374806+0000","last_fullsized":"2024-03-06T07:27:09.374806+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:32.055587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365624+0000","last_change":"2024-03-06T07:27:09.365624+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365624+0000","last_undegraded":"2024-03-06T07:27:09.365624+0000","last_fullsized":"2024-03-06T07:27:09.365624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371513+0000","last_change":"2024-03-06T07:27:09.367584+0000","last_active":"2024-03-06T07:27:09.371513+0000","last_peered":"2024-03-06T07:27:09.371513+0000","last_clean":"2024-03-06T07:27:09.371513+0000","last_became_active":"2024-03-06T07:27:09.366434+0000","last_became_peered":"2024-03-06T07:27:09.366434+0000","last_unstale":"2024-03-06T07:27:09.371513+0000","last_undegraded":"2024-03-06T07:27:09.371513+0000","last_fullsized":"2024-03-06T07:27:09.371513+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:44.994841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368264+0000","last_change":"2024-03-06T07:27:09.368264+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368264+0000","last_undegraded":"2024-03-06T07:27:09.368264+0000","last_fullsized":"2024-03-06T07:27:09.368264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374468+0000","last_change":"2024-03-06T07:27:09.373174+0000","last_active":"2024-03-06T07:27:09.374468+0000","last_peered":"2024-03-06T07:27:09.374468+0000","last_clean":"2024-03-06T07:27:09.374468+0000","last_became_active":"2024-03-06T07:27:09.372894+0000","last_became_peered":"2024-03-06T07:27:09.372894+0000","last_unstale":"2024-03-06T07:27:09.374468+0000","last_undegraded":"2024-03-06T07:27:09.374468+0000","last_fullsized":"2024-03-06T07:27:09.374468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:34.062787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359625+0000","last_change":"2024-03-06T07:27:09.359625+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359625+0000","last_undegraded":"2024-03-06T07:27:09.359625+0000","last_fullsized":"2024-03-06T07:27:09.359625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.377291+0000","last_change":"2024-03-06T07:27:09.374733+0000","last_active":"2024-03-06T07:27:09.377291+0000","last_peered":"2024-03-06T07:27:09.377291+0000","last_clean":"2024-03-06T07:27:09.377291+0000","last_became_active":"2024-03-06T07:27:09.374380+0000","last_became_peered":"2024-03-06T07:27:09.374380+0000","last_unstale":"2024-03-06T07:27:09.377291+0000","last_undegraded":"2024-03-06T07:27:09.377291+0000","last_fullsized":"2024-03-06T07:27:09.377291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:10.906485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364417+0000","last_change":"2024-03-06T07:27:09.364417+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364417+0000","last_undegraded":"2024-03-06T07:27:09.364417+0000","last_fullsized":"2024-03-06T07:27:09.364417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.377859+0000","last_change":"2024-03-06T07:27:09.375768+0000","last_active":"2024-03-06T07:27:09.377859+0000","last_peered":"2024-03-06T07:27:09.377859+0000","last_clean":"2024-03-06T07:27:09.377859+0000","last_became_active":"2024-03-06T07:27:09.375512+0000","last_became_peered":"2024-03-06T07:27:09.375512+0000","last_unstale":"2024-03-06T07:27:09.377859+0000","last_undegraded":"2024-03-06T07:27:09.377859+0000","last_fullsized":"2024-03-06T07:27:09.377859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:25.888556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360242+0000","last_change":"2024-03-06T07:27:09.360242+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360242+0000","last_undegraded":"2024-03-06T07:27:09.360242+0000","last_fullsized":"2024-03-06T07:27:09.360242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371369+0000","last_change":"2024-03-06T07:27:09.369058+0000","last_active":"2024-03-06T07:27:09.371369+0000","last_peered":"2024-03-06T07:27:09.371369+0000","last_clean":"2024-03-06T07:27:09.371369+0000","last_became_active":"2024-03-06T07:27:09.368762+0000","last_became_peered":"2024-03-06T07:27:09.368762+0000","last_unstale":"2024-03-06T07:27:09.371369+0000","last_undegraded":"2024-03-06T07:27:09.371369+0000","last_fullsized":"2024-03-06T07:27:09.371369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:20.708389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368719+0000","last_change":"2024-03-06T07:27:09.368719+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368719+0000","last_undegraded":"2024-03-06T07:27:09.368719+0000","last_fullsized":"2024-03-06T07:27:09.368719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.379184+0000","last_change":"2024-03-06T07:27:09.377310+0000","last_active":"2024-03-06T07:27:09.379184+0000","last_peered":"2024-03-06T07:27:09.379184+0000","last_clean":"2024-03-06T07:27:09.379184+0000","last_became_active":"2024-03-06T07:27:09.377048+0000","last_became_peered":"2024-03-06T07:27:09.377048+0000","last_unstale":"2024-03-06T07:27:09.379184+0000","last_undegraded":"2024-03-06T07:27:09.379184+0000","last_fullsized":"2024-03-06T07:27:09.379184+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:23.219552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.358988+0000","last_change":"2024-03-06T07:27:09.358988+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.358988+0000","last_undegraded":"2024-03-06T07:27:09.358988+0000","last_fullsized":"2024-03-06T07:27:09.358988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.381029+0000","last_change":"2024-03-06T07:27:09.380149+0000","last_active":"2024-03-06T07:27:09.381029+0000","last_peered":"2024-03-06T07:27:09.381029+0000","last_clean":"2024-03-06T07:27:09.381029+0000","last_became_active":"2024-03-06T07:27:09.379853+0000","last_became_peered":"2024-03-06T07:27:09.379853+0000","last_unstale":"2024-03-06T07:27:09.381029+0000","last_undegraded":"2024-03-06T07:27:09.381029+0000","last_fullsized":"2024-03-06T07:27:09.381029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:09.153513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364174+0000","last_change":"2024-03-06T07:27:09.364174+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364174+0000","last_undegraded":"2024-03-06T07:27:09.364174+0000","last_fullsized":"2024-03-06T07:27:09.364174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380994+0000","last_change":"2024-03-06T07:27:09.375900+0000","last_active":"2024-03-06T07:27:09.380994+0000","last_peered":"2024-03-06T07:27:09.380994+0000","last_clean":"2024-03-06T07:27:09.380994+0000","last_became_active":"2024-03-06T07:27:09.375583+0000","last_became_peered":"2024-03-06T07:27:09.375583+0000","last_unstale":"2024-03-06T07:27:09.380994+0000","last_undegraded":"2024-03-06T07:27:09.380994+0000","last_fullsized":"2024-03-06T07:27:09.380994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:53.571465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365127+0000","last_change":"2024-03-06T07:27:09.365127+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365127+0000","last_undegraded":"2024-03-06T07:27:09.365127+0000","last_fullsized":"2024-03-06T07:27:09.365127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.369385+0000","last_change":"2024-03-06T07:27:09.367702+0000","last_active":"2024-03-06T07:27:09.369385+0000","last_peered":"2024-03-06T07:27:09.369385+0000","last_clean":"2024-03-06T07:27:09.369385+0000","last_became_active":"2024-03-06T07:27:09.367456+0000","last_became_peered":"2024-03-06T07:27:09.367456+0000","last_unstale":"2024-03-06T07:27:09.369385+0000","last_undegraded":"2024-03-06T07:27:09.369385+0000","last_fullsized":"2024-03-06T07:27:09.369385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:17.715304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360340+0000","last_change":"2024-03-06T07:27:09.360340+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360340+0000","last_undegraded":"2024-03-06T07:27:09.360340+0000","last_fullsized":"2024-03-06T07:27:09.360340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371484+0000","last_change":"2024-03-06T07:27:09.367998+0000","last_active":"2024-03-06T07:27:09.371484+0000","last_peered":"2024-03-06T07:27:09.371484+0000","last_clean":"2024-03-06T07:27:09.371484+0000","last_became_active":"2024-03-06T07:27:09.367731+0000","last_became_peered":"2024-03-06T07:27:09.367731+0000","last_unstale":"2024-03-06T07:27:09.371484+0000","last_undegraded":"2024-03-06T07:27:09.371484+0000","last_fullsized":"2024-03-06T07:27:09.371484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:14.629790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365984+0000","last_change":"2024-03-06T07:27:09.365984+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365984+0000","last_undegraded":"2024-03-06T07:27:09.365984+0000","last_fullsized":"2024-03-06T07:27:09.365984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374853+0000","last_change":"2024-03-06T07:27:09.374468+0000","last_active":"2024-03-06T07:27:09.374853+0000","last_peered":"2024-03-06T07:27:09.374853+0000","last_clean":"2024-03-06T07:27:09.374853+0000","last_became_active":"2024-03-06T07:27:09.374183+0000","last_became_peered":"2024-03-06T07:27:09.374183+0000","last_unstale":"2024-03-06T07:27:09.374853+0000","last_undegraded":"2024-03-06T07:27:09.374853+0000","last_fullsized":"2024-03-06T07:27:09.374853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:49.597272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359152+0000","last_change":"2024-03-06T07:27:09.359152+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359152+0000","last_undegraded":"2024-03-06T07:27:09.359152+0000","last_fullsized":"2024-03-06T07:27:09.359152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.370888+0000","last_change":"2024-03-06T07:27:09.368796+0000","last_active":"2024-03-06T07:27:09.370888+0000","last_peered":"2024-03-06T07:27:09.370888+0000","last_clean":"2024-03-06T07:27:09.370888+0000","last_became_active":"2024-03-06T07:27:09.368017+0000","last_became_peered":"2024-03-06T07:27:09.368017+0000","last_unstale":"2024-03-06T07:27:09.370888+0000","last_undegraded":"2024-03-06T07:27:09.370888+0000","last_fullsized":"2024-03-06T07:27:09.370888+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:49.597272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.363467+0000","last_change":"2024-03-06T07:27:09.363467+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.363467+0000","last_undegraded":"2024-03-06T07:27:09.363467+0000","last_fullsized":"2024-03-06T07:27:09.363467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.373600+0000","last_change":"2024-03-06T07:27:09.373156+0000","last_active":"2024-03-06T07:27:09.373600+0000","last_peered":"2024-03-06T07:27:09.373600+0000","last_clean":"2024-03-06T07:27:09.373600+0000","last_became_active":"2024-03-06T07:27:09.372848+0000","last_became_peered":"2024-03-06T07:27:09.372848+0000","last_unstale":"2024-03-06T07:27:09.373600+0000","last_undegraded":"2024-03-06T07:27:09.373600+0000","last_fullsized":"2024-03-06T07:27:09.373600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:17.715304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.374217+0000","last_change":"2024-03-06T07:27:09.374217+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.374217+0000","last_undegraded":"2024-03-06T07:27:09.374217+0000","last_fullsized":"2024-03-06T07:27:09.374217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.375143+0000","last_change":"2024-03-06T07:27:09.371787+0000","last_active":"2024-03-06T07:27:09.375143+0000","last_peered":"2024-03-06T07:27:09.375143+0000","last_clean":"2024-03-06T07:27:09.375143+0000","last_became_active":"2024-03-06T07:27:09.371100+0000","last_became_peered":"2024-03-06T07:27:09.371100+0000","last_unstale":"2024-03-06T07:27:09.375143+0000","last_undegraded":"2024-03-06T07:27:09.375143+0000","last_fullsized":"2024-03-06T07:27:09.375143+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:03.753803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":288,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.361425+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:27:09.361425+0000","last_peered":"2024-03-06T07:27:09.361425+0000","last_clean":"2024-03-06T07:27:09.361425+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:27:09.361425+0000","last_undegraded":"2024-03-06T07:27:09.361425+0000","last_fullsized":"2024-03-06T07:27:09.361425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1014782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":418139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1014782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:14.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.638+0000 7f40ebaed640 1 Processor -- start 2024-03-06T07:27:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.638+0000 7f40ebaed640 1 -- start start 2024-03-06T07:27:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40ebaed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e415df00 0x7f40e40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40ebaed640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f40e40b95a0 0x7f40e40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40ebaed640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e40cea30 con 0x7f40e40b9a30 2024-03-06T07:27:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40ebaed640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e40d2af0 con 0x7f40e40b95a0 2024-03-06T07:27:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40ebaed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40cf3c0 con 0x7f40e415df00 2024-03-06T07:27:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40ea08a640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f40e40b9a30 0x7f40e40d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43310/0 (socket says 172.21.15.27:43310) 2024-03-06T07:27:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40ea08a640 1 -- 172.21.15.27:0/4249681427 learned_addr learned my addr 172.21.15.27:0/4249681427 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40e9088640 1 --2- 172.21.15.27:0/4249681427 >> v2:172.21.15.69:3300/0 conn(0x7f40e40b95a0 0x7f40e40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40e9889640 1 --2- 172.21.15.27:0/4249681427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e415df00 0x7f40e40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.639+0000 7f40e8887640 1 -- 172.21.15.27:0/4249681427 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986421515 0 0) 0x7f40e40cea30 con 0x7f40e40b9a30 2024-03-06T07:27:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.640+0000 7f40e8887640 1 -- 172.21.15.27:0/4249681427 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d0003520 con 0x7f40e40b9a30 2024-03-06T07:27:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.640+0000 7f40e9088640 1 -- 172.21.15.27:0/4249681427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e415df00 msgr2=0x7f40e40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.640+0000 7f40e9088640 1 --2- 172.21.15.27:0/4249681427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e415df00 0x7f40e40d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.640+0000 7f40e9088640 1 -- 172.21.15.27:0/4249681427 >> v1:172.21.15.27:6789/0 conn(0x7f40e40b9a30 legacy=0x7f40e40d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.640+0000 7f40e9088640 1 -- 172.21.15.27:0/4249681427 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40cf0e0 con 0x7f40e40b95a0 2024-03-06T07:27:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.640+0000 7f40e9889640 1 --2- 172.21.15.27:0/4249681427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e415df00 0x7f40e40d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.640+0000 7f40e9088640 1 --2- 172.21.15.27:0/4249681427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e40b95a0 0x7f40e40b5700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f40e00097a0 tx=0x7f40e002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18afc79115f2936 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.641+0000 7f40e8887640 1 -- 172.21.15.27:0/4249681427 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0035020 con 0x7f40e40b95a0 2024-03-06T07:27:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.641+0000 7f40e8887640 1 -- 172.21.15.27:0/4249681427 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e0031810 con 0x7f40e40b95a0 2024-03-06T07:27:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.641+0000 7f40e8887640 1 -- 172.21.15.27:0/4249681427 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0031bc0 con 0x7f40e40b95a0 2024-03-06T07:27:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.641+0000 7f40ebaed640 1 -- 172.21.15.27:0/4249681427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e40b95a0 msgr2=0x7f40e40b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.641+0000 7f40ebaed640 1 --2- 172.21.15.27:0/4249681427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e40b95a0 0x7f40e40b5700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f40e00097a0 tx=0x7f40e002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.641+0000 7f40ebaed640 1 -- 172.21.15.27:0/4249681427 shutdown_connections 2024-03-06T07:27:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.641+0000 7f40ebaed640 1 --2- 172.21.15.27:0/4249681427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e40b95a0 0x7f40e40b5700 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.641+0000 7f40ebaed640 1 --2- 172.21.15.27:0/4249681427 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e415df00 0x7f40e40d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.641+0000 7f40ebaed640 1 -- 172.21.15.27:0/4249681427 >> 172.21.15.27:0/4249681427 conn(0x7f40e4071960 msgr2=0x7f40e4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.642+0000 7f40ebaed640 1 -- 172.21.15.27:0/4249681427 shutdown_connections 2024-03-06T07:27:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.642+0000 7f40ebaed640 1 -- 172.21.15.27:0/4249681427 wait complete. 2024-03-06T07:27:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.644+0000 7f40ebaed640 1 Processor -- start 2024-03-06T07:27:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.644+0000 7f40ebaed640 1 -- start start 2024-03-06T07:27:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ebaed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e40b95a0 0x7f40e40b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ebaed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e415df00 0x7f40e41f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ebaed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40e41f32a0 0x7f40e41f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ebaed640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40cec70 con 0x7f40e41f32a0 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ebaed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40ce750 con 0x7f40e415df00 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ebaed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40cedf0 con 0x7f40e40b95a0 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ea08a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40e41f32a0 0x7f40e41f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ea08a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40e41f32a0 0x7f40e41f6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34290/0 (socket says 172.21.15.27:34290) 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ea08a640 1 -- 172.21.15.27:0/3404505316 learned_addr learned my addr 172.21.15.27:0/3404505316 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40e9088640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e415df00 0x7f40e41f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40e9889640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e40b95a0 0x7f40e40b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ea08a640 1 -- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e40b95a0 msgr2=0x7f40e40b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ea08a640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e40b95a0 0x7f40e40b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ea08a640 1 -- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e415df00 msgr2=0x7f40e41f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ea08a640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e415df00 0x7f40e41f2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.645+0000 7f40ea08a640 1 -- 172.21.15.27:0/3404505316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e4150e30 con 0x7f40e41f32a0 2024-03-06T07:27:14.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.646+0000 7f40ea08a640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40e41f32a0 0x7f40e41f6c20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00cbb0 tx=0x7f40dc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.646+0000 7f40da7fc640 1 -- 172.21.15.27:0/3404505316 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc017070 con 0x7f40e41f32a0 2024-03-06T07:27:14.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.646+0000 7f40da7fc640 1 -- 172.21.15.27:0/3404505316 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40dc00ad40 con 0x7f40e41f32a0 2024-03-06T07:27:14.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.646+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e40c89a0 con 0x7f40e41f32a0 2024-03-06T07:27:14.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.646+0000 7f40da7fc640 1 -- 172.21.15.27:0/3404505316 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc012600 con 0x7f40e41f32a0 2024-03-06T07:27:14.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.646+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e40c9240 con 0x7f40e41f32a0 2024-03-06T07:27:14.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.647+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e40c8b70 con 0x7f40e41f32a0 2024-03-06T07:27:14.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.647+0000 7f40da7fc640 1 -- 172.21.15.27:0/3404505316 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40dc01a070 con 0x7f40e41f32a0 2024-03-06T07:27:14.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.648+0000 7f40da7fc640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f40c806b4d0 0x7f40c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.648+0000 7f40da7fc640 1 -- 172.21.15.27:0/3404505316 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40dc089ec0 con 0x7f40e41f32a0 2024-03-06T07:27:14.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.648+0000 7f40e9889640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f40c806b4d0 0x7f40c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.648+0000 7f40e9889640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f40c806b4d0 0x7f40c806d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009560 tx=0x7f40d4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.652+0000 7f40da7fc640 1 -- 172.21.15.27:0/3404505316 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40dc058700 con 0x7f40e41f32a0 2024-03-06T07:27:14.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.829+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40e4067ff0 con 0x7f40c806b4d0 2024-03-06T07:27:14.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.836+0000 7f40da7fc640 1 -- 172.21.15.27:0/3404505316 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246760 (secure 0 0 0) 0x7f40e4067ff0 con 0x7f40c806b4d0 2024-03-06T07:27:14.839 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:14.841 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:27:14.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f40c806b4d0 msgr2=0x7f40c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f40c806b4d0 0x7f40c806d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009560 tx=0x7f40d4009190 comp rx=0 tx=0).stop 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40e41f32a0 msgr2=0x7f40e41f6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40e41f32a0 0x7f40e41f6c20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00cbb0 tx=0x7f40dc00a430 comp rx=0 tx=0).stop 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 shutdown_connections 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f40c806b4d0 0x7f40c806d990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40e41f32a0 0x7f40e41f6c20 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40e415df00 0x7f40e41f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 --2- 172.21.15.27:0/3404505316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40e40b95a0 0x7f40e40b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 >> 172.21.15.27:0/3404505316 conn(0x7f40e4071960 msgr2=0x7f40e40cb760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 shutdown_connections 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.839+0000 7f40ebaed640 1 -- 172.21.15.27:0/3404505316 wait complete. 2024-03-06T07:27:14.851 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-03-06T07:27:14.050694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258488,"kb_used_data":9020,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716232,"statfs":{"total":773094113280,"available":772829421568,"internally_reserved":0,"allocated":9236480,"data_stored":4569773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.694133"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366331+0000","last_change":"2024-03-06T07:27:09.366331+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366331+0000","last_undegraded":"2024-03-06T07:27:09.366331+0000","last_fullsized":"2024-03-06T07:27:09.366331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365065+0000","last_change":"2024-03-06T07:27:09.365065+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365065+0000","last_undegraded":"2024-03-06T07:27:09.365065+0000","last_fullsized":"2024-03-06T07:27:09.365065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361750+0000","last_change":"2024-03-06T07:27:09.361750+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361750+0000","last_undegraded":"2024-03-06T07:27:09.361750+0000","last_fullsized":"2024-03-06T07:27:09.361750+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368354+0000","last_change":"2024-03-06T07:27:09.368354+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368354+0000","last_undegraded":"2024-03-06T07:27:09.368354+0000","last_fullsized":"2024-03-06T07:27:09.368354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368139+0000","last_change":"2024-03-06T07:27:09.368139+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368139+0000","last_undegraded":"2024-03-06T07:27:09.368139+0000","last_fullsized":"2024-03-06T07:27:09.368139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360754+0000","last_change":"2024-03-06T07:27:09.360754+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360754+0000","last_undegraded":"2024-03-06T07:27:09.360754+0000","last_fullsized":"2024-03-06T07:27:09.360754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366900+0000","last_change":"2024-03-06T07:27:09.366900+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366900+0000","last_undegraded":"2024-03-06T07:27:09.366900+0000","last_fullsized":"2024-03-06T07:27:09.366900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360184+0000","last_change":"2024-03-06T07:27:09.360184+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360184+0000","last_undegraded":"2024-03-06T07:27:09.360184+0000","last_fullsized":"2024-03-06T07:27:09.360184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360603+0000","last_change":"2024-03-06T07:27:09.360603+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360603+0000","last_undegraded":"2024-03-06T07:27:09.360603+0000","last_fullsized":"2024-03-06T07:27:09.360603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364335+0000","last_change":"2024-03-06T07:27:09.364335+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364335+0000","last_undegraded":"2024-03-06T07:27:09.364335+0000","last_fullsized":"2024-03-06T07:27:09.364335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364708+0000","last_change":"2024-03-06T07:27:09.364708+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364708+0000","last_undegraded":"2024-03-06T07:27:09.364708+0000","last_fullsized":"2024-03-06T07:27:09.364708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366392+0000","last_change":"2024-03-06T07:27:09.366392+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366392+0000","last_undegraded":"2024-03-06T07:27:09.366392+0000","last_fullsized":"2024-03-06T07:27:09.366392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361471+0000","last_change":"2024-03-06T07:27:09.361471+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361471+0000","last_undegraded":"2024-03-06T07:27:09.361471+0000","last_fullsized":"2024-03-06T07:27:09.361471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368716+0000","last_change":"2024-03-06T07:27:09.368716+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368716+0000","last_undegraded":"2024-03-06T07:27:09.368716+0000","last_fullsized":"2024-03-06T07:27:09.368716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.362610+0000","last_change":"2024-03-06T07:27:09.362610+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.362610+0000","last_undegraded":"2024-03-06T07:27:09.362610+0000","last_fullsized":"2024-03-06T07:27:09.362610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365314+0000","last_change":"2024-03-06T07:27:09.365314+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365314+0000","last_undegraded":"2024-03-06T07:27:09.365314+0000","last_fullsized":"2024-03-06T07:27:09.365314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364605+0000","last_change":"2024-03-06T07:27:09.364605+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364605+0000","last_undegraded":"2024-03-06T07:27:09.364605+0000","last_fullsized":"2024-03-06T07:27:09.364605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366597+0000","last_change":"2024-03-06T07:27:09.366597+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366597+0000","last_undegraded":"2024-03-06T07:27:09.366597+0000","last_fullsized":"2024-03-06T07:27:09.366597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367523+0000","last_change":"2024-03-06T07:27:09.367523+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367523+0000","last_undegraded":"2024-03-06T07:27:09.367523+0000","last_fullsized":"2024-03-06T07:27:09.367523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367857+0000","last_change":"2024-03-06T07:27:09.367857+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367857+0000","last_undegraded":"2024-03-06T07:27:09.367857+0000","last_fullsized":"2024-03-06T07:27:09.367857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366174+0000","last_change":"2024-03-06T07:27:09.366174+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366174+0000","last_undegraded":"2024-03-06T07:27:09.366174+0000","last_fullsized":"2024-03-06T07:27:09.366174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369885+0000","last_change":"2024-03-06T07:27:09.369885+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369885+0000","last_undegraded":"2024-03-06T07:27:09.369885+0000","last_fullsized":"2024-03-06T07:27:09.369885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367067+0000","last_change":"2024-03-06T07:27:09.367067+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367067+0000","last_undegraded":"2024-03-06T07:27:09.367067+0000","last_fullsized":"2024-03-06T07:27:09.367067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364968+0000","last_change":"2024-03-06T07:27:09.364968+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364968+0000","last_undegraded":"2024-03-06T07:27:09.364968+0000","last_fullsized":"2024-03-06T07:27:09.364968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367295+0000","last_change":"2024-03-06T07:27:09.367295+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367295+0000","last_undegraded":"2024-03-06T07:27:09.367295+0000","last_fullsized":"2024-03-06T07:27:09.367295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360671+0000","last_change":"2024-03-06T07:27:09.360671+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360671+0000","last_undegraded":"2024-03-06T07:27:09.360671+0000","last_fullsized":"2024-03-06T07:27:09.360671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366535+0000","last_change":"2024-03-06T07:27:09.366535+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366535+0000","last_undegraded":"2024-03-06T07:27:09.366535+0000","last_fullsized":"2024-03-06T07:27:09.366535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365953+0000","last_change":"2024-03-06T07:27:09.365953+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365953+0000","last_undegraded":"2024-03-06T07:27:09.365953+0000","last_fullsized":"2024-03-06T07:27:09.365953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369681+0000","last_change":"2024-03-06T07:27:09.369681+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369681+0000","last_undegraded":"2024-03-06T07:27:09.369681+0000","last_fullsized":"2024-03-06T07:27:09.369681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369176+0000","last_change":"2024-03-06T07:27:09.369176+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369176+0000","last_undegraded":"2024-03-06T07:27:09.369176+0000","last_fullsized":"2024-03-06T07:27:09.369176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367187+0000","last_change":"2024-03-06T07:27:09.367187+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367187+0000","last_undegraded":"2024-03-06T07:27:09.367187+0000","last_fullsized":"2024-03-06T07:27:09.367187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.363528+0000","last_change":"2024-03-06T07:27:09.363528+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.363528+0000","last_undegraded":"2024-03-06T07:27:09.363528+0000","last_fullsized":"2024-03-06T07:27:09.363528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369136+0000","last_change":"2024-03-06T07:27:09.369136+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369136+0000","last_undegraded":"2024-03-06T07:27:09.369136+0000","last_fullsized":"2024-03-06T07:27:09.369136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383880+0000","last_change":"2024-03-06T07:27:09.378545+0000","last_active":"2024-03-06T07:27:09.383880+0000","last_peered":"2024-03-06T07:27:09.383880+0000","last_clean":"2024-03-06T07:27:09.383880+0000","last_became_active":"2024-03-06T07:27:09.378276+0000","last_became_peered":"2024-03-06T07:27:09.378276+0000","last_unstale":"2024-03-06T07:27:09.383880+0000","last_undegraded":"2024-03-06T07:27:09.383880+0000","last_fullsized":"2024-03-06T07:27:09.383880+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:09.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365340+0000","last_change":"2024-03-06T07:27:09.365340+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365340+0000","last_undegraded":"2024-03-06T07:27:09.365340+0000","last_fullsized":"2024-03-06T07:27:09.365340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.381029+0000","last_change":"2024-03-06T07:27:09.375904+0000","last_active":"2024-03-06T07:27:09.381029+0000","last_peered":"2024-03-06T07:27:09.381029+0000","last_clean":"2024-03-06T07:27:09.381029+0000","last_became_active":"2024-03-06T07:27:09.375566+0000","last_became_peered":"2024-03-06T07:27:09.375566+0000","last_unstale":"2024-03-06T07:27:09.381029+0000","last_undegraded":"2024-03-06T07:27:09.381029+0000","last_fullsized":"2024-03-06T07:27:09.381029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:07.197604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365542+0000","last_change":"2024-03-06T07:27:09.365542+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365542+0000","last_undegraded":"2024-03-06T07:27:09.365542+0000","last_fullsized":"2024-03-06T07:27:09.365542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374222+0000","last_change":"2024-03-06T07:27:09.371083+0000","last_active":"2024-03-06T07:27:09.374222+0000","last_peered":"2024-03-06T07:27:09.374222+0000","last_clean":"2024-03-06T07:27:09.374222+0000","last_became_active":"2024-03-06T07:27:09.370820+0000","last_became_peered":"2024-03-06T07:27:09.370820+0000","last_unstale":"2024-03-06T07:27:09.374222+0000","last_undegraded":"2024-03-06T07:27:09.374222+0000","last_fullsized":"2024-03-06T07:27:09.374222+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:29.304676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359572+0000","last_change":"2024-03-06T07:27:09.359572+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359572+0000","last_undegraded":"2024-03-06T07:27:09.359572+0000","last_fullsized":"2024-03-06T07:27:09.359572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.378612+0000","last_change":"2024-03-06T07:27:09.374749+0000","last_active":"2024-03-06T07:27:09.378612+0000","last_peered":"2024-03-06T07:27:09.378612+0000","last_clean":"2024-03-06T07:27:09.378612+0000","last_became_active":"2024-03-06T07:27:09.374421+0000","last_became_peered":"2024-03-06T07:27:09.374421+0000","last_unstale":"2024-03-06T07:27:09.378612+0000","last_undegraded":"2024-03-06T07:27:09.378612+0000","last_fullsized":"2024-03-06T07:27:09.378612+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:36.751221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368708+0000","last_change":"2024-03-06T07:27:09.368708+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368708+0000","last_undegraded":"2024-03-06T07:27:09.368708+0000","last_fullsized":"2024-03-06T07:27:09.368708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383926+0000","last_change":"2024-03-06T07:27:09.379796+0000","last_active":"2024-03-06T07:27:09.383926+0000","last_peered":"2024-03-06T07:27:09.383926+0000","last_clean":"2024-03-06T07:27:09.383926+0000","last_became_active":"2024-03-06T07:27:09.379609+0000","last_became_peered":"2024-03-06T07:27:09.379609+0000","last_unstale":"2024-03-06T07:27:09.383926+0000","last_undegraded":"2024-03-06T07:27:09.383926+0000","last_fullsized":"2024-03-06T07:27:09.383926+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:53.571465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367797+0000","last_change":"2024-03-06T07:27:09.367797+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367797+0000","last_undegraded":"2024-03-06T07:27:09.367797+0000","last_fullsized":"2024-03-06T07:27:09.367797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380992+0000","last_change":"2024-03-06T07:27:09.375952+0000","last_active":"2024-03-06T07:27:09.380992+0000","last_peered":"2024-03-06T07:27:09.380992+0000","last_clean":"2024-03-06T07:27:09.380992+0000","last_became_active":"2024-03-06T07:27:09.375584+0000","last_became_peered":"2024-03-06T07:27:09.375584+0000","last_unstale":"2024-03-06T07:27:09.380992+0000","last_undegraded":"2024-03-06T07:27:09.380992+0000","last_fullsized":"2024-03-06T07:27:09.380992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:03.362736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359735+0000","last_change":"2024-03-06T07:27:09.359735+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359735+0000","last_undegraded":"2024-03-06T07:27:09.359735+0000","last_fullsized":"2024-03-06T07:27:09.359735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.370532+0000","last_change":"2024-03-06T07:27:09.366681+0000","last_active":"2024-03-06T07:27:09.370532+0000","last_peered":"2024-03-06T07:27:09.370532+0000","last_clean":"2024-03-06T07:27:09.370532+0000","last_became_active":"2024-03-06T07:27:09.366329+0000","last_became_peered":"2024-03-06T07:27:09.366329+0000","last_unstale":"2024-03-06T07:27:09.370532+0000","last_undegraded":"2024-03-06T07:27:09.370532+0000","last_fullsized":"2024-03-06T07:27:09.370532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:07.928354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361649+0000","last_change":"2024-03-06T07:27:09.361649+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361649+0000","last_undegraded":"2024-03-06T07:27:09.361649+0000","last_fullsized":"2024-03-06T07:27:09.361649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380679+0000","last_change":"2024-03-06T07:27:09.378990+0000","last_active":"2024-03-06T07:27:09.380679+0000","last_peered":"2024-03-06T07:27:09.380679+0000","last_clean":"2024-03-06T07:27:09.380679+0000","last_became_active":"2024-03-06T07:27:09.378578+0000","last_became_peered":"2024-03-06T07:27:09.378578+0000","last_unstale":"2024-03-06T07:27:09.380679+0000","last_undegraded":"2024-03-06T07:27:09.380679+0000","last_fullsized":"2024-03-06T07:27:09.380679+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:41.025662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.362615+0000","last_change":"2024-03-06T07:27:09.362615+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.362615+0000","last_undegraded":"2024-03-06T07:27:09.362615+0000","last_fullsized":"2024-03-06T07:27:09.362615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.372532+0000","last_change":"2024-03-06T07:27:09.369411+0000","last_active":"2024-03-06T07:27:09.372532+0000","last_peered":"2024-03-06T07:27:09.372532+0000","last_clean":"2024-03-06T07:27:09.372532+0000","last_became_active":"2024-03-06T07:27:09.369100+0000","last_became_peered":"2024-03-06T07:27:09.369100+0000","last_unstale":"2024-03-06T07:27:09.372532+0000","last_undegraded":"2024-03-06T07:27:09.372532+0000","last_fullsized":"2024-03-06T07:27:09.372532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:07.197604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360078+0000","last_change":"2024-03-06T07:27:09.360078+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360078+0000","last_undegraded":"2024-03-06T07:27:09.360078+0000","last_fullsized":"2024-03-06T07:27:09.360078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.375998+0000","last_change":"2024-03-06T07:27:09.372377+0000","last_active":"2024-03-06T07:27:09.375998+0000","last_peered":"2024-03-06T07:27:09.375998+0000","last_clean":"2024-03-06T07:27:09.375998+0000","last_became_active":"2024-03-06T07:27:09.371991+0000","last_became_peered":"2024-03-06T07:27:09.371991+0000","last_unstale":"2024-03-06T07:27:09.375998+0000","last_undegraded":"2024-03-06T07:27:09.375998+0000","last_fullsized":"2024-03-06T07:27:09.375998+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:03.362736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367153+0000","last_change":"2024-03-06T07:27:09.367153+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367153+0000","last_undegraded":"2024-03-06T07:27:09.367153+0000","last_fullsized":"2024-03-06T07:27:09.367153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383917+0000","last_change":"2024-03-06T07:27:09.378597+0000","last_active":"2024-03-06T07:27:09.383917+0000","last_peered":"2024-03-06T07:27:09.383917+0000","last_clean":"2024-03-06T07:27:09.383917+0000","last_became_active":"2024-03-06T07:27:09.378334+0000","last_became_peered":"2024-03-06T07:27:09.378334+0000","last_unstale":"2024-03-06T07:27:09.383917+0000","last_undegraded":"2024-03-06T07:27:09.383917+0000","last_fullsized":"2024-03-06T07:27:09.383917+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:20.907611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366065+0000","last_change":"2024-03-06T07:27:09.366065+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366065+0000","last_undegraded":"2024-03-06T07:27:09.366065+0000","last_fullsized":"2024-03-06T07:27:09.366065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.368379+0000","last_change":"2024-03-06T07:27:09.365562+0000","last_active":"2024-03-06T07:27:09.368379+0000","last_peered":"2024-03-06T07:27:09.368379+0000","last_clean":"2024-03-06T07:27:09.368379+0000","last_became_active":"2024-03-06T07:27:09.365279+0000","last_became_peered":"2024-03-06T07:27:09.365279+0000","last_unstale":"2024-03-06T07:27:09.368379+0000","last_undegraded":"2024-03-06T07:27:09.368379+0000","last_fullsized":"2024-03-06T07:27:09.368379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:35.367786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366631+0000","last_change":"2024-03-06T07:27:09.366631+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366631+0000","last_undegraded":"2024-03-06T07:27:09.366631+0000","last_fullsized":"2024-03-06T07:27:09.366631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.379502+0000","last_change":"2024-03-06T07:27:09.378045+0000","last_active":"2024-03-06T07:27:09.379502+0000","last_peered":"2024-03-06T07:27:09.379502+0000","last_clean":"2024-03-06T07:27:09.379502+0000","last_became_active":"2024-03-06T07:27:09.377782+0000","last_became_peered":"2024-03-06T07:27:09.377782+0000","last_unstale":"2024-03-06T07:27:09.379502+0000","last_undegraded":"2024-03-06T07:27:09.379502+0000","last_fullsized":"2024-03-06T07:27:09.379502+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:47.722593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359370+0000","last_change":"2024-03-06T07:27:09.359370+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359370+0000","last_undegraded":"2024-03-06T07:27:09.359370+0000","last_fullsized":"2024-03-06T07:27:09.359370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380600+0000","last_change":"2024-03-06T07:27:09.378309+0000","last_active":"2024-03-06T07:27:09.380600+0000","last_peered":"2024-03-06T07:27:09.380600+0000","last_clean":"2024-03-06T07:27:09.380600+0000","last_became_active":"2024-03-06T07:27:09.378012+0000","last_became_peered":"2024-03-06T07:27:09.378012+0000","last_unstale":"2024-03-06T07:27:09.380600+0000","last_undegraded":"2024-03-06T07:27:09.380600+0000","last_fullsized":"2024-03-06T07:27:09.380600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:49.071313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365822+0000","last_change":"2024-03-06T07:27:09.365822+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365822+0000","last_undegraded":"2024-03-06T07:27:09.365822+0000","last_fullsized":"2024-03-06T07:27:09.365822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.372446+0000","last_change":"2024-03-06T07:27:09.370907+0000","last_active":"2024-03-06T07:27:09.372446+0000","last_peered":"2024-03-06T07:27:09.372446+0000","last_clean":"2024-03-06T07:27:09.372446+0000","last_became_active":"2024-03-06T07:27:09.370655+0000","last_became_peered":"2024-03-06T07:27:09.370655+0000","last_unstale":"2024-03-06T07:27:09.372446+0000","last_undegraded":"2024-03-06T07:27:09.372446+0000","last_fullsized":"2024-03-06T07:27:09.372446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:07.928354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368019+0000","last_change":"2024-03-06T07:27:09.368019+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368019+0000","last_undegraded":"2024-03-06T07:27:09.368019+0000","last_fullsized":"2024-03-06T07:27:09.368019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371431+0000","last_change":"2024-03-06T07:27:09.368550+0000","last_active":"2024-03-06T07:27:09.371431+0000","last_peered":"2024-03-06T07:27:09.371431+0000","last_clean":"2024-03-06T07:27:09.371431+0000","last_became_active":"2024-03-06T07:27:09.367237+0000","last_became_peered":"2024-03-06T07:27:09.367237+0000","last_unstale":"2024-03-06T07:27:09.371431+0000","last_undegraded":"2024-03-06T07:27:09.371431+0000","last_fullsized":"2024-03-06T07:27:09.371431+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:43.340074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359006+0000","last_change":"2024-03-06T07:27:09.359006+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359006+0000","last_undegraded":"2024-03-06T07:27:09.359006+0000","last_fullsized":"2024-03-06T07:27:09.359006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.369618+0000","last_change":"2024-03-06T07:27:09.367473+0000","last_active":"2024-03-06T07:27:09.369618+0000","last_peered":"2024-03-06T07:27:09.369618+0000","last_clean":"2024-03-06T07:27:09.369618+0000","last_became_active":"2024-03-06T07:27:09.367201+0000","last_became_peered":"2024-03-06T07:27:09.367201+0000","last_unstale":"2024-03-06T07:27:09.369618+0000","last_undegraded":"2024-03-06T07:27:09.369618+0000","last_fullsized":"2024-03-06T07:27:09.369618+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:26.071654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364913+0000","last_change":"2024-03-06T07:27:09.364913+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364913+0000","last_undegraded":"2024-03-06T07:27:09.364913+0000","last_fullsized":"2024-03-06T07:27:09.364913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374806+0000","last_change":"2024-03-06T07:27:09.373607+0000","last_active":"2024-03-06T07:27:09.374806+0000","last_peered":"2024-03-06T07:27:09.374806+0000","last_clean":"2024-03-06T07:27:09.374806+0000","last_became_active":"2024-03-06T07:27:09.373306+0000","last_became_peered":"2024-03-06T07:27:09.373306+0000","last_unstale":"2024-03-06T07:27:09.374806+0000","last_undegraded":"2024-03-06T07:27:09.374806+0000","last_fullsized":"2024-03-06T07:27:09.374806+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:32.055587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365624+0000","last_change":"2024-03-06T07:27:09.365624+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365624+0000","last_undegraded":"2024-03-06T07:27:09.365624+0000","last_fullsized":"2024-03-06T07:27:09.365624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371513+0000","last_change":"2024-03-06T07:27:09.367584+0000","last_active":"2024-03-06T07:27:09.371513+0000","last_peered":"2024-03-06T07:27:09.371513+0000","last_clean":"2024-03-06T07:27:09.371513+0000","last_became_active":"2024-03-06T07:27:09.366434+0000","last_became_peered":"2024-03-06T07:27:09.366434+0000","last_unstale":"2024-03-06T07:27:09.371513+0000","last_undegraded":"2024-03-06T07:27:09.371513+0000","last_fullsized":"2024-03-06T07:27:09.371513+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:44.994841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368264+0000","last_change":"2024-03-06T07:27:09.368264+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368264+0000","last_undegraded":"2024-03-06T07:27:09.368264+0000","last_fullsized":"2024-03-06T07:27:09.368264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374468+0000","last_change":"2024-03-06T07:27:09.373174+0000","last_active":"2024-03-06T07:27:09.374468+0000","last_peered":"2024-03-06T07:27:09.374468+0000","last_clean":"2024-03-06T07:27:09.374468+0000","last_became_active":"2024-03-06T07:27:09.372894+0000","last_became_peered":"2024-03-06T07:27:09.372894+0000","last_unstale":"2024-03-06T07:27:09.374468+0000","last_undegraded":"2024-03-06T07:27:09.374468+0000","last_fullsized":"2024-03-06T07:27:09.374468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:34.062787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359625+0000","last_change":"2024-03-06T07:27:09.359625+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359625+0000","last_undegraded":"2024-03-06T07:27:09.359625+0000","last_fullsized":"2024-03-06T07:27:09.359625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.377291+0000","last_change":"2024-03-06T07:27:09.374733+0000","last_active":"2024-03-06T07:27:09.377291+0000","last_peered":"2024-03-06T07:27:09.377291+0000","last_clean":"2024-03-06T07:27:09.377291+0000","last_became_active":"2024-03-06T07:27:09.374380+0000","last_became_peered":"2024-03-06T07:27:09.374380+0000","last_unstale":"2024-03-06T07:27:09.377291+0000","last_undegraded":"2024-03-06T07:27:09.377291+0000","last_fullsized":"2024-03-06T07:27:09.377291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:10.906485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364417+0000","last_change":"2024-03-06T07:27:09.364417+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364417+0000","last_undegraded":"2024-03-06T07:27:09.364417+0000","last_fullsized":"2024-03-06T07:27:09.364417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.377859+0000","last_change":"2024-03-06T07:27:09.375768+0000","last_active":"2024-03-06T07:27:09.377859+0000","last_peered":"2024-03-06T07:27:09.377859+0000","last_clean":"2024-03-06T07:27:09.377859+0000","last_became_active":"2024-03-06T07:27:09.375512+0000","last_became_peered":"2024-03-06T07:27:09.375512+0000","last_unstale":"2024-03-06T07:27:09.377859+0000","last_undegraded":"2024-03-06T07:27:09.377859+0000","last_fullsized":"2024-03-06T07:27:09.377859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:25.888556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360242+0000","last_change":"2024-03-06T07:27:09.360242+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360242+0000","last_undegraded":"2024-03-06T07:27:09.360242+0000","last_fullsized":"2024-03-06T07:27:09.360242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371369+0000","last_change":"2024-03-06T07:27:09.369058+0000","last_active":"2024-03-06T07:27:09.371369+0000","last_peered":"2024-03-06T07:27:09.371369+0000","last_clean":"2024-03-06T07:27:09.371369+0000","last_became_active":"2024-03-06T07:27:09.368762+0000","last_became_peered":"2024-03-06T07:27:09.368762+0000","last_unstale":"2024-03-06T07:27:09.371369+0000","last_undegraded":"2024-03-06T07:27:09.371369+0000","last_fullsized":"2024-03-06T07:27:09.371369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:20.708389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368719+0000","last_change":"2024-03-06T07:27:09.368719+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368719+0000","last_undegraded":"2024-03-06T07:27:09.368719+0000","last_fullsized":"2024-03-06T07:27:09.368719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.379184+0000","last_change":"2024-03-06T07:27:09.377310+0000","last_active":"2024-03-06T07:27:09.379184+0000","last_peered":"2024-03-06T07:27:09.379184+0000","last_clean":"2024-03-06T07:27:09.379184+0000","last_became_active":"2024-03-06T07:27:09.377048+0000","last_became_peered":"2024-03-06T07:27:09.377048+0000","last_unstale":"2024-03-06T07:27:09.379184+0000","last_undegraded":"2024-03-06T07:27:09.379184+0000","last_fullsized":"2024-03-06T07:27:09.379184+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:23.219552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.358988+0000","last_change":"2024-03-06T07:27:09.358988+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.358988+0000","last_undegraded":"2024-03-06T07:27:09.358988+0000","last_fullsized":"2024-03-06T07:27:09.358988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.381029+0000","last_change":"2024-03-06T07:27:09.380149+0000","last_active":"2024-03-06T07:27:09.381029+0000","last_peered":"2024-03-06T07:27:09.381029+0000","last_clean":"2024-03-06T07:27:09.381029+0000","last_became_active":"2024-03-06T07:27:09.379853+0000","last_became_peered":"2024-03-06T07:27:09.379853+0000","last_unstale":"2024-03-06T07:27:09.381029+0000","last_undegraded":"2024-03-06T07:27:09.381029+0000","last_fullsized":"2024-03-06T07:27:09.381029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:09.153513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364174+0000","last_change":"2024-03-06T07:27:09.364174+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364174+0000","last_undegraded":"2024-03-06T07:27:09.364174+0000","last_fullsized":"2024-03-06T07:27:09.364174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380994+0000","last_change":"2024-03-06T07:27:09.375900+0000","last_active":"2024-03-06T07:27:09.380994+0000","last_peered":"2024-03-06T07:27:09.380994+0000","last_clean":"2024-03-06T07:27:09.380994+0000","last_became_active":"2024-03-06T07:27:09.375583+0000","last_became_peered":"2024-03-06T07:27:09.375583+0000","last_unstale":"2024-03-06T07:27:09.380994+0000","last_undegraded":"2024-03-06T07:27:09.380994+0000","last_fullsized":"2024-03-06T07:27:09.380994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:53.571465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365127+0000","last_change":"2024-03-06T07:27:09.365127+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365127+0000","last_undegraded":"2024-03-06T07:27:09.365127+0000","last_fullsized":"2024-03-06T07:27:09.365127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.369385+0000","last_change":"2024-03-06T07:27:09.367702+0000","last_active":"2024-03-06T07:27:09.369385+0000","last_peered":"2024-03-06T07:27:09.369385+0000","last_clean":"2024-03-06T07:27:09.369385+0000","last_became_active":"2024-03-06T07:27:09.367456+0000","last_became_peered":"2024-03-06T07:27:09.367456+0000","last_unstale":"2024-03-06T07:27:09.369385+0000","last_undegraded":"2024-03-06T07:27:09.369385+0000","last_fullsized":"2024-03-06T07:27:09.369385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:17.715304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360340+0000","last_change":"2024-03-06T07:27:09.360340+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360340+0000","last_undegraded":"2024-03-06T07:27:09.360340+0000","last_fullsized":"2024-03-06T07:27:09.360340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371484+0000","last_change":"2024-03-06T07:27:09.367998+0000","last_active":"2024-03-06T07:27:09.371484+0000","last_peered":"2024-03-06T07:27:09.371484+0000","last_clean":"2024-03-06T07:27:09.371484+0000","last_became_active":"2024-03-06T07:27:09.367731+0000","last_became_peered":"2024-03-06T07:27:09.367731+0000","last_unstale":"2024-03-06T07:27:09.371484+0000","last_undegraded":"2024-03-06T07:27:09.371484+0000","last_fullsized":"2024-03-06T07:27:09.371484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:14.629790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365984+0000","last_change":"2024-03-06T07:27:09.365984+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365984+0000","last_undegraded":"2024-03-06T07:27:09.365984+0000","last_fullsized":"2024-03-06T07:27:09.365984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374853+0000","last_change":"2024-03-06T07:27:09.374468+0000","last_active":"2024-03-06T07:27:09.374853+0000","last_peered":"2024-03-06T07:27:09.374853+0000","last_clean":"2024-03-06T07:27:09.374853+0000","last_became_active":"2024-03-06T07:27:09.374183+0000","last_became_peered":"2024-03-06T07:27:09.374183+0000","last_unstale":"2024-03-06T07:27:09.374853+0000","last_undegraded":"2024-03-06T07:27:09.374853+0000","last_fullsized":"2024-03-06T07:27:09.374853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:49.597272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359152+0000","last_change":"2024-03-06T07:27:09.359152+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359152+0000","last_undegraded":"2024-03-06T07:27:09.359152+0000","last_fullsized":"2024-03-06T07:27:09.359152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.370888+0000","last_change":"2024-03-06T07:27:09.368796+0000","last_active":"2024-03-06T07:27:09.370888+0000","last_peered":"2024-03-06T07:27:09.370888+0000","last_clean":"2024-03-06T07:27:09.370888+0000","last_became_active":"2024-03-06T07:27:09.368017+0000","last_became_peered":"2024-03-06T07:27:09.368017+0000","last_unstale":"2024-03-06T07:27:09.370888+0000","last_undegraded":"2024-03-06T07:27:09.370888+0000","last_fullsized":"2024-03-06T07:27:09.370888+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:49.597272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.363467+0000","last_change":"2024-03-06T07:27:09.363467+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.363467+0000","last_undegraded":"2024-03-06T07:27:09.363467+0000","last_fullsized":"2024-03-06T07:27:09.363467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.373600+0000","last_change":"2024-03-06T07:27:09.373156+0000","last_active":"2024-03-06T07:27:09.373600+0000","last_peered":"2024-03-06T07:27:09.373600+0000","last_clean":"2024-03-06T07:27:09.373600+0000","last_became_active":"2024-03-06T07:27:09.372848+0000","last_became_peered":"2024-03-06T07:27:09.372848+0000","last_unstale":"2024-03-06T07:27:09.373600+0000","last_undegraded":"2024-03-06T07:27:09.373600+0000","last_fullsized":"2024-03-06T07:27:09.373600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:17.715304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.374217+0000","last_change":"2024-03-06T07:27:09.374217+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.374217+0000","last_undegraded":"2024-03-06T07:27:09.374217+0000","last_fullsized":"2024-03-06T07:27:09.374217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.375143+0000","last_change":"2024-03-06T07:27:09.371787+0000","last_active":"2024-03-06T07:27:09.375143+0000","last_peered":"2024-03-06T07:27:09.375143+0000","last_clean":"2024-03-06T07:27:09.375143+0000","last_became_active":"2024-03-06T07:27:09.371100+0000","last_became_peered":"2024-03-06T07:27:09.371100+0000","last_unstale":"2024-03-06T07:27:09.375143+0000","last_undegraded":"2024-03-06T07:27:09.375143+0000","last_fullsized":"2024-03-06T07:27:09.375143+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:03.753803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":288,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.361425+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:27:09.361425+0000","last_peered":"2024-03-06T07:27:09.361425+0000","last_clean":"2024-03-06T07:27:09.361425+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:27:09.361425+0000","last_undegraded":"2024-03-06T07:27:09.361425+0000","last_fullsized":"2024-03-06T07:27:09.361425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1014782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":418139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1014782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:14.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.996+0000 7f2078471640 1 Processor -- start 2024-03-06T07:27:15.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.996+0000 7f2078471640 1 -- start start 2024-03-06T07:27:15.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f2078471640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f20700d02e0 0x7f20700cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f2078471640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 0x7f20700b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f2078471640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f20700d06c0 0x7f20700b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f2078471640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700b0b80 con 0x7f20700d02e0 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f2078471640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700aeec0 con 0x7f20700d06c0 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f2078471640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20700b0dd0 con 0x7f20700b4260 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f207620d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f20700d02e0 0x7f20700cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f207620d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f20700d02e0 0x7f20700cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34296/0 (socket says 172.21.15.27:34296) 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f207620d640 1 -- 172.21.15.27:0/281361354 learned_addr learned my addr 172.21.15.27:0/281361354 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f2076a0e640 1 --2- 172.21.15.27:0/281361354 >> v2:172.21.15.69:3300/0 conn(0x7f20700d06c0 0x7f20700b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f2075a0c640 1 --2- 172.21.15.27:0/281361354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 0x7f20700b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f207620d640 1 -- 172.21.15.27:0/281361354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 msgr2=0x7f20700b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f207620d640 1 --2- 172.21.15.27:0/281361354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 0x7f20700b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f207620d640 1 -- 172.21.15.27:0/281361354 >> v2:172.21.15.69:3300/0 conn(0x7f20700d06c0 msgr2=0x7f20700b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f207620d640 1 --2- 172.21.15.27:0/281361354 >> v2:172.21.15.69:3300/0 conn(0x7f20700d06c0 0x7f20700b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.997+0000 7f207620d640 1 -- 172.21.15.27:0/281361354 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20700b10b0 con 0x7f20700d02e0 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.998+0000 7f207620d640 1 --2- 172.21.15.27:0/281361354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 0x7f20700cfa00 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2070071d50 tx=0x7f206c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a7549f8eaeda13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.998+0000 7f207520b640 1 -- 172.21.15.27:0/281361354 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c034070 con 0x7f20700d02e0 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.998+0000 7f207520b640 1 -- 172.21.15.27:0/281361354 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206c030860 con 0x7f20700d02e0 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.998+0000 7f207520b640 1 -- 172.21.15.27:0/281361354 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c030c30 con 0x7f20700d02e0 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.999+0000 7f2078471640 1 -- 172.21.15.27:0/281361354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 msgr2=0x7f20700cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.999+0000 7f2078471640 1 --2- 172.21.15.27:0/281361354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 0x7f20700cfa00 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2070071d50 tx=0x7f206c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.999+0000 7f2078471640 1 -- 172.21.15.27:0/281361354 shutdown_connections 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.999+0000 7f2078471640 1 --2- 172.21.15.27:0/281361354 >> v2:172.21.15.69:3300/0 conn(0x7f20700d06c0 0x7f20700b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.999+0000 7f2078471640 1 --2- 172.21.15.27:0/281361354 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 0x7f20700b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.999+0000 7f2078471640 1 --2- 172.21.15.27:0/281361354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 0x7f20700cfa00 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.999+0000 7f2078471640 1 -- 172.21.15.27:0/281361354 >> 172.21.15.27:0/281361354 conn(0x7f20700b36d0 msgr2=0x7f20700b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.999+0000 7f2078471640 1 -- 172.21.15.27:0/281361354 shutdown_connections 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:14.999+0000 7f2078471640 1 -- 172.21.15.27:0/281361354 wait complete. 2024-03-06T07:27:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.000+0000 7f2078471640 1 Processor -- start 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.000+0000 7f2078471640 1 -- start start 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2078471640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 0x7f20700cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2078471640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 0x7f20700c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2078471640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20700d06c0 0x7f2070154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2078471640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20700b2100 con 0x7f20700d02e0 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2078471640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20700b1f80 con 0x7f20700d06c0 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2078471640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20700b2280 con 0x7f20700b4260 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2075a0c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 0x7f20700c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2075a0c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 0x7f20700c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34300/0 (socket says 172.21.15.27:34300) 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2075a0c640 1 -- 172.21.15.27:0/2337587414 learned_addr learned my addr 172.21.15.27:0/2337587414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f207620d640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 0x7f20700cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2076a0e640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20700d06c0 0x7f2070154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2075a0c640 1 -- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 msgr2=0x7f20700cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2075a0c640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 0x7f20700cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2075a0c640 1 -- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20700d06c0 msgr2=0x7f2070154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2075a0c640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20700d06c0 0x7f2070154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2075a0c640 1 -- 172.21.15.27:0/2337587414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2070144a80 con 0x7f20700d02e0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.001+0000 7f2075a0c640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 0x7f20700c8bd0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f206000d5f0 tx=0x7f206000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.002+0000 7f2066ffd640 1 -- 172.21.15.27:0/2337587414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2060004240 con 0x7f20700d02e0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.002+0000 7f2066ffd640 1 -- 172.21.15.27:0/2337587414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20600043e0 con 0x7f20700d02e0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.002+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2070153bc0 con 0x7f20700d02e0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.002+0000 7f2066ffd640 1 -- 172.21.15.27:0/2337587414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2060003610 con 0x7f20700d02e0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.002+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2070154180 con 0x7f20700d02e0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.003+0000 7f2066ffd640 1 -- 172.21.15.27:0/2337587414 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f206001b020 con 0x7f20700d02e0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.003+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2070068700 con 0x7f20700d02e0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.003+0000 7f2066ffd640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f205406b4d0 0x7f205406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.004+0000 7f2066ffd640 1 -- 172.21.15.27:0/2337587414 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2060089e90 con 0x7f20700d02e0 2024-03-06T07:27:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.004+0000 7f207620d640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f205406b4d0 0x7f205406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.004+0000 7f207620d640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f205406b4d0 0x7f205406d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f206c036000 tx=0x7f206c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.008+0000 7f2066ffd640 1 -- 172.21.15.27:0/2337587414 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2060058da0 con 0x7f20700d02e0 2024-03-06T07:27:15.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.184+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20700c7a60 con 0x7f205406b4d0 2024-03-06T07:27:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.190+0000 7f2066ffd640 1 -- 172.21.15.27:0/2337587414 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246760 (secure 0 0 0) 0x7f206c030880 con 0x7f205406b4d0 2024-03-06T07:27:15.194 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:15.195 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f205406b4d0 msgr2=0x7f205406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f205406b4d0 0x7f205406d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f206c036000 tx=0x7f206c03b040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 msgr2=0x7f20700c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 0x7f20700c8bd0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f206000d5f0 tx=0x7f206000dab0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 shutdown_connections 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f205406b4d0 0x7f205406d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20700d06c0 0x7f2070154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20700d02e0 0x7f20700c8bd0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 --2- 172.21.15.27:0/2337587414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20700b4260 0x7f20700cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 >> 172.21.15.27:0/2337587414 conn(0x7f20700b36d0 msgr2=0x7f20700b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 shutdown_connections 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.193+0000 7f2078471640 1 -- 172.21.15.27:0/2337587414 wait complete. 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-03-06T07:27:14.050694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258488,"kb_used_data":9020,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716232,"statfs":{"total":773094113280,"available":772829421568,"internally_reserved":0,"allocated":9236480,"data_stored":4569773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.694133"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366331+0000","last_change":"2024-03-06T07:27:09.366331+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366331+0000","last_undegraded":"2024-03-06T07:27:09.366331+0000","last_fullsized":"2024-03-06T07:27:09.366331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365065+0000","last_change":"2024-03-06T07:27:09.365065+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365065+0000","last_undegraded":"2024-03-06T07:27:09.365065+0000","last_fullsized":"2024-03-06T07:27:09.365065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361750+0000","last_change":"2024-03-06T07:27:09.361750+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361750+0000","last_undegraded":"2024-03-06T07:27:09.361750+0000","last_fullsized":"2024-03-06T07:27:09.361750+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368354+0000","last_change":"2024-03-06T07:27:09.368354+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368354+0000","last_undegraded":"2024-03-06T07:27:09.368354+0000","last_fullsized":"2024-03-06T07:27:09.368354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368139+0000","last_change":"2024-03-06T07:27:09.368139+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368139+0000","last_undegraded":"2024-03-06T07:27:09.368139+0000","last_fullsized":"2024-03-06T07:27:09.368139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360754+0000","last_change":"2024-03-06T07:27:09.360754+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360754+0000","last_undegraded":"2024-03-06T07:27:09.360754+0000","last_fullsized":"2024-03-06T07:27:09.360754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366900+0000","last_change":"2024-03-06T07:27:09.366900+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366900+0000","last_undegraded":"2024-03-06T07:27:09.366900+0000","last_fullsized":"2024-03-06T07:27:09.366900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360184+0000","last_change":"2024-03-06T07:27:09.360184+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360184+0000","last_undegraded":"2024-03-06T07:27:09.360184+0000","last_fullsized":"2024-03-06T07:27:09.360184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360603+0000","last_change":"2024-03-06T07:27:09.360603+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360603+0000","last_undegraded":"2024-03-06T07:27:09.360603+0000","last_fullsized":"2024-03-06T07:27:09.360603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364335+0000","last_change":"2024-03-06T07:27:09.364335+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364335+0000","last_undegraded":"2024-03-06T07:27:09.364335+0000","last_fullsized":"2024-03-06T07:27:09.364335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364708+0000","last_change":"2024-03-06T07:27:09.364708+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364708+0000","last_undegraded":"2024-03-06T07:27:09.364708+0000","last_fullsized":"2024-03-06T07:27:09.364708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366392+0000","last_change":"2024-03-06T07:27:09.366392+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366392+0000","last_undegraded":"2024-03-06T07:27:09.366392+0000","last_fullsized":"2024-03-06T07:27:09.366392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361471+0000","last_change":"2024-03-06T07:27:09.361471+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361471+0000","last_undegraded":"2024-03-06T07:27:09.361471+0000","last_fullsized":"2024-03-06T07:27:09.361471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368716+0000","last_change":"2024-03-06T07:27:09.368716+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368716+0000","last_undegraded":"2024-03-06T07:27:09.368716+0000","last_fullsized":"2024-03-06T07:27:09.368716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.362610+0000","last_change":"2024-03-06T07:27:09.362610+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.362610+0000","last_undegraded":"2024-03-06T07:27:09.362610+0000","last_fullsized":"2024-03-06T07:27:09.362610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365314+0000","last_change":"2024-03-06T07:27:09.365314+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365314+0000","last_undegraded":"2024-03-06T07:27:09.365314+0000","last_fullsized":"2024-03-06T07:27:09.365314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364605+0000","last_change":"2024-03-06T07:27:09.364605+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364605+0000","last_undegraded":"2024-03-06T07:27:09.364605+0000","last_fullsized":"2024-03-06T07:27:09.364605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366597+0000","last_change":"2024-03-06T07:27:09.366597+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366597+0000","last_undegraded":"2024-03-06T07:27:09.366597+0000","last_fullsized":"2024-03-06T07:27:09.366597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367523+0000","last_change":"2024-03-06T07:27:09.367523+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367523+0000","last_undegraded":"2024-03-06T07:27:09.367523+0000","last_fullsized":"2024-03-06T07:27:09.367523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367857+0000","last_change":"2024-03-06T07:27:09.367857+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367857+0000","last_undegraded":"2024-03-06T07:27:09.367857+0000","last_fullsized":"2024-03-06T07:27:09.367857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366174+0000","last_change":"2024-03-06T07:27:09.366174+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366174+0000","last_undegraded":"2024-03-06T07:27:09.366174+0000","last_fullsized":"2024-03-06T07:27:09.366174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369885+0000","last_change":"2024-03-06T07:27:09.369885+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369885+0000","last_undegraded":"2024-03-06T07:27:09.369885+0000","last_fullsized":"2024-03-06T07:27:09.369885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367067+0000","last_change":"2024-03-06T07:27:09.367067+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367067+0000","last_undegraded":"2024-03-06T07:27:09.367067+0000","last_fullsized":"2024-03-06T07:27:09.367067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364968+0000","last_change":"2024-03-06T07:27:09.364968+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364968+0000","last_undegraded":"2024-03-06T07:27:09.364968+0000","last_fullsized":"2024-03-06T07:27:09.364968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367295+0000","last_change":"2024-03-06T07:27:09.367295+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367295+0000","last_undegraded":"2024-03-06T07:27:09.367295+0000","last_fullsized":"2024-03-06T07:27:09.367295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360671+0000","last_change":"2024-03-06T07:27:09.360671+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360671+0000","last_undegraded":"2024-03-06T07:27:09.360671+0000","last_fullsized":"2024-03-06T07:27:09.360671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366535+0000","last_change":"2024-03-06T07:27:09.366535+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366535+0000","last_undegraded":"2024-03-06T07:27:09.366535+0000","last_fullsized":"2024-03-06T07:27:09.366535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365953+0000","last_change":"2024-03-06T07:27:09.365953+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365953+0000","last_undegraded":"2024-03-06T07:27:09.365953+0000","last_fullsized":"2024-03-06T07:27:09.365953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369681+0000","last_change":"2024-03-06T07:27:09.369681+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369681+0000","last_undegraded":"2024-03-06T07:27:09.369681+0000","last_fullsized":"2024-03-06T07:27:09.369681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369176+0000","last_change":"2024-03-06T07:27:09.369176+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369176+0000","last_undegraded":"2024-03-06T07:27:09.369176+0000","last_fullsized":"2024-03-06T07:27:09.369176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367187+0000","last_change":"2024-03-06T07:27:09.367187+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367187+0000","last_undegraded":"2024-03-06T07:27:09.367187+0000","last_fullsized":"2024-03-06T07:27:09.367187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.363528+0000","last_change":"2024-03-06T07:27:09.363528+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.363528+0000","last_undegraded":"2024-03-06T07:27:09.363528+0000","last_fullsized":"2024-03-06T07:27:09.363528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369136+0000","last_change":"2024-03-06T07:27:09.369136+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369136+0000","last_undegraded":"2024-03-06T07:27:09.369136+0000","last_fullsized":"2024-03-06T07:27:09.369136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383880+0000","last_change":"2024-03-06T07:27:09.378545+0000","last_active":"2024-03-06T07:27:09.383880+0000","last_peered":"2024-03-06T07:27:09.383880+0000","last_clean":"2024-03-06T07:27:09.383880+0000","last_became_active":"2024-03-06T07:27:09.378276+0000","last_became_peered":"2024-03-06T07:27:09.378276+0000","last_unstale":"2024-03-06T07:27:09.383880+0000","last_undegraded":"2024-03-06T07:27:09.383880+0000","last_fullsized":"2024-03-06T07:27:09.383880+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:09.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365340+0000","last_change":"2024-03-06T07:27:09.365340+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365340+0000","last_undegraded":"2024-03-06T07:27:09.365340+0000","last_fullsized":"2024-03-06T07:27:09.365340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.381029+0000","last_change":"2024-03-06T07:27:09.375904+0000","last_active":"2024-03-06T07:27:09.381029+0000","last_peered":"2024-03-06T07:27:09.381029+0000","last_clean":"2024-03-06T07:27:09.381029+0000","last_became_active":"2024-03-06T07:27:09.375566+0000","last_became_peered":"2024-03-06T07:27:09.375566+0000","last_unstale":"2024-03-06T07:27:09.381029+0000","last_undegraded":"2024-03-06T07:27:09.381029+0000","last_fullsized":"2024-03-06T07:27:09.381029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:07.197604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365542+0000","last_change":"2024-03-06T07:27:09.365542+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365542+0000","last_undegraded":"2024-03-06T07:27:09.365542+0000","last_fullsized":"2024-03-06T07:27:09.365542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374222+0000","last_change":"2024-03-06T07:27:09.371083+0000","last_active":"2024-03-06T07:27:09.374222+0000","last_peered":"2024-03-06T07:27:09.374222+0000","last_clean":"2024-03-06T07:27:09.374222+0000","last_became_active":"2024-03-06T07:27:09.370820+0000","last_became_peered":"2024-03-06T07:27:09.370820+0000","last_unstale":"2024-03-06T07:27:09.374222+0000","last_undegraded":"2024-03-06T07:27:09.374222+0000","last_fullsized":"2024-03-06T07:27:09.374222+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:29.304676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359572+0000","last_change":"2024-03-06T07:27:09.359572+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359572+0000","last_undegraded":"2024-03-06T07:27:09.359572+0000","last_fullsized":"2024-03-06T07:27:09.359572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.378612+0000","last_change":"2024-03-06T07:27:09.374749+0000","last_active":"2024-03-06T07:27:09.378612+0000","last_peered":"2024-03-06T07:27:09.378612+0000","last_clean":"2024-03-06T07:27:09.378612+0000","last_became_active":"2024-03-06T07:27:09.374421+0000","last_became_peered":"2024-03-06T07:27:09.374421+0000","last_unstale":"2024-03-06T07:27:09.378612+0000","last_undegraded":"2024-03-06T07:27:09.378612+0000","last_fullsized":"2024-03-06T07:27:09.378612+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:36.751221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368708+0000","last_change":"2024-03-06T07:27:09.368708+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368708+0000","last_undegraded":"2024-03-06T07:27:09.368708+0000","last_fullsized":"2024-03-06T07:27:09.368708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383926+0000","last_change":"2024-03-06T07:27:09.379796+0000","last_active":"2024-03-06T07:27:09.383926+0000","last_peered":"2024-03-06T07:27:09.383926+0000","last_clean":"2024-03-06T07:27:09.383926+0000","last_became_active":"2024-03-06T07:27:09.379609+0000","last_became_peered":"2024-03-06T07:27:09.379609+0000","last_unstale":"2024-03-06T07:27:09.383926+0000","last_undegraded":"2024-03-06T07:27:09.383926+0000","last_fullsized":"2024-03-06T07:27:09.383926+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:53.571465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367797+0000","last_change":"2024-03-06T07:27:09.367797+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367797+0000","last_undegraded":"2024-03-06T07:27:09.367797+0000","last_fullsized":"2024-03-06T07:27:09.367797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380992+0000","last_change":"2024-03-06T07:27:09.375952+0000","last_active":"2024-03-06T07:27:09.380992+0000","last_peered":"2024-03-06T07:27:09.380992+0000","last_clean":"2024-03-06T07:27:09.380992+0000","last_became_active":"2024-03-06T07:27:09.375584+0000","last_became_peered":"2024-03-06T07:27:09.375584+0000","last_unstale":"2024-03-06T07:27:09.380992+0000","last_undegraded":"2024-03-06T07:27:09.380992+0000","last_fullsized":"2024-03-06T07:27:09.380992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:03.362736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359735+0000","last_change":"2024-03-06T07:27:09.359735+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359735+0000","last_undegraded":"2024-03-06T07:27:09.359735+0000","last_fullsized":"2024-03-06T07:27:09.359735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.370532+0000","last_change":"2024-03-06T07:27:09.366681+0000","last_active":"2024-03-06T07:27:09.370532+0000","last_peered":"2024-03-06T07:27:09.370532+0000","last_clean":"2024-03-06T07:27:09.370532+0000","last_became_active":"2024-03-06T07:27:09.366329+0000","last_became_peered":"2024-03-06T07:27:09.366329+0000","last_unstale":"2024-03-06T07:27:09.370532+0000","last_undegraded":"2024-03-06T07:27:09.370532+0000","last_fullsized":"2024-03-06T07:27:09.370532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:07.928354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361649+0000","last_change":"2024-03-06T07:27:09.361649+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361649+0000","last_undegraded":"2024-03-06T07:27:09.361649+0000","last_fullsized":"2024-03-06T07:27:09.361649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380679+0000","last_change":"2024-03-06T07:27:09.378990+0000","last_active":"2024-03-06T07:27:09.380679+0000","last_peered":"2024-03-06T07:27:09.380679+0000","last_clean":"2024-03-06T07:27:09.380679+0000","last_became_active":"2024-03-06T07:27:09.378578+0000","last_became_peered":"2024-03-06T07:27:09.378578+0000","last_unstale":"2024-03-06T07:27:09.380679+0000","last_undegraded":"2024-03-06T07:27:09.380679+0000","last_fullsized":"2024-03-06T07:27:09.380679+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:41.025662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.362615+0000","last_change":"2024-03-06T07:27:09.362615+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.362615+0000","last_undegraded":"2024-03-06T07:27:09.362615+0000","last_fullsized":"2024-03-06T07:27:09.362615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.372532+0000","last_change":"2024-03-06T07:27:09.369411+0000","last_active":"2024-03-06T07:27:09.372532+0000","last_peered":"2024-03-06T07:27:09.372532+0000","last_clean":"2024-03-06T07:27:09.372532+0000","last_became_active":"2024-03-06T07:27:09.369100+0000","last_became_peered":"2024-03-06T07:27:09.369100+0000","last_unstale":"2024-03-06T07:27:09.372532+0000","last_undegraded":"2024-03-06T07:27:09.372532+0000","last_fullsized":"2024-03-06T07:27:09.372532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:07.197604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360078+0000","last_change":"2024-03-06T07:27:09.360078+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360078+0000","last_undegraded":"2024-03-06T07:27:09.360078+0000","last_fullsized":"2024-03-06T07:27:09.360078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.375998+0000","last_change":"2024-03-06T07:27:09.372377+0000","last_active":"2024-03-06T07:27:09.375998+0000","last_peered":"2024-03-06T07:27:09.375998+0000","last_clean":"2024-03-06T07:27:09.375998+0000","last_became_active":"2024-03-06T07:27:09.371991+0000","last_became_peered":"2024-03-06T07:27:09.371991+0000","last_unstale":"2024-03-06T07:27:09.375998+0000","last_undegraded":"2024-03-06T07:27:09.375998+0000","last_fullsized":"2024-03-06T07:27:09.375998+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:03.362736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367153+0000","last_change":"2024-03-06T07:27:09.367153+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367153+0000","last_undegraded":"2024-03-06T07:27:09.367153+0000","last_fullsized":"2024-03-06T07:27:09.367153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383917+0000","last_change":"2024-03-06T07:27:09.378597+0000","last_active":"2024-03-06T07:27:09.383917+0000","last_peered":"2024-03-06T07:27:09.383917+0000","last_clean":"2024-03-06T07:27:09.383917+0000","last_became_active":"2024-03-06T07:27:09.378334+0000","last_became_peered":"2024-03-06T07:27:09.378334+0000","last_unstale":"2024-03-06T07:27:09.383917+0000","last_undegraded":"2024-03-06T07:27:09.383917+0000","last_fullsized":"2024-03-06T07:27:09.383917+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:20.907611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366065+0000","last_change":"2024-03-06T07:27:09.366065+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366065+0000","last_undegraded":"2024-03-06T07:27:09.366065+0000","last_fullsized":"2024-03-06T07:27:09.366065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.368379+0000","last_change":"2024-03-06T07:27:09.365562+0000","last_active":"2024-03-06T07:27:09.368379+0000","last_peered":"2024-03-06T07:27:09.368379+0000","last_clean":"2024-03-06T07:27:09.368379+0000","last_became_active":"2024-03-06T07:27:09.365279+0000","last_became_peered":"2024-03-06T07:27:09.365279+0000","last_unstale":"2024-03-06T07:27:09.368379+0000","last_undegraded":"2024-03-06T07:27:09.368379+0000","last_fullsized":"2024-03-06T07:27:09.368379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:35.367786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366631+0000","last_change":"2024-03-06T07:27:09.366631+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366631+0000","last_undegraded":"2024-03-06T07:27:09.366631+0000","last_fullsized":"2024-03-06T07:27:09.366631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.379502+0000","last_change":"2024-03-06T07:27:09.378045+0000","last_active":"2024-03-06T07:27:09.379502+0000","last_peered":"2024-03-06T07:27:09.379502+0000","last_clean":"2024-03-06T07:27:09.379502+0000","last_became_active":"2024-03-06T07:27:09.377782+0000","last_became_peered":"2024-03-06T07:27:09.377782+0000","last_unstale":"2024-03-06T07:27:09.379502+0000","last_undegraded":"2024-03-06T07:27:09.379502+0000","last_fullsized":"2024-03-06T07:27:09.379502+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:47.722593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359370+0000","last_change":"2024-03-06T07:27:09.359370+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359370+0000","last_undegraded":"2024-03-06T07:27:09.359370+0000","last_fullsized":"2024-03-06T07:27:09.359370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380600+0000","last_change":"2024-03-06T07:27:09.378309+0000","last_active":"2024-03-06T07:27:09.380600+0000","last_peered":"2024-03-06T07:27:09.380600+0000","last_clean":"2024-03-06T07:27:09.380600+0000","last_became_active":"2024-03-06T07:27:09.378012+0000","last_became_peered":"2024-03-06T07:27:09.378012+0000","last_unstale":"2024-03-06T07:27:09.380600+0000","last_undegraded":"2024-03-06T07:27:09.380600+0000","last_fullsized":"2024-03-06T07:27:09.380600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:49.071313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365822+0000","last_change":"2024-03-06T07:27:09.365822+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365822+0000","last_undegraded":"2024-03-06T07:27:09.365822+0000","last_fullsized":"2024-03-06T07:27:09.365822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.372446+0000","last_change":"2024-03-06T07:27:09.370907+0000","last_active":"2024-03-06T07:27:09.372446+0000","last_peered":"2024-03-06T07:27:09.372446+0000","last_clean":"2024-03-06T07:27:09.372446+0000","last_became_active":"2024-03-06T07:27:09.370655+0000","last_became_peered":"2024-03-06T07:27:09.370655+0000","last_unstale":"2024-03-06T07:27:09.372446+0000","last_undegraded":"2024-03-06T07:27:09.372446+0000","last_fullsized":"2024-03-06T07:27:09.372446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:07.928354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368019+0000","last_change":"2024-03-06T07:27:09.368019+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368019+0000","last_undegraded":"2024-03-06T07:27:09.368019+0000","last_fullsized":"2024-03-06T07:27:09.368019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371431+0000","last_change":"2024-03-06T07:27:09.368550+0000","last_active":"2024-03-06T07:27:09.371431+0000","last_peered":"2024-03-06T07:27:09.371431+0000","last_clean":"2024-03-06T07:27:09.371431+0000","last_became_active":"2024-03-06T07:27:09.367237+0000","last_became_peered":"2024-03-06T07:27:09.367237+0000","last_unstale":"2024-03-06T07:27:09.371431+0000","last_undegraded":"2024-03-06T07:27:09.371431+0000","last_fullsized":"2024-03-06T07:27:09.371431+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:43.340074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359006+0000","last_change":"2024-03-06T07:27:09.359006+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359006+0000","last_undegraded":"2024-03-06T07:27:09.359006+0000","last_fullsized":"2024-03-06T07:27:09.359006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.369618+0000","last_change":"2024-03-06T07:27:09.367473+0000","last_active":"2024-03-06T07:27:09.369618+0000","last_peered":"2024-03-06T07:27:09.369618+0000","last_clean":"2024-03-06T07:27:09.369618+0000","last_became_active":"2024-03-06T07:27:09.367201+0000","last_became_peered":"2024-03-06T07:27:09.367201+0000","last_unstale":"2024-03-06T07:27:09.369618+0000","last_undegraded":"2024-03-06T07:27:09.369618+0000","last_fullsized":"2024-03-06T07:27:09.369618+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:26.071654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364913+0000","last_change":"2024-03-06T07:27:09.364913+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364913+0000","last_undegraded":"2024-03-06T07:27:09.364913+0000","last_fullsized":"2024-03-06T07:27:09.364913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374806+0000","last_change":"2024-03-06T07:27:09.373607+0000","last_active":"2024-03-06T07:27:09.374806+0000","last_peered":"2024-03-06T07:27:09.374806+0000","last_clean":"2024-03-06T07:27:09.374806+0000","last_became_active":"2024-03-06T07:27:09.373306+0000","last_became_peered":"2024-03-06T07:27:09.373306+0000","last_unstale":"2024-03-06T07:27:09.374806+0000","last_undegraded":"2024-03-06T07:27:09.374806+0000","last_fullsized":"2024-03-06T07:27:09.374806+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:32.055587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365624+0000","last_change":"2024-03-06T07:27:09.365624+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365624+0000","last_undegraded":"2024-03-06T07:27:09.365624+0000","last_fullsized":"2024-03-06T07:27:09.365624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371513+0000","last_change":"2024-03-06T07:27:09.367584+0000","last_active":"2024-03-06T07:27:09.371513+0000","last_peered":"2024-03-06T07:27:09.371513+0000","last_clean":"2024-03-06T07:27:09.371513+0000","last_became_active":"2024-03-06T07:27:09.366434+0000","last_became_peered":"2024-03-06T07:27:09.366434+0000","last_unstale":"2024-03-06T07:27:09.371513+0000","last_undegraded":"2024-03-06T07:27:09.371513+0000","last_fullsized":"2024-03-06T07:27:09.371513+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:44.994841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368264+0000","last_change":"2024-03-06T07:27:09.368264+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368264+0000","last_undegraded":"2024-03-06T07:27:09.368264+0000","last_fullsized":"2024-03-06T07:27:09.368264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374468+0000","last_change":"2024-03-06T07:27:09.373174+0000","last_active":"2024-03-06T07:27:09.374468+0000","last_peered":"2024-03-06T07:27:09.374468+0000","last_clean":"2024-03-06T07:27:09.374468+0000","last_became_active":"2024-03-06T07:27:09.372894+0000","last_became_peered":"2024-03-06T07:27:09.372894+0000","last_unstale":"2024-03-06T07:27:09.374468+0000","last_undegraded":"2024-03-06T07:27:09.374468+0000","last_fullsized":"2024-03-06T07:27:09.374468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:34.062787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359625+0000","last_change":"2024-03-06T07:27:09.359625+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359625+0000","last_undegraded":"2024-03-06T07:27:09.359625+0000","last_fullsized":"2024-03-06T07:27:09.359625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.377291+0000","last_change":"2024-03-06T07:27:09.374733+0000","last_active":"2024-03-06T07:27:09.377291+0000","last_peered":"2024-03-06T07:27:09.377291+0000","last_clean":"2024-03-06T07:27:09.377291+0000","last_became_active":"2024-03-06T07:27:09.374380+0000","last_became_peered":"2024-03-06T07:27:09.374380+0000","last_unstale":"2024-03-06T07:27:09.377291+0000","last_undegraded":"2024-03-06T07:27:09.377291+0000","last_fullsized":"2024-03-06T07:27:09.377291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:10.906485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364417+0000","last_change":"2024-03-06T07:27:09.364417+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364417+0000","last_undegraded":"2024-03-06T07:27:09.364417+0000","last_fullsized":"2024-03-06T07:27:09.364417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.377859+0000","last_change":"2024-03-06T07:27:09.375768+0000","last_active":"2024-03-06T07:27:09.377859+0000","last_peered":"2024-03-06T07:27:09.377859+0000","last_clean":"2024-03-06T07:27:09.377859+0000","last_became_active":"2024-03-06T07:27:09.375512+0000","last_became_peered":"2024-03-06T07:27:09.375512+0000","last_unstale":"2024-03-06T07:27:09.377859+0000","last_undegraded":"2024-03-06T07:27:09.377859+0000","last_fullsized":"2024-03-06T07:27:09.377859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:25.888556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360242+0000","last_change":"2024-03-06T07:27:09.360242+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360242+0000","last_undegraded":"2024-03-06T07:27:09.360242+0000","last_fullsized":"2024-03-06T07:27:09.360242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371369+0000","last_change":"2024-03-06T07:27:09.369058+0000","last_active":"2024-03-06T07:27:09.371369+0000","last_peered":"2024-03-06T07:27:09.371369+0000","last_clean":"2024-03-06T07:27:09.371369+0000","last_became_active":"2024-03-06T07:27:09.368762+0000","last_became_peered":"2024-03-06T07:27:09.368762+0000","last_unstale":"2024-03-06T07:27:09.371369+0000","last_undegraded":"2024-03-06T07:27:09.371369+0000","last_fullsized":"2024-03-06T07:27:09.371369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:20.708389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368719+0000","last_change":"2024-03-06T07:27:09.368719+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368719+0000","last_undegraded":"2024-03-06T07:27:09.368719+0000","last_fullsized":"2024-03-06T07:27:09.368719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.379184+0000","last_change":"2024-03-06T07:27:09.377310+0000","last_active":"2024-03-06T07:27:09.379184+0000","last_peered":"2024-03-06T07:27:09.379184+0000","last_clean":"2024-03-06T07:27:09.379184+0000","last_became_active":"2024-03-06T07:27:09.377048+0000","last_became_peered":"2024-03-06T07:27:09.377048+0000","last_unstale":"2024-03-06T07:27:09.379184+0000","last_undegraded":"2024-03-06T07:27:09.379184+0000","last_fullsized":"2024-03-06T07:27:09.379184+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:23.219552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.358988+0000","last_change":"2024-03-06T07:27:09.358988+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.358988+0000","last_undegraded":"2024-03-06T07:27:09.358988+0000","last_fullsized":"2024-03-06T07:27:09.358988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.381029+0000","last_change":"2024-03-06T07:27:09.380149+0000","last_active":"2024-03-06T07:27:09.381029+0000","last_peered":"2024-03-06T07:27:09.381029+0000","last_clean":"2024-03-06T07:27:09.381029+0000","last_became_active":"2024-03-06T07:27:09.379853+0000","last_became_peered":"2024-03-06T07:27:09.379853+0000","last_unstale":"2024-03-06T07:27:09.381029+0000","last_undegraded":"2024-03-06T07:27:09.381029+0000","last_fullsized":"2024-03-06T07:27:09.381029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:09.153513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364174+0000","last_change":"2024-03-06T07:27:09.364174+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364174+0000","last_undegraded":"2024-03-06T07:27:09.364174+0000","last_fullsized":"2024-03-06T07:27:09.364174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380994+0000","last_change":"2024-03-06T07:27:09.375900+0000","last_active":"2024-03-06T07:27:09.380994+0000","last_peered":"2024-03-06T07:27:09.380994+0000","last_clean":"2024-03-06T07:27:09.380994+0000","last_became_active":"2024-03-06T07:27:09.375583+0000","last_became_peered":"2024-03-06T07:27:09.375583+0000","last_unstale":"2024-03-06T07:27:09.380994+0000","last_undegraded":"2024-03-06T07:27:09.380994+0000","last_fullsized":"2024-03-06T07:27:09.380994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:53.571465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365127+0000","last_change":"2024-03-06T07:27:09.365127+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365127+0000","last_undegraded":"2024-03-06T07:27:09.365127+0000","last_fullsized":"2024-03-06T07:27:09.365127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.369385+0000","last_change":"2024-03-06T07:27:09.367702+0000","last_active":"2024-03-06T07:27:09.369385+0000","last_peered":"2024-03-06T07:27:09.369385+0000","last_clean":"2024-03-06T07:27:09.369385+0000","last_became_active":"2024-03-06T07:27:09.367456+0000","last_became_peered":"2024-03-06T07:27:09.367456+0000","last_unstale":"2024-03-06T07:27:09.369385+0000","last_undegraded":"2024-03-06T07:27:09.369385+0000","last_fullsized":"2024-03-06T07:27:09.369385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:17.715304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360340+0000","last_change":"2024-03-06T07:27:09.360340+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360340+0000","last_undegraded":"2024-03-06T07:27:09.360340+0000","last_fullsized":"2024-03-06T07:27:09.360340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371484+0000","last_change":"2024-03-06T07:27:09.367998+0000","last_active":"2024-03-06T07:27:09.371484+0000","last_peered":"2024-03-06T07:27:09.371484+0000","last_clean":"2024-03-06T07:27:09.371484+0000","last_became_active":"2024-03-06T07:27:09.367731+0000","last_became_peered":"2024-03-06T07:27:09.367731+0000","last_unstale":"2024-03-06T07:27:09.371484+0000","last_undegraded":"2024-03-06T07:27:09.371484+0000","last_fullsized":"2024-03-06T07:27:09.371484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:14.629790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365984+0000","last_change":"2024-03-06T07:27:09.365984+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365984+0000","last_undegraded":"2024-03-06T07:27:09.365984+0000","last_fullsized":"2024-03-06T07:27:09.365984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374853+0000","last_change":"2024-03-06T07:27:09.374468+0000","last_active":"2024-03-06T07:27:09.374853+0000","last_peered":"2024-03-06T07:27:09.374853+0000","last_clean":"2024-03-06T07:27:09.374853+0000","last_became_active":"2024-03-06T07:27:09.374183+0000","last_became_peered":"2024-03-06T07:27:09.374183+0000","last_unstale":"2024-03-06T07:27:09.374853+0000","last_undegraded":"2024-03-06T07:27:09.374853+0000","last_fullsized":"2024-03-06T07:27:09.374853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:49.597272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359152+0000","last_change":"2024-03-06T07:27:09.359152+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359152+0000","last_undegraded":"2024-03-06T07:27:09.359152+0000","last_fullsized":"2024-03-06T07:27:09.359152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.370888+0000","last_change":"2024-03-06T07:27:09.368796+0000","last_active":"2024-03-06T07:27:09.370888+0000","last_peered":"2024-03-06T07:27:09.370888+0000","last_clean":"2024-03-06T07:27:09.370888+0000","last_became_active":"2024-03-06T07:27:09.368017+0000","last_became_peered":"2024-03-06T07:27:09.368017+0000","last_unstale":"2024-03-06T07:27:09.370888+0000","last_undegraded":"2024-03-06T07:27:09.370888+0000","last_fullsized":"2024-03-06T07:27:09.370888+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:49.597272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.363467+0000","last_change":"2024-03-06T07:27:09.363467+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.363467+0000","last_undegraded":"2024-03-06T07:27:09.363467+0000","last_fullsized":"2024-03-06T07:27:09.363467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.373600+0000","last_change":"2024-03-06T07:27:09.373156+0000","last_active":"2024-03-06T07:27:09.373600+0000","last_peered":"2024-03-06T07:27:09.373600+0000","last_clean":"2024-03-06T07:27:09.373600+0000","last_became_active":"2024-03-06T07:27:09.372848+0000","last_became_peered":"2024-03-06T07:27:09.372848+0000","last_unstale":"2024-03-06T07:27:09.373600+0000","last_undegraded":"2024-03-06T07:27:09.373600+0000","last_fullsized":"2024-03-06T07:27:09.373600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:17.715304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.374217+0000","last_change":"2024-03-06T07:27:09.374217+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.374217+0000","last_undegraded":"2024-03-06T07:27:09.374217+0000","last_fullsized":"2024-03-06T07:27:09.374217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.375143+0000","last_change":"2024-03-06T07:27:09.371787+0000","last_active":"2024-03-06T07:27:09.375143+0000","last_peered":"2024-03-06T07:27:09.375143+0000","last_clean":"2024-03-06T07:27:09.375143+0000","last_became_active":"2024-03-06T07:27:09.371100+0000","last_became_peered":"2024-03-06T07:27:09.371100+0000","last_unstale":"2024-03-06T07:27:09.375143+0000","last_undegraded":"2024-03-06T07:27:09.375143+0000","last_fullsized":"2024-03-06T07:27:09.375143+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:03.753803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":288,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.361425+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:27:09.361425+0000","last_peered":"2024-03-06T07:27:09.361425+0000","last_clean":"2024-03-06T07:27:09.361425+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:27:09.361425+0000","last_undegraded":"2024-03-06T07:27:09.361425+0000","last_fullsized":"2024-03-06T07:27:09.361425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1014782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":418139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1014782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:15.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3f is not active+clean 2024-03-06T07:27:15.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.366331+0000', 'last_change': '2024-03-06T07:27:09.366331+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.366331+0000', 'last_undegraded': '2024-03-06T07:27:09.366331+0000', 'last_fullsized': '2024-03-06T07:27:09.366331+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3e is not active+clean 2024-03-06T07:27:15.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.365065+0000', 'last_change': '2024-03-06T07:27:09.365065+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.365065+0000', 'last_undegraded': '2024-03-06T07:27:09.365065+0000', 'last_fullsized': '2024-03-06T07:27:09.365065+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3d is not active+clean 2024-03-06T07:27:15.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.361750+0000', 'last_change': '2024-03-06T07:27:09.361750+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.361750+0000', 'last_undegraded': '2024-03-06T07:27:09.361750+0000', 'last_fullsized': '2024-03-06T07:27:09.361750+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3c is not active+clean 2024-03-06T07:27:15.231 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.368354+0000', 'last_change': '2024-03-06T07:27:09.368354+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.368354+0000', 'last_undegraded': '2024-03-06T07:27:09.368354+0000', 'last_fullsized': '2024-03-06T07:27:09.368354+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.231 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3b is not active+clean 2024-03-06T07:27:15.231 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.368139+0000', 'last_change': '2024-03-06T07:27:09.368139+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.368139+0000', 'last_undegraded': '2024-03-06T07:27:09.368139+0000', 'last_fullsized': '2024-03-06T07:27:09.368139+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3a is not active+clean 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.360754+0000', 'last_change': '2024-03-06T07:27:09.360754+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.360754+0000', 'last_undegraded': '2024-03-06T07:27:09.360754+0000', 'last_fullsized': '2024-03-06T07:27:09.360754+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.39 is not active+clean 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.366900+0000', 'last_change': '2024-03-06T07:27:09.366900+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.366900+0000', 'last_undegraded': '2024-03-06T07:27:09.366900+0000', 'last_fullsized': '2024-03-06T07:27:09.366900+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.38 is not active+clean 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.360184+0000', 'last_change': '2024-03-06T07:27:09.360184+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.360184+0000', 'last_undegraded': '2024-03-06T07:27:09.360184+0000', 'last_fullsized': '2024-03-06T07:27:09.360184+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.37 is not active+clean 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.360603+0000', 'last_change': '2024-03-06T07:27:09.360603+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.360603+0000', 'last_undegraded': '2024-03-06T07:27:09.360603+0000', 'last_fullsized': '2024-03-06T07:27:09.360603+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.36 is not active+clean 2024-03-06T07:27:15.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.364335+0000', 'last_change': '2024-03-06T07:27:09.364335+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.364335+0000', 'last_undegraded': '2024-03-06T07:27:09.364335+0000', 'last_fullsized': '2024-03-06T07:27:09.364335+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.35 is not active+clean 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.364708+0000', 'last_change': '2024-03-06T07:27:09.364708+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.364708+0000', 'last_undegraded': '2024-03-06T07:27:09.364708+0000', 'last_fullsized': '2024-03-06T07:27:09.364708+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.34 is not active+clean 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.366392+0000', 'last_change': '2024-03-06T07:27:09.366392+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.366392+0000', 'last_undegraded': '2024-03-06T07:27:09.366392+0000', 'last_fullsized': '2024-03-06T07:27:09.366392+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.33 is not active+clean 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.361471+0000', 'last_change': '2024-03-06T07:27:09.361471+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.361471+0000', 'last_undegraded': '2024-03-06T07:27:09.361471+0000', 'last_fullsized': '2024-03-06T07:27:09.361471+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.32 is not active+clean 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.368716+0000', 'last_change': '2024-03-06T07:27:09.368716+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.368716+0000', 'last_undegraded': '2024-03-06T07:27:09.368716+0000', 'last_fullsized': '2024-03-06T07:27:09.368716+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.31 is not active+clean 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.362610+0000', 'last_change': '2024-03-06T07:27:09.362610+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.362610+0000', 'last_undegraded': '2024-03-06T07:27:09.362610+0000', 'last_fullsized': '2024-03-06T07:27:09.362610+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.30 is not active+clean 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.365314+0000', 'last_change': '2024-03-06T07:27:09.365314+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.365314+0000', 'last_undegraded': '2024-03-06T07:27:09.365314+0000', 'last_fullsized': '2024-03-06T07:27:09.365314+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2f is not active+clean 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.364605+0000', 'last_change': '2024-03-06T07:27:09.364605+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.364605+0000', 'last_undegraded': '2024-03-06T07:27:09.364605+0000', 'last_fullsized': '2024-03-06T07:27:09.364605+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2e is not active+clean 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.366597+0000', 'last_change': '2024-03-06T07:27:09.366597+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.366597+0000', 'last_undegraded': '2024-03-06T07:27:09.366597+0000', 'last_fullsized': '2024-03-06T07:27:09.366597+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2d is not active+clean 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.367523+0000', 'last_change': '2024-03-06T07:27:09.367523+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.367523+0000', 'last_undegraded': '2024-03-06T07:27:09.367523+0000', 'last_fullsized': '2024-03-06T07:27:09.367523+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2c is not active+clean 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.367857+0000', 'last_change': '2024-03-06T07:27:09.367857+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.367857+0000', 'last_undegraded': '2024-03-06T07:27:09.367857+0000', 'last_fullsized': '2024-03-06T07:27:09.367857+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2b is not active+clean 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.366174+0000', 'last_change': '2024-03-06T07:27:09.366174+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.366174+0000', 'last_undegraded': '2024-03-06T07:27:09.366174+0000', 'last_fullsized': '2024-03-06T07:27:09.366174+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2a is not active+clean 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.369885+0000', 'last_change': '2024-03-06T07:27:09.369885+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.369885+0000', 'last_undegraded': '2024-03-06T07:27:09.369885+0000', 'last_fullsized': '2024-03-06T07:27:09.369885+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.29 is not active+clean 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.367067+0000', 'last_change': '2024-03-06T07:27:09.367067+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.367067+0000', 'last_undegraded': '2024-03-06T07:27:09.367067+0000', 'last_fullsized': '2024-03-06T07:27:09.367067+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.28 is not active+clean 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.364968+0000', 'last_change': '2024-03-06T07:27:09.364968+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.364968+0000', 'last_undegraded': '2024-03-06T07:27:09.364968+0000', 'last_fullsized': '2024-03-06T07:27:09.364968+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.27 is not active+clean 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.367295+0000', 'last_change': '2024-03-06T07:27:09.367295+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.367295+0000', 'last_undegraded': '2024-03-06T07:27:09.367295+0000', 'last_fullsized': '2024-03-06T07:27:09.367295+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.26 is not active+clean 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.360671+0000', 'last_change': '2024-03-06T07:27:09.360671+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.360671+0000', 'last_undegraded': '2024-03-06T07:27:09.360671+0000', 'last_fullsized': '2024-03-06T07:27:09.360671+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.25 is not active+clean 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.366535+0000', 'last_change': '2024-03-06T07:27:09.366535+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.366535+0000', 'last_undegraded': '2024-03-06T07:27:09.366535+0000', 'last_fullsized': '2024-03-06T07:27:09.366535+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.24 is not active+clean 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.365953+0000', 'last_change': '2024-03-06T07:27:09.365953+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.365953+0000', 'last_undegraded': '2024-03-06T07:27:09.365953+0000', 'last_fullsized': '2024-03-06T07:27:09.365953+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.23 is not active+clean 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.369681+0000', 'last_change': '2024-03-06T07:27:09.369681+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.369681+0000', 'last_undegraded': '2024-03-06T07:27:09.369681+0000', 'last_fullsized': '2024-03-06T07:27:09.369681+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.22 is not active+clean 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.369176+0000', 'last_change': '2024-03-06T07:27:09.369176+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.369176+0000', 'last_undegraded': '2024-03-06T07:27:09.369176+0000', 'last_fullsized': '2024-03-06T07:27:09.369176+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.21 is not active+clean 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.367187+0000', 'last_change': '2024-03-06T07:27:09.367187+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.367187+0000', 'last_undegraded': '2024-03-06T07:27:09.367187+0000', 'last_fullsized': '2024-03-06T07:27:09.367187+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.20 is not active+clean 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.363528+0000', 'last_change': '2024-03-06T07:27:09.363528+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.363528+0000', 'last_undegraded': '2024-03-06T07:27:09.363528+0000', 'last_fullsized': '2024-03-06T07:27:09.363528+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1e is not active+clean 2024-03-06T07:27:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.369136+0000', 'last_change': '2024-03-06T07:27:09.369136+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.369136+0000', 'last_undegraded': '2024-03-06T07:27:09.369136+0000', 'last_fullsized': '2024-03-06T07:27:09.369136+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1f is not active+clean 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.365340+0000', 'last_change': '2024-03-06T07:27:09.365340+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.365340+0000', 'last_undegraded': '2024-03-06T07:27:09.365340+0000', 'last_fullsized': '2024-03-06T07:27:09.365340+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1c is not active+clean 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.365542+0000', 'last_change': '2024-03-06T07:27:09.365542+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.365542+0000', 'last_undegraded': '2024-03-06T07:27:09.365542+0000', 'last_fullsized': '2024-03-06T07:27:09.365542+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1d is not active+clean 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.359572+0000', 'last_change': '2024-03-06T07:27:09.359572+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.359572+0000', 'last_undegraded': '2024-03-06T07:27:09.359572+0000', 'last_fullsized': '2024-03-06T07:27:09.359572+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1a is not active+clean 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.368708+0000', 'last_change': '2024-03-06T07:27:09.368708+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.368708+0000', 'last_undegraded': '2024-03-06T07:27:09.368708+0000', 'last_fullsized': '2024-03-06T07:27:09.368708+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1b is not active+clean 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.367797+0000', 'last_change': '2024-03-06T07:27:09.367797+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.367797+0000', 'last_undegraded': '2024-03-06T07:27:09.367797+0000', 'last_fullsized': '2024-03-06T07:27:09.367797+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.18 is not active+clean 2024-03-06T07:27:15.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.359735+0000', 'last_change': '2024-03-06T07:27:09.359735+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.359735+0000', 'last_undegraded': '2024-03-06T07:27:09.359735+0000', 'last_fullsized': '2024-03-06T07:27:09.359735+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.19 is not active+clean 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.361649+0000', 'last_change': '2024-03-06T07:27:09.361649+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.361649+0000', 'last_undegraded': '2024-03-06T07:27:09.361649+0000', 'last_fullsized': '2024-03-06T07:27:09.361649+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.16 is not active+clean 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.362615+0000', 'last_change': '2024-03-06T07:27:09.362615+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.362615+0000', 'last_undegraded': '2024-03-06T07:27:09.362615+0000', 'last_fullsized': '2024-03-06T07:27:09.362615+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.17 is not active+clean 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.360078+0000', 'last_change': '2024-03-06T07:27:09.360078+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.360078+0000', 'last_undegraded': '2024-03-06T07:27:09.360078+0000', 'last_fullsized': '2024-03-06T07:27:09.360078+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.14 is not active+clean 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.367153+0000', 'last_change': '2024-03-06T07:27:09.367153+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.367153+0000', 'last_undegraded': '2024-03-06T07:27:09.367153+0000', 'last_fullsized': '2024-03-06T07:27:09.367153+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.15 is not active+clean 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.366065+0000', 'last_change': '2024-03-06T07:27:09.366065+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.366065+0000', 'last_undegraded': '2024-03-06T07:27:09.366065+0000', 'last_fullsized': '2024-03-06T07:27:09.366065+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.12 is not active+clean 2024-03-06T07:27:15.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.366631+0000', 'last_change': '2024-03-06T07:27:09.366631+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.366631+0000', 'last_undegraded': '2024-03-06T07:27:09.366631+0000', 'last_fullsized': '2024-03-06T07:27:09.366631+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.13 is not active+clean 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.359370+0000', 'last_change': '2024-03-06T07:27:09.359370+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.359370+0000', 'last_undegraded': '2024-03-06T07:27:09.359370+0000', 'last_fullsized': '2024-03-06T07:27:09.359370+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.10 is not active+clean 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.365822+0000', 'last_change': '2024-03-06T07:27:09.365822+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.365822+0000', 'last_undegraded': '2024-03-06T07:27:09.365822+0000', 'last_fullsized': '2024-03-06T07:27:09.365822+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.11 is not active+clean 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.368019+0000', 'last_change': '2024-03-06T07:27:09.368019+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.368019+0000', 'last_undegraded': '2024-03-06T07:27:09.368019+0000', 'last_fullsized': '2024-03-06T07:27:09.368019+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.e is not active+clean 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.359006+0000', 'last_change': '2024-03-06T07:27:09.359006+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.359006+0000', 'last_undegraded': '2024-03-06T07:27:09.359006+0000', 'last_fullsized': '2024-03-06T07:27:09.359006+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.f is not active+clean 2024-03-06T07:27:15.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.364913+0000', 'last_change': '2024-03-06T07:27:09.364913+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.364913+0000', 'last_undegraded': '2024-03-06T07:27:09.364913+0000', 'last_fullsized': '2024-03-06T07:27:09.364913+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.d is not active+clean 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.365624+0000', 'last_change': '2024-03-06T07:27:09.365624+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.365624+0000', 'last_undegraded': '2024-03-06T07:27:09.365624+0000', 'last_fullsized': '2024-03-06T07:27:09.365624+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.a is not active+clean 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.368264+0000', 'last_change': '2024-03-06T07:27:09.368264+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.368264+0000', 'last_undegraded': '2024-03-06T07:27:09.368264+0000', 'last_fullsized': '2024-03-06T07:27:09.368264+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.b is not active+clean 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.359625+0000', 'last_change': '2024-03-06T07:27:09.359625+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.359625+0000', 'last_undegraded': '2024-03-06T07:27:09.359625+0000', 'last_fullsized': '2024-03-06T07:27:09.359625+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.8 is not active+clean 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.364417+0000', 'last_change': '2024-03-06T07:27:09.364417+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.364417+0000', 'last_undegraded': '2024-03-06T07:27:09.364417+0000', 'last_fullsized': '2024-03-06T07:27:09.364417+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.9 is not active+clean 2024-03-06T07:27:15.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.360242+0000', 'last_change': '2024-03-06T07:27:09.360242+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.360242+0000', 'last_undegraded': '2024-03-06T07:27:09.360242+0000', 'last_fullsized': '2024-03-06T07:27:09.360242+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.6 is not active+clean 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.368719+0000', 'last_change': '2024-03-06T07:27:09.368719+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.368719+0000', 'last_undegraded': '2024-03-06T07:27:09.368719+0000', 'last_fullsized': '2024-03-06T07:27:09.368719+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.7 is not active+clean 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.358988+0000', 'last_change': '2024-03-06T07:27:09.358988+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.358988+0000', 'last_undegraded': '2024-03-06T07:27:09.358988+0000', 'last_fullsized': '2024-03-06T07:27:09.358988+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.4 is not active+clean 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.364174+0000', 'last_change': '2024-03-06T07:27:09.364174+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.364174+0000', 'last_undegraded': '2024-03-06T07:27:09.364174+0000', 'last_fullsized': '2024-03-06T07:27:09.364174+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.5 is not active+clean 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.365127+0000', 'last_change': '2024-03-06T07:27:09.365127+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.365127+0000', 'last_undegraded': '2024-03-06T07:27:09.365127+0000', 'last_fullsized': '2024-03-06T07:27:09.365127+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2 is not active+clean 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.360340+0000', 'last_change': '2024-03-06T07:27:09.360340+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.360340+0000', 'last_undegraded': '2024-03-06T07:27:09.360340+0000', 'last_fullsized': '2024-03-06T07:27:09.360340+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3 is not active+clean 2024-03-06T07:27:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.365984+0000', 'last_change': '2024-03-06T07:27:09.365984+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.365984+0000', 'last_undegraded': '2024-03-06T07:27:09.365984+0000', 'last_fullsized': '2024-03-06T07:27:09.365984+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:15.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.0 is not active+clean 2024-03-06T07:27:15.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.359152+0000', 'last_change': '2024-03-06T07:27:09.359152+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.359152+0000', 'last_undegraded': '2024-03-06T07:27:09.359152+0000', 'last_fullsized': '2024-03-06T07:27:09.359152+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:15.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1 is not active+clean 2024-03-06T07:27:15.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.363467+0000', 'last_change': '2024-03-06T07:27:09.363467+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.363467+0000', 'last_undegraded': '2024-03-06T07:27:09.363467+0000', 'last_fullsized': '2024-03-06T07:27:09.363467+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:15.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.c is not active+clean 2024-03-06T07:27:15.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:09.374217+0000', 'last_change': '2024-03-06T07:27:09.374217+0000', 'last_active': '2024-03-06T07:27:09.352918+0000', 'last_peered': '2024-03-06T07:27:09.352918+0000', 'last_clean': '2024-03-06T07:27:09.352918+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:09.374217+0000', 'last_undegraded': '2024-03-06T07:27:09.374217+0000', 'last_fullsized': '2024-03-06T07:27:09.374217+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:09.352918+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:15.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:27:15.347 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:27:15.345+0000 7faaa49fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:15.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.349+0000 7f1855e9a640 1 Processor -- start 2024-03-06T07:27:15.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.349+0000 7f1855e9a640 1 -- start start 2024-03-06T07:27:15.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f1855e9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f18500b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f1855e9a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18500b2b50 0x7f18500b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f1855e9a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18500ce610 con 0x7f1850071540 2024-03-06T07:27:15.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f1855e9a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18500d2f70 con 0x7f18500b2b50 2024-03-06T07:27:15.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f1855e9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18500d4be0 con 0x7f18500d3400 2024-03-06T07:27:15.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f184effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f18500b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f184ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18500b2b50 0x7f18500b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f184effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f18500b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:37438/0 (socket says 172.21.15.27:37438) 2024-03-06T07:27:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f184effd640 1 -- 172.21.15.27:0/3542907691 learned_addr learned my addr 172.21.15.27:0/3542907691 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.350+0000 7f184ffff640 1 --2- 172.21.15.27:0/3542907691 >> v2:172.21.15.69:3300/0 conn(0x7f18500b2b50 0x7f18500b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:47738/0 (socket says 172.21.15.27:47738) 2024-03-06T07:27:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.351+0000 7f184effd640 1 -- 172.21.15.27:0/3542907691 >> v2:172.21.15.69:3300/0 conn(0x7f18500b2b50 msgr2=0x7f18500b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.351+0000 7f184effd640 1 --2- 172.21.15.27:0/3542907691 >> v2:172.21.15.69:3300/0 conn(0x7f18500b2b50 0x7f18500b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.351+0000 7f184effd640 1 -- 172.21.15.27:0/3542907691 >> v1:172.21.15.69:6789/0 conn(0x7f1850071540 legacy=0x7f1850071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.351+0000 7f184effd640 1 -- 172.21.15.27:0/3542907691 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18500d2920 con 0x7f18500d3400 2024-03-06T07:27:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.351+0000 7f184ffff640 1 --2- 172.21.15.27:0/3542907691 >> v2:172.21.15.69:3300/0 conn(0x7f18500b2b50 0x7f18500b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f184effd640 1 --2- 172.21.15.27:0/3542907691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f18500b30a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1840009870 tx=0x7f184002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7235e7b9192a63df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f184e7fc640 1 -- 172.21.15.27:0/3542907691 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840034070 con 0x7f18500d3400 2024-03-06T07:27:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f184e7fc640 1 -- 172.21.15.27:0/3542907691 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1840030840 con 0x7f18500d3400 2024-03-06T07:27:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f184e7fc640 1 -- 172.21.15.27:0/3542907691 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840030c10 con 0x7f18500d3400 2024-03-06T07:27:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f1855e9a640 1 -- 172.21.15.27:0/3542907691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 msgr2=0x7f18500b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f1855e9a640 1 --2- 172.21.15.27:0/3542907691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f18500b30a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1840009870 tx=0x7f184002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f1855e9a640 1 -- 172.21.15.27:0/3542907691 shutdown_connections 2024-03-06T07:27:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f1855e9a640 1 --2- 172.21.15.27:0/3542907691 >> v2:172.21.15.69:3300/0 conn(0x7f18500b2b50 0x7f18500b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f1855e9a640 1 --2- 172.21.15.27:0/3542907691 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f18500b30a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f1855e9a640 1 -- 172.21.15.27:0/3542907691 >> 172.21.15.27:0/3542907691 conn(0x7f18500d3ca0 msgr2=0x7f18500d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f1855e9a640 1 -- 172.21.15.27:0/3542907691 shutdown_connections 2024-03-06T07:27:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.352+0000 7f1855e9a640 1 -- 172.21.15.27:0/3542907691 wait complete. 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.354+0000 7f1855e9a640 1 Processor -- start 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.354+0000 7f1855e9a640 1 -- start start 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.354+0000 7f1855e9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1850071540 0x7f1850156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.354+0000 7f1855e9a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18500b2b50 0x7f18500b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.354+0000 7f1855e9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f1850158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.354+0000 7f1855e9a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185015e370 con 0x7f18500b2b50 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.354+0000 7f1855e9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18500ce8f0 con 0x7f1850071540 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.354+0000 7f1855e9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18500b6580 con 0x7f18500d3400 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18500b2b50 0x7f18500b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18500b2b50 0x7f18500b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34312/0 (socket says 172.21.15.27:34312) 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184effd640 1 -- 172.21.15.27:0/67079650 learned_addr learned my addr 172.21.15.27:0/67079650 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184f7fe640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1850071540 0x7f1850156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184ffff640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f1850158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184effd640 1 -- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 msgr2=0x7f1850158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184effd640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f1850158f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184effd640 1 -- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1850071540 msgr2=0x7f1850156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184effd640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1850071540 0x7f1850156f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184effd640 1 -- 172.21.15.27:0/67079650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18500b54e0 con 0x7f18500b2b50 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f184effd640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18500b2b50 0x7f18500b4ab0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f184002b9c0 tx=0x7f18400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f1847fff640 1 -- 172.21.15.27:0/67079650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840034040 con 0x7f18500b2b50 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f1847fff640 1 -- 172.21.15.27:0/67079650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184003ada0 con 0x7f18500b2b50 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18500b57c0 con 0x7f18500b2b50 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f1847fff640 1 -- 172.21.15.27:0/67079650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840044610 con 0x7f18500b2b50 2024-03-06T07:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.355+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18500b5d80 con 0x7f18500b2b50 2024-03-06T07:27:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.356+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1850068700 con 0x7f18500b2b50 2024-03-06T07:27:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.357+0000 7f1847fff640 1 -- 172.21.15.27:0/67079650 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f184004b060 con 0x7f18500b2b50 2024-03-06T07:27:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.357+0000 7f1847fff640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f183006b4d0 0x7f183006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.357+0000 7f1847fff640 1 -- 172.21.15.27:0/67079650 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18400ac130 con 0x7f18500b2b50 2024-03-06T07:27:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.358+0000 7f184f7fe640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f183006b4d0 0x7f183006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.358+0000 7f184f7fe640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f183006b4d0 0x7f183006d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f18380015a0 tx=0x7f1838009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.362+0000 7f1847fff640 1 -- 172.21.15.27:0/67079650 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f184003f070 con 0x7f18500b2b50 2024-03-06T07:27:15.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.614+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f18501576e0 con 0x7f18500b2b50 2024-03-06T07:27:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.614+0000 7f1847fff640 1 -- 172.21.15.27:0/67079650 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7f184007afc0 con 0x7f18500b2b50 2024-03-06T07:27:15.616 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:15.616 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":291,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":84,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709754,"num_in_osds":8,"osd_in_since":1709709745,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":264691712,"bytes_avail":772829421568,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":52,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"a","status":"up:active","gid":7032}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-06T07:26:02.032664+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:27:15.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f183006b4d0 msgr2=0x7f183006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f183006b4d0 0x7f183006d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f18380015a0 tx=0x7f1838009040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18500b2b50 msgr2=0x7f18500b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18500b2b50 0x7f18500b4ab0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f184002b9c0 tx=0x7f18400046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 shutdown_connections 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f183006b4d0 0x7f183006d990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18500d3400 0x7f1850158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18500b2b50 0x7f18500b4ab0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 --2- 172.21.15.27:0/67079650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1850071540 0x7f1850156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.616+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 >> 172.21.15.27:0/67079650 conn(0x7f18500d3ca0 msgr2=0x7f185015e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.617+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 shutdown_connections 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.617+0000 7f1855e9a640 1 -- 172.21.15.27:0/67079650 wait complete. 2024-03-06T07:27:15.629 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:27:15.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.734+0000 7f714a508640 1 Processor -- start 2024-03-06T07:27:15.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.734+0000 7f714a508640 1 -- start start 2024-03-06T07:27:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f714a508640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d02e0 0x7f71440cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f714a508640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71440b4260 0x7f71440b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f714a508640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f71440d06c0 0x7f71440b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f714a508640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71440b0b80 con 0x7f71440b4260 2024-03-06T07:27:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f714a508640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71440aeec0 con 0x7f71440d06c0 2024-03-06T07:27:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f714a508640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71440b0dd0 con 0x7f71440d02e0 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f71437fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71440b4260 0x7f71440b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f71437fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71440b4260 0x7f71440b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34324/0 (socket says 172.21.15.27:34324) 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f71437fe640 1 -- 172.21.15.27:0/1606809983 learned_addr learned my addr 172.21.15.27:0/1606809983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f71437fe640 1 -- 172.21.15.27:0/1606809983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d02e0 msgr2=0x7f71440cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f7143fff640 1 --2- 172.21.15.27:0/1606809983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d02e0 0x7f71440cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f7148aa5640 1 --2- 172.21.15.27:0/1606809983 >> v2:172.21.15.69:3300/0 conn(0x7f71440d06c0 0x7f71440b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f71437fe640 1 --2- 172.21.15.27:0/1606809983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d02e0 0x7f71440cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f71437fe640 1 -- 172.21.15.27:0/1606809983 >> v2:172.21.15.69:3300/0 conn(0x7f71440d06c0 msgr2=0x7f71440b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f71437fe640 1 --2- 172.21.15.27:0/1606809983 >> v2:172.21.15.69:3300/0 conn(0x7f71440d06c0 0x7f71440b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f71437fe640 1 -- 172.21.15.27:0/1606809983 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71440b10b0 con 0x7f71440b4260 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.736+0000 7f7148aa5640 1 --2- 172.21.15.27:0/1606809983 >> v2:172.21.15.69:3300/0 conn(0x7f71440d06c0 0x7f71440b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.737+0000 7f71437fe640 1 --2- 172.21.15.27:0/1606809983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 0x7f71440b1500 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f7144071d50 tx=0x7f713802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a294f19b36d9531 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.737+0000 7f7142ffd640 1 -- 172.21.15.27:0/1606809983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138034070 con 0x7f71440b4260 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.737+0000 7f7142ffd640 1 -- 172.21.15.27:0/1606809983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7138030860 con 0x7f71440b4260 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.737+0000 7f7142ffd640 1 -- 172.21.15.27:0/1606809983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138030c30 con 0x7f71440b4260 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.738+0000 7f714a508640 1 -- 172.21.15.27:0/1606809983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 msgr2=0x7f71440b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.738+0000 7f714a508640 1 --2- 172.21.15.27:0/1606809983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 0x7f71440b1500 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f7144071d50 tx=0x7f713802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.738+0000 7f714a508640 1 -- 172.21.15.27:0/1606809983 shutdown_connections 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.738+0000 7f714a508640 1 --2- 172.21.15.27:0/1606809983 >> v2:172.21.15.69:3300/0 conn(0x7f71440d06c0 0x7f71440b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.738+0000 7f714a508640 1 --2- 172.21.15.27:0/1606809983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 0x7f71440b1500 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.738+0000 7f714a508640 1 --2- 172.21.15.27:0/1606809983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d02e0 0x7f71440cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.738+0000 7f714a508640 1 -- 172.21.15.27:0/1606809983 >> 172.21.15.27:0/1606809983 conn(0x7f71440b36d0 msgr2=0x7f71440b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.738+0000 7f714a508640 1 -- 172.21.15.27:0/1606809983 shutdown_connections 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.738+0000 7f714a508640 1 -- 172.21.15.27:0/1606809983 wait complete. 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.740+0000 7f714a508640 1 Processor -- start 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.740+0000 7f714a508640 1 -- start start 2024-03-06T07:27:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f714a508640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 0x7f71441f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f714a508640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d02e0 0x7f71440cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f714a508640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d06c0 0x7f7144140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f714a508640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71440b2100 con 0x7f71440b4260 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f714a508640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71440b1f80 con 0x7f71440d02e0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f714a508640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71440b2280 con 0x7f71440d06c0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7143fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 0x7f71441f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7143fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 0x7f71441f16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34326/0 (socket says 172.21.15.27:34326) 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7143fff640 1 -- 172.21.15.27:0/858349685 learned_addr learned my addr 172.21.15.27:0/858349685 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f71437fe640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d02e0 0x7f71440cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7148aa5640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d06c0 0x7f7144140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7143fff640 1 -- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d06c0 msgr2=0x7f7144140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7143fff640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d06c0 0x7f7144140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7143fff640 1 -- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d02e0 msgr2=0x7f71440cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7143fff640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d02e0 0x7f71440cb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7143fff640 1 -- 172.21.15.27:0/858349685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144144070 con 0x7f71440b4260 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7148aa5640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d06c0 0x7f7144140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.742+0000 7f7143fff640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 0x7f71441f16a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f713400c820 tx=0x7f713400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.743+0000 7f7140ff9640 1 -- 172.21.15.27:0/858349685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134015070 con 0x7f71440b4260 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.743+0000 7f7140ff9640 1 -- 172.21.15.27:0/858349685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7134004220 con 0x7f71440b4260 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.743+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7144142fd0 con 0x7f71440b4260 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.743+0000 7f7140ff9640 1 -- 172.21.15.27:0/858349685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713400ce10 con 0x7f71440b4260 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.743+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7144143590 con 0x7f71440b4260 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.744+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7144068700 con 0x7f71440b4260 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.744+0000 7f7140ff9640 1 -- 172.21.15.27:0/858349685 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7134007830 con 0x7f71440b4260 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.744+0000 7f7140ff9640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f713006b4d0 0x7f713006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.745+0000 7f7140ff9640 1 -- 172.21.15.27:0/858349685 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7134089f20 con 0x7f71440b4260 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.745+0000 7f71437fe640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f713006b4d0 0x7f713006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.745+0000 7f71437fe640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f713006b4d0 0x7f713006d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7138036000 tx=0x7f713803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.749+0000 7f7140ff9640 1 -- 172.21.15.27:0/858349685 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7134058750 con 0x7f71440b4260 2024-03-06T07:27:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.928+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71440af380 con 0x7f713006b4d0 2024-03-06T07:27:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.934+0000 7f7140ff9640 1 -- 172.21.15.27:0/858349685 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246760 (secure 0 0 0) 0x7f71440af380 con 0x7f713006b4d0 2024-03-06T07:27:15.938 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:27:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.936+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f713006b4d0 msgr2=0x7f713006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.936+0000 7f714a508640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f713006b4d0 0x7f713006d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7138036000 tx=0x7f713803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.936+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 msgr2=0x7f71441f16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.936+0000 7f714a508640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 0x7f71441f16a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f713400c820 tx=0x7f713400cce0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.937+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 shutdown_connections 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.937+0000 7f714a508640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f713006b4d0 0x7f713006d990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.937+0000 7f714a508640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f71440d06c0 0x7f7144140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.937+0000 7f714a508640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71440d02e0 0x7f71440cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.937+0000 7f714a508640 1 --2- 172.21.15.27:0/858349685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71440b4260 0x7f71441f16a0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.937+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 >> 172.21.15.27:0/858349685 conn(0x7f71440b36d0 msgr2=0x7f71440ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.937+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 shutdown_connections 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:15.937+0000 7f714a508640 1 -- 172.21.15.27:0/858349685 wait complete. 2024-03-06T07:27:15.949 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-03-06T07:27:14.050694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258488,"kb_used_data":9020,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716232,"statfs":{"total":773094113280,"available":772829421568,"internally_reserved":0,"allocated":9236480,"data_stored":4569773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.694133"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366331+0000","last_change":"2024-03-06T07:27:09.366331+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366331+0000","last_undegraded":"2024-03-06T07:27:09.366331+0000","last_fullsized":"2024-03-06T07:27:09.366331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365065+0000","last_change":"2024-03-06T07:27:09.365065+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365065+0000","last_undegraded":"2024-03-06T07:27:09.365065+0000","last_fullsized":"2024-03-06T07:27:09.365065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361750+0000","last_change":"2024-03-06T07:27:09.361750+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361750+0000","last_undegraded":"2024-03-06T07:27:09.361750+0000","last_fullsized":"2024-03-06T07:27:09.361750+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368354+0000","last_change":"2024-03-06T07:27:09.368354+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368354+0000","last_undegraded":"2024-03-06T07:27:09.368354+0000","last_fullsized":"2024-03-06T07:27:09.368354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368139+0000","last_change":"2024-03-06T07:27:09.368139+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368139+0000","last_undegraded":"2024-03-06T07:27:09.368139+0000","last_fullsized":"2024-03-06T07:27:09.368139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360754+0000","last_change":"2024-03-06T07:27:09.360754+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360754+0000","last_undegraded":"2024-03-06T07:27:09.360754+0000","last_fullsized":"2024-03-06T07:27:09.360754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366900+0000","last_change":"2024-03-06T07:27:09.366900+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366900+0000","last_undegraded":"2024-03-06T07:27:09.366900+0000","last_fullsized":"2024-03-06T07:27:09.366900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360184+0000","last_change":"2024-03-06T07:27:09.360184+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360184+0000","last_undegraded":"2024-03-06T07:27:09.360184+0000","last_fullsized":"2024-03-06T07:27:09.360184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360603+0000","last_change":"2024-03-06T07:27:09.360603+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360603+0000","last_undegraded":"2024-03-06T07:27:09.360603+0000","last_fullsized":"2024-03-06T07:27:09.360603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364335+0000","last_change":"2024-03-06T07:27:09.364335+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364335+0000","last_undegraded":"2024-03-06T07:27:09.364335+0000","last_fullsized":"2024-03-06T07:27:09.364335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364708+0000","last_change":"2024-03-06T07:27:09.364708+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364708+0000","last_undegraded":"2024-03-06T07:27:09.364708+0000","last_fullsized":"2024-03-06T07:27:09.364708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366392+0000","last_change":"2024-03-06T07:27:09.366392+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366392+0000","last_undegraded":"2024-03-06T07:27:09.366392+0000","last_fullsized":"2024-03-06T07:27:09.366392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361471+0000","last_change":"2024-03-06T07:27:09.361471+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361471+0000","last_undegraded":"2024-03-06T07:27:09.361471+0000","last_fullsized":"2024-03-06T07:27:09.361471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368716+0000","last_change":"2024-03-06T07:27:09.368716+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368716+0000","last_undegraded":"2024-03-06T07:27:09.368716+0000","last_fullsized":"2024-03-06T07:27:09.368716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.362610+0000","last_change":"2024-03-06T07:27:09.362610+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.362610+0000","last_undegraded":"2024-03-06T07:27:09.362610+0000","last_fullsized":"2024-03-06T07:27:09.362610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365314+0000","last_change":"2024-03-06T07:27:09.365314+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365314+0000","last_undegraded":"2024-03-06T07:27:09.365314+0000","last_fullsized":"2024-03-06T07:27:09.365314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364605+0000","last_change":"2024-03-06T07:27:09.364605+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364605+0000","last_undegraded":"2024-03-06T07:27:09.364605+0000","last_fullsized":"2024-03-06T07:27:09.364605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366597+0000","last_change":"2024-03-06T07:27:09.366597+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366597+0000","last_undegraded":"2024-03-06T07:27:09.366597+0000","last_fullsized":"2024-03-06T07:27:09.366597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367523+0000","last_change":"2024-03-06T07:27:09.367523+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367523+0000","last_undegraded":"2024-03-06T07:27:09.367523+0000","last_fullsized":"2024-03-06T07:27:09.367523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367857+0000","last_change":"2024-03-06T07:27:09.367857+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367857+0000","last_undegraded":"2024-03-06T07:27:09.367857+0000","last_fullsized":"2024-03-06T07:27:09.367857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366174+0000","last_change":"2024-03-06T07:27:09.366174+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366174+0000","last_undegraded":"2024-03-06T07:27:09.366174+0000","last_fullsized":"2024-03-06T07:27:09.366174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369885+0000","last_change":"2024-03-06T07:27:09.369885+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369885+0000","last_undegraded":"2024-03-06T07:27:09.369885+0000","last_fullsized":"2024-03-06T07:27:09.369885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367067+0000","last_change":"2024-03-06T07:27:09.367067+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367067+0000","last_undegraded":"2024-03-06T07:27:09.367067+0000","last_fullsized":"2024-03-06T07:27:09.367067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364968+0000","last_change":"2024-03-06T07:27:09.364968+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364968+0000","last_undegraded":"2024-03-06T07:27:09.364968+0000","last_fullsized":"2024-03-06T07:27:09.364968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367295+0000","last_change":"2024-03-06T07:27:09.367295+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367295+0000","last_undegraded":"2024-03-06T07:27:09.367295+0000","last_fullsized":"2024-03-06T07:27:09.367295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360671+0000","last_change":"2024-03-06T07:27:09.360671+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360671+0000","last_undegraded":"2024-03-06T07:27:09.360671+0000","last_fullsized":"2024-03-06T07:27:09.360671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366535+0000","last_change":"2024-03-06T07:27:09.366535+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366535+0000","last_undegraded":"2024-03-06T07:27:09.366535+0000","last_fullsized":"2024-03-06T07:27:09.366535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365953+0000","last_change":"2024-03-06T07:27:09.365953+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365953+0000","last_undegraded":"2024-03-06T07:27:09.365953+0000","last_fullsized":"2024-03-06T07:27:09.365953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369681+0000","last_change":"2024-03-06T07:27:09.369681+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369681+0000","last_undegraded":"2024-03-06T07:27:09.369681+0000","last_fullsized":"2024-03-06T07:27:09.369681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369176+0000","last_change":"2024-03-06T07:27:09.369176+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369176+0000","last_undegraded":"2024-03-06T07:27:09.369176+0000","last_fullsized":"2024-03-06T07:27:09.369176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367187+0000","last_change":"2024-03-06T07:27:09.367187+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367187+0000","last_undegraded":"2024-03-06T07:27:09.367187+0000","last_fullsized":"2024-03-06T07:27:09.367187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.363528+0000","last_change":"2024-03-06T07:27:09.363528+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.363528+0000","last_undegraded":"2024-03-06T07:27:09.363528+0000","last_fullsized":"2024-03-06T07:27:09.363528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.369136+0000","last_change":"2024-03-06T07:27:09.369136+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.369136+0000","last_undegraded":"2024-03-06T07:27:09.369136+0000","last_fullsized":"2024-03-06T07:27:09.369136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383880+0000","last_change":"2024-03-06T07:27:09.378545+0000","last_active":"2024-03-06T07:27:09.383880+0000","last_peered":"2024-03-06T07:27:09.383880+0000","last_clean":"2024-03-06T07:27:09.383880+0000","last_became_active":"2024-03-06T07:27:09.378276+0000","last_became_peered":"2024-03-06T07:27:09.378276+0000","last_unstale":"2024-03-06T07:27:09.383880+0000","last_undegraded":"2024-03-06T07:27:09.383880+0000","last_fullsized":"2024-03-06T07:27:09.383880+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:09.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365340+0000","last_change":"2024-03-06T07:27:09.365340+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365340+0000","last_undegraded":"2024-03-06T07:27:09.365340+0000","last_fullsized":"2024-03-06T07:27:09.365340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.381029+0000","last_change":"2024-03-06T07:27:09.375904+0000","last_active":"2024-03-06T07:27:09.381029+0000","last_peered":"2024-03-06T07:27:09.381029+0000","last_clean":"2024-03-06T07:27:09.381029+0000","last_became_active":"2024-03-06T07:27:09.375566+0000","last_became_peered":"2024-03-06T07:27:09.375566+0000","last_unstale":"2024-03-06T07:27:09.381029+0000","last_undegraded":"2024-03-06T07:27:09.381029+0000","last_fullsized":"2024-03-06T07:27:09.381029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:07.197604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365542+0000","last_change":"2024-03-06T07:27:09.365542+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365542+0000","last_undegraded":"2024-03-06T07:27:09.365542+0000","last_fullsized":"2024-03-06T07:27:09.365542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374222+0000","last_change":"2024-03-06T07:27:09.371083+0000","last_active":"2024-03-06T07:27:09.374222+0000","last_peered":"2024-03-06T07:27:09.374222+0000","last_clean":"2024-03-06T07:27:09.374222+0000","last_became_active":"2024-03-06T07:27:09.370820+0000","last_became_peered":"2024-03-06T07:27:09.370820+0000","last_unstale":"2024-03-06T07:27:09.374222+0000","last_undegraded":"2024-03-06T07:27:09.374222+0000","last_fullsized":"2024-03-06T07:27:09.374222+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:29.304676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359572+0000","last_change":"2024-03-06T07:27:09.359572+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359572+0000","last_undegraded":"2024-03-06T07:27:09.359572+0000","last_fullsized":"2024-03-06T07:27:09.359572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.378612+0000","last_change":"2024-03-06T07:27:09.374749+0000","last_active":"2024-03-06T07:27:09.378612+0000","last_peered":"2024-03-06T07:27:09.378612+0000","last_clean":"2024-03-06T07:27:09.378612+0000","last_became_active":"2024-03-06T07:27:09.374421+0000","last_became_peered":"2024-03-06T07:27:09.374421+0000","last_unstale":"2024-03-06T07:27:09.378612+0000","last_undegraded":"2024-03-06T07:27:09.378612+0000","last_fullsized":"2024-03-06T07:27:09.378612+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:36.751221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368708+0000","last_change":"2024-03-06T07:27:09.368708+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368708+0000","last_undegraded":"2024-03-06T07:27:09.368708+0000","last_fullsized":"2024-03-06T07:27:09.368708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383926+0000","last_change":"2024-03-06T07:27:09.379796+0000","last_active":"2024-03-06T07:27:09.383926+0000","last_peered":"2024-03-06T07:27:09.383926+0000","last_clean":"2024-03-06T07:27:09.383926+0000","last_became_active":"2024-03-06T07:27:09.379609+0000","last_became_peered":"2024-03-06T07:27:09.379609+0000","last_unstale":"2024-03-06T07:27:09.383926+0000","last_undegraded":"2024-03-06T07:27:09.383926+0000","last_fullsized":"2024-03-06T07:27:09.383926+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:53.571465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367797+0000","last_change":"2024-03-06T07:27:09.367797+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367797+0000","last_undegraded":"2024-03-06T07:27:09.367797+0000","last_fullsized":"2024-03-06T07:27:09.367797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380992+0000","last_change":"2024-03-06T07:27:09.375952+0000","last_active":"2024-03-06T07:27:09.380992+0000","last_peered":"2024-03-06T07:27:09.380992+0000","last_clean":"2024-03-06T07:27:09.380992+0000","last_became_active":"2024-03-06T07:27:09.375584+0000","last_became_peered":"2024-03-06T07:27:09.375584+0000","last_unstale":"2024-03-06T07:27:09.380992+0000","last_undegraded":"2024-03-06T07:27:09.380992+0000","last_fullsized":"2024-03-06T07:27:09.380992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:03.362736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359735+0000","last_change":"2024-03-06T07:27:09.359735+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359735+0000","last_undegraded":"2024-03-06T07:27:09.359735+0000","last_fullsized":"2024-03-06T07:27:09.359735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.370532+0000","last_change":"2024-03-06T07:27:09.366681+0000","last_active":"2024-03-06T07:27:09.370532+0000","last_peered":"2024-03-06T07:27:09.370532+0000","last_clean":"2024-03-06T07:27:09.370532+0000","last_became_active":"2024-03-06T07:27:09.366329+0000","last_became_peered":"2024-03-06T07:27:09.366329+0000","last_unstale":"2024-03-06T07:27:09.370532+0000","last_undegraded":"2024-03-06T07:27:09.370532+0000","last_fullsized":"2024-03-06T07:27:09.370532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:07.928354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.361649+0000","last_change":"2024-03-06T07:27:09.361649+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.361649+0000","last_undegraded":"2024-03-06T07:27:09.361649+0000","last_fullsized":"2024-03-06T07:27:09.361649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380679+0000","last_change":"2024-03-06T07:27:09.378990+0000","last_active":"2024-03-06T07:27:09.380679+0000","last_peered":"2024-03-06T07:27:09.380679+0000","last_clean":"2024-03-06T07:27:09.380679+0000","last_became_active":"2024-03-06T07:27:09.378578+0000","last_became_peered":"2024-03-06T07:27:09.378578+0000","last_unstale":"2024-03-06T07:27:09.380679+0000","last_undegraded":"2024-03-06T07:27:09.380679+0000","last_fullsized":"2024-03-06T07:27:09.380679+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:41.025662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.362615+0000","last_change":"2024-03-06T07:27:09.362615+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.362615+0000","last_undegraded":"2024-03-06T07:27:09.362615+0000","last_fullsized":"2024-03-06T07:27:09.362615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.372532+0000","last_change":"2024-03-06T07:27:09.369411+0000","last_active":"2024-03-06T07:27:09.372532+0000","last_peered":"2024-03-06T07:27:09.372532+0000","last_clean":"2024-03-06T07:27:09.372532+0000","last_became_active":"2024-03-06T07:27:09.369100+0000","last_became_peered":"2024-03-06T07:27:09.369100+0000","last_unstale":"2024-03-06T07:27:09.372532+0000","last_undegraded":"2024-03-06T07:27:09.372532+0000","last_fullsized":"2024-03-06T07:27:09.372532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:07.197604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360078+0000","last_change":"2024-03-06T07:27:09.360078+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360078+0000","last_undegraded":"2024-03-06T07:27:09.360078+0000","last_fullsized":"2024-03-06T07:27:09.360078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.375998+0000","last_change":"2024-03-06T07:27:09.372377+0000","last_active":"2024-03-06T07:27:09.375998+0000","last_peered":"2024-03-06T07:27:09.375998+0000","last_clean":"2024-03-06T07:27:09.375998+0000","last_became_active":"2024-03-06T07:27:09.371991+0000","last_became_peered":"2024-03-06T07:27:09.371991+0000","last_unstale":"2024-03-06T07:27:09.375998+0000","last_undegraded":"2024-03-06T07:27:09.375998+0000","last_fullsized":"2024-03-06T07:27:09.375998+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:03.362736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.367153+0000","last_change":"2024-03-06T07:27:09.367153+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.367153+0000","last_undegraded":"2024-03-06T07:27:09.367153+0000","last_fullsized":"2024-03-06T07:27:09.367153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.383917+0000","last_change":"2024-03-06T07:27:09.378597+0000","last_active":"2024-03-06T07:27:09.383917+0000","last_peered":"2024-03-06T07:27:09.383917+0000","last_clean":"2024-03-06T07:27:09.383917+0000","last_became_active":"2024-03-06T07:27:09.378334+0000","last_became_peered":"2024-03-06T07:27:09.378334+0000","last_unstale":"2024-03-06T07:27:09.383917+0000","last_undegraded":"2024-03-06T07:27:09.383917+0000","last_fullsized":"2024-03-06T07:27:09.383917+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:20.907611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366065+0000","last_change":"2024-03-06T07:27:09.366065+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366065+0000","last_undegraded":"2024-03-06T07:27:09.366065+0000","last_fullsized":"2024-03-06T07:27:09.366065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.368379+0000","last_change":"2024-03-06T07:27:09.365562+0000","last_active":"2024-03-06T07:27:09.368379+0000","last_peered":"2024-03-06T07:27:09.368379+0000","last_clean":"2024-03-06T07:27:09.368379+0000","last_became_active":"2024-03-06T07:27:09.365279+0000","last_became_peered":"2024-03-06T07:27:09.365279+0000","last_unstale":"2024-03-06T07:27:09.368379+0000","last_undegraded":"2024-03-06T07:27:09.368379+0000","last_fullsized":"2024-03-06T07:27:09.368379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:35.367786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.366631+0000","last_change":"2024-03-06T07:27:09.366631+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.366631+0000","last_undegraded":"2024-03-06T07:27:09.366631+0000","last_fullsized":"2024-03-06T07:27:09.366631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.379502+0000","last_change":"2024-03-06T07:27:09.378045+0000","last_active":"2024-03-06T07:27:09.379502+0000","last_peered":"2024-03-06T07:27:09.379502+0000","last_clean":"2024-03-06T07:27:09.379502+0000","last_became_active":"2024-03-06T07:27:09.377782+0000","last_became_peered":"2024-03-06T07:27:09.377782+0000","last_unstale":"2024-03-06T07:27:09.379502+0000","last_undegraded":"2024-03-06T07:27:09.379502+0000","last_fullsized":"2024-03-06T07:27:09.379502+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:47.722593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359370+0000","last_change":"2024-03-06T07:27:09.359370+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359370+0000","last_undegraded":"2024-03-06T07:27:09.359370+0000","last_fullsized":"2024-03-06T07:27:09.359370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380600+0000","last_change":"2024-03-06T07:27:09.378309+0000","last_active":"2024-03-06T07:27:09.380600+0000","last_peered":"2024-03-06T07:27:09.380600+0000","last_clean":"2024-03-06T07:27:09.380600+0000","last_became_active":"2024-03-06T07:27:09.378012+0000","last_became_peered":"2024-03-06T07:27:09.378012+0000","last_unstale":"2024-03-06T07:27:09.380600+0000","last_undegraded":"2024-03-06T07:27:09.380600+0000","last_fullsized":"2024-03-06T07:27:09.380600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:49.071313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365822+0000","last_change":"2024-03-06T07:27:09.365822+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365822+0000","last_undegraded":"2024-03-06T07:27:09.365822+0000","last_fullsized":"2024-03-06T07:27:09.365822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.372446+0000","last_change":"2024-03-06T07:27:09.370907+0000","last_active":"2024-03-06T07:27:09.372446+0000","last_peered":"2024-03-06T07:27:09.372446+0000","last_clean":"2024-03-06T07:27:09.372446+0000","last_became_active":"2024-03-06T07:27:09.370655+0000","last_became_peered":"2024-03-06T07:27:09.370655+0000","last_unstale":"2024-03-06T07:27:09.372446+0000","last_undegraded":"2024-03-06T07:27:09.372446+0000","last_fullsized":"2024-03-06T07:27:09.372446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:07.928354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368019+0000","last_change":"2024-03-06T07:27:09.368019+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368019+0000","last_undegraded":"2024-03-06T07:27:09.368019+0000","last_fullsized":"2024-03-06T07:27:09.368019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371431+0000","last_change":"2024-03-06T07:27:09.368550+0000","last_active":"2024-03-06T07:27:09.371431+0000","last_peered":"2024-03-06T07:27:09.371431+0000","last_clean":"2024-03-06T07:27:09.371431+0000","last_became_active":"2024-03-06T07:27:09.367237+0000","last_became_peered":"2024-03-06T07:27:09.367237+0000","last_unstale":"2024-03-06T07:27:09.371431+0000","last_undegraded":"2024-03-06T07:27:09.371431+0000","last_fullsized":"2024-03-06T07:27:09.371431+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:43.340074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359006+0000","last_change":"2024-03-06T07:27:09.359006+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359006+0000","last_undegraded":"2024-03-06T07:27:09.359006+0000","last_fullsized":"2024-03-06T07:27:09.359006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.369618+0000","last_change":"2024-03-06T07:27:09.367473+0000","last_active":"2024-03-06T07:27:09.369618+0000","last_peered":"2024-03-06T07:27:09.369618+0000","last_clean":"2024-03-06T07:27:09.369618+0000","last_became_active":"2024-03-06T07:27:09.367201+0000","last_became_peered":"2024-03-06T07:27:09.367201+0000","last_unstale":"2024-03-06T07:27:09.369618+0000","last_undegraded":"2024-03-06T07:27:09.369618+0000","last_fullsized":"2024-03-06T07:27:09.369618+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:26.071654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364913+0000","last_change":"2024-03-06T07:27:09.364913+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364913+0000","last_undegraded":"2024-03-06T07:27:09.364913+0000","last_fullsized":"2024-03-06T07:27:09.364913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374806+0000","last_change":"2024-03-06T07:27:09.373607+0000","last_active":"2024-03-06T07:27:09.374806+0000","last_peered":"2024-03-06T07:27:09.374806+0000","last_clean":"2024-03-06T07:27:09.374806+0000","last_became_active":"2024-03-06T07:27:09.373306+0000","last_became_peered":"2024-03-06T07:27:09.373306+0000","last_unstale":"2024-03-06T07:27:09.374806+0000","last_undegraded":"2024-03-06T07:27:09.374806+0000","last_fullsized":"2024-03-06T07:27:09.374806+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:32.055587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365624+0000","last_change":"2024-03-06T07:27:09.365624+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365624+0000","last_undegraded":"2024-03-06T07:27:09.365624+0000","last_fullsized":"2024-03-06T07:27:09.365624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371513+0000","last_change":"2024-03-06T07:27:09.367584+0000","last_active":"2024-03-06T07:27:09.371513+0000","last_peered":"2024-03-06T07:27:09.371513+0000","last_clean":"2024-03-06T07:27:09.371513+0000","last_became_active":"2024-03-06T07:27:09.366434+0000","last_became_peered":"2024-03-06T07:27:09.366434+0000","last_unstale":"2024-03-06T07:27:09.371513+0000","last_undegraded":"2024-03-06T07:27:09.371513+0000","last_fullsized":"2024-03-06T07:27:09.371513+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:44.994841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368264+0000","last_change":"2024-03-06T07:27:09.368264+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368264+0000","last_undegraded":"2024-03-06T07:27:09.368264+0000","last_fullsized":"2024-03-06T07:27:09.368264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374468+0000","last_change":"2024-03-06T07:27:09.373174+0000","last_active":"2024-03-06T07:27:09.374468+0000","last_peered":"2024-03-06T07:27:09.374468+0000","last_clean":"2024-03-06T07:27:09.374468+0000","last_became_active":"2024-03-06T07:27:09.372894+0000","last_became_peered":"2024-03-06T07:27:09.372894+0000","last_unstale":"2024-03-06T07:27:09.374468+0000","last_undegraded":"2024-03-06T07:27:09.374468+0000","last_fullsized":"2024-03-06T07:27:09.374468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:34.062787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359625+0000","last_change":"2024-03-06T07:27:09.359625+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359625+0000","last_undegraded":"2024-03-06T07:27:09.359625+0000","last_fullsized":"2024-03-06T07:27:09.359625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.377291+0000","last_change":"2024-03-06T07:27:09.374733+0000","last_active":"2024-03-06T07:27:09.377291+0000","last_peered":"2024-03-06T07:27:09.377291+0000","last_clean":"2024-03-06T07:27:09.377291+0000","last_became_active":"2024-03-06T07:27:09.374380+0000","last_became_peered":"2024-03-06T07:27:09.374380+0000","last_unstale":"2024-03-06T07:27:09.377291+0000","last_undegraded":"2024-03-06T07:27:09.377291+0000","last_fullsized":"2024-03-06T07:27:09.377291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:10.906485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364417+0000","last_change":"2024-03-06T07:27:09.364417+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364417+0000","last_undegraded":"2024-03-06T07:27:09.364417+0000","last_fullsized":"2024-03-06T07:27:09.364417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.377859+0000","last_change":"2024-03-06T07:27:09.375768+0000","last_active":"2024-03-06T07:27:09.377859+0000","last_peered":"2024-03-06T07:27:09.377859+0000","last_clean":"2024-03-06T07:27:09.377859+0000","last_became_active":"2024-03-06T07:27:09.375512+0000","last_became_peered":"2024-03-06T07:27:09.375512+0000","last_unstale":"2024-03-06T07:27:09.377859+0000","last_undegraded":"2024-03-06T07:27:09.377859+0000","last_fullsized":"2024-03-06T07:27:09.377859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:25.888556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360242+0000","last_change":"2024-03-06T07:27:09.360242+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360242+0000","last_undegraded":"2024-03-06T07:27:09.360242+0000","last_fullsized":"2024-03-06T07:27:09.360242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371369+0000","last_change":"2024-03-06T07:27:09.369058+0000","last_active":"2024-03-06T07:27:09.371369+0000","last_peered":"2024-03-06T07:27:09.371369+0000","last_clean":"2024-03-06T07:27:09.371369+0000","last_became_active":"2024-03-06T07:27:09.368762+0000","last_became_peered":"2024-03-06T07:27:09.368762+0000","last_unstale":"2024-03-06T07:27:09.371369+0000","last_undegraded":"2024-03-06T07:27:09.371369+0000","last_fullsized":"2024-03-06T07:27:09.371369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:20.708389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.368719+0000","last_change":"2024-03-06T07:27:09.368719+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.368719+0000","last_undegraded":"2024-03-06T07:27:09.368719+0000","last_fullsized":"2024-03-06T07:27:09.368719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.379184+0000","last_change":"2024-03-06T07:27:09.377310+0000","last_active":"2024-03-06T07:27:09.379184+0000","last_peered":"2024-03-06T07:27:09.379184+0000","last_clean":"2024-03-06T07:27:09.379184+0000","last_became_active":"2024-03-06T07:27:09.377048+0000","last_became_peered":"2024-03-06T07:27:09.377048+0000","last_unstale":"2024-03-06T07:27:09.379184+0000","last_undegraded":"2024-03-06T07:27:09.379184+0000","last_fullsized":"2024-03-06T07:27:09.379184+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:23.219552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.358988+0000","last_change":"2024-03-06T07:27:09.358988+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.358988+0000","last_undegraded":"2024-03-06T07:27:09.358988+0000","last_fullsized":"2024-03-06T07:27:09.358988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.381029+0000","last_change":"2024-03-06T07:27:09.380149+0000","last_active":"2024-03-06T07:27:09.381029+0000","last_peered":"2024-03-06T07:27:09.381029+0000","last_clean":"2024-03-06T07:27:09.381029+0000","last_became_active":"2024-03-06T07:27:09.379853+0000","last_became_peered":"2024-03-06T07:27:09.379853+0000","last_unstale":"2024-03-06T07:27:09.381029+0000","last_undegraded":"2024-03-06T07:27:09.381029+0000","last_fullsized":"2024-03-06T07:27:09.381029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:09.153513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.364174+0000","last_change":"2024-03-06T07:27:09.364174+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.364174+0000","last_undegraded":"2024-03-06T07:27:09.364174+0000","last_fullsized":"2024-03-06T07:27:09.364174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.380994+0000","last_change":"2024-03-06T07:27:09.375900+0000","last_active":"2024-03-06T07:27:09.380994+0000","last_peered":"2024-03-06T07:27:09.380994+0000","last_clean":"2024-03-06T07:27:09.380994+0000","last_became_active":"2024-03-06T07:27:09.375583+0000","last_became_peered":"2024-03-06T07:27:09.375583+0000","last_unstale":"2024-03-06T07:27:09.380994+0000","last_undegraded":"2024-03-06T07:27:09.380994+0000","last_fullsized":"2024-03-06T07:27:09.380994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:53.571465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365127+0000","last_change":"2024-03-06T07:27:09.365127+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365127+0000","last_undegraded":"2024-03-06T07:27:09.365127+0000","last_fullsized":"2024-03-06T07:27:09.365127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.369385+0000","last_change":"2024-03-06T07:27:09.367702+0000","last_active":"2024-03-06T07:27:09.369385+0000","last_peered":"2024-03-06T07:27:09.369385+0000","last_clean":"2024-03-06T07:27:09.369385+0000","last_became_active":"2024-03-06T07:27:09.367456+0000","last_became_peered":"2024-03-06T07:27:09.367456+0000","last_unstale":"2024-03-06T07:27:09.369385+0000","last_undegraded":"2024-03-06T07:27:09.369385+0000","last_fullsized":"2024-03-06T07:27:09.369385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:17.715304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.360340+0000","last_change":"2024-03-06T07:27:09.360340+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.360340+0000","last_undegraded":"2024-03-06T07:27:09.360340+0000","last_fullsized":"2024-03-06T07:27:09.360340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.371484+0000","last_change":"2024-03-06T07:27:09.367998+0000","last_active":"2024-03-06T07:27:09.371484+0000","last_peered":"2024-03-06T07:27:09.371484+0000","last_clean":"2024-03-06T07:27:09.371484+0000","last_became_active":"2024-03-06T07:27:09.367731+0000","last_became_peered":"2024-03-06T07:27:09.367731+0000","last_unstale":"2024-03-06T07:27:09.371484+0000","last_undegraded":"2024-03-06T07:27:09.371484+0000","last_fullsized":"2024-03-06T07:27:09.371484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:14.629790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.365984+0000","last_change":"2024-03-06T07:27:09.365984+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.365984+0000","last_undegraded":"2024-03-06T07:27:09.365984+0000","last_fullsized":"2024-03-06T07:27:09.365984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.374853+0000","last_change":"2024-03-06T07:27:09.374468+0000","last_active":"2024-03-06T07:27:09.374853+0000","last_peered":"2024-03-06T07:27:09.374853+0000","last_clean":"2024-03-06T07:27:09.374853+0000","last_became_active":"2024-03-06T07:27:09.374183+0000","last_became_peered":"2024-03-06T07:27:09.374183+0000","last_unstale":"2024-03-06T07:27:09.374853+0000","last_undegraded":"2024-03-06T07:27:09.374853+0000","last_fullsized":"2024-03-06T07:27:09.374853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:49.597272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.359152+0000","last_change":"2024-03-06T07:27:09.359152+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.359152+0000","last_undegraded":"2024-03-06T07:27:09.359152+0000","last_fullsized":"2024-03-06T07:27:09.359152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.370888+0000","last_change":"2024-03-06T07:27:09.368796+0000","last_active":"2024-03-06T07:27:09.370888+0000","last_peered":"2024-03-06T07:27:09.370888+0000","last_clean":"2024-03-06T07:27:09.370888+0000","last_became_active":"2024-03-06T07:27:09.368017+0000","last_became_peered":"2024-03-06T07:27:09.368017+0000","last_unstale":"2024-03-06T07:27:09.370888+0000","last_undegraded":"2024-03-06T07:27:09.370888+0000","last_fullsized":"2024-03-06T07:27:09.370888+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:49.597272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.363467+0000","last_change":"2024-03-06T07:27:09.363467+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.363467+0000","last_undegraded":"2024-03-06T07:27:09.363467+0000","last_fullsized":"2024-03-06T07:27:09.363467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.373600+0000","last_change":"2024-03-06T07:27:09.373156+0000","last_active":"2024-03-06T07:27:09.373600+0000","last_peered":"2024-03-06T07:27:09.373600+0000","last_clean":"2024-03-06T07:27:09.373600+0000","last_became_active":"2024-03-06T07:27:09.372848+0000","last_became_peered":"2024-03-06T07:27:09.372848+0000","last_unstale":"2024-03-06T07:27:09.373600+0000","last_undegraded":"2024-03-06T07:27:09.373600+0000","last_fullsized":"2024-03-06T07:27:09.373600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:17.715304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-03-06T07:27:09.374217+0000","last_change":"2024-03-06T07:27:09.374217+0000","last_active":"2024-03-06T07:27:09.352918+0000","last_peered":"2024-03-06T07:27:09.352918+0000","last_clean":"2024-03-06T07:27:09.352918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:09.374217+0000","last_undegraded":"2024-03-06T07:27:09.374217+0000","last_fullsized":"2024-03-06T07:27:09.374217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.375143+0000","last_change":"2024-03-06T07:27:09.371787+0000","last_active":"2024-03-06T07:27:09.375143+0000","last_peered":"2024-03-06T07:27:09.375143+0000","last_clean":"2024-03-06T07:27:09.375143+0000","last_became_active":"2024-03-06T07:27:09.371100+0000","last_became_peered":"2024-03-06T07:27:09.371100+0000","last_unstale":"2024-03-06T07:27:09.375143+0000","last_undegraded":"2024-03-06T07:27:09.375143+0000","last_fullsized":"2024-03-06T07:27:09.375143+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:03.753803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":288,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-06T07:27:09.361425+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:27:09.361425+0000","last_peered":"2024-03-06T07:27:09.361425+0000","last_clean":"2024-03-06T07:27:09.361425+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:27:09.361425+0000","last_undegraded":"2024-03-06T07:27:09.361425+0000","last_fullsized":"2024-03-06T07:27:09.361425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1014782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":418139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":424414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1014782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:18.267 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:18.270 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:18.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:19.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.058+0000 7f08afa39640 1 Processor -- start 2024-03-06T07:27:19.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.058+0000 7f08afa39640 1 -- start start 2024-03-06T07:27:19.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.059+0000 7f08afa39640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08a80b9a10 0x7f08a80d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.059+0000 7f08afa39640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a80cea30 con 0x7f08a815df00 2024-03-06T07:27:19.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.059+0000 7f08afa39640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a80cf3c0 con 0x7f08a80b95a0 2024-03-06T07:27:19.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.059+0000 7f08afa39640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a80cf0e0 con 0x7f08a80b9a10 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.059+0000 7f08ad7d5640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f08a815df00 0x7f08a80d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43314/0 (socket says 172.21.15.27:43314) 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.059+0000 7f08ad7d5640 1 -- 172.21.15.27:0/1493268601 learned_addr learned my addr 172.21.15.27:0/1493268601 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.059+0000 7f08adfd6640 1 --2- 172.21.15.27:0/1493268601 >> v2:172.21.15.69:3300/0 conn(0x7f08a80b9a10 0x7f08a80d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.059+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007233848 0 0) 0x7f08a80cea30 con 0x7f08a815df00 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.059+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0894003520 con 0x7f08a815df00 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3640488199 0 0) 0x7f0894003520 con 0x7f08a815df00 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 >> v2:172.21.15.69:3300/0 conn(0x7f08a80b9a10 msgr2=0x7f08a80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 --2- 172.21.15.27:0/1493268601 >> v2:172.21.15.69:3300/0 conn(0x7f08a80b9a10 0x7f08a80d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 >> v1:172.21.15.69:6789/0 conn(0x7f08a80b95a0 legacy=0x7f08a80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a80ce470 con 0x7f08a815df00 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f08a0001670 con 0x7f08a815df00 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 >> v1:172.21.15.27:6789/0 conn(0x7f08a815df00 legacy=0x7f08a80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08940039b0 0x7f0894003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f0894004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08940051c0 0x7f089400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80cf0e0 con 0x7f0894004800 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089400c090 con 0x7f08940039b0 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089400c290 con 0x7f08940051c0 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f08acfd4640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f0894004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f08adfd6640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08940051c0 0x7f089400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f08ad7d5640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08940039b0 0x7f0894003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f08acfd4640 1 -- 172.21.15.27:0/1493268601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08940051c0 msgr2=0x7f089400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f08acfd4640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08940051c0 0x7f089400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f08acfd4640 1 -- 172.21.15.27:0/1493268601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08940039b0 msgr2=0x7f0894003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f08acfd4640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08940039b0 0x7f0894003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f08adfd6640 1 -- 172.21.15.27:0/1493268601 reap_dead start 2024-03-06T07:27:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.060+0000 7f08acfd4640 1 -- 172.21.15.27:0/1493268601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0894003520 con 0x7f0894004800 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f08acfd4640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f0894004c80 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f0898001750 tx=0x7f0894005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cd0543d4f81c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0898001370 con 0x7f0894004800 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08980052d0 con 0x7f0894004800 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f089ffff640 1 -- 172.21.15.27:0/1493268601 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0898003830 con 0x7f0894004800 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f08afa39640 1 -- 172.21.15.27:0/1493268601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 msgr2=0x7f0894004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f08afa39640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f0894004c80 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f0898001750 tx=0x7f0894005830 comp rx=0 tx=0).stop 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f08afa39640 1 -- 172.21.15.27:0/1493268601 shutdown_connections 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f08afa39640 1 --2- 172.21.15.27:0/1493268601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f0894004c80 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f08afa39640 1 -- 172.21.15.27:0/1493268601 >> 172.21.15.27:0/1493268601 conn(0x7f08a8071960 msgr2=0x7f08a8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.062+0000 7f08afa39640 1 -- 172.21.15.27:0/1493268601 shutdown_connections 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.063+0000 7f08afa39640 1 -- 172.21.15.27:0/1493268601 wait complete. 2024-03-06T07:27:19.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.064+0000 7f08afa39640 1 Processor -- start 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.064+0000 7f08afa39640 1 -- start start 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.064+0000 7f08afa39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f08a8160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.064+0000 7f08afa39640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08a815a7f0 0x7f08a8159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.064+0000 7f08afa39640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08a815ead0 0x7f08a815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.064+0000 7f08afa39640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80cedf0 con 0x7f0894004800 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.064+0000 7f08afa39640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80cec70 con 0x7f08a815ead0 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.064+0000 7f08afa39640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80d25b0 con 0x7f08a815a7f0 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08ad7d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f08a8160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08ad7d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f08a8160b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34342/0 (socket says 172.21.15.27:34342) 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08ad7d5640 1 -- 172.21.15.27:0/199918413 learned_addr learned my addr 172.21.15.27:0/199918413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08adfd6640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08a815ead0 0x7f08a815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08acfd4640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08a815a7f0 0x7f08a8159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08ad7d5640 1 -- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08a815a7f0 msgr2=0x7f08a8159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08ad7d5640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08a815a7f0 0x7f08a8159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08ad7d5640 1 -- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08a815ead0 msgr2=0x7f08a815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08ad7d5640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08a815ead0 0x7f08a815ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08ad7d5640 1 -- 172.21.15.27:0/199918413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a80cc9f0 con 0x7f0894004800 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08adfd6640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08a815ead0 0x7f08a815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08ad7d5640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f08a8160b60 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f08a0004f20 tx=0x7f08a00074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f089dffb640 1 -- 172.21.15.27:0/199918413 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a000f750 con 0x7f0894004800 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f089dffb640 1 -- 172.21.15.27:0/199918413 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a0018070 con 0x7f0894004800 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a80cda50 con 0x7f0894004800 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.065+0000 7f089dffb640 1 -- 172.21.15.27:0/199918413 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a000fb30 con 0x7f0894004800 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.066+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a80ccc90 con 0x7f0894004800 2024-03-06T07:27:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.066+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a8067ff0 con 0x7f0894004800 2024-03-06T07:27:19.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.067+0000 7f089dffb640 1 -- 172.21.15.27:0/199918413 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08a000fcd0 con 0x7f0894004800 2024-03-06T07:27:19.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.067+0000 7f089dffb640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f088c06b4d0 0x7f088c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.067+0000 7f089dffb640 1 -- 172.21.15.27:0/199918413 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08a008a170 con 0x7f0894004800 2024-03-06T07:27:19.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.068+0000 7f08acfd4640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f088c06b4d0 0x7f088c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.068+0000 7f08acfd4640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f088c06b4d0 0x7f088c06d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0898001750 tx=0x7f0898009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.076+0000 7f089dffb640 1 -- 172.21.15.27:0/199918413 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08a00590b0 con 0x7f0894004800 2024-03-06T07:27:19.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.252+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08a815aff0 con 0x7f088c06b4d0 2024-03-06T07:27:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.259+0000 7f089dffb640 1 -- 172.21.15.27:0/199918413 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252086 (secure 0 0 0) 0x7f08a815aff0 con 0x7f088c06b4d0 2024-03-06T07:27:19.264 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:27:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f088c06b4d0 msgr2=0x7f088c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f088c06b4d0 0x7f088c06d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0898001750 tx=0x7f0898009030 comp rx=0 tx=0).stop 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 msgr2=0x7f08a8160b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f08a8160b60 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f08a0004f20 tx=0x7f08a00074f0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 shutdown_connections 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f088c06b4d0 0x7f088c06d990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08a815ead0 0x7f08a815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08a815a7f0 0x7f08a8159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 --2- 172.21.15.27:0/199918413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0894004800 0x7f08a8160b60 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 >> 172.21.15.27:0/199918413 conn(0x7f08a8071960 msgr2=0x7f08a80b7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 shutdown_connections 2024-03-06T07:27:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.262+0000 7f08afa39640 1 -- 172.21.15.27:0/199918413 wait complete. 2024-03-06T07:27:19.275 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-03-06T07:27:18.051600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258748,"kb_used_data":9288,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715972,"statfs":{"total":773094113280,"available":772829155328,"internally_reserved":0,"allocated":9510912,"data_stored":4685468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002055"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942083+0000","last_change":"2024-03-06T07:27:10.394265+0000","last_active":"2024-03-06T07:27:11.942083+0000","last_peered":"2024-03-06T07:27:11.942083+0000","last_clean":"2024-03-06T07:27:11.942083+0000","last_became_active":"2024-03-06T07:27:10.393971+0000","last_became_peered":"2024-03-06T07:27:10.393971+0000","last_unstale":"2024-03-06T07:27:11.942083+0000","last_undegraded":"2024-03-06T07:27:11.942083+0000","last_fullsized":"2024-03-06T07:27:11.942083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:49.740586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397787+0000","last_change":"2024-03-06T07:27:10.405318+0000","last_active":"2024-03-06T07:27:11.397787+0000","last_peered":"2024-03-06T07:27:11.397787+0000","last_clean":"2024-03-06T07:27:11.397787+0000","last_became_active":"2024-03-06T07:27:10.404871+0000","last_became_peered":"2024-03-06T07:27:10.404871+0000","last_unstale":"2024-03-06T07:27:11.397787+0000","last_undegraded":"2024-03-06T07:27:11.397787+0000","last_fullsized":"2024-03-06T07:27:11.397787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:08.206841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388886+0000","last_change":"2024-03-06T07:27:10.389157+0000","last_active":"2024-03-06T07:27:11.388886+0000","last_peered":"2024-03-06T07:27:11.388886+0000","last_clean":"2024-03-06T07:27:11.388886+0000","last_became_active":"2024-03-06T07:27:10.388860+0000","last_became_peered":"2024-03-06T07:27:10.388860+0000","last_unstale":"2024-03-06T07:27:11.388886+0000","last_undegraded":"2024-03-06T07:27:11.388886+0000","last_fullsized":"2024-03-06T07:27:11.388886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:35.985491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.940894+0000","last_change":"2024-03-06T07:27:10.406734+0000","last_active":"2024-03-06T07:27:11.940894+0000","last_peered":"2024-03-06T07:27:11.940894+0000","last_clean":"2024-03-06T07:27:11.940894+0000","last_became_active":"2024-03-06T07:27:10.406432+0000","last_became_peered":"2024-03-06T07:27:10.406432+0000","last_unstale":"2024-03-06T07:27:11.940894+0000","last_undegraded":"2024-03-06T07:27:11.940894+0000","last_fullsized":"2024-03-06T07:27:11.940894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:16.753769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389733+0000","last_change":"2024-03-06T07:27:10.391484+0000","last_active":"2024-03-06T07:27:11.389733+0000","last_peered":"2024-03-06T07:27:11.389733+0000","last_clean":"2024-03-06T07:27:11.389733+0000","last_became_active":"2024-03-06T07:27:10.391232+0000","last_became_peered":"2024-03-06T07:27:10.391232+0000","last_unstale":"2024-03-06T07:27:11.389733+0000","last_undegraded":"2024-03-06T07:27:11.389733+0000","last_fullsized":"2024-03-06T07:27:11.389733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:04.614996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941671+0000","last_change":"2024-03-06T07:27:10.409233+0000","last_active":"2024-03-06T07:27:11.941671+0000","last_peered":"2024-03-06T07:27:11.941671+0000","last_clean":"2024-03-06T07:27:11.941671+0000","last_became_active":"2024-03-06T07:27:10.408829+0000","last_became_peered":"2024-03-06T07:27:10.408829+0000","last_unstale":"2024-03-06T07:27:11.941671+0000","last_undegraded":"2024-03-06T07:27:11.941671+0000","last_fullsized":"2024-03-06T07:27:11.941671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:04.763040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.940601+0000","last_change":"2024-03-06T07:27:10.397993+0000","last_active":"2024-03-06T07:27:11.940601+0000","last_peered":"2024-03-06T07:27:11.940601+0000","last_clean":"2024-03-06T07:27:11.940601+0000","last_became_active":"2024-03-06T07:27:10.397676+0000","last_became_peered":"2024-03-06T07:27:10.397676+0000","last_unstale":"2024-03-06T07:27:11.940601+0000","last_undegraded":"2024-03-06T07:27:11.940601+0000","last_fullsized":"2024-03-06T07:27:11.940601+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:02.764085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941469+0000","last_change":"2024-03-06T07:27:10.395127+0000","last_active":"2024-03-06T07:27:11.941469+0000","last_peered":"2024-03-06T07:27:11.941469+0000","last_clean":"2024-03-06T07:27:11.941469+0000","last_became_active":"2024-03-06T07:27:10.394777+0000","last_became_peered":"2024-03-06T07:27:10.394777+0000","last_unstale":"2024-03-06T07:27:11.941469+0000","last_undegraded":"2024-03-06T07:27:11.941469+0000","last_fullsized":"2024-03-06T07:27:11.941469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:44.089432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941842+0000","last_change":"2024-03-06T07:27:10.396067+0000","last_active":"2024-03-06T07:27:11.941842+0000","last_peered":"2024-03-06T07:27:11.941842+0000","last_clean":"2024-03-06T07:27:11.941842+0000","last_became_active":"2024-03-06T07:27:10.395730+0000","last_became_peered":"2024-03-06T07:27:10.395730+0000","last_unstale":"2024-03-06T07:27:11.941842+0000","last_undegraded":"2024-03-06T07:27:11.941842+0000","last_fullsized":"2024-03-06T07:27:11.941842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:09.014322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942317+0000","last_change":"2024-03-06T07:27:10.390713+0000","last_active":"2024-03-06T07:27:11.942317+0000","last_peered":"2024-03-06T07:27:11.942317+0000","last_clean":"2024-03-06T07:27:11.942317+0000","last_became_active":"2024-03-06T07:27:10.390419+0000","last_became_peered":"2024-03-06T07:27:10.390419+0000","last_unstale":"2024-03-06T07:27:11.942317+0000","last_undegraded":"2024-03-06T07:27:11.942317+0000","last_fullsized":"2024-03-06T07:27:11.942317+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:34.968369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397623+0000","last_change":"2024-03-06T07:27:10.389603+0000","last_active":"2024-03-06T07:27:11.397623+0000","last_peered":"2024-03-06T07:27:11.397623+0000","last_clean":"2024-03-06T07:27:11.397623+0000","last_became_active":"2024-03-06T07:27:10.389305+0000","last_became_peered":"2024-03-06T07:27:10.389305+0000","last_unstale":"2024-03-06T07:27:11.397623+0000","last_undegraded":"2024-03-06T07:27:11.397623+0000","last_fullsized":"2024-03-06T07:27:11.397623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:50.606509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941602+0000","last_change":"2024-03-06T07:27:10.394431+0000","last_active":"2024-03-06T07:27:11.941602+0000","last_peered":"2024-03-06T07:27:11.941602+0000","last_clean":"2024-03-06T07:27:11.941602+0000","last_became_active":"2024-03-06T07:27:10.393968+0000","last_became_peered":"2024-03-06T07:27:10.393968+0000","last_unstale":"2024-03-06T07:27:11.941602+0000","last_undegraded":"2024-03-06T07:27:11.941602+0000","last_fullsized":"2024-03-06T07:27:11.941602+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:08.487015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389201+0000","last_change":"2024-03-06T07:27:10.388660+0000","last_active":"2024-03-06T07:27:11.389201+0000","last_peered":"2024-03-06T07:27:11.389201+0000","last_clean":"2024-03-06T07:27:11.389201+0000","last_became_active":"2024-03-06T07:27:10.388325+0000","last_became_peered":"2024-03-06T07:27:10.388325+0000","last_unstale":"2024-03-06T07:27:11.389201+0000","last_undegraded":"2024-03-06T07:27:11.389201+0000","last_fullsized":"2024-03-06T07:27:11.389201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:40.660680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941019+0000","last_change":"2024-03-06T07:27:10.399127+0000","last_active":"2024-03-06T07:27:11.941019+0000","last_peered":"2024-03-06T07:27:11.941019+0000","last_clean":"2024-03-06T07:27:11.941019+0000","last_became_active":"2024-03-06T07:27:10.398609+0000","last_became_peered":"2024-03-06T07:27:10.398609+0000","last_unstale":"2024-03-06T07:27:11.941019+0000","last_undegraded":"2024-03-06T07:27:11.941019+0000","last_fullsized":"2024-03-06T07:27:11.941019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:25.004410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.703991+0000","last_change":"2024-03-06T07:27:10.388019+0000","last_active":"2024-03-06T07:27:11.703991+0000","last_peered":"2024-03-06T07:27:11.703991+0000","last_clean":"2024-03-06T07:27:11.703991+0000","last_became_active":"2024-03-06T07:27:10.387715+0000","last_became_peered":"2024-03-06T07:27:10.387715+0000","last_unstale":"2024-03-06T07:27:11.703991+0000","last_undegraded":"2024-03-06T07:27:11.703991+0000","last_fullsized":"2024-03-06T07:27:11.703991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:17.311840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388995+0000","last_change":"2024-03-06T07:27:10.389909+0000","last_active":"2024-03-06T07:27:11.388995+0000","last_peered":"2024-03-06T07:27:11.388995+0000","last_clean":"2024-03-06T07:27:11.388995+0000","last_became_active":"2024-03-06T07:27:10.389536+0000","last_became_peered":"2024-03-06T07:27:10.389536+0000","last_unstale":"2024-03-06T07:27:11.388995+0000","last_undegraded":"2024-03-06T07:27:11.388995+0000","last_fullsized":"2024-03-06T07:27:11.388995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:37.760458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389506+0000","last_change":"2024-03-06T07:27:10.409063+0000","last_active":"2024-03-06T07:27:11.389506+0000","last_peered":"2024-03-06T07:27:11.389506+0000","last_clean":"2024-03-06T07:27:11.389506+0000","last_became_active":"2024-03-06T07:27:10.408750+0000","last_became_peered":"2024-03-06T07:27:10.408750+0000","last_unstale":"2024-03-06T07:27:11.389506+0000","last_undegraded":"2024-03-06T07:27:11.389506+0000","last_fullsized":"2024-03-06T07:27:11.389506+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:21.819419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.940774+0000","last_change":"2024-03-06T07:27:10.406970+0000","last_active":"2024-03-06T07:27:11.940774+0000","last_peered":"2024-03-06T07:27:11.940774+0000","last_clean":"2024-03-06T07:27:11.940774+0000","last_became_active":"2024-03-06T07:27:10.406645+0000","last_became_peered":"2024-03-06T07:27:10.406645+0000","last_unstale":"2024-03-06T07:27:11.940774+0000","last_undegraded":"2024-03-06T07:27:11.940774+0000","last_fullsized":"2024-03-06T07:27:11.940774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:07.227316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388496+0000","last_change":"2024-03-06T07:27:10.385944+0000","last_active":"2024-03-06T07:27:11.388496+0000","last_peered":"2024-03-06T07:27:11.388496+0000","last_clean":"2024-03-06T07:27:11.388496+0000","last_became_active":"2024-03-06T07:27:10.385682+0000","last_became_peered":"2024-03-06T07:27:10.385682+0000","last_unstale":"2024-03-06T07:27:11.388496+0000","last_undegraded":"2024-03-06T07:27:11.388496+0000","last_fullsized":"2024-03-06T07:27:11.388496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:34.968369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.940915+0000","last_change":"2024-03-06T07:27:10.406152+0000","last_active":"2024-03-06T07:27:11.940915+0000","last_peered":"2024-03-06T07:27:11.940915+0000","last_clean":"2024-03-06T07:27:11.940915+0000","last_became_active":"2024-03-06T07:27:10.405822+0000","last_became_peered":"2024-03-06T07:27:10.405822+0000","last_unstale":"2024-03-06T07:27:11.940915+0000","last_undegraded":"2024-03-06T07:27:11.940915+0000","last_fullsized":"2024-03-06T07:27:11.940915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:31.622931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.940749+0000","last_change":"2024-03-06T07:27:10.406447+0000","last_active":"2024-03-06T07:27:11.940749+0000","last_peered":"2024-03-06T07:27:11.940749+0000","last_clean":"2024-03-06T07:27:11.940749+0000","last_became_active":"2024-03-06T07:27:10.406110+0000","last_became_peered":"2024-03-06T07:27:10.406110+0000","last_unstale":"2024-03-06T07:27:11.940749+0000","last_undegraded":"2024-03-06T07:27:11.940749+0000","last_fullsized":"2024-03-06T07:27:11.940749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:58.107493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397360+0000","last_change":"2024-03-06T07:27:10.396306+0000","last_active":"2024-03-06T07:27:11.397360+0000","last_peered":"2024-03-06T07:27:11.397360+0000","last_clean":"2024-03-06T07:27:11.397360+0000","last_became_active":"2024-03-06T07:27:10.395921+0000","last_became_peered":"2024-03-06T07:27:10.395921+0000","last_unstale":"2024-03-06T07:27:11.397360+0000","last_undegraded":"2024-03-06T07:27:11.397360+0000","last_fullsized":"2024-03-06T07:27:11.397360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:09.500962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941462+0000","last_change":"2024-03-06T07:27:10.399827+0000","last_active":"2024-03-06T07:27:11.941462+0000","last_peered":"2024-03-06T07:27:11.941462+0000","last_clean":"2024-03-06T07:27:11.941462+0000","last_became_active":"2024-03-06T07:27:10.399444+0000","last_became_peered":"2024-03-06T07:27:10.399444+0000","last_unstale":"2024-03-06T07:27:11.941462+0000","last_undegraded":"2024-03-06T07:27:11.941462+0000","last_fullsized":"2024-03-06T07:27:11.941462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:51.286328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.940557+0000","last_change":"2024-03-06T07:27:10.397702+0000","last_active":"2024-03-06T07:27:11.940557+0000","last_peered":"2024-03-06T07:27:11.940557+0000","last_clean":"2024-03-06T07:27:11.940557+0000","last_became_active":"2024-03-06T07:27:10.397204+0000","last_became_peered":"2024-03-06T07:27:10.397204+0000","last_unstale":"2024-03-06T07:27:11.940557+0000","last_undegraded":"2024-03-06T07:27:11.940557+0000","last_fullsized":"2024-03-06T07:27:11.940557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:26.778735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397488+0000","last_change":"2024-03-06T07:27:10.404148+0000","last_active":"2024-03-06T07:27:11.397488+0000","last_peered":"2024-03-06T07:27:11.397488+0000","last_clean":"2024-03-06T07:27:11.397488+0000","last_became_active":"2024-03-06T07:27:10.403843+0000","last_became_peered":"2024-03-06T07:27:10.403843+0000","last_unstale":"2024-03-06T07:27:11.397488+0000","last_undegraded":"2024-03-06T07:27:11.397488+0000","last_fullsized":"2024-03-06T07:27:11.397488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:27.080891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389143+0000","last_change":"2024-03-06T07:27:10.397049+0000","last_active":"2024-03-06T07:27:11.389143+0000","last_peered":"2024-03-06T07:27:11.389143+0000","last_clean":"2024-03-06T07:27:11.389143+0000","last_became_active":"2024-03-06T07:27:10.396693+0000","last_became_peered":"2024-03-06T07:27:10.396693+0000","last_unstale":"2024-03-06T07:27:11.389143+0000","last_undegraded":"2024-03-06T07:27:11.389143+0000","last_fullsized":"2024-03-06T07:27:11.389143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:01.289849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389699+0000","last_change":"2024-03-06T07:27:10.390606+0000","last_active":"2024-03-06T07:27:11.389699+0000","last_peered":"2024-03-06T07:27:11.389699+0000","last_clean":"2024-03-06T07:27:11.389699+0000","last_became_active":"2024-03-06T07:27:10.390306+0000","last_became_peered":"2024-03-06T07:27:10.390306+0000","last_unstale":"2024-03-06T07:27:11.389699+0000","last_undegraded":"2024-03-06T07:27:11.389699+0000","last_fullsized":"2024-03-06T07:27:11.389699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:35.634856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941543+0000","last_change":"2024-03-06T07:27:10.394836+0000","last_active":"2024-03-06T07:27:11.941543+0000","last_peered":"2024-03-06T07:27:11.941543+0000","last_clean":"2024-03-06T07:27:11.941543+0000","last_became_active":"2024-03-06T07:27:10.394326+0000","last_became_peered":"2024-03-06T07:27:10.394326+0000","last_unstale":"2024-03-06T07:27:11.941543+0000","last_undegraded":"2024-03-06T07:27:11.941543+0000","last_fullsized":"2024-03-06T07:27:11.941543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:04.763040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.398207+0000","last_change":"2024-03-06T07:27:10.391115+0000","last_active":"2024-03-06T07:27:11.398207+0000","last_peered":"2024-03-06T07:27:11.398207+0000","last_clean":"2024-03-06T07:27:11.398207+0000","last_became_active":"2024-03-06T07:27:10.390835+0000","last_became_peered":"2024-03-06T07:27:10.390835+0000","last_unstale":"2024-03-06T07:27:11.398207+0000","last_undegraded":"2024-03-06T07:27:11.398207+0000","last_fullsized":"2024-03-06T07:27:11.398207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:54.580702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397378+0000","last_change":"2024-03-06T07:27:10.409660+0000","last_active":"2024-03-06T07:27:11.397378+0000","last_peered":"2024-03-06T07:27:11.397378+0000","last_clean":"2024-03-06T07:27:11.397378+0000","last_became_active":"2024-03-06T07:27:10.409321+0000","last_became_peered":"2024-03-06T07:27:10.409321+0000","last_unstale":"2024-03-06T07:27:11.397378+0000","last_undegraded":"2024-03-06T07:27:11.397378+0000","last_fullsized":"2024-03-06T07:27:11.397378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:33.750480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389186+0000","last_change":"2024-03-06T07:27:10.396108+0000","last_active":"2024-03-06T07:27:11.389186+0000","last_peered":"2024-03-06T07:27:11.389186+0000","last_clean":"2024-03-06T07:27:11.389186+0000","last_became_active":"2024-03-06T07:27:10.395808+0000","last_became_peered":"2024-03-06T07:27:10.395808+0000","last_unstale":"2024-03-06T07:27:11.389186+0000","last_undegraded":"2024-03-06T07:27:11.389186+0000","last_fullsized":"2024-03-06T07:27:11.389186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:57.214126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941392+0000","last_change":"2024-03-06T07:27:10.397335+0000","last_active":"2024-03-06T07:27:11.941392+0000","last_peered":"2024-03-06T07:27:11.941392+0000","last_clean":"2024-03-06T07:27:11.941392+0000","last_became_active":"2024-03-06T07:27:10.396718+0000","last_became_peered":"2024-03-06T07:27:10.396718+0000","last_unstale":"2024-03-06T07:27:11.941392+0000","last_undegraded":"2024-03-06T07:27:11.941392+0000","last_fullsized":"2024-03-06T07:27:11.941392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:44.089432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.390043+0000","last_change":"2024-03-06T07:27:10.389873+0000","last_active":"2024-03-06T07:27:11.390043+0000","last_peered":"2024-03-06T07:27:11.390043+0000","last_clean":"2024-03-06T07:27:11.390043+0000","last_became_active":"2024-03-06T07:27:10.389569+0000","last_became_peered":"2024-03-06T07:27:10.389569+0000","last_unstale":"2024-03-06T07:27:11.390043+0000","last_undegraded":"2024-03-06T07:27:11.390043+0000","last_fullsized":"2024-03-06T07:27:11.390043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:08.487015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"83'4","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942238+0000","last_change":"2024-03-06T07:27:09.378545+0000","last_active":"2024-03-06T07:27:11.942238+0000","last_peered":"2024-03-06T07:27:11.942238+0000","last_clean":"2024-03-06T07:27:11.942238+0000","last_became_active":"2024-03-06T07:27:09.378276+0000","last_became_peered":"2024-03-06T07:27:09.378276+0000","last_unstale":"2024-03-06T07:27:11.942238+0000","last_undegraded":"2024-03-06T07:27:11.942238+0000","last_fullsized":"2024-03-06T07:27:11.942238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:13.018870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942159+0000","last_change":"2024-03-06T07:27:10.400371+0000","last_active":"2024-03-06T07:27:11.942159+0000","last_peered":"2024-03-06T07:27:11.942159+0000","last_clean":"2024-03-06T07:27:11.942159+0000","last_became_active":"2024-03-06T07:27:10.399519+0000","last_became_peered":"2024-03-06T07:27:10.399519+0000","last_unstale":"2024-03-06T07:27:11.942159+0000","last_undegraded":"2024-03-06T07:27:11.942159+0000","last_fullsized":"2024-03-06T07:27:11.942159+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:07.868224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.390156+0000","last_change":"2024-03-06T07:27:09.375904+0000","last_active":"2024-03-06T07:27:11.390156+0000","last_peered":"2024-03-06T07:27:11.390156+0000","last_clean":"2024-03-06T07:27:11.390156+0000","last_became_active":"2024-03-06T07:27:09.375566+0000","last_became_peered":"2024-03-06T07:27:09.375566+0000","last_unstale":"2024-03-06T07:27:11.390156+0000","last_undegraded":"2024-03-06T07:27:11.390156+0000","last_fullsized":"2024-03-06T07:27:11.390156+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:49.139300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942504+0000","last_change":"2024-03-06T07:27:10.397433+0000","last_active":"2024-03-06T07:27:11.942504+0000","last_peered":"2024-03-06T07:27:11.942504+0000","last_clean":"2024-03-06T07:27:11.942504+0000","last_became_active":"2024-03-06T07:27:10.397172+0000","last_became_peered":"2024-03-06T07:27:10.397172+0000","last_unstale":"2024-03-06T07:27:11.942504+0000","last_undegraded":"2024-03-06T07:27:11.942504+0000","last_fullsized":"2024-03-06T07:27:11.942504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:09.284951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388697+0000","last_change":"2024-03-06T07:27:09.371083+0000","last_active":"2024-03-06T07:27:11.388697+0000","last_peered":"2024-03-06T07:27:11.388697+0000","last_clean":"2024-03-06T07:27:11.388697+0000","last_became_active":"2024-03-06T07:27:09.370820+0000","last_became_peered":"2024-03-06T07:27:09.370820+0000","last_unstale":"2024-03-06T07:27:11.388697+0000","last_undegraded":"2024-03-06T07:27:11.388697+0000","last_fullsized":"2024-03-06T07:27:11.388697+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:03.753803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389093+0000","last_change":"2024-03-06T07:27:10.407889+0000","last_active":"2024-03-06T07:27:11.389093+0000","last_peered":"2024-03-06T07:27:11.389093+0000","last_clean":"2024-03-06T07:27:11.389093+0000","last_became_active":"2024-03-06T07:27:10.407661+0000","last_became_peered":"2024-03-06T07:27:10.407661+0000","last_unstale":"2024-03-06T07:27:11.389093+0000","last_undegraded":"2024-03-06T07:27:11.389093+0000","last_fullsized":"2024-03-06T07:27:11.389093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:50.577701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388553+0000","last_change":"2024-03-06T07:27:09.374749+0000","last_active":"2024-03-06T07:27:11.388553+0000","last_peered":"2024-03-06T07:27:11.388553+0000","last_clean":"2024-03-06T07:27:11.388553+0000","last_became_active":"2024-03-06T07:27:09.374421+0000","last_became_peered":"2024-03-06T07:27:09.374421+0000","last_unstale":"2024-03-06T07:27:11.388553+0000","last_undegraded":"2024-03-06T07:27:11.388553+0000","last_fullsized":"2024-03-06T07:27:11.388553+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:34.625619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397516+0000","last_change":"2024-03-06T07:27:10.409592+0000","last_active":"2024-03-06T07:27:11.397516+0000","last_peered":"2024-03-06T07:27:11.397516+0000","last_clean":"2024-03-06T07:27:11.397516+0000","last_became_active":"2024-03-06T07:27:10.408978+0000","last_became_peered":"2024-03-06T07:27:10.408978+0000","last_unstale":"2024-03-06T07:27:11.397516+0000","last_undegraded":"2024-03-06T07:27:11.397516+0000","last_fullsized":"2024-03-06T07:27:11.397516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:18.724540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389906+0000","last_change":"2024-03-06T07:27:09.379796+0000","last_active":"2024-03-06T07:27:11.389906+0000","last_peered":"2024-03-06T07:27:11.389906+0000","last_clean":"2024-03-06T07:27:11.389906+0000","last_became_active":"2024-03-06T07:27:09.379609+0000","last_became_peered":"2024-03-06T07:27:09.379609+0000","last_unstale":"2024-03-06T07:27:11.389906+0000","last_undegraded":"2024-03-06T07:27:11.389906+0000","last_fullsized":"2024-03-06T07:27:11.389906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:53.589572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.390052+0000","last_change":"2024-03-06T07:27:10.406972+0000","last_active":"2024-03-06T07:27:11.390052+0000","last_peered":"2024-03-06T07:27:11.390052+0000","last_clean":"2024-03-06T07:27:11.390052+0000","last_became_active":"2024-03-06T07:27:10.406575+0000","last_became_peered":"2024-03-06T07:27:10.406575+0000","last_unstale":"2024-03-06T07:27:11.390052+0000","last_undegraded":"2024-03-06T07:27:11.390052+0000","last_fullsized":"2024-03-06T07:27:11.390052+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:09.014322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389479+0000","last_change":"2024-03-06T07:27:09.375952+0000","last_active":"2024-03-06T07:27:11.389479+0000","last_peered":"2024-03-06T07:27:11.389479+0000","last_clean":"2024-03-06T07:27:11.389479+0000","last_became_active":"2024-03-06T07:27:09.375584+0000","last_became_peered":"2024-03-06T07:27:09.375584+0000","last_unstale":"2024-03-06T07:27:11.389479+0000","last_undegraded":"2024-03-06T07:27:11.389479+0000","last_fullsized":"2024-03-06T07:27:11.389479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:09.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388712+0000","last_change":"2024-03-06T07:27:10.396113+0000","last_active":"2024-03-06T07:27:11.388712+0000","last_peered":"2024-03-06T07:27:11.388712+0000","last_clean":"2024-03-06T07:27:11.388712+0000","last_became_active":"2024-03-06T07:27:10.394332+0000","last_became_peered":"2024-03-06T07:27:10.394332+0000","last_unstale":"2024-03-06T07:27:11.388712+0000","last_undegraded":"2024-03-06T07:27:11.388712+0000","last_fullsized":"2024-03-06T07:27:11.388712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:50.080550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388111+0000","last_change":"2024-03-06T07:27:09.366681+0000","last_active":"2024-03-06T07:27:11.388111+0000","last_peered":"2024-03-06T07:27:11.388111+0000","last_clean":"2024-03-06T07:27:11.388111+0000","last_became_active":"2024-03-06T07:27:09.366329+0000","last_became_peered":"2024-03-06T07:27:09.366329+0000","last_unstale":"2024-03-06T07:27:11.388111+0000","last_undegraded":"2024-03-06T07:27:11.388111+0000","last_fullsized":"2024-03-06T07:27:11.388111+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:56.204889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.704027+0000","last_change":"2024-03-06T07:27:10.388876+0000","last_active":"2024-03-06T07:27:11.704027+0000","last_peered":"2024-03-06T07:27:11.704027+0000","last_clean":"2024-03-06T07:27:11.704027+0000","last_became_active":"2024-03-06T07:27:10.388610+0000","last_became_peered":"2024-03-06T07:27:10.388610+0000","last_unstale":"2024-03-06T07:27:11.704027+0000","last_undegraded":"2024-03-06T07:27:11.704027+0000","last_fullsized":"2024-03-06T07:27:11.704027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:23.887895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397247+0000","last_change":"2024-03-06T07:27:09.378990+0000","last_active":"2024-03-06T07:27:11.397247+0000","last_peered":"2024-03-06T07:27:11.397247+0000","last_clean":"2024-03-06T07:27:11.397247+0000","last_became_active":"2024-03-06T07:27:09.378578+0000","last_became_peered":"2024-03-06T07:27:09.378578+0000","last_unstale":"2024-03-06T07:27:11.397247+0000","last_undegraded":"2024-03-06T07:27:11.397247+0000","last_fullsized":"2024-03-06T07:27:11.397247+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:17.424123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.704314+0000","last_change":"2024-03-06T07:27:10.395052+0000","last_active":"2024-03-06T07:27:11.704314+0000","last_peered":"2024-03-06T07:27:11.704314+0000","last_clean":"2024-03-06T07:27:11.704314+0000","last_became_active":"2024-03-06T07:27:10.394788+0000","last_became_peered":"2024-03-06T07:27:10.394788+0000","last_unstale":"2024-03-06T07:27:11.704314+0000","last_undegraded":"2024-03-06T07:27:11.704314+0000","last_fullsized":"2024-03-06T07:27:11.704314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:35.870894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942307+0000","last_change":"2024-03-06T07:27:09.369411+0000","last_active":"2024-03-06T07:27:11.942307+0000","last_peered":"2024-03-06T07:27:11.942307+0000","last_clean":"2024-03-06T07:27:11.942307+0000","last_became_active":"2024-03-06T07:27:09.369100+0000","last_became_peered":"2024-03-06T07:27:09.369100+0000","last_unstale":"2024-03-06T07:27:11.942307+0000","last_undegraded":"2024-03-06T07:27:11.942307+0000","last_fullsized":"2024-03-06T07:27:11.942307+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:09.257168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941905+0000","last_change":"2024-03-06T07:27:10.409590+0000","last_active":"2024-03-06T07:27:11.941905+0000","last_peered":"2024-03-06T07:27:11.941905+0000","last_clean":"2024-03-06T07:27:11.941905+0000","last_became_active":"2024-03-06T07:27:10.408868+0000","last_became_peered":"2024-03-06T07:27:10.408868+0000","last_unstale":"2024-03-06T07:27:11.941905+0000","last_undegraded":"2024-03-06T07:27:11.941905+0000","last_fullsized":"2024-03-06T07:27:11.941905+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:49.740586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942424+0000","last_change":"2024-03-06T07:27:09.372377+0000","last_active":"2024-03-06T07:27:11.942424+0000","last_peered":"2024-03-06T07:27:11.942424+0000","last_clean":"2024-03-06T07:27:11.942424+0000","last_became_active":"2024-03-06T07:27:09.371991+0000","last_became_peered":"2024-03-06T07:27:09.371991+0000","last_unstale":"2024-03-06T07:27:11.942424+0000","last_undegraded":"2024-03-06T07:27:11.942424+0000","last_fullsized":"2024-03-06T07:27:11.942424+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:02.670033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388479+0000","last_change":"2024-03-06T07:27:10.393351+0000","last_active":"2024-03-06T07:27:11.388479+0000","last_peered":"2024-03-06T07:27:11.388479+0000","last_clean":"2024-03-06T07:27:11.388479+0000","last_became_active":"2024-03-06T07:27:10.393053+0000","last_became_peered":"2024-03-06T07:27:10.393053+0000","last_unstale":"2024-03-06T07:27:11.388479+0000","last_undegraded":"2024-03-06T07:27:11.388479+0000","last_fullsized":"2024-03-06T07:27:11.388479+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:09.153697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"83'3","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941770+0000","last_change":"2024-03-06T07:27:09.378597+0000","last_active":"2024-03-06T07:27:11.941770+0000","last_peered":"2024-03-06T07:27:11.941770+0000","last_clean":"2024-03-06T07:27:11.941770+0000","last_became_active":"2024-03-06T07:27:09.378334+0000","last_became_peered":"2024-03-06T07:27:09.378334+0000","last_unstale":"2024-03-06T07:27:11.941770+0000","last_undegraded":"2024-03-06T07:27:11.941770+0000","last_fullsized":"2024-03-06T07:27:11.941770+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:49.139300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389751+0000","last_change":"2024-03-06T07:27:10.397116+0000","last_active":"2024-03-06T07:27:11.389751+0000","last_peered":"2024-03-06T07:27:11.389751+0000","last_clean":"2024-03-06T07:27:11.389751+0000","last_became_active":"2024-03-06T07:27:10.396267+0000","last_became_peered":"2024-03-06T07:27:10.396267+0000","last_unstale":"2024-03-06T07:27:11.389751+0000","last_undegraded":"2024-03-06T07:27:11.389751+0000","last_fullsized":"2024-03-06T07:27:11.389751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:44.089432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"83'6","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941639+0000","last_change":"2024-03-06T07:27:09.365562+0000","last_active":"2024-03-06T07:27:11.941639+0000","last_peered":"2024-03-06T07:27:11.941639+0000","last_clean":"2024-03-06T07:27:11.941639+0000","last_became_active":"2024-03-06T07:27:09.365279+0000","last_became_peered":"2024-03-06T07:27:09.365279+0000","last_unstale":"2024-03-06T07:27:11.941639+0000","last_undegraded":"2024-03-06T07:27:11.941639+0000","last_fullsized":"2024-03-06T07:27:11.941639+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:44.994841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388298+0000","last_change":"2024-03-06T07:27:10.390241+0000","last_active":"2024-03-06T07:27:11.388298+0000","last_peered":"2024-03-06T07:27:11.388298+0000","last_clean":"2024-03-06T07:27:11.388298+0000","last_became_active":"2024-03-06T07:27:10.389430+0000","last_became_peered":"2024-03-06T07:27:10.389430+0000","last_unstale":"2024-03-06T07:27:11.388298+0000","last_undegraded":"2024-03-06T07:27:11.388298+0000","last_fullsized":"2024-03-06T07:27:11.388298+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:07.868224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.398197+0000","last_change":"2024-03-06T07:27:09.378045+0000","last_active":"2024-03-06T07:27:11.398197+0000","last_peered":"2024-03-06T07:27:11.398197+0000","last_clean":"2024-03-06T07:27:11.398197+0000","last_became_active":"2024-03-06T07:27:09.377782+0000","last_became_peered":"2024-03-06T07:27:09.377782+0000","last_unstale":"2024-03-06T07:27:11.398197+0000","last_undegraded":"2024-03-06T07:27:11.398197+0000","last_fullsized":"2024-03-06T07:27:11.398197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:12.105383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389380+0000","last_change":"2024-03-06T07:27:10.395103+0000","last_active":"2024-03-06T07:27:11.389380+0000","last_peered":"2024-03-06T07:27:11.389380+0000","last_clean":"2024-03-06T07:27:11.389380+0000","last_became_active":"2024-03-06T07:27:10.394823+0000","last_became_peered":"2024-03-06T07:27:10.394823+0000","last_unstale":"2024-03-06T07:27:11.389380+0000","last_undegraded":"2024-03-06T07:27:11.389380+0000","last_fullsized":"2024-03-06T07:27:11.389380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:00.874502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.704053+0000","last_change":"2024-03-06T07:27:09.378309+0000","last_active":"2024-03-06T07:27:11.704053+0000","last_peered":"2024-03-06T07:27:11.704053+0000","last_clean":"2024-03-06T07:27:11.704053+0000","last_became_active":"2024-03-06T07:27:09.378012+0000","last_became_peered":"2024-03-06T07:27:09.378012+0000","last_unstale":"2024-03-06T07:27:11.704053+0000","last_undegraded":"2024-03-06T07:27:11.704053+0000","last_fullsized":"2024-03-06T07:27:11.704053+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:49.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397278+0000","last_change":"2024-03-06T07:27:10.409588+0000","last_active":"2024-03-06T07:27:11.397278+0000","last_peered":"2024-03-06T07:27:11.397278+0000","last_clean":"2024-03-06T07:27:11.397278+0000","last_became_active":"2024-03-06T07:27:10.409246+0000","last_became_peered":"2024-03-06T07:27:10.409246+0000","last_unstale":"2024-03-06T07:27:11.397278+0000","last_undegraded":"2024-03-06T07:27:11.397278+0000","last_fullsized":"2024-03-06T07:27:11.397278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:31.622931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941874+0000","last_change":"2024-03-06T07:27:09.370907+0000","last_active":"2024-03-06T07:27:11.941874+0000","last_peered":"2024-03-06T07:27:11.941874+0000","last_clean":"2024-03-06T07:27:11.941874+0000","last_became_active":"2024-03-06T07:27:09.370655+0000","last_became_peered":"2024-03-06T07:27:09.370655+0000","last_unstale":"2024-03-06T07:27:11.941874+0000","last_undegraded":"2024-03-06T07:27:11.941874+0000","last_fullsized":"2024-03-06T07:27:11.941874+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:14.629790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397293+0000","last_change":"2024-03-06T07:27:10.407006+0000","last_active":"2024-03-06T07:27:11.397293+0000","last_peered":"2024-03-06T07:27:11.397293+0000","last_clean":"2024-03-06T07:27:11.397293+0000","last_became_active":"2024-03-06T07:27:10.406718+0000","last_became_peered":"2024-03-06T07:27:10.406718+0000","last_unstale":"2024-03-06T07:27:11.397293+0000","last_undegraded":"2024-03-06T07:27:11.397293+0000","last_fullsized":"2024-03-06T07:27:11.397293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:25.004410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"83'8","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388057+0000","last_change":"2024-03-06T07:27:09.368550+0000","last_active":"2024-03-06T07:27:11.388057+0000","last_peered":"2024-03-06T07:27:11.388057+0000","last_clean":"2024-03-06T07:27:11.388057+0000","last_became_active":"2024-03-06T07:27:09.367237+0000","last_became_peered":"2024-03-06T07:27:09.367237+0000","last_unstale":"2024-03-06T07:27:11.388057+0000","last_undegraded":"2024-03-06T07:27:11.388057+0000","last_fullsized":"2024-03-06T07:27:11.388057+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:20.810183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389365+0000","last_change":"2024-03-06T07:27:10.386144+0000","last_active":"2024-03-06T07:27:11.389365+0000","last_peered":"2024-03-06T07:27:11.389365+0000","last_clean":"2024-03-06T07:27:11.389365+0000","last_became_active":"2024-03-06T07:27:10.385864+0000","last_became_peered":"2024-03-06T07:27:10.385864+0000","last_unstale":"2024-03-06T07:27:11.389365+0000","last_undegraded":"2024-03-06T07:27:11.389365+0000","last_fullsized":"2024-03-06T07:27:11.389365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:17.601786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942363+0000","last_change":"2024-03-06T07:27:09.367473+0000","last_active":"2024-03-06T07:27:11.942363+0000","last_peered":"2024-03-06T07:27:11.942363+0000","last_clean":"2024-03-06T07:27:11.942363+0000","last_became_active":"2024-03-06T07:27:09.367201+0000","last_became_peered":"2024-03-06T07:27:09.367201+0000","last_unstale":"2024-03-06T07:27:11.942363+0000","last_undegraded":"2024-03-06T07:27:11.942363+0000","last_fullsized":"2024-03-06T07:27:11.942363+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:08.840813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942247+0000","last_change":"2024-03-06T07:27:10.393522+0000","last_active":"2024-03-06T07:27:11.942247+0000","last_peered":"2024-03-06T07:27:11.942247+0000","last_clean":"2024-03-06T07:27:11.942247+0000","last_became_active":"2024-03-06T07:27:10.393242+0000","last_became_peered":"2024-03-06T07:27:10.393242+0000","last_unstale":"2024-03-06T07:27:11.942247+0000","last_undegraded":"2024-03-06T07:27:11.942247+0000","last_fullsized":"2024-03-06T07:27:11.942247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:09.153697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388487+0000","last_change":"2024-03-06T07:27:09.373607+0000","last_active":"2024-03-06T07:27:11.388487+0000","last_peered":"2024-03-06T07:27:11.388487+0000","last_clean":"2024-03-06T07:27:11.388487+0000","last_became_active":"2024-03-06T07:27:09.373306+0000","last_became_peered":"2024-03-06T07:27:09.373306+0000","last_unstale":"2024-03-06T07:27:11.388487+0000","last_undegraded":"2024-03-06T07:27:11.388487+0000","last_fullsized":"2024-03-06T07:27:11.388487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:03.605759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389870+0000","last_change":"2024-03-06T07:27:10.394863+0000","last_active":"2024-03-06T07:27:11.389870+0000","last_peered":"2024-03-06T07:27:11.389870+0000","last_clean":"2024-03-06T07:27:11.389870+0000","last_became_active":"2024-03-06T07:27:10.394375+0000","last_became_peered":"2024-03-06T07:27:10.394375+0000","last_unstale":"2024-03-06T07:27:11.389870+0000","last_undegraded":"2024-03-06T07:27:11.389870+0000","last_fullsized":"2024-03-06T07:27:11.389870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:51.286328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388608+0000","last_change":"2024-03-06T07:27:09.367584+0000","last_active":"2024-03-06T07:27:11.388608+0000","last_peered":"2024-03-06T07:27:11.388608+0000","last_clean":"2024-03-06T07:27:11.388608+0000","last_became_active":"2024-03-06T07:27:09.366434+0000","last_became_peered":"2024-03-06T07:27:09.366434+0000","last_unstale":"2024-03-06T07:27:11.388608+0000","last_undegraded":"2024-03-06T07:27:11.388608+0000","last_fullsized":"2024-03-06T07:27:11.388608+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:43.080196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941018+0000","last_change":"2024-03-06T07:27:10.398783+0000","last_active":"2024-03-06T07:27:11.941018+0000","last_peered":"2024-03-06T07:27:11.941018+0000","last_clean":"2024-03-06T07:27:11.941018+0000","last_became_active":"2024-03-06T07:27:10.398490+0000","last_became_peered":"2024-03-06T07:27:10.398490+0000","last_unstale":"2024-03-06T07:27:11.941018+0000","last_undegraded":"2024-03-06T07:27:11.941018+0000","last_fullsized":"2024-03-06T07:27:11.941018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:09.500962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.704098+0000","last_change":"2024-03-06T07:27:09.373174+0000","last_active":"2024-03-06T07:27:11.704098+0000","last_peered":"2024-03-06T07:27:11.704098+0000","last_clean":"2024-03-06T07:27:11.704098+0000","last_became_active":"2024-03-06T07:27:09.372894+0000","last_became_peered":"2024-03-06T07:27:09.372894+0000","last_unstale":"2024-03-06T07:27:11.704098+0000","last_undegraded":"2024-03-06T07:27:11.704098+0000","last_fullsized":"2024-03-06T07:27:11.704098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:05.954280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941589+0000","last_change":"2024-03-06T07:27:10.389981+0000","last_active":"2024-03-06T07:27:11.941589+0000","last_peered":"2024-03-06T07:27:11.941589+0000","last_clean":"2024-03-06T07:27:11.941589+0000","last_became_active":"2024-03-06T07:27:10.389714+0000","last_became_peered":"2024-03-06T07:27:10.389714+0000","last_unstale":"2024-03-06T07:27:11.941589+0000","last_undegraded":"2024-03-06T07:27:11.941589+0000","last_fullsized":"2024-03-06T07:27:11.941589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:51.286328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941113+0000","last_change":"2024-03-06T07:27:09.374733+0000","last_active":"2024-03-06T07:27:11.941113+0000","last_peered":"2024-03-06T07:27:11.941113+0000","last_clean":"2024-03-06T07:27:11.941113+0000","last_became_active":"2024-03-06T07:27:09.374380+0000","last_became_peered":"2024-03-06T07:27:09.374380+0000","last_unstale":"2024-03-06T07:27:11.941113+0000","last_undegraded":"2024-03-06T07:27:11.941113+0000","last_fullsized":"2024-03-06T07:27:11.941113+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:09.153513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.940628+0000","last_change":"2024-03-06T07:27:10.397466+0000","last_active":"2024-03-06T07:27:11.940628+0000","last_peered":"2024-03-06T07:27:11.940628+0000","last_clean":"2024-03-06T07:27:11.940628+0000","last_became_active":"2024-03-06T07:27:10.396908+0000","last_became_peered":"2024-03-06T07:27:10.396908+0000","last_unstale":"2024-03-06T07:27:11.940628+0000","last_undegraded":"2024-03-06T07:27:11.940628+0000","last_fullsized":"2024-03-06T07:27:11.940628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:36.253150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.940611+0000","last_change":"2024-03-06T07:27:09.375768+0000","last_active":"2024-03-06T07:27:11.940611+0000","last_peered":"2024-03-06T07:27:11.940611+0000","last_clean":"2024-03-06T07:27:11.940611+0000","last_became_active":"2024-03-06T07:27:09.375512+0000","last_became_peered":"2024-03-06T07:27:09.375512+0000","last_unstale":"2024-03-06T07:27:11.940611+0000","last_undegraded":"2024-03-06T07:27:11.940611+0000","last_fullsized":"2024-03-06T07:27:11.940611+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:41.025662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941768+0000","last_change":"2024-03-06T07:27:10.396593+0000","last_active":"2024-03-06T07:27:11.941768+0000","last_peered":"2024-03-06T07:27:11.941768+0000","last_clean":"2024-03-06T07:27:11.941768+0000","last_became_active":"2024-03-06T07:27:10.396322+0000","last_became_peered":"2024-03-06T07:27:10.396322+0000","last_unstale":"2024-03-06T07:27:11.941768+0000","last_undegraded":"2024-03-06T07:27:11.941768+0000","last_fullsized":"2024-03-06T07:27:11.941768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:08.487015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.703882+0000","last_change":"2024-03-06T07:27:09.369058+0000","last_active":"2024-03-06T07:27:11.703882+0000","last_peered":"2024-03-06T07:27:11.703882+0000","last_clean":"2024-03-06T07:27:11.703882+0000","last_became_active":"2024-03-06T07:27:09.368762+0000","last_became_peered":"2024-03-06T07:27:09.368762+0000","last_unstale":"2024-03-06T07:27:11.703882+0000","last_undegraded":"2024-03-06T07:27:11.703882+0000","last_fullsized":"2024-03-06T07:27:11.703882+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:34.976254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.390134+0000","last_change":"2024-03-06T07:27:10.392951+0000","last_active":"2024-03-06T07:27:11.390134+0000","last_peered":"2024-03-06T07:27:11.390134+0000","last_clean":"2024-03-06T07:27:11.390134+0000","last_became_active":"2024-03-06T07:27:10.392357+0000","last_became_peered":"2024-03-06T07:27:10.392357+0000","last_unstale":"2024-03-06T07:27:11.390134+0000","last_undegraded":"2024-03-06T07:27:11.390134+0000","last_fullsized":"2024-03-06T07:27:11.390134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:14.134642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397518+0000","last_change":"2024-03-06T07:27:09.377310+0000","last_active":"2024-03-06T07:27:11.397518+0000","last_peered":"2024-03-06T07:27:11.397518+0000","last_clean":"2024-03-06T07:27:11.397518+0000","last_became_active":"2024-03-06T07:27:09.377048+0000","last_became_peered":"2024-03-06T07:27:09.377048+0000","last_unstale":"2024-03-06T07:27:11.397518+0000","last_undegraded":"2024-03-06T07:27:11.397518+0000","last_fullsized":"2024-03-06T07:27:11.397518+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:34.870654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389528+0000","last_change":"2024-03-06T07:27:10.391311+0000","last_active":"2024-03-06T07:27:11.389528+0000","last_peered":"2024-03-06T07:27:11.389528+0000","last_clean":"2024-03-06T07:27:11.389528+0000","last_became_active":"2024-03-06T07:27:10.390796+0000","last_became_peered":"2024-03-06T07:27:10.390796+0000","last_unstale":"2024-03-06T07:27:11.389528+0000","last_undegraded":"2024-03-06T07:27:11.389528+0000","last_fullsized":"2024-03-06T07:27:11.389528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:17.311840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"84'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.397815+0000","last_change":"2024-03-06T07:27:09.380149+0000","last_active":"2024-03-06T07:27:11.397815+0000","last_peered":"2024-03-06T07:27:11.397815+0000","last_clean":"2024-03-06T07:27:11.397815+0000","last_became_active":"2024-03-06T07:27:09.379853+0000","last_became_peered":"2024-03-06T07:27:09.379853+0000","last_unstale":"2024-03-06T07:27:11.397815+0000","last_undegraded":"2024-03-06T07:27:11.397815+0000","last_fullsized":"2024-03-06T07:27:11.397815+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:13.602002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941830+0000","last_change":"2024-03-06T07:27:10.409860+0000","last_active":"2024-03-06T07:27:11.941830+0000","last_peered":"2024-03-06T07:27:11.941830+0000","last_clean":"2024-03-06T07:27:11.941830+0000","last_became_active":"2024-03-06T07:27:10.409587+0000","last_became_peered":"2024-03-06T07:27:10.409587+0000","last_unstale":"2024-03-06T07:27:11.941830+0000","last_undegraded":"2024-03-06T07:27:11.941830+0000","last_fullsized":"2024-03-06T07:27:11.941830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:14.134642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942002+0000","last_change":"2024-03-06T07:27:09.375900+0000","last_active":"2024-03-06T07:27:11.942002+0000","last_peered":"2024-03-06T07:27:11.942002+0000","last_clean":"2024-03-06T07:27:11.942002+0000","last_became_active":"2024-03-06T07:27:09.375583+0000","last_became_peered":"2024-03-06T07:27:09.375583+0000","last_unstale":"2024-03-06T07:27:11.942002+0000","last_undegraded":"2024-03-06T07:27:11.942002+0000","last_fullsized":"2024-03-06T07:27:11.942002+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:53.589572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389892+0000","last_change":"2024-03-06T07:27:10.396161+0000","last_active":"2024-03-06T07:27:11.389892+0000","last_peered":"2024-03-06T07:27:11.389892+0000","last_clean":"2024-03-06T07:27:11.389892+0000","last_became_active":"2024-03-06T07:27:10.395835+0000","last_became_peered":"2024-03-06T07:27:10.395835+0000","last_unstale":"2024-03-06T07:27:11.389892+0000","last_undegraded":"2024-03-06T07:27:11.389892+0000","last_fullsized":"2024-03-06T07:27:11.389892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:04.763040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.942123+0000","last_change":"2024-03-06T07:27:09.367702+0000","last_active":"2024-03-06T07:27:11.942123+0000","last_peered":"2024-03-06T07:27:11.942123+0000","last_clean":"2024-03-06T07:27:11.942123+0000","last_became_active":"2024-03-06T07:27:09.367456+0000","last_became_peered":"2024-03-06T07:27:09.367456+0000","last_unstale":"2024-03-06T07:27:11.942123+0000","last_undegraded":"2024-03-06T07:27:11.942123+0000","last_fullsized":"2024-03-06T07:27:11.942123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:35.367786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941881+0000","last_change":"2024-03-06T07:27:10.390636+0000","last_active":"2024-03-06T07:27:11.941881+0000","last_peered":"2024-03-06T07:27:11.941881+0000","last_clean":"2024-03-06T07:27:11.941881+0000","last_became_active":"2024-03-06T07:27:10.390385+0000","last_became_peered":"2024-03-06T07:27:10.390385+0000","last_unstale":"2024-03-06T07:27:11.941881+0000","last_undegraded":"2024-03-06T07:27:11.941881+0000","last_fullsized":"2024-03-06T07:27:11.941881+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:14.134642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388497+0000","last_change":"2024-03-06T07:27:09.367998+0000","last_active":"2024-03-06T07:27:11.388497+0000","last_peered":"2024-03-06T07:27:11.388497+0000","last_clean":"2024-03-06T07:27:11.388497+0000","last_became_active":"2024-03-06T07:27:09.367731+0000","last_became_peered":"2024-03-06T07:27:09.367731+0000","last_unstale":"2024-03-06T07:27:11.388497+0000","last_undegraded":"2024-03-06T07:27:11.388497+0000","last_fullsized":"2024-03-06T07:27:11.388497+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:50.277092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388507+0000","last_change":"2024-03-06T07:27:10.392119+0000","last_active":"2024-03-06T07:27:11.388507+0000","last_peered":"2024-03-06T07:27:11.388507+0000","last_clean":"2024-03-06T07:27:11.388507+0000","last_became_active":"2024-03-06T07:27:10.391783+0000","last_became_peered":"2024-03-06T07:27:10.391783+0000","last_unstale":"2024-03-06T07:27:11.388507+0000","last_undegraded":"2024-03-06T07:27:11.388507+0000","last_fullsized":"2024-03-06T07:27:11.388507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:09.284951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389600+0000","last_change":"2024-03-06T07:27:09.374468+0000","last_active":"2024-03-06T07:27:11.389600+0000","last_peered":"2024-03-06T07:27:11.389600+0000","last_clean":"2024-03-06T07:27:11.389600+0000","last_became_active":"2024-03-06T07:27:09.374183+0000","last_became_peered":"2024-03-06T07:27:09.374183+0000","last_unstale":"2024-03-06T07:27:11.389600+0000","last_undegraded":"2024-03-06T07:27:11.389600+0000","last_fullsized":"2024-03-06T07:27:11.389600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:20.907611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388788+0000","last_change":"2024-03-06T07:27:10.389149+0000","last_active":"2024-03-06T07:27:11.388788+0000","last_peered":"2024-03-06T07:27:11.388788+0000","last_clean":"2024-03-06T07:27:11.388788+0000","last_became_active":"2024-03-06T07:27:10.388845+0000","last_became_peered":"2024-03-06T07:27:10.388845+0000","last_unstale":"2024-03-06T07:27:11.388788+0000","last_undegraded":"2024-03-06T07:27:11.388788+0000","last_fullsized":"2024-03-06T07:27:11.388788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:06.963517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941816+0000","last_change":"2024-03-06T07:27:09.368796+0000","last_active":"2024-03-06T07:27:11.941816+0000","last_peered":"2024-03-06T07:27:11.941816+0000","last_clean":"2024-03-06T07:27:11.941816+0000","last_became_active":"2024-03-06T07:27:09.368017+0000","last_became_peered":"2024-03-06T07:27:09.368017+0000","last_unstale":"2024-03-06T07:27:11.941816+0000","last_undegraded":"2024-03-06T07:27:11.941816+0000","last_fullsized":"2024-03-06T07:27:11.941816+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:08.447336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.941718+0000","last_change":"2024-03-06T07:27:10.409155+0000","last_active":"2024-03-06T07:27:11.941718+0000","last_peered":"2024-03-06T07:27:11.941718+0000","last_clean":"2024-03-06T07:27:11.941718+0000","last_became_active":"2024-03-06T07:27:10.408809+0000","last_became_peered":"2024-03-06T07:27:10.408809+0000","last_unstale":"2024-03-06T07:27:11.941718+0000","last_undegraded":"2024-03-06T07:27:11.941718+0000","last_fullsized":"2024-03-06T07:27:11.941718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:09.014322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389567+0000","last_change":"2024-03-06T07:27:09.373156+0000","last_active":"2024-03-06T07:27:11.389567+0000","last_peered":"2024-03-06T07:27:11.389567+0000","last_clean":"2024-03-06T07:27:11.389567+0000","last_became_active":"2024-03-06T07:27:09.372848+0000","last_became_peered":"2024-03-06T07:27:09.372848+0000","last_unstale":"2024-03-06T07:27:11.389567+0000","last_undegraded":"2024-03-06T07:27:11.389567+0000","last_fullsized":"2024-03-06T07:27:11.389567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:08.447336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.398068+0000","last_change":"2024-03-06T07:27:10.390235+0000","last_active":"2024-03-06T07:27:11.398068+0000","last_peered":"2024-03-06T07:27:11.398068+0000","last_clean":"2024-03-06T07:27:11.398068+0000","last_became_active":"2024-03-06T07:27:10.389895+0000","last_became_peered":"2024-03-06T07:27:10.389895+0000","last_unstale":"2024-03-06T07:27:11.398068+0000","last_undegraded":"2024-03-06T07:27:11.398068+0000","last_fullsized":"2024-03-06T07:27:11.398068+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:09.352918+0000","last_clean_scrub_stamp":"2024-03-06T07:27:09.352918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:04.371972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.389148+0000","last_change":"2024-03-06T07:27:09.371787+0000","last_active":"2024-03-06T07:27:11.389148+0000","last_peered":"2024-03-06T07:27:11.389148+0000","last_clean":"2024-03-06T07:27:11.389148+0000","last_became_active":"2024-03-06T07:27:09.371100+0000","last_became_peered":"2024-03-06T07:27:09.371100+0000","last_unstale":"2024-03-06T07:27:11.389148+0000","last_undegraded":"2024-03-06T07:27:11.389148+0000","last_fullsized":"2024-03-06T07:27:11.389148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:08.343681+0000","last_clean_scrub_stamp":"2024-03-06T07:27:08.343681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:07.477778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":292,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-06T07:27:11.388904+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:27:11.388904+0000","last_peered":"2024-03-06T07:27:11.388904+0000","last_clean":"2024-03-06T07:27:11.388904+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:27:11.388904+0000","last_undegraded":"2024-03-06T07:27:11.388904+0000","last_fullsized":"2024-03-06T07:27:11.388904+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1630208,"data_stored":1027927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1044480,"data_stored":438183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339632,"statfs":{"total":96636764160,"available":96603783168,"internally_reserved":0,"allocated":1048576,"data_stored":439123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1032192,"data_stored":437489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1044480,"data_stored":439101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1036288,"data_stored":438071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1044480,"data_stored":437683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1630208,"data_stored":1027891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:19.279 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:19.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:19.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.426+0000 7f5424fbf640 1 Processor -- start 2024-03-06T07:27:19.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.426+0000 7f5424fbf640 1 -- start start 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.428+0000 7f5424fbf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f54200d2d90 0x7f54200ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.428+0000 7f5424fbf640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542015c210 con 0x7f54200cf4d0 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.428+0000 7f5424fbf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542015ed30 con 0x7f542015be30 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.428+0000 7f5424fbf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54200ceb70 con 0x7f54200d2d90 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.428+0000 7f541dd9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f54200cf4d0 0x7f54200b27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43330/0 (socket says 172.21.15.27:43330) 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.428+0000 7f541dd9b640 1 -- 172.21.15.27:0/3903236132 learned_addr learned my addr 172.21.15.27:0/3903236132 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.428+0000 7f541ed9d640 1 --2- 172.21.15.27:0/3903236132 >> v2:172.21.15.69:3300/0 conn(0x7f54200d2d90 0x7f54200ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.428+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204415780 0 0) 0x7f542015c210 con 0x7f54200cf4d0 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.428+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5408003520 con 0x7f54200cf4d0 2024-03-06T07:27:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1778690236 0 0) 0x7f5408003520 con 0x7f54200cf4d0 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 >> v2:172.21.15.69:3300/0 conn(0x7f54200d2d90 msgr2=0x7f54200ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 --2- 172.21.15.27:0/3903236132 >> v2:172.21.15.69:3300/0 conn(0x7f54200d2d90 0x7f54200ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 >> v1:172.21.15.69:6789/0 conn(0x7f542015be30 legacy=0x7f54200cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542015f010 con 0x7f54200cf4d0 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f5410001670 con 0x7f54200cf4d0 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 >> v1:172.21.15.27:6789/0 conn(0x7f54200cf4d0 legacy=0x7f54200b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54080039b0 0x7f5408003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5408004800 0x7f5408004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54200ceb70 con 0x7f5408004800 2024-03-06T07:27:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540800c160 con 0x7f54080051c0 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540800c360 con 0x7f54080039b0 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541dd9b640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5408004800 0x7f5408004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541e59c640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54080039b0 0x7f5408003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541ed9d640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541dd9b640 1 -- 172.21.15.27:0/3903236132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54080039b0 msgr2=0x7f5408003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541dd9b640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f54080039b0 0x7f5408003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541dd9b640 1 -- 172.21.15.27:0/3903236132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54080051c0 msgr2=0x7f540800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541dd9b640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541dd9b640 1 -- 172.21.15.27:0/3903236132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5408003520 con 0x7f5408004800 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541ed9d640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.429+0000 7f541ed9d640 1 -- 172.21.15.27:0/3903236132 reap_dead start 2024-03-06T07:27:19.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.430+0000 7f541dd9b640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5408004800 0x7f5408004c80 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f541000c210 tx=0x7f5408005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c12d3bb1fcb39af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.430+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410007d30 con 0x7f5408004800 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.430+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410030070 con 0x7f5408004800 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.430+0000 7f541d59a640 1 -- 172.21.15.27:0/3903236132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410005d70 con 0x7f5408004800 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.431+0000 7f5424fbf640 1 -- 172.21.15.27:0/3903236132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5408004800 msgr2=0x7f5408004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.431+0000 7f5424fbf640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5408004800 0x7f5408004c80 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f541000c210 tx=0x7f5408005830 comp rx=0 tx=0).stop 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.431+0000 7f5424fbf640 1 -- 172.21.15.27:0/3903236132 shutdown_connections 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.431+0000 7f5424fbf640 1 --2- 172.21.15.27:0/3903236132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5408004800 0x7f5408004c80 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.431+0000 7f5424fbf640 1 -- 172.21.15.27:0/3903236132 >> 172.21.15.27:0/3903236132 conn(0x7f54200d3d60 msgr2=0x7f54200d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.431+0000 7f5424fbf640 1 -- 172.21.15.27:0/3903236132 shutdown_connections 2024-03-06T07:27:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.431+0000 7f5424fbf640 1 -- 172.21.15.27:0/3903236132 wait complete. 2024-03-06T07:27:19.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.432+0000 7f5424fbf640 1 Processor -- start 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.432+0000 7f5424fbf640 1 -- start start 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f5424fbf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5408004800 0x7f54200cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f5424fbf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54200b8dd0 0x7f54200b74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f5424fbf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5420159060 0x7f5420149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f5424fbf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542015f5d0 con 0x7f54200b8dd0 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f5424fbf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54200728a0 con 0x7f5408004800 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f5424fbf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542015e7c0 con 0x7f5420159060 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54200b8dd0 0x7f54200b74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54200b8dd0 0x7f54200b74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34368/0 (socket says 172.21.15.27:34368) 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541dd9b640 1 -- 172.21.15.27:0/2363808178 learned_addr learned my addr 172.21.15.27:0/2363808178 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541e59c640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5408004800 0x7f54200cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541ed9d640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5420159060 0x7f5420149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541dd9b640 1 -- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5420159060 msgr2=0x7f5420149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541dd9b640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5420159060 0x7f5420149dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541dd9b640 1 -- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5408004800 msgr2=0x7f54200cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541dd9b640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5408004800 0x7f54200cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541dd9b640 1 -- 172.21.15.27:0/2363808178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200b7d30 con 0x7f54200b8dd0 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541e59c640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5408004800 0x7f54200cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.433+0000 7f541ed9d640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5420159060 0x7f5420149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.434+0000 7f541dd9b640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54200b8dd0 0x7f54200b74e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f541000c210 tx=0x7f541002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.434+0000 7f540effd640 1 -- 172.21.15.27:0/2363808178 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410046070 con 0x7f54200b8dd0 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.434+0000 7f540effd640 1 -- 172.21.15.27:0/2363808178 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410004bf0 con 0x7f54200b8dd0 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.434+0000 7f540effd640 1 -- 172.21.15.27:0/2363808178 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410007d10 con 0x7f54200b8dd0 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.434+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54200b8010 con 0x7f54200b8dd0 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.434+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54200b88b0 con 0x7f54200b8dd0 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.435+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54200d05c0 con 0x7f54200b8dd0 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.435+0000 7f540effd640 1 -- 172.21.15.27:0/2363808178 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5410007070 con 0x7f54200b8dd0 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.436+0000 7f540effd640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f540006b4d0 0x7f540006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.436+0000 7f540effd640 1 -- 172.21.15.27:0/2363808178 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54100ab5f0 con 0x7f54200b8dd0 2024-03-06T07:27:19.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.436+0000 7f541e59c640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f540006b4d0 0x7f540006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.437+0000 7f541e59c640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f540006b4d0 0x7f540006d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f54140036b0 tx=0x7f5414009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.440+0000 7f540effd640 1 -- 172.21.15.27:0/2363808178 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f541007a3e0 con 0x7f54200b8dd0 2024-03-06T07:27:19.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.819+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f54200b81e0 con 0x7f54200b8dd0 2024-03-06T07:27:19.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.823+0000 7f540effd640 1 -- 172.21.15.27:0/2363808178 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5410079ce0 con 0x7f54200b8dd0 2024-03-06T07:27:19.825 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:27:19.825 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:27:19.825 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-03-06T07:27:19.825 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-03-06T07:27:19.825 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:19.826 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-03-06T07:27:19.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f540006b4d0 msgr2=0x7f540006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f540006b4d0 0x7f540006d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f54140036b0 tx=0x7f5414009040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54200b8dd0 msgr2=0x7f54200b74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54200b8dd0 0x7f54200b74e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f541000c210 tx=0x7f541002e070 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 shutdown_connections 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f540006b4d0 0x7f540006d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5420159060 0x7f5420149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54200b8dd0 0x7f54200b74e0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 --2- 172.21.15.27:0/2363808178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5408004800 0x7f54200cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 >> 172.21.15.27:0/2363808178 conn(0x7f54200d3d60 msgr2=0x7f54200d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 shutdown_connections 2024-03-06T07:27:19.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.825+0000 7f5424fbf640 1 -- 172.21.15.27:0/2363808178 wait complete. 2024-03-06T07:27:19.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:19.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.943+0000 7f5f25df6640 1 Processor -- start 2024-03-06T07:27:19.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.943+0000 7f5f25df6640 1 -- start start 2024-03-06T07:27:19.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.944+0000 7f5f25df6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f2015df00 0x7f5f200d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.944+0000 7f5f25df6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f200cea30 con 0x7f5f200b9a30 2024-03-06T07:27:19.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.944+0000 7f5f25df6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f200d2af0 con 0x7f5f200b95a0 2024-03-06T07:27:19.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.944+0000 7f5f25df6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200cf3c0 con 0x7f5f2015df00 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.944+0000 7f5f1ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5f200b9a30 0x7f5f200d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43340/0 (socket says 172.21.15.27:43340) 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.944+0000 7f5f1ffff640 1 -- 172.21.15.27:0/1967136704 learned_addr learned my addr 172.21.15.27:0/1967136704 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.944+0000 7f5f1f7fe640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f2015df00 0x7f5f200d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.944+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890455115 0 0) 0x7f5f200cea30 con 0x7f5f200b9a30 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.944+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f08003520 con 0x7f5f200b9a30 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671918458 0 0) 0x7f5f200d2af0 con 0x7f5f200b95a0 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f200cea30 con 0x7f5f200b95a0 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2182135039 0 0) 0x7f5f08003520 con 0x7f5f200b9a30 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f2015df00 msgr2=0x7f5f200d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f2015df00 0x7f5f200d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 >> v1:172.21.15.69:6789/0 conn(0x7f5f200b95a0 legacy=0x7f5f200b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f200cf0e0 con 0x7f5f200b9a30 2024-03-06T07:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f5f180019f0 con 0x7f5f200b9a30 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 >> v1:172.21.15.27:6789/0 conn(0x7f5f200b9a30 legacy=0x7f5f200d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f2015df00 0x7f5f08003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f080040d0 0x7f5f08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f08004d40 0x7f5f080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200cf3c0 con 0x7f5f08004d40 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0800baa0 con 0x7f5f080040d0 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0800bca0 con 0x7f5f2015df00 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1ffff640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f08004d40 0x7f5f080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1f7fe640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f2015df00 0x7f5f08003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1effd640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f080040d0 0x7f5f08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1ffff640 1 -- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f2015df00 msgr2=0x7f5f08003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1ffff640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f2015df00 0x7f5f08003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1ffff640 1 -- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f080040d0 msgr2=0x7f5f08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1ffff640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f080040d0 0x7f5f08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.945+0000 7f5f1ffff640 1 -- 172.21.15.27:0/1967136704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f08003520 con 0x7f5f08004d40 2024-03-06T07:27:19.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f1ffff640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f08004d40 0x7f5f080051c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800a7d0 tx=0x7f5f08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44038c6cf9e3aadf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1800a430 con 0x7f5f08004d40 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f18034630 con 0x7f5f08004d40 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f1e7fc640 1 -- 172.21.15.27:0/1967136704 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18034a60 con 0x7f5f08004d40 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f25df6640 1 -- 172.21.15.27:0/1967136704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f08004d40 msgr2=0x7f5f080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f25df6640 1 --2- 172.21.15.27:0/1967136704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f08004d40 0x7f5f080051c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800a7d0 tx=0x7f5f08005830 comp rx=0 tx=0).stop 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f1ffff640 1 -- 172.21.15.27:0/1967136704 reap_dead start 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f25df6640 1 -- 172.21.15.27:0/1967136704 shutdown_connections 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f25df6640 1 -- 172.21.15.27:0/1967136704 >> 172.21.15.27:0/1967136704 conn(0x7f5f20071960 msgr2=0x7f5f20071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f25df6640 1 -- 172.21.15.27:0/1967136704 shutdown_connections 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.947+0000 7f5f25df6640 1 -- 172.21.15.27:0/1967136704 wait complete. 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.948+0000 7f5f25df6640 1 Processor -- start 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f25df6640 1 -- start start 2024-03-06T07:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f25df6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f20155530 0x7f5f200d3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f25df6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f2015f1d0 0x7f5f200d3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f25df6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f2015f8d0 0x7f5f2015fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f25df6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200cec70 con 0x7f5f2015f1d0 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f25df6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200ce750 con 0x7f5f2015f8d0 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f25df6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200cedf0 con 0x7f5f20155530 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f2015f1d0 0x7f5f200d3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f1effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f2015f1d0 0x7f5f200d3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34380/0 (socket says 172.21.15.27:34380) 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.949+0000 7f5f1effd640 1 -- 172.21.15.27:0/1012629790 learned_addr learned my addr 172.21.15.27:0/1012629790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f1f7fe640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f20155530 0x7f5f200d3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f1ffff640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f2015f8d0 0x7f5f2015fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f1effd640 1 -- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f20155530 msgr2=0x7f5f200d3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f1effd640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f20155530 0x7f5f200d3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f1effd640 1 -- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f2015f8d0 msgr2=0x7f5f2015fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f1effd640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f2015f8d0 0x7f5f2015fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f1effd640 1 -- 172.21.15.27:0/1012629790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2015e410 con 0x7f5f2015f1d0 2024-03-06T07:27:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f1effd640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f2015f1d0 0x7f5f200d3db0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f5f1400f770 tx=0x7f5f1400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f07fff640 1 -- 172.21.15.27:0/1012629790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1400fe80 con 0x7f5f2015f1d0 2024-03-06T07:27:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f07fff640 1 -- 172.21.15.27:0/1012629790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f14018070 con 0x7f5f2015f1d0 2024-03-06T07:27:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2015e6f0 con 0x7f5f2015f1d0 2024-03-06T07:27:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f07fff640 1 -- 172.21.15.27:0/1012629790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1400d6d0 con 0x7f5f2015f1d0 2024-03-06T07:27:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.950+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2015ecb0 con 0x7f5f2015f1d0 2024-03-06T07:27:19.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.951+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2015eeb0 con 0x7f5f2015f1d0 2024-03-06T07:27:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.952+0000 7f5f07fff640 1 -- 172.21.15.27:0/1012629790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f1400d870 con 0x7f5f2015f1d0 2024-03-06T07:27:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.952+0000 7f5f07fff640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f0c06b4d0 0x7f5f0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.952+0000 7f5f07fff640 1 -- 172.21.15.27:0/1012629790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f1408a140 con 0x7f5f2015f1d0 2024-03-06T07:27:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.952+0000 7f5f1f7fe640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f0c06b4d0 0x7f5f0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.953+0000 7f5f1f7fe640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f0c06b4d0 0x7f5f0c06d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009b00 tx=0x7f5f10008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:19.961+0000 7f5f07fff640 1 -- 172.21.15.27:0/1012629790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f2015eeb0 con 0x7f5f2015f1d0 2024-03-06T07:27:20.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.181+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f20067ff0 con 0x7f5f2015f1d0 2024-03-06T07:27:20.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.182+0000 7f5f07fff640 1 -- 172.21.15.27:0/1012629790 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f5f1401a310 con 0x7f5f2015f1d0 2024-03-06T07:27:20.185 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:20.185 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/930650799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":930650799},{"type":"v1","addr":"172.21.15.69:6836","nonce":930650799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1050755919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1050755919},{"type":"v1","addr":"172.21.15.27:6835","nonce":1050755919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1173605349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1173605349},{"type":"v1","addr":"172.21.15.69:6837","nonce":1173605349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:09.729142+0000","modified":"2024-03-06T07:27:12.757645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7032},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7032":{"gid":7032,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3587849188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3587849188},{"type":"v1","addr":"172.21.15.27:6834","nonce":3587849188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7032,"qdb_cluster":[7032]},"id":6}]} 2024-03-06T07:27:20.185 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 52 2024-03-06T07:27:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f0c06b4d0 msgr2=0x7f5f0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f0c06b4d0 0x7f5f0c06d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009b00 tx=0x7f5f10008040 comp rx=0 tx=0).stop 2024-03-06T07:27:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f2015f1d0 msgr2=0x7f5f200d3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f2015f1d0 0x7f5f200d3db0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f5f1400f770 tx=0x7f5f1400fc30 comp rx=0 tx=0).stop 2024-03-06T07:27:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 shutdown_connections 2024-03-06T07:27:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f0c06b4d0 0x7f5f0c06d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f2015f8d0 0x7f5f2015fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f2015f1d0 0x7f5f200d3db0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 --2- 172.21.15.27:0/1012629790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f20155530 0x7f5f200d3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 >> 172.21.15.27:0/1012629790 conn(0x7f5f20071960 msgr2=0x7f5f200d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.184+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 shutdown_connections 2024-03-06T07:27:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.185+0000 7f5f25df6640 1 -- 172.21.15.27:0/1012629790 wait complete. 2024-03-06T07:27:20.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:09.729142+0000', 'modified': '2024-03-06T07:27:12.757645+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7032}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7032': {'gid': 7032, 'name': 'a', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/3587849188', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3587849188}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 3587849188}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7032, 'qdb_cluster': [7032]} 2024-03-06T07:27:20.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:20.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:27:20.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.706+0000 7f110569a640 1 Processor -- start 2024-03-06T07:27:20.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.706+0000 7f110569a640 1 -- start start 2024-03-06T07:27:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f110569a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f110006b5c0 0x7f11000b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f110569a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f11000ae9b0 0x7f11000d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f110569a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11000b3340 con 0x7f11000b0e90 2024-03-06T07:27:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f110569a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11000cfa00 con 0x7f110006b5c0 2024-03-06T07:27:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f110569a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11000cee90 con 0x7f11000ae9b0 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f10feffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f110006b5c0 0x7f11000b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f10fe7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f11000b0e90 0x7f110006b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43346/0 (socket says 172.21.15.27:43346) 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f10fe7fc640 1 -- 172.21.15.27:0/2013029050 learned_addr learned my addr 172.21.15.27:0/2013029050 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f10feffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f110006b5c0 0x7f11000b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34392/0 (socket says 172.21.15.27:34392) 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/2013029050 >> v2:172.21.15.69:3300/0 conn(0x7f11000ae9b0 0x7f11000d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f10feffd640 1 -- 172.21.15.27:0/2013029050 >> v2:172.21.15.69:3300/0 conn(0x7f11000ae9b0 msgr2=0x7f11000d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f10feffd640 1 --2- 172.21.15.27:0/2013029050 >> v2:172.21.15.69:3300/0 conn(0x7f11000ae9b0 0x7f11000d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f10feffd640 1 -- 172.21.15.27:0/2013029050 >> v1:172.21.15.27:6789/0 conn(0x7f11000b0e90 legacy=0x7f110006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.707+0000 7f10feffd640 1 -- 172.21.15.27:0/2013029050 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11000cfc50 con 0x7f110006b5c0 2024-03-06T07:27:20.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.708+0000 7f10feffd640 1 --2- 172.21.15.27:0/2013029050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 0x7f11000b2e00 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f1100072170 tx=0x7f10f402b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72ebdf75ebc5fba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.708+0000 7f10fdffb640 1 -- 172.21.15.27:0/2013029050 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4037070 con 0x7f110006b5c0 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.708+0000 7f10fdffb640 1 -- 172.21.15.27:0/2013029050 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f4030710 con 0x7f110006b5c0 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.708+0000 7f10fdffb640 1 -- 172.21.15.27:0/2013029050 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4030ae0 con 0x7f110006b5c0 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.709+0000 7f110569a640 1 -- 172.21.15.27:0/2013029050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 msgr2=0x7f11000b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.709+0000 7f110569a640 1 --2- 172.21.15.27:0/2013029050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 0x7f11000b2e00 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f1100072170 tx=0x7f10f402b680 comp rx=0 tx=0).stop 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.709+0000 7f110569a640 1 -- 172.21.15.27:0/2013029050 shutdown_connections 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.709+0000 7f110569a640 1 --2- 172.21.15.27:0/2013029050 >> v2:172.21.15.69:3300/0 conn(0x7f11000ae9b0 0x7f11000d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.709+0000 7f110569a640 1 --2- 172.21.15.27:0/2013029050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 0x7f11000b2e00 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.709+0000 7f110569a640 1 -- 172.21.15.27:0/2013029050 >> 172.21.15.27:0/2013029050 conn(0x7f11000b36d0 msgr2=0x7f11000b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.709+0000 7f110569a640 1 -- 172.21.15.27:0/2013029050 shutdown_connections 2024-03-06T07:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.709+0000 7f110569a640 1 -- 172.21.15.27:0/2013029050 wait complete. 2024-03-06T07:27:20.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.711+0000 7f110569a640 1 Processor -- start 2024-03-06T07:27:20.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.711+0000 7f110569a640 1 -- start start 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f110569a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 0x7f11000cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f110569a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11000ae9b0 0x7f11000c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f110569a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11000b0e90 0x7f11001e5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f110569a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110006c910 con 0x7f110006b5c0 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f110569a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110006c790 con 0x7f11000ae9b0 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f110569a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11000aedc0 con 0x7f11000b0e90 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 0x7f11000cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 0x7f11000cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34406/0 (socket says 172.21.15.27:34406) 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10feffd640 1 -- 172.21.15.27:0/3010774248 learned_addr learned my addr 172.21.15.27:0/3010774248 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10feffd640 1 -- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11000b0e90 msgr2=0x7f11001e5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10feffd640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11000b0e90 0x7f11001e5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10feffd640 1 -- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11000ae9b0 msgr2=0x7f11000c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10feffd640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11000ae9b0 0x7f11000c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10feffd640 1 -- 172.21.15.27:0/3010774248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11001f7730 con 0x7f110006b5c0 2024-03-06T07:27:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10feffd640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 0x7f11000cc5d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f10f402b6c0 tx=0x7f10f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10e77fe640 1 -- 172.21.15.27:0/3010774248 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4041030 con 0x7f110006b5c0 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10e77fe640 1 -- 172.21.15.27:0/3010774248 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f4030510 con 0x7f110006b5c0 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f110569a640 1 -- 172.21.15.27:0/3010774248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11001f7a10 con 0x7f110006b5c0 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f10e77fe640 1 -- 172.21.15.27:0/3010774248 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4037040 con 0x7f110006b5c0 2024-03-06T07:27:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.713+0000 7f110569a640 1 -- 172.21.15.27:0/3010774248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11001f7fd0 con 0x7f110006b5c0 2024-03-06T07:27:20.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.715+0000 7f10e77fe640 1 -- 172.21.15.27:0/3010774248 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10f4044070 con 0x7f110006b5c0 2024-03-06T07:27:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.715+0000 7f10e77fe640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ec06b4d0 0x7f10ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.716+0000 7f10e77fe640 1 -- 172.21.15.27:0/3010774248 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10f40abc30 con 0x7f110006b5c0 2024-03-06T07:27:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.716+0000 7f10fe7fc640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ec06b4d0 0x7f10ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.716+0000 7f10fe7fc640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ec06b4d0 0x7f10ec06d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f11000c5890 tx=0x7f10f0007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.716+0000 7f1107f22740 1 Processor -- start 2024-03-06T07:27:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f1107f22740 1 -- start start 2024-03-06T07:27:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f1107f22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6e3870 0x55ee4e6e3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f1107f22740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e6e78c0 0x55ee4e6e9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f1107f22740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 0x55ee4e713a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f1107f22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e64d6b0 con 0x55ee4e6e3870 2024-03-06T07:27:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f1107f22740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e5f6a60 con 0x55ee4e713600 2024-03-06T07:27:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f1107f22740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e649750 con 0x55ee4e6e78c0 2024-03-06T07:27:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f10feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6e3870 0x55ee4e6e3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f10feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6e3870 0x55ee4e6e3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34418/0 (socket says 172.21.15.27:34418) 2024-03-06T07:27:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f10feffd640 1 -- 172.21.15.27:0/1972412922 learned_addr learned my addr 172.21.15.27:0/1972412922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f10fe7fc640 1 --2- 172.21.15.27:0/1972412922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e6e78c0 0x55ee4e6e9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.717+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/1972412922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 0x55ee4e713a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.718+0000 7f10feffd640 1 -- 172.21.15.27:0/1972412922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e6e78c0 msgr2=0x55ee4e6e9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.718+0000 7f10feffd640 1 --2- 172.21.15.27:0/1972412922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e6e78c0 0x55ee4e6e9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.718+0000 7f10feffd640 1 -- 172.21.15.27:0/1972412922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 msgr2=0x55ee4e713a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.718+0000 7f10feffd640 1 --2- 172.21.15.27:0/1972412922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 0x55ee4e713a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.718+0000 7f10feffd640 1 -- 172.21.15.27:0/1972412922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee4e614f30 con 0x55ee4e6e3870 2024-03-06T07:27:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.718+0000 7f10feffd640 1 --2- 172.21.15.27:0/1972412922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6e3870 0x55ee4e6e3c50 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f10f4082620 tx=0x7f10f4076810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24188da7ff9692e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.718+0000 7f10e4ff9640 1 -- 172.21.15.27:0/1972412922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f407e590 con 0x55ee4e6e3870 2024-03-06T07:27:20.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.718+0000 7f10e4ff9640 1 -- 172.21.15.27:0/1972412922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f4082b50 con 0x55ee4e6e3870 2024-03-06T07:27:20.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.718+0000 7f10e4ff9640 1 -- 172.21.15.27:0/1972412922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4078740 con 0x55ee4e6e3870 2024-03-06T07:27:20.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 -- 172.21.15.27:0/1972412922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6e3870 msgr2=0x55ee4e6e3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 --2- 172.21.15.27:0/1972412922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6e3870 0x55ee4e6e3c50 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f10f4082620 tx=0x7f10f4076810 comp rx=0 tx=0).stop 2024-03-06T07:27:20.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 -- 172.21.15.27:0/1972412922 shutdown_connections 2024-03-06T07:27:20.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 --2- 172.21.15.27:0/1972412922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 0x55ee4e713a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 --2- 172.21.15.27:0/1972412922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e6e78c0 0x55ee4e6e9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 --2- 172.21.15.27:0/1972412922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e6e3870 0x55ee4e6e3c50 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 -- 172.21.15.27:0/1972412922 >> 172.21.15.27:0/1972412922 conn(0x55ee4e6e58f0 msgr2=0x55ee4e6e71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 -- 172.21.15.27:0/1972412922 shutdown_connections 2024-03-06T07:27:20.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 -- 172.21.15.27:0/1972412922 wait complete. 2024-03-06T07:27:20.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 -- start start 2024-03-06T07:27:20.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.719+0000 7f1107f22740 1 Processor -- start 2024-03-06T07:27:20.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f1107f22740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 0x55ee4e70fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f1107f22740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e715c90 0x55ee4e708bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f1107f22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e709110 0x55ee4e714770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f1107f22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e649750 con 0x55ee4e709110 2024-03-06T07:27:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f1107f22740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e609ea0 con 0x55ee4e713600 2024-03-06T07:27:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f1107f22740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e64d6b0 con 0x55ee4e715c90 2024-03-06T07:27:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f1107f22740 10 client.0 _ll_register_callbacks cb 0x55ee4e609870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e709110 0x55ee4e714770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e709110 0x55ee4e714770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34428/0 (socket says 172.21.15.27:34428) 2024-03-06T07:27:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f10ff7fe640 1 -- 172.21.15.27:0/2352304430 learned_addr learned my addr 172.21.15.27:0/2352304430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.720+0000 7f10feffd640 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 0x55ee4e70fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10fe7fc640 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e715c90 0x55ee4e708bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ff7fe640 1 -- 172.21.15.27:0/2352304430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e715c90 msgr2=0x55ee4e708bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e715c90 0x55ee4e708bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ff7fe640 1 -- 172.21.15.27:0/2352304430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 msgr2=0x55ee4e70fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 0x55ee4e70fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ff7fe640 1 -- 172.21.15.27:0/2352304430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee4e60a270 con 0x55ee4e709110 2024-03-06T07:27:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e709110 0x55ee4e714770 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f10f8004350 tx=0x7f10f8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2a9a42bb434d20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:27:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ddffb640 1 -- 172.21.15.27:0/2352304430 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8007aa0 con 0x55ee4e709110 2024-03-06T07:27:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ddffb640 1 -- 172.21.15.27:0/2352304430 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10f8007c40 con 0x55ee4e709110 2024-03-06T07:27:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f1107f22740 1 -- 172.21.15.27:0/2352304430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f10f4082b50 con 0x55ee4e709110 2024-03-06T07:27:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ddffb640 1 -- 172.21.15.27:0/2352304430 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10f8007e40 con 0x55ee4e709110 2024-03-06T07:27:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f1107f22740 10 client.7212 fetch_fsmap learned FSMap version 52 2024-03-06T07:27:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f1107f22740 1 -- 172.21.15.27:0/2352304430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) v3 -- 0x55ee4e3622d0 con 0x55ee4e709110 2024-03-06T07:27:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.721+0000 7f10ddffb640 1 -- 172.21.15.27:0/2352304430 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f4082b50 con 0x55ee4e709110 2024-03-06T07:27:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f10ddffb640 1 -- 172.21.15.27:0/2352304430 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 52) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f10f8025080 con 0x55ee4e709110 2024-03-06T07:27:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f1107f22740 10 client.7212 fetch_fsmap finished waiting for FSMap version 52 2024-03-06T07:27:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f1107f22740 10 client.7212 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f1107f22740 20 client.7212 populate_metadata read hostname 'smithi027' 2024-03-06T07:27:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f1107f22740 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] conn(0x55ee4e73fa00 0x55ee4e73fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f1107f22740 4 client.7212 mds_command: new command op to 7032 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f1107f22740 1 -- 172.21.15.27:0/2352304430 --> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee4e65abf0 con 0x55ee4e73fa00 2024-03-06T07:27:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f10feffd640 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] conn(0x55ee4e73fa00 0x55ee4e73fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f10ddffb640 10 client.7212 ms_handle_connect on v2:172.21.15.27:6832/3587849188 2024-03-06T07:27:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.722+0000 7f10feffd640 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] conn(0x55ee4e73fa00 0x55ee4e73fde0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4405949804a8084f server_cookie=b0e382670d812968 in_seq=0 out_seq=0 2024-03-06T07:27:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f10ddffb640 1 -- 172.21.15.27:0/2352304430 <== mds.0 v2:172.21.15.27:6832/3587849188 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ee4e65abf0 con 0x55ee4e73fa00 2024-03-06T07:27:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f10ddffb640 10 client.7212 handle_command_reply: tid=0 2024-03-06T07:27:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 client.7212 shutdown 2024-03-06T07:27:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 -- 172.21.15.27:0/2352304430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e709110 msgr2=0x55ee4e714770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e709110 0x55ee4e714770 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f10f8004350 tx=0x7f10f8003f90 comp rx=0 tx=0).stop 2024-03-06T07:27:20.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 -- 172.21.15.27:0/2352304430 shutdown_connections 2024-03-06T07:27:20.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] conn(0x55ee4e73fa00 0x55ee4e73fde0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e709110 0x55ee4e714770 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e715c90 0x55ee4e708bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f10ddffb640 0 client.7212 ms_handle_reset on v2:172.21.15.27:6832/3587849188 2024-03-06T07:27:20.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 --2- 172.21.15.27:0/2352304430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e713600 0x55ee4e70fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 -- 172.21.15.27:0/2352304430 >> 172.21.15.27:0/2352304430 conn(0x55ee4e6e3870 msgr2=0x55ee4e7115c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 -- 172.21.15.27:0/2352304430 shutdown_connections 2024-03-06T07:27:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 1 -- 172.21.15.27:0/2352304430 wait complete. 2024-03-06T07:27:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.723+0000 7f1107f22740 20 client.7212 trim_cache size 0 max 16384 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.740+0000 7f1107f22740 1 Processor -- start 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.740+0000 7f1107f22740 1 -- start start 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f1107f22740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 0x55ee4e793500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f1107f22740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 0x55ee4e70fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f1107f22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e710160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f1107f22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11001e44d0 con 0x55ee4e715c90 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f1107f22740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11001e38d0 con 0x55ee4e709110 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f1107f22740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11001e3ed0 con 0x55ee4e713600 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e710160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e710160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34432/0 (socket says 172.21.15.27:34432) 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10ff7fe640 1 -- 172.21.15.27:0/471895239 learned_addr learned my addr 172.21.15.27:0/471895239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:20.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10ff7fe640 1 -- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 msgr2=0x55ee4e70fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10feffd640 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 0x55ee4e793500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10fe7fc640 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 0x55ee4e70fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 0x55ee4e70fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10ff7fe640 1 -- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 msgr2=0x55ee4e793500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 0x55ee4e793500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10ff7fe640 1 -- 172.21.15.27:0/471895239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee4e7153b0 con 0x55ee4e715c90 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10fe7fc640 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 0x55ee4e70fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10feffd640 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 0x55ee4e793500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e710160 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f10f8007000 tx=0x7f10f800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b94c72c3afffbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10dffff640 1 -- 172.21.15.27:0/471895239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8007b20 con 0x55ee4e715c90 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10dffff640 1 -- 172.21.15.27:0/471895239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f801a070 con 0x55ee4e715c90 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.741+0000 7f10dffff640 1 -- 172.21.15.27:0/471895239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f80155b0 con 0x55ee4e715c90 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 -- 172.21.15.27:0/471895239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 msgr2=0x55ee4e710160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e710160 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f10f8007000 tx=0x7f10f800e000 comp rx=0 tx=0).stop 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 -- 172.21.15.27:0/471895239 shutdown_connections 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e710160 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 0x55ee4e70fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 --2- 172.21.15.27:0/471895239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 0x55ee4e793500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 -- 172.21.15.27:0/471895239 >> 172.21.15.27:0/471895239 conn(0x55ee4e6e3870 msgr2=0x55ee4e70f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 -- 172.21.15.27:0/471895239 shutdown_connections 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 -- 172.21.15.27:0/471895239 wait complete. 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.742+0000 7f1107f22740 1 -- start start 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f1107f22740 1 Processor -- start 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f1107f22740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 0x55ee4e6e8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f1107f22740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 0x55ee4e6e8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f1107f22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e6e2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f1107f22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e7169c0 con 0x55ee4e715c90 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f1107f22740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e5f6a60 con 0x55ee4e709110 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f1107f22740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee4e717000 con 0x55ee4e713600 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f1107f22740 10 client.0 _ll_register_callbacks cb 0x7f10f8003e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e6e2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e6e2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34442/0 (socket says 172.21.15.27:34442) 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.743+0000 7f10ff7fe640 1 -- 172.21.15.27:0/1182928461 learned_addr learned my addr 172.21.15.27:0/1182928461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10fe7fc640 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 0x55ee4e6e8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10feffd640 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 0x55ee4e6e8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ff7fe640 1 -- 172.21.15.27:0/1182928461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 msgr2=0x55ee4e6e8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 0x55ee4e6e8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ff7fe640 1 -- 172.21.15.27:0/1182928461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 msgr2=0x55ee4e6e8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 0x55ee4e6e8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ff7fe640 1 -- 172.21.15.27:0/1182928461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee4e42c6b0 con 0x55ee4e715c90 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ff7fe640 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e6e2510 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f10f80154e0 tx=0x7f10f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a416424a525dbce4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ddffb640 1 -- 172.21.15.27:0/1182928461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8015560 con 0x55ee4e715c90 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ddffb640 1 -- 172.21.15.27:0/1182928461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10f8015700 con 0x55ee4e715c90 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f1107f22740 1 -- 172.21.15.27:0/1182928461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f10f801a070 con 0x55ee4e715c90 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f10ddffb640 1 -- 172.21.15.27:0/1182928461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10f803f050 con 0x55ee4e715c90 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f1107f22740 10 client.7218 fetch_fsmap learned FSMap version 52 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.744+0000 7f1107f22740 1 -- 172.21.15.27:0/1182928461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) v3 -- 0x55ee4e72e5f0 con 0x55ee4e715c90 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f10ddffb640 1 -- 172.21.15.27:0/1182928461 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f801a070 con 0x55ee4e715c90 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f10ddffb640 1 -- 172.21.15.27:0/1182928461 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 52) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f10f8042020 con 0x55ee4e715c90 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 10 client.7218 fetch_fsmap finished waiting for FSMap version 52 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 10 client.7218 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 20 client.7218 populate_metadata read hostname 'smithi027' 2024-03-06T07:27:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] conn(0x55ee4e6e2aa0 0x55ee4e6e2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 4 client.7218 mds_command: new command op to 7032 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 -- 172.21.15.27:0/1182928461 --> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ee4e654f40 con 0x55ee4e6e2aa0 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f10feffd640 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] conn(0x55ee4e6e2aa0 0x55ee4e6e2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f10ddffb640 10 client.7218 ms_handle_connect on v2:172.21.15.27:6832/3587849188 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f10feffd640 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] conn(0x55ee4e6e2aa0 0x55ee4e6e2e80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7681957e90fa4b0 server_cookie=ac9ee6a135f5cbcb in_seq=0 out_seq=0 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f10ddffb640 1 -- 172.21.15.27:0/1182928461 <== mds.0 v2:172.21.15.27:6832/3587849188 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55ee4e654f40 con 0x55ee4e6e2aa0 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f10ddffb640 10 client.7218 handle_command_reply: tid=0 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 client.7218 shutdown 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 -- 172.21.15.27:0/1182928461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 msgr2=0x55ee4e6e2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e6e2510 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f10f80154e0 tx=0x7f10f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 -- 172.21.15.27:0/1182928461 shutdown_connections 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.27:6832/3587849188,v1:172.21.15.27:6834/3587849188] conn(0x55ee4e6e2aa0 0x55ee4e6e2e80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee4e715c90 0x55ee4e6e2510 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ee4e713600 0x55ee4e6e8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 --2- 172.21.15.27:0/1182928461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4e709110 0x55ee4e6e8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f1107f22740 1 -- 172.21.15.27:0/1182928461 >> 172.21.15.27:0/1182928461 conn(0x55ee4e6e3870 msgr2=0x55ee4e70fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.745+0000 7f10ddffb640 0 client.7218 ms_handle_reset on v2:172.21.15.27:6832/3587849188 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f1107f22740 1 -- 172.21.15.27:0/1182928461 shutdown_connections 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f1107f22740 1 -- 172.21.15.27:0/1182928461 wait complete. 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f1107f22740 20 client.7218 trim_cache size 0 max 16384 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7032, 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:27:20.764 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:27:20.764 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:27:20.764 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.363140204, 2024-03-06T07:27:20.764 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 52, 2024-03-06T07:27:20.764 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 84, 2024-03-06T07:27:20.764 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 84, 2024-03-06T07:27:20.764 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 16.369821053999999 2024-03-06T07:27:20.764 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:27:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 -- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ec06b4d0 msgr2=0x7f10ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ec06b4d0 0x7f10ec06d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f11000c5890 tx=0x7f10f0007a40 comp rx=0 tx=0).stop 2024-03-06T07:27:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 -- 172.21.15.27:0/3010774248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 msgr2=0x7f11000cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 0x7f11000cc5d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f10f402b6c0 tx=0x7f10f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 -- 172.21.15.27:0/3010774248 shutdown_connections 2024-03-06T07:27:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ec06b4d0 0x7f10ec06d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11000b0e90 0x7f11001e5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11000ae9b0 0x7f11000c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 --2- 172.21.15.27:0/3010774248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f110006b5c0 0x7f11000cc5d0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 -- 172.21.15.27:0/3010774248 >> 172.21.15.27:0/3010774248 conn(0x7f11000b36d0 msgr2=0x7f11001df9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 -- 172.21.15.27:0/3010774248 shutdown_connections 2024-03-06T07:27:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.746+0000 7f110569a640 1 -- 172.21.15.27:0/3010774248 wait complete. 2024-03-06T07:27:20.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:20.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.872+0000 7f2653c65640 1 Processor -- start 2024-03-06T07:27:20.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.872+0000 7f2653c65640 1 -- start start 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2653c65640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f264c0b4260 0x7f264c0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2653c65640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f264c0d06c0 0x7f264c0b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2653c65640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c0b0dd0 con 0x7f264c0d02e0 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2653c65640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264c0b0b80 con 0x7f264c0b4260 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2653c65640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264c0aeec0 con 0x7f264c0d06c0 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2651200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f264c0b4260 0x7f264c0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2651200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f264c0b4260 0x7f264c0cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34450/0 (socket says 172.21.15.27:34450) 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2651200640 1 -- 172.21.15.27:0/867674930 learned_addr learned my addr 172.21.15.27:0/867674930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2652202640 1 --2- 172.21.15.27:0/867674930 >> v2:172.21.15.69:3300/0 conn(0x7f264c0d06c0 0x7f264c0b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2651200640 1 -- 172.21.15.27:0/867674930 >> v2:172.21.15.69:3300/0 conn(0x7f264c0d06c0 msgr2=0x7f264c0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2651200640 1 --2- 172.21.15.27:0/867674930 >> v2:172.21.15.69:3300/0 conn(0x7f264c0d06c0 0x7f264c0b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2651200640 1 -- 172.21.15.27:0/867674930 >> v1:172.21.15.27:6789/0 conn(0x7f264c0d02e0 legacy=0x7f264c0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.873+0000 7f2651200640 1 -- 172.21.15.27:0/867674930 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c0b10b0 con 0x7f264c0b4260 2024-03-06T07:27:20.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.875+0000 7f2651200640 1 --2- 172.21.15.27:0/867674930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 0x7f264c0cfa00 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f264c0cc0e0 tx=0x7f263c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1e88fa40ee3398 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.875+0000 7f26509ff640 1 -- 172.21.15.27:0/867674930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c02f850 con 0x7f264c0b4260 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.875+0000 7f26509ff640 1 -- 172.21.15.27:0/867674930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c02f9f0 con 0x7f264c0b4260 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.875+0000 7f26509ff640 1 -- 172.21.15.27:0/867674930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c02fdc0 con 0x7f264c0b4260 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.876+0000 7f2653c65640 1 -- 172.21.15.27:0/867674930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 msgr2=0x7f264c0cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.876+0000 7f2653c65640 1 --2- 172.21.15.27:0/867674930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 0x7f264c0cfa00 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f264c0cc0e0 tx=0x7f263c02ba30 comp rx=0 tx=0).stop 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.876+0000 7f2653c65640 1 -- 172.21.15.27:0/867674930 shutdown_connections 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.876+0000 7f2653c65640 1 --2- 172.21.15.27:0/867674930 >> v2:172.21.15.69:3300/0 conn(0x7f264c0d06c0 0x7f264c0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.876+0000 7f2653c65640 1 --2- 172.21.15.27:0/867674930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 0x7f264c0cfa00 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.876+0000 7f2653c65640 1 -- 172.21.15.27:0/867674930 >> 172.21.15.27:0/867674930 conn(0x7f264c0b36d0 msgr2=0x7f264c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.876+0000 7f2653c65640 1 -- 172.21.15.27:0/867674930 shutdown_connections 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.876+0000 7f2653c65640 1 -- 172.21.15.27:0/867674930 wait complete. 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.877+0000 7f2653c65640 1 Processor -- start 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.877+0000 7f2653c65640 1 -- start start 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2653c65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 0x7f264c06bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2653c65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f264c0d02e0 0x7f264c09c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2653c65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f264c0d06c0 0x7f264c15e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2653c65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c0b22d0 con 0x7f264c0b4260 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2653c65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c0b2150 con 0x7f264c0d06c0 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2653c65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264c0af350 con 0x7f264c0d02e0 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651a01640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 0x7f264c06bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651a01640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 0x7f264c06bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34458/0 (socket says 172.21.15.27:34458) 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651a01640 1 -- 172.21.15.27:0/1522830203 learned_addr learned my addr 172.21.15.27:0/1522830203 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651200640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f264c0d02e0 0x7f264c09c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2652202640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f264c0d06c0 0x7f264c15e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651a01640 1 -- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f264c0d02e0 msgr2=0x7f264c09c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651a01640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f264c0d02e0 0x7f264c09c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651a01640 1 -- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f264c0d06c0 msgr2=0x7f264c15e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651a01640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f264c0d06c0 0x7f264c15e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651a01640 1 -- 172.21.15.27:0/1522830203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c0c9cc0 con 0x7f264c0b4260 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651200640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f264c0d02e0 0x7f264c09c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2652202640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f264c0d06c0 0x7f264c15e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2651a01640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 0x7f264c06bde0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f264800c400 tx=0x7f264800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f26427fc640 1 -- 172.21.15.27:0/1522830203 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26480052d0 con 0x7f264c0b4260 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f26427fc640 1 -- 172.21.15.27:0/1522830203 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2648005490 con 0x7f264c0b4260 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264c0c9fa0 con 0x7f264c0b4260 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.878+0000 7f26427fc640 1 -- 172.21.15.27:0/1522830203 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648007260 con 0x7f264c0b4260 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.879+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264c0ca560 con 0x7f264c0b4260 2024-03-06T07:27:20.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.880+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264c070a20 con 0x7f264c0b4260 2024-03-06T07:27:20.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.881+0000 7f26427fc640 1 -- 172.21.15.27:0/1522830203 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2648007400 con 0x7f264c0b4260 2024-03-06T07:27:20.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.881+0000 7f26427fc640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f263006b4d0 0x7f263006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.881+0000 7f26427fc640 1 -- 172.21.15.27:0/1522830203 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26480896a0 con 0x7f264c0b4260 2024-03-06T07:27:20.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.881+0000 7f2651200640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f263006b4d0 0x7f263006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.882+0000 7f2651200640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f263006b4d0 0x7f263006d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f263c036000 tx=0x7f263c039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:20.886+0000 7f26427fc640 1 -- 172.21.15.27:0/1522830203 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f264c070a20 con 0x7f264c0b4260 2024-03-06T07:27:21.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.107+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f264c0cf2e0 con 0x7f264c0b4260 2024-03-06T07:27:21.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.107+0000 7f26427fc640 1 -- 172.21.15.27:0/1522830203 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v52) v1 ==== 53+0+73 (secure 0 0 0) 0x7f264c0cf2e0 con 0x7f264c0b4260 2024-03-06T07:27:21.110 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f263006b4d0 msgr2=0x7f263006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f263006b4d0 0x7f263006d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f263c036000 tx=0x7f263c039040 comp rx=0 tx=0).stop 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 msgr2=0x7f264c06bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 0x7f264c06bde0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f264800c400 tx=0x7f264800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 shutdown_connections 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f263006b4d0 0x7f263006d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f264c0d06c0 0x7f264c15e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f264c0d02e0 0x7f264c09c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 --2- 172.21.15.27:0/1522830203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f264c0b4260 0x7f264c06bde0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 >> 172.21.15.27:0/1522830203 conn(0x7f264c0b36d0 msgr2=0x7f264c06c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 shutdown_connections 2024-03-06T07:27:21.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:21.109+0000 7f2653c65640 1 -- 172.21.15.27:0/1522830203 wait complete. 2024-03-06T07:27:21.122 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:21.122 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:21.122 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:27:21.123 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:21.123 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.123 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:21.123 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:21.123 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:21.123 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:21.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.123 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:21.139 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42449sec preferred_lft 42449sec 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:27:21.140 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:27:21.141 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:21.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.142 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:21.208 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:27:21.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.209 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:27:21.265 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:27:21.266 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:21.266 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:27:21.266 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:27:21.266 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:21.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.267 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:27:21.267 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:27:21.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:27:21.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:27:21.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:27:21.268 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:27:21.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.371 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:27:21.386 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:21.386 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:21.386 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:21.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:27:21.456 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:21.457 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:21.457 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:21.512 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:21.512 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:27:21.639 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:21.695 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:27:21.695 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:27:21.695 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:27:21.695 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:27:21.695 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:27:21.695 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:27:21.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:21.696 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.697 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.698 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:27:21.752 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:21.752 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:21.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:27:21.825 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:21.882 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:21.883 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:21.884 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:21.885 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:21.885 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:27:21.885 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.885 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:21.885 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.885 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:21.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.885 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:27:21.890 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:27:21.887+0000 7f8ffc647480 -1 init, newargv = 0x55c7f4fb53f0 newargc=15 2024-03-06T07:27:21.890 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[66106]: starting ceph client 2024-03-06T07:27:21.894 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[66106]: starting fuse 2024-03-06T07:27:21.906 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:27:21.906 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:27:21.907 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:21.908 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:27:21.908 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:21.908 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:27:22.032 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.66106.asok 2024-03-06T07:27:22.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:22.043 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.66106.asok 2024-03-06T07:27:22.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.66106.asok status 2024-03-06T07:27:22.190 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:27:22.190 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:27:22.190 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:22.190 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:22.190 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:27:22.190 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:27:22.190 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:22.190 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "66106", 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7233, 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 7233 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:27:22.191 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4130545623 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4130545623 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.7233 192.168.0.1:0/4130545623", 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/4130545623", 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 52, 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 84, 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.199 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:27:22.199 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:22.214 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:27:22.215 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:22.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:22.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.287 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:27:22.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.348 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:27:22.412 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:27:22.412 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:22.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.467 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:27:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:35+0 records in 2024-03-06T07:27:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:35+0 records out 2024-03-06T07:27:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.181505 s, 202 MB/s 2024-03-06T07:27:22.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:22.709 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:27:22.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:22.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.874+0000 7f24132df640 1 Processor -- start 2024-03-06T07:27:22.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.874+0000 7f24132df640 1 -- start start 2024-03-06T07:27:22.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.875+0000 7f24132df640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f240c15df00 0x7f240c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.875+0000 7f24132df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.875+0000 7f24132df640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c0cea30 con 0x7f240c0b9a30 2024-03-06T07:27:22.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.875+0000 7f24132df640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240c0d2af0 con 0x7f240c15df00 2024-03-06T07:27:22.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.875+0000 7f24132df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0cf3c0 con 0x7f240c0b95a0 2024-03-06T07:27:22.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.875+0000 7f241107b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f240c15df00 0x7f240c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.875+0000 7f241087a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.875+0000 7f241107b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f240c15df00 0x7f240c0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:47922/0 (socket says 172.21.15.27:47922) 2024-03-06T07:27:22.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.875+0000 7f241107b640 1 -- 172.21.15.27:0/3613837997 learned_addr learned my addr 172.21.15.27:0/3613837997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:22.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.876+0000 7f241107b640 1 -- 172.21.15.27:0/3613837997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 msgr2=0x7f240c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.876+0000 7f241107b640 1 --2- 172.21.15.27:0/3613837997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c0b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.876+0000 7f241107b640 1 -- 172.21.15.27:0/3613837997 >> v1:172.21.15.69:6789/0 conn(0x7f240c0b9a30 legacy=0x7f240c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.876+0000 7f241107b640 1 -- 172.21.15.27:0/3613837997 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0cf0e0 con 0x7f240c15df00 2024-03-06T07:27:22.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.876+0000 7f241087a640 1 --2- 172.21.15.27:0/3613837997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:22.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.877+0000 7f241107b640 1 --2- 172.21.15.27:0/3613837997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c15df00 0x7f240c0d25b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2408009870 tx=0x7f240802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2956cb6a5d2ef9fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.877+0000 7f2403fff640 1 -- 172.21.15.27:0/3613837997 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408034070 con 0x7f240c15df00 2024-03-06T07:27:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.877+0000 7f2403fff640 1 -- 172.21.15.27:0/3613837997 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2408030840 con 0x7f240c15df00 2024-03-06T07:27:22.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.877+0000 7f2403fff640 1 -- 172.21.15.27:0/3613837997 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408030c10 con 0x7f240c15df00 2024-03-06T07:27:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.878+0000 7f24132df640 1 -- 172.21.15.27:0/3613837997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c15df00 msgr2=0x7f240c0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.878+0000 7f24132df640 1 --2- 172.21.15.27:0/3613837997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c15df00 0x7f240c0d25b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2408009870 tx=0x7f240802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.878+0000 7f24132df640 1 -- 172.21.15.27:0/3613837997 shutdown_connections 2024-03-06T07:27:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.878+0000 7f24132df640 1 --2- 172.21.15.27:0/3613837997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.878+0000 7f24132df640 1 --2- 172.21.15.27:0/3613837997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c15df00 0x7f240c0d25b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.878+0000 7f24132df640 1 -- 172.21.15.27:0/3613837997 >> 172.21.15.27:0/3613837997 conn(0x7f240c071960 msgr2=0x7f240c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.878+0000 7f24132df640 1 -- 172.21.15.27:0/3613837997 shutdown_connections 2024-03-06T07:27:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.878+0000 7f24132df640 1 -- 172.21.15.27:0/3613837997 wait complete. 2024-03-06T07:27:22.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.879+0000 7f24132df640 1 Processor -- start 2024-03-06T07:27:22.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.879+0000 7f24132df640 1 -- start start 2024-03-06T07:27:22.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f24132df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f24132df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f240c15df00 0x7f240c0c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f24132df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c0c8700 0x7f240c15af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f24132df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0cec70 con 0x7f240c15df00 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f24132df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0ce750 con 0x7f240c0c8700 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f24132df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0cedf0 con 0x7f240c0b95a0 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241087a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f240c15df00 0x7f240c0c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241107b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241087a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f240c15df00 0x7f240c0c81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34468/0 (socket says 172.21.15.27:34468) 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241087a640 1 -- 172.21.15.27:0/2134928099 learned_addr learned my addr 172.21.15.27:0/2134928099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241187c640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c0c8700 0x7f240c15af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241107b640 1 -- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c0c8700 msgr2=0x7f240c15af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241107b640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c0c8700 0x7f240c15af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241107b640 1 -- 172.21.15.27:0/2134928099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f240c15df00 msgr2=0x7f240c0c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241107b640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f240c15df00 0x7f240c0c81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241107b640 1 -- 172.21.15.27:0/2134928099 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0c8c80 con 0x7f240c0b95a0 2024-03-06T07:27:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.880+0000 7f241087a640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f240c15df00 0x7f240c0c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.881+0000 7f241107b640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c159380 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f240c0b6870 tx=0x7f2408004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.881+0000 7f2401ffb640 1 -- 172.21.15.27:0/2134928099 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408048070 con 0x7f240c0b95a0 2024-03-06T07:27:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.881+0000 7f2401ffb640 1 -- 172.21.15.27:0/2134928099 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2408030640 con 0x7f240c0b95a0 2024-03-06T07:27:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.881+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c0c8f60 con 0x7f240c0b95a0 2024-03-06T07:27:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.881+0000 7f2401ffb640 1 -- 172.21.15.27:0/2134928099 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408034040 con 0x7f240c0b95a0 2024-03-06T07:27:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.881+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c0c9520 con 0x7f240c0b95a0 2024-03-06T07:27:22.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.882+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c067ff0 con 0x7f240c0b95a0 2024-03-06T07:27:22.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.882+0000 7f2401ffb640 1 -- 172.21.15.27:0/2134928099 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f240804b070 con 0x7f240c0b95a0 2024-03-06T07:27:22.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.883+0000 7f2401ffb640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23f006b500 0x7f23f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.883+0000 7f2401ffb640 1 -- 172.21.15.27:0/2134928099 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24080ac410 con 0x7f240c0b95a0 2024-03-06T07:27:22.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.883+0000 7f241087a640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23f006b500 0x7f23f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.883+0000 7f241087a640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23f006b500 0x7f23f006d9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f240c0cb770 tx=0x7f23fc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:22.888+0000 7f2401ffb640 1 -- 172.21.15.27:0/2134928099 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f240807ac30 con 0x7f240c0b95a0 2024-03-06T07:27:23.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.212+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f240c0008d0 con 0x7f240c0b95a0 2024-03-06T07:27:23.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.212+0000 7f2401ffb640 1 -- 172.21.15.27:0/2134928099 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f240807ecd0 con 0x7f240c0b95a0 2024-03-06T07:27:23.215 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:23.215 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/930650799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":930650799},{"type":"v1","addr":"172.21.15.69:6836","nonce":930650799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1050755919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1050755919},{"type":"v1","addr":"172.21.15.27:6835","nonce":1050755919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1173605349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1173605349},{"type":"v1","addr":"172.21.15.69:6837","nonce":1173605349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:09.729142+0000","modified":"2024-03-06T07:27:12.757645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7032},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7032":{"gid":7032,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3587849188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3587849188},{"type":"v1","addr":"172.21.15.27:6834","nonce":3587849188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7032,"qdb_cluster":[7032]},"id":6}]} 2024-03-06T07:27:23.215 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 52 2024-03-06T07:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23f006b500 msgr2=0x7f23f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23f006b500 0x7f23f006d9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f240c0cb770 tx=0x7f23fc009190 comp rx=0 tx=0).stop 2024-03-06T07:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 msgr2=0x7f240c159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c159380 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f240c0b6870 tx=0x7f2408004730 comp rx=0 tx=0).stop 2024-03-06T07:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 shutdown_connections 2024-03-06T07:27:23.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23f006b500 0x7f23f006d9c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c0c8700 0x7f240c15af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f240c15df00 0x7f240c0c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 --2- 172.21.15.27:0/2134928099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f240c0b95a0 0x7f240c159380 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.214+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 >> 172.21.15.27:0/2134928099 conn(0x7f240c071960 msgr2=0x7f240c0ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.215+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 shutdown_connections 2024-03-06T07:27:23.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.215+0000 7f24132df640 1 -- 172.21.15.27:0/2134928099 wait complete. 2024-03-06T07:27:23.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:27:23.365 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:27:23.365 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:23.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:27:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.469+0000 7f9f51765640 1 Processor -- start 2024-03-06T07:27:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.469+0000 7f9f51765640 1 -- start start 2024-03-06T07:27:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.470+0000 7f9f51765640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c15df00 0x7f9f4c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.470+0000 7f9f51765640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f4c0b95a0 0x7f9f4c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.470+0000 7f9f51765640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c0cea30 con 0x7f9f4c0b9a30 2024-03-06T07:27:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.470+0000 7f9f51765640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f4c0d2af0 con 0x7f9f4c0b95a0 2024-03-06T07:27:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.470+0000 7f9f51765640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0cf3c0 con 0x7f9f4c15df00 2024-03-06T07:27:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.470+0000 7f9f4a7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f4c0b95a0 0x7f9f4c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.470+0000 7f9f4a7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f4c0b95a0 0x7f9f4c0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34480/0 (socket says 172.21.15.27:34480) 2024-03-06T07:27:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.470+0000 7f9f4a7fc640 1 -- 172.21.15.27:0/1229443315 learned_addr learned my addr 172.21.15.27:0/1229443315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.470+0000 7f9f4affd640 1 --2- 172.21.15.27:0/1229443315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c15df00 0x7f9f4c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.471+0000 7f9f4a7fc640 1 -- 172.21.15.27:0/1229443315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c15df00 msgr2=0x7f9f4c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.471+0000 7f9f4a7fc640 1 --2- 172.21.15.27:0/1229443315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c15df00 0x7f9f4c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.471+0000 7f9f4a7fc640 1 -- 172.21.15.27:0/1229443315 >> v1:172.21.15.69:6789/0 conn(0x7f9f4c0b9a30 legacy=0x7f9f4c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.471+0000 7f9f4a7fc640 1 -- 172.21.15.27:0/1229443315 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0cf0e0 con 0x7f9f4c0b95a0 2024-03-06T07:27:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.471+0000 7f9f4affd640 1 --2- 172.21.15.27:0/1229443315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c15df00 0x7f9f4c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.471+0000 7f9f4a7fc640 1 --2- 172.21.15.27:0/1229443315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c0b95a0 0x7f9f4c0b5700 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c009870 tx=0x7f9f3c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413b85b650a2121e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.471+0000 7f9f49ffb640 1 -- 172.21.15.27:0/1229443315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c035020 con 0x7f9f4c0b95a0 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f49ffb640 1 -- 172.21.15.27:0/1229443315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c031640 con 0x7f9f4c0b95a0 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f49ffb640 1 -- 172.21.15.27:0/1229443315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c02d040 con 0x7f9f4c0b95a0 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f51765640 1 -- 172.21.15.27:0/1229443315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c0b95a0 msgr2=0x7f9f4c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f51765640 1 --2- 172.21.15.27:0/1229443315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c0b95a0 0x7f9f4c0b5700 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c009870 tx=0x7f9f3c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f51765640 1 -- 172.21.15.27:0/1229443315 shutdown_connections 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f51765640 1 --2- 172.21.15.27:0/1229443315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c0b95a0 0x7f9f4c0b5700 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f51765640 1 --2- 172.21.15.27:0/1229443315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c15df00 0x7f9f4c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f51765640 1 -- 172.21.15.27:0/1229443315 >> 172.21.15.27:0/1229443315 conn(0x7f9f4c071960 msgr2=0x7f9f4c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f51765640 1 -- 172.21.15.27:0/1229443315 shutdown_connections 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.472+0000 7f9f51765640 1 -- 172.21.15.27:0/1229443315 wait complete. 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.473+0000 7f9f51765640 1 Processor -- start 2024-03-06T07:27:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.473+0000 7f9f51765640 1 -- start start 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.474+0000 7f9f51765640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c0b95a0 0x7f9f4c15b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.474+0000 7f9f51765640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f4c15df00 0x7f9f4c150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.474+0000 7f9f51765640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c15b230 0x7f9f4c1f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.474+0000 7f9f51765640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0cec70 con 0x7f9f4c15b230 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.474+0000 7f9f51765640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0ce750 con 0x7f9f4c15df00 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.474+0000 7f9f51765640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0cedf0 con 0x7f9f4c0b95a0 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.474+0000 7f9f4b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c15b230 0x7f9f4c1f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.474+0000 7f9f4b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c15b230 0x7f9f4c1f3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34482/0 (socket says 172.21.15.27:34482) 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.474+0000 7f9f4b7fe640 1 -- 172.21.15.27:0/1754547615 learned_addr learned my addr 172.21.15.27:0/1754547615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f4a7fc640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f4c15df00 0x7f9f4c150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f4b7fe640 1 -- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c0b95a0 msgr2=0x7f9f4c15b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f4affd640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c0b95a0 0x7f9f4c15b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f4b7fe640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c0b95a0 0x7f9f4c15b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f4b7fe640 1 -- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f4c15df00 msgr2=0x7f9f4c150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f4b7fe640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f4c15df00 0x7f9f4c150ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f4b7fe640 1 -- 172.21.15.27:0/1754547615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c15e570 con 0x7f9f4c15b230 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f4b7fe640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c15b230 0x7f9f4c1f3b40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400c730 tx=0x7f9f4400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f337fe640 1 -- 172.21.15.27:0/1754547615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44016020 con 0x7f9f4c15b230 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f337fe640 1 -- 172.21.15.27:0/1754547615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4400aa50 con 0x7f9f4c15b230 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4c0c8b10 con 0x7f9f4c15b230 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f337fe640 1 -- 172.21.15.27:0/1754547615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4400ce40 con 0x7f9f4c15b230 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.475+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4c0c90d0 con 0x7f9f4c15b230 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.476+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4c067ff0 con 0x7f9f4c15b230 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.477+0000 7f9f337fe640 1 -- 172.21.15.27:0/1754547615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f44019020 con 0x7f9f4c15b230 2024-03-06T07:27:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.477+0000 7f9f337fe640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f3806b4d0 0x7f9f3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.477+0000 7f9f337fe640 1 -- 172.21.15.27:0/1754547615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f4408a1e0 con 0x7f9f4c15b230 2024-03-06T07:27:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.477+0000 7f9f4affd640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f3806b4d0 0x7f9f3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.478+0000 7f9f4affd640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f3806b4d0 0x7f9f3806d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000b840 tx=0x7f9f40008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.482+0000 7f9f337fe640 1 -- 172.21.15.27:0/1754547615 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f44020030 con 0x7f9f4c15b230 2024-03-06T07:27:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.735+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f9f4c06b780 con 0x7f9f4c15b230 2024-03-06T07:27:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.735+0000 7f9f337fe640 1 -- 172.21.15.27:0/1754547615 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1653 (secure 0 0 0) 0x7f9f4c15e570 con 0x7f9f4c15b230 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: "total_bytes": 773094113280, 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: "total_avail_bytes": 772829155328, 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_bytes": 264957952, 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_bytes": 264957952, 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_ratio": 0.00034272405900992453, 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: "num_per_pool_osds": 8, 2024-03-06T07:27:23.738 INFO:teuthology.orchestra.run.smithi027.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: "stats_by_class": { 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: "ssd": { 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: "total_bytes": 773094113280, 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: "total_avail_bytes": 772829155328, 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_bytes": 264957952, 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_bytes": 264957952, 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: "total_used_raw_ratio": 0.00034272405900992453 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:23.739 INFO:teuthology.orchestra.run.smithi027.stdout: "name": ".mgr", 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 1, 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "stored": 590368, 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "objects": 2, 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "kb_used": 1160, 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 1187840, 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "percent_used": 1.6179312751773978e-06, 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "max_avail": 367085453312 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs_metadata", 2024-03-06T07:27:23.740 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 12, 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "stored": 2410, 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "objects": 22, 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "kb_used": 64, 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 65536, 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "percent_used": 8.9265313363284804e-08, 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "max_avail": 367085453312 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs_data", 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 13, 2024-03-06T07:27:23.741 INFO:teuthology.orchestra.run.smithi027.stdout: "stats": { 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: "stored": 0, 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: "objects": 0, 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: "kb_used": 0, 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 0, 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: "percent_used": 0, 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: "max_avail": 367085453312 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:27:23.742 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f3806b4d0 msgr2=0x7f9f3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f3806b4d0 0x7f9f3806d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000b840 tx=0x7f9f40008040 comp rx=0 tx=0).stop 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c15b230 msgr2=0x7f9f4c1f3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c15b230 0x7f9f4c1f3b40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400c730 tx=0x7f9f4400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 shutdown_connections 2024-03-06T07:27:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f3806b4d0 0x7f9f3806d990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4c15b230 0x7f9f4c1f3b40 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f4c15df00 0x7f9f4c150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 --2- 172.21.15.27:0/1754547615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f4c0b95a0 0x7f9f4c15b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 >> 172.21.15.27:0/1754547615 conn(0x7f9f4c071960 msgr2=0x7f9f4c0b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 shutdown_connections 2024-03-06T07:27:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:23.737+0000 7f9f51765640 1 -- 172.21.15.27:0/1754547615 wait complete. 2024-03-06T07:27:23.750 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:27:23.750 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:23.750 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:23.802 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:23.850 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:23.850 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:27:23.850 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:23.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:24.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.011+0000 7f4983df6640 1 Processor -- start 2024-03-06T07:27:24.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.011+0000 7f4983df6640 1 -- start start 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.012+0000 7f4983df6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f497c071540 0x7f497c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.012+0000 7f4983df6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0b2b50 0x7f497c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.012+0000 7f4983df6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c0ce610 con 0x7f497c0d3400 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.012+0000 7f4983df6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f497c0d2f70 con 0x7f497c071540 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.012+0000 7f4983df6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0d4be0 con 0x7f497c0b2b50 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.012+0000 7f4981b92640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f497c071540 0x7f497c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.012+0000 7f4982393640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0b2b50 0x7f497c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.012+0000 7f4981b92640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f497c071540 0x7f497c0b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:44316/0 (socket says 172.21.15.27:44316) 2024-03-06T07:27:24.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.012+0000 7f4981b92640 1 -- 172.21.15.27:0/2752046711 learned_addr learned my addr 172.21.15.27:0/2752046711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.013+0000 7f4982393640 1 -- 172.21.15.27:0/2752046711 >> v2:172.21.15.69:3300/0 conn(0x7f497c071540 msgr2=0x7f497c0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.013+0000 7f4982393640 1 --2- 172.21.15.27:0/2752046711 >> v2:172.21.15.69:3300/0 conn(0x7f497c071540 0x7f497c0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.013+0000 7f4982393640 1 -- 172.21.15.27:0/2752046711 >> v1:172.21.15.69:6789/0 conn(0x7f497c0d3400 legacy=0x7f497c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.013+0000 7f4982393640 1 -- 172.21.15.27:0/2752046711 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c0d2920 con 0x7f497c0b2b50 2024-03-06T07:27:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.013+0000 7f4981b92640 1 --2- 172.21.15.27:0/2752046711 >> v2:172.21.15.69:3300/0 conn(0x7f497c071540 0x7f497c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.013+0000 7f4982393640 1 --2- 172.21.15.27:0/2752046711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0b2b50 0x7f497c0b6e20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4974009d90 tx=0x7f497402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b27bb2a86430a27a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4980b90640 1 -- 172.21.15.27:0/2752046711 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974039070 con 0x7f497c0b2b50 2024-03-06T07:27:24.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4980b90640 1 -- 172.21.15.27:0/2752046711 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497402cc50 con 0x7f497c0b2b50 2024-03-06T07:27:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4980b90640 1 -- 172.21.15.27:0/2752046711 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974034440 con 0x7f497c0b2b50 2024-03-06T07:27:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4983df6640 1 -- 172.21.15.27:0/2752046711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0b2b50 msgr2=0x7f497c0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4983df6640 1 --2- 172.21.15.27:0/2752046711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0b2b50 0x7f497c0b6e20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4974009d90 tx=0x7f497402bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4983df6640 1 -- 172.21.15.27:0/2752046711 shutdown_connections 2024-03-06T07:27:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4983df6640 1 --2- 172.21.15.27:0/2752046711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0b2b50 0x7f497c0b6e20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4983df6640 1 --2- 172.21.15.27:0/2752046711 >> v2:172.21.15.69:3300/0 conn(0x7f497c071540 0x7f497c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4983df6640 1 -- 172.21.15.27:0/2752046711 >> 172.21.15.27:0/2752046711 conn(0x7f497c0d3ca0 msgr2=0x7f497c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4983df6640 1 -- 172.21.15.27:0/2752046711 shutdown_connections 2024-03-06T07:27:24.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.014+0000 7f4983df6640 1 -- 172.21.15.27:0/2752046711 wait complete. 2024-03-06T07:27:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.015+0000 7f4983df6640 1 Processor -- start 2024-03-06T07:27:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.015+0000 7f4983df6640 1 -- start start 2024-03-06T07:27:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4983df6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c071540 0x7f497c0cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4983df6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f497c0b2b50 0x7f497c14c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4983df6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0d3400 0x7f497c14ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4983df6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c15e370 con 0x7f497c0b2b50 2024-03-06T07:27:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4983df6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0ce8f0 con 0x7f497c071540 2024-03-06T07:27:24.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4983df6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497c14add0 con 0x7f497c0d3400 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4981391640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f497c0b2b50 0x7f497c14c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4981391640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f497c0b2b50 0x7f497c14c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36036/0 (socket says 172.21.15.27:36036) 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4981391640 1 -- 172.21.15.27:0/3064151348 learned_addr learned my addr 172.21.15.27:0/3064151348 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4981b92640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c071540 0x7f497c0cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.016+0000 7f4982393640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0d3400 0x7f497c14ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f4982393640 1 -- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c071540 msgr2=0x7f497c0cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f4982393640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c071540 0x7f497c0cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f4982393640 1 -- 172.21.15.27:0/3064151348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f497c0b2b50 msgr2=0x7f497c14c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f4982393640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f497c0b2b50 0x7f497c14c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f4982393640 1 -- 172.21.15.27:0/3064151348 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c0cd200 con 0x7f497c0d3400 2024-03-06T07:27:24.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f4981391640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f497c0b2b50 0x7f497c14c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f4982393640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0d3400 0x7f497c14ce90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f497c0d30f0 tx=0x7f4974004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f49727fc640 1 -- 172.21.15.27:0/3064151348 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974039050 con 0x7f497c0d3400 2024-03-06T07:27:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f49727fc640 1 -- 172.21.15.27:0/3064151348 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4974045070 con 0x7f497c0d3400 2024-03-06T07:27:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c0cd4e0 con 0x7f497c0d3400 2024-03-06T07:27:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f49727fc640 1 -- 172.21.15.27:0/3064151348 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974009450 con 0x7f497c0d3400 2024-03-06T07:27:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.017+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c0ca170 con 0x7f497c0d3400 2024-03-06T07:27:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.018+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c0cd700 con 0x7f497c0d3400 2024-03-06T07:27:24.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.019+0000 7f49727fc640 1 -- 172.21.15.27:0/3064151348 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f497403e610 con 0x7f497c0d3400 2024-03-06T07:27:24.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.019+0000 7f49727fc640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f496006b4d0 0x7f496006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.019+0000 7f49727fc640 1 -- 172.21.15.27:0/3064151348 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49740ac9a0 con 0x7f497c0d3400 2024-03-06T07:27:24.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.019+0000 7f4981b92640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f496006b4d0 0x7f496006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.020+0000 7f4981b92640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f496006b4d0 0x7f496006d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4978006ca0 tx=0x7f4978009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.024+0000 7f49727fc640 1 -- 172.21.15.27:0/3064151348 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f497407b260 con 0x7f497c0d3400 2024-03-06T07:27:24.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.208+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f497c14af50 con 0x7f497c0d3400 2024-03-06T07:27:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.209+0000 7f49727fc640 1 -- 172.21.15.27:0/3064151348 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16588 (secure 0 0 0) 0x7f497407f300 con 0x7f497c0d3400 2024-03-06T07:27:24.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:24.212 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":84,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:11.381712+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:07.388472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:08.720233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:24.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f496006b4d0 msgr2=0x7f496006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f496006b4d0 0x7f496006d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4978006ca0 tx=0x7f4978009240 comp rx=0 tx=0).stop 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0d3400 msgr2=0x7f497c14ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0d3400 0x7f497c14ce90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f497c0d30f0 tx=0x7f4974004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 shutdown_connections 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f496006b4d0 0x7f496006d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f497c0d3400 0x7f497c14ce90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f497c0b2b50 0x7f497c14c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 --2- 172.21.15.27:0/3064151348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c071540 0x7f497c0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 >> 172.21.15.27:0/3064151348 conn(0x7f497c0d3ca0 msgr2=0x7f497c160590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 shutdown_connections 2024-03-06T07:27:24.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.211+0000 7f4983df6640 1 -- 172.21.15.27:0/3064151348 wait complete. 2024-03-06T07:27:24.224 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:07.388472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:08.720233+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:24.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:24.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.331+0000 7f2491088640 1 Processor -- start 2024-03-06T07:27:24.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.332+0000 7f2491088640 1 -- start start 2024-03-06T07:27:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.332+0000 7f2491088640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f248c0b2d30 0x7f248c0b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.332+0000 7f2491088640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248c0d4960 con 0x7f248c0b7820 2024-03-06T07:27:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.332+0000 7f2491088640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248c0d36c0 con 0x7f248c071540 2024-03-06T07:27:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.332+0000 7f2491088640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f248c0d3230 con 0x7f248c0b2d30 2024-03-06T07:27:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.332+0000 7f248b59e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f248c0b7820 0x7f248c0b7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39754/0 (socket says 172.21.15.27:39754) 2024-03-06T07:27:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.332+0000 7f248b59e640 1 -- 172.21.15.27:0/2073512511 learned_addr learned my addr 172.21.15.27:0/2073512511 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f248a59c640 1 --2- 172.21.15.27:0/2073512511 >> v2:172.21.15.69:3300/0 conn(0x7f248c0b2d30 0x7f248c0b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892270217 0 0) 0x7f248c0d4960 con 0x7f248c0b7820 2024-03-06T07:27:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2474003520 con 0x7f248c0b7820 2024-03-06T07:27:24.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 693824779 0 0) 0x7f2474003520 con 0x7f248c0b7820 2024-03-06T07:27:24.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 >> v2:172.21.15.69:3300/0 conn(0x7f248c0b2d30 msgr2=0x7f248c0b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 --2- 172.21.15.27:0/2073512511 >> v2:172.21.15.69:3300/0 conn(0x7f248c0b2d30 0x7f248c0b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 >> v1:172.21.15.69:6789/0 conn(0x7f248c071540 legacy=0x7f248c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c0d2620 con 0x7f248c0b7820 2024-03-06T07:27:24.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f24840019f0 con 0x7f248c0b7820 2024-03-06T07:27:24.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 >> v1:172.21.15.27:6789/0 conn(0x7f248c0b7820 legacy=0x7f248c0b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2474003a00 0x7f2474003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474004800 0x7f2474004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24740051c0 0x7f247400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c0d3230 con 0x7f24740051c0 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247400c0c0 con 0x7f2474004800 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247400c2c0 con 0x7f2474003a00 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f248b59e640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24740051c0 0x7f247400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.333+0000 7f248a59c640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474004800 0x7f2474004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.334+0000 7f248ad9d640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2474003a00 0x7f2474003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.334+0000 7f248b59e640 1 -- 172.21.15.27:0/2073512511 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2474003a00 msgr2=0x7f2474003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.334+0000 7f248b59e640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2474003a00 0x7f2474003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.334+0000 7f248b59e640 1 -- 172.21.15.27:0/2073512511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474004800 msgr2=0x7f2474004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.334+0000 7f248b59e640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2474004800 0x7f2474004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.334+0000 7f248b59e640 1 -- 172.21.15.27:0/2073512511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2474003520 con 0x7f24740051c0 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.334+0000 7f248ad9d640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2474003a00 0x7f2474003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.334+0000 7f248b59e640 1 -- 172.21.15.27:0/2073512511 reap_dead start 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.334+0000 7f248b59e640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24740051c0 0x7f247400b9c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f248400a7b0 tx=0x7f2474005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb76fcad53e952f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248400a3d0 con 0x7f24740051c0 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2484034670 con 0x7f24740051c0 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2489d9b640 1 -- 172.21.15.27:0/2073512511 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2484034a60 con 0x7f24740051c0 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2491088640 1 -- 172.21.15.27:0/2073512511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24740051c0 msgr2=0x7f247400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2491088640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24740051c0 0x7f247400b9c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f248400a7b0 tx=0x7f2474005830 comp rx=0 tx=0).stop 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2491088640 1 -- 172.21.15.27:0/2073512511 shutdown_connections 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2491088640 1 --2- 172.21.15.27:0/2073512511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24740051c0 0x7f247400b9c0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2491088640 1 -- 172.21.15.27:0/2073512511 >> 172.21.15.27:0/2073512511 conn(0x7f248c0d3d60 msgr2=0x7f248c0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2491088640 1 -- 172.21.15.27:0/2073512511 shutdown_connections 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.335+0000 7f2491088640 1 -- 172.21.15.27:0/2073512511 wait complete. 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.337+0000 7f2491088640 1 Processor -- start 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.337+0000 7f2491088640 1 -- start start 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.337+0000 7f2491088640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24740051c0 0x7f248c0cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.337+0000 7f2491088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f248c14cdf0 0x7f248c14d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.337+0000 7f2491088640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f248c0c7f40 0x7f248c0c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.337+0000 7f2491088640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c0d2ec0 con 0x7f248c0c7f40 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.337+0000 7f2491088640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c0d2be0 con 0x7f24740051c0 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.337+0000 7f2491088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248c0d22f0 con 0x7f248c14cdf0 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f248c0c7f40 0x7f248c0c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f248c0c7f40 0x7f248c0c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36050/0 (socket says 172.21.15.27:36050) 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248b59e640 1 -- 172.21.15.27:0/893257527 learned_addr learned my addr 172.21.15.27:0/893257527 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248a59c640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f248c14cdf0 0x7f248c14d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248ad9d640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24740051c0 0x7f248c0cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248b59e640 1 -- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f248c14cdf0 msgr2=0x7f248c14d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248b59e640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f248c14cdf0 0x7f248c14d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248b59e640 1 -- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24740051c0 msgr2=0x7f248c0cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248b59e640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24740051c0 0x7f248c0cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248b59e640 1 -- 172.21.15.27:0/893257527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c1e8b80 con 0x7f248c0c7f40 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248ad9d640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24740051c0 0x7f248c0cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f248b59e640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f248c0c7f40 0x7f248c0c8380 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f2484005c20 tx=0x7f2484001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f24737fe640 1 -- 172.21.15.27:0/893257527 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2484001250 con 0x7f248c0c7f40 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f24737fe640 1 -- 172.21.15.27:0/893257527 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2484045070 con 0x7f248c0c7f40 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248c1e8e60 con 0x7f248c0c7f40 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f24737fe640 1 -- 172.21.15.27:0/893257527 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2484034c30 con 0x7f248c0c7f40 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.338+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248c0c88c0 con 0x7f248c0c7f40 2024-03-06T07:27:24.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.339+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248c0d0520 con 0x7f248c0c7f40 2024-03-06T07:27:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.341+0000 7f24737fe640 1 -- 172.21.15.27:0/893257527 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f248404a460 con 0x7f248c0c7f40 2024-03-06T07:27:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.341+0000 7f24737fe640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f247806b670 0x7f247806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.341+0000 7f24737fe640 1 -- 172.21.15.27:0/893257527 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24840ac820 con 0x7f248c0c7f40 2024-03-06T07:27:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.341+0000 7f248ad9d640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f247806b670 0x7f247806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.342+0000 7f248ad9d640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f247806b670 0x7f247806db30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2480007fd0 tx=0x7f2480009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.345+0000 7f24737fe640 1 -- 172.21.15.27:0/893257527 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f248407b050 con 0x7f248c0c7f40 2024-03-06T07:27:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.549+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f248c072070 con 0x7f248c0c7f40 2024-03-06T07:27:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.549+0000 7f24737fe640 1 -- 172.21.15.27:0/893257527 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f248407f0f0 con 0x7f248c0c7f40 2024-03-06T07:27:24.552 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:27:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f247806b670 msgr2=0x7f247806db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f247806b670 0x7f247806db30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2480007fd0 tx=0x7f2480009040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f248c0c7f40 msgr2=0x7f248c0c8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f248c0c7f40 0x7f248c0c8380 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f2484005c20 tx=0x7f2484001cf0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 shutdown_connections 2024-03-06T07:27:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f247806b670 0x7f247806db30 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f248c0c7f40 0x7f248c0c8380 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f248c14cdf0 0x7f248c14d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 --2- 172.21.15.27:0/893257527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24740051c0 0x7f248c0cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.551+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 >> 172.21.15.27:0/893257527 conn(0x7f248c0d3d60 msgr2=0x7f248c0b4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.552+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 shutdown_connections 2024-03-06T07:27:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.552+0000 7f2491088640 1 -- 172.21.15.27:0/893257527 wait complete. 2024-03-06T07:27:24.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.672+0000 7f64e8cc9640 1 Processor -- start 2024-03-06T07:27:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.672+0000 7f64e8cc9640 1 -- start start 2024-03-06T07:27:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.673+0000 7f64e8cc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e415dd90 0x7f64e40ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.673+0000 7f64e8cc9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f64e40b6d10 0x7f64e40b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.673+0000 7f64e8cc9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e40d3740 con 0x7f64e40d29d0 2024-03-06T07:27:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.673+0000 7f64e8cc9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e40d3400 con 0x7f64e40b6d10 2024-03-06T07:27:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.673+0000 7f64e8cc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40d3120 con 0x7f64e415dd90 2024-03-06T07:27:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.673+0000 7f64e2d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f64e40b6d10 0x7f64e40b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.673+0000 7f64e2d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f64e40b6d10 0x7f64e40b71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36054/0 (socket says 172.21.15.27:36054) 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.674+0000 7f64e2d9d640 1 -- 172.21.15.27:0/2818314168 learned_addr learned my addr 172.21.15.27:0/2818314168 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.674+0000 7f64e259c640 1 --2- 172.21.15.27:0/2818314168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e415dd90 0x7f64e40ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.674+0000 7f64e2d9d640 1 -- 172.21.15.27:0/2818314168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e415dd90 msgr2=0x7f64e40ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.674+0000 7f64e2d9d640 1 --2- 172.21.15.27:0/2818314168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e415dd90 0x7f64e40ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.674+0000 7f64e2d9d640 1 -- 172.21.15.27:0/2818314168 >> v1:172.21.15.69:6789/0 conn(0x7f64e40d29d0 legacy=0x7f64e40d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.674+0000 7f64e2d9d640 1 -- 172.21.15.27:0/2818314168 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e40d39e0 con 0x7f64e40b6d10 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.674+0000 7f64e259c640 1 --2- 172.21.15.27:0/2818314168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e415dd90 0x7f64e40ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e2d9d640 1 --2- 172.21.15.27:0/2818314168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e40b6d10 0x7f64e40b71a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009e50 tx=0x7f64dc02bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935ee707a9aa0160 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e159a640 1 -- 172.21.15.27:0/2818314168 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc02cb80 con 0x7f64e40b6d10 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e159a640 1 -- 172.21.15.27:0/2818314168 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc02cd20 con 0x7f64e40b6d10 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e159a640 1 -- 172.21.15.27:0/2818314168 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc034610 con 0x7f64e40b6d10 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e8cc9640 1 -- 172.21.15.27:0/2818314168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e40b6d10 msgr2=0x7f64e40b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e8cc9640 1 --2- 172.21.15.27:0/2818314168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e40b6d10 0x7f64e40b71a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009e50 tx=0x7f64dc02bfe0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e8cc9640 1 -- 172.21.15.27:0/2818314168 shutdown_connections 2024-03-06T07:27:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e8cc9640 1 --2- 172.21.15.27:0/2818314168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e40b6d10 0x7f64e40b71a0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e8cc9640 1 --2- 172.21.15.27:0/2818314168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e415dd90 0x7f64e40ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e8cc9640 1 -- 172.21.15.27:0/2818314168 >> 172.21.15.27:0/2818314168 conn(0x7f64e4071960 msgr2=0x7f64e4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.675+0000 7f64e8cc9640 1 -- 172.21.15.27:0/2818314168 shutdown_connections 2024-03-06T07:27:24.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.676+0000 7f64e8cc9640 1 -- 172.21.15.27:0/2818314168 wait complete. 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.677+0000 7f64e8cc9640 1 Processor -- start 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.677+0000 7f64e8cc9640 1 -- start start 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.678+0000 7f64e8cc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64e40b6d10 0x7f64e41583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.678+0000 7f64e8cc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e40d29d0 0x7f64e4157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.678+0000 7f64e8cc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e415dd90 0x7f64e4160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.678+0000 7f64e8cc9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40d2df0 con 0x7f64e415dd90 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.678+0000 7f64e8cc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e415e170 con 0x7f64e40b6d10 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.678+0000 7f64e8cc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40d48f0 con 0x7f64e40d29d0 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.678+0000 7f64e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e415dd90 0x7f64e4160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.678+0000 7f64e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e415dd90 0x7f64e4160db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36064/0 (socket says 172.21.15.27:36064) 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.678+0000 7f64e2d9d640 1 -- 172.21.15.27:0/1389010599 learned_addr learned my addr 172.21.15.27:0/1389010599 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e1d9b640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e40d29d0 0x7f64e4157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e259c640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64e40b6d10 0x7f64e41583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e2d9d640 1 -- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e40d29d0 msgr2=0x7f64e4157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e2d9d640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e40d29d0 0x7f64e4157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e2d9d640 1 -- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64e40b6d10 msgr2=0x7f64e41583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e2d9d640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64e40b6d10 0x7f64e41583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e2d9d640 1 -- 172.21.15.27:0/1389010599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e415f0f0 con 0x7f64e415dd90 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e1d9b640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e40d29d0 0x7f64e4157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e259c640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64e40b6d10 0x7f64e41583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e2d9d640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e415dd90 0x7f64e4160db0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009e50 tx=0x7f64dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64d2ffd640 1 -- 172.21.15.27:0/1389010599 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc043020 con 0x7f64e415dd90 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64d2ffd640 1 -- 172.21.15.27:0/1389010599 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc03e450 con 0x7f64e415dd90 2024-03-06T07:27:24.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e415f3d0 con 0x7f64e415dd90 2024-03-06T07:27:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64d2ffd640 1 -- 172.21.15.27:0/1389010599 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc009510 con 0x7f64e415dd90 2024-03-06T07:27:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.679+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e41f3af0 con 0x7f64e415dd90 2024-03-06T07:27:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.680+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e4072a30 con 0x7f64e415dd90 2024-03-06T07:27:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.681+0000 7f64d2ffd640 1 -- 172.21.15.27:0/1389010599 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64dc046020 con 0x7f64e415dd90 2024-03-06T07:27:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.681+0000 7f64d2ffd640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f64c406b4d0 0x7f64c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.681+0000 7f64d2ffd640 1 -- 172.21.15.27:0/1389010599 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64dc0acb20 con 0x7f64e415dd90 2024-03-06T07:27:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.681+0000 7f64e259c640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f64c406b4d0 0x7f64c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.682+0000 7f64e259c640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f64c406b4d0 0x7f64c406d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f64d400a990 tx=0x7f64d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.686+0000 7f64d2ffd640 1 -- 172.21.15.27:0/1389010599 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64dc04d030 con 0x7f64e415dd90 2024-03-06T07:27:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.868+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f64e404f410 con 0x7f64e415dd90 2024-03-06T07:27:24.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.868+0000 7f64d2ffd640 1 -- 172.21.15.27:0/1389010599 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7f64dc07b9a0 con 0x7f64e415dd90 2024-03-06T07:27:24.871 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f64c406b4d0 msgr2=0x7f64c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f64c406b4d0 0x7f64c406d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f64d400a990 tx=0x7f64d4008040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e415dd90 msgr2=0x7f64e4160db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e415dd90 0x7f64e4160db0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009e50 tx=0x7f64dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 shutdown_connections 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f64c406b4d0 0x7f64c406d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64e415dd90 0x7f64e4160db0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f64e40d29d0 0x7f64e4157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 --2- 172.21.15.27:0/1389010599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64e40b6d10 0x7f64e41583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 >> 172.21.15.27:0/1389010599 conn(0x7f64e4071960 msgr2=0x7f64e40c7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 shutdown_connections 2024-03-06T07:27:24.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.870+0000 7f64e8cc9640 1 -- 172.21.15.27:0/1389010599 wait complete. 2024-03-06T07:27:24.884 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:24.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.991+0000 7fec5d5ac640 1 Processor -- start 2024-03-06T07:27:24.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.991+0000 7fec5d5ac640 1 -- start start 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.992+0000 7fec5d5ac640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec580b3ce0 0x7fec5806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.992+0000 7fec5d5ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec58067410 0x7fec5806e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.992+0000 7fec5d5ac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5806e710 con 0x7fec5815df40 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.992+0000 7fec5d5ac640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5815f000 con 0x7fec580b3ce0 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.992+0000 7fec5d5ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5815e6d0 con 0x7fec58067410 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.992+0000 7fec56ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fec5815df40 0x7fec58066d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39766/0 (socket says 172.21.15.27:39766) 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.992+0000 7fec56ffd640 1 -- 172.21.15.27:0/2618987612 learned_addr learned my addr 172.21.15.27:0/2618987612 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec567fc640 1 --2- 172.21.15.27:0/2618987612 >> v2:172.21.15.69:3300/0 conn(0x7fec580b3ce0 0x7fec5806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec577fe640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec58067410 0x7fec5806e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485925958 0 0) 0x7fec5806e710 con 0x7fec5815df40 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec44003520 con 0x7fec5815df40 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881734127 0 0) 0x7fec44003520 con 0x7fec5815df40 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec58067410 msgr2=0x7fec5806e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec58067410 0x7fec5806e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 >> v2:172.21.15.69:3300/0 conn(0x7fec580b3ce0 msgr2=0x7fec5806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 --2- 172.21.15.27:0/2618987612 >> v2:172.21.15.69:3300/0 conn(0x7fec580b3ce0 0x7fec5806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5815e9b0 con 0x7fec5815df40 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fec40001670 con 0x7fec5815df40 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 >> v1:172.21.15.27:6789/0 conn(0x7fec5815df40 legacy=0x7fec58066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec440039b0 0x7fec44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec44004800 0x7fec44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec58067410 0x7fec440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5815f000 con 0x7fec44004800 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5815e6d0 con 0x7fec440039b0 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4400b9c0 con 0x7fec58067410 2024-03-06T07:27:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec567fc640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec44004800 0x7fec44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec577fe640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec58067410 0x7fec440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.993+0000 7fec56ffd640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec440039b0 0x7fec44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.994+0000 7fec567fc640 1 -- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec58067410 msgr2=0x7fec440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.994+0000 7fec567fc640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec58067410 0x7fec440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.994+0000 7fec567fc640 1 -- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec440039b0 msgr2=0x7fec44003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.994+0000 7fec567fc640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec440039b0 0x7fec44003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.994+0000 7fec567fc640 1 -- 172.21.15.27:0/2618987612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec44003520 con 0x7fec44004800 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.994+0000 7fec567fc640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec44004800 0x7fec44004c80 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fec48009a90 tx=0x7fec44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e1e030bf054f3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec48039070 con 0x7fec44004800 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4802c6b0 con 0x7fec44004800 2024-03-06T07:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec55ffb640 1 -- 172.21.15.27:0/2618987612 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec48009690 con 0x7fec44004800 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec5d5ac640 1 -- 172.21.15.27:0/2618987612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec44004800 msgr2=0x7fec44004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec5d5ac640 1 --2- 172.21.15.27:0/2618987612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec44004800 0x7fec44004c80 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fec48009a90 tx=0x7fec44005830 comp rx=0 tx=0).stop 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec577fe640 1 -- 172.21.15.27:0/2618987612 reap_dead start 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec5d5ac640 1 -- 172.21.15.27:0/2618987612 shutdown_connections 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec5d5ac640 1 -- 172.21.15.27:0/2618987612 >> 172.21.15.27:0/2618987612 conn(0x7fec580d09c0 msgr2=0x7fec580d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec5d5ac640 1 -- 172.21.15.27:0/2618987612 shutdown_connections 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.995+0000 7fec5d5ac640 1 -- 172.21.15.27:0/2618987612 wait complete. 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.996+0000 7fec5d5ac640 1 Processor -- start 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec5d5ac640 1 -- start start 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec5d5ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec44004800 0x7fec58157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec5d5ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec58158530 0x7fec580af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec5d5ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec580afcb0 0x7fec580b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec5d5ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5815e320 con 0x7fec58158530 2024-03-06T07:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec5d5ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5806e1d0 con 0x7fec44004800 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec5d5ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec58158ad0 con 0x7fec580afcb0 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec567fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec58158530 0x7fec580af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec567fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec58158530 0x7fec580af770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36076/0 (socket says 172.21.15.27:36076) 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.997+0000 7fec567fc640 1 -- 172.21.15.27:0/1147318391 learned_addr learned my addr 172.21.15.27:0/1147318391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec567fc640 1 -- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec580afcb0 msgr2=0x7fec580b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec567fc640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec580afcb0 0x7fec580b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec567fc640 1 -- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec44004800 msgr2=0x7fec58157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec56ffd640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec44004800 0x7fec58157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec567fc640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec44004800 0x7fec58157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec567fc640 1 -- 172.21.15.27:0/1147318391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58158db0 con 0x7fec58158530 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec567fc640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec58158530 0x7fec580af770 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fec48000c00 tx=0x7fec48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec4f7fe640 1 -- 172.21.15.27:0/1147318391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec48039050 con 0x7fec58158530 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec4f7fe640 1 -- 172.21.15.27:0/1147318391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec48045070 con 0x7fec58158530 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec4f7fe640 1 -- 172.21.15.27:0/1147318391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec480044e0 con 0x7fec58158530 2024-03-06T07:27:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec58159090 con 0x7fec58158530 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.998+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec58159650 con 0x7fec58158530 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:24.999+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec580733d0 con 0x7fec58158530 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.000+0000 7fec4f7fe640 1 -- 172.21.15.27:0/1147318391 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec48003a50 con 0x7fec58158530 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.000+0000 7fec4f7fe640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec3806b4d0 0x7fec3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.000+0000 7fec4f7fe640 1 -- 172.21.15.27:0/1147318391 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec480abc70 con 0x7fec58158530 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.000+0000 7fec56ffd640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec3806b4d0 0x7fec3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.001+0000 7fec56ffd640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec3806b4d0 0x7fec3806d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fec400014f0 tx=0x7fec40009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.004+0000 7fec4f7fe640 1 -- 172.21.15.27:0/1147318391 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec4804e020 con 0x7fec58158530 2024-03-06T07:27:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.188+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fec580b5710 con 0x7fec58158530 2024-03-06T07:27:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.189+0000 7fec4f7fe640 1 -- 172.21.15.27:0/1147318391 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7fec4807aaf0 con 0x7fec58158530 2024-03-06T07:27:25.191 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec3806b4d0 msgr2=0x7fec3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec3806b4d0 0x7fec3806d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fec400014f0 tx=0x7fec40009040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec58158530 msgr2=0x7fec580af770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec58158530 0x7fec580af770 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fec48000c00 tx=0x7fec48004730 comp rx=0 tx=0).stop 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 shutdown_connections 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fec3806b4d0 0x7fec3806d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec580afcb0 0x7fec580b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec58158530 0x7fec580af770 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 --2- 172.21.15.27:0/1147318391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec44004800 0x7fec58157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 >> 172.21.15.27:0/1147318391 conn(0x7fec580d09c0 msgr2=0x7fec580ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 shutdown_connections 2024-03-06T07:27:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.191+0000 7fec5d5ac640 1 -- 172.21.15.27:0/1147318391 wait complete. 2024-03-06T07:27:25.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:25.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.331+0000 7f9d99fbc640 1 Processor -- start 2024-03-06T07:27:25.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.331+0000 7f9d99fbc640 1 -- start start 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d99fbc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9d94071540 0x7f9d940b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d99fbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940d3400 0x7f9d94071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d99fbc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9d940b2b50 0x7f9d940d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d99fbc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d940d2f70 con 0x7f9d940b2b50 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d99fbc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d940d4be0 con 0x7f9d94071540 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d99fbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d94071e60 con 0x7f9d940d3400 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d93fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9d940b2b50 0x7f9d940d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d93fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9d940b2b50 0x7f9d940d1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36080/0 (socket says 172.21.15.27:36080) 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d93fff640 1 -- 172.21.15.27:0/3531764379 learned_addr learned my addr 172.21.15.27:0/3531764379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d92ffd640 1 --2- 172.21.15.27:0/3531764379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940d3400 0x7f9d94071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d937fe640 1 --2- 172.21.15.27:0/3531764379 >> v2:172.21.15.69:3300/0 conn(0x7f9d94071540 0x7f9d940b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d93fff640 1 -- 172.21.15.27:0/3531764379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940d3400 msgr2=0x7f9d94071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d93fff640 1 --2- 172.21.15.27:0/3531764379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940d3400 0x7f9d94071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d93fff640 1 -- 172.21.15.27:0/3531764379 >> v2:172.21.15.69:3300/0 conn(0x7f9d94071540 msgr2=0x7f9d940b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d93fff640 1 --2- 172.21.15.27:0/3531764379 >> v2:172.21.15.69:3300/0 conn(0x7f9d94071540 0x7f9d940b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.332+0000 7f9d93fff640 1 -- 172.21.15.27:0/3531764379 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940ce610 con 0x7f9d940b2b50 2024-03-06T07:27:25.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.333+0000 7f9d93fff640 1 --2- 172.21.15.27:0/3531764379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940b2b50 0x7f9d940d1870 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f9d940d4da0 tx=0x7f9d8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f49ccccef4a0a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.333+0000 7f9d927fc640 1 -- 172.21.15.27:0/3531764379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c02ca70 con 0x7f9d940b2b50 2024-03-06T07:27:25.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.333+0000 7f9d927fc640 1 -- 172.21.15.27:0/3531764379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c02cc10 con 0x7f9d940b2b50 2024-03-06T07:27:25.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.333+0000 7f9d927fc640 1 -- 172.21.15.27:0/3531764379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0343f0 con 0x7f9d940b2b50 2024-03-06T07:27:25.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.334+0000 7f9d99fbc640 1 -- 172.21.15.27:0/3531764379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940b2b50 msgr2=0x7f9d940d1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.334+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/3531764379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940b2b50 0x7f9d940d1870 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f9d940d4da0 tx=0x7f9d8c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.334+0000 7f9d99fbc640 1 -- 172.21.15.27:0/3531764379 shutdown_connections 2024-03-06T07:27:25.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.334+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/3531764379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940b2b50 0x7f9d940d1870 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.334+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/3531764379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940d3400 0x7f9d94071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.334+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/3531764379 >> v2:172.21.15.69:3300/0 conn(0x7f9d94071540 0x7f9d940b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.334+0000 7f9d99fbc640 1 -- 172.21.15.27:0/3531764379 >> 172.21.15.27:0/3531764379 conn(0x7f9d940d3ca0 msgr2=0x7f9d940d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.334+0000 7f9d99fbc640 1 -- 172.21.15.27:0/3531764379 shutdown_connections 2024-03-06T07:27:25.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.334+0000 7f9d99fbc640 1 -- 172.21.15.27:0/3531764379 wait complete. 2024-03-06T07:27:25.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.335+0000 7f9d99fbc640 1 Processor -- start 2024-03-06T07:27:25.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.335+0000 7f9d99fbc640 1 -- start start 2024-03-06T07:27:25.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d99fbc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d94071540 0x7f9d940b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d99fbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940b2b50 0x7f9d9415fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d99fbc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940d3400 0x7f9d9414ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d99fbc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d940ce8f0 con 0x7f9d940d3400 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d99fbc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d940d2c00 con 0x7f9d94071540 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d99fbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9415e370 con 0x7f9d940b2b50 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d93fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940d3400 0x7f9d9414ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d93fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940d3400 0x7f9d9414ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36088/0 (socket says 172.21.15.27:36088) 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d93fff640 1 -- 172.21.15.27:0/2672372624 learned_addr learned my addr 172.21.15.27:0/2672372624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d92ffd640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940b2b50 0x7f9d9415fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d937fe640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d94071540 0x7f9d940b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d93fff640 1 -- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940b2b50 msgr2=0x7f9d9415fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d93fff640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940b2b50 0x7f9d9415fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d93fff640 1 -- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d94071540 msgr2=0x7f9d940b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d93fff640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d94071540 0x7f9d940b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.336+0000 7f9d93fff640 1 -- 172.21.15.27:0/2672372624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940b5f30 con 0x7f9d940d3400 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.337+0000 7f9d93fff640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940d3400 0x7f9d9414ecc0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c036040 tx=0x7f9d8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.337+0000 7f9d8bfff640 1 -- 172.21.15.27:0/2672372624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c043020 con 0x7f9d940d3400 2024-03-06T07:27:25.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.337+0000 7f9d8bfff640 1 -- 172.21.15.27:0/2672372624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c03e450 con 0x7f9d940d3400 2024-03-06T07:27:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.337+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d94158d60 con 0x7f9d940d3400 2024-03-06T07:27:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.337+0000 7f9d8bfff640 1 -- 172.21.15.27:0/2672372624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c009510 con 0x7f9d940d3400 2024-03-06T07:27:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.337+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d94159320 con 0x7f9d940d3400 2024-03-06T07:27:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.338+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d94068700 con 0x7f9d940d3400 2024-03-06T07:27:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.338+0000 7f9d8bfff640 1 -- 172.21.15.27:0/2672372624 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d8c046020 con 0x7f9d940d3400 2024-03-06T07:27:25.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.339+0000 7f9d8bfff640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9d7406b4d0 0x7f9d7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.339+0000 7f9d8bfff640 1 -- 172.21.15.27:0/2672372624 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d8c0ac770 con 0x7f9d940d3400 2024-03-06T07:27:25.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.339+0000 7f9d937fe640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9d7406b4d0 0x7f9d7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.340+0000 7f9d937fe640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9d7406b4d0 0x7f9d7406d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009ac0 tx=0x7f9d84009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.343+0000 7f9d8bfff640 1 -- 172.21.15.27:0/2672372624 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d8c04d030 con 0x7f9d940d3400 2024-03-06T07:27:25.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.545+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9d941596c0 con 0x7f9d940d3400 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.545+0000 7f9d8bfff640 1 -- 172.21.15.27:0/2672372624 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v84) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9d940b5f30 con 0x7f9d940d3400 2024-03-06T07:27:25.548 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:27:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9d7406b4d0 msgr2=0x7f9d7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9d7406b4d0 0x7f9d7406d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009ac0 tx=0x7f9d84009190 comp rx=0 tx=0).stop 2024-03-06T07:27:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940d3400 msgr2=0x7f9d9414ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940d3400 0x7f9d9414ecc0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c036040 tx=0x7f9d8c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 shutdown_connections 2024-03-06T07:27:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9d7406b4d0 0x7f9d7406d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d940d3400 0x7f9d9414ecc0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9d940b2b50 0x7f9d9415fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 --2- 172.21.15.27:0/2672372624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d94071540 0x7f9d940b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 >> 172.21.15.27:0/2672372624 conn(0x7f9d940d3ca0 msgr2=0x7f9d940ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 shutdown_connections 2024-03-06T07:27:25.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:25.547+0000 7f9d99fbc640 1 -- 172.21.15.27:0/2672372624 wait complete. 2024-03-06T07:27:25.560 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:25.590 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[66106]: fuse finished with error 0 and tester_r 0 2024-03-06T07:27:25.591 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:27:25.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.591 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:27:25.680 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:27:25.680 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:25.699 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:25.699 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:27:25.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.699 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:27:25.756 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:27:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:25.757 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:27:25.757 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:25.772 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:25.772 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:27:25.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.773 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:27:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:27:25.828 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:27:25.829 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:25.846 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:25.846 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:27:25.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.846 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:27:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:25.904 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:27:25.904 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.010+0000 7fe86f064640 1 Processor -- start 2024-03-06T07:27:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.010+0000 7fe86f064640 1 -- start start 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe86f064640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8680b4260 0x7fe8680cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe86f064640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 0x7fe8680b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe86f064640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8680b0dd0 con 0x7fe8680d02e0 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe86f064640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8680b0b80 con 0x7fe8680b4260 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe86f064640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8680aeec0 con 0x7fe8680d06c0 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe867fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8680b4260 0x7fe8680cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe867fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8680b4260 0x7fe8680cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36096/0 (socket says 172.21.15.27:36096) 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe867fff640 1 -- 172.21.15.27:0/4030738715 learned_addr learned my addr 172.21.15.27:0/4030738715 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe86d601640 1 --2- 172.21.15.27:0/4030738715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 0x7fe8680b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe867fff640 1 -- 172.21.15.27:0/4030738715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 msgr2=0x7fe8680b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe867fff640 1 --2- 172.21.15.27:0/4030738715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 0x7fe8680b1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.011+0000 7fe867fff640 1 -- 172.21.15.27:0/4030738715 >> v1:172.21.15.69:6789/0 conn(0x7fe8680d02e0 legacy=0x7fe8680b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.012+0000 7fe867fff640 1 -- 172.21.15.27:0/4030738715 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8680b10b0 con 0x7fe8680b4260 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.012+0000 7fe867fff640 1 --2- 172.21.15.27:0/4030738715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680b4260 0x7fe8680cfa00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fe854009870 tx=0x7fe85402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577070e6d0483810 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.012+0000 7fe8677fe640 1 -- 172.21.15.27:0/4030738715 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854034070 con 0x7fe8680b4260 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.012+0000 7fe8677fe640 1 -- 172.21.15.27:0/4030738715 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe854030840 con 0x7fe8680b4260 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.012+0000 7fe8677fe640 1 -- 172.21.15.27:0/4030738715 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854030c10 con 0x7fe8680b4260 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.013+0000 7fe86f064640 1 -- 172.21.15.27:0/4030738715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680b4260 msgr2=0x7fe8680cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.013+0000 7fe86f064640 1 --2- 172.21.15.27:0/4030738715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680b4260 0x7fe8680cfa00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fe854009870 tx=0x7fe85402b980 comp rx=0 tx=0).stop 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.013+0000 7fe86f064640 1 -- 172.21.15.27:0/4030738715 shutdown_connections 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.013+0000 7fe86f064640 1 --2- 172.21.15.27:0/4030738715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680b4260 0x7fe8680cfa00 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.013+0000 7fe86f064640 1 --2- 172.21.15.27:0/4030738715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 0x7fe8680b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.013+0000 7fe86f064640 1 -- 172.21.15.27:0/4030738715 >> 172.21.15.27:0/4030738715 conn(0x7fe8680b36d0 msgr2=0x7fe8680b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.013+0000 7fe86f064640 1 -- 172.21.15.27:0/4030738715 shutdown_connections 2024-03-06T07:27:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.013+0000 7fe86f064640 1 -- 172.21.15.27:0/4030738715 wait complete. 2024-03-06T07:27:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.015+0000 7fe86f064640 1 Processor -- start 2024-03-06T07:27:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.015+0000 7fe86f064640 1 -- start start 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe86f064640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8680b4260 0x7fe8681f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe86f064640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680d02e0 0x7fe8680c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe86f064640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 0x7fe8680ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe86f064640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8680b22d0 con 0x7fe8680d02e0 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe86f064640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8680b2150 con 0x7fe8680b4260 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe86f064640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8680af350 con 0x7fe8680d06c0 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe867fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680d02e0 0x7fe8680c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe867fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680d02e0 0x7fe8680c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36100/0 (socket says 172.21.15.27:36100) 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe867fff640 1 -- 172.21.15.27:0/2881674078 learned_addr learned my addr 172.21.15.27:0/2881674078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe86ce00640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8680b4260 0x7fe8681f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe86d601640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 0x7fe8680ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe867fff640 1 -- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 msgr2=0x7fe8680ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe867fff640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 0x7fe8680ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe867fff640 1 -- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8680b4260 msgr2=0x7fe8681f5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe867fff640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8680b4260 0x7fe8681f5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe867fff640 1 -- 172.21.15.27:0/2881674078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8680ca550 con 0x7fe8680d02e0 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.017+0000 7fe867fff640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680d02e0 0x7fe8680c9de0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fe854000c00 tx=0x7fe854004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.018+0000 7fe8657fa640 1 -- 172.21.15.27:0/2881674078 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854048070 con 0x7fe8680d02e0 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.018+0000 7fe8657fa640 1 -- 172.21.15.27:0/2881674078 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe854030640 con 0x7fe8680d02e0 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.018+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8680ca830 con 0x7fe8680d02e0 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.018+0000 7fe8657fa640 1 -- 172.21.15.27:0/2881674078 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854034040 con 0x7fe8680d02e0 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.018+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8680cadf0 con 0x7fe8680d02e0 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.019+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe868068700 con 0x7fe8680d02e0 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.019+0000 7fe8657fa640 1 -- 172.21.15.27:0/2881674078 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe85404b070 con 0x7fe8680d02e0 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.020+0000 7fe8657fa640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe84c06b4d0 0x7fe84c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.020+0000 7fe8657fa640 1 -- 172.21.15.27:0/2881674078 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8540ac280 con 0x7fe8680d02e0 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.020+0000 7fe86ce00640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe84c06b4d0 0x7fe84c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.021+0000 7fe86ce00640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe84c06b4d0 0x7fe84c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe85c004f60 tx=0x7fe85c007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.024+0000 7fe8657fa640 1 -- 172.21.15.27:0/2881674078 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe85407aaa0 con 0x7fe8680d02e0 2024-03-06T07:27:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.246+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe868158c50 con 0x7fe8680d02e0 2024-03-06T07:27:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.247+0000 7fe8657fa640 1 -- 172.21.15.27:0/2881674078 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fe85407eb40 con 0x7fe8680d02e0 2024-03-06T07:27:26.250 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:26.250 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/930650799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":930650799},{"type":"v1","addr":"172.21.15.69:6836","nonce":930650799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1050755919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1050755919},{"type":"v1","addr":"172.21.15.27:6835","nonce":1050755919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1173605349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1173605349},{"type":"v1","addr":"172.21.15.69:6837","nonce":1173605349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:09.729142+0000","modified":"2024-03-06T07:27:12.757645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7032},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7032":{"gid":7032,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3587849188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3587849188},{"type":"v1","addr":"172.21.15.27:6834","nonce":3587849188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7032,"qdb_cluster":[7032]},"id":6}]} 2024-03-06T07:27:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 52 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe84c06b4d0 msgr2=0x7fe84c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe84c06b4d0 0x7fe84c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe85c004f60 tx=0x7fe85c007e30 comp rx=0 tx=0).stop 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680d02e0 msgr2=0x7fe8680c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680d02e0 0x7fe8680c9de0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fe854000c00 tx=0x7fe854004730 comp rx=0 tx=0).stop 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 shutdown_connections 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe84c06b4d0 0x7fe84c06d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8680d06c0 0x7fe8680ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8680d02e0 0x7fe8680c9de0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 --2- 172.21.15.27:0/2881674078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8680b4260 0x7fe8681f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 >> 172.21.15.27:0/2881674078 conn(0x7fe8680b36d0 msgr2=0x7fe8680b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 shutdown_connections 2024-03-06T07:27:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.249+0000 7fe86f064640 1 -- 172.21.15.27:0/2881674078 wait complete. 2024-03-06T07:27:26.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.367+0000 7f9b10a1c640 1 Processor -- start 2024-03-06T07:27:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.367+0000 7f9b10a1c640 1 -- start start 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b10a1c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b0c0b95a0 0x7f9b0c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b10a1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b10a1c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c0cea30 con 0x7f9b0c15df00 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b10a1c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0c0cf3c0 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b10a1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0cf0e0 con 0x7f9b0c0b9a10 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b09d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b0c0b95a0 0x7f9b0c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b0a59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9b0c15df00 0x7f9b0c0d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39778/0 (socket says 172.21.15.27:39778) 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b0a59c640 1 -- 172.21.15.27:0/1740262326 learned_addr learned my addr 172.21.15.27:0/1740262326 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b0ad9d640 1 --2- 172.21.15.27:0/1740262326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b09d9b640 1 -- 172.21.15.27:0/1740262326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 msgr2=0x7f9b0c0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b09d9b640 1 --2- 172.21.15.27:0/1740262326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b09d9b640 1 -- 172.21.15.27:0/1740262326 >> v1:172.21.15.27:6789/0 conn(0x7f9b0c15df00 legacy=0x7f9b0c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b09d9b640 1 -- 172.21.15.27:0/1740262326 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c0ce470 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b0959a640 1 -- 172.21.15.27:0/1740262326 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845816042 0 0) 0x7f9b0c0cea30 con 0x7f9b0c15df00 2024-03-06T07:27:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.368+0000 7f9b0ad9d640 1 --2- 172.21.15.27:0/1740262326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.369+0000 7f9b09d9b640 1 --2- 172.21.15.27:0/1740262326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 0x7f9b0c0b5700 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f9af40097a0 tx=0x7f9af402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f86875054263e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.369+0000 7f9b0959a640 1 -- 172.21.15.27:0/1740262326 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4035020 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.369+0000 7f9b0959a640 1 -- 172.21.15.27:0/1740262326 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4031830 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.369+0000 7f9b0959a640 1 -- 172.21.15.27:0/1740262326 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4031c00 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.370+0000 7f9b10a1c640 1 -- 172.21.15.27:0/1740262326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 msgr2=0x7f9b0c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.370+0000 7f9b10a1c640 1 --2- 172.21.15.27:0/1740262326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 0x7f9b0c0b5700 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f9af40097a0 tx=0x7f9af402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.370+0000 7f9b10a1c640 1 -- 172.21.15.27:0/1740262326 shutdown_connections 2024-03-06T07:27:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.370+0000 7f9b10a1c640 1 --2- 172.21.15.27:0/1740262326 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.370+0000 7f9b10a1c640 1 --2- 172.21.15.27:0/1740262326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 0x7f9b0c0b5700 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.370+0000 7f9b10a1c640 1 -- 172.21.15.27:0/1740262326 >> 172.21.15.27:0/1740262326 conn(0x7f9b0c071960 msgr2=0x7f9b0c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.370+0000 7f9b10a1c640 1 -- 172.21.15.27:0/1740262326 shutdown_connections 2024-03-06T07:27:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.370+0000 7f9b10a1c640 1 -- 172.21.15.27:0/1740262326 wait complete. 2024-03-06T07:27:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.371+0000 7f9b10a1c640 1 Processor -- start 2024-03-06T07:27:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.371+0000 7f9b10a1c640 1 -- start start 2024-03-06T07:27:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b10a1c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 0x7f9b0c14c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b10a1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c15f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b10a1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c15df00 0x7f9b0c0aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b10a1c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0cedf0 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b10a1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0cec70 con 0x7f9b0c15df00 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b10a1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0d3220 con 0x7f9b0c0b9a10 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 0x7f9b0c14c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 0x7f9b0c14c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36120/0 (socket says 172.21.15.27:36120) 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0a59c640 1 -- 172.21.15.27:0/589037554 learned_addr learned my addr 172.21.15.27:0/589037554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b09d9b640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c15f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0ad9d640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c15df00 0x7f9b0c0aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0a59c640 1 -- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 msgr2=0x7f9b0c15f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0a59c640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c15f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0a59c640 1 -- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c15df00 msgr2=0x7f9b0c0aa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0a59c640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c15df00 0x7f9b0c0aa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0a59c640 1 -- 172.21.15.27:0/589037554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c15fd80 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b09d9b640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c15f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.372+0000 7f9b0ad9d640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c15df00 0x7f9b0c0aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.373+0000 7f9b0a59c640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 0x7f9b0c14c7c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00c460 tx=0x7f9afc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.373+0000 7f9b02ffd640 1 -- 172.21.15.27:0/589037554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc018070 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.373+0000 7f9b02ffd640 1 -- 172.21.15.27:0/589037554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc007500 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.373+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c160060 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.373+0000 7f9b02ffd640 1 -- 172.21.15.27:0/589037554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc004a20 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.373+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c160620 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.374+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0c067ff0 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.374+0000 7f9b02ffd640 1 -- 172.21.15.27:0/589037554 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9afc01b070 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.375+0000 7f9b02ffd640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9aec06b4d0 0x7f9aec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.375+0000 7f9b02ffd640 1 -- 172.21.15.27:0/589037554 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9afc0894d0 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.375+0000 7f9b09d9b640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9aec06b4d0 0x7f9aec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.376+0000 7f9b09d9b640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9aec06b4d0 0x7f9aec06d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9af4036000 tx=0x7f9af4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.381+0000 7f9b02ffd640 1 -- 172.21.15.27:0/589037554 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9afc058350 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.570+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b0c15d5b0 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.571+0000 7f9b02ffd640 1 -- 172.21.15.27:0/589037554 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16588 (secure 0 0 0) 0x7f9afc087e20 con 0x7f9b0c0b95a0 2024-03-06T07:27:26.574 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:26.574 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":84,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:11.381712+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:07.388472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:08.720233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9aec06b4d0 msgr2=0x7f9aec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9aec06b4d0 0x7f9aec06d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9af4036000 tx=0x7f9af4008030 comp rx=0 tx=0).stop 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 msgr2=0x7f9b0c14c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 0x7f9b0c14c7c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00c460 tx=0x7f9afc00c920 comp rx=0 tx=0).stop 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 shutdown_connections 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9aec06b4d0 0x7f9aec06d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b0c15df00 0x7f9b0c0aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b0c0b9a10 0x7f9b0c15f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 --2- 172.21.15.27:0/589037554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0b95a0 0x7f9b0c14c7c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 >> 172.21.15.27:0/589037554 conn(0x7f9b0c071960 msgr2=0x7f9b0c0ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 shutdown_connections 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.573+0000 7f9b10a1c640 1 -- 172.21.15.27:0/589037554 wait complete. 2024-03-06T07:27:26.587 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:07.388472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:08.720233+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:26.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:26.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.694+0000 7fc5cd93e640 1 Processor -- start 2024-03-06T07:27:26.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.694+0000 7fc5cd93e640 1 -- start start 2024-03-06T07:27:26.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.695+0000 7fc5cd93e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc5c8071540 0x7fc5c8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.695+0000 7fc5cd93e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5c80cee80 0x7fc5c8168900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.695+0000 7fc5cd93e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c8071e60 con 0x7fc5c80b2d30 2024-03-06T07:27:26.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.695+0000 7fc5cd93e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c80d3230 con 0x7fc5c80cee80 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.695+0000 7fc5cd93e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c80d4960 con 0x7fc5c8071540 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.695+0000 7fc5c77fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5c80cee80 0x7fc5c8168900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.695+0000 7fc5c77fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5c80cee80 0x7fc5c8168900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36132/0 (socket says 172.21.15.27:36132) 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.695+0000 7fc5be7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc5c80b2d30 0x7fc5c80ce770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39788/0 (socket says 172.21.15.27:39788) 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.695+0000 7fc5c77fe640 1 -- 172.21.15.27:0/3916087817 learned_addr learned my addr 172.21.15.27:0/3916087817 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.696+0000 7fc5c6ffd640 1 --2- 172.21.15.27:0/3916087817 >> v2:172.21.15.69:3300/0 conn(0x7fc5c8071540 0x7fc5c8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.696+0000 7fc5c77fe640 1 -- 172.21.15.27:0/3916087817 >> v2:172.21.15.69:3300/0 conn(0x7fc5c8071540 msgr2=0x7fc5c8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.696+0000 7fc5c77fe640 1 --2- 172.21.15.27:0/3916087817 >> v2:172.21.15.69:3300/0 conn(0x7fc5c8071540 0x7fc5c8071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.696+0000 7fc5c77fe640 1 -- 172.21.15.27:0/3916087817 >> v1:172.21.15.27:6789/0 conn(0x7fc5c80b2d30 legacy=0x7fc5c80ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.696+0000 7fc5c77fe640 1 -- 172.21.15.27:0/3916087817 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80d36c0 con 0x7fc5c80cee80 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.696+0000 7fc5c67fc640 1 -- 172.21.15.27:0/3916087817 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894091025 0 0) 0x7fc5c8071e60 con 0x7fc5c80b2d30 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.697+0000 7fc5c77fe640 1 --2- 172.21.15.27:0/3916087817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c80cee80 0x7fc5c8168900 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fc5c808ca00 tx=0x7fc5c002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f00bc66b6c52e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.697+0000 7fc5c67fc640 1 -- 172.21.15.27:0/3916087817 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c002cb20 con 0x7fc5c80cee80 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.697+0000 7fc5c67fc640 1 -- 172.21.15.27:0/3916087817 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c002ccc0 con 0x7fc5c80cee80 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.697+0000 7fc5c67fc640 1 -- 172.21.15.27:0/3916087817 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0034610 con 0x7fc5c80cee80 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.698+0000 7fc5cd93e640 1 -- 172.21.15.27:0/3916087817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c80cee80 msgr2=0x7fc5c8168900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.698+0000 7fc5cd93e640 1 --2- 172.21.15.27:0/3916087817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c80cee80 0x7fc5c8168900 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fc5c808ca00 tx=0x7fc5c002bf80 comp rx=0 tx=0).stop 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.698+0000 7fc5cd93e640 1 -- 172.21.15.27:0/3916087817 shutdown_connections 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.698+0000 7fc5cd93e640 1 --2- 172.21.15.27:0/3916087817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c80cee80 0x7fc5c8168900 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.698+0000 7fc5cd93e640 1 --2- 172.21.15.27:0/3916087817 >> v2:172.21.15.69:3300/0 conn(0x7fc5c8071540 0x7fc5c8071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.698+0000 7fc5cd93e640 1 -- 172.21.15.27:0/3916087817 >> 172.21.15.27:0/3916087817 conn(0x7fc5c80d3d60 msgr2=0x7fc5c80d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.698+0000 7fc5cd93e640 1 -- 172.21.15.27:0/3916087817 shutdown_connections 2024-03-06T07:27:26.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.698+0000 7fc5cd93e640 1 -- 172.21.15.27:0/3916087817 wait complete. 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.699+0000 7fc5cd93e640 1 Processor -- start 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.699+0000 7fc5cd93e640 1 -- start start 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5cd93e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8071540 0x7fc5c8156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5cd93e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5c80b2d30 0x7fc5c81f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5cd93e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5c80cee80 0x7fc5c81595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5cd93e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80d2be0 con 0x7fc5c8071540 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5cd93e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80d2900 con 0x7fc5c80cee80 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5cd93e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80d2ec0 con 0x7fc5c80b2d30 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5c6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8071540 0x7fc5c8156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5c6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8071540 0x7fc5c8156e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36136/0 (socket says 172.21.15.27:36136) 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5c6ffd640 1 -- 172.21.15.27:0/2999327664 learned_addr learned my addr 172.21.15.27:0/2999327664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5c77fe640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5c80cee80 0x7fc5c81595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5be7fc640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5c80b2d30 0x7fc5c81f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5c6ffd640 1 -- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5c80b2d30 msgr2=0x7fc5c81f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5c6ffd640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5c80b2d30 0x7fc5c81f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5c6ffd640 1 -- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5c80cee80 msgr2=0x7fc5c81595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5c6ffd640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5c80cee80 0x7fc5c81595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5c6ffd640 1 -- 172.21.15.27:0/2999327664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c8156770 con 0x7fc5c8071540 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.700+0000 7fc5be7fc640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5c80b2d30 0x7fc5c81f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.701+0000 7fc5c6ffd640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8071540 0x7fc5c8156e20 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c7f0 tx=0x7fc5b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.701+0000 7fc5c4ff9640 1 -- 172.21.15.27:0/2999327664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8015070 con 0x7fc5c8071540 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.701+0000 7fc5c4ff9640 1 -- 172.21.15.27:0/2999327664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b8004220 con 0x7fc5c8071540 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.701+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c8157360 con 0x7fc5c8071540 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.701+0000 7fc5c4ff9640 1 -- 172.21.15.27:0/2999327664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8007620 con 0x7fc5c8071540 2024-03-06T07:27:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.701+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c80cc3f0 con 0x7fc5c8071540 2024-03-06T07:27:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.702+0000 7fc5c4ff9640 1 -- 172.21.15.27:0/2999327664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5b80079d0 con 0x7fc5c8071540 2024-03-06T07:27:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.702+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c8148460 con 0x7fc5c8071540 2024-03-06T07:27:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.703+0000 7fc5c4ff9640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.703+0000 7fc5c4ff9640 1 -- 172.21.15.27:0/2999327664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5b808a480 con 0x7fc5c8071540 2024-03-06T07:27:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.703+0000 7fc5be7fc640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.704+0000 7fc5be7fc640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8156dd0 tx=0x7fc5b0007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.707+0000 7fc5c4ff9640 1 -- 172.21.15.27:0/2999327664 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5c8148460 con 0x7fc5c8071540 2024-03-06T07:27:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.889+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5c8072070 con 0x7fc5c8071540 2024-03-06T07:27:26.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.889+0000 7fc5c4ff9640 1 -- 172.21.15.27:0/2999327664 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5b8058cb0 con 0x7fc5c8071540 2024-03-06T07:27:26.892 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:27:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc5ac06b4d0 msgr2=0x7fc5ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8156dd0 tx=0x7fc5b0007ca0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8071540 msgr2=0x7fc5c8156e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8071540 0x7fc5c8156e20 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c7f0 tx=0x7fc5b800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 shutdown_connections 2024-03-06T07:27:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc5ac06b4d0 0x7fc5ac06d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5c80cee80 0x7fc5c81595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc5c80b2d30 0x7fc5c81f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 --2- 172.21.15.27:0/2999327664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8071540 0x7fc5c8156e20 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.892+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 >> 172.21.15.27:0/2999327664 conn(0x7fc5c80d3d60 msgr2=0x7fc5c815bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.893+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 shutdown_connections 2024-03-06T07:27:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:26.893+0000 7fc5cd93e640 1 -- 172.21.15.27:0/2999327664 wait complete. 2024-03-06T07:27:26.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.013+0000 7f573d478640 1 Processor -- start 2024-03-06T07:27:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.013+0000 7f573d478640 1 -- start start 2024-03-06T07:27:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f573d478640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f57380b4840 0x7f57380b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f573d478640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57380b1420 0x7f573806d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f573d478640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5738161b50 con 0x7f5738146fd0 2024-03-06T07:27:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f573d478640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57380b1890 con 0x7f57380b4840 2024-03-06T07:27:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f573d478640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5738161900 con 0x7f57380b1420 2024-03-06T07:27:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f5736ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5738146fd0 0x7f573806ef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39796/0 (socket says 172.21.15.27:39796) 2024-03-06T07:27:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f5736ffd640 1 -- 172.21.15.27:0/2778953916 learned_addr learned my addr 172.21.15.27:0/2778953916 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f57367fc640 1 --2- 172.21.15.27:0/2778953916 >> v2:172.21.15.69:3300/0 conn(0x7f57380b4840 0x7f57380b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f57377fe640 1 --2- 172.21.15.27:0/2778953916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57380b1420 0x7f573806d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f5735ffb640 1 -- 172.21.15.27:0/2778953916 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455149971 0 0) 0x7f5738161b50 con 0x7f5738146fd0 2024-03-06T07:27:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f5735ffb640 1 -- 172.21.15.27:0/2778953916 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5720003520 con 0x7f5738146fd0 2024-03-06T07:27:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f57367fc640 1 -- 172.21.15.27:0/2778953916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57380b1420 msgr2=0x7f573806d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f57367fc640 1 --2- 172.21.15.27:0/2778953916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57380b1420 0x7f573806d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f57367fc640 1 -- 172.21.15.27:0/2778953916 >> v1:172.21.15.27:6789/0 conn(0x7f5738146fd0 legacy=0x7f573806ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.014+0000 7f57367fc640 1 -- 172.21.15.27:0/2778953916 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5738161e30 con 0x7f57380b4840 2024-03-06T07:27:27.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.015+0000 7f57367fc640 1 --2- 172.21.15.27:0/2778953916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57380b4840 0x7f57380b0ee0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f572c0097a0 tx=0x7f572c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=413f6ca8d0f0ebbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.015+0000 7f5735ffb640 1 -- 172.21.15.27:0/2778953916 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c035020 con 0x7f57380b4840 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.015+0000 7f5735ffb640 1 -- 172.21.15.27:0/2778953916 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c031830 con 0x7f57380b4840 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.015+0000 7f5735ffb640 1 -- 172.21.15.27:0/2778953916 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c031be0 con 0x7f57380b4840 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.016+0000 7f573d478640 1 -- 172.21.15.27:0/2778953916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57380b4840 msgr2=0x7f57380b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.016+0000 7f573d478640 1 --2- 172.21.15.27:0/2778953916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57380b4840 0x7f57380b0ee0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f572c0097a0 tx=0x7f572c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.016+0000 7f573d478640 1 -- 172.21.15.27:0/2778953916 shutdown_connections 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.016+0000 7f573d478640 1 --2- 172.21.15.27:0/2778953916 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57380b1420 0x7f573806d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.016+0000 7f573d478640 1 --2- 172.21.15.27:0/2778953916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57380b4840 0x7f57380b0ee0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.016+0000 7f573d478640 1 -- 172.21.15.27:0/2778953916 >> 172.21.15.27:0/2778953916 conn(0x7f57380afac0 msgr2=0x7f5738072d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.016+0000 7f573d478640 1 -- 172.21.15.27:0/2778953916 shutdown_connections 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.016+0000 7f573d478640 1 -- 172.21.15.27:0/2778953916 wait complete. 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.017+0000 7f573d478640 1 Processor -- start 2024-03-06T07:27:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.017+0000 7f573d478640 1 -- start start 2024-03-06T07:27:27.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f573d478640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57380b1420 0x7f57381f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f573d478640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57380b4840 0x7f573815fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f573d478640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5738146fd0 0x7f5738072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f573d478640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573806f620 con 0x7f57380b4840 2024-03-06T07:27:27.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f573d478640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57381623f0 con 0x7f57380b1420 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f573d478640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57381473b0 con 0x7f5738146fd0 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f57367fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57380b4840 0x7f573815fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f57367fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57380b4840 0x7f573815fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36144/0 (socket says 172.21.15.27:36144) 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f57367fc640 1 -- 172.21.15.27:0/2237480435 learned_addr learned my addr 172.21.15.27:0/2237480435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f57377fe640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5738146fd0 0x7f5738072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f5736ffd640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57380b1420 0x7f57381f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f57367fc640 1 -- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5738146fd0 msgr2=0x7f5738072370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f57367fc640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5738146fd0 0x7f5738072370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f57367fc640 1 -- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57380b1420 msgr2=0x7f57381f6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f57367fc640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57380b1420 0x7f57381f6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.018+0000 7f57367fc640 1 -- 172.21.15.27:0/2237480435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57381f75f0 con 0x7f57380b4840 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.019+0000 7f57367fc640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57380b4840 0x7f573815fd30 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f572c0098c0 tx=0x7f572c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.019+0000 7f571f7fe640 1 -- 172.21.15.27:0/2237480435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c035040 con 0x7f57380b4840 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.019+0000 7f571f7fe640 1 -- 172.21.15.27:0/2237480435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c03b900 con 0x7f57380b4840 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.019+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57381f78d0 con 0x7f57380b4840 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.019+0000 7f571f7fe640 1 -- 172.21.15.27:0/2237480435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c03bcf0 con 0x7f57380b4840 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.019+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57381f7bb0 con 0x7f57380b4840 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.020+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5738073410 con 0x7f57380b4840 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.020+0000 7f571f7fe640 1 -- 172.21.15.27:0/2237480435 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f572c038020 con 0x7f57380b4840 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.021+0000 7f571f7fe640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f572406b4d0 0x7f572406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.021+0000 7f571f7fe640 1 -- 172.21.15.27:0/2237480435 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f572c0ac050 con 0x7f57380b4840 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.021+0000 7f5736ffd640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f572406b4d0 0x7f572406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.021+0000 7f5736ffd640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f572406b4d0 0x7f572406d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5728005880 tx=0x7f5728009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.025+0000 7f571f7fe640 1 -- 172.21.15.27:0/2237480435 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f572c07a910 con 0x7f57380b4840 2024-03-06T07:27:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.204+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5738142400 con 0x7f57380b4840 2024-03-06T07:27:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.204+0000 7f571f7fe640 1 -- 172.21.15.27:0/2237480435 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7f572c07e9b0 con 0x7f57380b4840 2024-03-06T07:27:27.207 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:27:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f572406b4d0 msgr2=0x7f572406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f572406b4d0 0x7f572406d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5728005880 tx=0x7f5728009040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57380b4840 msgr2=0x7f573815fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57380b4840 0x7f573815fd30 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f572c0098c0 tx=0x7f572c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 shutdown_connections 2024-03-06T07:27:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f572406b4d0 0x7f572406d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5738146fd0 0x7f5738072370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57380b4840 0x7f573815fd30 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 --2- 172.21.15.27:0/2237480435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57380b1420 0x7f57381f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 >> 172.21.15.27:0/2237480435 conn(0x7f57380afac0 msgr2=0x7f5738141850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 shutdown_connections 2024-03-06T07:27:27.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.206+0000 7f573d478640 1 -- 172.21.15.27:0/2237480435 wait complete. 2024-03-06T07:27:27.220 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.328+0000 7fccabb8a640 1 Processor -- start 2024-03-06T07:27:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.328+0000 7fccabb8a640 1 -- start start 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fccabb8a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcca40ceea0 0x7fcca40d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fccabb8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca40b6d10 0x7fcca40b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fccabb8a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca40d3740 con 0x7fcca415dd90 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fccabb8a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca40d3400 con 0x7fcca40ceea0 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fccabb8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40d3120 con 0x7fcca40b6d10 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca9926640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcca415dd90 0x7fcca40d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39810/0 (socket says 172.21.15.27:39810) 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca9926640 1 -- 172.21.15.27:0/1948782962 learned_addr learned my addr 172.21.15.27:0/1948782962 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fccaa127640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca40b6d10 0x7fcca40b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960881956 0 0) 0x7fcca40d3740 con 0x7fcca415dd90 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca9125640 1 --2- 172.21.15.27:0/1948782962 >> v2:172.21.15.69:3300/0 conn(0x7fcca40ceea0 0x7fcca40d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc90003520 con 0x7fcca415dd90 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4201746908 0 0) 0x7fcc90003520 con 0x7fcca415dd90 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca40b6d10 msgr2=0x7fcca40b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca40b6d10 0x7fcca40b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 >> v2:172.21.15.69:3300/0 conn(0x7fcca40ceea0 msgr2=0x7fcca40d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 --2- 172.21.15.27:0/1948782962 >> v2:172.21.15.69:3300/0 conn(0x7fcca40ceea0 0x7fcca40d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40d39e0 con 0x7fcca415dd90 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fcca0001690 con 0x7fcca415dd90 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 >> v1:172.21.15.27:6789/0 conn(0x7fcca415dd90 legacy=0x7fcca40d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.329+0000 7fcca8924640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca40b6d10 0x7fcc900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fcca8924640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc90003f30 0x7fcc90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fcca8924640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc90004d40 0x7fcc900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40d3400 con 0x7fcc90004d40 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40d3120 con 0x7fcc90003f30 2024-03-06T07:27:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9000b9f0 con 0x7fcca40b6d10 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fccaa127640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc90004d40 0x7fcc900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fcca9926640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca40b6d10 0x7fcc900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fcca9125640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc90003f30 0x7fcc90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fccaa127640 1 -- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca40b6d10 msgr2=0x7fcc900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fccaa127640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca40b6d10 0x7fcc900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fccaa127640 1 -- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc90003f30 msgr2=0x7fcc90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fccaa127640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc90003f30 0x7fcc90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.330+0000 7fccaa127640 1 -- 172.21.15.27:0/1948782962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc90003520 con 0x7fcc90004d40 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.331+0000 7fccaa127640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc90004d40 0x7fcc900051c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009fc0 tx=0x7fcc90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ff0f9588b564a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.331+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c039070 con 0x7fcc90004d40 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.331+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c02cc20 con 0x7fcc90004d40 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.331+0000 7fcca8924640 1 -- 172.21.15.27:0/1948782962 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c009c10 con 0x7fcc90004d40 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.331+0000 7fccabb8a640 1 -- 172.21.15.27:0/1948782962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc90004d40 msgr2=0x7fcc900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.331+0000 7fccabb8a640 1 --2- 172.21.15.27:0/1948782962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc90004d40 0x7fcc900051c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009fc0 tx=0x7fcc90005830 comp rx=0 tx=0).stop 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.331+0000 7fccaa127640 1 -- 172.21.15.27:0/1948782962 reap_dead start 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.331+0000 7fccabb8a640 1 -- 172.21.15.27:0/1948782962 shutdown_connections 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.331+0000 7fccabb8a640 1 -- 172.21.15.27:0/1948782962 >> 172.21.15.27:0/1948782962 conn(0x7fcca4071960 msgr2=0x7fcca4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.332+0000 7fccabb8a640 1 -- 172.21.15.27:0/1948782962 shutdown_connections 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.332+0000 7fccabb8a640 1 -- 172.21.15.27:0/1948782962 wait complete. 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.333+0000 7fccabb8a640 1 Processor -- start 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.333+0000 7fccabb8a640 1 -- start start 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.333+0000 7fccabb8a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca415ee90 0x7fcca41f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.333+0000 7fccabb8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415f270 0x7fcca4160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.333+0000 7fccabb8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcca414aca0 0x7fcca414b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.333+0000 7fccabb8a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40d48f0 con 0x7fcca414aca0 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.333+0000 7fccabb8a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca415e170 con 0x7fcca415ee90 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.333+0000 7fccabb8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40d4a70 con 0x7fcca415f270 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccaa127640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcca414aca0 0x7fcca414b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccaa127640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcca414aca0 0x7fcca414b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36162/0 (socket says 172.21.15.27:36162) 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccaa127640 1 -- 172.21.15.27:0/1965183495 learned_addr learned my addr 172.21.15.27:0/1965183495 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fcca9926640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca415ee90 0x7fcca41f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fcca9125640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415f270 0x7fcca4160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccaa127640 1 -- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415f270 msgr2=0x7fcca4160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccaa127640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415f270 0x7fcca4160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccaa127640 1 -- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca415ee90 msgr2=0x7fcca41f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccaa127640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca415ee90 0x7fcca41f37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccaa127640 1 -- 172.21.15.27:0/1965183495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca415e620 con 0x7fcca414aca0 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fcca9926640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca415ee90 0x7fcca41f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fcca9125640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415f270 0x7fcca4160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccaa127640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcca414aca0 0x7fcca414b120 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c02c690 tx=0x7fcc9c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fcc9a7fc640 1 -- 172.21.15.27:0/1965183495 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c039030 con 0x7fcca414aca0 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fcc9a7fc640 1 -- 172.21.15.27:0/1965183495 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c043070 con 0x7fcca414aca0 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca41f49d0 con 0x7fcca414aca0 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fcc9a7fc640 1 -- 172.21.15.27:0/1965183495 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c009c10 con 0x7fcca414aca0 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.334+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca41f40f0 con 0x7fcca414aca0 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.336+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca4072a30 con 0x7fcca414aca0 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.336+0000 7fcc9a7fc640 1 -- 172.21.15.27:0/1965183495 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc9c005d10 con 0x7fcca414aca0 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.336+0000 7fcc9a7fc640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc8806b4d0 0x7fcc8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.336+0000 7fcc9a7fc640 1 -- 172.21.15.27:0/1965183495 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc9c0ac380 con 0x7fcca414aca0 2024-03-06T07:27:27.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.336+0000 7fcca9926640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc8806b4d0 0x7fcc8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.337+0000 7fcca9926640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc8806b4d0 0x7fcc8806d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcca0004840 tx=0x7fcca0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.341+0000 7fcc9a7fc640 1 -- 172.21.15.27:0/1965183495 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcca415e620 con 0x7fcca414aca0 2024-03-06T07:27:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.521+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcca41f4ba0 con 0x7fcca414aca0 2024-03-06T07:27:27.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.521+0000 7fcc9a7fc640 1 -- 172.21.15.27:0/1965183495 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcc9c07b200 con 0x7fcca414aca0 2024-03-06T07:27:27.524 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc8806b4d0 msgr2=0x7fcc8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc8806b4d0 0x7fcc8806d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcca0004840 tx=0x7fcca0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcca414aca0 msgr2=0x7fcca414b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcca414aca0 0x7fcca414b120 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c02c690 tx=0x7fcc9c009750 comp rx=0 tx=0).stop 2024-03-06T07:27:27.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 shutdown_connections 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc8806b4d0 0x7fcc8806d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcca414aca0 0x7fcca414b120 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcca415f270 0x7fcca4160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 --2- 172.21.15.27:0/1965183495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca415ee90 0x7fcca41f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.523+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 >> 172.21.15.27:0/1965183495 conn(0x7fcca4071960 msgr2=0x7fcca415bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.524+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 shutdown_connections 2024-03-06T07:27:27.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.524+0000 7fccabb8a640 1 -- 172.21.15.27:0/1965183495 wait complete. 2024-03-06T07:27:27.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:27.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.640+0000 7fcc3304e640 1 Processor -- start 2024-03-06T07:27:27.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.641+0000 7fcc3304e640 1 -- start start 2024-03-06T07:27:27.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.641+0000 7fcc3304e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcc2c15df00 0x7fcc2c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.641+0000 7fcc3304e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b9a30 0x7fcc2c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.641+0000 7fcc3304e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c0cea30 con 0x7fcc2c0b95a0 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.641+0000 7fcc3304e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2c0d2af0 con 0x7fcc2c15df00 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.641+0000 7fcc3304e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0cf3c0 con 0x7fcc2c0b9a30 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc30dea640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcc2c15df00 0x7fcc2c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc315eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b9a30 0x7fcc2c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc30dea640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcc2c15df00 0x7fcc2c0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:44502/0 (socket says 172.21.15.27:44502) 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc315eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b9a30 0x7fcc2c0d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:55474/0 (socket says 172.21.15.27:55474) 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc30dea640 1 -- 172.21.15.27:0/3708013587 learned_addr learned my addr 172.21.15.27:0/3708013587 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc315eb640 1 -- 172.21.15.27:0/3708013587 >> v2:172.21.15.69:3300/0 conn(0x7fcc2c15df00 msgr2=0x7fcc2c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc315eb640 1 --2- 172.21.15.27:0/3708013587 >> v2:172.21.15.69:3300/0 conn(0x7fcc2c15df00 0x7fcc2c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc315eb640 1 -- 172.21.15.27:0/3708013587 >> v1:172.21.15.69:6789/0 conn(0x7fcc2c0b95a0 legacy=0x7fcc2c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc315eb640 1 -- 172.21.15.27:0/3708013587 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0cf0e0 con 0x7fcc2c0b9a30 2024-03-06T07:27:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.642+0000 7fcc30dea640 1 --2- 172.21.15.27:0/3708013587 >> v2:172.21.15.69:3300/0 conn(0x7fcc2c15df00 0x7fcc2c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.643+0000 7fcc315eb640 1 --2- 172.21.15.27:0/3708013587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b9a30 0x7fcc2c0d3220 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0097a0 tx=0x7fcc1c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cee2ad07cc2da29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.643+0000 7fcc2b7fe640 1 -- 172.21.15.27:0/3708013587 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c035020 con 0x7fcc2c0b9a30 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.643+0000 7fcc2b7fe640 1 -- 172.21.15.27:0/3708013587 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c031830 con 0x7fcc2c0b9a30 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.643+0000 7fcc2b7fe640 1 -- 172.21.15.27:0/3708013587 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c031c00 con 0x7fcc2c0b9a30 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.644+0000 7fcc3304e640 1 -- 172.21.15.27:0/3708013587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b9a30 msgr2=0x7fcc2c0d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.644+0000 7fcc3304e640 1 --2- 172.21.15.27:0/3708013587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b9a30 0x7fcc2c0d3220 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0097a0 tx=0x7fcc1c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.644+0000 7fcc3304e640 1 -- 172.21.15.27:0/3708013587 shutdown_connections 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.644+0000 7fcc3304e640 1 --2- 172.21.15.27:0/3708013587 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b9a30 0x7fcc2c0d3220 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.644+0000 7fcc3304e640 1 --2- 172.21.15.27:0/3708013587 >> v2:172.21.15.69:3300/0 conn(0x7fcc2c15df00 0x7fcc2c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.644+0000 7fcc3304e640 1 -- 172.21.15.27:0/3708013587 >> 172.21.15.27:0/3708013587 conn(0x7fcc2c071960 msgr2=0x7fcc2c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.644+0000 7fcc3304e640 1 -- 172.21.15.27:0/3708013587 shutdown_connections 2024-03-06T07:27:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.644+0000 7fcc3304e640 1 -- 172.21.15.27:0/3708013587 wait complete. 2024-03-06T07:27:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.645+0000 7fcc3304e640 1 Processor -- start 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.645+0000 7fcc3304e640 1 -- start start 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc3304e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c0b95a0 0x7fcc2c1f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc3304e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c15df00 0x7fcc2c15ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc3304e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b1df0 0x7fcc2c0b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc3304e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0cec70 con 0x7fcc2c15df00 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc3304e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0ce750 con 0x7fcc2c0b95a0 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc3304e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0cedf0 con 0x7fcc2c0b1df0 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc2bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c15df00 0x7fcc2c15ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc2bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c15df00 0x7fcc2c15ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36178/0 (socket says 172.21.15.27:36178) 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc2bfff640 1 -- 172.21.15.27:0/2284607559 learned_addr learned my addr 172.21.15.27:0/2284607559 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc30dea640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c0b95a0 0x7fcc2c1f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc315eb640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b1df0 0x7fcc2c0b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc2bfff640 1 -- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b1df0 msgr2=0x7fcc2c0b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc2bfff640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b1df0 0x7fcc2c0b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc2bfff640 1 -- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c0b95a0 msgr2=0x7fcc2c1f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc2bfff640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c0b95a0 0x7fcc2c1f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc2bfff640 1 -- 172.21.15.27:0/2284607559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0cbf20 con 0x7fcc2c15df00 2024-03-06T07:27:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.646+0000 7fcc2bfff640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c15df00 0x7fcc2c15ec00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000c400 tx=0x7fcc2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.647+0000 7fcc297fa640 1 -- 172.21.15.27:0/2284607559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc200052d0 con 0x7fcc2c15df00 2024-03-06T07:27:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.647+0000 7fcc297fa640 1 -- 172.21.15.27:0/2284607559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc20005490 con 0x7fcc2c15df00 2024-03-06T07:27:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.647+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2c0cc200 con 0x7fcc2c15df00 2024-03-06T07:27:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.647+0000 7fcc297fa640 1 -- 172.21.15.27:0/2284607559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20007260 con 0x7fcc2c15df00 2024-03-06T07:27:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.647+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2c1f34a0 con 0x7fcc2c15df00 2024-03-06T07:27:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.648+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2c067ff0 con 0x7fcc2c15df00 2024-03-06T07:27:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.648+0000 7fcc297fa640 1 -- 172.21.15.27:0/2284607559 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc20007400 con 0x7fcc2c15df00 2024-03-06T07:27:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.649+0000 7fcc297fa640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc1806b4d0 0x7fcc1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.649+0000 7fcc297fa640 1 -- 172.21.15.27:0/2284607559 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc200895d0 con 0x7fcc2c15df00 2024-03-06T07:27:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.649+0000 7fcc30dea640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc1806b4d0 0x7fcc1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.650+0000 7fcc30dea640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc1806b4d0 0x7fcc1806d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fcc240072c0 tx=0x7fcc2400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.653+0000 7fcc297fa640 1 -- 172.21.15.27:0/2284607559 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc20057e90 con 0x7fcc2c15df00 2024-03-06T07:27:27.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.873+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc2c15b820 con 0x7fcc2c15df00 2024-03-06T07:27:27.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.874+0000 7fcc297fa640 1 -- 172.21.15.27:0/2284607559 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fcc2005bf30 con 0x7fcc2c15df00 2024-03-06T07:27:27.876 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/930650799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":930650799},{"type":"v1","addr":"172.21.15.69:6836","nonce":930650799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1050755919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1050755919},{"type":"v1","addr":"172.21.15.27:6835","nonce":1050755919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1173605349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1173605349},{"type":"v1","addr":"172.21.15.69:6837","nonce":1173605349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:09.729142+0000","modified":"2024-03-06T07:27:12.757645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7032},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7032":{"gid":7032,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3587849188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3587849188},{"type":"v1","addr":"172.21.15.27:6834","nonce":3587849188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7032,"qdb_cluster":[7032]},"id":6}]} 2024-03-06T07:27:27.877 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 52 2024-03-06T07:27:27.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc1806b4d0 msgr2=0x7fcc1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc1806b4d0 0x7fcc1806d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fcc240072c0 tx=0x7fcc2400b040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c15df00 msgr2=0x7fcc2c15ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c15df00 0x7fcc2c15ec00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000c400 tx=0x7fcc2000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 shutdown_connections 2024-03-06T07:27:27.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc1806b4d0 0x7fcc1806d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc2c0b1df0 0x7fcc2c0b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c15df00 0x7fcc2c15ec00 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 --2- 172.21.15.27:0/2284607559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc2c0b95a0 0x7fcc2c1f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 >> 172.21.15.27:0/2284607559 conn(0x7fcc2c071960 msgr2=0x7fcc2c0b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 shutdown_connections 2024-03-06T07:27:27.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.876+0000 7fcc3304e640 1 -- 172.21.15.27:0/2284607559 wait complete. 2024-03-06T07:27:27.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:27.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.994+0000 7fbeffd56640 1 Processor -- start 2024-03-06T07:27:27.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.995+0000 7fbeffd56640 1 -- start start 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.995+0000 7fbeffd56640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbef80b9a10 0x7fbef80d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.995+0000 7fbeffd56640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef80cea30 con 0x7fbef80b95a0 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.995+0000 7fbeffd56640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef80cf3c0 con 0x7fbef815df00 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.995+0000 7fbeffd56640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef80cf0e0 con 0x7fbef80b9a10 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.995+0000 7fbefe2f3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbef80b9a10 0x7fbef80d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.995+0000 7fbefd2f1640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbef80b95a0 0x7fbef80b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39816/0 (socket says 172.21.15.27:39816) 2024-03-06T07:27:27.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.995+0000 7fbefd2f1640 1 -- 172.21.15.27:0/2230848808 learned_addr learned my addr 172.21.15.27:0/2230848808 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:27.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.996+0000 7fbefe2f3640 1 -- 172.21.15.27:0/2230848808 >> v1:172.21.15.69:6789/0 conn(0x7fbef815df00 legacy=0x7fbef80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.996+0000 7fbefe2f3640 1 -- 172.21.15.27:0/2230848808 >> v1:172.21.15.27:6789/0 conn(0x7fbef80b95a0 legacy=0x7fbef80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.996+0000 7fbefe2f3640 1 -- 172.21.15.27:0/2230848808 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef80ce470 con 0x7fbef80b9a10 2024-03-06T07:27:27.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.996+0000 7fbefcaf0640 1 -- 172.21.15.27:0/2230848808 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872538812 0 0) 0x7fbef80cea30 con 0x7fbef80b95a0 2024-03-06T07:27:27.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.997+0000 7fbefe2f3640 1 --2- 172.21.15.27:0/2230848808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef80b9a10 0x7fbef80d3220 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbef80cf590 tx=0x7fbef002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a2a8327b1a36dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.997+0000 7fbefcaf0640 1 -- 172.21.15.27:0/2230848808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0039070 con 0x7fbef80b9a10 2024-03-06T07:27:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.997+0000 7fbefcaf0640 1 -- 172.21.15.27:0/2230848808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef002cce0 con 0x7fbef80b9a10 2024-03-06T07:27:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.997+0000 7fbefcaf0640 1 -- 172.21.15.27:0/2230848808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0034660 con 0x7fbef80b9a10 2024-03-06T07:27:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.998+0000 7fbeffd56640 1 -- 172.21.15.27:0/2230848808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef80b9a10 msgr2=0x7fbef80d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.998+0000 7fbeffd56640 1 --2- 172.21.15.27:0/2230848808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef80b9a10 0x7fbef80d3220 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbef80cf590 tx=0x7fbef002bf30 comp rx=0 tx=0).stop 2024-03-06T07:27:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.998+0000 7fbeffd56640 1 -- 172.21.15.27:0/2230848808 shutdown_connections 2024-03-06T07:27:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.998+0000 7fbeffd56640 1 --2- 172.21.15.27:0/2230848808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef80b9a10 0x7fbef80d3220 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.998+0000 7fbeffd56640 1 -- 172.21.15.27:0/2230848808 >> 172.21.15.27:0/2230848808 conn(0x7fbef8071960 msgr2=0x7fbef8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.998+0000 7fbeffd56640 1 -- 172.21.15.27:0/2230848808 shutdown_connections 2024-03-06T07:27:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.998+0000 7fbeffd56640 1 -- 172.21.15.27:0/2230848808 wait complete. 2024-03-06T07:27:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.999+0000 7fbeffd56640 1 Processor -- start 2024-03-06T07:27:28.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:27.999+0000 7fbeffd56640 1 -- start start 2024-03-06T07:27:28.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbeffd56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef80b95a0 0x7fbef80d3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbeffd56640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef80b9a10 0x7fbef815eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbeffd56640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef815df00 0x7fbef80ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbeffd56640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80cedf0 con 0x7fbef815df00 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbeffd56640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80cec70 con 0x7fbef80b9a10 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbeffd56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80b5e10 con 0x7fbef80b95a0 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefe2f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef815df00 0x7fbef80ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefe2f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef815df00 0x7fbef80ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36206/0 (socket says 172.21.15.27:36206) 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefe2f3640 1 -- 172.21.15.27:0/2202431903 learned_addr learned my addr 172.21.15.27:0/2202431903 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefd2f1640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef80b9a10 0x7fbef815eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefdaf2640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef80b95a0 0x7fbef80d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefe2f3640 1 -- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef80b95a0 msgr2=0x7fbef80d3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefe2f3640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef80b95a0 0x7fbef80d3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefe2f3640 1 -- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef80b9a10 msgr2=0x7fbef815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefe2f3640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef80b9a10 0x7fbef815eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefe2f3640 1 -- 172.21.15.27:0/2202431903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8160b20 con 0x7fbef815df00 2024-03-06T07:27:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.000+0000 7fbefd2f1640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef80b9a10 0x7fbef815eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.001+0000 7fbefe2f3640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef815df00 0x7fbef80ccf20 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fbef0037040 tx=0x7fbef0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.001+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2202431903 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0039050 con 0x7fbef815df00 2024-03-06T07:27:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.001+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2202431903 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef0045070 con 0x7fbef815df00 2024-03-06T07:27:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.001+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef8160e00 con 0x7fbef815df00 2024-03-06T07:27:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.001+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2202431903 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef003e810 con 0x7fbef815df00 2024-03-06T07:27:28.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.001+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef81613c0 con 0x7fbef815df00 2024-03-06T07:27:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.002+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef815b000 con 0x7fbef815df00 2024-03-06T07:27:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.002+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2202431903 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbef003e9b0 con 0x7fbef815df00 2024-03-06T07:27:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.003+0000 7fbeee7fc640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b4d0 0x7fbedc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.003+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2202431903 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbef00ad2b0 con 0x7fbef815df00 2024-03-06T07:27:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.003+0000 7fbefdaf2640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b4d0 0x7fbedc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.004+0000 7fbefdaf2640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b4d0 0x7fbedc06d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000f60 tx=0x7fbee8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.007+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2202431903 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbef007c2e0 con 0x7fbef815df00 2024-03-06T07:27:28.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.189+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbef80d1b90 con 0x7fbef815df00 2024-03-06T07:27:28.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.191+0000 7fbeee7fc640 1 -- 172.21.15.27:0/2202431903 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16588 (secure 0 0 0) 0x7fbef007bbe0 con 0x7fbef815df00 2024-03-06T07:27:28.194 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:28.194 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":84,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:11.381712+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:07.388472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:08.720233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b4d0 msgr2=0x7fbedc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b4d0 0x7fbedc06d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000f60 tx=0x7fbee8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef815df00 msgr2=0x7fbef80ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef815df00 0x7fbef80ccf20 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fbef0037040 tx=0x7fbef0004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 shutdown_connections 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbedc06b4d0 0x7fbedc06d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbef815df00 0x7fbef80ccf20 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef80b9a10 0x7fbef815eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 --2- 172.21.15.27:0/2202431903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbef80b95a0 0x7fbef80d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 >> 172.21.15.27:0/2202431903 conn(0x7fbef8071960 msgr2=0x7fbef80b5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 shutdown_connections 2024-03-06T07:27:28.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.193+0000 7fbeffd56640 1 -- 172.21.15.27:0/2202431903 wait complete. 2024-03-06T07:27:28.207 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:27:28.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.313+0000 7f8a1e725640 1 Processor -- start 2024-03-06T07:27:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.313+0000 7f8a1e725640 1 -- start start 2024-03-06T07:27:28.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.313+0000 7f8a1e725640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a18071540 0x7f8a180b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.313+0000 7f8a1e725640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a180d3400 0x7f8a18071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.313+0000 7f8a1e725640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a18071e60 con 0x7f8a180b2b50 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.313+0000 7f8a1e725640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a180d2f70 con 0x7f8a180d3400 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.313+0000 7f8a1e725640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a180d4be0 con 0x7f8a18071540 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a177fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a180d3400 0x7f8a18071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a1ccc2640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8a180b2b50 0x7f8a180d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39822/0 (socket says 172.21.15.27:39822) 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a1ccc2640 1 -- 172.21.15.27:0/1181450349 learned_addr learned my addr 172.21.15.27:0/1181450349 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a17fff640 1 --2- 172.21.15.27:0/1181450349 >> v2:172.21.15.69:3300/0 conn(0x7f8a18071540 0x7f8a180b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a177fe640 1 -- 172.21.15.27:0/1181450349 >> v2:172.21.15.69:3300/0 conn(0x7f8a18071540 msgr2=0x7f8a180b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a177fe640 1 --2- 172.21.15.27:0/1181450349 >> v2:172.21.15.69:3300/0 conn(0x7f8a18071540 0x7f8a180b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a177fe640 1 -- 172.21.15.27:0/1181450349 >> v1:172.21.15.27:6789/0 conn(0x7f8a180b2b50 legacy=0x7f8a180d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a177fe640 1 -- 172.21.15.27:0/1181450349 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a180ce610 con 0x7f8a180d3400 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a16ffd640 1 -- 172.21.15.27:0/1181450349 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109677207 0 0) 0x7f8a18071e60 con 0x7f8a180b2b50 2024-03-06T07:27:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.314+0000 7f8a17fff640 1 --2- 172.21.15.27:0/1181450349 >> v2:172.21.15.69:3300/0 conn(0x7f8a18071540 0x7f8a180b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a177fe640 1 --2- 172.21.15.27:0/1181450349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 0x7f8a18071920 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f8a180d4da0 tx=0x7f8a0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97c6543c5cae3b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a16ffd640 1 -- 172.21.15.27:0/1181450349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c035020 con 0x7f8a180d3400 2024-03-06T07:27:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a16ffd640 1 -- 172.21.15.27:0/1181450349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c031830 con 0x7f8a180d3400 2024-03-06T07:27:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a16ffd640 1 -- 172.21.15.27:0/1181450349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c031c00 con 0x7f8a180d3400 2024-03-06T07:27:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a1e725640 1 -- 172.21.15.27:0/1181450349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 msgr2=0x7f8a18071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a1e725640 1 --2- 172.21.15.27:0/1181450349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 0x7f8a18071920 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f8a180d4da0 tx=0x7f8a0c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a1e725640 1 -- 172.21.15.27:0/1181450349 shutdown_connections 2024-03-06T07:27:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a1e725640 1 --2- 172.21.15.27:0/1181450349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 0x7f8a18071920 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a1e725640 1 --2- 172.21.15.27:0/1181450349 >> v2:172.21.15.69:3300/0 conn(0x7f8a18071540 0x7f8a180b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.315+0000 7f8a1e725640 1 -- 172.21.15.27:0/1181450349 >> 172.21.15.27:0/1181450349 conn(0x7f8a180d3ca0 msgr2=0x7f8a180d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.316+0000 7f8a1e725640 1 -- 172.21.15.27:0/1181450349 shutdown_connections 2024-03-06T07:27:28.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.316+0000 7f8a1e725640 1 -- 172.21.15.27:0/1181450349 wait complete. 2024-03-06T07:27:28.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.317+0000 7f8a1e725640 1 Processor -- start 2024-03-06T07:27:28.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.317+0000 7f8a1e725640 1 -- start start 2024-03-06T07:27:28.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1e725640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a18071540 0x7f8a180cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1e725640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180b2b50 0x7f8a180b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1e725640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 0x7f8a180ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1e725640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180ce8f0 con 0x7f8a180d3400 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1e725640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180d2c00 con 0x7f8a18071540 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1e725640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1815e370 con 0x7f8a180b2b50 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1ccc2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 0x7f8a180ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1ccc2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 0x7f8a180ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36226/0 (socket says 172.21.15.27:36226) 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1ccc2640 1 -- 172.21.15.27:0/2747704521 learned_addr learned my addr 172.21.15.27:0/2747704521 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a177fe640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180b2b50 0x7f8a180b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a17fff640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a18071540 0x7f8a180cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1ccc2640 1 -- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180b2b50 msgr2=0x7f8a180b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1ccc2640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180b2b50 0x7f8a180b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1ccc2640 1 -- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a18071540 msgr2=0x7f8a180cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1ccc2640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a18071540 0x7f8a180cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1ccc2640 1 -- 172.21.15.27:0/2747704521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a180cc570 con 0x7f8a180d3400 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1ccc2640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 0x7f8a180ca0e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000c7a0 tx=0x7f8a1000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a14ff9640 1 -- 172.21.15.27:0/2747704521 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10018070 con 0x7f8a180d3400 2024-03-06T07:27:28.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a14ff9640 1 -- 172.21.15.27:0/2747704521 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1000a640 con 0x7f8a180d3400 2024-03-06T07:27:28.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a180cc850 con 0x7f8a180d3400 2024-03-06T07:27:28.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.318+0000 7f8a14ff9640 1 -- 172.21.15.27:0/2747704521 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1000aab0 con 0x7f8a180d3400 2024-03-06T07:27:28.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.319+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a180cdb50 con 0x7f8a180d3400 2024-03-06T07:27:28.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.320+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a180cca20 con 0x7f8a180d3400 2024-03-06T07:27:28.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.320+0000 7f8a14ff9640 1 -- 172.21.15.27:0/2747704521 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a1001b070 con 0x7f8a180d3400 2024-03-06T07:27:28.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.320+0000 7f8a14ff9640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a0406b4d0 0x7f8a0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.321+0000 7f8a14ff9640 1 -- 172.21.15.27:0/2747704521 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a10089840 con 0x7f8a180d3400 2024-03-06T07:27:28.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.321+0000 7f8a17fff640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a0406b4d0 0x7f8a0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.321+0000 7f8a17fff640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a0406b4d0 0x7f8a0406d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8a08006fd0 tx=0x7f8a08008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.325+0000 7f8a14ff9640 1 -- 172.21.15.27:0/2747704521 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a100586c0 con 0x7f8a180d3400 2024-03-06T07:27:28.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.590+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a18068700 con 0x7f8a180d3400 2024-03-06T07:27:28.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.591+0000 7f8a14ff9640 1 -- 172.21.15.27:0/2747704521 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f8a10057fc0 con 0x7f8a180d3400 2024-03-06T07:27:28.593 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:28.593 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/930650799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":930650799},{"type":"v1","addr":"172.21.15.69:6836","nonce":930650799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1050755919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1050755919},{"type":"v1","addr":"172.21.15.27:6835","nonce":1050755919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1173605349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1173605349},{"type":"v1","addr":"172.21.15.69:6837","nonce":1173605349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:09.729142+0000","modified":"2024-03-06T07:27:12.757645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7032},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7032":{"gid":7032,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3587849188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3587849188},{"type":"v1","addr":"172.21.15.27:6834","nonce":3587849188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7032,"qdb_cluster":[7032]},"id":6}]} 2024-03-06T07:27:28.593 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 52 2024-03-06T07:27:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a0406b4d0 msgr2=0x7f8a0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a0406b4d0 0x7f8a0406d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8a08006fd0 tx=0x7f8a08008040 comp rx=0 tx=0).stop 2024-03-06T07:27:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 msgr2=0x7f8a180ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 0x7f8a180ca0e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000c7a0 tx=0x7f8a1000cc60 comp rx=0 tx=0).stop 2024-03-06T07:27:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 shutdown_connections 2024-03-06T07:27:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a0406b4d0 0x7f8a0406d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a180d3400 0x7f8a180ca0e0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a180b2b50 0x7f8a180b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 --2- 172.21.15.27:0/2747704521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a18071540 0x7f8a180cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.592+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 >> 172.21.15.27:0/2747704521 conn(0x7f8a180d3ca0 msgr2=0x7f8a181606b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.593+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 shutdown_connections 2024-03-06T07:27:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.593+0000 7f8a1e725640 1 -- 172.21.15.27:0/2747704521 wait complete. 2024-03-06T07:27:28.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.710+0000 7ff148264640 1 Processor -- start 2024-03-06T07:27:28.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.710+0000 7ff148264640 1 -- start start 2024-03-06T07:27:28.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.711+0000 7ff148264640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff14006b5c0 0x7ff1400b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.711+0000 7ff148264640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1400ae9b0 0x7ff1400d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.711+0000 7ff148264640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400b3340 con 0x7ff1400b0e90 2024-03-06T07:27:28.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.711+0000 7ff148264640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1400cfa00 con 0x7ff14006b5c0 2024-03-06T07:27:28.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.711+0000 7ff148264640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400cee90 con 0x7ff1400ae9b0 2024-03-06T07:27:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.711+0000 7ff146000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff14006b5c0 0x7ff1400b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.711+0000 7ff146000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff14006b5c0 0x7ff1400b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36234/0 (socket says 172.21.15.27:36234) 2024-03-06T07:27:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.711+0000 7ff146000640 1 -- 172.21.15.27:0/2141567594 learned_addr learned my addr 172.21.15.27:0/2141567594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.711+0000 7ff146801640 1 --2- 172.21.15.27:0/2141567594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1400ae9b0 0x7ff1400d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.712+0000 7ff146000640 1 -- 172.21.15.27:0/2141567594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1400ae9b0 msgr2=0x7ff1400d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.712+0000 7ff146000640 1 --2- 172.21.15.27:0/2141567594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1400ae9b0 0x7ff1400d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.712+0000 7ff146000640 1 -- 172.21.15.27:0/2141567594 >> v1:172.21.15.69:6789/0 conn(0x7ff1400b0e90 legacy=0x7ff14006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.712+0000 7ff146000640 1 -- 172.21.15.27:0/2141567594 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400cfc50 con 0x7ff14006b5c0 2024-03-06T07:27:28.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.712+0000 7ff146000640 1 --2- 172.21.15.27:0/2141567594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14006b5c0 0x7ff1400b2e00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009870 tx=0x7ff13c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38eaf321ae556b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff144ffe640 1 -- 172.21.15.27:0/2141567594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c035020 con 0x7ff14006b5c0 2024-03-06T07:27:28.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff144ffe640 1 -- 172.21.15.27:0/2141567594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c031640 con 0x7ff14006b5c0 2024-03-06T07:27:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff144ffe640 1 -- 172.21.15.27:0/2141567594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c02d040 con 0x7ff14006b5c0 2024-03-06T07:27:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff148264640 1 -- 172.21.15.27:0/2141567594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14006b5c0 msgr2=0x7ff1400b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff148264640 1 --2- 172.21.15.27:0/2141567594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14006b5c0 0x7ff1400b2e00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009870 tx=0x7ff13c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff148264640 1 -- 172.21.15.27:0/2141567594 shutdown_connections 2024-03-06T07:27:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff148264640 1 --2- 172.21.15.27:0/2141567594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1400ae9b0 0x7ff1400d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff148264640 1 --2- 172.21.15.27:0/2141567594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14006b5c0 0x7ff1400b2e00 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff148264640 1 -- 172.21.15.27:0/2141567594 >> 172.21.15.27:0/2141567594 conn(0x7ff1400b36d0 msgr2=0x7ff1400b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff148264640 1 -- 172.21.15.27:0/2141567594 shutdown_connections 2024-03-06T07:27:28.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.713+0000 7ff148264640 1 -- 172.21.15.27:0/2141567594 wait complete. 2024-03-06T07:27:28.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.715+0000 7ff148264640 1 Processor -- start 2024-03-06T07:27:28.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.715+0000 7ff148264640 1 -- start start 2024-03-06T07:27:28.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.715+0000 7ff148264640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14006b5c0 0x7ff1401da330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.715+0000 7ff148264640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1400ae9b0 0x7ff14006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.715+0000 7ff148264640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1400b0e90 0x7ff140070260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.715+0000 7ff148264640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14006c910 con 0x7ff1400b0e90 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.715+0000 7ff148264640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14006c790 con 0x7ff1400ae9b0 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.715+0000 7ff148264640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400aee00 con 0x7ff14006b5c0 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146801640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1400b0e90 0x7ff140070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146801640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1400b0e90 0x7ff140070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36242/0 (socket says 172.21.15.27:36242) 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146801640 1 -- 172.21.15.27:0/763648332 learned_addr learned my addr 172.21.15.27:0/763648332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146000640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14006b5c0 0x7ff1401da330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff1457ff640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1400ae9b0 0x7ff14006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146801640 1 -- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14006b5c0 msgr2=0x7ff1401da330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146801640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14006b5c0 0x7ff1401da330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146801640 1 -- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1400ae9b0 msgr2=0x7ff14006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146801640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1400ae9b0 0x7ff14006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146801640 1 -- 172.21.15.27:0/763648332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1401e3810 con 0x7ff1400b0e90 2024-03-06T07:27:28.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146000640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14006b5c0 0x7ff1401da330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:28.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff1457ff640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1400ae9b0 0x7ff14006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:28.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff146801640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1400b0e90 0x7ff140070260 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ff138007e70 tx=0x7ff13800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff1367fc640 1 -- 172.21.15.27:0/763648332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138014620 con 0x7ff1400b0e90 2024-03-06T07:27:28.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff1367fc640 1 -- 172.21.15.27:0/763648332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff138019070 con 0x7ff1400b0e90 2024-03-06T07:27:28.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1401e3af0 con 0x7ff1400b0e90 2024-03-06T07:27:28.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff1367fc640 1 -- 172.21.15.27:0/763648332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138014a20 con 0x7ff1400b0e90 2024-03-06T07:27:28.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.716+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1401da870 con 0x7ff1400b0e90 2024-03-06T07:27:28.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.717+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1401e3490 con 0x7ff1400b0e90 2024-03-06T07:27:28.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.718+0000 7ff1367fc640 1 -- 172.21.15.27:0/763648332 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff138014bc0 con 0x7ff1400b0e90 2024-03-06T07:27:28.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.718+0000 7ff1367fc640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff12406b4d0 0x7ff12406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.718+0000 7ff1367fc640 1 -- 172.21.15.27:0/763648332 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff13808aa40 con 0x7ff1400b0e90 2024-03-06T07:27:28.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.719+0000 7ff146000640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff12406b4d0 0x7ff12406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.719+0000 7ff146000640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff12406b4d0 0x7ff12406d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff13c036000 tx=0x7ff13c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.723+0000 7ff1367fc640 1 -- 172.21.15.27:0/763648332 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1401e3490 con 0x7ff1400b0e90 2024-03-06T07:27:28.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.943+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff140068700 con 0x7ff1400b0e90 2024-03-06T07:27:28.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.944+0000 7ff1367fc640 1 -- 172.21.15.27:0/763648332 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4976 (secure 0 0 0) 0x7ff13808ae20 con 0x7ff1400b0e90 2024-03-06T07:27:28.947 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:28.947 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/930650799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":930650799},{"type":"v1","addr":"172.21.15.69:6836","nonce":930650799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1050755919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1050755919},{"type":"v1","addr":"172.21.15.27:6835","nonce":1050755919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1173605349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1173605349},{"type":"v1","addr":"172.21.15.69:6837","nonce":1173605349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:09.729142+0000","modified":"2024-03-06T07:27:12.757645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7032},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7032":{"gid":7032,"name":"a","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3587849188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3587849188},{"type":"v1","addr":"172.21.15.27:6834","nonce":3587849188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7032,"qdb_cluster":[7032]},"id":6}]} 2024-03-06T07:27:28.947 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 52 2024-03-06T07:27:28.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff12406b4d0 msgr2=0x7ff12406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff12406b4d0 0x7ff12406d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff13c036000 tx=0x7ff13c03c040 comp rx=0 tx=0).stop 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1400b0e90 msgr2=0x7ff140070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1400b0e90 0x7ff140070260 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ff138007e70 tx=0x7ff13800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 shutdown_connections 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff12406b4d0 0x7ff12406d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1400b0e90 0x7ff140070260 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1400ae9b0 0x7ff14006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 --2- 172.21.15.27:0/763648332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff14006b5c0 0x7ff1401da330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 >> 172.21.15.27:0/763648332 conn(0x7ff1400b36d0 msgr2=0x7ff1401e6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 shutdown_connections 2024-03-06T07:27:28.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:28.946+0000 7ff148264640 1 -- 172.21.15.27:0/763648332 wait complete. 2024-03-06T07:27:28.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:29.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.064+0000 7f6cf2e17640 1 Processor -- start 2024-03-06T07:27:29.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.064+0000 7f6cf2e17640 1 -- start start 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.065+0000 7f6cf2e17640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cec15df00 0x7f6cec0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.065+0000 7f6cf2e17640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec0cea30 con 0x7f6cec0b9a30 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.065+0000 7f6cf2e17640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec0d2af0 con 0x7f6cec0b95a0 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.065+0000 7f6cf2e17640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0cf3c0 con 0x7f6cec15df00 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.065+0000 7f6cf13b4640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6cec0b9a30 0x7f6cec0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39832/0 (socket says 172.21.15.27:39832) 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.065+0000 7f6cf13b4640 1 -- 172.21.15.27:0/736225368 learned_addr learned my addr 172.21.15.27:0/736225368 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6cf0bb3640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cec15df00 0x7f6cec0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989851458 0 0) 0x7f6cec0cea30 con 0x7f6cec0b9a30 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cdc003520 con 0x7f6cec0b9a30 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2490735397 0 0) 0x7f6cdc003520 con 0x7f6cec0b9a30 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cec15df00 msgr2=0x7f6cec0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cec15df00 0x7f6cec0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 >> v1:172.21.15.69:6789/0 conn(0x7f6cec0b95a0 legacy=0x7f6cec0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0cf0e0 con 0x7f6cec0b9a30 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f6ce40019f0 con 0x7f6cec0b9a30 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 >> v1:172.21.15.27:6789/0 conn(0x7f6cec0b9a30 legacy=0x7f6cec0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039e0 0x7f6cdc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc004390 0x7f6cdc004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cec15df00 0x7f6cdc0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0cf3c0 con 0x7f6cdc004390 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc00b9c0 con 0x7f6cdc0039e0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc00bbc0 con 0x7f6cec15df00 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6cf0bb3640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039e0 0x7f6cdc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6cebfff640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc004390 0x7f6cdc004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.066+0000 7f6cf13b4640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cec15df00 0x7f6cdc0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.067+0000 7f6cebfff640 1 -- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cec15df00 msgr2=0x7f6cdc0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.067+0000 7f6cebfff640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cec15df00 0x7f6cdc0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.067+0000 7f6cebfff640 1 -- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039e0 msgr2=0x7f6cdc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.067+0000 7f6cebfff640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039e0 0x7f6cdc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.067+0000 7f6cebfff640 1 -- 172.21.15.27:0/736225368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc003520 con 0x7f6cdc004390 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.067+0000 7f6cf0bb3640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cdc0039e0 0x7f6cdc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.067+0000 7f6cebfff640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc004390 0x7f6cdc004860 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8001860 tx=0x7f6cdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9ec810bc46e5e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8038070 con 0x7f6cdc004390 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd8033410 con 0x7f6cdc004390 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6ceb7fe640 1 -- 172.21.15.27:0/736225368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8001430 con 0x7f6cdc004390 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6cf2e17640 1 -- 172.21.15.27:0/736225368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc004390 msgr2=0x7f6cdc004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6cf2e17640 1 --2- 172.21.15.27:0/736225368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cdc004390 0x7f6cdc004860 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8001860 tx=0x7f6cdc005830 comp rx=0 tx=0).stop 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6cf13b4640 1 -- 172.21.15.27:0/736225368 reap_dead start 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6cf2e17640 1 -- 172.21.15.27:0/736225368 shutdown_connections 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6cf2e17640 1 -- 172.21.15.27:0/736225368 >> 172.21.15.27:0/736225368 conn(0x7f6cec071960 msgr2=0x7f6cec071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6cf2e17640 1 -- 172.21.15.27:0/736225368 shutdown_connections 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.068+0000 7f6cf2e17640 1 -- 172.21.15.27:0/736225368 wait complete. 2024-03-06T07:27:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.070+0000 7f6cf2e17640 1 Processor -- start 2024-03-06T07:27:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.070+0000 7f6cf2e17640 1 -- start start 2024-03-06T07:27:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.071+0000 7f6cf2e17640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc004390 0x7f6cec1f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.071+0000 7f6cf2e17640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15e400 0x7f6cec15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.071+0000 7f6cf2e17640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cec0cb760 0x7f6cec15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.071+0000 7f6cf2e17640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0cec70 con 0x7f6cec15e400 2024-03-06T07:27:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.071+0000 7f6cf2e17640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0ce750 con 0x7f6cec0cb760 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.071+0000 7f6cf2e17640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0cedf0 con 0x7f6cdc004390 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.071+0000 7f6cebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15e400 0x7f6cec15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.071+0000 7f6cebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15e400 0x7f6cec15ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36268/0 (socket says 172.21.15.27:36268) 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.071+0000 7f6cebfff640 1 -- 172.21.15.27:0/1230352909 learned_addr learned my addr 172.21.15.27:0/1230352909 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cf13b4640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cec0cb760 0x7f6cec15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cf0bb3640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc004390 0x7f6cec1f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cebfff640 1 -- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc004390 msgr2=0x7f6cec1f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cebfff640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc004390 0x7f6cec1f3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cebfff640 1 -- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cec0cb760 msgr2=0x7f6cec15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cebfff640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cec0cb760 0x7f6cec15b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cebfff640 1 -- 172.21.15.27:0/1230352909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0cbd20 con 0x7f6cec15e400 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cebfff640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15e400 0x7f6cec15ad60 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f6cd802bdd0 tx=0x7f6cd8001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6ce97fa640 1 -- 172.21.15.27:0/1230352909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8041020 con 0x7f6cec15e400 2024-03-06T07:27:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6ce97fa640 1 -- 172.21.15.27:0/1230352909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd8006e00 con 0x7f6cec15e400 2024-03-06T07:27:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec0c77b0 con 0x7f6cec15e400 2024-03-06T07:27:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6ce97fa640 1 -- 172.21.15.27:0/1230352909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8038040 con 0x7f6cec15e400 2024-03-06T07:27:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.072+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec0c7d70 con 0x7f6cec15e400 2024-03-06T07:27:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.073+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec067ff0 con 0x7f6cec15e400 2024-03-06T07:27:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.074+0000 7f6ce97fa640 1 -- 172.21.15.27:0/1230352909 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6cd8044020 con 0x7f6cec15e400 2024-03-06T07:27:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.074+0000 7f6ce97fa640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6cd006b4d0 0x7f6cd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.074+0000 7f6ce97fa640 1 -- 172.21.15.27:0/1230352909 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6cd80ab7d0 con 0x7f6cec15e400 2024-03-06T07:27:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.075+0000 7f6cf0bb3640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6cd006b4d0 0x7f6cd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.075+0000 7f6cf0bb3640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6cd006b4d0 0x7f6cd006d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009c00 tx=0x7f6ce0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.083+0000 7f6ce97fa640 1 -- 172.21.15.27:0/1230352909 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6cd804b020 con 0x7f6cec15e400 2024-03-06T07:27:29.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.264+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cec0b2bd0 con 0x7f6cec15e400 2024-03-06T07:27:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.266+0000 7f6ce97fa640 1 -- 172.21.15.27:0/1230352909 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16638 (secure 0 0 0) 0x7f6cd807a7e0 con 0x7f6cec15e400 2024-03-06T07:27:29.268 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:29.269 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":85,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:28.432865+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:07.388472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:08.720233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.10","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:29.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6cd006b4d0 msgr2=0x7f6cd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6cd006b4d0 0x7f6cd006d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009c00 tx=0x7f6ce0008040 comp rx=0 tx=0).stop 2024-03-06T07:27:29.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15e400 msgr2=0x7f6cec15ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15e400 0x7f6cec15ad60 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f6cd802bdd0 tx=0x7f6cd8001230 comp rx=0 tx=0).stop 2024-03-06T07:27:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 shutdown_connections 2024-03-06T07:27:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6cd006b4d0 0x7f6cd006d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cec0cb760 0x7f6cec15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15e400 0x7f6cec15ad60 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 --2- 172.21.15.27:0/1230352909 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6cdc004390 0x7f6cec1f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 >> 172.21.15.27:0/1230352909 conn(0x7f6cec071960 msgr2=0x7f6cec1618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 shutdown_connections 2024-03-06T07:27:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.268+0000 7f6cf2e17640 1 -- 172.21.15.27:0/1230352909 wait complete. 2024-03-06T07:27:29.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:27:29.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.387+0000 7f7d5175c640 1 Processor -- start 2024-03-06T07:27:29.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.387+0000 7f7d5175c640 1 -- start start 2024-03-06T07:27:29.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.388+0000 7f7d5175c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c072090 0x7f7d4c15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.388+0000 7f7d5175c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c0b4160 con 0x7f7d4c06e700 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.388+0000 7f7d5175c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c0b5b40 con 0x7f7d4c072660 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.388+0000 7f7d5175c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0b3ba0 con 0x7f7d4c072090 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.388+0000 7f7d4affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7d4c06e700 0x7f7d4c06d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39834/0 (socket says 172.21.15.27:39834) 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.388+0000 7f7d4affd640 1 -- 172.21.15.27:0/970905714 learned_addr learned my addr 172.21.15.27:0/970905714 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4b7fe640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c072090 0x7f7d4c15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076823538 0 0) 0x7f7d4c0b4160 con 0x7f7d4c06e700 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d38003520 con 0x7f7d4c06e700 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3593101347 0 0) 0x7f7d38003520 con 0x7f7d4c06e700 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c072090 msgr2=0x7f7d4c15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c072090 0x7f7d4c15ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 >> v1:172.21.15.69:6789/0 conn(0x7f7d4c072660 legacy=0x7f7d4c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c0b3e80 con 0x7f7d4c06e700 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f7d3c001670 con 0x7f7d4c06e700 2024-03-06T07:27:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 >> v1:172.21.15.27:6789/0 conn(0x7f7d4c06e700 legacy=0x7f7d4c06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d380039e0 0x7f7d38003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c072090 0x7f7d38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d38004d40 0x7f7d380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0b3ba0 con 0x7f7d380039e0 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3800b9f0 con 0x7f7d38004d40 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3800bbf0 con 0x7f7d4c072090 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4affd640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d380039e0 0x7f7d38003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d43fff640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c072090 0x7f7d38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4b7fe640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d38004d40 0x7f7d380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4affd640 1 -- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c072090 msgr2=0x7f7d38004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4affd640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c072090 0x7f7d38004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4affd640 1 -- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d38004d40 msgr2=0x7f7d380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4affd640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d38004d40 0x7f7d380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4affd640 1 -- 172.21.15.27:0/970905714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38003520 con 0x7f7d380039e0 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d43fff640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c072090 0x7f7d38004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.389+0000 7f7d4b7fe640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d38004d40 0x7f7d380051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.390+0000 7f7d4affd640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d380039e0 0x7f7d38003dc0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00c290 tx=0x7f7d38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409996af44826472 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.390+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c007d50 con 0x7f7d380039e0 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.390+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c030070 con 0x7f7d380039e0 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.390+0000 7f7d4a7fc640 1 -- 172.21.15.27:0/970905714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c005d30 con 0x7f7d380039e0 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.391+0000 7f7d5175c640 1 -- 172.21.15.27:0/970905714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d380039e0 msgr2=0x7f7d38003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.391+0000 7f7d5175c640 1 --2- 172.21.15.27:0/970905714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d380039e0 0x7f7d38003dc0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00c290 tx=0x7f7d38005830 comp rx=0 tx=0).stop 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.391+0000 7f7d4b7fe640 1 -- 172.21.15.27:0/970905714 reap_dead start 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.391+0000 7f7d5175c640 1 -- 172.21.15.27:0/970905714 shutdown_connections 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.391+0000 7f7d5175c640 1 -- 172.21.15.27:0/970905714 >> 172.21.15.27:0/970905714 conn(0x7f7d4c0b2720 msgr2=0x7f7d4c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.391+0000 7f7d5175c640 1 -- 172.21.15.27:0/970905714 shutdown_connections 2024-03-06T07:27:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.391+0000 7f7d5175c640 1 -- 172.21.15.27:0/970905714 wait complete. 2024-03-06T07:27:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.392+0000 7f7d5175c640 1 Processor -- start 2024-03-06T07:27:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.392+0000 7f7d5175c640 1 -- start start 2024-03-06T07:27:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d5175c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d380039e0 0x7f7d4c15b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d5175c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4c158bc0 0x7f7d4c1545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d5175c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c1f3ad0 0x7f7d4c1f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d5175c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0b5750 con 0x7f7d4c158bc0 2024-03-06T07:27:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d5175c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0b55d0 con 0x7f7d380039e0 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d5175c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0b58d0 con 0x7f7d4c1f3ad0 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4c158bc0 0x7f7d4c1545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d43fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4c158bc0 0x7f7d4c1545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36274/0 (socket says 172.21.15.27:36274) 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d43fff640 1 -- 172.21.15.27:0/675033251 learned_addr learned my addr 172.21.15.27:0/675033251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d4affd640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d380039e0 0x7f7d4c15b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d4b7fe640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c1f3ad0 0x7f7d4c1f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d43fff640 1 -- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c1f3ad0 msgr2=0x7f7d4c1f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d43fff640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c1f3ad0 0x7f7d4c1f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d43fff640 1 -- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d380039e0 msgr2=0x7f7d4c15b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d43fff640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d380039e0 0x7f7d4c15b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.393+0000 7f7d43fff640 1 -- 172.21.15.27:0/675033251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c1f4490 con 0x7f7d4c158bc0 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.394+0000 7f7d43fff640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4c158bc0 0x7f7d4c1545c0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400e800 tx=0x7f7d3400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.394+0000 7f7d437fe640 1 -- 172.21.15.27:0/675033251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d340050d0 con 0x7f7d4c158bc0 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.394+0000 7f7d437fe640 1 -- 172.21.15.27:0/675033251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34018070 con 0x7f7d4c158bc0 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.394+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c159410 con 0x7f7d4c158bc0 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.394+0000 7f7d437fe640 1 -- 172.21.15.27:0/675033251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34005470 con 0x7f7d4c158bc0 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.394+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c1599d0 con 0x7f7d4c158bc0 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.395+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c158490 con 0x7f7d4c158bc0 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.395+0000 7f7d437fe640 1 -- 172.21.15.27:0/675033251 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d3400c6c0 con 0x7f7d4c158bc0 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.396+0000 7f7d437fe640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.396+0000 7f7d437fe640 1 -- 172.21.15.27:0/675033251 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7d34089ed0 con 0x7f7d4c158bc0 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.396+0000 7f7d4affd640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.396+0000 7f7d4affd640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c034000 tx=0x7f7d3c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.403+0000 7f7d437fe640 1 -- 172.21.15.27:0/675033251 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d4c158490 con 0x7f7d4c158bc0 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.701+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7d4c0b5b40 con 0x7f7d4c158bc0 2024-03-06T07:27:29.703 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:27:29.701+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.703+0000 7f7d437fe640 1 -- 172.21.15.27:0/675033251 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v53) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7d4c0b5b40 con 0x7f7d4c158bc0 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:27:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d2c06b500 msgr2=0x7f7d2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c034000 tx=0x7f7d3c009030 comp rx=0 tx=0).stop 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4c158bc0 msgr2=0x7f7d4c1545c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4c158bc0 0x7f7d4c1545c0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400e800 tx=0x7f7d3400ecc0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 shutdown_connections 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7d4c1f3ad0 0x7f7d4c1f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4c158bc0 0x7f7d4c1545c0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 --2- 172.21.15.27:0/675033251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d380039e0 0x7f7d4c15b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 >> 172.21.15.27:0/675033251 conn(0x7f7d4c0b2720 msgr2=0x7f7d4c06d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 shutdown_connections 2024-03-06T07:27:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.706+0000 7f7d5175c640 1 -- 172.21.15.27:0/675033251 wait complete. 2024-03-06T07:27:29.710 INFO:tasks.ceph.mds.a.smithi027.stderr: -2881> 2024-03-06T07:27:15.345+0000 7faaa49fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:29.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:27:29.738 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:27:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.827+0000 7efe05fa0640 1 Processor -- start 2024-03-06T07:27:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.827+0000 7efe05fa0640 1 -- start start 2024-03-06T07:27:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.828+0000 7efe05fa0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe00144640 0x7efe0006e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.828+0000 7efe05fa0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efe0006e940 0x7efe0015b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.828+0000 7efe05fa0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0015bc40 con 0x7efe0015b670 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.828+0000 7efe05fa0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0006df70 con 0x7efe00144640 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.828+0000 7efe05fa0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0015bf20 con 0x7efe0006e940 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.828+0000 7efdff7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe00144640 0x7efe0006e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.828+0000 7efdff7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe00144640 0x7efe0006e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36296/0 (socket says 172.21.15.27:36296) 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.828+0000 7efdff7fe640 1 -- 172.21.15.27:0/2294955581 learned_addr learned my addr 172.21.15.27:0/2294955581 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.829+0000 7efdfeffd640 1 --2- 172.21.15.27:0/2294955581 >> v2:172.21.15.69:3300/0 conn(0x7efe0006e940 0x7efe0015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.829+0000 7efdff7fe640 1 -- 172.21.15.27:0/2294955581 >> v2:172.21.15.69:3300/0 conn(0x7efe0006e940 msgr2=0x7efe0015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.829+0000 7efdff7fe640 1 --2- 172.21.15.27:0/2294955581 >> v2:172.21.15.69:3300/0 conn(0x7efe0006e940 0x7efe0015b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.829+0000 7efdff7fe640 1 -- 172.21.15.27:0/2294955581 >> v1:172.21.15.69:6789/0 conn(0x7efe0015b670 legacy=0x7efe000ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.829+0000 7efdff7fe640 1 -- 172.21.15.27:0/2294955581 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0006dc00 con 0x7efe00144640 2024-03-06T07:27:29.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efdff7fe640 1 --2- 172.21.15.27:0/2294955581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 0x7efe0006e400 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7efe000cf8a0 tx=0x7efdf002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a586e52860446c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efdfe7fc640 1 -- 172.21.15.27:0/2294955581 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0035020 con 0x7efe00144640 2024-03-06T07:27:29.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efdfe7fc640 1 -- 172.21.15.27:0/2294955581 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf0031870 con 0x7efe00144640 2024-03-06T07:27:29.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efdfe7fc640 1 -- 172.21.15.27:0/2294955581 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0031c40 con 0x7efe00144640 2024-03-06T07:27:29.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efe05fa0640 1 -- 172.21.15.27:0/2294955581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 msgr2=0x7efe0006e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efe05fa0640 1 --2- 172.21.15.27:0/2294955581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 0x7efe0006e400 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7efe000cf8a0 tx=0x7efdf002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:29.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efe05fa0640 1 -- 172.21.15.27:0/2294955581 shutdown_connections 2024-03-06T07:27:29.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efe05fa0640 1 --2- 172.21.15.27:0/2294955581 >> v2:172.21.15.69:3300/0 conn(0x7efe0006e940 0x7efe0015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efe05fa0640 1 --2- 172.21.15.27:0/2294955581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 0x7efe0006e400 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efe05fa0640 1 -- 172.21.15.27:0/2294955581 >> 172.21.15.27:0/2294955581 conn(0x7efe00066d00 msgr2=0x7efe00067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efe05fa0640 1 -- 172.21.15.27:0/2294955581 shutdown_connections 2024-03-06T07:27:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.830+0000 7efe05fa0640 1 -- 172.21.15.27:0/2294955581 wait complete. 2024-03-06T07:27:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.831+0000 7efe05fa0640 1 Processor -- start 2024-03-06T07:27:29.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efe05fa0640 1 -- start start 2024-03-06T07:27:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efe05fa0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0006e940 0x7efe0015a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efe05fa0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 0x7efe000a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efe05fa0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0015b670 0x7efe001e6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efe05fa0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe000cf190 con 0x7efe00144640 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efe05fa0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe000cf010 con 0x7efe0015b670 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efe05fa0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe000b36d0 con 0x7efe0006e940 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efdfeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 0x7efe000a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efdfeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 0x7efe000a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36306/0 (socket says 172.21.15.27:36306) 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.832+0000 7efdfeffd640 1 -- 172.21.15.27:0/3540645313 learned_addr learned my addr 172.21.15.27:0/3540645313 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdff7fe640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0006e940 0x7efe0015a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdfffff640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0015b670 0x7efe001e6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdfeffd640 1 -- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0006e940 msgr2=0x7efe0015a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdfeffd640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0006e940 0x7efe0015a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdfeffd640 1 -- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0015b670 msgr2=0x7efe001e6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdfeffd640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0015b670 0x7efe001e6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdfeffd640 1 -- 172.21.15.27:0/3540645313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe00132fc0 con 0x7efe00144640 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdfffff640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0015b670 0x7efe001e6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdff7fe640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0006e940 0x7efe0015a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efdfeffd640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 0x7efe000a94a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7efdf400d800 tx=0x7efdf400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efde7fff640 1 -- 172.21.15.27:0/3540645313 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf400b6c0 con 0x7efe00144640 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efde7fff640 1 -- 172.21.15.27:0/3540645313 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf4019070 con 0x7efe00144640 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe001332a0 con 0x7efe00144640 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efde7fff640 1 -- 172.21.15.27:0/3540645313 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf400ba80 con 0x7efe00144640 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.833+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe00133860 con 0x7efe00144640 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.834+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe000733c0 con 0x7efe00144640 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.835+0000 7efde7fff640 1 -- 172.21.15.27:0/3540645313 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdf400bc20 con 0x7efe00144640 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.835+0000 7efde7fff640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efdec06b4d0 0x7efdec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.835+0000 7efde7fff640 1 -- 172.21.15.27:0/3540645313 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5824+0+0 (secure 0 0 0) 0x7efdf408a6c0 con 0x7efe00144640 2024-03-06T07:27:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.835+0000 7efdff7fe640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efdec06b4d0 0x7efdec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.836+0000 7efdff7fe640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efdec06b4d0 0x7efdec06d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efdf0036000 tx=0x7efdf003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:29.839+0000 7efde7fff640 1 -- 172.21.15.27:0/3540645313 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdf401c360 con 0x7efe00144640 2024-03-06T07:27:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.059+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efe0004f410 con 0x7efe00144640 2024-03-06T07:27:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.708+0000 7efde7fff640 1 -- 172.21.15.27:0/3540645313 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v54) v1 ==== 104+0+0 (secure 0 0 0) 0x7efdf4059560 con 0x7efe00144640 2024-03-06T07:27:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efdec06b4d0 msgr2=0x7efdec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efdec06b4d0 0x7efdec06d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efdf0036000 tx=0x7efdf003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 msgr2=0x7efe000a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 0x7efe000a94a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7efdf400d800 tx=0x7efdf400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 shutdown_connections 2024-03-06T07:27:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efdec06b4d0 0x7efdec06d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0015b670 0x7efe001e6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe00144640 0x7efe000a94a0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 --2- 172.21.15.27:0/3540645313 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0006e940 0x7efe0015a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 >> 172.21.15.27:0/3540645313 conn(0x7efe00066d00 msgr2=0x7efe00142690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 shutdown_connections 2024-03-06T07:27:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.711+0000 7efe05fa0640 1 -- 172.21.15.27:0/3540645313 wait complete. 2024-03-06T07:27:30.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:27:30.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.883+0000 7f6fe5488640 1 Processor -- start 2024-03-06T07:27:30.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.883+0000 7f6fe5488640 1 -- start start 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fe5488640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6fe0144640 0x7f6fe006e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fe5488640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe006e940 0x7f6fe015b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fe5488640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe015bc40 con 0x7f6fe015b670 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fe5488640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe006df70 con 0x7f6fe0144640 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fe5488640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe015bf20 con 0x7f6fe006e940 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fdeffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6fe0144640 0x7f6fe006e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe006e940 0x7f6fe015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fdeffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6fe0144640 0x7f6fe006e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:44624/0 (socket says 172.21.15.27:44624) 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fdeffd640 1 -- 172.21.15.27:0/3559517815 learned_addr learned my addr 172.21.15.27:0/3559517815 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:30.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fdeffd640 1 -- 172.21.15.27:0/3559517815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe006e940 msgr2=0x7f6fe015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fdeffd640 1 --2- 172.21.15.27:0/3559517815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe006e940 0x7f6fe015b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.884+0000 7f6fdeffd640 1 -- 172.21.15.27:0/3559517815 >> v1:172.21.15.69:6789/0 conn(0x7f6fe015b670 legacy=0x7f6fe00ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.885+0000 7f6fdeffd640 1 -- 172.21.15.27:0/3559517815 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe006dc00 con 0x7f6fe0144640 2024-03-06T07:27:30.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.885+0000 7f6fd7fff640 1 --2- 172.21.15.27:0/3559517815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe006e940 0x7f6fe015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:30.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.885+0000 7f6fdeffd640 1 --2- 172.21.15.27:0/3559517815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0144640 0x7f6fe006e400 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000bb10 tx=0x7f6fd002e150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=616acc3d4f63eccd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.885+0000 7f6fde7fc640 1 -- 172.21.15.27:0/3559517815 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd003b070 con 0x7f6fe0144640 2024-03-06T07:27:30.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fde7fc640 1 -- 172.21.15.27:0/3559517815 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd0036440 con 0x7f6fe0144640 2024-03-06T07:27:30.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fde7fc640 1 -- 172.21.15.27:0/3559517815 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0036810 con 0x7f6fe0144640 2024-03-06T07:27:30.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fe5488640 1 -- 172.21.15.27:0/3559517815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0144640 msgr2=0x7f6fe006e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fe5488640 1 --2- 172.21.15.27:0/3559517815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0144640 0x7f6fe006e400 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000bb10 tx=0x7f6fd002e150 comp rx=0 tx=0).stop 2024-03-06T07:27:30.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fe5488640 1 -- 172.21.15.27:0/3559517815 shutdown_connections 2024-03-06T07:27:30.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fe5488640 1 --2- 172.21.15.27:0/3559517815 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe006e940 0x7f6fe015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fe5488640 1 --2- 172.21.15.27:0/3559517815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe0144640 0x7f6fe006e400 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fe5488640 1 -- 172.21.15.27:0/3559517815 >> 172.21.15.27:0/3559517815 conn(0x7f6fe0066d00 msgr2=0x7f6fe0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fe5488640 1 -- 172.21.15.27:0/3559517815 shutdown_connections 2024-03-06T07:27:30.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.886+0000 7f6fe5488640 1 -- 172.21.15.27:0/3559517815 wait complete. 2024-03-06T07:27:30.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.887+0000 7f6fe5488640 1 Processor -- start 2024-03-06T07:27:30.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.887+0000 7f6fe5488640 1 -- start start 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fe5488640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe006e940 0x7f6fe015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fe5488640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe0144640 0x7f6fe00b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fe5488640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe015b670 0x7f6fe00ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fe5488640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00cf190 con 0x7f6fe015b670 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fe5488640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00cf010 con 0x7f6fe006e940 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fe5488640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00b36d0 con 0x7f6fe0144640 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fdf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe015b670 0x7f6fe00ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fdeffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe006e940 0x7f6fe015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fdf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe015b670 0x7f6fe00ae0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36314/0 (socket says 172.21.15.27:36314) 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fdf7fe640 1 -- 172.21.15.27:0/419231138 learned_addr learned my addr 172.21.15.27:0/419231138 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.888+0000 7f6fd7fff640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe0144640 0x7f6fe00b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fdeffd640 1 -- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe0144640 msgr2=0x7f6fe00b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fdeffd640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe0144640 0x7f6fe00b0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fdeffd640 1 -- 172.21.15.27:0/419231138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe015b670 msgr2=0x7f6fe00ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fdeffd640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe015b670 0x7f6fe00ae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fdeffd640 1 -- 172.21.15.27:0/419231138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe00b0b80 con 0x7f6fe006e940 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fdeffd640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe006e940 0x7f6fe015a330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0035000 tx=0x7f6fd00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fd77fe640 1 -- 172.21.15.27:0/419231138 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd003b040 con 0x7f6fe006e940 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fd77fe640 1 -- 172.21.15.27:0/419231138 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd0046070 con 0x7f6fe006e940 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe00b0e60 con 0x7f6fe006e940 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fd77fe640 1 -- 172.21.15.27:0/419231138 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0034920 con 0x7f6fe006e940 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.889+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe00a95d0 con 0x7f6fe006e940 2024-03-06T07:27:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.890+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe00733c0 con 0x7f6fe006e940 2024-03-06T07:27:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.891+0000 7f6fd77fe640 1 -- 172.21.15.27:0/419231138 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fd0034ac0 con 0x7f6fe006e940 2024-03-06T07:27:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.891+0000 7f6fd77fe640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fc006b4d0 0x7f6fc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.891+0000 7f6fd77fe640 1 -- 172.21.15.27:0/419231138 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f6fd00aeab0 con 0x7f6fe006e940 2024-03-06T07:27:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.891+0000 7f6fd7fff640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fc006b4d0 0x7f6fc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.892+0000 7f6fd7fff640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fc006b4d0 0x7f6fc006d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009560 tx=0x7f6fc8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:30.896+0000 7f6fd77fe640 1 -- 172.21.15.27:0/419231138 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fd007d300 con 0x7f6fe006e940 2024-03-06T07:27:31.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.076+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6fe004f410 con 0x7f6fe006e940 2024-03-06T07:27:31.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.713+0000 7f6fd77fe640 1 -- 172.21.15.27:0/419231138 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v88) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6fd00813a0 con 0x7f6fe006e940 2024-03-06T07:27:31.716 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:27:31.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fc006b4d0 msgr2=0x7f6fc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fc006b4d0 0x7f6fc006d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009560 tx=0x7f6fc8009190 comp rx=0 tx=0).stop 2024-03-06T07:27:31.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe006e940 msgr2=0x7f6fe015a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe006e940 0x7f6fe015a330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0035000 tx=0x7f6fd00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 shutdown_connections 2024-03-06T07:27:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fc006b4d0 0x7f6fc006d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe015b670 0x7f6fe00ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fe0144640 0x7f6fe00b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 --2- 172.21.15.27:0/419231138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fe006e940 0x7f6fe015a330 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 >> 172.21.15.27:0/419231138 conn(0x7f6fe0066d00 msgr2=0x7f6fe015d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 shutdown_connections 2024-03-06T07:27:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.717+0000 7f6fe5488640 1 -- 172.21.15.27:0/419231138 wait complete. 2024-03-06T07:27:31.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:27:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.849+0000 7fecd19eb640 1 Processor -- start 2024-03-06T07:27:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.849+0000 7fecd19eb640 1 -- start start 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7fecd19eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feccc1618e0 0x7feccc06b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7fecd19eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0ce4f0 0x7feccc0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7fecd19eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feccc161d40 0x7feccc1621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7fecd19eb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feccc0cfce0 con 0x7feccc161d40 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7fecd19eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feccc0cee70 con 0x7feccc1618e0 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7fecd19eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc15f730 con 0x7feccc0ce4f0 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feccc161d40 0x7feccc1621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feccc161d40 0x7feccc1621d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36318/0 (socket says 172.21.15.27:36318) 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccb7fe640 1 -- 172.21.15.27:0/1336315477 learned_addr learned my addr 172.21.15.27:0/1336315477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccaffd640 1 --2- 172.21.15.27:0/1336315477 >> v2:172.21.15.69:3300/0 conn(0x7feccc1618e0 0x7feccc06b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7fecca7fc640 1 --2- 172.21.15.27:0/1336315477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0ce4f0 0x7feccc0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccb7fe640 1 -- 172.21.15.27:0/1336315477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0ce4f0 msgr2=0x7feccc0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccb7fe640 1 --2- 172.21.15.27:0/1336315477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0ce4f0 0x7feccc0ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccb7fe640 1 -- 172.21.15.27:0/1336315477 >> v2:172.21.15.69:3300/0 conn(0x7feccc1618e0 msgr2=0x7feccc06b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccb7fe640 1 --2- 172.21.15.27:0/1336315477 >> v2:172.21.15.69:3300/0 conn(0x7feccc1618e0 0x7feccc06b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccb7fe640 1 -- 172.21.15.27:0/1336315477 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc15ebb0 con 0x7feccc161d40 2024-03-06T07:27:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7feccaffd640 1 --2- 172.21.15.27:0/1336315477 >> v2:172.21.15.69:3300/0 conn(0x7feccc1618e0 0x7feccc06b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.850+0000 7fecca7fc640 1 --2- 172.21.15.27:0/1336315477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0ce4f0 0x7feccc0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:31.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.851+0000 7feccb7fe640 1 --2- 172.21.15.27:0/1336315477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 0x7feccc1621d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7feccc071250 tx=0x7fecc402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb73439e4aeec6d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.851+0000 7fecc9ffb640 1 -- 172.21.15.27:0/1336315477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc402cb00 con 0x7feccc161d40 2024-03-06T07:27:31.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.851+0000 7fecc9ffb640 1 -- 172.21.15.27:0/1336315477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc402cca0 con 0x7feccc161d40 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.851+0000 7fecc9ffb640 1 -- 172.21.15.27:0/1336315477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4034610 con 0x7feccc161d40 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.852+0000 7fecd19eb640 1 -- 172.21.15.27:0/1336315477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 msgr2=0x7feccc1621d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.852+0000 7fecd19eb640 1 --2- 172.21.15.27:0/1336315477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 0x7feccc1621d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7feccc071250 tx=0x7fecc402bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.852+0000 7fecd19eb640 1 -- 172.21.15.27:0/1336315477 shutdown_connections 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.852+0000 7fecd19eb640 1 --2- 172.21.15.27:0/1336315477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 0x7feccc1621d0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.852+0000 7fecd19eb640 1 --2- 172.21.15.27:0/1336315477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0ce4f0 0x7feccc0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.852+0000 7fecd19eb640 1 --2- 172.21.15.27:0/1336315477 >> v2:172.21.15.69:3300/0 conn(0x7feccc1618e0 0x7feccc06b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.852+0000 7fecd19eb640 1 -- 172.21.15.27:0/1336315477 >> 172.21.15.27:0/1336315477 conn(0x7feccc0b3660 msgr2=0x7feccc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.852+0000 7fecd19eb640 1 -- 172.21.15.27:0/1336315477 shutdown_connections 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.852+0000 7fecd19eb640 1 -- 172.21.15.27:0/1336315477 wait complete. 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.853+0000 7fecd19eb640 1 Processor -- start 2024-03-06T07:27:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.853+0000 7fecd19eb640 1 -- start start 2024-03-06T07:27:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecd19eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feccc0ce4f0 0x7feccc153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecd19eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 0x7feccc154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecd19eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0b4e70 0x7feccc0b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecd19eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc15f450 con 0x7feccc161d40 2024-03-06T07:27:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecd19eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc15f170 con 0x7feccc0ce4f0 2024-03-06T07:27:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecd19eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc06b670 con 0x7feccc0b4e70 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecca7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 0x7feccc154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecca7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 0x7feccc154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36332/0 (socket says 172.21.15.27:36332) 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecca7fc640 1 -- 172.21.15.27:0/4137944759 learned_addr learned my addr 172.21.15.27:0/4137944759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7feccaffd640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feccc0ce4f0 0x7feccc153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7feccb7fe640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0b4e70 0x7feccc0b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecca7fc640 1 -- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0b4e70 msgr2=0x7feccc0b5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecca7fc640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0b4e70 0x7feccc0b5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecca7fc640 1 -- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feccc0ce4f0 msgr2=0x7feccc153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecca7fc640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feccc0ce4f0 0x7feccc153c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecca7fc640 1 -- 172.21.15.27:0/4137944759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc0b77a0 con 0x7feccc161d40 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7feccb7fe640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0b4e70 0x7feccc0b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7feccaffd640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feccc0ce4f0 0x7feccc153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.854+0000 7fecca7fc640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 0x7feccc154190 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fecc000c820 tx=0x7fecc000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.855+0000 7fecb3fff640 1 -- 172.21.15.27:0/4137944759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc0015070 con 0x7feccc161d40 2024-03-06T07:27:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.855+0000 7fecb3fff640 1 -- 172.21.15.27:0/4137944759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc0007a30 con 0x7feccc161d40 2024-03-06T07:27:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.855+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feccc0c7fd0 con 0x7feccc161d40 2024-03-06T07:27:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.855+0000 7fecb3fff640 1 -- 172.21.15.27:0/4137944759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc0007e20 con 0x7feccc161d40 2024-03-06T07:27:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.855+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feccc0b6cc0 con 0x7feccc161d40 2024-03-06T07:27:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.856+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feccc09fc50 con 0x7feccc161d40 2024-03-06T07:27:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.856+0000 7fecb3fff640 1 -- 172.21.15.27:0/4137944759 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fecc0018070 con 0x7feccc161d40 2024-03-06T07:27:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.856+0000 7fecb3fff640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecb806b4d0 0x7fecb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.857+0000 7fecb3fff640 1 -- 172.21.15.27:0/4137944759 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5386+0+0 (secure 0 0 0) 0x7fecc0089f80 con 0x7feccc161d40 2024-03-06T07:27:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.857+0000 7feccaffd640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecb806b4d0 0x7fecb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.857+0000 7feccaffd640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecb806b4d0 0x7fecb806d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7feccc0c54e0 tx=0x7fecbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:31.861+0000 7fecb3fff640 1 -- 172.21.15.27:0/4137944759 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fecc0058f50 con 0x7feccc161d40 2024-03-06T07:27:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.047+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feccc0b7ad0 con 0x7feccc161d40 2024-03-06T07:27:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.714+0000 7fecb3fff640 1 -- 172.21.15.27:0/4137944759 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v89) v1 ==== 169+0+0 (secure 0 0 0) 0x7fecc0058850 con 0x7feccc161d40 2024-03-06T07:27:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:27:32.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecb806b4d0 msgr2=0x7fecb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecb806b4d0 0x7fecb806d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7feccc0c54e0 tx=0x7fecbc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:32.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 msgr2=0x7feccc154190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 0x7feccc154190 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fecc000c820 tx=0x7fecc000cce0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 shutdown_connections 2024-03-06T07:27:32.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecb806b4d0 0x7fecb806d990 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feccc0b4e70 0x7feccc0b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feccc161d40 0x7feccc154190 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 --2- 172.21.15.27:0/4137944759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feccc0ce4f0 0x7feccc153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 >> 172.21.15.27:0/4137944759 conn(0x7feccc0b3660 msgr2=0x7feccc0cc410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 shutdown_connections 2024-03-06T07:27:32.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.717+0000 7fecd19eb640 1 -- 172.21.15.27:0/4137944759 wait complete. 2024-03-06T07:27:32.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:27:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.850+0000 7ff6693f7640 1 Processor -- start 2024-03-06T07:27:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.850+0000 7ff6693f7640 1 -- start start 2024-03-06T07:27:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.850+0000 7ff6693f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff664166cb0 0x7ff6641690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.850+0000 7ff6693f7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff664161a40 con 0x7ff6640b2cd0 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.850+0000 7ff6693f7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6640b2370 con 0x7ff6640cdcd0 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.850+0000 7ff6693f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640d11e0 con 0x7ff664166cb0 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff65bfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff6640b2cd0 0x7ff6640b1c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39838/0 (socket says 172.21.15.27:39838) 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff65bfff640 1 -- 172.21.15.27:0/192665965 learned_addr learned my addr 172.21.15.27:0/192665965 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff6637fe640 1 --2- 172.21.15.27:0/192665965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff664166cb0 0x7ff6641690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff6627fc640 1 -- 172.21.15.27:0/192665965 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914822344 0 0) 0x7ff664161a40 con 0x7ff6640b2cd0 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff6627fc640 1 -- 172.21.15.27:0/192665965 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff650003520 con 0x7ff6640b2cd0 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff6627fc640 1 -- 172.21.15.27:0/192665965 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615068439 0 0) 0x7ff6640b2370 con 0x7ff6640cdcd0 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff6627fc640 1 -- 172.21.15.27:0/192665965 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff664161a40 con 0x7ff6640cdcd0 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff6637fe640 1 -- 172.21.15.27:0/192665965 >> v1:172.21.15.69:6789/0 conn(0x7ff6640cdcd0 legacy=0x7ff6640b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff6637fe640 1 -- 172.21.15.27:0/192665965 >> v1:172.21.15.27:6789/0 conn(0x7ff6640b2cd0 legacy=0x7ff6640b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff6627fc640 1 -- 172.21.15.27:0/192665965 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2789620760 0 0) 0x7ff650003520 con 0x7ff6640b2cd0 2024-03-06T07:27:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.851+0000 7ff6637fe640 1 -- 172.21.15.27:0/192665965 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66406c2d0 con 0x7ff664166cb0 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.852+0000 7ff6637fe640 1 --2- 172.21.15.27:0/192665965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff664166cb0 0x7ff6641690a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009dc0 tx=0x7ff65c02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be8b7982f381f982 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.852+0000 7ff6627fc640 1 -- 172.21.15.27:0/192665965 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c039070 con 0x7ff664166cb0 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.852+0000 7ff6627fc640 1 -- 172.21.15.27:0/192665965 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c02c880 con 0x7ff664166cb0 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.852+0000 7ff6627fc640 1 -- 172.21.15.27:0/192665965 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c001670 con 0x7ff664166cb0 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.853+0000 7ff6693f7640 1 -- 172.21.15.27:0/192665965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff664166cb0 msgr2=0x7ff6641690a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.853+0000 7ff6693f7640 1 --2- 172.21.15.27:0/192665965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff664166cb0 0x7ff6641690a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009dc0 tx=0x7ff65c02bd50 comp rx=0 tx=0).stop 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.853+0000 7ff6693f7640 1 -- 172.21.15.27:0/192665965 shutdown_connections 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.853+0000 7ff6693f7640 1 --2- 172.21.15.27:0/192665965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff664166cb0 0x7ff6641690a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.853+0000 7ff6693f7640 1 -- 172.21.15.27:0/192665965 >> 172.21.15.27:0/192665965 conn(0x7ff6640b3700 msgr2=0x7ff6640b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.853+0000 7ff6693f7640 1 -- 172.21.15.27:0/192665965 shutdown_connections 2024-03-06T07:27:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.853+0000 7ff6693f7640 1 -- 172.21.15.27:0/192665965 wait complete. 2024-03-06T07:27:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.854+0000 7ff6693f7640 1 Processor -- start 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.854+0000 7ff6693f7640 1 -- start start 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6693f7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6640b2cd0 0x7ff66415cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6693f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6640cdcd0 0x7ff6640ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6693f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664166cb0 0x7ff664148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6693f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640d0c20 con 0x7ff664166cb0 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6693f7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640d0940 con 0x7ff6640b2cd0 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6693f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640d0f00 con 0x7ff6640cdcd0 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664166cb0 0x7ff664148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6637fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664166cb0 0x7ff664148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36340/0 (socket says 172.21.15.27:36340) 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6637fe640 1 -- 172.21.15.27:0/3778270060 learned_addr learned my addr 172.21.15.27:0/3778270060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff65bfff640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6640cdcd0 0x7ff6640ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6637fe640 1 -- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6640cdcd0 msgr2=0x7ff6640ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6637fe640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6640cdcd0 0x7ff6640ca620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6637fe640 1 -- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6640b2cd0 msgr2=0x7ff66415cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6637fe640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6640b2cd0 0x7ff66415cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6637fe640 1 -- 172.21.15.27:0/3778270060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6641494a0 con 0x7ff664166cb0 2024-03-06T07:27:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff6637fe640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664166cb0 0x7ff664148f60 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ff65c037040 tx=0x7ff65c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3778270060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c039050 con 0x7ff664166cb0 2024-03-06T07:27:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.855+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3778270060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c045070 con 0x7ff664166cb0 2024-03-06T07:27:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.856+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66415d3b0 con 0x7ff664166cb0 2024-03-06T07:27:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.856+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3778270060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c009450 con 0x7ff664166cb0 2024-03-06T07:27:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.856+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66415d970 con 0x7ff664166cb0 2024-03-06T07:27:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.857+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66409fcf0 con 0x7ff664166cb0 2024-03-06T07:27:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.857+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3778270060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff65c040690 con 0x7ff664166cb0 2024-03-06T07:27:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.857+0000 7ff65b7fe640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff64406b4d0 0x7ff64406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.857+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3778270060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff65c0ac730 con 0x7ff664166cb0 2024-03-06T07:27:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.858+0000 7ff662ffd640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff64406b4d0 0x7ff64406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.858+0000 7ff662ffd640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff64406b4d0 0x7ff64406d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff6540010e0 tx=0x7ff654007950 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:32.862+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3778270060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff65c07b8b0 con 0x7ff664166cb0 2024-03-06T07:27:33.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.047+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff6640d0720 con 0x7ff664166cb0 2024-03-06T07:27:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.051+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3778270060 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff65c07b1b0 con 0x7ff664166cb0 2024-03-06T07:27:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff64406b4d0 msgr2=0x7ff64406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff64406b4d0 0x7ff64406d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff6540010e0 tx=0x7ff654007950 comp rx=0 tx=0).stop 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664166cb0 msgr2=0x7ff664148f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664166cb0 0x7ff664148f60 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ff65c037040 tx=0x7ff65c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 shutdown_connections 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff64406b4d0 0x7ff64406d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664166cb0 0x7ff664148f60 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff6640cdcd0 0x7ff6640ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 --2- 172.21.15.27:0/3778270060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6640b2cd0 0x7ff66415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 >> 172.21.15.27:0/3778270060 conn(0x7ff6640b3700 msgr2=0x7ff6640cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 shutdown_connections 2024-03-06T07:27:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.053+0000 7ff6693f7640 1 -- 172.21.15.27:0/3778270060 wait complete. 2024-03-06T07:27:33.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:27:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.172+0000 7f9af550f640 1 Processor -- start 2024-03-06T07:27:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.172+0000 7f9af550f640 1 -- start start 2024-03-06T07:27:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9af550f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9af00d0880 0x7f9af0168c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9af550f640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00cf130 con 0x7f9af015e060 2024-03-06T07:27:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9af550f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00d0dd0 con 0x7f9af006fe20 2024-03-06T07:27:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9af550f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af00d10b0 con 0x7f9af00d0880 2024-03-06T07:27:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aeeffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9af015e060 0x7f9af00ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39852/0 (socket says 172.21.15.27:39852) 2024-03-06T07:27:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aeeffd640 1 -- 172.21.15.27:0/3416892128 learned_addr learned my addr 172.21.15.27:0/3416892128 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aef7fe640 1 --2- 172.21.15.27:0/3416892128 >> v2:172.21.15.69:3300/0 conn(0x7f9af00d0880 0x7f9af0168c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513407315 0 0) 0x7f9af00cf130 con 0x7f9af015e060 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad8003520 con 0x7f9af015e060 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2538214560 0 0) 0x7f9ad8003520 con 0x7f9af015e060 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 >> v2:172.21.15.69:3300/0 conn(0x7f9af00d0880 msgr2=0x7f9af0168c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aedffb640 1 --2- 172.21.15.27:0/3416892128 >> v2:172.21.15.69:3300/0 conn(0x7f9af00d0880 0x7f9af0168c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 >> v1:172.21.15.69:6789/0 conn(0x7f9af006fe20 legacy=0x7f9af00cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af00cc0e0 con 0x7f9af015e060 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.173+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f9ae0001670 con 0x7f9af015e060 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 >> v1:172.21.15.27:6789/0 conn(0x7f9af015e060 legacy=0x7f9af00ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aedffb640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad80039b0 0x7f9ad8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aedffb640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ad8004800 0x7f9ad8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aedffb640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad80051c0 0x7f9ad800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00d10b0 con 0x7f9ad8004800 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad800c060 con 0x7f9ad80039b0 2024-03-06T07:27:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad800c260 con 0x7f9ad80051c0 2024-03-06T07:27:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aeeffd640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad80039b0 0x7f9ad8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aee7fc640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ad8004800 0x7f9ad8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aef7fe640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad80051c0 0x7f9ad800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aee7fc640 1 -- 172.21.15.27:0/3416892128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad80051c0 msgr2=0x7f9ad800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aee7fc640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad80051c0 0x7f9ad800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aee7fc640 1 -- 172.21.15.27:0/3416892128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad80039b0 msgr2=0x7f9ad8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aee7fc640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad80039b0 0x7f9ad8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aef7fe640 1 -- 172.21.15.27:0/3416892128 reap_dead start 2024-03-06T07:27:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aee7fc640 1 -- 172.21.15.27:0/3416892128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad8003520 con 0x7f9ad8004800 2024-03-06T07:27:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.174+0000 7f9aeeffd640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad80039b0 0x7f9ad8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.175+0000 7f9aee7fc640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ad8004800 0x7f9ad8004c80 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4001720 tx=0x7f9ad8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78506fa7f08ecef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.175+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4001370 con 0x7f9ad8004800 2024-03-06T07:27:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.175+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae40052d0 con 0x7f9ad8004800 2024-03-06T07:27:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.175+0000 7f9aedffb640 1 -- 172.21.15.27:0/3416892128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4003830 con 0x7f9ad8004800 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.175+0000 7f9af550f640 1 -- 172.21.15.27:0/3416892128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ad8004800 msgr2=0x7f9ad8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.175+0000 7f9af550f640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ad8004800 0x7f9ad8004c80 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4001720 tx=0x7f9ad8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.175+0000 7f9af550f640 1 -- 172.21.15.27:0/3416892128 shutdown_connections 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.175+0000 7f9af550f640 1 --2- 172.21.15.27:0/3416892128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ad8004800 0x7f9ad8004c80 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.175+0000 7f9af550f640 1 -- 172.21.15.27:0/3416892128 >> 172.21.15.27:0/3416892128 conn(0x7f9af00d1a90 msgr2=0x7f9af00d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.176+0000 7f9af550f640 1 -- 172.21.15.27:0/3416892128 shutdown_connections 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.176+0000 7f9af550f640 1 -- 172.21.15.27:0/3416892128 wait complete. 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.177+0000 7f9af550f640 1 Processor -- start 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.177+0000 7f9af550f640 1 -- start start 2024-03-06T07:27:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9af550f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8004800 0x7f9af00cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9af550f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af015bad0 0x7f9af00c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9af550f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af01600f0 0x7f9af0160570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9af550f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af015e440 con 0x7f9af015bad0 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9af550f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00cc6a0 con 0x7f9af01600f0 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9af550f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00ce310 con 0x7f9ad8004800 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af015bad0 0x7f9af00c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af015bad0 0x7f9af00c9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36354/0 (socket says 172.21.15.27:36354) 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aee7fc640 1 -- 172.21.15.27:0/1805126307 learned_addr learned my addr 172.21.15.27:0/1805126307 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aeeffd640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8004800 0x7f9af00cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aef7fe640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af01600f0 0x7f9af0160570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aee7fc640 1 -- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8004800 msgr2=0x7f9af00cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aee7fc640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8004800 0x7f9af00cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aee7fc640 1 -- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af01600f0 msgr2=0x7f9af0160570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aee7fc640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af01600f0 0x7f9af0160570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aee7fc640 1 -- 172.21.15.27:0/1805126307 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af00c9a60 con 0x7f9af015bad0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aeeffd640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8004800 0x7f9af00cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9aee7fc640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af015bad0 0x7f9af00c9520 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4000c00 tx=0x7f9ae4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9ad77fe640 1 -- 172.21.15.27:0/1805126307 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4041020 con 0x7f9af015bad0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9ad77fe640 1 -- 172.21.15.27:0/1805126307 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae4005480 con 0x7f9af015bad0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af00c9d30 con 0x7f9af015bad0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.178+0000 7f9ad77fe640 1 -- 172.21.15.27:0/1805126307 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4001370 con 0x7f9af015bad0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.179+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af00ca2f0 con 0x7f9af015bad0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.180+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af0074980 con 0x7f9af015bad0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.180+0000 7f9ad77fe640 1 -- 172.21.15.27:0/1805126307 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ae4044020 con 0x7f9af015bad0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.180+0000 7f9ad77fe640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9adc06b4d0 0x7f9adc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.180+0000 7f9ad77fe640 1 -- 172.21.15.27:0/1805126307 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9ae40ab830 con 0x7f9af015bad0 2024-03-06T07:27:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.181+0000 7f9aeeffd640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9adc06b4d0 0x7f9adc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.181+0000 7f9aeeffd640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9adc06b4d0 0x7f9adc06d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0007fd0 tx=0x7f9ae0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.185+0000 7f9ad77fe640 1 -- 172.21.15.27:0/1805126307 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9af0074980 con 0x7f9af015bad0 2024-03-06T07:27:33.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.441+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f9af0067ff0 con 0x7f9af015bad0 2024-03-06T07:27:33.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.716+0000 7f9ad77fe640 1 -- 172.21.15.27:0/1805126307 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v276) v1 ==== 132+0+0 (secure 0 0 0) 0x7f9ae404b030 con 0x7f9af015bad0 2024-03-06T07:27:33.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9adc06b4d0 msgr2=0x7f9adc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9adc06b4d0 0x7f9adc06d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0007fd0 tx=0x7f9ae0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af015bad0 msgr2=0x7f9af00c9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af015bad0 0x7f9af00c9520 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4000c00 tx=0x7f9ae4001120 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 shutdown_connections 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9adc06b4d0 0x7f9adc06d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af01600f0 0x7f9af0160570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af015bad0 0x7f9af00c9520 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 --2- 172.21.15.27:0/1805126307 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9ad8004800 0x7f9af00cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 >> 172.21.15.27:0/1805126307 conn(0x7f9af00d1a90 msgr2=0x7f9af006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 shutdown_connections 2024-03-06T07:27:33.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.718+0000 7f9af550f640 1 -- 172.21.15.27:0/1805126307 wait complete. 2024-03-06T07:27:33.732 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:27:33.732 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:27:33.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:33.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.838+0000 7f3a857c9640 1 Processor -- start 2024-03-06T07:27:33.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.838+0000 7f3a857c9640 1 -- start start 2024-03-06T07:27:33.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.839+0000 7f3a857c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a80071540 0x7f3a800b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.839+0000 7f3a857c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a800b2b50 0x7f3a800b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.839+0000 7f3a857c9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a800ce610 con 0x7f3a800d3400 2024-03-06T07:27:33.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.839+0000 7f3a857c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a800d2f70 con 0x7f3a800b2b50 2024-03-06T07:27:33.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.839+0000 7f3a857c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a800d4be0 con 0x7f3a80071540 2024-03-06T07:27:33.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.839+0000 7f3a7f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a800b2b50 0x7f3a800b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.839+0000 7f3a7e7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3a800d3400 0x7f3a80071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35670/0 (socket says 172.21.15.27:35670) 2024-03-06T07:27:33.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.839+0000 7f3a7e7fc640 1 -- 172.21.15.27:0/2094114186 learned_addr learned my addr 172.21.15.27:0/2094114186 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:33.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.839+0000 7f3a7effd640 1 --2- 172.21.15.27:0/2094114186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a80071540 0x7f3a800b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.840+0000 7f3a7f7fe640 1 -- 172.21.15.27:0/2094114186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a80071540 msgr2=0x7f3a800b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.840+0000 7f3a7f7fe640 1 --2- 172.21.15.27:0/2094114186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a80071540 0x7f3a800b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.840+0000 7f3a7f7fe640 1 -- 172.21.15.27:0/2094114186 >> v1:172.21.15.27:6789/0 conn(0x7f3a800d3400 legacy=0x7f3a80071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.840+0000 7f3a7f7fe640 1 -- 172.21.15.27:0/2094114186 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a800d2920 con 0x7f3a800b2b50 2024-03-06T07:27:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.840+0000 7f3a7dffb640 1 -- 172.21.15.27:0/2094114186 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385727631 0 0) 0x7f3a800ce610 con 0x7f3a800d3400 2024-03-06T07:27:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.840+0000 7f3a7f7fe640 1 --2- 172.21.15.27:0/2094114186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 0x7f3a800b6e20 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009d90 tx=0x7f3a7002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf92028b179bc34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.840+0000 7f3a7dffb640 1 -- 172.21.15.27:0/2094114186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70039070 con 0x7f3a800b2b50 2024-03-06T07:27:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a7dffb640 1 -- 172.21.15.27:0/2094114186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7002cc50 con 0x7f3a800b2b50 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a7dffb640 1 -- 172.21.15.27:0/2094114186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70034440 con 0x7f3a800b2b50 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a857c9640 1 -- 172.21.15.27:0/2094114186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 msgr2=0x7f3a800b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a857c9640 1 --2- 172.21.15.27:0/2094114186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 0x7f3a800b6e20 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009d90 tx=0x7f3a7002bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a857c9640 1 -- 172.21.15.27:0/2094114186 shutdown_connections 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a857c9640 1 --2- 172.21.15.27:0/2094114186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 0x7f3a800b6e20 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a857c9640 1 --2- 172.21.15.27:0/2094114186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a80071540 0x7f3a800b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a857c9640 1 -- 172.21.15.27:0/2094114186 >> 172.21.15.27:0/2094114186 conn(0x7f3a800d3ca0 msgr2=0x7f3a800d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a857c9640 1 -- 172.21.15.27:0/2094114186 shutdown_connections 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.841+0000 7f3a857c9640 1 -- 172.21.15.27:0/2094114186 wait complete. 2024-03-06T07:27:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.843+0000 7f3a857c9640 1 Processor -- start 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.843+0000 7f3a857c9640 1 -- start start 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.843+0000 7f3a857c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a80071540 0x7f3a8015c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.843+0000 7f3a857c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 0x7f3a800b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.843+0000 7f3a857c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a800d3400 0x7f3a800b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.843+0000 7f3a857c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8015e370 con 0x7f3a800b2b50 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.843+0000 7f3a857c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a800ce8f0 con 0x7f3a80071540 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.843+0000 7f3a857c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8015d5b0 con 0x7f3a800d3400 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 0x7f3a800b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 0x7f3a800b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46056/0 (socket says 172.21.15.27:46056) 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7e7fc640 1 -- 172.21.15.27:0/4110415129 learned_addr learned my addr 172.21.15.27:0/4110415129 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7effd640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a80071540 0x7f3a8015c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7f7fe640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a800d3400 0x7f3a800b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7e7fc640 1 -- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a800d3400 msgr2=0x7f3a800b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7e7fc640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a800d3400 0x7f3a800b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7e7fc640 1 -- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a80071540 msgr2=0x7f3a8015c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7e7fc640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a80071540 0x7f3a8015c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7e7fc640 1 -- 172.21.15.27:0/4110415129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a801f3fb0 con 0x7f3a800b2b50 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7f7fe640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a800d3400 0x7f3a800b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7effd640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a80071540 0x7f3a8015c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a7e7fc640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 0x7f3a800b54d0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800c410 tx=0x7f3a6800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a777fe640 1 -- 172.21.15.27:0/4110415129 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a680074e0 con 0x7f3a800b2b50 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a777fe640 1 -- 172.21.15.27:0/4110415129 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a680047e0 con 0x7f3a800b2b50 2024-03-06T07:27:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a777fe640 1 -- 172.21.15.27:0/4110415129 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68004bd0 con 0x7f3a800b2b50 2024-03-06T07:27:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a801f4290 con 0x7f3a800b2b50 2024-03-06T07:27:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.844+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a801f4d70 con 0x7f3a800b2b50 2024-03-06T07:27:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.845+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a800cf6f0 con 0x7f3a800b2b50 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.846+0000 7f3a777fe640 1 -- 172.21.15.27:0/4110415129 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a68011070 con 0x7f3a800b2b50 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.846+0000 7f3a777fe640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6006b4d0 0x7f3a6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.846+0000 7f3a777fe640 1 -- 172.21.15.27:0/4110415129 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3a68089b60 con 0x7f3a800b2b50 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.846+0000 7f3a7effd640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6006b4d0 0x7f3a6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.847+0000 7f3a7effd640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6006b4d0 0x7f3a6006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3a78006fd0 tx=0x7f3a78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:33.851+0000 7f3a777fe640 1 -- 172.21.15.27:0/4110415129 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a800cf6f0 con 0x7f3a800b2b50 2024-03-06T07:27:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.032+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a800b6ac0 con 0x7f3a800b2b50 2024-03-06T07:27:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.033+0000 7f3a777fe640 1 -- 172.21.15.27:0/4110415129 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+12949 (secure 0 0 0) 0x7f3a800b6ac0 con 0x7f3a800b2b50 2024-03-06T07:27:34.035 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:34.036 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:32.711580+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6834/3587849188":"2024-03-07T07:27:29.702926+0000","172.21.15.27:6832/3587849188":"2024-03-07T07:27:29.702926+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6006b4d0 msgr2=0x7f3a6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6006b4d0 0x7f3a6006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3a78006fd0 tx=0x7f3a78008040 comp rx=0 tx=0).stop 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 msgr2=0x7f3a800b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 0x7f3a800b54d0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800c410 tx=0x7f3a6800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 shutdown_connections 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6006b4d0 0x7f3a6006d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a800d3400 0x7f3a800b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a800b2b50 0x7f3a800b54d0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 --2- 172.21.15.27:0/4110415129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a80071540 0x7f3a8015c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 >> 172.21.15.27:0/4110415129 conn(0x7f3a800d3ca0 msgr2=0x7f3a8015e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 shutdown_connections 2024-03-06T07:27:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.035+0000 7f3a857c9640 1 -- 172.21.15.27:0/4110415129 wait complete. 2024-03-06T07:27:34.048 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:34.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.165+0000 7fac6673a640 1 Processor -- start 2024-03-06T07:27:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.165+0000 7fac6673a640 1 -- start start 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.165+0000 7fac6673a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac600b7420 0x7fac600d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.165+0000 7fac6673a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac600d3740 con 0x7fac600ceea0 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac6673a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac600d3120 con 0x7fac6015dd90 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac6673a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac600d3400 con 0x7fac600b7420 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fac600ceea0 0x7fac600b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35686/0 (socket says 172.21.15.27:35686) 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5f7fe640 1 -- 172.21.15.27:0/3335318458 learned_addr learned my addr 172.21.15.27:0/3335318458 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496761320 0 0) 0x7fac600d3740 con 0x7fac600ceea0 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac64cd7640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac600b7420 0x7fac600d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac48003520 con 0x7fac600ceea0 2024-03-06T07:27:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1488500658 0 0) 0x7fac48003520 con 0x7fac600ceea0 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac600b7420 msgr2=0x7fac600d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac600b7420 0x7fac600d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 >> v1:172.21.15.69:6789/0 conn(0x7fac6015dd90 legacy=0x7fac600d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac600d39e0 con 0x7fac600ceea0 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fac54001670 con 0x7fac600ceea0 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 >> v1:172.21.15.27:6789/0 conn(0x7fac600ceea0 legacy=0x7fac600b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac480039e0 0x7fac48003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac48004340 0x7fac48004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48005020 0x7fac4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac600d3400 con 0x7fac48005020 2024-03-06T07:27:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4800bf30 con 0x7fac48004340 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.166+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4800c130 con 0x7fac480039e0 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.167+0000 7fac5f7fe640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac48004340 0x7fac48004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.167+0000 7fac5ffff640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac480039e0 0x7fac48003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.167+0000 7fac64cd7640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48005020 0x7fac4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.167+0000 7fac5f7fe640 1 -- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac480039e0 msgr2=0x7fac48003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.167+0000 7fac5f7fe640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac480039e0 0x7fac48003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.167+0000 7fac5f7fe640 1 -- 172.21.15.27:0/3335318458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48005020 msgr2=0x7fac4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.167+0000 7fac5f7fe640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48005020 0x7fac4800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.167+0000 7fac5f7fe640 1 -- 172.21.15.27:0/3335318458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac48003520 con 0x7fac48004340 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.167+0000 7fac64cd7640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48005020 0x7fac4800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.168+0000 7fac5f7fe640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac48004340 0x7fac48004820 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fac5400c240 tx=0x7fac48005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4a32bb03688b900 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.168+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54007d30 con 0x7fac48004340 2024-03-06T07:27:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.168+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac54030070 con 0x7fac48004340 2024-03-06T07:27:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.168+0000 7fac5effd640 1 -- 172.21.15.27:0/3335318458 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5402cd60 con 0x7fac48004340 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.169+0000 7fac6673a640 1 -- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac48004340 msgr2=0x7fac48004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.169+0000 7fac6673a640 1 --2- 172.21.15.27:0/3335318458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac48004340 0x7fac48004820 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fac5400c240 tx=0x7fac48005830 comp rx=0 tx=0).stop 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.169+0000 7fac64cd7640 1 -- 172.21.15.27:0/3335318458 reap_dead start 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.169+0000 7fac6673a640 1 -- 172.21.15.27:0/3335318458 shutdown_connections 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.169+0000 7fac6673a640 1 -- 172.21.15.27:0/3335318458 >> 172.21.15.27:0/3335318458 conn(0x7fac60071960 msgr2=0x7fac60071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.169+0000 7fac6673a640 1 -- 172.21.15.27:0/3335318458 shutdown_connections 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.169+0000 7fac6673a640 1 -- 172.21.15.27:0/3335318458 wait complete. 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.170+0000 7fac6673a640 1 Processor -- start 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac6673a640 1 -- start start 2024-03-06T07:27:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac6673a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48004340 0x7fac601f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac6673a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac600b8d70 0x7fac600b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac6673a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac6015bb70 0x7fac6015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac6673a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac600d48f0 con 0x7fac48004340 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac6673a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6015e170 con 0x7fac600b8d70 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac6673a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac600d4a70 con 0x7fac6015bb70 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48004340 0x7fac601f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac5ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48004340 0x7fac601f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46082/0 (socket says 172.21.15.27:46082) 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.171+0000 7fac5ffff640 1 -- 172.21.15.27:0/627518106 learned_addr learned my addr 172.21.15.27:0/627518106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5f7fe640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac600b8d70 0x7fac600b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac64cd7640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac6015bb70 0x7fac6015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5ffff640 1 -- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac6015bb70 msgr2=0x7fac6015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5ffff640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac6015bb70 0x7fac6015bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5ffff640 1 -- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac600b8d70 msgr2=0x7fac600b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5ffff640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac600b8d70 0x7fac600b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5ffff640 1 -- 172.21.15.27:0/627518106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac600b98c0 con 0x7fac48004340 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac64cd7640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac6015bb70 0x7fac6015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5ffff640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48004340 0x7fac601f3200 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fac5000f5f0 tx=0x7fac5000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5cff9640 1 -- 172.21.15.27:0/627518106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac500052e0 con 0x7fac48004340 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5cff9640 1 -- 172.21.15.27:0/627518106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac50018070 con 0x7fac48004340 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac60144a10 con 0x7fac48004340 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac5cff9640 1 -- 172.21.15.27:0/627518106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50003810 con 0x7fac48004340 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.172+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac60144fd0 con 0x7fac48004340 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.173+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac60072a30 con 0x7fac48004340 2024-03-06T07:27:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.174+0000 7fac5cff9640 1 -- 172.21.15.27:0/627518106 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac500039b0 con 0x7fac48004340 2024-03-06T07:27:34.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.174+0000 7fac5cff9640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac4c06b4d0 0x7fac4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.174+0000 7fac5cff9640 1 -- 172.21.15.27:0/627518106 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fac50089be0 con 0x7fac48004340 2024-03-06T07:27:34.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.174+0000 7fac5f7fe640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac4c06b4d0 0x7fac4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.175+0000 7fac5f7fe640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac4c06b4d0 0x7fac4c06d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fac54034000 tx=0x7fac54009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.178+0000 7fac5cff9640 1 -- 172.21.15.27:0/627518106 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac50058df0 con 0x7fac48004340 2024-03-06T07:27:34.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.363+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac600b2aa0 con 0x7fac48004340 2024-03-06T07:27:34.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.363+0000 7fac5cff9640 1 -- 172.21.15.27:0/627518106 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac500586f0 con 0x7fac48004340 2024-03-06T07:27:34.366 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:27:34.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac4c06b4d0 msgr2=0x7fac4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac4c06b4d0 0x7fac4c06d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fac54034000 tx=0x7fac54009030 comp rx=0 tx=0).stop 2024-03-06T07:27:34.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48004340 msgr2=0x7fac601f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48004340 0x7fac601f3200 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fac5000f5f0 tx=0x7fac5000fab0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 shutdown_connections 2024-03-06T07:27:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac4c06b4d0 0x7fac4c06d990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac6015bb70 0x7fac6015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac600b8d70 0x7fac600b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 --2- 172.21.15.27:0/627518106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac48004340 0x7fac601f3200 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 >> 172.21.15.27:0/627518106 conn(0x7fac60071960 msgr2=0x7fac600d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 shutdown_connections 2024-03-06T07:27:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.365+0000 7fac6673a640 1 -- 172.21.15.27:0/627518106 wait complete. 2024-03-06T07:27:34.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:27:34.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.484+0000 7f70e6055640 1 Processor -- start 2024-03-06T07:27:34.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.484+0000 7f70e6055640 1 -- start start 2024-03-06T07:27:34.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70e6055640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70e015bf30 0x7f70e00d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70e6055640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e00d0890 con 0x7f70e00d2580 2024-03-06T07:27:34.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70e6055640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e006ec80 con 0x7f70e00cea40 2024-03-06T07:27:34.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70e6055640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e006ef60 con 0x7f70e015bf30 2024-03-06T07:27:34.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70deffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f70e00d2580 0x7f70e00ce330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35700/0 (socket says 172.21.15.27:35700) 2024-03-06T07:27:34.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70deffd640 1 -- 172.21.15.27:0/3953346689 learned_addr learned my addr 172.21.15.27:0/3953346689 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70df7fe640 1 --2- 172.21.15.27:0/3953346689 >> v2:172.21.15.69:3300/0 conn(0x7f70e015bf30 0x7f70e00d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305088698 0 0) 0x7f70e00d0890 con 0x7f70e00d2580 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c8003520 con 0x7f70e00d2580 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 467406513 0 0) 0x7f70c8003520 con 0x7f70e00d2580 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 >> v2:172.21.15.69:3300/0 conn(0x7f70e015bf30 msgr2=0x7f70e00d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 --2- 172.21.15.27:0/3953346689 >> v2:172.21.15.69:3300/0 conn(0x7f70e015bf30 0x7f70e00d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 >> v1:172.21.15.69:6789/0 conn(0x7f70e00cea40 legacy=0x7f70e006f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e006f240 con 0x7f70e00d2580 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f70d0001670 con 0x7f70e00d2580 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70df7fe640 1 --2- 172.21.15.27:0/3953346689 >> v2:172.21.15.69:3300/0 conn(0x7f70e015bf30 0x7f70e00d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 >> v1:172.21.15.27:6789/0 conn(0x7f70e00d2580 legacy=0x7f70e00ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c80039b0 0x7f70c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.485+0000 7f70de7fc640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8004800 0x7f70c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70de7fc640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e006ef60 con 0x7f70c80051c0 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c800c160 con 0x7f70c8004800 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c800c360 con 0x7f70c80039b0 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70dffff640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70deffd640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8004800 0x7f70c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70df7fe640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c80039b0 0x7f70c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70dffff640 1 -- 172.21.15.27:0/3953346689 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c80039b0 msgr2=0x7f70c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70dffff640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70c80039b0 0x7f70c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70dffff640 1 -- 172.21.15.27:0/3953346689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8004800 msgr2=0x7f70c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70dffff640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70c8004800 0x7f70c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.486+0000 7f70dffff640 1 -- 172.21.15.27:0/3953346689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8003520 con 0x7f70c80051c0 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70dffff640 1 -- 172.21.15.27:0/3953346689 reap_dead start 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70dffff640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70c800b9c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f70d8001e50 tx=0x7f70c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a2dfe6fba9ab9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d8001a70 con 0x7f70c80051c0 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d8003bb0 con 0x7f70c80051c0 2024-03-06T07:27:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70de7fc640 1 -- 172.21.15.27:0/3953346689 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d80050d0 con 0x7f70c80051c0 2024-03-06T07:27:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70e6055640 1 -- 172.21.15.27:0/3953346689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 msgr2=0x7f70c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70e6055640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70c800b9c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f70d8001e50 tx=0x7f70c8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70e6055640 1 -- 172.21.15.27:0/3953346689 shutdown_connections 2024-03-06T07:27:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70e6055640 1 --2- 172.21.15.27:0/3953346689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70c800b9c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.487+0000 7f70e6055640 1 -- 172.21.15.27:0/3953346689 >> 172.21.15.27:0/3953346689 conn(0x7f70e00d1a60 msgr2=0x7f70e00d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.488+0000 7f70e6055640 1 -- 172.21.15.27:0/3953346689 shutdown_connections 2024-03-06T07:27:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.488+0000 7f70e6055640 1 -- 172.21.15.27:0/3953346689 wait complete. 2024-03-06T07:27:34.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.489+0000 7f70e6055640 1 Processor -- start 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.489+0000 7f70e6055640 1 -- start start 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70e6055640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70e015c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70e6055640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e015ca60 0x7f70e006b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70e6055640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e006b700 0x7f70e015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70e6055640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00ce6c0 con 0x7f70c80051c0 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70e6055640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00ce540 con 0x7f70e006b700 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70e6055640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00ce840 con 0x7f70e015ca60 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70e015c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70e015c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46096/0 (socket says 172.21.15.27:46096) 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70df7fe640 1 -- 172.21.15.27:0/3485900596 learned_addr learned my addr 172.21.15.27:0/3485900596 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70df7fe640 1 -- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e015ca60 msgr2=0x7f70e006b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70deffd640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e015ca60 0x7f70e006b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70dffff640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e006b700 0x7f70e015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70df7fe640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e015ca60 0x7f70e006b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70df7fe640 1 -- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e006b700 msgr2=0x7f70e015cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70df7fe640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e006b700 0x7f70e015cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70df7fe640 1 -- 172.21.15.27:0/3485900596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e006bd20 con 0x7f70c80051c0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70dffff640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e006b700 0x7f70e015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70df7fe640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70e015c520 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f70d4011a90 tx=0x7f70d4011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70c7fff640 1 -- 172.21.15.27:0/3485900596 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d400fc10 con 0x7f70c80051c0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70c7fff640 1 -- 172.21.15.27:0/3485900596 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d400fdb0 con 0x7f70c80051c0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e006c000 con 0x7f70c80051c0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70c7fff640 1 -- 172.21.15.27:0/3485900596 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4004260 con 0x7f70c80051c0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.490+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e006c5c0 con 0x7f70c80051c0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.491+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e00b4650 con 0x7f70c80051c0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.492+0000 7f70c7fff640 1 -- 172.21.15.27:0/3485900596 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70d4012040 con 0x7f70c80051c0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.492+0000 7f70c7fff640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f70cc06b500 0x7f70cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.492+0000 7f70c7fff640 1 -- 172.21.15.27:0/3485900596 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f70d408a320 con 0x7f70c80051c0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.492+0000 7f70deffd640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f70cc06b500 0x7f70cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.493+0000 7f70deffd640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f70cc06b500 0x7f70cc06d9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f70e015ce40 tx=0x7f70d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.497+0000 7f70c7fff640 1 -- 172.21.15.27:0/3485900596 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70e00b4650 con 0x7f70c80051c0 2024-03-06T07:27:34.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:34.749+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f70e0067ff0 con 0x7f70c80051c0 2024-03-06T07:27:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.722+0000 7f70c7fff640 1 -- 172.21.15.27:0/3485900596 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v278) v1 ==== 137+0+0 (secure 0 0 0) 0x7f70d4019030 con 0x7f70c80051c0 2024-03-06T07:27:35.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.724+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f70cc06b500 msgr2=0x7f70cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.724+0000 7f70e6055640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f70cc06b500 0x7f70cc06d9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f70e015ce40 tx=0x7f70d0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:35.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 msgr2=0x7f70e015c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70e015c520 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f70d4011a90 tx=0x7f70d4011f50 comp rx=0 tx=0).stop 2024-03-06T07:27:35.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 shutdown_connections 2024-03-06T07:27:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f70cc06b500 0x7f70cc06d9c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e006b700 0x7f70e015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70e015ca60 0x7f70e006b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 --2- 172.21.15.27:0/3485900596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70c80051c0 0x7f70e015c520 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 >> 172.21.15.27:0/3485900596 conn(0x7f70e00d1a60 msgr2=0x7f70e00cd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 shutdown_connections 2024-03-06T07:27:35.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.725+0000 7f70e6055640 1 -- 172.21.15.27:0/3485900596 wait complete. 2024-03-06T07:27:35.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:27:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.855+0000 7f420839a640 1 Processor -- start 2024-03-06T07:27:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.855+0000 7f420839a640 1 -- start start 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f420839a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4200071540 0x7f4200071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f420839a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f42000b2d30 0x7f42000ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f420839a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200071e60 con 0x7f42000cecb0 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f420839a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42000d3230 con 0x7f4200071540 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f420839a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42000d4960 con 0x7f42000b2d30 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f4206136640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4200071540 0x7f4200071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f4206937640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f42000cecb0 0x7f42000b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35716/0 (socket says 172.21.15.27:35716) 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f4206937640 1 -- 172.21.15.27:0/3337648918 learned_addr learned my addr 172.21.15.27:0/3337648918 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f4205935640 1 --2- 172.21.15.27:0/3337648918 >> v2:172.21.15.69:3300/0 conn(0x7f42000b2d30 0x7f42000ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f4206136640 1 -- 172.21.15.27:0/3337648918 >> v2:172.21.15.69:3300/0 conn(0x7f42000b2d30 msgr2=0x7f42000ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f4206136640 1 --2- 172.21.15.27:0/3337648918 >> v2:172.21.15.69:3300/0 conn(0x7f42000b2d30 0x7f42000ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f4206136640 1 -- 172.21.15.27:0/3337648918 >> v1:172.21.15.27:6789/0 conn(0x7f42000cecb0 legacy=0x7f42000b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.857+0000 7f4206136640 1 -- 172.21.15.27:0/3337648918 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000d36c0 con 0x7f4200071540 2024-03-06T07:27:35.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.856+0000 7f4205134640 1 -- 172.21.15.27:0/3337648918 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265894093 0 0) 0x7f4200071e60 con 0x7f42000cecb0 2024-03-06T07:27:35.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.857+0000 7f4205935640 1 --2- 172.21.15.27:0/3337648918 >> v2:172.21.15.69:3300/0 conn(0x7f42000b2d30 0x7f42000ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.857+0000 7f4206136640 1 --2- 172.21.15.27:0/3337648918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4200071540 0x7f4200071920 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f42000c7690 tx=0x7f41f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3ad47dafbf4993 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.857+0000 7f4205134640 1 -- 172.21.15.27:0/3337648918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8039070 con 0x7f4200071540 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.857+0000 7f4205134640 1 -- 172.21.15.27:0/3337648918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f802cc80 con 0x7f4200071540 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.857+0000 7f4205134640 1 -- 172.21.15.27:0/3337648918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8034660 con 0x7f4200071540 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.858+0000 7f420839a640 1 -- 172.21.15.27:0/3337648918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4200071540 msgr2=0x7f4200071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.858+0000 7f420839a640 1 --2- 172.21.15.27:0/3337648918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4200071540 0x7f4200071920 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f42000c7690 tx=0x7f41f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.858+0000 7f420839a640 1 -- 172.21.15.27:0/3337648918 shutdown_connections 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.858+0000 7f420839a640 1 --2- 172.21.15.27:0/3337648918 >> v2:172.21.15.69:3300/0 conn(0x7f42000b2d30 0x7f42000ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.858+0000 7f420839a640 1 --2- 172.21.15.27:0/3337648918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4200071540 0x7f4200071920 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.858+0000 7f420839a640 1 -- 172.21.15.27:0/3337648918 >> 172.21.15.27:0/3337648918 conn(0x7f42000d3d60 msgr2=0x7f42000d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.858+0000 7f420839a640 1 -- 172.21.15.27:0/3337648918 shutdown_connections 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.858+0000 7f420839a640 1 -- 172.21.15.27:0/3337648918 wait complete. 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.859+0000 7f420839a640 1 Processor -- start 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.859+0000 7f420839a640 1 -- start start 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f420839a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4200071540 0x7f42001f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f420839a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42000b2d30 0x7f42001f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f420839a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42000cecb0 0x7f42001e6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f420839a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42000d2be0 con 0x7f42000cecb0 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f420839a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42000d2900 con 0x7f42000b2d30 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f420839a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42000d2ec0 con 0x7f4200071540 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f4206937640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42000cecb0 0x7f42001e6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f4206937640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42000cecb0 0x7f42001e6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46104/0 (socket says 172.21.15.27:46104) 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f4206937640 1 -- 172.21.15.27:0/3228185069 learned_addr learned my addr 172.21.15.27:0/3228185069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f4205935640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42000b2d30 0x7f42001f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f4206136640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4200071540 0x7f42001f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f4206937640 1 -- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4200071540 msgr2=0x7f42001f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f4206937640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4200071540 0x7f42001f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f4206937640 1 -- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42000b2d30 msgr2=0x7f42001f0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.860+0000 7f4206937640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42000b2d30 0x7f42001f0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.861+0000 7f4206937640 1 -- 172.21.15.27:0/3228185069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000c8bc0 con 0x7f42000cecb0 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.861+0000 7f4205935640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42000b2d30 0x7f42001f0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.861+0000 7f4206937640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42000cecb0 0x7f42001e6990 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f41f000c450 tx=0x7f41f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.861+0000 7f41feffd640 1 -- 172.21.15.27:0/3228185069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0018070 con 0x7f42000cecb0 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.861+0000 7f41feffd640 1 -- 172.21.15.27:0/3228185069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f0007500 con 0x7f42000cecb0 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.861+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42000c8ea0 con 0x7f42000cecb0 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.861+0000 7f41feffd640 1 -- 172.21.15.27:0/3228185069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0004a20 con 0x7f42000cecb0 2024-03-06T07:27:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.861+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42001f5320 con 0x7f42000cecb0 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.862+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4200072070 con 0x7f42000cecb0 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.862+0000 7f41feffd640 1 -- 172.21.15.27:0/3228185069 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41f001b070 con 0x7f42000cecb0 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.863+0000 7f41feffd640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41ec06b500 0x7f41ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.863+0000 7f41feffd640 1 -- 172.21.15.27:0/3228185069 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f41f0089230 con 0x7f42000cecb0 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.863+0000 7f4206136640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41ec06b500 0x7f41ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.864+0000 7f4206136640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41ec06b500 0x7f41ec06d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f41f8036000 tx=0x7f41f8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:35.867+0000 7f41feffd640 1 -- 172.21.15.27:0/3228185069 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41f0058310 con 0x7f42000cecb0 2024-03-06T07:27:36.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.052+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f42000ce9b0 con 0x7f42000cecb0 2024-03-06T07:27:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.055+0000 7f41feffd640 1 -- 172.21.15.27:0/3228185069 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f41f0057c10 con 0x7f42000cecb0 2024-03-06T07:27:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41ec06b500 msgr2=0x7f41ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41ec06b500 0x7f41ec06d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f41f8036000 tx=0x7f41f8030000 comp rx=0 tx=0).stop 2024-03-06T07:27:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42000cecb0 msgr2=0x7f42001e6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42000cecb0 0x7f42001e6990 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f41f000c450 tx=0x7f41f000c910 comp rx=0 tx=0).stop 2024-03-06T07:27:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 shutdown_connections 2024-03-06T07:27:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41ec06b500 0x7f41ec06d9c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42000cecb0 0x7f42001e6990 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42000b2d30 0x7f42001f0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 --2- 172.21.15.27:0/3228185069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4200071540 0x7f42001f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 >> 172.21.15.27:0/3228185069 conn(0x7f42000d3d60 msgr2=0x7f420015c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 shutdown_connections 2024-03-06T07:27:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.057+0000 7f420839a640 1 -- 172.21.15.27:0/3228185069 wait complete. 2024-03-06T07:27:36.071 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:36.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:36.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:36.164 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:36.164 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:27:36.164 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:36.180 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:36.180 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:27:36.180 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:36.221 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:36.221 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:27:36.222 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:36.237 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:36.237 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:27:36.237 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:36.278 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:36.278 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:27:36.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:36.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.383+0000 7fa579f56640 1 Processor -- start 2024-03-06T07:27:36.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.384+0000 7fa579f56640 1 -- start start 2024-03-06T07:27:36.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.384+0000 7fa579f56640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa57415e060 0x7fa5740ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.384+0000 7fa579f56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa57406fe20 0x7fa5740ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.384+0000 7fa579f56640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5740d0dd0 con 0x7fa5740d0880 2024-03-06T07:27:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.384+0000 7fa579f56640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5740d10b0 con 0x7fa57415e060 2024-03-06T07:27:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.384+0000 7fa579f56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740cc0e0 con 0x7fa57406fe20 2024-03-06T07:27:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.384+0000 7fa573fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa5740d0880 0x7fa57406ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35722/0 (socket says 172.21.15.27:35722) 2024-03-06T07:27:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.384+0000 7fa573fff640 1 -- 172.21.15.27:0/164586624 learned_addr learned my addr 172.21.15.27:0/164586624 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417494502 0 0) 0x7fa5740d0dd0 con 0x7fa5740d0880 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5737fe640 1 --2- 172.21.15.27:0/164586624 >> v2:172.21.15.69:3300/0 conn(0x7fa57415e060 0x7fa5740ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa55c003520 con 0x7fa5740d0880 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa572ffd640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa57406fe20 0x7fa5740ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2798584257 0 0) 0x7fa55c003520 con 0x7fa5740d0880 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa57406fe20 msgr2=0x7fa5740ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa57406fe20 0x7fa5740ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 >> v2:172.21.15.69:3300/0 conn(0x7fa57415e060 msgr2=0x7fa5740ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 --2- 172.21.15.27:0/164586624 >> v2:172.21.15.69:3300/0 conn(0x7fa57415e060 0x7fa5740ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5740cc3c0 con 0x7fa5740d0880 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fa56c0019f0 con 0x7fa5740d0880 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 >> v1:172.21.15.27:6789/0 conn(0x7fa5740d0880 legacy=0x7fa57406ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa57406fe20 0x7fa55c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa55c003f80 0x7fa55c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa55c004d40 0x7fa55c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740d10b0 con 0x7fa55c003f80 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740cc0e0 con 0x7fa55c004d40 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c00b9c0 con 0x7fa57406fe20 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa572ffd640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa55c003f80 0x7fa55c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa5737fe640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa57406fe20 0x7fa55c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.385+0000 7fa573fff640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa55c004d40 0x7fa55c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.386+0000 7fa572ffd640 1 -- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa57406fe20 msgr2=0x7fa55c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.386+0000 7fa572ffd640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa57406fe20 0x7fa55c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.386+0000 7fa572ffd640 1 -- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa55c004d40 msgr2=0x7fa55c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.386+0000 7fa572ffd640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa55c004d40 0x7fa55c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.386+0000 7fa572ffd640 1 -- 172.21.15.27:0/164586624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c003520 con 0x7fa55c003f80 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.386+0000 7fa572ffd640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa55c003f80 0x7fa55c004800 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa568009ac0 tx=0x7fa55c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fec75b09abf40c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.386+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568039070 con 0x7fa55c003f80 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.386+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56802c680 con 0x7fa55c003f80 2024-03-06T07:27:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.386+0000 7fa5727fc640 1 -- 172.21.15.27:0/164586624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568009690 con 0x7fa55c003f80 2024-03-06T07:27:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.387+0000 7fa579f56640 1 -- 172.21.15.27:0/164586624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa55c003f80 msgr2=0x7fa55c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.387+0000 7fa579f56640 1 --2- 172.21.15.27:0/164586624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa55c003f80 0x7fa55c004800 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa568009ac0 tx=0x7fa55c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.387+0000 7fa573fff640 1 -- 172.21.15.27:0/164586624 reap_dead start 2024-03-06T07:27:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.387+0000 7fa579f56640 1 -- 172.21.15.27:0/164586624 shutdown_connections 2024-03-06T07:27:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.387+0000 7fa579f56640 1 -- 172.21.15.27:0/164586624 >> 172.21.15.27:0/164586624 conn(0x7fa5740d1a90 msgr2=0x7fa5740d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.387+0000 7fa579f56640 1 -- 172.21.15.27:0/164586624 shutdown_connections 2024-03-06T07:27:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.387+0000 7fa579f56640 1 -- 172.21.15.27:0/164586624 wait complete. 2024-03-06T07:27:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.388+0000 7fa579f56640 1 Processor -- start 2024-03-06T07:27:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.388+0000 7fa579f56640 1 -- start start 2024-03-06T07:27:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa579f56640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa55c003f80 0x7fa57414df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa579f56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740b48e0 0x7fa5740cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa579f56640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740c94d0 0x7fa5740c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa579f56640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740cdbd0 con 0x7fa5740c94d0 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa579f56640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57415e440 con 0x7fa55c003f80 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa579f56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740cdd50 con 0x7fa5740b48e0 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa573fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740c94d0 0x7fa5740c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa573fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740c94d0 0x7fa5740c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46126/0 (socket says 172.21.15.27:46126) 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa573fff640 1 -- 172.21.15.27:0/1126857911 learned_addr learned my addr 172.21.15.27:0/1126857911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa5737fe640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa55c003f80 0x7fa57414df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.389+0000 7fa572ffd640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740b48e0 0x7fa5740cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa573fff640 1 -- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740b48e0 msgr2=0x7fa5740cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa573fff640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740b48e0 0x7fa5740cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa573fff640 1 -- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa55c003f80 msgr2=0x7fa57414df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa573fff640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa55c003f80 0x7fa57414df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa573fff640 1 -- 172.21.15.27:0/1126857911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57415f760 con 0x7fa5740c94d0 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa573fff640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740c94d0 0x7fa5740c9950 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0013d0 tx=0x7fa56c007a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa55bfff640 1 -- 172.21.15.27:0/1126857911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c00aad0 con 0x7fa5740c94d0 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa55bfff640 1 -- 172.21.15.27:0/1126857911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c019070 con 0x7fa5740c94d0 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57415fa40 con 0x7fa5740c94d0 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa55bfff640 1 -- 172.21.15.27:0/1126857911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0133f0 con 0x7fa5740c94d0 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.390+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa574160000 con 0x7fa5740c94d0 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.391+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa574067ff0 con 0x7fa5740c94d0 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.391+0000 7fa55bfff640 1 -- 172.21.15.27:0/1126857911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa56c013590 con 0x7fa5740c94d0 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.392+0000 7fa55bfff640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa56006b500 0x7fa56006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.392+0000 7fa55bfff640 1 -- 172.21.15.27:0/1126857911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa56c08a3a0 con 0x7fa5740c94d0 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.392+0000 7fa5737fe640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa56006b500 0x7fa56006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.393+0000 7fa5737fe640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa56006b500 0x7fa56006d9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa564009c40 tx=0x7fa564009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.396+0000 7fa55bfff640 1 -- 172.21.15.27:0/1126857911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa56c059490 con 0x7fa5740c94d0 2024-03-06T07:27:36.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.616+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5740cbe50 con 0x7fa5740c94d0 2024-03-06T07:27:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.616+0000 7fa55bfff640 1 -- 172.21.15.27:0/1126857911 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fa56c058d90 con 0x7fa5740c94d0 2024-03-06T07:27:36.619 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:36.619 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":54,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/930650799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":930650799},{"type":"v1","addr":"172.21.15.69:6836","nonce":930650799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1050755919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1050755919},{"type":"v1","addr":"172.21.15.27:6835","nonce":1050755919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1173605349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1173605349},{"type":"v1","addr":"172.21.15.69:6837","nonce":1173605349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1392345366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1392345366},{"type":"v1","addr":"172.21.15.27:6834","nonce":1392345366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[]} 2024-03-06T07:27:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 54 2024-03-06T07:27:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa56006b500 msgr2=0x7fa56006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa56006b500 0x7fa56006d9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa564009c40 tx=0x7fa564009190 comp rx=0 tx=0).stop 2024-03-06T07:27:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740c94d0 msgr2=0x7fa5740c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740c94d0 0x7fa5740c9950 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0013d0 tx=0x7fa56c007a60 comp rx=0 tx=0).stop 2024-03-06T07:27:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 shutdown_connections 2024-03-06T07:27:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa56006b500 0x7fa56006d9c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740c94d0 0x7fa5740c9950 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa5740b48e0 0x7fa5740cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 --2- 172.21.15.27:0/1126857911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa55c003f80 0x7fa57414df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 >> 172.21.15.27:0/1126857911 conn(0x7fa5740d1a90 msgr2=0x7fa5740b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 shutdown_connections 2024-03-06T07:27:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:36.619+0000 7fa579f56640 1 -- 172.21.15.27:0/1126857911 wait complete. 2024-03-06T07:27:36.632 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:36.633 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:27:36.633 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:36.633 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:36.633 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:36.633 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:27:36.633 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:36.633 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:36.633 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:36.634 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:27:36.634 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:36.634 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:36.634 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:36.634 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:27:36.634 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:36.634 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:36.642 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:36.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:36.643 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:36.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:36.646 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:36.646 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:36.684 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:36.685 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:36.685 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:36.685 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:36.689 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:36.689 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:36.766 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:27:36.766 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:27:36.772 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:27:36.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-06T07:27:42.646 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:42.646 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:42.649 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:42.650 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:42.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:27:42.733 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:27:42.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.769+0000 7f7f10d5e640 1 Processor -- start 2024-03-06T07:27:42.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.769+0000 7f7f10d5e640 1 -- start start 2024-03-06T07:27:42.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f10d5e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f0c158630 0x7f7f0c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f10d5e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f0c15bd70 0x7f7f0c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f10d5e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c0b2290 0x7f7f0c15e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f10d5e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0c0b26b0 con 0x7f7f0c158630 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f10d5e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0c0b3600 con 0x7f7f0c15bd70 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f10d5e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0b38b0 con 0x7f7f0c0b2290 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f0a59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f0c158630 0x7f7f0c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f0a59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f0c158630 0x7f7f0c0b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46154/0 (socket says 172.21.15.27:46154) 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f0a59c640 1 -- 172.21.15.27:0/2940930992 learned_addr learned my addr 172.21.15.27:0/2940930992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f0ad9d640 1 --2- 172.21.15.27:0/2940930992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c0b2290 0x7f7f0c15e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f09d9b640 1 --2- 172.21.15.27:0/2940930992 >> v2:172.21.15.69:3300/0 conn(0x7f7f0c15bd70 0x7f7f0c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f0a59c640 1 -- 172.21.15.27:0/2940930992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c0b2290 msgr2=0x7f7f0c15e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f0a59c640 1 --2- 172.21.15.27:0/2940930992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c0b2290 0x7f7f0c15e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f0a59c640 1 -- 172.21.15.27:0/2940930992 >> v2:172.21.15.69:3300/0 conn(0x7f7f0c15bd70 msgr2=0x7f7f0c0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.770+0000 7f7f0a59c640 1 --2- 172.21.15.27:0/2940930992 >> v2:172.21.15.69:3300/0 conn(0x7f7f0c15bd70 0x7f7f0c0b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.771+0000 7f7f0a59c640 1 -- 172.21.15.27:0/2940930992 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c0b3b70 con 0x7f7f0c158630 2024-03-06T07:27:42.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.771+0000 7f7f0a59c640 1 --2- 172.21.15.27:0/2940930992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c158630 0x7f7f0c0b30c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0b2bd0 tx=0x7f7f0402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f861009f6f68559 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.771+0000 7f7f0959a640 1 -- 172.21.15.27:0/2940930992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0403b070 con 0x7f7f0c158630 2024-03-06T07:27:42.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.771+0000 7f7f0959a640 1 -- 172.21.15.27:0/2940930992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0402ee00 con 0x7f7f0c158630 2024-03-06T07:27:42.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f0959a640 1 -- 172.21.15.27:0/2940930992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04036660 con 0x7f7f0c158630 2024-03-06T07:27:42.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f10d5e640 1 -- 172.21.15.27:0/2940930992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c158630 msgr2=0x7f7f0c0b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/2940930992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c158630 0x7f7f0c0b30c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0b2bd0 tx=0x7f7f0402e050 comp rx=0 tx=0).stop 2024-03-06T07:27:42.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f10d5e640 1 -- 172.21.15.27:0/2940930992 shutdown_connections 2024-03-06T07:27:42.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/2940930992 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c0b2290 0x7f7f0c15e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/2940930992 >> v2:172.21.15.69:3300/0 conn(0x7f7f0c15bd70 0x7f7f0c0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/2940930992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c158630 0x7f7f0c0b30c0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f10d5e640 1 -- 172.21.15.27:0/2940930992 >> 172.21.15.27:0/2940930992 conn(0x7f7f0c0b55e0 msgr2=0x7f7f0c0b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f10d5e640 1 -- 172.21.15.27:0/2940930992 shutdown_connections 2024-03-06T07:27:42.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.772+0000 7f7f10d5e640 1 -- 172.21.15.27:0/2940930992 wait complete. 2024-03-06T07:27:42.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.774+0000 7f7f10d5e640 1 Processor -- start 2024-03-06T07:27:42.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.774+0000 7f7f10d5e640 1 -- start start 2024-03-06T07:27:42.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.774+0000 7f7f10d5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c0b2290 0x7f7f0c0c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.774+0000 7f7f10d5e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c158630 0x7f7f0c06dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.774+0000 7f7f10d5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c15bd70 0x7f7f0c14b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.774+0000 7f7f10d5e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0b4410 con 0x7f7f0c15bd70 2024-03-06T07:27:42.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.774+0000 7f7f10d5e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0b4130 con 0x7f7f0c0b2290 2024-03-06T07:27:42.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.774+0000 7f7f10d5e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c158a10 con 0x7f7f0c158630 2024-03-06T07:27:42.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c15bd70 0x7f7f0c14b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c15bd70 0x7f7f0c14b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46158/0 (socket says 172.21.15.27:46158) 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0ad9d640 1 -- 172.21.15.27:0/922062113 learned_addr learned my addr 172.21.15.27:0/922062113 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0a59c640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c0b2290 0x7f7f0c0c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f09d9b640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c158630 0x7f7f0c06dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0ad9d640 1 -- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c158630 msgr2=0x7f7f0c06dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0ad9d640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c158630 0x7f7f0c06dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0ad9d640 1 -- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c0b2290 msgr2=0x7f7f0c0c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0ad9d640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c0b2290 0x7f7f0c0c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0ad9d640 1 -- 172.21.15.27:0/922062113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c0c9b20 con 0x7f7f0c15bd70 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f0ad9d640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c15bd70 0x7f7f0c14b7f0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00eed0 tx=0x7f7efc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f02ffd640 1 -- 172.21.15.27:0/922062113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc018020 con 0x7f7f0c15bd70 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f02ffd640 1 -- 172.21.15.27:0/922062113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc014400 con 0x7f7f0c15bd70 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f02ffd640 1 -- 172.21.15.27:0/922062113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc0147f0 con 0x7f7f0c15bd70 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0c0c9e00 con 0x7f7f0c15bd70 2024-03-06T07:27:42.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.775+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0c0ca3c0 con 0x7f7f0c15bd70 2024-03-06T07:27:42.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.776+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0c067ff0 con 0x7f7f0c15bd70 2024-03-06T07:27:42.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.777+0000 7f7f02ffd640 1 -- 172.21.15.27:0/922062113 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7efc005d10 con 0x7f7f0c15bd70 2024-03-06T07:27:42.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.777+0000 7f7f02ffd640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7eec06b500 0x7f7eec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.777+0000 7f7f02ffd640 1 -- 172.21.15.27:0/922062113 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7efc08a480 con 0x7f7f0c15bd70 2024-03-06T07:27:42.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.777+0000 7f7f0a59c640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7eec06b500 0x7f7eec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.778+0000 7f7f0a59c640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7eec06b500 0x7f7eec06d9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7f04038000 tx=0x7f7f04032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.782+0000 7f7f02ffd640 1 -- 172.21.15.27:0/922062113 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7efc022070 con 0x7f7f0c15bd70 2024-03-06T07:27:42.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:42.974+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7f0c1595b0 con 0x7f7f0c15bd70 2024-03-06T07:27:43.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-06T07:27:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.746+0000 7f7f02ffd640 1 -- 172.21.15.27:0/922062113 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v94) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7efc058fb0 con 0x7f7f0c15bd70 2024-03-06T07:27:44.748 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-03-06T07:27:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7eec06b500 msgr2=0x7f7eec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7eec06b500 0x7f7eec06d9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7f04038000 tx=0x7f7f04032000 comp rx=0 tx=0).stop 2024-03-06T07:27:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c15bd70 msgr2=0x7f7f0c14b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c15bd70 0x7f7f0c14b7f0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00eed0 tx=0x7f7efc00b480 comp rx=0 tx=0).stop 2024-03-06T07:27:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 shutdown_connections 2024-03-06T07:27:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7eec06b500 0x7f7eec06d9c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f0c15bd70 0x7f7f0c14b7f0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f0c158630 0x7f7f0c06dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 --2- 172.21.15.27:0/922062113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c0b2290 0x7f7f0c0c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 >> 172.21.15.27:0/922062113 conn(0x7f7f0c0b55e0 msgr2=0x7f7f0c1578f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 shutdown_connections 2024-03-06T07:27:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.749+0000 7f7f10d5e640 1 -- 172.21.15.27:0/922062113 wait complete. 2024-03-06T07:27:44.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:27:44.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.878+0000 7fbb06783640 1 Processor -- start 2024-03-06T07:27:44.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.878+0000 7fbb06783640 1 -- start start 2024-03-06T07:27:44.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.878+0000 7fbb06783640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb0015df00 0x7fbb000d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.878+0000 7fbb06783640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbb000b95a0 0x7fbb000b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.878+0000 7fbb06783640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb000b9a30 0x7fbb000b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.878+0000 7fbb06783640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb000cea30 con 0x7fbb0015df00 2024-03-06T07:27:44.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbb06783640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb000d2af0 con 0x7fbb000b95a0 2024-03-06T07:27:44.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbb06783640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000cf3c0 con 0x7fbb000b9a30 2024-03-06T07:27:44.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbafffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb0015df00 0x7fbb000d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbafffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb0015df00 0x7fbb000d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52516/0 (socket says 172.21.15.27:52516) 2024-03-06T07:27:44.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbafffff640 1 -- 172.21.15.27:0/4254892778 learned_addr learned my addr 172.21.15.27:0/4254892778 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:44.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbafffff640 1 -- 172.21.15.27:0/4254892778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb000b9a30 msgr2=0x7fbb000b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbafffff640 1 --2- 172.21.15.27:0/4254892778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb000b9a30 0x7fbb000b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbafffff640 1 -- 172.21.15.27:0/4254892778 >> v2:172.21.15.69:3300/0 conn(0x7fbb000b95a0 msgr2=0x7fbb000b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbafffff640 1 --2- 172.21.15.27:0/4254892778 >> v2:172.21.15.69:3300/0 conn(0x7fbb000b95a0 0x7fbb000b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.879+0000 7fbafffff640 1 -- 172.21.15.27:0/4254892778 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb000cf0e0 con 0x7fbb0015df00 2024-03-06T07:27:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.880+0000 7fbafffff640 1 --2- 172.21.15.27:0/4254892778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb0015df00 0x7fbb000d25b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fbb000d1760 tx=0x7fbaf002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8237185cd566b159 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.880+0000 7fbafeffd640 1 -- 172.21.15.27:0/4254892778 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0035020 con 0x7fbb0015df00 2024-03-06T07:27:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.880+0000 7fbafeffd640 1 -- 172.21.15.27:0/4254892778 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf0031870 con 0x7fbb0015df00 2024-03-06T07:27:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.880+0000 7fbafeffd640 1 -- 172.21.15.27:0/4254892778 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0031c40 con 0x7fbb0015df00 2024-03-06T07:27:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.880+0000 7fbb06783640 1 -- 172.21.15.27:0/4254892778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb0015df00 msgr2=0x7fbb000d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.880+0000 7fbb06783640 1 --2- 172.21.15.27:0/4254892778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb0015df00 0x7fbb000d25b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fbb000d1760 tx=0x7fbaf002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.880+0000 7fbb06783640 1 -- 172.21.15.27:0/4254892778 shutdown_connections 2024-03-06T07:27:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.880+0000 7fbb06783640 1 --2- 172.21.15.27:0/4254892778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb000b9a30 0x7fbb000b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.881+0000 7fbb06783640 1 --2- 172.21.15.27:0/4254892778 >> v2:172.21.15.69:3300/0 conn(0x7fbb000b95a0 0x7fbb000b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.881+0000 7fbb06783640 1 --2- 172.21.15.27:0/4254892778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb0015df00 0x7fbb000d25b0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.881+0000 7fbb06783640 1 -- 172.21.15.27:0/4254892778 >> 172.21.15.27:0/4254892778 conn(0x7fbb00071960 msgr2=0x7fbb00071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.881+0000 7fbb06783640 1 -- 172.21.15.27:0/4254892778 shutdown_connections 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.881+0000 7fbb06783640 1 -- 172.21.15.27:0/4254892778 wait complete. 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.882+0000 7fbb06783640 1 Processor -- start 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.882+0000 7fbb06783640 1 -- start start 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb06783640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b95a0 0x7fbb0015b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb06783640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb0015df00 0x7fbb0015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb06783640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb001f3df0 0x7fbb001f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb06783640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000cec70 con 0x7fbb001f3df0 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb06783640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000ce750 con 0x7fbb000b95a0 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb06783640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000cedf0 con 0x7fbb0015df00 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb04d20640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb001f3df0 0x7fbb001f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb04d20640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb001f3df0 0x7fbb001f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52528/0 (socket says 172.21.15.27:52528) 2024-03-06T07:27:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb04d20640 1 -- 172.21.15.27:0/818550782 learned_addr learned my addr 172.21.15.27:0/818550782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbafffff640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b95a0 0x7fbb0015b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbaff7fe640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb0015df00 0x7fbb0015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb04d20640 1 -- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb0015df00 msgr2=0x7fbb0015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb04d20640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb0015df00 0x7fbb0015ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb04d20640 1 -- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b95a0 msgr2=0x7fbb0015b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb04d20640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b95a0 0x7fbb0015b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb04d20640 1 -- 172.21.15.27:0/818550782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0015f230 con 0x7fbb001f3df0 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbafffff640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b95a0 0x7fbb0015b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb04d20640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb001f3df0 0x7fbb001f42a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800b970 tx=0x7fbaf800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbafcff9640 1 -- 172.21.15.27:0/818550782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8019070 con 0x7fbb001f3df0 2024-03-06T07:27:44.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbafcff9640 1 -- 172.21.15.27:0/818550782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf800cb00 con 0x7fbb001f3df0 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb000c7ae0 con 0x7fbb001f3df0 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.883+0000 7fbafcff9640 1 -- 172.21.15.27:0/818550782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8014440 con 0x7fbb001f3df0 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.884+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb000c80a0 con 0x7fbb001f3df0 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.884+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb00067ff0 con 0x7fbb001f3df0 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.885+0000 7fbafcff9640 1 -- 172.21.15.27:0/818550782 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaf801c070 con 0x7fbb001f3df0 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.885+0000 7fbafcff9640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b4d0 0x7fbaec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.885+0000 7fbafcff9640 1 -- 172.21.15.27:0/818550782 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbaf8089ba0 con 0x7fbb001f3df0 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.886+0000 7fbafffff640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b4d0 0x7fbaec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.886+0000 7fbafffff640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b4d0 0x7fbaec06d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0036000 tx=0x7fbaf003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:44.890+0000 7fbafcff9640 1 -- 172.21.15.27:0/818550782 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaf8058d70 con 0x7fbb001f3df0 2024-03-06T07:27:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.139+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbb000b3390 con 0x7fbb001f3df0 2024-03-06T07:27:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.140+0000 7fbafcff9640 1 -- 172.21.15.27:0/818550782 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbaf8058670 con 0x7fbb001f3df0 2024-03-06T07:27:45.141 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g==", 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ==", 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg==", 2024-03-06T07:27:45.143 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ==", 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:45.144 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw==", 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.145 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA==", 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA==", 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:45.146 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg==", 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.147 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A==", 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw==", 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:45.148 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA==", 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-03-06T07:27:45.149 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A==", 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA==", 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:45.150 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ==", 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-03-06T07:27:45.151 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ==", 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg==", 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:27:45.152 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlkc2iEhAA6BuYba0FzXeVjGPOP8sgyQ==", 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw==", 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg==", 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:27:45.154 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:27:45.155 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:45.155 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:27:45.155 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.155 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:45.155 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:27:45.155 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:27:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b4d0 msgr2=0x7fbaec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b4d0 0x7fbaec06d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0036000 tx=0x7fbaf003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb001f3df0 msgr2=0x7fbb001f42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb001f3df0 0x7fbb001f42a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800b970 tx=0x7fbaf800be30 comp rx=0 tx=0).stop 2024-03-06T07:27:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 shutdown_connections 2024-03-06T07:27:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b4d0 0x7fbaec06d990 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb001f3df0 0x7fbb001f42a0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb0015df00 0x7fbb0015ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 --2- 172.21.15.27:0/818550782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b95a0 0x7fbb0015b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 >> 172.21.15.27:0/818550782 conn(0x7fbb00071960 msgr2=0x7fbb000d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 shutdown_connections 2024-03-06T07:27:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.142+0000 7fbb06783640 1 -- 172.21.15.27:0/818550782 wait complete. 2024-03-06T07:27:45.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:45.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.263+0000 7fb461c81640 1 Processor -- start 2024-03-06T07:27:45.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.263+0000 7fb461c81640 1 -- start start 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb461c81640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb45c15e060 0x7fb45c0ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb461c81640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb45c06fe20 0x7fb45c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb461c81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 0x7fb45c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb461c81640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45c0d0dd0 con 0x7fb45c06fe20 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb461c81640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45c0d10b0 con 0x7fb45c15e060 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb461c81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c0cc0e0 con 0x7fb45c0d0880 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb45c06fe20 0x7fb45c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb45c06fe20 0x7fb45c0ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52544/0 (socket says 172.21.15.27:52544) 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45affd640 1 -- 172.21.15.27:0/4181947256 learned_addr learned my addr 172.21.15.27:0/4181947256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45b7fe640 1 --2- 172.21.15.27:0/4181947256 >> v2:172.21.15.69:3300/0 conn(0x7fb45c15e060 0x7fb45c0ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45bfff640 1 --2- 172.21.15.27:0/4181947256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 0x7fb45c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45affd640 1 -- 172.21.15.27:0/4181947256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 msgr2=0x7fb45c0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45affd640 1 --2- 172.21.15.27:0/4181947256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 0x7fb45c0ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45affd640 1 -- 172.21.15.27:0/4181947256 >> v2:172.21.15.69:3300/0 conn(0x7fb45c15e060 msgr2=0x7fb45c0ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45affd640 1 --2- 172.21.15.27:0/4181947256 >> v2:172.21.15.69:3300/0 conn(0x7fb45c15e060 0x7fb45c0ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45affd640 1 -- 172.21.15.27:0/4181947256 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c0cc3c0 con 0x7fb45c06fe20 2024-03-06T07:27:45.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.264+0000 7fb45b7fe640 1 --2- 172.21.15.27:0/4181947256 >> v2:172.21.15.69:3300/0 conn(0x7fb45c15e060 0x7fb45c0ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:45.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.265+0000 7fb45affd640 1 --2- 172.21.15.27:0/4181947256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 0x7fb45c0ce850 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0722f0 tx=0x7fb45002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a326910d6562494c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.265+0000 7fb45a7fc640 1 -- 172.21.15.27:0/4181947256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb450035020 con 0x7fb45c06fe20 2024-03-06T07:27:45.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.265+0000 7fb45a7fc640 1 -- 172.21.15.27:0/4181947256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb450031870 con 0x7fb45c06fe20 2024-03-06T07:27:45.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.265+0000 7fb45a7fc640 1 -- 172.21.15.27:0/4181947256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb450031c40 con 0x7fb45c06fe20 2024-03-06T07:27:45.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.266+0000 7fb461c81640 1 -- 172.21.15.27:0/4181947256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 msgr2=0x7fb45c0ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.266+0000 7fb461c81640 1 --2- 172.21.15.27:0/4181947256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 0x7fb45c0ce850 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0722f0 tx=0x7fb45002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:45.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.266+0000 7fb461c81640 1 -- 172.21.15.27:0/4181947256 shutdown_connections 2024-03-06T07:27:45.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.266+0000 7fb461c81640 1 --2- 172.21.15.27:0/4181947256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 0x7fb45c0ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.266+0000 7fb461c81640 1 --2- 172.21.15.27:0/4181947256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 0x7fb45c0ce850 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.266+0000 7fb461c81640 1 --2- 172.21.15.27:0/4181947256 >> v2:172.21.15.69:3300/0 conn(0x7fb45c15e060 0x7fb45c0ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.266+0000 7fb461c81640 1 -- 172.21.15.27:0/4181947256 >> 172.21.15.27:0/4181947256 conn(0x7fb45c0d1a90 msgr2=0x7fb45c0d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.266+0000 7fb461c81640 1 -- 172.21.15.27:0/4181947256 shutdown_connections 2024-03-06T07:27:45.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.266+0000 7fb461c81640 1 -- 172.21.15.27:0/4181947256 wait complete. 2024-03-06T07:27:45.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.267+0000 7fb461c81640 1 Processor -- start 2024-03-06T07:27:45.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.267+0000 7fb461c81640 1 -- start start 2024-03-06T07:27:45.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb461c81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 0x7fb45c14cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb461c81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 0x7fb45c0c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb461c81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb45c15e060 0x7fb45c0c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb461c81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c06ed30 con 0x7fb45c06fe20 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb461c81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c15e440 con 0x7fb45c15e060 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb461c81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c06eeb0 con 0x7fb45c0d0880 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 0x7fb45c14cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 0x7fb45c14cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52548/0 (socket says 172.21.15.27:52548) 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45b7fe640 1 -- 172.21.15.27:0/3698142943 learned_addr learned my addr 172.21.15.27:0/3698142943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45bfff640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb45c15e060 0x7fb45c0c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45affd640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 0x7fb45c0c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45b7fe640 1 -- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 msgr2=0x7fb45c0c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45b7fe640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 0x7fb45c0c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45b7fe640 1 -- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb45c15e060 msgr2=0x7fb45c0c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45b7fe640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb45c15e060 0x7fb45c0c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45b7fe640 1 -- 172.21.15.27:0/3698142943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c0c7230 con 0x7fb45c06fe20 2024-03-06T07:27:45.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45affd640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 0x7fb45c0c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.268+0000 7fb45b7fe640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 0x7fb45c14cc40 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00d7d0 tx=0x7fb44c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.269+0000 7fb443fff640 1 -- 172.21.15.27:0/3698142943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c004030 con 0x7fb45c06fe20 2024-03-06T07:27:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.269+0000 7fb443fff640 1 -- 172.21.15.27:0/3698142943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c0041d0 con 0x7fb45c06fe20 2024-03-06T07:27:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.269+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c0c7510 con 0x7fb45c06fe20 2024-03-06T07:27:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.269+0000 7fb443fff640 1 -- 172.21.15.27:0/3698142943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c00b6c0 con 0x7fb45c06fe20 2024-03-06T07:27:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.269+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c0c7ad0 con 0x7fb45c06fe20 2024-03-06T07:27:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.270+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c06c8f0 con 0x7fb45c06fe20 2024-03-06T07:27:45.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.270+0000 7fb443fff640 1 -- 172.21.15.27:0/3698142943 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb44c00b860 con 0x7fb45c06fe20 2024-03-06T07:27:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.270+0000 7fb443fff640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb44806b4d0 0x7fb44806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.271+0000 7fb443fff640 1 -- 172.21.15.27:0/3698142943 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb44c08a100 con 0x7fb45c06fe20 2024-03-06T07:27:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.271+0000 7fb45affd640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb44806b4d0 0x7fb44806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.271+0000 7fb45affd640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb44806b4d0 0x7fb44806d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb450036000 tx=0x7fb450008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.275+0000 7fb443fff640 1 -- 172.21.15.27:0/3698142943 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb45c06c8f0 con 0x7fb45c06fe20 2024-03-06T07:27:45.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.456+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb45c067ff0 con 0x7fb45c06fe20 2024-03-06T07:27:45.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.457+0000 7fb443fff640 1 -- 172.21.15.27:0/3698142943 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+12820 (secure 0 0 0) 0x7fb44c0592d0 con 0x7fb45c06fe20 2024-03-06T07:27:45.458 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:45.458 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:44.745446+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:45.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb44806b4d0 msgr2=0x7fb44806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb44806b4d0 0x7fb44806d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb450036000 tx=0x7fb450008030 comp rx=0 tx=0).stop 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 msgr2=0x7fb45c14cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 0x7fb45c14cc40 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00d7d0 tx=0x7fb44c00dc90 comp rx=0 tx=0).stop 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 shutdown_connections 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb44806b4d0 0x7fb44806d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb45c15e060 0x7fb45c0c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb45c0d0880 0x7fb45c0c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 --2- 172.21.15.27:0/3698142943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb45c06fe20 0x7fb45c14cc40 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 >> 172.21.15.27:0/3698142943 conn(0x7fb45c0d1a90 msgr2=0x7fb45c0cead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 shutdown_connections 2024-03-06T07:27:45.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.459+0000 7fb461c81640 1 -- 172.21.15.27:0/3698142943 wait complete. 2024-03-06T07:27:45.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:45.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.580+0000 7f19fee0e640 1 Processor -- start 2024-03-06T07:27:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.580+0000 7f19fee0e640 1 -- start start 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fee0e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19f815dd90 0x7f19f80ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fee0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f80d29d0 0x7f19f80d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fee0e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19f80d4720 0x7f19f80b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fee0e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f80d3400 con 0x7f19f815dd90 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fee0e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f80d3740 con 0x7f19f80d4720 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fee0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80d3120 con 0x7f19f80d29d0 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fcbaa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19f815dd90 0x7f19f80ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fcbaa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19f815dd90 0x7f19f80ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52550/0 (socket says 172.21.15.27:52550) 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fcbaa640 1 -- 172.21.15.27:0/555804107 learned_addr learned my addr 172.21.15.27:0/555804107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fd3ab640 1 --2- 172.21.15.27:0/555804107 >> v2:172.21.15.69:3300/0 conn(0x7f19f80d4720 0x7f19f80b6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19f7fff640 1 --2- 172.21.15.27:0/555804107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f80d29d0 0x7f19f80d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fcbaa640 1 -- 172.21.15.27:0/555804107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f80d29d0 msgr2=0x7f19f80d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fcbaa640 1 --2- 172.21.15.27:0/555804107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f80d29d0 0x7f19f80d41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fcbaa640 1 -- 172.21.15.27:0/555804107 >> v2:172.21.15.69:3300/0 conn(0x7f19f80d4720 msgr2=0x7f19f80b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fcbaa640 1 --2- 172.21.15.27:0/555804107 >> v2:172.21.15.69:3300/0 conn(0x7f19f80d4720 0x7f19f80b6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fcbaa640 1 -- 172.21.15.27:0/555804107 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f80d39e0 con 0x7f19f815dd90 2024-03-06T07:27:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.581+0000 7f19fd3ab640 1 --2- 172.21.15.27:0/555804107 >> v2:172.21.15.69:3300/0 conn(0x7f19f80d4720 0x7f19f80b6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.582+0000 7f19fcbaa640 1 --2- 172.21.15.27:0/555804107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f815dd90 0x7f19f80ceea0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f19f80b6c60 tx=0x7f19e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c361adcd7644011 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.582+0000 7f19f77fe640 1 -- 172.21.15.27:0/555804107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8034070 con 0x7f19f815dd90 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.582+0000 7f19f77fe640 1 -- 172.21.15.27:0/555804107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e8030860 con 0x7f19f815dd90 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.582+0000 7f19f77fe640 1 -- 172.21.15.27:0/555804107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8030c30 con 0x7f19f815dd90 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.583+0000 7f19fee0e640 1 -- 172.21.15.27:0/555804107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f815dd90 msgr2=0x7f19f80ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.583+0000 7f19fee0e640 1 --2- 172.21.15.27:0/555804107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f815dd90 0x7f19f80ceea0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f19f80b6c60 tx=0x7f19e802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.583+0000 7f19fee0e640 1 -- 172.21.15.27:0/555804107 shutdown_connections 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.583+0000 7f19fee0e640 1 --2- 172.21.15.27:0/555804107 >> v2:172.21.15.69:3300/0 conn(0x7f19f80d4720 0x7f19f80b6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.583+0000 7f19fee0e640 1 --2- 172.21.15.27:0/555804107 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f80d29d0 0x7f19f80d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.583+0000 7f19fee0e640 1 --2- 172.21.15.27:0/555804107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f815dd90 0x7f19f80ceea0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.583+0000 7f19fee0e640 1 -- 172.21.15.27:0/555804107 >> 172.21.15.27:0/555804107 conn(0x7f19f8071960 msgr2=0x7f19f8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.583+0000 7f19fee0e640 1 -- 172.21.15.27:0/555804107 shutdown_connections 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.583+0000 7f19fee0e640 1 -- 172.21.15.27:0/555804107 wait complete. 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.584+0000 7f19fee0e640 1 Processor -- start 2024-03-06T07:27:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.584+0000 7f19fee0e640 1 -- start start 2024-03-06T07:27:45.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19fee0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f80d29d0 0x7f19f80c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19fee0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f80d4720 0x7f19f8160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19fee0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f815dd90 0x7f19f815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19fee0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80d2df0 con 0x7f19f80d4720 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19fee0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f815e170 con 0x7f19f80d29d0 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19fee0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80b7250 con 0x7f19f815dd90 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f80d4720 0x7f19f8160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f80d4720 0x7f19f8160c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52558/0 (socket says 172.21.15.27:52558) 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19f7fff640 1 -- 172.21.15.27:0/1509151917 learned_addr learned my addr 172.21.15.27:0/1509151917 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19fcbaa640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f80d29d0 0x7f19f80c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.585+0000 7f19fd3ab640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f815dd90 0x7f19f815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19f7fff640 1 -- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f815dd90 msgr2=0x7f19f815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19f7fff640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f815dd90 0x7f19f815ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19f7fff640 1 -- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f80d29d0 msgr2=0x7f19f80c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19f7fff640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f80d29d0 0x7f19f80c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19f7fff640 1 -- 172.21.15.27:0/1509151917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f8155440 con 0x7f19f80d4720 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19f7fff640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f80d4720 0x7f19f8160c10 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c820 tx=0x7f19ec00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19f57fa640 1 -- 172.21.15.27:0/1509151917 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec015070 con 0x7f19f80d4720 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19f57fa640 1 -- 172.21.15.27:0/1509151917 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec004220 con 0x7f19f80d4720 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f8155720 con 0x7f19f80d4720 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19f57fa640 1 -- 172.21.15.27:0/1509151917 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec00ce10 con 0x7f19f80d4720 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.586+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f8155ce0 con 0x7f19f80d4720 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.587+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f8072a30 con 0x7f19f80d4720 2024-03-06T07:27:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.587+0000 7f19f57fa640 1 -- 172.21.15.27:0/1509151917 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19ec007830 con 0x7f19f80d4720 2024-03-06T07:27:45.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.588+0000 7f19f57fa640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19e406b4d0 0x7f19e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.588+0000 7f19f57fa640 1 -- 172.21.15.27:0/1509151917 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f19ec089e00 con 0x7f19f80d4720 2024-03-06T07:27:45.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.588+0000 7f19fcbaa640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19e406b4d0 0x7f19e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.588+0000 7f19fcbaa640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19e406b4d0 0x7f19e406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f19e8036000 tx=0x7f19e803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.592+0000 7f19f57fa640 1 -- 172.21.15.27:0/1509151917 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19ec058f40 con 0x7f19f80d4720 2024-03-06T07:27:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.775+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f19f804f410 con 0x7f19f80d4720 2024-03-06T07:27:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.775+0000 7f19f57fa640 1 -- 172.21.15.27:0/1509151917 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f19ec058840 con 0x7f19f80d4720 2024-03-06T07:27:45.777 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:27:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19e406b4d0 msgr2=0x7f19e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19e406b4d0 0x7f19e406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f19e8036000 tx=0x7f19e803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f80d4720 msgr2=0x7f19f8160c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f80d4720 0x7f19f8160c10 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c820 tx=0x7f19ec00cce0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 shutdown_connections 2024-03-06T07:27:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19e406b4d0 0x7f19e406d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f815dd90 0x7f19f815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f80d4720 0x7f19f8160c10 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 --2- 172.21.15.27:0/1509151917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f80d29d0 0x7f19f80c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 >> 172.21.15.27:0/1509151917 conn(0x7f19f8071960 msgr2=0x7f19f80b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 shutdown_connections 2024-03-06T07:27:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.777+0000 7f19fee0e640 1 -- 172.21.15.27:0/1509151917 wait complete. 2024-03-06T07:27:45.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:45.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.895+0000 7f4a5c68f640 1 Processor -- start 2024-03-06T07:27:45.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.895+0000 7f4a5c68f640 1 -- start start 2024-03-06T07:27:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.896+0000 7f4a5c68f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a54071c50 0x7f4a540d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.896+0000 7f4a5c68f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a540b2c50 0x7f4a540d2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.896+0000 7f4a5c68f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a540ce970 con 0x7f4a5415e000 2024-03-06T07:27:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.896+0000 7f4a5c68f640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a540cece0 con 0x7f4a54071c50 2024-03-06T07:27:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.896+0000 7f4a5c68f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540cef30 con 0x7f4a540b2c50 2024-03-06T07:27:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.897+0000 7f4a59c2a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a54071c50 0x7f4a540d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.897+0000 7f4a59c2a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a54071c50 0x7f4a540d25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52566/0 (socket says 172.21.15.27:52566) 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.897+0000 7f4a59c2a640 1 -- 172.21.15.27:0/3922376933 learned_addr learned my addr 172.21.15.27:0/3922376933 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.897+0000 7f4a5ac2c640 1 --2- 172.21.15.27:0/3922376933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a540b2c50 0x7f4a540d2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.897+0000 7f4a59c2a640 1 -- 172.21.15.27:0/3922376933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a540b2c50 msgr2=0x7f4a540d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.897+0000 7f4a59c2a640 1 --2- 172.21.15.27:0/3922376933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a540b2c50 0x7f4a540d2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.897+0000 7f4a59c2a640 1 -- 172.21.15.27:0/3922376933 >> v1:172.21.15.69:6789/0 conn(0x7f4a5415e000 legacy=0x7f4a54071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.897+0000 7f4a59c2a640 1 -- 172.21.15.27:0/3922376933 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a540cf210 con 0x7f4a54071c50 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a59c2a640 1 --2- 172.21.15.27:0/3922376933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a54071c50 0x7f4a540d25d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009870 tx=0x7f4a4402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454586a79a01f092 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a59429640 1 -- 172.21.15.27:0/3922376933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a440317c0 con 0x7f4a54071c50 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a59429640 1 -- 172.21.15.27:0/3922376933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a44037070 con 0x7f4a54071c50 2024-03-06T07:27:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a59429640 1 -- 172.21.15.27:0/3922376933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44031b80 con 0x7f4a54071c50 2024-03-06T07:27:45.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a5c68f640 1 -- 172.21.15.27:0/3922376933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a54071c50 msgr2=0x7f4a540d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a5c68f640 1 --2- 172.21.15.27:0/3922376933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a54071c50 0x7f4a540d25d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009870 tx=0x7f4a4402b860 comp rx=0 tx=0).stop 2024-03-06T07:27:45.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a5c68f640 1 -- 172.21.15.27:0/3922376933 shutdown_connections 2024-03-06T07:27:45.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a5c68f640 1 --2- 172.21.15.27:0/3922376933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a54071c50 0x7f4a540d25d0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a5c68f640 1 --2- 172.21.15.27:0/3922376933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a540b2c50 0x7f4a540d2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a5c68f640 1 -- 172.21.15.27:0/3922376933 >> 172.21.15.27:0/3922376933 conn(0x7f4a540d3c80 msgr2=0x7f4a540d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a5c68f640 1 -- 172.21.15.27:0/3922376933 shutdown_connections 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.898+0000 7f4a5c68f640 1 -- 172.21.15.27:0/3922376933 wait complete. 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a5c68f640 1 Processor -- start 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a5c68f640 1 -- start start 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a5c68f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54071c50 0x7f4a5414b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a5c68f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540b2c50 0x7f4a5415bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a5c68f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5415e000 0x7f4a5415c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a5c68f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540b3320 con 0x7f4a540b2c50 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a5c68f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540b31a0 con 0x7f4a5415e000 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a5c68f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5415e3e0 con 0x7f4a54071c50 2024-03-06T07:27:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a59c2a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540b2c50 0x7f4a5415bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a59c2a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540b2c50 0x7f4a5415bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52568/0 (socket says 172.21.15.27:52568) 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.900+0000 7f4a59c2a640 1 -- 172.21.15.27:0/445409521 learned_addr learned my addr 172.21.15.27:0/445409521 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a5ac2c640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5415e000 0x7f4a5415c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a5a42b640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54071c50 0x7f4a5414b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a59c2a640 1 -- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54071c50 msgr2=0x7f4a5414b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a59c2a640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54071c50 0x7f4a5414b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a59c2a640 1 -- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5415e000 msgr2=0x7f4a5415c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a59c2a640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5415e000 0x7f4a5415c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a59c2a640 1 -- 172.21.15.27:0/445409521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5414b6d0 con 0x7f4a540b2c50 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a5ac2c640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5415e000 0x7f4a5415c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a59c2a640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540b2c50 0x7f4a5415bf20 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402b8a0 tx=0x7f4a440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a4affd640 1 -- 172.21.15.27:0/445409521 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44032860 con 0x7f4a540b2c50 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a4affd640 1 -- 172.21.15.27:0/445409521 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a44032a00 con 0x7f4a540b2c50 2024-03-06T07:27:45.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5415a7e0 con 0x7f4a540b2c50 2024-03-06T07:27:45.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a4affd640 1 -- 172.21.15.27:0/445409521 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44032df0 con 0x7f4a540b2c50 2024-03-06T07:27:45.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.901+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5415ada0 con 0x7f4a540b2c50 2024-03-06T07:27:45.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.902+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a54067ff0 con 0x7f4a540b2c50 2024-03-06T07:27:45.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.903+0000 7f4a4affd640 1 -- 172.21.15.27:0/445409521 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a4404c060 con 0x7f4a540b2c50 2024-03-06T07:27:45.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.903+0000 7f4a4affd640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a3806b4d0 0x7f4a3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.903+0000 7f4a4affd640 1 -- 172.21.15.27:0/445409521 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4a440abc60 con 0x7f4a540b2c50 2024-03-06T07:27:45.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.903+0000 7f4a5a42b640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a3806b4d0 0x7f4a3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.904+0000 7f4a5a42b640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a3806b4d0 0x7f4a3806d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4a50004ea0 tx=0x7f4a50009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:45.907+0000 7f4a4affd640 1 -- 172.21.15.27:0/445409521 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a44040070 con 0x7f4a540b2c50 2024-03-06T07:27:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.093+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4a540b7bd0 con 0x7f4a540b2c50 2024-03-06T07:27:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.093+0000 7f4a4affd640 1 -- 172.21.15.27:0/445409521 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v94) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4a4407a870 con 0x7f4a540b2c50 2024-03-06T07:27:46.094 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:46.095 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:27:46.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:27:46.095 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:27:46.095 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:27:46.095 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:46.095 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a3806b4d0 msgr2=0x7f4a3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a3806b4d0 0x7f4a3806d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4a50004ea0 tx=0x7f4a50009040 comp rx=0 tx=0).stop 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540b2c50 msgr2=0x7f4a5415bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540b2c50 0x7f4a5415bf20 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402b8a0 tx=0x7f4a440046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 shutdown_connections 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a3806b4d0 0x7f4a3806d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a5415e000 0x7f4a5415c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540b2c50 0x7f4a5415bf20 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 --2- 172.21.15.27:0/445409521 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a54071c50 0x7f4a5414b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 >> 172.21.15.27:0/445409521 conn(0x7f4a540d3c80 msgr2=0x7f4a540cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 shutdown_connections 2024-03-06T07:27:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.095+0000 7f4a5c68f640 1 -- 172.21.15.27:0/445409521 wait complete. 2024-03-06T07:27:46.107 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:46.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:46.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.218+0000 7faf29977640 1 Processor -- start 2024-03-06T07:27:46.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.218+0000 7faf29977640 1 -- start start 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf29977640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf2415ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf29977640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf240b5840 0x7faf240b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf29977640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2415f3b0 con 0x7faf2415dfb0 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf29977640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf2406f070 con 0x7faf240b5840 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf29977640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf24071940 con 0x7faf2415dbd0 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf1a7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf240b5840 0x7faf240b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf22ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf2415ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf1a7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf240b5840 0x7faf240b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:50832/0 (socket says 172.21.15.27:50832) 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf237fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7faf2415dfb0 0x7faf24161a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:60886/0 (socket says 172.21.15.27:60886) 2024-03-06T07:27:46.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.219+0000 7faf1a7fc640 1 -- 172.21.15.27:0/2903769503 learned_addr learned my addr 172.21.15.27:0/2903769503 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.220+0000 7faf1a7fc640 1 -- 172.21.15.27:0/2903769503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 msgr2=0x7faf2415ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.220+0000 7faf1a7fc640 1 --2- 172.21.15.27:0/2903769503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf2415ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.220+0000 7faf1a7fc640 1 -- 172.21.15.27:0/2903769503 >> v1:172.21.15.69:6789/0 conn(0x7faf2415dfb0 legacy=0x7faf24161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.220+0000 7faf1a7fc640 1 -- 172.21.15.27:0/2903769503 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf240b5db0 con 0x7faf240b5840 2024-03-06T07:27:46.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.220+0000 7faf22ffd640 1 --2- 172.21.15.27:0/2903769503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf2415ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.220+0000 7faf1a7fc640 1 --2- 172.21.15.27:0/2903769503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf240b5840 0x7faf240b39b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7faf0c009870 tx=0x7faf0c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30e77411134cf560 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf227fc640 1 -- 172.21.15.27:0/2903769503 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c034070 con 0x7faf240b5840 2024-03-06T07:27:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf227fc640 1 -- 172.21.15.27:0/2903769503 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c030840 con 0x7faf240b5840 2024-03-06T07:27:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf227fc640 1 -- 172.21.15.27:0/2903769503 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c030c10 con 0x7faf240b5840 2024-03-06T07:27:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf29977640 1 -- 172.21.15.27:0/2903769503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf240b5840 msgr2=0x7faf240b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf29977640 1 --2- 172.21.15.27:0/2903769503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf240b5840 0x7faf240b39b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7faf0c009870 tx=0x7faf0c02b980 comp rx=0 tx=0).stop 2024-03-06T07:27:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf29977640 1 -- 172.21.15.27:0/2903769503 shutdown_connections 2024-03-06T07:27:46.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf29977640 1 --2- 172.21.15.27:0/2903769503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf240b5840 0x7faf240b39b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf29977640 1 --2- 172.21.15.27:0/2903769503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf2415ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf29977640 1 -- 172.21.15.27:0/2903769503 >> 172.21.15.27:0/2903769503 conn(0x7faf240b26b0 msgr2=0x7faf240b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf29977640 1 -- 172.21.15.27:0/2903769503 shutdown_connections 2024-03-06T07:27:46.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.221+0000 7faf29977640 1 -- 172.21.15.27:0/2903769503 wait complete. 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.222+0000 7faf29977640 1 Processor -- start 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf29977640 1 -- start start 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf29977640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf240b5840 0x7faf24146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf29977640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf240ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf29977640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf2415dfb0 0x7faf241e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf29977640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf240721c0 con 0x7faf240b5840 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf29977640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf24071ee0 con 0x7faf2415dfb0 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf29977640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf240726e0 con 0x7faf2415dbd0 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf22ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf240b5840 0x7faf24146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf22ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf240b5840 0x7faf24146480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52574/0 (socket says 172.21.15.27:52574) 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.223+0000 7faf22ffd640 1 -- 172.21.15.27:0/1203335314 learned_addr learned my addr 172.21.15.27:0/1203335314 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf1a7fc640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf240ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf237fe640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf2415dfb0 0x7faf241e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf22ffd640 1 -- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 msgr2=0x7faf240ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf22ffd640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf240ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf22ffd640 1 -- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf2415dfb0 msgr2=0x7faf241e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf22ffd640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf2415dfb0 0x7faf241e75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf22ffd640 1 -- 172.21.15.27:0/1203335314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf240ae3d0 con 0x7faf240b5840 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf1a7fc640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf240ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf22ffd640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf240b5840 0x7faf24146480 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7faf1400ec10 tx=0x7faf1400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf20ff9640 1 -- 172.21.15.27:0/1203335314 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14017070 con 0x7faf240b5840 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf240ae6b0 con 0x7faf240b5840 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf20ff9640 1 -- 172.21.15.27:0/1203335314 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1400cc30 con 0x7faf240b5840 2024-03-06T07:27:46.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf20ff9640 1 -- 172.21.15.27:0/1203335314 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14013470 con 0x7faf240b5840 2024-03-06T07:27:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.224+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf240aec70 con 0x7faf240b5840 2024-03-06T07:27:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.225+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf24073380 con 0x7faf240b5840 2024-03-06T07:27:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.226+0000 7faf20ff9640 1 -- 172.21.15.27:0/1203335314 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf1401a070 con 0x7faf240b5840 2024-03-06T07:27:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.226+0000 7faf20ff9640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faf0406b4d0 0x7faf0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.226+0000 7faf20ff9640 1 -- 172.21.15.27:0/1203335314 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faf14089ea0 con 0x7faf240b5840 2024-03-06T07:27:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.226+0000 7faf1a7fc640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faf0406b4d0 0x7faf0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.227+0000 7faf1a7fc640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faf0406b4d0 0x7faf0406d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7faf0c036000 tx=0x7faf0c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.230+0000 7faf20ff9640 1 -- 172.21.15.27:0/1203335314 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf14058ff0 con 0x7faf240b5840 2024-03-06T07:27:46.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.412+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7faf241613c0 con 0x7faf240b5840 2024-03-06T07:27:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.757+0000 7faf20ff9640 1 -- 172.21.15.27:0/1203335314 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7faf140588f0 con 0x7faf240b5840 2024-03-06T07:27:46.758 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.760+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faf0406b4d0 msgr2=0x7faf0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.760+0000 7faf29977640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faf0406b4d0 0x7faf0406d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7faf0c036000 tx=0x7faf0c03b040 comp rx=0 tx=0).stop 2024-03-06T07:27:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.760+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf240b5840 msgr2=0x7faf24146480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.760+0000 7faf29977640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf240b5840 0x7faf24146480 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7faf1400ec10 tx=0x7faf1400c520 comp rx=0 tx=0).stop 2024-03-06T07:27:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.761+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 shutdown_connections 2024-03-06T07:27:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.761+0000 7faf29977640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faf0406b4d0 0x7faf0406d990 secure :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7faf0c036000 tx=0x7faf0c03b040 comp rx=0 tx=0).stop 2024-03-06T07:27:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.761+0000 7faf29977640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf2415dfb0 0x7faf241e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.761+0000 7faf29977640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faf2415dbd0 0x7faf240ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.761+0000 7faf29977640 1 --2- 172.21.15.27:0/1203335314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf240b5840 0x7faf24146480 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.761+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 >> 172.21.15.27:0/1203335314 conn(0x7faf240b26b0 msgr2=0x7faf24160800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.761+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 shutdown_connections 2024-03-06T07:27:46.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.761+0000 7faf29977640 1 -- 172.21.15.27:0/1203335314 wait complete. 2024-03-06T07:27:46.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:46.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.890+0000 7fd0e784e640 1 Processor -- start 2024-03-06T07:27:46.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.890+0000 7fd0e784e640 1 -- start start 2024-03-06T07:27:46.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.891+0000 7fd0e784e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e0142170 0x7fd0e0142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.891+0000 7fd0e784e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e0070980 con 0x7fd0e00b29a0 2024-03-06T07:27:46.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.891+0000 7fd0e784e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e0142f60 con 0x7fd0e00b4830 2024-03-06T07:27:46.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.891+0000 7fd0e784e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e01458e0 con 0x7fd0e0142170 2024-03-06T07:27:46.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.891+0000 7fd0e5deb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd0e00b29a0 0x7fd0e00716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40722/0 (socket says 172.21.15.27:40722) 2024-03-06T07:27:46.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.891+0000 7fd0e5deb640 1 -- 172.21.15.27:0/4291963424 learned_addr learned my addr 172.21.15.27:0/4291963424 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:46.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.891+0000 7fd0e55ea640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e0142170 0x7fd0e0142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.891+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736242874 0 0) 0x7fd0e0070980 con 0x7fd0e00b29a0 2024-03-06T07:27:46.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.891+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0cc003520 con 0x7fd0e00b29a0 2024-03-06T07:27:46.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4211887283 0 0) 0x7fd0cc003520 con 0x7fd0e00b29a0 2024-03-06T07:27:46.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e0142170 msgr2=0x7fd0e0142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e0142170 0x7fd0e0142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 >> v1:172.21.15.69:6789/0 conn(0x7fd0e00b4830 legacy=0x7fd0e0160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e0070bf0 con 0x7fd0e00b29a0 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fd0d80019f0 con 0x7fd0e00b29a0 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 >> v1:172.21.15.27:6789/0 conn(0x7fd0e00b29a0 legacy=0x7fd0e00716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0cc0039e0 0x7fd0cc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0cc0051c0 0x7fd0cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc00c090 con 0x7fd0cc004800 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc00c210 con 0x7fd0cc0051c0 2024-03-06T07:27:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc00c410 con 0x7fd0cc0039e0 2024-03-06T07:27:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e55ea640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e0142170 0x7fd0e0142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e4de9640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e5deb640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0cc0051c0 0x7fd0cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e55ea640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0cc0039e0 0x7fd0cc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e4de9640 1 -- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0cc0039e0 msgr2=0x7fd0cc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e4de9640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0cc0039e0 0x7fd0cc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e4de9640 1 -- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0cc0051c0 msgr2=0x7fd0cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e4de9640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0cc0051c0 0x7fd0cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e4de9640 1 -- 172.21.15.27:0/4291963424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc003520 con 0x7fd0cc004800 2024-03-06T07:27:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.892+0000 7fd0e5deb640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0cc0051c0 0x7fd0cc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.893+0000 7fd0e4de9640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0cc004c80 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0001750 tx=0x7fd0cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6bb016663fdea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.893+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0001370 con 0x7fd0cc004800 2024-03-06T07:27:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.893+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d00052d0 con 0x7fd0cc004800 2024-03-06T07:27:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.893+0000 7fd0d7fff640 1 -- 172.21.15.27:0/4291963424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0003830 con 0x7fd0cc004800 2024-03-06T07:27:46.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.894+0000 7fd0e784e640 1 -- 172.21.15.27:0/4291963424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 msgr2=0x7fd0cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.894+0000 7fd0e784e640 1 --2- 172.21.15.27:0/4291963424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0cc004c80 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0001750 tx=0x7fd0cc005830 comp rx=0 tx=0).stop 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.894+0000 7fd0e5deb640 1 -- 172.21.15.27:0/4291963424 reap_dead start 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.894+0000 7fd0e784e640 1 -- 172.21.15.27:0/4291963424 shutdown_connections 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.894+0000 7fd0e784e640 1 -- 172.21.15.27:0/4291963424 >> 172.21.15.27:0/4291963424 conn(0x7fd0e0066d00 msgr2=0x7fd0e0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.894+0000 7fd0e784e640 1 -- 172.21.15.27:0/4291963424 shutdown_connections 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.894+0000 7fd0e784e640 1 -- 172.21.15.27:0/4291963424 wait complete. 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.895+0000 7fd0e784e640 1 Processor -- start 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.895+0000 7fd0e784e640 1 -- start start 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e784e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0e00ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e784e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e01e4cd0 0x7fd0e013f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e784e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e01e5790 0x7fd0e01e5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e784e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00b4df0 con 0x7fd0cc004800 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e784e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00711b0 con 0x7fd0e01e5790 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e784e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e0142550 con 0x7fd0e01e4cd0 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e55ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0e00ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e55ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0e00ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52586/0 (socket says 172.21.15.27:52586) 2024-03-06T07:27:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e55ea640 1 -- 172.21.15.27:0/746219983 learned_addr learned my addr 172.21.15.27:0/746219983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e5deb640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e01e5790 0x7fd0e01e5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e4de9640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e01e4cd0 0x7fd0e013f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e55ea640 1 -- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e01e4cd0 msgr2=0x7fd0e013f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e55ea640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e01e4cd0 0x7fd0e013f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e55ea640 1 -- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e01e5790 msgr2=0x7fd0e01e5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e55ea640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e01e5790 0x7fd0e01e5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e55ea640 1 -- 172.21.15.27:0/746219983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e01e51d0 con 0x7fd0cc004800 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e4de9640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e01e4cd0 0x7fd0e013f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e55ea640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0e00ade80 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc00f490 tx=0x7fd0dc00f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0d5ffb640 1 -- 172.21.15.27:0/746219983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc018020 con 0x7fd0cc004800 2024-03-06T07:27:46.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0d5ffb640 1 -- 172.21.15.27:0/746219983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc014890 con 0x7fd0cc004800 2024-03-06T07:27:46.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e01e54b0 con 0x7fd0cc004800 2024-03-06T07:27:46.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.896+0000 7fd0d5ffb640 1 -- 172.21.15.27:0/746219983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc014c80 con 0x7fd0cc004800 2024-03-06T07:27:46.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.897+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e01e6180 con 0x7fd0cc004800 2024-03-06T07:27:46.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.898+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e00733c0 con 0x7fd0cc004800 2024-03-06T07:27:46.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.899+0000 7fd0d5ffb640 1 -- 172.21.15.27:0/746219983 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0dc01b020 con 0x7fd0cc004800 2024-03-06T07:27:46.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.899+0000 7fd0d5ffb640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0c406b500 0x7fd0c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.899+0000 7fd0d5ffb640 1 -- 172.21.15.27:0/746219983 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd0dc08a2a0 con 0x7fd0cc004800 2024-03-06T07:27:46.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.899+0000 7fd0e4de9640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0c406b500 0x7fd0c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.900+0000 7fd0e4de9640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0c406b500 0x7fd0c406d9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0001750 tx=0x7fd0d0009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:46.904+0000 7fd0d5ffb640 1 -- 172.21.15.27:0/746219983 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0dc022020 con 0x7fd0cc004800 2024-03-06T07:27:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.085+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd0e015bc30 con 0x7fd0cc004800 2024-03-06T07:27:47.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.759+0000 7fd0d5ffb640 1 -- 172.21.15.27:0/746219983 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v96) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd0dc014e20 con 0x7fd0cc004800 2024-03-06T07:27:47.760 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:27:47.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0c406b500 msgr2=0x7fd0c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0c406b500 0x7fd0c406d9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0001750 tx=0x7fd0d0009030 comp rx=0 tx=0).stop 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 msgr2=0x7fd0e00ade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0e00ade80 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc00f490 tx=0x7fd0dc00f950 comp rx=0 tx=0).stop 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 shutdown_connections 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd0c406b500 0x7fd0c406d9c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e01e5790 0x7fd0e01e5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0e01e4cd0 0x7fd0e013f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 --2- 172.21.15.27:0/746219983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc004800 0x7fd0e00ade80 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 >> 172.21.15.27:0/746219983 conn(0x7fd0e0066d00 msgr2=0x7fd0e01449c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 shutdown_connections 2024-03-06T07:27:47.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.762+0000 7fd0e784e640 1 -- 172.21.15.27:0/746219983 wait complete. 2024-03-06T07:27:47.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:47.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.903+0000 7fa3c9615640 1 Processor -- start 2024-03-06T07:27:47.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.903+0000 7fa3c9615640 1 -- start start 2024-03-06T07:27:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.903+0000 7fa3c9615640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3c4141170 0x7fa3c415f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c9615640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 0x7fa3c40d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c9615640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c4141550 con 0x7fa3c40d18f0 2024-03-06T07:27:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c9615640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c40d1320 con 0x7fa3c4141170 2024-03-06T07:27:47.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c9615640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c415edc0 con 0x7fa3c4162360 2024-03-06T07:27:47.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c2ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3c4141170 0x7fa3c415f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 0x7fa3c40d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c27fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa3c40d18f0 0x7fa3c4161c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:60900/0 (socket says 172.21.15.27:60900) 2024-03-06T07:27:47.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c27fc640 1 -- 172.21.15.27:0/3772422637 learned_addr learned my addr 172.21.15.27:0/3772422637 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:47.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c1ffb640 1 -- 172.21.15.27:0/3772422637 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697148321 0 0) 0x7fa3c4141550 con 0x7fa3c40d18f0 2024-03-06T07:27:47.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c1ffb640 1 -- 172.21.15.27:0/3772422637 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac003520 con 0x7fa3c40d18f0 2024-03-06T07:27:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c2ffd640 1 -- 172.21.15.27:0/3772422637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 msgr2=0x7fa3c40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.904+0000 7fa3c2ffd640 1 --2- 172.21.15.27:0/3772422637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 0x7fa3c40d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.905+0000 7fa3c2ffd640 1 -- 172.21.15.27:0/3772422637 >> v1:172.21.15.69:6789/0 conn(0x7fa3c40d18f0 legacy=0x7fa3c4161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.905+0000 7fa3c2ffd640 1 -- 172.21.15.27:0/3772422637 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c4162860 con 0x7fa3c4141170 2024-03-06T07:27:47.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.905+0000 7fa3c37fe640 1 --2- 172.21.15.27:0/3772422637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 0x7fa3c40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.905+0000 7fa3c2ffd640 1 --2- 172.21.15.27:0/3772422637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c4141170 0x7fa3c415f250 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa3b40097a0 tx=0x7fa3b402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5966d76cf7f86ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.905+0000 7fa3c1ffb640 1 -- 172.21.15.27:0/3772422637 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4035020 con 0x7fa3c4141170 2024-03-06T07:27:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c1ffb640 1 -- 172.21.15.27:0/3772422637 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b4031830 con 0x7fa3c4141170 2024-03-06T07:27:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c1ffb640 1 -- 172.21.15.27:0/3772422637 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4031be0 con 0x7fa3c4141170 2024-03-06T07:27:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c9615640 1 -- 172.21.15.27:0/3772422637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c4141170 msgr2=0x7fa3c415f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c9615640 1 --2- 172.21.15.27:0/3772422637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c4141170 0x7fa3c415f250 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa3b40097a0 tx=0x7fa3b402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c9615640 1 -- 172.21.15.27:0/3772422637 shutdown_connections 2024-03-06T07:27:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c9615640 1 --2- 172.21.15.27:0/3772422637 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 0x7fa3c40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c9615640 1 --2- 172.21.15.27:0/3772422637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c4141170 0x7fa3c415f250 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c9615640 1 -- 172.21.15.27:0/3772422637 >> 172.21.15.27:0/3772422637 conn(0x7fa3c406eb00 msgr2=0x7fa3c406ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c9615640 1 -- 172.21.15.27:0/3772422637 shutdown_connections 2024-03-06T07:27:47.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.906+0000 7fa3c9615640 1 -- 172.21.15.27:0/3772422637 wait complete. 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.907+0000 7fa3c9615640 1 Processor -- start 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.907+0000 7fa3c9615640 1 -- start start 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c9615640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40d18f0 0x7fa3c4143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c9615640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3c4141170 0x7fa3c40a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c9615640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 0x7fa3c414e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c9615640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c415ea50 con 0x7fa3c4141170 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c9615640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c415e770 con 0x7fa3c40d18f0 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c9615640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c40702d0 con 0x7fa3c4162360 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3c4141170 0x7fa3c40a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3c4141170 0x7fa3c40a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52598/0 (socket says 172.21.15.27:52598) 2024-03-06T07:27:47.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c27fc640 1 -- 172.21.15.27:0/3286012163 learned_addr learned my addr 172.21.15.27:0/3286012163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:47.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40d18f0 0x7fa3c4143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.908+0000 7fa3c37fe640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 0x7fa3c414e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3c27fc640 1 -- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 msgr2=0x7fa3c414e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3c27fc640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 0x7fa3c414e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3c27fc640 1 -- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40d18f0 msgr2=0x7fa3c4143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3c27fc640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40d18f0 0x7fa3c4143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3c27fc640 1 -- 172.21.15.27:0/3286012163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c41429d0 con 0x7fa3c4141170 2024-03-06T07:27:47.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3c27fc640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3c4141170 0x7fa3c40a94a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800c430 tx=0x7fa3b800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3ab7fe640 1 -- 172.21.15.27:0/3286012163 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8017070 con 0x7fa3c4141170 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3ab7fe640 1 -- 172.21.15.27:0/3286012163 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b8005ce0 con 0x7fa3c4141170 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c4142f60 con 0x7fa3c4141170 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3ab7fe640 1 -- 172.21.15.27:0/3286012163 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b80072f0 con 0x7fa3c4141170 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.909+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c4143520 con 0x7fa3c4141170 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.910+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c40700c0 con 0x7fa3c4141170 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.911+0000 7fa3ab7fe640 1 -- 172.21.15.27:0/3286012163 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3b801a070 con 0x7fa3c4141170 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.911+0000 7fa3ab7fe640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3b006b4d0 0x7fa3b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.911+0000 7fa3ab7fe640 1 -- 172.21.15.27:0/3286012163 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fa3b80894d0 con 0x7fa3c4141170 2024-03-06T07:27:47.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.911+0000 7fa3c2ffd640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3b006b4d0 0x7fa3b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.912+0000 7fa3c2ffd640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3b006b4d0 0x7fa3b006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4036000 tx=0x7fa3b4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:47.916+0000 7fa3ab7fe640 1 -- 172.21.15.27:0/3286012163 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3b8057d60 con 0x7fa3c4141170 2024-03-06T07:27:48.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.136+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa3c40b6ac0 con 0x7fa3c4141170 2024-03-06T07:27:48.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:48.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:48.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.772+0000 7fa3ab7fe640 1 -- 172.21.15.27:0/3286012163 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v57) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa3b805be00 con 0x7fa3c4141170 2024-03-06T07:27:48.774 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:27:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.777+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3b006b4d0 msgr2=0x7fa3b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.777+0000 7fa3c9615640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3b006b4d0 0x7fa3b006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4036000 tx=0x7fa3b4008030 comp rx=0 tx=0).stop 2024-03-06T07:27:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.777+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3c4141170 msgr2=0x7fa3c40a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.777+0000 7fa3c9615640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3c4141170 0x7fa3c40a94a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800c430 tx=0x7fa3b800c830 comp rx=0 tx=0).stop 2024-03-06T07:27:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.781+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 shutdown_connections 2024-03-06T07:27:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.781+0000 7fa3c9615640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3b006b4d0 0x7fa3b006d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.781+0000 7fa3c9615640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3c4162360 0x7fa3c414e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.781+0000 7fa3c9615640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3c4141170 0x7fa3c40a94a0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.781+0000 7fa3c9615640 1 --2- 172.21.15.27:0/3286012163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c40d18f0 0x7fa3c4143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.781+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 >> 172.21.15.27:0/3286012163 conn(0x7fa3c406eb00 msgr2=0x7fa3c415c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.784+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 shutdown_connections 2024-03-06T07:27:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.786+0000 7fa3c9615640 1 -- 172.21.15.27:0/3286012163 wait complete. 2024-03-06T07:27:48.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:48.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.951+0000 7f42b860e640 1 Processor -- start 2024-03-06T07:27:48.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.951+0000 7f42b860e640 1 -- start start 2024-03-06T07:27:48.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b860e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f42b006e700 0x7f42b015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b860e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b00b55d0 0x7f42b0071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b860e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b00b4160 con 0x7f42b0072660 2024-03-06T07:27:48.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b860e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b00b5b40 con 0x7f42b006e700 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b860e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b00b3ba0 con 0x7f42b00b55d0 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b5ba9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f42b0072660 0x7f42b006d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40732/0 (socket says 172.21.15.27:40732) 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b5ba9640 1 -- 172.21.15.27:0/2005485141 learned_addr learned my addr 172.21.15.27:0/2005485141 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b63aa640 1 --2- 172.21.15.27:0/2005485141 >> v2:172.21.15.69:3300/0 conn(0x7f42b006e700 0x7f42b015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b6bab640 1 --2- 172.21.15.27:0/2005485141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b00b55d0 0x7f42b0071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b53a8640 1 -- 172.21.15.27:0/2005485141 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875908344 0 0) 0x7f42b00b4160 con 0x7f42b0072660 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b53a8640 1 -- 172.21.15.27:0/2005485141 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f429c003520 con 0x7f42b0072660 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b6bab640 1 -- 172.21.15.27:0/2005485141 >> v2:172.21.15.69:3300/0 conn(0x7f42b006e700 msgr2=0x7f42b015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b6bab640 1 --2- 172.21.15.27:0/2005485141 >> v2:172.21.15.69:3300/0 conn(0x7f42b006e700 0x7f42b015ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b6bab640 1 -- 172.21.15.27:0/2005485141 >> v1:172.21.15.27:6789/0 conn(0x7f42b0072660 legacy=0x7f42b006d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b6bab640 1 -- 172.21.15.27:0/2005485141 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b00b3e80 con 0x7f42b00b55d0 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.952+0000 7f42b63aa640 1 --2- 172.21.15.27:0/2005485141 >> v2:172.21.15.69:3300/0 conn(0x7f42b006e700 0x7f42b015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.953+0000 7f42b6bab640 1 --2- 172.21.15.27:0/2005485141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b00b55d0 0x7f42b0071980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f42a8009d90 tx=0x7f42a802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e4e579b397afab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.953+0000 7f42b53a8640 1 -- 172.21.15.27:0/2005485141 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8039070 con 0x7f42b00b55d0 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.953+0000 7f42b53a8640 1 -- 172.21.15.27:0/2005485141 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a802cc50 con 0x7f42b00b55d0 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.953+0000 7f42b53a8640 1 -- 172.21.15.27:0/2005485141 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8034440 con 0x7f42b00b55d0 2024-03-06T07:27:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.954+0000 7f42b860e640 1 -- 172.21.15.27:0/2005485141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b00b55d0 msgr2=0x7f42b0071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.954+0000 7f42b860e640 1 --2- 172.21.15.27:0/2005485141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b00b55d0 0x7f42b0071980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f42a8009d90 tx=0x7f42a802bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.954+0000 7f42b860e640 1 -- 172.21.15.27:0/2005485141 shutdown_connections 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.954+0000 7f42b860e640 1 --2- 172.21.15.27:0/2005485141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b00b55d0 0x7f42b0071980 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.954+0000 7f42b860e640 1 --2- 172.21.15.27:0/2005485141 >> v2:172.21.15.69:3300/0 conn(0x7f42b006e700 0x7f42b015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.954+0000 7f42b860e640 1 -- 172.21.15.27:0/2005485141 >> 172.21.15.27:0/2005485141 conn(0x7f42b00b2720 msgr2=0x7f42b00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.954+0000 7f42b860e640 1 -- 172.21.15.27:0/2005485141 shutdown_connections 2024-03-06T07:27:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.954+0000 7f42b860e640 1 -- 172.21.15.27:0/2005485141 wait complete. 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.955+0000 7f42b860e640 1 Processor -- start 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.955+0000 7f42b860e640 1 -- start start 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b860e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b006e700 0x7f42b0157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b860e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b0072660 0x7f42b00b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b860e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42b00b55d0 0x7f42b015ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b860e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b0072040 con 0x7f42b0072660 2024-03-06T07:27:48.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b860e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b0071ec0 con 0x7f42b00b55d0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b860e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b00721c0 con 0x7f42b006e700 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b5ba9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b0072660 0x7f42b00b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b5ba9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b0072660 0x7f42b00b0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52606/0 (socket says 172.21.15.27:52606) 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b5ba9640 1 -- 172.21.15.27:0/537335000 learned_addr learned my addr 172.21.15.27:0/537335000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b63aa640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b006e700 0x7f42b0157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b6bab640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42b00b55d0 0x7f42b015ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b5ba9640 1 -- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b006e700 msgr2=0x7f42b0157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b5ba9640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b006e700 0x7f42b0157a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b5ba9640 1 -- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42b00b55d0 msgr2=0x7f42b015ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b5ba9640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42b00b55d0 0x7f42b015ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b5ba9640 1 -- 172.21.15.27:0/537335000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b0157fc0 con 0x7f42b0072660 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.956+0000 7f42b5ba9640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b0072660 0x7f42b00b0830 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f42a000c430 tx=0x7f42a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.957+0000 7f42a6ffd640 1 -- 172.21.15.27:0/537335000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0017070 con 0x7f42b0072660 2024-03-06T07:27:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.957+0000 7f42a6ffd640 1 -- 172.21.15.27:0/537335000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a0005d00 con 0x7f42b0072660 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.957+0000 7f42a6ffd640 1 -- 172.21.15.27:0/537335000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0007310 con 0x7f42b0072660 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.957+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b00b1280 con 0x7f42b0072660 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.957+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b00cecc0 con 0x7f42b0072660 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.958+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b00b33b0 con 0x7f42b0072660 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.958+0000 7f42a6ffd640 1 -- 172.21.15.27:0/537335000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42a001a070 con 0x7f42b0072660 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.959+0000 7f42a6ffd640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f429406b4d0 0x7f429406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.959+0000 7f42a6ffd640 1 -- 172.21.15.27:0/537335000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42a00895e0 con 0x7f42b0072660 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.959+0000 7f42b63aa640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f429406b4d0 0x7f429406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.959+0000 7f42b63aa640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f429406b4d0 0x7f429406d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f42b00ce790 tx=0x7f42ac009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:48.963+0000 7f42a6ffd640 1 -- 172.21.15.27:0/537335000 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42b00b33b0 con 0x7f42b0072660 2024-03-06T07:27:49.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.149+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42b0067ff0 con 0x7f42b0072660 2024-03-06T07:27:49.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.150+0000 7f42a6ffd640 1 -- 172.21.15.27:0/537335000 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16549 (secure 0 0 0) 0x7f42a0057e10 con 0x7f42b0072660 2024-03-06T07:27:49.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:49.152 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:48.137576+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:46.443375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:47.116887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f429406b4d0 msgr2=0x7f429406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f429406b4d0 0x7f429406d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f42b00ce790 tx=0x7f42ac009190 comp rx=0 tx=0).stop 2024-03-06T07:27:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b0072660 msgr2=0x7f42b00b0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b0072660 0x7f42b00b0830 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f42a000c430 tx=0x7f42a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 shutdown_connections 2024-03-06T07:27:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f429406b4d0 0x7f429406d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f42b00b55d0 0x7f42b015ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b0072660 0x7f42b00b0830 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 --2- 172.21.15.27:0/537335000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f42b006e700 0x7f42b0157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.152+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 >> 172.21.15.27:0/537335000 conn(0x7f42b00b2720 msgr2=0x7f42b00b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.153+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 shutdown_connections 2024-03-06T07:27:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.153+0000 7f42b860e640 1 -- 172.21.15.27:0/537335000 wait complete. 2024-03-06T07:27:49.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:49.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.272+0000 7f482b064640 1 Processor -- start 2024-03-06T07:27:49.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.272+0000 7f482b064640 1 -- start start 2024-03-06T07:27:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f482b064640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f482415e060 0x7f48240ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f482b064640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f482406fe20 0x7f48240ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f482b064640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240d0dd0 con 0x7f48240d0880 2024-03-06T07:27:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f482b064640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48240d10b0 con 0x7f482415e060 2024-03-06T07:27:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f482b064640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48240cc0e0 con 0x7f482406fe20 2024-03-06T07:27:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f4828e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f482415e060 0x7f48240ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f4828e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f482415e060 0x7f48240ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52616/0 (socket says 172.21.15.27:52616) 2024-03-06T07:27:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f4828e00640 1 -- 172.21.15.27:0/441679054 learned_addr learned my addr 172.21.15.27:0/441679054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f4823fff640 1 --2- 172.21.15.27:0/441679054 >> v2:172.21.15.69:3300/0 conn(0x7f482406fe20 0x7f48240ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f4828e00640 1 -- 172.21.15.27:0/441679054 >> v2:172.21.15.69:3300/0 conn(0x7f482406fe20 msgr2=0x7f48240ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f4828e00640 1 --2- 172.21.15.27:0/441679054 >> v2:172.21.15.69:3300/0 conn(0x7f482406fe20 0x7f48240ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f4828e00640 1 -- 172.21.15.27:0/441679054 >> v1:172.21.15.27:6789/0 conn(0x7f48240d0880 legacy=0x7f482406ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f48237fe640 1 -- 172.21.15.27:0/441679054 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258881098 0 0) 0x7f48240d0dd0 con 0x7f48240d0880 2024-03-06T07:27:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.273+0000 7f4828e00640 1 -- 172.21.15.27:0/441679054 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48240cc3c0 con 0x7f482415e060 2024-03-06T07:27:49.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.274+0000 7f4828e00640 1 --2- 172.21.15.27:0/441679054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f482415e060 0x7f48240ce310 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f4824068cd0 tx=0x7f481802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb850f4feb4fbe60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.274+0000 7f48237fe640 1 -- 172.21.15.27:0/441679054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818034070 con 0x7f482415e060 2024-03-06T07:27:49.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.274+0000 7f48237fe640 1 -- 172.21.15.27:0/441679054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4818030860 con 0x7f482415e060 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.274+0000 7f48237fe640 1 -- 172.21.15.27:0/441679054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818030c30 con 0x7f482415e060 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.275+0000 7f482b064640 1 -- 172.21.15.27:0/441679054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f482415e060 msgr2=0x7f48240ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.275+0000 7f482b064640 1 --2- 172.21.15.27:0/441679054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f482415e060 0x7f48240ce310 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f4824068cd0 tx=0x7f481802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.275+0000 7f482b064640 1 -- 172.21.15.27:0/441679054 shutdown_connections 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.275+0000 7f482b064640 1 --2- 172.21.15.27:0/441679054 >> v2:172.21.15.69:3300/0 conn(0x7f482406fe20 0x7f48240ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.275+0000 7f482b064640 1 --2- 172.21.15.27:0/441679054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f482415e060 0x7f48240ce310 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.275+0000 7f482b064640 1 -- 172.21.15.27:0/441679054 >> 172.21.15.27:0/441679054 conn(0x7f48240d1a90 msgr2=0x7f48240d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.275+0000 7f482b064640 1 -- 172.21.15.27:0/441679054 shutdown_connections 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.275+0000 7f482b064640 1 -- 172.21.15.27:0/441679054 wait complete. 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.276+0000 7f482b064640 1 Processor -- start 2024-03-06T07:27:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.276+0000 7f482b064640 1 -- start start 2024-03-06T07:27:49.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f482b064640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482406fe20 0x7f48240cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f482b064640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48240d0880 0x7f48240b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f482b064640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482415e060 0x7f48240c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f482b064640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48240cdbd0 con 0x7f48240d0880 2024-03-06T07:27:49.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f482b064640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482415e440 con 0x7f482406fe20 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f482b064640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48240cdd50 con 0x7f482415e060 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4823fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48240d0880 0x7f48240b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4823fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48240d0880 0x7f48240b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52624/0 (socket says 172.21.15.27:52624) 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4823fff640 1 -- 172.21.15.27:0/2847548218 learned_addr learned my addr 172.21.15.27:0/2847548218 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4828e00640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482406fe20 0x7f48240cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4823fff640 1 -- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482415e060 msgr2=0x7f48240c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4829601640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482415e060 0x7f48240c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4823fff640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482415e060 0x7f48240c7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4823fff640 1 -- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482406fe20 msgr2=0x7f48240cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4823fff640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482406fe20 0x7f48240cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.277+0000 7f4823fff640 1 -- 172.21.15.27:0/2847548218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48240cba20 con 0x7f48240d0880 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.278+0000 7f4828e00640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482406fe20 0x7f48240cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.278+0000 7f4829601640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482415e060 0x7f48240c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.278+0000 7f4823fff640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48240d0880 0x7f48240b48e0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f481000e840 tx=0x7f481000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.278+0000 7f48217fa640 1 -- 172.21.15.27:0/2847548218 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810017070 con 0x7f48240d0880 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.278+0000 7f48217fa640 1 -- 172.21.15.27:0/2847548218 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481000c940 con 0x7f48240d0880 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.278+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48240c6750 con 0x7f48240d0880 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.278+0000 7f48217fa640 1 -- 172.21.15.27:0/2847548218 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481000cd10 con 0x7f48240d0880 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.278+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48240c6d10 con 0x7f48240d0880 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.279+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4824067ff0 con 0x7f48240d0880 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.279+0000 7f48217fa640 1 -- 172.21.15.27:0/2847548218 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f481001a070 con 0x7f48240d0880 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.280+0000 7f48217fa640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480806b4d0 0x7f480806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.280+0000 7f48217fa640 1 -- 172.21.15.27:0/2847548218 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f481008a410 con 0x7f48240d0880 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.280+0000 7f4828e00640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480806b4d0 0x7f480806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.281+0000 7f4828e00640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480806b4d0 0x7f480806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4818036000 tx=0x7f481803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.284+0000 7f48217fa640 1 -- 172.21.15.27:0/2847548218 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4810059320 con 0x7f48240d0880 2024-03-06T07:27:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.466+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48240b07d0 con 0x7f48240d0880 2024-03-06T07:27:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.467+0000 7f48217fa640 1 -- 172.21.15.27:0/2847548218 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16549 (secure 0 0 0) 0x7f4810058c20 con 0x7f48240d0880 2024-03-06T07:27:49.469 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:49.469 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:48.137576+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:46.443375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:47.116887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480806b4d0 msgr2=0x7f480806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480806b4d0 0x7f480806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4818036000 tx=0x7f481803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48240d0880 msgr2=0x7f48240b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48240d0880 0x7f48240b48e0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f481000e840 tx=0x7f481000ed00 comp rx=0 tx=0).stop 2024-03-06T07:27:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 shutdown_connections 2024-03-06T07:27:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f480806b4d0 0x7f480806d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f482415e060 0x7f48240c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48240d0880 0x7f48240b48e0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 --2- 172.21.15.27:0/2847548218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f482406fe20 0x7f48240cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.469+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 >> 172.21.15.27:0/2847548218 conn(0x7f48240d1a90 msgr2=0x7f48240d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.470+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 shutdown_connections 2024-03-06T07:27:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.470+0000 7f482b064640 1 -- 172.21.15.27:0/2847548218 wait complete. 2024-03-06T07:27:49.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.591+0000 7fd26dea4640 1 Processor -- start 2024-03-06T07:27:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.591+0000 7fd26dea4640 1 -- start start 2024-03-06T07:27:49.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd26dea4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd26806d960 0x7fd26815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd26dea4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd268161bb0 0x7fd268161fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd26dea4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd268162520 con 0x7fd2680cf1c0 2024-03-06T07:27:49.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd26dea4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26815ed20 con 0x7fd268161bb0 2024-03-06T07:27:49.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd26dea4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd268072890 con 0x7fd26806d960 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd267fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd268161bb0 0x7fd268161fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd2677fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd2680cf1c0 0x7fd26806d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40746/0 (socket says 172.21.15.27:40746) 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd2677fe640 1 -- 172.21.15.27:0/4085016843 learned_addr learned my addr 172.21.15.27:0/4085016843 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd266ffd640 1 --2- 172.21.15.27:0/4085016843 >> v2:172.21.15.69:3300/0 conn(0x7fd26806d960 0x7fd26815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd267fff640 1 -- 172.21.15.27:0/4085016843 >> v2:172.21.15.69:3300/0 conn(0x7fd26806d960 msgr2=0x7fd26815f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd267fff640 1 --2- 172.21.15.27:0/4085016843 >> v2:172.21.15.69:3300/0 conn(0x7fd26806d960 0x7fd26815f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd267fff640 1 -- 172.21.15.27:0/4085016843 >> v1:172.21.15.27:6789/0 conn(0x7fd2680cf1c0 legacy=0x7fd26806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd267fff640 1 -- 172.21.15.27:0/4085016843 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2680cf5a0 con 0x7fd268161bb0 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.592+0000 7fd2667fc640 1 -- 172.21.15.27:0/4085016843 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712795032 0 0) 0x7fd268162520 con 0x7fd2680cf1c0 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.593+0000 7fd267fff640 1 --2- 172.21.15.27:0/4085016843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 0x7fd268161fe0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fd26806ec10 tx=0x7fd26002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7456f655a132b4cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.593+0000 7fd2667fc640 1 -- 172.21.15.27:0/4085016843 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260039070 con 0x7fd268161bb0 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.593+0000 7fd2667fc640 1 -- 172.21.15.27:0/4085016843 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26002cc80 con 0x7fd268161bb0 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.593+0000 7fd2667fc640 1 -- 172.21.15.27:0/4085016843 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260034660 con 0x7fd268161bb0 2024-03-06T07:27:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.594+0000 7fd26dea4640 1 -- 172.21.15.27:0/4085016843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 msgr2=0x7fd268161fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.594+0000 7fd26dea4640 1 --2- 172.21.15.27:0/4085016843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 0x7fd268161fe0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fd26806ec10 tx=0x7fd26002bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.594+0000 7fd26dea4640 1 -- 172.21.15.27:0/4085016843 shutdown_connections 2024-03-06T07:27:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.594+0000 7fd26dea4640 1 --2- 172.21.15.27:0/4085016843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 0x7fd268161fe0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.594+0000 7fd26dea4640 1 --2- 172.21.15.27:0/4085016843 >> v2:172.21.15.69:3300/0 conn(0x7fd26806d960 0x7fd26815f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.594+0000 7fd26dea4640 1 -- 172.21.15.27:0/4085016843 >> 172.21.15.27:0/4085016843 conn(0x7fd268066d00 msgr2=0x7fd268067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.594+0000 7fd26dea4640 1 -- 172.21.15.27:0/4085016843 shutdown_connections 2024-03-06T07:27:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.594+0000 7fd26dea4640 1 -- 172.21.15.27:0/4085016843 wait complete. 2024-03-06T07:27:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.595+0000 7fd26dea4640 1 Processor -- start 2024-03-06T07:27:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.595+0000 7fd26dea4640 1 -- start start 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd26dea4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd26806d960 0x7fd268139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd26dea4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2680cf1c0 0x7fd26813cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd26dea4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 0x7fd26815fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd26dea4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26815e6d0 con 0x7fd268161bb0 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd26dea4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd268162a70 con 0x7fd26806d960 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd26dea4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26815e9b0 con 0x7fd2680cf1c0 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd267fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 0x7fd26815fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd267fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 0x7fd26815fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52648/0 (socket says 172.21.15.27:52648) 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd267fff640 1 -- 172.21.15.27:0/1679997697 learned_addr learned my addr 172.21.15.27:0/1679997697 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd2677fe640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd26806d960 0x7fd268139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd266ffd640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2680cf1c0 0x7fd26813cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd267fff640 1 -- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2680cf1c0 msgr2=0x7fd26813cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd267fff640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2680cf1c0 0x7fd26813cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd267fff640 1 -- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd26806d960 msgr2=0x7fd268139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd267fff640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd26806d960 0x7fd268139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd267fff640 1 -- 172.21.15.27:0/1679997697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268160700 con 0x7fd268161bb0 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd266ffd640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2680cf1c0 0x7fd26813cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.596+0000 7fd267fff640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 0x7fd26815fd90 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fd260037040 tx=0x7fd260004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.597+0000 7fd25ffff640 1 -- 172.21.15.27:0/1679997697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260044020 con 0x7fd268161bb0 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.597+0000 7fd25ffff640 1 -- 172.21.15.27:0/1679997697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26003a040 con 0x7fd268161bb0 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.597+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2681609e0 con 0x7fd268161bb0 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.597+0000 7fd25ffff640 1 -- 172.21.15.27:0/1679997697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260039050 con 0x7fd268161bb0 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.597+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd268160fa0 con 0x7fd268161bb0 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.598+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2680733c0 con 0x7fd268161bb0 2024-03-06T07:27:49.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.598+0000 7fd25ffff640 1 -- 172.21.15.27:0/1679997697 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd260003b90 con 0x7fd268161bb0 2024-03-06T07:27:49.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.598+0000 7fd25ffff640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd24806b4d0 0x7fd24806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.598+0000 7fd25ffff640 1 -- 172.21.15.27:0/1679997697 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2600ac9b0 con 0x7fd268161bb0 2024-03-06T07:27:49.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.599+0000 7fd2677fe640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd24806b4d0 0x7fd24806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.599+0000 7fd2677fe640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd24806b4d0 0x7fd24806d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd258007fd0 tx=0x7fd258007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.603+0000 7fd25ffff640 1 -- 172.21.15.27:0/1679997697 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd26004e020 con 0x7fd268161bb0 2024-03-06T07:27:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:49.827+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd2680cd8c0 con 0x7fd268161bb0 2024-03-06T07:27:50.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.802+0000 7fd25ffff640 1 -- 172.21.15.27:0/1679997697 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd2680cf5a0 con 0x7fd268161bb0 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd24806b4d0 msgr2=0x7fd24806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd24806b4d0 0x7fd24806d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd258007fd0 tx=0x7fd258007de0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 msgr2=0x7fd26815fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 0x7fd26815fd90 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fd260037040 tx=0x7fd260004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 shutdown_connections 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd24806b4d0 0x7fd24806d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd268161bb0 0x7fd26815fd90 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2680cf1c0 0x7fd26813cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 --2- 172.21.15.27:0/1679997697 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd26806d960 0x7fd268139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 >> 172.21.15.27:0/1679997697 conn(0x7fd268066d00 msgr2=0x7fd268070c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 shutdown_connections 2024-03-06T07:27:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.804+0000 7fd26dea4640 1 -- 172.21.15.27:0/1679997697 wait complete. 2024-03-06T07:27:50.817 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:50.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.925+0000 7f4d35264640 1 Processor -- start 2024-03-06T07:27:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.925+0000 7f4d35264640 1 -- start start 2024-03-06T07:27:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.926+0000 7f4d35264640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4d3015f500 0x7f4d300ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.926+0000 7f4d35264640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d300cd310 con 0x7f4d300c5e30 2024-03-06T07:27:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.926+0000 7f4d35264640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d300cd5b0 con 0x7f4d300cf740 2024-03-06T07:27:50.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.926+0000 7f4d35264640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3015f8e0 con 0x7f4d3015f500 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.926+0000 7f4d2f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4d300c5e30 0x7f4d300c6280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40748/0 (socket says 172.21.15.27:40748) 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.926+0000 7f4d2f7fe640 1 -- 172.21.15.27:0/456375322 learned_addr learned my addr 172.21.15.27:0/456375322 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2effd640 1 --2- 172.21.15.27:0/456375322 >> v2:172.21.15.69:3300/0 conn(0x7f4d3015f500 0x7f4d300ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650264923 0 0) 0x7f4d300cd310 con 0x7f4d300c5e30 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d18003520 con 0x7f4d300c5e30 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1465617577 0 0) 0x7f4d18003520 con 0x7f4d300c5e30 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 >> v2:172.21.15.69:3300/0 conn(0x7f4d3015f500 msgr2=0x7f4d300ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 --2- 172.21.15.27:0/456375322 >> v2:172.21.15.69:3300/0 conn(0x7f4d3015f500 0x7f4d300ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 >> v1:172.21.15.69:6789/0 conn(0x7f4d300cf740 legacy=0x7f4d300ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d300cd890 con 0x7f4d300c5e30 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f4d28001a10 con 0x7f4d300c5e30 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 >> v1:172.21.15.27:6789/0 conn(0x7f4d300c5e30 legacy=0x7f4d300c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d180039b0 0x7f4d18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180043b0 0x7f4d18004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d18005120 0x7f4d1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3015f8e0 con 0x7f4d180039b0 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1800c060 con 0x7f4d18005120 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1800c260 con 0x7f4d180043b0 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2effd640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d180039b0 0x7f4d18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2f7fe640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d18005120 0x7f4d1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.927+0000 7f4d2e7fc640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180043b0 0x7f4d18004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.928+0000 7f4d2effd640 1 -- 172.21.15.27:0/456375322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180043b0 msgr2=0x7f4d18004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.928+0000 7f4d2effd640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180043b0 0x7f4d18004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.928+0000 7f4d2effd640 1 -- 172.21.15.27:0/456375322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d18005120 msgr2=0x7f4d1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.928+0000 7f4d2effd640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d18005120 0x7f4d1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.928+0000 7f4d2effd640 1 -- 172.21.15.27:0/456375322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d18003520 con 0x7f4d180039b0 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.928+0000 7f4d2f7fe640 1 -- 172.21.15.27:0/456375322 reap_dead start 2024-03-06T07:27:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.928+0000 7f4d2e7fc640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180043b0 0x7f4d18004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.928+0000 7f4d2effd640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d180039b0 0x7f4d18003d90 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009ac0 tx=0x7f4d18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571f223f1d87832c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.928+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d20039070 con 0x7f4d180039b0 2024-03-06T07:27:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.929+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2002c6b0 con 0x7f4d180039b0 2024-03-06T07:27:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.929+0000 7f4d2dffb640 1 -- 172.21.15.27:0/456375322 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d20009690 con 0x7f4d180039b0 2024-03-06T07:27:50.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.929+0000 7f4d35264640 1 -- 172.21.15.27:0/456375322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d180039b0 msgr2=0x7f4d18003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.929+0000 7f4d35264640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d180039b0 0x7f4d18003d90 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009ac0 tx=0x7f4d18005830 comp rx=0 tx=0).stop 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.929+0000 7f4d35264640 1 -- 172.21.15.27:0/456375322 shutdown_connections 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.929+0000 7f4d35264640 1 --2- 172.21.15.27:0/456375322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d180039b0 0x7f4d18003d90 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.929+0000 7f4d35264640 1 -- 172.21.15.27:0/456375322 >> 172.21.15.27:0/456375322 conn(0x7f4d300b3700 msgr2=0x7f4d300b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.929+0000 7f4d35264640 1 -- 172.21.15.27:0/456375322 shutdown_connections 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.929+0000 7f4d35264640 1 -- 172.21.15.27:0/456375322 wait complete. 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.931+0000 7f4d35264640 1 Processor -- start 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.931+0000 7f4d35264640 1 -- start start 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d35264640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180039b0 0x7f4d30151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d35264640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d300b5e80 0x7f4d30152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d35264640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d300c8eb0 0x7f4d300c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d35264640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d300c59a0 con 0x7f4d300b5e80 2024-03-06T07:27:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d35264640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d300c56c0 con 0x7f4d300c8eb0 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d35264640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d300ce300 con 0x7f4d180039b0 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d300b5e80 0x7f4d30152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d300b5e80 0x7f4d30152040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52668/0 (socket says 172.21.15.27:52668) 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2e7fc640 1 -- 172.21.15.27:0/4137615726 learned_addr learned my addr 172.21.15.27:0/4137615726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2f7fe640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d300c8eb0 0x7f4d300c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2effd640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180039b0 0x7f4d30151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2e7fc640 1 -- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180039b0 msgr2=0x7f4d30151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2e7fc640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180039b0 0x7f4d30151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2e7fc640 1 -- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d300c8eb0 msgr2=0x7f4d300c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2e7fc640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d300c8eb0 0x7f4d300c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2e7fc640 1 -- 172.21.15.27:0/4137615726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d300c9cf0 con 0x7f4d300b5e80 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2effd640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180039b0 0x7f4d30151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d2e7fc640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d300b5e80 0x7f4d30152040 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400e550 tx=0x7f4d2400e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d177fe640 1 -- 172.21.15.27:0/4137615726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24019070 con 0x7f4d300b5e80 2024-03-06T07:27:50.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d177fe640 1 -- 172.21.15.27:0/4137615726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d240052e0 con 0x7f4d300b5e80 2024-03-06T07:27:50.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.933+0000 7f4d177fe640 1 -- 172.21.15.27:0/4137615726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d240037f0 con 0x7f4d300b5e80 2024-03-06T07:27:50.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.934+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d300b53a0 con 0x7f4d300b5e80 2024-03-06T07:27:50.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.934+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d300b5960 con 0x7f4d300b5e80 2024-03-06T07:27:50.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.935+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3009fcf0 con 0x7f4d300b5e80 2024-03-06T07:27:50.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.935+0000 7f4d177fe640 1 -- 172.21.15.27:0/4137615726 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d2401c070 con 0x7f4d300b5e80 2024-03-06T07:27:50.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.935+0000 7f4d177fe640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d1c06b4d0 0x7f4d1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.936+0000 7f4d177fe640 1 -- 172.21.15.27:0/4137615726 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d24089680 con 0x7f4d300b5e80 2024-03-06T07:27:50.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.936+0000 7f4d2effd640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d1c06b4d0 0x7f4d1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.936+0000 7f4d2effd640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d1c06b4d0 0x7f4d1c06d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4d20036000 tx=0x7f4d20030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:50.940+0000 7f4d177fe640 1 -- 172.21.15.27:0/4137615726 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d24058000 con 0x7f4d300b5e80 2024-03-06T07:27:51.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.161+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4d3004f410 con 0x7f4d300b5e80 2024-03-06T07:27:51.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.803+0000 7f4d177fe640 1 -- 172.21.15.27:0/4137615726 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4d2405c0a0 con 0x7f4d300b5e80 2024-03-06T07:27:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d1c06b4d0 msgr2=0x7f4d1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d1c06b4d0 0x7f4d1c06d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4d20036000 tx=0x7f4d20030000 comp rx=0 tx=0).stop 2024-03-06T07:27:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d300b5e80 msgr2=0x7f4d30152040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d300b5e80 0x7f4d30152040 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400e550 tx=0x7f4d2400e950 comp rx=0 tx=0).stop 2024-03-06T07:27:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 shutdown_connections 2024-03-06T07:27:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d1c06b4d0 0x7f4d1c06d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d300c8eb0 0x7f4d300c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d300b5e80 0x7f4d30152040 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 --2- 172.21.15.27:0/4137615726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d180039b0 0x7f4d30151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.805+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 >> 172.21.15.27:0/4137615726 conn(0x7f4d300b3700 msgr2=0x7f4d3006fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.806+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 shutdown_connections 2024-03-06T07:27:51.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.806+0000 7f4d35264640 1 -- 172.21.15.27:0/4137615726 wait complete. 2024-03-06T07:27:51.818 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:51.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.923+0000 7f83ba713640 1 Processor -- start 2024-03-06T07:27:51.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.923+0000 7f83ba713640 1 -- start start 2024-03-06T07:27:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83ba713640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83b40b95a0 0x7f83b40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83ba713640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b40b9a10 0x7f83b40b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83ba713640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b40cea30 con 0x7f83b415df00 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83ba713640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b40cf3c0 con 0x7f83b40b95a0 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83ba713640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40cf0e0 con 0x7f83b40b9a10 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83b37fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83b40b95a0 0x7f83b40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83b37fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83b40b95a0 0x7f83b40b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52684/0 (socket says 172.21.15.27:52684) 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83b37fe640 1 -- 172.21.15.27:0/3030418642 learned_addr learned my addr 172.21.15.27:0/3030418642 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83b8cb0640 1 --2- 172.21.15.27:0/3030418642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b40b9a10 0x7f83b40b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83b37fe640 1 -- 172.21.15.27:0/3030418642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b40b9a10 msgr2=0x7f83b40b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83b37fe640 1 --2- 172.21.15.27:0/3030418642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b40b9a10 0x7f83b40b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83b37fe640 1 -- 172.21.15.27:0/3030418642 >> v1:172.21.15.69:6789/0 conn(0x7f83b415df00 legacy=0x7f83b40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.925+0000 7f83b37fe640 1 -- 172.21.15.27:0/3030418642 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b40ce470 con 0x7f83b40b95a0 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.926+0000 7f83b37fe640 1 --2- 172.21.15.27:0/3030418642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 0x7f83b40b5700 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009870 tx=0x7f83a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fe299a12c5449c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.926+0000 7f83b2ffd640 1 -- 172.21.15.27:0/3030418642 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8034070 con 0x7f83b40b95a0 2024-03-06T07:27:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.926+0000 7f83b2ffd640 1 -- 172.21.15.27:0/3030418642 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8030840 con 0x7f83b40b95a0 2024-03-06T07:27:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.926+0000 7f83b2ffd640 1 -- 172.21.15.27:0/3030418642 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8030c10 con 0x7f83b40b95a0 2024-03-06T07:27:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.927+0000 7f83ba713640 1 -- 172.21.15.27:0/3030418642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 msgr2=0x7f83b40b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.927+0000 7f83ba713640 1 --2- 172.21.15.27:0/3030418642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 0x7f83b40b5700 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009870 tx=0x7f83a802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:51.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.927+0000 7f83ba713640 1 -- 172.21.15.27:0/3030418642 shutdown_connections 2024-03-06T07:27:51.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.927+0000 7f83ba713640 1 --2- 172.21.15.27:0/3030418642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 0x7f83b40b5700 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.927+0000 7f83ba713640 1 --2- 172.21.15.27:0/3030418642 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b40b9a10 0x7f83b40b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.927+0000 7f83ba713640 1 -- 172.21.15.27:0/3030418642 >> 172.21.15.27:0/3030418642 conn(0x7f83b4071960 msgr2=0x7f83b4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.927+0000 7f83ba713640 1 -- 172.21.15.27:0/3030418642 shutdown_connections 2024-03-06T07:27:51.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.927+0000 7f83ba713640 1 -- 172.21.15.27:0/3030418642 wait complete. 2024-03-06T07:27:51.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.928+0000 7f83ba713640 1 Processor -- start 2024-03-06T07:27:51.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83ba713640 1 -- start start 2024-03-06T07:27:51.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83ba713640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 0x7f83b415c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83ba713640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b40b9a10 0x7f83b4160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83ba713640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b415df00 0x7f83b414ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83ba713640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40cedf0 con 0x7f83b40b95a0 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83ba713640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40cec70 con 0x7f83b40b9a10 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83ba713640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40d3220 con 0x7f83b415df00 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83b3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 0x7f83b415c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83b3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 0x7f83b415c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52688/0 (socket says 172.21.15.27:52688) 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.929+0000 7f83b3fff640 1 -- 172.21.15.27:0/2330036506 learned_addr learned my addr 172.21.15.27:0/2330036506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b8cb0640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b415df00 0x7f83b414ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b37fe640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b40b9a10 0x7f83b4160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b3fff640 1 -- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b415df00 msgr2=0x7f83b414ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b3fff640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b415df00 0x7f83b414ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b3fff640 1 -- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b40b9a10 msgr2=0x7f83b4160db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b3fff640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b40b9a10 0x7f83b4160db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b3fff640 1 -- 172.21.15.27:0/2330036506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b4150480 con 0x7f83b40b95a0 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b8cb0640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b415df00 0x7f83b414ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b3fff640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 0x7f83b415c4d0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f83a400c430 tx=0x7f83a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b0ff9640 1 -- 172.21.15.27:0/2330036506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4017070 con 0x7f83b40b95a0 2024-03-06T07:27:51.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b0ff9640 1 -- 172.21.15.27:0/2330036506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a40052d0 con 0x7f83b40b95a0 2024-03-06T07:27:51.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b41e6760 con 0x7f83b40b95a0 2024-03-06T07:27:51.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83b0ff9640 1 -- 172.21.15.27:0/2330036506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4003830 con 0x7f83b40b95a0 2024-03-06T07:27:51.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.930+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b415f670 con 0x7f83b40b95a0 2024-03-06T07:27:51.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.931+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b4067ff0 con 0x7f83b40b95a0 2024-03-06T07:27:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.932+0000 7f83b0ff9640 1 -- 172.21.15.27:0/2330036506 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83a401a070 con 0x7f83b40b95a0 2024-03-06T07:27:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.932+0000 7f83b0ff9640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.932+0000 7f83b0ff9640 1 -- 172.21.15.27:0/2330036506 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83a40894f0 con 0x7f83b40b95a0 2024-03-06T07:27:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.932+0000 7f83b37fe640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.933+0000 7f83b37fe640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f83a8036000 tx=0x7f83a803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:51.936+0000 7f83b0ff9640 1 -- 172.21.15.27:0/2330036506 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83a4058400 con 0x7f83b40b95a0 2024-03-06T07:27:52.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.157+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b4071f90 con 0x7f83b40b95a0 2024-03-06T07:27:52.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.158+0000 7f83b0ff9640 1 -- 172.21.15.27:0/2330036506 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f83a4057d00 con 0x7f83b40b95a0 2024-03-06T07:27:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:52.160 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1540177819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":1540177819},{"type":"v1","addr":"172.21.15.69:6837","nonce":1540177819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6715,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3062817242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3062817242},{"type":"v1","addr":"172.21.15.69:6835","nonce":3062817242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7425,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2584284640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2584284640},{"type":"v1","addr":"172.21.15.27:6834","nonce":2584284640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:27:51.162771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7428":{"gid":7428,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/3147032864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3147032864},{"type":"v1","addr":"172.21.15.27:6835","nonce":3147032864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7428,"qdb_cluster":[7428]},"id":7}]} 2024-03-06T07:27:52.160 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 61 2024-03-06T07:27:52.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a006b4d0 msgr2=0x7f83a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f83a8036000 tx=0x7f83a803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 msgr2=0x7f83b415c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 0x7f83b415c4d0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f83a400c430 tx=0x7f83a400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 shutdown_connections 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f83b415df00 0x7f83b414ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f83b40b9a10 0x7f83b4160db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 --2- 172.21.15.27:0/2330036506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83b40b95a0 0x7f83b415c4d0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 >> 172.21.15.27:0/2330036506 conn(0x7f83b4071960 msgr2=0x7f83b40ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 shutdown_connections 2024-03-06T07:27:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.160+0000 7f83ba713640 1 -- 172.21.15.27:0/2330036506 wait complete. 2024-03-06T07:27:52.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.277+0000 7fa54fc0e640 1 Processor -- start 2024-03-06T07:27:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.278+0000 7fa54fc0e640 1 -- start start 2024-03-06T07:27:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.278+0000 7fa54fc0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548071540 0x7fa5480b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.278+0000 7fa54fc0e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5480b2b50 0x7fa5480b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.278+0000 7fa54fc0e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5480ce610 con 0x7fa5480d3400 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.278+0000 7fa54fc0e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5480d2f70 con 0x7fa5480b2b50 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.278+0000 7fa54fc0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480d4be0 con 0x7fa548071540 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.278+0000 7fa54d1a9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa5480d3400 0x7fa548071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40764/0 (socket says 172.21.15.27:40764) 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54d1a9640 1 -- 172.21.15.27:0/2668695700 learned_addr learned my addr 172.21.15.27:0/2668695700 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54e1ab640 1 --2- 172.21.15.27:0/2668695700 >> v2:172.21.15.69:3300/0 conn(0x7fa5480b2b50 0x7fa5480b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54d9aa640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548071540 0x7fa5480b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143886046 0 0) 0x7fa5480ce610 con 0x7fa5480d3400 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa534003520 con 0x7fa5480d3400 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868725669 0 0) 0x7fa534003520 con 0x7fa5480d3400 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548071540 msgr2=0x7fa5480b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548071540 0x7fa5480b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 >> v2:172.21.15.69:3300/0 conn(0x7fa5480b2b50 msgr2=0x7fa5480b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 --2- 172.21.15.27:0/2668695700 >> v2:172.21.15.69:3300/0 conn(0x7fa5480b2b50 0x7fa5480b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5480d2920 con 0x7fa5480d3400 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fa53c001670 con 0x7fa5480d3400 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 >> v1:172.21.15.27:6789/0 conn(0x7fa5480d3400 legacy=0x7fa548071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5340039b0 0x7fa534003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548071540 0x7fa534004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa534004d40 0x7fa5340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480d2f70 con 0x7fa534004d40 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480d4be0 con 0x7fa5340039b0 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53400b9c0 con 0x7fa548071540 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54e1ab640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa534004d40 0x7fa5340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54d9aa640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5340039b0 0x7fa534003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.279+0000 7fa54d1a9640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548071540 0x7fa534004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.280+0000 7fa54e1ab640 1 -- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548071540 msgr2=0x7fa534004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.280+0000 7fa54e1ab640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548071540 0x7fa534004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.280+0000 7fa54e1ab640 1 -- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5340039b0 msgr2=0x7fa534003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.280+0000 7fa54e1ab640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5340039b0 0x7fa534003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.280+0000 7fa54e1ab640 1 -- 172.21.15.27:0/2668695700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa534003520 con 0x7fa534004d40 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.280+0000 7fa54e1ab640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa534004d40 0x7fa5340051c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fa540009fc0 tx=0x7fa534005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770fd61705236a05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540039070 con 0x7fa534004d40 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54002cc60 con 0x7fa534004d40 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54c9a8640 1 -- 172.21.15.27:0/2668695700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540009c10 con 0x7fa534004d40 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54fc0e640 1 -- 172.21.15.27:0/2668695700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa534004d40 msgr2=0x7fa5340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54fc0e640 1 --2- 172.21.15.27:0/2668695700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa534004d40 0x7fa5340051c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fa540009fc0 tx=0x7fa534005830 comp rx=0 tx=0).stop 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54e1ab640 1 -- 172.21.15.27:0/2668695700 reap_dead start 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54fc0e640 1 -- 172.21.15.27:0/2668695700 shutdown_connections 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54fc0e640 1 -- 172.21.15.27:0/2668695700 >> 172.21.15.27:0/2668695700 conn(0x7fa5480d3ca0 msgr2=0x7fa5480d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54fc0e640 1 -- 172.21.15.27:0/2668695700 shutdown_connections 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.281+0000 7fa54fc0e640 1 -- 172.21.15.27:0/2668695700 wait complete. 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.283+0000 7fa54fc0e640 1 Processor -- start 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.283+0000 7fa54fc0e640 1 -- start start 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54fc0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa534004d40 0x7fa54814b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54fc0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548156b80 0x7fa5480b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54fc0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa548157000 0x7fa5480b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54fc0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54815e370 con 0x7fa548157000 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54fc0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480ce8f0 con 0x7fa534004d40 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54fc0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54814bfd0 con 0x7fa548156b80 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54e1ab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa548157000 0x7fa5480b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54e1ab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa548157000 0x7fa5480b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52706/0 (socket says 172.21.15.27:52706) 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54e1ab640 1 -- 172.21.15.27:0/1514765820 learned_addr learned my addr 172.21.15.27:0/1514765820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54d1a9640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548156b80 0x7fa5480b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54d9aa640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa534004d40 0x7fa54814b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54e1ab640 1 -- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548156b80 msgr2=0x7fa5480b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54e1ab640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548156b80 0x7fa5480b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54e1ab640 1 -- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa534004d40 msgr2=0x7fa54814b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54e1ab640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa534004d40 0x7fa54814b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54e1ab640 1 -- 172.21.15.27:0/1514765820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa548158680 con 0x7fa548157000 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54d9aa640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa534004d40 0x7fa54814b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54e1ab640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa548157000 0x7fa5480b5b80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fa540037040 tx=0x7fa540009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa53a7fc640 1 -- 172.21.15.27:0/1514765820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540039030 con 0x7fa548157000 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa53a7fc640 1 -- 172.21.15.27:0/1514765820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54003a040 con 0x7fa548157000 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa53a7fc640 1 -- 172.21.15.27:0/1514765820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540003990 con 0x7fa548157000 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.285+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa548155dc0 con 0x7fa548157000 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.286+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa548156380 con 0x7fa548157000 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.286+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa548068700 con 0x7fa548157000 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.287+0000 7fa53a7fc640 1 -- 172.21.15.27:0/1514765820 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa540045030 con 0x7fa548157000 2024-03-06T07:27:52.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.288+0000 7fa53a7fc640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa52c06b500 0x7fa52c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.288+0000 7fa53a7fc640 1 -- 172.21.15.27:0/1514765820 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5400ac110 con 0x7fa548157000 2024-03-06T07:27:52.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.288+0000 7fa54d9aa640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa52c06b500 0x7fa52c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.288+0000 7fa54d9aa640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa52c06b500 0x7fa52c06d9c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa544009a40 tx=0x7fa544008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.292+0000 7fa53a7fc640 1 -- 172.21.15.27:0/1514765820 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa54004c020 con 0x7fa548157000 2024-03-06T07:27:52.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.475+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa54815da20 con 0x7fa548157000 2024-03-06T07:27:52.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.476+0000 7fa53a7fc640 1 -- 172.21.15.27:0/1514765820 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16536 (secure 0 0 0) 0x7fa54007b020 con 0x7fa548157000 2024-03-06T07:27:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:52.478 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":98,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:49.783505+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:46.443375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:47.116887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:52.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa52c06b500 msgr2=0x7fa52c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa52c06b500 0x7fa52c06d9c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa544009a40 tx=0x7fa544008040 comp rx=0 tx=0).stop 2024-03-06T07:27:52.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa548157000 msgr2=0x7fa5480b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa548157000 0x7fa5480b5b80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fa540037040 tx=0x7fa540009750 comp rx=0 tx=0).stop 2024-03-06T07:27:52.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 shutdown_connections 2024-03-06T07:27:52.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa52c06b500 0x7fa52c06d9c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa548157000 0x7fa5480b5b80 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa548156b80 0x7fa5480b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 --2- 172.21.15.27:0/1514765820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa534004d40 0x7fa54814b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 >> 172.21.15.27:0/1514765820 conn(0x7fa5480d3ca0 msgr2=0x7fa54815fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 shutdown_connections 2024-03-06T07:27:52.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.478+0000 7fa54fc0e640 1 -- 172.21.15.27:0/1514765820 wait complete. 2024-03-06T07:27:52.491 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:52.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:52.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.595+0000 7f50f3d63640 1 Processor -- start 2024-03-06T07:27:52.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.596+0000 7f50f3d63640 1 -- start start 2024-03-06T07:27:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.597+0000 7f50f3d63640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f50ec0b95a0 0x7f50ec0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.597+0000 7f50f3d63640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec0cea30 con 0x7f50ec0b9a10 2024-03-06T07:27:52.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.597+0000 7f50f3d63640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec0cf3c0 con 0x7f50ec15df00 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.597+0000 7f50f3d63640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ec0cf0e0 con 0x7f50ec0b95a0 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.597+0000 7f50f2300640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f50ec0b9a10 0x7f50ec0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40778/0 (socket says 172.21.15.27:40778) 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.597+0000 7f50f2300640 1 -- 172.21.15.27:0/893615314 learned_addr learned my addr 172.21.15.27:0/893615314 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.597+0000 7f50f12fe640 1 --2- 172.21.15.27:0/893615314 >> v2:172.21.15.69:3300/0 conn(0x7f50ec0b95a0 0x7f50ec0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.598+0000 7f50f0afd640 1 -- 172.21.15.27:0/893615314 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427551555 0 0) 0x7f50ec0cea30 con 0x7f50ec0b9a10 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.598+0000 7f50f0afd640 1 -- 172.21.15.27:0/893615314 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d8003520 con 0x7f50ec0b9a10 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.598+0000 7f50f0afd640 1 -- 172.21.15.27:0/893615314 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632415393 0 0) 0x7f50ec0cf3c0 con 0x7f50ec15df00 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.598+0000 7f50f0afd640 1 -- 172.21.15.27:0/893615314 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec0cea30 con 0x7f50ec15df00 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.598+0000 7f50f12fe640 1 -- 172.21.15.27:0/893615314 >> v1:172.21.15.69:6789/0 conn(0x7f50ec15df00 legacy=0x7f50ec0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.598+0000 7f50f12fe640 1 -- 172.21.15.27:0/893615314 >> v1:172.21.15.27:6789/0 conn(0x7f50ec0b9a10 legacy=0x7f50ec0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.598+0000 7f50f12fe640 1 -- 172.21.15.27:0/893615314 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0ce470 con 0x7f50ec0b95a0 2024-03-06T07:27:52.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f12fe640 1 --2- 172.21.15.27:0/893615314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0b95a0 0x7f50ec0b5700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0cf590 tx=0x7f50e802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=221424f0a8ba6546 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f0afd640 1 -- 172.21.15.27:0/893615314 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8034070 con 0x7f50ec0b95a0 2024-03-06T07:27:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f0afd640 1 -- 172.21.15.27:0/893615314 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e8030610 con 0x7f50ec0b95a0 2024-03-06T07:27:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f0afd640 1 -- 172.21.15.27:0/893615314 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e80309e0 con 0x7f50ec0b95a0 2024-03-06T07:27:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f3d63640 1 -- 172.21.15.27:0/893615314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0b95a0 msgr2=0x7f50ec0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f3d63640 1 --2- 172.21.15.27:0/893615314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0b95a0 0x7f50ec0b5700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0cf590 tx=0x7f50e802b960 comp rx=0 tx=0).stop 2024-03-06T07:27:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f3d63640 1 -- 172.21.15.27:0/893615314 shutdown_connections 2024-03-06T07:27:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f3d63640 1 --2- 172.21.15.27:0/893615314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0b95a0 0x7f50ec0b5700 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f3d63640 1 -- 172.21.15.27:0/893615314 >> 172.21.15.27:0/893615314 conn(0x7f50ec071960 msgr2=0x7f50ec071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f3d63640 1 -- 172.21.15.27:0/893615314 shutdown_connections 2024-03-06T07:27:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.599+0000 7f50f3d63640 1 -- 172.21.15.27:0/893615314 wait complete. 2024-03-06T07:27:52.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f3d63640 1 Processor -- start 2024-03-06T07:27:52.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f3d63640 1 -- start start 2024-03-06T07:27:52.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f3d63640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0b95a0 0x7f50ec1f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f3d63640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0b9a10 0x7f50ec15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f3d63640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec15df00 0x7f50ec0c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f3d63640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec0cedf0 con 0x7f50ec0b95a0 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f3d63640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec0cec70 con 0x7f50ec0b9a10 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f3d63640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec0b5c40 con 0x7f50ec15df00 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f1aff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0b95a0 0x7f50ec1f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f1aff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0b95a0 0x7f50ec1f5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52708/0 (socket says 172.21.15.27:52708) 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.601+0000 7f50f1aff640 1 -- 172.21.15.27:0/1763762199 learned_addr learned my addr 172.21.15.27:0/1763762199 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f12fe640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0b9a10 0x7f50ec15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f2300640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec15df00 0x7f50ec0c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f1aff640 1 -- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec15df00 msgr2=0x7f50ec0c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f1aff640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec15df00 0x7f50ec0c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f1aff640 1 -- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0b9a10 msgr2=0x7f50ec15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f1aff640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0b9a10 0x7f50ec15c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f1aff640 1 -- 172.21.15.27:0/1763762199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec161110 con 0x7f50ec0b95a0 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f2300640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec15df00 0x7f50ec0c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:52.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f1aff640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0b95a0 0x7f50ec1f5430 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00c430 tx=0x7f50dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50e27fc640 1 -- 172.21.15.27:0/1763762199 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc017070 con 0x7f50ec0b95a0 2024-03-06T07:27:52.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50e27fc640 1 -- 172.21.15.27:0/1763762199 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc0052f0 con 0x7f50ec0b95a0 2024-03-06T07:27:52.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ec0c9ff0 con 0x7f50ec0b95a0 2024-03-06T07:27:52.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50e27fc640 1 -- 172.21.15.27:0/1763762199 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc003830 con 0x7f50ec0b95a0 2024-03-06T07:27:52.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.602+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ec0ca5b0 con 0x7f50ec0b95a0 2024-03-06T07:27:52.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.603+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ec0b86a0 con 0x7f50ec0b95a0 2024-03-06T07:27:52.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.604+0000 7f50e27fc640 1 -- 172.21.15.27:0/1763762199 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50dc01a070 con 0x7f50ec0b95a0 2024-03-06T07:27:52.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.604+0000 7f50e27fc640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f50d006b4d0 0x7f50d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.604+0000 7f50e27fc640 1 -- 172.21.15.27:0/1763762199 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50dc0895e0 con 0x7f50ec0b95a0 2024-03-06T07:27:52.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.605+0000 7f50f12fe640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f50d006b4d0 0x7f50d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.605+0000 7f50f12fe640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f50d006b4d0 0x7f50d006d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f50e8036000 tx=0x7f50e803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.609+0000 7f50e27fc640 1 -- 172.21.15.27:0/1763762199 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50ec0b86a0 con 0x7f50ec0b95a0 2024-03-06T07:27:52.738 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:27:52.737+0000 7f42e05cd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:52.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.785+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50ec0d04c0 con 0x7f50d006b4d0 2024-03-06T07:27:52.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.792+0000 7f50e27fc640 1 -- 172.21.15.27:0/1763762199 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252043 (secure 0 0 0) 0x7f50ec0d04c0 con 0x7f50d006b4d0 2024-03-06T07:27:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:52.797 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f50d006b4d0 msgr2=0x7f50d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f50d006b4d0 0x7f50d006d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f50e8036000 tx=0x7f50e803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0b95a0 msgr2=0x7f50ec1f5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0b95a0 0x7f50ec1f5430 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00c430 tx=0x7f50dc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 shutdown_connections 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f50d006b4d0 0x7f50d006d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50ec15df00 0x7f50ec0c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50ec0b9a10 0x7f50ec15c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 --2- 172.21.15.27:0/1763762199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0b95a0 0x7f50ec1f5430 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 >> 172.21.15.27:0/1763762199 conn(0x7f50ec071960 msgr2=0x7f50ec0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 shutdown_connections 2024-03-06T07:27:52.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.795+0000 7f50f3d63640 1 -- 172.21.15.27:0/1763762199 wait complete. 2024-03-06T07:27:52.807 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-03-06T07:27:52.059894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260100,"kb_used_data":10612,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714620,"statfs":{"total":773094113280,"available":772827770880,"internally_reserved":0,"allocated":10866688,"data_stored":5285392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.301543"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801580+0000","last_change":"2024-03-06T07:27:48.795851+0000","last_active":"2024-03-06T07:27:48.801580+0000","last_peered":"2024-03-06T07:27:48.801580+0000","last_clean":"2024-03-06T07:27:48.801580+0000","last_became_active":"2024-03-06T07:27:48.795333+0000","last_became_peered":"2024-03-06T07:27:48.795333+0000","last_unstale":"2024-03-06T07:27:48.801580+0000","last_undegraded":"2024-03-06T07:27:48.801580+0000","last_fullsized":"2024-03-06T07:27:48.801580+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:52.539081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787100+0000","last_change":"2024-03-06T07:27:48.797653+0000","last_active":"2024-03-06T07:27:49.787100+0000","last_peered":"2024-03-06T07:27:49.787100+0000","last_clean":"2024-03-06T07:27:49.787100+0000","last_became_active":"2024-03-06T07:27:48.797295+0000","last_became_peered":"2024-03-06T07:27:48.797295+0000","last_unstale":"2024-03-06T07:27:49.787100+0000","last_undegraded":"2024-03-06T07:27:49.787100+0000","last_fullsized":"2024-03-06T07:27:49.787100+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:47.706960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802294+0000","last_change":"2024-03-06T07:27:48.798199+0000","last_active":"2024-03-06T07:27:48.802294+0000","last_peered":"2024-03-06T07:27:48.802294+0000","last_clean":"2024-03-06T07:27:48.802294+0000","last_became_active":"2024-03-06T07:27:48.797938+0000","last_became_peered":"2024-03-06T07:27:48.797938+0000","last_unstale":"2024-03-06T07:27:48.802294+0000","last_undegraded":"2024-03-06T07:27:48.802294+0000","last_fullsized":"2024-03-06T07:27:48.802294+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:28.145024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789870+0000","last_change":"2024-03-06T07:27:48.801803+0000","last_active":"2024-03-06T07:27:49.789870+0000","last_peered":"2024-03-06T07:27:49.789870+0000","last_clean":"2024-03-06T07:27:49.789870+0000","last_became_active":"2024-03-06T07:27:48.800374+0000","last_became_peered":"2024-03-06T07:27:48.800374+0000","last_unstale":"2024-03-06T07:27:49.789870+0000","last_undegraded":"2024-03-06T07:27:49.789870+0000","last_fullsized":"2024-03-06T07:27:49.789870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:33.743145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804429+0000","last_change":"2024-03-06T07:27:48.801900+0000","last_active":"2024-03-06T07:27:48.804429+0000","last_peered":"2024-03-06T07:27:48.804429+0000","last_clean":"2024-03-06T07:27:48.804429+0000","last_became_active":"2024-03-06T07:27:48.801632+0000","last_became_peered":"2024-03-06T07:27:48.801632+0000","last_unstale":"2024-03-06T07:27:48.804429+0000","last_undegraded":"2024-03-06T07:27:48.804429+0000","last_fullsized":"2024-03-06T07:27:48.804429+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:59.992386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.803734+0000","last_change":"2024-03-06T07:27:48.798657+0000","last_active":"2024-03-06T07:27:48.803734+0000","last_peered":"2024-03-06T07:27:48.803734+0000","last_clean":"2024-03-06T07:27:48.803734+0000","last_became_active":"2024-03-06T07:27:48.798282+0000","last_became_peered":"2024-03-06T07:27:48.798282+0000","last_unstale":"2024-03-06T07:27:48.803734+0000","last_undegraded":"2024-03-06T07:27:48.803734+0000","last_fullsized":"2024-03-06T07:27:48.803734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:49.074891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.803411+0000","last_change":"2024-03-06T07:27:48.796710+0000","last_active":"2024-03-06T07:27:48.803411+0000","last_peered":"2024-03-06T07:27:48.803411+0000","last_clean":"2024-03-06T07:27:48.803411+0000","last_became_active":"2024-03-06T07:27:48.796108+0000","last_became_peered":"2024-03-06T07:27:48.796108+0000","last_unstale":"2024-03-06T07:27:48.803411+0000","last_undegraded":"2024-03-06T07:27:48.803411+0000","last_fullsized":"2024-03-06T07:27:48.803411+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:52.728980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786611+0000","last_change":"2024-03-06T07:27:48.796169+0000","last_active":"2024-03-06T07:27:49.786611+0000","last_peered":"2024-03-06T07:27:49.786611+0000","last_clean":"2024-03-06T07:27:49.786611+0000","last_became_active":"2024-03-06T07:27:48.795915+0000","last_became_peered":"2024-03-06T07:27:48.795915+0000","last_unstale":"2024-03-06T07:27:49.786611+0000","last_undegraded":"2024-03-06T07:27:49.786611+0000","last_fullsized":"2024-03-06T07:27:49.786611+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:19.744943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790560+0000","last_change":"2024-03-06T07:27:48.793854+0000","last_active":"2024-03-06T07:27:49.790560+0000","last_peered":"2024-03-06T07:27:49.790560+0000","last_clean":"2024-03-06T07:27:49.790560+0000","last_became_active":"2024-03-06T07:27:48.793564+0000","last_became_peered":"2024-03-06T07:27:48.793564+0000","last_unstale":"2024-03-06T07:27:49.790560+0000","last_undegraded":"2024-03-06T07:27:49.790560+0000","last_fullsized":"2024-03-06T07:27:49.790560+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:52.728980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.797389+0000","last_change":"2024-03-06T07:27:48.794510+0000","last_active":"2024-03-06T07:27:48.797389+0000","last_peered":"2024-03-06T07:27:48.797389+0000","last_clean":"2024-03-06T07:27:48.797389+0000","last_became_active":"2024-03-06T07:27:48.794221+0000","last_became_peered":"2024-03-06T07:27:48.794221+0000","last_unstale":"2024-03-06T07:27:48.797389+0000","last_undegraded":"2024-03-06T07:27:48.797389+0000","last_fullsized":"2024-03-06T07:27:48.797389+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:55.210682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808399+0000","last_change":"2024-03-06T07:27:48.804541+0000","last_active":"2024-03-06T07:27:48.808399+0000","last_peered":"2024-03-06T07:27:48.808399+0000","last_clean":"2024-03-06T07:27:48.808399+0000","last_became_active":"2024-03-06T07:27:48.804246+0000","last_became_peered":"2024-03-06T07:27:48.804246+0000","last_unstale":"2024-03-06T07:27:48.808399+0000","last_undegraded":"2024-03-06T07:27:48.808399+0000","last_fullsized":"2024-03-06T07:27:48.808399+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:29.734536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790348+0000","last_change":"2024-03-06T07:27:48.790705+0000","last_active":"2024-03-06T07:27:49.790348+0000","last_peered":"2024-03-06T07:27:49.790348+0000","last_clean":"2024-03-06T07:27:49.790348+0000","last_became_active":"2024-03-06T07:27:48.790431+0000","last_became_peered":"2024-03-06T07:27:48.790431+0000","last_unstale":"2024-03-06T07:27:49.790348+0000","last_undegraded":"2024-03-06T07:27:49.790348+0000","last_fullsized":"2024-03-06T07:27:49.790348+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:50.559585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789754+0000","last_change":"2024-03-06T07:27:48.792112+0000","last_active":"2024-03-06T07:27:49.789754+0000","last_peered":"2024-03-06T07:27:49.789754+0000","last_clean":"2024-03-06T07:27:49.789754+0000","last_became_active":"2024-03-06T07:27:48.791847+0000","last_became_peered":"2024-03-06T07:27:48.791847+0000","last_unstale":"2024-03-06T07:27:49.789754+0000","last_undegraded":"2024-03-06T07:27:49.789754+0000","last_fullsized":"2024-03-06T07:27:49.789754+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:53.226987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802433+0000","last_change":"2024-03-06T07:27:48.799504+0000","last_active":"2024-03-06T07:27:48.802433+0000","last_peered":"2024-03-06T07:27:48.802433+0000","last_clean":"2024-03-06T07:27:48.802433+0000","last_became_active":"2024-03-06T07:27:48.798944+0000","last_became_peered":"2024-03-06T07:27:48.798944+0000","last_unstale":"2024-03-06T07:27:48.802433+0000","last_undegraded":"2024-03-06T07:27:48.802433+0000","last_fullsized":"2024-03-06T07:27:48.802433+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:43.167479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789509+0000","last_change":"2024-03-06T07:27:48.801372+0000","last_active":"2024-03-06T07:27:49.789509+0000","last_peered":"2024-03-06T07:27:49.789509+0000","last_clean":"2024-03-06T07:27:49.789509+0000","last_became_active":"2024-03-06T07:27:48.800890+0000","last_became_peered":"2024-03-06T07:27:48.800890+0000","last_unstale":"2024-03-06T07:27:49.789509+0000","last_undegraded":"2024-03-06T07:27:49.789509+0000","last_fullsized":"2024-03-06T07:27:49.789509+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:01.705383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808351+0000","last_change":"2024-03-06T07:27:48.802940+0000","last_active":"2024-03-06T07:27:48.808351+0000","last_peered":"2024-03-06T07:27:48.808351+0000","last_clean":"2024-03-06T07:27:48.808351+0000","last_became_active":"2024-03-06T07:27:48.802679+0000","last_became_peered":"2024-03-06T07:27:48.802679+0000","last_unstale":"2024-03-06T07:27:48.808351+0000","last_undegraded":"2024-03-06T07:27:48.808351+0000","last_fullsized":"2024-03-06T07:27:48.808351+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:49.006905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787075+0000","last_change":"2024-03-06T07:27:48.793735+0000","last_active":"2024-03-06T07:27:49.787075+0000","last_peered":"2024-03-06T07:27:49.787075+0000","last_clean":"2024-03-06T07:27:49.787075+0000","last_became_active":"2024-03-06T07:27:48.793063+0000","last_became_peered":"2024-03-06T07:27:48.793063+0000","last_unstale":"2024-03-06T07:27:49.787075+0000","last_undegraded":"2024-03-06T07:27:49.787075+0000","last_fullsized":"2024-03-06T07:27:49.787075+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:02.703373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789777+0000","last_change":"2024-03-06T07:27:48.800359+0000","last_active":"2024-03-06T07:27:49.789777+0000","last_peered":"2024-03-06T07:27:49.789777+0000","last_clean":"2024-03-06T07:27:49.789777+0000","last_became_active":"2024-03-06T07:27:48.800062+0000","last_became_peered":"2024-03-06T07:27:48.800062+0000","last_unstale":"2024-03-06T07:27:49.789777+0000","last_undegraded":"2024-03-06T07:27:49.789777+0000","last_fullsized":"2024-03-06T07:27:49.789777+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:49.962734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798268+0000","last_change":"2024-03-06T07:27:48.797350+0000","last_active":"2024-03-06T07:27:48.798268+0000","last_peered":"2024-03-06T07:27:48.798268+0000","last_clean":"2024-03-06T07:27:48.798268+0000","last_became_active":"2024-03-06T07:27:48.793707+0000","last_became_peered":"2024-03-06T07:27:48.793707+0000","last_unstale":"2024-03-06T07:27:48.798268+0000","last_undegraded":"2024-03-06T07:27:48.798268+0000","last_fullsized":"2024-03-06T07:27:48.798268+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:35.618564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790486+0000","last_change":"2024-03-06T07:27:48.797707+0000","last_active":"2024-03-06T07:27:49.790486+0000","last_peered":"2024-03-06T07:27:49.790486+0000","last_clean":"2024-03-06T07:27:49.790486+0000","last_became_active":"2024-03-06T07:27:48.793867+0000","last_became_peered":"2024-03-06T07:27:48.793867+0000","last_unstale":"2024-03-06T07:27:49.790486+0000","last_undegraded":"2024-03-06T07:27:49.790486+0000","last_fullsized":"2024-03-06T07:27:49.790486+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:59.992386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802258+0000","last_change":"2024-03-06T07:27:48.799007+0000","last_active":"2024-03-06T07:27:48.802258+0000","last_peered":"2024-03-06T07:27:48.802258+0000","last_clean":"2024-03-06T07:27:48.802258+0000","last_became_active":"2024-03-06T07:27:48.798738+0000","last_became_peered":"2024-03-06T07:27:48.798738+0000","last_unstale":"2024-03-06T07:27:48.802258+0000","last_undegraded":"2024-03-06T07:27:48.802258+0000","last_fullsized":"2024-03-06T07:27:48.802258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:43.019435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786588+0000","last_change":"2024-03-06T07:27:48.796914+0000","last_active":"2024-03-06T07:27:49.786588+0000","last_peered":"2024-03-06T07:27:49.786588+0000","last_clean":"2024-03-06T07:27:49.786588+0000","last_became_active":"2024-03-06T07:27:48.796630+0000","last_became_peered":"2024-03-06T07:27:48.796630+0000","last_unstale":"2024-03-06T07:27:49.786588+0000","last_undegraded":"2024-03-06T07:27:49.786588+0000","last_fullsized":"2024-03-06T07:27:49.786588+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:33.743145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786926+0000","last_change":"2024-03-06T07:27:48.793763+0000","last_active":"2024-03-06T07:27:49.786926+0000","last_peered":"2024-03-06T07:27:49.786926+0000","last_clean":"2024-03-06T07:27:49.786926+0000","last_became_active":"2024-03-06T07:27:48.793517+0000","last_became_peered":"2024-03-06T07:27:48.793517+0000","last_unstale":"2024-03-06T07:27:49.786926+0000","last_undegraded":"2024-03-06T07:27:49.786926+0000","last_fullsized":"2024-03-06T07:27:49.786926+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:14.670440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.805508+0000","last_change":"2024-03-06T07:27:48.802332+0000","last_active":"2024-03-06T07:27:48.805508+0000","last_peered":"2024-03-06T07:27:48.805508+0000","last_clean":"2024-03-06T07:27:48.805508+0000","last_became_active":"2024-03-06T07:27:48.802032+0000","last_became_peered":"2024-03-06T07:27:48.802032+0000","last_unstale":"2024-03-06T07:27:48.805508+0000","last_undegraded":"2024-03-06T07:27:48.805508+0000","last_fullsized":"2024-03-06T07:27:48.805508+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:18.344411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798756+0000","last_change":"2024-03-06T07:27:48.793638+0000","last_active":"2024-03-06T07:27:48.798756+0000","last_peered":"2024-03-06T07:27:48.798756+0000","last_clean":"2024-03-06T07:27:48.798756+0000","last_became_active":"2024-03-06T07:27:48.793350+0000","last_became_peered":"2024-03-06T07:27:48.793350+0000","last_unstale":"2024-03-06T07:27:48.798756+0000","last_undegraded":"2024-03-06T07:27:48.798756+0000","last_fullsized":"2024-03-06T07:27:48.798756+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:59.800540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786828+0000","last_change":"2024-03-06T07:27:48.788989+0000","last_active":"2024-03-06T07:27:49.786828+0000","last_peered":"2024-03-06T07:27:49.786828+0000","last_clean":"2024-03-06T07:27:49.786828+0000","last_became_active":"2024-03-06T07:27:48.788710+0000","last_became_peered":"2024-03-06T07:27:48.788710+0000","last_unstale":"2024-03-06T07:27:49.786828+0000","last_undegraded":"2024-03-06T07:27:49.786828+0000","last_fullsized":"2024-03-06T07:27:49.786828+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:11.990008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.805489+0000","last_change":"2024-03-06T07:27:48.801347+0000","last_active":"2024-03-06T07:27:48.805489+0000","last_peered":"2024-03-06T07:27:48.805489+0000","last_clean":"2024-03-06T07:27:48.805489+0000","last_became_active":"2024-03-06T07:27:48.800813+0000","last_became_peered":"2024-03-06T07:27:48.800813+0000","last_unstale":"2024-03-06T07:27:48.805489+0000","last_undegraded":"2024-03-06T07:27:48.805489+0000","last_fullsized":"2024-03-06T07:27:48.805489+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:59.793165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804661+0000","last_change":"2024-03-06T07:27:48.802751+0000","last_active":"2024-03-06T07:27:48.804661+0000","last_peered":"2024-03-06T07:27:48.804661+0000","last_clean":"2024-03-06T07:27:48.804661+0000","last_became_active":"2024-03-06T07:27:48.802405+0000","last_became_peered":"2024-03-06T07:27:48.802405+0000","last_unstale":"2024-03-06T07:27:48.804661+0000","last_undegraded":"2024-03-06T07:27:48.804661+0000","last_fullsized":"2024-03-06T07:27:48.804661+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:52.728980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786823+0000","last_change":"2024-03-06T07:27:48.789405+0000","last_active":"2024-03-06T07:27:49.786823+0000","last_peered":"2024-03-06T07:27:49.786823+0000","last_clean":"2024-03-06T07:27:49.786823+0000","last_became_active":"2024-03-06T07:27:48.788692+0000","last_became_peered":"2024-03-06T07:27:48.788692+0000","last_unstale":"2024-03-06T07:27:49.786823+0000","last_undegraded":"2024-03-06T07:27:49.786823+0000","last_fullsized":"2024-03-06T07:27:49.786823+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:18.882652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802386+0000","last_change":"2024-03-06T07:27:48.799277+0000","last_active":"2024-03-06T07:27:48.802386+0000","last_peered":"2024-03-06T07:27:48.802386+0000","last_clean":"2024-03-06T07:27:48.802386+0000","last_became_active":"2024-03-06T07:27:48.798582+0000","last_became_peered":"2024-03-06T07:27:48.798582+0000","last_unstale":"2024-03-06T07:27:48.802386+0000","last_undegraded":"2024-03-06T07:27:48.802386+0000","last_fullsized":"2024-03-06T07:27:48.802386+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:22.493871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789558+0000","last_change":"2024-03-06T07:27:48.801175+0000","last_active":"2024-03-06T07:27:49.789558+0000","last_peered":"2024-03-06T07:27:49.789558+0000","last_clean":"2024-03-06T07:27:49.789558+0000","last_became_active":"2024-03-06T07:27:48.800614+0000","last_became_peered":"2024-03-06T07:27:48.800614+0000","last_unstale":"2024-03-06T07:27:49.789558+0000","last_undegraded":"2024-03-06T07:27:49.789558+0000","last_fullsized":"2024-03-06T07:27:49.789558+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:42.003832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.790700+0000","last_change":"2024-03-06T07:27:48.788164+0000","last_active":"2024-03-06T07:27:48.790700+0000","last_peered":"2024-03-06T07:27:48.790700+0000","last_clean":"2024-03-06T07:27:48.790700+0000","last_became_active":"2024-03-06T07:27:48.787907+0000","last_became_peered":"2024-03-06T07:27:48.787907+0000","last_unstale":"2024-03-06T07:27:48.790700+0000","last_undegraded":"2024-03-06T07:27:48.790700+0000","last_fullsized":"2024-03-06T07:27:48.790700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:29.690767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798628+0000","last_change":"2024-03-06T07:27:48.798020+0000","last_active":"2024-03-06T07:27:48.798628+0000","last_peered":"2024-03-06T07:27:48.798628+0000","last_clean":"2024-03-06T07:27:48.798628+0000","last_became_active":"2024-03-06T07:27:48.797674+0000","last_became_peered":"2024-03-06T07:27:48.797674+0000","last_unstale":"2024-03-06T07:27:48.798628+0000","last_undegraded":"2024-03-06T07:27:48.798628+0000","last_fullsized":"2024-03-06T07:27:48.798628+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:14.039295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"97'4","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786964+0000","last_change":"2024-03-06T07:27:47.777623+0000","last_active":"2024-03-06T07:27:49.786964+0000","last_peered":"2024-03-06T07:27:49.786964+0000","last_clean":"2024-03-06T07:27:49.786964+0000","last_became_active":"2024-03-06T07:27:47.777293+0000","last_became_peered":"2024-03-06T07:27:47.777293+0000","last_unstale":"2024-03-06T07:27:49.786964+0000","last_undegraded":"2024-03-06T07:27:49.786964+0000","last_fullsized":"2024-03-06T07:27:49.786964+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.070234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789934+0000","last_change":"2024-03-06T07:27:48.795542+0000","last_active":"2024-03-06T07:27:49.789934+0000","last_peered":"2024-03-06T07:27:49.789934+0000","last_clean":"2024-03-06T07:27:49.789934+0000","last_became_active":"2024-03-06T07:27:48.795254+0000","last_became_peered":"2024-03-06T07:27:48.795254+0000","last_unstale":"2024-03-06T07:27:49.789934+0000","last_undegraded":"2024-03-06T07:27:49.789934+0000","last_fullsized":"2024-03-06T07:27:49.789934+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:19.744943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801986+0000","last_change":"2024-03-06T07:27:47.784471+0000","last_active":"2024-03-06T07:27:48.801986+0000","last_peered":"2024-03-06T07:27:48.801986+0000","last_clean":"2024-03-06T07:27:48.801986+0000","last_became_active":"2024-03-06T07:27:47.784211+0000","last_became_peered":"2024-03-06T07:27:47.784211+0000","last_unstale":"2024-03-06T07:27:48.801986+0000","last_undegraded":"2024-03-06T07:27:48.801986+0000","last_fullsized":"2024-03-06T07:27:48.801986+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:11.336675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798378+0000","last_change":"2024-03-06T07:27:48.790902+0000","last_active":"2024-03-06T07:27:48.798378+0000","last_peered":"2024-03-06T07:27:48.798378+0000","last_clean":"2024-03-06T07:27:48.798378+0000","last_became_active":"2024-03-06T07:27:48.790480+0000","last_became_peered":"2024-03-06T07:27:48.790480+0000","last_unstale":"2024-03-06T07:27:48.798378+0000","last_undegraded":"2024-03-06T07:27:48.798378+0000","last_fullsized":"2024-03-06T07:27:48.798378+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:43.167479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786868+0000","last_change":"2024-03-06T07:27:47.783376+0000","last_active":"2024-03-06T07:27:49.786868+0000","last_peered":"2024-03-06T07:27:49.786868+0000","last_clean":"2024-03-06T07:27:49.786868+0000","last_became_active":"2024-03-06T07:27:47.783123+0000","last_became_peered":"2024-03-06T07:27:47.783123+0000","last_unstale":"2024-03-06T07:27:49.786868+0000","last_undegraded":"2024-03-06T07:27:49.786868+0000","last_fullsized":"2024-03-06T07:27:49.786868+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:58.987729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790580+0000","last_change":"2024-03-06T07:27:48.798612+0000","last_active":"2024-03-06T07:27:49.790580+0000","last_peered":"2024-03-06T07:27:49.790580+0000","last_clean":"2024-03-06T07:27:49.790580+0000","last_became_active":"2024-03-06T07:27:48.797841+0000","last_became_peered":"2024-03-06T07:27:48.797841+0000","last_unstale":"2024-03-06T07:27:49.790580+0000","last_undegraded":"2024-03-06T07:27:49.790580+0000","last_fullsized":"2024-03-06T07:27:49.790580+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:49.074891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786654+0000","last_change":"2024-03-06T07:27:47.780478+0000","last_active":"2024-03-06T07:27:49.786654+0000","last_peered":"2024-03-06T07:27:49.786654+0000","last_clean":"2024-03-06T07:27:49.786654+0000","last_became_active":"2024-03-06T07:27:47.780187+0000","last_became_peered":"2024-03-06T07:27:47.780187+0000","last_unstale":"2024-03-06T07:27:49.786654+0000","last_undegraded":"2024-03-06T07:27:49.786654+0000","last_fullsized":"2024-03-06T07:27:49.786654+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:54.206025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796937+0000","last_change":"2024-03-06T07:27:48.792623+0000","last_active":"2024-03-06T07:27:48.796937+0000","last_peered":"2024-03-06T07:27:48.796937+0000","last_clean":"2024-03-06T07:27:48.796937+0000","last_became_active":"2024-03-06T07:27:48.792379+0000","last_became_peered":"2024-03-06T07:27:48.792379+0000","last_unstale":"2024-03-06T07:27:48.796937+0000","last_undegraded":"2024-03-06T07:27:48.796937+0000","last_fullsized":"2024-03-06T07:27:48.796937+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:26.004240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.789991+0000","last_change":"2024-03-06T07:27:47.776984+0000","last_active":"2024-03-06T07:27:48.789991+0000","last_peered":"2024-03-06T07:27:48.789991+0000","last_clean":"2024-03-06T07:27:48.789991+0000","last_became_active":"2024-03-06T07:27:47.776679+0000","last_became_peered":"2024-03-06T07:27:47.776679+0000","last_unstale":"2024-03-06T07:27:48.789991+0000","last_undegraded":"2024-03-06T07:27:48.789991+0000","last_fullsized":"2024-03-06T07:27:48.789991+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.219201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808397+0000","last_change":"2024-03-06T07:27:48.804068+0000","last_active":"2024-03-06T07:27:48.808397+0000","last_peered":"2024-03-06T07:27:48.808397+0000","last_clean":"2024-03-06T07:27:48.808397+0000","last_became_active":"2024-03-06T07:27:48.803431+0000","last_became_peered":"2024-03-06T07:27:48.803431+0000","last_unstale":"2024-03-06T07:27:48.808397+0000","last_undegraded":"2024-03-06T07:27:48.808397+0000","last_fullsized":"2024-03-06T07:27:48.808397+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:44.063497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786697+0000","last_change":"2024-03-06T07:27:47.780233+0000","last_active":"2024-03-06T07:27:49.786697+0000","last_peered":"2024-03-06T07:27:49.786697+0000","last_clean":"2024-03-06T07:27:49.786697+0000","last_became_active":"2024-03-06T07:27:47.779968+0000","last_became_peered":"2024-03-06T07:27:47.779968+0000","last_unstale":"2024-03-06T07:27:49.786697+0000","last_undegraded":"2024-03-06T07:27:49.786697+0000","last_fullsized":"2024-03-06T07:27:49.786697+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:11.336675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789472+0000","last_change":"2024-03-06T07:27:48.796209+0000","last_active":"2024-03-06T07:27:49.789472+0000","last_peered":"2024-03-06T07:27:49.789472+0000","last_clean":"2024-03-06T07:27:49.789472+0000","last_became_active":"2024-03-06T07:27:48.795957+0000","last_became_peered":"2024-03-06T07:27:48.795957+0000","last_unstale":"2024-03-06T07:27:49.789472+0000","last_undegraded":"2024-03-06T07:27:49.789472+0000","last_fullsized":"2024-03-06T07:27:49.789472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:10.959320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.785904+0000","last_change":"2024-03-06T07:27:47.780756+0000","last_active":"2024-03-06T07:27:48.785904+0000","last_peered":"2024-03-06T07:27:48.785904+0000","last_clean":"2024-03-06T07:27:48.785904+0000","last_became_active":"2024-03-06T07:27:47.780417+0000","last_became_peered":"2024-03-06T07:27:47.780417+0000","last_unstale":"2024-03-06T07:27:48.785904+0000","last_undegraded":"2024-03-06T07:27:48.785904+0000","last_fullsized":"2024-03-06T07:27:48.785904+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:03.533825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789925+0000","last_change":"2024-03-06T07:27:48.798440+0000","last_active":"2024-03-06T07:27:49.789925+0000","last_peered":"2024-03-06T07:27:49.789925+0000","last_clean":"2024-03-06T07:27:49.789925+0000","last_became_active":"2024-03-06T07:27:48.798093+0000","last_became_peered":"2024-03-06T07:27:48.798093+0000","last_unstale":"2024-03-06T07:27:49.789925+0000","last_undegraded":"2024-03-06T07:27:49.789925+0000","last_fullsized":"2024-03-06T07:27:49.789925+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:55.210682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.780191+0000","last_change":"2024-03-06T07:27:47.777912+0000","last_active":"2024-03-06T07:27:48.780191+0000","last_peered":"2024-03-06T07:27:48.780191+0000","last_clean":"2024-03-06T07:27:48.780191+0000","last_became_active":"2024-03-06T07:27:47.777610+0000","last_became_peered":"2024-03-06T07:27:47.777610+0000","last_unstale":"2024-03-06T07:27:48.780191+0000","last_undegraded":"2024-03-06T07:27:48.780191+0000","last_fullsized":"2024-03-06T07:27:48.780191+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:42.014777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786976+0000","last_change":"2024-03-06T07:27:48.792774+0000","last_active":"2024-03-06T07:27:49.786976+0000","last_peered":"2024-03-06T07:27:49.786976+0000","last_clean":"2024-03-06T07:27:49.786976+0000","last_became_active":"2024-03-06T07:27:48.792074+0000","last_became_peered":"2024-03-06T07:27:48.792074+0000","last_unstale":"2024-03-06T07:27:49.786976+0000","last_undegraded":"2024-03-06T07:27:49.786976+0000","last_fullsized":"2024-03-06T07:27:49.786976+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:35.663777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.781439+0000","last_change":"2024-03-06T07:27:47.782069+0000","last_active":"2024-03-06T07:27:48.781439+0000","last_peered":"2024-03-06T07:27:48.781439+0000","last_clean":"2024-03-06T07:27:48.781439+0000","last_became_active":"2024-03-06T07:27:47.781790+0000","last_became_peered":"2024-03-06T07:27:47.781790+0000","last_unstale":"2024-03-06T07:27:48.781439+0000","last_undegraded":"2024-03-06T07:27:48.781439+0000","last_fullsized":"2024-03-06T07:27:48.781439+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.489214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790765+0000","last_change":"2024-03-06T07:27:48.796717+0000","last_active":"2024-03-06T07:27:49.790765+0000","last_peered":"2024-03-06T07:27:49.790765+0000","last_clean":"2024-03-06T07:27:49.790765+0000","last_became_active":"2024-03-06T07:27:48.796455+0000","last_became_peered":"2024-03-06T07:27:48.796455+0000","last_unstale":"2024-03-06T07:27:49.790765+0000","last_undegraded":"2024-03-06T07:27:49.790765+0000","last_fullsized":"2024-03-06T07:27:49.790765+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:59.793165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789831+0000","last_change":"2024-03-06T07:27:47.780974+0000","last_active":"2024-03-06T07:27:49.789831+0000","last_peered":"2024-03-06T07:27:49.789831+0000","last_clean":"2024-03-06T07:27:49.789831+0000","last_became_active":"2024-03-06T07:27:47.780701+0000","last_became_peered":"2024-03-06T07:27:47.780701+0000","last_unstale":"2024-03-06T07:27:49.789831+0000","last_undegraded":"2024-03-06T07:27:49.789831+0000","last_fullsized":"2024-03-06T07:27:49.789831+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:58.987729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796328+0000","last_change":"2024-03-06T07:27:48.793759+0000","last_active":"2024-03-06T07:27:48.796328+0000","last_peered":"2024-03-06T07:27:48.796328+0000","last_clean":"2024-03-06T07:27:48.796328+0000","last_became_active":"2024-03-06T07:27:48.793468+0000","last_became_peered":"2024-03-06T07:27:48.793468+0000","last_unstale":"2024-03-06T07:27:48.796328+0000","last_undegraded":"2024-03-06T07:27:48.796328+0000","last_fullsized":"2024-03-06T07:27:48.796328+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:13.207256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"97'3","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804150+0000","last_change":"2024-03-06T07:27:47.778360+0000","last_active":"2024-03-06T07:27:48.804150+0000","last_peered":"2024-03-06T07:27:48.804150+0000","last_clean":"2024-03-06T07:27:48.804150+0000","last_became_active":"2024-03-06T07:27:47.778069+0000","last_became_peered":"2024-03-06T07:27:47.778069+0000","last_unstale":"2024-03-06T07:27:48.804150+0000","last_undegraded":"2024-03-06T07:27:48.804150+0000","last_fullsized":"2024-03-06T07:27:48.804150+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:58.987729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808382+0000","last_change":"2024-03-06T07:27:48.803862+0000","last_active":"2024-03-06T07:27:48.808382+0000","last_peered":"2024-03-06T07:27:48.808382+0000","last_clean":"2024-03-06T07:27:48.808382+0000","last_became_active":"2024-03-06T07:27:48.803196+0000","last_became_peered":"2024-03-06T07:27:48.803196+0000","last_unstale":"2024-03-06T07:27:48.808382+0000","last_undegraded":"2024-03-06T07:27:48.808382+0000","last_fullsized":"2024-03-06T07:27:48.808382+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:12.157853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"97'6","reported_seq":20,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790470+0000","last_change":"2024-03-06T07:27:47.775011+0000","last_active":"2024-03-06T07:27:49.790470+0000","last_peered":"2024-03-06T07:27:49.790470+0000","last_clean":"2024-03-06T07:27:49.790470+0000","last_became_active":"2024-03-06T07:27:47.774698+0000","last_became_peered":"2024-03-06T07:27:47.774698+0000","last_unstale":"2024-03-06T07:27:49.790470+0000","last_undegraded":"2024-03-06T07:27:49.790470+0000","last_fullsized":"2024-03-06T07:27:49.790470+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:13.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789604+0000","last_change":"2024-03-06T07:27:48.796347+0000","last_active":"2024-03-06T07:27:49.789604+0000","last_peered":"2024-03-06T07:27:49.789604+0000","last_clean":"2024-03-06T07:27:49.789604+0000","last_became_active":"2024-03-06T07:27:48.796096+0000","last_became_peered":"2024-03-06T07:27:48.796096+0000","last_unstale":"2024-03-06T07:27:49.789604+0000","last_undegraded":"2024-03-06T07:27:49.789604+0000","last_fullsized":"2024-03-06T07:27:49.789604+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:19.208168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.793547+0000","last_change":"2024-03-06T07:27:47.774075+0000","last_active":"2024-03-06T07:27:48.793547+0000","last_peered":"2024-03-06T07:27:48.793547+0000","last_clean":"2024-03-06T07:27:48.793547+0000","last_became_active":"2024-03-06T07:27:47.773374+0000","last_became_peered":"2024-03-06T07:27:47.773374+0000","last_unstale":"2024-03-06T07:27:48.793547+0000","last_undegraded":"2024-03-06T07:27:48.793547+0000","last_fullsized":"2024-03-06T07:27:48.793547+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:23.403860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801405+0000","last_change":"2024-03-06T07:27:48.794407+0000","last_active":"2024-03-06T07:27:48.801405+0000","last_peered":"2024-03-06T07:27:48.801405+0000","last_clean":"2024-03-06T07:27:48.801405+0000","last_became_active":"2024-03-06T07:27:48.794139+0000","last_became_peered":"2024-03-06T07:27:48.794139+0000","last_unstale":"2024-03-06T07:27:48.801405+0000","last_undegraded":"2024-03-06T07:27:48.801405+0000","last_fullsized":"2024-03-06T07:27:48.801405+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:46.891453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.784249+0000","last_change":"2024-03-06T07:27:47.775045+0000","last_active":"2024-03-06T07:27:48.784249+0000","last_peered":"2024-03-06T07:27:48.784249+0000","last_clean":"2024-03-06T07:27:48.784249+0000","last_became_active":"2024-03-06T07:27:47.774751+0000","last_became_peered":"2024-03-06T07:27:47.774751+0000","last_unstale":"2024-03-06T07:27:48.784249+0000","last_undegraded":"2024-03-06T07:27:48.784249+0000","last_fullsized":"2024-03-06T07:27:48.784249+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:53.038808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789911+0000","last_change":"2024-03-06T07:27:48.795541+0000","last_active":"2024-03-06T07:27:49.789911+0000","last_peered":"2024-03-06T07:27:49.789911+0000","last_clean":"2024-03-06T07:27:49.789911+0000","last_became_active":"2024-03-06T07:27:48.793670+0000","last_became_peered":"2024-03-06T07:27:48.793670+0000","last_unstale":"2024-03-06T07:27:49.789911+0000","last_undegraded":"2024-03-06T07:27:49.789911+0000","last_fullsized":"2024-03-06T07:27:49.789911+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:50.898201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789945+0000","last_change":"2024-03-06T07:27:47.774018+0000","last_active":"2024-03-06T07:27:49.789945+0000","last_peered":"2024-03-06T07:27:49.789945+0000","last_clean":"2024-03-06T07:27:49.789945+0000","last_became_active":"2024-03-06T07:27:47.773537+0000","last_became_peered":"2024-03-06T07:27:47.773537+0000","last_unstale":"2024-03-06T07:27:49.789945+0000","last_undegraded":"2024-03-06T07:27:49.789945+0000","last_fullsized":"2024-03-06T07:27:49.789945+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:54.711622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804475+0000","last_change":"2024-03-06T07:27:48.791020+0000","last_active":"2024-03-06T07:27:48.804475+0000","last_peered":"2024-03-06T07:27:48.804475+0000","last_clean":"2024-03-06T07:27:48.804475+0000","last_became_active":"2024-03-06T07:27:48.790755+0000","last_became_peered":"2024-03-06T07:27:48.790755+0000","last_unstale":"2024-03-06T07:27:48.804475+0000","last_undegraded":"2024-03-06T07:27:48.804475+0000","last_fullsized":"2024-03-06T07:27:48.804475+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:08.718352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"97'8","reported_seq":20,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.792334+0000","last_change":"2024-03-06T07:27:47.779087+0000","last_active":"2024-03-06T07:27:48.792334+0000","last_peered":"2024-03-06T07:27:48.792334+0000","last_clean":"2024-03-06T07:27:48.792334+0000","last_became_active":"2024-03-06T07:27:47.778765+0000","last_became_peered":"2024-03-06T07:27:47.778765+0000","last_unstale":"2024-03-06T07:27:48.792334+0000","last_undegraded":"2024-03-06T07:27:48.792334+0000","last_fullsized":"2024-03-06T07:27:48.792334+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:13.034637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798779+0000","last_change":"2024-03-06T07:27:48.795848+0000","last_active":"2024-03-06T07:27:48.798779+0000","last_peered":"2024-03-06T07:27:48.798779+0000","last_clean":"2024-03-06T07:27:48.798779+0000","last_became_active":"2024-03-06T07:27:48.795500+0000","last_became_peered":"2024-03-06T07:27:48.795500+0000","last_unstale":"2024-03-06T07:27:48.798779+0000","last_undegraded":"2024-03-06T07:27:48.798779+0000","last_fullsized":"2024-03-06T07:27:48.798779+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:50.559585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790488+0000","last_change":"2024-03-06T07:27:47.778837+0000","last_active":"2024-03-06T07:27:49.790488+0000","last_peered":"2024-03-06T07:27:49.790488+0000","last_clean":"2024-03-06T07:27:49.790488+0000","last_became_active":"2024-03-06T07:27:47.777861+0000","last_became_peered":"2024-03-06T07:27:47.777861+0000","last_unstale":"2024-03-06T07:27:49.790488+0000","last_undegraded":"2024-03-06T07:27:49.790488+0000","last_fullsized":"2024-03-06T07:27:49.790488+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:01.287677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.797654+0000","last_change":"2024-03-06T07:27:48.795117+0000","last_active":"2024-03-06T07:27:48.797654+0000","last_peered":"2024-03-06T07:27:48.797654+0000","last_clean":"2024-03-06T07:27:48.797654+0000","last_became_active":"2024-03-06T07:27:48.794821+0000","last_became_peered":"2024-03-06T07:27:48.794821+0000","last_unstale":"2024-03-06T07:27:48.797654+0000","last_undegraded":"2024-03-06T07:27:48.797654+0000","last_fullsized":"2024-03-06T07:27:48.797654+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:42.682853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796657+0000","last_change":"2024-03-06T07:27:47.777867+0000","last_active":"2024-03-06T07:27:48.796657+0000","last_peered":"2024-03-06T07:27:48.796657+0000","last_clean":"2024-03-06T07:27:48.796657+0000","last_became_active":"2024-03-06T07:27:47.777524+0000","last_became_peered":"2024-03-06T07:27:47.777524+0000","last_unstale":"2024-03-06T07:27:48.796657+0000","last_undegraded":"2024-03-06T07:27:48.796657+0000","last_fullsized":"2024-03-06T07:27:48.796657+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:37.466064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808417+0000","last_change":"2024-03-06T07:27:48.804642+0000","last_active":"2024-03-06T07:27:48.808417+0000","last_peered":"2024-03-06T07:27:48.808417+0000","last_clean":"2024-03-06T07:27:48.808417+0000","last_became_active":"2024-03-06T07:27:48.804397+0000","last_became_peered":"2024-03-06T07:27:48.804397+0000","last_unstale":"2024-03-06T07:27:48.808417+0000","last_undegraded":"2024-03-06T07:27:48.808417+0000","last_fullsized":"2024-03-06T07:27:48.808417+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:09.768452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786754+0000","last_change":"2024-03-06T07:27:47.779625+0000","last_active":"2024-03-06T07:27:49.786754+0000","last_peered":"2024-03-06T07:27:49.786754+0000","last_clean":"2024-03-06T07:27:49.786754+0000","last_became_active":"2024-03-06T07:27:47.779219+0000","last_became_peered":"2024-03-06T07:27:47.779219+0000","last_unstale":"2024-03-06T07:27:49.786754+0000","last_undegraded":"2024-03-06T07:27:49.786754+0000","last_fullsized":"2024-03-06T07:27:49.786754+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:49.554927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802412+0000","last_change":"2024-03-06T07:27:48.799290+0000","last_active":"2024-03-06T07:27:48.802412+0000","last_peered":"2024-03-06T07:27:48.802412+0000","last_clean":"2024-03-06T07:27:48.802412+0000","last_became_active":"2024-03-06T07:27:48.798759+0000","last_became_peered":"2024-03-06T07:27:48.798759+0000","last_unstale":"2024-03-06T07:27:48.802412+0000","last_undegraded":"2024-03-06T07:27:48.802412+0000","last_fullsized":"2024-03-06T07:27:48.802412+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:29.690767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787148+0000","last_change":"2024-03-06T07:27:47.777764+0000","last_active":"2024-03-06T07:27:49.787148+0000","last_peered":"2024-03-06T07:27:49.787148+0000","last_clean":"2024-03-06T07:27:49.787148+0000","last_became_active":"2024-03-06T07:27:47.777485+0000","last_became_peered":"2024-03-06T07:27:47.777485+0000","last_unstale":"2024-03-06T07:27:49.787148+0000","last_undegraded":"2024-03-06T07:27:49.787148+0000","last_fullsized":"2024-03-06T07:27:49.787148+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:58.788508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.797776+0000","last_change":"2024-03-06T07:27:48.789978+0000","last_active":"2024-03-06T07:27:48.797776+0000","last_peered":"2024-03-06T07:27:48.797776+0000","last_clean":"2024-03-06T07:27:48.797776+0000","last_became_active":"2024-03-06T07:27:48.789690+0000","last_became_peered":"2024-03-06T07:27:48.789690+0000","last_unstale":"2024-03-06T07:27:48.797776+0000","last_undegraded":"2024-03-06T07:27:48.797776+0000","last_fullsized":"2024-03-06T07:27:48.797776+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:22.753749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.789634+0000","last_change":"2024-03-06T07:27:47.779895+0000","last_active":"2024-03-06T07:27:48.789634+0000","last_peered":"2024-03-06T07:27:48.789634+0000","last_clean":"2024-03-06T07:27:48.789634+0000","last_became_active":"2024-03-06T07:27:47.779631+0000","last_became_peered":"2024-03-06T07:27:47.779631+0000","last_unstale":"2024-03-06T07:27:48.789634+0000","last_undegraded":"2024-03-06T07:27:48.789634+0000","last_fullsized":"2024-03-06T07:27:48.789634+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:13.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808769+0000","last_change":"2024-03-06T07:27:48.803552+0000","last_active":"2024-03-06T07:27:48.808769+0000","last_peered":"2024-03-06T07:27:48.808769+0000","last_clean":"2024-03-06T07:27:48.808769+0000","last_became_active":"2024-03-06T07:27:48.803143+0000","last_became_peered":"2024-03-06T07:27:48.803143+0000","last_unstale":"2024-03-06T07:27:48.808769+0000","last_undegraded":"2024-03-06T07:27:48.808769+0000","last_fullsized":"2024-03-06T07:27:48.808769+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:46.891453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.789425+0000","last_change":"2024-03-06T07:27:47.776385+0000","last_active":"2024-03-06T07:27:48.789425+0000","last_peered":"2024-03-06T07:27:48.789425+0000","last_clean":"2024-03-06T07:27:48.789425+0000","last_became_active":"2024-03-06T07:27:47.776097+0000","last_became_peered":"2024-03-06T07:27:47.776097+0000","last_unstale":"2024-03-06T07:27:48.789425+0000","last_undegraded":"2024-03-06T07:27:48.789425+0000","last_fullsized":"2024-03-06T07:27:48.789425+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:01.287677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796346+0000","last_change":"2024-03-06T07:27:48.793821+0000","last_active":"2024-03-06T07:27:48.796346+0000","last_peered":"2024-03-06T07:27:48.796346+0000","last_clean":"2024-03-06T07:27:48.796346+0000","last_became_active":"2024-03-06T07:27:48.793533+0000","last_became_peered":"2024-03-06T07:27:48.793533+0000","last_unstale":"2024-03-06T07:27:48.796346+0000","last_undegraded":"2024-03-06T07:27:48.796346+0000","last_fullsized":"2024-03-06T07:27:48.796346+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:50.898201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786708+0000","last_change":"2024-03-06T07:27:47.780912+0000","last_active":"2024-03-06T07:27:49.786708+0000","last_peered":"2024-03-06T07:27:49.786708+0000","last_clean":"2024-03-06T07:27:49.786708+0000","last_became_active":"2024-03-06T07:27:47.780648+0000","last_became_peered":"2024-03-06T07:27:47.780648+0000","last_unstale":"2024-03-06T07:27:49.786708+0000","last_undegraded":"2024-03-06T07:27:49.786708+0000","last_fullsized":"2024-03-06T07:27:49.786708+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:49.893544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787085+0000","last_change":"2024-03-06T07:27:48.798936+0000","last_active":"2024-03-06T07:27:49.787085+0000","last_peered":"2024-03-06T07:27:49.787085+0000","last_clean":"2024-03-06T07:27:49.787085+0000","last_became_active":"2024-03-06T07:27:48.798238+0000","last_became_peered":"2024-03-06T07:27:48.798238+0000","last_unstale":"2024-03-06T07:27:49.787085+0000","last_undegraded":"2024-03-06T07:27:49.787085+0000","last_fullsized":"2024-03-06T07:27:49.787085+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:35.877874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801545+0000","last_change":"2024-03-06T07:27:47.776593+0000","last_active":"2024-03-06T07:27:48.801545+0000","last_peered":"2024-03-06T07:27:48.801545+0000","last_clean":"2024-03-06T07:27:48.801545+0000","last_became_active":"2024-03-06T07:27:47.776328+0000","last_became_peered":"2024-03-06T07:27:47.776328+0000","last_unstale":"2024-03-06T07:27:48.801545+0000","last_undegraded":"2024-03-06T07:27:48.801545+0000","last_fullsized":"2024-03-06T07:27:48.801545+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:15.672617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804597+0000","last_change":"2024-03-06T07:27:48.800716+0000","last_active":"2024-03-06T07:27:48.804597+0000","last_peered":"2024-03-06T07:27:48.804597+0000","last_clean":"2024-03-06T07:27:48.804597+0000","last_became_active":"2024-03-06T07:27:48.799508+0000","last_became_peered":"2024-03-06T07:27:48.799508+0000","last_unstale":"2024-03-06T07:27:48.804597+0000","last_undegraded":"2024-03-06T07:27:48.804597+0000","last_fullsized":"2024-03-06T07:27:48.804597+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:46.272663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787136+0000","last_change":"2024-03-06T07:27:47.777045+0000","last_active":"2024-03-06T07:27:49.787136+0000","last_peered":"2024-03-06T07:27:49.787136+0000","last_clean":"2024-03-06T07:27:49.787136+0000","last_became_active":"2024-03-06T07:27:47.776762+0000","last_became_peered":"2024-03-06T07:27:47.776762+0000","last_unstale":"2024-03-06T07:27:49.787136+0000","last_undegraded":"2024-03-06T07:27:49.787136+0000","last_fullsized":"2024-03-06T07:27:49.787136+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:48.002247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801496+0000","last_change":"2024-03-06T07:27:48.795605+0000","last_active":"2024-03-06T07:27:48.801496+0000","last_peered":"2024-03-06T07:27:48.801496+0000","last_clean":"2024-03-06T07:27:48.801496+0000","last_became_active":"2024-03-06T07:27:48.795341+0000","last_became_peered":"2024-03-06T07:27:48.795341+0000","last_unstale":"2024-03-06T07:27:48.801496+0000","last_undegraded":"2024-03-06T07:27:48.801496+0000","last_fullsized":"2024-03-06T07:27:48.801496+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:47.418761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786966+0000","last_change":"2024-03-06T07:27:47.780561+0000","last_active":"2024-03-06T07:27:49.786966+0000","last_peered":"2024-03-06T07:27:49.786966+0000","last_clean":"2024-03-06T07:27:49.786966+0000","last_became_active":"2024-03-06T07:27:47.780242+0000","last_became_peered":"2024-03-06T07:27:47.780242+0000","last_unstale":"2024-03-06T07:27:49.786966+0000","last_undegraded":"2024-03-06T07:27:49.786966+0000","last_fullsized":"2024-03-06T07:27:49.786966+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:51.724322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796971+0000","last_change":"2024-03-06T07:27:48.794742+0000","last_active":"2024-03-06T07:27:48.796971+0000","last_peered":"2024-03-06T07:27:48.796971+0000","last_clean":"2024-03-06T07:27:48.796971+0000","last_became_active":"2024-03-06T07:27:48.794446+0000","last_became_peered":"2024-03-06T07:27:48.794446+0000","last_unstale":"2024-03-06T07:27:48.796971+0000","last_undegraded":"2024-03-06T07:27:48.796971+0000","last_fullsized":"2024-03-06T07:27:48.796971+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:12.341333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.787571+0000","last_change":"2024-03-06T07:27:47.777152+0000","last_active":"2024-03-06T07:27:48.787571+0000","last_peered":"2024-03-06T07:27:48.787571+0000","last_clean":"2024-03-06T07:27:48.787571+0000","last_became_active":"2024-03-06T07:27:47.776679+0000","last_became_peered":"2024-03-06T07:27:47.776679+0000","last_unstale":"2024-03-06T07:27:48.787571+0000","last_undegraded":"2024-03-06T07:27:48.787571+0000","last_fullsized":"2024-03-06T07:27:48.787571+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:07.960133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.799177+0000","last_change":"2024-03-06T07:27:48.797957+0000","last_active":"2024-03-06T07:27:48.799177+0000","last_peered":"2024-03-06T07:27:48.799177+0000","last_clean":"2024-03-06T07:27:48.799177+0000","last_became_active":"2024-03-06T07:27:48.797504+0000","last_became_peered":"2024-03-06T07:27:48.797504+0000","last_unstale":"2024-03-06T07:27:48.799177+0000","last_undegraded":"2024-03-06T07:27:48.799177+0000","last_fullsized":"2024-03-06T07:27:48.799177+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:00.223858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786971+0000","last_change":"2024-03-06T07:27:47.775434+0000","last_active":"2024-03-06T07:27:49.786971+0000","last_peered":"2024-03-06T07:27:49.786971+0000","last_clean":"2024-03-06T07:27:49.786971+0000","last_became_active":"2024-03-06T07:27:47.775149+0000","last_became_peered":"2024-03-06T07:27:47.775149+0000","last_unstale":"2024-03-06T07:27:49.786971+0000","last_undegraded":"2024-03-06T07:27:49.786971+0000","last_fullsized":"2024-03-06T07:27:49.786971+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:17.339754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790099+0000","last_change":"2024-03-06T07:27:48.800230+0000","last_active":"2024-03-06T07:27:49.790099+0000","last_peered":"2024-03-06T07:27:49.790099+0000","last_clean":"2024-03-06T07:27:49.790099+0000","last_became_active":"2024-03-06T07:27:48.799946+0000","last_became_peered":"2024-03-06T07:27:48.799946+0000","last_unstale":"2024-03-06T07:27:49.790099+0000","last_undegraded":"2024-03-06T07:27:49.790099+0000","last_fullsized":"2024-03-06T07:27:49.790099+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:12.341333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.785461+0000","last_change":"2024-03-06T07:27:47.778202+0000","last_active":"2024-03-06T07:27:48.785461+0000","last_peered":"2024-03-06T07:27:48.785461+0000","last_clean":"2024-03-06T07:27:48.785461+0000","last_became_active":"2024-03-06T07:27:47.777909+0000","last_became_peered":"2024-03-06T07:27:47.777909+0000","last_unstale":"2024-03-06T07:27:48.785461+0000","last_undegraded":"2024-03-06T07:27:48.785461+0000","last_fullsized":"2024-03-06T07:27:48.785461+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:41.678195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.797870+0000","last_change":"2024-03-06T07:27:48.791256+0000","last_active":"2024-03-06T07:27:48.797870+0000","last_peered":"2024-03-06T07:27:48.797870+0000","last_clean":"2024-03-06T07:27:48.797870+0000","last_became_active":"2024-03-06T07:27:48.791003+0000","last_became_peered":"2024-03-06T07:27:48.791003+0000","last_unstale":"2024-03-06T07:27:48.797870+0000","last_undegraded":"2024-03-06T07:27:48.797870+0000","last_fullsized":"2024-03-06T07:27:48.797870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:11.469263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789695+0000","last_change":"2024-03-06T07:27:47.780344+0000","last_active":"2024-03-06T07:27:49.789695+0000","last_peered":"2024-03-06T07:27:49.789695+0000","last_clean":"2024-03-06T07:27:49.789695+0000","last_became_active":"2024-03-06T07:27:47.780075+0000","last_became_peered":"2024-03-06T07:27:47.780075+0000","last_unstale":"2024-03-06T07:27:49.789695+0000","last_undegraded":"2024-03-06T07:27:49.789695+0000","last_fullsized":"2024-03-06T07:27:49.789695+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:51.724322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804634+0000","last_change":"2024-03-06T07:27:48.803235+0000","last_active":"2024-03-06T07:27:48.804634+0000","last_peered":"2024-03-06T07:27:48.804634+0000","last_clean":"2024-03-06T07:27:48.804634+0000","last_became_active":"2024-03-06T07:27:48.802771+0000","last_became_peered":"2024-03-06T07:27:48.802771+0000","last_unstale":"2024-03-06T07:27:48.804634+0000","last_undegraded":"2024-03-06T07:27:48.804634+0000","last_fullsized":"2024-03-06T07:27:48.804634+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:49.074891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.778712+0000","last_change":"2024-03-06T07:27:47.778794+0000","last_active":"2024-03-06T07:27:48.778712+0000","last_peered":"2024-03-06T07:27:48.778712+0000","last_clean":"2024-03-06T07:27:48.778712+0000","last_became_active":"2024-03-06T07:27:47.778517+0000","last_became_peered":"2024-03-06T07:27:47.778517+0000","last_unstale":"2024-03-06T07:27:48.778712+0000","last_undegraded":"2024-03-06T07:27:48.778712+0000","last_fullsized":"2024-03-06T07:27:48.778712+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:46.337372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798175+0000","last_change":"2024-03-06T07:27:48.792574+0000","last_active":"2024-03-06T07:27:48.798175+0000","last_peered":"2024-03-06T07:27:48.798175+0000","last_clean":"2024-03-06T07:27:48.798175+0000","last_became_active":"2024-03-06T07:27:48.792286+0000","last_became_peered":"2024-03-06T07:27:48.792286+0000","last_unstale":"2024-03-06T07:27:48.798175+0000","last_undegraded":"2024-03-06T07:27:48.798175+0000","last_fullsized":"2024-03-06T07:27:48.798175+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:16.164897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804873+0000","last_change":"2024-03-06T07:27:47.776742+0000","last_active":"2024-03-06T07:27:48.804873+0000","last_peered":"2024-03-06T07:27:48.804873+0000","last_clean":"2024-03-06T07:27:48.804873+0000","last_became_active":"2024-03-06T07:27:47.776463+0000","last_became_peered":"2024-03-06T07:27:47.776463+0000","last_unstale":"2024-03-06T07:27:48.804873+0000","last_undegraded":"2024-03-06T07:27:48.804873+0000","last_fullsized":"2024-03-06T07:27:48.804873+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:49.554927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":316,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.792495+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:27:48.792495+0000","last_peered":"2024-03-06T07:27:48.792495+0000","last_clean":"2024-03-06T07:27:48.792495+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:27:48.792495+0000","last_undegraded":"2024-03-06T07:27:48.792495+0000","last_fullsized":"2024-03-06T07:27:48.792495+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32944,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338896,"statfs":{"total":96636764160,"available":96603029504,"internally_reserved":0,"allocated":1802240,"data_stored":1102172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1208320,"data_stored":512344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1204224,"data_stored":511762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1220608,"data_stored":513428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32380,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339460,"statfs":{"total":96636764160,"available":96603607040,"internally_reserved":0,"allocated":1228800,"data_stored":519626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1204224,"data_stored":511726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1200128,"data_stored":511692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32944,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338896,"statfs":{"total":96636764160,"available":96603029504,"internally_reserved":0,"allocated":1798144,"data_stored":1102642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:52.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:52.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.952+0000 7effcb4df640 1 Processor -- start 2024-03-06T07:27:52.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.953+0000 7effcb4df640 1 -- start start 2024-03-06T07:27:52.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.953+0000 7effcb4df640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7effc40b95a0 0x7effc40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.953+0000 7effcb4df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 0x7effc40b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.953+0000 7effcb4df640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc40cea30 con 0x7effc415df00 2024-03-06T07:27:52.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.953+0000 7effcb4df640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc40cf3c0 con 0x7effc40b95a0 2024-03-06T07:27:52.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.953+0000 7effcb4df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc40cf0e0 con 0x7effc40b9a10 2024-03-06T07:27:52.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.953+0000 7effc8a7a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7effc40b95a0 0x7effc40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.954+0000 7effc8a7a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7effc40b95a0 0x7effc40b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52710/0 (socket says 172.21.15.27:52710) 2024-03-06T07:27:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.954+0000 7effc927b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7effc415df00 0x7effc40d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40788/0 (socket says 172.21.15.27:40788) 2024-03-06T07:27:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.954+0000 7effc8a7a640 1 -- 172.21.15.27:0/4203427447 learned_addr learned my addr 172.21.15.27:0/4203427447 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.954+0000 7effc9a7c640 1 --2- 172.21.15.27:0/4203427447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 0x7effc40b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.954+0000 7effc8a7a640 1 -- 172.21.15.27:0/4203427447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 msgr2=0x7effc40b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.954+0000 7effc8a7a640 1 --2- 172.21.15.27:0/4203427447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 0x7effc40b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.954+0000 7effc8a7a640 1 -- 172.21.15.27:0/4203427447 >> v1:172.21.15.27:6789/0 conn(0x7effc415df00 legacy=0x7effc40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.954+0000 7effbbfff640 1 -- 172.21.15.27:0/4203427447 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194742874 0 0) 0x7effc40cea30 con 0x7effc415df00 2024-03-06T07:27:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.954+0000 7effc8a7a640 1 -- 172.21.15.27:0/4203427447 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40ce470 con 0x7effc40b95a0 2024-03-06T07:27:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.955+0000 7effc8a7a640 1 --2- 172.21.15.27:0/4203427447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 0x7effc40b5700 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7effc0009870 tx=0x7effc002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798fee5bca95bf88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.955+0000 7effbbfff640 1 -- 172.21.15.27:0/4203427447 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc0034070 con 0x7effc40b95a0 2024-03-06T07:27:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.955+0000 7effbbfff640 1 -- 172.21.15.27:0/4203427447 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc0030840 con 0x7effc40b95a0 2024-03-06T07:27:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.955+0000 7effbbfff640 1 -- 172.21.15.27:0/4203427447 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc0030c10 con 0x7effc40b95a0 2024-03-06T07:27:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.956+0000 7effcb4df640 1 -- 172.21.15.27:0/4203427447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 msgr2=0x7effc40b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.956+0000 7effcb4df640 1 --2- 172.21.15.27:0/4203427447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 0x7effc40b5700 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7effc0009870 tx=0x7effc002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.956+0000 7effcb4df640 1 -- 172.21.15.27:0/4203427447 shutdown_connections 2024-03-06T07:27:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.956+0000 7effcb4df640 1 --2- 172.21.15.27:0/4203427447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 0x7effc40b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.956+0000 7effcb4df640 1 --2- 172.21.15.27:0/4203427447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 0x7effc40b5700 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.956+0000 7effcb4df640 1 -- 172.21.15.27:0/4203427447 >> 172.21.15.27:0/4203427447 conn(0x7effc4071960 msgr2=0x7effc4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.956+0000 7effcb4df640 1 -- 172.21.15.27:0/4203427447 shutdown_connections 2024-03-06T07:27:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.956+0000 7effcb4df640 1 -- 172.21.15.27:0/4203427447 wait complete. 2024-03-06T07:27:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.957+0000 7effcb4df640 1 Processor -- start 2024-03-06T07:27:52.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.957+0000 7effcb4df640 1 -- start start 2024-03-06T07:27:52.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effcb4df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 0x7effc40b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effcb4df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 0x7effc40b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effcb4df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc415df00 0x7effc415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effcb4df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc40cedf0 con 0x7effc40b95a0 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effcb4df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc40cec70 con 0x7effc415df00 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effcb4df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc40d3220 con 0x7effc40b9a10 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc927b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 0x7effc40b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc927b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 0x7effc40b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52724/0 (socket says 172.21.15.27:52724) 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc927b640 1 -- 172.21.15.27:0/792503880 learned_addr learned my addr 172.21.15.27:0/792503880 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc8a7a640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 0x7effc40b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc9a7c640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc415df00 0x7effc415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc927b640 1 -- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 msgr2=0x7effc40b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc927b640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 0x7effc40b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc927b640 1 -- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc415df00 msgr2=0x7effc415e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc927b640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc415df00 0x7effc415e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.958+0000 7effc927b640 1 -- 172.21.15.27:0/792503880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc41f73c0 con 0x7effc40b95a0 2024-03-06T07:27:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.959+0000 7effc927b640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 0x7effc40b5640 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7effb400c430 tx=0x7effb400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.959+0000 7effb9ffb640 1 -- 172.21.15.27:0/792503880 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb40074e0 con 0x7effc40b95a0 2024-03-06T07:27:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.959+0000 7effb9ffb640 1 -- 172.21.15.27:0/792503880 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb40047e0 con 0x7effc40b95a0 2024-03-06T07:27:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.959+0000 7effb9ffb640 1 -- 172.21.15.27:0/792503880 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4004bd0 con 0x7effc40b95a0 2024-03-06T07:27:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.959+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc41f76a0 con 0x7effc40b95a0 2024-03-06T07:27:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.959+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc414d2d0 con 0x7effc40b95a0 2024-03-06T07:27:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.960+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc4067ff0 con 0x7effc40b95a0 2024-03-06T07:27:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.960+0000 7effb9ffb640 1 -- 172.21.15.27:0/792503880 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effb4011070 con 0x7effc40b95a0 2024-03-06T07:27:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.961+0000 7effb9ffb640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7effa806b4d0 0x7effa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.961+0000 7effb9ffb640 1 -- 172.21.15.27:0/792503880 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7effb4089ca0 con 0x7effc40b95a0 2024-03-06T07:27:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.961+0000 7effc8a7a640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7effa806b4d0 0x7effa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.962+0000 7effc8a7a640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7effa806b4d0 0x7effa806d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7effc0036000 tx=0x7effc0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:52.965+0000 7effb9ffb640 1 -- 172.21.15.27:0/792503880 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effb4058560 con 0x7effc40b95a0 2024-03-06T07:27:53.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.142+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effc41f7fd0 con 0x7effa806b4d0 2024-03-06T07:27:53.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.148+0000 7effb9ffb640 1 -- 172.21.15.27:0/792503880 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252043 (secure 0 0 0) 0x7effc41f7fd0 con 0x7effa806b4d0 2024-03-06T07:27:53.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:53.153 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:27:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7effa806b4d0 msgr2=0x7effa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7effa806b4d0 0x7effa806d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7effc0036000 tx=0x7effc0008030 comp rx=0 tx=0).stop 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 msgr2=0x7effc40b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 0x7effc40b5640 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7effb400c430 tx=0x7effb400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 shutdown_connections 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7effa806b4d0 0x7effa806d990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc415df00 0x7effc415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7effc40b9a10 0x7effc40b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 --2- 172.21.15.27:0/792503880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc40b95a0 0x7effc40b5640 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 >> 172.21.15.27:0/792503880 conn(0x7effc4071960 msgr2=0x7effc4071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 shutdown_connections 2024-03-06T07:27:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.151+0000 7effcb4df640 1 -- 172.21.15.27:0/792503880 wait complete. 2024-03-06T07:27:53.163 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-03-06T07:27:52.059894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260100,"kb_used_data":10612,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714620,"statfs":{"total":773094113280,"available":772827770880,"internally_reserved":0,"allocated":10866688,"data_stored":5285392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.301543"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801580+0000","last_change":"2024-03-06T07:27:48.795851+0000","last_active":"2024-03-06T07:27:48.801580+0000","last_peered":"2024-03-06T07:27:48.801580+0000","last_clean":"2024-03-06T07:27:48.801580+0000","last_became_active":"2024-03-06T07:27:48.795333+0000","last_became_peered":"2024-03-06T07:27:48.795333+0000","last_unstale":"2024-03-06T07:27:48.801580+0000","last_undegraded":"2024-03-06T07:27:48.801580+0000","last_fullsized":"2024-03-06T07:27:48.801580+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:52.539081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787100+0000","last_change":"2024-03-06T07:27:48.797653+0000","last_active":"2024-03-06T07:27:49.787100+0000","last_peered":"2024-03-06T07:27:49.787100+0000","last_clean":"2024-03-06T07:27:49.787100+0000","last_became_active":"2024-03-06T07:27:48.797295+0000","last_became_peered":"2024-03-06T07:27:48.797295+0000","last_unstale":"2024-03-06T07:27:49.787100+0000","last_undegraded":"2024-03-06T07:27:49.787100+0000","last_fullsized":"2024-03-06T07:27:49.787100+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:47.706960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802294+0000","last_change":"2024-03-06T07:27:48.798199+0000","last_active":"2024-03-06T07:27:48.802294+0000","last_peered":"2024-03-06T07:27:48.802294+0000","last_clean":"2024-03-06T07:27:48.802294+0000","last_became_active":"2024-03-06T07:27:48.797938+0000","last_became_peered":"2024-03-06T07:27:48.797938+0000","last_unstale":"2024-03-06T07:27:48.802294+0000","last_undegraded":"2024-03-06T07:27:48.802294+0000","last_fullsized":"2024-03-06T07:27:48.802294+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:28.145024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789870+0000","last_change":"2024-03-06T07:27:48.801803+0000","last_active":"2024-03-06T07:27:49.789870+0000","last_peered":"2024-03-06T07:27:49.789870+0000","last_clean":"2024-03-06T07:27:49.789870+0000","last_became_active":"2024-03-06T07:27:48.800374+0000","last_became_peered":"2024-03-06T07:27:48.800374+0000","last_unstale":"2024-03-06T07:27:49.789870+0000","last_undegraded":"2024-03-06T07:27:49.789870+0000","last_fullsized":"2024-03-06T07:27:49.789870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:33.743145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804429+0000","last_change":"2024-03-06T07:27:48.801900+0000","last_active":"2024-03-06T07:27:48.804429+0000","last_peered":"2024-03-06T07:27:48.804429+0000","last_clean":"2024-03-06T07:27:48.804429+0000","last_became_active":"2024-03-06T07:27:48.801632+0000","last_became_peered":"2024-03-06T07:27:48.801632+0000","last_unstale":"2024-03-06T07:27:48.804429+0000","last_undegraded":"2024-03-06T07:27:48.804429+0000","last_fullsized":"2024-03-06T07:27:48.804429+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:59.992386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.803734+0000","last_change":"2024-03-06T07:27:48.798657+0000","last_active":"2024-03-06T07:27:48.803734+0000","last_peered":"2024-03-06T07:27:48.803734+0000","last_clean":"2024-03-06T07:27:48.803734+0000","last_became_active":"2024-03-06T07:27:48.798282+0000","last_became_peered":"2024-03-06T07:27:48.798282+0000","last_unstale":"2024-03-06T07:27:48.803734+0000","last_undegraded":"2024-03-06T07:27:48.803734+0000","last_fullsized":"2024-03-06T07:27:48.803734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:49.074891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.803411+0000","last_change":"2024-03-06T07:27:48.796710+0000","last_active":"2024-03-06T07:27:48.803411+0000","last_peered":"2024-03-06T07:27:48.803411+0000","last_clean":"2024-03-06T07:27:48.803411+0000","last_became_active":"2024-03-06T07:27:48.796108+0000","last_became_peered":"2024-03-06T07:27:48.796108+0000","last_unstale":"2024-03-06T07:27:48.803411+0000","last_undegraded":"2024-03-06T07:27:48.803411+0000","last_fullsized":"2024-03-06T07:27:48.803411+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:52.728980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786611+0000","last_change":"2024-03-06T07:27:48.796169+0000","last_active":"2024-03-06T07:27:49.786611+0000","last_peered":"2024-03-06T07:27:49.786611+0000","last_clean":"2024-03-06T07:27:49.786611+0000","last_became_active":"2024-03-06T07:27:48.795915+0000","last_became_peered":"2024-03-06T07:27:48.795915+0000","last_unstale":"2024-03-06T07:27:49.786611+0000","last_undegraded":"2024-03-06T07:27:49.786611+0000","last_fullsized":"2024-03-06T07:27:49.786611+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:19.744943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790560+0000","last_change":"2024-03-06T07:27:48.793854+0000","last_active":"2024-03-06T07:27:49.790560+0000","last_peered":"2024-03-06T07:27:49.790560+0000","last_clean":"2024-03-06T07:27:49.790560+0000","last_became_active":"2024-03-06T07:27:48.793564+0000","last_became_peered":"2024-03-06T07:27:48.793564+0000","last_unstale":"2024-03-06T07:27:49.790560+0000","last_undegraded":"2024-03-06T07:27:49.790560+0000","last_fullsized":"2024-03-06T07:27:49.790560+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:52.728980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.797389+0000","last_change":"2024-03-06T07:27:48.794510+0000","last_active":"2024-03-06T07:27:48.797389+0000","last_peered":"2024-03-06T07:27:48.797389+0000","last_clean":"2024-03-06T07:27:48.797389+0000","last_became_active":"2024-03-06T07:27:48.794221+0000","last_became_peered":"2024-03-06T07:27:48.794221+0000","last_unstale":"2024-03-06T07:27:48.797389+0000","last_undegraded":"2024-03-06T07:27:48.797389+0000","last_fullsized":"2024-03-06T07:27:48.797389+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:55.210682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808399+0000","last_change":"2024-03-06T07:27:48.804541+0000","last_active":"2024-03-06T07:27:48.808399+0000","last_peered":"2024-03-06T07:27:48.808399+0000","last_clean":"2024-03-06T07:27:48.808399+0000","last_became_active":"2024-03-06T07:27:48.804246+0000","last_became_peered":"2024-03-06T07:27:48.804246+0000","last_unstale":"2024-03-06T07:27:48.808399+0000","last_undegraded":"2024-03-06T07:27:48.808399+0000","last_fullsized":"2024-03-06T07:27:48.808399+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:29.734536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790348+0000","last_change":"2024-03-06T07:27:48.790705+0000","last_active":"2024-03-06T07:27:49.790348+0000","last_peered":"2024-03-06T07:27:49.790348+0000","last_clean":"2024-03-06T07:27:49.790348+0000","last_became_active":"2024-03-06T07:27:48.790431+0000","last_became_peered":"2024-03-06T07:27:48.790431+0000","last_unstale":"2024-03-06T07:27:49.790348+0000","last_undegraded":"2024-03-06T07:27:49.790348+0000","last_fullsized":"2024-03-06T07:27:49.790348+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:50.559585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789754+0000","last_change":"2024-03-06T07:27:48.792112+0000","last_active":"2024-03-06T07:27:49.789754+0000","last_peered":"2024-03-06T07:27:49.789754+0000","last_clean":"2024-03-06T07:27:49.789754+0000","last_became_active":"2024-03-06T07:27:48.791847+0000","last_became_peered":"2024-03-06T07:27:48.791847+0000","last_unstale":"2024-03-06T07:27:49.789754+0000","last_undegraded":"2024-03-06T07:27:49.789754+0000","last_fullsized":"2024-03-06T07:27:49.789754+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:53.226987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802433+0000","last_change":"2024-03-06T07:27:48.799504+0000","last_active":"2024-03-06T07:27:48.802433+0000","last_peered":"2024-03-06T07:27:48.802433+0000","last_clean":"2024-03-06T07:27:48.802433+0000","last_became_active":"2024-03-06T07:27:48.798944+0000","last_became_peered":"2024-03-06T07:27:48.798944+0000","last_unstale":"2024-03-06T07:27:48.802433+0000","last_undegraded":"2024-03-06T07:27:48.802433+0000","last_fullsized":"2024-03-06T07:27:48.802433+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:43.167479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789509+0000","last_change":"2024-03-06T07:27:48.801372+0000","last_active":"2024-03-06T07:27:49.789509+0000","last_peered":"2024-03-06T07:27:49.789509+0000","last_clean":"2024-03-06T07:27:49.789509+0000","last_became_active":"2024-03-06T07:27:48.800890+0000","last_became_peered":"2024-03-06T07:27:48.800890+0000","last_unstale":"2024-03-06T07:27:49.789509+0000","last_undegraded":"2024-03-06T07:27:49.789509+0000","last_fullsized":"2024-03-06T07:27:49.789509+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:01.705383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808351+0000","last_change":"2024-03-06T07:27:48.802940+0000","last_active":"2024-03-06T07:27:48.808351+0000","last_peered":"2024-03-06T07:27:48.808351+0000","last_clean":"2024-03-06T07:27:48.808351+0000","last_became_active":"2024-03-06T07:27:48.802679+0000","last_became_peered":"2024-03-06T07:27:48.802679+0000","last_unstale":"2024-03-06T07:27:48.808351+0000","last_undegraded":"2024-03-06T07:27:48.808351+0000","last_fullsized":"2024-03-06T07:27:48.808351+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:49.006905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787075+0000","last_change":"2024-03-06T07:27:48.793735+0000","last_active":"2024-03-06T07:27:49.787075+0000","last_peered":"2024-03-06T07:27:49.787075+0000","last_clean":"2024-03-06T07:27:49.787075+0000","last_became_active":"2024-03-06T07:27:48.793063+0000","last_became_peered":"2024-03-06T07:27:48.793063+0000","last_unstale":"2024-03-06T07:27:49.787075+0000","last_undegraded":"2024-03-06T07:27:49.787075+0000","last_fullsized":"2024-03-06T07:27:49.787075+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:02.703373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789777+0000","last_change":"2024-03-06T07:27:48.800359+0000","last_active":"2024-03-06T07:27:49.789777+0000","last_peered":"2024-03-06T07:27:49.789777+0000","last_clean":"2024-03-06T07:27:49.789777+0000","last_became_active":"2024-03-06T07:27:48.800062+0000","last_became_peered":"2024-03-06T07:27:48.800062+0000","last_unstale":"2024-03-06T07:27:49.789777+0000","last_undegraded":"2024-03-06T07:27:49.789777+0000","last_fullsized":"2024-03-06T07:27:49.789777+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:49.962734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798268+0000","last_change":"2024-03-06T07:27:48.797350+0000","last_active":"2024-03-06T07:27:48.798268+0000","last_peered":"2024-03-06T07:27:48.798268+0000","last_clean":"2024-03-06T07:27:48.798268+0000","last_became_active":"2024-03-06T07:27:48.793707+0000","last_became_peered":"2024-03-06T07:27:48.793707+0000","last_unstale":"2024-03-06T07:27:48.798268+0000","last_undegraded":"2024-03-06T07:27:48.798268+0000","last_fullsized":"2024-03-06T07:27:48.798268+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:35.618564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790486+0000","last_change":"2024-03-06T07:27:48.797707+0000","last_active":"2024-03-06T07:27:49.790486+0000","last_peered":"2024-03-06T07:27:49.790486+0000","last_clean":"2024-03-06T07:27:49.790486+0000","last_became_active":"2024-03-06T07:27:48.793867+0000","last_became_peered":"2024-03-06T07:27:48.793867+0000","last_unstale":"2024-03-06T07:27:49.790486+0000","last_undegraded":"2024-03-06T07:27:49.790486+0000","last_fullsized":"2024-03-06T07:27:49.790486+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:59.992386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802258+0000","last_change":"2024-03-06T07:27:48.799007+0000","last_active":"2024-03-06T07:27:48.802258+0000","last_peered":"2024-03-06T07:27:48.802258+0000","last_clean":"2024-03-06T07:27:48.802258+0000","last_became_active":"2024-03-06T07:27:48.798738+0000","last_became_peered":"2024-03-06T07:27:48.798738+0000","last_unstale":"2024-03-06T07:27:48.802258+0000","last_undegraded":"2024-03-06T07:27:48.802258+0000","last_fullsized":"2024-03-06T07:27:48.802258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:43.019435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786588+0000","last_change":"2024-03-06T07:27:48.796914+0000","last_active":"2024-03-06T07:27:49.786588+0000","last_peered":"2024-03-06T07:27:49.786588+0000","last_clean":"2024-03-06T07:27:49.786588+0000","last_became_active":"2024-03-06T07:27:48.796630+0000","last_became_peered":"2024-03-06T07:27:48.796630+0000","last_unstale":"2024-03-06T07:27:49.786588+0000","last_undegraded":"2024-03-06T07:27:49.786588+0000","last_fullsized":"2024-03-06T07:27:49.786588+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:33.743145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786926+0000","last_change":"2024-03-06T07:27:48.793763+0000","last_active":"2024-03-06T07:27:49.786926+0000","last_peered":"2024-03-06T07:27:49.786926+0000","last_clean":"2024-03-06T07:27:49.786926+0000","last_became_active":"2024-03-06T07:27:48.793517+0000","last_became_peered":"2024-03-06T07:27:48.793517+0000","last_unstale":"2024-03-06T07:27:49.786926+0000","last_undegraded":"2024-03-06T07:27:49.786926+0000","last_fullsized":"2024-03-06T07:27:49.786926+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:14.670440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.805508+0000","last_change":"2024-03-06T07:27:48.802332+0000","last_active":"2024-03-06T07:27:48.805508+0000","last_peered":"2024-03-06T07:27:48.805508+0000","last_clean":"2024-03-06T07:27:48.805508+0000","last_became_active":"2024-03-06T07:27:48.802032+0000","last_became_peered":"2024-03-06T07:27:48.802032+0000","last_unstale":"2024-03-06T07:27:48.805508+0000","last_undegraded":"2024-03-06T07:27:48.805508+0000","last_fullsized":"2024-03-06T07:27:48.805508+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:18.344411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798756+0000","last_change":"2024-03-06T07:27:48.793638+0000","last_active":"2024-03-06T07:27:48.798756+0000","last_peered":"2024-03-06T07:27:48.798756+0000","last_clean":"2024-03-06T07:27:48.798756+0000","last_became_active":"2024-03-06T07:27:48.793350+0000","last_became_peered":"2024-03-06T07:27:48.793350+0000","last_unstale":"2024-03-06T07:27:48.798756+0000","last_undegraded":"2024-03-06T07:27:48.798756+0000","last_fullsized":"2024-03-06T07:27:48.798756+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:59.800540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786828+0000","last_change":"2024-03-06T07:27:48.788989+0000","last_active":"2024-03-06T07:27:49.786828+0000","last_peered":"2024-03-06T07:27:49.786828+0000","last_clean":"2024-03-06T07:27:49.786828+0000","last_became_active":"2024-03-06T07:27:48.788710+0000","last_became_peered":"2024-03-06T07:27:48.788710+0000","last_unstale":"2024-03-06T07:27:49.786828+0000","last_undegraded":"2024-03-06T07:27:49.786828+0000","last_fullsized":"2024-03-06T07:27:49.786828+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:11.990008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.805489+0000","last_change":"2024-03-06T07:27:48.801347+0000","last_active":"2024-03-06T07:27:48.805489+0000","last_peered":"2024-03-06T07:27:48.805489+0000","last_clean":"2024-03-06T07:27:48.805489+0000","last_became_active":"2024-03-06T07:27:48.800813+0000","last_became_peered":"2024-03-06T07:27:48.800813+0000","last_unstale":"2024-03-06T07:27:48.805489+0000","last_undegraded":"2024-03-06T07:27:48.805489+0000","last_fullsized":"2024-03-06T07:27:48.805489+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:59.793165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804661+0000","last_change":"2024-03-06T07:27:48.802751+0000","last_active":"2024-03-06T07:27:48.804661+0000","last_peered":"2024-03-06T07:27:48.804661+0000","last_clean":"2024-03-06T07:27:48.804661+0000","last_became_active":"2024-03-06T07:27:48.802405+0000","last_became_peered":"2024-03-06T07:27:48.802405+0000","last_unstale":"2024-03-06T07:27:48.804661+0000","last_undegraded":"2024-03-06T07:27:48.804661+0000","last_fullsized":"2024-03-06T07:27:48.804661+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:52.728980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786823+0000","last_change":"2024-03-06T07:27:48.789405+0000","last_active":"2024-03-06T07:27:49.786823+0000","last_peered":"2024-03-06T07:27:49.786823+0000","last_clean":"2024-03-06T07:27:49.786823+0000","last_became_active":"2024-03-06T07:27:48.788692+0000","last_became_peered":"2024-03-06T07:27:48.788692+0000","last_unstale":"2024-03-06T07:27:49.786823+0000","last_undegraded":"2024-03-06T07:27:49.786823+0000","last_fullsized":"2024-03-06T07:27:49.786823+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:18.882652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802386+0000","last_change":"2024-03-06T07:27:48.799277+0000","last_active":"2024-03-06T07:27:48.802386+0000","last_peered":"2024-03-06T07:27:48.802386+0000","last_clean":"2024-03-06T07:27:48.802386+0000","last_became_active":"2024-03-06T07:27:48.798582+0000","last_became_peered":"2024-03-06T07:27:48.798582+0000","last_unstale":"2024-03-06T07:27:48.802386+0000","last_undegraded":"2024-03-06T07:27:48.802386+0000","last_fullsized":"2024-03-06T07:27:48.802386+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:22.493871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789558+0000","last_change":"2024-03-06T07:27:48.801175+0000","last_active":"2024-03-06T07:27:49.789558+0000","last_peered":"2024-03-06T07:27:49.789558+0000","last_clean":"2024-03-06T07:27:49.789558+0000","last_became_active":"2024-03-06T07:27:48.800614+0000","last_became_peered":"2024-03-06T07:27:48.800614+0000","last_unstale":"2024-03-06T07:27:49.789558+0000","last_undegraded":"2024-03-06T07:27:49.789558+0000","last_fullsized":"2024-03-06T07:27:49.789558+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:42.003832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.790700+0000","last_change":"2024-03-06T07:27:48.788164+0000","last_active":"2024-03-06T07:27:48.790700+0000","last_peered":"2024-03-06T07:27:48.790700+0000","last_clean":"2024-03-06T07:27:48.790700+0000","last_became_active":"2024-03-06T07:27:48.787907+0000","last_became_peered":"2024-03-06T07:27:48.787907+0000","last_unstale":"2024-03-06T07:27:48.790700+0000","last_undegraded":"2024-03-06T07:27:48.790700+0000","last_fullsized":"2024-03-06T07:27:48.790700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:29.690767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798628+0000","last_change":"2024-03-06T07:27:48.798020+0000","last_active":"2024-03-06T07:27:48.798628+0000","last_peered":"2024-03-06T07:27:48.798628+0000","last_clean":"2024-03-06T07:27:48.798628+0000","last_became_active":"2024-03-06T07:27:48.797674+0000","last_became_peered":"2024-03-06T07:27:48.797674+0000","last_unstale":"2024-03-06T07:27:48.798628+0000","last_undegraded":"2024-03-06T07:27:48.798628+0000","last_fullsized":"2024-03-06T07:27:48.798628+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:14.039295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"97'4","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786964+0000","last_change":"2024-03-06T07:27:47.777623+0000","last_active":"2024-03-06T07:27:49.786964+0000","last_peered":"2024-03-06T07:27:49.786964+0000","last_clean":"2024-03-06T07:27:49.786964+0000","last_became_active":"2024-03-06T07:27:47.777293+0000","last_became_peered":"2024-03-06T07:27:47.777293+0000","last_unstale":"2024-03-06T07:27:49.786964+0000","last_undegraded":"2024-03-06T07:27:49.786964+0000","last_fullsized":"2024-03-06T07:27:49.786964+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:48.070234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789934+0000","last_change":"2024-03-06T07:27:48.795542+0000","last_active":"2024-03-06T07:27:49.789934+0000","last_peered":"2024-03-06T07:27:49.789934+0000","last_clean":"2024-03-06T07:27:49.789934+0000","last_became_active":"2024-03-06T07:27:48.795254+0000","last_became_peered":"2024-03-06T07:27:48.795254+0000","last_unstale":"2024-03-06T07:27:49.789934+0000","last_undegraded":"2024-03-06T07:27:49.789934+0000","last_fullsized":"2024-03-06T07:27:49.789934+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:19.744943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801986+0000","last_change":"2024-03-06T07:27:47.784471+0000","last_active":"2024-03-06T07:27:48.801986+0000","last_peered":"2024-03-06T07:27:48.801986+0000","last_clean":"2024-03-06T07:27:48.801986+0000","last_became_active":"2024-03-06T07:27:47.784211+0000","last_became_peered":"2024-03-06T07:27:47.784211+0000","last_unstale":"2024-03-06T07:27:48.801986+0000","last_undegraded":"2024-03-06T07:27:48.801986+0000","last_fullsized":"2024-03-06T07:27:48.801986+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:11.336675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798378+0000","last_change":"2024-03-06T07:27:48.790902+0000","last_active":"2024-03-06T07:27:48.798378+0000","last_peered":"2024-03-06T07:27:48.798378+0000","last_clean":"2024-03-06T07:27:48.798378+0000","last_became_active":"2024-03-06T07:27:48.790480+0000","last_became_peered":"2024-03-06T07:27:48.790480+0000","last_unstale":"2024-03-06T07:27:48.798378+0000","last_undegraded":"2024-03-06T07:27:48.798378+0000","last_fullsized":"2024-03-06T07:27:48.798378+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:43.167479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786868+0000","last_change":"2024-03-06T07:27:47.783376+0000","last_active":"2024-03-06T07:27:49.786868+0000","last_peered":"2024-03-06T07:27:49.786868+0000","last_clean":"2024-03-06T07:27:49.786868+0000","last_became_active":"2024-03-06T07:27:47.783123+0000","last_became_peered":"2024-03-06T07:27:47.783123+0000","last_unstale":"2024-03-06T07:27:49.786868+0000","last_undegraded":"2024-03-06T07:27:49.786868+0000","last_fullsized":"2024-03-06T07:27:49.786868+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:58.987729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790580+0000","last_change":"2024-03-06T07:27:48.798612+0000","last_active":"2024-03-06T07:27:49.790580+0000","last_peered":"2024-03-06T07:27:49.790580+0000","last_clean":"2024-03-06T07:27:49.790580+0000","last_became_active":"2024-03-06T07:27:48.797841+0000","last_became_peered":"2024-03-06T07:27:48.797841+0000","last_unstale":"2024-03-06T07:27:49.790580+0000","last_undegraded":"2024-03-06T07:27:49.790580+0000","last_fullsized":"2024-03-06T07:27:49.790580+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:49.074891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786654+0000","last_change":"2024-03-06T07:27:47.780478+0000","last_active":"2024-03-06T07:27:49.786654+0000","last_peered":"2024-03-06T07:27:49.786654+0000","last_clean":"2024-03-06T07:27:49.786654+0000","last_became_active":"2024-03-06T07:27:47.780187+0000","last_became_peered":"2024-03-06T07:27:47.780187+0000","last_unstale":"2024-03-06T07:27:49.786654+0000","last_undegraded":"2024-03-06T07:27:49.786654+0000","last_fullsized":"2024-03-06T07:27:49.786654+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:54.206025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796937+0000","last_change":"2024-03-06T07:27:48.792623+0000","last_active":"2024-03-06T07:27:48.796937+0000","last_peered":"2024-03-06T07:27:48.796937+0000","last_clean":"2024-03-06T07:27:48.796937+0000","last_became_active":"2024-03-06T07:27:48.792379+0000","last_became_peered":"2024-03-06T07:27:48.792379+0000","last_unstale":"2024-03-06T07:27:48.796937+0000","last_undegraded":"2024-03-06T07:27:48.796937+0000","last_fullsized":"2024-03-06T07:27:48.796937+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:26.004240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.789991+0000","last_change":"2024-03-06T07:27:47.776984+0000","last_active":"2024-03-06T07:27:48.789991+0000","last_peered":"2024-03-06T07:27:48.789991+0000","last_clean":"2024-03-06T07:27:48.789991+0000","last_became_active":"2024-03-06T07:27:47.776679+0000","last_became_peered":"2024-03-06T07:27:47.776679+0000","last_unstale":"2024-03-06T07:27:48.789991+0000","last_undegraded":"2024-03-06T07:27:48.789991+0000","last_fullsized":"2024-03-06T07:27:48.789991+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:59.219201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808397+0000","last_change":"2024-03-06T07:27:48.804068+0000","last_active":"2024-03-06T07:27:48.808397+0000","last_peered":"2024-03-06T07:27:48.808397+0000","last_clean":"2024-03-06T07:27:48.808397+0000","last_became_active":"2024-03-06T07:27:48.803431+0000","last_became_peered":"2024-03-06T07:27:48.803431+0000","last_unstale":"2024-03-06T07:27:48.808397+0000","last_undegraded":"2024-03-06T07:27:48.808397+0000","last_fullsized":"2024-03-06T07:27:48.808397+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:28:44.063497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786697+0000","last_change":"2024-03-06T07:27:47.780233+0000","last_active":"2024-03-06T07:27:49.786697+0000","last_peered":"2024-03-06T07:27:49.786697+0000","last_clean":"2024-03-06T07:27:49.786697+0000","last_became_active":"2024-03-06T07:27:47.779968+0000","last_became_peered":"2024-03-06T07:27:47.779968+0000","last_unstale":"2024-03-06T07:27:49.786697+0000","last_undegraded":"2024-03-06T07:27:49.786697+0000","last_fullsized":"2024-03-06T07:27:49.786697+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:11.336675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789472+0000","last_change":"2024-03-06T07:27:48.796209+0000","last_active":"2024-03-06T07:27:49.789472+0000","last_peered":"2024-03-06T07:27:49.789472+0000","last_clean":"2024-03-06T07:27:49.789472+0000","last_became_active":"2024-03-06T07:27:48.795957+0000","last_became_peered":"2024-03-06T07:27:48.795957+0000","last_unstale":"2024-03-06T07:27:49.789472+0000","last_undegraded":"2024-03-06T07:27:49.789472+0000","last_fullsized":"2024-03-06T07:27:49.789472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:10.959320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.785904+0000","last_change":"2024-03-06T07:27:47.780756+0000","last_active":"2024-03-06T07:27:48.785904+0000","last_peered":"2024-03-06T07:27:48.785904+0000","last_clean":"2024-03-06T07:27:48.785904+0000","last_became_active":"2024-03-06T07:27:47.780417+0000","last_became_peered":"2024-03-06T07:27:47.780417+0000","last_unstale":"2024-03-06T07:27:48.785904+0000","last_undegraded":"2024-03-06T07:27:48.785904+0000","last_fullsized":"2024-03-06T07:27:48.785904+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:03.533825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789925+0000","last_change":"2024-03-06T07:27:48.798440+0000","last_active":"2024-03-06T07:27:49.789925+0000","last_peered":"2024-03-06T07:27:49.789925+0000","last_clean":"2024-03-06T07:27:49.789925+0000","last_became_active":"2024-03-06T07:27:48.798093+0000","last_became_peered":"2024-03-06T07:27:48.798093+0000","last_unstale":"2024-03-06T07:27:49.789925+0000","last_undegraded":"2024-03-06T07:27:49.789925+0000","last_fullsized":"2024-03-06T07:27:49.789925+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:55.210682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.780191+0000","last_change":"2024-03-06T07:27:47.777912+0000","last_active":"2024-03-06T07:27:48.780191+0000","last_peered":"2024-03-06T07:27:48.780191+0000","last_clean":"2024-03-06T07:27:48.780191+0000","last_became_active":"2024-03-06T07:27:47.777610+0000","last_became_peered":"2024-03-06T07:27:47.777610+0000","last_unstale":"2024-03-06T07:27:48.780191+0000","last_undegraded":"2024-03-06T07:27:48.780191+0000","last_fullsized":"2024-03-06T07:27:48.780191+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:42.014777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786976+0000","last_change":"2024-03-06T07:27:48.792774+0000","last_active":"2024-03-06T07:27:49.786976+0000","last_peered":"2024-03-06T07:27:49.786976+0000","last_clean":"2024-03-06T07:27:49.786976+0000","last_became_active":"2024-03-06T07:27:48.792074+0000","last_became_peered":"2024-03-06T07:27:48.792074+0000","last_unstale":"2024-03-06T07:27:49.786976+0000","last_undegraded":"2024-03-06T07:27:49.786976+0000","last_fullsized":"2024-03-06T07:27:49.786976+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:35.663777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.781439+0000","last_change":"2024-03-06T07:27:47.782069+0000","last_active":"2024-03-06T07:27:48.781439+0000","last_peered":"2024-03-06T07:27:48.781439+0000","last_clean":"2024-03-06T07:27:48.781439+0000","last_became_active":"2024-03-06T07:27:47.781790+0000","last_became_peered":"2024-03-06T07:27:47.781790+0000","last_unstale":"2024-03-06T07:27:48.781439+0000","last_undegraded":"2024-03-06T07:27:48.781439+0000","last_fullsized":"2024-03-06T07:27:48.781439+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.489214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790765+0000","last_change":"2024-03-06T07:27:48.796717+0000","last_active":"2024-03-06T07:27:49.790765+0000","last_peered":"2024-03-06T07:27:49.790765+0000","last_clean":"2024-03-06T07:27:49.790765+0000","last_became_active":"2024-03-06T07:27:48.796455+0000","last_became_peered":"2024-03-06T07:27:48.796455+0000","last_unstale":"2024-03-06T07:27:49.790765+0000","last_undegraded":"2024-03-06T07:27:49.790765+0000","last_fullsized":"2024-03-06T07:27:49.790765+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:59.793165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789831+0000","last_change":"2024-03-06T07:27:47.780974+0000","last_active":"2024-03-06T07:27:49.789831+0000","last_peered":"2024-03-06T07:27:49.789831+0000","last_clean":"2024-03-06T07:27:49.789831+0000","last_became_active":"2024-03-06T07:27:47.780701+0000","last_became_peered":"2024-03-06T07:27:47.780701+0000","last_unstale":"2024-03-06T07:27:49.789831+0000","last_undegraded":"2024-03-06T07:27:49.789831+0000","last_fullsized":"2024-03-06T07:27:49.789831+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:58.987729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796328+0000","last_change":"2024-03-06T07:27:48.793759+0000","last_active":"2024-03-06T07:27:48.796328+0000","last_peered":"2024-03-06T07:27:48.796328+0000","last_clean":"2024-03-06T07:27:48.796328+0000","last_became_active":"2024-03-06T07:27:48.793468+0000","last_became_peered":"2024-03-06T07:27:48.793468+0000","last_unstale":"2024-03-06T07:27:48.796328+0000","last_undegraded":"2024-03-06T07:27:48.796328+0000","last_fullsized":"2024-03-06T07:27:48.796328+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:13.207256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"97'3","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804150+0000","last_change":"2024-03-06T07:27:47.778360+0000","last_active":"2024-03-06T07:27:48.804150+0000","last_peered":"2024-03-06T07:27:48.804150+0000","last_clean":"2024-03-06T07:27:48.804150+0000","last_became_active":"2024-03-06T07:27:47.778069+0000","last_became_peered":"2024-03-06T07:27:47.778069+0000","last_unstale":"2024-03-06T07:27:48.804150+0000","last_undegraded":"2024-03-06T07:27:48.804150+0000","last_fullsized":"2024-03-06T07:27:48.804150+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:58.987729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808382+0000","last_change":"2024-03-06T07:27:48.803862+0000","last_active":"2024-03-06T07:27:48.808382+0000","last_peered":"2024-03-06T07:27:48.808382+0000","last_clean":"2024-03-06T07:27:48.808382+0000","last_became_active":"2024-03-06T07:27:48.803196+0000","last_became_peered":"2024-03-06T07:27:48.803196+0000","last_unstale":"2024-03-06T07:27:48.808382+0000","last_undegraded":"2024-03-06T07:27:48.808382+0000","last_fullsized":"2024-03-06T07:27:48.808382+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:12.157853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"97'6","reported_seq":20,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790470+0000","last_change":"2024-03-06T07:27:47.775011+0000","last_active":"2024-03-06T07:27:49.790470+0000","last_peered":"2024-03-06T07:27:49.790470+0000","last_clean":"2024-03-06T07:27:49.790470+0000","last_became_active":"2024-03-06T07:27:47.774698+0000","last_became_peered":"2024-03-06T07:27:47.774698+0000","last_unstale":"2024-03-06T07:27:49.790470+0000","last_undegraded":"2024-03-06T07:27:49.790470+0000","last_fullsized":"2024-03-06T07:27:49.790470+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:13.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789604+0000","last_change":"2024-03-06T07:27:48.796347+0000","last_active":"2024-03-06T07:27:49.789604+0000","last_peered":"2024-03-06T07:27:49.789604+0000","last_clean":"2024-03-06T07:27:49.789604+0000","last_became_active":"2024-03-06T07:27:48.796096+0000","last_became_peered":"2024-03-06T07:27:48.796096+0000","last_unstale":"2024-03-06T07:27:49.789604+0000","last_undegraded":"2024-03-06T07:27:49.789604+0000","last_fullsized":"2024-03-06T07:27:49.789604+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:19.208168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.793547+0000","last_change":"2024-03-06T07:27:47.774075+0000","last_active":"2024-03-06T07:27:48.793547+0000","last_peered":"2024-03-06T07:27:48.793547+0000","last_clean":"2024-03-06T07:27:48.793547+0000","last_became_active":"2024-03-06T07:27:47.773374+0000","last_became_peered":"2024-03-06T07:27:47.773374+0000","last_unstale":"2024-03-06T07:27:48.793547+0000","last_undegraded":"2024-03-06T07:27:48.793547+0000","last_fullsized":"2024-03-06T07:27:48.793547+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:23.403860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801405+0000","last_change":"2024-03-06T07:27:48.794407+0000","last_active":"2024-03-06T07:27:48.801405+0000","last_peered":"2024-03-06T07:27:48.801405+0000","last_clean":"2024-03-06T07:27:48.801405+0000","last_became_active":"2024-03-06T07:27:48.794139+0000","last_became_peered":"2024-03-06T07:27:48.794139+0000","last_unstale":"2024-03-06T07:27:48.801405+0000","last_undegraded":"2024-03-06T07:27:48.801405+0000","last_fullsized":"2024-03-06T07:27:48.801405+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:46.891453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.784249+0000","last_change":"2024-03-06T07:27:47.775045+0000","last_active":"2024-03-06T07:27:48.784249+0000","last_peered":"2024-03-06T07:27:48.784249+0000","last_clean":"2024-03-06T07:27:48.784249+0000","last_became_active":"2024-03-06T07:27:47.774751+0000","last_became_peered":"2024-03-06T07:27:47.774751+0000","last_unstale":"2024-03-06T07:27:48.784249+0000","last_undegraded":"2024-03-06T07:27:48.784249+0000","last_fullsized":"2024-03-06T07:27:48.784249+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:53.038808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789911+0000","last_change":"2024-03-06T07:27:48.795541+0000","last_active":"2024-03-06T07:27:49.789911+0000","last_peered":"2024-03-06T07:27:49.789911+0000","last_clean":"2024-03-06T07:27:49.789911+0000","last_became_active":"2024-03-06T07:27:48.793670+0000","last_became_peered":"2024-03-06T07:27:48.793670+0000","last_unstale":"2024-03-06T07:27:49.789911+0000","last_undegraded":"2024-03-06T07:27:49.789911+0000","last_fullsized":"2024-03-06T07:27:49.789911+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:50.898201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789945+0000","last_change":"2024-03-06T07:27:47.774018+0000","last_active":"2024-03-06T07:27:49.789945+0000","last_peered":"2024-03-06T07:27:49.789945+0000","last_clean":"2024-03-06T07:27:49.789945+0000","last_became_active":"2024-03-06T07:27:47.773537+0000","last_became_peered":"2024-03-06T07:27:47.773537+0000","last_unstale":"2024-03-06T07:27:49.789945+0000","last_undegraded":"2024-03-06T07:27:49.789945+0000","last_fullsized":"2024-03-06T07:27:49.789945+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:54.711622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804475+0000","last_change":"2024-03-06T07:27:48.791020+0000","last_active":"2024-03-06T07:27:48.804475+0000","last_peered":"2024-03-06T07:27:48.804475+0000","last_clean":"2024-03-06T07:27:48.804475+0000","last_became_active":"2024-03-06T07:27:48.790755+0000","last_became_peered":"2024-03-06T07:27:48.790755+0000","last_unstale":"2024-03-06T07:27:48.804475+0000","last_undegraded":"2024-03-06T07:27:48.804475+0000","last_fullsized":"2024-03-06T07:27:48.804475+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:08.718352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"97'8","reported_seq":20,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.792334+0000","last_change":"2024-03-06T07:27:47.779087+0000","last_active":"2024-03-06T07:27:48.792334+0000","last_peered":"2024-03-06T07:27:48.792334+0000","last_clean":"2024-03-06T07:27:48.792334+0000","last_became_active":"2024-03-06T07:27:47.778765+0000","last_became_peered":"2024-03-06T07:27:47.778765+0000","last_unstale":"2024-03-06T07:27:48.792334+0000","last_undegraded":"2024-03-06T07:27:48.792334+0000","last_fullsized":"2024-03-06T07:27:48.792334+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:13.034637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798779+0000","last_change":"2024-03-06T07:27:48.795848+0000","last_active":"2024-03-06T07:27:48.798779+0000","last_peered":"2024-03-06T07:27:48.798779+0000","last_clean":"2024-03-06T07:27:48.798779+0000","last_became_active":"2024-03-06T07:27:48.795500+0000","last_became_peered":"2024-03-06T07:27:48.795500+0000","last_unstale":"2024-03-06T07:27:48.798779+0000","last_undegraded":"2024-03-06T07:27:48.798779+0000","last_fullsized":"2024-03-06T07:27:48.798779+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:50.559585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790488+0000","last_change":"2024-03-06T07:27:47.778837+0000","last_active":"2024-03-06T07:27:49.790488+0000","last_peered":"2024-03-06T07:27:49.790488+0000","last_clean":"2024-03-06T07:27:49.790488+0000","last_became_active":"2024-03-06T07:27:47.777861+0000","last_became_peered":"2024-03-06T07:27:47.777861+0000","last_unstale":"2024-03-06T07:27:49.790488+0000","last_undegraded":"2024-03-06T07:27:49.790488+0000","last_fullsized":"2024-03-06T07:27:49.790488+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:01.287677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.797654+0000","last_change":"2024-03-06T07:27:48.795117+0000","last_active":"2024-03-06T07:27:48.797654+0000","last_peered":"2024-03-06T07:27:48.797654+0000","last_clean":"2024-03-06T07:27:48.797654+0000","last_became_active":"2024-03-06T07:27:48.794821+0000","last_became_peered":"2024-03-06T07:27:48.794821+0000","last_unstale":"2024-03-06T07:27:48.797654+0000","last_undegraded":"2024-03-06T07:27:48.797654+0000","last_fullsized":"2024-03-06T07:27:48.797654+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:42.682853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796657+0000","last_change":"2024-03-06T07:27:47.777867+0000","last_active":"2024-03-06T07:27:48.796657+0000","last_peered":"2024-03-06T07:27:48.796657+0000","last_clean":"2024-03-06T07:27:48.796657+0000","last_became_active":"2024-03-06T07:27:47.777524+0000","last_became_peered":"2024-03-06T07:27:47.777524+0000","last_unstale":"2024-03-06T07:27:48.796657+0000","last_undegraded":"2024-03-06T07:27:48.796657+0000","last_fullsized":"2024-03-06T07:27:48.796657+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:37.466064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808417+0000","last_change":"2024-03-06T07:27:48.804642+0000","last_active":"2024-03-06T07:27:48.808417+0000","last_peered":"2024-03-06T07:27:48.808417+0000","last_clean":"2024-03-06T07:27:48.808417+0000","last_became_active":"2024-03-06T07:27:48.804397+0000","last_became_peered":"2024-03-06T07:27:48.804397+0000","last_unstale":"2024-03-06T07:27:48.808417+0000","last_undegraded":"2024-03-06T07:27:48.808417+0000","last_fullsized":"2024-03-06T07:27:48.808417+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:09.768452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786754+0000","last_change":"2024-03-06T07:27:47.779625+0000","last_active":"2024-03-06T07:27:49.786754+0000","last_peered":"2024-03-06T07:27:49.786754+0000","last_clean":"2024-03-06T07:27:49.786754+0000","last_became_active":"2024-03-06T07:27:47.779219+0000","last_became_peered":"2024-03-06T07:27:47.779219+0000","last_unstale":"2024-03-06T07:27:49.786754+0000","last_undegraded":"2024-03-06T07:27:49.786754+0000","last_fullsized":"2024-03-06T07:27:49.786754+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:49.554927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.802412+0000","last_change":"2024-03-06T07:27:48.799290+0000","last_active":"2024-03-06T07:27:48.802412+0000","last_peered":"2024-03-06T07:27:48.802412+0000","last_clean":"2024-03-06T07:27:48.802412+0000","last_became_active":"2024-03-06T07:27:48.798759+0000","last_became_peered":"2024-03-06T07:27:48.798759+0000","last_unstale":"2024-03-06T07:27:48.802412+0000","last_undegraded":"2024-03-06T07:27:48.802412+0000","last_fullsized":"2024-03-06T07:27:48.802412+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:29.690767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787148+0000","last_change":"2024-03-06T07:27:47.777764+0000","last_active":"2024-03-06T07:27:49.787148+0000","last_peered":"2024-03-06T07:27:49.787148+0000","last_clean":"2024-03-06T07:27:49.787148+0000","last_became_active":"2024-03-06T07:27:47.777485+0000","last_became_peered":"2024-03-06T07:27:47.777485+0000","last_unstale":"2024-03-06T07:27:49.787148+0000","last_undegraded":"2024-03-06T07:27:49.787148+0000","last_fullsized":"2024-03-06T07:27:49.787148+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:58.788508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.797776+0000","last_change":"2024-03-06T07:27:48.789978+0000","last_active":"2024-03-06T07:27:48.797776+0000","last_peered":"2024-03-06T07:27:48.797776+0000","last_clean":"2024-03-06T07:27:48.797776+0000","last_became_active":"2024-03-06T07:27:48.789690+0000","last_became_peered":"2024-03-06T07:27:48.789690+0000","last_unstale":"2024-03-06T07:27:48.797776+0000","last_undegraded":"2024-03-06T07:27:48.797776+0000","last_fullsized":"2024-03-06T07:27:48.797776+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:22.753749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.789634+0000","last_change":"2024-03-06T07:27:47.779895+0000","last_active":"2024-03-06T07:27:48.789634+0000","last_peered":"2024-03-06T07:27:48.789634+0000","last_clean":"2024-03-06T07:27:48.789634+0000","last_became_active":"2024-03-06T07:27:47.779631+0000","last_became_peered":"2024-03-06T07:27:47.779631+0000","last_unstale":"2024-03-06T07:27:48.789634+0000","last_undegraded":"2024-03-06T07:27:48.789634+0000","last_fullsized":"2024-03-06T07:27:48.789634+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:13.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.808769+0000","last_change":"2024-03-06T07:27:48.803552+0000","last_active":"2024-03-06T07:27:48.808769+0000","last_peered":"2024-03-06T07:27:48.808769+0000","last_clean":"2024-03-06T07:27:48.808769+0000","last_became_active":"2024-03-06T07:27:48.803143+0000","last_became_peered":"2024-03-06T07:27:48.803143+0000","last_unstale":"2024-03-06T07:27:48.808769+0000","last_undegraded":"2024-03-06T07:27:48.808769+0000","last_fullsized":"2024-03-06T07:27:48.808769+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:46.891453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.789425+0000","last_change":"2024-03-06T07:27:47.776385+0000","last_active":"2024-03-06T07:27:48.789425+0000","last_peered":"2024-03-06T07:27:48.789425+0000","last_clean":"2024-03-06T07:27:48.789425+0000","last_became_active":"2024-03-06T07:27:47.776097+0000","last_became_peered":"2024-03-06T07:27:47.776097+0000","last_unstale":"2024-03-06T07:27:48.789425+0000","last_undegraded":"2024-03-06T07:27:48.789425+0000","last_fullsized":"2024-03-06T07:27:48.789425+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:01.287677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796346+0000","last_change":"2024-03-06T07:27:48.793821+0000","last_active":"2024-03-06T07:27:48.796346+0000","last_peered":"2024-03-06T07:27:48.796346+0000","last_clean":"2024-03-06T07:27:48.796346+0000","last_became_active":"2024-03-06T07:27:48.793533+0000","last_became_peered":"2024-03-06T07:27:48.793533+0000","last_unstale":"2024-03-06T07:27:48.796346+0000","last_undegraded":"2024-03-06T07:27:48.796346+0000","last_fullsized":"2024-03-06T07:27:48.796346+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:50.898201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786708+0000","last_change":"2024-03-06T07:27:47.780912+0000","last_active":"2024-03-06T07:27:49.786708+0000","last_peered":"2024-03-06T07:27:49.786708+0000","last_clean":"2024-03-06T07:27:49.786708+0000","last_became_active":"2024-03-06T07:27:47.780648+0000","last_became_peered":"2024-03-06T07:27:47.780648+0000","last_unstale":"2024-03-06T07:27:49.786708+0000","last_undegraded":"2024-03-06T07:27:49.786708+0000","last_fullsized":"2024-03-06T07:27:49.786708+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:49.893544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787085+0000","last_change":"2024-03-06T07:27:48.798936+0000","last_active":"2024-03-06T07:27:49.787085+0000","last_peered":"2024-03-06T07:27:49.787085+0000","last_clean":"2024-03-06T07:27:49.787085+0000","last_became_active":"2024-03-06T07:27:48.798238+0000","last_became_peered":"2024-03-06T07:27:48.798238+0000","last_unstale":"2024-03-06T07:27:49.787085+0000","last_undegraded":"2024-03-06T07:27:49.787085+0000","last_fullsized":"2024-03-06T07:27:49.787085+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:35.877874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801545+0000","last_change":"2024-03-06T07:27:47.776593+0000","last_active":"2024-03-06T07:27:48.801545+0000","last_peered":"2024-03-06T07:27:48.801545+0000","last_clean":"2024-03-06T07:27:48.801545+0000","last_became_active":"2024-03-06T07:27:47.776328+0000","last_became_peered":"2024-03-06T07:27:47.776328+0000","last_unstale":"2024-03-06T07:27:48.801545+0000","last_undegraded":"2024-03-06T07:27:48.801545+0000","last_fullsized":"2024-03-06T07:27:48.801545+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:15.672617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804597+0000","last_change":"2024-03-06T07:27:48.800716+0000","last_active":"2024-03-06T07:27:48.804597+0000","last_peered":"2024-03-06T07:27:48.804597+0000","last_clean":"2024-03-06T07:27:48.804597+0000","last_became_active":"2024-03-06T07:27:48.799508+0000","last_became_peered":"2024-03-06T07:27:48.799508+0000","last_unstale":"2024-03-06T07:27:48.804597+0000","last_undegraded":"2024-03-06T07:27:48.804597+0000","last_fullsized":"2024-03-06T07:27:48.804597+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:46.272663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.787136+0000","last_change":"2024-03-06T07:27:47.777045+0000","last_active":"2024-03-06T07:27:49.787136+0000","last_peered":"2024-03-06T07:27:49.787136+0000","last_clean":"2024-03-06T07:27:49.787136+0000","last_became_active":"2024-03-06T07:27:47.776762+0000","last_became_peered":"2024-03-06T07:27:47.776762+0000","last_unstale":"2024-03-06T07:27:49.787136+0000","last_undegraded":"2024-03-06T07:27:49.787136+0000","last_fullsized":"2024-03-06T07:27:49.787136+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:48.002247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.801496+0000","last_change":"2024-03-06T07:27:48.795605+0000","last_active":"2024-03-06T07:27:48.801496+0000","last_peered":"2024-03-06T07:27:48.801496+0000","last_clean":"2024-03-06T07:27:48.801496+0000","last_became_active":"2024-03-06T07:27:48.795341+0000","last_became_peered":"2024-03-06T07:27:48.795341+0000","last_unstale":"2024-03-06T07:27:48.801496+0000","last_undegraded":"2024-03-06T07:27:48.801496+0000","last_fullsized":"2024-03-06T07:27:48.801496+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:47.418761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786966+0000","last_change":"2024-03-06T07:27:47.780561+0000","last_active":"2024-03-06T07:27:49.786966+0000","last_peered":"2024-03-06T07:27:49.786966+0000","last_clean":"2024-03-06T07:27:49.786966+0000","last_became_active":"2024-03-06T07:27:47.780242+0000","last_became_peered":"2024-03-06T07:27:47.780242+0000","last_unstale":"2024-03-06T07:27:49.786966+0000","last_undegraded":"2024-03-06T07:27:49.786966+0000","last_fullsized":"2024-03-06T07:27:49.786966+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:51.724322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.796971+0000","last_change":"2024-03-06T07:27:48.794742+0000","last_active":"2024-03-06T07:27:48.796971+0000","last_peered":"2024-03-06T07:27:48.796971+0000","last_clean":"2024-03-06T07:27:48.796971+0000","last_became_active":"2024-03-06T07:27:48.794446+0000","last_became_peered":"2024-03-06T07:27:48.794446+0000","last_unstale":"2024-03-06T07:27:48.796971+0000","last_undegraded":"2024-03-06T07:27:48.796971+0000","last_fullsized":"2024-03-06T07:27:48.796971+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:12.341333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.787571+0000","last_change":"2024-03-06T07:27:47.777152+0000","last_active":"2024-03-06T07:27:48.787571+0000","last_peered":"2024-03-06T07:27:48.787571+0000","last_clean":"2024-03-06T07:27:48.787571+0000","last_became_active":"2024-03-06T07:27:47.776679+0000","last_became_peered":"2024-03-06T07:27:47.776679+0000","last_unstale":"2024-03-06T07:27:48.787571+0000","last_undegraded":"2024-03-06T07:27:48.787571+0000","last_fullsized":"2024-03-06T07:27:48.787571+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:07.960133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.799177+0000","last_change":"2024-03-06T07:27:48.797957+0000","last_active":"2024-03-06T07:27:48.799177+0000","last_peered":"2024-03-06T07:27:48.799177+0000","last_clean":"2024-03-06T07:27:48.799177+0000","last_became_active":"2024-03-06T07:27:48.797504+0000","last_became_peered":"2024-03-06T07:27:48.797504+0000","last_unstale":"2024-03-06T07:27:48.799177+0000","last_undegraded":"2024-03-06T07:27:48.799177+0000","last_fullsized":"2024-03-06T07:27:48.799177+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:00.223858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.786971+0000","last_change":"2024-03-06T07:27:47.775434+0000","last_active":"2024-03-06T07:27:49.786971+0000","last_peered":"2024-03-06T07:27:49.786971+0000","last_clean":"2024-03-06T07:27:49.786971+0000","last_became_active":"2024-03-06T07:27:47.775149+0000","last_became_peered":"2024-03-06T07:27:47.775149+0000","last_unstale":"2024-03-06T07:27:49.786971+0000","last_undegraded":"2024-03-06T07:27:49.786971+0000","last_fullsized":"2024-03-06T07:27:49.786971+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:17.339754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.790099+0000","last_change":"2024-03-06T07:27:48.800230+0000","last_active":"2024-03-06T07:27:49.790099+0000","last_peered":"2024-03-06T07:27:49.790099+0000","last_clean":"2024-03-06T07:27:49.790099+0000","last_became_active":"2024-03-06T07:27:48.799946+0000","last_became_peered":"2024-03-06T07:27:48.799946+0000","last_unstale":"2024-03-06T07:27:49.790099+0000","last_undegraded":"2024-03-06T07:27:49.790099+0000","last_fullsized":"2024-03-06T07:27:49.790099+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:12.341333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.785461+0000","last_change":"2024-03-06T07:27:47.778202+0000","last_active":"2024-03-06T07:27:48.785461+0000","last_peered":"2024-03-06T07:27:48.785461+0000","last_clean":"2024-03-06T07:27:48.785461+0000","last_became_active":"2024-03-06T07:27:47.777909+0000","last_became_peered":"2024-03-06T07:27:47.777909+0000","last_unstale":"2024-03-06T07:27:48.785461+0000","last_undegraded":"2024-03-06T07:27:48.785461+0000","last_fullsized":"2024-03-06T07:27:48.785461+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:41.678195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.797870+0000","last_change":"2024-03-06T07:27:48.791256+0000","last_active":"2024-03-06T07:27:48.797870+0000","last_peered":"2024-03-06T07:27:48.797870+0000","last_clean":"2024-03-06T07:27:48.797870+0000","last_became_active":"2024-03-06T07:27:48.791003+0000","last_became_peered":"2024-03-06T07:27:48.791003+0000","last_unstale":"2024-03-06T07:27:48.797870+0000","last_undegraded":"2024-03-06T07:27:48.797870+0000","last_fullsized":"2024-03-06T07:27:48.797870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:11.469263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-06T07:27:49.789695+0000","last_change":"2024-03-06T07:27:47.780344+0000","last_active":"2024-03-06T07:27:49.789695+0000","last_peered":"2024-03-06T07:27:49.789695+0000","last_clean":"2024-03-06T07:27:49.789695+0000","last_became_active":"2024-03-06T07:27:47.780075+0000","last_became_peered":"2024-03-06T07:27:47.780075+0000","last_unstale":"2024-03-06T07:27:49.789695+0000","last_undegraded":"2024-03-06T07:27:49.789695+0000","last_fullsized":"2024-03-06T07:27:49.789695+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:51.724322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804634+0000","last_change":"2024-03-06T07:27:48.803235+0000","last_active":"2024-03-06T07:27:48.804634+0000","last_peered":"2024-03-06T07:27:48.804634+0000","last_clean":"2024-03-06T07:27:48.804634+0000","last_became_active":"2024-03-06T07:27:48.802771+0000","last_became_peered":"2024-03-06T07:27:48.802771+0000","last_unstale":"2024-03-06T07:27:48.804634+0000","last_undegraded":"2024-03-06T07:27:48.804634+0000","last_fullsized":"2024-03-06T07:27:48.804634+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:49.074891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.778712+0000","last_change":"2024-03-06T07:27:47.778794+0000","last_active":"2024-03-06T07:27:48.778712+0000","last_peered":"2024-03-06T07:27:48.778712+0000","last_clean":"2024-03-06T07:27:48.778712+0000","last_became_active":"2024-03-06T07:27:47.778517+0000","last_became_peered":"2024-03-06T07:27:47.778517+0000","last_unstale":"2024-03-06T07:27:48.778712+0000","last_undegraded":"2024-03-06T07:27:48.778712+0000","last_fullsized":"2024-03-06T07:27:48.778712+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:46.337372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.798175+0000","last_change":"2024-03-06T07:27:48.792574+0000","last_active":"2024-03-06T07:27:48.798175+0000","last_peered":"2024-03-06T07:27:48.798175+0000","last_clean":"2024-03-06T07:27:48.798175+0000","last_became_active":"2024-03-06T07:27:48.792286+0000","last_became_peered":"2024-03-06T07:27:48.792286+0000","last_unstale":"2024-03-06T07:27:48.798175+0000","last_undegraded":"2024-03-06T07:27:48.798175+0000","last_fullsized":"2024-03-06T07:27:48.798175+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:47.757357+0000","last_clean_scrub_stamp":"2024-03-06T07:27:47.757357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:16.164897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.804873+0000","last_change":"2024-03-06T07:27:47.776742+0000","last_active":"2024-03-06T07:27:48.804873+0000","last_peered":"2024-03-06T07:27:48.804873+0000","last_clean":"2024-03-06T07:27:48.804873+0000","last_became_active":"2024-03-06T07:27:47.776463+0000","last_became_peered":"2024-03-06T07:27:47.776463+0000","last_unstale":"2024-03-06T07:27:48.804873+0000","last_undegraded":"2024-03-06T07:27:48.804873+0000","last_fullsized":"2024-03-06T07:27:48.804873+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:46.752699+0000","last_clean_scrub_stamp":"2024-03-06T07:27:46.752699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:49.554927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":316,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-06T07:27:48.792495+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:27:48.792495+0000","last_peered":"2024-03-06T07:27:48.792495+0000","last_clean":"2024-03-06T07:27:48.792495+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:27:48.792495+0000","last_undegraded":"2024-03-06T07:27:48.792495+0000","last_fullsized":"2024-03-06T07:27:48.792495+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32944,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338896,"statfs":{"total":96636764160,"available":96603029504,"internally_reserved":0,"allocated":1802240,"data_stored":1102172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1208320,"data_stored":512344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1204224,"data_stored":511762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1220608,"data_stored":513428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32380,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339460,"statfs":{"total":96636764160,"available":96603607040,"internally_reserved":0,"allocated":1228800,"data_stored":519626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1204224,"data_stored":511726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1200128,"data_stored":511692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32944,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338896,"statfs":{"total":96636764160,"available":96603029504,"internally_reserved":0,"allocated":1798144,"data_stored":1102642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:53.167 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:53.167 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:53.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.317+0000 7f0107c5c640 1 Processor -- start 2024-03-06T07:27:53.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.317+0000 7f0107c5c640 1 -- start start 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.318+0000 7f0107c5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000b79f0 0x7f01000b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.318+0000 7f0107c5c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01000b83b0 con 0x7f01000b2d30 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.318+0000 7f0107c5c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01000d4960 con 0x7f0100071540 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.318+0000 7f0107c5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01000d3230 con 0x7f01000b79f0 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.318+0000 7f01051f7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f01000b2d30 0x7f01000b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40792/0 (socket says 172.21.15.27:40792) 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.318+0000 7f01051f7640 1 -- 172.21.15.27:0/3280628980 learned_addr learned my addr 172.21.15.27:0/3280628980 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01061f9640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000b79f0 0x7f01000b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366928837 0 0) 0x7f01000b83b0 con 0x7f01000b2d30 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ec003520 con 0x7f01000b2d30 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982236025 0 0) 0x7f01000d4960 con 0x7f0100071540 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01000b83b0 con 0x7f0100071540 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4055281759 0 0) 0x7f00ec003520 con 0x7f01000b2d30 2024-03-06T07:27:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000b79f0 msgr2=0x7f01000b7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000b79f0 0x7f01000b7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 >> v1:172.21.15.69:6789/0 conn(0x7f0100071540 legacy=0x7f0100071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01000d36c0 con 0x7f01000b2d30 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f00fc001670 con 0x7f01000b2d30 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 >> v1:172.21.15.27:6789/0 conn(0x7f01000b2d30 legacy=0x7f01000b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000b79f0 0x7f00ec003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ec0040d0 0x7f00ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec004d40 0x7f00ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01000d3230 con 0x7f00ec004d40 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec00baa0 con 0x7f00ec0040d0 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec00bca0 con 0x7f01000b79f0 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01061f9640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec004d40 0x7f00ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01051f7640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ec0040d0 0x7f00ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.319+0000 7f01059f8640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000b79f0 0x7f00ec003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.320+0000 7f01061f9640 1 -- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000b79f0 msgr2=0x7f00ec003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.320+0000 7f01061f9640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000b79f0 0x7f00ec003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.320+0000 7f01061f9640 1 -- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ec0040d0 msgr2=0x7f00ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.320+0000 7f01061f9640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ec0040d0 0x7f00ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.320+0000 7f01061f9640 1 -- 172.21.15.27:0/3280628980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec003520 con 0x7f00ec004d40 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.320+0000 7f01061f9640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec004d40 0x7f00ec0051c0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009fc0 tx=0x7f00ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b62ba526fe27117 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.320+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8039070 con 0x7f00ec004d40 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.320+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f802cc20 con 0x7f00ec004d40 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.321+0000 7f01049f6640 1 -- 172.21.15.27:0/3280628980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8009c10 con 0x7f00ec004d40 2024-03-06T07:27:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.321+0000 7f0107c5c640 1 -- 172.21.15.27:0/3280628980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec004d40 msgr2=0x7f00ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.321+0000 7f0107c5c640 1 --2- 172.21.15.27:0/3280628980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ec004d40 0x7f00ec0051c0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009fc0 tx=0x7f00ec005830 comp rx=0 tx=0).stop 2024-03-06T07:27:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.321+0000 7f01061f9640 1 -- 172.21.15.27:0/3280628980 reap_dead start 2024-03-06T07:27:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.321+0000 7f0107c5c640 1 -- 172.21.15.27:0/3280628980 shutdown_connections 2024-03-06T07:27:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.321+0000 7f0107c5c640 1 -- 172.21.15.27:0/3280628980 >> 172.21.15.27:0/3280628980 conn(0x7f01000d3d60 msgr2=0x7f01000d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.321+0000 7f0107c5c640 1 -- 172.21.15.27:0/3280628980 shutdown_connections 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.321+0000 7f0107c5c640 1 -- 172.21.15.27:0/3280628980 wait complete. 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.322+0000 7f0107c5c640 1 Processor -- start 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.322+0000 7f0107c5c640 1 -- start start 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f0107c5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010015a600 0x7f01000ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f0107c5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000ad9e0 0x7f01000cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f0107c5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01000cdf50 0x7f01001f8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f0107c5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01000d2be0 con 0x7f010015a600 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f0107c5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01000d2900 con 0x7f01000cdf50 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f0107c5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01000d2ec0 con 0x7f01000ad9e0 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f01059f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010015a600 0x7f01000ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f01059f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010015a600 0x7f01000ad4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52744/0 (socket says 172.21.15.27:52744) 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f01059f8640 1 -- 172.21.15.27:0/313588378 learned_addr learned my addr 172.21.15.27:0/313588378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f01061f9640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01000cdf50 0x7f01001f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.323+0000 7f01051f7640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000ad9e0 0x7f01000cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f01059f8640 1 -- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000ad9e0 msgr2=0x7f01000cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f01059f8640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000ad9e0 0x7f01000cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f01059f8640 1 -- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01000cdf50 msgr2=0x7f01001f8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f01059f8640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01000cdf50 0x7f01001f8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f01059f8640 1 -- 172.21.15.27:0/313588378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0100155640 con 0x7f010015a600 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f01059f8640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010015a600 0x7f01000ad4a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f00f000f7e0 tx=0x7f00f000fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f00f67fc640 1 -- 172.21.15.27:0/313588378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f00052e0 con 0x7f010015a600 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f00f67fc640 1 -- 172.21.15.27:0/313588378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f0018070 con 0x7f010015a600 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010015a9e0 con 0x7f010015a600 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f00f67fc640 1 -- 172.21.15.27:0/313588378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f000d910 con 0x7f010015a600 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.324+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01000aad70 con 0x7f010015a600 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.325+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01000addc0 con 0x7f010015a600 2024-03-06T07:27:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.326+0000 7f00f67fc640 1 -- 172.21.15.27:0/313588378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00f000dab0 con 0x7f010015a600 2024-03-06T07:27:53.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.327+0000 7f00f67fc640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f00e406b4d0 0x7f00e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.327+0000 7f00f67fc640 1 -- 172.21.15.27:0/313588378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00f008a090 con 0x7f010015a600 2024-03-06T07:27:53.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.327+0000 7f01051f7640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f00e406b4d0 0x7f00e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.327+0000 7f01051f7640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f00e406b4d0 0x7f00e406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0100155dc0 tx=0x7f00fc001be0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.331+0000 7f00f67fc640 1 -- 172.21.15.27:0/313588378 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00f00588c0 con 0x7f010015a600 2024-03-06T07:27:53.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.642+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0100072070 con 0x7f010015a600 2024-03-06T07:27:53.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.645+0000 7f00f67fc640 1 -- 172.21.15.27:0/313588378 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f00f005c960 con 0x7f010015a600 2024-03-06T07:27:53.646 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:27:53.647 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:27:53.647 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-03-06T07:27:53.647 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-03-06T07:27:53.647 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:53.647 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f00e406b4d0 msgr2=0x7f00e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f00e406b4d0 0x7f00e406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0100155dc0 tx=0x7f00fc001be0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010015a600 msgr2=0x7f01000ad4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010015a600 0x7f01000ad4a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f00f000f7e0 tx=0x7f00f000fca0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 shutdown_connections 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f00e406b4d0 0x7f00e406d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01000cdf50 0x7f01001f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01000ad9e0 0x7f01000cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 --2- 172.21.15.27:0/313588378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010015a600 0x7f01000ad4a0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 >> 172.21.15.27:0/313588378 conn(0x7f01000d3d60 msgr2=0x7f010015af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 shutdown_connections 2024-03-06T07:27:53.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.647+0000 7f0107c5c640 1 -- 172.21.15.27:0/313588378 wait complete. 2024-03-06T07:27:53.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.766+0000 7f006b41c640 1 Processor -- start 2024-03-06T07:27:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.766+0000 7f006b41c640 1 -- start start 2024-03-06T07:27:53.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.767+0000 7f006b41c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0064157b80 0x7f006415b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.767+0000 7f006b41c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f00640cbab0 0x7f0064070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.767+0000 7f006b41c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006406c7b0 con 0x7f0064157f60 2024-03-06T07:27:53.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.767+0000 7f006b41c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006406ca90 con 0x7f00640cbab0 2024-03-06T07:27:53.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.767+0000 7f006b41c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006406cd70 con 0x7f0064157b80 2024-03-06T07:27:53.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.767+0000 7f00699b9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f00640cbab0 0x7f0064070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.767+0000 7f00691b8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0064157b80 0x7f006415b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.768+0000 7f00689b7640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0064157f60 0x7f006415b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:32938/0 (socket says 172.21.15.27:32938) 2024-03-06T07:27:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.768+0000 7f00689b7640 1 -- 172.21.15.27:0/2643422296 learned_addr learned my addr 172.21.15.27:0/2643422296 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.768+0000 7f005bfff640 1 -- 172.21.15.27:0/2643422296 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213260593 0 0) 0x7f006406c7b0 con 0x7f0064157f60 2024-03-06T07:27:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.768+0000 7f00691b8640 1 -- 172.21.15.27:0/2643422296 >> v2:172.21.15.69:3300/0 conn(0x7f00640cbab0 msgr2=0x7f0064070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.768+0000 7f00691b8640 1 --2- 172.21.15.27:0/2643422296 >> v2:172.21.15.69:3300/0 conn(0x7f00640cbab0 0x7f0064070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.768+0000 7f00691b8640 1 -- 172.21.15.27:0/2643422296 >> v1:172.21.15.69:6789/0 conn(0x7f0064157f60 legacy=0x7f006415b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.768+0000 7f00691b8640 1 -- 172.21.15.27:0/2643422296 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006406d050 con 0x7f0064157b80 2024-03-06T07:27:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.769+0000 7f00691b8640 1 --2- 172.21.15.27:0/2643422296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0064157b80 0x7f006415b260 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f00600097a0 tx=0x7f006002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d70744dae1f5d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.769+0000 7f005bfff640 1 -- 172.21.15.27:0/2643422296 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060035020 con 0x7f0064157b80 2024-03-06T07:27:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.769+0000 7f005bfff640 1 -- 172.21.15.27:0/2643422296 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0060031830 con 0x7f0064157b80 2024-03-06T07:27:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.769+0000 7f005bfff640 1 -- 172.21.15.27:0/2643422296 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060031c00 con 0x7f0064157b80 2024-03-06T07:27:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.770+0000 7f006b41c640 1 -- 172.21.15.27:0/2643422296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0064157b80 msgr2=0x7f006415b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.770+0000 7f006b41c640 1 --2- 172.21.15.27:0/2643422296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0064157b80 0x7f006415b260 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f00600097a0 tx=0x7f006002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.770+0000 7f006b41c640 1 -- 172.21.15.27:0/2643422296 shutdown_connections 2024-03-06T07:27:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.770+0000 7f006b41c640 1 --2- 172.21.15.27:0/2643422296 >> v2:172.21.15.69:3300/0 conn(0x7f00640cbab0 0x7f0064070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.770+0000 7f006b41c640 1 --2- 172.21.15.27:0/2643422296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0064157b80 0x7f006415b260 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.770+0000 7f006b41c640 1 -- 172.21.15.27:0/2643422296 >> 172.21.15.27:0/2643422296 conn(0x7f0064066d00 msgr2=0x7f00640670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.770+0000 7f006b41c640 1 -- 172.21.15.27:0/2643422296 shutdown_connections 2024-03-06T07:27:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.770+0000 7f006b41c640 1 -- 172.21.15.27:0/2643422296 wait complete. 2024-03-06T07:27:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.772+0000 7f006b41c640 1 Processor -- start 2024-03-06T07:27:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.772+0000 7f006b41c640 1 -- start start 2024-03-06T07:27:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.773+0000 7f006b41c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00640cbab0 0x7f0064147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.773+0000 7f006b41c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064157b80 0x7f006414e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.773+0000 7f006b41c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0064157f60 0x7f0064147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.773+0000 7f006b41c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0064071080 con 0x7f0064157b80 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.773+0000 7f006b41c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00640cc260 con 0x7f0064157f60 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.773+0000 7f006b41c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0064071200 con 0x7f00640cbab0 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.773+0000 7f00689b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064157b80 0x7f006414e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00689b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064157b80 0x7f006414e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36108/0 (socket says 172.21.15.27:36108) 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00689b7640 1 -- 172.21.15.27:0/2909745746 learned_addr learned my addr 172.21.15.27:0/2909745746 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00691b8640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00640cbab0 0x7f0064147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00699b9640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0064157f60 0x7f0064147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00689b7640 1 -- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00640cbab0 msgr2=0x7f0064147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00689b7640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00640cbab0 0x7f0064147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00689b7640 1 -- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0064157f60 msgr2=0x7f0064147ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00689b7640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0064157f60 0x7f0064147ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00689b7640 1 -- 172.21.15.27:0/2909745746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00640cc4b0 con 0x7f0064157b80 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00699b9640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0064157f60 0x7f0064147ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f00689b7640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064157b80 0x7f006414e160 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f005400c430 tx=0x7f005400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f0059ffb640 1 -- 172.21.15.27:0/2909745746 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00540074e0 con 0x7f0064157b80 2024-03-06T07:27:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f0059ffb640 1 -- 172.21.15.27:0/2909745746 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00540047e0 con 0x7f0064157b80 2024-03-06T07:27:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00640cc790 con 0x7f0064157b80 2024-03-06T07:27:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.774+0000 7f0059ffb640 1 -- 172.21.15.27:0/2909745746 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054004bd0 con 0x7f0064157b80 2024-03-06T07:27:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.775+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00640ccd50 con 0x7f0064157b80 2024-03-06T07:27:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.775+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0064067ff0 con 0x7f0064157b80 2024-03-06T07:27:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.776+0000 7f0059ffb640 1 -- 172.21.15.27:0/2909745746 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0054011070 con 0x7f0064157b80 2024-03-06T07:27:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.776+0000 7f0059ffb640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f004806b500 0x7f004806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.776+0000 7f0059ffb640 1 -- 172.21.15.27:0/2909745746 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0054089e60 con 0x7f0064157b80 2024-03-06T07:27:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.777+0000 7f00691b8640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f004806b500 0x7f004806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.777+0000 7f00691b8640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f004806b500 0x7f004806d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0060036000 tx=0x7f0060008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.781+0000 7f0059ffb640 1 -- 172.21.15.27:0/2909745746 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0054058cf0 con 0x7f0064157b80 2024-03-06T07:27:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:53.999+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0064164f40 con 0x7f0064157b80 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.000+0000 7f0059ffb640 1 -- 172.21.15.27:0/2909745746 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f00540585f0 con 0x7f0064157b80 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:54.001 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1540177819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":1540177819},{"type":"v1","addr":"172.21.15.69:6837","nonce":1540177819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6715,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3062817242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3062817242},{"type":"v1","addr":"172.21.15.69:6835","nonce":3062817242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7425,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2584284640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2584284640},{"type":"v1","addr":"172.21.15.27:6834","nonce":2584284640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:27:51.162771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7428},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7428":{"gid":7428,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/3147032864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3147032864},{"type":"v1","addr":"172.21.15.27:6835","nonce":3147032864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7428,"qdb_cluster":[7428]},"id":7}]} 2024-03-06T07:27:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 61 2024-03-06T07:27:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f004806b500 msgr2=0x7f004806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f004806b500 0x7f004806d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0060036000 tx=0x7f0060008030 comp rx=0 tx=0).stop 2024-03-06T07:27:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064157b80 msgr2=0x7f006414e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064157b80 0x7f006414e160 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f005400c430 tx=0x7f005400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 shutdown_connections 2024-03-06T07:27:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f004806b500 0x7f004806d9c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0064157f60 0x7f0064147ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064157b80 0x7f006414e160 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 --2- 172.21.15.27:0/2909745746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00640cbab0 0x7f0064147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 >> 172.21.15.27:0/2909745746 conn(0x7f0064066d00 msgr2=0x7f0064158c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 shutdown_connections 2024-03-06T07:27:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.002+0000 7f006b41c640 1 -- 172.21.15.27:0/2909745746 wait complete. 2024-03-06T07:27:54.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:48.137883+0000', 'modified': '2024-03-06T07:27:51.162771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7428}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7428': {'gid': 7428, 'name': 'a', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/3147032864', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 3147032864}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 3147032864}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7428, 'qdb_cluster': [7428]} 2024-03-06T07:27:54.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:54.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:27:54.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.524+0000 7f5bfa14b640 1 Processor -- start 2024-03-06T07:27:54.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.524+0000 7f5bfa14b640 1 -- start start 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bfa14b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5bf406b5c0 0x7f5bf40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bfa14b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 0x7f5bf406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bfa14b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf40b3340 con 0x7f5bf40ae9b0 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bfa14b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf40cfa00 con 0x7f5bf406b5c0 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bfa14b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf40cee90 con 0x7f5bf40b0e90 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 0x7f5bf406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf3fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5bf40ae9b0 0x7f5bf40d03a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:32942/0 (socket says 172.21.15.27:32942) 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5bf406b5c0 0x7f5bf40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf3fff640 1 -- 172.21.15.27:0/564638751 learned_addr learned my addr 172.21.15.27:0/564638751 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf27fc640 1 -- 172.21.15.27:0/564638751 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051051285 0 0) 0x7f5bf40b3340 con 0x7f5bf40ae9b0 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf27fc640 1 -- 172.21.15.27:0/564638751 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc003520 con 0x7f5bf40ae9b0 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf37fe640 1 -- 172.21.15.27:0/564638751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 msgr2=0x7f5bf406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/564638751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 0x7f5bf406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf37fe640 1 -- 172.21.15.27:0/564638751 >> v1:172.21.15.69:6789/0 conn(0x7f5bf40ae9b0 legacy=0x7f5bf40d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf37fe640 1 -- 172.21.15.27:0/564638751 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf40cfc50 con 0x7f5bf406b5c0 2024-03-06T07:27:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.525+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/564638751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 0x7f5bf406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:54.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.526+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/564638751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf406b5c0 0x7f5bf40b2e00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5be40097a0 tx=0x7f5be402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a57da3f9213f187 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.526+0000 7f5bf27fc640 1 -- 172.21.15.27:0/564638751 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be40315a0 con 0x7f5bf406b5c0 2024-03-06T07:27:54.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.526+0000 7f5bf27fc640 1 -- 172.21.15.27:0/564638751 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be4037070 con 0x7f5bf406b5c0 2024-03-06T07:27:54.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.526+0000 7f5bf27fc640 1 -- 172.21.15.27:0/564638751 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4031960 con 0x7f5bf406b5c0 2024-03-06T07:27:54.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.527+0000 7f5bfa14b640 1 -- 172.21.15.27:0/564638751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf406b5c0 msgr2=0x7f5bf40b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.527+0000 7f5bfa14b640 1 --2- 172.21.15.27:0/564638751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf406b5c0 0x7f5bf40b2e00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5be40097a0 tx=0x7f5be402b840 comp rx=0 tx=0).stop 2024-03-06T07:27:54.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.527+0000 7f5bfa14b640 1 -- 172.21.15.27:0/564638751 shutdown_connections 2024-03-06T07:27:54.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.527+0000 7f5bfa14b640 1 --2- 172.21.15.27:0/564638751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 0x7f5bf406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.527+0000 7f5bfa14b640 1 --2- 172.21.15.27:0/564638751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf406b5c0 0x7f5bf40b2e00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.527+0000 7f5bfa14b640 1 -- 172.21.15.27:0/564638751 >> 172.21.15.27:0/564638751 conn(0x7f5bf40b36d0 msgr2=0x7f5bf40b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.527+0000 7f5bfa14b640 1 -- 172.21.15.27:0/564638751 shutdown_connections 2024-03-06T07:27:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.527+0000 7f5bfa14b640 1 -- 172.21.15.27:0/564638751 wait complete. 2024-03-06T07:27:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.529+0000 7f5bfa14b640 1 Processor -- start 2024-03-06T07:27:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.529+0000 7f5bfa14b640 1 -- start start 2024-03-06T07:27:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bfa14b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf406b5c0 0x7f5bf40c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bfa14b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf40ae9b0 0x7f5bf41f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bfa14b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 0x7f5bf41f4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bfa14b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf406c790 con 0x7f5bf406b5c0 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bfa14b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf406bee0 con 0x7f5bf40ae9b0 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bfa14b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf406c910 con 0x7f5bf40b0e90 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf406b5c0 0x7f5bf40c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf406b5c0 0x7f5bf40c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36118/0 (socket says 172.21.15.27:36118) 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf37fe640 1 -- 172.21.15.27:0/3159114673 learned_addr learned my addr 172.21.15.27:0/3159114673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf40ae9b0 0x7f5bf41f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf3fff640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 0x7f5bf41f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf37fe640 1 -- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 msgr2=0x7f5bf41f4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 0x7f5bf41f4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf37fe640 1 -- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf40ae9b0 msgr2=0x7f5bf41f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf40ae9b0 0x7f5bf41f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf37fe640 1 -- 172.21.15.27:0/3159114673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf41f4740 con 0x7f5bf406b5c0 2024-03-06T07:27:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.531+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf406b5c0 0x7f5bf40c7fd0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5be4034040 tx=0x7f5be40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.532+0000 7f5bdbfff640 1 -- 172.21.15.27:0/3159114673 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4032630 con 0x7f5bf406b5c0 2024-03-06T07:27:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.532+0000 7f5bdbfff640 1 -- 172.21.15.27:0/3159114673 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be40327d0 con 0x7f5bf406b5c0 2024-03-06T07:27:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.532+0000 7f5bfa14b640 1 -- 172.21.15.27:0/3159114673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf41f4a20 con 0x7f5bf406b5c0 2024-03-06T07:27:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.532+0000 7f5bdbfff640 1 -- 172.21.15.27:0/3159114673 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4032bc0 con 0x7f5bf406b5c0 2024-03-06T07:27:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.532+0000 7f5bfa14b640 1 -- 172.21.15.27:0/3159114673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf41f52e0 con 0x7f5bf406b5c0 2024-03-06T07:27:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.533+0000 7f5bdbfff640 1 -- 172.21.15.27:0/3159114673 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5be404e430 con 0x7f5bf406b5c0 2024-03-06T07:27:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.534+0000 7f5bdbfff640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.534+0000 7f5bdbfff640 1 -- 172.21.15.27:0/3159114673 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5be402f030 con 0x7f5bf406b5c0 2024-03-06T07:27:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.534+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.534+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009560 tx=0x7f5be8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.535+0000 7f5bfce0c740 1 Processor -- start 2024-03-06T07:27:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.535+0000 7f5bfce0c740 1 -- start start 2024-03-06T07:27:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf8676870 0x55ecf8676c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf867a8c0 0x55ecf867c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 0x55ecf86a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf85e06b0 con 0x55ecf8676870 2024-03-06T07:27:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf8589a60 con 0x55ecf86a6600 2024-03-06T07:27:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecf85dc750 con 0x55ecf867a8c0 2024-03-06T07:27:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf8676870 0x55ecf8676c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf8676870 0x55ecf8676c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36122/0 (socket says 172.21.15.27:36122) 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf37fe640 1 -- 172.21.15.27:0/3067320532 learned_addr learned my addr 172.21.15.27:0/3067320532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/3067320532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf867a8c0 0x55ecf867c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf3fff640 1 --2- 172.21.15.27:0/3067320532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 0x55ecf86a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf37fe640 1 -- 172.21.15.27:0/3067320532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf867a8c0 msgr2=0x55ecf867c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/3067320532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf867a8c0 0x55ecf867c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf37fe640 1 -- 172.21.15.27:0/3067320532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 msgr2=0x55ecf86a6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/3067320532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 0x55ecf86a6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf37fe640 1 -- 172.21.15.27:0/3067320532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecf85a7f30 con 0x55ecf8676870 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/3067320532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf8676870 0x55ecf8676c50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f5be40aa370 tx=0x7f5be4079fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4f1e560165f9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bd97fa640 1 -- 172.21.15.27:0/3067320532 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4082cb0 con 0x55ecf8676870 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bd97fa640 1 -- 172.21.15.27:0/3067320532 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be4037070 con 0x55ecf8676870 2024-03-06T07:27:54.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.536+0000 7f5bd97fa640 1 -- 172.21.15.27:0/3067320532 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4032d60 con 0x55ecf8676870 2024-03-06T07:27:54.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.537+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3067320532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf8676870 msgr2=0x55ecf8676c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.537+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3067320532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf8676870 0x55ecf8676c50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f5be40aa370 tx=0x7f5be4079fb0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.537+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3067320532 shutdown_connections 2024-03-06T07:27:54.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.537+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3067320532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 0x55ecf86a6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.537+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3067320532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf867a8c0 0x55ecf867c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.537+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3067320532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf8676870 0x55ecf8676c50 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.537+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3067320532 >> 172.21.15.27:0/3067320532 conn(0x55ecf86788f0 msgr2=0x55ecf867a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.537+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3067320532 shutdown_connections 2024-03-06T07:27:54.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.537+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3067320532 wait complete. 2024-03-06T07:27:54.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.538+0000 7f5bfce0c740 1 -- start start 2024-03-06T07:27:54.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.538+0000 7f5bfce0c740 1 Processor -- start 2024-03-06T07:27:54.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 0x55ecf869a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf869d500 0x55ecf86c2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a2190 0x55ecf86a2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf8589a60 con 0x55ecf869d500 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf859cea0 con 0x55ecf86a6600 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecf85e06b0 con 0x55ecf86a2190 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bfce0c740 10 client.0 _ll_register_callbacks cb 0x55ecf859c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf869d500 0x55ecf86c2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf869d500 0x55ecf86c2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36124/0 (socket says 172.21.15.27:36124) 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf2ffd640 1 -- 172.21.15.27:0/3415540015 learned_addr learned my addr 172.21.15.27:0/3415540015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf3fff640 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a2190 0x55ecf86a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 0x55ecf869a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf2ffd640 1 -- 172.21.15.27:0/3415540015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a2190 msgr2=0x55ecf86a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a2190 0x55ecf86a2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf2ffd640 1 -- 172.21.15.27:0/3415540015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 msgr2=0x55ecf869a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 0x55ecf869a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf2ffd640 1 -- 172.21.15.27:0/3415540015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecf859d270 con 0x55ecf869d500 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf869d500 0x55ecf86c2040 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f5be8003640 tx=0x7f5be80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edee3f79315d0a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bd27fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.539+0000 7f5bd27fc640 1 -- 172.21.15.27:0/3415540015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be8027070 con 0x55ecf869d500 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bd27fc640 1 -- 172.21.15.27:0/3415540015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5be8014650 con 0x55ecf869d500 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3415540015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5be4037070 con 0x55ecf869d500 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bd27fc640 1 -- 172.21.15.27:0/3415540015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5be8032050 con 0x55ecf869d500 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bfce0c740 10 client.7554 fetch_fsmap learned FSMap version 61 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3415540015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55ecf82f52d0 con 0x55ecf869d500 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bd27fc640 1 -- 172.21.15.27:0/3415540015 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be4037070 con 0x55ecf869d500 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bd27fc640 1 -- 172.21.15.27:0/3415540015 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5be8035020 con 0x55ecf869d500 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bfce0c740 10 client.7554 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bfce0c740 10 client.7554 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bfce0c740 20 client.7554 populate_metadata read hostname 'smithi027' 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] conn(0x55ecf86a1330 0x55ecf86e02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bfce0c740 4 client.7554 mds_command: new command op to 7428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3415540015 --> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecf85edbf0 con 0x55ecf86a1330 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.540+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] conn(0x55ecf86a1330 0x55ecf86e02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] conn(0x55ecf86a1330 0x55ecf86e02f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92afda9a63b5a8d server_cookie=d89f9b904ac157f6 in_seq=0 out_seq=0 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bd27fc640 10 client.7554 ms_handle_connect on v2:172.21.15.27:6833/3147032864 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bd27fc640 1 -- 172.21.15.27:0/3415540015 <== mds.0 v2:172.21.15.27:6833/3147032864 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ecf85edbf0 con 0x55ecf86a1330 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bd27fc640 10 client.7554 handle_command_reply: tid=0 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 client.7554 shutdown 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3415540015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf869d500 msgr2=0x55ecf86c2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf869d500 0x55ecf86c2040 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f5be8003640 tx=0x7f5be80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3415540015 shutdown_connections 2024-03-06T07:27:54.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a2190 0x55ecf86a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] conn(0x55ecf86a1330 0x55ecf86e02f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf869d500 0x55ecf86c2040 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/3415540015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a6600 0x55ecf869a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bd27fc640 0 client.7554 ms_handle_reset on v2:172.21.15.27:6833/3147032864 2024-03-06T07:27:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3415540015 >> 172.21.15.27:0/3415540015 conn(0x55ecf8676870 msgr2=0x55ecf86a45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3415540015 shutdown_connections 2024-03-06T07:27:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 1 -- 172.21.15.27:0/3415540015 wait complete. 2024-03-06T07:27:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.541+0000 7f5bfce0c740 20 client.7554 trim_cache size 0 max 16384 2024-03-06T07:27:54.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.558+0000 7f5bfce0c740 1 Processor -- start 2024-03-06T07:27:54.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.558+0000 7f5bfce0c740 1 -- start start 2024-03-06T07:27:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf869d500 0x55ecf86c1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a2190 0x55ecf86c16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a6600 0x55ecf867b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf41ffe90 con 0x55ecf86a6600 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf41ff030 con 0x55ecf86a2190 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf41fe460 con 0x55ecf869d500 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bf3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a6600 0x55ecf867b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bf3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a6600 0x55ecf867b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36128/0 (socket says 172.21.15.27:36128) 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bf3fff640 1 -- 172.21.15.27:0/509571364 learned_addr learned my addr 172.21.15.27:0/509571364 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/509571364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a2190 0x55ecf86c16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.559+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/509571364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf869d500 0x55ecf86c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bf3fff640 1 -- 172.21.15.27:0/509571364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf869d500 msgr2=0x55ecf86c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bf3fff640 1 --2- 172.21.15.27:0/509571364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf869d500 0x55ecf86c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bf3fff640 1 -- 172.21.15.27:0/509571364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a2190 msgr2=0x55ecf86c16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bf3fff640 1 --2- 172.21.15.27:0/509571364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a2190 0x55ecf86c16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bf3fff640 1 -- 172.21.15.27:0/509571364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecf83bf6b0 con 0x55ecf86a6600 2024-03-06T07:27:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bf3fff640 1 --2- 172.21.15.27:0/509571364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a6600 0x55ecf867b3f0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f5bec002110 tx=0x7f5bec007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0f9fc1e545da22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bd8ff9640 1 -- 172.21.15.27:0/509571364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec016630 con 0x55ecf86a6600 2024-03-06T07:27:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bd8ff9640 1 -- 172.21.15.27:0/509571364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec01a030 con 0x55ecf86a6600 2024-03-06T07:27:54.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bd8ff9640 1 -- 172.21.15.27:0/509571364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec016a10 con 0x55ecf86a6600 2024-03-06T07:27:54.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bfce0c740 1 -- 172.21.15.27:0/509571364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a6600 msgr2=0x55ecf867b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.560+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/509571364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a6600 0x55ecf867b3f0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f5bec002110 tx=0x7f5bec007410 comp rx=0 tx=0).stop 2024-03-06T07:27:54.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.561+0000 7f5bfce0c740 1 -- 172.21.15.27:0/509571364 shutdown_connections 2024-03-06T07:27:54.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.561+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/509571364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a6600 0x55ecf867b3f0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.561+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/509571364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf86a2190 0x55ecf86c16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.561+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/509571364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf869d500 0x55ecf86c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.561+0000 7f5bfce0c740 1 -- 172.21.15.27:0/509571364 >> 172.21.15.27:0/509571364 conn(0x55ecf8676870 msgr2=0x55ecf86a3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.561+0000 7f5bfce0c740 1 -- 172.21.15.27:0/509571364 shutdown_connections 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.561+0000 7f5bfce0c740 1 -- 172.21.15.27:0/509571364 wait complete. 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.561+0000 7f5bfce0c740 1 -- start start 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.561+0000 7f5bfce0c740 1 Processor -- start 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf869d500 0x55ecf869f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a2190 0x55ecf869fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bfce0c740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a6600 0x55ecf86a73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecf869b010 con 0x55ecf86a2190 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf41fea60 con 0x55ecf869d500 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bfce0c740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecf869da00 con 0x55ecf86a6600 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bfce0c740 10 client.0 _ll_register_callbacks cb 0x7f5be8004320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a2190 0x55ecf869fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a2190 0x55ecf869fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36138/0 (socket says 172.21.15.27:36138) 2024-03-06T07:27:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf2ffd640 1 -- 172.21.15.27:0/4077117855 learned_addr learned my addr 172.21.15.27:0/4077117855 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf3fff640 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a6600 0x55ecf86a73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf869d500 0x55ecf869f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf2ffd640 1 -- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a6600 msgr2=0x55ecf86a73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a6600 0x55ecf86a73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf2ffd640 1 -- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf869d500 msgr2=0x55ecf869f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf869d500 0x55ecf869f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf2ffd640 1 -- 172.21.15.27:0/4077117855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecf86c2da0 con 0x55ecf86a2190 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf3fff640 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a6600 0x55ecf86a73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.562+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf869d500 0x55ecf869f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bf2ffd640 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a2190 0x55ecf869fbd0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f5be8003640 tx=0x7f5be8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b432f8c22c2bcdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bd27fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bd27fc640 1 -- 172.21.15.27:0/4077117855 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be8027040 con 0x55ecf86a2190 2024-03-06T07:27:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bd27fc640 1 -- 172.21.15.27:0/4077117855 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5be802c650 con 0x55ecf86a2190 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bfce0c740 1 -- 172.21.15.27:0/4077117855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5bec01a030 con 0x55ecf86a2190 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bd27fc640 1 -- 172.21.15.27:0/4077117855 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5be8043050 con 0x55ecf86a2190 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bfce0c740 10 client.7560 fetch_fsmap learned FSMap version 61 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bfce0c740 1 -- 172.21.15.27:0/4077117855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55ecf86d09e0 con 0x55ecf86a2190 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bd27fc640 1 -- 172.21.15.27:0/4077117855 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec01a030 con 0x55ecf86a2190 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bd27fc640 1 -- 172.21.15.27:0/4077117855 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5be8046020 con 0x55ecf86a2190 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bfce0c740 10 client.7560 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bfce0c740 10 client.7560 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bfce0c740 20 client.7560 populate_metadata read hostname 'smithi027' 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] conn(0x55ecf8731110 0x55ecf8734650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bfce0c740 4 client.7560 mds_command: new command op to 7428 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bfce0c740 1 -- 172.21.15.27:0/4077117855 --> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ecf85e7f40 con 0x55ecf8731110 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.563+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] conn(0x55ecf8731110 0x55ecf8734650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bd27fc640 10 client.7560 ms_handle_connect on v2:172.21.15.27:6833/3147032864 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bf37fe640 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] conn(0x55ecf8731110 0x55ecf8734650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=352a3313f658439 server_cookie=8760611c738ffe25 in_seq=0 out_seq=0 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bd27fc640 1 -- 172.21.15.27:0/4077117855 <== mds.0 v2:172.21.15.27:6833/3147032864 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55ecf85e7f40 con 0x55ecf8731110 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bd27fc640 10 client.7560 handle_command_reply: tid=0 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 client.7560 shutdown 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 -- 172.21.15.27:0/4077117855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a2190 msgr2=0x55ecf869fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a2190 0x55ecf869fbd0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f5be8003640 tx=0x7f5be8019000 comp rx=0 tx=0).stop 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 -- 172.21.15.27:0/4077117855 shutdown_connections 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55ecf86a6600 0x55ecf86a73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.27:6833/3147032864,v1:172.21.15.27:6835/3147032864] conn(0x55ecf8731110 0x55ecf8734650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecf86a2190 0x55ecf869fbd0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 --2- 172.21.15.27:0/4077117855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ecf869d500 0x55ecf869f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bd27fc640 0 client.7560 ms_handle_reset on v2:172.21.15.27:6833/3147032864 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 -- 172.21.15.27:0/4077117855 >> 172.21.15.27:0/4077117855 conn(0x55ecf8676870 msgr2=0x55ecf86a3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 -- 172.21.15.27:0/4077117855 shutdown_connections 2024-03-06T07:27:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 1 -- 172.21.15.27:0/4077117855 wait complete. 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.564+0000 7f5bfce0c740 20 client.7560 trim_cache size 0 max 16384 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7428, 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 5.7911745129999996, 2024-03-06T07:27:54.572 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 61, 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 98, 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 98, 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 17.798026468 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:27:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 -- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 msgr2=0x7f5be006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009560 tx=0x7f5be8009190 comp rx=0 tx=0).stop 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 -- 172.21.15.27:0/3159114673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf406b5c0 msgr2=0x7f5bf40c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf406b5c0 0x7f5bf40c7fd0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5be4034040 tx=0x7f5be40046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 -- 172.21.15.27:0/3159114673 shutdown_connections 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bf40b0e90 0x7f5bf41f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf40ae9b0 0x7f5bf41f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 --2- 172.21.15.27:0/3159114673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf406b5c0 0x7f5bf40c7fd0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 -- 172.21.15.27:0/3159114673 >> 172.21.15.27:0/3159114673 conn(0x7f5bf40b36d0 msgr2=0x7f5bf40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 -- 172.21.15.27:0/3159114673 shutdown_connections 2024-03-06T07:27:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.565+0000 7f5bfa14b640 1 -- 172.21.15.27:0/3159114673 wait complete. 2024-03-06T07:27:54.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:54.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.683+0000 7fc35f525640 1 Processor -- start 2024-03-06T07:27:54.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.683+0000 7fc35f525640 1 -- start start 2024-03-06T07:27:54.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc35f525640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc358071250 0x7fc35815b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc35f525640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35806c7b0 con 0x7fc358157b80 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc35f525640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35806ca90 con 0x7fc358157f60 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc35f525640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35806cd70 con 0x7fc358071250 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc35d2c1640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc358157b80 0x7fc35815b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44468/0 (socket says 172.21.15.27:44468) 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc35d2c1640 1 -- 172.21.15.27:0/1498381875 learned_addr learned my addr 172.21.15.27:0/1498381875 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc35dac2640 1 --2- 172.21.15.27:0/1498381875 >> v2:172.21.15.69:3300/0 conn(0x7fc358071250 0x7fc35815b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570617383 0 0) 0x7fc35806c7b0 con 0x7fc358157b80 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc344003520 con 0x7fc358157b80 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835575193 0 0) 0x7fc35806ca90 con 0x7fc358157f60 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35806c7b0 con 0x7fc358157f60 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 149106985 0 0) 0x7fc344003520 con 0x7fc358157b80 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 >> v2:172.21.15.69:3300/0 conn(0x7fc358071250 msgr2=0x7fc35815b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 --2- 172.21.15.27:0/1498381875 >> v2:172.21.15.69:3300/0 conn(0x7fc358071250 0x7fc35815b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 >> v1:172.21.15.69:6789/0 conn(0x7fc358157f60 legacy=0x7fc358070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35806d050 con 0x7fc358157b80 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fc354001670 con 0x7fc358157b80 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc35dac2640 1 --2- 172.21.15.27:0/1498381875 >> v2:172.21.15.69:3300/0 conn(0x7fc358071250 0x7fc35815b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.684+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 >> v1:172.21.15.27:6789/0 conn(0x7fc358157b80 legacy=0x7fc35815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc34ffff640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc344003b40 0x7fc344003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc34ffff640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc344004800 0x7fc344004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc34ffff640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3440051c0 0x7fc34400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35806cd70 con 0x7fc344003b40 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34400e250 con 0x7fc344004800 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34400e450 con 0x7fc3440051c0 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc35d2c1640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc344003b40 0x7fc344003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc35cac0640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc344004800 0x7fc344004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc35dac2640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3440051c0 0x7fc34400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc35d2c1640 1 -- 172.21.15.27:0/1498381875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3440051c0 msgr2=0x7fc34400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc35d2c1640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3440051c0 0x7fc34400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc35d2c1640 1 -- 172.21.15.27:0/1498381875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc344004800 msgr2=0x7fc344004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc35d2c1640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc344004800 0x7fc344004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc35d2c1640 1 -- 172.21.15.27:0/1498381875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc344003520 con 0x7fc344003b40 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.685+0000 7fc35dac2640 1 -- 172.21.15.27:0/1498381875 reap_dead start 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.686+0000 7fc35d2c1640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc344003b40 0x7fc344003f40 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc354007fd0 tx=0x7fc344005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d245ba7c76a3894 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.686+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354007ca0 con 0x7fc344003b40 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.686+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc354030070 con 0x7fc344003b40 2024-03-06T07:27:54.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.686+0000 7fc34ffff640 1 -- 172.21.15.27:0/1498381875 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354005d70 con 0x7fc344003b40 2024-03-06T07:27:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.687+0000 7fc35f525640 1 -- 172.21.15.27:0/1498381875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc344003b40 msgr2=0x7fc344003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.687+0000 7fc35f525640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc344003b40 0x7fc344003f40 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc354007fd0 tx=0x7fc344005830 comp rx=0 tx=0).stop 2024-03-06T07:27:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.687+0000 7fc35f525640 1 -- 172.21.15.27:0/1498381875 shutdown_connections 2024-03-06T07:27:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.687+0000 7fc35f525640 1 --2- 172.21.15.27:0/1498381875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc344003b40 0x7fc344003f40 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.687+0000 7fc35f525640 1 -- 172.21.15.27:0/1498381875 >> 172.21.15.27:0/1498381875 conn(0x7fc358066d00 msgr2=0x7fc3580670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.687+0000 7fc35f525640 1 -- 172.21.15.27:0/1498381875 shutdown_connections 2024-03-06T07:27:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.687+0000 7fc35f525640 1 -- 172.21.15.27:0/1498381875 wait complete. 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.689+0000 7fc35f525640 1 Processor -- start 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.689+0000 7fc35f525640 1 -- start start 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.690+0000 7fc35f525640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc344003b40 0x7fc35814da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.690+0000 7fc35f525640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3580b2cf0 0x7fc3580b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.690+0000 7fc35f525640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3580b1ae0 0x7fc3580b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.690+0000 7fc35f525640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580cbab0 con 0x7fc3580b2cf0 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.690+0000 7fc35f525640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580cc260 con 0x7fc3580b1ae0 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.690+0000 7fc35f525640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580cbc30 con 0x7fc344003b40 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.690+0000 7fc35cac0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3580b2cf0 0x7fc3580b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.690+0000 7fc35cac0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3580b2cf0 0x7fc3580b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36158/0 (socket says 172.21.15.27:36158) 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.690+0000 7fc35cac0640 1 -- 172.21.15.27:0/2354701154 learned_addr learned my addr 172.21.15.27:0/2354701154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35d2c1640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc344003b40 0x7fc35814da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35dac2640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3580b1ae0 0x7fc3580b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35cac0640 1 -- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc344003b40 msgr2=0x7fc35814da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35cac0640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc344003b40 0x7fc35814da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35cac0640 1 -- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3580b1ae0 msgr2=0x7fc3580b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35cac0640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3580b1ae0 0x7fc3580b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35cac0640 1 -- 172.21.15.27:0/2354701154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3581ed370 con 0x7fc3580b2cf0 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35dac2640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3580b1ae0 0x7fc3580b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35cac0640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3580b2cf0 0x7fc3580b15a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc34800f7d0 tx=0x7fc34800fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc34dffb640 1 -- 172.21.15.27:0/2354701154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34801a070 con 0x7fc3580b2cf0 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc34dffb640 1 -- 172.21.15.27:0/2354701154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc348005110 con 0x7fc3580b2cf0 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35814e3a0 con 0x7fc3580b2cf0 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc34dffb640 1 -- 172.21.15.27:0/2354701154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348007050 con 0x7fc3580b2cf0 2024-03-06T07:27:54.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.691+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35814e960 con 0x7fc3580b2cf0 2024-03-06T07:27:54.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.692+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc358067ff0 con 0x7fc3580b2cf0 2024-03-06T07:27:54.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.693+0000 7fc34dffb640 1 -- 172.21.15.27:0/2354701154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc34801d070 con 0x7fc3580b2cf0 2024-03-06T07:27:54.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.693+0000 7fc34dffb640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc33c06b4d0 0x7fc33c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.693+0000 7fc34dffb640 1 -- 172.21.15.27:0/2354701154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc348089820 con 0x7fc3580b2cf0 2024-03-06T07:27:54.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.693+0000 7fc35d2c1640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc33c06b4d0 0x7fc33c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.694+0000 7fc35d2c1640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc33c06b4d0 0x7fc33c06d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc354034000 tx=0x7fc354007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.698+0000 7fc34dffb640 1 -- 172.21.15.27:0/2354701154 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3581ed370 con 0x7fc3580b2cf0 2024-03-06T07:27:54.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.919+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc3581f0b80 con 0x7fc3580b2cf0 2024-03-06T07:27:54.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.920+0000 7fc34dffb640 1 -- 172.21.15.27:0/2354701154 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc3480580e0 con 0x7fc3580b2cf0 2024-03-06T07:27:54.921 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:54.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc33c06b4d0 msgr2=0x7fc33c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc33c06b4d0 0x7fc33c06d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc354034000 tx=0x7fc354007000 comp rx=0 tx=0).stop 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3580b2cf0 msgr2=0x7fc3580b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3580b2cf0 0x7fc3580b15a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc34800f7d0 tx=0x7fc34800fc90 comp rx=0 tx=0).stop 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 shutdown_connections 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc33c06b4d0 0x7fc33c06d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3580b1ae0 0x7fc3580b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3580b2cf0 0x7fc3580b15a0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 --2- 172.21.15.27:0/2354701154 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc344003b40 0x7fc35814da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 >> 172.21.15.27:0/2354701154 conn(0x7fc358066d00 msgr2=0x7fc35815a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 shutdown_connections 2024-03-06T07:27:54.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:54.922+0000 7fc35f525640 1 -- 172.21.15.27:0/2354701154 wait complete. 2024-03-06T07:27:54.934 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:54.934 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:54.934 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:27:54.934 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:54.934 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:54.934 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:54.934 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:54.934 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:54.934 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:54.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:54.935 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:27:54.950 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:54.950 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:54.950 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:54.950 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:54.950 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:54.950 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42415sec preferred_lft 42415sec 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:54.951 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:27:54.952 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:54.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:54.953 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:55.019 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:27:55.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.021 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:27:55.077 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:27:55.077 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:55.077 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:27:55.077 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:27:55.077 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:55.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.077 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:27:55.078 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:27:55.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:27:55.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:27:55.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:27:55.078 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:27:55.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.183 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:27:55.198 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:55.199 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:55.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:27:55.269 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:55.270 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:55.270 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:55.325 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:55.325 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:55.325 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:55.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:27:55.449 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:55.506 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:27:55.506 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:27:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:55.508 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:55.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.509 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:27:55.564 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:55.565 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:55.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:27:55.637 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:27:55.694 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:55.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.696 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:27:55.702 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:27:55.700+0000 7fd6e6c49480 -1 init, newargv = 0x56493bb93650 newargc=15 2024-03-06T07:27:55.702 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[68919]: starting ceph client 2024-03-06T07:27:55.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[68919]: starting fuse 2024-03-06T07:27:55.717 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:27:55.717 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:27:55.718 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:55.719 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:27:55.845 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.68919.asok 2024-03-06T07:27:55.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:55.858 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.68919.asok 2024-03-06T07:27:55.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.68919.asok status 2024-03-06T07:27:56.000 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "68919", 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7575, 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:27:56.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 7575 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2978425134 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2978425134 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.7575 192.168.0.1:0/2978425134", 2024-03-06T07:27:56.002 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/2978425134", 2024-03-06T07:27:56.003 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:27:56.003 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 61, 2024-03-06T07:27:56.003 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 98, 2024-03-06T07:27:56.003 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:56.003 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:27:56.003 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:27:56.003 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:27:56.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:56.010 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:27:56.010 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:56.026 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:27:56.026 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:56.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:56.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:56.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:56.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:56.197 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-03-06T07:27:56.197 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-03-06T07:27:56.197 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:56.198 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:56.199 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:27:56.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:56.201 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-03-06T07:27:56.202 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:56.203 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.204 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:56.205 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-03-06T07:27:56.206 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-03-06T07:27:56.207 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.208 INFO:teuthology.orchestra.run.smithi069.stdout: mds_max_purge_files = 6 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.209 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:56.210 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-06T07:27:56.211 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.212 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-06T07:27:56.214 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.215 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.216 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.217 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-03-06T07:27:56.218 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.219 INFO:teuthology.orchestra.run.smithi027.stdout: mds_max_purge_files = 6 2024-03-06T07:27:56.221 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:56.263 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:56.316 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-03-06T07:27:56.316 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-03-06T07:27:56.316 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.317 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:56.318 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-03-06T07:27:56.319 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:56.320 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-03-06T07:27:56.321 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.322 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.323 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:56.324 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-03-06T07:27:56.325 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-03-06T07:27:56.326 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: mds_max_purge_files = 6 2024-03-06T07:27:56.327 INFO:teuthology.orchestra.run.smithi069.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-06T07:27:56.328 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:56.329 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-06T07:27:56.330 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:56.331 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-06T07:27:56.332 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-03-06T07:27:56.333 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.334 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:56.335 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:56.336 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-03-06T07:27:56.337 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-03-06T07:27:56.338 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: mds_max_purge_files = 6 2024-03-06T07:27:56.339 INFO:teuthology.orchestra.run.smithi027.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:27:56.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:56.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.489+0000 7fd734888640 1 Processor -- start 2024-03-06T07:27:56.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.489+0000 7fd734888640 1 -- start start 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.489+0000 7fd734888640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd73015e080 0x7fd7300d4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.489+0000 7fd734888640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd73015e460 0x7fd7300d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.489+0000 7fd734888640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd7300d1cc0 0x7fd7300d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.489+0000 7fd734888640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7300b36f0 con 0x7fd73015e460 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.489+0000 7fd734888640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7300cfd30 con 0x7fd7300d1cc0 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.489+0000 7fd734888640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300cf490 con 0x7fd73015e080 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72dd9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd73015e460 0x7fd7300d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72dd9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd73015e460 0x7fd7300d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36170/0 (socket says 172.21.15.27:36170) 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72dd9b640 1 -- 172.21.15.27:0/1820944741 learned_addr learned my addr 172.21.15.27:0/1820944741 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:56.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72ed9d640 1 --2- 172.21.15.27:0/1820944741 >> v2:172.21.15.69:3300/0 conn(0x7fd7300d1cc0 0x7fd7300d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72dd9b640 1 -- 172.21.15.27:0/1820944741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd73015e080 msgr2=0x7fd7300d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72e59c640 1 --2- 172.21.15.27:0/1820944741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd73015e080 0x7fd7300d4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72dd9b640 1 --2- 172.21.15.27:0/1820944741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd73015e080 0x7fd7300d4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72dd9b640 1 -- 172.21.15.27:0/1820944741 >> v2:172.21.15.69:3300/0 conn(0x7fd7300d1cc0 msgr2=0x7fd7300d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72dd9b640 1 --2- 172.21.15.27:0/1820944741 >> v2:172.21.15.69:3300/0 conn(0x7fd7300d1cc0 0x7fd7300d2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72dd9b640 1 -- 172.21.15.27:0/1820944741 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7300cf770 con 0x7fd73015e460 2024-03-06T07:27:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72ed9d640 1 --2- 172.21.15.27:0/1820944741 >> v2:172.21.15.69:3300/0 conn(0x7fd7300d1cc0 0x7fd7300d2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.490+0000 7fd72e59c640 1 --2- 172.21.15.27:0/1820944741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd73015e080 0x7fd7300d4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.491+0000 7fd72dd9b640 1 --2- 172.21.15.27:0/1820944741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 0x7fd7300d2450 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fd7300d1930 tx=0x7fd72002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b12a20bbc8d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.491+0000 7fd72d59a640 1 -- 172.21.15.27:0/1820944741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720034070 con 0x7fd73015e460 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.491+0000 7fd72d59a640 1 -- 172.21.15.27:0/1820944741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd720030860 con 0x7fd73015e460 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.491+0000 7fd72d59a640 1 -- 172.21.15.27:0/1820944741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720030c30 con 0x7fd73015e460 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.492+0000 7fd734888640 1 -- 172.21.15.27:0/1820944741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 msgr2=0x7fd7300d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.492+0000 7fd734888640 1 --2- 172.21.15.27:0/1820944741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 0x7fd7300d2450 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fd7300d1930 tx=0x7fd72002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.492+0000 7fd734888640 1 -- 172.21.15.27:0/1820944741 shutdown_connections 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.492+0000 7fd734888640 1 --2- 172.21.15.27:0/1820944741 >> v2:172.21.15.69:3300/0 conn(0x7fd7300d1cc0 0x7fd7300d2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.492+0000 7fd734888640 1 --2- 172.21.15.27:0/1820944741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 0x7fd7300d2450 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.492+0000 7fd734888640 1 --2- 172.21.15.27:0/1820944741 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd73015e080 0x7fd7300d4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.492+0000 7fd734888640 1 -- 172.21.15.27:0/1820944741 >> 172.21.15.27:0/1820944741 conn(0x7fd7300d3e70 msgr2=0x7fd7300d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.492+0000 7fd734888640 1 -- 172.21.15.27:0/1820944741 shutdown_connections 2024-03-06T07:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.492+0000 7fd734888640 1 -- 172.21.15.27:0/1820944741 wait complete. 2024-03-06T07:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.494+0000 7fd734888640 1 Processor -- start 2024-03-06T07:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.494+0000 7fd734888640 1 -- start start 2024-03-06T07:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.494+0000 7fd734888640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300d1cc0 0x7fd7300cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.494+0000 7fd734888640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015e080 0x7fd7300b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd734888640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 0x7fd7300b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd734888640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300d04a0 con 0x7fd73015e460 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd734888640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300b3380 con 0x7fd73015e080 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd734888640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300d0620 con 0x7fd7300d1cc0 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 0x7fd7300b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 0x7fd7300b6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36180/0 (socket says 172.21.15.27:36180) 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72ed9d640 1 -- 172.21.15.27:0/405926817 learned_addr learned my addr 172.21.15.27:0/405926817 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72dd9b640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015e080 0x7fd7300b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72e59c640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300d1cc0 0x7fd7300cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72ed9d640 1 -- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300d1cc0 msgr2=0x7fd7300cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72ed9d640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300d1cc0 0x7fd7300cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72ed9d640 1 -- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015e080 msgr2=0x7fd7300b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72ed9d640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015e080 0x7fd7300b6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72ed9d640 1 -- 172.21.15.27:0/405926817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7300cc450 con 0x7fd73015e460 2024-03-06T07:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd72ed9d640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 0x7fd7300b6f90 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd72800ce70 tx=0x7fd72800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd71effd640 1 -- 172.21.15.27:0/405926817 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72801a070 con 0x7fd73015e460 2024-03-06T07:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd71effd640 1 -- 172.21.15.27:0/405926817 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72800d840 con 0x7fd73015e460 2024-03-06T07:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7300cc730 con 0x7fd73015e460 2024-03-06T07:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.495+0000 7fd71effd640 1 -- 172.21.15.27:0/405926817 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72800dc30 con 0x7fd73015e460 2024-03-06T07:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.496+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7300cccf0 con 0x7fd73015e460 2024-03-06T07:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.497+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7300751b0 con 0x7fd73015e460 2024-03-06T07:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.497+0000 7fd71effd640 1 -- 172.21.15.27:0/405926817 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd72801d070 con 0x7fd73015e460 2024-03-06T07:27:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.498+0000 7fd71effd640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd71006b500 0x7fd71006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.498+0000 7fd71effd640 1 -- 172.21.15.27:0/405926817 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd72808a8e0 con 0x7fd73015e460 2024-03-06T07:27:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.498+0000 7fd72e59c640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd71006b500 0x7fd71006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.498+0000 7fd72e59c640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd71006b500 0x7fd71006d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd72400ab60 tx=0x7fd724008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.502+0000 7fd71effd640 1 -- 172.21.15.27:0/405926817 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd728059110 con 0x7fd73015e460 2024-03-06T07:27:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.681+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd73006ca10 con 0x7fd73015e460 2024-03-06T07:27:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.682+0000 7fd71effd640 1 -- 172.21.15.27:0/405926817 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd72805d1b0 con 0x7fd73015e460 2024-03-06T07:27:56.683 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:27:56.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd71006b500 msgr2=0x7fd71006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd71006b500 0x7fd71006d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd72400ab60 tx=0x7fd724008040 comp rx=0 tx=0).stop 2024-03-06T07:27:56.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 msgr2=0x7fd7300b6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 0x7fd7300b6f90 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd72800ce70 tx=0x7fd72800a710 comp rx=0 tx=0).stop 2024-03-06T07:27:56.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 shutdown_connections 2024-03-06T07:27:56.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd71006b500 0x7fd71006d9c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73015e460 0x7fd7300b6f90 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73015e080 0x7fd7300b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 --2- 172.21.15.27:0/405926817 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd7300d1cc0 0x7fd7300cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 >> 172.21.15.27:0/405926817 conn(0x7fd7300d3e70 msgr2=0x7fd7300d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 shutdown_connections 2024-03-06T07:27:56.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.684+0000 7fd734888640 1 -- 172.21.15.27:0/405926817 wait complete. 2024-03-06T07:27:56.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:56.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:56.753 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:56.754 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:56.755 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:27:56.756 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.757 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-03-06T07:27:56.758 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-03-06T07:27:56.759 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-03-06T07:27:56.760 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.761 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-03-06T07:27:56.762 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.763 INFO:teuthology.orchestra.run.smithi069.stdout: mds_max_purge_files = 6 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi069.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi069.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.764 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.765 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-06T07:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:56.767 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:56.768 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:56.769 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-06T07:27:56.770 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:56.771 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:56.772 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-03-06T07:27:56.773 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: mds_max_purge_files = 6 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:27:56.774 INFO:teuthology.orchestra.run.smithi027.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:56.777 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:56.778 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:56.778 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:56.778 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:56.778 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:56.778 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:56.779 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:56.779 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:56.826 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:56.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:27:56.827 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:56.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:27:56.829 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:56.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:27:56.830 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:56.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:27:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.939+0000 7fcc67f55640 1 Processor -- start 2024-03-06T07:27:56.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.940+0000 7fcc67f55640 1 -- start start 2024-03-06T07:27:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.941+0000 7fcc67f55640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc600d4c40 0x7fcc600d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.941+0000 7fcc67f55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcc600d2a30 0x7fcc600b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.941+0000 7fcc67f55640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc600d0100 con 0x7fcc6015e100 2024-03-06T07:27:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.941+0000 7fcc67f55640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc600cf860 con 0x7fcc600d4c40 2024-03-06T07:27:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.941+0000 7fcc67f55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc600cfb40 con 0x7fcc600d2a30 2024-03-06T07:27:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.941+0000 7fcc65cf1640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcc6015e100 0x7fcc600d2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44472/0 (socket says 172.21.15.27:44472) 2024-03-06T07:27:56.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.941+0000 7fcc65cf1640 1 -- 172.21.15.27:0/1462074657 learned_addr learned my addr 172.21.15.27:0/1462074657 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:56.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.941+0000 7fcc654f0640 1 --2- 172.21.15.27:0/1462074657 >> v2:172.21.15.27:3300/0 conn(0x7fcc600d4c40 0x7fcc600d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.941+0000 7fcc664f2640 1 --2- 172.21.15.27:0/1462074657 >> v2:172.21.15.69:3300/0 conn(0x7fcc600d2a30 0x7fcc600b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.942+0000 7fcc64cef640 1 -- 172.21.15.27:0/1462074657 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165171143 0 0) 0x7fcc600d0100 con 0x7fcc6015e100 2024-03-06T07:27:56.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.942+0000 7fcc64cef640 1 -- 172.21.15.27:0/1462074657 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4c003520 con 0x7fcc6015e100 2024-03-06T07:27:56.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.942+0000 7fcc654f0640 1 -- 172.21.15.27:0/1462074657 >> v2:172.21.15.69:3300/0 conn(0x7fcc600d2a30 msgr2=0x7fcc600b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.942+0000 7fcc654f0640 1 --2- 172.21.15.27:0/1462074657 >> v2:172.21.15.69:3300/0 conn(0x7fcc600d2a30 0x7fcc600b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.942+0000 7fcc654f0640 1 -- 172.21.15.27:0/1462074657 >> v1:172.21.15.27:6789/0 conn(0x7fcc6015e100 legacy=0x7fcc600d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.942+0000 7fcc654f0640 1 -- 172.21.15.27:0/1462074657 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc600cfe20 con 0x7fcc600d4c40 2024-03-06T07:27:56.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.942+0000 7fcc654f0640 1 --2- 172.21.15.27:0/1462074657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc600d4c40 0x7fcc600d1cd0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fcc600b3370 tx=0x7fcc5002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f25f6b8ce649db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.943+0000 7fcc64cef640 1 -- 172.21.15.27:0/1462074657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc500315a0 con 0x7fcc600d4c40 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.943+0000 7fcc64cef640 1 -- 172.21.15.27:0/1462074657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50037070 con 0x7fcc600d4c40 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.943+0000 7fcc64cef640 1 -- 172.21.15.27:0/1462074657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50031960 con 0x7fcc600d4c40 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.944+0000 7fcc67f55640 1 -- 172.21.15.27:0/1462074657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc600d4c40 msgr2=0x7fcc600d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.944+0000 7fcc67f55640 1 --2- 172.21.15.27:0/1462074657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc600d4c40 0x7fcc600d1cd0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fcc600b3370 tx=0x7fcc5002b840 comp rx=0 tx=0).stop 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.944+0000 7fcc67f55640 1 -- 172.21.15.27:0/1462074657 shutdown_connections 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.944+0000 7fcc67f55640 1 --2- 172.21.15.27:0/1462074657 >> v2:172.21.15.69:3300/0 conn(0x7fcc600d2a30 0x7fcc600b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.944+0000 7fcc67f55640 1 --2- 172.21.15.27:0/1462074657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc600d4c40 0x7fcc600d1cd0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.944+0000 7fcc67f55640 1 -- 172.21.15.27:0/1462074657 >> 172.21.15.27:0/1462074657 conn(0x7fcc600d3ef0 msgr2=0x7fcc600d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.944+0000 7fcc67f55640 1 -- 172.21.15.27:0/1462074657 shutdown_connections 2024-03-06T07:27:56.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.944+0000 7fcc67f55640 1 -- 172.21.15.27:0/1462074657 wait complete. 2024-03-06T07:27:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.945+0000 7fcc67f55640 1 Processor -- start 2024-03-06T07:27:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.945+0000 7fcc67f55640 1 -- start start 2024-03-06T07:27:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc67f55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc600d2a30 0x7fcc601533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc67f55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc600d4c40 0x7fcc601f4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc67f55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc6015e100 0x7fcc601f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc67f55640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6015e4e0 con 0x7fcc6015e100 2024-03-06T07:27:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc67f55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600cf510 con 0x7fcc600d4c40 2024-03-06T07:27:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc67f55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6015e660 con 0x7fcc600d2a30 2024-03-06T07:27:56.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc664f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc6015e100 0x7fcc601f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc654f0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc600d4c40 0x7fcc601f4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc65cf1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc600d2a30 0x7fcc601533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc654f0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc600d4c40 0x7fcc601f4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:36700/0 (socket says 172.21.15.27:36700) 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.946+0000 7fcc654f0640 1 -- 172.21.15.27:0/1497232438 learned_addr learned my addr 172.21.15.27:0/1497232438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc65cf1640 1 -- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc600d4c40 msgr2=0x7fcc601f4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc65cf1640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc600d4c40 0x7fcc601f4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc65cf1640 1 -- 172.21.15.27:0/1497232438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc6015e100 msgr2=0x7fcc601f50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc65cf1640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc6015e100 0x7fcc601f50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc65cf1640 1 -- 172.21.15.27:0/1497232438 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc601f3e50 con 0x7fcc600d2a30 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc654f0640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc600d4c40 0x7fcc601f4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc664f2640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc6015e100 0x7fcc601f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc65cf1640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc600d2a30 0x7fcc601533f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00c3e0 tx=0x7fcc5c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc567fc640 1 -- 172.21.15.27:0/1497232438 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c0074e0 con 0x7fcc600d2a30 2024-03-06T07:27:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc567fc640 1 -- 172.21.15.27:0/1497232438 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c0047e0 con 0x7fcc600d2a30 2024-03-06T07:27:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc567fc640 1 -- 172.21.15.27:0/1497232438 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c004bd0 con 0x7fcc600d2a30 2024-03-06T07:27:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc60153930 con 0x7fcc600d2a30 2024-03-06T07:27:56.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.947+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc60155050 con 0x7fcc600d2a30 2024-03-06T07:27:56.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.949+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc60154e40 con 0x7fcc600d2a30 2024-03-06T07:27:56.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.949+0000 7fcc567fc640 1 -- 172.21.15.27:0/1497232438 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc5c011070 con 0x7fcc600d2a30 2024-03-06T07:27:56.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.950+0000 7fcc567fc640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc4406b500 0x7fcc4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.950+0000 7fcc567fc640 1 -- 172.21.15.27:0/1497232438 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc5c089b00 con 0x7fcc600d2a30 2024-03-06T07:27:56.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.950+0000 7fcc654f0640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc4406b500 0x7fcc4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.950+0000 7fcc654f0640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc4406b500 0x7fcc4406d9c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcc60156c20 tx=0x7fcc5003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.955+0000 7fcc567fc640 1 -- 172.21.15.27:0/1497232438 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc60153930 con 0x7fcc600d2a30 2024-03-06T07:27:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.981+0000 7f67e8222640 1 Processor -- start 2024-03-06T07:27:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.981+0000 7f79bf165640 1 Processor -- start 2024-03-06T07:27:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.981+0000 7f6e7638c640 1 Processor -- start 2024-03-06T07:27:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.981+0000 7f79bf165640 1 -- start start 2024-03-06T07:27:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.981+0000 7f6e7638c640 1 -- start start 2024-03-06T07:27:56.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.981+0000 7f67e8222640 1 -- start start 2024-03-06T07:27:56.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f6e7638c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6e7015e100 0x7f6e700d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e8222640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67e015ad70 0x7f67e00b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e8222640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f67e015f2d0 0x7f67e00cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f6e7638c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 0x7f6e700d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f6e7638c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e700d0100 con 0x7f6e7015e4e0 2024-03-06T07:27:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f6e7638c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e700cf860 con 0x7f6e7015e100 2024-03-06T07:27:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f6e7638c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700cfb40 con 0x7f6e700d4c40 2024-03-06T07:27:56.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e8222640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e015b150 con 0x7f67e00b4120 2024-03-06T07:27:56.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e8222640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e00cd5b0 con 0x7f67e015f2d0 2024-03-06T07:27:56.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e8222640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e00b3080 con 0x7f67e015ad70 2024-03-06T07:27:56.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f79bf165640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79b8140710 0x7f79b8070b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f79bf165640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b8140af0 con 0x7f79b80b3870 2024-03-06T07:27:56.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f79bf165640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b80b3ee0 con 0x7f79b8071090 2024-03-06T07:27:56.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f79bf165640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b80714f0 con 0x7f79b8140710 2024-03-06T07:27:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e5fbe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67e015ad70 0x7f67e00b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e57bd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f67e015f2d0 0x7f67e00cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e57bd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f67e015f2d0 0x7f67e00cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36202/0 (socket says 172.21.15.27:36202) 2024-03-06T07:27:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e5fbe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67e015ad70 0x7f67e00b2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:36710/0 (socket says 172.21.15.27:36710) 2024-03-06T07:27:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f67e57bd640 1 -- 172.21.15.27:0/3870373268 learned_addr learned my addr 172.21.15.27:0/3870373268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f6e6ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6e7015e100 0x7f6e700d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f6e6f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 0x7f6e700d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f6e6ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6e7015e100 0x7f6e700d1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:36708/0 (socket says 172.21.15.27:36708) 2024-03-06T07:27:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f6e6ffff640 1 -- 172.21.15.27:0/2743323576 learned_addr learned my addr 172.21.15.27:0/2743323576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.982+0000 7f79bcf01640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79b8140710 0x7f79b8070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f79bcf01640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79b8140710 0x7f79b8070b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36218/0 (socket says 172.21.15.27:36218) 2024-03-06T07:27:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f79bd702640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f79b80b3870 0x7f79b806cc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44474/0 (socket says 172.21.15.27:44474) 2024-03-06T07:27:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f6e6ffff640 1 -- 172.21.15.27:0/2743323576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 msgr2=0x7f6e700d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f6e6ffff640 1 --2- 172.21.15.27:0/2743323576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 0x7f6e700d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f67e5fbe640 1 -- 172.21.15.27:0/3870373268 >> v2:172.21.15.27:3300/0 conn(0x7f67e015f2d0 msgr2=0x7f67e00cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f67e5fbe640 1 --2- 172.21.15.27:0/3870373268 >> v2:172.21.15.27:3300/0 conn(0x7f67e015f2d0 0x7f67e00cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f67e5fbe640 1 -- 172.21.15.27:0/3870373268 >> v1:172.21.15.27:6789/0 conn(0x7f67e00b4120 legacy=0x7f67e00b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f67e5fbe640 1 -- 172.21.15.27:0/3870373268 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e00b3360 con 0x7f67e015ad70 2024-03-06T07:27:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f79bcf01640 1 -- 172.21.15.27:0/2671405614 learned_addr learned my addr 172.21.15.27:0/2671405614 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f67e57bd640 1 --2- 172.21.15.27:0/3870373268 >> v2:172.21.15.27:3300/0 conn(0x7f67e015f2d0 0x7f67e00cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f6e6ffff640 1 -- 172.21.15.27:0/2743323576 >> v1:172.21.15.69:6789/0 conn(0x7f6e7015e4e0 legacy=0x7f6e700d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f6e6ffff640 1 -- 172.21.15.27:0/2743323576 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e700cfe20 con 0x7f6e7015e100 2024-03-06T07:27:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f79bcf01640 1 -- 172.21.15.27:0/2671405614 >> v1:172.21.15.69:6789/0 conn(0x7f79b8071090 legacy=0x7f79b80b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f79b77fe640 1 -- 172.21.15.27:0/2671405614 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092811552 0 0) 0x7f79b8140af0 con 0x7f79b80b3870 2024-03-06T07:27:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f79bcf01640 1 -- 172.21.15.27:0/2671405614 >> v1:172.21.15.27:6789/0 conn(0x7f79b80b3870 legacy=0x7f79b806cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.983+0000 7f79bcf01640 1 -- 172.21.15.27:0/2671405614 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b80b41c0 con 0x7f79b8140710 2024-03-06T07:27:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.984+0000 7f67e5fbe640 1 --2- 172.21.15.27:0/3870373268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015ad70 0x7f67e00b2aa0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0097a0 tx=0x7f67dc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c2975b2bb37b1d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.984+0000 7f6e6ffff640 1 --2- 172.21.15.27:0/2743323576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e7015e100 0x7f6e700d1cd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009870 tx=0x7f6e6002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12fcf264ed295da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.984+0000 7f67e4fbc640 1 -- 172.21.15.27:0/3870373268 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc035020 con 0x7f67e015ad70 2024-03-06T07:27:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.984+0000 7f67e4fbc640 1 -- 172.21.15.27:0/3870373268 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc031830 con 0x7f67e015ad70 2024-03-06T07:27:56.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.984+0000 7f67e4fbc640 1 -- 172.21.15.27:0/3870373268 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc031c00 con 0x7f67e015ad70 2024-03-06T07:27:56.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.984+0000 7f6e6effd640 1 -- 172.21.15.27:0/2743323576 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60034070 con 0x7f6e7015e100 2024-03-06T07:27:56.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.984+0000 7f6e6effd640 1 -- 172.21.15.27:0/2743323576 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60030840 con 0x7f6e7015e100 2024-03-06T07:27:56.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.984+0000 7f6e6effd640 1 -- 172.21.15.27:0/2743323576 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60030c10 con 0x7f6e7015e100 2024-03-06T07:27:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f79bcf01640 1 --2- 172.21.15.27:0/2671405614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b8140710 0x7f79b8070b50 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0097f0 tx=0x7f79ac02b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56898ce966df975c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f67e8222640 1 -- 172.21.15.27:0/3870373268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015ad70 msgr2=0x7f67e00b2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f67e8222640 1 --2- 172.21.15.27:0/3870373268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015ad70 0x7f67e00b2aa0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0097a0 tx=0x7f67dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f6e7638c640 1 -- 172.21.15.27:0/2743323576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e7015e100 msgr2=0x7f6e700d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f6e7638c640 1 --2- 172.21.15.27:0/2743323576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e7015e100 0x7f6e700d1cd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009870 tx=0x7f6e6002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f6e7638c640 1 -- 172.21.15.27:0/2743323576 shutdown_connections 2024-03-06T07:27:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f6e7638c640 1 --2- 172.21.15.27:0/2743323576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 0x7f6e700d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f6e7638c640 1 --2- 172.21.15.27:0/2743323576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e7015e100 0x7f6e700d1cd0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f6e7638c640 1 -- 172.21.15.27:0/2743323576 >> 172.21.15.27:0/2743323576 conn(0x7f6e700d3ef0 msgr2=0x7f6e700d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f6e7638c640 1 -- 172.21.15.27:0/2743323576 shutdown_connections 2024-03-06T07:27:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f67e8222640 1 -- 172.21.15.27:0/3870373268 shutdown_connections 2024-03-06T07:27:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f79b77fe640 1 -- 172.21.15.27:0/2671405614 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac031760 con 0x7f79b8140710 2024-03-06T07:27:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f79b77fe640 1 -- 172.21.15.27:0/2671405614 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ac037070 con 0x7f79b8140710 2024-03-06T07:27:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f79b77fe640 1 -- 172.21.15.27:0/2671405614 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac031b20 con 0x7f79b8140710 2024-03-06T07:27:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f6e7638c640 1 -- 172.21.15.27:0/2743323576 wait complete. 2024-03-06T07:27:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f67e8222640 1 --2- 172.21.15.27:0/3870373268 >> v2:172.21.15.27:3300/0 conn(0x7f67e015f2d0 0x7f67e00cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f67e8222640 1 --2- 172.21.15.27:0/3870373268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015ad70 0x7f67e00b2aa0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.985+0000 7f67e8222640 1 -- 172.21.15.27:0/3870373268 >> 172.21.15.27:0/3870373268 conn(0x7f67e00b17a0 msgr2=0x7f67e00b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f67e8222640 1 -- 172.21.15.27:0/3870373268 shutdown_connections 2024-03-06T07:27:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f67e8222640 1 -- 172.21.15.27:0/3870373268 wait complete. 2024-03-06T07:27:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f79bf165640 1 -- 172.21.15.27:0/2671405614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b8140710 msgr2=0x7f79b8070b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f79bf165640 1 --2- 172.21.15.27:0/2671405614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b8140710 0x7f79b8070b50 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0097f0 tx=0x7f79ac02b800 comp rx=0 tx=0).stop 2024-03-06T07:27:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f79bf165640 1 -- 172.21.15.27:0/2671405614 shutdown_connections 2024-03-06T07:27:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f79bf165640 1 --2- 172.21.15.27:0/2671405614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b8140710 0x7f79b8070b50 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f79bf165640 1 -- 172.21.15.27:0/2671405614 >> 172.21.15.27:0/2671405614 conn(0x7f79b80b17a0 msgr2=0x7f79b80b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f79bf165640 1 -- 172.21.15.27:0/2671405614 shutdown_connections 2024-03-06T07:27:56.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f79bf165640 1 -- 172.21.15.27:0/2671405614 wait complete. 2024-03-06T07:27:56.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.986+0000 7f6e7638c640 1 Processor -- start 2024-03-06T07:27:56.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.987+0000 7f6e7638c640 1 -- start start 2024-03-06T07:27:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.987+0000 7f67e8222640 1 Processor -- start 2024-03-06T07:27:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.987+0000 7f67e8222640 1 -- start start 2024-03-06T07:27:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e7638c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 0x7f6e7009dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e7638c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e7015e100 0x7f6e7015c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e7638c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e7015e4e0 0x7f6e7015c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e8222640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67e00b4120 0x7f67e01e5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e7638c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700b2080 con 0x7f6e7015e100 2024-03-06T07:27:57.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e7638c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700cf510 con 0x7f6e7015e4e0 2024-03-06T07:27:57.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e6f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e7015e100 0x7f6e7015c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e8222640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67e015ad70 0x7f67e01e6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e8222640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015f2d0 0x7f67e0146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e8222640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e00b3c00 con 0x7f67e015ad70 2024-03-06T07:27:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e8222640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e00b3920 con 0x7f67e015f2d0 2024-03-06T07:27:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e8222640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e015f730 con 0x7f67e00b4120 2024-03-06T07:27:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e6f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e7015e100 0x7f6e7015c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36232/0 (socket says 172.21.15.27:36232) 2024-03-06T07:27:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e5fbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67e00b4120 0x7f67e01e5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e5fbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67e00b4120 0x7f67e01e5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:51666/0 (socket says 172.21.15.27:51666) 2024-03-06T07:27:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e5fbe640 1 -- 172.21.15.27:0/3187147812 learned_addr learned my addr 172.21.15.27:0/3187147812 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e67bf640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015f2d0 0x7f67e0146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e6f7fe640 1 -- 172.21.15.27:0/2185064785 learned_addr learned my addr 172.21.15.27:0/2185064785 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e6ffff640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 0x7f6e7009dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e74929640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e7015e4e0 0x7f6e7015c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f6e7638c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700b2200 con 0x7f6e700d4c40 2024-03-06T07:27:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e57bd640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67e015ad70 0x7f67e01e6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e5fbe640 1 -- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015f2d0 msgr2=0x7f67e0146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e5fbe640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015f2d0 0x7f67e0146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e5fbe640 1 -- 172.21.15.27:0/3187147812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67e015ad70 msgr2=0x7f67e01e6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e5fbe640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67e015ad70 0x7f67e01e6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e5fbe640 1 -- 172.21.15.27:0/3187147812 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e01e5cd0 con 0x7f67e00b4120 2024-03-06T07:27:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.988+0000 7f67e67bf640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015f2d0 0x7f67e0146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6ffff640 1 -- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e7015e4e0 msgr2=0x7f6e7015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f79bf165640 1 Processor -- start 2024-03-06T07:27:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6ffff640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e7015e4e0 0x7f6e7015c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6ffff640 1 -- 172.21.15.27:0/2185064785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e7015e100 msgr2=0x7f6e7015c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6ffff640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e7015e100 0x7f6e7015c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6ffff640 1 -- 172.21.15.27:0/2185064785 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e700a15d0 con 0x7f6e700d4c40 2024-03-06T07:27:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6f7fe640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e7015e100 0x7f6e7015c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f67e5fbe640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67e00b4120 0x7f67e01e5790 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0098c0 tx=0x7f67dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6ffff640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 0x7f6e7009dea0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e60033000 tx=0x7f6e60004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f67d67fc640 1 -- 172.21.15.27:0/3187147812 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc040070 con 0x7f67e00b4120 2024-03-06T07:27:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f67d67fc640 1 -- 172.21.15.27:0/3187147812 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc03e070 con 0x7f67e00b4120 2024-03-06T07:27:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e01e46f0 con 0x7f67e00b4120 2024-03-06T07:27:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f67d67fc640 1 -- 172.21.15.27:0/3187147812 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc035040 con 0x7f67e00b4120 2024-03-06T07:27:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e01e4f90 con 0x7f67e00b4120 2024-03-06T07:27:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6cff9640 1 -- 172.21.15.27:0/2185064785 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60048070 con 0x7f6e700d4c40 2024-03-06T07:27:57.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6cff9640 1 -- 172.21.15.27:0/2185064785 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60030640 con 0x7f6e700d4c40 2024-03-06T07:27:57.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e6cff9640 1 -- 172.21.15.27:0/2185064785 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60034040 con 0x7f6e700d4c40 2024-03-06T07:27:57.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e700a18b0 con 0x7f6e700d4c40 2024-03-06T07:27:57.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.990+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e701e69a0 con 0x7f6e700d4c40 2024-03-06T07:27:57.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.989+0000 7f79bf165640 1 -- start start 2024-03-06T07:27:57.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.990+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e00752d0 con 0x7f67e00b4120 2024-03-06T07:27:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.990+0000 7f79bf165640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79b8071090 0x7f79b81f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bf165640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b80b3870 0x7f79b8143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bf165640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79b8140710 0x7f79b8143890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bf165640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b806eb90 con 0x7f79b80b3870 2024-03-06T07:27:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bf165640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b80b4780 con 0x7f79b8071090 2024-03-06T07:27:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bf165640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b806ed10 con 0x7f79b8140710 2024-03-06T07:27:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bcf01640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79b8071090 0x7f79b81f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bcf01640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79b8071090 0x7f79b81f7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:36728/0 (socket says 172.21.15.27:36728) 2024-03-06T07:27:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b80b3870 0x7f79b8143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bcf01640 1 -- 172.21.15.27:0/252758998 learned_addr learned my addr 172.21.15.27:0/252758998 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f67d67fc640 1 -- 172.21.15.27:0/3187147812 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67dc038020 con 0x7f67e00b4120 2024-03-06T07:27:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bd702640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79b8140710 0x7f79b8143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e700752d0 con 0x7f6e700d4c40 2024-03-06T07:27:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bd702640 1 -- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79b8071090 msgr2=0x7f79b81f7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bd702640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79b8071090 0x7f79b81f7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bd702640 1 -- 172.21.15.27:0/252758998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b80b3870 msgr2=0x7f79b8143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bd702640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b80b3870 0x7f79b8143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bd702640 1 -- 172.21.15.27:0/252758998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b813c2b0 con 0x7f79b8140710 2024-03-06T07:27:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bcf01640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79b8071090 0x7f79b81f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f6e6cff9640 1 -- 172.21.15.27:0/2185064785 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e6004b070 con 0x7f6e700d4c40 2024-03-06T07:27:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f67d67fc640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f67c406b4d0 0x7f67c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f79bd702640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79b8140710 0x7f79b8143890 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f79b000ca40 tx=0x7f79b000cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.991+0000 7f67d67fc640 1 -- 172.21.15.27:0/3187147812 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67dc0ab4a0 con 0x7f67e00b4120 2024-03-06T07:27:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.992+0000 7f67e57bd640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f67c406b4d0 0x7f67c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.992+0000 7f6e6cff9640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e5c06b520 0x7f6e5c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.992+0000 7f6e6cff9640 1 -- 172.21.15.27:0/2185064785 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e600ac0b0 con 0x7f6e700d4c40 2024-03-06T07:27:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.992+0000 7f6e6f7fe640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e5c06b520 0x7f6e5c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.992+0000 7f67e57bd640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f67c406b4d0 0x7f67c406d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f67e015a0e0 tx=0x7f67d0009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.992+0000 7f6e6f7fe640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e5c06b520 0x7f6e5c06d9e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6e700d19d0 tx=0x7f6e64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:56.996+0000 7f67d67fc640 1 -- 172.21.15.27:0/3187147812 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67dc031800 con 0x7f67e00b4120 2024-03-06T07:27:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.000+0000 7f79b57fa640 1 -- 172.21.15.27:0/252758998 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b000abd0 con 0x7f79b8140710 2024-03-06T07:27:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.000+0000 7f79b57fa640 1 -- 172.21.15.27:0/252758998 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b000adb0 con 0x7f79b8140710 2024-03-06T07:27:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.000+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b813c590 con 0x7f79b8140710 2024-03-06T07:27:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.000+0000 7f79b57fa640 1 -- 172.21.15.27:0/252758998 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b0012640 con 0x7f79b8140710 2024-03-06T07:27:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.000+0000 7f6e6cff9640 1 -- 172.21.15.27:0/2185064785 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e6007ab70 con 0x7f6e700d4c40 2024-03-06T07:27:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.000+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b8152ad0 con 0x7f79b8140710 2024-03-06T07:27:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.002+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b8152cd0 con 0x7f79b8140710 2024-03-06T07:27:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.002+0000 7f79b57fa640 1 -- 172.21.15.27:0/252758998 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79b00127e0 con 0x7f79b8140710 2024-03-06T07:27:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.003+0000 7f79b57fa640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f799c06b4d0 0x7f799c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.003+0000 7f79b57fa640 1 -- 172.21.15.27:0/252758998 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79b008a730 con 0x7f79b8140710 2024-03-06T07:27:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.003+0000 7f79bcf01640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f799c06b4d0 0x7f799c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.003+0000 7f79bcf01640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f799c06b4d0 0x7f799c06d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0090b0 tx=0x7f79ac03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.007+0000 7f79b57fa640 1 -- 172.21.15.27:0/252758998 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79b8152cd0 con 0x7f79b8140710 2024-03-06T07:27:57.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.222+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fcc600752d0 con 0x7fcc600d2a30 2024-03-06T07:27:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.254+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f6e700cefe0 con 0x7f6e700d4c40 2024-03-06T07:27:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.272+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f67e006c9d0 con 0x7f67e00b4120 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.279+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f79b80752d0 con 0x7f79b8140710 2024-03-06T07:27:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.831+0000 7fcc567fc640 1 -- 172.21.15.27:0/1497232438 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6080 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7fcc601f3e50 con 0x7fcc600d2a30 2024-03-06T07:27:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 6080 2024-03-06T07:27:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc4406b500 msgr2=0x7fcc4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc4406b500 0x7fcc4406d9c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcc60156c20 tx=0x7fcc5003d040 comp rx=0 tx=0).stop 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc600d2a30 msgr2=0x7fcc601533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc600d2a30 0x7fcc601533f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00c3e0 tx=0x7fcc5c00c8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 shutdown_connections 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc4406b500 0x7fcc4406d9c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc6015e100 0x7fcc601f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc600d4c40 0x7fcc601f4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 --2- 172.21.15.27:0/1497232438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc600d2a30 0x7fcc601533f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 >> 172.21.15.27:0/1497232438 conn(0x7fcc600d3ef0 msgr2=0x7fcc600d1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 shutdown_connections 2024-03-06T07:27:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.835+0000 7fcc67f55640 1 -- 172.21.15.27:0/1497232438 wait complete. 2024-03-06T07:27:57.841 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:27:57.839+0000 7fa8a7301640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.845+0000 7f6e6cff9640 1 -- 172.21.15.27:0/2185064785 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6715 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f6e6007ec10 con 0x7f6e700d4c40 2024-03-06T07:27:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.845+0000 7f67d67fc640 1 -- 172.21.15.27:0/3187147812 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7425 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f67dc079d40 con 0x7f67e00b4120 2024-03-06T07:27:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.845+0000 7f79b57fa640 1 -- 172.21.15.27:0/252758998 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7428 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f79b0058ec0 con 0x7f79b8140710 2024-03-06T07:27:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 6715 2024-03-06T07:27:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 7425 2024-03-06T07:27:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 7428 2024-03-06T07:27:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f799c06b4d0 msgr2=0x7f799c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f799c06b4d0 0x7f799c06d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0090b0 tx=0x7f79ac03d040 comp rx=0 tx=0).stop 2024-03-06T07:27:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79b8140710 msgr2=0x7f79b8143890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79b8140710 0x7f79b8143890 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f79b000ca40 tx=0x7f79b000cf00 comp rx=0 tx=0).stop 2024-03-06T07:27:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 shutdown_connections 2024-03-06T07:27:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f799c06b4d0 0x7f799c06d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79b8140710 0x7f79b8143890 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b80b3870 0x7f79b8143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 --2- 172.21.15.27:0/252758998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79b8071090 0x7f79b81f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 >> 172.21.15.27:0/252758998 conn(0x7f79b80b17a0 msgr2=0x7f79b806db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 shutdown_connections 2024-03-06T07:27:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f79bf165640 1 -- 172.21.15.27:0/252758998 wait complete. 2024-03-06T07:27:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.848+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f67c406b4d0 msgr2=0x7f67c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f67c406b4d0 0x7f67c406d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f67e015a0e0 tx=0x7f67d0009240 comp rx=0 tx=0).stop 2024-03-06T07:27:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67e00b4120 msgr2=0x7f67e01e5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67e00b4120 0x7f67e01e5790 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0098c0 tx=0x7f67dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 shutdown_connections 2024-03-06T07:27:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f67c406b4d0 0x7f67c406d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67e015f2d0 0x7f67e0146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67e015ad70 0x7f67e01e6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 --2- 172.21.15.27:0/3187147812 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67e00b4120 0x7f67e01e5790 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0098c0 tx=0x7f67dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 >> 172.21.15.27:0/3187147812 conn(0x7f67e00b17a0 msgr2=0x7f67e015b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e5c06b520 msgr2=0x7f6e5c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e5c06b520 0x7f6e5c06d9e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6e700d19d0 tx=0x7f6e64008040 comp rx=0 tx=0).stop 2024-03-06T07:27:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 msgr2=0x7f6e7009dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 0x7f6e7009dea0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e60033000 tx=0x7f6e60004730 comp rx=0 tx=0).stop 2024-03-06T07:27:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 shutdown_connections 2024-03-06T07:27:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6e5c06b520 0x7f6e5c06d9e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e7015e4e0 0x7f6e7015c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e7015e100 0x7f6e7015c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 --2- 172.21.15.27:0/2185064785 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6e700d4c40 0x7f6e7009dea0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 >> 172.21.15.27:0/2185064785 conn(0x7f6e700d3ef0 msgr2=0x7f6e700d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.849+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 shutdown_connections 2024-03-06T07:27:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.850+0000 7f67e8222640 1 -- 172.21.15.27:0/3187147812 wait complete. 2024-03-06T07:27:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.850+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 shutdown_connections 2024-03-06T07:27:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:57.850+0000 7f6e7638c640 1 -- 172.21.15.27:0/2185064785 wait complete. 2024-03-06T07:27:57.855 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:57.855 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:57.857 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:57.857 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:57.865 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:57.865 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:57.867 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:57.867 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:57.869 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:57.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:57.870 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:57.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:57.912 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:57.913 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:57.915 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:57.915 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:57.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:57.938 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:27:57.946 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:27:57.999 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:27:57.999 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:27:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.029+0000 7fdb4284e640 1 Processor -- start 2024-03-06T07:27:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.029+0000 7fdb4284e640 1 -- start start 2024-03-06T07:27:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.030+0000 7fdb4284e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb3c15e100 0x7fdb3c0d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.030+0000 7fdb4284e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c0d4c40 0x7fdb3c0d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.030+0000 7fdb4284e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0d0100 con 0x7fdb3c15e4e0 2024-03-06T07:27:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.030+0000 7fdb4284e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c0cf860 con 0x7fdb3c15e100 2024-03-06T07:27:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.030+0000 7fdb4284e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0cfb40 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.030+0000 7fdb3bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb3c15e100 0x7fdb3c0d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.030+0000 7fdb40deb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb3c15e4e0 0x7fdb3c0d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44494/0 (socket says 172.21.15.27:44494) 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.030+0000 7fdb40deb640 1 -- 172.21.15.27:0/654620031 learned_addr learned my addr 172.21.15.27:0/654620031 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.031+0000 7fdb3b7fe640 1 --2- 172.21.15.27:0/654620031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c0d4c40 0x7fdb3c0d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.031+0000 7fdb3bfff640 1 -- 172.21.15.27:0/654620031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c0d4c40 msgr2=0x7fdb3c0d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.031+0000 7fdb3bfff640 1 --2- 172.21.15.27:0/654620031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c0d4c40 0x7fdb3c0d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.031+0000 7fdb3bfff640 1 -- 172.21.15.27:0/654620031 >> v1:172.21.15.27:6789/0 conn(0x7fdb3c15e4e0 legacy=0x7fdb3c0d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.031+0000 7fdb3affd640 1 -- 172.21.15.27:0/654620031 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243299252 0 0) 0x7fdb3c0d0100 con 0x7fdb3c15e4e0 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.031+0000 7fdb3bfff640 1 -- 172.21.15.27:0/654620031 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0cfe20 con 0x7fdb3c15e100 2024-03-06T07:27:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.031+0000 7fdb3b7fe640 1 --2- 172.21.15.27:0/654620031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c0d4c40 0x7fdb3c0d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.031+0000 7fdb3bfff640 1 --2- 172.21.15.27:0/654620031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c15e100 0x7fdb3c0d1cd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c0097a0 tx=0x7fdb2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb9614be060620f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb3affd640 1 -- 172.21.15.27:0/654620031 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c035020 con 0x7fdb3c15e100 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb3affd640 1 -- 172.21.15.27:0/654620031 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2c031830 con 0x7fdb3c15e100 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb3affd640 1 -- 172.21.15.27:0/654620031 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c031c00 con 0x7fdb3c15e100 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb4284e640 1 -- 172.21.15.27:0/654620031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c15e100 msgr2=0x7fdb3c0d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb4284e640 1 --2- 172.21.15.27:0/654620031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c15e100 0x7fdb3c0d1cd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c0097a0 tx=0x7fdb2c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb4284e640 1 -- 172.21.15.27:0/654620031 shutdown_connections 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb4284e640 1 --2- 172.21.15.27:0/654620031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c0d4c40 0x7fdb3c0d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb4284e640 1 --2- 172.21.15.27:0/654620031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c15e100 0x7fdb3c0d1cd0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb4284e640 1 -- 172.21.15.27:0/654620031 >> 172.21.15.27:0/654620031 conn(0x7fdb3c0d3ef0 msgr2=0x7fdb3c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb4284e640 1 -- 172.21.15.27:0/654620031 shutdown_connections 2024-03-06T07:27:58.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.032+0000 7fdb4284e640 1 -- 172.21.15.27:0/654620031 wait complete. 2024-03-06T07:27:58.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.033+0000 7fdb4284e640 1 Processor -- start 2024-03-06T07:27:58.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.033+0000 7fdb4284e640 1 -- start start 2024-03-06T07:27:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb4284e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c0d4c40 0x7fdb3c1f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb4284e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb3c15e100 0x7fdb3c14c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb4284e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c15e4e0 0x7fdb3c0a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb4284e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0b2080 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb4284e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0cf510 con 0x7fdb3c15e100 2024-03-06T07:27:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb4284e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0b2200 con 0x7fdb3c15e4e0 2024-03-06T07:27:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c0d4c40 0x7fdb3c1f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c0d4c40 0x7fdb3c1f79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36310/0 (socket says 172.21.15.27:36310) 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb3bfff640 1 -- 172.21.15.27:0/3598395423 learned_addr learned my addr 172.21.15.27:0/3598395423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb3b7fe640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb3c15e100 0x7fdb3c14c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.034+0000 7fdb40deb640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c15e4e0 0x7fdb3c0a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb3bfff640 1 -- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c15e4e0 msgr2=0x7fdb3c0a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb3bfff640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c15e4e0 0x7fdb3c0a13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb3bfff640 1 -- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb3c15e100 msgr2=0x7fdb3c14c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb3bfff640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb3c15e100 0x7fdb3c14c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb3bfff640 1 -- 172.21.15.27:0/3598395423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0b36e0 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb3bfff640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c0d4c40 0x7fdb3c1f79e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c000c00 tx=0x7fdb2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb38ff9640 1 -- 172.21.15.27:0/3598395423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c035050 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb38ff9640 1 -- 172.21.15.27:0/3598395423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2c004230 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c14e780 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb38ff9640 1 -- 172.21.15.27:0/3598395423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c03b800 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.035+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c14ed40 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.036+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c0752d0 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.037+0000 7fdb38ff9640 1 -- 172.21.15.27:0/3598395423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb2c038020 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.037+0000 7fdb38ff9640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb2806b4d0 0x7fdb2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.037+0000 7fdb38ff9640 1 -- 172.21.15.27:0/3598395423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdb2c0abf30 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.037+0000 7fdb3b7fe640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb2806b4d0 0x7fdb2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.038+0000 7fdb3b7fe640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb2806b4d0 0x7fdb2806d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c06d050 tx=0x7fdb30008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.041+0000 7fdb38ff9640 1 -- 172.21.15.27:0/3598395423 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb2c040070 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.258+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb3c14c700 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.259+0000 7fdb38ff9640 1 -- 172.21.15.27:0/3598395423 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+1802 (secure 0 0 0) 0x7fdb2c07a700 con 0x7fdb3c0d4c40 2024-03-06T07:27:58.261 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:27:58.262 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":63,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:27:57.840080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:27:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 63 2024-03-06T07:27:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.260+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb2806b4d0 msgr2=0x7fdb2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.260+0000 7fdb4284e640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb2806b4d0 0x7fdb2806d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c06d050 tx=0x7fdb30008040 comp rx=0 tx=0).stop 2024-03-06T07:27:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.260+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c0d4c40 msgr2=0x7fdb3c1f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.260+0000 7fdb4284e640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c0d4c40 0x7fdb3c1f79e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c000c00 tx=0x7fdb2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.261+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 shutdown_connections 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.261+0000 7fdb4284e640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb2806b4d0 0x7fdb2806d990 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.261+0000 7fdb4284e640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb3c15e4e0 0x7fdb3c0a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.261+0000 7fdb4284e640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb3c15e100 0x7fdb3c14c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.261+0000 7fdb4284e640 1 --2- 172.21.15.27:0/3598395423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb3c0d4c40 0x7fdb3c1f79e0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.261+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 >> 172.21.15.27:0/3598395423 conn(0x7fdb3c0d3ef0 msgr2=0x7fdb3c15e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.261+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 shutdown_connections 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:58.261+0000 7fdb4284e640 1 -- 172.21.15.27:0/3598395423 wait complete. 2024-03-06T07:27:58.676 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:48.137883+0000', 'modified': '2024-03-06T07:27:57.840080+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 102, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:58.676 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:27:59.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:59.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.794+0000 7f5469916640 1 Processor -- start 2024-03-06T07:27:59.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.794+0000 7f5469916640 1 -- start start 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5469916640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f546415c0e0 0x7f54640cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5469916640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f54640cd980 0x7f5464165ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5469916640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54640ccbc0 con 0x7f54640d28d0 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5469916640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54640ccea0 con 0x7f546415c0e0 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5469916640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54640cd180 con 0x7f54640cd980 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5462ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f546415c0e0 0x7f54640cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f54627fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f54640d28d0 0x7f54640d0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44506/0 (socket says 172.21.15.27:44506) 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f54627fc640 1 -- 172.21.15.27:0/191436942 learned_addr learned my addr 172.21.15.27:0/191436942 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f54637fe640 1 --2- 172.21.15.27:0/191436942 >> v2:172.21.15.69:3300/0 conn(0x7f54640cd980 0x7f5464165ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5462ffd640 1 -- 172.21.15.27:0/191436942 >> v2:172.21.15.69:3300/0 conn(0x7f54640cd980 msgr2=0x7f5464165ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5462ffd640 1 --2- 172.21.15.27:0/191436942 >> v2:172.21.15.69:3300/0 conn(0x7f54640cd980 0x7f5464165ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5462ffd640 1 -- 172.21.15.27:0/191436942 >> v1:172.21.15.27:6789/0 conn(0x7f54640d28d0 legacy=0x7f54640d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f5462ffd640 1 -- 172.21.15.27:0/191436942 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54640cd460 con 0x7f546415c0e0 2024-03-06T07:27:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.795+0000 7f54637fe640 1 --2- 172.21.15.27:0/191436942 >> v2:172.21.15.69:3300/0 conn(0x7f54640cd980 0x7f5464165ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:59.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.796+0000 7f5462ffd640 1 --2- 172.21.15.27:0/191436942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546415c0e0 0x7f54640cf8a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5464072880 tx=0x7f545402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1799dfee9012204d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.796+0000 7f5461ffb640 1 -- 172.21.15.27:0/191436942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454035020 con 0x7f546415c0e0 2024-03-06T07:27:59.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.796+0000 7f5461ffb640 1 -- 172.21.15.27:0/191436942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5454031830 con 0x7f546415c0e0 2024-03-06T07:27:59.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.796+0000 7f5461ffb640 1 -- 172.21.15.27:0/191436942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454031c00 con 0x7f546415c0e0 2024-03-06T07:27:59.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.797+0000 7f5469916640 1 -- 172.21.15.27:0/191436942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546415c0e0 msgr2=0x7f54640cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.797+0000 7f5469916640 1 --2- 172.21.15.27:0/191436942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546415c0e0 0x7f54640cf8a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5464072880 tx=0x7f545402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.797+0000 7f5469916640 1 -- 172.21.15.27:0/191436942 shutdown_connections 2024-03-06T07:27:59.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.797+0000 7f5469916640 1 --2- 172.21.15.27:0/191436942 >> v2:172.21.15.69:3300/0 conn(0x7f54640cd980 0x7f5464165ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.797+0000 7f5469916640 1 --2- 172.21.15.27:0/191436942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546415c0e0 0x7f54640cf8a0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.797+0000 7f5469916640 1 -- 172.21.15.27:0/191436942 >> 172.21.15.27:0/191436942 conn(0x7f54640d1cd0 msgr2=0x7f54640d20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.797+0000 7f5469916640 1 -- 172.21.15.27:0/191436942 shutdown_connections 2024-03-06T07:27:59.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.797+0000 7f5469916640 1 -- 172.21.15.27:0/191436942 wait complete. 2024-03-06T07:27:59.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.798+0000 7f5469916640 1 Processor -- start 2024-03-06T07:27:59.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.798+0000 7f5469916640 1 -- start start 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f5469916640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54640cd980 0x7f54640c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f5469916640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640d28d0 0x7f54640c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f5469916640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f546415c0e0 0x7f54640c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f5469916640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54640cff60 con 0x7f54640cd980 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f5469916640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54640cfde0 con 0x7f54640d28d0 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f5469916640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54640d0890 con 0x7f546415c0e0 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f5462ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54640cd980 0x7f54640c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f5462ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54640cd980 0x7f54640c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36334/0 (socket says 172.21.15.27:36334) 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f5462ffd640 1 -- 172.21.15.27:0/1468639730 learned_addr learned my addr 172.21.15.27:0/1468639730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f54627fc640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640d28d0 0x7f54640c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.799+0000 7f54637fe640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f546415c0e0 0x7f54640c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f5462ffd640 1 -- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f546415c0e0 msgr2=0x7f54640c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f5462ffd640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f546415c0e0 0x7f54640c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f5462ffd640 1 -- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640d28d0 msgr2=0x7f54640c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f5462ffd640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640d28d0 0x7f54640c6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f5462ffd640 1 -- 172.21.15.27:0/1468639730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464151c30 con 0x7f54640cd980 2024-03-06T07:27:59.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f5462ffd640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54640cd980 0x7f54640c64f0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5454000c00 tx=0x7f54540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f544bfff640 1 -- 172.21.15.27:0/1468639730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454040070 con 0x7f54640cd980 2024-03-06T07:27:59.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f544bfff640 1 -- 172.21.15.27:0/1468639730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5454039a20 con 0x7f54640cd980 2024-03-06T07:27:59.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546414ee40 con 0x7f54640cd980 2024-03-06T07:27:59.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f544bfff640 1 -- 172.21.15.27:0/1468639730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454035040 con 0x7f54640cd980 2024-03-06T07:27:59.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.800+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546414f400 con 0x7f54640cd980 2024-03-06T07:27:59.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.801+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54640752d0 con 0x7f54640cd980 2024-03-06T07:27:59.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.801+0000 7f544bfff640 1 -- 172.21.15.27:0/1468639730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5454038020 con 0x7f54640cd980 2024-03-06T07:27:59.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.802+0000 7f544bfff640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f545006b4d0 0x7f545006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.802+0000 7f544bfff640 1 -- 172.21.15.27:0/1468639730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f54540abbc0 con 0x7f54640cd980 2024-03-06T07:27:59.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.802+0000 7f54627fc640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f545006b4d0 0x7f545006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.803+0000 7f54627fc640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f545006b4d0 0x7f545006d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f54640c91e0 tx=0x7f5458009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:27:59.806+0000 7f544bfff640 1 -- 172.21.15.27:0/1468639730 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f545407aa70 con 0x7f54640cd980 2024-03-06T07:28:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.025+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54640d15d0 con 0x7f54640cd980 2024-03-06T07:28:00.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.025+0000 7f544bfff640 1 -- 172.21.15.27:0/1468639730 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f545407a370 con 0x7f54640cd980 2024-03-06T07:28:00.028 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:00.028 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":65,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:27:58.540250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:28:00.028 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 65 2024-03-06T07:28:00.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f545006b4d0 msgr2=0x7f545006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f545006b4d0 0x7f545006d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f54640c91e0 tx=0x7f5458009040 comp rx=0 tx=0).stop 2024-03-06T07:28:00.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54640cd980 msgr2=0x7f54640c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54640cd980 0x7f54640c64f0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5454000c00 tx=0x7f54540046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 shutdown_connections 2024-03-06T07:28:00.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f545006b4d0 0x7f545006d990 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f546415c0e0 0x7f54640c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54640d28d0 0x7f54640c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 --2- 172.21.15.27:0/1468639730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54640cd980 0x7f54640c64f0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 >> 172.21.15.27:0/1468639730 conn(0x7f54640d1cd0 msgr2=0x7f546415c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 shutdown_connections 2024-03-06T07:28:00.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:00.027+0000 7f5469916640 1 -- 172.21.15.27:0/1468639730 wait complete. 2024-03-06T07:28:00.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:48.137883+0000', 'modified': '2024-03-06T07:27:58.540250+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 102, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7635}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7635': {'gid': 7635, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.27:6834/550876663', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 550876663}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 550876663}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:28:00.443 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7635:up:reconnect 2024-03-06T07:28:01.445 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:01.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.566+0000 7fa17ca84640 1 Processor -- start 2024-03-06T07:28:01.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.566+0000 7fa17ca84640 1 -- start start 2024-03-06T07:28:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.566+0000 7fa17ca84640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa1780b3b90 0x7fa1780d1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.566+0000 7fa17ca84640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 0x7fa17815eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.566+0000 7fa17ca84640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17806fa70 con 0x7fa1781553e0 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.566+0000 7fa17ca84640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17815f080 con 0x7fa1780b3b90 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.566+0000 7fa17ca84640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa178161f40 con 0x7fa17815e6b0 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.567+0000 7fa16ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa1780b3b90 0x7fa1780d1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.567+0000 7fa16ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa1780b3b90 0x7fa1780d1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36344/0 (socket says 172.21.15.27:36344) 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.567+0000 7fa16ffff640 1 -- 172.21.15.27:0/1735549257 learned_addr learned my addr 172.21.15.27:0/1735549257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.567+0000 7fa176d9d640 1 --2- 172.21.15.27:0/1735549257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 0x7fa17815eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.567+0000 7fa16ffff640 1 -- 172.21.15.27:0/1735549257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 msgr2=0x7fa17815eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.567+0000 7fa16ffff640 1 --2- 172.21.15.27:0/1735549257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 0x7fa17815eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.567+0000 7fa16ffff640 1 -- 172.21.15.27:0/1735549257 >> v1:172.21.15.69:6789/0 conn(0x7fa1781553e0 legacy=0x7fa1781623d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.567+0000 7fa16ffff640 1 -- 172.21.15.27:0/1735549257 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17806fd50 con 0x7fa1780b3b90 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.568+0000 7fa16ffff640 1 --2- 172.21.15.27:0/1735549257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 0x7fa1780d1750 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa160009870 tx=0x7fa16002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95b310f5e7a7c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.568+0000 7fa175d9b640 1 -- 172.21.15.27:0/1735549257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160034070 con 0x7fa1780b3b90 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.568+0000 7fa175d9b640 1 -- 172.21.15.27:0/1735549257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa160030840 con 0x7fa1780b3b90 2024-03-06T07:28:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.568+0000 7fa175d9b640 1 -- 172.21.15.27:0/1735549257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160030c10 con 0x7fa1780b3b90 2024-03-06T07:28:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.569+0000 7fa17ca84640 1 -- 172.21.15.27:0/1735549257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 msgr2=0x7fa1780d1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.569+0000 7fa17ca84640 1 --2- 172.21.15.27:0/1735549257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 0x7fa1780d1750 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa160009870 tx=0x7fa16002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.569+0000 7fa17ca84640 1 -- 172.21.15.27:0/1735549257 shutdown_connections 2024-03-06T07:28:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.569+0000 7fa17ca84640 1 --2- 172.21.15.27:0/1735549257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 0x7fa1780d1750 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.569+0000 7fa17ca84640 1 --2- 172.21.15.27:0/1735549257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 0x7fa17815eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.569+0000 7fa17ca84640 1 -- 172.21.15.27:0/1735549257 >> 172.21.15.27:0/1735549257 conn(0x7fa1780d0c30 msgr2=0x7fa1780d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.569+0000 7fa17ca84640 1 -- 172.21.15.27:0/1735549257 shutdown_connections 2024-03-06T07:28:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.569+0000 7fa17ca84640 1 -- 172.21.15.27:0/1735549257 wait complete. 2024-03-06T07:28:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.571+0000 7fa17ca84640 1 Processor -- start 2024-03-06T07:28:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.571+0000 7fa17ca84640 1 -- start start 2024-03-06T07:28:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.572+0000 7fa17ca84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 0x7fa178144660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.572+0000 7fa17ca84640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 0x7fa1781e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.572+0000 7fa17ca84640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa178143d30 0x7fa178141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.572+0000 7fa17ca84640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa178162ae0 con 0x7fa1780b3b90 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.572+0000 7fa17ca84640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa178070310 con 0x7fa178143d30 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.572+0000 7fa17ca84640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780b53c0 con 0x7fa17815e6b0 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.572+0000 7fa17659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 0x7fa178144660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.572+0000 7fa17659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 0x7fa178144660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36356/0 (socket says 172.21.15.27:36356) 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.572+0000 7fa17659c640 1 -- 172.21.15.27:0/2168152934 learned_addr learned my addr 172.21.15.27:0/2168152934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa16ffff640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 0x7fa1781e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa176d9d640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa178143d30 0x7fa178141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa17659c640 1 -- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 msgr2=0x7fa1781e7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa17659c640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 0x7fa1781e7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa17659c640 1 -- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa178143d30 msgr2=0x7fa178141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa17659c640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa178143d30 0x7fa178141a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa17659c640 1 -- 172.21.15.27:0/2168152934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1781e7eb0 con 0x7fa1780b3b90 2024-03-06T07:28:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa176d9d640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa178143d30 0x7fa178141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa17659c640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 0x7fa178144660 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c430 tx=0x7fa16800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa16effd640 1 -- 172.21.15.27:0/2168152934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1680052d0 con 0x7fa1780b3b90 2024-03-06T07:28:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa16effd640 1 -- 172.21.15.27:0/2168152934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168005490 con 0x7fa1780b3b90 2024-03-06T07:28:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1780ada50 con 0x7fa1780b3b90 2024-03-06T07:28:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa16effd640 1 -- 172.21.15.27:0/2168152934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168007260 con 0x7fa1780b3b90 2024-03-06T07:28:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.573+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1780ae010 con 0x7fa1780b3b90 2024-03-06T07:28:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.574+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1780752d0 con 0x7fa1780b3b90 2024-03-06T07:28:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.575+0000 7fa16effd640 1 -- 172.21.15.27:0/2168152934 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa168007400 con 0x7fa1780b3b90 2024-03-06T07:28:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.575+0000 7fa16effd640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa15806b500 0x7fa15806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.575+0000 7fa16effd640 1 -- 172.21.15.27:0/2168152934 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa1680897e0 con 0x7fa1780b3b90 2024-03-06T07:28:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.576+0000 7fa16ffff640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa15806b500 0x7fa15806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.576+0000 7fa16ffff640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa15806b500 0x7fa15806d9c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa160036000 tx=0x7fa16003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.580+0000 7fa16effd640 1 -- 172.21.15.27:0/2168152934 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa168058570 con 0x7fa1780b3b90 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.798+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1780b0470 con 0x7fa1780b3b90 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.799+0000 7fa16effd640 1 -- 172.21.15.27:0/2168152934 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fa168057e70 con 0x7fa1780b3b90 2024-03-06T07:28:01.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:01.802 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:01.802 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa15806b500 msgr2=0x7fa15806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa15806b500 0x7fa15806d9c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa160036000 tx=0x7fa16003b040 comp rx=0 tx=0).stop 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 msgr2=0x7fa178144660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 0x7fa178144660 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c430 tx=0x7fa16800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 shutdown_connections 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa15806b500 0x7fa15806d9c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa178143d30 0x7fa178141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa17815e6b0 0x7fa1781e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 --2- 172.21.15.27:0/2168152934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1780b3b90 0x7fa178144660 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 >> 172.21.15.27:0/2168152934 conn(0x7fa1780d0c30 msgr2=0x7fa1781623d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 shutdown_connections 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:01.801+0000 7fa17ca84640 1 -- 172.21.15.27:0/2168152934 wait complete. 2024-03-06T07:28:02.217 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:48.137883+0000', 'modified': '2024-03-06T07:28:01.537159+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 102, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7635}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7635': {'gid': 7635, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.27:6834/550876663', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 550876663}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 550876663}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7635, 'qdb_cluster': [7635]} 2024-03-06T07:28:02.217 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:02.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:28:02.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.322+0000 7fcc280f4640 1 Processor -- start 2024-03-06T07:28:02.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.323+0000 7fcc280f4640 1 -- start start 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.323+0000 7fcc280f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc200d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.323+0000 7fcc280f4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc200d0100 con 0x7fcc200d2a30 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.323+0000 7fcc280f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc200cf860 con 0x7fcc2015e100 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.323+0000 7fcc280f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc200cfb40 con 0x7fcc200d4c40 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.323+0000 7fcc26691640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcc200d2a30 0x7fcc200b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44514/0 (socket says 172.21.15.27:44514) 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.323+0000 7fcc26691640 1 -- 172.21.15.27:0/3056868735 learned_addr learned my addr 172.21.15.27:0/3056868735 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.324+0000 7fcc2568f640 1 --2- 172.21.15.27:0/3056868735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc200d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.324+0000 7fcc24e8e640 1 -- 172.21.15.27:0/3056868735 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677029681 0 0) 0x7fcc200d0100 con 0x7fcc200d2a30 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.324+0000 7fcc24e8e640 1 -- 172.21.15.27:0/3056868735 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc08003520 con 0x7fcc200d2a30 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.324+0000 7fcc24e8e640 1 -- 172.21.15.27:0/3056868735 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505558714 0 0) 0x7fcc200cf860 con 0x7fcc2015e100 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.324+0000 7fcc24e8e640 1 -- 172.21.15.27:0/3056868735 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc200d0100 con 0x7fcc2015e100 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.324+0000 7fcc2568f640 1 -- 172.21.15.27:0/3056868735 >> v1:172.21.15.69:6789/0 conn(0x7fcc2015e100 legacy=0x7fcc200d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.324+0000 7fcc24e8e640 1 -- 172.21.15.27:0/3056868735 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674448010 0 0) 0x7fcc08003520 con 0x7fcc200d2a30 2024-03-06T07:28:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.324+0000 7fcc2568f640 1 -- 172.21.15.27:0/3056868735 >> v1:172.21.15.27:6789/0 conn(0x7fcc200d2a30 legacy=0x7fcc200b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.324+0000 7fcc2568f640 1 -- 172.21.15.27:0/3056868735 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc200cfe20 con 0x7fcc200d4c40 2024-03-06T07:28:02.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.325+0000 7fcc2568f640 1 --2- 172.21.15.27:0/3056868735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc200d1cd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcc14009dc0 tx=0x7fcc1402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21c02a3d972a5432 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.325+0000 7fcc24e8e640 1 -- 172.21.15.27:0/3056868735 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc14039070 con 0x7fcc200d4c40 2024-03-06T07:28:02.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.325+0000 7fcc24e8e640 1 -- 172.21.15.27:0/3056868735 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1402c910 con 0x7fcc200d4c40 2024-03-06T07:28:02.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.325+0000 7fcc24e8e640 1 -- 172.21.15.27:0/3056868735 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18001670 con 0x7fcc200d4c40 2024-03-06T07:28:02.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.326+0000 7fcc280f4640 1 -- 172.21.15.27:0/3056868735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 msgr2=0x7fcc200d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.326+0000 7fcc280f4640 1 --2- 172.21.15.27:0/3056868735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc200d1cd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcc14009dc0 tx=0x7fcc1402bde0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.326+0000 7fcc280f4640 1 -- 172.21.15.27:0/3056868735 shutdown_connections 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.326+0000 7fcc280f4640 1 --2- 172.21.15.27:0/3056868735 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc200d1cd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.326+0000 7fcc280f4640 1 -- 172.21.15.27:0/3056868735 >> 172.21.15.27:0/3056868735 conn(0x7fcc200d3ef0 msgr2=0x7fcc200d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.326+0000 7fcc280f4640 1 -- 172.21.15.27:0/3056868735 shutdown_connections 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.326+0000 7fcc280f4640 1 -- 172.21.15.27:0/3056868735 wait complete. 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.327+0000 7fcc280f4640 1 Processor -- start 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.327+0000 7fcc280f4640 1 -- start start 2024-03-06T07:28:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc280f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc200d2a30 0x7fcc200cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc280f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc20152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc280f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2015e100 0x7fcc200cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc280f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2015e4e0 con 0x7fcc2015e100 2024-03-06T07:28:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc280f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc200cf510 con 0x7fcc200d2a30 2024-03-06T07:28:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc280f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2015e660 con 0x7fcc200d4c40 2024-03-06T07:28:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc26691640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2015e100 0x7fcc200cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc26691640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2015e100 0x7fcc200cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36370/0 (socket says 172.21.15.27:36370) 2024-03-06T07:28:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc26691640 1 -- 172.21.15.27:0/4045787712 learned_addr learned my addr 172.21.15.27:0/4045787712 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc2568f640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc20152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc25e90640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc200d2a30 0x7fcc200cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc26691640 1 -- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 msgr2=0x7fcc20152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc26691640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc20152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc26691640 1 -- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc200d2a30 msgr2=0x7fcc200cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc26691640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc200d2a30 0x7fcc200cc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc26691640 1 -- 172.21.15.27:0/4045787712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc200b9870 con 0x7fcc2015e100 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc2568f640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc20152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc26691640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2015e100 0x7fcc200cafc0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800c750 tx=0x7fcc1800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.328+0000 7fcc1e7fc640 1 -- 172.21.15.27:0/4045787712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18005ad0 con 0x7fcc2015e100 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.329+0000 7fcc1e7fc640 1 -- 172.21.15.27:0/4045787712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18005c70 con 0x7fcc2015e100 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.329+0000 7fcc280f4640 1 -- 172.21.15.27:0/4045787712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc200b8ad0 con 0x7fcc2015e100 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.329+0000 7fcc1e7fc640 1 -- 172.21.15.27:0/4045787712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18005e60 con 0x7fcc2015e100 2024-03-06T07:28:02.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.329+0000 7fcc280f4640 1 -- 172.21.15.27:0/4045787712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc200b8d90 con 0x7fcc2015e100 2024-03-06T07:28:02.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.331+0000 7fcc1e7fc640 1 -- 172.21.15.27:0/4045787712 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc180072b0 con 0x7fcc2015e100 2024-03-06T07:28:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.331+0000 7fcc1e7fc640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc0c06b4d0 0x7fcc0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.332+0000 7fcc1e7fc640 1 -- 172.21.15.27:0/4045787712 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcc1808a120 con 0x7fcc2015e100 2024-03-06T07:28:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.332+0000 7fcc2adbc740 1 Processor -- start 2024-03-06T07:28:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.332+0000 7fcc25e90640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc0c06b4d0 0x7fcc0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.332+0000 7fcc2adbc740 1 -- start start 2024-03-06T07:28:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.332+0000 7fcc25e90640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc0c06b4d0 0x7fcc0c06d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcc100010e0 tx=0x7fcc10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0c9a870 0x5652d0c9ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0c9e8c0 0x5652d0ca0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cca600 0x5652d0ccaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d0c046b0 con 0x5652d0c9a870 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d0bada60 con 0x5652d0cca600 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652d0c00750 con 0x5652d0c9e8c0 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc25e90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0c9a870 0x5652d0c9ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc25e90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0c9a870 0x5652d0c9ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36384/0 (socket says 172.21.15.27:36384) 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc25e90640 1 -- 172.21.15.27:0/3926905437 learned_addr learned my addr 172.21.15.27:0/3926905437 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc2568f640 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0c9e8c0 0x5652d0ca0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc26691640 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cca600 0x5652d0ccaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc25e90640 1 -- 172.21.15.27:0/3926905437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0c9e8c0 msgr2=0x5652d0ca0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc25e90640 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0c9e8c0 0x5652d0ca0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc25e90640 1 -- 172.21.15.27:0/3926905437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cca600 msgr2=0x5652d0ccaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc25e90640 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cca600 0x5652d0ccaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc25e90640 1 -- 172.21.15.27:0/3926905437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652d0bcbf30 con 0x5652d0c9a870 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc26691640 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cca600 0x5652d0ccaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc25e90640 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0c9a870 0x5652d0c9ac50 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fcc10014840 tx=0x7fcc10010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd2e7ee1fa98c85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.333+0000 7fcc077fe640 1 -- 172.21.15.27:0/3926905437 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10024070 con 0x5652d0c9a870 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc077fe640 1 -- 172.21.15.27:0/3926905437 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1000dbc0 con 0x5652d0c9a870 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc077fe640 1 -- 172.21.15.27:0/3926905437 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1001f400 con 0x5652d0c9a870 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc2adbc740 1 -- 172.21.15.27:0/3926905437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0c9a870 msgr2=0x5652d0c9ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0c9a870 0x5652d0c9ac50 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fcc10014840 tx=0x7fcc10010000 comp rx=0 tx=0).stop 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc2adbc740 1 -- 172.21.15.27:0/3926905437 shutdown_connections 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cca600 0x5652d0ccaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0c9e8c0 0x5652d0ca0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/3926905437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0c9a870 0x5652d0c9ac50 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc2adbc740 1 -- 172.21.15.27:0/3926905437 >> 172.21.15.27:0/3926905437 conn(0x5652d0c9c8f0 msgr2=0x5652d0c9e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc2adbc740 1 -- 172.21.15.27:0/3926905437 shutdown_connections 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.334+0000 7fcc2adbc740 1 -- 172.21.15.27:0/3926905437 wait complete. 2024-03-06T07:28:02.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.335+0000 7fcc2adbc740 1 -- start start 2024-03-06T07:28:02.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.335+0000 7fcc2adbc740 1 Processor -- start 2024-03-06T07:28:02.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cca600 0x5652d0ce56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cce5b0 0x5652d0ccbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 0x5652d0cc6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d0c9ed50 con 0x5652d0cca600 2024-03-06T07:28:02.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d0bc0ea0 con 0x5652d0cc6a70 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652d0c9f030 con 0x5652d0cce5b0 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc2adbc740 10 client.0 _ll_register_callbacks cb 0x5652d0bc0870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc25e90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cca600 0x5652d0ce56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc25e90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cca600 0x5652d0ce56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36394/0 (socket says 172.21.15.27:36394) 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc25e90640 1 -- 172.21.15.27:0/1398044954 learned_addr learned my addr 172.21.15.27:0/1398044954 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc2568f640 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cce5b0 0x5652d0ccbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc26691640 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 0x5652d0cc6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc25e90640 1 -- 172.21.15.27:0/1398044954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cce5b0 msgr2=0x5652d0ccbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc25e90640 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cce5b0 0x5652d0ccbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc25e90640 1 -- 172.21.15.27:0/1398044954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 msgr2=0x5652d0cc6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc25e90640 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 0x5652d0cc6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc25e90640 1 -- 172.21.15.27:0/1398044954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652d0bc1270 con 0x5652d0cca600 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc26691640 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 0x5652d0cc6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:02.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc25e90640 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cca600 0x5652d0ce56f0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000d810 tx=0x7fcc10000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2d67b8bf36154c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.336+0000 7fcc04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc04ff9640 1 -- 172.21.15.27:0/1398044954 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10024040 con 0x5652d0cca600 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc04ff9640 1 -- 172.21.15.27:0/1398044954 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcc10005270 con 0x5652d0cca600 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2adbc740 1 -- 172.21.15.27:0/1398044954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc1000dbc0 con 0x5652d0cca600 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc04ff9640 1 -- 172.21.15.27:0/1398044954 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc10035050 con 0x5652d0cca600 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2adbc740 10 client.7674 fetch_fsmap learned FSMap version 67 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2adbc740 1 -- 172.21.15.27:0/1398044954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x5652d09192d0 con 0x5652d0cca600 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc04ff9640 1 -- 172.21.15.27:0/1398044954 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1000dbc0 con 0x5652d0cca600 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc04ff9640 1 -- 172.21.15.27:0/1398044954 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcc10038020 con 0x5652d0cca600 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2adbc740 10 client.7674 fetch_fsmap finished waiting for FSMap version 67 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2adbc740 10 client.7674 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2adbc740 20 client.7674 populate_metadata read hostname 'smithi027' 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] conn(0x5652d0cc2d60 0x5652d0cc3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2adbc740 4 client.7674 mds_command: new command op to 7635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2adbc740 1 -- 172.21.15.27:0/1398044954 --> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652d0c11bf0 con 0x5652d0cc2d60 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.337+0000 7fcc2568f640 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] conn(0x5652d0cc2d60 0x5652d0cc3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc04ff9640 10 client.7674 ms_handle_connect on v2:172.21.15.27:6832/550876663 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2568f640 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] conn(0x5652d0cc2d60 0x5652d0cc3140 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c4d437f97e21cd0 server_cookie=5fc707f7889d9e88 in_seq=0 out_seq=0 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc04ff9640 1 -- 172.21.15.27:0/1398044954 <== mds.0 v2:172.21.15.27:6832/550876663 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5652d0c11bf0 con 0x5652d0cc2d60 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc04ff9640 10 client.7674 handle_command_reply: tid=0 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2adbc740 1 client.7674 shutdown 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2adbc740 1 -- 172.21.15.27:0/1398044954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cca600 msgr2=0x5652d0ce56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cca600 0x5652d0ce56f0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000d810 tx=0x7fcc10000ed0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2adbc740 1 -- 172.21.15.27:0/1398044954 shutdown_connections 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] conn(0x5652d0cc2d60 0x5652d0cc3140 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 0x5652d0cc6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc04ff9640 0 client.7674 ms_handle_reset on v2:172.21.15.27:6832/550876663 2024-03-06T07:28:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cce5b0 0x5652d0ccbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/1398044954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cca600 0x5652d0ce56f0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.338+0000 7fcc2adbc740 1 -- 172.21.15.27:0/1398044954 >> 172.21.15.27:0/1398044954 conn(0x5652d0c9a870 msgr2=0x5652d0cc85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.339+0000 7fcc2adbc740 1 -- 172.21.15.27:0/1398044954 shutdown_connections 2024-03-06T07:28:02.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.339+0000 7fcc2adbc740 1 -- 172.21.15.27:0/1398044954 wait complete. 2024-03-06T07:28:02.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.339+0000 7fcc2adbc740 20 client.7674 trim_cache size 0 max 16384 2024-03-06T07:28:02.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.356+0000 7fcc2adbc740 1 Processor -- start 2024-03-06T07:28:02.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.356+0000 7fcc2adbc740 1 -- start start 2024-03-06T07:28:02.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.356+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 0x5652d0cc4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.356+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 0x5652d0ca1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.356+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cce5b0 0x5652d0ce60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.356+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc20200350 con 0x5652d0cce5b0 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.356+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201ff490 con 0x5652d0cc6a70 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.356+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201fe8c0 con 0x5652d0cca600 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc26691640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cce5b0 0x5652d0ce60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc26691640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cce5b0 0x5652d0ce60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36398/0 (socket says 172.21.15.27:36398) 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc26691640 1 -- 172.21.15.27:0/2558152392 learned_addr learned my addr 172.21.15.27:0/2558152392 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc26691640 1 -- 172.21.15.27:0/2558152392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 msgr2=0x5652d0ca1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc25e90640 1 --2- 172.21.15.27:0/2558152392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 0x5652d0cc4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc2568f640 1 --2- 172.21.15.27:0/2558152392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 0x5652d0ca1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc26691640 1 --2- 172.21.15.27:0/2558152392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 0x5652d0ca1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc26691640 1 -- 172.21.15.27:0/2558152392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 msgr2=0x5652d0cc4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc26691640 1 --2- 172.21.15.27:0/2558152392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 0x5652d0cc4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc26691640 1 -- 172.21.15.27:0/2558152392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652d09e36b0 con 0x5652d0cce5b0 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc26691640 1 --2- 172.21.15.27:0/2558152392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cce5b0 0x5652d0ce60b0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800a430 tx=0x7fcc18077040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59ad0b0f78e51c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc06ffd640 1 -- 172.21.15.27:0/2558152392 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18060e30 con 0x5652d0cce5b0 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc06ffd640 1 -- 172.21.15.27:0/2558152392 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18056a20 con 0x5652d0cce5b0 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.357+0000 7fcc06ffd640 1 -- 172.21.15.27:0/2558152392 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18054d90 con 0x5652d0cce5b0 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2558152392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cce5b0 msgr2=0x5652d0ce60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2558152392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cce5b0 0x5652d0ce60b0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800a430 tx=0x7fcc18077040 comp rx=0 tx=0).stop 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2558152392 shutdown_connections 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2558152392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cce5b0 0x5652d0ce60b0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2558152392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 0x5652d0ca1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2558152392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cc6a70 0x5652d0cc4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2558152392 >> 172.21.15.27:0/2558152392 conn(0x5652d0c9a870 msgr2=0x5652d0c9e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2558152392 shutdown_connections 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2558152392 wait complete. 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 -- start start 2024-03-06T07:28:02.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.358+0000 7fcc2adbc740 1 Processor -- start 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cc6a70 0x5652d0d55140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 0x5652d0cf7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc2adbc740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cce5b0 0x5652d0d083a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d0c9ed50 con 0x5652d0cc6a70 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201feec0 con 0x5652d0cce5b0 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc2adbc740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652d0cc1fc0 con 0x5652d0cca600 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc2adbc740 10 client.0 _ll_register_callbacks cb 0x7fcc10014710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc25e90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cc6a70 0x5652d0d55140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc25e90640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cc6a70 0x5652d0d55140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36402/0 (socket says 172.21.15.27:36402) 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.359+0000 7fcc25e90640 1 -- 172.21.15.27:0/2393024730 learned_addr learned my addr 172.21.15.27:0/2393024730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc26691640 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cce5b0 0x5652d0d083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2568f640 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 0x5652d0cf7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc25e90640 1 -- 172.21.15.27:0/2393024730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 msgr2=0x5652d0cf7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc25e90640 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 0x5652d0cf7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc25e90640 1 -- 172.21.15.27:0/2393024730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cce5b0 msgr2=0x5652d0d083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc25e90640 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cce5b0 0x5652d0d083a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc25e90640 1 -- 172.21.15.27:0/2393024730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652d0cf7f80 con 0x5652d0cc6a70 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc26691640 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cce5b0 0x5652d0d083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc25e90640 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cc6a70 0x5652d0d55140 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000d8f0 tx=0x7fcc10013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f822724b5a757cfc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc04ff9640 1 -- 172.21.15.27:0/2393024730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10024070 con 0x5652d0cc6a70 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc04ff9640 1 -- 172.21.15.27:0/2393024730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcc100037f0 con 0x5652d0cc6a70 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2393024730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc18056a20 con 0x5652d0cc6a70 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc04ff9640 1 -- 172.21.15.27:0/2393024730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc10042050 con 0x5652d0cc6a70 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2adbc740 10 client.7680 fetch_fsmap learned FSMap version 67 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2393024730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x5652d0ce51c0 con 0x5652d0cc6a70 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc04ff9640 1 -- 172.21.15.27:0/2393024730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18056a20 con 0x5652d0cc6a70 2024-03-06T07:28:02.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc04ff9640 1 -- 172.21.15.27:0/2393024730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcc10045020 con 0x5652d0cc6a70 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2adbc740 10 client.7680 fetch_fsmap finished waiting for FSMap version 67 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2adbc740 10 client.7680 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2adbc740 20 client.7680 populate_metadata read hostname 'smithi027' 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] conn(0x5652d0cc72b0 0x5652d0cc4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2adbc740 4 client.7680 mds_command: new command op to 7635 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.360+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2393024730 --> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5652d0cccfe0 con 0x5652d0cc72b0 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2568f640 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] conn(0x5652d0cc72b0 0x5652d0cc4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2568f640 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] conn(0x5652d0cc72b0 0x5652d0cc4da0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=238284990d9546f3 server_cookie=a52670c468d39b3f in_seq=0 out_seq=0 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc04ff9640 10 client.7680 ms_handle_connect on v2:172.21.15.27:6832/550876663 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc04ff9640 1 -- 172.21.15.27:0/2393024730 <== mds.0 v2:172.21.15.27:6832/550876663 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5652d0cccfe0 con 0x5652d0cc72b0 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc04ff9640 10 client.7680 handle_command_reply: tid=0 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 client.7680 shutdown 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2393024730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cc6a70 msgr2=0x5652d0d55140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cc6a70 0x5652d0d55140 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000d8f0 tx=0x7fcc10013000 comp rx=0 tx=0).stop 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2393024730 shutdown_connections 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.27:6832/550876663,v1:172.21.15.27:6834/550876663] conn(0x5652d0cc72b0 0x5652d0cc4da0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5652d0cce5b0 0x5652d0d083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc04ff9640 0 client.7680 ms_handle_reset on v2:172.21.15.27:6832/550876663 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5652d0cca600 0x5652d0cf7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 --2- 172.21.15.27:0/2393024730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652d0cc6a70 0x5652d0d55140 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2393024730 >> 172.21.15.27:0/2393024730 conn(0x5652d0c9a870 msgr2=0x5652d0cc9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2393024730 shutdown_connections 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.361+0000 7fcc2adbc740 1 -- 172.21.15.27:0/2393024730 wait complete. 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc2adbc740 20 client.7680 trim_cache size 0 max 16384 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:28:02.369 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7635, 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 3.8324296549999999, 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 67, 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 102, 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 102, 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 4.3629866670000004 2024-03-06T07:28:02.370 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:28:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 -- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc0c06b4d0 msgr2=0x7fcc0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc0c06b4d0 0x7fcc0c06d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcc100010e0 tx=0x7fcc10009040 comp rx=0 tx=0).stop 2024-03-06T07:28:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 -- 172.21.15.27:0/4045787712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2015e100 msgr2=0x7fcc200cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2015e100 0x7fcc200cafc0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800c750 tx=0x7fcc1800cc10 comp rx=0 tx=0).stop 2024-03-06T07:28:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 -- 172.21.15.27:0/4045787712 shutdown_connections 2024-03-06T07:28:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc0c06b4d0 0x7fcc0c06d990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2015e100 0x7fcc200cafc0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc200d4c40 0x7fcc20152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 --2- 172.21.15.27:0/4045787712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc200d2a30 0x7fcc200cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 -- 172.21.15.27:0/4045787712 >> 172.21.15.27:0/4045787712 conn(0x7fcc200d3ef0 msgr2=0x7fcc2006c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 -- 172.21.15.27:0/4045787712 shutdown_connections 2024-03-06T07:28:02.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:02.362+0000 7fcc280f4640 1 -- 172.21.15.27:0/4045787712 wait complete. 2024-03-06T07:28:02.375 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:28:02.375 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:28:02.375 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:28:02.375 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> subdir = "delete_me" 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> size_unit = 1024 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> file_multiplier = 200 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> for i in range(0, file_multiplier): 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> f.write(size * '"'"'x'"'"') 2024-03-06T07:28:02.376 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:28:03.536 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:28:03.534+0000 7fb50f49b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:09.267 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json config get mds_max_purge_ops 2024-03-06T07:28:09.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:09.268 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:28:09.380 INFO:teuthology.orchestra.run.smithi069.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:28:09.380 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:28:09.381 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json config get mds_max_purge_files 2024-03-06T07:28:09.493 INFO:teuthology.orchestra.run.smithi069.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:28:09.493 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:28:09.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:09.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.612+0000 7f2b46b4b640 1 Processor -- start 2024-03-06T07:28:09.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.612+0000 7f2b46b4b640 1 -- start start 2024-03-06T07:28:09.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.613+0000 7f2b46b4b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2b400c9a80 0x7f2b400d49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.613+0000 7f2b46b4b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b400c79d0 con 0x7f2b400d1c60 2024-03-06T07:28:09.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.613+0000 7f2b46b4b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b400c7cb0 con 0x7f2b400d2460 2024-03-06T07:28:09.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.613+0000 7f2b46b4b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b400c85e0 con 0x7f2b400c9a80 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.613+0000 7f2b3ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2b400d1c60 0x7f2b400b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47466/0 (socket says 172.21.15.27:47466) 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.613+0000 7f2b3ffff640 1 -- 172.21.15.27:0/2748171428 learned_addr learned my addr 172.21.15.27:0/2748171428 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.613+0000 7f2b450e8640 1 --2- 172.21.15.27:0/2748171428 >> v2:172.21.15.69:3300/0 conn(0x7f2b400c9a80 0x7f2b400d49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.614+0000 7f2b3f7fe640 1 -- 172.21.15.27:0/2748171428 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485642478 0 0) 0x7f2b400c79d0 con 0x7f2b400d1c60 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.614+0000 7f2b3f7fe640 1 -- 172.21.15.27:0/2748171428 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b30003520 con 0x7f2b400d1c60 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.614+0000 7f2b3f7fe640 1 -- 172.21.15.27:0/2748171428 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563889619 0 0) 0x7f2b400c7cb0 con 0x7f2b400d2460 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.614+0000 7f2b3f7fe640 1 -- 172.21.15.27:0/2748171428 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b400c79d0 con 0x7f2b400d2460 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.614+0000 7f2b450e8640 1 -- 172.21.15.27:0/2748171428 >> v1:172.21.15.69:6789/0 conn(0x7f2b400d2460 legacy=0x7f2b400d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.614+0000 7f2b450e8640 1 -- 172.21.15.27:0/2748171428 >> v1:172.21.15.27:6789/0 conn(0x7f2b400d1c60 legacy=0x7f2b400b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.614+0000 7f2b450e8640 1 -- 172.21.15.27:0/2748171428 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b400c7f90 con 0x7f2b400c9a80 2024-03-06T07:28:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.615+0000 7f2b450e8640 1 --2- 172.21.15.27:0/2748171428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 0x7f2b400d49a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009dc0 tx=0x7f2b3802bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2dfef26ec1f2d02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.615+0000 7f2b3f7fe640 1 -- 172.21.15.27:0/2748171428 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38039070 con 0x7f2b400c9a80 2024-03-06T07:28:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.615+0000 7f2b3f7fe640 1 -- 172.21.15.27:0/2748171428 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3802ccd0 con 0x7f2b400c9a80 2024-03-06T07:28:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.615+0000 7f2b3f7fe640 1 -- 172.21.15.27:0/2748171428 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38034660 con 0x7f2b400c9a80 2024-03-06T07:28:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.616+0000 7f2b46b4b640 1 -- 172.21.15.27:0/2748171428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 msgr2=0x7f2b400d49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.616+0000 7f2b46b4b640 1 --2- 172.21.15.27:0/2748171428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 0x7f2b400d49a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009dc0 tx=0x7f2b3802bf20 comp rx=0 tx=0).stop 2024-03-06T07:28:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.616+0000 7f2b46b4b640 1 -- 172.21.15.27:0/2748171428 shutdown_connections 2024-03-06T07:28:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.616+0000 7f2b46b4b640 1 --2- 172.21.15.27:0/2748171428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 0x7f2b400d49a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.616+0000 7f2b46b4b640 1 -- 172.21.15.27:0/2748171428 >> 172.21.15.27:0/2748171428 conn(0x7f2b400d3e80 msgr2=0x7f2b400d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.616+0000 7f2b46b4b640 1 -- 172.21.15.27:0/2748171428 shutdown_connections 2024-03-06T07:28:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.616+0000 7f2b46b4b640 1 -- 172.21.15.27:0/2748171428 wait complete. 2024-03-06T07:28:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.617+0000 7f2b46b4b640 1 Processor -- start 2024-03-06T07:28:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.617+0000 7f2b46b4b640 1 -- start start 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b46b4b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 0x7f2b400cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b46b4b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b400d1c60 0x7f2b40154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b46b4b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b400d2460 0x7f2b400cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b46b4b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b400d1280 con 0x7f2b400d1c60 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b46b4b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b400d1100 con 0x7f2b400c9a80 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b46b4b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4015e550 con 0x7f2b400d2460 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b3ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b400d1c60 0x7f2b40154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b3ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b400d1c60 0x7f2b40154080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53636/0 (socket says 172.21.15.27:53636) 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b3ffff640 1 -- 172.21.15.27:0/3079518802 learned_addr learned my addr 172.21.15.27:0/3079518802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b448e7640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 0x7f2b400cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.618+0000 7f2b450e8640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b400d2460 0x7f2b400cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b3ffff640 1 -- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b400d2460 msgr2=0x7f2b400cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b3ffff640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b400d2460 0x7f2b400cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b3ffff640 1 -- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 msgr2=0x7f2b400cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b3ffff640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 0x7f2b400cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b3ffff640 1 -- 172.21.15.27:0/3079518802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b400cb460 con 0x7f2b400d1c60 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b450e8640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b400d2460 0x7f2b400cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b448e7640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 0x7f2b400cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b3ffff640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b400d1c60 0x7f2b40154080 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00c440 tx=0x7f2b2c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b3d7fa640 1 -- 172.21.15.27:0/3079518802 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c017070 con 0x7f2b400d1c60 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b3d7fa640 1 -- 172.21.15.27:0/3079518802 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2c005d00 con 0x7f2b400d1c60 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b3d7fa640 1 -- 172.21.15.27:0/3079518802 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c0072b0 con 0x7f2b400d1c60 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b400cb740 con 0x7f2b400d1c60 2024-03-06T07:28:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.619+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b400cbd00 con 0x7f2b400d1c60 2024-03-06T07:28:09.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.621+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b400752d0 con 0x7f2b400d1c60 2024-03-06T07:28:09.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.622+0000 7f2b3d7fa640 1 -- 172.21.15.27:0/3079518802 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b2c01a070 con 0x7f2b400d1c60 2024-03-06T07:28:09.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.622+0000 7f2b3d7fa640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b2406b4d0 0x7f2b2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.622+0000 7f2b3d7fa640 1 -- 172.21.15.27:0/3079518802 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2b2c089510 con 0x7f2b400d1c60 2024-03-06T07:28:09.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.622+0000 7f2b448e7640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b2406b4d0 0x7f2b2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.623+0000 7f2b448e7640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b2406b4d0 0x7f2b2406d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2b34007fd0 tx=0x7f2b34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.627+0000 7f2b3d7fa640 1 -- 172.21.15.27:0/3079518802 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b2c057db0 con 0x7f2b400d1c60 2024-03-06T07:28:09.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.917+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b400d2bc0 con 0x7f2b400d1c60 2024-03-06T07:28:09.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.917+0000 7f2b3d7fa640 1 -- 172.21.15.27:0/3079518802 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f2b2c05be50 con 0x7f2b400d1c60 2024-03-06T07:28:09.920 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:09.920 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:09.920 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b2406b4d0 msgr2=0x7f2b2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b2406b4d0 0x7f2b2406d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2b34007fd0 tx=0x7f2b34009040 comp rx=0 tx=0).stop 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b400d1c60 msgr2=0x7f2b40154080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b400d1c60 0x7f2b40154080 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00c440 tx=0x7f2b2c00c900 comp rx=0 tx=0).stop 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 shutdown_connections 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b2406b4d0 0x7f2b2406d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b400d2460 0x7f2b400cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b400d1c60 0x7f2b40154080 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 --2- 172.21.15.27:0/3079518802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b400c9a80 0x7f2b400cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 >> 172.21.15.27:0/3079518802 conn(0x7f2b400d3e80 msgr2=0x7f2b400b8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 shutdown_connections 2024-03-06T07:28:09.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:09.920+0000 7f2b46b4b640 1 -- 172.21.15.27:0/3079518802 wait complete. 2024-03-06T07:28:09.935 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:28:10.081 INFO:teuthology.orchestra.run.smithi027.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":24,"msgr_send_messages":24,"msgr_recv_bytes":15337,"msgr_send_bytes":4223506,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.015750600,"msgr_running_send_time":0.006810693,"msgr_running_recv_time":0.005606979,"msgr_running_fast_dispatch_time":0.002157338,"msgr_send_messages_queue_lat":{"avgcount":23,"sum":0.009547115,"avgtime":0.000415091},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000206,"avgtime":0.000000103},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":4509,"msgr_send_messages":10607,"msgr_recv_bytes":1560107,"msgr_send_bytes":10120048,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.519991669,"msgr_running_send_time":0.280854842,"msgr_running_recv_time":0.270438163,"msgr_running_fast_dispatch_time":0.003754344,"msgr_send_messages_queue_lat":{"avgcount":10606,"sum":0.144620725,"avgtime":0.000013635},"msgr_handle_ack_lat":{"avgcount":14887,"sum":0.002970521,"avgtime":0.000000199},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":92224},"AsyncMessenger::Worker-2":{"msgr_recv_messages":27,"msgr_send_messages":24,"msgr_recv_bytes":108770,"msgr_send_bytes":8944,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.004104319,"msgr_running_send_time":0.000813867,"msgr_running_recv_time":0.005069749,"msgr_running_fast_dispatch_time":0.000301858,"msgr_send_messages_queue_lat":{"avgcount":23,"sum":0.004089936,"avgtime":0.000177823},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":104992,"msgr_send_encrypted_bytes":6032},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":352,"complete_latency":{"avgcount":76,"sum":1.192663215,"avgtime":0.015692937}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000088667,"avgtime":0.000022166}},"mds":{"request":3249,"reply":3247,"reply_latency":{"avgcount":3247,"sum":3.701995960,"avgtime":0.001140128},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3232,"inodes_top":0,"inodes_bottom":3232,"inodes_pin_tail":0,"inodes_pinned":3232,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3254,"traverse_hit":3253,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3199,"root_rbytes":5463040,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":32,"ceph_cap_op_grant":3239,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":1211,"handle_client_caps_dirty":1132,"handle_client_cap_release":0,"process_request_cap_release":3276},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":37,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":37,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":4454,"evex":0,"evtrm":0,"ev":4458,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":12946905,"rdpos":4195685,"jlat":{"avgcount":12,"sum":0.054519627,"avgtime":0.004543302},"replayed":4},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3248,"dn+":3248,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":79176,"heap":248164},"mds_server":{"dispatch_client_request":3251,"dispatch_server_request":0,"handle_client_request":3249,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.174951590,"avgtime":0.000992172},"req_getattr_latency":{"avgcount":6,"sum":0.009656876,"avgtime":0.001609479},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002708406,"avgtime":0.002708406},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.230416875,"avgtime":0.076805625},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":37,"sum":0.284262213,"avgtime":0.007682762},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3087,"average_load":3087,"avg_session_uptime":817,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":9578777,"buffer_anon_items":2987,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":9905912,"mds_co_items":47849,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":65,"op_send_bytes":8648088,"op_resend":0,"op_reply":65,"op_latency":{"avgcount":65,"sum":0.121996950,"avgtime":0.001876876},"op_inflight":0,"oplen_avg":{"avgcount":65,"sum":121},"op":65,"op_r":36,"op_w":29,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":12,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":7,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":16,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":297,"max":104857600,"get_started":0,"get":4556,"get_sum":1271649,"get_or_fail_fail":0,"get_or_fail_success":4556,"take":0,"take_sum":0,"put":4555,"put_sum":1271352,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":65,"take_sum":8650020,"put":65,"put_sum":8650020,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":65,"take_sum":65,"put":65,"put_sum":65,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a619c9a820":{"val":363993,"max":3758096384,"get_started":0,"get":4454,"get_sum":8750405,"get_or_fail_fail":0,"get_or_fail_success":4454,"take":0,"take_sum":0,"put":12,"put_sum":8386412,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:28:10.082 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.03e-07, "sum": 2.06e-07 }, "msgr_recv_bytes": 15337, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 24, "msgr_running_fast_dispatch_time": 0.002157338, "msgr_running_recv_time": 0.005606979, "msgr_running_send_time": 0.006810693, "msgr_running_total_time": 0.0157506, "msgr_send_bytes": 4223506, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 24, "msgr_send_messages_queue_lat": { "avgcount": 23, "avgtime": 0.000415091, "sum": 0.009547115 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 14887, "avgtime": 1.99e-07, "sum": 0.002970521 }, "msgr_recv_bytes": 1560107, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 4509, "msgr_running_fast_dispatch_time": 0.003754344, "msgr_running_recv_time": 0.270438163, "msgr_running_send_time": 0.280854842, "msgr_running_total_time": 0.519991669, "msgr_send_bytes": 10120048, "msgr_send_encrypted_bytes": 92224, "msgr_send_messages": 10607, "msgr_send_messages_queue_lat": { "avgcount": 10606, "avgtime": 1.3635e-05, "sum": 0.144620725 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 108770, "msgr_recv_encrypted_bytes": 104992, "msgr_recv_messages": 27, "msgr_running_fast_dispatch_time": 0.000301858, "msgr_running_recv_time": 0.005069749, "msgr_running_send_time": 0.000813867, "msgr_running_total_time": 0.004104319, "msgr_send_bytes": 8944, "msgr_send_encrypted_bytes": 6032, "msgr_send_messages": 24, "msgr_send_messages_queue_lat": { "avgcount": 23, "avgtime": 0.000177823, "sum": 0.004089936 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 76, "avgtime": 0.015692937, "sum": 1.192663215 }, "queue_len": 352 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.2166e-05, "sum": 8.8667e-05 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3239, "ceph_cap_op_revoke": 32, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 1211, "handle_client_caps_dirty": 1132, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3232, "inodes_bottom": 3232, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3232, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3276, "q": 0, "reply": 3247, "reply_latency": { "avgcount": 3247, "avgtime": 0.001140128, "sum": 3.70199596 }, "request": 3249, "root_rbytes": 5463040, "root_rfiles": 3199, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3254, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3253, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 37, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 37, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 4458, "evadd": 4454, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.004543302, "sum": 0.054519627 }, "rdpos": 4195685, "replayed": 4, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 12946905 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3248, "dn+": 3248, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 79176 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3251, "dispatch_server_request": 0, "handle_client_request": 3249, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000992172, "sum": 3.17495159 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.001609479, "sum": 0.009656876 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002708406, "sum": 0.002708406 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.076805625, "sum": 0.230416875 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 37, "avgtime": 0.007682762, "sum": 0.284262213 } }, "mds_sessions": { "average_load": 3087, "avg_session_uptime": 817, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3087 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 9578777, "buffer_anon_items": 2987, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 9905912, "mds_co_items": 47849, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 16, "op": 65, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 65, "avgtime": 0.001876876, "sum": 0.12199695 }, "op_pg": 0, "op_r": 36, "op_reply": 65, "op_resend": 0, "op_rmw": 0, "op_send": 65, "op_send_bytes": 8648088, "op_w": 29, "oplen_avg": { "avgcount": 65, "sum": 121 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 7, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 12, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 4556, "get_or_fail_fail": 0, "get_or_fail_success": 4556, "get_started": 0, "get_sum": 1271649, "max": 104857600, "put": 4555, "put_sum": 1271352, "take": 0, "take_sum": 0, "val": 297, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 65, "put_sum": 8650020, "take": 65, "take_sum": 8650020, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 65, "put_sum": 65, "take": 65, "take_sum": 65, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a619c9a820": { "get": 4454, "get_or_fail_fail": 0, "get_or_fail_success": 4454, "get_started": 0, "get_sum": 8750405, "max": 3758096384, "put": 12, "put_sum": 8386412, "take": 0, "take_sum": 0, "val": 363993, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:28:10.082 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/37, 0/37 2024-03-06T07:28:11.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:11.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.215+0000 7f73f20c9640 1 Processor -- start 2024-03-06T07:28:11.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.215+0000 7f73f20c9640 1 -- start start 2024-03-06T07:28:11.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.215+0000 7f73f20c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73f20c9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec0d0100 con 0x7f73ec0d4c40 2024-03-06T07:28:11.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73f20c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec0cf860 con 0x7f73ec0d2a30 2024-03-06T07:28:11.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73f20c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec0cfb40 con 0x7f73ec15e100 2024-03-06T07:28:11.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73eaffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f73ec0d4c40 0x7f73ec0d2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47472/0 (socket says 172.21.15.27:47472) 2024-03-06T07:28:11.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73eaffd640 1 -- 172.21.15.27:0/664606397 learned_addr learned my addr 172.21.15.27:0/664606397 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:11.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73eb7fe640 1 --2- 172.21.15.27:0/664606397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73ea7fc640 1 -- 172.21.15.27:0/664606397 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980299874 0 0) 0x7f73ec0d0100 con 0x7f73ec0d4c40 2024-03-06T07:28:11.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73ea7fc640 1 -- 172.21.15.27:0/664606397 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d4003520 con 0x7f73ec0d4c40 2024-03-06T07:28:11.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73eb7fe640 1 -- 172.21.15.27:0/664606397 >> v1:172.21.15.69:6789/0 conn(0x7f73ec0d2a30 legacy=0x7f73ec0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73ea7fc640 1 -- 172.21.15.27:0/664606397 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043080849 0 0) 0x7f73ec0cf860 con 0x7f73ec0d2a30 2024-03-06T07:28:11.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73eb7fe640 1 -- 172.21.15.27:0/664606397 >> v1:172.21.15.27:6789/0 conn(0x7f73ec0d4c40 legacy=0x7f73ec0d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.216+0000 7f73eb7fe640 1 -- 172.21.15.27:0/664606397 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec0cfe20 con 0x7f73ec15e100 2024-03-06T07:28:11.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.218+0000 7f73eb7fe640 1 --2- 172.21.15.27:0/664606397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0d1cd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0097d0 tx=0x7f73dc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17ff07d170b8d153 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.218+0000 7f73ea7fc640 1 -- 172.21.15.27:0/664606397 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc035020 con 0x7f73ec15e100 2024-03-06T07:28:11.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.218+0000 7f73ea7fc640 1 -- 172.21.15.27:0/664606397 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc031890 con 0x7f73ec15e100 2024-03-06T07:28:11.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.218+0000 7f73ea7fc640 1 -- 172.21.15.27:0/664606397 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc031c60 con 0x7f73ec15e100 2024-03-06T07:28:11.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.219+0000 7f73f20c9640 1 -- 172.21.15.27:0/664606397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 msgr2=0x7f73ec0d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.219+0000 7f73f20c9640 1 --2- 172.21.15.27:0/664606397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0d1cd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0097d0 tx=0x7f73dc02b930 comp rx=0 tx=0).stop 2024-03-06T07:28:11.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.219+0000 7f73f20c9640 1 -- 172.21.15.27:0/664606397 shutdown_connections 2024-03-06T07:28:11.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.219+0000 7f73f20c9640 1 --2- 172.21.15.27:0/664606397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0d1cd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.219+0000 7f73f20c9640 1 -- 172.21.15.27:0/664606397 >> 172.21.15.27:0/664606397 conn(0x7f73ec0d3ef0 msgr2=0x7f73ec0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.219+0000 7f73f20c9640 1 -- 172.21.15.27:0/664606397 shutdown_connections 2024-03-06T07:28:11.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.219+0000 7f73f20c9640 1 -- 172.21.15.27:0/664606397 wait complete. 2024-03-06T07:28:11.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.220+0000 7f73f20c9640 1 Processor -- start 2024-03-06T07:28:11.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.221+0000 7f73f20c9640 1 -- start start 2024-03-06T07:28:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.221+0000 7f73f20c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ec0d2a30 0x7f73ec1f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.221+0000 7f73f20c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0d4c40 0x7f73ec0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.221+0000 7f73f20c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73f20c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec15e4e0 con 0x7f73ec0d2a30 2024-03-06T07:28:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73f20c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec0cf510 con 0x7f73ec0d4c40 2024-03-06T07:28:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73f20c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec15e660 con 0x7f73ec15e100 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ec0d2a30 0x7f73ec1f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ec0d2a30 0x7f73ec1f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53650/0 (socket says 172.21.15.27:53650) 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eb7fe640 1 -- 172.21.15.27:0/390867671 learned_addr learned my addr 172.21.15.27:0/390867671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eaffd640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0d4c40 0x7f73ec0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73ebfff640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eb7fe640 1 -- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 msgr2=0x7f73ec0b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eb7fe640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eb7fe640 1 -- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0d4c40 msgr2=0x7f73ec0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eb7fe640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0d4c40 0x7f73ec0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eb7fe640 1 -- 172.21.15.27:0/390867671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec09e840 con 0x7f73ec0d2a30 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73ebfff640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eaffd640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0d4c40 0x7f73ec0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73eb7fe640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ec0d2a30 0x7f73ec1f4190 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0097d0 tx=0x7f73dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73d3fff640 1 -- 172.21.15.27:0/390867671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc040070 con 0x7f73ec0d2a30 2024-03-06T07:28:11.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73d3fff640 1 -- 172.21.15.27:0/390867671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc030ca0 con 0x7f73ec0d2a30 2024-03-06T07:28:11.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73d3fff640 1 -- 172.21.15.27:0/390867671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc035040 con 0x7f73ec0d2a30 2024-03-06T07:28:11.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.222+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ec09eb20 con 0x7f73ec0d2a30 2024-03-06T07:28:11.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.223+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ec09f0e0 con 0x7f73ec0d2a30 2024-03-06T07:28:11.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.224+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ec0752d0 con 0x7f73ec0d2a30 2024-03-06T07:28:11.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.225+0000 7f73d3fff640 1 -- 172.21.15.27:0/390867671 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73dc038020 con 0x7f73ec0d2a30 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.225+0000 7f73d3fff640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73d806b4d0 0x7f73d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.225+0000 7f73d3fff640 1 -- 172.21.15.27:0/390867671 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f73dc0ac1b0 con 0x7f73ec0d2a30 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.226+0000 7f73eaffd640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73d806b4d0 0x7f73d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.226+0000 7f73eaffd640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73d806b4d0 0x7f73d806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0b3760 tx=0x7f73e0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.234+0000 7f73d3fff640 1 -- 172.21.15.27:0/390867671 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73dc02d040 con 0x7f73ec0d2a30 2024-03-06T07:28:11.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.597+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73ec06cc30 con 0x7f73ec0d2a30 2024-03-06T07:28:11.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.597+0000 7f73d3fff640 1 -- 172.21.15.27:0/390867671 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f73dc07af50 con 0x7f73ec0d2a30 2024-03-06T07:28:11.600 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:11.600 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:11.600 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73d806b4d0 msgr2=0x7f73d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73d806b4d0 0x7f73d806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0b3760 tx=0x7f73e0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ec0d2a30 msgr2=0x7f73ec1f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ec0d2a30 0x7f73ec1f4190 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0097d0 tx=0x7f73dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 shutdown_connections 2024-03-06T07:28:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f73d806b4d0 0x7f73d806d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73ec15e100 0x7f73ec0b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73ec0d4c40 0x7f73ec0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 --2- 172.21.15.27:0/390867671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ec0d2a30 0x7f73ec1f4190 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 >> 172.21.15.27:0/390867671 conn(0x7f73ec0d3ef0 msgr2=0x7f73ec0b95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 shutdown_connections 2024-03-06T07:28:11.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:11.600+0000 7f73f20c9640 1 -- 172.21.15.27:0/390867671 wait complete. 2024-03-06T07:28:11.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:28:11.768 INFO:teuthology.orchestra.run.smithi027.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":25,"msgr_send_messages":25,"msgr_recv_bytes":15583,"msgr_send_bytes":4223824,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.015841204,"msgr_running_send_time":0.006846405,"msgr_running_recv_time":0.005623995,"msgr_running_fast_dispatch_time":0.002189000,"msgr_send_messages_queue_lat":{"avgcount":24,"sum":0.009567408,"avgtime":0.000398642},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000206,"avgtime":0.000000103},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6247,"msgr_send_messages":12525,"msgr_recv_bytes":2199251,"msgr_send_bytes":15165847,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.651071965,"msgr_running_send_time":0.350746066,"msgr_running_recv_time":0.349748525,"msgr_running_fast_dispatch_time":0.005166302,"msgr_send_messages_queue_lat":{"avgcount":12524,"sum":0.179229319,"avgtime":0.000014310},"msgr_handle_ack_lat":{"avgcount":18418,"sum":0.003523350,"avgtime":0.000000191},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":92224},"AsyncMessenger::Worker-2":{"msgr_recv_messages":27,"msgr_send_messages":24,"msgr_recv_bytes":108770,"msgr_send_bytes":8944,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.004182769,"msgr_running_send_time":0.000853051,"msgr_running_recv_time":0.005083538,"msgr_running_fast_dispatch_time":0.000301858,"msgr_send_messages_queue_lat":{"avgcount":23,"sum":0.004089936,"avgtime":0.000177823},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":104992,"msgr_send_encrypted_bytes":6032},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1703,"complete_latency":{"avgcount":80,"sum":1.471176471,"avgtime":0.018389705}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000088667,"avgtime":0.000022166}},"mds":{"request":4089,"reply":4087,"reply_latency":{"avgcount":4087,"sum":4.913434816,"avgtime":0.001202210},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4088,"inodes_top":0,"inodes_bottom":4088,"inodes_pin_tail":0,"inodes_pinned":4088,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":4095,"traverse_hit":4094,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3199,"root_rbytes":5463040,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":492,"ceph_cap_op_grant":3401,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2105,"handle_client_caps_dirty":1972,"handle_client_cap_release":0,"process_request_cap_release":4957},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":878,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":878,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":6190,"evex":0,"evtrm":0,"ev":6194,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":17376973,"rdpos":4195685,"jlat":{"avgcount":13,"sum":0.075291314,"avgtime":0.005791639},"replayed":4},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4088,"dn+":4088,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":106568,"heap":248164},"mds_server":{"dispatch_client_request":4092,"dispatch_server_request":0,"handle_client_request":4089,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.174951590,"avgtime":0.000992172},"req_getattr_latency":{"avgcount":6,"sum":0.009656876,"avgtime":0.001609479},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002708406,"avgtime":0.002708406},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.230416875,"avgtime":0.076805625},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":877,"sum":1.495701069,"avgtime":0.001705474},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3859,"average_load":3859,"avg_session_uptime":817,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":9578777,"buffer_anon_items":2987,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":9905912,"mds_co_items":47849,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":69,"op_send_bytes":12862870,"op_resend":0,"op_reply":69,"op_latency":{"avgcount":69,"sum":0.145996350,"avgtime":0.002115889},"op_inflight":0,"oplen_avg":{"avgcount":69,"sum":129},"op":69,"op_r":36,"op_w":33,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":13,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":20,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":296,"max":104857600,"get_started":0,"get":6295,"get_sum":1754529,"get_or_fail_fail":0,"get_or_fail_success":6295,"take":0,"take_sum":0,"put":6294,"put_sum":1754233,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":69,"take_sum":12864802,"put":69,"put_sum":12864802,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":69,"take_sum":69,"put":69,"put_sum":69,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a619c9a820":{"val":599757,"max":3758096384,"get_started":0,"get":6190,"get_sum":13180473,"get_or_fail_fail":0,"get_or_fail_success":6190,"take":0,"take_sum":0,"put":13,"put_sum":12580716,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:28:11.769 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.03e-07, "sum": 2.06e-07 }, "msgr_recv_bytes": 15583, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 25, "msgr_running_fast_dispatch_time": 0.002189, "msgr_running_recv_time": 0.005623995, "msgr_running_send_time": 0.006846405, "msgr_running_total_time": 0.015841204, "msgr_send_bytes": 4223824, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 25, "msgr_send_messages_queue_lat": { "avgcount": 24, "avgtime": 0.000398642, "sum": 0.009567408 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 18418, "avgtime": 1.91e-07, "sum": 0.00352335 }, "msgr_recv_bytes": 2199251, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 6247, "msgr_running_fast_dispatch_time": 0.005166302, "msgr_running_recv_time": 0.349748525, "msgr_running_send_time": 0.350746066, "msgr_running_total_time": 0.651071965, "msgr_send_bytes": 15165847, "msgr_send_encrypted_bytes": 92224, "msgr_send_messages": 12525, "msgr_send_messages_queue_lat": { "avgcount": 12524, "avgtime": 1.431e-05, "sum": 0.179229319 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 108770, "msgr_recv_encrypted_bytes": 104992, "msgr_recv_messages": 27, "msgr_running_fast_dispatch_time": 0.000301858, "msgr_running_recv_time": 0.005083538, "msgr_running_send_time": 0.000853051, "msgr_running_total_time": 0.004182769, "msgr_send_bytes": 8944, "msgr_send_encrypted_bytes": 6032, "msgr_send_messages": 24, "msgr_send_messages_queue_lat": { "avgcount": 23, "avgtime": 0.000177823, "sum": 0.004089936 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 80, "avgtime": 0.018389705, "sum": 1.471176471 }, "queue_len": 1703 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.2166e-05, "sum": 8.8667e-05 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3401, "ceph_cap_op_revoke": 492, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2105, "handle_client_caps_dirty": 1972, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4088, "inodes_bottom": 4088, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4088, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 4957, "q": 0, "reply": 4087, "reply_latency": { "avgcount": 4087, "avgtime": 0.00120221, "sum": 4.913434816 }, "request": 4089, "root_rbytes": 5463040, "root_rfiles": 3199, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4095, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4094, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 878, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 878, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6194, "evadd": 6190, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 13, "avgtime": 0.005791639, "sum": 0.075291314 }, "rdpos": 4195685, "replayed": 4, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 17376973 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4088, "dn+": 4088, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 106568 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4092, "dispatch_server_request": 0, "handle_client_request": 4089, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000992172, "sum": 3.17495159 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.001609479, "sum": 0.009656876 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002708406, "sum": 0.002708406 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.076805625, "sum": 0.230416875 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 877, "avgtime": 0.001705474, "sum": 1.495701069 } }, "mds_sessions": { "average_load": 3859, "avg_session_uptime": 817, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3859 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 9578777, "buffer_anon_items": 2987, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 9905912, "mds_co_items": 47849, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 20, "op": 69, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 69, "avgtime": 0.002115889, "sum": 0.14599635 }, "op_pg": 0, "op_r": 36, "op_reply": 69, "op_resend": 0, "op_rmw": 0, "op_send": 69, "op_send_bytes": 12862870, "op_w": 33, "oplen_avg": { "avgcount": 69, "sum": 129 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 13, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6295, "get_or_fail_fail": 0, "get_or_fail_success": 6295, "get_started": 0, "get_sum": 1754529, "max": 104857600, "put": 6294, "put_sum": 1754233, "take": 0, "take_sum": 0, "val": 296, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 69, "put_sum": 12864802, "take": 69, "take_sum": 12864802, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 69, "put_sum": 69, "take": 69, "take_sum": 69, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a619c9a820": { "get": 6190, "get_or_fail_fail": 0, "get_or_fail_success": 6190, "get_started": 0, "get_sum": 13180473, "max": 3758096384, "put": 13, "put_sum": 12580716, "take": 0, "take_sum": 0, "val": 599757, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:28:11.769 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/878, 0/878 2024-03-06T07:28:12.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:12.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.913+0000 7f0ec5f13640 1 Processor -- start 2024-03-06T07:28:12.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.914+0000 7f0ec5f13640 1 -- start start 2024-03-06T07:28:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.914+0000 7f0ec5f13640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ec00d2a30 0x7f0ec00d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.914+0000 7f0ec5f13640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec00d0100 con 0x7f0ec015e100 2024-03-06T07:28:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.914+0000 7f0ec5f13640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec00cf860 con 0x7f0ec00d4c40 2024-03-06T07:28:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.914+0000 7f0ec5f13640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec00cfb40 con 0x7f0ec00d2a30 2024-03-06T07:28:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.915+0000 7f0ebf7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0ec015e100 0x7f0ec00d2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47478/0 (socket says 172.21.15.27:47478) 2024-03-06T07:28:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.915+0000 7f0ebf7fe640 1 -- 172.21.15.27:0/3267988915 learned_addr learned my addr 172.21.15.27:0/3267988915 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.915+0000 7f0ebffff640 1 --2- 172.21.15.27:0/3267988915 >> v2:172.21.15.27:3300/0 conn(0x7f0ec00d2a30 0x7f0ec00d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.915+0000 7f0ebe7fc640 1 -- 172.21.15.27:0/3267988915 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904752697 0 0) 0x7f0ec00d0100 con 0x7f0ec015e100 2024-03-06T07:28:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.915+0000 7f0ebe7fc640 1 -- 172.21.15.27:0/3267988915 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea8003520 con 0x7f0ec015e100 2024-03-06T07:28:12.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.915+0000 7f0ebffff640 1 -- 172.21.15.27:0/3267988915 >> v1:172.21.15.69:6789/0 conn(0x7f0ec00d4c40 legacy=0x7f0ec00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.915+0000 7f0ebffff640 1 -- 172.21.15.27:0/3267988915 >> v1:172.21.15.27:6789/0 conn(0x7f0ec015e100 legacy=0x7f0ec00d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.915+0000 7f0ebffff640 1 -- 172.21.15.27:0/3267988915 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00cfe20 con 0x7f0ec00d2a30 2024-03-06T07:28:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.916+0000 7f0ebffff640 1 --2- 172.21.15.27:0/3267988915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec00d2a30 0x7f0ec00d1cd0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009d90 tx=0x7f0eb802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785853f77c81afbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.916+0000 7f0ebe7fc640 1 -- 172.21.15.27:0/3267988915 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8039070 con 0x7f0ec00d2a30 2024-03-06T07:28:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.916+0000 7f0ebe7fc640 1 -- 172.21.15.27:0/3267988915 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb802cd00 con 0x7f0ec00d2a30 2024-03-06T07:28:12.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.916+0000 7f0ebe7fc640 1 -- 172.21.15.27:0/3267988915 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8034640 con 0x7f0ec00d2a30 2024-03-06T07:28:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.917+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3267988915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec00d2a30 msgr2=0x7f0ec00d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.917+0000 7f0ec5f13640 1 --2- 172.21.15.27:0/3267988915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec00d2a30 0x7f0ec00d1cd0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009d90 tx=0x7f0eb802bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.917+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3267988915 shutdown_connections 2024-03-06T07:28:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.917+0000 7f0ec5f13640 1 --2- 172.21.15.27:0/3267988915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec00d2a30 0x7f0ec00d1cd0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.917+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3267988915 >> 172.21.15.27:0/3267988915 conn(0x7f0ec00d3ef0 msgr2=0x7f0ec00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.917+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3267988915 shutdown_connections 2024-03-06T07:28:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.917+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3267988915 wait complete. 2024-03-06T07:28:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.918+0000 7f0ec5f13640 1 Processor -- start 2024-03-06T07:28:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.918+0000 7f0ec5f13640 1 -- start start 2024-03-06T07:28:12.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ec5f13640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec00d2a30 0x7f0ec0153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ec5f13640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec00d4c40 0x7f0ec01550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ec5f13640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec015e100 0x7f0ec01555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ec5f13640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec015e4e0 con 0x7f0ec015e100 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ec5f13640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec00cf510 con 0x7f0ec00d4c40 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ec5f13640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec015e660 con 0x7f0ec00d2a30 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec015e100 0x7f0ec01555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec015e100 0x7f0ec01555f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53672/0 (socket says 172.21.15.27:53672) 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebffff640 1 -- 172.21.15.27:0/3465751930 learned_addr learned my addr 172.21.15.27:0/3465751930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebf7fe640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec00d2a30 0x7f0ec0153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebeffd640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec00d4c40 0x7f0ec01550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebffff640 1 -- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec00d2a30 msgr2=0x7f0ec0153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebffff640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec00d2a30 0x7f0ec0153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebffff640 1 -- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec00d4c40 msgr2=0x7f0ec01550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebffff640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec00d4c40 0x7f0ec01550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebffff640 1 -- 172.21.15.27:0/3465751930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec0154360 con 0x7f0ec015e100 2024-03-06T07:28:12.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ebffff640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec015e100 0x7f0ec01555f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009d90 tx=0x7f0eb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.920+0000 7f0ea7fff640 1 -- 172.21.15.27:0/3465751930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8044020 con 0x7f0ec015e100 2024-03-06T07:28:12.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.921+0000 7f0ea7fff640 1 -- 172.21.15.27:0/3465751930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb803a040 con 0x7f0ec015e100 2024-03-06T07:28:12.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.921+0000 7f0ea7fff640 1 -- 172.21.15.27:0/3465751930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8039050 con 0x7f0ec015e100 2024-03-06T07:28:12.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.921+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec01539c0 con 0x7f0ec015e100 2024-03-06T07:28:12.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.921+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec01545d0 con 0x7f0ec015e100 2024-03-06T07:28:12.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.922+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec00752d0 con 0x7f0ec015e100 2024-03-06T07:28:12.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.922+0000 7f0ea7fff640 1 -- 172.21.15.27:0/3465751930 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0eb8003bd0 con 0x7f0ec015e100 2024-03-06T07:28:12.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.923+0000 7f0ea7fff640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0eac06b4d0 0x7f0eac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.923+0000 7f0ea7fff640 1 -- 172.21.15.27:0/3465751930 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0eb80ac8a0 con 0x7f0ec015e100 2024-03-06T07:28:12.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.923+0000 7f0ebf7fe640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0eac06b4d0 0x7f0eac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.924+0000 7f0ebf7fe640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0eac06b4d0 0x7f0eac06d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00011e0 tx=0x7f0eb0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:12.929+0000 7f0ea7fff640 1 -- 172.21.15.27:0/3465751930 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0eb804e060 con 0x7f0ec015e100 2024-03-06T07:28:13.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.265+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ec015d690 con 0x7f0ec015e100 2024-03-06T07:28:13.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.266+0000 7f0ea7fff640 1 -- 172.21.15.27:0/3465751930 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f0ec01539c0 con 0x7f0ec015e100 2024-03-06T07:28:13.268 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:13.269 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:13.269 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:13.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0eac06b4d0 msgr2=0x7f0eac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0eac06b4d0 0x7f0eac06d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00011e0 tx=0x7f0eb0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec015e100 msgr2=0x7f0ec01555f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec015e100 0x7f0ec01555f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009d90 tx=0x7f0eb8004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 shutdown_connections 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0eac06b4d0 0x7f0eac06d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec015e100 0x7f0ec01555f0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ec00d4c40 0x7f0ec01550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 --2- 172.21.15.27:0/3465751930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0ec00d2a30 0x7f0ec0153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.269+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 >> 172.21.15.27:0/3465751930 conn(0x7f0ec00d3ef0 msgr2=0x7f0ec015f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.270+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 shutdown_connections 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:13.270+0000 7f0ec5f13640 1 -- 172.21.15.27:0/3465751930 wait complete. 2024-03-06T07:28:13.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:28:13.445 INFO:teuthology.orchestra.run.smithi027.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":40,"msgr_send_messages":40,"msgr_recv_bytes":19385,"msgr_send_bytes":4228706,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.016801885,"msgr_running_send_time":0.007208702,"msgr_running_recv_time":0.005847085,"msgr_running_fast_dispatch_time":0.002498072,"msgr_send_messages_queue_lat":{"avgcount":39,"sum":0.010081221,"avgtime":0.000258492},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000206,"avgtime":0.000000103},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7503,"msgr_send_messages":15864,"msgr_recv_bytes":2658497,"msgr_send_bytes":19196119,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.788083166,"msgr_running_send_time":0.432534101,"msgr_running_recv_time":0.407739036,"msgr_running_fast_dispatch_time":0.007363709,"msgr_send_messages_queue_lat":{"avgcount":15863,"sum":0.260498570,"avgtime":0.000016421},"msgr_handle_ack_lat":{"avgcount":22162,"sum":0.004303116,"avgtime":0.000000194},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":92224},"AsyncMessenger::Worker-2":{"msgr_recv_messages":38,"msgr_send_messages":35,"msgr_recv_bytes":113663,"msgr_send_bytes":15089,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.004931152,"msgr_running_send_time":0.001121639,"msgr_running_recv_time":0.005251349,"msgr_running_fast_dispatch_time":0.000564092,"msgr_send_messages_queue_lat":{"avgcount":34,"sum":0.004366921,"avgtime":0.000128438},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":105264,"msgr_send_encrypted_bytes":6656},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1192,"complete_latency":{"avgcount":82,"sum":3.191116371,"avgtime":0.038916053}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":49,"sum":0.019582814,"avgtime":0.000399649}},"mds":{"request":4552,"reply":4551,"reply_latency":{"avgcount":4551,"sum":6.265866533,"avgtime":0.001376810},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4551,"inodes_top":0,"inodes_bottom":4551,"inodes_pin_tail":0,"inodes_pinned":4551,"inodes_expired":0,"inodes_with_caps":3049,"caps":3049,"subtrees":2,"traverse":4558,"traverse_hit":4557,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2335,"root_rbytes":9521152,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":633,"ceph_cap_op_grant":4572,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2887,"handle_client_caps_dirty":2701,"handle_client_cap_release":2,"process_request_cap_release":5883},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1320,"num_strays_delayed":132,"num_strays_enqueuing":0,"strays_created":1341,"strays_enqueued":21,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7458,"evex":0,"evtrm":0,"ev":7462,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":20386947,"rdpos":4195685,"jlat":{"avgcount":14,"sum":0.211875116,"avgtime":0.015133936},"replayed":4},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4552,"dn+":4552,"dn-":0,"cap":3049,"cap+":3202,"cap-":153,"rss":106568,"heap":248164},"mds_server":{"dispatch_client_request":4555,"dispatch_server_request":0,"handle_client_request":4552,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.174951590,"avgtime":0.000992172},"req_getattr_latency":{"avgcount":6,"sum":0.009656876,"avgtime":0.001609479},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002708406,"avgtime":0.002708406},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.230416875,"avgtime":0.076805625},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1341,"sum":2.848132786,"avgtime":0.002123887},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4245,"average_load":4245,"avg_session_uptime":822,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18867153,"buffer_anon_items":5473,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14609957,"mds_co_items":126722,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":101,"op_send_bytes":15803007,"op_resend":0,"op_reply":101,"op_latency":{"avgcount":101,"sum":0.305992350,"avgtime":0.003029627},"op_inflight":0,"oplen_avg":{"avgcount":101,"sum":161},"op":101,"op_r":37,"op_w":64,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":9,"osdop_write":15,"osdop_writefull":3,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":35,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":20,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":21,"pq_executed":21,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":298,"max":104857600,"get_started":0,"get":7578,"get_sum":2107336,"get_or_fail_fail":0,"get_or_fail_success":7578,"take":0,"take_sum":0,"put":7577,"put_sum":2107038,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":101,"take_sum":15807060,"put":101,"put_sum":15807060,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":101,"take_sum":101,"put":101,"put_sum":101,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":21,"get_sum":2121,"get_or_fail_fail":0,"get_or_fail_success":21,"take":0,"take_sum":0,"put":1,"put_sum":2121,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a619c9a820":{"val":671895,"max":3758096384,"get_started":0,"get":7458,"get_sum":16190447,"get_or_fail_fail":0,"get_or_fail_success":7458,"take":0,"take_sum":0,"put":14,"put_sum":15518552,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:28:13.446 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.03e-07, "sum": 2.06e-07 }, "msgr_recv_bytes": 19385, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 40, "msgr_running_fast_dispatch_time": 0.002498072, "msgr_running_recv_time": 0.005847085, "msgr_running_send_time": 0.007208702, "msgr_running_total_time": 0.016801885, "msgr_send_bytes": 4228706, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 40, "msgr_send_messages_queue_lat": { "avgcount": 39, "avgtime": 0.000258492, "sum": 0.010081221 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 22162, "avgtime": 1.94e-07, "sum": 0.004303116 }, "msgr_recv_bytes": 2658497, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 7503, "msgr_running_fast_dispatch_time": 0.007363709, "msgr_running_recv_time": 0.407739036, "msgr_running_send_time": 0.432534101, "msgr_running_total_time": 0.788083166, "msgr_send_bytes": 19196119, "msgr_send_encrypted_bytes": 92224, "msgr_send_messages": 15864, "msgr_send_messages_queue_lat": { "avgcount": 15863, "avgtime": 1.6421e-05, "sum": 0.26049857 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 113663, "msgr_recv_encrypted_bytes": 105264, "msgr_recv_messages": 38, "msgr_running_fast_dispatch_time": 0.000564092, "msgr_running_recv_time": 0.005251349, "msgr_running_send_time": 0.001121639, "msgr_running_total_time": 0.004931152, "msgr_send_bytes": 15089, "msgr_send_encrypted_bytes": 6656, "msgr_send_messages": 35, "msgr_send_messages_queue_lat": { "avgcount": 34, "avgtime": 0.000128438, "sum": 0.004366921 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 82, "avgtime": 0.038916053, "sum": 3.191116371 }, "queue_len": 1192 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 49, "avgtime": 0.000399649, "sum": 0.019582814 }, "queue_len": 0 }, "mds": { "caps": 3049, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4572, "ceph_cap_op_revoke": 633, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 2, "handle_client_caps": 2887, "handle_client_caps_dirty": 2701, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4551, "inodes_bottom": 4551, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4551, "inodes_top": 0, "inodes_with_caps": 3049, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 5883, "q": 0, "reply": 4551, "reply_latency": { "avgcount": 4551, "avgtime": 0.00137681, "sum": 6.265866533 }, "request": 4552, "root_rbytes": 9521152, "root_rfiles": 2335, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4558, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4557, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1320, "num_strays_delayed": 132, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1341, "strays_enqueued": 21, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7462, "evadd": 7458, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 14, "avgtime": 0.015133936, "sum": 0.211875116 }, "rdpos": 4195685, "replayed": 4, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 20386947 }, "mds_mem": { "cap": 3049, "cap+": 3202, "cap-": 153, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4552, "dn+": 4552, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 106568 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4555, "dispatch_server_request": 0, "handle_client_request": 4552, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000992172, "sum": 3.17495159 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.001609479, "sum": 0.009656876 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002708406, "sum": 0.002708406 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.076805625, "sum": 0.230416875 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1341, "avgtime": 0.002123887, "sum": 2.848132786 } }, "mds_sessions": { "average_load": 4245, "avg_session_uptime": 822, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4245 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18867153, "buffer_anon_items": 5473, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 14609957, "mds_co_items": 126722, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 20, "op": 101, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 101, "avgtime": 0.003029627, "sum": 0.30599235 }, "op_pg": 0, "op_r": 37, "op_reply": 101, "op_resend": 0, "op_rmw": 0, "op_send": 101, "op_send_bytes": 15803007, "op_w": 64, "oplen_avg": { "avgcount": 101, "sum": 161 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 35, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 9, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 15, "osdop_writefull": 3, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 21, "pq_executed_ops": 21, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7578, "get_or_fail_fail": 0, "get_or_fail_success": 7578, "get_started": 0, "get_sum": 2107336, "max": 104857600, "put": 7577, "put_sum": 2107038, "take": 0, "take_sum": 0, "val": 298, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 101, "put_sum": 15807060, "take": 101, "take_sum": 15807060, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 101, "put_sum": 101, "take": 101, "take_sum": 101, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 21, "get_or_fail_fail": 0, "get_or_fail_success": 21, "get_started": 0, "get_sum": 2121, "max": 3758096384, "put": 1, "put_sum": 2121, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a619c9a820": { "get": 7458, "get_or_fail_fail": 0, "get_or_fail_success": 7458, "get_started": 0, "get_sum": 16190447, "max": 3758096384, "put": 14, "put_sum": 15518552, "take": 0, "take_sum": 0, "val": 671895, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:28:13.446 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1320, 21/1341 2024-03-06T07:28:14.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.582+0000 7fad7eaaf640 1 Processor -- start 2024-03-06T07:28:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.582+0000 7fad7eaaf640 1 -- start start 2024-03-06T07:28:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.583+0000 7fad7eaaf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fad7815e100 0x7fad780d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.583+0000 7fad7eaaf640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780d0100 con 0x7fad780d4c40 2024-03-06T07:28:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.583+0000 7fad7eaaf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780cf860 con 0x7fad780d2a30 2024-03-06T07:28:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.583+0000 7fad7eaaf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad780cfb40 con 0x7fad7815e100 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.583+0000 7fad77fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fad780d4c40 0x7fad780d2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57032/0 (socket says 172.21.15.27:57032) 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.583+0000 7fad77fff640 1 -- 172.21.15.27:0/2103321277 learned_addr learned my addr 172.21.15.27:0/2103321277 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad7c84b640 1 --2- 172.21.15.27:0/2103321277 >> v2:172.21.15.69:3300/0 conn(0x7fad7815e100 0x7fad780d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410799033 0 0) 0x7fad780d0100 con 0x7fad780d4c40 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad68003520 con 0x7fad780d4c40 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547356978 0 0) 0x7fad780cf860 con 0x7fad780d2a30 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad780d0100 con 0x7fad780d2a30 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1350428182 0 0) 0x7fad68003520 con 0x7fad780d4c40 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 >> v2:172.21.15.69:3300/0 conn(0x7fad7815e100 msgr2=0x7fad780d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 --2- 172.21.15.27:0/2103321277 >> v2:172.21.15.69:3300/0 conn(0x7fad7815e100 0x7fad780d1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 >> v1:172.21.15.69:6789/0 conn(0x7fad780d2a30 legacy=0x7fad780b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad780cfe20 con 0x7fad780d4c40 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fad64001670 con 0x7fad780d4c40 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 >> v1:172.21.15.27:6789/0 conn(0x7fad780d4c40 legacy=0x7fad780d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad68003b40 0x7fad68003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad68004800 0x7fad68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad680051c0 0x7fad6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad780cfb40 con 0x7fad680051c0 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6800c040 con 0x7fad68004800 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6800c240 con 0x7fad68003b40 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad7d04c640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad680051c0 0x7fad6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad77fff640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad68004800 0x7fad68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.584+0000 7fad7c84b640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad68003b40 0x7fad68003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.585+0000 7fad7d04c640 1 -- 172.21.15.27:0/2103321277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad68003b40 msgr2=0x7fad68003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.585+0000 7fad7d04c640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad68003b40 0x7fad68003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.585+0000 7fad7d04c640 1 -- 172.21.15.27:0/2103321277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad68004800 msgr2=0x7fad68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.585+0000 7fad7d04c640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad68004800 0x7fad68004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.585+0000 7fad7d04c640 1 -- 172.21.15.27:0/2103321277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad68003520 con 0x7fad680051c0 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.585+0000 7fad7d04c640 1 -- 172.21.15.27:0/2103321277 reap_dead start 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.586+0000 7fad7d04c640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad680051c0 0x7fad6800b9c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fad70001fd0 tx=0x7fad68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a33307d049dc0b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.586+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad70001bf0 con 0x7fad680051c0 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.586+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad70003b70 con 0x7fad680051c0 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.586+0000 7fad777fe640 1 -- 172.21.15.27:0/2103321277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad700050d0 con 0x7fad680051c0 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.586+0000 7fad7eaaf640 1 -- 172.21.15.27:0/2103321277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad680051c0 msgr2=0x7fad6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.586+0000 7fad7eaaf640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad680051c0 0x7fad6800b9c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fad70001fd0 tx=0x7fad68005830 comp rx=0 tx=0).stop 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.586+0000 7fad7eaaf640 1 -- 172.21.15.27:0/2103321277 shutdown_connections 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.586+0000 7fad7eaaf640 1 --2- 172.21.15.27:0/2103321277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad680051c0 0x7fad6800b9c0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.586+0000 7fad7eaaf640 1 -- 172.21.15.27:0/2103321277 >> 172.21.15.27:0/2103321277 conn(0x7fad780d3ef0 msgr2=0x7fad780d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.587+0000 7fad7eaaf640 1 -- 172.21.15.27:0/2103321277 shutdown_connections 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.587+0000 7fad7eaaf640 1 -- 172.21.15.27:0/2103321277 wait complete. 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.588+0000 7fad7eaaf640 1 Processor -- start 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.588+0000 7fad7eaaf640 1 -- start start 2024-03-06T07:28:14.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.589+0000 7fad7eaaf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad680051c0 0x7fad78157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.589+0000 7fad7eaaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad780cd780 0x7fad780cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.589+0000 7fad7eaaf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad78159710 0x7fad78159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.589+0000 7fad7eaaf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7815e4e0 con 0x7fad780cd780 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.589+0000 7fad7eaaf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad780cf510 con 0x7fad78159710 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.589+0000 7fad7eaaf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7815e660 con 0x7fad680051c0 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.589+0000 7fad77fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad780cd780 0x7fad780cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.589+0000 7fad77fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad780cd780 0x7fad780cdbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47724/0 (socket says 172.21.15.27:47724) 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.589+0000 7fad77fff640 1 -- 172.21.15.27:0/3454827072 learned_addr learned my addr 172.21.15.27:0/3454827072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad7c84b640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad680051c0 0x7fad78157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad7d04c640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad78159710 0x7fad78159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad77fff640 1 -- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad680051c0 msgr2=0x7fad78157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad77fff640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad680051c0 0x7fad78157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad77fff640 1 -- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad78159710 msgr2=0x7fad78159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad77fff640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad78159710 0x7fad78159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad77fff640 1 -- 172.21.15.27:0/3454827072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7815a420 con 0x7fad780cd780 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad7d04c640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad78159710 0x7fad78159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad77fff640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad780cd780 0x7fad780cdbe0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fad64007310 tx=0x7fad64007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad757fa640 1 -- 172.21.15.27:0/3454827072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64001650 con 0x7fad780cd780 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad757fa640 1 -- 172.21.15.27:0/3454827072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad64018070 con 0x7fad780cd780 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad78157da0 con 0x7fad780cd780 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad757fa640 1 -- 172.21.15.27:0/3454827072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6400fba0 con 0x7fad780cd780 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.590+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad78158360 con 0x7fad780cd780 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.592+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad780752d0 con 0x7fad780cd780 2024-03-06T07:28:14.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.592+0000 7fad757fa640 1 -- 172.21.15.27:0/3454827072 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad6400fd40 con 0x7fad780cd780 2024-03-06T07:28:14.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.594+0000 7fad757fa640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fad5c06b4d0 0x7fad5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.594+0000 7fad757fa640 1 -- 172.21.15.27:0/3454827072 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fad6408a1c0 con 0x7fad780cd780 2024-03-06T07:28:14.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.594+0000 7fad7c84b640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fad5c06b4d0 0x7fad5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.594+0000 7fad7c84b640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fad5c06b4d0 0x7fad5c06d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009b40 tx=0x7fad6c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.601+0000 7fad757fa640 1 -- 172.21.15.27:0/3454827072 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad7815a420 con 0x7fad780cd780 2024-03-06T07:28:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.990+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad780b68b0 con 0x7fad780cd780 2024-03-06T07:28:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.991+0000 7fad757fa640 1 -- 172.21.15.27:0/3454827072 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fad64058fd0 con 0x7fad780cd780 2024-03-06T07:28:14.993 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:14.993 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.993+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fad5c06b4d0 msgr2=0x7fad5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.993+0000 7fad7eaaf640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fad5c06b4d0 0x7fad5c06d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009b40 tx=0x7fad6c009190 comp rx=0 tx=0).stop 2024-03-06T07:28:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.993+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad780cd780 msgr2=0x7fad780cdbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.993+0000 7fad7eaaf640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad780cd780 0x7fad780cdbe0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fad64007310 tx=0x7fad64007710 comp rx=0 tx=0).stop 2024-03-06T07:28:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.994+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 shutdown_connections 2024-03-06T07:28:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.994+0000 7fad7eaaf640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fad5c06b4d0 0x7fad5c06d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.994+0000 7fad7eaaf640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad78159710 0x7fad78159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.994+0000 7fad7eaaf640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad780cd780 0x7fad780cdbe0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.994+0000 7fad7eaaf640 1 --2- 172.21.15.27:0/3454827072 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fad680051c0 0x7fad78157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.994+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 >> 172.21.15.27:0/3454827072 conn(0x7fad780d3ef0 msgr2=0x7fad7806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.994+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 shutdown_connections 2024-03-06T07:28:14.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:14.994+0000 7fad7eaaf640 1 -- 172.21.15.27:0/3454827072 wait complete. 2024-03-06T07:28:15.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:28:15.155 INFO:teuthology.orchestra.run.smithi027.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":271,"msgr_send_messages":271,"msgr_recv_bytes":78051,"msgr_send_bytes":4304004,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.051530199,"msgr_running_send_time":0.027371852,"msgr_running_recv_time":0.011853883,"msgr_running_fast_dispatch_time":0.009461708,"msgr_send_messages_queue_lat":{"avgcount":270,"sum":0.032028643,"avgtime":0.000118624},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000206,"avgtime":0.000000103},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8693,"msgr_send_messages":18902,"msgr_recv_bytes":3109315,"msgr_send_bytes":21548791,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.934214775,"msgr_running_send_time":0.515791944,"msgr_running_recv_time":0.485798030,"msgr_running_fast_dispatch_time":0.013907044,"msgr_send_messages_queue_lat":{"avgcount":18901,"sum":0.342949245,"avgtime":0.000018144},"msgr_handle_ack_lat":{"avgcount":25412,"sum":0.004957129,"avgtime":0.000000195},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94016},"AsyncMessenger::Worker-2":{"msgr_recv_messages":218,"msgr_send_messages":215,"msgr_recv_bytes":213003,"msgr_send_bytes":127569,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.033432436,"msgr_running_send_time":0.015923907,"msgr_running_recv_time":0.010837737,"msgr_running_fast_dispatch_time":0.007231091,"msgr_send_messages_queue_lat":{"avgcount":214,"sum":0.025892234,"avgtime":0.000120991},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":105264,"msgr_send_encrypted_bytes":6656},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":86,"sum":4.898266675,"avgtime":0.056956589}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":860,"sum":0.475401791,"avgtime":0.000552792}},"mds":{"request":5282,"reply":5281,"reply_latency":{"avgcount":5281,"sum":7.492926367,"avgtime":0.001418846},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5002,"inodes_top":0,"inodes_bottom":5002,"inodes_pin_tail":0,"inodes_pinned":5002,"inodes_expired":0,"inodes_with_caps":2649,"caps":2649,"subtrees":2,"traverse":5288,"traverse_hit":5287,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1722,"root_rbytes":10988544,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":683,"ceph_cap_op_grant":5337,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3204,"handle_client_caps_dirty":3000,"handle_client_cap_release":4,"process_request_cap_release":7343},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1518,"num_strays_delayed":474,"num_strays_enqueuing":0,"strays_created":2071,"strays_enqueued":553,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":9039,"evex":0,"evtrm":0,"ev":9043,"evexg":0,"evexd":0,"segadd":11,"segex":0,"segtrm":0,"seg":13,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":23647723,"rdpos":4195685,"jlat":{"avgcount":15,"sum":0.756108004,"avgtime":0.050407200},"replayed":4},"mds_mem":{"ino":3061,"ino+":3214,"ino-":153,"dir":13,"dir+":13,"dir-":0,"dn":5129,"dn+":5282,"dn-":153,"cap":2175,"cap+":3202,"cap-":1027,"rss":106568,"heap":248164},"mds_server":{"dispatch_client_request":5285,"dispatch_server_request":0,"handle_client_request":5283,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.174951590,"avgtime":0.000992172},"req_getattr_latency":{"avgcount":6,"sum":0.009656876,"avgtime":0.001609479},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002708406,"avgtime":0.002708406},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.230416875,"avgtime":0.076805625},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2071,"sum":4.075192620,"avgtime":0.001967741},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4884,"average_load":4884,"avg_session_uptime":822,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18867153,"buffer_anon_items":5473,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14609957,"mds_co_items":126722,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":651,"op_send_bytes":17129749,"op_resend":0,"op_reply":651,"op_latency":{"avgcount":651,"sum":1.338966525,"avgtime":0.002056784},"op_inflight":0,"oplen_avg":{"avgcount":651,"sum":715},"op":651,"op_r":43,"op_w":608,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":15,"osdop_write":22,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":568,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":24,"omap_rd":48,"omap_del":6},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":553,"pq_executed":553,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":297,"max":104857600,"get_started":0,"get":9179,"get_sum":2572069,"get_or_fail_fail":0,"get_or_fail_success":9179,"take":0,"take_sum":0,"put":9178,"put_sum":2571772,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":651,"take_sum":17187534,"put":651,"put_sum":17187534,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":651,"take_sum":651,"put":651,"put_sum":651,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":553,"get_sum":55853,"get_or_fail_fail":0,"get_or_fail_success":553,"take":0,"take_sum":0,"put":7,"put_sum":55853,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a619c9a820":{"val":2676203,"max":3758096384,"get_started":0,"get":9039,"get_sum":19451223,"get_or_fail_fail":0,"get_or_fail_success":9039,"take":0,"take_sum":0,"put":15,"put_sum":16775020,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:28:15.156 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.03e-07, "sum": 2.06e-07 }, "msgr_recv_bytes": 78051, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 271, "msgr_running_fast_dispatch_time": 0.009461708, "msgr_running_recv_time": 0.011853883, "msgr_running_send_time": 0.027371852, "msgr_running_total_time": 0.051530199, "msgr_send_bytes": 4304004, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 271, "msgr_send_messages_queue_lat": { "avgcount": 270, "avgtime": 0.000118624, "sum": 0.032028643 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 25412, "avgtime": 1.95e-07, "sum": 0.004957129 }, "msgr_recv_bytes": 3109315, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 8693, "msgr_running_fast_dispatch_time": 0.013907044, "msgr_running_recv_time": 0.48579803, "msgr_running_send_time": 0.515791944, "msgr_running_total_time": 0.934214775, "msgr_send_bytes": 21548791, "msgr_send_encrypted_bytes": 94016, "msgr_send_messages": 18902, "msgr_send_messages_queue_lat": { "avgcount": 18901, "avgtime": 1.8144e-05, "sum": 0.342949245 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 213003, "msgr_recv_encrypted_bytes": 105264, "msgr_recv_messages": 218, "msgr_running_fast_dispatch_time": 0.007231091, "msgr_running_recv_time": 0.010837737, "msgr_running_send_time": 0.015923907, "msgr_running_total_time": 0.033432436, "msgr_send_bytes": 127569, "msgr_send_encrypted_bytes": 6656, "msgr_send_messages": 215, "msgr_send_messages_queue_lat": { "avgcount": 214, "avgtime": 0.000120991, "sum": 0.025892234 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 86, "avgtime": 0.056956589, "sum": 4.898266675 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 860, "avgtime": 0.000552792, "sum": 0.475401791 }, "queue_len": 0 }, "mds": { "caps": 2649, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5337, "ceph_cap_op_revoke": 683, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 4, "handle_client_caps": 3204, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5002, "inodes_bottom": 5002, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5002, "inodes_top": 0, "inodes_with_caps": 2649, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 7343, "q": 0, "reply": 5281, "reply_latency": { "avgcount": 5281, "avgtime": 0.001418846, "sum": 7.492926367 }, "request": 5282, "root_rbytes": 10988544, "root_rfiles": 1722, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5288, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5287, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1518, "num_strays_delayed": 474, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2071, "strays_enqueued": 553, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9043, "evadd": 9039, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 15, "avgtime": 0.0504072, "sum": 0.756108004 }, "rdpos": 4195685, "replayed": 4, "seg": 13, "segadd": 11, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 23647723 }, "mds_mem": { "cap": 2175, "cap+": 3202, "cap-": 1027, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5129, "dn+": 5282, "dn-": 153, "heap": 248164, "ino": 3061, "ino+": 3214, "ino-": 153, "rss": 106568 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5285, "dispatch_server_request": 0, "handle_client_request": 5283, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000992172, "sum": 3.17495159 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.001609479, "sum": 0.009656876 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002708406, "sum": 0.002708406 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.076805625, "sum": 0.230416875 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2071, "avgtime": 0.001967741, "sum": 4.07519262 } }, "mds_sessions": { "average_load": 4884, "avg_session_uptime": 822, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4884 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18867153, "buffer_anon_items": 5473, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 14609957, "mds_co_items": 126722, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 6, "omap_rd": 48, "omap_wr": 24, "op": 651, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 651, "avgtime": 0.002056784, "sum": 1.338966525 }, "op_pg": 0, "op_r": 43, "op_reply": 651, "op_resend": 0, "op_rmw": 0, "op_send": 651, "op_send_bytes": 17129749, "op_w": 608, "oplen_avg": { "avgcount": 651, "sum": 715 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 568, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 15, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 22, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 553, "pq_executed_ops": 553, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 9179, "get_or_fail_fail": 0, "get_or_fail_success": 9179, "get_started": 0, "get_sum": 2572069, "max": 104857600, "put": 9178, "put_sum": 2571772, "take": 0, "take_sum": 0, "val": 297, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 651, "put_sum": 17187534, "take": 651, "take_sum": 17187534, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 651, "put_sum": 651, "take": 651, "take_sum": 651, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 553, "get_or_fail_fail": 0, "get_or_fail_success": 553, "get_started": 0, "get_sum": 55853, "max": 3758096384, "put": 7, "put_sum": 55853, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a619c9a820": { "get": 9039, "get_or_fail_fail": 0, "get_or_fail_success": 9039, "get_started": 0, "get_sum": 19451223, "max": 3758096384, "put": 15, "put_sum": 16775020, "take": 0, "take_sum": 0, "val": 2676203, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:28:15.156 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1518, 553/2071 2024-03-06T07:28:16.157 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:16.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.298+0000 7f8d8195e640 1 Processor -- start 2024-03-06T07:28:16.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.298+0000 7f8d8195e640 1 -- start start 2024-03-06T07:28:16.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.299+0000 7f8d8195e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d7c0b7710 0x7f8d7c06c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.299+0000 7f8d8195e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d7c0d27c0 0x7f8d7c06cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.299+0000 7f8d8195e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c06c6c0 con 0x7f8d7c0cd0b0 2024-03-06T07:28:16.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.299+0000 7f8d8195e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c0ccc20 con 0x7f8d7c0d27c0 2024-03-06T07:28:16.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.299+0000 7f8d8195e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c0cc2f0 con 0x7f8d7c0b7710 2024-03-06T07:28:16.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.299+0000 7f8d73fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d7c0d27c0 0x7f8d7c06cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.299+0000 7f8d73fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d7c0d27c0 0x7f8d7c06cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47726/0 (socket says 172.21.15.27:47726) 2024-03-06T07:28:16.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.299+0000 7f8d73fff640 1 -- 172.21.15.27:0/575540086 learned_addr learned my addr 172.21.15.27:0/575540086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:16.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.299+0000 7f8d7affd640 1 --2- 172.21.15.27:0/575540086 >> v2:172.21.15.69:3300/0 conn(0x7f8d7c0b7710 0x7f8d7c06c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.300+0000 7f8d7a7fc640 1 -- 172.21.15.27:0/575540086 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199337803 0 0) 0x7f8d7c06c6c0 con 0x7f8d7c0cd0b0 2024-03-06T07:28:16.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.300+0000 7f8d7a7fc640 1 -- 172.21.15.27:0/575540086 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d68003520 con 0x7f8d7c0cd0b0 2024-03-06T07:28:16.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.300+0000 7f8d73fff640 1 -- 172.21.15.27:0/575540086 >> v2:172.21.15.69:3300/0 conn(0x7f8d7c0b7710 msgr2=0x7f8d7c06c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.300+0000 7f8d73fff640 1 --2- 172.21.15.27:0/575540086 >> v2:172.21.15.69:3300/0 conn(0x7f8d7c0b7710 0x7f8d7c06c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.300+0000 7f8d73fff640 1 -- 172.21.15.27:0/575540086 >> v1:172.21.15.69:6789/0 conn(0x7f8d7c0cd0b0 legacy=0x7f8d7c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.300+0000 7f8d73fff640 1 -- 172.21.15.27:0/575540086 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0cc5d0 con 0x7f8d7c0d27c0 2024-03-06T07:28:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.300+0000 7f8d7affd640 1 --2- 172.21.15.27:0/575540086 >> v2:172.21.15.69:3300/0 conn(0x7f8d7c0b7710 0x7f8d7c06c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.301+0000 7f8d73fff640 1 --2- 172.21.15.27:0/575540086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0d27c0 0x7f8d7c06cf00 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0691e0 tx=0x7f8d6402bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805ea8d80bc5a5a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.301+0000 7f8d7a7fc640 1 -- 172.21.15.27:0/575540086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d64039070 con 0x7f8d7c0d27c0 2024-03-06T07:28:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.301+0000 7f8d7a7fc640 1 -- 172.21.15.27:0/575540086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6402c710 con 0x7f8d7c0d27c0 2024-03-06T07:28:16.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.301+0000 7f8d7a7fc640 1 -- 172.21.15.27:0/575540086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6402cae0 con 0x7f8d7c0d27c0 2024-03-06T07:28:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.302+0000 7f8d8195e640 1 -- 172.21.15.27:0/575540086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0d27c0 msgr2=0x7f8d7c06cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.302+0000 7f8d8195e640 1 --2- 172.21.15.27:0/575540086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0d27c0 0x7f8d7c06cf00 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0691e0 tx=0x7f8d6402bb70 comp rx=0 tx=0).stop 2024-03-06T07:28:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.302+0000 7f8d8195e640 1 -- 172.21.15.27:0/575540086 shutdown_connections 2024-03-06T07:28:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.302+0000 7f8d8195e640 1 --2- 172.21.15.27:0/575540086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0d27c0 0x7f8d7c06cf00 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.302+0000 7f8d8195e640 1 --2- 172.21.15.27:0/575540086 >> v2:172.21.15.69:3300/0 conn(0x7f8d7c0b7710 0x7f8d7c06c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.302+0000 7f8d8195e640 1 -- 172.21.15.27:0/575540086 >> 172.21.15.27:0/575540086 conn(0x7f8d7c0d1ca0 msgr2=0x7f8d7c0d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.302+0000 7f8d8195e640 1 -- 172.21.15.27:0/575540086 shutdown_connections 2024-03-06T07:28:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.302+0000 7f8d8195e640 1 -- 172.21.15.27:0/575540086 wait complete. 2024-03-06T07:28:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.304+0000 7f8d8195e640 1 Processor -- start 2024-03-06T07:28:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.304+0000 7f8d8195e640 1 -- start start 2024-03-06T07:28:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d8195e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b7710 0x7f8d7c0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d8195e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d7c0cd0b0 0x7f8d7c06f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d8195e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7c0d27c0 0x7f8d7c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d8195e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c06c370 con 0x7f8d7c0b7710 2024-03-06T07:28:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d8195e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c0b7af0 con 0x7f8d7c0cd0b0 2024-03-06T07:28:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d8195e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c0b4af0 con 0x7f8d7c0d27c0 2024-03-06T07:28:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b7710 0x7f8d7c0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b7710 0x7f8d7c0c5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47742/0 (socket says 172.21.15.27:47742) 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7affd640 1 -- 172.21.15.27:0/3879490617 learned_addr learned my addr 172.21.15.27:0/3879490617 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d73fff640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d7c0cd0b0 0x7f8d7c06f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7b7fe640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7c0d27c0 0x7f8d7c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7affd640 1 -- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7c0d27c0 msgr2=0x7f8d7c0b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7affd640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7c0d27c0 0x7f8d7c0b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7affd640 1 -- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d7c0cd0b0 msgr2=0x7f8d7c06f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7affd640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d7c0cd0b0 0x7f8d7c06f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7affd640 1 -- 172.21.15.27:0/3879490617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0c7490 con 0x7f8d7c0b7710 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d73fff640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d7c0cd0b0 0x7f8d7c06f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7b7fe640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7c0d27c0 0x7f8d7c0b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.305+0000 7f8d7affd640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b7710 0x7f8d7c0c5790 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00ebe0 tx=0x7f8d6c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.306+0000 7f8d78ff9640 1 -- 172.21.15.27:0/3879490617 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c017070 con 0x7f8d7c0b7710 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.306+0000 7f8d78ff9640 1 -- 172.21.15.27:0/3879490617 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c00cc30 con 0x7f8d7c0b7710 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.306+0000 7f8d78ff9640 1 -- 172.21.15.27:0/3879490617 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c013470 con 0x7f8d7c0b7710 2024-03-06T07:28:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.306+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c0c7770 con 0x7f8d7c0b7710 2024-03-06T07:28:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.306+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c0c7d30 con 0x7f8d7c0b7710 2024-03-06T07:28:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.307+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7c0752d0 con 0x7f8d7c0b7710 2024-03-06T07:28:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.308+0000 7f8d78ff9640 1 -- 172.21.15.27:0/3879490617 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d6c01a070 con 0x7f8d7c0b7710 2024-03-06T07:28:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.308+0000 7f8d78ff9640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5c06b520 0x7f8d5c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.308+0000 7f8d78ff9640 1 -- 172.21.15.27:0/3879490617 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8d6c08a4c0 con 0x7f8d7c0b7710 2024-03-06T07:28:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.308+0000 7f8d73fff640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5c06b520 0x7f8d5c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.310+0000 7f8d73fff640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5c06b520 0x7f8d5c06d9e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8d64036000 tx=0x7f8d64030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.314+0000 7f8d78ff9640 1 -- 172.21.15.27:0/3879490617 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d6c059390 con 0x7f8d7c0b7710 2024-03-06T07:28:16.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.670+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d7c0ce6f0 con 0x7f8d7c0b7710 2024-03-06T07:28:16.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.670+0000 7f8d78ff9640 1 -- 172.21.15.27:0/3879490617 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f8d6c058c90 con 0x7f8d7c0b7710 2024-03-06T07:28:16.673 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:16.673 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:16.673 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:16.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.672+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5c06b520 msgr2=0x7f8d5c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.672+0000 7f8d8195e640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5c06b520 0x7f8d5c06d9e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8d64036000 tx=0x7f8d64030000 comp rx=0 tx=0).stop 2024-03-06T07:28:16.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.672+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b7710 msgr2=0x7f8d7c0c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.672+0000 7f8d8195e640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b7710 0x7f8d7c0c5790 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00ebe0 tx=0x7f8d6c00c520 comp rx=0 tx=0).stop 2024-03-06T07:28:16.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.672+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 shutdown_connections 2024-03-06T07:28:16.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.672+0000 7f8d8195e640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5c06b520 0x7f8d5c06d9e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.672+0000 7f8d8195e640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7c0d27c0 0x7f8d7c0b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.672+0000 7f8d8195e640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d7c0cd0b0 0x7f8d7c06f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.672+0000 7f8d8195e640 1 --2- 172.21.15.27:0/3879490617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b7710 0x7f8d7c0c5790 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.673+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 >> 172.21.15.27:0/3879490617 conn(0x7f8d7c0d1ca0 msgr2=0x7f8d7c0d0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.673+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 shutdown_connections 2024-03-06T07:28:16.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:16.673+0000 7f8d8195e640 1 -- 172.21.15.27:0/3879490617 wait complete. 2024-03-06T07:28:16.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:28:16.845 INFO:teuthology.orchestra.run.smithi027.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":489,"msgr_send_messages":489,"msgr_recv_bytes":133415,"msgr_send_bytes":4375064,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.077564061,"msgr_running_send_time":0.041099125,"msgr_running_recv_time":0.016657357,"msgr_running_fast_dispatch_time":0.015150001,"msgr_send_messages_queue_lat":{"avgcount":488,"sum":0.041694250,"avgtime":0.000085439},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000206,"avgtime":0.000000103},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9661,"msgr_send_messages":20982,"msgr_recv_bytes":3476936,"msgr_send_bytes":26549333,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.058196689,"msgr_running_send_time":0.587498258,"msgr_running_recv_time":0.541022249,"msgr_running_fast_dispatch_time":0.019807798,"msgr_send_messages_queue_lat":{"avgcount":20981,"sum":0.382407328,"avgtime":0.000018226},"msgr_handle_ack_lat":{"avgcount":27714,"sum":0.005428194,"avgtime":0.000000195},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94016},"AsyncMessenger::Worker-2":{"msgr_recv_messages":360,"msgr_send_messages":357,"msgr_recv_bytes":343117,"msgr_send_bytes":268277,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.051960969,"msgr_running_send_time":0.024456200,"msgr_running_recv_time":0.014838523,"msgr_running_fast_dispatch_time":0.011869218,"msgr_send_messages_queue_lat":{"avgcount":356,"sum":0.036877767,"avgtime":0.000103589},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":105536,"msgr_send_encrypted_bytes":7280},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":2145,"complete_latency":{"avgcount":89,"sum":4.898802966,"avgtime":0.055042729}},"finisher-PurgeQueue":{"queue_len":425,"complete_latency":{"avgcount":1628,"sum":0.624454078,"avgtime":0.000383571}},"mds":{"request":6115,"reply":6114,"reply_latency":{"avgcount":6114,"sum":8.712291620,"avgtime":0.001424974},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5704,"inodes_top":0,"inodes_bottom":5704,"inodes_pin_tail":0,"inodes_pinned":5704,"inodes_expired":0,"inodes_with_caps":1717,"caps":1717,"subtrees":2,"traverse":6121,"traverse_hit":6120,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1310,"root_rbytes":9660416,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":683,"ceph_cap_op_grant":5795,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3204,"handle_client_caps_dirty":3000,"handle_client_cap_release":5,"process_request_cap_release":9009},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1532,"num_strays_delayed":0,"num_strays_enqueuing":113,"strays_created":2904,"strays_enqueued":1372,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10693,"evex":0,"evtrm":0,"ev":10697,"evexg":0,"evexd":0,"segadd":13,"segex":0,"segtrm":0,"seg":15,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26793208,"rdpos":4195685,"jlat":{"avgcount":16,"sum":0.780048564,"avgtime":0.048753035},"replayed":4},"mds_mem":{"ino":2804,"ino+":3214,"ino-":410,"dir":13,"dir+":13,"dir-":0,"dn":5705,"dn+":6115,"dn-":410,"cap":1717,"cap+":3202,"cap-":1485,"rss":115912,"heap":248164},"mds_server":{"dispatch_client_request":6118,"dispatch_server_request":0,"handle_client_request":6115,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.174951590,"avgtime":0.000992172},"req_getattr_latency":{"avgcount":6,"sum":0.009656876,"avgtime":0.001609479},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002708406,"avgtime":0.002708406},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.230416875,"avgtime":0.076805625},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2904,"sum":5.294557873,"avgtime":0.001823194},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5615,"average_load":5615,"avg_session_uptime":822,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18867153,"buffer_anon_items":5473,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14609957,"mds_co_items":126722,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1143,"op_send_bytes":21419518,"op_resend":0,"op_reply":1143,"op_latency":{"avgcount":1143,"sum":2.163945900,"avgtime":0.001893216},"op_inflight":0,"oplen_avg":{"avgcount":1143,"sum":1210},"op":1143,"op_r":49,"op_w":1094,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":21,"osdop_write":29,"osdop_writefull":6,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1044,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":27,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":1,"pq_executing_ops_high_water":6,"pq_executing":1,"pq_executing_high_water":6,"pq_executed_ops":1027,"pq_executed":1027,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":296,"max":104857600,"get_started":0,"get":10507,"get_sum":3005595,"get_or_fail_fail":0,"get_or_fail_success":10507,"take":0,"take_sum":0,"put":10506,"put_sum":3005299,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1143,"take_sum":21571435,"put":1143,"put_sum":21571435,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1143,"take_sum":1143,"put":1143,"put_sum":1143,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1485,"get_sum":149985,"get_or_fail_fail":0,"get_or_fail_success":1485,"take":0,"take_sum":0,"put":13,"put_sum":149985,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a619c9a820":{"val":1627384,"max":3758096384,"get_started":0,"get":10693,"get_sum":22596708,"get_or_fail_fail":0,"get_or_fail_success":10693,"take":0,"take_sum":0,"put":16,"put_sum":20969324,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:28:16.847 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.03e-07, "sum": 2.06e-07 }, "msgr_recv_bytes": 133415, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 489, "msgr_running_fast_dispatch_time": 0.015150001, "msgr_running_recv_time": 0.016657357, "msgr_running_send_time": 0.041099125, "msgr_running_total_time": 0.077564061, "msgr_send_bytes": 4375064, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 489, "msgr_send_messages_queue_lat": { "avgcount": 488, "avgtime": 8.5439e-05, "sum": 0.04169425 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 27714, "avgtime": 1.95e-07, "sum": 0.005428194 }, "msgr_recv_bytes": 3476936, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9661, "msgr_running_fast_dispatch_time": 0.019807798, "msgr_running_recv_time": 0.541022249, "msgr_running_send_time": 0.587498258, "msgr_running_total_time": 1.058196689, "msgr_send_bytes": 26549333, "msgr_send_encrypted_bytes": 94016, "msgr_send_messages": 20982, "msgr_send_messages_queue_lat": { "avgcount": 20981, "avgtime": 1.8226e-05, "sum": 0.382407328 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 343117, "msgr_recv_encrypted_bytes": 105536, "msgr_recv_messages": 360, "msgr_running_fast_dispatch_time": 0.011869218, "msgr_running_recv_time": 0.014838523, "msgr_running_send_time": 0.0244562, "msgr_running_total_time": 0.051960969, "msgr_send_bytes": 268277, "msgr_send_encrypted_bytes": 7280, "msgr_send_messages": 357, "msgr_send_messages_queue_lat": { "avgcount": 356, "avgtime": 0.000103589, "sum": 0.036877767 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 89, "avgtime": 0.055042729, "sum": 4.898802966 }, "queue_len": 2145 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1628, "avgtime": 0.000383571, "sum": 0.624454078 }, "queue_len": 425 }, "mds": { "caps": 1717, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5795, "ceph_cap_op_revoke": 683, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 5, "handle_client_caps": 3204, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5704, "inodes_bottom": 5704, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5704, "inodes_top": 0, "inodes_with_caps": 1717, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9009, "q": 0, "reply": 6114, "reply_latency": { "avgcount": 6114, "avgtime": 0.001424974, "sum": 8.71229162 }, "request": 6115, "root_rbytes": 9660416, "root_rfiles": 1310, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6121, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6120, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1532, "num_strays_delayed": 0, "num_strays_enqueuing": 113, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2904, "strays_enqueued": 1372, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10697, "evadd": 10693, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 16, "avgtime": 0.048753035, "sum": 0.780048564 }, "rdpos": 4195685, "replayed": 4, "seg": 15, "segadd": 13, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26793208 }, "mds_mem": { "cap": 1717, "cap+": 3202, "cap-": 1485, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5705, "dn+": 6115, "dn-": 410, "heap": 248164, "ino": 2804, "ino+": 3214, "ino-": 410, "rss": 115912 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6118, "dispatch_server_request": 0, "handle_client_request": 6115, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000992172, "sum": 3.17495159 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.001609479, "sum": 0.009656876 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002708406, "sum": 0.002708406 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.076805625, "sum": 0.230416875 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2904, "avgtime": 0.001823194, "sum": 5.294557873 } }, "mds_sessions": { "average_load": 5615, "avg_session_uptime": 822, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5615 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18867153, "buffer_anon_items": 5473, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 14609957, "mds_co_items": 126722, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 27, "op": 1143, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1143, "avgtime": 0.001893216, "sum": 2.1639459 }, "op_pg": 0, "op_r": 49, "op_reply": 1143, "op_resend": 0, "op_rmw": 0, "op_send": 1143, "op_send_bytes": 21419518, "op_w": 1094, "oplen_avg": { "avgcount": 1143, "sum": 1210 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1044, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 21, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 29, "osdop_writefull": 6, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1027, "pq_executed_ops": 1027, "pq_executing": 1, "pq_executing_high_water": 6, "pq_executing_ops": 1, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10507, "get_or_fail_fail": 0, "get_or_fail_success": 10507, "get_started": 0, "get_sum": 3005595, "max": 104857600, "put": 10506, "put_sum": 3005299, "take": 0, "take_sum": 0, "val": 296, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1143, "put_sum": 21571435, "take": 1143, "take_sum": 21571435, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1143, "put_sum": 1143, "take": 1143, "take_sum": 1143, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1485, "get_or_fail_fail": 0, "get_or_fail_success": 1485, "get_started": 0, "get_sum": 149985, "max": 3758096384, "put": 13, "put_sum": 149985, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a619c9a820": { "get": 10693, "get_or_fail_fail": 0, "get_or_fail_success": 10693, "get_started": 0, "get_sum": 22596708, "max": 3758096384, "put": 16, "put_sum": 20969324, "take": 0, "take_sum": 0, "val": 1627384, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:28:16.847 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 1/1532, 1027/2904 2024-03-06T07:28:17.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:17.913 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.970+0000 7efea59a0640 1 Processor -- start 2024-03-06T07:28:17.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.970+0000 7efea59a0640 1 -- start start 2024-03-06T07:28:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea59a0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efea015e0b0 0x7efea00d21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea59a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea00d48a0 0x7efea00d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea59a0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efea00c8fd0 0x7efea00c9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea59a0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea00c8210 con 0x7efea00c8fd0 2024-03-06T07:28:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea59a0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea00c84f0 con 0x7efea015e0b0 2024-03-06T07:28:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea59a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea00c87d0 con 0x7efea00d48a0 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea519f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efea00c8fd0 0x7efea00c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea519f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efea00c8fd0 0x7efea00c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47756/0 (socket says 172.21.15.27:47756) 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea519f640 1 -- 172.21.15.27:0/1332510798 learned_addr learned my addr 172.21.15.27:0/1332510798 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efe9ffff640 1 --2- 172.21.15.27:0/1332510798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea00d48a0 0x7efea00d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.971+0000 7efea499e640 1 --2- 172.21.15.27:0/1332510798 >> v2:172.21.15.69:3300/0 conn(0x7efea015e0b0 0x7efea00d21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.972+0000 7efea519f640 1 -- 172.21.15.27:0/1332510798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea00d48a0 msgr2=0x7efea00d2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.972+0000 7efea519f640 1 --2- 172.21.15.27:0/1332510798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea00d48a0 0x7efea00d2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.972+0000 7efea519f640 1 -- 172.21.15.27:0/1332510798 >> v2:172.21.15.69:3300/0 conn(0x7efea015e0b0 msgr2=0x7efea00d21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.972+0000 7efea519f640 1 --2- 172.21.15.27:0/1332510798 >> v2:172.21.15.69:3300/0 conn(0x7efea015e0b0 0x7efea00d21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.972+0000 7efea519f640 1 -- 172.21.15.27:0/1332510798 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea00c8ab0 con 0x7efea00c8fd0 2024-03-06T07:28:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.973+0000 7efea519f640 1 --2- 172.21.15.27:0/1332510798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 0x7efea00c9460 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7efea008c260 tx=0x7efe9802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf114b8fb4c70e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.973+0000 7efe9f7fe640 1 -- 172.21.15.27:0/1332510798 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9802ca80 con 0x7efea00c8fd0 2024-03-06T07:28:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.973+0000 7efe9f7fe640 1 -- 172.21.15.27:0/1332510798 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9802cc20 con 0x7efea00c8fd0 2024-03-06T07:28:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.973+0000 7efe9f7fe640 1 -- 172.21.15.27:0/1332510798 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe980343f0 con 0x7efea00c8fd0 2024-03-06T07:28:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.974+0000 7efea59a0640 1 -- 172.21.15.27:0/1332510798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 msgr2=0x7efea00c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.974+0000 7efea59a0640 1 --2- 172.21.15.27:0/1332510798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 0x7efea00c9460 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7efea008c260 tx=0x7efe9802bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.974+0000 7efea59a0640 1 -- 172.21.15.27:0/1332510798 shutdown_connections 2024-03-06T07:28:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.974+0000 7efea59a0640 1 --2- 172.21.15.27:0/1332510798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 0x7efea00c9460 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.974+0000 7efea59a0640 1 --2- 172.21.15.27:0/1332510798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea00d48a0 0x7efea00d2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.974+0000 7efea59a0640 1 --2- 172.21.15.27:0/1332510798 >> v2:172.21.15.69:3300/0 conn(0x7efea015e0b0 0x7efea00d21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.974+0000 7efea59a0640 1 -- 172.21.15.27:0/1332510798 >> 172.21.15.27:0/1332510798 conn(0x7efea00d3ca0 msgr2=0x7efea00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.974+0000 7efea59a0640 1 -- 172.21.15.27:0/1332510798 shutdown_connections 2024-03-06T07:28:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.974+0000 7efea59a0640 1 -- 172.21.15.27:0/1332510798 wait complete. 2024-03-06T07:28:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.975+0000 7efea59a0640 1 Processor -- start 2024-03-06T07:28:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.975+0000 7efea59a0640 1 -- start start 2024-03-06T07:28:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.976+0000 7efea59a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 0x7efea0154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.976+0000 7efea59a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efea00d48a0 0x7efea00a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.976+0000 7efea59a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea015e0b0 0x7efea00cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.976+0000 7efea59a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea015e610 con 0x7efea00c8fd0 2024-03-06T07:28:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.976+0000 7efea59a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea015e490 con 0x7efea00d48a0 2024-03-06T07:28:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.976+0000 7efea59a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea00d19f0 con 0x7efea015e0b0 2024-03-06T07:28:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.976+0000 7efea499e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 0x7efea0154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea499e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 0x7efea0154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47762/0 (socket says 172.21.15.27:47762) 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea499e640 1 -- 172.21.15.27:0/3354183878 learned_addr learned my addr 172.21.15.27:0/3354183878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efe9ffff640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efea00d48a0 0x7efea00a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea519f640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea015e0b0 0x7efea00cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea499e640 1 -- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea015e0b0 msgr2=0x7efea00cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea499e640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea015e0b0 0x7efea00cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea499e640 1 -- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efea00d48a0 msgr2=0x7efea00a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea499e640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efea00d48a0 0x7efea00a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea499e640 1 -- 172.21.15.27:0/3354183878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea01f4080 con 0x7efea00c8fd0 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea499e640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 0x7efea0154120 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7efe9400e870 tx=0x7efe9400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efe9d7fa640 1 -- 172.21.15.27:0/3354183878 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94017070 con 0x7efea00c8fd0 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efe9d7fa640 1 -- 172.21.15.27:0/3354183878 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9400c940 con 0x7efea00c8fd0 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea0154660 con 0x7efea00c8fd0 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.977+0000 7efe9d7fa640 1 -- 172.21.15.27:0/3354183878 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9400cd30 con 0x7efea00c8fd0 2024-03-06T07:28:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.978+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea00a1650 con 0x7efea00c8fd0 2024-03-06T07:28:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.978+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea0075260 con 0x7efea00c8fd0 2024-03-06T07:28:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.979+0000 7efe9d7fa640 1 -- 172.21.15.27:0/3354183878 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe9401a070 con 0x7efea00c8fd0 2024-03-06T07:28:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.980+0000 7efe9d7fa640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe8406b4d0 0x7efe8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.980+0000 7efe9d7fa640 1 -- 172.21.15.27:0/3354183878 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efe9408a7f0 con 0x7efea00c8fd0 2024-03-06T07:28:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.980+0000 7efe9ffff640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe8406b4d0 0x7efe8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.980+0000 7efe9ffff640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe8406b4d0 0x7efe8406d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efea015cfd0 tx=0x7efe8c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:17.988+0000 7efe9d7fa640 1 -- 172.21.15.27:0/3354183878 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe940590a0 con 0x7efea00c8fd0 2024-03-06T07:28:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.034+0000 7f9e4c80e640 1 Processor -- start 2024-03-06T07:28:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.034+0000 7f9e4c80e640 1 -- start start 2024-03-06T07:28:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e4c80e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e4815e100 0x7f9e480d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e4c80e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e480d0100 con 0x7f9e480d4c40 2024-03-06T07:28:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e4c80e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e480cf860 con 0x7f9e480d2a30 2024-03-06T07:28:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e4c80e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e480cfb40 con 0x7f9e4815e100 2024-03-06T07:28:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e4659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e4815e100 0x7f9e480d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e3ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9e480d4c40 0x7f9e480d2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57040/0 (socket says 172.21.15.27:57040) 2024-03-06T07:28:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e4659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e4815e100 0x7f9e480d1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47770/0 (socket says 172.21.15.27:47770) 2024-03-06T07:28:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e3ffff640 1 -- 172.21.15.27:0/4194516933 learned_addr learned my addr 172.21.15.27:0/4194516933 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e4659c640 1 -- 172.21.15.27:0/4194516933 >> v1:172.21.15.69:6789/0 conn(0x7f9e480d2a30 legacy=0x7f9e480b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e4659c640 1 -- 172.21.15.27:0/4194516933 >> v1:172.21.15.27:6789/0 conn(0x7f9e480d4c40 legacy=0x7f9e480d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.035+0000 7f9e4659c640 1 -- 172.21.15.27:0/4194516933 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e480cfe20 con 0x7f9e4815e100 2024-03-06T07:28:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.036+0000 7f9e4659c640 1 --2- 172.21.15.27:0/4194516933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e4815e100 0x7f9e480d1cd0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f9e380097a0 tx=0x7f9e3802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b58a743a53937e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.036+0000 7f9e45d9b640 1 -- 172.21.15.27:0/4194516933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e38035020 con 0x7f9e4815e100 2024-03-06T07:28:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.037+0000 7f9e45d9b640 1 -- 172.21.15.27:0/4194516933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e38031830 con 0x7f9e4815e100 2024-03-06T07:28:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.037+0000 7f9e45d9b640 1 -- 172.21.15.27:0/4194516933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e38031c00 con 0x7f9e4815e100 2024-03-06T07:28:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.037+0000 7f9e4c80e640 1 -- 172.21.15.27:0/4194516933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e4815e100 msgr2=0x7f9e480d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.037+0000 7f9e4c80e640 1 --2- 172.21.15.27:0/4194516933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e4815e100 0x7f9e480d1cd0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f9e380097a0 tx=0x7f9e3802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.037+0000 7f9e4c80e640 1 -- 172.21.15.27:0/4194516933 shutdown_connections 2024-03-06T07:28:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.037+0000 7f9e4c80e640 1 --2- 172.21.15.27:0/4194516933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e4815e100 0x7f9e480d1cd0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.037+0000 7f9e4c80e640 1 -- 172.21.15.27:0/4194516933 >> 172.21.15.27:0/4194516933 conn(0x7f9e480d3ef0 msgr2=0x7f9e480d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.037+0000 7f9e4c80e640 1 -- 172.21.15.27:0/4194516933 shutdown_connections 2024-03-06T07:28:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.037+0000 7f9e4c80e640 1 -- 172.21.15.27:0/4194516933 wait complete. 2024-03-06T07:28:18.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.040+0000 7f9e4c80e640 1 Processor -- start 2024-03-06T07:28:18.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.040+0000 7f9e4c80e640 1 -- start start 2024-03-06T07:28:18.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e4c80e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e480d2a30 0x7f9e480a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e4c80e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e480d4c40 0x7f9e4815cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e4c80e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e4815e100 0x7f9e481f7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e4c80e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4815e4e0 con 0x7f9e480d4c40 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e4c80e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e480cf510 con 0x7f9e480d2a30 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e4c80e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4815e660 con 0x7f9e4815e100 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e3ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e480d4c40 0x7f9e4815cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e3ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e480d4c40 0x7f9e4815cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47774/0 (socket says 172.21.15.27:47774) 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e3ffff640 1 -- 172.21.15.27:0/596802391 learned_addr learned my addr 172.21.15.27:0/596802391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e4659c640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e480d2a30 0x7f9e480a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e46d9d640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e4815e100 0x7f9e481f7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e3ffff640 1 -- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e4815e100 msgr2=0x7f9e481f7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e3ffff640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e4815e100 0x7f9e481f7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e3ffff640 1 -- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e480d2a30 msgr2=0x7f9e480a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e3ffff640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e480d2a30 0x7f9e480a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.041+0000 7f9e3ffff640 1 -- 172.21.15.27:0/596802391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4809e640 con 0x7f9e480d4c40 2024-03-06T07:28:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.042+0000 7f9e3ffff640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e480d4c40 0x7f9e4815cb30 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f9e3000c430 tx=0x7f9e3000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.042+0000 7f9e3effd640 1 -- 172.21.15.27:0/596802391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e30017070 con 0x7f9e480d4c40 2024-03-06T07:28:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.042+0000 7f9e3effd640 1 -- 172.21.15.27:0/596802391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e30005ce0 con 0x7f9e480d4c40 2024-03-06T07:28:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.042+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4809e920 con 0x7f9e480d4c40 2024-03-06T07:28:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.042+0000 7f9e3effd640 1 -- 172.21.15.27:0/596802391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e300072f0 con 0x7f9e480d4c40 2024-03-06T07:28:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.042+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4809eee0 con 0x7f9e480d4c40 2024-03-06T07:28:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.043+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e480752d0 con 0x7f9e480d4c40 2024-03-06T07:28:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.044+0000 7f9e3effd640 1 -- 172.21.15.27:0/596802391 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e3001a070 con 0x7f9e480d4c40 2024-03-06T07:28:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.044+0000 7f9e3effd640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e2806b4d0 0x7f9e2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.044+0000 7f9e3effd640 1 -- 172.21.15.27:0/596802391 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9e300895d0 con 0x7f9e480d4c40 2024-03-06T07:28:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.044+0000 7f9e4659c640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e2806b4d0 0x7f9e2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.045+0000 7f9e4659c640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e2806b4d0 0x7f9e2806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9e38036000 tx=0x7f9e38008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.053+0000 7f9e3effd640 1 -- 172.21.15.27:0/596802391 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e30057da0 con 0x7f9e480d4c40 2024-03-06T07:28:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.232+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efea006be40 con 0x7efea00c8fd0 2024-03-06T07:28:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.233+0000 7efe9d7fa640 1 -- 172.21.15.27:0/3354183878 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7efe9405d140 con 0x7efea00c8fd0 2024-03-06T07:28:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:18.235 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:18.236 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe8406b4d0 msgr2=0x7efe8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe8406b4d0 0x7efe8406d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efea015cfd0 tx=0x7efe8c009190 comp rx=0 tx=0).stop 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 msgr2=0x7efea0154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 0x7efea0154120 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7efe9400e870 tx=0x7efe9400ed30 comp rx=0 tx=0).stop 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 shutdown_connections 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe8406b4d0 0x7efe8406d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efea015e0b0 0x7efea00cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efea00d48a0 0x7efea00a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 --2- 172.21.15.27:0/3354183878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efea00c8fd0 0x7efea0154120 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 >> 172.21.15.27:0/3354183878 conn(0x7efea00d3ca0 msgr2=0x7efea00d32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 shutdown_connections 2024-03-06T07:28:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.235+0000 7efea59a0640 1 -- 172.21.15.27:0/3354183878 wait complete. 2024-03-06T07:28:18.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:28:18.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.272+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e480b9aa0 con 0x7f9e480d4c40 2024-03-06T07:28:18.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.272+0000 7f9e3effd640 1 -- 172.21.15.27:0/596802391 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f9e3005be40 con 0x7f9e480d4c40 2024-03-06T07:28:18.275 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:18.275 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:18.275 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:18.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e2806b4d0 msgr2=0x7f9e2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e2806b4d0 0x7f9e2806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9e38036000 tx=0x7f9e38008030 comp rx=0 tx=0).stop 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e480d4c40 msgr2=0x7f9e4815cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e480d4c40 0x7f9e4815cb30 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f9e3000c430 tx=0x7f9e3000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 shutdown_connections 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e2806b4d0 0x7f9e2806d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9e4815e100 0x7f9e481f7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e480d4c40 0x7f9e4815cb30 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 --2- 172.21.15.27:0/596802391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e480d2a30 0x7f9e480a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 >> 172.21.15.27:0/596802391 conn(0x7f9e480d3ef0 msgr2=0x7f9e4815e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 shutdown_connections 2024-03-06T07:28:18.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:18.275+0000 7f9e4c80e640 1 -- 172.21.15.27:0/596802391 wait complete. 2024-03-06T07:28:18.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:28:18.375 INFO:teuthology.orchestra.run.smithi027.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":676,"msgr_send_messages":676,"msgr_recv_bytes":180913,"msgr_send_bytes":4436026,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.099776897,"msgr_running_send_time":0.052068417,"msgr_running_recv_time":0.021310045,"msgr_running_fast_dispatch_time":0.020451641,"msgr_send_messages_queue_lat":{"avgcount":675,"sum":0.053222372,"avgtime":0.000078847},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000206,"avgtime":0.000000103},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10094,"msgr_send_messages":24036,"msgr_recv_bytes":3637851,"msgr_send_bytes":27381183,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.160050704,"msgr_running_send_time":0.651745015,"msgr_running_recv_time":0.572090385,"msgr_running_fast_dispatch_time":0.024153173,"msgr_send_messages_queue_lat":{"avgcount":24035,"sum":0.449227912,"avgtime":0.000018690},"msgr_handle_ack_lat":{"avgcount":29636,"sum":0.005842262,"avgtime":0.000000197},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94016},"AsyncMessenger::Worker-2":{"msgr_recv_messages":506,"msgr_send_messages":503,"msgr_recv_bytes":380161,"msgr_send_bytes":548403,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.077090854,"msgr_running_send_time":0.038291612,"msgr_running_recv_time":0.018086738,"msgr_running_fast_dispatch_time":0.018379807,"msgr_send_messages_queue_lat":{"avgcount":502,"sum":0.047026784,"avgtime":0.000093678},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":105536,"msgr_send_encrypted_bytes":7280},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":97,"sum":6.881954401,"avgtime":0.070947983}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2324,"sum":0.927146014,"avgtime":0.000398944}},"mds":{"request":6412,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.602585490,"avgtime":0.001497829},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4927,"inodes_top":0,"inodes_bottom":4927,"inodes_pin_tail":0,"inodes_pinned":4927,"inodes_expired":0,"inodes_with_caps":1248,"caps":1248,"subtrees":2,"traverse":6418,"traverse_hit":6417,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":685,"ceph_cap_op_grant":7081,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":6,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1716,"num_strays_delayed":469,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":1485,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11106,"evex":0,"evtrm":0,"ev":11110,"evexg":0,"evexd":0,"segadd":14,"segex":0,"segtrm":0,"seg":16,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":27813699,"rdpos":4195685,"jlat":{"avgcount":20,"sum":0.931256281,"avgtime":0.046562814},"replayed":4},"mds_mem":{"ino":1729,"ino+":3214,"ino-":1485,"dir":13,"dir+":13,"dir-":0,"dn":4927,"dn+":6412,"dn-":1485,"cap":1248,"cap+":3202,"cap-":1954,"rss":104360,"heap":248164},"mds_server":{"dispatch_client_request":6418,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.174951590,"avgtime":0.000992172},"req_getattr_latency":{"avgcount":6,"sum":0.009656876,"avgtime":0.001609479},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002708406,"avgtime":0.002708406},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.230416875,"avgtime":0.076805625},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.476149395,"avgtime":0.476149395},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.708702348,"avgtime":0.001783969},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5840,"average_load":5840,"avg_session_uptime":827,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":267469,"buffer_anon_items":97,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":7719515,"mds_co_items":112656,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1606,"op_send_bytes":24067573,"op_resend":0,"op_reply":1606,"op_latency":{"avgcount":1606,"sum":2.988925274,"avgtime":0.001861099},"op_inflight":0,"oplen_avg":{"avgcount":1606,"sum":1673},"op":1606,"op_r":49,"op_w":1557,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":21,"osdop_write":33,"osdop_writefull":8,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1501,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":27,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1485,"pq_executed":1485,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":11272,"get_sum":3181816,"get_or_fail_fail":0,"get_or_fail_success":11272,"take":0,"take_sum":0,"put":11272,"put_sum":3181816,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1606,"take_sum":24219490,"put":1606,"put_sum":24219490,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1606,"take_sum":1606,"put":1606,"put_sum":1606,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1485,"get_sum":149985,"get_or_fail_fail":0,"get_or_fail_success":1485,"take":0,"take_sum":0,"put":13,"put_sum":149985,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a619c9a820":{"val":0,"max":3758096384,"get_started":0,"get":11106,"get_sum":23617199,"get_or_fail_fail":0,"get_or_fail_success":11106,"take":0,"take_sum":0,"put":20,"put_sum":23617199,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:28:18.377 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.03e-07, "sum": 2.06e-07 }, "msgr_recv_bytes": 180913, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 676, "msgr_running_fast_dispatch_time": 0.020451641, "msgr_running_recv_time": 0.021310045, "msgr_running_send_time": 0.052068417, "msgr_running_total_time": 0.099776897, "msgr_send_bytes": 4436026, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 676, "msgr_send_messages_queue_lat": { "avgcount": 675, "avgtime": 7.8847e-05, "sum": 0.053222372 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29636, "avgtime": 1.97e-07, "sum": 0.005842262 }, "msgr_recv_bytes": 3637851, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 10094, "msgr_running_fast_dispatch_time": 0.024153173, "msgr_running_recv_time": 0.572090385, "msgr_running_send_time": 0.651745015, "msgr_running_total_time": 1.160050704, "msgr_send_bytes": 27381183, "msgr_send_encrypted_bytes": 94016, "msgr_send_messages": 24036, "msgr_send_messages_queue_lat": { "avgcount": 24035, "avgtime": 1.869e-05, "sum": 0.449227912 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 380161, "msgr_recv_encrypted_bytes": 105536, "msgr_recv_messages": 506, "msgr_running_fast_dispatch_time": 0.018379807, "msgr_running_recv_time": 0.018086738, "msgr_running_send_time": 0.038291612, "msgr_running_total_time": 0.077090854, "msgr_send_bytes": 548403, "msgr_send_encrypted_bytes": 7280, "msgr_send_messages": 503, "msgr_send_messages_queue_lat": { "avgcount": 502, "avgtime": 9.3678e-05, "sum": 0.047026784 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 97, "avgtime": 0.070947983, "sum": 6.881954401 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2324, "avgtime": 0.000398944, "sum": 0.927146014 }, "queue_len": 0 }, "mds": { "caps": 1248, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7081, "ceph_cap_op_revoke": 685, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 6, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4927, "inodes_bottom": 4927, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4927, "inodes_top": 0, "inodes_with_caps": 1248, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001497829, "sum": 9.60258549 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6418, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6417, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1716, "num_strays_delayed": 469, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 1485, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11110, "evadd": 11106, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 20, "avgtime": 0.046562814, "sum": 0.931256281 }, "rdpos": 4195685, "replayed": 4, "seg": 16, "segadd": 14, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 27813699 }, "mds_mem": { "cap": 1248, "cap+": 3202, "cap-": 1954, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4927, "dn+": 6412, "dn-": 1485, "heap": 248164, "ino": 1729, "ino+": 3214, "ino-": 1485, "rss": 104360 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6418, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000992172, "sum": 3.17495159 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.001609479, "sum": 0.009656876 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002708406, "sum": 0.002708406 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.076805625, "sum": 0.230416875 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.476149395, "sum": 0.476149395 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001783969, "sum": 5.708702348 } }, "mds_sessions": { "average_load": 5840, "avg_session_uptime": 827, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5840 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 267469, "buffer_anon_items": 97, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 7719515, "mds_co_items": 112656, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 27, "op": 1606, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1606, "avgtime": 0.001861099, "sum": 2.988925274 }, "op_pg": 0, "op_r": 49, "op_reply": 1606, "op_resend": 0, "op_rmw": 0, "op_send": 1606, "op_send_bytes": 24067573, "op_w": 1557, "oplen_avg": { "avgcount": 1606, "sum": 1673 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1501, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 21, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 33, "osdop_writefull": 8, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1485, "pq_executed_ops": 1485, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11272, "get_or_fail_fail": 0, "get_or_fail_success": 11272, "get_started": 0, "get_sum": 3181816, "max": 104857600, "put": 11272, "put_sum": 3181816, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1606, "put_sum": 24219490, "take": 1606, "take_sum": 24219490, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1606, "put_sum": 1606, "take": 1606, "take_sum": 1606, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1485, "get_or_fail_fail": 0, "get_or_fail_success": 1485, "get_started": 0, "get_sum": 149985, "max": 3758096384, "put": 13, "put_sum": 149985, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a619c9a820": { "get": 11106, "get_or_fail_fail": 0, "get_or_fail_success": 11106, "get_started": 0, "get_sum": 23617199, "max": 3758096384, "put": 20, "put_sum": 23617199, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:28:18.377 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1716, 1485/3201 2024-03-06T07:28:18.603 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:18.605 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:18.761 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:28:18.761 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:19.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.504+0000 7f98ef781640 1 Processor -- start 2024-03-06T07:28:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.504+0000 7f98ef781640 1 -- start start 2024-03-06T07:28:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ef781640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816ad10 0x7f98e80728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ef781640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f98e816e800 0x7f98e816ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ef781640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e816b0f0 con 0x7f98e816f1a0 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ef781640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e8070aa0 con 0x7f98e816e800 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ef781640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80b4050 con 0x7f98e816ad10 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ecd1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f98e816e800 0x7f98e816ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ed51d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816ad10 0x7f98e80728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ecd1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f98e816e800 0x7f98e816ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:37854/0 (socket says 172.21.15.27:37854) 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98edd1e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f98e816f1a0 0x7f98e80b2df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:52570/0 (socket says 172.21.15.27:52570) 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ecd1c640 1 -- 172.21.15.27:0/1195204732 learned_addr learned my addr 172.21.15.27:0/1195204732 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.505+0000 7f98ed51d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816ad10 0x7f98e80728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:48290/0 (socket says 172.21.15.27:48290) 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.548+0000 7f98ecd1c640 1 -- 172.21.15.27:0/1195204732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816ad10 msgr2=0x7f98e80728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.548+0000 7f98ecd1c640 1 --2- 172.21.15.27:0/1195204732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816ad10 0x7f98e80728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.548+0000 7f98ecd1c640 1 -- 172.21.15.27:0/1195204732 >> v1:172.21.15.69:6789/0 conn(0x7f98e816f1a0 legacy=0x7f98e80b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.548+0000 7f98ecd1c640 1 -- 172.21.15.27:0/1195204732 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e80b4330 con 0x7f98e816e800 2024-03-06T07:28:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.548+0000 7f98ed51d640 1 --2- 172.21.15.27:0/1195204732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816ad10 0x7f98e80728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.550+0000 7f98ecd1c640 1 --2- 172.21.15.27:0/1195204732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e816e800 0x7f98e816ec60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f98e400b970 tx=0x7f98e402c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79767856bbd512fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.550+0000 7f98dbfff640 1 -- 172.21.15.27:0/1195204732 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e4032070 con 0x7f98e816e800 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.550+0000 7f98dbfff640 1 -- 172.21.15.27:0/1195204732 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e402ce40 con 0x7f98e816e800 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.550+0000 7f98dbfff640 1 -- 172.21.15.27:0/1195204732 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e4036620 con 0x7f98e816e800 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.551+0000 7f98ef781640 1 -- 172.21.15.27:0/1195204732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e816e800 msgr2=0x7f98e816ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.551+0000 7f98ef781640 1 --2- 172.21.15.27:0/1195204732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e816e800 0x7f98e816ec60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f98e400b970 tx=0x7f98e402c0f0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.551+0000 7f98ef781640 1 -- 172.21.15.27:0/1195204732 shutdown_connections 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.551+0000 7f98ef781640 1 --2- 172.21.15.27:0/1195204732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e816e800 0x7f98e816ec60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.551+0000 7f98ef781640 1 --2- 172.21.15.27:0/1195204732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816ad10 0x7f98e80728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.551+0000 7f98ef781640 1 -- 172.21.15.27:0/1195204732 >> 172.21.15.27:0/1195204732 conn(0x7f98e80b17a0 msgr2=0x7f98e80b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.551+0000 7f98ef781640 1 -- 172.21.15.27:0/1195204732 shutdown_connections 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.551+0000 7f98ef781640 1 -- 172.21.15.27:0/1195204732 wait complete. 2024-03-06T07:28:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.552+0000 7f98ef781640 1 Processor -- start 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.552+0000 7f98ef781640 1 -- start start 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ef781640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e816ad10 0x7f98e815e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ef781640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e816e800 0x7f98e81f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ef781640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816f1a0 0x7f98e815d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ef781640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e816f630 con 0x7f98e816ad10 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ef781640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e8070fd0 con 0x7f98e816e800 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ef781640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e806d830 con 0x7f98e816f1a0 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ed51d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e816ad10 0x7f98e815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ed51d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e816ad10 0x7f98e815e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47786/0 (socket says 172.21.15.27:47786) 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ed51d640 1 -- 172.21.15.27:0/2267388657 learned_addr learned my addr 172.21.15.27:0/2267388657 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ecd1c640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e816e800 0x7f98e81f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98edd1e640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816f1a0 0x7f98e815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ed51d640 1 -- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816f1a0 msgr2=0x7f98e815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ed51d640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816f1a0 0x7f98e815d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ed51d640 1 -- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e816e800 msgr2=0x7f98e81f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ed51d640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e816e800 0x7f98e81f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ed51d640 1 -- 172.21.15.27:0/2267388657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e8163400 con 0x7f98e816ad10 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98edd1e640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816f1a0 0x7f98e815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98ed51d640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e816ad10 0x7f98e815e080 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00dd90 tx=0x7f98dc010490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.553+0000 7f98d9ffb640 1 -- 172.21.15.27:0/2267388657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc010da0 con 0x7f98e816ad10 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.554+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e81636e0 con 0x7f98e816ad10 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.554+0000 7f98d9ffb640 1 -- 172.21.15.27:0/2267388657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc01d070 con 0x7f98e816ad10 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.554+0000 7f98d9ffb640 1 -- 172.21.15.27:0/2267388657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc0186a0 con 0x7f98e816ad10 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.554+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e816b780 con 0x7f98e816ad10 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.555+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e81638b0 con 0x7f98e816ad10 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.555+0000 7f98d9ffb640 1 -- 172.21.15.27:0/2267388657 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98dc016050 con 0x7f98e816ad10 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.556+0000 7f98d9ffb640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98cc06b4d0 0x7f98cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.556+0000 7f98d9ffb640 1 -- 172.21.15.27:0/2267388657 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f98dc08e150 con 0x7f98e816ad10 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.556+0000 7f98ecd1c640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98cc06b4d0 0x7f98cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.557+0000 7f98ecd1c640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98cc06b4d0 0x7f98cc06d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f98e4037000 tx=0x7f98e4032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.560+0000 7f98d9ffb640 1 -- 172.21.15.27:0/2267388657 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98dc026020 con 0x7f98e816ad10 2024-03-06T07:28:19.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.915+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98e80752d0 con 0x7f98e816ad10 2024-03-06T07:28:19.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.916+0000 7f98d9ffb640 1 -- 172.21.15.27:0/2267388657 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f98dc014040 con 0x7f98e816ad10 2024-03-06T07:28:19.918 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:19.918 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:19.919 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98cc06b4d0 msgr2=0x7f98cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98cc06b4d0 0x7f98cc06d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f98e4037000 tx=0x7f98e4032000 comp rx=0 tx=0).stop 2024-03-06T07:28:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e816ad10 msgr2=0x7f98e815e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e816ad10 0x7f98e815e080 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00dd90 tx=0x7f98dc010490 comp rx=0 tx=0).stop 2024-03-06T07:28:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 shutdown_connections 2024-03-06T07:28:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98cc06b4d0 0x7f98cc06d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98e816f1a0 0x7f98e815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98e816e800 0x7f98e81f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 --2- 172.21.15.27:0/2267388657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e816ad10 0x7f98e815e080 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 >> 172.21.15.27:0/2267388657 conn(0x7f98e80b17a0 msgr2=0x7f98e80b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 shutdown_connections 2024-03-06T07:28:19.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:19.919+0000 7f98ef781640 1 -- 172.21.15.27:0/2267388657 wait complete. 2024-03-06T07:28:19.939 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:28:20.145 INFO:teuthology.orchestra.run.smithi027.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2039,"msgr_send_messages":2041,"msgr_recv_bytes":589726,"msgr_send_bytes":5885645,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.187018644,"msgr_running_send_time":0.082477731,"msgr_running_recv_time":0.039502841,"msgr_running_fast_dispatch_time":0.055172010,"msgr_send_messages_queue_lat":{"avgcount":2040,"sum":1.968684839,"avgtime":0.000965041},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000206,"avgtime":0.000000103},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10950,"msgr_send_messages":24892,"msgr_recv_bytes":3924348,"msgr_send_bytes":27782833,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.240360808,"msgr_running_send_time":0.678905799,"msgr_running_recv_time":0.583055019,"msgr_running_fast_dispatch_time":0.044911450,"msgr_send_messages_queue_lat":{"avgcount":24891,"sum":2.446244629,"avgtime":0.000098278},"msgr_handle_ack_lat":{"avgcount":29640,"sum":0.005842675,"avgtime":0.000000197},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1458,"msgr_send_messages":1456,"msgr_recv_bytes":839020,"msgr_send_bytes":1243627,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.139158338,"msgr_running_send_time":0.062058336,"msgr_running_recv_time":0.031001148,"msgr_running_fast_dispatch_time":0.039092300,"msgr_send_messages_queue_lat":{"avgcount":1455,"sum":0.242169334,"avgtime":0.000166439},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":105536,"msgr_send_encrypted_bytes":7280},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":123,"sum":7.126831924,"avgtime":0.057941722}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4703,"sum":1.106842529,"avgtime":0.000235348}},"mds":{"request":6412,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.602585490,"avgtime":0.001497829},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":4927,"inodes_top":0,"inodes_bottom":4927,"inodes_pin_tail":0,"inodes_pinned":4927,"inodes_expired":0,"inodes_with_caps":1248,"caps":1248,"subtrees":2,"traverse":6418,"traverse_hit":6417,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":685,"ceph_cap_op_grant":7081,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1,"num_strays_delayed":1,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3200,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12824,"evex":11110,"evtrm":11110,"ev":1718,"evexg":0,"evexd":11109,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":27813699,"wrpos":28848156,"rdpos":4195685,"jlat":{"avgcount":22,"sum":0.984137748,"avgtime":0.044733534},"replayed":4},"mds_mem":{"ino":1729,"ino+":3214,"ino-":1485,"dir":13,"dir+":13,"dir-":0,"dn":1727,"dn+":6412,"dn-":4685,"cap":1,"cap+":3202,"cap-":3201,"rss":120616,"heap":248164},"mds_server":{"dispatch_client_request":6418,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.174951590,"avgtime":0.000992172},"req_getattr_latency":{"avgcount":6,"sum":0.009656876,"avgtime":0.001609479},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002708406,"avgtime":0.002708406},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.230416875,"avgtime":0.076805625},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.476149395,"avgtime":0.476149395},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.708702348,"avgtime":0.001783969},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5840,"average_load":5840,"avg_session_uptime":827,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":267469,"buffer_anon_items":97,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":7719515,"mds_co_items":112656,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":4781,"op_send_bytes":25446196,"op_resend":0,"op_reply":4775,"op_latency":{"avgcount":4775,"sum":227.464313975,"avgtime":0.047636505},"op_inflight":5,"oplen_avg":{"avgcount":4781,"sum":8322},"op":4781,"op_r":52,"op_w":4729,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1726,"osdop_read":24,"osdop_write":38,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2921,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3452,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":51,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":2894,"pq_executed":2894,"pq_item_in_journal":306},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14445,"get_sum":4050923,"get_or_fail_fail":0,"get_or_fail_success":14445,"take":0,"take_sum":0,"put":14445,"put_sum":4050923,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4781,"take_sum":25771328,"put":4775,"put_sum":25771328,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":6,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4781,"take_sum":4781,"put":4775,"put_sum":4775,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3200,"get_sum":323200,"get_or_fail_fail":0,"get_or_fail_success":3200,"take":0,"take_sum":0,"put":16,"put_sum":323200,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a619c9a820":{"val":1032427,"max":3758096384,"get_started":0,"get":12824,"get_sum":24651656,"get_or_fail_fail":0,"get_or_fail_success":12824,"take":0,"take_sum":0,"put":22,"put_sum":23619229,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:28:20.147 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.03e-07, "sum": 2.06e-07 }, "msgr_recv_bytes": 589726, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 2039, "msgr_running_fast_dispatch_time": 0.05517201, "msgr_running_recv_time": 0.039502841, "msgr_running_send_time": 0.082477731, "msgr_running_total_time": 0.187018644, "msgr_send_bytes": 5885645, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 2041, "msgr_send_messages_queue_lat": { "avgcount": 2040, "avgtime": 0.000965041, "sum": 1.968684839 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29640, "avgtime": 1.97e-07, "sum": 0.005842675 }, "msgr_recv_bytes": 3924348, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 10950, "msgr_running_fast_dispatch_time": 0.04491145, "msgr_running_recv_time": 0.583055019, "msgr_running_send_time": 0.678905799, "msgr_running_total_time": 1.240360808, "msgr_send_bytes": 27782833, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 24892, "msgr_send_messages_queue_lat": { "avgcount": 24891, "avgtime": 9.8278e-05, "sum": 2.446244629 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 839020, "msgr_recv_encrypted_bytes": 105536, "msgr_recv_messages": 1458, "msgr_running_fast_dispatch_time": 0.0390923, "msgr_running_recv_time": 0.031001148, "msgr_running_send_time": 0.062058336, "msgr_running_total_time": 0.139158338, "msgr_send_bytes": 1243627, "msgr_send_encrypted_bytes": 7280, "msgr_send_messages": 1456, "msgr_send_messages_queue_lat": { "avgcount": 1455, "avgtime": 0.000166439, "sum": 0.242169334 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 123, "avgtime": 0.057941722, "sum": 7.126831924 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4703, "avgtime": 0.000235348, "sum": 1.106842529 }, "queue_len": 0 }, "mds": { "caps": 1248, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7081, "ceph_cap_op_revoke": 685, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4927, "inodes_bottom": 4927, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4927, "inodes_top": 0, "inodes_with_caps": 1248, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001497829, "sum": 9.60258549 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6418, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6417, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1, "num_strays_delayed": 1, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3200, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1718, "evadd": 12824, "evex": 11110, "evexd": 11109, "evexg": 0, "evlrg": 0, "evtrm": 11110, "expos": 27813699, "jlat": { "avgcount": 22, "avgtime": 0.044733534, "sum": 0.984137748 }, "rdpos": 4195685, "replayed": 4, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28848156 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1727, "dn+": 6412, "dn-": 4685, "heap": 248164, "ino": 1729, "ino+": 3214, "ino-": 1485, "rss": 120616 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6418, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000992172, "sum": 3.17495159 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.001609479, "sum": 0.009656876 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002708406, "sum": 0.002708406 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.076805625, "sum": 0.230416875 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.476149395, "sum": 0.476149395 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001783969, "sum": 5.708702348 } }, "mds_sessions": { "average_load": 5840, "avg_session_uptime": 827, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5840 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 267469, "buffer_anon_items": 97, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 7719515, "mds_co_items": 112656, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 51, "op": 4781, "op_Infinitylight": 5, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 4775, "avgtime": 0.047636505, "sum": 227.464313975 }, "op_pg": 0, "op_r": 52, "op_reply": 4775, "op_resend": 0, "op_rmw": 0, "op_send": 4781, "op_send_bytes": 25446196, "op_w": 4729, "oplen_avg": { "avgcount": 4781, "sum": 8322 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1726, "osdop_delete": 2921, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 24, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3452, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 38, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2894, "pq_executed_ops": 2894, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 306 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14445, "get_or_fail_fail": 0, "get_or_fail_success": 14445, "get_started": 0, "get_sum": 4050923, "max": 104857600, "put": 14445, "put_sum": 4050923, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4775, "put_sum": 25771328, "take": 4781, "take_sum": 25771328, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4775, "put_sum": 4775, "take": 4781, "take_sum": 4781, "val": 6, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3200, "get_or_fail_fail": 0, "get_or_fail_success": 3200, "get_started": 0, "get_sum": 323200, "max": 3758096384, "put": 16, "put_sum": 323200, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a619c9a820": { "get": 12824, "get_or_fail_fail": 0, "get_or_fail_success": 12824, "get_started": 0, "get_sum": 24651656, "max": 3758096384, "put": 22, "put_sum": 23619229, "take": 0, "take_sum": 0, "val": 1032427, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:28:20.147 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/1, 2894/3201 2024-03-06T07:28:21.148 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:21.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.254+0000 7fc015bf6640 1 Processor -- start 2024-03-06T07:28:21.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.254+0000 7fc015bf6640 1 -- start start 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.255+0000 7fc015bf6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0100d2680 0x7fc0100d4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.255+0000 7fc015bf6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0100b3050 0x7fc0100b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.255+0000 7fc015bf6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0100ceab0 con 0x7fc0100b1040 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.255+0000 7fc015bf6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0100cf380 con 0x7fc0100d2680 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.255+0000 7fc015bf6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0100d1d10 con 0x7fc0100b3050 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.255+0000 7fc00f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0100d2680 0x7fc0100d4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.255+0000 7fc00ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc0100b1040 0x7fc0100cf810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57050/0 (socket says 172.21.15.27:57050) 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.255+0000 7fc00ffff640 1 -- 172.21.15.27:0/1150612285 learned_addr learned my addr 172.21.15.27:0/1150612285 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.256+0000 7fc00effd640 1 --2- 172.21.15.27:0/1150612285 >> v2:172.21.15.69:3300/0 conn(0x7fc0100b3050 0x7fc0100b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.256+0000 7fc00f7fe640 1 -- 172.21.15.27:0/1150612285 >> v2:172.21.15.69:3300/0 conn(0x7fc0100b3050 msgr2=0x7fc0100b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.256+0000 7fc00f7fe640 1 --2- 172.21.15.27:0/1150612285 >> v2:172.21.15.69:3300/0 conn(0x7fc0100b3050 0x7fc0100b0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.256+0000 7fc00f7fe640 1 -- 172.21.15.27:0/1150612285 >> v1:172.21.15.27:6789/0 conn(0x7fc0100b1040 legacy=0x7fc0100cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.256+0000 7fc00e7fc640 1 -- 172.21.15.27:0/1150612285 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058468435 0 0) 0x7fc008001490 con 0x7fc0100b1040 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.256+0000 7fc00f7fe640 1 -- 172.21.15.27:0/1150612285 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100d1ff0 con 0x7fc0100d2680 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.256+0000 7fc00f7fe640 1 --2- 172.21.15.27:0/1150612285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 0x7fc0100d4a50 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc0000097a0 tx=0x7fc00002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fa3bc83cef563a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc00e7fc640 1 -- 172.21.15.27:0/1150612285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000035020 con 0x7fc0100d2680 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc00e7fc640 1 -- 172.21.15.27:0/1150612285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc000031830 con 0x7fc0100d2680 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc00e7fc640 1 -- 172.21.15.27:0/1150612285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000031c00 con 0x7fc0100d2680 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc015bf6640 1 -- 172.21.15.27:0/1150612285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 msgr2=0x7fc0100d4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc015bf6640 1 --2- 172.21.15.27:0/1150612285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 0x7fc0100d4a50 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc0000097a0 tx=0x7fc00002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc015bf6640 1 -- 172.21.15.27:0/1150612285 shutdown_connections 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc015bf6640 1 --2- 172.21.15.27:0/1150612285 >> v2:172.21.15.69:3300/0 conn(0x7fc0100b3050 0x7fc0100b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc015bf6640 1 --2- 172.21.15.27:0/1150612285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 0x7fc0100d4a50 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc015bf6640 1 -- 172.21.15.27:0/1150612285 >> 172.21.15.27:0/1150612285 conn(0x7fc0100d3f30 msgr2=0x7fc0100d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc015bf6640 1 -- 172.21.15.27:0/1150612285 shutdown_connections 2024-03-06T07:28:21.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.257+0000 7fc015bf6640 1 -- 172.21.15.27:0/1150612285 wait complete. 2024-03-06T07:28:21.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.258+0000 7fc015bf6640 1 Processor -- start 2024-03-06T07:28:21.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.259+0000 7fc015bf6640 1 -- start start 2024-03-06T07:28:21.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.259+0000 7fc015bf6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0100b1040 0x7fc010155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.259+0000 7fc015bf6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0100b3050 0x7fc01015dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.259+0000 7fc015bf6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 0x7fc01015e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.259+0000 7fc015bf6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100cff20 con 0x7fc0100d2680 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.259+0000 7fc015bf6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100cf010 con 0x7fc0100b3050 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.259+0000 7fc015bf6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100d00a0 con 0x7fc0100b1040 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 0x7fc01015e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 0x7fc01015e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47808/0 (socket says 172.21.15.27:47808) 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00ffff640 1 -- 172.21.15.27:0/3191116535 learned_addr learned my addr 172.21.15.27:0/3191116535 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00effd640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0100b3050 0x7fc01015dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00f7fe640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0100b1040 0x7fc010155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00ffff640 1 -- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0100b1040 msgr2=0x7fc010155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00ffff640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0100b1040 0x7fc010155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00ffff640 1 -- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0100b3050 msgr2=0x7fc01015dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00ffff640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0100b3050 0x7fc01015dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00ffff640 1 -- 172.21.15.27:0/3191116535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100cd0f0 con 0x7fc0100d2680 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00effd640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0100b3050 0x7fc01015dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc00ffff640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 0x7fc01015e400 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc00800c730 tx=0x7fc00800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fbff7fff640 1 -- 172.21.15.27:0/3191116535 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008018070 con 0x7fc0100d2680 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fbff7fff640 1 -- 172.21.15.27:0/3191116535 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00800ce40 con 0x7fc0100d2680 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0100cd3d0 con 0x7fc0100d2680 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fbff7fff640 1 -- 172.21.15.27:0/3191116535 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00800a860 con 0x7fc0100d2680 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.260+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0100cd990 con 0x7fc0100d2680 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.262+0000 7fbff7fff640 1 -- 172.21.15.27:0/3191116535 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc00801b070 con 0x7fc0100d2680 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.262+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0100752d0 con 0x7fc0100d2680 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.262+0000 7fbff7fff640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbffc06b4d0 0x7fbffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.262+0000 7fbff7fff640 1 -- 172.21.15.27:0/3191116535 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc008089860 con 0x7fc0100d2680 2024-03-06T07:28:21.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.263+0000 7fc00f7fe640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbffc06b4d0 0x7fbffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.263+0000 7fc00f7fe640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbffc06b4d0 0x7fbffc06d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc000036000 tx=0x7fc00003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.267+0000 7fbff7fff640 1 -- 172.21.15.27:0/3191116535 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0100ceab0 con 0x7fc0100d2680 2024-03-06T07:28:21.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.620+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0100b5800 con 0x7fc0100d2680 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.620+0000 7fbff7fff640 1 -- 172.21.15.27:0/3191116535 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fc008058680 con 0x7fc0100d2680 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:21.622 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbffc06b4d0 msgr2=0x7fbffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbffc06b4d0 0x7fbffc06d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc000036000 tx=0x7fc00003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 msgr2=0x7fc01015e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 0x7fc01015e400 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc00800c730 tx=0x7fc00800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 shutdown_connections 2024-03-06T07:28:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbffc06b4d0 0x7fbffc06d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0100d2680 0x7fc01015e400 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0100b3050 0x7fc01015dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 --2- 172.21.15.27:0/3191116535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0100b1040 0x7fc010155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.622+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 >> 172.21.15.27:0/3191116535 conn(0x7fc0100d3f30 msgr2=0x7fc01006b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.623+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 shutdown_connections 2024-03-06T07:28:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.623+0000 7fc015bf6640 1 -- 172.21.15.27:0/3191116535 wait complete. 2024-03-06T07:28:21.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:28:21.757 INFO:teuthology.orchestra.run.smithi027.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2162,"msgr_send_messages":2162,"msgr_recv_bytes":620968,"msgr_send_bytes":5925091,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.195021240,"msgr_running_send_time":0.085386597,"msgr_running_recv_time":0.041569853,"msgr_running_fast_dispatch_time":0.057651356,"msgr_send_messages_queue_lat":{"avgcount":2161,"sum":1.971847818,"avgtime":0.000912470},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000206,"avgtime":0.000000103},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11040,"msgr_send_messages":24978,"msgr_recv_bytes":3947410,"msgr_send_bytes":27810978,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.246901537,"msgr_running_send_time":0.681450639,"msgr_running_recv_time":0.584568954,"msgr_running_fast_dispatch_time":0.046890639,"msgr_send_messages_queue_lat":{"avgcount":24977,"sum":2.448292863,"avgtime":0.000098021},"msgr_handle_ack_lat":{"avgcount":29641,"sum":0.005842786,"avgtime":0.000000197},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1561,"msgr_send_messages":1558,"msgr_recv_bytes":865269,"msgr_send_bytes":1277426,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.145762700,"msgr_running_send_time":0.064268624,"msgr_running_recv_time":0.032628104,"msgr_running_fast_dispatch_time":0.041167530,"msgr_send_messages_queue_lat":{"avgcount":1557,"sum":0.244606523,"avgtime":0.000157101},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":105808,"msgr_send_encrypted_bytes":7904},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":7.126842763,"avgtime":0.057474538}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5228,"sum":1.123715218,"avgtime":0.000214941}},"mds":{"request":6412,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.602585490,"avgtime":0.001497829},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":4927,"inodes_top":0,"inodes_bottom":4927,"inodes_pin_tail":0,"inodes_pinned":4927,"inodes_expired":0,"inodes_with_caps":1248,"caps":1248,"subtrees":2,"traverse":6418,"traverse_hit":6417,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":685,"ceph_cap_op_grant":7081,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12825,"evex":11110,"evtrm":11110,"ev":1719,"evexg":0,"evexd":11109,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":27813699,"wrpos":28848755,"rdpos":4195685,"jlat":{"avgcount":22,"sum":0.984137748,"avgtime":0.044733534},"replayed":4},"mds_mem":{"ino":1729,"ino+":3214,"ino-":1485,"dir":12,"dir+":13,"dir-":1,"dn":1727,"dn+":6412,"dn-":4685,"cap":1,"cap+":3202,"cap-":3201,"rss":120616,"heap":248164},"mds_server":{"dispatch_client_request":6418,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.174951590,"avgtime":0.000992172},"req_getattr_latency":{"avgcount":6,"sum":0.009656876,"avgtime":0.001609479},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002708406,"avgtime":0.002708406},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.230416875,"avgtime":0.076805625},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.476149395,"avgtime":0.476149395},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.708702348,"avgtime":0.001783969},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5840,"average_load":5840,"avg_session_uptime":827,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":267469,"buffer_anon_items":97,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":7719515,"mds_co_items":112656,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5087,"op_send_bytes":25446552,"op_resend":0,"op_reply":5087,"op_latency":{"avgcount":5087,"sum":227.940302075,"avgtime":0.044808394},"op_inflight":0,"oplen_avg":{"avgcount":5087,"sum":8629},"op":5087,"op_r":53,"op_w":5034,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1726,"osdop_read":25,"osdop_write":39,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3452,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":52,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14759,"get_sum":4102656,"get_or_fail_fail":0,"get_or_fail_success":14759,"take":0,"take_sum":0,"put":14759,"put_sum":4102656,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5087,"take_sum":25771785,"put":5087,"put_sum":25771785,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5087,"take_sum":5087,"put":5087,"put_sum":5087,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":17,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a619c9a820":{"val":1033026,"max":3758096384,"get_started":0,"get":12825,"get_sum":24652255,"get_or_fail_fail":0,"get_or_fail_success":12825,"take":0,"take_sum":0,"put":22,"put_sum":23619229,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:28:21.759 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.03e-07, "sum": 2.06e-07 }, "msgr_recv_bytes": 620968, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 2162, "msgr_running_fast_dispatch_time": 0.057651356, "msgr_running_recv_time": 0.041569853, "msgr_running_send_time": 0.085386597, "msgr_running_total_time": 0.19502124, "msgr_send_bytes": 5925091, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 2162, "msgr_send_messages_queue_lat": { "avgcount": 2161, "avgtime": 0.00091247, "sum": 1.971847818 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29641, "avgtime": 1.97e-07, "sum": 0.005842786 }, "msgr_recv_bytes": 3947410, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 11040, "msgr_running_fast_dispatch_time": 0.046890639, "msgr_running_recv_time": 0.584568954, "msgr_running_send_time": 0.681450639, "msgr_running_total_time": 1.246901537, "msgr_send_bytes": 27810978, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 24978, "msgr_send_messages_queue_lat": { "avgcount": 24977, "avgtime": 9.8021e-05, "sum": 2.448292863 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 865269, "msgr_recv_encrypted_bytes": 105808, "msgr_recv_messages": 1561, "msgr_running_fast_dispatch_time": 0.04116753, "msgr_running_recv_time": 0.032628104, "msgr_running_send_time": 0.064268624, "msgr_running_total_time": 0.1457627, "msgr_send_bytes": 1277426, "msgr_send_encrypted_bytes": 7904, "msgr_send_messages": 1558, "msgr_send_messages_queue_lat": { "avgcount": 1557, "avgtime": 0.000157101, "sum": 0.244606523 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.057474538, "sum": 7.126842763 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5228, "avgtime": 0.000214941, "sum": 1.123715218 }, "queue_len": 0 }, "mds": { "caps": 1248, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7081, "ceph_cap_op_revoke": 685, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4927, "inodes_bottom": 4927, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4927, "inodes_top": 0, "inodes_with_caps": 1248, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001497829, "sum": 9.60258549 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6418, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6417, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1719, "evadd": 12825, "evex": 11110, "evexd": 11109, "evexg": 0, "evlrg": 0, "evtrm": 11110, "expos": 27813699, "jlat": { "avgcount": 22, "avgtime": 0.044733534, "sum": 0.984137748 }, "rdpos": 4195685, "replayed": 4, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28848755 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 1727, "dn+": 6412, "dn-": 4685, "heap": 248164, "ino": 1729, "ino+": 3214, "ino-": 1485, "rss": 120616 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6418, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000992172, "sum": 3.17495159 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.001609479, "sum": 0.009656876 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002708406, "sum": 0.002708406 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.076805625, "sum": 0.230416875 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.476149395, "sum": 0.476149395 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001783969, "sum": 5.708702348 } }, "mds_sessions": { "average_load": 5840, "avg_session_uptime": 827, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5840 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 267469, "buffer_anon_items": 97, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 7719515, "mds_co_items": 112656, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 52, "op": 5087, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5087, "avgtime": 0.044808394, "sum": 227.940302075 }, "op_pg": 0, "op_r": 53, "op_reply": 5087, "op_resend": 0, "op_rmw": 0, "op_send": 5087, "op_send_bytes": 25446552, "op_w": 5034, "oplen_avg": { "avgcount": 5087, "sum": 8629 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1726, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 25, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3452, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 39, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14759, "get_or_fail_fail": 0, "get_or_fail_success": 14759, "get_started": 0, "get_sum": 4102656, "max": 104857600, "put": 14759, "put_sum": 4102656, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5087, "put_sum": 25771785, "take": 5087, "take_sum": 25771785, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5087, "put_sum": 5087, "take": 5087, "take_sum": 5087, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 17, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a619c9a820": { "get": 12825, "get_or_fail_fail": 0, "get_or_fail_success": 12825, "get_started": 0, "get_sum": 24652255, "max": 3758096384, "put": 22, "put_sum": 23619229, "take": 0, "take_sum": 0, "val": 1033026, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:28:21.759 INFO:tasks.cephfs.test_strays:Complete purge in 7 seconds 2024-03-06T07:28:21.759 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:21.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.870+0000 7f209117a640 1 Processor -- start 2024-03-06T07:28:21.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.870+0000 7f209117a640 1 -- start start 2024-03-06T07:28:21.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f209117a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f208c0b3050 0x7f208c0d4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f209117a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f208c0d2a60 0x7f208c0cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f209117a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0ceab0 con 0x7f208c0d2680 2024-03-06T07:28:21.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f209117a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c0cf380 con 0x7f208c0d2a60 2024-03-06T07:28:21.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f209117a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c0d1d10 con 0x7f208c0b3050 2024-03-06T07:28:21.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f208b59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f208c0d2a60 0x7f208c0cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f208ad9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f208c0d2680 0x7f208c0b0b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57058/0 (socket says 172.21.15.27:57058) 2024-03-06T07:28:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f208ad9d640 1 -- 172.21.15.27:0/2321923190 learned_addr learned my addr 172.21.15.27:0/2321923190 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f208a59c640 1 --2- 172.21.15.27:0/2321923190 >> v2:172.21.15.69:3300/0 conn(0x7f208c0b3050 0x7f208c0d4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f208b59e640 1 -- 172.21.15.27:0/2321923190 >> v2:172.21.15.69:3300/0 conn(0x7f208c0b3050 msgr2=0x7f208c0d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f208b59e640 1 --2- 172.21.15.27:0/2321923190 >> v2:172.21.15.69:3300/0 conn(0x7f208c0b3050 0x7f208c0d4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f208b59e640 1 -- 172.21.15.27:0/2321923190 >> v1:172.21.15.27:6789/0 conn(0x7f208c0d2680 legacy=0x7f208c0b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.871+0000 7f208b59e640 1 -- 172.21.15.27:0/2321923190 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0d1ff0 con 0x7f208c0d2a60 2024-03-06T07:28:21.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.872+0000 7f208b59e640 1 --2- 172.21.15.27:0/2321923190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 0x7f208c0cf810 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f208c0b9fe0 tx=0x7f208402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e068bc5c76c82c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.872+0000 7f2089d9b640 1 -- 172.21.15.27:0/2321923190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084039070 con 0x7f208c0d2a60 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.872+0000 7f2089d9b640 1 -- 172.21.15.27:0/2321923190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208402cc80 con 0x7f208c0d2a60 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.872+0000 7f2089d9b640 1 -- 172.21.15.27:0/2321923190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084034660 con 0x7f208c0d2a60 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.873+0000 7f209117a640 1 -- 172.21.15.27:0/2321923190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 msgr2=0x7f208c0cf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.873+0000 7f209117a640 1 --2- 172.21.15.27:0/2321923190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 0x7f208c0cf810 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f208c0b9fe0 tx=0x7f208402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.873+0000 7f209117a640 1 -- 172.21.15.27:0/2321923190 shutdown_connections 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.873+0000 7f209117a640 1 --2- 172.21.15.27:0/2321923190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 0x7f208c0cf810 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.873+0000 7f209117a640 1 --2- 172.21.15.27:0/2321923190 >> v2:172.21.15.69:3300/0 conn(0x7f208c0b3050 0x7f208c0d4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.873+0000 7f209117a640 1 -- 172.21.15.27:0/2321923190 >> 172.21.15.27:0/2321923190 conn(0x7f208c0d3f30 msgr2=0x7f208c0d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.873+0000 7f209117a640 1 -- 172.21.15.27:0/2321923190 shutdown_connections 2024-03-06T07:28:21.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.873+0000 7f209117a640 1 -- 172.21.15.27:0/2321923190 wait complete. 2024-03-06T07:28:21.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.875+0000 7f209117a640 1 Processor -- start 2024-03-06T07:28:21.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.875+0000 7f209117a640 1 -- start start 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f209117a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f208c0b3050 0x7f208c15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f209117a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208c0d2680 0x7f208c0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f209117a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 0x7f208c15b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f209117a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0b1210 con 0x7f208c0d2a60 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f209117a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0cf010 con 0x7f208c0d2680 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f209117a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0cfd50 con 0x7f208c0b3050 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 0x7f208c15b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 0x7f208c15b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47814/0 (socket says 172.21.15.27:47814) 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208b59e640 1 -- 172.21.15.27:0/3919701802 learned_addr learned my addr 172.21.15.27:0/3919701802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:21.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208a59c640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208c0d2680 0x7f208c0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208ad9d640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f208c0b3050 0x7f208c15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208b59e640 1 -- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f208c0b3050 msgr2=0x7f208c15cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208b59e640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f208c0b3050 0x7f208c15cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208b59e640 1 -- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208c0d2680 msgr2=0x7f208c0cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208b59e640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208c0d2680 0x7f208c0cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208b59e640 1 -- 172.21.15.27:0/3919701802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0ce180 con 0x7f208c0d2a60 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.877+0000 7f208b59e640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 0x7f208c15b6c0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f2084009e80 tx=0x7f2084004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.878+0000 7f20737fe640 1 -- 172.21.15.27:0/3919701802 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084039050 con 0x7f208c0d2a60 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.878+0000 7f20737fe640 1 -- 172.21.15.27:0/3919701802 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2084045070 con 0x7f208c0d2a60 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.878+0000 7f20737fe640 1 -- 172.21.15.27:0/3919701802 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208403e830 con 0x7f208c0d2a60 2024-03-06T07:28:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.878+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c15d630 con 0x7f208c0d2a60 2024-03-06T07:28:21.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.878+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c15dbf0 con 0x7f208c0d2a60 2024-03-06T07:28:21.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.879+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c0752d0 con 0x7f208c0d2a60 2024-03-06T07:28:21.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.879+0000 7f20737fe640 1 -- 172.21.15.27:0/3919701802 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f208403e9d0 con 0x7f208c0d2a60 2024-03-06T07:28:21.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.880+0000 7f20737fe640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207806b4d0 0x7f207806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.880+0000 7f20737fe640 1 -- 172.21.15.27:0/3919701802 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f20840ad190 con 0x7f208c0d2a60 2024-03-06T07:28:21.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.880+0000 7f208ad9d640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207806b4d0 0x7f207806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.881+0000 7f208ad9d640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207806b4d0 0x7f207806d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2080007fd0 tx=0x7f2080007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:21.884+0000 7f20737fe640 1 -- 172.21.15.27:0/3919701802 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f208c0ce180 con 0x7f208c0d2a60 2024-03-06T07:28:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.105+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f208c0b5800 con 0x7f208c0d2a60 2024-03-06T07:28:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.106+0000 7f20737fe640 1 -- 172.21.15.27:0/3919701802 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f208407ba80 con 0x7f208c0d2a60 2024-03-06T07:28:22.108 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:22.108 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:22.108 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207806b4d0 msgr2=0x7f207806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207806b4d0 0x7f207806d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2080007fd0 tx=0x7f2080007e80 comp rx=0 tx=0).stop 2024-03-06T07:28:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 msgr2=0x7f208c15b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 0x7f208c15b6c0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f2084009e80 tx=0x7f2084004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 shutdown_connections 2024-03-06T07:28:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207806b4d0 0x7f207806d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208c0d2a60 0x7f208c15b6c0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208c0d2680 0x7f208c0cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 --2- 172.21.15.27:0/3919701802 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f208c0b3050 0x7f208c15cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 >> 172.21.15.27:0/3919701802 conn(0x7f208c0d3f30 msgr2=0x7f208c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 shutdown_connections 2024-03-06T07:28:22.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.108+0000 7f209117a640 1 -- 172.21.15.27:0/3919701802 wait complete. 2024-03-06T07:28:22.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:28:22.243 INFO:teuthology.orchestra.run.smithi027.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2162,"msgr_send_messages":2162,"msgr_recv_bytes":620968,"msgr_send_bytes":5925091,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.195021240,"msgr_running_send_time":0.085386597,"msgr_running_recv_time":0.041569853,"msgr_running_fast_dispatch_time":0.057651356,"msgr_send_messages_queue_lat":{"avgcount":2161,"sum":1.971847818,"avgtime":0.000912470},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000206,"avgtime":0.000000103},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11041,"msgr_send_messages":24978,"msgr_recv_bytes":3947828,"msgr_send_bytes":27810978,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.246987663,"msgr_running_send_time":0.681477184,"msgr_running_recv_time":0.584591273,"msgr_running_fast_dispatch_time":0.046925138,"msgr_send_messages_queue_lat":{"avgcount":24977,"sum":2.448292863,"avgtime":0.000098021},"msgr_handle_ack_lat":{"avgcount":29642,"sum":0.005843027,"avgtime":0.000000197},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1561,"msgr_send_messages":1558,"msgr_recv_bytes":865269,"msgr_send_bytes":1277426,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.145762700,"msgr_running_send_time":0.064268624,"msgr_running_recv_time":0.032628104,"msgr_running_fast_dispatch_time":0.041167530,"msgr_send_messages_queue_lat":{"avgcount":1557,"sum":0.244606523,"avgtime":0.000157101},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":105808,"msgr_send_encrypted_bytes":7904},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":124,"sum":7.126842763,"avgtime":0.057474538}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5228,"sum":1.123715218,"avgtime":0.000214941}},"mds":{"request":6412,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.602585490,"avgtime":0.001497829},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":4927,"inodes_top":0,"inodes_bottom":4927,"inodes_pin_tail":0,"inodes_pinned":4927,"inodes_expired":0,"inodes_with_caps":1248,"caps":1248,"subtrees":2,"traverse":6418,"traverse_hit":6417,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":685,"ceph_cap_op_grant":7081,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12825,"evex":11110,"evtrm":11110,"ev":1719,"evexg":0,"evexd":11109,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":27813699,"wrpos":28848755,"rdpos":4195685,"jlat":{"avgcount":22,"sum":0.984137748,"avgtime":0.044733534},"replayed":4},"mds_mem":{"ino":1729,"ino+":3214,"ino-":1485,"dir":12,"dir+":13,"dir-":1,"dn":1727,"dn+":6412,"dn-":4685,"cap":1,"cap+":3202,"cap-":3201,"rss":120616,"heap":248164},"mds_server":{"dispatch_client_request":6418,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.174951590,"avgtime":0.000992172},"req_getattr_latency":{"avgcount":6,"sum":0.009656876,"avgtime":0.001609479},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002708406,"avgtime":0.002708406},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.230416875,"avgtime":0.076805625},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.476149395,"avgtime":0.476149395},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.708702348,"avgtime":0.001783969},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5840,"average_load":5840,"avg_session_uptime":827,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":267469,"buffer_anon_items":97,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":7719515,"mds_co_items":112656,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5087,"op_send_bytes":25446552,"op_resend":0,"op_reply":5087,"op_latency":{"avgcount":5087,"sum":227.940302075,"avgtime":0.044808394},"op_inflight":0,"oplen_avg":{"avgcount":5087,"sum":8629},"op":5087,"op_r":53,"op_w":5034,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1726,"osdop_read":25,"osdop_write":39,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3452,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":52,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14760,"get_sum":4102984,"get_or_fail_fail":0,"get_or_fail_success":14760,"take":0,"take_sum":0,"put":14760,"put_sum":4102984,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5087,"take_sum":25771785,"put":5087,"put_sum":25771785,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5087,"take_sum":5087,"put":5087,"put_sum":5087,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":17,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a619c9a820":{"val":1033026,"max":3758096384,"get_started":0,"get":12825,"get_sum":24652255,"get_or_fail_fail":0,"get_or_fail_success":12825,"take":0,"take_sum":0,"put":22,"put_sum":23619229,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:28:22.244 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.03e-07, "sum": 2.06e-07 }, "msgr_recv_bytes": 620968, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 2162, "msgr_running_fast_dispatch_time": 0.057651356, "msgr_running_recv_time": 0.041569853, "msgr_running_send_time": 0.085386597, "msgr_running_total_time": 0.19502124, "msgr_send_bytes": 5925091, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 2162, "msgr_send_messages_queue_lat": { "avgcount": 2161, "avgtime": 0.00091247, "sum": 1.971847818 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29642, "avgtime": 1.97e-07, "sum": 0.005843027 }, "msgr_recv_bytes": 3947828, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 11041, "msgr_running_fast_dispatch_time": 0.046925138, "msgr_running_recv_time": 0.584591273, "msgr_running_send_time": 0.681477184, "msgr_running_total_time": 1.246987663, "msgr_send_bytes": 27810978, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 24978, "msgr_send_messages_queue_lat": { "avgcount": 24977, "avgtime": 9.8021e-05, "sum": 2.448292863 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 865269, "msgr_recv_encrypted_bytes": 105808, "msgr_recv_messages": 1561, "msgr_running_fast_dispatch_time": 0.04116753, "msgr_running_recv_time": 0.032628104, "msgr_running_send_time": 0.064268624, "msgr_running_total_time": 0.1457627, "msgr_send_bytes": 1277426, "msgr_send_encrypted_bytes": 7904, "msgr_send_messages": 1558, "msgr_send_messages_queue_lat": { "avgcount": 1557, "avgtime": 0.000157101, "sum": 0.244606523 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 124, "avgtime": 0.057474538, "sum": 7.126842763 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5228, "avgtime": 0.000214941, "sum": 1.123715218 }, "queue_len": 0 }, "mds": { "caps": 1248, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7081, "ceph_cap_op_revoke": 685, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4927, "inodes_bottom": 4927, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4927, "inodes_top": 0, "inodes_with_caps": 1248, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001497829, "sum": 9.60258549 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6418, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6417, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1719, "evadd": 12825, "evex": 11110, "evexd": 11109, "evexg": 0, "evlrg": 0, "evtrm": 11110, "expos": 27813699, "jlat": { "avgcount": 22, "avgtime": 0.044733534, "sum": 0.984137748 }, "rdpos": 4195685, "replayed": 4, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28848755 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 1727, "dn+": 6412, "dn-": 4685, "heap": 248164, "ino": 1729, "ino+": 3214, "ino-": 1485, "rss": 120616 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6418, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000992172, "sum": 3.17495159 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.001609479, "sum": 0.009656876 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002708406, "sum": 0.002708406 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.076805625, "sum": 0.230416875 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.476149395, "sum": 0.476149395 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001783969, "sum": 5.708702348 } }, "mds_sessions": { "average_load": 5840, "avg_session_uptime": 827, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5840 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 267469, "buffer_anon_items": 97, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 7719515, "mds_co_items": 112656, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 52, "op": 5087, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5087, "avgtime": 0.044808394, "sum": 227.940302075 }, "op_pg": 0, "op_r": 53, "op_reply": 5087, "op_resend": 0, "op_rmw": 0, "op_send": 5087, "op_send_bytes": 25446552, "op_w": 5034, "oplen_avg": { "avgcount": 5087, "sum": 8629 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1726, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 25, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3452, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 39, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14760, "get_or_fail_fail": 0, "get_or_fail_success": 14760, "get_started": 0, "get_sum": 4102984, "max": 104857600, "put": 14760, "put_sum": 4102984, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5087, "put_sum": 25771785, "take": 5087, "take_sum": 25771785, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5087, "put_sum": 5087, "take": 5087, "take_sum": 5087, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 17, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a619c9a820": { "get": 12825, "get_or_fail_fail": 0, "get_or_fail_success": 12825, "get_started": 0, "get_sum": 24652255, "max": 3758096384, "put": 22, "put_sum": 23619229, "take": 0, "take_sum": 0, "val": 1033026, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:28:22.245 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:22.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:22.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:22.353 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:22.353 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:28:22.353 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:22.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:22.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.514+0000 7f4e183e4640 1 Processor -- start 2024-03-06T07:28:22.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.514+0000 7f4e183e4640 1 -- start start 2024-03-06T07:28:22.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e183e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e100d2a30 0x7f4e100d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e183e4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e100d0100 con 0x7f4e100d4c40 2024-03-06T07:28:22.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e183e4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e100cf860 con 0x7f4e1015e100 2024-03-06T07:28:22.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e183e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100cfb40 con 0x7f4e100d2a30 2024-03-06T07:28:22.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1597f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4e100d4c40 0x7f4e100b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57070/0 (socket says 172.21.15.27:57070) 2024-03-06T07:28:22.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1597f640 1 -- 172.21.15.27:0/3812709664 learned_addr learned my addr 172.21.15.27:0/3812709664 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:22.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e16981640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e100d2a30 0x7f4e100d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025470834 0 0) 0x7f4e100d0100 con 0x7f4e100d4c40 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dfc003520 con 0x7f4e100d4c40 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3827682977 0 0) 0x7f4dfc003520 con 0x7f4e100d4c40 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e100d2a30 msgr2=0x7f4e100d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e100d2a30 0x7f4e100d1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 >> v1:172.21.15.69:6789/0 conn(0x7f4e1015e100 legacy=0x7f4e100d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100cfe20 con 0x7f4e100d4c40 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f4e00001670 con 0x7f4e100d4c40 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 >> v1:172.21.15.27:6789/0 conn(0x7f4e100d4c40 legacy=0x7f4e100b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e100d2a30 0x7f4dfc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dfc003f20 0x7f4dfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.515+0000 7f4e1517e640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dfc004d40 0x7f4dfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100cfb40 con 0x7f4dfc003f20 2024-03-06T07:28:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc00ba20 con 0x7f4dfc004d40 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc00bc20 con 0x7f4e100d2a30 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e1597f640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dfc003f20 0x7f4dfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e16180640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e100d2a30 0x7f4dfc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e16981640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dfc004d40 0x7f4dfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e1597f640 1 -- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e100d2a30 msgr2=0x7f4dfc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e1597f640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e100d2a30 0x7f4dfc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e1597f640 1 -- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dfc004d40 msgr2=0x7f4dfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e1597f640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dfc004d40 0x7f4dfc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.516+0000 7f4e1597f640 1 -- 172.21.15.27:0/3812709664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc003520 con 0x7f4dfc003f20 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e1597f640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dfc003f20 0x7f4dfc004800 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000c240 tx=0x7f4dfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2671c6735570e0f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00007d10 con 0x7f4dfc003f20 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e00030070 con 0x7f4dfc003f20 2024-03-06T07:28:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e1517e640 1 -- 172.21.15.27:0/3812709664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00005d70 con 0x7f4dfc003f20 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e183e4640 1 -- 172.21.15.27:0/3812709664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dfc003f20 msgr2=0x7f4dfc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e183e4640 1 --2- 172.21.15.27:0/3812709664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dfc003f20 0x7f4dfc004800 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000c240 tx=0x7f4dfc005830 comp rx=0 tx=0).stop 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e16981640 1 -- 172.21.15.27:0/3812709664 reap_dead start 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e183e4640 1 -- 172.21.15.27:0/3812709664 shutdown_connections 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e183e4640 1 -- 172.21.15.27:0/3812709664 >> 172.21.15.27:0/3812709664 conn(0x7f4e100d3ef0 msgr2=0x7f4e100d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.517+0000 7f4e183e4640 1 -- 172.21.15.27:0/3812709664 shutdown_connections 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.518+0000 7f4e183e4640 1 -- 172.21.15.27:0/3812709664 wait complete. 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.519+0000 7f4e183e4640 1 Processor -- start 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.519+0000 7f4e183e4640 1 -- start start 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.519+0000 7f4e183e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dfc003f20 0x7f4e1009f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.519+0000 7f4e183e4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1009e8e0 0x7f4e100b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.519+0000 7f4e183e4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e101f6a00 0x7f4e100b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.519+0000 7f4e183e4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1015e4e0 con 0x7f4e1009e8e0 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.519+0000 7f4e183e4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100cf510 con 0x7f4e101f6a00 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.519+0000 7f4e183e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1015e660 con 0x7f4dfc003f20 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.519+0000 7f4e1597f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1009e8e0 0x7f4e100b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e1597f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1009e8e0 0x7f4e100b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47818/0 (socket says 172.21.15.27:47818) 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e1597f640 1 -- 172.21.15.27:0/3846526187 learned_addr learned my addr 172.21.15.27:0/3846526187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e16981640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e101f6a00 0x7f4e100b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e16180640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dfc003f20 0x7f4e1009f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e1597f640 1 -- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dfc003f20 msgr2=0x7f4e1009f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e1597f640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dfc003f20 0x7f4e1009f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e1597f640 1 -- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e101f6a00 msgr2=0x7f4e100b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e1597f640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e101f6a00 0x7f4e100b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e1597f640 1 -- 172.21.15.27:0/3846526187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1014c230 con 0x7f4e1009e8e0 2024-03-06T07:28:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e16981640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e101f6a00 0x7f4e100b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e1597f640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1009e8e0 0x7f4e100b34e0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000c2a0 tx=0x7f4e100cfff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e06ffd640 1 -- 172.21.15.27:0/3846526187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00046070 con 0x7f4e1009e8e0 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e06ffd640 1 -- 172.21.15.27:0/3846526187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e00004bf0 con 0x7f4e1009e8e0 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e100b3a20 con 0x7f4e1009e8e0 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e06ffd640 1 -- 172.21.15.27:0/3846526187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00007d10 con 0x7f4e1009e8e0 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.520+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1014c7d0 con 0x7f4e1009e8e0 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.522+0000 7f4e06ffd640 1 -- 172.21.15.27:0/3846526187 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e00007070 con 0x7f4e1009e8e0 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.522+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e100752d0 con 0x7f4e1009e8e0 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.522+0000 7f4e06ffd640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4df406b500 0x7f4df406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.522+0000 7f4e06ffd640 1 -- 172.21.15.27:0/3846526187 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4e000ab2c0 con 0x7f4e1009e8e0 2024-03-06T07:28:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.523+0000 7f4e16180640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4df406b500 0x7f4df406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.523+0000 7f4e16180640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4df406b500 0x7f4df406d9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4e1014c100 tx=0x7f4e0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.527+0000 7f4e06ffd640 1 -- 172.21.15.27:0/3846526187 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e00079c00 con 0x7f4e1009e8e0 2024-03-06T07:28:22.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.710+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e1015e960 con 0x7f4e1009e8e0 2024-03-06T07:28:22.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.711+0000 7f4e06ffd640 1 -- 172.21.15.27:0/3846526187 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f4e0007dca0 con 0x7f4e1009e8e0 2024-03-06T07:28:22.713 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:22.713 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:57.839900+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:46.443375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:47.116887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/3147032864":"2024-03-07T07:27:57.839873+0000","172.21.15.27:6833/3147032864":"2024-03-07T07:27:57.839873+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4df406b500 msgr2=0x7f4df406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4df406b500 0x7f4df406d9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4e1014c100 tx=0x7f4e0c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1009e8e0 msgr2=0x7f4e100b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1009e8e0 0x7f4e100b34e0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000c2a0 tx=0x7f4e100cfff0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 shutdown_connections 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4df406b500 0x7f4df406d9c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e101f6a00 0x7f4e100b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1009e8e0 0x7f4e100b34e0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 --2- 172.21.15.27:0/3846526187 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dfc003f20 0x7f4e1009f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 >> 172.21.15.27:0/3846526187 conn(0x7f4e100d3ef0 msgr2=0x7f4e1015eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 shutdown_connections 2024-03-06T07:28:22.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.713+0000 7f4e183e4640 1 -- 172.21.15.27:0/3846526187 wait complete. 2024-03-06T07:28:22.725 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:46.443375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:47.116887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:22.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:22.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.835+0000 7f5cec11e640 1 Processor -- start 2024-03-06T07:28:22.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.835+0000 7f5cec11e640 1 -- start start 2024-03-06T07:28:22.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5cec11e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ce40d2c10 0x7f5ce40b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5cec11e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d1080 0x7f5ce40b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5cec11e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce40d4e20 con 0x7f5ce415e460 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5cec11e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40b3410 con 0x7f5ce40d2c10 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5cec11e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40b36f0 con 0x7f5ce40d1080 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5ce96b9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ce40d2c10 0x7f5ce40b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5ce9eba640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5ce415e460 0x7f5ce40d2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57076/0 (socket says 172.21.15.27:57076) 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5ce9eba640 1 -- 172.21.15.27:0/2612062829 learned_addr learned my addr 172.21.15.27:0/2612062829 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5ce96b9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ce40d2c10 0x7f5ce40b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47820/0 (socket says 172.21.15.27:47820) 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5cea6bb640 1 --2- 172.21.15.27:0/2612062829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d1080 0x7f5ce40b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5ce96b9640 1 -- 172.21.15.27:0/2612062829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d1080 msgr2=0x7f5ce40b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5ce96b9640 1 --2- 172.21.15.27:0/2612062829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d1080 0x7f5ce40b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5ce96b9640 1 -- 172.21.15.27:0/2612062829 >> v1:172.21.15.27:6789/0 conn(0x7f5ce415e460 legacy=0x7f5ce40d2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5ce8eb8640 1 -- 172.21.15.27:0/2612062829 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211741437 0 0) 0x7f5ce40d4e20 con 0x7f5ce415e460 2024-03-06T07:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.836+0000 7f5ce96b9640 1 -- 172.21.15.27:0/2612062829 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40b39d0 con 0x7f5ce40d2c10 2024-03-06T07:28:22.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.837+0000 7f5ce96b9640 1 --2- 172.21.15.27:0/2612062829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d2c10 0x7f5ce40b41d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097a0 tx=0x7f5cd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbe85a18d754870 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.837+0000 7f5ce8eb8640 1 -- 172.21.15.27:0/2612062829 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4035020 con 0x7f5ce40d2c10 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.837+0000 7f5ce8eb8640 1 -- 172.21.15.27:0/2612062829 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd4031830 con 0x7f5ce40d2c10 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.837+0000 7f5ce8eb8640 1 -- 172.21.15.27:0/2612062829 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4031c00 con 0x7f5ce40d2c10 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.838+0000 7f5cec11e640 1 -- 172.21.15.27:0/2612062829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d2c10 msgr2=0x7f5ce40b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.838+0000 7f5cec11e640 1 --2- 172.21.15.27:0/2612062829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d2c10 0x7f5ce40b41d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097a0 tx=0x7f5cd402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.838+0000 7f5cec11e640 1 -- 172.21.15.27:0/2612062829 shutdown_connections 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.838+0000 7f5cec11e640 1 --2- 172.21.15.27:0/2612062829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d1080 0x7f5ce40b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.838+0000 7f5cec11e640 1 --2- 172.21.15.27:0/2612062829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d2c10 0x7f5ce40b41d0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.838+0000 7f5cec11e640 1 -- 172.21.15.27:0/2612062829 >> 172.21.15.27:0/2612062829 conn(0x7f5ce40d3f20 msgr2=0x7f5ce40d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.838+0000 7f5cec11e640 1 -- 172.21.15.27:0/2612062829 shutdown_connections 2024-03-06T07:28:22.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.838+0000 7f5cec11e640 1 -- 172.21.15.27:0/2612062829 wait complete. 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.839+0000 7f5cec11e640 1 Processor -- start 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.839+0000 7f5cec11e640 1 -- start start 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5cec11e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d1080 0x7f5ce415b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5cec11e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d2c10 0x7f5ce40c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5cec11e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ce415e460 0x7f5ce41611e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5cec11e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40d4bc0 con 0x7f5ce40d1080 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5cec11e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40d4a40 con 0x7f5ce415e460 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5cec11e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40d1d00 con 0x7f5ce40d2c10 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce9eba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d1080 0x7f5ce415b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce9eba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d1080 0x7f5ce415b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47830/0 (socket says 172.21.15.27:47830) 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce9eba640 1 -- 172.21.15.27:0/453071225 learned_addr learned my addr 172.21.15.27:0/453071225 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5cea6bb640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ce415e460 0x7f5ce41611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce96b9640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d2c10 0x7f5ce40c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce9eba640 1 -- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d2c10 msgr2=0x7f5ce40c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce9eba640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d2c10 0x7f5ce40c7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce9eba640 1 -- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ce415e460 msgr2=0x7f5ce41611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce9eba640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ce415e460 0x7f5ce41611e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce9eba640 1 -- 172.21.15.27:0/453071225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40c8150 con 0x7f5ce40d1080 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce96b9640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d2c10 0x7f5ce40c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5cea6bb640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ce415e460 0x7f5ce41611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.840+0000 7f5ce9eba640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d1080 0x7f5ce415b200 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c400 tx=0x7f5ce000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.841+0000 7f5cda7fc640 1 -- 172.21.15.27:0/453071225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0017070 con 0x7f5ce40d1080 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.841+0000 7f5cda7fc640 1 -- 172.21.15.27:0/453071225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce0005ce0 con 0x7f5ce40d1080 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.841+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce40c8430 con 0x7f5ce40d1080 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.841+0000 7f5cda7fc640 1 -- 172.21.15.27:0/453071225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0007310 con 0x7f5ce40d1080 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.841+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce40c89f0 con 0x7f5ce40d1080 2024-03-06T07:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.841+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce4068760 con 0x7f5ce40d1080 2024-03-06T07:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.843+0000 7f5cda7fc640 1 -- 172.21.15.27:0/453071225 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ce001a070 con 0x7f5ce40d1080 2024-03-06T07:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.843+0000 7f5cda7fc640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cc806b4d0 0x7f5cc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.843+0000 7f5cda7fc640 1 -- 172.21.15.27:0/453071225 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5ce0089720 con 0x7f5ce40d1080 2024-03-06T07:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.843+0000 7f5ce96b9640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cc806b4d0 0x7f5cc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.844+0000 7f5ce96b9640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cc806b4d0 0x7f5cc806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4036000 tx=0x7f5cd4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:22.848+0000 7f5cda7fc640 1 -- 172.21.15.27:0/453071225 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ce00584c0 con 0x7f5ce40d1080 2024-03-06T07:28:23.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.028+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ce40b9ad0 con 0x7f5ce40d1080 2024-03-06T07:28:23.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.029+0000 7f5cda7fc640 1 -- 172.21.15.27:0/453071225 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ce0057dc0 con 0x7f5ce40d1080 2024-03-06T07:28:23.030 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:28:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cc806b4d0 msgr2=0x7f5cc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cc806b4d0 0x7f5cc806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4036000 tx=0x7f5cd4008030 comp rx=0 tx=0).stop 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d1080 msgr2=0x7f5ce415b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d1080 0x7f5ce415b200 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c400 tx=0x7f5ce000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 shutdown_connections 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cc806b4d0 0x7f5cc806d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ce415e460 0x7f5ce41611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ce40d2c10 0x7f5ce40c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 --2- 172.21.15.27:0/453071225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce40d1080 0x7f5ce415b200 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 >> 172.21.15.27:0/453071225 conn(0x7f5ce40d3f20 msgr2=0x7f5ce40b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 shutdown_connections 2024-03-06T07:28:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.031+0000 7f5cec11e640 1 -- 172.21.15.27:0/453071225 wait complete. 2024-03-06T07:28:23.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:23.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.152+0000 7f02bd7a0640 1 Processor -- start 2024-03-06T07:28:23.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.153+0000 7f02bd7a0640 1 -- start start 2024-03-06T07:28:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.153+0000 7f02bd7a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b80d4830 0x7f02b80d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.153+0000 7f02bd7a0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b80b3440 con 0x7f02b80d2a00 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.153+0000 7f02bd7a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b80b3720 con 0x7f02b815e120 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.153+0000 7f02bd7a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80b31f0 con 0x7f02b80d4830 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.153+0000 7f02b77fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f02b80d2a00 0x7f02b80b4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57092/0 (socket says 172.21.15.27:57092) 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.153+0000 7f02b77fe640 1 -- 172.21.15.27:0/1925956714 learned_addr learned my addr 172.21.15.27:0/1925956714 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02bcf9f640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b80d4830 0x7f02b80d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654009970 0 0) 0x7f02b80b3440 con 0x7f02b80d2a00 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a0003520 con 0x7f02b80d2a00 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 711805790 0 0) 0x7f02a0003520 con 0x7f02b80d2a00 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b80d4830 msgr2=0x7f02b80d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b80d4830 0x7f02b80d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 >> v1:172.21.15.69:6789/0 conn(0x7f02b815e120 legacy=0x7f02b80d22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80b3a00 con 0x7f02b80d2a00 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f02a8001670 con 0x7f02b80d2a00 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 >> v1:172.21.15.27:6789/0 conn(0x7f02b80d2a00 legacy=0x7f02b80b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a00039e0 0x7f02a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02a0004390 0x7f02a0004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b80d4830 0x7f02a0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80b31f0 con 0x7f02a0004390 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a000ba70 con 0x7f02a00039e0 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a000bc70 con 0x7f02b80d4830 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b77fe640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02a0004390 0x7f02a0004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02b7fff640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a00039e0 0x7f02a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.154+0000 7f02bcf9f640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b80d4830 0x7f02a0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.155+0000 7f02b77fe640 1 -- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b80d4830 msgr2=0x7f02a0005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.155+0000 7f02b77fe640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b80d4830 0x7f02a0005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.155+0000 7f02b77fe640 1 -- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a00039e0 msgr2=0x7f02a0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.155+0000 7f02b77fe640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a00039e0 0x7f02a0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.155+0000 7f02b77fe640 1 -- 172.21.15.27:0/1925956714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a0003520 con 0x7f02a0004390 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.155+0000 7f02b77fe640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02a0004390 0x7f02a0004860 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f02a800c290 tx=0x7f02a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44979bef689fad0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8007d50 con 0x7f02a0004390 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a8030070 con 0x7f02a0004390 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02b6ffd640 1 -- 172.21.15.27:0/1925956714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a802cd60 con 0x7f02a0004390 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1925956714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02a0004390 msgr2=0x7f02a0004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02bd7a0640 1 --2- 172.21.15.27:0/1925956714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02a0004390 0x7f02a0004860 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f02a800c290 tx=0x7f02a0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02bcf9f640 1 -- 172.21.15.27:0/1925956714 reap_dead start 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1925956714 shutdown_connections 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1925956714 >> 172.21.15.27:0/1925956714 conn(0x7f02b80d3d10 msgr2=0x7f02b80d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1925956714 shutdown_connections 2024-03-06T07:28:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.156+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1925956714 wait complete. 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.157+0000 7f02bd7a0640 1 Processor -- start 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02bd7a0640 1 -- start start 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02bd7a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02a0004390 0x7f02b81f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02bd7a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b814aa60 0x7f02b81f7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02bd7a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b80c7840 0x7f02b80c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02bd7a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80d1af0 con 0x7f02b814aa60 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02bd7a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b815e500 con 0x7f02b80c7840 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02bd7a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80d1c70 con 0x7f02a0004390 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b814aa60 0x7f02b81f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b814aa60 0x7f02b81f7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47846/0 (socket says 172.21.15.27:47846) 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.158+0000 7f02b77fe640 1 -- 172.21.15.27:0/1495928244 learned_addr learned my addr 172.21.15.27:0/1495928244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02bcf9f640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b80c7840 0x7f02b80c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b7fff640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02a0004390 0x7f02b81f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b77fe640 1 -- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02a0004390 msgr2=0x7f02b81f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b77fe640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02a0004390 0x7f02b81f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b77fe640 1 -- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b80c7840 msgr2=0x7f02b80c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b77fe640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b80c7840 0x7f02b80c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b77fe640 1 -- 172.21.15.27:0/1495928244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80c99e0 con 0x7f02b814aa60 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02bcf9f640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b80c7840 0x7f02b80c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b7fff640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02a0004390 0x7f02b81f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b77fe640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b814aa60 0x7f02b81f7690 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f02a8000f60 tx=0x7f02a8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b4ff9640 1 -- 172.21.15.27:0/1495928244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8007d50 con 0x7f02b814aa60 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b4ff9640 1 -- 172.21.15.27:0/1495928244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80049e0 con 0x7f02b814aa60 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b80c9cc0 con 0x7f02b814aa60 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02b4ff9640 1 -- 172.21.15.27:0/1495928244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8004dd0 con 0x7f02b814aa60 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.159+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b80ca280 con 0x7f02b814aa60 2024-03-06T07:28:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.160+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b8075260 con 0x7f02b814aa60 2024-03-06T07:28:23.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.161+0000 7f02b4ff9640 1 -- 172.21.15.27:0/1495928244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02a8007070 con 0x7f02b814aa60 2024-03-06T07:28:23.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.161+0000 7f02b4ff9640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a406b4d0 0x7f02a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.161+0000 7f02b4ff9640 1 -- 172.21.15.27:0/1495928244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f02a80ab5a0 con 0x7f02b814aa60 2024-03-06T07:28:23.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.162+0000 7f02b7fff640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a406b4d0 0x7f02a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.162+0000 7f02b7fff640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a406b4d0 0x7f02a406d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f02b80ce2d0 tx=0x7f02ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.166+0000 7f02b4ff9640 1 -- 172.21.15.27:0/1495928244 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02a807a3d0 con 0x7f02b814aa60 2024-03-06T07:28:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.348+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f02b80cf960 con 0x7f02b814aa60 2024-03-06T07:28:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.348+0000 7f02b4ff9640 1 -- 172.21.15.27:0/1495928244 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7f02a8079cd0 con 0x7f02b814aa60 2024-03-06T07:28:23.353 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:28:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a406b4d0 msgr2=0x7f02a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a406b4d0 0x7f02a406d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f02b80ce2d0 tx=0x7f02ac009040 comp rx=0 tx=0).stop 2024-03-06T07:28:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b814aa60 msgr2=0x7f02b81f7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b814aa60 0x7f02b81f7690 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f02a8000f60 tx=0x7f02a8001a40 comp rx=0 tx=0).stop 2024-03-06T07:28:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 shutdown_connections 2024-03-06T07:28:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a406b4d0 0x7f02a406d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b80c7840 0x7f02b80c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b814aa60 0x7f02b81f7690 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 --2- 172.21.15.27:0/1495928244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02a0004390 0x7f02b81f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 >> 172.21.15.27:0/1495928244 conn(0x7f02b80d3d10 msgr2=0x7f02b80d22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 shutdown_connections 2024-03-06T07:28:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.351+0000 7f02bd7a0640 1 -- 172.21.15.27:0/1495928244 wait complete. 2024-03-06T07:28:23.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.471+0000 7f065b26d640 1 Processor -- start 2024-03-06T07:28:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.471+0000 7f065b26d640 1 -- start start 2024-03-06T07:28:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f065b26d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06540cec20 0x7f06540cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f065b26d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f06540d0c40 0x7f065406c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f065b26d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06540b24a0 con 0x7f06540b2010 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f065b26d640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06540b2780 con 0x7f06540cec20 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f065b26d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06540b2a60 con 0x7f06540d0c40 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f0659009640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06540cec20 0x7f06540cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f0659009640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06540cec20 0x7f06540cfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47850/0 (socket says 172.21.15.27:47850) 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f0659009640 1 -- 172.21.15.27:0/1898032123 learned_addr learned my addr 172.21.15.27:0/1898032123 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f065980a640 1 --2- 172.21.15.27:0/1898032123 >> v2:172.21.15.69:3300/0 conn(0x7f06540d0c40 0x7f065406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f0659009640 1 -- 172.21.15.27:0/1898032123 >> v2:172.21.15.69:3300/0 conn(0x7f06540d0c40 msgr2=0x7f065406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f0659009640 1 --2- 172.21.15.27:0/1898032123 >> v2:172.21.15.69:3300/0 conn(0x7f06540d0c40 0x7f065406c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f0659009640 1 -- 172.21.15.27:0/1898032123 >> v1:172.21.15.69:6789/0 conn(0x7f06540b2010 legacy=0x7f06540d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.472+0000 7f0659009640 1 -- 172.21.15.27:0/1898032123 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06540b2d40 con 0x7f06540cec20 2024-03-06T07:28:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.473+0000 7f0659009640 1 --2- 172.21.15.27:0/1898032123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540cec20 0x7f06540cfff0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f0654161e60 tx=0x7f065002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3e8a401ed513f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.473+0000 7f064bfff640 1 -- 172.21.15.27:0/1898032123 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065002ca80 con 0x7f06540cec20 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.473+0000 7f064bfff640 1 -- 172.21.15.27:0/1898032123 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065002cc20 con 0x7f06540cec20 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.473+0000 7f064bfff640 1 -- 172.21.15.27:0/1898032123 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06500343f0 con 0x7f06540cec20 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.474+0000 7f065b26d640 1 -- 172.21.15.27:0/1898032123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540cec20 msgr2=0x7f06540cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.474+0000 7f065b26d640 1 --2- 172.21.15.27:0/1898032123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540cec20 0x7f06540cfff0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f0654161e60 tx=0x7f065002bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.474+0000 7f065b26d640 1 -- 172.21.15.27:0/1898032123 shutdown_connections 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.474+0000 7f065b26d640 1 --2- 172.21.15.27:0/1898032123 >> v2:172.21.15.69:3300/0 conn(0x7f06540d0c40 0x7f065406c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.474+0000 7f065b26d640 1 --2- 172.21.15.27:0/1898032123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540cec20 0x7f06540cfff0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.474+0000 7f065b26d640 1 -- 172.21.15.27:0/1898032123 >> 172.21.15.27:0/1898032123 conn(0x7f06540d1cc0 msgr2=0x7f06540d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.474+0000 7f065b26d640 1 -- 172.21.15.27:0/1898032123 shutdown_connections 2024-03-06T07:28:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.474+0000 7f065b26d640 1 -- 172.21.15.27:0/1898032123 wait complete. 2024-03-06T07:28:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.475+0000 7f065b26d640 1 Processor -- start 2024-03-06T07:28:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.475+0000 7f065b26d640 1 -- start start 2024-03-06T07:28:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f065b26d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06540cec20 0x7f0654157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f065b26d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540d0c40 0x7f065415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f065b26d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06540c8140 0x7f0654156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f065b26d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540cf000 con 0x7f06540d0c40 2024-03-06T07:28:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f065b26d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540af9c0 con 0x7f06540cec20 2024-03-06T07:28:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f065b26d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06540b3540 con 0x7f06540c8140 2024-03-06T07:28:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0658808640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540d0c40 0x7f065415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0658808640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540d0c40 0x7f065415e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47860/0 (socket says 172.21.15.27:47860) 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0658808640 1 -- 172.21.15.27:0/2319996632 learned_addr learned my addr 172.21.15.27:0/2319996632 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0659009640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06540cec20 0x7f0654157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f065980a640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06540c8140 0x7f0654156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0658808640 1 -- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06540c8140 msgr2=0x7f0654156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0658808640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06540c8140 0x7f0654156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0658808640 1 -- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06540cec20 msgr2=0x7f0654157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0658808640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06540cec20 0x7f0654157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0658808640 1 -- 172.21.15.27:0/2319996632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06540b5940 con 0x7f06540d0c40 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0658808640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540d0c40 0x7f065415e9e0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f064400b3f0 tx=0x7f064400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.476+0000 7f0649ffb640 1 -- 172.21.15.27:0/2319996632 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644015020 con 0x7f06540d0c40 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.477+0000 7f0649ffb640 1 -- 172.21.15.27:0/2319996632 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0644011830 con 0x7f06540d0c40 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.477+0000 7f0649ffb640 1 -- 172.21.15.27:0/2319996632 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644011c20 con 0x7f06540d0c40 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.477+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065415f0c0 con 0x7f06540d0c40 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.477+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06540b5be0 con 0x7f06540d0c40 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.478+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0654068760 con 0x7f06540d0c40 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.478+0000 7f0649ffb640 1 -- 172.21.15.27:0/2319996632 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0644018020 con 0x7f06540d0c40 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.479+0000 7f0649ffb640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f063806b4d0 0x7f063806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.479+0000 7f0649ffb640 1 -- 172.21.15.27:0/2319996632 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0644089d40 con 0x7f06540d0c40 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.479+0000 7f0659009640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f063806b4d0 0x7f063806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.479+0000 7f0659009640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f063806b4d0 0x7f063806d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0650035000 tx=0x7f0650030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.488+0000 7f0649ffb640 1 -- 172.21.15.27:0/2319996632 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f064401f030 con 0x7f06540d0c40 2024-03-06T07:28:23.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.672+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f06540b5db0 con 0x7f06540d0c40 2024-03-06T07:28:23.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.673+0000 7f0649ffb640 1 -- 172.21.15.27:0/2319996632 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0644058510 con 0x7f06540d0c40 2024-03-06T07:28:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:28:23.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f063806b4d0 msgr2=0x7f063806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f063806b4d0 0x7f063806d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0650035000 tx=0x7f0650030000 comp rx=0 tx=0).stop 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540d0c40 msgr2=0x7f065415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540d0c40 0x7f065415e9e0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f064400b3f0 tx=0x7f064400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 shutdown_connections 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f063806b4d0 0x7f063806d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06540c8140 0x7f0654156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06540d0c40 0x7f065415e9e0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 --2- 172.21.15.27:0/2319996632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06540cec20 0x7f0654157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 >> 172.21.15.27:0/2319996632 conn(0x7f06540d1cc0 msgr2=0x7f06540b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 shutdown_connections 2024-03-06T07:28:23.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.675+0000 7f065b26d640 1 -- 172.21.15.27:0/2319996632 wait complete. 2024-03-06T07:28:23.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:23.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.798+0000 7ff429c78640 1 Processor -- start 2024-03-06T07:28:23.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.798+0000 7ff429c78640 1 -- start start 2024-03-06T07:28:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.799+0000 7ff429c78640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff42406d0d0 0x7ff4240cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.799+0000 7ff429c78640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42406c6c0 con 0x7ff4240d27c0 2024-03-06T07:28:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.799+0000 7ff429c78640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4240cc2f0 con 0x7ff4240b7710 2024-03-06T07:28:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.799+0000 7ff429c78640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4240ccc20 con 0x7ff42406d0d0 2024-03-06T07:28:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.799+0000 7ff422ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff42406d0d0 0x7ff4240cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.799+0000 7ff423fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff4240d27c0 0x7ff4240d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46100/0 (socket says 172.21.15.27:46100) 2024-03-06T07:28:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.799+0000 7ff423fff640 1 -- 172.21.15.27:0/3124265324 learned_addr learned my addr 172.21.15.27:0/3124265324 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.800+0000 7ff422ffd640 1 -- 172.21.15.27:0/3124265324 >> v1:172.21.15.69:6789/0 conn(0x7ff4240b7710 legacy=0x7ff42406c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.800+0000 7ff422ffd640 1 -- 172.21.15.27:0/3124265324 >> v1:172.21.15.27:6789/0 conn(0x7ff4240d27c0 legacy=0x7ff4240d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.800+0000 7ff422ffd640 1 -- 172.21.15.27:0/3124265324 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4240cc5d0 con 0x7ff42406d0d0 2024-03-06T07:28:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.800+0000 7ff4227fc640 1 -- 172.21.15.27:0/3124265324 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299004119 0 0) 0x7ff42406c6c0 con 0x7ff4240d27c0 2024-03-06T07:28:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.800+0000 7ff422ffd640 1 --2- 172.21.15.27:0/3124265324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff42406d0d0 0x7ff4240cffd0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff4140097a0 tx=0x7ff41402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdce96fa94fdec09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff4227fc640 1 -- 172.21.15.27:0/3124265324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414034070 con 0x7ff42406d0d0 2024-03-06T07:28:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff4227fc640 1 -- 172.21.15.27:0/3124265324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff414030610 con 0x7ff42406d0d0 2024-03-06T07:28:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff4227fc640 1 -- 172.21.15.27:0/3124265324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4140309e0 con 0x7ff42406d0d0 2024-03-06T07:28:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff429c78640 1 -- 172.21.15.27:0/3124265324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff42406d0d0 msgr2=0x7ff4240cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff429c78640 1 --2- 172.21.15.27:0/3124265324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff42406d0d0 0x7ff4240cffd0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff4140097a0 tx=0x7ff41402b960 comp rx=0 tx=0).stop 2024-03-06T07:28:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff429c78640 1 -- 172.21.15.27:0/3124265324 shutdown_connections 2024-03-06T07:28:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff429c78640 1 --2- 172.21.15.27:0/3124265324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff42406d0d0 0x7ff4240cffd0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff429c78640 1 -- 172.21.15.27:0/3124265324 >> 172.21.15.27:0/3124265324 conn(0x7ff4240d1ca0 msgr2=0x7ff4240d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff429c78640 1 -- 172.21.15.27:0/3124265324 shutdown_connections 2024-03-06T07:28:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.801+0000 7ff429c78640 1 -- 172.21.15.27:0/3124265324 wait complete. 2024-03-06T07:28:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.803+0000 7ff429c78640 1 Processor -- start 2024-03-06T07:28:23.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.804+0000 7ff429c78640 1 -- start start 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff429c78640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff42406d0d0 0x7ff424157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff429c78640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240b7710 0x7ff4240b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff429c78640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4240d27c0 0x7ff4240b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff429c78640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240b7af0 con 0x7ff4240b7710 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff429c78640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240ce0c0 con 0x7ff4240d27c0 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff429c78640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42406c370 con 0x7ff42406d0d0 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff422ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240b7710 0x7ff4240b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff422ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240b7710 0x7ff4240b3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60028/0 (socket says 172.21.15.27:60028) 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff422ffd640 1 -- 172.21.15.27:0/1609917346 learned_addr learned my addr 172.21.15.27:0/1609917346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff423fff640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4240d27c0 0x7ff4240b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff4237fe640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff42406d0d0 0x7ff424157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff422ffd640 1 -- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff42406d0d0 msgr2=0x7ff424157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff422ffd640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff42406d0d0 0x7ff424157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff422ffd640 1 -- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4240d27c0 msgr2=0x7ff4240b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff422ffd640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4240d27c0 0x7ff4240b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.805+0000 7ff422ffd640 1 -- 172.21.15.27:0/1609917346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42415fc50 con 0x7ff4240b7710 2024-03-06T07:28:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.806+0000 7ff422ffd640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240b7710 0x7ff4240b3620 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7ff4140098c0 tx=0x7ff4140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.806+0000 7ff41bfff640 1 -- 172.21.15.27:0/1609917346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414034040 con 0x7ff4240b7710 2024-03-06T07:28:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.806+0000 7ff41bfff640 1 -- 172.21.15.27:0/1609917346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff414039a10 con 0x7ff4240b7710 2024-03-06T07:28:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.806+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42415ff30 con 0x7ff4240b7710 2024-03-06T07:28:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.806+0000 7ff41bfff640 1 -- 172.21.15.27:0/1609917346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414039de0 con 0x7ff4240b7710 2024-03-06T07:28:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.806+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4241604f0 con 0x7ff4240b7710 2024-03-06T07:28:23.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.807+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4240752d0 con 0x7ff4240b7710 2024-03-06T07:28:23.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.807+0000 7ff41bfff640 1 -- 172.21.15.27:0/1609917346 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff414037070 con 0x7ff4240b7710 2024-03-06T07:28:23.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.808+0000 7ff41bfff640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff40406b4d0 0x7ff40406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.808+0000 7ff41bfff640 1 -- 172.21.15.27:0/1609917346 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff4140ac1a0 con 0x7ff4240b7710 2024-03-06T07:28:23.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.808+0000 7ff4237fe640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff40406b4d0 0x7ff40406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.809+0000 7ff4237fe640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff40406b4d0 0x7ff40406d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff40c000f60 tx=0x7ff40c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:23.812+0000 7ff41bfff640 1 -- 172.21.15.27:0/1609917346 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff41407aa00 con 0x7ff4240b7710 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.007+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff4240d23c0 con 0x7ff4240b7710 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.007+0000 7ff41bfff640 1 -- 172.21.15.27:0/1609917346 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v102) v1 ==== 80+0+122 (secure 0 0 0) 0x7ff41407eaa0 con 0x7ff4240b7710 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6835/3147032864 2024-03-07T07:27:57.839873+0000 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6833/3147032864 2024-03-07T07:27:57.839873+0000 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-03-06T07:28:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff40406b4d0 msgr2=0x7ff40406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff40406b4d0 0x7ff40406d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff40c000f60 tx=0x7ff40c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240b7710 msgr2=0x7ff4240b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240b7710 0x7ff4240b3620 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7ff4140098c0 tx=0x7ff4140046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 shutdown_connections 2024-03-06T07:28:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff40406b4d0 0x7ff40406d990 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4240d27c0 0x7ff4240b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240b7710 0x7ff4240b3620 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 --2- 172.21.15.27:0/1609917346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff42406d0d0 0x7ff424157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 >> 172.21.15.27:0/1609917346 conn(0x7ff4240d1ca0 msgr2=0x7ff4240b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 shutdown_connections 2024-03-06T07:28:24.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.009+0000 7ff429c78640 1 -- 172.21.15.27:0/1609917346 wait complete. 2024-03-06T07:28:24.021 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:28:24.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:24.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[68919]: fuse finished with error 0 and tester_r 0 2024-03-06T07:28:24.052 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:28:24.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.052 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:24.121 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:28:24.121 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:24.138 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:24.138 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:28:24.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.139 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:28:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:24.194 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:28:24.195 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:28:24.195 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:24.210 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:24.210 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:28:24.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.210 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:28:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:24.265 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:28:24.265 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:28:24.266 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:24.282 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:24.282 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:28:24.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.282 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:28:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:24.338 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:28:24.338 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:24.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.443+0000 7f0b095e9640 1 Processor -- start 2024-03-06T07:28:24.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.443+0000 7f0b095e9640 1 -- start start 2024-03-06T07:28:24.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.444+0000 7f0b095e9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b040b5030 0x7f0b040ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.444+0000 7f0b095e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b040b6030 0x7f0b040ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.444+0000 7f0b095e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b040b5b80 con 0x7f0b0415f7d0 2024-03-06T07:28:24.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.444+0000 7f0b095e9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b040b1d20 con 0x7f0b040b5030 2024-03-06T07:28:24.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.444+0000 7f0b095e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b040b55b0 con 0x7f0b040b6030 2024-03-06T07:28:24.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.444+0000 7f0afa7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b040b5030 0x7f0b040ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.444+0000 7f0b02ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0b0415f7d0 0x7f0b0406aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46112/0 (socket says 172.21.15.27:46112) 2024-03-06T07:28:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.444+0000 7f0b02ffd640 1 -- 172.21.15.27:0/1815023036 learned_addr learned my addr 172.21.15.27:0/1815023036 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.444+0000 7f0b037fe640 1 --2- 172.21.15.27:0/1815023036 >> v2:172.21.15.69:3300/0 conn(0x7f0b040b6030 0x7f0b040ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.445+0000 7f0afa7fc640 1 -- 172.21.15.27:0/1815023036 >> v2:172.21.15.69:3300/0 conn(0x7f0b040b6030 msgr2=0x7f0b040ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.445+0000 7f0afa7fc640 1 --2- 172.21.15.27:0/1815023036 >> v2:172.21.15.69:3300/0 conn(0x7f0b040b6030 0x7f0b040ad6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.445+0000 7f0afa7fc640 1 -- 172.21.15.27:0/1815023036 >> v1:172.21.15.27:6789/0 conn(0x7f0b0415f7d0 legacy=0x7f0b0406aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.445+0000 7f0afa7fc640 1 -- 172.21.15.27:0/1815023036 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b040732f0 con 0x7f0b040b5030 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.445+0000 7f0afa7fc640 1 --2- 172.21.15.27:0/1815023036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b5030 0x7f0b040ad1b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f0b040c3c10 tx=0x7f0aec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6dc7f963112e92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.445+0000 7f0b027fc640 1 -- 172.21.15.27:0/1815023036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec034070 con 0x7f0b040b5030 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b027fc640 1 -- 172.21.15.27:0/1815023036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aec030610 con 0x7f0b040b5030 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b027fc640 1 -- 172.21.15.27:0/1815023036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec0309e0 con 0x7f0b040b5030 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b095e9640 1 -- 172.21.15.27:0/1815023036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b5030 msgr2=0x7f0b040ad1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b095e9640 1 --2- 172.21.15.27:0/1815023036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b5030 0x7f0b040ad1b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f0b040c3c10 tx=0x7f0aec02b960 comp rx=0 tx=0).stop 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b095e9640 1 -- 172.21.15.27:0/1815023036 shutdown_connections 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b095e9640 1 --2- 172.21.15.27:0/1815023036 >> v2:172.21.15.69:3300/0 conn(0x7f0b040b6030 0x7f0b040ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b095e9640 1 --2- 172.21.15.27:0/1815023036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b5030 0x7f0b040ad1b0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b095e9640 1 -- 172.21.15.27:0/1815023036 >> 172.21.15.27:0/1815023036 conn(0x7f0b04072650 msgr2=0x7f0b04072a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b095e9640 1 -- 172.21.15.27:0/1815023036 shutdown_connections 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.446+0000 7f0b095e9640 1 -- 172.21.15.27:0/1815023036 wait complete. 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.448+0000 7f0b095e9640 1 Processor -- start 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.449+0000 7f0b095e9640 1 -- start start 2024-03-06T07:28:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0b095e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b040b5030 0x7f0b040c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0b095e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b6030 0x7f0b04155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0b095e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b0415f7d0 0x7f0b041588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0b095e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0406b750 con 0x7f0b040b6030 2024-03-06T07:28:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0b095e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0406b5d0 con 0x7f0b040b5030 2024-03-06T07:28:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0b095e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b040ccd10 con 0x7f0b0415f7d0 2024-03-06T07:28:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0afa7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b6030 0x7f0b04155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0afa7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b6030 0x7f0b04155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60046/0 (socket says 172.21.15.27:60046) 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0afa7fc640 1 -- 172.21.15.27:0/2855536670 learned_addr learned my addr 172.21.15.27:0/2855536670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0b02ffd640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b040b5030 0x7f0b040c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0b037fe640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b0415f7d0 0x7f0b041588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0afa7fc640 1 -- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b0415f7d0 msgr2=0x7f0b041588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0afa7fc640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b0415f7d0 0x7f0b041588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0afa7fc640 1 -- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b040b5030 msgr2=0x7f0b040c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0afa7fc640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b040b5030 0x7f0b040c5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0afa7fc640 1 -- 172.21.15.27:0/2855536670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b04158de0 con 0x7f0b040b6030 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.450+0000 7f0b02ffd640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b040b5030 0x7f0b040c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.451+0000 7f0afa7fc640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b6030 0x7f0b04155540 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f0aec02bf70 tx=0x7f0aec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.451+0000 7f0afbfff640 1 -- 172.21.15.27:0/2855536670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec046020 con 0x7f0b040b6030 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.451+0000 7f0afbfff640 1 -- 172.21.15.27:0/2855536670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aec039a10 con 0x7f0b040b6030 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.451+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b040c5fc0 con 0x7f0b040b6030 2024-03-06T07:28:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.451+0000 7f0afbfff640 1 -- 172.21.15.27:0/2855536670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec034040 con 0x7f0b040b6030 2024-03-06T07:28:24.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.451+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b040c6580 con 0x7f0b040b6030 2024-03-06T07:28:24.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.452+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b040752d0 con 0x7f0b040b6030 2024-03-06T07:28:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.452+0000 7f0afbfff640 1 -- 172.21.15.27:0/2855536670 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0aec037070 con 0x7f0b040b6030 2024-03-06T07:28:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.453+0000 7f0afbfff640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ae406b500 0x7f0ae406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.453+0000 7f0afbfff640 1 -- 172.21.15.27:0/2855536670 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0aec0abf90 con 0x7f0b040b6030 2024-03-06T07:28:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.453+0000 7f0b02ffd640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ae406b500 0x7f0ae406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.453+0000 7f0b02ffd640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ae406b500 0x7f0ae406d9c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0b0415bed0 tx=0x7f0af4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.457+0000 7f0afbfff640 1 -- 172.21.15.27:0/2855536670 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0aec050020 con 0x7f0b040b6030 2024-03-06T07:28:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.674+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b04074510 con 0x7f0b040b6030 2024-03-06T07:28:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.675+0000 7f0afbfff640 1 -- 172.21.15.27:0/2855536670 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f0aec039e20 con 0x7f0b040b6030 2024-03-06T07:28:24.677 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:24.677 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:24.677 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ae406b500 msgr2=0x7f0ae406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ae406b500 0x7f0ae406d9c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0b0415bed0 tx=0x7f0af4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b6030 msgr2=0x7f0b04155540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b6030 0x7f0b04155540 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f0aec02bf70 tx=0x7f0aec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 shutdown_connections 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ae406b500 0x7f0ae406d9c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b0415f7d0 0x7f0b041588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b040b6030 0x7f0b04155540 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 --2- 172.21.15.27:0/2855536670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b040b5030 0x7f0b040c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 >> 172.21.15.27:0/2855536670 conn(0x7f0b04072650 msgr2=0x7f0b0406d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 shutdown_connections 2024-03-06T07:28:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.677+0000 7f0b095e9640 1 -- 172.21.15.27:0/2855536670 wait complete. 2024-03-06T07:28:24.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:24.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.803+0000 7ff85d72c640 1 Processor -- start 2024-03-06T07:28:24.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.803+0000 7ff85d72c640 1 -- start start 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff85d72c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d1c60 0x7ff8580d49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff85d72c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8580b4af0 0x7ff8580b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff85d72c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8580c79d0 con 0x7ff8580d2460 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff85d72c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8580c85e0 con 0x7ff8580b4af0 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff85d72c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580c7cb0 con 0x7ff8580d1c60 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff856ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff8580d2460 0x7ff8580d2840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46126/0 (socket says 172.21.15.27:46126) 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff856ffd640 1 -- 172.21.15.27:0/855356410 learned_addr learned my addr 172.21.15.27:0/855356410 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff8577fe640 1 --2- 172.21.15.27:0/855356410 >> v2:172.21.15.69:3300/0 conn(0x7ff8580b4af0 0x7ff8580b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff8567fc640 1 --2- 172.21.15.27:0/855356410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d1c60 0x7ff8580d49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff855ffb640 1 -- 172.21.15.27:0/855356410 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867759313 0 0) 0x7ff8580c79d0 con 0x7ff8580d2460 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff855ffb640 1 -- 172.21.15.27:0/855356410 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff840003520 con 0x7ff8580d2460 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff8567fc640 1 -- 172.21.15.27:0/855356410 >> v2:172.21.15.69:3300/0 conn(0x7ff8580b4af0 msgr2=0x7ff8580b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff8567fc640 1 --2- 172.21.15.27:0/855356410 >> v2:172.21.15.69:3300/0 conn(0x7ff8580b4af0 0x7ff8580b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff8567fc640 1 -- 172.21.15.27:0/855356410 >> v1:172.21.15.27:6789/0 conn(0x7ff8580d2460 legacy=0x7ff8580d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.804+0000 7ff8567fc640 1 -- 172.21.15.27:0/855356410 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8580c7f90 con 0x7ff8580d1c60 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.805+0000 7ff8567fc640 1 --2- 172.21.15.27:0/855356410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d1c60 0x7ff8580d49a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0097d0 tx=0x7ff84c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3572d90f3846671b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.805+0000 7ff855ffb640 1 -- 172.21.15.27:0/855356410 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c035020 con 0x7ff8580d1c60 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.805+0000 7ff855ffb640 1 -- 172.21.15.27:0/855356410 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c031800 con 0x7ff8580d1c60 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.805+0000 7ff855ffb640 1 -- 172.21.15.27:0/855356410 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c031bd0 con 0x7ff8580d1c60 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.806+0000 7ff85d72c640 1 -- 172.21.15.27:0/855356410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d1c60 msgr2=0x7ff8580d49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.806+0000 7ff85d72c640 1 --2- 172.21.15.27:0/855356410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d1c60 0x7ff8580d49a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0097d0 tx=0x7ff84c02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.806+0000 7ff85d72c640 1 -- 172.21.15.27:0/855356410 shutdown_connections 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.806+0000 7ff85d72c640 1 --2- 172.21.15.27:0/855356410 >> v2:172.21.15.69:3300/0 conn(0x7ff8580b4af0 0x7ff8580b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.806+0000 7ff85d72c640 1 --2- 172.21.15.27:0/855356410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d1c60 0x7ff8580d49a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.806+0000 7ff85d72c640 1 -- 172.21.15.27:0/855356410 >> 172.21.15.27:0/855356410 conn(0x7ff8580d3e80 msgr2=0x7ff8580d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.806+0000 7ff85d72c640 1 -- 172.21.15.27:0/855356410 shutdown_connections 2024-03-06T07:28:24.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.806+0000 7ff85d72c640 1 -- 172.21.15.27:0/855356410 wait complete. 2024-03-06T07:28:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.807+0000 7ff85d72c640 1 Processor -- start 2024-03-06T07:28:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.807+0000 7ff85d72c640 1 -- start start 2024-03-06T07:28:24.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff85d72c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8580b4af0 0x7ff8580cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff85d72c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8580d1c60 0x7ff8581f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff85d72c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d2460 0x7ff85815c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff85d72c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580d1280 con 0x7ff8580d1c60 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff85d72c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580d1100 con 0x7ff8580b4af0 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff85d72c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85815e550 con 0x7ff8580d2460 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff8567fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8580d1c60 0x7ff8581f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff8567fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8580d1c60 0x7ff8581f3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60062/0 (socket says 172.21.15.27:60062) 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff8567fc640 1 -- 172.21.15.27:0/4228801275 learned_addr learned my addr 172.21.15.27:0/4228801275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff8577fe640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d2460 0x7ff85815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff856ffd640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8580b4af0 0x7ff8580cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff8567fc640 1 -- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d2460 msgr2=0x7ff85815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff8567fc640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d2460 0x7ff85815c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff8567fc640 1 -- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8580b4af0 msgr2=0x7ff8580cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff8567fc640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8580b4af0 0x7ff8580cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.808+0000 7ff8567fc640 1 -- 172.21.15.27:0/4228801275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8580ce1f0 con 0x7ff8580d1c60 2024-03-06T07:28:24.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.809+0000 7ff8567fc640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8580d1c60 0x7ff8581f3e20 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0096f0 tx=0x7ff84c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.809+0000 7ff83f7fe640 1 -- 172.21.15.27:0/4228801275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c041070 con 0x7ff8580d1c60 2024-03-06T07:28:24.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.809+0000 7ff83f7fe640 1 -- 172.21.15.27:0/4228801275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c031e30 con 0x7ff8580d1c60 2024-03-06T07:28:24.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.809+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8580b81e0 con 0x7ff8580d1c60 2024-03-06T07:28:24.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.809+0000 7ff83f7fe640 1 -- 172.21.15.27:0/4228801275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c035040 con 0x7ff8580d1c60 2024-03-06T07:28:24.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.809+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8580b87a0 con 0x7ff8580d1c60 2024-03-06T07:28:24.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.810+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8580752d0 con 0x7ff8580d1c60 2024-03-06T07:28:24.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.810+0000 7ff83f7fe640 1 -- 172.21.15.27:0/4228801275 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff84c04c060 con 0x7ff8580d1c60 2024-03-06T07:28:24.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.811+0000 7ff83f7fe640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff84406b4d0 0x7ff84406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.811+0000 7ff83f7fe640 1 -- 172.21.15.27:0/4228801275 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff84c0abc20 con 0x7ff8580d1c60 2024-03-06T07:28:24.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.811+0000 7ff856ffd640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff84406b4d0 0x7ff84406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.812+0000 7ff856ffd640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff84406b4d0 0x7ff84406d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff848005880 tx=0x7ff848009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:24.820+0000 7ff83f7fe640 1 -- 172.21.15.27:0/4228801275 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff84c04b2c0 con 0x7ff8580d1c60 2024-03-06T07:28:25.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.063+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8580b8970 con 0x7ff8580d1c60 2024-03-06T07:28:25.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.064+0000 7ff83f7fe640 1 -- 172.21.15.27:0/4228801275 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16666 (secure 0 0 0) 0x7ff84c07aad0 con 0x7ff8580d1c60 2024-03-06T07:28:25.066 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:25.066 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:57.839900+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:46.443375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:47.116887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/3147032864":"2024-03-07T07:27:57.839873+0000","172.21.15.27:6833/3147032864":"2024-03-07T07:27:57.839873+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff84406b4d0 msgr2=0x7ff84406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff84406b4d0 0x7ff84406d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff848005880 tx=0x7ff848009040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8580d1c60 msgr2=0x7ff8581f3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8580d1c60 0x7ff8581f3e20 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0096f0 tx=0x7ff84c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 shutdown_connections 2024-03-06T07:28:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff84406b4d0 0x7ff84406d990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8580d2460 0x7ff85815c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8580d1c60 0x7ff8581f3e20 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 --2- 172.21.15.27:0/4228801275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8580b4af0 0x7ff8580cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 >> 172.21.15.27:0/4228801275 conn(0x7ff8580d3e80 msgr2=0x7ff85806ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 shutdown_connections 2024-03-06T07:28:25.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.066+0000 7ff85d72c640 1 -- 172.21.15.27:0/4228801275 wait complete. 2024-03-06T07:28:25.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:46.443375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:47.116887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:25.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.193+0000 7f1b9c44e640 1 Processor -- start 2024-03-06T07:28:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.193+0000 7f1b9c44e640 1 -- start start 2024-03-06T07:28:25.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b9c44e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b94161830 0x7f1b940d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b9c44e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b940d2100 con 0x7f1b94161c10 2024-03-06T07:28:25.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b9c44e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b940b5000 con 0x7f1b940d0fc0 2024-03-06T07:28:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b9c44e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b940b52c0 con 0x7f1b94161830 2024-03-06T07:28:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b9a1ea640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b94161830 0x7f1b940d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b999e9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1b94161c10 0x7f1b940c7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46130/0 (socket says 172.21.15.27:46130) 2024-03-06T07:28:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b999e9640 1 -- 172.21.15.27:0/1365518556 learned_addr learned my addr 172.21.15.27:0/1365518556 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b9a1ea640 1 -- 172.21.15.27:0/1365518556 >> v1:172.21.15.69:6789/0 conn(0x7f1b940d0fc0 legacy=0x7f1b940c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b9a1ea640 1 -- 172.21.15.27:0/1365518556 >> v1:172.21.15.27:6789/0 conn(0x7f1b94161c10 legacy=0x7f1b940c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b991e8640 1 -- 172.21.15.27:0/1365518556 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641298490 0 0) 0x7f1b940d2100 con 0x7f1b94161c10 2024-03-06T07:28:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.194+0000 7f1b9a1ea640 1 -- 172.21.15.27:0/1365518556 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b940d1e20 con 0x7f1b94161830 2024-03-06T07:28:25.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.195+0000 7f1b9a1ea640 1 --2- 172.21.15.27:0/1365518556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b94161830 0x7f1b940d24f0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f1b900097a0 tx=0x7f1b9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59ca4900780ff49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.195+0000 7f1b991e8640 1 -- 172.21.15.27:0/1365518556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90035020 con 0x7f1b94161830 2024-03-06T07:28:25.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.195+0000 7f1b991e8640 1 -- 172.21.15.27:0/1365518556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b90031830 con 0x7f1b94161830 2024-03-06T07:28:25.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.195+0000 7f1b991e8640 1 -- 172.21.15.27:0/1365518556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90031c00 con 0x7f1b94161830 2024-03-06T07:28:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.196+0000 7f1b9c44e640 1 -- 172.21.15.27:0/1365518556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b94161830 msgr2=0x7f1b940d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.196+0000 7f1b9c44e640 1 --2- 172.21.15.27:0/1365518556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b94161830 0x7f1b940d24f0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f1b900097a0 tx=0x7f1b9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.196+0000 7f1b9c44e640 1 -- 172.21.15.27:0/1365518556 shutdown_connections 2024-03-06T07:28:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.196+0000 7f1b9c44e640 1 --2- 172.21.15.27:0/1365518556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b94161830 0x7f1b940d24f0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.196+0000 7f1b9c44e640 1 -- 172.21.15.27:0/1365518556 >> 172.21.15.27:0/1365518556 conn(0x7f1b940d3f10 msgr2=0x7f1b940d4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.196+0000 7f1b9c44e640 1 -- 172.21.15.27:0/1365518556 shutdown_connections 2024-03-06T07:28:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.196+0000 7f1b9c44e640 1 -- 172.21.15.27:0/1365518556 wait complete. 2024-03-06T07:28:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.197+0000 7f1b9c44e640 1 Processor -- start 2024-03-06T07:28:25.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.197+0000 7f1b9c44e640 1 -- start start 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9c44e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b940d0fc0 0x7f1b9415b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9c44e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94161830 0x7f1b940b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9c44e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b94159c40 0x7f1b940b7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9c44e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940cfb30 con 0x7f1b940d0fc0 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9c44e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940d4bd0 con 0x7f1b94161830 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9c44e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940cfcb0 con 0x7f1b94159c40 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a1ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b940d0fc0 0x7f1b9415b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a1ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b940d0fc0 0x7f1b9415b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60084/0 (socket says 172.21.15.27:60084) 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a1ea640 1 -- 172.21.15.27:0/387490378 learned_addr learned my addr 172.21.15.27:0/387490378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b999e9640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94161830 0x7f1b940b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a9eb640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b94159c40 0x7f1b940b7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a1ea640 1 -- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b94159c40 msgr2=0x7f1b940b7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a1ea640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b94159c40 0x7f1b940b7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a1ea640 1 -- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94161830 msgr2=0x7f1b940b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a1ea640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94161830 0x7f1b940b6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a1ea640 1 -- 172.21.15.27:0/387490378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b941f8450 con 0x7f1b940d0fc0 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a9eb640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b94159c40 0x7f1b940b7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.198+0000 7f1b9a1ea640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b940d0fc0 0x7f1b9415b8e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009890 tx=0x7f1b900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.199+0000 7f1b8affd640 1 -- 172.21.15.27:0/387490378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90035040 con 0x7f1b940d0fc0 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.199+0000 7f1b8affd640 1 -- 172.21.15.27:0/387490378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9003aa30 con 0x7f1b940d0fc0 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.199+0000 7f1b8affd640 1 -- 172.21.15.27:0/387490378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9003ae20 con 0x7f1b940d0fc0 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.199+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b940cb160 con 0x7f1b940d0fc0 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.199+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b940cb720 con 0x7f1b940d0fc0 2024-03-06T07:28:25.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.200+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b94068050 con 0x7f1b940d0fc0 2024-03-06T07:28:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.200+0000 7f1b8affd640 1 -- 172.21.15.27:0/387490378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b90038020 con 0x7f1b940d0fc0 2024-03-06T07:28:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.201+0000 7f1b8affd640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b7806b4d0 0x7f1b7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.201+0000 7f1b8affd640 1 -- 172.21.15.27:0/387490378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1b900ac020 con 0x7f1b940d0fc0 2024-03-06T07:28:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.201+0000 7f1b999e9640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b7806b4d0 0x7f1b7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.201+0000 7f1b999e9640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b7806b4d0 0x7f1b7806d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1b940b4780 tx=0x7f1b84007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.205+0000 7f1b8affd640 1 -- 172.21.15.27:0/387490378 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b9007aed0 con 0x7f1b940d0fc0 2024-03-06T07:28:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.451+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b940d22d0 con 0x7f1b940d0fc0 2024-03-06T07:28:25.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.452+0000 7f1b8affd640 1 -- 172.21.15.27:0/387490378 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b9007a7d0 con 0x7f1b940d0fc0 2024-03-06T07:28:25.453 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:28:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b7806b4d0 msgr2=0x7f1b7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b7806b4d0 0x7f1b7806d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1b940b4780 tx=0x7f1b84007e40 comp rx=0 tx=0).stop 2024-03-06T07:28:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b940d0fc0 msgr2=0x7f1b9415b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b940d0fc0 0x7f1b9415b8e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009890 tx=0x7f1b900046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 shutdown_connections 2024-03-06T07:28:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b7806b4d0 0x7f1b7806d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b94159c40 0x7f1b940b7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94161830 0x7f1b940b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 --2- 172.21.15.27:0/387490378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b940d0fc0 0x7f1b9415b8e0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.453+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 >> 172.21.15.27:0/387490378 conn(0x7f1b940d3f10 msgr2=0x7f1b94160aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.454+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 shutdown_connections 2024-03-06T07:28:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.454+0000 7f1b9c44e640 1 -- 172.21.15.27:0/387490378 wait complete. 2024-03-06T07:28:25.466 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:25.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.574+0000 7f595118a640 1 Processor -- start 2024-03-06T07:28:25.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.574+0000 7f595118a640 1 -- start start 2024-03-06T07:28:25.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.574+0000 7f595118a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f594c13ee70 0x7f594c144f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.574+0000 7f595118a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f594c1454b0 0x7f594c0ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.574+0000 7f595118a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c0ce850 0x7f594c141750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.574+0000 7f595118a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c145920 con 0x7f594c13ee70 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.574+0000 7f595118a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c15c890 con 0x7f594c1454b0 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.574+0000 7f595118a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c06cb00 con 0x7f594c0ce850 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594ad9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f594c13ee70 0x7f594c144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594ad9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f594c13ee70 0x7f594c144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60098/0 (socket says 172.21.15.27:60098) 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594ad9d640 1 -- 172.21.15.27:0/3503499243 learned_addr learned my addr 172.21.15.27:0/3503499243 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594b59e640 1 --2- 172.21.15.27:0/3503499243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c0ce850 0x7f594c141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594a59c640 1 --2- 172.21.15.27:0/3503499243 >> v2:172.21.15.69:3300/0 conn(0x7f594c1454b0 0x7f594c0ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594ad9d640 1 -- 172.21.15.27:0/3503499243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c0ce850 msgr2=0x7f594c141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594ad9d640 1 --2- 172.21.15.27:0/3503499243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c0ce850 0x7f594c141750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594ad9d640 1 -- 172.21.15.27:0/3503499243 >> v2:172.21.15.69:3300/0 conn(0x7f594c1454b0 msgr2=0x7f594c0ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594ad9d640 1 --2- 172.21.15.27:0/3503499243 >> v2:172.21.15.69:3300/0 conn(0x7f594c1454b0 0x7f594c0ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.575+0000 7f594ad9d640 1 -- 172.21.15.27:0/3503499243 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c0cefd0 con 0x7f594c13ee70 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.576+0000 7f594ad9d640 1 --2- 172.21.15.27:0/3503499243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c13ee70 0x7f594c144f70 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f594c0693d0 tx=0x7f593c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a276336ff27fa090 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.576+0000 7f5949d9b640 1 -- 172.21.15.27:0/3503499243 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c035020 con 0x7f594c13ee70 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.576+0000 7f5949d9b640 1 -- 172.21.15.27:0/3503499243 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c031640 con 0x7f594c13ee70 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.576+0000 7f5949d9b640 1 -- 172.21.15.27:0/3503499243 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c02d040 con 0x7f594c13ee70 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.577+0000 7f595118a640 1 -- 172.21.15.27:0/3503499243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c13ee70 msgr2=0x7f594c144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.577+0000 7f595118a640 1 --2- 172.21.15.27:0/3503499243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c13ee70 0x7f594c144f70 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f594c0693d0 tx=0x7f593c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.577+0000 7f595118a640 1 -- 172.21.15.27:0/3503499243 shutdown_connections 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.577+0000 7f595118a640 1 --2- 172.21.15.27:0/3503499243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c0ce850 0x7f594c141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.577+0000 7f595118a640 1 --2- 172.21.15.27:0/3503499243 >> v2:172.21.15.69:3300/0 conn(0x7f594c1454b0 0x7f594c0ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.577+0000 7f595118a640 1 --2- 172.21.15.27:0/3503499243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c13ee70 0x7f594c144f70 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.577+0000 7f595118a640 1 -- 172.21.15.27:0/3503499243 >> 172.21.15.27:0/3503499243 conn(0x7f594c0b17a0 msgr2=0x7f594c0b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.577+0000 7f595118a640 1 -- 172.21.15.27:0/3503499243 shutdown_connections 2024-03-06T07:28:25.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.577+0000 7f595118a640 1 -- 172.21.15.27:0/3503499243 wait complete. 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.578+0000 7f595118a640 1 Processor -- start 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.578+0000 7f595118a640 1 -- start start 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f595118a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c0ce850 0x7f594c06d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f595118a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c13ee70 0x7f594c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f595118a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c1454b0 0x7f594c140330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f595118a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c06cd50 con 0x7f594c0ce850 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f595118a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0cf590 con 0x7f594c1454b0 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f595118a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0cecc0 con 0x7f594c13ee70 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c0ce850 0x7f594c06d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c0ce850 0x7f594c06d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60102/0 (socket says 172.21.15.27:60102) 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594ad9d640 1 -- 172.21.15.27:0/1730569811 learned_addr learned my addr 172.21.15.27:0/1730569811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594a59c640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c13ee70 0x7f594c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594b59e640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c1454b0 0x7f594c140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594ad9d640 1 -- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c13ee70 msgr2=0x7f594c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594ad9d640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c13ee70 0x7f594c06d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594ad9d640 1 -- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c1454b0 msgr2=0x7f594c140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594ad9d640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c1454b0 0x7f594c140330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594ad9d640 1 -- 172.21.15.27:0/1730569811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c143a10 con 0x7f594c0ce850 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.579+0000 7f594b59e640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c1454b0 0x7f594c140330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.580+0000 7f594ad9d640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c0ce850 0x7f594c06d170 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f593c000c00 tx=0x7f593c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.580+0000 7f59337fe640 1 -- 172.21.15.27:0/1730569811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c040070 con 0x7f594c0ce850 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.580+0000 7f59337fe640 1 -- 172.21.15.27:0/1730569811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c0399f0 con 0x7f594c0ce850 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.580+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c143cf0 con 0x7f594c0ce850 2024-03-06T07:28:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.580+0000 7f59337fe640 1 -- 172.21.15.27:0/1730569811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c035040 con 0x7f594c0ce850 2024-03-06T07:28:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.580+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c1442b0 con 0x7f594c0ce850 2024-03-06T07:28:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.581+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c068050 con 0x7f594c0ce850 2024-03-06T07:28:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.581+0000 7f59337fe640 1 -- 172.21.15.27:0/1730569811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f593c04b060 con 0x7f594c0ce850 2024-03-06T07:28:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.582+0000 7f59337fe640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593806b4d0 0x7f593806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.582+0000 7f59337fe640 1 -- 172.21.15.27:0/1730569811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f593c0aba80 con 0x7f594c0ce850 2024-03-06T07:28:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.582+0000 7f594a59c640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593806b4d0 0x7f593806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.582+0000 7f594a59c640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593806b4d0 0x7f593806d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f594c141fc0 tx=0x7f5940008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.586+0000 7f59337fe640 1 -- 172.21.15.27:0/1730569811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f593c07a8a0 con 0x7f594c0ce850 2024-03-06T07:28:25.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.768+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f594c06e840 con 0x7f594c0ce850 2024-03-06T07:28:25.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.769+0000 7f59337fe640 1 -- 172.21.15.27:0/1730569811 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7f593c07a1a0 con 0x7f594c0ce850 2024-03-06T07:28:25.770 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:28:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593806b4d0 msgr2=0x7f593806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593806b4d0 0x7f593806d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f594c141fc0 tx=0x7f5940008040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c0ce850 msgr2=0x7f594c06d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c0ce850 0x7f594c06d170 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f593c000c00 tx=0x7f593c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 shutdown_connections 2024-03-06T07:28:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f593806b4d0 0x7f593806d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f594c1454b0 0x7f594c140330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f594c13ee70 0x7f594c06d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 --2- 172.21.15.27:0/1730569811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594c0ce850 0x7f594c06d170 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 >> 172.21.15.27:0/1730569811 conn(0x7f594c0b17a0 msgr2=0x7f594c0b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 shutdown_connections 2024-03-06T07:28:25.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.771+0000 7f595118a640 1 -- 172.21.15.27:0/1730569811 wait complete. 2024-03-06T07:28:25.788 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:25.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.932+0000 7fa253466640 1 Processor -- start 2024-03-06T07:28:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.932+0000 7fa253466640 1 -- start start 2024-03-06T07:28:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.933+0000 7fa253466640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa24c161c10 0x7fa24c0ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.933+0000 7fa253466640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24c0d2100 con 0x7fa24c0d2c00 2024-03-06T07:28:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.933+0000 7fa253466640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24c0b5000 con 0x7fa24c161830 2024-03-06T07:28:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.933+0000 7fa253466640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24c0b52c0 con 0x7fa24c161c10 2024-03-06T07:28:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.933+0000 7fa250a01640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa24c161c10 0x7fa24c0ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.933+0000 7fa251a03640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa24c0d2c00 0x7fa24c0cf060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46142/0 (socket says 172.21.15.27:46142) 2024-03-06T07:28:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.933+0000 7fa251202640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa24c161830 0x7fa24c0d24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:54458/0 (socket says 172.21.15.27:54458) 2024-03-06T07:28:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.934+0000 7fa251a03640 1 -- 172.21.15.27:0/1708929261 learned_addr learned my addr 172.21.15.27:0/1708929261 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:25.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.934+0000 7fa23ffff640 1 -- 172.21.15.27:0/1708929261 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350856162 0 0) 0x7fa24c0d2100 con 0x7fa24c0d2c00 2024-03-06T07:28:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.934+0000 7fa23ffff640 1 -- 172.21.15.27:0/1708929261 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa238003520 con 0x7fa24c0d2c00 2024-03-06T07:28:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.934+0000 7fa250a01640 1 -- 172.21.15.27:0/1708929261 >> v1:172.21.15.69:6789/0 conn(0x7fa24c161830 legacy=0x7fa24c0d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.934+0000 7fa23ffff640 1 -- 172.21.15.27:0/1708929261 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995086751 0 0) 0x7fa24c0b5000 con 0x7fa24c161830 2024-03-06T07:28:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.934+0000 7fa250a01640 1 -- 172.21.15.27:0/1708929261 >> v1:172.21.15.27:6789/0 conn(0x7fa24c0d2c00 legacy=0x7fa24c0cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.934+0000 7fa250a01640 1 -- 172.21.15.27:0/1708929261 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c0d1e20 con 0x7fa24c161c10 2024-03-06T07:28:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.935+0000 7fa250a01640 1 --2- 172.21.15.27:0/1708929261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c161c10 0x7fa24c0ceb20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa2480097a0 tx=0x7fa24802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a08fef2310fcbb01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.935+0000 7fa23ffff640 1 -- 172.21.15.27:0/1708929261 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480315a0 con 0x7fa24c161c10 2024-03-06T07:28:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.935+0000 7fa23ffff640 1 -- 172.21.15.27:0/1708929261 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa248037070 con 0x7fa24c161c10 2024-03-06T07:28:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.935+0000 7fa23ffff640 1 -- 172.21.15.27:0/1708929261 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248031960 con 0x7fa24c161c10 2024-03-06T07:28:25.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.936+0000 7fa253466640 1 -- 172.21.15.27:0/1708929261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c161c10 msgr2=0x7fa24c0ceb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.936+0000 7fa253466640 1 --2- 172.21.15.27:0/1708929261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c161c10 0x7fa24c0ceb20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa2480097a0 tx=0x7fa24802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:25.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.936+0000 7fa253466640 1 -- 172.21.15.27:0/1708929261 shutdown_connections 2024-03-06T07:28:25.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.936+0000 7fa253466640 1 --2- 172.21.15.27:0/1708929261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c161c10 0x7fa24c0ceb20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.936+0000 7fa253466640 1 -- 172.21.15.27:0/1708929261 >> 172.21.15.27:0/1708929261 conn(0x7fa24c0d3f10 msgr2=0x7fa24c0d4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.936+0000 7fa253466640 1 -- 172.21.15.27:0/1708929261 shutdown_connections 2024-03-06T07:28:25.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.936+0000 7fa253466640 1 -- 172.21.15.27:0/1708929261 wait complete. 2024-03-06T07:28:25.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.937+0000 7fa253466640 1 Processor -- start 2024-03-06T07:28:25.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.937+0000 7fa253466640 1 -- start start 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa253466640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c0d2c00 0x7fa24c0cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa253466640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24c161830 0x7fa24c0b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa253466640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c1f5750 0x7fa24c0b8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa253466640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0c8a70 con 0x7fa24c1f5750 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa253466640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0d4bd0 con 0x7fa24c0d2c00 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa253466640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0c8bf0 con 0x7fa24c161830 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251a03640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c1f5750 0x7fa24c0b8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251a03640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c1f5750 0x7fa24c0b8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60114/0 (socket says 172.21.15.27:60114) 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251a03640 1 -- 172.21.15.27:0/714667745 learned_addr learned my addr 172.21.15.27:0/714667745 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251202640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c0d2c00 0x7fa24c0cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251a03640 1 -- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24c161830 msgr2=0x7fa24c0b8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa250a01640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24c161830 0x7fa24c0b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251a03640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24c161830 0x7fa24c0b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251a03640 1 -- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c0d2c00 msgr2=0x7fa24c0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251a03640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c0d2c00 0x7fa24c0cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251a03640 1 -- 172.21.15.27:0/714667745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c1f8c70 con 0x7fa24c1f5750 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251202640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c0d2c00 0x7fa24c0cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa251a03640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c1f5750 0x7fa24c0b8f80 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa24400cb70 tx=0x7fa24400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.938+0000 7fa23dffb640 1 -- 172.21.15.27:0/714667745 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244017070 con 0x7fa24c1f5750 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.939+0000 7fa23dffb640 1 -- 172.21.15.27:0/714667745 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24400ad40 con 0x7fa24c1f5750 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.939+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24c1f4990 con 0x7fa24c1f5750 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.939+0000 7fa23dffb640 1 -- 172.21.15.27:0/714667745 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244012620 con 0x7fa24c1f5750 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.939+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24c1f4f50 con 0x7fa24c1f5750 2024-03-06T07:28:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.940+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24c0689a0 con 0x7fa24c1f5750 2024-03-06T07:28:25.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.940+0000 7fa23dffb640 1 -- 172.21.15.27:0/714667745 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa24401a070 con 0x7fa24c1f5750 2024-03-06T07:28:25.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.941+0000 7fa23dffb640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23006b500 0x7fa23006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.941+0000 7fa23dffb640 1 -- 172.21.15.27:0/714667745 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa244089f70 con 0x7fa24c1f5750 2024-03-06T07:28:25.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.941+0000 7fa251202640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23006b500 0x7fa23006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.942+0000 7fa251202640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23006b500 0x7fa23006d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa240000f60 tx=0x7fa240009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:25.945+0000 7fa23dffb640 1 -- 172.21.15.27:0/714667745 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa24c0689a0 con 0x7fa24c1f5750 2024-03-06T07:28:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.126+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa24c0b8820 con 0x7fa24c1f5750 2024-03-06T07:28:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.126+0000 7fa23dffb640 1 -- 172.21.15.27:0/714667745 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa244058e20 con 0x7fa24c1f5750 2024-03-06T07:28:26.128 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:28:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23006b500 msgr2=0x7fa23006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23006b500 0x7fa23006d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa240000f60 tx=0x7fa240009040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c1f5750 msgr2=0x7fa24c0b8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c1f5750 0x7fa24c0b8f80 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa24400cb70 tx=0x7fa24400a430 comp rx=0 tx=0).stop 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 shutdown_connections 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa23006b500 0x7fa23006d9c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c1f5750 0x7fa24c0b8f80 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa24c161830 0x7fa24c0b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 --2- 172.21.15.27:0/714667745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa24c0d2c00 0x7fa24c0cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 >> 172.21.15.27:0/714667745 conn(0x7fa24c0d3f10 msgr2=0x7fa24c06bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 shutdown_connections 2024-03-06T07:28:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.128+0000 7fa253466640 1 -- 172.21.15.27:0/714667745 wait complete. 2024-03-06T07:28:26.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.245+0000 7fb407302640 1 Processor -- start 2024-03-06T07:28:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.245+0000 7fb407302640 1 -- start start 2024-03-06T07:28:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.246+0000 7fb407302640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4001553e0 0x7fb4000d1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.246+0000 7fb407302640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb40015e6b0 0x7fb40015eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.246+0000 7fb407302640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40006fa70 con 0x7fb4000b3b90 2024-03-06T07:28:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.246+0000 7fb407302640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb400161f40 con 0x7fb4001553e0 2024-03-06T07:28:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.246+0000 7fb407302640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40015f060 con 0x7fb40015e6b0 2024-03-06T07:28:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.246+0000 7fb40509e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4001553e0 0x7fb4000d1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.246+0000 7fb40489d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb4000b3b90 0x7fb4001623d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46152/0 (socket says 172.21.15.27:46152) 2024-03-06T07:28:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.246+0000 7fb40489d640 1 -- 172.21.15.27:0/3582619283 learned_addr learned my addr 172.21.15.27:0/3582619283 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.247+0000 7fb40589f640 1 --2- 172.21.15.27:0/3582619283 >> v2:172.21.15.69:3300/0 conn(0x7fb40015e6b0 0x7fb40015eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.247+0000 7fb40509e640 1 -- 172.21.15.27:0/3582619283 >> v2:172.21.15.69:3300/0 conn(0x7fb40015e6b0 msgr2=0x7fb40015eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.247+0000 7fb40509e640 1 --2- 172.21.15.27:0/3582619283 >> v2:172.21.15.69:3300/0 conn(0x7fb40015e6b0 0x7fb40015eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.247+0000 7fb40509e640 1 -- 172.21.15.27:0/3582619283 >> v1:172.21.15.27:6789/0 conn(0x7fb4000b3b90 legacy=0x7fb4001623d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.247+0000 7fb40509e640 1 -- 172.21.15.27:0/3582619283 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40006fd50 con 0x7fb4001553e0 2024-03-06T07:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb40509e640 1 --2- 172.21.15.27:0/3582619283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4001553e0 0x7fb4000d1750 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fb4000b0740 tx=0x7fb3fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b652c7b5105903 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb3f7fff640 1 -- 172.21.15.27:0/3582619283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc035020 con 0x7fb4001553e0 2024-03-06T07:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb3f7fff640 1 -- 172.21.15.27:0/3582619283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3fc031830 con 0x7fb4001553e0 2024-03-06T07:28:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb3f7fff640 1 -- 172.21.15.27:0/3582619283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc031c00 con 0x7fb4001553e0 2024-03-06T07:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb407302640 1 -- 172.21.15.27:0/3582619283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4001553e0 msgr2=0x7fb4000d1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb407302640 1 --2- 172.21.15.27:0/3582619283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4001553e0 0x7fb4000d1750 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fb4000b0740 tx=0x7fb3fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb407302640 1 -- 172.21.15.27:0/3582619283 shutdown_connections 2024-03-06T07:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb407302640 1 --2- 172.21.15.27:0/3582619283 >> v2:172.21.15.69:3300/0 conn(0x7fb40015e6b0 0x7fb40015eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb407302640 1 --2- 172.21.15.27:0/3582619283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4001553e0 0x7fb4000d1750 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb407302640 1 -- 172.21.15.27:0/3582619283 >> 172.21.15.27:0/3582619283 conn(0x7fb4000d0c30 msgr2=0x7fb4000d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb407302640 1 -- 172.21.15.27:0/3582619283 shutdown_connections 2024-03-06T07:28:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.248+0000 7fb407302640 1 -- 172.21.15.27:0/3582619283 wait complete. 2024-03-06T07:28:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb407302640 1 Processor -- start 2024-03-06T07:28:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb407302640 1 -- start start 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb407302640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4000b3b90 0x7fb4001527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb407302640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb40015e6b0 0x7fb400153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb407302640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb400155a00 0x7fb40014be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb407302640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb400162ae0 con 0x7fb400155a00 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb407302640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb400070310 con 0x7fb40015e6b0 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb407302640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4000b53c0 con 0x7fb4000b3b90 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb40589f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb400155a00 0x7fb40014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb40589f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb400155a00 0x7fb40014be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60142/0 (socket says 172.21.15.27:60142) 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.250+0000 7fb40589f640 1 -- 172.21.15.27:0/2409784358 learned_addr learned my addr 172.21.15.27:0/2409784358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40489d640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb40015e6b0 0x7fb400153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40509e640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4000b3b90 0x7fb4001527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40589f640 1 -- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4000b3b90 msgr2=0x7fb4001527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40589f640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4000b3b90 0x7fb4001527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40589f640 1 -- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb40015e6b0 msgr2=0x7fb400153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40589f640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb40015e6b0 0x7fb400153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40589f640 1 -- 172.21.15.27:0/2409784358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb400152e70 con 0x7fb400155a00 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40509e640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4000b3b90 0x7fb4001527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40489d640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb40015e6b0 0x7fb400153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb40589f640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb400155a00 0x7fb40014be20 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8007e60 tx=0x7fb3f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb3f5ffb640 1 -- 172.21.15.27:0/2409784358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8014420 con 0x7fb400155a00 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb3f5ffb640 1 -- 172.21.15.27:0/2409784358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f8019070 con 0x7fb400155a00 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb400153150 con 0x7fb400155a00 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb3f5ffb640 1 -- 172.21.15.27:0/2409784358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8014820 con 0x7fb400155a00 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.251+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb400153710 con 0x7fb400155a00 2024-03-06T07:28:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.252+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4000752d0 con 0x7fb400155a00 2024-03-06T07:28:26.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.253+0000 7fb3f5ffb640 1 -- 172.21.15.27:0/2409784358 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3f80149c0 con 0x7fb400155a00 2024-03-06T07:28:26.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.254+0000 7fb3f5ffb640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3e406b500 0x7fb3e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.254+0000 7fb3f5ffb640 1 -- 172.21.15.27:0/2409784358 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb3f808aa00 con 0x7fb400155a00 2024-03-06T07:28:26.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.254+0000 7fb40509e640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3e406b500 0x7fb3e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.255+0000 7fb40509e640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3e406b500 0x7fb3e406d9c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc036000 tx=0x7fb3fc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.258+0000 7fb3f5ffb640 1 -- 172.21.15.27:0/2409784358 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3f80598b0 con 0x7fb400155a00 2024-03-06T07:28:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.478+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4000b0470 con 0x7fb400155a00 2024-03-06T07:28:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.479+0000 7fb3f5ffb640 1 -- 172.21.15.27:0/2409784358 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fb3f80591b0 con 0x7fb400155a00 2024-03-06T07:28:26.481 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:26.481 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:26.481 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3e406b500 msgr2=0x7fb3e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3e406b500 0x7fb3e406d9c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc036000 tx=0x7fb3fc008030 comp rx=0 tx=0).stop 2024-03-06T07:28:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb400155a00 msgr2=0x7fb40014be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb400155a00 0x7fb40014be20 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8007e60 tx=0x7fb3f800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 shutdown_connections 2024-03-06T07:28:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3e406b500 0x7fb3e406d9c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb400155a00 0x7fb40014be20 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb40015e6b0 0x7fb400153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 --2- 172.21.15.27:0/2409784358 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4000b3b90 0x7fb4001527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 >> 172.21.15.27:0/2409784358 conn(0x7fb4000d0c30 msgr2=0x7fb4000ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 shutdown_connections 2024-03-06T07:28:26.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.481+0000 7fb407302640 1 -- 172.21.15.27:0/2409784358 wait complete. 2024-03-06T07:28:26.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:26.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.599+0000 7f7fc641c640 1 Processor -- start 2024-03-06T07:28:26.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.599+0000 7f7fc641c640 1 -- start start 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fc641c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7fc015e100 0x7f7fc00d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fc641c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fc00d4c40 0x7f7fc00d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fc641c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e4e0 0x7f7fc00d2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fc641c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc00d0100 con 0x7f7fc015e100 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fc641c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc00cf860 con 0x7f7fc00d4c40 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fc641c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00cfb40 con 0x7f7fc015e4e0 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fbffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7fc015e100 0x7f7fc00d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fbffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7fc015e100 0x7f7fc00d1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60158/0 (socket says 172.21.15.27:60158) 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fbffff640 1 -- 172.21.15.27:0/1177318682 learned_addr learned my addr 172.21.15.27:0/1177318682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:26.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fbf7fe640 1 --2- 172.21.15.27:0/1177318682 >> v2:172.21.15.69:3300/0 conn(0x7f7fc00d4c40 0x7f7fc00d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.600+0000 7f7fc49b9640 1 --2- 172.21.15.27:0/1177318682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e4e0 0x7f7fc00d2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.601+0000 7f7fbffff640 1 -- 172.21.15.27:0/1177318682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e4e0 msgr2=0x7f7fc00d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.601+0000 7f7fbffff640 1 --2- 172.21.15.27:0/1177318682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e4e0 0x7f7fc00d2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.601+0000 7f7fbffff640 1 -- 172.21.15.27:0/1177318682 >> v2:172.21.15.69:3300/0 conn(0x7f7fc00d4c40 msgr2=0x7f7fc00d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.601+0000 7f7fbffff640 1 --2- 172.21.15.27:0/1177318682 >> v2:172.21.15.69:3300/0 conn(0x7f7fc00d4c40 0x7f7fc00d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.601+0000 7f7fbffff640 1 -- 172.21.15.27:0/1177318682 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc00cfe20 con 0x7f7fc015e100 2024-03-06T07:28:26.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.601+0000 7f7fbffff640 1 --2- 172.21.15.27:0/1177318682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e100 0x7f7fc00d1cd0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00677b0 tx=0x7f7fb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699473b2fc1fbf49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.601+0000 7f7fbeffd640 1 -- 172.21.15.27:0/1177318682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0035020 con 0x7f7fc015e100 2024-03-06T07:28:26.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fbeffd640 1 -- 172.21.15.27:0/1177318682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb0031870 con 0x7f7fc015e100 2024-03-06T07:28:26.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fbeffd640 1 -- 172.21.15.27:0/1177318682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0031c40 con 0x7f7fc015e100 2024-03-06T07:28:26.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fc641c640 1 -- 172.21.15.27:0/1177318682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e100 msgr2=0x7f7fc00d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fc641c640 1 --2- 172.21.15.27:0/1177318682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e100 0x7f7fc00d1cd0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00677b0 tx=0x7f7fb002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:26.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fc641c640 1 -- 172.21.15.27:0/1177318682 shutdown_connections 2024-03-06T07:28:26.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fc641c640 1 --2- 172.21.15.27:0/1177318682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e4e0 0x7f7fc00d2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fc641c640 1 --2- 172.21.15.27:0/1177318682 >> v2:172.21.15.69:3300/0 conn(0x7f7fc00d4c40 0x7f7fc00d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fc641c640 1 --2- 172.21.15.27:0/1177318682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e100 0x7f7fc00d1cd0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fc641c640 1 -- 172.21.15.27:0/1177318682 >> 172.21.15.27:0/1177318682 conn(0x7f7fc00d3ef0 msgr2=0x7f7fc00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fc641c640 1 -- 172.21.15.27:0/1177318682 shutdown_connections 2024-03-06T07:28:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.602+0000 7f7fc641c640 1 -- 172.21.15.27:0/1177318682 wait complete. 2024-03-06T07:28:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.604+0000 7f7fc641c640 1 Processor -- start 2024-03-06T07:28:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.604+0000 7f7fc641c640 1 -- start start 2024-03-06T07:28:26.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fc641c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fc00d4c40 0x7f7fc009e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fc641c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e100 0x7f7fc00b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fc641c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e4e0 0x7f7fc014ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fc641c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00d0650 con 0x7f7fc015e4e0 2024-03-06T07:28:26.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fc641c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00cf510 con 0x7f7fc00d4c40 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fc641c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00d07d0 con 0x7f7fc015e100 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fc49b9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e4e0 0x7f7fc014ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fc49b9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e4e0 0x7f7fc014ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60170/0 (socket says 172.21.15.27:60170) 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fc49b9640 1 -- 172.21.15.27:0/494155148 learned_addr learned my addr 172.21.15.27:0/494155148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fbf7fe640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e100 0x7f7fc00b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.605+0000 7f7fbffff640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fc00d4c40 0x7f7fc009e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.606+0000 7f7fc49b9640 1 -- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e100 msgr2=0x7f7fc00b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.606+0000 7f7fc49b9640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e100 0x7f7fc00b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.606+0000 7f7fc49b9640 1 -- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fc00d4c40 msgr2=0x7f7fc009e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.606+0000 7f7fc49b9640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fc00d4c40 0x7f7fc009e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.606+0000 7f7fc49b9640 1 -- 172.21.15.27:0/494155148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc00b38d0 con 0x7f7fc015e4e0 2024-03-06T07:28:26.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.606+0000 7f7fc49b9640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e4e0 0x7f7fc014ba20 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8007e60 tx=0x7f7fb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.609+0000 7f7fbcff9640 1 -- 172.21.15.27:0/494155148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8014420 con 0x7f7fc015e4e0 2024-03-06T07:28:26.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.609+0000 7f7fbcff9640 1 -- 172.21.15.27:0/494155148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb8019070 con 0x7f7fc015e4e0 2024-03-06T07:28:26.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.609+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc014c4d0 con 0x7f7fc015e4e0 2024-03-06T07:28:26.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.609+0000 7f7fbcff9640 1 -- 172.21.15.27:0/494155148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8014820 con 0x7f7fc015e4e0 2024-03-06T07:28:26.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.609+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc014ca90 con 0x7f7fc015e4e0 2024-03-06T07:28:26.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.610+0000 7f7fbcff9640 1 -- 172.21.15.27:0/494155148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fb80149c0 con 0x7f7fc015e4e0 2024-03-06T07:28:26.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.610+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc00752d0 con 0x7f7fc015e4e0 2024-03-06T07:28:26.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.611+0000 7f7fbcff9640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7fa006b500 0x7f7fa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.611+0000 7f7fbcff9640 1 -- 172.21.15.27:0/494155148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7fb808a990 con 0x7f7fc015e4e0 2024-03-06T07:28:26.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.611+0000 7f7fbffff640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7fa006b500 0x7f7fa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.611+0000 7f7fbffff640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7fa006b500 0x7f7fa006d9c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0036000 tx=0x7f7fb003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.615+0000 7f7fbcff9640 1 -- 172.21.15.27:0/494155148 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fb80590d0 con 0x7f7fc015e4e0 2024-03-06T07:28:26.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.800+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fc0068260 con 0x7f7fc015e4e0 2024-03-06T07:28:26.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.801+0000 7f7fbcff9640 1 -- 172.21.15.27:0/494155148 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f7fb805d170 con 0x7f7fc015e4e0 2024-03-06T07:28:26.803 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:26.803 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:57.839900+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:46.443375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:47.116887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/3147032864":"2024-03-07T07:27:57.839873+0000","172.21.15.27:6833/3147032864":"2024-03-07T07:27:57.839873+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:26.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7fa006b500 msgr2=0x7f7fa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7fa006b500 0x7f7fa006d9c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0036000 tx=0x7f7fb003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e4e0 msgr2=0x7f7fc014ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e4e0 0x7f7fc014ba20 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8007e60 tx=0x7f7fb800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 shutdown_connections 2024-03-06T07:28:26.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7fa006b500 0x7f7fa006d9c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fc015e4e0 0x7f7fc014ba20 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7fc015e100 0x7f7fc00b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 --2- 172.21.15.27:0/494155148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fc00d4c40 0x7f7fc009e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.804+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 >> 172.21.15.27:0/494155148 conn(0x7f7fc00d3ef0 msgr2=0x7f7fc015ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.805+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 shutdown_connections 2024-03-06T07:28:26.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.805+0000 7f7fc641c640 1 -- 172.21.15.27:0/494155148 wait complete. 2024-03-06T07:28:26.818 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:26.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.924+0000 7f515fd93640 1 Processor -- start 2024-03-06T07:28:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.924+0000 7f515fd93640 1 -- start start 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515fd93640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f515815e100 0x7f51580d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515fd93640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f51580d4c40 0x7f51580d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515fd93640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580d0100 con 0x7f515815e4e0 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515fd93640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51580cf860 con 0x7f515815e100 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515fd93640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51580cfb40 con 0x7f51580d4c40 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515db2f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f515815e100 0x7f51580d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515e330640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f515815e4e0 0x7f51580d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46156/0 (socket says 172.21.15.27:46156) 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515e330640 1 -- 172.21.15.27:0/1591555165 learned_addr learned my addr 172.21.15.27:0/1591555165 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515d32e640 1 --2- 172.21.15.27:0/1591555165 >> v2:172.21.15.69:3300/0 conn(0x7f51580d4c40 0x7f51580d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515db2f640 1 -- 172.21.15.27:0/1591555165 >> v2:172.21.15.69:3300/0 conn(0x7f51580d4c40 msgr2=0x7f51580d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515db2f640 1 --2- 172.21.15.27:0/1591555165 >> v2:172.21.15.69:3300/0 conn(0x7f51580d4c40 0x7f51580d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515db2f640 1 -- 172.21.15.27:0/1591555165 >> v1:172.21.15.27:6789/0 conn(0x7f515815e4e0 legacy=0x7f51580d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515cb2d640 1 -- 172.21.15.27:0/1591555165 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802678749 0 0) 0x7f51580d0100 con 0x7f515815e4e0 2024-03-06T07:28:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.925+0000 7f515db2f640 1 -- 172.21.15.27:0/1591555165 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580cfe20 con 0x7f515815e100 2024-03-06T07:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.926+0000 7f515db2f640 1 --2- 172.21.15.27:0/1591555165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f515815e100 0x7f51580d1cd0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f51580677b0 tx=0x7f515402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baae712a3a584277 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.926+0000 7f515cb2d640 1 -- 172.21.15.27:0/1591555165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154035020 con 0x7f515815e100 2024-03-06T07:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.926+0000 7f515cb2d640 1 -- 172.21.15.27:0/1591555165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5154031830 con 0x7f515815e100 2024-03-06T07:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.926+0000 7f515cb2d640 1 -- 172.21.15.27:0/1591555165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154031c00 con 0x7f515815e100 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.927+0000 7f515fd93640 1 -- 172.21.15.27:0/1591555165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f515815e100 msgr2=0x7f51580d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.927+0000 7f515fd93640 1 --2- 172.21.15.27:0/1591555165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f515815e100 0x7f51580d1cd0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f51580677b0 tx=0x7f515402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.927+0000 7f515fd93640 1 -- 172.21.15.27:0/1591555165 shutdown_connections 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.927+0000 7f515fd93640 1 --2- 172.21.15.27:0/1591555165 >> v2:172.21.15.69:3300/0 conn(0x7f51580d4c40 0x7f51580d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.927+0000 7f515fd93640 1 --2- 172.21.15.27:0/1591555165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f515815e100 0x7f51580d1cd0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.927+0000 7f515fd93640 1 -- 172.21.15.27:0/1591555165 >> 172.21.15.27:0/1591555165 conn(0x7f51580d3ef0 msgr2=0x7f51580d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.927+0000 7f515fd93640 1 -- 172.21.15.27:0/1591555165 shutdown_connections 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.927+0000 7f515fd93640 1 -- 172.21.15.27:0/1591555165 wait complete. 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.928+0000 7f515fd93640 1 Processor -- start 2024-03-06T07:28:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.928+0000 7f515fd93640 1 -- start start 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515fd93640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51580d4c40 0x7f51580a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515fd93640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515815e100 0x7f515815c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515fd93640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515815e4e0 0x7f51580b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515fd93640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51580b2080 con 0x7f51580d4c40 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515fd93640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51580cf510 con 0x7f515815e100 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515fd93640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51580b2200 con 0x7f515815e4e0 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515db2f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51580d4c40 0x7f51580a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515db2f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51580d4c40 0x7f51580a15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60186/0 (socket says 172.21.15.27:60186) 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515db2f640 1 -- 172.21.15.27:0/2964783044 learned_addr learned my addr 172.21.15.27:0/2964783044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515d32e640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515815e100 0x7f515815c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515e330640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515815e4e0 0x7f51580b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515db2f640 1 -- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515815e4e0 msgr2=0x7f51580b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515db2f640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515815e4e0 0x7f51580b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515db2f640 1 -- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515815e100 msgr2=0x7f515815c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515db2f640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515815e100 0x7f515815c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515db2f640 1 -- 172.21.15.27:0/2964783044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580a07e0 con 0x7f51580d4c40 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515e330640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515815e4e0 0x7f51580b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f515db2f640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51580d4c40 0x7f51580a15a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f5154000c00 tx=0x7f51540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.929+0000 7f514e7fc640 1 -- 172.21.15.27:0/2964783044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154035070 con 0x7f51580d4c40 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.930+0000 7f514e7fc640 1 -- 172.21.15.27:0/2964783044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5154004250 con 0x7f51580d4c40 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.930+0000 7f514e7fc640 1 -- 172.21.15.27:0/2964783044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515403a6f0 con 0x7f51580d4c40 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.930+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51580a0ac0 con 0x7f51580d4c40 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.930+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51580b4830 con 0x7f51580d4c40 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.931+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51580752d0 con 0x7f51580d4c40 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.931+0000 7f514e7fc640 1 -- 172.21.15.27:0/2964783044 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5154038020 con 0x7f51580d4c40 2024-03-06T07:28:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.931+0000 7f514e7fc640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f513c06b4d0 0x7f513c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.932+0000 7f514e7fc640 1 -- 172.21.15.27:0/2964783044 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f51540abf60 con 0x7f51580d4c40 2024-03-06T07:28:26.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.932+0000 7f515d32e640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f513c06b4d0 0x7f513c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.932+0000 7f515d32e640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f513c06b4d0 0x7f513c06d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f51580cd7b0 tx=0x7f5148008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:26.936+0000 7f514e7fc640 1 -- 172.21.15.27:0/2964783044 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5154040070 con 0x7f51580d4c40 2024-03-06T07:28:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.153+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5158074790 con 0x7f51580d4c40 2024-03-06T07:28:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.153+0000 7f514e7fc640 1 -- 172.21.15.27:0/2964783044 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f515407ae10 con 0x7f51580d4c40 2024-03-06T07:28:27.155 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:27.155 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f513c06b4d0 msgr2=0x7f513c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f513c06b4d0 0x7f513c06d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f51580cd7b0 tx=0x7f5148008040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51580d4c40 msgr2=0x7f51580a15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51580d4c40 0x7f51580a15a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f5154000c00 tx=0x7f51540046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 shutdown_connections 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f513c06b4d0 0x7f513c06d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f515815e4e0 0x7f51580b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f515815e100 0x7f515815c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 --2- 172.21.15.27:0/2964783044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51580d4c40 0x7f51580a15a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 >> 172.21.15.27:0/2964783044 conn(0x7f51580d3ef0 msgr2=0x7f51580d1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 shutdown_connections 2024-03-06T07:28:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.155+0000 7f515fd93640 1 -- 172.21.15.27:0/2964783044 wait complete. 2024-03-06T07:28:27.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:27.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.274+0000 7f985b19a640 1 Processor -- start 2024-03-06T07:28:27.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.274+0000 7f985b19a640 1 -- start start 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f985b19a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f985415e100 0x7f98540d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f985b19a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98540d4c40 0x7f98540d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f985b19a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98540d0100 con 0x7f985415e4e0 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f985b19a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98540cf860 con 0x7f98540d4c40 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f985b19a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98540cfb40 con 0x7f985415e100 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f9853fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98540d4c40 0x7f98540d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f9859737640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f985415e4e0 0x7f98540d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46172/0 (socket says 172.21.15.27:46172) 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f9859737640 1 -- 172.21.15.27:0/350681831 learned_addr learned my addr 172.21.15.27:0/350681831 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f9858f36640 1 --2- 172.21.15.27:0/350681831 >> v2:172.21.15.69:3300/0 conn(0x7f985415e100 0x7f98540d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f9853fff640 1 -- 172.21.15.27:0/350681831 >> v2:172.21.15.69:3300/0 conn(0x7f985415e100 msgr2=0x7f98540d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f9853fff640 1 --2- 172.21.15.27:0/350681831 >> v2:172.21.15.69:3300/0 conn(0x7f985415e100 0x7f98540d1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f9853fff640 1 -- 172.21.15.27:0/350681831 >> v1:172.21.15.27:6789/0 conn(0x7f985415e4e0 legacy=0x7f98540d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f9853fff640 1 -- 172.21.15.27:0/350681831 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98540cfe20 con 0x7f98540d4c40 2024-03-06T07:28:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f98537fe640 1 -- 172.21.15.27:0/350681831 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697559680 0 0) 0x7f98540d0100 con 0x7f985415e4e0 2024-03-06T07:28:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.275+0000 7f9858f36640 1 --2- 172.21.15.27:0/350681831 >> v2:172.21.15.69:3300/0 conn(0x7f985415e100 0x7f98540d1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.276+0000 7f9853fff640 1 --2- 172.21.15.27:0/350681831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540d4c40 0x7f98540d2320 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f985408e3a0 tx=0x7f984002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdff2194fa070d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.276+0000 7f98537fe640 1 -- 172.21.15.27:0/350681831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840035020 con 0x7f98540d4c40 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.276+0000 7f98537fe640 1 -- 172.21.15.27:0/350681831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9840031830 con 0x7f98540d4c40 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.276+0000 7f98537fe640 1 -- 172.21.15.27:0/350681831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840031c00 con 0x7f98540d4c40 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.277+0000 7f985b19a640 1 -- 172.21.15.27:0/350681831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540d4c40 msgr2=0x7f98540d2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.277+0000 7f985b19a640 1 --2- 172.21.15.27:0/350681831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540d4c40 0x7f98540d2320 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f985408e3a0 tx=0x7f984002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.277+0000 7f985b19a640 1 -- 172.21.15.27:0/350681831 shutdown_connections 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.277+0000 7f985b19a640 1 --2- 172.21.15.27:0/350681831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540d4c40 0x7f98540d2320 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.277+0000 7f985b19a640 1 --2- 172.21.15.27:0/350681831 >> v2:172.21.15.69:3300/0 conn(0x7f985415e100 0x7f98540d1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.277+0000 7f985b19a640 1 -- 172.21.15.27:0/350681831 >> 172.21.15.27:0/350681831 conn(0x7f98540d3ef0 msgr2=0x7f98540d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.277+0000 7f985b19a640 1 -- 172.21.15.27:0/350681831 shutdown_connections 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.277+0000 7f985b19a640 1 -- 172.21.15.27:0/350681831 wait complete. 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.278+0000 7f985b19a640 1 Processor -- start 2024-03-06T07:28:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.278+0000 7f985b19a640 1 -- start start 2024-03-06T07:28:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f985b19a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98540d4c40 0x7f9854160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f985b19a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f985415e100 0x7f985415f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f985b19a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f985415e4e0 0x7f98540b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f985b19a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98540b2080 con 0x7f985415e100 2024-03-06T07:28:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f985b19a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98540cf510 con 0x7f985415e4e0 2024-03-06T07:28:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f985b19a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98540b2200 con 0x7f98540d4c40 2024-03-06T07:28:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9853fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f985415e100 0x7f985415f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9853fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f985415e100 0x7f985415f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60206/0 (socket says 172.21.15.27:60206) 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9853fff640 1 -- 172.21.15.27:0/2362771670 learned_addr learned my addr 172.21.15.27:0/2362771670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9859737640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f985415e4e0 0x7f98540b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9858f36640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98540d4c40 0x7f9854160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9853fff640 1 -- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98540d4c40 msgr2=0x7f9854160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9853fff640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98540d4c40 0x7f9854160670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9853fff640 1 -- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f985415e4e0 msgr2=0x7f98540b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9853fff640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f985415e4e0 0x7f98540b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9853fff640 1 -- 172.21.15.27:0/2362771670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98540b8120 con 0x7f985415e100 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.279+0000 7f9853fff640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f985415e100 0x7f985415f580 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f9840009890 tx=0x7f98400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.280+0000 7f98517fa640 1 -- 172.21.15.27:0/2362771670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840041070 con 0x7f985415e100 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.280+0000 7f98517fa640 1 -- 172.21.15.27:0/2362771670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984003aa30 con 0x7f985415e100 2024-03-06T07:28:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.280+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98540b83f0 con 0x7f985415e100 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.280+0000 7f98517fa640 1 -- 172.21.15.27:0/2362771670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840035040 con 0x7f985415e100 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.280+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985415ed80 con 0x7f985415e100 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.281+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98540752d0 con 0x7f985415e100 2024-03-06T07:28:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.281+0000 7f98517fa640 1 -- 172.21.15.27:0/2362771670 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9840038020 con 0x7f985415e100 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.282+0000 7f98517fa640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f983806b4d0 0x7f983806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.282+0000 7f98517fa640 1 -- 172.21.15.27:0/2362771670 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f98400abcd0 con 0x7f985415e100 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.282+0000 7f9858f36640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f983806b4d0 0x7f983806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.283+0000 7f9858f36640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f983806b4d0 0x7f983806d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f985408d7d0 tx=0x7f9848008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.286+0000 7f98517fa640 1 -- 172.21.15.27:0/2362771670 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f984007ab80 con 0x7f985415e100 2024-03-06T07:28:27.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.506+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98540b2f20 con 0x7f985415e100 2024-03-06T07:28:27.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.507+0000 7f98517fa640 1 -- 172.21.15.27:0/2362771670 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f984007a480 con 0x7f985415e100 2024-03-06T07:28:27.508 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:27.508 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:48.137883+0000","modified":"2024-03-06T07:28:01.537159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6834/550876663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":550876663},{"type":"v1","addr":"172.21.15.27:6834","nonce":550876663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:28:27.509 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f983806b4d0 msgr2=0x7f983806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f983806b4d0 0x7f983806d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f985408d7d0 tx=0x7f9848008040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f985415e100 msgr2=0x7f985415f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f985415e100 0x7f985415f580 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f9840009890 tx=0x7f98400046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 shutdown_connections 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f983806b4d0 0x7f983806d990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f985415e4e0 0x7f98540b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f985415e100 0x7f985415f580 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 --2- 172.21.15.27:0/2362771670 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98540d4c40 0x7f9854160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 >> 172.21.15.27:0/2362771670 conn(0x7f98540d3ef0 msgr2=0x7f985406b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 shutdown_connections 2024-03-06T07:28:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.509+0000 7f985b19a640 1 -- 172.21.15.27:0/2362771670 wait complete. 2024-03-06T07:28:27.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.627+0000 7f4500fbc640 1 Processor -- start 2024-03-06T07:28:27.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.627+0000 7f4500fbc640 1 -- start start 2024-03-06T07:28:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.628+0000 7f4500fbc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44fc15e100 0x7f44fc0d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.628+0000 7f4500fbc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f44fc0d4c40 0x7f44fc0d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.628+0000 7f4500fbc640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44fc0d0100 con 0x7f44fc15e4e0 2024-03-06T07:28:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.628+0000 7f4500fbc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44fc0cf860 con 0x7f44fc15e100 2024-03-06T07:28:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.628+0000 7f4500fbc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44fc0cfb40 con 0x7f44fc0d4c40 2024-03-06T07:28:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.628+0000 7f44fa59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44fc15e100 0x7f44fc0d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.628+0000 7f44fad9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f44fc15e4e0 0x7f44fc0d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46174/0 (socket says 172.21.15.27:46174) 2024-03-06T07:28:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.628+0000 7f44fad9d640 1 -- 172.21.15.27:0/2919986107 learned_addr learned my addr 172.21.15.27:0/2919986107 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.628+0000 7f44f9d9b640 1 --2- 172.21.15.27:0/2919986107 >> v2:172.21.15.69:3300/0 conn(0x7f44fc0d4c40 0x7f44fc0d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.629+0000 7f44fa59c640 1 -- 172.21.15.27:0/2919986107 >> v2:172.21.15.69:3300/0 conn(0x7f44fc0d4c40 msgr2=0x7f44fc0d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.629+0000 7f44fa59c640 1 --2- 172.21.15.27:0/2919986107 >> v2:172.21.15.69:3300/0 conn(0x7f44fc0d4c40 0x7f44fc0d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.629+0000 7f44fa59c640 1 -- 172.21.15.27:0/2919986107 >> v1:172.21.15.27:6789/0 conn(0x7f44fc15e4e0 legacy=0x7f44fc0d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.629+0000 7f44f959a640 1 -- 172.21.15.27:0/2919986107 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466886620 0 0) 0x7f44fc0d0100 con 0x7f44fc15e4e0 2024-03-06T07:28:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.629+0000 7f44fa59c640 1 -- 172.21.15.27:0/2919986107 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc0cfe20 con 0x7f44fc15e100 2024-03-06T07:28:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.629+0000 7f44fa59c640 1 --2- 172.21.15.27:0/2919986107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc15e100 0x7f44fc0d1cd0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0677b0 tx=0x7f44ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04d129746cd5644 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f44f959a640 1 -- 172.21.15.27:0/2919986107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec035020 con 0x7f44fc15e100 2024-03-06T07:28:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f44f959a640 1 -- 172.21.15.27:0/2919986107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec031830 con 0x7f44fc15e100 2024-03-06T07:28:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f44f959a640 1 -- 172.21.15.27:0/2919986107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec031c00 con 0x7f44fc15e100 2024-03-06T07:28:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f4500fbc640 1 -- 172.21.15.27:0/2919986107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc15e100 msgr2=0x7f44fc0d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f4500fbc640 1 --2- 172.21.15.27:0/2919986107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc15e100 0x7f44fc0d1cd0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0677b0 tx=0x7f44ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f4500fbc640 1 -- 172.21.15.27:0/2919986107 shutdown_connections 2024-03-06T07:28:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f4500fbc640 1 --2- 172.21.15.27:0/2919986107 >> v2:172.21.15.69:3300/0 conn(0x7f44fc0d4c40 0x7f44fc0d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f4500fbc640 1 --2- 172.21.15.27:0/2919986107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc15e100 0x7f44fc0d1cd0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f4500fbc640 1 -- 172.21.15.27:0/2919986107 >> 172.21.15.27:0/2919986107 conn(0x7f44fc0d3ef0 msgr2=0x7f44fc0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.630+0000 7f4500fbc640 1 -- 172.21.15.27:0/2919986107 shutdown_connections 2024-03-06T07:28:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.631+0000 7f4500fbc640 1 -- 172.21.15.27:0/2919986107 wait complete. 2024-03-06T07:28:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.632+0000 7f4500fbc640 1 Processor -- start 2024-03-06T07:28:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.632+0000 7f4500fbc640 1 -- start start 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.632+0000 7f4500fbc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc0d4c40 0x7f44fc0a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.632+0000 7f4500fbc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44fc15e100 0x7f44fc15c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.632+0000 7f4500fbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44fc15e4e0 0x7f44fc0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.632+0000 7f4500fbc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc0b2080 con 0x7f44fc0d4c40 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.632+0000 7f4500fbc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc0cf510 con 0x7f44fc15e100 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f4500fbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc0b2200 con 0x7f44fc15e4e0 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc0d4c40 0x7f44fc0a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc0d4c40 0x7f44fc0a15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60232/0 (socket says 172.21.15.27:60232) 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fa59c640 1 -- 172.21.15.27:0/1971260715 learned_addr learned my addr 172.21.15.27:0/1971260715 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44f9d9b640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44fc15e100 0x7f44fc15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fad9d640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44fc15e4e0 0x7f44fc0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fa59c640 1 -- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44fc15e4e0 msgr2=0x7f44fc0b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fa59c640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44fc15e4e0 0x7f44fc0b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fa59c640 1 -- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44fc15e100 msgr2=0x7f44fc15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fa59c640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44fc15e100 0x7f44fc15c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fa59c640 1 -- 172.21.15.27:0/1971260715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc0a07e0 con 0x7f44fc0d4c40 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44f9d9b640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44fc15e100 0x7f44fc15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44fa59c640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc0d4c40 0x7f44fc0a15a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009890 tx=0x7f44ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44eaffd640 1 -- 172.21.15.27:0/1971260715 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec041070 con 0x7f44fc0d4c40 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44eaffd640 1 -- 172.21.15.27:0/1971260715 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec03aa30 con 0x7f44fc0d4c40 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44fc0a0ac0 con 0x7f44fc0d4c40 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f44eaffd640 1 -- 172.21.15.27:0/1971260715 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec035040 con 0x7f44fc0d4c40 2024-03-06T07:28:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.633+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44fc0b4830 con 0x7f44fc0d4c40 2024-03-06T07:28:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.635+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc0752d0 con 0x7f44fc0d4c40 2024-03-06T07:28:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.635+0000 7f44eaffd640 1 -- 172.21.15.27:0/1971260715 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44ec038020 con 0x7f44fc0d4c40 2024-03-06T07:28:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.635+0000 7f44eaffd640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44dc06b4d0 0x7f44dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.635+0000 7f44eaffd640 1 -- 172.21.15.27:0/1971260715 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f44ec0abbe0 con 0x7f44fc0d4c40 2024-03-06T07:28:27.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.636+0000 7f44f9d9b640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44dc06b4d0 0x7f44dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.636+0000 7f44f9d9b640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44dc06b4d0 0x7f44dc06d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0cd7b0 tx=0x7f44f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.640+0000 7f44eaffd640 1 -- 172.21.15.27:0/1971260715 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44ec07aa90 con 0x7f44fc0d4c40 2024-03-06T07:28:27.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.826+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44fc074790 con 0x7f44fc0d4c40 2024-03-06T07:28:27.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.827+0000 7f44eaffd640 1 -- 172.21.15.27:0/1971260715 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f44ec07a390 con 0x7f44fc0d4c40 2024-03-06T07:28:27.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:27.829 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:27:57.839900+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:46.443375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:47.116887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/3147032864":"2024-03-07T07:27:57.839873+0000","172.21.15.27:6833/3147032864":"2024-03-07T07:27:57.839873+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:27.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44dc06b4d0 msgr2=0x7f44dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44dc06b4d0 0x7f44dc06d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0cd7b0 tx=0x7f44f0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc0d4c40 msgr2=0x7f44fc0a15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc0d4c40 0x7f44fc0a15a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009890 tx=0x7f44ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 shutdown_connections 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44dc06b4d0 0x7f44dc06d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44fc15e4e0 0x7f44fc0b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44fc15e100 0x7f44fc15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 --2- 172.21.15.27:0/1971260715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44fc0d4c40 0x7f44fc0a15a0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 >> 172.21.15.27:0/1971260715 conn(0x7f44fc0d3ef0 msgr2=0x7f44fc0d1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 shutdown_connections 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.829+0000 7f4500fbc640 1 -- 172.21.15.27:0/1971260715 wait complete. 2024-03-06T07:28:27.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:27.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.968+0000 7f8343847640 1 Processor -- start 2024-03-06T07:28:27.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.968+0000 7f8343847640 1 -- start start 2024-03-06T07:28:27.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f8343847640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0d2680 0x7f833c0d4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f8343847640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f833c0b3050 0x7f833c0b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f8343847640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0ceab0 con 0x7f833c0b1040 2024-03-06T07:28:27.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f8343847640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0cf380 con 0x7f833c0b3050 2024-03-06T07:28:27.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f8343847640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0d1d10 con 0x7f833c0d2680 2024-03-06T07:28:27.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f8340de2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f833c0b3050 0x7f833c0b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f83415e3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0d2680 0x7f833c0d4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f8340de2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f833c0b3050 0x7f833c0b0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:44636/0 (socket says 172.21.15.27:44636) 2024-03-06T07:28:27.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f83415e3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0d2680 0x7f833c0d4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:41726/0 (socket says 172.21.15.27:41726) 2024-03-06T07:28:27.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f8341de4640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f833c0b1040 0x7f833c0cf810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:54466/0 (socket says 172.21.15.27:54466) 2024-03-06T07:28:27.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.969+0000 7f8340de2640 1 -- 172.21.15.27:0/2496746769 learned_addr learned my addr 172.21.15.27:0/2496746769 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:27.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.970+0000 7f8333fff640 1 -- 172.21.15.27:0/2496746769 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456537407 0 0) 0x7f8334001490 con 0x7f833c0b1040 2024-03-06T07:28:27.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.970+0000 7f8333fff640 1 -- 172.21.15.27:0/2496746769 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8328003520 con 0x7f833c0b1040 2024-03-06T07:28:27.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.970+0000 7f8340de2640 1 -- 172.21.15.27:0/2496746769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0d2680 msgr2=0x7f833c0d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.970+0000 7f8340de2640 1 --2- 172.21.15.27:0/2496746769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0d2680 0x7f833c0d4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.970+0000 7f8340de2640 1 -- 172.21.15.27:0/2496746769 >> v1:172.21.15.69:6789/0 conn(0x7f833c0b1040 legacy=0x7f833c0cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.970+0000 7f8340de2640 1 -- 172.21.15.27:0/2496746769 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0d1ff0 con 0x7f833c0b3050 2024-03-06T07:28:27.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.970+0000 7f83415e3640 1 --2- 172.21.15.27:0/2496746769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0d2680 0x7f833c0d4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:27.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.970+0000 7f8340de2640 1 --2- 172.21.15.27:0/2496746769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833c0b3050 0x7f833c0b0b00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f832c009870 tx=0x7f832c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad7e73a3dbbd188f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.971+0000 7f8333fff640 1 -- 172.21.15.27:0/2496746769 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c034070 con 0x7f833c0b3050 2024-03-06T07:28:27.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.971+0000 7f8333fff640 1 -- 172.21.15.27:0/2496746769 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c030840 con 0x7f833c0b3050 2024-03-06T07:28:27.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.971+0000 7f8333fff640 1 -- 172.21.15.27:0/2496746769 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c030c10 con 0x7f833c0b3050 2024-03-06T07:28:27.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.971+0000 7f8343847640 1 -- 172.21.15.27:0/2496746769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833c0b3050 msgr2=0x7f833c0b0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.971+0000 7f8343847640 1 --2- 172.21.15.27:0/2496746769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833c0b3050 0x7f833c0b0b00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f832c009870 tx=0x7f832c02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:27.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.971+0000 7f8343847640 1 -- 172.21.15.27:0/2496746769 shutdown_connections 2024-03-06T07:28:27.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.971+0000 7f8343847640 1 --2- 172.21.15.27:0/2496746769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833c0b3050 0x7f833c0b0b00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.972+0000 7f8343847640 1 --2- 172.21.15.27:0/2496746769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0d2680 0x7f833c0d4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.972+0000 7f8343847640 1 -- 172.21.15.27:0/2496746769 >> 172.21.15.27:0/2496746769 conn(0x7f833c0d3f30 msgr2=0x7f833c0d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.972+0000 7f8343847640 1 -- 172.21.15.27:0/2496746769 shutdown_connections 2024-03-06T07:28:27.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.972+0000 7f8343847640 1 -- 172.21.15.27:0/2496746769 wait complete. 2024-03-06T07:28:27.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.974+0000 7f8343847640 1 Processor -- start 2024-03-06T07:28:27.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.974+0000 7f8343847640 1 -- start start 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f8343847640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f833c0b1040 0x7f833c0b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f8343847640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0b3050 0x7f833c151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f8343847640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833c0d2680 0x7f833c0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f8343847640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0cff20 con 0x7f833c0b1040 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f8343847640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0cf010 con 0x7f833c0d2680 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f8343847640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0d00a0 con 0x7f833c0b3050 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f83415e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f833c0b1040 0x7f833c0b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f83415e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f833c0b1040 0x7f833c0b96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60246/0 (socket says 172.21.15.27:60246) 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f83415e3640 1 -- 172.21.15.27:0/1160534404 learned_addr learned my addr 172.21.15.27:0/1160534404 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f8340de2640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0b3050 0x7f833c151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.975+0000 7f8341de4640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833c0d2680 0x7f833c0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f83415e3640 1 -- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0b3050 msgr2=0x7f833c151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f83415e3640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0b3050 0x7f833c151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f83415e3640 1 -- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833c0d2680 msgr2=0x7f833c0c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f83415e3640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833c0d2680 0x7f833c0c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f83415e3640 1 -- 172.21.15.27:0/1160534404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0c8ce0 con 0x7f833c0b1040 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f83415e3640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f833c0b1040 0x7f833c0b96e0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f833800e960 tx=0x7f833800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f8331ffb640 1 -- 172.21.15.27:0/1160534404 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338017070 con 0x7f833c0b1040 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f8331ffb640 1 -- 172.21.15.27:0/1160534404 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833800ca00 con 0x7f833c0b1040 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c0c8fc0 con 0x7f833c0b1040 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f8331ffb640 1 -- 172.21.15.27:0/1160534404 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833800cdd0 con 0x7f833c0b1040 2024-03-06T07:28:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.976+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c0c9fe0 con 0x7f833c0b1040 2024-03-06T07:28:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.978+0000 7f8331ffb640 1 -- 172.21.15.27:0/1160534404 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f833801a070 con 0x7f833c0b1040 2024-03-06T07:28:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.978+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c0752d0 con 0x7f833c0b1040 2024-03-06T07:28:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.978+0000 7f8331ffb640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f832006b500 0x7f832006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.978+0000 7f8331ffb640 1 -- 172.21.15.27:0/1160534404 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f833808a350 con 0x7f833c0b1040 2024-03-06T07:28:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.978+0000 7f8340de2640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f832006b500 0x7f832006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.979+0000 7f8340de2640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f832006b500 0x7f832006d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f832c036000 tx=0x7f832c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:27.982+0000 7f8331ffb640 1 -- 172.21.15.27:0/1160534404 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8338058bb0 con 0x7f833c0b1040 2024-03-06T07:28:28.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.202+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f833c0b5800 con 0x7f833c0b1040 2024-03-06T07:28:28.204 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:28:28.203+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:28.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.204+0000 7f8331ffb640 1 -- 172.21.15.27:0/1160534404 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) v1 ==== 161+0+0 (secure 0 0 0) 0x7f833805cc50 con 0x7f833c0b1040 2024-03-06T07:28:28.206 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.207+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f832006b500 msgr2=0x7f832006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.207+0000 7f8343847640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f832006b500 0x7f832006d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f832c036000 tx=0x7f832c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.207+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f833c0b1040 msgr2=0x7f833c0b96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.207+0000 7f8343847640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f833c0b1040 0x7f833c0b96e0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f833800e960 tx=0x7f833800ee20 comp rx=0 tx=0).stop 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.208+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 shutdown_connections 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.208+0000 7f8343847640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f832006b500 0x7f832006d9c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.208+0000 7f8343847640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f833c0d2680 0x7f833c0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.208+0000 7f8343847640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f833c0b3050 0x7f833c151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.208+0000 7f8343847640 1 --2- 172.21.15.27:0/1160534404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f833c0b1040 0x7f833c0b96e0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.208+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 >> 172.21.15.27:0/1160534404 conn(0x7f833c0d3f30 msgr2=0x7f833c160860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.208+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 shutdown_connections 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.208+0000 7f8343847640 1 -- 172.21.15.27:0/1160534404 wait complete. 2024-03-06T07:28:28.222 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:28.250 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:28:28.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.331+0000 7f778951e640 1 Processor -- start 2024-03-06T07:28:28.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.331+0000 7f778951e640 1 -- start start 2024-03-06T07:28:28.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f778951e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f77840cd9f0 0x7f7784167050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f778951e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840cf6c0 con 0x7f778415f730 2024-03-06T07:28:28.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f778951e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840b1e10 con 0x7f77840d28c0 2024-03-06T07:28:28.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f778951e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77840cc8b0 con 0x7f77840cd9f0 2024-03-06T07:28:28.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7782ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f778415f730 0x7f77840b1700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46180/0 (socket says 172.21.15.27:46180) 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7782ffd640 1 -- 172.21.15.27:0/1779798999 learned_addr learned my addr 172.21.15.27:0/1779798999 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f77837fe640 1 --2- 172.21.15.27:0/1779798999 >> v2:172.21.15.69:3300/0 conn(0x7f77840cd9f0 0x7f7784167050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275656244 0 0) 0x7f77840cf6c0 con 0x7f778415f730 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f776c003520 con 0x7f778415f730 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575019476 0 0) 0x7f776c003520 con 0x7f778415f730 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 >> v2:172.21.15.69:3300/0 conn(0x7f77840cd9f0 msgr2=0x7f7784167050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7781ffb640 1 --2- 172.21.15.27:0/1779798999 >> v2:172.21.15.69:3300/0 conn(0x7f77840cd9f0 0x7f7784167050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 >> v1:172.21.15.69:6789/0 conn(0x7f77840d28c0 legacy=0x7f77840cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77840cf100 con 0x7f778415f730 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.332+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f7774001670 con 0x7f778415f730 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 >> v1:172.21.15.27:6789/0 conn(0x7f778415f730 legacy=0x7f77840b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f7781ffb640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f776c0039b0 0x7f776c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f7781ffb640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f776c004800 0x7f776c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f7781ffb640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f776c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c00c040 con 0x7f776c0051c0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c00c1f0 con 0x7f776c0039b0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776c00c3f0 con 0x7f776c004800 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f77837fe640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f776c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f7782ffd640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f776c0039b0 0x7f776c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f77827fc640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f776c004800 0x7f776c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f77837fe640 1 -- 172.21.15.27:0/1779798999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f776c004800 msgr2=0x7f776c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f77837fe640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f776c004800 0x7f776c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f77837fe640 1 -- 172.21.15.27:0/1779798999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f776c0039b0 msgr2=0x7f776c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f77837fe640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f776c0039b0 0x7f776c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.333+0000 7f77837fe640 1 -- 172.21.15.27:0/1779798999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c003520 con 0x7f776c0051c0 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f77837fe640 1 -- 172.21.15.27:0/1779798999 reap_dead start 2024-03-06T07:28:28.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f77837fe640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f776c00b9c0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f777c009fc0 tx=0x7f776c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41669ca86a1008b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c009c10 con 0x7f776c0051c0 2024-03-06T07:28:28.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c02cca0 con 0x7f776c0051c0 2024-03-06T07:28:28.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f7781ffb640 1 -- 172.21.15.27:0/1779798999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c003bd0 con 0x7f776c0051c0 2024-03-06T07:28:28.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f778951e640 1 -- 172.21.15.27:0/1779798999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 msgr2=0x7f776c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f778951e640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f776c00b9c0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f777c009fc0 tx=0x7f776c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:28.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f778951e640 1 -- 172.21.15.27:0/1779798999 shutdown_connections 2024-03-06T07:28:28.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f778951e640 1 --2- 172.21.15.27:0/1779798999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f776c00b9c0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.334+0000 7f778951e640 1 -- 172.21.15.27:0/1779798999 >> 172.21.15.27:0/1779798999 conn(0x7f77840d1da0 msgr2=0x7f77840d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.335+0000 7f778951e640 1 -- 172.21.15.27:0/1779798999 shutdown_connections 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.335+0000 7f778951e640 1 -- 172.21.15.27:0/1779798999 wait complete. 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.336+0000 7f778951e640 1 Processor -- start 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.336+0000 7f778951e640 1 -- start start 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f778951e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f77840cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f778951e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77841552e0 0x7f77840c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f778951e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77840c6d70 0x7f77840c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f778951e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77840ccae0 con 0x7f776c0051c0 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f778951e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778415fb10 con 0x7f77841552e0 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f778951e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77840ccdc0 con 0x7f77840c6d70 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f7782ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f77840cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f7782ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f77840cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60292/0 (socket says 172.21.15.27:60292) 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f7782ffd640 1 -- 172.21.15.27:0/3324649774 learned_addr learned my addr 172.21.15.27:0/3324649774 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f77837fe640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77840c6d70 0x7f77840c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f77827fc640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77841552e0 0x7f77840c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f7782ffd640 1 -- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77840c6d70 msgr2=0x7f77840c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f7782ffd640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77840c6d70 0x7f77840c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f7782ffd640 1 -- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77841552e0 msgr2=0x7f77840c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f7782ffd640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77841552e0 0x7f77840c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f7782ffd640 1 -- 172.21.15.27:0/3324649774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778406c4e0 con 0x7f776c0051c0 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f77827fc640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77841552e0 0x7f77840c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f7782ffd640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f77840cb610 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f77740011e0 tx=0x7f7774007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f776b7fe640 1 -- 172.21.15.27:0/3324649774 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777400f960 con 0x7f776c0051c0 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f776b7fe640 1 -- 172.21.15.27:0/3324649774 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774018070 con 0x7f776c0051c0 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f776b7fe640 1 -- 172.21.15.27:0/3324649774 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777400fd60 con 0x7f776c0051c0 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77840b5d00 con 0x7f776c0051c0 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.337+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77840b62c0 con 0x7f776c0051c0 2024-03-06T07:28:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.338+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7784068050 con 0x7f776c0051c0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.339+0000 7f776b7fe640 1 -- 172.21.15.27:0/3324649774 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f777401d460 con 0x7f776c0051c0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.339+0000 7f776b7fe640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777006b500 0x7f777006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.339+0000 7f776b7fe640 1 -- 172.21.15.27:0/3324649774 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f777408a3b0 con 0x7f776c0051c0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.339+0000 7f77827fc640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777006b500 0x7f777006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.340+0000 7f77827fc640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777006b500 0x7f777006d9c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7778000c00 tx=0x7f7778009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.344+0000 7f776b7fe640 1 -- 172.21.15.27:0/3324649774 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77740590f0 con 0x7f776c0051c0 2024-03-06T07:28:28.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:28.562+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f77840b7d20 con 0x7f776c0051c0 2024-03-06T07:28:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.205+0000 7f776b7fe640 1 -- 172.21.15.27:0/3324649774 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7f77740589f0 con 0x7f776c0051c0 2024-03-06T07:28:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777006b500 msgr2=0x7f777006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777006b500 0x7f777006d9c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7778000c00 tx=0x7f7778009040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 msgr2=0x7f77840cb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f77840cb610 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f77740011e0 tx=0x7f7774007720 comp rx=0 tx=0).stop 2024-03-06T07:28:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 shutdown_connections 2024-03-06T07:28:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f777006b500 0x7f777006d9c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77840c6d70 0x7f77840c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77841552e0 0x7f77840c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 --2- 172.21.15.27:0/3324649774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f776c0051c0 0x7f77840cb610 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 >> 172.21.15.27:0/3324649774 conn(0x7f77840d1da0 msgr2=0x7f778406eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 shutdown_connections 2024-03-06T07:28:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.209+0000 7f778951e640 1 -- 172.21.15.27:0/3324649774 wait complete. 2024-03-06T07:28:29.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:29.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.376+0000 7f2e1b22a640 1 Processor -- start 2024-03-06T07:28:29.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.376+0000 7f2e1b22a640 1 -- start start 2024-03-06T07:28:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e1b22a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e140b29f0 0x7f2e140ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e1b22a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140cd1d0 0x7f2e140b0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e1b22a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e140cd5e0 con 0x7f2e1415e390 2024-03-06T07:28:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e1b22a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1415e770 con 0x7f2e140b29f0 2024-03-06T07:28:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e1b22a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140b5c40 con 0x7f2e140cd1d0 2024-03-06T07:28:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e13fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e140b29f0 0x7f2e140ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e13fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e140b29f0 0x7f2e140ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60306/0 (socket says 172.21.15.27:60306) 2024-03-06T07:28:29.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e13fff640 1 -- 172.21.15.27:0/3584717957 learned_addr learned my addr 172.21.15.27:0/3584717957 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:29.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e197c7640 1 --2- 172.21.15.27:0/3584717957 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140cd1d0 0x7f2e140b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.377+0000 7f2e13fff640 1 -- 172.21.15.27:0/3584717957 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140cd1d0 msgr2=0x7f2e140b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.378+0000 7f2e13fff640 1 --2- 172.21.15.27:0/3584717957 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140cd1d0 0x7f2e140b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.378+0000 7f2e13fff640 1 -- 172.21.15.27:0/3584717957 >> v1:172.21.15.69:6789/0 conn(0x7f2e1415e390 legacy=0x7f2e14066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.378+0000 7f2e13fff640 1 -- 172.21.15.27:0/3584717957 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e140b5030 con 0x7f2e140b29f0 2024-03-06T07:28:29.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.378+0000 7f2e197c7640 1 --2- 172.21.15.27:0/3584717957 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140cd1d0 0x7f2e140b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.378+0000 7f2e13fff640 1 --2- 172.21.15.27:0/3584717957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140b29f0 0x7f2e140ccc90 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f2e140b17e0 tx=0x7f2e0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25e0ab58eadb34c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.378+0000 7f2e137fe640 1 -- 172.21.15.27:0/3584717957 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e080315a0 con 0x7f2e140b29f0 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.378+0000 7f2e137fe640 1 -- 172.21.15.27:0/3584717957 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e08037070 con 0x7f2e140b29f0 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.379+0000 7f2e137fe640 1 -- 172.21.15.27:0/3584717957 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08031960 con 0x7f2e140b29f0 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.379+0000 7f2e1b22a640 1 -- 172.21.15.27:0/3584717957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140b29f0 msgr2=0x7f2e140ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.379+0000 7f2e1b22a640 1 --2- 172.21.15.27:0/3584717957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140b29f0 0x7f2e140ccc90 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f2e140b17e0 tx=0x7f2e0802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.379+0000 7f2e1b22a640 1 -- 172.21.15.27:0/3584717957 shutdown_connections 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.379+0000 7f2e1b22a640 1 --2- 172.21.15.27:0/3584717957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140b29f0 0x7f2e140ccc90 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.379+0000 7f2e1b22a640 1 --2- 172.21.15.27:0/3584717957 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140cd1d0 0x7f2e140b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.379+0000 7f2e1b22a640 1 -- 172.21.15.27:0/3584717957 >> 172.21.15.27:0/3584717957 conn(0x7f2e140b1ed0 msgr2=0x7f2e140b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.379+0000 7f2e1b22a640 1 -- 172.21.15.27:0/3584717957 shutdown_connections 2024-03-06T07:28:29.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.379+0000 7f2e1b22a640 1 -- 172.21.15.27:0/3584717957 wait complete. 2024-03-06T07:28:29.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.380+0000 7f2e1b22a640 1 Processor -- start 2024-03-06T07:28:29.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.380+0000 7f2e1b22a640 1 -- start start 2024-03-06T07:28:29.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e1b22a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140b29f0 0x7f2e140ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e1b22a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140cd1d0 0x7f2e140c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e1b22a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e1415e390 0x7f2e140cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e1b22a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140b58d0 con 0x7f2e140cd1d0 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e1b22a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140b55f0 con 0x7f2e1415e390 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e1b22a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140b60d0 con 0x7f2e140b29f0 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e13fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140cd1d0 0x7f2e140c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e13fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140cd1d0 0x7f2e140c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60322/0 (socket says 172.21.15.27:60322) 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e13fff640 1 -- 172.21.15.27:0/2837369312 learned_addr learned my addr 172.21.15.27:0/2837369312 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e18fc6640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140b29f0 0x7f2e140ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e197c7640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e1415e390 0x7f2e140cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e13fff640 1 -- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140b29f0 msgr2=0x7f2e140ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.381+0000 7f2e13fff640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140b29f0 0x7f2e140ca9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.382+0000 7f2e13fff640 1 -- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e1415e390 msgr2=0x7f2e140cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.382+0000 7f2e13fff640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e1415e390 0x7f2e140cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.382+0000 7f2e13fff640 1 -- 172.21.15.27:0/2837369312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e141f3ac0 con 0x7f2e140cd1d0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.382+0000 7f2e13fff640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140cd1d0 0x7f2e140c3e30 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f2e0802b880 tx=0x7f2e080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.382+0000 7f2e117fa640 1 -- 172.21.15.27:0/2837369312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08032840 con 0x7f2e140cd1d0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.382+0000 7f2e117fa640 1 -- 172.21.15.27:0/2837369312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e080329e0 con 0x7f2e140cd1d0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.382+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e140c7620 con 0x7f2e140cd1d0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.382+0000 7f2e117fa640 1 -- 172.21.15.27:0/2837369312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08032dd0 con 0x7f2e140cd1d0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.382+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e140cb3c0 con 0x7f2e140cd1d0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.383+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e140cb5c0 con 0x7f2e140cd1d0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.383+0000 7f2e117fa640 1 -- 172.21.15.27:0/2837369312 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e0803b3f0 con 0x7f2e140cd1d0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.384+0000 7f2e117fa640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e0006b4d0 0x7f2e0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.384+0000 7f2e117fa640 1 -- 172.21.15.27:0/2837369312 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f2e0802f030 con 0x7f2e140cd1d0 2024-03-06T07:28:29.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.384+0000 7f2e18fc6640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e0006b4d0 0x7f2e0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.385+0000 7f2e18fc6640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e0006b4d0 0x7f2e0006d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00b450 tx=0x7f2e0c00d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.388+0000 7f2e117fa640 1 -- 172.21.15.27:0/2837369312 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e140cb5c0 con 0x7f2e140cd1d0 2024-03-06T07:28:29.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:29.570+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2e140752d0 con 0x7f2e140cd1d0 2024-03-06T07:28:30.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.207+0000 7f2e117fa640 1 -- 172.21.15.27:0/2837369312 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2e08040070 con 0x7f2e140cd1d0 2024-03-06T07:28:30.208 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:30.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e0006b4d0 msgr2=0x7f2e0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e0006b4d0 0x7f2e0006d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00b450 tx=0x7f2e0c00d040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140cd1d0 msgr2=0x7f2e140c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140cd1d0 0x7f2e140c3e30 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f2e0802b880 tx=0x7f2e080046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 shutdown_connections 2024-03-06T07:28:30.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e0006b4d0 0x7f2e0006d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e1415e390 0x7f2e140cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e140cd1d0 0x7f2e140c3e30 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 --2- 172.21.15.27:0/2837369312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e140b29f0 0x7f2e140ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 >> 172.21.15.27:0/2837369312 conn(0x7f2e140b1ed0 msgr2=0x7f2e140cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.210+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 shutdown_connections 2024-03-06T07:28:30.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.211+0000 7f2e1b22a640 1 -- 172.21.15.27:0/2837369312 wait complete. 2024-03-06T07:28:30.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:30.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.341+0000 7f33d7193640 1 Processor -- start 2024-03-06T07:28:30.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.341+0000 7f33d7193640 1 -- start start 2024-03-06T07:28:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.341+0000 7f33d7193640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33d015e380 0x7f33d00ced70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.341+0000 7f33d7193640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d015e760 con 0x7f33d00cde30 2024-03-06T07:28:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.341+0000 7f33d7193640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d00cb530 con 0x7f33d00cac50 2024-03-06T07:28:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.341+0000 7f33d7193640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d00cb810 con 0x7f33d015e380 2024-03-06T07:28:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.342+0000 7f33d4f2f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33d015e380 0x7f33d00ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.342+0000 7f33d5730640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f33d00cde30 0x7f33d0164a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46182/0 (socket says 172.21.15.27:46182) 2024-03-06T07:28:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.342+0000 7f33d5730640 1 -- 172.21.15.27:0/1479791609 learned_addr learned my addr 172.21.15.27:0/1479791609 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.342+0000 7f33d4f2f640 1 -- 172.21.15.27:0/1479791609 >> v1:172.21.15.69:6789/0 conn(0x7f33d00cac50 legacy=0x7f33d00cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.342+0000 7f33d4f2f640 1 -- 172.21.15.27:0/1479791609 >> v1:172.21.15.27:6789/0 conn(0x7f33d00cde30 legacy=0x7f33d0164a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.342+0000 7f33d4f2f640 1 -- 172.21.15.27:0/1479791609 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d00cbaf0 con 0x7f33d015e380 2024-03-06T07:28:30.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33d4f2f640 1 --2- 172.21.15.27:0/1479791609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d015e380 0x7f33d00ced70 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f33c40097d0 tx=0x7f33c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128b84ad41397754 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33cf7fe640 1 -- 172.21.15.27:0/1479791609 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c40315a0 con 0x7f33d015e380 2024-03-06T07:28:30.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33cf7fe640 1 -- 172.21.15.27:0/1479791609 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c4037070 con 0x7f33d015e380 2024-03-06T07:28:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33cf7fe640 1 -- 172.21.15.27:0/1479791609 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4031960 con 0x7f33d015e380 2024-03-06T07:28:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33d7193640 1 -- 172.21.15.27:0/1479791609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d015e380 msgr2=0x7f33d00ced70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33d7193640 1 --2- 172.21.15.27:0/1479791609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d015e380 0x7f33d00ced70 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f33c40097d0 tx=0x7f33c402b840 comp rx=0 tx=0).stop 2024-03-06T07:28:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33d7193640 1 -- 172.21.15.27:0/1479791609 shutdown_connections 2024-03-06T07:28:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33d7193640 1 --2- 172.21.15.27:0/1479791609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d015e380 0x7f33d00ced70 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33d7193640 1 -- 172.21.15.27:0/1479791609 >> 172.21.15.27:0/1479791609 conn(0x7f33d00cfb50 msgr2=0x7f33d00cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33d7193640 1 -- 172.21.15.27:0/1479791609 shutdown_connections 2024-03-06T07:28:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.343+0000 7f33d7193640 1 -- 172.21.15.27:0/1479791609 wait complete. 2024-03-06T07:28:30.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.345+0000 7f33d7193640 1 Processor -- start 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.345+0000 7f33d7193640 1 -- start start 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d7193640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33d00cac50 0x7f33d00709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d7193640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d00cde30 0x7f33d01e7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d7193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33d015e380 0x7f33d01577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d7193640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d00cb260 con 0x7f33d00cde30 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d7193640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d00cb0e0 con 0x7f33d00cac50 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d7193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d006dc10 con 0x7f33d015e380 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d00cde30 0x7f33d01e7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d00cde30 0x7f33d01e7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60346/0 (socket says 172.21.15.27:60346) 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cffff640 1 -- 172.21.15.27:0/1454023476 learned_addr learned my addr 172.21.15.27:0/1454023476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d4f2f640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33d00cac50 0x7f33d00709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d5730640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33d015e380 0x7f33d01577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cffff640 1 -- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33d015e380 msgr2=0x7f33d01577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cffff640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33d015e380 0x7f33d01577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cffff640 1 -- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33d00cac50 msgr2=0x7f33d00709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cffff640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33d00cac50 0x7f33d00709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cffff640 1 -- 172.21.15.27:0/1454023476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d00b2c70 con 0x7f33d00cde30 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d5730640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33d015e380 0x7f33d01577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cffff640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d00cde30 0x7f33d01e7d20 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c430 tx=0x7f33bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cd7fa640 1 -- 172.21.15.27:0/1454023476 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc011790 con 0x7f33d00cde30 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cd7fa640 1 -- 172.21.15.27:0/1454023476 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc017070 con 0x7f33d00cde30 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33cd7fa640 1 -- 172.21.15.27:0/1454023476 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc011bb0 con 0x7f33d00cde30 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.346+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d00b2f50 con 0x7f33d00cde30 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.347+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d00b3510 con 0x7f33d00cde30 2024-03-06T07:28:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.347+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d00752d0 con 0x7f33d00cde30 2024-03-06T07:28:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.348+0000 7f33cd7fa640 1 -- 172.21.15.27:0/1454023476 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33bc01c460 con 0x7f33d00cde30 2024-03-06T07:28:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.348+0000 7f33cd7fa640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f33b406b4d0 0x7f33b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.348+0000 7f33cd7fa640 1 -- 172.21.15.27:0/1454023476 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5443+0+0 (secure 0 0 0) 0x7f33bc089e60 con 0x7f33d00cde30 2024-03-06T07:28:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.349+0000 7f33d4f2f640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f33b406b4d0 0x7f33b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.349+0000 7f33d4f2f640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f33b406b4d0 0x7f33b406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f33c4036000 tx=0x7f33c403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.353+0000 7f33cd7fa640 1 -- 172.21.15.27:0/1454023476 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33bc058e70 con 0x7f33d00cde30 2024-03-06T07:28:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:30.538+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f33d00c9ee0 con 0x7f33d00cde30 2024-03-06T07:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.215+0000 7f33cd7fa640 1 -- 172.21.15.27:0/1454023476 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) v1 ==== 169+0+0 (secure 0 0 0) 0x7f33bc058770 con 0x7f33d00cde30 2024-03-06T07:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f33b406b4d0 msgr2=0x7f33b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f33b406b4d0 0x7f33b406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f33c4036000 tx=0x7f33c403d040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d00cde30 msgr2=0x7f33d01e7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d00cde30 0x7f33d01e7d20 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c430 tx=0x7f33bc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 shutdown_connections 2024-03-06T07:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f33b406b4d0 0x7f33b406d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33d015e380 0x7f33d01577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d00cde30 0x7f33d01e7d20 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 --2- 172.21.15.27:0/1454023476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33d00cac50 0x7f33d00709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.217+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 >> 172.21.15.27:0/1454023476 conn(0x7f33d00cfb50 msgr2=0x7f33d00c5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.218+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 shutdown_connections 2024-03-06T07:28:31.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.218+0000 7f33d7193640 1 -- 172.21.15.27:0/1454023476 wait complete. 2024-03-06T07:28:31.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:28:31.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:28:31.296 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-03-06T07:28:31.296 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-03-06T07:28:31.296 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:28:31.296 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-03-06T07:28:31.296 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.296 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-03-06T07:28:31.296 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-03-06T07:28:31.297 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.298 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-03-06T07:28:31.299 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.300 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-03-06T07:28:31.301 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:28:31.302 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:28:31.303 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.304 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-03-06T07:28:31.305 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-03-06T07:28:31.306 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.307 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.308 INFO:teuthology.orchestra.run.smithi069.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:28:31.308 INFO:teuthology.orchestra.run.smithi069.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:28:31.308 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-06T07:28:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-06T07:28:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:28:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-06T07:28:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.308 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.309 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:28:31.310 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:28:31.311 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-06T07:28:31.312 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:28:31.313 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-06T07:28:31.314 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:28:31.315 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-06T07:28:31.317 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.318 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-03-06T07:28:31.319 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:28:31.320 INFO:teuthology.orchestra.run.smithi027.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:28:31.322 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:28:31.365 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.417 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.418 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-03-06T07:28:31.419 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:28:31.420 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.421 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.422 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-03-06T07:28:31.424 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-03-06T07:28:31.427 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-03-06T07:28:31.428 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi069.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:28:31.431 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-06T07:28:31.432 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-06T07:28:31.433 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-06T07:28:31.434 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.435 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-03-06T07:28:31.436 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-03-06T07:28:31.437 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-06T07:28:31.438 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.439 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:28:31.440 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.441 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.442 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-03-06T07:28:31.443 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-03-06T07:28:31.443 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-03-06T07:28:31.443 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-03-06T07:28:31.443 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:28:31.443 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.443 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.443 INFO:teuthology.orchestra.run.smithi027.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:28:31.445 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:28:31.487 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:28:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:28:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-03-06T07:28:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-03-06T07:28:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-03-06T07:28:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-03-06T07:28:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-03-06T07:28:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-03-06T07:28:31.547 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790]" 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.553 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-06T07:28:31.558 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-06T07:28:31.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:28:31.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.714+0000 7fd17f7c9640 1 Processor -- start 2024-03-06T07:28:31.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.714+0000 7fd17f7c9640 1 -- start start 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17f7c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd17815e2f0 0x7fd1780b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17f7c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806b4e0 0x7fd17806b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17f7c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd17806be80 0x7fd1780cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17f7c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17815e6d0 con 0x7fd17806be80 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17f7c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1780ccd00 con 0x7fd17815e2f0 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17f7c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1780ccfe0 con 0x7fd17806b4e0 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17dd66640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd17806be80 0x7fd1780cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17dd66640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd17806be80 0x7fd1780cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60362/0 (socket says 172.21.15.27:60362) 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17dd66640 1 -- 172.21.15.27:0/1748154120 learned_addr learned my addr 172.21.15.27:0/1748154120 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17cd64640 1 --2- 172.21.15.27:0/1748154120 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806b4e0 0x7fd17806b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17d565640 1 --2- 172.21.15.27:0/1748154120 >> v2:172.21.15.69:3300/0 conn(0x7fd17815e2f0 0x7fd1780b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17dd66640 1 -- 172.21.15.27:0/1748154120 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806b4e0 msgr2=0x7fd17806b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17dd66640 1 --2- 172.21.15.27:0/1748154120 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806b4e0 0x7fd17806b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17dd66640 1 -- 172.21.15.27:0/1748154120 >> v2:172.21.15.69:3300/0 conn(0x7fd17815e2f0 msgr2=0x7fd1780b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17dd66640 1 --2- 172.21.15.27:0/1748154120 >> v2:172.21.15.69:3300/0 conn(0x7fd17815e2f0 0x7fd1780b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17dd66640 1 -- 172.21.15.27:0/1748154120 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1780cd2c0 con 0x7fd17806be80 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.715+0000 7fd17d565640 1 --2- 172.21.15.27:0/1748154120 >> v2:172.21.15.69:3300/0 conn(0x7fd17815e2f0 0x7fd1780b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.716+0000 7fd17dd66640 1 --2- 172.21.15.27:0/1748154120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806be80 0x7fd1780cfa00 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd178071dc0 tx=0x7fd17002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2a209f9ee0b156 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.716+0000 7fd16ffff640 1 -- 172.21.15.27:0/1748154120 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17002c950 con 0x7fd17806be80 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.716+0000 7fd16ffff640 1 -- 172.21.15.27:0/1748154120 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17002caf0 con 0x7fd17806be80 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.716+0000 7fd16ffff640 1 -- 172.21.15.27:0/1748154120 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700343f0 con 0x7fd17806be80 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.717+0000 7fd17f7c9640 1 -- 172.21.15.27:0/1748154120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806be80 msgr2=0x7fd1780cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.717+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/1748154120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806be80 0x7fd1780cfa00 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd178071dc0 tx=0x7fd17002bdb0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.717+0000 7fd17f7c9640 1 -- 172.21.15.27:0/1748154120 shutdown_connections 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.717+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/1748154120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806be80 0x7fd1780cfa00 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.717+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/1748154120 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806b4e0 0x7fd17806b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.717+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/1748154120 >> v2:172.21.15.69:3300/0 conn(0x7fd17815e2f0 0x7fd1780b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.717+0000 7fd17f7c9640 1 -- 172.21.15.27:0/1748154120 >> 172.21.15.27:0/1748154120 conn(0x7fd1780b36d0 msgr2=0x7fd1780b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.717+0000 7fd17f7c9640 1 -- 172.21.15.27:0/1748154120 shutdown_connections 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.717+0000 7fd17f7c9640 1 -- 172.21.15.27:0/1748154120 wait complete. 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.718+0000 7fd17f7c9640 1 Processor -- start 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.718+0000 7fd17f7c9640 1 -- start start 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17f7c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806b4e0 0x7fd17815e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17f7c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806be80 0x7fd17814cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17f7c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd17815e2f0 0x7fd1780b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17f7c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1780cff40 con 0x7fd17806b4e0 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17f7c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17806c310 con 0x7fd17815e2f0 2024-03-06T07:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17f7c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1780d00c0 con 0x7fd17806be80 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17d565640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806b4e0 0x7fd17815e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17d565640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806b4e0 0x7fd17815e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60372/0 (socket says 172.21.15.27:60372) 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17d565640 1 -- 172.21.15.27:0/3340502841 learned_addr learned my addr 172.21.15.27:0/3340502841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17dd66640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd17815e2f0 0x7fd1780b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17cd64640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806be80 0x7fd17814cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17d565640 1 -- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806be80 msgr2=0x7fd17814cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17d565640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806be80 0x7fd17814cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17d565640 1 -- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd17815e2f0 msgr2=0x7fd1780b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17d565640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd17815e2f0 0x7fd1780b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17d565640 1 -- 172.21.15.27:0/3340502841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17814d490 con 0x7fd17806b4e0 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17d565640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806b4e0 0x7fd17815e970 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c7f0 tx=0x7fd17400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd16dffb640 1 -- 172.21.15.27:0/3340502841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd174015070 con 0x7fd17806b4e0 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd16dffb640 1 -- 172.21.15.27:0/3340502841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd174007820 con 0x7fd17806b4e0 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17815ef80 con 0x7fd17806b4e0 2024-03-06T07:28:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.719+0000 7fd16dffb640 1 -- 172.21.15.27:0/3340502841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd174007bd0 con 0x7fd17806b4e0 2024-03-06T07:28:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.720+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17815f540 con 0x7fd17806b4e0 2024-03-06T07:28:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.721+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17809fd30 con 0x7fd17806b4e0 2024-03-06T07:28:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.721+0000 7fd16dffb640 1 -- 172.21.15.27:0/3340502841 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd174018070 con 0x7fd17806b4e0 2024-03-06T07:28:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.721+0000 7fd16dffb640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd15c06b4d0 0x7fd15c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.721+0000 7fd16dffb640 1 -- 172.21.15.27:0/3340502841 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd17408a060 con 0x7fd17806b4e0 2024-03-06T07:28:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.721+0000 7fd17cd64640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd15c06b4d0 0x7fd15c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.722+0000 7fd17cd64640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd15c06b4d0 0x7fd15c06d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd168006fd0 tx=0x7fd168008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.726+0000 7fd16dffb640 1 -- 172.21.15.27:0/3340502841 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd17814d490 con 0x7fd17806b4e0 2024-03-06T07:28:31.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.907+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd17806caa0 con 0x7fd17806b4e0 2024-03-06T07:28:31.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.910+0000 7fd16dffb640 1 -- 172.21.15.27:0/3340502841 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd174059180 con 0x7fd17806b4e0 2024-03-06T07:28:31.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd15c06b4d0 msgr2=0x7fd15c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd15c06b4d0 0x7fd15c06d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd168006fd0 tx=0x7fd168008040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806b4e0 msgr2=0x7fd17815e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806b4e0 0x7fd17815e970 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c7f0 tx=0x7fd17400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 shutdown_connections 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd15c06b4d0 0x7fd15c06d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd17815e2f0 0x7fd1780b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd17806be80 0x7fd17814cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 --2- 172.21.15.27:0/3340502841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17806b4e0 0x7fd17815e970 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 >> 172.21.15.27:0/3340502841 conn(0x7fd1780b36d0 msgr2=0x7fd17815bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 shutdown_connections 2024-03-06T07:28:31.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:31.912+0000 7fd17f7c9640 1 -- 172.21.15.27:0/3340502841 wait complete. 2024-03-06T07:28:31.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:28:32.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.030+0000 7f0bf9888640 1 Processor -- start 2024-03-06T07:28:32.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.030+0000 7f0bf9888640 1 -- start start 2024-03-06T07:28:32.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf9888640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bf415aa30 0x7f0bf40cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf9888640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0bf40aef60 0x7f0bf40b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf9888640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf415b160 con 0x7f0bf40d0000 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf9888640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf40d0460 con 0x7f0bf415aa30 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf9888640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf415b440 con 0x7f0bf40aef60 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf2ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bf415aa30 0x7f0bf40cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bea7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0bf40d0000 0x7f0bf40ae850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46188/0 (socket says 172.21.15.27:46188) 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bea7fc640 1 -- 172.21.15.27:0/2043699664 learned_addr learned my addr 172.21.15.27:0/2043699664 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf37fe640 1 --2- 172.21.15.27:0/2043699664 >> v2:172.21.15.69:3300/0 conn(0x7f0bf40aef60 0x7f0bf40b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2043699664 >> v2:172.21.15.69:3300/0 conn(0x7f0bf40aef60 msgr2=0x7f0bf40b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf2ffd640 1 --2- 172.21.15.27:0/2043699664 >> v2:172.21.15.69:3300/0 conn(0x7f0bf40aef60 0x7f0bf40b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2043699664 >> v1:172.21.15.27:6789/0 conn(0x7f0bf40d0000 legacy=0x7f0bf40ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf27fc640 1 -- 172.21.15.27:0/2043699664 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180413634 0 0) 0x7f0bf415b160 con 0x7f0bf40d0000 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.031+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2043699664 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf415b720 con 0x7f0bf415aa30 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.032+0000 7f0bf2ffd640 1 --2- 172.21.15.27:0/2043699664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 0x7f0bf40cfac0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4072a30 tx=0x7f0be402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9541227b79ad4eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.032+0000 7f0bf27fc640 1 -- 172.21.15.27:0/2043699664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4035020 con 0x7f0bf415aa30 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.032+0000 7f0bf27fc640 1 -- 172.21.15.27:0/2043699664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be4031830 con 0x7f0bf415aa30 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.032+0000 7f0bf27fc640 1 -- 172.21.15.27:0/2043699664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4031c00 con 0x7f0bf415aa30 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.033+0000 7f0bf9888640 1 -- 172.21.15.27:0/2043699664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 msgr2=0x7f0bf40cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.033+0000 7f0bf9888640 1 --2- 172.21.15.27:0/2043699664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 0x7f0bf40cfac0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4072a30 tx=0x7f0be402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.033+0000 7f0bf9888640 1 -- 172.21.15.27:0/2043699664 shutdown_connections 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.033+0000 7f0bf9888640 1 --2- 172.21.15.27:0/2043699664 >> v2:172.21.15.69:3300/0 conn(0x7f0bf40aef60 0x7f0bf40b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.033+0000 7f0bf9888640 1 --2- 172.21.15.27:0/2043699664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 0x7f0bf40cfac0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.033+0000 7f0bf9888640 1 -- 172.21.15.27:0/2043699664 >> 172.21.15.27:0/2043699664 conn(0x7f0bf4066ee0 msgr2=0x7f0bf40672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.033+0000 7f0bf9888640 1 -- 172.21.15.27:0/2043699664 shutdown_connections 2024-03-06T07:28:32.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.033+0000 7f0bf9888640 1 -- 172.21.15.27:0/2043699664 wait complete. 2024-03-06T07:28:32.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.034+0000 7f0bf9888640 1 Processor -- start 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.034+0000 7f0bf9888640 1 -- start start 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf9888640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf40aef60 0x7f0bf4158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf9888640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40d0000 0x7f0bf4151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf9888640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 0x7f0bf41540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf9888640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf415ae10 con 0x7f0bf415aa30 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf9888640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf415bce0 con 0x7f0bf40d0000 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf9888640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40b3b20 con 0x7f0bf40aef60 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 0x7f0bf41540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 0x7f0bf41540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60382/0 (socket says 172.21.15.27:60382) 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf37fe640 1 -- 172.21.15.27:0/1712325932 learned_addr learned my addr 172.21.15.27:0/1712325932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf2ffd640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf40aef60 0x7f0bf4158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bea7fc640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40d0000 0x7f0bf4151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf37fe640 1 -- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf40aef60 msgr2=0x7f0bf4158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf37fe640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf40aef60 0x7f0bf4158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf37fe640 1 -- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40d0000 msgr2=0x7f0bf4151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf37fe640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40d0000 0x7f0bf4151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf37fe640 1 -- 172.21.15.27:0/1712325932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf4159320 con 0x7f0bf415aa30 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.035+0000 7f0bf37fe640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 0x7f0bf41540f0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00efc0 tx=0x7f0bec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.036+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1712325932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec018020 con 0x7f0bf415aa30 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.036+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1712325932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec014480 con 0x7f0bf415aa30 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.036+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1712325932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec014870 con 0x7f0bf415aa30 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.036+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf41f1750 con 0x7f0bf415aa30 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.036+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf4158d60 con 0x7f0bf415aa30 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.037+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf41f1570 con 0x7f0bf415aa30 2024-03-06T07:28:32.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.037+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1712325932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bec005d10 con 0x7f0bf415aa30 2024-03-06T07:28:32.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.037+0000 7f0bf0ff9640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bd406b4d0 0x7f0bd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.037+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1712325932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0bec08ab60 con 0x7f0bf415aa30 2024-03-06T07:28:32.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.038+0000 7f0bf2ffd640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bd406b4d0 0x7f0bd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.038+0000 7f0bf2ffd640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bd406b4d0 0x7f0bd406d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0be4036000 tx=0x7f0be403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.042+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1712325932 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bf41f1570 con 0x7f0bf415aa30 2024-03-06T07:28:32.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:32.292+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f0bf41f1920 con 0x7f0bf415aa30 2024-03-06T07:28:33.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.218+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1712325932 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v327) v1 ==== 134+0+0 (secure 0 0 0) 0x7f0bec022030 con 0x7f0bf415aa30 2024-03-06T07:28:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bd406b4d0 msgr2=0x7f0bd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bd406b4d0 0x7f0bd406d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0be4036000 tx=0x7f0be403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 msgr2=0x7f0bf41540f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 0x7f0bf41540f0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00efc0 tx=0x7f0bec00c490 comp rx=0 tx=0).stop 2024-03-06T07:28:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 shutdown_connections 2024-03-06T07:28:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0bd406b4d0 0x7f0bd406d990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415aa30 0x7f0bf41540f0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40d0000 0x7f0bf4151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 --2- 172.21.15.27:0/1712325932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bf40aef60 0x7f0bf4158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 >> 172.21.15.27:0/1712325932 conn(0x7f0bf4066ee0 msgr2=0x7f0bf4156a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 shutdown_connections 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.221+0000 7f0bf9888640 1 -- 172.21.15.27:0/1712325932 wait complete. 2024-03-06T07:28:33.238 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:28:33.238 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:28:33.238 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:28:33.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.353+0000 7fdb61088640 1 Processor -- start 2024-03-06T07:28:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.353+0000 7fdb61088640 1 -- start start 2024-03-06T07:28:33.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb61088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c0ceed0 0x7fdb5c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb61088640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0cf3d0 con 0x7fdb5c145780 2024-03-06T07:28:33.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb61088640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0cf6b0 con 0x7fdb5c066d00 2024-03-06T07:28:33.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb61088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c072c30 con 0x7fdb5c0ceed0 2024-03-06T07:28:33.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5ad9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb5c145780 0x7fdb5c072520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46200/0 (socket says 172.21.15.27:46200) 2024-03-06T07:28:33.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5ad9d640 1 -- 172.21.15.27:0/1866011669 learned_addr learned my addr 172.21.15.27:0/1866011669 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:33.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb53fff640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c0ceed0 0x7fdb5c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267305360 0 0) 0x7fdb5c0cf3d0 con 0x7fdb5c145780 2024-03-06T07:28:33.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb48003520 con 0x7fdb5c145780 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 104307876 0 0) 0x7fdb48003520 con 0x7fdb5c145780 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c0ceed0 msgr2=0x7fdb5c0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c0ceed0 0x7fdb5c0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 >> v1:172.21.15.69:6789/0 conn(0x7fdb5c066d00 legacy=0x7fdb5c06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0cf990 con 0x7fdb5c145780 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fdb4c001670 con 0x7fdb5c145780 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 >> v1:172.21.15.27:6789/0 conn(0x7fdb5c145780 legacy=0x7fdb5c072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb48003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c0ceed0 0x7fdb48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb48004d40 0x7fdb480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c072c30 con 0x7fdb480039e0 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4800ba70 con 0x7fdb48004d40 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.354+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4800bc70 con 0x7fdb5c0ceed0 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb5ad9d640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb48003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb53fff640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c0ceed0 0x7fdb48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb5b59e640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb48004d40 0x7fdb480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb5ad9d640 1 -- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c0ceed0 msgr2=0x7fdb48004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb5ad9d640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c0ceed0 0x7fdb48004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb5ad9d640 1 -- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb48004d40 msgr2=0x7fdb480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb5ad9d640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb48004d40 0x7fdb480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb5ad9d640 1 -- 172.21.15.27:0/1866011669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb48003520 con 0x7fdb480039e0 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb53fff640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c0ceed0 0x7fdb48004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.355+0000 7fdb5b59e640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb48004d40 0x7fdb480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb5ad9d640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb48003dc0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c190 tx=0x7fdb48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775232918da4d532 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c007cc0 con 0x7fdb480039e0 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c030070 con 0x7fdb480039e0 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb5a59c640 1 -- 172.21.15.27:0/1866011669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c005d10 con 0x7fdb480039e0 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb61088640 1 -- 172.21.15.27:0/1866011669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 msgr2=0x7fdb48003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb61088640 1 --2- 172.21.15.27:0/1866011669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb48003dc0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c190 tx=0x7fdb48005830 comp rx=0 tx=0).stop 2024-03-06T07:28:33.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb5b59e640 1 -- 172.21.15.27:0/1866011669 reap_dead start 2024-03-06T07:28:33.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb61088640 1 -- 172.21.15.27:0/1866011669 shutdown_connections 2024-03-06T07:28:33.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb61088640 1 -- 172.21.15.27:0/1866011669 >> 172.21.15.27:0/1866011669 conn(0x7fdb5c0b2720 msgr2=0x7fdb5c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb61088640 1 -- 172.21.15.27:0/1866011669 shutdown_connections 2024-03-06T07:28:33.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.356+0000 7fdb61088640 1 -- 172.21.15.27:0/1866011669 wait complete. 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.358+0000 7fdb61088640 1 Processor -- start 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.358+0000 7fdb61088640 1 -- start start 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.358+0000 7fdb61088640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb5c0c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.358+0000 7fdb61088640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb5c1f46c0 0x7fdb5c140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.358+0000 7fdb61088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c13d8f0 0x7fdb5c13dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.358+0000 7fdb61088640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0d05f0 con 0x7fdb480039e0 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.358+0000 7fdb61088640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0d0470 con 0x7fdb5c1f46c0 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.358+0000 7fdb61088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0d0770 con 0x7fdb5c13d8f0 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb5c0c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb5c0c6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60400/0 (socket says 172.21.15.27:60400) 2024-03-06T07:28:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5ad9d640 1 -- 172.21.15.27:0/1805614481 learned_addr learned my addr 172.21.15.27:0/1805614481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb53fff640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb5c1f46c0 0x7fdb5c140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5b59e640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c13d8f0 0x7fdb5c13dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5ad9d640 1 -- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c13d8f0 msgr2=0x7fdb5c13dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5ad9d640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c13d8f0 0x7fdb5c13dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5ad9d640 1 -- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb5c1f46c0 msgr2=0x7fdb5c140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5ad9d640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb5c1f46c0 0x7fdb5c140bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5ad9d640 1 -- 172.21.15.27:0/1805614481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c1f4b40 con 0x7fdb480039e0 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb53fff640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb5c1f46c0 0x7fdb5c140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb5ad9d640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb5c0c6a50 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c190 tx=0x7fdb4c0019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb537fe640 1 -- 172.21.15.27:0/1805614481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c046070 con 0x7fdb480039e0 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb537fe640 1 -- 172.21.15.27:0/1805614481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c001740 con 0x7fdb480039e0 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c1413e0 con 0x7fdb480039e0 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb537fe640 1 -- 172.21.15.27:0/1805614481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c02cc20 con 0x7fdb480039e0 2024-03-06T07:28:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.359+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c1419a0 con 0x7fdb480039e0 2024-03-06T07:28:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.360+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c067ff0 con 0x7fdb480039e0 2024-03-06T07:28:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.361+0000 7fdb537fe640 1 -- 172.21.15.27:0/1805614481 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb4c007070 con 0x7fdb480039e0 2024-03-06T07:28:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.361+0000 7fdb537fe640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb3c06b500 0x7fdb3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.361+0000 7fdb537fe640 1 -- 172.21.15.27:0/1805614481 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdb4c0ab0c0 con 0x7fdb480039e0 2024-03-06T07:28:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.362+0000 7fdb53fff640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb3c06b500 0x7fdb3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.362+0000 7fdb53fff640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb3c06b500 0x7fdb3c06d9c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c06f060 tx=0x7fdb44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.365+0000 7fdb537fe640 1 -- 172.21.15.27:0/1805614481 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb4c07a1e0 con 0x7fdb480039e0 2024-03-06T07:28:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.549+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb5c06f3d0 con 0x7fdb480039e0 2024-03-06T07:28:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.550+0000 7fdb537fe640 1 -- 172.21.15.27:0/1805614481 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+13078 (secure 0 0 0) 0x7fdb4c079ae0 con 0x7fdb480039e0 2024-03-06T07:28:33.551 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":105,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:28:31.207793+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6832/550876663":"2024-03-07T07:28:28.203384+0000","172.21.15.27:6834/550876663":"2024-03-07T07:28:28.203384+0000","172.21.15.27:6835/3147032864":"2024-03-07T07:27:57.839873+0000","172.21.15.27:6833/3147032864":"2024-03-07T07:27:57.839873+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb3c06b500 msgr2=0x7fdb3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb3c06b500 0x7fdb3c06d9c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c06f060 tx=0x7fdb44008040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 msgr2=0x7fdb5c0c6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb5c0c6a50 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c190 tx=0x7fdb4c0019c0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 shutdown_connections 2024-03-06T07:28:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdb3c06b500 0x7fdb3c06d9c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb5c13d8f0 0x7fdb5c13dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb5c1f46c0 0x7fdb5c140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 --2- 172.21.15.27:0/1805614481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb480039e0 0x7fdb5c0c6a50 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.551+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 >> 172.21.15.27:0/1805614481 conn(0x7fdb5c0b2720 msgr2=0x7fdb5c0b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.552+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 shutdown_connections 2024-03-06T07:28:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.552+0000 7fdb61088640 1 -- 172.21.15.27:0/1805614481 wait complete. 2024-03-06T07:28:33.564 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:33.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.673+0000 7f0582b7a640 1 Processor -- start 2024-03-06T07:28:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.673+0000 7f0582b7a640 1 -- start start 2024-03-06T07:28:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f0582b7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f057c144640 0x7f057c06e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f0582b7a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c15bc40 con 0x7f057c0ce900 2024-03-06T07:28:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f0582b7a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c15bf20 con 0x7f057c06e940 2024-03-06T07:28:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f0582b7a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057c06df70 con 0x7f057c144640 2024-03-06T07:28:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f0581117640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f057c0ce900 0x7f057c0b36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58034/0 (socket says 172.21.15.27:58034) 2024-03-06T07:28:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f0581117640 1 -- 172.21.15.27:0/254146374 learned_addr learned my addr 172.21.15.27:0/254146374 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580506458 0 0) 0x7f057c15bc40 con 0x7f057c0ce900 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f0580916640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f057c144640 0x7f057c06e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056c003520 con 0x7f057c0ce900 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 340158753 0 0) 0x7f056c003520 con 0x7f057c0ce900 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f057c144640 msgr2=0x7f057c06e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f057c144640 0x7f057c06e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 >> v1:172.21.15.69:6789/0 conn(0x7f057c06e940 legacy=0x7f057c15b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c06dc00 con 0x7f057c0ce900 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f05740019f0 con 0x7f057c0ce900 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 >> v1:172.21.15.27:6789/0 conn(0x7f057c0ce900 legacy=0x7f057c0b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f056c0039e0 0x7f056c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c004390 0x7f056c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.674+0000 7f057b7fe640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f057c144640 0x7f056c0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c06df70 con 0x7f056c004390 2024-03-06T07:28:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00b9c0 con 0x7f056c0039e0 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00ddd0 con 0x7f057c144640 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f0580916640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f056c0039e0 0x7f056c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f057bfff640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c004390 0x7f056c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f0581117640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f057c144640 0x7f056c0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f057bfff640 1 -- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f057c144640 msgr2=0x7f056c0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f057bfff640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f057c144640 0x7f056c0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f057bfff640 1 -- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f056c0039e0 msgr2=0x7f056c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f057bfff640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f056c0039e0 0x7f056c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.675+0000 7f057bfff640 1 -- 172.21.15.27:0/254146374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c003520 con 0x7f056c004390 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f057bfff640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c004390 0x7f056c004860 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f0568001830 tx=0x7f056c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc9628a69a6aec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568001430 con 0x7f056c004390 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0568033610 con 0x7f056c004390 2024-03-06T07:28:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f057b7fe640 1 -- 172.21.15.27:0/254146374 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568033a00 con 0x7f056c004390 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f0582b7a640 1 -- 172.21.15.27:0/254146374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c004390 msgr2=0x7f056c004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f0582b7a640 1 --2- 172.21.15.27:0/254146374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c004390 0x7f056c004860 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f0568001830 tx=0x7f056c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f0581117640 1 -- 172.21.15.27:0/254146374 reap_dead start 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f0582b7a640 1 -- 172.21.15.27:0/254146374 shutdown_connections 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f0582b7a640 1 -- 172.21.15.27:0/254146374 >> 172.21.15.27:0/254146374 conn(0x7f057c066d00 msgr2=0x7f057c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f0582b7a640 1 -- 172.21.15.27:0/254146374 shutdown_connections 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.676+0000 7f0582b7a640 1 -- 172.21.15.27:0/254146374 wait complete. 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.678+0000 7f0582b7a640 1 Processor -- start 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.678+0000 7f0582b7a640 1 -- start start 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.678+0000 7f0582b7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c004390 0x7f057c142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.678+0000 7f0582b7a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c15a320 0x7f057c136560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.678+0000 7f0582b7a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f057c14fb40 0x7f057c14ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.678+0000 7f0582b7a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c15b9c0 con 0x7f057c14fb40 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0582b7a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c15b840 con 0x7f057c15a320 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0582b7a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057c0b3de0 con 0x7f056c004390 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0581117640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f057c14fb40 0x7f057c14ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0581117640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f057c14fb40 0x7f057c14ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54614/0 (socket says 172.21.15.27:54614) 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0581117640 1 -- 172.21.15.27:0/2218227783 learned_addr learned my addr 172.21.15.27:0/2218227783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0580916640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c004390 0x7f057c142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f057bfff640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c15a320 0x7f057c136560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0581117640 1 -- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c004390 msgr2=0x7f057c142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0581117640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c004390 0x7f057c142d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0581117640 1 -- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c15a320 msgr2=0x7f057c136560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0581117640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c15a320 0x7f057c136560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0581117640 1 -- 172.21.15.27:0/2218227783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c13a5c0 con 0x7f057c14fb40 2024-03-06T07:28:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0581117640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f057c14fb40 0x7f057c14ffc0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f0574000f80 tx=0x7f0574007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f05797fa640 1 -- 172.21.15.27:0/2218227783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574017020 con 0x7f057c14fb40 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f05797fa640 1 -- 172.21.15.27:0/2218227783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057400ac20 con 0x7f057c14fb40 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c143630 con 0x7f057c14fb40 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f05797fa640 1 -- 172.21.15.27:0/2218227783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05740133f0 con 0x7f057c14fb40 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.679+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c143bf0 con 0x7f057c14fb40 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.680+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c0733c0 con 0x7f057c14fb40 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.681+0000 7f05797fa640 1 -- 172.21.15.27:0/2218227783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f057401a020 con 0x7f057c14fb40 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.681+0000 7f05797fa640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f056006b4d0 0x7f056006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.681+0000 7f05797fa640 1 -- 172.21.15.27:0/2218227783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f057408a5d0 con 0x7f057c14fb40 2024-03-06T07:28:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.681+0000 7f0580916640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f056006b4d0 0x7f056006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.682+0000 7f0580916640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f056006b4d0 0x7f056006d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0570009b40 tx=0x7f0570009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.686+0000 7f05797fa640 1 -- 172.21.15.27:0/2218227783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0574021020 con 0x7f057c14fb40 2024-03-06T07:28:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.868+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f057c04f410 con 0x7f057c14fb40 2024-03-06T07:28:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.869+0000 7f05797fa640 1 -- 172.21.15.27:0/2218227783 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0574059780 con 0x7f057c14fb40 2024-03-06T07:28:33.870 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:28:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f056006b4d0 msgr2=0x7f056006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f056006b4d0 0x7f056006d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0570009b40 tx=0x7f0570009190 comp rx=0 tx=0).stop 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f057c14fb40 msgr2=0x7f057c14ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f057c14fb40 0x7f057c14ffc0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f0574000f80 tx=0x7f0574007b10 comp rx=0 tx=0).stop 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 shutdown_connections 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f056006b4d0 0x7f056006d990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f057c14fb40 0x7f057c14ffc0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c15a320 0x7f057c136560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 --2- 172.21.15.27:0/2218227783 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c004390 0x7f057c142d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 >> 172.21.15.27:0/2218227783 conn(0x7f057c066d00 msgr2=0x7f057c0b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 shutdown_connections 2024-03-06T07:28:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.871+0000 7f0582b7a640 1 -- 172.21.15.27:0/2218227783 wait complete. 2024-03-06T07:28:33.884 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:28:33.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.990+0000 7f046653a640 1 Processor -- start 2024-03-06T07:28:33.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.990+0000 7f046653a640 1 -- start start 2024-03-06T07:28:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.990+0000 7f046653a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f046006df30 0x7f04600ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.990+0000 7f046653a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04600aec40 0x7f04600b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.990+0000 7f046653a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0460071240 con 0x7f0460158520 2024-03-06T07:28:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.990+0000 7f046653a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04600af0f0 con 0x7f046006df30 2024-03-06T07:28:33.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.990+0000 7f046653a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0460071b70 con 0x7f04600aec40 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0460158520 0x7f046006d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58044/0 (socket says 172.21.15.27:58044) 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045ffff640 1 -- 172.21.15.27:0/2525633144 learned_addr learned my addr 172.21.15.27:0/2525633144 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f0464ad7640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04600aec40 0x7f04600b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045f7fe640 1 --2- 172.21.15.27:0/2525633144 >> v2:172.21.15.69:3300/0 conn(0x7f046006df30 0x7f04600ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549450918 0 0) 0x7f0460071240 con 0x7f0460158520 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0448003520 con 0x7f0460158520 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3234090875 0 0) 0x7f0448003520 con 0x7f0460158520 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04600aec40 msgr2=0x7f04600b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04600aec40 0x7f04600b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 >> v2:172.21.15.69:3300/0 conn(0x7f046006df30 msgr2=0x7f04600ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 --2- 172.21.15.27:0/2525633144 >> v2:172.21.15.69:3300/0 conn(0x7f046006df30 0x7f04600ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460071520 con 0x7f0460158520 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f0450001670 con 0x7f0460158520 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 >> v1:172.21.15.27:6789/0 conn(0x7f0460158520 legacy=0x7f046006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04480039b0 0x7f0448003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0448004800 0x7f0448004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04600aec40 0x7f04480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04600af0f0 con 0x7f04480039b0 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0460071b70 con 0x7f0448004800 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044800b9c0 con 0x7f04600aec40 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045ffff640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04480039b0 0x7f0448003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.991+0000 7f045f7fe640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0448004800 0x7f0448004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.992+0000 7f0464ad7640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04600aec40 0x7f04480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.992+0000 7f045ffff640 1 -- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04600aec40 msgr2=0x7f04480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.992+0000 7f045ffff640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04600aec40 0x7f04480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.992+0000 7f045ffff640 1 -- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0448004800 msgr2=0x7f0448004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.992+0000 7f045ffff640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0448004800 0x7f0448004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.992+0000 7f045ffff640 1 -- 172.21.15.27:0/2525633144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448003520 con 0x7f04480039b0 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.992+0000 7f045ffff640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04480039b0 0x7f0448003dd0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f045000c240 tx=0x7f0448005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47238e58c21fbea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.992+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450007d30 con 0x7f04480039b0 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.993+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450030070 con 0x7f04480039b0 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.993+0000 7f045effd640 1 -- 172.21.15.27:0/2525633144 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045002cd60 con 0x7f04480039b0 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.993+0000 7f046653a640 1 -- 172.21.15.27:0/2525633144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04480039b0 msgr2=0x7f0448003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.993+0000 7f046653a640 1 --2- 172.21.15.27:0/2525633144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04480039b0 0x7f0448003dd0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f045000c240 tx=0x7f0448005830 comp rx=0 tx=0).stop 2024-03-06T07:28:33.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.993+0000 7f0464ad7640 1 -- 172.21.15.27:0/2525633144 reap_dead start 2024-03-06T07:28:33.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.993+0000 7f046653a640 1 -- 172.21.15.27:0/2525633144 shutdown_connections 2024-03-06T07:28:33.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.993+0000 7f046653a640 1 -- 172.21.15.27:0/2525633144 >> 172.21.15.27:0/2525633144 conn(0x7f0460066d00 msgr2=0x7f04600670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.993+0000 7f046653a640 1 -- 172.21.15.27:0/2525633144 shutdown_connections 2024-03-06T07:28:33.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.993+0000 7f046653a640 1 -- 172.21.15.27:0/2525633144 wait complete. 2024-03-06T07:28:34.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.994+0000 7f046653a640 1 Processor -- start 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.994+0000 7f046653a640 1 -- start start 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f046653a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04480039b0 0x7f0460157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f046653a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04601f5c00 0x7f04601f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f046653a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04601f2410 0x7f0460154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f046653a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0460158900 con 0x7f04601f5c00 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f046653a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0460072000 con 0x7f04480039b0 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f046653a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0460158a80 con 0x7f04601f2410 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f045f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04601f5c00 0x7f04601f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f045f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04601f5c00 0x7f04601f5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54642/0 (socket says 172.21.15.27:54642) 2024-03-06T07:28:34.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f045f7fe640 1 -- 172.21.15.27:0/2018849902 learned_addr learned my addr 172.21.15.27:0/2018849902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f045ffff640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04480039b0 0x7f0460157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f0464ad7640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04601f2410 0x7f0460154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f045f7fe640 1 -- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04601f2410 msgr2=0x7f0460154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f045f7fe640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04601f2410 0x7f0460154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.995+0000 7f045f7fe640 1 -- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04480039b0 msgr2=0x7f0460157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.996+0000 7f045f7fe640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04480039b0 0x7f0460157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.996+0000 7f045f7fe640 1 -- 172.21.15.27:0/2018849902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04601567e0 con 0x7f04601f5c00 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.996+0000 7f045ffff640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04480039b0 0x7f0460157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.996+0000 7f045f7fe640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04601f5c00 0x7f04601f5220 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f04540118e0 tx=0x7f0454011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.996+0000 7f045cff9640 1 -- 172.21.15.27:0/2018849902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045400faf0 con 0x7f04601f5c00 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.996+0000 7f045cff9640 1 -- 172.21.15.27:0/2018849902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045400fc90 con 0x7f04601f5c00 2024-03-06T07:28:34.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.996+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0460156ac0 con 0x7f04601f5c00 2024-03-06T07:28:34.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.996+0000 7f045cff9640 1 -- 172.21.15.27:0/2018849902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454014440 con 0x7f04601f5c00 2024-03-06T07:28:34.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.996+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0460157080 con 0x7f04601f5c00 2024-03-06T07:28:34.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.997+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04600d0720 con 0x7f04601f5c00 2024-03-06T07:28:34.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.998+0000 7f045cff9640 1 -- 172.21.15.27:0/2018849902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0454012040 con 0x7f04601f5c00 2024-03-06T07:28:34.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.998+0000 7f045cff9640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f044c06b500 0x7f044c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.998+0000 7f045cff9640 1 -- 172.21.15.27:0/2018849902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f045408a530 con 0x7f04601f5c00 2024-03-06T07:28:34.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.998+0000 7f045ffff640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f044c06b500 0x7f044c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:33.999+0000 7f045ffff640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f044c06b500 0x7f044c06d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0450034000 tx=0x7f0450007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:34.003+0000 7f045cff9640 1 -- 172.21.15.27:0/2018849902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04600d0720 con 0x7f04601f5c00 2024-03-06T07:28:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:34.255+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f0460067ff0 con 0x7f04601f5c00 2024-03-06T07:28:35.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.223+0000 7f045cff9640 1 -- 172.21.15.27:0/2018849902 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v329) v1 ==== 145+0+0 (secure 0 0 0) 0x7f0454019030 con 0x7f04601f5c00 2024-03-06T07:28:35.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f044c06b500 msgr2=0x7f044c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f044c06b500 0x7f044c06d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0450034000 tx=0x7f0450007000 comp rx=0 tx=0).stop 2024-03-06T07:28:35.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04601f5c00 msgr2=0x7f04601f5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04601f5c00 0x7f04601f5220 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f04540118e0 tx=0x7f0454011da0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 shutdown_connections 2024-03-06T07:28:35.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f044c06b500 0x7f044c06d9c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04601f2410 0x7f0460154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04601f5c00 0x7f04601f5220 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 --2- 172.21.15.27:0/2018849902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04480039b0 0x7f0460157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 >> 172.21.15.27:0/2018849902 conn(0x7f0460066d00 msgr2=0x7f046006d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 shutdown_connections 2024-03-06T07:28:35.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.226+0000 7f046653a640 1 -- 172.21.15.27:0/2018849902 wait complete. 2024-03-06T07:28:35.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.356+0000 7fb5e759a640 1 Processor -- start 2024-03-06T07:28:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.356+0000 7fb5e759a640 1 -- start start 2024-03-06T07:28:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.357+0000 7fb5e759a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e015b130 0x7fb5e015b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.357+0000 7fb5e759a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5e00ce900 0x7fb5e00ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.357+0000 7fb5e759a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e015bc40 con 0x7fb5e0144640 2024-03-06T07:28:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.357+0000 7fb5e759a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e006df70 con 0x7fb5e00ce900 2024-03-06T07:28:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.357+0000 7fb5e759a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e015bf20 con 0x7fb5e015b130 2024-03-06T07:28:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.357+0000 7fb5e5b37640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5e00ce900 0x7fb5e00ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.357+0000 7fb5e5336640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb5e0144640 0x7fb5e006e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:33000/0 (socket says 172.21.15.27:33000) 2024-03-06T07:28:35.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.357+0000 7fb5e4b35640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e015b130 0x7fb5e015b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.357+0000 7fb5e5336640 1 -- 172.21.15.27:0/2757927284 learned_addr learned my addr 172.21.15.27:0/2757927284 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:35.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.358+0000 7fb5e5b37640 1 -- 172.21.15.27:0/2757927284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e015b130 msgr2=0x7fb5e015b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.358+0000 7fb5e5b37640 1 --2- 172.21.15.27:0/2757927284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e015b130 0x7fb5e015b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.358+0000 7fb5e5b37640 1 -- 172.21.15.27:0/2757927284 >> v1:172.21.15.69:6789/0 conn(0x7fb5e0144640 legacy=0x7fb5e006e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.358+0000 7fb5e5b37640 1 -- 172.21.15.27:0/2757927284 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e006dc00 con 0x7fb5e00ce900 2024-03-06T07:28:35.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.358+0000 7fb5d7fff640 1 -- 172.21.15.27:0/2757927284 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109150437 0 0) 0x7fb5e015bc40 con 0x7fb5e0144640 2024-03-06T07:28:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5e5b37640 1 --2- 172.21.15.27:0/2757927284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e00ce900 0x7fb5e00ced90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009d90 tx=0x7fb5d802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba5e45289ca6c536 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5d7fff640 1 -- 172.21.15.27:0/2757927284 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8039070 con 0x7fb5e00ce900 2024-03-06T07:28:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5d7fff640 1 -- 172.21.15.27:0/2757927284 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d802cc50 con 0x7fb5e00ce900 2024-03-06T07:28:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5d7fff640 1 -- 172.21.15.27:0/2757927284 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8034440 con 0x7fb5e00ce900 2024-03-06T07:28:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5e759a640 1 -- 172.21.15.27:0/2757927284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e00ce900 msgr2=0x7fb5e00ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5e759a640 1 --2- 172.21.15.27:0/2757927284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e00ce900 0x7fb5e00ced90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009d90 tx=0x7fb5d802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5e759a640 1 -- 172.21.15.27:0/2757927284 shutdown_connections 2024-03-06T07:28:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5e759a640 1 --2- 172.21.15.27:0/2757927284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e00ce900 0x7fb5e00ced90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5e759a640 1 --2- 172.21.15.27:0/2757927284 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e015b130 0x7fb5e015b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5e759a640 1 -- 172.21.15.27:0/2757927284 >> 172.21.15.27:0/2757927284 conn(0x7fb5e0066d00 msgr2=0x7fb5e0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5e759a640 1 -- 172.21.15.27:0/2757927284 shutdown_connections 2024-03-06T07:28:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.359+0000 7fb5e759a640 1 -- 172.21.15.27:0/2757927284 wait complete. 2024-03-06T07:28:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.361+0000 7fb5e759a640 1 Processor -- start 2024-03-06T07:28:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.361+0000 7fb5e759a640 1 -- start start 2024-03-06T07:28:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.361+0000 7fb5e759a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e00ce900 0x7fb5e0159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.361+0000 7fb5e759a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e0144640 0x7fb5e01f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.361+0000 7fb5e759a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e015b130 0x7fb5e00a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.361+0000 7fb5e759a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0070c20 con 0x7fb5e015b130 2024-03-06T07:28:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.361+0000 7fb5e759a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0070aa0 con 0x7fb5e0144640 2024-03-06T07:28:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.361+0000 7fb5e759a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00b36d0 con 0x7fb5e00ce900 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5b37640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e015b130 0x7fb5e00a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5b37640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e015b130 0x7fb5e00a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54658/0 (socket says 172.21.15.27:54658) 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5b37640 1 -- 172.21.15.27:0/1515338256 learned_addr learned my addr 172.21.15.27:0/1515338256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e4b35640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e0144640 0x7fb5e01f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5336640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e00ce900 0x7fb5e0159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5b37640 1 -- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e00ce900 msgr2=0x7fb5e0159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5b37640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e00ce900 0x7fb5e0159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5b37640 1 -- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e0144640 msgr2=0x7fb5e01f1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5b37640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e0144640 0x7fb5e01f1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5b37640 1 -- 172.21.15.27:0/1515338256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e01f4e60 con 0x7fb5e015b130 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e5b37640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e015b130 0x7fb5e00a94a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009d90 tx=0x7fb5d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5d5ffb640 1 -- 172.21.15.27:0/1515338256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8039050 con 0x7fb5e015b130 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5d5ffb640 1 -- 172.21.15.27:0/1515338256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d8045070 con 0x7fb5e015b130 2024-03-06T07:28:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5d5ffb640 1 -- 172.21.15.27:0/1515338256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8009450 con 0x7fb5e015b130 2024-03-06T07:28:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.362+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e015a6c0 con 0x7fb5e015b130 2024-03-06T07:28:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.363+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e00aa2f0 con 0x7fb5e015b130 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.364+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e00733c0 con 0x7fb5e015b130 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.364+0000 7fb5d5ffb640 1 -- 172.21.15.27:0/1515338256 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5d803e610 con 0x7fb5e015b130 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.364+0000 7fb5d5ffb640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5c406b4d0 0x7fb5c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.365+0000 7fb5d5ffb640 1 -- 172.21.15.27:0/1515338256 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb5d80ac8d0 con 0x7fb5e015b130 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.365+0000 7fb5e5336640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5c406b4d0 0x7fb5c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.365+0000 7fb5e5336640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5c406b4d0 0x7fb5c406d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc000f60 tx=0x7fb5dc007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.369+0000 7fb5d5ffb640 1 -- 172.21.15.27:0/1515338256 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5d807b3a0 con 0x7fb5e015b130 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.550+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb5e013ff70 con 0x7fb5e015b130 2024-03-06T07:28:35.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.553+0000 7fb5d5ffb640 1 -- 172.21.15.27:0/1515338256 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb5d807f440 con 0x7fb5e015b130 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5c406b4d0 msgr2=0x7fb5c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5c406b4d0 0x7fb5c406d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc000f60 tx=0x7fb5dc007d50 comp rx=0 tx=0).stop 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e015b130 msgr2=0x7fb5e00a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e015b130 0x7fb5e00a94a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009d90 tx=0x7fb5d8004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 shutdown_connections 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb5c406b4d0 0x7fb5c406d990 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5e015b130 0x7fb5e00a94a0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5e0144640 0x7fb5e01f1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 --2- 172.21.15.27:0/1515338256 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5e00ce900 0x7fb5e0159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 >> 172.21.15.27:0/1515338256 conn(0x7fb5e0066d00 msgr2=0x7fb5e0140770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.555+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 shutdown_connections 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.556+0000 7fb5e759a640 1 -- 172.21.15.27:0/1515338256 wait complete. 2024-03-06T07:28:35.568 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:35.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:35.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:35.666 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:35.666 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:28:35.666 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:35.682 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:35.682 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:28:35.682 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:35.699 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:35.699 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:28:35.700 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:35.737 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:35.737 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:28:35.738 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:35.756 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:35.756 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:28:35.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:35.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.883+0000 7f7e25563640 1 Processor -- start 2024-03-06T07:28:35.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.883+0000 7f7e25563640 1 -- start start 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e25563640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2015e060 0x7f7e200ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e25563640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e200d0dd0 con 0x7f7e200d0880 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e25563640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e200d10b0 con 0x7f7e2006fe20 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e25563640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e200cc0e0 con 0x7f7e2015e060 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e1f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e200d0880 0x7f7e200ccbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58056/0 (socket says 172.21.15.27:58056) 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e1f7fe640 1 -- 172.21.15.27:0/439765258 learned_addr learned my addr 172.21.15.27:0/439765258 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e1effd640 1 --2- 172.21.15.27:0/439765258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2015e060 0x7f7e200ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e1e7fc640 1 -- 172.21.15.27:0/439765258 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212010838 0 0) 0x7f7e200d0dd0 con 0x7f7e200d0880 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e1e7fc640 1 -- 172.21.15.27:0/439765258 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e0c003520 con 0x7f7e200d0880 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e1e7fc640 1 -- 172.21.15.27:0/439765258 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191383855 0 0) 0x7f7e200d10b0 con 0x7f7e2006fe20 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e1effd640 1 -- 172.21.15.27:0/439765258 >> v1:172.21.15.69:6789/0 conn(0x7f7e2006fe20 legacy=0x7f7e200ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e1effd640 1 -- 172.21.15.27:0/439765258 >> v1:172.21.15.27:6789/0 conn(0x7f7e200d0880 legacy=0x7f7e200ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.884+0000 7f7e1effd640 1 -- 172.21.15.27:0/439765258 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e200cc3c0 con 0x7f7e2015e060 2024-03-06T07:28:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.885+0000 7f7e1effd640 1 --2- 172.21.15.27:0/439765258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2015e060 0x7f7e200ce310 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7e100097d0 tx=0x7f7e1002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3ab54f41c023004 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.885+0000 7f7e1e7fc640 1 -- 172.21.15.27:0/439765258 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10035020 con 0x7f7e2015e060 2024-03-06T07:28:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.885+0000 7f7e1e7fc640 1 -- 172.21.15.27:0/439765258 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e10031890 con 0x7f7e2015e060 2024-03-06T07:28:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.885+0000 7f7e1e7fc640 1 -- 172.21.15.27:0/439765258 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10031c60 con 0x7f7e2015e060 2024-03-06T07:28:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.886+0000 7f7e25563640 1 -- 172.21.15.27:0/439765258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2015e060 msgr2=0x7f7e200ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.886+0000 7f7e25563640 1 --2- 172.21.15.27:0/439765258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2015e060 0x7f7e200ce310 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7e100097d0 tx=0x7f7e1002b930 comp rx=0 tx=0).stop 2024-03-06T07:28:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.886+0000 7f7e25563640 1 -- 172.21.15.27:0/439765258 shutdown_connections 2024-03-06T07:28:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.886+0000 7f7e25563640 1 --2- 172.21.15.27:0/439765258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e2015e060 0x7f7e200ce310 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.886+0000 7f7e25563640 1 -- 172.21.15.27:0/439765258 >> 172.21.15.27:0/439765258 conn(0x7f7e200d1a90 msgr2=0x7f7e200d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.886+0000 7f7e25563640 1 -- 172.21.15.27:0/439765258 shutdown_connections 2024-03-06T07:28:35.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.886+0000 7f7e25563640 1 -- 172.21.15.27:0/439765258 wait complete. 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.887+0000 7f7e25563640 1 Processor -- start 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.887+0000 7f7e25563640 1 -- start start 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e25563640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2006fe20 0x7f7e200af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e25563640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e200d0880 0x7f7e2014bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e25563640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e2015e060 0x7f7e2014c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e25563640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e200cef60 con 0x7f7e2015e060 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e25563640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2015e440 con 0x7f7e2006fe20 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e25563640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e200cf0e0 con 0x7f7e200d0880 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e2015e060 0x7f7e2014c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e2015e060 0x7f7e2014c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54660/0 (socket says 172.21.15.27:54660) 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1f7fe640 1 -- 172.21.15.27:0/3662891123 learned_addr learned my addr 172.21.15.27:0/3662891123 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1effd640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2006fe20 0x7f7e200af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e17fff640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e200d0880 0x7f7e2014bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1f7fe640 1 -- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e200d0880 msgr2=0x7f7e2014bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1f7fe640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e200d0880 0x7f7e2014bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1f7fe640 1 -- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2006fe20 msgr2=0x7f7e200af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1f7fe640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2006fe20 0x7f7e200af270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1f7fe640 1 -- 172.21.15.27:0/3662891123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2014f890 con 0x7f7e2015e060 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.888+0000 7f7e1f7fe640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e2015e060 0x7f7e2014c2a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800cb40 tx=0x7f7e1800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.889+0000 7f7e177fe640 1 -- 172.21.15.27:0/3662891123 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18017070 con 0x7f7e2015e060 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.889+0000 7f7e177fe640 1 -- 172.21.15.27:0/3662891123 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1800aca0 con 0x7f7e2015e060 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.889+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2015fd30 con 0x7f7e2015e060 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.889+0000 7f7e177fe640 1 -- 172.21.15.27:0/3662891123 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18012620 con 0x7f7e2015e060 2024-03-06T07:28:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.889+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2015f970 con 0x7f7e2015e060 2024-03-06T07:28:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.890+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e20067ff0 con 0x7f7e2015e060 2024-03-06T07:28:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.890+0000 7f7e177fe640 1 -- 172.21.15.27:0/3662891123 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e1801a070 con 0x7f7e2015e060 2024-03-06T07:28:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.891+0000 7f7e177fe640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e0006b4d0 0x7f7e0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.891+0000 7f7e177fe640 1 -- 172.21.15.27:0/3662891123 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7e18089cb0 con 0x7f7e2015e060 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.891+0000 7f7e1effd640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e0006b4d0 0x7f7e0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.891+0000 7f7e1effd640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e0006b4d0 0x7f7e0006d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7e10036000 tx=0x7f7e1003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:35.895+0000 7f7e177fe640 1 -- 172.21.15.27:0/3662891123 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e18058810 con 0x7f7e2015e060 2024-03-06T07:28:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.116+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e200d20c0 con 0x7f7e2015e060 2024-03-06T07:28:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.117+0000 7f7e177fe640 1 -- 172.21.15.27:0/3662891123 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f7e1805c8b0 con 0x7f7e2015e060 2024-03-06T07:28:36.118 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:36.119 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/109211083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":109211083},{"type":"v1","addr":"172.21.15.27:6835","nonce":109211083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6895,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3201775065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3201775065},{"type":"v1","addr":"172.21.15.69:6835","nonce":3201775065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":6901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/2932892673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":2932892673},{"type":"v1","addr":"172.21.15.69:6837","nonce":2932892673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3524025245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3524025245},{"type":"v1","addr":"172.21.15.27:6834","nonce":3524025245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[]} 2024-03-06T07:28:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 69 2024-03-06T07:28:36.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e0006b4d0 msgr2=0x7f7e0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e0006b4d0 0x7f7e0006d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7e10036000 tx=0x7f7e1003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e2015e060 msgr2=0x7f7e2014c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e2015e060 0x7f7e2014c2a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800cb40 tx=0x7f7e1800a060 comp rx=0 tx=0).stop 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 shutdown_connections 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e0006b4d0 0x7f7e0006d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e2015e060 0x7f7e2014c2a0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e200d0880 0x7f7e2014bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 --2- 172.21.15.27:0/3662891123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e2006fe20 0x7f7e200af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 >> 172.21.15.27:0/3662891123 conn(0x7f7e200d1a90 msgr2=0x7f7e2006eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 shutdown_connections 2024-03-06T07:28:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.119+0000 7f7e25563640 1 -- 172.21.15.27:0/3662891123 wait complete. 2024-03-06T07:28:36.132 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:36.132 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:36.132 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:36.132 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:36.132 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:36.132 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:36.133 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:36.133 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:36.133 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:36.133 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:36.133 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:36.133 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:36.133 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:36.134 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:36.134 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:36.134 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:36.141 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:36.141 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:36.142 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:36.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:36.144 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:36.145 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:36.145 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:36.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:36.184 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:36.184 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:36.184 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:36.185 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:36.187 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:36.187 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:36.187 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:36.188 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:36.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:36.265 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:28:36.265 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:28:36.270 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:28:36.270 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:28:36.270 INFO:tasks.ceph.mds.b.smithi069.stderr:2024-03-06T07:28:36.269+0000 7fc80633b640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:28:36.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.299+0000 7f308c33a640 1 Processor -- start 2024-03-06T07:28:36.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.299+0000 7f308c33a640 1 -- start start 2024-03-06T07:28:36.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.300+0000 7f308c33a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f308406df30 0x7f30840ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.300+0000 7f308c33a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f30840aec40 0x7f30840b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.300+0000 7f308c33a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3084071240 con 0x7f3084158520 2024-03-06T07:28:36.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.300+0000 7f308c33a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30840af0f0 con 0x7f308406df30 2024-03-06T07:28:36.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.300+0000 7f308c33a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3084071b70 con 0x7f30840aec40 2024-03-06T07:28:36.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.300+0000 7f30898d5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f308406df30 0x7f30840ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.300+0000 7f30898d5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f308406df30 0x7f30840ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54708/0 (socket says 172.21.15.27:54708) 2024-03-06T07:28:36.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.300+0000 7f30898d5640 1 -- 172.21.15.27:0/2157084366 learned_addr learned my addr 172.21.15.27:0/2157084366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:36.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.301+0000 7f308a8d7640 1 --2- 172.21.15.27:0/2157084366 >> v2:172.21.15.69:3300/0 conn(0x7f30840aec40 0x7f30840b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.301+0000 7f30898d5640 1 -- 172.21.15.27:0/2157084366 >> v2:172.21.15.69:3300/0 conn(0x7f30840aec40 msgr2=0x7f30840b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.301+0000 7f30898d5640 1 --2- 172.21.15.27:0/2157084366 >> v2:172.21.15.69:3300/0 conn(0x7f30840aec40 0x7f30840b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.301+0000 7f30898d5640 1 -- 172.21.15.27:0/2157084366 >> v1:172.21.15.69:6789/0 conn(0x7f3084158520 legacy=0x7f308406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.301+0000 7f30898d5640 1 -- 172.21.15.27:0/2157084366 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3084071520 con 0x7f308406df30 2024-03-06T07:28:36.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.301+0000 7f30898d5640 1 --2- 172.21.15.27:0/2157084366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f308406df30 0x7f30840ae700 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f3080009870 tx=0x7f308002b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7ddb65cda8fa2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f30890d4640 1 -- 172.21.15.27:0/2157084366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080034070 con 0x7f308406df30 2024-03-06T07:28:36.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f30890d4640 1 -- 172.21.15.27:0/2157084366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3080030880 con 0x7f308406df30 2024-03-06T07:28:36.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f30890d4640 1 -- 172.21.15.27:0/2157084366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080030c90 con 0x7f308406df30 2024-03-06T07:28:36.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f308c33a640 1 -- 172.21.15.27:0/2157084366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f308406df30 msgr2=0x7f30840ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f308c33a640 1 --2- 172.21.15.27:0/2157084366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f308406df30 0x7f30840ae700 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f3080009870 tx=0x7f308002b9c0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f308c33a640 1 -- 172.21.15.27:0/2157084366 shutdown_connections 2024-03-06T07:28:36.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f308c33a640 1 --2- 172.21.15.27:0/2157084366 >> v2:172.21.15.69:3300/0 conn(0x7f30840aec40 0x7f30840b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f308c33a640 1 --2- 172.21.15.27:0/2157084366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f308406df30 0x7f30840ae700 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f308c33a640 1 -- 172.21.15.27:0/2157084366 >> 172.21.15.27:0/2157084366 conn(0x7f3084066d00 msgr2=0x7f30840670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f308c33a640 1 -- 172.21.15.27:0/2157084366 shutdown_connections 2024-03-06T07:28:36.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.302+0000 7f308c33a640 1 -- 172.21.15.27:0/2157084366 wait complete. 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308c33a640 1 Processor -- start 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308c33a640 1 -- start start 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308c33a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f308406df30 0x7f308414c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308c33a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30840aec40 0x7f30841e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308c33a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3084158520 0x7f30840cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308c33a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3084158900 con 0x7f3084158520 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308c33a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3084072000 con 0x7f308406df30 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308c33a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3084158a80 con 0x7f30840aec40 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308a8d7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3084158520 0x7f30840cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308a8d7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3084158520 0x7f30840cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54722/0 (socket says 172.21.15.27:54722) 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.304+0000 7f308a8d7640 1 -- 172.21.15.27:0/132371828 learned_addr learned my addr 172.21.15.27:0/132371828 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f30898d5640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30840aec40 0x7f30841e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f308a8d7640 1 -- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30840aec40 msgr2=0x7f30841e3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f308a0d6640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f308406df30 0x7f308414c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f308a8d7640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30840aec40 0x7f30841e3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f308a8d7640 1 -- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f308406df30 msgr2=0x7f308414c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f308a8d7640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f308406df30 0x7f308414c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f308a8d7640 1 -- 172.21.15.27:0/132371828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30841e47e0 con 0x7f3084158520 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f308a8d7640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3084158520 0x7f30840cf250 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f307c010fa0 tx=0x7f307c00e550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f307affd640 1 -- 172.21.15.27:0/132371828 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c00b2d0 con 0x7f3084158520 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f307affd640 1 -- 172.21.15.27:0/132371828 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c01b070 con 0x7f3084158520 2024-03-06T07:28:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30840ce490 con 0x7f3084158520 2024-03-06T07:28:36.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f307affd640 1 -- 172.21.15.27:0/132371828 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c009780 con 0x7f3084158520 2024-03-06T07:28:36.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.305+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30840cea50 con 0x7f3084158520 2024-03-06T07:28:36.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.306+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3084067ff0 con 0x7f3084158520 2024-03-06T07:28:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.307+0000 7f307affd640 1 -- 172.21.15.27:0/132371828 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f307c005d10 con 0x7f3084158520 2024-03-06T07:28:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.307+0000 7f307affd640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f306806b4d0 0x7f306806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.307+0000 7f307affd640 1 -- 172.21.15.27:0/132371828 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f307c08c650 con 0x7f3084158520 2024-03-06T07:28:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.307+0000 7f308a0d6640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f306806b4d0 0x7f306806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.308+0000 7f308a0d6640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f306806b4d0 0x7f306806d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f30840ae9d0 tx=0x7f3074009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.312+0000 7f307affd640 1 -- 172.21.15.27:0/132371828 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30841e47e0 con 0x7f3084158520 2024-03-06T07:28:36.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:36.507+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f30840b31f0 con 0x7f3084158520 2024-03-06T07:28:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.567+0000 7f307affd640 1 -- 172.21.15.27:0/132371828 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v110) v1 ==== 97+0+0 (secure 0 0 0) 0x7f307c012040 con 0x7f3084158520 2024-03-06T07:28:37.569 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-03-06T07:28:37.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f306806b4d0 msgr2=0x7f306806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f306806b4d0 0x7f306806d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f30840ae9d0 tx=0x7f3074009040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3084158520 msgr2=0x7f30840cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3084158520 0x7f30840cf250 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f307c010fa0 tx=0x7f307c00e550 comp rx=0 tx=0).stop 2024-03-06T07:28:37.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 shutdown_connections 2024-03-06T07:28:37.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f306806b4d0 0x7f306806d990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3084158520 0x7f30840cf250 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30840aec40 0x7f30841e3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 --2- 172.21.15.27:0/132371828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f308406df30 0x7f308414c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 >> 172.21.15.27:0/132371828 conn(0x7f3084066d00 msgr2=0x7f30840b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 shutdown_connections 2024-03-06T07:28:37.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.571+0000 7f308c33a640 1 -- 172.21.15.27:0/132371828 wait complete. 2024-03-06T07:28:37.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:37.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.698+0000 7fc6a4dc9640 1 Processor -- start 2024-03-06T07:28:37.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.698+0000 7fc6a4dc9640 1 -- start start 2024-03-06T07:28:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc6a4dc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a00b1d50 0x7fc6a00b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc6a4dc9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6a006d210 0x7fc6a006d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc6a4dc9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a015e410 con 0x7fc6a015e030 2024-03-06T07:28:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc6a4dc9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a00b26f0 con 0x7fc6a006d210 2024-03-06T07:28:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc6a4dc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0162700 con 0x7fc6a00b1d50 2024-03-06T07:28:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69e59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc6a015e030 0x7fc6a006ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58070/0 (socket says 172.21.15.27:58070) 2024-03-06T07:28:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69e59c640 1 -- 172.21.15.27:0/1201410792 learned_addr learned my addr 172.21.15.27:0/1201410792 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:37.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69ed9d640 1 --2- 172.21.15.27:0/1201410792 >> v2:172.21.15.27:3300/0 conn(0x7fc6a006d210 0x7fc6a006d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69d59a640 1 -- 172.21.15.27:0/1201410792 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440538886 0 0) 0x7fc690001490 con 0x7fc6a015e030 2024-03-06T07:28:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69dd9b640 1 --2- 172.21.15.27:0/1201410792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a00b1d50 0x7fc6a00b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69d59a640 1 -- 172.21.15.27:0/1201410792 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc688003520 con 0x7fc6a015e030 2024-03-06T07:28:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69ed9d640 1 -- 172.21.15.27:0/1201410792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a00b1d50 msgr2=0x7fc6a00b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69ed9d640 1 --2- 172.21.15.27:0/1201410792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a00b1d50 0x7fc6a00b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69ed9d640 1 -- 172.21.15.27:0/1201410792 >> v1:172.21.15.27:6789/0 conn(0x7fc6a015e030 legacy=0x7fc6a006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.699+0000 7fc69ed9d640 1 -- 172.21.15.27:0/1201410792 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a006f140 con 0x7fc6a006d210 2024-03-06T07:28:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.700+0000 7fc69ed9d640 1 --2- 172.21.15.27:0/1201410792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 0x7fc6a006d690 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc6980097a0 tx=0x7fc69802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e787455f25c10f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.700+0000 7fc69d59a640 1 -- 172.21.15.27:0/1201410792 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698035020 con 0x7fc6a006d210 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.700+0000 7fc69d59a640 1 -- 172.21.15.27:0/1201410792 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc698031830 con 0x7fc6a006d210 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.700+0000 7fc69d59a640 1 -- 172.21.15.27:0/1201410792 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698031be0 con 0x7fc6a006d210 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.701+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/1201410792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 msgr2=0x7fc6a006d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.701+0000 7fc6a4dc9640 1 --2- 172.21.15.27:0/1201410792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 0x7fc6a006d690 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc6980097a0 tx=0x7fc69802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.701+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/1201410792 shutdown_connections 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.701+0000 7fc6a4dc9640 1 --2- 172.21.15.27:0/1201410792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 0x7fc6a006d690 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.701+0000 7fc6a4dc9640 1 --2- 172.21.15.27:0/1201410792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a00b1d50 0x7fc6a00b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.701+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/1201410792 >> 172.21.15.27:0/1201410792 conn(0x7fc6a00b55e0 msgr2=0x7fc6a00b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.701+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/1201410792 shutdown_connections 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.701+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/1201410792 wait complete. 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.702+0000 7fc6a4dc9640 1 Processor -- start 2024-03-06T07:28:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.702+0000 7fc6a4dc9640 1 -- start start 2024-03-06T07:28:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc6a4dc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 0x7fc6a0147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc6a4dc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6a00b1d50 0x7fc6a009a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc6a4dc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a015e030 0x7fc6a0158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc6a4dc9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a01620b0 con 0x7fc6a006d210 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc6a4dc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0161dd0 con 0x7fc6a00b1d50 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc6a4dc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0162390 con 0x7fc6a015e030 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc69e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 0x7fc6a0147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc69e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 0x7fc6a0147140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54740/0 (socket says 172.21.15.27:54740) 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc69e59c640 1 -- 172.21.15.27:0/2194077240 learned_addr learned my addr 172.21.15.27:0/2194077240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc69ed9d640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a015e030 0x7fc6a0158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.703+0000 7fc69dd9b640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6a00b1d50 0x7fc6a009a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc69e59c640 1 -- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a015e030 msgr2=0x7fc6a0158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc69e59c640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a015e030 0x7fc6a0158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc69e59c640 1 -- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6a00b1d50 msgr2=0x7fc6a009a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc69e59c640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6a00b1d50 0x7fc6a009a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc69e59c640 1 -- 172.21.15.27:0/2194077240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a009d330 con 0x7fc6a006d210 2024-03-06T07:28:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc69e59c640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 0x7fc6a0147140 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fc69000cb50 tx=0x7fc69000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc68effd640 1 -- 172.21.15.27:0/2194077240 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690017070 con 0x7fc6a006d210 2024-03-06T07:28:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc68effd640 1 -- 172.21.15.27:0/2194077240 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69000ad40 con 0x7fc6a006d210 2024-03-06T07:28:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a009d610 con 0x7fc6a006d210 2024-03-06T07:28:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc68effd640 1 -- 172.21.15.27:0/2194077240 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690012640 con 0x7fc6a006d210 2024-03-06T07:28:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.704+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a009dbd0 con 0x7fc6a006d210 2024-03-06T07:28:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.705+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a0067ff0 con 0x7fc6a006d210 2024-03-06T07:28:37.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.705+0000 7fc68effd640 1 -- 172.21.15.27:0/2194077240 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc69001a070 con 0x7fc6a006d210 2024-03-06T07:28:37.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.706+0000 7fc68effd640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc68006b500 0x7fc68006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.706+0000 7fc68effd640 1 -- 172.21.15.27:0/2194077240 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc690089d80 con 0x7fc6a006d210 2024-03-06T07:28:37.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.706+0000 7fc69dd9b640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc68006b500 0x7fc68006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.707+0000 7fc69dd9b640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc68006b500 0x7fc68006d9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0067280 tx=0x7fc694008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.710+0000 7fc68effd640 1 -- 172.21.15.27:0/2194077240 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6a015e410 con 0x7fc6a006d210 2024-03-06T07:28:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.958+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc6a00b2910 con 0x7fc6a006d210 2024-03-06T07:28:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.958+0000 7fc68effd640 1 -- 172.21.15.27:0/2194077240 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc690058ed0 con 0x7fc6a006d210 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g==", 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:28:37.960 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ==", 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.961 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg==", 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ==", 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:28:37.962 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw==", 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.963 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA==", 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA==", 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:28:37.964 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg==", 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A==", 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.965 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw==", 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-03-06T07:28:37.966 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA==", 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A==", 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA==", 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ==", 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:28:37.968 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ==", 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.969 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg==", 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlkc2iEhAA6BuYba0FzXeVjGPOP8sgyQ==", 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:28:37.970 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw==", 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-03-06T07:28:37.971 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg==", 2024-03-06T07:28:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:28:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:28:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:28:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:28:37.972 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:28:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc68006b500 msgr2=0x7fc68006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc68006b500 0x7fc68006d9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0067280 tx=0x7fc694008040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 msgr2=0x7fc6a0147140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 0x7fc6a0147140 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fc69000cb50 tx=0x7fc69000a430 comp rx=0 tx=0).stop 2024-03-06T07:28:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 shutdown_connections 2024-03-06T07:28:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc68006b500 0x7fc68006d9c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6a015e030 0x7fc6a0158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6a00b1d50 0x7fc6a009a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 --2- 172.21.15.27:0/2194077240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6a006d210 0x7fc6a0147140 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 >> 172.21.15.27:0/2194077240 conn(0x7fc6a00b55e0 msgr2=0x7fc6a00ae550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 shutdown_connections 2024-03-06T07:28:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:37.960+0000 7fc6a4dc9640 1 -- 172.21.15.27:0/2194077240 wait complete. 2024-03-06T07:28:37.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:38.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.080+0000 7f96ee62a640 1 Processor -- start 2024-03-06T07:28:38.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.080+0000 7f96ee62a640 1 -- start start 2024-03-06T07:28:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.081+0000 7f96ee62a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 0x7f96e8168c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.081+0000 7f96ee62a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e80cf130 con 0x7f96e815e060 2024-03-06T07:28:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.081+0000 7f96ee62a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e80d0dd0 con 0x7f96e806fe20 2024-03-06T07:28:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.081+0000 7f96ee62a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e80d10b0 con 0x7f96e80d0880 2024-03-06T07:28:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.081+0000 7f96e7fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f96e815e060 0x7f96e80ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58082/0 (socket says 172.21.15.27:58082) 2024-03-06T07:28:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.081+0000 7f96e7fff640 1 -- 172.21.15.27:0/2904306988 learned_addr learned my addr 172.21.15.27:0/2904306988 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.081+0000 7f96ecbc7640 1 --2- 172.21.15.27:0/2904306988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 0x7f96e8168c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.082+0000 7f96e6ffd640 1 -- 172.21.15.27:0/2904306988 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948635398 0 0) 0x7f96e80cf130 con 0x7f96e815e060 2024-03-06T07:28:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.082+0000 7f96e6ffd640 1 -- 172.21.15.27:0/2904306988 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d4003520 con 0x7f96e815e060 2024-03-06T07:28:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.082+0000 7f96e6ffd640 1 -- 172.21.15.27:0/2904306988 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314038996 0 0) 0x7f96e80d0dd0 con 0x7f96e806fe20 2024-03-06T07:28:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.082+0000 7f96ecbc7640 1 -- 172.21.15.27:0/2904306988 >> v1:172.21.15.69:6789/0 conn(0x7f96e806fe20 legacy=0x7f96e80cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.082+0000 7f96ecbc7640 1 -- 172.21.15.27:0/2904306988 >> v1:172.21.15.27:6789/0 conn(0x7f96e815e060 legacy=0x7f96e80ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.082+0000 7f96ecbc7640 1 -- 172.21.15.27:0/2904306988 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e80cc0e0 con 0x7f96e80d0880 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.082+0000 7f96ecbc7640 1 --2- 172.21.15.27:0/2904306988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 0x7f96e8168c20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009dc0 tx=0x7f96e002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfc7d1925003f178 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96e6ffd640 1 -- 172.21.15.27:0/2904306988 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0039070 con 0x7f96e80d0880 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96e6ffd640 1 -- 172.21.15.27:0/2904306988 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e002ccd0 con 0x7f96e80d0880 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96e6ffd640 1 -- 172.21.15.27:0/2904306988 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0034660 con 0x7f96e80d0880 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96ee62a640 1 -- 172.21.15.27:0/2904306988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 msgr2=0x7f96e8168c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96ee62a640 1 --2- 172.21.15.27:0/2904306988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 0x7f96e8168c20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009dc0 tx=0x7f96e002bf20 comp rx=0 tx=0).stop 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96ee62a640 1 -- 172.21.15.27:0/2904306988 shutdown_connections 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96ee62a640 1 --2- 172.21.15.27:0/2904306988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 0x7f96e8168c20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96ee62a640 1 -- 172.21.15.27:0/2904306988 >> 172.21.15.27:0/2904306988 conn(0x7f96e80d1a90 msgr2=0x7f96e80d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96ee62a640 1 -- 172.21.15.27:0/2904306988 shutdown_connections 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.083+0000 7f96ee62a640 1 -- 172.21.15.27:0/2904306988 wait complete. 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.084+0000 7f96ee62a640 1 Processor -- start 2024-03-06T07:28:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.085+0000 7f96ee62a640 1 -- start start 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.085+0000 7f96ee62a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96e806fe20 0x7f96e81f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.085+0000 7f96ee62a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 0x7f96e80b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.085+0000 7f96ee62a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e815e060 0x7f96e80ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.085+0000 7f96ee62a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e815e440 con 0x7f96e815e060 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.085+0000 7f96ee62a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e80cc6a0 con 0x7f96e806fe20 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.085+0000 7f96ee62a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e80cdbd0 con 0x7f96e80d0880 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.085+0000 7f96ecbc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e815e060 0x7f96e80ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ecbc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e815e060 0x7f96e80ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54756/0 (socket says 172.21.15.27:54756) 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ecbc7640 1 -- 172.21.15.27:0/4004999630 learned_addr learned my addr 172.21.15.27:0/4004999630 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96e77fe640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 0x7f96e80b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96e7fff640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96e806fe20 0x7f96e81f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ecbc7640 1 -- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 msgr2=0x7f96e80b6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ecbc7640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 0x7f96e80b6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ecbc7640 1 -- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96e806fe20 msgr2=0x7f96e81f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ecbc7640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96e806fe20 0x7f96e81f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ecbc7640 1 -- 172.21.15.27:0/4004999630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e80b7370 con 0x7f96e815e060 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96e7fff640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96e806fe20 0x7f96e81f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ecbc7640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e815e060 0x7f96e80ce000 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009ee0 tx=0x7f96e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96e4ff9640 1 -- 172.21.15.27:0/4004999630 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0039050 con 0x7f96e815e060 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96e4ff9640 1 -- 172.21.15.27:0/4004999630 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e0045070 con 0x7f96e815e060 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96e4ff9640 1 -- 172.21.15.27:0/4004999630 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0009450 con 0x7f96e815e060 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e80b7650 con 0x7f96e815e060 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.086+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e80b6f40 con 0x7f96e815e060 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.087+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e8067ff0 con 0x7f96e815e060 2024-03-06T07:28:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.088+0000 7f96e4ff9640 1 -- 172.21.15.27:0/4004999630 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96e003e850 con 0x7f96e815e060 2024-03-06T07:28:38.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.088+0000 7f96e4ff9640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b500 0x7f96c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.088+0000 7f96e4ff9640 1 -- 172.21.15.27:0/4004999630 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f96e00acab0 con 0x7f96e815e060 2024-03-06T07:28:38.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.088+0000 7f96e7fff640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b500 0x7f96c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.089+0000 7f96e7fff640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b500 0x7f96c806d9c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f96d0005880 tx=0x7f96d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.092+0000 7f96e4ff9640 1 -- 172.21.15.27:0/4004999630 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96e0032e20 con 0x7f96e815e060 2024-03-06T07:28:38.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.274+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96e80cec60 con 0x7f96e815e060 2024-03-06T07:28:38.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.275+0000 7f96e4ff9640 1 -- 172.21.15.27:0/4004999630 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+12821 (secure 0 0 0) 0x7f96e007bd10 con 0x7f96e815e060 2024-03-06T07:28:38.277 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:38.277 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":110,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:28:37.565973+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:38.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b500 msgr2=0x7f96c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b500 0x7f96c806d9c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f96d0005880 tx=0x7f96d0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e815e060 msgr2=0x7f96e80ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e815e060 0x7f96e80ce000 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009ee0 tx=0x7f96e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 shutdown_connections 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b500 0x7f96c806d9c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e815e060 0x7f96e80ce000 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96e80d0880 0x7f96e80b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 --2- 172.21.15.27:0/4004999630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96e806fe20 0x7f96e81f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 >> 172.21.15.27:0/4004999630 conn(0x7f96e80d1a90 msgr2=0x7f96e80b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 shutdown_connections 2024-03-06T07:28:38.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.277+0000 7f96ee62a640 1 -- 172.21.15.27:0/4004999630 wait complete. 2024-03-06T07:28:38.291 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:38.291 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:38.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.399+0000 7f8d18278640 1 Processor -- start 2024-03-06T07:28:38.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.399+0000 7f8d18278640 1 -- start start 2024-03-06T07:28:38.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d18278640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d10142170 0x7f8d10142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d18278640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d100b4830 0x7f8d10160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d18278640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d100b29a0 0x7f8d100b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d18278640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d101458e0 con 0x7f8d100b4830 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d18278640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d100b3310 con 0x7f8d100b29a0 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d18278640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d10070980 con 0x7f8d10142170 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d15813640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d100b4830 0x7f8d10160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d15813640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d100b4830 0x7f8d10160280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54766/0 (socket says 172.21.15.27:54766) 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d15813640 1 -- 172.21.15.27:0/2545862983 learned_addr learned my addr 172.21.15.27:0/2545862983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d15813640 1 -- 172.21.15.27:0/2545862983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d10142170 msgr2=0x7f8d10142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d16815640 1 --2- 172.21.15.27:0/2545862983 >> v2:172.21.15.69:3300/0 conn(0x7f8d100b29a0 0x7f8d100b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d16014640 1 --2- 172.21.15.27:0/2545862983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d10142170 0x7f8d10142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d15813640 1 --2- 172.21.15.27:0/2545862983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d10142170 0x7f8d10142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d15813640 1 -- 172.21.15.27:0/2545862983 >> v2:172.21.15.69:3300/0 conn(0x7f8d100b29a0 msgr2=0x7f8d100b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d15813640 1 --2- 172.21.15.27:0/2545862983 >> v2:172.21.15.69:3300/0 conn(0x7f8d100b29a0 0x7f8d100b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d15813640 1 -- 172.21.15.27:0/2545862983 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d10142f60 con 0x7f8d100b4830 2024-03-06T07:28:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d16014640 1 --2- 172.21.15.27:0/2545862983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d10142170 0x7f8d10142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.400+0000 7f8d16815640 1 --2- 172.21.15.27:0/2545862983 >> v2:172.21.15.69:3300/0 conn(0x7f8d100b29a0 0x7f8d100b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.401+0000 7f8d15813640 1 --2- 172.21.15.27:0/2545862983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d100b4830 0x7f8d10160280 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f8d100cf5f0 tx=0x7f8d0002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484d8646121ff2ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.401+0000 7f8d15012640 1 -- 172.21.15.27:0/2545862983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d00035020 con 0x7f8d100b4830 2024-03-06T07:28:38.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.401+0000 7f8d15012640 1 -- 172.21.15.27:0/2545862983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d00031640 con 0x7f8d100b4830 2024-03-06T07:28:38.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.401+0000 7f8d15012640 1 -- 172.21.15.27:0/2545862983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0002d040 con 0x7f8d100b4830 2024-03-06T07:28:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.401+0000 7f8d18278640 1 -- 172.21.15.27:0/2545862983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d100b4830 msgr2=0x7f8d10160280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.401+0000 7f8d18278640 1 --2- 172.21.15.27:0/2545862983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d100b4830 0x7f8d10160280 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f8d100cf5f0 tx=0x7f8d0002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.402+0000 7f8d18278640 1 -- 172.21.15.27:0/2545862983 shutdown_connections 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.402+0000 7f8d18278640 1 --2- 172.21.15.27:0/2545862983 >> v2:172.21.15.69:3300/0 conn(0x7f8d100b29a0 0x7f8d100b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.402+0000 7f8d18278640 1 --2- 172.21.15.27:0/2545862983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d100b4830 0x7f8d10160280 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.402+0000 7f8d18278640 1 --2- 172.21.15.27:0/2545862983 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d10142170 0x7f8d10142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.402+0000 7f8d18278640 1 -- 172.21.15.27:0/2545862983 >> 172.21.15.27:0/2545862983 conn(0x7f8d10066d00 msgr2=0x7f8d10067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.402+0000 7f8d18278640 1 -- 172.21.15.27:0/2545862983 shutdown_connections 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.402+0000 7f8d18278640 1 -- 172.21.15.27:0/2545862983 wait complete. 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.403+0000 7f8d18278640 1 Processor -- start 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.403+0000 7f8d18278640 1 -- start start 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.403+0000 7f8d18278640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d100b29a0 0x7f8d1015cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.403+0000 7f8d18278640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d100b4830 0x7f8d101f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.403+0000 7f8d18278640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d10142170 0x7f8d1015d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.403+0000 7f8d18278640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d10070ed0 con 0x7f8d10142170 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.403+0000 7f8d18278640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d101607c0 con 0x7f8d100b29a0 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d18278640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d100711b0 con 0x7f8d100b4830 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16815640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d10142170 0x7f8d1015d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16815640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d10142170 0x7f8d1015d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54778/0 (socket says 172.21.15.27:54778) 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16815640 1 -- 172.21.15.27:0/1469979671 learned_addr learned my addr 172.21.15.27:0/1469979671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d15813640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d100b4830 0x7f8d101f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16014640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d100b29a0 0x7f8d1015cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16815640 1 -- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d100b4830 msgr2=0x7f8d101f21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16815640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d100b4830 0x7f8d101f21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16815640 1 -- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d100b29a0 msgr2=0x7f8d1015cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16815640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d100b29a0 0x7f8d1015cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16815640 1 -- 172.21.15.27:0/1469979671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d101f4f60 con 0x7f8d10142170 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16014640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d100b29a0 0x7f8d1015cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d16815640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d10142170 0x7f8d1015d710 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800cef0 tx=0x7f8d0800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d06ffd640 1 -- 172.21.15.27:0/1469979671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0801a070 con 0x7f8d10142170 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d06ffd640 1 -- 172.21.15.27:0/1469979671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0800d860 con 0x7f8d10142170 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d101f5240 con 0x7f8d10142170 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d06ffd640 1 -- 172.21.15.27:0/1469979671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0800dc50 con 0x7f8d10142170 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.404+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d101f5800 con 0x7f8d10142170 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.405+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d100733c0 con 0x7f8d10142170 2024-03-06T07:28:38.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.406+0000 7f8d06ffd640 1 -- 172.21.15.27:0/1469979671 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d0801d070 con 0x7f8d10142170 2024-03-06T07:28:38.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.406+0000 7f8d06ffd640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cf406b4d0 0x7f8cf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.406+0000 7f8d06ffd640 1 -- 172.21.15.27:0/1469979671 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8d0808a6d0 con 0x7f8d10142170 2024-03-06T07:28:38.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.407+0000 7f8d16014640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cf406b4d0 0x7f8cf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.407+0000 7f8d16014640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cf406b4d0 0x7f8cf406d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00a910 tx=0x7f8d0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.411+0000 7f8d06ffd640 1 -- 172.21.15.27:0/1469979671 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d08059310 con 0x7f8d10142170 2024-03-06T07:28:38.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.591+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d1004f410 con 0x7f8d10142170 2024-03-06T07:28:38.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.591+0000 7f8d06ffd640 1 -- 172.21.15.27:0/1469979671 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d0805d3b0 con 0x7f8d10142170 2024-03-06T07:28:38.593 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:28:38.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cf406b4d0 msgr2=0x7f8cf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cf406b4d0 0x7f8cf406d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00a910 tx=0x7f8d0c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d10142170 msgr2=0x7f8d1015d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d10142170 0x7f8d1015d710 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800cef0 tx=0x7f8d0800a710 comp rx=0 tx=0).stop 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 shutdown_connections 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cf406b4d0 0x7f8cf406d990 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d10142170 0x7f8d1015d710 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d100b4830 0x7f8d101f21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 --2- 172.21.15.27:0/1469979671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d100b29a0 0x7f8d1015cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 >> 172.21.15.27:0/1469979671 conn(0x7f8d10066d00 msgr2=0x7f8d1015fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 shutdown_connections 2024-03-06T07:28:38.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.593+0000 7f8d18278640 1 -- 172.21.15.27:0/1469979671 wait complete. 2024-03-06T07:28:38.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:38.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.711+0000 7ff20df56640 1 Processor -- start 2024-03-06T07:28:38.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.711+0000 7ff20df56640 1 -- start start 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff20df56640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff208158520 0x7ff20806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff20df56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff20806dd60 0x7ff2080ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff20df56640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080af080 con 0x7ff2080aec40 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff20df56640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20806e1c0 con 0x7ff208158520 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff20df56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff208071b70 con 0x7ff20806dd60 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff206ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff20806dd60 0x7ff2080ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff2077fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff208158520 0x7ff20806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff206ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff20806dd60 0x7ff2080ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:57482/0 (socket says 172.21.15.27:57482) 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff206ffd640 1 -- 172.21.15.27:0/1641523829 learned_addr learned my addr 172.21.15.27:0/1641523829 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:38.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff206ffd640 1 -- 172.21.15.27:0/1641523829 >> v2:172.21.15.69:3300/0 conn(0x7ff208158520 msgr2=0x7ff20806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff206ffd640 1 --2- 172.21.15.27:0/1641523829 >> v2:172.21.15.69:3300/0 conn(0x7ff208158520 0x7ff20806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff206ffd640 1 -- 172.21.15.27:0/1641523829 >> v1:172.21.15.69:6789/0 conn(0x7ff2080aec40 legacy=0x7ff20815e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.712+0000 7ff206ffd640 1 -- 172.21.15.27:0/1641523829 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208071240 con 0x7ff20806dd60 2024-03-06T07:28:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.713+0000 7ff206ffd640 1 --2- 172.21.15.27:0/1641523829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff20806dd60 0x7ff2080ae700 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009870 tx=0x7ff1fc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba5c15bafdd0b24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.713+0000 7ff2067fc640 1 -- 172.21.15.27:0/1641523829 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc034070 con 0x7ff20806dd60 2024-03-06T07:28:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.713+0000 7ff2067fc640 1 -- 172.21.15.27:0/1641523829 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc030840 con 0x7ff20806dd60 2024-03-06T07:28:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.713+0000 7ff2067fc640 1 -- 172.21.15.27:0/1641523829 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc030c10 con 0x7ff20806dd60 2024-03-06T07:28:38.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.714+0000 7ff20df56640 1 -- 172.21.15.27:0/1641523829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff20806dd60 msgr2=0x7ff2080ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.714+0000 7ff20df56640 1 --2- 172.21.15.27:0/1641523829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff20806dd60 0x7ff2080ae700 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009870 tx=0x7ff1fc02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.714+0000 7ff20df56640 1 -- 172.21.15.27:0/1641523829 shutdown_connections 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.714+0000 7ff20df56640 1 --2- 172.21.15.27:0/1641523829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff20806dd60 0x7ff2080ae700 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.714+0000 7ff20df56640 1 --2- 172.21.15.27:0/1641523829 >> v2:172.21.15.69:3300/0 conn(0x7ff208158520 0x7ff20806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.714+0000 7ff20df56640 1 -- 172.21.15.27:0/1641523829 >> 172.21.15.27:0/1641523829 conn(0x7ff208066d00 msgr2=0x7ff2080670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.714+0000 7ff20df56640 1 -- 172.21.15.27:0/1641523829 shutdown_connections 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.714+0000 7ff20df56640 1 -- 172.21.15.27:0/1641523829 wait complete. 2024-03-06T07:28:38.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.715+0000 7ff20df56640 1 Processor -- start 2024-03-06T07:28:38.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.715+0000 7ff20df56640 1 -- start start 2024-03-06T07:28:38.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff20df56640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff20806dd60 0x7ff208146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff20df56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2080aec40 0x7ff20815dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff20df56640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff208158520 0x7ff2080ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff20df56640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080b4650 con 0x7ff20806dd60 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff20df56640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff208071800 con 0x7ff208158520 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff20df56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff208072000 con 0x7ff2080aec40 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff2077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff20806dd60 0x7ff208146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff2077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff20806dd60 0x7ff208146420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54786/0 (socket says 172.21.15.27:54786) 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff2077fe640 1 -- 172.21.15.27:0/410510797 learned_addr learned my addr 172.21.15.27:0/410510797 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff206ffd640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2080aec40 0x7ff20815dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff207fff640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff208158520 0x7ff2080ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff2077fe640 1 -- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2080aec40 msgr2=0x7ff20815dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.716+0000 7ff2077fe640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2080aec40 0x7ff20815dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.717+0000 7ff2077fe640 1 -- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff208158520 msgr2=0x7ff2080ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.717+0000 7ff2077fe640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff208158520 0x7ff2080ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.717+0000 7ff2077fe640 1 -- 172.21.15.27:0/410510797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208149500 con 0x7ff20806dd60 2024-03-06T07:28:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.717+0000 7ff2077fe640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff20806dd60 0x7ff208146420 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800e870 tx=0x7ff1f800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.717+0000 7ff1effff640 1 -- 172.21.15.27:0/410510797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8017070 con 0x7ff20806dd60 2024-03-06T07:28:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.717+0000 7ff1effff640 1 -- 172.21.15.27:0/410510797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f800c940 con 0x7ff20806dd60 2024-03-06T07:28:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.717+0000 7ff1effff640 1 -- 172.21.15.27:0/410510797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f800cd30 con 0x7ff20806dd60 2024-03-06T07:28:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.717+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2081497e0 con 0x7ff20806dd60 2024-03-06T07:28:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.717+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff208149da0 con 0x7ff20806dd60 2024-03-06T07:28:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.718+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff208067ff0 con 0x7ff20806dd60 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.718+0000 7ff1effff640 1 -- 172.21.15.27:0/410510797 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1f801a070 con 0x7ff20806dd60 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.719+0000 7ff1effff640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1f406b4d0 0x7ff1f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.719+0000 7ff1effff640 1 -- 172.21.15.27:0/410510797 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff1f808a130 con 0x7ff20806dd60 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.719+0000 7ff206ffd640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1f406b4d0 0x7ff1f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.720+0000 7ff206ffd640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1f406b4d0 0x7ff1f406d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc036000 tx=0x7ff1fc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.723+0000 7ff1effff640 1 -- 172.21.15.27:0/410510797 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1f8059300 con 0x7ff20806dd60 2024-03-06T07:28:38.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.908+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff208157cd0 con 0x7ff20806dd60 2024-03-06T07:28:38.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.909+0000 7ff1effff640 1 -- 172.21.15.27:0/410510797 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v110) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff1f8058c00 con 0x7ff20806dd60 2024-03-06T07:28:38.911 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:38.911 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:28:38.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:38.911 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:28:38.912 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:28:38.912 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:38.912 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1f406b4d0 msgr2=0x7ff1f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1f406b4d0 0x7ff1f406d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc036000 tx=0x7ff1fc03b040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff20806dd60 msgr2=0x7ff208146420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff20806dd60 0x7ff208146420 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800e870 tx=0x7ff1f800ed30 comp rx=0 tx=0).stop 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 shutdown_connections 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1f406b4d0 0x7ff1f406d990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff208158520 0x7ff2080ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2080aec40 0x7ff20815dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 --2- 172.21.15.27:0/410510797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff20806dd60 0x7ff208146420 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 >> 172.21.15.27:0/410510797 conn(0x7ff208066d00 msgr2=0x7ff2080af310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 shutdown_connections 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:38.911+0000 7ff20df56640 1 -- 172.21.15.27:0/410510797 wait complete. 2024-03-06T07:28:38.925 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:38.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:39.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.033+0000 7fbb01e4e640 1 Processor -- start 2024-03-06T07:28:39.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.033+0000 7fbb01e4e640 1 -- start start 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.034+0000 7fbb01e4e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbafc15ae50 0x7fbafc0708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.034+0000 7fbb01e4e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15b230 0x7fbafc071160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.034+0000 7fbb01e4e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc070e20 con 0x7fbafc15b690 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.034+0000 7fbb01e4e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbafc0b2e30 con 0x7fbafc15ae50 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.034+0000 7fbb01e4e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc06dfe0 con 0x7fbafc15b230 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.034+0000 7fbafbfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbafc15b690 0x7fbafc150a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58084/0 (socket says 172.21.15.27:58084) 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.034+0000 7fbafbfff640 1 -- 172.21.15.27:0/3224542903 learned_addr learned my addr 172.21.15.27:0/3224542903 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.034+0000 7fbafb7fe640 1 --2- 172.21.15.27:0/3224542903 >> v2:172.21.15.69:3300/0 conn(0x7fbafc15ae50 0x7fbafc0708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.034+0000 7fbafaffd640 1 --2- 172.21.15.27:0/3224542903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15b230 0x7fbafc071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.035+0000 7fbafa7fc640 1 -- 172.21.15.27:0/3224542903 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580862418 0 0) 0x7fbafc070e20 con 0x7fbafc15b690 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.035+0000 7fbafa7fc640 1 -- 172.21.15.27:0/3224542903 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae4003520 con 0x7fbafc15b690 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.035+0000 7fbafb7fe640 1 -- 172.21.15.27:0/3224542903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15b230 msgr2=0x7fbafc071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.035+0000 7fbafb7fe640 1 --2- 172.21.15.27:0/3224542903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15b230 0x7fbafc071160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.035+0000 7fbafb7fe640 1 -- 172.21.15.27:0/3224542903 >> v1:172.21.15.27:6789/0 conn(0x7fbafc15b690 legacy=0x7fbafc150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.035+0000 7fbafb7fe640 1 -- 172.21.15.27:0/3224542903 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0b30b0 con 0x7fbafc15ae50 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbafb7fe640 1 --2- 172.21.15.27:0/3224542903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbafc15ae50 0x7fbafc0708e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0097a0 tx=0x7fbaec02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=753b7a0bd2a72bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbafa7fc640 1 -- 172.21.15.27:0/3224542903 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec0315a0 con 0x7fbafc15ae50 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbafa7fc640 1 -- 172.21.15.27:0/3224542903 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec037070 con 0x7fbafc15ae50 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbafa7fc640 1 -- 172.21.15.27:0/3224542903 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec031940 con 0x7fbafc15ae50 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3224542903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbafc15ae50 msgr2=0x7fbafc0708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbb01e4e640 1 --2- 172.21.15.27:0/3224542903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbafc15ae50 0x7fbafc0708e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0097a0 tx=0x7fbaec02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3224542903 shutdown_connections 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbb01e4e640 1 --2- 172.21.15.27:0/3224542903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15b230 0x7fbafc071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbb01e4e640 1 --2- 172.21.15.27:0/3224542903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbafc15ae50 0x7fbafc0708e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3224542903 >> 172.21.15.27:0/3224542903 conn(0x7fbafc066d00 msgr2=0x7fbafc067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3224542903 shutdown_connections 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.036+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3224542903 wait complete. 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.037+0000 7fbb01e4e640 1 Processor -- start 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.038+0000 7fbb01e4e640 1 -- start start 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.038+0000 7fbb01e4e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15ae50 0x7fbafc168840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.038+0000 7fbb01e4e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbafc15b230 0x7fbafc168f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.038+0000 7fbb01e4e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbafc15b690 0x7fbafc13e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.038+0000 7fbb01e4e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0b3950 con 0x7fbafc15b690 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.038+0000 7fbb01e4e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0b3670 con 0x7fbafc15b230 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.038+0000 7fbb01e4e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0b4150 con 0x7fbafc15ae50 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbafc15b690 0x7fbafc13e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbafc15b690 0x7fbafc13e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54802/0 (socket says 172.21.15.27:54802) 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafbfff640 1 -- 172.21.15.27:0/3525764716 learned_addr learned my addr 172.21.15.27:0/3525764716 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafaffd640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbafc15b230 0x7fbafc168f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafb7fe640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15ae50 0x7fbafc168840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafbfff640 1 -- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15ae50 msgr2=0x7fbafc168840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafbfff640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15ae50 0x7fbafc168840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafbfff640 1 -- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbafc15b230 msgr2=0x7fbafc168f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafbfff640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbafc15b230 0x7fbafc168f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafbfff640 1 -- 172.21.15.27:0/3525764716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc167da0 con 0x7fbafc15b690 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafb7fe640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15ae50 0x7fbafc168840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbafbfff640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbafc15b690 0x7fbafc13e9b0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400cad0 tx=0x7fbaf400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbae3fff640 1 -- 172.21.15.27:0/3525764716 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4017070 con 0x7fbafc15b690 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbae3fff640 1 -- 172.21.15.27:0/3525764716 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf400abd0 con 0x7fbafc15b690 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbae3fff640 1 -- 172.21.15.27:0/3525764716 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4012400 con 0x7fbafc15b690 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc13fdc0 con 0x7fbafc15b690 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.039+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc168040 con 0x7fbafc15b690 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.040+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbafc0733c0 con 0x7fbafc15b690 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.041+0000 7fbae3fff640 1 -- 172.21.15.27:0/3525764716 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaf401a070 con 0x7fbafc15b690 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.041+0000 7fbae3fff640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbae806b500 0x7fbae806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.041+0000 7fbae3fff640 1 -- 172.21.15.27:0/3525764716 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbaf4089a90 con 0x7fbafc15b690 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.041+0000 7fbafb7fe640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbae806b500 0x7fbae806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.042+0000 7fbafb7fe640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbae806b500 0x7fbae806d9c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbaec036000 tx=0x7fbaec03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.045+0000 7fbae3fff640 1 -- 172.21.15.27:0/3525764716 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaf40586a0 con 0x7fbafc15b690 2024-03-06T07:28:39.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.228+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbafc04f410 con 0x7fbafc15b690 2024-03-06T07:28:39.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.568+0000 7fbae3fff640 1 -- 172.21.15.27:0/3525764716 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v111) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbaf405c740 con 0x7fbafc15b690 2024-03-06T07:28:39.571 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:39.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbae806b500 msgr2=0x7fbae806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbae806b500 0x7fbae806d9c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbaec036000 tx=0x7fbaec03d040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbafc15b690 msgr2=0x7fbafc13e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbafc15b690 0x7fbafc13e9b0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400cad0 tx=0x7fbaf400cf90 comp rx=0 tx=0).stop 2024-03-06T07:28:39.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 shutdown_connections 2024-03-06T07:28:39.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbae806b500 0x7fbae806d9c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbafc15b690 0x7fbafc13e9b0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbafc15b230 0x7fbafc168f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 --2- 172.21.15.27:0/3525764716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbafc15ae50 0x7fbafc168840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 >> 172.21.15.27:0/3525764716 conn(0x7fbafc066d00 msgr2=0x7fbafc0cd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 shutdown_connections 2024-03-06T07:28:39.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.571+0000 7fbb01e4e640 1 -- 172.21.15.27:0/3525764716 wait complete. 2024-03-06T07:28:39.589 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:39.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.701+0000 7f7abeb5c640 1 Processor -- start 2024-03-06T07:28:39.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.701+0000 7f7abeb5c640 1 -- start start 2024-03-06T07:28:39.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.702+0000 7f7abeb5c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7ab8072a30 0x7f7ab81624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.702+0000 7f7abeb5c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab80b4950 con 0x7f7ab80d07c0 2024-03-06T07:28:39.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.702+0000 7f7abeb5c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab806d4d0 con 0x7f7ab80cf9f0 2024-03-06T07:28:39.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.702+0000 7f7abeb5c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab806dba0 con 0x7f7ab8072a30 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.702+0000 7f7abd0f9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7ab80d07c0 0x7f7ab815d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58098/0 (socket says 172.21.15.27:58098) 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.702+0000 7f7abd0f9640 1 -- 172.21.15.27:0/897191216 learned_addr learned my addr 172.21.15.27:0/897191216 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.702+0000 7f7abc8f8640 1 --2- 172.21.15.27:0/897191216 >> v2:172.21.15.69:3300/0 conn(0x7f7ab8072a30 0x7f7ab81624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.702+0000 7f7ab77fe640 1 -- 172.21.15.27:0/897191216 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149316807 0 0) 0x7f7ab80b4950 con 0x7f7ab80d07c0 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.703+0000 7f7ab77fe640 1 -- 172.21.15.27:0/897191216 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa0003520 con 0x7f7ab80d07c0 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.703+0000 7f7ab77fe640 1 -- 172.21.15.27:0/897191216 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310238351 0 0) 0x7f7ab806d4d0 con 0x7f7ab80cf9f0 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.703+0000 7f7ab77fe640 1 -- 172.21.15.27:0/897191216 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab80b4950 con 0x7f7ab80cf9f0 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.703+0000 7f7ab77fe640 1 -- 172.21.15.27:0/897191216 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2091962523 0 0) 0x7f7aa0003520 con 0x7f7ab80d07c0 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.703+0000 7f7abc8f8640 1 -- 172.21.15.27:0/897191216 >> v1:172.21.15.69:6789/0 conn(0x7f7ab80cf9f0 legacy=0x7f7ab80d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.703+0000 7f7abc8f8640 1 -- 172.21.15.27:0/897191216 >> v1:172.21.15.27:6789/0 conn(0x7f7ab80d07c0 legacy=0x7f7ab815d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.703+0000 7f7abc8f8640 1 -- 172.21.15.27:0/897191216 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab806d7b0 con 0x7f7ab8072a30 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7abc8f8640 1 --2- 172.21.15.27:0/897191216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab8072a30 0x7f7ab81624c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009d90 tx=0x7f7ab002bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d504e95916d769b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7ab77fe640 1 -- 172.21.15.27:0/897191216 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0039070 con 0x7f7ab8072a30 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7ab77fe640 1 -- 172.21.15.27:0/897191216 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab002c820 con 0x7f7ab8072a30 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7ab77fe640 1 -- 172.21.15.27:0/897191216 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8001670 con 0x7f7ab8072a30 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7abeb5c640 1 -- 172.21.15.27:0/897191216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab8072a30 msgr2=0x7f7ab81624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7abeb5c640 1 --2- 172.21.15.27:0/897191216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab8072a30 0x7f7ab81624c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009d90 tx=0x7f7ab002bcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7abeb5c640 1 -- 172.21.15.27:0/897191216 shutdown_connections 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7abeb5c640 1 --2- 172.21.15.27:0/897191216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab8072a30 0x7f7ab81624c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7abeb5c640 1 -- 172.21.15.27:0/897191216 >> 172.21.15.27:0/897191216 conn(0x7f7ab806af20 msgr2=0x7f7ab806b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7abeb5c640 1 -- 172.21.15.27:0/897191216 shutdown_connections 2024-03-06T07:28:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.704+0000 7f7abeb5c640 1 -- 172.21.15.27:0/897191216 wait complete. 2024-03-06T07:28:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.706+0000 7f7abeb5c640 1 Processor -- start 2024-03-06T07:28:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.706+0000 7f7abeb5c640 1 -- start start 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.706+0000 7f7abeb5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ab8072a30 0x7f7ab8156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.706+0000 7f7abeb5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab80cf9f0 0x7f7ab8158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.706+0000 7f7abeb5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80d07c0 0x7f7ab815d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.706+0000 7f7abeb5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab8072e10 con 0x7f7ab8072a30 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.706+0000 7f7abeb5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab8071130 con 0x7f7ab80d07c0 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.706+0000 7f7abeb5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab8072f90 con 0x7f7ab80cf9f0 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abc8f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ab8072a30 0x7f7ab8156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abc8f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ab8072a30 0x7f7ab8156c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54808/0 (socket says 172.21.15.27:54808) 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abc8f8640 1 -- 172.21.15.27:0/76849320 learned_addr learned my addr 172.21.15.27:0/76849320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abd0f9640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80d07c0 0x7f7ab815d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7ab7fff640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab80cf9f0 0x7f7ab8158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abc8f8640 1 -- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab80cf9f0 msgr2=0x7f7ab8158980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abc8f8640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab80cf9f0 0x7f7ab8158980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abc8f8640 1 -- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80d07c0 msgr2=0x7f7ab815d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abc8f8640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80d07c0 0x7f7ab815d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abc8f8640 1 -- 172.21.15.27:0/76849320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab8159460 con 0x7f7ab8072a30 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abc8f8640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ab8072a30 0x7f7ab8156c20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009eb0 tx=0x7f7ab0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7ab57fa640 1 -- 172.21.15.27:0/76849320 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0039050 con 0x7f7ab8072a30 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7ab57fa640 1 -- 172.21.15.27:0/76849320 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab0045070 con 0x7f7ab8072a30 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab8159740 con 0x7f7ab8072a30 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7ab57fa640 1 -- 172.21.15.27:0/76849320 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0004670 con 0x7f7ab8072a30 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.707+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab8159d00 con 0x7f7ab8072a30 2024-03-06T07:28:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.708+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab809e320 con 0x7f7ab8072a30 2024-03-06T07:28:39.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.709+0000 7f7ab57fa640 1 -- 172.21.15.27:0/76849320 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ab004a460 con 0x7f7ab8072a30 2024-03-06T07:28:39.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.709+0000 7f7ab57fa640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7aa406b4d0 0x7f7aa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.710+0000 7f7ab57fa640 1 -- 172.21.15.27:0/76849320 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7ab00ac5e0 con 0x7f7ab8072a30 2024-03-06T07:28:39.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.710+0000 7f7ab7fff640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7aa406b4d0 0x7f7aa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.710+0000 7f7ab7fff640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7aa406b4d0 0x7f7aa406d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7ab815ad00 tx=0x7f7aac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.714+0000 7f7ab57fa640 1 -- 172.21.15.27:0/76849320 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ab004f020 con 0x7f7ab8072a30 2024-03-06T07:28:39.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:39.897+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7ab804f410 con 0x7f7ab8072a30 2024-03-06T07:28:40.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.576+0000 7f7ab57fa640 1 -- 172.21.15.27:0/76849320 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v112) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7ab003d360 con 0x7f7ab8072a30 2024-03-06T07:28:40.578 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:28:40.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.579+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7aa406b4d0 msgr2=0x7f7aa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.579+0000 7f7abeb5c640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7aa406b4d0 0x7f7aa406d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7ab815ad00 tx=0x7f7aac009040 comp rx=0 tx=0).stop 2024-03-06T07:28:40.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.579+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ab8072a30 msgr2=0x7f7ab8156c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.579+0000 7f7abeb5c640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ab8072a30 0x7f7ab8156c20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009eb0 tx=0x7f7ab0004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.580+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 shutdown_connections 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.580+0000 7f7abeb5c640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7aa406b4d0 0x7f7aa406d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.580+0000 7f7abeb5c640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ab80d07c0 0x7f7ab815d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.580+0000 7f7abeb5c640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ab80cf9f0 0x7f7ab8158980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.580+0000 7f7abeb5c640 1 --2- 172.21.15.27:0/76849320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ab8072a30 0x7f7ab8156c20 secure :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009eb0 tx=0x7f7ab0004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.580+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 >> 172.21.15.27:0/76849320 conn(0x7f7ab806af20 msgr2=0x7f7ab80b27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.581+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 shutdown_connections 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.581+0000 7f7abeb5c640 1 -- 172.21.15.27:0/76849320 wait complete. 2024-03-06T07:28:40.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:28:40.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.717+0000 7f85b75a1640 1 Processor -- start 2024-03-06T07:28:40.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.717+0000 7f85b75a1640 1 -- start start 2024-03-06T07:28:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.718+0000 7f85b75a1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85b00d2000 0x7f85b015f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.718+0000 7f85b75a1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f85b0161c50 0x7f85b01620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.718+0000 7f85b75a1640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b01625e0 con 0x7f85b0141170 2024-03-06T07:28:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.718+0000 7f85b75a1640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b00d1320 con 0x7f85b00d2000 2024-03-06T07:28:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.718+0000 7f85b75a1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b015edc0 con 0x7f85b0161c50 2024-03-06T07:28:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.718+0000 7f85b4b3c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85b00d2000 0x7f85b015f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.718+0000 7f85b533d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f85b0141170 0x7f85b00d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58102/0 (socket says 172.21.15.27:58102) 2024-03-06T07:28:40.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.718+0000 7f85b533d640 1 -- 172.21.15.27:0/3636538961 learned_addr learned my addr 172.21.15.27:0/3636538961 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:40.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.719+0000 7f85b5b3e640 1 --2- 172.21.15.27:0/3636538961 >> v2:172.21.15.69:3300/0 conn(0x7f85b0161c50 0x7f85b01620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.719+0000 7f85b4b3c640 1 -- 172.21.15.27:0/3636538961 >> v2:172.21.15.69:3300/0 conn(0x7f85b0161c50 msgr2=0x7f85b01620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.719+0000 7f85b4b3c640 1 --2- 172.21.15.27:0/3636538961 >> v2:172.21.15.69:3300/0 conn(0x7f85b0161c50 0x7f85b01620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.719+0000 7f85b4b3c640 1 -- 172.21.15.27:0/3636538961 >> v1:172.21.15.27:6789/0 conn(0x7f85b0141170 legacy=0x7f85b00d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.719+0000 7f85b4b3c640 1 -- 172.21.15.27:0/3636538961 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b0141550 con 0x7f85b00d2000 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.719+0000 7f85b4b3c640 1 --2- 172.21.15.27:0/3636538961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b00d2000 0x7f85b015f250 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f85b015ef40 tx=0x7f85a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2158d9269ace1e22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85affff640 1 -- 172.21.15.27:0/3636538961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a4035020 con 0x7f85b00d2000 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85affff640 1 -- 172.21.15.27:0/3636538961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a4031830 con 0x7f85b00d2000 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85affff640 1 -- 172.21.15.27:0/3636538961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a4031c00 con 0x7f85b00d2000 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85b75a1640 1 -- 172.21.15.27:0/3636538961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b00d2000 msgr2=0x7f85b015f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85b75a1640 1 --2- 172.21.15.27:0/3636538961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b00d2000 0x7f85b015f250 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f85b015ef40 tx=0x7f85a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85b75a1640 1 -- 172.21.15.27:0/3636538961 shutdown_connections 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85b75a1640 1 --2- 172.21.15.27:0/3636538961 >> v2:172.21.15.69:3300/0 conn(0x7f85b0161c50 0x7f85b01620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85b75a1640 1 --2- 172.21.15.27:0/3636538961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b00d2000 0x7f85b015f250 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85b75a1640 1 -- 172.21.15.27:0/3636538961 >> 172.21.15.27:0/3636538961 conn(0x7f85b006eb00 msgr2=0x7f85b006ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85b75a1640 1 -- 172.21.15.27:0/3636538961 shutdown_connections 2024-03-06T07:28:40.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.720+0000 7f85b75a1640 1 -- 172.21.15.27:0/3636538961 wait complete. 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.721+0000 7f85b75a1640 1 Processor -- start 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.721+0000 7f85b75a1640 1 -- start start 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b75a1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b00d2000 0x7f85b01f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b75a1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85b0141170 0x7f85b01442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b75a1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b0161c50 0x7f85b0140280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b75a1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b015e770 con 0x7f85b0161c50 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b75a1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b0162b40 con 0x7f85b0141170 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b75a1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b015ea50 con 0x7f85b00d2000 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b5b3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b0161c50 0x7f85b0140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b5b3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b0161c50 0x7f85b0140280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54826/0 (socket says 172.21.15.27:54826) 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b5b3e640 1 -- 172.21.15.27:0/1648332085 learned_addr learned my addr 172.21.15.27:0/1648332085 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.722+0000 7f85b533d640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b00d2000 0x7f85b01f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b4b3c640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85b0141170 0x7f85b01442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b5b3e640 1 -- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b00d2000 msgr2=0x7f85b01f7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b5b3e640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b00d2000 0x7f85b01f7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b5b3e640 1 -- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85b0141170 msgr2=0x7f85b01442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b5b3e640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85b0141170 0x7f85b01442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b5b3e640 1 -- 172.21.15.27:0/1648332085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b01f8980 con 0x7f85b0161c50 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b4b3c640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85b0141170 0x7f85b01442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b533d640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b00d2000 0x7f85b01f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:40.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b5b3e640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b0161c50 0x7f85b0140280 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f85a800d800 tx=0x7f85a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85adffb640 1 -- 172.21.15.27:0/1648332085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a800b6c0 con 0x7f85b0161c50 2024-03-06T07:28:40.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85adffb640 1 -- 172.21.15.27:0/1648332085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a8019070 con 0x7f85b0161c50 2024-03-06T07:28:40.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85adffb640 1 -- 172.21.15.27:0/1648332085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a800bac0 con 0x7f85b0161c50 2024-03-06T07:28:40.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b01f80e0 con 0x7f85b0161c50 2024-03-06T07:28:40.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.723+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b01f86a0 con 0x7f85b0161c50 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.724+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b00700c0 con 0x7f85b0161c50 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.725+0000 7f85adffb640 1 -- 172.21.15.27:0/1648332085 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85a800bc60 con 0x7f85b0161c50 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.725+0000 7f85adffb640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f859c06b4d0 0x7f859c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.725+0000 7f85adffb640 1 -- 172.21.15.27:0/1648332085 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f85a808a5a0 con 0x7f85b0161c50 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.725+0000 7f85b533d640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f859c06b4d0 0x7f859c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.726+0000 7f85b533d640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f859c06b4d0 0x7f859c06d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f85a00012e0 tx=0x7f85a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.730+0000 7f85adffb640 1 -- 172.21.15.27:0/1648332085 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85a8059510 con 0x7f85b0161c50 2024-03-06T07:28:40.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:40.949+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f85b00b6ac0 con 0x7f85b0161c50 2024-03-06T07:28:41.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.577+0000 7f85adffb640 1 -- 172.21.15.27:0/1648332085 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f85a8058e10 con 0x7f85b0161c50 2024-03-06T07:28:41.580 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:28:41.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.582+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f859c06b4d0 msgr2=0x7f859c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.582+0000 7f85b75a1640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f859c06b4d0 0x7f859c06d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f85a00012e0 tx=0x7f85a0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:41.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.582+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b0161c50 msgr2=0x7f85b0140280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.582+0000 7f85b75a1640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b0161c50 0x7f85b0140280 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f85a800d800 tx=0x7f85a800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.586+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 shutdown_connections 2024-03-06T07:28:41.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.586+0000 7f85b75a1640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f859c06b4d0 0x7f859c06d990 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.586+0000 7f85b75a1640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b0161c50 0x7f85b0140280 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.586+0000 7f85b75a1640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85b0141170 0x7f85b01442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.586+0000 7f85b75a1640 1 --2- 172.21.15.27:0/1648332085 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85b00d2000 0x7f85b01f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.586+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 >> 172.21.15.27:0/1648332085 conn(0x7f85b006eb00 msgr2=0x7f85b015db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.586+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 shutdown_connections 2024-03-06T07:28:41.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.586+0000 7f85b75a1640 1 -- 172.21.15.27:0/1648332085 wait complete. 2024-03-06T07:28:41.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:41.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.731+0000 7fbfb8ec9640 1 Processor -- start 2024-03-06T07:28:41.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.731+0000 7fbfb8ec9640 1 -- start start 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.732+0000 7fbfb8ec9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 0x7fbfb406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.732+0000 7fbfb8ec9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbfb40aee10 0x7fbfb40b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.732+0000 7fbfb8ec9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb4071240 con 0x7fbfb406dd60 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.732+0000 7fbfb8ec9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb406e1c0 con 0x7fbfb40aee10 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.732+0000 7fbfb8ec9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb4071b70 con 0x7fbfb4158520 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb2d9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbfb40aee10 0x7fbfb40b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb259c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 0x7fbfb406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb1d9b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbfb406dd60 0x7fbfb40ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:33058/0 (socket says 172.21.15.27:33058) 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb1d9b640 1 -- 172.21.15.27:0/898195672 learned_addr learned my addr 172.21.15.27:0/898195672 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb159a640 1 -- 172.21.15.27:0/898195672 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863429129 0 0) 0x7fbfb4071240 con 0x7fbfb406dd60 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb159a640 1 -- 172.21.15.27:0/898195672 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9c003520 con 0x7fbfb406dd60 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb259c640 1 -- 172.21.15.27:0/898195672 >> v2:172.21.15.69:3300/0 conn(0x7fbfb40aee10 msgr2=0x7fbfb40b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb259c640 1 --2- 172.21.15.27:0/898195672 >> v2:172.21.15.69:3300/0 conn(0x7fbfb40aee10 0x7fbfb40b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb259c640 1 -- 172.21.15.27:0/898195672 >> v1:172.21.15.69:6789/0 conn(0x7fbfb406dd60 legacy=0x7fbfb40ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb259c640 1 -- 172.21.15.27:0/898195672 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb4071520 con 0x7fbfb4158520 2024-03-06T07:28:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.733+0000 7fbfb2d9d640 1 --2- 172.21.15.27:0/898195672 >> v2:172.21.15.69:3300/0 conn(0x7fbfb40aee10 0x7fbfb40b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.734+0000 7fbfb259c640 1 --2- 172.21.15.27:0/898195672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 0x7fbfb406d820 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40097a0 tx=0x7fbfa402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bce59bd21938fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.734+0000 7fbfb159a640 1 -- 172.21.15.27:0/898195672 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4035020 con 0x7fbfb4158520 2024-03-06T07:28:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.734+0000 7fbfb159a640 1 -- 172.21.15.27:0/898195672 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa4031830 con 0x7fbfb4158520 2024-03-06T07:28:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.734+0000 7fbfb159a640 1 -- 172.21.15.27:0/898195672 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4031be0 con 0x7fbfb4158520 2024-03-06T07:28:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.735+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/898195672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 msgr2=0x7fbfb406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.735+0000 7fbfb8ec9640 1 --2- 172.21.15.27:0/898195672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 0x7fbfb406d820 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40097a0 tx=0x7fbfa402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.735+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/898195672 shutdown_connections 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.735+0000 7fbfb8ec9640 1 --2- 172.21.15.27:0/898195672 >> v2:172.21.15.69:3300/0 conn(0x7fbfb40aee10 0x7fbfb40b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.735+0000 7fbfb8ec9640 1 --2- 172.21.15.27:0/898195672 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 0x7fbfb406d820 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.735+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/898195672 >> 172.21.15.27:0/898195672 conn(0x7fbfb4066d00 msgr2=0x7fbfb40670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.735+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/898195672 shutdown_connections 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.735+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/898195672 wait complete. 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.736+0000 7fbfb8ec9640 1 Processor -- start 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.736+0000 7fbfb8ec9640 1 -- start start 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb8ec9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb406dd60 0x7fbfb41f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb8ec9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfb40aee10 0x7fbfb41f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb8ec9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 0x7fbfb4155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb8ec9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb4158900 con 0x7fbfb406dd60 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb8ec9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb4072000 con 0x7fbfb40aee10 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb8ec9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb4158a80 con 0x7fbfb4158520 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb406dd60 0x7fbfb41f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb406dd60 0x7fbfb41f0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54838/0 (socket says 172.21.15.27:54838) 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb259c640 1 -- 172.21.15.27:0/768193084 learned_addr learned my addr 172.21.15.27:0/768193084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb2d9d640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 0x7fbfb4155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb1d9b640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfb40aee10 0x7fbfb41f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb259c640 1 -- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 msgr2=0x7fbfb4155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb259c640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 0x7fbfb4155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb259c640 1 -- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfb40aee10 msgr2=0x7fbfb41f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb259c640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfb40aee10 0x7fbfb41f3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb259c640 1 -- 172.21.15.27:0/768193084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb41f39c0 con 0x7fbfb406dd60 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.737+0000 7fbfb259c640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb406dd60 0x7fbfb41f0720 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4000c00 tx=0x7fbfa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.738+0000 7fbfa2ffd640 1 -- 172.21.15.27:0/768193084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4035050 con 0x7fbfb406dd60 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.738+0000 7fbfa2ffd640 1 -- 172.21.15.27:0/768193084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa403e070 con 0x7fbfb406dd60 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.738+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb41f3ca0 con 0x7fbfb406dd60 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.738+0000 7fbfa2ffd640 1 -- 172.21.15.27:0/768193084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4003610 con 0x7fbfb406dd60 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.738+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb4141b00 con 0x7fbfb406dd60 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.739+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb4067ff0 con 0x7fbfb406dd60 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.739+0000 7fbfa2ffd640 1 -- 172.21.15.27:0/768193084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfa4038020 con 0x7fbfb406dd60 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.740+0000 7fbfa2ffd640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf9406b4d0 0x7fbf9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.740+0000 7fbfa2ffd640 1 -- 172.21.15.27:0/768193084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfa40abc70 con 0x7fbfb406dd60 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.740+0000 7fbfb1d9b640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf9406b4d0 0x7fbf9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.740+0000 7fbfb1d9b640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf9406b4d0 0x7fbf9406d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbfb40b0f00 tx=0x7fbfa8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.744+0000 7fbfa2ffd640 1 -- 172.21.15.27:0/768193084 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfa4031800 con 0x7fbfb406dd60 2024-03-06T07:28:41.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.926+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfb40b2dc0 con 0x7fbfb406dd60 2024-03-06T07:28:41.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.927+0000 7fbfa2ffd640 1 -- 172.21.15.27:0/768193084 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16568 (secure 0 0 0) 0x7fbfa407aaf0 con 0x7fbfb406dd60 2024-03-06T07:28:41.929 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:41.930 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":113,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:28:40.951277+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.260334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.929245+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:41.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf9406b4d0 msgr2=0x7fbf9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf9406b4d0 0x7fbf9406d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbfb40b0f00 tx=0x7fbfa8009040 comp rx=0 tx=0).stop 2024-03-06T07:28:41.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb406dd60 msgr2=0x7fbfb41f0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb406dd60 0x7fbfb41f0720 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4000c00 tx=0x7fbfa40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 shutdown_connections 2024-03-06T07:28:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf9406b4d0 0x7fbf9406d990 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbfb4158520 0x7fbfb4155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfb40aee10 0x7fbfb41f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 --2- 172.21.15.27:0/768193084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb406dd60 0x7fbfb41f0720 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 >> 172.21.15.27:0/768193084 conn(0x7fbfb4066d00 msgr2=0x7fbfb4152b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 shutdown_connections 2024-03-06T07:28:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:41.929+0000 7fbfb8ec9640 1 -- 172.21.15.27:0/768193084 wait complete. 2024-03-06T07:28:41.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.047+0000 7f429b79a640 1 Processor -- start 2024-03-06T07:28:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.047+0000 7f429b79a640 1 -- start start 2024-03-06T07:28:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.048+0000 7f429b79a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42940d0880 0x7f4294168c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.048+0000 7f429b79a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42940cf130 con 0x7f429415e060 2024-03-06T07:28:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.048+0000 7f429b79a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42940d0dd0 con 0x7f429406fe20 2024-03-06T07:28:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.048+0000 7f429b79a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940d10b0 con 0x7f42940d0880 2024-03-06T07:28:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.048+0000 7f4299d37640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42940d0880 0x7f4294168c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.048+0000 7f4299536640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f429415e060 0x7f42940ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58104/0 (socket says 172.21.15.27:58104) 2024-03-06T07:28:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.048+0000 7f4299536640 1 -- 172.21.15.27:0/3878756402 learned_addr learned my addr 172.21.15.27:0/3878756402 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.048+0000 7f4299d37640 1 -- 172.21.15.27:0/3878756402 >> v1:172.21.15.69:6789/0 conn(0x7f429406fe20 legacy=0x7f42940cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.049+0000 7f4299d37640 1 -- 172.21.15.27:0/3878756402 >> v1:172.21.15.27:6789/0 conn(0x7f429415e060 legacy=0x7f42940ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.049+0000 7f428bfff640 1 -- 172.21.15.27:0/3878756402 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714399220 0 0) 0x7f42940cf130 con 0x7f429415e060 2024-03-06T07:28:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.049+0000 7f4299d37640 1 -- 172.21.15.27:0/3878756402 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940cc0e0 con 0x7f42940d0880 2024-03-06T07:28:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.049+0000 7f4299d37640 1 --2- 172.21.15.27:0/3878756402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 0x7f4294168c20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f428c009d90 tx=0x7f428c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ca8f896c034bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.049+0000 7f428bfff640 1 -- 172.21.15.27:0/3878756402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c039070 con 0x7f42940d0880 2024-03-06T07:28:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.049+0000 7f428bfff640 1 -- 172.21.15.27:0/3878756402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428c02cd00 con 0x7f42940d0880 2024-03-06T07:28:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.049+0000 7f428bfff640 1 -- 172.21.15.27:0/3878756402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c034660 con 0x7f42940d0880 2024-03-06T07:28:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.050+0000 7f429b79a640 1 -- 172.21.15.27:0/3878756402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 msgr2=0x7f4294168c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.050+0000 7f429b79a640 1 --2- 172.21.15.27:0/3878756402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 0x7f4294168c20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f428c009d90 tx=0x7f428c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.050+0000 7f429b79a640 1 -- 172.21.15.27:0/3878756402 shutdown_connections 2024-03-06T07:28:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.050+0000 7f429b79a640 1 --2- 172.21.15.27:0/3878756402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 0x7f4294168c20 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.050+0000 7f429b79a640 1 -- 172.21.15.27:0/3878756402 >> 172.21.15.27:0/3878756402 conn(0x7f42940d1a90 msgr2=0x7f42940d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.050+0000 7f429b79a640 1 -- 172.21.15.27:0/3878756402 shutdown_connections 2024-03-06T07:28:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.050+0000 7f429b79a640 1 -- 172.21.15.27:0/3878756402 wait complete. 2024-03-06T07:28:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.051+0000 7f429b79a640 1 Processor -- start 2024-03-06T07:28:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.051+0000 7f429b79a640 1 -- start start 2024-03-06T07:28:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f429b79a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f429406fe20 0x7f429415e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f429b79a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 0x7f42940b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f429b79a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f429415e060 0x7f4294160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f429b79a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429415e440 con 0x7f42940d0880 2024-03-06T07:28:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f429b79a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42940cc6a0 con 0x7f429415e060 2024-03-06T07:28:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f429b79a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42940cdbd0 con 0x7f429406fe20 2024-03-06T07:28:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4298d35640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 0x7f42940b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4298d35640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 0x7f42940b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54854/0 (socket says 172.21.15.27:54854) 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4298d35640 1 -- 172.21.15.27:0/2777135743 learned_addr learned my addr 172.21.15.27:0/2777135743 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4299d37640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f429415e060 0x7f4294160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4299536640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f429406fe20 0x7f429415e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4298d35640 1 -- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f429406fe20 msgr2=0x7f429415e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4298d35640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f429406fe20 0x7f429415e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4298d35640 1 -- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f429415e060 msgr2=0x7f4294160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4298d35640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f429415e060 0x7f4294160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4298d35640 1 -- 172.21.15.27:0/2777135743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940c5e10 con 0x7f42940d0880 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.052+0000 7f4299536640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f429406fe20 0x7f429415e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.053+0000 7f4298d35640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 0x7f42940b2180 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f428400c430 tx=0x7f428400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.053+0000 7f4289ffb640 1 -- 172.21.15.27:0/2777135743 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4284017070 con 0x7f42940d0880 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.053+0000 7f4289ffb640 1 -- 172.21.15.27:0/2777135743 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42840052d0 con 0x7f42940d0880 2024-03-06T07:28:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.053+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42940c60f0 con 0x7f42940d0880 2024-03-06T07:28:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.053+0000 7f4289ffb640 1 -- 172.21.15.27:0/2777135743 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4284003830 con 0x7f42940d0880 2024-03-06T07:28:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.053+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42940c66b0 con 0x7f42940d0880 2024-03-06T07:28:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.054+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4294067ff0 con 0x7f42940d0880 2024-03-06T07:28:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.054+0000 7f4289ffb640 1 -- 172.21.15.27:0/2777135743 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f428401a070 con 0x7f42940d0880 2024-03-06T07:28:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.055+0000 7f4289ffb640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f427806b4d0 0x7f427806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.055+0000 7f4289ffb640 1 -- 172.21.15.27:0/2777135743 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4284089540 con 0x7f42940d0880 2024-03-06T07:28:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.055+0000 7f4299536640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f427806b4d0 0x7f427806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.056+0000 7f4299536640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f427806b4d0 0x7f427806d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4290007fd0 tx=0x7f4290007dc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.060+0000 7f4289ffb640 1 -- 172.21.15.27:0/2777135743 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4284057d70 con 0x7f42940d0880 2024-03-06T07:28:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.245+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42940d20c0 con 0x7f42940d0880 2024-03-06T07:28:42.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.246+0000 7f4289ffb640 1 -- 172.21.15.27:0/2777135743 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16568 (secure 0 0 0) 0x7f428405be10 con 0x7f42940d0880 2024-03-06T07:28:42.249 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:42.249 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":113,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:28:40.951277+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.260334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.929245+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.248+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f427806b4d0 msgr2=0x7f427806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.248+0000 7f429b79a640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f427806b4d0 0x7f427806d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4290007fd0 tx=0x7f4290007dc0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.248+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 msgr2=0x7f42940b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.248+0000 7f429b79a640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 0x7f42940b2180 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f428400c430 tx=0x7f428400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.249+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 shutdown_connections 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.249+0000 7f429b79a640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f427806b4d0 0x7f427806d990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.249+0000 7f429b79a640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f429415e060 0x7f4294160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.249+0000 7f429b79a640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42940d0880 0x7f42940b2180 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.249+0000 7f429b79a640 1 --2- 172.21.15.27:0/2777135743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f429406fe20 0x7f429415e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.249+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 >> 172.21.15.27:0/2777135743 conn(0x7f42940d1a90 msgr2=0x7f429406f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.249+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 shutdown_connections 2024-03-06T07:28:42.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.249+0000 7f429b79a640 1 -- 172.21.15.27:0/2777135743 wait complete. 2024-03-06T07:28:42.262 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:28:42.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.370+0000 7f1983ccf640 1 Processor -- start 2024-03-06T07:28:42.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.370+0000 7f1983ccf640 1 -- start start 2024-03-06T07:28:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f1983ccf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f197c15b130 0x7f197c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f1983ccf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f197c0ce900 0x7f197c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f1983ccf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197c15bc40 con 0x7f197c144640 2024-03-06T07:28:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f1983ccf640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197c06df70 con 0x7f197c0ce900 2024-03-06T07:28:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f1983ccf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197c0cf270 con 0x7f197c15b130 2024-03-06T07:28:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f198226c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f197c0ce900 0x7f197c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f198226c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f197c0ce900 0x7f197c0ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54856/0 (socket says 172.21.15.27:54856) 2024-03-06T07:28:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f198226c640 1 -- 172.21.15.27:0/1529264426 learned_addr learned my addr 172.21.15.27:0/1529264426 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f198126a640 1 --2- 172.21.15.27:0/1529264426 >> v2:172.21.15.69:3300/0 conn(0x7f197c15b130 0x7f197c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f198226c640 1 -- 172.21.15.27:0/1529264426 >> v2:172.21.15.69:3300/0 conn(0x7f197c15b130 msgr2=0x7f197c15b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f198226c640 1 --2- 172.21.15.27:0/1529264426 >> v2:172.21.15.69:3300/0 conn(0x7f197c15b130 0x7f197c15b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f198226c640 1 -- 172.21.15.27:0/1529264426 >> v1:172.21.15.69:6789/0 conn(0x7f197c144640 legacy=0x7f197c06e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f198226c640 1 -- 172.21.15.27:0/1529264426 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c15bf20 con 0x7f197c0ce900 2024-03-06T07:28:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.371+0000 7f198126a640 1 --2- 172.21.15.27:0/1529264426 >> v2:172.21.15.69:3300/0 conn(0x7f197c15b130 0x7f197c15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.372+0000 7f198226c640 1 --2- 172.21.15.27:0/1529264426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 0x7f197c0ced30 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f197c0706a0 tx=0x7f196c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80f7aeb406326a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.372+0000 7f1980a69640 1 -- 172.21.15.27:0/1529264426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c035020 con 0x7f197c0ce900 2024-03-06T07:28:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.372+0000 7f1980a69640 1 -- 172.21.15.27:0/1529264426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196c031870 con 0x7f197c0ce900 2024-03-06T07:28:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.372+0000 7f1980a69640 1 -- 172.21.15.27:0/1529264426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c031c40 con 0x7f197c0ce900 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.373+0000 7f1983ccf640 1 -- 172.21.15.27:0/1529264426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 msgr2=0x7f197c0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.373+0000 7f1983ccf640 1 --2- 172.21.15.27:0/1529264426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 0x7f197c0ced30 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f197c0706a0 tx=0x7f196c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.373+0000 7f1983ccf640 1 -- 172.21.15.27:0/1529264426 shutdown_connections 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.373+0000 7f1983ccf640 1 --2- 172.21.15.27:0/1529264426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 0x7f197c0ced30 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.373+0000 7f1983ccf640 1 --2- 172.21.15.27:0/1529264426 >> v2:172.21.15.69:3300/0 conn(0x7f197c15b130 0x7f197c15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.373+0000 7f1983ccf640 1 -- 172.21.15.27:0/1529264426 >> 172.21.15.27:0/1529264426 conn(0x7f197c066d00 msgr2=0x7f197c067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.373+0000 7f1983ccf640 1 -- 172.21.15.27:0/1529264426 shutdown_connections 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.373+0000 7f1983ccf640 1 -- 172.21.15.27:0/1529264426 wait complete. 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.374+0000 7f1983ccf640 1 Processor -- start 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.374+0000 7f1983ccf640 1 -- start start 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1983ccf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 0x7f197c13c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1983ccf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c144640 0x7f197c0cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1983ccf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f197c15b130 0x7f197c142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1983ccf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c070aa0 con 0x7f197c0ce900 2024-03-06T07:28:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1983ccf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c06eb10 con 0x7f197c144640 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1983ccf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c070c20 con 0x7f197c15b130 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1981a6b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 0x7f197c13c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1981a6b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 0x7f197c13c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54866/0 (socket says 172.21.15.27:54866) 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1981a6b640 1 -- 172.21.15.27:0/3379123993 learned_addr learned my addr 172.21.15.27:0/3379123993 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f198226c640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f197c15b130 0x7f197c142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1981a6b640 1 -- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f197c15b130 msgr2=0x7f197c142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1981a6b640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f197c15b130 0x7f197c142840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f198126a640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c144640 0x7f197c0cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1981a6b640 1 -- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c144640 msgr2=0x7f197c0cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1981a6b640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c144640 0x7f197c0cb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1981a6b640 1 -- 172.21.15.27:0/3379123993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c0cc2c0 con 0x7f197c0ce900 2024-03-06T07:28:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.375+0000 7f1981a6b640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 0x7f197c13c5c0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f197400c7c0 tx=0x7f197400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.376+0000 7f197a7fc640 1 -- 172.21.15.27:0/3379123993 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974016020 con 0x7f197c0ce900 2024-03-06T07:28:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.376+0000 7f197a7fc640 1 -- 172.21.15.27:0/3379123993 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197400aa50 con 0x7f197c0ce900 2024-03-06T07:28:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.376+0000 7f197a7fc640 1 -- 172.21.15.27:0/3379123993 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974012420 con 0x7f197c0ce900 2024-03-06T07:28:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.376+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197c0cc5a0 con 0x7f197c0ce900 2024-03-06T07:28:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.376+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197c0c68c0 con 0x7f197c0ce900 2024-03-06T07:28:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.377+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197c0733c0 con 0x7f197c0ce900 2024-03-06T07:28:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.377+0000 7f197a7fc640 1 -- 172.21.15.27:0/3379123993 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1974019020 con 0x7f197c0ce900 2024-03-06T07:28:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.377+0000 7f197a7fc640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f196806b4d0 0x7f196806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.378+0000 7f197a7fc640 1 -- 172.21.15.27:0/3379123993 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1974089fd0 con 0x7f197c0ce900 2024-03-06T07:28:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.378+0000 7f198126a640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f196806b4d0 0x7f196806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.378+0000 7f198126a640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f196806b4d0 0x7f196806d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f197c143b00 tx=0x7f1970008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.382+0000 7f197a7fc640 1 -- 172.21.15.27:0/3379123993 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1974020070 con 0x7f197c0ce900 2024-03-06T07:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:42.611+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f197c04f410 con 0x7f197c0ce900 2024-03-06T07:28:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.599+0000 7f197a7fc640 1 -- 172.21.15.27:0/3379123993 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v74) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1974058e50 con 0x7f197c0ce900 2024-03-06T07:28:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f196806b4d0 msgr2=0x7f196806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f196806b4d0 0x7f196806d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f197c143b00 tx=0x7f1970008040 comp rx=0 tx=0).stop 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 msgr2=0x7f197c13c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 0x7f197c13c5c0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f197400c7c0 tx=0x7f197400cc80 comp rx=0 tx=0).stop 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 shutdown_connections 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f196806b4d0 0x7f196806d990 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f197c15b130 0x7f197c142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c144640 0x7f197c0cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 --2- 172.21.15.27:0/3379123993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f197c0ce900 0x7f197c13c5c0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 >> 172.21.15.27:0/3379123993 conn(0x7f197c066d00 msgr2=0x7f197c15d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 shutdown_connections 2024-03-06T07:28:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.602+0000 7f1983ccf640 1 -- 172.21.15.27:0/3379123993 wait complete. 2024-03-06T07:28:43.615 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:28:43.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:28:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.723+0000 7fca8dd83640 1 Processor -- start 2024-03-06T07:28:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.723+0000 7fca8dd83640 1 -- start start 2024-03-06T07:28:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.723+0000 7fca8dd83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8815dd90 0x7fca880ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.723+0000 7fca8dd83640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca880d29d0 0x7fca880d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca8dd83640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca880d3740 con 0x7fca880d4720 2024-03-06T07:28:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca8dd83640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca880d3400 con 0x7fca880d29d0 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca8dd83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca880d3120 con 0x7fca8815dd90 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca86ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca880d29d0 0x7fca880d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca86ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca880d29d0 0x7fca880d41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44552/0 (socket says 172.21.15.27:44552) 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca86ffd640 1 -- 172.21.15.27:0/390435395 learned_addr learned my addr 172.21.15.27:0/390435395 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca877fe640 1 --2- 172.21.15.27:0/390435395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8815dd90 0x7fca880ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca86ffd640 1 -- 172.21.15.27:0/390435395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8815dd90 msgr2=0x7fca880ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca86ffd640 1 --2- 172.21.15.27:0/390435395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8815dd90 0x7fca880ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca86ffd640 1 -- 172.21.15.27:0/390435395 >> v1:172.21.15.69:6789/0 conn(0x7fca880d4720 legacy=0x7fca880b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca86ffd640 1 -- 172.21.15.27:0/390435395 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca880d39e0 con 0x7fca880d29d0 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.724+0000 7fca877fe640 1 --2- 172.21.15.27:0/390435395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8815dd90 0x7fca880ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca86ffd640 1 --2- 172.21.15.27:0/390435395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 0x7fca880d41e0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fca7c009870 tx=0x7fca7c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b746d2923b8891 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca867fc640 1 -- 172.21.15.27:0/390435395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c034070 con 0x7fca880d29d0 2024-03-06T07:28:43.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca867fc640 1 -- 172.21.15.27:0/390435395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c030840 con 0x7fca880d29d0 2024-03-06T07:28:43.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca867fc640 1 -- 172.21.15.27:0/390435395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c030c10 con 0x7fca880d29d0 2024-03-06T07:28:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca8dd83640 1 -- 172.21.15.27:0/390435395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 msgr2=0x7fca880d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca8dd83640 1 --2- 172.21.15.27:0/390435395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 0x7fca880d41e0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fca7c009870 tx=0x7fca7c02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca8dd83640 1 -- 172.21.15.27:0/390435395 shutdown_connections 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca8dd83640 1 --2- 172.21.15.27:0/390435395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 0x7fca880d41e0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca8dd83640 1 --2- 172.21.15.27:0/390435395 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca8815dd90 0x7fca880ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.725+0000 7fca8dd83640 1 -- 172.21.15.27:0/390435395 >> 172.21.15.27:0/390435395 conn(0x7fca88071960 msgr2=0x7fca88071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.726+0000 7fca8dd83640 1 -- 172.21.15.27:0/390435395 shutdown_connections 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.726+0000 7fca8dd83640 1 -- 172.21.15.27:0/390435395 wait complete. 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.727+0000 7fca8dd83640 1 Processor -- start 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.727+0000 7fca8dd83640 1 -- start start 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca8dd83640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 0x7fca88156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca8dd83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca880d4720 0x7fca88160b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca8dd83640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca8815dd90 0x7fca880b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca8dd83640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca880d2df0 con 0x7fca880d29d0 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca8dd83640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8815e170 con 0x7fca8815dd90 2024-03-06T07:28:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca8dd83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca880b7420 con 0x7fca880d4720 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 0x7fca88156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 0x7fca88156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44568/0 (socket says 172.21.15.27:44568) 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca877fe640 1 -- 172.21.15.27:0/4035694826 learned_addr learned my addr 172.21.15.27:0/4035694826 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca86ffd640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca880d4720 0x7fca88160b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca87fff640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca8815dd90 0x7fca880b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca877fe640 1 -- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca880d4720 msgr2=0x7fca88160b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca877fe640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca880d4720 0x7fca88160b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca877fe640 1 -- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca8815dd90 msgr2=0x7fca880b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca877fe640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca8815dd90 0x7fca880b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca877fe640 1 -- 172.21.15.27:0/4035694826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca881e7c30 con 0x7fca880d29d0 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca87fff640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca8815dd90 0x7fca880b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca877fe640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 0x7fca88156b70 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fca7800d7d0 tx=0x7fca7800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca6ffff640 1 -- 172.21.15.27:0/4035694826 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78004030 con 0x7fca880d29d0 2024-03-06T07:28:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca6ffff640 1 -- 172.21.15.27:0/4035694826 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca780041d0 con 0x7fca880d29d0 2024-03-06T07:28:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca881e7f10 con 0x7fca880d29d0 2024-03-06T07:28:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.728+0000 7fca6ffff640 1 -- 172.21.15.27:0/4035694826 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca780096a0 con 0x7fca880d29d0 2024-03-06T07:28:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.729+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca881e87b0 con 0x7fca880d29d0 2024-03-06T07:28:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.730+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca881e8300 con 0x7fca880d29d0 2024-03-06T07:28:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.730+0000 7fca6ffff640 1 -- 172.21.15.27:0/4035694826 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca78009840 con 0x7fca880d29d0 2024-03-06T07:28:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.730+0000 7fca6ffff640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca7406b4d0 0x7fca7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.730+0000 7fca6ffff640 1 -- 172.21.15.27:0/4035694826 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca7808a440 con 0x7fca880d29d0 2024-03-06T07:28:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.731+0000 7fca86ffd640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca7406b4d0 0x7fca7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.731+0000 7fca86ffd640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca7406b4d0 0x7fca7406d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fca7c036000 tx=0x7fca7c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.735+0000 7fca6ffff640 1 -- 172.21.15.27:0/4035694826 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca780592d0 con 0x7fca880d29d0 2024-03-06T07:28:43.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:43.954+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fca880cf170 con 0x7fca880d29d0 2024-03-06T07:28:44.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.604+0000 7fca6ffff640 1 -- 172.21.15.27:0/4035694826 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v75) v1 ==== 122+0+0 (secure 0 0 0) 0x7fca78058bd0 con 0x7fca880d29d0 2024-03-06T07:28:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca7406b4d0 msgr2=0x7fca7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca7406b4d0 0x7fca7406d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fca7c036000 tx=0x7fca7c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 msgr2=0x7fca88156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 0x7fca88156b70 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fca7800d7d0 tx=0x7fca7800dc90 comp rx=0 tx=0).stop 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 shutdown_connections 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca7406b4d0 0x7fca7406d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca8815dd90 0x7fca880b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca880d4720 0x7fca88160b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 --2- 172.21.15.27:0/4035694826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca880d29d0 0x7fca88156b70 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 >> 172.21.15.27:0/4035694826 conn(0x7fca88071960 msgr2=0x7fca880b6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 shutdown_connections 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.606+0000 7fca8dd83640 1 -- 172.21.15.27:0/4035694826 wait complete. 2024-03-06T07:28:44.620 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:28:44.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:44.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.725+0000 7fde05f93640 1 Processor -- start 2024-03-06T07:28:44.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.725+0000 7fde05f93640 1 -- start start 2024-03-06T07:28:44.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.725+0000 7fde05f93640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde000b9a10 0x7fde000d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.725+0000 7fde05f93640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde000cea30 con 0x7fde000b95a0 2024-03-06T07:28:44.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.725+0000 7fde05f93640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde000cf3c0 con 0x7fde0015df00 2024-03-06T07:28:44.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.725+0000 7fde05f93640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde000cf0e0 con 0x7fde000b9a10 2024-03-06T07:28:44.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfeffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fde000b95a0 0x7fde000b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34788/0 (socket says 172.21.15.27:34788) 2024-03-06T07:28:44.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfeffd640 1 -- 172.21.15.27:0/3136098590 learned_addr learned my addr 172.21.15.27:0/3136098590 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:44.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfffff640 1 --2- 172.21.15.27:0/3136098590 >> v2:172.21.15.69:3300/0 conn(0x7fde000b9a10 0x7fde000d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfe7fc640 1 -- 172.21.15.27:0/3136098590 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274823989 0 0) 0x7fde000cea30 con 0x7fde000b95a0 2024-03-06T07:28:44.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfe7fc640 1 -- 172.21.15.27:0/3136098590 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde8003520 con 0x7fde000b95a0 2024-03-06T07:28:44.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfe7fc640 1 -- 172.21.15.27:0/3136098590 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181615821 0 0) 0x7fde000cf3c0 con 0x7fde0015df00 2024-03-06T07:28:44.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfe7fc640 1 -- 172.21.15.27:0/3136098590 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde000cea30 con 0x7fde0015df00 2024-03-06T07:28:44.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfe7fc640 1 -- 172.21.15.27:0/3136098590 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1456157798 0 0) 0x7fdde8003520 con 0x7fde000b95a0 2024-03-06T07:28:44.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfffff640 1 -- 172.21.15.27:0/3136098590 >> v1:172.21.15.69:6789/0 conn(0x7fde0015df00 legacy=0x7fde000d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfffff640 1 -- 172.21.15.27:0/3136098590 >> v1:172.21.15.27:6789/0 conn(0x7fde000b95a0 legacy=0x7fde000b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.726+0000 7fddfffff640 1 -- 172.21.15.27:0/3136098590 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde000ce470 con 0x7fde000b9a10 2024-03-06T07:28:44.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.727+0000 7fddfffff640 1 --2- 172.21.15.27:0/3136098590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde000b9a10 0x7fde000d3220 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fde000cf590 tx=0x7fddf802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddf6dbe27be3eb24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.727+0000 7fddfe7fc640 1 -- 172.21.15.27:0/3136098590 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8039070 con 0x7fde000b9a10 2024-03-06T07:28:44.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.727+0000 7fddfe7fc640 1 -- 172.21.15.27:0/3136098590 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf802c930 con 0x7fde000b9a10 2024-03-06T07:28:44.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.727+0000 7fddfe7fc640 1 -- 172.21.15.27:0/3136098590 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4001670 con 0x7fde000b9a10 2024-03-06T07:28:44.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.728+0000 7fde05f93640 1 -- 172.21.15.27:0/3136098590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde000b9a10 msgr2=0x7fde000d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.728+0000 7fde05f93640 1 --2- 172.21.15.27:0/3136098590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde000b9a10 0x7fde000d3220 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fde000cf590 tx=0x7fddf802be00 comp rx=0 tx=0).stop 2024-03-06T07:28:44.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.728+0000 7fde05f93640 1 -- 172.21.15.27:0/3136098590 shutdown_connections 2024-03-06T07:28:44.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.728+0000 7fde05f93640 1 --2- 172.21.15.27:0/3136098590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde000b9a10 0x7fde000d3220 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.728+0000 7fde05f93640 1 -- 172.21.15.27:0/3136098590 >> 172.21.15.27:0/3136098590 conn(0x7fde00071960 msgr2=0x7fde00071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.728+0000 7fde05f93640 1 -- 172.21.15.27:0/3136098590 shutdown_connections 2024-03-06T07:28:44.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.728+0000 7fde05f93640 1 -- 172.21.15.27:0/3136098590 wait complete. 2024-03-06T07:28:44.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.730+0000 7fde05f93640 1 Processor -- start 2024-03-06T07:28:44.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.730+0000 7fde05f93640 1 -- start start 2024-03-06T07:28:44.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fde05f93640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde000b95a0 0x7fde000c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fde05f93640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde000b9a10 0x7fde0014e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fde05f93640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde0015df00 0x7fde000cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fde05f93640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde000cedf0 con 0x7fde000b9a10 2024-03-06T07:28:44.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fde05f93640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde000cec70 con 0x7fde000b95a0 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fde05f93640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde000b5e10 con 0x7fde0015df00 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde000b9a10 0x7fde0014e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde000b9a10 0x7fde0014e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44570/0 (socket says 172.21.15.27:44570) 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfeffd640 1 -- 172.21.15.27:0/2573242851 learned_addr learned my addr 172.21.15.27:0/2573242851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfffff640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde0015df00 0x7fde000cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddff7fe640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde000b95a0 0x7fde000c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfeffd640 1 -- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde0015df00 msgr2=0x7fde000cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfeffd640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde0015df00 0x7fde000cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfeffd640 1 -- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde000b95a0 msgr2=0x7fde000c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfeffd640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde000b95a0 0x7fde000c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfeffd640 1 -- 172.21.15.27:0/2573242851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde000cc030 con 0x7fde000b9a10 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfffff640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde0015df00 0x7fde000cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:44.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fddfeffd640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde000b9a10 0x7fde0014e1e0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fddf400c7b0 tx=0x7fddf400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fdde7fff640 1 -- 172.21.15.27:0/2573242851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4017070 con 0x7fde000b9a10 2024-03-06T07:28:44.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fdde7fff640 1 -- 172.21.15.27:0/2573242851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf4005ad0 con 0x7fde000b9a10 2024-03-06T07:28:44.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde000cc310 con 0x7fde000b9a10 2024-03-06T07:28:44.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fdde7fff640 1 -- 172.21.15.27:0/2573242851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4007070 con 0x7fde000b9a10 2024-03-06T07:28:44.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.732+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde000cc8d0 con 0x7fde000b9a10 2024-03-06T07:28:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.734+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde00067ff0 con 0x7fde000b9a10 2024-03-06T07:28:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.734+0000 7fdde7fff640 1 -- 172.21.15.27:0/2573242851 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddf401a070 con 0x7fde000b9a10 2024-03-06T07:28:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.735+0000 7fdde7fff640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddec06b4d0 0x7fddec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.735+0000 7fdde7fff640 1 -- 172.21.15.27:0/2573242851 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddf4089700 con 0x7fde000b9a10 2024-03-06T07:28:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.735+0000 7fddff7fe640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddec06b4d0 0x7fddec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.735+0000 7fddff7fe640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddec06b4d0 0x7fddec06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fddf00010e0 tx=0x7fddf0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.739+0000 7fdde7fff640 1 -- 172.21.15.27:0/2573242851 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddf4058610 con 0x7fde000b9a10 2024-03-06T07:28:44.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.957+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde000d2a50 con 0x7fde000b9a10 2024-03-06T07:28:44.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.958+0000 7fdde7fff640 1 -- 172.21.15.27:0/2573242851 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fddf4057f10 con 0x7fde000b9a10 2024-03-06T07:28:44.960 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:44.960 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/103754153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":103754153},{"type":"v1","addr":"172.21.15.69:6836","nonce":103754153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4078702997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":4078702997},{"type":"v1","addr":"172.21.15.69:6837","nonce":4078702997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7956,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3926923781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3926923781},{"type":"v1","addr":"172.21.15.27:6834","nonce":3926923781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:28:43.956535+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7959":{"gid":7959,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1655684069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1655684069},{"type":"v1","addr":"172.21.15.27:6835","nonce":1655684069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7959,"qdb_cluster":[7959]},"id":8}]} 2024-03-06T07:28:44.961 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 75 2024-03-06T07:28:44.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.959+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddec06b4d0 msgr2=0x7fddec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.959+0000 7fde05f93640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddec06b4d0 0x7fddec06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fddf00010e0 tx=0x7fddf0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.959+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde000b9a10 msgr2=0x7fde0014e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.959+0000 7fde05f93640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde000b9a10 0x7fde0014e1e0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fddf400c7b0 tx=0x7fddf400cc70 comp rx=0 tx=0).stop 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.960+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 shutdown_connections 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.960+0000 7fde05f93640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fddec06b4d0 0x7fddec06d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.960+0000 7fde05f93640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde0015df00 0x7fde000cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.960+0000 7fde05f93640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde000b9a10 0x7fde0014e1e0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.960+0000 7fde05f93640 1 --2- 172.21.15.27:0/2573242851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde000b95a0 0x7fde000c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.960+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 >> 172.21.15.27:0/2573242851 conn(0x7fde00071960 msgr2=0x7fde001618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.960+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 shutdown_connections 2024-03-06T07:28:44.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:44.960+0000 7fde05f93640 1 -- 172.21.15.27:0/2573242851 wait complete. 2024-03-06T07:28:44.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.087+0000 7ff9ff3a0640 1 Processor -- start 2024-03-06T07:28:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.087+0000 7ff9ff3a0640 1 -- start start 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9ff3a0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9f81612b0 0x7ff9f80b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9ff3a0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f80cf370 con 0x7ff9f80b5c40 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9ff3a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f80d1040 con 0x7ff9f8161690 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9ff3a0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f80d1320 con 0x7ff9f81612b0 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9fe39e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9f81612b0 0x7ff9f80b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9feb9f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff9f80b5c40 0x7ff9f80d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34804/0 (socket says 172.21.15.27:34804) 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9feb9f640 1 -- 172.21.15.27:0/1967195856 learned_addr learned my addr 172.21.15.27:0/1967195856 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9fe39e640 1 -- 172.21.15.27:0/1967195856 >> v1:172.21.15.69:6789/0 conn(0x7ff9f8161690 legacy=0x7ff9f80b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9fe39e640 1 -- 172.21.15.27:0/1967195856 >> v1:172.21.15.27:6789/0 conn(0x7ff9f80b5c40 legacy=0x7ff9f80d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9fe39e640 1 -- 172.21.15.27:0/1967195856 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f80ced20 con 0x7ff9f81612b0 2024-03-06T07:28:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.088+0000 7ff9fd39c640 1 -- 172.21.15.27:0/1967195856 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986608780 0 0) 0x7ff9f80cf370 con 0x7ff9f80b5c40 2024-03-06T07:28:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.089+0000 7ff9fe39e640 1 --2- 172.21.15.27:0/1967195856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 0x7ff9f80b2990 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40097a0 tx=0x7ff9f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34dbb781b695dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.089+0000 7ff9fd39c640 1 -- 172.21.15.27:0/1967195856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4035020 con 0x7ff9f81612b0 2024-03-06T07:28:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.089+0000 7ff9fd39c640 1 -- 172.21.15.27:0/1967195856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f4031830 con 0x7ff9f81612b0 2024-03-06T07:28:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.089+0000 7ff9fd39c640 1 -- 172.21.15.27:0/1967195856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4031c00 con 0x7ff9f81612b0 2024-03-06T07:28:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.090+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/1967195856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 msgr2=0x7ff9f80b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.090+0000 7ff9ff3a0640 1 --2- 172.21.15.27:0/1967195856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 0x7ff9f80b2990 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40097a0 tx=0x7ff9f402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.090+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/1967195856 shutdown_connections 2024-03-06T07:28:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.090+0000 7ff9ff3a0640 1 --2- 172.21.15.27:0/1967195856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 0x7ff9f80b2990 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.090+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/1967195856 >> 172.21.15.27:0/1967195856 conn(0x7ff9f8067120 msgr2=0x7ff9f8067530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.090+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/1967195856 shutdown_connections 2024-03-06T07:28:45.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.090+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/1967195856 wait complete. 2024-03-06T07:28:45.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.092+0000 7ff9ff3a0640 1 Processor -- start 2024-03-06T07:28:45.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.092+0000 7ff9ff3a0640 1 -- start start 2024-03-06T07:28:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.093+0000 7ff9ff3a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f80b5c40 0x7ff9f8155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.093+0000 7ff9ff3a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 0x7ff9f815f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.093+0000 7ff9ff3a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8161690 0x7ff9f8152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.093+0000 7ff9ff3a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80b3150 con 0x7ff9f81612b0 2024-03-06T07:28:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.093+0000 7ff9ff3a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80b2fd0 con 0x7ff9f8161690 2024-03-06T07:28:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.093+0000 7ff9ff3a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80d3560 con 0x7ff9f80b5c40 2024-03-06T07:28:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.093+0000 7ff9fdb9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 0x7ff9f815f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.093+0000 7ff9fdb9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 0x7ff9f815f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44590/0 (socket says 172.21.15.27:44590) 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.093+0000 7ff9fdb9d640 1 -- 172.21.15.27:0/2276294573 learned_addr learned my addr 172.21.15.27:0/2276294573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9fe39e640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f80b5c40 0x7ff9f8155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9feb9f640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8161690 0x7ff9f8152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9fdb9d640 1 -- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f80b5c40 msgr2=0x7ff9f8155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9fdb9d640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f80b5c40 0x7ff9f8155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9fdb9d640 1 -- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8161690 msgr2=0x7ff9f8152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9fdb9d640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8161690 0x7ff9f8152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9fdb9d640 1 -- 172.21.15.27:0/2276294573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f8154010 con 0x7ff9f81612b0 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9feb9f640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8161690 0x7ff9f8152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9fe39e640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f80b5c40 0x7ff9f8155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9fdb9d640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 0x7ff9f815f5e0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800c3a0 tx=0x7ff9e800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9eeffd640 1 -- 172.21.15.27:0/2276294573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8017070 con 0x7ff9f81612b0 2024-03-06T07:28:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9eeffd640 1 -- 172.21.15.27:0/2276294573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e80052f0 con 0x7ff9f81612b0 2024-03-06T07:28:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f81542d0 con 0x7ff9f81612b0 2024-03-06T07:28:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9eeffd640 1 -- 172.21.15.27:0/2276294573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8003640 con 0x7ff9f81612b0 2024-03-06T07:28:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.094+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f8154890 con 0x7ff9f81612b0 2024-03-06T07:28:45.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.095+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f804f330 con 0x7ff9f81612b0 2024-03-06T07:28:45.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.096+0000 7ff9eeffd640 1 -- 172.21.15.27:0/2276294573 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9e801a070 con 0x7ff9f81612b0 2024-03-06T07:28:45.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.096+0000 7ff9eeffd640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff9dc06b500 0x7ff9dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.096+0000 7ff9eeffd640 1 -- 172.21.15.27:0/2276294573 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9e8089590 con 0x7ff9f81612b0 2024-03-06T07:28:45.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.096+0000 7ff9fe39e640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff9dc06b500 0x7ff9dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.097+0000 7ff9fe39e640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff9dc06b500 0x7ff9dc06d9c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4036000 tx=0x7ff9f4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.100+0000 7ff9eeffd640 1 -- 172.21.15.27:0/2276294573 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9e8058420 con 0x7ff9f81612b0 2024-03-06T07:28:45.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.285+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9f80b6990 con 0x7ff9f81612b0 2024-03-06T07:28:45.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.286+0000 7ff9eeffd640 1 -- 172.21.15.27:0/2276294573 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16555 (secure 0 0 0) 0x7ff9e8057d20 con 0x7ff9f81612b0 2024-03-06T07:28:45.289 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:45.289 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":114,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:28:42.579409+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.260334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.929245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff9dc06b500 msgr2=0x7ff9dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff9dc06b500 0x7ff9dc06d9c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4036000 tx=0x7ff9f4008030 comp rx=0 tx=0).stop 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 msgr2=0x7ff9f815f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 0x7ff9f815f5e0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800c3a0 tx=0x7ff9e800c860 comp rx=0 tx=0).stop 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 shutdown_connections 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff9dc06b500 0x7ff9dc06d9c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9f8161690 0x7ff9f8152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9f81612b0 0x7ff9f815f5e0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 --2- 172.21.15.27:0/2276294573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9f80b5c40 0x7ff9f8155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 >> 172.21.15.27:0/2276294573 conn(0x7ff9f8067120 msgr2=0x7ff9f80b7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 shutdown_connections 2024-03-06T07:28:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.288+0000 7ff9ff3a0640 1 -- 172.21.15.27:0/2276294573 wait complete. 2024-03-06T07:28:45.306 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:28:45.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.412+0000 7f60d07b5640 1 Processor -- start 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.412+0000 7f60d07b5640 1 -- start start 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.413+0000 7f60d07b5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f60c80b2b50 0x7f60c80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.413+0000 7f60d07b5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c80ce610 con 0x7f60c80d3400 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.413+0000 7f60d07b5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c80d2920 con 0x7f60c8071540 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.413+0000 7f60d07b5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c80d2f70 con 0x7f60c80b2b50 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60cdd50640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f60c80d3400 0x7f60c80b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34810/0 (socket says 172.21.15.27:34810) 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60cdd50640 1 -- 172.21.15.27:0/373926901 learned_addr learned my addr 172.21.15.27:0/373926901 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60ced52640 1 --2- 172.21.15.27:0/373926901 >> v2:172.21.15.69:3300/0 conn(0x7f60c80b2b50 0x7f60c80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60cd54f640 1 -- 172.21.15.27:0/373926901 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094058804 0 0) 0x7f60c80ce610 con 0x7f60c80d3400 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60cd54f640 1 -- 172.21.15.27:0/373926901 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b4003520 con 0x7f60c80d3400 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60cd54f640 1 -- 172.21.15.27:0/373926901 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442950768 0 0) 0x7f60c80d2920 con 0x7f60c8071540 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60cd54f640 1 -- 172.21.15.27:0/373926901 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c80ce610 con 0x7f60c8071540 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60ced52640 1 -- 172.21.15.27:0/373926901 >> v1:172.21.15.69:6789/0 conn(0x7f60c8071540 legacy=0x7f60c8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60ced52640 1 -- 172.21.15.27:0/373926901 >> v1:172.21.15.27:6789/0 conn(0x7f60c80d3400 legacy=0x7f60c80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.414+0000 7f60ced52640 1 -- 172.21.15.27:0/373926901 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c80d2c00 con 0x7f60c80b2b50 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.415+0000 7f60ced52640 1 --2- 172.21.15.27:0/373926901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c80b2b50 0x7f60c80b30a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009d90 tx=0x7f60c002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98334e33bfdb9eb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.415+0000 7f60cd54f640 1 -- 172.21.15.27:0/373926901 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0039070 con 0x7f60c80b2b50 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.415+0000 7f60cd54f640 1 -- 172.21.15.27:0/373926901 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c002cd00 con 0x7f60c80b2b50 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.415+0000 7f60cd54f640 1 -- 172.21.15.27:0/373926901 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0034660 con 0x7f60c80b2b50 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.416+0000 7f60d07b5640 1 -- 172.21.15.27:0/373926901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c80b2b50 msgr2=0x7f60c80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.416+0000 7f60d07b5640 1 --2- 172.21.15.27:0/373926901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c80b2b50 0x7f60c80b30a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009d90 tx=0x7f60c002bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.416+0000 7f60d07b5640 1 -- 172.21.15.27:0/373926901 shutdown_connections 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.416+0000 7f60d07b5640 1 --2- 172.21.15.27:0/373926901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c80b2b50 0x7f60c80b30a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.416+0000 7f60d07b5640 1 -- 172.21.15.27:0/373926901 >> 172.21.15.27:0/373926901 conn(0x7f60c80d3ca0 msgr2=0x7f60c80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.416+0000 7f60d07b5640 1 -- 172.21.15.27:0/373926901 shutdown_connections 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.416+0000 7f60d07b5640 1 -- 172.21.15.27:0/373926901 wait complete. 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.417+0000 7f60d07b5640 1 Processor -- start 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.417+0000 7f60d07b5640 1 -- start start 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60d07b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c8071540 0x7f60c80cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60d07b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60c80b2b50 0x7f60c8149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60d07b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c80d3400 0x7f60c814a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60d07b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c815e370 con 0x7f60c80b2b50 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60d07b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c80d4be0 con 0x7f60c80d3400 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60d07b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c81f4c30 con 0x7f60c8071540 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60cdd50640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60c80b2b50 0x7f60c8149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60cdd50640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60c80b2b50 0x7f60c8149c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44596/0 (socket says 172.21.15.27:44596) 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60cdd50640 1 -- 172.21.15.27:0/1614671418 learned_addr learned my addr 172.21.15.27:0/1614671418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60ce551640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c8071540 0x7f60c80cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60ced52640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c80d3400 0x7f60c814a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60cdd50640 1 -- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c8071540 msgr2=0x7f60c80cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60cdd50640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c8071540 0x7f60c80cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60cdd50640 1 -- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c80d3400 msgr2=0x7f60c814a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60cdd50640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c80d3400 0x7f60c814a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60cdd50640 1 -- 172.21.15.27:0/1614671418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c81f3e70 con 0x7f60c80b2b50 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.418+0000 7f60cdd50640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60c80b2b50 0x7f60c8149c90 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f60b800c3e0 tx=0x7f60b800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.419+0000 7f60beffd640 1 -- 172.21.15.27:0/1614671418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b8017070 con 0x7f60c80b2b50 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.419+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c81f4150 con 0x7f60c80b2b50 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.419+0000 7f60beffd640 1 -- 172.21.15.27:0/1614671418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b8005ce0 con 0x7f60c80b2b50 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.419+0000 7f60beffd640 1 -- 172.21.15.27:0/1614671418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b80072d0 con 0x7f60c80b2b50 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.419+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c81f4710 con 0x7f60c80b2b50 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.419+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c8068700 con 0x7f60c80b2b50 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.420+0000 7f60beffd640 1 -- 172.21.15.27:0/1614671418 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60b801a070 con 0x7f60c80b2b50 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.420+0000 7f60beffd640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60ac06b4d0 0x7f60ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.420+0000 7f60beffd640 1 -- 172.21.15.27:0/1614671418 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60b8089570 con 0x7f60c80b2b50 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.421+0000 7f60ce551640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60ac06b4d0 0x7f60ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.421+0000 7f60ce551640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60ac06b4d0 0x7f60ac06d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f60c4007fd0 tx=0x7f60c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.430+0000 7f60beffd640 1 -- 172.21.15.27:0/1614671418 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60b8057d20 con 0x7f60c80b2b50 2024-03-06T07:28:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.612+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f60c80b4700 con 0x7f60ac06b4d0 2024-03-06T07:28:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.618+0000 7f60beffd640 1 -- 172.21.15.27:0/1614671418 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240257 (secure 0 0 0) 0x7f60c80b4700 con 0x7f60ac06b4d0 2024-03-06T07:28:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:28:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60ac06b4d0 msgr2=0x7f60ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60ac06b4d0 0x7f60ac06d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f60c4007fd0 tx=0x7f60c4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60c80b2b50 msgr2=0x7f60c8149c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60c80b2b50 0x7f60c8149c90 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f60b800c3e0 tx=0x7f60b800c8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 shutdown_connections 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f60ac06b4d0 0x7f60ac06d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60c80d3400 0x7f60c814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60c80b2b50 0x7f60c8149c90 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 --2- 172.21.15.27:0/1614671418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60c8071540 0x7f60c80cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 >> 172.21.15.27:0/1614671418 conn(0x7f60c80d3ca0 msgr2=0x7f60c80ceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 shutdown_connections 2024-03-06T07:28:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.621+0000 7f60d07b5640 1 -- 172.21.15.27:0/1614671418 wait complete. 2024-03-06T07:28:45.635 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-03-06T07:28:44.071756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323760,"kb_used_data":11896,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754650960,"statfs":{"total":773094113280,"available":772762583040,"internally_reserved":0,"allocated":12181504,"data_stored":5890564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":319213106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.499232"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577740+0000","last_change":"2024-03-06T07:28:39.577740+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577740+0000","last_undegraded":"2024-03-06T07:28:39.577740+0000","last_fullsized":"2024-03-06T07:28:39.577740+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.581194+0000","last_change":"2024-03-06T07:28:39.581194+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.581194+0000","last_undegraded":"2024-03-06T07:28:39.581194+0000","last_fullsized":"2024-03-06T07:28:39.581194+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578117+0000","last_change":"2024-03-06T07:28:39.578117+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578117+0000","last_undegraded":"2024-03-06T07:28:39.578117+0000","last_fullsized":"2024-03-06T07:28:39.578117+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577683+0000","last_change":"2024-03-06T07:28:39.577683+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577683+0000","last_undegraded":"2024-03-06T07:28:39.577683+0000","last_fullsized":"2024-03-06T07:28:39.577683+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578644+0000","last_change":"2024-03-06T07:28:39.578644+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578644+0000","last_undegraded":"2024-03-06T07:28:39.578644+0000","last_fullsized":"2024-03-06T07:28:39.578644+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.573807+0000","last_change":"2024-03-06T07:28:39.573807+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.573807+0000","last_undegraded":"2024-03-06T07:28:39.573807+0000","last_fullsized":"2024-03-06T07:28:39.573807+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.574565+0000","last_change":"2024-03-06T07:28:39.574565+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.574565+0000","last_undegraded":"2024-03-06T07:28:39.574565+0000","last_fullsized":"2024-03-06T07:28:39.574565+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578890+0000","last_change":"2024-03-06T07:28:39.578890+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578890+0000","last_undegraded":"2024-03-06T07:28:39.578890+0000","last_fullsized":"2024-03-06T07:28:39.578890+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577537+0000","last_change":"2024-03-06T07:28:39.577537+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577537+0000","last_undegraded":"2024-03-06T07:28:39.577537+0000","last_fullsized":"2024-03-06T07:28:39.577537+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.573819+0000","last_change":"2024-03-06T07:28:39.573819+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.573819+0000","last_undegraded":"2024-03-06T07:28:39.573819+0000","last_fullsized":"2024-03-06T07:28:39.573819+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578867+0000","last_change":"2024-03-06T07:28:39.578867+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578867+0000","last_undegraded":"2024-03-06T07:28:39.578867+0000","last_fullsized":"2024-03-06T07:28:39.578867+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576993+0000","last_change":"2024-03-06T07:28:39.576993+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576993+0000","last_undegraded":"2024-03-06T07:28:39.576993+0000","last_fullsized":"2024-03-06T07:28:39.576993+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578310+0000","last_change":"2024-03-06T07:28:39.578310+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578310+0000","last_undegraded":"2024-03-06T07:28:39.578310+0000","last_fullsized":"2024-03-06T07:28:39.578310+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.575936+0000","last_change":"2024-03-06T07:28:39.575936+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.575936+0000","last_undegraded":"2024-03-06T07:28:39.575936+0000","last_fullsized":"2024-03-06T07:28:39.575936+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577496+0000","last_change":"2024-03-06T07:28:39.577496+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577496+0000","last_undegraded":"2024-03-06T07:28:39.577496+0000","last_fullsized":"2024-03-06T07:28:39.577496+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577185+0000","last_change":"2024-03-06T07:28:39.577185+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577185+0000","last_undegraded":"2024-03-06T07:28:39.577185+0000","last_fullsized":"2024-03-06T07:28:39.577185+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577627+0000","last_change":"2024-03-06T07:28:39.577627+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577627+0000","last_undegraded":"2024-03-06T07:28:39.577627+0000","last_fullsized":"2024-03-06T07:28:39.577627+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.575953+0000","last_change":"2024-03-06T07:28:39.575953+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.575953+0000","last_undegraded":"2024-03-06T07:28:39.575953+0000","last_fullsized":"2024-03-06T07:28:39.575953+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577636+0000","last_change":"2024-03-06T07:28:39.577636+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577636+0000","last_undegraded":"2024-03-06T07:28:39.577636+0000","last_fullsized":"2024-03-06T07:28:39.577636+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577881+0000","last_change":"2024-03-06T07:28:39.577881+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577881+0000","last_undegraded":"2024-03-06T07:28:39.577881+0000","last_fullsized":"2024-03-06T07:28:39.577881+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578485+0000","last_change":"2024-03-06T07:28:39.578485+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578485+0000","last_undegraded":"2024-03-06T07:28:39.578485+0000","last_fullsized":"2024-03-06T07:28:39.578485+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576363+0000","last_change":"2024-03-06T07:28:39.576363+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576363+0000","last_undegraded":"2024-03-06T07:28:39.576363+0000","last_fullsized":"2024-03-06T07:28:39.576363+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577000+0000","last_change":"2024-03-06T07:28:39.577000+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577000+0000","last_undegraded":"2024-03-06T07:28:39.577000+0000","last_fullsized":"2024-03-06T07:28:39.577000+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.574127+0000","last_change":"2024-03-06T07:28:39.574127+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.574127+0000","last_undegraded":"2024-03-06T07:28:39.574127+0000","last_fullsized":"2024-03-06T07:28:39.574127+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.575817+0000","last_change":"2024-03-06T07:28:39.575817+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.575817+0000","last_undegraded":"2024-03-06T07:28:39.575817+0000","last_fullsized":"2024-03-06T07:28:39.575817+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576774+0000","last_change":"2024-03-06T07:28:39.576774+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576774+0000","last_undegraded":"2024-03-06T07:28:39.576774+0000","last_fullsized":"2024-03-06T07:28:39.576774+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.573807+0000","last_change":"2024-03-06T07:28:39.573807+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.573807+0000","last_undegraded":"2024-03-06T07:28:39.573807+0000","last_fullsized":"2024-03-06T07:28:39.573807+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577740+0000","last_change":"2024-03-06T07:28:39.577740+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577740+0000","last_undegraded":"2024-03-06T07:28:39.577740+0000","last_fullsized":"2024-03-06T07:28:39.577740+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.574565+0000","last_change":"2024-03-06T07:28:39.574565+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.574565+0000","last_undegraded":"2024-03-06T07:28:39.574565+0000","last_fullsized":"2024-03-06T07:28:39.574565+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576392+0000","last_change":"2024-03-06T07:28:39.576392+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576392+0000","last_undegraded":"2024-03-06T07:28:39.576392+0000","last_fullsized":"2024-03-06T07:28:39.576392+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576435+0000","last_change":"2024-03-06T07:28:39.576435+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576435+0000","last_undegraded":"2024-03-06T07:28:39.576435+0000","last_fullsized":"2024-03-06T07:28:39.576435+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577730+0000","last_change":"2024-03-06T07:28:39.577730+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577730+0000","last_undegraded":"2024-03-06T07:28:39.577730+0000","last_fullsized":"2024-03-06T07:28:39.577730+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":342,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.581590+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:28:39.581590+0000","last_peered":"2024-03-06T07:28:39.581590+0000","last_clean":"2024-03-06T07:28:39.581590+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:28:39.581590+0000","last_undegraded":"2024-03-06T07:28:39.581590+0000","last_fullsized":"2024-03-06T07:28:39.581590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37268,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334572,"statfs":{"total":96636764160,"available":96598601728,"internally_reserved":0,"allocated":1974272,"data_stored":1182005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40836,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331004,"statfs":{"total":96636764160,"available":96594948096,"internally_reserved":0,"allocated":1368064,"data_stored":585820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574924,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40852,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330988,"statfs":{"total":96636764160,"available":96594931712,"internally_reserved":0,"allocated":1380352,"data_stored":591637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40852,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330988,"statfs":{"total":96636764160,"available":96594931712,"internally_reserved":0,"allocated":1368064,"data_stored":585820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40836,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331004,"statfs":{"total":96636764160,"available":96594948096,"internally_reserved":0,"allocated":1368064,"data_stored":585820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40852,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330988,"statfs":{"total":96636764160,"available":96594931712,"internally_reserved":0,"allocated":1380352,"data_stored":591637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40836,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331004,"statfs":{"total":96636764160,"available":96594948096,"internally_reserved":0,"allocated":1368064,"data_stored":585820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41428,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330412,"statfs":{"total":96636764160,"available":96594341888,"internally_reserved":0,"allocated":1974272,"data_stored":1182005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:45.640 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.780+0000 7fa7488c3640 1 Processor -- start 2024-03-06T07:28:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.780+0000 7fa7488c3640 1 -- start start 2024-03-06T07:28:45.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.780+0000 7fa7488c3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa74415df00 0x7fa7440d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa7488c3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa7440b95a0 0x7fa7440b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa7488c3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7440cea30 con 0x7fa7440b9a30 2024-03-06T07:28:45.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa7488c3640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7440d2af0 con 0x7fa74415df00 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa7488c3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7440cf3c0 con 0x7fa7440b95a0 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa74259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa74415df00 0x7fa7440d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa74259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa74415df00 0x7fa7440d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44600/0 (socket says 172.21.15.27:44600) 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa74259c640 1 -- 172.21.15.27:0/2550370969 learned_addr learned my addr 172.21.15.27:0/2550370969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa741d9b640 1 --2- 172.21.15.27:0/2550370969 >> v2:172.21.15.69:3300/0 conn(0x7fa7440b95a0 0x7fa7440b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa74259c640 1 -- 172.21.15.27:0/2550370969 >> v2:172.21.15.69:3300/0 conn(0x7fa7440b95a0 msgr2=0x7fa7440b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa74259c640 1 --2- 172.21.15.27:0/2550370969 >> v2:172.21.15.69:3300/0 conn(0x7fa7440b95a0 0x7fa7440b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa74259c640 1 -- 172.21.15.27:0/2550370969 >> v1:172.21.15.69:6789/0 conn(0x7fa7440b9a30 legacy=0x7fa7440d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.781+0000 7fa74259c640 1 -- 172.21.15.27:0/2550370969 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7440cf0e0 con 0x7fa74415df00 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.782+0000 7fa74259c640 1 --2- 172.21.15.27:0/2550370969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74415df00 0x7fa7440d25b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fa7440d1760 tx=0x7fa73402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9609a0cf4c150fed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.782+0000 7fa74159a640 1 -- 172.21.15.27:0/2550370969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734034070 con 0x7fa74415df00 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.782+0000 7fa74159a640 1 -- 172.21.15.27:0/2550370969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa734030860 con 0x7fa74415df00 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.782+0000 7fa74159a640 1 -- 172.21.15.27:0/2550370969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734030c30 con 0x7fa74415df00 2024-03-06T07:28:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.783+0000 7fa7488c3640 1 -- 172.21.15.27:0/2550370969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74415df00 msgr2=0x7fa7440d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.783+0000 7fa7488c3640 1 --2- 172.21.15.27:0/2550370969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74415df00 0x7fa7440d25b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fa7440d1760 tx=0x7fa73402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.783+0000 7fa7488c3640 1 -- 172.21.15.27:0/2550370969 shutdown_connections 2024-03-06T07:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.783+0000 7fa7488c3640 1 --2- 172.21.15.27:0/2550370969 >> v2:172.21.15.69:3300/0 conn(0x7fa7440b95a0 0x7fa7440b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.783+0000 7fa7488c3640 1 --2- 172.21.15.27:0/2550370969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74415df00 0x7fa7440d25b0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.783+0000 7fa7488c3640 1 -- 172.21.15.27:0/2550370969 >> 172.21.15.27:0/2550370969 conn(0x7fa744071960 msgr2=0x7fa744071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.783+0000 7fa7488c3640 1 -- 172.21.15.27:0/2550370969 shutdown_connections 2024-03-06T07:28:45.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.783+0000 7fa7488c3640 1 -- 172.21.15.27:0/2550370969 wait complete. 2024-03-06T07:28:45.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.784+0000 7fa7488c3640 1 Processor -- start 2024-03-06T07:28:45.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.784+0000 7fa7488c3640 1 -- start start 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa7488c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7440b95a0 0x7fa7440cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa7488c3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa74415df00 0x7fa7440c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa7488c3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7440cb760 0x7fa7440b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa7488c3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440cec70 con 0x7fa7440b95a0 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa7488c3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440ce750 con 0x7fa74415df00 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa7488c3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440cedf0 con 0x7fa7440cb760 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa74259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7440b95a0 0x7fa7440cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa74259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7440b95a0 0x7fa7440cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44614/0 (socket says 172.21.15.27:44614) 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa74259c640 1 -- 172.21.15.27:0/998134507 learned_addr learned my addr 172.21.15.27:0/998134507 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa742d9d640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7440cb760 0x7fa7440b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa741d9b640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa74415df00 0x7fa7440c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa74259c640 1 -- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7440cb760 msgr2=0x7fa7440b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa74259c640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7440cb760 0x7fa7440b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa74259c640 1 -- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa74415df00 msgr2=0x7fa7440c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa74259c640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa74415df00 0x7fa7440c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa74259c640 1 -- 172.21.15.27:0/998134507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7440c9260 con 0x7fa7440b95a0 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.785+0000 7fa74259c640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7440b95a0 0x7fa7440cd6a0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fa734000c00 tx=0x7fa7340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.786+0000 7fa732ffd640 1 -- 172.21.15.27:0/998134507 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734034040 con 0x7fa7440b95a0 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.786+0000 7fa732ffd640 1 -- 172.21.15.27:0/998134507 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa734039d90 con 0x7fa7440b95a0 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.786+0000 7fa732ffd640 1 -- 172.21.15.27:0/998134507 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734043610 con 0x7fa7440b95a0 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.786+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7440ccf30 con 0x7fa7440b95a0 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.786+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7440c7f60 con 0x7fa7440b95a0 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.787+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa744067ff0 con 0x7fa7440b95a0 2024-03-06T07:28:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.787+0000 7fa732ffd640 1 -- 172.21.15.27:0/998134507 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa734037070 con 0x7fa7440b95a0 2024-03-06T07:28:45.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.788+0000 7fa732ffd640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa72406b4d0 0x7fa72406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.788+0000 7fa732ffd640 1 -- 172.21.15.27:0/998134507 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7340ac2f0 con 0x7fa7440b95a0 2024-03-06T07:28:45.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.788+0000 7fa741d9b640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa72406b4d0 0x7fa72406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.788+0000 7fa741d9b640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa72406b4d0 0x7fa72406d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa7441610b0 tx=0x7fa738008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.792+0000 7fa732ffd640 1 -- 172.21.15.27:0/998134507 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7440c9260 con 0x7fa7440b95a0 2024-03-06T07:28:45.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.971+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7440d30e0 con 0x7fa72406b4d0 2024-03-06T07:28:45.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.977+0000 7fa732ffd640 1 -- 172.21.15.27:0/998134507 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240257 (secure 0 0 0) 0x7fa7440d30e0 con 0x7fa72406b4d0 2024-03-06T07:28:45.982 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:45.983 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:28:45.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa72406b4d0 msgr2=0x7fa72406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa72406b4d0 0x7fa72406d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa7441610b0 tx=0x7fa738008040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7440b95a0 msgr2=0x7fa7440cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7440b95a0 0x7fa7440cd6a0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fa734000c00 tx=0x7fa7340046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 shutdown_connections 2024-03-06T07:28:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa72406b4d0 0x7fa72406d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa7440cb760 0x7fa7440b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa74415df00 0x7fa7440c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 --2- 172.21.15.27:0/998134507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7440b95a0 0x7fa7440cd6a0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 >> 172.21.15.27:0/998134507 conn(0x7fa744071960 msgr2=0x7fa74415bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 shutdown_connections 2024-03-06T07:28:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:45.981+0000 7fa7488c3640 1 -- 172.21.15.27:0/998134507 wait complete. 2024-03-06T07:28:45.995 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-03-06T07:28:44.071756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323760,"kb_used_data":11896,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754650960,"statfs":{"total":773094113280,"available":772762583040,"internally_reserved":0,"allocated":12181504,"data_stored":5890564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":319213106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.499232"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577740+0000","last_change":"2024-03-06T07:28:39.577740+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577740+0000","last_undegraded":"2024-03-06T07:28:39.577740+0000","last_fullsized":"2024-03-06T07:28:39.577740+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.581194+0000","last_change":"2024-03-06T07:28:39.581194+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.581194+0000","last_undegraded":"2024-03-06T07:28:39.581194+0000","last_fullsized":"2024-03-06T07:28:39.581194+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578117+0000","last_change":"2024-03-06T07:28:39.578117+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578117+0000","last_undegraded":"2024-03-06T07:28:39.578117+0000","last_fullsized":"2024-03-06T07:28:39.578117+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577683+0000","last_change":"2024-03-06T07:28:39.577683+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577683+0000","last_undegraded":"2024-03-06T07:28:39.577683+0000","last_fullsized":"2024-03-06T07:28:39.577683+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578644+0000","last_change":"2024-03-06T07:28:39.578644+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578644+0000","last_undegraded":"2024-03-06T07:28:39.578644+0000","last_fullsized":"2024-03-06T07:28:39.578644+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.573807+0000","last_change":"2024-03-06T07:28:39.573807+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.573807+0000","last_undegraded":"2024-03-06T07:28:39.573807+0000","last_fullsized":"2024-03-06T07:28:39.573807+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.574565+0000","last_change":"2024-03-06T07:28:39.574565+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.574565+0000","last_undegraded":"2024-03-06T07:28:39.574565+0000","last_fullsized":"2024-03-06T07:28:39.574565+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578890+0000","last_change":"2024-03-06T07:28:39.578890+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578890+0000","last_undegraded":"2024-03-06T07:28:39.578890+0000","last_fullsized":"2024-03-06T07:28:39.578890+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577537+0000","last_change":"2024-03-06T07:28:39.577537+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577537+0000","last_undegraded":"2024-03-06T07:28:39.577537+0000","last_fullsized":"2024-03-06T07:28:39.577537+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.573819+0000","last_change":"2024-03-06T07:28:39.573819+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.573819+0000","last_undegraded":"2024-03-06T07:28:39.573819+0000","last_fullsized":"2024-03-06T07:28:39.573819+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578867+0000","last_change":"2024-03-06T07:28:39.578867+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578867+0000","last_undegraded":"2024-03-06T07:28:39.578867+0000","last_fullsized":"2024-03-06T07:28:39.578867+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576993+0000","last_change":"2024-03-06T07:28:39.576993+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576993+0000","last_undegraded":"2024-03-06T07:28:39.576993+0000","last_fullsized":"2024-03-06T07:28:39.576993+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578310+0000","last_change":"2024-03-06T07:28:39.578310+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578310+0000","last_undegraded":"2024-03-06T07:28:39.578310+0000","last_fullsized":"2024-03-06T07:28:39.578310+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.575936+0000","last_change":"2024-03-06T07:28:39.575936+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.575936+0000","last_undegraded":"2024-03-06T07:28:39.575936+0000","last_fullsized":"2024-03-06T07:28:39.575936+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577496+0000","last_change":"2024-03-06T07:28:39.577496+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577496+0000","last_undegraded":"2024-03-06T07:28:39.577496+0000","last_fullsized":"2024-03-06T07:28:39.577496+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577185+0000","last_change":"2024-03-06T07:28:39.577185+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577185+0000","last_undegraded":"2024-03-06T07:28:39.577185+0000","last_fullsized":"2024-03-06T07:28:39.577185+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577627+0000","last_change":"2024-03-06T07:28:39.577627+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577627+0000","last_undegraded":"2024-03-06T07:28:39.577627+0000","last_fullsized":"2024-03-06T07:28:39.577627+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.575953+0000","last_change":"2024-03-06T07:28:39.575953+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.575953+0000","last_undegraded":"2024-03-06T07:28:39.575953+0000","last_fullsized":"2024-03-06T07:28:39.575953+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577636+0000","last_change":"2024-03-06T07:28:39.577636+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577636+0000","last_undegraded":"2024-03-06T07:28:39.577636+0000","last_fullsized":"2024-03-06T07:28:39.577636+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577881+0000","last_change":"2024-03-06T07:28:39.577881+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577881+0000","last_undegraded":"2024-03-06T07:28:39.577881+0000","last_fullsized":"2024-03-06T07:28:39.577881+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.578485+0000","last_change":"2024-03-06T07:28:39.578485+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.578485+0000","last_undegraded":"2024-03-06T07:28:39.578485+0000","last_fullsized":"2024-03-06T07:28:39.578485+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576363+0000","last_change":"2024-03-06T07:28:39.576363+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576363+0000","last_undegraded":"2024-03-06T07:28:39.576363+0000","last_fullsized":"2024-03-06T07:28:39.576363+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577000+0000","last_change":"2024-03-06T07:28:39.577000+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577000+0000","last_undegraded":"2024-03-06T07:28:39.577000+0000","last_fullsized":"2024-03-06T07:28:39.577000+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.574127+0000","last_change":"2024-03-06T07:28:39.574127+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.574127+0000","last_undegraded":"2024-03-06T07:28:39.574127+0000","last_fullsized":"2024-03-06T07:28:39.574127+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.575817+0000","last_change":"2024-03-06T07:28:39.575817+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.575817+0000","last_undegraded":"2024-03-06T07:28:39.575817+0000","last_fullsized":"2024-03-06T07:28:39.575817+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576774+0000","last_change":"2024-03-06T07:28:39.576774+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576774+0000","last_undegraded":"2024-03-06T07:28:39.576774+0000","last_fullsized":"2024-03-06T07:28:39.576774+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.573807+0000","last_change":"2024-03-06T07:28:39.573807+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.573807+0000","last_undegraded":"2024-03-06T07:28:39.573807+0000","last_fullsized":"2024-03-06T07:28:39.573807+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577740+0000","last_change":"2024-03-06T07:28:39.577740+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577740+0000","last_undegraded":"2024-03-06T07:28:39.577740+0000","last_fullsized":"2024-03-06T07:28:39.577740+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.574565+0000","last_change":"2024-03-06T07:28:39.574565+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.574565+0000","last_undegraded":"2024-03-06T07:28:39.574565+0000","last_fullsized":"2024-03-06T07:28:39.574565+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576392+0000","last_change":"2024-03-06T07:28:39.576392+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576392+0000","last_undegraded":"2024-03-06T07:28:39.576392+0000","last_fullsized":"2024-03-06T07:28:39.576392+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.576435+0000","last_change":"2024-03-06T07:28:39.576435+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.576435+0000","last_undegraded":"2024-03-06T07:28:39.576435+0000","last_fullsized":"2024-03-06T07:28:39.576435+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:40.570920+0000","last_change":"2024-03-06T07:28:40.570920+0000","last_active":"2024-03-06T07:28:40.570920+0000","last_peered":"2024-03-06T07:28:40.570920+0000","last_clean":"2024-03-06T07:28:40.570920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.570920+0000","last_undegraded":"2024-03-06T07:28:40.570920+0000","last_fullsized":"2024-03-06T07:28:40.570920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-03-06T07:28:39.577730+0000","last_change":"2024-03-06T07:28:39.577730+0000","last_active":"2024-03-06T07:28:39.568828+0000","last_peered":"2024-03-06T07:28:39.568828+0000","last_clean":"2024-03-06T07:28:39.568828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:39.577730+0000","last_undegraded":"2024-03-06T07:28:39.577730+0000","last_fullsized":"2024-03-06T07:28:39.577730+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":342,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.581590+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:28:39.581590+0000","last_peered":"2024-03-06T07:28:39.581590+0000","last_clean":"2024-03-06T07:28:39.581590+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:28:39.581590+0000","last_undegraded":"2024-03-06T07:28:39.581590+0000","last_fullsized":"2024-03-06T07:28:39.581590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37268,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334572,"statfs":{"total":96636764160,"available":96598601728,"internally_reserved":0,"allocated":1974272,"data_stored":1182005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40836,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331004,"statfs":{"total":96636764160,"available":96594948096,"internally_reserved":0,"allocated":1368064,"data_stored":585820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574924,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40852,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330988,"statfs":{"total":96636764160,"available":96594931712,"internally_reserved":0,"allocated":1380352,"data_stored":591637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40852,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330988,"statfs":{"total":96636764160,"available":96594931712,"internally_reserved":0,"allocated":1368064,"data_stored":585820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40836,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331004,"statfs":{"total":96636764160,"available":96594948096,"internally_reserved":0,"allocated":1368064,"data_stored":585820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40852,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330988,"statfs":{"total":96636764160,"available":96594931712,"internally_reserved":0,"allocated":1380352,"data_stored":591637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40836,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331004,"statfs":{"total":96636764160,"available":96594948096,"internally_reserved":0,"allocated":1368064,"data_stored":585820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41428,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330412,"statfs":{"total":96636764160,"available":96594341888,"internally_reserved":0,"allocated":1974272,"data_stored":1182005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:46.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:46.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.140+0000 7f8d1f08d640 1 Processor -- start 2024-03-06T07:28:46.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.140+0000 7f8d1f08d640 1 -- start start 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.141+0000 7f8d1f08d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d1815df00 0x7f8d180d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.141+0000 7f8d1f08d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180b95a0 0x7f8d180b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.141+0000 7f8d1f08d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d180cea30 con 0x7f8d180b9a30 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.141+0000 7f8d1f08d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d180d2af0 con 0x7f8d1815df00 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.141+0000 7f8d1f08d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180cf3c0 con 0x7f8d180b95a0 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d17fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180b95a0 0x7f8d180b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d1ce29640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d1815df00 0x7f8d180d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d17fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180b95a0 0x7f8d180b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:48866/0 (socket says 172.21.15.27:48866) 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d17fff640 1 -- 172.21.15.27:0/489084676 learned_addr learned my addr 172.21.15.27:0/489084676 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:46.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d17fff640 1 -- 172.21.15.27:0/489084676 >> v2:172.21.15.69:3300/0 conn(0x7f8d1815df00 msgr2=0x7f8d180d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d17fff640 1 --2- 172.21.15.27:0/489084676 >> v2:172.21.15.69:3300/0 conn(0x7f8d1815df00 0x7f8d180d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d17fff640 1 -- 172.21.15.27:0/489084676 >> v1:172.21.15.69:6789/0 conn(0x7f8d180b9a30 legacy=0x7f8d180d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d177fe640 1 -- 172.21.15.27:0/489084676 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246898027 0 0) 0x7f8d180cea30 con 0x7f8d180b9a30 2024-03-06T07:28:46.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d17fff640 1 -- 172.21.15.27:0/489084676 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d180cf0e0 con 0x7f8d180b95a0 2024-03-06T07:28:46.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.142+0000 7f8d1ce29640 1 --2- 172.21.15.27:0/489084676 >> v2:172.21.15.69:3300/0 conn(0x7f8d1815df00 0x7f8d180d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.143+0000 7f8d17fff640 1 --2- 172.21.15.27:0/489084676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180b95a0 0x7f8d180b5700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009870 tx=0x7f8d0402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77574bc1ab01eebd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.143+0000 7f8d177fe640 1 -- 172.21.15.27:0/489084676 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04034070 con 0x7f8d180b95a0 2024-03-06T07:28:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.143+0000 7f8d177fe640 1 -- 172.21.15.27:0/489084676 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d04030840 con 0x7f8d180b95a0 2024-03-06T07:28:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.143+0000 7f8d177fe640 1 -- 172.21.15.27:0/489084676 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04030c10 con 0x7f8d180b95a0 2024-03-06T07:28:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.144+0000 7f8d1f08d640 1 -- 172.21.15.27:0/489084676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180b95a0 msgr2=0x7f8d180b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.144+0000 7f8d1f08d640 1 --2- 172.21.15.27:0/489084676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180b95a0 0x7f8d180b5700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009870 tx=0x7f8d0402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.144+0000 7f8d1f08d640 1 -- 172.21.15.27:0/489084676 shutdown_connections 2024-03-06T07:28:46.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.144+0000 7f8d1f08d640 1 --2- 172.21.15.27:0/489084676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d180b95a0 0x7f8d180b5700 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.144+0000 7f8d1f08d640 1 --2- 172.21.15.27:0/489084676 >> v2:172.21.15.69:3300/0 conn(0x7f8d1815df00 0x7f8d180d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.144+0000 7f8d1f08d640 1 -- 172.21.15.27:0/489084676 >> 172.21.15.27:0/489084676 conn(0x7f8d18071960 msgr2=0x7f8d18071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.144+0000 7f8d1f08d640 1 -- 172.21.15.27:0/489084676 shutdown_connections 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.144+0000 7f8d1f08d640 1 -- 172.21.15.27:0/489084676 wait complete. 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.145+0000 7f8d1f08d640 1 Processor -- start 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.145+0000 7f8d1f08d640 1 -- start start 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1f08d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d180b95a0 0x7f8d18155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1f08d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d1815df00 0x7f8d180cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1f08d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18160dc0 0x7f8d180b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1f08d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180cec70 con 0x7f8d180b95a0 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1f08d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180ce750 con 0x7f8d18160dc0 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1f08d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180cedf0 con 0x7f8d1815df00 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1ce29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d180b95a0 0x7f8d18155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1ce29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d180b95a0 0x7f8d18155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44618/0 (socket says 172.21.15.27:44618) 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1ce29640 1 -- 172.21.15.27:0/1456901850 learned_addr learned my addr 172.21.15.27:0/1456901850 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d17fff640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d1815df00 0x7f8d180cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1d62a640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18160dc0 0x7f8d180b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1ce29640 1 -- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d1815df00 msgr2=0x7f8d180cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1ce29640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d1815df00 0x7f8d180cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1ce29640 1 -- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18160dc0 msgr2=0x7f8d180b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1ce29640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18160dc0 0x7f8d180b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d1ce29640 1 -- 172.21.15.27:0/1456901850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d180cb6c0 con 0x7f8d180b95a0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.146+0000 7f8d17fff640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d1815df00 0x7f8d180cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.147+0000 7f8d1ce29640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d180b95a0 0x7f8d18155330 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00e960 tx=0x7f8d0c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.147+0000 7f8d157fa640 1 -- 172.21.15.27:0/1456901850 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c017070 con 0x7f8d180b95a0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.147+0000 7f8d157fa640 1 -- 172.21.15.27:0/1456901850 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0c00ca00 con 0x7f8d180b95a0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.147+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d180cb9a0 con 0x7f8d180b95a0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.147+0000 7f8d157fa640 1 -- 172.21.15.27:0/1456901850 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c00cdd0 con 0x7f8d180b95a0 2024-03-06T07:28:46.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.147+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d180cbf60 con 0x7f8d180b95a0 2024-03-06T07:28:46.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.148+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d18067ff0 con 0x7f8d180b95a0 2024-03-06T07:28:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.148+0000 7f8d157fa640 1 -- 172.21.15.27:0/1456901850 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d0c01a070 con 0x7f8d180b95a0 2024-03-06T07:28:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.149+0000 7f8d157fa640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cfc06b4d0 0x7f8cfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.149+0000 7f8d157fa640 1 -- 172.21.15.27:0/1456901850 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d0c08a320 con 0x7f8d180b95a0 2024-03-06T07:28:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.149+0000 7f8d17fff640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cfc06b4d0 0x7f8cfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.150+0000 7f8d17fff640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cfc06b4d0 0x7f8cfc06d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8d04036000 tx=0x7f8d04008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.153+0000 7f8d157fa640 1 -- 172.21.15.27:0/1456901850 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d0c058be0 con 0x7f8d180b95a0 2024-03-06T07:28:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.330+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d180b30a0 con 0x7f8cfc06b4d0 2024-03-06T07:28:46.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.337+0000 7f8d157fa640 1 -- 172.21.15.27:0/1456901850 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252482 (secure 0 0 0) 0x7f8d180b30a0 con 0x7f8cfc06b4d0 2024-03-06T07:28:46.342 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:46.343 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:28:46.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cfc06b4d0 msgr2=0x7f8cfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cfc06b4d0 0x7f8cfc06d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8d04036000 tx=0x7f8d04008030 comp rx=0 tx=0).stop 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d180b95a0 msgr2=0x7f8d18155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d180b95a0 0x7f8d18155330 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00e960 tx=0x7f8d0c00ee20 comp rx=0 tx=0).stop 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 shutdown_connections 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8cfc06b4d0 0x7f8cfc06d990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18160dc0 0x7f8d180b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d1815df00 0x7f8d180cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 --2- 172.21.15.27:0/1456901850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d180b95a0 0x7f8d18155330 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 >> 172.21.15.27:0/1456901850 conn(0x7f8d18071960 msgr2=0x7f8d1815d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 shutdown_connections 2024-03-06T07:28:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.341+0000 7f8d1f08d640 1 -- 172.21.15.27:0/1456901850 wait complete. 2024-03-06T07:28:46.354 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-03-06T07:28:46.072316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324188,"kb_used_data":12296,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754650532,"statfs":{"total":773094113280,"available":772762144768,"internally_reserved":0,"allocated":12591104,"data_stored":6073452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":319213106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001546"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615443+0000","last_change":"2024-03-06T07:28:41.605491+0000","last_active":"2024-03-06T07:28:42.615443+0000","last_peered":"2024-03-06T07:28:42.615443+0000","last_clean":"2024-03-06T07:28:42.615443+0000","last_became_active":"2024-03-06T07:28:41.605207+0000","last_became_peered":"2024-03-06T07:28:41.605207+0000","last_unstale":"2024-03-06T07:28:42.615443+0000","last_undegraded":"2024-03-06T07:28:42.615443+0000","last_fullsized":"2024-03-06T07:28:42.615443+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:41.754749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584798+0000","last_change":"2024-03-06T07:28:41.598081+0000","last_active":"2024-03-06T07:28:42.584798+0000","last_peered":"2024-03-06T07:28:42.584798+0000","last_clean":"2024-03-06T07:28:42.584798+0000","last_became_active":"2024-03-06T07:28:41.597746+0000","last_became_peered":"2024-03-06T07:28:41.597746+0000","last_unstale":"2024-03-06T07:28:42.584798+0000","last_undegraded":"2024-03-06T07:28:42.584798+0000","last_fullsized":"2024-03-06T07:28:42.584798+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:36.645183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616191+0000","last_change":"2024-03-06T07:28:41.604865+0000","last_active":"2024-03-06T07:28:42.616191+0000","last_peered":"2024-03-06T07:28:42.616191+0000","last_clean":"2024-03-06T07:28:42.616191+0000","last_became_active":"2024-03-06T07:28:41.603741+0000","last_became_peered":"2024-03-06T07:28:41.603741+0000","last_unstale":"2024-03-06T07:28:42.616191+0000","last_undegraded":"2024-03-06T07:28:42.616191+0000","last_fullsized":"2024-03-06T07:28:42.616191+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.995900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615181+0000","last_change":"2024-03-06T07:28:41.606885+0000","last_active":"2024-03-06T07:28:42.615181+0000","last_peered":"2024-03-06T07:28:42.615181+0000","last_clean":"2024-03-06T07:28:42.615181+0000","last_became_active":"2024-03-06T07:28:41.606582+0000","last_became_peered":"2024-03-06T07:28:41.606582+0000","last_unstale":"2024-03-06T07:28:42.615181+0000","last_undegraded":"2024-03-06T07:28:42.615181+0000","last_fullsized":"2024-03-06T07:28:42.615181+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:40.695072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585291+0000","last_change":"2024-03-06T07:28:41.611053+0000","last_active":"2024-03-06T07:28:42.585291+0000","last_peered":"2024-03-06T07:28:42.585291+0000","last_clean":"2024-03-06T07:28:42.585291+0000","last_became_active":"2024-03-06T07:28:41.610636+0000","last_became_peered":"2024-03-06T07:28:41.610636+0000","last_unstale":"2024-03-06T07:28:42.585291+0000","last_undegraded":"2024-03-06T07:28:42.585291+0000","last_fullsized":"2024-03-06T07:28:42.585291+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:54.518946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614777+0000","last_change":"2024-03-06T07:28:41.588032+0000","last_active":"2024-03-06T07:28:42.614777+0000","last_peered":"2024-03-06T07:28:42.614777+0000","last_clean":"2024-03-06T07:28:42.614777+0000","last_became_active":"2024-03-06T07:28:41.587772+0000","last_became_peered":"2024-03-06T07:28:41.587772+0000","last_unstale":"2024-03-06T07:28:42.614777+0000","last_undegraded":"2024-03-06T07:28:42.614777+0000","last_fullsized":"2024-03-06T07:28:42.614777+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:34.817395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615081+0000","last_change":"2024-03-06T07:28:41.597654+0000","last_active":"2024-03-06T07:28:42.615081+0000","last_peered":"2024-03-06T07:28:42.615081+0000","last_clean":"2024-03-06T07:28:42.615081+0000","last_became_active":"2024-03-06T07:28:41.597375+0000","last_became_peered":"2024-03-06T07:28:41.597375+0000","last_unstale":"2024-03-06T07:28:42.615081+0000","last_undegraded":"2024-03-06T07:28:42.615081+0000","last_fullsized":"2024-03-06T07:28:42.615081+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:55.516936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615462+0000","last_change":"2024-03-06T07:28:41.598969+0000","last_active":"2024-03-06T07:28:42.615462+0000","last_peered":"2024-03-06T07:28:42.615462+0000","last_clean":"2024-03-06T07:28:42.615462+0000","last_became_active":"2024-03-06T07:28:41.598680+0000","last_became_peered":"2024-03-06T07:28:41.598680+0000","last_unstale":"2024-03-06T07:28:42.615462+0000","last_undegraded":"2024-03-06T07:28:42.615462+0000","last_fullsized":"2024-03-06T07:28:42.615462+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:07.931449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615598+0000","last_change":"2024-03-06T07:28:41.607892+0000","last_active":"2024-03-06T07:28:42.615598+0000","last_peered":"2024-03-06T07:28:42.615598+0000","last_clean":"2024-03-06T07:28:42.615598+0000","last_became_active":"2024-03-06T07:28:41.607625+0000","last_became_peered":"2024-03-06T07:28:41.607625+0000","last_unstale":"2024-03-06T07:28:42.615598+0000","last_undegraded":"2024-03-06T07:28:42.615598+0000","last_fullsized":"2024-03-06T07:28:42.615598+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:15.356128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615335+0000","last_change":"2024-03-06T07:28:41.613674+0000","last_active":"2024-03-06T07:28:42.615335+0000","last_peered":"2024-03-06T07:28:42.615335+0000","last_clean":"2024-03-06T07:28:42.615335+0000","last_became_active":"2024-03-06T07:28:41.613392+0000","last_became_peered":"2024-03-06T07:28:41.613392+0000","last_unstale":"2024-03-06T07:28:42.615335+0000","last_undegraded":"2024-03-06T07:28:42.615335+0000","last_fullsized":"2024-03-06T07:28:42.615335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:56.109795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585034+0000","last_change":"2024-03-06T07:28:41.599288+0000","last_active":"2024-03-06T07:28:42.585034+0000","last_peered":"2024-03-06T07:28:42.585034+0000","last_clean":"2024-03-06T07:28:42.585034+0000","last_became_active":"2024-03-06T07:28:41.599023+0000","last_became_peered":"2024-03-06T07:28:41.599023+0000","last_unstale":"2024-03-06T07:28:42.585034+0000","last_undegraded":"2024-03-06T07:28:42.585034+0000","last_fullsized":"2024-03-06T07:28:42.585034+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:12.604285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615406+0000","last_change":"2024-03-06T07:28:41.603785+0000","last_active":"2024-03-06T07:28:42.615406+0000","last_peered":"2024-03-06T07:28:42.615406+0000","last_clean":"2024-03-06T07:28:42.615406+0000","last_became_active":"2024-03-06T07:28:41.603478+0000","last_became_peered":"2024-03-06T07:28:41.603478+0000","last_unstale":"2024-03-06T07:28:42.615406+0000","last_undegraded":"2024-03-06T07:28:42.615406+0000","last_fullsized":"2024-03-06T07:28:42.615406+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:18.198067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616026+0000","last_change":"2024-03-06T07:28:41.612243+0000","last_active":"2024-03-06T07:28:42.616026+0000","last_peered":"2024-03-06T07:28:42.616026+0000","last_clean":"2024-03-06T07:28:42.616026+0000","last_became_active":"2024-03-06T07:28:41.611872+0000","last_became_peered":"2024-03-06T07:28:41.611872+0000","last_unstale":"2024-03-06T07:28:42.616026+0000","last_undegraded":"2024-03-06T07:28:42.616026+0000","last_fullsized":"2024-03-06T07:28:42.616026+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:55.837869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615140+0000","last_change":"2024-03-06T07:28:41.597885+0000","last_active":"2024-03-06T07:28:42.615140+0000","last_peered":"2024-03-06T07:28:42.615140+0000","last_clean":"2024-03-06T07:28:42.615140+0000","last_became_active":"2024-03-06T07:28:41.597613+0000","last_became_peered":"2024-03-06T07:28:41.597613+0000","last_unstale":"2024-03-06T07:28:42.615140+0000","last_undegraded":"2024-03-06T07:28:42.615140+0000","last_fullsized":"2024-03-06T07:28:42.615140+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:50.061502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614475+0000","last_change":"2024-03-06T07:28:41.587309+0000","last_active":"2024-03-06T07:28:42.614475+0000","last_peered":"2024-03-06T07:28:42.614475+0000","last_clean":"2024-03-06T07:28:42.614475+0000","last_became_active":"2024-03-06T07:28:41.587034+0000","last_became_peered":"2024-03-06T07:28:41.587034+0000","last_unstale":"2024-03-06T07:28:42.614475+0000","last_undegraded":"2024-03-06T07:28:42.614475+0000","last_fullsized":"2024-03-06T07:28:42.614475+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:02.582015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614882+0000","last_change":"2024-03-06T07:28:41.610442+0000","last_active":"2024-03-06T07:28:42.614882+0000","last_peered":"2024-03-06T07:28:42.614882+0000","last_clean":"2024-03-06T07:28:42.614882+0000","last_became_active":"2024-03-06T07:28:41.605967+0000","last_became_peered":"2024-03-06T07:28:41.605967+0000","last_unstale":"2024-03-06T07:28:42.614882+0000","last_undegraded":"2024-03-06T07:28:42.614882+0000","last_fullsized":"2024-03-06T07:28:42.614882+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:50.576369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584911+0000","last_change":"2024-03-06T07:28:41.600051+0000","last_active":"2024-03-06T07:28:42.584911+0000","last_peered":"2024-03-06T07:28:42.584911+0000","last_clean":"2024-03-06T07:28:42.584911+0000","last_became_active":"2024-03-06T07:28:41.599530+0000","last_became_peered":"2024-03-06T07:28:41.599530+0000","last_unstale":"2024-03-06T07:28:42.584911+0000","last_undegraded":"2024-03-06T07:28:42.584911+0000","last_fullsized":"2024-03-06T07:28:42.584911+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:19.178731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615853+0000","last_change":"2024-03-06T07:28:41.602134+0000","last_active":"2024-03-06T07:28:42.615853+0000","last_peered":"2024-03-06T07:28:42.615853+0000","last_clean":"2024-03-06T07:28:42.615853+0000","last_became_active":"2024-03-06T07:28:41.601846+0000","last_became_peered":"2024-03-06T07:28:41.601846+0000","last_unstale":"2024-03-06T07:28:42.615853+0000","last_undegraded":"2024-03-06T07:28:42.615853+0000","last_fullsized":"2024-03-06T07:28:42.615853+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:13.831484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616246+0000","last_change":"2024-03-06T07:28:41.612404+0000","last_active":"2024-03-06T07:28:42.616246+0000","last_peered":"2024-03-06T07:28:42.616246+0000","last_clean":"2024-03-06T07:28:42.616246+0000","last_became_active":"2024-03-06T07:28:41.612121+0000","last_became_peered":"2024-03-06T07:28:41.612121+0000","last_unstale":"2024-03-06T07:28:42.616246+0000","last_undegraded":"2024-03-06T07:28:42.616246+0000","last_fullsized":"2024-03-06T07:28:42.616246+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:05.095081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584952+0000","last_change":"2024-03-06T07:28:41.593131+0000","last_active":"2024-03-06T07:28:42.584952+0000","last_peered":"2024-03-06T07:28:42.584952+0000","last_clean":"2024-03-06T07:28:42.584952+0000","last_became_active":"2024-03-06T07:28:41.592855+0000","last_became_peered":"2024-03-06T07:28:41.592855+0000","last_unstale":"2024-03-06T07:28:42.584952+0000","last_undegraded":"2024-03-06T07:28:42.584952+0000","last_fullsized":"2024-03-06T07:28:42.584952+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:40.354302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585104+0000","last_change":"2024-03-06T07:28:41.594310+0000","last_active":"2024-03-06T07:28:42.585104+0000","last_peered":"2024-03-06T07:28:42.585104+0000","last_clean":"2024-03-06T07:28:42.585104+0000","last_became_active":"2024-03-06T07:28:41.593649+0000","last_became_peered":"2024-03-06T07:28:41.593649+0000","last_unstale":"2024-03-06T07:28:42.585104+0000","last_undegraded":"2024-03-06T07:28:42.585104+0000","last_fullsized":"2024-03-06T07:28:42.585104+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:34.600777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584533+0000","last_change":"2024-03-06T07:28:41.599908+0000","last_active":"2024-03-06T07:28:42.584533+0000","last_peered":"2024-03-06T07:28:42.584533+0000","last_clean":"2024-03-06T07:28:42.584533+0000","last_became_active":"2024-03-06T07:28:41.599609+0000","last_became_peered":"2024-03-06T07:28:41.599609+0000","last_unstale":"2024-03-06T07:28:42.584533+0000","last_undegraded":"2024-03-06T07:28:42.584533+0000","last_fullsized":"2024-03-06T07:28:42.584533+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:56.356422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584612+0000","last_change":"2024-03-06T07:28:41.592523+0000","last_active":"2024-03-06T07:28:42.584612+0000","last_peered":"2024-03-06T07:28:42.584612+0000","last_clean":"2024-03-06T07:28:42.584612+0000","last_became_active":"2024-03-06T07:28:41.592223+0000","last_became_peered":"2024-03-06T07:28:41.592223+0000","last_unstale":"2024-03-06T07:28:42.584612+0000","last_undegraded":"2024-03-06T07:28:42.584612+0000","last_fullsized":"2024-03-06T07:28:42.584612+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:21.109449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584132+0000","last_change":"2024-03-06T07:28:41.593471+0000","last_active":"2024-03-06T07:28:42.584132+0000","last_peered":"2024-03-06T07:28:42.584132+0000","last_clean":"2024-03-06T07:28:42.584132+0000","last_became_active":"2024-03-06T07:28:41.593153+0000","last_became_peered":"2024-03-06T07:28:41.593153+0000","last_unstale":"2024-03-06T07:28:42.584132+0000","last_undegraded":"2024-03-06T07:28:42.584132+0000","last_fullsized":"2024-03-06T07:28:42.584132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:16.974584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585034+0000","last_change":"2024-03-06T07:28:41.599835+0000","last_active":"2024-03-06T07:28:42.585034+0000","last_peered":"2024-03-06T07:28:42.585034+0000","last_clean":"2024-03-06T07:28:42.585034+0000","last_became_active":"2024-03-06T07:28:41.599196+0000","last_became_peered":"2024-03-06T07:28:41.599196+0000","last_unstale":"2024-03-06T07:28:42.585034+0000","last_undegraded":"2024-03-06T07:28:42.585034+0000","last_fullsized":"2024-03-06T07:28:42.585034+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:42.380673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615615+0000","last_change":"2024-03-06T07:28:41.604648+0000","last_active":"2024-03-06T07:28:42.615615+0000","last_peered":"2024-03-06T07:28:42.615615+0000","last_clean":"2024-03-06T07:28:42.615615+0000","last_became_active":"2024-03-06T07:28:41.604346+0000","last_became_peered":"2024-03-06T07:28:41.604346+0000","last_unstale":"2024-03-06T07:28:42.615615+0000","last_undegraded":"2024-03-06T07:28:42.615615+0000","last_fullsized":"2024-03-06T07:28:42.615615+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:27.688649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614784+0000","last_change":"2024-03-06T07:28:41.604347+0000","last_active":"2024-03-06T07:28:42.614784+0000","last_peered":"2024-03-06T07:28:42.614784+0000","last_clean":"2024-03-06T07:28:42.614784+0000","last_became_active":"2024-03-06T07:28:41.604078+0000","last_became_peered":"2024-03-06T07:28:41.604078+0000","last_unstale":"2024-03-06T07:28:42.614784+0000","last_undegraded":"2024-03-06T07:28:42.614784+0000","last_fullsized":"2024-03-06T07:28:42.614784+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:12.021731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584403+0000","last_change":"2024-03-06T07:28:41.610412+0000","last_active":"2024-03-06T07:28:42.584403+0000","last_peered":"2024-03-06T07:28:42.584403+0000","last_clean":"2024-03-06T07:28:42.584403+0000","last_became_active":"2024-03-06T07:28:41.607523+0000","last_became_peered":"2024-03-06T07:28:41.607523+0000","last_unstale":"2024-03-06T07:28:42.584403+0000","last_undegraded":"2024-03-06T07:28:42.584403+0000","last_fullsized":"2024-03-06T07:28:42.584403+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:19.391795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584714+0000","last_change":"2024-03-06T07:28:41.592755+0000","last_active":"2024-03-06T07:28:42.584714+0000","last_peered":"2024-03-06T07:28:42.584714+0000","last_clean":"2024-03-06T07:28:42.584714+0000","last_became_active":"2024-03-06T07:28:41.592455+0000","last_became_peered":"2024-03-06T07:28:41.592455+0000","last_unstale":"2024-03-06T07:28:42.584714+0000","last_undegraded":"2024-03-06T07:28:42.584714+0000","last_fullsized":"2024-03-06T07:28:42.584714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:37.201552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615460+0000","last_change":"2024-03-06T07:28:41.602207+0000","last_active":"2024-03-06T07:28:42.615460+0000","last_peered":"2024-03-06T07:28:42.615460+0000","last_clean":"2024-03-06T07:28:42.615460+0000","last_became_active":"2024-03-06T07:28:41.601914+0000","last_became_peered":"2024-03-06T07:28:41.601914+0000","last_unstale":"2024-03-06T07:28:42.615460+0000","last_undegraded":"2024-03-06T07:28:42.615460+0000","last_fullsized":"2024-03-06T07:28:42.615460+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:40.520524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614940+0000","last_change":"2024-03-06T07:28:41.605636+0000","last_active":"2024-03-06T07:28:42.614940+0000","last_peered":"2024-03-06T07:28:42.614940+0000","last_clean":"2024-03-06T07:28:42.614940+0000","last_became_active":"2024-03-06T07:28:41.605101+0000","last_became_peered":"2024-03-06T07:28:41.605101+0000","last_unstale":"2024-03-06T07:28:42.614940+0000","last_undegraded":"2024-03-06T07:28:42.614940+0000","last_fullsized":"2024-03-06T07:28:42.614940+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:51.706297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614469+0000","last_change":"2024-03-06T07:28:41.592526+0000","last_active":"2024-03-06T07:28:42.614469+0000","last_peered":"2024-03-06T07:28:42.614469+0000","last_clean":"2024-03-06T07:28:42.614469+0000","last_became_active":"2024-03-06T07:28:41.592212+0000","last_became_peered":"2024-03-06T07:28:41.592212+0000","last_unstale":"2024-03-06T07:28:42.614469+0000","last_undegraded":"2024-03-06T07:28:42.614469+0000","last_fullsized":"2024-03-06T07:28:42.614469+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:22.548099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614673+0000","last_change":"2024-03-06T07:28:41.612948+0000","last_active":"2024-03-06T07:28:42.614673+0000","last_peered":"2024-03-06T07:28:42.614673+0000","last_clean":"2024-03-06T07:28:42.614673+0000","last_became_active":"2024-03-06T07:28:41.612634+0000","last_became_peered":"2024-03-06T07:28:41.612634+0000","last_unstale":"2024-03-06T07:28:42.614673+0000","last_undegraded":"2024-03-06T07:28:42.614673+0000","last_fullsized":"2024-03-06T07:28:42.614673+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:03.772883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615429+0000","last_change":"2024-03-06T07:28:40.591430+0000","last_active":"2024-03-06T07:28:42.615429+0000","last_peered":"2024-03-06T07:28:42.615429+0000","last_clean":"2024-03-06T07:28:42.615429+0000","last_became_active":"2024-03-06T07:28:40.590954+0000","last_became_peered":"2024-03-06T07:28:40.590954+0000","last_unstale":"2024-03-06T07:28:42.615429+0000","last_undegraded":"2024-03-06T07:28:42.615429+0000","last_fullsized":"2024-03-06T07:28:42.615429+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:41.774205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615738+0000","last_change":"2024-03-06T07:28:41.601165+0000","last_active":"2024-03-06T07:28:42.615738+0000","last_peered":"2024-03-06T07:28:42.615738+0000","last_clean":"2024-03-06T07:28:42.615738+0000","last_became_active":"2024-03-06T07:28:41.600834+0000","last_became_peered":"2024-03-06T07:28:41.600834+0000","last_unstale":"2024-03-06T07:28:42.615738+0000","last_undegraded":"2024-03-06T07:28:42.615738+0000","last_fullsized":"2024-03-06T07:28:42.615738+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:34.600777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616100+0000","last_change":"2024-03-06T07:28:40.592943+0000","last_active":"2024-03-06T07:28:42.616100+0000","last_peered":"2024-03-06T07:28:42.616100+0000","last_clean":"2024-03-06T07:28:42.616100+0000","last_became_active":"2024-03-06T07:28:40.592617+0000","last_became_peered":"2024-03-06T07:28:40.592617+0000","last_unstale":"2024-03-06T07:28:42.616100+0000","last_undegraded":"2024-03-06T07:28:42.616100+0000","last_fullsized":"2024-03-06T07:28:42.616100+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:35.643091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614680+0000","last_change":"2024-03-06T07:28:41.591740+0000","last_active":"2024-03-06T07:28:42.614680+0000","last_peered":"2024-03-06T07:28:42.614680+0000","last_clean":"2024-03-06T07:28:42.614680+0000","last_became_active":"2024-03-06T07:28:41.591478+0000","last_became_peered":"2024-03-06T07:28:41.591478+0000","last_unstale":"2024-03-06T07:28:42.614680+0000","last_undegraded":"2024-03-06T07:28:42.614680+0000","last_fullsized":"2024-03-06T07:28:42.614680+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:54.518946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585315+0000","last_change":"2024-03-06T07:28:40.601530+0000","last_active":"2024-03-06T07:28:42.585315+0000","last_peered":"2024-03-06T07:28:42.585315+0000","last_clean":"2024-03-06T07:28:42.585315+0000","last_became_active":"2024-03-06T07:28:40.601235+0000","last_became_peered":"2024-03-06T07:28:40.601235+0000","last_unstale":"2024-03-06T07:28:42.585315+0000","last_undegraded":"2024-03-06T07:28:42.585315+0000","last_fullsized":"2024-03-06T07:28:42.585315+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:10.155882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616345+0000","last_change":"2024-03-06T07:28:41.612533+0000","last_active":"2024-03-06T07:28:42.616345+0000","last_peered":"2024-03-06T07:28:42.616345+0000","last_clean":"2024-03-06T07:28:42.616345+0000","last_became_active":"2024-03-06T07:28:41.612195+0000","last_became_peered":"2024-03-06T07:28:41.612195+0000","last_unstale":"2024-03-06T07:28:42.616345+0000","last_undegraded":"2024-03-06T07:28:42.616345+0000","last_fullsized":"2024-03-06T07:28:42.616345+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:19.391795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585158+0000","last_change":"2024-03-06T07:28:40.601199+0000","last_active":"2024-03-06T07:28:42.585158+0000","last_peered":"2024-03-06T07:28:42.585158+0000","last_clean":"2024-03-06T07:28:42.585158+0000","last_became_active":"2024-03-06T07:28:40.600924+0000","last_became_peered":"2024-03-06T07:28:40.600924+0000","last_unstale":"2024-03-06T07:28:42.585158+0000","last_undegraded":"2024-03-06T07:28:42.585158+0000","last_fullsized":"2024-03-06T07:28:42.585158+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:51.604636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615023+0000","last_change":"2024-03-06T07:28:41.613286+0000","last_active":"2024-03-06T07:28:42.615023+0000","last_peered":"2024-03-06T07:28:42.615023+0000","last_clean":"2024-03-06T07:28:42.615023+0000","last_became_active":"2024-03-06T07:28:41.612678+0000","last_became_peered":"2024-03-06T07:28:41.612678+0000","last_unstale":"2024-03-06T07:28:42.615023+0000","last_undegraded":"2024-03-06T07:28:42.615023+0000","last_fullsized":"2024-03-06T07:28:42.615023+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:08.661708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615029+0000","last_change":"2024-03-06T07:28:40.596714+0000","last_active":"2024-03-06T07:28:42.615029+0000","last_peered":"2024-03-06T07:28:42.615029+0000","last_clean":"2024-03-06T07:28:42.615029+0000","last_became_active":"2024-03-06T07:28:40.596484+0000","last_became_peered":"2024-03-06T07:28:40.596484+0000","last_unstale":"2024-03-06T07:28:42.615029+0000","last_undegraded":"2024-03-06T07:28:42.615029+0000","last_fullsized":"2024-03-06T07:28:42.615029+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:33.815303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584885+0000","last_change":"2024-03-06T07:28:41.598081+0000","last_active":"2024-03-06T07:28:42.584885+0000","last_peered":"2024-03-06T07:28:42.584885+0000","last_clean":"2024-03-06T07:28:42.584885+0000","last_became_active":"2024-03-06T07:28:41.597755+0000","last_became_peered":"2024-03-06T07:28:41.597755+0000","last_unstale":"2024-03-06T07:28:42.584885+0000","last_undegraded":"2024-03-06T07:28:42.584885+0000","last_fullsized":"2024-03-06T07:28:42.584885+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:17.415589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584596+0000","last_change":"2024-03-06T07:28:40.593599+0000","last_active":"2024-03-06T07:28:42.584596+0000","last_peered":"2024-03-06T07:28:42.584596+0000","last_clean":"2024-03-06T07:28:42.584596+0000","last_became_active":"2024-03-06T07:28:40.593318+0000","last_became_peered":"2024-03-06T07:28:40.593318+0000","last_unstale":"2024-03-06T07:28:42.584596+0000","last_undegraded":"2024-03-06T07:28:42.584596+0000","last_fullsized":"2024-03-06T07:28:42.584596+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:18.176638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615176+0000","last_change":"2024-03-06T07:28:41.611042+0000","last_active":"2024-03-06T07:28:42.615176+0000","last_peered":"2024-03-06T07:28:42.615176+0000","last_clean":"2024-03-06T07:28:42.615176+0000","last_became_active":"2024-03-06T07:28:41.610758+0000","last_became_peered":"2024-03-06T07:28:41.610758+0000","last_unstale":"2024-03-06T07:28:42.615176+0000","last_undegraded":"2024-03-06T07:28:42.615176+0000","last_fullsized":"2024-03-06T07:28:42.615176+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:22.849251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584484+0000","last_change":"2024-03-06T07:28:40.587951+0000","last_active":"2024-03-06T07:28:42.584484+0000","last_peered":"2024-03-06T07:28:42.584484+0000","last_clean":"2024-03-06T07:28:42.584484+0000","last_became_active":"2024-03-06T07:28:40.587671+0000","last_became_peered":"2024-03-06T07:28:40.587671+0000","last_unstale":"2024-03-06T07:28:42.584484+0000","last_undegraded":"2024-03-06T07:28:42.584484+0000","last_fullsized":"2024-03-06T07:28:42.584484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:06.481910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615021+0000","last_change":"2024-03-06T07:28:41.602157+0000","last_active":"2024-03-06T07:28:42.615021+0000","last_peered":"2024-03-06T07:28:42.615021+0000","last_clean":"2024-03-06T07:28:42.615021+0000","last_became_active":"2024-03-06T07:28:41.601862+0000","last_became_peered":"2024-03-06T07:28:41.601862+0000","last_unstale":"2024-03-06T07:28:42.615021+0000","last_undegraded":"2024-03-06T07:28:42.615021+0000","last_fullsized":"2024-03-06T07:28:42.615021+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:28.477340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615227+0000","last_change":"2024-03-06T07:28:40.590486+0000","last_active":"2024-03-06T07:28:42.615227+0000","last_peered":"2024-03-06T07:28:42.615227+0000","last_clean":"2024-03-06T07:28:42.615227+0000","last_became_active":"2024-03-06T07:28:40.590113+0000","last_became_peered":"2024-03-06T07:28:40.590113+0000","last_unstale":"2024-03-06T07:28:42.615227+0000","last_undegraded":"2024-03-06T07:28:42.615227+0000","last_fullsized":"2024-03-06T07:28:42.615227+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:35.804062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615546+0000","last_change":"2024-03-06T07:28:41.613036+0000","last_active":"2024-03-06T07:28:42.615546+0000","last_peered":"2024-03-06T07:28:42.615546+0000","last_clean":"2024-03-06T07:28:42.615546+0000","last_became_active":"2024-03-06T07:28:41.612725+0000","last_became_peered":"2024-03-06T07:28:41.612725+0000","last_unstale":"2024-03-06T07:28:42.615546+0000","last_undegraded":"2024-03-06T07:28:42.615546+0000","last_fullsized":"2024-03-06T07:28:42.615546+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:20.966815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585400+0000","last_change":"2024-03-06T07:28:40.600729+0000","last_active":"2024-03-06T07:28:42.585400+0000","last_peered":"2024-03-06T07:28:42.585400+0000","last_clean":"2024-03-06T07:28:42.585400+0000","last_became_active":"2024-03-06T07:28:40.600441+0000","last_became_peered":"2024-03-06T07:28:40.600441+0000","last_unstale":"2024-03-06T07:28:42.585400+0000","last_undegraded":"2024-03-06T07:28:42.585400+0000","last_fullsized":"2024-03-06T07:28:42.585400+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:40.818375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615866+0000","last_change":"2024-03-06T07:28:41.606308+0000","last_active":"2024-03-06T07:28:42.615866+0000","last_peered":"2024-03-06T07:28:42.615866+0000","last_clean":"2024-03-06T07:28:42.615866+0000","last_became_active":"2024-03-06T07:28:41.605734+0000","last_became_peered":"2024-03-06T07:28:41.605734+0000","last_unstale":"2024-03-06T07:28:42.615866+0000","last_undegraded":"2024-03-06T07:28:42.615866+0000","last_fullsized":"2024-03-06T07:28:42.615866+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:18.073915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584417+0000","last_change":"2024-03-06T07:28:40.592798+0000","last_active":"2024-03-06T07:28:42.584417+0000","last_peered":"2024-03-06T07:28:42.584417+0000","last_clean":"2024-03-06T07:28:42.584417+0000","last_became_active":"2024-03-06T07:28:40.592483+0000","last_became_peered":"2024-03-06T07:28:40.592483+0000","last_unstale":"2024-03-06T07:28:42.584417+0000","last_undegraded":"2024-03-06T07:28:42.584417+0000","last_fullsized":"2024-03-06T07:28:42.584417+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:57.654186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584539+0000","last_change":"2024-03-06T07:28:41.606477+0000","last_active":"2024-03-06T07:28:42.584539+0000","last_peered":"2024-03-06T07:28:42.584539+0000","last_clean":"2024-03-06T07:28:42.584539+0000","last_became_active":"2024-03-06T07:28:41.606142+0000","last_became_peered":"2024-03-06T07:28:41.606142+0000","last_unstale":"2024-03-06T07:28:42.584539+0000","last_undegraded":"2024-03-06T07:28:42.584539+0000","last_fullsized":"2024-03-06T07:28:42.584539+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:03.190401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":17,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615153+0000","last_change":"2024-03-06T07:28:40.592800+0000","last_active":"2024-03-06T07:28:42.615153+0000","last_peered":"2024-03-06T07:28:42.615153+0000","last_clean":"2024-03-06T07:28:42.615153+0000","last_became_active":"2024-03-06T07:28:40.592525+0000","last_became_peered":"2024-03-06T07:28:40.592525+0000","last_unstale":"2024-03-06T07:28:42.615153+0000","last_undegraded":"2024-03-06T07:28:42.615153+0000","last_fullsized":"2024-03-06T07:28:42.615153+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:23.047792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615295+0000","last_change":"2024-03-06T07:28:41.602882+0000","last_active":"2024-03-06T07:28:42.615295+0000","last_peered":"2024-03-06T07:28:42.615295+0000","last_clean":"2024-03-06T07:28:42.615295+0000","last_became_active":"2024-03-06T07:28:41.602571+0000","last_became_peered":"2024-03-06T07:28:41.602571+0000","last_unstale":"2024-03-06T07:28:42.615295+0000","last_undegraded":"2024-03-06T07:28:42.615295+0000","last_fullsized":"2024-03-06T07:28:42.615295+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:28.691437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":20,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615479+0000","last_change":"2024-03-06T07:28:40.589011+0000","last_active":"2024-03-06T07:28:42.615479+0000","last_peered":"2024-03-06T07:28:42.615479+0000","last_clean":"2024-03-06T07:28:42.615479+0000","last_became_active":"2024-03-06T07:28:40.588731+0000","last_became_peered":"2024-03-06T07:28:40.588731+0000","last_unstale":"2024-03-06T07:28:42.615479+0000","last_undegraded":"2024-03-06T07:28:42.615479+0000","last_fullsized":"2024-03-06T07:28:42.615479+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:06.929357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584676+0000","last_change":"2024-03-06T07:28:41.603914+0000","last_active":"2024-03-06T07:28:42.584676+0000","last_peered":"2024-03-06T07:28:42.584676+0000","last_clean":"2024-03-06T07:28:42.584676+0000","last_became_active":"2024-03-06T07:28:41.603566+0000","last_became_peered":"2024-03-06T07:28:41.603566+0000","last_unstale":"2024-03-06T07:28:42.584676+0000","last_undegraded":"2024-03-06T07:28:42.584676+0000","last_fullsized":"2024-03-06T07:28:42.584676+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:24.371626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614486+0000","last_change":"2024-03-06T07:28:40.596849+0000","last_active":"2024-03-06T07:28:42.614486+0000","last_peered":"2024-03-06T07:28:42.614486+0000","last_clean":"2024-03-06T07:28:42.614486+0000","last_became_active":"2024-03-06T07:28:40.596315+0000","last_became_peered":"2024-03-06T07:28:40.596315+0000","last_unstale":"2024-03-06T07:28:42.614486+0000","last_undegraded":"2024-03-06T07:28:42.614486+0000","last_fullsized":"2024-03-06T07:28:42.614486+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:04.152803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584355+0000","last_change":"2024-03-06T07:28:41.593085+0000","last_active":"2024-03-06T07:28:42.584355+0000","last_peered":"2024-03-06T07:28:42.584355+0000","last_clean":"2024-03-06T07:28:42.584355+0000","last_became_active":"2024-03-06T07:28:41.592769+0000","last_became_peered":"2024-03-06T07:28:41.592769+0000","last_unstale":"2024-03-06T07:28:42.584355+0000","last_undegraded":"2024-03-06T07:28:42.584355+0000","last_fullsized":"2024-03-06T07:28:42.584355+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:55.787493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615402+0000","last_change":"2024-03-06T07:28:40.595680+0000","last_active":"2024-03-06T07:28:42.615402+0000","last_peered":"2024-03-06T07:28:42.615402+0000","last_clean":"2024-03-06T07:28:42.615402+0000","last_became_active":"2024-03-06T07:28:40.595437+0000","last_became_peered":"2024-03-06T07:28:40.595437+0000","last_unstale":"2024-03-06T07:28:42.615402+0000","last_undegraded":"2024-03-06T07:28:42.615402+0000","last_fullsized":"2024-03-06T07:28:42.615402+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:20.107356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584232+0000","last_change":"2024-03-06T07:28:41.604230+0000","last_active":"2024-03-06T07:28:42.584232+0000","last_peered":"2024-03-06T07:28:42.584232+0000","last_clean":"2024-03-06T07:28:42.584232+0000","last_became_active":"2024-03-06T07:28:41.602845+0000","last_became_peered":"2024-03-06T07:28:41.602845+0000","last_unstale":"2024-03-06T07:28:42.584232+0000","last_undegraded":"2024-03-06T07:28:42.584232+0000","last_fullsized":"2024-03-06T07:28:42.584232+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.995900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":22,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584998+0000","last_change":"2024-03-06T07:28:40.600673+0000","last_active":"2024-03-06T07:28:42.584998+0000","last_peered":"2024-03-06T07:28:42.584998+0000","last_clean":"2024-03-06T07:28:42.584998+0000","last_became_active":"2024-03-06T07:28:40.600373+0000","last_became_peered":"2024-03-06T07:28:40.600373+0000","last_unstale":"2024-03-06T07:28:42.584998+0000","last_undegraded":"2024-03-06T07:28:42.584998+0000","last_fullsized":"2024-03-06T07:28:42.584998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:03.969324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585624+0000","last_change":"2024-03-06T07:28:41.610614+0000","last_active":"2024-03-06T07:28:42.585624+0000","last_peered":"2024-03-06T07:28:42.585624+0000","last_clean":"2024-03-06T07:28:42.585624+0000","last_became_active":"2024-03-06T07:28:41.610307+0000","last_became_peered":"2024-03-06T07:28:41.610307+0000","last_unstale":"2024-03-06T07:28:42.585624+0000","last_undegraded":"2024-03-06T07:28:42.585624+0000","last_fullsized":"2024-03-06T07:28:42.585624+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:34.817395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615670+0000","last_change":"2024-03-06T07:28:40.591386+0000","last_active":"2024-03-06T07:28:42.615670+0000","last_peered":"2024-03-06T07:28:42.615670+0000","last_clean":"2024-03-06T07:28:42.615670+0000","last_became_active":"2024-03-06T07:28:40.591132+0000","last_became_peered":"2024-03-06T07:28:40.591132+0000","last_unstale":"2024-03-06T07:28:42.615670+0000","last_undegraded":"2024-03-06T07:28:42.615670+0000","last_fullsized":"2024-03-06T07:28:42.615670+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:54.835777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614664+0000","last_change":"2024-03-06T07:28:41.605192+0000","last_active":"2024-03-06T07:28:42.614664+0000","last_peered":"2024-03-06T07:28:42.614664+0000","last_clean":"2024-03-06T07:28:42.614664+0000","last_became_active":"2024-03-06T07:28:41.604812+0000","last_became_peered":"2024-03-06T07:28:41.604812+0000","last_unstale":"2024-03-06T07:28:42.614664+0000","last_undegraded":"2024-03-06T07:28:42.614664+0000","last_fullsized":"2024-03-06T07:28:42.614664+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:03.509633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585603+0000","last_change":"2024-03-06T07:28:40.598436+0000","last_active":"2024-03-06T07:28:42.585603+0000","last_peered":"2024-03-06T07:28:42.585603+0000","last_clean":"2024-03-06T07:28:42.585603+0000","last_became_active":"2024-03-06T07:28:40.598075+0000","last_became_peered":"2024-03-06T07:28:40.598075+0000","last_unstale":"2024-03-06T07:28:42.585603+0000","last_undegraded":"2024-03-06T07:28:42.585603+0000","last_fullsized":"2024-03-06T07:28:42.585603+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:21.546007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616566+0000","last_change":"2024-03-06T07:28:41.603468+0000","last_active":"2024-03-06T07:28:42.616566+0000","last_peered":"2024-03-06T07:28:42.616566+0000","last_clean":"2024-03-06T07:28:42.616566+0000","last_became_active":"2024-03-06T07:28:41.603081+0000","last_became_peered":"2024-03-06T07:28:41.603081+0000","last_unstale":"2024-03-06T07:28:42.616566+0000","last_undegraded":"2024-03-06T07:28:42.616566+0000","last_fullsized":"2024-03-06T07:28:42.616566+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:03.190401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615332+0000","last_change":"2024-03-06T07:28:40.594727+0000","last_active":"2024-03-06T07:28:42.615332+0000","last_peered":"2024-03-06T07:28:42.615332+0000","last_clean":"2024-03-06T07:28:42.615332+0000","last_became_active":"2024-03-06T07:28:40.594455+0000","last_became_peered":"2024-03-06T07:28:40.594455+0000","last_unstale":"2024-03-06T07:28:42.615332+0000","last_undegraded":"2024-03-06T07:28:42.615332+0000","last_fullsized":"2024-03-06T07:28:42.615332+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:36.199459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616429+0000","last_change":"2024-03-06T07:28:41.605423+0000","last_active":"2024-03-06T07:28:42.616429+0000","last_peered":"2024-03-06T07:28:42.616429+0000","last_clean":"2024-03-06T07:28:42.616429+0000","last_became_active":"2024-03-06T07:28:41.604962+0000","last_became_peered":"2024-03-06T07:28:41.604962+0000","last_unstale":"2024-03-06T07:28:42.616429+0000","last_undegraded":"2024-03-06T07:28:42.616429+0000","last_fullsized":"2024-03-06T07:28:42.616429+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:56.356422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614955+0000","last_change":"2024-03-06T07:28:40.598996+0000","last_active":"2024-03-06T07:28:42.614955+0000","last_peered":"2024-03-06T07:28:42.614955+0000","last_clean":"2024-03-06T07:28:42.614955+0000","last_became_active":"2024-03-06T07:28:40.598707+0000","last_became_peered":"2024-03-06T07:28:40.598707+0000","last_unstale":"2024-03-06T07:28:42.614955+0000","last_undegraded":"2024-03-06T07:28:42.614955+0000","last_fullsized":"2024-03-06T07:28:42.614955+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:53.516854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616329+0000","last_change":"2024-03-06T07:28:41.602357+0000","last_active":"2024-03-06T07:28:42.616329+0000","last_peered":"2024-03-06T07:28:42.616329+0000","last_clean":"2024-03-06T07:28:42.616329+0000","last_became_active":"2024-03-06T07:28:41.602093+0000","last_became_peered":"2024-03-06T07:28:41.602093+0000","last_unstale":"2024-03-06T07:28:42.616329+0000","last_undegraded":"2024-03-06T07:28:42.616329+0000","last_fullsized":"2024-03-06T07:28:42.616329+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:55.787493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614653+0000","last_change":"2024-03-06T07:28:40.600309+0000","last_active":"2024-03-06T07:28:42.614653+0000","last_peered":"2024-03-06T07:28:42.614653+0000","last_clean":"2024-03-06T07:28:42.614653+0000","last_became_active":"2024-03-06T07:28:40.599976+0000","last_became_peered":"2024-03-06T07:28:40.599976+0000","last_unstale":"2024-03-06T07:28:42.614653+0000","last_undegraded":"2024-03-06T07:28:42.614653+0000","last_fullsized":"2024-03-06T07:28:42.614653+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:42.371056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615062+0000","last_change":"2024-03-06T07:28:41.599242+0000","last_active":"2024-03-06T07:28:42.615062+0000","last_peered":"2024-03-06T07:28:42.615062+0000","last_clean":"2024-03-06T07:28:42.615062+0000","last_became_active":"2024-03-06T07:28:41.598920+0000","last_became_peered":"2024-03-06T07:28:41.598920+0000","last_unstale":"2024-03-06T07:28:42.615062+0000","last_undegraded":"2024-03-06T07:28:42.615062+0000","last_fullsized":"2024-03-06T07:28:42.615062+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:51.489658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585293+0000","last_change":"2024-03-06T07:28:40.601354+0000","last_active":"2024-03-06T07:28:42.585293+0000","last_peered":"2024-03-06T07:28:42.585293+0000","last_clean":"2024-03-06T07:28:42.585293+0000","last_became_active":"2024-03-06T07:28:40.601113+0000","last_became_peered":"2024-03-06T07:28:40.601113+0000","last_unstale":"2024-03-06T07:28:42.585293+0000","last_undegraded":"2024-03-06T07:28:42.585293+0000","last_fullsized":"2024-03-06T07:28:42.585293+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:35.874968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615717+0000","last_change":"2024-03-06T07:28:41.612198+0000","last_active":"2024-03-06T07:28:42.615717+0000","last_peered":"2024-03-06T07:28:42.615717+0000","last_clean":"2024-03-06T07:28:42.615717+0000","last_became_active":"2024-03-06T07:28:41.611905+0000","last_became_peered":"2024-03-06T07:28:41.611905+0000","last_unstale":"2024-03-06T07:28:42.615717+0000","last_undegraded":"2024-03-06T07:28:42.615717+0000","last_fullsized":"2024-03-06T07:28:42.615717+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:24.049884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614539+0000","last_change":"2024-03-06T07:28:40.599514+0000","last_active":"2024-03-06T07:28:42.614539+0000","last_peered":"2024-03-06T07:28:42.614539+0000","last_clean":"2024-03-06T07:28:42.614539+0000","last_became_active":"2024-03-06T07:28:40.599217+0000","last_became_peered":"2024-03-06T07:28:40.599217+0000","last_unstale":"2024-03-06T07:28:42.614539+0000","last_undegraded":"2024-03-06T07:28:42.614539+0000","last_fullsized":"2024-03-06T07:28:42.614539+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:47.022153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584335+0000","last_change":"2024-03-06T07:28:41.607422+0000","last_active":"2024-03-06T07:28:42.584335+0000","last_peered":"2024-03-06T07:28:42.584335+0000","last_clean":"2024-03-06T07:28:42.584335+0000","last_became_active":"2024-03-06T07:28:41.607072+0000","last_became_peered":"2024-03-06T07:28:41.607072+0000","last_unstale":"2024-03-06T07:28:42.584335+0000","last_undegraded":"2024-03-06T07:28:42.584335+0000","last_fullsized":"2024-03-06T07:28:42.584335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:05.095081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615350+0000","last_change":"2024-03-06T07:28:40.596316+0000","last_active":"2024-03-06T07:28:42.615350+0000","last_peered":"2024-03-06T07:28:42.615350+0000","last_clean":"2024-03-06T07:28:42.615350+0000","last_became_active":"2024-03-06T07:28:40.595681+0000","last_became_peered":"2024-03-06T07:28:40.595681+0000","last_unstale":"2024-03-06T07:28:42.615350+0000","last_undegraded":"2024-03-06T07:28:42.615350+0000","last_fullsized":"2024-03-06T07:28:42.615350+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:58.815160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616022+0000","last_change":"2024-03-06T07:28:41.605871+0000","last_active":"2024-03-06T07:28:42.616022+0000","last_peered":"2024-03-06T07:28:42.616022+0000","last_clean":"2024-03-06T07:28:42.616022+0000","last_became_active":"2024-03-06T07:28:41.605577+0000","last_became_peered":"2024-03-06T07:28:41.605577+0000","last_unstale":"2024-03-06T07:28:42.616022+0000","last_undegraded":"2024-03-06T07:28:42.616022+0000","last_fullsized":"2024-03-06T07:28:42.616022+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:36.806154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584550+0000","last_change":"2024-03-06T07:28:40.590129+0000","last_active":"2024-03-06T07:28:42.584550+0000","last_peered":"2024-03-06T07:28:42.584550+0000","last_clean":"2024-03-06T07:28:42.584550+0000","last_became_active":"2024-03-06T07:28:40.589864+0000","last_became_peered":"2024-03-06T07:28:40.589864+0000","last_unstale":"2024-03-06T07:28:42.584550+0000","last_undegraded":"2024-03-06T07:28:42.584550+0000","last_fullsized":"2024-03-06T07:28:42.584550+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:39.352210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615655+0000","last_change":"2024-03-06T07:28:41.607050+0000","last_active":"2024-03-06T07:28:42.615655+0000","last_peered":"2024-03-06T07:28:42.615655+0000","last_clean":"2024-03-06T07:28:42.615655+0000","last_became_active":"2024-03-06T07:28:41.606639+0000","last_became_peered":"2024-03-06T07:28:41.606639+0000","last_unstale":"2024-03-06T07:28:42.615655+0000","last_undegraded":"2024-03-06T07:28:42.615655+0000","last_fullsized":"2024-03-06T07:28:42.615655+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:09.115278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"114'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616105+0000","last_change":"2024-03-06T07:28:40.598310+0000","last_active":"2024-03-06T07:28:42.616105+0000","last_peered":"2024-03-06T07:28:42.616105+0000","last_clean":"2024-03-06T07:28:42.616105+0000","last_became_active":"2024-03-06T07:28:40.597952+0000","last_became_peered":"2024-03-06T07:28:40.597952+0000","last_unstale":"2024-03-06T07:28:42.616105+0000","last_undegraded":"2024-03-06T07:28:42.616105+0000","last_fullsized":"2024-03-06T07:28:42.616105+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:11.242574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615326+0000","last_change":"2024-03-06T07:28:41.600904+0000","last_active":"2024-03-06T07:28:42.615326+0000","last_peered":"2024-03-06T07:28:42.615326+0000","last_clean":"2024-03-06T07:28:42.615326+0000","last_became_active":"2024-03-06T07:28:41.600603+0000","last_became_peered":"2024-03-06T07:28:41.600603+0000","last_unstale":"2024-03-06T07:28:42.615326+0000","last_undegraded":"2024-03-06T07:28:42.615326+0000","last_fullsized":"2024-03-06T07:28:42.615326+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:45.519515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615204+0000","last_change":"2024-03-06T07:28:40.595293+0000","last_active":"2024-03-06T07:28:42.615204+0000","last_peered":"2024-03-06T07:28:42.615204+0000","last_clean":"2024-03-06T07:28:42.615204+0000","last_became_active":"2024-03-06T07:28:40.595028+0000","last_became_peered":"2024-03-06T07:28:40.595028+0000","last_unstale":"2024-03-06T07:28:42.615204+0000","last_undegraded":"2024-03-06T07:28:42.615204+0000","last_fullsized":"2024-03-06T07:28:42.615204+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:15.972492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615314+0000","last_change":"2024-03-06T07:28:41.603184+0000","last_active":"2024-03-06T07:28:42.615314+0000","last_peered":"2024-03-06T07:28:42.615314+0000","last_clean":"2024-03-06T07:28:42.615314+0000","last_became_active":"2024-03-06T07:28:41.602913+0000","last_became_peered":"2024-03-06T07:28:41.602913+0000","last_unstale":"2024-03-06T07:28:42.615314+0000","last_undegraded":"2024-03-06T07:28:42.615314+0000","last_fullsized":"2024-03-06T07:28:42.615314+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:12.604285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584462+0000","last_change":"2024-03-06T07:28:40.593167+0000","last_active":"2024-03-06T07:28:42.584462+0000","last_peered":"2024-03-06T07:28:42.584462+0000","last_clean":"2024-03-06T07:28:42.584462+0000","last_became_active":"2024-03-06T07:28:40.592882+0000","last_became_peered":"2024-03-06T07:28:40.592882+0000","last_unstale":"2024-03-06T07:28:42.584462+0000","last_undegraded":"2024-03-06T07:28:42.584462+0000","last_fullsized":"2024-03-06T07:28:42.584462+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:41.378581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584514+0000","last_change":"2024-03-06T07:28:41.603668+0000","last_active":"2024-03-06T07:28:42.584514+0000","last_peered":"2024-03-06T07:28:42.584514+0000","last_clean":"2024-03-06T07:28:42.584514+0000","last_became_active":"2024-03-06T07:28:41.603374+0000","last_became_peered":"2024-03-06T07:28:41.603374+0000","last_unstale":"2024-03-06T07:28:42.584514+0000","last_undegraded":"2024-03-06T07:28:42.584514+0000","last_fullsized":"2024-03-06T07:28:42.584514+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:12.244667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615144+0000","last_change":"2024-03-06T07:28:40.593051+0000","last_active":"2024-03-06T07:28:42.615144+0000","last_peered":"2024-03-06T07:28:42.615144+0000","last_clean":"2024-03-06T07:28:42.615144+0000","last_became_active":"2024-03-06T07:28:40.592727+0000","last_became_peered":"2024-03-06T07:28:40.592727+0000","last_unstale":"2024-03-06T07:28:42.615144+0000","last_undegraded":"2024-03-06T07:28:42.615144+0000","last_fullsized":"2024-03-06T07:28:42.615144+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:45.038458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584929+0000","last_change":"2024-03-06T07:28:41.612638+0000","last_active":"2024-03-06T07:28:42.584929+0000","last_peered":"2024-03-06T07:28:42.584929+0000","last_clean":"2024-03-06T07:28:42.584929+0000","last_became_active":"2024-03-06T07:28:41.612281+0000","last_became_peered":"2024-03-06T07:28:41.612281+0000","last_unstale":"2024-03-06T07:28:42.584929+0000","last_undegraded":"2024-03-06T07:28:42.584929+0000","last_fullsized":"2024-03-06T07:28:42.584929+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.656279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584484+0000","last_change":"2024-03-06T07:28:40.593507+0000","last_active":"2024-03-06T07:28:42.584484+0000","last_peered":"2024-03-06T07:28:42.584484+0000","last_clean":"2024-03-06T07:28:42.584484+0000","last_became_active":"2024-03-06T07:28:40.593219+0000","last_became_peered":"2024-03-06T07:28:40.593219+0000","last_unstale":"2024-03-06T07:28:42.584484+0000","last_undegraded":"2024-03-06T07:28:42.584484+0000","last_fullsized":"2024-03-06T07:28:42.584484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:27.475248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615978+0000","last_change":"2024-03-06T07:28:41.598292+0000","last_active":"2024-03-06T07:28:42.615978+0000","last_peered":"2024-03-06T07:28:42.615978+0000","last_clean":"2024-03-06T07:28:42.615978+0000","last_became_active":"2024-03-06T07:28:41.598002+0000","last_became_peered":"2024-03-06T07:28:41.598002+0000","last_unstale":"2024-03-06T07:28:42.615978+0000","last_undegraded":"2024-03-06T07:28:42.615978+0000","last_fullsized":"2024-03-06T07:28:42.615978+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:16.974584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585079+0000","last_change":"2024-03-06T07:28:40.597305+0000","last_active":"2024-03-06T07:28:42.585079+0000","last_peered":"2024-03-06T07:28:42.585079+0000","last_clean":"2024-03-06T07:28:42.585079+0000","last_became_active":"2024-03-06T07:28:40.596994+0000","last_became_peered":"2024-03-06T07:28:40.596994+0000","last_unstale":"2024-03-06T07:28:42.585079+0000","last_undegraded":"2024-03-06T07:28:42.585079+0000","last_fullsized":"2024-03-06T07:28:42.585079+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:40.886362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614978+0000","last_change":"2024-03-06T07:28:41.605397+0000","last_active":"2024-03-06T07:28:42.614978+0000","last_peered":"2024-03-06T07:28:42.614978+0000","last_clean":"2024-03-06T07:28:42.614978+0000","last_became_active":"2024-03-06T07:28:41.604725+0000","last_became_peered":"2024-03-06T07:28:41.604725+0000","last_unstale":"2024-03-06T07:28:42.614978+0000","last_undegraded":"2024-03-06T07:28:42.614978+0000","last_fullsized":"2024-03-06T07:28:42.614978+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:21.039498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615436+0000","last_change":"2024-03-06T07:28:40.596580+0000","last_active":"2024-03-06T07:28:42.615436+0000","last_peered":"2024-03-06T07:28:42.615436+0000","last_clean":"2024-03-06T07:28:42.615436+0000","last_became_active":"2024-03-06T07:28:40.596067+0000","last_became_peered":"2024-03-06T07:28:40.596067+0000","last_unstale":"2024-03-06T07:28:42.615436+0000","last_undegraded":"2024-03-06T07:28:42.615436+0000","last_fullsized":"2024-03-06T07:28:42.615436+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:49.059409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584162+0000","last_change":"2024-03-06T07:28:41.593409+0000","last_active":"2024-03-06T07:28:42.584162+0000","last_peered":"2024-03-06T07:28:42.584162+0000","last_clean":"2024-03-06T07:28:42.584162+0000","last_became_active":"2024-03-06T07:28:41.592822+0000","last_became_peered":"2024-03-06T07:28:41.592822+0000","last_unstale":"2024-03-06T07:28:42.584162+0000","last_undegraded":"2024-03-06T07:28:42.584162+0000","last_fullsized":"2024-03-06T07:28:42.584162+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:55.837869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615076+0000","last_change":"2024-03-06T07:28:40.598826+0000","last_active":"2024-03-06T07:28:42.615076+0000","last_peered":"2024-03-06T07:28:42.615076+0000","last_clean":"2024-03-06T07:28:42.615076+0000","last_became_active":"2024-03-06T07:28:40.598534+0000","last_became_peered":"2024-03-06T07:28:40.598534+0000","last_unstale":"2024-03-06T07:28:42.615076+0000","last_undegraded":"2024-03-06T07:28:42.615076+0000","last_fullsized":"2024-03-06T07:28:42.615076+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:11.019639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":348,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615864+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:28:42.615864+0000","last_peered":"2024-03-06T07:28:42.615864+0000","last_clean":"2024-03-06T07:28:42.615864+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:28:42.615864+0000","last_undegraded":"2024-03-06T07:28:42.615864+0000","last_fullsized":"2024-03-06T07:28:42.615864+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37316,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334524,"statfs":{"total":96636764160,"available":96598552576,"internally_reserved":0,"allocated":2019328,"data_stored":1201971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574926,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40896,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330944,"statfs":{"total":96636764160,"available":96594886656,"internally_reserved":0,"allocated":1421312,"data_stored":611057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40900,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330940,"statfs":{"total":96636764160,"available":96594882560,"internally_reserved":0,"allocated":1425408,"data_stored":611591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40888,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330952,"statfs":{"total":96636764160,"available":96594894848,"internally_reserved":0,"allocated":1417216,"data_stored":610987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574925,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40908,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330932,"statfs":{"total":96636764160,"available":96594874368,"internally_reserved":0,"allocated":1433600,"data_stored":612669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40892,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330948,"statfs":{"total":96636764160,"available":96594890752,"internally_reserved":0,"allocated":1417216,"data_stored":610987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40900,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330940,"statfs":{"total":96636764160,"available":96594882560,"internally_reserved":0,"allocated":1425408,"data_stored":611111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41488,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330352,"statfs":{"total":96636764160,"available":96594280448,"internally_reserved":0,"allocated":2031616,"data_stored":1203079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:46.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:28:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.499+0000 7fdc81181640 1 Processor -- start 2024-03-06T07:28:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.499+0000 7fdc81181640 1 -- start start 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc81181640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc7c0b95a0 0x7fdc7c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc81181640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdc7c0b9a10 0x7fdc7c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc81181640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0cea30 con 0x7fdc7c15df00 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc81181640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c0cf3c0 con 0x7fdc7c0b95a0 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc81181640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c0cf0e0 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc7a59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc7c0b95a0 0x7fdc7c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc7a59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc7c0b95a0 0x7fdc7c0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44624/0 (socket says 172.21.15.27:44624) 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc7a59c640 1 -- 172.21.15.27:0/3796550792 learned_addr learned my addr 172.21.15.27:0/3796550792 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc7b59e640 1 --2- 172.21.15.27:0/3796550792 >> v2:172.21.15.69:3300/0 conn(0x7fdc7c0b9a10 0x7fdc7c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc7a59c640 1 -- 172.21.15.27:0/3796550792 >> v2:172.21.15.69:3300/0 conn(0x7fdc7c0b9a10 msgr2=0x7fdc7c0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc7a59c640 1 --2- 172.21.15.27:0/3796550792 >> v2:172.21.15.69:3300/0 conn(0x7fdc7c0b9a10 0x7fdc7c0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc7a59c640 1 -- 172.21.15.27:0/3796550792 >> v1:172.21.15.69:6789/0 conn(0x7fdc7c15df00 legacy=0x7fdc7c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.500+0000 7fdc7a59c640 1 -- 172.21.15.27:0/3796550792 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c0ce470 con 0x7fdc7c0b95a0 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.501+0000 7fdc7a59c640 1 --2- 172.21.15.27:0/3796550792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b95a0 0x7fdc7c0b5700 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0d4d20 tx=0x7fdc7002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b45be36527f9f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.501+0000 7fdc79d9b640 1 -- 172.21.15.27:0/3796550792 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70034070 con 0x7fdc7c0b95a0 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.501+0000 7fdc79d9b640 1 -- 172.21.15.27:0/3796550792 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc70030860 con 0x7fdc7c0b95a0 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.501+0000 7fdc79d9b640 1 -- 172.21.15.27:0/3796550792 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70030c30 con 0x7fdc7c0b95a0 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.502+0000 7fdc81181640 1 -- 172.21.15.27:0/3796550792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b95a0 msgr2=0x7fdc7c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.502+0000 7fdc81181640 1 --2- 172.21.15.27:0/3796550792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b95a0 0x7fdc7c0b5700 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0d4d20 tx=0x7fdc7002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.502+0000 7fdc81181640 1 -- 172.21.15.27:0/3796550792 shutdown_connections 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.502+0000 7fdc81181640 1 --2- 172.21.15.27:0/3796550792 >> v2:172.21.15.69:3300/0 conn(0x7fdc7c0b9a10 0x7fdc7c0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.502+0000 7fdc81181640 1 --2- 172.21.15.27:0/3796550792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b95a0 0x7fdc7c0b5700 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.502+0000 7fdc81181640 1 -- 172.21.15.27:0/3796550792 >> 172.21.15.27:0/3796550792 conn(0x7fdc7c071960 msgr2=0x7fdc7c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.502+0000 7fdc81181640 1 -- 172.21.15.27:0/3796550792 shutdown_connections 2024-03-06T07:28:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.502+0000 7fdc81181640 1 -- 172.21.15.27:0/3796550792 wait complete. 2024-03-06T07:28:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.503+0000 7fdc81181640 1 Processor -- start 2024-03-06T07:28:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.503+0000 7fdc81181640 1 -- start start 2024-03-06T07:28:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc81181640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc7c0b95a0 0x7fdc7c155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc81181640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b9a10 0x7fdc7c0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc81181640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc7c15df00 0x7fdc7c1618b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc81181640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0cedf0 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc81181640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0cec70 con 0x7fdc7c0b95a0 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc81181640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0d3220 con 0x7fdc7c15df00 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc7a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b9a10 0x7fdc7c0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc7a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b9a10 0x7fdc7c0cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44632/0 (socket says 172.21.15.27:44632) 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc7a59c640 1 -- 172.21.15.27:0/1804947265 learned_addr learned my addr 172.21.15.27:0/1804947265 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc7ad9d640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc7c0b95a0 0x7fdc7c155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.504+0000 7fdc7b59e640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc7c15df00 0x7fdc7c1618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc7a59c640 1 -- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc7c15df00 msgr2=0x7fdc7c1618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc7a59c640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc7c15df00 0x7fdc7c1618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc7a59c640 1 -- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc7c0b95a0 msgr2=0x7fdc7c155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc7a59c640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc7c0b95a0 0x7fdc7c155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc7a59c640 1 -- 172.21.15.27:0/1804947265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c155770 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc7a59c640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b9a10 0x7fdc7c0cad50 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fdc7002b9e0 tx=0x7fdc70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc637fe640 1 -- 172.21.15.27:0/1804947265 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70048070 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc637fe640 1 -- 172.21.15.27:0/1804947265 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc70030660 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c0cb7f0 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc637fe640 1 -- 172.21.15.27:0/1804947265 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70034040 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.505+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c0cc0d0 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.506+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c067ff0 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.507+0000 7fdc637fe640 1 -- 172.21.15.27:0/1804947265 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc7004b070 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.507+0000 7fdc637fe640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc6806b500 0x7fdc6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.507+0000 7fdc637fe640 1 -- 172.21.15.27:0/1804947265 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc700abfe0 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.507+0000 7fdc7ad9d640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc6806b500 0x7fdc6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.508+0000 7fdc7ad9d640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc6806b500 0x7fdc6806d9c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0c7930 tx=0x7fdc6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.512+0000 7fdc637fe640 1 -- 172.21.15.27:0/1804947265 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc7c155770 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.765+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fdc7c0b7d40 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.765+0000 7fdc637fe640 1 -- 172.21.15.27:0/1804947265 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1014 (secure 0 0 0) 0x7fdc70030830 con 0x7fdc7c0b9a10 2024-03-06T07:28:46.767 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:46.768 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":383,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":114,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709754,"num_in_osds":8,"osd_in_since":1709709745,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":331968512,"bytes_avail":772762144768,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":75,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"a","status":"up:active","gid":7959}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-03-06T07:27:58.061277+0000","services":{}},"progress_events":{}} 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc6806b500 msgr2=0x7fdc6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc6806b500 0x7fdc6806d9c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0c7930 tx=0x7fdc6c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b9a10 msgr2=0x7fdc7c0cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b9a10 0x7fdc7c0cad50 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fdc7002b9e0 tx=0x7fdc70004730 comp rx=0 tx=0).stop 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 shutdown_connections 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc6806b500 0x7fdc6806d9c0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc7c15df00 0x7fdc7c1618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc7c0b9a10 0x7fdc7c0cad50 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 --2- 172.21.15.27:0/1804947265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc7c0b95a0 0x7fdc7c155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 >> 172.21.15.27:0/1804947265 conn(0x7fdc7c071960 msgr2=0x7fdc7c15c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 shutdown_connections 2024-03-06T07:28:46.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.767+0000 7fdc81181640 1 -- 172.21.15.27:0/1804947265 wait complete. 2024-03-06T07:28:46.781 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:28:46.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:46.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.886+0000 7fb668793640 1 Processor -- start 2024-03-06T07:28:46.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.886+0000 7fb668793640 1 -- start start 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb668793640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb660071540 0x7fb6600b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb668793640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6600b2b50 0x7fb6600b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb668793640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6600ce610 con 0x7fb6600d3400 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb668793640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6600d2f70 con 0x7fb6600b2b50 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb668793640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600d4be0 con 0x7fb660071540 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb666d30640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6600b2b50 0x7fb6600b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb665d2e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb6600d3400 0x7fb660071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34826/0 (socket says 172.21.15.27:34826) 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb666d30640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6600b2b50 0x7fb6600b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44638/0 (socket says 172.21.15.27:44638) 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb665d2e640 1 -- 172.21.15.27:0/2321863465 learned_addr learned my addr 172.21.15.27:0/2321863465 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb66652f640 1 --2- 172.21.15.27:0/2321863465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb660071540 0x7fb6600b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb666d30640 1 -- 172.21.15.27:0/2321863465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb660071540 msgr2=0x7fb6600b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb666d30640 1 --2- 172.21.15.27:0/2321863465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb660071540 0x7fb6600b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb666d30640 1 -- 172.21.15.27:0/2321863465 >> v1:172.21.15.27:6789/0 conn(0x7fb6600d3400 legacy=0x7fb660071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb66552d640 1 -- 172.21.15.27:0/2321863465 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942731665 0 0) 0x7fb6600ce610 con 0x7fb6600d3400 2024-03-06T07:28:46.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb666d30640 1 -- 172.21.15.27:0/2321863465 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6600d2920 con 0x7fb6600b2b50 2024-03-06T07:28:46.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.887+0000 7fb66652f640 1 --2- 172.21.15.27:0/2321863465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb660071540 0x7fb6600b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:46.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.888+0000 7fb666d30640 1 --2- 172.21.15.27:0/2321863465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 0x7fb6600b6e20 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb658009d90 tx=0x7fb65802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa3ee3a364e4d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb66552d640 1 -- 172.21.15.27:0/2321863465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb658039070 con 0x7fb6600b2b50 2024-03-06T07:28:46.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb66552d640 1 -- 172.21.15.27:0/2321863465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65802cc50 con 0x7fb6600b2b50 2024-03-06T07:28:46.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb66552d640 1 -- 172.21.15.27:0/2321863465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb658034440 con 0x7fb6600b2b50 2024-03-06T07:28:46.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb668793640 1 -- 172.21.15.27:0/2321863465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 msgr2=0x7fb6600b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb668793640 1 --2- 172.21.15.27:0/2321863465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 0x7fb6600b6e20 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb658009d90 tx=0x7fb65802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb668793640 1 -- 172.21.15.27:0/2321863465 shutdown_connections 2024-03-06T07:28:46.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb668793640 1 --2- 172.21.15.27:0/2321863465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 0x7fb6600b6e20 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb668793640 1 --2- 172.21.15.27:0/2321863465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb660071540 0x7fb6600b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb668793640 1 -- 172.21.15.27:0/2321863465 >> 172.21.15.27:0/2321863465 conn(0x7fb6600d3ca0 msgr2=0x7fb6600d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb668793640 1 -- 172.21.15.27:0/2321863465 shutdown_connections 2024-03-06T07:28:46.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.889+0000 7fb668793640 1 -- 172.21.15.27:0/2321863465 wait complete. 2024-03-06T07:28:46.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.891+0000 7fb668793640 1 Processor -- start 2024-03-06T07:28:46.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.891+0000 7fb668793640 1 -- start start 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb668793640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb660071540 0x7fb660155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb668793640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 0x7fb6600cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb668793640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6600d3400 0x7fb6601f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb668793640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66015e370 con 0x7fb6600b2b50 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb668793640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600ce8f0 con 0x7fb660071540 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb668793640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6601f3d20 con 0x7fb6600d3400 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb665d2e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 0x7fb6600cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb665d2e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 0x7fb6600cc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44650/0 (socket says 172.21.15.27:44650) 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb665d2e640 1 -- 172.21.15.27:0/494181995 learned_addr learned my addr 172.21.15.27:0/494181995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb66652f640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb660071540 0x7fb660155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb666d30640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6600d3400 0x7fb6601f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb665d2e640 1 -- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6600d3400 msgr2=0x7fb6601f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb665d2e640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6600d3400 0x7fb6601f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb665d2e640 1 -- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb660071540 msgr2=0x7fb660155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb665d2e640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb660071540 0x7fb660155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb665d2e640 1 -- 172.21.15.27:0/494181995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6601f4800 con 0x7fb6600b2b50 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb665d2e640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 0x7fb6600cc7d0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb65000c410 tx=0x7fb65000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb656ffd640 1 -- 172.21.15.27:0/494181995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500074e0 con 0x7fb6600b2b50 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb656ffd640 1 -- 172.21.15.27:0/494181995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6500047e0 con 0x7fb6600b2b50 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.893+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6601f3f60 con 0x7fb6600b2b50 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.894+0000 7fb656ffd640 1 -- 172.21.15.27:0/494181995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb650004bd0 con 0x7fb6600b2b50 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.894+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6601f4520 con 0x7fb6600b2b50 2024-03-06T07:28:46.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.895+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb660068700 con 0x7fb6600b2b50 2024-03-06T07:28:46.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.895+0000 7fb656ffd640 1 -- 172.21.15.27:0/494181995 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb650011070 con 0x7fb6600b2b50 2024-03-06T07:28:46.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.896+0000 7fb656ffd640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb64406b4d0 0x7fb64406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.896+0000 7fb656ffd640 1 -- 172.21.15.27:0/494181995 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb650089e60 con 0x7fb6600b2b50 2024-03-06T07:28:46.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.896+0000 7fb66652f640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb64406b4d0 0x7fb64406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.896+0000 7fb66652f640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb64406b4d0 0x7fb64406d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb66015cf80 tx=0x7fb65c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:46.900+0000 7fb656ffd640 1 -- 172.21.15.27:0/494181995 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb650058ce0 con 0x7fb6600b2b50 2024-03-06T07:28:47.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.104+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb66015e820 con 0x7fb64406b4d0 2024-03-06T07:28:47.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.111+0000 7fb656ffd640 1 -- 172.21.15.27:0/494181995 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252482 (secure 0 0 0) 0x7fb66015e820 con 0x7fb64406b4d0 2024-03-06T07:28:47.116 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:47.117 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:28:47.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb64406b4d0 msgr2=0x7fb64406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb64406b4d0 0x7fb64406d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb66015cf80 tx=0x7fb65c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 msgr2=0x7fb6600cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 0x7fb6600cc7d0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb65000c410 tx=0x7fb65000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 shutdown_connections 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb64406b4d0 0x7fb64406d990 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6600d3400 0x7fb6601f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6600b2b50 0x7fb6600cc7d0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 --2- 172.21.15.27:0/494181995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb660071540 0x7fb660155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 >> 172.21.15.27:0/494181995 conn(0x7fb6600d3ca0 msgr2=0x7fb660071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 shutdown_connections 2024-03-06T07:28:47.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:47.114+0000 7fb668793640 1 -- 172.21.15.27:0/494181995 wait complete. 2024-03-06T07:28:47.127 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-03-06T07:28:46.072316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324188,"kb_used_data":12296,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754650532,"statfs":{"total":773094113280,"available":772762144768,"internally_reserved":0,"allocated":12591104,"data_stored":6073452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":319213106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001546"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615443+0000","last_change":"2024-03-06T07:28:41.605491+0000","last_active":"2024-03-06T07:28:42.615443+0000","last_peered":"2024-03-06T07:28:42.615443+0000","last_clean":"2024-03-06T07:28:42.615443+0000","last_became_active":"2024-03-06T07:28:41.605207+0000","last_became_peered":"2024-03-06T07:28:41.605207+0000","last_unstale":"2024-03-06T07:28:42.615443+0000","last_undegraded":"2024-03-06T07:28:42.615443+0000","last_fullsized":"2024-03-06T07:28:42.615443+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:41.754749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584798+0000","last_change":"2024-03-06T07:28:41.598081+0000","last_active":"2024-03-06T07:28:42.584798+0000","last_peered":"2024-03-06T07:28:42.584798+0000","last_clean":"2024-03-06T07:28:42.584798+0000","last_became_active":"2024-03-06T07:28:41.597746+0000","last_became_peered":"2024-03-06T07:28:41.597746+0000","last_unstale":"2024-03-06T07:28:42.584798+0000","last_undegraded":"2024-03-06T07:28:42.584798+0000","last_fullsized":"2024-03-06T07:28:42.584798+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:36.645183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616191+0000","last_change":"2024-03-06T07:28:41.604865+0000","last_active":"2024-03-06T07:28:42.616191+0000","last_peered":"2024-03-06T07:28:42.616191+0000","last_clean":"2024-03-06T07:28:42.616191+0000","last_became_active":"2024-03-06T07:28:41.603741+0000","last_became_peered":"2024-03-06T07:28:41.603741+0000","last_unstale":"2024-03-06T07:28:42.616191+0000","last_undegraded":"2024-03-06T07:28:42.616191+0000","last_fullsized":"2024-03-06T07:28:42.616191+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.995900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615181+0000","last_change":"2024-03-06T07:28:41.606885+0000","last_active":"2024-03-06T07:28:42.615181+0000","last_peered":"2024-03-06T07:28:42.615181+0000","last_clean":"2024-03-06T07:28:42.615181+0000","last_became_active":"2024-03-06T07:28:41.606582+0000","last_became_peered":"2024-03-06T07:28:41.606582+0000","last_unstale":"2024-03-06T07:28:42.615181+0000","last_undegraded":"2024-03-06T07:28:42.615181+0000","last_fullsized":"2024-03-06T07:28:42.615181+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:40.695072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585291+0000","last_change":"2024-03-06T07:28:41.611053+0000","last_active":"2024-03-06T07:28:42.585291+0000","last_peered":"2024-03-06T07:28:42.585291+0000","last_clean":"2024-03-06T07:28:42.585291+0000","last_became_active":"2024-03-06T07:28:41.610636+0000","last_became_peered":"2024-03-06T07:28:41.610636+0000","last_unstale":"2024-03-06T07:28:42.585291+0000","last_undegraded":"2024-03-06T07:28:42.585291+0000","last_fullsized":"2024-03-06T07:28:42.585291+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:54.518946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614777+0000","last_change":"2024-03-06T07:28:41.588032+0000","last_active":"2024-03-06T07:28:42.614777+0000","last_peered":"2024-03-06T07:28:42.614777+0000","last_clean":"2024-03-06T07:28:42.614777+0000","last_became_active":"2024-03-06T07:28:41.587772+0000","last_became_peered":"2024-03-06T07:28:41.587772+0000","last_unstale":"2024-03-06T07:28:42.614777+0000","last_undegraded":"2024-03-06T07:28:42.614777+0000","last_fullsized":"2024-03-06T07:28:42.614777+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:34.817395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615081+0000","last_change":"2024-03-06T07:28:41.597654+0000","last_active":"2024-03-06T07:28:42.615081+0000","last_peered":"2024-03-06T07:28:42.615081+0000","last_clean":"2024-03-06T07:28:42.615081+0000","last_became_active":"2024-03-06T07:28:41.597375+0000","last_became_peered":"2024-03-06T07:28:41.597375+0000","last_unstale":"2024-03-06T07:28:42.615081+0000","last_undegraded":"2024-03-06T07:28:42.615081+0000","last_fullsized":"2024-03-06T07:28:42.615081+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:55.516936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615462+0000","last_change":"2024-03-06T07:28:41.598969+0000","last_active":"2024-03-06T07:28:42.615462+0000","last_peered":"2024-03-06T07:28:42.615462+0000","last_clean":"2024-03-06T07:28:42.615462+0000","last_became_active":"2024-03-06T07:28:41.598680+0000","last_became_peered":"2024-03-06T07:28:41.598680+0000","last_unstale":"2024-03-06T07:28:42.615462+0000","last_undegraded":"2024-03-06T07:28:42.615462+0000","last_fullsized":"2024-03-06T07:28:42.615462+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:07.931449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615598+0000","last_change":"2024-03-06T07:28:41.607892+0000","last_active":"2024-03-06T07:28:42.615598+0000","last_peered":"2024-03-06T07:28:42.615598+0000","last_clean":"2024-03-06T07:28:42.615598+0000","last_became_active":"2024-03-06T07:28:41.607625+0000","last_became_peered":"2024-03-06T07:28:41.607625+0000","last_unstale":"2024-03-06T07:28:42.615598+0000","last_undegraded":"2024-03-06T07:28:42.615598+0000","last_fullsized":"2024-03-06T07:28:42.615598+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:15.356128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615335+0000","last_change":"2024-03-06T07:28:41.613674+0000","last_active":"2024-03-06T07:28:42.615335+0000","last_peered":"2024-03-06T07:28:42.615335+0000","last_clean":"2024-03-06T07:28:42.615335+0000","last_became_active":"2024-03-06T07:28:41.613392+0000","last_became_peered":"2024-03-06T07:28:41.613392+0000","last_unstale":"2024-03-06T07:28:42.615335+0000","last_undegraded":"2024-03-06T07:28:42.615335+0000","last_fullsized":"2024-03-06T07:28:42.615335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:56.109795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585034+0000","last_change":"2024-03-06T07:28:41.599288+0000","last_active":"2024-03-06T07:28:42.585034+0000","last_peered":"2024-03-06T07:28:42.585034+0000","last_clean":"2024-03-06T07:28:42.585034+0000","last_became_active":"2024-03-06T07:28:41.599023+0000","last_became_peered":"2024-03-06T07:28:41.599023+0000","last_unstale":"2024-03-06T07:28:42.585034+0000","last_undegraded":"2024-03-06T07:28:42.585034+0000","last_fullsized":"2024-03-06T07:28:42.585034+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:12.604285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615406+0000","last_change":"2024-03-06T07:28:41.603785+0000","last_active":"2024-03-06T07:28:42.615406+0000","last_peered":"2024-03-06T07:28:42.615406+0000","last_clean":"2024-03-06T07:28:42.615406+0000","last_became_active":"2024-03-06T07:28:41.603478+0000","last_became_peered":"2024-03-06T07:28:41.603478+0000","last_unstale":"2024-03-06T07:28:42.615406+0000","last_undegraded":"2024-03-06T07:28:42.615406+0000","last_fullsized":"2024-03-06T07:28:42.615406+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:18.198067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616026+0000","last_change":"2024-03-06T07:28:41.612243+0000","last_active":"2024-03-06T07:28:42.616026+0000","last_peered":"2024-03-06T07:28:42.616026+0000","last_clean":"2024-03-06T07:28:42.616026+0000","last_became_active":"2024-03-06T07:28:41.611872+0000","last_became_peered":"2024-03-06T07:28:41.611872+0000","last_unstale":"2024-03-06T07:28:42.616026+0000","last_undegraded":"2024-03-06T07:28:42.616026+0000","last_fullsized":"2024-03-06T07:28:42.616026+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:55.837869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615140+0000","last_change":"2024-03-06T07:28:41.597885+0000","last_active":"2024-03-06T07:28:42.615140+0000","last_peered":"2024-03-06T07:28:42.615140+0000","last_clean":"2024-03-06T07:28:42.615140+0000","last_became_active":"2024-03-06T07:28:41.597613+0000","last_became_peered":"2024-03-06T07:28:41.597613+0000","last_unstale":"2024-03-06T07:28:42.615140+0000","last_undegraded":"2024-03-06T07:28:42.615140+0000","last_fullsized":"2024-03-06T07:28:42.615140+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:50.061502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614475+0000","last_change":"2024-03-06T07:28:41.587309+0000","last_active":"2024-03-06T07:28:42.614475+0000","last_peered":"2024-03-06T07:28:42.614475+0000","last_clean":"2024-03-06T07:28:42.614475+0000","last_became_active":"2024-03-06T07:28:41.587034+0000","last_became_peered":"2024-03-06T07:28:41.587034+0000","last_unstale":"2024-03-06T07:28:42.614475+0000","last_undegraded":"2024-03-06T07:28:42.614475+0000","last_fullsized":"2024-03-06T07:28:42.614475+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:02.582015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614882+0000","last_change":"2024-03-06T07:28:41.610442+0000","last_active":"2024-03-06T07:28:42.614882+0000","last_peered":"2024-03-06T07:28:42.614882+0000","last_clean":"2024-03-06T07:28:42.614882+0000","last_became_active":"2024-03-06T07:28:41.605967+0000","last_became_peered":"2024-03-06T07:28:41.605967+0000","last_unstale":"2024-03-06T07:28:42.614882+0000","last_undegraded":"2024-03-06T07:28:42.614882+0000","last_fullsized":"2024-03-06T07:28:42.614882+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:50.576369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584911+0000","last_change":"2024-03-06T07:28:41.600051+0000","last_active":"2024-03-06T07:28:42.584911+0000","last_peered":"2024-03-06T07:28:42.584911+0000","last_clean":"2024-03-06T07:28:42.584911+0000","last_became_active":"2024-03-06T07:28:41.599530+0000","last_became_peered":"2024-03-06T07:28:41.599530+0000","last_unstale":"2024-03-06T07:28:42.584911+0000","last_undegraded":"2024-03-06T07:28:42.584911+0000","last_fullsized":"2024-03-06T07:28:42.584911+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:19.178731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615853+0000","last_change":"2024-03-06T07:28:41.602134+0000","last_active":"2024-03-06T07:28:42.615853+0000","last_peered":"2024-03-06T07:28:42.615853+0000","last_clean":"2024-03-06T07:28:42.615853+0000","last_became_active":"2024-03-06T07:28:41.601846+0000","last_became_peered":"2024-03-06T07:28:41.601846+0000","last_unstale":"2024-03-06T07:28:42.615853+0000","last_undegraded":"2024-03-06T07:28:42.615853+0000","last_fullsized":"2024-03-06T07:28:42.615853+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:13.831484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616246+0000","last_change":"2024-03-06T07:28:41.612404+0000","last_active":"2024-03-06T07:28:42.616246+0000","last_peered":"2024-03-06T07:28:42.616246+0000","last_clean":"2024-03-06T07:28:42.616246+0000","last_became_active":"2024-03-06T07:28:41.612121+0000","last_became_peered":"2024-03-06T07:28:41.612121+0000","last_unstale":"2024-03-06T07:28:42.616246+0000","last_undegraded":"2024-03-06T07:28:42.616246+0000","last_fullsized":"2024-03-06T07:28:42.616246+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:05.095081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584952+0000","last_change":"2024-03-06T07:28:41.593131+0000","last_active":"2024-03-06T07:28:42.584952+0000","last_peered":"2024-03-06T07:28:42.584952+0000","last_clean":"2024-03-06T07:28:42.584952+0000","last_became_active":"2024-03-06T07:28:41.592855+0000","last_became_peered":"2024-03-06T07:28:41.592855+0000","last_unstale":"2024-03-06T07:28:42.584952+0000","last_undegraded":"2024-03-06T07:28:42.584952+0000","last_fullsized":"2024-03-06T07:28:42.584952+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:40.354302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585104+0000","last_change":"2024-03-06T07:28:41.594310+0000","last_active":"2024-03-06T07:28:42.585104+0000","last_peered":"2024-03-06T07:28:42.585104+0000","last_clean":"2024-03-06T07:28:42.585104+0000","last_became_active":"2024-03-06T07:28:41.593649+0000","last_became_peered":"2024-03-06T07:28:41.593649+0000","last_unstale":"2024-03-06T07:28:42.585104+0000","last_undegraded":"2024-03-06T07:28:42.585104+0000","last_fullsized":"2024-03-06T07:28:42.585104+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:34.600777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584533+0000","last_change":"2024-03-06T07:28:41.599908+0000","last_active":"2024-03-06T07:28:42.584533+0000","last_peered":"2024-03-06T07:28:42.584533+0000","last_clean":"2024-03-06T07:28:42.584533+0000","last_became_active":"2024-03-06T07:28:41.599609+0000","last_became_peered":"2024-03-06T07:28:41.599609+0000","last_unstale":"2024-03-06T07:28:42.584533+0000","last_undegraded":"2024-03-06T07:28:42.584533+0000","last_fullsized":"2024-03-06T07:28:42.584533+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:56.356422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584612+0000","last_change":"2024-03-06T07:28:41.592523+0000","last_active":"2024-03-06T07:28:42.584612+0000","last_peered":"2024-03-06T07:28:42.584612+0000","last_clean":"2024-03-06T07:28:42.584612+0000","last_became_active":"2024-03-06T07:28:41.592223+0000","last_became_peered":"2024-03-06T07:28:41.592223+0000","last_unstale":"2024-03-06T07:28:42.584612+0000","last_undegraded":"2024-03-06T07:28:42.584612+0000","last_fullsized":"2024-03-06T07:28:42.584612+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:21.109449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584132+0000","last_change":"2024-03-06T07:28:41.593471+0000","last_active":"2024-03-06T07:28:42.584132+0000","last_peered":"2024-03-06T07:28:42.584132+0000","last_clean":"2024-03-06T07:28:42.584132+0000","last_became_active":"2024-03-06T07:28:41.593153+0000","last_became_peered":"2024-03-06T07:28:41.593153+0000","last_unstale":"2024-03-06T07:28:42.584132+0000","last_undegraded":"2024-03-06T07:28:42.584132+0000","last_fullsized":"2024-03-06T07:28:42.584132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:16.974584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585034+0000","last_change":"2024-03-06T07:28:41.599835+0000","last_active":"2024-03-06T07:28:42.585034+0000","last_peered":"2024-03-06T07:28:42.585034+0000","last_clean":"2024-03-06T07:28:42.585034+0000","last_became_active":"2024-03-06T07:28:41.599196+0000","last_became_peered":"2024-03-06T07:28:41.599196+0000","last_unstale":"2024-03-06T07:28:42.585034+0000","last_undegraded":"2024-03-06T07:28:42.585034+0000","last_fullsized":"2024-03-06T07:28:42.585034+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:42.380673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615615+0000","last_change":"2024-03-06T07:28:41.604648+0000","last_active":"2024-03-06T07:28:42.615615+0000","last_peered":"2024-03-06T07:28:42.615615+0000","last_clean":"2024-03-06T07:28:42.615615+0000","last_became_active":"2024-03-06T07:28:41.604346+0000","last_became_peered":"2024-03-06T07:28:41.604346+0000","last_unstale":"2024-03-06T07:28:42.615615+0000","last_undegraded":"2024-03-06T07:28:42.615615+0000","last_fullsized":"2024-03-06T07:28:42.615615+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:27.688649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614784+0000","last_change":"2024-03-06T07:28:41.604347+0000","last_active":"2024-03-06T07:28:42.614784+0000","last_peered":"2024-03-06T07:28:42.614784+0000","last_clean":"2024-03-06T07:28:42.614784+0000","last_became_active":"2024-03-06T07:28:41.604078+0000","last_became_peered":"2024-03-06T07:28:41.604078+0000","last_unstale":"2024-03-06T07:28:42.614784+0000","last_undegraded":"2024-03-06T07:28:42.614784+0000","last_fullsized":"2024-03-06T07:28:42.614784+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:12.021731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584403+0000","last_change":"2024-03-06T07:28:41.610412+0000","last_active":"2024-03-06T07:28:42.584403+0000","last_peered":"2024-03-06T07:28:42.584403+0000","last_clean":"2024-03-06T07:28:42.584403+0000","last_became_active":"2024-03-06T07:28:41.607523+0000","last_became_peered":"2024-03-06T07:28:41.607523+0000","last_unstale":"2024-03-06T07:28:42.584403+0000","last_undegraded":"2024-03-06T07:28:42.584403+0000","last_fullsized":"2024-03-06T07:28:42.584403+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:19.391795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584714+0000","last_change":"2024-03-06T07:28:41.592755+0000","last_active":"2024-03-06T07:28:42.584714+0000","last_peered":"2024-03-06T07:28:42.584714+0000","last_clean":"2024-03-06T07:28:42.584714+0000","last_became_active":"2024-03-06T07:28:41.592455+0000","last_became_peered":"2024-03-06T07:28:41.592455+0000","last_unstale":"2024-03-06T07:28:42.584714+0000","last_undegraded":"2024-03-06T07:28:42.584714+0000","last_fullsized":"2024-03-06T07:28:42.584714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:37.201552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615460+0000","last_change":"2024-03-06T07:28:41.602207+0000","last_active":"2024-03-06T07:28:42.615460+0000","last_peered":"2024-03-06T07:28:42.615460+0000","last_clean":"2024-03-06T07:28:42.615460+0000","last_became_active":"2024-03-06T07:28:41.601914+0000","last_became_peered":"2024-03-06T07:28:41.601914+0000","last_unstale":"2024-03-06T07:28:42.615460+0000","last_undegraded":"2024-03-06T07:28:42.615460+0000","last_fullsized":"2024-03-06T07:28:42.615460+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:40.520524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614940+0000","last_change":"2024-03-06T07:28:41.605636+0000","last_active":"2024-03-06T07:28:42.614940+0000","last_peered":"2024-03-06T07:28:42.614940+0000","last_clean":"2024-03-06T07:28:42.614940+0000","last_became_active":"2024-03-06T07:28:41.605101+0000","last_became_peered":"2024-03-06T07:28:41.605101+0000","last_unstale":"2024-03-06T07:28:42.614940+0000","last_undegraded":"2024-03-06T07:28:42.614940+0000","last_fullsized":"2024-03-06T07:28:42.614940+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:51.706297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614469+0000","last_change":"2024-03-06T07:28:41.592526+0000","last_active":"2024-03-06T07:28:42.614469+0000","last_peered":"2024-03-06T07:28:42.614469+0000","last_clean":"2024-03-06T07:28:42.614469+0000","last_became_active":"2024-03-06T07:28:41.592212+0000","last_became_peered":"2024-03-06T07:28:41.592212+0000","last_unstale":"2024-03-06T07:28:42.614469+0000","last_undegraded":"2024-03-06T07:28:42.614469+0000","last_fullsized":"2024-03-06T07:28:42.614469+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:22.548099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614673+0000","last_change":"2024-03-06T07:28:41.612948+0000","last_active":"2024-03-06T07:28:42.614673+0000","last_peered":"2024-03-06T07:28:42.614673+0000","last_clean":"2024-03-06T07:28:42.614673+0000","last_became_active":"2024-03-06T07:28:41.612634+0000","last_became_peered":"2024-03-06T07:28:41.612634+0000","last_unstale":"2024-03-06T07:28:42.614673+0000","last_undegraded":"2024-03-06T07:28:42.614673+0000","last_fullsized":"2024-03-06T07:28:42.614673+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:03.772883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615429+0000","last_change":"2024-03-06T07:28:40.591430+0000","last_active":"2024-03-06T07:28:42.615429+0000","last_peered":"2024-03-06T07:28:42.615429+0000","last_clean":"2024-03-06T07:28:42.615429+0000","last_became_active":"2024-03-06T07:28:40.590954+0000","last_became_peered":"2024-03-06T07:28:40.590954+0000","last_unstale":"2024-03-06T07:28:42.615429+0000","last_undegraded":"2024-03-06T07:28:42.615429+0000","last_fullsized":"2024-03-06T07:28:42.615429+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:41.774205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615738+0000","last_change":"2024-03-06T07:28:41.601165+0000","last_active":"2024-03-06T07:28:42.615738+0000","last_peered":"2024-03-06T07:28:42.615738+0000","last_clean":"2024-03-06T07:28:42.615738+0000","last_became_active":"2024-03-06T07:28:41.600834+0000","last_became_peered":"2024-03-06T07:28:41.600834+0000","last_unstale":"2024-03-06T07:28:42.615738+0000","last_undegraded":"2024-03-06T07:28:42.615738+0000","last_fullsized":"2024-03-06T07:28:42.615738+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:34.600777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616100+0000","last_change":"2024-03-06T07:28:40.592943+0000","last_active":"2024-03-06T07:28:42.616100+0000","last_peered":"2024-03-06T07:28:42.616100+0000","last_clean":"2024-03-06T07:28:42.616100+0000","last_became_active":"2024-03-06T07:28:40.592617+0000","last_became_peered":"2024-03-06T07:28:40.592617+0000","last_unstale":"2024-03-06T07:28:42.616100+0000","last_undegraded":"2024-03-06T07:28:42.616100+0000","last_fullsized":"2024-03-06T07:28:42.616100+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:35.643091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614680+0000","last_change":"2024-03-06T07:28:41.591740+0000","last_active":"2024-03-06T07:28:42.614680+0000","last_peered":"2024-03-06T07:28:42.614680+0000","last_clean":"2024-03-06T07:28:42.614680+0000","last_became_active":"2024-03-06T07:28:41.591478+0000","last_became_peered":"2024-03-06T07:28:41.591478+0000","last_unstale":"2024-03-06T07:28:42.614680+0000","last_undegraded":"2024-03-06T07:28:42.614680+0000","last_fullsized":"2024-03-06T07:28:42.614680+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:54.518946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585315+0000","last_change":"2024-03-06T07:28:40.601530+0000","last_active":"2024-03-06T07:28:42.585315+0000","last_peered":"2024-03-06T07:28:42.585315+0000","last_clean":"2024-03-06T07:28:42.585315+0000","last_became_active":"2024-03-06T07:28:40.601235+0000","last_became_peered":"2024-03-06T07:28:40.601235+0000","last_unstale":"2024-03-06T07:28:42.585315+0000","last_undegraded":"2024-03-06T07:28:42.585315+0000","last_fullsized":"2024-03-06T07:28:42.585315+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:10.155882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616345+0000","last_change":"2024-03-06T07:28:41.612533+0000","last_active":"2024-03-06T07:28:42.616345+0000","last_peered":"2024-03-06T07:28:42.616345+0000","last_clean":"2024-03-06T07:28:42.616345+0000","last_became_active":"2024-03-06T07:28:41.612195+0000","last_became_peered":"2024-03-06T07:28:41.612195+0000","last_unstale":"2024-03-06T07:28:42.616345+0000","last_undegraded":"2024-03-06T07:28:42.616345+0000","last_fullsized":"2024-03-06T07:28:42.616345+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:19.391795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585158+0000","last_change":"2024-03-06T07:28:40.601199+0000","last_active":"2024-03-06T07:28:42.585158+0000","last_peered":"2024-03-06T07:28:42.585158+0000","last_clean":"2024-03-06T07:28:42.585158+0000","last_became_active":"2024-03-06T07:28:40.600924+0000","last_became_peered":"2024-03-06T07:28:40.600924+0000","last_unstale":"2024-03-06T07:28:42.585158+0000","last_undegraded":"2024-03-06T07:28:42.585158+0000","last_fullsized":"2024-03-06T07:28:42.585158+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:51.604636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615023+0000","last_change":"2024-03-06T07:28:41.613286+0000","last_active":"2024-03-06T07:28:42.615023+0000","last_peered":"2024-03-06T07:28:42.615023+0000","last_clean":"2024-03-06T07:28:42.615023+0000","last_became_active":"2024-03-06T07:28:41.612678+0000","last_became_peered":"2024-03-06T07:28:41.612678+0000","last_unstale":"2024-03-06T07:28:42.615023+0000","last_undegraded":"2024-03-06T07:28:42.615023+0000","last_fullsized":"2024-03-06T07:28:42.615023+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:08.661708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615029+0000","last_change":"2024-03-06T07:28:40.596714+0000","last_active":"2024-03-06T07:28:42.615029+0000","last_peered":"2024-03-06T07:28:42.615029+0000","last_clean":"2024-03-06T07:28:42.615029+0000","last_became_active":"2024-03-06T07:28:40.596484+0000","last_became_peered":"2024-03-06T07:28:40.596484+0000","last_unstale":"2024-03-06T07:28:42.615029+0000","last_undegraded":"2024-03-06T07:28:42.615029+0000","last_fullsized":"2024-03-06T07:28:42.615029+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:33.815303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584885+0000","last_change":"2024-03-06T07:28:41.598081+0000","last_active":"2024-03-06T07:28:42.584885+0000","last_peered":"2024-03-06T07:28:42.584885+0000","last_clean":"2024-03-06T07:28:42.584885+0000","last_became_active":"2024-03-06T07:28:41.597755+0000","last_became_peered":"2024-03-06T07:28:41.597755+0000","last_unstale":"2024-03-06T07:28:42.584885+0000","last_undegraded":"2024-03-06T07:28:42.584885+0000","last_fullsized":"2024-03-06T07:28:42.584885+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:17.415589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584596+0000","last_change":"2024-03-06T07:28:40.593599+0000","last_active":"2024-03-06T07:28:42.584596+0000","last_peered":"2024-03-06T07:28:42.584596+0000","last_clean":"2024-03-06T07:28:42.584596+0000","last_became_active":"2024-03-06T07:28:40.593318+0000","last_became_peered":"2024-03-06T07:28:40.593318+0000","last_unstale":"2024-03-06T07:28:42.584596+0000","last_undegraded":"2024-03-06T07:28:42.584596+0000","last_fullsized":"2024-03-06T07:28:42.584596+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:18.176638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615176+0000","last_change":"2024-03-06T07:28:41.611042+0000","last_active":"2024-03-06T07:28:42.615176+0000","last_peered":"2024-03-06T07:28:42.615176+0000","last_clean":"2024-03-06T07:28:42.615176+0000","last_became_active":"2024-03-06T07:28:41.610758+0000","last_became_peered":"2024-03-06T07:28:41.610758+0000","last_unstale":"2024-03-06T07:28:42.615176+0000","last_undegraded":"2024-03-06T07:28:42.615176+0000","last_fullsized":"2024-03-06T07:28:42.615176+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:22.849251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584484+0000","last_change":"2024-03-06T07:28:40.587951+0000","last_active":"2024-03-06T07:28:42.584484+0000","last_peered":"2024-03-06T07:28:42.584484+0000","last_clean":"2024-03-06T07:28:42.584484+0000","last_became_active":"2024-03-06T07:28:40.587671+0000","last_became_peered":"2024-03-06T07:28:40.587671+0000","last_unstale":"2024-03-06T07:28:42.584484+0000","last_undegraded":"2024-03-06T07:28:42.584484+0000","last_fullsized":"2024-03-06T07:28:42.584484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:06.481910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615021+0000","last_change":"2024-03-06T07:28:41.602157+0000","last_active":"2024-03-06T07:28:42.615021+0000","last_peered":"2024-03-06T07:28:42.615021+0000","last_clean":"2024-03-06T07:28:42.615021+0000","last_became_active":"2024-03-06T07:28:41.601862+0000","last_became_peered":"2024-03-06T07:28:41.601862+0000","last_unstale":"2024-03-06T07:28:42.615021+0000","last_undegraded":"2024-03-06T07:28:42.615021+0000","last_fullsized":"2024-03-06T07:28:42.615021+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:28.477340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615227+0000","last_change":"2024-03-06T07:28:40.590486+0000","last_active":"2024-03-06T07:28:42.615227+0000","last_peered":"2024-03-06T07:28:42.615227+0000","last_clean":"2024-03-06T07:28:42.615227+0000","last_became_active":"2024-03-06T07:28:40.590113+0000","last_became_peered":"2024-03-06T07:28:40.590113+0000","last_unstale":"2024-03-06T07:28:42.615227+0000","last_undegraded":"2024-03-06T07:28:42.615227+0000","last_fullsized":"2024-03-06T07:28:42.615227+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:35.804062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615546+0000","last_change":"2024-03-06T07:28:41.613036+0000","last_active":"2024-03-06T07:28:42.615546+0000","last_peered":"2024-03-06T07:28:42.615546+0000","last_clean":"2024-03-06T07:28:42.615546+0000","last_became_active":"2024-03-06T07:28:41.612725+0000","last_became_peered":"2024-03-06T07:28:41.612725+0000","last_unstale":"2024-03-06T07:28:42.615546+0000","last_undegraded":"2024-03-06T07:28:42.615546+0000","last_fullsized":"2024-03-06T07:28:42.615546+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:20.966815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585400+0000","last_change":"2024-03-06T07:28:40.600729+0000","last_active":"2024-03-06T07:28:42.585400+0000","last_peered":"2024-03-06T07:28:42.585400+0000","last_clean":"2024-03-06T07:28:42.585400+0000","last_became_active":"2024-03-06T07:28:40.600441+0000","last_became_peered":"2024-03-06T07:28:40.600441+0000","last_unstale":"2024-03-06T07:28:42.585400+0000","last_undegraded":"2024-03-06T07:28:42.585400+0000","last_fullsized":"2024-03-06T07:28:42.585400+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:40.818375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615866+0000","last_change":"2024-03-06T07:28:41.606308+0000","last_active":"2024-03-06T07:28:42.615866+0000","last_peered":"2024-03-06T07:28:42.615866+0000","last_clean":"2024-03-06T07:28:42.615866+0000","last_became_active":"2024-03-06T07:28:41.605734+0000","last_became_peered":"2024-03-06T07:28:41.605734+0000","last_unstale":"2024-03-06T07:28:42.615866+0000","last_undegraded":"2024-03-06T07:28:42.615866+0000","last_fullsized":"2024-03-06T07:28:42.615866+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:18.073915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584417+0000","last_change":"2024-03-06T07:28:40.592798+0000","last_active":"2024-03-06T07:28:42.584417+0000","last_peered":"2024-03-06T07:28:42.584417+0000","last_clean":"2024-03-06T07:28:42.584417+0000","last_became_active":"2024-03-06T07:28:40.592483+0000","last_became_peered":"2024-03-06T07:28:40.592483+0000","last_unstale":"2024-03-06T07:28:42.584417+0000","last_undegraded":"2024-03-06T07:28:42.584417+0000","last_fullsized":"2024-03-06T07:28:42.584417+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:57.654186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584539+0000","last_change":"2024-03-06T07:28:41.606477+0000","last_active":"2024-03-06T07:28:42.584539+0000","last_peered":"2024-03-06T07:28:42.584539+0000","last_clean":"2024-03-06T07:28:42.584539+0000","last_became_active":"2024-03-06T07:28:41.606142+0000","last_became_peered":"2024-03-06T07:28:41.606142+0000","last_unstale":"2024-03-06T07:28:42.584539+0000","last_undegraded":"2024-03-06T07:28:42.584539+0000","last_fullsized":"2024-03-06T07:28:42.584539+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:03.190401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":17,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615153+0000","last_change":"2024-03-06T07:28:40.592800+0000","last_active":"2024-03-06T07:28:42.615153+0000","last_peered":"2024-03-06T07:28:42.615153+0000","last_clean":"2024-03-06T07:28:42.615153+0000","last_became_active":"2024-03-06T07:28:40.592525+0000","last_became_peered":"2024-03-06T07:28:40.592525+0000","last_unstale":"2024-03-06T07:28:42.615153+0000","last_undegraded":"2024-03-06T07:28:42.615153+0000","last_fullsized":"2024-03-06T07:28:42.615153+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:23.047792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615295+0000","last_change":"2024-03-06T07:28:41.602882+0000","last_active":"2024-03-06T07:28:42.615295+0000","last_peered":"2024-03-06T07:28:42.615295+0000","last_clean":"2024-03-06T07:28:42.615295+0000","last_became_active":"2024-03-06T07:28:41.602571+0000","last_became_peered":"2024-03-06T07:28:41.602571+0000","last_unstale":"2024-03-06T07:28:42.615295+0000","last_undegraded":"2024-03-06T07:28:42.615295+0000","last_fullsized":"2024-03-06T07:28:42.615295+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:28.691437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":20,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615479+0000","last_change":"2024-03-06T07:28:40.589011+0000","last_active":"2024-03-06T07:28:42.615479+0000","last_peered":"2024-03-06T07:28:42.615479+0000","last_clean":"2024-03-06T07:28:42.615479+0000","last_became_active":"2024-03-06T07:28:40.588731+0000","last_became_peered":"2024-03-06T07:28:40.588731+0000","last_unstale":"2024-03-06T07:28:42.615479+0000","last_undegraded":"2024-03-06T07:28:42.615479+0000","last_fullsized":"2024-03-06T07:28:42.615479+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:06.929357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584676+0000","last_change":"2024-03-06T07:28:41.603914+0000","last_active":"2024-03-06T07:28:42.584676+0000","last_peered":"2024-03-06T07:28:42.584676+0000","last_clean":"2024-03-06T07:28:42.584676+0000","last_became_active":"2024-03-06T07:28:41.603566+0000","last_became_peered":"2024-03-06T07:28:41.603566+0000","last_unstale":"2024-03-06T07:28:42.584676+0000","last_undegraded":"2024-03-06T07:28:42.584676+0000","last_fullsized":"2024-03-06T07:28:42.584676+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:24.371626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614486+0000","last_change":"2024-03-06T07:28:40.596849+0000","last_active":"2024-03-06T07:28:42.614486+0000","last_peered":"2024-03-06T07:28:42.614486+0000","last_clean":"2024-03-06T07:28:42.614486+0000","last_became_active":"2024-03-06T07:28:40.596315+0000","last_became_peered":"2024-03-06T07:28:40.596315+0000","last_unstale":"2024-03-06T07:28:42.614486+0000","last_undegraded":"2024-03-06T07:28:42.614486+0000","last_fullsized":"2024-03-06T07:28:42.614486+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:04.152803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584355+0000","last_change":"2024-03-06T07:28:41.593085+0000","last_active":"2024-03-06T07:28:42.584355+0000","last_peered":"2024-03-06T07:28:42.584355+0000","last_clean":"2024-03-06T07:28:42.584355+0000","last_became_active":"2024-03-06T07:28:41.592769+0000","last_became_peered":"2024-03-06T07:28:41.592769+0000","last_unstale":"2024-03-06T07:28:42.584355+0000","last_undegraded":"2024-03-06T07:28:42.584355+0000","last_fullsized":"2024-03-06T07:28:42.584355+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:55.787493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615402+0000","last_change":"2024-03-06T07:28:40.595680+0000","last_active":"2024-03-06T07:28:42.615402+0000","last_peered":"2024-03-06T07:28:42.615402+0000","last_clean":"2024-03-06T07:28:42.615402+0000","last_became_active":"2024-03-06T07:28:40.595437+0000","last_became_peered":"2024-03-06T07:28:40.595437+0000","last_unstale":"2024-03-06T07:28:42.615402+0000","last_undegraded":"2024-03-06T07:28:42.615402+0000","last_fullsized":"2024-03-06T07:28:42.615402+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:20.107356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584232+0000","last_change":"2024-03-06T07:28:41.604230+0000","last_active":"2024-03-06T07:28:42.584232+0000","last_peered":"2024-03-06T07:28:42.584232+0000","last_clean":"2024-03-06T07:28:42.584232+0000","last_became_active":"2024-03-06T07:28:41.602845+0000","last_became_peered":"2024-03-06T07:28:41.602845+0000","last_unstale":"2024-03-06T07:28:42.584232+0000","last_undegraded":"2024-03-06T07:28:42.584232+0000","last_fullsized":"2024-03-06T07:28:42.584232+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.995900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":22,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584998+0000","last_change":"2024-03-06T07:28:40.600673+0000","last_active":"2024-03-06T07:28:42.584998+0000","last_peered":"2024-03-06T07:28:42.584998+0000","last_clean":"2024-03-06T07:28:42.584998+0000","last_became_active":"2024-03-06T07:28:40.600373+0000","last_became_peered":"2024-03-06T07:28:40.600373+0000","last_unstale":"2024-03-06T07:28:42.584998+0000","last_undegraded":"2024-03-06T07:28:42.584998+0000","last_fullsized":"2024-03-06T07:28:42.584998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:03.969324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585624+0000","last_change":"2024-03-06T07:28:41.610614+0000","last_active":"2024-03-06T07:28:42.585624+0000","last_peered":"2024-03-06T07:28:42.585624+0000","last_clean":"2024-03-06T07:28:42.585624+0000","last_became_active":"2024-03-06T07:28:41.610307+0000","last_became_peered":"2024-03-06T07:28:41.610307+0000","last_unstale":"2024-03-06T07:28:42.585624+0000","last_undegraded":"2024-03-06T07:28:42.585624+0000","last_fullsized":"2024-03-06T07:28:42.585624+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:34.817395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615670+0000","last_change":"2024-03-06T07:28:40.591386+0000","last_active":"2024-03-06T07:28:42.615670+0000","last_peered":"2024-03-06T07:28:42.615670+0000","last_clean":"2024-03-06T07:28:42.615670+0000","last_became_active":"2024-03-06T07:28:40.591132+0000","last_became_peered":"2024-03-06T07:28:40.591132+0000","last_unstale":"2024-03-06T07:28:42.615670+0000","last_undegraded":"2024-03-06T07:28:42.615670+0000","last_fullsized":"2024-03-06T07:28:42.615670+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:54.835777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614664+0000","last_change":"2024-03-06T07:28:41.605192+0000","last_active":"2024-03-06T07:28:42.614664+0000","last_peered":"2024-03-06T07:28:42.614664+0000","last_clean":"2024-03-06T07:28:42.614664+0000","last_became_active":"2024-03-06T07:28:41.604812+0000","last_became_peered":"2024-03-06T07:28:41.604812+0000","last_unstale":"2024-03-06T07:28:42.614664+0000","last_undegraded":"2024-03-06T07:28:42.614664+0000","last_fullsized":"2024-03-06T07:28:42.614664+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:03.509633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585603+0000","last_change":"2024-03-06T07:28:40.598436+0000","last_active":"2024-03-06T07:28:42.585603+0000","last_peered":"2024-03-06T07:28:42.585603+0000","last_clean":"2024-03-06T07:28:42.585603+0000","last_became_active":"2024-03-06T07:28:40.598075+0000","last_became_peered":"2024-03-06T07:28:40.598075+0000","last_unstale":"2024-03-06T07:28:42.585603+0000","last_undegraded":"2024-03-06T07:28:42.585603+0000","last_fullsized":"2024-03-06T07:28:42.585603+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:21.546007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616566+0000","last_change":"2024-03-06T07:28:41.603468+0000","last_active":"2024-03-06T07:28:42.616566+0000","last_peered":"2024-03-06T07:28:42.616566+0000","last_clean":"2024-03-06T07:28:42.616566+0000","last_became_active":"2024-03-06T07:28:41.603081+0000","last_became_peered":"2024-03-06T07:28:41.603081+0000","last_unstale":"2024-03-06T07:28:42.616566+0000","last_undegraded":"2024-03-06T07:28:42.616566+0000","last_fullsized":"2024-03-06T07:28:42.616566+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:03.190401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615332+0000","last_change":"2024-03-06T07:28:40.594727+0000","last_active":"2024-03-06T07:28:42.615332+0000","last_peered":"2024-03-06T07:28:42.615332+0000","last_clean":"2024-03-06T07:28:42.615332+0000","last_became_active":"2024-03-06T07:28:40.594455+0000","last_became_peered":"2024-03-06T07:28:40.594455+0000","last_unstale":"2024-03-06T07:28:42.615332+0000","last_undegraded":"2024-03-06T07:28:42.615332+0000","last_fullsized":"2024-03-06T07:28:42.615332+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:36.199459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616429+0000","last_change":"2024-03-06T07:28:41.605423+0000","last_active":"2024-03-06T07:28:42.616429+0000","last_peered":"2024-03-06T07:28:42.616429+0000","last_clean":"2024-03-06T07:28:42.616429+0000","last_became_active":"2024-03-06T07:28:41.604962+0000","last_became_peered":"2024-03-06T07:28:41.604962+0000","last_unstale":"2024-03-06T07:28:42.616429+0000","last_undegraded":"2024-03-06T07:28:42.616429+0000","last_fullsized":"2024-03-06T07:28:42.616429+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:56.356422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614955+0000","last_change":"2024-03-06T07:28:40.598996+0000","last_active":"2024-03-06T07:28:42.614955+0000","last_peered":"2024-03-06T07:28:42.614955+0000","last_clean":"2024-03-06T07:28:42.614955+0000","last_became_active":"2024-03-06T07:28:40.598707+0000","last_became_peered":"2024-03-06T07:28:40.598707+0000","last_unstale":"2024-03-06T07:28:42.614955+0000","last_undegraded":"2024-03-06T07:28:42.614955+0000","last_fullsized":"2024-03-06T07:28:42.614955+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:53.516854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616329+0000","last_change":"2024-03-06T07:28:41.602357+0000","last_active":"2024-03-06T07:28:42.616329+0000","last_peered":"2024-03-06T07:28:42.616329+0000","last_clean":"2024-03-06T07:28:42.616329+0000","last_became_active":"2024-03-06T07:28:41.602093+0000","last_became_peered":"2024-03-06T07:28:41.602093+0000","last_unstale":"2024-03-06T07:28:42.616329+0000","last_undegraded":"2024-03-06T07:28:42.616329+0000","last_fullsized":"2024-03-06T07:28:42.616329+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:55.787493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614653+0000","last_change":"2024-03-06T07:28:40.600309+0000","last_active":"2024-03-06T07:28:42.614653+0000","last_peered":"2024-03-06T07:28:42.614653+0000","last_clean":"2024-03-06T07:28:42.614653+0000","last_became_active":"2024-03-06T07:28:40.599976+0000","last_became_peered":"2024-03-06T07:28:40.599976+0000","last_unstale":"2024-03-06T07:28:42.614653+0000","last_undegraded":"2024-03-06T07:28:42.614653+0000","last_fullsized":"2024-03-06T07:28:42.614653+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:42.371056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615062+0000","last_change":"2024-03-06T07:28:41.599242+0000","last_active":"2024-03-06T07:28:42.615062+0000","last_peered":"2024-03-06T07:28:42.615062+0000","last_clean":"2024-03-06T07:28:42.615062+0000","last_became_active":"2024-03-06T07:28:41.598920+0000","last_became_peered":"2024-03-06T07:28:41.598920+0000","last_unstale":"2024-03-06T07:28:42.615062+0000","last_undegraded":"2024-03-06T07:28:42.615062+0000","last_fullsized":"2024-03-06T07:28:42.615062+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:51.489658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585293+0000","last_change":"2024-03-06T07:28:40.601354+0000","last_active":"2024-03-06T07:28:42.585293+0000","last_peered":"2024-03-06T07:28:42.585293+0000","last_clean":"2024-03-06T07:28:42.585293+0000","last_became_active":"2024-03-06T07:28:40.601113+0000","last_became_peered":"2024-03-06T07:28:40.601113+0000","last_unstale":"2024-03-06T07:28:42.585293+0000","last_undegraded":"2024-03-06T07:28:42.585293+0000","last_fullsized":"2024-03-06T07:28:42.585293+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:35.874968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615717+0000","last_change":"2024-03-06T07:28:41.612198+0000","last_active":"2024-03-06T07:28:42.615717+0000","last_peered":"2024-03-06T07:28:42.615717+0000","last_clean":"2024-03-06T07:28:42.615717+0000","last_became_active":"2024-03-06T07:28:41.611905+0000","last_became_peered":"2024-03-06T07:28:41.611905+0000","last_unstale":"2024-03-06T07:28:42.615717+0000","last_undegraded":"2024-03-06T07:28:42.615717+0000","last_fullsized":"2024-03-06T07:28:42.615717+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:24.049884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614539+0000","last_change":"2024-03-06T07:28:40.599514+0000","last_active":"2024-03-06T07:28:42.614539+0000","last_peered":"2024-03-06T07:28:42.614539+0000","last_clean":"2024-03-06T07:28:42.614539+0000","last_became_active":"2024-03-06T07:28:40.599217+0000","last_became_peered":"2024-03-06T07:28:40.599217+0000","last_unstale":"2024-03-06T07:28:42.614539+0000","last_undegraded":"2024-03-06T07:28:42.614539+0000","last_fullsized":"2024-03-06T07:28:42.614539+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:47.022153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584335+0000","last_change":"2024-03-06T07:28:41.607422+0000","last_active":"2024-03-06T07:28:42.584335+0000","last_peered":"2024-03-06T07:28:42.584335+0000","last_clean":"2024-03-06T07:28:42.584335+0000","last_became_active":"2024-03-06T07:28:41.607072+0000","last_became_peered":"2024-03-06T07:28:41.607072+0000","last_unstale":"2024-03-06T07:28:42.584335+0000","last_undegraded":"2024-03-06T07:28:42.584335+0000","last_fullsized":"2024-03-06T07:28:42.584335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:05.095081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615350+0000","last_change":"2024-03-06T07:28:40.596316+0000","last_active":"2024-03-06T07:28:42.615350+0000","last_peered":"2024-03-06T07:28:42.615350+0000","last_clean":"2024-03-06T07:28:42.615350+0000","last_became_active":"2024-03-06T07:28:40.595681+0000","last_became_peered":"2024-03-06T07:28:40.595681+0000","last_unstale":"2024-03-06T07:28:42.615350+0000","last_undegraded":"2024-03-06T07:28:42.615350+0000","last_fullsized":"2024-03-06T07:28:42.615350+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:58.815160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616022+0000","last_change":"2024-03-06T07:28:41.605871+0000","last_active":"2024-03-06T07:28:42.616022+0000","last_peered":"2024-03-06T07:28:42.616022+0000","last_clean":"2024-03-06T07:28:42.616022+0000","last_became_active":"2024-03-06T07:28:41.605577+0000","last_became_peered":"2024-03-06T07:28:41.605577+0000","last_unstale":"2024-03-06T07:28:42.616022+0000","last_undegraded":"2024-03-06T07:28:42.616022+0000","last_fullsized":"2024-03-06T07:28:42.616022+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:36.806154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584550+0000","last_change":"2024-03-06T07:28:40.590129+0000","last_active":"2024-03-06T07:28:42.584550+0000","last_peered":"2024-03-06T07:28:42.584550+0000","last_clean":"2024-03-06T07:28:42.584550+0000","last_became_active":"2024-03-06T07:28:40.589864+0000","last_became_peered":"2024-03-06T07:28:40.589864+0000","last_unstale":"2024-03-06T07:28:42.584550+0000","last_undegraded":"2024-03-06T07:28:42.584550+0000","last_fullsized":"2024-03-06T07:28:42.584550+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:39.352210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615655+0000","last_change":"2024-03-06T07:28:41.607050+0000","last_active":"2024-03-06T07:28:42.615655+0000","last_peered":"2024-03-06T07:28:42.615655+0000","last_clean":"2024-03-06T07:28:42.615655+0000","last_became_active":"2024-03-06T07:28:41.606639+0000","last_became_peered":"2024-03-06T07:28:41.606639+0000","last_unstale":"2024-03-06T07:28:42.615655+0000","last_undegraded":"2024-03-06T07:28:42.615655+0000","last_fullsized":"2024-03-06T07:28:42.615655+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:09.115278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"114'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616105+0000","last_change":"2024-03-06T07:28:40.598310+0000","last_active":"2024-03-06T07:28:42.616105+0000","last_peered":"2024-03-06T07:28:42.616105+0000","last_clean":"2024-03-06T07:28:42.616105+0000","last_became_active":"2024-03-06T07:28:40.597952+0000","last_became_peered":"2024-03-06T07:28:40.597952+0000","last_unstale":"2024-03-06T07:28:42.616105+0000","last_undegraded":"2024-03-06T07:28:42.616105+0000","last_fullsized":"2024-03-06T07:28:42.616105+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:11.242574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615326+0000","last_change":"2024-03-06T07:28:41.600904+0000","last_active":"2024-03-06T07:28:42.615326+0000","last_peered":"2024-03-06T07:28:42.615326+0000","last_clean":"2024-03-06T07:28:42.615326+0000","last_became_active":"2024-03-06T07:28:41.600603+0000","last_became_peered":"2024-03-06T07:28:41.600603+0000","last_unstale":"2024-03-06T07:28:42.615326+0000","last_undegraded":"2024-03-06T07:28:42.615326+0000","last_fullsized":"2024-03-06T07:28:42.615326+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:45.519515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615204+0000","last_change":"2024-03-06T07:28:40.595293+0000","last_active":"2024-03-06T07:28:42.615204+0000","last_peered":"2024-03-06T07:28:42.615204+0000","last_clean":"2024-03-06T07:28:42.615204+0000","last_became_active":"2024-03-06T07:28:40.595028+0000","last_became_peered":"2024-03-06T07:28:40.595028+0000","last_unstale":"2024-03-06T07:28:42.615204+0000","last_undegraded":"2024-03-06T07:28:42.615204+0000","last_fullsized":"2024-03-06T07:28:42.615204+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:15.972492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615314+0000","last_change":"2024-03-06T07:28:41.603184+0000","last_active":"2024-03-06T07:28:42.615314+0000","last_peered":"2024-03-06T07:28:42.615314+0000","last_clean":"2024-03-06T07:28:42.615314+0000","last_became_active":"2024-03-06T07:28:41.602913+0000","last_became_peered":"2024-03-06T07:28:41.602913+0000","last_unstale":"2024-03-06T07:28:42.615314+0000","last_undegraded":"2024-03-06T07:28:42.615314+0000","last_fullsized":"2024-03-06T07:28:42.615314+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:12.604285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584462+0000","last_change":"2024-03-06T07:28:40.593167+0000","last_active":"2024-03-06T07:28:42.584462+0000","last_peered":"2024-03-06T07:28:42.584462+0000","last_clean":"2024-03-06T07:28:42.584462+0000","last_became_active":"2024-03-06T07:28:40.592882+0000","last_became_peered":"2024-03-06T07:28:40.592882+0000","last_unstale":"2024-03-06T07:28:42.584462+0000","last_undegraded":"2024-03-06T07:28:42.584462+0000","last_fullsized":"2024-03-06T07:28:42.584462+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:41.378581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584514+0000","last_change":"2024-03-06T07:28:41.603668+0000","last_active":"2024-03-06T07:28:42.584514+0000","last_peered":"2024-03-06T07:28:42.584514+0000","last_clean":"2024-03-06T07:28:42.584514+0000","last_became_active":"2024-03-06T07:28:41.603374+0000","last_became_peered":"2024-03-06T07:28:41.603374+0000","last_unstale":"2024-03-06T07:28:42.584514+0000","last_undegraded":"2024-03-06T07:28:42.584514+0000","last_fullsized":"2024-03-06T07:28:42.584514+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:12.244667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615144+0000","last_change":"2024-03-06T07:28:40.593051+0000","last_active":"2024-03-06T07:28:42.615144+0000","last_peered":"2024-03-06T07:28:42.615144+0000","last_clean":"2024-03-06T07:28:42.615144+0000","last_became_active":"2024-03-06T07:28:40.592727+0000","last_became_peered":"2024-03-06T07:28:40.592727+0000","last_unstale":"2024-03-06T07:28:42.615144+0000","last_undegraded":"2024-03-06T07:28:42.615144+0000","last_fullsized":"2024-03-06T07:28:42.615144+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:45.038458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584929+0000","last_change":"2024-03-06T07:28:41.612638+0000","last_active":"2024-03-06T07:28:42.584929+0000","last_peered":"2024-03-06T07:28:42.584929+0000","last_clean":"2024-03-06T07:28:42.584929+0000","last_became_active":"2024-03-06T07:28:41.612281+0000","last_became_peered":"2024-03-06T07:28:41.612281+0000","last_unstale":"2024-03-06T07:28:42.584929+0000","last_undegraded":"2024-03-06T07:28:42.584929+0000","last_fullsized":"2024-03-06T07:28:42.584929+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.656279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584484+0000","last_change":"2024-03-06T07:28:40.593507+0000","last_active":"2024-03-06T07:28:42.584484+0000","last_peered":"2024-03-06T07:28:42.584484+0000","last_clean":"2024-03-06T07:28:42.584484+0000","last_became_active":"2024-03-06T07:28:40.593219+0000","last_became_peered":"2024-03-06T07:28:40.593219+0000","last_unstale":"2024-03-06T07:28:42.584484+0000","last_undegraded":"2024-03-06T07:28:42.584484+0000","last_fullsized":"2024-03-06T07:28:42.584484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:27.475248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615978+0000","last_change":"2024-03-06T07:28:41.598292+0000","last_active":"2024-03-06T07:28:42.615978+0000","last_peered":"2024-03-06T07:28:42.615978+0000","last_clean":"2024-03-06T07:28:42.615978+0000","last_became_active":"2024-03-06T07:28:41.598002+0000","last_became_peered":"2024-03-06T07:28:41.598002+0000","last_unstale":"2024-03-06T07:28:42.615978+0000","last_undegraded":"2024-03-06T07:28:42.615978+0000","last_fullsized":"2024-03-06T07:28:42.615978+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:16.974584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585079+0000","last_change":"2024-03-06T07:28:40.597305+0000","last_active":"2024-03-06T07:28:42.585079+0000","last_peered":"2024-03-06T07:28:42.585079+0000","last_clean":"2024-03-06T07:28:42.585079+0000","last_became_active":"2024-03-06T07:28:40.596994+0000","last_became_peered":"2024-03-06T07:28:40.596994+0000","last_unstale":"2024-03-06T07:28:42.585079+0000","last_undegraded":"2024-03-06T07:28:42.585079+0000","last_fullsized":"2024-03-06T07:28:42.585079+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:40.886362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614978+0000","last_change":"2024-03-06T07:28:41.605397+0000","last_active":"2024-03-06T07:28:42.614978+0000","last_peered":"2024-03-06T07:28:42.614978+0000","last_clean":"2024-03-06T07:28:42.614978+0000","last_became_active":"2024-03-06T07:28:41.604725+0000","last_became_peered":"2024-03-06T07:28:41.604725+0000","last_unstale":"2024-03-06T07:28:42.614978+0000","last_undegraded":"2024-03-06T07:28:42.614978+0000","last_fullsized":"2024-03-06T07:28:42.614978+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:21.039498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615436+0000","last_change":"2024-03-06T07:28:40.596580+0000","last_active":"2024-03-06T07:28:42.615436+0000","last_peered":"2024-03-06T07:28:42.615436+0000","last_clean":"2024-03-06T07:28:42.615436+0000","last_became_active":"2024-03-06T07:28:40.596067+0000","last_became_peered":"2024-03-06T07:28:40.596067+0000","last_unstale":"2024-03-06T07:28:42.615436+0000","last_undegraded":"2024-03-06T07:28:42.615436+0000","last_fullsized":"2024-03-06T07:28:42.615436+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:49.059409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584162+0000","last_change":"2024-03-06T07:28:41.593409+0000","last_active":"2024-03-06T07:28:42.584162+0000","last_peered":"2024-03-06T07:28:42.584162+0000","last_clean":"2024-03-06T07:28:42.584162+0000","last_became_active":"2024-03-06T07:28:41.592822+0000","last_became_peered":"2024-03-06T07:28:41.592822+0000","last_unstale":"2024-03-06T07:28:42.584162+0000","last_undegraded":"2024-03-06T07:28:42.584162+0000","last_fullsized":"2024-03-06T07:28:42.584162+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:55.837869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615076+0000","last_change":"2024-03-06T07:28:40.598826+0000","last_active":"2024-03-06T07:28:42.615076+0000","last_peered":"2024-03-06T07:28:42.615076+0000","last_clean":"2024-03-06T07:28:42.615076+0000","last_became_active":"2024-03-06T07:28:40.598534+0000","last_became_peered":"2024-03-06T07:28:40.598534+0000","last_unstale":"2024-03-06T07:28:42.615076+0000","last_undegraded":"2024-03-06T07:28:42.615076+0000","last_fullsized":"2024-03-06T07:28:42.615076+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:11.019639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":348,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615864+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:28:42.615864+0000","last_peered":"2024-03-06T07:28:42.615864+0000","last_clean":"2024-03-06T07:28:42.615864+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:28:42.615864+0000","last_undegraded":"2024-03-06T07:28:42.615864+0000","last_fullsized":"2024-03-06T07:28:42.615864+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37316,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334524,"statfs":{"total":96636764160,"available":96598552576,"internally_reserved":0,"allocated":2019328,"data_stored":1201971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574926,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40896,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330944,"statfs":{"total":96636764160,"available":96594886656,"internally_reserved":0,"allocated":1421312,"data_stored":611057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40900,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330940,"statfs":{"total":96636764160,"available":96594882560,"internally_reserved":0,"allocated":1425408,"data_stored":611591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40888,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330952,"statfs":{"total":96636764160,"available":96594894848,"internally_reserved":0,"allocated":1417216,"data_stored":610987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574925,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40908,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330932,"statfs":{"total":96636764160,"available":96594874368,"internally_reserved":0,"allocated":1433600,"data_stored":612669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40892,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330948,"statfs":{"total":96636764160,"available":96594890752,"internally_reserved":0,"allocated":1417216,"data_stored":610987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40900,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330940,"statfs":{"total":96636764160,"available":96594882560,"internally_reserved":0,"allocated":1425408,"data_stored":611111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41488,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330352,"statfs":{"total":96636764160,"available":96594280448,"internally_reserved":0,"allocated":2031616,"data_stored":1203079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:47.237 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:28:47.234+0000 7f9dcf52f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:48.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:49.000 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:50.132 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:50.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.237+0000 7fb202f1e640 1 Processor -- start 2024-03-06T07:28:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.237+0000 7fb202f1e640 1 -- start start 2024-03-06T07:28:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb202f1e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1fc071540 0x7fb1fc0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb202f1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0d3400 0x7fb1fc071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb202f1e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc071e60 con 0x7fb1fc0b2b50 2024-03-06T07:28:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb202f1e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1fc0d2f70 con 0x7fb1fc071540 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb202f1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0d4be0 con 0x7fb1fc0d3400 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb200cba640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1fc071540 0x7fb1fc0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb200cba640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1fc071540 0x7fb1fc0b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44658/0 (socket says 172.21.15.27:44658) 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb200cba640 1 -- 172.21.15.27:0/3126212757 learned_addr learned my addr 172.21.15.27:0/3126212757 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb1fbfff640 1 --2- 172.21.15.27:0/3126212757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0d3400 0x7fb1fc071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb200cba640 1 -- 172.21.15.27:0/3126212757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0d3400 msgr2=0x7fb1fc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb200cba640 1 --2- 172.21.15.27:0/3126212757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0d3400 0x7fb1fc071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb200cba640 1 -- 172.21.15.27:0/3126212757 >> v1:172.21.15.69:6789/0 conn(0x7fb1fc0b2b50 legacy=0x7fb1fc0d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.238+0000 7fb200cba640 1 -- 172.21.15.27:0/3126212757 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc0ce610 con 0x7fb1fc071540 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.239+0000 7fb200cba640 1 --2- 172.21.15.27:0/3126212757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc071540 0x7fb1fc0b30a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009870 tx=0x7fb1f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8350ce984f495d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.239+0000 7fb1fb7fe640 1 -- 172.21.15.27:0/3126212757 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0035020 con 0x7fb1fc071540 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.239+0000 7fb1fb7fe640 1 -- 172.21.15.27:0/3126212757 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f0031640 con 0x7fb1fc071540 2024-03-06T07:28:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.239+0000 7fb1fb7fe640 1 -- 172.21.15.27:0/3126212757 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f002d040 con 0x7fb1fc071540 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.240+0000 7fb202f1e640 1 -- 172.21.15.27:0/3126212757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc071540 msgr2=0x7fb1fc0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.240+0000 7fb202f1e640 1 --2- 172.21.15.27:0/3126212757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc071540 0x7fb1fc0b30a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009870 tx=0x7fb1f002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.240+0000 7fb202f1e640 1 -- 172.21.15.27:0/3126212757 shutdown_connections 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.240+0000 7fb202f1e640 1 --2- 172.21.15.27:0/3126212757 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0d3400 0x7fb1fc071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.240+0000 7fb202f1e640 1 --2- 172.21.15.27:0/3126212757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc071540 0x7fb1fc0b30a0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.240+0000 7fb202f1e640 1 -- 172.21.15.27:0/3126212757 >> 172.21.15.27:0/3126212757 conn(0x7fb1fc0d3ca0 msgr2=0x7fb1fc0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.240+0000 7fb202f1e640 1 -- 172.21.15.27:0/3126212757 shutdown_connections 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.240+0000 7fb202f1e640 1 -- 172.21.15.27:0/3126212757 wait complete. 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.241+0000 7fb202f1e640 1 Processor -- start 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.241+0000 7fb202f1e640 1 -- start start 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb202f1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc071540 0x7fb1fc15cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb202f1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0b2b50 0x7fb1fc0ab030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb202f1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc0d3400 0x7fb1fc1f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb202f1e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0ce8f0 con 0x7fb1fc0d3400 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb202f1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0d2c00 con 0x7fb1fc071540 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb202f1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc15e370 con 0x7fb1fc0b2b50 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb2014bb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc0d3400 0x7fb1fc1f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb2014bb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc0d3400 0x7fb1fc1f4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44664/0 (socket says 172.21.15.27:44664) 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb2014bb640 1 -- 172.21.15.27:0/2655131931 learned_addr learned my addr 172.21.15.27:0/2655131931 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb1fbfff640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0b2b50 0x7fb1fc0ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb200cba640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc071540 0x7fb1fc15cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb2014bb640 1 -- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0b2b50 msgr2=0x7fb1fc0ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb2014bb640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0b2b50 0x7fb1fc0ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb2014bb640 1 -- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc071540 msgr2=0x7fb1fc15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb2014bb640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc071540 0x7fb1fc15cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb2014bb640 1 -- 172.21.15.27:0/2655131931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc0ac7d0 con 0x7fb1fc0d3400 2024-03-06T07:28:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.242+0000 7fb2014bb640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc0d3400 0x7fb1fc1f4220 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400c7b0 tx=0x7fb1f400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.243+0000 7fb1f97fa640 1 -- 172.21.15.27:0/2655131931 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4004b60 con 0x7fb1fc0d3400 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.243+0000 7fb1f97fa640 1 -- 172.21.15.27:0/2655131931 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f4004d00 con 0x7fb1fc0d3400 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.243+0000 7fb1f97fa640 1 -- 172.21.15.27:0/2655131931 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400aa90 con 0x7fb1fc0d3400 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.243+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc0ab730 con 0x7fb1fc0d3400 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.243+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc0abcf0 con 0x7fb1fc0d3400 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.244+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1fc068700 con 0x7fb1fc0d3400 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.244+0000 7fb1f97fa640 1 -- 172.21.15.27:0/2655131931 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1f400ac30 con 0x7fb1fc0d3400 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.245+0000 7fb1f97fa640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1e006b500 0x7fb1e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.245+0000 7fb1f97fa640 1 -- 172.21.15.27:0/2655131931 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1f408a320 con 0x7fb1fc0d3400 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.245+0000 7fb200cba640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1e006b500 0x7fb1e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.246+0000 7fb200cba640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1e006b500 0x7fb1e006d9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0036000 tx=0x7fb1f003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.249+0000 7fb1f97fa640 1 -- 172.21.15.27:0/2655131931 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1f4020070 con 0x7fb1fc0d3400 2024-03-06T07:28:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.429+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1fc06b8e0 con 0x7fb1e006b500 2024-03-06T07:28:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.435+0000 7fb1f97fa640 1 -- 172.21.15.27:0/2655131931 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252371 (secure 0 0 0) 0x7fb1fc06b8e0 con 0x7fb1e006b500 2024-03-06T07:28:50.440 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:50.441 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1e006b500 msgr2=0x7fb1e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1e006b500 0x7fb1e006d9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0036000 tx=0x7fb1f003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc0d3400 msgr2=0x7fb1fc1f4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc0d3400 0x7fb1fc1f4220 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400c7b0 tx=0x7fb1f400cc70 comp rx=0 tx=0).stop 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 shutdown_connections 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb1e006b500 0x7fb1e006d9c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1fc0d3400 0x7fb1fc1f4220 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1fc0b2b50 0x7fb1fc0ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 --2- 172.21.15.27:0/2655131931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc071540 0x7fb1fc15cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 >> 172.21.15.27:0/2655131931 conn(0x7fb1fc0d3ca0 msgr2=0x7fb1fc0b8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 shutdown_connections 2024-03-06T07:28:50.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.439+0000 7fb202f1e640 1 -- 172.21.15.27:0/2655131931 wait complete. 2024-03-06T07:28:50.453 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-03-06T07:28:50.073328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324188,"kb_used_data":12296,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754650532,"statfs":{"total":773094113280,"available":772762144768,"internally_reserved":0,"allocated":12591104,"data_stored":6073452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":319213106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.492883"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615443+0000","last_change":"2024-03-06T07:28:41.605491+0000","last_active":"2024-03-06T07:28:42.615443+0000","last_peered":"2024-03-06T07:28:42.615443+0000","last_clean":"2024-03-06T07:28:42.615443+0000","last_became_active":"2024-03-06T07:28:41.605207+0000","last_became_peered":"2024-03-06T07:28:41.605207+0000","last_unstale":"2024-03-06T07:28:42.615443+0000","last_undegraded":"2024-03-06T07:28:42.615443+0000","last_fullsized":"2024-03-06T07:28:42.615443+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:41.754749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584798+0000","last_change":"2024-03-06T07:28:41.598081+0000","last_active":"2024-03-06T07:28:42.584798+0000","last_peered":"2024-03-06T07:28:42.584798+0000","last_clean":"2024-03-06T07:28:42.584798+0000","last_became_active":"2024-03-06T07:28:41.597746+0000","last_became_peered":"2024-03-06T07:28:41.597746+0000","last_unstale":"2024-03-06T07:28:42.584798+0000","last_undegraded":"2024-03-06T07:28:42.584798+0000","last_fullsized":"2024-03-06T07:28:42.584798+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:36.645183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616191+0000","last_change":"2024-03-06T07:28:41.604865+0000","last_active":"2024-03-06T07:28:42.616191+0000","last_peered":"2024-03-06T07:28:42.616191+0000","last_clean":"2024-03-06T07:28:42.616191+0000","last_became_active":"2024-03-06T07:28:41.603741+0000","last_became_peered":"2024-03-06T07:28:41.603741+0000","last_unstale":"2024-03-06T07:28:42.616191+0000","last_undegraded":"2024-03-06T07:28:42.616191+0000","last_fullsized":"2024-03-06T07:28:42.616191+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.995900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615181+0000","last_change":"2024-03-06T07:28:41.606885+0000","last_active":"2024-03-06T07:28:42.615181+0000","last_peered":"2024-03-06T07:28:42.615181+0000","last_clean":"2024-03-06T07:28:42.615181+0000","last_became_active":"2024-03-06T07:28:41.606582+0000","last_became_peered":"2024-03-06T07:28:41.606582+0000","last_unstale":"2024-03-06T07:28:42.615181+0000","last_undegraded":"2024-03-06T07:28:42.615181+0000","last_fullsized":"2024-03-06T07:28:42.615181+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:40.695072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585291+0000","last_change":"2024-03-06T07:28:41.611053+0000","last_active":"2024-03-06T07:28:42.585291+0000","last_peered":"2024-03-06T07:28:42.585291+0000","last_clean":"2024-03-06T07:28:42.585291+0000","last_became_active":"2024-03-06T07:28:41.610636+0000","last_became_peered":"2024-03-06T07:28:41.610636+0000","last_unstale":"2024-03-06T07:28:42.585291+0000","last_undegraded":"2024-03-06T07:28:42.585291+0000","last_fullsized":"2024-03-06T07:28:42.585291+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:54.518946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614777+0000","last_change":"2024-03-06T07:28:41.588032+0000","last_active":"2024-03-06T07:28:42.614777+0000","last_peered":"2024-03-06T07:28:42.614777+0000","last_clean":"2024-03-06T07:28:42.614777+0000","last_became_active":"2024-03-06T07:28:41.587772+0000","last_became_peered":"2024-03-06T07:28:41.587772+0000","last_unstale":"2024-03-06T07:28:42.614777+0000","last_undegraded":"2024-03-06T07:28:42.614777+0000","last_fullsized":"2024-03-06T07:28:42.614777+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:34.817395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615081+0000","last_change":"2024-03-06T07:28:41.597654+0000","last_active":"2024-03-06T07:28:42.615081+0000","last_peered":"2024-03-06T07:28:42.615081+0000","last_clean":"2024-03-06T07:28:42.615081+0000","last_became_active":"2024-03-06T07:28:41.597375+0000","last_became_peered":"2024-03-06T07:28:41.597375+0000","last_unstale":"2024-03-06T07:28:42.615081+0000","last_undegraded":"2024-03-06T07:28:42.615081+0000","last_fullsized":"2024-03-06T07:28:42.615081+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:55.516936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615462+0000","last_change":"2024-03-06T07:28:41.598969+0000","last_active":"2024-03-06T07:28:42.615462+0000","last_peered":"2024-03-06T07:28:42.615462+0000","last_clean":"2024-03-06T07:28:42.615462+0000","last_became_active":"2024-03-06T07:28:41.598680+0000","last_became_peered":"2024-03-06T07:28:41.598680+0000","last_unstale":"2024-03-06T07:28:42.615462+0000","last_undegraded":"2024-03-06T07:28:42.615462+0000","last_fullsized":"2024-03-06T07:28:42.615462+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:07.931449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615598+0000","last_change":"2024-03-06T07:28:41.607892+0000","last_active":"2024-03-06T07:28:42.615598+0000","last_peered":"2024-03-06T07:28:42.615598+0000","last_clean":"2024-03-06T07:28:42.615598+0000","last_became_active":"2024-03-06T07:28:41.607625+0000","last_became_peered":"2024-03-06T07:28:41.607625+0000","last_unstale":"2024-03-06T07:28:42.615598+0000","last_undegraded":"2024-03-06T07:28:42.615598+0000","last_fullsized":"2024-03-06T07:28:42.615598+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:15.356128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615335+0000","last_change":"2024-03-06T07:28:41.613674+0000","last_active":"2024-03-06T07:28:42.615335+0000","last_peered":"2024-03-06T07:28:42.615335+0000","last_clean":"2024-03-06T07:28:42.615335+0000","last_became_active":"2024-03-06T07:28:41.613392+0000","last_became_peered":"2024-03-06T07:28:41.613392+0000","last_unstale":"2024-03-06T07:28:42.615335+0000","last_undegraded":"2024-03-06T07:28:42.615335+0000","last_fullsized":"2024-03-06T07:28:42.615335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:56.109795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585034+0000","last_change":"2024-03-06T07:28:41.599288+0000","last_active":"2024-03-06T07:28:42.585034+0000","last_peered":"2024-03-06T07:28:42.585034+0000","last_clean":"2024-03-06T07:28:42.585034+0000","last_became_active":"2024-03-06T07:28:41.599023+0000","last_became_peered":"2024-03-06T07:28:41.599023+0000","last_unstale":"2024-03-06T07:28:42.585034+0000","last_undegraded":"2024-03-06T07:28:42.585034+0000","last_fullsized":"2024-03-06T07:28:42.585034+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:12.604285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615406+0000","last_change":"2024-03-06T07:28:41.603785+0000","last_active":"2024-03-06T07:28:42.615406+0000","last_peered":"2024-03-06T07:28:42.615406+0000","last_clean":"2024-03-06T07:28:42.615406+0000","last_became_active":"2024-03-06T07:28:41.603478+0000","last_became_peered":"2024-03-06T07:28:41.603478+0000","last_unstale":"2024-03-06T07:28:42.615406+0000","last_undegraded":"2024-03-06T07:28:42.615406+0000","last_fullsized":"2024-03-06T07:28:42.615406+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:18.198067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616026+0000","last_change":"2024-03-06T07:28:41.612243+0000","last_active":"2024-03-06T07:28:42.616026+0000","last_peered":"2024-03-06T07:28:42.616026+0000","last_clean":"2024-03-06T07:28:42.616026+0000","last_became_active":"2024-03-06T07:28:41.611872+0000","last_became_peered":"2024-03-06T07:28:41.611872+0000","last_unstale":"2024-03-06T07:28:42.616026+0000","last_undegraded":"2024-03-06T07:28:42.616026+0000","last_fullsized":"2024-03-06T07:28:42.616026+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:55.837869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615140+0000","last_change":"2024-03-06T07:28:41.597885+0000","last_active":"2024-03-06T07:28:42.615140+0000","last_peered":"2024-03-06T07:28:42.615140+0000","last_clean":"2024-03-06T07:28:42.615140+0000","last_became_active":"2024-03-06T07:28:41.597613+0000","last_became_peered":"2024-03-06T07:28:41.597613+0000","last_unstale":"2024-03-06T07:28:42.615140+0000","last_undegraded":"2024-03-06T07:28:42.615140+0000","last_fullsized":"2024-03-06T07:28:42.615140+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:50.061502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614475+0000","last_change":"2024-03-06T07:28:41.587309+0000","last_active":"2024-03-06T07:28:42.614475+0000","last_peered":"2024-03-06T07:28:42.614475+0000","last_clean":"2024-03-06T07:28:42.614475+0000","last_became_active":"2024-03-06T07:28:41.587034+0000","last_became_peered":"2024-03-06T07:28:41.587034+0000","last_unstale":"2024-03-06T07:28:42.614475+0000","last_undegraded":"2024-03-06T07:28:42.614475+0000","last_fullsized":"2024-03-06T07:28:42.614475+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:02.582015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614882+0000","last_change":"2024-03-06T07:28:41.610442+0000","last_active":"2024-03-06T07:28:42.614882+0000","last_peered":"2024-03-06T07:28:42.614882+0000","last_clean":"2024-03-06T07:28:42.614882+0000","last_became_active":"2024-03-06T07:28:41.605967+0000","last_became_peered":"2024-03-06T07:28:41.605967+0000","last_unstale":"2024-03-06T07:28:42.614882+0000","last_undegraded":"2024-03-06T07:28:42.614882+0000","last_fullsized":"2024-03-06T07:28:42.614882+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:50.576369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584911+0000","last_change":"2024-03-06T07:28:41.600051+0000","last_active":"2024-03-06T07:28:42.584911+0000","last_peered":"2024-03-06T07:28:42.584911+0000","last_clean":"2024-03-06T07:28:42.584911+0000","last_became_active":"2024-03-06T07:28:41.599530+0000","last_became_peered":"2024-03-06T07:28:41.599530+0000","last_unstale":"2024-03-06T07:28:42.584911+0000","last_undegraded":"2024-03-06T07:28:42.584911+0000","last_fullsized":"2024-03-06T07:28:42.584911+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:19.178731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615853+0000","last_change":"2024-03-06T07:28:41.602134+0000","last_active":"2024-03-06T07:28:42.615853+0000","last_peered":"2024-03-06T07:28:42.615853+0000","last_clean":"2024-03-06T07:28:42.615853+0000","last_became_active":"2024-03-06T07:28:41.601846+0000","last_became_peered":"2024-03-06T07:28:41.601846+0000","last_unstale":"2024-03-06T07:28:42.615853+0000","last_undegraded":"2024-03-06T07:28:42.615853+0000","last_fullsized":"2024-03-06T07:28:42.615853+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:13.831484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616246+0000","last_change":"2024-03-06T07:28:41.612404+0000","last_active":"2024-03-06T07:28:42.616246+0000","last_peered":"2024-03-06T07:28:42.616246+0000","last_clean":"2024-03-06T07:28:42.616246+0000","last_became_active":"2024-03-06T07:28:41.612121+0000","last_became_peered":"2024-03-06T07:28:41.612121+0000","last_unstale":"2024-03-06T07:28:42.616246+0000","last_undegraded":"2024-03-06T07:28:42.616246+0000","last_fullsized":"2024-03-06T07:28:42.616246+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:05.095081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584952+0000","last_change":"2024-03-06T07:28:41.593131+0000","last_active":"2024-03-06T07:28:42.584952+0000","last_peered":"2024-03-06T07:28:42.584952+0000","last_clean":"2024-03-06T07:28:42.584952+0000","last_became_active":"2024-03-06T07:28:41.592855+0000","last_became_peered":"2024-03-06T07:28:41.592855+0000","last_unstale":"2024-03-06T07:28:42.584952+0000","last_undegraded":"2024-03-06T07:28:42.584952+0000","last_fullsized":"2024-03-06T07:28:42.584952+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:40.354302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585104+0000","last_change":"2024-03-06T07:28:41.594310+0000","last_active":"2024-03-06T07:28:42.585104+0000","last_peered":"2024-03-06T07:28:42.585104+0000","last_clean":"2024-03-06T07:28:42.585104+0000","last_became_active":"2024-03-06T07:28:41.593649+0000","last_became_peered":"2024-03-06T07:28:41.593649+0000","last_unstale":"2024-03-06T07:28:42.585104+0000","last_undegraded":"2024-03-06T07:28:42.585104+0000","last_fullsized":"2024-03-06T07:28:42.585104+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:34.600777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584533+0000","last_change":"2024-03-06T07:28:41.599908+0000","last_active":"2024-03-06T07:28:42.584533+0000","last_peered":"2024-03-06T07:28:42.584533+0000","last_clean":"2024-03-06T07:28:42.584533+0000","last_became_active":"2024-03-06T07:28:41.599609+0000","last_became_peered":"2024-03-06T07:28:41.599609+0000","last_unstale":"2024-03-06T07:28:42.584533+0000","last_undegraded":"2024-03-06T07:28:42.584533+0000","last_fullsized":"2024-03-06T07:28:42.584533+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:56.356422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584612+0000","last_change":"2024-03-06T07:28:41.592523+0000","last_active":"2024-03-06T07:28:42.584612+0000","last_peered":"2024-03-06T07:28:42.584612+0000","last_clean":"2024-03-06T07:28:42.584612+0000","last_became_active":"2024-03-06T07:28:41.592223+0000","last_became_peered":"2024-03-06T07:28:41.592223+0000","last_unstale":"2024-03-06T07:28:42.584612+0000","last_undegraded":"2024-03-06T07:28:42.584612+0000","last_fullsized":"2024-03-06T07:28:42.584612+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:21.109449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584132+0000","last_change":"2024-03-06T07:28:41.593471+0000","last_active":"2024-03-06T07:28:42.584132+0000","last_peered":"2024-03-06T07:28:42.584132+0000","last_clean":"2024-03-06T07:28:42.584132+0000","last_became_active":"2024-03-06T07:28:41.593153+0000","last_became_peered":"2024-03-06T07:28:41.593153+0000","last_unstale":"2024-03-06T07:28:42.584132+0000","last_undegraded":"2024-03-06T07:28:42.584132+0000","last_fullsized":"2024-03-06T07:28:42.584132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:16.974584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585034+0000","last_change":"2024-03-06T07:28:41.599835+0000","last_active":"2024-03-06T07:28:42.585034+0000","last_peered":"2024-03-06T07:28:42.585034+0000","last_clean":"2024-03-06T07:28:42.585034+0000","last_became_active":"2024-03-06T07:28:41.599196+0000","last_became_peered":"2024-03-06T07:28:41.599196+0000","last_unstale":"2024-03-06T07:28:42.585034+0000","last_undegraded":"2024-03-06T07:28:42.585034+0000","last_fullsized":"2024-03-06T07:28:42.585034+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:42.380673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615615+0000","last_change":"2024-03-06T07:28:41.604648+0000","last_active":"2024-03-06T07:28:42.615615+0000","last_peered":"2024-03-06T07:28:42.615615+0000","last_clean":"2024-03-06T07:28:42.615615+0000","last_became_active":"2024-03-06T07:28:41.604346+0000","last_became_peered":"2024-03-06T07:28:41.604346+0000","last_unstale":"2024-03-06T07:28:42.615615+0000","last_undegraded":"2024-03-06T07:28:42.615615+0000","last_fullsized":"2024-03-06T07:28:42.615615+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:27.688649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614784+0000","last_change":"2024-03-06T07:28:41.604347+0000","last_active":"2024-03-06T07:28:42.614784+0000","last_peered":"2024-03-06T07:28:42.614784+0000","last_clean":"2024-03-06T07:28:42.614784+0000","last_became_active":"2024-03-06T07:28:41.604078+0000","last_became_peered":"2024-03-06T07:28:41.604078+0000","last_unstale":"2024-03-06T07:28:42.614784+0000","last_undegraded":"2024-03-06T07:28:42.614784+0000","last_fullsized":"2024-03-06T07:28:42.614784+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:12.021731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584403+0000","last_change":"2024-03-06T07:28:41.610412+0000","last_active":"2024-03-06T07:28:42.584403+0000","last_peered":"2024-03-06T07:28:42.584403+0000","last_clean":"2024-03-06T07:28:42.584403+0000","last_became_active":"2024-03-06T07:28:41.607523+0000","last_became_peered":"2024-03-06T07:28:41.607523+0000","last_unstale":"2024-03-06T07:28:42.584403+0000","last_undegraded":"2024-03-06T07:28:42.584403+0000","last_fullsized":"2024-03-06T07:28:42.584403+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:19.391795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584714+0000","last_change":"2024-03-06T07:28:41.592755+0000","last_active":"2024-03-06T07:28:42.584714+0000","last_peered":"2024-03-06T07:28:42.584714+0000","last_clean":"2024-03-06T07:28:42.584714+0000","last_became_active":"2024-03-06T07:28:41.592455+0000","last_became_peered":"2024-03-06T07:28:41.592455+0000","last_unstale":"2024-03-06T07:28:42.584714+0000","last_undegraded":"2024-03-06T07:28:42.584714+0000","last_fullsized":"2024-03-06T07:28:42.584714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:37.201552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615460+0000","last_change":"2024-03-06T07:28:41.602207+0000","last_active":"2024-03-06T07:28:42.615460+0000","last_peered":"2024-03-06T07:28:42.615460+0000","last_clean":"2024-03-06T07:28:42.615460+0000","last_became_active":"2024-03-06T07:28:41.601914+0000","last_became_peered":"2024-03-06T07:28:41.601914+0000","last_unstale":"2024-03-06T07:28:42.615460+0000","last_undegraded":"2024-03-06T07:28:42.615460+0000","last_fullsized":"2024-03-06T07:28:42.615460+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:40.520524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614940+0000","last_change":"2024-03-06T07:28:41.605636+0000","last_active":"2024-03-06T07:28:42.614940+0000","last_peered":"2024-03-06T07:28:42.614940+0000","last_clean":"2024-03-06T07:28:42.614940+0000","last_became_active":"2024-03-06T07:28:41.605101+0000","last_became_peered":"2024-03-06T07:28:41.605101+0000","last_unstale":"2024-03-06T07:28:42.614940+0000","last_undegraded":"2024-03-06T07:28:42.614940+0000","last_fullsized":"2024-03-06T07:28:42.614940+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:51.706297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614469+0000","last_change":"2024-03-06T07:28:41.592526+0000","last_active":"2024-03-06T07:28:42.614469+0000","last_peered":"2024-03-06T07:28:42.614469+0000","last_clean":"2024-03-06T07:28:42.614469+0000","last_became_active":"2024-03-06T07:28:41.592212+0000","last_became_peered":"2024-03-06T07:28:41.592212+0000","last_unstale":"2024-03-06T07:28:42.614469+0000","last_undegraded":"2024-03-06T07:28:42.614469+0000","last_fullsized":"2024-03-06T07:28:42.614469+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:22.548099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614673+0000","last_change":"2024-03-06T07:28:41.612948+0000","last_active":"2024-03-06T07:28:42.614673+0000","last_peered":"2024-03-06T07:28:42.614673+0000","last_clean":"2024-03-06T07:28:42.614673+0000","last_became_active":"2024-03-06T07:28:41.612634+0000","last_became_peered":"2024-03-06T07:28:41.612634+0000","last_unstale":"2024-03-06T07:28:42.614673+0000","last_undegraded":"2024-03-06T07:28:42.614673+0000","last_fullsized":"2024-03-06T07:28:42.614673+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:03.772883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615429+0000","last_change":"2024-03-06T07:28:40.591430+0000","last_active":"2024-03-06T07:28:42.615429+0000","last_peered":"2024-03-06T07:28:42.615429+0000","last_clean":"2024-03-06T07:28:42.615429+0000","last_became_active":"2024-03-06T07:28:40.590954+0000","last_became_peered":"2024-03-06T07:28:40.590954+0000","last_unstale":"2024-03-06T07:28:42.615429+0000","last_undegraded":"2024-03-06T07:28:42.615429+0000","last_fullsized":"2024-03-06T07:28:42.615429+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:41.774205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615738+0000","last_change":"2024-03-06T07:28:41.601165+0000","last_active":"2024-03-06T07:28:42.615738+0000","last_peered":"2024-03-06T07:28:42.615738+0000","last_clean":"2024-03-06T07:28:42.615738+0000","last_became_active":"2024-03-06T07:28:41.600834+0000","last_became_peered":"2024-03-06T07:28:41.600834+0000","last_unstale":"2024-03-06T07:28:42.615738+0000","last_undegraded":"2024-03-06T07:28:42.615738+0000","last_fullsized":"2024-03-06T07:28:42.615738+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:34.600777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616100+0000","last_change":"2024-03-06T07:28:40.592943+0000","last_active":"2024-03-06T07:28:42.616100+0000","last_peered":"2024-03-06T07:28:42.616100+0000","last_clean":"2024-03-06T07:28:42.616100+0000","last_became_active":"2024-03-06T07:28:40.592617+0000","last_became_peered":"2024-03-06T07:28:40.592617+0000","last_unstale":"2024-03-06T07:28:42.616100+0000","last_undegraded":"2024-03-06T07:28:42.616100+0000","last_fullsized":"2024-03-06T07:28:42.616100+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:35.643091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614680+0000","last_change":"2024-03-06T07:28:41.591740+0000","last_active":"2024-03-06T07:28:42.614680+0000","last_peered":"2024-03-06T07:28:42.614680+0000","last_clean":"2024-03-06T07:28:42.614680+0000","last_became_active":"2024-03-06T07:28:41.591478+0000","last_became_peered":"2024-03-06T07:28:41.591478+0000","last_unstale":"2024-03-06T07:28:42.614680+0000","last_undegraded":"2024-03-06T07:28:42.614680+0000","last_fullsized":"2024-03-06T07:28:42.614680+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:54.518946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585315+0000","last_change":"2024-03-06T07:28:40.601530+0000","last_active":"2024-03-06T07:28:42.585315+0000","last_peered":"2024-03-06T07:28:42.585315+0000","last_clean":"2024-03-06T07:28:42.585315+0000","last_became_active":"2024-03-06T07:28:40.601235+0000","last_became_peered":"2024-03-06T07:28:40.601235+0000","last_unstale":"2024-03-06T07:28:42.585315+0000","last_undegraded":"2024-03-06T07:28:42.585315+0000","last_fullsized":"2024-03-06T07:28:42.585315+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:10.155882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616345+0000","last_change":"2024-03-06T07:28:41.612533+0000","last_active":"2024-03-06T07:28:42.616345+0000","last_peered":"2024-03-06T07:28:42.616345+0000","last_clean":"2024-03-06T07:28:42.616345+0000","last_became_active":"2024-03-06T07:28:41.612195+0000","last_became_peered":"2024-03-06T07:28:41.612195+0000","last_unstale":"2024-03-06T07:28:42.616345+0000","last_undegraded":"2024-03-06T07:28:42.616345+0000","last_fullsized":"2024-03-06T07:28:42.616345+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:19.391795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585158+0000","last_change":"2024-03-06T07:28:40.601199+0000","last_active":"2024-03-06T07:28:42.585158+0000","last_peered":"2024-03-06T07:28:42.585158+0000","last_clean":"2024-03-06T07:28:42.585158+0000","last_became_active":"2024-03-06T07:28:40.600924+0000","last_became_peered":"2024-03-06T07:28:40.600924+0000","last_unstale":"2024-03-06T07:28:42.585158+0000","last_undegraded":"2024-03-06T07:28:42.585158+0000","last_fullsized":"2024-03-06T07:28:42.585158+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:51.604636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615023+0000","last_change":"2024-03-06T07:28:41.613286+0000","last_active":"2024-03-06T07:28:42.615023+0000","last_peered":"2024-03-06T07:28:42.615023+0000","last_clean":"2024-03-06T07:28:42.615023+0000","last_became_active":"2024-03-06T07:28:41.612678+0000","last_became_peered":"2024-03-06T07:28:41.612678+0000","last_unstale":"2024-03-06T07:28:42.615023+0000","last_undegraded":"2024-03-06T07:28:42.615023+0000","last_fullsized":"2024-03-06T07:28:42.615023+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:08.661708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615029+0000","last_change":"2024-03-06T07:28:40.596714+0000","last_active":"2024-03-06T07:28:42.615029+0000","last_peered":"2024-03-06T07:28:42.615029+0000","last_clean":"2024-03-06T07:28:42.615029+0000","last_became_active":"2024-03-06T07:28:40.596484+0000","last_became_peered":"2024-03-06T07:28:40.596484+0000","last_unstale":"2024-03-06T07:28:42.615029+0000","last_undegraded":"2024-03-06T07:28:42.615029+0000","last_fullsized":"2024-03-06T07:28:42.615029+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:33.815303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584885+0000","last_change":"2024-03-06T07:28:41.598081+0000","last_active":"2024-03-06T07:28:42.584885+0000","last_peered":"2024-03-06T07:28:42.584885+0000","last_clean":"2024-03-06T07:28:42.584885+0000","last_became_active":"2024-03-06T07:28:41.597755+0000","last_became_peered":"2024-03-06T07:28:41.597755+0000","last_unstale":"2024-03-06T07:28:42.584885+0000","last_undegraded":"2024-03-06T07:28:42.584885+0000","last_fullsized":"2024-03-06T07:28:42.584885+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:17.415589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584596+0000","last_change":"2024-03-06T07:28:40.593599+0000","last_active":"2024-03-06T07:28:42.584596+0000","last_peered":"2024-03-06T07:28:42.584596+0000","last_clean":"2024-03-06T07:28:42.584596+0000","last_became_active":"2024-03-06T07:28:40.593318+0000","last_became_peered":"2024-03-06T07:28:40.593318+0000","last_unstale":"2024-03-06T07:28:42.584596+0000","last_undegraded":"2024-03-06T07:28:42.584596+0000","last_fullsized":"2024-03-06T07:28:42.584596+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:18.176638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615176+0000","last_change":"2024-03-06T07:28:41.611042+0000","last_active":"2024-03-06T07:28:42.615176+0000","last_peered":"2024-03-06T07:28:42.615176+0000","last_clean":"2024-03-06T07:28:42.615176+0000","last_became_active":"2024-03-06T07:28:41.610758+0000","last_became_peered":"2024-03-06T07:28:41.610758+0000","last_unstale":"2024-03-06T07:28:42.615176+0000","last_undegraded":"2024-03-06T07:28:42.615176+0000","last_fullsized":"2024-03-06T07:28:42.615176+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:22.849251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584484+0000","last_change":"2024-03-06T07:28:40.587951+0000","last_active":"2024-03-06T07:28:42.584484+0000","last_peered":"2024-03-06T07:28:42.584484+0000","last_clean":"2024-03-06T07:28:42.584484+0000","last_became_active":"2024-03-06T07:28:40.587671+0000","last_became_peered":"2024-03-06T07:28:40.587671+0000","last_unstale":"2024-03-06T07:28:42.584484+0000","last_undegraded":"2024-03-06T07:28:42.584484+0000","last_fullsized":"2024-03-06T07:28:42.584484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:06.481910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615021+0000","last_change":"2024-03-06T07:28:41.602157+0000","last_active":"2024-03-06T07:28:42.615021+0000","last_peered":"2024-03-06T07:28:42.615021+0000","last_clean":"2024-03-06T07:28:42.615021+0000","last_became_active":"2024-03-06T07:28:41.601862+0000","last_became_peered":"2024-03-06T07:28:41.601862+0000","last_unstale":"2024-03-06T07:28:42.615021+0000","last_undegraded":"2024-03-06T07:28:42.615021+0000","last_fullsized":"2024-03-06T07:28:42.615021+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:28.477340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615227+0000","last_change":"2024-03-06T07:28:40.590486+0000","last_active":"2024-03-06T07:28:42.615227+0000","last_peered":"2024-03-06T07:28:42.615227+0000","last_clean":"2024-03-06T07:28:42.615227+0000","last_became_active":"2024-03-06T07:28:40.590113+0000","last_became_peered":"2024-03-06T07:28:40.590113+0000","last_unstale":"2024-03-06T07:28:42.615227+0000","last_undegraded":"2024-03-06T07:28:42.615227+0000","last_fullsized":"2024-03-06T07:28:42.615227+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:35.804062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615546+0000","last_change":"2024-03-06T07:28:41.613036+0000","last_active":"2024-03-06T07:28:42.615546+0000","last_peered":"2024-03-06T07:28:42.615546+0000","last_clean":"2024-03-06T07:28:42.615546+0000","last_became_active":"2024-03-06T07:28:41.612725+0000","last_became_peered":"2024-03-06T07:28:41.612725+0000","last_unstale":"2024-03-06T07:28:42.615546+0000","last_undegraded":"2024-03-06T07:28:42.615546+0000","last_fullsized":"2024-03-06T07:28:42.615546+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:20.966815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585400+0000","last_change":"2024-03-06T07:28:40.600729+0000","last_active":"2024-03-06T07:28:42.585400+0000","last_peered":"2024-03-06T07:28:42.585400+0000","last_clean":"2024-03-06T07:28:42.585400+0000","last_became_active":"2024-03-06T07:28:40.600441+0000","last_became_peered":"2024-03-06T07:28:40.600441+0000","last_unstale":"2024-03-06T07:28:42.585400+0000","last_undegraded":"2024-03-06T07:28:42.585400+0000","last_fullsized":"2024-03-06T07:28:42.585400+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:40.818375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615866+0000","last_change":"2024-03-06T07:28:41.606308+0000","last_active":"2024-03-06T07:28:42.615866+0000","last_peered":"2024-03-06T07:28:42.615866+0000","last_clean":"2024-03-06T07:28:42.615866+0000","last_became_active":"2024-03-06T07:28:41.605734+0000","last_became_peered":"2024-03-06T07:28:41.605734+0000","last_unstale":"2024-03-06T07:28:42.615866+0000","last_undegraded":"2024-03-06T07:28:42.615866+0000","last_fullsized":"2024-03-06T07:28:42.615866+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:18.073915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584417+0000","last_change":"2024-03-06T07:28:40.592798+0000","last_active":"2024-03-06T07:28:42.584417+0000","last_peered":"2024-03-06T07:28:42.584417+0000","last_clean":"2024-03-06T07:28:42.584417+0000","last_became_active":"2024-03-06T07:28:40.592483+0000","last_became_peered":"2024-03-06T07:28:40.592483+0000","last_unstale":"2024-03-06T07:28:42.584417+0000","last_undegraded":"2024-03-06T07:28:42.584417+0000","last_fullsized":"2024-03-06T07:28:42.584417+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:57.654186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584539+0000","last_change":"2024-03-06T07:28:41.606477+0000","last_active":"2024-03-06T07:28:42.584539+0000","last_peered":"2024-03-06T07:28:42.584539+0000","last_clean":"2024-03-06T07:28:42.584539+0000","last_became_active":"2024-03-06T07:28:41.606142+0000","last_became_peered":"2024-03-06T07:28:41.606142+0000","last_unstale":"2024-03-06T07:28:42.584539+0000","last_undegraded":"2024-03-06T07:28:42.584539+0000","last_fullsized":"2024-03-06T07:28:42.584539+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:03.190401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":17,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615153+0000","last_change":"2024-03-06T07:28:40.592800+0000","last_active":"2024-03-06T07:28:42.615153+0000","last_peered":"2024-03-06T07:28:42.615153+0000","last_clean":"2024-03-06T07:28:42.615153+0000","last_became_active":"2024-03-06T07:28:40.592525+0000","last_became_peered":"2024-03-06T07:28:40.592525+0000","last_unstale":"2024-03-06T07:28:42.615153+0000","last_undegraded":"2024-03-06T07:28:42.615153+0000","last_fullsized":"2024-03-06T07:28:42.615153+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:23.047792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615295+0000","last_change":"2024-03-06T07:28:41.602882+0000","last_active":"2024-03-06T07:28:42.615295+0000","last_peered":"2024-03-06T07:28:42.615295+0000","last_clean":"2024-03-06T07:28:42.615295+0000","last_became_active":"2024-03-06T07:28:41.602571+0000","last_became_peered":"2024-03-06T07:28:41.602571+0000","last_unstale":"2024-03-06T07:28:42.615295+0000","last_undegraded":"2024-03-06T07:28:42.615295+0000","last_fullsized":"2024-03-06T07:28:42.615295+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:28.691437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":20,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615479+0000","last_change":"2024-03-06T07:28:40.589011+0000","last_active":"2024-03-06T07:28:42.615479+0000","last_peered":"2024-03-06T07:28:42.615479+0000","last_clean":"2024-03-06T07:28:42.615479+0000","last_became_active":"2024-03-06T07:28:40.588731+0000","last_became_peered":"2024-03-06T07:28:40.588731+0000","last_unstale":"2024-03-06T07:28:42.615479+0000","last_undegraded":"2024-03-06T07:28:42.615479+0000","last_fullsized":"2024-03-06T07:28:42.615479+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:06.929357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584676+0000","last_change":"2024-03-06T07:28:41.603914+0000","last_active":"2024-03-06T07:28:42.584676+0000","last_peered":"2024-03-06T07:28:42.584676+0000","last_clean":"2024-03-06T07:28:42.584676+0000","last_became_active":"2024-03-06T07:28:41.603566+0000","last_became_peered":"2024-03-06T07:28:41.603566+0000","last_unstale":"2024-03-06T07:28:42.584676+0000","last_undegraded":"2024-03-06T07:28:42.584676+0000","last_fullsized":"2024-03-06T07:28:42.584676+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:24.371626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614486+0000","last_change":"2024-03-06T07:28:40.596849+0000","last_active":"2024-03-06T07:28:42.614486+0000","last_peered":"2024-03-06T07:28:42.614486+0000","last_clean":"2024-03-06T07:28:42.614486+0000","last_became_active":"2024-03-06T07:28:40.596315+0000","last_became_peered":"2024-03-06T07:28:40.596315+0000","last_unstale":"2024-03-06T07:28:42.614486+0000","last_undegraded":"2024-03-06T07:28:42.614486+0000","last_fullsized":"2024-03-06T07:28:42.614486+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:04.152803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584355+0000","last_change":"2024-03-06T07:28:41.593085+0000","last_active":"2024-03-06T07:28:42.584355+0000","last_peered":"2024-03-06T07:28:42.584355+0000","last_clean":"2024-03-06T07:28:42.584355+0000","last_became_active":"2024-03-06T07:28:41.592769+0000","last_became_peered":"2024-03-06T07:28:41.592769+0000","last_unstale":"2024-03-06T07:28:42.584355+0000","last_undegraded":"2024-03-06T07:28:42.584355+0000","last_fullsized":"2024-03-06T07:28:42.584355+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:55.787493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615402+0000","last_change":"2024-03-06T07:28:40.595680+0000","last_active":"2024-03-06T07:28:42.615402+0000","last_peered":"2024-03-06T07:28:42.615402+0000","last_clean":"2024-03-06T07:28:42.615402+0000","last_became_active":"2024-03-06T07:28:40.595437+0000","last_became_peered":"2024-03-06T07:28:40.595437+0000","last_unstale":"2024-03-06T07:28:42.615402+0000","last_undegraded":"2024-03-06T07:28:42.615402+0000","last_fullsized":"2024-03-06T07:28:42.615402+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:20.107356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584232+0000","last_change":"2024-03-06T07:28:41.604230+0000","last_active":"2024-03-06T07:28:42.584232+0000","last_peered":"2024-03-06T07:28:42.584232+0000","last_clean":"2024-03-06T07:28:42.584232+0000","last_became_active":"2024-03-06T07:28:41.602845+0000","last_became_peered":"2024-03-06T07:28:41.602845+0000","last_unstale":"2024-03-06T07:28:42.584232+0000","last_undegraded":"2024-03-06T07:28:42.584232+0000","last_fullsized":"2024-03-06T07:28:42.584232+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:38.995900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":22,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584998+0000","last_change":"2024-03-06T07:28:40.600673+0000","last_active":"2024-03-06T07:28:42.584998+0000","last_peered":"2024-03-06T07:28:42.584998+0000","last_clean":"2024-03-06T07:28:42.584998+0000","last_became_active":"2024-03-06T07:28:40.600373+0000","last_became_peered":"2024-03-06T07:28:40.600373+0000","last_unstale":"2024-03-06T07:28:42.584998+0000","last_undegraded":"2024-03-06T07:28:42.584998+0000","last_fullsized":"2024-03-06T07:28:42.584998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:03.969324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585624+0000","last_change":"2024-03-06T07:28:41.610614+0000","last_active":"2024-03-06T07:28:42.585624+0000","last_peered":"2024-03-06T07:28:42.585624+0000","last_clean":"2024-03-06T07:28:42.585624+0000","last_became_active":"2024-03-06T07:28:41.610307+0000","last_became_peered":"2024-03-06T07:28:41.610307+0000","last_unstale":"2024-03-06T07:28:42.585624+0000","last_undegraded":"2024-03-06T07:28:42.585624+0000","last_fullsized":"2024-03-06T07:28:42.585624+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:34.817395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615670+0000","last_change":"2024-03-06T07:28:40.591386+0000","last_active":"2024-03-06T07:28:42.615670+0000","last_peered":"2024-03-06T07:28:42.615670+0000","last_clean":"2024-03-06T07:28:42.615670+0000","last_became_active":"2024-03-06T07:28:40.591132+0000","last_became_peered":"2024-03-06T07:28:40.591132+0000","last_unstale":"2024-03-06T07:28:42.615670+0000","last_undegraded":"2024-03-06T07:28:42.615670+0000","last_fullsized":"2024-03-06T07:28:42.615670+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:54.835777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614664+0000","last_change":"2024-03-06T07:28:41.605192+0000","last_active":"2024-03-06T07:28:42.614664+0000","last_peered":"2024-03-06T07:28:42.614664+0000","last_clean":"2024-03-06T07:28:42.614664+0000","last_became_active":"2024-03-06T07:28:41.604812+0000","last_became_peered":"2024-03-06T07:28:41.604812+0000","last_unstale":"2024-03-06T07:28:42.614664+0000","last_undegraded":"2024-03-06T07:28:42.614664+0000","last_fullsized":"2024-03-06T07:28:42.614664+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:03.509633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585603+0000","last_change":"2024-03-06T07:28:40.598436+0000","last_active":"2024-03-06T07:28:42.585603+0000","last_peered":"2024-03-06T07:28:42.585603+0000","last_clean":"2024-03-06T07:28:42.585603+0000","last_became_active":"2024-03-06T07:28:40.598075+0000","last_became_peered":"2024-03-06T07:28:40.598075+0000","last_unstale":"2024-03-06T07:28:42.585603+0000","last_undegraded":"2024-03-06T07:28:42.585603+0000","last_fullsized":"2024-03-06T07:28:42.585603+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:21.546007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616566+0000","last_change":"2024-03-06T07:28:41.603468+0000","last_active":"2024-03-06T07:28:42.616566+0000","last_peered":"2024-03-06T07:28:42.616566+0000","last_clean":"2024-03-06T07:28:42.616566+0000","last_became_active":"2024-03-06T07:28:41.603081+0000","last_became_peered":"2024-03-06T07:28:41.603081+0000","last_unstale":"2024-03-06T07:28:42.616566+0000","last_undegraded":"2024-03-06T07:28:42.616566+0000","last_fullsized":"2024-03-06T07:28:42.616566+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:03.190401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615332+0000","last_change":"2024-03-06T07:28:40.594727+0000","last_active":"2024-03-06T07:28:42.615332+0000","last_peered":"2024-03-06T07:28:42.615332+0000","last_clean":"2024-03-06T07:28:42.615332+0000","last_became_active":"2024-03-06T07:28:40.594455+0000","last_became_peered":"2024-03-06T07:28:40.594455+0000","last_unstale":"2024-03-06T07:28:42.615332+0000","last_undegraded":"2024-03-06T07:28:42.615332+0000","last_fullsized":"2024-03-06T07:28:42.615332+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:36.199459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616429+0000","last_change":"2024-03-06T07:28:41.605423+0000","last_active":"2024-03-06T07:28:42.616429+0000","last_peered":"2024-03-06T07:28:42.616429+0000","last_clean":"2024-03-06T07:28:42.616429+0000","last_became_active":"2024-03-06T07:28:41.604962+0000","last_became_peered":"2024-03-06T07:28:41.604962+0000","last_unstale":"2024-03-06T07:28:42.616429+0000","last_undegraded":"2024-03-06T07:28:42.616429+0000","last_fullsized":"2024-03-06T07:28:42.616429+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:56.356422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614955+0000","last_change":"2024-03-06T07:28:40.598996+0000","last_active":"2024-03-06T07:28:42.614955+0000","last_peered":"2024-03-06T07:28:42.614955+0000","last_clean":"2024-03-06T07:28:42.614955+0000","last_became_active":"2024-03-06T07:28:40.598707+0000","last_became_peered":"2024-03-06T07:28:40.598707+0000","last_unstale":"2024-03-06T07:28:42.614955+0000","last_undegraded":"2024-03-06T07:28:42.614955+0000","last_fullsized":"2024-03-06T07:28:42.614955+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:53.516854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616329+0000","last_change":"2024-03-06T07:28:41.602357+0000","last_active":"2024-03-06T07:28:42.616329+0000","last_peered":"2024-03-06T07:28:42.616329+0000","last_clean":"2024-03-06T07:28:42.616329+0000","last_became_active":"2024-03-06T07:28:41.602093+0000","last_became_peered":"2024-03-06T07:28:41.602093+0000","last_unstale":"2024-03-06T07:28:42.616329+0000","last_undegraded":"2024-03-06T07:28:42.616329+0000","last_fullsized":"2024-03-06T07:28:42.616329+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:55.787493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614653+0000","last_change":"2024-03-06T07:28:40.600309+0000","last_active":"2024-03-06T07:28:42.614653+0000","last_peered":"2024-03-06T07:28:42.614653+0000","last_clean":"2024-03-06T07:28:42.614653+0000","last_became_active":"2024-03-06T07:28:40.599976+0000","last_became_peered":"2024-03-06T07:28:40.599976+0000","last_unstale":"2024-03-06T07:28:42.614653+0000","last_undegraded":"2024-03-06T07:28:42.614653+0000","last_fullsized":"2024-03-06T07:28:42.614653+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:42.371056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615062+0000","last_change":"2024-03-06T07:28:41.599242+0000","last_active":"2024-03-06T07:28:42.615062+0000","last_peered":"2024-03-06T07:28:42.615062+0000","last_clean":"2024-03-06T07:28:42.615062+0000","last_became_active":"2024-03-06T07:28:41.598920+0000","last_became_peered":"2024-03-06T07:28:41.598920+0000","last_unstale":"2024-03-06T07:28:42.615062+0000","last_undegraded":"2024-03-06T07:28:42.615062+0000","last_fullsized":"2024-03-06T07:28:42.615062+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:51.489658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585293+0000","last_change":"2024-03-06T07:28:40.601354+0000","last_active":"2024-03-06T07:28:42.585293+0000","last_peered":"2024-03-06T07:28:42.585293+0000","last_clean":"2024-03-06T07:28:42.585293+0000","last_became_active":"2024-03-06T07:28:40.601113+0000","last_became_peered":"2024-03-06T07:28:40.601113+0000","last_unstale":"2024-03-06T07:28:42.585293+0000","last_undegraded":"2024-03-06T07:28:42.585293+0000","last_fullsized":"2024-03-06T07:28:42.585293+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:35.874968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615717+0000","last_change":"2024-03-06T07:28:41.612198+0000","last_active":"2024-03-06T07:28:42.615717+0000","last_peered":"2024-03-06T07:28:42.615717+0000","last_clean":"2024-03-06T07:28:42.615717+0000","last_became_active":"2024-03-06T07:28:41.611905+0000","last_became_peered":"2024-03-06T07:28:41.611905+0000","last_unstale":"2024-03-06T07:28:42.615717+0000","last_undegraded":"2024-03-06T07:28:42.615717+0000","last_fullsized":"2024-03-06T07:28:42.615717+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:24.049884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614539+0000","last_change":"2024-03-06T07:28:40.599514+0000","last_active":"2024-03-06T07:28:42.614539+0000","last_peered":"2024-03-06T07:28:42.614539+0000","last_clean":"2024-03-06T07:28:42.614539+0000","last_became_active":"2024-03-06T07:28:40.599217+0000","last_became_peered":"2024-03-06T07:28:40.599217+0000","last_unstale":"2024-03-06T07:28:42.614539+0000","last_undegraded":"2024-03-06T07:28:42.614539+0000","last_fullsized":"2024-03-06T07:28:42.614539+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:47.022153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584335+0000","last_change":"2024-03-06T07:28:41.607422+0000","last_active":"2024-03-06T07:28:42.584335+0000","last_peered":"2024-03-06T07:28:42.584335+0000","last_clean":"2024-03-06T07:28:42.584335+0000","last_became_active":"2024-03-06T07:28:41.607072+0000","last_became_peered":"2024-03-06T07:28:41.607072+0000","last_unstale":"2024-03-06T07:28:42.584335+0000","last_undegraded":"2024-03-06T07:28:42.584335+0000","last_fullsized":"2024-03-06T07:28:42.584335+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:05.095081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615350+0000","last_change":"2024-03-06T07:28:40.596316+0000","last_active":"2024-03-06T07:28:42.615350+0000","last_peered":"2024-03-06T07:28:42.615350+0000","last_clean":"2024-03-06T07:28:42.615350+0000","last_became_active":"2024-03-06T07:28:40.595681+0000","last_became_peered":"2024-03-06T07:28:40.595681+0000","last_unstale":"2024-03-06T07:28:42.615350+0000","last_undegraded":"2024-03-06T07:28:42.615350+0000","last_fullsized":"2024-03-06T07:28:42.615350+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:58.815160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616022+0000","last_change":"2024-03-06T07:28:41.605871+0000","last_active":"2024-03-06T07:28:42.616022+0000","last_peered":"2024-03-06T07:28:42.616022+0000","last_clean":"2024-03-06T07:28:42.616022+0000","last_became_active":"2024-03-06T07:28:41.605577+0000","last_became_peered":"2024-03-06T07:28:41.605577+0000","last_unstale":"2024-03-06T07:28:42.616022+0000","last_undegraded":"2024-03-06T07:28:42.616022+0000","last_fullsized":"2024-03-06T07:28:42.616022+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:36.806154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584550+0000","last_change":"2024-03-06T07:28:40.590129+0000","last_active":"2024-03-06T07:28:42.584550+0000","last_peered":"2024-03-06T07:28:42.584550+0000","last_clean":"2024-03-06T07:28:42.584550+0000","last_became_active":"2024-03-06T07:28:40.589864+0000","last_became_peered":"2024-03-06T07:28:40.589864+0000","last_unstale":"2024-03-06T07:28:42.584550+0000","last_undegraded":"2024-03-06T07:28:42.584550+0000","last_fullsized":"2024-03-06T07:28:42.584550+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:39.352210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615655+0000","last_change":"2024-03-06T07:28:41.607050+0000","last_active":"2024-03-06T07:28:42.615655+0000","last_peered":"2024-03-06T07:28:42.615655+0000","last_clean":"2024-03-06T07:28:42.615655+0000","last_became_active":"2024-03-06T07:28:41.606639+0000","last_became_peered":"2024-03-06T07:28:41.606639+0000","last_unstale":"2024-03-06T07:28:42.615655+0000","last_undegraded":"2024-03-06T07:28:42.615655+0000","last_fullsized":"2024-03-06T07:28:42.615655+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:09.115278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"114'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.616105+0000","last_change":"2024-03-06T07:28:40.598310+0000","last_active":"2024-03-06T07:28:42.616105+0000","last_peered":"2024-03-06T07:28:42.616105+0000","last_clean":"2024-03-06T07:28:42.616105+0000","last_became_active":"2024-03-06T07:28:40.597952+0000","last_became_peered":"2024-03-06T07:28:40.597952+0000","last_unstale":"2024-03-06T07:28:42.616105+0000","last_undegraded":"2024-03-06T07:28:42.616105+0000","last_fullsized":"2024-03-06T07:28:42.616105+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:11.242574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615326+0000","last_change":"2024-03-06T07:28:41.600904+0000","last_active":"2024-03-06T07:28:42.615326+0000","last_peered":"2024-03-06T07:28:42.615326+0000","last_clean":"2024-03-06T07:28:42.615326+0000","last_became_active":"2024-03-06T07:28:41.600603+0000","last_became_peered":"2024-03-06T07:28:41.600603+0000","last_unstale":"2024-03-06T07:28:42.615326+0000","last_undegraded":"2024-03-06T07:28:42.615326+0000","last_fullsized":"2024-03-06T07:28:42.615326+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:45.519515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615204+0000","last_change":"2024-03-06T07:28:40.595293+0000","last_active":"2024-03-06T07:28:42.615204+0000","last_peered":"2024-03-06T07:28:42.615204+0000","last_clean":"2024-03-06T07:28:42.615204+0000","last_became_active":"2024-03-06T07:28:40.595028+0000","last_became_peered":"2024-03-06T07:28:40.595028+0000","last_unstale":"2024-03-06T07:28:42.615204+0000","last_undegraded":"2024-03-06T07:28:42.615204+0000","last_fullsized":"2024-03-06T07:28:42.615204+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:15.972492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615314+0000","last_change":"2024-03-06T07:28:41.603184+0000","last_active":"2024-03-06T07:28:42.615314+0000","last_peered":"2024-03-06T07:28:42.615314+0000","last_clean":"2024-03-06T07:28:42.615314+0000","last_became_active":"2024-03-06T07:28:41.602913+0000","last_became_peered":"2024-03-06T07:28:41.602913+0000","last_unstale":"2024-03-06T07:28:42.615314+0000","last_undegraded":"2024-03-06T07:28:42.615314+0000","last_fullsized":"2024-03-06T07:28:42.615314+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:12.604285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584462+0000","last_change":"2024-03-06T07:28:40.593167+0000","last_active":"2024-03-06T07:28:42.584462+0000","last_peered":"2024-03-06T07:28:42.584462+0000","last_clean":"2024-03-06T07:28:42.584462+0000","last_became_active":"2024-03-06T07:28:40.592882+0000","last_became_peered":"2024-03-06T07:28:40.592882+0000","last_unstale":"2024-03-06T07:28:42.584462+0000","last_undegraded":"2024-03-06T07:28:42.584462+0000","last_fullsized":"2024-03-06T07:28:42.584462+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:41.378581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584514+0000","last_change":"2024-03-06T07:28:41.603668+0000","last_active":"2024-03-06T07:28:42.584514+0000","last_peered":"2024-03-06T07:28:42.584514+0000","last_clean":"2024-03-06T07:28:42.584514+0000","last_became_active":"2024-03-06T07:28:41.603374+0000","last_became_peered":"2024-03-06T07:28:41.603374+0000","last_unstale":"2024-03-06T07:28:42.584514+0000","last_undegraded":"2024-03-06T07:28:42.584514+0000","last_fullsized":"2024-03-06T07:28:42.584514+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:12.244667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615144+0000","last_change":"2024-03-06T07:28:40.593051+0000","last_active":"2024-03-06T07:28:42.615144+0000","last_peered":"2024-03-06T07:28:42.615144+0000","last_clean":"2024-03-06T07:28:42.615144+0000","last_became_active":"2024-03-06T07:28:40.592727+0000","last_became_peered":"2024-03-06T07:28:40.592727+0000","last_unstale":"2024-03-06T07:28:42.615144+0000","last_undegraded":"2024-03-06T07:28:42.615144+0000","last_fullsized":"2024-03-06T07:28:42.615144+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:45.038458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584929+0000","last_change":"2024-03-06T07:28:41.612638+0000","last_active":"2024-03-06T07:28:42.584929+0000","last_peered":"2024-03-06T07:28:42.584929+0000","last_clean":"2024-03-06T07:28:42.584929+0000","last_became_active":"2024-03-06T07:28:41.612281+0000","last_became_peered":"2024-03-06T07:28:41.612281+0000","last_unstale":"2024-03-06T07:28:42.584929+0000","last_undegraded":"2024-03-06T07:28:42.584929+0000","last_fullsized":"2024-03-06T07:28:42.584929+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:58.656279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584484+0000","last_change":"2024-03-06T07:28:40.593507+0000","last_active":"2024-03-06T07:28:42.584484+0000","last_peered":"2024-03-06T07:28:42.584484+0000","last_clean":"2024-03-06T07:28:42.584484+0000","last_became_active":"2024-03-06T07:28:40.593219+0000","last_became_peered":"2024-03-06T07:28:40.593219+0000","last_unstale":"2024-03-06T07:28:42.584484+0000","last_undegraded":"2024-03-06T07:28:42.584484+0000","last_fullsized":"2024-03-06T07:28:42.584484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:27.475248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615978+0000","last_change":"2024-03-06T07:28:41.598292+0000","last_active":"2024-03-06T07:28:42.615978+0000","last_peered":"2024-03-06T07:28:42.615978+0000","last_clean":"2024-03-06T07:28:42.615978+0000","last_became_active":"2024-03-06T07:28:41.598002+0000","last_became_peered":"2024-03-06T07:28:41.598002+0000","last_unstale":"2024-03-06T07:28:42.615978+0000","last_undegraded":"2024-03-06T07:28:42.615978+0000","last_fullsized":"2024-03-06T07:28:42.615978+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:16.974584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.585079+0000","last_change":"2024-03-06T07:28:40.597305+0000","last_active":"2024-03-06T07:28:42.585079+0000","last_peered":"2024-03-06T07:28:42.585079+0000","last_clean":"2024-03-06T07:28:42.585079+0000","last_became_active":"2024-03-06T07:28:40.596994+0000","last_became_peered":"2024-03-06T07:28:40.596994+0000","last_unstale":"2024-03-06T07:28:42.585079+0000","last_undegraded":"2024-03-06T07:28:42.585079+0000","last_fullsized":"2024-03-06T07:28:42.585079+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:40.886362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.614978+0000","last_change":"2024-03-06T07:28:41.605397+0000","last_active":"2024-03-06T07:28:42.614978+0000","last_peered":"2024-03-06T07:28:42.614978+0000","last_clean":"2024-03-06T07:28:42.614978+0000","last_became_active":"2024-03-06T07:28:41.604725+0000","last_became_peered":"2024-03-06T07:28:41.604725+0000","last_unstale":"2024-03-06T07:28:42.614978+0000","last_undegraded":"2024-03-06T07:28:42.614978+0000","last_fullsized":"2024-03-06T07:28:42.614978+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:21.039498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615436+0000","last_change":"2024-03-06T07:28:40.596580+0000","last_active":"2024-03-06T07:28:42.615436+0000","last_peered":"2024-03-06T07:28:42.615436+0000","last_clean":"2024-03-06T07:28:42.615436+0000","last_became_active":"2024-03-06T07:28:40.596067+0000","last_became_peered":"2024-03-06T07:28:40.596067+0000","last_unstale":"2024-03-06T07:28:42.615436+0000","last_undegraded":"2024-03-06T07:28:42.615436+0000","last_fullsized":"2024-03-06T07:28:42.615436+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:49.059409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.584162+0000","last_change":"2024-03-06T07:28:41.593409+0000","last_active":"2024-03-06T07:28:42.584162+0000","last_peered":"2024-03-06T07:28:42.584162+0000","last_clean":"2024-03-06T07:28:42.584162+0000","last_became_active":"2024-03-06T07:28:41.592822+0000","last_became_peered":"2024-03-06T07:28:41.592822+0000","last_unstale":"2024-03-06T07:28:42.584162+0000","last_undegraded":"2024-03-06T07:28:42.584162+0000","last_fullsized":"2024-03-06T07:28:42.584162+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.570920+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.570920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:55.837869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615076+0000","last_change":"2024-03-06T07:28:40.598826+0000","last_active":"2024-03-06T07:28:42.615076+0000","last_peered":"2024-03-06T07:28:42.615076+0000","last_clean":"2024-03-06T07:28:42.615076+0000","last_became_active":"2024-03-06T07:28:40.598534+0000","last_became_peered":"2024-03-06T07:28:40.598534+0000","last_unstale":"2024-03-06T07:28:42.615076+0000","last_undegraded":"2024-03-06T07:28:42.615076+0000","last_fullsized":"2024-03-06T07:28:42.615076+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.568828+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.568828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:11.019639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":348,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.615864+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:28:42.615864+0000","last_peered":"2024-03-06T07:28:42.615864+0000","last_clean":"2024-03-06T07:28:42.615864+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:28:42.615864+0000","last_undegraded":"2024-03-06T07:28:42.615864+0000","last_fullsized":"2024-03-06T07:28:42.615864+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37316,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334524,"statfs":{"total":96636764160,"available":96598552576,"internally_reserved":0,"allocated":2019328,"data_stored":1201971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574927,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40896,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330944,"statfs":{"total":96636764160,"available":96594886656,"internally_reserved":0,"allocated":1421312,"data_stored":611057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40900,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330940,"statfs":{"total":96636764160,"available":96594882560,"internally_reserved":0,"allocated":1425408,"data_stored":611591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40888,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330952,"statfs":{"total":96636764160,"available":96594894848,"internally_reserved":0,"allocated":1417216,"data_stored":610987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40908,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330932,"statfs":{"total":96636764160,"available":96594874368,"internally_reserved":0,"allocated":1433600,"data_stored":612669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40892,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330948,"statfs":{"total":96636764160,"available":96594890752,"internally_reserved":0,"allocated":1417216,"data_stored":610987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40900,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330940,"statfs":{"total":96636764160,"available":96594882560,"internally_reserved":0,"allocated":1425408,"data_stored":611111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574927,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41488,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330352,"statfs":{"total":96636764160,"available":96594280448,"internally_reserved":0,"allocated":2031616,"data_stored":1203079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:50.456 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:28:50.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:28:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.605+0000 7faa8c015640 1 Processor -- start 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.605+0000 7faa8c015640 1 -- start start 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.606+0000 7faa8c015640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa840cf0f0 0x7faa84071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.606+0000 7faa8c015640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa840724b0 0x7faa840d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.606+0000 7faa8c015640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa84072cf0 con 0x7faa8415dd50 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.606+0000 7faa8c015640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa84072fd0 con 0x7faa840724b0 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.606+0000 7faa8c015640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa840d1f60 con 0x7faa840cf0f0 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.607+0000 7faa895b0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa840cf0f0 0x7faa84071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.607+0000 7faa8a5b2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa840724b0 0x7faa840d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.607+0000 7faa8a5b2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faa840724b0 0x7faa840d2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:40278/0 (socket says 172.21.15.27:40278) 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.607+0000 7faa8a5b2640 1 -- 172.21.15.27:0/2018090388 learned_addr learned my addr 172.21.15.27:0/2018090388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.607+0000 7faa8a5b2640 1 -- 172.21.15.27:0/2018090388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa840cf0f0 msgr2=0x7faa84071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.607+0000 7faa8a5b2640 1 --2- 172.21.15.27:0/2018090388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa840cf0f0 0x7faa84071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.607+0000 7faa8a5b2640 1 -- 172.21.15.27:0/2018090388 >> v1:172.21.15.69:6789/0 conn(0x7faa8415dd50 legacy=0x7faa840ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.607+0000 7faa8a5b2640 1 -- 172.21.15.27:0/2018090388 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa840d2240 con 0x7faa840724b0 2024-03-06T07:28:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.607+0000 7faa895b0640 1 --2- 172.21.15.27:0/2018090388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa840cf0f0 0x7faa84071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.608+0000 7faa8a5b2640 1 --2- 172.21.15.27:0/2018090388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa840724b0 0x7faa840d2760 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009e50 tx=0x7faa7c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aec836f079ffb37e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.608+0000 7faa88daf640 1 -- 172.21.15.27:0/2018090388 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c02ca70 con 0x7faa840724b0 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.608+0000 7faa88daf640 1 -- 172.21.15.27:0/2018090388 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c02cc10 con 0x7faa840724b0 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.608+0000 7faa88daf640 1 -- 172.21.15.27:0/2018090388 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c0343f0 con 0x7faa840724b0 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.608+0000 7faa8c015640 1 -- 172.21.15.27:0/2018090388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa840724b0 msgr2=0x7faa840d2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.608+0000 7faa8c015640 1 --2- 172.21.15.27:0/2018090388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa840724b0 0x7faa840d2760 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009e50 tx=0x7faa7c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.608+0000 7faa8c015640 1 -- 172.21.15.27:0/2018090388 shutdown_connections 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.608+0000 7faa8c015640 1 --2- 172.21.15.27:0/2018090388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa840724b0 0x7faa840d2760 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.608+0000 7faa8c015640 1 --2- 172.21.15.27:0/2018090388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa840cf0f0 0x7faa84071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.609+0000 7faa8c015640 1 -- 172.21.15.27:0/2018090388 >> 172.21.15.27:0/2018090388 conn(0x7faa840d3ef0 msgr2=0x7faa840d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.609+0000 7faa8c015640 1 -- 172.21.15.27:0/2018090388 shutdown_connections 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.609+0000 7faa8c015640 1 -- 172.21.15.27:0/2018090388 wait complete. 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.610+0000 7faa8c015640 1 Processor -- start 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.610+0000 7faa8c015640 1 -- start start 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.610+0000 7faa8c015640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa840724b0 0x7faa84149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.610+0000 7faa8c015640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa840cf0f0 0x7faa840cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa8c015640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8415dd50 0x7faa8415bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa8c015640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8415e2b0 con 0x7faa840724b0 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa8c015640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8415e130 con 0x7faa840cf0f0 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa8c015640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa840d2ca0 con 0x7faa8415dd50 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa89db1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa840724b0 0x7faa84149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa89db1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa840724b0 0x7faa84149070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44678/0 (socket says 172.21.15.27:44678) 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa89db1640 1 -- 172.21.15.27:0/3241325464 learned_addr learned my addr 172.21.15.27:0/3241325464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa8a5b2640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8415dd50 0x7faa8415bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa895b0640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa840cf0f0 0x7faa840cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa89db1640 1 -- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8415dd50 msgr2=0x7faa8415bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa89db1640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8415dd50 0x7faa8415bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa89db1640 1 -- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa840cf0f0 msgr2=0x7faa840cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa89db1640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa840cf0f0 0x7faa840cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa89db1640 1 -- 172.21.15.27:0/3241325464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84159a30 con 0x7faa840724b0 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa89db1640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa840724b0 0x7faa84149070 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7faa8000c3f0 tx=0x7faa8000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa767fc640 1 -- 172.21.15.27:0/3241325464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa800052d0 con 0x7faa840724b0 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa767fc640 1 -- 172.21.15.27:0/3241325464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa80005490 con 0x7faa840724b0 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa84159d10 con 0x7faa840724b0 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa767fc640 1 -- 172.21.15.27:0/3241325464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa80007260 con 0x7faa840724b0 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.611+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8415a2d0 con 0x7faa840724b0 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.612+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8406cdd0 con 0x7faa840724b0 2024-03-06T07:28:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.613+0000 7faa767fc640 1 -- 172.21.15.27:0/3241325464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa80007400 con 0x7faa840724b0 2024-03-06T07:28:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.614+0000 7faa767fc640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa6806b4d0 0x7faa6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.614+0000 7faa767fc640 1 -- 172.21.15.27:0/3241325464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa800896b0 con 0x7faa840724b0 2024-03-06T07:28:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.614+0000 7faa895b0640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa6806b4d0 0x7faa6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.615+0000 7faa895b0640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa6806b4d0 0x7faa6806d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faa78009560 tx=0x7faa78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.618+0000 7faa767fc640 1 -- 172.21.15.27:0/3241325464 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa80058530 con 0x7faa840724b0 2024-03-06T07:28:50.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.870+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7faa8404f410 con 0x7faa840724b0 2024-03-06T07:28:50.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.872+0000 7faa767fc640 1 -- 172.21.15.27:0/3241325464 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7faa80057e30 con 0x7faa840724b0 2024-03-06T07:28:50.874 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:28:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:28:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-03-06T07:28:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-03-06T07:28:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:50.875 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-03-06T07:28:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa6806b4d0 msgr2=0x7faa6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa6806b4d0 0x7faa6806d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faa78009560 tx=0x7faa78008040 comp rx=0 tx=0).stop 2024-03-06T07:28:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa840724b0 msgr2=0x7faa84149070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa840724b0 0x7faa84149070 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7faa8000c3f0 tx=0x7faa8000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 shutdown_connections 2024-03-06T07:28:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa6806b4d0 0x7faa6806d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa8415dd50 0x7faa8415bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa840cf0f0 0x7faa840cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 --2- 172.21.15.27:0/3241325464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa840724b0 0x7faa84149070 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 >> 172.21.15.27:0/3241325464 conn(0x7faa840d3ef0 msgr2=0x7faa841582f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.874+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 shutdown_connections 2024-03-06T07:28:50.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.875+0000 7faa8c015640 1 -- 172.21.15.27:0/3241325464 wait complete. 2024-03-06T07:28:50.888 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:50.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.994+0000 7f6c593c7640 1 Processor -- start 2024-03-06T07:28:50.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.994+0000 7f6c593c7640 1 -- start start 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.994+0000 7f6c593c7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6c540cea70 0x7f6c540d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.994+0000 7f6c593c7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c540cd350 con 0x7f6c540b0ed0 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.994+0000 7f6c593c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c540cd630 con 0x7f6c5415bda0 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.994+0000 7f6c593c7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c540b1360 con 0x7f6c540cea70 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c537fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6c540b0ed0 0x7f6c5406f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34832/0 (socket says 172.21.15.27:34832) 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c537fe640 1 -- 172.21.15.27:0/3390220109 learned_addr learned my addr 172.21.15.27:0/3390220109 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990837552 0 0) 0x7f6c540cd350 con 0x7f6c540b0ed0 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c527fc640 1 --2- 172.21.15.27:0/3390220109 >> v2:172.21.15.69:3300/0 conn(0x7f6c540cea70 0x7f6c540d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c003520 con 0x7f6c540b0ed0 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 102637972 0 0) 0x7f6c3c003520 con 0x7f6c540b0ed0 2024-03-06T07:28:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 >> v2:172.21.15.69:3300/0 conn(0x7f6c540cea70 msgr2=0x7f6c540d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 --2- 172.21.15.27:0/3390220109 >> v2:172.21.15.69:3300/0 conn(0x7f6c540cea70 0x7f6c540d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 >> v1:172.21.15.69:6789/0 conn(0x7f6c5415bda0 legacy=0x7f6c540ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540cd910 con 0x7f6c540b0ed0 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f6c4c0019f0 con 0x7f6c540b0ed0 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 >> v1:172.21.15.27:6789/0 conn(0x7f6c540b0ed0 legacy=0x7f6c5406f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c3c0039b0 0x7f6c3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c004800 0x7f6c3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c0051c0 0x7f6c3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c00c030 con 0x7f6c3c0039b0 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c00c1e0 con 0x7f6c3c004800 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c00c3e0 con 0x7f6c3c0051c0 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c52ffd640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c3c0039b0 0x7f6c3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.995+0000 7f6c527fc640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c004800 0x7f6c3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.996+0000 7f6c537fe640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c0051c0 0x7f6c3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.996+0000 7f6c52ffd640 1 -- 172.21.15.27:0/3390220109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c0051c0 msgr2=0x7f6c3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.996+0000 7f6c52ffd640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c0051c0 0x7f6c3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.996+0000 7f6c52ffd640 1 -- 172.21.15.27:0/3390220109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c004800 msgr2=0x7f6c3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.996+0000 7f6c52ffd640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c3c004800 0x7f6c3c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.996+0000 7f6c52ffd640 1 -- 172.21.15.27:0/3390220109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c003520 con 0x7f6c3c0039b0 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.996+0000 7f6c537fe640 1 -- 172.21.15.27:0/3390220109 reap_dead start 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.996+0000 7f6c52ffd640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c3c0039b0 0x7f6c3c003dd0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6c44001750 tx=0x7f6c3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864a5b6ade484193 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44001370 con 0x7f6c3c0039b0 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c440052d0 con 0x7f6c3c0039b0 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c51ffb640 1 -- 172.21.15.27:0/3390220109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44003830 con 0x7f6c3c0039b0 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c593c7640 1 -- 172.21.15.27:0/3390220109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c3c0039b0 msgr2=0x7f6c3c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c593c7640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c3c0039b0 0x7f6c3c003dd0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6c44001750 tx=0x7f6c3c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c593c7640 1 -- 172.21.15.27:0/3390220109 shutdown_connections 2024-03-06T07:28:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c593c7640 1 --2- 172.21.15.27:0/3390220109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c3c0039b0 0x7f6c3c003dd0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c593c7640 1 -- 172.21.15.27:0/3390220109 >> 172.21.15.27:0/3390220109 conn(0x7f6c540d1a90 msgr2=0x7f6c540d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c593c7640 1 -- 172.21.15.27:0/3390220109 shutdown_connections 2024-03-06T07:28:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.997+0000 7f6c593c7640 1 -- 172.21.15.27:0/3390220109 wait complete. 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.998+0000 7f6c593c7640 1 Processor -- start 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.998+0000 7f6c593c7640 1 -- start start 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c593c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c0039b0 0x7f6c540b5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c593c7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c54156320 0x7f6c541555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c593c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c54155d60 0x7f6c540b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c593c7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5415c180 con 0x7f6c54156320 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c593c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5406eb20 con 0x7f6c54155d60 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c593c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5415c300 con 0x7f6c3c0039b0 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c527fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c54156320 0x7f6c541555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c527fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c54156320 0x7f6c541555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44702/0 (socket says 172.21.15.27:44702) 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c527fc640 1 -- 172.21.15.27:0/1087077059 learned_addr learned my addr 172.21.15.27:0/1087077059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c52ffd640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c0039b0 0x7f6c540b5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:50.999+0000 7f6c537fe640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c54155d60 0x7f6c540b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c527fc640 1 -- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c0039b0 msgr2=0x7f6c540b5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c527fc640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c0039b0 0x7f6c540b5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c527fc640 1 -- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c54155d60 msgr2=0x7f6c540b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c527fc640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c54155d60 0x7f6c540b6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c527fc640 1 -- 172.21.15.27:0/1087077059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540b62a0 con 0x7f6c54156320 2024-03-06T07:28:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c527fc640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c54156320 0x7f6c541555b0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f6c48011aa0 tx=0x7f6c48011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c3b7fe640 1 -- 172.21.15.27:0/1087077059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4800fb00 con 0x7f6c54156320 2024-03-06T07:28:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c3b7fe640 1 -- 172.21.15.27:0/1087077059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4800fca0 con 0x7f6c54156320 2024-03-06T07:28:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c540b4cc0 con 0x7f6c54156320 2024-03-06T07:28:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c3b7fe640 1 -- 172.21.15.27:0/1087077059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48014660 con 0x7f6c54156320 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.000+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c540b5280 con 0x7f6c54156320 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.001+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c54067ff0 con 0x7f6c54156320 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.002+0000 7f6c3b7fe640 1 -- 172.21.15.27:0/1087077059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c48012040 con 0x7f6c54156320 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.002+0000 7f6c3b7fe640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.002+0000 7f6c3b7fe640 1 -- 172.21.15.27:0/1087077059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c4808a6d0 con 0x7f6c54156320 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.002+0000 7f6c52ffd640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.003+0000 7f6c52ffd640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c4006b4d0 0x7f6c4006d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6c44033000 tx=0x7f6c44009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.006+0000 7f6c3b7fe640 1 -- 172.21.15.27:0/1087077059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c48019030 con 0x7f6c54156320 2024-03-06T07:28:51.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.224+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c540ce560 con 0x7f6c54156320 2024-03-06T07:28:51.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.225+0000 7f6c3b7fe640 1 -- 172.21.15.27:0/1087077059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f6c48058e70 con 0x7f6c54156320 2024-03-06T07:28:51.227 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:51.227 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/103754153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":103754153},{"type":"v1","addr":"172.21.15.69:6836","nonce":103754153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4078702997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":4078702997},{"type":"v1","addr":"172.21.15.69:6837","nonce":4078702997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7956,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3926923781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3926923781},{"type":"v1","addr":"172.21.15.27:6834","nonce":3926923781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:28:43.956535+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7959":{"gid":7959,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1655684069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1655684069},{"type":"v1","addr":"172.21.15.27:6835","nonce":1655684069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7959,"qdb_cluster":[7959]},"id":8}]} 2024-03-06T07:28:51.227 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 75 2024-03-06T07:28:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c4006b4d0 msgr2=0x7f6c4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c4006b4d0 0x7f6c4006d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6c44033000 tx=0x7f6c44009030 comp rx=0 tx=0).stop 2024-03-06T07:28:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c54156320 msgr2=0x7f6c541555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c54156320 0x7f6c541555b0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f6c48011aa0 tx=0x7f6c48011f60 comp rx=0 tx=0).stop 2024-03-06T07:28:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 shutdown_connections 2024-03-06T07:28:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c54155d60 0x7f6c540b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c54156320 0x7f6c541555b0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 --2- 172.21.15.27:0/1087077059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6c3c0039b0 0x7f6c540b5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 >> 172.21.15.27:0/1087077059 conn(0x7f6c540d1a90 msgr2=0x7f6c540ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 shutdown_connections 2024-03-06T07:28:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.226+0000 7f6c593c7640 1 -- 172.21.15.27:0/1087077059 wait complete. 2024-03-06T07:28:51.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:40.951614+0000', 'modified': '2024-03-06T07:28:43.956535+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7959}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7959': {'gid': 7959, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/1655684069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 1655684069}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 1655684069}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7959, 'qdb_cluster': [7959]} 2024-03-06T07:28:51.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:51.644 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:28:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.749+0000 7f7b93ba0640 1 Processor -- start 2024-03-06T07:28:51.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.749+0000 7f7b93ba0640 1 -- start start 2024-03-06T07:28:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b93ba0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b8c158520 0x7f7b8c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b93ba0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c06dd60 0x7f7b8c0ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b93ba0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8c0af080 con 0x7f7b8c0aec40 2024-03-06T07:28:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b93ba0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8c06e1c0 con 0x7f7b8c158520 2024-03-06T07:28:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b93ba0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c071b70 con 0x7f7b8c06dd60 2024-03-06T07:28:51.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b9193c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b8c158520 0x7f7b8c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b9193c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b8c158520 0x7f7b8c06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44716/0 (socket says 172.21.15.27:44716) 2024-03-06T07:28:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b9213d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7b8c0aec40 0x7f7b8c15e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34846/0 (socket says 172.21.15.27:34846) 2024-03-06T07:28:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b9193c640 1 -- 172.21.15.27:0/3245864996 learned_addr learned my addr 172.21.15.27:0/3245864996 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.750+0000 7f7b9113b640 1 --2- 172.21.15.27:0/3245864996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c06dd60 0x7f7b8c0ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.751+0000 7f7b9193c640 1 -- 172.21.15.27:0/3245864996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c06dd60 msgr2=0x7f7b8c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.751+0000 7f7b9193c640 1 --2- 172.21.15.27:0/3245864996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c06dd60 0x7f7b8c0ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.751+0000 7f7b9193c640 1 -- 172.21.15.27:0/3245864996 >> v1:172.21.15.27:6789/0 conn(0x7f7b8c0aec40 legacy=0x7f7b8c15e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.751+0000 7f7b9193c640 1 -- 172.21.15.27:0/3245864996 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c071240 con 0x7f7b8c158520 2024-03-06T07:28:51.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.751+0000 7f7b9093a640 1 -- 172.21.15.27:0/3245864996 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198930628 0 0) 0x7f7b8c0af080 con 0x7f7b8c0aec40 2024-03-06T07:28:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.751+0000 7f7b9193c640 1 --2- 172.21.15.27:0/3245864996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c158520 0x7f7b8c06d820 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009870 tx=0x7f7b8802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b247ff5e65871 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.751+0000 7f7b9093a640 1 -- 172.21.15.27:0/3245864996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88034070 con 0x7f7b8c158520 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b9093a640 1 -- 172.21.15.27:0/3245864996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b88030840 con 0x7f7b8c158520 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b9093a640 1 -- 172.21.15.27:0/3245864996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88030c10 con 0x7f7b8c158520 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3245864996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c158520 msgr2=0x7f7b8c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b93ba0640 1 --2- 172.21.15.27:0/3245864996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c158520 0x7f7b8c06d820 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009870 tx=0x7f7b8802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3245864996 shutdown_connections 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b93ba0640 1 --2- 172.21.15.27:0/3245864996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c06dd60 0x7f7b8c0ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b93ba0640 1 --2- 172.21.15.27:0/3245864996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c158520 0x7f7b8c06d820 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3245864996 >> 172.21.15.27:0/3245864996 conn(0x7f7b8c066d00 msgr2=0x7f7b8c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3245864996 shutdown_connections 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.752+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3245864996 wait complete. 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.753+0000 7f7b93ba0640 1 Processor -- start 2024-03-06T07:28:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.753+0000 7f7b93ba0640 1 -- start start 2024-03-06T07:28:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b93ba0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b8c06dd60 0x7f7b8c140e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b93ba0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c0aec40 0x7f7b8c1f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b93ba0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c158520 0x7f7b8c1f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b93ba0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c0b4650 con 0x7f7b8c0aec40 2024-03-06T07:28:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b93ba0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c071800 con 0x7f7b8c06dd60 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b93ba0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c072000 con 0x7f7b8c158520 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9113b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c0aec40 0x7f7b8c1f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9113b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c0aec40 0x7f7b8c1f7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44722/0 (socket says 172.21.15.27:44722) 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9113b640 1 -- 172.21.15.27:0/3801218738 learned_addr learned my addr 172.21.15.27:0/3801218738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9213d640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c158520 0x7f7b8c1f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9193c640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b8c06dd60 0x7f7b8c140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9113b640 1 -- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c158520 msgr2=0x7f7b8c1f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9113b640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c158520 0x7f7b8c1f7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9113b640 1 -- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b8c06dd60 msgr2=0x7f7b8c140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9113b640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b8c06dd60 0x7f7b8c140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.754+0000 7f7b9113b640 1 -- 172.21.15.27:0/3801218738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c0aff00 con 0x7f7b8c0aec40 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.755+0000 7f7b9113b640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c0aec40 0x7f7b8c1f7210 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e9b0 tx=0x7f7b7c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.755+0000 7f7b827fc640 1 -- 172.21.15.27:0/3801218738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c017070 con 0x7f7b8c0aec40 2024-03-06T07:28:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.755+0000 7f7b827fc640 1 -- 172.21.15.27:0/3801218738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7c00c9a0 con 0x7f7b8c0aec40 2024-03-06T07:28:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.755+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3801218738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c0b01e0 con 0x7f7b8c0aec40 2024-03-06T07:28:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.755+0000 7f7b827fc640 1 -- 172.21.15.27:0/3801218738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c00cd90 con 0x7f7b8c0aec40 2024-03-06T07:28:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.755+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3801218738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c0b07a0 con 0x7f7b8c0aec40 2024-03-06T07:28:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.757+0000 7f7b827fc640 1 -- 172.21.15.27:0/3801218738 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b7c01a070 con 0x7f7b8c0aec40 2024-03-06T07:28:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.757+0000 7f7b827fc640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b7006b4d0 0x7f7b7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.757+0000 7f7b827fc640 1 -- 172.21.15.27:0/3801218738 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b7c08a530 con 0x7f7b8c0aec40 2024-03-06T07:28:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.758+0000 7f7b9193c640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b7006b4d0 0x7f7b7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.758+0000 7f7b9193c640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b7006b4d0 0x7f7b7006d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7b88036000 tx=0x7f7b8803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.758+0000 7f7b96781740 1 Processor -- start 2024-03-06T07:28:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.758+0000 7f7b96781740 1 -- start start 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b96781740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e784a870 0x5604e784ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b96781740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e784e8c0 0x5604e7850360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b96781740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 0x5604e787aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b96781740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e77b46b0 con 0x5604e784a870 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b96781740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e775da60 con 0x5604e784e8c0 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b96781740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604e77b0750 con 0x5604e787a600 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9193c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e784a870 0x5604e784ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9193c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e784a870 0x5604e784ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44730/0 (socket says 172.21.15.27:44730) 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9193c640 1 -- 172.21.15.27:0/1975363778 learned_addr learned my addr 172.21.15.27:0/1975363778 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9213d640 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 0x5604e787aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9113b640 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e784e8c0 0x5604e7850360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9193c640 1 -- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 msgr2=0x5604e787aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9193c640 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 0x5604e787aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9193c640 1 -- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e784e8c0 msgr2=0x5604e7850360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9193c640 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e784e8c0 0x5604e7850360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9193c640 1 -- 172.21.15.27:0/1975363778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604e777bf30 con 0x5604e784a870 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9213d640 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 0x5604e787aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9113b640 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e784e8c0 0x5604e7850360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.759+0000 7f7b9193c640 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e784a870 0x5604e784ac50 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009960 tx=0x7f7b88030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8045a9f6ba1e93df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b6f7fe640 1 -- 172.21.15.27:0/1975363778 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88053070 con 0x5604e784a870 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b6f7fe640 1 -- 172.21.15.27:0/1975363778 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b88030e00 con 0x5604e784a870 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b6f7fe640 1 -- 172.21.15.27:0/1975363778 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88034040 con 0x5604e784a870 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b96781740 1 -- 172.21.15.27:0/1975363778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e784a870 msgr2=0x5604e784ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b96781740 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e784a870 0x5604e784ac50 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009960 tx=0x7f7b88030000 comp rx=0 tx=0).stop 2024-03-06T07:28:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b96781740 1 -- 172.21.15.27:0/1975363778 shutdown_connections 2024-03-06T07:28:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b96781740 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 0x5604e787aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b96781740 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e784e8c0 0x5604e7850360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b96781740 1 --2- 172.21.15.27:0/1975363778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e784a870 0x5604e784ac50 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b96781740 1 -- 172.21.15.27:0/1975363778 >> 172.21.15.27:0/1975363778 conn(0x5604e784c8f0 msgr2=0x5604e784e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b96781740 1 -- 172.21.15.27:0/1975363778 shutdown_connections 2024-03-06T07:28:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.760+0000 7f7b96781740 1 -- 172.21.15.27:0/1975363778 wait complete. 2024-03-06T07:28:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.761+0000 7f7b96781740 1 -- start start 2024-03-06T07:28:51.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.761+0000 7f7b96781740 1 Processor -- start 2024-03-06T07:28:51.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b96781740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7860af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b96781740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78605d0 0x5604e787cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b96781740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78755a0 0x5604e7875a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b96781740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e784ed50 con 0x5604e787a600 2024-03-06T07:28:51.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b96781740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e7770ea0 con 0x5604e78605d0 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b96781740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604e784f030 con 0x5604e78755a0 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b96781740 10 client.0 _ll_register_callbacks cb 0x5604e7770870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9193c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7860af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9193c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7860af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44744/0 (socket says 172.21.15.27:44744) 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9193c640 1 -- 172.21.15.27:0/1323772369 learned_addr learned my addr 172.21.15.27:0/1323772369 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9213d640 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78755a0 0x5604e7875a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9113b640 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78605d0 0x5604e787cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9193c640 1 -- 172.21.15.27:0/1323772369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78755a0 msgr2=0x5604e7875a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9193c640 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78755a0 0x5604e7875a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9193c640 1 -- 172.21.15.27:0/1323772369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78605d0 msgr2=0x5604e787cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9193c640 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78605d0 0x5604e787cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.762+0000 7f7b9193c640 1 -- 172.21.15.27:0/1323772369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5604e7771270 con 0x5604e787a600 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b9193c640 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7860af0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f7b880090b0 tx=0x7f7b88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91dcafc417ead90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:28:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b6cff9640 1 -- 172.21.15.27:0/1323772369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8805c060 con 0x5604e787a600 2024-03-06T07:28:51.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b6cff9640 1 -- 172.21.15.27:0/1323772369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b8804e940 con 0x5604e787a600 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b96781740 1 -- 172.21.15.27:0/1323772369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b88030e00 con 0x5604e787a600 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b6cff9640 1 -- 172.21.15.27:0/1323772369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b88034070 con 0x5604e787a600 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b96781740 10 client.8115 fetch_fsmap learned FSMap version 75 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b96781740 1 -- 172.21.15.27:0/1323772369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x5604e74c92d0 con 0x5604e787a600 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b6cff9640 1 -- 172.21.15.27:0/1323772369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b88030e00 con 0x5604e787a600 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b6cff9640 1 -- 172.21.15.27:0/1323772369 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 75) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7b8804eb40 con 0x5604e787a600 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b96781740 10 client.8115 fetch_fsmap finished waiting for FSMap version 75 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b96781740 10 client.8115 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b96781740 20 client.8115 populate_metadata read hostname 'smithi027' 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b96781740 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] conn(0x5604e78a9710 0x5604e7891c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b96781740 4 client.8115 mds_command: new command op to 7959 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:51.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b96781740 1 -- 172.21.15.27:0/1323772369 --> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604e77c1bf0 con 0x5604e78a9710 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.763+0000 7f7b9113b640 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] conn(0x5604e78a9710 0x5604e7891c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.764+0000 7f7b6cff9640 10 client.8115 ms_handle_connect on v2:172.21.15.27:6833/1655684069 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.764+0000 7f7b9113b640 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] conn(0x5604e78a9710 0x5604e7891c50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ffcf7ef720a40a server_cookie=704736bfe59bceea in_seq=0 out_seq=0 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.764+0000 7f7b6cff9640 1 -- 172.21.15.27:0/1323772369 <== mds.0 v2:172.21.15.27:6833/1655684069 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5604e77c1bf0 con 0x5604e78a9710 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.764+0000 7f7b6cff9640 10 client.8115 handle_command_reply: tid=0 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.764+0000 7f7b96781740 1 client.8115 shutdown 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 -- 172.21.15.27:0/1323772369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 msgr2=0x5604e7860af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7860af0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f7b880090b0 tx=0x7f7b88004730 comp rx=0 tx=0).stop 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 -- 172.21.15.27:0/1323772369 shutdown_connections 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] conn(0x5604e78a9710 0x5604e7891c50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78755a0 0x5604e7875a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b6cff9640 0 client.8115 ms_handle_reset on v2:172.21.15.27:6833/1655684069 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78605d0 0x5604e787cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 --2- 172.21.15.27:0/1323772369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7860af0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 -- 172.21.15.27:0/1323772369 >> 172.21.15.27:0/1323772369 conn(0x5604e784a870 msgr2=0x5604e78785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 -- 172.21.15.27:0/1323772369 shutdown_connections 2024-03-06T07:28:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 1 -- 172.21.15.27:0/1323772369 wait complete. 2024-03-06T07:28:51.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.765+0000 7f7b96781740 20 client.8115 trim_cache size 0 max 16384 2024-03-06T07:28:51.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.786+0000 7f7b96781740 1 Processor -- start 2024-03-06T07:28:51.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b96781740 1 -- start start 2024-03-06T07:28:51.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b96781740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78605d0 0x5604e7873f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b96781740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 0x5604e7879970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b96781740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7879eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b96781740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c1ffd90 con 0x5604e787a600 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b96781740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c168420 con 0x5604e78755a0 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b96781740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c0cdd40 con 0x5604e78605d0 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b9213d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7879eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b9213d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7879eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44758/0 (socket says 172.21.15.27:44758) 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.787+0000 7f7b9213d640 1 -- 172.21.15.27:0/2414440457 learned_addr learned my addr 172.21.15.27:0/2414440457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b9113b640 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 0x5604e7879970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b9193c640 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78605d0 0x5604e7873f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b9213d640 1 -- 172.21.15.27:0/2414440457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78605d0 msgr2=0x5604e7873f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b9213d640 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78605d0 0x5604e7873f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b9213d640 1 -- 172.21.15.27:0/2414440457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 msgr2=0x5604e7879970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b9213d640 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 0x5604e7879970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b9213d640 1 -- 172.21.15.27:0/2414440457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604e7875d90 con 0x5604e787a600 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b9113b640 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 0x5604e7879970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b9213d640 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7879eb0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f7b84001cc0 tx=0x7f7b84009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd666864271db2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b6effd640 1 -- 172.21.15.27:0/2414440457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84007830 con 0x5604e787a600 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b6effd640 1 -- 172.21.15.27:0/2414440457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b840079d0 con 0x5604e787a600 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.788+0000 7f7b6effd640 1 -- 172.21.15.27:0/2414440457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84016650 con 0x5604e787a600 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 -- 172.21.15.27:0/2414440457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 msgr2=0x5604e7879eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7879eb0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f7b84001cc0 tx=0x7f7b84009040 comp rx=0 tx=0).stop 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 -- 172.21.15.27:0/2414440457 shutdown_connections 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e787a600 0x5604e7879eb0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 0x5604e7879970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 --2- 172.21.15.27:0/2414440457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e78605d0 0x5604e7873f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 -- 172.21.15.27:0/2414440457 >> 172.21.15.27:0/2414440457 conn(0x5604e784a870 msgr2=0x5604e788efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 -- 172.21.15.27:0/2414440457 shutdown_connections 2024-03-06T07:28:51.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 -- 172.21.15.27:0/2414440457 wait complete. 2024-03-06T07:28:51.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 -- start start 2024-03-06T07:28:51.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.789+0000 7f7b96781740 1 Processor -- start 2024-03-06T07:28:51.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b96781740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e78605d0 0x5604e7879740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b96781740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 0x5604e7879c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b96781740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 0x5604e78afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b96781740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e787bdc0 con 0x5604e78605d0 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b96781740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c167e20 con 0x5604e78755a0 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b96781740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604e7861320 con 0x5604e787a600 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b96781740 10 client.0 _ll_register_callbacks cb 0x7f7b88039ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b9193c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e78605d0 0x5604e7879740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b9193c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e78605d0 0x5604e7879740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44764/0 (socket says 172.21.15.27:44764) 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b9193c640 1 -- 172.21.15.27:0/307027921 learned_addr learned my addr 172.21.15.27:0/307027921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b9213d640 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 0x5604e78afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.790+0000 7f7b9113b640 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 0x5604e7879c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b9193c640 1 -- 172.21.15.27:0/307027921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 msgr2=0x5604e78afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b9193c640 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 0x5604e78afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b9193c640 1 -- 172.21.15.27:0/307027921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 msgr2=0x5604e7879c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b9193c640 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 0x5604e7879c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b9193c640 1 -- 172.21.15.27:0/307027921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5604e75936b0 con 0x5604e78605d0 2024-03-06T07:28:51.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b9193c640 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e78605d0 0x5604e7879740 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f7b88000c00 tx=0x7f7b88008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b073929a292f06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b6cff9640 1 -- 172.21.15.27:0/307027921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8806b060 con 0x5604e78605d0 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b6cff9640 1 -- 172.21.15.27:0/307027921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b88030e00 con 0x5604e78605d0 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b96781740 1 -- 172.21.15.27:0/307027921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b840079d0 con 0x5604e78605d0 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b6cff9640 1 -- 172.21.15.27:0/307027921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b88071050 con 0x5604e78605d0 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b96781740 10 client.8121 fetch_fsmap learned FSMap version 75 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b96781740 1 -- 172.21.15.27:0/307027921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x5604e78b0900 con 0x5604e78605d0 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b6cff9640 1 -- 172.21.15.27:0/307027921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b840079d0 con 0x5604e78605d0 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b6cff9640 1 -- 172.21.15.27:0/307027921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 75) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7b88074020 con 0x5604e78605d0 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.791+0000 7f7b96781740 10 client.8121 fetch_fsmap finished waiting for FSMap version 75 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 10 client.8121 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 20 client.8121 populate_metadata read hostname 'smithi027' 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] conn(0x5604e78b02c0 0x5604e784ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 4 client.8121 mds_command: new command op to 7959 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 -- 172.21.15.27:0/307027921 --> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5604e77bbf40 con 0x5604e78b02c0 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b9113b640 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] conn(0x5604e78b02c0 0x5604e784ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b9113b640 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] conn(0x5604e78b02c0 0x5604e784ef60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a927d8224926c22 server_cookie=6f081f5c4066deba in_seq=0 out_seq=0 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b6cff9640 10 client.8121 ms_handle_connect on v2:172.21.15.27:6833/1655684069 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b6cff9640 1 -- 172.21.15.27:0/307027921 <== mds.0 v2:172.21.15.27:6833/1655684069 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5604e77bbf40 con 0x5604e78b02c0 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b6cff9640 10 client.8121 handle_command_reply: tid=0 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 client.8121 shutdown 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 -- 172.21.15.27:0/307027921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e78605d0 msgr2=0x5604e7879740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e78605d0 0x5604e7879740 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f7b88000c00 tx=0x7f7b88008030 comp rx=0 tx=0).stop 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 -- 172.21.15.27:0/307027921 shutdown_connections 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.27:6833/1655684069,v1:172.21.15.27:6835/1655684069] conn(0x5604e78b02c0 0x5604e784ef60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5604e787a600 0x5604e78afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5604e78755a0 0x5604e7879c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b6cff9640 0 client.8121 ms_handle_reset on v2:172.21.15.27:6833/1655684069 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 --2- 172.21.15.27:0/307027921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604e78605d0 0x5604e7879740 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 -- 172.21.15.27:0/307027921 >> 172.21.15.27:0/307027921 conn(0x5604e784a870 msgr2=0x5604e7874470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 -- 172.21.15.27:0/307027921 shutdown_connections 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.792+0000 7f7b96781740 1 -- 172.21.15.27:0/307027921 wait complete. 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b96781740 20 client.8121 trim_cache size 0 max 16384 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:28:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7959, 2024-03-06T07:28:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:28:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:28:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:28:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.214188351000001, 2024-03-06T07:28:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 75, 2024-03-06T07:28:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 114, 2024-03-06T07:28:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 114, 2024-03-06T07:28:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 15.526955515999999 2024-03-06T07:28:51.802 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:28:51.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b7006b4d0 msgr2=0x7f7b7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b7006b4d0 0x7f7b7006d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7b88036000 tx=0x7f7b8803b040 comp rx=0 tx=0).stop 2024-03-06T07:28:51.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3801218738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c0aec40 msgr2=0x7f7b8c1f7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c0aec40 0x7f7b8c1f7210 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e9b0 tx=0x7f7b7c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:28:51.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3801218738 shutdown_connections 2024-03-06T07:28:51.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b7006b4d0 0x7f7b7006d990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b8c158520 0x7f7b8c1f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b8c0aec40 0x7f7b8c1f7210 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 --2- 172.21.15.27:0/3801218738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b8c06dd60 0x7f7b8c140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3801218738 >> 172.21.15.27:0/3801218738 conn(0x7f7b8c066d00 msgr2=0x7f7b8c158d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3801218738 shutdown_connections 2024-03-06T07:28:51.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.793+0000 7f7b93ba0640 1 -- 172.21.15.27:0/3801218738 wait complete. 2024-03-06T07:28:51.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:51.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.916+0000 7f6b11b81640 1 Processor -- start 2024-03-06T07:28:51.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.916+0000 7f6b11b81640 1 -- start start 2024-03-06T07:28:51.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.917+0000 7f6b11b81640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6b0c0b2b50 0x7f6b0c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.917+0000 7f6b11b81640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0c0ce610 con 0x7f6b0c071540 2024-03-06T07:28:51.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.917+0000 7f6b11b81640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0c0d2920 con 0x7f6b0c0d3400 2024-03-06T07:28:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.917+0000 7f6b11b81640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0c0d2f70 con 0x7f6b0c0b2b50 2024-03-06T07:28:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.917+0000 7f6b0b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6b0c071540 0x7f6b0c071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34852/0 (socket says 172.21.15.27:34852) 2024-03-06T07:28:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.917+0000 7f6b0b7fe640 1 -- 172.21.15.27:0/4002059565 learned_addr learned my addr 172.21.15.27:0/4002059565 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.918+0000 7f6b0bfff640 1 --2- 172.21.15.27:0/4002059565 >> v2:172.21.15.69:3300/0 conn(0x7f6b0c0b2b50 0x7f6b0c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.918+0000 7f6b0a7fc640 1 -- 172.21.15.27:0/4002059565 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743748538 0 0) 0x7f6b0c0ce610 con 0x7f6b0c071540 2024-03-06T07:28:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.918+0000 7f6b0a7fc640 1 -- 172.21.15.27:0/4002059565 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af4003520 con 0x7f6b0c071540 2024-03-06T07:28:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.918+0000 7f6b0bfff640 1 -- 172.21.15.27:0/4002059565 >> v1:172.21.15.69:6789/0 conn(0x7f6b0c0d3400 legacy=0x7f6b0c0b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.918+0000 7f6b0bfff640 1 -- 172.21.15.27:0/4002059565 >> v1:172.21.15.27:6789/0 conn(0x7f6b0c071540 legacy=0x7f6b0c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.918+0000 7f6b0bfff640 1 -- 172.21.15.27:0/4002059565 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c0d2c00 con 0x7f6b0c0b2b50 2024-03-06T07:28:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b0bfff640 1 --2- 172.21.15.27:0/4002059565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 0x7f6b0c0b30a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009d90 tx=0x7f6b0402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d74cb3b57c25404c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b0a7fc640 1 -- 172.21.15.27:0/4002059565 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04039070 con 0x7f6b0c0b2b50 2024-03-06T07:28:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b0a7fc640 1 -- 172.21.15.27:0/4002059565 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0402cd00 con 0x7f6b0c0b2b50 2024-03-06T07:28:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b0a7fc640 1 -- 172.21.15.27:0/4002059565 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04034640 con 0x7f6b0c0b2b50 2024-03-06T07:28:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b11b81640 1 -- 172.21.15.27:0/4002059565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 msgr2=0x7f6b0c0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b11b81640 1 --2- 172.21.15.27:0/4002059565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 0x7f6b0c0b30a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009d90 tx=0x7f6b0402bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:51.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b11b81640 1 -- 172.21.15.27:0/4002059565 shutdown_connections 2024-03-06T07:28:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b11b81640 1 --2- 172.21.15.27:0/4002059565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 0x7f6b0c0b30a0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b11b81640 1 -- 172.21.15.27:0/4002059565 >> 172.21.15.27:0/4002059565 conn(0x7f6b0c0d3ca0 msgr2=0x7f6b0c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b11b81640 1 -- 172.21.15.27:0/4002059565 shutdown_connections 2024-03-06T07:28:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.919+0000 7f6b11b81640 1 -- 172.21.15.27:0/4002059565 wait complete. 2024-03-06T07:28:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.921+0000 7f6b11b81640 1 Processor -- start 2024-03-06T07:28:51.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.921+0000 7f6b11b81640 1 -- start start 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.921+0000 7f6b11b81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b0c071540 0x7f6b0c14d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.921+0000 7f6b11b81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 0x7f6b0c0ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.921+0000 7f6b11b81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0d3400 0x7f6b0c0ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.921+0000 7f6b11b81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c15e370 con 0x7f6b0c071540 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.921+0000 7f6b11b81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c0d4be0 con 0x7f6b0c0b2b50 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.921+0000 7f6b11b81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c1509b0 con 0x7f6b0c0d3400 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b0c071540 0x7f6b0c14d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b0c071540 0x7f6b0c14d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44780/0 (socket says 172.21.15.27:44780) 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0b7fe640 1 -- 172.21.15.27:0/2170455175 learned_addr learned my addr 172.21.15.27:0/2170455175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0affd640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 0x7f6b0c0ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0bfff640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0d3400 0x7f6b0c0ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0b7fe640 1 -- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0d3400 msgr2=0x7f6b0c0ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0b7fe640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0d3400 0x7f6b0c0ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0b7fe640 1 -- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 msgr2=0x7f6b0c0ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0b7fe640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 0x7f6b0c0ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0b7fe640 1 -- 172.21.15.27:0/2170455175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c14d610 con 0x7f6b0c071540 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0affd640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 0x7f6b0c0ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b0b7fe640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b0c071540 0x7f6b0c14d0d0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00c500 tx=0x7f6afc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6af3fff640 1 -- 172.21.15.27:0/2170455175 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc017070 con 0x7f6b0c071540 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6af3fff640 1 -- 172.21.15.27:0/2170455175 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc005ce0 con 0x7f6b0c071540 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0c0b3580 con 0x7f6b0c071540 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6af3fff640 1 -- 172.21.15.27:0/2170455175 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc007330 con 0x7f6b0c071540 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.922+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0c0b3b40 con 0x7f6b0c071540 2024-03-06T07:28:51.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.923+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0c0d2640 con 0x7f6b0c071540 2024-03-06T07:28:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.924+0000 7f6af3fff640 1 -- 172.21.15.27:0/2170455175 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6afc01a070 con 0x7f6b0c071540 2024-03-06T07:28:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.924+0000 7f6af3fff640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6af806b4d0 0x7f6af806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.925+0000 7f6af3fff640 1 -- 172.21.15.27:0/2170455175 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6afc089630 con 0x7f6b0c071540 2024-03-06T07:28:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.925+0000 7f6b0affd640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6af806b4d0 0x7f6af806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.925+0000 7f6b0affd640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6af806b4d0 0x7f6af806d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0c84c0 tx=0x7f6b00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:51.929+0000 7f6af3fff640 1 -- 172.21.15.27:0/2170455175 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b0c0d2640 con 0x7f6b0c071540 2024-03-06T07:28:52.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.151+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6b0c0b49b0 con 0x7f6b0c071540 2024-03-06T07:28:52.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.152+0000 7f6af3fff640 1 -- 172.21.15.27:0/2170455175 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v75) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6b0c14d610 con 0x7f6b0c071540 2024-03-06T07:28:52.154 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6af806b4d0 msgr2=0x7f6af806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6af806b4d0 0x7f6af806d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0c84c0 tx=0x7f6b00009040 comp rx=0 tx=0).stop 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b0c071540 msgr2=0x7f6b0c14d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b0c071540 0x7f6b0c14d0d0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00c500 tx=0x7f6afc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 shutdown_connections 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6af806b4d0 0x7f6af806d990 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b0c0d3400 0x7f6b0c0ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b0c0b2b50 0x7f6b0c0ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 --2- 172.21.15.27:0/2170455175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b0c071540 0x7f6b0c14d0d0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 >> 172.21.15.27:0/2170455175 conn(0x7f6b0c0d3ca0 msgr2=0x7f6b0c160bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 shutdown_connections 2024-03-06T07:28:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:52.154+0000 7f6b11b81640 1 -- 172.21.15.27:0/2170455175 wait complete. 2024-03-06T07:28:52.167 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:52.168 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:52.168 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:28:52.168 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:28:52.168 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:52.168 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:52.168 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:52.168 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:52.168 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:52.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.168 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:28:52.184 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.184 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42358sec preferred_lft 42358sec 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:28:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.187 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:28:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.187 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:52.253 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:28:52.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.255 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:28:52.311 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:28:52.311 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:52.311 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:28:52.311 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:28:52.311 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:52.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.311 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:28:52.312 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:28:52.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:28:52.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:28:52.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:28:52.312 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:28:52.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.417 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:28:52.432 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:28:52.432 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:28:52.433 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:52.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:28:52.503 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:28:52.504 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:52.504 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:52.559 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:52.559 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:52.559 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:52.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:28:52.685 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:28:52.741 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:28:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:52.743 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:52.743 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:52.743 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:52.743 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:28:52.743 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:52.743 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:52.743 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:52.743 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:52.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.743 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:28:52.796 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:28:52.796 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:28:52.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:28:52.869 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:52.926 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:28:52.926 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:28:52.926 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:28:52.926 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:28:52.926 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:28:52.926 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:28:52.926 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:28:52.926 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:28:52.926 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:52.927 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:52.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.929 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:28:52.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:28:52.931+0000 7fdfb2579480 -1 init, newargv = 0x5617852ba7f0 newargc=15 2024-03-06T07:28:52.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[73048]: starting ceph client 2024-03-06T07:28:52.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[73048]: starting fuse 2024-03-06T07:28:52.950 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:28:52.951 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:28:52.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:28:52.951 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:28:52.951 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:28:52.951 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:28:52.951 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:28:52.951 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:28:52.951 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:28:52.951 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:28:52.952 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:28:52.953 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:28:52.953 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:28:52.953 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:28:52.953 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:28:52.953 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:28:53.076 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.73048.asok 2024-03-06T07:28:53.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:53.087 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.73048.asok 2024-03-06T07:28:53.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.73048.asok status 2024-03-06T07:28:53.230 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:28:53.230 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:28:53.230 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:28:53.230 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:28:53.230 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "73048", 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8133, 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 8133 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:53.231 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2641926029 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2641926029 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.8133 192.168.0.1:0/2641926029", 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/2641926029", 2024-03-06T07:28:53.232 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:28:53.233 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 75, 2024-03-06T07:28:53.233 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 114, 2024-03-06T07:28:53.233 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:28:53.233 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:28:53.233 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:28:53.233 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:28:53.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.239 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:28:53.239 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:53.254 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:28:53.254 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:53.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:53.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.325 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:28:53.386 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:28:53.386 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:53.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.441 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:28:53.564 INFO:teuthology.orchestra.run.smithi027.stderr:8+0 records in 2024-03-06T07:28:53.564 INFO:teuthology.orchestra.run.smithi027.stderr:8+0 records out 2024-03-06T07:28:53.564 INFO:teuthology.orchestra.run.smithi027.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0661082 s, 127 MB/s 2024-03-06T07:28:53.565 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:28:53.565 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:28:53.565 DEBUG:teuthology.orchestra.run.smithi027:> import stat 2024-03-06T07:28:53.565 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:28:53.565 DEBUG:teuthology.orchestra.run.smithi027:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:28:53.565 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:28:53.631 INFO:teuthology.orchestra.run.smithi027.stdout:1099511627777 2024-03-06T07:28:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:53.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.641 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:28:53.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.698 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:28:53.758 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:28:53.758 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:28:53.758 DEBUG:teuthology.orchestra.run.smithi027:> import stat 2024-03-06T07:28:53.758 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:28:53.758 DEBUG:teuthology.orchestra.run.smithi027:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:28:53.758 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:28:53.864 INFO:teuthology.orchestra.run.smithi027.stdout:1099511627777 2024-03-06T07:28:53.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:53.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.022+0000 7f5e62f16640 1 Processor -- start 2024-03-06T07:28:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.022+0000 7f5e62f16640 1 -- start start 2024-03-06T07:28:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e62f16640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5e5c15df00 0x7f5e5c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e62f16640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b9a30 0x7f5e5c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e62f16640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c0cea30 con 0x7f5e5c0b95a0 2024-03-06T07:28:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e62f16640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c0d2af0 con 0x7f5e5c15df00 2024-03-06T07:28:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e62f16640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0cf3c0 con 0x7f5e5c0b9a30 2024-03-06T07:28:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e60cb2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5e5c15df00 0x7f5e5c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e614b3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b9a30 0x7f5e5c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e5bfff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5e5c0b95a0 0x7f5e5c0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:37040/0 (socket says 172.21.15.27:37040) 2024-03-06T07:28:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e5bfff640 1 -- 172.21.15.27:0/3888940831 learned_addr learned my addr 172.21.15.27:0/3888940831 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e5b7fe640 1 -- 172.21.15.27:0/3888940831 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768001483 0 0) 0x7f5e5c0cea30 con 0x7f5e5c0b95a0 2024-03-06T07:28:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e5b7fe640 1 -- 172.21.15.27:0/3888940831 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e4c003520 con 0x7f5e5c0b95a0 2024-03-06T07:28:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e60cb2640 1 -- 172.21.15.27:0/3888940831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b9a30 msgr2=0x7f5e5c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e60cb2640 1 --2- 172.21.15.27:0/3888940831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b9a30 0x7f5e5c0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e60cb2640 1 -- 172.21.15.27:0/3888940831 >> v1:172.21.15.69:6789/0 conn(0x7f5e5c0b95a0 legacy=0x7f5e5c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e60cb2640 1 -- 172.21.15.27:0/3888940831 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0cf0e0 con 0x7f5e5c15df00 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.023+0000 7f5e614b3640 1 --2- 172.21.15.27:0/3888940831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b9a30 0x7f5e5c0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.024+0000 7f5e60cb2640 1 --2- 172.21.15.27:0/3888940831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e5c15df00 0x7f5e5c0d25b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5e500097a0 tx=0x7f5e5002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62acd7f9787cc815 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.024+0000 7f5e5b7fe640 1 -- 172.21.15.27:0/3888940831 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50035020 con 0x7f5e5c15df00 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.024+0000 7f5e5b7fe640 1 -- 172.21.15.27:0/3888940831 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e50031810 con 0x7f5e5c15df00 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.024+0000 7f5e5b7fe640 1 -- 172.21.15.27:0/3888940831 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50031bc0 con 0x7f5e5c15df00 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.025+0000 7f5e62f16640 1 -- 172.21.15.27:0/3888940831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e5c15df00 msgr2=0x7f5e5c0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.025+0000 7f5e62f16640 1 --2- 172.21.15.27:0/3888940831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e5c15df00 0x7f5e5c0d25b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5e500097a0 tx=0x7f5e5002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.025+0000 7f5e62f16640 1 -- 172.21.15.27:0/3888940831 shutdown_connections 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.025+0000 7f5e62f16640 1 --2- 172.21.15.27:0/3888940831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b9a30 0x7f5e5c0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.025+0000 7f5e62f16640 1 --2- 172.21.15.27:0/3888940831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e5c15df00 0x7f5e5c0d25b0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.025+0000 7f5e62f16640 1 -- 172.21.15.27:0/3888940831 >> 172.21.15.27:0/3888940831 conn(0x7f5e5c071960 msgr2=0x7f5e5c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.025+0000 7f5e62f16640 1 -- 172.21.15.27:0/3888940831 shutdown_connections 2024-03-06T07:28:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.025+0000 7f5e62f16640 1 -- 172.21.15.27:0/3888940831 wait complete. 2024-03-06T07:28:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.027+0000 7f5e62f16640 1 Processor -- start 2024-03-06T07:28:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.027+0000 7f5e62f16640 1 -- start start 2024-03-06T07:28:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e62f16640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b95a0 0x7f5e5c14e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e62f16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c15df00 0x7f5e5c0b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e62f16640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e5c0b83d0 0x7f5e5c1603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e62f16640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0cec70 con 0x7f5e5c15df00 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e62f16640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0ce750 con 0x7f5e5c0b83d0 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e62f16640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0cedf0 con 0x7f5e5c0b95a0 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e5bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c15df00 0x7f5e5c0b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e5bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c15df00 0x7f5e5c0b7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57270/0 (socket says 172.21.15.27:57270) 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e5bfff640 1 -- 172.21.15.27:0/3254490238 learned_addr learned my addr 172.21.15.27:0/3254490238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e60cb2640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b95a0 0x7f5e5c14e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e614b3640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e5c0b83d0 0x7f5e5c1603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e5bfff640 1 -- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b95a0 msgr2=0x7f5e5c14e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e5bfff640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b95a0 0x7f5e5c14e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e5bfff640 1 -- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e5c0b83d0 msgr2=0x7f5e5c1603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e5bfff640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e5c0b83d0 0x7f5e5c1603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e5bfff640 1 -- 172.21.15.27:0/3254490238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c1610a0 con 0x7f5e5c15df00 2024-03-06T07:28:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.029+0000 7f5e5bfff640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c15df00 0x7f5e5c0b7e90 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800c390 tx=0x7f5e4800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.030+0000 7f5e597fa640 1 -- 172.21.15.27:0/3254490238 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48017070 con 0x7f5e5c15df00 2024-03-06T07:28:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.030+0000 7f5e597fa640 1 -- 172.21.15.27:0/3254490238 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e48005ce0 con 0x7f5e5c15df00 2024-03-06T07:28:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.030+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c14eb20 con 0x7f5e5c15df00 2024-03-06T07:28:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.030+0000 7f5e597fa640 1 -- 172.21.15.27:0/3254490238 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e480072f0 con 0x7f5e5c15df00 2024-03-06T07:28:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.030+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c14f0e0 con 0x7f5e5c15df00 2024-03-06T07:28:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.031+0000 7f5e597fa640 1 -- 172.21.15.27:0/3254490238 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e4801a070 con 0x7f5e5c15df00 2024-03-06T07:28:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.031+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c1614c0 con 0x7f5e5c15df00 2024-03-06T07:28:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.032+0000 7f5e597fa640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e4006b4d0 0x7f5e4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.032+0000 7f5e597fa640 1 -- 172.21.15.27:0/3254490238 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e48089410 con 0x7f5e5c15df00 2024-03-06T07:28:54.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.032+0000 7f5e60cb2640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e4006b4d0 0x7f5e4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.032+0000 7f5e60cb2640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e4006b4d0 0x7f5e4006d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5e50036000 tx=0x7f5e50008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.036+0000 7f5e597fa640 1 -- 172.21.15.27:0/3254490238 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e5c1614c0 con 0x7f5e5c15df00 2024-03-06T07:28:54.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.254+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e5c067ff0 con 0x7f5e5c15df00 2024-03-06T07:28:54.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.255+0000 7f5e597fa640 1 -- 172.21.15.27:0/3254490238 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f5e48057cd0 con 0x7f5e5c15df00 2024-03-06T07:28:54.257 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:54.257 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/103754153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":103754153},{"type":"v1","addr":"172.21.15.69:6836","nonce":103754153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4078702997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":4078702997},{"type":"v1","addr":"172.21.15.69:6837","nonce":4078702997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7956,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3926923781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3926923781},{"type":"v1","addr":"172.21.15.27:6834","nonce":3926923781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:28:43.956535+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7959":{"gid":7959,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1655684069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1655684069},{"type":"v1","addr":"172.21.15.27:6835","nonce":1655684069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7959,"qdb_cluster":[7959]},"id":8}]} 2024-03-06T07:28:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 75 2024-03-06T07:28:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e4006b4d0 msgr2=0x7f5e4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e4006b4d0 0x7f5e4006d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5e50036000 tx=0x7f5e50008030 comp rx=0 tx=0).stop 2024-03-06T07:28:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c15df00 msgr2=0x7f5e5c0b7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c15df00 0x7f5e5c0b7e90 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800c390 tx=0x7f5e4800c790 comp rx=0 tx=0).stop 2024-03-06T07:28:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 shutdown_connections 2024-03-06T07:28:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5e4006b4d0 0x7f5e4006d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e5c0b83d0 0x7f5e5c1603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c15df00 0x7f5e5c0b7e90 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 --2- 172.21.15.27:0/3254490238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5e5c0b95a0 0x7f5e5c14e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.256+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 >> 172.21.15.27:0/3254490238 conn(0x7f5e5c071960 msgr2=0x7f5e5c0b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.257+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 shutdown_connections 2024-03-06T07:28:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.257+0000 7f5e62f16640 1 -- 172.21.15.27:0/3254490238 wait complete. 2024-03-06T07:28:54.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:28:54.409 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:28:54.409 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:54.409 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:54.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.451+0000 7f28cfd6ff40 1 Processor -- start 2024-03-06T07:28:54.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.451+0000 7f28cfd6ff40 1 -- start start 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cfd6ff40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b80030f0 0x561a0a7fe520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cfd6ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561a0a7e5dc0 0x561a0a7fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cfd6ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561a0a75d590 0x561a0a75d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cfd6ff40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a0a6d9b70 con 0x7f28b80030f0 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cfd6ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a0a6ddad0 con 0x561a0a75d590 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cfd6ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a0a8098b0 con 0x561a0a7e5dc0 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cd870640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b80030f0 0x561a0a7fe520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cd870640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b80030f0 0x561a0a7fe520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57302/0 (socket says 172.21.15.27:57302) 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cd870640 1 -- 172.21.15.27:0/1872707268 learned_addr learned my addr 172.21.15.27:0/1872707268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cd06f640 1 --2- 172.21.15.27:0/1872707268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561a0a7e5dc0 0x561a0a7fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28ce071640 1 --2- 172.21.15.27:0/1872707268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561a0a75d590 0x561a0a75d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cd870640 1 -- 172.21.15.27:0/1872707268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561a0a7e5dc0 msgr2=0x561a0a7fdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cd870640 1 --2- 172.21.15.27:0/1872707268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561a0a7e5dc0 0x561a0a7fdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cd870640 1 -- 172.21.15.27:0/1872707268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561a0a75d590 msgr2=0x561a0a75d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cd870640 1 --2- 172.21.15.27:0/1872707268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561a0a75d590 0x561a0a75d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.452+0000 7f28cd870640 1 -- 172.21.15.27:0/1872707268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a0a7342b0 con 0x7f28b80030f0 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.453+0000 7f28cd870640 1 --2- 172.21.15.27:0/1872707268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b80030f0 0x561a0a7fe520 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009740 tx=0x7f28c0005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b2df2581e55d5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.453+0000 7f28bffff640 1 -- 172.21.15.27:0/1872707268 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0008890 con 0x7f28b80030f0 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.453+0000 7f28bffff640 1 -- 172.21.15.27:0/1872707268 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c001a070 con 0x7f28b80030f0 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.453+0000 7f28bffff640 1 -- 172.21.15.27:0/1872707268 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0008c70 con 0x7f28b80030f0 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.453+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/1872707268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b80030f0 msgr2=0x561a0a7fe520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 --2- 172.21.15.27:0/1872707268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b80030f0 0x561a0a7fe520 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009740 tx=0x7f28c0005a00 comp rx=0 tx=0).stop 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/1872707268 shutdown_connections 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 --2- 172.21.15.27:0/1872707268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561a0a75d590 0x561a0a75d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 --2- 172.21.15.27:0/1872707268 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561a0a7e5dc0 0x561a0a7fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 --2- 172.21.15.27:0/1872707268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b80030f0 0x561a0a7fe520 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/1872707268 >> 172.21.15.27:0/1872707268 conn(0x561a0a7325e0 msgr2=0x561a0a75cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/1872707268 shutdown_connections 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/1872707268 wait complete. 2024-03-06T07:28:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 Processor -- start 2024-03-06T07:28:54.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.454+0000 7f28cfd6ff40 1 -- start start 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cfd6ff40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561a0a75d590 0x561a0a800f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cfd6ff40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a0a7e5dc0 0x561a0a8a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cfd6ff40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28b80030f0 0x561a0a8a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cfd6ff40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a0a734060 con 0x561a0a7e5dc0 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cfd6ff40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a0a77b610 con 0x7f28b80030f0 2024-03-06T07:28:54.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cfd6ff40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a0a77b8b0 con 0x561a0a75d590 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cd06f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a0a7e5dc0 0x561a0a8a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cd06f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a0a7e5dc0 0x561a0a8a0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57316/0 (socket says 172.21.15.27:57316) 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cd06f640 1 -- 172.21.15.27:0/2320331406 learned_addr learned my addr 172.21.15.27:0/2320331406 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cd870640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561a0a75d590 0x561a0a800f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28ce071640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28b80030f0 0x561a0a8a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cd06f640 1 -- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561a0a75d590 msgr2=0x561a0a800f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cd06f640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x561a0a75d590 0x561a0a800f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cd06f640 1 -- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28b80030f0 msgr2=0x561a0a8a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cd06f640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28b80030f0 0x561a0a8a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.455+0000 7f28cd06f640 1 -- 172.21.15.27:0/2320331406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a0a761ee0 con 0x561a0a7e5dc0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.456+0000 7f28cd06f640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a0a7e5dc0 0x561a0a8a0f00 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f28c4000c00 tx=0x7f28b8004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.456+0000 7f28beffd640 1 -- 172.21.15.27:0/2320331406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4061070 con 0x561a0a7e5dc0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.456+0000 7f28beffd640 1 -- 172.21.15.27:0/2320331406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c4052ad0 con 0x561a0a7e5dc0 2024-03-06T07:28:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.456+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561a0a7621c0 con 0x561a0a7e5dc0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.456+0000 7f28beffd640 1 -- 172.21.15.27:0/2320331406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4057040 con 0x561a0a7e5dc0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.456+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a0a806dd0 con 0x561a0a7e5dc0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.457+0000 7f28beffd640 1 -- 172.21.15.27:0/2320331406 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28c4064070 con 0x561a0a7e5dc0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.458+0000 7f28beffd640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28b0069eb0 0x7f28b006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.458+0000 7f28beffd640 1 -- 172.21.15.27:0/2320331406 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28c40ca350 con 0x561a0a7e5dc0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.458+0000 7f28cd870640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28b0069eb0 0x7f28b006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.458+0000 7f28cfd6ff40 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x561a0a8072f0 0x561a0a807760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.458+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x561a0a807ca0 con 0x561a0a8072f0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.458+0000 7f28ce071640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x561a0a8072f0 0x561a0a807760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cd870640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28b0069eb0 0x7f28b006c370 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f28c0017040 tx=0x7f28c0008000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28ce071640 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x561a0a8072f0 0x561a0a807760 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28ce071640 1 -- 172.21.15.27:0/2320331406 <== osd.0 v2:172.21.15.27:6800/3926946944 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f28c80060c0 con 0x561a0a8072f0 2024-03-06T07:28:54.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x561a0a8072f0 msgr2=0x561a0a807760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x561a0a8072f0 0x561a0a807760 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28b0069eb0 msgr2=0x7f28b006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28b0069eb0 0x7f28b006c370 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f28c0017040 tx=0x7f28c0008000 comp rx=0 tx=0).stop 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a0a7e5dc0 msgr2=0x561a0a8a0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 --2- 172.21.15.27:0/2320331406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561a0a7e5dc0 0x561a0a8a0f00 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f28c4000c00 tx=0x7f28b8004120 comp rx=0 tx=0).stop 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28ce071640 1 -- 172.21.15.27:0/2320331406 reap_dead start 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 shutdown_connections 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 >> 172.21.15.27:0/2320331406 conn(0x561a0a7325e0 msgr2=0x561a0a75cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 shutdown_connections 2024-03-06T07:28:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.459+0000 7f28cfd6ff40 1 -- 172.21.15.27:0/2320331406 wait complete. 2024-03-06T07:28:54.466 INFO:teuthology.orchestra.run.smithi027.stdout:Wfile_a 2024-03-06T07:28:54.466 INFO:teuthology.orchestra.run.smithi027.stdout:dir_1 2024-03-06T07:28:54.466 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:54.601 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:28:54.601 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511627777, 2024-03-06T07:28:54.601 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2024-03-06T07:28:54.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:54.601 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1099511627776, 2024-03-06T07:28:54.601 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "file_a", 2024-03-06T07:28:54.601 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 10 2024-03-06T07:28:54.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:28:54.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:28:54.602 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1, 2024-03-06T07:28:54.602 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "dir_1", 2024-03-06T07:28:54.602 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 17 2024-03-06T07:28:54.602 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:28:54.602 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-03-06T07:28:54.602 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 17, 2024-03-06T07:28:54.602 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [] 2024-03-06T07:28:54.602 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:28:54.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:54.614 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:28:54.614 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:54.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:54.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.773+0000 7fe93bb25640 1 Processor -- start 2024-03-06T07:28:54.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.774+0000 7fe93bb25640 1 -- start start 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.775+0000 7fe93bb25640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9340b95a0 0x7fe9340b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.775+0000 7fe93bb25640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9340b9a10 0x7fe9340b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.775+0000 7fe93bb25640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9340cea30 con 0x7fe93415df00 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.775+0000 7fe93bb25640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9340cf3c0 con 0x7fe9340b9a10 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.775+0000 7fe93bb25640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9340cf0e0 con 0x7fe9340b95a0 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.775+0000 7fe93a0c2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9340b9a10 0x7fe9340b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.775+0000 7fe9398c1640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe93415df00 0x7fe9340d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59824/0 (socket says 172.21.15.27:59824) 2024-03-06T07:28:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.775+0000 7fe9398c1640 1 -- 172.21.15.27:0/608936283 learned_addr learned my addr 172.21.15.27:0/608936283 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.775+0000 7fe9390c0640 1 --2- 172.21.15.27:0/608936283 >> v2:172.21.15.69:3300/0 conn(0x7fe9340b95a0 0x7fe9340b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.776+0000 7fe93a0c2640 1 -- 172.21.15.27:0/608936283 >> v2:172.21.15.69:3300/0 conn(0x7fe9340b95a0 msgr2=0x7fe9340b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.776+0000 7fe93a0c2640 1 --2- 172.21.15.27:0/608936283 >> v2:172.21.15.69:3300/0 conn(0x7fe9340b95a0 0x7fe9340b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.776+0000 7fe93a0c2640 1 -- 172.21.15.27:0/608936283 >> v1:172.21.15.27:6789/0 conn(0x7fe93415df00 legacy=0x7fe9340d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.776+0000 7fe93a0c2640 1 -- 172.21.15.27:0/608936283 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9340ce470 con 0x7fe9340b9a10 2024-03-06T07:28:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.776+0000 7fe9388bf640 1 -- 172.21.15.27:0/608936283 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708253182 0 0) 0x7fe9340cea30 con 0x7fe93415df00 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.776+0000 7fe93a0c2640 1 --2- 172.21.15.27:0/608936283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9340b9a10 0x7fe9340b5c40 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fe9340d1760 tx=0x7fe92c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e8424d6f118d7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe9388bf640 1 -- 172.21.15.27:0/608936283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c039070 con 0x7fe9340b9a10 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe9388bf640 1 -- 172.21.15.27:0/608936283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c02cc80 con 0x7fe9340b9a10 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe9388bf640 1 -- 172.21.15.27:0/608936283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c034660 con 0x7fe9340b9a10 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe93bb25640 1 -- 172.21.15.27:0/608936283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9340b9a10 msgr2=0x7fe9340b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe93bb25640 1 --2- 172.21.15.27:0/608936283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9340b9a10 0x7fe9340b5c40 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fe9340d1760 tx=0x7fe92c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe93bb25640 1 -- 172.21.15.27:0/608936283 shutdown_connections 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe93bb25640 1 --2- 172.21.15.27:0/608936283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9340b9a10 0x7fe9340b5c40 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe93bb25640 1 --2- 172.21.15.27:0/608936283 >> v2:172.21.15.69:3300/0 conn(0x7fe9340b95a0 0x7fe9340b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe93bb25640 1 -- 172.21.15.27:0/608936283 >> 172.21.15.27:0/608936283 conn(0x7fe934071960 msgr2=0x7fe934071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe93bb25640 1 -- 172.21.15.27:0/608936283 shutdown_connections 2024-03-06T07:28:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.777+0000 7fe93bb25640 1 -- 172.21.15.27:0/608936283 wait complete. 2024-03-06T07:28:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.778+0000 7fe93bb25640 1 Processor -- start 2024-03-06T07:28:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.778+0000 7fe93bb25640 1 -- start start 2024-03-06T07:28:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe93bb25640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9340b95a0 0x7fe93415fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe93bb25640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9340b9a10 0x7fe93415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe93bb25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe93415df00 0x7fe93414e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe93bb25640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9340cedf0 con 0x7fe93415df00 2024-03-06T07:28:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe93bb25640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9340cec70 con 0x7fe9340b95a0 2024-03-06T07:28:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe93bb25640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9340d3220 con 0x7fe9340b9a10 2024-03-06T07:28:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe93a0c2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe93415df00 0x7fe93414e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe93a0c2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe93415df00 0x7fe93414e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57344/0 (socket says 172.21.15.27:57344) 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe93a0c2640 1 -- 172.21.15.27:0/1624373200 learned_addr learned my addr 172.21.15.27:0/1624373200 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe9398c1640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9340b95a0 0x7fe93415fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.779+0000 7fe9390c0640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9340b9a10 0x7fe93415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe93a0c2640 1 -- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9340b9a10 msgr2=0x7fe93415e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe93a0c2640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9340b9a10 0x7fe93415e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe93a0c2640 1 -- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9340b95a0 msgr2=0x7fe93415fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe93a0c2640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9340b95a0 0x7fe93415fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe93a0c2640 1 -- 172.21.15.27:0/1624373200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93415eca0 con 0x7fe93415df00 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe9390c0640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9340b9a10 0x7fe93415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe93a0c2640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe93415df00 0x7fe93414e610 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009e80 tx=0x7fe92c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe92a7fc640 1 -- 172.21.15.27:0/1624373200 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c039050 con 0x7fe93415df00 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe92a7fc640 1 -- 172.21.15.27:0/1624373200 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c03a040 con 0x7fe93415df00 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93415ef80 con 0x7fe93415df00 2024-03-06T07:28:54.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe92a7fc640 1 -- 172.21.15.27:0/1624373200 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c03e610 con 0x7fe93415df00 2024-03-06T07:28:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.780+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9340cadf0 con 0x7fe93415df00 2024-03-06T07:28:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.781+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe934067ff0 con 0x7fe93415df00 2024-03-06T07:28:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.782+0000 7fe92a7fc640 1 -- 172.21.15.27:0/1624373200 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe92c047020 con 0x7fe93415df00 2024-03-06T07:28:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.782+0000 7fe92a7fc640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe91806b500 0x7fe91806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.782+0000 7fe92a7fc640 1 -- 172.21.15.27:0/1624373200 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe92c0ac4e0 con 0x7fe93415df00 2024-03-06T07:28:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.782+0000 7fe9398c1640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe91806b500 0x7fe91806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.783+0000 7fe9398c1640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe91806b500 0x7fe91806d9c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe930007fd0 tx=0x7fe930007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.786+0000 7fe92a7fc640 1 -- 172.21.15.27:0/1624373200 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe92c04e020 con 0x7fe93415df00 2024-03-06T07:28:54.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.970+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9340674f0 con 0x7fe93415df00 2024-03-06T07:28:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.971+0000 7fe92a7fc640 1 -- 172.21.15.27:0/1624373200 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16555 (secure 0 0 0) 0x7fe92c07b360 con 0x7fe93415df00 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":114,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:28:42.579409+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.260334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.929245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe91806b500 msgr2=0x7fe91806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe91806b500 0x7fe91806d9c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe930007fd0 tx=0x7fe930007de0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe93415df00 msgr2=0x7fe93414e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe93415df00 0x7fe93414e610 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009e80 tx=0x7fe92c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 shutdown_connections 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe91806b500 0x7fe91806d9c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe93415df00 0x7fe93414e610 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9340b9a10 0x7fe93415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 --2- 172.21.15.27:0/1624373200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9340b95a0 0x7fe93415fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 >> 172.21.15.27:0/1624373200 conn(0x7fe934071960 msgr2=0x7fe934071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.973+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 shutdown_connections 2024-03-06T07:28:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:54.974+0000 7fe93bb25640 1 -- 172.21.15.27:0/1624373200 wait complete. 2024-03-06T07:28:54.991 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:39.260334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:39.929245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:54.992 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.099+0000 7f5c88565640 1 Processor -- start 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.099+0000 7f5c88565640 1 -- start start 2024-03-06T07:28:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.100+0000 7f5c88565640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5c8015dd90 0x7f5c800ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.100+0000 7f5c88565640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d29d0 0x7f5c800d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.100+0000 7f5c88565640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c800d3740 con 0x7f5c800d4720 2024-03-06T07:28:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.100+0000 7f5c88565640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c800d3400 con 0x7f5c8015dd90 2024-03-06T07:28:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.100+0000 7f5c88565640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c800d3120 con 0x7f5c800d29d0 2024-03-06T07:28:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.100+0000 7f5c85b00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d29d0 0x7f5c800d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.100+0000 7f5c86301640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5c8015dd90 0x7f5c800ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.100+0000 7f5c86b02640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5c800d4720 0x7f5c800b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:37050/0 (socket says 172.21.15.27:37050) 2024-03-06T07:28:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.100+0000 7f5c86b02640 1 -- 172.21.15.27:0/3211977006 learned_addr learned my addr 172.21.15.27:0/3211977006 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.101+0000 7f5c86301640 1 -- 172.21.15.27:0/3211977006 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d29d0 msgr2=0x7f5c800d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.101+0000 7f5c86301640 1 --2- 172.21.15.27:0/3211977006 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d29d0 0x7f5c800d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.101+0000 7f5c86301640 1 -- 172.21.15.27:0/3211977006 >> v1:172.21.15.69:6789/0 conn(0x7f5c800d4720 legacy=0x7f5c800b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.101+0000 7f5c852ff640 1 -- 172.21.15.27:0/3211977006 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582470298 0 0) 0x7f5c800d3740 con 0x7f5c800d4720 2024-03-06T07:28:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.101+0000 7f5c86301640 1 -- 172.21.15.27:0/3211977006 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c800d39e0 con 0x7f5c8015dd90 2024-03-06T07:28:55.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.102+0000 7f5c86301640 1 --2- 172.21.15.27:0/3211977006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 0x7f5c800ceea0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5c700097a0 tx=0x7f5c7002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0043c0c8bf682da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.102+0000 7f5c852ff640 1 -- 172.21.15.27:0/3211977006 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70034070 con 0x7f5c8015dd90 2024-03-06T07:28:55.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.102+0000 7f5c852ff640 1 -- 172.21.15.27:0/3211977006 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c70030610 con 0x7f5c8015dd90 2024-03-06T07:28:55.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.102+0000 7f5c852ff640 1 -- 172.21.15.27:0/3211977006 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c700309e0 con 0x7f5c8015dd90 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.103+0000 7f5c88565640 1 -- 172.21.15.27:0/3211977006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 msgr2=0x7f5c800ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.103+0000 7f5c88565640 1 --2- 172.21.15.27:0/3211977006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 0x7f5c800ceea0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5c700097a0 tx=0x7f5c7002b960 comp rx=0 tx=0).stop 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.103+0000 7f5c88565640 1 -- 172.21.15.27:0/3211977006 shutdown_connections 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.103+0000 7f5c88565640 1 --2- 172.21.15.27:0/3211977006 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d29d0 0x7f5c800d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.103+0000 7f5c88565640 1 --2- 172.21.15.27:0/3211977006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 0x7f5c800ceea0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.103+0000 7f5c88565640 1 -- 172.21.15.27:0/3211977006 >> 172.21.15.27:0/3211977006 conn(0x7f5c80071960 msgr2=0x7f5c80071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.103+0000 7f5c88565640 1 -- 172.21.15.27:0/3211977006 shutdown_connections 2024-03-06T07:28:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.103+0000 7f5c88565640 1 -- 172.21.15.27:0/3211977006 wait complete. 2024-03-06T07:28:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.105+0000 7f5c88565640 1 Processor -- start 2024-03-06T07:28:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.105+0000 7f5c88565640 1 -- start start 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c88565640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c800d29d0 0x7f5c801f7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c88565640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d4720 0x7f5c800b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c88565640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 0x7f5c800b8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c88565640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c800d2df0 con 0x7f5c800d29d0 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c88565640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8015e170 con 0x7f5c8015dd90 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c88565640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c800b7420 con 0x7f5c800d4720 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c86301640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c800d29d0 0x7f5c801f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c86301640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c800d29d0 0x7f5c801f7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57358/0 (socket says 172.21.15.27:57358) 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c86301640 1 -- 172.21.15.27:0/2078841997 learned_addr learned my addr 172.21.15.27:0/2078841997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c85b00640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d4720 0x7f5c800b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.106+0000 7f5c86b02640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 0x7f5c800b8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c86301640 1 -- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d4720 msgr2=0x7f5c800b89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c86301640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d4720 0x7f5c800b89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c86301640 1 -- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 msgr2=0x7f5c800b8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c86301640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 0x7f5c800b8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c86301640 1 -- 172.21.15.27:0/2078841997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c80160e50 con 0x7f5c800d29d0 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c86b02640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 0x7f5c800b8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c86301640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c800d29d0 0x7f5c801f7840 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f5c70000c00 tx=0x7f5c700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c76ffd640 1 -- 172.21.15.27:0/2078841997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70047070 con 0x7f5c800d29d0 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c76ffd640 1 -- 172.21.15.27:0/2078841997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c70030410 con 0x7f5c800d29d0 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c80161130 con 0x7f5c800d29d0 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c76ffd640 1 -- 172.21.15.27:0/2078841997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70034040 con 0x7f5c800d29d0 2024-03-06T07:28:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.107+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c801f7320 con 0x7f5c800d29d0 2024-03-06T07:28:55.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.108+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c80072a30 con 0x7f5c800d29d0 2024-03-06T07:28:55.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.108+0000 7f5c76ffd640 1 -- 172.21.15.27:0/2078841997 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c7004a070 con 0x7f5c800d29d0 2024-03-06T07:28:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.109+0000 7f5c76ffd640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c6406b4d0 0x7f5c6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.109+0000 7f5c76ffd640 1 -- 172.21.15.27:0/2078841997 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c700ab9f0 con 0x7f5c800d29d0 2024-03-06T07:28:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.109+0000 7f5c85b00640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c6406b4d0 0x7f5c6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.110+0000 7f5c85b00640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c6406b4d0 0x7f5c6406d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5c800b6460 tx=0x7f5c7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.113+0000 7f5c76ffd640 1 -- 172.21.15.27:0/2078841997 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c7007a900 con 0x7f5c800d29d0 2024-03-06T07:28:55.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.375+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c8004f410 con 0x7f5c800d29d0 2024-03-06T07:28:55.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.376+0000 7f5c76ffd640 1 -- 172.21.15.27:0/2078841997 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c700305e0 con 0x7f5c800d29d0 2024-03-06T07:28:55.380 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:28:55.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.377+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c6406b4d0 msgr2=0x7f5c6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.377+0000 7f5c88565640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c6406b4d0 0x7f5c6406d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5c800b6460 tx=0x7f5c7c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:55.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.377+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c800d29d0 msgr2=0x7f5c801f7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.377+0000 7f5c88565640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c800d29d0 0x7f5c801f7840 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f5c70000c00 tx=0x7f5c700046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.378+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 shutdown_connections 2024-03-06T07:28:55.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.378+0000 7f5c88565640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c6406b4d0 0x7f5c6406d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.378+0000 7f5c88565640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c8015dd90 0x7f5c800b8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.378+0000 7f5c88565640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5c800d4720 0x7f5c800b89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.378+0000 7f5c88565640 1 --2- 172.21.15.27:0/2078841997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c800d29d0 0x7f5c801f7840 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.378+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 >> 172.21.15.27:0/2078841997 conn(0x7f5c80071960 msgr2=0x7f5c800d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.378+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 shutdown_connections 2024-03-06T07:28:55.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.378+0000 7f5c88565640 1 -- 172.21.15.27:0/2078841997 wait complete. 2024-03-06T07:28:55.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.500+0000 7f96e2b65640 1 Processor -- start 2024-03-06T07:28:55.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.500+0000 7f96e2b65640 1 -- start start 2024-03-06T07:28:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96e2b65640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96dc0ceea0 0x7f96dc0d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96e2b65640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96dc0d3740 con 0x7f96dc15dd90 2024-03-06T07:28:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96e2b65640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96dc0d3120 con 0x7f96dc0b6d10 2024-03-06T07:28:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96e2b65640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96dc0d3400 con 0x7f96dc0ceea0 2024-03-06T07:28:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96e0901640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f96dc15dd90 0x7f96dc0d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59838/0 (socket says 172.21.15.27:59838) 2024-03-06T07:28:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96e0901640 1 -- 172.21.15.27:0/4054870427 learned_addr learned my addr 172.21.15.27:0/4054870427 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96dbfff640 1 --2- 172.21.15.27:0/4054870427 >> v2:172.21.15.69:3300/0 conn(0x7f96dc0ceea0 0x7f96dc0d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96db7fe640 1 -- 172.21.15.27:0/4054870427 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783964869 0 0) 0x7f96dc0d3740 con 0x7f96dc15dd90 2024-03-06T07:28:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96db7fe640 1 -- 172.21.15.27:0/4054870427 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c4003520 con 0x7f96dc15dd90 2024-03-06T07:28:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96db7fe640 1 -- 172.21.15.27:0/4054870427 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083184288 0 0) 0x7f96dc0d3120 con 0x7f96dc0b6d10 2024-03-06T07:28:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.501+0000 7f96db7fe640 1 -- 172.21.15.27:0/4054870427 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96dc0d3740 con 0x7f96dc0b6d10 2024-03-06T07:28:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.502+0000 7f96dbfff640 1 -- 172.21.15.27:0/4054870427 >> v1:172.21.15.69:6789/0 conn(0x7f96dc0b6d10 legacy=0x7f96dc0d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.502+0000 7f96dbfff640 1 -- 172.21.15.27:0/4054870427 >> v1:172.21.15.27:6789/0 conn(0x7f96dc15dd90 legacy=0x7f96dc0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.502+0000 7f96dbfff640 1 -- 172.21.15.27:0/4054870427 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc0d39e0 con 0x7f96dc0ceea0 2024-03-06T07:28:55.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.502+0000 7f96dbfff640 1 --2- 172.21.15.27:0/4054870427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0ceea0 0x7f96dc0d29d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f96d00097a0 tx=0x7f96d002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f888ac37c7a8902 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96db7fe640 1 -- 172.21.15.27:0/4054870427 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d002f810 con 0x7f96dc0ceea0 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96db7fe640 1 -- 172.21.15.27:0/4054870427 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d002f9b0 con 0x7f96dc0ceea0 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96db7fe640 1 -- 172.21.15.27:0/4054870427 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d002fd80 con 0x7f96dc0ceea0 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96e2b65640 1 -- 172.21.15.27:0/4054870427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0ceea0 msgr2=0x7f96dc0d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96e2b65640 1 --2- 172.21.15.27:0/4054870427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0ceea0 0x7f96dc0d29d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f96d00097a0 tx=0x7f96d002b9f0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96e2b65640 1 -- 172.21.15.27:0/4054870427 shutdown_connections 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96e2b65640 1 --2- 172.21.15.27:0/4054870427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0ceea0 0x7f96dc0d29d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96e2b65640 1 -- 172.21.15.27:0/4054870427 >> 172.21.15.27:0/4054870427 conn(0x7f96dc071960 msgr2=0x7f96dc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96e2b65640 1 -- 172.21.15.27:0/4054870427 shutdown_connections 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.503+0000 7f96e2b65640 1 -- 172.21.15.27:0/4054870427 wait complete. 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.504+0000 7f96e2b65640 1 Processor -- start 2024-03-06T07:28:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.504+0000 7f96e2b65640 1 -- start start 2024-03-06T07:28:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96e2b65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0b6d10 0x7f96dc0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96e2b65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96dc0ceea0 0x7f96dc149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96e2b65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc15dd90 0x7f96dc160c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96e2b65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc0d48f0 con 0x7f96dc0ceea0 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96e2b65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc15e170 con 0x7f96dc0b6d10 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96e2b65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc0d4a70 con 0x7f96dc15dd90 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96dbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96dc0ceea0 0x7f96dc149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96dbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96dc0ceea0 0x7f96dc149bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57364/0 (socket says 172.21.15.27:57364) 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96dbfff640 1 -- 172.21.15.27:0/4120623906 learned_addr learned my addr 172.21.15.27:0/4120623906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96e1102640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc15dd90 0x7f96dc160c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.505+0000 7f96e0901640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0b6d10 0x7f96dc0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96dbfff640 1 -- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc15dd90 msgr2=0x7f96dc160c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96dbfff640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc15dd90 0x7f96dc160c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96dbfff640 1 -- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0b6d10 msgr2=0x7f96dc0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96dbfff640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0b6d10 0x7f96dc0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96dbfff640 1 -- 172.21.15.27:0/4120623906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc0cce00 con 0x7f96dc0ceea0 2024-03-06T07:28:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96e1102640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc15dd90 0x7f96dc160c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96e0901640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0b6d10 0x7f96dc0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96dbfff640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96dc0ceea0 0x7f96dc149bd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009890 tx=0x7f96d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96d97fa640 1 -- 172.21.15.27:0/4120623906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0046070 con 0x7f96dc0ceea0 2024-03-06T07:28:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96d97fa640 1 -- 172.21.15.27:0/4120623906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d00378f0 con 0x7f96dc0ceea0 2024-03-06T07:28:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96dc0cd0e0 con 0x7f96dc0ceea0 2024-03-06T07:28:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96d97fa640 1 -- 172.21.15.27:0/4120623906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0037ce0 con 0x7f96dc0ceea0 2024-03-06T07:28:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.506+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96dc0cd6a0 con 0x7f96dc0ceea0 2024-03-06T07:28:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.507+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96dc072a30 con 0x7f96dc0ceea0 2024-03-06T07:28:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.508+0000 7f96d97fa640 1 -- 172.21.15.27:0/4120623906 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96d00403e0 con 0x7f96dc0ceea0 2024-03-06T07:28:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.508+0000 7f96d97fa640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b4d0 0x7f96c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.508+0000 7f96d97fa640 1 -- 172.21.15.27:0/4120623906 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96d00abc00 con 0x7f96dc0ceea0 2024-03-06T07:28:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.508+0000 7f96e0901640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b4d0 0x7f96c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.509+0000 7f96e0901640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b4d0 0x7f96c806d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f96cc000f60 tx=0x7f96cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.512+0000 7f96d97fa640 1 -- 172.21.15.27:0/4120623906 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96d00b2080 con 0x7f96dc0ceea0 2024-03-06T07:28:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.696+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96dc04f410 con 0x7f96dc0ceea0 2024-03-06T07:28:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.697+0000 7f96d97fa640 1 -- 172.21.15.27:0/4120623906 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f96d0033030 con 0x7f96dc0ceea0 2024-03-06T07:28:55.699 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:28:55.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b4d0 msgr2=0x7f96c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b4d0 0x7f96c806d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f96cc000f60 tx=0x7f96cc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:55.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96dc0ceea0 msgr2=0x7f96dc149bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96dc0ceea0 0x7f96dc149bd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009890 tx=0x7f96d0004730 comp rx=0 tx=0).stop 2024-03-06T07:28:55.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 shutdown_connections 2024-03-06T07:28:55.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96c806b4d0 0x7f96c806d990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96dc15dd90 0x7f96dc160c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96dc0ceea0 0x7f96dc149bd0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 --2- 172.21.15.27:0/4120623906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96dc0b6d10 0x7f96dc0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 >> 172.21.15.27:0/4120623906 conn(0x7f96dc071960 msgr2=0x7f96dc0b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 shutdown_connections 2024-03-06T07:28:55.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.699+0000 7f96e2b65640 1 -- 172.21.15.27:0/4120623906 wait complete. 2024-03-06T07:28:55.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:55.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.820+0000 7f909700e640 1 Processor -- start 2024-03-06T07:28:55.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.820+0000 7f909700e640 1 -- start start 2024-03-06T07:28:55.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f909700e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9090071540 0x7f9090071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f909700e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f90900b2d30 0x7f90900ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f909700e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9090071e60 con 0x7f90900cecb0 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f909700e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90900d3230 con 0x7f9090071540 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f909700e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90900d4960 con 0x7f90900b2d30 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f9094daa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9090071540 0x7f9090071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f9094daa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9090071540 0x7f9090071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57368/0 (socket says 172.21.15.27:57368) 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f9094daa640 1 -- 172.21.15.27:0/4245310530 learned_addr learned my addr 172.21.15.27:0/4245310530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f908ffff640 1 --2- 172.21.15.27:0/4245310530 >> v2:172.21.15.69:3300/0 conn(0x7f90900b2d30 0x7f90900ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f9094daa640 1 -- 172.21.15.27:0/4245310530 >> v2:172.21.15.69:3300/0 conn(0x7f90900b2d30 msgr2=0x7f90900ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f9094daa640 1 --2- 172.21.15.27:0/4245310530 >> v2:172.21.15.69:3300/0 conn(0x7f90900b2d30 0x7f90900ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f9094daa640 1 -- 172.21.15.27:0/4245310530 >> v1:172.21.15.69:6789/0 conn(0x7f90900cecb0 legacy=0x7f90900b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.821+0000 7f9094daa640 1 -- 172.21.15.27:0/4245310530 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90900d36c0 con 0x7f9090071540 2024-03-06T07:28:55.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.822+0000 7f9094daa640 1 --2- 172.21.15.27:0/4245310530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 0x7f9090071920 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f90900c7690 tx=0x7f908402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bed271b2d9227d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.822+0000 7f908f7fe640 1 -- 172.21.15.27:0/4245310530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084034070 con 0x7f9090071540 2024-03-06T07:28:55.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.822+0000 7f908f7fe640 1 -- 172.21.15.27:0/4245310530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084030860 con 0x7f9090071540 2024-03-06T07:28:55.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.822+0000 7f908f7fe640 1 -- 172.21.15.27:0/4245310530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084030c30 con 0x7f9090071540 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.823+0000 7f909700e640 1 -- 172.21.15.27:0/4245310530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 msgr2=0x7f9090071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.823+0000 7f909700e640 1 --2- 172.21.15.27:0/4245310530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 0x7f9090071920 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f90900c7690 tx=0x7f908402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.823+0000 7f909700e640 1 -- 172.21.15.27:0/4245310530 shutdown_connections 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.823+0000 7f909700e640 1 --2- 172.21.15.27:0/4245310530 >> v2:172.21.15.69:3300/0 conn(0x7f90900b2d30 0x7f90900ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.823+0000 7f909700e640 1 --2- 172.21.15.27:0/4245310530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 0x7f9090071920 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.823+0000 7f909700e640 1 -- 172.21.15.27:0/4245310530 >> 172.21.15.27:0/4245310530 conn(0x7f90900d3d60 msgr2=0x7f90900d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.823+0000 7f909700e640 1 -- 172.21.15.27:0/4245310530 shutdown_connections 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.823+0000 7f909700e640 1 -- 172.21.15.27:0/4245310530 wait complete. 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.824+0000 7f909700e640 1 Processor -- start 2024-03-06T07:28:55.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.824+0000 7f909700e640 1 -- start start 2024-03-06T07:28:55.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f909700e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 0x7f90900c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f909700e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90900b2d30 0x7f9090156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f909700e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90900cecb0 0x7f9090158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f909700e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90900d2be0 con 0x7f9090071540 2024-03-06T07:28:55.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f909700e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90900d2900 con 0x7f90900cecb0 2024-03-06T07:28:55.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f909700e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90900d2ec0 con 0x7f90900b2d30 2024-03-06T07:28:55.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f9094daa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 0x7f90900c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f9094daa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 0x7f90900c8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57378/0 (socket says 172.21.15.27:57378) 2024-03-06T07:28:55.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f9094daa640 1 -- 172.21.15.27:0/1179784605 learned_addr learned my addr 172.21.15.27:0/1179784605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f90955ab640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90900cecb0 0x7f9090158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f908ffff640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90900b2d30 0x7f9090156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f9094daa640 1 -- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90900b2d30 msgr2=0x7f9090156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f9094daa640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90900b2d30 0x7f9090156fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f9094daa640 1 -- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90900cecb0 msgr2=0x7f9090158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f9094daa640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90900cecb0 0x7f9090158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f9094daa640 1 -- 172.21.15.27:0/1179784605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90900c9830 con 0x7f9090071540 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.825+0000 7f90955ab640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90900cecb0 0x7f9090158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.826+0000 7f9094daa640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 0x7f90900c8840 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f908402b9e0 tx=0x7f90840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.826+0000 7f908d7fa640 1 -- 172.21.15.27:0/1179784605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084034040 con 0x7f9090071540 2024-03-06T07:28:55.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.826+0000 7f908d7fa640 1 -- 172.21.15.27:0/1179784605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084039d90 con 0x7f9090071540 2024-03-06T07:28:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.826+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90900c8f90 con 0x7f9090071540 2024-03-06T07:28:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.826+0000 7f908d7fa640 1 -- 172.21.15.27:0/1179784605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084043610 con 0x7f9090071540 2024-03-06T07:28:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.826+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90900c9550 con 0x7f9090071540 2024-03-06T07:28:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.827+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9090072070 con 0x7f9090071540 2024-03-06T07:28:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.827+0000 7f908d7fa640 1 -- 172.21.15.27:0/1179784605 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f908404a060 con 0x7f9090071540 2024-03-06T07:28:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.828+0000 7f908d7fa640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907406b4d0 0x7f907406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.828+0000 7f908d7fa640 1 -- 172.21.15.27:0/1179784605 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90840ac2a0 con 0x7f9090071540 2024-03-06T07:28:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.828+0000 7f908ffff640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907406b4d0 0x7f907406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.829+0000 7f908ffff640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907406b4d0 0x7f907406d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f90901566a0 tx=0x7f907c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:55.837+0000 7f908d7fa640 1 -- 172.21.15.27:0/1179784605 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f908403e070 con 0x7f9090071540 2024-03-06T07:28:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.018+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f909004f410 con 0x7f9090071540 2024-03-06T07:28:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.019+0000 7f908d7fa640 1 -- 172.21.15.27:0/1179784605 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9084037360 con 0x7f9090071540 2024-03-06T07:28:56.021 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:28:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.020+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907406b4d0 msgr2=0x7f907406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.020+0000 7f909700e640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907406b4d0 0x7f907406d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f90901566a0 tx=0x7f907c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.020+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 msgr2=0x7f90900c8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.020+0000 7f909700e640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 0x7f90900c8840 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f908402b9e0 tx=0x7f90840046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.021+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 shutdown_connections 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.021+0000 7f909700e640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907406b4d0 0x7f907406d990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.021+0000 7f909700e640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90900cecb0 0x7f9090158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.021+0000 7f909700e640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90900b2d30 0x7f9090156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.021+0000 7f909700e640 1 --2- 172.21.15.27:0/1179784605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9090071540 0x7f90900c8840 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.021+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 >> 172.21.15.27:0/1179784605 conn(0x7f90900d3d60 msgr2=0x7f90900ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.021+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 shutdown_connections 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.021+0000 7f909700e640 1 -- 172.21.15.27:0/1179784605 wait complete. 2024-03-06T07:28:56.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.141+0000 7f2402639640 1 Processor -- start 2024-03-06T07:28:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.141+0000 7f2402639640 1 -- start start 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.142+0000 7f2402639640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23fc0b0e90 0x7f23fc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.142+0000 7f2402639640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc0cee90 con 0x7f23fc06b5c0 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.142+0000 7f2402639640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc0cfc50 con 0x7f23fc0ae9b0 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.142+0000 7f2402639640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23fc0cfa00 con 0x7f23fc0b0e90 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.142+0000 7f23fb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23fc0b0e90 0x7f23fc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.142+0000 7f23fbfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f23fc06b5c0 0x7f23fc0b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59846/0 (socket says 172.21.15.27:59846) 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.142+0000 7f23fb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23fc0b0e90 0x7f23fc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57390/0 (socket says 172.21.15.27:57390) 2024-03-06T07:28:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.142+0000 7f23fbfff640 1 -- 172.21.15.27:0/3501227386 learned_addr learned my addr 172.21.15.27:0/3501227386 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.143+0000 7f23fb7fe640 1 -- 172.21.15.27:0/3501227386 >> v1:172.21.15.69:6789/0 conn(0x7f23fc0ae9b0 legacy=0x7f23fc0d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.143+0000 7f23fb7fe640 1 -- 172.21.15.27:0/3501227386 >> v1:172.21.15.27:6789/0 conn(0x7f23fc06b5c0 legacy=0x7f23fc0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.143+0000 7f23fb7fe640 1 -- 172.21.15.27:0/3501227386 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0cff30 con 0x7f23fc0b0e90 2024-03-06T07:28:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.143+0000 7f23fb7fe640 1 --2- 172.21.15.27:0/3501227386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 0x7f23fc06b9a0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f23ec0097a0 tx=0x7f23ec02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8e1186aa0b9121 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f23faffd640 1 -- 172.21.15.27:0/3501227386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec037070 con 0x7f23fc0b0e90 2024-03-06T07:28:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f23faffd640 1 -- 172.21.15.27:0/3501227386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ec030710 con 0x7f23fc0b0e90 2024-03-06T07:28:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f23faffd640 1 -- 172.21.15.27:0/3501227386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec030ae0 con 0x7f23fc0b0e90 2024-03-06T07:28:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f2402639640 1 -- 172.21.15.27:0/3501227386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 msgr2=0x7f23fc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f2402639640 1 --2- 172.21.15.27:0/3501227386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 0x7f23fc06b9a0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f23ec0097a0 tx=0x7f23ec02b710 comp rx=0 tx=0).stop 2024-03-06T07:28:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f2402639640 1 -- 172.21.15.27:0/3501227386 shutdown_connections 2024-03-06T07:28:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f2402639640 1 --2- 172.21.15.27:0/3501227386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 0x7f23fc06b9a0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f2402639640 1 -- 172.21.15.27:0/3501227386 >> 172.21.15.27:0/3501227386 conn(0x7f23fc0b36d0 msgr2=0x7f23fc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f2402639640 1 -- 172.21.15.27:0/3501227386 shutdown_connections 2024-03-06T07:28:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.144+0000 7f2402639640 1 -- 172.21.15.27:0/3501227386 wait complete. 2024-03-06T07:28:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.146+0000 7f2402639640 1 Processor -- start 2024-03-06T07:28:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.147+0000 7f2402639640 1 -- start start 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2402639640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23fc06b5c0 0x7f23fc158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2402639640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23fc0ae9b0 0x7f23fc0b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2402639640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 0x7f23fc0c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2402639640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc06c910 con 0x7f23fc0b0e90 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2402639640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc06c790 con 0x7f23fc0ae9b0 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2402639640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc0d0ab0 con 0x7f23fc06b5c0 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2400bd6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 0x7f23fc0c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2400bd6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 0x7f23fc0c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57392/0 (socket says 172.21.15.27:57392) 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2400bd6640 1 -- 172.21.15.27:0/3838340795 learned_addr learned my addr 172.21.15.27:0/3838340795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f23fb7fe640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23fc0ae9b0 0x7f23fc0b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f23fbfff640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23fc06b5c0 0x7f23fc158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2400bd6640 1 -- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23fc06b5c0 msgr2=0x7f23fc158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2400bd6640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23fc06b5c0 0x7f23fc158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2400bd6640 1 -- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23fc0ae9b0 msgr2=0x7f23fc0b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2400bd6640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23fc0ae9b0 0x7f23fc0b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.148+0000 7f2400bd6640 1 -- 172.21.15.27:0/3838340795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0b5a00 con 0x7f23fc0b0e90 2024-03-06T07:28:56.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.149+0000 7f2400bd6640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 0x7f23fc0c9cd0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f23f400c7a0 tx=0x7f23f400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.149+0000 7f23f8ff9640 1 -- 172.21.15.27:0/3838340795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4003b70 con 0x7f23fc0b0e90 2024-03-06T07:28:56.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.149+0000 7f23f8ff9640 1 -- 172.21.15.27:0/3838340795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f4003d30 con 0x7f23fc0b0e90 2024-03-06T07:28:56.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.149+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc0c56d0 con 0x7f23fc0b0e90 2024-03-06T07:28:56.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.149+0000 7f23f8ff9640 1 -- 172.21.15.27:0/3838340795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f400aa90 con 0x7f23fc0b0e90 2024-03-06T07:28:56.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.149+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc0c5c90 con 0x7f23fc0b0e90 2024-03-06T07:28:56.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.150+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23fc068700 con 0x7f23fc0b0e90 2024-03-06T07:28:56.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.151+0000 7f23f8ff9640 1 -- 172.21.15.27:0/3838340795 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23f400ac30 con 0x7f23fc0b0e90 2024-03-06T07:28:56.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.151+0000 7f23f8ff9640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e806b4d0 0x7f23e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.151+0000 7f23f8ff9640 1 -- 172.21.15.27:0/3838340795 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23f408a000 con 0x7f23fc0b0e90 2024-03-06T07:28:56.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.151+0000 7f23fbfff640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e806b4d0 0x7f23e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.152+0000 7f23fbfff640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e806b4d0 0x7f23e806d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f23f0000f60 tx=0x7f23f0007e00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.160+0000 7f23f8ff9640 1 -- 172.21.15.27:0/3838340795 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23f4058830 con 0x7f23fc0b0e90 2024-03-06T07:28:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.354+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f23fc15e760 con 0x7f23fc0b0e90 2024-03-06T07:28:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.355+0000 7f23f8ff9640 1 -- 172.21.15.27:0/3838340795 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v114) v1 ==== 80+0+0 (secure 0 0 0) 0x7f23f405c8d0 con 0x7f23fc0b0e90 2024-03-06T07:28:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e806b4d0 msgr2=0x7f23e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e806b4d0 0x7f23e806d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f23f0000f60 tx=0x7f23f0007e00 comp rx=0 tx=0).stop 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 msgr2=0x7f23fc0c9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 0x7f23fc0c9cd0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f23f400c7a0 tx=0x7f23f400cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 shutdown_connections 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e806b4d0 0x7f23e806d990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23fc0b0e90 0x7f23fc0c9cd0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23fc0ae9b0 0x7f23fc0b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 --2- 172.21.15.27:0/3838340795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f23fc06b5c0 0x7f23fc158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 >> 172.21.15.27:0/3838340795 conn(0x7f23fc0b36d0 msgr2=0x7f23fc06ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 shutdown_connections 2024-03-06T07:28:56.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.357+0000 7f2402639640 1 -- 172.21.15.27:0/3838340795 wait complete. 2024-03-06T07:28:56.370 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:28:56.370 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:56.388 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:56.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.531+0000 7fee7968f640 1 Processor -- start 2024-03-06T07:28:56.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.531+0000 7fee7968f640 1 -- start start 2024-03-06T07:28:56.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.532+0000 7fee7968f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fee740d3400 0x7fee740b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.532+0000 7fee7968f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 0x7fee740b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.532+0000 7fee7968f640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740ce610 con 0x7fee74071540 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.532+0000 7fee7968f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee740d2f70 con 0x7fee740d3400 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.532+0000 7fee7968f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee740d4be0 con 0x7fee740b2b50 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.532+0000 7fee72ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fee74071540 0x7fee74071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59854/0 (socket says 172.21.15.27:59854) 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.532+0000 7fee72ffd640 1 -- 172.21.15.27:0/594623913 learned_addr learned my addr 172.21.15.27:0/594623913 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.533+0000 7fee737fe640 1 --2- 172.21.15.27:0/594623913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 0x7fee740b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.533+0000 7fee727fc640 1 --2- 172.21.15.27:0/594623913 >> v2:172.21.15.69:3300/0 conn(0x7fee740d3400 0x7fee740b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.533+0000 7fee71ffb640 1 -- 172.21.15.27:0/594623913 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031671626 0 0) 0x7fee740ce610 con 0x7fee74071540 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.533+0000 7fee71ffb640 1 -- 172.21.15.27:0/594623913 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee5c003520 con 0x7fee74071540 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.533+0000 7fee727fc640 1 -- 172.21.15.27:0/594623913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 msgr2=0x7fee740b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.533+0000 7fee727fc640 1 --2- 172.21.15.27:0/594623913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 0x7fee740b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.533+0000 7fee727fc640 1 -- 172.21.15.27:0/594623913 >> v1:172.21.15.27:6789/0 conn(0x7fee74071540 legacy=0x7fee74071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.533+0000 7fee727fc640 1 -- 172.21.15.27:0/594623913 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee740d2920 con 0x7fee740d3400 2024-03-06T07:28:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.533+0000 7fee737fe640 1 --2- 172.21.15.27:0/594623913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 0x7fee740b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:56.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee727fc640 1 --2- 172.21.15.27:0/594623913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee740d3400 0x7fee740b30a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fee640097a0 tx=0x7fee6402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2e02982ca0c7b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee71ffb640 1 -- 172.21.15.27:0/594623913 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64035020 con 0x7fee740d3400 2024-03-06T07:28:56.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee71ffb640 1 -- 172.21.15.27:0/594623913 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee64031830 con 0x7fee740d3400 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee71ffb640 1 -- 172.21.15.27:0/594623913 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64031be0 con 0x7fee740d3400 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee7968f640 1 -- 172.21.15.27:0/594623913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee740d3400 msgr2=0x7fee740b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee7968f640 1 --2- 172.21.15.27:0/594623913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee740d3400 0x7fee740b30a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fee640097a0 tx=0x7fee6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee7968f640 1 -- 172.21.15.27:0/594623913 shutdown_connections 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee7968f640 1 --2- 172.21.15.27:0/594623913 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 0x7fee740b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee7968f640 1 --2- 172.21.15.27:0/594623913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee740d3400 0x7fee740b30a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.534+0000 7fee7968f640 1 -- 172.21.15.27:0/594623913 >> 172.21.15.27:0/594623913 conn(0x7fee740d3ca0 msgr2=0x7fee740d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.535+0000 7fee7968f640 1 -- 172.21.15.27:0/594623913 shutdown_connections 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.535+0000 7fee7968f640 1 -- 172.21.15.27:0/594623913 wait complete. 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.536+0000 7fee7968f640 1 Processor -- start 2024-03-06T07:28:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.536+0000 7fee7968f640 1 -- start start 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee7968f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee74071540 0x7fee741f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee7968f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 0x7fee740aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee7968f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee740d3400 0x7fee740aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee7968f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7415e370 con 0x7fee74071540 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee7968f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee740ce8f0 con 0x7fee740d3400 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee7968f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7414da00 con 0x7fee740b2b50 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee72ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee74071540 0x7fee741f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee72ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee74071540 0x7fee741f2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57400/0 (socket says 172.21.15.27:57400) 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee72ffd640 1 -- 172.21.15.27:0/497401528 learned_addr learned my addr 172.21.15.27:0/497401528 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee727fc640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 0x7fee740aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee737fe640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee740d3400 0x7fee740aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee72ffd640 1 -- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 msgr2=0x7fee740aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee72ffd640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 0x7fee740aa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee72ffd640 1 -- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee740d3400 msgr2=0x7fee740aabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee72ffd640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee740d3400 0x7fee740aabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee72ffd640 1 -- 172.21.15.27:0/497401528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7414dce0 con 0x7fee74071540 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee72ffd640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee74071540 0x7fee741f2de0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c430 tx=0x7fee6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee5b7fe640 1 -- 172.21.15.27:0/497401528 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68017070 con 0x7fee74071540 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee5b7fe640 1 -- 172.21.15.27:0/497401528 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee68005ce0 con 0x7fee74071540 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7414dfc0 con 0x7fee74071540 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.537+0000 7fee5b7fe640 1 -- 172.21.15.27:0/497401528 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee680072f0 con 0x7fee74071540 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.538+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7414e580 con 0x7fee74071540 2024-03-06T07:28:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.539+0000 7fee5b7fe640 1 -- 172.21.15.27:0/497401528 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee6801a070 con 0x7fee74071540 2024-03-06T07:28:56.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.539+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee74068700 con 0x7fee74071540 2024-03-06T07:28:56.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.539+0000 7fee5b7fe640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee6006b4d0 0x7fee6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.539+0000 7fee5b7fe640 1 -- 172.21.15.27:0/497401528 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee68089690 con 0x7fee74071540 2024-03-06T07:28:56.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.540+0000 7fee727fc640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee6006b4d0 0x7fee6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.540+0000 7fee727fc640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee6006b4d0 0x7fee6006d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fee64036000 tx=0x7fee64008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.544+0000 7fee5b7fe640 1 -- 172.21.15.27:0/497401528 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee68058540 con 0x7fee74071540 2024-03-06T07:28:56.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.749+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee7414e920 con 0x7fee74071540 2024-03-06T07:28:56.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.750+0000 7fee5b7fe640 1 -- 172.21.15.27:0/497401528 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16555 (secure 0 0 0) 0x7fee68057e40 con 0x7fee74071540 2024-03-06T07:28:56.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":114,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:28:42.579409+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.260334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.929245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee6006b4d0 msgr2=0x7fee6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee6006b4d0 0x7fee6006d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fee64036000 tx=0x7fee64008030 comp rx=0 tx=0).stop 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee74071540 msgr2=0x7fee741f2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee74071540 0x7fee741f2de0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c430 tx=0x7fee6800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 shutdown_connections 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fee6006b4d0 0x7fee6006d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee740d3400 0x7fee740aabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fee740b2b50 0x7fee740aa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 --2- 172.21.15.27:0/497401528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee74071540 0x7fee741f2de0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 >> 172.21.15.27:0/497401528 conn(0x7fee740d3ca0 msgr2=0x7fee7415f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 shutdown_connections 2024-03-06T07:28:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.752+0000 7fee7968f640 1 -- 172.21.15.27:0/497401528 wait complete. 2024-03-06T07:28:56.765 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:39.260334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:39.929245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:56.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.873+0000 7fe82c6b8640 1 Processor -- start 2024-03-06T07:28:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.873+0000 7fe82c6b8640 1 -- start start 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82c6b8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 0x7fe824071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82c6b8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8240b2d30 0x7fe8240ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82c6b8640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8240cecb0 0x7fe8240cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82c6b8640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8240d3230 con 0x7fe8240cecb0 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82c6b8640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe824071e60 con 0x7fe8240b2d30 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82c6b8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8240d4960 con 0x7fe824071540 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82ac55640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8240cecb0 0x7fe8240cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82ac55640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8240cecb0 0x7fe8240cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57406/0 (socket says 172.21.15.27:57406) 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82ac55640 1 -- 172.21.15.27:0/2266166906 learned_addr learned my addr 172.21.15.27:0/2266166906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82a454640 1 --2- 172.21.15.27:0/2266166906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 0x7fe824071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe829c53640 1 --2- 172.21.15.27:0/2266166906 >> v2:172.21.15.69:3300/0 conn(0x7fe8240b2d30 0x7fe8240ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82ac55640 1 -- 172.21.15.27:0/2266166906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 msgr2=0x7fe824071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82ac55640 1 --2- 172.21.15.27:0/2266166906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 0x7fe824071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82ac55640 1 -- 172.21.15.27:0/2266166906 >> v2:172.21.15.69:3300/0 conn(0x7fe8240b2d30 msgr2=0x7fe8240ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82ac55640 1 --2- 172.21.15.27:0/2266166906 >> v2:172.21.15.69:3300/0 conn(0x7fe8240b2d30 0x7fe8240ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.874+0000 7fe82ac55640 1 -- 172.21.15.27:0/2266166906 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8240d36c0 con 0x7fe8240cecb0 2024-03-06T07:28:56.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.875+0000 7fe82ac55640 1 --2- 172.21.15.27:0/2266166906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 0x7fe8240cf130 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fe8240b68f0 tx=0x7fe81c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83c925d806e680 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe829452640 1 -- 172.21.15.27:0/2266166906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c02ca70 con 0x7fe8240cecb0 2024-03-06T07:28:56.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe829452640 1 -- 172.21.15.27:0/2266166906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c02cc10 con 0x7fe8240cecb0 2024-03-06T07:28:56.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe829452640 1 -- 172.21.15.27:0/2266166906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0343f0 con 0x7fe8240cecb0 2024-03-06T07:28:56.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe82c6b8640 1 -- 172.21.15.27:0/2266166906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 msgr2=0x7fe8240cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/2266166906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 0x7fe8240cf130 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fe8240b68f0 tx=0x7fe81c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe82c6b8640 1 -- 172.21.15.27:0/2266166906 shutdown_connections 2024-03-06T07:28:56.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/2266166906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 0x7fe8240cf130 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/2266166906 >> v2:172.21.15.69:3300/0 conn(0x7fe8240b2d30 0x7fe8240ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/2266166906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 0x7fe824071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe82c6b8640 1 -- 172.21.15.27:0/2266166906 >> 172.21.15.27:0/2266166906 conn(0x7fe8240d3d60 msgr2=0x7fe8240d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe82c6b8640 1 -- 172.21.15.27:0/2266166906 shutdown_connections 2024-03-06T07:28:56.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.876+0000 7fe82c6b8640 1 -- 172.21.15.27:0/2266166906 wait complete. 2024-03-06T07:28:56.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.878+0000 7fe82c6b8640 1 Processor -- start 2024-03-06T07:28:56.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.878+0000 7fe82c6b8640 1 -- start start 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.878+0000 7fe82c6b8640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 0x7fe8241e8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.878+0000 7fe82c6b8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8240b2d30 0x7fe8240cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.878+0000 7fe82c6b8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 0x7fe8241e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.878+0000 7fe82c6b8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8240d2be0 con 0x7fe8240cecb0 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.878+0000 7fe82c6b8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8240d2900 con 0x7fe8240b2d30 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.878+0000 7fe82c6b8640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8240d2ec0 con 0x7fe824071540 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.878+0000 7fe82ac55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 0x7fe8241e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82ac55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 0x7fe8241e49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57414/0 (socket says 172.21.15.27:57414) 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82ac55640 1 -- 172.21.15.27:0/1909189446 learned_addr learned my addr 172.21.15.27:0/1909189446 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe829c53640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8240b2d30 0x7fe8240cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82a454640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 0x7fe8241e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82ac55640 1 -- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 msgr2=0x7fe8241e8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82ac55640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 0x7fe8241e8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82ac55640 1 -- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8240b2d30 msgr2=0x7fe8240cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82ac55640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8240b2d30 0x7fe8240cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82ac55640 1 -- 172.21.15.27:0/1909189446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8241e3910 con 0x7fe8240cecb0 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82a454640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 0x7fe8241e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe829c53640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8240b2d30 0x7fe8240cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82ac55640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 0x7fe8241e49c0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fe81c02c4e0 tx=0x7fe81c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe81affd640 1 -- 172.21.15.27:0/1909189446 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c044020 con 0x7fe8240cecb0 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe81affd640 1 -- 172.21.15.27:0/1909189446 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c040690 con 0x7fe8240cecb0 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8241e3bf0 con 0x7fe8240cecb0 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe81affd640 1 -- 172.21.15.27:0/1909189446 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c009510 con 0x7fe8240cecb0 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.879+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8241e41b0 con 0x7fe8240cecb0 2024-03-06T07:28:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.880+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe824072070 con 0x7fe8240cecb0 2024-03-06T07:28:56.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.881+0000 7fe81affd640 1 -- 172.21.15.27:0/1909189446 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe81c047020 con 0x7fe8240cecb0 2024-03-06T07:28:56.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.882+0000 7fe81affd640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe80806b4d0 0x7fe80806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.882+0000 7fe81affd640 1 -- 172.21.15.27:0/1909189446 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe81c0accb0 con 0x7fe8240cecb0 2024-03-06T07:28:56.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.882+0000 7fe82a454640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe80806b4d0 0x7fe80806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.883+0000 7fe82a454640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe80806b4d0 0x7fe80806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe82000ba50 tx=0x7fe820008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:56.886+0000 7fe81affd640 1 -- 172.21.15.27:0/1909189446 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe81c04e030 con 0x7fe8240cecb0 2024-03-06T07:28:57.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.068+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe82404f410 con 0x7fe8240cecb0 2024-03-06T07:28:57.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.068+0000 7fe81affd640 1 -- 172.21.15.27:0/1909189446 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe81c07bce0 con 0x7fe8240cecb0 2024-03-06T07:28:57.071 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe80806b4d0 msgr2=0x7fe80806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe80806b4d0 0x7fe80806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe82000ba50 tx=0x7fe820008040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 msgr2=0x7fe8241e49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 0x7fe8241e49c0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fe81c02c4e0 tx=0x7fe81c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 shutdown_connections 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe80806b4d0 0x7fe80806d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8240cecb0 0x7fe8241e49c0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8240b2d30 0x7fe8240cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 --2- 172.21.15.27:0/1909189446 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe824071540 0x7fe8241e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 >> 172.21.15.27:0/1909189446 conn(0x7fe8240d3d60 msgr2=0x7fe82415bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 shutdown_connections 2024-03-06T07:28:57.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.070+0000 7fe82c6b8640 1 -- 172.21.15.27:0/1909189446 wait complete. 2024-03-06T07:28:57.084 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:57.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.192+0000 7f9035372640 1 Processor -- start 2024-03-06T07:28:57.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.192+0000 7f9035372640 1 -- start start 2024-03-06T07:28:57.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.193+0000 7f9035372640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9030072a30 0x7f90301624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.193+0000 7f9035372640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90300d07c0 0x7f903015d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.193+0000 7f9035372640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903006d4d0 con 0x7f90300cf9f0 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.193+0000 7f9035372640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903006dba0 con 0x7f90300d07c0 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.193+0000 7f9035372640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90300b4950 con 0x7f9030072a30 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.194+0000 7f902f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90300d07c0 0x7f903015d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.194+0000 7f902f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90300d07c0 0x7f903015d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57424/0 (socket says 172.21.15.27:57424) 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.194+0000 7f902f7fe640 1 -- 172.21.15.27:0/633204577 learned_addr learned my addr 172.21.15.27:0/633204577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.194+0000 7f902effd640 1 --2- 172.21.15.27:0/633204577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9030072a30 0x7f90301624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.194+0000 7f902f7fe640 1 -- 172.21.15.27:0/633204577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9030072a30 msgr2=0x7f90301624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.194+0000 7f902f7fe640 1 --2- 172.21.15.27:0/633204577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9030072a30 0x7f90301624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.194+0000 7f902f7fe640 1 -- 172.21.15.27:0/633204577 >> v1:172.21.15.69:6789/0 conn(0x7f90300cf9f0 legacy=0x7f90300d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.194+0000 7f902f7fe640 1 -- 172.21.15.27:0/633204577 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903006d7b0 con 0x7f90300d07c0 2024-03-06T07:28:57.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.194+0000 7f902effd640 1 --2- 172.21.15.27:0/633204577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9030072a30 0x7f90301624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.195+0000 7f902f7fe640 1 --2- 172.21.15.27:0/633204577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90300d07c0 0x7f903015d660 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f9028009e50 tx=0x7f902802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d161c9e89099bdf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.195+0000 7f902dffb640 1 -- 172.21.15.27:0/633204577 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902802c9e0 con 0x7f90300d07c0 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.195+0000 7f902dffb640 1 -- 172.21.15.27:0/633204577 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902802cb80 con 0x7f90300d07c0 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.195+0000 7f902dffb640 1 -- 172.21.15.27:0/633204577 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90280343f0 con 0x7f90300d07c0 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.196+0000 7f9035372640 1 -- 172.21.15.27:0/633204577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90300d07c0 msgr2=0x7f903015d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.196+0000 7f9035372640 1 --2- 172.21.15.27:0/633204577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90300d07c0 0x7f903015d660 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f9028009e50 tx=0x7f902802be40 comp rx=0 tx=0).stop 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.196+0000 7f9035372640 1 -- 172.21.15.27:0/633204577 shutdown_connections 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.196+0000 7f9035372640 1 --2- 172.21.15.27:0/633204577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90300d07c0 0x7f903015d660 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.196+0000 7f9035372640 1 --2- 172.21.15.27:0/633204577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9030072a30 0x7f90301624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.196+0000 7f9035372640 1 -- 172.21.15.27:0/633204577 >> 172.21.15.27:0/633204577 conn(0x7f903006af20 msgr2=0x7f903006b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.196+0000 7f9035372640 1 -- 172.21.15.27:0/633204577 shutdown_connections 2024-03-06T07:28:57.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.196+0000 7f9035372640 1 -- 172.21.15.27:0/633204577 wait complete. 2024-03-06T07:28:57.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.198+0000 7f9035372640 1 Processor -- start 2024-03-06T07:28:57.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.198+0000 7f9035372640 1 -- start start 2024-03-06T07:28:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.199+0000 7f9035372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9030072a30 0x7f903014eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.199+0000 7f9035372640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90300cf9f0 0x7f903015e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.199+0000 7f9035372640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90300d07c0 0x7f9030156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.199+0000 7f9035372640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9030072e10 con 0x7f9030072a30 2024-03-06T07:28:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.199+0000 7f9035372640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9030071130 con 0x7f90300cf9f0 2024-03-06T07:28:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.199+0000 7f9035372640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9030072f90 con 0x7f90300d07c0 2024-03-06T07:28:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.199+0000 7f902effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9030072a30 0x7f903014eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.199+0000 7f902effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9030072a30 0x7f903014eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57434/0 (socket says 172.21.15.27:57434) 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.199+0000 7f902effd640 1 -- 172.21.15.27:0/1950439315 learned_addr learned my addr 172.21.15.27:0/1950439315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902f7fe640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90300d07c0 0x7f9030156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902e7fc640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90300cf9f0 0x7f903015e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902effd640 1 -- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90300d07c0 msgr2=0x7f9030156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902effd640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90300d07c0 0x7f9030156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902effd640 1 -- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90300cf9f0 msgr2=0x7f903015e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902effd640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90300cf9f0 0x7f903015e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902effd640 1 -- 172.21.15.27:0/1950439315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90301574f0 con 0x7f9030072a30 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902e7fc640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90300cf9f0 0x7f903015e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902f7fe640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90300d07c0 0x7f9030156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f902effd640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9030072a30 0x7f903014eff0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f902400d7d0 tx=0x7f902400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f90177fe640 1 -- 172.21.15.27:0/1950439315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9024004030 con 0x7f9030072a30 2024-03-06T07:28:57.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f90177fe640 1 -- 172.21.15.27:0/1950439315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90240041d0 con 0x7f9030072a30 2024-03-06T07:28:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f90177fe640 1 -- 172.21.15.27:0/1950439315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902400b6c0 con 0x7f9030072a30 2024-03-06T07:28:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903009c780 con 0x7f9030072a30 2024-03-06T07:28:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.200+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90301571e0 con 0x7f9030072a30 2024-03-06T07:28:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.201+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903009ce80 con 0x7f9030072a30 2024-03-06T07:28:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.202+0000 7f90177fe640 1 -- 172.21.15.27:0/1950439315 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f902400b860 con 0x7f9030072a30 2024-03-06T07:28:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.202+0000 7f90177fe640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f901c06b4d0 0x7f901c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.202+0000 7f90177fe640 1 -- 172.21.15.27:0/1950439315 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f902408a4c0 con 0x7f9030072a30 2024-03-06T07:28:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.202+0000 7f902e7fc640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f901c06b4d0 0x7f901c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.203+0000 7f902e7fc640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f901c06b4d0 0x7f901c06d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f903015da40 tx=0x7f9020009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.212+0000 7f90177fe640 1 -- 172.21.15.27:0/1950439315 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9024059340 con 0x7f9030072a30 2024-03-06T07:28:57.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.391+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f903004f410 con 0x7f9030072a30 2024-03-06T07:28:57.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.392+0000 7f90177fe640 1 -- 172.21.15.27:0/1950439315 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9024058c40 con 0x7f9030072a30 2024-03-06T07:28:57.394 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:28:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.393+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f901c06b4d0 msgr2=0x7f901c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.393+0000 7f9035372640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f901c06b4d0 0x7f901c06d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f903015da40 tx=0x7f9020009040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.393+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9030072a30 msgr2=0x7f903014eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.393+0000 7f9035372640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9030072a30 0x7f903014eff0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f902400d7d0 tx=0x7f902400dc90 comp rx=0 tx=0).stop 2024-03-06T07:28:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.394+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 shutdown_connections 2024-03-06T07:28:57.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.394+0000 7f9035372640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f901c06b4d0 0x7f901c06d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.394+0000 7f9035372640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f90300d07c0 0x7f9030156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.394+0000 7f9035372640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90300cf9f0 0x7f903015e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.394+0000 7f9035372640 1 --2- 172.21.15.27:0/1950439315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9030072a30 0x7f903014eff0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.394+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 >> 172.21.15.27:0/1950439315 conn(0x7f903006af20 msgr2=0x7f90300afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.394+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 shutdown_connections 2024-03-06T07:28:57.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.394+0000 7f9035372640 1 -- 172.21.15.27:0/1950439315 wait complete. 2024-03-06T07:28:57.407 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.514+0000 7f1838a81640 1 Processor -- start 2024-03-06T07:28:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.514+0000 7f1838a81640 1 -- start start 2024-03-06T07:28:57.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.515+0000 7f1838a81640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18340ceea0 0x7f18340d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.515+0000 7f1838a81640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18340d3740 con 0x7f18340b6d10 2024-03-06T07:28:57.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.515+0000 7f1838a81640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18340d3120 con 0x7f183415dd90 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.515+0000 7f1838a81640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18340d3400 con 0x7f18340ceea0 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.515+0000 7f1832d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f18340b6d10 0x7f18340d0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59862/0 (socket says 172.21.15.27:59862) 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.515+0000 7f1832d9d640 1 -- 172.21.15.27:0/1253226548 learned_addr learned my addr 172.21.15.27:0/1253226548 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.515+0000 7f1831d9b640 1 --2- 172.21.15.27:0/1253226548 >> v2:172.21.15.69:3300/0 conn(0x7f18340ceea0 0x7f18340d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.515+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491437402 0 0) 0x7f18340d3740 con 0x7f18340b6d10 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.515+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1820003520 con 0x7f18340b6d10 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159648156 0 0) 0x7f1820003520 con 0x7f18340b6d10 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 >> v2:172.21.15.69:3300/0 conn(0x7f18340ceea0 msgr2=0x7f18340d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 --2- 172.21.15.27:0/1253226548 >> v2:172.21.15.69:3300/0 conn(0x7f18340ceea0 0x7f18340d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 >> v1:172.21.15.69:6789/0 conn(0x7f183415dd90 legacy=0x7f18340d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18340d39e0 con 0x7f18340b6d10 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f182c0019f0 con 0x7f18340b6d10 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 >> v1:172.21.15.27:6789/0 conn(0x7f18340b6d10 legacy=0x7f18340d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18200039b0 0x7f1820003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1820004800 0x7f1820004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18200051c0 0x7f182000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18340d3400 con 0x7f1820004800 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182000c070 con 0x7f18200039b0 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182000c270 con 0x7f18200051c0 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f1831d9b640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1820004800 0x7f1820004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f183259c640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18200039b0 0x7f1820003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f1832d9d640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18200051c0 0x7f182000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f1831d9b640 1 -- 172.21.15.27:0/1253226548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18200051c0 msgr2=0x7f182000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f1831d9b640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18200051c0 0x7f182000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f1831d9b640 1 -- 172.21.15.27:0/1253226548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18200039b0 msgr2=0x7f1820003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f1831d9b640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18200039b0 0x7f1820003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f1831d9b640 1 -- 172.21.15.27:0/1253226548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1820003520 con 0x7f1820004800 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f1832d9d640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18200051c0 0x7f182000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.516+0000 7f1832d9d640 1 -- 172.21.15.27:0/1253226548 reap_dead start 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.517+0000 7f1831d9b640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1820004800 0x7f1820004c80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f1824009ac0 tx=0x7f1820005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8010dca3a5c9c0ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.517+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824039070 con 0x7f1820004800 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.517+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182402c6b0 con 0x7f1820004800 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.517+0000 7f183159a640 1 -- 172.21.15.27:0/1253226548 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824009690 con 0x7f1820004800 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.518+0000 7f1838a81640 1 -- 172.21.15.27:0/1253226548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1820004800 msgr2=0x7f1820004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.518+0000 7f1838a81640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1820004800 0x7f1820004c80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f1824009ac0 tx=0x7f1820005830 comp rx=0 tx=0).stop 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.518+0000 7f1838a81640 1 -- 172.21.15.27:0/1253226548 shutdown_connections 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.518+0000 7f1838a81640 1 --2- 172.21.15.27:0/1253226548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1820004800 0x7f1820004c80 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.518+0000 7f1838a81640 1 -- 172.21.15.27:0/1253226548 >> 172.21.15.27:0/1253226548 conn(0x7f1834071960 msgr2=0x7f1834071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.518+0000 7f1838a81640 1 -- 172.21.15.27:0/1253226548 shutdown_connections 2024-03-06T07:28:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.518+0000 7f1838a81640 1 -- 172.21.15.27:0/1253226548 wait complete. 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.519+0000 7f1838a81640 1 Processor -- start 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.519+0000 7f1838a81640 1 -- start start 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1838a81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820004800 0x7f18341604a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1838a81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18340b8db0 0x7f1834157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1838a81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18341f4b00 0x7f18341f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1838a81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18340b71a0 con 0x7f18340b8db0 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1838a81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183415e170 con 0x7f1820004800 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1838a81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18340b7320 con 0x7f18341f4b00 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1831d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18340b8db0 0x7f1834157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1831d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18340b8db0 0x7f1834157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57454/0 (socket says 172.21.15.27:57454) 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1831d9b640 1 -- 172.21.15.27:0/2305750252 learned_addr learned my addr 172.21.15.27:0/2305750252 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1832d9d640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18341f4b00 0x7f18341f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f183259c640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820004800 0x7f18341604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1831d9b640 1 -- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18341f4b00 msgr2=0x7f18341f4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1831d9b640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18341f4b00 0x7f18341f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1831d9b640 1 -- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820004800 msgr2=0x7f18341604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1831d9b640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820004800 0x7f18341604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.520+0000 7f1831d9b640 1 -- 172.21.15.27:0/2305750252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18341f3b50 con 0x7f18340b8db0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.521+0000 7f1831d9b640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18340b8db0 0x7f1834157300 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f1824009be0 tx=0x7f1824004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.521+0000 7f182affd640 1 -- 172.21.15.27:0/2305750252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824042020 con 0x7f18340b8db0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.521+0000 7f182affd640 1 -- 172.21.15.27:0/2305750252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182403a040 con 0x7f18340b8db0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.521+0000 7f182affd640 1 -- 172.21.15.27:0/2305750252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824039070 con 0x7f18340b8db0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.521+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1834157840 con 0x7f18340b8db0 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.521+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1834160d80 con 0x7f18340b8db0 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.522+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1834072a30 con 0x7f18340b8db0 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.522+0000 7f182affd640 1 -- 172.21.15.27:0/2305750252 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1824045020 con 0x7f18340b8db0 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.523+0000 7f182affd640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f181406b4d0 0x7f181406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.523+0000 7f182affd640 1 -- 172.21.15.27:0/2305750252 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18240ab730 con 0x7f18340b8db0 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.523+0000 7f183259c640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f181406b4d0 0x7f181406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.524+0000 7f183259c640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f181406b4d0 0x7f181406d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f181c000c00 tx=0x7f181c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.527+0000 7f182affd640 1 -- 172.21.15.27:0/2305750252 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f182404c030 con 0x7f18340b8db0 2024-03-06T07:28:57.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.712+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f183404f410 con 0x7f18340b8db0 2024-03-06T07:28:57.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.712+0000 7f182affd640 1 -- 172.21.15.27:0/2305750252 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1824079ff0 con 0x7f18340b8db0 2024-03-06T07:28:57.715 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f181406b4d0 msgr2=0x7f181406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f181406b4d0 0x7f181406d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f181c000c00 tx=0x7f181c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18340b8db0 msgr2=0x7f1834157300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18340b8db0 0x7f1834157300 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f1824009be0 tx=0x7f1824004730 comp rx=0 tx=0).stop 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 shutdown_connections 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f181406b4d0 0x7f181406d990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18341f4b00 0x7f18341f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18340b8db0 0x7f1834157300 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 --2- 172.21.15.27:0/2305750252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820004800 0x7f18341604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 >> 172.21.15.27:0/2305750252 conn(0x7f1834071960 msgr2=0x7f18340b7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 shutdown_connections 2024-03-06T07:28:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.714+0000 7f1838a81640 1 -- 172.21.15.27:0/2305750252 wait complete. 2024-03-06T07:28:57.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.835+0000 7fdd59ec9640 1 Processor -- start 2024-03-06T07:28:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.835+0000 7fdd59ec9640 1 -- start start 2024-03-06T07:28:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.836+0000 7fdd59ec9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd540b2b50 0x7fdd540b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.836+0000 7fdd59ec9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd540ce610 con 0x7fdd540d3400 2024-03-06T07:28:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.836+0000 7fdd59ec9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd540d2920 con 0x7fdd54071540 2024-03-06T07:28:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.836+0000 7fdd59ec9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd540d2f70 con 0x7fdd540b2b50 2024-03-06T07:28:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.836+0000 7fdd52ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdd540d3400 0x7fdd540b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59864/0 (socket says 172.21.15.27:59864) 2024-03-06T07:28:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.836+0000 7fdd52ffd640 1 -- 172.21.15.27:0/2836783357 learned_addr learned my addr 172.21.15.27:0/2836783357 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.836+0000 7fdd53fff640 1 --2- 172.21.15.27:0/2836783357 >> v2:172.21.15.69:3300/0 conn(0x7fdd540b2b50 0x7fdd540b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981488097 0 0) 0x7fdd540ce610 con 0x7fdd540d3400 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd3c003520 con 0x7fdd540d3400 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258220005 0 0) 0x7fdd540d2920 con 0x7fdd54071540 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd540ce610 con 0x7fdd54071540 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 776228822 0 0) 0x7fdd3c003520 con 0x7fdd540d3400 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 >> v2:172.21.15.69:3300/0 conn(0x7fdd540b2b50 msgr2=0x7fdd540b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 --2- 172.21.15.27:0/2836783357 >> v2:172.21.15.69:3300/0 conn(0x7fdd540b2b50 0x7fdd540b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 >> v1:172.21.15.69:6789/0 conn(0x7fdd54071540 legacy=0x7fdd54071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd540d2c00 con 0x7fdd540d3400 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fdd480015f0 con 0x7fdd540d3400 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 >> v1:172.21.15.27:6789/0 conn(0x7fdd540d3400 legacy=0x7fdd540b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd3c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c004800 0x7fdd3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0051c0 0x7fdd3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540d2f70 con 0x7fdd3c003b90 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c00c090 con 0x7fdd3c004800 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c00c290 con 0x7fdd3c0051c0 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd537fe640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd3c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd52ffd640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c004800 0x7fdd3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd53fff640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0051c0 0x7fdd3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd537fe640 1 -- 172.21.15.27:0/2836783357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0051c0 msgr2=0x7fdd3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd537fe640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0051c0 0x7fdd3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd537fe640 1 -- 172.21.15.27:0/2836783357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c004800 msgr2=0x7fdd3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd537fe640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c004800 0x7fdd3c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd537fe640 1 -- 172.21.15.27:0/2836783357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c003520 con 0x7fdd3c003b90 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd53fff640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c0051c0 0x7fdd3c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.837+0000 7fdd53fff640 1 -- 172.21.15.27:0/2836783357 reap_dead start 2024-03-06T07:28:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd537fe640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd3c003f90 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fdd440018d0 tx=0x7fdd3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8ed4c6c42dc09b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd440014f0 con 0x7fdd3c003b90 2024-03-06T07:28:57.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4402f050 con 0x7fdd3c003b90 2024-03-06T07:28:57.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd527fc640 1 -- 172.21.15.27:0/2836783357 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd440035f0 con 0x7fdd3c003b90 2024-03-06T07:28:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd59ec9640 1 -- 172.21.15.27:0/2836783357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 msgr2=0x7fdd3c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd59ec9640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd3c003f90 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fdd440018d0 tx=0x7fdd3c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd59ec9640 1 -- 172.21.15.27:0/2836783357 shutdown_connections 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd59ec9640 1 --2- 172.21.15.27:0/2836783357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd3c003f90 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd59ec9640 1 -- 172.21.15.27:0/2836783357 >> 172.21.15.27:0/2836783357 conn(0x7fdd540d3ca0 msgr2=0x7fdd540d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd59ec9640 1 -- 172.21.15.27:0/2836783357 shutdown_connections 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.839+0000 7fdd59ec9640 1 -- 172.21.15.27:0/2836783357 wait complete. 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.841+0000 7fdd59ec9640 1 Processor -- start 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.841+0000 7fdd59ec9640 1 -- start start 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.841+0000 7fdd59ec9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd5415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.841+0000 7fdd59ec9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd541f6bd0 0x7fdd540b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.841+0000 7fdd59ec9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd540b7f50 0x7fdd540b8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.841+0000 7fdd59ec9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5415e370 con 0x7fdd3c003b90 2024-03-06T07:28:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.841+0000 7fdd59ec9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540d4be0 con 0x7fdd541f6bd0 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd59ec9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540cbdc0 con 0x7fdd540b7f50 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd537fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd5415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd537fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd5415a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57474/0 (socket says 172.21.15.27:57474) 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd537fe640 1 -- 172.21.15.27:0/1996861379 learned_addr learned my addr 172.21.15.27:0/1996861379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd52ffd640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd541f6bd0 0x7fdd540b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd53fff640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd540b7f50 0x7fdd540b8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd537fe640 1 -- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd540b7f50 msgr2=0x7fdd540b8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd537fe640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd540b7f50 0x7fdd540b8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd537fe640 1 -- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd541f6bd0 msgr2=0x7fdd540b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd537fe640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd541f6bd0 0x7fdd540b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd537fe640 1 -- 172.21.15.27:0/1996861379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd541f7090 con 0x7fdd3c003b90 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd52ffd640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd541f6bd0 0x7fdd540b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd537fe640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd5415a130 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fdd44035040 tx=0x7fdd440012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd3bfff640 1 -- 172.21.15.27:0/1996861379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44043020 con 0x7fdd3c003b90 2024-03-06T07:28:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd3bfff640 1 -- 172.21.15.27:0/1996861379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd44006e50 con 0x7fdd3c003b90 2024-03-06T07:28:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd541f7370 con 0x7fdd3c003b90 2024-03-06T07:28:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd3bfff640 1 -- 172.21.15.27:0/1996861379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd440014f0 con 0x7fdd3c003b90 2024-03-06T07:28:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.842+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd541f7930 con 0x7fdd3c003b90 2024-03-06T07:28:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.843+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd540d2640 con 0x7fdd3c003b90 2024-03-06T07:28:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.844+0000 7fdd3bfff640 1 -- 172.21.15.27:0/1996861379 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd44046020 con 0x7fdd3c003b90 2024-03-06T07:28:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.844+0000 7fdd3bfff640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd4006b4d0 0x7fdd4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.845+0000 7fdd3bfff640 1 -- 172.21.15.27:0/1996861379 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd440ac230 con 0x7fdd3c003b90 2024-03-06T07:28:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.845+0000 7fdd52ffd640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd4006b4d0 0x7fdd4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.845+0000 7fdd52ffd640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd4006b4d0 0x7fdd4006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdd540cde70 tx=0x7fdd48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:57.849+0000 7fdd3bfff640 1 -- 172.21.15.27:0/1996861379 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd540d2640 con 0x7fdd3c003b90 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.043+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdd54068700 con 0x7fdd3c003b90 2024-03-06T07:28:58.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.043+0000 7fdd3bfff640 1 -- 172.21.15.27:0/1996861379 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v114) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdd4407b1d0 con 0x7fdd3c003b90 2024-03-06T07:28:58.046 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:28:58.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd4006b4d0 msgr2=0x7fdd4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd4006b4d0 0x7fdd4006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdd540cde70 tx=0x7fdd48009040 comp rx=0 tx=0).stop 2024-03-06T07:28:58.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 msgr2=0x7fdd5415a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd5415a130 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fdd44035040 tx=0x7fdd440012d0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 shutdown_connections 2024-03-06T07:28:58.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd4006b4d0 0x7fdd4006d990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd540b7f50 0x7fdd540b8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd541f6bd0 0x7fdd540b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 --2- 172.21.15.27:0/1996861379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd3c003b90 0x7fdd5415a130 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 >> 172.21.15.27:0/1996861379 conn(0x7fdd540d3ca0 msgr2=0x7fdd540ca1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 shutdown_connections 2024-03-06T07:28:58.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.045+0000 7fdd59ec9640 1 -- 172.21.15.27:0/1996861379 wait complete. 2024-03-06T07:28:58.058 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:28:58.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:58.059 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:58.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[73048]: fuse finished with error 0 and tester_r 0 2024-03-06T07:28:58.091 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:58.133 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:28:58.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:58.134 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:58.150 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.294+0000 7ff690047640 1 Processor -- start 2024-03-06T07:28:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.295+0000 7ff690047640 1 -- start start 2024-03-06T07:28:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.295+0000 7ff690047640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815df00 0x7ff6880d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.295+0000 7ff690047640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff6880b95a0 0x7ff6880b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.295+0000 7ff690047640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6880cea30 con 0x7ff6880b9a30 2024-03-06T07:28:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.295+0000 7ff690047640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6880d2af0 con 0x7ff6880b95a0 2024-03-06T07:28:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.295+0000 7ff690047640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880cf3c0 con 0x7ff68815df00 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68e5e4640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff6880b9a30 0x7ff6880d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59874/0 (socket says 172.21.15.27:59874) 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68e5e4640 1 -- 172.21.15.27:0/2762652941 learned_addr learned my addr 172.21.15.27:0/2762652941 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68d5e2640 1 --2- 172.21.15.27:0/2762652941 >> v2:172.21.15.69:3300/0 conn(0x7ff6880b95a0 0x7ff6880b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68dde3640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815df00 0x7ff6880d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286490916 0 0) 0x7ff6880cea30 con 0x7ff6880b9a30 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff674003520 con 0x7ff6880b9a30 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1086437320 0 0) 0x7ff674003520 con 0x7ff6880b9a30 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815df00 msgr2=0x7ff6880d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815df00 0x7ff6880d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 >> v2:172.21.15.69:3300/0 conn(0x7ff6880b95a0 msgr2=0x7ff6880b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 --2- 172.21.15.27:0/2762652941 >> v2:172.21.15.69:3300/0 conn(0x7ff6880b95a0 0x7ff6880b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6880cf0e0 con 0x7ff6880b9a30 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7ff6800019f0 con 0x7ff6880b9a30 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 >> v1:172.21.15.27:6789/0 conn(0x7ff6880b9a30 legacy=0x7ff6880d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6740039b0 0x7ff674003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff674004800 0x7ff674004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815df00 0x7ff6740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880d2af0 con 0x7ff6740039b0 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880cf3c0 con 0x7ff674004800 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67400b9c0 con 0x7ff68815df00 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.296+0000 7ff68dde3640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6740039b0 0x7ff674003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.297+0000 7ff68e5e4640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815df00 0x7ff6740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.297+0000 7ff68d5e2640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff674004800 0x7ff674004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.297+0000 7ff68dde3640 1 -- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815df00 msgr2=0x7ff6740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.297+0000 7ff68dde3640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815df00 0x7ff6740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.297+0000 7ff68dde3640 1 -- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff674004800 msgr2=0x7ff674004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.297+0000 7ff68dde3640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff674004800 0x7ff674004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.297+0000 7ff68dde3640 1 -- 172.21.15.27:0/2762652941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff674003520 con 0x7ff6740039b0 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.297+0000 7ff68d5e2640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff674004800 0x7ff674004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.297+0000 7ff68dde3640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6740039b0 0x7ff674003dd0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7ff684009ac0 tx=0x7ff674005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4274811753bd6069 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.298+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684039070 con 0x7ff6740039b0 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.298+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68402c6b0 con 0x7ff6740039b0 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.298+0000 7ff68cde1640 1 -- 172.21.15.27:0/2762652941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684009690 con 0x7ff6740039b0 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.299+0000 7ff690047640 1 -- 172.21.15.27:0/2762652941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6740039b0 msgr2=0x7ff674003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.299+0000 7ff690047640 1 --2- 172.21.15.27:0/2762652941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6740039b0 0x7ff674003dd0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7ff684009ac0 tx=0x7ff674005830 comp rx=0 tx=0).stop 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.299+0000 7ff68e5e4640 1 -- 172.21.15.27:0/2762652941 reap_dead start 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.299+0000 7ff690047640 1 -- 172.21.15.27:0/2762652941 shutdown_connections 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.299+0000 7ff690047640 1 -- 172.21.15.27:0/2762652941 >> 172.21.15.27:0/2762652941 conn(0x7ff688071960 msgr2=0x7ff688071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.299+0000 7ff690047640 1 -- 172.21.15.27:0/2762652941 shutdown_connections 2024-03-06T07:28:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.299+0000 7ff690047640 1 -- 172.21.15.27:0/2762652941 wait complete. 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.301+0000 7ff690047640 1 Processor -- start 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.301+0000 7ff690047640 1 -- start start 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff690047640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6740039b0 0x7ff68814dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff690047640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff68815ecd0 0x7ff6880b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff690047640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815f150 0x7ff6880b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff690047640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880cec70 con 0x7ff68815ecd0 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff690047640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880ce750 con 0x7ff6740039b0 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff690047640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880cedf0 con 0x7ff68815f150 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff68d5e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff68815ecd0 0x7ff6880b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff68d5e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff68815ecd0 0x7ff6880b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57498/0 (socket says 172.21.15.27:57498) 2024-03-06T07:28:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff68dde3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6740039b0 0x7ff68814dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff68d5e2640 1 -- 172.21.15.27:0/1517997814 learned_addr learned my addr 172.21.15.27:0/1517997814 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.302+0000 7ff68e5e4640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815f150 0x7ff6880b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff68d5e2640 1 -- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815f150 msgr2=0x7ff6880b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff68d5e2640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815f150 0x7ff6880b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff68d5e2640 1 -- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6740039b0 msgr2=0x7ff68814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff68d5e2640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6740039b0 0x7ff68814dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff68d5e2640 1 -- 172.21.15.27:0/1517997814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68814e4b0 con 0x7ff68815ecd0 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff68dde3640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6740039b0 0x7ff68814dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff68d5e2640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff68815ecd0 0x7ff6880b48e0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff678011540 tx=0x7ff678011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff67e7fc640 1 -- 172.21.15.27:0/1517997814 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678015870 con 0x7ff68815ecd0 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff67e7fc640 1 -- 172.21.15.27:0/1517997814 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff678015a10 con 0x7ff68815ecd0 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6881f35c0 con 0x7ff68815ecd0 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff67e7fc640 1 -- 172.21.15.27:0/1517997814 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678015dc0 con 0x7ff68815ecd0 2024-03-06T07:28:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.303+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6880cb410 con 0x7ff68815ecd0 2024-03-06T07:28:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.304+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff688067ff0 con 0x7ff68815ecd0 2024-03-06T07:28:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.305+0000 7ff67e7fc640 1 -- 172.21.15.27:0/1517997814 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff67800f460 con 0x7ff68815ecd0 2024-03-06T07:28:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.305+0000 7ff67e7fc640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff66c06b500 0x7ff66c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.305+0000 7ff67e7fc640 1 -- 172.21.15.27:0/1517997814 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff67808a850 con 0x7ff68815ecd0 2024-03-06T07:28:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.305+0000 7ff68dde3640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff66c06b500 0x7ff66c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.306+0000 7ff68dde3640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff66c06b500 0x7ff66c06d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff684036000 tx=0x7ff684030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.309+0000 7ff67e7fc640 1 -- 172.21.15.27:0/1517997814 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff678019030 con 0x7ff68815ecd0 2024-03-06T07:28:58.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.533+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff68815d540 con 0x7ff68815ecd0 2024-03-06T07:28:58.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.534+0000 7ff67e7fc640 1 -- 172.21.15.27:0/1517997814 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4976 (secure 0 0 0) 0x7ff678059080 con 0x7ff68815ecd0 2024-03-06T07:28:58.536 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:28:58.537 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/103754153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":103754153},{"type":"v1","addr":"172.21.15.69:6836","nonce":103754153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4078702997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":4078702997},{"type":"v1","addr":"172.21.15.69:6837","nonce":4078702997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7956,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3926923781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3926923781},{"type":"v1","addr":"172.21.15.27:6834","nonce":3926923781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:28:43.956535+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7959":{"gid":7959,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1655684069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1655684069},{"type":"v1","addr":"172.21.15.27:6835","nonce":1655684069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7959,"qdb_cluster":[7959]},"id":8}]} 2024-03-06T07:28:58.537 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 75 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff66c06b500 msgr2=0x7ff66c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff66c06b500 0x7ff66c06d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff684036000 tx=0x7ff684030000 comp rx=0 tx=0).stop 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff68815ecd0 msgr2=0x7ff6880b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff68815ecd0 0x7ff6880b48e0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff678011540 tx=0x7ff678011a00 comp rx=0 tx=0).stop 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 shutdown_connections 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff66c06b500 0x7ff66c06d9c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff68815f150 0x7ff6880b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff68815ecd0 0x7ff6880b48e0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 --2- 172.21.15.27:0/1517997814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6740039b0 0x7ff68814dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 >> 172.21.15.27:0/1517997814 conn(0x7ff688071960 msgr2=0x7ff6880b8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 shutdown_connections 2024-03-06T07:28:58.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.536+0000 7ff690047640 1 -- 172.21.15.27:0/1517997814 wait complete. 2024-03-06T07:28:58.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:28:58.687 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:28:58.687 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:58.688 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:58.688 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:58.688 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:58.688 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:58.689 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:58.689 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:58.689 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:58.689 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:58.698 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:58.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:28:58.698 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:58.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:28:58.700 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:58.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:28:58.701 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:58.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:28:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.848+0000 7f63f4ab5640 1 Processor -- start 2024-03-06T07:28:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.848+0000 7f63f4ab5640 1 -- start start 2024-03-06T07:28:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.848+0000 7f5faa53a640 1 Processor -- start 2024-03-06T07:28:58.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.848+0000 7f66981bc640 1 Processor -- start 2024-03-06T07:28:58.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.848+0000 7f5faa53a640 1 -- start start 2024-03-06T07:28:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.848+0000 7f707d40e640 1 Processor -- start 2024-03-06T07:28:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.848+0000 7f66981bc640 1 -- start start 2024-03-06T07:28:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.848+0000 7f707d40e640 1 -- start start 2024-03-06T07:28:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f5faa53a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 0x7f5fa40b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f63f4ab5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f00aef90 0x7f63f006c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f63f4ab5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f006ccf0 con 0x7f63f0070f80 2024-03-06T07:28:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f63f4ab5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f006cfd0 con 0x7f63f0159d50 2024-03-06T07:28:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f63f4ab5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f00b28c0 con 0x7f63f00aef90 2024-03-06T07:28:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f5faa53a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa40b8100 con 0x7f5fa40b7c70 2024-03-06T07:28:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f5faa53a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa40b83c0 con 0x7f5fa40b5700 2024-03-06T07:28:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f5faa53a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa40b8fd0 con 0x7f5fa415dd70 2024-03-06T07:28:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f707d40e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7078147880 0x7f707815eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f63edd9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f63f0070f80 0x7f63f00b2d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59884/0 (socket says 172.21.15.27:59884) 2024-03-06T07:28:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f63edd9b640 1 -- 172.21.15.27:0/117339441 learned_addr learned my addr 172.21.15.27:0/117339441 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f66981bc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f669015ac30 0x7f66900b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f5fa3fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 0x7f5fa40b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f66981bc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66900afb30 0x7f66900719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f63eed9d640 1 --2- 172.21.15.27:0/117339441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f00aef90 0x7f63f006c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f707d40e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5ba0 0x7f707806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f5fa37fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5fa40b5700 0x7f5fa40d1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:37122/0 (socket says 172.21.15.27:37122) 2024-03-06T07:28:58.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f5fa37fe640 1 -- 172.21.15.27:0/2739823114 learned_addr learned my addr 172.21.15.27:0/2739823114 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.849+0000 7f5fa8ad7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5fa40b7c70 0x7f5fa40d2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59898/0 (socket says 172.21.15.27:59898) 2024-03-06T07:28:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f707d40e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707815e700 con 0x7f70780b5740 2024-03-06T07:28:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f66981bc640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66900b3e80 con 0x7f6690071ef0 2024-03-06T07:28:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f707d40e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707815f400 con 0x7f7078147880 2024-03-06T07:28:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6695757640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66900afb30 0x7f66900719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f63ed59a640 1 -- 172.21.15.27:0/117339441 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170637916 0 0) 0x7f63f006ccf0 con 0x7f63f0070f80 2024-03-06T07:28:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f63ed59a640 1 -- 172.21.15.27:0/117339441 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d8003520 con 0x7f63f0070f80 2024-03-06T07:28:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6696759640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6690071ef0 0x7f669006d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59904/0 (socket says 172.21.15.27:59904) 2024-03-06T07:28:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6696759640 1 -- 172.21.15.27:0/907049295 learned_addr learned my addr 172.21.15.27:0/907049295 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:28:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f707d40e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780b3a90 con 0x7f70780b5ba0 2024-03-06T07:28:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f7076ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7078147880 0x7f707815eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f70777fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5ba0 0x7f707806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f7076ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7078147880 0x7f707815eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57510/0 (socket says 172.21.15.27:57510) 2024-03-06T07:28:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f7076ffd640 1 -- 172.21.15.27:0/2063281031 learned_addr learned my addr 172.21.15.27:0/2063281031 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f5fa2ffd640 1 -- 172.21.15.27:0/2739823114 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120762134 0 0) 0x7f5fa40b8100 con 0x7f5fa40b7c70 2024-03-06T07:28:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f66981bc640 1 -- 172.21.15.27:0/907049295 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66900722d0 con 0x7f669015ac30 2024-03-06T07:28:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f66981bc640 1 -- 172.21.15.27:0/907049295 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66900b4120 con 0x7f66900afb30 2024-03-06T07:28:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6695f58640 1 --2- 172.21.15.27:0/907049295 >> v2:172.21.15.27:3300/0 conn(0x7f669015ac30 0x7f66900b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f63ed59a640 1 -- 172.21.15.27:0/117339441 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104185683 0 0) 0x7f63f006cfd0 con 0x7f63f0159d50 2024-03-06T07:28:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f63ed59a640 1 -- 172.21.15.27:0/117339441 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f006ccf0 con 0x7f63f0159d50 2024-03-06T07:28:58.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f5fa2ffd640 1 -- 172.21.15.27:0/2739823114 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f8c003520 con 0x7f5fa40b7c70 2024-03-06T07:28:58.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f5fa2ffd640 1 -- 172.21.15.27:0/2739823114 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112166419 0 0) 0x7f5fa40b83c0 con 0x7f5fa40b5700 2024-03-06T07:28:58.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f5fa3fff640 1 -- 172.21.15.27:0/2739823114 >> v1:172.21.15.69:6789/0 conn(0x7f5fa40b5700 legacy=0x7f5fa40d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f5fa3fff640 1 -- 172.21.15.27:0/2739823114 >> v1:172.21.15.27:6789/0 conn(0x7f5fa40b7c70 legacy=0x7f5fa40d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f63eed9d640 1 -- 172.21.15.27:0/117339441 >> v1:172.21.15.69:6789/0 conn(0x7f63f0159d50 legacy=0x7f63f0070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f63eed9d640 1 -- 172.21.15.27:0/117339441 >> v1:172.21.15.27:6789/0 conn(0x7f63f0070f80 legacy=0x7f63f00b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f63eed9d640 1 -- 172.21.15.27:0/117339441 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f006d2b0 con 0x7f63f00aef90 2024-03-06T07:28:58.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f7076ffd640 1 -- 172.21.15.27:0/2063281031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5ba0 msgr2=0x7f707806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f7076ffd640 1 --2- 172.21.15.27:0/2063281031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5ba0 0x7f707806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f7076ffd640 1 -- 172.21.15.27:0/2063281031 >> v1:172.21.15.69:6789/0 conn(0x7f70780b5740 legacy=0x7f7078066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f5fa3fff640 1 -- 172.21.15.27:0/2739823114 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa40b86a0 con 0x7f5fa415dd70 2024-03-06T07:28:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6694f56640 1 -- 172.21.15.27:0/907049295 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798907261 0 0) 0x7f66900b3e80 con 0x7f6690071ef0 2024-03-06T07:28:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6694f56640 1 -- 172.21.15.27:0/907049295 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f667c003520 con 0x7f6690071ef0 2024-03-06T07:28:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f7076ffd640 1 -- 172.21.15.27:0/2063281031 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70780b42a0 con 0x7f7078147880 2024-03-06T07:28:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f70777fe640 1 --2- 172.21.15.27:0/2063281031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5ba0 0x7f707806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6695757640 1 -- 172.21.15.27:0/907049295 >> v2:172.21.15.27:3300/0 conn(0x7f669015ac30 msgr2=0x7f66900b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6695757640 1 --2- 172.21.15.27:0/907049295 >> v2:172.21.15.27:3300/0 conn(0x7f669015ac30 0x7f66900b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6695757640 1 -- 172.21.15.27:0/907049295 >> v1:172.21.15.27:6789/0 conn(0x7f6690071ef0 legacy=0x7f669006d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.850+0000 7f6695757640 1 -- 172.21.15.27:0/907049295 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66900b4400 con 0x7f66900afb30 2024-03-06T07:28:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.851+0000 7f63eed9d640 1 --2- 172.21.15.27:0/117339441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f00aef90 0x7f63f006c710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f63f00b2a40 tx=0x7f63e802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12dcdf72fa30c53e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.851+0000 7f5fa3fff640 1 --2- 172.21.15.27:0/2739823114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 0x7f5fa40b9650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5f940097d0 tx=0x7f5f9402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e3ec0c5dacb7919 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.851+0000 7f6695757640 1 --2- 172.21.15.27:0/907049295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 0x7f66900719b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f66900d0940 tx=0x7f668002fcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddc3b9bf42a423d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.851+0000 7f63ed59a640 1 -- 172.21.15.27:0/117339441 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8039070 con 0x7f63f00aef90 2024-03-06T07:28:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.851+0000 7f63ed59a640 1 -- 172.21.15.27:0/117339441 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e802ccd0 con 0x7f63f00aef90 2024-03-06T07:28:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.851+0000 7f63ed59a640 1 -- 172.21.15.27:0/117339441 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8034660 con 0x7f63f00aef90 2024-03-06T07:28:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.851+0000 7f6694f56640 1 -- 172.21.15.27:0/907049295 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668003d070 con 0x7f66900afb30 2024-03-06T07:28:58.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f6694f56640 1 -- 172.21.15.27:0/907049295 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6680030aa0 con 0x7f66900afb30 2024-03-06T07:28:58.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f5fa2ffd640 1 -- 172.21.15.27:0/2739823114 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94034070 con 0x7f5fa415dd70 2024-03-06T07:28:58.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f5fa2ffd640 1 -- 172.21.15.27:0/2739823114 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f94030880 con 0x7f5fa415dd70 2024-03-06T07:28:58.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f5fa2ffd640 1 -- 172.21.15.27:0/2739823114 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94030c50 con 0x7f5fa415dd70 2024-03-06T07:28:58.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f6694f56640 1 -- 172.21.15.27:0/907049295 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680038440 con 0x7f66900afb30 2024-03-06T07:28:58.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f7076ffd640 1 --2- 172.21.15.27:0/2063281031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7078147880 0x7f707815eec0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f7068009870 tx=0x7f706802b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47113e925e0428f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f7075ffb640 1 -- 172.21.15.27:0/2063281031 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068037070 con 0x7f7078147880 2024-03-06T07:28:58.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f7075ffb640 1 -- 172.21.15.27:0/2063281031 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7068030710 con 0x7f7078147880 2024-03-06T07:28:58.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f7075ffb640 1 -- 172.21.15.27:0/2063281031 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068030ae0 con 0x7f7078147880 2024-03-06T07:28:58.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f5faa53a640 1 -- 172.21.15.27:0/2739823114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 msgr2=0x7f5fa40b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f5faa53a640 1 --2- 172.21.15.27:0/2739823114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 0x7f5fa40b9650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5f940097d0 tx=0x7f5f9402b9c0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f63f4ab5640 1 -- 172.21.15.27:0/117339441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f00aef90 msgr2=0x7f63f006c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f63f4ab5640 1 --2- 172.21.15.27:0/117339441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f00aef90 0x7f63f006c710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f63f00b2a40 tx=0x7f63e802bf20 comp rx=0 tx=0).stop 2024-03-06T07:28:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f66981bc640 1 -- 172.21.15.27:0/907049295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 msgr2=0x7f66900719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f66981bc640 1 --2- 172.21.15.27:0/907049295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 0x7f66900719b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f66900d0940 tx=0x7f668002fcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f5faa53a640 1 -- 172.21.15.27:0/2739823114 shutdown_connections 2024-03-06T07:28:58.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f5faa53a640 1 --2- 172.21.15.27:0/2739823114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 0x7f5fa40b9650 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.852+0000 7f5faa53a640 1 -- 172.21.15.27:0/2739823114 >> 172.21.15.27:0/2739823114 conn(0x7f5fa4071960 msgr2=0x7f5fa4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f66981bc640 1 -- 172.21.15.27:0/907049295 shutdown_connections 2024-03-06T07:28:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f66981bc640 1 --2- 172.21.15.27:0/907049295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 0x7f66900719b0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f66981bc640 1 --2- 172.21.15.27:0/907049295 >> v2:172.21.15.27:3300/0 conn(0x7f669015ac30 0x7f66900b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f66981bc640 1 -- 172.21.15.27:0/907049295 >> 172.21.15.27:0/907049295 conn(0x7f66900d0990 msgr2=0x7f66900d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f63f4ab5640 1 -- 172.21.15.27:0/117339441 shutdown_connections 2024-03-06T07:28:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f63f4ab5640 1 --2- 172.21.15.27:0/117339441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f00aef90 0x7f63f006c710 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f5faa53a640 1 -- 172.21.15.27:0/2739823114 shutdown_connections 2024-03-06T07:28:58.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f63f4ab5640 1 -- 172.21.15.27:0/117339441 >> 172.21.15.27:0/117339441 conn(0x7f63f0066d00 msgr2=0x7f63f00670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f66981bc640 1 -- 172.21.15.27:0/907049295 shutdown_connections 2024-03-06T07:28:58.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f5faa53a640 1 -- 172.21.15.27:0/2739823114 wait complete. 2024-03-06T07:28:58.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f66981bc640 1 -- 172.21.15.27:0/907049295 wait complete. 2024-03-06T07:28:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f63f4ab5640 1 -- 172.21.15.27:0/117339441 shutdown_connections 2024-03-06T07:28:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f707d40e640 1 -- 172.21.15.27:0/2063281031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7078147880 msgr2=0x7f707815eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f707d40e640 1 --2- 172.21.15.27:0/2063281031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7078147880 0x7f707815eec0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f7068009870 tx=0x7f706802b740 comp rx=0 tx=0).stop 2024-03-06T07:28:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f707d40e640 1 -- 172.21.15.27:0/2063281031 shutdown_connections 2024-03-06T07:28:58.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f707d40e640 1 --2- 172.21.15.27:0/2063281031 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5ba0 0x7f707806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f63f4ab5640 1 -- 172.21.15.27:0/117339441 wait complete. 2024-03-06T07:28:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f707d40e640 1 --2- 172.21.15.27:0/2063281031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7078147880 0x7f707815eec0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f707d40e640 1 -- 172.21.15.27:0/2063281031 >> 172.21.15.27:0/2063281031 conn(0x7f70780d0a60 msgr2=0x7f70780d0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f707d40e640 1 -- 172.21.15.27:0/2063281031 shutdown_connections 2024-03-06T07:28:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.853+0000 7f707d40e640 1 -- 172.21.15.27:0/2063281031 wait complete. 2024-03-06T07:28:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.854+0000 7f5faa53a640 1 Processor -- start 2024-03-06T07:28:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.854+0000 7f5faa53a640 1 -- start start 2024-03-06T07:28:58.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.854+0000 7f66981bc640 1 Processor -- start 2024-03-06T07:28:58.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.854+0000 7f66981bc640 1 -- start start 2024-03-06T07:28:58.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f63f4ab5640 1 Processor -- start 2024-03-06T07:28:58.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f63f4ab5640 1 -- start start 2024-03-06T07:28:58.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f707d40e640 1 Processor -- start 2024-03-06T07:28:58.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f707d40e640 1 -- start start 2024-03-06T07:28:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5faa53a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40b5700 0x7f5fa4160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5faa53a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fa40b7c70 0x7f5fa415e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5faa53a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 0x7f5fa41f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5faa53a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa40b9b90 con 0x7f5fa40b5700 2024-03-06T07:28:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5faa53a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa40b8c60 con 0x7f5fa40b7c70 2024-03-06T07:28:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5faa53a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa415e150 con 0x7f5fa415dd70 2024-03-06T07:28:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5fa3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40b5700 0x7f5fa4160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5fa3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40b5700 0x7f5fa4160ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57522/0 (socket says 172.21.15.27:57522) 2024-03-06T07:28:58.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5fa3fff640 1 -- 172.21.15.27:0/2647621303 learned_addr learned my addr 172.21.15.27:0/2647621303 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.855+0000 7f5fa37fe640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fa40b7c70 0x7f5fa415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa8ad7640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 0x7f5fa41f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa3fff640 1 -- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 msgr2=0x7f5fa41f3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa3fff640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 0x7f5fa41f3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa3fff640 1 -- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fa40b7c70 msgr2=0x7f5fa415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa3fff640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fa40b7c70 0x7f5fa415e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa3fff640 1 -- 172.21.15.27:0/2647621303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa415ff50 con 0x7f5fa40b5700 2024-03-06T07:28:58.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f66981bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6690071ef0 0x7f66900aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63f4ab5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63f0070f80 0x7f63f013a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f66981bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 0x7f669015cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f66981bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f669015ac30 0x7f66901eeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa3fff640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40b5700 0x7f5fa4160ff0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000c00 tx=0x7f5f940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63f4ab5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63f00aef90 0x7f63f015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63ee59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63f0070f80 0x7f63f013a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f707d40e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5740 0x7f707815a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f707d40e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780af690 0x7f70780ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa0ff9640 1 -- 172.21.15.27:0/2647621303 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94034040 con 0x7f5fa40b5700 2024-03-06T07:28:58.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f707d40e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780ce750 0x7f7078145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63f4ab5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f0159d50 0x7f63f015ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63f4ab5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f006e420 con 0x7f63f0070f80 2024-03-06T07:28:58.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63f4ab5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f015a130 con 0x7f63f00aef90 2024-03-06T07:28:58.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63f4ab5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f006e960 con 0x7f63f0159d50 2024-03-06T07:28:58.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63ee59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63f0070f80 0x7f63f013a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57524/0 (socket says 172.21.15.27:57524) 2024-03-06T07:28:58.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63ee59c640 1 -- 172.21.15.27:0/740803440 learned_addr learned my addr 172.21.15.27:0/740803440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa0ff9640 1 -- 172.21.15.27:0/2647621303 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f940443f0 con 0x7f5fa40b5700 2024-03-06T07:28:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5fa0ff9640 1 -- 172.21.15.27:0/2647621303 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f940447e0 con 0x7f5fa40b5700 2024-03-06T07:28:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa4160230 con 0x7f5fa40b5700 2024-03-06T07:28:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63eed9d640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f0159d50 0x7f63f015ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f63edd9b640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63f00aef90 0x7f63f015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5740 0x7f707815a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f70767fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780af690 0x7f70780ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f70767fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780af690 0x7f70780ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:52450/0 (socket says 172.21.15.27:52450) 2024-03-06T07:28:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5740 0x7f707815a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:49494/0 (socket says 172.21.15.27:49494) 2024-03-06T07:28:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f707d40e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707815e9a0 con 0x7f70780ce750 2024-03-06T07:28:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f70767fc640 1 -- 172.21.15.27:0/1399304907 learned_addr learned my addr 172.21.15.27:0/1399304907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780b3fc0 con 0x7f70780af690 2024-03-06T07:28:58.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780d1670 con 0x7f70780b5740 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f6695f58640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6690071ef0 0x7f66900aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f6695f58640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6690071ef0 0x7f66900aa6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:49482/0 (socket says 172.21.15.27:49482) 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f6695757640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 0x7f669015cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.856+0000 7f6695757640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 0x7f669015cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:52422/0 (socket says 172.21.15.27:52422) 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f66981bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66900b4ca0 con 0x7f669015ac30 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f66981bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66900b49c0 con 0x7f66900afb30 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f66981bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66900cefa0 con 0x7f6690071ef0 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f6695f58640 1 -- 172.21.15.27:0/2232625833 learned_addr learned my addr 172.21.15.27:0/2232625833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f6696759640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f669015ac30 0x7f66901eeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f70777fe640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780ce750 0x7f7078145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa41607f0 con 0x7f5fa40b5700 2024-03-06T07:28:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f63ee59c640 1 -- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f0159d50 msgr2=0x7f63f015ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f63ee59c640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f0159d50 0x7f63f015ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f63ee59c640 1 -- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63f00aef90 msgr2=0x7f63f015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f63ee59c640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63f00aef90 0x7f63f015b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f63ee59c640 1 -- 172.21.15.27:0/740803440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f013b720 con 0x7f63f0070f80 2024-03-06T07:28:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f63eed9d640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f0159d50 0x7f63f015ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f63edd9b640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63f00aef90 0x7f63f015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f63ee59c640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63f0070f80 0x7f63f013a970 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f63e000c430 tx=0x7f63e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f7076ffd640 1 -- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780af690 msgr2=0x7f70780ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f7076ffd640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780af690 0x7f70780ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f7076ffd640 1 -- 172.21.15.27:0/1399304907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780ce750 msgr2=0x7f7078145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f7076ffd640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780ce750 0x7f7078145eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f7076ffd640 1 -- 172.21.15.27:0/1399304907 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70780b0190 con 0x7f70780b5740 2024-03-06T07:28:58.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f6695f58640 1 -- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 msgr2=0x7f669015cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f6695f58640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 0x7f669015cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f6695f58640 1 -- 172.21.15.27:0/2232625833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f669015ac30 msgr2=0x7f66901eeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f6695f58640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f669015ac30 0x7f66901eeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f6695f58640 1 -- 172.21.15.27:0/2232625833 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669015d550 con 0x7f6690071ef0 2024-03-06T07:28:58.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.857+0000 7f6695f58640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6690071ef0 0x7f66900aa6a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f668c00e8a0 tx=0x7f668c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f66867fc640 1 -- 172.21.15.27:0/2232625833 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c017070 con 0x7f6690071ef0 2024-03-06T07:28:58.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f66867fc640 1 -- 172.21.15.27:0/2232625833 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668c00c940 con 0x7f6690071ef0 2024-03-06T07:28:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669015d830 con 0x7f6690071ef0 2024-03-06T07:28:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f63deffd640 1 -- 172.21.15.27:0/740803440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0017070 con 0x7f63f0070f80 2024-03-06T07:28:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f66867fc640 1 -- 172.21.15.27:0/2232625833 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c00cd10 con 0x7f6690071ef0 2024-03-06T07:28:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f63deffd640 1 -- 172.21.15.27:0/740803440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e00052f0 con 0x7f63f0070f80 2024-03-06T07:28:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f015a450 con 0x7f63f0070f80 2024-03-06T07:28:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f7076ffd640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5740 0x7f707815a990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7078071650 tx=0x7f70680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669015ddf0 con 0x7f6690071ef0 2024-03-06T07:28:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f63deffd640 1 -- 172.21.15.27:0/740803440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0003830 con 0x7f63f0070f80 2024-03-06T07:28:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f015aa10 con 0x7f63f0070f80 2024-03-06T07:28:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.858+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa4067ff0 con 0x7f5fa40b5700 2024-03-06T07:28:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.859+0000 7f705f7fe640 1 -- 172.21.15.27:0/1399304907 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068041020 con 0x7f70780b5740 2024-03-06T07:28:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.859+0000 7f705f7fe640 1 -- 172.21.15.27:0/1399304907 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706803c870 con 0x7f70780b5740 2024-03-06T07:28:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.859+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70780ae1b0 con 0x7f70780b5740 2024-03-06T07:28:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.859+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70780ae770 con 0x7f70780b5740 2024-03-06T07:28:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.859+0000 7f705f7fe640 1 -- 172.21.15.27:0/1399304907 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068041020 con 0x7f70780b5740 2024-03-06T07:28:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.859+0000 7f5fa0ff9640 1 -- 172.21.15.27:0/2647621303 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f9404b060 con 0x7f5fa40b5700 2024-03-06T07:28:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.859+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6690068700 con 0x7f6690071ef0 2024-03-06T07:28:58.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.859+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f0067ff0 con 0x7f63f0070f80 2024-03-06T07:28:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.860+0000 7f5fa0ff9640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f9006b4d0 0x7f5f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.860+0000 7f66867fc640 1 -- 172.21.15.27:0/2232625833 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f668c01a070 con 0x7f6690071ef0 2024-03-06T07:28:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.860+0000 7f5fa0ff9640 1 -- 172.21.15.27:0/2647621303 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f940ac530 con 0x7f5fa40b5700 2024-03-06T07:28:58.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.860+0000 7f5fa37fe640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f9006b4d0 0x7f5f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.860+0000 7f66867fc640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f667406b4d0 0x7f667406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.860+0000 7f66867fc640 1 -- 172.21.15.27:0/2232625833 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f668c089fa0 con 0x7f6690071ef0 2024-03-06T07:28:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.860+0000 7f6695757640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f667406b4d0 0x7f667406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.861+0000 7f63deffd640 1 -- 172.21.15.27:0/740803440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63e001a070 con 0x7f63f0070f80 2024-03-06T07:28:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.861+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7078068700 con 0x7f70780b5740 2024-03-06T07:28:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.861+0000 7f5fa37fe640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f9006b4d0 0x7f5f9006d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5fa415a3b0 tx=0x7f5f98007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.861+0000 7f6695757640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f667406b4d0 0x7f667406d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f668003a000 tx=0x7f6680034000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.861+0000 7f63deffd640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63d006b4d0 0x7f63d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.861+0000 7f63deffd640 1 -- 172.21.15.27:0/740803440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63e0089640 con 0x7f63f0070f80 2024-03-06T07:28:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.861+0000 7f705f7fe640 1 -- 172.21.15.27:0/1399304907 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7068044020 con 0x7f70780b5740 2024-03-06T07:28:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.861+0000 7f63edd9b640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63d006b4d0 0x7f63d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.862+0000 7f705f7fe640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f706406b6d0 0x7f706406db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.862+0000 7f705f7fe640 1 -- 172.21.15.27:0/1399304907 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70680ab500 con 0x7f70780b5740 2024-03-06T07:28:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.862+0000 7f63edd9b640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63d006b4d0 0x7f63d006d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f63f015d320 tx=0x7f63e4007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.862+0000 7f70767fc640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f706406b6d0 0x7f706406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.862+0000 7f70767fc640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f706406b6d0 0x7f706406db90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7078143580 tx=0x7f706c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.865+0000 7f5fa0ff9640 1 -- 172.21.15.27:0/2647621303 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f9403f070 con 0x7f5fa40b5700 2024-03-06T07:28:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.865+0000 7f66867fc640 1 -- 172.21.15.27:0/2232625833 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f668c058860 con 0x7f6690071ef0 2024-03-06T07:28:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.870+0000 7f63deffd640 1 -- 172.21.15.27:0/740803440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63e0057d50 con 0x7f63f0070f80 2024-03-06T07:28:58.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:58.870+0000 7f705f7fe640 1 -- 172.21.15.27:0/1399304907 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f706804b030 con 0x7f70780b5740 2024-03-06T07:28:59.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.121+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f669006e550 con 0x7f6690071ef0 2024-03-06T07:28:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.123+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f63f00cec90 con 0x7f63f0070f80 2024-03-06T07:28:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.124+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f5fa40cf1d0 con 0x7f5fa40b5700 2024-03-06T07:28:59.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.126+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f70780afd00 con 0x7f70780b5740 2024-03-06T07:28:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.656+0000 7f66867fc640 1 -- 172.21.15.27:0/2232625833 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6371 v76) v1 ==== 95+0+0 (secure 0 0 0) 0x7f668c05c900 con 0x7f6690071ef0 2024-03-06T07:28:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 6371 2024-03-06T07:28:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f667406b4d0 msgr2=0x7f667406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f667406b4d0 0x7f667406d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f668003a000 tx=0x7f6680034000 comp rx=0 tx=0).stop 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6690071ef0 msgr2=0x7f66900aa6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6690071ef0 0x7f66900aa6a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f668c00e8a0 tx=0x7f668c00ed60 comp rx=0 tx=0).stop 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 shutdown_connections 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f667406b4d0 0x7f667406d990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f669015ac30 0x7f66901eeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900afb30 0x7f669015cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 --2- 172.21.15.27:0/2232625833 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6690071ef0 0x7f66900aa6a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 >> 172.21.15.27:0/2232625833 conn(0x7f66900d0990 msgr2=0x7f6690066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 shutdown_connections 2024-03-06T07:28:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.659+0000 7f66981bc640 1 -- 172.21.15.27:0/2232625833 wait complete. 2024-03-06T07:28:59.664 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:28:59.662+0000 7fa8a7301640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:59.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.665+0000 7f63deffd640 1 -- 172.21.15.27:0/740803440 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7959 v77) v1 ==== 95+0+0 (secure 0 0 0) 0x7f63e005bdf0 con 0x7f63f0070f80 2024-03-06T07:28:59.667 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 7959 2024-03-06T07:28:59.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.667+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63d006b4d0 msgr2=0x7f63d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.667+0000 7f63f4ab5640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63d006b4d0 0x7f63d006d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f63f015d320 tx=0x7f63e4007d30 comp rx=0 tx=0).stop 2024-03-06T07:28:59.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.667+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63f0070f80 msgr2=0x7f63f013a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.667+0000 7f63f4ab5640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63f0070f80 0x7f63f013a970 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f63e000c430 tx=0x7f63e000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.667+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 shutdown_connections 2024-03-06T07:28:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.667+0000 7f63f4ab5640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f63d006b4d0 0x7f63d006d990 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.667+0000 7f63f4ab5640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63f0159d50 0x7f63f015ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.667+0000 7f63f4ab5640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63f00aef90 0x7f63f015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.667+0000 7f63f4ab5640 1 --2- 172.21.15.27:0/740803440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63f0070f80 0x7f63f013a970 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.668+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 >> 172.21.15.27:0/740803440 conn(0x7f63f0066d00 msgr2=0x7f63f00b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.668+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 shutdown_connections 2024-03-06T07:28:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.668+0000 7f63f4ab5640 1 -- 172.21.15.27:0/740803440 wait complete. 2024-03-06T07:28:59.676 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:59.676 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.674+0000 7f5fa0ff9640 1 -- 172.21.15.27:0/2647621303 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6368 v78) v1 ==== 95+0+0 (secure 0 0 0) 0x7f5f9407ac40 con 0x7f5fa40b5700 2024-03-06T07:28:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 6368 2024-03-06T07:28:59.679 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:59.679 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f9006b4d0 msgr2=0x7f5f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f9006b4d0 0x7f5f9006d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5fa415a3b0 tx=0x7f5f98007f00 comp rx=0 tx=0).stop 2024-03-06T07:28:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40b5700 msgr2=0x7f5fa4160ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40b5700 0x7f5fa4160ff0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000c00 tx=0x7f5f940046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 shutdown_connections 2024-03-06T07:28:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f9006b4d0 0x7f5f9006d990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fa415dd70 0x7f5fa41f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fa40b7c70 0x7f5fa415e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 --2- 172.21.15.27:0/2647621303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40b5700 0x7f5fa4160ff0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 >> 172.21.15.27:0/2647621303 conn(0x7f5fa4071960 msgr2=0x7f5fa40b9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.678+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 shutdown_connections 2024-03-06T07:28:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:28:59.679+0000 7f5faa53a640 1 -- 172.21.15.27:0/2647621303 wait complete. 2024-03-06T07:28:59.687 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:59.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:59.694 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:59.694 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:59.697 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:59.697 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:59.736 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:59.736 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:59.763 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:28:59.777 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:28:59.820 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:29:00.664 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:29:00.663+0000 7fa8a7301640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:29:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.668+0000 7f705f7fe640 1 -- 172.21.15.27:0/1399304907 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7956 v79) v1 ==== 95+0+0 (secure 0 0 0) 0x7f706807a3c0 con 0x7f70780b5740 2024-03-06T07:29:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 7956 2024-03-06T07:29:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f706406b6d0 msgr2=0x7f706406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f706406b6d0 0x7f706406db90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7078143580 tx=0x7f706c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5740 msgr2=0x7f707815a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5740 0x7f707815a990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7078071650 tx=0x7f70680046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 shutdown_connections 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f706406b6d0 0x7f706406db90 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780ce750 0x7f7078145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780af690 0x7f70780ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 --2- 172.21.15.27:0/1399304907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b5740 0x7f707815a990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 >> 172.21.15.27:0/1399304907 conn(0x7f70780d0a60 msgr2=0x7f707813f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 shutdown_connections 2024-03-06T07:29:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.671+0000 7f707d40e640 1 -- 172.21.15.27:0/1399304907 wait complete. 2024-03-06T07:29:00.690 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:29:00.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:29:00.733 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:29:00.733 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:00.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:00.821 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:29:00.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.848+0000 7f9f29b5e640 1 Processor -- start 2024-03-06T07:29:00.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.848+0000 7f9f29b5e640 1 -- start start 2024-03-06T07:29:00.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f29b5e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f2415df00 0x7f9f240d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f29b5e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9f240b9a30 0x7f9f240d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f29b5e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f240cea30 con 0x7f9f240b95a0 2024-03-06T07:29:00.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f29b5e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f240d2af0 con 0x7f9f240b9a30 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f29b5e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240cf3c0 con 0x7f9f2415df00 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f22ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9f240b95a0 0x7f9f240b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59932/0 (socket says 172.21.15.27:59932) 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f22ffd640 1 -- 172.21.15.27:0/2286508429 learned_addr learned my addr 172.21.15.27:0/2286508429 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f23fff640 1 --2- 172.21.15.27:0/2286508429 >> v2:172.21.15.69:3300/0 conn(0x7f9f240b9a30 0x7f9f240d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f237fe640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f2415df00 0x7f9f240d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97179464 0 0) 0x7f9f240cea30 con 0x7f9f240b95a0 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c003520 con 0x7f9f240b95a0 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2858886361 0 0) 0x7f9f0c003520 con 0x7f9f240b95a0 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f2415df00 msgr2=0x7f9f240d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f2415df00 0x7f9f240d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 >> v2:172.21.15.69:3300/0 conn(0x7f9f240b9a30 msgr2=0x7f9f240d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 --2- 172.21.15.27:0/2286508429 >> v2:172.21.15.69:3300/0 conn(0x7f9f240b9a30 0x7f9f240d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f240cf0e0 con 0x7f9f240b95a0 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f9f18001670 con 0x7f9f240b95a0 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.849+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 >> v1:172.21.15.27:6789/0 conn(0x7f9f240b95a0 legacy=0x7f9f240b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f227fc640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f0c0039b0 0x7f9f0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f227fc640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f2415df00 0x7f9f0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f227fc640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c004d40 0x7f9f0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240d2af0 con 0x7f9f0c004d40 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240cf3c0 con 0x7f9f0c0039b0 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c00b9c0 con 0x7f9f2415df00 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f23fff640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c004d40 0x7f9f0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f237fe640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f0c0039b0 0x7f9f0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f22ffd640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f2415df00 0x7f9f0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f23fff640 1 -- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f2415df00 msgr2=0x7f9f0c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f23fff640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f2415df00 0x7f9f0c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f23fff640 1 -- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f0c0039b0 msgr2=0x7f9f0c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f23fff640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f0c0039b0 0x7f9f0c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.850+0000 7f9f23fff640 1 -- 172.21.15.27:0/2286508429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c003520 con 0x7f9f0c004d40 2024-03-06T07:29:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.851+0000 7f9f23fff640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c004d40 0x7f9f0c0051c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009f90 tx=0x7f9f0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4fba15cf47dcc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.851+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c039070 con 0x7f9f0c004d40 2024-03-06T07:29:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.851+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1c02cc20 con 0x7f9f0c004d40 2024-03-06T07:29:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.851+0000 7f9f227fc640 1 -- 172.21.15.27:0/2286508429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c009c10 con 0x7f9f0c004d40 2024-03-06T07:29:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.851+0000 7f9f29b5e640 1 -- 172.21.15.27:0/2286508429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c004d40 msgr2=0x7f9f0c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.851+0000 7f9f29b5e640 1 --2- 172.21.15.27:0/2286508429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c004d40 0x7f9f0c0051c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009f90 tx=0x7f9f0c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.851+0000 7f9f23fff640 1 -- 172.21.15.27:0/2286508429 reap_dead start 2024-03-06T07:29:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.851+0000 7f9f29b5e640 1 -- 172.21.15.27:0/2286508429 shutdown_connections 2024-03-06T07:29:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.851+0000 7f9f29b5e640 1 -- 172.21.15.27:0/2286508429 >> 172.21.15.27:0/2286508429 conn(0x7f9f24071960 msgr2=0x7f9f24071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.852+0000 7f9f29b5e640 1 -- 172.21.15.27:0/2286508429 shutdown_connections 2024-03-06T07:29:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.852+0000 7f9f29b5e640 1 -- 172.21.15.27:0/2286508429 wait complete. 2024-03-06T07:29:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.853+0000 7f9f29b5e640 1 Processor -- start 2024-03-06T07:29:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.853+0000 7f9f29b5e640 1 -- start start 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f29b5e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f0c004d40 0x7f9f240b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f29b5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f240b2e20 0x7f9f24154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f29b5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f240b32a0 0x7f9f24155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f29b5e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240cec70 con 0x7f9f240b32a0 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f29b5e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240ce750 con 0x7f9f240b2e20 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f29b5e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240cedf0 con 0x7f9f0c004d40 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f23fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f240b32a0 0x7f9f24155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f23fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f240b32a0 0x7f9f24155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57600/0 (socket says 172.21.15.27:57600) 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f23fff640 1 -- 172.21.15.27:0/1947696804 learned_addr learned my addr 172.21.15.27:0/1947696804 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f237fe640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f0c004d40 0x7f9f240b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f22ffd640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f240b2e20 0x7f9f24154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f23fff640 1 -- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f0c004d40 msgr2=0x7f9f240b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f23fff640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f0c004d40 0x7f9f240b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f23fff640 1 -- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f240b2e20 msgr2=0x7f9f24154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f23fff640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f240b2e20 0x7f9f24154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f23fff640 1 -- 172.21.15.27:0/1947696804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2415e530 con 0x7f9f240b32a0 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f237fe640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f0c004d40 0x7f9f240b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f22ffd640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f240b2e20 0x7f9f24154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f23fff640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f240b32a0 0x7f9f24155510 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c002230 tx=0x7f9f1c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f0bfff640 1 -- 172.21.15.27:0/1947696804 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c039030 con 0x7f9f240b32a0 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.854+0000 7f9f0bfff640 1 -- 172.21.15.27:0/1947696804 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1c043070 con 0x7f9f240b32a0 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.855+0000 7f9f0bfff640 1 -- 172.21.15.27:0/1947696804 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c03d7d0 con 0x7f9f240b32a0 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.855+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2415e810 con 0x7f9f240b32a0 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.855+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2415edd0 con 0x7f9f240b32a0 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.856+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f24067ff0 con 0x7f9f240b32a0 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.856+0000 7f9f0bfff640 1 -- 172.21.15.27:0/1947696804 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f1c005d10 con 0x7f9f240b32a0 2024-03-06T07:29:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.856+0000 7f9f0bfff640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b500 0x7f9f1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.857+0000 7f9f0bfff640 1 -- 172.21.15.27:0/1947696804 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9f1c0ac4b0 con 0x7f9f240b32a0 2024-03-06T07:29:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.857+0000 7f9f237fe640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b500 0x7f9f1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.857+0000 7f9f237fe640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b500 0x7f9f1006d9c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009a40 tx=0x7f9f14008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:00.861+0000 7f9f0bfff640 1 -- 172.21.15.27:0/1947696804 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f1c07b310 con 0x7f9f240b32a0 2024-03-06T07:29:01.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.084+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f240b7530 con 0x7f9f240b32a0 2024-03-06T07:29:01.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.084+0000 7f9f0bfff640 1 -- 172.21.15.27:0/1947696804 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4183 (secure 0 0 0) 0x7f9f1c07ac10 con 0x7f9f240b32a0 2024-03-06T07:29:01.086 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:01.086 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:00.667970+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:29:01.086 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 80 2024-03-06T07:29:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b500 msgr2=0x7f9f1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b500 0x7f9f1006d9c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009a40 tx=0x7f9f14008040 comp rx=0 tx=0).stop 2024-03-06T07:29:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f240b32a0 msgr2=0x7f9f24155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f240b32a0 0x7f9f24155510 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c002230 tx=0x7f9f1c009750 comp rx=0 tx=0).stop 2024-03-06T07:29:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 shutdown_connections 2024-03-06T07:29:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9f1006b500 0x7f9f1006d9c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f240b32a0 0x7f9f24155510 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f240b2e20 0x7f9f24154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 --2- 172.21.15.27:0/1947696804 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9f0c004d40 0x7f9f240b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 >> 172.21.15.27:0/1947696804 conn(0x7f9f24071960 msgr2=0x7f9f240d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 shutdown_connections 2024-03-06T07:29:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:01.086+0000 7f9f29b5e640 1 -- 172.21.15.27:0/1947696804 wait complete. 2024-03-06T07:29:01.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:40.951614+0000', 'modified': '2024-03-06T07:29:00.667970+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 118, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8256': {'gid': 8256, 'name': 'a', 'rank': 0, 'incarnation': 80, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.27:6833/2669617374', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2669617374}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 2669617374}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:29:01.501 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8256:up:replay 2024-03-06T07:29:02.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.620+0000 7f120285c640 1 Processor -- start 2024-03-06T07:29:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.620+0000 7f120285c640 1 -- start start 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f120285c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc0b1020 0x7f11fc06af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f120285c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11fc0cd890 0x7f11fc15d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f120285c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0cc0e0 con 0x7f11fc06ed10 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f120285c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc0d28b0 con 0x7f11fc0cd890 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f120285c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0cf320 con 0x7f11fc0b1020 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f1200df9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11fc0cd890 0x7f11fc15d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f11fb7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f11fc06ed10 0x7f11fc0cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59934/0 (socket says 172.21.15.27:59934) 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f1200df9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11fc0cd890 0x7f11fc15d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57612/0 (socket says 172.21.15.27:57612) 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f11fb7fe640 1 -- 172.21.15.27:0/3011693215 learned_addr learned my addr 172.21.15.27:0/3011693215 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f11fbfff640 1 --2- 172.21.15.27:0/3011693215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc0b1020 0x7f11fc06af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f1200df9640 1 -- 172.21.15.27:0/3011693215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc0b1020 msgr2=0x7f11fc06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f1200df9640 1 --2- 172.21.15.27:0/3011693215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc0b1020 0x7f11fc06af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f1200df9640 1 -- 172.21.15.27:0/3011693215 >> v1:172.21.15.27:6789/0 conn(0x7f11fc06ed10 legacy=0x7f11fc0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f11faffd640 1 -- 172.21.15.27:0/3011693215 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295699554 0 0) 0x7f11fc0cc0e0 con 0x7f11fc06ed10 2024-03-06T07:29:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.621+0000 7f1200df9640 1 -- 172.21.15.27:0/3011693215 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc0cc3c0 con 0x7f11fc0cd890 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.622+0000 7f1200df9640 1 --2- 172.21.15.27:0/3011693215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 0x7f11fc15d700 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009d90 tx=0x7f11f402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94631be99c23b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.622+0000 7f11faffd640 1 -- 172.21.15.27:0/3011693215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4039070 con 0x7f11fc0cd890 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.622+0000 7f11faffd640 1 -- 172.21.15.27:0/3011693215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f402cc50 con 0x7f11fc0cd890 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.622+0000 7f11faffd640 1 -- 172.21.15.27:0/3011693215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4034440 con 0x7f11fc0cd890 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.623+0000 7f120285c640 1 -- 172.21.15.27:0/3011693215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 msgr2=0x7f11fc15d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.623+0000 7f120285c640 1 --2- 172.21.15.27:0/3011693215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 0x7f11fc15d700 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009d90 tx=0x7f11f402bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.623+0000 7f120285c640 1 -- 172.21.15.27:0/3011693215 shutdown_connections 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.623+0000 7f120285c640 1 --2- 172.21.15.27:0/3011693215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 0x7f11fc15d700 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.623+0000 7f120285c640 1 --2- 172.21.15.27:0/3011693215 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc0b1020 0x7f11fc06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.623+0000 7f120285c640 1 -- 172.21.15.27:0/3011693215 >> 172.21.15.27:0/3011693215 conn(0x7f11fc0d1a90 msgr2=0x7f11fc0d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.623+0000 7f120285c640 1 -- 172.21.15.27:0/3011693215 shutdown_connections 2024-03-06T07:29:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.623+0000 7f120285c640 1 -- 172.21.15.27:0/3011693215 wait complete. 2024-03-06T07:29:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.624+0000 7f120285c640 1 Processor -- start 2024-03-06T07:29:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.624+0000 7f120285c640 1 -- start start 2024-03-06T07:29:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f120285c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc06ed10 0x7f11fc0b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f120285c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc0b1020 0x7f11fc15d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f120285c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 0x7f11fc15dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f120285c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0ccc60 con 0x7f11fc0cd890 2024-03-06T07:29:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f120285c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0cc980 con 0x7f11fc0b1020 2024-03-06T07:29:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f120285c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0cdcc0 con 0x7f11fc06ed10 2024-03-06T07:29:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f1200df9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 0x7f11fc15dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f1200df9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 0x7f11fc15dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57622/0 (socket says 172.21.15.27:57622) 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f1200df9640 1 -- 172.21.15.27:0/2255392971 learned_addr learned my addr 172.21.15.27:0/2255392971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f11fbfff640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc06ed10 0x7f11fc0b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f11fb7fe640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc0b1020 0x7f11fc15d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f1200df9640 1 -- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc06ed10 msgr2=0x7f11fc0b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f1200df9640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc06ed10 0x7f11fc0b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.625+0000 7f1200df9640 1 -- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc0b1020 msgr2=0x7f11fc15d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.626+0000 7f1200df9640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc0b1020 0x7f11fc15d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.626+0000 7f1200df9640 1 -- 172.21.15.27:0/2255392971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc0b2580 con 0x7f11fc0cd890 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.626+0000 7f11fb7fe640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc0b1020 0x7f11fc15d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.626+0000 7f1200df9640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 0x7f11fc15dbf0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009d90 tx=0x7f11f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.626+0000 7f11f8ff9640 1 -- 172.21.15.27:0/2255392971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4039050 con 0x7f11fc0cd890 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.626+0000 7f11f8ff9640 1 -- 172.21.15.27:0/2255392971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4045070 con 0x7f11fc0cd890 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.626+0000 7f11f8ff9640 1 -- 172.21.15.27:0/2255392971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f403e610 con 0x7f11fc0cd890 2024-03-06T07:29:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.626+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc0b2860 con 0x7f11fc0cd890 2024-03-06T07:29:02.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.626+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc0b2e20 con 0x7f11fc0cd890 2024-03-06T07:29:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.627+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc067ff0 con 0x7f11fc0cd890 2024-03-06T07:29:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.627+0000 7f11f8ff9640 1 -- 172.21.15.27:0/2255392971 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11f403e7b0 con 0x7f11fc0cd890 2024-03-06T07:29:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.628+0000 7f11f8ff9640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11e806b4d0 0x7f11e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.628+0000 7f11f8ff9640 1 -- 172.21.15.27:0/2255392971 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f11f40acdf0 con 0x7f11fc0cd890 2024-03-06T07:29:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.628+0000 7f11fbfff640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11e806b4d0 0x7f11e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.629+0000 7f11fbfff640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11e806b4d0 0x7f11e806d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f11f0006fd0 tx=0x7f11f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.632+0000 7f11f8ff9640 1 -- 172.21.15.27:0/2255392971 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11f4032e20 con 0x7f11fc0cd890 2024-03-06T07:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.852+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11fc06f2f0 con 0x7f11fc0cd890 2024-03-06T07:29:02.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.853+0000 7f11f8ff9640 1 -- 172.21.15.27:0/2255392971 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f11f407bbb0 con 0x7f11fc0cd890 2024-03-06T07:29:02.854 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:02.854 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:01.680790+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:29:02.854 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 82 2024-03-06T07:29:02.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.854+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11e806b4d0 msgr2=0x7f11e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.854+0000 7f120285c640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11e806b4d0 0x7f11e806d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f11f0006fd0 tx=0x7f11f0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:02.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.854+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 msgr2=0x7f11fc15dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.854+0000 7f120285c640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 0x7f11fc15dbf0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009d90 tx=0x7f11f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:02.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.854+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 shutdown_connections 2024-03-06T07:29:02.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.855+0000 7f120285c640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f11e806b4d0 0x7f11e806d990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.855+0000 7f120285c640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11fc0cd890 0x7f11fc15dbf0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.855+0000 7f120285c640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11fc0b1020 0x7f11fc15d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.855+0000 7f120285c640 1 --2- 172.21.15.27:0/2255392971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11fc06ed10 0x7f11fc0b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.855+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 >> 172.21.15.27:0/2255392971 conn(0x7f11fc0d1a90 msgr2=0x7f11fc06f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.855+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 shutdown_connections 2024-03-06T07:29:02.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:02.855+0000 7f120285c640 1 -- 172.21.15.27:0/2255392971 wait complete. 2024-03-06T07:29:03.270 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:40.951614+0000', 'modified': '2024-03-06T07:29:01.680790+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 118, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8256': {'gid': 8256, 'name': 'a', 'rank': 0, 'incarnation': 80, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.27:6833/2669617374', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2669617374}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 2669617374}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:29:03.270 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8256:up:rejoin 2024-03-06T07:29:04.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.376+0000 7f824ed0c640 1 Processor -- start 2024-03-06T07:29:04.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.377+0000 7f824ed0c640 1 -- start start 2024-03-06T07:29:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.377+0000 7f824ed0c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f82480b2b50 0x7f82480b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.377+0000 7f824ed0c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82480ce610 con 0x7f8248071540 2024-03-06T07:29:04.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.377+0000 7f824ed0c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82480d2920 con 0x7f82480d3400 2024-03-06T07:29:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.377+0000 7f824ed0c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82480d2f70 con 0x7f82480b2b50 2024-03-06T07:29:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.377+0000 7f824caa8640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8248071540 0x7f8248071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57186/0 (socket says 172.21.15.27:57186) 2024-03-06T07:29:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.377+0000 7f824caa8640 1 -- 172.21.15.27:0/3290218085 learned_addr learned my addr 172.21.15.27:0/3290218085 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.378+0000 7f824d2a9640 1 --2- 172.21.15.27:0/3290218085 >> v2:172.21.15.69:3300/0 conn(0x7f82480b2b50 0x7f82480b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.378+0000 7f82477fe640 1 -- 172.21.15.27:0/3290218085 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645676080 0 0) 0x7f82480ce610 con 0x7f8248071540 2024-03-06T07:29:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.378+0000 7f82477fe640 1 -- 172.21.15.27:0/3290218085 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8238003520 con 0x7f8248071540 2024-03-06T07:29:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.378+0000 7f824d2a9640 1 -- 172.21.15.27:0/3290218085 >> v1:172.21.15.69:6789/0 conn(0x7f82480d3400 legacy=0x7f82480b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.378+0000 7f824d2a9640 1 -- 172.21.15.27:0/3290218085 >> v1:172.21.15.27:6789/0 conn(0x7f8248071540 legacy=0x7f8248071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.378+0000 7f824d2a9640 1 -- 172.21.15.27:0/3290218085 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82480d2c00 con 0x7f82480b2b50 2024-03-06T07:29:04.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.379+0000 7f824d2a9640 1 --2- 172.21.15.27:0/3290218085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82480b2b50 0x7f82480b30a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8240009d90 tx=0x7f824002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35bbf5efb72fa1cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.379+0000 7f82477fe640 1 -- 172.21.15.27:0/3290218085 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8240039070 con 0x7f82480b2b50 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.379+0000 7f82477fe640 1 -- 172.21.15.27:0/3290218085 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824002cd00 con 0x7f82480b2b50 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.379+0000 7f82477fe640 1 -- 172.21.15.27:0/3290218085 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8240034640 con 0x7f82480b2b50 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.380+0000 7f824ed0c640 1 -- 172.21.15.27:0/3290218085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82480b2b50 msgr2=0x7f82480b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.380+0000 7f824ed0c640 1 --2- 172.21.15.27:0/3290218085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82480b2b50 0x7f82480b30a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8240009d90 tx=0x7f824002bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.380+0000 7f824ed0c640 1 -- 172.21.15.27:0/3290218085 shutdown_connections 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.380+0000 7f824ed0c640 1 --2- 172.21.15.27:0/3290218085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82480b2b50 0x7f82480b30a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.380+0000 7f824ed0c640 1 -- 172.21.15.27:0/3290218085 >> 172.21.15.27:0/3290218085 conn(0x7f82480d3ca0 msgr2=0x7f82480d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.380+0000 7f824ed0c640 1 -- 172.21.15.27:0/3290218085 shutdown_connections 2024-03-06T07:29:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.380+0000 7f824ed0c640 1 -- 172.21.15.27:0/3290218085 wait complete. 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.381+0000 7f824ed0c640 1 Processor -- start 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.381+0000 7f824ed0c640 1 -- start start 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824ed0c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8248071540 0x7f824814d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824ed0c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82480b2b50 0x7f824815bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824ed0c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82480d3400 0x7f824815c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824ed0c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824815e370 con 0x7f82480d3400 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824ed0c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82480d4be0 con 0x7f8248071540 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824ed0c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82481f3d90 con 0x7f82480b2b50 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824d2a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82480d3400 0x7f824815c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824d2a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82480d3400 0x7f824815c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33550/0 (socket says 172.21.15.27:33550) 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824d2a9640 1 -- 172.21.15.27:0/254831295 learned_addr learned my addr 172.21.15.27:0/254831295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824caa8640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8248071540 0x7f824814d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f8247fff640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82480b2b50 0x7f824815bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824d2a9640 1 -- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82480b2b50 msgr2=0x7f824815bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824d2a9640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82480b2b50 0x7f824815bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824d2a9640 1 -- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8248071540 msgr2=0x7f824814d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824d2a9640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8248071540 0x7f824814d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824d2a9640 1 -- 172.21.15.27:0/254831295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82481f4070 con 0x7f82480d3400 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f8247fff640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82480b2b50 0x7f824815bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.382+0000 7f824d2a9640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82480d3400 0x7f824815c0e0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8240009d90 tx=0x7f8240004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.383+0000 7f82457fa640 1 -- 172.21.15.27:0/254831295 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8240044020 con 0x7f82480d3400 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.383+0000 7f82457fa640 1 -- 172.21.15.27:0/254831295 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824003a040 con 0x7f82480d3400 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.383+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82481f4350 con 0x7f82480d3400 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.383+0000 7f82457fa640 1 -- 172.21.15.27:0/254831295 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8240039050 con 0x7f82480d3400 2024-03-06T07:29:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.383+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82480c8170 con 0x7f82480d3400 2024-03-06T07:29:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.384+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82480d2640 con 0x7f82480d3400 2024-03-06T07:29:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.384+0000 7f82457fa640 1 -- 172.21.15.27:0/254831295 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8240003bd0 con 0x7f82480d3400 2024-03-06T07:29:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.385+0000 7f82457fa640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f822c06b500 0x7f822c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.385+0000 7f82457fa640 1 -- 172.21.15.27:0/254831295 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f82400ac7b0 con 0x7f82480d3400 2024-03-06T07:29:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.385+0000 7f824caa8640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f822c06b500 0x7f822c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.385+0000 7f824caa8640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f822c06b500 0x7f822c06d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f823c0011e0 tx=0x7f823c007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.389+0000 7f82457fa640 1 -- 172.21.15.27:0/254831295 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82480d2640 con 0x7f82480d3400 2024-03-06T07:29:04.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.627+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8248068700 con 0x7f82480d3400 2024-03-06T07:29:04.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.628+0000 7f82457fa640 1 -- 172.21.15.27:0/254831295 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f824004e060 con 0x7f82480d3400 2024-03-06T07:29:04.630 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:04.630 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f822c06b500 msgr2=0x7f822c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f822c06b500 0x7f822c06d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f823c0011e0 tx=0x7f823c007d30 comp rx=0 tx=0).stop 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82480d3400 msgr2=0x7f824815c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82480d3400 0x7f824815c0e0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8240009d90 tx=0x7f8240004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 shutdown_connections 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f822c06b500 0x7f822c06d9c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82480d3400 0x7f824815c0e0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82480b2b50 0x7f824815bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 --2- 172.21.15.27:0/254831295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8248071540 0x7f824814d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 >> 172.21.15.27:0/254831295 conn(0x7f82480d3ca0 msgr2=0x7f82480b8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 shutdown_connections 2024-03-06T07:29:04.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:04.630+0000 7f824ed0c640 1 -- 172.21.15.27:0/254831295 wait complete. 2024-03-06T07:29:05.045 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:40.951614+0000', 'modified': '2024-03-06T07:29:03.684342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 118, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8256': {'gid': 8256, 'name': 'a', 'rank': 0, 'incarnation': 80, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.27:6833/2669617374', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2669617374}, {'type': 'v1', 'addr': '172.21.15.27:6833', 'nonce': 2669617374}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8256, 'qdb_cluster': [8256]} 2024-03-06T07:29:05.045 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:05.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:29:05.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.151+0000 7f6d77f65640 1 Processor -- start 2024-03-06T07:29:05.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.152+0000 7f6d77f65640 1 -- start start 2024-03-06T07:29:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.152+0000 7f6d77f65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700d0940 0x7f6d70162390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.152+0000 7f6d77f65640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d700d1090 con 0x7f6d700cebb0 2024-03-06T07:29:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.152+0000 7f6d77f65640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7006b0a0 con 0x7f6d7006ef80 2024-03-06T07:29:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.152+0000 7f6d77f65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7006b380 con 0x7f6d700d0940 2024-03-06T07:29:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d75d01640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6d700cebb0 0x7f6d70161c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57196/0 (socket says 172.21.15.27:57196) 2024-03-06T07:29:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d75d01640 1 -- 172.21.15.27:0/899576470 learned_addr learned my addr 172.21.15.27:0/899576470 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d75500640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700d0940 0x7f6d70162390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818853986 0 0) 0x7f6d700d1090 con 0x7f6d700cebb0 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d5c003520 con 0x7f6d700cebb0 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718407 0 0) 0x7f6d5c003520 con 0x7f6d700cebb0 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700d0940 msgr2=0x7f6d70162390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700d0940 0x7f6d70162390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 >> v1:172.21.15.69:6789/0 conn(0x7f6d7006ef80 legacy=0x7f6d7006f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7006b660 con 0x7f6d700cebb0 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f6d6c001670 con 0x7f6d700cebb0 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 >> v1:172.21.15.27:6789/0 conn(0x7f6d700cebb0 legacy=0x7f6d70161c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d5c0039e0 0x7f6d5c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700d0940 0x7f6d5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c004d40 0x7f6d5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7006b380 con 0x7f6d5c004d40 2024-03-06T07:29:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c00b9c0 con 0x7f6d5c0039e0 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c00bbc0 con 0x7f6d700d0940 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.153+0000 7f6d76502640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c004d40 0x7f6d5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.154+0000 7f6d75500640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700d0940 0x7f6d5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.154+0000 7f6d75d01640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d5c0039e0 0x7f6d5c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.154+0000 7f6d75500640 1 -- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d5c0039e0 msgr2=0x7f6d5c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.154+0000 7f6d75500640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d5c0039e0 0x7f6d5c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.154+0000 7f6d75500640 1 -- 172.21.15.27:0/899576470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c004d40 msgr2=0x7f6d5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.154+0000 7f6d75500640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c004d40 0x7f6d5c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.154+0000 7f6d75500640 1 -- 172.21.15.27:0/899576470 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c003520 con 0x7f6d700d0940 2024-03-06T07:29:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.154+0000 7f6d76502640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c004d40 0x7f6d5c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:05.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.155+0000 7f6d75500640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700d0940 0x7f6d5c004800 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d60009ac0 tx=0x7f6d5c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3670d85c7b569b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.155+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60039070 con 0x7f6d700d0940 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.155+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6002c630 con 0x7f6d700d0940 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.155+0000 7f6d74cff640 1 -- 172.21.15.27:0/899576470 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60009690 con 0x7f6d700d0940 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.156+0000 7f6d77f65640 1 -- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700d0940 msgr2=0x7f6d5c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.156+0000 7f6d77f65640 1 --2- 172.21.15.27:0/899576470 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700d0940 0x7f6d5c004800 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d60009ac0 tx=0x7f6d5c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.156+0000 7f6d76502640 1 -- 172.21.15.27:0/899576470 reap_dead start 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.156+0000 7f6d77f65640 1 -- 172.21.15.27:0/899576470 shutdown_connections 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.156+0000 7f6d77f65640 1 -- 172.21.15.27:0/899576470 >> 172.21.15.27:0/899576470 conn(0x7f6d700b35d0 msgr2=0x7f6d700b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.156+0000 7f6d77f65640 1 -- 172.21.15.27:0/899576470 shutdown_connections 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.156+0000 7f6d77f65640 1 -- 172.21.15.27:0/899576470 wait complete. 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.157+0000 7f6d77f65640 1 Processor -- start 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.157+0000 7f6d77f65640 1 -- start start 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d77f65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d700d0940 0x7f6d700b7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d77f65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700c55f0 0x7f6d7015c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d77f65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d700c7070 0x7f6d700c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d77f65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7006cd10 con 0x7f6d700c7070 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d77f65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7006cb90 con 0x7f6d700d0940 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d77f65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7006ce90 con 0x7f6d700c55f0 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d76502640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d700c7070 0x7f6d700c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d76502640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d700c7070 0x7f6d700c74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33572/0 (socket says 172.21.15.27:33572) 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d76502640 1 -- 172.21.15.27:0/1450397647 learned_addr learned my addr 172.21.15.27:0/1450397647 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d75d01640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d700d0940 0x7f6d700b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d75500640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700c55f0 0x7f6d7015c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d76502640 1 -- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700c55f0 msgr2=0x7f6d7015c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d76502640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700c55f0 0x7f6d7015c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d76502640 1 -- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d700d0940 msgr2=0x7f6d700b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d76502640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d700d0940 0x7f6d700b7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d76502640 1 -- 172.21.15.27:0/1450397647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d700c5a70 con 0x7f6d700c7070 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.158+0000 7f6d75d01640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d700d0940 0x7f6d700b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.159+0000 7f6d76502640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d700c7070 0x7f6d700c74f0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f6d70072190 tx=0x7f6d6800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.159+0000 7f6d667fc640 1 -- 172.21.15.27:0/1450397647 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d68001b30 con 0x7f6d700c7070 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.159+0000 7f6d667fc640 1 -- 172.21.15.27:0/1450397647 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d68018070 con 0x7f6d700c7070 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.159+0000 7f6d667fc640 1 -- 172.21.15.27:0/1450397647 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6800d8d0 con 0x7f6d700c7070 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.159+0000 7f6d77f65640 1 -- 172.21.15.27:0/1450397647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d700c7f80 con 0x7f6d700c7070 2024-03-06T07:29:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.159+0000 7f6d77f65640 1 -- 172.21.15.27:0/1450397647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d700c8540 con 0x7f6d700c7070 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.161+0000 7f6d667fc640 1 -- 172.21.15.27:0/1450397647 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d6800da70 con 0x7f6d700c7070 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.161+0000 7f6d667fc640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d5406b4d0 0x7f6d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.161+0000 7f6d667fc640 1 -- 172.21.15.27:0/1450397647 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6d6808a270 con 0x7f6d700c7070 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.161+0000 7f6d75d01640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d5406b4d0 0x7f6d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.162+0000 7f6d75d01640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d5406b4d0 0x7f6d5406d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c000c00 tx=0x7f6d6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.162+0000 7f6d7ac2f740 1 Processor -- start 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.162+0000 7f6d7ac2f740 1 -- start start 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a470870 0x55d41a470c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4748c0 0x55d41a476360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41a3da6b0 con 0x55d41a4748c0 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41a383a60 con 0x55d41a4a0600 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d41a3d6750 con 0x55d41a470870 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d75500640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4748c0 0x55d41a476360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d75500640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4748c0 0x55d41a476360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33578/0 (socket says 172.21.15.27:33578) 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d75d01640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a470870 0x55d41a470c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d75500640 1 -- 172.21.15.27:0/311009506 learned_addr learned my addr 172.21.15.27:0/311009506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d76502640 1 --2- 172.21.15.27:0/311009506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d75500640 1 -- 172.21.15.27:0/311009506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a470870 msgr2=0x55d41a470c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d75500640 1 --2- 172.21.15.27:0/311009506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a470870 0x55d41a470c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d75500640 1 -- 172.21.15.27:0/311009506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 msgr2=0x55d41a4a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.163+0000 7f6d75500640 1 --2- 172.21.15.27:0/311009506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.164+0000 7f6d75500640 1 -- 172.21.15.27:0/311009506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d41a3a1f30 con 0x55d41a4748c0 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.164+0000 7f6d75500640 1 --2- 172.21.15.27:0/311009506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4748c0 0x55d41a476360 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f6d600099e0 tx=0x7f6d6002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88541b087261f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.164+0000 7f6d537fe640 1 -- 172.21.15.27:0/311009506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60039070 con 0x55d41a4748c0 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.164+0000 7f6d537fe640 1 -- 172.21.15.27:0/311009506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d60045070 con 0x55d41a4748c0 2024-03-06T07:29:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.164+0000 7f6d537fe640 1 -- 172.21.15.27:0/311009506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60004440 con 0x55d41a4748c0 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.164+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/311009506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4748c0 msgr2=0x55d41a476360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/311009506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4748c0 0x55d41a476360 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f6d600099e0 tx=0x7f6d6002e000 comp rx=0 tx=0).stop 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/311009506 shutdown_connections 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/311009506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/311009506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4748c0 0x55d41a476360 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/311009506 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a470870 0x55d41a470c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/311009506 >> 172.21.15.27:0/311009506 conn(0x55d41a4728f0 msgr2=0x55d41a4741b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/311009506 shutdown_connections 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/311009506 wait complete. 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 -- start start 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.165+0000 7f6d7ac2f740 1 Processor -- start 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a499260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 0x55d41a4997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41a474d50 con 0x55d41a4937d0 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41a396ea0 con 0x55d41a4a0600 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d41a475030 con 0x55d41a493280 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d7ac2f740 10 client.0 _ll_register_callbacks cb 0x55d41a396870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d75500640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a499260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d75500640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a499260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33584/0 (socket says 172.21.15.27:33584) 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d75500640 1 -- 172.21.15.27:0/3072608655 learned_addr learned my addr 172.21.15.27:0/3072608655 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d76502640 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 0x55d41a4997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.166+0000 7f6d75d01640 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d75500640 1 -- 172.21.15.27:0/3072608655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 msgr2=0x55d41a4997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d75500640 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 0x55d41a4997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d75500640 1 -- 172.21.15.27:0/3072608655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 msgr2=0x55d41a4a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d75500640 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d75500640 1 -- 172.21.15.27:0/3072608655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d41a397270 con 0x55d41a4937d0 2024-03-06T07:29:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d75500640 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a499260 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f6d60042040 tx=0x7f6d600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abec7923b47c010 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d50ff9640 1 -- 172.21.15.27:0/3072608655 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6004d020 con 0x55d41a4937d0 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d50ff9640 1 -- 172.21.15.27:0/3072608655 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6d60005b90 con 0x55d41a4937d0 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/3072608655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d60045070 con 0x55d41a4937d0 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d50ff9640 1 -- 172.21.15.27:0/3072608655 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d60057050 con 0x55d41a4937d0 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d7ac2f740 10 client.8298 fetch_fsmap learned FSMap version 83 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/3072608655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) v3 -- 0x55d41a0ef2d0 con 0x55d41a4937d0 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d50ff9640 1 -- 172.21.15.27:0/3072608655 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d60045070 con 0x55d41a4937d0 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.167+0000 7f6d50ff9640 1 -- 172.21.15.27:0/3072608655 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 83) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6d6005a020 con 0x55d41a4937d0 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.168+0000 7f6d7ac2f740 10 client.8298 fetch_fsmap finished waiting for FSMap version 83 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.168+0000 7f6d7ac2f740 10 client.8298 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.168+0000 7f6d7ac2f740 20 client.8298 populate_metadata read hostname 'smithi027' 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.168+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] conn(0x55d41a475a30 0x55d41a4da7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.168+0000 7f6d7ac2f740 4 client.8298 mds_command: new command op to 8256 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.168+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/3072608655 --> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d41a3e7bf0 con 0x55d41a475a30 2024-03-06T07:29:05.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.168+0000 7f6d75d01640 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] conn(0x55d41a475a30 0x55d41a4da7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.168+0000 7f6d50ff9640 10 client.8298 ms_handle_connect on v2:172.21.15.27:6832/2669617374 2024-03-06T07:29:05.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.168+0000 7f6d75d01640 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] conn(0x55d41a475a30 0x55d41a4da7a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=424d4a3c375573f3 server_cookie=6b0f600550960681 in_seq=0 out_seq=0 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d50ff9640 1 -- 172.21.15.27:0/3072608655 <== mds.0 v2:172.21.15.27:6832/2669617374 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d41a3e7bf0 con 0x55d41a475a30 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d50ff9640 10 client.8298 handle_command_reply: tid=0 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d7ac2f740 1 client.8298 shutdown 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/3072608655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 msgr2=0x55d41a499260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a499260 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f6d60042040 tx=0x7f6d600046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/3072608655 shutdown_connections 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] conn(0x55d41a475a30 0x55d41a4da7a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 0x55d41a4997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a499260 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/3072608655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/3072608655 >> 172.21.15.27:0/3072608655 conn(0x55d41a470870 msgr2=0x55d41a46f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.169+0000 7f6d50ff9640 0 client.8298 ms_handle_reset on v2:172.21.15.27:6832/2669617374 2024-03-06T07:29:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.170+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/3072608655 shutdown_connections 2024-03-06T07:29:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.170+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/3072608655 wait complete. 2024-03-06T07:29:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.170+0000 7f6d7ac2f740 20 client.8298 trim_cache size 0 max 16384 2024-03-06T07:29:05.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.187+0000 7f6d7ac2f740 1 Processor -- start 2024-03-06T07:29:05.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.187+0000 7f6d7ac2f740 1 -- start start 2024-03-06T07:29:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.187+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 0x55d41a4955c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.187+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a4c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.187+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.187+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d701ffbd0 con 0x55d41a4937d0 2024-03-06T07:29:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.187+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d700b5910 con 0x55d41a4a0600 2024-03-06T07:29:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.187+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d700b5f10 con 0x55d41a493280 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75500640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a4c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75500640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a4c38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33590/0 (socket says 172.21.15.27:33590) 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75500640 1 -- 172.21.15.27:0/396145176 learned_addr learned my addr 172.21.15.27:0/396145176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75d01640 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 0x55d41a4955c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d76502640 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75500640 1 -- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 msgr2=0x55d41a4955c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75500640 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 0x55d41a4955c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75500640 1 -- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 msgr2=0x55d41a4a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75500640 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75500640 1 -- 172.21.15.27:0/396145176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d41a1b96b0 con 0x55d41a4937d0 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75d01640 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 0x55d41a4955c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d76502640 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d75500640 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a4c38e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f6d60001eb0 tx=0x7f6d60049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3658b9eb1ff63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d52ffd640 1 -- 172.21.15.27:0/396145176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6004d030 con 0x55d41a4937d0 2024-03-06T07:29:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d52ffd640 1 -- 172.21.15.27:0/396145176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d60045070 con 0x55d41a4937d0 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.188+0000 7f6d52ffd640 1 -- 172.21.15.27:0/396145176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60039040 con 0x55d41a4937d0 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/396145176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 msgr2=0x55d41a4c38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a4c38e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f6d60001eb0 tx=0x7f6d60049000 comp rx=0 tx=0).stop 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/396145176 shutdown_connections 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a4a0600 0x55d41a4a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a4c38e0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/396145176 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a493280 0x55d41a4955c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/396145176 >> 172.21.15.27:0/396145176 conn(0x55d41a470870 msgr2=0x55d41a4df9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/396145176 shutdown_connections 2024-03-06T07:29:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/396145176 wait complete. 2024-03-06T07:29:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 -- start start 2024-03-06T07:29:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.189+0000 7f6d7ac2f740 1 Processor -- start 2024-03-06T07:29:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a493280 0x55d41a52a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a495050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d7ac2f740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a4a0600 0x55d41a4a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41a474d50 con 0x55d41a4937d0 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41a475030 con 0x55d41a493280 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d7ac2f740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d41a498b00 con 0x55d41a4a0600 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d7ac2f740 10 client.0 _ll_register_callbacks cb 0x7f6d60009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d75500640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a495050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d75500640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a495050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33596/0 (socket says 172.21.15.27:33596) 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.190+0000 7f6d75500640 1 -- 172.21.15.27:0/1747742914 learned_addr learned my addr 172.21.15.27:0/1747742914 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d75d01640 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a493280 0x55d41a52a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d76502640 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a4a0600 0x55d41a4a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d75500640 1 -- 172.21.15.27:0/1747742914 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a4a0600 msgr2=0x55d41a4a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d75500640 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a4a0600 0x55d41a4a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d75500640 1 -- 172.21.15.27:0/1747742914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a493280 msgr2=0x55d41a52a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d75500640 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a493280 0x55d41a52a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d75500640 1 -- 172.21.15.27:0/1747742914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d41a492390 con 0x55d41a4937d0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d75500640 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a495050 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f6d60009380 tx=0x7f6d60030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf070c01d3314fb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d50ff9640 1 -- 172.21.15.27:0/1747742914 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60062060 con 0x55d41a4937d0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d50ff9640 1 -- 172.21.15.27:0/1747742914 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6d6002c770 con 0x55d41a4937d0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/1747742914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d60045070 con 0x55d41a4937d0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d50ff9640 1 -- 172.21.15.27:0/1747742914 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d60068050 con 0x55d41a4937d0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d7ac2f740 10 client.8304 fetch_fsmap learned FSMap version 83 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/1747742914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) v3 -- 0x55d41a4973f0 con 0x55d41a4937d0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.191+0000 7f6d50ff9640 1 -- 172.21.15.27:0/1747742914 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d60045070 con 0x55d41a4937d0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d50ff9640 1 -- 172.21.15.27:0/1747742914 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 83) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6d6006b020 con 0x55d41a4937d0 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 10 client.8304 fetch_fsmap finished waiting for FSMap version 83 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 10 client.8304 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 20 client.8304 populate_metadata read hostname 'smithi027' 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] conn(0x55d41a4969b0 0x55d41a496d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 4 client.8304 mds_command: new command op to 8256 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/1747742914 --> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d41a49b180 con 0x55d41a4969b0 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d75d01640 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] conn(0x55d41a4969b0 0x55d41a496d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d50ff9640 10 client.8304 ms_handle_connect on v2:172.21.15.27:6832/2669617374 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d75d01640 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] conn(0x55d41a4969b0 0x55d41a496d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6191ced1d0feb6a server_cookie=5acf18ec399f9d9c in_seq=0 out_seq=0 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d50ff9640 1 -- 172.21.15.27:0/1747742914 <== mds.0 v2:172.21.15.27:6832/2669617374 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55d41a49b180 con 0x55d41a4969b0 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d50ff9640 10 client.8304 handle_command_reply: tid=0 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 client.8304 shutdown 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/1747742914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 msgr2=0x55d41a495050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a495050 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f6d60009380 tx=0x7f6d60030000 comp rx=0 tx=0).stop 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/1747742914 shutdown_connections 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.27:6832/2669617374,v1:172.21.15.27:6833/2669617374] conn(0x55d41a4969b0 0x55d41a496d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d41a4a0600 0x55d41a4a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d41a4937d0 0x55d41a495050 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d50ff9640 0 client.8304 ms_handle_reset on v2:172.21.15.27:6832/2669617374 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 --2- 172.21.15.27:0/1747742914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d41a493280 0x55d41a52a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.192+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/1747742914 >> 172.21.15.27:0/1747742914 conn(0x55d41a470870 msgr2=0x55d41a49b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/1747742914 shutdown_connections 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d7ac2f740 1 -- 172.21.15.27:0/1747742914 wait complete. 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d7ac2f740 20 client.8304 trim_cache size 0 max 16384 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8256, 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:29:05.199 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 4.5190956680000003, 2024-03-06T07:29:05.200 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 83, 2024-03-06T07:29:05.200 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 118, 2024-03-06T07:29:05.200 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 118, 2024-03-06T07:29:05.200 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 5.3720141430000004 2024-03-06T07:29:05.200 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:29:05.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 -- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d5406b4d0 msgr2=0x7f6d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d5406b4d0 0x7f6d5406d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c000c00 tx=0x7f6d6c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 -- 172.21.15.27:0/1450397647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d700c7070 msgr2=0x7f6d700c74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d700c7070 0x7f6d700c74f0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f6d70072190 tx=0x7f6d6800d150 comp rx=0 tx=0).stop 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 -- 172.21.15.27:0/1450397647 shutdown_connections 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d5406b4d0 0x7f6d5406d990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d700c7070 0x7f6d700c74f0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d700c55f0 0x7f6d7015c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 --2- 172.21.15.27:0/1450397647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d700d0940 0x7f6d700b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 -- 172.21.15.27:0/1450397647 >> 172.21.15.27:0/1450397647 conn(0x7f6d700b35d0 msgr2=0x7f6d700cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 -- 172.21.15.27:0/1450397647 shutdown_connections 2024-03-06T07:29:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.193+0000 7f6d77f65640 1 -- 172.21.15.27:0/1450397647 wait complete. 2024-03-06T07:29:05.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.310+0000 7f253ec0f640 1 Processor -- start 2024-03-06T07:29:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.310+0000 7f253ec0f640 1 -- start start 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f253ec0f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f253815df00 0x7f25380d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f253ec0f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25380b95a0 0x7f25380b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f253ec0f640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25380cea30 con 0x7f25380b9a30 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f253ec0f640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25380d2af0 con 0x7f25380b95a0 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f253ec0f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25380cf3c0 con 0x7f253815df00 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f2537fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25380b95a0 0x7f25380b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f253d1ac640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f25380b9a30 0x7f25380d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57202/0 (socket says 172.21.15.27:57202) 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f2537fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25380b95a0 0x7f25380b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33604/0 (socket says 172.21.15.27:33604) 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f253d1ac640 1 -- 172.21.15.27:0/4203958732 learned_addr learned my addr 172.21.15.27:0/4203958732 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f253c9ab640 1 --2- 172.21.15.27:0/4203958732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f253815df00 0x7f25380d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f2537fff640 1 -- 172.21.15.27:0/4203958732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f253815df00 msgr2=0x7f25380d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f2537fff640 1 --2- 172.21.15.27:0/4203958732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f253815df00 0x7f25380d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f2537fff640 1 -- 172.21.15.27:0/4203958732 >> v1:172.21.15.27:6789/0 conn(0x7f25380b9a30 legacy=0x7f25380d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f25377fe640 1 -- 172.21.15.27:0/4203958732 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042686285 0 0) 0x7f25380cea30 con 0x7f25380b9a30 2024-03-06T07:29:05.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.312+0000 7f2537fff640 1 -- 172.21.15.27:0/4203958732 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25380cf0e0 con 0x7f25380b95a0 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.313+0000 7f2537fff640 1 --2- 172.21.15.27:0/4203958732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25380b95a0 0x7f25380b5700 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f25240097a0 tx=0x7f252402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10218731d21a32e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.313+0000 7f25377fe640 1 -- 172.21.15.27:0/4203958732 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524035020 con 0x7f25380b95a0 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.313+0000 7f25377fe640 1 -- 172.21.15.27:0/4203958732 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2524031830 con 0x7f25380b95a0 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.313+0000 7f25377fe640 1 -- 172.21.15.27:0/4203958732 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524031c00 con 0x7f25380b95a0 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.314+0000 7f253ec0f640 1 -- 172.21.15.27:0/4203958732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25380b95a0 msgr2=0x7f25380b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.314+0000 7f253ec0f640 1 --2- 172.21.15.27:0/4203958732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25380b95a0 0x7f25380b5700 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f25240097a0 tx=0x7f252402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.314+0000 7f253ec0f640 1 -- 172.21.15.27:0/4203958732 shutdown_connections 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.314+0000 7f253ec0f640 1 --2- 172.21.15.27:0/4203958732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25380b95a0 0x7f25380b5700 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.314+0000 7f253ec0f640 1 --2- 172.21.15.27:0/4203958732 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f253815df00 0x7f25380d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.314+0000 7f253ec0f640 1 -- 172.21.15.27:0/4203958732 >> 172.21.15.27:0/4203958732 conn(0x7f2538071960 msgr2=0x7f2538071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.314+0000 7f253ec0f640 1 -- 172.21.15.27:0/4203958732 shutdown_connections 2024-03-06T07:29:05.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.314+0000 7f253ec0f640 1 -- 172.21.15.27:0/4203958732 wait complete. 2024-03-06T07:29:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.315+0000 7f253ec0f640 1 Processor -- start 2024-03-06T07:29:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.315+0000 7f253ec0f640 1 -- start start 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f253ec0f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25380b95a0 0x7f25380cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f253ec0f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f253815df00 0x7f253815f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f253ec0f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f253814c0a0 0x7f25380cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f253ec0f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25380cec70 con 0x7f253815df00 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f253ec0f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25380ce750 con 0x7f253814c0a0 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f253ec0f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25380cedf0 con 0x7f25380b95a0 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f2537fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f253815df00 0x7f253815f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f2537fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f253815df00 0x7f253815f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33612/0 (socket says 172.21.15.27:33612) 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f2537fff640 1 -- 172.21.15.27:0/4124601793 learned_addr learned my addr 172.21.15.27:0/4124601793 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f253d1ac640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f253814c0a0 0x7f25380cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f253c9ab640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25380b95a0 0x7f25380cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f2537fff640 1 -- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25380b95a0 msgr2=0x7f25380cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f2537fff640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25380b95a0 0x7f25380cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f2537fff640 1 -- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f253814c0a0 msgr2=0x7f25380cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f2537fff640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f253814c0a0 0x7f25380cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.316+0000 7f2537fff640 1 -- 172.21.15.27:0/4124601793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253815f950 con 0x7f253815df00 2024-03-06T07:29:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.317+0000 7f2537fff640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f253815df00 0x7f253815f410 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f2524034040 tx=0x7f25240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.317+0000 7f25357fa640 1 -- 172.21.15.27:0/4124601793 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524035040 con 0x7f253815df00 2024-03-06T07:29:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.317+0000 7f25357fa640 1 -- 172.21.15.27:0/4124601793 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2524004250 con 0x7f253815df00 2024-03-06T07:29:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.317+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25380cb3e0 con 0x7f253815df00 2024-03-06T07:29:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.317+0000 7f25357fa640 1 -- 172.21.15.27:0/4124601793 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252403a700 con 0x7f253815df00 2024-03-06T07:29:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.317+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25380cb9a0 con 0x7f253815df00 2024-03-06T07:29:05.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.318+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2538067ff0 con 0x7f253815df00 2024-03-06T07:29:05.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.318+0000 7f25357fa640 1 -- 172.21.15.27:0/4124601793 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2524038020 con 0x7f253815df00 2024-03-06T07:29:05.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.319+0000 7f25357fa640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f251c06b4d0 0x7f251c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.319+0000 7f25357fa640 1 -- 172.21.15.27:0/4124601793 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f25240abf50 con 0x7f253815df00 2024-03-06T07:29:05.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.319+0000 7f253c9ab640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f251c06b4d0 0x7f251c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.319+0000 7f253c9ab640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f251c06b4d0 0x7f251c06d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f252c006fd0 tx=0x7f252c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.323+0000 7f25357fa640 1 -- 172.21.15.27:0/4124601793 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2524031800 con 0x7f253815df00 2024-03-06T07:29:05.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.549+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f253815d3f0 con 0x7f253815df00 2024-03-06T07:29:05.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.550+0000 7f25357fa640 1 -- 172.21.15.27:0/4124601793 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v83) v1 ==== 53+0+73 (secure 0 0 0) 0x7f252407a640 con 0x7f253815df00 2024-03-06T07:29:05.551 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f251c06b4d0 msgr2=0x7f251c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f251c06b4d0 0x7f251c06d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f252c006fd0 tx=0x7f252c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f253815df00 msgr2=0x7f253815f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f253815df00 0x7f253815f410 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f2524034040 tx=0x7f25240046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 shutdown_connections 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f251c06b4d0 0x7f251c06d990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f253814c0a0 0x7f25380cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f253815df00 0x7f253815f410 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 --2- 172.21.15.27:0/4124601793 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25380b95a0 0x7f25380cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 >> 172.21.15.27:0/4124601793 conn(0x7f2538071960 msgr2=0x7f25380b8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 shutdown_connections 2024-03-06T07:29:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:05.552+0000 7f253ec0f640 1 -- 172.21.15.27:0/4124601793 wait complete. 2024-03-06T07:29:05.564 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:29:05.565 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:05.565 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:29:05.565 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:29:05.565 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:05.565 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:29:05.565 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:29:05.565 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:05.565 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:05.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:05.565 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:29:05.581 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42344sec preferred_lft 42344sec 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:29:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:05.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:05.584 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:05.650 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:29:05.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:05.652 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:29:05.675 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:29:05.674+0000 7f61c459e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:05.690 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:29:05.691 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:05.691 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:29:05.691 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:29:05.691 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:05.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:05.691 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:29:05.691 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:29:05.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:29:05.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:29:05.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:29:05.691 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:29:05.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:05.797 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:29:05.813 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:29:05.813 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:29:05.813 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:05.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:05.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:29:05.884 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:29:05.884 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:05.884 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:05.939 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:05.939 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:05.939 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:05.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:29:06.071 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:29:06.128 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:29:06.128 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:29:06.128 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:29:06.128 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:29:06.128 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:06.131 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:29:06.183 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:29:06.183 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:29:06.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:29:06.263 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:29:06.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:29:06.319+0000 7f4bec46a480 -1 init, newargv = 0x55be914169d0 newargc=15 2024-03-06T07:29:06.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[74699]: starting ceph client 2024-03-06T07:29:06.321 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:29:06.321 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:29:06.321 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:29:06.321 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:29:06.321 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:29:06.321 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:29:06.321 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:29:06.322 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:06.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:06.324 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:29:06.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[74699]: starting fuse 2024-03-06T07:29:06.340 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:29:06.340 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:29:06.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:29:06.340 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:29:06.341 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:29:06.342 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:29:06.342 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:29:06.342 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:29:06.342 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:29:06.342 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:29:06.342 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:29:06.342 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:29:06.342 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:29:06.342 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:29:06.465 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.74699.asok 2024-03-06T07:29:06.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:29:06.477 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.74699.asok 2024-03-06T07:29:06.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:06.477 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.74699.asok status 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "74699", 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:29:06.635 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8319, 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 8319 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3708923859 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:29:06.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3708923859 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.8319 192.168.0.1:0/3708923859", 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/3708923859", 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 83, 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 118, 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:29:06.637 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:29:06.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:06.643 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:29:06.643 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:06.659 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:29:06.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:06.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:06.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:06.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:06.730 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:29:06.789 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.934+0000 7f1543dac640 1 Processor -- start 2024-03-06T07:29:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.934+0000 7f1543dac640 1 -- start start 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1543dac640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f153c15df00 0x7f153c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1543dac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f153c0b9a30 0x7f153c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1543dac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0cea30 con 0x7f153c0b95a0 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1543dac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c0d2af0 con 0x7f153c0b9a30 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1543dac640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c0cf3c0 con 0x7f153c15df00 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1542349640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f153c0b9a30 0x7f153c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1542349640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f153c0b9a30 0x7f153c0d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33622/0 (socket says 172.21.15.27:33622) 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1542349640 1 -- 172.21.15.27:0/1474999080 learned_addr learned my addr 172.21.15.27:0/1474999080 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1541b48640 1 --2- 172.21.15.27:0/1474999080 >> v2:172.21.15.69:3300/0 conn(0x7f153c15df00 0x7f153c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1542349640 1 -- 172.21.15.27:0/1474999080 >> v2:172.21.15.69:3300/0 conn(0x7f153c15df00 msgr2=0x7f153c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1542349640 1 --2- 172.21.15.27:0/1474999080 >> v2:172.21.15.69:3300/0 conn(0x7f153c15df00 0x7f153c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1542349640 1 -- 172.21.15.27:0/1474999080 >> v1:172.21.15.27:6789/0 conn(0x7f153c0b95a0 legacy=0x7f153c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1542349640 1 -- 172.21.15.27:0/1474999080 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0cf0e0 con 0x7f153c0b9a30 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1540b46640 1 -- 172.21.15.27:0/1474999080 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098851412 0 0) 0x7f153c0cea30 con 0x7f153c0b95a0 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.935+0000 7f1541b48640 1 --2- 172.21.15.27:0/1474999080 >> v2:172.21.15.69:3300/0 conn(0x7f153c15df00 0x7f153c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.936+0000 7f1542349640 1 --2- 172.21.15.27:0/1474999080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0b9a30 0x7f153c0d3220 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f153c0d11b0 tx=0x7f153402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69fd19962742dc52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.936+0000 7f1540b46640 1 -- 172.21.15.27:0/1474999080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153402cb20 con 0x7f153c0b9a30 2024-03-06T07:29:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.936+0000 7f1540b46640 1 -- 172.21.15.27:0/1474999080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153402ccc0 con 0x7f153c0b9a30 2024-03-06T07:29:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.936+0000 7f1540b46640 1 -- 172.21.15.27:0/1474999080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534034610 con 0x7f153c0b9a30 2024-03-06T07:29:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.937+0000 7f1543dac640 1 -- 172.21.15.27:0/1474999080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0b9a30 msgr2=0x7f153c0d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.937+0000 7f1543dac640 1 --2- 172.21.15.27:0/1474999080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0b9a30 0x7f153c0d3220 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f153c0d11b0 tx=0x7f153402bf80 comp rx=0 tx=0).stop 2024-03-06T07:29:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.937+0000 7f1543dac640 1 -- 172.21.15.27:0/1474999080 shutdown_connections 2024-03-06T07:29:06.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.937+0000 7f1543dac640 1 --2- 172.21.15.27:0/1474999080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0b9a30 0x7f153c0d3220 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.937+0000 7f1543dac640 1 --2- 172.21.15.27:0/1474999080 >> v2:172.21.15.69:3300/0 conn(0x7f153c15df00 0x7f153c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.937+0000 7f1543dac640 1 -- 172.21.15.27:0/1474999080 >> 172.21.15.27:0/1474999080 conn(0x7f153c071960 msgr2=0x7f153c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.937+0000 7f1543dac640 1 -- 172.21.15.27:0/1474999080 shutdown_connections 2024-03-06T07:29:06.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.937+0000 7f1543dac640 1 -- 172.21.15.27:0/1474999080 wait complete. 2024-03-06T07:29:06.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.939+0000 7f1543dac640 1 Processor -- start 2024-03-06T07:29:06.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.939+0000 7f1543dac640 1 -- start start 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1543dac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b95a0 0x7f153c1f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1543dac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c15df00 0x7f153c0cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1543dac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0cd550 0x7f153c15fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1543dac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0cec70 con 0x7f153c0cd550 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1543dac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0ce750 con 0x7f153c0b95a0 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1543dac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0cedf0 con 0x7f153c15df00 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1542349640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0cd550 0x7f153c15fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1542349640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0cd550 0x7f153c15fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33626/0 (socket says 172.21.15.27:33626) 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1542349640 1 -- 172.21.15.27:0/1603521231 learned_addr learned my addr 172.21.15.27:0/1603521231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1541347640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c15df00 0x7f153c0cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1541b48640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b95a0 0x7f153c1f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1542349640 1 -- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c15df00 msgr2=0x7f153c0cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1542349640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c15df00 0x7f153c0cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1542349640 1 -- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b95a0 msgr2=0x7f153c1f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1542349640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b95a0 0x7f153c1f2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1542349640 1 -- 172.21.15.27:0/1603521231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c1f7ae0 con 0x7f153c0cd550 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1542349640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0cd550 0x7f153c15fce0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1534009f40 tx=0x7f1534004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f15327fc640 1 -- 172.21.15.27:0/1603521231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534043030 con 0x7f153c0cd550 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f15327fc640 1 -- 172.21.15.27:0/1603521231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153403e450 con 0x7f153c0cd550 2024-03-06T07:29:06.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.941+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c1f6d20 con 0x7f153c0cd550 2024-03-06T07:29:06.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.942+0000 7f15327fc640 1 -- 172.21.15.27:0/1603521231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534009510 con 0x7f153c0cd550 2024-03-06T07:29:06.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.942+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c1f72e0 con 0x7f153c0cd550 2024-03-06T07:29:06.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.942+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c067ff0 con 0x7f153c0cd550 2024-03-06T07:29:06.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.943+0000 7f15327fc640 1 -- 172.21.15.27:0/1603521231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1534046070 con 0x7f153c0cd550 2024-03-06T07:29:06.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.943+0000 7f15327fc640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.944+0000 7f15327fc640 1 -- 172.21.15.27:0/1603521231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f15340ac810 con 0x7f153c0cd550 2024-03-06T07:29:06.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.944+0000 7f1541b48640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.944+0000 7f1541b48640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f152c006fd0 tx=0x7f152c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:06.948+0000 7f15327fc640 1 -- 172.21.15.27:0/1603521231 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f153404d070 con 0x7f153c0cd550 2024-03-06T07:29:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.169+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f153c0688a0 con 0x7f153c0cd550 2024-03-06T07:29:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.170+0000 7f15327fc640 1 -- 172.21.15.27:0/1603521231 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f153407b790 con 0x7f153c0cd550 2024-03-06T07:29:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:07.172 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 msgr2=0x7f152006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f152c006fd0 tx=0x7f152c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0cd550 msgr2=0x7f153c15fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0cd550 0x7f153c15fce0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1534009f40 tx=0x7f1534004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 shutdown_connections 2024-03-06T07:29:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f152006b4d0 0x7f152006d990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0cd550 0x7f153c15fce0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f153c15df00 0x7f153c0cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 --2- 172.21.15.27:0/1603521231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f153c0b95a0 0x7f153c1f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 >> 172.21.15.27:0/1603521231 conn(0x7f153c071960 msgr2=0x7f153c0b8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 shutdown_connections 2024-03-06T07:29:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.173+0000 7f1543dac640 1 -- 172.21.15.27:0/1603521231 wait complete. 2024-03-06T07:29:07.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:29:07.313 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:29:07.314 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:29:07.314 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.420+0000 7f2a6b6f4640 1 Processor -- start 2024-03-06T07:29:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.421+0000 7f2a6b6f4640 1 -- start start 2024-03-06T07:29:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.421+0000 7f2a6b6f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6415df00 0x7f2a640d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.421+0000 7f2a6b6f4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a640b95a0 0x7f2a640b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.421+0000 7f2a6b6f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a640cea30 con 0x7f2a640b9a30 2024-03-06T07:29:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.421+0000 7f2a6b6f4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a640d2af0 con 0x7f2a640b95a0 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.421+0000 7f2a6b6f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a640cf3c0 con 0x7f2a6415df00 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a69490640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6415df00 0x7f2a640d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a68c8f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a640b95a0 0x7f2a640b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a69490640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6415df00 0x7f2a640d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:42252/0 (socket says 172.21.15.27:42252) 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a68c8f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a640b95a0 0x7f2a640b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:48462/0 (socket says 172.21.15.27:48462) 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a69490640 1 -- 172.21.15.27:0/2619999705 learned_addr learned my addr 172.21.15.27:0/2619999705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a68c8f640 1 -- 172.21.15.27:0/2619999705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6415df00 msgr2=0x7f2a640d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a68c8f640 1 --2- 172.21.15.27:0/2619999705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6415df00 0x7f2a640d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a68c8f640 1 -- 172.21.15.27:0/2619999705 >> v1:172.21.15.69:6789/0 conn(0x7f2a640b9a30 legacy=0x7f2a640d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a68c8f640 1 -- 172.21.15.27:0/2619999705 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a640cf0e0 con 0x7f2a640b95a0 2024-03-06T07:29:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.422+0000 7f2a69490640 1 --2- 172.21.15.27:0/2619999705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6415df00 0x7f2a640d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.423+0000 7f2a68c8f640 1 --2- 172.21.15.27:0/2619999705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a640b95a0 0x7f2a640b5700 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2a540097a0 tx=0x7f2a5402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2deda1dbe0b2b57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.423+0000 7f2a5bfff640 1 -- 172.21.15.27:0/2619999705 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54035020 con 0x7f2a640b95a0 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.423+0000 7f2a5bfff640 1 -- 172.21.15.27:0/2619999705 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a54031830 con 0x7f2a640b95a0 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.423+0000 7f2a5bfff640 1 -- 172.21.15.27:0/2619999705 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54031c00 con 0x7f2a640b95a0 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.424+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2619999705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a640b95a0 msgr2=0x7f2a640b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.424+0000 7f2a6b6f4640 1 --2- 172.21.15.27:0/2619999705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a640b95a0 0x7f2a640b5700 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2a540097a0 tx=0x7f2a5402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.424+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2619999705 shutdown_connections 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.424+0000 7f2a6b6f4640 1 --2- 172.21.15.27:0/2619999705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a640b95a0 0x7f2a640b5700 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.424+0000 7f2a6b6f4640 1 --2- 172.21.15.27:0/2619999705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a6415df00 0x7f2a640d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.424+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2619999705 >> 172.21.15.27:0/2619999705 conn(0x7f2a64071960 msgr2=0x7f2a64071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.424+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2619999705 shutdown_connections 2024-03-06T07:29:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.424+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2619999705 wait complete. 2024-03-06T07:29:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.425+0000 7f2a6b6f4640 1 Processor -- start 2024-03-06T07:29:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.425+0000 7f2a6b6f4640 1 -- start start 2024-03-06T07:29:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a6b6f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a640b95a0 0x7f2a640b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a6b6f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6415df00 0x7f2a64160200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a6b6f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c77b0 0x7f2a64150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a6b6f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a640cec70 con 0x7f2a640c77b0 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a6b6f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a640ce750 con 0x7f2a6415df00 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a6b6f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a640cedf0 con 0x7f2a640b95a0 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a69c91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c77b0 0x7f2a64150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a69c91640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c77b0 0x7f2a64150ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33630/0 (socket says 172.21.15.27:33630) 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a69c91640 1 -- 172.21.15.27:0/2057675359 learned_addr learned my addr 172.21.15.27:0/2057675359 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a69490640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a640b95a0 0x7f2a640b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a68c8f640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6415df00 0x7f2a64160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a69c91640 1 -- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a640b95a0 msgr2=0x7f2a640b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a69c91640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a640b95a0 0x7f2a640b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a69c91640 1 -- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6415df00 msgr2=0x7f2a64160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a69c91640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6415df00 0x7f2a64160200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.426+0000 7f2a69c91640 1 -- 172.21.15.27:0/2057675359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a640c7f70 con 0x7f2a640c77b0 2024-03-06T07:29:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.427+0000 7f2a69c91640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c77b0 0x7f2a64150ec0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00c730 tx=0x7f2a5c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.427+0000 7f2a59ffb640 1 -- 172.21.15.27:0/2057675359 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c003b70 con 0x7f2a640c77b0 2024-03-06T07:29:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.427+0000 7f2a59ffb640 1 -- 172.21.15.27:0/2057675359 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5c003d10 con 0x7f2a640c77b0 2024-03-06T07:29:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.427+0000 7f2a59ffb640 1 -- 172.21.15.27:0/2057675359 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c00aa90 con 0x7f2a640c77b0 2024-03-06T07:29:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.427+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a640c8250 con 0x7f2a640c77b0 2024-03-06T07:29:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.427+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a640c8810 con 0x7f2a640c77b0 2024-03-06T07:29:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.428+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a64067ff0 con 0x7f2a640c77b0 2024-03-06T07:29:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.428+0000 7f2a59ffb640 1 -- 172.21.15.27:0/2057675359 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a5c00ac30 con 0x7f2a640c77b0 2024-03-06T07:29:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.429+0000 7f2a59ffb640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a4806b4d0 0x7f2a4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.429+0000 7f2a59ffb640 1 -- 172.21.15.27:0/2057675359 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f2a5c08a130 con 0x7f2a640c77b0 2024-03-06T07:29:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.429+0000 7f2a69490640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a4806b4d0 0x7f2a4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.430+0000 7f2a69490640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a4806b4d0 0x7f2a4806d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2a60006ca0 tx=0x7f2a60009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.438+0000 7f2a59ffb640 1 -- 172.21.15.27:0/2057675359 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a5c058810 con 0x7f2a640c77b0 2024-03-06T07:29:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.656+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a640688b0 con 0x7f2a640c77b0 2024-03-06T07:29:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.656+0000 7f2a59ffb640 1 -- 172.21.15.27:0/2057675359 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f2a5c05c8b0 con 0x7f2a640c77b0 2024-03-06T07:29:07.658 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:07.658 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:07.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a4806b4d0 msgr2=0x7f2a4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a4806b4d0 0x7f2a4806d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2a60006ca0 tx=0x7f2a60009240 comp rx=0 tx=0).stop 2024-03-06T07:29:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c77b0 msgr2=0x7f2a64150ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c77b0 0x7f2a64150ec0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00c730 tx=0x7f2a5c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 shutdown_connections 2024-03-06T07:29:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2a4806b4d0 0x7f2a4806d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c77b0 0x7f2a64150ec0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6415df00 0x7f2a64160200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 --2- 172.21.15.27:0/2057675359 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a640b95a0 0x7f2a640b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.658+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 >> 172.21.15.27:0/2057675359 conn(0x7f2a64071960 msgr2=0x7f2a6415bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.659+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 shutdown_connections 2024-03-06T07:29:07.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:07.659+0000 7f2a6b6f4640 1 -- 172.21.15.27:0/2057675359 wait complete. 2024-03-06T07:29:07.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:29:07.794 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:29:07.794 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:29:19.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:19.151 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:37.796 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:29:37.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebda67f40 1 Processor -- start 2024-03-06T07:29:37.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebda67f40 1 -- start start 2024-03-06T07:29:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebda67f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d19565b2e0 0x55d195731a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebda67f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d195731010 0x55d195712550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebda67f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 0x55d19569cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebda67f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1956a1d20 con 0x55d19565b2e0 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebda67f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1955ffb70 con 0x55d1957313f0 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebda67f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1956a1fc0 con 0x55d195731010 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebb568640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d19565b2e0 0x55d195731a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebb568640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d19565b2e0 0x55d195731a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35466/0 (socket says 172.21.15.27:35466) 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.837+0000 7f7ebb568640 1 -- 172.21.15.27:0/1242379688 learned_addr learned my addr 172.21.15.27:0/1242379688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebad67640 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d195731010 0x55d195712550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 0x55d19569cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebb568640 1 -- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d195731010 msgr2=0x55d195712550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebb568640 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d195731010 0x55d195712550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebb568640 1 -- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 msgr2=0x55d19569cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebb568640 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 0x55d19569cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebb568640 1 -- 172.21.15.27:0/1242379688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d195688230 con 0x55d19565b2e0 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebad67640 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d195731010 0x55d195712550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 0x55d19569cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebb568640 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d19565b2e0 0x55d195731a60 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009790 tx=0x7f7eac009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe2be4e1eee4516 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7eb9d65640 1 -- 172.21.15.27:0/1242379688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac017070 con 0x55d19565b2e0 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7eb9d65640 1 -- 172.21.15.27:0/1242379688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac0090d0 con 0x55d19565b2e0 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7eb9d65640 1 -- 172.21.15.27:0/1242379688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac006b60 con 0x55d19565b2e0 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebda67f40 1 -- 172.21.15.27:0/1242379688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d19565b2e0 msgr2=0x55d195731a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d19565b2e0 0x55d195731a60 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009790 tx=0x7f7eac009c50 comp rx=0 tx=0).stop 2024-03-06T07:29:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebda67f40 1 -- 172.21.15.27:0/1242379688 shutdown_connections 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 0x55d19569cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d195731010 0x55d195712550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/1242379688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d19565b2e0 0x55d195731a60 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.838+0000 7f7ebda67f40 1 -- 172.21.15.27:0/1242379688 >> 172.21.15.27:0/1242379688 conn(0x55d1956585d0 msgr2=0x55d19569f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.839+0000 7f7ebda67f40 1 -- 172.21.15.27:0/1242379688 shutdown_connections 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.839+0000 7f7ebda67f40 1 -- 172.21.15.27:0/1242379688 wait complete. 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.839+0000 7f7ebda67f40 1 Processor -- start 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.839+0000 7f7ebda67f40 1 -- start start 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebda67f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d19565b2e0 0x55d195683370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebda67f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d195731010 0x55d195710e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebda67f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 0x55d19572ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebda67f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d195715c20 con 0x55d195731010 2024-03-06T07:29:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebda67f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19570e800 con 0x55d1957313f0 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebda67f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d195712a90 con 0x55d19565b2e0 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebad67640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d195731010 0x55d195710e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebad67640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d195731010 0x55d195710e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35478/0 (socket says 172.21.15.27:35478) 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 learned_addr learned my addr 172.21.15.27:0/3568641617 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d19565b2e0 0x55d195683370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d19565b2e0 msgr2=0x55d195683370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebad67640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d19565b2e0 0x55d195683370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 0x55d19572ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 msgr2=0x55d19572ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebad67640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 0x55d19572ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d195687a30 con 0x55d195731010 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55d19565b2e0 0x55d195683370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1957313f0 0x55d19572ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.840+0000 7f7ebad67640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d195731010 0x55d195710e10 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4000c00 tx=0x7f7ea4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.841+0000 7f7eb8d63640 1 -- 172.21.15.27:0/3568641617 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4065020 con 0x55d195731010 2024-03-06T07:29:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.841+0000 7f7eb8d63640 1 -- 172.21.15.27:0/3568641617 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea404ddb0 con 0x55d195731010 2024-03-06T07:29:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.841+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d195687d10 con 0x55d195731010 2024-03-06T07:29:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.841+0000 7f7eb8d63640 1 -- 172.21.15.27:0/3568641617 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea405a040 con 0x55d195731010 2024-03-06T07:29:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.841+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d19572a2e0 con 0x55d195731010 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.842+0000 7f7eb8d63640 1 -- 172.21.15.27:0/3568641617 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ea4068020 con 0x55d195731010 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.842+0000 7f7eb8d63640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e9c069eb0 0x7f7e9c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.842+0000 7f7eb8d63640 1 -- 172.21.15.27:0/3568641617 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7ea40cd330 con 0x55d195731010 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.843+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55d1956a0040 0x55d1957ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.843+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.843+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e9c069eb0 0x7f7e9c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.843+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55d1956a0040 0x55d1957ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.843+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e9c069eb0 0x7f7e9c06c370 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x55d195715bf0 tx=0x7f7eac00f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.843+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55d1956a0040 0x55d1957ca570 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.844+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 1 ==== osd_op_reply(1 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x55d1956a0040 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.844+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7eb400f820 0x7f7eb400e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.844+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb400f820 2024-03-06T07:29:37.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.844+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7eb400f820 0x7f7eb400e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.844+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7eb400f820 0x7f7eb400e2a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.845+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.1 v2:172.21.15.27:6809/746548704 1 ==== osd_op_reply(2 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb400f820 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.845+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f7eac012410 0x7f7eac012850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.845+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac012410 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.845+0000 7f7ebad67640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f7eac012410 0x7f7eac012850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.845+0000 7f7ebad67640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f7eac012410 0x7f7eac012850 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.845+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.3 v2:172.21.15.27:6824/4277055095 1 ==== osd_op_reply(3 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac012410 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.845+0000 7f7ebad67640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f7ea406a020 0x7f7ea409a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.845+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.846+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f7ea406a020 0x7f7ea409a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.846+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f7ea406a020 0x7f7ea409a910 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.846+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 1 ==== osd_op_reply(4 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x7f7ea406a020 2024-03-06T07:29:37.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.846+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.847+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 2 ==== osd_op_reply(5 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x55d1956a0040 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.847+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac012410 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.847+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.3 v2:172.21.15.27:6824/4277055095 2 ==== osd_op_reply(6 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac012410 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.847+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.847+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 2 ==== osd_op_reply(7 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x7f7ea406a020 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.847+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.848+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 3 ==== osd_op_reply(8 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x55d1956a0040 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.848+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.848+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 3 ==== osd_op_reply(9 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x7f7ea406a020 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.848+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac012410 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.848+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.3 v2:172.21.15.27:6824/4277055095 3 ==== osd_op_reply(10 [pgnls start_epoch 120 out=49b] v114'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac012410 2024-03-06T07:29:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.848+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac012410 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.849+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.3 v2:172.21.15.27:6824/4277055095 4 ==== osd_op_reply(11 [pgnls start_epoch 120 out=49b] v114'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac012410 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.849+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.849+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 4 ==== osd_op_reply(12 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x7f7ea406a020 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.849+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.849+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 4 ==== osd_op_reply(13 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x55d1956a0040 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.849+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f7eb4019970 0x7f7eb4017440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.849+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.850+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f7eb4019970 0x7f7eb4017440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.850+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f7eb4019970 0x7f7eb4017440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.851+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 1 ==== osd_op_reply(14 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.851+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb400f820 2024-03-06T07:29:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.851+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.1 v2:172.21.15.27:6809/746548704 2 ==== osd_op_reply(15 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb400f820 2024-03-06T07:29:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.851+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.851+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 5 ==== osd_op_reply(16 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x7f7ea406a020 2024-03-06T07:29:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.851+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.852+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 2 ==== osd_op_reply(17 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.852+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7f7eac02e920 0x7f7eac01c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.852+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac02e920 2024-03-06T07:29:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.852+0000 7f7ebad67640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7f7eac02e920 0x7f7eac01c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.852+0000 7f7ebad67640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7f7eac02e920 0x7f7eac01c400 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.853+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.5 v2:172.21.15.69:6819/715204514 1 ==== osd_op_reply(18 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac02e920 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.853+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac012410 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.853+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.3 v2:172.21.15.27:6824/4277055095 5 ==== osd_op_reply(19 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac012410 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.853+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.853+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 5 ==== osd_op_reply(20 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x55d1956a0040 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.853+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.854+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 6 ==== osd_op_reply(21 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x55d1956a0040 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.854+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac012410 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.854+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.3 v2:172.21.15.27:6824/4277055095 6 ==== osd_op_reply(22 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac012410 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.854+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.854+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 6 ==== osd_op_reply(23 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb400c640 con 0x7f7ea406a020 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.854+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb400f820 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.855+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.1 v2:172.21.15.27:6809/746548704 3 ==== osd_op_reply(24 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb400f820 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.855+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f7eac0073f0 0x7f7eac007870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.855+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac0073f0 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.855+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f7eac0073f0 0x7f7eac007870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.855+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f7eac0073f0 0x7f7eac007870 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.856+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.6 v2:172.21.15.69:6800/288062332 1 ==== osd_op_reply(25 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb4036040 con 0x7f7eac0073f0 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.856+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.856+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 7 ==== osd_op_reply(26 [pgnls start_epoch 120 out=49b] v114'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb4036040 con 0x7f7ea406a020 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.856+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac02e920 2024-03-06T07:29:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.857+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.5 v2:172.21.15.69:6819/715204514 2 ==== osd_op_reply(27 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac02e920 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.857+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.857+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 8 ==== osd_op_reply(28 [pgnls start_epoch 120 out=49b] v114'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb4036040 con 0x7f7ea406a020 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.857+0000 7f7ebbd69640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f7eb4039040 0x7f7eb402a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.857+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4039040 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.857+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f7eb4039040 0x7f7eb402a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.857+0000 7f7ebb568640 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f7eb4039040 0x7f7eb402a940 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.858+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.2 v2:172.21.15.27:6808/3892975520 1 ==== osd_op_reply(29 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4039040 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.858+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac0073f0 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.858+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.6 v2:172.21.15.69:6800/288062332 2 ==== osd_op_reply(30 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb4036040 con 0x7f7eac0073f0 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.858+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb400f820 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.858+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.1 v2:172.21.15.27:6809/746548704 4 ==== osd_op_reply(31 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb400f820 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.858+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.859+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 3 ==== osd_op_reply(32 [pgnls start_epoch 120 out=49b] v114'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.859+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.859+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 9 ==== osd_op_reply(33 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb4036040 con 0x7f7ea406a020 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.859+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.859+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 10 ==== osd_op_reply(34 [pgnls start_epoch 120 out=49b] v114'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb4036040 con 0x7f7ea406a020 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.859+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac012410 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.860+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.3 v2:172.21.15.27:6824/4277055095 7 ==== osd_op_reply(35 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac012410 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.860+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb400f820 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.860+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.1 v2:172.21.15.27:6809/746548704 5 ==== osd_op_reply(36 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb400f820 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.860+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac012410 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.860+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.3 v2:172.21.15.27:6824/4277055095 8 ==== osd_op_reply(37 [pgnls start_epoch 120 out=49b] v114'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac012410 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.860+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.861+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 4 ==== osd_op_reply(38 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.861+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac02e920 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.861+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.5 v2:172.21.15.69:6819/715204514 3 ==== osd_op_reply(39 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac02e920 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.861+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac02e920 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.861+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.5 v2:172.21.15.69:6819/715204514 4 ==== osd_op_reply(40 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac02e920 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.861+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac0073f0 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.862+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.6 v2:172.21.15.69:6800/288062332 3 ==== osd_op_reply(41 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb4036040 con 0x7f7eac0073f0 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.862+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb400f820 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.862+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.1 v2:172.21.15.27:6809/746548704 6 ==== osd_op_reply(42 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb400f820 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.862+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac02e920 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.862+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.5 v2:172.21.15.69:6819/715204514 5 ==== osd_op_reply(43 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac02e920 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.863+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.863+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 5 ==== osd_op_reply(44 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.863+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.863+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 7 ==== osd_op_reply(45 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb4036040 con 0x55d1956a0040 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.863+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.864+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 8 ==== osd_op_reply(46 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb4036040 con 0x55d1956a0040 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.864+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.864+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 9 ==== osd_op_reply(47 [pgnls start_epoch 120 out=49b] v114'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb404a040 con 0x55d1956a0040 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.864+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.864+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 10 ==== osd_op_reply(48 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb404a040 con 0x55d1956a0040 2024-03-06T07:29:37.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.864+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.864+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 6 ==== osd_op_reply(49 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.864+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.865+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 7 ==== osd_op_reply(50 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.865+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac012410 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.865+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.3 v2:172.21.15.27:6824/4277055095 9 ==== osd_op_reply(51 [pgnls start_epoch 120 out=49b] v114'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac012410 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.865+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x55d1956a0040 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.865+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.7 v2:172.21.15.69:6816/1392567864 11 ==== osd_op_reply(52 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb404a040 con 0x55d1956a0040 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.865+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.866+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 11 ==== osd_op_reply(53 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb404a040 con 0x7f7ea406a020 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.866+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.866+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 8 ==== osd_op_reply(54 [pgnls start_epoch 120 out=49b] v114'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.866+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.866+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 9 ==== osd_op_reply(55 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.866+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4039040 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.867+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.2 v2:172.21.15.27:6808/3892975520 2 ==== osd_op_reply(56 [pgnls start_epoch 120 out=49b] v114'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4039040 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.867+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.867+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 12 ==== osd_op_reply(57 [pgnls start_epoch 120 out=81b] v114'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f7eb404a040 con 0x7f7ea406a020 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.867+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.867+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 10 ==== osd_op_reply(58 [pgnls start_epoch 120 out=49b] v114'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.867+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac0073f0 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.868+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.6 v2:172.21.15.69:6800/288062332 4 ==== osd_op_reply(59 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb404a040 con 0x7f7eac0073f0 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.868+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7ea406a020 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.868+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.0 v2:172.21.15.27:6800/3926946944 13 ==== osd_op_reply(60 [pgnls start_epoch 120 out=49b] v114'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb404a040 con 0x7f7ea406a020 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.868+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb400f820 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.868+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.1 v2:172.21.15.27:6809/746548704 7 ==== osd_op_reply(61 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb400f820 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.868+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eb4019970 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.869+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 <== osd.4 v2:172.21.15.69:6808/1626703870 11 ==== osd_op_reply(62 [pgnls start_epoch 120 out=81b] v114'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f7eac01a070 con 0x7f7eb4019970 2024-03-06T07:29:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.869+0000 7f7ebb568640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac0073f0 2024-03-06T07:29:37.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.869+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 <== osd.6 v2:172.21.15.69:6800/288062332 5 ==== osd_op_reply(63 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7eb404a040 con 0x7f7eac0073f0 2024-03-06T07:29:37.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.869+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55d1956838b0 con 0x7f7eac02e920 2024-03-06T07:29:37.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.869+0000 7f7ebad67640 1 -- 172.21.15.27:0/3568641617 <== osd.5 v2:172.21.15.69:6819/715204514 6 ==== osd_op_reply(64 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7ea405d040 con 0x7f7eac02e920 2024-03-06T07:29:37.877 INFO:teuthology.orchestra.run.smithi027.stdout:10000000001.00000000 2024-03-06T07:29:37.877 INFO:teuthology.orchestra.run.smithi027.stdout:10000000001.00000001 2024-03-06T07:29:37.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f7ea406a020 msgr2=0x7f7ea409a910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f7ea406a020 0x7f7ea409a910 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7eb400f820 msgr2=0x7f7eb400e2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7eb400f820 0x7f7eb400e2a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f7eb4039040 msgr2=0x7f7eb402a940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7f7eb4039040 0x7f7eb402a940 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f7eac012410 msgr2=0x7f7eac012850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 reap_dead start 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f7eac012410 0x7f7eac012850 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f7eb4019970 msgr2=0x7f7eb4017440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f7eb4019970 0x7f7eb4017440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7f7eac02e920 msgr2=0x7f7eac01c400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7f7eac02e920 0x7f7eac01c400 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f7eac0073f0 msgr2=0x7f7eac007870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f7eac0073f0 0x7f7eac007870 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55d1956a0040 msgr2=0x55d1957ca570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55d1956a0040 0x55d1957ca570 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebbd69640 1 -- 172.21.15.27:0/3568641617 reap_dead start 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e9c069eb0 msgr2=0x7f7e9c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e9c069eb0 0x7f7e9c06c370 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x55d195715bf0 tx=0x7f7eac00f000 comp rx=0 tx=0).stop 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d195731010 msgr2=0x55d195710e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d195731010 0x55d195710e10 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4000c00 tx=0x7f7ea4002fb0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 shutdown_connections 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e9c069eb0 0x7f7e9c06c370 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 --2- 172.21.15.27:0/3568641617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d195731010 0x55d195710e10 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 >> 172.21.15.27:0/3568641617 conn(0x55d1956585d0 msgr2=0x55d19570de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 shutdown_connections 2024-03-06T07:29:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:37.870+0000 7f7ebda67f40 1 -- 172.21.15.27:0/3568641617 wait complete. 2024-03-06T07:29:37.881 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:29:37.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:38.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.029+0000 7f1f308f4640 1 Processor -- start 2024-03-06T07:29:38.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.029+0000 7f1f308f4640 1 -- start start 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f308f4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f2c0aee10 0x7f1f2c15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f308f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f308f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c0729b0 con 0x7f1f2c15bac0 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f308f4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c06cf70 con 0x7f1f2c0aee10 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f308f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c06c920 con 0x7f1f2c06d400 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f2ad9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f29d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f2c0aee10 0x7f1f2c15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f2ad9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:34212/0 (socket says 172.21.15.27:34212) 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f2a59c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1f2c15bac0 0x7f1f2c0ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:55550/0 (socket says 172.21.15.27:55550) 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f2ad9d640 1 -- 172.21.15.27:0/2082823330 learned_addr learned my addr 172.21.15.27:0/2082823330 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.030+0000 7f1f29d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f2c0aee10 0x7f1f2c15cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:48166/0 (socket says 172.21.15.27:48166) 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.031+0000 7f1f2ad9d640 1 -- 172.21.15.27:0/2082823330 >> v2:172.21.15.69:3300/0 conn(0x7f1f2c0aee10 msgr2=0x7f1f2c15cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.031+0000 7f1f2ad9d640 1 --2- 172.21.15.27:0/2082823330 >> v2:172.21.15.69:3300/0 conn(0x7f1f2c0aee10 0x7f1f2c15cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.031+0000 7f1f2ad9d640 1 -- 172.21.15.27:0/2082823330 >> v1:172.21.15.69:6789/0 conn(0x7f1f2c15bac0 legacy=0x7f1f2c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.031+0000 7f1f2ad9d640 1 -- 172.21.15.27:0/2082823330 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c06cc00 con 0x7f1f2c06d400 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.031+0000 7f1f29d9b640 1 --2- 172.21.15.27:0/2082823330 >> v2:172.21.15.69:3300/0 conn(0x7f1f2c0aee10 0x7f1f2c15cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f2ad9d640 1 --2- 172.21.15.27:0/2082823330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c070870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009e50 tx=0x7f1f2402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca17e7bf268319ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f2959a640 1 -- 172.21.15.27:0/2082823330 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2402cb00 con 0x7f1f2c06d400 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f2959a640 1 -- 172.21.15.27:0/2082823330 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2402cca0 con 0x7f1f2c06d400 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f2959a640 1 -- 172.21.15.27:0/2082823330 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24034610 con 0x7f1f2c06d400 2024-03-06T07:29:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f308f4640 1 -- 172.21.15.27:0/2082823330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 msgr2=0x7f1f2c070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f308f4640 1 --2- 172.21.15.27:0/2082823330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c070870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009e50 tx=0x7f1f2402bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f308f4640 1 -- 172.21.15.27:0/2082823330 shutdown_connections 2024-03-06T07:29:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f308f4640 1 --2- 172.21.15.27:0/2082823330 >> v2:172.21.15.69:3300/0 conn(0x7f1f2c0aee10 0x7f1f2c15cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f308f4640 1 --2- 172.21.15.27:0/2082823330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c070870 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.032+0000 7f1f308f4640 1 -- 172.21.15.27:0/2082823330 >> 172.21.15.27:0/2082823330 conn(0x7f1f2c066d00 msgr2=0x7f1f2c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.033+0000 7f1f308f4640 1 -- 172.21.15.27:0/2082823330 shutdown_connections 2024-03-06T07:29:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.033+0000 7f1f308f4640 1 -- 172.21.15.27:0/2082823330 wait complete. 2024-03-06T07:29:38.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.034+0000 7f1f308f4640 1 Processor -- start 2024-03-06T07:29:38.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.034+0000 7f1f308f4640 1 -- start start 2024-03-06T07:29:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f308f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f308f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c0aee10 0x7f1f2c13fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f308f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f2c15bac0 0x7f1f2c1e42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f308f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c15c020 con 0x7f1f2c0aee10 2024-03-06T07:29:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f308f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c15bea0 con 0x7f1f2c15bac0 2024-03-06T07:29:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f308f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c072580 con 0x7f1f2c06d400 2024-03-06T07:29:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f29d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c0aee10 0x7f1f2c13fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f29d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c0aee10 0x7f1f2c13fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35488/0 (socket says 172.21.15.27:35488) 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f29d9b640 1 -- 172.21.15.27:0/12674846 learned_addr learned my addr 172.21.15.27:0/12674846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f2a59c640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f2ad9d640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f2c15bac0 0x7f1f2c1e42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f29d9b640 1 -- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 msgr2=0x7f1f2c158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f29d9b640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f29d9b640 1 -- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f2c15bac0 msgr2=0x7f1f2c1e42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f29d9b640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f2c15bac0 0x7f1f2c1e42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f29d9b640 1 -- 172.21.15.27:0/12674846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c140630 con 0x7f1f2c0aee10 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.035+0000 7f1f29d9b640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c0aee10 0x7f1f2c13fe70 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000ebe0 tx=0x7f1f2000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.036+0000 7f1f1affd640 1 -- 172.21.15.27:0/12674846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20017070 con 0x7f1f2c0aee10 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.036+0000 7f1f1affd640 1 -- 172.21.15.27:0/12674846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2000cc30 con 0x7f1f2c0aee10 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.036+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c140910 con 0x7f1f2c0aee10 2024-03-06T07:29:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.036+0000 7f1f1affd640 1 -- 172.21.15.27:0/12674846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20013470 con 0x7f1f2c0aee10 2024-03-06T07:29:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.036+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c140ed0 con 0x7f1f2c0aee10 2024-03-06T07:29:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.037+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c067ff0 con 0x7f1f2c0aee10 2024-03-06T07:29:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.037+0000 7f1f1affd640 1 -- 172.21.15.27:0/12674846 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f2001a070 con 0x7f1f2c0aee10 2024-03-06T07:29:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.038+0000 7f1f1affd640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0c06b4d0 0x7f1f0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.038+0000 7f1f1affd640 1 -- 172.21.15.27:0/12674846 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1f2008a530 con 0x7f1f2c0aee10 2024-03-06T07:29:38.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.038+0000 7f1f2a59c640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0c06b4d0 0x7f1f0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.038+0000 7f1f2a59c640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0c06b4d0 0x7f1f0c06d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c000f60 tx=0x7f1f1c007e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.042+0000 7f1f1affd640 1 -- 172.21.15.27:0/12674846 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f20058c70 con 0x7f1f2c0aee10 2024-03-06T07:29:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.262+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f2c0cfb80 con 0x7f1f2c0aee10 2024-03-06T07:29:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.262+0000 7f1f1affd640 1 -- 172.21.15.27:0/12674846 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f1f2005cd10 con 0x7f1f2c0aee10 2024-03-06T07:29:38.265 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:38.265 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0c06b4d0 msgr2=0x7f1f0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0c06b4d0 0x7f1f0c06d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c000f60 tx=0x7f1f1c007e20 comp rx=0 tx=0).stop 2024-03-06T07:29:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c0aee10 msgr2=0x7f1f2c13fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c0aee10 0x7f1f2c13fe70 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000ebe0 tx=0x7f1f2000c520 comp rx=0 tx=0).stop 2024-03-06T07:29:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 shutdown_connections 2024-03-06T07:29:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0c06b4d0 0x7f1f0c06d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f2c15bac0 0x7f1f2c1e42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c0aee10 0x7f1f2c13fe70 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 --2- 172.21.15.27:0/12674846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f2c06d400 0x7f1f2c158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 >> 172.21.15.27:0/12674846 conn(0x7f1f2c066d00 msgr2=0x7f1f2c15e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 shutdown_connections 2024-03-06T07:29:38.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.264+0000 7f1f308f4640 1 -- 172.21.15.27:0/12674846 wait complete. 2024-03-06T07:29:38.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:38.403 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:29:38.403 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:29:38.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.507+0000 7fecffb20640 1 Processor -- start 2024-03-06T07:29:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.508+0000 7fecffb20640 1 -- start start 2024-03-06T07:29:38.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.508+0000 7fecffb20640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fecf80b2b50 0x7fecf80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.508+0000 7fecffb20640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf80ce610 con 0x7fecf80d3400 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.508+0000 7fecffb20640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf80d2920 con 0x7fecf8071540 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.508+0000 7fecffb20640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf80d2f70 con 0x7fecf80b2b50 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfd0bb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fecf80d3400 0x7fecf80b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51916/0 (socket says 172.21.15.27:51916) 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfd0bb640 1 -- 172.21.15.27:0/1404111849 learned_addr learned my addr 172.21.15.27:0/1404111849 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfe0bd640 1 --2- 172.21.15.27:0/1404111849 >> v2:172.21.15.69:3300/0 conn(0x7fecf80b2b50 0x7fecf80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfc8ba640 1 -- 172.21.15.27:0/1404111849 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134788190 0 0) 0x7fecf80ce610 con 0x7fecf80d3400 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfc8ba640 1 -- 172.21.15.27:0/1404111849 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece4003520 con 0x7fecf80d3400 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfc8ba640 1 -- 172.21.15.27:0/1404111849 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663070337 0 0) 0x7fecf80d2920 con 0x7fecf8071540 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfc8ba640 1 -- 172.21.15.27:0/1404111849 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf80ce610 con 0x7fecf8071540 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfe0bd640 1 -- 172.21.15.27:0/1404111849 >> v1:172.21.15.69:6789/0 conn(0x7fecf8071540 legacy=0x7fecf8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfc8ba640 1 -- 172.21.15.27:0/1404111849 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1390397552 0 0) 0x7fece4003520 con 0x7fecf80d3400 2024-03-06T07:29:38.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfe0bd640 1 -- 172.21.15.27:0/1404111849 >> v1:172.21.15.27:6789/0 conn(0x7fecf80d3400 legacy=0x7fecf80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.509+0000 7fecfe0bd640 1 -- 172.21.15.27:0/1404111849 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf80d2c00 con 0x7fecf80b2b50 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.510+0000 7fecfe0bd640 1 --2- 172.21.15.27:0/1404111849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf80b2b50 0x7fecf80b30a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009d90 tx=0x7fecf002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c049f2756c99316 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecfc8ba640 1 -- 172.21.15.27:0/1404111849 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0039070 con 0x7fecf80b2b50 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecfc8ba640 1 -- 172.21.15.27:0/1404111849 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf002c940 con 0x7fecf80b2b50 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecfc8ba640 1 -- 172.21.15.27:0/1404111849 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4001670 con 0x7fecf80b2b50 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecffb20640 1 -- 172.21.15.27:0/1404111849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf80b2b50 msgr2=0x7fecf80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecffb20640 1 --2- 172.21.15.27:0/1404111849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf80b2b50 0x7fecf80b30a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009d90 tx=0x7fecf002be10 comp rx=0 tx=0).stop 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecffb20640 1 -- 172.21.15.27:0/1404111849 shutdown_connections 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecffb20640 1 --2- 172.21.15.27:0/1404111849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf80b2b50 0x7fecf80b30a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecffb20640 1 -- 172.21.15.27:0/1404111849 >> 172.21.15.27:0/1404111849 conn(0x7fecf80d3ca0 msgr2=0x7fecf80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecffb20640 1 -- 172.21.15.27:0/1404111849 shutdown_connections 2024-03-06T07:29:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.511+0000 7fecffb20640 1 -- 172.21.15.27:0/1404111849 wait complete. 2024-03-06T07:29:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.512+0000 7fecffb20640 1 Processor -- start 2024-03-06T07:29:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.512+0000 7fecffb20640 1 -- start start 2024-03-06T07:29:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecffb20640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8071540 0x7fecf814c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecffb20640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80b2b50 0x7fecf8149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecffb20640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf80d3400 0x7fecf814a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecffb20640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf815e370 con 0x7fecf8071540 2024-03-06T07:29:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecffb20640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf80d4be0 con 0x7fecf80d3400 2024-03-06T07:29:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecffb20640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf814b2a0 con 0x7fecf80b2b50 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecfd8bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8071540 0x7fecf814c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecfd8bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8071540 0x7fecf814c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35504/0 (socket says 172.21.15.27:35504) 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecfd8bc640 1 -- 172.21.15.27:0/3854073608 learned_addr learned my addr 172.21.15.27:0/3854073608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecfd0bb640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80b2b50 0x7fecf8149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.513+0000 7fecfe0bd640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf80d3400 0x7fecf814a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecfd8bc640 1 -- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80b2b50 msgr2=0x7fecf8149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecfd8bc640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80b2b50 0x7fecf8149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecfd8bc640 1 -- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf80d3400 msgr2=0x7fecf814a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecfd8bc640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf80d3400 0x7fecf814a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecfd8bc640 1 -- 172.21.15.27:0/3854073608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf814b580 con 0x7fecf8071540 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecfd8bc640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8071540 0x7fecf814c340 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fece800c4c0 tx=0x7fece800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecee7fc640 1 -- 172.21.15.27:0/3854073608 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece80052d0 con 0x7fecf8071540 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecee7fc640 1 -- 172.21.15.27:0/3854073608 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece8005470 con 0x7fecf8071540 2024-03-06T07:29:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf814b860 con 0x7fecf8071540 2024-03-06T07:29:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecee7fc640 1 -- 172.21.15.27:0/3854073608 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8012870 con 0x7fecf8071540 2024-03-06T07:29:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.514+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf814be20 con 0x7fecf8071540 2024-03-06T07:29:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.515+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf8068700 con 0x7fecf8071540 2024-03-06T07:29:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.516+0000 7fecee7fc640 1 -- 172.21.15.27:0/3854073608 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fece8012a10 con 0x7fecf8071540 2024-03-06T07:29:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.516+0000 7fecee7fc640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecdc06b4d0 0x7fecdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.516+0000 7fecee7fc640 1 -- 172.21.15.27:0/3854073608 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fece8089ee0 con 0x7fecf8071540 2024-03-06T07:29:38.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.516+0000 7fecfd0bb640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecdc06b4d0 0x7fecdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.517+0000 7fecfd0bb640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecdc06b4d0 0x7fecdc06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fecf80b5dd0 tx=0x7fecf40016b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.520+0000 7fecee7fc640 1 -- 172.21.15.27:0/3854073608 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fece80585a0 con 0x7fecf8071540 2024-03-06T07:29:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.741+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecf81611e0 con 0x7fecf8071540 2024-03-06T07:29:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.742+0000 7fecee7fc640 1 -- 172.21.15.27:0/3854073608 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fece805c640 con 0x7fecf8071540 2024-03-06T07:29:38.744 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:38.745 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecdc06b4d0 msgr2=0x7fecdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecdc06b4d0 0x7fecdc06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fecf80b5dd0 tx=0x7fecf40016b0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8071540 msgr2=0x7fecf814c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8071540 0x7fecf814c340 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fece800c4c0 tx=0x7fece800c980 comp rx=0 tx=0).stop 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 shutdown_connections 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fecdc06b4d0 0x7fecdc06d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecf80d3400 0x7fecf814a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fecf80b2b50 0x7fecf8149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 --2- 172.21.15.27:0/3854073608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf8071540 0x7fecf814c340 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 >> 172.21.15.27:0/3854073608 conn(0x7fecf80d3ca0 msgr2=0x7fecf81603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 shutdown_connections 2024-03-06T07:29:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.744+0000 7fecffb20640 1 -- 172.21.15.27:0/3854073608 wait complete. 2024-03-06T07:29:38.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:29:38.890 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:29:38.891 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:38.891 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b36b8f40 1 Processor -- start 2024-03-06T07:29:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b36b8f40 1 -- start start 2024-03-06T07:29:38.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b36b8f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0b72913c0 0x55a0b733bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b36b8f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0b7325df0 0x55a0b728dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b36b8f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b73261d0 0x55a0b7337e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b36b8f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b72917d0 con 0x55a0b73261d0 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b36b8f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b7207b70 con 0x55a0b7325df0 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b36b8f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0b72aaa70 con 0x55a0b72913c0 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b19ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b73261d0 0x55a0b7337e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b19ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b73261d0 0x55a0b7337e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35530/0 (socket says 172.21.15.27:35530) 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.931+0000 7fb9b19ba640 1 -- 172.21.15.27:0/3302175750 learned_addr learned my addr 172.21.15.27:0/3302175750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9b19ba640 1 -- 172.21.15.27:0/3302175750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0b72913c0 msgr2=0x55a0b733bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9b19ba640 1 --2- 172.21.15.27:0/3302175750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0b72913c0 0x55a0b733bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9b19ba640 1 -- 172.21.15.27:0/3302175750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0b7325df0 msgr2=0x55a0b728dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9b19ba640 1 --2- 172.21.15.27:0/3302175750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0b7325df0 0x55a0b728dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9b19ba640 1 -- 172.21.15.27:0/3302175750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0b733f210 con 0x55a0b73261d0 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9b19ba640 1 --2- 172.21.15.27:0/3302175750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b73261d0 0x55a0b7337e20 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00a4b0 tx=0x7fb9ac00a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60550a3d5d109e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9a37fe640 1 -- 172.21.15.27:0/3302175750 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac00f7b0 con 0x55a0b73261d0 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9a37fe640 1 -- 172.21.15.27:0/3302175750 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ac015070 con 0x55a0b73261d0 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9a37fe640 1 -- 172.21.15.27:0/3302175750 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac00fb70 con 0x55a0b73261d0 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/3302175750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b73261d0 msgr2=0x55a0b7337e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.932+0000 7fb9b36b8f40 1 --2- 172.21.15.27:0/3302175750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b73261d0 0x55a0b7337e20 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00a4b0 tx=0x7fb9ac00a970 comp rx=0 tx=0).stop 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.933+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/3302175750 shutdown_connections 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.933+0000 7fb9b36b8f40 1 --2- 172.21.15.27:0/3302175750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b73261d0 0x55a0b7337e20 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.933+0000 7fb9b36b8f40 1 --2- 172.21.15.27:0/3302175750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0b7325df0 0x55a0b728dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.933+0000 7fb9b36b8f40 1 --2- 172.21.15.27:0/3302175750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0b72913c0 0x55a0b733bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.933+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/3302175750 >> 172.21.15.27:0/3302175750 conn(0x55a0b728e9c0 msgr2=0x55a0b7325050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.933+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/3302175750 shutdown_connections 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.933+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/3302175750 wait complete. 2024-03-06T07:29:38.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.933+0000 7fb9b36b8f40 1 Processor -- start 2024-03-06T07:29:38.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.933+0000 7fb9b36b8f40 1 -- start start 2024-03-06T07:29:38.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b36b8f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0b72913c0 0x55a0b7320b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b36b8f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b7325df0 0x55a0b731bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b36b8f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0b73261d0 0x55a0b731c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b36b8f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b728e1e0 con 0x55a0b7325df0 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b36b8f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b7264fb0 con 0x55a0b73261d0 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b36b8f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0b731c630 con 0x55a0b72913c0 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b09b8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b7325df0 0x55a0b731bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b09b8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b7325df0 0x55a0b731bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35532/0 (socket says 172.21.15.27:35532) 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b09b8640 1 -- 172.21.15.27:0/4272941773 learned_addr learned my addr 172.21.15.27:0/4272941773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b19ba640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0b73261d0 0x55a0b731c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b11b9640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0b72913c0 0x55a0b7320b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b09b8640 1 -- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0b72913c0 msgr2=0x55a0b7320b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b09b8640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0b72913c0 0x55a0b7320b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b09b8640 1 -- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0b73261d0 msgr2=0x55a0b731c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b09b8640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0b73261d0 0x55a0b731c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b09b8640 1 -- 172.21.15.27:0/4272941773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0b731d1c0 con 0x55a0b7325df0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b19ba640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0b73261d0 0x55a0b731c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b11b9640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a0b72913c0 0x55a0b7320b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.934+0000 7fb9b09b8640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b7325df0 0x55a0b731bbb0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb9a801e3f0 tx=0x7fb9a804e6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.935+0000 7fb9a27fc640 1 -- 172.21.15.27:0/4272941773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8060430 con 0x55a0b7325df0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.935+0000 7fb9a27fc640 1 -- 172.21.15.27:0/4272941773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a8069070 con 0x55a0b7325df0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.935+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a0b731d460 con 0x55a0b7325df0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.935+0000 7fb9a27fc640 1 -- 172.21.15.27:0/4272941773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8060810 con 0x55a0b7325df0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.935+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0b731dd00 con 0x55a0b7325df0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.936+0000 7fb9a27fc640 1 -- 172.21.15.27:0/4272941773 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9a80609b0 con 0x55a0b7325df0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.936+0000 7fb9a27fc640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb994069eb0 0x7fb99406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.936+0000 7fb9a27fc640 1 -- 172.21.15.27:0/4272941773 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb9a80ccfe0 con 0x55a0b7325df0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.937+0000 7fb9b36b8f40 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a0b7338ff0 0x55a0b73393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.937+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120) v9 -- 0x55a0b731fa20 con 0x55a0b7338ff0 2024-03-06T07:29:38.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.937+0000 7fb9b11b9640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb994069eb0 0x7fb99406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.937+0000 7fb9b19ba640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a0b7338ff0 0x55a0b73393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.937+0000 7fb9b19ba640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a0b7338ff0 0x55a0b73393d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.937+0000 7fb9b11b9640 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb994069eb0 0x7fb99406c370 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400d370 tx=0x7fb9a4001530 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.937+0000 7fb9b19ba640 1 -- 172.21.15.27:0/4272941773 <== osd.0 v2:172.21.15.27:6800/3926946944 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7fb9ac00e070 con 0x55a0b7338ff0 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a0b7338ff0 msgr2=0x55a0b73393d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a0b7338ff0 0x55a0b73393d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb994069eb0 msgr2=0x7fb99406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb994069eb0 0x7fb99406c370 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400d370 tx=0x7fb9a4001530 comp rx=0 tx=0).stop 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b7325df0 msgr2=0x55a0b731bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 --2- 172.21.15.27:0/4272941773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a0b7325df0 0x55a0b731bbb0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb9a801e3f0 tx=0x7fb9a804e6a0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b19ba640 1 -- 172.21.15.27:0/4272941773 reap_dead start 2024-03-06T07:29:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 shutdown_connections 2024-03-06T07:29:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 >> 172.21.15.27:0/4272941773 conn(0x55a0b728e9c0 msgr2=0x55a0b7264720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 shutdown_connections 2024-03-06T07:29:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:38.938+0000 7fb9b36b8f40 1 -- 172.21.15.27:0/4272941773 wait complete. 2024-03-06T07:29:38.945 INFO:teuthology.orchestra.run.smithi027.stdout:] 10000000001 stray0 2024-03-06T07:29:38.946 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511627777, 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1536, 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "10000000001", 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 11 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 256, 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "stray0", 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 5 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 17, 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [] 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:29:39.033 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:39.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:39.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.179+0000 7f6212cc9640 1 Processor -- start 2024-03-06T07:29:39.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.179+0000 7f6212cc9640 1 -- start start 2024-03-06T07:29:39.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6212cc9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f620c0b95a0 0x7f620c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6212cc9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620c0cea30 con 0x7f620c0b9a10 2024-03-06T07:29:39.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6212cc9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620c0cf3c0 con 0x7f620c15df00 2024-03-06T07:29:39.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6212cc9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620c0cf0e0 con 0x7f620c0b95a0 2024-03-06T07:29:39.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6203fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f620c0b95a0 0x7f620c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6203fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f620c0b95a0 0x7f620c0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35544/0 (socket says 172.21.15.27:35544) 2024-03-06T07:29:39.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6203fff640 1 -- 172.21.15.27:0/2940817192 learned_addr learned my addr 172.21.15.27:0/2940817192 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:39.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6203fff640 1 -- 172.21.15.27:0/2940817192 >> v1:172.21.15.69:6789/0 conn(0x7f620c15df00 legacy=0x7f620c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6203fff640 1 -- 172.21.15.27:0/2940817192 >> v1:172.21.15.27:6789/0 conn(0x7f620c0b9a10 legacy=0x7f620c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.180+0000 7f6203fff640 1 -- 172.21.15.27:0/2940817192 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c0ce470 con 0x7f620c0b95a0 2024-03-06T07:29:39.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.181+0000 7f6203fff640 1 --2- 172.21.15.27:0/2940817192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 0x7f620c0b5700 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f620c0cf590 tx=0x7f61fc02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da5aff91af24d28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.181+0000 7f62037fe640 1 -- 172.21.15.27:0/2940817192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc02f840 con 0x7f620c0b95a0 2024-03-06T07:29:39.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.181+0000 7f62037fe640 1 -- 172.21.15.27:0/2940817192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61fc02f9e0 con 0x7f620c0b95a0 2024-03-06T07:29:39.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.181+0000 7f62037fe640 1 -- 172.21.15.27:0/2940817192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc02fdb0 con 0x7f620c0b95a0 2024-03-06T07:29:39.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.181+0000 7f6212cc9640 1 -- 172.21.15.27:0/2940817192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 msgr2=0x7f620c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.181+0000 7f6212cc9640 1 --2- 172.21.15.27:0/2940817192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 0x7f620c0b5700 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f620c0cf590 tx=0x7f61fc02ba20 comp rx=0 tx=0).stop 2024-03-06T07:29:39.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.182+0000 7f6212cc9640 1 -- 172.21.15.27:0/2940817192 shutdown_connections 2024-03-06T07:29:39.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.182+0000 7f6212cc9640 1 --2- 172.21.15.27:0/2940817192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 0x7f620c0b5700 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.182+0000 7f6212cc9640 1 -- 172.21.15.27:0/2940817192 >> 172.21.15.27:0/2940817192 conn(0x7f620c071960 msgr2=0x7f620c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.182+0000 7f6212cc9640 1 -- 172.21.15.27:0/2940817192 shutdown_connections 2024-03-06T07:29:39.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.182+0000 7f6212cc9640 1 -- 172.21.15.27:0/2940817192 wait complete. 2024-03-06T07:29:39.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.183+0000 7f6212cc9640 1 Processor -- start 2024-03-06T07:29:39.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.183+0000 7f6212cc9640 1 -- start start 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6212cc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 0x7f620c14b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6212cc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c0b9a10 0x7f620c149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6212cc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f620c15df00 0x7f620c1f77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6212cc9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0cedf0 con 0x7f620c0b95a0 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6212cc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0cec70 con 0x7f620c0b9a10 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6212cc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0b5c40 con 0x7f620c15df00 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6210a65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 0x7f620c14b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6210a65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 0x7f620c14b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35548/0 (socket says 172.21.15.27:35548) 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6210a65640 1 -- 172.21.15.27:0/2838543181 learned_addr learned my addr 172.21.15.27:0/2838543181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6211266640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f620c15df00 0x7f620c1f77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6203fff640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c0b9a10 0x7f620c149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6210a65640 1 -- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f620c15df00 msgr2=0x7f620c1f77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6210a65640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f620c15df00 0x7f620c1f77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6210a65640 1 -- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c0b9a10 msgr2=0x7f620c149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6210a65640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c0b9a10 0x7f620c149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6210a65640 1 -- 172.21.15.27:0/2838543181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c14bf20 con 0x7f620c0b95a0 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6211266640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f620c15df00 0x7f620c1f77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6203fff640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c0b9a10 0x7f620c149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.184+0000 7f6210a65640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 0x7f620c14b940 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f620800c730 tx=0x7f620800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.185+0000 7f62017fa640 1 -- 172.21.15.27:0/2838543181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6208016020 con 0x7f620c0b95a0 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.185+0000 7f62017fa640 1 -- 172.21.15.27:0/2838543181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6208006d50 con 0x7f620c0b95a0 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.185+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c14c200 con 0x7f620c0b95a0 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.185+0000 7f62017fa640 1 -- 172.21.15.27:0/2838543181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6208003bb0 con 0x7f620c0b95a0 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.185+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c14c7c0 con 0x7f620c0b95a0 2024-03-06T07:29:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.186+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c067ff0 con 0x7f620c0b95a0 2024-03-06T07:29:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.186+0000 7f62017fa640 1 -- 172.21.15.27:0/2838543181 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f620800a850 con 0x7f620c0b95a0 2024-03-06T07:29:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.187+0000 7f62017fa640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61f806b4d0 0x7f61f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.187+0000 7f62017fa640 1 -- 172.21.15.27:0/2838543181 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f6208089f40 con 0x7f620c0b95a0 2024-03-06T07:29:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.187+0000 7f6203fff640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61f806b4d0 0x7f61f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.188+0000 7f6203fff640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61f806b4d0 0x7f61f806d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f61fc036000 tx=0x7f61fc039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.196+0000 7f62017fa640 1 -- 172.21.15.27:0/2838543181 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6208020020 con 0x7f620c0b95a0 2024-03-06T07:29:39.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.495+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f620c06b780 con 0x7f620c0b95a0 2024-03-06T07:29:39.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.495+0000 7f62017fa640 1 -- 172.21.15.27:0/2838543181 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f6208019310 con 0x7f620c0b95a0 2024-03-06T07:29:39.498 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:39.498 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:39.498 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61f806b4d0 msgr2=0x7f61f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61f806b4d0 0x7f61f806d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f61fc036000 tx=0x7f61fc039040 comp rx=0 tx=0).stop 2024-03-06T07:29:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 msgr2=0x7f620c14b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 0x7f620c14b940 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f620800c730 tx=0x7f620800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 shutdown_connections 2024-03-06T07:29:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61f806b4d0 0x7f61f806d990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f620c15df00 0x7f620c1f77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c0b9a10 0x7f620c149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 --2- 172.21.15.27:0/2838543181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f620c0b95a0 0x7f620c14b940 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 >> 172.21.15.27:0/2838543181 conn(0x7f620c071960 msgr2=0x7f620c0b7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 shutdown_connections 2024-03-06T07:29:39.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.497+0000 7f6212cc9640 1 -- 172.21.15.27:0/2838543181 wait complete. 2024-03-06T07:29:39.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:29:39.634 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:29:39.634 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:29:39.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:39.634 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:29:39.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:39.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.804+0000 7febb369a640 1 Processor -- start 2024-03-06T07:29:39.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.804+0000 7febb369a640 1 -- start start 2024-03-06T07:29:39.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb369a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb369a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febac0b0e90 0x7febac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb369a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febac0b3340 con 0x7febac0ae9b0 2024-03-06T07:29:39.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb369a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febac0cfa00 con 0x7febac0b0e90 2024-03-06T07:29:39.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb369a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac0cee90 con 0x7febac06b5c0 2024-03-06T07:29:39.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb1436640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb0c35640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febac0b0e90 0x7febac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb1436640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:34282/0 (socket says 172.21.15.27:34282) 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb0c35640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febac0b0e90 0x7febac06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:48222/0 (socket says 172.21.15.27:48222) 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb1436640 1 -- 172.21.15.27:0/2853200710 learned_addr learned my addr 172.21.15.27:0/2853200710 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb1c37640 1 --1- 172.21.15.27:0/2853200710 >> v1:172.21.15.69:6789/0 conn(0x7febac0ae9b0 0x7febac0d03a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:55574/0 (socket says 172.21.15.27:55574) 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7feba3fff640 1 -- 172.21.15.27:0/2853200710 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412423951 0 0) 0x7febac0b3340 con 0x7febac0ae9b0 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7feba3fff640 1 -- 172.21.15.27:0/2853200710 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb98003520 con 0x7febac0ae9b0 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb1436640 1 -- 172.21.15.27:0/2853200710 >> v2:172.21.15.69:3300/0 conn(0x7febac0b0e90 msgr2=0x7febac06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb1436640 1 --2- 172.21.15.27:0/2853200710 >> v2:172.21.15.69:3300/0 conn(0x7febac0b0e90 0x7febac06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb1436640 1 -- 172.21.15.27:0/2853200710 >> v1:172.21.15.69:6789/0 conn(0x7febac0ae9b0 legacy=0x7febac0d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb1436640 1 -- 172.21.15.27:0/2853200710 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac0cfc50 con 0x7febac06b5c0 2024-03-06T07:29:39.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.805+0000 7febb0c35640 1 --2- 172.21.15.27:0/2853200710 >> v2:172.21.15.69:3300/0 conn(0x7febac0b0e90 0x7febac06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:39.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.807+0000 7febb1436640 1 --2- 172.21.15.27:0/2853200710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0b2e00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feba8009870 tx=0x7feba802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5257768f0ef4cd65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.807+0000 7feba3fff640 1 -- 172.21.15.27:0/2853200710 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba8035020 con 0x7febac06b5c0 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.807+0000 7feba3fff640 1 -- 172.21.15.27:0/2853200710 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba8031640 con 0x7febac06b5c0 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.807+0000 7feba3fff640 1 -- 172.21.15.27:0/2853200710 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba802d040 con 0x7febac06b5c0 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.807+0000 7febb369a640 1 -- 172.21.15.27:0/2853200710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 msgr2=0x7febac0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.807+0000 7febb369a640 1 --2- 172.21.15.27:0/2853200710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0b2e00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feba8009870 tx=0x7feba802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.808+0000 7febb369a640 1 -- 172.21.15.27:0/2853200710 shutdown_connections 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.808+0000 7febb369a640 1 --2- 172.21.15.27:0/2853200710 >> v2:172.21.15.69:3300/0 conn(0x7febac0b0e90 0x7febac06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.808+0000 7febb369a640 1 --2- 172.21.15.27:0/2853200710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0b2e00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.808+0000 7febb369a640 1 -- 172.21.15.27:0/2853200710 >> 172.21.15.27:0/2853200710 conn(0x7febac0b36d0 msgr2=0x7febac0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.808+0000 7febb369a640 1 -- 172.21.15.27:0/2853200710 shutdown_connections 2024-03-06T07:29:39.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.808+0000 7febb369a640 1 -- 172.21.15.27:0/2853200710 wait complete. 2024-03-06T07:29:39.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.809+0000 7febb369a640 1 Processor -- start 2024-03-06T07:29:39.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.809+0000 7febb369a640 1 -- start start 2024-03-06T07:29:39.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb369a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0700a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb369a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac0ae9b0 0x7febac154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb369a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febac0b0e90 0x7febac155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb369a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac06c790 con 0x7febac0ae9b0 2024-03-06T07:29:39.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb369a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac06bee0 con 0x7febac0b0e90 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb369a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac06c910 con 0x7febac06b5c0 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb0c35640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac0ae9b0 0x7febac154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb0c35640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac0ae9b0 0x7febac154b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35556/0 (socket says 172.21.15.27:35556) 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb0c35640 1 -- 172.21.15.27:0/2599731552 learned_addr learned my addr 172.21.15.27:0/2599731552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb1436640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0700a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb1c37640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febac0b0e90 0x7febac155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb0c35640 1 -- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 msgr2=0x7febac0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb0c35640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0700a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb0c35640 1 -- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febac0b0e90 msgr2=0x7febac155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb0c35640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febac0b0e90 0x7febac155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb0c35640 1 -- 172.21.15.27:0/2599731552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac1f3e40 con 0x7febac0ae9b0 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb0c35640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac0ae9b0 0x7febac154b10 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00ebe0 tx=0x7feb9c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7feba1ffb640 1 -- 172.21.15.27:0/2599731552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c017070 con 0x7febac0ae9b0 2024-03-06T07:29:39.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7feba1ffb640 1 -- 172.21.15.27:0/2599731552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9c00cc30 con 0x7febac0ae9b0 2024-03-06T07:29:39.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.810+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febac1f4120 con 0x7febac0ae9b0 2024-03-06T07:29:39.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.811+0000 7feba1ffb640 1 -- 172.21.15.27:0/2599731552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c013470 con 0x7febac0ae9b0 2024-03-06T07:29:39.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.811+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac1f46e0 con 0x7febac0ae9b0 2024-03-06T07:29:39.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.812+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac068700 con 0x7febac0ae9b0 2024-03-06T07:29:39.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.812+0000 7feba1ffb640 1 -- 172.21.15.27:0/2599731552 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb9c01a070 con 0x7febac0ae9b0 2024-03-06T07:29:39.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.813+0000 7feba1ffb640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feb9006b4d0 0x7feb9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.813+0000 7feba1ffb640 1 -- 172.21.15.27:0/2599731552 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7feb9c08a5d0 con 0x7febac0ae9b0 2024-03-06T07:29:39.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.813+0000 7febb1436640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feb9006b4d0 0x7feb9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.814+0000 7febb1436640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feb9006b4d0 0x7feb9006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7feba8036000 tx=0x7feba803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:39.817+0000 7feba1ffb640 1 -- 172.21.15.27:0/2599731552 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb9c059370 con 0x7febac0ae9b0 2024-03-06T07:29:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.038+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febac0b1880 con 0x7febac0ae9b0 2024-03-06T07:29:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.039+0000 7feba1ffb640 1 -- 172.21.15.27:0/2599731552 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7feb9c058c70 con 0x7febac0ae9b0 2024-03-06T07:29:40.042 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:40.042 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:40.042 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feb9006b4d0 msgr2=0x7feb9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feb9006b4d0 0x7feb9006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7feba8036000 tx=0x7feba803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac0ae9b0 msgr2=0x7febac154b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac0ae9b0 0x7febac154b10 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00ebe0 tx=0x7feb9c00c520 comp rx=0 tx=0).stop 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 shutdown_connections 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feb9006b4d0 0x7feb9006d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febac0b0e90 0x7febac155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febac0ae9b0 0x7febac154b10 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 --2- 172.21.15.27:0/2599731552 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7febac06b5c0 0x7febac0700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 >> 172.21.15.27:0/2599731552 conn(0x7febac0b36d0 msgr2=0x7febac0aee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 shutdown_connections 2024-03-06T07:29:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.041+0000 7febb369a640 1 -- 172.21.15.27:0/2599731552 wait complete. 2024-03-06T07:29:40.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:29:40.216 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:29:40.217 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:29:40.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:40.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.322+0000 7f3c953bd640 1 Processor -- start 2024-03-06T07:29:40.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.322+0000 7f3c953bd640 1 -- start start 2024-03-06T07:29:40.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c953bd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c900b1020 0x7f3c9006af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c953bd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3c9006ed10 0x7f3c900cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c953bd640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c900cc0e0 con 0x7f3c900cd6c0 2024-03-06T07:29:40.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c953bd640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c900d28b0 con 0x7f3c9006ed10 2024-03-06T07:29:40.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c953bd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900cf320 con 0x7f3c900b1020 2024-03-06T07:29:40.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c8f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3c900cd6c0 0x7f3c9015d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51934/0 (socket says 172.21.15.27:51934) 2024-03-06T07:29:40.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c8f7fe640 1 -- 172.21.15.27:0/2655757166 learned_addr learned my addr 172.21.15.27:0/2655757166 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c8effd640 1 --2- 172.21.15.27:0/2655757166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c900b1020 0x7f3c9006af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c87fff640 1 --2- 172.21.15.27:0/2655757166 >> v2:172.21.15.69:3300/0 conn(0x7f3c9006ed10 0x7f3c900cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c8e7fc640 1 -- 172.21.15.27:0/2655757166 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427711280 0 0) 0x7f3c900cc0e0 con 0x7f3c900cd6c0 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c8e7fc640 1 -- 172.21.15.27:0/2655757166 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c7c003520 con 0x7f3c900cd6c0 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c8effd640 1 -- 172.21.15.27:0/2655757166 >> v2:172.21.15.69:3300/0 conn(0x7f3c9006ed10 msgr2=0x7f3c900cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c8effd640 1 --2- 172.21.15.27:0/2655757166 >> v2:172.21.15.69:3300/0 conn(0x7f3c9006ed10 0x7f3c900cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c8effd640 1 -- 172.21.15.27:0/2655757166 >> v1:172.21.15.27:6789/0 conn(0x7f3c900cd6c0 legacy=0x7f3c9015d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c8effd640 1 -- 172.21.15.27:0/2655757166 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c900cc3c0 con 0x7f3c900b1020 2024-03-06T07:29:40.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.323+0000 7f3c87fff640 1 --2- 172.21.15.27:0/2655757166 >> v2:172.21.15.69:3300/0 conn(0x7f3c9006ed10 0x7f3c900cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.324+0000 7f3c8effd640 1 --2- 172.21.15.27:0/2655757166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c900b1020 0x7f3c9006af20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3c800097a0 tx=0x7f3c8002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad545ac24503633 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.324+0000 7f3c8e7fc640 1 -- 172.21.15.27:0/2655757166 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80035020 con 0x7f3c900b1020 2024-03-06T07:29:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.324+0000 7f3c8e7fc640 1 -- 172.21.15.27:0/2655757166 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c80031830 con 0x7f3c900b1020 2024-03-06T07:29:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.324+0000 7f3c8e7fc640 1 -- 172.21.15.27:0/2655757166 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80031be0 con 0x7f3c900b1020 2024-03-06T07:29:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.325+0000 7f3c953bd640 1 -- 172.21.15.27:0/2655757166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c900b1020 msgr2=0x7f3c9006af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.325+0000 7f3c953bd640 1 --2- 172.21.15.27:0/2655757166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c900b1020 0x7f3c9006af20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3c800097a0 tx=0x7f3c8002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.325+0000 7f3c953bd640 1 -- 172.21.15.27:0/2655757166 shutdown_connections 2024-03-06T07:29:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.325+0000 7f3c953bd640 1 --2- 172.21.15.27:0/2655757166 >> v2:172.21.15.69:3300/0 conn(0x7f3c9006ed10 0x7f3c900cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.325+0000 7f3c953bd640 1 --2- 172.21.15.27:0/2655757166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c900b1020 0x7f3c9006af20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.325+0000 7f3c953bd640 1 -- 172.21.15.27:0/2655757166 >> 172.21.15.27:0/2655757166 conn(0x7f3c900d1a90 msgr2=0x7f3c900d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.325+0000 7f3c953bd640 1 -- 172.21.15.27:0/2655757166 shutdown_connections 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.325+0000 7f3c953bd640 1 -- 172.21.15.27:0/2655757166 wait complete. 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.326+0000 7f3c953bd640 1 Processor -- start 2024-03-06T07:29:40.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.326+0000 7f3c953bd640 1 -- start start 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c953bd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c9006ed10 0x7f3c9015dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c953bd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c900b1020 0x7f3c900b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c953bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c900cd6c0 0x7f3c901f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c953bd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900ccc60 con 0x7f3c900cd6c0 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c953bd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900cc980 con 0x7f3c900b1020 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c953bd640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900cdaf0 con 0x7f3c9006ed10 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c8f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c900cd6c0 0x7f3c901f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c8f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c900cd6c0 0x7f3c901f6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35558/0 (socket says 172.21.15.27:35558) 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c8f7fe640 1 -- 172.21.15.27:0/2662276988 learned_addr learned my addr 172.21.15.27:0/2662276988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:40.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c8effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c9006ed10 0x7f3c9015dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c87fff640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c900b1020 0x7f3c900b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c8f7fe640 1 -- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c9006ed10 msgr2=0x7f3c9015dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c8f7fe640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c9006ed10 0x7f3c9015dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c8f7fe640 1 -- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c900b1020 msgr2=0x7f3c900b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.327+0000 7f3c8f7fe640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c900b1020 0x7f3c900b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.328+0000 7f3c8f7fe640 1 -- 172.21.15.27:0/2662276988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c900b5d70 con 0x7f3c900cd6c0 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.328+0000 7f3c8f7fe640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c900cd6c0 0x7f3c901f6f60 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800caa0 tx=0x7f3c8800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.328+0000 7f3c877fe640 1 -- 172.21.15.27:0/2662276988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88017070 con 0x7f3c900cd6c0 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.328+0000 7f3c877fe640 1 -- 172.21.15.27:0/2662276988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8800ac00 con 0x7f3c900cd6c0 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.328+0000 7f3c877fe640 1 -- 172.21.15.27:0/2662276988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88012400 con 0x7f3c900cd6c0 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.328+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c900b6050 con 0x7f3c900cd6c0 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.328+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c901f4cd0 con 0x7f3c900cd6c0 2024-03-06T07:29:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.329+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c900b6570 con 0x7f3c900cd6c0 2024-03-06T07:29:40.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.330+0000 7f3c877fe640 1 -- 172.21.15.27:0/2662276988 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c8801a070 con 0x7f3c900cd6c0 2024-03-06T07:29:40.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.330+0000 7f3c877fe640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c7006b4d0 0x7f3c7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.331+0000 7f3c877fe640 1 -- 172.21.15.27:0/2662276988 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f3c8808a040 con 0x7f3c900cd6c0 2024-03-06T07:29:40.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.331+0000 7f3c8effd640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c7006b4d0 0x7f3c7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.331+0000 7f3c8effd640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c7006b4d0 0x7f3c7006d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3c80036000 tx=0x7f3c80008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.335+0000 7f3c877fe640 1 -- 172.21.15.27:0/2662276988 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c900b6570 con 0x7f3c900cd6c0 2024-03-06T07:29:40.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.554+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c900d2130 con 0x7f3c900cd6c0 2024-03-06T07:29:40.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.555+0000 7f3c877fe640 1 -- 172.21.15.27:0/2662276988 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f3c900d2130 con 0x7f3c900cd6c0 2024-03-06T07:29:40.556 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:40.556 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:40.557 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:40.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.556+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c7006b4d0 msgr2=0x7f3c7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.556+0000 7f3c953bd640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c7006b4d0 0x7f3c7006d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3c80036000 tx=0x7f3c80008030 comp rx=0 tx=0).stop 2024-03-06T07:29:40.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.556+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c900cd6c0 msgr2=0x7f3c901f6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.556+0000 7f3c953bd640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c900cd6c0 0x7f3c901f6f60 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800caa0 tx=0x7f3c8800cf60 comp rx=0 tx=0).stop 2024-03-06T07:29:40.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.557+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 shutdown_connections 2024-03-06T07:29:40.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.557+0000 7f3c953bd640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c7006b4d0 0x7f3c7006d990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.557+0000 7f3c953bd640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c900cd6c0 0x7f3c901f6f60 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.557+0000 7f3c953bd640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c900b1020 0x7f3c900b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.557+0000 7f3c953bd640 1 --2- 172.21.15.27:0/2662276988 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c9006ed10 0x7f3c9015dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.557+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 >> 172.21.15.27:0/2662276988 conn(0x7f3c900d1a90 msgr2=0x7f3c9006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.557+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 shutdown_connections 2024-03-06T07:29:40.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.557+0000 7f3c953bd640 1 -- 172.21.15.27:0/2662276988 wait complete. 2024-03-06T07:29:40.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:29:40.692 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:29:40.692 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:29:40.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.800+0000 7f1086cdf640 1 Processor -- start 2024-03-06T07:29:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.800+0000 7f1086cdf640 1 -- start start 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.800+0000 7f1086cdf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f108015df00 0x7f10800d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.800+0000 7f1086cdf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10800b95a0 0x7f10800b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.800+0000 7f1086cdf640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10800cea30 con 0x7f10800b9a30 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.800+0000 7f1086cdf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10800d2af0 con 0x7f108015df00 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.800+0000 7f1086cdf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10800cf3c0 con 0x7f10800b95a0 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f108527c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f10800b9a30 0x7f10800d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51946/0 (socket says 172.21.15.27:51946) 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f108527c640 1 -- 172.21.15.27:0/306028755 learned_addr learned my addr 172.21.15.27:0/306028755 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f107ffff640 1 --2- 172.21.15.27:0/306028755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10800b95a0 0x7f10800b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f1084a7b640 1 --2- 172.21.15.27:0/306028755 >> v2:172.21.15.69:3300/0 conn(0x7f108015df00 0x7f10800d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f107f7fe640 1 -- 172.21.15.27:0/306028755 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336759736 0 0) 0x7f10800cea30 con 0x7f10800b9a30 2024-03-06T07:29:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f107f7fe640 1 -- 172.21.15.27:0/306028755 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1070003520 con 0x7f10800b9a30 2024-03-06T07:29:40.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f107ffff640 1 -- 172.21.15.27:0/306028755 >> v2:172.21.15.69:3300/0 conn(0x7f108015df00 msgr2=0x7f10800d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f107ffff640 1 --2- 172.21.15.27:0/306028755 >> v2:172.21.15.69:3300/0 conn(0x7f108015df00 0x7f10800d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f107ffff640 1 -- 172.21.15.27:0/306028755 >> v1:172.21.15.27:6789/0 conn(0x7f10800b9a30 legacy=0x7f10800d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f107f7fe640 1 -- 172.21.15.27:0/306028755 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1844747823 0 0) 0x7f1070003520 con 0x7f10800b9a30 2024-03-06T07:29:40.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.801+0000 7f107ffff640 1 -- 172.21.15.27:0/306028755 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10800cf0e0 con 0x7f10800b95a0 2024-03-06T07:29:40.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.802+0000 7f107ffff640 1 --2- 172.21.15.27:0/306028755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10800b95a0 0x7f10800b5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f106c0097a0 tx=0x7f106c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a64762769ad491 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.802+0000 7f107f7fe640 1 -- 172.21.15.27:0/306028755 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c037070 con 0x7f10800b95a0 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.802+0000 7f107f7fe640 1 -- 172.21.15.27:0/306028755 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c030510 con 0x7f10800b95a0 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.802+0000 7f107f7fe640 1 -- 172.21.15.27:0/306028755 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780019f0 con 0x7f10800b95a0 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.803+0000 7f1086cdf640 1 -- 172.21.15.27:0/306028755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10800b95a0 msgr2=0x7f10800b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.803+0000 7f1086cdf640 1 --2- 172.21.15.27:0/306028755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10800b95a0 0x7f10800b5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f106c0097a0 tx=0x7f106c02b790 comp rx=0 tx=0).stop 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.803+0000 7f1086cdf640 1 -- 172.21.15.27:0/306028755 shutdown_connections 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.803+0000 7f1086cdf640 1 --2- 172.21.15.27:0/306028755 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10800b95a0 0x7f10800b5700 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.803+0000 7f1086cdf640 1 --2- 172.21.15.27:0/306028755 >> v2:172.21.15.69:3300/0 conn(0x7f108015df00 0x7f10800d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.803+0000 7f1086cdf640 1 -- 172.21.15.27:0/306028755 >> 172.21.15.27:0/306028755 conn(0x7f1080071960 msgr2=0x7f1080071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.803+0000 7f1086cdf640 1 -- 172.21.15.27:0/306028755 shutdown_connections 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.803+0000 7f1086cdf640 1 -- 172.21.15.27:0/306028755 wait complete. 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.804+0000 7f1086cdf640 1 Processor -- start 2024-03-06T07:29:40.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.804+0000 7f1086cdf640 1 -- start start 2024-03-06T07:29:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f1086cdf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10800b95a0 0x7f10800c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f1086cdf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f108015df00 0x7f108014af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f1086cdf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10800b4520 0x7f10800b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f1086cdf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10800cec70 con 0x7f10800b4520 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f1086cdf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10800ce750 con 0x7f10800b95a0 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f1086cdf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10800cedf0 con 0x7f108015df00 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f108527c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10800b4520 0x7f10800b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f108527c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10800b4520 0x7f10800b49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35572/0 (socket says 172.21.15.27:35572) 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f108527c640 1 -- 172.21.15.27:0/1002159039 learned_addr learned my addr 172.21.15.27:0/1002159039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f1084a7b640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10800b95a0 0x7f10800c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f107ffff640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f108015df00 0x7f108014af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f108527c640 1 -- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f108015df00 msgr2=0x7f108014af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f108527c640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f108015df00 0x7f108014af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f108527c640 1 -- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10800b95a0 msgr2=0x7f10800c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f108527c640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10800b95a0 0x7f10800c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.805+0000 7f108527c640 1 -- 172.21.15.27:0/1002159039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10800c7cf0 con 0x7f10800b4520 2024-03-06T07:29:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.806+0000 7f108527c640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10800b4520 0x7f10800b49b0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f1078001560 tx=0x7f107800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.806+0000 7f107d7fa640 1 -- 172.21.15.27:0/1002159039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780019f0 con 0x7f10800b4520 2024-03-06T07:29:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.806+0000 7f107d7fa640 1 -- 172.21.15.27:0/1002159039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1078012400 con 0x7f10800b4520 2024-03-06T07:29:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.806+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10800cb930 con 0x7f10800b4520 2024-03-06T07:29:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.806+0000 7f107d7fa640 1 -- 172.21.15.27:0/1002159039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780127f0 con 0x7f10800b4520 2024-03-06T07:29:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.806+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108015f7e0 con 0x7f10800b4520 2024-03-06T07:29:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.807+0000 7f107d7fa640 1 -- 172.21.15.27:0/1002159039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1078012990 con 0x7f10800b4520 2024-03-06T07:29:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.807+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1080067ff0 con 0x7f10800b4520 2024-03-06T07:29:40.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.808+0000 7f107d7fa640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f106406b500 0x7f106406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.808+0000 7f107d7fa640 1 -- 172.21.15.27:0/1002159039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f107808a8a0 con 0x7f10800b4520 2024-03-06T07:29:40.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.808+0000 7f1084a7b640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f106406b500 0x7f106406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.808+0000 7f1084a7b640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f106406b500 0x7f106406d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1074009aa0 tx=0x7f1074009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:40.812+0000 7f107d7fa640 1 -- 172.21.15.27:0/1002159039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1078059070 con 0x7f10800b4520 2024-03-06T07:29:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.039+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10800b72c0 con 0x7f10800b4520 2024-03-06T07:29:41.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.040+0000 7f107d7fa640 1 -- 172.21.15.27:0/1002159039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f107805d110 con 0x7f10800b4520 2024-03-06T07:29:41.042 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:41.043 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:41.044 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:41.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f106406b500 msgr2=0x7f106406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f106406b500 0x7f106406d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1074009aa0 tx=0x7f1074009190 comp rx=0 tx=0).stop 2024-03-06T07:29:41.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10800b4520 msgr2=0x7f10800b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10800b4520 0x7f10800b49b0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f1078001560 tx=0x7f107800a430 comp rx=0 tx=0).stop 2024-03-06T07:29:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 shutdown_connections 2024-03-06T07:29:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f106406b500 0x7f106406d9c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10800b4520 0x7f10800b49b0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f108015df00 0x7f108014af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 --2- 172.21.15.27:0/1002159039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10800b95a0 0x7f10800c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 >> 172.21.15.27:0/1002159039 conn(0x7f1080071960 msgr2=0x7f10800b8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 shutdown_connections 2024-03-06T07:29:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.042+0000 7f1086cdf640 1 -- 172.21.15.27:0/1002159039 wait complete. 2024-03-06T07:29:41.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:29:41.190 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:29:41.190 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:41.190 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.231+0000 7ff41f9b7f40 1 Processor -- start 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.231+0000 7ff41f9b7f40 1 -- start start 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41f9b7f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5581140c6a10 0x558114195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41f9b7f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 0x5581141a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41f9b7f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558114199650 0x5581140ee190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41f9b7f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811410bc60 con 0x558114199650 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41f9b7f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811406ab70 con 0x558114198d30 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41f9b7f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55811410bde0 con 0x5581140c6a10 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41dcbb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558114199650 0x5581140ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41dcbb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558114199650 0x5581140ee190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35592/0 (socket says 172.21.15.27:35592) 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41dcbb640 1 -- 172.21.15.27:0/2288385266 learned_addr learned my addr 172.21.15.27:0/2288385266 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41d4ba640 1 --2- 172.21.15.27:0/2288385266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5581140c6a10 0x558114195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41ccb9640 1 --2- 172.21.15.27:0/2288385266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 0x5581141a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41dcbb640 1 -- 172.21.15.27:0/2288385266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5581140c6a10 msgr2=0x558114195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41dcbb640 1 --2- 172.21.15.27:0/2288385266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5581140c6a10 0x558114195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41dcbb640 1 -- 172.21.15.27:0/2288385266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 msgr2=0x5581141a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41dcbb640 1 --2- 172.21.15.27:0/2288385266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 0x5581141a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41dcbb640 1 -- 172.21.15.27:0/2288385266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581141963e0 con 0x558114199650 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff41dcbb640 1 --2- 172.21.15.27:0/2288385266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558114199650 0x5581140ee190 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7ff4180447e0 tx=0x7ff41804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75ec0e3fe1c3dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.232+0000 7ff4177fe640 1 -- 172.21.15.27:0/2288385266 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff418057040 con 0x558114199650 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff4177fe640 1 -- 172.21.15.27:0/2288385266 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff418061070 con 0x558114199650 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff4177fe640 1 -- 172.21.15.27:0/2288385266 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff418052b40 con 0x558114199650 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2288385266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558114199650 msgr2=0x5581140ee190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff41f9b7f40 1 --2- 172.21.15.27:0/2288385266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558114199650 0x5581140ee190 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7ff4180447e0 tx=0x7ff41804c000 comp rx=0 tx=0).stop 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2288385266 shutdown_connections 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff41f9b7f40 1 --2- 172.21.15.27:0/2288385266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558114199650 0x5581140ee190 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff41f9b7f40 1 --2- 172.21.15.27:0/2288385266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 0x5581141a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff41f9b7f40 1 --2- 172.21.15.27:0/2288385266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x5581140c6a10 0x558114195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2288385266 >> 172.21.15.27:0/2288385266 conn(0x5581140c35e0 msgr2=0x55811410c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2288385266 shutdown_connections 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.233+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2288385266 wait complete. 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41f9b7f40 1 Processor -- start 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41f9b7f40 1 -- start start 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41f9b7f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581140c6a10 0x558114179d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41f9b7f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 0x5581140ee160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41f9b7f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558114199650 0x5581141a2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41f9b7f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581141a3420 con 0x5581140c6a10 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41f9b7f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811419d8b0 con 0x558114198d30 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41f9b7f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581140edbd0 con 0x558114199650 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41d4ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581140c6a10 0x558114179d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41d4ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581140c6a10 0x558114179d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35598/0 (socket says 172.21.15.27:35598) 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41d4ba640 1 -- 172.21.15.27:0/2003730162 learned_addr learned my addr 172.21.15.27:0/2003730162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.234+0000 7ff41dcbb640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558114199650 0x5581141a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41ccb9640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 0x5581140ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41d4ba640 1 -- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558114199650 msgr2=0x5581141a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41d4ba640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x558114199650 0x5581141a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41d4ba640 1 -- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 msgr2=0x5581140ee160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41d4ba640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 0x5581140ee160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41d4ba640 1 -- 172.21.15.27:0/2003730162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581140eec20 con 0x5581140c6a10 2024-03-06T07:29:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41ccb9640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558114198d30 0x5581140ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41d4ba640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581140c6a10 0x558114179d00 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7ff410009f40 tx=0x7ff410009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff4167fc640 1 -- 172.21.15.27:0/2003730162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4100143f0 con 0x5581140c6a10 2024-03-06T07:29:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff4167fc640 1 -- 172.21.15.27:0/2003730162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff410019070 con 0x5581140c6a10 2024-03-06T07:29:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff4167fc640 1 -- 172.21.15.27:0/2003730162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4100147d0 con 0x5581140c6a10 2024-03-06T07:29:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5581140eef00 con 0x5581140c6a10 2024-03-06T07:29:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.235+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581140ef4c0 con 0x5581140c6a10 2024-03-06T07:29:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.237+0000 7ff4167fc640 1 -- 172.21.15.27:0/2003730162 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff410014970 con 0x5581140c6a10 2024-03-06T07:29:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.238+0000 7ff4167fc640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff400069eb0 0x7ff40006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.238+0000 7ff4167fc640 1 -- 172.21.15.27:0/2003730162 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff410088d60 con 0x5581140c6a10 2024-03-06T07:29:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.238+0000 7ff41f9b7f40 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55811419dd10 0x55811419e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.238+0000 7ff41ccb9640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff400069eb0 0x7ff40006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.238+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120) v9 -- 0x55811419e6a0 con 0x55811419dd10 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.238+0000 7ff41dcbb640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55811419dd10 0x55811419e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.238+0000 7ff41dcbb640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55811419dd10 0x55811419e160 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.238+0000 7ff41ccb9640 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff400069eb0 0x7ff40006c370 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x55811419d730 tx=0x7ff408003050 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41dcbb640 1 -- 172.21.15.27:0/2003730162 <== osd.0 v2:172.21.15.27:6800/3926946944 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7ff41805a040 con 0x55811419dd10 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55811419dd10 msgr2=0x55811419e160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55811419dd10 0x55811419e160 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff400069eb0 msgr2=0x7ff40006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff400069eb0 0x7ff40006c370 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x55811419d730 tx=0x7ff408003050 comp rx=0 tx=0).stop 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581140c6a10 msgr2=0x558114179d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 --2- 172.21.15.27:0/2003730162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581140c6a10 0x558114179d00 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7ff410009f40 tx=0x7ff410009f70 comp rx=0 tx=0).stop 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41dcbb640 1 -- 172.21.15.27:0/2003730162 reap_dead start 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 shutdown_connections 2024-03-06T07:29:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 >> 172.21.15.27:0/2003730162 conn(0x5581140c35e0 msgr2=0x558114194ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 shutdown_connections 2024-03-06T07:29:41.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.239+0000 7ff41f9b7f40 1 -- 172.21.15.27:0/2003730162 wait complete. 2024-03-06T07:29:41.245 INFO:teuthology.orchestra.run.smithi027.stdout:W�file_cdir_2 2024-03-06T07:29:41.246 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:29:41.332 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:29:41.332 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511627777, 2024-03-06T07:29:41.332 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2024-03-06T07:29:41.332 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:29:41.332 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1099511628278, 2024-03-06T07:29:41.332 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "file_c", 2024-03-06T07:29:41.332 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 15 2024-03-06T07:29:41.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:29:41.332 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:29:41.333 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1, 2024-03-06T07:29:41.333 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "dir_2", 2024-03-06T07:29:41.333 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 31 2024-03-06T07:29:41.333 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:29:41.333 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-03-06T07:29:41.333 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 17, 2024-03-06T07:29:41.333 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [] 2024-03-06T07:29:41.333 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:29:41.333 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:41.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:41.335 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:29:41.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:41.393 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:29:41.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:41.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.597+0000 7fa6dcb3e640 1 Processor -- start 2024-03-06T07:29:41.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.597+0000 7fa6dcb3e640 1 -- start start 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6dcb3e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6d815df00 0x7fa6d80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6dcb3e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80cea30 con 0x7fa6d80b95a0 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6dcb3e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80d2af0 con 0x7fa6d80b9a30 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6dcb3e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d80cf3c0 con 0x7fa6d815df00 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d5d9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6d80b95a0 0x7fa6d80b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51960/0 (socket says 172.21.15.27:51960) 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d5d9b640 1 -- 172.21.15.27:0/607157017 learned_addr learned my addr 172.21.15.27:0/607157017 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d659c640 1 --2- 172.21.15.27:0/607157017 >> v2:172.21.15.69:3300/0 conn(0x7fa6d815df00 0x7fa6d80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155849741 0 0) 0x7fa6d80cea30 con 0x7fa6d80b95a0 2024-03-06T07:29:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c0003520 con 0x7fa6d80b95a0 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528632528 0 0) 0x7fa6d80d2af0 con 0x7fa6d80b9a30 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d80cea30 con 0x7fa6d80b9a30 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439788374 0 0) 0x7fa6c0003520 con 0x7fa6d80b95a0 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 >> v2:172.21.15.69:3300/0 conn(0x7fa6d815df00 msgr2=0x7fa6d80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 --2- 172.21.15.27:0/607157017 >> v2:172.21.15.69:3300/0 conn(0x7fa6d815df00 0x7fa6d80d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 >> v1:172.21.15.69:6789/0 conn(0x7fa6d80b9a30 legacy=0x7fa6d80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80cf0e0 con 0x7fa6d80b95a0 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fa6cc001670 con 0x7fa6d80b95a0 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 >> v1:172.21.15.27:6789/0 conn(0x7fa6d80b95a0 legacy=0x7fa6d80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6c0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c0004800 0x7fa6c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.598+0000 7fa6d559a640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c00051c0 0x7fa6c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d80cf3c0 con 0x7fa6c0003b90 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c000c090 con 0x7fa6c0004800 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c000c290 con 0x7fa6c00051c0 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d659c640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6c0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d5d9b640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c0004800 0x7fa6c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d6d9d640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c00051c0 0x7fa6c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d659c640 1 -- 172.21.15.27:0/607157017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c00051c0 msgr2=0x7fa6c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d659c640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c00051c0 0x7fa6c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d659c640 1 -- 172.21.15.27:0/607157017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c0004800 msgr2=0x7fa6c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d659c640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c0004800 0x7fa6c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d659c640 1 -- 172.21.15.27:0/607157017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c0003520 con 0x7fa6c0003b90 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d6d9d640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c00051c0 0x7fa6c000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.599+0000 7fa6d6d9d640 1 -- 172.21.15.27:0/607157017 reap_dead start 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6d659c640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6c0003f90 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009ac0 tx=0x7fa6c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56dc5b56de714bd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8039070 con 0x7fa6c0003b90 2024-03-06T07:29:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c802c6b0 con 0x7fa6c0003b90 2024-03-06T07:29:41.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6d559a640 1 -- 172.21.15.27:0/607157017 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8009690 con 0x7fa6c0003b90 2024-03-06T07:29:41.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/607157017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 msgr2=0x7fa6c0003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6dcb3e640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6c0003f90 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009ac0 tx=0x7fa6c0005830 comp rx=0 tx=0).stop 2024-03-06T07:29:41.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/607157017 shutdown_connections 2024-03-06T07:29:41.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6dcb3e640 1 --2- 172.21.15.27:0/607157017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6c0003f90 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/607157017 >> 172.21.15.27:0/607157017 conn(0x7fa6d8071960 msgr2=0x7fa6d8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/607157017 shutdown_connections 2024-03-06T07:29:41.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.600+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/607157017 wait complete. 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.602+0000 7fa6dcb3e640 1 Processor -- start 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.602+0000 7fa6dcb3e640 1 -- start start 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.602+0000 7fa6dcb3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6d81552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.602+0000 7fa6dcb3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d80cc570 0x7fa6d80cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6dcb3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d80b84c0 0x7fa6d80b8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6dcb3e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d80cec70 con 0x7fa6c0003b90 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6dcb3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d80ce750 con 0x7fa6d80cc570 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6dcb3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d80cedf0 con 0x7fa6d80b84c0 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6d81552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6d81552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35620/0 (socket says 172.21.15.27:35620) 2024-03-06T07:29:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d659c640 1 -- 172.21.15.27:0/1313130742 learned_addr learned my addr 172.21.15.27:0/1313130742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d6d9d640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d80b84c0 0x7fa6d80b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d5d9b640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d80cc570 0x7fa6d80cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d659c640 1 -- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d80b84c0 msgr2=0x7fa6d80b8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d659c640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d80b84c0 0x7fa6d80b8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d659c640 1 -- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d80cc570 msgr2=0x7fa6d80cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d659c640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d80cc570 0x7fa6d80cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d659c640 1 -- 172.21.15.27:0/1313130742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d815e530 con 0x7fa6c0003b90 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d5d9b640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d80cc570 0x7fa6d80cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6d659c640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6d81552c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8037040 tx=0x7fa6c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/1313130742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8042020 con 0x7fa6c0003b90 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/1313130742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c803a040 con 0x7fa6c0003b90 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/1313130742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8039070 con 0x7fa6c0003b90 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d815e810 con 0x7fa6c0003b90 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.603+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d815edd0 con 0x7fa6c0003b90 2024-03-06T07:29:41.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.604+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d80b8e80 con 0x7fa6c0003b90 2024-03-06T07:29:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.605+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/1313130742 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6c8045020 con 0x7fa6c0003b90 2024-03-06T07:29:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.605+0000 7fa6c6ffd640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b806b500 0x7fa6b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.605+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/1313130742 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa6c80ab710 con 0x7fa6c0003b90 2024-03-06T07:29:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.605+0000 7fa6d5d9b640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b806b500 0x7fa6b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.606+0000 7fa6d5d9b640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b806b500 0x7fa6b806d9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa6d81f7020 tx=0x7fa6cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.609+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/1313130742 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6c804c070 con 0x7fa6c0003b90 2024-03-06T07:29:41.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.832+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6d8067ff0 con 0x7fa6c0003b90 2024-03-06T07:29:41.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.832+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/1313130742 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fa6c807a4a0 con 0x7fa6c0003b90 2024-03-06T07:29:41.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:41.834 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:41.834 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:41.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.834+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b806b500 msgr2=0x7fa6b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.834+0000 7fa6dcb3e640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b806b500 0x7fa6b806d9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa6d81f7020 tx=0x7fa6cc009040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.834+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 msgr2=0x7fa6d81552c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.834+0000 7fa6dcb3e640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6d81552c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8037040 tx=0x7fa6c8004730 comp rx=0 tx=0).stop 2024-03-06T07:29:41.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.835+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 shutdown_connections 2024-03-06T07:29:41.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.835+0000 7fa6dcb3e640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b806b500 0x7fa6b806d9c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.835+0000 7fa6dcb3e640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6d80b84c0 0x7fa6d80b8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.835+0000 7fa6dcb3e640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6d80cc570 0x7fa6d80cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.835+0000 7fa6dcb3e640 1 --2- 172.21.15.27:0/1313130742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c0003b90 0x7fa6d81552c0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.835+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 >> 172.21.15.27:0/1313130742 conn(0x7fa6d8071960 msgr2=0x7fa6d80b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.835+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 shutdown_connections 2024-03-06T07:29:41.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:41.835+0000 7fa6dcb3e640 1 -- 172.21.15.27:0/1313130742 wait complete. 2024-03-06T07:29:41.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:29:41.970 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:29:41.971 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:29:41.971 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:29:46.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:47.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.077+0000 7fbc461b6640 1 Processor -- start 2024-03-06T07:29:47.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.077+0000 7fbc461b6640 1 -- start start 2024-03-06T07:29:47.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.078+0000 7fbc461b6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc400b95a0 0x7fbc400b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.078+0000 7fbc461b6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc400cea30 con 0x7fbc400b9a10 2024-03-06T07:29:47.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.078+0000 7fbc461b6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc400cf3c0 con 0x7fbc4015df00 2024-03-06T07:29:47.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.078+0000 7fbc461b6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc400cf0e0 con 0x7fbc400b95a0 2024-03-06T07:29:47.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.078+0000 7fbc3effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc400b95a0 0x7fbc400b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.078+0000 7fbc3ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbc400b9a10 0x7fbc400d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39942/0 (socket says 172.21.15.27:39942) 2024-03-06T07:29:47.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.078+0000 7fbc3ffff640 1 -- 172.21.15.27:0/438761479 learned_addr learned my addr 172.21.15.27:0/438761479 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:47.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.079+0000 7fbc3effd640 1 -- 172.21.15.27:0/438761479 >> v1:172.21.15.69:6789/0 conn(0x7fbc4015df00 legacy=0x7fbc400d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.079+0000 7fbc3effd640 1 -- 172.21.15.27:0/438761479 >> v1:172.21.15.27:6789/0 conn(0x7fbc400b9a10 legacy=0x7fbc400d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.079+0000 7fbc3e7fc640 1 -- 172.21.15.27:0/438761479 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706961296 0 0) 0x7fbc400cea30 con 0x7fbc400b9a10 2024-03-06T07:29:47.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.079+0000 7fbc3effd640 1 -- 172.21.15.27:0/438761479 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc400ce470 con 0x7fbc400b95a0 2024-03-06T07:29:47.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.079+0000 7fbc3effd640 1 --2- 172.21.15.27:0/438761479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b95a0 0x7fbc400b5700 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fbc400cf590 tx=0x7fbc2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20efc613b7abe138 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc3e7fc640 1 -- 172.21.15.27:0/438761479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28035020 con 0x7fbc400b95a0 2024-03-06T07:29:47.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc3e7fc640 1 -- 172.21.15.27:0/438761479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc28031830 con 0x7fbc400b95a0 2024-03-06T07:29:47.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc3e7fc640 1 -- 172.21.15.27:0/438761479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28031c00 con 0x7fbc400b95a0 2024-03-06T07:29:47.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc461b6640 1 -- 172.21.15.27:0/438761479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b95a0 msgr2=0x7fbc400b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc461b6640 1 --2- 172.21.15.27:0/438761479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b95a0 0x7fbc400b5700 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fbc400cf590 tx=0x7fbc2802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc461b6640 1 -- 172.21.15.27:0/438761479 shutdown_connections 2024-03-06T07:29:47.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc461b6640 1 --2- 172.21.15.27:0/438761479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b95a0 0x7fbc400b5700 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc461b6640 1 -- 172.21.15.27:0/438761479 >> 172.21.15.27:0/438761479 conn(0x7fbc40071960 msgr2=0x7fbc40071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc461b6640 1 -- 172.21.15.27:0/438761479 shutdown_connections 2024-03-06T07:29:47.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.080+0000 7fbc461b6640 1 -- 172.21.15.27:0/438761479 wait complete. 2024-03-06T07:29:47.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.081+0000 7fbc461b6640 1 Processor -- start 2024-03-06T07:29:47.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc461b6640 1 -- start start 2024-03-06T07:29:47.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc461b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc400b95a0 0x7fbc400cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc461b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b9a10 0x7fbc400c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc461b6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc4015df00 0x7fbc400d2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc461b6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc400cedf0 con 0x7fbc400b9a10 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc461b6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc400cec70 con 0x7fbc4015df00 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc461b6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc400b5c40 con 0x7fbc400b95a0 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b9a10 0x7fbc400c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc3effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b9a10 0x7fbc400c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50954/0 (socket says 172.21.15.27:50954) 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.082+0000 7fbc3effd640 1 -- 172.21.15.27:0/2113495487 learned_addr learned my addr 172.21.15.27:0/2113495487 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc3ffff640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc4015df00 0x7fbc400d2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc3f7fe640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc400b95a0 0x7fbc400cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc3effd640 1 -- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc400b95a0 msgr2=0x7fbc400cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc3effd640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc400b95a0 0x7fbc400cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc3effd640 1 -- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc4015df00 msgr2=0x7fbc400d2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc3effd640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc4015df00 0x7fbc400d2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc3effd640 1 -- 172.21.15.27:0/2113495487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc400c7d00 con 0x7fbc400b9a10 2024-03-06T07:29:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc3effd640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b9a10 0x7fbc400c66a0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009890 tx=0x7fbc280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc37fff640 1 -- 172.21.15.27:0/2113495487 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28004230 con 0x7fbc400b9a10 2024-03-06T07:29:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc37fff640 1 -- 172.21.15.27:0/2113495487 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc280043d0 con 0x7fbc400b9a10 2024-03-06T07:29:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc400c7fe0 con 0x7fbc400b9a10 2024-03-06T07:29:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc37fff640 1 -- 172.21.15.27:0/2113495487 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2803cad0 con 0x7fbc400b9a10 2024-03-06T07:29:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.083+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc400c85a0 con 0x7fbc400b9a10 2024-03-06T07:29:47.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.084+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc40067ff0 con 0x7fbc400b9a10 2024-03-06T07:29:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.085+0000 7fbc37fff640 1 -- 172.21.15.27:0/2113495487 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc28038020 con 0x7fbc400b9a10 2024-03-06T07:29:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.085+0000 7fbc37fff640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.085+0000 7fbc37fff640 1 -- 172.21.15.27:0/2113495487 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fbc280ac5c0 con 0x7fbc400b9a10 2024-03-06T07:29:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.085+0000 7fbc3f7fe640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.086+0000 7fbc3f7fe640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc2006b4d0 0x7fbc2006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fbc30006fd0 tx=0x7fbc30008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.090+0000 7fbc37fff640 1 -- 172.21.15.27:0/2113495487 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc28041070 con 0x7fbc400b9a10 2024-03-06T07:29:47.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.308+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc400d4110 con 0x7fbc400b9a10 2024-03-06T07:29:47.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.309+0000 7fbc37fff640 1 -- 172.21.15.27:0/2113495487 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fbc280aae20 con 0x7fbc400b9a10 2024-03-06T07:29:47.310 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:47.310 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:47.311 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:47.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.310+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc2006b4d0 msgr2=0x7fbc2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.310+0000 7fbc461b6640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc2006b4d0 0x7fbc2006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fbc30006fd0 tx=0x7fbc30008040 comp rx=0 tx=0).stop 2024-03-06T07:29:47.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b9a10 msgr2=0x7fbc400c66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b9a10 0x7fbc400c66a0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009890 tx=0x7fbc280046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 shutdown_connections 2024-03-06T07:29:47.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc4015df00 0x7fbc400d2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400b9a10 0x7fbc400c66a0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 --2- 172.21.15.27:0/2113495487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc400b95a0 0x7fbc400cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 >> 172.21.15.27:0/2113495487 conn(0x7fbc40071960 msgr2=0x7fbc4015ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 shutdown_connections 2024-03-06T07:29:47.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.311+0000 7fbc461b6640 1 -- 172.21.15.27:0/2113495487 wait complete. 2024-03-06T07:29:47.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:29:47.447 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:29:47.447 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:29:47.448 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.552+0000 7f237afd7640 1 Processor -- start 2024-03-06T07:29:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.553+0000 7f237afd7640 1 -- start start 2024-03-06T07:29:47.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.553+0000 7f237afd7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 0x7f23740b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.553+0000 7f237afd7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23740d3400 0x7f2374071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.553+0000 7f237afd7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2374071e60 con 0x7f23740b2b50 2024-03-06T07:29:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.553+0000 7f237afd7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23740d2f70 con 0x7f23740d3400 2024-03-06T07:29:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.553+0000 7f237afd7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23740d4be0 con 0x7f2374071540 2024-03-06T07:29:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.553+0000 7f2373fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23740d3400 0x7f2374071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.553+0000 7f2379574640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f23740b2b50 0x7f23740d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39946/0 (socket says 172.21.15.27:39946) 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.553+0000 7f2379574640 1 -- 172.21.15.27:0/4202440931 learned_addr learned my addr 172.21.15.27:0/4202440931 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.554+0000 7f2378d73640 1 --2- 172.21.15.27:0/4202440931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 0x7f23740b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.554+0000 7f2373fff640 1 -- 172.21.15.27:0/4202440931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 msgr2=0x7f23740b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.554+0000 7f2373fff640 1 --2- 172.21.15.27:0/4202440931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 0x7f23740b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.554+0000 7f2373fff640 1 -- 172.21.15.27:0/4202440931 >> v1:172.21.15.27:6789/0 conn(0x7f23740b2b50 legacy=0x7f23740d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.554+0000 7f2373fff640 1 -- 172.21.15.27:0/4202440931 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23740ce610 con 0x7f23740d3400 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.554+0000 7f23737fe640 1 -- 172.21.15.27:0/4202440931 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466575736 0 0) 0x7f2374071e60 con 0x7f23740b2b50 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.555+0000 7f2373fff640 1 --2- 172.21.15.27:0/4202440931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740d3400 0x7f2374071920 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f235c0097a0 tx=0x7f235c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951229836280aef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.555+0000 7f23737fe640 1 -- 172.21.15.27:0/4202440931 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c035020 con 0x7f23740d3400 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.555+0000 7f23737fe640 1 -- 172.21.15.27:0/4202440931 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c031830 con 0x7f23740d3400 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.555+0000 7f23737fe640 1 -- 172.21.15.27:0/4202440931 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c031c00 con 0x7f23740d3400 2024-03-06T07:29:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.556+0000 7f237afd7640 1 -- 172.21.15.27:0/4202440931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740d3400 msgr2=0x7f2374071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.556+0000 7f237afd7640 1 --2- 172.21.15.27:0/4202440931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740d3400 0x7f2374071920 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f235c0097a0 tx=0x7f235c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.556+0000 7f237afd7640 1 -- 172.21.15.27:0/4202440931 shutdown_connections 2024-03-06T07:29:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.556+0000 7f237afd7640 1 --2- 172.21.15.27:0/4202440931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740d3400 0x7f2374071920 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.556+0000 7f237afd7640 1 --2- 172.21.15.27:0/4202440931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 0x7f23740b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.556+0000 7f237afd7640 1 -- 172.21.15.27:0/4202440931 >> 172.21.15.27:0/4202440931 conn(0x7f23740d3ca0 msgr2=0x7f23740d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.556+0000 7f237afd7640 1 -- 172.21.15.27:0/4202440931 shutdown_connections 2024-03-06T07:29:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.556+0000 7f237afd7640 1 -- 172.21.15.27:0/4202440931 wait complete. 2024-03-06T07:29:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.557+0000 7f237afd7640 1 Processor -- start 2024-03-06T07:29:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.557+0000 7f237afd7640 1 -- start start 2024-03-06T07:29:47.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f237afd7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 0x7f23740b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f237afd7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740b2b50 0x7f23740c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f237afd7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23740d3400 0x7f2374151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f237afd7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23740ce8f0 con 0x7f23740b2b50 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f237afd7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23740d2c00 con 0x7f23740d3400 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f237afd7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237415e370 con 0x7f2374071540 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2373fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740b2b50 0x7f23740c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2373fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740b2b50 0x7f23740c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50974/0 (socket says 172.21.15.27:50974) 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2373fff640 1 -- 172.21.15.27:0/1350784021 learned_addr learned my addr 172.21.15.27:0/1350784021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2378d73640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 0x7f23740b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2379574640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23740d3400 0x7f2374151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2373fff640 1 -- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 msgr2=0x7f23740b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2373fff640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 0x7f23740b5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2373fff640 1 -- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23740d3400 msgr2=0x7f2374151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2373fff640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23740d3400 0x7f2374151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.558+0000 7f2373fff640 1 -- 172.21.15.27:0/1350784021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2374151650 con 0x7f23740b2b50 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.559+0000 7f2373fff640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740b2b50 0x7f23740c8d90 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f235c0096f0 tx=0x7f235c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.559+0000 7f23717fa640 1 -- 172.21.15.27:0/1350784021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c040070 con 0x7f23740b2b50 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.559+0000 7f23717fa640 1 -- 172.21.15.27:0/1350784021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c03a910 con 0x7f23740b2b50 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.559+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23740c9500 con 0x7f23740b2b50 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.559+0000 7f23717fa640 1 -- 172.21.15.27:0/1350784021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c035040 con 0x7f23740b2b50 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.559+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23740c9ac0 con 0x7f23740b2b50 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.560+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2374068700 con 0x7f23740b2b50 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.560+0000 7f23717fa640 1 -- 172.21.15.27:0/1350784021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f235c04b060 con 0x7f23740b2b50 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.561+0000 7f23717fa640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f235406b4d0 0x7f235406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.561+0000 7f23717fa640 1 -- 172.21.15.27:0/1350784021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f235c0aba90 con 0x7f23740b2b50 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.561+0000 7f2378d73640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f235406b4d0 0x7f235406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.561+0000 7f2378d73640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f235406b4d0 0x7f235406d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f236c007480 tx=0x7f236c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.570+0000 7f23717fa640 1 -- 172.21.15.27:0/1350784021 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2374151650 con 0x7f23740b2b50 2024-03-06T07:29:47.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.795+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23741612a0 con 0x7f23740b2b50 2024-03-06T07:29:47.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.795+0000 7f23717fa640 1 -- 172.21.15.27:0/1350784021 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f235c07a820 con 0x7f23740b2b50 2024-03-06T07:29:47.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:47.798 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:47.798 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:47.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f235406b4d0 msgr2=0x7f235406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f235406b4d0 0x7f235406d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f236c007480 tx=0x7f236c00b040 comp rx=0 tx=0).stop 2024-03-06T07:29:47.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740b2b50 msgr2=0x7f23740c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740b2b50 0x7f23740c8d90 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f235c0096f0 tx=0x7f235c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 shutdown_connections 2024-03-06T07:29:47.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f235406b4d0 0x7f235406d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23740d3400 0x7f2374151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23740b2b50 0x7f23740c8d90 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 --2- 172.21.15.27:0/1350784021 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2374071540 0x7f23740b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 >> 172.21.15.27:0/1350784021 conn(0x7f23740d3ca0 msgr2=0x7f2374071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 shutdown_connections 2024-03-06T07:29:47.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:47.797+0000 7f237afd7640 1 -- 172.21.15.27:0/1350784021 wait complete. 2024-03-06T07:29:47.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:29:47.932 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:29:47.932 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:29:47.932 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:48.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.039+0000 7f74c3ed4640 1 Processor -- start 2024-03-06T07:29:48.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.039+0000 7f74c3ed4640 1 -- start start 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.039+0000 7f74c3ed4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc071540 0x7f74bc0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.039+0000 7f74c3ed4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74bc0d3400 0x7f74bc071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c3ed4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74bc071e60 con 0x7f74bc0b2b50 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c3ed4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74bc0d2f70 con 0x7f74bc0d3400 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c3ed4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0d4be0 con 0x7f74bc071540 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c146f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74bc0d3400 0x7f74bc071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c2471640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f74bc0b2b50 0x7f74bc0d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39952/0 (socket says 172.21.15.27:39952) 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c2471640 1 -- 172.21.15.27:0/3674643007 learned_addr learned my addr 172.21.15.27:0/3674643007 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c1c70640 1 --2- 172.21.15.27:0/3674643007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc071540 0x7f74bc0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c146f640 1 -- 172.21.15.27:0/3674643007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc071540 msgr2=0x7f74bc0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c146f640 1 --2- 172.21.15.27:0/3674643007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc071540 0x7f74bc0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c146f640 1 -- 172.21.15.27:0/3674643007 >> v1:172.21.15.27:6789/0 conn(0x7f74bc0b2b50 legacy=0x7f74bc0d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c146f640 1 -- 172.21.15.27:0/3674643007 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc0ce610 con 0x7f74bc0d3400 2024-03-06T07:29:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.040+0000 7f74c0c6e640 1 -- 172.21.15.27:0/3674643007 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641989349 0 0) 0x7f74bc071e60 con 0x7f74bc0b2b50 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.041+0000 7f74c146f640 1 --2- 172.21.15.27:0/3674643007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0d3400 0x7f74bc071920 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0097a0 tx=0x7f74ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62de1380386806e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.041+0000 7f74c0c6e640 1 -- 172.21.15.27:0/3674643007 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac035020 con 0x7f74bc0d3400 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.041+0000 7f74c0c6e640 1 -- 172.21.15.27:0/3674643007 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac031830 con 0x7f74bc0d3400 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.041+0000 7f74c0c6e640 1 -- 172.21.15.27:0/3674643007 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac031c00 con 0x7f74bc0d3400 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.042+0000 7f74c3ed4640 1 -- 172.21.15.27:0/3674643007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0d3400 msgr2=0x7f74bc071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.042+0000 7f74c3ed4640 1 --2- 172.21.15.27:0/3674643007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0d3400 0x7f74bc071920 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0097a0 tx=0x7f74ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.042+0000 7f74c3ed4640 1 -- 172.21.15.27:0/3674643007 shutdown_connections 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.042+0000 7f74c3ed4640 1 --2- 172.21.15.27:0/3674643007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0d3400 0x7f74bc071920 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.042+0000 7f74c3ed4640 1 --2- 172.21.15.27:0/3674643007 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc071540 0x7f74bc0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.042+0000 7f74c3ed4640 1 -- 172.21.15.27:0/3674643007 >> 172.21.15.27:0/3674643007 conn(0x7f74bc0d3ca0 msgr2=0x7f74bc0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.042+0000 7f74c3ed4640 1 -- 172.21.15.27:0/3674643007 shutdown_connections 2024-03-06T07:29:48.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.042+0000 7f74c3ed4640 1 -- 172.21.15.27:0/3674643007 wait complete. 2024-03-06T07:29:48.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.044+0000 7f74c3ed4640 1 Processor -- start 2024-03-06T07:29:48.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.044+0000 7f74c3ed4640 1 -- start start 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c3ed4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74bc071540 0x7f74bc0b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c3ed4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0b2b50 0x7f74bc0c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c3ed4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc0d3400 0x7f74bc151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c3ed4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0ce8f0 con 0x7f74bc0b2b50 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c3ed4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0d2c00 con 0x7f74bc071540 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c3ed4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc15e370 con 0x7f74bc0d3400 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c146f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0b2b50 0x7f74bc0c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c146f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0b2b50 0x7f74bc0c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50998/0 (socket says 172.21.15.27:50998) 2024-03-06T07:29:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c146f640 1 -- 172.21.15.27:0/4223506529 learned_addr learned my addr 172.21.15.27:0/4223506529 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c2471640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc0d3400 0x7f74bc151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c1c70640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74bc071540 0x7f74bc0b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c146f640 1 -- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc0d3400 msgr2=0x7f74bc151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c146f640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc0d3400 0x7f74bc151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c146f640 1 -- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74bc071540 msgr2=0x7f74bc0b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c146f640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74bc071540 0x7f74bc0b5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c146f640 1 -- 172.21.15.27:0/4223506529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc151650 con 0x7f74bc0b2b50 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c1c70640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74bc071540 0x7f74bc0b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c2471640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc0d3400 0x7f74bc151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.045+0000 7f74c146f640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0b2b50 0x7f74bc0c8d90 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0096f0 tx=0x7f74ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.046+0000 7f74b27fc640 1 -- 172.21.15.27:0/4223506529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac040070 con 0x7f74bc0b2b50 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.046+0000 7f74b27fc640 1 -- 172.21.15.27:0/4223506529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac03a910 con 0x7f74bc0b2b50 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.046+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74bc0c9500 con 0x7f74bc0b2b50 2024-03-06T07:29:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.046+0000 7f74b27fc640 1 -- 172.21.15.27:0/4223506529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac035040 con 0x7f74bc0b2b50 2024-03-06T07:29:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.046+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74bc0c9ac0 con 0x7f74bc0b2b50 2024-03-06T07:29:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.047+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74bc068700 con 0x7f74bc0b2b50 2024-03-06T07:29:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.047+0000 7f74b27fc640 1 -- 172.21.15.27:0/4223506529 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74ac04b060 con 0x7f74bc0b2b50 2024-03-06T07:29:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.048+0000 7f74b27fc640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f74a006b4d0 0x7f74a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.048+0000 7f74b27fc640 1 -- 172.21.15.27:0/4223506529 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f74ac0abb00 con 0x7f74bc0b2b50 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.048+0000 7f74c1c70640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f74a006b4d0 0x7f74a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.049+0000 7f74c1c70640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f74a006b4d0 0x7f74a006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f74b8006fd0 tx=0x7f74b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.052+0000 7f74b27fc640 1 -- 172.21.15.27:0/4223506529 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74bc151650 con 0x7f74bc0b2b50 2024-03-06T07:29:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.272+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74bc1612a0 con 0x7f74bc0b2b50 2024-03-06T07:29:48.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.272+0000 7f74b27fc640 1 -- 172.21.15.27:0/4223506529 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f74ac031800 con 0x7f74bc0b2b50 2024-03-06T07:29:48.274 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:48.274 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:48.274 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f74a006b4d0 msgr2=0x7f74a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f74a006b4d0 0x7f74a006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f74b8006fd0 tx=0x7f74b8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0b2b50 msgr2=0x7f74bc0c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0b2b50 0x7f74bc0c8d90 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0096f0 tx=0x7f74ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 shutdown_connections 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f74a006b4d0 0x7f74a006d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f74bc0d3400 0x7f74bc151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74bc0b2b50 0x7f74bc0c8d90 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 --2- 172.21.15.27:0/4223506529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74bc071540 0x7f74bc0b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 >> 172.21.15.27:0/4223506529 conn(0x7f74bc0d3ca0 msgr2=0x7f74bc071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 shutdown_connections 2024-03-06T07:29:48.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.274+0000 7f74c3ed4640 1 -- 172.21.15.27:0/4223506529 wait complete. 2024-03-06T07:29:48.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:29:48.422 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:29:48.423 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:48.423 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:29:48.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.464+0000 7fbe8eb2ef40 1 Processor -- start 2024-03-06T07:29:48.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.464+0000 7fbe8eb2ef40 1 -- start start 2024-03-06T07:29:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8eb2ef40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a1a0777590 0x55a1a0804180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8eb2ef40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0792770 0x55a1a0822ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8eb2ef40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a1a081a280 0x55a1a081ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8eb2ef40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a07932d0 con 0x55a1a0792770 2024-03-06T07:29:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8eb2ef40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a06f2b70 con 0x55a1a0777590 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8eb2ef40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a1a0793450 con 0x55a1a081a280 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8be2e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0792770 0x55a1a0822ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8be2e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0792770 0x55a1a0822ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51026/0 (socket says 172.21.15.27:51026) 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8be2e640 1 -- 172.21.15.27:0/1294876201 learned_addr learned my addr 172.21.15.27:0/1294876201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8c62f640 1 --2- 172.21.15.27:0/1294876201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a1a0777590 0x55a1a0804180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8ce30640 1 --2- 172.21.15.27:0/1294876201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a1a081a280 0x55a1a081ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8be2e640 1 -- 172.21.15.27:0/1294876201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a1a081a280 msgr2=0x55a1a081ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8be2e640 1 --2- 172.21.15.27:0/1294876201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a1a081a280 0x55a1a081ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8be2e640 1 -- 172.21.15.27:0/1294876201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a1a0777590 msgr2=0x55a1a0804180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8be2e640 1 --2- 172.21.15.27:0/1294876201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a1a0777590 0x55a1a0804180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8be2e640 1 -- 172.21.15.27:0/1294876201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1a08046c0 con 0x55a1a0792770 2024-03-06T07:29:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8be2e640 1 --2- 172.21.15.27:0/1294876201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0792770 0x55a1a0822ad0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fbe80000c00 tx=0x7fbe8004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984de558e2e33e7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.466+0000 7fbe8ae2c640 1 -- 172.21.15.27:0/1294876201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80057070 con 0x55a1a0792770 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8ae2c640 1 -- 172.21.15.27:0/1294876201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe80061070 con 0x55a1a0792770 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8ae2c640 1 -- 172.21.15.27:0/1294876201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80052b50 con 0x55a1a0792770 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/1294876201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0792770 msgr2=0x55a1a0822ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8eb2ef40 1 --2- 172.21.15.27:0/1294876201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0792770 0x55a1a0822ad0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fbe80000c00 tx=0x7fbe8004c000 comp rx=0 tx=0).stop 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/1294876201 shutdown_connections 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8eb2ef40 1 --2- 172.21.15.27:0/1294876201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a1a081a280 0x55a1a081ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8eb2ef40 1 --2- 172.21.15.27:0/1294876201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0792770 0x55a1a0822ad0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8eb2ef40 1 --2- 172.21.15.27:0/1294876201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a1a0777590 0x55a1a0804180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/1294876201 >> 172.21.15.27:0/1294876201 conn(0x55a1a0775a60 msgr2=0x55a1a08050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/1294876201 shutdown_connections 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.467+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/1294876201 wait complete. 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.468+0000 7fbe8eb2ef40 1 Processor -- start 2024-03-06T07:29:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.468+0000 7fbe8eb2ef40 1 -- start start 2024-03-06T07:29:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.468+0000 7fbe8eb2ef40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0777590 0x55a1a0819560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.468+0000 7fbe8eb2ef40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a1a0792770 0x55a1a074d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.468+0000 7fbe8eb2ef40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a1a081a280 0x55a1a074de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.468+0000 7fbe8eb2ef40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a07fc700 con 0x55a1a0777590 2024-03-06T07:29:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.468+0000 7fbe8eb2ef40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a0823010 con 0x55a1a0792770 2024-03-06T07:29:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.468+0000 7fbe8eb2ef40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a1a073a7d0 con 0x55a1a081a280 2024-03-06T07:29:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.468+0000 7fbe8c62f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0777590 0x55a1a0819560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8c62f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0777590 0x55a1a0819560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51028/0 (socket says 172.21.15.27:51028) 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8c62f640 1 -- 172.21.15.27:0/3324465680 learned_addr learned my addr 172.21.15.27:0/3324465680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8be2e640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a1a0792770 0x55a1a074d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8ce30640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a1a081a280 0x55a1a074de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8c62f640 1 -- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a1a081a280 msgr2=0x55a1a074de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8c62f640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55a1a081a280 0x55a1a074de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8c62f640 1 -- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a1a0792770 msgr2=0x55a1a074d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8c62f640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a1a0792770 0x55a1a074d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8c62f640 1 -- 172.21.15.27:0/3324465680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1a0819aa0 con 0x55a1a0777590 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8c62f640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0777590 0x55a1a0819560 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c00df70 tx=0x7fbe7c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe89e2a640 1 -- 172.21.15.27:0/3324465680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0032d0 con 0x55a1a0777590 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe89e2a640 1 -- 172.21.15.27:0/3324465680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c016070 con 0x55a1a0777590 2024-03-06T07:29:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a1a0819c90 con 0x55a1a0777590 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe89e2a640 1 -- 172.21.15.27:0/3324465680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0116a0 con 0x55a1a0777590 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.469+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1a074d0c0 con 0x55a1a0777590 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.471+0000 7fbe89e2a640 1 -- 172.21.15.27:0/3324465680 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe7c011840 con 0x55a1a0777590 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.471+0000 7fbe89e2a640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe78069eb0 0x7fbe7806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.471+0000 7fbe89e2a640 1 -- 172.21.15.27:0/3324465680 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fbe7c088e00 con 0x55a1a0777590 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.471+0000 7fbe8be2e640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe78069eb0 0x7fbe7806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.471+0000 7fbe8eb2ef40 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a1a0793e90 0x55a1a08224a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.471+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120) v9 -- 0x55a1a081f4a0 con 0x55a1a0793e90 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.471+0000 7fbe8ce30640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a1a0793e90 0x55a1a08224a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8ce30640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a1a0793e90 0x55a1a08224a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8be2e640 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe78069eb0 0x7fbe7806c370 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fbe80060000 tx=0x7fbe8004f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8ce30640 1 -- 172.21.15.27:0/3324465680 <== osd.0 v2:172.21.15.27:6800/3926946944 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fbe840029d0 con 0x55a1a0793e90 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a1a0793e90 msgr2=0x55a1a08224a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x55a1a0793e90 0x55a1a08224a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe78069eb0 msgr2=0x7fbe7806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbe78069eb0 0x7fbe7806c370 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fbe80060000 tx=0x7fbe8004f000 comp rx=0 tx=0).stop 2024-03-06T07:29:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0777590 msgr2=0x55a1a0819560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 --2- 172.21.15.27:0/3324465680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a1a0777590 0x55a1a0819560 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c00df70 tx=0x7fbe7c00b5b0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8ce30640 1 -- 172.21.15.27:0/3324465680 reap_dead start 2024-03-06T07:29:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 shutdown_connections 2024-03-06T07:29:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 >> 172.21.15.27:0/3324465680 conn(0x55a1a0775a60 msgr2=0x55a1a08050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 shutdown_connections 2024-03-06T07:29:48.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.472+0000 7fbe8eb2ef40 1 -- 172.21.15.27:0/3324465680 wait complete. 2024-03-06T07:29:48.479 INFO:teuthology.orchestra.run.smithi027.stdout:W�file_ddir_2* 2024-03-06T07:29:48.479 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:29:48.565 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:29:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511627777, 2024-03-06T07:29:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2024-03-06T07:29:48.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1099511628278, 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "file_d", 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 25 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1, 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "dir_2", 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 42 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 17, 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [] 2024-03-06T07:29:48.566 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:29:48.567 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:48.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:48.568 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:29:48.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:48.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.774+0000 7ff796ff6640 1 Processor -- start 2024-03-06T07:29:48.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.774+0000 7ff796ff6640 1 -- start start 2024-03-06T07:29:48.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.775+0000 7ff796ff6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff7900b0e90 0x7ff79006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.775+0000 7ff796ff6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900ae9b0 0x7ff7900d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.775+0000 7ff796ff6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7900cee90 con 0x7ff79006b5c0 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.775+0000 7ff796ff6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7900cfa00 con 0x7ff7900b0e90 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.775+0000 7ff796ff6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900cfc50 con 0x7ff7900ae9b0 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.775+0000 7ff795593640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900ae9b0 0x7ff7900d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff78ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff7900b0e90 0x7ff79006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff795593640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900ae9b0 0x7ff7900d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:36406/0 (socket says 172.21.15.27:36406) 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff78ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff7900b0e90 0x7ff79006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:49330/0 (socket says 172.21.15.27:49330) 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff795593640 1 -- 172.21.15.27:0/1013435613 learned_addr learned my addr 172.21.15.27:0/1013435613 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff794d92640 1 --1- 172.21.15.27:0/1013435613 >> v1:172.21.15.69:6789/0 conn(0x7ff79006b5c0 0x7ff7900b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:58224/0 (socket says 172.21.15.27:58224) 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff78f7fe640 1 -- 172.21.15.27:0/1013435613 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293680132 0 0) 0x7ff7900cee90 con 0x7ff79006b5c0 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff78f7fe640 1 -- 172.21.15.27:0/1013435613 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff778003520 con 0x7ff79006b5c0 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff78ffff640 1 -- 172.21.15.27:0/1013435613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900ae9b0 msgr2=0x7ff7900d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff78ffff640 1 --2- 172.21.15.27:0/1013435613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900ae9b0 0x7ff7900d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff78ffff640 1 -- 172.21.15.27:0/1013435613 >> v1:172.21.15.69:6789/0 conn(0x7ff79006b5c0 legacy=0x7ff7900b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.776+0000 7ff78ffff640 1 -- 172.21.15.27:0/1013435613 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7900cff30 con 0x7ff7900b0e90 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.777+0000 7ff78ffff640 1 --2- 172.21.15.27:0/1013435613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7900b0e90 0x7ff79006b9a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff7840097a0 tx=0x7ff78402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbb6bb9364d21496 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.777+0000 7ff78f7fe640 1 -- 172.21.15.27:0/1013435613 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7840315a0 con 0x7ff7900b0e90 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.777+0000 7ff78f7fe640 1 -- 172.21.15.27:0/1013435613 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff784037070 con 0x7ff7900b0e90 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.777+0000 7ff78f7fe640 1 -- 172.21.15.27:0/1013435613 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784031940 con 0x7ff7900b0e90 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.778+0000 7ff796ff6640 1 -- 172.21.15.27:0/1013435613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7900b0e90 msgr2=0x7ff79006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.778+0000 7ff796ff6640 1 --2- 172.21.15.27:0/1013435613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7900b0e90 0x7ff79006b9a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff7840097a0 tx=0x7ff78402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.778+0000 7ff796ff6640 1 -- 172.21.15.27:0/1013435613 shutdown_connections 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.778+0000 7ff796ff6640 1 --2- 172.21.15.27:0/1013435613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7900b0e90 0x7ff79006b9a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.778+0000 7ff796ff6640 1 --2- 172.21.15.27:0/1013435613 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900ae9b0 0x7ff7900d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.778+0000 7ff796ff6640 1 -- 172.21.15.27:0/1013435613 >> 172.21.15.27:0/1013435613 conn(0x7ff7900b36d0 msgr2=0x7ff7900b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.778+0000 7ff796ff6640 1 -- 172.21.15.27:0/1013435613 shutdown_connections 2024-03-06T07:29:48.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.778+0000 7ff796ff6640 1 -- 172.21.15.27:0/1013435613 wait complete. 2024-03-06T07:29:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.779+0000 7ff796ff6640 1 Processor -- start 2024-03-06T07:29:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.779+0000 7ff796ff6640 1 -- start start 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff796ff6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79006b5c0 0x7ff7901de400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff796ff6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7900ae9b0 0x7ff7900c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff796ff6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900b0e90 0x7ff7900c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff796ff6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79006c910 con 0x7ff79006b5c0 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff796ff6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79006c790 con 0x7ff7900ae9b0 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff796ff6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900aee00 con 0x7ff7900b0e90 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff794d92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79006b5c0 0x7ff7901de400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff794d92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79006b5c0 0x7ff7901de400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51032/0 (socket says 172.21.15.27:51032) 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff794d92640 1 -- 172.21.15.27:0/229264057 learned_addr learned my addr 172.21.15.27:0/229264057 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff795593640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900b0e90 0x7ff7900c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff78ffff640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7900ae9b0 0x7ff7900c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff794d92640 1 -- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900b0e90 msgr2=0x7ff7900c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff794d92640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900b0e90 0x7ff7900c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.780+0000 7ff794d92640 1 -- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7900ae9b0 msgr2=0x7ff7900c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.781+0000 7ff794d92640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7900ae9b0 0x7ff7900c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.781+0000 7ff794d92640 1 -- 172.21.15.27:0/229264057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7900c6ab0 con 0x7ff79006b5c0 2024-03-06T07:29:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.781+0000 7ff794d92640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79006b5c0 0x7ff7901de400 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7ff78000c730 tx=0x7ff78000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.783+0000 7ff78d7fa640 1 -- 172.21.15.27:0/229264057 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780017070 con 0x7ff79006b5c0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.783+0000 7ff78d7fa640 1 -- 172.21.15.27:0/229264057 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78000ce40 con 0x7ff79006b5c0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.783+0000 7ff78d7fa640 1 -- 172.21.15.27:0/229264057 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78000a860 con 0x7ff79006b5c0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.783+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7900c6d90 con 0x7ff79006b5c0 2024-03-06T07:29:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.784+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7900c7350 con 0x7ff79006b5c0 2024-03-06T07:29:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.785+0000 7ff78d7fa640 1 -- 172.21.15.27:0/229264057 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff78001a070 con 0x7ff79006b5c0 2024-03-06T07:29:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.785+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7900d16a0 con 0x7ff79006b5c0 2024-03-06T07:29:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.785+0000 7ff78d7fa640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.785+0000 7ff78d7fa640 1 -- 172.21.15.27:0/229264057 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff7800898c0 con 0x7ff79006b5c0 2024-03-06T07:29:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.786+0000 7ff78ffff640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.786+0000 7ff78ffff640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff77c06b4d0 0x7ff77c06d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff784036000 tx=0x7ff78403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:48.790+0000 7ff78d7fa640 1 -- 172.21.15.27:0/229264057 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7900d16a0 con 0x7ff79006b5c0 2024-03-06T07:29:49.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.008+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7901e61c0 con 0x7ff79006b5c0 2024-03-06T07:29:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.008+0000 7ff78d7fa640 1 -- 172.21.15.27:0/229264057 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7ff7901e61c0 con 0x7ff79006b5c0 2024-03-06T07:29:49.010 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:49.010 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:49.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff77c06b4d0 msgr2=0x7ff77c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff77c06b4d0 0x7ff77c06d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff784036000 tx=0x7ff78403d040 comp rx=0 tx=0).stop 2024-03-06T07:29:49.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79006b5c0 msgr2=0x7ff7901de400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79006b5c0 0x7ff7901de400 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7ff78000c730 tx=0x7ff78000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 shutdown_connections 2024-03-06T07:29:49.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff7900b0e90 0x7ff7900c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7900ae9b0 0x7ff7900c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 --2- 172.21.15.27:0/229264057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff79006b5c0 0x7ff7901de400 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 >> 172.21.15.27:0/229264057 conn(0x7ff7900b36d0 msgr2=0x7ff7901e2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 shutdown_connections 2024-03-06T07:29:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:49.010+0000 7ff796ff6640 1 -- 172.21.15.27:0/229264057 wait complete. 2024-03-06T07:29:49.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:49.146 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:29:49.146 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:29:49.146 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:29:49.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:49.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:54.148 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:54.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.254+0000 7f1b79e65640 1 Processor -- start 2024-03-06T07:29:54.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.255+0000 7f1b79e65640 1 -- start start 2024-03-06T07:29:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.255+0000 7f1b79e65640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1b7415df00 0x7f1b740d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.255+0000 7f1b79e65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b740b95a0 0x7f1b740b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.255+0000 7f1b79e65640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b740b9a30 0x7f1b740b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.255+0000 7f1b79e65640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b740cea30 con 0x7f1b740b9a30 2024-03-06T07:29:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.255+0000 7f1b79e65640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b740d2af0 con 0x7f1b7415df00 2024-03-06T07:29:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.255+0000 7f1b79e65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b740cf3c0 con 0x7f1b740b95a0 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.255+0000 7f1b73fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b740b9a30 0x7f1b740b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.256+0000 7f1b73fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b740b9a30 0x7f1b740b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40240/0 (socket says 172.21.15.27:40240) 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.256+0000 7f1b73fff640 1 -- 172.21.15.27:0/1475758170 learned_addr learned my addr 172.21.15.27:0/1475758170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.256+0000 7f1b72ffd640 1 --2- 172.21.15.27:0/1475758170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b740b95a0 0x7f1b740b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.256+0000 7f1b737fe640 1 --2- 172.21.15.27:0/1475758170 >> v2:172.21.15.69:3300/0 conn(0x7f1b7415df00 0x7f1b740d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.256+0000 7f1b73fff640 1 -- 172.21.15.27:0/1475758170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b740b95a0 msgr2=0x7f1b740b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.256+0000 7f1b73fff640 1 --2- 172.21.15.27:0/1475758170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b740b95a0 0x7f1b740b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.256+0000 7f1b73fff640 1 -- 172.21.15.27:0/1475758170 >> v2:172.21.15.69:3300/0 conn(0x7f1b7415df00 msgr2=0x7f1b740d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.256+0000 7f1b73fff640 1 --2- 172.21.15.27:0/1475758170 >> v2:172.21.15.69:3300/0 conn(0x7f1b7415df00 0x7f1b740d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.256+0000 7f1b73fff640 1 -- 172.21.15.27:0/1475758170 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b740cf0e0 con 0x7f1b740b9a30 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b73fff640 1 --2- 172.21.15.27:0/1475758170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b740b9a30 0x7f1b740b5c40 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f1b740d11b0 tx=0x7f1b6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7de78b5fe17d6ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b727fc640 1 -- 172.21.15.27:0/1475758170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c02ca70 con 0x7f1b740b9a30 2024-03-06T07:29:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b727fc640 1 -- 172.21.15.27:0/1475758170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6c02cc10 con 0x7f1b740b9a30 2024-03-06T07:29:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b727fc640 1 -- 172.21.15.27:0/1475758170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c0343f0 con 0x7f1b740b9a30 2024-03-06T07:29:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b79e65640 1 -- 172.21.15.27:0/1475758170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b740b9a30 msgr2=0x7f1b740b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b79e65640 1 --2- 172.21.15.27:0/1475758170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b740b9a30 0x7f1b740b5c40 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f1b740d11b0 tx=0x7f1b6c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b79e65640 1 -- 172.21.15.27:0/1475758170 shutdown_connections 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b79e65640 1 --2- 172.21.15.27:0/1475758170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b740b9a30 0x7f1b740b5c40 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b79e65640 1 --2- 172.21.15.27:0/1475758170 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b740b95a0 0x7f1b740b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b79e65640 1 --2- 172.21.15.27:0/1475758170 >> v2:172.21.15.69:3300/0 conn(0x7f1b7415df00 0x7f1b740d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b79e65640 1 -- 172.21.15.27:0/1475758170 >> 172.21.15.27:0/1475758170 conn(0x7f1b74071960 msgr2=0x7f1b74071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b79e65640 1 -- 172.21.15.27:0/1475758170 shutdown_connections 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.257+0000 7f1b79e65640 1 -- 172.21.15.27:0/1475758170 wait complete. 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.259+0000 7f1b79e65640 1 Processor -- start 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.259+0000 7f1b79e65640 1 -- start start 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.259+0000 7f1b79e65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b740b95a0 0x7f1b741527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.259+0000 7f1b79e65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7415df00 0x7f1b741e7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.259+0000 7f1b79e65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b741e9c60 0x7f1b741ea0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.259+0000 7f1b79e65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b740cec70 con 0x7f1b741e9c60 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.259+0000 7f1b79e65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b740ce750 con 0x7f1b740b95a0 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.259+0000 7f1b79e65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b740cedf0 con 0x7f1b7415df00 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b73fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b741e9c60 0x7f1b741ea0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b73fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b741e9c60 0x7f1b741ea0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40242/0 (socket says 172.21.15.27:40242) 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b73fff640 1 -- 172.21.15.27:0/2032224139 learned_addr learned my addr 172.21.15.27:0/2032224139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b72ffd640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7415df00 0x7f1b741e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b737fe640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b740b95a0 0x7f1b741527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b73fff640 1 -- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7415df00 msgr2=0x7f1b741e7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b73fff640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7415df00 0x7f1b741e7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b73fff640 1 -- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b740b95a0 msgr2=0x7f1b741527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b73fff640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b740b95a0 0x7f1b741527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b73fff640 1 -- 172.21.15.27:0/2032224139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b74153900 con 0x7f1b741e9c60 2024-03-06T07:29:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b73fff640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b741e9c60 0x7f1b741ea0e0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c036040 tx=0x7f1b6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b5bfff640 1 -- 172.21.15.27:0/2032224139 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c043020 con 0x7f1b741e9c60 2024-03-06T07:29:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b5bfff640 1 -- 172.21.15.27:0/2032224139 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6c03e450 con 0x7f1b741e9c60 2024-03-06T07:29:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7415eec0 con 0x7f1b741e9c60 2024-03-06T07:29:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.260+0000 7f1b5bfff640 1 -- 172.21.15.27:0/2032224139 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c009510 con 0x7f1b741e9c60 2024-03-06T07:29:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.261+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b740b3de0 con 0x7f1b741e9c60 2024-03-06T07:29:54.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.262+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b74067ff0 con 0x7f1b741e9c60 2024-03-06T07:29:54.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.263+0000 7f1b5bfff640 1 -- 172.21.15.27:0/2032224139 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b6c046020 con 0x7f1b741e9c60 2024-03-06T07:29:54.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.263+0000 7f1b5bfff640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b6006b4d0 0x7f1b6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.263+0000 7f1b5bfff640 1 -- 172.21.15.27:0/2032224139 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1b6c0aca50 con 0x7f1b741e9c60 2024-03-06T07:29:54.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.263+0000 7f1b737fe640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b6006b4d0 0x7f1b6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.264+0000 7f1b737fe640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b6006b4d0 0x7f1b6006d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006fd0 tx=0x7f1b64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.268+0000 7f1b5bfff640 1 -- 172.21.15.27:0/2032224139 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b6c04d030 con 0x7f1b741e9c60 2024-03-06T07:29:54.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.489+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b740b6e10 con 0x7f1b741e9c60 2024-03-06T07:29:54.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.489+0000 7f1b5bfff640 1 -- 172.21.15.27:0/2032224139 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f1b6c07b130 con 0x7f1b741e9c60 2024-03-06T07:29:54.492 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:54.492 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:54.492 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b6006b4d0 msgr2=0x7f1b6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b6006b4d0 0x7f1b6006d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006fd0 tx=0x7f1b64008040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b741e9c60 msgr2=0x7f1b741ea0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b741e9c60 0x7f1b741ea0e0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c036040 tx=0x7f1b6c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 shutdown_connections 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1b6006b4d0 0x7f1b6006d990 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b741e9c60 0x7f1b741ea0e0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b7415df00 0x7f1b741e7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 --2- 172.21.15.27:0/2032224139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b740b95a0 0x7f1b741527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.491+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 >> 172.21.15.27:0/2032224139 conn(0x7f1b74071960 msgr2=0x7f1b740b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.492+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 shutdown_connections 2024-03-06T07:29:54.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.492+0000 7f1b79e65640 1 -- 172.21.15.27:0/2032224139 wait complete. 2024-03-06T07:29:54.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:29:54.627 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:29:54.627 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:54.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.733+0000 7fced21d7640 1 Processor -- start 2024-03-06T07:29:54.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.733+0000 7fced21d7640 1 -- start start 2024-03-06T07:29:54.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.733+0000 7fced21d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc15df00 0x7fcecc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.733+0000 7fced21d7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcecc0b9a30 0x7fcecc0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.733+0000 7fced21d7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc0cea30 con 0x7fcecc0b95a0 2024-03-06T07:29:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.733+0000 7fced21d7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcecc0d2af0 con 0x7fcecc0b9a30 2024-03-06T07:29:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.733+0000 7fced21d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc0cf3c0 con 0x7fcecc15df00 2024-03-06T07:29:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.734+0000 7fcecbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcecc0b9a30 0x7fcecc0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.734+0000 7fcecbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcecc0b9a30 0x7fcecc0d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40250/0 (socket says 172.21.15.27:40250) 2024-03-06T07:29:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.734+0000 7fcecbfff640 1 -- 172.21.15.27:0/2440488250 learned_addr learned my addr 172.21.15.27:0/2440488250 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.734+0000 7fcecb7fe640 1 --2- 172.21.15.27:0/2440488250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc15df00 0x7fcecc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.734+0000 7fcecbfff640 1 -- 172.21.15.27:0/2440488250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc15df00 msgr2=0x7fcecc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.734+0000 7fcecbfff640 1 --2- 172.21.15.27:0/2440488250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc15df00 0x7fcecc0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.734+0000 7fcecbfff640 1 -- 172.21.15.27:0/2440488250 >> v1:172.21.15.69:6789/0 conn(0x7fcecc0b95a0 legacy=0x7fcecc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.734+0000 7fcecbfff640 1 -- 172.21.15.27:0/2440488250 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc0cf0e0 con 0x7fcecc0b9a30 2024-03-06T07:29:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.735+0000 7fcecbfff640 1 --2- 172.21.15.27:0/2440488250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b9a30 0x7fcecc0d3220 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009e50 tx=0x7fcec402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6562aa94cbfd9b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.735+0000 7fceca7fc640 1 -- 172.21.15.27:0/2440488250 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec402cb00 con 0x7fcecc0b9a30 2024-03-06T07:29:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.735+0000 7fceca7fc640 1 -- 172.21.15.27:0/2440488250 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec402cca0 con 0x7fcecc0b9a30 2024-03-06T07:29:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.735+0000 7fceca7fc640 1 -- 172.21.15.27:0/2440488250 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4034610 con 0x7fcecc0b9a30 2024-03-06T07:29:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.736+0000 7fced21d7640 1 -- 172.21.15.27:0/2440488250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b9a30 msgr2=0x7fcecc0d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.736+0000 7fced21d7640 1 --2- 172.21.15.27:0/2440488250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b9a30 0x7fcecc0d3220 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009e50 tx=0x7fcec402bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.736+0000 7fced21d7640 1 -- 172.21.15.27:0/2440488250 shutdown_connections 2024-03-06T07:29:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.736+0000 7fced21d7640 1 --2- 172.21.15.27:0/2440488250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b9a30 0x7fcecc0d3220 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.736+0000 7fced21d7640 1 --2- 172.21.15.27:0/2440488250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc15df00 0x7fcecc0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.736+0000 7fced21d7640 1 -- 172.21.15.27:0/2440488250 >> 172.21.15.27:0/2440488250 conn(0x7fcecc071960 msgr2=0x7fcecc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.736+0000 7fced21d7640 1 -- 172.21.15.27:0/2440488250 shutdown_connections 2024-03-06T07:29:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.736+0000 7fced21d7640 1 -- 172.21.15.27:0/2440488250 wait complete. 2024-03-06T07:29:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.737+0000 7fced21d7640 1 Processor -- start 2024-03-06T07:29:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fced21d7640 1 -- start start 2024-03-06T07:29:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fced21d7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b95a0 0x7fcecc1f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fced21d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcecc15df00 0x7fcecc0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fced21d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc0b4510 0x7fcecc15e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fced21d7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc0cec70 con 0x7fcecc0b95a0 2024-03-06T07:29:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fced21d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc0ce750 con 0x7fcecc15df00 2024-03-06T07:29:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fced21d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc0cedf0 con 0x7fcecc0b4510 2024-03-06T07:29:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fcecb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b95a0 0x7fcecc1f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fcecb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b95a0 0x7fcecc1f3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40260/0 (socket says 172.21.15.27:40260) 2024-03-06T07:29:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.738+0000 7fcecb7fe640 1 -- 172.21.15.27:0/62104756 learned_addr learned my addr 172.21.15.27:0/62104756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecb7fe640 1 -- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc0b4510 msgr2=0x7fcecc15e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecbfff640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc0b4510 0x7fcecc15e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecaffd640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcecc15df00 0x7fcecc0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecb7fe640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc0b4510 0x7fcecc15e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecb7fe640 1 -- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcecc15df00 msgr2=0x7fcecc0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecb7fe640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcecc15df00 0x7fcecc0b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecb7fe640 1 -- 172.21.15.27:0/62104756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc0b4a20 con 0x7fcecc0b95a0 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecbfff640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc0b4510 0x7fcecc15e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecaffd640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcecc15df00 0x7fcecc0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fcecb7fe640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b95a0 0x7fcecc1f3900 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00e9b0 tx=0x7fcebc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fceb3fff640 1 -- 172.21.15.27:0/62104756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc017070 con 0x7fcecc0b95a0 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fceb3fff640 1 -- 172.21.15.27:0/62104756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcebc00c9a0 con 0x7fcecc0b95a0 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcecc0b4d00 con 0x7fcecc0b95a0 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fceb3fff640 1 -- 172.21.15.27:0/62104756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc00cd90 con 0x7fcecc0b95a0 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.739+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcecc0b52c0 con 0x7fcecc0b95a0 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.740+0000 7fceb3fff640 1 -- 172.21.15.27:0/62104756 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcebc01a070 con 0x7fcecc0b95a0 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.741+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcecc067ff0 con 0x7fcecc0b95a0 2024-03-06T07:29:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.741+0000 7fceb3fff640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fceb806b4d0 0x7fceb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.741+0000 7fceb3fff640 1 -- 172.21.15.27:0/62104756 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fcebc08a1f0 con 0x7fcecc0b95a0 2024-03-06T07:29:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.741+0000 7fcecaffd640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fceb806b4d0 0x7fceb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.742+0000 7fcecaffd640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fceb806b4d0 0x7fceb806d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fcecc15a7f0 tx=0x7fcec0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.746+0000 7fceb3fff640 1 -- 172.21.15.27:0/62104756 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcebc059010 con 0x7fcecc0b95a0 2024-03-06T07:29:54.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.968+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcecc15ae90 con 0x7fcecc0b95a0 2024-03-06T07:29:54.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.969+0000 7fceb3fff640 1 -- 172.21.15.27:0/62104756 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fcebc058910 con 0x7fcecc0b95a0 2024-03-06T07:29:54.971 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:54.971 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:54.971 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:54.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fceb806b4d0 msgr2=0x7fceb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fceb806b4d0 0x7fceb806d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fcecc15a7f0 tx=0x7fcec0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b95a0 msgr2=0x7fcecc1f3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b95a0 0x7fcecc1f3900 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00e9b0 tx=0x7fcebc00ee70 comp rx=0 tx=0).stop 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 shutdown_connections 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fceb806b4d0 0x7fceb806d990 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcecc0b4510 0x7fcecc15e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcecc15df00 0x7fcecc0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 --2- 172.21.15.27:0/62104756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcecc0b95a0 0x7fcecc1f3900 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 >> 172.21.15.27:0/62104756 conn(0x7fcecc071960 msgr2=0x7fcecc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 shutdown_connections 2024-03-06T07:29:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:54.971+0000 7fced21d7640 1 -- 172.21.15.27:0/62104756 wait complete. 2024-03-06T07:29:54.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:29:55.105 INFO:teuthology.orchestra.run.smithi027.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:29:55.106 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:29:55.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.215+0000 7fdf53e88640 1 Processor -- start 2024-03-06T07:29:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.215+0000 7fdf53e88640 1 -- start start 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf53e88640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdf4c15df00 0x7fdf4c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf53e88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf4c0b9a30 0x7fdf4c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf53e88640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c0cea30 con 0x7fdf4c0b95a0 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf53e88640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c0d2af0 con 0x7fdf4c15df00 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf53e88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0cf3c0 con 0x7fdf4c0b9a30 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf51423640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdf4c0b95a0 0x7fdf4c0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34072/0 (socket says 172.21.15.27:34072) 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf51423640 1 -- 172.21.15.27:0/1917500165 learned_addr learned my addr 172.21.15.27:0/1917500165 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf51c24640 1 --2- 172.21.15.27:0/1917500165 >> v2:172.21.15.69:3300/0 conn(0x7fdf4c15df00 0x7fdf4c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf52425640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf4c0b9a30 0x7fdf4c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300472574 0 0) 0x7fdf4c0cea30 con 0x7fdf4c0b95a0 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf38003520 con 0x7fdf4c0b95a0 2024-03-06T07:29:55.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067411334 0 0) 0x7fdf38003520 con 0x7fdf4c0b95a0 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf4c0b9a30 msgr2=0x7fdf4c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.216+0000 7fdf50c22640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf4c0b9a30 0x7fdf4c0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 >> v2:172.21.15.69:3300/0 conn(0x7fdf4c15df00 msgr2=0x7fdf4c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 --2- 172.21.15.27:0/1917500165 >> v2:172.21.15.69:3300/0 conn(0x7fdf4c15df00 0x7fdf4c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c0cf0e0 con 0x7fdf4c0b95a0 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fdf3c001670 con 0x7fdf4c0b95a0 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf51c24640 1 --2- 172.21.15.27:0/1917500165 >> v2:172.21.15.69:3300/0 conn(0x7fdf4c15df00 0x7fdf4c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 >> v1:172.21.15.27:6789/0 conn(0x7fdf4c0b95a0 legacy=0x7fdf4c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf380039b0 0x7fdf38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf38004e50 0x7fdf380052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0d2af0 con 0x7fdf38004310 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0cf3c0 con 0x7fdf38004e50 2024-03-06T07:29:55.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3800bd40 con 0x7fdf380039b0 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf51423640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf51c24640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf380039b0 0x7fdf38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf52425640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf38004e50 0x7fdf380052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf51423640 1 -- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf380039b0 msgr2=0x7fdf38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf51423640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf380039b0 0x7fdf38003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf51423640 1 -- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf38004e50 msgr2=0x7fdf380052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf51423640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf38004e50 0x7fdf380052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.217+0000 7fdf51423640 1 -- 172.21.15.27:0/1917500165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf38003520 con 0x7fdf38004310 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.218+0000 7fdf51423640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf38004800 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c290 tx=0x7fdf38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1c389a3d22fd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.218+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c007d50 con 0x7fdf38004310 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.218+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c030070 con 0x7fdf38004310 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.218+0000 7fdf50c22640 1 -- 172.21.15.27:0/1917500165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c02cdc0 con 0x7fdf38004310 2024-03-06T07:29:55.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.218+0000 7fdf53e88640 1 -- 172.21.15.27:0/1917500165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 msgr2=0x7fdf38004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.218+0000 7fdf53e88640 1 --2- 172.21.15.27:0/1917500165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf38004800 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c290 tx=0x7fdf38005830 comp rx=0 tx=0).stop 2024-03-06T07:29:55.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.219+0000 7fdf52425640 1 -- 172.21.15.27:0/1917500165 reap_dead start 2024-03-06T07:29:55.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.219+0000 7fdf53e88640 1 -- 172.21.15.27:0/1917500165 shutdown_connections 2024-03-06T07:29:55.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.219+0000 7fdf53e88640 1 -- 172.21.15.27:0/1917500165 >> 172.21.15.27:0/1917500165 conn(0x7fdf4c071960 msgr2=0x7fdf4c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.219+0000 7fdf53e88640 1 -- 172.21.15.27:0/1917500165 shutdown_connections 2024-03-06T07:29:55.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.219+0000 7fdf53e88640 1 -- 172.21.15.27:0/1917500165 wait complete. 2024-03-06T07:29:55.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.221+0000 7fdf53e88640 1 Processor -- start 2024-03-06T07:29:55.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.221+0000 7fdf53e88640 1 -- start start 2024-03-06T07:29:55.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf53e88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf4c15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf53e88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf4c15ef60 0x7fdf4c15ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf53e88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf4c1f4480 0x7fdf4c0ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf53e88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0cec70 con 0x7fdf38004310 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf53e88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0ce750 con 0x7fdf4c1f4480 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf53e88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0cedf0 con 0x7fdf4c15ef60 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf51c24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf4c15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf51c24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf4c15bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40268/0 (socket says 172.21.15.27:40268) 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf51c24640 1 -- 172.21.15.27:0/1395380209 learned_addr learned my addr 172.21.15.27:0/1395380209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf51423640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf4c15ef60 0x7fdf4c15ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf52425640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf4c1f4480 0x7fdf4c0ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf51c24640 1 -- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf4c15ef60 msgr2=0x7fdf4c15ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf51c24640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf4c15ef60 0x7fdf4c15ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf51c24640 1 -- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf4c1f4480 msgr2=0x7fdf4c0ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.222+0000 7fdf51c24640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf4c1f4480 0x7fdf4c0ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.223+0000 7fdf51c24640 1 -- 172.21.15.27:0/1395380209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c160650 con 0x7fdf38004310 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.223+0000 7fdf51c24640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf4c15bb20 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fdf48011d00 tx=0x7fdf4800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.223+0000 7fdf427fc640 1 -- 172.21.15.27:0/1395380209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4800fcc0 con 0x7fdf38004310 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.223+0000 7fdf427fc640 1 -- 172.21.15.27:0/1395380209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4800fe60 con 0x7fdf38004310 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.223+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c15e480 con 0x7fdf38004310 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.223+0000 7fdf427fc640 1 -- 172.21.15.27:0/1395380209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48004270 con 0x7fdf38004310 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.223+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c1f4900 con 0x7fdf38004310 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.224+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4c15f340 con 0x7fdf38004310 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.224+0000 7fdf427fc640 1 -- 172.21.15.27:0/1395380209 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf48012040 con 0x7fdf38004310 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.225+0000 7fdf427fc640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf3006b4d0 0x7fdf3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.225+0000 7fdf427fc640 1 -- 172.21.15.27:0/1395380209 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fdf4808aa90 con 0x7fdf38004310 2024-03-06T07:29:55.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.225+0000 7fdf51423640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf3006b4d0 0x7fdf3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.225+0000 7fdf51423640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf3006b4d0 0x7fdf3006d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c034000 tx=0x7fdf3c007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.229+0000 7fdf427fc640 1 -- 172.21.15.27:0/1395380209 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf4c15f340 con 0x7fdf38004310 2024-03-06T07:29:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.447+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf4c067ff0 con 0x7fdf38004310 2024-03-06T07:29:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.448+0000 7fdf427fc640 1 -- 172.21.15.27:0/1395380209 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fdf48019070 con 0x7fdf38004310 2024-03-06T07:29:55.449 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:55.449 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf3006b4d0 msgr2=0x7fdf3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf3006b4d0 0x7fdf3006d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c034000 tx=0x7fdf3c007000 comp rx=0 tx=0).stop 2024-03-06T07:29:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 msgr2=0x7fdf4c15bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf4c15bb20 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fdf48011d00 tx=0x7fdf4800f610 comp rx=0 tx=0).stop 2024-03-06T07:29:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 shutdown_connections 2024-03-06T07:29:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf3006b4d0 0x7fdf3006d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf4c1f4480 0x7fdf4c0ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf4c15ef60 0x7fdf4c15ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 --2- 172.21.15.27:0/1395380209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf38004310 0x7fdf4c15bb20 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 >> 172.21.15.27:0/1395380209 conn(0x7fdf4c071960 msgr2=0x7fdf4c0b8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 shutdown_connections 2024-03-06T07:29:55.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.450+0000 7fdf53e88640 1 -- 172.21.15.27:0/1395380209 wait complete. 2024-03-06T07:29:55.462 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache 2024-03-06T07:29:55.584 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:29:55.584 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:29:55.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:55.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.690+0000 7f3f16d8a640 1 Processor -- start 2024-03-06T07:29:55.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.690+0000 7f3f16d8a640 1 -- start start 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f16d8a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3f1015df00 0x7f3f100d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f16d8a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f100b95a0 0x7f3f100b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f16d8a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f100cea30 con 0x7f3f100b9a30 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f16d8a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f100d2af0 con 0x7f3f100b95a0 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f16d8a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f100cf3c0 con 0x7f3f1015df00 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f07fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f100b95a0 0x7f3f100b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f07fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f100b95a0 0x7f3f100b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40272/0 (socket says 172.21.15.27:40272) 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f07fff640 1 -- 172.21.15.27:0/547449339 learned_addr learned my addr 172.21.15.27:0/547449339 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f14b26640 1 --2- 172.21.15.27:0/547449339 >> v2:172.21.15.69:3300/0 conn(0x7f3f1015df00 0x7f3f100d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f07fff640 1 -- 172.21.15.27:0/547449339 >> v2:172.21.15.69:3300/0 conn(0x7f3f1015df00 msgr2=0x7f3f100d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f07fff640 1 --2- 172.21.15.27:0/547449339 >> v2:172.21.15.69:3300/0 conn(0x7f3f1015df00 0x7f3f100d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f07fff640 1 -- 172.21.15.27:0/547449339 >> v1:172.21.15.69:6789/0 conn(0x7f3f100b9a30 legacy=0x7f3f100d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f07fff640 1 -- 172.21.15.27:0/547449339 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f100cf0e0 con 0x7f3f100b95a0 2024-03-06T07:29:55.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.691+0000 7f3f14b26640 1 --2- 172.21.15.27:0/547449339 >> v2:172.21.15.69:3300/0 conn(0x7f3f1015df00 0x7f3f100d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:55.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.692+0000 7f3f07fff640 1 --2- 172.21.15.27:0/547449339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 0x7f3f100b5700 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f3f100d11b0 tx=0x7f3f0002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1c5b2120d8020 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f077fe640 1 -- 172.21.15.27:0/547449339 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f00035020 con 0x7f3f100b95a0 2024-03-06T07:29:55.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f077fe640 1 -- 172.21.15.27:0/547449339 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f00031870 con 0x7f3f100b95a0 2024-03-06T07:29:55.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f077fe640 1 -- 172.21.15.27:0/547449339 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f00031c40 con 0x7f3f100b95a0 2024-03-06T07:29:55.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f16d8a640 1 -- 172.21.15.27:0/547449339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 msgr2=0x7f3f100b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f16d8a640 1 --2- 172.21.15.27:0/547449339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 0x7f3f100b5700 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f3f100d11b0 tx=0x7f3f0002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:55.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f16d8a640 1 -- 172.21.15.27:0/547449339 shutdown_connections 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f16d8a640 1 --2- 172.21.15.27:0/547449339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 0x7f3f100b5700 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f16d8a640 1 --2- 172.21.15.27:0/547449339 >> v2:172.21.15.69:3300/0 conn(0x7f3f1015df00 0x7f3f100d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f16d8a640 1 -- 172.21.15.27:0/547449339 >> 172.21.15.27:0/547449339 conn(0x7f3f10071960 msgr2=0x7f3f10071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f16d8a640 1 -- 172.21.15.27:0/547449339 shutdown_connections 2024-03-06T07:29:55.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.693+0000 7f3f16d8a640 1 -- 172.21.15.27:0/547449339 wait complete. 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.695+0000 7f3f16d8a640 1 Processor -- start 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.695+0000 7f3f16d8a640 1 -- start start 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.695+0000 7f3f16d8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 0x7f3f10160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.695+0000 7f3f16d8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f1015df00 0x7f3f1015f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.695+0000 7f3f16d8a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f1015b290 0x7f3f100c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.695+0000 7f3f16d8a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f100cec70 con 0x7f3f100b95a0 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.695+0000 7f3f16d8a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f100ce750 con 0x7f3f1015b290 2024-03-06T07:29:55.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.695+0000 7f3f16d8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f100cedf0 con 0x7f3f1015df00 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f14b26640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 0x7f3f10160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f14b26640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 0x7f3f10160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40286/0 (socket says 172.21.15.27:40286) 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f14b26640 1 -- 172.21.15.27:0/2915603723 learned_addr learned my addr 172.21.15.27:0/2915603723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f07fff640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f1015df00 0x7f3f1015f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f15327640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f1015b290 0x7f3f100c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f14b26640 1 -- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f1015df00 msgr2=0x7f3f1015f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f14b26640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f1015df00 0x7f3f1015f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f14b26640 1 -- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f1015b290 msgr2=0x7f3f100c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f14b26640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f1015b290 0x7f3f100c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f14b26640 1 -- 172.21.15.27:0/2915603723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1015fce0 con 0x7f3f100b95a0 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f07fff640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f1015df00 0x7f3f1015f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f14b26640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 0x7f3f10160d80 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00c7f0 tx=0x7f3f0c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f057fa640 1 -- 172.21.15.27:0/2915603723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c015070 con 0x7f3f100b95a0 2024-03-06T07:29:55.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f057fa640 1 -- 172.21.15.27:0/2915603723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c007820 con 0x7f3f100b95a0 2024-03-06T07:29:55.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1015ffc0 con 0x7f3f100b95a0 2024-03-06T07:29:55.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f057fa640 1 -- 172.21.15.27:0/2915603723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c007bd0 con 0x7f3f100b95a0 2024-03-06T07:29:55.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.696+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f10160580 con 0x7f3f100b95a0 2024-03-06T07:29:55.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.697+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f10067ff0 con 0x7f3f100b95a0 2024-03-06T07:29:55.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.698+0000 7f3f057fa640 1 -- 172.21.15.27:0/2915603723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f0c018070 con 0x7f3f100b95a0 2024-03-06T07:29:55.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.699+0000 7f3f057fa640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3efc06b500 0x7f3efc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.699+0000 7f3f057fa640 1 -- 172.21.15.27:0/2915603723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f3f0c08a400 con 0x7f3f100b95a0 2024-03-06T07:29:55.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.699+0000 7f3f07fff640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3efc06b500 0x7f3efc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.700+0000 7f3f07fff640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3efc06b500 0x7f3efc06d9c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3f00036000 tx=0x7f3f0003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.703+0000 7f3f057fa640 1 -- 172.21.15.27:0/2915603723 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f0c059190 con 0x7f3f100b95a0 2024-03-06T07:29:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.927+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f100c7560 con 0x7f3f100b95a0 2024-03-06T07:29:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.928+0000 7f3f057fa640 1 -- 172.21.15.27:0/2915603723 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f3f0c058a90 con 0x7f3f100b95a0 2024-03-06T07:29:55.929 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:55.929 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:55.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3efc06b500 msgr2=0x7f3efc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3efc06b500 0x7f3efc06d9c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3f00036000 tx=0x7f3f0003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 msgr2=0x7f3f10160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 0x7f3f10160d80 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00c7f0 tx=0x7f3f0c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 shutdown_connections 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3efc06b500 0x7f3efc06d9c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f1015b290 0x7f3f100c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f1015df00 0x7f3f1015f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 --2- 172.21.15.27:0/2915603723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f100b95a0 0x7f3f10160d80 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 >> 172.21.15.27:0/2915603723 conn(0x7f3f10071960 msgr2=0x7f3f100b5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 shutdown_connections 2024-03-06T07:29:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:55.930+0000 7f3f16d8a640 1 -- 172.21.15.27:0/2915603723 wait complete. 2024-03-06T07:29:55.944 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue 2024-03-06T07:29:56.066 INFO:teuthology.orchestra.run.smithi027.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:29:56.067 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:29:56.067 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:29:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.107+0000 7fd7ba986f40 1 Processor -- start 2024-03-06T07:29:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.107+0000 7fd7ba986f40 1 -- start start 2024-03-06T07:29:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7ba986f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb051a0 0x55671eb7f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7ba986f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb92310 0x55671eabd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7ba986f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55671eb926f0 0x55671eae8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7ba986f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55671eaea790 con 0x55671eb051a0 2024-03-06T07:29:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7ba986f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55671ea60b70 con 0x55671eb92310 2024-03-06T07:29:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7ba986f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55671eabb500 con 0x55671eb926f0 2024-03-06T07:29:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8489640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb051a0 0x55671eb7f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8489640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb051a0 0x55671eb7f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40298/0 (socket says 172.21.15.27:40298) 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8489640 1 -- 172.21.15.27:0/2316208876 learned_addr learned my addr 172.21.15.27:0/2316208876 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55671eb926f0 0x55671eae8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb92310 0x55671eabd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8489640 1 -- 172.21.15.27:0/2316208876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55671eb926f0 msgr2=0x55671eae8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8489640 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55671eb926f0 0x55671eae8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8489640 1 -- 172.21.15.27:0/2316208876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb92310 msgr2=0x55671eabd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8489640 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb92310 0x55671eabd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8489640 1 -- 172.21.15.27:0/2316208876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55671eb94130 con 0x55671eb051a0 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb92310 0x55671eabd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.108+0000 7fd7b8489640 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb051a0 0x55671eb7f360 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd7a801bb20 tx=0x7fd7a804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3f1cbe95f581d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7b6c86640 1 -- 172.21.15.27:0/2316208876 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8057070 con 0x55671eb051a0 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7b6c86640 1 -- 172.21.15.27:0/2316208876 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a8061070 con 0x55671eb051a0 2024-03-06T07:29:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7b6c86640 1 -- 172.21.15.27:0/2316208876 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8052960 con 0x55671eb051a0 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7ba986f40 1 -- 172.21.15.27:0/2316208876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb051a0 msgr2=0x55671eb7f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb051a0 0x55671eb7f360 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd7a801bb20 tx=0x7fd7a804c000 comp rx=0 tx=0).stop 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7ba986f40 1 -- 172.21.15.27:0/2316208876 shutdown_connections 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55671eb926f0 0x55671eae8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb92310 0x55671eabd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/2316208876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb051a0 0x55671eb7f360 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7ba986f40 1 -- 172.21.15.27:0/2316208876 >> 172.21.15.27:0/2316208876 conn(0x55671eae79b0 msgr2=0x55671eae9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7ba986f40 1 -- 172.21.15.27:0/2316208876 shutdown_connections 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.109+0000 7fd7ba986f40 1 -- 172.21.15.27:0/2316208876 wait complete. 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.110+0000 7fd7ba986f40 1 Processor -- start 2024-03-06T07:29:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.110+0000 7fd7ba986f40 1 -- start start 2024-03-06T07:29:56.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.110+0000 7fd7ba986f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb051a0 0x55671eb7d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.110+0000 7fd7ba986f40 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb92310 0x55671eb80650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.110+0000 7fd7ba986f40 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55671eb926f0 0x55671eb8bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.110+0000 7fd7ba986f40 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55671eb8ce80 con 0x55671eb92310 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.110+0000 7fd7ba986f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55671eb91960 con 0x55671eb051a0 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.110+0000 7fd7ba986f40 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55671eb8d120 con 0x55671eb926f0 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b7c88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb92310 0x55671eb80650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b7c88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb92310 0x55671eb80650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40314/0 (socket says 172.21.15.27:40314) 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 learned_addr learned my addr 172.21.15.27:0/1619545842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb051a0 0x55671eb7d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55671eb926f0 0x55671eb8bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55671eb926f0 msgr2=0x55671eb8bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55671eb926f0 0x55671eb8bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb051a0 msgr2=0x55671eb7d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55671eb051a0 0x55671eb7d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55671eb7e3d0 con 0x55671eb92310 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb92310 0x55671eb80650 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0001bf0 tx=0x7fd7a000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b5c84640 1 -- 172.21.15.27:0/1619545842 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a000f650 con 0x55671eb92310 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b5c84640 1 -- 172.21.15.27:0/1619545842 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a000f7f0 con 0x55671eb92310 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55671eb8ae60 con 0x55671eb92310 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7b5c84640 1 -- 172.21.15.27:0/1619545842 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a000fba0 con 0x55671eb92310 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.111+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55671eb8b420 con 0x55671eb92310 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.113+0000 7fd7b5c84640 1 -- 172.21.15.27:0/1619545842 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7a00203f0 con 0x55671eb92310 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.113+0000 7fd7b5c84640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd798069ee0 0x7fd79806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.113+0000 7fd7b5c84640 1 -- 172.21.15.27:0/1619545842 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd7a0089620 con 0x55671eb92310 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.113+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55671eb8c160 0x55671ec35130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.113+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.113+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd798069ee0 0x7fd79806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.114+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55671eb8c160 0x55671ec35130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.114+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd798069ee0 0x7fd79806c3a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8055040 tx=0x7fd7a804f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.114+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55671eb8c160 0x55671ec35130 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.114+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 1 ==== osd_op_reply(1 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.114+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7fd7b0014040 0x7fd7b00133b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.114+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b0014040 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.115+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7fd7b0014040 0x7fd7b00133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.115+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7fd7b0014040 0x7fd7b00133b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.115+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.1 v2:172.21.15.27:6809/746548704 1 ==== osd_op_reply(2 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b0014040 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.115+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7fd7a8071920 0x7fd7a8066400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.115+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8071920 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.115+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7fd7a8071920 0x7fd7a8066400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.116+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7fd7a8071920 0x7fd7a8066400 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.116+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.3 v2:172.21.15.27:6824/4277055095 1 ==== osd_op_reply(3 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8071920 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.116+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7fd7a0058050 0x7fd7a0056cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.116+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.116+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7fd7a0058050 0x7fd7a0056cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.117+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7fd7a0058050 0x7fd7a0056cb0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.117+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 1 ==== osd_op_reply(4 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.117+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.117+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 2 ==== osd_op_reply(5 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.117+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8071920 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.117+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.3 v2:172.21.15.27:6824/4277055095 2 ==== osd_op_reply(6 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8071920 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.117+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.118+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 2 ==== osd_op_reply(7 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.118+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.118+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 3 ==== osd_op_reply(8 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.118+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.118+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 3 ==== osd_op_reply(9 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.118+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8071920 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.119+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.3 v2:172.21.15.27:6824/4277055095 3 ==== osd_op_reply(10 [pgnls start_epoch 120 out=49b] v114'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8071920 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.119+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8071920 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.119+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.3 v2:172.21.15.27:6824/4277055095 4 ==== osd_op_reply(11 [pgnls start_epoch 120 out=49b] v114'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8071920 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.119+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.119+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 4 ==== osd_op_reply(12 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.119+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.120+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 4 ==== osd_op_reply(13 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.120+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7fd7b002c940 0x7fd7b0022400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.120+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.120+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7fd7b002c940 0x7fd7b0022400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.121+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7fd7b002c940 0x7fd7b0022400 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.121+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 1 ==== osd_op_reply(14 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.121+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b0014040 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.121+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.1 v2:172.21.15.27:6809/746548704 2 ==== osd_op_reply(15 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b0014040 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.121+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.121+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 5 ==== osd_op_reply(16 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.121+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.122+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 2 ==== osd_op_reply(17 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.122+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fd7a8052c10 0x7fd7a805c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.122+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8052c10 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.122+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fd7a8052c10 0x7fd7a805c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.122+0000 7fd7b7c88640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fd7a8052c10 0x7fd7a805c910 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.123+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.5 v2:172.21.15.69:6819/715204514 1 ==== osd_op_reply(18 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8052c10 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.123+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8071920 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.123+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.3 v2:172.21.15.27:6824/4277055095 5 ==== osd_op_reply(19 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8071920 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.123+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.123+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 5 ==== osd_op_reply(20 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.123+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.124+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 6 ==== osd_op_reply(21 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.124+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8071920 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.124+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.3 v2:172.21.15.27:6824/4277055095 6 ==== osd_op_reply(22 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8071920 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.124+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.124+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 6 ==== osd_op_reply(23 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.124+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b0014040 2024-03-06T07:29:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.125+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.1 v2:172.21.15.27:6809/746548704 3 ==== osd_op_reply(24 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b0014040 2024-03-06T07:29:56.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.125+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fd7a8080910 0x7fd7a807f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.125+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8080910 2024-03-06T07:29:56.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.125+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fd7a8080910 0x7fd7a807f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.125+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fd7a8080910 0x7fd7a807f390 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.126+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.6 v2:172.21.15.69:6800/288062332 1 ==== osd_op_reply(25 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a8080910 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.126+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.126+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 7 ==== osd_op_reply(26 [pgnls start_epoch 120 out=49b] v114'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.126+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8052c10 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.126+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.5 v2:172.21.15.69:6819/715204514 2 ==== osd_op_reply(27 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8052c10 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.126+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.126+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 8 ==== osd_op_reply(28 [pgnls start_epoch 120 out=49b] v114'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.126+0000 7fd7b8c8a640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fd7b0038950 0x7fd7b00443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.126+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b0038950 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.127+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fd7b0038950 0x7fd7b00443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.127+0000 7fd7b8489640 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fd7b0038950 0x7fd7b00443f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.127+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.2 v2:172.21.15.27:6808/3892975520 1 ==== osd_op_reply(29 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b0038950 2024-03-06T07:29:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.127+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8080910 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.128+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.6 v2:172.21.15.69:6800/288062332 2 ==== osd_op_reply(30 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a8080910 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.128+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b0014040 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.128+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.1 v2:172.21.15.27:6809/746548704 4 ==== osd_op_reply(31 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b0014040 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.128+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.128+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 3 ==== osd_op_reply(32 [pgnls start_epoch 120 out=49b] v114'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.128+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.129+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 9 ==== osd_op_reply(33 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.129+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.129+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 10 ==== osd_op_reply(34 [pgnls start_epoch 120 out=49b] v114'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.129+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8071920 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.129+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.3 v2:172.21.15.27:6824/4277055095 7 ==== osd_op_reply(35 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8071920 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.129+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b0014040 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.129+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.1 v2:172.21.15.27:6809/746548704 5 ==== osd_op_reply(36 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b0014040 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.129+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8071920 2024-03-06T07:29:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.130+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.3 v2:172.21.15.27:6824/4277055095 8 ==== osd_op_reply(37 [pgnls start_epoch 120 out=49b] v114'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8071920 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.130+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.130+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 4 ==== osd_op_reply(38 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.130+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8052c10 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.131+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.5 v2:172.21.15.69:6819/715204514 3 ==== osd_op_reply(39 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8052c10 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.131+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8052c10 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.131+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.5 v2:172.21.15.69:6819/715204514 4 ==== osd_op_reply(40 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8052c10 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.131+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8080910 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.131+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.6 v2:172.21.15.69:6800/288062332 3 ==== osd_op_reply(41 [pgnls start_epoch 120 out=49b] v114'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a8080910 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.131+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b0014040 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.132+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.1 v2:172.21.15.27:6809/746548704 6 ==== osd_op_reply(42 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b0014040 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.132+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8052c10 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.132+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.5 v2:172.21.15.69:6819/715204514 5 ==== osd_op_reply(43 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8052c10 2024-03-06T07:29:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.132+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.132+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 5 ==== osd_op_reply(44 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.132+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.133+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 7 ==== osd_op_reply(45 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.133+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.133+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 8 ==== osd_op_reply(46 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.133+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.133+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 9 ==== osd_op_reply(47 [pgnls start_epoch 120 out=49b] v114'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.133+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.133+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 10 ==== osd_op_reply(48 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.133+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.134+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 6 ==== osd_op_reply(49 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.134+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.134+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 7 ==== osd_op_reply(50 [pgnls start_epoch 120 out=49b] v114'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.134+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8071920 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.134+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.3 v2:172.21.15.27:6824/4277055095 9 ==== osd_op_reply(51 [pgnls start_epoch 120 out=49b] v114'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8071920 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.134+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x55671eb8c160 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.135+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.7 v2:172.21.15.69:6816/1392567864 11 ==== osd_op_reply(52 [pgnls start_epoch 120 out=49b] v114'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x55671eb8c160 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.135+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.135+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 11 ==== osd_op_reply(53 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.135+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.135+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 8 ==== osd_op_reply(54 [pgnls start_epoch 120 out=49b] v114'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.135+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.135+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 9 ==== osd_op_reply(55 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.135+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b0038950 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.136+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.2 v2:172.21.15.27:6808/3892975520 2 ==== osd_op_reply(56 [pgnls start_epoch 120 out=49b] v114'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b0038950 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.136+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.136+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 12 ==== osd_op_reply(57 [pgnls start_epoch 120 out=49b] v120'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.136+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.136+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 10 ==== osd_op_reply(58 [pgnls start_epoch 120 out=49b] v114'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.136+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8080910 2024-03-06T07:29:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.137+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.6 v2:172.21.15.69:6800/288062332 4 ==== osd_op_reply(59 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a8080910 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.137+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a0058050 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.137+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.0 v2:172.21.15.27:6800/3926946944 13 ==== osd_op_reply(60 [pgnls start_epoch 120 out=49b] v114'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a0058050 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.137+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b0014040 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.137+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.1 v2:172.21.15.27:6809/746548704 7 ==== osd_op_reply(61 [pgnls start_epoch 120 out=49b] v114'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b0014040 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.137+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7b002c940 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 <== osd.4 v2:172.21.15.69:6808/1626703870 11 ==== osd_op_reply(62 [pgnls start_epoch 120 out=49b] v120'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a805a070 con 0x7fd7b002c940 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7b8489640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8080910 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 <== osd.6 v2:172.21.15.69:6800/288062332 5 ==== osd_op_reply(63 [pgnls start_epoch 120 out=49b] v114'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7b000ea10 con 0x7fd7a8080910 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 120 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e120) v9 -- 0x55671ec35670 con 0x7fd7a8052c10 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7b7c88640 1 -- 172.21.15.27:0/1619545842 <== osd.5 v2:172.21.15.69:6819/715204514 6 ==== osd_op_reply(64 [pgnls start_epoch 120 out=49b] v114'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd7a005e050 con 0x7fd7a8052c10 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7fd7a0058050 msgr2=0x7fd7a0056cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7fd7a0058050 0x7fd7a0056cb0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7fd7b0014040 msgr2=0x7fd7b00133b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7fd7b0014040 0x7fd7b00133b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fd7b0038950 msgr2=0x7fd7b00443f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fd7b0038950 0x7fd7b00443f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7fd7a8071920 msgr2=0x7fd7a8066400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7fd7a8071920 0x7fd7a8066400 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7fd7b002c940 msgr2=0x7fd7b0022400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7fd7b002c940 0x7fd7b0022400 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fd7a8052c10 msgr2=0x7fd7a805c910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fd7a8052c10 0x7fd7a805c910 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 reap_dead start 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.138+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fd7a8080910 msgr2=0x7fd7a807f390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7fd7a8080910 0x7fd7a807f390 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7b8c8a640 1 -- 172.21.15.27:0/1619545842 reap_dead start 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55671eb8c160 msgr2=0x55671ec35130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55671eb8c160 0x55671ec35130 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd798069ee0 msgr2=0x7fd79806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd798069ee0 0x7fd79806c3a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8055040 tx=0x7fd7a804f000 comp rx=0 tx=0).stop 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb92310 msgr2=0x55671eb80650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb92310 0x55671eb80650 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0001bf0 tx=0x7fd7a000b6c0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 shutdown_connections 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x55671eb8c160 0x55671ec35130 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd798069ee0 0x7fd79806c3a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 --2- 172.21.15.27:0/1619545842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55671eb92310 0x55671eb80650 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 >> 172.21.15.27:0/1619545842 conn(0x55671eae79b0 msgr2=0x55671eb7ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 shutdown_connections 2024-03-06T07:29:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.139+0000 7fd7ba986f40 1 -- 172.21.15.27:0/1619545842 wait complete. 2024-03-06T07:29:56.148 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:29:56.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:56.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.291+0000 7f98cc355640 1 Processor -- start 2024-03-06T07:29:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.291+0000 7f98cc355640 1 -- start start 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.292+0000 7f98cc355640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f98c415bac0 0x7f98c415cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.292+0000 7f98cc355640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98c40aee10 0x7f98c4070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.292+0000 7f98cc355640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c40729b0 con 0x7f98c406d400 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.292+0000 7f98cc355640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c406cf70 con 0x7f98c40aee10 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.292+0000 7f98cc355640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c406c920 con 0x7f98c415bac0 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.292+0000 7f98ca8f2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98c40aee10 0x7f98c4070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.292+0000 7f98ca8f2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98c40aee10 0x7f98c4070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40328/0 (socket says 172.21.15.27:40328) 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.292+0000 7f98ca8f2640 1 -- 172.21.15.27:0/2600078274 learned_addr learned my addr 172.21.15.27:0/2600078274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.292+0000 7f98ca0f1640 1 --2- 172.21.15.27:0/2600078274 >> v2:172.21.15.69:3300/0 conn(0x7f98c415bac0 0x7f98c415cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.293+0000 7f98ca8f2640 1 -- 172.21.15.27:0/2600078274 >> v2:172.21.15.69:3300/0 conn(0x7f98c415bac0 msgr2=0x7f98c415cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.293+0000 7f98ca8f2640 1 --2- 172.21.15.27:0/2600078274 >> v2:172.21.15.69:3300/0 conn(0x7f98c415bac0 0x7f98c415cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.293+0000 7f98ca8f2640 1 -- 172.21.15.27:0/2600078274 >> v1:172.21.15.27:6789/0 conn(0x7f98c406d400 legacy=0x7f98c40ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.293+0000 7f98ca8f2640 1 -- 172.21.15.27:0/2600078274 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c406cc00 con 0x7f98c40aee10 2024-03-06T07:29:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.293+0000 7f98c90ef640 1 -- 172.21.15.27:0/2600078274 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604760465 0 0) 0x7f98c40729b0 con 0x7f98c406d400 2024-03-06T07:29:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.293+0000 7f98ca0f1640 1 --2- 172.21.15.27:0/2600078274 >> v2:172.21.15.69:3300/0 conn(0x7f98c415bac0 0x7f98c415cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.293+0000 7f98ca8f2640 1 --2- 172.21.15.27:0/2600078274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c40aee10 0x7f98c4070870 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f98c406d0f0 tx=0x7f98bc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d71be59574a8ae1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.293+0000 7f98c90ef640 1 -- 172.21.15.27:0/2600078274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc02cb00 con 0x7f98c40aee10 2024-03-06T07:29:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98c90ef640 1 -- 172.21.15.27:0/2600078274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc02cca0 con 0x7f98c40aee10 2024-03-06T07:29:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98c90ef640 1 -- 172.21.15.27:0/2600078274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc034610 con 0x7f98c40aee10 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98cc355640 1 -- 172.21.15.27:0/2600078274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c40aee10 msgr2=0x7f98c4070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98cc355640 1 --2- 172.21.15.27:0/2600078274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c40aee10 0x7f98c4070870 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f98c406d0f0 tx=0x7f98bc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98cc355640 1 -- 172.21.15.27:0/2600078274 shutdown_connections 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98cc355640 1 --2- 172.21.15.27:0/2600078274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c40aee10 0x7f98c4070870 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98cc355640 1 --2- 172.21.15.27:0/2600078274 >> v2:172.21.15.69:3300/0 conn(0x7f98c415bac0 0x7f98c415cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98cc355640 1 -- 172.21.15.27:0/2600078274 >> 172.21.15.27:0/2600078274 conn(0x7f98c4066d00 msgr2=0x7f98c40670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98cc355640 1 -- 172.21.15.27:0/2600078274 shutdown_connections 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.294+0000 7f98cc355640 1 -- 172.21.15.27:0/2600078274 wait complete. 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.295+0000 7f98cc355640 1 Processor -- start 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.295+0000 7f98cc355640 1 -- start start 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98cc355640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c406d400 0x7f98c4145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98cc355640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40aee10 0x7f98c40ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98cc355640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c415bac0 0x7f98c40acc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98cc355640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c415c020 con 0x7f98c406d400 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98cc355640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c415bea0 con 0x7f98c415bac0 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98cc355640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c4072580 con 0x7f98c40aee10 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98ca0f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c406d400 0x7f98c4145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98ca0f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c406d400 0x7f98c4145ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40344/0 (socket says 172.21.15.27:40344) 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98ca0f1640 1 -- 172.21.15.27:0/3583803669 learned_addr learned my addr 172.21.15.27:0/3583803669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98c98f0640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40aee10 0x7f98c40ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.296+0000 7f98ca8f2640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c415bac0 0x7f98c40acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98ca0f1640 1 -- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40aee10 msgr2=0x7f98c40ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98ca0f1640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40aee10 0x7f98c40ac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98ca0f1640 1 -- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c415bac0 msgr2=0x7f98c40acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98ca0f1640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c415bac0 0x7f98c40acc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98ca0f1640 1 -- 172.21.15.27:0/3583803669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c4146530 con 0x7f98c406d400 2024-03-06T07:29:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98ca0f1640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c406d400 0x7f98c4145ff0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f98c000c7f0 tx=0x7f98c000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98baffd640 1 -- 172.21.15.27:0/3583803669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0015070 con 0x7f98c406d400 2024-03-06T07:29:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98baffd640 1 -- 172.21.15.27:0/3583803669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c0007a30 con 0x7f98c406d400 2024-03-06T07:29:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c415b1a0 con 0x7f98c406d400 2024-03-06T07:29:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98baffd640 1 -- 172.21.15.27:0/3583803669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0007e00 con 0x7f98c406d400 2024-03-06T07:29:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.297+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c4146bc0 con 0x7f98c406d400 2024-03-06T07:29:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.298+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c4067ff0 con 0x7f98c406d400 2024-03-06T07:29:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.299+0000 7f98baffd640 1 -- 172.21.15.27:0/3583803669 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98c0018070 con 0x7f98c406d400 2024-03-06T07:29:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.299+0000 7f98baffd640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98a806b4d0 0x7f98a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.299+0000 7f98baffd640 1 -- 172.21.15.27:0/3583803669 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f98c008a1e0 con 0x7f98c406d400 2024-03-06T07:29:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.299+0000 7f98c98f0640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98a806b4d0 0x7f98a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.300+0000 7f98c98f0640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98a806b4d0 0x7f98a806d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f98c40cbbb0 tx=0x7f98b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.303+0000 7f98baffd640 1 -- 172.21.15.27:0/3583803669 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98c0058930 con 0x7f98c406d400 2024-03-06T07:29:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.527+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98c406de50 con 0x7f98c406d400 2024-03-06T07:29:56.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.527+0000 7f98baffd640 1 -- 172.21.15.27:0/3583803669 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f98c005c9d0 con 0x7f98c406d400 2024-03-06T07:29:56.529 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:56.529 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:56.529 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98a806b4d0 msgr2=0x7f98a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98a806b4d0 0x7f98a806d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f98c40cbbb0 tx=0x7f98b4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c406d400 msgr2=0x7f98c4145ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c406d400 0x7f98c4145ff0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f98c000c7f0 tx=0x7f98c000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 shutdown_connections 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f98a806b4d0 0x7f98a806d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c415bac0 0x7f98c40acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f98c40aee10 0x7f98c40ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 --2- 172.21.15.27:0/3583803669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c406d400 0x7f98c4145ff0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 >> 172.21.15.27:0/3583803669 conn(0x7f98c4066d00 msgr2=0x7f98c415e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 shutdown_connections 2024-03-06T07:29:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.530+0000 7f98cc355640 1 -- 172.21.15.27:0/3583803669 wait complete. 2024-03-06T07:29:56.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:29:56.664 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:29:56.665 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:29:56.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:56.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.770+0000 7fc394e39640 1 Processor -- start 2024-03-06T07:29:56.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.770+0000 7fc394e39640 1 -- start start 2024-03-06T07:29:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.771+0000 7fc394e39640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015ac30 0x7fc3900b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.771+0000 7fc394e39640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3900afb30 0x7fc3900719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.771+0000 7fc394e39640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3900b3e80 con 0x7fc390071ef0 2024-03-06T07:29:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.771+0000 7fc394e39640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc390072320 con 0x7fc3900afb30 2024-03-06T07:29:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.771+0000 7fc394e39640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900b4120 con 0x7fc39015ac30 2024-03-06T07:29:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.771+0000 7fc387fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3900afb30 0x7fc3900719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.772+0000 7fc38ed9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc390071ef0 0x7fc39006d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34096/0 (socket says 172.21.15.27:34096) 2024-03-06T07:29:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.772+0000 7fc38ed9d640 1 -- 172.21.15.27:0/3677219022 learned_addr learned my addr 172.21.15.27:0/3677219022 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.772+0000 7fc38e59c640 1 --2- 172.21.15.27:0/3677219022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015ac30 0x7fc3900b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.772+0000 7fc387fff640 1 -- 172.21.15.27:0/3677219022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015ac30 msgr2=0x7fc3900b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.772+0000 7fc387fff640 1 --2- 172.21.15.27:0/3677219022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015ac30 0x7fc3900b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.772+0000 7fc387fff640 1 -- 172.21.15.27:0/3677219022 >> v1:172.21.15.27:6789/0 conn(0x7fc390071ef0 legacy=0x7fc39006d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.772+0000 7fc387fff640 1 -- 172.21.15.27:0/3677219022 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3900b4400 con 0x7fc3900afb30 2024-03-06T07:29:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.772+0000 7fc38dd9b640 1 -- 172.21.15.27:0/3677219022 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907027337 0 0) 0x7fc3900b3e80 con 0x7fc390071ef0 2024-03-06T07:29:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc387fff640 1 --2- 172.21.15.27:0/3677219022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900afb30 0x7fc3900719b0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fc37800ba40 tx=0x7fc37802e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e6fca56a0e6fda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc38dd9b640 1 -- 172.21.15.27:0/3677219022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37803b070 con 0x7fc3900afb30 2024-03-06T07:29:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc38dd9b640 1 -- 172.21.15.27:0/3677219022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37802ee30 con 0x7fc3900afb30 2024-03-06T07:29:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc38dd9b640 1 -- 172.21.15.27:0/3677219022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc378036640 con 0x7fc3900afb30 2024-03-06T07:29:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc394e39640 1 -- 172.21.15.27:0/3677219022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900afb30 msgr2=0x7fc3900719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc394e39640 1 --2- 172.21.15.27:0/3677219022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900afb30 0x7fc3900719b0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fc37800ba40 tx=0x7fc37802e080 comp rx=0 tx=0).stop 2024-03-06T07:29:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc394e39640 1 -- 172.21.15.27:0/3677219022 shutdown_connections 2024-03-06T07:29:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc394e39640 1 --2- 172.21.15.27:0/3677219022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900afb30 0x7fc3900719b0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc394e39640 1 --2- 172.21.15.27:0/3677219022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015ac30 0x7fc3900b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc394e39640 1 -- 172.21.15.27:0/3677219022 >> 172.21.15.27:0/3677219022 conn(0x7fc3900d0990 msgr2=0x7fc3900d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc394e39640 1 -- 172.21.15.27:0/3677219022 shutdown_connections 2024-03-06T07:29:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.773+0000 7fc394e39640 1 -- 172.21.15.27:0/3677219022 wait complete. 2024-03-06T07:29:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.775+0000 7fc394e39640 1 Processor -- start 2024-03-06T07:29:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.775+0000 7fc394e39640 1 -- start start 2024-03-06T07:29:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc394e39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390071ef0 0x7fc39015cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc394e39640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900afb30 0x7fc3900aa660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc394e39640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015ac30 0x7fc3900aaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc394e39640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900b4ca0 con 0x7fc390071ef0 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc394e39640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900b49c0 con 0x7fc39015ac30 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc394e39640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900cefa0 con 0x7fc3900afb30 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390071ef0 0x7fc39015cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390071ef0 0x7fc39015cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40376/0 (socket says 172.21.15.27:40376) 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38e59c640 1 -- 172.21.15.27:0/1359784527 learned_addr learned my addr 172.21.15.27:0/1359784527 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc387fff640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900afb30 0x7fc3900aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38ed9d640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015ac30 0x7fc3900aaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38e59c640 1 -- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900afb30 msgr2=0x7fc3900aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38e59c640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900afb30 0x7fc3900aa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38e59c640 1 -- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015ac30 msgr2=0x7fc3900aaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38e59c640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015ac30 0x7fc3900aaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38e59c640 1 -- 172.21.15.27:0/1359784527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3900a9540 con 0x7fc390071ef0 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc387fff640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900afb30 0x7fc3900aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38ed9d640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015ac30 0x7fc3900aaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc38e59c640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390071ef0 0x7fc39015cbe0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc38000e870 tx=0x7fc38000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc386ffd640 1 -- 172.21.15.27:0/1359784527 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380017070 con 0x7fc390071ef0 2024-03-06T07:29:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3900a9820 con 0x7fc390071ef0 2024-03-06T07:29:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc386ffd640 1 -- 172.21.15.27:0/1359784527 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38000c940 con 0x7fc390071ef0 2024-03-06T07:29:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc386ffd640 1 -- 172.21.15.27:0/1359784527 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38000cd30 con 0x7fc390071ef0 2024-03-06T07:29:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.776+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3900a9de0 con 0x7fc390071ef0 2024-03-06T07:29:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.778+0000 7fc386ffd640 1 -- 172.21.15.27:0/1359784527 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc38001a070 con 0x7fc390071ef0 2024-03-06T07:29:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.778+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc390068700 con 0x7fc390071ef0 2024-03-06T07:29:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.778+0000 7fc386ffd640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b4d0 0x7fc37006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.778+0000 7fc386ffd640 1 -- 172.21.15.27:0/1359784527 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc38008a5f0 con 0x7fc390071ef0 2024-03-06T07:29:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.778+0000 7fc387fff640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b4d0 0x7fc37006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.779+0000 7fc387fff640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b4d0 0x7fc37006d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc378038000 tx=0x7fc378032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:56.783+0000 7fc386ffd640 1 -- 172.21.15.27:0/1359784527 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc380059300 con 0x7fc390071ef0 2024-03-06T07:29:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.002+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc39006fde0 con 0x7fc390071ef0 2024-03-06T07:29:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.002+0000 7fc386ffd640 1 -- 172.21.15.27:0/1359784527 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fc380058c00 con 0x7fc390071ef0 2024-03-06T07:29:57.004 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:57.004 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b4d0 msgr2=0x7fc37006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b4d0 0x7fc37006d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc378038000 tx=0x7fc378032000 comp rx=0 tx=0).stop 2024-03-06T07:29:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390071ef0 msgr2=0x7fc39015cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390071ef0 0x7fc39015cbe0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc38000e870 tx=0x7fc38000ed30 comp rx=0 tx=0).stop 2024-03-06T07:29:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 shutdown_connections 2024-03-06T07:29:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b4d0 0x7fc37006d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015ac30 0x7fc3900aaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900afb30 0x7fc3900aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 --2- 172.21.15.27:0/1359784527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390071ef0 0x7fc39015cbe0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 >> 172.21.15.27:0/1359784527 conn(0x7fc3900d0990 msgr2=0x7fc39006f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 shutdown_connections 2024-03-06T07:29:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.004+0000 7fc394e39640 1 -- 172.21.15.27:0/1359784527 wait complete. 2024-03-06T07:29:57.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:29:57.140 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:29:57.140 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:57.141 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:57.184 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:57.238 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:57.239 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:29:57.239 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:57.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.399+0000 7f5d15ac9640 1 Processor -- start 2024-03-06T07:29:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.399+0000 7f5d15ac9640 1 -- start start 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.400+0000 7f5d15ac9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d100b95a0 0x7f5d100b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.400+0000 7f5d15ac9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100cea30 con 0x7f5d100b9a10 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.400+0000 7f5d15ac9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100cf3c0 con 0x7f5d1015df00 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.400+0000 7f5d15ac9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d100cf0e0 con 0x7f5d100b95a0 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.400+0000 7f5d0ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5d100b9a10 0x7f5d100d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34104/0 (socket says 172.21.15.27:34104) 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.400+0000 7f5d0ffff640 1 -- 172.21.15.27:0/2904564666 learned_addr learned my addr 172.21.15.27:0/2904564666 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.400+0000 7f5d0effd640 1 --2- 172.21.15.27:0/2904564666 >> v2:172.21.15.69:3300/0 conn(0x7f5d100b95a0 0x7f5d100b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.400+0000 7f5d0e7fc640 1 -- 172.21.15.27:0/2904564666 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310546836 0 0) 0x7f5d100cea30 con 0x7f5d100b9a10 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.400+0000 7f5d0e7fc640 1 -- 172.21.15.27:0/2904564666 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf8003520 con 0x7f5d100b9a10 2024-03-06T07:29:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.401+0000 7f5d0effd640 1 -- 172.21.15.27:0/2904564666 >> v1:172.21.15.69:6789/0 conn(0x7f5d1015df00 legacy=0x7f5d100d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.401+0000 7f5d0effd640 1 -- 172.21.15.27:0/2904564666 >> v1:172.21.15.27:6789/0 conn(0x7f5d100b9a10 legacy=0x7f5d100d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.401+0000 7f5d0e7fc640 1 -- 172.21.15.27:0/2904564666 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1295088014 0 0) 0x7f5cf8003520 con 0x7f5d100b9a10 2024-03-06T07:29:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.401+0000 7f5d0effd640 1 -- 172.21.15.27:0/2904564666 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100ce470 con 0x7f5d100b95a0 2024-03-06T07:29:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.401+0000 7f5d0effd640 1 --2- 172.21.15.27:0/2904564666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d100b95a0 0x7f5d100b5700 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5d100cf590 tx=0x7f5d0402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9196627a09e08e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.402+0000 7f5d0e7fc640 1 -- 172.21.15.27:0/2904564666 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04039070 con 0x7f5d100b95a0 2024-03-06T07:29:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.402+0000 7f5d0e7fc640 1 -- 172.21.15.27:0/2904564666 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d040314f0 con 0x7f5d100b95a0 2024-03-06T07:29:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.402+0000 7f5d0e7fc640 1 -- 172.21.15.27:0/2904564666 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d080019f0 con 0x7f5d100b95a0 2024-03-06T07:29:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.402+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2904564666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d100b95a0 msgr2=0x7f5d100b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.402+0000 7f5d15ac9640 1 --2- 172.21.15.27:0/2904564666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d100b95a0 0x7f5d100b5700 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5d100cf590 tx=0x7f5d0402b810 comp rx=0 tx=0).stop 2024-03-06T07:29:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.402+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2904564666 shutdown_connections 2024-03-06T07:29:57.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.402+0000 7f5d15ac9640 1 --2- 172.21.15.27:0/2904564666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d100b95a0 0x7f5d100b5700 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.402+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2904564666 >> 172.21.15.27:0/2904564666 conn(0x7f5d10071960 msgr2=0x7f5d10071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.402+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2904564666 shutdown_connections 2024-03-06T07:29:57.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.403+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2904564666 wait complete. 2024-03-06T07:29:57.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.404+0000 7f5d15ac9640 1 Processor -- start 2024-03-06T07:29:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.405+0000 7f5d15ac9640 1 -- start start 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d15ac9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d100b95a0 0x7f5d100b6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d15ac9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d100b9a10 0x7f5d100cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d15ac9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d1015df00 0x7f5d100cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d15ac9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100cedf0 con 0x7f5d100b95a0 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d15ac9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100cec70 con 0x7f5d1015df00 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d15ac9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100b5c40 con 0x7f5d100b9a10 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d100b95a0 0x7f5d100b6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d100b95a0 0x7f5d100b6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40378/0 (socket says 172.21.15.27:40378) 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0f7fe640 1 -- 172.21.15.27:0/2099338208 learned_addr learned my addr 172.21.15.27:0/2099338208 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0ffff640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d1015df00 0x7f5d100cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0effd640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d100b9a10 0x7f5d100cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0f7fe640 1 -- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d100b9a10 msgr2=0x7f5d100cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0f7fe640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d100b9a10 0x7f5d100cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0f7fe640 1 -- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d1015df00 msgr2=0x7f5d100cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0f7fe640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d1015df00 0x7f5d100cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.406+0000 7f5d0f7fe640 1 -- 172.21.15.27:0/2099338208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100cb060 con 0x7f5d100b95a0 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.407+0000 7f5d0f7fe640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d100b95a0 0x7f5d100b6fe0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000c430 tx=0x7f5d0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.407+0000 7f5cf7fff640 1 -- 172.21.15.27:0/2099338208 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00017070 con 0x7f5d100b95a0 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.407+0000 7f5cf7fff640 1 -- 172.21.15.27:0/2099338208 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d000052d0 con 0x7f5d100b95a0 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.407+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d100ccaf0 con 0x7f5d100b95a0 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.407+0000 7f5cf7fff640 1 -- 172.21.15.27:0/2099338208 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00003830 con 0x7f5d100b95a0 2024-03-06T07:29:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.407+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d100cd0b0 con 0x7f5d100b95a0 2024-03-06T07:29:57.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.408+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d10067ff0 con 0x7f5d100b95a0 2024-03-06T07:29:57.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.408+0000 7f5cf7fff640 1 -- 172.21.15.27:0/2099338208 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d0001a070 con 0x7f5d100b95a0 2024-03-06T07:29:57.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.409+0000 7f5cf7fff640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.409+0000 7f5cf7fff640 1 -- 172.21.15.27:0/2099338208 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f5d000895c0 con 0x7f5d100b95a0 2024-03-06T07:29:57.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.409+0000 7f5d0effd640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.410+0000 7f5d0effd640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5d04036000 tx=0x7f5d04008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.413+0000 7f5cf7fff640 1 -- 172.21.15.27:0/2099338208 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d00057d00 con 0x7f5d100b95a0 2024-03-06T07:29:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.596+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d100cd770 con 0x7f5d100b95a0 2024-03-06T07:29:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.597+0000 7f5cf7fff640 1 -- 172.21.15.27:0/2099338208 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16967 (secure 0 0 0) 0x7f5d00087e20 con 0x7f5d100b95a0 2024-03-06T07:29:57.598 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:57.598 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":120,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:29:29.730083+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.260334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.929245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6832/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6834/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6835/1655684069":"2024-03-07T07:28:59.663475+0000","172.21.15.27:6833/1655684069":"2024-03-07T07:28:59.663475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cfc06b4d0 msgr2=0x7f5cfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5d04036000 tx=0x7f5d04008030 comp rx=0 tx=0).stop 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d100b95a0 msgr2=0x7f5d100b6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d100b95a0 0x7f5d100b6fe0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000c430 tx=0x7f5d0000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 shutdown_connections 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d1015df00 0x7f5d100cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5d100b9a10 0x7f5d100cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 --2- 172.21.15.27:0/2099338208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d100b95a0 0x7f5d100b6fe0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 >> 172.21.15.27:0/2099338208 conn(0x7f5d10071960 msgr2=0x7f5d100d4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 shutdown_connections 2024-03-06T07:29:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.599+0000 7f5d15ac9640 1 -- 172.21.15.27:0/2099338208 wait complete. 2024-03-06T07:29:57.611 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:39.260334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:39.929245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:57.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.720+0000 7f3cbd4c9640 1 Processor -- start 2024-03-06T07:29:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.721+0000 7f3cbd4c9640 1 -- start start 2024-03-06T07:29:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.721+0000 7f3cbd4c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cb80717b0 0x7f3cb8071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.721+0000 7f3cbd4c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b2b10 0x7f3cb80d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.721+0000 7f3cbd4c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb815e4b0 con 0x7f3cb80b3060 2024-03-06T07:29:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.721+0000 7f3cbd4c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb80d2e10 con 0x7f3cb80717b0 2024-03-06T07:29:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.721+0000 7f3cbd4c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80b97e0 con 0x7f3cb80b2b10 2024-03-06T07:29:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.721+0000 7f3cb6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cb80717b0 0x7f3cb8071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.722+0000 7f3cb6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cb80717b0 0x7f3cb8071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40390/0 (socket says 172.21.15.27:40390) 2024-03-06T07:29:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.722+0000 7f3cb6ffd640 1 -- 172.21.15.27:0/3781945311 learned_addr learned my addr 172.21.15.27:0/3781945311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.722+0000 7f3cb77fe640 1 --2- 172.21.15.27:0/3781945311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b2b10 0x7f3cb80d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.722+0000 7f3cb6ffd640 1 -- 172.21.15.27:0/3781945311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b2b10 msgr2=0x7f3cb80d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.722+0000 7f3cb6ffd640 1 --2- 172.21.15.27:0/3781945311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b2b10 0x7f3cb80d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.722+0000 7f3cb6ffd640 1 -- 172.21.15.27:0/3781945311 >> v1:172.21.15.69:6789/0 conn(0x7f3cb80b3060 legacy=0x7f3cb80b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.722+0000 7f3cb6ffd640 1 -- 172.21.15.27:0/3781945311 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb80b9a80 con 0x7f3cb80717b0 2024-03-06T07:29:57.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cb6ffd640 1 --2- 172.21.15.27:0/3781945311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80717b0 0x7f3cb8071b90 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009870 tx=0x7f3ca802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca5a461ab032d8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cb5ffb640 1 -- 172.21.15.27:0/3781945311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8034070 con 0x7f3cb80717b0 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cb5ffb640 1 -- 172.21.15.27:0/3781945311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca8030840 con 0x7f3cb80717b0 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cb5ffb640 1 -- 172.21.15.27:0/3781945311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8030c10 con 0x7f3cb80717b0 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/3781945311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80717b0 msgr2=0x7f3cb8071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cbd4c9640 1 --2- 172.21.15.27:0/3781945311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80717b0 0x7f3cb8071b90 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009870 tx=0x7f3ca802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/3781945311 shutdown_connections 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cbd4c9640 1 --2- 172.21.15.27:0/3781945311 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b2b10 0x7f3cb80d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cbd4c9640 1 --2- 172.21.15.27:0/3781945311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80717b0 0x7f3cb8071b90 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/3781945311 >> 172.21.15.27:0/3781945311 conn(0x7f3cb80d3d20 msgr2=0x7f3cb80d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/3781945311 shutdown_connections 2024-03-06T07:29:57.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.723+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/3781945311 wait complete. 2024-03-06T07:29:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.725+0000 7f3cbd4c9640 1 Processor -- start 2024-03-06T07:29:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.725+0000 7f3cbd4c9640 1 -- start start 2024-03-06T07:29:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.725+0000 7f3cbd4c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80717b0 0x7f3cb814b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.725+0000 7f3cbd4c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b2b10 0x7f3cb8149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.725+0000 7f3cbd4c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b3060 0x7f3cb80cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.725+0000 7f3cbd4c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80b7560 con 0x7f3cb80b2b10 2024-03-06T07:29:57.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.725+0000 7f3cbd4c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80d4bf0 con 0x7f3cb80b3060 2024-03-06T07:29:57.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.725+0000 7f3cbd4c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80b76e0 con 0x7f3cb80717b0 2024-03-06T07:29:57.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b2b10 0x7f3cb8149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b2b10 0x7f3cb8149fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40392/0 (socket says 172.21.15.27:40392) 2024-03-06T07:29:57.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb67fc640 1 -- 172.21.15.27:0/471864124 learned_addr learned my addr 172.21.15.27:0/471864124 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:57.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb6ffd640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80717b0 0x7f3cb814b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb77fe640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b3060 0x7f3cb80cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb67fc640 1 -- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80717b0 msgr2=0x7f3cb814b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb67fc640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80717b0 0x7f3cb814b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb67fc640 1 -- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b3060 msgr2=0x7f3cb80cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb67fc640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b3060 0x7f3cb80cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb67fc640 1 -- 172.21.15.27:0/471864124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb814bde0 con 0x7f3cb80b2b10 2024-03-06T07:29:57.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cb67fc640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b2b10 0x7f3cb8149fb0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00c3c0 tx=0x7f3cac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3c9f7fe640 1 -- 172.21.15.27:0/471864124 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0052d0 con 0x7f3cb80b2b10 2024-03-06T07:29:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3c9f7fe640 1 -- 172.21.15.27:0/471864124 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac005490 con 0x7f3cb80b2b10 2024-03-06T07:29:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb814a800 con 0x7f3cb80b2b10 2024-03-06T07:29:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3c9f7fe640 1 -- 172.21.15.27:0/471864124 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac007260 con 0x7f3cb80b2b10 2024-03-06T07:29:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.726+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb814b0a0 con 0x7f3cb80b2b10 2024-03-06T07:29:57.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.727+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb806ce40 con 0x7f3cb80b2b10 2024-03-06T07:29:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.728+0000 7f3c9f7fe640 1 -- 172.21.15.27:0/471864124 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cac007400 con 0x7f3cb80b2b10 2024-03-06T07:29:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.729+0000 7f3c9f7fe640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ca406b4d0 0x7f3ca406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.729+0000 7f3c9f7fe640 1 -- 172.21.15.27:0/471864124 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f3cac089690 con 0x7f3cb80b2b10 2024-03-06T07:29:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.729+0000 7f3cb6ffd640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ca406b4d0 0x7f3ca406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.729+0000 7f3cb6ffd640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ca406b4d0 0x7f3ca406d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8036000 tx=0x7f3ca8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.733+0000 7f3c9f7fe640 1 -- 172.21.15.27:0/471864124 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cb815e4b0 con 0x7f3cb80b2b10 2024-03-06T07:29:57.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.980+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3cb804f410 con 0x7f3cb80b2b10 2024-03-06T07:29:57.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.980+0000 7f3c9f7fe640 1 -- 172.21.15.27:0/471864124 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3cac05ce20 con 0x7f3cb80b2b10 2024-03-06T07:29:57.982 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:29:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ca406b4d0 msgr2=0x7f3ca406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ca406b4d0 0x7f3ca406d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8036000 tx=0x7f3ca8008030 comp rx=0 tx=0).stop 2024-03-06T07:29:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b2b10 msgr2=0x7f3cb8149fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b2b10 0x7f3cb8149fb0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00c3c0 tx=0x7f3cac00c880 comp rx=0 tx=0).stop 2024-03-06T07:29:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 shutdown_connections 2024-03-06T07:29:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3ca406b4d0 0x7f3ca406d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b3060 0x7f3cb80cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b2b10 0x7f3cb8149fb0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 --2- 172.21.15.27:0/471864124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80717b0 0x7f3cb814b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.982+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 >> 172.21.15.27:0/471864124 conn(0x7f3cb80d3d20 msgr2=0x7f3cb815d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.983+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 shutdown_connections 2024-03-06T07:29:57.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:57.983+0000 7f3cbd4c9640 1 -- 172.21.15.27:0/471864124 wait complete. 2024-03-06T07:29:57.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:58.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.109+0000 7f7426193640 1 Processor -- start 2024-03-06T07:29:58.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.109+0000 7f7426193640 1 -- start start 2024-03-06T07:29:58.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f7426193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015dd90 0x7f74200ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f7426193640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f74200d29d0 0x7f74200d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f7426193640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200d3740 con 0x7f74200d4720 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f7426193640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74200d3400 con 0x7f74200d29d0 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f7426193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74200d3120 con 0x7f742015dd90 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f741ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f74200d4720 0x7f74200b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34116/0 (socket says 172.21.15.27:34116) 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f741ffff640 1 -- 172.21.15.27:0/1696332079 learned_addr learned my addr 172.21.15.27:0/1696332079 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f741f7fe640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015dd90 0x7f74200ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613023954 0 0) 0x7f74200d3740 con 0x7f74200d4720 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7408003520 con 0x7f74200d4720 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.110+0000 7f741effd640 1 --2- 172.21.15.27:0/1696332079 >> v2:172.21.15.69:3300/0 conn(0x7f74200d29d0 0x7f74200d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425899681 0 0) 0x7f7408003520 con 0x7f74200d4720 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015dd90 msgr2=0x7f74200ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015dd90 0x7f74200ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 >> v2:172.21.15.69:3300/0 conn(0x7f74200d29d0 msgr2=0x7f74200d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 --2- 172.21.15.27:0/1696332079 >> v2:172.21.15.69:3300/0 conn(0x7f74200d29d0 0x7f74200d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74200d39e0 con 0x7f74200d4720 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f74180019f0 con 0x7f74200d4720 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 >> v1:172.21.15.27:6789/0 conn(0x7f74200d4720 legacy=0x7f74200b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015dd90 0x7f74080039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f7408004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7408004d40 0x7f74080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74200d3400 con 0x7f7408003f30 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74200d3120 con 0x7f7408004d40 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740800b9f0 con 0x7f742015dd90 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741effd640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f7408004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741f7fe640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015dd90 0x7f74080039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741ffff640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7408004d40 0x7f74080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741effd640 1 -- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015dd90 msgr2=0x7f74080039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741effd640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015dd90 0x7f74080039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741effd640 1 -- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7408004d40 msgr2=0x7f74080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741effd640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7408004d40 0x7f74080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741effd640 1 -- 172.21.15.27:0/1696332079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408003520 con 0x7f7408003f30 2024-03-06T07:29:58.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.111+0000 7f741ffff640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7408004d40 0x7f74080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.112+0000 7f741effd640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f7408004800 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f7414009ac0 tx=0x7f7408005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47dc9728a71ebb84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.112+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414039070 con 0x7f7408003f30 2024-03-06T07:29:58.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.112+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741402c710 con 0x7f7408003f30 2024-03-06T07:29:58.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.112+0000 7f741e7fc640 1 -- 172.21.15.27:0/1696332079 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414009690 con 0x7f7408003f30 2024-03-06T07:29:58.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.113+0000 7f7426193640 1 -- 172.21.15.27:0/1696332079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 msgr2=0x7f7408004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.113+0000 7f7426193640 1 --2- 172.21.15.27:0/1696332079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f7408004800 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f7414009ac0 tx=0x7f7408005830 comp rx=0 tx=0).stop 2024-03-06T07:29:58.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.113+0000 7f741ffff640 1 -- 172.21.15.27:0/1696332079 reap_dead start 2024-03-06T07:29:58.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.113+0000 7f7426193640 1 -- 172.21.15.27:0/1696332079 shutdown_connections 2024-03-06T07:29:58.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.113+0000 7f7426193640 1 -- 172.21.15.27:0/1696332079 >> 172.21.15.27:0/1696332079 conn(0x7f7420071960 msgr2=0x7f7420071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.113+0000 7f7426193640 1 -- 172.21.15.27:0/1696332079 shutdown_connections 2024-03-06T07:29:58.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.113+0000 7f7426193640 1 -- 172.21.15.27:0/1696332079 wait complete. 2024-03-06T07:29:58.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.114+0000 7f7426193640 1 Processor -- start 2024-03-06T07:29:58.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.114+0000 7f7426193640 1 -- start start 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f7426193640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f74201f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f7426193640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f742015cbe0 0x7f7420160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f7426193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015c050 0x7f742015c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f7426193640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74200d2df0 con 0x7f7408003f30 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f7426193640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742015e170 con 0x7f742015cbe0 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f7426193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74200b7420 con 0x7f742015c050 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f74201f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f74201f73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40414/0 (socket says 172.21.15.27:40414) 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741f7fe640 1 -- 172.21.15.27:0/1650310537 learned_addr learned my addr 172.21.15.27:0/1650310537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741ffff640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015c050 0x7f742015c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741effd640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f742015cbe0 0x7f7420160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741f7fe640 1 -- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015c050 msgr2=0x7f742015c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741f7fe640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015c050 0x7f742015c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741f7fe640 1 -- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f742015cbe0 msgr2=0x7f7420160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741f7fe640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f742015cbe0 0x7f7420160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741f7fe640 1 -- 172.21.15.27:0/1650310537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74201f7910 con 0x7f7408003f30 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.115+0000 7f741effd640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f742015cbe0 0x7f7420160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.116+0000 7f741f7fe640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f74201f73d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f74100119d0 tx=0x7f7410011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.116+0000 7f7407fff640 1 -- 172.21.15.27:0/1650310537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741000faf0 con 0x7f7408003f30 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.116+0000 7f7407fff640 1 -- 172.21.15.27:0/1650310537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741000fc90 con 0x7f7408003f30 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.116+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74200cc9c0 con 0x7f7408003f30 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.116+0000 7f7407fff640 1 -- 172.21.15.27:0/1650310537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410014440 con 0x7f7408003f30 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.116+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74201f7e90 con 0x7f7408003f30 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.117+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7420072a30 con 0x7f7408003f30 2024-03-06T07:29:58.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.118+0000 7f7407fff640 1 -- 172.21.15.27:0/1650310537 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7410012040 con 0x7f7408003f30 2024-03-06T07:29:58.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.118+0000 7f7407fff640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f740c06b4d0 0x7f740c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.118+0000 7f7407fff640 1 -- 172.21.15.27:0/1650310537 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f741008a7d0 con 0x7f7408003f30 2024-03-06T07:29:58.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.119+0000 7f741effd640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f740c06b4d0 0x7f740c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.119+0000 7f741effd640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f740c06b4d0 0x7f740c06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7414036000 tx=0x7f7414030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.123+0000 7f7407fff640 1 -- 172.21.15.27:0/1650310537 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7410019030 con 0x7f7408003f30 2024-03-06T07:29:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.304+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f742004f410 con 0x7f7408003f30 2024-03-06T07:29:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.304+0000 7f7407fff640 1 -- 172.21.15.27:0/1650310537 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7410058f10 con 0x7f7408003f30 2024-03-06T07:29:58.306 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:29:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f740c06b4d0 msgr2=0x7f740c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f740c06b4d0 0x7f740c06d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7414036000 tx=0x7f7414030000 comp rx=0 tx=0).stop 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 msgr2=0x7f74201f73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f74201f73d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f74100119d0 tx=0x7f7410011dd0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 shutdown_connections 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f740c06b4d0 0x7f740c06d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f742015c050 0x7f742015c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f742015cbe0 0x7f7420160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 --2- 172.21.15.27:0/1650310537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7408003f30 0x7f74201f73d0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 >> 172.21.15.27:0/1650310537 conn(0x7f7420071960 msgr2=0x7f7420068e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 shutdown_connections 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.307+0000 7f7426193640 1 -- 172.21.15.27:0/1650310537 wait complete. 2024-03-06T07:29:58.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:58.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.432+0000 7f9d021e2640 1 Processor -- start 2024-03-06T07:29:58.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.432+0000 7f9d021e2640 1 -- start start 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9d021e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 0x7f9cfc0d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9d021e2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9cfc0b6d10 0x7f9cfc0b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9d021e2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cfc0d3740 con 0x7f9cfc15dd90 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9d021e2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cfc0d3400 con 0x7f9cfc0b6d10 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9d021e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0d3120 con 0x7f9cfc0ceea0 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9cfbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9cfc0b6d10 0x7f9cfc0b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9cfb7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9cfc15dd90 0x7f9cfc0d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34122/0 (socket says 172.21.15.27:34122) 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9cfb7fe640 1 -- 172.21.15.27:0/3341166940 learned_addr learned my addr 172.21.15.27:0/3341166940 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9cfaffd640 1 --2- 172.21.15.27:0/3341166940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 0x7f9cfc0d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9cfbfff640 1 -- 172.21.15.27:0/3341166940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 msgr2=0x7f9cfc0d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9cfbfff640 1 --2- 172.21.15.27:0/3341166940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 0x7f9cfc0d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9cfbfff640 1 -- 172.21.15.27:0/3341166940 >> v1:172.21.15.27:6789/0 conn(0x7f9cfc15dd90 legacy=0x7f9cfc0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9cfbfff640 1 -- 172.21.15.27:0/3341166940 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0d39e0 con 0x7f9cfc0b6d10 2024-03-06T07:29:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.433+0000 7f9cfa7fc640 1 -- 172.21.15.27:0/3341166940 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322442026 0 0) 0x7f9cfc0d3740 con 0x7f9cfc15dd90 2024-03-06T07:29:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9cfbfff640 1 --2- 172.21.15.27:0/3341166940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc0b6d10 0x7f9cfc0b71a0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009dc0 tx=0x7f9cf402bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a6fee618b71e55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9cfa7fc640 1 -- 172.21.15.27:0/3341166940 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4039070 con 0x7f9cfc0b6d10 2024-03-06T07:29:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9cfa7fc640 1 -- 172.21.15.27:0/3341166940 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf402ccd0 con 0x7f9cfc0b6d10 2024-03-06T07:29:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9cfa7fc640 1 -- 172.21.15.27:0/3341166940 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4034660 con 0x7f9cfc0b6d10 2024-03-06T07:29:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9d021e2640 1 -- 172.21.15.27:0/3341166940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc0b6d10 msgr2=0x7f9cfc0b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9d021e2640 1 --2- 172.21.15.27:0/3341166940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc0b6d10 0x7f9cfc0b71a0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009dc0 tx=0x7f9cf402bf20 comp rx=0 tx=0).stop 2024-03-06T07:29:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9d021e2640 1 -- 172.21.15.27:0/3341166940 shutdown_connections 2024-03-06T07:29:58.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9d021e2640 1 --2- 172.21.15.27:0/3341166940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc0b6d10 0x7f9cfc0b71a0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9d021e2640 1 --2- 172.21.15.27:0/3341166940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 0x7f9cfc0d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9d021e2640 1 -- 172.21.15.27:0/3341166940 >> 172.21.15.27:0/3341166940 conn(0x7f9cfc071960 msgr2=0x7f9cfc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9d021e2640 1 -- 172.21.15.27:0/3341166940 shutdown_connections 2024-03-06T07:29:58.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.435+0000 7f9d021e2640 1 -- 172.21.15.27:0/3341166940 wait complete. 2024-03-06T07:29:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.436+0000 7f9d021e2640 1 Processor -- start 2024-03-06T07:29:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.437+0000 7f9d021e2640 1 -- start start 2024-03-06T07:29:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.437+0000 7f9d021e2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cfc0b6d10 0x7f9cfc153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.437+0000 7f9d021e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 0x7f9cfc0cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.437+0000 7f9d021e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc15dd90 0x7f9cfc0cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.437+0000 7f9d021e2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0d48f0 con 0x7f9cfc15dd90 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.437+0000 7f9d021e2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc15e170 con 0x7f9cfc0b6d10 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.437+0000 7f9d021e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0d4a70 con 0x7f9cfc0ceea0 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.437+0000 7f9cfbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc15dd90 0x7f9cfc0cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc15dd90 0x7f9cfc0cd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40432/0 (socket says 172.21.15.27:40432) 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfbfff640 1 -- 172.21.15.27:0/4116416907 learned_addr learned my addr 172.21.15.27:0/4116416907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfaffd640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 0x7f9cfc0cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfb7fe640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cfc0b6d10 0x7f9cfc153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfbfff640 1 -- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 msgr2=0x7f9cfc0cb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfbfff640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 0x7f9cfc0cb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfbfff640 1 -- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cfc0b6d10 msgr2=0x7f9cfc153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfbfff640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cfc0b6d10 0x7f9cfc153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfbfff640 1 -- 172.21.15.27:0/4116416907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0cc240 con 0x7f9cfc15dd90 2024-03-06T07:29:58.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfb7fe640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cfc0b6d10 0x7f9cfc153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9cfbfff640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc15dd90 0x7f9cfc0cd290 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4037040 tx=0x7f9cf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9ce3fff640 1 -- 172.21.15.27:0/4116416907 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4039050 con 0x7f9cfc15dd90 2024-03-06T07:29:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9ce3fff640 1 -- 172.21.15.27:0/4116416907 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf4045070 con 0x7f9cfc15dd90 2024-03-06T07:29:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc0cc520 con 0x7f9cfc15dd90 2024-03-06T07:29:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9ce3fff640 1 -- 172.21.15.27:0/4116416907 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf403e850 con 0x7f9cfc15dd90 2024-03-06T07:29:58.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.438+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc0c7ed0 con 0x7f9cfc15dd90 2024-03-06T07:29:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.439+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc072a30 con 0x7f9cfc15dd90 2024-03-06T07:29:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.440+0000 7f9ce3fff640 1 -- 172.21.15.27:0/4116416907 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cf403e9f0 con 0x7f9cfc15dd90 2024-03-06T07:29:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.440+0000 7f9ce3fff640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ce806b500 0x7f9ce806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.440+0000 7f9ce3fff640 1 -- 172.21.15.27:0/4116416907 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f9cf40ad140 con 0x7f9cfc15dd90 2024-03-06T07:29:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.441+0000 7f9cfb7fe640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ce806b500 0x7f9ce806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.441+0000 7f9cfb7fe640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ce806b500 0x7f9ce806d9c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0000f60 tx=0x7f9cf0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.445+0000 7f9ce3fff640 1 -- 172.21.15.27:0/4116416907 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cf4032e20 con 0x7f9cfc15dd90 2024-03-06T07:29:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.714+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9cfc04f410 con 0x7f9cfc15dd90 2024-03-06T07:29:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.714+0000 7f9ce3fff640 1 -- 172.21.15.27:0/4116416907 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9cf407c080 con 0x7f9cfc15dd90 2024-03-06T07:29:58.717 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ce806b500 msgr2=0x7f9ce806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ce806b500 0x7f9ce806d9c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0000f60 tx=0x7f9cf0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc15dd90 msgr2=0x7f9cfc0cd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc15dd90 0x7f9cfc0cd290 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4037040 tx=0x7f9cf4004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 shutdown_connections 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ce806b500 0x7f9ce806d9c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cfc15dd90 0x7f9cfc0cd290 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cfc0ceea0 0x7f9cfc0cb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 --2- 172.21.15.27:0/4116416907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cfc0b6d10 0x7f9cfc153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.717+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 >> 172.21.15.27:0/4116416907 conn(0x7f9cfc071960 msgr2=0x7f9cfc0b7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.718+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 shutdown_connections 2024-03-06T07:29:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.718+0000 7f9d021e2640 1 -- 172.21.15.27:0/4116416907 wait complete. 2024-03-06T07:29:58.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.841+0000 7f1e99102640 1 Processor -- start 2024-03-06T07:29:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.841+0000 7f1e99102640 1 -- start start 2024-03-06T07:29:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.842+0000 7f1e99102640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e940b95a0 0x7f1e940b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.842+0000 7f1e99102640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e940cea30 con 0x7f1e9415df00 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.842+0000 7f1e99102640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e940cf3c0 con 0x7f1e940b9a10 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.842+0000 7f1e99102640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940cf0e0 con 0x7f1e940b95a0 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.842+0000 7f1e92d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1e9415df00 0x7f1e940d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34128/0 (socket says 172.21.15.27:34128) 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.842+0000 7f1e92d9d640 1 -- 172.21.15.27:0/2877472919 learned_addr learned my addr 172.21.15.27:0/2877472919 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e9259c640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e940b95a0 0x7f1e940b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691920170 0 0) 0x7f1e940cea30 con 0x7f1e9415df00 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7c003520 con 0x7f1e9415df00 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 515758892 0 0) 0x7f1e7c003520 con 0x7f1e9415df00 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e940b95a0 msgr2=0x7f1e940b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e940b95a0 0x7f1e940b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 >> v1:172.21.15.69:6789/0 conn(0x7f1e940b9a10 legacy=0x7f1e940d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e940ce470 con 0x7f1e9415df00 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f1e88001670 con 0x7f1e9415df00 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 >> v1:172.21.15.27:6789/0 conn(0x7f1e9415df00 legacy=0x7f1e940d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e940b95a0 0x7f1e7c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e7c003f60 0x7f1e7c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c004d40 0x7f1e7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940cf0e0 con 0x7f1e7c004d40 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c00baa0 con 0x7f1e7c003f60 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c00bca0 con 0x7f1e940b95a0 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e9359e640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c004d40 0x7f1e7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e92d9d640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e940b95a0 0x7f1e7c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e9259c640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e7c003f60 0x7f1e7c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e9359e640 1 -- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e940b95a0 msgr2=0x7f1e7c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e9359e640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e940b95a0 0x7f1e7c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e9359e640 1 -- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e7c003f60 msgr2=0x7f1e7c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e9359e640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e7c003f60 0x7f1e7c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e9359e640 1 -- 172.21.15.27:0/2877472919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c003520 con 0x7f1e7c004d40 2024-03-06T07:29:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.843+0000 7f1e92d9d640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e940b95a0 0x7f1e7c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.844+0000 7f1e9359e640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c004d40 0x7f1e7c0051c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c001e20 tx=0x7f1e7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52b6981558d2f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.844+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c001a70 con 0x7f1e7c004d40 2024-03-06T07:29:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.844+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8c003b70 con 0x7f1e7c004d40 2024-03-06T07:29:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.844+0000 7f1e91d9b640 1 -- 172.21.15.27:0/2877472919 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c0050d0 con 0x7f1e7c004d40 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.845+0000 7f1e99102640 1 -- 172.21.15.27:0/2877472919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c004d40 msgr2=0x7f1e7c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.845+0000 7f1e99102640 1 --2- 172.21.15.27:0/2877472919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c004d40 0x7f1e7c0051c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c001e20 tx=0x7f1e7c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.845+0000 7f1e9359e640 1 -- 172.21.15.27:0/2877472919 reap_dead start 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.845+0000 7f1e99102640 1 -- 172.21.15.27:0/2877472919 shutdown_connections 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.845+0000 7f1e99102640 1 -- 172.21.15.27:0/2877472919 >> 172.21.15.27:0/2877472919 conn(0x7f1e94071960 msgr2=0x7f1e94071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.845+0000 7f1e99102640 1 -- 172.21.15.27:0/2877472919 shutdown_connections 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.845+0000 7f1e99102640 1 -- 172.21.15.27:0/2877472919 wait complete. 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.847+0000 7f1e99102640 1 Processor -- start 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.847+0000 7f1e99102640 1 -- start start 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.847+0000 7f1e99102640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e941551c0 0x7f1e940cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.847+0000 7f1e99102640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e941555a0 0x7f1e94155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.847+0000 7f1e99102640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e9415a170 0x7f1e9415a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.847+0000 7f1e99102640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940cedf0 con 0x7f1e9415a170 2024-03-06T07:29:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.847+0000 7f1e99102640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940cec70 con 0x7f1e941555a0 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.847+0000 7f1e99102640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940d3930 con 0x7f1e941551c0 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e9415a170 0x7f1e9415a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e9415a170 0x7f1e9415a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40458/0 (socket says 172.21.15.27:40458) 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9359e640 1 -- 172.21.15.27:0/1721555576 learned_addr learned my addr 172.21.15.27:0/1721555576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e92d9d640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e941551c0 0x7f1e940cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9259c640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e941555a0 0x7f1e94155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9359e640 1 -- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e941551c0 msgr2=0x7f1e940cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9359e640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e941551c0 0x7f1e940cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9359e640 1 -- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e941555a0 msgr2=0x7f1e94155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9359e640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e941555a0 0x7f1e94155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9359e640 1 -- 172.21.15.27:0/1721555576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e940cbca0 con 0x7f1e9415a170 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e9359e640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e9415a170 0x7f1e9415a5f0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c001f10 tx=0x7f1e8c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e7b7fe640 1 -- 172.21.15.27:0/1721555576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88007d30 con 0x7f1e9415a170 2024-03-06T07:29:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e7b7fe640 1 -- 172.21.15.27:0/1721555576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8c003970 con 0x7f1e9415a170 2024-03-06T07:29:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e7b7fe640 1 -- 172.21.15.27:0/1721555576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c001a70 con 0x7f1e9415a170 2024-03-06T07:29:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9415e650 con 0x7f1e9415a170 2024-03-06T07:29:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.848+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9415ec10 con 0x7f1e9415a170 2024-03-06T07:29:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.849+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9415ab30 con 0x7f1e9415a170 2024-03-06T07:29:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.850+0000 7f1e7b7fe640 1 -- 172.21.15.27:0/1721555576 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e8c007030 con 0x7f1e9415a170 2024-03-06T07:29:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.850+0000 7f1e7b7fe640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1e8006b500 0x7f1e8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.850+0000 7f1e7b7fe640 1 -- 172.21.15.27:0/1721555576 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1e8c0ab0a0 con 0x7f1e9415a170 2024-03-06T07:29:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.851+0000 7f1e92d9d640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1e8006b500 0x7f1e8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.851+0000 7f1e92d9d640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1e8006b500 0x7f1e8006d9c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1e88001030 tx=0x7f1e88001bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:58.855+0000 7f1e7b7fe640 1 -- 172.21.15.27:0/1721555576 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e8c033020 con 0x7f1e9415a170 2024-03-06T07:29:59.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.051+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1e9415d760 con 0x7f1e9415a170 2024-03-06T07:29:59.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.051+0000 7f1e7b7fe640 1 -- 172.21.15.27:0/1721555576 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v120) v1 ==== 80+0+244 (secure 0 0 0) 0x7f1e9415d760 con 0x7f1e9415a170 2024-03-06T07:29:59.053 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6832/3926923781 2024-03-07T07:29:00.663556+0000 2024-03-06T07:29:59.054 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6834/3926923781 2024-03-07T07:29:00.663556+0000 2024-03-06T07:29:59.054 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6835/1655684069 2024-03-07T07:28:59.663475+0000 2024-03-06T07:29:59.054 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6833/1655684069 2024-03-07T07:28:59.663475+0000 2024-03-06T07:29:59.054 INFO:teuthology.orchestra.run.smithi027.stderr:listed 4 entries 2024-03-06T07:29:59.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1e8006b500 msgr2=0x7f1e8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1e8006b500 0x7f1e8006d9c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1e88001030 tx=0x7f1e88001bc0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e9415a170 msgr2=0x7f1e9415a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e9415a170 0x7f1e9415a5f0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c001f10 tx=0x7f1e8c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 shutdown_connections 2024-03-06T07:29:59.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1e8006b500 0x7f1e8006d9c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e9415a170 0x7f1e9415a5f0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e941555a0 0x7f1e94155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 --2- 172.21.15.27:0/1721555576 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1e941551c0 0x7f1e940cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 >> 172.21.15.27:0/1721555576 conn(0x7f1e94071960 msgr2=0x7f1e940d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 shutdown_connections 2024-03-06T07:29:59.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.053+0000 7f1e99102640 1 -- 172.21.15.27:0/1721555576 wait complete. 2024-03-06T07:29:59.067 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:29:59.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:59.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:59.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[74699]: fuse finished with error 0 and tester_r 0 2024-03-06T07:29:59.099 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:29:59.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:59.100 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:29:59.164 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:29:59.164 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:59.183 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:59.183 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:29:59.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:59.183 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:29:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:59.242 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:29:59.242 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:29:59.242 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:59.259 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:59.259 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:29:59.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:59.259 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:29:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:59.315 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:29:59.316 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:29:59.316 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:59.334 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:59.335 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:29:59.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:59.335 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:29:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:59.392 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:29:59.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.508+0000 7f7f8c7bf640 1 Processor -- start 2024-03-06T07:29:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.508+0000 7f7f8c7bf640 1 -- start start 2024-03-06T07:29:59.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.509+0000 7f7f8c7bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f84158520 0x7f7f8406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.509+0000 7f7f8c7bf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f8406dd60 0x7f7f840ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.509+0000 7f7f8c7bf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f840af080 con 0x7f7f840aec40 2024-03-06T07:29:59.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.509+0000 7f7f8c7bf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8406e1c0 con 0x7f7f8406dd60 2024-03-06T07:29:59.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.509+0000 7f7f8c7bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f84071b70 con 0x7f7f84158520 2024-03-06T07:29:59.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.509+0000 7f7f8a55b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f84158520 0x7f7f8406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.509+0000 7f7f89d5a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f8406dd60 0x7f7f840ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.509+0000 7f7f8a55b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f84158520 0x7f7f8406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:54898/0 (socket says 172.21.15.27:54898) 2024-03-06T07:29:59.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.509+0000 7f7f8a55b640 1 -- 172.21.15.27:0/2285655131 learned_addr learned my addr 172.21.15.27:0/2285655131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:59.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.510+0000 7f7f8a55b640 1 -- 172.21.15.27:0/2285655131 >> v2:172.21.15.69:3300/0 conn(0x7f7f8406dd60 msgr2=0x7f7f840ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.510+0000 7f7f8a55b640 1 --2- 172.21.15.27:0/2285655131 >> v2:172.21.15.69:3300/0 conn(0x7f7f8406dd60 0x7f7f840ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.510+0000 7f7f8a55b640 1 -- 172.21.15.27:0/2285655131 >> v1:172.21.15.69:6789/0 conn(0x7f7f840aec40 legacy=0x7f7f8415e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.510+0000 7f7f8a55b640 1 -- 172.21.15.27:0/2285655131 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f84071240 con 0x7f7f84158520 2024-03-06T07:29:59.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.510+0000 7f7f89d5a640 1 --2- 172.21.15.27:0/2285655131 >> v2:172.21.15.69:3300/0 conn(0x7f7f8406dd60 0x7f7f840ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:59.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.510+0000 7f7f8a55b640 1 --2- 172.21.15.27:0/2285655131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f84158520 0x7f7f8406d820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009870 tx=0x7f7f8002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31dc5caaad1ebf3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f89559640 1 -- 172.21.15.27:0/2285655131 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80034070 con 0x7f7f84158520 2024-03-06T07:29:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f89559640 1 -- 172.21.15.27:0/2285655131 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f80030840 con 0x7f7f84158520 2024-03-06T07:29:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f89559640 1 -- 172.21.15.27:0/2285655131 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80030c10 con 0x7f7f84158520 2024-03-06T07:29:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2285655131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f84158520 msgr2=0x7f7f8406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f8c7bf640 1 --2- 172.21.15.27:0/2285655131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f84158520 0x7f7f8406d820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009870 tx=0x7f7f8002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2285655131 shutdown_connections 2024-03-06T07:29:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f8c7bf640 1 --2- 172.21.15.27:0/2285655131 >> v2:172.21.15.69:3300/0 conn(0x7f7f8406dd60 0x7f7f840ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f8c7bf640 1 --2- 172.21.15.27:0/2285655131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f84158520 0x7f7f8406d820 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2285655131 >> 172.21.15.27:0/2285655131 conn(0x7f7f84066d00 msgr2=0x7f7f840670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2285655131 shutdown_connections 2024-03-06T07:29:59.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.511+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2285655131 wait complete. 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.512+0000 7f7f8c7bf640 1 Processor -- start 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.512+0000 7f7f8c7bf640 1 -- start start 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f8c7bf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f8406dd60 0x7f7f840b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f8c7bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f840aec40 0x7f7f841481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f8c7bf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f84158520 0x7f7f84159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f8c7bf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840b4650 con 0x7f7f8406dd60 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f8c7bf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f84071800 con 0x7f7f84158520 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f8c7bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f84072000 con 0x7f7f840aec40 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f8a55b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f8406dd60 0x7f7f840b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f8a55b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f8406dd60 0x7f7f840b0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40464/0 (socket says 172.21.15.27:40464) 2024-03-06T07:29:59.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f8a55b640 1 -- 172.21.15.27:0/2922893653 learned_addr learned my addr 172.21.15.27:0/2922893653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.513+0000 7f7f89d5a640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f840aec40 0x7f7f841481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f8ad5c640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f84158520 0x7f7f84159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f8a55b640 1 -- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f840aec40 msgr2=0x7f7f841481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f8a55b640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f840aec40 0x7f7f841481f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f8a55b640 1 -- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f84158520 msgr2=0x7f7f84159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f8a55b640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f84158520 0x7f7f84159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f8a55b640 1 -- 172.21.15.27:0/2922893653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f840b1310 con 0x7f7f8406dd60 2024-03-06T07:29:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f89d5a640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f840aec40 0x7f7f841481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f8a55b640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f8406dd60 0x7f7f840b0d30 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f7f800090b0 tx=0x7f7f800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f7affd640 1 -- 172.21.15.27:0/2922893653 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80034040 con 0x7f7f8406dd60 2024-03-06T07:29:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f7affd640 1 -- 172.21.15.27:0/2922893653 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8003ada0 con 0x7f7f8406dd60 2024-03-06T07:29:59.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f841f3a50 con 0x7f7f8406dd60 2024-03-06T07:29:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f7affd640 1 -- 172.21.15.27:0/2922893653 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80044610 con 0x7f7f8406dd60 2024-03-06T07:29:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.514+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f840cd2c0 con 0x7f7f8406dd60 2024-03-06T07:29:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.515+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f84067ff0 con 0x7f7f8406dd60 2024-03-06T07:29:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.516+0000 7f7f7affd640 1 -- 172.21.15.27:0/2922893653 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f8004b060 con 0x7f7f8406dd60 2024-03-06T07:29:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.516+0000 7f7f7affd640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f6806b4d0 0x7f7f6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.516+0000 7f7f7affd640 1 -- 172.21.15.27:0/2922893653 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7f800ac200 con 0x7f7f8406dd60 2024-03-06T07:29:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.516+0000 7f7f89d5a640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f6806b4d0 0x7f7f6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.517+0000 7f7f89d5a640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f6806b4d0 0x7f7f6806d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7f840ade10 tx=0x7f7f74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.520+0000 7f7f7affd640 1 -- 172.21.15.27:0/2922893653 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f8003f070 con 0x7f7f8406dd60 2024-03-06T07:29:59.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.740+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f840ccdb0 con 0x7f7f8406dd60 2024-03-06T07:29:59.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.740+0000 7f7f7affd640 1 -- 172.21.15.27:0/2922893653 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f7f8007b140 con 0x7f7f8406dd60 2024-03-06T07:29:59.743 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:29:59.743 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:29:59.743 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f6806b4d0 msgr2=0x7f7f6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f6806b4d0 0x7f7f6806d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7f840ade10 tx=0x7f7f74008040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f8406dd60 msgr2=0x7f7f840b0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f8406dd60 0x7f7f840b0d30 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f7f800090b0 tx=0x7f7f800046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 shutdown_connections 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f6806b4d0 0x7f7f6806d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f84158520 0x7f7f84159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f840aec40 0x7f7f841481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 --2- 172.21.15.27:0/2922893653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f8406dd60 0x7f7f840b0d30 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 >> 172.21.15.27:0/2922893653 conn(0x7f7f84066d00 msgr2=0x7f7f8415a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 shutdown_connections 2024-03-06T07:29:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.742+0000 7f7f8c7bf640 1 -- 172.21.15.27:0/2922893653 wait complete. 2024-03-06T07:29:59.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:59.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.862+0000 7f0148cfb640 1 Processor -- start 2024-03-06T07:29:59.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.862+0000 7f0148cfb640 1 -- start start 2024-03-06T07:29:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f0148cfb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f0148cfb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f01440b95a0 0x7f01440b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f0148cfb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01440cea30 con 0x7f01440b9a30 2024-03-06T07:29:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f0148cfb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01440d2af0 con 0x7f01440b95a0 2024-03-06T07:29:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f0148cfb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01440cf3c0 con 0x7f014415df00 2024-03-06T07:29:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f014259c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f0141d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f01440b95a0 0x7f01440b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f014259c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:54916/0 (socket says 172.21.15.27:54916) 2024-03-06T07:29:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f014259c640 1 -- 172.21.15.27:0/4013462522 learned_addr learned my addr 172.21.15.27:0/4013462522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f014259c640 1 -- 172.21.15.27:0/4013462522 >> v2:172.21.15.69:3300/0 conn(0x7f01440b95a0 msgr2=0x7f01440b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f014259c640 1 --2- 172.21.15.27:0/4013462522 >> v2:172.21.15.69:3300/0 conn(0x7f01440b95a0 0x7f01440b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f014259c640 1 -- 172.21.15.27:0/4013462522 >> v1:172.21.15.69:6789/0 conn(0x7f01440b9a30 legacy=0x7f01440d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f014159a640 1 -- 172.21.15.27:0/4013462522 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941089235 0 0) 0x7f01440cea30 con 0x7f01440b9a30 2024-03-06T07:29:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f014259c640 1 -- 172.21.15.27:0/4013462522 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01440cf0e0 con 0x7f014415df00 2024-03-06T07:29:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.864+0000 7f0141d9b640 1 --2- 172.21.15.27:0/4013462522 >> v2:172.21.15.69:3300/0 conn(0x7f01440b95a0 0x7f01440b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:59.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.865+0000 7f014259c640 1 --2- 172.21.15.27:0/4013462522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440d25b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0134009870 tx=0x7f013402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e7ecdbe53f1ae27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.865+0000 7f014159a640 1 -- 172.21.15.27:0/4013462522 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0134034070 con 0x7f014415df00 2024-03-06T07:29:59.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.865+0000 7f014159a640 1 -- 172.21.15.27:0/4013462522 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0134030840 con 0x7f014415df00 2024-03-06T07:29:59.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.865+0000 7f014159a640 1 -- 172.21.15.27:0/4013462522 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0134030c10 con 0x7f014415df00 2024-03-06T07:29:59.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.866+0000 7f0148cfb640 1 -- 172.21.15.27:0/4013462522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 msgr2=0x7f01440d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.866+0000 7f0148cfb640 1 --2- 172.21.15.27:0/4013462522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440d25b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0134009870 tx=0x7f013402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:59.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.866+0000 7f0148cfb640 1 -- 172.21.15.27:0/4013462522 shutdown_connections 2024-03-06T07:29:59.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.866+0000 7f0148cfb640 1 --2- 172.21.15.27:0/4013462522 >> v2:172.21.15.69:3300/0 conn(0x7f01440b95a0 0x7f01440b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.866+0000 7f0148cfb640 1 --2- 172.21.15.27:0/4013462522 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440d25b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.866+0000 7f0148cfb640 1 -- 172.21.15.27:0/4013462522 >> 172.21.15.27:0/4013462522 conn(0x7f0144071960 msgr2=0x7f0144071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.866+0000 7f0148cfb640 1 -- 172.21.15.27:0/4013462522 shutdown_connections 2024-03-06T07:29:59.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.866+0000 7f0148cfb640 1 -- 172.21.15.27:0/4013462522 wait complete. 2024-03-06T07:29:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.868+0000 7f0148cfb640 1 Processor -- start 2024-03-06T07:29:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.869+0000 7f0148cfb640 1 -- start start 2024-03-06T07:29:59.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0148cfb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440b95a0 0x7f014415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0148cfb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0148cfb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f014415ecc0 0x7f01441602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0148cfb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01440cec70 con 0x7f014415ecc0 2024-03-06T07:29:59.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0148cfb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01440ce750 con 0x7f01440b95a0 2024-03-06T07:29:59.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0148cfb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01440cedf0 con 0x7f014415df00 2024-03-06T07:29:59.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0142d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f014415ecc0 0x7f01441602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0142d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f014415ecc0 0x7f01441602c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40470/0 (socket says 172.21.15.27:40470) 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0142d9d640 1 -- 172.21.15.27:0/3740577410 learned_addr learned my addr 172.21.15.27:0/3740577410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0141d9b640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f014259c640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440b95a0 0x7f014415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0142d9d640 1 -- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 msgr2=0x7f01440b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0142d9d640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0142d9d640 1 -- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440b95a0 msgr2=0x7f014415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0142d9d640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440b95a0 0x7f014415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.870+0000 7f0142d9d640 1 -- 172.21.15.27:0/3740577410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014415f150 con 0x7f014415ecc0 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.871+0000 7f0142d9d640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f014415ecc0 0x7f01441602c0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f013c00c730 tx=0x7f013c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.871+0000 7f0132ffd640 1 -- 172.21.15.27:0/3740577410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c003da0 con 0x7f014415ecc0 2024-03-06T07:29:59.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.871+0000 7f0132ffd640 1 -- 172.21.15.27:0/3740577410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013c00ce40 con 0x7f014415ecc0 2024-03-06T07:29:59.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.871+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014415f3f0 con 0x7f014415ecc0 2024-03-06T07:29:59.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.871+0000 7f0132ffd640 1 -- 172.21.15.27:0/3740577410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c0052b0 con 0x7f014415ecc0 2024-03-06T07:29:59.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.871+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014415f9b0 con 0x7f014415ecc0 2024-03-06T07:29:59.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.872+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014415fbf0 con 0x7f014415ecc0 2024-03-06T07:29:59.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.872+0000 7f0132ffd640 1 -- 172.21.15.27:0/3740577410 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f013c005450 con 0x7f014415ecc0 2024-03-06T07:29:59.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.873+0000 7f0132ffd640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b500 0x7f012406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.873+0000 7f0132ffd640 1 -- 172.21.15.27:0/3740577410 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f013c08a160 con 0x7f014415ecc0 2024-03-06T07:29:59.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.873+0000 7f014259c640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b500 0x7f012406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.874+0000 7f014259c640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b500 0x7f012406d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0134036000 tx=0x7f0134008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:29:59.877+0000 7f0132ffd640 1 -- 172.21.15.27:0/3740577410 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f014415fbf0 con 0x7f014415ecc0 2024-03-06T07:30:00.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.063+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f014415d390 con 0x7f014415ecc0 2024-03-06T07:30:00.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.064+0000 7f0132ffd640 1 -- 172.21.15.27:0/3740577410 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16967 (secure 0 0 0) 0x7f013c058820 con 0x7f014415ecc0 2024-03-06T07:30:00.067 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:00.067 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":120,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:29:29.730083+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.260334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.929245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6832/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6834/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6835/1655684069":"2024-03-07T07:28:59.663475+0000","172.21.15.27:6833/1655684069":"2024-03-07T07:28:59.663475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:00.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b500 msgr2=0x7f012406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b500 0x7f012406d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0134036000 tx=0x7f0134008030 comp rx=0 tx=0).stop 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f014415ecc0 msgr2=0x7f01441602c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f014415ecc0 0x7f01441602c0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f013c00c730 tx=0x7f013c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 shutdown_connections 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b500 0x7f012406d9c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f014415ecc0 0x7f01441602c0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415df00 0x7f01440b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 --2- 172.21.15.27:0/3740577410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440b95a0 0x7f014415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 >> 172.21.15.27:0/3740577410 conn(0x7f0144071960 msgr2=0x7f01441618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 shutdown_connections 2024-03-06T07:30:00.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.067+0000 7f0148cfb640 1 -- 172.21.15.27:0/3740577410 wait complete. 2024-03-06T07:30:00.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:39.260334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:39.929245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:00.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:00.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.189+0000 7fc0eb555640 1 Processor -- start 2024-03-06T07:30:00.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.190+0000 7fc0eb555640 1 -- start start 2024-03-06T07:30:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.190+0000 7fc0eb555640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0e4144640 0x7fc0e406e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.190+0000 7fc0eb555640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0e40ce900 0x7fc0e40ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.190+0000 7fc0eb555640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e415bc40 con 0x7fc0e406e940 2024-03-06T07:30:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.190+0000 7fc0eb555640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e406df70 con 0x7fc0e40ce900 2024-03-06T07:30:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.190+0000 7fc0eb555640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e40cf270 con 0x7fc0e4144640 2024-03-06T07:30:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.191+0000 7fc0e9af2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0e40ce900 0x7fc0e40ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.191+0000 7fc0e9af2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0e40ce900 0x7fc0e40ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40484/0 (socket says 172.21.15.27:40484) 2024-03-06T07:30:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.191+0000 7fc0e9af2640 1 -- 172.21.15.27:0/2145304619 learned_addr learned my addr 172.21.15.27:0/2145304619 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:00.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.191+0000 7fc0e92f1640 1 --2- 172.21.15.27:0/2145304619 >> v2:172.21.15.69:3300/0 conn(0x7fc0e4144640 0x7fc0e406e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.191+0000 7fc0e9af2640 1 -- 172.21.15.27:0/2145304619 >> v2:172.21.15.69:3300/0 conn(0x7fc0e4144640 msgr2=0x7fc0e406e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.191+0000 7fc0e9af2640 1 --2- 172.21.15.27:0/2145304619 >> v2:172.21.15.69:3300/0 conn(0x7fc0e4144640 0x7fc0e406e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.191+0000 7fc0e9af2640 1 -- 172.21.15.27:0/2145304619 >> v1:172.21.15.69:6789/0 conn(0x7fc0e406e940 legacy=0x7fc0e415b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.191+0000 7fc0e9af2640 1 -- 172.21.15.27:0/2145304619 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e415bf20 con 0x7fc0e40ce900 2024-03-06T07:30:00.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.191+0000 7fc0e92f1640 1 --2- 172.21.15.27:0/2145304619 >> v2:172.21.15.69:3300/0 conn(0x7fc0e4144640 0x7fc0e406e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.192+0000 7fc0e9af2640 1 --2- 172.21.15.27:0/2145304619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e40ce900 0x7fc0e40ced30 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fc0e406d360 tx=0x7fc0dc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62b1b62177bc7f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.192+0000 7fc0dbfff640 1 -- 172.21.15.27:0/2145304619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc02ca80 con 0x7fc0e40ce900 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.192+0000 7fc0dbfff640 1 -- 172.21.15.27:0/2145304619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0dc02cc20 con 0x7fc0e40ce900 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.192+0000 7fc0dbfff640 1 -- 172.21.15.27:0/2145304619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc0343f0 con 0x7fc0e40ce900 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.193+0000 7fc0eb555640 1 -- 172.21.15.27:0/2145304619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e40ce900 msgr2=0x7fc0e40ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.193+0000 7fc0eb555640 1 --2- 172.21.15.27:0/2145304619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e40ce900 0x7fc0e40ced30 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fc0e406d360 tx=0x7fc0dc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.193+0000 7fc0eb555640 1 -- 172.21.15.27:0/2145304619 shutdown_connections 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.193+0000 7fc0eb555640 1 --2- 172.21.15.27:0/2145304619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e40ce900 0x7fc0e40ced30 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.193+0000 7fc0eb555640 1 --2- 172.21.15.27:0/2145304619 >> v2:172.21.15.69:3300/0 conn(0x7fc0e4144640 0x7fc0e406e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.193+0000 7fc0eb555640 1 -- 172.21.15.27:0/2145304619 >> 172.21.15.27:0/2145304619 conn(0x7fc0e4066d00 msgr2=0x7fc0e4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.193+0000 7fc0eb555640 1 -- 172.21.15.27:0/2145304619 shutdown_connections 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.193+0000 7fc0eb555640 1 -- 172.21.15.27:0/2145304619 wait complete. 2024-03-06T07:30:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.194+0000 7fc0eb555640 1 Processor -- start 2024-03-06T07:30:00.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.194+0000 7fc0eb555640 1 -- start start 2024-03-06T07:30:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0eb555640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e406e940 0x7fc0e41f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0eb555640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e40ce900 0x7fc0e4159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0eb555640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e4144640 0x7fc0e415a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0eb555640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e4070c20 con 0x7fc0e406e940 2024-03-06T07:30:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0eb555640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e4070aa0 con 0x7fc0e4144640 2024-03-06T07:30:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0eb555640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e415b840 con 0x7fc0e40ce900 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e92f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e406e940 0x7fc0e41f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e92f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e406e940 0x7fc0e41f4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40496/0 (socket says 172.21.15.27:40496) 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e92f1640 1 -- 172.21.15.27:0/622630206 learned_addr learned my addr 172.21.15.27:0/622630206 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e8af0640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e40ce900 0x7fc0e4159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e9af2640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e4144640 0x7fc0e415a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e92f1640 1 -- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e40ce900 msgr2=0x7fc0e4159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e92f1640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e40ce900 0x7fc0e4159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e92f1640 1 -- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e4144640 msgr2=0x7fc0e415a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e92f1640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e4144640 0x7fc0e415a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e92f1640 1 -- 172.21.15.27:0/622630206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e4139820 con 0x7fc0e406e940 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.195+0000 7fc0e9af2640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e4144640 0x7fc0e415a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.196+0000 7fc0e92f1640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e406e940 0x7fc0e41f4e30 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000c7f0 tx=0x7fc0e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.196+0000 7fc0d9ffb640 1 -- 172.21.15.27:0/622630206 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0015070 con 0x7fc0e406e940 2024-03-06T07:30:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.196+0000 7fc0d9ffb640 1 -- 172.21.15.27:0/622630206 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e0007820 con 0x7fc0e406e940 2024-03-06T07:30:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.196+0000 7fc0d9ffb640 1 -- 172.21.15.27:0/622630206 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0007bd0 con 0x7fc0e406e940 2024-03-06T07:30:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.196+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e4139ad0 con 0x7fc0e406e940 2024-03-06T07:30:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.196+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e413a090 con 0x7fc0e406e940 2024-03-06T07:30:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.197+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e40733c0 con 0x7fc0e406e940 2024-03-06T07:30:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.197+0000 7fc0d9ffb640 1 -- 172.21.15.27:0/622630206 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0e0018070 con 0x7fc0e406e940 2024-03-06T07:30:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.198+0000 7fc0d9ffb640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc0c806b4d0 0x7fc0c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.198+0000 7fc0d9ffb640 1 -- 172.21.15.27:0/622630206 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc0e008a2e0 con 0x7fc0e406e940 2024-03-06T07:30:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.198+0000 7fc0e8af0640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc0c806b4d0 0x7fc0c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.199+0000 7fc0e8af0640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc0c806b4d0 0x7fc0c806d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc0e413c0d0 tx=0x7fc0d4007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.206+0000 7fc0d9ffb640 1 -- 172.21.15.27:0/622630206 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0e0058ab0 con 0x7fc0e406e940 2024-03-06T07:30:00.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.493+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0e40cd1e0 con 0x7fc0e406e940 2024-03-06T07:30:00.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.494+0000 7fc0d9ffb640 1 -- 172.21.15.27:0/622630206 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0e005cb50 con 0x7fc0e406e940 2024-03-06T07:30:00.496 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:30:00.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc0c806b4d0 msgr2=0x7fc0c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc0c806b4d0 0x7fc0c806d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc0e413c0d0 tx=0x7fc0d4007e30 comp rx=0 tx=0).stop 2024-03-06T07:30:00.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e406e940 msgr2=0x7fc0e41f4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e406e940 0x7fc0e41f4e30 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000c7f0 tx=0x7fc0e000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 shutdown_connections 2024-03-06T07:30:00.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc0c806b4d0 0x7fc0c806d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0e4144640 0x7fc0e415a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0e40ce900 0x7fc0e4159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 --2- 172.21.15.27:0/622630206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e406e940 0x7fc0e41f4e30 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 >> 172.21.15.27:0/622630206 conn(0x7fc0e4066d00 msgr2=0x7fc0e40670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 shutdown_connections 2024-03-06T07:30:00.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.496+0000 7fc0eb555640 1 -- 172.21.15.27:0/622630206 wait complete. 2024-03-06T07:30:00.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:00.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.626+0000 7f2af3fc7640 1 Processor -- start 2024-03-06T07:30:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.626+0000 7f2af3fc7640 1 -- start start 2024-03-06T07:30:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.627+0000 7f2af3fc7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2aec0717b0 0x7f2aec071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.627+0000 7f2af3fc7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2aec0b3060 0x7f2aec0d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.627+0000 7f2af3fc7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec15e4b0 con 0x7f2aec0b2b10 2024-03-06T07:30:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.627+0000 7f2af3fc7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aec0d2e10 con 0x7f2aec0b3060 2024-03-06T07:30:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.627+0000 7f2af3fc7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aec0b97e0 con 0x7f2aec0717b0 2024-03-06T07:30:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.627+0000 7f2af1562640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2aec0b3060 0x7f2aec0d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.627+0000 7f2af1562640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2aec0b3060 0x7f2aec0d32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40498/0 (socket says 172.21.15.27:40498) 2024-03-06T07:30:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.627+0000 7f2af1562640 1 -- 172.21.15.27:0/1378431635 learned_addr learned my addr 172.21.15.27:0/1378431635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.628+0000 7f2af1d63640 1 --2- 172.21.15.27:0/1378431635 >> v2:172.21.15.69:3300/0 conn(0x7f2aec0717b0 0x7f2aec071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.628+0000 7f2af1562640 1 -- 172.21.15.27:0/1378431635 >> v2:172.21.15.69:3300/0 conn(0x7f2aec0717b0 msgr2=0x7f2aec071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.628+0000 7f2af1562640 1 --2- 172.21.15.27:0/1378431635 >> v2:172.21.15.69:3300/0 conn(0x7f2aec0717b0 0x7f2aec071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.628+0000 7f2af1562640 1 -- 172.21.15.27:0/1378431635 >> v1:172.21.15.69:6789/0 conn(0x7f2aec0b2b10 legacy=0x7f2aec0b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.628+0000 7f2af1562640 1 -- 172.21.15.27:0/1378431635 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec0b9a80 con 0x7f2aec0b3060 2024-03-06T07:30:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.628+0000 7f2af1562640 1 --2- 172.21.15.27:0/1378431635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b3060 0x7f2aec0d32a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f2aec04fd20 tx=0x7f2adc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399ebe9b636d69da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af0d61640 1 -- 172.21.15.27:0/1378431635 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc034070 con 0x7f2aec0b3060 2024-03-06T07:30:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af0d61640 1 -- 172.21.15.27:0/1378431635 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc030860 con 0x7f2aec0b3060 2024-03-06T07:30:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af0d61640 1 -- 172.21.15.27:0/1378431635 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc030c30 con 0x7f2aec0b3060 2024-03-06T07:30:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af3fc7640 1 -- 172.21.15.27:0/1378431635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b3060 msgr2=0x7f2aec0d32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af3fc7640 1 --2- 172.21.15.27:0/1378431635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b3060 0x7f2aec0d32a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f2aec04fd20 tx=0x7f2adc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af3fc7640 1 -- 172.21.15.27:0/1378431635 shutdown_connections 2024-03-06T07:30:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af3fc7640 1 --2- 172.21.15.27:0/1378431635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b3060 0x7f2aec0d32a0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af3fc7640 1 --2- 172.21.15.27:0/1378431635 >> v2:172.21.15.69:3300/0 conn(0x7f2aec0717b0 0x7f2aec071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af3fc7640 1 -- 172.21.15.27:0/1378431635 >> 172.21.15.27:0/1378431635 conn(0x7f2aec0d3d20 msgr2=0x7f2aec0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af3fc7640 1 -- 172.21.15.27:0/1378431635 shutdown_connections 2024-03-06T07:30:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.629+0000 7f2af3fc7640 1 -- 172.21.15.27:0/1378431635 wait complete. 2024-03-06T07:30:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.630+0000 7f2af3fc7640 1 Processor -- start 2024-03-06T07:30:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.630+0000 7f2af3fc7640 1 -- start start 2024-03-06T07:30:00.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.631+0000 7f2af3fc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aec0717b0 0x7f2aec1f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.631+0000 7f2af3fc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b2b10 0x7f2aec156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.631+0000 7f2af3fc7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aec0b3060 0x7f2aec0ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.631+0000 7f2af3fc7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0b7560 con 0x7f2aec0b2b10 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.631+0000 7f2af3fc7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0d4bf0 con 0x7f2aec0b3060 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.631+0000 7f2af3fc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0b76e0 con 0x7f2aec0717b0 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.631+0000 7f2af1562640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b2b10 0x7f2aec156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.631+0000 7f2af1562640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b2b10 0x7f2aec156c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40512/0 (socket says 172.21.15.27:40512) 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.631+0000 7f2af1562640 1 -- 172.21.15.27:0/4053490371 learned_addr learned my addr 172.21.15.27:0/4053490371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af1d63640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aec0717b0 0x7f2aec1f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af2564640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aec0b3060 0x7f2aec0ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af1562640 1 -- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aec0717b0 msgr2=0x7f2aec1f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af1562640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aec0717b0 0x7f2aec1f3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af1562640 1 -- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aec0b3060 msgr2=0x7f2aec0ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af1562640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aec0b3060 0x7f2aec0ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af1562640 1 -- 172.21.15.27:0/4053490371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec0c8320 con 0x7f2aec0b2b10 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af1562640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b2b10 0x7f2aec156c00 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f2adc02b9e0 tx=0x7f2adc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2ae27fc640 1 -- 172.21.15.27:0/4053490371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc034040 con 0x7f2aec0b2b10 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2ae27fc640 1 -- 172.21.15.27:0/4053490371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc039d90 con 0x7f2aec0b2b10 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec0c85c0 con 0x7f2aec0b2b10 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2ae27fc640 1 -- 172.21.15.27:0/4053490371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc043630 con 0x7f2aec0b2b10 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.632+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec0c8b80 con 0x7f2aec0b2b10 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.633+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aec06ce40 con 0x7f2aec0b2b10 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.634+0000 7f2ae27fc640 1 -- 172.21.15.27:0/4053490371 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2adc037070 con 0x7f2aec0b2b10 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.634+0000 7f2ae27fc640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2ad006b4d0 0x7f2ad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.634+0000 7f2ae27fc640 1 -- 172.21.15.27:0/4053490371 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f2adc0acc80 con 0x7f2aec0b2b10 2024-03-06T07:30:00.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.634+0000 7f2af1d63640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2ad006b4d0 0x7f2ad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.635+0000 7f2af1d63640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2ad006b4d0 0x7f2ad006d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4006fd0 tx=0x7f2ae4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.638+0000 7f2ae27fc640 1 -- 172.21.15.27:0/4053490371 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2adc03e070 con 0x7f2aec0b2b10 2024-03-06T07:30:00.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.874+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2aec0b8a80 con 0x7f2aec0b2b10 2024-03-06T07:30:00.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.874+0000 7f2ae27fc640 1 -- 172.21.15.27:0/4053490371 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2adc07baf0 con 0x7f2aec0b2b10 2024-03-06T07:30:00.877 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:30:00.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2ad006b4d0 msgr2=0x7f2ad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2ad006b4d0 0x7f2ad006d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4006fd0 tx=0x7f2ae4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:00.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b2b10 msgr2=0x7f2aec156c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b2b10 0x7f2aec156c00 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f2adc02b9e0 tx=0x7f2adc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 shutdown_connections 2024-03-06T07:30:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2ad006b4d0 0x7f2ad006d990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aec0b3060 0x7f2aec0ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec0b2b10 0x7f2aec156c00 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 --2- 172.21.15.27:0/4053490371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aec0717b0 0x7f2aec1f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.876+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 >> 172.21.15.27:0/4053490371 conn(0x7f2aec0d3d20 msgr2=0x7f2aec0ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.877+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 shutdown_connections 2024-03-06T07:30:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:00.877+0000 7f2af3fc7640 1 -- 172.21.15.27:0/4053490371 wait complete. 2024-03-06T07:30:00.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.018+0000 7fb53cac2640 1 Processor -- start 2024-03-06T07:30:01.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.018+0000 7fb53cac2640 1 -- start start 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb53cac2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5380b7420 0x7fb5380d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb53cac2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5380d3740 con 0x7fb5380ceea0 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb53cac2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5380d3120 con 0x7fb53815dd90 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb53cac2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5380d3400 con 0x7fb5380b7420 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb536d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5380b7420 0x7fb5380d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb52ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb5380ceea0 0x7fb5380b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34130/0 (socket says 172.21.15.27:34130) 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb52ffff640 1 -- 172.21.15.27:0/2069068733 learned_addr learned my addr 172.21.15.27:0/2069068733 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb536d9d640 1 -- 172.21.15.27:0/2069068733 >> v1:172.21.15.69:6789/0 conn(0x7fb53815dd90 legacy=0x7fb5380d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb536d9d640 1 -- 172.21.15.27:0/2069068733 >> v1:172.21.15.27:6789/0 conn(0x7fb5380ceea0 legacy=0x7fb5380b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb536d9d640 1 -- 172.21.15.27:0/2069068733 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380d39e0 con 0x7fb5380b7420 2024-03-06T07:30:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.020+0000 7fb535d9b640 1 -- 172.21.15.27:0/2069068733 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629468585 0 0) 0x7fb5380d3740 con 0x7fb5380ceea0 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.021+0000 7fb536d9d640 1 --2- 172.21.15.27:0/2069068733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5380b7420 0x7fb5380d29d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb530009d90 tx=0x7fb53002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c79d18c518f756 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.021+0000 7fb535d9b640 1 -- 172.21.15.27:0/2069068733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530039070 con 0x7fb5380b7420 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.021+0000 7fb535d9b640 1 -- 172.21.15.27:0/2069068733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53002cd00 con 0x7fb5380b7420 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.021+0000 7fb535d9b640 1 -- 172.21.15.27:0/2069068733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530034660 con 0x7fb5380b7420 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.021+0000 7fb53cac2640 1 -- 172.21.15.27:0/2069068733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5380b7420 msgr2=0x7fb5380d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.021+0000 7fb53cac2640 1 --2- 172.21.15.27:0/2069068733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5380b7420 0x7fb5380d29d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb530009d90 tx=0x7fb53002bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.022+0000 7fb53cac2640 1 -- 172.21.15.27:0/2069068733 shutdown_connections 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.022+0000 7fb53cac2640 1 --2- 172.21.15.27:0/2069068733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5380b7420 0x7fb5380d29d0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.022+0000 7fb53cac2640 1 -- 172.21.15.27:0/2069068733 >> 172.21.15.27:0/2069068733 conn(0x7fb538071960 msgr2=0x7fb538071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.022+0000 7fb53cac2640 1 -- 172.21.15.27:0/2069068733 shutdown_connections 2024-03-06T07:30:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.022+0000 7fb53cac2640 1 -- 172.21.15.27:0/2069068733 wait complete. 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.023+0000 7fb53cac2640 1 Processor -- start 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.023+0000 7fb53cac2640 1 -- start start 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb53cac2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5380ceea0 0x7fb538149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb53cac2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53815dd90 0x7fb53815e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb53cac2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb538160fe0 0x7fb5380c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb53cac2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380d48f0 con 0x7fb53815dd90 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb53cac2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53815e170 con 0x7fb538160fe0 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb53cac2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380d4a70 con 0x7fb5380ceea0 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53815dd90 0x7fb53815e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53815dd90 0x7fb53815e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40532/0 (socket says 172.21.15.27:40532) 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb52ffff640 1 -- 172.21.15.27:0/3087149149 learned_addr learned my addr 172.21.15.27:0/3087149149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb53659c640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5380ceea0 0x7fb538149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb536d9d640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb538160fe0 0x7fb5380c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb52ffff640 1 -- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5380ceea0 msgr2=0x7fb538149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb52ffff640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5380ceea0 0x7fb538149600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb52ffff640 1 -- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb538160fe0 msgr2=0x7fb5380c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb52ffff640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb538160fe0 0x7fb5380c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb52ffff640 1 -- 172.21.15.27:0/3087149149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53815ff80 con 0x7fb53815dd90 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.024+0000 7fb52ffff640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53815dd90 0x7fb53815e380 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb52000c490 tx=0x7fb52000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.025+0000 7fb52effd640 1 -- 172.21.15.27:0/3087149149 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520005ce0 con 0x7fb53815dd90 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.025+0000 7fb52effd640 1 -- 172.21.15.27:0/3087149149 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520007070 con 0x7fb53815dd90 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.025+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb538160220 con 0x7fb53815dd90 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.025+0000 7fb52effd640 1 -- 172.21.15.27:0/3087149149 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520007460 con 0x7fb53815dd90 2024-03-06T07:30:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.025+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5381607e0 con 0x7fb53815dd90 2024-03-06T07:30:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.026+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb538072a30 con 0x7fb53815dd90 2024-03-06T07:30:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.027+0000 7fb52effd640 1 -- 172.21.15.27:0/3087149149 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb520007600 con 0x7fb53815dd90 2024-03-06T07:30:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.027+0000 7fb52effd640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb51806b4d0 0x7fb51806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.027+0000 7fb52effd640 1 -- 172.21.15.27:0/3087149149 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb520089d30 con 0x7fb53815dd90 2024-03-06T07:30:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.027+0000 7fb53659c640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb51806b4d0 0x7fb51806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.028+0000 7fb53659c640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb51806b4d0 0x7fb51806d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb528000f60 tx=0x7fb528009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.032+0000 7fb52effd640 1 -- 172.21.15.27:0/3087149149 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb520058500 con 0x7fb53815dd90 2024-03-06T07:30:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.213+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb5380d2630 con 0x7fb53815dd90 2024-03-06T07:30:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.213+0000 7fb52effd640 1 -- 172.21.15.27:0/3087149149 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb52005c5a0 con 0x7fb53815dd90 2024-03-06T07:30:01.216 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:30:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb51806b4d0 msgr2=0x7fb51806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb51806b4d0 0x7fb51806d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb528000f60 tx=0x7fb528009040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53815dd90 msgr2=0x7fb53815e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53815dd90 0x7fb53815e380 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb52000c490 tx=0x7fb52000c950 comp rx=0 tx=0).stop 2024-03-06T07:30:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 shutdown_connections 2024-03-06T07:30:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb51806b4d0 0x7fb51806d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb538160fe0 0x7fb5380c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53815dd90 0x7fb53815e380 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 --2- 172.21.15.27:0/3087149149 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5380ceea0 0x7fb538149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 >> 172.21.15.27:0/3087149149 conn(0x7fb538071960 msgr2=0x7fb5380b96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 shutdown_connections 2024-03-06T07:30:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.215+0000 7fb53cac2640 1 -- 172.21.15.27:0/3087149149 wait complete. 2024-03-06T07:30:01.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:01.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.333+0000 7efcb2302640 1 Processor -- start 2024-03-06T07:30:01.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.333+0000 7efcb2302640 1 -- start start 2024-03-06T07:30:01.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb2302640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efcac071540 0x7efcac0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb2302640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac0b2b50 0x7efcac0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb2302640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcac0ce610 con 0x7efcac0d3400 2024-03-06T07:30:01.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb2302640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcac0d2f70 con 0x7efcac071540 2024-03-06T07:30:01.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb2302640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcac0d4be0 con 0x7efcac0b2b50 2024-03-06T07:30:01.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcab7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efcac0d3400 0x7efcac071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34134/0 (socket says 172.21.15.27:34134) 2024-03-06T07:30:01.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcab7fe640 1 -- 172.21.15.27:0/2255140266 learned_addr learned my addr 172.21.15.27:0/2255140266 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:01.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb089f640 1 --2- 172.21.15.27:0/2255140266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac0b2b50 0x7efcac0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcabfff640 1 --2- 172.21.15.27:0/2255140266 >> v2:172.21.15.69:3300/0 conn(0x7efcac071540 0x7efcac0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcaaffd640 1 -- 172.21.15.27:0/2255140266 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480094434 0 0) 0x7efcac0ce610 con 0x7efcac0d3400 2024-03-06T07:30:01.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcaaffd640 1 -- 172.21.15.27:0/2255140266 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc94003520 con 0x7efcac0d3400 2024-03-06T07:30:01.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb089f640 1 -- 172.21.15.27:0/2255140266 >> v2:172.21.15.69:3300/0 conn(0x7efcac071540 msgr2=0x7efcac0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb089f640 1 --2- 172.21.15.27:0/2255140266 >> v2:172.21.15.69:3300/0 conn(0x7efcac071540 0x7efcac0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb089f640 1 -- 172.21.15.27:0/2255140266 >> v1:172.21.15.27:6789/0 conn(0x7efcac0d3400 legacy=0x7efcac071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.334+0000 7efcb089f640 1 -- 172.21.15.27:0/2255140266 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac0d2920 con 0x7efcac0b2b50 2024-03-06T07:30:01.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.335+0000 7efcb089f640 1 --2- 172.21.15.27:0/2255140266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac0b2b50 0x7efcac0b6e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efca4009d90 tx=0x7efca402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2205bd9ec5b19183 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.335+0000 7efcaaffd640 1 -- 172.21.15.27:0/2255140266 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4039070 con 0x7efcac0b2b50 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.335+0000 7efcaaffd640 1 -- 172.21.15.27:0/2255140266 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca402cc50 con 0x7efcac0b2b50 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.335+0000 7efcaaffd640 1 -- 172.21.15.27:0/2255140266 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4034440 con 0x7efcac0b2b50 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.336+0000 7efcb2302640 1 -- 172.21.15.27:0/2255140266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac0b2b50 msgr2=0x7efcac0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.336+0000 7efcb2302640 1 --2- 172.21.15.27:0/2255140266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac0b2b50 0x7efcac0b6e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efca4009d90 tx=0x7efca402bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.336+0000 7efcb2302640 1 -- 172.21.15.27:0/2255140266 shutdown_connections 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.336+0000 7efcb2302640 1 --2- 172.21.15.27:0/2255140266 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac0b2b50 0x7efcac0b6e20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.336+0000 7efcb2302640 1 --2- 172.21.15.27:0/2255140266 >> v2:172.21.15.69:3300/0 conn(0x7efcac071540 0x7efcac0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.336+0000 7efcb2302640 1 -- 172.21.15.27:0/2255140266 >> 172.21.15.27:0/2255140266 conn(0x7efcac0d3ca0 msgr2=0x7efcac0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.336+0000 7efcb2302640 1 -- 172.21.15.27:0/2255140266 shutdown_connections 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.336+0000 7efcb2302640 1 -- 172.21.15.27:0/2255140266 wait complete. 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.337+0000 7efcb2302640 1 Processor -- start 2024-03-06T07:30:01.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.337+0000 7efcb2302640 1 -- start start 2024-03-06T07:30:01.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcb2302640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac071540 0x7efcac14b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcb2302640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcac0b2b50 0x7efcac0b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcb2302640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac0d3400 0x7efcac0b8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcb2302640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac15e370 con 0x7efcac0b2b50 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcb2302640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac0ce8f0 con 0x7efcac0d3400 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcb2302640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcac0c85e0 con 0x7efcac071540 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcac0b2b50 0x7efcac0b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcabfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac071540 0x7efcac14b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcac0b2b50 0x7efcac0b7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40534/0 (socket says 172.21.15.27:40534) 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcab7fe640 1 -- 172.21.15.27:0/1663681979 learned_addr learned my addr 172.21.15.27:0/1663681979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.338+0000 7efcb089f640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac0d3400 0x7efcac0b8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efcab7fe640 1 -- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac071540 msgr2=0x7efcac14b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efcab7fe640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac071540 0x7efcac14b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efcab7fe640 1 -- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac0d3400 msgr2=0x7efcac0b8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efcab7fe640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac0d3400 0x7efcac0b8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efcab7fe640 1 -- 172.21.15.27:0/1663681979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac0cb0a0 con 0x7efcac0b2b50 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efcabfff640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac071540 0x7efcac14b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efcab7fe640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcac0b2b50 0x7efcac0b7de0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7efc9c001eb0 tx=0x7efc9c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efca8ff9640 1 -- 172.21.15.27:0/1663681979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c017070 con 0x7efcac0b2b50 2024-03-06T07:30:01.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efca8ff9640 1 -- 172.21.15.27:0/1663681979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9c005d00 con 0x7efcac0b2b50 2024-03-06T07:30:01.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efca8ff9640 1 -- 172.21.15.27:0/1663681979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c007310 con 0x7efcac0b2b50 2024-03-06T07:30:01.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcac0c8da0 con 0x7efcac0b2b50 2024-03-06T07:30:01.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.339+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcac0c8300 con 0x7efcac0b2b50 2024-03-06T07:30:01.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.340+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcac068700 con 0x7efcac0b2b50 2024-03-06T07:30:01.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.341+0000 7efca8ff9640 1 -- 172.21.15.27:0/1663681979 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc9c01a070 con 0x7efcac0b2b50 2024-03-06T07:30:01.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.341+0000 7efca8ff9640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc9806b4d0 0x7efc9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.341+0000 7efca8ff9640 1 -- 172.21.15.27:0/1663681979 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7efc9c089610 con 0x7efcac0b2b50 2024-03-06T07:30:01.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.341+0000 7efcabfff640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc9806b4d0 0x7efc9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.342+0000 7efcabfff640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc9806b4d0 0x7efc9806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efca0009f30 tx=0x7efca0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.345+0000 7efca8ff9640 1 -- 172.21.15.27:0/1663681979 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc9c057d50 con 0x7efcac0b2b50 2024-03-06T07:30:01.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.565+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcac06b8e0 con 0x7efcac0b2b50 2024-03-06T07:30:01.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.565+0000 7efca8ff9640 1 -- 172.21.15.27:0/1663681979 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7efc9c05bdf0 con 0x7efcac0b2b50 2024-03-06T07:30:01.568 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:01.568 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:30:01.568 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc9806b4d0 msgr2=0x7efc9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc9806b4d0 0x7efc9806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efca0009f30 tx=0x7efca0009190 comp rx=0 tx=0).stop 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcac0b2b50 msgr2=0x7efcac0b7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcac0b2b50 0x7efcac0b7de0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7efc9c001eb0 tx=0x7efc9c00c820 comp rx=0 tx=0).stop 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 shutdown_connections 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efc9806b4d0 0x7efc9806d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac0d3400 0x7efcac0b8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcac0b2b50 0x7efcac0b7de0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 --2- 172.21.15.27:0/1663681979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efcac071540 0x7efcac14b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 >> 172.21.15.27:0/1663681979 conn(0x7efcac0d3ca0 msgr2=0x7efcac15e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 shutdown_connections 2024-03-06T07:30:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.567+0000 7efcb2302640 1 -- 172.21.15.27:0/1663681979 wait complete. 2024-03-06T07:30:01.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:01.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.688+0000 7f79ce088640 1 Processor -- start 2024-03-06T07:30:01.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.688+0000 7f79ce088640 1 -- start start 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79ce088640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79c815df00 0x7f79c80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79ce088640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79c80b95a0 0x7f79c80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79ce088640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c80cea30 con 0x7f79c80b9a30 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79ce088640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c80d2af0 con 0x7f79c815df00 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79ce088640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c80cf3c0 con 0x7f79c80b95a0 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79c77fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79c815df00 0x7f79c80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79c7fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f79c80b9a30 0x7f79c80d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34148/0 (socket says 172.21.15.27:34148) 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79c7fff640 1 -- 172.21.15.27:0/3626403152 learned_addr learned my addr 172.21.15.27:0/3626403152 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79c6ffd640 1 --2- 172.21.15.27:0/3626403152 >> v2:172.21.15.69:3300/0 conn(0x7f79c80b95a0 0x7f79c80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79c77fe640 1 -- 172.21.15.27:0/3626403152 >> v2:172.21.15.69:3300/0 conn(0x7f79c80b95a0 msgr2=0x7f79c80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79c77fe640 1 --2- 172.21.15.27:0/3626403152 >> v2:172.21.15.69:3300/0 conn(0x7f79c80b95a0 0x7f79c80b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79c77fe640 1 -- 172.21.15.27:0/3626403152 >> v1:172.21.15.27:6789/0 conn(0x7f79c80b9a30 legacy=0x7f79c80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.689+0000 7f79c77fe640 1 -- 172.21.15.27:0/3626403152 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c80cf0e0 con 0x7f79c815df00 2024-03-06T07:30:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79c77fe640 1 --2- 172.21.15.27:0/3626403152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c815df00 0x7f79c80d25b0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f79c80d1760 tx=0x7f79b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f6d6b1873a52d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79c67fc640 1 -- 172.21.15.27:0/3626403152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8039070 con 0x7f79c815df00 2024-03-06T07:30:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79c67fc640 1 -- 172.21.15.27:0/3626403152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b802cc80 con 0x7f79c815df00 2024-03-06T07:30:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79c67fc640 1 -- 172.21.15.27:0/3626403152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8034660 con 0x7f79c815df00 2024-03-06T07:30:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79ce088640 1 -- 172.21.15.27:0/3626403152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c815df00 msgr2=0x7f79c80d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79ce088640 1 --2- 172.21.15.27:0/3626403152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c815df00 0x7f79c80d25b0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f79c80d1760 tx=0x7f79b802bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79ce088640 1 -- 172.21.15.27:0/3626403152 shutdown_connections 2024-03-06T07:30:01.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79ce088640 1 --2- 172.21.15.27:0/3626403152 >> v2:172.21.15.69:3300/0 conn(0x7f79c80b95a0 0x7f79c80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79ce088640 1 --2- 172.21.15.27:0/3626403152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c815df00 0x7f79c80d25b0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79ce088640 1 -- 172.21.15.27:0/3626403152 >> 172.21.15.27:0/3626403152 conn(0x7f79c8071960 msgr2=0x7f79c8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79ce088640 1 -- 172.21.15.27:0/3626403152 shutdown_connections 2024-03-06T07:30:01.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.691+0000 7f79ce088640 1 -- 172.21.15.27:0/3626403152 wait complete. 2024-03-06T07:30:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.692+0000 7f79ce088640 1 Processor -- start 2024-03-06T07:30:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79ce088640 1 -- start start 2024-03-06T07:30:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79ce088640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c80b95a0 0x7f79c815aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79ce088640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c815df00 0x7f79c81f3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79ce088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79c81f3ba0 0x7f79c80ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79ce088640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c80cec70 con 0x7f79c80b95a0 2024-03-06T07:30:01.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79ce088640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c80ce750 con 0x7f79c815df00 2024-03-06T07:30:01.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79ce088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c80cedf0 con 0x7f79c81f3ba0 2024-03-06T07:30:01.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c80b95a0 0x7f79c815aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c80b95a0 0x7f79c815aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40566/0 (socket says 172.21.15.27:40566) 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.693+0000 7f79c77fe640 1 -- 172.21.15.27:0/1689654736 learned_addr learned my addr 172.21.15.27:0/1689654736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79c77fe640 1 -- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79c81f3ba0 msgr2=0x7f79c80ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79c77fe640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79c81f3ba0 0x7f79c80ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79c77fe640 1 -- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c815df00 msgr2=0x7f79c81f3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79c77fe640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c815df00 0x7f79c81f3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79c77fe640 1 -- 172.21.15.27:0/1689654736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c80c7f10 con 0x7f79c80b95a0 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79c77fe640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c80b95a0 0x7f79c815aed0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f79b8037040 tx=0x7f79b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79affff640 1 -- 172.21.15.27:0/1689654736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8039050 con 0x7f79c80b95a0 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79affff640 1 -- 172.21.15.27:0/1689654736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b8045070 con 0x7f79c80b95a0 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c815f280 con 0x7f79c80b95a0 2024-03-06T07:30:01.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79affff640 1 -- 172.21.15.27:0/1689654736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b803e850 con 0x7f79c80b95a0 2024-03-06T07:30:01.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.694+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c81f4590 con 0x7f79c80b95a0 2024-03-06T07:30:01.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.695+0000 7f79affff640 1 -- 172.21.15.27:0/1689654736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79b803e9f0 con 0x7f79c80b95a0 2024-03-06T07:30:01.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.696+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c8067ff0 con 0x7f79c80b95a0 2024-03-06T07:30:01.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.696+0000 7f79affff640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f79b406b4d0 0x7f79b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.696+0000 7f79affff640 1 -- 172.21.15.27:0/1689654736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f79b80ad160 con 0x7f79c80b95a0 2024-03-06T07:30:01.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.696+0000 7f79c6ffd640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f79b406b4d0 0x7f79b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.697+0000 7f79c6ffd640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f79b406b4d0 0x7f79b406d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f79c80c73d0 tx=0x7f79bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.701+0000 7f79affff640 1 -- 172.21.15.27:0/1689654736 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79c80c7f10 con 0x7f79c80b95a0 2024-03-06T07:30:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.900+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79c81f3ff0 con 0x7f79c80b95a0 2024-03-06T07:30:01.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.901+0000 7f79affff640 1 -- 172.21.15.27:0/1689654736 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16967 (secure 0 0 0) 0x7f79b807c080 con 0x7f79c80b95a0 2024-03-06T07:30:01.904 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:01.904 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":120,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:29:29.730083+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.260334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.929245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6832/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6834/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6835/1655684069":"2024-03-07T07:28:59.663475+0000","172.21.15.27:6833/1655684069":"2024-03-07T07:28:59.663475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f79b406b4d0 msgr2=0x7f79b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f79b406b4d0 0x7f79b406d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f79c80c73d0 tx=0x7f79bc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c80b95a0 msgr2=0x7f79c815aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c80b95a0 0x7f79c815aed0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f79b8037040 tx=0x7f79b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 shutdown_connections 2024-03-06T07:30:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f79b406b4d0 0x7f79b406d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79c81f3ba0 0x7f79c80ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c815df00 0x7f79c81f3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 --2- 172.21.15.27:0/1689654736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79c80b95a0 0x7f79c815aed0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.903+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 >> 172.21.15.27:0/1689654736 conn(0x7f79c8071960 msgr2=0x7f79c815bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.904+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 shutdown_connections 2024-03-06T07:30:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:01.904+0000 7f79ce088640 1 -- 172.21.15.27:0/1689654736 wait complete. 2024-03-06T07:30:01.918 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:30:01.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:02.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.022+0000 7f860948f640 1 Processor -- start 2024-03-06T07:30:02.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.022+0000 7f860948f640 1 -- start start 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.023+0000 7f860948f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8604158520 0x7f860406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.023+0000 7f860948f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86040aee10 0x7f86040b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.023+0000 7f860948f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8604071240 con 0x7f860406dd60 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.023+0000 7f860948f640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860406e1c0 con 0x7f86040aee10 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.023+0000 7f860948f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8604071b70 con 0x7f8604158520 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.023+0000 7f86037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86040aee10 0x7f86040b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.023+0000 7f86037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86040aee10 0x7f86040b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40580/0 (socket says 172.21.15.27:40580) 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.023+0000 7f86037fe640 1 -- 172.21.15.27:0/227911649 learned_addr learned my addr 172.21.15.27:0/227911649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.024+0000 7f8602ffd640 1 --2- 172.21.15.27:0/227911649 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8604158520 0x7f860406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.024+0000 7f86037fe640 1 -- 172.21.15.27:0/227911649 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8604158520 msgr2=0x7f860406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.024+0000 7f86037fe640 1 --2- 172.21.15.27:0/227911649 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8604158520 0x7f860406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.024+0000 7f86037fe640 1 -- 172.21.15.27:0/227911649 >> v1:172.21.15.69:6789/0 conn(0x7f860406dd60 legacy=0x7f86040ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.024+0000 7f86037fe640 1 -- 172.21.15.27:0/227911649 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8604071520 con 0x7f86040aee10 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f86037fe640 1 --2- 172.21.15.27:0/227911649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040aee10 0x7f86040b3640 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00d150 tx=0x7f85fc02e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16430470bad0cac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f8601ffb640 1 -- 172.21.15.27:0/227911649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00b0d0 con 0x7f86040aee10 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f8601ffb640 1 -- 172.21.15.27:0/227911649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc00b270 con 0x7f86040aee10 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f8601ffb640 1 -- 172.21.15.27:0/227911649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc009600 con 0x7f86040aee10 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f860948f640 1 -- 172.21.15.27:0/227911649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040aee10 msgr2=0x7f86040b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f860948f640 1 --2- 172.21.15.27:0/227911649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040aee10 0x7f86040b3640 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00d150 tx=0x7f85fc02e160 comp rx=0 tx=0).stop 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f860948f640 1 -- 172.21.15.27:0/227911649 shutdown_connections 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f860948f640 1 --2- 172.21.15.27:0/227911649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040aee10 0x7f86040b3640 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f860948f640 1 --2- 172.21.15.27:0/227911649 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8604158520 0x7f860406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f860948f640 1 -- 172.21.15.27:0/227911649 >> 172.21.15.27:0/227911649 conn(0x7f8604066d00 msgr2=0x7f86040670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f860948f640 1 -- 172.21.15.27:0/227911649 shutdown_connections 2024-03-06T07:30:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.025+0000 7f860948f640 1 -- 172.21.15.27:0/227911649 wait complete. 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.027+0000 7f860948f640 1 Processor -- start 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.027+0000 7f860948f640 1 -- start start 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.027+0000 7f860948f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860406dd60 0x7f86041f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.027+0000 7f860948f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86040aee10 0x7f86040cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.027+0000 7f860948f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8604158520 0x7f86040cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.027+0000 7f860948f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8604158900 con 0x7f860406dd60 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.027+0000 7f860948f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8604072000 con 0x7f8604158520 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.027+0000 7f860948f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8604158a80 con 0x7f86040aee10 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.027+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860406dd60 0x7f86041f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860406dd60 0x7f86041f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40592/0 (socket says 172.21.15.27:40592) 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f8602ffd640 1 -- 172.21.15.27:0/2569721278 learned_addr learned my addr 172.21.15.27:0/2569721278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f86027fc640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86040aee10 0x7f86040cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f86037fe640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8604158520 0x7f86040cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f8602ffd640 1 -- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86040aee10 msgr2=0x7f86040cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f8602ffd640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86040aee10 0x7f86040cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f8602ffd640 1 -- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8604158520 msgr2=0x7f86040cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f8602ffd640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8604158520 0x7f86040cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f8602ffd640 1 -- 172.21.15.27:0/2569721278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040cdaa0 con 0x7f860406dd60 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f8602ffd640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860406dd60 0x7f86041f6560 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f85f800e870 tx=0x7f85f800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f85eb7fe640 1 -- 172.21.15.27:0/2569721278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8017070 con 0x7f860406dd60 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f85eb7fe640 1 -- 172.21.15.27:0/2569721278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f800c940 con 0x7f860406dd60 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86041e3e50 con 0x7f860406dd60 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f85eb7fe640 1 -- 172.21.15.27:0/2569721278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f800cd30 con 0x7f860406dd60 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.028+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86041e4410 con 0x7f860406dd60 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.029+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8604067ff0 con 0x7f860406dd60 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.030+0000 7f85eb7fe640 1 -- 172.21.15.27:0/2569721278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85f801a070 con 0x7f860406dd60 2024-03-06T07:30:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.030+0000 7f85eb7fe640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.030+0000 7f85eb7fe640 1 -- 172.21.15.27:0/2569721278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f85f808a530 con 0x7f860406dd60 2024-03-06T07:30:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.031+0000 7f86027fc640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.031+0000 7f86027fc640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8604067300 tx=0x7f85f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.040+0000 7f85eb7fe640 1 -- 172.21.15.27:0/2569721278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85f80592d0 con 0x7f860406dd60 2024-03-06T07:30:02.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.257+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86040b31f0 con 0x7f860406dd60 2024-03-06T07:30:02.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.258+0000 7f85eb7fe640 1 -- 172.21.15.27:0/2569721278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f85f8058bd0 con 0x7f860406dd60 2024-03-06T07:30:02.261 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:02.261 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:30:02.261 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:30:02.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.259+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 msgr2=0x7f85f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.259+0000 7f860948f640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8604067300 tx=0x7f85f4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860406dd60 msgr2=0x7f86041f6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860406dd60 0x7f86041f6560 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f85f800e870 tx=0x7f85f800ed30 comp rx=0 tx=0).stop 2024-03-06T07:30:02.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 shutdown_connections 2024-03-06T07:30:02.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8604158520 0x7f86040cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86040aee10 0x7f86040cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 --2- 172.21.15.27:0/2569721278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f860406dd60 0x7f86041f6560 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 >> 172.21.15.27:0/2569721278 conn(0x7f8604066d00 msgr2=0x7f86040b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 shutdown_connections 2024-03-06T07:30:02.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.260+0000 7f860948f640 1 -- 172.21.15.27:0/2569721278 wait complete. 2024-03-06T07:30:02.273 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:02.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.377+0000 7fe4cca88640 1 Processor -- start 2024-03-06T07:30:02.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.377+0000 7fe4cca88640 1 -- start start 2024-03-06T07:30:02.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.378+0000 7fe4cca88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 0x7fe4c80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.378+0000 7fe4cca88640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe4c80b9a30 0x7fe4c80d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.378+0000 7fe4cca88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c80cea30 con 0x7fe4c80b95a0 2024-03-06T07:30:02.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.378+0000 7fe4cca88640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c80d2af0 con 0x7fe4c80b9a30 2024-03-06T07:30:02.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.378+0000 7fe4cca88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80cf3c0 con 0x7fe4c815df00 2024-03-06T07:30:02.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.378+0000 7fe4c6d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe4c80b9a30 0x7fe4c80d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.378+0000 7fe4c6d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe4c80b9a30 0x7fe4c80d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40602/0 (socket says 172.21.15.27:40602) 2024-03-06T07:30:02.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.378+0000 7fe4c6d9d640 1 -- 172.21.15.27:0/3891441978 learned_addr learned my addr 172.21.15.27:0/3891441978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:02.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.379+0000 7fe4c659c640 1 --2- 172.21.15.27:0/3891441978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 0x7fe4c80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.379+0000 7fe4c6d9d640 1 -- 172.21.15.27:0/3891441978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 msgr2=0x7fe4c80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.379+0000 7fe4c6d9d640 1 --2- 172.21.15.27:0/3891441978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 0x7fe4c80d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.379+0000 7fe4c6d9d640 1 -- 172.21.15.27:0/3891441978 >> v1:172.21.15.69:6789/0 conn(0x7fe4c80b95a0 legacy=0x7fe4c80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.379+0000 7fe4c6d9d640 1 -- 172.21.15.27:0/3891441978 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c80cf0e0 con 0x7fe4c80b9a30 2024-03-06T07:30:02.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4c6d9d640 1 --2- 172.21.15.27:0/3891441978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b9a30 0x7fe4c80d3220 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009e50 tx=0x7fe4c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be05c689e2638aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4c559a640 1 -- 172.21.15.27:0/3891441978 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c002ca70 con 0x7fe4c80b9a30 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4c559a640 1 -- 172.21.15.27:0/3891441978 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c002cc10 con 0x7fe4c80b9a30 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4c559a640 1 -- 172.21.15.27:0/3891441978 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c00343f0 con 0x7fe4c80b9a30 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4cca88640 1 -- 172.21.15.27:0/3891441978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b9a30 msgr2=0x7fe4c80d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4cca88640 1 --2- 172.21.15.27:0/3891441978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b9a30 0x7fe4c80d3220 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009e50 tx=0x7fe4c002bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4cca88640 1 -- 172.21.15.27:0/3891441978 shutdown_connections 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4cca88640 1 --2- 172.21.15.27:0/3891441978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b9a30 0x7fe4c80d3220 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4cca88640 1 --2- 172.21.15.27:0/3891441978 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 0x7fe4c80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4cca88640 1 -- 172.21.15.27:0/3891441978 >> 172.21.15.27:0/3891441978 conn(0x7fe4c8071960 msgr2=0x7fe4c8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4cca88640 1 -- 172.21.15.27:0/3891441978 shutdown_connections 2024-03-06T07:30:02.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.380+0000 7fe4cca88640 1 -- 172.21.15.27:0/3891441978 wait complete. 2024-03-06T07:30:02.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.382+0000 7fe4cca88640 1 Processor -- start 2024-03-06T07:30:02.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.382+0000 7fe4cca88640 1 -- start start 2024-03-06T07:30:02.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4cca88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b95a0 0x7fe4c81f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4cca88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 0x7fe4c815af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4cca88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c815b4b0 0x7fe4c80ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4cca88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80cec70 con 0x7fe4c80b95a0 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4cca88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80ce750 con 0x7fe4c815b4b0 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4cca88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80cedf0 con 0x7fe4c815df00 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b95a0 0x7fe4c81f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b95a0 0x7fe4c81f64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40618/0 (socket says 172.21.15.27:40618) 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c659c640 1 -- 172.21.15.27:0/645813080 learned_addr learned my addr 172.21.15.27:0/645813080 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c5d9b640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 0x7fe4c815af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c6d9d640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c815b4b0 0x7fe4c80ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c659c640 1 -- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 msgr2=0x7fe4c815af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c659c640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 0x7fe4c815af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c659c640 1 -- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c815b4b0 msgr2=0x7fe4c80ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c659c640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c815b4b0 0x7fe4c80ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c659c640 1 -- 172.21.15.27:0/645813080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c81f3a20 con 0x7fe4c80b95a0 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c5d9b640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 0x7fe4c815af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4c659c640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b95a0 0x7fe4c81f64e0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00d800 tx=0x7fe4bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4b6ffd640 1 -- 172.21.15.27:0/645813080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc00b6c0 con 0x7fe4c80b95a0 2024-03-06T07:30:02.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4b6ffd640 1 -- 172.21.15.27:0/645813080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc019070 con 0x7fe4c80b95a0 2024-03-06T07:30:02.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c81f3d00 con 0x7fe4c80b95a0 2024-03-06T07:30:02.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4b6ffd640 1 -- 172.21.15.27:0/645813080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc00ba80 con 0x7fe4c80b95a0 2024-03-06T07:30:02.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.383+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c81f42c0 con 0x7fe4c80b95a0 2024-03-06T07:30:02.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.384+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c8067ff0 con 0x7fe4c80b95a0 2024-03-06T07:30:02.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.385+0000 7fe4b6ffd640 1 -- 172.21.15.27:0/645813080 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4bc00bc20 con 0x7fe4c80b95a0 2024-03-06T07:30:02.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.386+0000 7fe4b6ffd640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4a806b4d0 0x7fe4a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.386+0000 7fe4b6ffd640 1 -- 172.21.15.27:0/645813080 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fe4bc08a7e0 con 0x7fe4c80b95a0 2024-03-06T07:30:02.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.386+0000 7fe4c5d9b640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4a806b4d0 0x7fe4a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.387+0000 7fe4c5d9b640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4a806b4d0 0x7fe4a806d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8160e90 tx=0x7fe4b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.391+0000 7fe4b6ffd640 1 -- 172.21.15.27:0/645813080 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4bc0594f0 con 0x7fe4c80b95a0 2024-03-06T07:30:02.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.609+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4c806b780 con 0x7fe4c80b95a0 2024-03-06T07:30:02.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.609+0000 7fe4b6ffd640 1 -- 172.21.15.27:0/645813080 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fe4bc058df0 con 0x7fe4c80b95a0 2024-03-06T07:30:02.612 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:02.612 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.951614+0000","modified":"2024-03-06T07:29:03.684342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8256":{"gid":8256,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6833/2669617374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2669617374},{"type":"v1","addr":"172.21.15.27:6833","nonce":2669617374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8256,"qdb_cluster":[8256]},"id":8}]} 2024-03-06T07:30:02.612 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 83 2024-03-06T07:30:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4a806b4d0 msgr2=0x7fe4a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4a806b4d0 0x7fe4a806d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8160e90 tx=0x7fe4b8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b95a0 msgr2=0x7fe4c81f64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b95a0 0x7fe4c81f64e0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00d800 tx=0x7fe4bc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 shutdown_connections 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe4a806b4d0 0x7fe4a806d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c815b4b0 0x7fe4c80ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c815df00 0x7fe4c815af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 --2- 172.21.15.27:0/645813080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4c80b95a0 0x7fe4c81f64e0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 >> 172.21.15.27:0/645813080 conn(0x7fe4c8071960 msgr2=0x7fe4c80d35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 shutdown_connections 2024-03-06T07:30:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.612+0000 7fe4cca88640 1 -- 172.21.15.27:0/645813080 wait complete. 2024-03-06T07:30:02.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:02.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.729+0000 7f17afaed640 1 Processor -- start 2024-03-06T07:30:02.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.730+0000 7f17afaed640 1 -- start start 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.730+0000 7f17afaed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a8071540 0x7f17a80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.730+0000 7f17afaed640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f17a80d3400 0x7f17a8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.730+0000 7f17afaed640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a8071e60 con 0x7f17a80b2b50 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.730+0000 7f17afaed640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a80d2f70 con 0x7f17a80d3400 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.730+0000 7f17afaed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80d4be0 con 0x7f17a8071540 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ad889640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a8071540 0x7f17a80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ad088640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f17a80d3400 0x7f17a8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ae08a640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f17a80b2b50 0x7f17a80d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:39544/0 (socket says 172.21.15.27:39544) 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ae08a640 1 -- 172.21.15.27:0/2708276401 learned_addr learned my addr 172.21.15.27:0/2708276401 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ac887640 1 -- 172.21.15.27:0/2708276401 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269751170 0 0) 0x7f17a8071e60 con 0x7f17a80b2b50 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ac887640 1 -- 172.21.15.27:0/2708276401 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1794003520 con 0x7f17a80b2b50 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ad889640 1 -- 172.21.15.27:0/2708276401 >> v2:172.21.15.69:3300/0 conn(0x7f17a80d3400 msgr2=0x7f17a8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ad889640 1 --2- 172.21.15.27:0/2708276401 >> v2:172.21.15.69:3300/0 conn(0x7f17a80d3400 0x7f17a8071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ad889640 1 -- 172.21.15.27:0/2708276401 >> v1:172.21.15.69:6789/0 conn(0x7f17a80b2b50 legacy=0x7f17a80d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ad889640 1 -- 172.21.15.27:0/2708276401 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a80ce610 con 0x7f17a8071540 2024-03-06T07:30:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.731+0000 7f17ad088640 1 --2- 172.21.15.27:0/2708276401 >> v2:172.21.15.69:3300/0 conn(0x7f17a80d3400 0x7f17a8071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.732+0000 7f17ad889640 1 --2- 172.21.15.27:0/2708276401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a8071540 0x7f17a80b30a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f17a40097a0 tx=0x7f17a402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7461ede0c10aa5aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17ac887640 1 -- 172.21.15.27:0/2708276401 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a4035020 con 0x7f17a8071540 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17ac887640 1 -- 172.21.15.27:0/2708276401 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a4031830 con 0x7f17a8071540 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17ac887640 1 -- 172.21.15.27:0/2708276401 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a4031c00 con 0x7f17a8071540 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17afaed640 1 -- 172.21.15.27:0/2708276401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a8071540 msgr2=0x7f17a80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17afaed640 1 --2- 172.21.15.27:0/2708276401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a8071540 0x7f17a80b30a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f17a40097a0 tx=0x7f17a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17afaed640 1 -- 172.21.15.27:0/2708276401 shutdown_connections 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17afaed640 1 --2- 172.21.15.27:0/2708276401 >> v2:172.21.15.69:3300/0 conn(0x7f17a80d3400 0x7f17a8071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17afaed640 1 --2- 172.21.15.27:0/2708276401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a8071540 0x7f17a80b30a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17afaed640 1 -- 172.21.15.27:0/2708276401 >> 172.21.15.27:0/2708276401 conn(0x7f17a80d3ca0 msgr2=0x7f17a80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17afaed640 1 -- 172.21.15.27:0/2708276401 shutdown_connections 2024-03-06T07:30:02.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.733+0000 7f17afaed640 1 -- 172.21.15.27:0/2708276401 wait complete. 2024-03-06T07:30:02.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17afaed640 1 Processor -- start 2024-03-06T07:30:02.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17afaed640 1 -- start start 2024-03-06T07:30:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17afaed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a8071540 0x7f17a80abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17afaed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a80b2b50 0x7f17a81f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17afaed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a80d3400 0x7f17a80b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17afaed640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80ce8f0 con 0x7f17a80d3400 2024-03-06T07:30:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17afaed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80d2c00 con 0x7f17a8071540 2024-03-06T07:30:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17afaed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a815e370 con 0x7f17a80b2b50 2024-03-06T07:30:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17ae08a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a80d3400 0x7f17a80b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17ae08a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a80d3400 0x7f17a80b1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40632/0 (socket says 172.21.15.27:40632) 2024-03-06T07:30:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.735+0000 7f17ae08a640 1 -- 172.21.15.27:0/3392963066 learned_addr learned my addr 172.21.15.27:0/3392963066 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17ad088640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a80b2b50 0x7f17a81f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17ad889640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a8071540 0x7f17a80abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17ae08a640 1 -- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a80b2b50 msgr2=0x7f17a81f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17ae08a640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a80b2b50 0x7f17a81f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17ae08a640 1 -- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a8071540 msgr2=0x7f17a80abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17ae08a640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a8071540 0x7f17a80abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17ae08a640 1 -- 172.21.15.27:0/3392963066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a80ac520 con 0x7f17a80d3400 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17ae08a640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a80d3400 0x7f17a80b1a60 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f17a000cb70 tx=0x7f17a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f179e7fc640 1 -- 172.21.15.27:0/3392963066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0017070 con 0x7f17a80d3400 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f179e7fc640 1 -- 172.21.15.27:0/3392963066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a000ad40 con 0x7f17a80d3400 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a80ac7a0 con 0x7f17a80d3400 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f179e7fc640 1 -- 172.21.15.27:0/3392963066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0012620 con 0x7f17a80d3400 2024-03-06T07:30:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.736+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a81f6f60 con 0x7f17a80d3400 2024-03-06T07:30:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.737+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a8068700 con 0x7f17a80d3400 2024-03-06T07:30:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.738+0000 7f179e7fc640 1 -- 172.21.15.27:0/3392963066 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17a001a070 con 0x7f17a80d3400 2024-03-06T07:30:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.738+0000 7f179e7fc640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178c06b4d0 0x7f178c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.738+0000 7f179e7fc640 1 -- 172.21.15.27:0/3392963066 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f17a008a030 con 0x7f17a80d3400 2024-03-06T07:30:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.738+0000 7f17ad889640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178c06b4d0 0x7f178c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.739+0000 7f17ad889640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178c06b4d0 0x7f178c06d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f17a4036000 tx=0x7f17a4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.743+0000 7f179e7fc640 1 -- 172.21.15.27:0/3392963066 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17a80ac520 con 0x7f17a80d3400 2024-03-06T07:30:02.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.926+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a80d18e0 con 0x7f17a80d3400 2024-03-06T07:30:02.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.927+0000 7f179e7fc640 1 -- 172.21.15.27:0/3392963066 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16967 (secure 0 0 0) 0x7f17a0058800 con 0x7f17a80d3400 2024-03-06T07:30:02.930 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:02.930 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":120,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:29:29.730083+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:39.260334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.929245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6832/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6834/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6835/1655684069":"2024-03-07T07:28:59.663475+0000","172.21.15.27:6833/1655684069":"2024-03-07T07:28:59.663475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:02.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178c06b4d0 msgr2=0x7f178c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178c06b4d0 0x7f178c06d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f17a4036000 tx=0x7f17a4008030 comp rx=0 tx=0).stop 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a80d3400 msgr2=0x7f17a80b1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a80d3400 0x7f17a80b1a60 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f17a000cb70 tx=0x7f17a000a430 comp rx=0 tx=0).stop 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 shutdown_connections 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178c06b4d0 0x7f178c06d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a80d3400 0x7f17a80b1a60 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a80b2b50 0x7f17a81f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 --2- 172.21.15.27:0/3392963066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a8071540 0x7f17a80abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 >> 172.21.15.27:0/3392963066 conn(0x7f17a80d3ca0 msgr2=0x7f17a80b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 shutdown_connections 2024-03-06T07:30:02.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:02.929+0000 7f17afaed640 1 -- 172.21.15.27:0/3392963066 wait complete. 2024-03-06T07:30:02.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:30:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.050+0000 7f7b25ac9640 1 Processor -- start 2024-03-06T07:30:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.050+0000 7f7b25ac9640 1 -- start start 2024-03-06T07:30:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.051+0000 7f7b25ac9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b20071540 0x7f7b200b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.051+0000 7f7b25ac9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b200d3400 0x7f7b20071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.051+0000 7f7b25ac9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b20071e60 con 0x7f7b200b2b50 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.051+0000 7f7b25ac9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b200d2f70 con 0x7f7b200d3400 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.051+0000 7f7b25ac9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200d4be0 con 0x7f7b20071540 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b20071540 0x7f7b200b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1effd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b200d3400 0x7f7b20071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b20071540 0x7f7b200b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:55024/0 (socket says 172.21.15.27:55024) 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1ffff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7b200b2b50 0x7f7b200d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:39546/0 (socket says 172.21.15.27:39546) 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1f7fe640 1 -- 172.21.15.27:0/3087421362 learned_addr learned my addr 172.21.15.27:0/3087421362 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1effd640 1 -- 172.21.15.27:0/3087421362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b20071540 msgr2=0x7f7b200b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1effd640 1 --2- 172.21.15.27:0/3087421362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b20071540 0x7f7b200b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1effd640 1 -- 172.21.15.27:0/3087421362 >> v1:172.21.15.69:6789/0 conn(0x7f7b200b2b50 legacy=0x7f7b200d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1effd640 1 -- 172.21.15.27:0/3087421362 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b200ce610 con 0x7f7b200d3400 2024-03-06T07:30:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.052+0000 7f7b1f7fe640 1 --2- 172.21.15.27:0/3087421362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b20071540 0x7f7b200b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.053+0000 7f7b1effd640 1 --2- 172.21.15.27:0/3087421362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b200d3400 0x7f7b20071920 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7b100097a0 tx=0x7f7b1002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a3a0ce669625cbe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.053+0000 7f7b1e7fc640 1 -- 172.21.15.27:0/3087421362 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b10035020 con 0x7f7b200d3400 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.053+0000 7f7b1e7fc640 1 -- 172.21.15.27:0/3087421362 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b10031830 con 0x7f7b200d3400 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.053+0000 7f7b1e7fc640 1 -- 172.21.15.27:0/3087421362 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b10031c00 con 0x7f7b200d3400 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.054+0000 7f7b25ac9640 1 -- 172.21.15.27:0/3087421362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b200d3400 msgr2=0x7f7b20071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.054+0000 7f7b25ac9640 1 --2- 172.21.15.27:0/3087421362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b200d3400 0x7f7b20071920 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7b100097a0 tx=0x7f7b1002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.054+0000 7f7b25ac9640 1 -- 172.21.15.27:0/3087421362 shutdown_connections 2024-03-06T07:30:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.054+0000 7f7b25ac9640 1 --2- 172.21.15.27:0/3087421362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b200d3400 0x7f7b20071920 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.054+0000 7f7b25ac9640 1 --2- 172.21.15.27:0/3087421362 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b20071540 0x7f7b200b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.054+0000 7f7b25ac9640 1 -- 172.21.15.27:0/3087421362 >> 172.21.15.27:0/3087421362 conn(0x7f7b200d3ca0 msgr2=0x7f7b200d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.054+0000 7f7b25ac9640 1 -- 172.21.15.27:0/3087421362 shutdown_connections 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.054+0000 7f7b25ac9640 1 -- 172.21.15.27:0/3087421362 wait complete. 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.055+0000 7f7b25ac9640 1 Processor -- start 2024-03-06T07:30:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.055+0000 7f7b25ac9640 1 -- start start 2024-03-06T07:30:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b25ac9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b20071540 0x7f7b201540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b25ac9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b200b2b50 0x7f7b200ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b25ac9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b200d3400 0x7f7b200cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b25ac9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200ce8f0 con 0x7f7b20071540 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b25ac9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200d2c00 con 0x7f7b200d3400 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b25ac9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2015e370 con 0x7f7b200b2b50 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b20071540 0x7f7b201540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b20071540 0x7f7b201540b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40644/0 (socket says 172.21.15.27:40644) 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1f7fe640 1 -- 172.21.15.27:0/1544088743 learned_addr learned my addr 172.21.15.27:0/1544088743 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1effd640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b200b2b50 0x7f7b200ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1ffff640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b200d3400 0x7f7b200cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1f7fe640 1 -- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b200b2b50 msgr2=0x7f7b200ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1f7fe640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b200b2b50 0x7f7b200ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1f7fe640 1 -- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b200d3400 msgr2=0x7f7b200cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1f7fe640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b200d3400 0x7f7b200cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1f7fe640 1 -- 172.21.15.27:0/1544088743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b200c9a00 con 0x7f7b20071540 2024-03-06T07:30:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1effd640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b200b2b50 0x7f7b200ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.056+0000 7f7b1f7fe640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b20071540 0x7f7b201540b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800e9d0 tx=0x7f7b0800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.057+0000 7f7b17fff640 1 -- 172.21.15.27:0/1544088743 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08017070 con 0x7f7b20071540 2024-03-06T07:30:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.057+0000 7f7b17fff640 1 -- 172.21.15.27:0/1544088743 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0800ca60 con 0x7f7b20071540 2024-03-06T07:30:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.057+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b200c9ce0 con 0x7f7b20071540 2024-03-06T07:30:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.057+0000 7f7b17fff640 1 -- 172.21.15.27:0/1544088743 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0800ce10 con 0x7f7b20071540 2024-03-06T07:30:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.057+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b200ca2a0 con 0x7f7b20071540 2024-03-06T07:30:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.058+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b200ca4e0 con 0x7f7b20071540 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.059+0000 7f7b17fff640 1 -- 172.21.15.27:0/1544088743 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b0801a070 con 0x7f7b20071540 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.059+0000 7f7b17fff640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b0006b4d0 0x7f7b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.059+0000 7f7b17fff640 1 -- 172.21.15.27:0/1544088743 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7b0808a6a0 con 0x7f7b20071540 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.059+0000 7f7b1effd640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b0006b4d0 0x7f7b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.060+0000 7f7b1effd640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b0006b4d0 0x7f7b0006d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7b10036000 tx=0x7f7b1003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.064+0000 7f7b17fff640 1 -- 172.21.15.27:0/1544088743 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b200ca4e0 con 0x7f7b20071540 2024-03-06T07:30:03.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.282+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7b2015f0c0 con 0x7f7b20071540 2024-03-06T07:30:03.285 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:30:03.283+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:30:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.285+0000 7f7b17fff640 1 -- 172.21.15.27:0/1544088743 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7b080593b0 con 0x7f7b20071540 2024-03-06T07:30:03.287 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:30:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b0006b4d0 msgr2=0x7f7b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b0006b4d0 0x7f7b0006d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7b10036000 tx=0x7f7b1003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b20071540 msgr2=0x7f7b201540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b20071540 0x7f7b201540b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800e9d0 tx=0x7f7b0800ee90 comp rx=0 tx=0).stop 2024-03-06T07:30:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 shutdown_connections 2024-03-06T07:30:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b0006b4d0 0x7f7b0006d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b200d3400 0x7f7b200cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b200b2b50 0x7f7b200ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 --2- 172.21.15.27:0/1544088743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b20071540 0x7f7b201540b0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 >> 172.21.15.27:0/1544088743 conn(0x7f7b200d3ca0 msgr2=0x7f7b2015e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 shutdown_connections 2024-03-06T07:30:03.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.288+0000 7f7b25ac9640 1 -- 172.21.15.27:0/1544088743 wait complete. 2024-03-06T07:30:03.294 INFO:tasks.ceph.mds.a.smithi027.stderr: -8792> 2024-03-06T07:29:05.674+0000 7f61c459e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:03.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:30:03.328 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:30:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.410+0000 7f3cbf193640 1 Processor -- start 2024-03-06T07:30:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.410+0000 7f3cbf193640 1 -- start start 2024-03-06T07:30:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.410+0000 7f3cbf193640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cb8142170 0x7f3cb8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.410+0000 7f3cbf193640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3cb80b4830 0x7f3cb8160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbf193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b29a0 0x7f3cb80b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbf193640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb81458e0 con 0x7f3cb8142170 2024-03-06T07:30:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbf193640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb8070980 con 0x7f3cb80b4830 2024-03-06T07:30:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbf193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8142f60 con 0x7f3cb80b29a0 2024-03-06T07:30:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbcf2f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cb8142170 0x7f3cb8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbcf2f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cb8142170 0x7f3cb8142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40666/0 (socket says 172.21.15.27:40666) 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbcf2f640 1 -- 172.21.15.27:0/3435494366 learned_addr learned my addr 172.21.15.27:0/3435494366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbd730640 1 --2- 172.21.15.27:0/3435494366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b29a0 0x7f3cb80b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cb7fff640 1 --2- 172.21.15.27:0/3435494366 >> v2:172.21.15.69:3300/0 conn(0x7f3cb80b4830 0x7f3cb8160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbcf2f640 1 -- 172.21.15.27:0/3435494366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b29a0 msgr2=0x7f3cb80b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbcf2f640 1 --2- 172.21.15.27:0/3435494366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b29a0 0x7f3cb80b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbcf2f640 1 -- 172.21.15.27:0/3435494366 >> v2:172.21.15.69:3300/0 conn(0x7f3cb80b4830 msgr2=0x7f3cb8160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbcf2f640 1 --2- 172.21.15.27:0/3435494366 >> v2:172.21.15.69:3300/0 conn(0x7f3cb80b4830 0x7f3cb8160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbcf2f640 1 -- 172.21.15.27:0/3435494366 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb81607c0 con 0x7f3cb8142170 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.411+0000 7f3cbd730640 1 --2- 172.21.15.27:0/3435494366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b29a0 0x7f3cb80b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbcf2f640 1 --2- 172.21.15.27:0/3435494366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb8142170 0x7f3cb8142a20 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80af240 tx=0x7f3cb002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c423c4910006efb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cb77fe640 1 -- 172.21.15.27:0/3435494366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb002c9e0 con 0x7f3cb8142170 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cb77fe640 1 -- 172.21.15.27:0/3435494366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb002cb80 con 0x7f3cb8142170 2024-03-06T07:30:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cb77fe640 1 -- 172.21.15.27:0/3435494366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb00343f0 con 0x7f3cb8142170 2024-03-06T07:30:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbf193640 1 -- 172.21.15.27:0/3435494366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb8142170 msgr2=0x7f3cb8142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbf193640 1 --2- 172.21.15.27:0/3435494366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb8142170 0x7f3cb8142a20 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80af240 tx=0x7f3cb002be40 comp rx=0 tx=0).stop 2024-03-06T07:30:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbf193640 1 -- 172.21.15.27:0/3435494366 shutdown_connections 2024-03-06T07:30:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbf193640 1 --2- 172.21.15.27:0/3435494366 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb80b29a0 0x7f3cb80b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbf193640 1 --2- 172.21.15.27:0/3435494366 >> v2:172.21.15.69:3300/0 conn(0x7f3cb80b4830 0x7f3cb8160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbf193640 1 --2- 172.21.15.27:0/3435494366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb8142170 0x7f3cb8142a20 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbf193640 1 -- 172.21.15.27:0/3435494366 >> 172.21.15.27:0/3435494366 conn(0x7f3cb8066d00 msgr2=0x7f3cb8067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbf193640 1 -- 172.21.15.27:0/3435494366 shutdown_connections 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.412+0000 7f3cbf193640 1 -- 172.21.15.27:0/3435494366 wait complete. 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.414+0000 7f3cbf193640 1 Processor -- start 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.414+0000 7f3cbf193640 1 -- start start 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.414+0000 7f3cbf193640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b29a0 0x7f3cb80b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.414+0000 7f3cbf193640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b4830 0x7f3cb81402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.414+0000 7f3cbf193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb8142170 0x7f3cb80b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.414+0000 7f3cbf193640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8070ed0 con 0x7f3cb80b4830 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.414+0000 7f3cbf193640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80b32f0 con 0x7f3cb80b29a0 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.414+0000 7f3cbf193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80711b0 con 0x7f3cb8142170 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b4830 0x7f3cb81402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b4830 0x7f3cb81402b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40676/0 (socket says 172.21.15.27:40676) 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb7fff640 1 -- 172.21.15.27:0/2214945035 learned_addr learned my addr 172.21.15.27:0/2214945035 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cbd730640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb8142170 0x7f3cb80b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cbcf2f640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b29a0 0x7f3cb80b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb7fff640 1 -- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb8142170 msgr2=0x7f3cb80b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb7fff640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb8142170 0x7f3cb80b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb7fff640 1 -- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b29a0 msgr2=0x7f3cb80b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb7fff640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b29a0 0x7f3cb80b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb7fff640 1 -- 172.21.15.27:0/2214945035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8140a00 con 0x7f3cb80b4830 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb7fff640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b4830 0x7f3cb81402b0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800d800 tx=0x7f3ca800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb57fa640 1 -- 172.21.15.27:0/2214945035 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca800b6c0 con 0x7f3cb80b4830 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb57fa640 1 -- 172.21.15.27:0/2214945035 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca8019070 con 0x7f3cb80b4830 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb813e6a0 con 0x7f3cb80b4830 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cb57fa640 1 -- 172.21.15.27:0/2214945035 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca800ba80 con 0x7f3cb80b4830 2024-03-06T07:30:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.415+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb813ec60 con 0x7f3cb80b4830 2024-03-06T07:30:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.416+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb80733c0 con 0x7f3cb80b4830 2024-03-06T07:30:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.417+0000 7f3cb57fa640 1 -- 172.21.15.27:0/2214945035 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ca800bc20 con 0x7f3cb80b4830 2024-03-06T07:30:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.417+0000 7f3cb57fa640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c9806b4d0 0x7f3c9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.417+0000 7f3cb57fa640 1 -- 172.21.15.27:0/2214945035 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f3ca808a780 con 0x7f3cb80b4830 2024-03-06T07:30:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.417+0000 7f3cbcf2f640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c9806b4d0 0x7f3c9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.418+0000 7f3cbcf2f640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c9806b4d0 0x7f3c9806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0035000 tx=0x7f3cb0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.421+0000 7f3cb57fa640 1 -- 172.21.15.27:0/2214945035 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ca80594c0 con 0x7f3cb80b4830 2024-03-06T07:30:03.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:03.646+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3cb804f410 con 0x7f3cb80b4830 2024-03-06T07:30:04.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.286+0000 7f3cb57fa640 1 -- 172.21.15.27:0/2214945035 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3ca8058dc0 con 0x7f3cb80b4830 2024-03-06T07:30:04.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.288+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c9806b4d0 msgr2=0x7f3c9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.288+0000 7f3cbf193640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c9806b4d0 0x7f3c9806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0035000 tx=0x7f3cb0030000 comp rx=0 tx=0).stop 2024-03-06T07:30:04.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.288+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b4830 msgr2=0x7f3cb81402b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.288+0000 7f3cbf193640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b4830 0x7f3cb81402b0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800d800 tx=0x7f3ca800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.289+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 shutdown_connections 2024-03-06T07:30:04.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.289+0000 7f3cbf193640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c9806b4d0 0x7f3c9806d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.289+0000 7f3cbf193640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3cb8142170 0x7f3cb80b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.289+0000 7f3cbf193640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cb80b4830 0x7f3cb81402b0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.289+0000 7f3cbf193640 1 --2- 172.21.15.27:0/2214945035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb80b29a0 0x7f3cb80b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.289+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 >> 172.21.15.27:0/2214945035 conn(0x7f3cb8066d00 msgr2=0x7f3cb81438c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.289+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 shutdown_connections 2024-03-06T07:30:04.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.289+0000 7f3cbf193640 1 -- 172.21.15.27:0/2214945035 wait complete. 2024-03-06T07:30:04.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:30:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.456+0000 7f79411d0640 1 Processor -- start 2024-03-06T07:30:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.456+0000 7f79411d0640 1 -- start start 2024-03-06T07:30:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.456+0000 7f79411d0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f793c15dd90 0x7f793c0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.456+0000 7f79411d0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f793c0b6d10 0x7f793c0b7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.456+0000 7f79411d0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793c0d3740 con 0x7f793c0d29d0 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.456+0000 7f79411d0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f793c0d3400 con 0x7f793c0b6d10 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.456+0000 7f79411d0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f793c0b7690 con 0x7f793c15dd90 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.457+0000 7f793b59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f793c0b6d10 0x7f793c0b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.457+0000 7f7933fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f793c0d29d0 0x7f793c0d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43216/0 (socket says 172.21.15.27:43216) 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.457+0000 7f7933fff640 1 -- 172.21.15.27:0/110045894 learned_addr learned my addr 172.21.15.27:0/110045894 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.457+0000 7f793ad9d640 1 --2- 172.21.15.27:0/110045894 >> v2:172.21.15.69:3300/0 conn(0x7f793c15dd90 0x7f793c0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.457+0000 7f793b59e640 1 -- 172.21.15.27:0/110045894 >> v2:172.21.15.69:3300/0 conn(0x7f793c15dd90 msgr2=0x7f793c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.457+0000 7f793b59e640 1 --2- 172.21.15.27:0/110045894 >> v2:172.21.15.69:3300/0 conn(0x7f793c15dd90 0x7f793c0ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.457+0000 7f793b59e640 1 -- 172.21.15.27:0/110045894 >> v1:172.21.15.27:6789/0 conn(0x7f793c0d29d0 legacy=0x7f793c0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.457+0000 7f793a59c640 1 -- 172.21.15.27:0/110045894 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480069079 0 0) 0x7f793c0d3740 con 0x7f793c0d29d0 2024-03-06T07:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.457+0000 7f793b59e640 1 -- 172.21.15.27:0/110045894 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793c0d3120 con 0x7f793c0b6d10 2024-03-06T07:30:04.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f793b59e640 1 --2- 172.21.15.27:0/110045894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 0x7f793c0b7150 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f793c0d1f90 tx=0x7f793402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d92c316a642903b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f793a59c640 1 -- 172.21.15.27:0/110045894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934039070 con 0x7f793c0b6d10 2024-03-06T07:30:04.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f793a59c640 1 -- 172.21.15.27:0/110045894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793402cd00 con 0x7f793c0b6d10 2024-03-06T07:30:04.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f793a59c640 1 -- 172.21.15.27:0/110045894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934034660 con 0x7f793c0b6d10 2024-03-06T07:30:04.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f79411d0640 1 -- 172.21.15.27:0/110045894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 msgr2=0x7f793c0b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f79411d0640 1 --2- 172.21.15.27:0/110045894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 0x7f793c0b7150 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f793c0d1f90 tx=0x7f793402bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f79411d0640 1 -- 172.21.15.27:0/110045894 shutdown_connections 2024-03-06T07:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f79411d0640 1 --2- 172.21.15.27:0/110045894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 0x7f793c0b7150 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f79411d0640 1 --2- 172.21.15.27:0/110045894 >> v2:172.21.15.69:3300/0 conn(0x7f793c15dd90 0x7f793c0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f79411d0640 1 -- 172.21.15.27:0/110045894 >> 172.21.15.27:0/110045894 conn(0x7f793c071960 msgr2=0x7f793c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f79411d0640 1 -- 172.21.15.27:0/110045894 shutdown_connections 2024-03-06T07:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.458+0000 7f79411d0640 1 -- 172.21.15.27:0/110045894 wait complete. 2024-03-06T07:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.460+0000 7f79411d0640 1 Processor -- start 2024-03-06T07:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.460+0000 7f79411d0640 1 -- start start 2024-03-06T07:30:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.460+0000 7f79411d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 0x7f793c0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.460+0000 7f79411d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f793c0d29d0 0x7f793c160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.460+0000 7f79411d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793c15dd90 0x7f793c0d43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.460+0000 7f79411d0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793c15e170 con 0x7f793c0b6d10 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.460+0000 7f79411d0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793c0d3cc0 con 0x7f793c15dd90 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.460+0000 7f79411d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f793c0d2df0 con 0x7f793c0d29d0 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 0x7f793c0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 0x7f793c0ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57582/0 (socket says 172.21.15.27:57582) 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793ad9d640 1 -- 172.21.15.27:0/3170096230 learned_addr learned my addr 172.21.15.27:0/3170096230 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f7933fff640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f793c0d29d0 0x7f793c160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793b59e640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793c15dd90 0x7f793c0d43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793ad9d640 1 -- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f793c0d29d0 msgr2=0x7f793c160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793ad9d640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f793c0d29d0 0x7f793c160ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793ad9d640 1 -- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793c15dd90 msgr2=0x7f793c0d43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793ad9d640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793c15dd90 0x7f793c0d43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793ad9d640 1 -- 172.21.15.27:0/3170096230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793c0c92f0 con 0x7f793c0b6d10 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f7933fff640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f793c0d29d0 0x7f793c160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793b59e640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793c15dd90 0x7f793c0d43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f793ad9d640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 0x7f793c0ca0b0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f792c00d5f0 tx=0x7f792c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f79337fe640 1 -- 172.21.15.27:0/3170096230 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c0125f0 con 0x7f793c0b6d10 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f79337fe640 1 -- 172.21.15.27:0/3170096230 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792c019070 con 0x7f793c0b6d10 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f793c0c95d0 con 0x7f793c0b6d10 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f79337fe640 1 -- 172.21.15.27:0/3170096230 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c0129d0 con 0x7f793c0b6d10 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.461+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793c0d48f0 con 0x7f793c0b6d10 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.462+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793c0b47a0 con 0x7f793c0b6d10 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.463+0000 7f79337fe640 1 -- 172.21.15.27:0/3170096230 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f792c012b70 con 0x7f793c0b6d10 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.463+0000 7f79337fe640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f791c06b4d0 0x7f791c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.463+0000 7f79337fe640 1 -- 172.21.15.27:0/3170096230 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f792c08a7c0 con 0x7f793c0b6d10 2024-03-06T07:30:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.463+0000 7f7933fff640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f791c06b4d0 0x7f791c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.464+0000 7f7933fff640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f791c06b4d0 0x7f791c06d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f793c15ad30 tx=0x7f7924009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.468+0000 7f79337fe640 1 -- 172.21.15.27:0/3170096230 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f792c059470 con 0x7f793c0b6d10 2024-03-06T07:30:04.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:04.649+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f793c04f410 con 0x7f793c0b6d10 2024-03-06T07:30:05.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.287+0000 7f79337fe640 1 -- 172.21.15.27:0/3170096230 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v122) v1 ==== 181+0+0 (secure 0 0 0) 0x7f792c058d70 con 0x7f793c0b6d10 2024-03-06T07:30:05.290 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:30:05.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f791c06b4d0 msgr2=0x7f791c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f791c06b4d0 0x7f791c06d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f793c15ad30 tx=0x7f7924009040 comp rx=0 tx=0).stop 2024-03-06T07:30:05.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 msgr2=0x7f793c0ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 0x7f793c0ca0b0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f792c00d5f0 tx=0x7f792c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 shutdown_connections 2024-03-06T07:30:05.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f791c06b4d0 0x7f791c06d990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f793c15dd90 0x7f793c0d43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f793c0d29d0 0x7f793c160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 --2- 172.21.15.27:0/3170096230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f793c0b6d10 0x7f793c0ca0b0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 >> 172.21.15.27:0/3170096230 conn(0x7f793c071960 msgr2=0x7f793c0b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 shutdown_connections 2024-03-06T07:30:05.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.291+0000 7f79411d0640 1 -- 172.21.15.27:0/3170096230 wait complete. 2024-03-06T07:30:05.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:30:05.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.418+0000 7fec0ea0e640 1 Processor -- start 2024-03-06T07:30:05.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.418+0000 7fec0ea0e640 1 -- start start 2024-03-06T07:30:05.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.419+0000 7fec0ea0e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec081618e0 0x7fec0806b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.419+0000 7fec0ea0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec080ce960 0x7fec08162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.419+0000 7fec0ea0e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0815f730 con 0x7fec080ce4f0 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.419+0000 7fec0ea0e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec080cfce0 con 0x7fec081618e0 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.419+0000 7fec0ea0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0815ebb0 con 0x7fec080ce960 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.419+0000 7fec077fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fec080ce4f0 0x7fec08161d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43230/0 (socket says 172.21.15.27:43230) 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.419+0000 7fec077fe640 1 -- 172.21.15.27:0/2765948465 learned_addr learned my addr 172.21.15.27:0/2765948465 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.419+0000 7fec0cfab640 1 --2- 172.21.15.27:0/2765948465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec080ce960 0x7fec08162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.420+0000 7fec07fff640 1 --2- 172.21.15.27:0/2765948465 >> v2:172.21.15.69:3300/0 conn(0x7fec081618e0 0x7fec0806b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.420+0000 7fec06ffd640 1 -- 172.21.15.27:0/2765948465 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757978269 0 0) 0x7fec0815f730 con 0x7fec080ce4f0 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.420+0000 7fec06ffd640 1 -- 172.21.15.27:0/2765948465 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf0003520 con 0x7fec080ce4f0 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.420+0000 7fec0cfab640 1 -- 172.21.15.27:0/2765948465 >> v2:172.21.15.69:3300/0 conn(0x7fec081618e0 msgr2=0x7fec0806b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.420+0000 7fec0cfab640 1 --2- 172.21.15.27:0/2765948465 >> v2:172.21.15.69:3300/0 conn(0x7fec081618e0 0x7fec0806b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.420+0000 7fec0cfab640 1 -- 172.21.15.27:0/2765948465 >> v1:172.21.15.27:6789/0 conn(0x7fec080ce4f0 legacy=0x7fec08161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.420+0000 7fec0cfab640 1 -- 172.21.15.27:0/2765948465 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0815ee90 con 0x7fec080ce960 2024-03-06T07:30:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.420+0000 7fec07fff640 1 --2- 172.21.15.27:0/2765948465 >> v2:172.21.15.69:3300/0 conn(0x7fec081618e0 0x7fec0806b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:05.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec0cfab640 1 --2- 172.21.15.27:0/2765948465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec080ce960 0x7fec08162450 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fec00009d90 tx=0x7fec0002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8c6684810ba5b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec06ffd640 1 -- 172.21.15.27:0/2765948465 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec00039070 con 0x7fec080ce960 2024-03-06T07:30:05.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec06ffd640 1 -- 172.21.15.27:0/2765948465 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0002cce0 con 0x7fec080ce960 2024-03-06T07:30:05.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec06ffd640 1 -- 172.21.15.27:0/2765948465 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec00034640 con 0x7fec080ce960 2024-03-06T07:30:05.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec0ea0e640 1 -- 172.21.15.27:0/2765948465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec080ce960 msgr2=0x7fec08162450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec0ea0e640 1 --2- 172.21.15.27:0/2765948465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec080ce960 0x7fec08162450 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fec00009d90 tx=0x7fec0002bf30 comp rx=0 tx=0).stop 2024-03-06T07:30:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec0ea0e640 1 -- 172.21.15.27:0/2765948465 shutdown_connections 2024-03-06T07:30:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec0ea0e640 1 --2- 172.21.15.27:0/2765948465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec080ce960 0x7fec08162450 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec0ea0e640 1 --2- 172.21.15.27:0/2765948465 >> v2:172.21.15.69:3300/0 conn(0x7fec081618e0 0x7fec0806b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec0ea0e640 1 -- 172.21.15.27:0/2765948465 >> 172.21.15.27:0/2765948465 conn(0x7fec080b3660 msgr2=0x7fec080b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.421+0000 7fec0ea0e640 1 -- 172.21.15.27:0/2765948465 shutdown_connections 2024-03-06T07:30:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.422+0000 7fec0ea0e640 1 -- 172.21.15.27:0/2765948465 wait complete. 2024-03-06T07:30:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.423+0000 7fec0ea0e640 1 Processor -- start 2024-03-06T07:30:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.423+0000 7fec0ea0e640 1 -- start start 2024-03-06T07:30:05.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec0ea0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec080ce4f0 0x7fec08154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec0ea0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec080ce960 0x7fec080c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec0ea0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec081558b0 0x7fec0815f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec0ea0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0806b670 con 0x7fec080ce960 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec0ea0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0815f450 con 0x7fec080ce4f0 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec0ea0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec080cedf0 con 0x7fec081558b0 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec080ce960 0x7fec080c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec080ce960 0x7fec080c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57592/0 (socket says 172.21.15.27:57592) 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec077fe640 1 -- 172.21.15.27:0/1154381238 learned_addr learned my addr 172.21.15.27:0/1154381238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec07fff640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec080ce4f0 0x7fec08154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec0cfab640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec081558b0 0x7fec0815f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec077fe640 1 -- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec081558b0 msgr2=0x7fec0815f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec077fe640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec081558b0 0x7fec0815f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec077fe640 1 -- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec080ce4f0 msgr2=0x7fec08154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec077fe640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec080ce4f0 0x7fec08154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec077fe640 1 -- 172.21.15.27:0/1154381238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec080c65f0 con 0x7fec080ce960 2024-03-06T07:30:05.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec0cfab640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec081558b0 0x7fec0815f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec07fff640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec080ce4f0 0x7fec08154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec077fe640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec080ce960 0x7fec080c80f0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7febfc00c430 tx=0x7febfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec04ff9640 1 -- 172.21.15.27:0/1154381238 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc017070 con 0x7fec080ce960 2024-03-06T07:30:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec04ff9640 1 -- 172.21.15.27:0/1154381238 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc005ce0 con 0x7fec080ce960 2024-03-06T07:30:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.424+0000 7fec04ff9640 1 -- 172.21.15.27:0/1154381238 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc0072f0 con 0x7fec080ce960 2024-03-06T07:30:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.425+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec080b59b0 con 0x7fec080ce960 2024-03-06T07:30:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.425+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec080b5c50 con 0x7fec080ce960 2024-03-06T07:30:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.425+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0809fc50 con 0x7fec080ce960 2024-03-06T07:30:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.426+0000 7fec04ff9640 1 -- 172.21.15.27:0/1154381238 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febfc01a070 con 0x7fec080ce960 2024-03-06T07:30:05.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.426+0000 7fec04ff9640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febf406b4d0 0x7febf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.427+0000 7fec04ff9640 1 -- 172.21.15.27:0/1154381238 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5587+0+0 (secure 0 0 0) 0x7febfc089670 con 0x7fec080ce960 2024-03-06T07:30:05.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.427+0000 7fec07fff640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febf406b4d0 0x7febf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.427+0000 7fec07fff640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febf406b4d0 0x7febf406d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7febf8009af0 tx=0x7febf8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.431+0000 7fec04ff9640 1 -- 172.21.15.27:0/1154381238 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febfc058560 con 0x7fec080ce960 2024-03-06T07:30:05.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:05.611+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fec080b2910 con 0x7fec080ce960 2024-03-06T07:30:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.289+0000 7fec04ff9640 1 -- 172.21.15.27:0/1154381238 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v123) v1 ==== 169+0+0 (secure 0 0 0) 0x7febfc057e60 con 0x7fec080ce960 2024-03-06T07:30:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:30:06.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.292+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febf406b4d0 msgr2=0x7febf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febf406b4d0 0x7febf406d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7febf8009af0 tx=0x7febf8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec080ce960 msgr2=0x7fec080c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec080ce960 0x7fec080c80f0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7febfc00c430 tx=0x7febfc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 shutdown_connections 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7febf406b4d0 0x7febf406d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fec081558b0 0x7fec0815f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec080ce960 0x7fec080c80f0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 --2- 172.21.15.27:0/1154381238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec080ce4f0 0x7fec08154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 >> 172.21.15.27:0/1154381238 conn(0x7fec080b3660 msgr2=0x7fec0806f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 shutdown_connections 2024-03-06T07:30:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.293+0000 7fec0ea0e640 1 -- 172.21.15.27:0/1154381238 wait complete. 2024-03-06T07:30:06.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:30:06.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.417+0000 7f99d15d0640 1 Processor -- start 2024-03-06T07:30:06.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.417+0000 7f99d15d0640 1 -- start start 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.417+0000 7f99d15d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc15b130 0x7f99cc15b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99d15d0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99cc15bc40 con 0x7f99cc144640 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99d15d0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99cc15bf20 con 0x7f99cc0ce900 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99d15d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc06df70 con 0x7f99cc15b130 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99caffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f99cc144640 0x7f99cc06e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43236/0 (socket says 172.21.15.27:43236) 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99caffd640 1 -- 172.21.15.27:0/2801807497 learned_addr learned my addr 172.21.15.27:0/2801807497 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99c3fff640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc15b130 0x7f99cc15b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652825439 0 0) 0x7f99cc15bc40 con 0x7f99cc144640 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99b8003520 con 0x7f99cc144640 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2643062484 0 0) 0x7f99b8003520 con 0x7f99cc144640 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc15b130 msgr2=0x7f99cc15b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc15b130 0x7f99cc15b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 >> v1:172.21.15.69:6789/0 conn(0x7f99cc0ce900 legacy=0x7f99cc0b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc06dc00 con 0x7f99cc144640 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f99bc001670 con 0x7f99cc144640 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 >> v1:172.21.15.27:6789/0 conn(0x7f99cc144640 legacy=0x7f99cc06e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b80039e0 0x7f99b8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc15b130 0x7f99b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc06df70 con 0x7f99b8004800 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b800baa0 con 0x7f99b80039e0 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b800deb0 con 0x7f99cc15b130 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99c3fff640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.418+0000 7f99caffd640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b80039e0 0x7f99b8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.419+0000 7f99cb7fe640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc15b130 0x7f99b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.419+0000 7f99c3fff640 1 -- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc15b130 msgr2=0x7f99b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.419+0000 7f99c3fff640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc15b130 0x7f99b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.419+0000 7f99c3fff640 1 -- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b80039e0 msgr2=0x7f99b8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.419+0000 7f99c3fff640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99b80039e0 0x7f99b8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.419+0000 7f99c3fff640 1 -- 172.21.15.27:0/2801807497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b8003520 con 0x7f99b8004800 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.419+0000 7f99cb7fe640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc15b130 0x7f99b80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.419+0000 7f99c3fff640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99b8004c80 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009a90 tx=0x7f99b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2761ccfe6fe57e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b4039070 con 0x7f99b8004800 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b402c680 con 0x7f99b8004800 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99ca7fc640 1 -- 172.21.15.27:0/2801807497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b4009690 con 0x7f99b8004800 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99d15d0640 1 -- 172.21.15.27:0/2801807497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 msgr2=0x7f99b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99d15d0640 1 --2- 172.21.15.27:0/2801807497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99b8004c80 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009a90 tx=0x7f99b8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:06.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99cb7fe640 1 -- 172.21.15.27:0/2801807497 reap_dead start 2024-03-06T07:30:06.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99d15d0640 1 -- 172.21.15.27:0/2801807497 shutdown_connections 2024-03-06T07:30:06.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99d15d0640 1 -- 172.21.15.27:0/2801807497 >> 172.21.15.27:0/2801807497 conn(0x7f99cc066d00 msgr2=0x7f99cc067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99d15d0640 1 -- 172.21.15.27:0/2801807497 shutdown_connections 2024-03-06T07:30:06.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.420+0000 7f99d15d0640 1 -- 172.21.15.27:0/2801807497 wait complete. 2024-03-06T07:30:06.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.421+0000 7f99d15d0640 1 Processor -- start 2024-03-06T07:30:06.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.421+0000 7f99d15d0640 1 -- start start 2024-03-06T07:30:06.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99d15d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99cc158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99d15d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc138620 0x7f99cc137b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99d15d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc15cb50 0x7f99cc15cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99d15d0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc070c20 con 0x7f99b8004800 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99d15d0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc070aa0 con 0x7f99cc15cb50 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99d15d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc0b3de0 con 0x7f99cc138620 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99caffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99cc158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99caffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99cc158490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57618/0 (socket says 172.21.15.27:57618) 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99caffd640 1 -- 172.21.15.27:0/4269055659 learned_addr learned my addr 172.21.15.27:0/4269055659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99caffd640 1 -- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc138620 msgr2=0x7f99cc137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99c3fff640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc138620 0x7f99cc137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99cb7fe640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc15cb50 0x7f99cc15cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99caffd640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc138620 0x7f99cc137b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99caffd640 1 -- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc15cb50 msgr2=0x7f99cc15cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99caffd640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc15cb50 0x7f99cc15cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.422+0000 7f99caffd640 1 -- 172.21.15.27:0/4269055659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc1f0140 con 0x7f99b8004800 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.423+0000 7f99caffd640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99cc158490 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f99bc007290 tx=0x7f99bc007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.423+0000 7f99c37fe640 1 -- 172.21.15.27:0/4269055659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc00f960 con 0x7f99b8004800 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.423+0000 7f99c37fe640 1 -- 172.21.15.27:0/4269055659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99bc018070 con 0x7f99b8004800 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.423+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99cc158c80 con 0x7f99b8004800 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.423+0000 7f99c37fe640 1 -- 172.21.15.27:0/4269055659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc00fd40 con 0x7f99b8004800 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.423+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99cc159240 con 0x7f99b8004800 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.424+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99cc0733c0 con 0x7f99b8004800 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.424+0000 7f99c37fe640 1 -- 172.21.15.27:0/4269055659 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99bc01d460 con 0x7f99b8004800 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.425+0000 7f99c37fe640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99ac06b4d0 0x7f99ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.425+0000 7f99c37fe640 1 -- 172.21.15.27:0/4269055659 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f99bc08a0b0 con 0x7f99b8004800 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.425+0000 7f99c3fff640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99ac06b4d0 0x7f99ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.425+0000 7f99c3fff640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99ac06b4d0 0x7f99ac06d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f99b4036000 tx=0x7f99b4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.429+0000 7f99c37fe640 1 -- 172.21.15.27:0/4269055659 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99bc0591f0 con 0x7f99b8004800 2024-03-06T07:30:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.610+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f99cc15da00 con 0x7f99b8004800 2024-03-06T07:30:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.613+0000 7f99c37fe640 1 -- 172.21.15.27:0/4269055659 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f99bc058af0 con 0x7f99b8004800 2024-03-06T07:30:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99ac06b4d0 msgr2=0x7f99ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99ac06b4d0 0x7f99ac06d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f99b4036000 tx=0x7f99b4030000 comp rx=0 tx=0).stop 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 msgr2=0x7f99cc158490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99cc158490 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f99bc007290 tx=0x7f99bc007750 comp rx=0 tx=0).stop 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 shutdown_connections 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99ac06b4d0 0x7f99ac06d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99cc15cb50 0x7f99cc15cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99cc138620 0x7f99cc137b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 --2- 172.21.15.27:0/4269055659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b8004800 0x7f99cc158490 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 >> 172.21.15.27:0/4269055659 conn(0x7f99cc066d00 msgr2=0x7f99cc15c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 shutdown_connections 2024-03-06T07:30:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.615+0000 7f99d15d0640 1 -- 172.21.15.27:0/4269055659 wait complete. 2024-03-06T07:30:06.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:30:06.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.734+0000 7f56f04df640 1 Processor -- start 2024-03-06T07:30:06.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.734+0000 7f56f04df640 1 -- start start 2024-03-06T07:30:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.735+0000 7f56f04df640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56e80d0880 0x7f56e8168c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.735+0000 7f56f04df640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e80cf130 con 0x7f56e815e060 2024-03-06T07:30:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.735+0000 7f56f04df640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e80d0dd0 con 0x7f56e806fe20 2024-03-06T07:30:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.735+0000 7f56f04df640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e80d10b0 con 0x7f56e80d0880 2024-03-06T07:30:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.735+0000 7f56eea7c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56e80d0880 0x7f56e8168c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.735+0000 7f56ee27b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f56e815e060 0x7f56e80ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43238/0 (socket says 172.21.15.27:43238) 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.735+0000 7f56eea7c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56e80d0880 0x7f56e8168c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57628/0 (socket says 172.21.15.27:57628) 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.735+0000 7f56ee27b640 1 -- 172.21.15.27:0/3678378533 learned_addr learned my addr 172.21.15.27:0/3678378533 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.736+0000 7f56eea7c640 1 -- 172.21.15.27:0/3678378533 >> v1:172.21.15.69:6789/0 conn(0x7f56e806fe20 legacy=0x7f56e80cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.736+0000 7f56eea7c640 1 -- 172.21.15.27:0/3678378533 >> v1:172.21.15.27:6789/0 conn(0x7f56e815e060 legacy=0x7f56e80ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.736+0000 7f56eea7c640 1 -- 172.21.15.27:0/3678378533 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e80cc0e0 con 0x7f56e80d0880 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.736+0000 7f56ed279640 1 -- 172.21.15.27:0/3678378533 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059190240 0 0) 0x7f56e80cf130 con 0x7f56e815e060 2024-03-06T07:30:06.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.736+0000 7f56eea7c640 1 --2- 172.21.15.27:0/3678378533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e80d0880 0x7f56e8168c20 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009d90 tx=0x7f56d802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ff098a059dcffd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56ed279640 1 -- 172.21.15.27:0/3678378533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8039070 con 0x7f56e80d0880 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56ed279640 1 -- 172.21.15.27:0/3678378533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d802cd00 con 0x7f56e80d0880 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56ed279640 1 -- 172.21.15.27:0/3678378533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8034660 con 0x7f56e80d0880 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56f04df640 1 -- 172.21.15.27:0/3678378533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e80d0880 msgr2=0x7f56e8168c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56f04df640 1 --2- 172.21.15.27:0/3678378533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e80d0880 0x7f56e8168c20 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009d90 tx=0x7f56d802bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56f04df640 1 -- 172.21.15.27:0/3678378533 shutdown_connections 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56f04df640 1 --2- 172.21.15.27:0/3678378533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e80d0880 0x7f56e8168c20 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56f04df640 1 -- 172.21.15.27:0/3678378533 >> 172.21.15.27:0/3678378533 conn(0x7f56e80d1a90 msgr2=0x7f56e80d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56f04df640 1 -- 172.21.15.27:0/3678378533 shutdown_connections 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.737+0000 7f56f04df640 1 -- 172.21.15.27:0/3678378533 wait complete. 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.738+0000 7f56f04df640 1 Processor -- start 2024-03-06T07:30:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.738+0000 7f56f04df640 1 -- start start 2024-03-06T07:30:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.739+0000 7f56f04df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e806fe20 0x7f56e815e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.739+0000 7f56f04df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56e80d0880 0x7f56e80b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.739+0000 7f56f04df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56e815e060 0x7f56e8160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.739+0000 7f56f04df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e815e440 con 0x7f56e806fe20 2024-03-06T07:30:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.739+0000 7f56f04df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e80cc6a0 con 0x7f56e80d0880 2024-03-06T07:30:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.739+0000 7f56f04df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e80cdbd0 con 0x7f56e815e060 2024-03-06T07:30:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.739+0000 7f56ee27b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e806fe20 0x7f56e815e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.739+0000 7f56ee27b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e806fe20 0x7f56e815e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57634/0 (socket says 172.21.15.27:57634) 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.739+0000 7f56ee27b640 1 -- 172.21.15.27:0/110743128 learned_addr learned my addr 172.21.15.27:0/110743128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56eda7a640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56e80d0880 0x7f56e80b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56eea7c640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56e815e060 0x7f56e8160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56ee27b640 1 -- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56e815e060 msgr2=0x7f56e8160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56ee27b640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56e815e060 0x7f56e8160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56ee27b640 1 -- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56e80d0880 msgr2=0x7f56e80b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56ee27b640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56e80d0880 0x7f56e80b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56ee27b640 1 -- 172.21.15.27:0/110743128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e80c5e10 con 0x7f56e806fe20 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56eea7c640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56e815e060 0x7f56e8160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56eda7a640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56e80d0880 0x7f56e80b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56ee27b640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e806fe20 0x7f56e815e830 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f56e000c420 tx=0x7f56e000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56e6ffd640 1 -- 172.21.15.27:0/110743128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e00074e0 con 0x7f56e806fe20 2024-03-06T07:30:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56e6ffd640 1 -- 172.21.15.27:0/110743128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e00047e0 con 0x7f56e806fe20 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e80c60f0 con 0x7f56e806fe20 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56e6ffd640 1 -- 172.21.15.27:0/110743128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e0004c10 con 0x7f56e806fe20 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.740+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e80c66b0 con 0x7f56e806fe20 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.741+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e8067ff0 con 0x7f56e806fe20 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.741+0000 7f56e6ffd640 1 -- 172.21.15.27:0/110743128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56e0011070 con 0x7f56e806fe20 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.742+0000 7f56e6ffd640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56d406b4d0 0x7f56d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.742+0000 7f56e6ffd640 1 -- 172.21.15.27:0/110743128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f56e0089bd0 con 0x7f56e806fe20 2024-03-06T07:30:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.742+0000 7f56eda7a640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56d406b4d0 0x7f56d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.743+0000 7f56eda7a640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56d406b4d0 0x7f56d406d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f56e80b2bf0 tx=0x7f56dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.746+0000 7f56e6ffd640 1 -- 172.21.15.27:0/110743128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56e000e280 con 0x7f56e806fe20 2024-03-06T07:30:06.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:06.997+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f56e80d20c0 con 0x7f56e806fe20 2024-03-06T07:30:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.291+0000 7f56e6ffd640 1 -- 172.21.15.27:0/110743128 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v401) v1 ==== 142+0+0 (secure 0 0 0) 0x7f56e0058ca0 con 0x7f56e806fe20 2024-03-06T07:30:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56d406b4d0 msgr2=0x7f56d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56d406b4d0 0x7f56d406d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f56e80b2bf0 tx=0x7f56dc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e806fe20 msgr2=0x7f56e815e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e806fe20 0x7f56e815e830 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f56e000c420 tx=0x7f56e000c8e0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 shutdown_connections 2024-03-06T07:30:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56d406b4d0 0x7f56d406d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f56e815e060 0x7f56e8160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56e80d0880 0x7f56e80b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 --2- 172.21.15.27:0/110743128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56e806fe20 0x7f56e815e830 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 >> 172.21.15.27:0/110743128 conn(0x7f56e80d1a90 msgr2=0x7f56e806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 shutdown_connections 2024-03-06T07:30:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.295+0000 7f56f04df640 1 -- 172.21.15.27:0/110743128 wait complete. 2024-03-06T07:30:07.309 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:30:07.309 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:30:07.309 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:30:07.310 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:07.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.415+0000 7ff89dbf7640 1 Processor -- start 2024-03-06T07:30:07.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.415+0000 7ff89dbf7640 1 -- start start 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.415+0000 7ff89dbf7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff89806fe20 0x7ff8980cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.415+0000 7ff89dbf7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980d0880 0x7ff8980ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.415+0000 7ff89dbf7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8980d0dd0 con 0x7ff89815e060 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.415+0000 7ff89dbf7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8980d10b0 con 0x7ff89806fe20 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.415+0000 7ff89dbf7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980cc0e0 con 0x7ff8980d0880 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff896ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff89806fe20 0x7ff8980cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff8977fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff89815e060 0x7ff8980ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43246/0 (socket says 172.21.15.27:43246) 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff8977fe640 1 -- 172.21.15.27:0/2528686898 learned_addr learned my addr 172.21.15.27:0/2528686898 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff897fff640 1 --2- 172.21.15.27:0/2528686898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980d0880 0x7ff8980ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff896ffd640 1 -- 172.21.15.27:0/2528686898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980d0880 msgr2=0x7ff8980ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff896ffd640 1 --2- 172.21.15.27:0/2528686898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980d0880 0x7ff8980ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff896ffd640 1 -- 172.21.15.27:0/2528686898 >> v1:172.21.15.27:6789/0 conn(0x7ff89815e060 legacy=0x7ff8980ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff8967fc640 1 -- 172.21.15.27:0/2528686898 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210120302 0 0) 0x7ff8980d0dd0 con 0x7ff89815e060 2024-03-06T07:30:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff896ffd640 1 -- 172.21.15.27:0/2528686898 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980cc3c0 con 0x7ff89806fe20 2024-03-06T07:30:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.416+0000 7ff897fff640 1 --2- 172.21.15.27:0/2528686898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980d0880 0x7ff8980ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:07.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.417+0000 7ff896ffd640 1 --2- 172.21.15.27:0/2528686898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff89806fe20 0x7ff8980cea20 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7ff8880097a0 tx=0x7ff88802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898b685418b17ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.417+0000 7ff8967fc640 1 -- 172.21.15.27:0/2528686898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888035020 con 0x7ff89806fe20 2024-03-06T07:30:07.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.417+0000 7ff8967fc640 1 -- 172.21.15.27:0/2528686898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff888031830 con 0x7ff89806fe20 2024-03-06T07:30:07.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.417+0000 7ff8967fc640 1 -- 172.21.15.27:0/2528686898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888031c00 con 0x7ff89806fe20 2024-03-06T07:30:07.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.417+0000 7ff89dbf7640 1 -- 172.21.15.27:0/2528686898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff89806fe20 msgr2=0x7ff8980cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.417+0000 7ff89dbf7640 1 --2- 172.21.15.27:0/2528686898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff89806fe20 0x7ff8980cea20 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7ff8880097a0 tx=0x7ff88802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.417+0000 7ff89dbf7640 1 -- 172.21.15.27:0/2528686898 shutdown_connections 2024-03-06T07:30:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.417+0000 7ff89dbf7640 1 --2- 172.21.15.27:0/2528686898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff8980d0880 0x7ff8980ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.417+0000 7ff89dbf7640 1 --2- 172.21.15.27:0/2528686898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff89806fe20 0x7ff8980cea20 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.418+0000 7ff89dbf7640 1 -- 172.21.15.27:0/2528686898 >> 172.21.15.27:0/2528686898 conn(0x7ff8980d1a90 msgr2=0x7ff8980d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.418+0000 7ff89dbf7640 1 -- 172.21.15.27:0/2528686898 shutdown_connections 2024-03-06T07:30:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.418+0000 7ff89dbf7640 1 -- 172.21.15.27:0/2528686898 wait complete. 2024-03-06T07:30:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.419+0000 7ff89dbf7640 1 Processor -- start 2024-03-06T07:30:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.419+0000 7ff89dbf7640 1 -- start start 2024-03-06T07:30:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff89dbf7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff89806fe20 0x7ff8980c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff89dbf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8980d0880 0x7ff8980b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff89dbf7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff89815e060 0x7ff898153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff89dbf7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980cef60 con 0x7ff8980d0880 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff89dbf7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89815e440 con 0x7ff89806fe20 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff89dbf7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980cf0e0 con 0x7ff89815e060 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff896ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8980d0880 0x7ff8980b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff896ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8980d0880 0x7ff8980b72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57646/0 (socket says 172.21.15.27:57646) 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff896ffd640 1 -- 172.21.15.27:0/4208143011 learned_addr learned my addr 172.21.15.27:0/4208143011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff897fff640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff89815e060 0x7ff898153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff896ffd640 1 -- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff89815e060 msgr2=0x7ff898153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff896ffd640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff89815e060 0x7ff898153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff896ffd640 1 -- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff89806fe20 msgr2=0x7ff8980c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff896ffd640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff89806fe20 0x7ff8980c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff896ffd640 1 -- 172.21.15.27:0/4208143011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff898153ad0 con 0x7ff8980d0880 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff897fff640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff89815e060 0x7ff898153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff896ffd640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8980d0880 0x7ff8980b72a0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7ff888034040 tx=0x7ff8880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff87ffff640 1 -- 172.21.15.27:0/4208143011 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888040070 con 0x7ff8980d0880 2024-03-06T07:30:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff87ffff640 1 -- 172.21.15.27:0/4208143011 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff888039a20 con 0x7ff8980d0880 2024-03-06T07:30:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89815e5c0 con 0x7ff8980d0880 2024-03-06T07:30:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.420+0000 7ff87ffff640 1 -- 172.21.15.27:0/4208143011 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888035040 con 0x7ff8980d0880 2024-03-06T07:30:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.421+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8980c5ed0 con 0x7ff8980d0880 2024-03-06T07:30:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.422+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff898067ff0 con 0x7ff8980d0880 2024-03-06T07:30:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.422+0000 7ff87ffff640 1 -- 172.21.15.27:0/4208143011 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff888038020 con 0x7ff8980d0880 2024-03-06T07:30:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.423+0000 7ff87ffff640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff88406b4d0 0x7ff88406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.423+0000 7ff87ffff640 1 -- 172.21.15.27:0/4208143011 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5115+0+0 (secure 0 0 0) 0x7ff8880ab990 con 0x7ff8980d0880 2024-03-06T07:30:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.423+0000 7ff8977fe640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff88406b4d0 0x7ff88406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.424+0000 7ff8977fe640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff88406b4d0 0x7ff88406d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff88c007fd0 tx=0x7ff88c007920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.427+0000 7ff87ffff640 1 -- 172.21.15.27:0/4208143011 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff898153ad0 con 0x7ff8980d0880 2024-03-06T07:30:07.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.608+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8980ce3b0 con 0x7ff8980d0880 2024-03-06T07:30:07.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.609+0000 7ff87ffff640 1 -- 172.21.15.27:0/4208143011 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+13218 (secure 0 0 0) 0x7ff88807aaf0 con 0x7ff8980d0880 2024-03-06T07:30:07.611 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:07.611 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:30:06.289255+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6833/2669617374":"2024-03-07T07:30:03.284721+0000","172.21.15.27:6832/2669617374":"2024-03-07T07:30:03.284721+0000","172.21.15.27:6832/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6834/3926923781":"2024-03-07T07:29:00.663556+0000","172.21.15.27:6835/1655684069":"2024-03-07T07:28:59.663475+0000","172.21.15.27:6833/1655684069":"2024-03-07T07:28:59.663475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:07.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff88406b4d0 msgr2=0x7ff88406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff88406b4d0 0x7ff88406d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff88c007fd0 tx=0x7ff88c007920 comp rx=0 tx=0).stop 2024-03-06T07:30:07.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8980d0880 msgr2=0x7ff8980b72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8980d0880 0x7ff8980b72a0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7ff888034040 tx=0x7ff8880046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 shutdown_connections 2024-03-06T07:30:07.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff88406b4d0 0x7ff88406d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff89815e060 0x7ff898153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8980d0880 0x7ff8980b72a0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 --2- 172.21.15.27:0/4208143011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff89806fe20 0x7ff8980c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 >> 172.21.15.27:0/4208143011 conn(0x7ff8980d1a90 msgr2=0x7ff8980d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 shutdown_connections 2024-03-06T07:30:07.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.611+0000 7ff89dbf7640 1 -- 172.21.15.27:0/4208143011 wait complete. 2024-03-06T07:30:07.624 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:07.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.732+0000 7fdf49246640 1 Processor -- start 2024-03-06T07:30:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.732+0000 7fdf49246640 1 -- start start 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf49246640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdf440717b0 0x7fdf44071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf49246640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b3060 0x7fdf440d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf49246640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4415e4b0 con 0x7fdf440b2b10 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf49246640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf440d2e10 con 0x7fdf440717b0 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf49246640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf440b97e0 con 0x7fdf440b3060 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf42d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdf440717b0 0x7fdf44071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf42d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdf440717b0 0x7fdf44071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57656/0 (socket says 172.21.15.27:57656) 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf42d9d640 1 -- 172.21.15.27:0/1901745310 learned_addr learned my addr 172.21.15.27:0/1901745310 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf3bfff640 1 --2- 172.21.15.27:0/1901745310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b3060 0x7fdf440d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf42d9d640 1 -- 172.21.15.27:0/1901745310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b3060 msgr2=0x7fdf440d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf42d9d640 1 --2- 172.21.15.27:0/1901745310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b3060 0x7fdf440d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf42d9d640 1 -- 172.21.15.27:0/1901745310 >> v1:172.21.15.69:6789/0 conn(0x7fdf440b2b10 legacy=0x7fdf440b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.733+0000 7fdf42d9d640 1 -- 172.21.15.27:0/1901745310 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf440b9a80 con 0x7fdf440717b0 2024-03-06T07:30:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.734+0000 7fdf42d9d640 1 --2- 172.21.15.27:0/1901745310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440717b0 0x7fdf44071b90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009870 tx=0x7fdf3402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d42d163a6791fc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.734+0000 7fdf4259c640 1 -- 172.21.15.27:0/1901745310 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34034070 con 0x7fdf440717b0 2024-03-06T07:30:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.734+0000 7fdf4259c640 1 -- 172.21.15.27:0/1901745310 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf34030840 con 0x7fdf440717b0 2024-03-06T07:30:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.734+0000 7fdf4259c640 1 -- 172.21.15.27:0/1901745310 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34030c10 con 0x7fdf440717b0 2024-03-06T07:30:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.735+0000 7fdf49246640 1 -- 172.21.15.27:0/1901745310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440717b0 msgr2=0x7fdf44071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.735+0000 7fdf49246640 1 --2- 172.21.15.27:0/1901745310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440717b0 0x7fdf44071b90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009870 tx=0x7fdf3402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.735+0000 7fdf49246640 1 -- 172.21.15.27:0/1901745310 shutdown_connections 2024-03-06T07:30:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.735+0000 7fdf49246640 1 --2- 172.21.15.27:0/1901745310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b3060 0x7fdf440d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.735+0000 7fdf49246640 1 --2- 172.21.15.27:0/1901745310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440717b0 0x7fdf44071b90 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.735+0000 7fdf49246640 1 -- 172.21.15.27:0/1901745310 >> 172.21.15.27:0/1901745310 conn(0x7fdf440d3d20 msgr2=0x7fdf440d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.735+0000 7fdf49246640 1 -- 172.21.15.27:0/1901745310 shutdown_connections 2024-03-06T07:30:07.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.735+0000 7fdf49246640 1 -- 172.21.15.27:0/1901745310 wait complete. 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.736+0000 7fdf49246640 1 Processor -- start 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.736+0000 7fdf49246640 1 -- start start 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf49246640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf440717b0 0x7fdf4414dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf49246640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b2b10 0x7fdf440cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf49246640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440b3060 0x7fdf440cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf49246640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf440b7560 con 0x7fdf440b3060 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf49246640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf440d4bf0 con 0x7fdf440717b0 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf49246640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf440b76e0 con 0x7fdf440b2b10 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf4359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440b3060 0x7fdf440cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf4359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440b3060 0x7fdf440cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57664/0 (socket says 172.21.15.27:57664) 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf4359e640 1 -- 172.21.15.27:0/2494971015 learned_addr learned my addr 172.21.15.27:0/2494971015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf3bfff640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b2b10 0x7fdf440cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf42d9d640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf440717b0 0x7fdf4414dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf4359e640 1 -- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b2b10 msgr2=0x7fdf440cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf4359e640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b2b10 0x7fdf440cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf4359e640 1 -- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf440717b0 msgr2=0x7fdf4414dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf4359e640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf440717b0 0x7fdf4414dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.737+0000 7fdf4359e640 1 -- 172.21.15.27:0/2494971015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf440cc770 con 0x7fdf440b3060 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.738+0000 7fdf4359e640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440b3060 0x7fdf440cd090 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c740 tx=0x7fdf3c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.738+0000 7fdf3b7fe640 1 -- 172.21.15.27:0/2494971015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c016020 con 0x7fdf440b3060 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.738+0000 7fdf3b7fe640 1 -- 172.21.15.27:0/2494971015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c00ce50 con 0x7fdf440b3060 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.738+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf440cb6d0 con 0x7fdf440b3060 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.738+0000 7fdf3b7fe640 1 -- 172.21.15.27:0/2494971015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c003970 con 0x7fdf440b3060 2024-03-06T07:30:07.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.738+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf440cbc90 con 0x7fdf440b3060 2024-03-06T07:30:07.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.739+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4406ce40 con 0x7fdf440b3060 2024-03-06T07:30:07.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.739+0000 7fdf3b7fe640 1 -- 172.21.15.27:0/2494971015 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf3c003b10 con 0x7fdf440b3060 2024-03-06T07:30:07.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.740+0000 7fdf3b7fe640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf2406b4d0 0x7fdf2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.740+0000 7fdf3b7fe640 1 -- 172.21.15.27:0/2494971015 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fdf3c089a20 con 0x7fdf440b3060 2024-03-06T07:30:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.740+0000 7fdf42d9d640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf2406b4d0 0x7fdf2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.740+0000 7fdf42d9d640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf2406b4d0 0x7fdf2406d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdf34036000 tx=0x7fdf34008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.744+0000 7fdf3b7fe640 1 -- 172.21.15.27:0/2494971015 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf3c020020 con 0x7fdf440b3060 2024-03-06T07:30:07.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.927+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf4404f410 con 0x7fdf440b3060 2024-03-06T07:30:07.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.927+0000 7fdf3b7fe640 1 -- 172.21.15.27:0/2494971015 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf4415e4b0 con 0x7fdf440b3060 2024-03-06T07:30:07.930 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:30:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf2406b4d0 msgr2=0x7fdf2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf2406b4d0 0x7fdf2406d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdf34036000 tx=0x7fdf34008030 comp rx=0 tx=0).stop 2024-03-06T07:30:07.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440b3060 msgr2=0x7fdf440cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440b3060 0x7fdf440cd090 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c740 tx=0x7fdf3c00cc00 comp rx=0 tx=0).stop 2024-03-06T07:30:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 shutdown_connections 2024-03-06T07:30:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf2406b4d0 0x7fdf2406d990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf440b3060 0x7fdf440cd090 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf440b2b10 0x7fdf440cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 --2- 172.21.15.27:0/2494971015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf440717b0 0x7fdf4414dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 >> 172.21.15.27:0/2494971015 conn(0x7fdf440d3d20 msgr2=0x7fdf4415cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 shutdown_connections 2024-03-06T07:30:07.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:07.929+0000 7fdf49246640 1 -- 172.21.15.27:0/2494971015 wait complete. 2024-03-06T07:30:07.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:30:08.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.048+0000 7f782d41c640 1 Processor -- start 2024-03-06T07:30:08.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.048+0000 7f782d41c640 1 -- start start 2024-03-06T07:30:08.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f782d41c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7828071540 0x7f78280b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f782d41c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 0x7f7828071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f782d41c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78280b2b50 0x7f78280d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f782d41c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78280d2f70 con 0x7f7828071540 2024-03-06T07:30:08.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f782d41c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78280d4be0 con 0x7f78280b2b50 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f782d41c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7828071e60 con 0x7f78280d3400 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f7826ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7828071540 0x7f78280b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f7826ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7828071540 0x7f78280b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57672/0 (socket says 172.21.15.27:57672) 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f7826ffd640 1 -- 172.21.15.27:0/776229903 learned_addr learned my addr 172.21.15.27:0/776229903 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f78277fe640 1 --2- 172.21.15.27:0/776229903 >> v2:172.21.15.69:3300/0 conn(0x7f78280b2b50 0x7f78280d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.049+0000 7f78267fc640 1 --2- 172.21.15.27:0/776229903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 0x7f7828071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.050+0000 7f7826ffd640 1 -- 172.21.15.27:0/776229903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 msgr2=0x7f7828071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.050+0000 7f7826ffd640 1 --2- 172.21.15.27:0/776229903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 0x7f7828071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.050+0000 7f7826ffd640 1 -- 172.21.15.27:0/776229903 >> v2:172.21.15.69:3300/0 conn(0x7f78280b2b50 msgr2=0x7f78280d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.050+0000 7f7826ffd640 1 --2- 172.21.15.27:0/776229903 >> v2:172.21.15.69:3300/0 conn(0x7f78280b2b50 0x7f78280d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.050+0000 7f7826ffd640 1 -- 172.21.15.27:0/776229903 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78280ce610 con 0x7f7828071540 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.050+0000 7f78267fc640 1 --2- 172.21.15.27:0/776229903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 0x7f7828071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.050+0000 7f7826ffd640 1 --2- 172.21.15.27:0/776229903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 0x7f78280b30a0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f782806c230 tx=0x7f782002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb8e4d9be8f7f3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f7825ffb640 1 -- 172.21.15.27:0/776229903 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820035020 con 0x7f7828071540 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f7825ffb640 1 -- 172.21.15.27:0/776229903 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7820031870 con 0x7f7828071540 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f7825ffb640 1 -- 172.21.15.27:0/776229903 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820031c40 con 0x7f7828071540 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f782d41c640 1 -- 172.21.15.27:0/776229903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 msgr2=0x7f78280b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f782d41c640 1 --2- 172.21.15.27:0/776229903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 0x7f78280b30a0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f782806c230 tx=0x7f782002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f782d41c640 1 -- 172.21.15.27:0/776229903 shutdown_connections 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f782d41c640 1 --2- 172.21.15.27:0/776229903 >> v2:172.21.15.69:3300/0 conn(0x7f78280b2b50 0x7f78280d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f782d41c640 1 --2- 172.21.15.27:0/776229903 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 0x7f7828071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f782d41c640 1 --2- 172.21.15.27:0/776229903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 0x7f78280b30a0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f782d41c640 1 -- 172.21.15.27:0/776229903 >> 172.21.15.27:0/776229903 conn(0x7f78280d3ca0 msgr2=0x7f78280d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f782d41c640 1 -- 172.21.15.27:0/776229903 shutdown_connections 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.051+0000 7f782d41c640 1 -- 172.21.15.27:0/776229903 wait complete. 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f782d41c640 1 Processor -- start 2024-03-06T07:30:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f782d41c640 1 -- start start 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f782d41c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 0x7f782815cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f782d41c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78280b2b50 0x7f78281f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f782d41c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 0x7f78280ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f782d41c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78280ce8f0 con 0x7f7828071540 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f782d41c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78280d2c00 con 0x7f78280b2b50 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f782d41c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782815e370 con 0x7f78280d3400 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f7826ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 0x7f782815cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f7826ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 0x7f782815cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57678/0 (socket says 172.21.15.27:57678) 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.053+0000 7f7826ffd640 1 -- 172.21.15.27:0/1330698041 learned_addr learned my addr 172.21.15.27:0/1330698041 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f78267fc640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78280b2b50 0x7f78281f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f78277fe640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 0x7f78280ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f7826ffd640 1 -- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 msgr2=0x7f78280ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f7826ffd640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 0x7f78280ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f7826ffd640 1 -- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78280b2b50 msgr2=0x7f78281f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f7826ffd640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78280b2b50 0x7f78281f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f7826ffd640 1 -- 172.21.15.27:0/1330698041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78281f3ed0 con 0x7f7828071540 2024-03-06T07:30:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f7826ffd640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 0x7f782815cd80 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f78200097c0 tx=0x7f78200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f780f7fe640 1 -- 172.21.15.27:0/1330698041 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820040070 con 0x7f7828071540 2024-03-06T07:30:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f780f7fe640 1 -- 172.21.15.27:0/1330698041 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7820030d00 con 0x7f7828071540 2024-03-06T07:30:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78281f41b0 con 0x7f7828071540 2024-03-06T07:30:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f780f7fe640 1 -- 172.21.15.27:0/1330698041 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820035040 con 0x7f7828071540 2024-03-06T07:30:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.054+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78281f4770 con 0x7f7828071540 2024-03-06T07:30:08.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.055+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7828068700 con 0x7f7828071540 2024-03-06T07:30:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.056+0000 7f780f7fe640 1 -- 172.21.15.27:0/1330698041 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f782004b060 con 0x7f7828071540 2024-03-06T07:30:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.056+0000 7f780f7fe640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f781406b4d0 0x7f781406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.056+0000 7f780f7fe640 1 -- 172.21.15.27:0/1330698041 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f78200abb80 con 0x7f7828071540 2024-03-06T07:30:08.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.056+0000 7f78267fc640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f781406b4d0 0x7f781406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.057+0000 7f78267fc640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f781406b4d0 0x7f781406d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f78280b7af0 tx=0x7f781c009870 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.060+0000 7f780f7fe640 1 -- 172.21.15.27:0/1330698041 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f782007a580 con 0x7f7828071540 2024-03-06T07:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:08.314+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f782806b8e0 con 0x7f7828071540 2024-03-06T07:30:09.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.298+0000 7f780f7fe640 1 -- 172.21.15.27:0/1330698041 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v403) v1 ==== 143+0+0 (secure 0 0 0) 0x7f7820099120 con 0x7f7828071540 2024-03-06T07:30:09.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f781406b4d0 msgr2=0x7f781406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f781406b4d0 0x7f781406d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f78280b7af0 tx=0x7f781c009870 comp rx=0 tx=0).stop 2024-03-06T07:30:09.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 msgr2=0x7f782815cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 0x7f782815cd80 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f78200097c0 tx=0x7f78200046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 shutdown_connections 2024-03-06T07:30:09.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f781406b4d0 0x7f781406d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f78280d3400 0x7f78280ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78280b2b50 0x7f78281f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 --2- 172.21.15.27:0/1330698041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828071540 0x7f782815cd80 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 >> 172.21.15.27:0/1330698041 conn(0x7f78280d3ca0 msgr2=0x7f782815f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 shutdown_connections 2024-03-06T07:30:09.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.301+0000 7f782d41c640 1 -- 172.21.15.27:0/1330698041 wait complete. 2024-03-06T07:30:09.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:30:09.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.432+0000 7f4e5580e640 1 Processor -- start 2024-03-06T07:30:09.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.432+0000 7f4e5580e640 1 -- start start 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.433+0000 7f4e5580e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4e5015dd90 0x7f4e500ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.433+0000 7f4e5580e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500d3740 con 0x7f4e500d29d0 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.433+0000 7f4e5580e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500d3120 con 0x7f4e500b6d10 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.433+0000 7f4e5580e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e500d3400 con 0x7f4e5015dd90 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.433+0000 7f4e4e7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4e500d29d0 0x7f4e500d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43262/0 (socket says 172.21.15.27:43262) 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.433+0000 7f4e4e7fc640 1 -- 172.21.15.27:0/3051254383 learned_addr learned my addr 172.21.15.27:0/3051254383 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.433+0000 7f4e4effd640 1 --2- 172.21.15.27:0/3051254383 >> v2:172.21.15.69:3300/0 conn(0x7f4e5015dd90 0x7f4e500ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295275215 0 0) 0x7f4e500d3740 con 0x7f4e500d29d0 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e3c003520 con 0x7f4e500d29d0 2024-03-06T07:30:09.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339928735 0 0) 0x7f4e500d3120 con 0x7f4e500b6d10 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e500d3740 con 0x7f4e500b6d10 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4108458638 0 0) 0x7f4e3c003520 con 0x7f4e500d29d0 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 >> v2:172.21.15.69:3300/0 conn(0x7f4e5015dd90 msgr2=0x7f4e500ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 --2- 172.21.15.27:0/3051254383 >> v2:172.21.15.69:3300/0 conn(0x7f4e5015dd90 0x7f4e500ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 >> v1:172.21.15.69:6789/0 conn(0x7f4e500b6d10 legacy=0x7f4e500d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e500d39e0 con 0x7f4e500d29d0 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f4e38001670 con 0x7f4e500d29d0 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 >> v1:172.21.15.27:6789/0 conn(0x7f4e500d29d0 legacy=0x7f4e500d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e3c003b90 0x7f4e3c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e3c004800 0x7f4e3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c0051c0 0x7f4e3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500d3400 con 0x7f4e3c003b90 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c00bfe0 con 0x7f4e3c004800 2024-03-06T07:30:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c00c1e0 con 0x7f4e3c0051c0 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4effd640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e3c003b90 0x7f4e3c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4e7fc640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e3c004800 0x7f4e3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4f7fe640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c0051c0 0x7f4e3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4effd640 1 -- 172.21.15.27:0/3051254383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c0051c0 msgr2=0x7f4e3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4effd640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c0051c0 0x7f4e3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4effd640 1 -- 172.21.15.27:0/3051254383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e3c004800 msgr2=0x7f4e3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4effd640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e3c004800 0x7f4e3c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4effd640 1 -- 172.21.15.27:0/3051254383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c003520 con 0x7f4e3c003b90 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4f7fe640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c0051c0 0x7f4e3c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.434+0000 7f4e4f7fe640 1 -- 172.21.15.27:0/3051254383 reap_dead start 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.435+0000 7f4e4effd640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e3c003b90 0x7f4e3c003f90 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f4e48009ac0 tx=0x7f4e3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d622e2d7c02d8694 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.435+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48039070 con 0x7f4e3c003b90 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.435+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4802c6b0 con 0x7f4e3c003b90 2024-03-06T07:30:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.435+0000 7f4e4dffb640 1 -- 172.21.15.27:0/3051254383 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48009690 con 0x7f4e3c003b90 2024-03-06T07:30:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.436+0000 7f4e5580e640 1 -- 172.21.15.27:0/3051254383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e3c003b90 msgr2=0x7f4e3c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.436+0000 7f4e5580e640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e3c003b90 0x7f4e3c003f90 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f4e48009ac0 tx=0x7f4e3c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.436+0000 7f4e5580e640 1 -- 172.21.15.27:0/3051254383 shutdown_connections 2024-03-06T07:30:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.436+0000 7f4e5580e640 1 --2- 172.21.15.27:0/3051254383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e3c003b90 0x7f4e3c003f90 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.436+0000 7f4e5580e640 1 -- 172.21.15.27:0/3051254383 >> 172.21.15.27:0/3051254383 conn(0x7f4e50071960 msgr2=0x7f4e50071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.436+0000 7f4e5580e640 1 -- 172.21.15.27:0/3051254383 shutdown_connections 2024-03-06T07:30:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.436+0000 7f4e5580e640 1 -- 172.21.15.27:0/3051254383 wait complete. 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.437+0000 7f4e5580e640 1 Processor -- start 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.437+0000 7f4e5580e640 1 -- start start 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e5580e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c003b90 0x7f4e500cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e5580e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e500c8800 0x7f4e500c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e5580e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e501473d0 0x7f4e50147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e5580e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500d2df0 con 0x7f4e500c8800 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e5580e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5015e170 con 0x7f4e501473d0 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e5580e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500ceea0 con 0x7f4e3c003b90 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e500c8800 0x7f4e500c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e500c8800 0x7f4e500c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57708/0 (socket says 172.21.15.27:57708) 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4e7fc640 1 -- 172.21.15.27:0/3784680165 learned_addr learned my addr 172.21.15.27:0/3784680165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4effd640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c003b90 0x7f4e500cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4f7fe640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e501473d0 0x7f4e50147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4e7fc640 1 -- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c003b90 msgr2=0x7f4e500cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4e7fc640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c003b90 0x7f4e500cafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4e7fc640 1 -- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e501473d0 msgr2=0x7f4e50147850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4e7fc640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e501473d0 0x7f4e50147850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4e7fc640 1 -- 172.21.15.27:0/3784680165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e500ca360 con 0x7f4e500c8800 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.438+0000 7f4e4f7fe640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e501473d0 0x7f4e50147850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.439+0000 7f4e4e7fc640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e500c8800 0x7f4e500c8c80 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f4e38004ea0 tx=0x7f4e38004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.439+0000 7f4e47fff640 1 -- 172.21.15.27:0/3784680165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3800f620 con 0x7f4e500c8800 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.439+0000 7f4e47fff640 1 -- 172.21.15.27:0/3784680165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e38018070 con 0x7f4e500c8800 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.439+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e500c92c0 con 0x7f4e500c8800 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.439+0000 7f4e47fff640 1 -- 172.21.15.27:0/3784680165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3800fa20 con 0x7f4e500c8800 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.439+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e500c9880 con 0x7f4e500c8800 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.440+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e50072a30 con 0x7f4e500c8800 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.440+0000 7f4e47fff640 1 -- 172.21.15.27:0/3784680165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e3800fbc0 con 0x7f4e500c8800 2024-03-06T07:30:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.441+0000 7f4e47fff640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e3006b500 0x7f4e3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.441+0000 7f4e47fff640 1 -- 172.21.15.27:0/3784680165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4e38089d50 con 0x7f4e500c8800 2024-03-06T07:30:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.441+0000 7f4e4effd640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e3006b500 0x7f4e3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.442+0000 7f4e4effd640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e3006b500 0x7f4e3006d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4e48036000 tx=0x7f4e48030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.446+0000 7f4e47fff640 1 -- 172.21.15.27:0/3784680165 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e3808d030 con 0x7f4e500c8800 2024-03-06T07:30:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.628+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4e500d4a60 con 0x7f4e500c8800 2024-03-06T07:30:09.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.633+0000 7f4e47fff640 1 -- 172.21.15.27:0/3784680165 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4e38058e20 con 0x7f4e500c8800 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e3006b500 msgr2=0x7f4e3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e3006b500 0x7f4e3006d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4e48036000 tx=0x7f4e48030000 comp rx=0 tx=0).stop 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e500c8800 msgr2=0x7f4e500c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e500c8800 0x7f4e500c8c80 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f4e38004ea0 tx=0x7f4e38004f90 comp rx=0 tx=0).stop 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 shutdown_connections 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4e3006b500 0x7f4e3006d9c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e501473d0 0x7f4e50147850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e500c8800 0x7f4e500c8c80 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 --2- 172.21.15.27:0/3784680165 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4e3c003b90 0x7f4e500cafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 >> 172.21.15.27:0/3784680165 conn(0x7f4e50071960 msgr2=0x7f4e50161fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 shutdown_connections 2024-03-06T07:30:09.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.635+0000 7f4e5580e640 1 -- 172.21.15.27:0/3784680165 wait complete. 2024-03-06T07:30:09.649 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:09.649 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:09.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:09.744 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:09.744 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:30:09.744 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:09.760 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:09.760 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:30:09.760 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:09.778 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:09.778 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:30:09.778 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:09.815 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:09.815 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:30:09.815 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:09.836 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:09.836 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:30:09.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.960+0000 7fa051eeb640 1 Processor -- start 2024-03-06T07:30:09.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.960+0000 7fa051eeb640 1 -- start start 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa051eeb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa04c1451a0 0x7fa04c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa051eeb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa04c1456f0 0x7fa04c0b2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa051eeb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c0b3c70 con 0x7fa04c15bde0 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa051eeb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04c0b4840 con 0x7fa04c1451a0 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa051eeb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04c0b53e0 con 0x7fa04c1456f0 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa04affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa04c1451a0 0x7fa04c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa04b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa04c15bde0 0x7fa04c0b4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43278/0 (socket says 172.21.15.27:43278) 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa04b7fe640 1 -- 172.21.15.27:0/3243721913 learned_addr learned my addr 172.21.15.27:0/3243721913 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa04bfff640 1 --2- 172.21.15.27:0/3243721913 >> v2:172.21.15.69:3300/0 conn(0x7fa04c1456f0 0x7fa04c0b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa04affd640 1 -- 172.21.15.27:0/3243721913 >> v2:172.21.15.69:3300/0 conn(0x7fa04c1456f0 msgr2=0x7fa04c0b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa04affd640 1 --2- 172.21.15.27:0/3243721913 >> v2:172.21.15.69:3300/0 conn(0x7fa04c1456f0 0x7fa04c0b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa04affd640 1 -- 172.21.15.27:0/3243721913 >> v1:172.21.15.27:6789/0 conn(0x7fa04c15bde0 legacy=0x7fa04c0b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.961+0000 7fa04affd640 1 -- 172.21.15.27:0/3243721913 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c0afad0 con 0x7fa04c1451a0 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.962+0000 7fa04affd640 1 --2- 172.21.15.27:0/3243721913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1451a0 0x7fa04c0b1d50 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0b49c0 tx=0x7fa04002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3013cac406a421c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.962+0000 7fa04a7fc640 1 -- 172.21.15.27:0/3243721913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040035020 con 0x7fa04c1451a0 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.962+0000 7fa04a7fc640 1 -- 172.21.15.27:0/3243721913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa040031830 con 0x7fa04c1451a0 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.962+0000 7fa04a7fc640 1 -- 172.21.15.27:0/3243721913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040031c00 con 0x7fa04c1451a0 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.963+0000 7fa051eeb640 1 -- 172.21.15.27:0/3243721913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1451a0 msgr2=0x7fa04c0b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.963+0000 7fa051eeb640 1 --2- 172.21.15.27:0/3243721913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1451a0 0x7fa04c0b1d50 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0b49c0 tx=0x7fa04002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.963+0000 7fa051eeb640 1 -- 172.21.15.27:0/3243721913 shutdown_connections 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.963+0000 7fa051eeb640 1 --2- 172.21.15.27:0/3243721913 >> v2:172.21.15.69:3300/0 conn(0x7fa04c1456f0 0x7fa04c0b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.963+0000 7fa051eeb640 1 --2- 172.21.15.27:0/3243721913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1451a0 0x7fa04c0b1d50 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.963+0000 7fa051eeb640 1 -- 172.21.15.27:0/3243721913 >> 172.21.15.27:0/3243721913 conn(0x7fa04c0b55e0 msgr2=0x7fa04c0b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.963+0000 7fa051eeb640 1 -- 172.21.15.27:0/3243721913 shutdown_connections 2024-03-06T07:30:09.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.963+0000 7fa051eeb640 1 -- 172.21.15.27:0/3243721913 wait complete. 2024-03-06T07:30:09.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.964+0000 7fa051eeb640 1 Processor -- start 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.964+0000 7fa051eeb640 1 -- start start 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa051eeb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa04c1451a0 0x7fa04c1f1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa051eeb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1456f0 0x7fa04c1f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa051eeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa04c15bde0 0x7fa04c154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa051eeb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0b44d0 con 0x7fa04c1456f0 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa051eeb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0b41f0 con 0x7fa04c1451a0 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa051eeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c15c1c0 con 0x7fa04c15bde0 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1456f0 0x7fa04c1f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1456f0 0x7fa04c1f1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57714/0 (socket says 172.21.15.27:57714) 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04affd640 1 -- 172.21.15.27:0/3591116428 learned_addr learned my addr 172.21.15.27:0/3591116428 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04b7fe640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa04c1451a0 0x7fa04c1f1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04bfff640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa04c15bde0 0x7fa04c154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04affd640 1 -- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa04c15bde0 msgr2=0x7fa04c154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04affd640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa04c15bde0 0x7fa04c154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04affd640 1 -- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa04c1451a0 msgr2=0x7fa04c1f1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04affd640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa04c1451a0 0x7fa04c1f1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04affd640 1 -- 172.21.15.27:0/3591116428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c1f22b0 con 0x7fa04c1456f0 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04bfff640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa04c15bde0 0x7fa04c154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04b7fe640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa04c1451a0 0x7fa04c1f1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.965+0000 7fa04affd640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1456f0 0x7fa04c1f1c90 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fa040009890 tx=0x7fa0400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.966+0000 7fa033fff640 1 -- 172.21.15.27:0/3591116428 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040041070 con 0x7fa04c1456f0 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.966+0000 7fa033fff640 1 -- 172.21.15.27:0/3591116428 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04003aa30 con 0x7fa04c1456f0 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.966+0000 7fa033fff640 1 -- 172.21.15.27:0/3591116428 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040035040 con 0x7fa04c1456f0 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.966+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c1f2590 con 0x7fa04c1456f0 2024-03-06T07:30:09.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.966+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c1f3240 con 0x7fa04c1456f0 2024-03-06T07:30:09.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.967+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c067ff0 con 0x7fa04c1456f0 2024-03-06T07:30:09.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.967+0000 7fa033fff640 1 -- 172.21.15.27:0/3591116428 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa040038020 con 0x7fa04c1456f0 2024-03-06T07:30:09.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.967+0000 7fa033fff640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.968+0000 7fa033fff640 1 -- 172.21.15.27:0/3591116428 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fa0400aba40 con 0x7fa04c1456f0 2024-03-06T07:30:09.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.968+0000 7fa04b7fe640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.969+0000 7fa04b7fe640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa03806b4d0 0x7fa03806d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa03c000f60 tx=0x7fa03c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:09.972+0000 7fa033fff640 1 -- 172.21.15.27:0/3591116428 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa04007ab10 con 0x7fa04c1456f0 2024-03-06T07:30:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.196+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa04c0b4cd0 con 0x7fa04c1456f0 2024-03-06T07:30:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.197+0000 7fa033fff640 1 -- 172.21.15.27:0/3591116428 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fa0400abe20 con 0x7fa04c1456f0 2024-03-06T07:30:10.199 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:10.200 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":85,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1937860664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1937860664},{"type":"v1","addr":"172.21.15.69:6835","nonce":1937860664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4018969021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4018969021},{"type":"v1","addr":"172.21.15.69:6837","nonce":4018969021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":8265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3793648808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3793648808},{"type":"v1","addr":"172.21.15.27:6835","nonce":3793648808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":8589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/1940832143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1940832143},{"type":"v1","addr":"172.21.15.27:6833","nonce":1940832143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[]} 2024-03-06T07:30:10.200 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 85 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa03806b4d0 msgr2=0x7fa03806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa03806b4d0 0x7fa03806d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa03c000f60 tx=0x7fa03c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1456f0 msgr2=0x7fa04c1f1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1456f0 0x7fa04c1f1c90 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fa040009890 tx=0x7fa0400046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 shutdown_connections 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa04c15bde0 0x7fa04c154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa04c1456f0 0x7fa04c1f1c90 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 --2- 172.21.15.27:0/3591116428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa04c1451a0 0x7fa04c1f1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 >> 172.21.15.27:0/3591116428 conn(0x7fa04c0b55e0 msgr2=0x7fa04c0ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 shutdown_connections 2024-03-06T07:30:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:10.199+0000 7fa051eeb640 1 -- 172.21.15.27:0/3591116428 wait complete. 2024-03-06T07:30:10.213 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:30:10.213 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:30:10.213 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:30:10.213 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:10.213 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:30:10.213 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:30:10.214 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:30:10.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:10.214 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:30:10.214 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:30:10.214 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:10.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:10.214 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:30:10.215 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:30:10.215 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:10.215 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:10.222 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:30:10.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:30:10.223 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:30:10.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:30:10.227 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:10.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:30:10.265 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:30:10.266 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:10.266 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:30:10.266 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:10.270 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:30:10.270 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:10.347 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:30:10.347 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:30:10.353 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:30:11.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-06T07:30:16.227 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:16.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:30:16.231 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:30:16.231 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:16.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:30:16.315 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:30:16.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.347+0000 7f383a688640 1 Processor -- start 2024-03-06T07:30:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.347+0000 7f383a688640 1 -- start start 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f383a688640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f383406ed10 0x7f383406af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f383a688640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38340cd890 0x7f383415d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f383a688640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38340cc0e0 con 0x7f38340b1020 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f383a688640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38340d28b0 con 0x7f383406ed10 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f383a688640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38340cf320 con 0x7f38340cd890 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f38337fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f383406ed10 0x7f383406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f38337fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f383406ed10 0x7f383406af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55442/0 (socket says 172.21.15.27:55442) 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f38337fe640 1 -- 172.21.15.27:0/134898255 learned_addr learned my addr 172.21.15.27:0/134898255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f3838c25640 1 --2- 172.21.15.27:0/134898255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38340cd890 0x7f383415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f38337fe640 1 -- 172.21.15.27:0/134898255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38340cd890 msgr2=0x7f383415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f38337fe640 1 --2- 172.21.15.27:0/134898255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38340cd890 0x7f383415d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f38337fe640 1 -- 172.21.15.27:0/134898255 >> v1:172.21.15.69:6789/0 conn(0x7f38340b1020 legacy=0x7f38340cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.348+0000 7f38337fe640 1 -- 172.21.15.27:0/134898255 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38340cc3c0 con 0x7f383406ed10 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.349+0000 7f38337fe640 1 --2- 172.21.15.27:0/134898255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383406ed10 0x7f383406af20 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f3824009870 tx=0x7f382402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2711e124f161af12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.349+0000 7f3832ffd640 1 -- 172.21.15.27:0/134898255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824034070 con 0x7f383406ed10 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.349+0000 7f3832ffd640 1 -- 172.21.15.27:0/134898255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3824030840 con 0x7f383406ed10 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.349+0000 7f3832ffd640 1 -- 172.21.15.27:0/134898255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824030c10 con 0x7f383406ed10 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.350+0000 7f383a688640 1 -- 172.21.15.27:0/134898255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383406ed10 msgr2=0x7f383406af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.350+0000 7f383a688640 1 --2- 172.21.15.27:0/134898255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383406ed10 0x7f383406af20 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f3824009870 tx=0x7f382402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.350+0000 7f383a688640 1 -- 172.21.15.27:0/134898255 shutdown_connections 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.350+0000 7f383a688640 1 --2- 172.21.15.27:0/134898255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383406ed10 0x7f383406af20 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.350+0000 7f383a688640 1 --2- 172.21.15.27:0/134898255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f38340cd890 0x7f383415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.350+0000 7f383a688640 1 -- 172.21.15.27:0/134898255 >> 172.21.15.27:0/134898255 conn(0x7f38340d1a90 msgr2=0x7f38340d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.350+0000 7f383a688640 1 -- 172.21.15.27:0/134898255 shutdown_connections 2024-03-06T07:30:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.350+0000 7f383a688640 1 -- 172.21.15.27:0/134898255 wait complete. 2024-03-06T07:30:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.351+0000 7f383a688640 1 Processor -- start 2024-03-06T07:30:16.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f383a688640 1 -- start start 2024-03-06T07:30:16.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f383a688640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383406ed10 0x7f3834151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f383a688640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38340b1020 0x7f38340c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f383a688640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38340cd890 0x7f38340b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f383a688640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38340ccc60 con 0x7f38340cd890 2024-03-06T07:30:16.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f383a688640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38340cc980 con 0x7f38340b1020 2024-03-06T07:30:16.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f383a688640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38340cdca0 con 0x7f383406ed10 2024-03-06T07:30:16.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f3838c25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38340cd890 0x7f38340b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f3838c25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38340cd890 0x7f38340b3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55446/0 (socket says 172.21.15.27:55446) 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.352+0000 7f3838c25640 1 -- 172.21.15.27:0/3982182540 learned_addr learned my addr 172.21.15.27:0/3982182540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f38337fe640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38340b1020 0x7f38340c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3833fff640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383406ed10 0x7f3834151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3838c25640 1 -- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383406ed10 msgr2=0x7f3834151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3838c25640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383406ed10 0x7f3834151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3838c25640 1 -- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38340b1020 msgr2=0x7f38340c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3838c25640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38340b1020 0x7f38340c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3838c25640 1 -- 172.21.15.27:0/3982182540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38340b15a0 con 0x7f38340cd890 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f38337fe640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38340b1020 0x7f38340c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3833fff640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383406ed10 0x7f3834151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3838c25640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38340cd890 0x7f38340b3930 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f382c00efc0 tx=0x7f382c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3830ff9640 1 -- 172.21.15.27:0/3982182540 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c018020 con 0x7f38340cd890 2024-03-06T07:30:16.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3830ff9640 1 -- 172.21.15.27:0/3982182540 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c014480 con 0x7f38340cd890 2024-03-06T07:30:16.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38340b1880 con 0x7f38340cd890 2024-03-06T07:30:16.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f3830ff9640 1 -- 172.21.15.27:0/3982182540 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c014870 con 0x7f38340cd890 2024-03-06T07:30:16.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.353+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38340b1e40 con 0x7f38340cd890 2024-03-06T07:30:16.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.354+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3834067ff0 con 0x7f38340cd890 2024-03-06T07:30:16.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.355+0000 7f3830ff9640 1 -- 172.21.15.27:0/3982182540 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f382c005d10 con 0x7f38340cd890 2024-03-06T07:30:16.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.355+0000 7f3830ff9640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f382006b4d0 0x7f382006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.355+0000 7f3830ff9640 1 -- 172.21.15.27:0/3982182540 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f382c08aa50 con 0x7f38340cd890 2024-03-06T07:30:16.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.356+0000 7f3833fff640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f382006b4d0 0x7f382006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.356+0000 7f3833fff640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f382006b4d0 0x7f382006d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3828001680 tx=0x7f3828009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.360+0000 7f3830ff9640 1 -- 172.21.15.27:0/3982182540 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f382c022070 con 0x7f38340cd890 2024-03-06T07:30:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:16.554+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3834161160 con 0x7f38340cd890 2024-03-06T07:30:17.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-06T07:30:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.324+0000 7f3830ff9640 1 -- 172.21.15.27:0/3982182540 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v128) v1 ==== 97+0+0 (secure 0 0 0) 0x7f382c010040 con 0x7f38340cd890 2024-03-06T07:30:18.327 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-03-06T07:30:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f382006b4d0 msgr2=0x7f382006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f382006b4d0 0x7f382006d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3828001680 tx=0x7f3828009040 comp rx=0 tx=0).stop 2024-03-06T07:30:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38340cd890 msgr2=0x7f38340b3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38340cd890 0x7f38340b3930 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f382c00efc0 tx=0x7f382c00c490 comp rx=0 tx=0).stop 2024-03-06T07:30:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 shutdown_connections 2024-03-06T07:30:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f382006b4d0 0x7f382006d990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38340cd890 0x7f38340b3930 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38340b1020 0x7f38340c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 --2- 172.21.15.27:0/3982182540 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f383406ed10 0x7f3834151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.327+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 >> 172.21.15.27:0/3982182540 conn(0x7f38340d1a90 msgr2=0x7f38340d1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.328+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 shutdown_connections 2024-03-06T07:30:18.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.328+0000 7f383a688640 1 -- 172.21.15.27:0/3982182540 wait complete. 2024-03-06T07:30:18.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:30:18.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.449+0000 7f0e7ef8c640 1 Processor -- start 2024-03-06T07:30:18.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.449+0000 7f0e7ef8c640 1 -- start start 2024-03-06T07:30:18.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.450+0000 7f0e7ef8c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e78147930 0x7f0e78072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.450+0000 7f0e7ef8c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e780adfc0 0x7f0e78161f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.450+0000 7f0e7ef8c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e78072a70 con 0x7f0e78161ac0 2024-03-06T07:30:18.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.450+0000 7f0e7ef8c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e78162460 con 0x7f0e78147930 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.450+0000 7f0e7ef8c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e780ae450 con 0x7f0e780adfc0 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.451+0000 7f0e7cd28640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e78147930 0x7f0e78072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.451+0000 7f0e7d529640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e780adfc0 0x7f0e78161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.451+0000 7f0e77fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0e78161ac0 0x7f0e780ad8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:49108/0 (socket says 172.21.15.27:49108) 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.451+0000 7f0e77fff640 1 -- 172.21.15.27:0/1993044553 learned_addr learned my addr 172.21.15.27:0/1993044553 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.451+0000 7f0e7cd28640 1 -- 172.21.15.27:0/1993044553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e780adfc0 msgr2=0x7f0e78161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.451+0000 7f0e7cd28640 1 --2- 172.21.15.27:0/1993044553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e780adfc0 0x7f0e78161f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.451+0000 7f0e7cd28640 1 -- 172.21.15.27:0/1993044553 >> v1:172.21.15.69:6789/0 conn(0x7f0e78161ac0 legacy=0x7f0e780ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.451+0000 7f0e7cd28640 1 -- 172.21.15.27:0/1993044553 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7806dbe0 con 0x7f0e78147930 2024-03-06T07:30:18.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.451+0000 7f0e777fe640 1 -- 172.21.15.27:0/1993044553 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449800120 0 0) 0x7f0e78072a70 con 0x7f0e78161ac0 2024-03-06T07:30:18.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.452+0000 7f0e7cd28640 1 --2- 172.21.15.27:0/1993044553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e78147930 0x7f0e78072530 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0097a0 tx=0x7f0e6c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ea96570ebe17cf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.452+0000 7f0e777fe640 1 -- 172.21.15.27:0/1993044553 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c035020 con 0x7f0e78147930 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.452+0000 7f0e777fe640 1 -- 172.21.15.27:0/1993044553 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6c031830 con 0x7f0e78147930 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.452+0000 7f0e777fe640 1 -- 172.21.15.27:0/1993044553 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c031c00 con 0x7f0e78147930 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.453+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/1993044553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e78147930 msgr2=0x7f0e78072530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.453+0000 7f0e7ef8c640 1 --2- 172.21.15.27:0/1993044553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e78147930 0x7f0e78072530 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0097a0 tx=0x7f0e6c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.453+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/1993044553 shutdown_connections 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.453+0000 7f0e7ef8c640 1 --2- 172.21.15.27:0/1993044553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e780adfc0 0x7f0e78161f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.453+0000 7f0e7ef8c640 1 --2- 172.21.15.27:0/1993044553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e78147930 0x7f0e78072530 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.453+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/1993044553 >> 172.21.15.27:0/1993044553 conn(0x7f0e7806cd20 msgr2=0x7f0e7806d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.453+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/1993044553 shutdown_connections 2024-03-06T07:30:18.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.453+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/1993044553 wait complete. 2024-03-06T07:30:18.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.454+0000 7f0e7ef8c640 1 Processor -- start 2024-03-06T07:30:18.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.454+0000 7f0e7ef8c640 1 -- start start 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7ef8c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e780adfc0 0x7f0e780b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7ef8c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e78147930 0x7f0e7813d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7ef8c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e78161ac0 0x7f0e7813d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7ef8c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e78072010 con 0x7f0e780adfc0 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7ef8c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e78071d30 con 0x7f0e78147930 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7ef8c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e78147d10 con 0x7f0e78161ac0 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7cd28640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e780adfc0 0x7f0e780b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7cd28640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e780adfc0 0x7f0e780b0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55460/0 (socket says 172.21.15.27:55460) 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7cd28640 1 -- 172.21.15.27:0/86213894 learned_addr learned my addr 172.21.15.27:0/86213894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e77fff640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e78147930 0x7f0e7813d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7d529640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e78161ac0 0x7f0e7813d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7cd28640 1 -- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e78161ac0 msgr2=0x7f0e7813d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7cd28640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e78161ac0 0x7f0e7813d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7cd28640 1 -- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e78147930 msgr2=0x7f0e7813d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7cd28640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e78147930 0x7f0e7813d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7cd28640 1 -- 172.21.15.27:0/86213894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e781508a0 con 0x7f0e780adfc0 2024-03-06T07:30:18.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.455+0000 7f0e7cd28640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e780adfc0 0x7f0e780b0650 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009890 tx=0x7f0e6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.456+0000 7f0e757fa640 1 -- 172.21.15.27:0/86213894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c035040 con 0x7f0e780adfc0 2024-03-06T07:30:18.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.456+0000 7f0e757fa640 1 -- 172.21.15.27:0/86213894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6c03b920 con 0x7f0e780adfc0 2024-03-06T07:30:18.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.456+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e78150b80 con 0x7f0e780adfc0 2024-03-06T07:30:18.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.456+0000 7f0e757fa640 1 -- 172.21.15.27:0/86213894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c03bd10 con 0x7f0e780adfc0 2024-03-06T07:30:18.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.456+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e78151140 con 0x7f0e780adfc0 2024-03-06T07:30:18.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.457+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e78067ff0 con 0x7f0e780adfc0 2024-03-06T07:30:18.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.457+0000 7f0e757fa640 1 -- 172.21.15.27:0/86213894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e6c038020 con 0x7f0e780adfc0 2024-03-06T07:30:18.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.458+0000 7f0e757fa640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.458+0000 7f0e757fa640 1 -- 172.21.15.27:0/86213894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0e6c0abef0 con 0x7f0e780adfc0 2024-03-06T07:30:18.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.458+0000 7f0e77fff640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.459+0000 7f0e77fff640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0e780cf840 tx=0x7f0e64007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.462+0000 7f0e757fa640 1 -- 172.21.15.27:0/86213894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e6c031800 con 0x7f0e780adfc0 2024-03-06T07:30:18.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.713+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0e780cc5d0 con 0x7f0e780adfc0 2024-03-06T07:30:18.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.713+0000 7f0e757fa640 1 -- 172.21.15.27:0/86213894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0e6c07aa80 con 0x7f0e780adfc0 2024-03-06T07:30:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:18.716 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:30:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-03-06T07:30:18.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g==", 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ==", 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:30:18.717 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg==", 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.718 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ==", 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw==", 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:30:18.719 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA==", 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA==", 2024-03-06T07:30:18.720 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg==", 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A==", 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw==", 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:18.722 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA==", 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.723 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A==", 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA==", 2024-03-06T07:30:18.724 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ==", 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:30:18.725 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ==", 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg==", 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlkc2iEhAA6BuYba0FzXeVjGPOP8sgyQ==", 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.727 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw==", 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:18.728 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg==", 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:30:18.729 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:30:18.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e5c06b4d0 msgr2=0x7f0e5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0e780cf840 tx=0x7f0e64007d50 comp rx=0 tx=0).stop 2024-03-06T07:30:18.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e780adfc0 msgr2=0x7f0e780b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e780adfc0 0x7f0e780b0650 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009890 tx=0x7f0e6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 shutdown_connections 2024-03-06T07:30:18.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e78161ac0 0x7f0e7813d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e78147930 0x7f0e7813d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 --2- 172.21.15.27:0/86213894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e780adfc0 0x7f0e780b0650 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 >> 172.21.15.27:0/86213894 conn(0x7f0e7806cd20 msgr2=0x7f0e780b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 shutdown_connections 2024-03-06T07:30:18.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.716+0000 7f0e7ef8c640 1 -- 172.21.15.27:0/86213894 wait complete. 2024-03-06T07:30:18.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:18.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.838+0000 7f19f813e640 1 Processor -- start 2024-03-06T07:30:18.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.838+0000 7f19f813e640 1 -- start start 2024-03-06T07:30:18.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f813e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19f0158520 0x7f19f006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f813e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19f006dd60 0x7f19f00ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f813e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f0071240 con 0x7f19f00aec40 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f813e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f0071b70 con 0x7f19f006dd60 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f813e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f00af0b0 con 0x7f19f0158520 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f56d9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19f006dd60 0x7f19f00ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f66db640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19f00aec40 0x7f19f015e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50924/0 (socket says 172.21.15.27:50924) 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f56d9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19f006dd60 0x7f19f00ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55466/0 (socket says 172.21.15.27:55466) 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f66db640 1 -- 172.21.15.27:0/3287860183 learned_addr learned my addr 172.21.15.27:0/3287860183 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f5eda640 1 --2- 172.21.15.27:0/3287860183 >> v2:172.21.15.69:3300/0 conn(0x7f19f0158520 0x7f19f006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f56d9640 1 -- 172.21.15.27:0/3287860183 >> v2:172.21.15.69:3300/0 conn(0x7f19f0158520 msgr2=0x7f19f006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f56d9640 1 --2- 172.21.15.27:0/3287860183 >> v2:172.21.15.69:3300/0 conn(0x7f19f0158520 0x7f19f006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f56d9640 1 -- 172.21.15.27:0/3287860183 >> v1:172.21.15.27:6789/0 conn(0x7f19f00aec40 legacy=0x7f19f015e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f4ed8640 1 -- 172.21.15.27:0/3287860183 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540289453 0 0) 0x7f19f0071240 con 0x7f19f00aec40 2024-03-06T07:30:18.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.839+0000 7f19f56d9640 1 -- 172.21.15.27:0/3287860183 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f0071520 con 0x7f19f006dd60 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.840+0000 7f19f56d9640 1 --2- 172.21.15.27:0/3287860183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 0x7f19f00ae700 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f19f006e900 tx=0x7f19e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb87bd872b6fba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.840+0000 7f19f4ed8640 1 -- 172.21.15.27:0/3287860183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0035020 con 0x7f19f006dd60 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.840+0000 7f19f4ed8640 1 -- 172.21.15.27:0/3287860183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0031830 con 0x7f19f006dd60 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.840+0000 7f19f4ed8640 1 -- 172.21.15.27:0/3287860183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0031c00 con 0x7f19f006dd60 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.841+0000 7f19f813e640 1 -- 172.21.15.27:0/3287860183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 msgr2=0x7f19f00ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.841+0000 7f19f813e640 1 --2- 172.21.15.27:0/3287860183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 0x7f19f00ae700 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f19f006e900 tx=0x7f19e002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.841+0000 7f19f813e640 1 -- 172.21.15.27:0/3287860183 shutdown_connections 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.841+0000 7f19f813e640 1 --2- 172.21.15.27:0/3287860183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 0x7f19f00ae700 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.841+0000 7f19f813e640 1 --2- 172.21.15.27:0/3287860183 >> v2:172.21.15.69:3300/0 conn(0x7f19f0158520 0x7f19f006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.841+0000 7f19f813e640 1 -- 172.21.15.27:0/3287860183 >> 172.21.15.27:0/3287860183 conn(0x7f19f0066d00 msgr2=0x7f19f00670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.841+0000 7f19f813e640 1 -- 172.21.15.27:0/3287860183 shutdown_connections 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.841+0000 7f19f813e640 1 -- 172.21.15.27:0/3287860183 wait complete. 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.842+0000 7f19f813e640 1 Processor -- start 2024-03-06T07:30:18.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.842+0000 7f19f813e640 1 -- start start 2024-03-06T07:30:18.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f813e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 0x7f19f01f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f813e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f00aec40 0x7f19f01ed8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f813e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f0158520 0x7f19f0159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f813e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0072000 con 0x7f19f006dd60 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f813e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f00b4650 con 0x7f19f00aec40 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f813e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0158900 con 0x7f19f0158520 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f5eda640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 0x7f19f01f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f5eda640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 0x7f19f01f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55472/0 (socket says 172.21.15.27:55472) 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f5eda640 1 -- 172.21.15.27:0/1280495899 learned_addr learned my addr 172.21.15.27:0/1280495899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f56d9640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f00aec40 0x7f19f01ed8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f66db640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f0158520 0x7f19f0159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f5eda640 1 -- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f0158520 msgr2=0x7f19f0159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f5eda640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f0158520 0x7f19f0159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f5eda640 1 -- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f00aec40 msgr2=0x7f19f01ed8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f5eda640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f00aec40 0x7f19f01ed8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f5eda640 1 -- 172.21.15.27:0/1280495899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f01e35b0 con 0x7f19f006dd60 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f56d9640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f00aec40 0x7f19f01ed8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.843+0000 7f19f5eda640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 0x7f19f01f0fe0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00d620 tx=0x7f19ec00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.844+0000 7f19e67fc640 1 -- 172.21.15.27:0/1280495899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0125f0 con 0x7f19f006dd60 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.844+0000 7f19e67fc640 1 -- 172.21.15.27:0/1280495899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec019070 con 0x7f19f006dd60 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.844+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f01430a0 con 0x7f19f006dd60 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.844+0000 7f19e67fc640 1 -- 172.21.15.27:0/1280495899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0129d0 con 0x7f19f006dd60 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.844+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f0143660 con 0x7f19f006dd60 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.845+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f0067ff0 con 0x7f19f006dd60 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.845+0000 7f19e67fc640 1 -- 172.21.15.27:0/1280495899 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19ec012b70 con 0x7f19f006dd60 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.846+0000 7f19e67fc640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19d406b4d0 0x7f19d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.846+0000 7f19e67fc640 1 -- 172.21.15.27:0/1280495899 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f19ec08a270 con 0x7f19f006dd60 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.846+0000 7f19f56d9640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19d406b4d0 0x7f19d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.846+0000 7f19f56d9640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19d406b4d0 0x7f19d406d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:18.850+0000 7f19e67fc640 1 -- 172.21.15.27:0/1280495899 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19ec059440 con 0x7f19f006dd60 2024-03-06T07:30:19.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.037+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19f00b31f0 con 0x7f19f006dd60 2024-03-06T07:30:19.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.037+0000 7f19e67fc640 1 -- 172.21.15.27:0/1280495899 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+12829 (secure 0 0 0) 0x7f19ec058d40 con 0x7f19f006dd60 2024-03-06T07:30:19.040 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:19.040 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":128,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:30:18.321487+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:19.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.039+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19d406b4d0 msgr2=0x7f19d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.039+0000 7f19f813e640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19d406b4d0 0x7f19d406d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e0008030 comp rx=0 tx=0).stop 2024-03-06T07:30:19.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.039+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 msgr2=0x7f19f01f0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.040+0000 7f19f813e640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 0x7f19f01f0fe0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00d620 tx=0x7f19ec00dae0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.040+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 shutdown_connections 2024-03-06T07:30:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.040+0000 7f19f813e640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f19d406b4d0 0x7f19d406d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.040+0000 7f19f813e640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19f0158520 0x7f19f0159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.040+0000 7f19f813e640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f00aec40 0x7f19f01ed8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.040+0000 7f19f813e640 1 --2- 172.21.15.27:0/1280495899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19f006dd60 0x7f19f01f0fe0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.040+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 >> 172.21.15.27:0/1280495899 conn(0x7f19f0066d00 msgr2=0x7f19f00670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.040+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 shutdown_connections 2024-03-06T07:30:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.040+0000 7f19f813e640 1 -- 172.21.15.27:0/1280495899 wait complete. 2024-03-06T07:30:19.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:19.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.161+0000 7f074b5e9640 1 Processor -- start 2024-03-06T07:30:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.161+0000 7f074b5e9640 1 -- start start 2024-03-06T07:30:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.162+0000 7f074b5e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 0x7f07440ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.162+0000 7f074b5e9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07440d29d0 0x7f07440d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.162+0000 7f074b5e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07440d3740 con 0x7f07440d4720 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.162+0000 7f074b5e9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07440d3400 con 0x7f07440d29d0 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.162+0000 7f074b5e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07440d3120 con 0x7f074415dd90 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.162+0000 7f0749b86640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f07440d4720 0x7f07440b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50934/0 (socket says 172.21.15.27:50934) 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.162+0000 7f0749b86640 1 -- 172.21.15.27:0/918228131 learned_addr learned my addr 172.21.15.27:0/918228131 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.162+0000 7f0748b84640 1 --2- 172.21.15.27:0/918228131 >> v2:172.21.15.27:3300/0 conn(0x7f07440d29d0 0x7f07440d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.163+0000 7f0743fff640 1 -- 172.21.15.27:0/918228131 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910863554 0 0) 0x7f07440d3740 con 0x7f07440d4720 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.163+0000 7f0743fff640 1 -- 172.21.15.27:0/918228131 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f072c003520 con 0x7f07440d4720 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.163+0000 7f0749385640 1 --2- 172.21.15.27:0/918228131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 0x7f07440ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.163+0000 7f0748b84640 1 -- 172.21.15.27:0/918228131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 msgr2=0x7f07440ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.163+0000 7f0748b84640 1 --2- 172.21.15.27:0/918228131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 0x7f07440ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.163+0000 7f0748b84640 1 -- 172.21.15.27:0/918228131 >> v1:172.21.15.27:6789/0 conn(0x7f07440d4720 legacy=0x7f07440b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.163+0000 7f0748b84640 1 -- 172.21.15.27:0/918228131 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07440d39e0 con 0x7f07440d29d0 2024-03-06T07:30:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.163+0000 7f0749385640 1 --2- 172.21.15.27:0/918228131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 0x7f07440ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:19.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f0748b84640 1 --2- 172.21.15.27:0/918228131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 0x7f07440d41e0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f07380097a0 tx=0x7f073802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36826a09fdf6801a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f0743fff640 1 -- 172.21.15.27:0/918228131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738034070 con 0x7f07440d29d0 2024-03-06T07:30:19.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f0743fff640 1 -- 172.21.15.27:0/918228131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738030610 con 0x7f07440d29d0 2024-03-06T07:30:19.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f0743fff640 1 -- 172.21.15.27:0/918228131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380309c0 con 0x7f07440d29d0 2024-03-06T07:30:19.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f074b5e9640 1 -- 172.21.15.27:0/918228131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 msgr2=0x7f07440d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f074b5e9640 1 --2- 172.21.15.27:0/918228131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 0x7f07440d41e0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f07380097a0 tx=0x7f073802b960 comp rx=0 tx=0).stop 2024-03-06T07:30:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f074b5e9640 1 -- 172.21.15.27:0/918228131 shutdown_connections 2024-03-06T07:30:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f074b5e9640 1 --2- 172.21.15.27:0/918228131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 0x7f07440d41e0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f074b5e9640 1 --2- 172.21.15.27:0/918228131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 0x7f07440ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f074b5e9640 1 -- 172.21.15.27:0/918228131 >> 172.21.15.27:0/918228131 conn(0x7f0744071960 msgr2=0x7f0744071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f074b5e9640 1 -- 172.21.15.27:0/918228131 shutdown_connections 2024-03-06T07:30:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.164+0000 7f074b5e9640 1 -- 172.21.15.27:0/918228131 wait complete. 2024-03-06T07:30:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f074b5e9640 1 Processor -- start 2024-03-06T07:30:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f074b5e9640 1 -- start start 2024-03-06T07:30:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f074b5e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 0x7f07441565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f074b5e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07440d4720 0x7f07441f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f074b5e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 0x7f074414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f074b5e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07440d2df0 con 0x7f07440d29d0 2024-03-06T07:30:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f074b5e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074415e170 con 0x7f07440d4720 2024-03-06T07:30:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f074b5e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07440b7420 con 0x7f074415dd90 2024-03-06T07:30:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f0749385640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 0x7f07441565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f0749385640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 0x7f07441565d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55502/0 (socket says 172.21.15.27:55502) 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.166+0000 7f0749385640 1 -- 172.21.15.27:0/2088106336 learned_addr learned my addr 172.21.15.27:0/2088106336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0748b84640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07440d4720 0x7f07441f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0749b86640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 0x7f074414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0749385640 1 -- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 msgr2=0x7f074414b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0749385640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 0x7f074414b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0749385640 1 -- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07440d4720 msgr2=0x7f07441f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0749385640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07440d4720 0x7f07441f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0749385640 1 -- 172.21.15.27:0/2088106336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0744156bb0 con 0x7f07440d29d0 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0748b84640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07440d4720 0x7f07441f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0749385640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 0x7f07441565d0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f073c00ce30 tx=0x7f073c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0741ffb640 1 -- 172.21.15.27:0/2088106336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c00f890 con 0x7f07440d29d0 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0741ffb640 1 -- 172.21.15.27:0/2088106336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073c00fa30 con 0x7f07440d29d0 2024-03-06T07:30:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f0741ffb640 1 -- 172.21.15.27:0/2088106336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c00fe00 con 0x7f07440d29d0 2024-03-06T07:30:19.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0744156e90 con 0x7f07440d29d0 2024-03-06T07:30:19.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.167+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0744158a10 con 0x7f07440d29d0 2024-03-06T07:30:19.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.168+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074404f410 con 0x7f07440d29d0 2024-03-06T07:30:19.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.168+0000 7f0741ffb640 1 -- 172.21.15.27:0/2088106336 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f073c018400 con 0x7f07440d29d0 2024-03-06T07:30:19.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.169+0000 7f0741ffb640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f073006b4d0 0x7f073006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.169+0000 7f0741ffb640 1 -- 172.21.15.27:0/2088106336 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f073c08ac40 con 0x7f07440d29d0 2024-03-06T07:30:19.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.169+0000 7f0748b84640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f073006b4d0 0x7f073006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.170+0000 7f0748b84640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f073006b4d0 0x7f073006d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0738036000 tx=0x7f073803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.173+0000 7f0741ffb640 1 -- 172.21.15.27:0/2088106336 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f073c014030 con 0x7f07440d29d0 2024-03-06T07:30:19.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:19.299 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:19.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.354+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0744072a10 con 0x7f07440d29d0 2024-03-06T07:30:19.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.354+0000 7f0741ffb640 1 -- 172.21.15.27:0/2088106336 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v128) v1 ==== 93+0+10 (secure 0 0 0) 0x7f073c00ae20 con 0x7f07440d29d0 2024-03-06T07:30:19.357 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:30:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f073006b4d0 msgr2=0x7f073006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f073006b4d0 0x7f073006d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0738036000 tx=0x7f073803b040 comp rx=0 tx=0).stop 2024-03-06T07:30:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 msgr2=0x7f07441565d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 0x7f07441565d0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f073c00ce30 tx=0x7f073c00a650 comp rx=0 tx=0).stop 2024-03-06T07:30:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 shutdown_connections 2024-03-06T07:30:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f073006b4d0 0x7f073006d990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f074415dd90 0x7f074414b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07440d4720 0x7f07441f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 --2- 172.21.15.27:0/2088106336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07440d29d0 0x7f07441565d0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 >> 172.21.15.27:0/2088106336 conn(0x7f0744071960 msgr2=0x7f07440d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.356+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 shutdown_connections 2024-03-06T07:30:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.357+0000 7f074b5e9640 1 -- 172.21.15.27:0/2088106336 wait complete. 2024-03-06T07:30:19.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:30:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.515+0000 7f167155c640 1 Processor -- start 2024-03-06T07:30:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.515+0000 7f167155c640 1 -- start start 2024-03-06T07:30:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f167155c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f166c15e030 0x7f166c06ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f167155c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c0b2460 0x7f166c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f167155c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c15e410 con 0x7f166c06ef70 2024-03-06T07:30:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f167155c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166c162700 con 0x7f166c15e030 2024-03-06T07:30:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f167155c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c161af0 con 0x7f166c0b2460 2024-03-06T07:30:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f166a7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f166c06ef70 0x7f166c0b1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50944/0 (socket says 172.21.15.27:50944) 2024-03-06T07:30:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f166a7fc640 1 -- 172.21.15.27:0/3786158512 learned_addr learned my addr 172.21.15.27:0/3786158512 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f166b7fe640 1 --2- 172.21.15.27:0/3786158512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c0b2460 0x7f166c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f166affd640 1 --2- 172.21.15.27:0/3786158512 >> v2:172.21.15.69:3300/0 conn(0x7f166c15e030 0x7f166c06ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f1669ffb640 1 -- 172.21.15.27:0/3786158512 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643752943 0 0) 0x7f1654001110 con 0x7f166c06ef70 2024-03-06T07:30:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f1669ffb640 1 -- 172.21.15.27:0/3786158512 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1658003520 con 0x7f166c06ef70 2024-03-06T07:30:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f1669ffb640 1 -- 172.21.15.27:0/3786158512 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57966762 0 0) 0x7f1658003520 con 0x7f166c06ef70 2024-03-06T07:30:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f166b7fe640 1 -- 172.21.15.27:0/3786158512 >> v2:172.21.15.69:3300/0 conn(0x7f166c15e030 msgr2=0x7f166c06ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f166b7fe640 1 --2- 172.21.15.27:0/3786158512 >> v2:172.21.15.69:3300/0 conn(0x7f166c15e030 0x7f166c06ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f166b7fe640 1 -- 172.21.15.27:0/3786158512 >> v1:172.21.15.27:6789/0 conn(0x7f166c06ef70 legacy=0x7f166c0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.516+0000 7f166b7fe640 1 -- 172.21.15.27:0/3786158512 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c161dd0 con 0x7f166c0b2460 2024-03-06T07:30:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.517+0000 7f166affd640 1 --2- 172.21.15.27:0/3786158512 >> v2:172.21.15.69:3300/0 conn(0x7f166c15e030 0x7f166c06ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.517+0000 7f166b7fe640 1 --2- 172.21.15.27:0/3786158512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c0b2460 0x7f166c06d210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f165c009d90 tx=0x7f165c02bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd04441f121b42ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.517+0000 7f1669ffb640 1 -- 172.21.15.27:0/3786158512 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c039070 con 0x7f166c0b2460 2024-03-06T07:30:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f1669ffb640 1 -- 172.21.15.27:0/3786158512 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c02c8a0 con 0x7f166c0b2460 2024-03-06T07:30:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f1669ffb640 1 -- 172.21.15.27:0/3786158512 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654001840 con 0x7f166c0b2460 2024-03-06T07:30:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f167155c640 1 -- 172.21.15.27:0/3786158512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c0b2460 msgr2=0x7f166c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f167155c640 1 --2- 172.21.15.27:0/3786158512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c0b2460 0x7f166c06d210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f165c009d90 tx=0x7f165c02bd70 comp rx=0 tx=0).stop 2024-03-06T07:30:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f167155c640 1 -- 172.21.15.27:0/3786158512 shutdown_connections 2024-03-06T07:30:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f167155c640 1 --2- 172.21.15.27:0/3786158512 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c0b2460 0x7f166c06d210 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f167155c640 1 --2- 172.21.15.27:0/3786158512 >> v2:172.21.15.69:3300/0 conn(0x7f166c15e030 0x7f166c06ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f167155c640 1 -- 172.21.15.27:0/3786158512 >> 172.21.15.27:0/3786158512 conn(0x7f166c0b55e0 msgr2=0x7f166c0b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f167155c640 1 -- 172.21.15.27:0/3786158512 shutdown_connections 2024-03-06T07:30:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.518+0000 7f167155c640 1 -- 172.21.15.27:0/3786158512 wait complete. 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.519+0000 7f167155c640 1 Processor -- start 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.519+0000 7f167155c640 1 -- start start 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f167155c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c06ef70 0x7f166c155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f167155c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f166c0b2460 0x7f166c1f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f167155c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f166c15e030 0x7f166c1f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f167155c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b31e0 con 0x7f166c0b2460 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f167155c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c162390 con 0x7f166c15e030 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f167155c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b3360 con 0x7f166c06ef70 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f166a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f166c0b2460 0x7f166c1f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f166a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f166c0b2460 0x7f166c1f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55504/0 (socket says 172.21.15.27:55504) 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f166a7fc640 1 -- 172.21.15.27:0/2307281378 learned_addr learned my addr 172.21.15.27:0/2307281378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f166b7fe640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f166c15e030 0x7f166c1f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.520+0000 7f166affd640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c06ef70 0x7f166c155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f166a7fc640 1 -- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c06ef70 msgr2=0x7f166c155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f166a7fc640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c06ef70 0x7f166c155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f166a7fc640 1 -- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f166c15e030 msgr2=0x7f166c1f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f166a7fc640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f166c15e030 0x7f166c1f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f166a7fc640 1 -- 172.21.15.27:0/2307281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c156010 con 0x7f166c0b2460 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f166a7fc640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f166c0b2460 0x7f166c1f3a10 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f165400c750 tx=0x7f165400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f16637fe640 1 -- 172.21.15.27:0/2307281378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654005ad0 con 0x7f166c0b2460 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f16637fe640 1 -- 172.21.15.27:0/2307281378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1654005c70 con 0x7f166c0b2460 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c155a50 con 0x7f166c0b2460 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f16637fe640 1 -- 172.21.15.27:0/2307281378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16540072b0 con 0x7f166c0b2460 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.521+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c097550 con 0x7f166c0b2460 2024-03-06T07:30:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.522+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c067ff0 con 0x7f166c0b2460 2024-03-06T07:30:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.523+0000 7f16637fe640 1 -- 172.21.15.27:0/2307281378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1654007450 con 0x7f166c0b2460 2024-03-06T07:30:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.523+0000 7f16637fe640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f164c06b4d0 0x7f164c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.523+0000 7f16637fe640 1 -- 172.21.15.27:0/2307281378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1654089bc0 con 0x7f166c0b2460 2024-03-06T07:30:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.523+0000 7f166affd640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f164c06b4d0 0x7f164c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.524+0000 7f166affd640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f164c06b4d0 0x7f164c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1664009560 tx=0x7f1664009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.527+0000 7f16637fe640 1 -- 172.21.15.27:0/2307281378 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f166c15e410 con 0x7f166c0b2460 2024-03-06T07:30:19.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.709+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f166c0b2910 con 0x7f166c0b2460 2024-03-06T07:30:19.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.710+0000 7f16637fe640 1 -- 172.21.15.27:0/2307281378 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v128) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1654058740 con 0x7f166c0b2460 2024-03-06T07:30:19.712 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:19.712 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:30:19.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:30:19.713 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:30:19.713 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:30:19.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:19.713 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:30:19.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.711+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f164c06b4d0 msgr2=0x7f164c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.711+0000 7f167155c640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f164c06b4d0 0x7f164c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1664009560 tx=0x7f1664009190 comp rx=0 tx=0).stop 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.711+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f166c0b2460 msgr2=0x7f166c1f3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.711+0000 7f167155c640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f166c0b2460 0x7f166c1f3a10 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f165400c750 tx=0x7f165400cc10 comp rx=0 tx=0).stop 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.712+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 shutdown_connections 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.712+0000 7f167155c640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f164c06b4d0 0x7f164c06d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.712+0000 7f167155c640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f166c15e030 0x7f166c1f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.712+0000 7f167155c640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f166c0b2460 0x7f166c1f3a10 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.712+0000 7f167155c640 1 --2- 172.21.15.27:0/2307281378 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f166c06ef70 0x7f166c155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.712+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 >> 172.21.15.27:0/2307281378 conn(0x7f166c0b55e0 msgr2=0x7f166c15fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.712+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 shutdown_connections 2024-03-06T07:30:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.712+0000 7f167155c640 1 -- 172.21.15.27:0/2307281378 wait complete. 2024-03-06T07:30:19.725 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:30:19.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:30:19.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.834+0000 7ff3e4271640 1 Processor -- start 2024-03-06T07:30:19.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.834+0000 7ff3e4271640 1 -- start start 2024-03-06T07:30:19.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e4271640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff3dc0d0110 0x7ff3dc0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e4271640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc0d05a0 0x7ff3dc0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e4271640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3dc0b33b0 con 0x7ff3dc15f5c0 2024-03-06T07:30:19.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e4271640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc0b0af0 con 0x7ff3dc0d0110 2024-03-06T07:30:19.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e4271640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc0b0d40 con 0x7ff3dc0d05a0 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e200d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff3dc15f5c0 0x7ff3dc0cfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50958/0 (socket says 172.21.15.27:50958) 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e200d640 1 -- 172.21.15.27:0/3133652769 learned_addr learned my addr 172.21.15.27:0/3133652769 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888415168 0 0) 0x7ff3dc0b33b0 con 0x7ff3dc15f5c0 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c8003520 con 0x7ff3dc15f5c0 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e180c640 1 --2- 172.21.15.27:0/3133652769 >> v2:172.21.15.69:3300/0 conn(0x7ff3dc0d0110 0x7ff3dc0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e280e640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc0d05a0 0x7ff3dc0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4151294834 0 0) 0x7ff3c8003520 con 0x7ff3dc15f5c0 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc0d05a0 msgr2=0x7ff3dc0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e100b640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc0d05a0 0x7ff3dc0ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 >> v2:172.21.15.69:3300/0 conn(0x7ff3dc0d0110 msgr2=0x7ff3dc0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.835+0000 7ff3e100b640 1 --2- 172.21.15.27:0/3133652769 >> v2:172.21.15.69:3300/0 conn(0x7ff3dc0d0110 0x7ff3dc0cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc0b1020 con 0x7ff3dc15f5c0 2024-03-06T07:30:19.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7ff3d4001690 con 0x7ff3dc15f5c0 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 >> v1:172.21.15.27:6789/0 conn(0x7ff3dc15f5c0 legacy=0x7ff3dc0cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e100b640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3c80039b0 0x7ff3c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e100b640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c8004800 0x7ff3c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e100b640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc0d05a0 0x7ff3c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc0b0af0 con 0x7ff3c8004800 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc0b0d40 con 0x7ff3c80039b0 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c800b9f0 con 0x7ff3dc0d05a0 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e180c640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c8004800 0x7ff3c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e200d640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3c80039b0 0x7ff3c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e280e640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc0d05a0 0x7ff3c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e180c640 1 -- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc0d05a0 msgr2=0x7ff3c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e180c640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc0d05a0 0x7ff3c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e180c640 1 -- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3c80039b0 msgr2=0x7ff3c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e180c640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3c80039b0 0x7ff3c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e180c640 1 -- 172.21.15.27:0/3133652769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c8003520 con 0x7ff3c8004800 2024-03-06T07:30:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.836+0000 7ff3e280e640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3dc0d05a0 0x7ff3c80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.837+0000 7ff3e180c640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c8004800 0x7ff3c8004c80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc009ac0 tx=0x7ff3c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e171ccae694508 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.837+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc039070 con 0x7ff3c8004800 2024-03-06T07:30:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.837+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc02c6d0 con 0x7ff3c8004800 2024-03-06T07:30:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.837+0000 7ff3e100b640 1 -- 172.21.15.27:0/3133652769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc009690 con 0x7ff3c8004800 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.837+0000 7ff3e4271640 1 -- 172.21.15.27:0/3133652769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c8004800 msgr2=0x7ff3c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.837+0000 7ff3e4271640 1 --2- 172.21.15.27:0/3133652769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3c8004800 0x7ff3c8004c80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc009ac0 tx=0x7ff3c8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.837+0000 7ff3e280e640 1 -- 172.21.15.27:0/3133652769 reap_dead start 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.838+0000 7ff3e4271640 1 -- 172.21.15.27:0/3133652769 shutdown_connections 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.838+0000 7ff3e4271640 1 -- 172.21.15.27:0/3133652769 >> 172.21.15.27:0/3133652769 conn(0x7ff3dc0b36d0 msgr2=0x7ff3dc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.838+0000 7ff3e4271640 1 -- 172.21.15.27:0/3133652769 shutdown_connections 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.838+0000 7ff3e4271640 1 -- 172.21.15.27:0/3133652769 wait complete. 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.839+0000 7ff3e4271640 1 Processor -- start 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.839+0000 7ff3e4271640 1 -- start start 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.839+0000 7ff3e4271640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3c8004800 0x7ff3dc06b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.839+0000 7ff3e4271640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3dc0b53b0 0x7ff3dc150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.839+0000 7ff3e4271640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc06c780 0x7ff3dc06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.839+0000 7ff3e4271640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc15f9a0 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.839+0000 7ff3e4271640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc0ced80 con 0x7ff3dc06c780 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.839+0000 7ff3e4271640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc0cf2a0 con 0x7ff3c8004800 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e180c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3dc0b53b0 0x7ff3dc150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e180c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3dc0b53b0 0x7ff3dc150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55532/0 (socket says 172.21.15.27:55532) 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e180c640 1 -- 172.21.15.27:0/1789977438 learned_addr learned my addr 172.21.15.27:0/1789977438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e180c640 1 -- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3c8004800 msgr2=0x7ff3dc06b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e180c640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3c8004800 0x7ff3dc06b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e180c640 1 -- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc06c780 msgr2=0x7ff3dc06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e280e640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc06c780 0x7ff3dc06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e180c640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc06c780 0x7ff3dc06eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e180c640 1 -- 172.21.15.27:0/1789977438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc06b6e0 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e180c640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3dc0b53b0 0x7ff3dc150f00 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc009be0 tx=0x7ff3cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3d2ffd640 1 -- 172.21.15.27:0/1789977438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc042020 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3d2ffd640 1 -- 172.21.15.27:0/1789977438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc03a040 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc06b9c0 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3d2ffd640 1 -- 172.21.15.27:0/1789977438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc039070 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.840+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc06bf80 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.841+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc09fd30 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.842+0000 7ff3d2ffd640 1 -- 172.21.15.27:0/1789977438 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3cc045020 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.842+0000 7ff3d2ffd640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3c006b4d0 0x7ff3c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.842+0000 7ff3d2ffd640 1 -- 172.21.15.27:0/1789977438 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff3cc0ab6c0 con 0x7ff3dc0b53b0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.842+0000 7ff3e200d640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3c006b4d0 0x7ff3c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.843+0000 7ff3e200d640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3c006b4d0 0x7ff3c006d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4001200 tx=0x7ff3d4001190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:19.846+0000 7ff3d2ffd640 1 -- 172.21.15.27:0/1789977438 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3cc04c030 con 0x7ff3dc0b53b0 2024-03-06T07:30:20.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.027+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff3dc04f410 con 0x7ff3dc0b53b0 2024-03-06T07:30:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.333+0000 7ff3d2ffd640 1 -- 172.21.15.27:0/1789977438 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v129) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff3cc07a890 con 0x7ff3dc0b53b0 2024-03-06T07:30:20.335 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:30:20.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3c006b4d0 msgr2=0x7ff3c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3c006b4d0 0x7ff3c006d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4001200 tx=0x7ff3d4001190 comp rx=0 tx=0).stop 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3dc0b53b0 msgr2=0x7ff3dc150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3dc0b53b0 0x7ff3dc150f00 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc009be0 tx=0x7ff3cc004730 comp rx=0 tx=0).stop 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 shutdown_connections 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3c006b4d0 0x7ff3c006d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3dc06c780 0x7ff3dc06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3dc0b53b0 0x7ff3dc150f00 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 --2- 172.21.15.27:0/1789977438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3c8004800 0x7ff3dc06b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 >> 172.21.15.27:0/1789977438 conn(0x7ff3dc0b36d0 msgr2=0x7ff3dc06fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 shutdown_connections 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.335+0000 7ff3e4271640 1 -- 172.21.15.27:0/1789977438 wait complete. 2024-03-06T07:30:20.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:30:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.464+0000 7fb863b55640 1 Processor -- start 2024-03-06T07:30:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.464+0000 7fb863b55640 1 -- start start 2024-03-06T07:30:20.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.465+0000 7fb863b55640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb85c0cead0 0x7fb85c168e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.465+0000 7fb863b55640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c15e6d0 con 0x7fb85c15e2f0 2024-03-06T07:30:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.465+0000 7fb863b55640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c0ccd00 con 0x7fb85c0b41f0 2024-03-06T07:30:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.465+0000 7fb863b55640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85c0d0a10 con 0x7fb85c0cead0 2024-03-06T07:30:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.465+0000 7fb8618f1640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb85c15e2f0 0x7fb85c06b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50972/0 (socket says 172.21.15.27:50972) 2024-03-06T07:30:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.465+0000 7fb8620f2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb85c0cead0 0x7fb85c168e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.465+0000 7fb8618f1640 1 -- 172.21.15.27:0/69472925 learned_addr learned my addr 172.21.15.27:0/69472925 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.466+0000 7fb8620f2640 1 -- 172.21.15.27:0/69472925 >> v1:172.21.15.69:6789/0 conn(0x7fb85c0b41f0 legacy=0x7fb85c06bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.466+0000 7fb8620f2640 1 -- 172.21.15.27:0/69472925 >> v1:172.21.15.27:6789/0 conn(0x7fb85c15e2f0 legacy=0x7fb85c06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.466+0000 7fb8620f2640 1 -- 172.21.15.27:0/69472925 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c0ccfe0 con 0x7fb85c0cead0 2024-03-06T07:30:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.466+0000 7fb8608ef640 1 -- 172.21.15.27:0/69472925 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082529025 0 0) 0x7fb85c15e6d0 con 0x7fb85c15e2f0 2024-03-06T07:30:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.466+0000 7fb8620f2640 1 --2- 172.21.15.27:0/69472925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0cead0 0x7fb85c168e30 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fb8540097a0 tx=0x7fb85402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dbbc3604d5b742 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb8608ef640 1 -- 172.21.15.27:0/69472925 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854035020 con 0x7fb85c0cead0 2024-03-06T07:30:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb8608ef640 1 -- 172.21.15.27:0/69472925 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb854031830 con 0x7fb85c0cead0 2024-03-06T07:30:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb8608ef640 1 -- 172.21.15.27:0/69472925 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854031c00 con 0x7fb85c0cead0 2024-03-06T07:30:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb863b55640 1 -- 172.21.15.27:0/69472925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0cead0 msgr2=0x7fb85c168e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb863b55640 1 --2- 172.21.15.27:0/69472925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0cead0 0x7fb85c168e30 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fb8540097a0 tx=0x7fb85402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb863b55640 1 -- 172.21.15.27:0/69472925 shutdown_connections 2024-03-06T07:30:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb863b55640 1 --2- 172.21.15.27:0/69472925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0cead0 0x7fb85c168e30 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb863b55640 1 -- 172.21.15.27:0/69472925 >> 172.21.15.27:0/69472925 conn(0x7fb85c0b36d0 msgr2=0x7fb85c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb863b55640 1 -- 172.21.15.27:0/69472925 shutdown_connections 2024-03-06T07:30:20.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.467+0000 7fb863b55640 1 -- 172.21.15.27:0/69472925 wait complete. 2024-03-06T07:30:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.468+0000 7fb863b55640 1 Processor -- start 2024-03-06T07:30:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb863b55640 1 -- start start 2024-03-06T07:30:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb863b55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0b41f0 0x7fb85c14e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb863b55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85c0cead0 0x7fb85c149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb863b55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb85c15e2f0 0x7fb85c14f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb863b55640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c06c300 con 0x7fb85c0b41f0 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb863b55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c0cd5a0 con 0x7fb85c15e2f0 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb863b55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c169370 con 0x7fb85c0cead0 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb8618f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0b41f0 0x7fb85c14e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb8618f1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0b41f0 0x7fb85c14e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55552/0 (socket says 172.21.15.27:55552) 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb8618f1640 1 -- 172.21.15.27:0/788654219 learned_addr learned my addr 172.21.15.27:0/788654219 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.469+0000 7fb8620f2640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb85c15e2f0 0x7fb85c14f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb8610f0640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85c0cead0 0x7fb85c149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb8618f1640 1 -- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85c0cead0 msgr2=0x7fb85c149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb8618f1640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85c0cead0 0x7fb85c149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb8618f1640 1 -- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb85c15e2f0 msgr2=0x7fb85c14f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb8618f1640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb85c15e2f0 0x7fb85c14f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb8618f1640 1 -- 172.21.15.27:0/788654219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c14d910 con 0x7fb85c0b41f0 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb8610f0640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85c0cead0 0x7fb85c149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb8618f1640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0b41f0 0x7fb85c14e9b0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00c780 tx=0x7fb84c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb85a7fc640 1 -- 172.21.15.27:0/788654219 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c018070 con 0x7fb85c0b41f0 2024-03-06T07:30:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb85a7fc640 1 -- 172.21.15.27:0/788654219 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c00a640 con 0x7fb85c0b41f0 2024-03-06T07:30:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c14dbf0 con 0x7fb85c0b41f0 2024-03-06T07:30:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb85a7fc640 1 -- 172.21.15.27:0/788654219 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c00aa70 con 0x7fb85c0b41f0 2024-03-06T07:30:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.470+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c14e1b0 con 0x7fb85c0b41f0 2024-03-06T07:30:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.471+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c09fd30 con 0x7fb85c0b41f0 2024-03-06T07:30:20.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.471+0000 7fb85a7fc640 1 -- 172.21.15.27:0/788654219 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb84c01b070 con 0x7fb85c0b41f0 2024-03-06T07:30:20.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.472+0000 7fb85a7fc640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb84806b4d0 0x7fb84806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.472+0000 7fb85a7fc640 1 -- 172.21.15.27:0/788654219 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb84c0895b0 con 0x7fb85c0b41f0 2024-03-06T07:30:20.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.472+0000 7fb8610f0640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb84806b4d0 0x7fb84806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.473+0000 7fb8610f0640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb84806b4d0 0x7fb84806d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb85c150ca0 tx=0x7fb850009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.477+0000 7fb85a7fc640 1 -- 172.21.15.27:0/788654219 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb84c0586a0 con 0x7fb85c0b41f0 2024-03-06T07:30:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:20.657+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb85c06c0a0 con 0x7fb85c0b41f0 2024-03-06T07:30:21.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.341+0000 7fb85a7fc640 1 -- 172.21.15.27:0/788654219 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v130) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb84c057fa0 con 0x7fb85c0b41f0 2024-03-06T07:30:21.343 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb84806b4d0 msgr2=0x7fb84806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb84806b4d0 0x7fb84806d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb85c150ca0 tx=0x7fb850009040 comp rx=0 tx=0).stop 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0b41f0 msgr2=0x7fb85c14e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0b41f0 0x7fb85c14e9b0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00c780 tx=0x7fb84c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 shutdown_connections 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb84806b4d0 0x7fb84806d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb85c15e2f0 0x7fb85c14f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb85c0cead0 0x7fb85c149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 --2- 172.21.15.27:0/788654219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb85c0b41f0 0x7fb85c14e9b0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 >> 172.21.15.27:0/788654219 conn(0x7fb85c0b36d0 msgr2=0x7fb85c0cef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 shutdown_connections 2024-03-06T07:30:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.344+0000 7fb863b55640 1 -- 172.21.15.27:0/788654219 wait complete. 2024-03-06T07:30:21.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:30:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.483+0000 7ff200e5c640 1 Processor -- start 2024-03-06T07:30:21.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.483+0000 7ff200e5c640 1 -- start start 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff200e5c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff1fc15def0 0x7ff1fc0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff200e5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1fc1621b0 0x7ff1fc066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff200e5c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc1627d0 con 0x7ff1fc0b3ef0 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff200e5c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1fc0b4350 con 0x7ff1fc15def0 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff200e5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc071940 con 0x7ff1fc1621b0 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f3fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff1fc0b3ef0 0x7ff1fc161aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50984/0 (socket says 172.21.15.27:50984) 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f3fff640 1 -- 172.21.15.27:0/186643964 learned_addr learned my addr 172.21.15.27:0/186643964 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1fad9d640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1fc1621b0 0x7ff1fc066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1fa59c640 1 --2- 172.21.15.27:0/186643964 >> v2:172.21.15.69:3300/0 conn(0x7ff1fc15def0 0x7ff1fc0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45121079 0 0) 0x7ff1fc1627d0 con 0x7ff1fc0b3ef0 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e8003520 con 0x7ff1fc0b3ef0 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 810778343 0 0) 0x7ff1e8003520 con 0x7ff1fc0b3ef0 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1fc1621b0 msgr2=0x7ff1fc066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1fc1621b0 0x7ff1fc066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 >> v2:172.21.15.69:3300/0 conn(0x7ff1fc15def0 msgr2=0x7ff1fc0b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 --2- 172.21.15.27:0/186643964 >> v2:172.21.15.69:3300/0 conn(0x7ff1fc15def0 0x7ff1fc0b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc071b90 con 0x7ff1fc0b3ef0 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7ff1e4001670 con 0x7ff1fc0b3ef0 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1fa59c640 1 --2- 172.21.15.27:0/186643964 >> v2:172.21.15.69:3300/0 conn(0x7ff1fc15def0 0x7ff1fc0b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 >> v1:172.21.15.27:6789/0 conn(0x7ff1fc0b3ef0 legacy=0x7ff1fc161aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1e80039b0 0x7ff1e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.484+0000 7ff1f9d9b640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e8004800 0x7ff1e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1f9d9b640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1fc1621b0 0x7ff1e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc071940 con 0x7ff1e80039b0 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e800b9c0 con 0x7ff1e8004800 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e800bb40 con 0x7ff1fc1621b0 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1fa59c640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1e80039b0 0x7ff1e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1f3fff640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e8004800 0x7ff1e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1fad9d640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1fc1621b0 0x7ff1e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1fa59c640 1 -- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1fc1621b0 msgr2=0x7ff1e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1fa59c640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1fc1621b0 0x7ff1e80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1fa59c640 1 -- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e8004800 msgr2=0x7ff1e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1fa59c640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e8004800 0x7ff1e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1fa59c640 1 -- 172.21.15.27:0/186643964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e8003520 con 0x7ff1e80039b0 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.485+0000 7ff1fad9d640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1fc1621b0 0x7ff1e80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff1fa59c640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1e80039b0 0x7ff1e8003dd0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009c90 tx=0x7ff1e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b38d2a35238a677 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec039070 con 0x7ff1e80039b0 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ec02c980 con 0x7ff1e80039b0 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff1f9d9b640 1 -- 172.21.15.27:0/186643964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec009860 con 0x7ff1e80039b0 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff200e5c640 1 -- 172.21.15.27:0/186643964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1e80039b0 msgr2=0x7ff1e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff200e5c640 1 --2- 172.21.15.27:0/186643964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1e80039b0 0x7ff1e8003dd0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009c90 tx=0x7ff1e8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff1fad9d640 1 -- 172.21.15.27:0/186643964 reap_dead start 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff200e5c640 1 -- 172.21.15.27:0/186643964 shutdown_connections 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff200e5c640 1 -- 172.21.15.27:0/186643964 >> 172.21.15.27:0/186643964 conn(0x7ff1fc0b26b0 msgr2=0x7ff1fc0b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff200e5c640 1 -- 172.21.15.27:0/186643964 shutdown_connections 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.486+0000 7ff200e5c640 1 -- 172.21.15.27:0/186643964 wait complete. 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.488+0000 7ff200e5c640 1 Processor -- start 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.488+0000 7ff200e5c640 1 -- start start 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff200e5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1e80039b0 0x7ff1fc1412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff200e5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1fc1f2df0 0x7ff1fc0aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff200e5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc1f3270 0x7ff1fc0aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff200e5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc15e2d0 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff200e5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc0721e0 con 0x7ff1fc1f3270 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff200e5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc072670 con 0x7ff1e80039b0 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1fc1f2df0 0x7ff1fc0aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1fc1f2df0 0x7ff1fc0aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55574/0 (socket says 172.21.15.27:55574) 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f3fff640 1 -- 172.21.15.27:0/1570044615 learned_addr learned my addr 172.21.15.27:0/1570044615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1fa59c640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1e80039b0 0x7ff1fc1412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1fad9d640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc1f3270 0x7ff1fc0aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f3fff640 1 -- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1e80039b0 msgr2=0x7ff1fc1412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f3fff640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1e80039b0 0x7ff1fc1412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f3fff640 1 -- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc1f3270 msgr2=0x7ff1fc0aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f3fff640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc1f3270 0x7ff1fc0aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f3fff640 1 -- 172.21.15.27:0/1570044615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc141820 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1fad9d640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc1f3270 0x7ff1fc0aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f3fff640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1fc1f2df0 0x7ff1fc0aeab0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4007340 tx=0x7ff1e4007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f2ffd640 1 -- 172.21.15.27:0/1570044615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e400f9c0 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f2ffd640 1 -- 172.21.15.27:0/1570044615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e4018070 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc15f0d0 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff1f2ffd640 1 -- 172.21.15.27:0/1570044615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e400fda0 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.489+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc160000 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.490+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc073380 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.491+0000 7ff1f2ffd640 1 -- 172.21.15.27:0/1570044615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1e401d460 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.491+0000 7ff1f2ffd640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1dc06b500 0x7ff1dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.491+0000 7ff1f2ffd640 1 -- 172.21.15.27:0/1570044615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff1e408a200 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.491+0000 7ff1fa59c640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1dc06b500 0x7ff1dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.492+0000 7ff1fa59c640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1dc06b500 0x7ff1dc06d9c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec036000 tx=0x7ff1ec030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.496+0000 7ff1f2ffd640 1 -- 172.21.15.27:0/1570044615 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1e40590e0 con 0x7ff1fc1f2df0 2024-03-06T07:30:21.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:21.717+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff1fc04f410 con 0x7ff1fc1f2df0 2024-03-06T07:30:22.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.359+0000 7ff1f2ffd640 1 -- 172.21.15.27:0/1570044615 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v88) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff1e40589e0 con 0x7ff1fc1f2df0 2024-03-06T07:30:22.361 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.366+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1dc06b500 msgr2=0x7ff1dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.367+0000 7ff200e5c640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1dc06b500 0x7ff1dc06d9c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec036000 tx=0x7ff1ec030000 comp rx=0 tx=0).stop 2024-03-06T07:30:22.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.367+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1fc1f2df0 msgr2=0x7ff1fc0aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.367+0000 7ff200e5c640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1fc1f2df0 0x7ff1fc0aeab0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4007340 tx=0x7ff1e4007800 comp rx=0 tx=0).stop 2024-03-06T07:30:22.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.369+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 shutdown_connections 2024-03-06T07:30:22.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.369+0000 7ff200e5c640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff1dc06b500 0x7ff1dc06d9c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.369+0000 7ff200e5c640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc1f3270 0x7ff1fc0aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.370+0000 7ff200e5c640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1fc1f2df0 0x7ff1fc0aeab0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.370+0000 7ff200e5c640 1 --2- 172.21.15.27:0/1570044615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff1e80039b0 0x7ff1fc1412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.370+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 >> 172.21.15.27:0/1570044615 conn(0x7ff1fc0b26b0 msgr2=0x7ff1fc06d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.370+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 shutdown_connections 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.370+0000 7ff200e5c640 1 -- 172.21.15.27:0/1570044615 wait complete. 2024-03-06T07:30:22.388 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:22.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.496+0000 7fde4d79a640 1 Processor -- start 2024-03-06T07:30:22.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.496+0000 7fde4d79a640 1 -- start start 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde4d79a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde4815e060 0x7fde480ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde4d79a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde480d0880 0x7fde480ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde4d79a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde480d0dd0 con 0x7fde4806fe20 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde4d79a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde480d10b0 con 0x7fde4815e060 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde4d79a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde480cc0e0 con 0x7fde480d0880 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde477fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde480d0880 0x7fde480ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde46ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde4815e060 0x7fde480ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde3e7fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fde4806fe20 0x7fde480ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:49124/0 (socket says 172.21.15.27:49124) 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde3e7fc640 1 -- 172.21.15.27:0/1573782766 learned_addr learned my addr 172.21.15.27:0/1573782766 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde46ffd640 1 -- 172.21.15.27:0/1573782766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde480d0880 msgr2=0x7fde480ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde46ffd640 1 --2- 172.21.15.27:0/1573782766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde480d0880 0x7fde480ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde46ffd640 1 -- 172.21.15.27:0/1573782766 >> v1:172.21.15.69:6789/0 conn(0x7fde4806fe20 legacy=0x7fde480ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.497+0000 7fde46ffd640 1 -- 172.21.15.27:0/1573782766 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde480cc3c0 con 0x7fde4815e060 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.498+0000 7fde46ffd640 1 --2- 172.21.15.27:0/1573782766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde4815e060 0x7fde480ce310 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fde300097a0 tx=0x7fde3002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8ebbfcc974de17c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.498+0000 7fde467fc640 1 -- 172.21.15.27:0/1573782766 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde30035020 con 0x7fde4815e060 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.498+0000 7fde467fc640 1 -- 172.21.15.27:0/1573782766 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde30031830 con 0x7fde4815e060 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.498+0000 7fde467fc640 1 -- 172.21.15.27:0/1573782766 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde30031c00 con 0x7fde4815e060 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.499+0000 7fde4d79a640 1 -- 172.21.15.27:0/1573782766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde4815e060 msgr2=0x7fde480ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.499+0000 7fde4d79a640 1 --2- 172.21.15.27:0/1573782766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde4815e060 0x7fde480ce310 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fde300097a0 tx=0x7fde3002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.499+0000 7fde4d79a640 1 -- 172.21.15.27:0/1573782766 shutdown_connections 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.499+0000 7fde4d79a640 1 --2- 172.21.15.27:0/1573782766 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde480d0880 0x7fde480ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.499+0000 7fde4d79a640 1 --2- 172.21.15.27:0/1573782766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde4815e060 0x7fde480ce310 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.499+0000 7fde4d79a640 1 -- 172.21.15.27:0/1573782766 >> 172.21.15.27:0/1573782766 conn(0x7fde480d1a90 msgr2=0x7fde480d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.499+0000 7fde4d79a640 1 -- 172.21.15.27:0/1573782766 shutdown_connections 2024-03-06T07:30:22.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.499+0000 7fde4d79a640 1 -- 172.21.15.27:0/1573782766 wait complete. 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.500+0000 7fde4d79a640 1 Processor -- start 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.500+0000 7fde4d79a640 1 -- start start 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde4d79a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde4806fe20 0x7fde48157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde4d79a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde480d0880 0x7fde481600f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde4d79a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde4815e060 0x7fde48160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde4d79a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde480cef60 con 0x7fde4815e060 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde4d79a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4815e440 con 0x7fde480d0880 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde4d79a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde480cf0e0 con 0x7fde4806fe20 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde4815e060 0x7fde48160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde4815e060 0x7fde48160630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55580/0 (socket says 172.21.15.27:55580) 2024-03-06T07:30:22.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde477fe640 1 -- 172.21.15.27:0/3654168100 learned_addr learned my addr 172.21.15.27:0/3654168100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde3e7fc640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde480d0880 0x7fde481600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.501+0000 7fde46ffd640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde4806fe20 0x7fde48157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde477fe640 1 -- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde4806fe20 msgr2=0x7fde48157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde477fe640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde4806fe20 0x7fde48157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde477fe640 1 -- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde480d0880 msgr2=0x7fde481600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde477fe640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde480d0880 0x7fde481600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde477fe640 1 -- 172.21.15.27:0/3654168100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde481585c0 con 0x7fde4815e060 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde46ffd640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde4806fe20 0x7fde48157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde477fe640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde4815e060 0x7fde48160630 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fde4000efc0 tx=0x7fde4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde3ffff640 1 -- 172.21.15.27:0/3654168100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40018020 con 0x7fde4815e060 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde3ffff640 1 -- 172.21.15.27:0/3654168100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde40014480 con 0x7fde4815e060 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4815e810 con 0x7fde4815e060 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde3ffff640 1 -- 172.21.15.27:0/3654168100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40014870 con 0x7fde4815e060 2024-03-06T07:30:22.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.502+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4815edd0 con 0x7fde4815e060 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.503+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde480c9310 con 0x7fde4815e060 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.503+0000 7fde3ffff640 1 -- 172.21.15.27:0/3654168100 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde40005d10 con 0x7fde4815e060 2024-03-06T07:30:22.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.504+0000 7fde3ffff640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde2c06b4d0 0x7fde2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.504+0000 7fde3ffff640 1 -- 172.21.15.27:0/3654168100 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde4008af30 con 0x7fde4815e060 2024-03-06T07:30:22.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.504+0000 7fde46ffd640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde2c06b4d0 0x7fde2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.505+0000 7fde46ffd640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde2c06b4d0 0x7fde2c06d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fde30036000 tx=0x7fde30008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.508+0000 7fde3ffff640 1 -- 172.21.15.27:0/3654168100 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde40022030 con 0x7fde4815e060 2024-03-06T07:30:22.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.689+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde48067ff0 con 0x7fde4815e060 2024-03-06T07:30:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.690+0000 7fde3ffff640 1 -- 172.21.15.27:0/3654168100 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16620 (secure 0 0 0) 0x7fde40010040 con 0x7fde4815e060 2024-03-06T07:30:22.692 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:22.692 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":131,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:30:21.718596+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:20.059798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.689553+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde2c06b4d0 msgr2=0x7fde2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde2c06b4d0 0x7fde2c06d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fde30036000 tx=0x7fde30008030 comp rx=0 tx=0).stop 2024-03-06T07:30:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde4815e060 msgr2=0x7fde48160630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde4815e060 0x7fde48160630 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fde4000efc0 tx=0x7fde4000c490 comp rx=0 tx=0).stop 2024-03-06T07:30:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 shutdown_connections 2024-03-06T07:30:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde2c06b4d0 0x7fde2c06d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde4815e060 0x7fde48160630 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde480d0880 0x7fde481600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 --2- 172.21.15.27:0/3654168100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde4806fe20 0x7fde48157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.692+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 >> 172.21.15.27:0/3654168100 conn(0x7fde480d1a90 msgr2=0x7fde480ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.693+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 shutdown_connections 2024-03-06T07:30:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.693+0000 7fde4d79a640 1 -- 172.21.15.27:0/3654168100 wait complete. 2024-03-06T07:30:22.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:22.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.811+0000 7f6617f1e640 1 Processor -- start 2024-03-06T07:30:22.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.811+0000 7f6617f1e640 1 -- start start 2024-03-06T07:30:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f6617f1e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66100aee10 0x7f6610165a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f6617f1e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6610071240 con 0x7f661006df30 2024-03-06T07:30:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f6617f1e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6610071520 con 0x7f6610158520 2024-03-06T07:30:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f6617f1e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6610071b70 con 0x7f66100aee10 2024-03-06T07:30:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f66164bb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66100aee10 0x7f6610165a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f66164bb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66100aee10 0x7f6610165a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55582/0 (socket says 172.21.15.27:55582) 2024-03-06T07:30:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f66164bb640 1 -- 172.21.15.27:0/539615958 learned_addr learned my addr 172.21.15.27:0/539615958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f66154b9640 1 --1- 172.21.15.27:0/539615958 >> v1:172.21.15.27:6789/0 conn(0x7f661006df30 0x7f66100ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50998/0 (socket says 172.21.15.27:50998) 2024-03-06T07:30:22.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f66164bb640 1 -- 172.21.15.27:0/539615958 >> v1:172.21.15.69:6789/0 conn(0x7f6610158520 legacy=0x7f661006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f66164bb640 1 -- 172.21.15.27:0/539615958 >> v1:172.21.15.27:6789/0 conn(0x7f661006df30 legacy=0x7f66100ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f6614cb8640 1 -- 172.21.15.27:0/539615958 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021266972 0 0) 0x7f6610071240 con 0x7f661006df30 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.812+0000 7f66164bb640 1 -- 172.21.15.27:0/539615958 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6610071800 con 0x7f66100aee10 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.813+0000 7f66164bb640 1 --2- 172.21.15.27:0/539615958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 0x7f6610165a00 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f660800d150 tx=0x7f660802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2518b9aa6028527d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.813+0000 7f6614cb8640 1 -- 172.21.15.27:0/539615958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660800b0d0 con 0x7f66100aee10 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.813+0000 7f6614cb8640 1 -- 172.21.15.27:0/539615958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660800b270 con 0x7f66100aee10 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.813+0000 7f6614cb8640 1 -- 172.21.15.27:0/539615958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6608009600 con 0x7f66100aee10 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.814+0000 7f6617f1e640 1 -- 172.21.15.27:0/539615958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 msgr2=0x7f6610165a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.814+0000 7f6617f1e640 1 --2- 172.21.15.27:0/539615958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 0x7f6610165a00 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f660800d150 tx=0x7f660802e170 comp rx=0 tx=0).stop 2024-03-06T07:30:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.814+0000 7f6617f1e640 1 -- 172.21.15.27:0/539615958 shutdown_connections 2024-03-06T07:30:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.814+0000 7f6617f1e640 1 --2- 172.21.15.27:0/539615958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 0x7f6610165a00 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.814+0000 7f6617f1e640 1 -- 172.21.15.27:0/539615958 >> 172.21.15.27:0/539615958 conn(0x7f6610066d00 msgr2=0x7f66100670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.814+0000 7f6617f1e640 1 -- 172.21.15.27:0/539615958 shutdown_connections 2024-03-06T07:30:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.814+0000 7f6617f1e640 1 -- 172.21.15.27:0/539615958 wait complete. 2024-03-06T07:30:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.815+0000 7f6617f1e640 1 Processor -- start 2024-03-06T07:30:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.815+0000 7f6617f1e640 1 -- start start 2024-03-06T07:30:22.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f6617f1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f661006df30 0x7f66101493e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f6617f1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 0x7f66100cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f6617f1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6610158520 0x7f66100cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f6617f1e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6610158900 con 0x7f66100aee10 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f6617f1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6610072000 con 0x7f661006df30 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f6617f1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6610158a80 con 0x7f6610158520 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66154b9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 0x7f66100cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66154b9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 0x7f66100cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55586/0 (socket says 172.21.15.27:55586) 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66154b9640 1 -- 172.21.15.27:0/633423666 learned_addr learned my addr 172.21.15.27:0/633423666 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f6615cba640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f661006df30 0x7f66101493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66164bb640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6610158520 0x7f66100cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66154b9640 1 -- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6610158520 msgr2=0x7f66100cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66154b9640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6610158520 0x7f66100cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66154b9640 1 -- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f661006df30 msgr2=0x7f66101493e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66154b9640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f661006df30 0x7f66101493e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66154b9640 1 -- 172.21.15.27:0/633423666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6610155570 con 0x7f66100aee10 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66164bb640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6610158520 0x7f66100cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f6615cba640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f661006df30 0x7f66101493e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66154b9640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 0x7f66100cd650 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f660000c4a0 tx=0x7f660000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66067fc640 1 -- 172.21.15.27:0/633423666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6600018070 con 0x7f66100aee10 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f66067fc640 1 -- 172.21.15.27:0/633423666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66000074e0 con 0x7f66100aee10 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.816+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66101557f0 con 0x7f66100aee10 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.817+0000 7f66067fc640 1 -- 172.21.15.27:0/633423666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6600004a20 con 0x7f66100aee10 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.817+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6610155db0 con 0x7f66100aee10 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.818+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6610067ff0 con 0x7f66100aee10 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.818+0000 7f66067fc640 1 -- 172.21.15.27:0/633423666 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f660001b070 con 0x7f66100aee10 2024-03-06T07:30:22.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.818+0000 7f66067fc640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65f406b500 0x7f65f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.819+0000 7f66067fc640 1 -- 172.21.15.27:0/633423666 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6600089420 con 0x7f66100aee10 2024-03-06T07:30:22.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.819+0000 7f6615cba640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65f406b500 0x7f65f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.819+0000 7f6615cba640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65f406b500 0x7f65f406d9c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f661006d830 tx=0x7f660c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:22.823+0000 7f66067fc640 1 -- 172.21.15.27:0/633423666 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6610155570 con 0x7f66100aee10 2024-03-06T07:30:23.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.006+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66100b31f0 con 0x7f66100aee10 2024-03-06T07:30:23.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.008+0000 7f66067fc640 1 -- 172.21.15.27:0/633423666 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f6600058330 con 0x7f66100aee10 2024-03-06T07:30:23.009 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:23.009 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":131,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:30:21.718596+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:20.059798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.689553+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:23.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65f406b500 msgr2=0x7f65f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65f406b500 0x7f65f406d9c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f661006d830 tx=0x7f660c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 msgr2=0x7f66100cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 0x7f66100cd650 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f660000c4a0 tx=0x7f660000c960 comp rx=0 tx=0).stop 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 shutdown_connections 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65f406b500 0x7f65f406d9c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6610158520 0x7f66100cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100aee10 0x7f66100cd650 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 --2- 172.21.15.27:0/633423666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f661006df30 0x7f66101493e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 >> 172.21.15.27:0/633423666 conn(0x7f6610066d00 msgr2=0x7f6610158c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 shutdown_connections 2024-03-06T07:30:23.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.010+0000 7f6617f1e640 1 -- 172.21.15.27:0/633423666 wait complete. 2024-03-06T07:30:23.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:30:23.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.131+0000 7fcf12981640 1 Processor -- start 2024-03-06T07:30:23.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.131+0000 7fcf12981640 1 -- start start 2024-03-06T07:30:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf12981640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 0x7fcf0c0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf12981640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcf0c15dfb0 0x7fcf0c0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf12981640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0c06f070 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf12981640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0c071940 con 0x7fcf0c15dfb0 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf12981640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c0b5db0 con 0x7fcf0c0b5840 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf10f1e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcf0c15dfb0 0x7fcf0c0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf10f1e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcf0c15dfb0 0x7fcf0c0b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55594/0 (socket says 172.21.15.27:55594) 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf10f1e640 1 -- 172.21.15.27:0/3249111610 learned_addr learned my addr 172.21.15.27:0/3249111610 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf0b7fe640 1 --2- 172.21.15.27:0/3249111610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 0x7fcf0c0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf10f1e640 1 -- 172.21.15.27:0/3249111610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 msgr2=0x7fcf0c0b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf10f1e640 1 --2- 172.21.15.27:0/3249111610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 0x7fcf0c0b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf10f1e640 1 -- 172.21.15.27:0/3249111610 >> v1:172.21.15.69:6789/0 conn(0x7fcf0c15dbd0 legacy=0x7fcf0c15ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.132+0000 7fcf10f1e640 1 -- 172.21.15.27:0/3249111610 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c071c00 con 0x7fcf0c15dfb0 2024-03-06T07:30:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf10f1e640 1 --2- 172.21.15.27:0/3249111610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dfb0 0x7fcf0c0b3ef0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fcf04009e50 tx=0x7fcf0402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8c64c8562a1a3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf0affd640 1 -- 172.21.15.27:0/3249111610 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0402cb00 con 0x7fcf0c15dfb0 2024-03-06T07:30:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf0affd640 1 -- 172.21.15.27:0/3249111610 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0402cca0 con 0x7fcf0c15dfb0 2024-03-06T07:30:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf0affd640 1 -- 172.21.15.27:0/3249111610 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04034610 con 0x7fcf0c15dfb0 2024-03-06T07:30:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf12981640 1 -- 172.21.15.27:0/3249111610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dfb0 msgr2=0x7fcf0c0b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf12981640 1 --2- 172.21.15.27:0/3249111610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dfb0 0x7fcf0c0b3ef0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fcf04009e50 tx=0x7fcf0402bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf12981640 1 -- 172.21.15.27:0/3249111610 shutdown_connections 2024-03-06T07:30:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf12981640 1 --2- 172.21.15.27:0/3249111610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dfb0 0x7fcf0c0b3ef0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf12981640 1 --2- 172.21.15.27:0/3249111610 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 0x7fcf0c0b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.133+0000 7fcf12981640 1 -- 172.21.15.27:0/3249111610 >> 172.21.15.27:0/3249111610 conn(0x7fcf0c0b26b0 msgr2=0x7fcf0c0b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.134+0000 7fcf12981640 1 -- 172.21.15.27:0/3249111610 shutdown_connections 2024-03-06T07:30:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.134+0000 7fcf12981640 1 -- 172.21.15.27:0/3249111610 wait complete. 2024-03-06T07:30:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.135+0000 7fcf12981640 1 Processor -- start 2024-03-06T07:30:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.135+0000 7fcf12981640 1 -- start start 2024-03-06T07:30:23.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf12981640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 0x7fcf0c14b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf12981640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dbd0 0x7fcf0c15e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf12981640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf0c15dfb0 0x7fcf0c1600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf12981640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c0726e0 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf12981640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c0721c0 con 0x7fcf0c15dfb0 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf12981640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c072860 con 0x7fcf0c0b5840 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dbd0 0x7fcf0c15e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dbd0 0x7fcf0c15e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55608/0 (socket says 172.21.15.27:55608) 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0b7fe640 1 -- 172.21.15.27:0/1406295084 learned_addr learned my addr 172.21.15.27:0/1406295084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf10f1e640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf0c15dfb0 0x7fcf0c1600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0bfff640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 0x7fcf0c14b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0b7fe640 1 -- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 msgr2=0x7fcf0c14b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0b7fe640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 0x7fcf0c14b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0b7fe640 1 -- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf0c15dfb0 msgr2=0x7fcf0c1600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0b7fe640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf0c15dfb0 0x7fcf0c1600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0b7fe640 1 -- 172.21.15.27:0/1406295084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c14b740 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.136+0000 7fcf0b7fe640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dbd0 0x7fcf0c15e7f0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000d620 tx=0x7fcf0000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.137+0000 7fcf08ff9640 1 -- 172.21.15.27:0/1406295084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0000b620 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.137+0000 7fcf08ff9640 1 -- 172.21.15.27:0/1406295084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf00019070 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.137+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c15ed30 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.137+0000 7fcf08ff9640 1 -- 172.21.15.27:0/1406295084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0000ba00 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.137+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0c15f2e0 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.138+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0c073380 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.138+0000 7fcf08ff9640 1 -- 172.21.15.27:0/1406295084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf0000bba0 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.139+0000 7fcf08ff9640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcef806b4d0 0x7fcef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.139+0000 7fcf08ff9640 1 -- 172.21.15.27:0/1406295084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf0008a430 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.139+0000 7fcf0bfff640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcef806b4d0 0x7fcef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.139+0000 7fcf0bfff640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcef806b4d0 0x7fcef806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fcefc000f60 tx=0x7fcefc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.143+0000 7fcf08ff9640 1 -- 172.21.15.27:0/1406295084 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf00058cf0 con 0x7fcf0c15dbd0 2024-03-06T07:30:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:23.369+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcf0c04f410 con 0x7fcf0c15dbd0 2024-03-06T07:30:24.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.379+0000 7fcf08ff9640 1 -- 172.21.15.27:0/1406295084 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v91) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcf0005cd90 con 0x7fcf0c15dbd0 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcef806b4d0 msgr2=0x7fcef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcef806b4d0 0x7fcef806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fcefc000f60 tx=0x7fcefc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dbd0 msgr2=0x7fcf0c15e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dbd0 0x7fcf0c15e7f0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000d620 tx=0x7fcf0000dae0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 shutdown_connections 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcef806b4d0 0x7fcef806d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf0c15dfb0 0x7fcf0c1600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf0c15dbd0 0x7fcf0c15e7f0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 --2- 172.21.15.27:0/1406295084 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf0c0b5840 0x7fcf0c14b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 >> 172.21.15.27:0/1406295084 conn(0x7fcf0c0b26b0 msgr2=0x7fcf0c162230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 shutdown_connections 2024-03-06T07:30:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.381+0000 7fcf12981640 1 -- 172.21.15.27:0/1406295084 wait complete. 2024-03-06T07:30:24.397 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:30:24.398 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:30:24.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.505+0000 7f82a4878640 1 Processor -- start 2024-03-06T07:30:24.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.505+0000 7f82a4878640 1 -- start start 2024-03-06T07:30:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.506+0000 7f82a4878640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f82a00b2d30 0x7f82a00b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.506+0000 7f82a4878640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00d4960 con 0x7f82a0071540 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.506+0000 7f82a4878640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00d36c0 con 0x7f82a00b7820 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.506+0000 7f82a4878640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a00d3230 con 0x7f82a00b2d30 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.506+0000 7f829e59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f82a0071540 0x7f82a0071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54670/0 (socket says 172.21.15.27:54670) 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.506+0000 7f829e59c640 1 -- 172.21.15.27:0/1623922665 learned_addr learned my addr 172.21.15.27:0/1623922665 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829dd9b640 1 --2- 172.21.15.27:0/1623922665 >> v2:172.21.15.69:3300/0 conn(0x7f82a00b2d30 0x7f82a00b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759179335 0 0) 0x7f82a00d4960 con 0x7f82a0071540 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f828c003520 con 0x7f82a0071540 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 365188038 0 0) 0x7f828c003520 con 0x7f82a0071540 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 >> v2:172.21.15.69:3300/0 conn(0x7f82a00b2d30 msgr2=0x7f82a00b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 --2- 172.21.15.27:0/1623922665 >> v2:172.21.15.69:3300/0 conn(0x7f82a00b2d30 0x7f82a00b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 >> v1:172.21.15.69:6789/0 conn(0x7f82a00b7820 legacy=0x7f82a00b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a00d2620 con 0x7f82a0071540 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f8290001670 con 0x7f82a0071540 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 >> v1:172.21.15.27:6789/0 conn(0x7f82a0071540 legacy=0x7f82a0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f828c0039b0 0x7f828c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f828c004800 0x7f828c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f828c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a00d3230 con 0x7f828c0051c0 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c00c060 con 0x7f828c0039b0 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828c00c260 con 0x7f828c004800 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829ed9d640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f828c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829e59c640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f828c0039b0 0x7f828c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.507+0000 7f829dd9b640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f828c004800 0x7f828c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.508+0000 7f829ed9d640 1 -- 172.21.15.27:0/1623922665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f828c004800 msgr2=0x7f828c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.508+0000 7f829ed9d640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f828c004800 0x7f828c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.508+0000 7f829ed9d640 1 -- 172.21.15.27:0/1623922665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f828c0039b0 msgr2=0x7f828c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.508+0000 7f829ed9d640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f828c0039b0 0x7f828c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.508+0000 7f829ed9d640 1 -- 172.21.15.27:0/1623922665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c003520 con 0x7f828c0051c0 2024-03-06T07:30:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.509+0000 7f829ed9d640 1 -- 172.21.15.27:0/1623922665 reap_dead start 2024-03-06T07:30:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.509+0000 7f829ed9d640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f828c00b9c0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f8298001e50 tx=0x7f828c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb0d19ffc61d64e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.509+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298001a70 con 0x7f828c0051c0 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.509+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8298003b90 con 0x7f828c0051c0 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.509+0000 7f829d59a640 1 -- 172.21.15.27:0/1623922665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82980050d0 con 0x7f828c0051c0 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.509+0000 7f82a4878640 1 -- 172.21.15.27:0/1623922665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 msgr2=0x7f828c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.509+0000 7f82a4878640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f828c00b9c0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f8298001e50 tx=0x7f828c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.510+0000 7f82a4878640 1 -- 172.21.15.27:0/1623922665 shutdown_connections 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.510+0000 7f82a4878640 1 --2- 172.21.15.27:0/1623922665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f828c00b9c0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.510+0000 7f82a4878640 1 -- 172.21.15.27:0/1623922665 >> 172.21.15.27:0/1623922665 conn(0x7f82a00d3d60 msgr2=0x7f82a00d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.510+0000 7f82a4878640 1 -- 172.21.15.27:0/1623922665 shutdown_connections 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.510+0000 7f82a4878640 1 -- 172.21.15.27:0/1623922665 wait complete. 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.511+0000 7f82a4878640 1 Processor -- start 2024-03-06T07:30:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.511+0000 7f82a4878640 1 -- start start 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f82a4878640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f82a01e8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f82a4878640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82a0159ac0 0x7f82a0155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f82a4878640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a0159f40 0x7f82a01560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f82a4878640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a00d2ec0 con 0x7f828c0051c0 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f82a4878640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a00d2be0 con 0x7f82a0159ac0 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f82a4878640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a00d22f0 con 0x7f82a0159f40 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f82a01e8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f82a01e8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56324/0 (socket says 172.21.15.27:56324) 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829e59c640 1 -- 172.21.15.27:0/445856716 learned_addr learned my addr 172.21.15.27:0/445856716 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829dd9b640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82a0159ac0 0x7f82a0155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829ed9d640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a0159f40 0x7f82a01560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829e59c640 1 -- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a0159f40 msgr2=0x7f82a01560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829e59c640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a0159f40 0x7f82a01560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829e59c640 1 -- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82a0159ac0 msgr2=0x7f82a0155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829e59c640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82a0159ac0 0x7f82a0155b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829e59c640 1 -- 172.21.15.27:0/445856716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a00cceb0 con 0x7f828c0051c0 2024-03-06T07:30:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f829e59c640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f82a01e8350 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f8290001050 tx=0x7f8290007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f8296ffd640 1 -- 172.21.15.27:0/445856716 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000f960 con 0x7f828c0051c0 2024-03-06T07:30:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.512+0000 7f8296ffd640 1 -- 172.21.15.27:0/445856716 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8290018070 con 0x7f828c0051c0 2024-03-06T07:30:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.513+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a00cd190 con 0x7f828c0051c0 2024-03-06T07:30:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.513+0000 7f8296ffd640 1 -- 172.21.15.27:0/445856716 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000fd80 con 0x7f828c0051c0 2024-03-06T07:30:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.513+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a00cd750 con 0x7f828c0051c0 2024-03-06T07:30:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.514+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a00d4be0 con 0x7f828c0051c0 2024-03-06T07:30:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.514+0000 7f8296ffd640 1 -- 172.21.15.27:0/445856716 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f829001d460 con 0x7f828c0051c0 2024-03-06T07:30:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.514+0000 7f8296ffd640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828006b500 0x7f828006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.515+0000 7f8296ffd640 1 -- 172.21.15.27:0/445856716 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f829008a300 con 0x7f828c0051c0 2024-03-06T07:30:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.515+0000 7f829dd9b640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828006b500 0x7f828006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.515+0000 7f829dd9b640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828006b500 0x7f828006d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f82a00cdf50 tx=0x7f8288008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.519+0000 7f8296ffd640 1 -- 172.21.15.27:0/445856716 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82a00d4be0 con 0x7f828c0051c0 2024-03-06T07:30:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:24.738+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f82a0072070 con 0x7f828c0051c0 2024-03-06T07:30:25.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.380+0000 7f8296ffd640 1 -- 172.21.15.27:0/445856716 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v92) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8290059100 con 0x7f828c0051c0 2024-03-06T07:30:25.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828006b500 msgr2=0x7f828006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828006b500 0x7f828006d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f82a00cdf50 tx=0x7f8288008040 comp rx=0 tx=0).stop 2024-03-06T07:30:25.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 msgr2=0x7f82a01e8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f82a01e8350 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f8290001050 tx=0x7f8290007720 comp rx=0 tx=0).stop 2024-03-06T07:30:25.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 shutdown_connections 2024-03-06T07:30:25.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828006b500 0x7f828006d9c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f82a0159f40 0x7f82a01560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82a0159ac0 0x7f82a0155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 --2- 172.21.15.27:0/445856716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f828c0051c0 0x7f82a01e8350 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 >> 172.21.15.27:0/445856716 conn(0x7f82a00d3d60 msgr2=0x7f82a00b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 shutdown_connections 2024-03-06T07:30:25.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.382+0000 7f82a4878640 1 -- 172.21.15.27:0/445856716 wait complete. 2024-03-06T07:30:25.395 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:30:25.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:25.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.501+0000 7f884e48a640 1 Processor -- start 2024-03-06T07:30:25.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.501+0000 7f884e48a640 1 -- start start 2024-03-06T07:30:25.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f884e48a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f884815df00 0x7f88480d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f884e48a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88480b95a0 0x7f88480b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f884e48a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88480cea30 con 0x7f88480b9a30 2024-03-06T07:30:25.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f884e48a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88480d2af0 con 0x7f88480b95a0 2024-03-06T07:30:25.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f884e48a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88480cf3c0 con 0x7f884815df00 2024-03-06T07:30:25.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f88477fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88480b95a0 0x7f88480b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f884ca27640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f88480b9a30 0x7f88480d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54684/0 (socket says 172.21.15.27:54684) 2024-03-06T07:30:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f884ca27640 1 -- 172.21.15.27:0/2432117638 learned_addr learned my addr 172.21.15.27:0/2432117638 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f8847fff640 1 --2- 172.21.15.27:0/2432117638 >> v2:172.21.15.69:3300/0 conn(0x7f884815df00 0x7f88480d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f88477fe640 1 -- 172.21.15.27:0/2432117638 >> v2:172.21.15.69:3300/0 conn(0x7f884815df00 msgr2=0x7f88480d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f88477fe640 1 --2- 172.21.15.27:0/2432117638 >> v2:172.21.15.69:3300/0 conn(0x7f884815df00 0x7f88480d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f88477fe640 1 -- 172.21.15.27:0/2432117638 >> v1:172.21.15.27:6789/0 conn(0x7f88480b9a30 legacy=0x7f88480d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f8846ffd640 1 -- 172.21.15.27:0/2432117638 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236029045 0 0) 0x7f88480cea30 con 0x7f88480b9a30 2024-03-06T07:30:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.502+0000 7f88477fe640 1 -- 172.21.15.27:0/2432117638 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88480cf0e0 con 0x7f88480b95a0 2024-03-06T07:30:25.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.503+0000 7f8847fff640 1 --2- 172.21.15.27:0/2432117638 >> v2:172.21.15.69:3300/0 conn(0x7f884815df00 0x7f88480d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f88477fe640 1 --2- 172.21.15.27:0/2432117638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88480b95a0 0x7f88480b5700 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f88480d11b0 tx=0x7f883802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966984537939e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f8846ffd640 1 -- 172.21.15.27:0/2432117638 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838035020 con 0x7f88480b95a0 2024-03-06T07:30:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f8846ffd640 1 -- 172.21.15.27:0/2432117638 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8838031830 con 0x7f88480b95a0 2024-03-06T07:30:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f8846ffd640 1 -- 172.21.15.27:0/2432117638 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838031c00 con 0x7f88480b95a0 2024-03-06T07:30:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f884e48a640 1 -- 172.21.15.27:0/2432117638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88480b95a0 msgr2=0x7f88480b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f884e48a640 1 --2- 172.21.15.27:0/2432117638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88480b95a0 0x7f88480b5700 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f88480d11b0 tx=0x7f883802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f884e48a640 1 -- 172.21.15.27:0/2432117638 shutdown_connections 2024-03-06T07:30:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f884e48a640 1 --2- 172.21.15.27:0/2432117638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88480b95a0 0x7f88480b5700 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f884e48a640 1 --2- 172.21.15.27:0/2432117638 >> v2:172.21.15.69:3300/0 conn(0x7f884815df00 0x7f88480d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f884e48a640 1 -- 172.21.15.27:0/2432117638 >> 172.21.15.27:0/2432117638 conn(0x7f8848071960 msgr2=0x7f8848071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f884e48a640 1 -- 172.21.15.27:0/2432117638 shutdown_connections 2024-03-06T07:30:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.504+0000 7f884e48a640 1 -- 172.21.15.27:0/2432117638 wait complete. 2024-03-06T07:30:25.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.506+0000 7f884e48a640 1 Processor -- start 2024-03-06T07:30:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.507+0000 7f884e48a640 1 -- start start 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f884e48a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88480b95a0 0x7f8848149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f884e48a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884815df00 0x7f88480c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f884e48a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88480c7cf0 0x7f88480c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f884e48a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88480cec70 con 0x7f884815df00 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f884e48a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88480ce750 con 0x7f88480c7cf0 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f884e48a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88480cedf0 con 0x7f88480b95a0 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f88477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884815df00 0x7f88480c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f88477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884815df00 0x7f88480c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56346/0 (socket says 172.21.15.27:56346) 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f88477fe640 1 -- 172.21.15.27:0/4173808976 learned_addr learned my addr 172.21.15.27:0/4173808976 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f8847fff640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88480b95a0 0x7f8848149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f884ca27640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88480c7cf0 0x7f88480c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f88477fe640 1 -- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88480b95a0 msgr2=0x7f8848149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f88477fe640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88480b95a0 0x7f8848149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f88477fe640 1 -- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88480c7cf0 msgr2=0x7f88480c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f88477fe640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88480c7cf0 0x7f88480c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f88477fe640 1 -- 172.21.15.27:0/4173808976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884815ebb0 con 0x7f884815df00 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f8847fff640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88480b95a0 0x7f8848149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.508+0000 7f884ca27640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88480c7cf0 0x7f88480c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.509+0000 7f88477fe640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884815df00 0x7f88480c77b0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f883802bee0 tx=0x7f88380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.509+0000 7f8844ff9640 1 -- 172.21.15.27:0/4173808976 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838041070 con 0x7f884815df00 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.509+0000 7f8844ff9640 1 -- 172.21.15.27:0/4173808976 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883803aa30 con 0x7f884815df00 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.509+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88480ca340 con 0x7f884815df00 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.509+0000 7f8844ff9640 1 -- 172.21.15.27:0/4173808976 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838035040 con 0x7f884815df00 2024-03-06T07:30:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.509+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88480ca900 con 0x7f884815df00 2024-03-06T07:30:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.510+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88480cab00 con 0x7f884815df00 2024-03-06T07:30:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.510+0000 7f8844ff9640 1 -- 172.21.15.27:0/4173808976 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8838038020 con 0x7f884815df00 2024-03-06T07:30:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.511+0000 7f8844ff9640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f883406b4d0 0x7f883406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.511+0000 7f8844ff9640 1 -- 172.21.15.27:0/4173808976 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88380ab960 con 0x7f884815df00 2024-03-06T07:30:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.511+0000 7f8847fff640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f883406b4d0 0x7f883406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.512+0000 7f8847fff640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f883406b4d0 0x7f883406d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f883c006fd0 tx=0x7f883c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.520+0000 7f8844ff9640 1 -- 172.21.15.27:0/4173808976 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88480cab00 con 0x7f884815df00 2024-03-06T07:30:25.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.741+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8848067ff0 con 0x7f884815df00 2024-03-06T07:30:25.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.742+0000 7f8844ff9640 1 -- 172.21.15.27:0/4173808976 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f883807a7e0 con 0x7f884815df00 2024-03-06T07:30:25.744 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:25.744 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/643534493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":643534493},{"type":"v1","addr":"172.21.15.69:6837","nonce":643534493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":7783,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3158495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3158495906},{"type":"v1","addr":"172.21.15.69:6835","nonce":3158495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":8670,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:24.739429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:25.744 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-06T07:30:25.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f883406b4d0 msgr2=0x7f883406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f883406b4d0 0x7f883406d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f883c006fd0 tx=0x7f883c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:25.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884815df00 msgr2=0x7f88480c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884815df00 0x7f88480c77b0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f883802bee0 tx=0x7f88380046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 shutdown_connections 2024-03-06T07:30:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f883406b4d0 0x7f883406d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88480c7cf0 0x7f88480c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f884815df00 0x7f88480c77b0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 --2- 172.21.15.27:0/4173808976 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88480b95a0 0x7f8848149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 >> 172.21.15.27:0/4173808976 conn(0x7f8848071960 msgr2=0x7f88480b8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 shutdown_connections 2024-03-06T07:30:25.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.744+0000 7f884e48a640 1 -- 172.21.15.27:0/4173808976 wait complete. 2024-03-06T07:30:25.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:25.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.867+0000 7f7f473bc640 1 Processor -- start 2024-03-06T07:30:25.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.867+0000 7f7f473bc640 1 -- start start 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f473bc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f400b95a0 0x7f7f400b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f473bc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f400b9a10 0x7f7f400b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f473bc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f400cea30 con 0x7f7f4015df00 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f473bc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f400cf3c0 con 0x7f7f400b95a0 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f473bc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f400cf0e0 con 0x7f7f400b9a10 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f44957640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f400b95a0 0x7f7f400b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f44957640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f400b95a0 0x7f7f400b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56354/0 (socket says 172.21.15.27:56354) 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f44957640 1 -- 172.21.15.27:0/3638724252 learned_addr learned my addr 172.21.15.27:0/3638724252 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f45959640 1 --2- 172.21.15.27:0/3638724252 >> v2:172.21.15.69:3300/0 conn(0x7f7f400b9a10 0x7f7f400b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f44957640 1 -- 172.21.15.27:0/3638724252 >> v2:172.21.15.69:3300/0 conn(0x7f7f400b9a10 msgr2=0x7f7f400b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f44957640 1 --2- 172.21.15.27:0/3638724252 >> v2:172.21.15.69:3300/0 conn(0x7f7f400b9a10 0x7f7f400b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f44957640 1 -- 172.21.15.27:0/3638724252 >> v1:172.21.15.69:6789/0 conn(0x7f7f4015df00 legacy=0x7f7f400d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f44957640 1 -- 172.21.15.27:0/3638724252 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f400ce470 con 0x7f7f400b95a0 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.868+0000 7f7f45959640 1 --2- 172.21.15.27:0/3638724252 >> v2:172.21.15.69:3300/0 conn(0x7f7f400b9a10 0x7f7f400b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.869+0000 7f7f44957640 1 --2- 172.21.15.27:0/3638724252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f400b95a0 0x7f7f400b5700 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f7f400d4d20 tx=0x7f7f3002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be5b606bc8da84c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.869+0000 7f7f37fff640 1 -- 172.21.15.27:0/3638724252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30034070 con 0x7f7f400b95a0 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.869+0000 7f7f37fff640 1 -- 172.21.15.27:0/3638724252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f30030860 con 0x7f7f400b95a0 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.869+0000 7f7f37fff640 1 -- 172.21.15.27:0/3638724252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30030c30 con 0x7f7f400b95a0 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.870+0000 7f7f473bc640 1 -- 172.21.15.27:0/3638724252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f400b95a0 msgr2=0x7f7f400b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.870+0000 7f7f473bc640 1 --2- 172.21.15.27:0/3638724252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f400b95a0 0x7f7f400b5700 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f7f400d4d20 tx=0x7f7f3002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.870+0000 7f7f473bc640 1 -- 172.21.15.27:0/3638724252 shutdown_connections 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.870+0000 7f7f473bc640 1 --2- 172.21.15.27:0/3638724252 >> v2:172.21.15.69:3300/0 conn(0x7f7f400b9a10 0x7f7f400b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.870+0000 7f7f473bc640 1 --2- 172.21.15.27:0/3638724252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f400b95a0 0x7f7f400b5700 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.870+0000 7f7f473bc640 1 -- 172.21.15.27:0/3638724252 >> 172.21.15.27:0/3638724252 conn(0x7f7f40071960 msgr2=0x7f7f40071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.870+0000 7f7f473bc640 1 -- 172.21.15.27:0/3638724252 shutdown_connections 2024-03-06T07:30:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.870+0000 7f7f473bc640 1 -- 172.21.15.27:0/3638724252 wait complete. 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.872+0000 7f7f473bc640 1 Processor -- start 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.872+0000 7f7f473bc640 1 -- start start 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.873+0000 7f7f473bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f400b95a0 0x7f7f4014b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.873+0000 7f7f473bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f400b9a10 0x7f7f4015e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.873+0000 7f7f473bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4015df00 0x7f7f400b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.873+0000 7f7f473bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f400cedf0 con 0x7f7f4015df00 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.873+0000 7f7f473bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f400cec70 con 0x7f7f400b95a0 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.873+0000 7f7f473bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f400d3220 con 0x7f7f400b9a10 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.873+0000 7f7f45959640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4015df00 0x7f7f400b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.873+0000 7f7f45959640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4015df00 0x7f7f400b4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56360/0 (socket says 172.21.15.27:56360) 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.873+0000 7f7f45959640 1 -- 172.21.15.27:0/3316842859 learned_addr learned my addr 172.21.15.27:0/3316842859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f45158640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f400b95a0 0x7f7f4014b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f44957640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f400b9a10 0x7f7f4015e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f45959640 1 -- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f400b9a10 msgr2=0x7f7f4015e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f45959640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f400b9a10 0x7f7f4015e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f45959640 1 -- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f400b95a0 msgr2=0x7f7f4014b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f45959640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f400b95a0 0x7f7f4014b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f45959640 1 -- 172.21.15.27:0/3316842859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4014b5d0 con 0x7f7f4015df00 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f45158640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f400b95a0 0x7f7f4014b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f45959640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4015df00 0x7f7f400b4d20 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009fa0 tx=0x7f7f3800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f35ffb640 1 -- 172.21.15.27:0/3316842859 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3801a070 con 0x7f7f4015df00 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f35ffb640 1 -- 172.21.15.27:0/3316842859 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f38015600 con 0x7f7f4015df00 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f401f71f0 con 0x7f7f4015df00 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f35ffb640 1 -- 172.21.15.27:0/3316842859 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f380159b0 con 0x7f7f4015df00 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.874+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f401f77b0 con 0x7f7f4015df00 2024-03-06T07:30:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.875+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f40067ff0 con 0x7f7f4015df00 2024-03-06T07:30:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.876+0000 7f7f35ffb640 1 -- 172.21.15.27:0/3316842859 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f3801d070 con 0x7f7f4015df00 2024-03-06T07:30:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.877+0000 7f7f35ffb640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f2406b4d0 0x7f7f2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.877+0000 7f7f35ffb640 1 -- 172.21.15.27:0/3316842859 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f3808a320 con 0x7f7f4015df00 2024-03-06T07:30:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.877+0000 7f7f45158640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f2406b4d0 0x7f7f2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.877+0000 7f7f45158640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f2406b4d0 0x7f7f2406d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c001660 tx=0x7f7f3c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:25.881+0000 7f7f35ffb640 1 -- 172.21.15.27:0/3316842859 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f4014b5d0 con 0x7f7f4015df00 2024-03-06T07:30:26.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.063+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f400cbe80 con 0x7f7f4015df00 2024-03-06T07:30:26.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.064+0000 7f7f35ffb640 1 -- 172.21.15.27:0/3316842859 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16607 (secure 0 0 0) 0x7f7f38010070 con 0x7f7f4015df00 2024-03-06T07:30:26.066 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:26.066 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:30:23.343374+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:20.059798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.689553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f2406b4d0 msgr2=0x7f7f2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f2406b4d0 0x7f7f2406d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c001660 tx=0x7f7f3c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4015df00 msgr2=0x7f7f400b4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4015df00 0x7f7f400b4d20 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009fa0 tx=0x7f7f3800d560 comp rx=0 tx=0).stop 2024-03-06T07:30:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 shutdown_connections 2024-03-06T07:30:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f2406b4d0 0x7f7f2406d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f4015df00 0x7f7f400b4d20 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f400b9a10 0x7f7f4015e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 --2- 172.21.15.27:0/3316842859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f400b95a0 0x7f7f4014b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 >> 172.21.15.27:0/3316842859 conn(0x7f7f40071960 msgr2=0x7f7f40071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 shutdown_connections 2024-03-06T07:30:26.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.066+0000 7f7f473bc640 1 -- 172.21.15.27:0/3316842859 wait complete. 2024-03-06T07:30:26.079 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:30:26.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.185+0000 7f0cfdaf4640 1 Processor -- start 2024-03-06T07:30:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.185+0000 7f0cfdaf4640 1 -- start start 2024-03-06T07:30:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.185+0000 7f0cfdaf4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0cf8071540 0x7f0cf80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.185+0000 7f0cfdaf4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cf80d3400 0x7f0cf8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.185+0000 7f0cfdaf4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf8071e60 con 0x7f0cf80b2b50 2024-03-06T07:30:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.185+0000 7f0cfdaf4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf80d2f70 con 0x7f0cf80d3400 2024-03-06T07:30:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.185+0000 7f0cfdaf4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf80d4be0 con 0x7f0cf8071540 2024-03-06T07:30:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.186+0000 7f0cf6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cf80d3400 0x7f0cf8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.186+0000 7f0cf6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cf80d3400 0x7f0cf8071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56374/0 (socket says 172.21.15.27:56374) 2024-03-06T07:30:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.186+0000 7f0cf6ffd640 1 -- 172.21.15.27:0/756048044 learned_addr learned my addr 172.21.15.27:0/756048044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.186+0000 7f0cf77fe640 1 --2- 172.21.15.27:0/756048044 >> v2:172.21.15.69:3300/0 conn(0x7f0cf8071540 0x7f0cf80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.186+0000 7f0cf6ffd640 1 -- 172.21.15.27:0/756048044 >> v2:172.21.15.69:3300/0 conn(0x7f0cf8071540 msgr2=0x7f0cf80b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.186+0000 7f0cf6ffd640 1 --2- 172.21.15.27:0/756048044 >> v2:172.21.15.69:3300/0 conn(0x7f0cf8071540 0x7f0cf80b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.186+0000 7f0cf6ffd640 1 -- 172.21.15.27:0/756048044 >> v1:172.21.15.69:6789/0 conn(0x7f0cf80b2b50 legacy=0x7f0cf80d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.186+0000 7f0cf6ffd640 1 -- 172.21.15.27:0/756048044 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80ce610 con 0x7f0cf80d3400 2024-03-06T07:30:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.187+0000 7f0cf6ffd640 1 --2- 172.21.15.27:0/756048044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80d3400 0x7f0cf8071920 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80d4da0 tx=0x7f0cec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ee29ab9e2ee239 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.187+0000 7f0cf67fc640 1 -- 172.21.15.27:0/756048044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec035020 con 0x7f0cf80d3400 2024-03-06T07:30:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.187+0000 7f0cf67fc640 1 -- 172.21.15.27:0/756048044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec031870 con 0x7f0cf80d3400 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.187+0000 7f0cf67fc640 1 -- 172.21.15.27:0/756048044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec031c40 con 0x7f0cf80d3400 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.188+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/756048044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80d3400 msgr2=0x7f0cf8071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.188+0000 7f0cfdaf4640 1 --2- 172.21.15.27:0/756048044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80d3400 0x7f0cf8071920 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80d4da0 tx=0x7f0cec02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.188+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/756048044 shutdown_connections 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.188+0000 7f0cfdaf4640 1 --2- 172.21.15.27:0/756048044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80d3400 0x7f0cf8071920 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.188+0000 7f0cfdaf4640 1 --2- 172.21.15.27:0/756048044 >> v2:172.21.15.69:3300/0 conn(0x7f0cf8071540 0x7f0cf80b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.188+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/756048044 >> 172.21.15.27:0/756048044 conn(0x7f0cf80d3ca0 msgr2=0x7f0cf80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.188+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/756048044 shutdown_connections 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.188+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/756048044 wait complete. 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.189+0000 7f0cfdaf4640 1 Processor -- start 2024-03-06T07:30:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.189+0000 7f0cfdaf4640 1 -- start start 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cfdaf4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cf8071540 0x7f0cf815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cfdaf4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80b2b50 0x7f0cf80b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cfdaf4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cf80d3400 0x7f0cf80ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cfdaf4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf80ce8f0 con 0x7f0cf80b2b50 2024-03-06T07:30:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cfdaf4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf80d2c00 con 0x7f0cf8071540 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cfdaf4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf815e370 con 0x7f0cf80d3400 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80b2b50 0x7f0cf80b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80b2b50 0x7f0cf80b1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56382/0 (socket says 172.21.15.27:56382) 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf6ffd640 1 -- 172.21.15.27:0/698987545 learned_addr learned my addr 172.21.15.27:0/698987545 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf77fe640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cf8071540 0x7f0cf815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf7fff640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cf80d3400 0x7f0cf80ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf6ffd640 1 -- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cf80d3400 msgr2=0x7f0cf80ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf6ffd640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cf80d3400 0x7f0cf80ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf6ffd640 1 -- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cf8071540 msgr2=0x7f0cf815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf6ffd640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cf8071540 0x7f0cf815c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.190+0000 7f0cf6ffd640 1 -- 172.21.15.27:0/698987545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80b2570 con 0x7f0cf80b2b50 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.191+0000 7f0cf6ffd640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80b2b50 0x7f0cf80b1c00 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f0cec02b950 tx=0x7f0cec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.191+0000 7f0cdffff640 1 -- 172.21.15.27:0/698987545 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec040070 con 0x7f0cf80b2b50 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.191+0000 7f0cdffff640 1 -- 172.21.15.27:0/698987545 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec03a420 con 0x7f0cf80b2b50 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.191+0000 7f0cdffff640 1 -- 172.21.15.27:0/698987545 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec035040 con 0x7f0cf80b2b50 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.191+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf80b3190 con 0x7f0cf80b2b50 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.191+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf80b3750 con 0x7f0cf80b2b50 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.192+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf8068700 con 0x7f0cf80b2b50 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.192+0000 7f0cdffff640 1 -- 172.21.15.27:0/698987545 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cec04b060 con 0x7f0cf80b2b50 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.193+0000 7f0cdffff640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ce406b4d0 0x7f0ce406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.193+0000 7f0cdffff640 1 -- 172.21.15.27:0/698987545 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cec0abf80 con 0x7f0cf80b2b50 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.193+0000 7f0cf77fe640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ce406b4d0 0x7f0ce406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.193+0000 7f0cf77fe640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ce406b4d0 0x7f0ce406d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800aac0 tx=0x7f0ce8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.202+0000 7f0cdffff640 1 -- 172.21.15.27:0/698987545 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cec07a840 con 0x7f0cf80b2b50 2024-03-06T07:30:26.313 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:30:26.312+0000 7fd868f37640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.383+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cf80b3920 con 0x7f0ce406b4d0 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.390+0000 7f0cdffff640 1 -- 172.21.15.27:0/698987545 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252482 (secure 0 0 0) 0x7f0cf80b3920 con 0x7f0ce406b4d0 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ce406b4d0 msgr2=0x7f0ce406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ce406b4d0 0x7f0ce406d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800aac0 tx=0x7f0ce8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80b2b50 msgr2=0x7f0cf80b1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80b2b50 0x7f0cf80b1c00 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f0cec02b950 tx=0x7f0cec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 shutdown_connections 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0ce406b4d0 0x7f0ce406d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0cf80d3400 0x7f0cf80ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cf80b2b50 0x7f0cf80b1c00 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 --2- 172.21.15.27:0/698987545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cf8071540 0x7f0cf815c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 >> 172.21.15.27:0/698987545 conn(0x7f0cf80d3ca0 msgr2=0x7f0cf815f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 shutdown_connections 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.394+0000 7f0cfdaf4640 1 -- 172.21.15.27:0/698987545 wait complete. 2024-03-06T07:30:26.406 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-03-06T07:30:26.097010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330092,"kb_used_data":14064,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644628,"statfs":{"total":773094113280,"available":772756099072,"internally_reserved":0,"allocated":14401536,"data_stored":6912956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.757546"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431259+0000","last_change":"2024-03-06T07:30:22.353235+0000","last_active":"2024-03-06T07:30:23.431259+0000","last_peered":"2024-03-06T07:30:23.431259+0000","last_clean":"2024-03-06T07:30:23.431259+0000","last_became_active":"2024-03-06T07:30:22.352948+0000","last_became_peered":"2024-03-06T07:30:22.352948+0000","last_unstale":"2024-03-06T07:30:23.431259+0000","last_undegraded":"2024-03-06T07:30:23.431259+0000","last_fullsized":"2024-03-06T07:30:23.431259+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.265886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346780+0000","last_change":"2024-03-06T07:30:22.362649+0000","last_active":"2024-03-06T07:30:23.346780+0000","last_peered":"2024-03-06T07:30:23.346780+0000","last_clean":"2024-03-06T07:30:23.346780+0000","last_became_active":"2024-03-06T07:30:22.362391+0000","last_became_peered":"2024-03-06T07:30:22.362391+0000","last_unstale":"2024-03-06T07:30:23.346780+0000","last_undegraded":"2024-03-06T07:30:23.346780+0000","last_fullsized":"2024-03-06T07:30:23.346780+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:03.613030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387133+0000","last_change":"2024-03-06T07:30:22.361827+0000","last_active":"2024-03-06T07:30:24.387133+0000","last_peered":"2024-03-06T07:30:24.387133+0000","last_clean":"2024-03-06T07:30:24.387133+0000","last_became_active":"2024-03-06T07:30:22.361513+0000","last_became_peered":"2024-03-06T07:30:22.361513+0000","last_unstale":"2024-03-06T07:30:24.387133+0000","last_undegraded":"2024-03-06T07:30:24.387133+0000","last_fullsized":"2024-03-06T07:30:24.387133+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:16.844706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431088+0000","last_change":"2024-03-06T07:30:22.362829+0000","last_active":"2024-03-06T07:30:23.431088+0000","last_peered":"2024-03-06T07:30:23.431088+0000","last_clean":"2024-03-06T07:30:23.431088+0000","last_became_active":"2024-03-06T07:30:22.362520+0000","last_became_peered":"2024-03-06T07:30:22.362520+0000","last_unstale":"2024-03-06T07:30:23.431088+0000","last_undegraded":"2024-03-06T07:30:23.431088+0000","last_fullsized":"2024-03-06T07:30:23.431088+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:03.592119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372164+0000","last_change":"2024-03-06T07:30:22.365431+0000","last_active":"2024-03-06T07:30:24.372164+0000","last_peered":"2024-03-06T07:30:24.372164+0000","last_clean":"2024-03-06T07:30:24.372164+0000","last_became_active":"2024-03-06T07:30:22.365173+0000","last_became_peered":"2024-03-06T07:30:22.365173+0000","last_unstale":"2024-03-06T07:30:24.372164+0000","last_undegraded":"2024-03-06T07:30:24.372164+0000","last_fullsized":"2024-03-06T07:30:24.372164+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:42.135831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374756+0000","last_change":"2024-03-06T07:30:22.356741+0000","last_active":"2024-03-06T07:30:24.374756+0000","last_peered":"2024-03-06T07:30:24.374756+0000","last_clean":"2024-03-06T07:30:24.374756+0000","last_became_active":"2024-03-06T07:30:22.355933+0000","last_became_peered":"2024-03-06T07:30:22.355933+0000","last_unstale":"2024-03-06T07:30:24.374756+0000","last_undegraded":"2024-03-06T07:30:24.374756+0000","last_fullsized":"2024-03-06T07:30:24.374756+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:20.362560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374487+0000","last_change":"2024-03-06T07:30:22.360723+0000","last_active":"2024-03-06T07:30:24.374487+0000","last_peered":"2024-03-06T07:30:24.374487+0000","last_clean":"2024-03-06T07:30:24.374487+0000","last_became_active":"2024-03-06T07:30:22.360423+0000","last_became_peered":"2024-03-06T07:30:22.360423+0000","last_unstale":"2024-03-06T07:30:24.374487+0000","last_undegraded":"2024-03-06T07:30:24.374487+0000","last_fullsized":"2024-03-06T07:30:24.374487+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:05.146007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387316+0000","last_change":"2024-03-06T07:30:22.367831+0000","last_active":"2024-03-06T07:30:24.387316+0000","last_peered":"2024-03-06T07:30:24.387316+0000","last_clean":"2024-03-06T07:30:24.387316+0000","last_became_active":"2024-03-06T07:30:22.367173+0000","last_became_peered":"2024-03-06T07:30:22.367173+0000","last_unstale":"2024-03-06T07:30:24.387316+0000","last_undegraded":"2024-03-06T07:30:24.387316+0000","last_fullsized":"2024-03-06T07:30:24.387316+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:54.397591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387195+0000","last_change":"2024-03-06T07:30:22.365076+0000","last_active":"2024-03-06T07:30:24.387195+0000","last_peered":"2024-03-06T07:30:24.387195+0000","last_clean":"2024-03-06T07:30:24.387195+0000","last_became_active":"2024-03-06T07:30:22.364772+0000","last_became_peered":"2024-03-06T07:30:22.364772+0000","last_unstale":"2024-03-06T07:30:24.387195+0000","last_undegraded":"2024-03-06T07:30:24.387195+0000","last_fullsized":"2024-03-06T07:30:24.387195+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:00.645432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346922+0000","last_change":"2024-03-06T07:30:22.360201+0000","last_active":"2024-03-06T07:30:23.346922+0000","last_peered":"2024-03-06T07:30:23.346922+0000","last_clean":"2024-03-06T07:30:23.346922+0000","last_became_active":"2024-03-06T07:30:22.359914+0000","last_became_peered":"2024-03-06T07:30:22.359914+0000","last_unstale":"2024-03-06T07:30:23.346922+0000","last_undegraded":"2024-03-06T07:30:23.346922+0000","last_fullsized":"2024-03-06T07:30:23.346922+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:26.283295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884084+0000","last_change":"2024-03-06T07:30:22.366235+0000","last_active":"2024-03-06T07:30:24.884084+0000","last_peered":"2024-03-06T07:30:24.884084+0000","last_clean":"2024-03-06T07:30:24.884084+0000","last_became_active":"2024-03-06T07:30:22.365937+0000","last_became_peered":"2024-03-06T07:30:22.365937+0000","last_unstale":"2024-03-06T07:30:24.884084+0000","last_undegraded":"2024-03-06T07:30:24.884084+0000","last_fullsized":"2024-03-06T07:30:24.884084+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:51.646354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387174+0000","last_change":"2024-03-06T07:30:22.364672+0000","last_active":"2024-03-06T07:30:24.387174+0000","last_peered":"2024-03-06T07:30:24.387174+0000","last_clean":"2024-03-06T07:30:24.387174+0000","last_became_active":"2024-03-06T07:30:22.364328+0000","last_became_peered":"2024-03-06T07:30:22.364328+0000","last_unstale":"2024-03-06T07:30:24.387174+0000","last_undegraded":"2024-03-06T07:30:24.387174+0000","last_fullsized":"2024-03-06T07:30:24.387174+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:42.982021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431114+0000","last_change":"2024-03-06T07:30:22.355786+0000","last_active":"2024-03-06T07:30:23.431114+0000","last_peered":"2024-03-06T07:30:23.431114+0000","last_clean":"2024-03-06T07:30:23.431114+0000","last_became_active":"2024-03-06T07:30:22.355486+0000","last_became_peered":"2024-03-06T07:30:22.355486+0000","last_unstale":"2024-03-06T07:30:23.431114+0000","last_undegraded":"2024-03-06T07:30:23.431114+0000","last_fullsized":"2024-03-06T07:30:23.431114+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:15.086879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346495+0000","last_change":"2024-03-06T07:30:22.368385+0000","last_active":"2024-03-06T07:30:23.346495+0000","last_peered":"2024-03-06T07:30:23.346495+0000","last_clean":"2024-03-06T07:30:23.346495+0000","last_became_active":"2024-03-06T07:30:22.367547+0000","last_became_peered":"2024-03-06T07:30:22.367547+0000","last_unstale":"2024-03-06T07:30:23.346495+0000","last_undegraded":"2024-03-06T07:30:23.346495+0000","last_fullsized":"2024-03-06T07:30:23.346495+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:01.803277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.430962+0000","last_change":"2024-03-06T07:30:22.360407+0000","last_active":"2024-03-06T07:30:23.430962+0000","last_peered":"2024-03-06T07:30:23.430962+0000","last_clean":"2024-03-06T07:30:23.430962+0000","last_became_active":"2024-03-06T07:30:22.360125+0000","last_became_peered":"2024-03-06T07:30:22.360125+0000","last_unstale":"2024-03-06T07:30:23.430962+0000","last_undegraded":"2024-03-06T07:30:23.430962+0000","last_fullsized":"2024-03-06T07:30:23.430962+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:05.135405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349733+0000","last_change":"2024-03-06T07:30:22.355315+0000","last_active":"2024-03-06T07:30:23.349733+0000","last_peered":"2024-03-06T07:30:23.349733+0000","last_clean":"2024-03-06T07:30:23.349733+0000","last_became_active":"2024-03-06T07:30:22.355019+0000","last_became_peered":"2024-03-06T07:30:22.355019+0000","last_unstale":"2024-03-06T07:30:23.349733+0000","last_undegraded":"2024-03-06T07:30:23.349733+0000","last_fullsized":"2024-03-06T07:30:23.349733+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:17.390996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374926+0000","last_change":"2024-03-06T07:30:22.360820+0000","last_active":"2024-03-06T07:30:24.374926+0000","last_peered":"2024-03-06T07:30:24.374926+0000","last_clean":"2024-03-06T07:30:24.374926+0000","last_became_active":"2024-03-06T07:30:22.360496+0000","last_became_peered":"2024-03-06T07:30:22.360496+0000","last_unstale":"2024-03-06T07:30:24.374926+0000","last_undegraded":"2024-03-06T07:30:24.374926+0000","last_fullsized":"2024-03-06T07:30:24.374926+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:25.165735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350060+0000","last_change":"2024-03-06T07:30:22.361923+0000","last_active":"2024-03-06T07:30:23.350060+0000","last_peered":"2024-03-06T07:30:23.350060+0000","last_clean":"2024-03-06T07:30:23.350060+0000","last_became_active":"2024-03-06T07:30:22.361640+0000","last_became_peered":"2024-03-06T07:30:22.361640+0000","last_unstale":"2024-03-06T07:30:23.350060+0000","last_undegraded":"2024-03-06T07:30:23.350060+0000","last_fullsized":"2024-03-06T07:30:23.350060+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:56.874819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884161+0000","last_change":"2024-03-06T07:30:22.363949+0000","last_active":"2024-03-06T07:30:24.884161+0000","last_peered":"2024-03-06T07:30:24.884161+0000","last_clean":"2024-03-06T07:30:24.884161+0000","last_became_active":"2024-03-06T07:30:22.363671+0000","last_became_peered":"2024-03-06T07:30:22.363671+0000","last_unstale":"2024-03-06T07:30:24.884161+0000","last_undegraded":"2024-03-06T07:30:24.884161+0000","last_fullsized":"2024-03-06T07:30:24.884161+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:45.367191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350027+0000","last_change":"2024-03-06T07:30:22.362617+0000","last_active":"2024-03-06T07:30:23.350027+0000","last_peered":"2024-03-06T07:30:23.350027+0000","last_clean":"2024-03-06T07:30:23.350027+0000","last_became_active":"2024-03-06T07:30:22.362338+0000","last_became_peered":"2024-03-06T07:30:22.362338+0000","last_unstale":"2024-03-06T07:30:23.350027+0000","last_undegraded":"2024-03-06T07:30:23.350027+0000","last_fullsized":"2024-03-06T07:30:23.350027+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:55.018122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371820+0000","last_change":"2024-03-06T07:30:22.364833+0000","last_active":"2024-03-06T07:30:24.371820+0000","last_peered":"2024-03-06T07:30:24.371820+0000","last_clean":"2024-03-06T07:30:24.371820+0000","last_became_active":"2024-03-06T07:30:22.364548+0000","last_became_peered":"2024-03-06T07:30:22.364548+0000","last_unstale":"2024-03-06T07:30:24.371820+0000","last_undegraded":"2024-03-06T07:30:24.371820+0000","last_fullsized":"2024-03-06T07:30:24.371820+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:56.874819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884006+0000","last_change":"2024-03-06T07:30:22.358843+0000","last_active":"2024-03-06T07:30:24.884006+0000","last_peered":"2024-03-06T07:30:24.884006+0000","last_clean":"2024-03-06T07:30:24.884006+0000","last_became_active":"2024-03-06T07:30:22.358546+0000","last_became_peered":"2024-03-06T07:30:22.358546+0000","last_unstale":"2024-03-06T07:30:24.884006+0000","last_undegraded":"2024-03-06T07:30:24.884006+0000","last_fullsized":"2024-03-06T07:30:24.884006+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:30.124102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387095+0000","last_change":"2024-03-06T07:30:22.366000+0000","last_active":"2024-03-06T07:30:24.387095+0000","last_peered":"2024-03-06T07:30:24.387095+0000","last_clean":"2024-03-06T07:30:24.387095+0000","last_became_active":"2024-03-06T07:30:22.364173+0000","last_became_peered":"2024-03-06T07:30:22.364173+0000","last_unstale":"2024-03-06T07:30:24.387095+0000","last_undegraded":"2024-03-06T07:30:24.387095+0000","last_fullsized":"2024-03-06T07:30:24.387095+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:20.362560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.883947+0000","last_change":"2024-03-06T07:30:22.364682+0000","last_active":"2024-03-06T07:30:24.883947+0000","last_peered":"2024-03-06T07:30:24.883947+0000","last_clean":"2024-03-06T07:30:24.883947+0000","last_became_active":"2024-03-06T07:30:22.364362+0000","last_became_peered":"2024-03-06T07:30:22.364362+0000","last_unstale":"2024-03-06T07:30:24.883947+0000","last_undegraded":"2024-03-06T07:30:24.883947+0000","last_fullsized":"2024-03-06T07:30:24.883947+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:38.013591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.347068+0000","last_change":"2024-03-06T07:30:22.361478+0000","last_active":"2024-03-06T07:30:23.347068+0000","last_peered":"2024-03-06T07:30:23.347068+0000","last_clean":"2024-03-06T07:30:23.347068+0000","last_became_active":"2024-03-06T07:30:22.361151+0000","last_became_peered":"2024-03-06T07:30:22.361151+0000","last_unstale":"2024-03-06T07:30:23.347068+0000","last_undegraded":"2024-03-06T07:30:23.347068+0000","last_fullsized":"2024-03-06T07:30:23.347068+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:58.961847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350100+0000","last_change":"2024-03-06T07:30:22.366826+0000","last_active":"2024-03-06T07:30:23.350100+0000","last_peered":"2024-03-06T07:30:23.350100+0000","last_clean":"2024-03-06T07:30:23.350100+0000","last_became_active":"2024-03-06T07:30:22.366560+0000","last_became_peered":"2024-03-06T07:30:22.366560+0000","last_unstale":"2024-03-06T07:30:23.350100+0000","last_undegraded":"2024-03-06T07:30:23.350100+0000","last_fullsized":"2024-03-06T07:30:23.350100+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:42.135831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431119+0000","last_change":"2024-03-06T07:30:22.355315+0000","last_active":"2024-03-06T07:30:23.431119+0000","last_peered":"2024-03-06T07:30:23.431119+0000","last_clean":"2024-03-06T07:30:23.431119+0000","last_became_active":"2024-03-06T07:30:22.354999+0000","last_became_peered":"2024-03-06T07:30:22.354999+0000","last_unstale":"2024-03-06T07:30:23.431119+0000","last_undegraded":"2024-03-06T07:30:23.431119+0000","last_fullsized":"2024-03-06T07:30:23.431119+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:26.564855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352848+0000","last_change":"2024-03-06T07:30:22.363204+0000","last_active":"2024-03-06T07:30:23.352848+0000","last_peered":"2024-03-06T07:30:23.352848+0000","last_clean":"2024-03-06T07:30:23.352848+0000","last_became_active":"2024-03-06T07:30:22.362848+0000","last_became_peered":"2024-03-06T07:30:22.362848+0000","last_unstale":"2024-03-06T07:30:23.352848+0000","last_undegraded":"2024-03-06T07:30:23.352848+0000","last_fullsized":"2024-03-06T07:30:23.352848+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:26.283295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431445+0000","last_change":"2024-03-06T07:30:22.366099+0000","last_active":"2024-03-06T07:30:23.431445+0000","last_peered":"2024-03-06T07:30:23.431445+0000","last_clean":"2024-03-06T07:30:23.431445+0000","last_became_active":"2024-03-06T07:30:22.365744+0000","last_became_peered":"2024-03-06T07:30:22.365744+0000","last_unstale":"2024-03-06T07:30:23.431445+0000","last_undegraded":"2024-03-06T07:30:23.431445+0000","last_fullsized":"2024-03-06T07:30:23.431445+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:32.470076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.430942+0000","last_change":"2024-03-06T07:30:22.357198+0000","last_active":"2024-03-06T07:30:23.430942+0000","last_peered":"2024-03-06T07:30:23.430942+0000","last_clean":"2024-03-06T07:30:23.430942+0000","last_became_active":"2024-03-06T07:30:22.356926+0000","last_became_peered":"2024-03-06T07:30:22.356926+0000","last_unstale":"2024-03-06T07:30:23.430942+0000","last_undegraded":"2024-03-06T07:30:23.430942+0000","last_fullsized":"2024-03-06T07:30:23.430942+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:43.954181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372235+0000","last_change":"2024-03-06T07:30:22.366699+0000","last_active":"2024-03-06T07:30:24.372235+0000","last_peered":"2024-03-06T07:30:24.372235+0000","last_clean":"2024-03-06T07:30:24.372235+0000","last_became_active":"2024-03-06T07:30:22.366429+0000","last_became_peered":"2024-03-06T07:30:22.366429+0000","last_unstale":"2024-03-06T07:30:24.372235+0000","last_undegraded":"2024-03-06T07:30:24.372235+0000","last_fullsized":"2024-03-06T07:30:24.372235+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:25.419177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374445+0000","last_change":"2024-03-06T07:30:22.356487+0000","last_active":"2024-03-06T07:30:24.374445+0000","last_peered":"2024-03-06T07:30:24.374445+0000","last_clean":"2024-03-06T07:30:24.374445+0000","last_became_active":"2024-03-06T07:30:22.356162+0000","last_became_peered":"2024-03-06T07:30:22.356162+0000","last_unstale":"2024-03-06T07:30:24.374445+0000","last_undegraded":"2024-03-06T07:30:24.374445+0000","last_fullsized":"2024-03-06T07:30:24.374445+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:44.856072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350204+0000","last_change":"2024-03-06T07:30:22.364976+0000","last_active":"2024-03-06T07:30:23.350204+0000","last_peered":"2024-03-06T07:30:23.350204+0000","last_clean":"2024-03-06T07:30:23.350204+0000","last_became_active":"2024-03-06T07:30:22.364725+0000","last_became_peered":"2024-03-06T07:30:22.364725+0000","last_unstale":"2024-03-06T07:30:23.350204+0000","last_undegraded":"2024-03-06T07:30:23.350204+0000","last_fullsized":"2024-03-06T07:30:23.350204+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:25.419177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"131'4","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372163+0000","last_change":"2024-03-06T07:30:21.356722+0000","last_active":"2024-03-06T07:30:24.372163+0000","last_peered":"2024-03-06T07:30:24.372163+0000","last_clean":"2024-03-06T07:30:24.372163+0000","last_became_active":"2024-03-06T07:30:21.356457+0000","last_became_peered":"2024-03-06T07:30:21.356457+0000","last_unstale":"2024-03-06T07:30:24.372163+0000","last_undegraded":"2024-03-06T07:30:24.372163+0000","last_fullsized":"2024-03-06T07:30:24.372163+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:34.007530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387212+0000","last_change":"2024-03-06T07:30:22.367501+0000","last_active":"2024-03-06T07:30:24.387212+0000","last_peered":"2024-03-06T07:30:24.387212+0000","last_clean":"2024-03-06T07:30:24.387212+0000","last_became_active":"2024-03-06T07:30:22.367131+0000","last_became_peered":"2024-03-06T07:30:22.367131+0000","last_unstale":"2024-03-06T07:30:24.387212+0000","last_undegraded":"2024-03-06T07:30:24.387212+0000","last_fullsized":"2024-03-06T07:30:24.387212+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:35.014807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352689+0000","last_change":"2024-03-06T07:30:21.362891+0000","last_active":"2024-03-06T07:30:23.352689+0000","last_peered":"2024-03-06T07:30:23.352689+0000","last_clean":"2024-03-06T07:30:23.352689+0000","last_became_active":"2024-03-06T07:30:21.362627+0000","last_became_peered":"2024-03-06T07:30:21.362627+0000","last_unstale":"2024-03-06T07:30:23.352689+0000","last_undegraded":"2024-03-06T07:30:23.352689+0000","last_fullsized":"2024-03-06T07:30:23.352689+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:25.557578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372282+0000","last_change":"2024-03-06T07:30:22.364459+0000","last_active":"2024-03-06T07:30:24.372282+0000","last_peered":"2024-03-06T07:30:24.372282+0000","last_clean":"2024-03-06T07:30:24.372282+0000","last_became_active":"2024-03-06T07:30:22.364191+0000","last_became_peered":"2024-03-06T07:30:22.364191+0000","last_unstale":"2024-03-06T07:30:24.372282+0000","last_undegraded":"2024-03-06T07:30:24.372282+0000","last_fullsized":"2024-03-06T07:30:24.372282+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:13.440811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884193+0000","last_change":"2024-03-06T07:30:21.358292+0000","last_active":"2024-03-06T07:30:24.884193+0000","last_peered":"2024-03-06T07:30:24.884193+0000","last_clean":"2024-03-06T07:30:24.884193+0000","last_became_active":"2024-03-06T07:30:21.358003+0000","last_became_peered":"2024-03-06T07:30:21.358003+0000","last_unstale":"2024-03-06T07:30:24.884193+0000","last_undegraded":"2024-03-06T07:30:24.884193+0000","last_fullsized":"2024-03-06T07:30:24.884193+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:52.695460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371760+0000","last_change":"2024-03-06T07:30:22.359793+0000","last_active":"2024-03-06T07:30:24.371760+0000","last_peered":"2024-03-06T07:30:24.371760+0000","last_clean":"2024-03-06T07:30:24.371760+0000","last_became_active":"2024-03-06T07:30:22.359513+0000","last_became_peered":"2024-03-06T07:30:22.359513+0000","last_unstale":"2024-03-06T07:30:24.371760+0000","last_undegraded":"2024-03-06T07:30:24.371760+0000","last_fullsized":"2024-03-06T07:30:24.371760+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:55.018122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346809+0000","last_change":"2024-03-06T07:30:21.360537+0000","last_active":"2024-03-06T07:30:23.346809+0000","last_peered":"2024-03-06T07:30:23.346809+0000","last_clean":"2024-03-06T07:30:23.346809+0000","last_became_active":"2024-03-06T07:30:21.359798+0000","last_became_peered":"2024-03-06T07:30:21.359798+0000","last_unstale":"2024-03-06T07:30:23.346809+0000","last_undegraded":"2024-03-06T07:30:23.346809+0000","last_fullsized":"2024-03-06T07:30:23.346809+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:35.594372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349748+0000","last_change":"2024-03-06T07:30:22.362520+0000","last_active":"2024-03-06T07:30:23.349748+0000","last_peered":"2024-03-06T07:30:23.349748+0000","last_clean":"2024-03-06T07:30:23.349748+0000","last_became_active":"2024-03-06T07:30:22.362258+0000","last_became_peered":"2024-03-06T07:30:22.362258+0000","last_unstale":"2024-03-06T07:30:23.349748+0000","last_undegraded":"2024-03-06T07:30:23.349748+0000","last_fullsized":"2024-03-06T07:30:23.349748+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:20.391472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346528+0000","last_change":"2024-03-06T07:30:21.357328+0000","last_active":"2024-03-06T07:30:23.346528+0000","last_peered":"2024-03-06T07:30:23.346528+0000","last_clean":"2024-03-06T07:30:23.346528+0000","last_became_active":"2024-03-06T07:30:21.357066+0000","last_became_peered":"2024-03-06T07:30:21.357066+0000","last_unstale":"2024-03-06T07:30:23.346528+0000","last_undegraded":"2024-03-06T07:30:23.346528+0000","last_fullsized":"2024-03-06T07:30:23.346528+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.806387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374639+0000","last_change":"2024-03-06T07:30:22.357870+0000","last_active":"2024-03-06T07:30:24.374639+0000","last_peered":"2024-03-06T07:30:24.374639+0000","last_clean":"2024-03-06T07:30:24.374639+0000","last_became_active":"2024-03-06T07:30:22.357613+0000","last_became_peered":"2024-03-06T07:30:22.357613+0000","last_unstale":"2024-03-06T07:30:24.374639+0000","last_undegraded":"2024-03-06T07:30:24.374639+0000","last_fullsized":"2024-03-06T07:30:24.374639+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:38.023851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349839+0000","last_change":"2024-03-06T07:30:21.362638+0000","last_active":"2024-03-06T07:30:23.349839+0000","last_peered":"2024-03-06T07:30:23.349839+0000","last_clean":"2024-03-06T07:30:23.349839+0000","last_became_active":"2024-03-06T07:30:21.362325+0000","last_became_peered":"2024-03-06T07:30:21.362325+0000","last_unstale":"2024-03-06T07:30:23.349839+0000","last_undegraded":"2024-03-06T07:30:23.349839+0000","last_fullsized":"2024-03-06T07:30:23.349839+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:39.237665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387288+0000","last_change":"2024-03-06T07:30:22.368329+0000","last_active":"2024-03-06T07:30:24.387288+0000","last_peered":"2024-03-06T07:30:24.387288+0000","last_clean":"2024-03-06T07:30:24.387288+0000","last_became_active":"2024-03-06T07:30:22.367042+0000","last_became_peered":"2024-03-06T07:30:22.367042+0000","last_unstale":"2024-03-06T07:30:24.387288+0000","last_undegraded":"2024-03-06T07:30:24.387288+0000","last_fullsized":"2024-03-06T07:30:24.387288+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:25.239461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374410+0000","last_change":"2024-03-06T07:30:21.354843+0000","last_active":"2024-03-06T07:30:24.374410+0000","last_peered":"2024-03-06T07:30:24.374410+0000","last_clean":"2024-03-06T07:30:24.374410+0000","last_became_active":"2024-03-06T07:30:21.354561+0000","last_became_peered":"2024-03-06T07:30:21.354561+0000","last_unstale":"2024-03-06T07:30:24.374410+0000","last_undegraded":"2024-03-06T07:30:24.374410+0000","last_fullsized":"2024-03-06T07:30:24.374410+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:31.246161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346473+0000","last_change":"2024-03-06T07:30:22.364106+0000","last_active":"2024-03-06T07:30:23.346473+0000","last_peered":"2024-03-06T07:30:23.346473+0000","last_clean":"2024-03-06T07:30:23.346473+0000","last_became_active":"2024-03-06T07:30:22.363504+0000","last_became_peered":"2024-03-06T07:30:22.363504+0000","last_unstale":"2024-03-06T07:30:23.346473+0000","last_undegraded":"2024-03-06T07:30:23.346473+0000","last_fullsized":"2024-03-06T07:30:23.346473+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:32.470076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346552+0000","last_change":"2024-03-06T07:30:21.353154+0000","last_active":"2024-03-06T07:30:23.346552+0000","last_peered":"2024-03-06T07:30:23.346552+0000","last_clean":"2024-03-06T07:30:23.346552+0000","last_became_active":"2024-03-06T07:30:21.352853+0000","last_became_peered":"2024-03-06T07:30:21.352853+0000","last_unstale":"2024-03-06T07:30:23.346552+0000","last_undegraded":"2024-03-06T07:30:23.346552+0000","last_fullsized":"2024-03-06T07:30:23.346552+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:18.752403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431311+0000","last_change":"2024-03-06T07:30:22.358294+0000","last_active":"2024-03-06T07:30:23.431311+0000","last_peered":"2024-03-06T07:30:23.431311+0000","last_clean":"2024-03-06T07:30:23.431311+0000","last_became_active":"2024-03-06T07:30:22.357962+0000","last_became_peered":"2024-03-06T07:30:22.357962+0000","last_unstale":"2024-03-06T07:30:23.431311+0000","last_undegraded":"2024-03-06T07:30:23.431311+0000","last_fullsized":"2024-03-06T07:30:23.431311+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:23.195487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352878+0000","last_change":"2024-03-06T07:30:21.367059+0000","last_active":"2024-03-06T07:30:23.352878+0000","last_peered":"2024-03-06T07:30:23.352878+0000","last_clean":"2024-03-06T07:30:23.352878+0000","last_became_active":"2024-03-06T07:30:21.366691+0000","last_became_peered":"2024-03-06T07:30:21.366691+0000","last_unstale":"2024-03-06T07:30:23.352878+0000","last_undegraded":"2024-03-06T07:30:23.352878+0000","last_fullsized":"2024-03-06T07:30:23.352878+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:30.332872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352770+0000","last_change":"2024-03-06T07:30:22.372180+0000","last_active":"2024-03-06T07:30:23.352770+0000","last_peered":"2024-03-06T07:30:23.352770+0000","last_clean":"2024-03-06T07:30:23.352770+0000","last_became_active":"2024-03-06T07:30:22.371861+0000","last_became_peered":"2024-03-06T07:30:22.371861+0000","last_unstale":"2024-03-06T07:30:23.352770+0000","last_undegraded":"2024-03-06T07:30:23.352770+0000","last_fullsized":"2024-03-06T07:30:23.352770+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:21.458852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350175+0000","last_change":"2024-03-06T07:30:21.363328+0000","last_active":"2024-03-06T07:30:23.350175+0000","last_peered":"2024-03-06T07:30:23.350175+0000","last_clean":"2024-03-06T07:30:23.350175+0000","last_became_active":"2024-03-06T07:30:21.362738+0000","last_became_peered":"2024-03-06T07:30:21.362738+0000","last_unstale":"2024-03-06T07:30:23.350175+0000","last_undegraded":"2024-03-06T07:30:23.350175+0000","last_fullsized":"2024-03-06T07:30:23.350175+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:24.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372309+0000","last_change":"2024-03-06T07:30:22.362197+0000","last_active":"2024-03-06T07:30:24.372309+0000","last_peered":"2024-03-06T07:30:24.372309+0000","last_clean":"2024-03-06T07:30:24.372309+0000","last_became_active":"2024-03-06T07:30:22.361837+0000","last_became_peered":"2024-03-06T07:30:22.361837+0000","last_unstale":"2024-03-06T07:30:24.372309+0000","last_undegraded":"2024-03-06T07:30:24.372309+0000","last_fullsized":"2024-03-06T07:30:24.372309+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:05.657203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"131'3","reported_seq":17,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352551+0000","last_change":"2024-03-06T07:30:21.366972+0000","last_active":"2024-03-06T07:30:23.352551+0000","last_peered":"2024-03-06T07:30:23.352551+0000","last_clean":"2024-03-06T07:30:23.352551+0000","last_became_active":"2024-03-06T07:30:21.366673+0000","last_became_peered":"2024-03-06T07:30:21.366673+0000","last_unstale":"2024-03-06T07:30:23.352551+0000","last_undegraded":"2024-03-06T07:30:23.352551+0000","last_fullsized":"2024-03-06T07:30:23.352551+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:14.079602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371972+0000","last_change":"2024-03-06T07:30:22.366102+0000","last_active":"2024-03-06T07:30:24.371972+0000","last_peered":"2024-03-06T07:30:24.371972+0000","last_clean":"2024-03-06T07:30:24.371972+0000","last_became_active":"2024-03-06T07:30:22.365791+0000","last_became_peered":"2024-03-06T07:30:22.365791+0000","last_unstale":"2024-03-06T07:30:24.371972+0000","last_undegraded":"2024-03-06T07:30:24.371972+0000","last_fullsized":"2024-03-06T07:30:24.371972+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:55.176659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"131'6","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387084+0000","last_change":"2024-03-06T07:30:21.356331+0000","last_active":"2024-03-06T07:30:24.387084+0000","last_peered":"2024-03-06T07:30:24.387084+0000","last_clean":"2024-03-06T07:30:24.387084+0000","last_became_active":"2024-03-06T07:30:21.355951+0000","last_became_peered":"2024-03-06T07:30:21.355951+0000","last_unstale":"2024-03-06T07:30:24.387084+0000","last_undegraded":"2024-03-06T07:30:24.387084+0000","last_fullsized":"2024-03-06T07:30:24.387084+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:20.451575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431362+0000","last_change":"2024-03-06T07:30:22.358246+0000","last_active":"2024-03-06T07:30:23.431362+0000","last_peered":"2024-03-06T07:30:23.431362+0000","last_clean":"2024-03-06T07:30:23.431362+0000","last_became_active":"2024-03-06T07:30:22.357980+0000","last_became_peered":"2024-03-06T07:30:22.357980+0000","last_unstale":"2024-03-06T07:30:23.431362+0000","last_undegraded":"2024-03-06T07:30:23.431362+0000","last_fullsized":"2024-03-06T07:30:23.431362+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:31.340149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431053+0000","last_change":"2024-03-06T07:30:21.361081+0000","last_active":"2024-03-06T07:30:23.431053+0000","last_peered":"2024-03-06T07:30:23.431053+0000","last_clean":"2024-03-06T07:30:23.431053+0000","last_became_active":"2024-03-06T07:30:21.360771+0000","last_became_peered":"2024-03-06T07:30:21.360771+0000","last_unstale":"2024-03-06T07:30:23.431053+0000","last_undegraded":"2024-03-06T07:30:23.431053+0000","last_fullsized":"2024-03-06T07:30:23.431053+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:53.587987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346448+0000","last_change":"2024-03-06T07:30:22.369634+0000","last_active":"2024-03-06T07:30:23.346448+0000","last_peered":"2024-03-06T07:30:23.346448+0000","last_clean":"2024-03-06T07:30:23.346448+0000","last_became_active":"2024-03-06T07:30:22.369006+0000","last_became_peered":"2024-03-06T07:30:22.369006+0000","last_unstale":"2024-03-06T07:30:23.346448+0000","last_undegraded":"2024-03-06T07:30:23.346448+0000","last_fullsized":"2024-03-06T07:30:23.346448+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:31.340149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346583+0000","last_change":"2024-03-06T07:30:21.354002+0000","last_active":"2024-03-06T07:30:23.346583+0000","last_peered":"2024-03-06T07:30:23.346583+0000","last_clean":"2024-03-06T07:30:23.346583+0000","last_became_active":"2024-03-06T07:30:21.353735+0000","last_became_peered":"2024-03-06T07:30:21.353735+0000","last_unstale":"2024-03-06T07:30:23.346583+0000","last_undegraded":"2024-03-06T07:30:23.346583+0000","last_fullsized":"2024-03-06T07:30:23.346583+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:35.543995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387033+0000","last_change":"2024-03-06T07:30:22.365206+0000","last_active":"2024-03-06T07:30:24.387033+0000","last_peered":"2024-03-06T07:30:24.387033+0000","last_clean":"2024-03-06T07:30:24.387033+0000","last_became_active":"2024-03-06T07:30:22.364923+0000","last_became_peered":"2024-03-06T07:30:22.364923+0000","last_unstale":"2024-03-06T07:30:24.387033+0000","last_undegraded":"2024-03-06T07:30:24.387033+0000","last_fullsized":"2024-03-06T07:30:24.387033+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:09.380233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"131'8","reported_seq":22,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387349+0000","last_change":"2024-03-06T07:30:21.361469+0000","last_active":"2024-03-06T07:30:24.387349+0000","last_peered":"2024-03-06T07:30:24.387349+0000","last_clean":"2024-03-06T07:30:24.387349+0000","last_became_active":"2024-03-06T07:30:21.361140+0000","last_became_peered":"2024-03-06T07:30:21.361140+0000","last_unstale":"2024-03-06T07:30:24.387349+0000","last_undegraded":"2024-03-06T07:30:24.387349+0000","last_fullsized":"2024-03-06T07:30:24.387349+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:25.276018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387150+0000","last_change":"2024-03-06T07:30:22.366631+0000","last_active":"2024-03-06T07:30:24.387150+0000","last_peered":"2024-03-06T07:30:24.387150+0000","last_clean":"2024-03-06T07:30:24.387150+0000","last_became_active":"2024-03-06T07:30:22.366379+0000","last_became_peered":"2024-03-06T07:30:22.366379+0000","last_unstale":"2024-03-06T07:30:24.387150+0000","last_undegraded":"2024-03-06T07:30:24.387150+0000","last_fullsized":"2024-03-06T07:30:24.387150+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:25.165735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352913+0000","last_change":"2024-03-06T07:30:21.358387+0000","last_active":"2024-03-06T07:30:23.352913+0000","last_peered":"2024-03-06T07:30:23.352913+0000","last_clean":"2024-03-06T07:30:23.352913+0000","last_became_active":"2024-03-06T07:30:21.358111+0000","last_became_peered":"2024-03-06T07:30:21.358111+0000","last_unstale":"2024-03-06T07:30:23.352913+0000","last_undegraded":"2024-03-06T07:30:23.352913+0000","last_fullsized":"2024-03-06T07:30:23.352913+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:22.188210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352844+0000","last_change":"2024-03-06T07:30:22.368146+0000","last_active":"2024-03-06T07:30:23.352844+0000","last_peered":"2024-03-06T07:30:23.352844+0000","last_clean":"2024-03-06T07:30:23.352844+0000","last_became_active":"2024-03-06T07:30:22.367832+0000","last_became_peered":"2024-03-06T07:30:22.367832+0000","last_unstale":"2024-03-06T07:30:23.352844+0000","last_undegraded":"2024-03-06T07:30:23.352844+0000","last_fullsized":"2024-03-06T07:30:23.352844+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:36.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884244+0000","last_change":"2024-03-06T07:30:21.363854+0000","last_active":"2024-03-06T07:30:24.884244+0000","last_peered":"2024-03-06T07:30:24.884244+0000","last_clean":"2024-03-06T07:30:24.884244+0000","last_became_active":"2024-03-06T07:30:21.363482+0000","last_became_peered":"2024-03-06T07:30:21.363482+0000","last_unstale":"2024-03-06T07:30:24.884244+0000","last_undegraded":"2024-03-06T07:30:24.884244+0000","last_fullsized":"2024-03-06T07:30:24.884244+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:07.251047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350076+0000","last_change":"2024-03-06T07:30:22.366657+0000","last_active":"2024-03-06T07:30:23.350076+0000","last_peered":"2024-03-06T07:30:23.350076+0000","last_clean":"2024-03-06T07:30:23.350076+0000","last_became_active":"2024-03-06T07:30:22.366369+0000","last_became_peered":"2024-03-06T07:30:22.366369+0000","last_unstale":"2024-03-06T07:30:23.350076+0000","last_undegraded":"2024-03-06T07:30:23.350076+0000","last_fullsized":"2024-03-06T07:30:23.350076+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:55.176659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374763+0000","last_change":"2024-03-06T07:30:21.359923+0000","last_active":"2024-03-06T07:30:24.374763+0000","last_peered":"2024-03-06T07:30:24.374763+0000","last_clean":"2024-03-06T07:30:24.374763+0000","last_became_active":"2024-03-06T07:30:21.359625+0000","last_became_peered":"2024-03-06T07:30:21.359625+0000","last_unstale":"2024-03-06T07:30:24.374763+0000","last_undegraded":"2024-03-06T07:30:24.374763+0000","last_fullsized":"2024-03-06T07:30:24.374763+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:02.605754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.347038+0000","last_change":"2024-03-06T07:30:22.363836+0000","last_active":"2024-03-06T07:30:23.347038+0000","last_peered":"2024-03-06T07:30:23.347038+0000","last_clean":"2024-03-06T07:30:23.347038+0000","last_became_active":"2024-03-06T07:30:22.363542+0000","last_became_peered":"2024-03-06T07:30:22.363542+0000","last_unstale":"2024-03-06T07:30:23.347038+0000","last_undegraded":"2024-03-06T07:30:23.347038+0000","last_fullsized":"2024-03-06T07:30:23.347038+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:36.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352767+0000","last_change":"2024-03-06T07:30:21.362922+0000","last_active":"2024-03-06T07:30:23.352767+0000","last_peered":"2024-03-06T07:30:23.352767+0000","last_clean":"2024-03-06T07:30:23.352767+0000","last_became_active":"2024-03-06T07:30:21.362655+0000","last_became_peered":"2024-03-06T07:30:21.362655+0000","last_unstale":"2024-03-06T07:30:23.352767+0000","last_undegraded":"2024-03-06T07:30:23.352767+0000","last_fullsized":"2024-03-06T07:30:23.352767+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:54.618131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431020+0000","last_change":"2024-03-06T07:30:22.351632+0000","last_active":"2024-03-06T07:30:23.431020+0000","last_peered":"2024-03-06T07:30:23.431020+0000","last_clean":"2024-03-06T07:30:23.431020+0000","last_became_active":"2024-03-06T07:30:22.351317+0000","last_became_peered":"2024-03-06T07:30:22.351317+0000","last_unstale":"2024-03-06T07:30:23.431020+0000","last_undegraded":"2024-03-06T07:30:23.431020+0000","last_fullsized":"2024-03-06T07:30:23.431020+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:31.183751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352785+0000","last_change":"2024-03-06T07:30:21.366829+0000","last_active":"2024-03-06T07:30:23.352785+0000","last_peered":"2024-03-06T07:30:23.352785+0000","last_clean":"2024-03-06T07:30:23.352785+0000","last_became_active":"2024-03-06T07:30:21.366553+0000","last_became_peered":"2024-03-06T07:30:21.366553+0000","last_unstale":"2024-03-06T07:30:23.352785+0000","last_undegraded":"2024-03-06T07:30:23.352785+0000","last_fullsized":"2024-03-06T07:30:23.352785+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:34.258609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374742+0000","last_change":"2024-03-06T07:30:22.359828+0000","last_active":"2024-03-06T07:30:24.374742+0000","last_peered":"2024-03-06T07:30:24.374742+0000","last_clean":"2024-03-06T07:30:24.374742+0000","last_became_active":"2024-03-06T07:30:22.359522+0000","last_became_peered":"2024-03-06T07:30:22.359522+0000","last_unstale":"2024-03-06T07:30:24.374742+0000","last_undegraded":"2024-03-06T07:30:24.374742+0000","last_fullsized":"2024-03-06T07:30:24.374742+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:09.380233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349866+0000","last_change":"2024-03-06T07:30:21.362535+0000","last_active":"2024-03-06T07:30:23.349866+0000","last_peered":"2024-03-06T07:30:23.349866+0000","last_clean":"2024-03-06T07:30:23.349866+0000","last_became_active":"2024-03-06T07:30:21.357316+0000","last_became_peered":"2024-03-06T07:30:21.357316+0000","last_unstale":"2024-03-06T07:30:23.349866+0000","last_undegraded":"2024-03-06T07:30:23.349866+0000","last_fullsized":"2024-03-06T07:30:23.349866+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:53.390314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346633+0000","last_change":"2024-03-06T07:30:22.357248+0000","last_active":"2024-03-06T07:30:23.346633+0000","last_peered":"2024-03-06T07:30:23.346633+0000","last_clean":"2024-03-06T07:30:23.346633+0000","last_became_active":"2024-03-06T07:30:22.356994+0000","last_became_peered":"2024-03-06T07:30:22.356994+0000","last_unstale":"2024-03-06T07:30:23.346633+0000","last_undegraded":"2024-03-06T07:30:23.346633+0000","last_fullsized":"2024-03-06T07:30:23.346633+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:32.253438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349665+0000","last_change":"2024-03-06T07:30:21.360791+0000","last_active":"2024-03-06T07:30:23.349665+0000","last_peered":"2024-03-06T07:30:23.349665+0000","last_clean":"2024-03-06T07:30:23.349665+0000","last_became_active":"2024-03-06T07:30:21.360285+0000","last_became_peered":"2024-03-06T07:30:21.360285+0000","last_unstale":"2024-03-06T07:30:23.349665+0000","last_undegraded":"2024-03-06T07:30:23.349665+0000","last_fullsized":"2024-03-06T07:30:23.349665+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:59.638155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371639+0000","last_change":"2024-03-06T07:30:22.366620+0000","last_active":"2024-03-06T07:30:24.371639+0000","last_peered":"2024-03-06T07:30:24.371639+0000","last_clean":"2024-03-06T07:30:24.371639+0000","last_became_active":"2024-03-06T07:30:22.366345+0000","last_became_peered":"2024-03-06T07:30:22.366345+0000","last_unstale":"2024-03-06T07:30:24.371639+0000","last_undegraded":"2024-03-06T07:30:24.371639+0000","last_fullsized":"2024-03-06T07:30:24.371639+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:17.390996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.883893+0000","last_change":"2024-03-06T07:30:21.356249+0000","last_active":"2024-03-06T07:30:24.883893+0000","last_peered":"2024-03-06T07:30:24.883893+0000","last_clean":"2024-03-06T07:30:24.883893+0000","last_became_active":"2024-03-06T07:30:21.355930+0000","last_became_peered":"2024-03-06T07:30:21.355930+0000","last_unstale":"2024-03-06T07:30:24.883893+0000","last_undegraded":"2024-03-06T07:30:24.883893+0000","last_fullsized":"2024-03-06T07:30:24.883893+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:24.632711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374949+0000","last_change":"2024-03-06T07:30:22.363606+0000","last_active":"2024-03-06T07:30:24.374949+0000","last_peered":"2024-03-06T07:30:24.374949+0000","last_clean":"2024-03-06T07:30:24.374949+0000","last_became_active":"2024-03-06T07:30:22.363190+0000","last_became_peered":"2024-03-06T07:30:22.363190+0000","last_unstale":"2024-03-06T07:30:24.374949+0000","last_undegraded":"2024-03-06T07:30:24.374949+0000","last_fullsized":"2024-03-06T07:30:24.374949+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:42.982021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431325+0000","last_change":"2024-03-06T07:30:21.365785+0000","last_active":"2024-03-06T07:30:23.431325+0000","last_peered":"2024-03-06T07:30:23.431325+0000","last_clean":"2024-03-06T07:30:23.431325+0000","last_became_active":"2024-03-06T07:30:21.365483+0000","last_became_peered":"2024-03-06T07:30:21.365483+0000","last_unstale":"2024-03-06T07:30:23.431325+0000","last_undegraded":"2024-03-06T07:30:23.431325+0000","last_fullsized":"2024-03-06T07:30:23.431325+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:57.830418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431267+0000","last_change":"2024-03-06T07:30:22.359840+0000","last_active":"2024-03-06T07:30:23.431267+0000","last_peered":"2024-03-06T07:30:23.431267+0000","last_clean":"2024-03-06T07:30:23.431267+0000","last_became_active":"2024-03-06T07:30:22.359489+0000","last_became_peered":"2024-03-06T07:30:22.359489+0000","last_unstale":"2024-03-06T07:30:23.431267+0000","last_undegraded":"2024-03-06T07:30:23.431267+0000","last_fullsized":"2024-03-06T07:30:23.431267+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:55.625408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"132'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.373736+0000","last_change":"2024-03-06T07:30:21.355058+0000","last_active":"2024-03-06T07:30:24.373736+0000","last_peered":"2024-03-06T07:30:24.373736+0000","last_clean":"2024-03-06T07:30:24.373736+0000","last_became_active":"2024-03-06T07:30:21.354785+0000","last_became_peered":"2024-03-06T07:30:21.354785+0000","last_unstale":"2024-03-06T07:30:24.373736+0000","last_undegraded":"2024-03-06T07:30:24.373736+0000","last_fullsized":"2024-03-06T07:30:24.373736+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:59.638155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346843+0000","last_change":"2024-03-06T07:30:22.361590+0000","last_active":"2024-03-06T07:30:23.346843+0000","last_peered":"2024-03-06T07:30:23.346843+0000","last_clean":"2024-03-06T07:30:23.346843+0000","last_became_active":"2024-03-06T07:30:22.361304+0000","last_became_peered":"2024-03-06T07:30:22.361304+0000","last_unstale":"2024-03-06T07:30:23.346843+0000","last_undegraded":"2024-03-06T07:30:23.346843+0000","last_fullsized":"2024-03-06T07:30:23.346843+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:21.458852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.347039+0000","last_change":"2024-03-06T07:30:21.357925+0000","last_active":"2024-03-06T07:30:23.347039+0000","last_peered":"2024-03-06T07:30:23.347039+0000","last_clean":"2024-03-06T07:30:23.347039+0000","last_became_active":"2024-03-06T07:30:21.357621+0000","last_became_peered":"2024-03-06T07:30:21.357621+0000","last_unstale":"2024-03-06T07:30:23.347039+0000","last_undegraded":"2024-03-06T07:30:23.347039+0000","last_fullsized":"2024-03-06T07:30:23.347039+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.851584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374816+0000","last_change":"2024-03-06T07:30:22.358228+0000","last_active":"2024-03-06T07:30:24.374816+0000","last_peered":"2024-03-06T07:30:24.374816+0000","last_clean":"2024-03-06T07:30:24.374816+0000","last_became_active":"2024-03-06T07:30:22.357597+0000","last_became_peered":"2024-03-06T07:30:22.357597+0000","last_unstale":"2024-03-06T07:30:24.374816+0000","last_undegraded":"2024-03-06T07:30:24.374816+0000","last_fullsized":"2024-03-06T07:30:24.374816+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:16.844706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884150+0000","last_change":"2024-03-06T07:30:21.364224+0000","last_active":"2024-03-06T07:30:24.884150+0000","last_peered":"2024-03-06T07:30:24.884150+0000","last_clean":"2024-03-06T07:30:24.884150+0000","last_became_active":"2024-03-06T07:30:21.363924+0000","last_became_peered":"2024-03-06T07:30:21.363924+0000","last_unstale":"2024-03-06T07:30:24.884150+0000","last_undegraded":"2024-03-06T07:30:24.884150+0000","last_fullsized":"2024-03-06T07:30:24.884150+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:29.122685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371708+0000","last_change":"2024-03-06T07:30:22.362379+0000","last_active":"2024-03-06T07:30:24.371708+0000","last_peered":"2024-03-06T07:30:24.371708+0000","last_clean":"2024-03-06T07:30:24.371708+0000","last_became_active":"2024-03-06T07:30:22.362091+0000","last_became_peered":"2024-03-06T07:30:22.362091+0000","last_unstale":"2024-03-06T07:30:24.371708+0000","last_undegraded":"2024-03-06T07:30:24.371708+0000","last_fullsized":"2024-03-06T07:30:24.371708+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:20.391472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346626+0000","last_change":"2024-03-06T07:30:21.354681+0000","last_active":"2024-03-06T07:30:23.346626+0000","last_peered":"2024-03-06T07:30:23.346626+0000","last_clean":"2024-03-06T07:30:23.346626+0000","last_became_active":"2024-03-06T07:30:21.354399+0000","last_became_peered":"2024-03-06T07:30:21.354399+0000","last_unstale":"2024-03-06T07:30:23.346626+0000","last_undegraded":"2024-03-06T07:30:23.346626+0000","last_fullsized":"2024-03-06T07:30:23.346626+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:56.731087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352625+0000","last_change":"2024-03-06T07:30:22.368255+0000","last_active":"2024-03-06T07:30:23.352625+0000","last_peered":"2024-03-06T07:30:23.352625+0000","last_clean":"2024-03-06T07:30:23.352625+0000","last_became_active":"2024-03-06T07:30:22.367968+0000","last_became_peered":"2024-03-06T07:30:22.367968+0000","last_unstale":"2024-03-06T07:30:23.352625+0000","last_undegraded":"2024-03-06T07:30:23.352625+0000","last_fullsized":"2024-03-06T07:30:23.352625+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:03.613030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371788+0000","last_change":"2024-03-06T07:30:21.359774+0000","last_active":"2024-03-06T07:30:24.371788+0000","last_peered":"2024-03-06T07:30:24.371788+0000","last_clean":"2024-03-06T07:30:24.371788+0000","last_became_active":"2024-03-06T07:30:21.359457+0000","last_became_peered":"2024-03-06T07:30:21.359457+0000","last_unstale":"2024-03-06T07:30:24.371788+0000","last_undegraded":"2024-03-06T07:30:24.371788+0000","last_fullsized":"2024-03-06T07:30:24.371788+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:41.974744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349743+0000","last_change":"2024-03-06T07:30:22.369678+0000","last_active":"2024-03-06T07:30:23.349743+0000","last_peered":"2024-03-06T07:30:23.349743+0000","last_clean":"2024-03-06T07:30:23.349743+0000","last_became_active":"2024-03-06T07:30:22.368569+0000","last_became_peered":"2024-03-06T07:30:22.368569+0000","last_unstale":"2024-03-06T07:30:23.349743+0000","last_undegraded":"2024-03-06T07:30:23.349743+0000","last_fullsized":"2024-03-06T07:30:23.349743+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:55.690989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349753+0000","last_change":"2024-03-06T07:30:21.359533+0000","last_active":"2024-03-06T07:30:23.349753+0000","last_peered":"2024-03-06T07:30:23.349753+0000","last_clean":"2024-03-06T07:30:23.349753+0000","last_became_active":"2024-03-06T07:30:21.359211+0000","last_became_peered":"2024-03-06T07:30:21.359211+0000","last_unstale":"2024-03-06T07:30:23.349753+0000","last_undegraded":"2024-03-06T07:30:23.349753+0000","last_fullsized":"2024-03-06T07:30:23.349753+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:34.007530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346584+0000","last_change":"2024-03-06T07:30:22.364308+0000","last_active":"2024-03-06T07:30:23.346584+0000","last_peered":"2024-03-06T07:30:23.346584+0000","last_clean":"2024-03-06T07:30:23.346584+0000","last_became_active":"2024-03-06T07:30:22.363848+0000","last_became_peered":"2024-03-06T07:30:22.363848+0000","last_unstale":"2024-03-06T07:30:23.346584+0000","last_undegraded":"2024-03-06T07:30:23.346584+0000","last_fullsized":"2024-03-06T07:30:23.346584+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:49.425487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884014+0000","last_change":"2024-03-06T07:30:21.364208+0000","last_active":"2024-03-06T07:30:24.884014+0000","last_peered":"2024-03-06T07:30:24.884014+0000","last_clean":"2024-03-06T07:30:24.884014+0000","last_became_active":"2024-03-06T07:30:21.363905+0000","last_became_peered":"2024-03-06T07:30:21.363905+0000","last_unstale":"2024-03-06T07:30:24.884014+0000","last_undegraded":"2024-03-06T07:30:24.884014+0000","last_fullsized":"2024-03-06T07:30:24.884014+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:56.526496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371667+0000","last_change":"2024-03-06T07:30:22.367439+0000","last_active":"2024-03-06T07:30:24.371667+0000","last_peered":"2024-03-06T07:30:24.371667+0000","last_clean":"2024-03-06T07:30:24.371667+0000","last_became_active":"2024-03-06T07:30:22.367142+0000","last_became_peered":"2024-03-06T07:30:22.367142+0000","last_unstale":"2024-03-06T07:30:24.371667+0000","last_undegraded":"2024-03-06T07:30:24.371667+0000","last_fullsized":"2024-03-06T07:30:24.371667+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:55.690989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372386+0000","last_change":"2024-03-06T07:30:21.355999+0000","last_active":"2024-03-06T07:30:24.372386+0000","last_peered":"2024-03-06T07:30:24.372386+0000","last_clean":"2024-03-06T07:30:24.372386+0000","last_became_active":"2024-03-06T07:30:21.355716+0000","last_became_peered":"2024-03-06T07:30:21.355716+0000","last_unstale":"2024-03-06T07:30:24.372386+0000","last_undegraded":"2024-03-06T07:30:24.372386+0000","last_fullsized":"2024-03-06T07:30:24.372386+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:24.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":382,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371479+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:30:24.371479+0000","last_peered":"2024-03-06T07:30:24.371479+0000","last_clean":"2024-03-06T07:30:24.371479+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:30:24.371479+0000","last_undegraded":"2024-03-06T07:30:24.371479+0000","last_fullsized":"2024-03-06T07:30:24.371479+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574945,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41712,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330128,"statfs":{"total":96636764160,"available":96594051072,"internally_reserved":0,"allocated":2265088,"data_stored":1310118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574946,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41116,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330724,"statfs":{"total":96636764160,"available":96594661376,"internally_reserved":0,"allocated":1646592,"data_stored":712812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41116,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330724,"statfs":{"total":96636764160,"available":96594661376,"internally_reserved":0,"allocated":1654784,"data_stored":717612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574945,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":1650688,"data_stored":717570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41120,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330720,"statfs":{"total":96636764160,"available":96594657280,"internally_reserved":0,"allocated":1654784,"data_stored":717660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574945,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41120,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330720,"statfs":{"total":96636764160,"available":96594657280,"internally_reserved":0,"allocated":1654784,"data_stored":718116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":1646592,"data_stored":717500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574946,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41684,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330156,"statfs":{"total":96636764160,"available":96594079744,"internally_reserved":0,"allocated":2228224,"data_stored":1301568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:26.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:26.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.551+0000 7f86e73f6640 1 Processor -- start 2024-03-06T07:30:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.551+0000 7f86e73f6640 1 -- start start 2024-03-06T07:30:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e73f6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86e0071540 0x7f86e00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e73f6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f86e00d3400 0x7f86e0071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e73f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e00b2b50 0x7f86e00d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e73f6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e00d2f70 con 0x7f86e0071540 2024-03-06T07:30:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e73f6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e00d4be0 con 0x7f86e00d3400 2024-03-06T07:30:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e73f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e0071e60 con 0x7f86e00b2b50 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e5192640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86e0071540 0x7f86e00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e5192640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86e0071540 0x7f86e00b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56390/0 (socket says 172.21.15.27:56390) 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e5192640 1 -- 172.21.15.27:0/154631675 learned_addr learned my addr 172.21.15.27:0/154631675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e4991640 1 --2- 172.21.15.27:0/154631675 >> v2:172.21.15.69:3300/0 conn(0x7f86e00d3400 0x7f86e0071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.553+0000 7f86e5192640 1 -- 172.21.15.27:0/154631675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e00b2b50 msgr2=0x7f86e00d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.552+0000 7f86e5993640 1 --2- 172.21.15.27:0/154631675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e00b2b50 0x7f86e00d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.553+0000 7f86e5192640 1 --2- 172.21.15.27:0/154631675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e00b2b50 0x7f86e00d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.553+0000 7f86e5192640 1 -- 172.21.15.27:0/154631675 >> v2:172.21.15.69:3300/0 conn(0x7f86e00d3400 msgr2=0x7f86e0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.553+0000 7f86e5192640 1 --2- 172.21.15.27:0/154631675 >> v2:172.21.15.69:3300/0 conn(0x7f86e00d3400 0x7f86e0071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.553+0000 7f86e5192640 1 -- 172.21.15.27:0/154631675 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e00ce610 con 0x7f86e0071540 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.553+0000 7f86e4991640 1 --2- 172.21.15.27:0/154631675 >> v2:172.21.15.69:3300/0 conn(0x7f86e00d3400 0x7f86e0071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.553+0000 7f86e5993640 1 --2- 172.21.15.27:0/154631675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e00b2b50 0x7f86e00d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.553+0000 7f86e5192640 1 --2- 172.21.15.27:0/154631675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e0071540 0x7f86e00b30a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f86e00b6860 tx=0x7f86d002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbcfe5916c954285 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.553+0000 7f86d7fff640 1 -- 172.21.15.27:0/154631675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d0035020 con 0x7f86e0071540 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86d7fff640 1 -- 172.21.15.27:0/154631675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d002d040 con 0x7f86e0071540 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86d7fff640 1 -- 172.21.15.27:0/154631675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d0031a80 con 0x7f86e0071540 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86e73f6640 1 -- 172.21.15.27:0/154631675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e0071540 msgr2=0x7f86e00b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86e73f6640 1 --2- 172.21.15.27:0/154631675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e0071540 0x7f86e00b30a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f86e00b6860 tx=0x7f86d002b900 comp rx=0 tx=0).stop 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86e73f6640 1 -- 172.21.15.27:0/154631675 shutdown_connections 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86e73f6640 1 --2- 172.21.15.27:0/154631675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e00b2b50 0x7f86e00d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86e73f6640 1 --2- 172.21.15.27:0/154631675 >> v2:172.21.15.69:3300/0 conn(0x7f86e00d3400 0x7f86e0071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86e73f6640 1 --2- 172.21.15.27:0/154631675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e0071540 0x7f86e00b30a0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86e73f6640 1 -- 172.21.15.27:0/154631675 >> 172.21.15.27:0/154631675 conn(0x7f86e00d3ca0 msgr2=0x7f86e00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86e73f6640 1 -- 172.21.15.27:0/154631675 shutdown_connections 2024-03-06T07:30:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.554+0000 7f86e73f6640 1 -- 172.21.15.27:0/154631675 wait complete. 2024-03-06T07:30:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.555+0000 7f86e73f6640 1 Processor -- start 2024-03-06T07:30:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.555+0000 7f86e73f6640 1 -- start start 2024-03-06T07:30:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e73f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e0071540 0x7f86e00adc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e73f6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e00b2b50 0x7f86e00ab070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e73f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86e00d3400 0x7f86e00b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e73f6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e00ce8f0 con 0x7f86e00b2b50 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e73f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e00d2c00 con 0x7f86e00d3400 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e73f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e015e370 con 0x7f86e0071540 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e4991640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e00b2b50 0x7f86e00ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e4991640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e00b2b50 0x7f86e00ab070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56402/0 (socket says 172.21.15.27:56402) 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e4991640 1 -- 172.21.15.27:0/1819831479 learned_addr learned my addr 172.21.15.27:0/1819831479 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e5192640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e0071540 0x7f86e00adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e5993640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86e00d3400 0x7f86e00b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e4991640 1 -- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e0071540 msgr2=0x7f86e00adc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e4991640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e0071540 0x7f86e00adc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e4991640 1 -- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86e00d3400 msgr2=0x7f86e00b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e4991640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86e00d3400 0x7f86e00b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e4991640 1 -- 172.21.15.27:0/1819831479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e00abb30 con 0x7f86e00b2b50 2024-03-06T07:30:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e4991640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e00b2b50 0x7f86e00ab070 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00c7f0 tx=0x7f86dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86d5ffb640 1 -- 172.21.15.27:0/1819831479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc015070 con 0x7f86e00b2b50 2024-03-06T07:30:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86d5ffb640 1 -- 172.21.15.27:0/1819831479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc007820 con 0x7f86e00b2b50 2024-03-06T07:30:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e00abe10 con 0x7f86e00b2b50 2024-03-06T07:30:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.557+0000 7f86d5ffb640 1 -- 172.21.15.27:0/1819831479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc007c10 con 0x7f86e00b2b50 2024-03-06T07:30:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.558+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e00ac3d0 con 0x7f86e00b2b50 2024-03-06T07:30:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.559+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e0068700 con 0x7f86e00b2b50 2024-03-06T07:30:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.559+0000 7f86d5ffb640 1 -- 172.21.15.27:0/1819831479 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86dc018070 con 0x7f86e00b2b50 2024-03-06T07:30:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.559+0000 7f86d5ffb640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f86c406b4d0 0x7f86c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.559+0000 7f86d5ffb640 1 -- 172.21.15.27:0/1819831479 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86dc08a270 con 0x7f86e00b2b50 2024-03-06T07:30:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.560+0000 7f86e5192640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f86c406b4d0 0x7f86c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.560+0000 7f86e5192640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f86c406b4d0 0x7f86c406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f86d0036000 tx=0x7f86d003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.564+0000 7f86d5ffb640 1 -- 172.21.15.27:0/1819831479 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86dc059180 con 0x7f86e00b2b50 2024-03-06T07:30:26.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.744+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f86e015e990 con 0x7f86c406b4d0 2024-03-06T07:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.751+0000 7f86d5ffb640 1 -- 172.21.15.27:0/1819831479 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252482 (secure 0 0 0) 0x7f86e015e990 con 0x7f86c406b4d0 2024-03-06T07:30:26.755 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:26.756 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:30:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f86c406b4d0 msgr2=0x7f86c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f86c406b4d0 0x7f86c406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f86d0036000 tx=0x7f86d003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e00b2b50 msgr2=0x7f86e00ab070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e00b2b50 0x7f86e00ab070 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00c7f0 tx=0x7f86dc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 shutdown_connections 2024-03-06T07:30:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f86c406b4d0 0x7f86c406d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86e00d3400 0x7f86e00b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86e00b2b50 0x7f86e00ab070 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 --2- 172.21.15.27:0/1819831479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86e0071540 0x7f86e00adc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 >> 172.21.15.27:0/1819831479 conn(0x7f86e00d3ca0 msgr2=0x7f86e0160be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 shutdown_connections 2024-03-06T07:30:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.754+0000 7f86e73f6640 1 -- 172.21.15.27:0/1819831479 wait complete. 2024-03-06T07:30:26.767 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-03-06T07:30:26.097010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330092,"kb_used_data":14064,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644628,"statfs":{"total":773094113280,"available":772756099072,"internally_reserved":0,"allocated":14401536,"data_stored":6912956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.757546"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431259+0000","last_change":"2024-03-06T07:30:22.353235+0000","last_active":"2024-03-06T07:30:23.431259+0000","last_peered":"2024-03-06T07:30:23.431259+0000","last_clean":"2024-03-06T07:30:23.431259+0000","last_became_active":"2024-03-06T07:30:22.352948+0000","last_became_peered":"2024-03-06T07:30:22.352948+0000","last_unstale":"2024-03-06T07:30:23.431259+0000","last_undegraded":"2024-03-06T07:30:23.431259+0000","last_fullsized":"2024-03-06T07:30:23.431259+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.265886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346780+0000","last_change":"2024-03-06T07:30:22.362649+0000","last_active":"2024-03-06T07:30:23.346780+0000","last_peered":"2024-03-06T07:30:23.346780+0000","last_clean":"2024-03-06T07:30:23.346780+0000","last_became_active":"2024-03-06T07:30:22.362391+0000","last_became_peered":"2024-03-06T07:30:22.362391+0000","last_unstale":"2024-03-06T07:30:23.346780+0000","last_undegraded":"2024-03-06T07:30:23.346780+0000","last_fullsized":"2024-03-06T07:30:23.346780+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:03.613030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387133+0000","last_change":"2024-03-06T07:30:22.361827+0000","last_active":"2024-03-06T07:30:24.387133+0000","last_peered":"2024-03-06T07:30:24.387133+0000","last_clean":"2024-03-06T07:30:24.387133+0000","last_became_active":"2024-03-06T07:30:22.361513+0000","last_became_peered":"2024-03-06T07:30:22.361513+0000","last_unstale":"2024-03-06T07:30:24.387133+0000","last_undegraded":"2024-03-06T07:30:24.387133+0000","last_fullsized":"2024-03-06T07:30:24.387133+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:16.844706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431088+0000","last_change":"2024-03-06T07:30:22.362829+0000","last_active":"2024-03-06T07:30:23.431088+0000","last_peered":"2024-03-06T07:30:23.431088+0000","last_clean":"2024-03-06T07:30:23.431088+0000","last_became_active":"2024-03-06T07:30:22.362520+0000","last_became_peered":"2024-03-06T07:30:22.362520+0000","last_unstale":"2024-03-06T07:30:23.431088+0000","last_undegraded":"2024-03-06T07:30:23.431088+0000","last_fullsized":"2024-03-06T07:30:23.431088+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:03.592119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372164+0000","last_change":"2024-03-06T07:30:22.365431+0000","last_active":"2024-03-06T07:30:24.372164+0000","last_peered":"2024-03-06T07:30:24.372164+0000","last_clean":"2024-03-06T07:30:24.372164+0000","last_became_active":"2024-03-06T07:30:22.365173+0000","last_became_peered":"2024-03-06T07:30:22.365173+0000","last_unstale":"2024-03-06T07:30:24.372164+0000","last_undegraded":"2024-03-06T07:30:24.372164+0000","last_fullsized":"2024-03-06T07:30:24.372164+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:42.135831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374756+0000","last_change":"2024-03-06T07:30:22.356741+0000","last_active":"2024-03-06T07:30:24.374756+0000","last_peered":"2024-03-06T07:30:24.374756+0000","last_clean":"2024-03-06T07:30:24.374756+0000","last_became_active":"2024-03-06T07:30:22.355933+0000","last_became_peered":"2024-03-06T07:30:22.355933+0000","last_unstale":"2024-03-06T07:30:24.374756+0000","last_undegraded":"2024-03-06T07:30:24.374756+0000","last_fullsized":"2024-03-06T07:30:24.374756+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:20.362560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374487+0000","last_change":"2024-03-06T07:30:22.360723+0000","last_active":"2024-03-06T07:30:24.374487+0000","last_peered":"2024-03-06T07:30:24.374487+0000","last_clean":"2024-03-06T07:30:24.374487+0000","last_became_active":"2024-03-06T07:30:22.360423+0000","last_became_peered":"2024-03-06T07:30:22.360423+0000","last_unstale":"2024-03-06T07:30:24.374487+0000","last_undegraded":"2024-03-06T07:30:24.374487+0000","last_fullsized":"2024-03-06T07:30:24.374487+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:05.146007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387316+0000","last_change":"2024-03-06T07:30:22.367831+0000","last_active":"2024-03-06T07:30:24.387316+0000","last_peered":"2024-03-06T07:30:24.387316+0000","last_clean":"2024-03-06T07:30:24.387316+0000","last_became_active":"2024-03-06T07:30:22.367173+0000","last_became_peered":"2024-03-06T07:30:22.367173+0000","last_unstale":"2024-03-06T07:30:24.387316+0000","last_undegraded":"2024-03-06T07:30:24.387316+0000","last_fullsized":"2024-03-06T07:30:24.387316+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:54.397591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387195+0000","last_change":"2024-03-06T07:30:22.365076+0000","last_active":"2024-03-06T07:30:24.387195+0000","last_peered":"2024-03-06T07:30:24.387195+0000","last_clean":"2024-03-06T07:30:24.387195+0000","last_became_active":"2024-03-06T07:30:22.364772+0000","last_became_peered":"2024-03-06T07:30:22.364772+0000","last_unstale":"2024-03-06T07:30:24.387195+0000","last_undegraded":"2024-03-06T07:30:24.387195+0000","last_fullsized":"2024-03-06T07:30:24.387195+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:00.645432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346922+0000","last_change":"2024-03-06T07:30:22.360201+0000","last_active":"2024-03-06T07:30:23.346922+0000","last_peered":"2024-03-06T07:30:23.346922+0000","last_clean":"2024-03-06T07:30:23.346922+0000","last_became_active":"2024-03-06T07:30:22.359914+0000","last_became_peered":"2024-03-06T07:30:22.359914+0000","last_unstale":"2024-03-06T07:30:23.346922+0000","last_undegraded":"2024-03-06T07:30:23.346922+0000","last_fullsized":"2024-03-06T07:30:23.346922+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:26.283295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884084+0000","last_change":"2024-03-06T07:30:22.366235+0000","last_active":"2024-03-06T07:30:24.884084+0000","last_peered":"2024-03-06T07:30:24.884084+0000","last_clean":"2024-03-06T07:30:24.884084+0000","last_became_active":"2024-03-06T07:30:22.365937+0000","last_became_peered":"2024-03-06T07:30:22.365937+0000","last_unstale":"2024-03-06T07:30:24.884084+0000","last_undegraded":"2024-03-06T07:30:24.884084+0000","last_fullsized":"2024-03-06T07:30:24.884084+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:51.646354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387174+0000","last_change":"2024-03-06T07:30:22.364672+0000","last_active":"2024-03-06T07:30:24.387174+0000","last_peered":"2024-03-06T07:30:24.387174+0000","last_clean":"2024-03-06T07:30:24.387174+0000","last_became_active":"2024-03-06T07:30:22.364328+0000","last_became_peered":"2024-03-06T07:30:22.364328+0000","last_unstale":"2024-03-06T07:30:24.387174+0000","last_undegraded":"2024-03-06T07:30:24.387174+0000","last_fullsized":"2024-03-06T07:30:24.387174+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:42.982021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431114+0000","last_change":"2024-03-06T07:30:22.355786+0000","last_active":"2024-03-06T07:30:23.431114+0000","last_peered":"2024-03-06T07:30:23.431114+0000","last_clean":"2024-03-06T07:30:23.431114+0000","last_became_active":"2024-03-06T07:30:22.355486+0000","last_became_peered":"2024-03-06T07:30:22.355486+0000","last_unstale":"2024-03-06T07:30:23.431114+0000","last_undegraded":"2024-03-06T07:30:23.431114+0000","last_fullsized":"2024-03-06T07:30:23.431114+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:15.086879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346495+0000","last_change":"2024-03-06T07:30:22.368385+0000","last_active":"2024-03-06T07:30:23.346495+0000","last_peered":"2024-03-06T07:30:23.346495+0000","last_clean":"2024-03-06T07:30:23.346495+0000","last_became_active":"2024-03-06T07:30:22.367547+0000","last_became_peered":"2024-03-06T07:30:22.367547+0000","last_unstale":"2024-03-06T07:30:23.346495+0000","last_undegraded":"2024-03-06T07:30:23.346495+0000","last_fullsized":"2024-03-06T07:30:23.346495+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:01.803277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.430962+0000","last_change":"2024-03-06T07:30:22.360407+0000","last_active":"2024-03-06T07:30:23.430962+0000","last_peered":"2024-03-06T07:30:23.430962+0000","last_clean":"2024-03-06T07:30:23.430962+0000","last_became_active":"2024-03-06T07:30:22.360125+0000","last_became_peered":"2024-03-06T07:30:22.360125+0000","last_unstale":"2024-03-06T07:30:23.430962+0000","last_undegraded":"2024-03-06T07:30:23.430962+0000","last_fullsized":"2024-03-06T07:30:23.430962+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:05.135405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349733+0000","last_change":"2024-03-06T07:30:22.355315+0000","last_active":"2024-03-06T07:30:23.349733+0000","last_peered":"2024-03-06T07:30:23.349733+0000","last_clean":"2024-03-06T07:30:23.349733+0000","last_became_active":"2024-03-06T07:30:22.355019+0000","last_became_peered":"2024-03-06T07:30:22.355019+0000","last_unstale":"2024-03-06T07:30:23.349733+0000","last_undegraded":"2024-03-06T07:30:23.349733+0000","last_fullsized":"2024-03-06T07:30:23.349733+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:17.390996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374926+0000","last_change":"2024-03-06T07:30:22.360820+0000","last_active":"2024-03-06T07:30:24.374926+0000","last_peered":"2024-03-06T07:30:24.374926+0000","last_clean":"2024-03-06T07:30:24.374926+0000","last_became_active":"2024-03-06T07:30:22.360496+0000","last_became_peered":"2024-03-06T07:30:22.360496+0000","last_unstale":"2024-03-06T07:30:24.374926+0000","last_undegraded":"2024-03-06T07:30:24.374926+0000","last_fullsized":"2024-03-06T07:30:24.374926+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:25.165735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350060+0000","last_change":"2024-03-06T07:30:22.361923+0000","last_active":"2024-03-06T07:30:23.350060+0000","last_peered":"2024-03-06T07:30:23.350060+0000","last_clean":"2024-03-06T07:30:23.350060+0000","last_became_active":"2024-03-06T07:30:22.361640+0000","last_became_peered":"2024-03-06T07:30:22.361640+0000","last_unstale":"2024-03-06T07:30:23.350060+0000","last_undegraded":"2024-03-06T07:30:23.350060+0000","last_fullsized":"2024-03-06T07:30:23.350060+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:56.874819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884161+0000","last_change":"2024-03-06T07:30:22.363949+0000","last_active":"2024-03-06T07:30:24.884161+0000","last_peered":"2024-03-06T07:30:24.884161+0000","last_clean":"2024-03-06T07:30:24.884161+0000","last_became_active":"2024-03-06T07:30:22.363671+0000","last_became_peered":"2024-03-06T07:30:22.363671+0000","last_unstale":"2024-03-06T07:30:24.884161+0000","last_undegraded":"2024-03-06T07:30:24.884161+0000","last_fullsized":"2024-03-06T07:30:24.884161+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:45.367191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350027+0000","last_change":"2024-03-06T07:30:22.362617+0000","last_active":"2024-03-06T07:30:23.350027+0000","last_peered":"2024-03-06T07:30:23.350027+0000","last_clean":"2024-03-06T07:30:23.350027+0000","last_became_active":"2024-03-06T07:30:22.362338+0000","last_became_peered":"2024-03-06T07:30:22.362338+0000","last_unstale":"2024-03-06T07:30:23.350027+0000","last_undegraded":"2024-03-06T07:30:23.350027+0000","last_fullsized":"2024-03-06T07:30:23.350027+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:55.018122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371820+0000","last_change":"2024-03-06T07:30:22.364833+0000","last_active":"2024-03-06T07:30:24.371820+0000","last_peered":"2024-03-06T07:30:24.371820+0000","last_clean":"2024-03-06T07:30:24.371820+0000","last_became_active":"2024-03-06T07:30:22.364548+0000","last_became_peered":"2024-03-06T07:30:22.364548+0000","last_unstale":"2024-03-06T07:30:24.371820+0000","last_undegraded":"2024-03-06T07:30:24.371820+0000","last_fullsized":"2024-03-06T07:30:24.371820+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:56.874819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884006+0000","last_change":"2024-03-06T07:30:22.358843+0000","last_active":"2024-03-06T07:30:24.884006+0000","last_peered":"2024-03-06T07:30:24.884006+0000","last_clean":"2024-03-06T07:30:24.884006+0000","last_became_active":"2024-03-06T07:30:22.358546+0000","last_became_peered":"2024-03-06T07:30:22.358546+0000","last_unstale":"2024-03-06T07:30:24.884006+0000","last_undegraded":"2024-03-06T07:30:24.884006+0000","last_fullsized":"2024-03-06T07:30:24.884006+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:30.124102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387095+0000","last_change":"2024-03-06T07:30:22.366000+0000","last_active":"2024-03-06T07:30:24.387095+0000","last_peered":"2024-03-06T07:30:24.387095+0000","last_clean":"2024-03-06T07:30:24.387095+0000","last_became_active":"2024-03-06T07:30:22.364173+0000","last_became_peered":"2024-03-06T07:30:22.364173+0000","last_unstale":"2024-03-06T07:30:24.387095+0000","last_undegraded":"2024-03-06T07:30:24.387095+0000","last_fullsized":"2024-03-06T07:30:24.387095+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:20.362560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.883947+0000","last_change":"2024-03-06T07:30:22.364682+0000","last_active":"2024-03-06T07:30:24.883947+0000","last_peered":"2024-03-06T07:30:24.883947+0000","last_clean":"2024-03-06T07:30:24.883947+0000","last_became_active":"2024-03-06T07:30:22.364362+0000","last_became_peered":"2024-03-06T07:30:22.364362+0000","last_unstale":"2024-03-06T07:30:24.883947+0000","last_undegraded":"2024-03-06T07:30:24.883947+0000","last_fullsized":"2024-03-06T07:30:24.883947+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:38.013591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.347068+0000","last_change":"2024-03-06T07:30:22.361478+0000","last_active":"2024-03-06T07:30:23.347068+0000","last_peered":"2024-03-06T07:30:23.347068+0000","last_clean":"2024-03-06T07:30:23.347068+0000","last_became_active":"2024-03-06T07:30:22.361151+0000","last_became_peered":"2024-03-06T07:30:22.361151+0000","last_unstale":"2024-03-06T07:30:23.347068+0000","last_undegraded":"2024-03-06T07:30:23.347068+0000","last_fullsized":"2024-03-06T07:30:23.347068+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:58.961847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350100+0000","last_change":"2024-03-06T07:30:22.366826+0000","last_active":"2024-03-06T07:30:23.350100+0000","last_peered":"2024-03-06T07:30:23.350100+0000","last_clean":"2024-03-06T07:30:23.350100+0000","last_became_active":"2024-03-06T07:30:22.366560+0000","last_became_peered":"2024-03-06T07:30:22.366560+0000","last_unstale":"2024-03-06T07:30:23.350100+0000","last_undegraded":"2024-03-06T07:30:23.350100+0000","last_fullsized":"2024-03-06T07:30:23.350100+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:42.135831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431119+0000","last_change":"2024-03-06T07:30:22.355315+0000","last_active":"2024-03-06T07:30:23.431119+0000","last_peered":"2024-03-06T07:30:23.431119+0000","last_clean":"2024-03-06T07:30:23.431119+0000","last_became_active":"2024-03-06T07:30:22.354999+0000","last_became_peered":"2024-03-06T07:30:22.354999+0000","last_unstale":"2024-03-06T07:30:23.431119+0000","last_undegraded":"2024-03-06T07:30:23.431119+0000","last_fullsized":"2024-03-06T07:30:23.431119+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:26.564855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352848+0000","last_change":"2024-03-06T07:30:22.363204+0000","last_active":"2024-03-06T07:30:23.352848+0000","last_peered":"2024-03-06T07:30:23.352848+0000","last_clean":"2024-03-06T07:30:23.352848+0000","last_became_active":"2024-03-06T07:30:22.362848+0000","last_became_peered":"2024-03-06T07:30:22.362848+0000","last_unstale":"2024-03-06T07:30:23.352848+0000","last_undegraded":"2024-03-06T07:30:23.352848+0000","last_fullsized":"2024-03-06T07:30:23.352848+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:26.283295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431445+0000","last_change":"2024-03-06T07:30:22.366099+0000","last_active":"2024-03-06T07:30:23.431445+0000","last_peered":"2024-03-06T07:30:23.431445+0000","last_clean":"2024-03-06T07:30:23.431445+0000","last_became_active":"2024-03-06T07:30:22.365744+0000","last_became_peered":"2024-03-06T07:30:22.365744+0000","last_unstale":"2024-03-06T07:30:23.431445+0000","last_undegraded":"2024-03-06T07:30:23.431445+0000","last_fullsized":"2024-03-06T07:30:23.431445+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:32.470076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.430942+0000","last_change":"2024-03-06T07:30:22.357198+0000","last_active":"2024-03-06T07:30:23.430942+0000","last_peered":"2024-03-06T07:30:23.430942+0000","last_clean":"2024-03-06T07:30:23.430942+0000","last_became_active":"2024-03-06T07:30:22.356926+0000","last_became_peered":"2024-03-06T07:30:22.356926+0000","last_unstale":"2024-03-06T07:30:23.430942+0000","last_undegraded":"2024-03-06T07:30:23.430942+0000","last_fullsized":"2024-03-06T07:30:23.430942+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:43.954181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372235+0000","last_change":"2024-03-06T07:30:22.366699+0000","last_active":"2024-03-06T07:30:24.372235+0000","last_peered":"2024-03-06T07:30:24.372235+0000","last_clean":"2024-03-06T07:30:24.372235+0000","last_became_active":"2024-03-06T07:30:22.366429+0000","last_became_peered":"2024-03-06T07:30:22.366429+0000","last_unstale":"2024-03-06T07:30:24.372235+0000","last_undegraded":"2024-03-06T07:30:24.372235+0000","last_fullsized":"2024-03-06T07:30:24.372235+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:25.419177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374445+0000","last_change":"2024-03-06T07:30:22.356487+0000","last_active":"2024-03-06T07:30:24.374445+0000","last_peered":"2024-03-06T07:30:24.374445+0000","last_clean":"2024-03-06T07:30:24.374445+0000","last_became_active":"2024-03-06T07:30:22.356162+0000","last_became_peered":"2024-03-06T07:30:22.356162+0000","last_unstale":"2024-03-06T07:30:24.374445+0000","last_undegraded":"2024-03-06T07:30:24.374445+0000","last_fullsized":"2024-03-06T07:30:24.374445+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:44.856072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350204+0000","last_change":"2024-03-06T07:30:22.364976+0000","last_active":"2024-03-06T07:30:23.350204+0000","last_peered":"2024-03-06T07:30:23.350204+0000","last_clean":"2024-03-06T07:30:23.350204+0000","last_became_active":"2024-03-06T07:30:22.364725+0000","last_became_peered":"2024-03-06T07:30:22.364725+0000","last_unstale":"2024-03-06T07:30:23.350204+0000","last_undegraded":"2024-03-06T07:30:23.350204+0000","last_fullsized":"2024-03-06T07:30:23.350204+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:25.419177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"131'4","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372163+0000","last_change":"2024-03-06T07:30:21.356722+0000","last_active":"2024-03-06T07:30:24.372163+0000","last_peered":"2024-03-06T07:30:24.372163+0000","last_clean":"2024-03-06T07:30:24.372163+0000","last_became_active":"2024-03-06T07:30:21.356457+0000","last_became_peered":"2024-03-06T07:30:21.356457+0000","last_unstale":"2024-03-06T07:30:24.372163+0000","last_undegraded":"2024-03-06T07:30:24.372163+0000","last_fullsized":"2024-03-06T07:30:24.372163+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:34.007530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387212+0000","last_change":"2024-03-06T07:30:22.367501+0000","last_active":"2024-03-06T07:30:24.387212+0000","last_peered":"2024-03-06T07:30:24.387212+0000","last_clean":"2024-03-06T07:30:24.387212+0000","last_became_active":"2024-03-06T07:30:22.367131+0000","last_became_peered":"2024-03-06T07:30:22.367131+0000","last_unstale":"2024-03-06T07:30:24.387212+0000","last_undegraded":"2024-03-06T07:30:24.387212+0000","last_fullsized":"2024-03-06T07:30:24.387212+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:35.014807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352689+0000","last_change":"2024-03-06T07:30:21.362891+0000","last_active":"2024-03-06T07:30:23.352689+0000","last_peered":"2024-03-06T07:30:23.352689+0000","last_clean":"2024-03-06T07:30:23.352689+0000","last_became_active":"2024-03-06T07:30:21.362627+0000","last_became_peered":"2024-03-06T07:30:21.362627+0000","last_unstale":"2024-03-06T07:30:23.352689+0000","last_undegraded":"2024-03-06T07:30:23.352689+0000","last_fullsized":"2024-03-06T07:30:23.352689+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:25.557578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372282+0000","last_change":"2024-03-06T07:30:22.364459+0000","last_active":"2024-03-06T07:30:24.372282+0000","last_peered":"2024-03-06T07:30:24.372282+0000","last_clean":"2024-03-06T07:30:24.372282+0000","last_became_active":"2024-03-06T07:30:22.364191+0000","last_became_peered":"2024-03-06T07:30:22.364191+0000","last_unstale":"2024-03-06T07:30:24.372282+0000","last_undegraded":"2024-03-06T07:30:24.372282+0000","last_fullsized":"2024-03-06T07:30:24.372282+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:13.440811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884193+0000","last_change":"2024-03-06T07:30:21.358292+0000","last_active":"2024-03-06T07:30:24.884193+0000","last_peered":"2024-03-06T07:30:24.884193+0000","last_clean":"2024-03-06T07:30:24.884193+0000","last_became_active":"2024-03-06T07:30:21.358003+0000","last_became_peered":"2024-03-06T07:30:21.358003+0000","last_unstale":"2024-03-06T07:30:24.884193+0000","last_undegraded":"2024-03-06T07:30:24.884193+0000","last_fullsized":"2024-03-06T07:30:24.884193+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:52.695460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371760+0000","last_change":"2024-03-06T07:30:22.359793+0000","last_active":"2024-03-06T07:30:24.371760+0000","last_peered":"2024-03-06T07:30:24.371760+0000","last_clean":"2024-03-06T07:30:24.371760+0000","last_became_active":"2024-03-06T07:30:22.359513+0000","last_became_peered":"2024-03-06T07:30:22.359513+0000","last_unstale":"2024-03-06T07:30:24.371760+0000","last_undegraded":"2024-03-06T07:30:24.371760+0000","last_fullsized":"2024-03-06T07:30:24.371760+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:55.018122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346809+0000","last_change":"2024-03-06T07:30:21.360537+0000","last_active":"2024-03-06T07:30:23.346809+0000","last_peered":"2024-03-06T07:30:23.346809+0000","last_clean":"2024-03-06T07:30:23.346809+0000","last_became_active":"2024-03-06T07:30:21.359798+0000","last_became_peered":"2024-03-06T07:30:21.359798+0000","last_unstale":"2024-03-06T07:30:23.346809+0000","last_undegraded":"2024-03-06T07:30:23.346809+0000","last_fullsized":"2024-03-06T07:30:23.346809+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:35.594372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349748+0000","last_change":"2024-03-06T07:30:22.362520+0000","last_active":"2024-03-06T07:30:23.349748+0000","last_peered":"2024-03-06T07:30:23.349748+0000","last_clean":"2024-03-06T07:30:23.349748+0000","last_became_active":"2024-03-06T07:30:22.362258+0000","last_became_peered":"2024-03-06T07:30:22.362258+0000","last_unstale":"2024-03-06T07:30:23.349748+0000","last_undegraded":"2024-03-06T07:30:23.349748+0000","last_fullsized":"2024-03-06T07:30:23.349748+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:20.391472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346528+0000","last_change":"2024-03-06T07:30:21.357328+0000","last_active":"2024-03-06T07:30:23.346528+0000","last_peered":"2024-03-06T07:30:23.346528+0000","last_clean":"2024-03-06T07:30:23.346528+0000","last_became_active":"2024-03-06T07:30:21.357066+0000","last_became_peered":"2024-03-06T07:30:21.357066+0000","last_unstale":"2024-03-06T07:30:23.346528+0000","last_undegraded":"2024-03-06T07:30:23.346528+0000","last_fullsized":"2024-03-06T07:30:23.346528+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.806387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374639+0000","last_change":"2024-03-06T07:30:22.357870+0000","last_active":"2024-03-06T07:30:24.374639+0000","last_peered":"2024-03-06T07:30:24.374639+0000","last_clean":"2024-03-06T07:30:24.374639+0000","last_became_active":"2024-03-06T07:30:22.357613+0000","last_became_peered":"2024-03-06T07:30:22.357613+0000","last_unstale":"2024-03-06T07:30:24.374639+0000","last_undegraded":"2024-03-06T07:30:24.374639+0000","last_fullsized":"2024-03-06T07:30:24.374639+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:38.023851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349839+0000","last_change":"2024-03-06T07:30:21.362638+0000","last_active":"2024-03-06T07:30:23.349839+0000","last_peered":"2024-03-06T07:30:23.349839+0000","last_clean":"2024-03-06T07:30:23.349839+0000","last_became_active":"2024-03-06T07:30:21.362325+0000","last_became_peered":"2024-03-06T07:30:21.362325+0000","last_unstale":"2024-03-06T07:30:23.349839+0000","last_undegraded":"2024-03-06T07:30:23.349839+0000","last_fullsized":"2024-03-06T07:30:23.349839+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:39.237665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387288+0000","last_change":"2024-03-06T07:30:22.368329+0000","last_active":"2024-03-06T07:30:24.387288+0000","last_peered":"2024-03-06T07:30:24.387288+0000","last_clean":"2024-03-06T07:30:24.387288+0000","last_became_active":"2024-03-06T07:30:22.367042+0000","last_became_peered":"2024-03-06T07:30:22.367042+0000","last_unstale":"2024-03-06T07:30:24.387288+0000","last_undegraded":"2024-03-06T07:30:24.387288+0000","last_fullsized":"2024-03-06T07:30:24.387288+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:25.239461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374410+0000","last_change":"2024-03-06T07:30:21.354843+0000","last_active":"2024-03-06T07:30:24.374410+0000","last_peered":"2024-03-06T07:30:24.374410+0000","last_clean":"2024-03-06T07:30:24.374410+0000","last_became_active":"2024-03-06T07:30:21.354561+0000","last_became_peered":"2024-03-06T07:30:21.354561+0000","last_unstale":"2024-03-06T07:30:24.374410+0000","last_undegraded":"2024-03-06T07:30:24.374410+0000","last_fullsized":"2024-03-06T07:30:24.374410+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:31.246161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346473+0000","last_change":"2024-03-06T07:30:22.364106+0000","last_active":"2024-03-06T07:30:23.346473+0000","last_peered":"2024-03-06T07:30:23.346473+0000","last_clean":"2024-03-06T07:30:23.346473+0000","last_became_active":"2024-03-06T07:30:22.363504+0000","last_became_peered":"2024-03-06T07:30:22.363504+0000","last_unstale":"2024-03-06T07:30:23.346473+0000","last_undegraded":"2024-03-06T07:30:23.346473+0000","last_fullsized":"2024-03-06T07:30:23.346473+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:32.470076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346552+0000","last_change":"2024-03-06T07:30:21.353154+0000","last_active":"2024-03-06T07:30:23.346552+0000","last_peered":"2024-03-06T07:30:23.346552+0000","last_clean":"2024-03-06T07:30:23.346552+0000","last_became_active":"2024-03-06T07:30:21.352853+0000","last_became_peered":"2024-03-06T07:30:21.352853+0000","last_unstale":"2024-03-06T07:30:23.346552+0000","last_undegraded":"2024-03-06T07:30:23.346552+0000","last_fullsized":"2024-03-06T07:30:23.346552+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:18.752403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431311+0000","last_change":"2024-03-06T07:30:22.358294+0000","last_active":"2024-03-06T07:30:23.431311+0000","last_peered":"2024-03-06T07:30:23.431311+0000","last_clean":"2024-03-06T07:30:23.431311+0000","last_became_active":"2024-03-06T07:30:22.357962+0000","last_became_peered":"2024-03-06T07:30:22.357962+0000","last_unstale":"2024-03-06T07:30:23.431311+0000","last_undegraded":"2024-03-06T07:30:23.431311+0000","last_fullsized":"2024-03-06T07:30:23.431311+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:23.195487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352878+0000","last_change":"2024-03-06T07:30:21.367059+0000","last_active":"2024-03-06T07:30:23.352878+0000","last_peered":"2024-03-06T07:30:23.352878+0000","last_clean":"2024-03-06T07:30:23.352878+0000","last_became_active":"2024-03-06T07:30:21.366691+0000","last_became_peered":"2024-03-06T07:30:21.366691+0000","last_unstale":"2024-03-06T07:30:23.352878+0000","last_undegraded":"2024-03-06T07:30:23.352878+0000","last_fullsized":"2024-03-06T07:30:23.352878+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:30.332872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352770+0000","last_change":"2024-03-06T07:30:22.372180+0000","last_active":"2024-03-06T07:30:23.352770+0000","last_peered":"2024-03-06T07:30:23.352770+0000","last_clean":"2024-03-06T07:30:23.352770+0000","last_became_active":"2024-03-06T07:30:22.371861+0000","last_became_peered":"2024-03-06T07:30:22.371861+0000","last_unstale":"2024-03-06T07:30:23.352770+0000","last_undegraded":"2024-03-06T07:30:23.352770+0000","last_fullsized":"2024-03-06T07:30:23.352770+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:21.458852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350175+0000","last_change":"2024-03-06T07:30:21.363328+0000","last_active":"2024-03-06T07:30:23.350175+0000","last_peered":"2024-03-06T07:30:23.350175+0000","last_clean":"2024-03-06T07:30:23.350175+0000","last_became_active":"2024-03-06T07:30:21.362738+0000","last_became_peered":"2024-03-06T07:30:21.362738+0000","last_unstale":"2024-03-06T07:30:23.350175+0000","last_undegraded":"2024-03-06T07:30:23.350175+0000","last_fullsized":"2024-03-06T07:30:23.350175+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:24.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372309+0000","last_change":"2024-03-06T07:30:22.362197+0000","last_active":"2024-03-06T07:30:24.372309+0000","last_peered":"2024-03-06T07:30:24.372309+0000","last_clean":"2024-03-06T07:30:24.372309+0000","last_became_active":"2024-03-06T07:30:22.361837+0000","last_became_peered":"2024-03-06T07:30:22.361837+0000","last_unstale":"2024-03-06T07:30:24.372309+0000","last_undegraded":"2024-03-06T07:30:24.372309+0000","last_fullsized":"2024-03-06T07:30:24.372309+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:05.657203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"131'3","reported_seq":17,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352551+0000","last_change":"2024-03-06T07:30:21.366972+0000","last_active":"2024-03-06T07:30:23.352551+0000","last_peered":"2024-03-06T07:30:23.352551+0000","last_clean":"2024-03-06T07:30:23.352551+0000","last_became_active":"2024-03-06T07:30:21.366673+0000","last_became_peered":"2024-03-06T07:30:21.366673+0000","last_unstale":"2024-03-06T07:30:23.352551+0000","last_undegraded":"2024-03-06T07:30:23.352551+0000","last_fullsized":"2024-03-06T07:30:23.352551+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:14.079602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371972+0000","last_change":"2024-03-06T07:30:22.366102+0000","last_active":"2024-03-06T07:30:24.371972+0000","last_peered":"2024-03-06T07:30:24.371972+0000","last_clean":"2024-03-06T07:30:24.371972+0000","last_became_active":"2024-03-06T07:30:22.365791+0000","last_became_peered":"2024-03-06T07:30:22.365791+0000","last_unstale":"2024-03-06T07:30:24.371972+0000","last_undegraded":"2024-03-06T07:30:24.371972+0000","last_fullsized":"2024-03-06T07:30:24.371972+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:55.176659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"131'6","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387084+0000","last_change":"2024-03-06T07:30:21.356331+0000","last_active":"2024-03-06T07:30:24.387084+0000","last_peered":"2024-03-06T07:30:24.387084+0000","last_clean":"2024-03-06T07:30:24.387084+0000","last_became_active":"2024-03-06T07:30:21.355951+0000","last_became_peered":"2024-03-06T07:30:21.355951+0000","last_unstale":"2024-03-06T07:30:24.387084+0000","last_undegraded":"2024-03-06T07:30:24.387084+0000","last_fullsized":"2024-03-06T07:30:24.387084+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:20.451575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431362+0000","last_change":"2024-03-06T07:30:22.358246+0000","last_active":"2024-03-06T07:30:23.431362+0000","last_peered":"2024-03-06T07:30:23.431362+0000","last_clean":"2024-03-06T07:30:23.431362+0000","last_became_active":"2024-03-06T07:30:22.357980+0000","last_became_peered":"2024-03-06T07:30:22.357980+0000","last_unstale":"2024-03-06T07:30:23.431362+0000","last_undegraded":"2024-03-06T07:30:23.431362+0000","last_fullsized":"2024-03-06T07:30:23.431362+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:31.340149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431053+0000","last_change":"2024-03-06T07:30:21.361081+0000","last_active":"2024-03-06T07:30:23.431053+0000","last_peered":"2024-03-06T07:30:23.431053+0000","last_clean":"2024-03-06T07:30:23.431053+0000","last_became_active":"2024-03-06T07:30:21.360771+0000","last_became_peered":"2024-03-06T07:30:21.360771+0000","last_unstale":"2024-03-06T07:30:23.431053+0000","last_undegraded":"2024-03-06T07:30:23.431053+0000","last_fullsized":"2024-03-06T07:30:23.431053+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:53.587987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346448+0000","last_change":"2024-03-06T07:30:22.369634+0000","last_active":"2024-03-06T07:30:23.346448+0000","last_peered":"2024-03-06T07:30:23.346448+0000","last_clean":"2024-03-06T07:30:23.346448+0000","last_became_active":"2024-03-06T07:30:22.369006+0000","last_became_peered":"2024-03-06T07:30:22.369006+0000","last_unstale":"2024-03-06T07:30:23.346448+0000","last_undegraded":"2024-03-06T07:30:23.346448+0000","last_fullsized":"2024-03-06T07:30:23.346448+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:31.340149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346583+0000","last_change":"2024-03-06T07:30:21.354002+0000","last_active":"2024-03-06T07:30:23.346583+0000","last_peered":"2024-03-06T07:30:23.346583+0000","last_clean":"2024-03-06T07:30:23.346583+0000","last_became_active":"2024-03-06T07:30:21.353735+0000","last_became_peered":"2024-03-06T07:30:21.353735+0000","last_unstale":"2024-03-06T07:30:23.346583+0000","last_undegraded":"2024-03-06T07:30:23.346583+0000","last_fullsized":"2024-03-06T07:30:23.346583+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:35.543995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387033+0000","last_change":"2024-03-06T07:30:22.365206+0000","last_active":"2024-03-06T07:30:24.387033+0000","last_peered":"2024-03-06T07:30:24.387033+0000","last_clean":"2024-03-06T07:30:24.387033+0000","last_became_active":"2024-03-06T07:30:22.364923+0000","last_became_peered":"2024-03-06T07:30:22.364923+0000","last_unstale":"2024-03-06T07:30:24.387033+0000","last_undegraded":"2024-03-06T07:30:24.387033+0000","last_fullsized":"2024-03-06T07:30:24.387033+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:09.380233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"131'8","reported_seq":22,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387349+0000","last_change":"2024-03-06T07:30:21.361469+0000","last_active":"2024-03-06T07:30:24.387349+0000","last_peered":"2024-03-06T07:30:24.387349+0000","last_clean":"2024-03-06T07:30:24.387349+0000","last_became_active":"2024-03-06T07:30:21.361140+0000","last_became_peered":"2024-03-06T07:30:21.361140+0000","last_unstale":"2024-03-06T07:30:24.387349+0000","last_undegraded":"2024-03-06T07:30:24.387349+0000","last_fullsized":"2024-03-06T07:30:24.387349+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:25.276018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.387150+0000","last_change":"2024-03-06T07:30:22.366631+0000","last_active":"2024-03-06T07:30:24.387150+0000","last_peered":"2024-03-06T07:30:24.387150+0000","last_clean":"2024-03-06T07:30:24.387150+0000","last_became_active":"2024-03-06T07:30:22.366379+0000","last_became_peered":"2024-03-06T07:30:22.366379+0000","last_unstale":"2024-03-06T07:30:24.387150+0000","last_undegraded":"2024-03-06T07:30:24.387150+0000","last_fullsized":"2024-03-06T07:30:24.387150+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:25.165735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352913+0000","last_change":"2024-03-06T07:30:21.358387+0000","last_active":"2024-03-06T07:30:23.352913+0000","last_peered":"2024-03-06T07:30:23.352913+0000","last_clean":"2024-03-06T07:30:23.352913+0000","last_became_active":"2024-03-06T07:30:21.358111+0000","last_became_peered":"2024-03-06T07:30:21.358111+0000","last_unstale":"2024-03-06T07:30:23.352913+0000","last_undegraded":"2024-03-06T07:30:23.352913+0000","last_fullsized":"2024-03-06T07:30:23.352913+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:22.188210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352844+0000","last_change":"2024-03-06T07:30:22.368146+0000","last_active":"2024-03-06T07:30:23.352844+0000","last_peered":"2024-03-06T07:30:23.352844+0000","last_clean":"2024-03-06T07:30:23.352844+0000","last_became_active":"2024-03-06T07:30:22.367832+0000","last_became_peered":"2024-03-06T07:30:22.367832+0000","last_unstale":"2024-03-06T07:30:23.352844+0000","last_undegraded":"2024-03-06T07:30:23.352844+0000","last_fullsized":"2024-03-06T07:30:23.352844+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:36.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884244+0000","last_change":"2024-03-06T07:30:21.363854+0000","last_active":"2024-03-06T07:30:24.884244+0000","last_peered":"2024-03-06T07:30:24.884244+0000","last_clean":"2024-03-06T07:30:24.884244+0000","last_became_active":"2024-03-06T07:30:21.363482+0000","last_became_peered":"2024-03-06T07:30:21.363482+0000","last_unstale":"2024-03-06T07:30:24.884244+0000","last_undegraded":"2024-03-06T07:30:24.884244+0000","last_fullsized":"2024-03-06T07:30:24.884244+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:07.251047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.350076+0000","last_change":"2024-03-06T07:30:22.366657+0000","last_active":"2024-03-06T07:30:23.350076+0000","last_peered":"2024-03-06T07:30:23.350076+0000","last_clean":"2024-03-06T07:30:23.350076+0000","last_became_active":"2024-03-06T07:30:22.366369+0000","last_became_peered":"2024-03-06T07:30:22.366369+0000","last_unstale":"2024-03-06T07:30:23.350076+0000","last_undegraded":"2024-03-06T07:30:23.350076+0000","last_fullsized":"2024-03-06T07:30:23.350076+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:55.176659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"131'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374763+0000","last_change":"2024-03-06T07:30:21.359923+0000","last_active":"2024-03-06T07:30:24.374763+0000","last_peered":"2024-03-06T07:30:24.374763+0000","last_clean":"2024-03-06T07:30:24.374763+0000","last_became_active":"2024-03-06T07:30:21.359625+0000","last_became_peered":"2024-03-06T07:30:21.359625+0000","last_unstale":"2024-03-06T07:30:24.374763+0000","last_undegraded":"2024-03-06T07:30:24.374763+0000","last_fullsized":"2024-03-06T07:30:24.374763+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:02.605754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.347038+0000","last_change":"2024-03-06T07:30:22.363836+0000","last_active":"2024-03-06T07:30:23.347038+0000","last_peered":"2024-03-06T07:30:23.347038+0000","last_clean":"2024-03-06T07:30:23.347038+0000","last_became_active":"2024-03-06T07:30:22.363542+0000","last_became_peered":"2024-03-06T07:30:22.363542+0000","last_unstale":"2024-03-06T07:30:23.347038+0000","last_undegraded":"2024-03-06T07:30:23.347038+0000","last_fullsized":"2024-03-06T07:30:23.347038+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:36.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352767+0000","last_change":"2024-03-06T07:30:21.362922+0000","last_active":"2024-03-06T07:30:23.352767+0000","last_peered":"2024-03-06T07:30:23.352767+0000","last_clean":"2024-03-06T07:30:23.352767+0000","last_became_active":"2024-03-06T07:30:21.362655+0000","last_became_peered":"2024-03-06T07:30:21.362655+0000","last_unstale":"2024-03-06T07:30:23.352767+0000","last_undegraded":"2024-03-06T07:30:23.352767+0000","last_fullsized":"2024-03-06T07:30:23.352767+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:54.618131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431020+0000","last_change":"2024-03-06T07:30:22.351632+0000","last_active":"2024-03-06T07:30:23.431020+0000","last_peered":"2024-03-06T07:30:23.431020+0000","last_clean":"2024-03-06T07:30:23.431020+0000","last_became_active":"2024-03-06T07:30:22.351317+0000","last_became_peered":"2024-03-06T07:30:22.351317+0000","last_unstale":"2024-03-06T07:30:23.431020+0000","last_undegraded":"2024-03-06T07:30:23.431020+0000","last_fullsized":"2024-03-06T07:30:23.431020+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:31.183751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352785+0000","last_change":"2024-03-06T07:30:21.366829+0000","last_active":"2024-03-06T07:30:23.352785+0000","last_peered":"2024-03-06T07:30:23.352785+0000","last_clean":"2024-03-06T07:30:23.352785+0000","last_became_active":"2024-03-06T07:30:21.366553+0000","last_became_peered":"2024-03-06T07:30:21.366553+0000","last_unstale":"2024-03-06T07:30:23.352785+0000","last_undegraded":"2024-03-06T07:30:23.352785+0000","last_fullsized":"2024-03-06T07:30:23.352785+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:34.258609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374742+0000","last_change":"2024-03-06T07:30:22.359828+0000","last_active":"2024-03-06T07:30:24.374742+0000","last_peered":"2024-03-06T07:30:24.374742+0000","last_clean":"2024-03-06T07:30:24.374742+0000","last_became_active":"2024-03-06T07:30:22.359522+0000","last_became_peered":"2024-03-06T07:30:22.359522+0000","last_unstale":"2024-03-06T07:30:24.374742+0000","last_undegraded":"2024-03-06T07:30:24.374742+0000","last_fullsized":"2024-03-06T07:30:24.374742+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:09.380233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349866+0000","last_change":"2024-03-06T07:30:21.362535+0000","last_active":"2024-03-06T07:30:23.349866+0000","last_peered":"2024-03-06T07:30:23.349866+0000","last_clean":"2024-03-06T07:30:23.349866+0000","last_became_active":"2024-03-06T07:30:21.357316+0000","last_became_peered":"2024-03-06T07:30:21.357316+0000","last_unstale":"2024-03-06T07:30:23.349866+0000","last_undegraded":"2024-03-06T07:30:23.349866+0000","last_fullsized":"2024-03-06T07:30:23.349866+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:53.390314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346633+0000","last_change":"2024-03-06T07:30:22.357248+0000","last_active":"2024-03-06T07:30:23.346633+0000","last_peered":"2024-03-06T07:30:23.346633+0000","last_clean":"2024-03-06T07:30:23.346633+0000","last_became_active":"2024-03-06T07:30:22.356994+0000","last_became_peered":"2024-03-06T07:30:22.356994+0000","last_unstale":"2024-03-06T07:30:23.346633+0000","last_undegraded":"2024-03-06T07:30:23.346633+0000","last_fullsized":"2024-03-06T07:30:23.346633+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:32.253438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349665+0000","last_change":"2024-03-06T07:30:21.360791+0000","last_active":"2024-03-06T07:30:23.349665+0000","last_peered":"2024-03-06T07:30:23.349665+0000","last_clean":"2024-03-06T07:30:23.349665+0000","last_became_active":"2024-03-06T07:30:21.360285+0000","last_became_peered":"2024-03-06T07:30:21.360285+0000","last_unstale":"2024-03-06T07:30:23.349665+0000","last_undegraded":"2024-03-06T07:30:23.349665+0000","last_fullsized":"2024-03-06T07:30:23.349665+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:59.638155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371639+0000","last_change":"2024-03-06T07:30:22.366620+0000","last_active":"2024-03-06T07:30:24.371639+0000","last_peered":"2024-03-06T07:30:24.371639+0000","last_clean":"2024-03-06T07:30:24.371639+0000","last_became_active":"2024-03-06T07:30:22.366345+0000","last_became_peered":"2024-03-06T07:30:22.366345+0000","last_unstale":"2024-03-06T07:30:24.371639+0000","last_undegraded":"2024-03-06T07:30:24.371639+0000","last_fullsized":"2024-03-06T07:30:24.371639+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:17.390996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.883893+0000","last_change":"2024-03-06T07:30:21.356249+0000","last_active":"2024-03-06T07:30:24.883893+0000","last_peered":"2024-03-06T07:30:24.883893+0000","last_clean":"2024-03-06T07:30:24.883893+0000","last_became_active":"2024-03-06T07:30:21.355930+0000","last_became_peered":"2024-03-06T07:30:21.355930+0000","last_unstale":"2024-03-06T07:30:24.883893+0000","last_undegraded":"2024-03-06T07:30:24.883893+0000","last_fullsized":"2024-03-06T07:30:24.883893+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:24.632711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374949+0000","last_change":"2024-03-06T07:30:22.363606+0000","last_active":"2024-03-06T07:30:24.374949+0000","last_peered":"2024-03-06T07:30:24.374949+0000","last_clean":"2024-03-06T07:30:24.374949+0000","last_became_active":"2024-03-06T07:30:22.363190+0000","last_became_peered":"2024-03-06T07:30:22.363190+0000","last_unstale":"2024-03-06T07:30:24.374949+0000","last_undegraded":"2024-03-06T07:30:24.374949+0000","last_fullsized":"2024-03-06T07:30:24.374949+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:42.982021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431325+0000","last_change":"2024-03-06T07:30:21.365785+0000","last_active":"2024-03-06T07:30:23.431325+0000","last_peered":"2024-03-06T07:30:23.431325+0000","last_clean":"2024-03-06T07:30:23.431325+0000","last_became_active":"2024-03-06T07:30:21.365483+0000","last_became_peered":"2024-03-06T07:30:21.365483+0000","last_unstale":"2024-03-06T07:30:23.431325+0000","last_undegraded":"2024-03-06T07:30:23.431325+0000","last_fullsized":"2024-03-06T07:30:23.431325+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:57.830418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.431267+0000","last_change":"2024-03-06T07:30:22.359840+0000","last_active":"2024-03-06T07:30:23.431267+0000","last_peered":"2024-03-06T07:30:23.431267+0000","last_clean":"2024-03-06T07:30:23.431267+0000","last_became_active":"2024-03-06T07:30:22.359489+0000","last_became_peered":"2024-03-06T07:30:22.359489+0000","last_unstale":"2024-03-06T07:30:23.431267+0000","last_undegraded":"2024-03-06T07:30:23.431267+0000","last_fullsized":"2024-03-06T07:30:23.431267+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:55.625408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"132'1","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.373736+0000","last_change":"2024-03-06T07:30:21.355058+0000","last_active":"2024-03-06T07:30:24.373736+0000","last_peered":"2024-03-06T07:30:24.373736+0000","last_clean":"2024-03-06T07:30:24.373736+0000","last_became_active":"2024-03-06T07:30:21.354785+0000","last_became_peered":"2024-03-06T07:30:21.354785+0000","last_unstale":"2024-03-06T07:30:24.373736+0000","last_undegraded":"2024-03-06T07:30:24.373736+0000","last_fullsized":"2024-03-06T07:30:24.373736+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:59.638155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346843+0000","last_change":"2024-03-06T07:30:22.361590+0000","last_active":"2024-03-06T07:30:23.346843+0000","last_peered":"2024-03-06T07:30:23.346843+0000","last_clean":"2024-03-06T07:30:23.346843+0000","last_became_active":"2024-03-06T07:30:22.361304+0000","last_became_peered":"2024-03-06T07:30:22.361304+0000","last_unstale":"2024-03-06T07:30:23.346843+0000","last_undegraded":"2024-03-06T07:30:23.346843+0000","last_fullsized":"2024-03-06T07:30:23.346843+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:21.458852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.347039+0000","last_change":"2024-03-06T07:30:21.357925+0000","last_active":"2024-03-06T07:30:23.347039+0000","last_peered":"2024-03-06T07:30:23.347039+0000","last_clean":"2024-03-06T07:30:23.347039+0000","last_became_active":"2024-03-06T07:30:21.357621+0000","last_became_peered":"2024-03-06T07:30:21.357621+0000","last_unstale":"2024-03-06T07:30:23.347039+0000","last_undegraded":"2024-03-06T07:30:23.347039+0000","last_fullsized":"2024-03-06T07:30:23.347039+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:44.851584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.374816+0000","last_change":"2024-03-06T07:30:22.358228+0000","last_active":"2024-03-06T07:30:24.374816+0000","last_peered":"2024-03-06T07:30:24.374816+0000","last_clean":"2024-03-06T07:30:24.374816+0000","last_became_active":"2024-03-06T07:30:22.357597+0000","last_became_peered":"2024-03-06T07:30:22.357597+0000","last_unstale":"2024-03-06T07:30:24.374816+0000","last_undegraded":"2024-03-06T07:30:24.374816+0000","last_fullsized":"2024-03-06T07:30:24.374816+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:16.844706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884150+0000","last_change":"2024-03-06T07:30:21.364224+0000","last_active":"2024-03-06T07:30:24.884150+0000","last_peered":"2024-03-06T07:30:24.884150+0000","last_clean":"2024-03-06T07:30:24.884150+0000","last_became_active":"2024-03-06T07:30:21.363924+0000","last_became_peered":"2024-03-06T07:30:21.363924+0000","last_unstale":"2024-03-06T07:30:24.884150+0000","last_undegraded":"2024-03-06T07:30:24.884150+0000","last_fullsized":"2024-03-06T07:30:24.884150+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:29.122685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371708+0000","last_change":"2024-03-06T07:30:22.362379+0000","last_active":"2024-03-06T07:30:24.371708+0000","last_peered":"2024-03-06T07:30:24.371708+0000","last_clean":"2024-03-06T07:30:24.371708+0000","last_became_active":"2024-03-06T07:30:22.362091+0000","last_became_peered":"2024-03-06T07:30:22.362091+0000","last_unstale":"2024-03-06T07:30:24.371708+0000","last_undegraded":"2024-03-06T07:30:24.371708+0000","last_fullsized":"2024-03-06T07:30:24.371708+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:20.391472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346626+0000","last_change":"2024-03-06T07:30:21.354681+0000","last_active":"2024-03-06T07:30:23.346626+0000","last_peered":"2024-03-06T07:30:23.346626+0000","last_clean":"2024-03-06T07:30:23.346626+0000","last_became_active":"2024-03-06T07:30:21.354399+0000","last_became_peered":"2024-03-06T07:30:21.354399+0000","last_unstale":"2024-03-06T07:30:23.346626+0000","last_undegraded":"2024-03-06T07:30:23.346626+0000","last_fullsized":"2024-03-06T07:30:23.346626+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:56.731087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.352625+0000","last_change":"2024-03-06T07:30:22.368255+0000","last_active":"2024-03-06T07:30:23.352625+0000","last_peered":"2024-03-06T07:30:23.352625+0000","last_clean":"2024-03-06T07:30:23.352625+0000","last_became_active":"2024-03-06T07:30:22.367968+0000","last_became_peered":"2024-03-06T07:30:22.367968+0000","last_unstale":"2024-03-06T07:30:23.352625+0000","last_undegraded":"2024-03-06T07:30:23.352625+0000","last_fullsized":"2024-03-06T07:30:23.352625+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:03.613030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371788+0000","last_change":"2024-03-06T07:30:21.359774+0000","last_active":"2024-03-06T07:30:24.371788+0000","last_peered":"2024-03-06T07:30:24.371788+0000","last_clean":"2024-03-06T07:30:24.371788+0000","last_became_active":"2024-03-06T07:30:21.359457+0000","last_became_peered":"2024-03-06T07:30:21.359457+0000","last_unstale":"2024-03-06T07:30:24.371788+0000","last_undegraded":"2024-03-06T07:30:24.371788+0000","last_fullsized":"2024-03-06T07:30:24.371788+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:41.974744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349743+0000","last_change":"2024-03-06T07:30:22.369678+0000","last_active":"2024-03-06T07:30:23.349743+0000","last_peered":"2024-03-06T07:30:23.349743+0000","last_clean":"2024-03-06T07:30:23.349743+0000","last_became_active":"2024-03-06T07:30:22.368569+0000","last_became_peered":"2024-03-06T07:30:22.368569+0000","last_unstale":"2024-03-06T07:30:23.349743+0000","last_undegraded":"2024-03-06T07:30:23.349743+0000","last_fullsized":"2024-03-06T07:30:23.349743+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:55.690989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.349753+0000","last_change":"2024-03-06T07:30:21.359533+0000","last_active":"2024-03-06T07:30:23.349753+0000","last_peered":"2024-03-06T07:30:23.349753+0000","last_clean":"2024-03-06T07:30:23.349753+0000","last_became_active":"2024-03-06T07:30:21.359211+0000","last_became_peered":"2024-03-06T07:30:21.359211+0000","last_unstale":"2024-03-06T07:30:23.349753+0000","last_undegraded":"2024-03-06T07:30:23.349753+0000","last_fullsized":"2024-03-06T07:30:23.349753+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:34.007530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:23.346584+0000","last_change":"2024-03-06T07:30:22.364308+0000","last_active":"2024-03-06T07:30:23.346584+0000","last_peered":"2024-03-06T07:30:23.346584+0000","last_clean":"2024-03-06T07:30:23.346584+0000","last_became_active":"2024-03-06T07:30:22.363848+0000","last_became_peered":"2024-03-06T07:30:22.363848+0000","last_unstale":"2024-03-06T07:30:23.346584+0000","last_undegraded":"2024-03-06T07:30:23.346584+0000","last_fullsized":"2024-03-06T07:30:23.346584+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:49.425487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.884014+0000","last_change":"2024-03-06T07:30:21.364208+0000","last_active":"2024-03-06T07:30:24.884014+0000","last_peered":"2024-03-06T07:30:24.884014+0000","last_clean":"2024-03-06T07:30:24.884014+0000","last_became_active":"2024-03-06T07:30:21.363905+0000","last_became_peered":"2024-03-06T07:30:21.363905+0000","last_unstale":"2024-03-06T07:30:24.884014+0000","last_undegraded":"2024-03-06T07:30:24.884014+0000","last_fullsized":"2024-03-06T07:30:24.884014+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:56.526496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371667+0000","last_change":"2024-03-06T07:30:22.367439+0000","last_active":"2024-03-06T07:30:24.371667+0000","last_peered":"2024-03-06T07:30:24.371667+0000","last_clean":"2024-03-06T07:30:24.371667+0000","last_became_active":"2024-03-06T07:30:22.367142+0000","last_became_peered":"2024-03-06T07:30:22.367142+0000","last_unstale":"2024-03-06T07:30:24.371667+0000","last_undegraded":"2024-03-06T07:30:24.371667+0000","last_fullsized":"2024-03-06T07:30:24.371667+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:21.334700+0000","last_clean_scrub_stamp":"2024-03-06T07:30:21.334700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:55.690989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"131'2","reported_seq":16,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.372386+0000","last_change":"2024-03-06T07:30:21.355999+0000","last_active":"2024-03-06T07:30:24.372386+0000","last_peered":"2024-03-06T07:30:24.372386+0000","last_clean":"2024-03-06T07:30:24.372386+0000","last_became_active":"2024-03-06T07:30:21.355716+0000","last_became_peered":"2024-03-06T07:30:21.355716+0000","last_unstale":"2024-03-06T07:30:24.372386+0000","last_undegraded":"2024-03-06T07:30:24.372386+0000","last_fullsized":"2024-03-06T07:30:24.372386+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.327423+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.327423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:24.232184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":382,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-06T07:30:24.371479+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:30:24.371479+0000","last_peered":"2024-03-06T07:30:24.371479+0000","last_clean":"2024-03-06T07:30:24.371479+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:30:24.371479+0000","last_undegraded":"2024-03-06T07:30:24.371479+0000","last_fullsized":"2024-03-06T07:30:24.371479+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574945,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41712,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330128,"statfs":{"total":96636764160,"available":96594051072,"internally_reserved":0,"allocated":2265088,"data_stored":1310118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574946,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41116,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330724,"statfs":{"total":96636764160,"available":96594661376,"internally_reserved":0,"allocated":1646592,"data_stored":712812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41116,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330724,"statfs":{"total":96636764160,"available":96594661376,"internally_reserved":0,"allocated":1654784,"data_stored":717612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574945,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":1650688,"data_stored":717570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41120,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330720,"statfs":{"total":96636764160,"available":96594657280,"internally_reserved":0,"allocated":1654784,"data_stored":717660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574945,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41120,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330720,"statfs":{"total":96636764160,"available":96594657280,"internally_reserved":0,"allocated":1654784,"data_stored":718116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":1646592,"data_stored":717500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574946,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41684,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330156,"statfs":{"total":96636764160,"available":96594079744,"internally_reserved":0,"allocated":2228224,"data_stored":1301568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:26.770 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:30:26.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:30:26.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.920+0000 7f057189a640 1 Processor -- start 2024-03-06T07:30:26.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.920+0000 7f057189a640 1 -- start start 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f057189a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f056c0afad0 0x7f056c0b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f057189a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 0x7f056c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f057189a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f056c066d00 0x7f056c067170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f057189a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f056c0676b0 con 0x7f056c066d00 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f057189a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f056c0afeb0 con 0x7f056c0afad0 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f057189a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c06dd50 con 0x7f056c1478c0 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f056c066d00 0x7f056c067170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f056c066d00 0x7f056c067170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56404/0 (socket says 172.21.15.27:56404) 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056b7fe640 1 -- 172.21.15.27:0/721551573 learned_addr learned my addr 172.21.15.27:0/721551573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056affd640 1 --2- 172.21.15.27:0/721551573 >> v2:172.21.15.69:3300/0 conn(0x7f056c0afad0 0x7f056c0b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056a7fc640 1 --2- 172.21.15.27:0/721551573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 0x7f056c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056b7fe640 1 -- 172.21.15.27:0/721551573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 msgr2=0x7f056c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056b7fe640 1 --2- 172.21.15.27:0/721551573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 0x7f056c0ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056b7fe640 1 -- 172.21.15.27:0/721551573 >> v2:172.21.15.69:3300/0 conn(0x7f056c0afad0 msgr2=0x7f056c0b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056b7fe640 1 --2- 172.21.15.27:0/721551573 >> v2:172.21.15.69:3300/0 conn(0x7f056c0afad0 0x7f056c0b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056b7fe640 1 -- 172.21.15.27:0/721551573 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c06e030 con 0x7f056c066d00 2024-03-06T07:30:26.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.921+0000 7f056a7fc640 1 --2- 172.21.15.27:0/721551573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 0x7f056c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.922+0000 7f056b7fe640 1 --2- 172.21.15.27:0/721551573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 0x7f056c067170 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f056c073710 tx=0x7f056402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b3ab182e7f6ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.922+0000 7f0569ffb640 1 -- 172.21.15.27:0/721551573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056402ca70 con 0x7f056c066d00 2024-03-06T07:30:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.922+0000 7f0569ffb640 1 -- 172.21.15.27:0/721551573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056402cc10 con 0x7f056c066d00 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.922+0000 7f0569ffb640 1 -- 172.21.15.27:0/721551573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05640343f0 con 0x7f056c066d00 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.923+0000 7f057189a640 1 -- 172.21.15.27:0/721551573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 msgr2=0x7f056c067170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.923+0000 7f057189a640 1 --2- 172.21.15.27:0/721551573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 0x7f056c067170 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f056c073710 tx=0x7f056402bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.923+0000 7f057189a640 1 -- 172.21.15.27:0/721551573 shutdown_connections 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.923+0000 7f057189a640 1 --2- 172.21.15.27:0/721551573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 0x7f056c067170 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.923+0000 7f057189a640 1 --2- 172.21.15.27:0/721551573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 0x7f056c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.923+0000 7f057189a640 1 --2- 172.21.15.27:0/721551573 >> v2:172.21.15.69:3300/0 conn(0x7f056c0afad0 0x7f056c0b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.923+0000 7f057189a640 1 -- 172.21.15.27:0/721551573 >> 172.21.15.27:0/721551573 conn(0x7f056c0b2790 msgr2=0x7f056c0b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.923+0000 7f057189a640 1 -- 172.21.15.27:0/721551573 shutdown_connections 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.923+0000 7f057189a640 1 -- 172.21.15.27:0/721551573 wait complete. 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.924+0000 7f057189a640 1 Processor -- start 2024-03-06T07:30:26.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.924+0000 7f057189a640 1 -- start start 2024-03-06T07:30:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f057189a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 0x7f056c0c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f057189a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f056c0afad0 0x7f056c145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f057189a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 0x7f056c146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f057189a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c06e8d0 con 0x7f056c066d00 2024-03-06T07:30:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f057189a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c06e5f0 con 0x7f056c0afad0 2024-03-06T07:30:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f057189a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c06da00 con 0x7f056c1478c0 2024-03-06T07:30:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 0x7f056c0c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 0x7f056c0c8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56410/0 (socket says 172.21.15.27:56410) 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056affd640 1 -- 172.21.15.27:0/613723759 learned_addr learned my addr 172.21.15.27:0/613723759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056a7fc640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f056c0afad0 0x7f056c145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056b7fe640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 0x7f056c146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056affd640 1 -- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 msgr2=0x7f056c146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056affd640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 0x7f056c146210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056affd640 1 -- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f056c0afad0 msgr2=0x7f056c145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056affd640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f056c0afad0 0x7f056c145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.925+0000 7f056affd640 1 -- 172.21.15.27:0/613723759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0d04b0 con 0x7f056c066d00 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.926+0000 7f056affd640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 0x7f056c0c8780 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f055400e870 tx=0x7f055400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.926+0000 7f0563fff640 1 -- 172.21.15.27:0/613723759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0554017070 con 0x7f056c066d00 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.926+0000 7f0563fff640 1 -- 172.21.15.27:0/613723759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055400ee60 con 0x7f056c066d00 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.926+0000 7f0563fff640 1 -- 172.21.15.27:0/613723759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055400cb80 con 0x7f056c066d00 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.926+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c0c6be0 con 0x7f056c066d00 2024-03-06T07:30:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.926+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c0c7480 con 0x7f056c066d00 2024-03-06T07:30:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.927+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c04f410 con 0x7f056c066d00 2024-03-06T07:30:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.928+0000 7f0563fff640 1 -- 172.21.15.27:0/613723759 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f055401a070 con 0x7f056c066d00 2024-03-06T07:30:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.928+0000 7f0563fff640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f054c06b4d0 0x7f054c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.928+0000 7f0563fff640 1 -- 172.21.15.27:0/613723759 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0554089fd0 con 0x7f056c066d00 2024-03-06T07:30:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.929+0000 7f056a7fc640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f054c06b4d0 0x7f054c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.929+0000 7f056a7fc640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f054c06b4d0 0x7f054c06d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f056c0c7af0 tx=0x7f055c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:26.933+0000 7f0563fff640 1 -- 172.21.15.27:0/613723759 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0554058800 con 0x7f056c066d00 2024-03-06T07:30:27.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.181+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f056c0ae440 con 0x7f056c066d00 2024-03-06T07:30:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.183+0000 7f0563fff640 1 -- 172.21.15.27:0/613723759 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f055405c8a0 con 0x7f056c066d00 2024-03-06T07:30:27.185 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:30:27.185 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:30:27.185 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-03-06T07:30:27.185 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-03-06T07:30:27.185 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-03-06T07:30:27.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f054c06b4d0 msgr2=0x7f054c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f054c06b4d0 0x7f054c06d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f056c0c7af0 tx=0x7f055c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 msgr2=0x7f056c0c8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 0x7f056c0c8780 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f055400e870 tx=0x7f055400ed30 comp rx=0 tx=0).stop 2024-03-06T07:30:27.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 shutdown_connections 2024-03-06T07:30:27.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f054c06b4d0 0x7f054c06d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f056c1478c0 0x7f056c146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f056c0afad0 0x7f056c145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 --2- 172.21.15.27:0/613723759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f056c066d00 0x7f056c0c8780 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.185+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 >> 172.21.15.27:0/613723759 conn(0x7f056c0b2790 msgr2=0x7f056c0adac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.186+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 shutdown_connections 2024-03-06T07:30:27.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.186+0000 7f057189a640 1 -- 172.21.15.27:0/613723759 wait complete. 2024-03-06T07:30:27.198 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.321+0000 7fb04ff7a640 1 Processor -- start 2024-03-06T07:30:27.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.321+0000 7fb04ff7a640 1 -- start start 2024-03-06T07:30:27.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.322+0000 7fb04ff7a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb048157f60 0x7fb04815b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.322+0000 7fb04ff7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 0x7fb048070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.322+0000 7fb04ff7a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04806c7b0 con 0x7fb048157b80 2024-03-06T07:30:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.322+0000 7fb04ff7a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb04806ca90 con 0x7fb048157f60 2024-03-06T07:30:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.322+0000 7fb04ff7a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04806cd70 con 0x7fb0480cbab0 2024-03-06T07:30:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.323+0000 7fb04e517640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 0x7fb048070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.323+0000 7fb04d515640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb048157f60 0x7fb04815b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.323+0000 7fb04dd16640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb048157b80 0x7fb04815b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:33882/0 (socket says 172.21.15.27:33882) 2024-03-06T07:30:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.323+0000 7fb04d515640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb048157f60 0x7fb04815b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:38160/0 (socket says 172.21.15.27:38160) 2024-03-06T07:30:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.323+0000 7fb04dd16640 1 -- 172.21.15.27:0/1475185823 learned_addr learned my addr 172.21.15.27:0/1475185823 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.323+0000 7fb04d515640 1 -- 172.21.15.27:0/1475185823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 msgr2=0x7fb048070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.323+0000 7fb04d515640 1 --2- 172.21.15.27:0/1475185823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 0x7fb048070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.323+0000 7fb04d515640 1 -- 172.21.15.27:0/1475185823 >> v1:172.21.15.69:6789/0 conn(0x7fb048157b80 legacy=0x7fb04815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.323+0000 7fb04d515640 1 -- 172.21.15.27:0/1475185823 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04806d050 con 0x7fb048157f60 2024-03-06T07:30:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.324+0000 7fb04d515640 1 --2- 172.21.15.27:0/1475185823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb048157f60 0x7fb04815b260 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb0380097a0 tx=0x7fb03802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=224045554a0f3000 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.324+0000 7fb04cd14640 1 -- 172.21.15.27:0/1475185823 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb038035020 con 0x7fb048157f60 2024-03-06T07:30:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.324+0000 7fb04cd14640 1 -- 172.21.15.27:0/1475185823 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb038031830 con 0x7fb048157f60 2024-03-06T07:30:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.324+0000 7fb04cd14640 1 -- 172.21.15.27:0/1475185823 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb038031c00 con 0x7fb048157f60 2024-03-06T07:30:27.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.325+0000 7fb04ff7a640 1 -- 172.21.15.27:0/1475185823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb048157f60 msgr2=0x7fb04815b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.325+0000 7fb04ff7a640 1 --2- 172.21.15.27:0/1475185823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb048157f60 0x7fb04815b260 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb0380097a0 tx=0x7fb03802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.325+0000 7fb04ff7a640 1 -- 172.21.15.27:0/1475185823 shutdown_connections 2024-03-06T07:30:27.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.325+0000 7fb04ff7a640 1 --2- 172.21.15.27:0/1475185823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb048157f60 0x7fb04815b260 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.325+0000 7fb04ff7a640 1 --2- 172.21.15.27:0/1475185823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 0x7fb048070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.325+0000 7fb04ff7a640 1 -- 172.21.15.27:0/1475185823 >> 172.21.15.27:0/1475185823 conn(0x7fb048066d00 msgr2=0x7fb0480670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.325+0000 7fb04ff7a640 1 -- 172.21.15.27:0/1475185823 shutdown_connections 2024-03-06T07:30:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.325+0000 7fb04ff7a640 1 -- 172.21.15.27:0/1475185823 wait complete. 2024-03-06T07:30:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.326+0000 7fb04ff7a640 1 Processor -- start 2024-03-06T07:30:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.326+0000 7fb04ff7a640 1 -- start start 2024-03-06T07:30:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04ff7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 0x7fb048151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04ff7a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb048157b80 0x7fb048148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04ff7a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb048157f60 0x7fb048153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04ff7a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb048071080 con 0x7fb048157f60 2024-03-06T07:30:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04ff7a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0480cc260 con 0x7fb048157b80 2024-03-06T07:30:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04ff7a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb048071200 con 0x7fb0480cbab0 2024-03-06T07:30:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04e517640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb048157f60 0x7fb048153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04e517640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb048157f60 0x7fb048153540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56422/0 (socket says 172.21.15.27:56422) 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04e517640 1 -- 172.21.15.27:0/3236227254 learned_addr learned my addr 172.21.15.27:0/3236227254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04dd16640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 0x7fb048151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04d515640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb048157b80 0x7fb048148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04e517640 1 -- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 msgr2=0x7fb048151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04e517640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 0x7fb048151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04e517640 1 -- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb048157b80 msgr2=0x7fb048148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04e517640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb048157b80 0x7fb048148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04e517640 1 -- 172.21.15.27:0/3236227254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb048153a80 con 0x7fb048157f60 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04e517640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb048157f60 0x7fb048153540 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fb044009e00 tx=0x7fb0440075e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb03e7fc640 1 -- 172.21.15.27:0/3236227254 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044019070 con 0x7fb048157f60 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb03e7fc640 1 -- 172.21.15.27:0/3236227254 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb044003b90 con 0x7fb048157f60 2024-03-06T07:30:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb03e7fc640 1 -- 172.21.15.27:0/3236227254 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044004030 con 0x7fb048157f60 2024-03-06T07:30:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0481521d0 con 0x7fb048157f60 2024-03-06T07:30:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.327+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb048152790 con 0x7fb048157f60 2024-03-06T07:30:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.329+0000 7fb03e7fc640 1 -- 172.21.15.27:0/3236227254 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb044005d10 con 0x7fb048157f60 2024-03-06T07:30:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.329+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb048067ff0 con 0x7fb048157f60 2024-03-06T07:30:27.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.329+0000 7fb03e7fc640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb02c06b500 0x7fb02c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.329+0000 7fb03e7fc640 1 -- 172.21.15.27:0/3236227254 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb04408a180 con 0x7fb048157f60 2024-03-06T07:30:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.330+0000 7fb04dd16640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb02c06b500 0x7fb02c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.330+0000 7fb04dd16640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb02c06b500 0x7fb02c06d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb0481f6f10 tx=0x7fb040007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.334+0000 7fb03e7fc640 1 -- 172.21.15.27:0/3236227254 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb044011070 con 0x7fb048157f60 2024-03-06T07:30:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.595+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0480d07a0 con 0x7fb048157f60 2024-03-06T07:30:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.596+0000 7fb03e7fc640 1 -- 172.21.15.27:0/3236227254 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fb044058a40 con 0x7fb048157f60 2024-03-06T07:30:27.597 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:27.597 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/643534493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":643534493},{"type":"v1","addr":"172.21.15.69:6837","nonce":643534493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":7783,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3158495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3158495906},{"type":"v1","addr":"172.21.15.69:6835","nonce":3158495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":8670,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:24.739429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb02c06b500 msgr2=0x7fb02c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb02c06b500 0x7fb02c06d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb0481f6f10 tx=0x7fb040007e80 comp rx=0 tx=0).stop 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb048157f60 msgr2=0x7fb048153540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb048157f60 0x7fb048153540 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fb044009e00 tx=0x7fb0440075e0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 shutdown_connections 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb02c06b500 0x7fb02c06d9c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb048157f60 0x7fb048153540 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb048157b80 0x7fb048148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 --2- 172.21.15.27:0/3236227254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0480cbab0 0x7fb048151bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 >> 172.21.15.27:0/3236227254 conn(0x7fb048066d00 msgr2=0x7fb048153fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 shutdown_connections 2024-03-06T07:30:27.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:27.598+0000 7fb04ff7a640 1 -- 172.21.15.27:0/3236227254 wait complete. 2024-03-06T07:30:28.012 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:24.739429+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8679}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:30:28.012 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:30:28.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:30:28.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.119+0000 7fa49d2a4640 1 Processor -- start 2024-03-06T07:30:28.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.119+0000 7fa49d2a4640 1 -- start start 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa49d2a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 0x7fa49806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa49d2a4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4980aee10 0x7fa4980b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa49d2a4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa498071240 con 0x7fa49806dd60 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa49d2a4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa49806e1c0 con 0x7fa4980aee10 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa49d2a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa498071b70 con 0x7fa498158520 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa4977fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4980aee10 0x7fa4980b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa48ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa49806dd60 0x7fa4980ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54696/0 (socket says 172.21.15.27:54696) 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa48ffff640 1 -- 172.21.15.27:0/3297747059 learned_addr learned my addr 172.21.15.27:0/3297747059 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa496ffd640 1 --2- 172.21.15.27:0/3297747059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 0x7fa49806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa4977fe640 1 -- 172.21.15.27:0/3297747059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 msgr2=0x7fa49806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa4977fe640 1 --2- 172.21.15.27:0/3297747059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 0x7fa49806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa4977fe640 1 -- 172.21.15.27:0/3297747059 >> v1:172.21.15.27:6789/0 conn(0x7fa49806dd60 legacy=0x7fa4980ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa4977fe640 1 -- 172.21.15.27:0/3297747059 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498071520 con 0x7fa4980aee10 2024-03-06T07:30:28.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa4967fc640 1 -- 172.21.15.27:0/3297747059 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650141728 0 0) 0x7fa498071240 con 0x7fa49806dd60 2024-03-06T07:30:28.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.120+0000 7fa496ffd640 1 --2- 172.21.15.27:0/3297747059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 0x7fa49806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.121+0000 7fa4977fe640 1 --2- 172.21.15.27:0/3297747059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 0x7fa4980b3640 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fa490009d90 tx=0x7fa49002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8517b40799fc34bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.121+0000 7fa4967fc640 1 -- 172.21.15.27:0/3297747059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490039070 con 0x7fa4980aee10 2024-03-06T07:30:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.121+0000 7fa4967fc640 1 -- 172.21.15.27:0/3297747059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49002cc50 con 0x7fa4980aee10 2024-03-06T07:30:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.121+0000 7fa4967fc640 1 -- 172.21.15.27:0/3297747059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490034440 con 0x7fa4980aee10 2024-03-06T07:30:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.122+0000 7fa49d2a4640 1 -- 172.21.15.27:0/3297747059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 msgr2=0x7fa4980b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.122+0000 7fa49d2a4640 1 --2- 172.21.15.27:0/3297747059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 0x7fa4980b3640 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fa490009d90 tx=0x7fa49002bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.122+0000 7fa49d2a4640 1 -- 172.21.15.27:0/3297747059 shutdown_connections 2024-03-06T07:30:28.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.122+0000 7fa49d2a4640 1 --2- 172.21.15.27:0/3297747059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 0x7fa4980b3640 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.122+0000 7fa49d2a4640 1 --2- 172.21.15.27:0/3297747059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 0x7fa49806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.122+0000 7fa49d2a4640 1 -- 172.21.15.27:0/3297747059 >> 172.21.15.27:0/3297747059 conn(0x7fa498066d00 msgr2=0x7fa4980670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.122+0000 7fa49d2a4640 1 -- 172.21.15.27:0/3297747059 shutdown_connections 2024-03-06T07:30:28.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.122+0000 7fa49d2a4640 1 -- 172.21.15.27:0/3297747059 wait complete. 2024-03-06T07:30:28.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.124+0000 7fa49d2a4640 1 Processor -- start 2024-03-06T07:30:28.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.124+0000 7fa49d2a4640 1 -- start start 2024-03-06T07:30:28.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.125+0000 7fa49d2a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa49806dd60 0x7fa49814a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.125+0000 7fa49d2a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 0x7fa4980a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.125+0000 7fa49d2a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 0x7fa4980a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.125+0000 7fa49d2a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa498158900 con 0x7fa4980aee10 2024-03-06T07:30:28.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.125+0000 7fa49d2a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa498072000 con 0x7fa49806dd60 2024-03-06T07:30:28.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.125+0000 7fa49d2a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa498158a80 con 0x7fa498158520 2024-03-06T07:30:28.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.125+0000 7fa48ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 0x7fa4980a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.125+0000 7fa48ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 0x7fa4980a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56436/0 (socket says 172.21.15.27:56436) 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.125+0000 7fa48ffff640 1 -- 172.21.15.27:0/4270433751 learned_addr learned my addr 172.21.15.27:0/4270433751 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa496ffd640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa49806dd60 0x7fa49814a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa4977fe640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 0x7fa4980a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa48ffff640 1 -- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 msgr2=0x7fa4980a99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa48ffff640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 0x7fa4980a99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa48ffff640 1 -- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa49806dd60 msgr2=0x7fa49814a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa48ffff640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa49806dd60 0x7fa49814a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa48ffff640 1 -- 172.21.15.27:0/4270433751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49815d780 con 0x7fa4980aee10 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa48ffff640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 0x7fa4980a94a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fa48000c4a0 tx=0x7fa48000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa48f7fe640 1 -- 172.21.15.27:0/4270433751 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480018070 con 0x7fa4980aee10 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa48f7fe640 1 -- 172.21.15.27:0/4270433751 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4800074e0 con 0x7fa4980aee10 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa49d2a4640 1 -- 172.21.15.27:0/4270433751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49815da60 con 0x7fa4980aee10 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa48f7fe640 1 -- 172.21.15.27:0/4270433751 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480004a20 con 0x7fa4980aee10 2024-03-06T07:30:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.126+0000 7fa49d2a4640 1 -- 172.21.15.27:0/4270433751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4980cd960 con 0x7fa4980aee10 2024-03-06T07:30:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.128+0000 7fa48f7fe640 1 -- 172.21.15.27:0/4270433751 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa48001b070 con 0x7fa4980aee10 2024-03-06T07:30:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.128+0000 7fa48f7fe640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa47806b4d0 0x7fa47806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.128+0000 7fa48f7fe640 1 -- 172.21.15.27:0/4270433751 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4800894e0 con 0x7fa4980aee10 2024-03-06T07:30:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.128+0000 7fa496ffd640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa47806b4d0 0x7fa47806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.129+0000 7fa496ffd640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa47806b4d0 0x7fa47806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa488006fd0 tx=0x7fa488008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.130+0000 7fa49ff6b740 1 Processor -- start 2024-03-06T07:30:28.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.130+0000 7fa49ff6b740 1 -- start start 2024-03-06T07:30:28.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.131+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240c94870 0x55c240c94c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.131+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240c988c0 0x55c240c9a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.131+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.131+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c240bfe6b0 con 0x55c240cc4600 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.131+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c240ba7a60 con 0x55c240c988c0 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.131+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c240bfa750 con 0x55c240c94870 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.131+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.131+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56452/0 (socket says 172.21.15.27:56452) 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.131+0000 7fa4977fe640 1 -- 172.21.15.27:0/1816733106 learned_addr learned my addr 172.21.15.27:0/1816733106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa48ffff640 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240c988c0 0x55c240c9a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa496ffd640 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240c94870 0x55c240c94c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa4977fe640 1 -- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240c94870 msgr2=0x55c240c94c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa4977fe640 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240c94870 0x55c240c94c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa4977fe640 1 -- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240c988c0 msgr2=0x55c240c9a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa4977fe640 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240c988c0 0x55c240c9a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa4977fe640 1 -- 172.21.15.27:0/1816733106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c240bc5f30 con 0x55c240cc4600 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa48ffff640 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240c988c0 0x55c240c9a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa496ffd640 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240c94870 0x55c240c94c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa4977fe640 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc4a80 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fa490009d90 tx=0x7fa49002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33c568c2fbe741a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa48cff9640 1 -- 172.21.15.27:0/1816733106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490039070 con 0x55c240cc4600 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa48cff9640 1 -- 172.21.15.27:0/1816733106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49003a040 con 0x55c240cc4600 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.132+0000 7fa48cff9640 1 -- 172.21.15.27:0/1816733106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49002cad0 con 0x55c240cc4600 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.133+0000 7fa49ff6b740 1 -- 172.21.15.27:0/1816733106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 msgr2=0x55c240cc4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.133+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc4a80 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fa490009d90 tx=0x7fa49002e000 comp rx=0 tx=0).stop 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.133+0000 7fa49ff6b740 1 -- 172.21.15.27:0/1816733106 shutdown_connections 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.133+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc4a80 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.133+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240c988c0 0x55c240c9a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.133+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/1816733106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240c94870 0x55c240c94c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.133+0000 7fa49ff6b740 1 -- 172.21.15.27:0/1816733106 >> 172.21.15.27:0/1816733106 conn(0x55c240c968f0 msgr2=0x55c240c981b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.133+0000 7fa49ff6b740 1 -- 172.21.15.27:0/1816733106 shutdown_connections 2024-03-06T07:30:28.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.133+0000 7fa49ff6b740 1 -- 172.21.15.27:0/1816733106 wait complete. 2024-03-06T07:30:28.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa49ff6b740 1 -- start start 2024-03-06T07:30:28.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa49ff6b740 1 Processor -- start 2024-03-06T07:30:28.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc4600 0x55c240cc6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc65b0 0x55c240cb9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 0x55c240cbcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c240c98d50 con 0x55c240cc65b0 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c240bbaea0 con 0x55c240cc4600 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c240c99030 con 0x55c240cb9d90 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa49ff6b740 10 client.0 _ll_register_callbacks cb 0x55c240bba870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.134+0000 7fa48ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc65b0 0x55c240cb9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa48ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc65b0 0x55c240cb9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56454/0 (socket says 172.21.15.27:56454) 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa48ffff640 1 -- 172.21.15.27:0/3414187419 learned_addr learned my addr 172.21.15.27:0/3414187419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa496ffd640 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc4600 0x55c240cc6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa4977fe640 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 0x55c240cbcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa48ffff640 1 -- 172.21.15.27:0/3414187419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 msgr2=0x55c240cbcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa48ffff640 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 0x55c240cbcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa48ffff640 1 -- 172.21.15.27:0/3414187419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc4600 msgr2=0x55c240cc6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa48ffff640 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc4600 0x55c240cc6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa48ffff640 1 -- 172.21.15.27:0/3414187419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c240bbb270 con 0x55c240cc65b0 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa48ffff640 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc65b0 0x55c240cb9850 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fa480020030 tx=0x7fa480058530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd036b0e22c8d0e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa475ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa475ffb640 1 -- 172.21.15.27:0/3414187419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480060400 con 0x55c240cc65b0 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa475ffb640 1 -- 172.21.15.27:0/3414187419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa48006f800 con 0x55c240cc65b0 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3414187419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa49003a040 con 0x55c240cc65b0 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa475ffb640 1 -- 172.21.15.27:0/3414187419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa480086050 con 0x55c240cc65b0 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa49ff6b740 10 client.8817 fetch_fsmap learned FSMap version 92 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.135+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3414187419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x55c2409132d0 con 0x55c240cc65b0 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa475ffb640 1 -- 172.21.15.27:0/3414187419 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49003a040 con 0x55c240cc65b0 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa475ffb640 1 -- 172.21.15.27:0/3414187419 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa48005ff70 con 0x55c240cc65b0 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa49ff6b740 10 client.8817 fetch_fsmap finished waiting for FSMap version 92 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa49ff6b740 10 client.8817 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa49ff6b740 20 client.8817 populate_metadata read hostname 'smithi027' 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55c240cf34f0 0x55c240cfe070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa49ff6b740 4 client.8817 mds_command: new command op to 8679 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:30:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3414187419 --> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c240c0bbf0 con 0x55c240cf34f0 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa496ffd640 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55c240cf34f0 0x55c240cfe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa475ffb640 10 client.8817 ms_handle_connect on v2:172.21.15.27:6833/4036525552 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.136+0000 7fa496ffd640 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55c240cf34f0 0x55c240cfe070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f69e8887e9b4de6b server_cookie=a4a5158811c194a in_seq=0 out_seq=0 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa475ffb640 1 -- 172.21.15.27:0/3414187419 <== mds.0 v2:172.21.15.27:6833/4036525552 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c240c0bbf0 con 0x55c240cf34f0 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa475ffb640 10 client.8817 handle_command_reply: tid=0 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 client.8817 shutdown 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3414187419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc65b0 msgr2=0x55c240cb9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc65b0 0x55c240cb9850 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fa480020030 tx=0x7fa480058530 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3414187419 shutdown_connections 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55c240cf34f0 0x55c240cfe070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 0x55c240cbcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc65b0 0x55c240cb9850 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa475ffb640 0 client.8817 ms_handle_reset on v2:172.21.15.27:6833/4036525552 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3414187419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc4600 0x55c240cc6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3414187419 >> 172.21.15.27:0/3414187419 conn(0x55c240c94870 msgr2=0x55c240cc25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3414187419 shutdown_connections 2024-03-06T07:30:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3414187419 wait complete. 2024-03-06T07:30:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.137+0000 7fa49ff6b740 20 client.8817 trim_cache size 0 max 16384 2024-03-06T07:30:28.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.154+0000 7fa49ff6b740 1 Processor -- start 2024-03-06T07:30:28.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.154+0000 7fa49ff6b740 1 -- start start 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cb9d90 0x55c240cb94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cc65b0 0x55c240cdf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49806d140 con 0x55c240cc4600 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980cf470 con 0x55c240cb9d90 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49806cb40 con 0x55c240cc65b0 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa48ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa48ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56470/0 (socket says 172.21.15.27:56470) 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa48ffff640 1 -- 172.21.15.27:0/2698341695 learned_addr learned my addr 172.21.15.27:0/2698341695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa496ffd640 1 --2- 172.21.15.27:0/2698341695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cb9d90 0x55c240cb94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa48ffff640 1 -- 172.21.15.27:0/2698341695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cc65b0 msgr2=0x55c240cdf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa48ffff640 1 --2- 172.21.15.27:0/2698341695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cc65b0 0x55c240cdf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa48ffff640 1 -- 172.21.15.27:0/2698341695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cb9d90 msgr2=0x55c240cb94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa48ffff640 1 --2- 172.21.15.27:0/2698341695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cb9d90 0x55c240cb94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa48ffff640 1 -- 172.21.15.27:0/2698341695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2409dd6b0 con 0x55c240cc4600 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa48ffff640 1 --2- 172.21.15.27:0/2698341695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc2820 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fa480054150 tx=0x7fa48005e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b3d791cb4d7833 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa477fff640 1 -- 172.21.15.27:0/2698341695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480004c60 con 0x55c240cc4600 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa477fff640 1 -- 172.21.15.27:0/2698341695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480060400 con 0x55c240cc4600 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.155+0000 7fa477fff640 1 -- 172.21.15.27:0/2698341695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480076bc0 con 0x55c240cc4600 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.156+0000 7fa49ff6b740 1 -- 172.21.15.27:0/2698341695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 msgr2=0x55c240cc2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.156+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/2698341695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc2820 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fa480054150 tx=0x7fa48005e000 comp rx=0 tx=0).stop 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.156+0000 7fa49ff6b740 1 -- 172.21.15.27:0/2698341695 shutdown_connections 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.156+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/2698341695 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cc65b0 0x55c240cdf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.156+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/2698341695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240cc2820 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.156+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/2698341695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cb9d90 0x55c240cb94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.156+0000 7fa49ff6b740 1 -- 172.21.15.27:0/2698341695 >> 172.21.15.27:0/2698341695 conn(0x55c240c94870 msgr2=0x55c240c8fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.156+0000 7fa49ff6b740 1 -- 172.21.15.27:0/2698341695 shutdown_connections 2024-03-06T07:30:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.156+0000 7fa49ff6b740 1 -- 172.21.15.27:0/2698341695 wait complete. 2024-03-06T07:30:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.157+0000 7fa49ff6b740 1 -- start start 2024-03-06T07:30:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.157+0000 7fa49ff6b740 1 Processor -- start 2024-03-06T07:30:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 0x55c240d4ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240c94c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa49ff6b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc65b0 0x55c240c95190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c240cbb050 con 0x55c240cc4600 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c240ce00d0 con 0x55c240cc65b0 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa49ff6b740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c240cb8a70 con 0x55c240cb9d90 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa49ff6b740 10 client.0 _ll_register_callbacks cb 0x7fa480057740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa48ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240c94c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa48ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240c94c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56486/0 (socket says 172.21.15.27:56486) 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa48ffff640 1 -- 172.21.15.27:0/3861013473 learned_addr learned my addr 172.21.15.27:0/3861013473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa4977fe640 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc65b0 0x55c240c95190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa496ffd640 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 0x55c240d4ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa48ffff640 1 -- 172.21.15.27:0/3861013473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 msgr2=0x55c240d4ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa48ffff640 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 0x55c240d4ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa48ffff640 1 -- 172.21.15.27:0/3861013473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc65b0 msgr2=0x55c240c95190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa48ffff640 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc65b0 0x55c240c95190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa48ffff640 1 -- 172.21.15.27:0/3861013473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c240cbd850 con 0x55c240cc4600 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa4977fe640 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc65b0 0x55c240c95190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa48ffff640 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240c94c50 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fa4800585a0 tx=0x7fa480087970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679662bfe605503 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa475ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa475ffb640 1 -- 172.21.15.27:0/3861013473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48007e060 con 0x55c240cc4600 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa475ffb640 1 -- 172.21.15.27:0/3861013473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa480057340 con 0x55c240cc4600 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.159+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3861013473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa480060400 con 0x55c240cc4600 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa475ffb640 1 -- 172.21.15.27:0/3861013473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa480081080 con 0x55c240cc4600 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa49ff6b740 10 client.8823 fetch_fsmap learned FSMap version 92 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3861013473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x55c240cbd850 con 0x55c240cc4600 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa475ffb640 1 -- 172.21.15.27:0/3861013473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480060400 con 0x55c240cc4600 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa475ffb640 1 -- 172.21.15.27:0/3861013473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa480053a40 con 0x55c240cc4600 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa49ff6b740 10 client.8823 fetch_fsmap finished waiting for FSMap version 92 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa49ff6b740 10 client.8823 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa49ff6b740 20 client.8823 populate_metadata read hostname 'smithi027' 2024-03-06T07:30:28.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55c240d529e0 0x55c240d52dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa49ff6b740 4 client.8823 mds_command: new command op to 8679 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3861013473 --> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c240c97050 con 0x55c240d529e0 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa496ffd640 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55c240d529e0 0x55c240d52dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa496ffd640 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55c240d529e0 0x55c240d52dc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc980121ec881c2e server_cookie=760c4195e51c78b1 in_seq=0 out_seq=0 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa475ffb640 10 client.8823 ms_handle_connect on v2:172.21.15.27:6833/4036525552 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa475ffb640 1 -- 172.21.15.27:0/3861013473 <== mds.0 v2:172.21.15.27:6833/4036525552 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55c240c97050 con 0x55c240d529e0 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa475ffb640 10 client.8823 handle_command_reply: tid=0 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.160+0000 7fa49ff6b740 1 client.8823 shutdown 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3861013473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 msgr2=0x55c240c94c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240c94c50 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fa4800585a0 tx=0x7fa480087970 comp rx=0 tx=0).stop 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3861013473 shutdown_connections 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55c240d529e0 0x55c240d52dc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c240cc65b0 0x55c240c95190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c240cc4600 0x55c240c94c50 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa475ffb640 0 client.8823 ms_handle_reset on v2:172.21.15.27:6833/4036525552 2024-03-06T07:30:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 --2- 172.21.15.27:0/3861013473 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55c240cb9d90 0x55c240d4ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3861013473 >> 172.21.15.27:0/3861013473 conn(0x55c240c94870 msgr2=0x55c240cb9670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3861013473 shutdown_connections 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 1 -- 172.21.15.27:0/3861013473 wait complete. 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49ff6b740 20 client.8823 trim_cache size 0 max 16384 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8679, 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:30:28.167 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:30:28.168 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 5.8008303200000002, 2024-03-06T07:30:28.168 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 92, 2024-03-06T07:30:28.168 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 132, 2024-03-06T07:30:28.168 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 132, 2024-03-06T07:30:28.168 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 17.812826038000001 2024-03-06T07:30:28.168 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 -- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa47806b4d0 msgr2=0x7fa47806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa47806b4d0 0x7fa47806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa488006fd0 tx=0x7fa488008040 comp rx=0 tx=0).stop 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 -- 172.21.15.27:0/4270433751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 msgr2=0x7fa4980a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 0x7fa4980a94a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fa48000c4a0 tx=0x7fa48000c960 comp rx=0 tx=0).stop 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 -- 172.21.15.27:0/4270433751 shutdown_connections 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa47806b4d0 0x7fa47806d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa498158520 0x7fa4980a99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4980aee10 0x7fa4980a94a0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 --2- 172.21.15.27:0/4270433751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa49806dd60 0x7fa49814a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 -- 172.21.15.27:0/4270433751 >> 172.21.15.27:0/4270433751 conn(0x7fa498066d00 msgr2=0x7fa49815a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 -- 172.21.15.27:0/4270433751 shutdown_connections 2024-03-06T07:30:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.161+0000 7fa49d2a4640 1 -- 172.21.15.27:0/4270433751 wait complete. 2024-03-06T07:30:28.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.280+0000 7fb8b2793640 1 Processor -- start 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.280+0000 7fb8b2793640 1 -- start start 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.280+0000 7fb8b2793640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac071540 0x7fb8ac0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8b2793640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8ac0b2b50 0x7fb8ac0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8b2793640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac0ce610 con 0x7fb8ac0d3400 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8b2793640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ac0d2f70 con 0x7fb8ac0b2b50 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8b2793640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac0d4be0 con 0x7fb8ac071540 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8b0d30640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8ac0b2b50 0x7fb8ac0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8ab7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb8ac0d3400 0x7fb8ac071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54708/0 (socket says 172.21.15.27:54708) 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8ab7fe640 1 -- 172.21.15.27:0/1285697181 learned_addr learned my addr 172.21.15.27:0/1285697181 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8abfff640 1 --2- 172.21.15.27:0/1285697181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac071540 0x7fb8ac0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8b0d30640 1 -- 172.21.15.27:0/1285697181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac071540 msgr2=0x7fb8ac0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8b0d30640 1 --2- 172.21.15.27:0/1285697181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac071540 0x7fb8ac0b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8b0d30640 1 -- 172.21.15.27:0/1285697181 >> v1:172.21.15.27:6789/0 conn(0x7fb8ac0d3400 legacy=0x7fb8ac071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8aaffd640 1 -- 172.21.15.27:0/1285697181 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055177768 0 0) 0x7fb8ac0ce610 con 0x7fb8ac0d3400 2024-03-06T07:30:28.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.281+0000 7fb8b0d30640 1 -- 172.21.15.27:0/1285697181 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0d2920 con 0x7fb8ac0b2b50 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.282+0000 7fb8b0d30640 1 --2- 172.21.15.27:0/1285697181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac0b2b50 0x7fb8ac0b6e20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4009d90 tx=0x7fb8a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e333f6d947e39e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.282+0000 7fb8aaffd640 1 -- 172.21.15.27:0/1285697181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4039070 con 0x7fb8ac0b2b50 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.282+0000 7fb8aaffd640 1 -- 172.21.15.27:0/1285697181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a402cc50 con 0x7fb8ac0b2b50 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.283+0000 7fb8aaffd640 1 -- 172.21.15.27:0/1285697181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4034440 con 0x7fb8ac0b2b50 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.283+0000 7fb8b2793640 1 -- 172.21.15.27:0/1285697181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac0b2b50 msgr2=0x7fb8ac0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.283+0000 7fb8b2793640 1 --2- 172.21.15.27:0/1285697181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac0b2b50 0x7fb8ac0b6e20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4009d90 tx=0x7fb8a402bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.283+0000 7fb8b2793640 1 -- 172.21.15.27:0/1285697181 shutdown_connections 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.283+0000 7fb8b2793640 1 --2- 172.21.15.27:0/1285697181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac0b2b50 0x7fb8ac0b6e20 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.283+0000 7fb8b2793640 1 --2- 172.21.15.27:0/1285697181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac071540 0x7fb8ac0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.283+0000 7fb8b2793640 1 -- 172.21.15.27:0/1285697181 >> 172.21.15.27:0/1285697181 conn(0x7fb8ac0d3ca0 msgr2=0x7fb8ac0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.283+0000 7fb8b2793640 1 -- 172.21.15.27:0/1285697181 shutdown_connections 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.283+0000 7fb8b2793640 1 -- 172.21.15.27:0/1285697181 wait complete. 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.284+0000 7fb8b2793640 1 Processor -- start 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.284+0000 7fb8b2793640 1 -- start start 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8b2793640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac071540 0x7fb8ac155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8b2793640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac0b2b50 0x7fb8ac0cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8b2793640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ac0d3400 0x7fb8ac1f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8b2793640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac15e370 con 0x7fb8ac071540 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8b2793640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac0ce8f0 con 0x7fb8ac0d3400 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8b2793640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac1f3d20 con 0x7fb8ac0b2b50 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac071540 0x7fb8ac155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8abfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac071540 0x7fb8ac155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56500/0 (socket says 172.21.15.27:56500) 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8abfff640 1 -- 172.21.15.27:0/346000472 learned_addr learned my addr 172.21.15.27:0/346000472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8ab7fe640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac0b2b50 0x7fb8ac0cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8b0d30640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ac0d3400 0x7fb8ac1f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8abfff640 1 -- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac0b2b50 msgr2=0x7fb8ac0cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.285+0000 7fb8abfff640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac0b2b50 0x7fb8ac0cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8abfff640 1 -- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ac0d3400 msgr2=0x7fb8ac1f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8abfff640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ac0d3400 0x7fb8ac1f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8abfff640 1 -- 172.21.15.27:0/346000472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac1f4800 con 0x7fb8ac071540 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8b0d30640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ac0d3400 0x7fb8ac1f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8ab7fe640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac0b2b50 0x7fb8ac0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8abfff640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac071540 0x7fb8ac155830 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00c7f0 tx=0x7fb89c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8a8ff9640 1 -- 172.21.15.27:0/346000472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c015070 con 0x7fb8ac071540 2024-03-06T07:30:28.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8a8ff9640 1 -- 172.21.15.27:0/346000472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c004240 con 0x7fb8ac071540 2024-03-06T07:30:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac1f3f60 con 0x7fb8ac071540 2024-03-06T07:30:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8a8ff9640 1 -- 172.21.15.27:0/346000472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c007620 con 0x7fb8ac071540 2024-03-06T07:30:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.286+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac1f4520 con 0x7fb8ac071540 2024-03-06T07:30:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.287+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac068700 con 0x7fb8ac071540 2024-03-06T07:30:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.287+0000 7fb8a8ff9640 1 -- 172.21.15.27:0/346000472 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb89c0079d0 con 0x7fb8ac071540 2024-03-06T07:30:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.288+0000 7fb8a8ff9640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb89806b4d0 0x7fb89806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.288+0000 7fb8a8ff9640 1 -- 172.21.15.27:0/346000472 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb89c08a680 con 0x7fb8ac071540 2024-03-06T07:30:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.288+0000 7fb8ab7fe640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb89806b4d0 0x7fb89806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.288+0000 7fb8ab7fe640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb89806b4d0 0x7fb89806d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0cde70 tx=0x7fb8a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.292+0000 7fb8a8ff9640 1 -- 172.21.15.27:0/346000472 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb89c059500 con 0x7fb8ac071540 2024-03-06T07:30:28.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.515+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb8ac15e820 con 0x7fb8ac071540 2024-03-06T07:30:28.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.516+0000 7fb8a8ff9640 1 -- 172.21.15.27:0/346000472 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v92) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb89c058e00 con 0x7fb8ac071540 2024-03-06T07:30:28.517 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb89806b4d0 msgr2=0x7fb89806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb89806b4d0 0x7fb89806d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0cde70 tx=0x7fb8a0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac071540 msgr2=0x7fb8ac155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac071540 0x7fb8ac155830 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00c7f0 tx=0x7fb89c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 shutdown_connections 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb89806b4d0 0x7fb89806d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ac0d3400 0x7fb8ac1f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8ac0b2b50 0x7fb8ac0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 --2- 172.21.15.27:0/346000472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8ac071540 0x7fb8ac155830 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 >> 172.21.15.27:0/346000472 conn(0x7fb8ac0d3ca0 msgr2=0x7fb8ac071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 shutdown_connections 2024-03-06T07:30:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:28.518+0000 7fb8b2793640 1 -- 172.21.15.27:0/346000472 wait complete. 2024-03-06T07:30:28.530 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:30:28.531 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:28.531 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:30:28.531 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:28.531 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:28.531 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:30:28.531 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:30:28.531 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:28.531 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:28.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.531 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:30:28.547 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42261sec preferred_lft 42261sec 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:30:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:28.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.550 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:28.617 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:30:28.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.618 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:30:28.673 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:30:28.673 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:28.674 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:30:28.674 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:30:28.674 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:28.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.674 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:30:28.674 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:30:28.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:30:28.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:30:28.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:30:28.674 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:30:28.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.780 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:30:28.796 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:28.796 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:28.796 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:28.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:28.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:30:28.867 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:30:28.867 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:28.867 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:28.922 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:28.922 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:28.922 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:28.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:30:29.048 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:29.103 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:30:29.103 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:30:29.103 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:30:29.104 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:29.105 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:29.106 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:29.106 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:29.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:29.106 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:30:29.162 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:30:29.162 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:30:29.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:30:29.236 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:30:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:30:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:30:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:30:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:30:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:29.293 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:29.294 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:29.295 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:29.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:29.295 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:30:29.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:30:29.307+0000 7f7b0add5480 -1 init, newargv = 0x55be4ba3f6c0 newargc=15 2024-03-06T07:30:29.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[78861]: starting ceph client 2024-03-06T07:30:29.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[78861]: starting fuse 2024-03-06T07:30:29.324 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:30:29.324 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:30:29.325 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:30:29.326 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:30:29.452 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.78861.asok 2024-03-06T07:30:29.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:29.463 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78861.asok 2024-03-06T07:30:29.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:29.463 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78861.asok status 2024-03-06T07:30:29.646 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "78861", 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8838, 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:30:29.647 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 8838 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 962192100 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 962192100 2024-03-06T07:30:29.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.8838 192.168.0.1:0/962192100", 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/962192100", 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 92, 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 132, 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:30:29.649 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:30:29.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:29.655 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:30:29.655 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:29.672 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:30:29.672 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:29.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:29.672 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:29.744 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:30:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.892+0000 7f0a3155e640 1 Processor -- start 2024-03-06T07:30:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.892+0000 7f0a3155e640 1 -- start start 2024-03-06T07:30:29.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.893+0000 7f0a3155e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a2c071540 0x7f0a2c071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.893+0000 7f0a3155e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0b2d30 0x7f0a2c0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.893+0000 7f0a3155e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2c071e60 con 0x7f0a2c0cecb0 2024-03-06T07:30:29.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.893+0000 7f0a3155e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a2c0d3230 con 0x7f0a2c071540 2024-03-06T07:30:29.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.893+0000 7f0a3155e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0d4960 con 0x7f0a2c0b2d30 2024-03-06T07:30:29.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.893+0000 7f0a2affd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a2c071540 0x7f0a2c071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.893+0000 7f0a2b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0a2c0cecb0 0x7f0a2c0b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54710/0 (socket says 172.21.15.27:54710) 2024-03-06T07:30:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.893+0000 7f0a2b7fe640 1 -- 172.21.15.27:0/3669885022 learned_addr learned my addr 172.21.15.27:0/3669885022 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.893+0000 7f0a2a7fc640 1 --2- 172.21.15.27:0/3669885022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0b2d30 0x7f0a2c0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.894+0000 7f0a2affd640 1 -- 172.21.15.27:0/3669885022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0b2d30 msgr2=0x7f0a2c0ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.894+0000 7f0a2affd640 1 --2- 172.21.15.27:0/3669885022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0b2d30 0x7f0a2c0ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.894+0000 7f0a2affd640 1 -- 172.21.15.27:0/3669885022 >> v1:172.21.15.27:6789/0 conn(0x7f0a2c0cecb0 legacy=0x7f0a2c0b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.894+0000 7f0a2affd640 1 -- 172.21.15.27:0/3669885022 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c0d36c0 con 0x7f0a2c071540 2024-03-06T07:30:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.894+0000 7f0a29ffb640 1 -- 172.21.15.27:0/3669885022 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462232266 0 0) 0x7f0a2c071e60 con 0x7f0a2c0cecb0 2024-03-06T07:30:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.894+0000 7f0a2a7fc640 1 --2- 172.21.15.27:0/3669885022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0b2d30 0x7f0a2c0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:29.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.894+0000 7f0a2affd640 1 --2- 172.21.15.27:0/3669885022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 0x7f0a2c071920 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f0a200097a0 tx=0x7f0a2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5a691b89ef2aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a29ffb640 1 -- 172.21.15.27:0/3669885022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20035020 con 0x7f0a2c071540 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a29ffb640 1 -- 172.21.15.27:0/3669885022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a20031830 con 0x7f0a2c071540 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a29ffb640 1 -- 172.21.15.27:0/3669885022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20031c00 con 0x7f0a2c071540 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a3155e640 1 -- 172.21.15.27:0/3669885022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 msgr2=0x7f0a2c071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a3155e640 1 --2- 172.21.15.27:0/3669885022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 0x7f0a2c071920 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f0a200097a0 tx=0x7f0a2002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a3155e640 1 -- 172.21.15.27:0/3669885022 shutdown_connections 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a3155e640 1 --2- 172.21.15.27:0/3669885022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0b2d30 0x7f0a2c0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a3155e640 1 --2- 172.21.15.27:0/3669885022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 0x7f0a2c071920 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a3155e640 1 -- 172.21.15.27:0/3669885022 >> 172.21.15.27:0/3669885022 conn(0x7f0a2c0d3d60 msgr2=0x7f0a2c0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a3155e640 1 -- 172.21.15.27:0/3669885022 shutdown_connections 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.895+0000 7f0a3155e640 1 -- 172.21.15.27:0/3669885022 wait complete. 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.896+0000 7f0a3155e640 1 Processor -- start 2024-03-06T07:30:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a3155e640 1 -- start start 2024-03-06T07:30:29.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a3155e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 0x7f0a2c1f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a3155e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a2c0b2d30 0x7f0a2c0c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a3155e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0cecb0 0x7f0a2c0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a3155e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0d2be0 con 0x7f0a2c071540 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a3155e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0d2900 con 0x7f0a2c0b2d30 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a3155e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0d2ec0 con 0x7f0a2c0cecb0 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a2affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 0x7f0a2c1f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a2affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 0x7f0a2c1f6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56524/0 (socket says 172.21.15.27:56524) 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.897+0000 7f0a2affd640 1 -- 172.21.15.27:0/381821194 learned_addr learned my addr 172.21.15.27:0/381821194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a2b7fe640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0cecb0 0x7f0a2c0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a2a7fc640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a2c0b2d30 0x7f0a2c0c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a2affd640 1 -- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0cecb0 msgr2=0x7f0a2c0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a2affd640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0cecb0 0x7f0a2c0ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a2affd640 1 -- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a2c0b2d30 msgr2=0x7f0a2c0c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a2affd640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a2c0b2d30 0x7f0a2c0c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a2affd640 1 -- 172.21.15.27:0/381821194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c0c80b0 con 0x7f0a2c071540 2024-03-06T07:30:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a2b7fe640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0cecb0 0x7f0a2c0ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a2affd640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 0x7f0a2c1f6b10 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f0a20000c00 tx=0x7f0a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a137fe640 1 -- 172.21.15.27:0/381821194 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20040070 con 0x7f0a2c071540 2024-03-06T07:30:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a137fe640 1 -- 172.21.15.27:0/381821194 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a20039a00 con 0x7f0a2c071540 2024-03-06T07:30:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2c0c8390 con 0x7f0a2c071540 2024-03-06T07:30:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a137fe640 1 -- 172.21.15.27:0/381821194 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20035040 con 0x7f0a2c071540 2024-03-06T07:30:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.898+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2c0c8950 con 0x7f0a2c071540 2024-03-06T07:30:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.899+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2c072070 con 0x7f0a2c071540 2024-03-06T07:30:29.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.900+0000 7f0a137fe640 1 -- 172.21.15.27:0/381821194 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a2004b060 con 0x7f0a2c071540 2024-03-06T07:30:29.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.901+0000 7f0a137fe640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a1806b4d0 0x7f0a1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.901+0000 7f0a137fe640 1 -- 172.21.15.27:0/381821194 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0a200aba20 con 0x7f0a2c071540 2024-03-06T07:30:29.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.901+0000 7f0a2a7fc640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a1806b4d0 0x7f0a1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.902+0000 7f0a2a7fc640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a1806b4d0 0x7f0a1806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0cabd0 tx=0x7f0a1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:29.906+0000 7f0a137fe640 1 -- 172.21.15.27:0/381821194 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a2007a860 con 0x7f0a2c071540 2024-03-06T07:30:30.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.125+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f0a2c04f410 con 0x7f0a2c071540 2024-03-06T07:30:30.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.399+0000 7f0a137fe640 1 -- 172.21.15.27:0/381821194 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v93) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0a2007a160 con 0x7f0a2c071540 2024-03-06T07:30:30.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a1806b4d0 msgr2=0x7f0a1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a1806b4d0 0x7f0a1806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0cabd0 tx=0x7f0a1c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 msgr2=0x7f0a2c1f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 0x7f0a2c1f6b10 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f0a20000c00 tx=0x7f0a200046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 shutdown_connections 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a1806b4d0 0x7f0a1806d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a2c0cecb0 0x7f0a2c0ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a2c0b2d30 0x7f0a2c0c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 --2- 172.21.15.27:0/381821194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a2c071540 0x7f0a2c1f6b10 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 >> 172.21.15.27:0/381821194 conn(0x7f0a2c0d3d60 msgr2=0x7f0a2c15bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 shutdown_connections 2024-03-06T07:30:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.401+0000 7f0a3155e640 1 -- 172.21.15.27:0/381821194 wait complete. 2024-03-06T07:30:30.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.526+0000 7fa6ce913640 1 Processor -- start 2024-03-06T07:30:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.526+0000 7fa6ce913640 1 -- start start 2024-03-06T07:30:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6ce913640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6ce913640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6c80b9a10 0x7fa6c80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6ce913640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c80cea30 con 0x7fa6c815df00 2024-03-06T07:30:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6ce913640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c80cf3c0 con 0x7fa6c80b9a10 2024-03-06T07:30:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6ce913640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c80cf0e0 con 0x7fa6c80b95a0 2024-03-06T07:30:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6cceb0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6c80b9a10 0x7fa6c80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6cceb0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6c80b9a10 0x7fa6c80b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56528/0 (socket says 172.21.15.27:56528) 2024-03-06T07:30:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6cceb0640 1 -- 172.21.15.27:0/3270947154 learned_addr learned my addr 172.21.15.27:0/3270947154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6c77fe640 1 --2- 172.21.15.27:0/3270947154 >> v2:172.21.15.69:3300/0 conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6cceb0640 1 -- 172.21.15.27:0/3270947154 >> v2:172.21.15.69:3300/0 conn(0x7fa6c80b95a0 msgr2=0x7fa6c80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6cceb0640 1 --2- 172.21.15.27:0/3270947154 >> v2:172.21.15.69:3300/0 conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6cceb0640 1 -- 172.21.15.27:0/3270947154 >> v1:172.21.15.69:6789/0 conn(0x7fa6c815df00 legacy=0x7fa6c80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6cceb0640 1 -- 172.21.15.27:0/3270947154 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c80ce470 con 0x7fa6c80b9a10 2024-03-06T07:30:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.527+0000 7fa6c77fe640 1 --2- 172.21.15.27:0/3270947154 >> v2:172.21.15.69:3300/0 conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.528+0000 7fa6cceb0640 1 --2- 172.21.15.27:0/3270947154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80b5c40 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80d1760 tx=0x7fa6c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b85500aec86f46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.528+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/3270947154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c002ca80 con 0x7fa6c80b9a10 2024-03-06T07:30:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.528+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/3270947154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c002cc20 con 0x7fa6c80b9a10 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.528+0000 7fa6c6ffd640 1 -- 172.21.15.27:0/3270947154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c00343f0 con 0x7fa6c80b9a10 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.529+0000 7fa6ce913640 1 -- 172.21.15.27:0/3270947154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 msgr2=0x7fa6c80b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.529+0000 7fa6ce913640 1 --2- 172.21.15.27:0/3270947154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80b5c40 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80d1760 tx=0x7fa6c002bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.529+0000 7fa6ce913640 1 -- 172.21.15.27:0/3270947154 shutdown_connections 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.529+0000 7fa6ce913640 1 --2- 172.21.15.27:0/3270947154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80b5c40 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.529+0000 7fa6ce913640 1 --2- 172.21.15.27:0/3270947154 >> v2:172.21.15.69:3300/0 conn(0x7fa6c80b95a0 0x7fa6c80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.529+0000 7fa6ce913640 1 -- 172.21.15.27:0/3270947154 >> 172.21.15.27:0/3270947154 conn(0x7fa6c8071960 msgr2=0x7fa6c8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.529+0000 7fa6ce913640 1 -- 172.21.15.27:0/3270947154 shutdown_connections 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.529+0000 7fa6ce913640 1 -- 172.21.15.27:0/3270947154 wait complete. 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.530+0000 7fa6ce913640 1 Processor -- start 2024-03-06T07:30:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.530+0000 7fa6ce913640 1 -- start start 2024-03-06T07:30:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6ce913640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b95a0 0x7fa6c81508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6ce913640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6ce913640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c815df00 0x7fa6c8149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6ce913640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80cedf0 con 0x7fa6c80b9a10 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6ce913640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80cec70 con 0x7fa6c80b95a0 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6ce913640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80d3220 con 0x7fa6c815df00 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80cc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56540/0 (socket says 172.21.15.27:56540) 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c77fe640 1 -- 172.21.15.27:0/1915254156 learned_addr learned my addr 172.21.15.27:0/1915254156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c7fff640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b95a0 0x7fa6c81508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6cceb0640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c815df00 0x7fa6c8149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c77fe640 1 -- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c815df00 msgr2=0x7fa6c8149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c77fe640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c815df00 0x7fa6c8149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c77fe640 1 -- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b95a0 msgr2=0x7fa6c81508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c77fe640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b95a0 0x7fa6c81508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c77fe640 1 -- 172.21.15.27:0/1915254156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c80c9f70 con 0x7fa6c80b9a10 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6cceb0640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c815df00 0x7fa6c8149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.531+0000 7fa6c7fff640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b95a0 0x7fa6c81508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.532+0000 7fa6c77fe640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80cc520 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800c820 tx=0x7fa6b800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.532+0000 7fa6c4ff9640 1 -- 172.21.15.27:0/1915254156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8015070 con 0x7fa6c80b9a10 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.532+0000 7fa6c4ff9640 1 -- 172.21.15.27:0/1915254156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b8007a30 con 0x7fa6c80b9a10 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.532+0000 7fa6c4ff9640 1 -- 172.21.15.27:0/1915254156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8007e00 con 0x7fa6c80b9a10 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.532+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c80ca250 con 0x7fa6c80b9a10 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.532+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c80ca810 con 0x7fa6c80b9a10 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.533+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c80cab50 con 0x7fa6c80b9a10 2024-03-06T07:30:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.533+0000 7fa6c4ff9640 1 -- 172.21.15.27:0/1915254156 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6b8018070 con 0x7fa6c80b9a10 2024-03-06T07:30:30.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.534+0000 7fa6c4ff9640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b406b4d0 0x7fa6b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.534+0000 7fa6c4ff9640 1 -- 172.21.15.27:0/1915254156 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa6b808a180 con 0x7fa6c80b9a10 2024-03-06T07:30:30.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.534+0000 7fa6c7fff640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b406b4d0 0x7fa6b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.535+0000 7fa6c7fff640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b406b4d0 0x7fa6b406d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc006fd0 tx=0x7fa6bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.538+0000 7fa6c4ff9640 1 -- 172.21.15.27:0/1915254156 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6b8058fc0 con 0x7fa6c80b9a10 2024-03-06T07:30:30.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.832+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6c8067ff0 con 0x7fa6c80b9a10 2024-03-06T07:30:30.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.832+0000 7fa6c4ff9640 1 -- 172.21.15.27:0/1915254156 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4992 (secure 0 0 0) 0x7fa6b80588c0 con 0x7fa6c80b9a10 2024-03-06T07:30:30.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:30.834 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/643534493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":643534493},{"type":"v1","addr":"172.21.15.69:6837","nonce":643534493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":7783,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3158495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3158495906},{"type":"v1","addr":"172.21.15.69:6835","nonce":3158495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:30.401665+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:creating","state_seq":1,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:30.834 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 94 2024-03-06T07:30:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.834+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b406b4d0 msgr2=0x7fa6b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.834+0000 7fa6ce913640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b406b4d0 0x7fa6b406d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc006fd0 tx=0x7fa6bc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.834+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 msgr2=0x7fa6c80cc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.834+0000 7fa6ce913640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80cc520 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800c820 tx=0x7fa6b800cc20 comp rx=0 tx=0).stop 2024-03-06T07:30:30.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.834+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 shutdown_connections 2024-03-06T07:30:30.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.835+0000 7fa6ce913640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6b406b4d0 0x7fa6b406d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.835+0000 7fa6ce913640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6c815df00 0x7fa6c8149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.835+0000 7fa6ce913640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6c80b9a10 0x7fa6c80cc520 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.835+0000 7fa6ce913640 1 --2- 172.21.15.27:0/1915254156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6c80b95a0 0x7fa6c81508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.835+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 >> 172.21.15.27:0/1915254156 conn(0x7fa6c8071960 msgr2=0x7fa6c80d3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.835+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 shutdown_connections 2024-03-06T07:30:30.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:30.835+0000 7fa6ce913640 1 -- 172.21.15.27:0/1915254156 wait complete. 2024-03-06T07:30:30.847 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:30:35.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.953+0000 7fcbc8884640 1 Processor -- start 2024-03-06T07:30:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.953+0000 7fcbc8884640 1 -- start start 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc8884640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b95a0 0x7fcbc40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc8884640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcbc40b9a10 0x7fcbc40b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc8884640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc40cea30 con 0x7fcbc415df00 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc8884640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc40cf3c0 con 0x7fcbc40b9a10 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc8884640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40cf0e0 con 0x7fcbc40b95a0 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc259c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcbc415df00 0x7fcbc40d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45444/0 (socket says 172.21.15.27:45444) 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc259c640 1 -- 172.21.15.27:0/759935420 learned_addr learned my addr 172.21.15.27:0/759935420 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc2d9d640 1 --2- 172.21.15.27:0/759935420 >> v2:172.21.15.69:3300/0 conn(0x7fcbc40b9a10 0x7fcbc40b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc1d9b640 1 --2- 172.21.15.27:0/759935420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b95a0 0x7fcbc40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc159a640 1 -- 172.21.15.27:0/759935420 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069226727 0 0) 0x7fcbc40cea30 con 0x7fcbc415df00 2024-03-06T07:30:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc159a640 1 -- 172.21.15.27:0/759935420 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbac003520 con 0x7fcbc415df00 2024-03-06T07:30:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc2d9d640 1 -- 172.21.15.27:0/759935420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b95a0 msgr2=0x7fcbc40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc2d9d640 1 --2- 172.21.15.27:0/759935420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b95a0 0x7fcbc40b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc2d9d640 1 -- 172.21.15.27:0/759935420 >> v1:172.21.15.27:6789/0 conn(0x7fcbc415df00 legacy=0x7fcbc40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.955+0000 7fcbc2d9d640 1 -- 172.21.15.27:0/759935420 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc40ce470 con 0x7fcbc40b9a10 2024-03-06T07:30:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.957+0000 7fcbc2d9d640 1 --2- 172.21.15.27:0/759935420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b9a10 0x7fcbc40b5c40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009d90 tx=0x7fcbbc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1006bed615c7de7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.957+0000 7fcbc159a640 1 -- 172.21.15.27:0/759935420 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc039070 con 0x7fcbc40b9a10 2024-03-06T07:30:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.957+0000 7fcbc159a640 1 -- 172.21.15.27:0/759935420 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbbc02cc60 con 0x7fcbc40b9a10 2024-03-06T07:30:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.957+0000 7fcbc159a640 1 -- 172.21.15.27:0/759935420 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc034660 con 0x7fcbc40b9a10 2024-03-06T07:30:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.958+0000 7fcbc8884640 1 -- 172.21.15.27:0/759935420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b9a10 msgr2=0x7fcbc40b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.958+0000 7fcbc8884640 1 --2- 172.21.15.27:0/759935420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b9a10 0x7fcbc40b5c40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009d90 tx=0x7fcbbc02beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.958+0000 7fcbc8884640 1 -- 172.21.15.27:0/759935420 shutdown_connections 2024-03-06T07:30:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.958+0000 7fcbc8884640 1 --2- 172.21.15.27:0/759935420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b9a10 0x7fcbc40b5c40 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.958+0000 7fcbc8884640 1 --2- 172.21.15.27:0/759935420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b95a0 0x7fcbc40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.958+0000 7fcbc8884640 1 -- 172.21.15.27:0/759935420 >> 172.21.15.27:0/759935420 conn(0x7fcbc4071960 msgr2=0x7fcbc4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.958+0000 7fcbc8884640 1 -- 172.21.15.27:0/759935420 shutdown_connections 2024-03-06T07:30:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.958+0000 7fcbc8884640 1 -- 172.21.15.27:0/759935420 wait complete. 2024-03-06T07:30:35.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.959+0000 7fcbc8884640 1 Processor -- start 2024-03-06T07:30:35.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.959+0000 7fcbc8884640 1 -- start start 2024-03-06T07:30:35.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.960+0000 7fcbc8884640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b95a0 0x7fcbc41f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.960+0000 7fcbc8884640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b9a10 0x7fcbc40c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.960+0000 7fcbc8884640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbc415df00 0x7fcbc415ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.960+0000 7fcbc8884640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40cedf0 con 0x7fcbc415df00 2024-03-06T07:30:35.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.960+0000 7fcbc8884640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40cec70 con 0x7fcbc40b95a0 2024-03-06T07:30:35.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.960+0000 7fcbc8884640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40d3220 con 0x7fcbc40b9a10 2024-03-06T07:30:35.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbc415df00 0x7fcbc415ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbc415df00 0x7fcbc415ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41816/0 (socket says 172.21.15.27:41816) 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc2d9d640 1 -- 172.21.15.27:0/2141866166 learned_addr learned my addr 172.21.15.27:0/2141866166 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc259c640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b95a0 0x7fcbc41f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc1d9b640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b9a10 0x7fcbc40c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc2d9d640 1 -- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b9a10 msgr2=0x7fcbc40c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc2d9d640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b9a10 0x7fcbc40c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc2d9d640 1 -- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b95a0 msgr2=0x7fcbc41f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc2d9d640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b95a0 0x7fcbc41f30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc2d9d640 1 -- 172.21.15.27:0/2141866166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc40cc140 con 0x7fcbc415df00 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc259c640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b95a0 0x7fcbc41f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc2d9d640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbc415df00 0x7fcbc415ef00 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009d90 tx=0x7fcbbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbb2ffd640 1 -- 172.21.15.27:0/2141866166 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc039050 con 0x7fcbc415df00 2024-03-06T07:30:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbb2ffd640 1 -- 172.21.15.27:0/2141866166 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbbc045070 con 0x7fcbc415df00 2024-03-06T07:30:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc40cc420 con 0x7fcbc415df00 2024-03-06T07:30:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.961+0000 7fcbb2ffd640 1 -- 172.21.15.27:0/2141866166 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc009450 con 0x7fcbc415df00 2024-03-06T07:30:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.962+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc40cc9e0 con 0x7fcbc415df00 2024-03-06T07:30:35.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.963+0000 7fcbb2ffd640 1 -- 172.21.15.27:0/2141866166 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbbc03e850 con 0x7fcbc415df00 2024-03-06T07:30:35.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.963+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc4067ff0 con 0x7fcbc415df00 2024-03-06T07:30:35.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.963+0000 7fcbb2ffd640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcba406b4d0 0x7fcba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.963+0000 7fcbb2ffd640 1 -- 172.21.15.27:0/2141866166 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcbbc0acf00 con 0x7fcbc415df00 2024-03-06T07:30:35.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.964+0000 7fcbc259c640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcba406b4d0 0x7fcba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.964+0000 7fcbc259c640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcba406b4d0 0x7fcba406d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4005880 tx=0x7fcbb4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:35.972+0000 7fcbb2ffd640 1 -- 172.21.15.27:0/2141866166 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbbc07b780 con 0x7fcbc415df00 2024-03-06T07:30:36.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.349+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbc40d2730 con 0x7fcbc415df00 2024-03-06T07:30:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.350+0000 7fcbb2ffd640 1 -- 172.21.15.27:0/2141866166 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4995 (secure 0 0 0) 0x7fcbbc07f820 con 0x7fcbc415df00 2024-03-06T07:30:36.351 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:36.351 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":95,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/643534493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":643534493},{"type":"v1","addr":"172.21.15.69:6837","nonce":643534493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":7783,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3158495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3158495906},{"type":"v1","addr":"172.21.15.69:6835","nonce":3158495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:31.403642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8670,8679]},"id":9}]} 2024-03-06T07:30:36.351 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 95 2024-03-06T07:30:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcba406b4d0 msgr2=0x7fcba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcba406b4d0 0x7fcba406d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4005880 tx=0x7fcbb4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbc415df00 msgr2=0x7fcbc415ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbc415df00 0x7fcbc415ef00 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009d90 tx=0x7fcbbc004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 shutdown_connections 2024-03-06T07:30:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcba406b4d0 0x7fcba406d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbc415df00 0x7fcbc415ef00 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcbc40b9a10 0x7fcbc40c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 --2- 172.21.15.27:0/2141866166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbc40b95a0 0x7fcbc41f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.351+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 >> 172.21.15.27:0/2141866166 conn(0x7fcbc4071960 msgr2=0x7fcbc4071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.352+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 shutdown_connections 2024-03-06T07:30:36.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.352+0000 7fcbc8884640 1 -- 172.21.15.27:0/2141866166 wait complete. 2024-03-06T07:30:36.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:36.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.507+0000 7f2e0135e640 1 Processor -- start 2024-03-06T07:30:36.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.507+0000 7f2e0135e640 1 -- start start 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.508+0000 7f2e0135e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc0b95a0 0x7f2dfc0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.508+0000 7f2e0135e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dfc0cea30 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.508+0000 7f2e0135e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dfc0cf3c0 con 0x7f2dfc15df00 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.508+0000 7f2e0135e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0cf0e0 con 0x7f2dfc0b95a0 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.508+0000 7f2dfb7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2dfc0b9a10 0x7f2dfc0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45456/0 (socket says 172.21.15.27:45456) 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.508+0000 7f2dfb7fe640 1 -- 172.21.15.27:0/2529503046 learned_addr learned my addr 172.21.15.27:0/2529503046 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.508+0000 7f2dfa7fc640 1 --2- 172.21.15.27:0/2529503046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc0b95a0 0x7f2dfc0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.508+0000 7f2df9ffb640 1 -- 172.21.15.27:0/2529503046 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639359097 0 0) 0x7f2dfc0cea30 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.508+0000 7f2df9ffb640 1 -- 172.21.15.27:0/2529503046 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de4003520 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.509+0000 7f2df9ffb640 1 -- 172.21.15.27:0/2529503046 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100314770 0 0) 0x7f2dfc0cf3c0 con 0x7f2dfc15df00 2024-03-06T07:30:36.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.509+0000 7f2df9ffb640 1 -- 172.21.15.27:0/2529503046 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dfc0cea30 con 0x7f2dfc15df00 2024-03-06T07:30:36.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.509+0000 7f2df9ffb640 1 -- 172.21.15.27:0/2529503046 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1303474450 0 0) 0x7f2de4003520 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.509+0000 7f2dfa7fc640 1 -- 172.21.15.27:0/2529503046 >> v1:172.21.15.69:6789/0 conn(0x7f2dfc15df00 legacy=0x7f2dfc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.509+0000 7f2dfa7fc640 1 -- 172.21.15.27:0/2529503046 >> v1:172.21.15.27:6789/0 conn(0x7f2dfc0b9a10 legacy=0x7f2dfc0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.509+0000 7f2dfa7fc640 1 -- 172.21.15.27:0/2529503046 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc0ce470 con 0x7f2dfc0b95a0 2024-03-06T07:30:36.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.509+0000 7f2dfa7fc640 1 --2- 172.21.15.27:0/2529503046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc0b95a0 0x7f2dfc0b5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2df00097d0 tx=0x7f2df002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=543ce4208257efe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2df9ffb640 1 -- 172.21.15.27:0/2529503046 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0037070 con 0x7f2dfc0b95a0 2024-03-06T07:30:36.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2df9ffb640 1 -- 172.21.15.27:0/2529503046 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df0030510 con 0x7f2dfc0b95a0 2024-03-06T07:30:36.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2df9ffb640 1 -- 172.21.15.27:0/2529503046 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df40019f0 con 0x7f2dfc0b95a0 2024-03-06T07:30:36.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2e0135e640 1 -- 172.21.15.27:0/2529503046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc0b95a0 msgr2=0x7f2dfc0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2e0135e640 1 --2- 172.21.15.27:0/2529503046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc0b95a0 0x7f2dfc0b5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2df00097d0 tx=0x7f2df002b800 comp rx=0 tx=0).stop 2024-03-06T07:30:36.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2e0135e640 1 -- 172.21.15.27:0/2529503046 shutdown_connections 2024-03-06T07:30:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2e0135e640 1 --2- 172.21.15.27:0/2529503046 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc0b95a0 0x7f2dfc0b5700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2e0135e640 1 -- 172.21.15.27:0/2529503046 >> 172.21.15.27:0/2529503046 conn(0x7f2dfc071960 msgr2=0x7f2dfc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2e0135e640 1 -- 172.21.15.27:0/2529503046 shutdown_connections 2024-03-06T07:30:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.510+0000 7f2e0135e640 1 -- 172.21.15.27:0/2529503046 wait complete. 2024-03-06T07:30:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.512+0000 7f2e0135e640 1 Processor -- start 2024-03-06T07:30:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.512+0000 7f2e0135e640 1 -- start start 2024-03-06T07:30:36.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.513+0000 7f2e0135e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dfc0b95a0 0x7f2dfc154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.513+0000 7f2e0135e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dfc0b9a10 0x7f2dfc15f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.513+0000 7f2e0135e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc15df00 0x7f2dfc0b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.513+0000 7f2e0135e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0cedf0 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.513+0000 7f2e0135e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0cec70 con 0x7f2dfc0b95a0 2024-03-06T07:30:36.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.513+0000 7f2e0135e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0b5c40 con 0x7f2dfc15df00 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfa7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dfc0b9a10 0x7f2dfc15f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfa7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dfc0b9a10 0x7f2dfc15f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41820/0 (socket says 172.21.15.27:41820) 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfa7fc640 1 -- 172.21.15.27:0/3242903754 learned_addr learned my addr 172.21.15.27:0/3242903754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfaffd640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dfc0b95a0 0x7f2dfc154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfb7fe640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc15df00 0x7f2dfc0b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfa7fc640 1 -- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc15df00 msgr2=0x7f2dfc0b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfa7fc640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc15df00 0x7f2dfc0b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfa7fc640 1 -- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dfc0b95a0 msgr2=0x7f2dfc154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfa7fc640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dfc0b95a0 0x7f2dfc154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfa7fc640 1 -- 172.21.15.27:0/3242903754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc154bb0 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2dfa7fc640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dfc0b9a10 0x7f2dfc15f530 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f2df0000c00 tx=0x7f2df00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2de37fe640 1 -- 172.21.15.27:0/3242903754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0037040 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2de37fe640 1 -- 172.21.15.27:0/3242903754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df0004250 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc15fea0 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2de37fe640 1 -- 172.21.15.27:0/3242903754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df003c670 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.514+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc160460 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.516+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc067ff0 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.516+0000 7f2de37fe640 1 -- 172.21.15.27:0/3242903754 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2df003a070 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.517+0000 7f2de37fe640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2de806b4d0 0x7f2de806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.517+0000 7f2de37fe640 1 -- 172.21.15.27:0/3242903754 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2df00abb10 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.517+0000 7f2dfaffd640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2de806b4d0 0x7f2de806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.518+0000 7f2dfaffd640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2de806b4d0 0x7f2de806d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0011e0 tx=0x7f2dec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.522+0000 7f2de37fe640 1 -- 172.21.15.27:0/3242903754 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2df0041020 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.745+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dfc071e70 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.745+0000 7f2de37fe640 1 -- 172.21.15.27:0/3242903754 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4995 (secure 0 0 0) 0x7f2df007a300 con 0x7f2dfc0b9a10 2024-03-06T07:30:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:36.747 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":95,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/643534493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":643534493},{"type":"v1","addr":"172.21.15.69:6837","nonce":643534493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":7783,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3158495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3158495906},{"type":"v1","addr":"172.21.15.69:6835","nonce":3158495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:31.403642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8670,8679]},"id":9}]} 2024-03-06T07:30:36.747 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 95 2024-03-06T07:30:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2de806b4d0 msgr2=0x7f2de806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2de806b4d0 0x7f2de806d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0011e0 tx=0x7f2dec009040 comp rx=0 tx=0).stop 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dfc0b9a10 msgr2=0x7f2dfc15f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dfc0b9a10 0x7f2dfc15f530 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f2df0000c00 tx=0x7f2df00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 shutdown_connections 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2de806b4d0 0x7f2de806d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dfc15df00 0x7f2dfc0b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dfc0b9a10 0x7f2dfc15f530 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 --2- 172.21.15.27:0/3242903754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dfc0b95a0 0x7f2dfc154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 >> 172.21.15.27:0/3242903754 conn(0x7f2dfc071960 msgr2=0x7f2dfc0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 shutdown_connections 2024-03-06T07:30:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.748+0000 7f2e0135e640 1 -- 172.21.15.27:0/3242903754 wait complete. 2024-03-06T07:30:36.761 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and c 2024-03-06T07:30:36.761 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:36.761 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:36.772 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:36.772 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:30:36.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.879+0000 7fa29a3b5640 1 Processor -- start 2024-03-06T07:30:36.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.879+0000 7fa29a3b5640 1 -- start start 2024-03-06T07:30:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.880+0000 7fa29a3b5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa29415df00 0x7fa2940d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.880+0000 7fa29a3b5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa2940b95a0 0x7fa2940b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.880+0000 7fa29a3b5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940cea30 con 0x7fa2940b9a30 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.880+0000 7fa29a3b5640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2940d2af0 con 0x7fa29415df00 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.880+0000 7fa29a3b5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2940cf3c0 con 0x7fa2940b95a0 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.880+0000 7fa293fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa29415df00 0x7fa2940d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.880+0000 7fa298952640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa2940b9a30 0x7fa2940d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45468/0 (socket says 172.21.15.27:45468) 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.880+0000 7fa298952640 1 -- 172.21.15.27:0/2874977337 learned_addr learned my addr 172.21.15.27:0/2874977337 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.881+0000 7fa2937fe640 1 --2- 172.21.15.27:0/2874977337 >> v2:172.21.15.69:3300/0 conn(0x7fa2940b95a0 0x7fa2940b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.881+0000 7fa292ffd640 1 -- 172.21.15.27:0/2874977337 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984271959 0 0) 0x7fa2940cea30 con 0x7fa2940b9a30 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.881+0000 7fa292ffd640 1 -- 172.21.15.27:0/2874977337 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c003520 con 0x7fa2940b9a30 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.881+0000 7fa293fff640 1 -- 172.21.15.27:0/2874977337 >> v2:172.21.15.69:3300/0 conn(0x7fa2940b95a0 msgr2=0x7fa2940b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.881+0000 7fa293fff640 1 --2- 172.21.15.27:0/2874977337 >> v2:172.21.15.69:3300/0 conn(0x7fa2940b95a0 0x7fa2940b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.881+0000 7fa293fff640 1 -- 172.21.15.27:0/2874977337 >> v1:172.21.15.27:6789/0 conn(0x7fa2940b9a30 legacy=0x7fa2940d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.881+0000 7fa293fff640 1 -- 172.21.15.27:0/2874977337 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940cf0e0 con 0x7fa29415df00 2024-03-06T07:30:36.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.882+0000 7fa293fff640 1 --2- 172.21.15.27:0/2874977337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa29415df00 0x7fa2940d25b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fa2940d1760 tx=0x7fa28402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba808bb1b0cf452 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.882+0000 7fa292ffd640 1 -- 172.21.15.27:0/2874977337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa284035020 con 0x7fa29415df00 2024-03-06T07:30:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.882+0000 7fa292ffd640 1 -- 172.21.15.27:0/2874977337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa284031830 con 0x7fa29415df00 2024-03-06T07:30:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.882+0000 7fa292ffd640 1 -- 172.21.15.27:0/2874977337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa284031c00 con 0x7fa29415df00 2024-03-06T07:30:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.883+0000 7fa29a3b5640 1 -- 172.21.15.27:0/2874977337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa29415df00 msgr2=0x7fa2940d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.883+0000 7fa29a3b5640 1 --2- 172.21.15.27:0/2874977337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa29415df00 0x7fa2940d25b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fa2940d1760 tx=0x7fa28402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.883+0000 7fa29a3b5640 1 -- 172.21.15.27:0/2874977337 shutdown_connections 2024-03-06T07:30:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.883+0000 7fa29a3b5640 1 --2- 172.21.15.27:0/2874977337 >> v2:172.21.15.69:3300/0 conn(0x7fa2940b95a0 0x7fa2940b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.883+0000 7fa29a3b5640 1 --2- 172.21.15.27:0/2874977337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa29415df00 0x7fa2940d25b0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.883+0000 7fa29a3b5640 1 -- 172.21.15.27:0/2874977337 >> 172.21.15.27:0/2874977337 conn(0x7fa294071960 msgr2=0x7fa294071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.883+0000 7fa29a3b5640 1 -- 172.21.15.27:0/2874977337 shutdown_connections 2024-03-06T07:30:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.883+0000 7fa29a3b5640 1 -- 172.21.15.27:0/2874977337 wait complete. 2024-03-06T07:30:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.885+0000 7fa29a3b5640 1 Processor -- start 2024-03-06T07:30:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.885+0000 7fa29a3b5640 1 -- start start 2024-03-06T07:30:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa29a3b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2940b95a0 0x7fa294157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa29a3b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa29415df00 0x7fa2940b5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa29a3b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2941584a0 0x7fa2940b5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa29a3b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940cec70 con 0x7fa2941584a0 2024-03-06T07:30:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa29a3b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940ce750 con 0x7fa29415df00 2024-03-06T07:30:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa29a3b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940cedf0 con 0x7fa2940b95a0 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa298952640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2941584a0 0x7fa2940b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa298952640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2941584a0 0x7fa2940b5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41838/0 (socket says 172.21.15.27:41838) 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa298952640 1 -- 172.21.15.27:0/1633134810 learned_addr learned my addr 172.21.15.27:0/1633134810 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa2937fe640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa29415df00 0x7fa2940b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa293fff640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2940b95a0 0x7fa294157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa298952640 1 -- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2940b95a0 msgr2=0x7fa294157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa298952640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2940b95a0 0x7fa294157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa298952640 1 -- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa29415df00 msgr2=0x7fa2940b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa298952640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa29415df00 0x7fa2940b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa298952640 1 -- 172.21.15.27:0/1633134810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29415f930 con 0x7fa2941584a0 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa2937fe640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa29415df00 0x7fa2940b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa298952640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2941584a0 0x7fa2940b5990 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00cb40 tx=0x7fa28c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.887+0000 7fa290ff9640 1 -- 172.21.15.27:0/1633134810 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c017070 con 0x7fa2941584a0 2024-03-06T07:30:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.888+0000 7fa290ff9640 1 -- 172.21.15.27:0/1633134810 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28c00aca0 con 0x7fa2941584a0 2024-03-06T07:30:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.888+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2940cb920 con 0x7fa2941584a0 2024-03-06T07:30:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.888+0000 7fa290ff9640 1 -- 172.21.15.27:0/1633134810 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c012620 con 0x7fa2941584a0 2024-03-06T07:30:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.888+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2940cbee0 con 0x7fa2941584a0 2024-03-06T07:30:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.889+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2940cc0e0 con 0x7fa2941584a0 2024-03-06T07:30:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.890+0000 7fa290ff9640 1 -- 172.21.15.27:0/1633134810 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa28c01a070 con 0x7fa2941584a0 2024-03-06T07:30:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.890+0000 7fa290ff9640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa28006b500 0x7fa28006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.890+0000 7fa290ff9640 1 -- 172.21.15.27:0/1633134810 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa28c08a010 con 0x7fa2941584a0 2024-03-06T07:30:36.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.890+0000 7fa293fff640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa28006b500 0x7fa28006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.891+0000 7fa293fff640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa28006b500 0x7fa28006d9c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa284036000 tx=0x7fa28403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:36.894+0000 7fa290ff9640 1 -- 172.21.15.27:0/1633134810 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2940cc0e0 con 0x7fa2941584a0 2024-03-06T07:30:37.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.146+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fa2940b3390 con 0x7fa2941584a0 2024-03-06T07:30:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.440+0000 7fa290ff9640 1 -- 172.21.15.27:0/1633134810 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 7783 v96) v1 ==== 95+0+0 (secure 0 0 0) 0x7fa28c058ee0 con 0x7fa2941584a0 2024-03-06T07:30:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 7783 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa28006b500 msgr2=0x7fa28006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa28006b500 0x7fa28006d9c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa284036000 tx=0x7fa28403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2941584a0 msgr2=0x7fa2940b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2941584a0 0x7fa2940b5990 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00cb40 tx=0x7fa28c00a060 comp rx=0 tx=0).stop 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 shutdown_connections 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa28006b500 0x7fa28006d9c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2941584a0 0x7fa2940b5990 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa29415df00 0x7fa2940b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 --2- 172.21.15.27:0/1633134810 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2940b95a0 0x7fa294157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.442+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 >> 172.21.15.27:0/1633134810 conn(0x7fa294071960 msgr2=0x7fa2941618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.443+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 shutdown_connections 2024-03-06T07:30:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.443+0000 7fa29a3b5640 1 -- 172.21.15.27:0/1633134810 wait complete. 2024-03-06T07:30:37.455 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:37.455 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:37.467 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:37.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:30:37.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.574+0000 7f0c5b702640 1 Processor -- start 2024-03-06T07:30:37.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.574+0000 7f0c5b702640 1 -- start start 2024-03-06T07:30:37.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.575+0000 7f0c5b702640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0c5415df00 0x7f0c540d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.575+0000 7f0c5b702640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c540cea30 con 0x7f0c540b9a30 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.575+0000 7f0c5b702640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c540d2af0 con 0x7f0c540b95a0 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.575+0000 7f0c5b702640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c540cf3c0 con 0x7f0c5415df00 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.575+0000 7f0c59c9f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0c540b9a30 0x7f0c540d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45480/0 (socket says 172.21.15.27:45480) 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.575+0000 7f0c59c9f640 1 -- 172.21.15.27:0/2792600764 learned_addr learned my addr 172.21.15.27:0/2792600764 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262399545 0 0) 0x7f0c540cea30 con 0x7f0c540b9a30 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c40003520 con 0x7f0c540b9a30 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c5949e640 1 --2- 172.21.15.27:0/2792600764 >> v2:172.21.15.69:3300/0 conn(0x7f0c5415df00 0x7f0c540d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1087313165 0 0) 0x7f0c40003520 con 0x7f0c540b9a30 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 >> v2:172.21.15.69:3300/0 conn(0x7f0c5415df00 msgr2=0x7f0c540d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 --2- 172.21.15.27:0/2792600764 >> v2:172.21.15.69:3300/0 conn(0x7f0c5415df00 0x7f0c540d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 >> v1:172.21.15.69:6789/0 conn(0x7f0c540b95a0 legacy=0x7f0c540b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c540cf0e0 con 0x7f0c540b9a30 2024-03-06T07:30:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f0c4c0019f0 con 0x7f0c540b9a30 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 >> v1:172.21.15.27:6789/0 conn(0x7f0c540b9a30 legacy=0x7f0c540d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c400039b0 0x7f0c40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c40004800 0x7f0c40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400051c0 0x7f0c4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540cf3c0 con 0x7f0c400039b0 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4000c060 con 0x7f0c400051c0 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4000c260 con 0x7f0c40004800 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c5949e640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c400039b0 0x7f0c40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c59c9f640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400051c0 0x7f0c4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c58c9d640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c40004800 0x7f0c40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c5949e640 1 -- 172.21.15.27:0/2792600764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c40004800 msgr2=0x7f0c40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c5949e640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c40004800 0x7f0c40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c5949e640 1 -- 172.21.15.27:0/2792600764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400051c0 msgr2=0x7f0c4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c5949e640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400051c0 0x7f0c4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c5949e640 1 -- 172.21.15.27:0/2792600764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c40003520 con 0x7f0c400039b0 2024-03-06T07:30:37.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c59c9f640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400051c0 0x7f0c4000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:37.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.576+0000 7f0c59c9f640 1 -- 172.21.15.27:0/2792600764 reap_dead start 2024-03-06T07:30:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.578+0000 7f0c5949e640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c400039b0 0x7f0c40003dd0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009ac0 tx=0x7f0c40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3917b327a18b969 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.578+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44039070 con 0x7f0c400039b0 2024-03-06T07:30:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.578+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4402c6b0 con 0x7f0c400039b0 2024-03-06T07:30:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.578+0000 7f0c4bfff640 1 -- 172.21.15.27:0/2792600764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44009690 con 0x7f0c400039b0 2024-03-06T07:30:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.578+0000 7f0c5b702640 1 -- 172.21.15.27:0/2792600764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c400039b0 msgr2=0x7f0c40003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.578+0000 7f0c5b702640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c400039b0 0x7f0c40003dd0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009ac0 tx=0x7f0c40005830 comp rx=0 tx=0).stop 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.578+0000 7f0c5b702640 1 -- 172.21.15.27:0/2792600764 shutdown_connections 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.578+0000 7f0c5b702640 1 --2- 172.21.15.27:0/2792600764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c400039b0 0x7f0c40003dd0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.578+0000 7f0c5b702640 1 -- 172.21.15.27:0/2792600764 >> 172.21.15.27:0/2792600764 conn(0x7f0c54071960 msgr2=0x7f0c54071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.579+0000 7f0c5b702640 1 -- 172.21.15.27:0/2792600764 shutdown_connections 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.579+0000 7f0c5b702640 1 -- 172.21.15.27:0/2792600764 wait complete. 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.580+0000 7f0c5b702640 1 Processor -- start 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.580+0000 7f0c5b702640 1 -- start start 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c5b702640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400039b0 0x7f0c541551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c5b702640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c540b4aa0 0x7f0c540b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c5b702640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c540cc710 0x7f0c540ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c5b702640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540cec70 con 0x7f0c540b4aa0 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c5b702640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540ce750 con 0x7f0c400039b0 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c5b702640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540cedf0 con 0x7f0c540cc710 2024-03-06T07:30:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c58c9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c540b4aa0 0x7f0c540b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c58c9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c540b4aa0 0x7f0c540b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41860/0 (socket says 172.21.15.27:41860) 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c58c9d640 1 -- 172.21.15.27:0/3720853449 learned_addr learned my addr 172.21.15.27:0/3720853449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c59c9f640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c540cc710 0x7f0c540ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c5949e640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400039b0 0x7f0c541551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c58c9d640 1 -- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c540cc710 msgr2=0x7f0c540ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c58c9d640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c540cc710 0x7f0c540ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c58c9d640 1 -- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400039b0 msgr2=0x7f0c541551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c58c9d640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400039b0 0x7f0c541551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c58c9d640 1 -- 172.21.15.27:0/3720853449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c540cafa0 con 0x7f0c540b4aa0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c5949e640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400039b0 0x7f0c541551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c59c9f640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c540cc710 0x7f0c540ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c58c9d640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c540b4aa0 0x7f0c540b4f00 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000e500 tx=0x7f0c5000e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.581+0000 7f0c49ffb640 1 -- 172.21.15.27:0/3720853449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c500053a0 con 0x7f0c540b4aa0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.582+0000 7f0c49ffb640 1 -- 172.21.15.27:0/3720853449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c50018070 con 0x7f0c540b4aa0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.582+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c540cb280 con 0x7f0c540b4aa0 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.582+0000 7f0c49ffb640 1 -- 172.21.15.27:0/3720853449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50003810 con 0x7f0c540b4aa0 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.582+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c540cb840 con 0x7f0c540b4aa0 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.583+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c540cad10 con 0x7f0c540b4aa0 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.584+0000 7f0c49ffb640 1 -- 172.21.15.27:0/3720853449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c500039b0 con 0x7f0c540b4aa0 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.584+0000 7f0c49ffb640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3806b4d0 0x7f0c3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.584+0000 7f0c49ffb640 1 -- 172.21.15.27:0/3720853449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0c50089f40 con 0x7f0c540b4aa0 2024-03-06T07:30:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.584+0000 7f0c5949e640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3806b4d0 0x7f0c3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.585+0000 7f0c5949e640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3806b4d0 0x7f0c3806d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0c44036000 tx=0x7f0c44030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.593+0000 7f0c49ffb640 1 -- 172.21.15.27:0/3720853449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c50058d00 con 0x7f0c540b4aa0 2024-03-06T07:30:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:37.954+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f0c54067ff0 con 0x7f0c540b4aa0 2024-03-06T07:30:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.441+0000 7f0c49ffb640 1 -- 172.21.15.27:0/3720853449 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6851 v97) v1 ==== 95+0+0 (secure 0 0 0) 0x7f0c50058600 con 0x7f0c540b4aa0 2024-03-06T07:30:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 6851 2024-03-06T07:30:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3806b4d0 msgr2=0x7f0c3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3806b4d0 0x7f0c3806d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0c44036000 tx=0x7f0c44030000 comp rx=0 tx=0).stop 2024-03-06T07:30:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c540b4aa0 msgr2=0x7f0c540b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c540b4aa0 0x7f0c540b4f00 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000e500 tx=0x7f0c5000e9c0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 shutdown_connections 2024-03-06T07:30:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3806b4d0 0x7f0c3806d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c540cc710 0x7f0c540ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c540b4aa0 0x7f0c540b4f00 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 --2- 172.21.15.27:0/3720853449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c400039b0 0x7f0c541551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 >> 172.21.15.27:0/3720853449 conn(0x7f0c54071960 msgr2=0x7f0c541618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 shutdown_connections 2024-03-06T07:30:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.444+0000 7f0c5b702640 1 -- 172.21.15.27:0/3720853449 wait complete. 2024-03-06T07:30:38.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:38.456 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:30:38.457 DEBUG:teuthology.orchestra.run.smithi027:> mkdir pin 2024-03-06T07:30:38.457 DEBUG:teuthology.orchestra.run.smithi027:> touch pin/placeholder 2024-03-06T07:30:38.457 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:30:38.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:38.486 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:30:38.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.691+0000 7f17a751e640 1 Processor -- start 2024-03-06T07:30:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.691+0000 7f17a751e640 1 -- start start 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a751e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17a006df30 0x7f17a00ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a751e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a0071240 con 0x7f17a0158520 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a751e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a0071520 con 0x7f17a00aec40 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a751e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a00af0f0 con 0x7f17a006df30 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a4ab9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17a006df30 0x7f17a00ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a52ba640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f17a0158520 0x7f17a006d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45488/0 (socket says 172.21.15.27:45488) 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a52ba640 1 -- 172.21.15.27:0/1176586183 learned_addr learned my addr 172.21.15.27:0/1176586183 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a4ab9640 1 -- 172.21.15.27:0/1176586183 >> v1:172.21.15.69:6789/0 conn(0x7f17a00aec40 legacy=0x7f17a00b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a4ab9640 1 -- 172.21.15.27:0/1176586183 >> v1:172.21.15.27:6789/0 conn(0x7f17a0158520 legacy=0x7f17a006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f1797fff640 1 -- 172.21.15.27:0/1176586183 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520004869 0 0) 0x7f17a0071240 con 0x7f17a0158520 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.692+0000 7f17a4ab9640 1 -- 172.21.15.27:0/1176586183 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a0071800 con 0x7f17a006df30 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.693+0000 7f17a4ab9640 1 --2- 172.21.15.27:0/1176586183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a006df30 0x7f17a00ae700 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f17900097d0 tx=0x7f179002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6de17a6827aa5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.693+0000 7f1797fff640 1 -- 172.21.15.27:0/1176586183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790034070 con 0x7f17a006df30 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.693+0000 7f1797fff640 1 -- 172.21.15.27:0/1176586183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1790030610 con 0x7f17a006df30 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.693+0000 7f1797fff640 1 -- 172.21.15.27:0/1176586183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17900309e0 con 0x7f17a006df30 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.694+0000 7f17a751e640 1 -- 172.21.15.27:0/1176586183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a006df30 msgr2=0x7f17a00ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.694+0000 7f17a751e640 1 --2- 172.21.15.27:0/1176586183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a006df30 0x7f17a00ae700 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f17900097d0 tx=0x7f179002b960 comp rx=0 tx=0).stop 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.694+0000 7f17a751e640 1 -- 172.21.15.27:0/1176586183 shutdown_connections 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.694+0000 7f17a751e640 1 --2- 172.21.15.27:0/1176586183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a006df30 0x7f17a00ae700 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.694+0000 7f17a751e640 1 -- 172.21.15.27:0/1176586183 >> 172.21.15.27:0/1176586183 conn(0x7f17a0066d00 msgr2=0x7f17a00670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.694+0000 7f17a751e640 1 -- 172.21.15.27:0/1176586183 shutdown_connections 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.694+0000 7f17a751e640 1 -- 172.21.15.27:0/1176586183 wait complete. 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.696+0000 7f17a751e640 1 Processor -- start 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.696+0000 7f17a751e640 1 -- start start 2024-03-06T07:30:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.697+0000 7f17a751e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a006df30 0x7f17a00cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.697+0000 7f17a751e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a00aec40 0x7f17a00a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.697+0000 7f17a751e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a0158520 0x7f17a00cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.697+0000 7f17a751e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a0158900 con 0x7f17a0158520 2024-03-06T07:30:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.697+0000 7f17a751e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a0071b70 con 0x7f17a00aec40 2024-03-06T07:30:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.697+0000 7f17a751e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a0158a80 con 0x7f17a006df30 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a5abb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a0158520 0x7f17a00cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a5abb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a0158520 0x7f17a00cf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41870/0 (socket says 172.21.15.27:41870) 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a5abb640 1 -- 172.21.15.27:0/681750514 learned_addr learned my addr 172.21.15.27:0/681750514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a4ab9640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a00aec40 0x7f17a00a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a52ba640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a006df30 0x7f17a00cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a5abb640 1 -- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a006df30 msgr2=0x7f17a00cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a5abb640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a006df30 0x7f17a00cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a5abb640 1 -- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a00aec40 msgr2=0x7f17a00a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a5abb640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a00aec40 0x7f17a00a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a5abb640 1 -- 172.21.15.27:0/681750514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a00ce570 con 0x7f17a0158520 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a5abb640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a0158520 0x7f17a00cf610 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f179800c7a0 tx=0x7f179800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f1795ffb640 1 -- 172.21.15.27:0/681750514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798003b70 con 0x7f17a0158520 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f1795ffb640 1 -- 172.21.15.27:0/681750514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1798003d30 con 0x7f17a0158520 2024-03-06T07:30:38.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a00ce850 con 0x7f17a0158520 2024-03-06T07:30:38.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.698+0000 7f1795ffb640 1 -- 172.21.15.27:0/681750514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179800aa90 con 0x7f17a0158520 2024-03-06T07:30:38.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.699+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a00cee10 con 0x7f17a0158520 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.700+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a0067ff0 con 0x7f17a0158520 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.700+0000 7f1795ffb640 1 -- 172.21.15.27:0/681750514 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f179800ac30 con 0x7f17a0158520 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.701+0000 7f1795ffb640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178406b4d0 0x7f178406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.701+0000 7f1795ffb640 1 -- 172.21.15.27:0/681750514 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f179808a040 con 0x7f17a0158520 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.701+0000 7f17a52ba640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178406b4d0 0x7f178406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.702+0000 7f17a52ba640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178406b4d0 0x7f178406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f179c007f70 tx=0x7f179c007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.705+0000 7f1795ffb640 1 -- 172.21.15.27:0/681750514 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1798058830 con 0x7f17a0158520 2024-03-06T07:30:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.925+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17a0159530 con 0x7f17a0158520 2024-03-06T07:30:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.925+0000 7f1795ffb640 1 -- 172.21.15.27:0/681750514 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f179805c8d0 con 0x7f17a0158520 2024-03-06T07:30:38.927 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:38.927 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:31.403642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8670,8679]},"id":9}]} 2024-03-06T07:30:38.927 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-06T07:30:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.928+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178406b4d0 msgr2=0x7f178406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.928+0000 7f17a751e640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178406b4d0 0x7f178406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f179c007f70 tx=0x7f179c007f00 comp rx=0 tx=0).stop 2024-03-06T07:30:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.928+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a0158520 msgr2=0x7f17a00cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.928+0000 7f17a751e640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a0158520 0x7f17a00cf610 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f179800c7a0 tx=0x7f179800cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.929+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 shutdown_connections 2024-03-06T07:30:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.929+0000 7f17a751e640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f178406b4d0 0x7f178406d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.929+0000 7f17a751e640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17a0158520 0x7f17a00cf610 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.929+0000 7f17a751e640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17a00aec40 0x7f17a00a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.929+0000 7f17a751e640 1 --2- 172.21.15.27:0/681750514 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f17a006df30 0x7f17a00cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.929+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 >> 172.21.15.27:0/681750514 conn(0x7f17a0066d00 msgr2=0x7f17a0159f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.929+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 shutdown_connections 2024-03-06T07:30:38.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:38.929+0000 7f17a751e640 1 -- 172.21.15.27:0/681750514 wait complete. 2024-03-06T07:30:38.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:39.067 INFO:teuthology.orchestra.run.smithi027.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:39.067 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:39.067 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:30:41.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:41.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.173+0000 7f8a37c1c640 1 Processor -- start 2024-03-06T07:30:41.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.174+0000 7f8a37c1c640 1 -- start start 2024-03-06T07:30:41.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.174+0000 7f8a37c1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a3015df00 0x7f8a300d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.174+0000 7f8a37c1c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a300b95a0 0x7f8a300b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.174+0000 7f8a37c1c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a300cea30 con 0x7f8a300b9a30 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.174+0000 7f8a37c1c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a300d2af0 con 0x7f8a300b95a0 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.174+0000 7f8a37c1c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a300cf3c0 con 0x7f8a3015df00 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a351b7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a300b95a0 0x7f8a300b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a361b9640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8a300b9a30 0x7f8a300d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45494/0 (socket says 172.21.15.27:45494) 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a361b9640 1 -- 172.21.15.27:0/2719678726 learned_addr learned my addr 172.21.15.27:0/2719678726 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a359b8640 1 --2- 172.21.15.27:0/2719678726 >> v2:172.21.15.69:3300/0 conn(0x7f8a3015df00 0x7f8a300d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a351b7640 1 -- 172.21.15.27:0/2719678726 >> v2:172.21.15.69:3300/0 conn(0x7f8a3015df00 msgr2=0x7f8a300d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a351b7640 1 --2- 172.21.15.27:0/2719678726 >> v2:172.21.15.69:3300/0 conn(0x7f8a3015df00 0x7f8a300d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a351b7640 1 -- 172.21.15.27:0/2719678726 >> v1:172.21.15.27:6789/0 conn(0x7f8a300b9a30 legacy=0x7f8a300d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a351b7640 1 -- 172.21.15.27:0/2719678726 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a300cf0e0 con 0x7f8a300b95a0 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a349b6640 1 -- 172.21.15.27:0/2719678726 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384647885 0 0) 0x7f8a300cea30 con 0x7f8a300b9a30 2024-03-06T07:30:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.175+0000 7f8a359b8640 1 --2- 172.21.15.27:0/2719678726 >> v2:172.21.15.69:3300/0 conn(0x7f8a3015df00 0x7f8a300d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:41.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a351b7640 1 --2- 172.21.15.27:0/2719678726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a300b95a0 0x7f8a300b5700 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f8a300d11b0 tx=0x7f8a2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f2c3ed1c70abc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a349b6640 1 -- 172.21.15.27:0/2719678726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20035020 con 0x7f8a300b95a0 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a349b6640 1 -- 172.21.15.27:0/2719678726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a20031830 con 0x7f8a300b95a0 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a349b6640 1 -- 172.21.15.27:0/2719678726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20031c00 con 0x7f8a300b95a0 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a37c1c640 1 -- 172.21.15.27:0/2719678726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a300b95a0 msgr2=0x7f8a300b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a37c1c640 1 --2- 172.21.15.27:0/2719678726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a300b95a0 0x7f8a300b5700 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f8a300d11b0 tx=0x7f8a2002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a37c1c640 1 -- 172.21.15.27:0/2719678726 shutdown_connections 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a37c1c640 1 --2- 172.21.15.27:0/2719678726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a300b95a0 0x7f8a300b5700 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a37c1c640 1 --2- 172.21.15.27:0/2719678726 >> v2:172.21.15.69:3300/0 conn(0x7f8a3015df00 0x7f8a300d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.176+0000 7f8a37c1c640 1 -- 172.21.15.27:0/2719678726 >> 172.21.15.27:0/2719678726 conn(0x7f8a30071960 msgr2=0x7f8a30071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.177+0000 7f8a37c1c640 1 -- 172.21.15.27:0/2719678726 shutdown_connections 2024-03-06T07:30:41.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.177+0000 7f8a37c1c640 1 -- 172.21.15.27:0/2719678726 wait complete. 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.178+0000 7f8a37c1c640 1 Processor -- start 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.178+0000 7f8a37c1c640 1 -- start start 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.178+0000 7f8a37c1c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a300b95a0 0x7f8a30149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.178+0000 7f8a37c1c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a3015df00 0x7f8a300c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a37c1c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300c7cf0 0x7f8a300c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a37c1c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a300cec70 con 0x7f8a3015df00 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a37c1c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a300ce750 con 0x7f8a300c7cf0 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a37c1c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a300cedf0 con 0x7f8a300b95a0 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a351b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a3015df00 0x7f8a300c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a351b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a3015df00 0x7f8a300c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41884/0 (socket says 172.21.15.27:41884) 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a351b7640 1 -- 172.21.15.27:0/3681965837 learned_addr learned my addr 172.21.15.27:0/3681965837 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a359b8640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a300b95a0 0x7f8a30149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a351b7640 1 -- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a300b95a0 msgr2=0x7f8a30149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a351b7640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a300b95a0 0x7f8a30149d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a351b7640 1 -- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300c7cf0 msgr2=0x7f8a300c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a361b9640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300c7cf0 0x7f8a300c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a351b7640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300c7cf0 0x7f8a300c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a351b7640 1 -- 172.21.15.27:0/3681965837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3015ebb0 con 0x7f8a3015df00 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a351b7640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a3015df00 0x7f8a300c77b0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009890 tx=0x7f8a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a267fc640 1 -- 172.21.15.27:0/3681965837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20035050 con 0x7f8a3015df00 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a267fc640 1 -- 172.21.15.27:0/3681965837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a20004230 con 0x7f8a3015df00 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a300ca340 con 0x7f8a3015df00 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a267fc640 1 -- 172.21.15.27:0/3681965837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2003b800 con 0x7f8a3015df00 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.179+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a300ca900 con 0x7f8a3015df00 2024-03-06T07:30:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.180+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a300cab00 con 0x7f8a3015df00 2024-03-06T07:30:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.181+0000 7f8a267fc640 1 -- 172.21.15.27:0/3681965837 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a20038020 con 0x7f8a3015df00 2024-03-06T07:30:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.181+0000 7f8a267fc640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a1406b4d0 0x7f8a1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.182+0000 7f8a267fc640 1 -- 172.21.15.27:0/3681965837 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8a200abfb0 con 0x7f8a3015df00 2024-03-06T07:30:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.182+0000 7f8a359b8640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a1406b4d0 0x7f8a1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.182+0000 7f8a359b8640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a1406b4d0 0x7f8a1406d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00b840 tx=0x7f8a2c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.186+0000 7f8a267fc640 1 -- 172.21.15.27:0/3681965837 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a300cab00 con 0x7f8a3015df00 2024-03-06T07:30:41.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.459+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a30067ff0 con 0x7f8a3015df00 2024-03-06T07:30:41.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.460+0000 7f8a267fc640 1 -- 172.21.15.27:0/3681965837 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f8a20040070 con 0x7f8a3015df00 2024-03-06T07:30:41.462 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:41.463 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:31.403642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8670,8679]},"id":9}]} 2024-03-06T07:30:41.463 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a1406b4d0 msgr2=0x7f8a1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a1406b4d0 0x7f8a1406d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00b840 tx=0x7f8a2c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a3015df00 msgr2=0x7f8a300c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a3015df00 0x7f8a300c77b0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009890 tx=0x7f8a200046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 shutdown_connections 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8a1406b4d0 0x7f8a1406d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300c7cf0 0x7f8a300c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a3015df00 0x7f8a300c77b0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 --2- 172.21.15.27:0/3681965837 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a300b95a0 0x7f8a30149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 >> 172.21.15.27:0/3681965837 conn(0x7f8a30071960 msgr2=0x7f8a300b8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 shutdown_connections 2024-03-06T07:30:41.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:41.462+0000 7f8a37c1c640 1 -- 172.21.15.27:0/3681965837 wait complete. 2024-03-06T07:30:41.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:41.597 INFO:teuthology.orchestra.run.smithi027.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:41.597 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:41.597 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:30:43.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:43.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.703+0000 7fb4094f4640 1 Processor -- start 2024-03-06T07:30:43.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.704+0000 7fb4094f4640 1 -- start start 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.704+0000 7fb4094f4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb40415df00 0x7fb4040d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.704+0000 7fb4094f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4040b95a0 0x7fb4040b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.704+0000 7fb4094f4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4040b9a30 0x7fb4040b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.704+0000 7fb4094f4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4040cea30 con 0x7fb4040b9a30 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.704+0000 7fb4094f4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4040d2af0 con 0x7fb40415df00 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.704+0000 7fb4094f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040cf3c0 con 0x7fb4040b95a0 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.704+0000 7fb4037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4040b9a30 0x7fb4040b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.705+0000 7fb4037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4040b9a30 0x7fb4040b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55234/0 (socket says 172.21.15.27:55234) 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.705+0000 7fb4037fe640 1 -- 172.21.15.27:0/398378678 learned_addr learned my addr 172.21.15.27:0/398378678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.705+0000 7fb402ffd640 1 --2- 172.21.15.27:0/398378678 >> v2:172.21.15.69:3300/0 conn(0x7fb40415df00 0x7fb4040d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.705+0000 7fb4037fe640 1 -- 172.21.15.27:0/398378678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4040b95a0 msgr2=0x7fb4040b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.705+0000 7fb4037fe640 1 --2- 172.21.15.27:0/398378678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4040b95a0 0x7fb4040b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.705+0000 7fb4037fe640 1 -- 172.21.15.27:0/398378678 >> v2:172.21.15.69:3300/0 conn(0x7fb40415df00 msgr2=0x7fb4040d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.705+0000 7fb4037fe640 1 --2- 172.21.15.27:0/398378678 >> v2:172.21.15.69:3300/0 conn(0x7fb40415df00 0x7fb4040d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.705+0000 7fb4037fe640 1 -- 172.21.15.27:0/398378678 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4040cf0e0 con 0x7fb4040b9a30 2024-03-06T07:30:43.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4037fe640 1 --2- 172.21.15.27:0/398378678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b9a30 0x7fb4040b5c40 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fb4040d11b0 tx=0x7fb3fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9772349b37f12e57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb401ffb640 1 -- 172.21.15.27:0/398378678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc02ca70 con 0x7fb4040b9a30 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb401ffb640 1 -- 172.21.15.27:0/398378678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3fc02cc10 con 0x7fb4040b9a30 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb401ffb640 1 -- 172.21.15.27:0/398378678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc0343f0 con 0x7fb4040b9a30 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4094f4640 1 -- 172.21.15.27:0/398378678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b9a30 msgr2=0x7fb4040b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4094f4640 1 --2- 172.21.15.27:0/398378678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b9a30 0x7fb4040b5c40 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fb4040d11b0 tx=0x7fb3fc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4094f4640 1 -- 172.21.15.27:0/398378678 shutdown_connections 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4094f4640 1 --2- 172.21.15.27:0/398378678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b9a30 0x7fb4040b5c40 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4094f4640 1 --2- 172.21.15.27:0/398378678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb4040b95a0 0x7fb4040b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4094f4640 1 --2- 172.21.15.27:0/398378678 >> v2:172.21.15.69:3300/0 conn(0x7fb40415df00 0x7fb4040d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4094f4640 1 -- 172.21.15.27:0/398378678 >> 172.21.15.27:0/398378678 conn(0x7fb404071960 msgr2=0x7fb404071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4094f4640 1 -- 172.21.15.27:0/398378678 shutdown_connections 2024-03-06T07:30:43.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.706+0000 7fb4094f4640 1 -- 172.21.15.27:0/398378678 wait complete. 2024-03-06T07:30:43.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.708+0000 7fb4094f4640 1 Processor -- start 2024-03-06T07:30:43.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.709+0000 7fb4094f4640 1 -- start start 2024-03-06T07:30:43.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb4094f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b95a0 0x7fb404158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb4094f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb40415df00 0x7fb4041f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb4094f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4041f3ce0 0x7fb4040cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb4094f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040cec70 con 0x7fb4040b95a0 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb4094f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040ce750 con 0x7fb4041f3ce0 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb4094f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040cedf0 con 0x7fb40415df00 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb402ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b95a0 0x7fb404158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb402ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b95a0 0x7fb404158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55248/0 (socket says 172.21.15.27:55248) 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb402ffd640 1 -- 172.21.15.27:0/3318664312 learned_addr learned my addr 172.21.15.27:0/3318664312 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb4037fe640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4041f3ce0 0x7fb4040cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb4027fc640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb40415df00 0x7fb4041f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb402ffd640 1 -- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb40415df00 msgr2=0x7fb4041f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb402ffd640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb40415df00 0x7fb4041f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb402ffd640 1 -- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4041f3ce0 msgr2=0x7fb4040cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb402ffd640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4041f3ce0 0x7fb4040cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb402ffd640 1 -- 172.21.15.27:0/3318664312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4041f4200 con 0x7fb4040b95a0 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.710+0000 7fb402ffd640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b95a0 0x7fb404158880 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c7f0 tx=0x7fb3f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.711+0000 7fb3eb7fe640 1 -- 172.21.15.27:0/3318664312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4015070 con 0x7fb4040b95a0 2024-03-06T07:30:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.711+0000 7fb3eb7fe640 1 -- 172.21.15.27:0/3318664312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4004240 con 0x7fb4040b95a0 2024-03-06T07:30:43.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.711+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4041f44e0 con 0x7fb4040b95a0 2024-03-06T07:30:43.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.711+0000 7fb3eb7fe640 1 -- 172.21.15.27:0/3318664312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4007620 con 0x7fb4040b95a0 2024-03-06T07:30:43.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.711+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb404158dc0 con 0x7fb4040b95a0 2024-03-06T07:30:43.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.712+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb404067ff0 con 0x7fb4040b95a0 2024-03-06T07:30:43.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.713+0000 7fb3eb7fe640 1 -- 172.21.15.27:0/3318664312 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3f40079d0 con 0x7fb4040b95a0 2024-03-06T07:30:43.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.713+0000 7fb3eb7fe640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3f006b500 0x7fb3f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.714+0000 7fb3eb7fe640 1 -- 172.21.15.27:0/3318664312 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb3f408a2b0 con 0x7fb4040b95a0 2024-03-06T07:30:43.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.714+0000 7fb4027fc640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3f006b500 0x7fb3f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.714+0000 7fb4027fc640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3f006b500 0x7fb3f006d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb40415d540 tx=0x7fb3f8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:43.723+0000 7fb3eb7fe640 1 -- 172.21.15.27:0/3318664312 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3f4059180 con 0x7fb4040b95a0 2024-03-06T07:30:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.013+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb40406b780 con 0x7fb4040b95a0 2024-03-06T07:30:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.013+0000 7fb3eb7fe640 1 -- 172.21.15.27:0/3318664312 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3417 (secure 0 0 0) 0x7fb3f4058a80 con 0x7fb4040b95a0 2024-03-06T07:30:44.016 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:44.016 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:31.403642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8670,8679]},"id":9}]} 2024-03-06T07:30:44.016 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-06T07:30:44.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3f006b500 msgr2=0x7fb3f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3f006b500 0x7fb3f006d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb40415d540 tx=0x7fb3f8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:44.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b95a0 msgr2=0x7fb404158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b95a0 0x7fb404158880 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c7f0 tx=0x7fb3f400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 shutdown_connections 2024-03-06T07:30:44.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb3f006b500 0x7fb3f006d9c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4041f3ce0 0x7fb4040cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb40415df00 0x7fb4041f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 --2- 172.21.15.27:0/3318664312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4040b95a0 0x7fb404158880 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 >> 172.21.15.27:0/3318664312 conn(0x7fb404071960 msgr2=0x7fb4040b8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 shutdown_connections 2024-03-06T07:30:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:44.015+0000 7fb4094f4640 1 -- 172.21.15.27:0/3318664312 wait complete. 2024-03-06T07:30:44.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:44.150 INFO:teuthology.orchestra.run.smithi027.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:44.150 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:44.151 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:30:46.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.258+0000 7fe7e7925640 1 Processor -- start 2024-03-06T07:30:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.259+0000 7fe7e7925640 1 -- start start 2024-03-06T07:30:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.260+0000 7fe7e7925640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 0x7fe7e00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.260+0000 7fe7e7925640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e00cea30 con 0x7fe7e00b9a30 2024-03-06T07:30:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.260+0000 7fe7e7925640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e00d2af0 con 0x7fe7e00b95a0 2024-03-06T07:30:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.260+0000 7fe7e7925640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00cf3c0 con 0x7fe7e015df00 2024-03-06T07:30:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.260+0000 7fe7e5ec2640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe7e00b9a30 0x7fe7e00d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39870/0 (socket says 172.21.15.27:39870) 2024-03-06T07:30:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.260+0000 7fe7e5ec2640 1 -- 172.21.15.27:0/35193450 learned_addr learned my addr 172.21.15.27:0/35193450 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.260+0000 7fe7e56c1640 1 --2- 172.21.15.27:0/35193450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 0x7fe7e00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.260+0000 7fe7d7fff640 1 -- 172.21.15.27:0/35193450 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154053074 0 0) 0x7fe7e00cea30 con 0x7fe7e00b9a30 2024-03-06T07:30:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.261+0000 7fe7d7fff640 1 -- 172.21.15.27:0/35193450 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7cc003520 con 0x7fe7e00b9a30 2024-03-06T07:30:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.261+0000 7fe7d7fff640 1 -- 172.21.15.27:0/35193450 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561247015 0 0) 0x7fe7e00d2af0 con 0x7fe7e00b95a0 2024-03-06T07:30:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.261+0000 7fe7d7fff640 1 -- 172.21.15.27:0/35193450 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e00cea30 con 0x7fe7e00b95a0 2024-03-06T07:30:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.261+0000 7fe7d7fff640 1 -- 172.21.15.27:0/35193450 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2338206084 0 0) 0x7fe7cc003520 con 0x7fe7e00b9a30 2024-03-06T07:30:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.261+0000 7fe7e56c1640 1 -- 172.21.15.27:0/35193450 >> v1:172.21.15.69:6789/0 conn(0x7fe7e00b95a0 legacy=0x7fe7e00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.261+0000 7fe7e56c1640 1 -- 172.21.15.27:0/35193450 >> v1:172.21.15.27:6789/0 conn(0x7fe7e00b9a30 legacy=0x7fe7e00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.261+0000 7fe7e56c1640 1 -- 172.21.15.27:0/35193450 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e00cf0e0 con 0x7fe7e015df00 2024-03-06T07:30:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.262+0000 7fe7e56c1640 1 --2- 172.21.15.27:0/35193450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 0x7fe7e00d25b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0097d0 tx=0x7fe7dc02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9873da2d5a24c193 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.262+0000 7fe7d7fff640 1 -- 172.21.15.27:0/35193450 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc037070 con 0x7fe7e015df00 2024-03-06T07:30:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.262+0000 7fe7d7fff640 1 -- 172.21.15.27:0/35193450 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc030510 con 0x7fe7e015df00 2024-03-06T07:30:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.262+0000 7fe7d7fff640 1 -- 172.21.15.27:0/35193450 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80019f0 con 0x7fe7e015df00 2024-03-06T07:30:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.262+0000 7fe7e7925640 1 -- 172.21.15.27:0/35193450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 msgr2=0x7fe7e00d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.262+0000 7fe7e7925640 1 --2- 172.21.15.27:0/35193450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 0x7fe7e00d25b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0097d0 tx=0x7fe7dc02b800 comp rx=0 tx=0).stop 2024-03-06T07:30:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.262+0000 7fe7e7925640 1 -- 172.21.15.27:0/35193450 shutdown_connections 2024-03-06T07:30:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.262+0000 7fe7e7925640 1 --2- 172.21.15.27:0/35193450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 0x7fe7e00d25b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.262+0000 7fe7e7925640 1 -- 172.21.15.27:0/35193450 >> 172.21.15.27:0/35193450 conn(0x7fe7e0071960 msgr2=0x7fe7e0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.263+0000 7fe7e7925640 1 -- 172.21.15.27:0/35193450 shutdown_connections 2024-03-06T07:30:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.263+0000 7fe7e7925640 1 -- 172.21.15.27:0/35193450 wait complete. 2024-03-06T07:30:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.264+0000 7fe7e7925640 1 Processor -- start 2024-03-06T07:30:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.264+0000 7fe7e7925640 1 -- start start 2024-03-06T07:30:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.264+0000 7fe7e7925640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e00b95a0 0x7fe7e0155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.264+0000 7fe7e7925640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 0x7fe7e00c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.264+0000 7fe7e7925640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0155d90 0x7fe7e00c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.264+0000 7fe7e7925640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00cec70 con 0x7fe7e00b95a0 2024-03-06T07:30:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.264+0000 7fe7e7925640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00ce750 con 0x7fe7e0155d90 2024-03-06T07:30:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.264+0000 7fe7e7925640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00cedf0 con 0x7fe7e015df00 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.264+0000 7fe7e56c1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e00b95a0 0x7fe7e0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e56c1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e00b95a0 0x7fe7e0155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55258/0 (socket says 172.21.15.27:55258) 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e56c1640 1 -- 172.21.15.27:0/1726213789 learned_addr learned my addr 172.21.15.27:0/1726213789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e5ec2640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0155d90 0x7fe7e00c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e56c1640 1 -- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 msgr2=0x7fe7e00c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e4ec0640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 0x7fe7e00c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e56c1640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 0x7fe7e00c7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e56c1640 1 -- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0155d90 msgr2=0x7fe7e00c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e56c1640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0155d90 0x7fe7e00c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e56c1640 1 -- 172.21.15.27:0/1726213789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e01602b0 con 0x7fe7e00b95a0 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e5ec2640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0155d90 0x7fe7e00c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e56c1640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e00b95a0 0x7fe7e0155850 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0097d0 tx=0x7fe7dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7d5ffb640 1 -- 172.21.15.27:0/1726213789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc042070 con 0x7fe7e00b95a0 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7d5ffb640 1 -- 172.21.15.27:0/1726213789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc004220 con 0x7fe7e00b95a0 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7d5ffb640 1 -- 172.21.15.27:0/1726213789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc037040 con 0x7fe7e00b95a0 2024-03-06T07:30:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.265+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e0160590 con 0x7fe7e00b95a0 2024-03-06T07:30:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.266+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e0156460 con 0x7fe7e00b95a0 2024-03-06T07:30:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.267+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e0067ff0 con 0x7fe7e00b95a0 2024-03-06T07:30:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.268+0000 7fe7d5ffb640 1 -- 172.21.15.27:0/1726213789 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7dc003610 con 0x7fe7e00b95a0 2024-03-06T07:30:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.268+0000 7fe7d5ffb640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7c406b500 0x7fe7c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.268+0000 7fe7d5ffb640 1 -- 172.21.15.27:0/1726213789 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe7dc0abe90 con 0x7fe7e00b95a0 2024-03-06T07:30:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.268+0000 7fe7e4ec0640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7c406b500 0x7fe7c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.269+0000 7fe7e4ec0640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7c406b500 0x7fe7c406d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00010e0 tx=0x7fe7d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.277+0000 7fe7d5ffb640 1 -- 172.21.15.27:0/1726213789 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7dc07acd0 con 0x7fe7e00b95a0 2024-03-06T07:30:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.512+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7e00b2710 con 0x7fe7e00b95a0 2024-03-06T07:30:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.512+0000 7fe7d5ffb640 1 -- 172.21.15.27:0/1726213789 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3418 (secure 0 0 0) 0x7fe7dc046e90 con 0x7fe7e00b95a0 2024-03-06T07:30:46.515 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:46.515 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:45.351386+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679,8670]},"id":9}]} 2024-03-06T07:30:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 98 2024-03-06T07:30:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7c406b500 msgr2=0x7fe7c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7c406b500 0x7fe7c406d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00010e0 tx=0x7fe7d0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e00b95a0 msgr2=0x7fe7e0155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e00b95a0 0x7fe7e0155850 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0097d0 tx=0x7fe7dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 shutdown_connections 2024-03-06T07:30:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe7c406b500 0x7fe7c406d9c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7e0155d90 0x7fe7e00c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7e015df00 0x7fe7e00c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 --2- 172.21.15.27:0/1726213789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e00b95a0 0x7fe7e0155850 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 >> 172.21.15.27:0/1726213789 conn(0x7fe7e0071960 msgr2=0x7fe7e00b8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 shutdown_connections 2024-03-06T07:30:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:46.514+0000 7fe7e7925640 1 -- 172.21.15.27:0/1726213789 wait complete. 2024-03-06T07:30:46.533 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:46.655 INFO:teuthology.orchestra.run.smithi027.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"10","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:30:46.656 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "10" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:46.656 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:30:46.656 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:30:46.656 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:46.672 DEBUG:teuthology.orchestra.run.smithi027:> stdin-killer -- python3 -c ' 2024-03-06T07:30:46.672 DEBUG:teuthology.orchestra.run.smithi027:> import fcntl 2024-03-06T07:30:46.672 DEBUG:teuthology.orchestra.run.smithi027:> import sys 2024-03-06T07:30:46.672 DEBUG:teuthology.orchestra.run.smithi027:> import time 2024-03-06T07:30:46.672 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:30:46.672 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:30:46.672 DEBUG:teuthology.orchestra.run.smithi027:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> while True: 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> try: 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> if os.read(0, 4096) == b"": 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> break 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> except BlockingIOError: 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> pass 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> time.sleep(2) 2024-03-06T07:30:46.673 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:30:46.714 DEBUG:teuthology.orchestra.run.smithi027:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:30:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:30:46.734 INFO:teuthology.orchestra.run.smithi027.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:30:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:47.735 DEBUG:teuthology.orchestra.run.smithi027:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 2bh/43d Inode: 2199023255552 Links: 2 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-06 07:30:46.782018849 +0000 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-06 07:30:46.782018849 +0000 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-06 07:30:46.782018849 +0000 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-03-06T07:30:47.752 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:30:47.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:47.752 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:30:47.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:30:48.015 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:30:48.058 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:30:48.059 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:30:48.181 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:30:48.182 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:30:48.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:30:48.321 INFO:teuthology.orchestra.run.smithi027.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:30:48.321 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:30:48.322 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:30:48.459 INFO:teuthology.orchestra.run.smithi027.stdout:{"message":"","return_code":0} 2024-03-06T07:30:48.460 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:30:48.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json cache drop 2024-03-06T07:30:48.589 INFO:teuthology.orchestra.run.smithi027.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":323,"bytes":64695}},"duration":0.0057013740000000004} 2024-03-06T07:30:48.589 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 64695, "items": 323 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.005701374, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:30:48.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:30:48.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.697+0000 7fe763c50640 1 Processor -- start 2024-03-06T07:30:48.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.698+0000 7fe763c50640 1 -- start start 2024-03-06T07:30:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.698+0000 7fe763c50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c0ceea0 0x7fe75c0d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.698+0000 7fe763c50640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0d3740 con 0x7fe75c0b6d10 2024-03-06T07:30:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.698+0000 7fe763c50640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0d3120 con 0x7fe75c15dd90 2024-03-06T07:30:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.698+0000 7fe763c50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0d3400 con 0x7fe75c0ceea0 2024-03-06T07:30:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7621ed640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe75c0b6d10 0x7fe75c0d0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39878/0 (socket says 172.21.15.27:39878) 2024-03-06T07:30:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7621ed640 1 -- 172.21.15.27:0/4251857343 learned_addr learned my addr 172.21.15.27:0/4251857343 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:48.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7611eb640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c0ceea0 0x7fe75c0d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535737430 0 0) 0x7fe75c0d3740 con 0x7fe75c0b6d10 2024-03-06T07:30:48.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe748003520 con 0x7fe75c0b6d10 2024-03-06T07:30:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1418038330 0 0) 0x7fe748003520 con 0x7fe75c0b6d10 2024-03-06T07:30:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c0ceea0 msgr2=0x7fe75c0d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c0ceea0 0x7fe75c0d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 >> v1:172.21.15.69:6789/0 conn(0x7fe75c15dd90 legacy=0x7fe75c0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0d39e0 con 0x7fe75c0b6d10 2024-03-06T07:30:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fe7540019f0 con 0x7fe75c0b6d10 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 >> v1:172.21.15.27:6789/0 conn(0x7fe75c0b6d10 legacy=0x7fe75c0d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7480039e0 0x7fe748003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe748004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c0ceea0 0x7fe7480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0d3400 con 0x7fe748004800 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74800bb00 con 0x7fe7480039e0 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.699+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74800bd00 con 0x7fe75c0ceea0 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.700+0000 7fe7611eb640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe748004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.700+0000 7fe7621ed640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c0ceea0 0x7fe7480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.700+0000 7fe7619ec640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7480039e0 0x7fe748003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.700+0000 7fe7611eb640 1 -- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c0ceea0 msgr2=0x7fe7480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.700+0000 7fe7611eb640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c0ceea0 0x7fe7480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.700+0000 7fe7611eb640 1 -- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7480039e0 msgr2=0x7fe748003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.700+0000 7fe7611eb640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7480039e0 0x7fe748003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.700+0000 7fe7611eb640 1 -- 172.21.15.27:0/4251857343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748003520 con 0x7fe748004800 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.700+0000 7fe7619ec640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7480039e0 0x7fe748003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.701+0000 7fe7611eb640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe748004c80 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009a90 tx=0x7fe748005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bbe4863c4469cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.701+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c039070 con 0x7fe748004800 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.701+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c02c680 con 0x7fe748004800 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.701+0000 7fe7609ea640 1 -- 172.21.15.27:0/4251857343 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c009690 con 0x7fe748004800 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.701+0000 7fe763c50640 1 -- 172.21.15.27:0/4251857343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 msgr2=0x7fe748004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.701+0000 7fe763c50640 1 --2- 172.21.15.27:0/4251857343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe748004c80 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009a90 tx=0x7fe748005830 comp rx=0 tx=0).stop 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.701+0000 7fe7621ed640 1 -- 172.21.15.27:0/4251857343 reap_dead start 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.701+0000 7fe763c50640 1 -- 172.21.15.27:0/4251857343 shutdown_connections 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.701+0000 7fe763c50640 1 -- 172.21.15.27:0/4251857343 >> 172.21.15.27:0/4251857343 conn(0x7fe75c071960 msgr2=0x7fe75c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.702+0000 7fe763c50640 1 -- 172.21.15.27:0/4251857343 shutdown_connections 2024-03-06T07:30:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.702+0000 7fe763c50640 1 -- 172.21.15.27:0/4251857343 wait complete. 2024-03-06T07:30:48.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.703+0000 7fe763c50640 1 Processor -- start 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.703+0000 7fe763c50640 1 -- start start 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.703+0000 7fe763c50640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe75c151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.703+0000 7fe763c50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe75c1601b0 0x7fe75c160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.703+0000 7fe763c50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c160b70 0x7fe75c154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe763c50640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0d48f0 con 0x7fe748004800 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe763c50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c15e170 con 0x7fe75c1601b0 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe763c50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0d4a70 con 0x7fe75c160b70 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7619ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe75c151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7619ec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe75c151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55278/0 (socket says 172.21.15.27:55278) 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7619ec640 1 -- 172.21.15.27:0/1063321710 learned_addr learned my addr 172.21.15.27:0/1063321710 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7611eb640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe75c1601b0 0x7fe75c160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7621ed640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c160b70 0x7fe75c154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7619ec640 1 -- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c160b70 msgr2=0x7fe75c154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7619ec640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c160b70 0x7fe75c154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7619ec640 1 -- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe75c1601b0 msgr2=0x7fe75c160630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7619ec640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe75c1601b0 0x7fe75c160630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7619ec640 1 -- 172.21.15.27:0/1063321710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0c8260 con 0x7fe748004800 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7619ec640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe75c151e80 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe75800e530 tx=0x7fe75800e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7527fc640 1 -- 172.21.15.27:0/1063321710 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7580052e0 con 0x7fe748004800 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7527fc640 1 -- 172.21.15.27:0/1063321710 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758018070 con 0x7fe748004800 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c0c8500 con 0x7fe748004800 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe7527fc640 1 -- 172.21.15.27:0/1063321710 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758003810 con 0x7fe748004800 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.704+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c161230 con 0x7fe748004800 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.706+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75c072a30 con 0x7fe748004800 2024-03-06T07:30:48.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.706+0000 7fe7527fc640 1 -- 172.21.15.27:0/1063321710 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7580039b0 con 0x7fe748004800 2024-03-06T07:30:48.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.706+0000 7fe7527fc640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74006b4d0 0x7fe74006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.707+0000 7fe7527fc640 1 -- 172.21.15.27:0/1063321710 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe758089e40 con 0x7fe748004800 2024-03-06T07:30:48.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.707+0000 7fe7611eb640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74006b4d0 0x7fe74006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.707+0000 7fe7611eb640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74006b4d0 0x7fe74006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe74c036000 tx=0x7fe74c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.716+0000 7fe7527fc640 1 -- 172.21.15.27:0/1063321710 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe758058c80 con 0x7fe748004800 2024-03-06T07:30:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:48.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:48.994+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fe75c04f410 con 0x7fe748004800 2024-03-06T07:30:49.357 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:49.360 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.462+0000 7fe7527fc640 1 -- 172.21.15.27:0/1063321710 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v99) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe758058580 con 0x7fe748004800 2024-03-06T07:30:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74006b4d0 msgr2=0x7fe74006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74006b4d0 0x7fe74006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe74c036000 tx=0x7fe74c030000 comp rx=0 tx=0).stop 2024-03-06T07:30:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 msgr2=0x7fe75c151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe75c151e80 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe75800e530 tx=0x7fe75800e930 comp rx=0 tx=0).stop 2024-03-06T07:30:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 shutdown_connections 2024-03-06T07:30:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74006b4d0 0x7fe74006d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe75c160b70 0x7fe75c154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe75c1601b0 0x7fe75c160630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 --2- 172.21.15.27:0/1063321710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748004800 0x7fe75c151e80 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 >> 172.21.15.27:0/1063321710 conn(0x7fe75c071960 msgr2=0x7fe75c15be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 shutdown_connections 2024-03-06T07:30:49.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.465+0000 7fe763c50640 1 -- 172.21.15.27:0/1063321710 wait complete. 2024-03-06T07:30:49.478 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:49.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.584+0000 7f7ee6fb6640 1 Processor -- start 2024-03-06T07:30:49.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.584+0000 7f7ee6fb6640 1 -- start start 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee6fb6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ee015df00 0x7f7ee00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee6fb6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee00b95a0 0x7f7ee00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee6fb6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee00cea30 con 0x7f7ee00b9a30 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee6fb6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee00d2af0 con 0x7f7ee015df00 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee6fb6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee00cf3c0 con 0x7f7ee00b95a0 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee4d52640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ee015df00 0x7f7ee00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee4d52640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ee015df00 0x7f7ee00d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55282/0 (socket says 172.21.15.27:55282) 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee4d52640 1 -- 172.21.15.27:0/3712917652 learned_addr learned my addr 172.21.15.27:0/3712917652 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7edffff640 1 --2- 172.21.15.27:0/3712917652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee00b95a0 0x7f7ee00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee4d52640 1 -- 172.21.15.27:0/3712917652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee00b95a0 msgr2=0x7f7ee00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee4d52640 1 --2- 172.21.15.27:0/3712917652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee00b95a0 0x7f7ee00b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee4d52640 1 -- 172.21.15.27:0/3712917652 >> v1:172.21.15.69:6789/0 conn(0x7f7ee00b9a30 legacy=0x7f7ee00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.585+0000 7f7ee4d52640 1 -- 172.21.15.27:0/3712917652 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee00cf0e0 con 0x7f7ee015df00 2024-03-06T07:30:49.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.586+0000 7f7ee4d52640 1 --2- 172.21.15.27:0/3712917652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 0x7f7ee00d25b0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009870 tx=0x7f7ed402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d27497727f3d303 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.586+0000 7f7edf7fe640 1 -- 172.21.15.27:0/3712917652 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4035020 con 0x7f7ee015df00 2024-03-06T07:30:49.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.586+0000 7f7edf7fe640 1 -- 172.21.15.27:0/3712917652 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed4031640 con 0x7f7ee015df00 2024-03-06T07:30:49.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.586+0000 7f7edf7fe640 1 -- 172.21.15.27:0/3712917652 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed402d040 con 0x7f7ee015df00 2024-03-06T07:30:49.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.587+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3712917652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 msgr2=0x7f7ee00d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.587+0000 7f7ee6fb6640 1 --2- 172.21.15.27:0/3712917652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 0x7f7ee00d25b0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009870 tx=0x7f7ed402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.587+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3712917652 shutdown_connections 2024-03-06T07:30:49.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.587+0000 7f7ee6fb6640 1 --2- 172.21.15.27:0/3712917652 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee00b95a0 0x7f7ee00b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.587+0000 7f7ee6fb6640 1 --2- 172.21.15.27:0/3712917652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 0x7f7ee00d25b0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.587+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3712917652 >> 172.21.15.27:0/3712917652 conn(0x7f7ee0071960 msgr2=0x7f7ee0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.587+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3712917652 shutdown_connections 2024-03-06T07:30:49.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.587+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3712917652 wait complete. 2024-03-06T07:30:49.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.588+0000 7f7ee6fb6640 1 Processor -- start 2024-03-06T07:30:49.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.588+0000 7f7ee6fb6640 1 -- start start 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7ee6fb6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ee00b95a0 0x7f7ee015a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7ee6fb6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 0x7f7ee015e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7ee6fb6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee015abc0 0x7f7ee015b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7ee6fb6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee00cec70 con 0x7f7ee015df00 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7ee6fb6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee00ce750 con 0x7f7ee00b95a0 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7ee6fb6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee00cedf0 con 0x7f7ee015abc0 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7edffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 0x7f7ee015e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7edffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 0x7f7ee015e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55298/0 (socket says 172.21.15.27:55298) 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7edffff640 1 -- 172.21.15.27:0/3604000866 learned_addr learned my addr 172.21.15.27:0/3604000866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7ee4d52640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ee00b95a0 0x7f7ee015a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7ee5553640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee015abc0 0x7f7ee015b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7edffff640 1 -- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee015abc0 msgr2=0x7f7ee015b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7edffff640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee015abc0 0x7f7ee015b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7edffff640 1 -- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ee00b95a0 msgr2=0x7f7ee015a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7edffff640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ee00b95a0 0x7f7ee015a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7edffff640 1 -- 172.21.15.27:0/3604000866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee015a970 con 0x7f7ee015df00 2024-03-06T07:30:49.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.589+0000 7f7edffff640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 0x7f7ee015e2e0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00c850 tx=0x7f7ecc00cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.590+0000 7f7edd7fa640 1 -- 172.21.15.27:0/3604000866 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc015070 con 0x7f7ee015df00 2024-03-06T07:30:49.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.590+0000 7f7edd7fa640 1 -- 172.21.15.27:0/3604000866 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc004020 con 0x7f7ee015df00 2024-03-06T07:30:49.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.590+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee00d4090 con 0x7f7ee015df00 2024-03-06T07:30:49.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.590+0000 7f7edd7fa640 1 -- 172.21.15.27:0/3604000866 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc0043f0 con 0x7f7ee015df00 2024-03-06T07:30:49.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.590+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee00c93b0 con 0x7f7ee015df00 2024-03-06T07:30:49.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.591+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee0067ff0 con 0x7f7ee015df00 2024-03-06T07:30:49.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.591+0000 7f7edd7fa640 1 -- 172.21.15.27:0/3604000866 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ecc007830 con 0x7f7ee015df00 2024-03-06T07:30:49.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.592+0000 7f7edd7fa640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ec406b4d0 0x7f7ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.592+0000 7f7edd7fa640 1 -- 172.21.15.27:0/3604000866 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7ecc08a0c0 con 0x7f7ee015df00 2024-03-06T07:30:49.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.592+0000 7f7ee4d52640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ec406b4d0 0x7f7ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.593+0000 7f7ee4d52640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ec406b4d0 0x7f7ec406d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4036000 tx=0x7f7ed403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.596+0000 7f7edd7fa640 1 -- 172.21.15.27:0/3604000866 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ecc058830 con 0x7f7ee015df00 2024-03-06T07:30:49.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.815+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ee006b780 con 0x7f7ee015df00 2024-03-06T07:30:49.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.816+0000 7f7edd7fa640 1 -- 172.21.15.27:0/3604000866 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f7ecc05c8d0 con 0x7f7ee015df00 2024-03-06T07:30:49.818 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:49.818 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:49.466385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:49.819 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 100 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ec406b4d0 msgr2=0x7f7ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ec406b4d0 0x7f7ec406d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4036000 tx=0x7f7ed403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 msgr2=0x7f7ee015e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 0x7f7ee015e2e0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00c850 tx=0x7f7ecc00cd10 comp rx=0 tx=0).stop 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 shutdown_connections 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ec406b4d0 0x7f7ec406d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ee015abc0 0x7f7ee015b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015df00 0x7f7ee015e2e0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 --2- 172.21.15.27:0/3604000866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ee00b95a0 0x7f7ee015a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 >> 172.21.15.27:0/3604000866 conn(0x7f7ee0071960 msgr2=0x7f7ee00d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 shutdown_connections 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:49.818+0000 7f7ee6fb6640 1 -- 172.21.15.27:0/3604000866 wait complete. 2024-03-06T07:30:50.032 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:49.466385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:30:50.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:30:50.793 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:51.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:51.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.139+0000 7fedc2fd6640 1 Processor -- start 2024-03-06T07:30:51.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.139+0000 7fedc2fd6640 1 -- start start 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.139+0000 7fedc2fd6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fedbc15df00 0x7fedbc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.139+0000 7fedc2fd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc0b9a30 0x7fedbc0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.139+0000 7fedc2fd6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc0cea30 con 0x7fedbc0b95a0 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.139+0000 7fedc2fd6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedbc0d2af0 con 0x7fedbc15df00 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.139+0000 7fedc2fd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0cf3c0 con 0x7fedbc0b9a30 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbbfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fedbc0b95a0 0x7fedbc0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39886/0 (socket says 172.21.15.27:39886) 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbbfff640 1 -- 172.21.15.27:0/3561079708 learned_addr learned my addr 172.21.15.27:0/3561079708 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedc1573640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc0b9a30 0x7fedbc0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedc0d72640 1 --2- 172.21.15.27:0/3561079708 >> v2:172.21.15.69:3300/0 conn(0x7fedbc15df00 0x7fedbc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113148828 0 0) 0x7fedbc0cea30 con 0x7fedbc0b95a0 2024-03-06T07:30:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedac003520 con 0x7fedbc0b95a0 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2612005643 0 0) 0x7fedac003520 con 0x7fedbc0b95a0 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc0b9a30 msgr2=0x7fedbc0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc0b9a30 0x7fedbc0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 >> v2:172.21.15.69:3300/0 conn(0x7fedbc15df00 msgr2=0x7fedbc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 --2- 172.21.15.27:0/3561079708 >> v2:172.21.15.69:3300/0 conn(0x7fedbc15df00 0x7fedbc0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc0cf0e0 con 0x7fedbc0b95a0 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7feda8001670 con 0x7fedbc0b95a0 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 >> v1:172.21.15.27:6789/0 conn(0x7fedbc0b95a0 legacy=0x7fedbc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0039b0 0x7fedac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedac004800 0x7fedac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedac005180 0x7fedac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0d2af0 con 0x7fedac0039b0 2024-03-06T07:30:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0cf3c0 con 0x7fedac005180 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac004600 con 0x7fedac004800 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedc0d72640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0039b0 0x7fedac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedbbfff640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedac004800 0x7fedac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.140+0000 7fedc1573640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedac005180 0x7fedac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.141+0000 7fedc0d72640 1 -- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedac004800 msgr2=0x7fedac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.141+0000 7fedc0d72640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedac004800 0x7fedac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.141+0000 7fedc0d72640 1 -- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedac005180 msgr2=0x7fedac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.141+0000 7fedc0d72640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedac005180 0x7fedac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.141+0000 7fedc0d72640 1 -- 172.21.15.27:0/3561079708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac003520 con 0x7fedac0039b0 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.141+0000 7fedc0d72640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0039b0 0x7fedac003dd0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009ac0 tx=0x7fedac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd21be85704bf0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0039070 con 0x7fedac0039b0 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb002c710 con 0x7fedac0039b0 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedbb7fe640 1 -- 172.21.15.27:0/3561079708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0009690 con 0x7fedac0039b0 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedc2fd6640 1 -- 172.21.15.27:0/3561079708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0039b0 msgr2=0x7fedac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedc2fd6640 1 --2- 172.21.15.27:0/3561079708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0039b0 0x7fedac003dd0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009ac0 tx=0x7fedac005830 comp rx=0 tx=0).stop 2024-03-06T07:30:51.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedc1573640 1 -- 172.21.15.27:0/3561079708 reap_dead start 2024-03-06T07:30:51.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedc2fd6640 1 -- 172.21.15.27:0/3561079708 shutdown_connections 2024-03-06T07:30:51.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedc2fd6640 1 -- 172.21.15.27:0/3561079708 >> 172.21.15.27:0/3561079708 conn(0x7fedbc071960 msgr2=0x7fedbc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedc2fd6640 1 -- 172.21.15.27:0/3561079708 shutdown_connections 2024-03-06T07:30:51.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.142+0000 7fedc2fd6640 1 -- 172.21.15.27:0/3561079708 wait complete. 2024-03-06T07:30:51.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.143+0000 7fedc2fd6640 1 Processor -- start 2024-03-06T07:30:51.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.143+0000 7fedc2fd6640 1 -- start start 2024-03-06T07:30:51.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.144+0000 7fedc2fd6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedac0039b0 0x7fedbc0cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.144+0000 7fedc2fd6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedbc150eb0 0x7fedbc1506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.144+0000 7fedc2fd6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc15ed40 0x7fedbc0c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.144+0000 7fedc2fd6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0cec70 con 0x7fedbc150eb0 2024-03-06T07:30:51.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.144+0000 7fedc2fd6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0ce750 con 0x7fedac0039b0 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.144+0000 7fedc2fd6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0cedf0 con 0x7fedbc15ed40 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.144+0000 7fedbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedbc150eb0 0x7fedbc1506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.144+0000 7fedbbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedbc150eb0 0x7fedbc1506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55314/0 (socket says 172.21.15.27:55314) 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.144+0000 7fedbbfff640 1 -- 172.21.15.27:0/224681114 learned_addr learned my addr 172.21.15.27:0/224681114 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedc1573640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc15ed40 0x7fedbc0c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedc0d72640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedac0039b0 0x7fedbc0cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedbbfff640 1 -- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc15ed40 msgr2=0x7fedbc0c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedbbfff640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc15ed40 0x7fedbc0c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedbbfff640 1 -- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedac0039b0 msgr2=0x7fedbc0cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedbbfff640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedac0039b0 0x7fedbc0cd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedbbfff640 1 -- 172.21.15.27:0/224681114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc0c8330 con 0x7fedbc150eb0 2024-03-06T07:30:51.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedc1573640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc15ed40 0x7fedbc0c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedbbfff640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedbc150eb0 0x7fedbc1506c0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7feda8007340 tx=0x7feda8007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedb97fa640 1 -- 172.21.15.27:0/224681114 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda800f9c0 con 0x7fedbc150eb0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedb97fa640 1 -- 172.21.15.27:0/224681114 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda8018070 con 0x7fedbc150eb0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedbc0c8610 con 0x7fedbc150eb0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedb97fa640 1 -- 172.21.15.27:0/224681114 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda800fda0 con 0x7fedbc150eb0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.145+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedbc0cca90 con 0x7fedbc150eb0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.146+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedbc067ff0 con 0x7fedbc150eb0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.147+0000 7fedb97fa640 1 -- 172.21.15.27:0/224681114 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feda801d460 con 0x7fedbc150eb0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.147+0000 7fedb97fa640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda006b4d0 0x7feda006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.147+0000 7fedb97fa640 1 -- 172.21.15.27:0/224681114 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7feda808a3d0 con 0x7fedbc150eb0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.147+0000 7fedc0d72640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda006b4d0 0x7feda006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.148+0000 7fedc0d72640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda006b4d0 0x7feda006d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fedb0036000 tx=0x7fedb0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.156+0000 7fedb97fa640 1 -- 172.21.15.27:0/224681114 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feda8059210 con 0x7fedbc150eb0 2024-03-06T07:30:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.375+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedbc161480 con 0x7fedbc150eb0 2024-03-06T07:30:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.376+0000 7fedb97fa640 1 -- 172.21.15.27:0/224681114 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3417 (secure 0 0 0) 0x7feda8058b10 con 0x7fedbc150eb0 2024-03-06T07:30:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:51.379 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:49.466385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:51.379 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 100 2024-03-06T07:30:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda006b4d0 msgr2=0x7feda006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda006b4d0 0x7feda006d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fedb0036000 tx=0x7fedb0030000 comp rx=0 tx=0).stop 2024-03-06T07:30:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedbc150eb0 msgr2=0x7fedbc1506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedbc150eb0 0x7fedbc1506c0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7feda8007340 tx=0x7feda8007800 comp rx=0 tx=0).stop 2024-03-06T07:30:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 shutdown_connections 2024-03-06T07:30:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda006b4d0 0x7feda006d990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedbc15ed40 0x7fedbc0c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedbc150eb0 0x7fedbc1506c0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 --2- 172.21.15.27:0/224681114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedac0039b0 0x7fedbc0cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 >> 172.21.15.27:0/224681114 conn(0x7fedbc071960 msgr2=0x7fedbc1618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 shutdown_connections 2024-03-06T07:30:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:51.378+0000 7fedc2fd6640 1 -- 172.21.15.27:0/224681114 wait complete. 2024-03-06T07:30:51.593 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:49.466385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:30:51.593 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:30:52.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:52.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.699+0000 7fbd5d402640 1 Processor -- start 2024-03-06T07:30:52.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.699+0000 7fbd5d402640 1 -- start start 2024-03-06T07:30:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.700+0000 7fbd5d402640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806ca90 0x7fbd581623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.700+0000 7fbd5d402640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd580cfa70 0x7fbd5806f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.700+0000 7fbd5d402640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580cff10 con 0x7fbd5806f8b0 2024-03-06T07:30:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.700+0000 7fbd5d402640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd581628f0 con 0x7fbd580cfa70 2024-03-06T07:30:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.700+0000 7fbd5d402640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580d01f0 con 0x7fbd5806ca90 2024-03-06T07:30:52.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.700+0000 7fbd567fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd580cfa70 0x7fbd5806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.700+0000 7fbd567fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd580cfa70 0x7fbd5806f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55318/0 (socket says 172.21.15.27:55318) 2024-03-06T07:30:52.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.700+0000 7fbd567fc640 1 -- 172.21.15.27:0/32468958 learned_addr learned my addr 172.21.15.27:0/32468958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:52.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.700+0000 7fbd56ffd640 1 --2- 172.21.15.27:0/32468958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806ca90 0x7fbd581623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.701+0000 7fbd567fc640 1 -- 172.21.15.27:0/32468958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806ca90 msgr2=0x7fbd581623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.701+0000 7fbd567fc640 1 --2- 172.21.15.27:0/32468958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806ca90 0x7fbd581623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.701+0000 7fbd567fc640 1 -- 172.21.15.27:0/32468958 >> v1:172.21.15.69:6789/0 conn(0x7fbd5806f8b0 legacy=0x7fbd5815d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.701+0000 7fbd567fc640 1 -- 172.21.15.27:0/32468958 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd580d04d0 con 0x7fbd580cfa70 2024-03-06T07:30:52.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.702+0000 7fbd567fc640 1 --2- 172.21.15.27:0/32468958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd580cfa70 0x7fbd5806f370 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009870 tx=0x7fbd4c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6395093e1065da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.702+0000 7fbd55ffb640 1 -- 172.21.15.27:0/32468958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c034070 con 0x7fbd580cfa70 2024-03-06T07:30:52.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.702+0000 7fbd55ffb640 1 -- 172.21.15.27:0/32468958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c030840 con 0x7fbd580cfa70 2024-03-06T07:30:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.702+0000 7fbd55ffb640 1 -- 172.21.15.27:0/32468958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c030c10 con 0x7fbd580cfa70 2024-03-06T07:30:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.703+0000 7fbd5d402640 1 -- 172.21.15.27:0/32468958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd580cfa70 msgr2=0x7fbd5806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.703+0000 7fbd5d402640 1 --2- 172.21.15.27:0/32468958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd580cfa70 0x7fbd5806f370 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009870 tx=0x7fbd4c02b980 comp rx=0 tx=0).stop 2024-03-06T07:30:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.703+0000 7fbd5d402640 1 -- 172.21.15.27:0/32468958 shutdown_connections 2024-03-06T07:30:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.703+0000 7fbd5d402640 1 --2- 172.21.15.27:0/32468958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd580cfa70 0x7fbd5806f370 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.703+0000 7fbd5d402640 1 --2- 172.21.15.27:0/32468958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806ca90 0x7fbd581623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.703+0000 7fbd5d402640 1 -- 172.21.15.27:0/32468958 >> 172.21.15.27:0/32468958 conn(0x7fbd580b3410 msgr2=0x7fbd580b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.703+0000 7fbd5d402640 1 -- 172.21.15.27:0/32468958 shutdown_connections 2024-03-06T07:30:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.703+0000 7fbd5d402640 1 -- 172.21.15.27:0/32468958 wait complete. 2024-03-06T07:30:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.704+0000 7fbd5d402640 1 Processor -- start 2024-03-06T07:30:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.704+0000 7fbd5d402640 1 -- start start 2024-03-06T07:30:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd5d402640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5806ca90 0x7fbd581591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd5d402640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806f8b0 0x7fbd580b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd5d402640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd580cfa70 0x7fbd5815f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd5d402640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5806ce70 con 0x7fbd5806ca90 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd5d402640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580d0a90 con 0x7fbd580cfa70 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd5d402640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580b4150 con 0x7fbd5806f8b0 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd56ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5806ca90 0x7fbd581591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd56ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5806ca90 0x7fbd581591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55330/0 (socket says 172.21.15.27:55330) 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd56ffd640 1 -- 172.21.15.27:0/1391693558 learned_addr learned my addr 172.21.15.27:0/1391693558 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd567fc640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806f8b0 0x7fbd580b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd577fe640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd580cfa70 0x7fbd5815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd56ffd640 1 -- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806f8b0 msgr2=0x7fbd580b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd56ffd640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806f8b0 0x7fbd580b6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd56ffd640 1 -- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd580cfa70 msgr2=0x7fbd5815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd56ffd640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd580cfa70 0x7fbd5815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd56ffd640 1 -- 172.21.15.27:0/1391693558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd581f7a70 con 0x7fbd5806ca90 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd567fc640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806f8b0 0x7fbd580b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.705+0000 7fbd56ffd640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5806ca90 0x7fbd581591a0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800e9b0 tx=0x7fbd4800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.706+0000 7fbd3f7fe640 1 -- 172.21.15.27:0/1391693558 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd48017070 con 0x7fbd5806ca90 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.706+0000 7fbd3f7fe640 1 -- 172.21.15.27:0/1391693558 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4800c9a0 con 0x7fbd5806ca90 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.706+0000 7fbd3f7fe640 1 -- 172.21.15.27:0/1391693558 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4800cd90 con 0x7fbd5806ca90 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.706+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd581f7d50 con 0x7fbd5806ca90 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.706+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd581f8310 con 0x7fbd5806ca90 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.707+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd580cf130 con 0x7fbd5806ca90 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.708+0000 7fbd3f7fe640 1 -- 172.21.15.27:0/1391693558 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd4801a070 con 0x7fbd5806ca90 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.708+0000 7fbd3f7fe640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbd4406b4d0 0x7fbd4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.708+0000 7fbd3f7fe640 1 -- 172.21.15.27:0/1391693558 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbd4808a1f0 con 0x7fbd5806ca90 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.708+0000 7fbd567fc640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbd4406b4d0 0x7fbd4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.709+0000 7fbd567fc640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbd4406b4d0 0x7fbd4406d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c036000 tx=0x7fbd4c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.713+0000 7fbd3f7fe640 1 -- 172.21.15.27:0/1391693558 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd580cf130 con 0x7fbd5806ca90 2024-03-06T07:30:52.795 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.934+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd5804f330 con 0x7fbd5806ca90 2024-03-06T07:30:52.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.935+0000 7fbd3f7fe640 1 -- 172.21.15.27:0/1391693558 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3417 (secure 0 0 0) 0x7fbd480590c0 con 0x7fbd5806ca90 2024-03-06T07:30:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:52.937 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:49.466385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:52.938 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 100 2024-03-06T07:30:52.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbd4406b4d0 msgr2=0x7fbd4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbd4406b4d0 0x7fbd4406d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c036000 tx=0x7fbd4c03b040 comp rx=0 tx=0).stop 2024-03-06T07:30:52.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5806ca90 msgr2=0x7fbd581591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5806ca90 0x7fbd581591a0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800e9b0 tx=0x7fbd4800ee70 comp rx=0 tx=0).stop 2024-03-06T07:30:52.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 shutdown_connections 2024-03-06T07:30:52.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbd4406b4d0 0x7fbd4406d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd580cfa70 0x7fbd5815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbd5806f8b0 0x7fbd580b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 --2- 172.21.15.27:0/1391693558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5806ca90 0x7fbd581591a0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 >> 172.21.15.27:0/1391693558 conn(0x7fbd580b3410 msgr2=0x7fbd580d0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 shutdown_connections 2024-03-06T07:30:52.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:52.937+0000 7fbd5d402640 1 -- 172.21.15.27:0/1391693558 wait complete. 2024-03-06T07:30:53.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:49.466385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:30:53.151 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:30:54.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.258+0000 7f89bde2a640 1 Processor -- start 2024-03-06T07:30:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.258+0000 7f89bde2a640 1 -- start start 2024-03-06T07:30:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89bde2a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89b80b9a10 0x7f89b80d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89bde2a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b80cea30 con 0x7f89b80b95a0 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89bde2a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b80cf3c0 con 0x7f89b815df00 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89bde2a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b80cf0e0 con 0x7f89b80b9a10 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89b7fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89b80b9a10 0x7f89b80d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89b6ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f89b80b95a0 0x7f89b80b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53274/0 (socket says 172.21.15.27:53274) 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89b6ffd640 1 -- 172.21.15.27:0/1636944573 learned_addr learned my addr 172.21.15.27:0/1636944573 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89b7fff640 1 -- 172.21.15.27:0/1636944573 >> v1:172.21.15.69:6789/0 conn(0x7f89b815df00 legacy=0x7f89b80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89b7fff640 1 -- 172.21.15.27:0/1636944573 >> v1:172.21.15.27:6789/0 conn(0x7f89b80b95a0 legacy=0x7f89b80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89b7fff640 1 -- 172.21.15.27:0/1636944573 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b80ce470 con 0x7f89b80b9a10 2024-03-06T07:30:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.259+0000 7f89b67fc640 1 -- 172.21.15.27:0/1636944573 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225752191 0 0) 0x7f89b80cea30 con 0x7f89b80b95a0 2024-03-06T07:30:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.260+0000 7f89b7fff640 1 --2- 172.21.15.27:0/1636944573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 0x7f89b80d3220 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f89b80cf590 tx=0x7f89b002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbe2871ddaddc57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.260+0000 7f89b67fc640 1 -- 172.21.15.27:0/1636944573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0039070 con 0x7f89b80b9a10 2024-03-06T07:30:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.260+0000 7f89b67fc640 1 -- 172.21.15.27:0/1636944573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b002cce0 con 0x7f89b80b9a10 2024-03-06T07:30:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.260+0000 7f89b67fc640 1 -- 172.21.15.27:0/1636944573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0034660 con 0x7f89b80b9a10 2024-03-06T07:30:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.261+0000 7f89bde2a640 1 -- 172.21.15.27:0/1636944573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 msgr2=0x7f89b80d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.261+0000 7f89bde2a640 1 --2- 172.21.15.27:0/1636944573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 0x7f89b80d3220 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f89b80cf590 tx=0x7f89b002bf30 comp rx=0 tx=0).stop 2024-03-06T07:30:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.261+0000 7f89bde2a640 1 -- 172.21.15.27:0/1636944573 shutdown_connections 2024-03-06T07:30:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.261+0000 7f89bde2a640 1 --2- 172.21.15.27:0/1636944573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 0x7f89b80d3220 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.261+0000 7f89bde2a640 1 -- 172.21.15.27:0/1636944573 >> 172.21.15.27:0/1636944573 conn(0x7f89b8071960 msgr2=0x7f89b8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.261+0000 7f89bde2a640 1 -- 172.21.15.27:0/1636944573 shutdown_connections 2024-03-06T07:30:54.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.261+0000 7f89bde2a640 1 -- 172.21.15.27:0/1636944573 wait complete. 2024-03-06T07:30:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.262+0000 7f89bde2a640 1 Processor -- start 2024-03-06T07:30:54.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.262+0000 7f89bde2a640 1 -- start start 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89bde2a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b80b95a0 0x7f89b814da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89bde2a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 0x7f89b8160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89bde2a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b815df00 0x7f89b80b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89bde2a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80cedf0 con 0x7f89b80b9a10 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89bde2a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80cec70 con 0x7f89b80b95a0 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89bde2a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80b5e10 con 0x7f89b815df00 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 0x7f89b8160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 0x7f89b8160bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56906/0 (socket says 172.21.15.27:56906) 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b6ffd640 1 -- 172.21.15.27:0/2641963496 learned_addr learned my addr 172.21.15.27:0/2641963496 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b77fe640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b80b95a0 0x7f89b814da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b7fff640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b815df00 0x7f89b80b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b6ffd640 1 -- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b815df00 msgr2=0x7f89b80b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b6ffd640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b815df00 0x7f89b80b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b6ffd640 1 -- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b80b95a0 msgr2=0x7f89b814da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b6ffd640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b80b95a0 0x7f89b814da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b6ffd640 1 -- 172.21.15.27:0/2641963496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b81f3560 con 0x7f89b80b9a10 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.263+0000 7f89b77fe640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b80b95a0 0x7f89b814da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.264+0000 7f89b6ffd640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 0x7f89b8160bc0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f89a800c4a0 tx=0x7f89a800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.264+0000 7f89affff640 1 -- 172.21.15.27:0/2641963496 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8018070 con 0x7f89b80b9a10 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.264+0000 7f89affff640 1 -- 172.21.15.27:0/2641963496 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a80074e0 con 0x7f89b80b9a10 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.264+0000 7f89affff640 1 -- 172.21.15.27:0/2641963496 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8004a20 con 0x7f89b80b9a10 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.264+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b814e450 con 0x7f89b80b9a10 2024-03-06T07:30:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.264+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b80cb310 con 0x7f89b80b9a10 2024-03-06T07:30:54.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.265+0000 7f89affff640 1 -- 172.21.15.27:0/2641963496 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89a801b070 con 0x7f89b80b9a10 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.265+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b8067ff0 con 0x7f89b80b9a10 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.266+0000 7f89affff640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f899806b4d0 0x7f899806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.266+0000 7f89affff640 1 -- 172.21.15.27:0/2641963496 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f89a8089460 con 0x7f89b80b9a10 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.266+0000 7f89b77fe640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f899806b4d0 0x7f899806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.266+0000 7f89b77fe640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f899806b4d0 0x7f899806d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f89a0007fd0 tx=0x7f89a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.270+0000 7f89affff640 1 -- 172.21.15.27:0/2641963496 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89b814e450 con 0x7f89b80b9a10 2024-03-06T07:30:54.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.528+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89b806b780 con 0x7f89b80b9a10 2024-03-06T07:30:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.529+0000 7f89affff640 1 -- 172.21.15.27:0/2641963496 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f89a8058330 con 0x7f89b80b9a10 2024-03-06T07:30:54.531 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:54.531 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:49.466385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 100 2024-03-06T07:30:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.530+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f899806b4d0 msgr2=0x7f899806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.530+0000 7f89bde2a640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f899806b4d0 0x7f899806d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f89a0007fd0 tx=0x7f89a0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.530+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 msgr2=0x7f89b8160bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.530+0000 7f89bde2a640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 0x7f89b8160bc0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f89a800c4a0 tx=0x7f89a800c960 comp rx=0 tx=0).stop 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.531+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 shutdown_connections 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.531+0000 7f89bde2a640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f899806b4d0 0x7f899806d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.531+0000 7f89bde2a640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89b815df00 0x7f89b80b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.531+0000 7f89bde2a640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b80b9a10 0x7f89b8160bc0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.531+0000 7f89bde2a640 1 --2- 172.21.15.27:0/2641963496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b80b95a0 0x7f89b814da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.531+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 >> 172.21.15.27:0/2641963496 conn(0x7f89b8071960 msgr2=0x7f89b815b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.531+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 shutdown_connections 2024-03-06T07:30:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:54.531+0000 7f89bde2a640 1 -- 172.21.15.27:0/2641963496 wait complete. 2024-03-06T07:30:54.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:49.466385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:30:54.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:30:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:55.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:55.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.850+0000 7f3ecf655640 1 Processor -- start 2024-03-06T07:30:55.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.851+0000 7f3ecf655640 1 -- start start 2024-03-06T07:30:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.851+0000 7f3ecf655640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec80b9a10 0x7f3ec80d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.851+0000 7f3ecf655640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec80cea30 con 0x7f3ec80b95a0 2024-03-06T07:30:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.851+0000 7f3ecf655640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec80cf3c0 con 0x7f3ec815df00 2024-03-06T07:30:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ecf655640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80cf0e0 con 0x7f3ec80b9a10 2024-03-06T07:30:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3eccbf0640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3ec80b95a0 0x7f3ec80b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53278/0 (socket says 172.21.15.27:53278) 2024-03-06T07:30:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3eccbf0640 1 -- 172.21.15.27:0/2567128338 learned_addr learned my addr 172.21.15.27:0/2567128338 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:30:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ecdbf2640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec80b9a10 0x7f3ec80d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184643190 0 0) 0x7f3ec80cea30 con 0x7f3ec80b95a0 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb4003520 con 0x7f3ec80b95a0 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1151788435 0 0) 0x7f3eb4003520 con 0x7f3ec80b95a0 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec80b9a10 msgr2=0x7f3ec80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec80b9a10 0x7f3ec80d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 >> v1:172.21.15.69:6789/0 conn(0x7f3ec815df00 legacy=0x7f3ec80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec80ce470 con 0x7f3ec80b95a0 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f3eb8001670 con 0x7f3ec80b95a0 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 >> v1:172.21.15.27:6789/0 conn(0x7f3ec80b95a0 legacy=0x7f3ec80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3eb40039e0 0x7f3eb4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb4004800 0x7f3eb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec80b9a10 0x7f3eb40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80cf0e0 con 0x7f3eb4004800 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb400ba70 con 0x7f3eb40039e0 2024-03-06T07:30:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.852+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb400bc70 con 0x7f3ec80b9a10 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.853+0000 7f3eccbf0640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb4004800 0x7f3eb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.853+0000 7f3ecd3f1640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3eb40039e0 0x7f3eb4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.853+0000 7f3ecdbf2640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec80b9a10 0x7f3eb40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.853+0000 7f3eccbf0640 1 -- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec80b9a10 msgr2=0x7f3eb40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.853+0000 7f3eccbf0640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec80b9a10 0x7f3eb40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.853+0000 7f3eccbf0640 1 -- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3eb40039e0 msgr2=0x7f3eb4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.853+0000 7f3eccbf0640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3eb40039e0 0x7f3eb4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.853+0000 7f3eccbf0640 1 -- 172.21.15.27:0/2567128338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb4003520 con 0x7f3eb4004800 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3eccbf0640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb4004800 0x7f3eb4004c80 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8001eb0 tx=0x7f3eb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7085e0568880d82b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8007d30 con 0x7f3eb4004800 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb8030070 con 0x7f3eb4004800 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ebffff640 1 -- 172.21.15.27:0/2567128338 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8005d30 con 0x7f3eb4004800 2024-03-06T07:30:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ecf655640 1 -- 172.21.15.27:0/2567128338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb4004800 msgr2=0x7f3eb4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ecf655640 1 --2- 172.21.15.27:0/2567128338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb4004800 0x7f3eb4004c80 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8001eb0 tx=0x7f3eb4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ecdbf2640 1 -- 172.21.15.27:0/2567128338 reap_dead start 2024-03-06T07:30:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ecf655640 1 -- 172.21.15.27:0/2567128338 shutdown_connections 2024-03-06T07:30:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ecf655640 1 -- 172.21.15.27:0/2567128338 >> 172.21.15.27:0/2567128338 conn(0x7f3ec8071960 msgr2=0x7f3ec8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ecf655640 1 -- 172.21.15.27:0/2567128338 shutdown_connections 2024-03-06T07:30:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.854+0000 7f3ecf655640 1 -- 172.21.15.27:0/2567128338 wait complete. 2024-03-06T07:30:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.856+0000 7f3ecf655640 1 Processor -- start 2024-03-06T07:30:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.856+0000 7f3ecf655640 1 -- start start 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecf655640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3eb4004800 0x7f3ec815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecf655640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec8158480 0x7f3ec80d3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecf655640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ec8158900 0x7f3ec81606c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecf655640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80cedf0 con 0x7f3ec8158900 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecf655640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80cec70 con 0x7f3eb4004800 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecf655640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80b5e10 con 0x7f3ec8158480 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecdbf2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ec8158900 0x7f3ec81606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecdbf2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ec8158900 0x7f3ec81606c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56926/0 (socket says 172.21.15.27:56926) 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecdbf2640 1 -- 172.21.15.27:0/1848564682 learned_addr learned my addr 172.21.15.27:0/1848564682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecd3f1640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3eb4004800 0x7f3ec815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3eccbf0640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec8158480 0x7f3ec80d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecdbf2640 1 -- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec8158480 msgr2=0x7f3ec80d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecdbf2640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec8158480 0x7f3ec80d3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecdbf2640 1 -- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3eb4004800 msgr2=0x7f3ec815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecdbf2640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3eb4004800 0x7f3ec815e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecdbf2640 1 -- 172.21.15.27:0/1848564682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec80c7970 con 0x7f3ec8158900 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.858+0000 7f3ecdbf2640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ec8158900 0x7f3ec81606c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80d0f40 tx=0x7f3ec000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.859+0000 7f3ebdffb640 1 -- 172.21.15.27:0/1848564682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0015430 con 0x7f3ec8158900 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.859+0000 7f3ebdffb640 1 -- 172.21.15.27:0/1848564682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec00155f0 con 0x7f3ec8158900 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.859+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec80c7c50 con 0x7f3ec8158900 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.859+0000 7f3ebdffb640 1 -- 172.21.15.27:0/1848564682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00159e0 con 0x7f3ec8158900 2024-03-06T07:30:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.859+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec80c8210 con 0x7f3ec8158900 2024-03-06T07:30:55.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.860+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec80b8fb0 con 0x7f3ec8158900 2024-03-06T07:30:55.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.860+0000 7f3ebdffb640 1 -- 172.21.15.27:0/1848564682 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ec0015b80 con 0x7f3ec8158900 2024-03-06T07:30:55.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.861+0000 7f3ebdffb640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3eac06b4d0 0x7f3eac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.861+0000 7f3ebdffb640 1 -- 172.21.15.27:0/1848564682 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3ec008b0c0 con 0x7f3ec8158900 2024-03-06T07:30:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.861+0000 7f3ecd3f1640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3eac06b4d0 0x7f3eac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.862+0000 7f3ecd3f1640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3eac06b4d0 0x7f3eac06d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005590 tx=0x7f3ec4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:55.865+0000 7f3ebdffb640 1 -- 172.21.15.27:0/1848564682 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ec80b8fb0 con 0x7f3ec8158900 2024-03-06T07:30:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.087+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ec80b2710 con 0x7f3ec8158900 2024-03-06T07:30:56.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.088+0000 7f3ebdffb640 1 -- 172.21.15.27:0/1848564682 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f3ec0019030 con 0x7f3ec8158900 2024-03-06T07:30:56.090 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:56.090 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3eac06b4d0 msgr2=0x7f3eac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3eac06b4d0 0x7f3eac06d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005590 tx=0x7f3ec4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ec8158900 msgr2=0x7f3ec81606c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ec8158900 0x7f3ec81606c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80d0f40 tx=0x7f3ec000f580 comp rx=0 tx=0).stop 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 shutdown_connections 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3eac06b4d0 0x7f3eac06d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ec8158900 0x7f3ec81606c0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3ec8158480 0x7f3ec80d3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 --2- 172.21.15.27:0/1848564682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3eb4004800 0x7f3ec815e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 >> 172.21.15.27:0/1848564682 conn(0x7f3ec8071960 msgr2=0x7f3ec80b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 shutdown_connections 2024-03-06T07:30:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:56.090+0000 7f3ecf655640 1 -- 172.21.15.27:0/1848564682 wait complete. 2024-03-06T07:30:56.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:30:56.305 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:30:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:57.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:57.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.410+0000 7f87c615c640 1 Processor -- start 2024-03-06T07:30:57.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.410+0000 7f87c615c640 1 -- start start 2024-03-06T07:30:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.411+0000 7f87c615c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c0071540 0x7f87c00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.411+0000 7f87c615c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f87c00d3400 0x7f87c0071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.411+0000 7f87c615c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87c00b2b50 0x7f87c00d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.411+0000 7f87c615c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c00d2f70 con 0x7f87c00b2b50 2024-03-06T07:30:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.411+0000 7f87c615c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c00d4be0 con 0x7f87c00d3400 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.411+0000 7f87c615c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c0071e60 con 0x7f87c0071540 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.411+0000 7f87bffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87c00b2b50 0x7f87c00d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.411+0000 7f87bffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87c00b2b50 0x7f87c00d1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56930/0 (socket says 172.21.15.27:56930) 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.411+0000 7f87bffff640 1 -- 172.21.15.27:0/2818445231 learned_addr learned my addr 172.21.15.27:0/2818445231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.412+0000 7f87bffff640 1 -- 172.21.15.27:0/2818445231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c0071540 msgr2=0x7f87c00b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.412+0000 7f87bf7fe640 1 --2- 172.21.15.27:0/2818445231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c0071540 0x7f87c00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.412+0000 7f87beffd640 1 --2- 172.21.15.27:0/2818445231 >> v2:172.21.15.69:3300/0 conn(0x7f87c00d3400 0x7f87c0071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.412+0000 7f87bffff640 1 --2- 172.21.15.27:0/2818445231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c0071540 0x7f87c00b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.412+0000 7f87bffff640 1 -- 172.21.15.27:0/2818445231 >> v2:172.21.15.69:3300/0 conn(0x7f87c00d3400 msgr2=0x7f87c0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.412+0000 7f87bffff640 1 --2- 172.21.15.27:0/2818445231 >> v2:172.21.15.69:3300/0 conn(0x7f87c00d3400 0x7f87c0071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.412+0000 7f87bffff640 1 -- 172.21.15.27:0/2818445231 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c00ce610 con 0x7f87c00b2b50 2024-03-06T07:30:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87bffff640 1 --2- 172.21.15.27:0/2818445231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 0x7f87c00d1870 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f87c008c810 tx=0x7f87b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9730376560823b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87be7fc640 1 -- 172.21.15.27:0/2818445231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b802ca70 con 0x7f87c00b2b50 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87be7fc640 1 -- 172.21.15.27:0/2818445231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b802cc10 con 0x7f87c00b2b50 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87be7fc640 1 -- 172.21.15.27:0/2818445231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b80343f0 con 0x7f87c00b2b50 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87c615c640 1 -- 172.21.15.27:0/2818445231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 msgr2=0x7f87c00d1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87c615c640 1 --2- 172.21.15.27:0/2818445231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 0x7f87c00d1870 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f87c008c810 tx=0x7f87b802bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87c615c640 1 -- 172.21.15.27:0/2818445231 shutdown_connections 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87c615c640 1 --2- 172.21.15.27:0/2818445231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 0x7f87c00d1870 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87c615c640 1 --2- 172.21.15.27:0/2818445231 >> v2:172.21.15.69:3300/0 conn(0x7f87c00d3400 0x7f87c0071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87c615c640 1 --2- 172.21.15.27:0/2818445231 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c0071540 0x7f87c00b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87c615c640 1 -- 172.21.15.27:0/2818445231 >> 172.21.15.27:0/2818445231 conn(0x7f87c00d3ca0 msgr2=0x7f87c00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87c615c640 1 -- 172.21.15.27:0/2818445231 shutdown_connections 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.413+0000 7f87c615c640 1 -- 172.21.15.27:0/2818445231 wait complete. 2024-03-06T07:30:57.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.414+0000 7f87c615c640 1 Processor -- start 2024-03-06T07:30:57.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.415+0000 7f87c615c640 1 -- start start 2024-03-06T07:30:57.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.415+0000 7f87c615c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87c0071540 0x7f87c00b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.415+0000 7f87c615c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 0x7f87c01f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.415+0000 7f87c615c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c00d3400 0x7f87c00cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.415+0000 7f87c615c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00ce8f0 con 0x7f87c00b2b50 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.415+0000 7f87c615c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00d2c00 con 0x7f87c0071540 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.415+0000 7f87c615c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c015e370 con 0x7f87c00d3400 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.415+0000 7f87beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 0x7f87c01f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87beffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 0x7f87c01f0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56934/0 (socket says 172.21.15.27:56934) 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87beffd640 1 -- 172.21.15.27:0/3943614422 learned_addr learned my addr 172.21.15.27:0/3943614422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87bf7fe640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87c0071540 0x7f87c00b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87bffff640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c00d3400 0x7f87c00cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87beffd640 1 -- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c00d3400 msgr2=0x7f87c00cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87beffd640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c00d3400 0x7f87c00cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87beffd640 1 -- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87c0071540 msgr2=0x7f87c00b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87beffd640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87c0071540 0x7f87c00b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87beffd640 1 -- 172.21.15.27:0/3943614422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c00b5c80 con 0x7f87c00b2b50 2024-03-06T07:30:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87bf7fe640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87c0071540 0x7f87c00b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87beffd640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 0x7f87c01f0ae0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c7f0 tx=0x7f87b400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87a7fff640 1 -- 172.21.15.27:0/3943614422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4004230 con 0x7f87c00b2b50 2024-03-06T07:30:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87a7fff640 1 -- 172.21.15.27:0/3943614422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b40043d0 con 0x7f87c00b2b50 2024-03-06T07:30:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c014c350 con 0x7f87c00b2b50 2024-03-06T07:30:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87a7fff640 1 -- 172.21.15.27:0/3943614422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4007860 con 0x7f87c00b2b50 2024-03-06T07:30:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.416+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c014c910 con 0x7f87c00b2b50 2024-03-06T07:30:57.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.417+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c0068700 con 0x7f87c00b2b50 2024-03-06T07:30:57.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.418+0000 7f87a7fff640 1 -- 172.21.15.27:0/3943614422 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87b4007c10 con 0x7f87c00b2b50 2024-03-06T07:30:57.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.419+0000 7f87a7fff640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f87ac06b4d0 0x7f87ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.419+0000 7f87a7fff640 1 -- 172.21.15.27:0/3943614422 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f87b408a430 con 0x7f87c00b2b50 2024-03-06T07:30:57.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.419+0000 7f87bf7fe640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f87ac06b4d0 0x7f87ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.420+0000 7f87bf7fe640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f87ac06b4d0 0x7f87ac06d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f87b000aa50 tx=0x7f87b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.423+0000 7f87a7fff640 1 -- 172.21.15.27:0/3943614422 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87b4015020 con 0x7f87c00b2b50 2024-03-06T07:30:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.743+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87c00b9a60 con 0x7f87c00b2b50 2024-03-06T07:30:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.743+0000 7f87a7fff640 1 -- 172.21.15.27:0/3943614422 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f87b4059270 con 0x7f87c00b2b50 2024-03-06T07:30:57.746 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:57.746 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:30:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f87ac06b4d0 msgr2=0x7f87ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f87ac06b4d0 0x7f87ac06d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f87b000aa50 tx=0x7f87b0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 msgr2=0x7f87c01f0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 0x7f87c01f0ae0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c7f0 tx=0x7f87b400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 shutdown_connections 2024-03-06T07:30:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f87ac06b4d0 0x7f87ac06d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f87c00d3400 0x7f87c00cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c00b2b50 0x7f87c01f0ae0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 --2- 172.21.15.27:0/3943614422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87c0071540 0x7f87c00b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.745+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 >> 172.21.15.27:0/3943614422 conn(0x7f87c00d3ca0 msgr2=0x7f87c00ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.746+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 shutdown_connections 2024-03-06T07:30:57.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:57.746+0000 7f87c615c640 1 -- 172.21.15.27:0/3943614422 wait complete. 2024-03-06T07:30:57.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:30:57.960 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:30:58.801 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:58.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:59.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.066+0000 7f195175c640 1 Processor -- start 2024-03-06T07:30:59.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.066+0000 7f195175c640 1 -- start start 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.066+0000 7f195175c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f194c0aee10 0x7f194c15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.066+0000 7f195175c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f194c06d400 0x7f194c070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.066+0000 7f195175c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194c0729b0 con 0x7f194c15bac0 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.066+0000 7f195175c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194c06cf70 con 0x7f194c0aee10 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.066+0000 7f195175c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194c06c920 con 0x7f194c06d400 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.067+0000 7f194a7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f194c0aee10 0x7f194c15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.067+0000 7f194a7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f194c0aee10 0x7f194c15cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56942/0 (socket says 172.21.15.27:56942) 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.067+0000 7f194a7fc640 1 -- 172.21.15.27:0/759083351 learned_addr learned my addr 172.21.15.27:0/759083351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.067+0000 7f194b7fe640 1 --2- 172.21.15.27:0/759083351 >> v2:172.21.15.69:3300/0 conn(0x7f194c06d400 0x7f194c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.067+0000 7f194a7fc640 1 -- 172.21.15.27:0/759083351 >> v2:172.21.15.69:3300/0 conn(0x7f194c06d400 msgr2=0x7f194c070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.067+0000 7f194a7fc640 1 --2- 172.21.15.27:0/759083351 >> v2:172.21.15.69:3300/0 conn(0x7f194c06d400 0x7f194c070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.067+0000 7f194a7fc640 1 -- 172.21.15.27:0/759083351 >> v1:172.21.15.69:6789/0 conn(0x7f194c15bac0 legacy=0x7f194c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.067+0000 7f194a7fc640 1 -- 172.21.15.27:0/759083351 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c06cc00 con 0x7f194c0aee10 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.067+0000 7f194b7fe640 1 --2- 172.21.15.27:0/759083351 >> v2:172.21.15.69:3300/0 conn(0x7f194c06d400 0x7f194c070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.068+0000 7f194a7fc640 1 --2- 172.21.15.27:0/759083351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 0x7f194c15cbe0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f194c06ea40 tx=0x7f194002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f865bac9dcd5c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.068+0000 7f1949ffb640 1 -- 172.21.15.27:0/759083351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940035020 con 0x7f194c0aee10 2024-03-06T07:30:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.068+0000 7f1949ffb640 1 -- 172.21.15.27:0/759083351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1940031870 con 0x7f194c0aee10 2024-03-06T07:30:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.068+0000 7f1949ffb640 1 -- 172.21.15.27:0/759083351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940031c40 con 0x7f194c0aee10 2024-03-06T07:30:59.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.069+0000 7f195175c640 1 -- 172.21.15.27:0/759083351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 msgr2=0x7f194c15cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.069+0000 7f195175c640 1 --2- 172.21.15.27:0/759083351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 0x7f194c15cbe0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f194c06ea40 tx=0x7f194002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:59.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.069+0000 7f195175c640 1 -- 172.21.15.27:0/759083351 shutdown_connections 2024-03-06T07:30:59.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.069+0000 7f195175c640 1 --2- 172.21.15.27:0/759083351 >> v2:172.21.15.69:3300/0 conn(0x7f194c06d400 0x7f194c070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.069+0000 7f195175c640 1 --2- 172.21.15.27:0/759083351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 0x7f194c15cbe0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.069+0000 7f195175c640 1 -- 172.21.15.27:0/759083351 >> 172.21.15.27:0/759083351 conn(0x7f194c066d00 msgr2=0x7f194c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.069+0000 7f195175c640 1 -- 172.21.15.27:0/759083351 shutdown_connections 2024-03-06T07:30:59.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.069+0000 7f195175c640 1 -- 172.21.15.27:0/759083351 wait complete. 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.070+0000 7f195175c640 1 Processor -- start 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.070+0000 7f195175c640 1 -- start start 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f195175c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f194c06d400 0x7f194c144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f195175c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 0x7f194c0aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f195175c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f194c15bac0 0x7f194c15e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f195175c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c15c020 con 0x7f194c0aee10 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f195175c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c15bea0 con 0x7f194c15bac0 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f195175c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c072580 con 0x7f194c06d400 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 0x7f194c0aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 0x7f194c0aaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56948/0 (socket says 172.21.15.27:56948) 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194a7fc640 1 -- 172.21.15.27:0/3160255330 learned_addr learned my addr 172.21.15.27:0/3160255330 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194b7fe640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f194c15bac0 0x7f194c15e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194affd640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f194c06d400 0x7f194c144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194a7fc640 1 -- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f194c06d400 msgr2=0x7f194c144ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194a7fc640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f194c06d400 0x7f194c144ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194a7fc640 1 -- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f194c15bac0 msgr2=0x7f194c15e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194a7fc640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f194c15bac0 0x7f194c15e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194a7fc640 1 -- 172.21.15.27:0/3160255330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c1e54e0 con 0x7f194c0aee10 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.071+0000 7f194b7fe640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f194c15bac0 0x7f194c15e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.072+0000 7f194a7fc640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 0x7f194c0aaa30 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f1940000c00 tx=0x7f19400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.072+0000 7f19337fe640 1 -- 172.21.15.27:0/3160255330 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940040070 con 0x7f194c0aee10 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.072+0000 7f19337fe640 1 -- 172.21.15.27:0/3160255330 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194003a420 con 0x7f194c0aee10 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.072+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194c1e57c0 con 0x7f194c0aee10 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.072+0000 7f19337fe640 1 -- 172.21.15.27:0/3160255330 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940035040 con 0x7f194c0aee10 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.072+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194c1e5d80 con 0x7f194c0aee10 2024-03-06T07:30:59.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.073+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194c067ff0 con 0x7f194c0aee10 2024-03-06T07:30:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.073+0000 7f19337fe640 1 -- 172.21.15.27:0/3160255330 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1940038020 con 0x7f194c0aee10 2024-03-06T07:30:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.074+0000 7f19337fe640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f193806b4d0 0x7f193806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.074+0000 7f19337fe640 1 -- 172.21.15.27:0/3160255330 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f19400ac2a0 con 0x7f194c0aee10 2024-03-06T07:30:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.074+0000 7f194affd640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f193806b4d0 0x7f193806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.075+0000 7f194affd640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f193806b4d0 0x7f193806d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f193c006fd0 tx=0x7f193c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.078+0000 7f19337fe640 1 -- 172.21.15.27:0/3160255330 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f194007b0e0 con 0x7f194c0aee10 2024-03-06T07:30:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.296+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f194c06de50 con 0x7f194c0aee10 2024-03-06T07:30:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.297+0000 7f19337fe640 1 -- 172.21.15.27:0/3160255330 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f19400ac0c0 con 0x7f194c0aee10 2024-03-06T07:30:59.299 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:30:59.299 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:30:59.300 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f193806b4d0 msgr2=0x7f193806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f193806b4d0 0x7f193806d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f193c006fd0 tx=0x7f193c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 msgr2=0x7f194c0aaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 0x7f194c0aaa30 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f1940000c00 tx=0x7f19400046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 shutdown_connections 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f193806b4d0 0x7f193806d990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f194c15bac0 0x7f194c15e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194c0aee10 0x7f194c0aaa30 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 --2- 172.21.15.27:0/3160255330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f194c06d400 0x7f194c144ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 >> 172.21.15.27:0/3160255330 conn(0x7f194c066d00 msgr2=0x7f194c15cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 shutdown_connections 2024-03-06T07:30:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:30:59.299+0000 7f195175c640 1 -- 172.21.15.27:0/3160255330 wait complete. 2024-03-06T07:30:59.513 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:30:59.514 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:00.514 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.619+0000 7f8d78f13640 1 Processor -- start 2024-03-06T07:31:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.619+0000 7f8d78f13640 1 -- start start 2024-03-06T07:31:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.620+0000 7f8d78f13640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d740b95a0 0x7f8d740b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.620+0000 7f8d78f13640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d740b9a10 0x7f8d740b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.620+0000 7f8d78f13640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d740cea30 con 0x7f8d7415df00 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.620+0000 7f8d78f13640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d740cf3c0 con 0x7f8d740b95a0 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.620+0000 7f8d78f13640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d740cf0e0 con 0x7f8d740b9a10 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d71d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d740b95a0 0x7f8d740b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d72d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d740b9a10 0x7f8d740b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d71d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d740b95a0 0x7f8d740b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:34824/0 (socket says 172.21.15.27:34824) 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d7259c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8d7415df00 0x7f8d740d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:51630/0 (socket says 172.21.15.27:51630) 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d71d9b640 1 -- 172.21.15.27:0/4014940998 learned_addr learned my addr 172.21.15.27:0/4014940998 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d71d9b640 1 -- 172.21.15.27:0/4014940998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d740b9a10 msgr2=0x7f8d740b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d71d9b640 1 --2- 172.21.15.27:0/4014940998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d740b9a10 0x7f8d740b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d71d9b640 1 -- 172.21.15.27:0/4014940998 >> v1:172.21.15.69:6789/0 conn(0x7f8d7415df00 legacy=0x7f8d740d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d7159a640 1 -- 172.21.15.27:0/4014940998 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169931166 0 0) 0x7f8d740cea30 con 0x7f8d7415df00 2024-03-06T07:31:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d71d9b640 1 -- 172.21.15.27:0/4014940998 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d740ce470 con 0x7f8d740b95a0 2024-03-06T07:31:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.621+0000 7f8d72d9d640 1 --2- 172.21.15.27:0/4014940998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d740b9a10 0x7f8d740b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.622+0000 7f8d71d9b640 1 --2- 172.21.15.27:0/4014940998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d740b95a0 0x7f8d740b5700 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009870 tx=0x7f8d6802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6b76a214c56ecdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d7159a640 1 -- 172.21.15.27:0/4014940998 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68034070 con 0x7f8d740b95a0 2024-03-06T07:31:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d7159a640 1 -- 172.21.15.27:0/4014940998 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d68030840 con 0x7f8d740b95a0 2024-03-06T07:31:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d7159a640 1 -- 172.21.15.27:0/4014940998 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68030c10 con 0x7f8d740b95a0 2024-03-06T07:31:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d78f13640 1 -- 172.21.15.27:0/4014940998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d740b95a0 msgr2=0x7f8d740b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d78f13640 1 --2- 172.21.15.27:0/4014940998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d740b95a0 0x7f8d740b5700 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009870 tx=0x7f8d6802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d78f13640 1 -- 172.21.15.27:0/4014940998 shutdown_connections 2024-03-06T07:31:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d78f13640 1 --2- 172.21.15.27:0/4014940998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d740b95a0 0x7f8d740b5700 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d78f13640 1 --2- 172.21.15.27:0/4014940998 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d740b9a10 0x7f8d740b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d78f13640 1 -- 172.21.15.27:0/4014940998 >> 172.21.15.27:0/4014940998 conn(0x7f8d74071960 msgr2=0x7f8d74071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d78f13640 1 -- 172.21.15.27:0/4014940998 shutdown_connections 2024-03-06T07:31:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.623+0000 7f8d78f13640 1 -- 172.21.15.27:0/4014940998 wait complete. 2024-03-06T07:31:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.624+0000 7f8d78f13640 1 Processor -- start 2024-03-06T07:31:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d78f13640 1 -- start start 2024-03-06T07:31:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d78f13640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d740b95a0 0x7f8d740ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d78f13640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740b9a10 0x7f8d7415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d78f13640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7415df00 0x7f8d740aa850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d78f13640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d740cedf0 con 0x7f8d740b9a10 2024-03-06T07:31:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d78f13640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d740cec70 con 0x7f8d740b95a0 2024-03-06T07:31:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d78f13640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d740d3220 con 0x7f8d7415df00 2024-03-06T07:31:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d71d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740b9a10 0x7f8d7415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d71d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740b9a10 0x7f8d7415e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56958/0 (socket says 172.21.15.27:56958) 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d71d9b640 1 -- 172.21.15.27:0/211744041 learned_addr learned my addr 172.21.15.27:0/211744041 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d72d9d640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7415df00 0x7f8d740aa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.625+0000 7f8d7259c640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d740b95a0 0x7f8d740ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d71d9b640 1 -- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7415df00 msgr2=0x7f8d740aa850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d71d9b640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7415df00 0x7f8d740aa850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d71d9b640 1 -- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d740b95a0 msgr2=0x7f8d740ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d71d9b640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d740b95a0 0x7f8d740ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d71d9b640 1 -- 172.21.15.27:0/211744041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7414ce30 con 0x7f8d740b9a10 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d71d9b640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740b9a10 0x7f8d7415e2e0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000c00 tx=0x7f8d680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d62ffd640 1 -- 172.21.15.27:0/211744041 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68034040 con 0x7f8d740b9a10 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d62ffd640 1 -- 172.21.15.27:0/211744041 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d68030640 con 0x7f8d740b9a10 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7414d110 con 0x7f8d740b9a10 2024-03-06T07:31:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d62ffd640 1 -- 172.21.15.27:0/211744041 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68039d20 con 0x7f8d740b9a10 2024-03-06T07:31:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.626+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7414d6d0 con 0x7f8d740b9a10 2024-03-06T07:31:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.627+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d74067ff0 con 0x7f8d740b9a10 2024-03-06T07:31:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.628+0000 7f8d62ffd640 1 -- 172.21.15.27:0/211744041 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d6804b060 con 0x7f8d740b9a10 2024-03-06T07:31:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.628+0000 7f8d62ffd640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5406b4d0 0x7f8d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.628+0000 7f8d62ffd640 1 -- 172.21.15.27:0/211744041 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8d680abd50 con 0x7f8d740b9a10 2024-03-06T07:31:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.629+0000 7f8d7259c640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5406b4d0 0x7f8d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.629+0000 7f8d7259c640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5406b4d0 0x7f8d5406d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8d64007fd0 tx=0x7f8d64007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.633+0000 7f8d62ffd640 1 -- 172.21.15.27:0/211744041 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d680af030 con 0x7f8d740b9a10 2024-03-06T07:31:00.803 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.853+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d740cdb60 con 0x7f8d740b9a10 2024-03-06T07:31:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.854+0000 7f8d62ffd640 1 -- 172.21.15.27:0/211744041 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f8d6807acb0 con 0x7f8d740b9a10 2024-03-06T07:31:00.856 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:00.857 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:00.857 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5406b4d0 msgr2=0x7f8d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5406b4d0 0x7f8d5406d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8d64007fd0 tx=0x7f8d64007de0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740b9a10 msgr2=0x7f8d7415e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740b9a10 0x7f8d7415e2e0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000c00 tx=0x7f8d680046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 shutdown_connections 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d5406b4d0 0x7f8d5406d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d7415df00 0x7f8d740aa850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740b9a10 0x7f8d7415e2e0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 --2- 172.21.15.27:0/211744041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d740b95a0 0x7f8d740ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 >> 172.21.15.27:0/211744041 conn(0x7f8d74071960 msgr2=0x7f8d74071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 shutdown_connections 2024-03-06T07:31:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:00.856+0000 7f8d78f13640 1 -- 172.21.15.27:0/211744041 wait complete. 2024-03-06T07:31:01.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:01.071 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:02.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:02.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.178+0000 7f10c5a6b640 1 Processor -- start 2024-03-06T07:31:02.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.178+0000 7f10c5a6b640 1 -- start start 2024-03-06T07:31:02.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.178+0000 7f10c5a6b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10c006b5c0 0x7f10c00b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.178+0000 7f10c5a6b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00b0e90 0x7f10c006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.178+0000 7f10c5a6b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00b3340 con 0x7f10c00ae9b0 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.178+0000 7f10c5a6b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c00cfa00 con 0x7f10c006b5c0 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.178+0000 7f10c5a6b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00cee90 con 0x7f10c00b0e90 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10bf7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10c006b5c0 0x7f10c00b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10beffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00b0e90 0x7f10c006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10bffff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f10c00ae9b0 0x7f10c00d03a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:51636/0 (socket says 172.21.15.27:51636) 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10beffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00b0e90 0x7f10c006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:41400/0 (socket says 172.21.15.27:41400) 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10bf7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10c006b5c0 0x7f10c00b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:34832/0 (socket says 172.21.15.27:34832) 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10bffff640 1 -- 172.21.15.27:0/3201576688 learned_addr learned my addr 172.21.15.27:0/3201576688 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10bf7fe640 1 -- 172.21.15.27:0/3201576688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00b0e90 msgr2=0x7f10c006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10bf7fe640 1 --2- 172.21.15.27:0/3201576688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00b0e90 0x7f10c006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10bf7fe640 1 -- 172.21.15.27:0/3201576688 >> v1:172.21.15.69:6789/0 conn(0x7f10c00ae9b0 legacy=0x7f10c00d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10bf7fe640 1 -- 172.21.15.27:0/3201576688 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c00cfc50 con 0x7f10c006b5c0 2024-03-06T07:31:02.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.179+0000 7f10beffd640 1 --2- 172.21.15.27:0/3201576688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00b0e90 0x7f10c006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.180+0000 7f10bf7fe640 1 --2- 172.21.15.27:0/3201576688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10c006b5c0 0x7f10c00b2e00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f10b00097a0 tx=0x7f10b002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9694896e8945ab71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.180+0000 7f10be7fc640 1 -- 172.21.15.27:0/3201576688 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0031780 con 0x7f10c006b5c0 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.180+0000 7f10be7fc640 1 -- 172.21.15.27:0/3201576688 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b0037070 con 0x7f10c006b5c0 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.180+0000 7f10be7fc640 1 -- 172.21.15.27:0/3201576688 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0031b40 con 0x7f10c006b5c0 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.181+0000 7f10c5a6b640 1 -- 172.21.15.27:0/3201576688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10c006b5c0 msgr2=0x7f10c00b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.181+0000 7f10c5a6b640 1 --2- 172.21.15.27:0/3201576688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10c006b5c0 0x7f10c00b2e00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f10b00097a0 tx=0x7f10b002b820 comp rx=0 tx=0).stop 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.181+0000 7f10c5a6b640 1 -- 172.21.15.27:0/3201576688 shutdown_connections 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.181+0000 7f10c5a6b640 1 --2- 172.21.15.27:0/3201576688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00b0e90 0x7f10c006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.181+0000 7f10c5a6b640 1 --2- 172.21.15.27:0/3201576688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10c006b5c0 0x7f10c00b2e00 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.181+0000 7f10c5a6b640 1 -- 172.21.15.27:0/3201576688 >> 172.21.15.27:0/3201576688 conn(0x7f10c00b36d0 msgr2=0x7f10c00b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.181+0000 7f10c5a6b640 1 -- 172.21.15.27:0/3201576688 shutdown_connections 2024-03-06T07:31:02.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.181+0000 7f10c5a6b640 1 -- 172.21.15.27:0/3201576688 wait complete. 2024-03-06T07:31:02.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.182+0000 7f10c5a6b640 1 Processor -- start 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.182+0000 7f10c5a6b640 1 -- start start 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10c5a6b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10c006b5c0 0x7f10c00cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10c5a6b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00ae9b0 0x7f10c00ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10c5a6b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c00b0e90 0x7f10c00aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10c5a6b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c006c790 con 0x7f10c00b0e90 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10c5a6b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c006bee0 con 0x7f10c006b5c0 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10c5a6b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c006c910 con 0x7f10c00ae9b0 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c00b0e90 0x7f10c00aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c00b0e90 0x7f10c00aee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56960/0 (socket says 172.21.15.27:56960) 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10bffff640 1 -- 172.21.15.27:0/1291778862 learned_addr learned my addr 172.21.15.27:0/1291778862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10beffd640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00ae9b0 0x7f10c00ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10bf7fe640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10c006b5c0 0x7f10c00cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10bffff640 1 -- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00ae9b0 msgr2=0x7f10c00ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10bffff640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00ae9b0 0x7f10c00ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10bffff640 1 -- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10c006b5c0 msgr2=0x7f10c00cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10bffff640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10c006b5c0 0x7f10c00cb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.183+0000 7f10bffff640 1 -- 172.21.15.27:0/1291778862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c01f7460 con 0x7f10c00b0e90 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.184+0000 7f10bffff640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c00b0e90 0x7f10c00aee00 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f10b800cbc0 tx=0x7f10b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.184+0000 7f10a7fff640 1 -- 172.21.15.27:0/1291778862 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8017070 con 0x7f10c00b0e90 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.184+0000 7f10a7fff640 1 -- 172.21.15.27:0/1291778862 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b800ad40 con 0x7f10c00b0e90 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.184+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c00c6320 con 0x7f10c00b0e90 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.184+0000 7f10a7fff640 1 -- 172.21.15.27:0/1291778862 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8012600 con 0x7f10c00b0e90 2024-03-06T07:31:02.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.184+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c00c6b60 con 0x7f10c00b0e90 2024-03-06T07:31:02.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.185+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c0068700 con 0x7f10c00b0e90 2024-03-06T07:31:02.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.185+0000 7f10a7fff640 1 -- 172.21.15.27:0/1291778862 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10b801a070 con 0x7f10c00b0e90 2024-03-06T07:31:02.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.186+0000 7f10a7fff640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ac06b4d0 0x7f10ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.186+0000 7f10a7fff640 1 -- 172.21.15.27:0/1291778862 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f10b8089f10 con 0x7f10c00b0e90 2024-03-06T07:31:02.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.186+0000 7f10bf7fe640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ac06b4d0 0x7f10ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.187+0000 7f10bf7fe640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ac06b4d0 0x7f10ac06d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f10b0036000 tx=0x7f10b003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.190+0000 7f10a7fff640 1 -- 172.21.15.27:0/1291778862 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10c00c6320 con 0x7f10c00b0e90 2024-03-06T07:31:02.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.416+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10c00d0aa0 con 0x7f10c00b0e90 2024-03-06T07:31:02.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.416+0000 7f10a7fff640 1 -- 172.21.15.27:0/1291778862 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f10b8058680 con 0x7f10c00b0e90 2024-03-06T07:31:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:02.418 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:02.418 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:02.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.418+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ac06b4d0 msgr2=0x7f10ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.418+0000 7f10c5a6b640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ac06b4d0 0x7f10ac06d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f10b0036000 tx=0x7f10b003d040 comp rx=0 tx=0).stop 2024-03-06T07:31:02.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.418+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c00b0e90 msgr2=0x7f10c00aee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.418+0000 7f10c5a6b640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c00b0e90 0x7f10c00aee00 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f10b800cbc0 tx=0x7f10b800a430 comp rx=0 tx=0).stop 2024-03-06T07:31:02.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.419+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 shutdown_connections 2024-03-06T07:31:02.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.419+0000 7f10c5a6b640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10ac06b4d0 0x7f10ac06d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.419+0000 7f10c5a6b640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10c00b0e90 0x7f10c00aee00 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.419+0000 7f10c5a6b640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10c00ae9b0 0x7f10c00ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.419+0000 7f10c5a6b640 1 --2- 172.21.15.27:0/1291778862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10c006b5c0 0x7f10c00cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.419+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 >> 172.21.15.27:0/1291778862 conn(0x7f10c00b36d0 msgr2=0x7f10c015e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.419+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 shutdown_connections 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:02.419+0000 7f10c5a6b640 1 -- 172.21.15.27:0/1291778862 wait complete. 2024-03-06T07:31:02.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:02.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:02.805 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:03.634 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:03.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.739+0000 7fc4624e1640 1 Processor -- start 2024-03-06T07:31:03.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.739+0000 7fc4624e1640 1 -- start start 2024-03-06T07:31:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc4624e1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c15ac30 0x7fc45c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc4624e1640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc45c0afb30 0x7fc45c0719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc4624e1640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc45c071ef0 0x7fc45c06d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc4624e1640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45c072320 con 0x7fc45c071ef0 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc4624e1640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45c0b3e80 con 0x7fc45c0afb30 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc4624e1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0b4120 con 0x7fc45c15ac30 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc460a7e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc45c071ef0 0x7fc45c06d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc460a7e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc45c071ef0 0x7fc45c06d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55072/0 (socket says 172.21.15.27:55072) 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc460a7e640 1 -- 172.21.15.27:0/977290830 learned_addr learned my addr 172.21.15.27:0/977290830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc45bfff640 1 --2- 172.21.15.27:0/977290830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c15ac30 0x7fc45c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc45b7fe640 1 --2- 172.21.15.27:0/977290830 >> v2:172.21.15.69:3300/0 conn(0x7fc45c0afb30 0x7fc45c0719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc460a7e640 1 -- 172.21.15.27:0/977290830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c15ac30 msgr2=0x7fc45c0b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc460a7e640 1 --2- 172.21.15.27:0/977290830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c15ac30 0x7fc45c0b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc460a7e640 1 -- 172.21.15.27:0/977290830 >> v2:172.21.15.69:3300/0 conn(0x7fc45c0afb30 msgr2=0x7fc45c0719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc460a7e640 1 --2- 172.21.15.27:0/977290830 >> v2:172.21.15.69:3300/0 conn(0x7fc45c0afb30 0x7fc45c0719b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.740+0000 7fc460a7e640 1 -- 172.21.15.27:0/977290830 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c0b4400 con 0x7fc45c071ef0 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.741+0000 7fc460a7e640 1 --2- 172.21.15.27:0/977290830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c071ef0 0x7fc45c06d9b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fc45c073650 tx=0x7fc45402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4169b1a1fe748886 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.741+0000 7fc45affd640 1 -- 172.21.15.27:0/977290830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45402ca70 con 0x7fc45c071ef0 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.741+0000 7fc45affd640 1 -- 172.21.15.27:0/977290830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45402cc10 con 0x7fc45c071ef0 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.741+0000 7fc45affd640 1 -- 172.21.15.27:0/977290830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4540343f0 con 0x7fc45c071ef0 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.742+0000 7fc4624e1640 1 -- 172.21.15.27:0/977290830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c071ef0 msgr2=0x7fc45c06d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.742+0000 7fc4624e1640 1 --2- 172.21.15.27:0/977290830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c071ef0 0x7fc45c06d9b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fc45c073650 tx=0x7fc45402bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.742+0000 7fc4624e1640 1 -- 172.21.15.27:0/977290830 shutdown_connections 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.742+0000 7fc4624e1640 1 --2- 172.21.15.27:0/977290830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c071ef0 0x7fc45c06d9b0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.742+0000 7fc4624e1640 1 --2- 172.21.15.27:0/977290830 >> v2:172.21.15.69:3300/0 conn(0x7fc45c0afb30 0x7fc45c0719b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.742+0000 7fc4624e1640 1 --2- 172.21.15.27:0/977290830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c15ac30 0x7fc45c0b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.742+0000 7fc4624e1640 1 -- 172.21.15.27:0/977290830 >> 172.21.15.27:0/977290830 conn(0x7fc45c0d0990 msgr2=0x7fc45c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.742+0000 7fc4624e1640 1 -- 172.21.15.27:0/977290830 shutdown_connections 2024-03-06T07:31:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.742+0000 7fc4624e1640 1 -- 172.21.15.27:0/977290830 wait complete. 2024-03-06T07:31:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.743+0000 7fc4624e1640 1 Processor -- start 2024-03-06T07:31:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.743+0000 7fc4624e1640 1 -- start start 2024-03-06T07:31:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.744+0000 7fc4624e1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c071ef0 0x7fc45c15db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.744+0000 7fc4624e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c0afb30 0x7fc45c14f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc4624e1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45c15ac30 0x7fc45c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc4624e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0b4ca0 con 0x7fc45c0afb30 2024-03-06T07:31:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc4624e1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0b49c0 con 0x7fc45c15ac30 2024-03-06T07:31:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc4624e1640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c06def0 con 0x7fc45c071ef0 2024-03-06T07:31:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c0afb30 0x7fc45c14f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c0afb30 0x7fc45c14f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55086/0 (socket says 172.21.15.27:55086) 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45b7fe640 1 -- 172.21.15.27:0/1985022693 learned_addr learned my addr 172.21.15.27:0/1985022693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45bfff640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c071ef0 0x7fc45c15db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc460a7e640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45c15ac30 0x7fc45c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45b7fe640 1 -- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c071ef0 msgr2=0x7fc45c15db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45b7fe640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c071ef0 0x7fc45c15db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45b7fe640 1 -- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45c15ac30 msgr2=0x7fc45c1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45b7fe640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45c15ac30 0x7fc45c1575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45b7fe640 1 -- 172.21.15.27:0/1985022693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c152f20 con 0x7fc45c0afb30 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45bfff640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c071ef0 0x7fc45c15db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc460a7e640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45c15ac30 0x7fc45c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc45b7fe640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c0afb30 0x7fc45c14f050 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fc45000c820 tx=0x7fc45000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc458ff9640 1 -- 172.21.15.27:0/1985022693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450015070 con 0x7fc45c0afb30 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc458ff9640 1 -- 172.21.15.27:0/1985022693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450004220 con 0x7fc45c0afb30 2024-03-06T07:31:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c15e190 con 0x7fc45c0afb30 2024-03-06T07:31:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc458ff9640 1 -- 172.21.15.27:0/1985022693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45000ce10 con 0x7fc45c0afb30 2024-03-06T07:31:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.745+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c156d20 con 0x7fc45c0afb30 2024-03-06T07:31:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.746+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c068700 con 0x7fc45c0afb30 2024-03-06T07:31:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.747+0000 7fc458ff9640 1 -- 172.21.15.27:0/1985022693 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc450007830 con 0x7fc45c0afb30 2024-03-06T07:31:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.747+0000 7fc458ff9640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc44806b4d0 0x7fc44806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.748+0000 7fc458ff9640 1 -- 172.21.15.27:0/1985022693 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc45008a140 con 0x7fc45c0afb30 2024-03-06T07:31:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.748+0000 7fc45bfff640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc44806b4d0 0x7fc44806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.748+0000 7fc45bfff640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc44806b4d0 0x7fc44806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc44c006fd0 tx=0x7fc44c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.757+0000 7fc458ff9640 1 -- 172.21.15.27:0/1985022693 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc450058e50 con 0x7fc45c0afb30 2024-03-06T07:31:03.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.979+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc45c15b010 con 0x7fc45c0afb30 2024-03-06T07:31:03.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.979+0000 7fc458ff9640 1 -- 172.21.15.27:0/1985022693 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7fc450058750 con 0x7fc45c0afb30 2024-03-06T07:31:03.981 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:03.981 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:03.981 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc44806b4d0 msgr2=0x7fc44806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc44806b4d0 0x7fc44806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc44c006fd0 tx=0x7fc44c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c0afb30 msgr2=0x7fc45c14f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c0afb30 0x7fc45c14f050 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fc45000c820 tx=0x7fc45000cce0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 shutdown_connections 2024-03-06T07:31:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc44806b4d0 0x7fc44806d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc45c15ac30 0x7fc45c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc45c0afb30 0x7fc45c14f050 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 --2- 172.21.15.27:0/1985022693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc45c071ef0 0x7fc45c15db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 >> 172.21.15.27:0/1985022693 conn(0x7fc45c0d0990 msgr2=0x7fc45c159230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.981+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 shutdown_connections 2024-03-06T07:31:03.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:03.982+0000 7fc4624e1640 1 -- 172.21.15.27:0/1985022693 wait complete. 2024-03-06T07:31:04.195 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:04.195 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:04.807 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:05.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.301+0000 7f857417a640 1 Processor -- start 2024-03-06T07:31:05.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.302+0000 7f857417a640 1 -- start start 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.302+0000 7f857417a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0b9a10 0x7f856c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.302+0000 7f857417a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c0cea30 con 0x7f856c0b95a0 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.302+0000 7f857417a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c0cf3c0 con 0x7f856c15df00 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.302+0000 7f857417a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0cf0e0 con 0x7f856c0b9a10 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8571715640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f856c0b95a0 0x7f856c0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37918/0 (socket says 172.21.15.27:37918) 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8571715640 1 -- 172.21.15.27:0/1932176457 learned_addr learned my addr 172.21.15.27:0/1932176457 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8572717640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0b9a10 0x7f856c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206482627 0 0) 0x7f856c0cea30 con 0x7f856c0b95a0 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8558003520 con 0x7f856c0b95a0 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744287216 0 0) 0x7f856c0cf3c0 con 0x7f856c15df00 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f856c0cea30 con 0x7f856c15df00 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 444770374 0 0) 0x7f8558003520 con 0x7f856c0b95a0 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0b9a10 msgr2=0x7f856c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0b9a10 0x7f856c0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 >> v1:172.21.15.69:6789/0 conn(0x7f856c15df00 legacy=0x7f856c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0ce470 con 0x7f856c0b95a0 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f855c001670 con 0x7f856c0b95a0 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 >> v1:172.21.15.27:6789/0 conn(0x7f856c0b95a0 legacy=0x7f856c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8558003b70 0x7f8558003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0b9a10 0x7f8558004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8558004d40 0x7f85580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0cf0e0 con 0x7f8558004d40 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855800ba70 con 0x7f8558003b70 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.303+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855800bc70 con 0x7f856c0b9a10 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.304+0000 7f8572717640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8558004d40 0x7f85580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.304+0000 7f8571f16640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8558003b70 0x7f8558003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.304+0000 7f8571715640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0b9a10 0x7f8558004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.304+0000 7f8572717640 1 -- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0b9a10 msgr2=0x7f8558004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.304+0000 7f8572717640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0b9a10 0x7f8558004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.304+0000 7f8572717640 1 -- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8558003b70 msgr2=0x7f8558003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.304+0000 7f8572717640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8558003b70 0x7f8558003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.304+0000 7f8572717640 1 -- 172.21.15.27:0/1932176457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8558003520 con 0x7f8558004d40 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f8572717640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8558004d40 0x7f85580051c0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f8564009fc0 tx=0x7f8558005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57666fc02ecd2f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8564039070 con 0x7f8558004d40 2024-03-06T07:31:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856402cc20 con 0x7f8558004d40 2024-03-06T07:31:05.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f8570f14640 1 -- 172.21.15.27:0/1932176457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8564009c10 con 0x7f8558004d40 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f857417a640 1 -- 172.21.15.27:0/1932176457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8558004d40 msgr2=0x7f85580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f857417a640 1 --2- 172.21.15.27:0/1932176457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8558004d40 0x7f85580051c0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f8564009fc0 tx=0x7f8558005830 comp rx=0 tx=0).stop 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f8572717640 1 -- 172.21.15.27:0/1932176457 reap_dead start 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f857417a640 1 -- 172.21.15.27:0/1932176457 shutdown_connections 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f857417a640 1 -- 172.21.15.27:0/1932176457 >> 172.21.15.27:0/1932176457 conn(0x7f856c071960 msgr2=0x7f856c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f857417a640 1 -- 172.21.15.27:0/1932176457 shutdown_connections 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.305+0000 7f857417a640 1 -- 172.21.15.27:0/1932176457 wait complete. 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.306+0000 7f857417a640 1 Processor -- start 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f857417a640 1 -- start start 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f857417a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8558004d40 0x7f856c157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f857417a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0c7e20 0x7f856c0c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f857417a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15ff10 0x7f856c1603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f857417a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0cedf0 con 0x7f856c15ff10 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f857417a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0cec70 con 0x7f8558004d40 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f857417a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0b5e10 con 0x7f856c0c7e20 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f8572717640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15ff10 0x7f856c1603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f8572717640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15ff10 0x7f856c1603c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55090/0 (socket says 172.21.15.27:55090) 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f8572717640 1 -- 172.21.15.27:0/1708346503 learned_addr learned my addr 172.21.15.27:0/1708346503 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.307+0000 7f8571f16640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8558004d40 0x7f856c157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f8571715640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0c7e20 0x7f856c0c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f8572717640 1 -- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0c7e20 msgr2=0x7f856c0c82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f8572717640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0c7e20 0x7f856c0c82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f8572717640 1 -- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8558004d40 msgr2=0x7f856c157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f8572717640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8558004d40 0x7f856c157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f8572717640 1 -- 172.21.15.27:0/1708346503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0cae80 con 0x7f856c15ff10 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f8572717640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15ff10 0x7f856c1603c0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f8564002230 tx=0x7f8564009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f85627fc640 1 -- 172.21.15.27:0/1708346503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8564039030 con 0x7f856c15ff10 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f85627fc640 1 -- 172.21.15.27:0/1708346503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8564043070 con 0x7f856c15ff10 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c0cb160 con 0x7f856c15ff10 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f85627fc640 1 -- 172.21.15.27:0/1708346503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856403d7f0 con 0x7f856c15ff10 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.308+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c0cb720 con 0x7f856c15ff10 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.309+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c067ff0 con 0x7f856c15ff10 2024-03-06T07:31:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.310+0000 7f85627fc640 1 -- 172.21.15.27:0/1708346503 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8564005d10 con 0x7f856c15ff10 2024-03-06T07:31:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.311+0000 7f85627fc640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f855006b4d0 0x7f855006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.311+0000 7f85627fc640 1 -- 172.21.15.27:0/1708346503 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f85640ac570 con 0x7f856c15ff10 2024-03-06T07:31:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.311+0000 7f8571f16640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f855006b4d0 0x7f855006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.311+0000 7f8571f16640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f855006b4d0 0x7f855006d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8568001940 tx=0x7f8568009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.320+0000 7f85627fc640 1 -- 172.21.15.27:0/1708346503 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f856404d020 con 0x7f856c15ff10 2024-03-06T07:31:05.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.541+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f856c1623a0 con 0x7f856c15ff10 2024-03-06T07:31:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.542+0000 7f85627fc640 1 -- 172.21.15.27:0/1708346503 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f856407adf0 con 0x7f856c15ff10 2024-03-06T07:31:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:05.543 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:05.544 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.543+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f855006b4d0 msgr2=0x7f855006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f855006b4d0 0x7f855006d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8568001940 tx=0x7f8568009040 comp rx=0 tx=0).stop 2024-03-06T07:31:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15ff10 msgr2=0x7f856c1603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15ff10 0x7f856c1603c0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f8564002230 tx=0x7f8564009750 comp rx=0 tx=0).stop 2024-03-06T07:31:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 shutdown_connections 2024-03-06T07:31:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f855006b4d0 0x7f855006d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15ff10 0x7f856c1603c0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f856c0c7e20 0x7f856c0c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 --2- 172.21.15.27:0/1708346503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8558004d40 0x7f856c157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 >> 172.21.15.27:0/1708346503 conn(0x7f856c071960 msgr2=0x7f856c0cf320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 shutdown_connections 2024-03-06T07:31:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:05.544+0000 7f857417a640 1 -- 172.21.15.27:0/1708346503 wait complete. 2024-03-06T07:31:05.757 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:05.757 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:06.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:06.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.864+0000 7fae5dc76640 1 Processor -- start 2024-03-06T07:31:06.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.864+0000 7fae5dc76640 1 -- start start 2024-03-06T07:31:06.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.865+0000 7fae5dc76640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.865+0000 7fae5dc76640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae580ce610 con 0x7fae580b2b50 2024-03-06T07:31:06.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.865+0000 7fae5dc76640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae580d2920 con 0x7fae58071540 2024-03-06T07:31:06.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.865+0000 7fae5dc76640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae580d2f70 con 0x7fae580d3400 2024-03-06T07:31:06.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.865+0000 7fae57fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fae580b2b50 0x7fae580b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37928/0 (socket says 172.21.15.27:37928) 2024-03-06T07:31:06.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.865+0000 7fae57fff640 1 -- 172.21.15.27:0/1744914975 learned_addr learned my addr 172.21.15.27:0/1744914975 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.865+0000 7fae56ffd640 1 --2- 172.21.15.27:0/1744914975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.866+0000 7fae567fc640 1 -- 172.21.15.27:0/1744914975 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467820709 0 0) 0x7fae580ce610 con 0x7fae580b2b50 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.866+0000 7fae567fc640 1 -- 172.21.15.27:0/1744914975 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae40003520 con 0x7fae580b2b50 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.866+0000 7fae567fc640 1 -- 172.21.15.27:0/1744914975 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567923539 0 0) 0x7fae580d2920 con 0x7fae58071540 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.866+0000 7fae567fc640 1 -- 172.21.15.27:0/1744914975 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae580ce610 con 0x7fae58071540 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.866+0000 7fae56ffd640 1 -- 172.21.15.27:0/1744914975 >> v1:172.21.15.69:6789/0 conn(0x7fae58071540 legacy=0x7fae58071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.866+0000 7fae56ffd640 1 -- 172.21.15.27:0/1744914975 >> v1:172.21.15.27:6789/0 conn(0x7fae580b2b50 legacy=0x7fae580b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.866+0000 7fae567fc640 1 -- 172.21.15.27:0/1744914975 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1960235023 0 0) 0x7fae40003520 con 0x7fae580b2b50 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.866+0000 7fae56ffd640 1 -- 172.21.15.27:0/1744914975 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580d2c00 con 0x7fae580d3400 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.867+0000 7fae56ffd640 1 --2- 172.21.15.27:0/1744914975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b30a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0097d0 tx=0x7fae4c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=200c7b99a6dbeb0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.867+0000 7fae567fc640 1 -- 172.21.15.27:0/1744914975 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c037070 con 0x7fae580d3400 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.867+0000 7fae567fc640 1 -- 172.21.15.27:0/1744914975 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c030510 con 0x7fae580d3400 2024-03-06T07:31:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.867+0000 7fae567fc640 1 -- 172.21.15.27:0/1744914975 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae500019f0 con 0x7fae580d3400 2024-03-06T07:31:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.868+0000 7fae5dc76640 1 -- 172.21.15.27:0/1744914975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 msgr2=0x7fae580b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.868+0000 7fae5dc76640 1 --2- 172.21.15.27:0/1744914975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b30a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0097d0 tx=0x7fae4c02b800 comp rx=0 tx=0).stop 2024-03-06T07:31:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.868+0000 7fae5dc76640 1 -- 172.21.15.27:0/1744914975 shutdown_connections 2024-03-06T07:31:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.868+0000 7fae5dc76640 1 --2- 172.21.15.27:0/1744914975 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b30a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.868+0000 7fae5dc76640 1 -- 172.21.15.27:0/1744914975 >> 172.21.15.27:0/1744914975 conn(0x7fae580d3ca0 msgr2=0x7fae580d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.868+0000 7fae5dc76640 1 -- 172.21.15.27:0/1744914975 shutdown_connections 2024-03-06T07:31:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.868+0000 7fae5dc76640 1 -- 172.21.15.27:0/1744914975 wait complete. 2024-03-06T07:31:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.870+0000 7fae5dc76640 1 Processor -- start 2024-03-06T07:31:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.870+0000 7fae5dc76640 1 -- start start 2024-03-06T07:31:06.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae5dc76640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae58071540 0x7fae581f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae5dc76640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae580b2b50 0x7fae581f4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae5dc76640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae5dc76640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5815e370 con 0x7fae58071540 2024-03-06T07:31:06.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae5dc76640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae580d4be0 con 0x7fae580b2b50 2024-03-06T07:31:06.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae5dc76640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae580b8380 con 0x7fae580d3400 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae58071540 0x7fae581f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae577fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae58071540 0x7fae581f0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55094/0 (socket says 172.21.15.27:55094) 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae577fe640 1 -- 172.21.15.27:0/1237283022 learned_addr learned my addr 172.21.15.27:0/1237283022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae56ffd640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae580b2b50 0x7fae581f4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.871+0000 7fae57fff640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae577fe640 1 -- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 msgr2=0x7fae580b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae577fe640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae577fe640 1 -- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae580b2b50 msgr2=0x7fae581f4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae577fe640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae580b2b50 0x7fae581f4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae577fe640 1 -- 172.21.15.27:0/1237283022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580b3ab0 con 0x7fae58071540 2024-03-06T07:31:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae57fff640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:06.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae577fe640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae58071540 0x7fae581f0d40 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fae4800c530 tx=0x7fae4800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae3ffff640 1 -- 172.21.15.27:0/1237283022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae480052d0 con 0x7fae58071540 2024-03-06T07:31:06.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae3ffff640 1 -- 172.21.15.27:0/1237283022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae48005490 con 0x7fae58071540 2024-03-06T07:31:06.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae580b3d90 con 0x7fae58071540 2024-03-06T07:31:06.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae3ffff640 1 -- 172.21.15.27:0/1237283022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48012870 con 0x7fae58071540 2024-03-06T07:31:06.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.872+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae581f1280 con 0x7fae58071540 2024-03-06T07:31:06.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.873+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae58068700 con 0x7fae58071540 2024-03-06T07:31:06.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.874+0000 7fae3ffff640 1 -- 172.21.15.27:0/1237283022 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae48012a10 con 0x7fae58071540 2024-03-06T07:31:06.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.874+0000 7fae3ffff640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fae4406b4d0 0x7fae4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.874+0000 7fae3ffff640 1 -- 172.21.15.27:0/1237283022 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fae48089e00 con 0x7fae58071540 2024-03-06T07:31:06.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.874+0000 7fae56ffd640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fae4406b4d0 0x7fae4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.875+0000 7fae56ffd640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fae4406b4d0 0x7fae4406d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fae4c034000 tx=0x7fae4c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:06.878+0000 7fae3ffff640 1 -- 172.21.15.27:0/1237283022 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae48058d00 con 0x7fae58071540 2024-03-06T07:31:07.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.100+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae580caae0 con 0x7fae58071540 2024-03-06T07:31:07.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.100+0000 7fae3ffff640 1 -- 172.21.15.27:0/1237283022 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7fae48058600 con 0x7fae58071540 2024-03-06T07:31:07.102 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:07.102 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:07.102 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fae4406b4d0 msgr2=0x7fae4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fae4406b4d0 0x7fae4406d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fae4c034000 tx=0x7fae4c008030 comp rx=0 tx=0).stop 2024-03-06T07:31:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae58071540 msgr2=0x7fae581f0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae58071540 0x7fae581f0d40 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fae4800c530 tx=0x7fae4800c9f0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 shutdown_connections 2024-03-06T07:31:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fae4406b4d0 0x7fae4406d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fae580d3400 0x7fae580b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae580b2b50 0x7fae581f4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 --2- 172.21.15.27:0/1237283022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae58071540 0x7fae581f0d40 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 >> 172.21.15.27:0/1237283022 conn(0x7fae580d3ca0 msgr2=0x7fae5815f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 shutdown_connections 2024-03-06T07:31:07.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:07.102+0000 7fae5dc76640 1 -- 172.21.15.27:0/1237283022 wait complete. 2024-03-06T07:31:07.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:07.320 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:08.322 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.427+0000 7f1b092f4640 1 Processor -- start 2024-03-06T07:31:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.427+0000 7f1b092f4640 1 -- start start 2024-03-06T07:31:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.428+0000 7f1b092f4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b04071540 0x7f1b040b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.428+0000 7f1b092f4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1b040d3400 0x7f1b04071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.428+0000 7f1b092f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b040b2b50 0x7f1b040d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.428+0000 7f1b092f4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b040d2f70 con 0x7f1b04071540 2024-03-06T07:31:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.428+0000 7f1b092f4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b040d4be0 con 0x7f1b040d3400 2024-03-06T07:31:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.428+0000 7f1b092f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b04071e60 con 0x7f1b040b2b50 2024-03-06T07:31:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.428+0000 7f1b02ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b04071540 0x7f1b040b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.428+0000 7f1b02ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b04071540 0x7f1b040b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55098/0 (socket says 172.21.15.27:55098) 2024-03-06T07:31:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.428+0000 7f1b02ffd640 1 -- 172.21.15.27:0/1544198015 learned_addr learned my addr 172.21.15.27:0/1544198015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.429+0000 7f1b02ffd640 1 -- 172.21.15.27:0/1544198015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b040b2b50 msgr2=0x7f1b040d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.429+0000 7f1b02ffd640 1 --2- 172.21.15.27:0/1544198015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b040b2b50 0x7f1b040d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.429+0000 7f1b02ffd640 1 -- 172.21.15.27:0/1544198015 >> v2:172.21.15.69:3300/0 conn(0x7f1b040d3400 msgr2=0x7f1b04071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.429+0000 7f1b02ffd640 1 --2- 172.21.15.27:0/1544198015 >> v2:172.21.15.69:3300/0 conn(0x7f1b040d3400 0x7f1b04071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.429+0000 7f1b02ffd640 1 -- 172.21.15.27:0/1544198015 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b040ce610 con 0x7f1b04071540 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.429+0000 7f1b02ffd640 1 --2- 172.21.15.27:0/1544198015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b04071540 0x7f1b040b30a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f1b040b6860 tx=0x7f1aec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c742bd4ec3bd69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b01ffb640 1 -- 172.21.15.27:0/1544198015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec035020 con 0x7f1b04071540 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b01ffb640 1 -- 172.21.15.27:0/1544198015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aec031870 con 0x7f1b04071540 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b01ffb640 1 -- 172.21.15.27:0/1544198015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec031c40 con 0x7f1b04071540 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b092f4640 1 -- 172.21.15.27:0/1544198015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b04071540 msgr2=0x7f1b040b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b092f4640 1 --2- 172.21.15.27:0/1544198015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b04071540 0x7f1b040b30a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f1b040b6860 tx=0x7f1aec02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b092f4640 1 -- 172.21.15.27:0/1544198015 shutdown_connections 2024-03-06T07:31:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b092f4640 1 --2- 172.21.15.27:0/1544198015 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b040b2b50 0x7f1b040d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b092f4640 1 --2- 172.21.15.27:0/1544198015 >> v2:172.21.15.69:3300/0 conn(0x7f1b040d3400 0x7f1b04071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b092f4640 1 --2- 172.21.15.27:0/1544198015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b04071540 0x7f1b040b30a0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b092f4640 1 -- 172.21.15.27:0/1544198015 >> 172.21.15.27:0/1544198015 conn(0x7f1b040d3ca0 msgr2=0x7f1b040d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b092f4640 1 -- 172.21.15.27:0/1544198015 shutdown_connections 2024-03-06T07:31:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.430+0000 7f1b092f4640 1 -- 172.21.15.27:0/1544198015 wait complete. 2024-03-06T07:31:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b092f4640 1 Processor -- start 2024-03-06T07:31:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b092f4640 1 -- start start 2024-03-06T07:31:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b092f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b04071540 0x7f1b041e6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b092f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040b2b50 0x7f1b040c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b092f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b040d3400 0x7f1b040cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b092f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040ce8f0 con 0x7f1b040b2b50 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b092f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040d2c00 con 0x7f1b040d3400 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b092f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0415e370 con 0x7f1b04071540 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b027fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040b2b50 0x7f1b040c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b027fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040b2b50 0x7f1b040c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55110/0 (socket says 172.21.15.27:55110) 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.432+0000 7f1b027fc640 1 -- 172.21.15.27:0/313092616 learned_addr learned my addr 172.21.15.27:0/313092616 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b02ffd640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b04071540 0x7f1b041e6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b037fe640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b040d3400 0x7f1b040cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b027fc640 1 -- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b04071540 msgr2=0x7f1b041e6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b027fc640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b04071540 0x7f1b041e6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b027fc640 1 -- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b040d3400 msgr2=0x7f1b040cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b027fc640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b040d3400 0x7f1b040cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b027fc640 1 -- 172.21.15.27:0/313092616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b04149c30 con 0x7f1b040b2b50 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b037fe640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b040d3400 0x7f1b040cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b02ffd640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b04071540 0x7f1b041e6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b027fc640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040b2b50 0x7f1b040c96d0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f1af400b440 tx=0x7f1af400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1afb7fe640 1 -- 172.21.15.27:0/313092616 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af4015020 con 0x7f1b040b2b50 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1afb7fe640 1 -- 172.21.15.27:0/313092616 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af40118b0 con 0x7f1b040b2b50 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b04149f10 con 0x7f1b040b2b50 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1afb7fe640 1 -- 172.21.15.27:0/313092616 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af4011ca0 con 0x7f1b040b2b50 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.433+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0414a4d0 con 0x7f1b040b2b50 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.434+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b04068700 con 0x7f1b040b2b50 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.435+0000 7f1afb7fe640 1 -- 172.21.15.27:0/313092616 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1af4018020 con 0x7f1b040b2b50 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.435+0000 7f1afb7fe640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ae406b4d0 0x7f1ae406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.436+0000 7f1afb7fe640 1 -- 172.21.15.27:0/313092616 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1af408a370 con 0x7f1b040b2b50 2024-03-06T07:31:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.436+0000 7f1b02ffd640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ae406b4d0 0x7f1ae406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.436+0000 7f1b02ffd640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ae406b4d0 0x7f1ae406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1aec036000 tx=0x7f1aec03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.440+0000 7f1afb7fe640 1 -- 172.21.15.27:0/313092616 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1af401f020 con 0x7f1b040b2b50 2024-03-06T07:31:08.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.704+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b0415f0e0 con 0x7f1b040b2b50 2024-03-06T07:31:08.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.705+0000 7f1afb7fe640 1 -- 172.21.15.27:0/313092616 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f1af400d040 con 0x7f1b040b2b50 2024-03-06T07:31:08.707 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:08.707 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ae406b4d0 msgr2=0x7f1ae406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ae406b4d0 0x7f1ae406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1aec036000 tx=0x7f1aec03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040b2b50 msgr2=0x7f1b040c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040b2b50 0x7f1b040c96d0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f1af400b440 tx=0x7f1af400b900 comp rx=0 tx=0).stop 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 shutdown_connections 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ae406b4d0 0x7f1ae406d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b040d3400 0x7f1b040cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040b2b50 0x7f1b040c96d0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 --2- 172.21.15.27:0/313092616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1b04071540 0x7f1b041e6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 >> 172.21.15.27:0/313092616 conn(0x7f1b040d3ca0 msgr2=0x7f1b04160c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 shutdown_connections 2024-03-06T07:31:08.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:08.707+0000 7f1b092f4640 1 -- 172.21.15.27:0/313092616 wait complete. 2024-03-06T07:31:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:08.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:08.922 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:09.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.028+0000 7fe79b9a0640 1 Processor -- start 2024-03-06T07:31:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.028+0000 7fe79b9a0640 1 -- start start 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.029+0000 7fe79b9a0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7940ae9b0 0x7fe7940d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.029+0000 7fe79b9a0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940cee90 con 0x7fe7940b0e90 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.029+0000 7fe79b9a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940cfc50 con 0x7fe79406b5c0 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.029+0000 7fe79b9a0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7940cfa00 con 0x7fe7940ae9b0 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.029+0000 7fe798f3b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe7940b0e90 0x7fe79406b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37938/0 (socket says 172.21.15.27:37938) 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.029+0000 7fe798f3b640 1 -- 172.21.15.27:0/681842309 learned_addr learned my addr 172.21.15.27:0/681842309 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe799f3d640 1 --2- 172.21.15.27:0/681842309 >> v2:172.21.15.69:3300/0 conn(0x7fe7940ae9b0 0x7fe7940d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860543860 0 0) 0x7fe7940cee90 con 0x7fe7940b0e90 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe780003520 con 0x7fe7940b0e90 2024-03-06T07:31:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2625827658 0 0) 0x7fe780003520 con 0x7fe7940b0e90 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 >> v2:172.21.15.69:3300/0 conn(0x7fe7940ae9b0 msgr2=0x7fe7940d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 --2- 172.21.15.27:0/681842309 >> v2:172.21.15.69:3300/0 conn(0x7fe7940ae9b0 0x7fe7940d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 >> v1:172.21.15.69:6789/0 conn(0x7fe79406b5c0 legacy=0x7fe7940b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940cff30 con 0x7fe7940b0e90 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fe790001670 con 0x7fe7940b0e90 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 >> v1:172.21.15.27:6789/0 conn(0x7fe7940b0e90 legacy=0x7fe79406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7800039b0 0x7fe780003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe780004800 0x7fe780004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7800051c0 0x7fe78000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940cfa00 con 0x7fe780004800 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78000c160 con 0x7fe7800039b0 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78000c360 con 0x7fe7800051c0 2024-03-06T07:31:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe798f3b640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe780004800 0x7fe780004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe79973c640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7800039b0 0x7fe780003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.030+0000 7fe799f3d640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7800051c0 0x7fe78000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe798f3b640 1 -- 172.21.15.27:0/681842309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7800051c0 msgr2=0x7fe78000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe798f3b640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7800051c0 0x7fe78000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe798f3b640 1 -- 172.21.15.27:0/681842309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7800039b0 msgr2=0x7fe780003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe798f3b640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7800039b0 0x7fe780003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe798f3b640 1 -- 172.21.15.27:0/681842309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe780003520 con 0x7fe780004800 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe799f3d640 1 -- 172.21.15.27:0/681842309 reap_dead start 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe798f3b640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe780004800 0x7fe780004c80 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c210 tx=0x7fe780005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81346d522d1adfc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe790007d30 con 0x7fe780004800 2024-03-06T07:31:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe790030070 con 0x7fe780004800 2024-03-06T07:31:10.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.031+0000 7fe78bfff640 1 -- 172.21.15.27:0/681842309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe790005d70 con 0x7fe780004800 2024-03-06T07:31:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.032+0000 7fe79b9a0640 1 -- 172.21.15.27:0/681842309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe780004800 msgr2=0x7fe780004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.032+0000 7fe79b9a0640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe780004800 0x7fe780004c80 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c210 tx=0x7fe780005830 comp rx=0 tx=0).stop 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.032+0000 7fe79b9a0640 1 -- 172.21.15.27:0/681842309 shutdown_connections 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.032+0000 7fe79b9a0640 1 --2- 172.21.15.27:0/681842309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe780004800 0x7fe780004c80 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.032+0000 7fe79b9a0640 1 -- 172.21.15.27:0/681842309 >> 172.21.15.27:0/681842309 conn(0x7fe7940b36d0 msgr2=0x7fe7940b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.032+0000 7fe79b9a0640 1 -- 172.21.15.27:0/681842309 shutdown_connections 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.032+0000 7fe79b9a0640 1 -- 172.21.15.27:0/681842309 wait complete. 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.034+0000 7fe79b9a0640 1 Processor -- start 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.034+0000 7fe79b9a0640 1 -- start start 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe79b9a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe780004800 0x7fe7940c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe79b9a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7941dbf00 0x7fe7941e4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe79b9a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7941db440 0x7fe7940c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe79b9a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940aef40 con 0x7fe7941dbf00 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe79b9a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940aedc0 con 0x7fe7941db440 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe79b9a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940b2e00 con 0x7fe780004800 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe798f3b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7941dbf00 0x7fe7941e4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe798f3b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7941dbf00 0x7fe7941e4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55128/0 (socket says 172.21.15.27:55128) 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe798f3b640 1 -- 172.21.15.27:0/2194936995 learned_addr learned my addr 172.21.15.27:0/2194936995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe79973c640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe780004800 0x7fe7940c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe799f3d640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7941db440 0x7fe7940c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe798f3b640 1 -- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe780004800 msgr2=0x7fe7940c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe798f3b640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe780004800 0x7fe7940c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe798f3b640 1 -- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7941db440 msgr2=0x7fe7940c66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe798f3b640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7941db440 0x7fe7940c66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe798f3b640 1 -- 172.21.15.27:0/2194936995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940c5630 con 0x7fe7941dbf00 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe798f3b640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7941dbf00 0x7fe7941e4360 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c210 tx=0x7fe79002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe789ffb640 1 -- 172.21.15.27:0/2194936995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe790046070 con 0x7fe7941dbf00 2024-03-06T07:31:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe789ffb640 1 -- 172.21.15.27:0/2194936995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe790000ed0 con 0x7fe7941dbf00 2024-03-06T07:31:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7940c5910 con 0x7fe7941dbf00 2024-03-06T07:31:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.035+0000 7fe789ffb640 1 -- 172.21.15.27:0/2194936995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe790004e50 con 0x7fe7941dbf00 2024-03-06T07:31:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.036+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7940c5ed0 con 0x7fe7941dbf00 2024-03-06T07:31:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.037+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe794068700 con 0x7fe7941dbf00 2024-03-06T07:31:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.037+0000 7fe789ffb640 1 -- 172.21.15.27:0/2194936995 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe790007070 con 0x7fe7941dbf00 2024-03-06T07:31:10.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.038+0000 7fe789ffb640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77806b670 0x7fe77806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.038+0000 7fe789ffb640 1 -- 172.21.15.27:0/2194936995 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe7900ab580 con 0x7fe7941dbf00 2024-03-06T07:31:10.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.038+0000 7fe79973c640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77806b670 0x7fe77806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.039+0000 7fe79973c640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77806b670 0x7fe77806db30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe784000c00 tx=0x7fe784009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.042+0000 7fe789ffb640 1 -- 172.21.15.27:0/2194936995 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe790079d70 con 0x7fe7941dbf00 2024-03-06T07:31:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.417+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7940af960 con 0x7fe7941dbf00 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.417+0000 7fe789ffb640 1 -- 172.21.15.27:0/2194936995 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7fe79007de10 con 0x7fe7941dbf00 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77806b670 msgr2=0x7fe77806db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77806b670 0x7fe77806db30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe784000c00 tx=0x7fe784009040 comp rx=0 tx=0).stop 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7941dbf00 msgr2=0x7fe7941e4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7941dbf00 0x7fe7941e4360 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c210 tx=0x7fe79002e070 comp rx=0 tx=0).stop 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 shutdown_connections 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe77806b670 0x7fe77806db30 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7941db440 0x7fe7940c66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7941dbf00 0x7fe7941e4360 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 --2- 172.21.15.27:0/2194936995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe780004800 0x7fe7940c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 >> 172.21.15.27:0/2194936995 conn(0x7fe7940b36d0 msgr2=0x7fe7940b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 shutdown_connections 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:10.421+0000 7fe79b9a0640 1 -- 172.21.15.27:0/2194936995 wait complete. 2024-03-06T07:31:10.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:10.634 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:10.813 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:11.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:11.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.741+0000 7fd0064b5640 1 Processor -- start 2024-03-06T07:31:11.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.741+0000 7fd0064b5640 1 -- start start 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.742+0000 7fd0064b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd0000b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.742+0000 7fd0064b5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0000b9a10 0x7fd0000b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.742+0000 7fd0064b5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0000cea30 con 0x7fd00015df00 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.742+0000 7fd0064b5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0000cf3c0 con 0x7fd0000b9a10 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.742+0000 7fd0064b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000cf0e0 con 0x7fd0000b95a0 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fd004a52640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0000b9a10 0x7fd0000b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd0000b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fd004a52640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0000b9a10 0x7fd0000b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:59662/0 (socket says 172.21.15.27:59662) 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fcffffff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd00015df00 0x7fd0000d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:59134/0 (socket says 172.21.15.27:59134) 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fd004a52640 1 -- 172.21.15.27:0/2180248070 learned_addr learned my addr 172.21.15.27:0/2180248070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:11.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd0000b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:56522/0 (socket says 172.21.15.27:56522) 2024-03-06T07:31:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2180248070 >> v2:172.21.15.69:3300/0 conn(0x7fd0000b9a10 msgr2=0x7fd0000b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/2180248070 >> v2:172.21.15.69:3300/0 conn(0x7fd0000b9a10 0x7fd0000b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2180248070 >> v1:172.21.15.69:6789/0 conn(0x7fd00015df00 legacy=0x7fd0000d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2180248070 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0000ce470 con 0x7fd0000b95a0 2024-03-06T07:31:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.743+0000 7fd004a52640 1 --2- 172.21.15.27:0/2180248070 >> v2:172.21.15.69:3300/0 conn(0x7fd0000b9a10 0x7fd0000b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.744+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/2180248070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd0000b5700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcff00097a0 tx=0x7fcff002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23cf2d935e360d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.744+0000 7fcffeffd640 1 -- 172.21.15.27:0/2180248070 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0035020 con 0x7fd0000b95a0 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.744+0000 7fcffeffd640 1 -- 172.21.15.27:0/2180248070 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff0031830 con 0x7fd0000b95a0 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.744+0000 7fcffeffd640 1 -- 172.21.15.27:0/2180248070 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0031c00 con 0x7fd0000b95a0 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.744+0000 7fd0064b5640 1 -- 172.21.15.27:0/2180248070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 msgr2=0x7fd0000b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.744+0000 7fd0064b5640 1 --2- 172.21.15.27:0/2180248070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd0000b5700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcff00097a0 tx=0x7fcff002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.745+0000 7fd0064b5640 1 -- 172.21.15.27:0/2180248070 shutdown_connections 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.745+0000 7fd0064b5640 1 --2- 172.21.15.27:0/2180248070 >> v2:172.21.15.69:3300/0 conn(0x7fd0000b9a10 0x7fd0000b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.745+0000 7fd0064b5640 1 --2- 172.21.15.27:0/2180248070 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd0000b5700 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.745+0000 7fd0064b5640 1 -- 172.21.15.27:0/2180248070 >> 172.21.15.27:0/2180248070 conn(0x7fd000071960 msgr2=0x7fd000071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.745+0000 7fd0064b5640 1 -- 172.21.15.27:0/2180248070 shutdown_connections 2024-03-06T07:31:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.745+0000 7fd0064b5640 1 -- 172.21.15.27:0/2180248070 wait complete. 2024-03-06T07:31:11.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.746+0000 7fd0064b5640 1 Processor -- start 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.746+0000 7fd0064b5640 1 -- start start 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fd0064b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd000159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fd0064b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0000b9a10 0x7fd0001f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fd0064b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00015df00 0x7fd00015bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fd0064b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000cedf0 con 0x7fd0000b9a10 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fd0064b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000cec70 con 0x7fd00015df00 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fd0064b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000d3220 con 0x7fd0000b95a0 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0000b9a10 0x7fd0001f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0000b9a10 0x7fd0001f42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55132/0 (socket says 172.21.15.27:55132) 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2331718030 learned_addr learned my addr 172.21.15.27:0/2331718030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcffffff640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd000159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fd004a52640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00015df00 0x7fd00015bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 msgr2=0x7fd000159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd000159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00015df00 msgr2=0x7fd00015bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00015df00 0x7fd00015bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2331718030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00015a170 con 0x7fd0000b9a10 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.747+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0000b9a10 0x7fd0001f42f0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fcff002bee0 tx=0x7fcff00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.748+0000 7fcffcff9640 1 -- 172.21.15.27:0/2331718030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0041070 con 0x7fd0000b9a10 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.748+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00015a450 con 0x7fd0000b9a10 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.748+0000 7fcffcff9640 1 -- 172.21.15.27:0/2331718030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff003f070 con 0x7fd0000b9a10 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.748+0000 7fcffcff9640 1 -- 172.21.15.27:0/2331718030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0035040 con 0x7fd0000b9a10 2024-03-06T07:31:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.748+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00015aa10 con 0x7fd0000b9a10 2024-03-06T07:31:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.749+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd000067ff0 con 0x7fd0000b9a10 2024-03-06T07:31:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.749+0000 7fcffcff9640 1 -- 172.21.15.27:0/2331718030 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcff0038020 con 0x7fd0000b9a10 2024-03-06T07:31:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.749+0000 7fcffcff9640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.750+0000 7fcffcff9640 1 -- 172.21.15.27:0/2331718030 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcff00abac0 con 0x7fd0000b9a10 2024-03-06T07:31:11.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.750+0000 7fcffffff640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.750+0000 7fcffffff640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd00015fdf0 tx=0x7fcff4007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.754+0000 7fcffcff9640 1 -- 172.21.15.27:0/2331718030 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcff0031800 con 0x7fd0000b9a10 2024-03-06T07:31:11.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.976+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd00015ad20 con 0x7fd0000b9a10 2024-03-06T07:31:11.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.977+0000 7fcffcff9640 1 -- 172.21.15.27:0/2331718030 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7fcff007a230 con 0x7fd0000b9a10 2024-03-06T07:31:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:11.979 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 msgr2=0x7fcfec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd00015fdf0 tx=0x7fcff4007e80 comp rx=0 tx=0).stop 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0000b9a10 msgr2=0x7fd0001f42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0000b9a10 0x7fd0001f42f0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fcff002bee0 tx=0x7fcff00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 shutdown_connections 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfec06b4d0 0x7fcfec06d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd00015df00 0x7fd00015bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0000b9a10 0x7fd0001f42f0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 --2- 172.21.15.27:0/2331718030 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0000b95a0 0x7fd000159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 >> 172.21.15.27:0/2331718030 conn(0x7fd000071960 msgr2=0x7fd0000ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 shutdown_connections 2024-03-06T07:31:11.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:11.979+0000 7fd0064b5640 1 -- 172.21.15.27:0/2331718030 wait complete. 2024-03-06T07:31:12.193 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:12.193 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:12.815 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:13.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.300+0000 7fe16ce9a640 1 Processor -- start 2024-03-06T07:31:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.300+0000 7fe16ce9a640 1 -- start start 2024-03-06T07:31:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.301+0000 7fe16ce9a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1680b95a0 0x7fe1680b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.301+0000 7fe16ce9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 0x7fe1680b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.301+0000 7fe16ce9a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1680cea30 con 0x7fe16815df00 2024-03-06T07:31:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.301+0000 7fe16ce9a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1680cf3c0 con 0x7fe1680b95a0 2024-03-06T07:31:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.301+0000 7fe16ce9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680cf0e0 con 0x7fe1680b9a10 2024-03-06T07:31:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.301+0000 7fe165d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1680b95a0 0x7fe1680b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.301+0000 7fe16659c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe16815df00 0x7fe1680d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37942/0 (socket says 172.21.15.27:37942) 2024-03-06T07:31:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.301+0000 7fe16659c640 1 -- 172.21.15.27:0/3524121273 learned_addr learned my addr 172.21.15.27:0/3524121273 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.302+0000 7fe166d9d640 1 --2- 172.21.15.27:0/3524121273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 0x7fe1680b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.302+0000 7fe165d9b640 1 -- 172.21.15.27:0/3524121273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 msgr2=0x7fe1680b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.302+0000 7fe165d9b640 1 --2- 172.21.15.27:0/3524121273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 0x7fe1680b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.302+0000 7fe165d9b640 1 -- 172.21.15.27:0/3524121273 >> v1:172.21.15.27:6789/0 conn(0x7fe16815df00 legacy=0x7fe1680d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.302+0000 7fe16559a640 1 -- 172.21.15.27:0/3524121273 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266518080 0 0) 0x7fe1680cea30 con 0x7fe16815df00 2024-03-06T07:31:13.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.302+0000 7fe165d9b640 1 -- 172.21.15.27:0/3524121273 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680ce470 con 0x7fe1680b95a0 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.302+0000 7fe165d9b640 1 --2- 172.21.15.27:0/3524121273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 0x7fe1680b5700 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fe1580097a0 tx=0x7fe15802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53a4499f3f60db9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16559a640 1 -- 172.21.15.27:0/3524121273 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158035020 con 0x7fe1680b95a0 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16559a640 1 -- 172.21.15.27:0/3524121273 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe158031830 con 0x7fe1680b95a0 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16559a640 1 -- 172.21.15.27:0/3524121273 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158031c00 con 0x7fe1680b95a0 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16ce9a640 1 -- 172.21.15.27:0/3524121273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 msgr2=0x7fe1680b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16ce9a640 1 --2- 172.21.15.27:0/3524121273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 0x7fe1680b5700 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fe1580097a0 tx=0x7fe15802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16ce9a640 1 -- 172.21.15.27:0/3524121273 shutdown_connections 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16ce9a640 1 --2- 172.21.15.27:0/3524121273 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 0x7fe1680b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16ce9a640 1 --2- 172.21.15.27:0/3524121273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 0x7fe1680b5700 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16ce9a640 1 -- 172.21.15.27:0/3524121273 >> 172.21.15.27:0/3524121273 conn(0x7fe168071960 msgr2=0x7fe168071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16ce9a640 1 -- 172.21.15.27:0/3524121273 shutdown_connections 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.303+0000 7fe16ce9a640 1 -- 172.21.15.27:0/3524121273 wait complete. 2024-03-06T07:31:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.304+0000 7fe16ce9a640 1 Processor -- start 2024-03-06T07:31:13.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16ce9a640 1 -- start start 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16ce9a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 0x7fe1680b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16ce9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 0x7fe1680b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16ce9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16815df00 0x7fe16815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16ce9a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680cedf0 con 0x7fe1680b95a0 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16ce9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680cec70 con 0x7fe16815df00 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16ce9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680d3220 con 0x7fe1680b9a10 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 0x7fe1680b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 0x7fe1680b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55144/0 (socket says 172.21.15.27:55144) 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.305+0000 7fe16659c640 1 -- 172.21.15.27:0/1134859588 learned_addr learned my addr 172.21.15.27:0/1134859588 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe166d9d640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16815df00 0x7fe16815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe165d9b640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 0x7fe1680b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe16659c640 1 -- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 msgr2=0x7fe1680b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe16659c640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 0x7fe1680b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe16659c640 1 -- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16815df00 msgr2=0x7fe16815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe16659c640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16815df00 0x7fe16815e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe16659c640 1 -- 172.21.15.27:0/1134859588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1681f73c0 con 0x7fe1680b95a0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe166d9d640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16815df00 0x7fe16815e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe16659c640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 0x7fe1680b5640 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fe15000c430 tx=0x7fe15000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe15effd640 1 -- 172.21.15.27:0/1134859588 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1500074e0 con 0x7fe1680b95a0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe15effd640 1 -- 172.21.15.27:0/1134859588 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1500047e0 con 0x7fe1680b95a0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1681f76a0 con 0x7fe1680b95a0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe15effd640 1 -- 172.21.15.27:0/1134859588 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe150004bd0 con 0x7fe1680b95a0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.306+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16814d2d0 con 0x7fe1680b95a0 2024-03-06T07:31:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.307+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe168067ff0 con 0x7fe1680b95a0 2024-03-06T07:31:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.308+0000 7fe15effd640 1 -- 172.21.15.27:0/1134859588 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe150011070 con 0x7fe1680b95a0 2024-03-06T07:31:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.308+0000 7fe15effd640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe14806b4d0 0x7fe14806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.308+0000 7fe15effd640 1 -- 172.21.15.27:0/1134859588 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe150089e70 con 0x7fe1680b95a0 2024-03-06T07:31:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.309+0000 7fe165d9b640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe14806b4d0 0x7fe14806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.309+0000 7fe165d9b640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe14806b4d0 0x7fe14806d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe158036000 tx=0x7fe158008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.313+0000 7fe15effd640 1 -- 172.21.15.27:0/1134859588 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe150058d40 con 0x7fe1680b95a0 2024-03-06T07:31:13.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.534+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1680c7530 con 0x7fe1680b95a0 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.535+0000 7fe15effd640 1 -- 172.21.15.27:0/1134859588 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7fe150058640 con 0x7fe1680b95a0 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:13.537 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe14806b4d0 msgr2=0x7fe14806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe14806b4d0 0x7fe14806d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe158036000 tx=0x7fe158008030 comp rx=0 tx=0).stop 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 msgr2=0x7fe1680b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 0x7fe1680b5640 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fe15000c430 tx=0x7fe15000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 shutdown_connections 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe14806b4d0 0x7fe14806d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16815df00 0x7fe16815e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1680b9a10 0x7fe1680b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 --2- 172.21.15.27:0/1134859588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1680b95a0 0x7fe1680b5640 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 >> 172.21.15.27:0/1134859588 conn(0x7fe168071960 msgr2=0x7fe168071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 shutdown_connections 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:13.537+0000 7fe16ce9a640 1 -- 172.21.15.27:0/1134859588 wait complete. 2024-03-06T07:31:13.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:13.755 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:14.755 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:14.817 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.862+0000 7f0bfa55e640 1 Processor -- start 2024-03-06T07:31:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.862+0000 7f0bfa55e640 1 -- start start 2024-03-06T07:31:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.862+0000 7f0bfa55e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0bf40b2b50 0x7f0bf40b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.862+0000 7f0bfa55e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf40ce610 con 0x7f0bf40d3400 2024-03-06T07:31:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.862+0000 7f0bfa55e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf40d2920 con 0x7f0bf4071540 2024-03-06T07:31:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.862+0000 7f0bfa55e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf40d2f70 con 0x7f0bf40b2b50 2024-03-06T07:31:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.862+0000 7f0bf37fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0bf40d3400 0x7f0bf40b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60362/0 (socket says 172.21.15.27:60362) 2024-03-06T07:31:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.862+0000 7f0bf37fe640 1 -- 172.21.15.27:0/2362812562 learned_addr learned my addr 172.21.15.27:0/2362812562 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf8afb640 1 --2- 172.21.15.27:0/2362812562 >> v2:172.21.15.69:3300/0 conn(0x7f0bf40b2b50 0x7f0bf40b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320397047 0 0) 0x7f0bf40ce610 con 0x7f0bf40d3400 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bdc003520 con 0x7f0bf40d3400 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2843319774 0 0) 0x7f0bdc003520 con 0x7f0bf40d3400 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 >> v2:172.21.15.69:3300/0 conn(0x7f0bf40b2b50 msgr2=0x7f0bf40b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 --2- 172.21.15.27:0/2362812562 >> v2:172.21.15.69:3300/0 conn(0x7f0bf40b2b50 0x7f0bf40b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 >> v1:172.21.15.69:6789/0 conn(0x7f0bf4071540 legacy=0x7f0bf4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf40d2c00 con 0x7f0bf40d3400 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f0be8001670 con 0x7f0bf40d3400 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf8afb640 1 --2- 172.21.15.27:0/2362812562 >> v2:172.21.15.69:3300/0 conn(0x7f0bf40b2b50 0x7f0bf40b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 >> v1:172.21.15.27:6789/0 conn(0x7f0bf40d3400 legacy=0x7f0bf40b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc0039b0 0x7f0bdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bdc004800 0x7f0bdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bdc0051c0 0x7f0bdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40d2f70 con 0x7f0bdc0039b0 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc00c060 con 0x7f0bdc004800 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc00c260 con 0x7f0bdc0051c0 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf3fff640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc0039b0 0x7f0bdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf37fe640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bdc004800 0x7f0bdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.863+0000 7f0bf8afb640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bdc0051c0 0x7f0bdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.864+0000 7f0bf3fff640 1 -- 172.21.15.27:0/2362812562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bdc0051c0 msgr2=0x7f0bdc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.864+0000 7f0bf3fff640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bdc0051c0 0x7f0bdc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.864+0000 7f0bf3fff640 1 -- 172.21.15.27:0/2362812562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bdc004800 msgr2=0x7f0bdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.864+0000 7f0bf3fff640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bdc004800 0x7f0bdc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.864+0000 7f0bf3fff640 1 -- 172.21.15.27:0/2362812562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc003520 con 0x7f0bdc0039b0 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.864+0000 7f0bf8afb640 1 -- 172.21.15.27:0/2362812562 reap_dead start 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.864+0000 7f0bf3fff640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc0039b0 0x7f0bdc003dd0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f0be4001750 tx=0x7f0bdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f104817afd19a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4001370 con 0x7f0bdc0039b0 2024-03-06T07:31:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be40052d0 con 0x7f0bdc0039b0 2024-03-06T07:31:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bf2ffd640 1 -- 172.21.15.27:0/2362812562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4003830 con 0x7f0bdc0039b0 2024-03-06T07:31:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bfa55e640 1 -- 172.21.15.27:0/2362812562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc0039b0 msgr2=0x7f0bdc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bfa55e640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc0039b0 0x7f0bdc003dd0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f0be4001750 tx=0x7f0bdc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bfa55e640 1 -- 172.21.15.27:0/2362812562 shutdown_connections 2024-03-06T07:31:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bfa55e640 1 --2- 172.21.15.27:0/2362812562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc0039b0 0x7f0bdc003dd0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bfa55e640 1 -- 172.21.15.27:0/2362812562 >> 172.21.15.27:0/2362812562 conn(0x7f0bf40d3ca0 msgr2=0x7f0bf40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bfa55e640 1 -- 172.21.15.27:0/2362812562 shutdown_connections 2024-03-06T07:31:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.865+0000 7f0bfa55e640 1 -- 172.21.15.27:0/2362812562 wait complete. 2024-03-06T07:31:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.867+0000 7f0bfa55e640 1 Processor -- start 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.867+0000 7f0bfa55e640 1 -- start start 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bfa55e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bdc0039b0 0x7f0bf414d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bfa55e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40b5b20 0x7f0bf40aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bfa55e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415e680 0x7f0bf415eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bfa55e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf415e370 con 0x7f0bf415e680 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bfa55e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40d4be0 con 0x7f0bf40b5b20 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bfa55e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf414e860 con 0x7f0bdc0039b0 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf8afb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415e680 0x7f0bf415eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf8afb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415e680 0x7f0bf415eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36068/0 (socket says 172.21.15.27:36068) 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf8afb640 1 -- 172.21.15.27:0/1996442450 learned_addr learned my addr 172.21.15.27:0/1996442450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf3fff640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bdc0039b0 0x7f0bf414d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf37fe640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40b5b20 0x7f0bf40aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf8afb640 1 -- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bdc0039b0 msgr2=0x7f0bf414d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf8afb640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bdc0039b0 0x7f0bf414d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf8afb640 1 -- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40b5b20 msgr2=0x7f0bf40aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf8afb640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40b5b20 0x7f0bf40aa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf8afb640 1 -- 172.21.15.27:0/1996442450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf414d930 con 0x7f0bf415e680 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf8afb640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415e680 0x7f0bf415eb00 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f0bf408ca70 tx=0x7f0bec009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1996442450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec003990 con 0x7f0bf415e680 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1996442450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec003b30 con 0x7f0bf415e680 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1996442450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec004020 con 0x7f0bf415e680 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf414eab0 con 0x7f0bf415e680 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.868+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf414f070 con 0x7f0bf415e680 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.869+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf415f040 con 0x7f0bf415e680 2024-03-06T07:31:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.870+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1996442450 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bec0041c0 con 0x7f0bf415e680 2024-03-06T07:31:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.871+0000 7f0bf0ff9640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0be006b4d0 0x7f0be006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.871+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1996442450 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0bec08b450 con 0x7f0bf415e680 2024-03-06T07:31:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.871+0000 7f0bf3fff640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0be006b4d0 0x7f0be006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.871+0000 7f0bf3fff640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0be006b4d0 0x7f0be006d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0be4033000 tx=0x7f0be4009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:14.875+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1996442450 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bec019030 con 0x7f0bf415e680 2024-03-06T07:31:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.093+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bf4068700 con 0x7f0bf415e680 2024-03-06T07:31:15.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.094+0000 7f0bf0ff9640 1 -- 172.21.15.27:0/1996442450 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f0bec010040 con 0x7f0bf415e680 2024-03-06T07:31:15.095 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:15.095 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0be006b4d0 msgr2=0x7f0be006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0be006b4d0 0x7f0be006d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0be4033000 tx=0x7f0be4009030 comp rx=0 tx=0).stop 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415e680 msgr2=0x7f0bf415eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415e680 0x7f0bf415eb00 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f0bf408ca70 tx=0x7f0bec009f90 comp rx=0 tx=0).stop 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 shutdown_connections 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0be006b4d0 0x7f0be006d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bf415e680 0x7f0bf415eb00 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bf40b5b20 0x7f0bf40aa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 --2- 172.21.15.27:0/1996442450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0bdc0039b0 0x7f0bf414d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 >> 172.21.15.27:0/1996442450 conn(0x7f0bf40d3ca0 msgr2=0x7f0bf4160ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 shutdown_connections 2024-03-06T07:31:15.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:15.096+0000 7f0bfa55e640 1 -- 172.21.15.27:0/1996442450 wait complete. 2024-03-06T07:31:15.310 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:15.310 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:16.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:16.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.417+0000 7f6bd748a640 1 Processor -- start 2024-03-06T07:31:16.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.417+0000 7f6bd748a640 1 -- start start 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd748a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd015df00 0x7f6bd00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd748a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00b95a0 0x7f6bd00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd748a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6bd00b9a30 0x7f6bd00b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd748a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd00cea30 con 0x7f6bd015df00 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd748a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd00d2af0 con 0x7f6bd00b9a30 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd748a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00cf3c0 con 0x7f6bd00b95a0 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd5226640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd015df00 0x7f6bd00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd5226640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd015df00 0x7f6bd00d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36070/0 (socket says 172.21.15.27:36070) 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd5226640 1 -- 172.21.15.27:0/1068583181 learned_addr learned my addr 172.21.15.27:0/1068583181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:16.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd5a27640 1 --2- 172.21.15.27:0/1068583181 >> v2:172.21.15.69:3300/0 conn(0x7f6bd00b9a30 0x7f6bd00b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd4a25640 1 --2- 172.21.15.27:0/1068583181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00b95a0 0x7f6bd00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd5226640 1 -- 172.21.15.27:0/1068583181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00b95a0 msgr2=0x7f6bd00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd5226640 1 --2- 172.21.15.27:0/1068583181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00b95a0 0x7f6bd00b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd5226640 1 -- 172.21.15.27:0/1068583181 >> v2:172.21.15.69:3300/0 conn(0x7f6bd00b9a30 msgr2=0x7f6bd00b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd5226640 1 --2- 172.21.15.27:0/1068583181 >> v2:172.21.15.69:3300/0 conn(0x7f6bd00b9a30 0x7f6bd00b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.418+0000 7f6bd5226640 1 -- 172.21.15.27:0/1068583181 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd00cf0e0 con 0x7f6bd015df00 2024-03-06T07:31:16.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.419+0000 7f6bd5226640 1 --2- 172.21.15.27:0/1068583181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 0x7f6bd00d25b0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00d1160 tx=0x7f6bcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c388bbe37db4f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bc7fff640 1 -- 172.21.15.27:0/1068583181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc035020 con 0x7f6bd015df00 2024-03-06T07:31:16.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bc7fff640 1 -- 172.21.15.27:0/1068583181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc031870 con 0x7f6bd015df00 2024-03-06T07:31:16.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bc7fff640 1 -- 172.21.15.27:0/1068583181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc031c40 con 0x7f6bd015df00 2024-03-06T07:31:16.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bd748a640 1 -- 172.21.15.27:0/1068583181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 msgr2=0x7f6bd00d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bd748a640 1 --2- 172.21.15.27:0/1068583181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 0x7f6bd00d25b0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00d1160 tx=0x7f6bcc02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:16.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bd748a640 1 -- 172.21.15.27:0/1068583181 shutdown_connections 2024-03-06T07:31:16.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bd748a640 1 --2- 172.21.15.27:0/1068583181 >> v2:172.21.15.69:3300/0 conn(0x7f6bd00b9a30 0x7f6bd00b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bd748a640 1 --2- 172.21.15.27:0/1068583181 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00b95a0 0x7f6bd00b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bd748a640 1 --2- 172.21.15.27:0/1068583181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 0x7f6bd00d25b0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.420+0000 7f6bd748a640 1 -- 172.21.15.27:0/1068583181 >> 172.21.15.27:0/1068583181 conn(0x7f6bd0071960 msgr2=0x7f6bd0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.421+0000 7f6bd748a640 1 -- 172.21.15.27:0/1068583181 shutdown_connections 2024-03-06T07:31:16.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.421+0000 7f6bd748a640 1 -- 172.21.15.27:0/1068583181 wait complete. 2024-03-06T07:31:16.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.422+0000 7f6bd748a640 1 Processor -- start 2024-03-06T07:31:16.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.423+0000 7f6bd748a640 1 -- start start 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd748a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b95a0 0x7f6bd015fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd748a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 0x7f6bd00cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd748a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00cd200 0x7f6bd015e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd748a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00cec70 con 0x7f6bd015df00 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd748a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00ce750 con 0x7f6bd00b95a0 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd748a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00cedf0 con 0x7f6bd00cd200 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd4a25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 0x7f6bd00cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd4a25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 0x7f6bd00cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36084/0 (socket says 172.21.15.27:36084) 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd4a25640 1 -- 172.21.15.27:0/463094739 learned_addr learned my addr 172.21.15.27:0/463094739 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd5226640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b95a0 0x7f6bd015fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd5a27640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00cd200 0x7f6bd015e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd4a25640 1 -- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00cd200 msgr2=0x7f6bd015e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd4a25640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00cd200 0x7f6bd015e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd4a25640 1 -- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b95a0 msgr2=0x7f6bd015fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd4a25640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b95a0 0x7f6bd015fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd4a25640 1 -- 172.21.15.27:0/463094739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd01603e0 con 0x7f6bd015df00 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd5226640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b95a0 0x7f6bd015fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.424+0000 7f6bd4a25640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 0x7f6bd00cccc0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f6bc000e9b0 tx=0x7f6bc000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.425+0000 7f6bc5ffb640 1 -- 172.21.15.27:0/463094739 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0017070 con 0x7f6bd015df00 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.425+0000 7f6bc5ffb640 1 -- 172.21.15.27:0/463094739 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc000c9a0 con 0x7f6bd015df00 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.425+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd0160660 con 0x7f6bd015df00 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.425+0000 7f6bc5ffb640 1 -- 172.21.15.27:0/463094739 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc000cd90 con 0x7f6bd015df00 2024-03-06T07:31:16.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.425+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd0160c20 con 0x7f6bd015df00 2024-03-06T07:31:16.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.426+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd0067ff0 con 0x7f6bd015df00 2024-03-06T07:31:16.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.426+0000 7f6bc5ffb640 1 -- 172.21.15.27:0/463094739 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bc001a070 con 0x7f6bd015df00 2024-03-06T07:31:16.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.427+0000 7f6bc5ffb640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bb406b4d0 0x7f6bb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.427+0000 7f6bc5ffb640 1 -- 172.21.15.27:0/463094739 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6bc008a280 con 0x7f6bd015df00 2024-03-06T07:31:16.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.427+0000 7f6bd5226640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bb406b4d0 0x7f6bb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.428+0000 7f6bd5226640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bb406b4d0 0x7f6bb406d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036000 tx=0x7f6bcc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.431+0000 7f6bc5ffb640 1 -- 172.21.15.27:0/463094739 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bd01603e0 con 0x7f6bd015df00 2024-03-06T07:31:16.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.652+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bd015d450 con 0x7f6bd015df00 2024-03-06T07:31:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.652+0000 7f6bc5ffb640 1 -- 172.21.15.27:0/463094739 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f6bc00590c0 con 0x7f6bd015df00 2024-03-06T07:31:16.655 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:16.655 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bb406b4d0 msgr2=0x7f6bb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bb406b4d0 0x7f6bb406d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036000 tx=0x7f6bcc03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 msgr2=0x7f6bd00cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 0x7f6bd00cccc0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f6bc000e9b0 tx=0x7f6bc000ee70 comp rx=0 tx=0).stop 2024-03-06T07:31:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 shutdown_connections 2024-03-06T07:31:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6bb406b4d0 0x7f6bb406d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6bd00cd200 0x7f6bd015e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd015df00 0x7f6bd00cccc0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 --2- 172.21.15.27:0/463094739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bd00b95a0 0x7f6bd015fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 >> 172.21.15.27:0/463094739 conn(0x7f6bd0071960 msgr2=0x7f6bd00cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 shutdown_connections 2024-03-06T07:31:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:16.654+0000 7f6bd748a640 1 -- 172.21.15.27:0/463094739 wait complete. 2024-03-06T07:31:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:16.867 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:16.868 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:17.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.974+0000 7fa9532df640 1 Processor -- start 2024-03-06T07:31:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.975+0000 7fa9532df640 1 -- start start 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.975+0000 7fa9532df640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa94c06b5c0 0x7fa94c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.975+0000 7fa9532df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 0x7fa94c0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.975+0000 7fa9532df640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94c0b3340 con 0x7fa94c0b0e90 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.975+0000 7fa9532df640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa94c0cfa00 con 0x7fa94c06b5c0 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.975+0000 7fa9532df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c0cee90 con 0x7fa94c0ae9b0 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.976+0000 7fa95107b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa94c06b5c0 0x7fa94c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.976+0000 7fa95087a640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa94c0b0e90 0x7fa94c06b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60368/0 (socket says 172.21.15.27:60368) 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.976+0000 7fa95087a640 1 -- 172.21.15.27:0/3428913113 learned_addr learned my addr 172.21.15.27:0/3428913113 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.976+0000 7fa95187c640 1 --2- 172.21.15.27:0/3428913113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 0x7fa94c0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.976+0000 7fa95107b640 1 -- 172.21.15.27:0/3428913113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 msgr2=0x7fa94c0d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.976+0000 7fa95107b640 1 --2- 172.21.15.27:0/3428913113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 0x7fa94c0d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.976+0000 7fa95107b640 1 -- 172.21.15.27:0/3428913113 >> v1:172.21.15.27:6789/0 conn(0x7fa94c0b0e90 legacy=0x7fa94c06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.976+0000 7fa95107b640 1 -- 172.21.15.27:0/3428913113 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c0cfc50 con 0x7fa94c06b5c0 2024-03-06T07:31:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa95107b640 1 --2- 172.21.15.27:0/3428913113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c06b5c0 0x7fa94c0b2e00 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fa948009d90 tx=0x7fa94802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a569795ec99b27ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa943fff640 1 -- 172.21.15.27:0/3428913113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa948039070 con 0x7fa94c06b5c0 2024-03-06T07:31:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa943fff640 1 -- 172.21.15.27:0/3428913113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94802cbc0 con 0x7fa94c06b5c0 2024-03-06T07:31:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa943fff640 1 -- 172.21.15.27:0/3428913113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa948034440 con 0x7fa94c06b5c0 2024-03-06T07:31:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa9532df640 1 -- 172.21.15.27:0/3428913113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c06b5c0 msgr2=0x7fa94c0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa9532df640 1 --2- 172.21.15.27:0/3428913113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c06b5c0 0x7fa94c0b2e00 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fa948009d90 tx=0x7fa94802be10 comp rx=0 tx=0).stop 2024-03-06T07:31:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa9532df640 1 -- 172.21.15.27:0/3428913113 shutdown_connections 2024-03-06T07:31:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa9532df640 1 --2- 172.21.15.27:0/3428913113 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 0x7fa94c0d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa9532df640 1 --2- 172.21.15.27:0/3428913113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c06b5c0 0x7fa94c0b2e00 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa9532df640 1 -- 172.21.15.27:0/3428913113 >> 172.21.15.27:0/3428913113 conn(0x7fa94c0b36d0 msgr2=0x7fa94c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.977+0000 7fa9532df640 1 -- 172.21.15.27:0/3428913113 shutdown_connections 2024-03-06T07:31:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.978+0000 7fa9532df640 1 -- 172.21.15.27:0/3428913113 wait complete. 2024-03-06T07:31:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.979+0000 7fa9532df640 1 Processor -- start 2024-03-06T07:31:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.979+0000 7fa9532df640 1 -- start start 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa9532df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa94c06b5c0 0x7fa94c0cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa9532df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 0x7fa94c1d63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa9532df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c0b0e90 0x7fa94c0cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa9532df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c06c910 con 0x7fa94c0b0e90 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa9532df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c06c790 con 0x7fa94c06b5c0 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa9532df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c0aee00 con 0x7fa94c0ae9b0 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa95187c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c0b0e90 0x7fa94c0cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa95187c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c0b0e90 0x7fa94c0cc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36094/0 (socket says 172.21.15.27:36094) 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa95187c640 1 -- 172.21.15.27:0/2461369269 learned_addr learned my addr 172.21.15.27:0/2461369269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa95087a640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 0x7fa94c1d63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.980+0000 7fa95107b640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa94c06b5c0 0x7fa94c0cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa95187c640 1 -- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 msgr2=0x7fa94c1d63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa95187c640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 0x7fa94c1d63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa95187c640 1 -- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa94c06b5c0 msgr2=0x7fa94c0cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa95187c640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa94c06b5c0 0x7fa94c0cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa95187c640 1 -- 172.21.15.27:0/2461369269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c1d81e0 con 0x7fa94c0b0e90 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa95187c640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c0b0e90 0x7fa94c0cc940 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fa94400d830 tx=0x7fa94400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa941ffb640 1 -- 172.21.15.27:0/2461369269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94400b6c0 con 0x7fa94c0b0e90 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa941ffb640 1 -- 172.21.15.27:0/2461369269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa944019070 con 0x7fa94c0b0e90 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c1d84c0 con 0x7fa94c0b0e90 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa941ffb640 1 -- 172.21.15.27:0/2461369269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94400bac0 con 0x7fa94c0b0e90 2024-03-06T07:31:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.981+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c1d8a80 con 0x7fa94c0b0e90 2024-03-06T07:31:17.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.982+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94c068700 con 0x7fa94c0b0e90 2024-03-06T07:31:17.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.983+0000 7fa941ffb640 1 -- 172.21.15.27:0/2461369269 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa94401e460 con 0x7fa94c0b0e90 2024-03-06T07:31:17.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.983+0000 7fa941ffb640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa93006b4d0 0x7fa93006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.984+0000 7fa941ffb640 1 -- 172.21.15.27:0/2461369269 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa94408a7a0 con 0x7fa94c0b0e90 2024-03-06T07:31:17.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.984+0000 7fa95107b640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa93006b4d0 0x7fa93006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.984+0000 7fa95107b640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa93006b4d0 0x7fa93006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa948036000 tx=0x7fa948030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:17.988+0000 7fa941ffb640 1 -- 172.21.15.27:0/2461369269 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa944058f90 con 0x7fa94c0b0e90 2024-03-06T07:31:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.209+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa94c06f830 con 0x7fa94c0b0e90 2024-03-06T07:31:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.210+0000 7fa941ffb640 1 -- 172.21.15.27:0/2461369269 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7fa94405d030 con 0x7fa94c0b0e90 2024-03-06T07:31:18.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:18.211 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa93006b4d0 msgr2=0x7fa93006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa93006b4d0 0x7fa93006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa948036000 tx=0x7fa948030000 comp rx=0 tx=0).stop 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c0b0e90 msgr2=0x7fa94c0cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c0b0e90 0x7fa94c0cc940 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fa94400d830 tx=0x7fa94400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 shutdown_connections 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa93006b4d0 0x7fa93006d990 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa94c0b0e90 0x7fa94c0cc940 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa94c0ae9b0 0x7fa94c1d63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 --2- 172.21.15.27:0/2461369269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa94c06b5c0 0x7fa94c0cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 >> 172.21.15.27:0/2461369269 conn(0x7fa94c0b36d0 msgr2=0x7fa94c0af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 shutdown_connections 2024-03-06T07:31:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:18.212+0000 7fa9532df640 1 -- 172.21.15.27:0/2461369269 wait complete. 2024-03-06T07:31:18.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:18.425 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:18.821 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:19.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.532+0000 7f2f79fd7640 1 Processor -- start 2024-03-06T07:31:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.532+0000 7f2f79fd7640 1 -- start start 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.533+0000 7f2f79fd7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2f740b95a0 0x7f2f740b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.533+0000 7f2f79fd7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f740cea30 con 0x7f2f7415df00 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.533+0000 7f2f79fd7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f740cf3c0 con 0x7f2f740b9a10 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.533+0000 7f2f79fd7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f740cf0e0 con 0x7f2f740b95a0 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.533+0000 7f2f737fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2f7415df00 0x7f2f740d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60374/0 (socket says 172.21.15.27:60374) 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.533+0000 7f2f737fe640 1 -- 172.21.15.27:0/2691857526 learned_addr learned my addr 172.21.15.27:0/2691857526 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.533+0000 7f2f72ffd640 1 --2- 172.21.15.27:0/2691857526 >> v2:172.21.15.69:3300/0 conn(0x7f2f740b95a0 0x7f2f740b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.533+0000 7f2f727fc640 1 -- 172.21.15.27:0/2691857526 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806891030 0 0) 0x7f2f740cea30 con 0x7f2f7415df00 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.533+0000 7f2f727fc640 1 -- 172.21.15.27:0/2691857526 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c003520 con 0x7f2f7415df00 2024-03-06T07:31:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.534+0000 7f2f72ffd640 1 -- 172.21.15.27:0/2691857526 >> v1:172.21.15.69:6789/0 conn(0x7f2f740b9a10 legacy=0x7f2f740d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.534+0000 7f2f72ffd640 1 -- 172.21.15.27:0/2691857526 >> v1:172.21.15.27:6789/0 conn(0x7f2f7415df00 legacy=0x7f2f740d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.534+0000 7f2f72ffd640 1 -- 172.21.15.27:0/2691857526 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f740ce470 con 0x7f2f740b95a0 2024-03-06T07:31:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.534+0000 7f2f72ffd640 1 --2- 172.21.15.27:0/2691857526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f740b95a0 0x7f2f740b5700 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2f740cf590 tx=0x7f2f6402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2aefc21119e443 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f727fc640 1 -- 172.21.15.27:0/2691857526 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64034070 con 0x7f2f740b95a0 2024-03-06T07:31:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f727fc640 1 -- 172.21.15.27:0/2691857526 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64030800 con 0x7f2f740b95a0 2024-03-06T07:31:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f727fc640 1 -- 172.21.15.27:0/2691857526 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64030bb0 con 0x7f2f740b95a0 2024-03-06T07:31:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f79fd7640 1 -- 172.21.15.27:0/2691857526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f740b95a0 msgr2=0x7f2f740b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f79fd7640 1 --2- 172.21.15.27:0/2691857526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f740b95a0 0x7f2f740b5700 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2f740cf590 tx=0x7f2f6402b940 comp rx=0 tx=0).stop 2024-03-06T07:31:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f79fd7640 1 -- 172.21.15.27:0/2691857526 shutdown_connections 2024-03-06T07:31:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f79fd7640 1 --2- 172.21.15.27:0/2691857526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f740b95a0 0x7f2f740b5700 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f79fd7640 1 -- 172.21.15.27:0/2691857526 >> 172.21.15.27:0/2691857526 conn(0x7f2f74071960 msgr2=0x7f2f74071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f79fd7640 1 -- 172.21.15.27:0/2691857526 shutdown_connections 2024-03-06T07:31:19.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.535+0000 7f2f79fd7640 1 -- 172.21.15.27:0/2691857526 wait complete. 2024-03-06T07:31:19.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.536+0000 7f2f79fd7640 1 Processor -- start 2024-03-06T07:31:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.537+0000 7f2f79fd7640 1 -- start start 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.537+0000 7f2f79fd7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f740b95a0 0x7f2f740c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.537+0000 7f2f79fd7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f740b9a10 0x7f2f741605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.537+0000 7f2f79fd7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f7415df00 0x7f2f740ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.537+0000 7f2f79fd7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740cedf0 con 0x7f2f740b9a10 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.537+0000 7f2f79fd7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740cec70 con 0x7f2f740b95a0 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.537+0000 7f2f79fd7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740b5c40 con 0x7f2f7415df00 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.537+0000 7f2f72ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f740b9a10 0x7f2f741605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.537+0000 7f2f72ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f740b9a10 0x7f2f741605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36106/0 (socket says 172.21.15.27:36106) 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f72ffd640 1 -- 172.21.15.27:0/4085104891 learned_addr learned my addr 172.21.15.27:0/4085104891 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f737fe640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f740b95a0 0x7f2f740c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f73fff640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f7415df00 0x7f2f740ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f72ffd640 1 -- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f7415df00 msgr2=0x7f2f740ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f72ffd640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f7415df00 0x7f2f740ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f72ffd640 1 -- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f740b95a0 msgr2=0x7f2f740c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f72ffd640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f740b95a0 0x7f2f740c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f72ffd640 1 -- 172.21.15.27:0/4085104891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f740c7d90 con 0x7f2f740b9a10 2024-03-06T07:31:19.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f73fff640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f7415df00 0x7f2f740ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:19.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f72ffd640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f740b9a10 0x7f2f741605a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009890 tx=0x7f2f640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f5bfff640 1 -- 172.21.15.27:0/4085104891 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64048070 con 0x7f2f740b9a10 2024-03-06T07:31:19.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f5bfff640 1 -- 172.21.15.27:0/4085104891 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64030600 con 0x7f2f740b9a10 2024-03-06T07:31:19.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f5bfff640 1 -- 172.21.15.27:0/4085104891 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64034070 con 0x7f2f740b9a10 2024-03-06T07:31:19.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f740c8070 con 0x7f2f740b9a10 2024-03-06T07:31:19.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.538+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f740cbda0 con 0x7f2f740b9a10 2024-03-06T07:31:19.543 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.539+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f74067ff0 con 0x7f2f740b9a10 2024-03-06T07:31:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.540+0000 7f2f5bfff640 1 -- 172.21.15.27:0/4085104891 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f6404b070 con 0x7f2f740b9a10 2024-03-06T07:31:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.540+0000 7f2f5bfff640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f6006b500 0x7f2f6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.541+0000 7f2f5bfff640 1 -- 172.21.15.27:0/4085104891 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2f640abd80 con 0x7f2f740b9a10 2024-03-06T07:31:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.541+0000 7f2f737fe640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f6006b500 0x7f2f6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.541+0000 7f2f737fe640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f6006b500 0x7f2f6006d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2f680011e0 tx=0x7f2f68007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.545+0000 7f2f5bfff640 1 -- 172.21.15.27:0/4085104891 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f6407a690 con 0x7f2f740b9a10 2024-03-06T07:31:19.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.764+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f740b8ed0 con 0x7f2f740b9a10 2024-03-06T07:31:19.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.765+0000 7f2f5bfff640 1 -- 172.21.15.27:0/4085104891 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f2f6407e730 con 0x7f2f740b9a10 2024-03-06T07:31:19.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:19.766 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:19.767 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f6006b500 msgr2=0x7f2f6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f6006b500 0x7f2f6006d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2f680011e0 tx=0x7f2f68007d30 comp rx=0 tx=0).stop 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f740b9a10 msgr2=0x7f2f741605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f740b9a10 0x7f2f741605a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009890 tx=0x7f2f640046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 shutdown_connections 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2f6006b500 0x7f2f6006d9c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2f7415df00 0x7f2f740ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f740b9a10 0x7f2f741605a0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 --2- 172.21.15.27:0/4085104891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f740b95a0 0x7f2f740c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 >> 172.21.15.27:0/4085104891 conn(0x7f2f74071960 msgr2=0x7f2f740b7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 shutdown_connections 2024-03-06T07:31:19.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:19.767+0000 7f2f79fd7640 1 -- 172.21.15.27:0/4085104891 wait complete. 2024-03-06T07:31:19.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:19.981 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:20.823 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:20.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:21.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.087+0000 7f566a11e640 1 Processor -- start 2024-03-06T07:31:21.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.087+0000 7f566a11e640 1 -- start start 2024-03-06T07:31:21.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.088+0000 7f566a11e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f56640aee10 0x7f566415cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.088+0000 7f566a11e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56640729b0 con 0x7f566415bac0 2024-03-06T07:31:21.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.088+0000 7f566a11e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566406c920 con 0x7f566406d400 2024-03-06T07:31:21.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.088+0000 7f566a11e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566406cf70 con 0x7f56640aee10 2024-03-06T07:31:21.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.088+0000 7f56637fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f566415bac0 0x7f56640ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60378/0 (socket says 172.21.15.27:60378) 2024-03-06T07:31:21.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.088+0000 7f56637fe640 1 -- 172.21.15.27:0/170124604 learned_addr learned my addr 172.21.15.27:0/170124604 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:21.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.088+0000 7f5662ffd640 1 --2- 172.21.15.27:0/170124604 >> v2:172.21.15.69:3300/0 conn(0x7f56640aee10 0x7f566415cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.089+0000 7f56627fc640 1 -- 172.21.15.27:0/170124604 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224221909 0 0) 0x7f56640729b0 con 0x7f566415bac0 2024-03-06T07:31:21.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.089+0000 7f56627fc640 1 -- 172.21.15.27:0/170124604 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f564c003520 con 0x7f566415bac0 2024-03-06T07:31:21.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.089+0000 7f5662ffd640 1 -- 172.21.15.27:0/170124604 >> v1:172.21.15.69:6789/0 conn(0x7f566406d400 legacy=0x7f5664070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.089+0000 7f5662ffd640 1 -- 172.21.15.27:0/170124604 >> v1:172.21.15.27:6789/0 conn(0x7f566415bac0 legacy=0x7f56640ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.089+0000 7f5662ffd640 1 -- 172.21.15.27:0/170124604 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566406cc00 con 0x7f56640aee10 2024-03-06T07:31:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.090+0000 7f5662ffd640 1 --2- 172.21.15.27:0/170124604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56640aee10 0x7f566415cbe0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f56580097a0 tx=0x7f565802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35fdb9f3f1805736 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.090+0000 7f56627fc640 1 -- 172.21.15.27:0/170124604 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658034070 con 0x7f56640aee10 2024-03-06T07:31:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.090+0000 7f56627fc640 1 -- 172.21.15.27:0/170124604 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5658030610 con 0x7f56640aee10 2024-03-06T07:31:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.090+0000 7f56627fc640 1 -- 172.21.15.27:0/170124604 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56580309e0 con 0x7f56640aee10 2024-03-06T07:31:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.090+0000 7f566a11e640 1 -- 172.21.15.27:0/170124604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56640aee10 msgr2=0x7f566415cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.090+0000 7f566a11e640 1 --2- 172.21.15.27:0/170124604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56640aee10 0x7f566415cbe0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f56580097a0 tx=0x7f565802b960 comp rx=0 tx=0).stop 2024-03-06T07:31:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.090+0000 7f566a11e640 1 -- 172.21.15.27:0/170124604 shutdown_connections 2024-03-06T07:31:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.090+0000 7f566a11e640 1 --2- 172.21.15.27:0/170124604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56640aee10 0x7f566415cbe0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.090+0000 7f566a11e640 1 -- 172.21.15.27:0/170124604 >> 172.21.15.27:0/170124604 conn(0x7f5664066d00 msgr2=0x7f56640670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.091+0000 7f566a11e640 1 -- 172.21.15.27:0/170124604 shutdown_connections 2024-03-06T07:31:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.091+0000 7f566a11e640 1 -- 172.21.15.27:0/170124604 wait complete. 2024-03-06T07:31:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.092+0000 7f566a11e640 1 Processor -- start 2024-03-06T07:31:21.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.093+0000 7f566a11e640 1 -- start start 2024-03-06T07:31:21.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f566a11e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f566406d400 0x7f5664154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f566a11e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56640aee10 0x7f5664152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f566a11e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566415bac0 0x7f5664159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f566a11e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566415c020 con 0x7f566415bac0 2024-03-06T07:31:21.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f566a11e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566415bea0 con 0x7f56640aee10 2024-03-06T07:31:21.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f566a11e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5664072580 con 0x7f566406d400 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f5663fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566415bac0 0x7f5664159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f5663fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566415bac0 0x7f5664159750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36118/0 (socket says 172.21.15.27:36118) 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f5663fff640 1 -- 172.21.15.27:0/4095388539 learned_addr learned my addr 172.21.15.27:0/4095388539 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f5662ffd640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56640aee10 0x7f5664152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.094+0000 7f56637fe640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f566406d400 0x7f5664154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f5663fff640 1 -- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f566406d400 msgr2=0x7f5664154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f5663fff640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f566406d400 0x7f5664154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f5663fff640 1 -- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56640aee10 msgr2=0x7f5664152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f5663fff640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56640aee10 0x7f5664152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f5663fff640 1 -- 172.21.15.27:0/4095388539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5664159c90 con 0x7f566415bac0 2024-03-06T07:31:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f5663fff640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566415bac0 0x7f5664159750 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f565c00c7a0 tx=0x7f565c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f564bfff640 1 -- 172.21.15.27:0/4095388539 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c003b70 con 0x7f566415bac0 2024-03-06T07:31:21.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f564bfff640 1 -- 172.21.15.27:0/4095388539 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c003d30 con 0x7f566415bac0 2024-03-06T07:31:21.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5664154d10 con 0x7f566415bac0 2024-03-06T07:31:21.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f564bfff640 1 -- 172.21.15.27:0/4095388539 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c00aa90 con 0x7f566415bac0 2024-03-06T07:31:21.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.095+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56641552d0 con 0x7f566415bac0 2024-03-06T07:31:21.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.096+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5664067ff0 con 0x7f566415bac0 2024-03-06T07:31:21.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.097+0000 7f564bfff640 1 -- 172.21.15.27:0/4095388539 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f565c00ac30 con 0x7f566415bac0 2024-03-06T07:31:21.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.097+0000 7f564bfff640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f565006b4d0 0x7f565006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.097+0000 7f564bfff640 1 -- 172.21.15.27:0/4095388539 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f565c08a040 con 0x7f566415bac0 2024-03-06T07:31:21.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.097+0000 7f56637fe640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f565006b4d0 0x7f565006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.098+0000 7f56637fe640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f565006b4d0 0x7f565006d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f56540011e0 tx=0x7f5654007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.106+0000 7f564bfff640 1 -- 172.21.15.27:0/4095388539 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f565c058830 con 0x7f566415bac0 2024-03-06T07:31:21.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.406+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f566406de50 con 0x7f566415bac0 2024-03-06T07:31:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.407+0000 7f564bfff640 1 -- 172.21.15.27:0/4095388539 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f565c05c8d0 con 0x7f566415bac0 2024-03-06T07:31:21.410 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:21.411 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f565006b4d0 msgr2=0x7f565006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f565006b4d0 0x7f565006d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f56540011e0 tx=0x7f5654007d30 comp rx=0 tx=0).stop 2024-03-06T07:31:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566415bac0 msgr2=0x7f5664159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566415bac0 0x7f5664159750 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f565c00c7a0 tx=0x7f565c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 shutdown_connections 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f565006b4d0 0x7f565006d990 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566415bac0 0x7f5664159750 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56640aee10 0x7f5664152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 --2- 172.21.15.27:0/4095388539 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f566406d400 0x7f5664154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 >> 172.21.15.27:0/4095388539 conn(0x7f5664066d00 msgr2=0x7f56640ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 shutdown_connections 2024-03-06T07:31:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:21.410+0000 7f566a11e640 1 -- 172.21.15.27:0/4095388539 wait complete. 2024-03-06T07:31:21.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:21.625 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:22.627 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.732+0000 7f290654b640 1 Processor -- start 2024-03-06T07:31:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.732+0000 7f290654b640 1 -- start start 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f290654b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2900071540 0x7f29000b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f290654b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f29000b2b50 0x7f29000b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f290654b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000ce610 con 0x7f29000d3400 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f290654b640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29000d2f70 con 0x7f2900071540 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f290654b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29000d4be0 con 0x7f29000b2b50 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f28fffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2900071540 0x7f29000b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f28fffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2900071540 0x7f29000b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36128/0 (socket says 172.21.15.27:36128) 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f28fffff640 1 -- 172.21.15.27:0/2572961671 learned_addr learned my addr 172.21.15.27:0/2572961671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f2904ae8640 1 --2- 172.21.15.27:0/2572961671 >> v2:172.21.15.69:3300/0 conn(0x7f29000b2b50 0x7f29000b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f28fffff640 1 -- 172.21.15.27:0/2572961671 >> v2:172.21.15.69:3300/0 conn(0x7f29000b2b50 msgr2=0x7f29000b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f28fffff640 1 --2- 172.21.15.27:0/2572961671 >> v2:172.21.15.69:3300/0 conn(0x7f29000b2b50 0x7f29000b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f28fffff640 1 -- 172.21.15.27:0/2572961671 >> v1:172.21.15.69:6789/0 conn(0x7f29000d3400 legacy=0x7f2900071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.733+0000 7f28fffff640 1 -- 172.21.15.27:0/2572961671 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000d2920 con 0x7f2900071540 2024-03-06T07:31:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.734+0000 7f28fffff640 1 --2- 172.21.15.27:0/2572961671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2900071540 0x7f29000b30a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f29000683b0 tx=0x7f28f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7e7a022b01fddc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.734+0000 7f28feffd640 1 -- 172.21.15.27:0/2572961671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0034070 con 0x7f2900071540 2024-03-06T07:31:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.734+0000 7f28feffd640 1 -- 172.21.15.27:0/2572961671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f0030860 con 0x7f2900071540 2024-03-06T07:31:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.734+0000 7f28feffd640 1 -- 172.21.15.27:0/2572961671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0030c30 con 0x7f2900071540 2024-03-06T07:31:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.735+0000 7f290654b640 1 -- 172.21.15.27:0/2572961671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2900071540 msgr2=0x7f29000b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.735+0000 7f290654b640 1 --2- 172.21.15.27:0/2572961671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2900071540 0x7f29000b30a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f29000683b0 tx=0x7f28f002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.735+0000 7f290654b640 1 -- 172.21.15.27:0/2572961671 shutdown_connections 2024-03-06T07:31:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.735+0000 7f290654b640 1 --2- 172.21.15.27:0/2572961671 >> v2:172.21.15.69:3300/0 conn(0x7f29000b2b50 0x7f29000b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.735+0000 7f290654b640 1 --2- 172.21.15.27:0/2572961671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2900071540 0x7f29000b30a0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.735+0000 7f290654b640 1 -- 172.21.15.27:0/2572961671 >> 172.21.15.27:0/2572961671 conn(0x7f29000d3ca0 msgr2=0x7f29000d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.735+0000 7f290654b640 1 -- 172.21.15.27:0/2572961671 shutdown_connections 2024-03-06T07:31:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.735+0000 7f290654b640 1 -- 172.21.15.27:0/2572961671 wait complete. 2024-03-06T07:31:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.736+0000 7f290654b640 1 Processor -- start 2024-03-06T07:31:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.736+0000 7f290654b640 1 -- start start 2024-03-06T07:31:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f290654b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900071540 0x7f29000cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f290654b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29000b2b50 0x7f29001e7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f290654b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29000d3400 0x7f29001f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f290654b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290015e370 con 0x7f29000b2b50 2024-03-06T07:31:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f290654b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000ce8f0 con 0x7f2900071540 2024-03-06T07:31:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f290654b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29000cc230 con 0x7f29000d3400 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29000b2b50 0x7f29001e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29000b2b50 0x7f29001e7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36138/0 (socket says 172.21.15.27:36138) 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28ff7fe640 1 -- 172.21.15.27:0/2640256434 learned_addr learned my addr 172.21.15.27:0/2640256434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28fffff640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900071540 0x7f29000cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f2904ae8640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29000d3400 0x7f29001f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28ff7fe640 1 -- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29000d3400 msgr2=0x7f29001f3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28ff7fe640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29000d3400 0x7f29001f3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28ff7fe640 1 -- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900071540 msgr2=0x7f29000cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28ff7fe640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900071540 0x7f29000cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28ff7fe640 1 -- 172.21.15.27:0/2640256434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000cc460 con 0x7f29000b2b50 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28ff7fe640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29000b2b50 0x7f29001e7030 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f28f400c440 tx=0x7f28f400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28fcff9640 1 -- 172.21.15.27:0/2640256434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f40052d0 con 0x7f29000b2b50 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28fcff9640 1 -- 172.21.15.27:0/2640256434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4005470 con 0x7f29000b2b50 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29000cc740 con 0x7f29000b2b50 2024-03-06T07:31:22.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.737+0000 7f28fcff9640 1 -- 172.21.15.27:0/2640256434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4007260 con 0x7f29000b2b50 2024-03-06T07:31:22.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.738+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29000ccd00 con 0x7f29000b2b50 2024-03-06T07:31:22.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.738+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2900068700 con 0x7f29000b2b50 2024-03-06T07:31:22.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.739+0000 7f28fcff9640 1 -- 172.21.15.27:0/2640256434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28f40074c0 con 0x7f29000b2b50 2024-03-06T07:31:22.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.739+0000 7f28fcff9640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28ec06b4d0 0x7f28ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.739+0000 7f28fcff9640 1 -- 172.21.15.27:0/2640256434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f28f4089550 con 0x7f29000b2b50 2024-03-06T07:31:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.740+0000 7f28fffff640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28ec06b4d0 0x7f28ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.740+0000 7f28fffff640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28ec06b4d0 0x7f28ec06d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f28f0036000 tx=0x7f28f0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.744+0000 7f28fcff9640 1 -- 172.21.15.27:0/2640256434 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28f4057dd0 con 0x7f29000b2b50 2024-03-06T07:31:22.825 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.982+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2900071a30 con 0x7f29000b2b50 2024-03-06T07:31:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.983+0000 7f28fcff9640 1 -- 172.21.15.27:0/2640256434 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f28f405be70 con 0x7f29000b2b50 2024-03-06T07:31:22.985 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:22.985 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.984+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28ec06b4d0 msgr2=0x7f28ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.984+0000 7f290654b640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28ec06b4d0 0x7f28ec06d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f28f0036000 tx=0x7f28f0008030 comp rx=0 tx=0).stop 2024-03-06T07:31:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.984+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29000b2b50 msgr2=0x7f29001e7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.984+0000 7f290654b640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29000b2b50 0x7f29001e7030 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f28f400c440 tx=0x7f28f400c900 comp rx=0 tx=0).stop 2024-03-06T07:31:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.985+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 shutdown_connections 2024-03-06T07:31:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.985+0000 7f290654b640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f28ec06b4d0 0x7f28ec06d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.985+0000 7f290654b640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f29000d3400 0x7f29001f3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.985+0000 7f290654b640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29000b2b50 0x7f29001e7030 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.985+0000 7f290654b640 1 --2- 172.21.15.27:0/2640256434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900071540 0x7f29000cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.985+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 >> 172.21.15.27:0/2640256434 conn(0x7f29000d3ca0 msgr2=0x7f290015ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.985+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 shutdown_connections 2024-03-06T07:31:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:22.985+0000 7f290654b640 1 -- 172.21.15.27:0/2640256434 wait complete. 2024-03-06T07:31:23.199 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:23.199 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:24.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:24.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.311+0000 7f4870839640 1 Processor -- start 2024-03-06T07:31:24.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.311+0000 7f4870839640 1 -- start start 2024-03-06T07:31:24.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.312+0000 7f4870839640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b95a0 0x7f486c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.312+0000 7f4870839640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f486c0b9a10 0x7f486c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.312+0000 7f4870839640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486c0cea30 con 0x7f486c15df00 2024-03-06T07:31:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.312+0000 7f4870839640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f486c0cf3c0 con 0x7f486c0b9a10 2024-03-06T07:31:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.312+0000 7f4870839640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486c0cf0e0 con 0x7f486c0b95a0 2024-03-06T07:31:24.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f486ad9d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f486c0b9a10 0x7f486c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f4869d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b95a0 0x7f486c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f486a59c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f486c15df00 0x7f486c0d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:44450/0 (socket says 172.21.15.27:44450) 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f486a59c640 1 -- 172.21.15.27:0/1951066055 learned_addr learned my addr 172.21.15.27:0/1951066055 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f486959a640 1 -- 172.21.15.27:0/1951066055 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664386154 0 0) 0x7f486c0cea30 con 0x7f486c15df00 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f486959a640 1 -- 172.21.15.27:0/1951066055 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4858003520 con 0x7f486c15df00 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f486ad9d640 1 -- 172.21.15.27:0/1951066055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b95a0 msgr2=0x7f486c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f486ad9d640 1 --2- 172.21.15.27:0/1951066055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b95a0 0x7f486c0b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f486ad9d640 1 -- 172.21.15.27:0/1951066055 >> v1:172.21.15.69:6789/0 conn(0x7f486c15df00 legacy=0x7f486c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f486ad9d640 1 -- 172.21.15.27:0/1951066055 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c0ce470 con 0x7f486c0b9a10 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.313+0000 7f4869d9b640 1 --2- 172.21.15.27:0/1951066055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b95a0 0x7f486c0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.314+0000 7f486ad9d640 1 --2- 172.21.15.27:0/1951066055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f486c0b9a10 0x7f486c0b5c40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f485c0097a0 tx=0x7f485c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c135dbf06ada51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.314+0000 7f486959a640 1 -- 172.21.15.27:0/1951066055 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c035020 con 0x7f486c0b9a10 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.314+0000 7f486959a640 1 -- 172.21.15.27:0/1951066055 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485c031810 con 0x7f486c0b9a10 2024-03-06T07:31:24.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.314+0000 7f486959a640 1 -- 172.21.15.27:0/1951066055 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c031be0 con 0x7f486c0b9a10 2024-03-06T07:31:24.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.315+0000 7f4870839640 1 -- 172.21.15.27:0/1951066055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f486c0b9a10 msgr2=0x7f486c0b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.315+0000 7f4870839640 1 --2- 172.21.15.27:0/1951066055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f486c0b9a10 0x7f486c0b5c40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f485c0097a0 tx=0x7f485c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.315+0000 7f4870839640 1 -- 172.21.15.27:0/1951066055 shutdown_connections 2024-03-06T07:31:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.315+0000 7f4870839640 1 --2- 172.21.15.27:0/1951066055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f486c0b9a10 0x7f486c0b5c40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.315+0000 7f4870839640 1 --2- 172.21.15.27:0/1951066055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b95a0 0x7f486c0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.315+0000 7f4870839640 1 -- 172.21.15.27:0/1951066055 >> 172.21.15.27:0/1951066055 conn(0x7f486c071960 msgr2=0x7f486c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.315+0000 7f4870839640 1 -- 172.21.15.27:0/1951066055 shutdown_connections 2024-03-06T07:31:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.315+0000 7f4870839640 1 -- 172.21.15.27:0/1951066055 wait complete. 2024-03-06T07:31:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.316+0000 7f4870839640 1 Processor -- start 2024-03-06T07:31:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.316+0000 7f4870839640 1 -- start start 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f4870839640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f486c0b95a0 0x7f486c1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f4870839640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b9a10 0x7f486c0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f4870839640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486c15df00 0x7f486c0ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f4870839640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c0cedf0 con 0x7f486c15df00 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f4870839640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c0cec70 con 0x7f486c0b95a0 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f4870839640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486c0d3220 con 0x7f486c0b9a10 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f486ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486c15df00 0x7f486c0ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f486ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486c15df00 0x7f486c0ccb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35682/0 (socket says 172.21.15.27:35682) 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f486ad9d640 1 -- 172.21.15.27:0/1670715629 learned_addr learned my addr 172.21.15.27:0/1670715629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f486a59c640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f486c0b95a0 0x7f486c1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f4869d9b640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b9a10 0x7f486c0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f486ad9d640 1 -- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b9a10 msgr2=0x7f486c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f486ad9d640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b9a10 0x7f486c0cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f486ad9d640 1 -- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f486c0b95a0 msgr2=0x7f486c1551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f486ad9d640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f486c0b95a0 0x7f486c1551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.317+0000 7f486ad9d640 1 -- 172.21.15.27:0/1670715629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c0cd1b0 con 0x7f486c15df00 2024-03-06T07:31:24.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.318+0000 7f486ad9d640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486c15df00 0x7f486c0ccb30 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f485c0098c0 tx=0x7f485c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.318+0000 7f4862ffd640 1 -- 172.21.15.27:0/1670715629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c041070 con 0x7f486c15df00 2024-03-06T07:31:24.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.318+0000 7f4862ffd640 1 -- 172.21.15.27:0/1670715629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485c031e40 con 0x7f486c15df00 2024-03-06T07:31:24.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.318+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486c0cd490 con 0x7f486c15df00 2024-03-06T07:31:24.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.318+0000 7f4862ffd640 1 -- 172.21.15.27:0/1670715629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c035040 con 0x7f486c15df00 2024-03-06T07:31:24.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.318+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486c1556e0 con 0x7f486c15df00 2024-03-06T07:31:24.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.319+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486c067ff0 con 0x7f486c15df00 2024-03-06T07:31:24.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.319+0000 7f4862ffd640 1 -- 172.21.15.27:0/1670715629 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f485c038020 con 0x7f486c15df00 2024-03-06T07:31:24.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.320+0000 7f4862ffd640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484c06b500 0x7f484c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.320+0000 7f4862ffd640 1 -- 172.21.15.27:0/1670715629 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f485c0abc30 con 0x7f486c15df00 2024-03-06T07:31:24.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.320+0000 7f486a59c640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484c06b500 0x7f484c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.320+0000 7f486a59c640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484c06b500 0x7f484c06d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4864005c00 tx=0x7f4864009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.324+0000 7f4862ffd640 1 -- 172.21.15.27:0/1670715629 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f485c07a430 con 0x7f486c15df00 2024-03-06T07:31:24.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.662+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f486c15cad0 con 0x7f486c15df00 2024-03-06T07:31:24.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.663+0000 7f4862ffd640 1 -- 172.21.15.27:0/1670715629 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f485c07e4d0 con 0x7f486c15df00 2024-03-06T07:31:24.666 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:24.666 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:30:55.351353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8679,"mds_1":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":7,"addr":"172.21.15.27:6834/1144600655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1144600655},{"type":"v1","addr":"172.21.15.27:6834","nonce":1144600655}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484c06b500 msgr2=0x7f484c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484c06b500 0x7f484c06d9c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4864005c00 tx=0x7f4864009040 comp rx=0 tx=0).stop 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486c15df00 msgr2=0x7f486c0ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486c15df00 0x7f486c0ccb30 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f485c0098c0 tx=0x7f485c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 shutdown_connections 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f484c06b500 0x7f484c06d9c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486c15df00 0x7f486c0ccb30 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f486c0b9a10 0x7f486c0cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 --2- 172.21.15.27:0/1670715629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f486c0b95a0 0x7f486c1551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 >> 172.21.15.27:0/1670715629 conn(0x7f486c071960 msgr2=0x7f486c071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 shutdown_connections 2024-03-06T07:31:24.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:24.665+0000 7f4870839640 1 -- 172.21.15.27:0/1670715629 wait complete. 2024-03-06T07:31:24.827 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:24.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:30:55.351353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8679, 'mds_1': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 7, 'addr': '172.21.15.27:6834/1144600655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1144600655}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1144600655}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:24.881 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8670:up:stopping 2024-03-06T07:31:25.646 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:31:25.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.987+0000 7f356d4df640 1 Processor -- start 2024-03-06T07:31:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.987+0000 7f356d4df640 1 -- start start 2024-03-06T07:31:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.988+0000 7f356d4df640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35680b0e90 0x7f356806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.988+0000 7f356d4df640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35680ae9b0 0x7f35680d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.988+0000 7f356d4df640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35680cee90 con 0x7f356806b5c0 2024-03-06T07:31:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.988+0000 7f356d4df640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35680cfa00 con 0x7f35680ae9b0 2024-03-06T07:31:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.988+0000 7f356d4df640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35680cfc50 con 0x7f35680b0e90 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.988+0000 7f35677fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35680ae9b0 0x7f35680d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.988+0000 7f35677fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35680ae9b0 0x7f35680d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35708/0 (socket says 172.21.15.27:35708) 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.988+0000 7f35677fe640 1 -- 172.21.15.27:0/3785175905 learned_addr learned my addr 172.21.15.27:0/3785175905 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.988+0000 7f3566ffd640 1 --1- 172.21.15.27:0/3785175905 >> v1:172.21.15.27:6789/0 conn(0x7f356806b5c0 0x7f35680b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42314/0 (socket says 172.21.15.27:42314) 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.989+0000 7f35677fe640 1 -- 172.21.15.27:0/3785175905 >> v2:172.21.15.69:3300/0 conn(0x7f35680b0e90 msgr2=0x7f356806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.989+0000 7f35667fc640 1 --2- 172.21.15.27:0/3785175905 >> v2:172.21.15.69:3300/0 conn(0x7f35680b0e90 0x7f356806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.989+0000 7f35677fe640 1 --2- 172.21.15.27:0/3785175905 >> v2:172.21.15.69:3300/0 conn(0x7f35680b0e90 0x7f356806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.989+0000 7f35677fe640 1 -- 172.21.15.27:0/3785175905 >> v1:172.21.15.27:6789/0 conn(0x7f356806b5c0 legacy=0x7f35680b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.989+0000 7f3565ffb640 1 -- 172.21.15.27:0/3785175905 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944721397 0 0) 0x7f35680cee90 con 0x7f356806b5c0 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.989+0000 7f35677fe640 1 -- 172.21.15.27:0/3785175905 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35680cff30 con 0x7f35680ae9b0 2024-03-06T07:31:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.989+0000 7f35667fc640 1 --2- 172.21.15.27:0/3785175905 >> v2:172.21.15.69:3300/0 conn(0x7f35680b0e90 0x7f356806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.989+0000 7f35677fe640 1 --2- 172.21.15.27:0/3785175905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680ae9b0 0x7f35680d03a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f35680b0e40 tx=0x7f356002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2c4d695a6f32d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.989+0000 7f3565ffb640 1 -- 172.21.15.27:0/3785175905 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356002ca80 con 0x7f35680ae9b0 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f3565ffb640 1 -- 172.21.15.27:0/3785175905 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356002cc20 con 0x7f35680ae9b0 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f3565ffb640 1 -- 172.21.15.27:0/3785175905 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35600343f0 con 0x7f35680ae9b0 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f356d4df640 1 -- 172.21.15.27:0/3785175905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680ae9b0 msgr2=0x7f35680d03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f356d4df640 1 --2- 172.21.15.27:0/3785175905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680ae9b0 0x7f35680d03a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f35680b0e40 tx=0x7f356002bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f356d4df640 1 -- 172.21.15.27:0/3785175905 shutdown_connections 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f356d4df640 1 --2- 172.21.15.27:0/3785175905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680ae9b0 0x7f35680d03a0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f356d4df640 1 --2- 172.21.15.27:0/3785175905 >> v2:172.21.15.69:3300/0 conn(0x7f35680b0e90 0x7f356806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f356d4df640 1 -- 172.21.15.27:0/3785175905 >> 172.21.15.27:0/3785175905 conn(0x7f35680b36d0 msgr2=0x7f35680b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f356d4df640 1 -- 172.21.15.27:0/3785175905 shutdown_connections 2024-03-06T07:31:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.990+0000 7f356d4df640 1 -- 172.21.15.27:0/3785175905 wait complete. 2024-03-06T07:31:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.991+0000 7f356d4df640 1 Processor -- start 2024-03-06T07:31:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f356d4df640 1 -- start start 2024-03-06T07:31:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f356d4df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f356806b5c0 0x7f35681dbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f356d4df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35680ae9b0 0x7f35681e8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f356d4df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680b0e90 0x7f35680c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f356d4df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356806c910 con 0x7f35680b0e90 2024-03-06T07:31:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f356d4df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356806c790 con 0x7f356806b5c0 2024-03-06T07:31:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f356d4df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35680aedc0 con 0x7f35680ae9b0 2024-03-06T07:31:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f35677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680b0e90 0x7f35680c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f35677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680b0e90 0x7f35680c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35724/0 (socket says 172.21.15.27:35724) 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.992+0000 7f35677fe640 1 -- 172.21.15.27:0/4141039219 learned_addr learned my addr 172.21.15.27:0/4141039219 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f35667fc640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35680ae9b0 0x7f35681e8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f3566ffd640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f356806b5c0 0x7f35681dbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f35677fe640 1 -- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35680ae9b0 msgr2=0x7f35681e8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f35677fe640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35680ae9b0 0x7f35681e8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f35677fe640 1 -- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f356806b5c0 msgr2=0x7f35681dbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f35677fe640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f356806b5c0 0x7f35681dbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f35677fe640 1 -- 172.21.15.27:0/4141039219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35681e8140 con 0x7f35680b0e90 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f35667fc640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35680ae9b0 0x7f35681e8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f3566ffd640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f356806b5c0 0x7f35681dbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f35677fe640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680b0e90 0x7f35680c5f40 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f3560036040 tx=0x7f3560004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f355f7fe640 1 -- 172.21.15.27:0/4141039219 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560043020 con 0x7f35680b0e90 2024-03-06T07:31:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f355f7fe640 1 -- 172.21.15.27:0/4141039219 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356003e450 con 0x7f35680b0e90 2024-03-06T07:31:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35681e8420 con 0x7f35680b0e90 2024-03-06T07:31:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f355f7fe640 1 -- 172.21.15.27:0/4141039219 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560009510 con 0x7f35680b0e90 2024-03-06T07:31:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.993+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35680cca30 con 0x7f35680b0e90 2024-03-06T07:31:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.994+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3568068700 con 0x7f35680b0e90 2024-03-06T07:31:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.995+0000 7f355f7fe640 1 -- 172.21.15.27:0/4141039219 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3560046020 con 0x7f35680b0e90 2024-03-06T07:31:26.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.995+0000 7f355f7fe640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f354806b4d0 0x7f354806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.995+0000 7f355f7fe640 1 -- 172.21.15.27:0/4141039219 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f35600acad0 con 0x7f35680b0e90 2024-03-06T07:31:26.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.996+0000 7f3566ffd640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f354806b4d0 0x7f354806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:25.996+0000 7f3566ffd640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f354806b4d0 0x7f354806d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3558000f60 tx=0x7f3558007c10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.000+0000 7f355f7fe640 1 -- 172.21.15.27:0/4141039219 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f356004d030 con 0x7f35680b0e90 2024-03-06T07:31:26.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.220+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35681e6460 con 0x7f35680b0e90 2024-03-06T07:31:26.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.220+0000 7f355f7fe640 1 -- 172.21.15.27:0/4141039219 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2611 (secure 0 0 0) 0x7f356007b9a0 con 0x7f35680b0e90 2024-03-06T07:31:26.223 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:26.223 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:31:25.351787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:26.223 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-03-06T07:31:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f354806b4d0 msgr2=0x7f354806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f354806b4d0 0x7f354806d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3558000f60 tx=0x7f3558007c10 comp rx=0 tx=0).stop 2024-03-06T07:31:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680b0e90 msgr2=0x7f35680c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680b0e90 0x7f35680c5f40 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f3560036040 tx=0x7f3560004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 shutdown_connections 2024-03-06T07:31:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f354806b4d0 0x7f354806d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35680b0e90 0x7f35680c5f40 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35680ae9b0 0x7f35681e8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 --2- 172.21.15.27:0/4141039219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f356806b5c0 0x7f35681dbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 >> 172.21.15.27:0/4141039219 conn(0x7f35680b36d0 msgr2=0x7f35681e9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 shutdown_connections 2024-03-06T07:31:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.222+0000 7f356d4df640 1 -- 172.21.15.27:0/4141039219 wait complete. 2024-03-06T07:31:26.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.718897+0000', 'modified': '2024-03-06T07:31:25.351787+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8679}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_8679': {'gid': 8679, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6835/4036525552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4036525552}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4036525552}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-03-06T07:31:26.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:31:26.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:31:26.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.543+0000 7fac9968a640 1 Processor -- start 2024-03-06T07:31:26.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.543+0000 7fac9968a640 1 -- start start 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac9968a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac940d18c0 0x7fac94071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac9968a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac94071a80 0x7fac940b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac9968a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940b6f20 con 0x7fac9415df70 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac9968a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac940d3030 con 0x7fac94071a80 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac9968a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac940d3310 con 0x7fac940d18c0 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac937fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac94071a80 0x7fac940b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac937fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac94071a80 0x7fac940b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35736/0 (socket says 172.21.15.27:35736) 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac937fe640 1 -- 172.21.15.27:0/1821771803 learned_addr learned my addr 172.21.15.27:0/1821771803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac927fc640 1 --2- 172.21.15.27:0/1821771803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac940d18c0 0x7fac94071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac937fe640 1 -- 172.21.15.27:0/1821771803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac940d18c0 msgr2=0x7fac94071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac937fe640 1 --2- 172.21.15.27:0/1821771803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac940d18c0 0x7fac94071540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac937fe640 1 -- 172.21.15.27:0/1821771803 >> v1:172.21.15.69:6789/0 conn(0x7fac9415df70 legacy=0x7fac940d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.544+0000 7fac937fe640 1 -- 172.21.15.27:0/1821771803 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940d35f0 con 0x7fac94071a80 2024-03-06T07:31:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.545+0000 7fac937fe640 1 --2- 172.21.15.27:0/1821771803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 0x7fac940b2700 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009e50 tx=0x7fac8c02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be649f9a86e179a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.545+0000 7fac91ffb640 1 -- 172.21.15.27:0/1821771803 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c02cb90 con 0x7fac94071a80 2024-03-06T07:31:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.545+0000 7fac91ffb640 1 -- 172.21.15.27:0/1821771803 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c02cd30 con 0x7fac94071a80 2024-03-06T07:31:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.545+0000 7fac91ffb640 1 -- 172.21.15.27:0/1821771803 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c034610 con 0x7fac94071a80 2024-03-06T07:31:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.546+0000 7fac9968a640 1 -- 172.21.15.27:0/1821771803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 msgr2=0x7fac940b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.546+0000 7fac9968a640 1 --2- 172.21.15.27:0/1821771803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 0x7fac940b2700 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009e50 tx=0x7fac8c02bff0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.546+0000 7fac9968a640 1 -- 172.21.15.27:0/1821771803 shutdown_connections 2024-03-06T07:31:26.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.546+0000 7fac9968a640 1 --2- 172.21.15.27:0/1821771803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 0x7fac940b2700 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.546+0000 7fac9968a640 1 --2- 172.21.15.27:0/1821771803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac940d18c0 0x7fac94071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.546+0000 7fac9968a640 1 -- 172.21.15.27:0/1821771803 >> 172.21.15.27:0/1821771803 conn(0x7fac940d3cf0 msgr2=0x7fac940d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.546+0000 7fac9968a640 1 -- 172.21.15.27:0/1821771803 shutdown_connections 2024-03-06T07:31:26.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.546+0000 7fac9968a640 1 -- 172.21.15.27:0/1821771803 wait complete. 2024-03-06T07:31:26.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.547+0000 7fac9968a640 1 Processor -- start 2024-03-06T07:31:26.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.547+0000 7fac9968a640 1 -- start start 2024-03-06T07:31:26.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac9968a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 0x7fac94150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac9968a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac940d18c0 0x7fac940ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac9968a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9415df70 0x7fac940c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac9968a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9415e4d0 con 0x7fac94071a80 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac9968a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9415e350 con 0x7fac940d18c0 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac9968a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac940b2c40 con 0x7fac9415df70 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac92ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 0x7fac94150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac92ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 0x7fac94150250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35744/0 (socket says 172.21.15.27:35744) 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac92ffd640 1 -- 172.21.15.27:0/4237404244 learned_addr learned my addr 172.21.15.27:0/4237404244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac927fc640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac940d18c0 0x7fac940ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac92ffd640 1 -- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9415df70 msgr2=0x7fac940c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac92ffd640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9415df70 0x7fac940c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac92ffd640 1 -- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac940d18c0 msgr2=0x7fac940ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac92ffd640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac940d18c0 0x7fac940ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac92ffd640 1 -- 172.21.15.27:0/4237404244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940cc980 con 0x7fac94071a80 2024-03-06T07:31:26.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.548+0000 7fac92ffd640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 0x7fac94150250 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fac8400c3c0 tx=0x7fac8400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.549+0000 7fac7b7fe640 1 -- 172.21.15.27:0/4237404244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac840052d0 con 0x7fac94071a80 2024-03-06T07:31:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.549+0000 7fac7b7fe640 1 -- 172.21.15.27:0/4237404244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84005490 con 0x7fac94071a80 2024-03-06T07:31:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.549+0000 7fac7b7fe640 1 -- 172.21.15.27:0/4237404244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84007260 con 0x7fac94071a80 2024-03-06T07:31:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.549+0000 7fac9968a640 1 -- 172.21.15.27:0/4237404244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac940ccc60 con 0x7fac94071a80 2024-03-06T07:31:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.549+0000 7fac9968a640 1 -- 172.21.15.27:0/4237404244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac940cd220 con 0x7fac94071a80 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.550+0000 7fac7b7fe640 1 -- 172.21.15.27:0/4237404244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac84007400 con 0x7fac94071a80 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.551+0000 7fac7b7fe640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac8006b500 0x7fac8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.551+0000 7fac7b7fe640 1 -- 172.21.15.27:0/4237404244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fac84089700 con 0x7fac94071a80 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.551+0000 7fac927fc640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac8006b500 0x7fac8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.551+0000 7fac9c26a740 1 Processor -- start 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac9c26a740 1 -- start start 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac927fc640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac8006b500 0x7fac8006d9c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fac940ce150 tx=0x7fac88008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e409870 0x55962e409c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e40d8c0 0x55962e40f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e439600 0x55962e439a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac9c26a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962e3736b0 con 0x55962e40d8c0 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac9c26a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962e31ca60 con 0x55962e439600 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac9c26a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55962e36f750 con 0x55962e409870 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e40d8c0 0x55962e40f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e40d8c0 0x55962e40f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35756/0 (socket says 172.21.15.27:35756) 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.552+0000 7fac927fc640 1 -- 172.21.15.27:0/1597770866 learned_addr learned my addr 172.21.15.27:0/1597770866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac92ffd640 1 --2- 172.21.15.27:0/1597770866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e409870 0x55962e409c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac937fe640 1 --2- 172.21.15.27:0/1597770866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e439600 0x55962e439a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac927fc640 1 -- 172.21.15.27:0/1597770866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e409870 msgr2=0x55962e409c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac927fc640 1 --2- 172.21.15.27:0/1597770866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e409870 0x55962e409c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac927fc640 1 -- 172.21.15.27:0/1597770866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e439600 msgr2=0x55962e439a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac927fc640 1 --2- 172.21.15.27:0/1597770866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e439600 0x55962e439a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac927fc640 1 -- 172.21.15.27:0/1597770866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55962e33af30 con 0x55962e40d8c0 2024-03-06T07:31:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac927fc640 1 --2- 172.21.15.27:0/1597770866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e40d8c0 0x55962e40f360 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fac8800e920 tx=0x7fac88011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02dcf9f9e8008a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac78ff9640 1 -- 172.21.15.27:0/1597770866 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88025070 con 0x55962e40d8c0 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac78ff9640 1 -- 172.21.15.27:0/1597770866 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac88015700 con 0x55962e40d8c0 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.553+0000 7fac78ff9640 1 -- 172.21.15.27:0/1597770866 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8800f040 con 0x55962e40d8c0 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.554+0000 7fac9c26a740 1 -- 172.21.15.27:0/1597770866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e40d8c0 msgr2=0x55962e40f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.554+0000 7fac9c26a740 1 --2- 172.21.15.27:0/1597770866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e40d8c0 0x55962e40f360 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fac8800e920 tx=0x7fac88011000 comp rx=0 tx=0).stop 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.554+0000 7fac9c26a740 1 -- 172.21.15.27:0/1597770866 shutdown_connections 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.554+0000 7fac9c26a740 1 --2- 172.21.15.27:0/1597770866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e439600 0x55962e439a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.554+0000 7fac9c26a740 1 --2- 172.21.15.27:0/1597770866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e40d8c0 0x55962e40f360 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.554+0000 7fac9c26a740 1 --2- 172.21.15.27:0/1597770866 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e409870 0x55962e409c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.554+0000 7fac9c26a740 1 -- 172.21.15.27:0/1597770866 >> 172.21.15.27:0/1597770866 conn(0x55962e40b8f0 msgr2=0x55962e40d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.554+0000 7fac9c26a740 1 -- 172.21.15.27:0/1597770866 shutdown_connections 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.554+0000 7fac9c26a740 1 -- 172.21.15.27:0/1597770866 wait complete. 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac9c26a740 1 -- start start 2024-03-06T07:31:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac9c26a740 1 Processor -- start 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e439600 0x55962e41e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e42d180 0x55962e40e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 0x55962e40e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac9c26a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962e40dd50 con 0x55962e439600 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac9c26a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962e32fea0 con 0x55962e42d560 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac9c26a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55962e40e030 con 0x55962e42d180 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac9c26a740 10 client.0 _ll_register_callbacks cb 0x55962e32f870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.555+0000 7fac92ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e439600 0x55962e41e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac92ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e439600 0x55962e41e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35768/0 (socket says 172.21.15.27:35768) 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac92ffd640 1 -- 172.21.15.27:0/3047907982 learned_addr learned my addr 172.21.15.27:0/3047907982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac937fe640 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 0x55962e40e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac927fc640 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e42d180 0x55962e40e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac92ffd640 1 -- 172.21.15.27:0/3047907982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e42d180 msgr2=0x55962e40e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac92ffd640 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e42d180 0x55962e40e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac92ffd640 1 -- 172.21.15.27:0/3047907982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 msgr2=0x55962e40e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac92ffd640 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 0x55962e40e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac92ffd640 1 -- 172.21.15.27:0/3047907982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55962e330270 con 0x55962e439600 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac927fc640 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e42d180 0x55962e40e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac92ffd640 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e439600 0x55962e41e980 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fac84060020 tx=0x7fac8405c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2401bf1d4d7adfc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac71ffb640 1 -- 172.21.15.27:0/3047907982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8405a040 con 0x55962e439600 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac71ffb640 1 -- 172.21.15.27:0/3047907982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fac840544b0 con 0x55962e439600 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac9c26a740 1 -- 172.21.15.27:0/3047907982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac88015700 con 0x55962e439600 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac71ffb640 1 -- 172.21.15.27:0/3047907982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac84056200 con 0x55962e439600 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac9c26a740 10 client.9081 fetch_fsmap learned FSMap version 102 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac9c26a740 1 -- 172.21.15.27:0/3047907982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x55962e0882d0 con 0x55962e439600 2024-03-06T07:31:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.556+0000 7fac71ffb640 1 -- 172.21.15.27:0/3047907982 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac88015700 con 0x55962e439600 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac71ffb640 1 -- 172.21.15.27:0/3047907982 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 102) v1 ==== 1597+0+0 (secure 0 0 0) 0x7fac8406fbd0 con 0x55962e439600 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac9c26a740 10 client.9081 fetch_fsmap finished waiting for FSMap version 102 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac9c26a740 10 client.9081 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac9c26a740 20 client.9081 populate_metadata read hostname 'smithi027' 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55962e43baa0 0x55962e43be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac9c26a740 4 client.9081 mds_command: new command op to 8679 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac9c26a740 1 -- 172.21.15.27:0/3047907982 --> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55962e380bf0 con 0x55962e43baa0 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac927fc640 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55962e43baa0 0x55962e43be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac71ffb640 10 client.9081 ms_handle_connect on v2:172.21.15.27:6833/4036525552 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.557+0000 7fac927fc640 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55962e43baa0 0x55962e43be80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a3c1ec699228027 server_cookie=5b954d9b5fe065fd in_seq=0 out_seq=0 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac71ffb640 1 -- 172.21.15.27:0/3047907982 <== mds.0 v2:172.21.15.27:6833/4036525552 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55962e380bf0 con 0x55962e43baa0 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac71ffb640 10 client.9081 handle_command_reply: tid=0 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 client.9081 shutdown 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 -- 172.21.15.27:0/3047907982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e439600 msgr2=0x55962e41e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e439600 0x55962e41e980 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fac84060020 tx=0x7fac8405c050 comp rx=0 tx=0).stop 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 -- 172.21.15.27:0/3047907982 shutdown_connections 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55962e43baa0 0x55962e43be80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 0x55962e40e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e42d180 0x55962e40e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3047907982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e439600 0x55962e41e980 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac71ffb640 0 client.9081 ms_handle_reset on v2:172.21.15.27:6833/4036525552 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 -- 172.21.15.27:0/3047907982 >> 172.21.15.27:0/3047907982 conn(0x55962e409870 msgr2=0x55962e408510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 -- 172.21.15.27:0/3047907982 shutdown_connections 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 1 -- 172.21.15.27:0/3047907982 wait complete. 2024-03-06T07:31:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.558+0000 7fac9c26a740 20 client.9081 trim_cache size 0 max 16384 2024-03-06T07:31:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.575+0000 7fac9c26a740 1 Processor -- start 2024-03-06T07:31:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.575+0000 7fac9c26a740 1 -- start start 2024-03-06T07:31:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d180 0x55962e408700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 0x55962e477d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e439600 0x55962e4782b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac9c26a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac941f48a0 con 0x55962e42d180 2024-03-06T07:31:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac9c26a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962e40d2a0 con 0x55962e42d560 2024-03-06T07:31:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac9c26a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55962e432b50 con 0x55962e439600 2024-03-06T07:31:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac92ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d180 0x55962e408700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac92ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d180 0x55962e408700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35770/0 (socket says 172.21.15.27:35770) 2024-03-06T07:31:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac92ffd640 1 -- 172.21.15.27:0/3634958963 learned_addr learned my addr 172.21.15.27:0/3634958963 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac92ffd640 1 -- 172.21.15.27:0/3634958963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e439600 msgr2=0x55962e4782b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac92ffd640 1 --2- 172.21.15.27:0/3634958963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e439600 0x55962e4782b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac927fc640 1 --2- 172.21.15.27:0/3634958963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 0x55962e477d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac92ffd640 1 -- 172.21.15.27:0/3634958963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 msgr2=0x55962e477d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac92ffd640 1 --2- 172.21.15.27:0/3634958963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 0x55962e477d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac92ffd640 1 -- 172.21.15.27:0/3634958963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55962e43ae40 con 0x55962e42d180 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac927fc640 1 --2- 172.21.15.27:0/3634958963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 0x55962e477d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.576+0000 7fac92ffd640 1 --2- 172.21.15.27:0/3634958963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d180 0x55962e408700 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fac84058670 tx=0x7fac8406f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b0e5f458d1fcbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac73fff640 1 -- 172.21.15.27:0/3634958963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8405a040 con 0x55962e42d180 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac73fff640 1 -- 172.21.15.27:0/3634958963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84057530 con 0x55962e42d180 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac73fff640 1 -- 172.21.15.27:0/3634958963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8406fbd0 con 0x55962e42d180 2024-03-06T07:31:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac9c26a740 1 -- 172.21.15.27:0/3634958963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d180 msgr2=0x55962e408700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3634958963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d180 0x55962e408700 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fac84058670 tx=0x7fac8406f000 comp rx=0 tx=0).stop 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac9c26a740 1 -- 172.21.15.27:0/3634958963 shutdown_connections 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3634958963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e439600 0x55962e4782b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3634958963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d560 0x55962e477d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac9c26a740 1 --2- 172.21.15.27:0/3634958963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d180 0x55962e408700 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac9c26a740 1 -- 172.21.15.27:0/3634958963 >> 172.21.15.27:0/3634958963 conn(0x55962e409870 msgr2=0x55962e43d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac9c26a740 1 -- 172.21.15.27:0/3634958963 shutdown_connections 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.577+0000 7fac9c26a740 1 -- 172.21.15.27:0/3634958963 wait complete. 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.578+0000 7fac9c26a740 1 -- start start 2024-03-06T07:31:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.578+0000 7fac9c26a740 1 Processor -- start 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d180 0x55962e4322d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d560 0x55962e4c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac9c26a740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e439600 0x55962e4c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac9c26a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962e405610 con 0x55962e42d560 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac9c26a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962e40d2a0 con 0x55962e42d180 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac9c26a740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55962e36a600 con 0x55962e439600 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac9c26a740 10 client.0 _ll_register_callbacks cb 0x7fac84057ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d560 0x55962e4c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d560 0x55962e4c78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35772/0 (socket says 172.21.15.27:35772) 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac927fc640 1 -- 172.21.15.27:0/245031351 learned_addr learned my addr 172.21.15.27:0/245031351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac937fe640 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e439600 0x55962e4c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac92ffd640 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d180 0x55962e4322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac927fc640 1 -- 172.21.15.27:0/245031351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e439600 msgr2=0x55962e4c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac927fc640 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e439600 0x55962e4c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac927fc640 1 -- 172.21.15.27:0/245031351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d180 msgr2=0x55962e4322d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac927fc640 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d180 0x55962e4322d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac927fc640 1 -- 172.21.15.27:0/245031351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55962e1526b0 con 0x55962e42d560 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac927fc640 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d560 0x55962e4c78d0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fac88020000 tx=0x7fac88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f63f333d875d8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac71ffb640 1 -- 172.21.15.27:0/245031351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88025040 con 0x55962e42d560 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac71ffb640 1 -- 172.21.15.27:0/245031351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fac88004230 con 0x55962e42d560 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.579+0000 7fac9c26a740 1 -- 172.21.15.27:0/245031351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac84057530 con 0x55962e42d560 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac71ffb640 1 -- 172.21.15.27:0/245031351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac88038050 con 0x55962e42d560 2024-03-06T07:31:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 10 client.9087 fetch_fsmap learned FSMap version 102 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 1 -- 172.21.15.27:0/245031351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x55962e43a830 con 0x55962e42d560 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac71ffb640 1 -- 172.21.15.27:0/245031351 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84057530 con 0x55962e42d560 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac71ffb640 1 -- 172.21.15.27:0/245031351 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 102) v1 ==== 1597+0+0 (secure 0 0 0) 0x7fac8803b020 con 0x55962e42d560 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 10 client.9087 fetch_fsmap finished waiting for FSMap version 102 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 10 client.9087 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 20 client.9087 populate_metadata read hostname 'smithi027' 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55962e478b50 0x55962e4399e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 4 client.9087 mds_command: new command op to 8679 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 1 -- 172.21.15.27:0/245031351 --> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55962e40c050 con 0x55962e478b50 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac92ffd640 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55962e478b50 0x55962e4399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac71ffb640 10 client.9087 ms_handle_connect on v2:172.21.15.27:6833/4036525552 2024-03-06T07:31:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac92ffd640 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55962e478b50 0x55962e4399e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e75ba548c62c310 server_cookie=427e81d6f503964e in_seq=0 out_seq=0 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac71ffb640 1 -- 172.21.15.27:0/245031351 <== mds.0 v2:172.21.15.27:6833/4036525552 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55962e40c050 con 0x55962e478b50 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac71ffb640 10 client.9087 handle_command_reply: tid=0 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 1 client.9087 shutdown 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 1 -- 172.21.15.27:0/245031351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d560 msgr2=0x55962e4c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.580+0000 7fac9c26a740 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d560 0x55962e4c78d0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fac88020000 tx=0x7fac88004730 comp rx=0 tx=0).stop 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9c26a740 1 -- 172.21.15.27:0/245031351 shutdown_connections 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9c26a740 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.27:6833/4036525552,v1:172.21.15.27:6835/4036525552] conn(0x55962e478b50 0x55962e4399e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9c26a740 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55962e439600 0x55962e4c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9c26a740 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55962e42d560 0x55962e4c78d0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac71ffb640 0 client.9087 ms_handle_reset on v2:172.21.15.27:6833/4036525552 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9c26a740 1 --2- 172.21.15.27:0/245031351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55962e42d180 0x55962e4322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9c26a740 1 -- 172.21.15.27:0/245031351 >> 172.21.15.27:0/245031351 conn(0x55962e409870 msgr2=0x55962e42c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9c26a740 1 -- 172.21.15.27:0/245031351 shutdown_connections 2024-03-06T07:31:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9c26a740 1 -- 172.21.15.27:0/245031351 wait complete. 2024-03-06T07:31:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9c26a740 20 client.9087 trim_cache size 0 max 16384 2024-03-06T07:31:26.590 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:31:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:31:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:31:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8679, 2024-03-06T07:31:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:31:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:31:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:31:26.590 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 64.221733181000005, 2024-03-06T07:31:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 102, 2024-03-06T07:31:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 136, 2024-03-06T07:31:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 136, 2024-03-06T07:31:26.591 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 76.233729542999995 2024-03-06T07:31:26.591 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:31:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 -- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac8006b500 msgr2=0x7fac8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac8006b500 0x7fac8006d9c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fac940ce150 tx=0x7fac88008040 comp rx=0 tx=0).stop 2024-03-06T07:31:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 -- 172.21.15.27:0/4237404244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 msgr2=0x7fac94150250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 0x7fac94150250 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fac8400c3c0 tx=0x7fac8400c880 comp rx=0 tx=0).stop 2024-03-06T07:31:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 -- 172.21.15.27:0/4237404244 shutdown_connections 2024-03-06T07:31:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fac8006b500 0x7fac8006d9c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fac9415df70 0x7fac940c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac940d18c0 0x7fac940ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 --2- 172.21.15.27:0/4237404244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac94071a80 0x7fac94150250 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 -- 172.21.15.27:0/4237404244 >> 172.21.15.27:0/4237404244 conn(0x7fac940d3cf0 msgr2=0x7fac94160940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 -- 172.21.15.27:0/4237404244 shutdown_connections 2024-03-06T07:31:26.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26.581+0000 7fac9968a640 1 -- 172.21.15.27:0/4237404244 wait complete. 2024-03-06T07:31:26.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:31:26.829 INFO:teuthology.orchestra.run.smithi027.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:26.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:31:26.840 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:26.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:26.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:26.966 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:26.966 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:31:26.967 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:26.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:27.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.127+0000 7f6a366d4640 1 Processor -- start 2024-03-06T07:31:27.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.127+0000 7f6a366d4640 1 -- start start 2024-03-06T07:31:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.128+0000 7f6a366d4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a300b95a0 0x7f6a300b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.128+0000 7f6a366d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 0x7f6a300b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.128+0000 7f6a366d4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a300cea30 con 0x7f6a3015df00 2024-03-06T07:31:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.128+0000 7f6a366d4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a300cf3c0 con 0x7f6a300b95a0 2024-03-06T07:31:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.128+0000 7f6a366d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a300cf0e0 con 0x7f6a300b9a10 2024-03-06T07:31:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.129+0000 7f6a2f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a300b95a0 0x7f6a300b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.129+0000 7f6a34c71640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 0x7f6a300b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.129+0000 7f6a2ffff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6a3015df00 0x7f6a300d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:44476/0 (socket says 172.21.15.27:44476) 2024-03-06T07:31:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.129+0000 7f6a2ffff640 1 -- 172.21.15.27:0/711719940 learned_addr learned my addr 172.21.15.27:0/711719940 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.129+0000 7f6a2f7fe640 1 -- 172.21.15.27:0/711719940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 msgr2=0x7f6a300b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.129+0000 7f6a2f7fe640 1 --2- 172.21.15.27:0/711719940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 0x7f6a300b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.129+0000 7f6a2f7fe640 1 -- 172.21.15.27:0/711719940 >> v1:172.21.15.69:6789/0 conn(0x7f6a3015df00 legacy=0x7f6a300d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.129+0000 7f6a2f7fe640 1 -- 172.21.15.27:0/711719940 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a300ce470 con 0x7f6a300b95a0 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.129+0000 7f6a2effd640 1 -- 172.21.15.27:0/711719940 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302880433 0 0) 0x7f6a300cea30 con 0x7f6a3015df00 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.130+0000 7f6a2f7fe640 1 --2- 172.21.15.27:0/711719940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 0x7f6a300b5700 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6a240097a0 tx=0x7f6a2402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3462e595c012d865 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.130+0000 7f6a2effd640 1 -- 172.21.15.27:0/711719940 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a24035020 con 0x7f6a300b95a0 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.130+0000 7f6a2effd640 1 -- 172.21.15.27:0/711719940 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a24031830 con 0x7f6a300b95a0 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.130+0000 7f6a2effd640 1 -- 172.21.15.27:0/711719940 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a24031c00 con 0x7f6a300b95a0 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.131+0000 7f6a366d4640 1 -- 172.21.15.27:0/711719940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 msgr2=0x7f6a300b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.131+0000 7f6a366d4640 1 --2- 172.21.15.27:0/711719940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 0x7f6a300b5700 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6a240097a0 tx=0x7f6a2402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.131+0000 7f6a366d4640 1 -- 172.21.15.27:0/711719940 shutdown_connections 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.131+0000 7f6a366d4640 1 --2- 172.21.15.27:0/711719940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 0x7f6a300b5700 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.131+0000 7f6a366d4640 1 --2- 172.21.15.27:0/711719940 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 0x7f6a300b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.131+0000 7f6a366d4640 1 -- 172.21.15.27:0/711719940 >> 172.21.15.27:0/711719940 conn(0x7f6a30071960 msgr2=0x7f6a30071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.131+0000 7f6a366d4640 1 -- 172.21.15.27:0/711719940 shutdown_connections 2024-03-06T07:31:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.131+0000 7f6a366d4640 1 -- 172.21.15.27:0/711719940 wait complete. 2024-03-06T07:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.133+0000 7f6a366d4640 1 Processor -- start 2024-03-06T07:31:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.133+0000 7f6a366d4640 1 -- start start 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a366d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 0x7f6a301541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a366d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 0x7f6a301551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a366d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a3015df00 0x7f6a301556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a366d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a300cedf0 con 0x7f6a3015df00 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a366d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a300cec70 con 0x7f6a300b95a0 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a366d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a300d3220 con 0x7f6a300b9a10 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a34c71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a3015df00 0x7f6a301556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a34c71640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a3015df00 0x7f6a301556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35784/0 (socket says 172.21.15.27:35784) 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a34c71640 1 -- 172.21.15.27:0/3676894874 learned_addr learned my addr 172.21.15.27:0/3676894874 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a2ffff640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 0x7f6a301541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.134+0000 7f6a2f7fe640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 0x7f6a301551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a34c71640 1 -- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 msgr2=0x7f6a301551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a34c71640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 0x7f6a301551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a34c71640 1 -- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 msgr2=0x7f6a301541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a34c71640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 0x7f6a301541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a34c71640 1 -- 172.21.15.27:0/3676894874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a30155f10 con 0x7f6a3015df00 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a2ffff640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 0x7f6a301541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a34c71640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a3015df00 0x7f6a301556e0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009dd0 tx=0x7f6a2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a2cff9640 1 -- 172.21.15.27:0/3676894874 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28018020 con 0x7f6a3015df00 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a2cff9640 1 -- 172.21.15.27:0/3676894874 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a28014480 con 0x7f6a3015df00 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a2cff9640 1 -- 172.21.15.27:0/3676894874 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28014870 con 0x7f6a3015df00 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a301561f0 con 0x7f6a3015df00 2024-03-06T07:31:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.135+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a301567b0 con 0x7f6a3015df00 2024-03-06T07:31:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.136+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a300d2eb0 con 0x7f6a3015df00 2024-03-06T07:31:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.137+0000 7f6a2cff9640 1 -- 172.21.15.27:0/3676894874 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a2801b020 con 0x7f6a3015df00 2024-03-06T07:31:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.137+0000 7f6a2cff9640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.137+0000 7f6a2cff9640 1 -- 172.21.15.27:0/3676894874 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6a2808adf0 con 0x7f6a3015df00 2024-03-06T07:31:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.138+0000 7f6a2ffff640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.138+0000 7f6a2ffff640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6a20000f60 tx=0x7f6a20007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.141+0000 7f6a2cff9640 1 -- 172.21.15.27:0/3676894874 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a300d2eb0 con 0x7f6a3015df00 2024-03-06T07:31:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.323+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a30067ff0 con 0x7f6a3015df00 2024-03-06T07:31:27.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.324+0000 7f6a2cff9640 1 -- 172.21.15.27:0/3676894874 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f6a28022030 con 0x7f6a3015df00 2024-03-06T07:31:27.327 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:27.327 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":136,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:30:37.955761+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:20.059798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.689553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.a","mappings":[{"from":2,"to":0}]},{"pgid":"19.1f","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a1c06b4d0 msgr2=0x7f6a1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6a20000f60 tx=0x7f6a20007ca0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a3015df00 msgr2=0x7f6a301556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a3015df00 0x7f6a301556e0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009dd0 tx=0x7f6a2800c490 comp rx=0 tx=0).stop 2024-03-06T07:31:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 shutdown_connections 2024-03-06T07:31:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a3015df00 0x7f6a301556e0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a300b9a10 0x7f6a301551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 --2- 172.21.15.27:0/3676894874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a300b95a0 0x7f6a301541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.326+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 >> 172.21.15.27:0/3676894874 conn(0x7f6a30071960 msgr2=0x7f6a300b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.327+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 shutdown_connections 2024-03-06T07:31:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.327+0000 7f6a366d4640 1 -- 172.21.15.27:0/3676894874 wait complete. 2024-03-06T07:31:27.340 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:20.059798+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:20.689553+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:27.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:27.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.452+0000 7f4d7748d640 1 Processor -- start 2024-03-06T07:31:27.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.452+0000 7f4d7748d640 1 -- start start 2024-03-06T07:31:27.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.453+0000 7f4d7748d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d70071540 0x7f4d70071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.453+0000 7f4d7748d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 0x7f4d700ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.453+0000 7f4d7748d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d70071e60 con 0x7f4d700cecb0 2024-03-06T07:31:27.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.453+0000 7f4d7748d640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d700d3230 con 0x7f4d70071540 2024-03-06T07:31:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.453+0000 7f4d7748d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700d4960 con 0x7f4d700b2d30 2024-03-06T07:31:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.453+0000 7f4d75229640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d70071540 0x7f4d70071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.453+0000 7f4d75229640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d70071540 0x7f4d70071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35800/0 (socket says 172.21.15.27:35800) 2024-03-06T07:31:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.453+0000 7f4d75229640 1 -- 172.21.15.27:0/603009338 learned_addr learned my addr 172.21.15.27:0/603009338 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.454+0000 7f4d75229640 1 -- 172.21.15.27:0/603009338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 msgr2=0x7f4d700ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.454+0000 7f4d74a28640 1 --2- 172.21.15.27:0/603009338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 0x7f4d700ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.454+0000 7f4d75229640 1 --2- 172.21.15.27:0/603009338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 0x7f4d700ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.454+0000 7f4d75229640 1 -- 172.21.15.27:0/603009338 >> v1:172.21.15.69:6789/0 conn(0x7f4d700cecb0 legacy=0x7f4d700b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.454+0000 7f4d75229640 1 -- 172.21.15.27:0/603009338 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d700d36c0 con 0x7f4d70071540 2024-03-06T07:31:27.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.454+0000 7f4d75229640 1 --2- 172.21.15.27:0/603009338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 0x7f4d70071920 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f4d64009870 tx=0x7f4d6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb738b5e4fc36c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d63fff640 1 -- 172.21.15.27:0/603009338 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64035020 con 0x7f4d70071540 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d63fff640 1 -- 172.21.15.27:0/603009338 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d64031640 con 0x7f4d70071540 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d63fff640 1 -- 172.21.15.27:0/603009338 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6402d040 con 0x7f4d70071540 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d7748d640 1 -- 172.21.15.27:0/603009338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 msgr2=0x7f4d70071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d7748d640 1 --2- 172.21.15.27:0/603009338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 0x7f4d70071920 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f4d64009870 tx=0x7f4d6402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d7748d640 1 -- 172.21.15.27:0/603009338 shutdown_connections 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d7748d640 1 --2- 172.21.15.27:0/603009338 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 0x7f4d700ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d7748d640 1 --2- 172.21.15.27:0/603009338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 0x7f4d70071920 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d7748d640 1 -- 172.21.15.27:0/603009338 >> 172.21.15.27:0/603009338 conn(0x7f4d700d3d60 msgr2=0x7f4d700d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d7748d640 1 -- 172.21.15.27:0/603009338 shutdown_connections 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.455+0000 7f4d7748d640 1 -- 172.21.15.27:0/603009338 wait complete. 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.456+0000 7f4d7748d640 1 Processor -- start 2024-03-06T07:31:27.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.456+0000 7f4d7748d640 1 -- start start 2024-03-06T07:31:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d7748d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 0x7f4d700c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d7748d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 0x7f4d701583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d7748d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d700cecb0 0x7f4d701f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d7748d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700d2be0 con 0x7f4d70071540 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d7748d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700d2900 con 0x7f4d700cecb0 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d7748d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700d2ec0 con 0x7f4d700b2d30 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d75229640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 0x7f4d700c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d75229640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 0x7f4d700c8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35806/0 (socket says 172.21.15.27:35806) 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d75229640 1 -- 172.21.15.27:0/1320472782 learned_addr learned my addr 172.21.15.27:0/1320472782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d74a28640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 0x7f4d701583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.457+0000 7f4d75a2a640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d700cecb0 0x7f4d701f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d75229640 1 -- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 msgr2=0x7f4d701583d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d75229640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 0x7f4d701583d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d75229640 1 -- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d700cecb0 msgr2=0x7f4d701f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d75229640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d700cecb0 0x7f4d701f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d75229640 1 -- 172.21.15.27:0/1320472782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d700cafb0 con 0x7f4d70071540 2024-03-06T07:31:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d75a2a640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d700cecb0 0x7f4d701f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d74a28640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 0x7f4d701583d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d75229640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 0x7f4d700c8310 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f4d6402b930 tx=0x7f4d640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d61ffb640 1 -- 172.21.15.27:0/1320472782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64040070 con 0x7f4d70071540 2024-03-06T07:31:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d61ffb640 1 -- 172.21.15.27:0/1320472782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d64039c10 con 0x7f4d70071540 2024-03-06T07:31:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d700cb290 con 0x7f4d70071540 2024-03-06T07:31:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d61ffb640 1 -- 172.21.15.27:0/1320472782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64035040 con 0x7f4d70071540 2024-03-06T07:31:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.458+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d700cb850 con 0x7f4d70071540 2024-03-06T07:31:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.459+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70072070 con 0x7f4d70071540 2024-03-06T07:31:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.460+0000 7f4d61ffb640 1 -- 172.21.15.27:0/1320472782 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d6404b060 con 0x7f4d70071540 2024-03-06T07:31:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.460+0000 7f4d61ffb640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d5406b4d0 0x7f4d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.461+0000 7f4d61ffb640 1 -- 172.21.15.27:0/1320472782 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4d640abcc0 con 0x7f4d70071540 2024-03-06T07:31:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.461+0000 7f4d74a28640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d5406b4d0 0x7f4d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.461+0000 7f4d74a28640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d5406b4d0 0x7f4d5406d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c006fd0 tx=0x7f4d6c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.465+0000 7f4d61ffb640 1 -- 172.21.15.27:0/1320472782 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d6402f300 con 0x7f4d70071540 2024-03-06T07:31:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.647+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d7004f410 con 0x7f4d70071540 2024-03-06T07:31:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.648+0000 7f4d61ffb640 1 -- 172.21.15.27:0/1320472782 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v136) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4d6407ab90 con 0x7f4d70071540 2024-03-06T07:31:27.650 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d5406b4d0 msgr2=0x7f4d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d5406b4d0 0x7f4d5406d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c006fd0 tx=0x7f4d6c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 msgr2=0x7f4d700c8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 0x7f4d700c8310 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f4d6402b930 tx=0x7f4d640046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 shutdown_connections 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4d5406b4d0 0x7f4d5406d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d700cecb0 0x7f4d701f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4d700b2d30 0x7f4d701583d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 --2- 172.21.15.27:0/1320472782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d70071540 0x7f4d700c8310 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 >> 172.21.15.27:0/1320472782 conn(0x7f4d700d3d60 msgr2=0x7f4d7015b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 shutdown_connections 2024-03-06T07:31:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.650+0000 7f4d7748d640 1 -- 172.21.15.27:0/1320472782 wait complete. 2024-03-06T07:31:27.663 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:27.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.771+0000 7f1fd7c29640 1 Processor -- start 2024-03-06T07:31:27.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.771+0000 7f1fd7c29640 1 -- start start 2024-03-06T07:31:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.772+0000 7f1fd7c29640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1fd00b7420 0x7f1fd00d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.772+0000 7f1fd7c29640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd00d3740 con 0x7f1fd00ceea0 2024-03-06T07:31:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.772+0000 7f1fd7c29640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd00d3120 con 0x7f1fd015dd90 2024-03-06T07:31:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.772+0000 7f1fd7c29640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd00d3400 con 0x7f1fd00b7420 2024-03-06T07:31:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.772+0000 7f1fd61c6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1fd00b7420 0x7f1fd00d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.772+0000 7f1fd51c4640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1fd00ceea0 0x7f1fd00b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42330/0 (socket says 172.21.15.27:42330) 2024-03-06T07:31:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.772+0000 7f1fd51c4640 1 -- 172.21.15.27:0/1142763460 learned_addr learned my addr 172.21.15.27:0/1142763460 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.773+0000 7f1fd61c6640 1 -- 172.21.15.27:0/1142763460 >> v1:172.21.15.69:6789/0 conn(0x7f1fd015dd90 legacy=0x7f1fd00d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.773+0000 7f1fd61c6640 1 -- 172.21.15.27:0/1142763460 >> v1:172.21.15.27:6789/0 conn(0x7f1fd00ceea0 legacy=0x7f1fd00b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.773+0000 7f1fd61c6640 1 -- 172.21.15.27:0/1142763460 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd00d39e0 con 0x7f1fd00b7420 2024-03-06T07:31:27.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.773+0000 7f1fd49c3640 1 -- 172.21.15.27:0/1142763460 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771053954 0 0) 0x7f1fd00d3740 con 0x7f1fd00ceea0 2024-03-06T07:31:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd61c6640 1 --2- 172.21.15.27:0/1142763460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd00b7420 0x7f1fd00d29d0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009d90 tx=0x7f1fc802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ba370f94032a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd49c3640 1 -- 172.21.15.27:0/1142763460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8039070 con 0x7f1fd00b7420 2024-03-06T07:31:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd49c3640 1 -- 172.21.15.27:0/1142763460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc802cd00 con 0x7f1fd00b7420 2024-03-06T07:31:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd49c3640 1 -- 172.21.15.27:0/1142763460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8034660 con 0x7f1fd00b7420 2024-03-06T07:31:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1142763460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd00b7420 msgr2=0x7f1fd00d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd7c29640 1 --2- 172.21.15.27:0/1142763460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd00b7420 0x7f1fd00d29d0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009d90 tx=0x7f1fc802bf50 comp rx=0 tx=0).stop 2024-03-06T07:31:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1142763460 shutdown_connections 2024-03-06T07:31:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd7c29640 1 --2- 172.21.15.27:0/1142763460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd00b7420 0x7f1fd00d29d0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1142763460 >> 172.21.15.27:0/1142763460 conn(0x7f1fd0071960 msgr2=0x7f1fd0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1142763460 shutdown_connections 2024-03-06T07:31:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.774+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1142763460 wait complete. 2024-03-06T07:31:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd7c29640 1 Processor -- start 2024-03-06T07:31:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd7c29640 1 -- start start 2024-03-06T07:31:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd7c29640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd00ceea0 0x7f1fd01f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd7c29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd015dd90 0x7f1fd00cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd7c29640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd01f4a10 0x7f1fd00cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd7c29640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd00d48f0 con 0x7f1fd015dd90 2024-03-06T07:31:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd7c29640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd015e170 con 0x7f1fd00ceea0 2024-03-06T07:31:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd7c29640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd00d4a70 con 0x7f1fd01f4a10 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd51c4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd015dd90 0x7f1fd00cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd51c4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd015dd90 0x7f1fd00cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35826/0 (socket says 172.21.15.27:35826) 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.776+0000 7f1fd51c4640 1 -- 172.21.15.27:0/1257487118 learned_addr learned my addr 172.21.15.27:0/1257487118 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd51c4640 1 -- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd01f4a10 msgr2=0x7f1fd00cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd59c5640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd00ceea0 0x7f1fd01f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd61c6640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd01f4a10 0x7f1fd00cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd51c4640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd01f4a10 0x7f1fd00cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd51c4640 1 -- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd00ceea0 msgr2=0x7f1fd01f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd51c4640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd00ceea0 0x7f1fd01f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd51c4640 1 -- 172.21.15.27:0/1257487118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd01f6c50 con 0x7f1fd015dd90 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd61c6640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd01f4a10 0x7f1fd00cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd59c5640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd00ceea0 0x7f1fd01f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd51c4640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd015dd90 0x7f1fd00cad70 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00c460 tx=0x7f1fcc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fc67fc640 1 -- 172.21.15.27:0/1257487118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc005ce0 con 0x7f1fd015dd90 2024-03-06T07:31:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fc67fc640 1 -- 172.21.15.27:0/1257487118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc007070 con 0x7f1fd015dd90 2024-03-06T07:31:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd01f7190 con 0x7f1fd015dd90 2024-03-06T07:31:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fc67fc640 1 -- 172.21.15.27:0/1257487118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc007460 con 0x7f1fd015dd90 2024-03-06T07:31:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.777+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd01f47a0 con 0x7f1fd015dd90 2024-03-06T07:31:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.778+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd0072a30 con 0x7f1fd015dd90 2024-03-06T07:31:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.779+0000 7f1fc67fc640 1 -- 172.21.15.27:0/1257487118 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fcc007600 con 0x7f1fd015dd90 2024-03-06T07:31:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.779+0000 7f1fc67fc640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1fb406b4d0 0x7f1fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.779+0000 7f1fc67fc640 1 -- 172.21.15.27:0/1257487118 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1fcc089df0 con 0x7f1fd015dd90 2024-03-06T07:31:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.779+0000 7f1fd59c5640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1fb406b4d0 0x7f1fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.780+0000 7f1fd59c5640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1fb406b4d0 0x7f1fb406d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0000f60 tx=0x7f1fc0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.783+0000 7f1fc67fc640 1 -- 172.21.15.27:0/1257487118 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fcc058cc0 con 0x7f1fd015dd90 2024-03-06T07:31:27.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.965+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1fd00b4290 con 0x7f1fd015dd90 2024-03-06T07:31:27.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.965+0000 7f1fc67fc640 1 -- 172.21.15.27:0/1257487118 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v136) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1fcc0585c0 con 0x7f1fd015dd90 2024-03-06T07:31:27.968 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:31:27.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1fb406b4d0 msgr2=0x7f1fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1fb406b4d0 0x7f1fb406d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0000f60 tx=0x7f1fc0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd015dd90 msgr2=0x7f1fd00cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd015dd90 0x7f1fd00cad70 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00c460 tx=0x7f1fcc00c920 comp rx=0 tx=0).stop 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 shutdown_connections 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1fb406b4d0 0x7f1fb406d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1fd01f4a10 0x7f1fd00cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fd015dd90 0x7f1fd00cad70 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 --2- 172.21.15.27:0/1257487118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd00ceea0 0x7f1fd01f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 >> 172.21.15.27:0/1257487118 conn(0x7f1fd0071960 msgr2=0x7f1fd00b7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 shutdown_connections 2024-03-06T07:31:27.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:27.967+0000 7f1fd7c29640 1 -- 172.21.15.27:0/1257487118 wait complete. 2024-03-06T07:31:27.980 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:28.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.088+0000 7fb1327c9640 1 Processor -- start 2024-03-06T07:31:28.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.088+0000 7fb1327c9640 1 -- start start 2024-03-06T07:31:28.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.089+0000 7fb1327c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb12c15dd90 0x7fb12c0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.089+0000 7fb1327c9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb12c0d3740 con 0x7fb12c0d29d0 2024-03-06T07:31:28.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.089+0000 7fb1327c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb12c0d3120 con 0x7fb12c0b6d10 2024-03-06T07:31:28.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.089+0000 7fb1327c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb12c0d3400 con 0x7fb12c15dd90 2024-03-06T07:31:28.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.089+0000 7fb12bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb12c15dd90 0x7fb12c0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.089+0000 7fb12b7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb12c0d29d0 0x7fb12c0d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42340/0 (socket says 172.21.15.27:42340) 2024-03-06T07:31:28.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.089+0000 7fb12b7fe640 1 -- 172.21.15.27:0/2914041199 learned_addr learned my addr 172.21.15.27:0/2914041199 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:28.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.089+0000 7fb12bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb12c15dd90 0x7fb12c0ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35828/0 (socket says 172.21.15.27:35828) 2024-03-06T07:31:28.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.090+0000 7fb12bfff640 1 -- 172.21.15.27:0/2914041199 >> v1:172.21.15.69:6789/0 conn(0x7fb12c0b6d10 legacy=0x7fb12c0d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.090+0000 7fb12bfff640 1 -- 172.21.15.27:0/2914041199 >> v1:172.21.15.27:6789/0 conn(0x7fb12c0d29d0 legacy=0x7fb12c0d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.090+0000 7fb12bfff640 1 -- 172.21.15.27:0/2914041199 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c0d39e0 con 0x7fb12c15dd90 2024-03-06T07:31:28.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.090+0000 7fb12affd640 1 -- 172.21.15.27:0/2914041199 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914900537 0 0) 0x7fb12c0d3740 con 0x7fb12c0d29d0 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.090+0000 7fb12bfff640 1 --2- 172.21.15.27:0/2914041199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c15dd90 0x7fb12c0ceea0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb1200097a0 tx=0x7fb12002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6e5ce9a77359e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb12affd640 1 -- 172.21.15.27:0/2914041199 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12002f810 con 0x7fb12c15dd90 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb12affd640 1 -- 172.21.15.27:0/2914041199 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12002f9b0 con 0x7fb12c15dd90 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb12affd640 1 -- 172.21.15.27:0/2914041199 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12002fd80 con 0x7fb12c15dd90 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb1327c9640 1 -- 172.21.15.27:0/2914041199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c15dd90 msgr2=0x7fb12c0ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb1327c9640 1 --2- 172.21.15.27:0/2914041199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c15dd90 0x7fb12c0ceea0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb1200097a0 tx=0x7fb12002b9f0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb1327c9640 1 -- 172.21.15.27:0/2914041199 shutdown_connections 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb1327c9640 1 --2- 172.21.15.27:0/2914041199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c15dd90 0x7fb12c0ceea0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb1327c9640 1 -- 172.21.15.27:0/2914041199 >> 172.21.15.27:0/2914041199 conn(0x7fb12c071960 msgr2=0x7fb12c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb1327c9640 1 -- 172.21.15.27:0/2914041199 shutdown_connections 2024-03-06T07:31:28.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.091+0000 7fb1327c9640 1 -- 172.21.15.27:0/2914041199 wait complete. 2024-03-06T07:31:28.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.092+0000 7fb1327c9640 1 Processor -- start 2024-03-06T07:31:28.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb1327c9640 1 -- start start 2024-03-06T07:31:28.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb1327c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb12c0b6d10 0x7fb12c0cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb1327c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c0d29d0 0x7fb12c1e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb1327c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb12c15dd90 0x7fb12c15f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb1327c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c0d2df0 con 0x7fb12c0d29d0 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb1327c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c15e170 con 0x7fb12c0b6d10 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb1327c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c0d48f0 con 0x7fb12c15dd90 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb12b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c0d29d0 0x7fb12c1e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb12b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c0d29d0 0x7fb12c1e8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35832/0 (socket says 172.21.15.27:35832) 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.093+0000 7fb12b7fe640 1 -- 172.21.15.27:0/4087237776 learned_addr learned my addr 172.21.15.27:0/4087237776 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb12bfff640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb12c0b6d10 0x7fb12c0cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb130d66640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb12c15dd90 0x7fb12c15f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb12b7fe640 1 -- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb12c15dd90 msgr2=0x7fb12c15f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb12b7fe640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb12c15dd90 0x7fb12c15f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb12b7fe640 1 -- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb12c0b6d10 msgr2=0x7fb12c0cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb12b7fe640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb12c0b6d10 0x7fb12c0cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb12b7fe640 1 -- 172.21.15.27:0/4087237776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c15d2f0 con 0x7fb12c0d29d0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb130d66640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb12c15dd90 0x7fb12c15f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb12b7fe640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c0d29d0 0x7fb12c1e8ae0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00c4a0 tx=0x7fb11c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb128ff9640 1 -- 172.21.15.27:0/4087237776 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c018070 con 0x7fb12c0d29d0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb128ff9640 1 -- 172.21.15.27:0/4087237776 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c0074e0 con 0x7fb12c0d29d0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12c1e7760 con 0x7fb12c0d29d0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb128ff9640 1 -- 172.21.15.27:0/4087237776 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c004a20 con 0x7fb12c0d29d0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.094+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12c1e7d20 con 0x7fb12c0d29d0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.095+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12c072a30 con 0x7fb12c0d29d0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.096+0000 7fb128ff9640 1 -- 172.21.15.27:0/4087237776 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb11c01b070 con 0x7fb12c0d29d0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.096+0000 7fb128ff9640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.096+0000 7fb128ff9640 1 -- 172.21.15.27:0/4087237776 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb11c089500 con 0x7fb12c0d29d0 2024-03-06T07:31:28.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.096+0000 7fb12bfff640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.097+0000 7fb12bfff640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11806b4d0 0x7fb11806d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb120036000 tx=0x7fb120008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.101+0000 7fb128ff9640 1 -- 172.21.15.27:0/4087237776 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb12c15d2f0 con 0x7fb12c0d29d0 2024-03-06T07:31:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.280+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb12c0b5500 con 0x7fb12c0d29d0 2024-03-06T07:31:28.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.280+0000 7fb128ff9640 1 -- 172.21.15.27:0/4087237776 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v136) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb11c057cf0 con 0x7fb12c0d29d0 2024-03-06T07:31:28.283 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:31:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11806b4d0 msgr2=0x7fb11806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11806b4d0 0x7fb11806d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb120036000 tx=0x7fb120008030 comp rx=0 tx=0).stop 2024-03-06T07:31:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c0d29d0 msgr2=0x7fb12c1e8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c0d29d0 0x7fb12c1e8ae0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00c4a0 tx=0x7fb11c00c960 comp rx=0 tx=0).stop 2024-03-06T07:31:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 shutdown_connections 2024-03-06T07:31:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb12c15dd90 0x7fb12c15f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb12c0d29d0 0x7fb12c1e8ae0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 --2- 172.21.15.27:0/4087237776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb12c0b6d10 0x7fb12c0cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 >> 172.21.15.27:0/4087237776 conn(0x7fb12c071960 msgr2=0x7fb12c0b7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 shutdown_connections 2024-03-06T07:31:28.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.282+0000 7fb1327c9640 1 -- 172.21.15.27:0/4087237776 wait complete. 2024-03-06T07:31:28.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:31:28.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.401+0000 7f36be13a640 1 Processor -- start 2024-03-06T07:31:28.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.401+0000 7f36be13a640 1 -- start start 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36be13a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f36b8071540 0x7f36b80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36be13a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36b80d3400 0x7f36b8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36be13a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b8071e60 con 0x7f36b80b2b50 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36be13a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b80d2f70 con 0x7f36b8071540 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36be13a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b80d4be0 con 0x7f36b80d3400 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36b77fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f36b8071540 0x7f36b80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36b77fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f36b8071540 0x7f36b80b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35836/0 (socket says 172.21.15.27:35836) 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36b77fe640 1 -- 172.21.15.27:0/1138437692 learned_addr learned my addr 172.21.15.27:0/1138437692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36b77fe640 1 -- 172.21.15.27:0/1138437692 >> v2:172.21.15.69:3300/0 conn(0x7f36b80d3400 msgr2=0x7f36b8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36b77fe640 1 --2- 172.21.15.27:0/1138437692 >> v2:172.21.15.69:3300/0 conn(0x7f36b80d3400 0x7f36b8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36b77fe640 1 -- 172.21.15.27:0/1138437692 >> v1:172.21.15.69:6789/0 conn(0x7f36b80b2b50 legacy=0x7f36b80d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.402+0000 7f36b77fe640 1 -- 172.21.15.27:0/1138437692 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b80ce610 con 0x7f36b8071540 2024-03-06T07:31:28.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.403+0000 7f36b77fe640 1 --2- 172.21.15.27:0/1138437692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 0x7f36b80b30a0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f36b80b6860 tx=0x7f36a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e925945b9fd013f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.403+0000 7f36b67fc640 1 -- 172.21.15.27:0/1138437692 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8035020 con 0x7f36b8071540 2024-03-06T07:31:28.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.403+0000 7f36b67fc640 1 -- 172.21.15.27:0/1138437692 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a8031870 con 0x7f36b8071540 2024-03-06T07:31:28.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.403+0000 7f36b67fc640 1 -- 172.21.15.27:0/1138437692 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8031c40 con 0x7f36b8071540 2024-03-06T07:31:28.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.404+0000 7f36be13a640 1 -- 172.21.15.27:0/1138437692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 msgr2=0x7f36b80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.404+0000 7f36be13a640 1 --2- 172.21.15.27:0/1138437692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 0x7f36b80b30a0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f36b80b6860 tx=0x7f36a802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.404+0000 7f36be13a640 1 -- 172.21.15.27:0/1138437692 shutdown_connections 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.404+0000 7f36be13a640 1 --2- 172.21.15.27:0/1138437692 >> v2:172.21.15.69:3300/0 conn(0x7f36b80d3400 0x7f36b8071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.404+0000 7f36be13a640 1 --2- 172.21.15.27:0/1138437692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 0x7f36b80b30a0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.404+0000 7f36be13a640 1 -- 172.21.15.27:0/1138437692 >> 172.21.15.27:0/1138437692 conn(0x7f36b80d3ca0 msgr2=0x7f36b80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.404+0000 7f36be13a640 1 -- 172.21.15.27:0/1138437692 shutdown_connections 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.404+0000 7f36be13a640 1 -- 172.21.15.27:0/1138437692 wait complete. 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.405+0000 7f36be13a640 1 Processor -- start 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.405+0000 7f36be13a640 1 -- start start 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36be13a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 0x7f36b80cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36be13a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36b80b2b50 0x7f36b80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36be13a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36b80d3400 0x7f36b814ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36be13a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b80ce8f0 con 0x7f36b8071540 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36be13a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b80d2c00 con 0x7f36b80d3400 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36be13a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b815e370 con 0x7f36b80b2b50 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 0x7f36b80cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 0x7f36b80cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35852/0 (socket says 172.21.15.27:35852) 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b77fe640 1 -- 172.21.15.27:0/3594854682 learned_addr learned my addr 172.21.15.27:0/3594854682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b7fff640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36b80d3400 0x7f36b814ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b6ffd640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36b80b2b50 0x7f36b80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b77fe640 1 -- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36b80b2b50 msgr2=0x7f36b80b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b77fe640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36b80b2b50 0x7f36b80b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b77fe640 1 -- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36b80d3400 msgr2=0x7f36b814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b77fe640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36b80d3400 0x7f36b814ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b77fe640 1 -- 172.21.15.27:0/3594854682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b8158dd0 con 0x7f36b8071540 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b7fff640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36b80d3400 0x7f36b814ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.406+0000 7f36b77fe640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 0x7f36b80cb6f0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f36a8000c00 tx=0x7f36a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.407+0000 7f369ffff640 1 -- 172.21.15.27:0/3594854682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a803ad60 con 0x7f36b8071540 2024-03-06T07:31:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.407+0000 7f369ffff640 1 -- 172.21.15.27:0/3594854682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a80443f0 con 0x7f36b8071540 2024-03-06T07:31:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.407+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b81590b0 con 0x7f36b8071540 2024-03-06T07:31:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.407+0000 7f369ffff640 1 -- 172.21.15.27:0/3594854682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a80447e0 con 0x7f36b8071540 2024-03-06T07:31:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.407+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b8159670 con 0x7f36b8071540 2024-03-06T07:31:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.408+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b8068700 con 0x7f36b8071540 2024-03-06T07:31:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.408+0000 7f369ffff640 1 -- 172.21.15.27:0/3594854682 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36a804b060 con 0x7f36b8071540 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.409+0000 7f369ffff640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a406b4d0 0x7f36a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.409+0000 7f369ffff640 1 -- 172.21.15.27:0/3594854682 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f36a80ac590 con 0x7f36b8071540 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.409+0000 7f36b6ffd640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a406b4d0 0x7f36a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.409+0000 7f36b6ffd640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a406b4d0 0x7f36a406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f36b8157960 tx=0x7f36ac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.413+0000 7f369ffff640 1 -- 172.21.15.27:0/3594854682 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36a803f070 con 0x7f36b8071540 2024-03-06T07:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.610+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f36b80b62f0 con 0x7f36b8071540 2024-03-06T07:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.610+0000 7f369ffff640 1 -- 172.21.15.27:0/3594854682 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v136) v1 ==== 80+0+0 (secure 0 0 0) 0x7f36a802f320 con 0x7f36b8071540 2024-03-06T07:31:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a406b4d0 msgr2=0x7f36a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a406b4d0 0x7f36a406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f36b8157960 tx=0x7f36ac008040 comp rx=0 tx=0).stop 2024-03-06T07:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 msgr2=0x7f36b80cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 0x7f36b80cb6f0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f36a8000c00 tx=0x7f36a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 shutdown_connections 2024-03-06T07:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f36a406b4d0 0x7f36a406d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36b80d3400 0x7f36b814ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36b80b2b50 0x7f36b80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 --2- 172.21.15.27:0/3594854682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36b8071540 0x7f36b80cb6f0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 >> 172.21.15.27:0/3594854682 conn(0x7f36b80d3ca0 msgr2=0x7f36b8160b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 shutdown_connections 2024-03-06T07:31:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:28.612+0000 7f36be13a640 1 -- 172.21.15.27:0/3594854682 wait complete. 2024-03-06T07:31:28.626 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:31:28.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:28.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[78861]: fuse finished with error 0 and tester_r 0 2024-03-06T07:31:28.656 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:31:28.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.657 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:31:28.747 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:31:28.747 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:28.764 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:28.764 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:31:28.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.764 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:31:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:28.819 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:31:28.819 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:31:28.820 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:28.835 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:28.835 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:31:28.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.835 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:31:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:28.891 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:31:28.891 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:31:28.891 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:28.907 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:28.907 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:31:28.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.907 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:31:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:31:28.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.068+0000 7f2406bd7640 1 Processor -- start 2024-03-06T07:31:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.068+0000 7f2406bd7640 1 -- start start 2024-03-06T07:31:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f2406bd7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24000b95a0 0x7f24000b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f2406bd7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b9a10 0x7f24000b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f2406bd7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24000cea30 con 0x7f240015df00 2024-03-06T07:31:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f2406bd7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24000cf3c0 con 0x7f24000b95a0 2024-03-06T07:31:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f2406bd7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24000cf0e0 con 0x7f24000b9a10 2024-03-06T07:31:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f23fffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24000b95a0 0x7f24000b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f23fffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24000b95a0 0x7f24000b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35858/0 (socket says 172.21.15.27:35858) 2024-03-06T07:31:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f23fffff640 1 -- 172.21.15.27:0/4241242987 learned_addr learned my addr 172.21.15.27:0/4241242987 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f2405174640 1 --2- 172.21.15.27:0/4241242987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b9a10 0x7f24000b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f23fffff640 1 -- 172.21.15.27:0/4241242987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b9a10 msgr2=0x7f24000b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f23fffff640 1 --2- 172.21.15.27:0/4241242987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b9a10 0x7f24000b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f23fffff640 1 -- 172.21.15.27:0/4241242987 >> v1:172.21.15.69:6789/0 conn(0x7f240015df00 legacy=0x7f24000d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.069+0000 7f23fffff640 1 -- 172.21.15.27:0/4241242987 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24000ce470 con 0x7f24000b95a0 2024-03-06T07:31:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.070+0000 7f23fffff640 1 --2- 172.21.15.27:0/4241242987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b95a0 0x7f24000b5700 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009870 tx=0x7f23ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8ea7d3ab207cda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f23ff7fe640 1 -- 172.21.15.27:0/4241242987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec034070 con 0x7f24000b95a0 2024-03-06T07:31:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f23ff7fe640 1 -- 172.21.15.27:0/4241242987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ec030840 con 0x7f24000b95a0 2024-03-06T07:31:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f23ff7fe640 1 -- 172.21.15.27:0/4241242987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec030c10 con 0x7f24000b95a0 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f2406bd7640 1 -- 172.21.15.27:0/4241242987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b95a0 msgr2=0x7f24000b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f2406bd7640 1 --2- 172.21.15.27:0/4241242987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b95a0 0x7f24000b5700 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009870 tx=0x7f23ec02b980 comp rx=0 tx=0).stop 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f2406bd7640 1 -- 172.21.15.27:0/4241242987 shutdown_connections 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f2406bd7640 1 --2- 172.21.15.27:0/4241242987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b95a0 0x7f24000b5700 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f2406bd7640 1 --2- 172.21.15.27:0/4241242987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b9a10 0x7f24000b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f2406bd7640 1 -- 172.21.15.27:0/4241242987 >> 172.21.15.27:0/4241242987 conn(0x7f2400071960 msgr2=0x7f2400071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f2406bd7640 1 -- 172.21.15.27:0/4241242987 shutdown_connections 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.071+0000 7f2406bd7640 1 -- 172.21.15.27:0/4241242987 wait complete. 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.072+0000 7f2406bd7640 1 Processor -- start 2024-03-06T07:31:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.072+0000 7f2406bd7640 1 -- start start 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f2406bd7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b95a0 0x7f24001509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f2406bd7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b9a10 0x7f240014d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f2406bd7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240015df00 0x7f24000cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f2406bd7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24000cedf0 con 0x7f24000b9a10 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f2406bd7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24000cec70 con 0x7f240015df00 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f2406bd7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24000d3220 con 0x7f24000b95a0 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f23fffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b9a10 0x7f240014d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f23fffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b9a10 0x7f240014d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35862/0 (socket says 172.21.15.27:35862) 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f23fffff640 1 -- 172.21.15.27:0/784862952 learned_addr learned my addr 172.21.15.27:0/784862952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f2405174640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240015df00 0x7f24000cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.073+0000 7f2404973640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b95a0 0x7f24001509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f23fffff640 1 -- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b95a0 msgr2=0x7f24001509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f23fffff640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b95a0 0x7f24001509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f23fffff640 1 -- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240015df00 msgr2=0x7f24000cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f23fffff640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240015df00 0x7f24000cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f23fffff640 1 -- 172.21.15.27:0/784862952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24000c9b80 con 0x7f24000b9a10 2024-03-06T07:31:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f23fffff640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b9a10 0x7f240014d2b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f23ec000c00 tx=0x7f23ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f23fd7fa640 1 -- 172.21.15.27:0/784862952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec048070 con 0x7f24000b9a10 2024-03-06T07:31:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f23fd7fa640 1 -- 172.21.15.27:0/784862952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ec030640 con 0x7f24000b9a10 2024-03-06T07:31:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f23fd7fa640 1 -- 172.21.15.27:0/784862952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec034040 con 0x7f24000b9a10 2024-03-06T07:31:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24000c9e60 con 0x7f24000b9a10 2024-03-06T07:31:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.074+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24000ca420 con 0x7f24000b9a10 2024-03-06T07:31:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.075+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2400067ff0 con 0x7f24000b9a10 2024-03-06T07:31:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.076+0000 7f23fd7fa640 1 -- 172.21.15.27:0/784862952 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23ec04b070 con 0x7f24000b9a10 2024-03-06T07:31:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.076+0000 7f23fd7fa640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e406b500 0x7f23e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.077+0000 7f23fd7fa640 1 -- 172.21.15.27:0/784862952 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f23ec0ac150 con 0x7f24000b9a10 2024-03-06T07:31:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.077+0000 7f2404973640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e406b500 0x7f23e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.077+0000 7f2404973640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e406b500 0x7f23e406d9c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f23f40016e0 tx=0x7f23f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.081+0000 7f23fd7fa640 1 -- 172.21.15.27:0/784862952 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23ec030810 con 0x7f24000b9a10 2024-03-06T07:31:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.376+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f240015cc60 con 0x7f24000b9a10 2024-03-06T07:31:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.376+0000 7f23fd7fa640 1 -- 172.21.15.27:0/784862952 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3402 (secure 0 0 0) 0x7f23ec07aaf0 con 0x7f24000b9a10 2024-03-06T07:31:29.380 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:29.380 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9060,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3131328741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3131328741},{"type":"v1","addr":"172.21.15.27:6834","nonce":3131328741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:31:25.351787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 103 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e406b500 msgr2=0x7f23e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e406b500 0x7f23e406d9c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f23f40016e0 tx=0x7f23f4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b9a10 msgr2=0x7f240014d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b9a10 0x7f240014d2b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f23ec000c00 tx=0x7f23ec004730 comp rx=0 tx=0).stop 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 shutdown_connections 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f23e406b500 0x7f23e406d9c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240015df00 0x7f24000cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24000b9a10 0x7f240014d2b0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 --2- 172.21.15.27:0/784862952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24000b95a0 0x7f24001509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 >> 172.21.15.27:0/784862952 conn(0x7f2400071960 msgr2=0x7f24000d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 shutdown_connections 2024-03-06T07:31:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.378+0000 7f2406bd7640 1 -- 172.21.15.27:0/784862952 wait complete. 2024-03-06T07:31:29.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:29.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.496+0000 7fb87b5bc640 1 Processor -- start 2024-03-06T07:31:29.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.497+0000 7fb87b5bc640 1 -- start start 2024-03-06T07:31:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.497+0000 7fb87b5bc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8740b95a0 0x7fb8740b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.497+0000 7fb87b5bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 0x7fb8740b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.497+0000 7fb87b5bc640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8740cea30 con 0x7fb87415df00 2024-03-06T07:31:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.497+0000 7fb87b5bc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8740cf3c0 con 0x7fb8740b95a0 2024-03-06T07:31:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.497+0000 7fb87b5bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8740cf0e0 con 0x7fb8740b9a10 2024-03-06T07:31:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.497+0000 7fb878b57640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8740b95a0 0x7fb8740b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.497+0000 7fb879358640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb87415df00 0x7fb8740d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42344/0 (socket says 172.21.15.27:42344) 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.497+0000 7fb879358640 1 -- 172.21.15.27:0/93393584 learned_addr learned my addr 172.21.15.27:0/93393584 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.498+0000 7fb879b59640 1 --2- 172.21.15.27:0/93393584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 0x7fb8740b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.498+0000 7fb878b57640 1 -- 172.21.15.27:0/93393584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 msgr2=0x7fb8740b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.498+0000 7fb878b57640 1 --2- 172.21.15.27:0/93393584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 0x7fb8740b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.498+0000 7fb878b57640 1 -- 172.21.15.27:0/93393584 >> v1:172.21.15.27:6789/0 conn(0x7fb87415df00 legacy=0x7fb8740d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.498+0000 7fb878b57640 1 -- 172.21.15.27:0/93393584 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8740ce470 con 0x7fb8740b95a0 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.498+0000 7fb879b59640 1 --2- 172.21.15.27:0/93393584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 0x7fb8740b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb878b57640 1 --2- 172.21.15.27:0/93393584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8740b95a0 0x7fb8740b5700 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fb8700097a0 tx=0x7fb87002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc76930b27bfd81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb86bfff640 1 -- 172.21.15.27:0/93393584 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb870035020 con 0x7fb8740b95a0 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb86bfff640 1 -- 172.21.15.27:0/93393584 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb870031830 con 0x7fb8740b95a0 2024-03-06T07:31:29.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb86bfff640 1 -- 172.21.15.27:0/93393584 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb870031c00 con 0x7fb8740b95a0 2024-03-06T07:31:29.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb87b5bc640 1 -- 172.21.15.27:0/93393584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8740b95a0 msgr2=0x7fb8740b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb87b5bc640 1 --2- 172.21.15.27:0/93393584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8740b95a0 0x7fb8740b5700 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fb8700097a0 tx=0x7fb87002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb87b5bc640 1 -- 172.21.15.27:0/93393584 shutdown_connections 2024-03-06T07:31:29.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb87b5bc640 1 --2- 172.21.15.27:0/93393584 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 0x7fb8740b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb87b5bc640 1 --2- 172.21.15.27:0/93393584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8740b95a0 0x7fb8740b5700 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb87b5bc640 1 -- 172.21.15.27:0/93393584 >> 172.21.15.27:0/93393584 conn(0x7fb874071960 msgr2=0x7fb874071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb87b5bc640 1 -- 172.21.15.27:0/93393584 shutdown_connections 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.499+0000 7fb87b5bc640 1 -- 172.21.15.27:0/93393584 wait complete. 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.501+0000 7fb87b5bc640 1 Processor -- start 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.501+0000 7fb87b5bc640 1 -- start start 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.501+0000 7fb87b5bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8740b95a0 0x7fb8740cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.501+0000 7fb87b5bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 0x7fb87415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.501+0000 7fb87b5bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb87415df00 0x7fb8740c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.501+0000 7fb87b5bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8740cedf0 con 0x7fb87415df00 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.501+0000 7fb87b5bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8740cec70 con 0x7fb8740b95a0 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.501+0000 7fb87b5bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8740d3220 con 0x7fb8740b9a10 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.501+0000 7fb879b59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb87415df00 0x7fb8740c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879b59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb87415df00 0x7fb8740c9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35880/0 (socket says 172.21.15.27:35880) 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879b59640 1 -- 172.21.15.27:0/3313777615 learned_addr learned my addr 172.21.15.27:0/3313777615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb878b57640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 0x7fb87415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879358640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8740b95a0 0x7fb8740cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879b59640 1 -- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 msgr2=0x7fb87415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879b59640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 0x7fb87415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879b59640 1 -- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8740b95a0 msgr2=0x7fb8740cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879b59640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8740b95a0 0x7fb8740cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879b59640 1 -- 172.21.15.27:0/3313777615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8740cb330 con 0x7fb87415df00 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879358640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8740b95a0 0x7fb8740cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb879b59640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb87415df00 0x7fb8740c9b00 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009fd0 tx=0x7fb86c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb869ffb640 1 -- 172.21.15.27:0/3313777615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c01a070 con 0x7fb87415df00 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb869ffb640 1 -- 172.21.15.27:0/3313777615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c003bb0 con 0x7fb87415df00 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb869ffb640 1 -- 172.21.15.27:0/3313777615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c004030 con 0x7fb87415df00 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.502+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8740cb670 con 0x7fb87415df00 2024-03-06T07:31:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.503+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8740cbeb0 con 0x7fb87415df00 2024-03-06T07:31:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.503+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8740cc0b0 con 0x7fb87415df00 2024-03-06T07:31:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.504+0000 7fb869ffb640 1 -- 172.21.15.27:0/3313777615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb86c01d070 con 0x7fb87415df00 2024-03-06T07:31:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.504+0000 7fb869ffb640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb85806b4d0 0x7fb85806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.504+0000 7fb869ffb640 1 -- 172.21.15.27:0/3313777615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb86c08a550 con 0x7fb87415df00 2024-03-06T07:31:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.504+0000 7fb879358640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb85806b4d0 0x7fb85806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.505+0000 7fb879358640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb85806b4d0 0x7fb85806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb874068c90 tx=0x7fb864009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.513+0000 7fb869ffb640 1 -- 172.21.15.27:0/3313777615 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8740cc0b0 con 0x7fb87415df00 2024-03-06T07:31:29.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.695+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb874067ff0 con 0x7fb87415df00 2024-03-06T07:31:29.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.696+0000 7fb869ffb640 1 -- 172.21.15.27:0/3313777615 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+16648 (secure 0 0 0) 0x7fb8740cb670 con 0x7fb87415df00 2024-03-06T07:31:29.699 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:29.699 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":136,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:30:37.955761+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:20.059798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.689553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.a","mappings":[{"from":2,"to":0}]},{"pgid":"19.1f","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb85806b4d0 msgr2=0x7fb85806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb85806b4d0 0x7fb85806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb874068c90 tx=0x7fb864009040 comp rx=0 tx=0).stop 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb87415df00 msgr2=0x7fb8740c9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb87415df00 0x7fb8740c9b00 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009fd0 tx=0x7fb86c00d560 comp rx=0 tx=0).stop 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 shutdown_connections 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb85806b4d0 0x7fb85806d990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb87415df00 0x7fb8740c9b00 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb8740b9a10 0x7fb87415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 --2- 172.21.15.27:0/3313777615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8740b95a0 0x7fb8740cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 >> 172.21.15.27:0/3313777615 conn(0x7fb874071960 msgr2=0x7fb8740b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 shutdown_connections 2024-03-06T07:31:29.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.698+0000 7fb87b5bc640 1 -- 172.21.15.27:0/3313777615 wait complete. 2024-03-06T07:31:29.712 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:20.059798+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:20.689553+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:29.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:29.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.819+0000 7f322d5ac640 1 Processor -- start 2024-03-06T07:31:29.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.819+0000 7f322d5ac640 1 -- start start 2024-03-06T07:31:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f322d5ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280cdcd0 0x7f32280b1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f322d5ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32280b2cd0 0x7f32280b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f322d5ac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3228161a40 con 0x7f32280b1fd0 2024-03-06T07:31:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f322d5ac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32280d11e0 con 0x7f32280b2cd0 2024-03-06T07:31:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f322d5ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322806c2d0 con 0x7f32280cdcd0 2024-03-06T07:31:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f32267fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32280b2cd0 0x7f32280b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f32267fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32280b2cd0 0x7f32280b1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35888/0 (socket says 172.21.15.27:35888) 2024-03-06T07:31:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f32267fc640 1 -- 172.21.15.27:0/1274509418 learned_addr learned my addr 172.21.15.27:0/1274509418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:29.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f32277fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f32280b1fd0 0x7f32280ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42346/0 (socket says 172.21.15.27:42346) 2024-03-06T07:31:29.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.820+0000 7f3226ffd640 1 --2- 172.21.15.27:0/1274509418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280cdcd0 0x7f32280b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.821+0000 7f32267fc640 1 -- 172.21.15.27:0/1274509418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280cdcd0 msgr2=0x7f32280b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.821+0000 7f32267fc640 1 --2- 172.21.15.27:0/1274509418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280cdcd0 0x7f32280b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.821+0000 7f32267fc640 1 -- 172.21.15.27:0/1274509418 >> v1:172.21.15.27:6789/0 conn(0x7f32280b1fd0 legacy=0x7f32280ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.821+0000 7f32267fc640 1 -- 172.21.15.27:0/1274509418 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32280b2a20 con 0x7f32280b2cd0 2024-03-06T07:31:29.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.821+0000 7f3225ffb640 1 -- 172.21.15.27:0/1274509418 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414474769 0 0) 0x7f3228161a40 con 0x7f32280b1fd0 2024-03-06T07:31:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.821+0000 7f32267fc640 1 --2- 172.21.15.27:0/1274509418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b2cd0 0x7f32280b1a90 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f321c009870 tx=0x7f321c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc83ebd60007ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f3225ffb640 1 -- 172.21.15.27:0/1274509418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c035020 con 0x7f32280b2cd0 2024-03-06T07:31:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f3225ffb640 1 -- 172.21.15.27:0/1274509418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c031640 con 0x7f32280b2cd0 2024-03-06T07:31:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f3225ffb640 1 -- 172.21.15.27:0/1274509418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c02d040 con 0x7f32280b2cd0 2024-03-06T07:31:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f322d5ac640 1 -- 172.21.15.27:0/1274509418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b2cd0 msgr2=0x7f32280b1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f322d5ac640 1 --2- 172.21.15.27:0/1274509418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b2cd0 0x7f32280b1a90 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f321c009870 tx=0x7f321c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f322d5ac640 1 -- 172.21.15.27:0/1274509418 shutdown_connections 2024-03-06T07:31:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f322d5ac640 1 --2- 172.21.15.27:0/1274509418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b2cd0 0x7f32280b1a90 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f322d5ac640 1 --2- 172.21.15.27:0/1274509418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280cdcd0 0x7f32280b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f322d5ac640 1 -- 172.21.15.27:0/1274509418 >> 172.21.15.27:0/1274509418 conn(0x7f32280b3700 msgr2=0x7f32280b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f322d5ac640 1 -- 172.21.15.27:0/1274509418 shutdown_connections 2024-03-06T07:31:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.822+0000 7f322d5ac640 1 -- 172.21.15.27:0/1274509418 wait complete. 2024-03-06T07:31:29.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.823+0000 7f322d5ac640 1 Processor -- start 2024-03-06T07:31:29.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.823+0000 7f322d5ac640 1 -- start start 2024-03-06T07:31:29.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f322d5ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b1fd0 0x7f32281f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f322d5ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280b2cd0 0x7f32281f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f322d5ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32280cdcd0 0x7f32281f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f322d5ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32280d0f00 con 0x7f32280b1fd0 2024-03-06T07:31:29.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f322d5ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32280d0c20 con 0x7f32280cdcd0 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f322d5ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32280cc1c0 con 0x7f32280b2cd0 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f3226ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b1fd0 0x7f32281f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f3226ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b1fd0 0x7f32281f4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35902/0 (socket says 172.21.15.27:35902) 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f3226ffd640 1 -- 172.21.15.27:0/1969851612 learned_addr learned my addr 172.21.15.27:0/1969851612 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f32267fc640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280b2cd0 0x7f32281f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.824+0000 7f32277fe640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32280cdcd0 0x7f32281f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f3226ffd640 1 -- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280b2cd0 msgr2=0x7f32281f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f3226ffd640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280b2cd0 0x7f32281f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f3226ffd640 1 -- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32280cdcd0 msgr2=0x7f32281f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f3226ffd640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32280cdcd0 0x7f32281f69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f3226ffd640 1 -- 172.21.15.27:0/1969851612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32281f4b60 con 0x7f32280b1fd0 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f3226ffd640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b1fd0 0x7f32281f4d60 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f321800d620 tx=0x7f321800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f320f7fe640 1 -- 172.21.15.27:0/1969851612 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321800b620 con 0x7f32280b1fd0 2024-03-06T07:31:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f320f7fe640 1 -- 172.21.15.27:0/1969851612 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3218019070 con 0x7f32280b1fd0 2024-03-06T07:31:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32281f6f20 con 0x7f32280b1fd0 2024-03-06T07:31:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f320f7fe640 1 -- 172.21.15.27:0/1969851612 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321800ba00 con 0x7f32280b1fd0 2024-03-06T07:31:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.825+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32280b4e00 con 0x7f32280b1fd0 2024-03-06T07:31:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.826+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322809fcf0 con 0x7f32280b1fd0 2024-03-06T07:31:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.827+0000 7f320f7fe640 1 -- 172.21.15.27:0/1969851612 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f321800bba0 con 0x7f32280b1fd0 2024-03-06T07:31:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.827+0000 7f320f7fe640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f321406b4d0 0x7f321406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.827+0000 7f320f7fe640 1 -- 172.21.15.27:0/1969851612 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f321808a510 con 0x7f32280b1fd0 2024-03-06T07:31:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.827+0000 7f32267fc640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f321406b4d0 0x7f321406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.828+0000 7f32267fc640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f321406b4d0 0x7f321406d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f321c036000 tx=0x7f321c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:29.836+0000 7f320f7fe640 1 -- 172.21.15.27:0/1969851612 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32281f4b60 con 0x7f32280b1fd0 2024-03-06T07:31:30.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.018+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32280d1690 con 0x7f32280b1fd0 2024-03-06T07:31:30.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.018+0000 7f320f7fe640 1 -- 172.21.15.27:0/1969851612 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v136) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3218088e20 con 0x7f32280b1fd0 2024-03-06T07:31:30.021 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:31:30.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f321406b4d0 msgr2=0x7f321406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f321406b4d0 0x7f321406d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f321c036000 tx=0x7f321c008030 comp rx=0 tx=0).stop 2024-03-06T07:31:30.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b1fd0 msgr2=0x7f32281f4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b1fd0 0x7f32281f4d60 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f321800d620 tx=0x7f321800dae0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 shutdown_connections 2024-03-06T07:31:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f321406b4d0 0x7f321406d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32280cdcd0 0x7f32281f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f32280b2cd0 0x7f32281f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 --2- 172.21.15.27:0/1969851612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32280b1fd0 0x7f32281f4d60 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.020+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 >> 172.21.15.27:0/1969851612 conn(0x7f32280b3700 msgr2=0x7f322806fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.021+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 shutdown_connections 2024-03-06T07:31:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.021+0000 7f322d5ac640 1 -- 172.21.15.27:0/1969851612 wait complete. 2024-03-06T07:31:30.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.142+0000 7f7f56bc7640 1 Processor -- start 2024-03-06T07:31:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.143+0000 7f7f56bc7640 1 -- start start 2024-03-06T07:31:30.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.143+0000 7f7f56bc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f50071540 0x7f7f50071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.143+0000 7f7f56bc7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f500cee80 0x7f7f50168900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.143+0000 7f7f56bc7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f50071e60 con 0x7f7f500b2d30 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.143+0000 7f7f56bc7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f500d3230 con 0x7f7f500cee80 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.143+0000 7f7f56bc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500d4960 con 0x7f7f50071540 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f54963640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f50071540 0x7f7f50071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f54963640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f50071540 0x7f7f50071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:40468/0 (socket says 172.21.15.27:40468) 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f55164640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f500cee80 0x7f7f50168900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f54963640 1 -- 172.21.15.27:0/4064653486 learned_addr learned my addr 172.21.15.27:0/4064653486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f4ffff640 1 --1- 172.21.15.27:0/4064653486 >> v1:172.21.15.69:6789/0 conn(0x7f7f500b2d30 0x7f7f500ce770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:44532/0 (socket says 172.21.15.27:44532) 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f4f7fe640 1 -- 172.21.15.27:0/4064653486 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773536229 0 0) 0x7f7f50071e60 con 0x7f7f500b2d30 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f54963640 1 -- 172.21.15.27:0/4064653486 >> v2:172.21.15.69:3300/0 conn(0x7f7f500cee80 msgr2=0x7f7f50168900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f54963640 1 --2- 172.21.15.27:0/4064653486 >> v2:172.21.15.69:3300/0 conn(0x7f7f500cee80 0x7f7f50168900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f54963640 1 -- 172.21.15.27:0/4064653486 >> v1:172.21.15.69:6789/0 conn(0x7f7f500b2d30 legacy=0x7f7f500ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.144+0000 7f7f54963640 1 -- 172.21.15.27:0/4064653486 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f500d36c0 con 0x7f7f50071540 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.145+0000 7f7f54963640 1 --2- 172.21.15.27:0/4064653486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f50071540 0x7f7f50071920 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7f44009870 tx=0x7f7f4402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e3fe38acfea44a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f4f7fe640 1 -- 172.21.15.27:0/4064653486 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f44034070 con 0x7f7f50071540 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f4f7fe640 1 -- 172.21.15.27:0/4064653486 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f44030840 con 0x7f7f50071540 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f4f7fe640 1 -- 172.21.15.27:0/4064653486 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f44030c10 con 0x7f7f50071540 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f56bc7640 1 -- 172.21.15.27:0/4064653486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f50071540 msgr2=0x7f7f50071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f56bc7640 1 --2- 172.21.15.27:0/4064653486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f50071540 0x7f7f50071920 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7f44009870 tx=0x7f7f4402b980 comp rx=0 tx=0).stop 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f56bc7640 1 -- 172.21.15.27:0/4064653486 shutdown_connections 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f56bc7640 1 --2- 172.21.15.27:0/4064653486 >> v2:172.21.15.69:3300/0 conn(0x7f7f500cee80 0x7f7f50168900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f56bc7640 1 --2- 172.21.15.27:0/4064653486 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f50071540 0x7f7f50071920 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f56bc7640 1 -- 172.21.15.27:0/4064653486 >> 172.21.15.27:0/4064653486 conn(0x7f7f500d3d60 msgr2=0x7f7f500d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f56bc7640 1 -- 172.21.15.27:0/4064653486 shutdown_connections 2024-03-06T07:31:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.146+0000 7f7f56bc7640 1 -- 172.21.15.27:0/4064653486 wait complete. 2024-03-06T07:31:30.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.147+0000 7f7f56bc7640 1 Processor -- start 2024-03-06T07:31:30.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.147+0000 7f7f56bc7640 1 -- start start 2024-03-06T07:31:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f56bc7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f50071540 0x7f7f501e4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f56bc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f500b2d30 0x7f7f500c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f56bc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f500cee80 0x7f7f501e7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f56bc7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500d2be0 con 0x7f7f500cee80 2024-03-06T07:31:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f56bc7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500d2900 con 0x7f7f50071540 2024-03-06T07:31:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f56bc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500d2ec0 con 0x7f7f500b2d30 2024-03-06T07:31:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f55164640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f500cee80 0x7f7f501e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f55164640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f500cee80 0x7f7f501e7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35918/0 (socket says 172.21.15.27:35918) 2024-03-06T07:31:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f55164640 1 -- 172.21.15.27:0/1312021734 learned_addr learned my addr 172.21.15.27:0/1312021734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.148+0000 7f7f54963640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f50071540 0x7f7f501e4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f4ffff640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f500b2d30 0x7f7f500c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f55164640 1 -- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f500b2d30 msgr2=0x7f7f500c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f55164640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f500b2d30 0x7f7f500c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f55164640 1 -- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f50071540 msgr2=0x7f7f501e4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f55164640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f50071540 0x7f7f501e4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f55164640 1 -- 172.21.15.27:0/1312021734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f50148320 con 0x7f7f500cee80 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f55164640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f500cee80 0x7f7f501e7480 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009dd0 tx=0x7f7f4800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f4d7fa640 1 -- 172.21.15.27:0/1312021734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48018020 con 0x7f7f500cee80 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f4d7fa640 1 -- 172.21.15.27:0/1312021734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f48014480 con 0x7f7f500cee80 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f501485c0 con 0x7f7f500cee80 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f4d7fa640 1 -- 172.21.15.27:0/1312021734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48014870 con 0x7f7f500cee80 2024-03-06T07:31:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.149+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f50148b80 con 0x7f7f500cee80 2024-03-06T07:31:30.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.150+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f50072070 con 0x7f7f500cee80 2024-03-06T07:31:30.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.151+0000 7f7f4d7fa640 1 -- 172.21.15.27:0/1312021734 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f48005d10 con 0x7f7f500cee80 2024-03-06T07:31:30.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.151+0000 7f7f4d7fa640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f3406b500 0x7f7f3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.151+0000 7f7f4d7fa640 1 -- 172.21.15.27:0/1312021734 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f4808ac10 con 0x7f7f500cee80 2024-03-06T07:31:30.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.151+0000 7f7f54963640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f3406b500 0x7f7f3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.152+0000 7f7f54963640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f3406b500 0x7f7f3406d9c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f7f44036000 tx=0x7f7f4403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.156+0000 7f7f4d7fa640 1 -- 172.21.15.27:0/1312021734 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f48022070 con 0x7f7f500cee80 2024-03-06T07:31:30.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.338+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7f500b4ba0 con 0x7f7f500cee80 2024-03-06T07:31:30.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.338+0000 7f7f4d7fa640 1 -- 172.21.15.27:0/1312021734 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v136) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7f48010040 con 0x7f7f500cee80 2024-03-06T07:31:30.341 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f3406b500 msgr2=0x7f7f3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f3406b500 0x7f7f3406d9c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f7f44036000 tx=0x7f7f4403c040 comp rx=0 tx=0).stop 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f500cee80 msgr2=0x7f7f501e7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f500cee80 0x7f7f501e7480 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009dd0 tx=0x7f7f4800c490 comp rx=0 tx=0).stop 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 shutdown_connections 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7f3406b500 0x7f7f3406d9c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f500cee80 0x7f7f501e7480 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7f500b2d30 0x7f7f500c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 --2- 172.21.15.27:0/1312021734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f50071540 0x7f7f501e4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 >> 172.21.15.27:0/1312021734 conn(0x7f7f500d3d60 msgr2=0x7f7f500c96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 shutdown_connections 2024-03-06T07:31:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.340+0000 7f7f56bc7640 1 -- 172.21.15.27:0/1312021734 wait complete. 2024-03-06T07:31:30.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:30.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.461+0000 7fca7677a640 1 Processor -- start 2024-03-06T07:31:30.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.461+0000 7fca7677a640 1 -- start start 2024-03-06T07:31:30.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca7677a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca700ceea0 0x7fca700d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca7677a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca700b6d10 0x7fca700b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca7677a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca700d3740 con 0x7fca7015dd90 2024-03-06T07:31:30.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca7677a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca700d3400 con 0x7fca700b6d10 2024-03-06T07:31:30.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca7677a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca700d3120 con 0x7fca700ceea0 2024-03-06T07:31:30.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca74d17640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca700b6d10 0x7fca700b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca74d17640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca700b6d10 0x7fca700b71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35930/0 (socket says 172.21.15.27:35930) 2024-03-06T07:31:30.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca74d17640 1 -- 172.21.15.27:0/3272456852 learned_addr learned my addr 172.21.15.27:0/3272456852 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:30.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca6f7fe640 1 --2- 172.21.15.27:0/3272456852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca700ceea0 0x7fca700d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca74d17640 1 -- 172.21.15.27:0/3272456852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca700ceea0 msgr2=0x7fca700d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca74d17640 1 --2- 172.21.15.27:0/3272456852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca700ceea0 0x7fca700d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca74d17640 1 -- 172.21.15.27:0/3272456852 >> v1:172.21.15.69:6789/0 conn(0x7fca7015dd90 legacy=0x7fca700d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca74d17640 1 -- 172.21.15.27:0/3272456852 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700d39e0 con 0x7fca700b6d10 2024-03-06T07:31:30.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.462+0000 7fca6f7fe640 1 --2- 172.21.15.27:0/3272456852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca700ceea0 0x7fca700d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:30.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.463+0000 7fca74d17640 1 --2- 172.21.15.27:0/3272456852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700b6d10 0x7fca700b71a0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fca68009e80 tx=0x7fca6802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf1ae0115ea926b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.463+0000 7fca6effd640 1 -- 172.21.15.27:0/3272456852 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6802caf0 con 0x7fca700b6d10 2024-03-06T07:31:30.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.463+0000 7fca6effd640 1 -- 172.21.15.27:0/3272456852 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6802cc90 con 0x7fca700b6d10 2024-03-06T07:31:30.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.463+0000 7fca6effd640 1 -- 172.21.15.27:0/3272456852 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68034610 con 0x7fca700b6d10 2024-03-06T07:31:30.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.464+0000 7fca7677a640 1 -- 172.21.15.27:0/3272456852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700b6d10 msgr2=0x7fca700b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.464+0000 7fca7677a640 1 --2- 172.21.15.27:0/3272456852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700b6d10 0x7fca700b71a0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fca68009e80 tx=0x7fca6802bf50 comp rx=0 tx=0).stop 2024-03-06T07:31:30.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.464+0000 7fca7677a640 1 -- 172.21.15.27:0/3272456852 shutdown_connections 2024-03-06T07:31:30.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.464+0000 7fca7677a640 1 --2- 172.21.15.27:0/3272456852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700b6d10 0x7fca700b71a0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.464+0000 7fca7677a640 1 --2- 172.21.15.27:0/3272456852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca700ceea0 0x7fca700d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.464+0000 7fca7677a640 1 -- 172.21.15.27:0/3272456852 >> 172.21.15.27:0/3272456852 conn(0x7fca70071960 msgr2=0x7fca70071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.464+0000 7fca7677a640 1 -- 172.21.15.27:0/3272456852 shutdown_connections 2024-03-06T07:31:30.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.464+0000 7fca7677a640 1 -- 172.21.15.27:0/3272456852 wait complete. 2024-03-06T07:31:30.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.465+0000 7fca7677a640 1 Processor -- start 2024-03-06T07:31:30.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.465+0000 7fca7677a640 1 -- start start 2024-03-06T07:31:30.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca7677a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca700b6d10 0x7fca7014f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca7677a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700ceea0 0x7fca700ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca7677a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca7015dd90 0x7fca701f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca7677a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca700d48f0 con 0x7fca700ceea0 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca7677a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7015e170 con 0x7fca700b6d10 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca7677a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca700d4a70 con 0x7fca7015dd90 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca6f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700ceea0 0x7fca700ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca6f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700ceea0 0x7fca700ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35936/0 (socket says 172.21.15.27:35936) 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca6f7fe640 1 -- 172.21.15.27:0/1531468564 learned_addr learned my addr 172.21.15.27:0/1531468564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca74d17640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca7015dd90 0x7fca701f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.466+0000 7fca6ffff640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca700b6d10 0x7fca7014f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca6f7fe640 1 -- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca7015dd90 msgr2=0x7fca701f6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca6f7fe640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca7015dd90 0x7fca701f6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca6f7fe640 1 -- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca700b6d10 msgr2=0x7fca7014f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca6f7fe640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca700b6d10 0x7fca7014f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca6f7fe640 1 -- 172.21.15.27:0/1531468564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700ca5f0 con 0x7fca700ceea0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca6f7fe640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700ceea0 0x7fca700ca0b0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c7f0 tx=0x7fca6000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca6cff9640 1 -- 172.21.15.27:0/1531468564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60015070 con 0x7fca700ceea0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca6cff9640 1 -- 172.21.15.27:0/1531468564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca60007a30 con 0x7fca700ceea0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca6cff9640 1 -- 172.21.15.27:0/1531468564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60007e00 con 0x7fca700ceea0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca701f7870 con 0x7fca700ceea0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.467+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca700c98b0 con 0x7fca700ceea0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.468+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca700c96a0 con 0x7fca700ceea0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.469+0000 7fca6cff9640 1 -- 172.21.15.27:0/1531468564 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca60018070 con 0x7fca700ceea0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.469+0000 7fca6cff9640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca5c06b4d0 0x7fca5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.469+0000 7fca6cff9640 1 -- 172.21.15.27:0/1531468564 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fca6008a140 con 0x7fca700ceea0 2024-03-06T07:31:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.469+0000 7fca6ffff640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca5c06b4d0 0x7fca5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.470+0000 7fca6ffff640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca5c06b4d0 0x7fca5c06d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fca64006fd0 tx=0x7fca64009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.473+0000 7fca6cff9640 1 -- 172.21.15.27:0/1531468564 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca60058930 con 0x7fca700ceea0 2024-03-06T07:31:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.687+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fca7004f410 con 0x7fca700ceea0 2024-03-06T07:31:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.688+0000 7fca6cff9640 1 -- 172.21.15.27:0/1531468564 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v136) v1 ==== 100+0+11 (secure 0 0 0) 0x7fca6005c9d0 con 0x7fca700ceea0 2024-03-06T07:31:30.690 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca5c06b4d0 msgr2=0x7fca5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca5c06b4d0 0x7fca5c06d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fca64006fd0 tx=0x7fca64009060 comp rx=0 tx=0).stop 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700ceea0 msgr2=0x7fca700ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700ceea0 0x7fca700ca0b0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c7f0 tx=0x7fca6000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 shutdown_connections 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fca5c06b4d0 0x7fca5c06d990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fca7015dd90 0x7fca701f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca700ceea0 0x7fca700ca0b0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 --2- 172.21.15.27:0/1531468564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca700b6d10 0x7fca7014f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 >> 172.21.15.27:0/1531468564 conn(0x7fca70071960 msgr2=0x7fca700b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 shutdown_connections 2024-03-06T07:31:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.690+0000 7fca7677a640 1 -- 172.21.15.27:0/1531468564 wait complete. 2024-03-06T07:31:30.703 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:30.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.808+0000 7f73bc42a640 1 Processor -- start 2024-03-06T07:31:30.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.808+0000 7f73bc42a640 1 -- start start 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73bc42a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b4071540 0x7f73b40b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73bc42a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f73b40d3400 0x7f73b4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73bc42a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b4071e60 con 0x7f73b40b2b50 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73bc42a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b40d2f70 con 0x7f73b40d3400 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73bc42a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40d4be0 con 0x7f73b4071540 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73ba9c7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f73b40b2b50 0x7f73b40d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42350/0 (socket says 172.21.15.27:42350) 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73ba9c7640 1 -- 172.21.15.27:0/1060668452 learned_addr learned my addr 172.21.15.27:0/1060668452 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73b99c5640 1 --2- 172.21.15.27:0/1060668452 >> v2:172.21.15.69:3300/0 conn(0x7f73b40d3400 0x7f73b4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73ba1c6640 1 --2- 172.21.15.27:0/1060668452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b4071540 0x7f73b40b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73b91c4640 1 -- 172.21.15.27:0/1060668452 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674802804 0 0) 0x7f73b4071e60 con 0x7f73b40b2b50 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.809+0000 7f73b91c4640 1 -- 172.21.15.27:0/1060668452 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a0003520 con 0x7f73b40b2b50 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.810+0000 7f73b99c5640 1 -- 172.21.15.27:0/1060668452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b4071540 msgr2=0x7f73b40b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.810+0000 7f73b99c5640 1 --2- 172.21.15.27:0/1060668452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b4071540 0x7f73b40b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.810+0000 7f73b99c5640 1 -- 172.21.15.27:0/1060668452 >> v1:172.21.15.27:6789/0 conn(0x7f73b40b2b50 legacy=0x7f73b40d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.810+0000 7f73b99c5640 1 -- 172.21.15.27:0/1060668452 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b40ce610 con 0x7f73b40d3400 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.810+0000 7f73ba1c6640 1 --2- 172.21.15.27:0/1060668452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b4071540 0x7f73b40b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.810+0000 7f73b99c5640 1 --2- 172.21.15.27:0/1060668452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b40d3400 0x7f73b4071920 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f73b00097a0 tx=0x7f73b002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4918d52c5644744b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73b91c4640 1 -- 172.21.15.27:0/1060668452 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0035020 con 0x7f73b40d3400 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73b91c4640 1 -- 172.21.15.27:0/1060668452 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0031810 con 0x7f73b40d3400 2024-03-06T07:31:30.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73b91c4640 1 -- 172.21.15.27:0/1060668452 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0031bc0 con 0x7f73b40d3400 2024-03-06T07:31:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73bc42a640 1 -- 172.21.15.27:0/1060668452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b40d3400 msgr2=0x7f73b4071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73bc42a640 1 --2- 172.21.15.27:0/1060668452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b40d3400 0x7f73b4071920 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f73b00097a0 tx=0x7f73b002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73bc42a640 1 -- 172.21.15.27:0/1060668452 shutdown_connections 2024-03-06T07:31:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73bc42a640 1 --2- 172.21.15.27:0/1060668452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b40d3400 0x7f73b4071920 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73bc42a640 1 --2- 172.21.15.27:0/1060668452 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b4071540 0x7f73b40b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73bc42a640 1 -- 172.21.15.27:0/1060668452 >> 172.21.15.27:0/1060668452 conn(0x7f73b40d3ca0 msgr2=0x7f73b40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73bc42a640 1 -- 172.21.15.27:0/1060668452 shutdown_connections 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.811+0000 7f73bc42a640 1 -- 172.21.15.27:0/1060668452 wait complete. 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.812+0000 7f73bc42a640 1 Processor -- start 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.812+0000 7f73bc42a640 1 -- start start 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73bc42a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b4071540 0x7f73b41f5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73bc42a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b2b50 0x7f73b414ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73bc42a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b40d3400 0x7f73b41e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73bc42a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40ce8f0 con 0x7f73b40d3400 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73bc42a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40d2c00 con 0x7f73b4071540 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73bc42a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b415e370 con 0x7f73b40b2b50 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73ba9c7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b40d3400 0x7f73b41e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73ba9c7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b40d3400 0x7f73b41e6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35944/0 (socket says 172.21.15.27:35944) 2024-03-06T07:31:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73ba9c7640 1 -- 172.21.15.27:0/1797642737 learned_addr learned my addr 172.21.15.27:0/1797642737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73ba1c6640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b4071540 0x7f73b41f5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.813+0000 7f73b99c5640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b2b50 0x7f73b414ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73ba9c7640 1 -- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b2b50 msgr2=0x7f73b414ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73ba9c7640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b2b50 0x7f73b414ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73ba9c7640 1 -- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b4071540 msgr2=0x7f73b41f5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73ba9c7640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b4071540 0x7f73b41f5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73ba9c7640 1 -- 172.21.15.27:0/1797642737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b41e7ba0 con 0x7f73b40d3400 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73ba9c7640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b40d3400 0x7f73b41e6dc0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00cb40 tx=0x7f73ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73aaffd640 1 -- 172.21.15.27:0/1797642737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac017070 con 0x7f73b40d3400 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73aaffd640 1 -- 172.21.15.27:0/1797642737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac00aca0 con 0x7f73b40d3400 2024-03-06T07:31:30.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b4150480 con 0x7f73b40d3400 2024-03-06T07:31:30.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73aaffd640 1 -- 172.21.15.27:0/1797642737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac012620 con 0x7f73b40d3400 2024-03-06T07:31:30.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.814+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b41f60b0 con 0x7f73b40d3400 2024-03-06T07:31:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.815+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b4068700 con 0x7f73b40d3400 2024-03-06T07:31:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.816+0000 7f73aaffd640 1 -- 172.21.15.27:0/1797642737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73ac01a070 con 0x7f73b40d3400 2024-03-06T07:31:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.816+0000 7f73aaffd640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b500 0x7f739806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.816+0000 7f73aaffd640 1 -- 172.21.15.27:0/1797642737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f73ac089ff0 con 0x7f73b40d3400 2024-03-06T07:31:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.817+0000 7f73ba1c6640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b500 0x7f739806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.817+0000 7f73ba1c6640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b500 0x7f739806d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009560 tx=0x7f73a4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:30.821+0000 7f73aaffd640 1 -- 172.21.15.27:0/1797642737 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73b41e7ba0 con 0x7f73b40d3400 2024-03-06T07:31:31.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.040+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73b40b95f0 con 0x7f73b40d3400 2024-03-06T07:31:31.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.041+0000 7f73aaffd640 1 -- 172.21.15.27:0/1797642737 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3402 (secure 0 0 0) 0x7f73ac058760 con 0x7f73b40d3400 2024-03-06T07:31:31.043 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:31.044 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9060,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3131328741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3131328741},{"type":"v1","addr":"172.21.15.27:6834","nonce":3131328741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:31:25.351787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:31.044 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 103 2024-03-06T07:31:31.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b500 msgr2=0x7f739806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b500 0x7f739806d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009560 tx=0x7f73a4009190 comp rx=0 tx=0).stop 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b40d3400 msgr2=0x7f73b41e6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b40d3400 0x7f73b41e6dc0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00cb40 tx=0x7f73ac00a060 comp rx=0 tx=0).stop 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 shutdown_connections 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f739806b500 0x7f739806d9c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b40d3400 0x7f73b41e6dc0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f73b40b2b50 0x7f73b414ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 --2- 172.21.15.27:0/1797642737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73b4071540 0x7f73b41f5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 >> 172.21.15.27:0/1797642737 conn(0x7f73b40d3ca0 msgr2=0x7f73b415f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 shutdown_connections 2024-03-06T07:31:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.043+0000 7f73bc42a640 1 -- 172.21.15.27:0/1797642737 wait complete. 2024-03-06T07:31:31.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.161+0000 7f8bb508a640 1 Processor -- start 2024-03-06T07:31:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.161+0000 7f8bb508a640 1 -- start start 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8bb508a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b95a0 0x7f8bb00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8bb508a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bb00b9a10 0x7f8bb00b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8bb508a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb00cea30 con 0x7f8bb015df00 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8bb508a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb00cf3c0 con 0x7f8bb00b9a10 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8bb508a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb00cf0e0 con 0x7f8bb00b95a0 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8baf59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bb00b9a10 0x7f8bb00b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8baf59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bb00b9a10 0x7f8bb00b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35952/0 (socket says 172.21.15.27:35952) 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8baf59e640 1 -- 172.21.15.27:0/3386114656 learned_addr learned my addr 172.21.15.27:0/3386114656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8bae59c640 1 --2- 172.21.15.27:0/3386114656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b95a0 0x7f8bb00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8baf59e640 1 -- 172.21.15.27:0/3386114656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b95a0 msgr2=0x7f8bb00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8baf59e640 1 --2- 172.21.15.27:0/3386114656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b95a0 0x7f8bb00b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8baf59e640 1 -- 172.21.15.27:0/3386114656 >> v1:172.21.15.69:6789/0 conn(0x7f8bb015df00 legacy=0x7f8bb00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.162+0000 7f8baf59e640 1 -- 172.21.15.27:0/3386114656 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb00ce470 con 0x7f8bb00b9a10 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.163+0000 7f8baf59e640 1 --2- 172.21.15.27:0/3386114656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb00b9a10 0x7f8bb00b5c40 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009e50 tx=0x7f8ba802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18b28c40819de8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.163+0000 7f8badd9b640 1 -- 172.21.15.27:0/3386114656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba802cb00 con 0x7f8bb00b9a10 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.163+0000 7f8badd9b640 1 -- 172.21.15.27:0/3386114656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba802cca0 con 0x7f8bb00b9a10 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.163+0000 7f8badd9b640 1 -- 172.21.15.27:0/3386114656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8034610 con 0x7f8bb00b9a10 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.164+0000 7f8bb508a640 1 -- 172.21.15.27:0/3386114656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb00b9a10 msgr2=0x7f8bb00b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.164+0000 7f8bb508a640 1 --2- 172.21.15.27:0/3386114656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb00b9a10 0x7f8bb00b5c40 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009e50 tx=0x7f8ba802bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.164+0000 7f8bb508a640 1 -- 172.21.15.27:0/3386114656 shutdown_connections 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.164+0000 7f8bb508a640 1 --2- 172.21.15.27:0/3386114656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb00b9a10 0x7f8bb00b5c40 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.164+0000 7f8bb508a640 1 --2- 172.21.15.27:0/3386114656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b95a0 0x7f8bb00b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.164+0000 7f8bb508a640 1 -- 172.21.15.27:0/3386114656 >> 172.21.15.27:0/3386114656 conn(0x7f8bb0071960 msgr2=0x7f8bb0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.164+0000 7f8bb508a640 1 -- 172.21.15.27:0/3386114656 shutdown_connections 2024-03-06T07:31:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.164+0000 7f8bb508a640 1 -- 172.21.15.27:0/3386114656 wait complete. 2024-03-06T07:31:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.166+0000 7f8bb508a640 1 Processor -- start 2024-03-06T07:31:31.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.166+0000 7f8bb508a640 1 -- start start 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.167+0000 7f8bb508a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb00b95a0 0x7f8bb014f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.167+0000 7f8bb508a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b9a10 0x7f8bb01f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.167+0000 7f8bb508a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb015df00 0x7f8bb00b7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.167+0000 7f8bb508a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb00cedf0 con 0x7f8bb015df00 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.167+0000 7f8bb508a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb00cec70 con 0x7f8bb00b95a0 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.167+0000 7f8bb508a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb00d3220 con 0x7f8bb00b9a10 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb015df00 0x7f8bb00b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb015df00 0x7f8bb00b7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35954/0 (socket says 172.21.15.27:35954) 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baf59e640 1 -- 172.21.15.27:0/2099247093 learned_addr learned my addr 172.21.15.27:0/2099247093 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baed9d640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb00b95a0 0x7f8bb014f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8bae59c640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b9a10 0x7f8bb01f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baf59e640 1 -- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b9a10 msgr2=0x7f8bb01f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baf59e640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b9a10 0x7f8bb01f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baf59e640 1 -- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb00b95a0 msgr2=0x7f8bb014f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baf59e640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb00b95a0 0x7f8bb014f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baf59e640 1 -- 172.21.15.27:0/2099247093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb00c89a0 con 0x7f8bb015df00 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8baf59e640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb015df00 0x7f8bb00b7590 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009e50 tx=0x7f8ba8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8b977fe640 1 -- 172.21.15.27:0/2099247093 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8043020 con 0x7f8bb015df00 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8b977fe640 1 -- 172.21.15.27:0/2099247093 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba802ce50 con 0x7f8bb015df00 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb00c8c80 con 0x7f8bb015df00 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8b977fe640 1 -- 172.21.15.27:0/2099247093 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8009510 con 0x7f8bb015df00 2024-03-06T07:31:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.168+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb00c9240 con 0x7f8bb015df00 2024-03-06T07:31:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.169+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb0067ff0 con 0x7f8bb015df00 2024-03-06T07:31:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.170+0000 7f8b977fe640 1 -- 172.21.15.27:0/2099247093 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ba8046020 con 0x7f8bb015df00 2024-03-06T07:31:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.170+0000 7f8b977fe640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b9c06b4d0 0x7f8b9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.170+0000 7f8b977fe640 1 -- 172.21.15.27:0/2099247093 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8ba80ac720 con 0x7f8bb015df00 2024-03-06T07:31:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.170+0000 7f8baed9d640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b9c06b4d0 0x7f8b9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.171+0000 7f8baed9d640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b9c06b4d0 0x7f8b9c06d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40016e0 tx=0x7f8ba4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.179+0000 7f8b977fe640 1 -- 172.21.15.27:0/2099247093 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ba804d030 con 0x7f8bb015df00 2024-03-06T07:31:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.364+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bb00c9410 con 0x7f8bb015df00 2024-03-06T07:31:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.365+0000 7f8b977fe640 1 -- 172.21.15.27:0/2099247093 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f8ba807af10 con 0x7f8bb015df00 2024-03-06T07:31:31.368 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:31.368 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":136,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:30:37.955761+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:20.059798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.689553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.a","mappings":[{"from":2,"to":0}]},{"pgid":"19.1f","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b9c06b4d0 msgr2=0x7f8b9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b9c06b4d0 0x7f8b9c06d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40016e0 tx=0x7f8ba4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb015df00 msgr2=0x7f8bb00b7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb015df00 0x7f8bb00b7590 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009e50 tx=0x7f8ba8004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 shutdown_connections 2024-03-06T07:31:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b9c06b4d0 0x7f8b9c06d990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb015df00 0x7f8bb00b7590 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8bb00b9a10 0x7f8bb01f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 --2- 172.21.15.27:0/2099247093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bb00b95a0 0x7f8bb014f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 >> 172.21.15.27:0/2099247093 conn(0x7f8bb0071960 msgr2=0x7f8bb015d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 shutdown_connections 2024-03-06T07:31:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.368+0000 7f8bb508a640 1 -- 172.21.15.27:0/2099247093 wait complete. 2024-03-06T07:31:31.382 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:31:31.382 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.487+0000 7f7eccec0640 1 Processor -- start 2024-03-06T07:31:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.488+0000 7f7eccec0640 1 -- start start 2024-03-06T07:31:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.488+0000 7f7eccec0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ec80b2b50 0x7f7ec80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.488+0000 7f7eccec0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80ce610 con 0x7f7ec8071540 2024-03-06T07:31:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.488+0000 7f7eccec0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80d2920 con 0x7f7ec80d3400 2024-03-06T07:31:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.488+0000 7f7eccec0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80d2f70 con 0x7f7ec80b2b50 2024-03-06T07:31:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec659c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7ec8071540 0x7f7ec8071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42354/0 (socket says 172.21.15.27:42354) 2024-03-06T07:31:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec659c640 1 -- 172.21.15.27:0/1387797501 learned_addr learned my addr 172.21.15.27:0/1387797501 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec6d9d640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ec80b2b50 0x7f7ec80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439585891 0 0) 0x7f7ec80ce610 con 0x7f7ec8071540 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb0003520 con 0x7f7ec8071540 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1237311551 0 0) 0x7f7eb0003520 con 0x7f7ec8071540 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ec80b2b50 msgr2=0x7f7ec80b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ec80b2b50 0x7f7ec80b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 >> v1:172.21.15.69:6789/0 conn(0x7f7ec80d3400 legacy=0x7f7ec80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec80d2c00 con 0x7f7ec8071540 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f7eb8001670 con 0x7f7ec8071540 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 >> v1:172.21.15.27:6789/0 conn(0x7f7ec8071540 legacy=0x7f7ec8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb00039e0 0x7f7eb0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb0004390 0x7f7eb0004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ec80b2b50 0x7f7eb00050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80d2f70 con 0x7f7eb00039e0 2024-03-06T07:31:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb000b9f0 con 0x7f7eb0004390 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.489+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb000bbf0 con 0x7f7ec80b2b50 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.490+0000 7f7ec659c640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb00039e0 0x7f7eb0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.490+0000 7f7ec5d9b640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb0004390 0x7f7eb0004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.490+0000 7f7ec6d9d640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ec80b2b50 0x7f7eb00050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.490+0000 7f7ec659c640 1 -- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ec80b2b50 msgr2=0x7f7eb00050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.490+0000 7f7ec659c640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ec80b2b50 0x7f7eb00050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.490+0000 7f7ec659c640 1 -- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb0004390 msgr2=0x7f7eb0004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.490+0000 7f7ec659c640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb0004390 0x7f7eb0004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.490+0000 7f7ec659c640 1 -- 172.21.15.27:0/1387797501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb0003520 con 0x7f7eb00039e0 2024-03-06T07:31:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7ec659c640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb00039e0 0x7f7eb0003dc0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800c290 tx=0x7f7eb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac299ea110501ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8007d50 con 0x7f7eb00039e0 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb8030070 con 0x7f7eb00039e0 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7ec559a640 1 -- 172.21.15.27:0/1387797501 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb802cd80 con 0x7f7eb00039e0 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7eccec0640 1 -- 172.21.15.27:0/1387797501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb00039e0 msgr2=0x7f7eb0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7eccec0640 1 --2- 172.21.15.27:0/1387797501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb00039e0 0x7f7eb0003dc0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800c290 tx=0x7f7eb0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7ec6d9d640 1 -- 172.21.15.27:0/1387797501 reap_dead start 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7eccec0640 1 -- 172.21.15.27:0/1387797501 shutdown_connections 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7eccec0640 1 -- 172.21.15.27:0/1387797501 >> 172.21.15.27:0/1387797501 conn(0x7f7ec80d3ca0 msgr2=0x7f7ec80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7eccec0640 1 -- 172.21.15.27:0/1387797501 shutdown_connections 2024-03-06T07:31:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.491+0000 7f7eccec0640 1 -- 172.21.15.27:0/1387797501 wait complete. 2024-03-06T07:31:31.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.493+0000 7f7eccec0640 1 Processor -- start 2024-03-06T07:31:31.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.493+0000 7f7eccec0640 1 -- start start 2024-03-06T07:31:31.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eccec0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00039e0 0x7f7ec81f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eccec0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec80c8490 0x7f7ec80c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eccec0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ec80c8e50 0x7f7ec80b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eccec0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec815e370 con 0x7f7ec80c8490 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eccec0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80d4be0 con 0x7f7ec80c8e50 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eccec0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80c92d0 con 0x7f7eb00039e0 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec5d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec80c8490 0x7f7ec80c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec5d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec80c8490 0x7f7ec80c8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35970/0 (socket says 172.21.15.27:35970) 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec5d9b640 1 -- 172.21.15.27:0/801776557 learned_addr learned my addr 172.21.15.27:0/801776557 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec6d9d640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ec80c8e50 0x7f7ec80b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec659c640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00039e0 0x7f7ec81f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec5d9b640 1 -- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00039e0 msgr2=0x7f7ec81f2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec5d9b640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00039e0 0x7f7ec81f2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec5d9b640 1 -- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ec80c8e50 msgr2=0x7f7ec80b55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec5d9b640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ec80c8e50 0x7f7ec80b55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec5d9b640 1 -- 172.21.15.27:0/801776557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec81f34c0 con 0x7f7ec80c8490 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7ec5d9b640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec80c8490 0x7f7ec80c8910 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00f820 tx=0x7f7ebc00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eb6ffd640 1 -- 172.21.15.27:0/801776557 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc01a070 con 0x7f7ec80c8490 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eb6ffd640 1 -- 172.21.15.27:0/801776557 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc005110 con 0x7f7ec80c8490 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec80b3e40 con 0x7f7ec80c8490 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.494+0000 7f7eb6ffd640 1 -- 172.21.15.27:0/801776557 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc007050 con 0x7f7ec80c8490 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.495+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec80cbda0 con 0x7f7ec80c8490 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.496+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8068700 con 0x7f7ec80c8490 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.496+0000 7f7eb6ffd640 1 -- 172.21.15.27:0/801776557 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ebc01d070 con 0x7f7ec80c8490 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.496+0000 7f7eb6ffd640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ea806b500 0x7f7ea806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.496+0000 7f7eb6ffd640 1 -- 172.21.15.27:0/801776557 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7ebc089860 con 0x7f7ec80c8490 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.496+0000 7f7ec659c640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ea806b500 0x7f7ea806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.497+0000 7f7ec659c640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ea806b500 0x7f7ea806d9c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8034000 tx=0x7f7eb8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.505+0000 7f7eb6ffd640 1 -- 172.21.15.27:0/801776557 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ebc058730 con 0x7f7ec80c8490 2024-03-06T07:31:31.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.724+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ec81613b0 con 0x7f7ec80c8490 2024-03-06T07:31:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.725+0000 7f7eb6ffd640 1 -- 172.21.15.27:0/801776557 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3402 (secure 0 0 0) 0x7f7ebc058030 con 0x7f7ec80c8490 2024-03-06T07:31:31.727 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:31.727 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9060,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3131328741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3131328741},{"type":"v1","addr":"172.21.15.27:6834","nonce":3131328741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:31:25.351787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:31.728 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 103 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ea806b500 msgr2=0x7f7ea806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ea806b500 0x7f7ea806d9c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8034000 tx=0x7f7eb8009030 comp rx=0 tx=0).stop 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec80c8490 msgr2=0x7f7ec80c8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec80c8490 0x7f7ec80c8910 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00f820 tx=0x7f7ebc00fce0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 shutdown_connections 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7ea806b500 0x7f7ea806d9c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ec80c8e50 0x7f7ec80b55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec80c8490 0x7f7ec80c8910 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 --2- 172.21.15.27:0/801776557 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7eb00039e0 0x7f7ec81f2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 >> 172.21.15.27:0/801776557 conn(0x7f7ec80d3ca0 msgr2=0x7f7ec815fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 shutdown_connections 2024-03-06T07:31:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.727+0000 7f7eccec0640 1 -- 172.21.15.27:0/801776557 wait complete. 2024-03-06T07:31:31.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.846+0000 7f411955e640 1 Processor -- start 2024-03-06T07:31:31.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.846+0000 7f411955e640 1 -- start start 2024-03-06T07:31:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f411955e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f411406b5c0 0x7f41140b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f411955e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f41140ae9b0 0x7f41140d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f411955e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41140b3340 con 0x7f41140b0e90 2024-03-06T07:31:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f411955e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41140cfa00 con 0x7f411406b5c0 2024-03-06T07:31:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f411955e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41140cee90 con 0x7f41140ae9b0 2024-03-06T07:31:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f4112ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f411406b5c0 0x7f41140b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f4112ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f411406b5c0 0x7f41140b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35974/0 (socket says 172.21.15.27:35974) 2024-03-06T07:31:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f4112ffd640 1 -- 172.21.15.27:0/1887850618 learned_addr learned my addr 172.21.15.27:0/1887850618 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f4112ffd640 1 -- 172.21.15.27:0/1887850618 >> v2:172.21.15.69:3300/0 conn(0x7f41140ae9b0 msgr2=0x7f41140d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.847+0000 7f41137fe640 1 --2- 172.21.15.27:0/1887850618 >> v2:172.21.15.69:3300/0 conn(0x7f41140ae9b0 0x7f41140d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.848+0000 7f4112ffd640 1 --2- 172.21.15.27:0/1887850618 >> v2:172.21.15.69:3300/0 conn(0x7f41140ae9b0 0x7f41140d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.848+0000 7f4112ffd640 1 -- 172.21.15.27:0/1887850618 >> v1:172.21.15.69:6789/0 conn(0x7f41140b0e90 legacy=0x7f411406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.848+0000 7f4112ffd640 1 -- 172.21.15.27:0/1887850618 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41140cfc50 con 0x7f411406b5c0 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.848+0000 7f4112ffd640 1 --2- 172.21.15.27:0/1887850618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f411406b5c0 0x7f41140b2e00 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f4114071b00 tx=0x7f410402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11dd7e2981991c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.848+0000 7f4111ffb640 1 -- 172.21.15.27:0/1887850618 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410402e070 con 0x7f411406b5c0 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.848+0000 7f4111ffb640 1 -- 172.21.15.27:0/1887850618 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104037070 con 0x7f411406b5c0 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.849+0000 7f4111ffb640 1 -- 172.21.15.27:0/1887850618 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104031a00 con 0x7f411406b5c0 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.849+0000 7f411955e640 1 -- 172.21.15.27:0/1887850618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f411406b5c0 msgr2=0x7f41140b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.849+0000 7f411955e640 1 --2- 172.21.15.27:0/1887850618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f411406b5c0 0x7f41140b2e00 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f4114071b00 tx=0x7f410402b880 comp rx=0 tx=0).stop 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.849+0000 7f411955e640 1 -- 172.21.15.27:0/1887850618 shutdown_connections 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.849+0000 7f411955e640 1 --2- 172.21.15.27:0/1887850618 >> v2:172.21.15.69:3300/0 conn(0x7f41140ae9b0 0x7f41140d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.849+0000 7f411955e640 1 --2- 172.21.15.27:0/1887850618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f411406b5c0 0x7f41140b2e00 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.849+0000 7f411955e640 1 -- 172.21.15.27:0/1887850618 >> 172.21.15.27:0/1887850618 conn(0x7f41140b36d0 msgr2=0x7f41140b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.849+0000 7f411955e640 1 -- 172.21.15.27:0/1887850618 shutdown_connections 2024-03-06T07:31:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.849+0000 7f411955e640 1 -- 172.21.15.27:0/1887850618 wait complete. 2024-03-06T07:31:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.850+0000 7f411955e640 1 Processor -- start 2024-03-06T07:31:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.850+0000 7f411955e640 1 -- start start 2024-03-06T07:31:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f411955e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f411406b5c0 0x7f41141de500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f411955e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41140ae9b0 0x7f41140caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f411955e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41140b0e90 0x7f41140cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f411955e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411406c910 con 0x7f41140b0e90 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f411955e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411406c790 con 0x7f411406b5c0 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f411955e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140aedc0 con 0x7f41140ae9b0 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f41137fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41140b0e90 0x7f41140cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f41137fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41140b0e90 0x7f41140cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35982/0 (socket says 172.21.15.27:35982) 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f41137fe640 1 -- 172.21.15.27:0/3725626856 learned_addr learned my addr 172.21.15.27:0/3725626856 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f4112ffd640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f411406b5c0 0x7f41141de500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.851+0000 7f41127fc640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41140ae9b0 0x7f41140caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f41137fe640 1 -- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41140ae9b0 msgr2=0x7f41140caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f41137fe640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41140ae9b0 0x7f41140caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f41137fe640 1 -- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f411406b5c0 msgr2=0x7f41141de500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f41137fe640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f411406b5c0 0x7f41141de500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f41137fe640 1 -- 172.21.15.27:0/3725626856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41141f5370 con 0x7f41140b0e90 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f41127fc640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41140ae9b0 0x7f41140caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f41137fe640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41140b0e90 0x7f41140cb3e0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f410c00cd70 tx=0x7f410c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f40fb7fe640 1 -- 172.21.15.27:0/3725626856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c003b90 con 0x7f41140b0e90 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f40fb7fe640 1 -- 172.21.15.27:0/3725626856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c003d30 con 0x7f41140b0e90 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f40fb7fe640 1 -- 172.21.15.27:0/3725626856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c004240 con 0x7f41140b0e90 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41140cbfd0 con 0x7f41140b0e90 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.852+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41140cc590 con 0x7f41140b0e90 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.853+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41140cc790 con 0x7f41140b0e90 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.854+0000 7f40fb7fe640 1 -- 172.21.15.27:0/3725626856 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f410c0043e0 con 0x7f41140b0e90 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.854+0000 7f40fb7fe640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f410006b4d0 0x7f410006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.854+0000 7f40fb7fe640 1 -- 172.21.15.27:0/3725626856 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f410c08ab60 con 0x7f41140b0e90 2024-03-06T07:31:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.854+0000 7f4112ffd640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f410006b4d0 0x7f410006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.855+0000 7f4112ffd640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f410006b4d0 0x7f410006d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4104036000 tx=0x7f410403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:31.858+0000 7f40fb7fe640 1 -- 172.21.15.27:0/3725626856 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41140cc790 con 0x7f41140b0e90 2024-03-06T07:31:32.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.213+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4114068700 con 0x7f41140b0e90 2024-03-06T07:31:32.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.213+0000 7f40fb7fe640 1 -- 172.21.15.27:0/3725626856 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3402 (secure 0 0 0) 0x7f410c059a30 con 0x7f41140b0e90 2024-03-06T07:31:32.216 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:32.216 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9060,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3131328741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3131328741},{"type":"v1","addr":"172.21.15.27:6834","nonce":3131328741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.718897+0000","modified":"2024-03-06T07:31:25.351787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8679":{"gid":8679,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6835/4036525552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4036525552},{"type":"v1","addr":"172.21.15.27:6835","nonce":4036525552}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":9}]} 2024-03-06T07:31:32.216 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 103 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f410006b4d0 msgr2=0x7f410006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f410006b4d0 0x7f410006d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4104036000 tx=0x7f410403d040 comp rx=0 tx=0).stop 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41140b0e90 msgr2=0x7f41140cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41140b0e90 0x7f41140cb3e0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f410c00cd70 tx=0x7f410c00a660 comp rx=0 tx=0).stop 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 shutdown_connections 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f410006b4d0 0x7f410006d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41140b0e90 0x7f41140cb3e0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41140ae9b0 0x7f41140caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 --2- 172.21.15.27:0/3725626856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f411406b5c0 0x7f41141de500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 >> 172.21.15.27:0/3725626856 conn(0x7f41140b36d0 msgr2=0x7f411406ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 shutdown_connections 2024-03-06T07:31:32.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.215+0000 7f411955e640 1 -- 172.21.15.27:0/3725626856 wait complete. 2024-03-06T07:31:32.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.333+0000 7f0b7e447640 1 Processor -- start 2024-03-06T07:31:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.333+0000 7f0b7e447640 1 -- start start 2024-03-06T07:31:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.334+0000 7f0b7e447640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b78158520 0x7f0b7806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.334+0000 7f0b7e447640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b780aee10 0x7f0b780b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.334+0000 7f0b7e447640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b78071240 con 0x7f0b7806dd60 2024-03-06T07:31:32.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.334+0000 7f0b7e447640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b78071b70 con 0x7f0b78158520 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.334+0000 7f0b7e447640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b78071520 con 0x7f0b780aee10 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.334+0000 7f0b77fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b78158520 0x7f0b7806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.334+0000 7f0b777fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0b7806dd60 0x7f0b780ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42358/0 (socket says 172.21.15.27:42358) 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.334+0000 7f0b777fe640 1 -- 172.21.15.27:0/2934878461 learned_addr learned my addr 172.21.15.27:0/2934878461 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.335+0000 7f0b7c9e4640 1 --2- 172.21.15.27:0/2934878461 >> v2:172.21.15.69:3300/0 conn(0x7f0b780aee10 0x7f0b780b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.335+0000 7f0b77fff640 1 -- 172.21.15.27:0/2934878461 >> v2:172.21.15.69:3300/0 conn(0x7f0b780aee10 msgr2=0x7f0b780b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.335+0000 7f0b77fff640 1 --2- 172.21.15.27:0/2934878461 >> v2:172.21.15.69:3300/0 conn(0x7f0b780aee10 0x7f0b780b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.335+0000 7f0b77fff640 1 -- 172.21.15.27:0/2934878461 >> v1:172.21.15.27:6789/0 conn(0x7f0b7806dd60 legacy=0x7f0b780ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.335+0000 7f0b77fff640 1 -- 172.21.15.27:0/2934878461 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78071800 con 0x7f0b78158520 2024-03-06T07:31:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.335+0000 7f0b7c9e4640 1 --2- 172.21.15.27:0/2934878461 >> v2:172.21.15.69:3300/0 conn(0x7f0b780aee10 0x7f0b780b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.335+0000 7f0b77fff640 1 --2- 172.21.15.27:0/2934878461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b78158520 0x7f0b7806d820 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f0b780b2d50 tx=0x7f0b6002e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e3d0a3c99ff980 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b76ffd640 1 -- 172.21.15.27:0/2934878461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6003b070 con 0x7f0b78158520 2024-03-06T07:31:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b76ffd640 1 -- 172.21.15.27:0/2934878461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6002ee50 con 0x7f0b78158520 2024-03-06T07:31:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b76ffd640 1 -- 172.21.15.27:0/2934878461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60036660 con 0x7f0b78158520 2024-03-06T07:31:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b7e447640 1 -- 172.21.15.27:0/2934878461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b78158520 msgr2=0x7f0b7806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b7e447640 1 --2- 172.21.15.27:0/2934878461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b78158520 0x7f0b7806d820 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f0b780b2d50 tx=0x7f0b6002e0a0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b7e447640 1 -- 172.21.15.27:0/2934878461 shutdown_connections 2024-03-06T07:31:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b7e447640 1 --2- 172.21.15.27:0/2934878461 >> v2:172.21.15.69:3300/0 conn(0x7f0b780aee10 0x7f0b780b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b7e447640 1 --2- 172.21.15.27:0/2934878461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b78158520 0x7f0b7806d820 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b7e447640 1 -- 172.21.15.27:0/2934878461 >> 172.21.15.27:0/2934878461 conn(0x7f0b78066d00 msgr2=0x7f0b780670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b7e447640 1 -- 172.21.15.27:0/2934878461 shutdown_connections 2024-03-06T07:31:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.336+0000 7f0b7e447640 1 -- 172.21.15.27:0/2934878461 wait complete. 2024-03-06T07:31:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.337+0000 7f0b7e447640 1 Processor -- start 2024-03-06T07:31:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.338+0000 7f0b7e447640 1 -- start start 2024-03-06T07:31:32.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.338+0000 7f0b7e447640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7806dd60 0x7f0b780cd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.338+0000 7f0b7e447640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780aee10 0x7f0b781450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.338+0000 7f0b7e447640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b78158520 0x7f0b78159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.338+0000 7f0b7e447640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78158a80 con 0x7f0b7806dd60 2024-03-06T07:31:32.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.338+0000 7f0b7e447640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78158900 con 0x7f0b78158520 2024-03-06T07:31:32.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.338+0000 7f0b7e447640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780b3b80 con 0x7f0b780aee10 2024-03-06T07:31:32.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.338+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7806dd60 0x7f0b780cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7806dd60 0x7f0b780cd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36008/0 (socket says 172.21.15.27:36008) 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b77fff640 1 -- 172.21.15.27:0/3422093218 learned_addr learned my addr 172.21.15.27:0/3422093218 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b7c9e4640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b78158520 0x7f0b78159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b77fff640 1 -- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780aee10 msgr2=0x7f0b781450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b77fff640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780aee10 0x7f0b781450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b77fff640 1 -- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b78158520 msgr2=0x7f0b78159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b77fff640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b78158520 0x7f0b78159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b77fff640 1 -- 172.21.15.27:0/3422093218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b780ccbe0 con 0x7f0b7806dd60 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b7c9e4640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b78158520 0x7f0b78159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b77fff640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7806dd60 0x7f0b780cd960 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000bb30 tx=0x7f0b600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b74ff9640 1 -- 172.21.15.27:0/3422093218 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6003b040 con 0x7f0b7806dd60 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b74ff9640 1 -- 172.21.15.27:0/3422093218 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b60047070 con 0x7f0b7806dd60 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b74ff9640 1 -- 172.21.15.27:0/3422093218 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60042820 con 0x7f0b7806dd60 2024-03-06T07:31:32.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b780cce80 con 0x7f0b7806dd60 2024-03-06T07:31:32.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.339+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b780cf3c0 con 0x7f0b7806dd60 2024-03-06T07:31:32.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.340+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b78067ff0 con 0x7f0b7806dd60 2024-03-06T07:31:32.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.341+0000 7f0b74ff9640 1 -- 172.21.15.27:0/3422093218 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b600429c0 con 0x7f0b7806dd60 2024-03-06T07:31:32.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.341+0000 7f0b74ff9640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b5806b4d0 0x7f0b5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.341+0000 7f0b74ff9640 1 -- 172.21.15.27:0/3422093218 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0b600aeab0 con 0x7f0b7806dd60 2024-03-06T07:31:32.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.341+0000 7f0b777fe640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b5806b4d0 0x7f0b5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.342+0000 7f0b777fe640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b5806b4d0 0x7f0b5806d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0b68005880 tx=0x7f0b68007aa0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.346+0000 7f0b74ff9640 1 -- 172.21.15.27:0/3422093218 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b600353a0 con 0x7f0b7806dd60 2024-03-06T07:31:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.529+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b780b47d0 con 0x7f0b7806dd60 2024-03-06T07:31:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.530+0000 7f0b74ff9640 1 -- 172.21.15.27:0/3422093218 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f0b6007da10 con 0x7f0b7806dd60 2024-03-06T07:31:32.533 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:32.533 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":136,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:30:37.955761+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:20.059798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.689553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.a","mappings":[{"from":2,"to":0}]},{"pgid":"19.1f","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b5806b4d0 msgr2=0x7f0b5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b5806b4d0 0x7f0b5806d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0b68005880 tx=0x7f0b68007aa0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7806dd60 msgr2=0x7f0b780cd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7806dd60 0x7f0b780cd960 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000bb30 tx=0x7f0b600046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 shutdown_connections 2024-03-06T07:31:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0b5806b4d0 0x7f0b5806d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b78158520 0x7f0b78159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0b780aee10 0x7f0b781450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 --2- 172.21.15.27:0/3422093218 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b7806dd60 0x7f0b780cd960 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 >> 172.21.15.27:0/3422093218 conn(0x7f0b78066d00 msgr2=0x7f0b7806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 shutdown_connections 2024-03-06T07:31:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.532+0000 7f0b7e447640 1 -- 172.21.15.27:0/3422093218 wait complete. 2024-03-06T07:31:32.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:31:32.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.654+0000 7fb63a372640 1 Processor -- start 2024-03-06T07:31:32.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.654+0000 7fb63a372640 1 -- start start 2024-03-06T07:31:32.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.655+0000 7fb63a372640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb63406b5c0 0x7fb6340b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.655+0000 7fb63a372640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6340b0e90 0x7fb63406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.655+0000 7fb63a372640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 0x7fb6340d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.655+0000 7fb63a372640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6340cfa00 con 0x7fb6340b0e90 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.655+0000 7fb63a372640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6340b3340 con 0x7fb63406b5c0 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.655+0000 7fb63a372640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340cee90 con 0x7fb6340ae9b0 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.655+0000 7fb6337fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6340b0e90 0x7fb63406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.655+0000 7fb6337fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6340b0e90 0x7fb63406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36024/0 (socket says 172.21.15.27:36024) 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.655+0000 7fb6337fe640 1 -- 172.21.15.27:0/3112996765 learned_addr learned my addr 172.21.15.27:0/3112996765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.656+0000 7fb633fff640 1 --2- 172.21.15.27:0/3112996765 >> v2:172.21.15.69:3300/0 conn(0x7fb63406b5c0 0x7fb6340b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.656+0000 7fb63890f640 1 --2- 172.21.15.27:0/3112996765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 0x7fb6340d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.656+0000 7fb6337fe640 1 -- 172.21.15.27:0/3112996765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 msgr2=0x7fb6340d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.656+0000 7fb6337fe640 1 --2- 172.21.15.27:0/3112996765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 0x7fb6340d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.656+0000 7fb6337fe640 1 -- 172.21.15.27:0/3112996765 >> v2:172.21.15.69:3300/0 conn(0x7fb63406b5c0 msgr2=0x7fb6340b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.656+0000 7fb6337fe640 1 --2- 172.21.15.27:0/3112996765 >> v2:172.21.15.69:3300/0 conn(0x7fb63406b5c0 0x7fb6340b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.656+0000 7fb6337fe640 1 -- 172.21.15.27:0/3112996765 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6340cfc50 con 0x7fb6340b0e90 2024-03-06T07:31:32.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.656+0000 7fb63890f640 1 --2- 172.21.15.27:0/3112996765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 0x7fb6340d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:32.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.656+0000 7fb6337fe640 1 --2- 172.21.15.27:0/3112996765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340b0e90 0x7fb63406b9a0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fb634071b00 tx=0x7fb61c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8346ec4abd6cbf3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb632ffd640 1 -- 172.21.15.27:0/3112996765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c02e070 con 0x7fb6340b0e90 2024-03-06T07:31:32.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb632ffd640 1 -- 172.21.15.27:0/3112996765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c037070 con 0x7fb6340b0e90 2024-03-06T07:31:32.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb632ffd640 1 -- 172.21.15.27:0/3112996765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c031a00 con 0x7fb6340b0e90 2024-03-06T07:31:32.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb63a372640 1 -- 172.21.15.27:0/3112996765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340b0e90 msgr2=0x7fb63406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb63a372640 1 --2- 172.21.15.27:0/3112996765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340b0e90 0x7fb63406b9a0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fb634071b00 tx=0x7fb61c02b880 comp rx=0 tx=0).stop 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb63a372640 1 -- 172.21.15.27:0/3112996765 shutdown_connections 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb63a372640 1 --2- 172.21.15.27:0/3112996765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 0x7fb6340d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb63a372640 1 --2- 172.21.15.27:0/3112996765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340b0e90 0x7fb63406b9a0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb63a372640 1 --2- 172.21.15.27:0/3112996765 >> v2:172.21.15.69:3300/0 conn(0x7fb63406b5c0 0x7fb6340b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb63a372640 1 -- 172.21.15.27:0/3112996765 >> 172.21.15.27:0/3112996765 conn(0x7fb6340b36d0 msgr2=0x7fb6340b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb63a372640 1 -- 172.21.15.27:0/3112996765 shutdown_connections 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.657+0000 7fb63a372640 1 -- 172.21.15.27:0/3112996765 wait complete. 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.659+0000 7fb63a372640 1 Processor -- start 2024-03-06T07:31:32.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.659+0000 7fb63a372640 1 -- start start 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.659+0000 7fb63a372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb63406b5c0 0x7fb6340c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.659+0000 7fb63a372640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 0x7fb6340b6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.659+0000 7fb63a372640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6340b0e90 0x7fb634157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.659+0000 7fb63a372640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63406c790 con 0x7fb63406b5c0 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.659+0000 7fb63a372640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63406bee0 con 0x7fb6340b0e90 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.659+0000 7fb63a372640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63406c910 con 0x7fb6340ae9b0 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb633fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb63406b5c0 0x7fb6340c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb633fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb63406b5c0 0x7fb6340c5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36030/0 (socket says 172.21.15.27:36030) 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb633fff640 1 -- 172.21.15.27:0/2311313468 learned_addr learned my addr 172.21.15.27:0/2311313468 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb63890f640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6340b0e90 0x7fb634157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb6337fe640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 0x7fb6340b6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb633fff640 1 -- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 msgr2=0x7fb6340b6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb633fff640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 0x7fb6340b6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb633fff640 1 -- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6340b0e90 msgr2=0x7fb634157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb633fff640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6340b0e90 0x7fb634157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb633fff640 1 -- 172.21.15.27:0/2311313468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6340c5c10 con 0x7fb63406b5c0 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb63890f640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6340b0e90 0x7fb634157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb633fff640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb63406b5c0 0x7fb6340c5630 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fb62400e870 tx=0x7fb62400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb630ff9640 1 -- 172.21.15.27:0/2311313468 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb624017070 con 0x7fb63406b5c0 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb630ff9640 1 -- 172.21.15.27:0/2311313468 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62400ee60 con 0x7fb63406b5c0 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb630ff9640 1 -- 172.21.15.27:0/2311313468 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62400cb80 con 0x7fb63406b5c0 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6340c5ef0 con 0x7fb63406b5c0 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.660+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6340c64b0 con 0x7fb63406b5c0 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.662+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb634068700 con 0x7fb63406b5c0 2024-03-06T07:31:32.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.662+0000 7fb630ff9640 1 -- 172.21.15.27:0/2311313468 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb62401a070 con 0x7fb63406b5c0 2024-03-06T07:31:32.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.662+0000 7fb630ff9640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb61406b500 0x7fb61406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.662+0000 7fb630ff9640 1 -- 172.21.15.27:0/2311313468 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb62408a400 con 0x7fb63406b5c0 2024-03-06T07:31:32.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.662+0000 7fb6337fe640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb61406b500 0x7fb61406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.663+0000 7fb6337fe640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb61406b500 0x7fb61406d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb61c036000 tx=0x7fb61c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.667+0000 7fb630ff9640 1 -- 172.21.15.27:0/2311313468 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb624059240 con 0x7fb63406b5c0 2024-03-06T07:31:32.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.966+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb6340b6bc0 con 0x7fb63406b5c0 2024-03-06T07:31:32.969 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:31:32.966+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:31:32.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.968+0000 7fb630ff9640 1 -- 172.21.15.27:0/2311313468 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v104) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb624058b40 con 0x7fb63406b5c0 2024-03-06T07:31:32.970 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:31:32.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.971+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb61406b500 msgr2=0x7fb61406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.971+0000 7fb63a372640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb61406b500 0x7fb61406d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb61c036000 tx=0x7fb61c03d040 comp rx=0 tx=0).stop 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.971+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb63406b5c0 msgr2=0x7fb6340c5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.971+0000 7fb63a372640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb63406b5c0 0x7fb6340c5630 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fb62400e870 tx=0x7fb62400ed30 comp rx=0 tx=0).stop 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.972+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 shutdown_connections 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.972+0000 7fb63a372640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb61406b500 0x7fb61406d9c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.972+0000 7fb63a372640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6340b0e90 0x7fb634157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.972+0000 7fb63a372640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6340ae9b0 0x7fb6340b6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.972+0000 7fb63a372640 1 --2- 172.21.15.27:0/2311313468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb63406b5c0 0x7fb6340c5630 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.972+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 >> 172.21.15.27:0/2311313468 conn(0x7fb6340b36d0 msgr2=0x7fb6340cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.972+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 shutdown_connections 2024-03-06T07:31:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:32.972+0000 7fb63a372640 1 -- 172.21.15.27:0/2311313468 wait complete. 2024-03-06T07:31:32.978 INFO:tasks.ceph.mds.a.smithi027.stderr: -8611> 2024-03-06T07:30:26.312+0000 7fd868f37640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:31:32.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:31:33.011 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:31:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.094+0000 7f02b9319640 1 Processor -- start 2024-03-06T07:31:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.095+0000 7f02b9319640 1 -- start start 2024-03-06T07:31:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.095+0000 7f02b9319640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02b415f500 0x7f02b40ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.095+0000 7f02b9319640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 0x7f02b40ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.095+0000 7f02b9319640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b40cd310 con 0x7f02b40ced80 2024-03-06T07:31:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.095+0000 7f02b9319640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b415f8e0 con 0x7f02b415f500 2024-03-06T07:31:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.095+0000 7f02b9319640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40c56c0 con 0x7f02b40cf740 2024-03-06T07:31:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.095+0000 7f02b2ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02b415f500 0x7f02b40ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.095+0000 7f02b2ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02b415f500 0x7f02b40ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36038/0 (socket says 172.21.15.27:36038) 2024-03-06T07:31:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.095+0000 7f02b2ffd640 1 -- 172.21.15.27:0/3712091075 learned_addr learned my addr 172.21.15.27:0/3712091075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.096+0000 7f02b27fc640 1 --2- 172.21.15.27:0/3712091075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 0x7f02b40ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.096+0000 7f02b2ffd640 1 -- 172.21.15.27:0/3712091075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 msgr2=0x7f02b40ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.096+0000 7f02b2ffd640 1 --2- 172.21.15.27:0/3712091075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 0x7f02b40ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.096+0000 7f02b2ffd640 1 -- 172.21.15.27:0/3712091075 >> v1:172.21.15.69:6789/0 conn(0x7f02b40ced80 legacy=0x7f02b40cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.096+0000 7f02b2ffd640 1 -- 172.21.15.27:0/3712091075 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b40cd5b0 con 0x7f02b415f500 2024-03-06T07:31:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b2ffd640 1 --2- 172.21.15.27:0/3712091075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b415f500 0x7f02b40ce300 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009e50 tx=0x7f02a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e332a3ef968aacea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b1ffb640 1 -- 172.21.15.27:0/3712091075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a402ca70 con 0x7f02b415f500 2024-03-06T07:31:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b1ffb640 1 -- 172.21.15.27:0/3712091075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a402cc10 con 0x7f02b415f500 2024-03-06T07:31:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b1ffb640 1 -- 172.21.15.27:0/3712091075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a40343f0 con 0x7f02b415f500 2024-03-06T07:31:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b9319640 1 -- 172.21.15.27:0/3712091075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b415f500 msgr2=0x7f02b40ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b9319640 1 --2- 172.21.15.27:0/3712091075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b415f500 0x7f02b40ce300 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009e50 tx=0x7f02a402bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b9319640 1 -- 172.21.15.27:0/3712091075 shutdown_connections 2024-03-06T07:31:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b9319640 1 --2- 172.21.15.27:0/3712091075 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 0x7f02b40ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b9319640 1 --2- 172.21.15.27:0/3712091075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b415f500 0x7f02b40ce300 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b9319640 1 -- 172.21.15.27:0/3712091075 >> 172.21.15.27:0/3712091075 conn(0x7f02b40b3700 msgr2=0x7f02b40b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b9319640 1 -- 172.21.15.27:0/3712091075 shutdown_connections 2024-03-06T07:31:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.097+0000 7f02b9319640 1 -- 172.21.15.27:0/3712091075 wait complete. 2024-03-06T07:31:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b9319640 1 Processor -- start 2024-03-06T07:31:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b9319640 1 -- start start 2024-03-06T07:31:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b9319640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b40ced80 0x7f02b40b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b9319640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 0x7f02b414cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b9319640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b415f500 0x7f02b406fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b9319640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40b43d0 con 0x7f02b40ced80 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b9319640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40c59a0 con 0x7f02b415f500 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b9319640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40c6e40 con 0x7f02b40cf740 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b40ced80 0x7f02b40b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b40ced80 0x7f02b40b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36042/0 (socket says 172.21.15.27:36042) 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.099+0000 7f02b2ffd640 1 -- 172.21.15.27:0/1498599296 learned_addr learned my addr 172.21.15.27:0/1498599296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b27fc640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 0x7f02b414cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b37fe640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b415f500 0x7f02b406fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b2ffd640 1 -- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 msgr2=0x7f02b414cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b2ffd640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 0x7f02b414cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b2ffd640 1 -- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b415f500 msgr2=0x7f02b406fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b2ffd640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b415f500 0x7f02b406fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b2ffd640 1 -- 172.21.15.27:0/1498599296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b41f7390 con 0x7f02b40ced80 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b37fe640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b415f500 0x7f02b406fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b2ffd640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b40ced80 0x7f02b40b51d0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f02a4000f80 tx=0x7f02a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f029b7fe640 1 -- 172.21.15.27:0/1498599296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4043020 con 0x7f02b40ced80 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f029b7fe640 1 -- 172.21.15.27:0/1498599296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a403e450 con 0x7f02b40ced80 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f029b7fe640 1 -- 172.21.15.27:0/1498599296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4009510 con 0x7f02b40ced80 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b41f7670 con 0x7f02b40ced80 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.100+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b41f7c30 con 0x7f02b40ced80 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.101+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b409fcf0 con 0x7f02b40ced80 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.102+0000 7f029b7fe640 1 -- 172.21.15.27:0/1498599296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02a4046020 con 0x7f02b40ced80 2024-03-06T07:31:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.102+0000 7f029b7fe640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a006b4d0 0x7f02a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.102+0000 7f029b7fe640 1 -- 172.21.15.27:0/1498599296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f02a40aca80 con 0x7f02b40ced80 2024-03-06T07:31:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.102+0000 7f02b27fc640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a006b4d0 0x7f02a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.103+0000 7f02b27fc640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a006b4d0 0x7f02a006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f02b4158590 tx=0x7f02a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.106+0000 7f029b7fe640 1 -- 172.21.15.27:0/1498599296 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02a404d030 con 0x7f02b40ced80 2024-03-06T07:31:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.326+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f02b404f410 con 0x7f02b40ced80 2024-03-06T07:31:33.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.969+0000 7f029b7fe640 1 -- 172.21.15.27:0/1498599296 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v105) v1 ==== 104+0+0 (secure 0 0 0) 0x7f02a407b880 con 0x7f02b40ced80 2024-03-06T07:31:33.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a006b4d0 msgr2=0x7f02a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a006b4d0 0x7f02a006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f02b4158590 tx=0x7f02a8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b40ced80 msgr2=0x7f02b40b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b40ced80 0x7f02b40b51d0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f02a4000f80 tx=0x7f02a4004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 shutdown_connections 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f02a006b4d0 0x7f02a006d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b415f500 0x7f02b406fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02b40cf740 0x7f02b414cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 --2- 172.21.15.27:0/1498599296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b40ced80 0x7f02b40b51d0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 >> 172.21.15.27:0/1498599296 conn(0x7f02b40b3700 msgr2=0x7f02b40cc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 shutdown_connections 2024-03-06T07:31:33.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:33.973+0000 7f02b9319640 1 -- 172.21.15.27:0/1498599296 wait complete. 2024-03-06T07:31:33.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:31:34.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.147+0000 7f6f7f0eb640 1 Processor -- start 2024-03-06T07:31:34.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.147+0000 7f6f7f0eb640 1 -- start start 2024-03-06T07:31:34.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.148+0000 7f6f7f0eb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f780b2d30 0x7f6f780b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.148+0000 7f6f7f0eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6f780b7820 0x7f6f780b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.148+0000 7f6f7f0eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f780d4960 con 0x7f6f78071540 2024-03-06T07:31:34.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.148+0000 7f6f7f0eb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f780d3230 con 0x7f6f780b7820 2024-03-06T07:31:34.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.148+0000 7f6f7f0eb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f780d36c0 con 0x7f6f780b2d30 2024-03-06T07:31:34.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.148+0000 7f6f7d688640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6f780b7820 0x7f6f780b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.148+0000 7f6f7d688640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6f780b7820 0x7f6f780b7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39420/0 (socket says 172.21.15.27:39420) 2024-03-06T07:31:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.148+0000 7f6f7d688640 1 -- 172.21.15.27:0/620168197 learned_addr learned my addr 172.21.15.27:0/620168197 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.149+0000 7f6f77fff640 1 --2- 172.21.15.27:0/620168197 >> v2:172.21.15.69:3300/0 conn(0x7f6f780b2d30 0x7f6f780b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.149+0000 7f6f7d688640 1 -- 172.21.15.27:0/620168197 >> v2:172.21.15.69:3300/0 conn(0x7f6f780b2d30 msgr2=0x7f6f780b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.149+0000 7f6f7d688640 1 --2- 172.21.15.27:0/620168197 >> v2:172.21.15.69:3300/0 conn(0x7f6f780b2d30 0x7f6f780b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.149+0000 7f6f7d688640 1 -- 172.21.15.27:0/620168197 >> v1:172.21.15.69:6789/0 conn(0x7f6f78071540 legacy=0x7f6f78071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.149+0000 7f6f7d688640 1 -- 172.21.15.27:0/620168197 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f780d2620 con 0x7f6f780b7820 2024-03-06T07:31:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f7d688640 1 --2- 172.21.15.27:0/620168197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f780b7820 0x7f6f780b7cd0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f6f780c7690 tx=0x7f6f6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970bbabc29f4d678 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f777fe640 1 -- 172.21.15.27:0/620168197 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f68035020 con 0x7f6f780b7820 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f777fe640 1 -- 172.21.15.27:0/620168197 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f68031870 con 0x7f6f780b7820 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f777fe640 1 -- 172.21.15.27:0/620168197 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f68031c40 con 0x7f6f780b7820 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/620168197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f780b7820 msgr2=0x7f6f780b7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f7f0eb640 1 --2- 172.21.15.27:0/620168197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f780b7820 0x7f6f780b7cd0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f6f780c7690 tx=0x7f6f6802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/620168197 shutdown_connections 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f7f0eb640 1 --2- 172.21.15.27:0/620168197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f780b7820 0x7f6f780b7cd0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f7f0eb640 1 --2- 172.21.15.27:0/620168197 >> v2:172.21.15.69:3300/0 conn(0x7f6f780b2d30 0x7f6f780b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/620168197 >> 172.21.15.27:0/620168197 conn(0x7f6f780d3d60 msgr2=0x7f6f780d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/620168197 shutdown_connections 2024-03-06T07:31:34.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.150+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/620168197 wait complete. 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.152+0000 7f6f7f0eb640 1 Processor -- start 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.152+0000 7f6f7f0eb640 1 -- start start 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.152+0000 7f6f7f0eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f78071540 0x7f6f781f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.152+0000 7f6f7f0eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f780b2d30 0x7f6f780aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.152+0000 7f6f7f0eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f780b7820 0x7f6f780c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.152+0000 7f6f7f0eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780d2ec0 con 0x7f6f78071540 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.152+0000 7f6f7f0eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780d2be0 con 0x7f6f780b7820 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.152+0000 7f6f7f0eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780d22f0 con 0x7f6f780b2d30 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7ce87640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f78071540 0x7f6f781f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7ce87640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f78071540 0x7f6f781f3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39432/0 (socket says 172.21.15.27:39432) 2024-03-06T07:31:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7ce87640 1 -- 172.21.15.27:0/1103395782 learned_addr learned my addr 172.21.15.27:0/1103395782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7ce87640 1 -- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f780b2d30 msgr2=0x7f6f780aa6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7d688640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f780b7820 0x7f6f780c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f77fff640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f780b2d30 0x7f6f780aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7ce87640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f780b2d30 0x7f6f780aa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7ce87640 1 -- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f780b7820 msgr2=0x7f6f780c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7ce87640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f780b7820 0x7f6f780c95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7ce87640 1 -- 172.21.15.27:0/1103395782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f780ab120 con 0x7f6f78071540 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7ce87640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f78071540 0x7f6f781f3c10 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000c740 tx=0x7f6f7000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f757fa640 1 -- 172.21.15.27:0/1103395782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70016020 con 0x7f6f78071540 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f757fa640 1 -- 172.21.15.27:0/1103395782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7000ce50 con 0x7f6f78071540 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f780ab400 con 0x7f6f78071540 2024-03-06T07:31:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f757fa640 1 -- 172.21.15.27:0/1103395782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7000ac90 con 0x7f6f78071540 2024-03-06T07:31:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.153+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f780b0c60 con 0x7f6f78071540 2024-03-06T07:31:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.154+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f780b58e0 con 0x7f6f78071540 2024-03-06T07:31:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.155+0000 7f6f757fa640 1 -- 172.21.15.27:0/1103395782 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f70019020 con 0x7f6f78071540 2024-03-06T07:31:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.155+0000 7f6f757fa640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f6406b4d0 0x7f6f6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.155+0000 7f6f757fa640 1 -- 172.21.15.27:0/1103395782 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f6f7008a5c0 con 0x7f6f78071540 2024-03-06T07:31:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.156+0000 7f6f77fff640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f6406b4d0 0x7f6f6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.156+0000 7f6f77fff640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f6406b4d0 0x7f6f6406d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6f7815cff0 tx=0x7f6f6c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.160+0000 7f6f757fa640 1 -- 172.21.15.27:0/1103395782 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f780b58e0 con 0x7f6f78071540 2024-03-06T07:31:34.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.342+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6f78072070 con 0x7f6f78071540 2024-03-06T07:31:34.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.974+0000 7f6f757fa640 1 -- 172.21.15.27:0/1103395782 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v138) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6f70020020 con 0x7f6f78071540 2024-03-06T07:31:34.976 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:31:34.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.977+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f6406b4d0 msgr2=0x7f6f6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.977+0000 7f6f7f0eb640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f6406b4d0 0x7f6f6406d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6f7815cff0 tx=0x7f6f6c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:34.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.977+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f78071540 msgr2=0x7f6f781f3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.977+0000 7f6f7f0eb640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f78071540 0x7f6f781f3c10 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000c740 tx=0x7f6f7000cc00 comp rx=0 tx=0).stop 2024-03-06T07:31:34.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.978+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 shutdown_connections 2024-03-06T07:31:34.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.978+0000 7f6f7f0eb640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f6406b4d0 0x7f6f6406d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.978+0000 7f6f7f0eb640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f780b7820 0x7f6f780c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.978+0000 7f6f7f0eb640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f780b2d30 0x7f6f780aa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.978+0000 7f6f7f0eb640 1 --2- 172.21.15.27:0/1103395782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f78071540 0x7f6f781f3c10 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.978+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 >> 172.21.15.27:0/1103395782 conn(0x7f6f780d3d60 msgr2=0x7f6f780b4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.979+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 shutdown_connections 2024-03-06T07:31:34.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:34.980+0000 7f6f7f0eb640 1 -- 172.21.15.27:0/1103395782 wait complete. 2024-03-06T07:31:34.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:31:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.109+0000 7f5f90239640 1 Processor -- start 2024-03-06T07:31:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.110+0000 7f5f90239640 1 -- start start 2024-03-06T07:31:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.110+0000 7f5f90239640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f88071540 0x7f5f88071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.110+0000 7f5f90239640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880cee80 0x7f5f88168900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.110+0000 7f5f90239640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f88071e60 con 0x7f5f880b2d30 2024-03-06T07:31:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.110+0000 7f5f90239640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f880d3230 con 0x7f5f88071540 2024-03-06T07:31:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.110+0000 7f5f90239640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880d4960 con 0x7f5f880cee80 2024-03-06T07:31:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.111+0000 7f5f8dfd5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f88071540 0x7f5f88071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.111+0000 7f5f8dfd5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f88071540 0x7f5f88071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39444/0 (socket says 172.21.15.27:39444) 2024-03-06T07:31:35.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.111+0000 7f5f8dfd5640 1 -- 172.21.15.27:0/4237277711 learned_addr learned my addr 172.21.15.27:0/4237277711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.111+0000 7f5f8e7d6640 1 --2- 172.21.15.27:0/4237277711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880cee80 0x7f5f88168900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.111+0000 7f5f8dfd5640 1 -- 172.21.15.27:0/4237277711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880cee80 msgr2=0x7f5f88168900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.111+0000 7f5f8dfd5640 1 --2- 172.21.15.27:0/4237277711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880cee80 0x7f5f88168900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.111+0000 7f5f8dfd5640 1 -- 172.21.15.27:0/4237277711 >> v1:172.21.15.69:6789/0 conn(0x7f5f880b2d30 legacy=0x7f5f880ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.111+0000 7f5f8dfd5640 1 -- 172.21.15.27:0/4237277711 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f880d36c0 con 0x7f5f88071540 2024-03-06T07:31:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f8dfd5640 1 --2- 172.21.15.27:0/4237277711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 0x7f5f88071920 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009870 tx=0x7f5f7c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad95e16f5fb1f7fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f8cfd3640 1 -- 172.21.15.27:0/4237277711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c034070 con 0x7f5f88071540 2024-03-06T07:31:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f8cfd3640 1 -- 172.21.15.27:0/4237277711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c030840 con 0x7f5f88071540 2024-03-06T07:31:35.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f8cfd3640 1 -- 172.21.15.27:0/4237277711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c030c10 con 0x7f5f88071540 2024-03-06T07:31:35.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f90239640 1 -- 172.21.15.27:0/4237277711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 msgr2=0x7f5f88071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f90239640 1 --2- 172.21.15.27:0/4237277711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 0x7f5f88071920 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009870 tx=0x7f5f7c02b980 comp rx=0 tx=0).stop 2024-03-06T07:31:35.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f90239640 1 -- 172.21.15.27:0/4237277711 shutdown_connections 2024-03-06T07:31:35.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f90239640 1 --2- 172.21.15.27:0/4237277711 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880cee80 0x7f5f88168900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f90239640 1 --2- 172.21.15.27:0/4237277711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 0x7f5f88071920 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.112+0000 7f5f90239640 1 -- 172.21.15.27:0/4237277711 >> 172.21.15.27:0/4237277711 conn(0x7f5f880d3d60 msgr2=0x7f5f880d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.113+0000 7f5f90239640 1 -- 172.21.15.27:0/4237277711 shutdown_connections 2024-03-06T07:31:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.113+0000 7f5f90239640 1 -- 172.21.15.27:0/4237277711 wait complete. 2024-03-06T07:31:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f90239640 1 Processor -- start 2024-03-06T07:31:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f90239640 1 -- start start 2024-03-06T07:31:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f90239640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 0x7f5f881f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f90239640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880b2d30 0x7f5f8814f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f90239640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f880cee80 0x7f5f8815ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f90239640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880d2be0 con 0x7f5f88071540 2024-03-06T07:31:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f90239640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880d2900 con 0x7f5f880cee80 2024-03-06T07:31:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f90239640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880d2ec0 con 0x7f5f880b2d30 2024-03-06T07:31:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f8dfd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 0x7f5f881f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f8dfd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 0x7f5f881f6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39448/0 (socket says 172.21.15.27:39448) 2024-03-06T07:31:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.114+0000 7f5f8dfd5640 1 -- 172.21.15.27:0/3829332545 learned_addr learned my addr 172.21.15.27:0/3829332545 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f8d7d4640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880b2d30 0x7f5f8814f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f8e7d6640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f880cee80 0x7f5f8815ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f8dfd5640 1 -- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880b2d30 msgr2=0x7f5f8814f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f8dfd5640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880b2d30 0x7f5f8814f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f8dfd5640 1 -- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f880cee80 msgr2=0x7f5f8815ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f8dfd5640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f880cee80 0x7f5f8815ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f8dfd5640 1 -- 172.21.15.27:0/3829332545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8814fd60 con 0x7f5f88071540 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f8dfd5640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 0x7f5f881f6f20 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c000c00 tx=0x7f5f7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f7a7fc640 1 -- 172.21.15.27:0/3829332545 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c034040 con 0x7f5f88071540 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f7a7fc640 1 -- 172.21.15.27:0/3829332545 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c039d90 con 0x7f5f88071540 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f88150040 con 0x7f5f88071540 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f7a7fc640 1 -- 172.21.15.27:0/3829332545 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c043610 con 0x7f5f88071540 2024-03-06T07:31:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.115+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f88150600 con 0x7f5f88071540 2024-03-06T07:31:35.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.116+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f88072070 con 0x7f5f88071540 2024-03-06T07:31:35.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.117+0000 7f5f7a7fc640 1 -- 172.21.15.27:0/3829332545 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f7c04a060 con 0x7f5f88071540 2024-03-06T07:31:35.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.117+0000 7f5f7a7fc640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f6c06b4d0 0x7f5f6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.117+0000 7f5f7a7fc640 1 -- 172.21.15.27:0/3829332545 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f5f7c0ac190 con 0x7f5f88071540 2024-03-06T07:31:35.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.117+0000 7f5f8d7d4640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f6c06b4d0 0x7f5f6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.118+0000 7f5f8d7d4640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f6c06b4d0 0x7f5f6c06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5f8815da30 tx=0x7f5f84009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.122+0000 7f5f7a7fc640 1 -- 172.21.15.27:0/3829332545 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f7c03e070 con 0x7f5f88071540 2024-03-06T07:31:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.306+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5f8804f410 con 0x7f5f88071540 2024-03-06T07:31:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.975+0000 7f5f7a7fc640 1 -- 172.21.15.27:0/3829332545 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v139) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5f7c02d330 con 0x7f5f88071540 2024-03-06T07:31:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:31:35.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f6c06b4d0 msgr2=0x7f5f6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f6c06b4d0 0x7f5f6c06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5f8815da30 tx=0x7f5f84009040 comp rx=0 tx=0).stop 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 msgr2=0x7f5f881f6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 0x7f5f881f6f20 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c000c00 tx=0x7f5f7c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 shutdown_connections 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5f6c06b4d0 0x7f5f6c06d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f880cee80 0x7f5f8815ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5f880b2d30 0x7f5f8814f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 --2- 172.21.15.27:0/3829332545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f88071540 0x7f5f881f6f20 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 >> 172.21.15.27:0/3829332545 conn(0x7f5f880d3d60 msgr2=0x7f5f8815b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 shutdown_connections 2024-03-06T07:31:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:35.980+0000 7f5f90239640 1 -- 172.21.15.27:0/3829332545 wait complete. 2024-03-06T07:31:36.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:31:36.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.112+0000 7fa6729bf640 1 Processor -- start 2024-03-06T07:31:36.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.112+0000 7fa6729bf640 1 -- start start 2024-03-06T07:31:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.113+0000 7fa6729bf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa66c15dbd0 0x7fa66c15ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.113+0000 7fa6729bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 0x7fa66c0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.113+0000 7fa6729bf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c15f3b0 con 0x7fa66c15dfb0 2024-03-06T07:31:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.113+0000 7fa6729bf640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66c06f070 con 0x7fa66c15dbd0 2024-03-06T07:31:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.113+0000 7fa6729bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c071940 con 0x7fa66c0b5840 2024-03-06T07:31:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.113+0000 7fa66bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa66c15dbd0 0x7fa66c15ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.113+0000 7fa66bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa66c15dbd0 0x7fa66c15ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39454/0 (socket says 172.21.15.27:39454) 2024-03-06T07:31:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.113+0000 7fa66bfff640 1 -- 172.21.15.27:0/1495510631 learned_addr learned my addr 172.21.15.27:0/1495510631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.114+0000 7fa66b7fe640 1 --2- 172.21.15.27:0/1495510631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 0x7fa66c0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.114+0000 7fa66bfff640 1 -- 172.21.15.27:0/1495510631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 msgr2=0x7fa66c0b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.114+0000 7fa66bfff640 1 --2- 172.21.15.27:0/1495510631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 0x7fa66c0b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.114+0000 7fa66bfff640 1 -- 172.21.15.27:0/1495510631 >> v1:172.21.15.69:6789/0 conn(0x7fa66c15dfb0 legacy=0x7fa66c161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.114+0000 7fa66bfff640 1 -- 172.21.15.27:0/1495510631 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c0b5db0 con 0x7fa66c15dbd0 2024-03-06T07:31:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.114+0000 7fa66b7fe640 1 --2- 172.21.15.27:0/1495510631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 0x7fa66c0b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.114+0000 7fa66bfff640 1 --2- 172.21.15.27:0/1495510631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dbd0 0x7fa66c15ee70 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fa654009870 tx=0x7fa65402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3f8ae9a2760173 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa66affd640 1 -- 172.21.15.27:0/1495510631 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa654035020 con 0x7fa66c15dbd0 2024-03-06T07:31:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa66affd640 1 -- 172.21.15.27:0/1495510631 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa654031640 con 0x7fa66c15dbd0 2024-03-06T07:31:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa66affd640 1 -- 172.21.15.27:0/1495510631 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65402d040 con 0x7fa66c15dbd0 2024-03-06T07:31:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa6729bf640 1 -- 172.21.15.27:0/1495510631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dbd0 msgr2=0x7fa66c15ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa6729bf640 1 --2- 172.21.15.27:0/1495510631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dbd0 0x7fa66c15ee70 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fa654009870 tx=0x7fa65402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa6729bf640 1 -- 172.21.15.27:0/1495510631 shutdown_connections 2024-03-06T07:31:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa6729bf640 1 --2- 172.21.15.27:0/1495510631 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 0x7fa66c0b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa6729bf640 1 --2- 172.21.15.27:0/1495510631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dbd0 0x7fa66c15ee70 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa6729bf640 1 -- 172.21.15.27:0/1495510631 >> 172.21.15.27:0/1495510631 conn(0x7fa66c0b26b0 msgr2=0x7fa66c0b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa6729bf640 1 -- 172.21.15.27:0/1495510631 shutdown_connections 2024-03-06T07:31:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.115+0000 7fa6729bf640 1 -- 172.21.15.27:0/1495510631 wait complete. 2024-03-06T07:31:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.116+0000 7fa6729bf640 1 Processor -- start 2024-03-06T07:31:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa6729bf640 1 -- start start 2024-03-06T07:31:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa6729bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 0x7fa66c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa6729bf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa66c15dbd0 0x7fa66c1494b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa6729bf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dfb0 0x7fa66c0af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa6729bf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0721c0 con 0x7fa66c15dfb0 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa6729bf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c071ee0 con 0x7fa66c15dbd0 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa6729bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0726e0 con 0x7fa66c0b5840 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa670f5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dfb0 0x7fa66c0af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa670f5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dfb0 0x7fa66c0af050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39464/0 (socket says 172.21.15.27:39464) 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.117+0000 7fa670f5c640 1 -- 172.21.15.27:0/3493524747 learned_addr learned my addr 172.21.15.27:0/3493524747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa66bfff640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 0x7fa66c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa66b7fe640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa66c15dbd0 0x7fa66c1494b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa670f5c640 1 -- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 msgr2=0x7fa66c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa670f5c640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 0x7fa66c0ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa670f5c640 1 -- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa66c15dbd0 msgr2=0x7fa66c1494b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa670f5c640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa66c15dbd0 0x7fa66c1494b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa670f5c640 1 -- 172.21.15.27:0/3493524747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c155cd0 con 0x7fa66c15dfb0 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa670f5c640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dfb0 0x7fa66c0af050 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fa66400b970 tx=0x7fa66400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa668ff9640 1 -- 172.21.15.27:0/3493524747 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664019070 con 0x7fa66c15dfb0 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa668ff9640 1 -- 172.21.15.27:0/3493524747 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66400cb40 con 0x7fa66c15dfb0 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66c1499f0 con 0x7fa66c15dfb0 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa668ff9640 1 -- 172.21.15.27:0/3493524747 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664014440 con 0x7fa66c15dfb0 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.118+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66c0adfb0 con 0x7fa66c15dfb0 2024-03-06T07:31:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.119+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66c073380 con 0x7fa66c15dfb0 2024-03-06T07:31:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.120+0000 7fa668ff9640 1 -- 172.21.15.27:0/3493524747 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa66401c070 con 0x7fa66c15dfb0 2024-03-06T07:31:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.120+0000 7fa668ff9640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.120+0000 7fa668ff9640 1 -- 172.21.15.27:0/3493524747 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa664089ad0 con 0x7fa66c15dfb0 2024-03-06T07:31:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.120+0000 7fa66bfff640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.121+0000 7fa66bfff640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa654036000 tx=0x7fa65403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.125+0000 7fa668ff9640 1 -- 172.21.15.27:0/3493524747 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa664058690 con 0x7fa66c15dfb0 2024-03-06T07:31:36.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.306+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa66c04f410 con 0x7fa66c15dfb0 2024-03-06T07:31:36.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.309+0000 7fa668ff9640 1 -- 172.21.15.27:0/3493524747 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa66405c730 con 0x7fa66c15dfb0 2024-03-06T07:31:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.311+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 msgr2=0x7fa64c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.311+0000 7fa6729bf640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa654036000 tx=0x7fa65403c040 comp rx=0 tx=0).stop 2024-03-06T07:31:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.311+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dfb0 msgr2=0x7fa66c0af050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.311+0000 7fa6729bf640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dfb0 0x7fa66c0af050 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fa66400b970 tx=0x7fa66400be30 comp rx=0 tx=0).stop 2024-03-06T07:31:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.311+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 shutdown_connections 2024-03-06T07:31:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.312+0000 7fa6729bf640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa64c06b4d0 0x7fa64c06d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.312+0000 7fa6729bf640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa66c15dfb0 0x7fa66c0af050 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.312+0000 7fa6729bf640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa66c15dbd0 0x7fa66c1494b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.312+0000 7fa6729bf640 1 --2- 172.21.15.27:0/3493524747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa66c0b5840 0x7fa66c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.312+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 >> 172.21.15.27:0/3493524747 conn(0x7fa66c0b26b0 msgr2=0x7fa66c15e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.312+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 shutdown_connections 2024-03-06T07:31:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.312+0000 7fa6729bf640 1 -- 172.21.15.27:0/3493524747 wait complete. 2024-03-06T07:31:36.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:31:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.433+0000 7fd9a933e640 1 Processor -- start 2024-03-06T07:31:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.433+0000 7fd9a933e640 1 -- start start 2024-03-06T07:31:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a933e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9a415bda0 0x7fd9a40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a933e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40cea70 0x7fd9a406f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a933e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40cd350 con 0x7fd9a40b0ed0 2024-03-06T07:31:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a933e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a406eb20 con 0x7fd9a415bda0 2024-03-06T07:31:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a933e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40b1330 con 0x7fd9a40cea70 2024-03-06T07:31:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a2ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9a415bda0 0x7fd9a40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a37fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40cea70 0x7fd9a406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a27fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd9a40b0ed0 0x7fd9a40ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:45844/0 (socket says 172.21.15.27:45844) 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a27fc640 1 -- 172.21.15.27:0/317591629 learned_addr learned my addr 172.21.15.27:0/317591629 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a37fe640 1 --2- 172.21.15.27:0/317591629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40cea70 0x7fd9a406f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:37754/0 (socket says 172.21.15.27:37754) 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.434+0000 7fd9a2ffd640 1 --2- 172.21.15.27:0/317591629 >> v2:172.21.15.69:3300/0 conn(0x7fd9a415bda0 0x7fd9a40d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:58046/0 (socket says 172.21.15.27:58046) 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.435+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/317591629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40cea70 msgr2=0x7fd9a406f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.435+0000 7fd9a2ffd640 1 --2- 172.21.15.27:0/317591629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40cea70 0x7fd9a406f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.435+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/317591629 >> v1:172.21.15.69:6789/0 conn(0x7fd9a40b0ed0 legacy=0x7fd9a40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.435+0000 7fd9a2ffd640 1 -- 172.21.15.27:0/317591629 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a40cd630 con 0x7fd9a415bda0 2024-03-06T07:31:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.435+0000 7fd9a37fe640 1 --2- 172.21.15.27:0/317591629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40cea70 0x7fd9a406f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:36.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.435+0000 7fd9a2ffd640 1 --2- 172.21.15.27:0/317591629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9a415bda0 0x7fd9a40d25b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd9940097a0 tx=0x7fd99402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0e1d765f284dc3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a1ffb640 1 -- 172.21.15.27:0/317591629 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994035020 con 0x7fd9a415bda0 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a1ffb640 1 -- 172.21.15.27:0/317591629 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd994031830 con 0x7fd9a415bda0 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a1ffb640 1 -- 172.21.15.27:0/317591629 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994031c00 con 0x7fd9a415bda0 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a933e640 1 -- 172.21.15.27:0/317591629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9a415bda0 msgr2=0x7fd9a40d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a933e640 1 --2- 172.21.15.27:0/317591629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9a415bda0 0x7fd9a40d25b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd9940097a0 tx=0x7fd99402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a933e640 1 -- 172.21.15.27:0/317591629 shutdown_connections 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a933e640 1 --2- 172.21.15.27:0/317591629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40cea70 0x7fd9a406f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a933e640 1 --2- 172.21.15.27:0/317591629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9a415bda0 0x7fd9a40d25b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a933e640 1 -- 172.21.15.27:0/317591629 >> 172.21.15.27:0/317591629 conn(0x7fd9a40d1a90 msgr2=0x7fd9a40d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a933e640 1 -- 172.21.15.27:0/317591629 shutdown_connections 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.436+0000 7fd9a933e640 1 -- 172.21.15.27:0/317591629 wait complete. 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.437+0000 7fd9a933e640 1 Processor -- start 2024-03-06T07:31:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.437+0000 7fd9a933e640 1 -- start start 2024-03-06T07:31:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a933e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40b0ed0 0x7fd9a40c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a933e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a40cea70 0x7fd9a406b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a933e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9a415bda0 0x7fd9a40b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a933e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a415c180 con 0x7fd9a40cea70 2024-03-06T07:31:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a933e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a406ed70 con 0x7fd9a415bda0 2024-03-06T07:31:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a933e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a415c300 con 0x7fd9a40b0ed0 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a40cea70 0x7fd9a406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a40cea70 0x7fd9a406b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39470/0 (socket says 172.21.15.27:39470) 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a27fc640 1 -- 172.21.15.27:0/2913152971 learned_addr learned my addr 172.21.15.27:0/2913152971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a37fe640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9a415bda0 0x7fd9a40b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.438+0000 7fd9a2ffd640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40b0ed0 0x7fd9a40c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd9a27fc640 1 -- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40b0ed0 msgr2=0x7fd9a40c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd9a27fc640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40b0ed0 0x7fd9a40c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd9a27fc640 1 -- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9a415bda0 msgr2=0x7fd9a40b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd9a27fc640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9a415bda0 0x7fd9a40b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd9a27fc640 1 -- 172.21.15.27:0/2913152971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a415da70 con 0x7fd9a40cea70 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd9a27fc640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a40cea70 0x7fd9a406b900 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fd99800c3c0 tx=0x7fd99800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd98b7fe640 1 -- 172.21.15.27:0/2913152971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd998017070 con 0x7fd9a40cea70 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd98b7fe640 1 -- 172.21.15.27:0/2913152971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd998005ce0 con 0x7fd9a40cea70 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a415dd50 con 0x7fd9a40cea70 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd98b7fe640 1 -- 172.21.15.27:0/2913152971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9980072f0 con 0x7fd9a40cea70 2024-03-06T07:31:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.439+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a415e310 con 0x7fd9a40cea70 2024-03-06T07:31:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.440+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a4067ff0 con 0x7fd9a40cea70 2024-03-06T07:31:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.441+0000 7fd98b7fe640 1 -- 172.21.15.27:0/2913152971 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd99801a070 con 0x7fd9a40cea70 2024-03-06T07:31:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.441+0000 7fd98b7fe640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.441+0000 7fd98b7fe640 1 -- 172.21.15.27:0/2913152971 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd998089110 con 0x7fd9a40cea70 2024-03-06T07:31:36.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.441+0000 7fd9a2ffd640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.442+0000 7fd9a2ffd640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd994036000 tx=0x7fd994008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.445+0000 7fd98b7fe640 1 -- 172.21.15.27:0/2913152971 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd998057c40 con 0x7fd9a40cea70 2024-03-06T07:31:36.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.696+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7fd9a40ccca0 con 0x7fd9a40cea70 2024-03-06T07:31:36.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.980+0000 7fd98b7fe640 1 -- 172.21.15.27:0/2913152971 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v476) v1 ==== 140+0+0 (secure 0 0 0) 0x7fd99805bce0 con 0x7fd9a40cea70 2024-03-06T07:31:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 msgr2=0x7fd99006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd994036000 tx=0x7fd994008030 comp rx=0 tx=0).stop 2024-03-06T07:31:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a40cea70 msgr2=0x7fd9a406b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a40cea70 0x7fd9a406b900 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fd99800c3c0 tx=0x7fd99800c7c0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 shutdown_connections 2024-03-06T07:31:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd99006b4d0 0x7fd99006d990 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9a415bda0 0x7fd9a40b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9a40cea70 0x7fd9a406b900 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 --2- 172.21.15.27:0/2913152971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9a40b0ed0 0x7fd9a40c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.982+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 >> 172.21.15.27:0/2913152971 conn(0x7fd9a40d1a90 msgr2=0x7fd9a40b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.983+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 shutdown_connections 2024-03-06T07:31:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:36.983+0000 7fd9a933e640 1 -- 172.21.15.27:0/2913152971 wait complete. 2024-03-06T07:31:36.997 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 ... ok 2024-03-06T07:31:36.997 INFO:tasks.cephfs_test_runner:Starting test: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:31:36.997 INFO:tasks.cephfs_test_runner:test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:31:36.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:37.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.104+0000 7fa40cc29640 1 Processor -- start 2024-03-06T07:31:37.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.104+0000 7fa40cc29640 1 -- start start 2024-03-06T07:31:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa40cc29640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa408071c50 0x7fa4080b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa40cc29640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4080b6f20 con 0x7fa4080d18c0 2024-03-06T07:31:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa40cc29640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4080d3030 con 0x7fa40815df70 2024-03-06T07:31:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa40cc29640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4080d3310 con 0x7fa408071c50 2024-03-06T07:31:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa3fdd9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa4080d18c0 0x7fa408071540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53114/0 (socket says 172.21.15.27:53114) 2024-03-06T07:31:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa3fdd9b640 1 -- 172.21.15.27:0/2046206074 learned_addr learned my addr 172.21.15.27:0/2046206074 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa406d9d640 1 --2- 172.21.15.27:0/2046206074 >> v2:172.21.15.69:3300/0 conn(0x7fa408071c50 0x7fa4080b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211113405 0 0) 0x7fa4080b6f20 con 0x7fa4080d18c0 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f4003520 con 0x7fa4080d18c0 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3473097649 0 0) 0x7fa3f4003520 con 0x7fa4080d18c0 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 >> v2:172.21.15.69:3300/0 conn(0x7fa408071c50 msgr2=0x7fa4080b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 --2- 172.21.15.27:0/2046206074 >> v2:172.21.15.69:3300/0 conn(0x7fa408071c50 0x7fa4080b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 >> v1:172.21.15.69:6789/0 conn(0x7fa40815df70 legacy=0x7fa4080d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4080d35f0 con 0x7fa4080d18c0 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fa3f0001670 con 0x7fa4080d18c0 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 >> v1:172.21.15.27:6789/0 conn(0x7fa4080d18c0 legacy=0x7fa408071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3f40039b0 0x7fa3f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4004800 0x7fa3f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa3f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080d3310 con 0x7fa3f40051c0 2024-03-06T07:31:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f400c070 con 0x7fa3f4004800 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.105+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f400c270 con 0x7fa3f40039b0 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.106+0000 7fa406d9d640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa3f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.106+0000 7fa3fdd9b640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4004800 0x7fa3f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.106+0000 7fa40659c640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3f40039b0 0x7fa3f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.106+0000 7fa406d9d640 1 -- 172.21.15.27:0/2046206074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3f40039b0 msgr2=0x7fa3f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.106+0000 7fa406d9d640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3f40039b0 0x7fa3f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.106+0000 7fa406d9d640 1 -- 172.21.15.27:0/2046206074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4004800 msgr2=0x7fa3f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.106+0000 7fa406d9d640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4004800 0x7fa3f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.106+0000 7fa406d9d640 1 -- 172.21.15.27:0/2046206074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f4003520 con 0x7fa3f40051c0 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.106+0000 7fa406d9d640 1 -- 172.21.15.27:0/2046206074 reap_dead start 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa406d9d640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa3f400b9c0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa400009f90 tx=0x7fa3f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c00fe039a47e4b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400039070 con 0x7fa3f40051c0 2024-03-06T07:31:37.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40002cc20 con 0x7fa3f40051c0 2024-03-06T07:31:37.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa405d9b640 1 -- 172.21.15.27:0/2046206074 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400009c10 con 0x7fa3f40051c0 2024-03-06T07:31:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa40cc29640 1 -- 172.21.15.27:0/2046206074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 msgr2=0x7fa3f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa40cc29640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa3f400b9c0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa400009f90 tx=0x7fa3f4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa40cc29640 1 -- 172.21.15.27:0/2046206074 shutdown_connections 2024-03-06T07:31:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa40cc29640 1 --2- 172.21.15.27:0/2046206074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa3f400b9c0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa40cc29640 1 -- 172.21.15.27:0/2046206074 >> 172.21.15.27:0/2046206074 conn(0x7fa4080d3cf0 msgr2=0x7fa4080d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa40cc29640 1 -- 172.21.15.27:0/2046206074 shutdown_connections 2024-03-06T07:31:37.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.107+0000 7fa40cc29640 1 -- 172.21.15.27:0/2046206074 wait complete. 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.109+0000 7fa40cc29640 1 Processor -- start 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.109+0000 7fa40cc29640 1 -- start start 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.109+0000 7fa40cc29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa4081510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.109+0000 7fa40cc29640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa40815a7e0 0x7fa4080c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.109+0000 7fa40cc29640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4080c8d90 0x7fa4081f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.109+0000 7fa40cc29640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40815e4d0 con 0x7fa3f40051c0 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.109+0000 7fa40cc29640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40815e350 con 0x7fa4080c8d90 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.109+0000 7fa40cc29640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa408071540 con 0x7fa40815a7e0 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa4081510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa4081510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39482/0 (socket says 172.21.15.27:39482) 2024-03-06T07:31:37.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40659c640 1 -- 172.21.15.27:0/571554458 learned_addr learned my addr 172.21.15.27:0/571554458 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa3fdd9b640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa40815a7e0 0x7fa4080c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa406d9d640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4080c8d90 0x7fa4081f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40659c640 1 -- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa40815a7e0 msgr2=0x7fa4080c8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40659c640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa40815a7e0 0x7fa4080c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40659c640 1 -- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4080c8d90 msgr2=0x7fa4081f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40659c640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4080c8d90 0x7fa4081f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40659c640 1 -- 172.21.15.27:0/571554458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4080c8350 con 0x7fa3f40051c0 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa406d9d640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4080c8d90 0x7fa4081f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40659c640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa4081510d0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800f5f0 tx=0x7fa3f800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa3ff7fe640 1 -- 172.21.15.27:0/571554458 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80052e0 con 0x7fa3f40051c0 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa3ff7fe640 1 -- 172.21.15.27:0/571554458 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f8018070 con 0x7fa3f40051c0 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4080ca230 con 0x7fa3f40051c0 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa3ff7fe640 1 -- 172.21.15.27:0/571554458 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8003810 con 0x7fa3f40051c0 2024-03-06T07:31:37.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.110+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40815a290 con 0x7fa3f40051c0 2024-03-06T07:31:37.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.111+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4080c9170 con 0x7fa3f40051c0 2024-03-06T07:31:37.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.112+0000 7fa3ff7fe640 1 -- 172.21.15.27:0/571554458 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3f80039b0 con 0x7fa3f40051c0 2024-03-06T07:31:37.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.112+0000 7fa3ff7fe640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3e806b4d0 0x7fa3e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.112+0000 7fa3ff7fe640 1 -- 172.21.15.27:0/571554458 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa3f8089be0 con 0x7fa3f40051c0 2024-03-06T07:31:37.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.112+0000 7fa3fdd9b640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3e806b4d0 0x7fa3e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.113+0000 7fa3fdd9b640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3e806b4d0 0x7fa3e806d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa4080c7770 tx=0x7fa3f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.117+0000 7fa3ff7fe640 1 -- 172.21.15.27:0/571554458 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3f8058d60 con 0x7fa3f40051c0 2024-03-06T07:31:37.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.300+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa408067ff0 con 0x7fa3f40051c0 2024-03-06T07:31:37.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.301+0000 7fa3ff7fe640 1 -- 172.21.15.27:0/571554458 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+12958 (secure 0 0 0) 0x7fa3f8058660 con 0x7fa3f40051c0 2024-03-06T07:31:37.304 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:37.304 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":139,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:31:35.975356+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/4036525552":"2024-03-07T07:31:32.968014+0000","172.21.15.27:6833/4036525552":"2024-03-07T07:31:32.968014+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3e806b4d0 msgr2=0x7fa3e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3e806b4d0 0x7fa3e806d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa4080c7770 tx=0x7fa3f0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 msgr2=0x7fa4081510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa4081510d0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800f5f0 tx=0x7fa3f800fab0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 shutdown_connections 2024-03-06T07:31:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3e806b4d0 0x7fa3e806d990 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4080c8d90 0x7fa4081f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa40815a7e0 0x7fa4080c8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 --2- 172.21.15.27:0/571554458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3f40051c0 0x7fa4081510d0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.303+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 >> 172.21.15.27:0/571554458 conn(0x7fa4080d3cf0 msgr2=0x7fa408160f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.304+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 shutdown_connections 2024-03-06T07:31:37.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.304+0000 7fa40cc29640 1 -- 172.21.15.27:0/571554458 wait complete. 2024-03-06T07:31:37.317 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:37.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:37.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.425+0000 7fbf08df7640 1 Processor -- start 2024-03-06T07:31:37.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.425+0000 7fbf08df7640 1 -- start start 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf08df7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf0415dbd0 0x7fbf0415ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf08df7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 0x7fbf040b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf08df7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0415f3b0 con 0x7fbf0415dfb0 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf08df7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0406f070 con 0x7fbf0415dbd0 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf08df7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf04071940 con 0x7fbf040b5840 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf0259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf0415dbd0 0x7fbf0415ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf0259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf0415dbd0 0x7fbf0415ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39494/0 (socket says 172.21.15.27:39494) 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf0259c640 1 -- 172.21.15.27:0/2256155958 learned_addr learned my addr 172.21.15.27:0/2256155958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf0259c640 1 -- 172.21.15.27:0/2256155958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 msgr2=0x7fbf040b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf0259c640 1 --2- 172.21.15.27:0/2256155958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 0x7fbf040b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf0259c640 1 -- 172.21.15.27:0/2256155958 >> v1:172.21.15.69:6789/0 conn(0x7fbf0415dfb0 legacy=0x7fbf04161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.426+0000 7fbf0259c640 1 -- 172.21.15.27:0/2256155958 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf040b5db0 con 0x7fbf0415dbd0 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.427+0000 7fbf0259c640 1 --2- 172.21.15.27:0/2256155958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dbd0 0x7fbf0415ee70 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009870 tx=0x7fbefc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79dadf14396dbeaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.427+0000 7fbf0159a640 1 -- 172.21.15.27:0/2256155958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc035020 con 0x7fbf0415dbd0 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.427+0000 7fbf0159a640 1 -- 172.21.15.27:0/2256155958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc031640 con 0x7fbf0415dbd0 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.427+0000 7fbf0159a640 1 -- 172.21.15.27:0/2256155958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc02d040 con 0x7fbf0415dbd0 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.428+0000 7fbf08df7640 1 -- 172.21.15.27:0/2256155958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dbd0 msgr2=0x7fbf0415ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.428+0000 7fbf08df7640 1 --2- 172.21.15.27:0/2256155958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dbd0 0x7fbf0415ee70 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009870 tx=0x7fbefc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.428+0000 7fbf08df7640 1 -- 172.21.15.27:0/2256155958 shutdown_connections 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.428+0000 7fbf08df7640 1 --2- 172.21.15.27:0/2256155958 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 0x7fbf040b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.428+0000 7fbf08df7640 1 --2- 172.21.15.27:0/2256155958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dbd0 0x7fbf0415ee70 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.428+0000 7fbf08df7640 1 -- 172.21.15.27:0/2256155958 >> 172.21.15.27:0/2256155958 conn(0x7fbf040b26b0 msgr2=0x7fbf040b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.428+0000 7fbf08df7640 1 -- 172.21.15.27:0/2256155958 shutdown_connections 2024-03-06T07:31:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.428+0000 7fbf08df7640 1 -- 172.21.15.27:0/2256155958 wait complete. 2024-03-06T07:31:37.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.429+0000 7fbf08df7640 1 Processor -- start 2024-03-06T07:31:37.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.429+0000 7fbf08df7640 1 -- start start 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf08df7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 0x7fbf04149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf08df7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0415dbd0 0x7fbf040b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf08df7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dfb0 0x7fbf040b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf08df7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040721c0 con 0x7fbf0415dfb0 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf08df7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf04071ee0 con 0x7fbf0415dbd0 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf08df7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040726e0 con 0x7fbf040b5840 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dfb0 0x7fbf040b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dfb0 0x7fbf040b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39510/0 (socket says 172.21.15.27:39510) 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf02d9d640 1 -- 172.21.15.27:0/2695468463 learned_addr learned my addr 172.21.15.27:0/2695468463 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf0259c640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 0x7fbf04149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf01d9b640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0415dbd0 0x7fbf040b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf02d9d640 1 -- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 msgr2=0x7fbf04149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf02d9d640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 0x7fbf04149440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf02d9d640 1 -- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0415dbd0 msgr2=0x7fbf040b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.430+0000 7fbf02d9d640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0415dbd0 0x7fbf040b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.431+0000 7fbf02d9d640 1 -- 172.21.15.27:0/2695468463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf04156df0 con 0x7fbf0415dfb0 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.431+0000 7fbf0259c640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 0x7fbf04149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.431+0000 7fbf01d9b640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0415dbd0 0x7fbf040b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.431+0000 7fbf02d9d640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dfb0 0x7fbf040b0e80 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fbef400b990 tx=0x7fbef400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.431+0000 7fbefaffd640 1 -- 172.21.15.27:0/2695468463 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4019070 con 0x7fbf0415dfb0 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.431+0000 7fbefaffd640 1 -- 172.21.15.27:0/2695468463 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef400cba0 con 0x7fbf0415dfb0 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.431+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf041570d0 con 0x7fbf0415dfb0 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.431+0000 7fbefaffd640 1 -- 172.21.15.27:0/2695468463 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4014440 con 0x7fbf0415dfb0 2024-03-06T07:31:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.431+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf04157690 con 0x7fbf0415dfb0 2024-03-06T07:31:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.432+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf04073380 con 0x7fbf0415dfb0 2024-03-06T07:31:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.432+0000 7fbefaffd640 1 -- 172.21.15.27:0/2695468463 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbef401c070 con 0x7fbf0415dfb0 2024-03-06T07:31:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.433+0000 7fbefaffd640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbee406b4d0 0x7fbee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.433+0000 7fbefaffd640 1 -- 172.21.15.27:0/2695468463 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbef4089b90 con 0x7fbf0415dfb0 2024-03-06T07:31:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.433+0000 7fbf0259c640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbee406b4d0 0x7fbee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.434+0000 7fbf0259c640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbee406b4d0 0x7fbee406d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fbefc036000 tx=0x7fbefc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.437+0000 7fbefaffd640 1 -- 172.21.15.27:0/2695468463 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbef4058da0 con 0x7fbf0415dfb0 2024-03-06T07:31:37.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.616+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbf04158ce0 con 0x7fbf0415dfb0 2024-03-06T07:31:37.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.616+0000 7fbefaffd640 1 -- 172.21.15.27:0/2695468463 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v139) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbef40586a0 con 0x7fbf0415dfb0 2024-03-06T07:31:37.618 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:31:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbee406b4d0 msgr2=0x7fbee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbee406b4d0 0x7fbee406d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fbefc036000 tx=0x7fbefc03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dfb0 msgr2=0x7fbf040b0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dfb0 0x7fbf040b0e80 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fbef400b990 tx=0x7fbef400be50 comp rx=0 tx=0).stop 2024-03-06T07:31:37.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 shutdown_connections 2024-03-06T07:31:37.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbee406b4d0 0x7fbee406d990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf0415dfb0 0x7fbf040b0e80 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf0415dbd0 0x7fbf040b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 --2- 172.21.15.27:0/2695468463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf040b5840 0x7fbf04149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 >> 172.21.15.27:0/2695468463 conn(0x7fbf040b26b0 msgr2=0x7fbf0406e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 shutdown_connections 2024-03-06T07:31:37.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.618+0000 7fbf08df7640 1 -- 172.21.15.27:0/2695468463 wait complete. 2024-03-06T07:31:37.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:31:37.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.736+0000 7f41e3b3a640 1 Processor -- start 2024-03-06T07:31:37.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.736+0000 7f41e3b3a640 1 -- start start 2024-03-06T07:31:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e3b3a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f41dc145780 0x7f41dc0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e3b3a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc0cf3d0 con 0x7f41dc066d00 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e3b3a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc0cf6b0 con 0x7f41dc0ceed0 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e3b3a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41dc072c30 con 0x7f41dc145780 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e20d7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f41dc066d00 0x7f41dc06d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53126/0 (socket says 172.21.15.27:53126) 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e20d7640 1 -- 172.21.15.27:0/2117857414 learned_addr learned my addr 172.21.15.27:0/2117857414 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759220406 0 0) 0x7f41dc0cf3d0 con 0x7f41dc066d00 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e18d6640 1 --2- 172.21.15.27:0/2117857414 >> v2:172.21.15.69:3300/0 conn(0x7f41dc145780 0x7f41dc0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c8003520 con 0x7f41dc066d00 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1712056887 0 0) 0x7f41c8003520 con 0x7f41dc066d00 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 >> v2:172.21.15.69:3300/0 conn(0x7f41dc145780 msgr2=0x7f41dc0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e08d4640 1 --2- 172.21.15.27:0/2117857414 >> v2:172.21.15.69:3300/0 conn(0x7f41dc145780 0x7f41dc0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 >> v1:172.21.15.69:6789/0 conn(0x7f41dc0ceed0 legacy=0x7f41dc072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc0cf990 con 0x7f41dc066d00 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.737+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f41d40019f0 con 0x7f41dc066d00 2024-03-06T07:31:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 >> v1:172.21.15.27:6789/0 conn(0x7f41dc066d00 legacy=0x7f41dc06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e08d4640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e08d4640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41c8004800 0x7f41c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e08d4640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41c8005210 0x7f41c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc072c30 con 0x7f41c8003a00 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c800c160 con 0x7f41c8005210 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c800c360 con 0x7f41c8004800 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e18d6640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e20d7640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41c8005210 0x7f41c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e10d5640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41c8004800 0x7f41c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e18d6640 1 -- 172.21.15.27:0/2117857414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41c8004800 msgr2=0x7f41c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e18d6640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41c8004800 0x7f41c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e18d6640 1 -- 172.21.15.27:0/2117857414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41c8005210 msgr2=0x7f41c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e18d6640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41c8005210 0x7f41c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e18d6640 1 -- 172.21.15.27:0/2117857414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c8003520 con 0x7f41c8003a00 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.738+0000 7f41e20d7640 1 -- 172.21.15.27:0/2117857414 reap_dead start 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.739+0000 7f41e18d6640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41c8003e20 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009a90 tx=0x7f41c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55947cad38ba3109 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.739+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8039070 con 0x7f41c8003a00 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.739+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d802c680 con 0x7f41c8003a00 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.739+0000 7f41e08d4640 1 -- 172.21.15.27:0/2117857414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8009690 con 0x7f41c8003a00 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.739+0000 7f41e3b3a640 1 -- 172.21.15.27:0/2117857414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 msgr2=0x7f41c8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.739+0000 7f41e3b3a640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41c8003e20 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009a90 tx=0x7f41c8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.739+0000 7f41e3b3a640 1 -- 172.21.15.27:0/2117857414 shutdown_connections 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.739+0000 7f41e3b3a640 1 --2- 172.21.15.27:0/2117857414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41c8003e20 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.739+0000 7f41e3b3a640 1 -- 172.21.15.27:0/2117857414 >> 172.21.15.27:0/2117857414 conn(0x7f41dc0b2720 msgr2=0x7f41dc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.740+0000 7f41e3b3a640 1 -- 172.21.15.27:0/2117857414 shutdown_connections 2024-03-06T07:31:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.740+0000 7f41e3b3a640 1 -- 172.21.15.27:0/2117857414 wait complete. 2024-03-06T07:31:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.741+0000 7f41e3b3a640 1 Processor -- start 2024-03-06T07:31:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.741+0000 7f41e3b3a640 1 -- start start 2024-03-06T07:31:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.741+0000 7f41e3b3a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41dc1e41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.741+0000 7f41e3b3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc13ece0 0x7f41dc1f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.741+0000 7f41e3b3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc1432b0 0x7f41dc143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.741+0000 7f41e3b3a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0d05f0 con 0x7f41c8003a00 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.741+0000 7f41e3b3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0d0470 con 0x7f41dc13ece0 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.741+0000 7f41e3b3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0d0770 con 0x7f41dc1432b0 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e18d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41dc1e41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e18d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41dc1e41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39532/0 (socket says 172.21.15.27:39532) 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e18d6640 1 -- 172.21.15.27:0/289862922 learned_addr learned my addr 172.21.15.27:0/289862922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e10d5640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc13ece0 0x7f41dc1f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e18d6640 1 -- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc1432b0 msgr2=0x7f41dc143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e20d7640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc1432b0 0x7f41dc143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e18d6640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc1432b0 0x7f41dc143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e18d6640 1 -- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc13ece0 msgr2=0x7f41dc1f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e18d6640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc13ece0 0x7f41dc1f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e18d6640 1 -- 172.21.15.27:0/289862922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc13f4c0 con 0x7f41c8003a00 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e10d5640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc13ece0 0x7f41dc1f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e20d7640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc1432b0 0x7f41dc143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e18d6640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41dc1e41b0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009a90 tx=0x7f41d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41d27fc640 1 -- 172.21.15.27:0/289862922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8039050 con 0x7f41c8003a00 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41d27fc640 1 -- 172.21.15.27:0/289862922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d803a040 con 0x7f41c8003a00 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41d27fc640 1 -- 172.21.15.27:0/289862922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d80044e0 con 0x7f41c8003a00 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc13f7a0 con 0x7f41c8003a00 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.742+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc13fd60 con 0x7f41c8003a00 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.743+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc067ff0 con 0x7f41c8003a00 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.744+0000 7f41d27fc640 1 -- 172.21.15.27:0/289862922 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41d8047020 con 0x7f41c8003a00 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.744+0000 7f41d27fc640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41c006b4d0 0x7f41c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.744+0000 7f41d27fc640 1 -- 172.21.15.27:0/289862922 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f41d80ab600 con 0x7f41c8003a00 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.745+0000 7f41e10d5640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41c006b4d0 0x7f41c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.745+0000 7f41e10d5640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41c006b4d0 0x7f41c006d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f41cc000c00 tx=0x7f41cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:37.749+0000 7f41d27fc640 1 -- 172.21.15.27:0/289862922 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41d804e020 con 0x7f41c8003a00 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.004+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7f41dc06c960 con 0x7f41c8003a00 2024-03-06T07:31:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.987+0000 7f41d27fc640 1 -- 172.21.15.27:0/289862922 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v478) v1 ==== 144+0+0 (secure 0 0 0) 0x7f41d807a780 con 0x7f41c8003a00 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.989+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41c006b4d0 msgr2=0x7f41c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.989+0000 7f41e3b3a640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41c006b4d0 0x7f41c006d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f41cc000c00 tx=0x7f41cc009040 comp rx=0 tx=0).stop 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 msgr2=0x7f41dc1e41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41dc1e41b0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009a90 tx=0x7f41d8004730 comp rx=0 tx=0).stop 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 shutdown_connections 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f41c006b4d0 0x7f41c006d990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f41dc1432b0 0x7f41dc143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41dc13ece0 0x7f41dc1f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 --2- 172.21.15.27:0/289862922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41c8003a00 0x7f41dc1e41b0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 >> 172.21.15.27:0/289862922 conn(0x7f41dc0b2720 msgr2=0x7f41dc072310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 shutdown_connections 2024-03-06T07:31:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:38.990+0000 7f41e3b3a640 1 -- 172.21.15.27:0/289862922 wait complete. 2024-03-06T07:31:39.006 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:31:39.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.118+0000 7fc564447640 1 Processor -- start 2024-03-06T07:31:39.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.119+0000 7fc564447640 1 -- start start 2024-03-06T07:31:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.119+0000 7fc564447640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc55c072a30 0x7fc55c1624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc564447640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc55c0d07c0 0x7fc55c15d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc564447640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c06d4d0 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc564447640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc55c06dba0 con 0x7fc55c072a30 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc564447640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc55c0b4950 con 0x7fc55c0d07c0 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc5619e2640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc55c0cf9f0 0x7fc55c0d00b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53142/0 (socket says 172.21.15.27:53142) 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc5621e3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc55c072a30 0x7fc55c1624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc5619e2640 1 -- 172.21.15.27:0/3045469668 learned_addr learned my addr 172.21.15.27:0/3045469668 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc5629e4640 1 --2- 172.21.15.27:0/3045469668 >> v2:172.21.15.69:3300/0 conn(0x7fc55c0d07c0 0x7fc55c15d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc5621e3640 1 -- 172.21.15.27:0/3045469668 >> v2:172.21.15.69:3300/0 conn(0x7fc55c0d07c0 msgr2=0x7fc55c15d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc5621e3640 1 --2- 172.21.15.27:0/3045469668 >> v2:172.21.15.69:3300/0 conn(0x7fc55c0d07c0 0x7fc55c15d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc5621e3640 1 -- 172.21.15.27:0/3045469668 >> v1:172.21.15.27:6789/0 conn(0x7fc55c0cf9f0 legacy=0x7fc55c0d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc5621e3640 1 -- 172.21.15.27:0/3045469668 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c06d7b0 con 0x7fc55c072a30 2024-03-06T07:31:39.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.120+0000 7fc5611e1640 1 -- 172.21.15.27:0/3045469668 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180773282 0 0) 0x7fc55c06d4d0 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.121+0000 7fc5621e3640 1 --2- 172.21.15.27:0/3045469668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c072a30 0x7fc55c1624c0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0710c0 tx=0x7fc55002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57f88c099dac018 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.121+0000 7fc5611e1640 1 -- 172.21.15.27:0/3045469668 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550038070 con 0x7fc55c072a30 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.121+0000 7fc5611e1640 1 -- 172.21.15.27:0/3045469668 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc550031710 con 0x7fc55c072a30 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.121+0000 7fc5611e1640 1 -- 172.21.15.27:0/3045469668 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550031ae0 con 0x7fc55c072a30 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.122+0000 7fc564447640 1 -- 172.21.15.27:0/3045469668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c072a30 msgr2=0x7fc55c1624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.122+0000 7fc564447640 1 --2- 172.21.15.27:0/3045469668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c072a30 0x7fc55c1624c0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0710c0 tx=0x7fc55002b7b0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.122+0000 7fc564447640 1 -- 172.21.15.27:0/3045469668 shutdown_connections 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.122+0000 7fc564447640 1 --2- 172.21.15.27:0/3045469668 >> v2:172.21.15.69:3300/0 conn(0x7fc55c0d07c0 0x7fc55c15d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.122+0000 7fc564447640 1 --2- 172.21.15.27:0/3045469668 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c072a30 0x7fc55c1624c0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.122+0000 7fc564447640 1 -- 172.21.15.27:0/3045469668 >> 172.21.15.27:0/3045469668 conn(0x7fc55c06af20 msgr2=0x7fc55c06b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.122+0000 7fc564447640 1 -- 172.21.15.27:0/3045469668 shutdown_connections 2024-03-06T07:31:39.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.122+0000 7fc564447640 1 -- 172.21.15.27:0/3045469668 wait complete. 2024-03-06T07:31:39.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.123+0000 7fc564447640 1 Processor -- start 2024-03-06T07:31:39.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.123+0000 7fc564447640 1 -- start start 2024-03-06T07:31:39.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc564447640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc55c072a30 0x7fc55c1560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc564447640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0cf9f0 0x7fc55c09d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc564447640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c0d07c0 0x7fc55c1f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc564447640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c072e10 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc564447640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c071130 con 0x7fc55c0d07c0 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc564447640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c072f90 con 0x7fc55c072a30 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc5619e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0cf9f0 0x7fc55c09d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc5619e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0cf9f0 0x7fc55c09d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39548/0 (socket says 172.21.15.27:39548) 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc5619e2640 1 -- 172.21.15.27:0/3654844493 learned_addr learned my addr 172.21.15.27:0/3654844493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc5619e2640 1 -- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc55c072a30 msgr2=0x7fc55c1560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc5619e2640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc55c072a30 0x7fc55c1560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc5619e2640 1 -- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c0d07c0 msgr2=0x7fc55c1f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc5619e2640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c0d07c0 0x7fc55c1f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc5619e2640 1 -- 172.21.15.27:0/3654844493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c1598b0 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc5619e2640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0cf9f0 0x7fc55c09d590 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fc55800c430 tx=0x7fc55800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc54effd640 1 -- 172.21.15.27:0/3654844493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580074e0 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc54effd640 1 -- 172.21.15.27:0/3654844493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5580047e0 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c0ae480 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc54effd640 1 -- 172.21.15.27:0/3654844493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558004bd0 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.124+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c0ae740 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.126+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c09e320 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.126+0000 7fc54effd640 1 -- 172.21.15.27:0/3654844493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc558011070 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.126+0000 7fc54effd640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc54006b4d0 0x7fc54006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.126+0000 7fc54effd640 1 -- 172.21.15.27:0/3654844493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc558089a80 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.127+0000 7fc5621e3640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc54006b4d0 0x7fc54006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.127+0000 7fc5621e3640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc54006b4d0 0x7fc54006d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc550035000 tx=0x7fc550030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.131+0000 7fc54effd640 1 -- 172.21.15.27:0/3654844493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc55c1598b0 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.312+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc55c04f410 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.315+0000 7fc54effd640 1 -- 172.21.15.27:0/3654844493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc55800e280 con 0x7fc55c0cf9f0 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc54006b4d0 msgr2=0x7fc54006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc54006b4d0 0x7fc54006d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc550035000 tx=0x7fc550030000 comp rx=0 tx=0).stop 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0cf9f0 msgr2=0x7fc55c09d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0cf9f0 0x7fc55c09d590 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fc55800c430 tx=0x7fc55800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 shutdown_connections 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc54006b4d0 0x7fc54006d990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c0d07c0 0x7fc55c1f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0cf9f0 0x7fc55c09d590 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 --2- 172.21.15.27:0/3654844493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc55c072a30 0x7fc55c1560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 >> 172.21.15.27:0/3654844493 conn(0x7fc55c06af20 msgr2=0x7fc55c0ce050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 shutdown_connections 2024-03-06T07:31:39.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.317+0000 7fc564447640 1 -- 172.21.15.27:0/3654844493 wait complete. 2024-03-06T07:31:39.330 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:39.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:39.379 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:39.425 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:39.425 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:31:39.425 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:39.441 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:39.441 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:31:39.441 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:39.458 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:39.458 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:31:39.458 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:39.496 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:39.496 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:31:39.496 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:39.515 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:39.515 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:31:39.515 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:39.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.641+0000 7f1635b47640 1 Processor -- start 2024-03-06T07:31:39.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.641+0000 7f1635b47640 1 -- start start 2024-03-06T07:31:39.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f1635b47640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163015bac0 0x7f163015cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f1635b47640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f163006d400 0x7f16300ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f1635b47640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16300729b0 con 0x7f16300aec40 2024-03-06T07:31:39.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f1635b47640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16300af0b0 con 0x7f163006d400 2024-03-06T07:31:39.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f1635b47640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163006cf70 con 0x7f163015bac0 2024-03-06T07:31:39.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f16300aec40 0x7f1630070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53158/0 (socket says 172.21.15.27:53158) 2024-03-06T07:31:39.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162ffff640 1 -- 172.21.15.27:0/4039730504 learned_addr learned my addr 172.21.15.27:0/4039730504 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:39.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162effd640 1 --2- 172.21.15.27:0/4039730504 >> v2:172.21.15.69:3300/0 conn(0x7f163006d400 0x7f16300ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162f7fe640 1 --2- 172.21.15.27:0/4039730504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163015bac0 0x7f163015cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162e7fc640 1 -- 172.21.15.27:0/4039730504 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749260202 0 0) 0x7f16300729b0 con 0x7f16300aec40 2024-03-06T07:31:39.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162e7fc640 1 -- 172.21.15.27:0/4039730504 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1618003520 con 0x7f16300aec40 2024-03-06T07:31:39.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162e7fc640 1 -- 172.21.15.27:0/4039730504 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 680934037 0 0) 0x7f1618003520 con 0x7f16300aec40 2024-03-06T07:31:39.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162effd640 1 -- 172.21.15.27:0/4039730504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163015bac0 msgr2=0x7f163015cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162effd640 1 --2- 172.21.15.27:0/4039730504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163015bac0 0x7f163015cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162effd640 1 -- 172.21.15.27:0/4039730504 >> v1:172.21.15.27:6789/0 conn(0x7f16300aec40 legacy=0x7f1630070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.642+0000 7f162effd640 1 -- 172.21.15.27:0/4039730504 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163006c920 con 0x7f163006d400 2024-03-06T07:31:39.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.643+0000 7f162effd640 1 --2- 172.21.15.27:0/4039730504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f163006d400 0x7f16300ae700 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f16240097a0 tx=0x7f162402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66138d5982ff669 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.643+0000 7f162e7fc640 1 -- 172.21.15.27:0/4039730504 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624037070 con 0x7f163006d400 2024-03-06T07:31:39.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.643+0000 7f162e7fc640 1 -- 172.21.15.27:0/4039730504 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1624030510 con 0x7f163006d400 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.643+0000 7f162e7fc640 1 -- 172.21.15.27:0/4039730504 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16280019f0 con 0x7f163006d400 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.644+0000 7f1635b47640 1 -- 172.21.15.27:0/4039730504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f163006d400 msgr2=0x7f16300ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.644+0000 7f1635b47640 1 --2- 172.21.15.27:0/4039730504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f163006d400 0x7f16300ae700 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f16240097a0 tx=0x7f162402b790 comp rx=0 tx=0).stop 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.644+0000 7f1635b47640 1 -- 172.21.15.27:0/4039730504 shutdown_connections 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.644+0000 7f1635b47640 1 --2- 172.21.15.27:0/4039730504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f163006d400 0x7f16300ae700 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.644+0000 7f1635b47640 1 --2- 172.21.15.27:0/4039730504 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163015bac0 0x7f163015cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.644+0000 7f1635b47640 1 -- 172.21.15.27:0/4039730504 >> 172.21.15.27:0/4039730504 conn(0x7f1630066d00 msgr2=0x7f16300670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.644+0000 7f1635b47640 1 -- 172.21.15.27:0/4039730504 shutdown_connections 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.644+0000 7f1635b47640 1 -- 172.21.15.27:0/4039730504 wait complete. 2024-03-06T07:31:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.645+0000 7f1635b47640 1 Processor -- start 2024-03-06T07:31:39.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f1635b47640 1 -- start start 2024-03-06T07:31:39.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f1635b47640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163006d400 0x7f1630142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f1635b47640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16300aec40 0x7f1630145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f1635b47640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f163015bac0 0x7f1630145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f1635b47640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163015bea0 con 0x7f16300aec40 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f1635b47640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16300cc490 con 0x7f163015bac0 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f1635b47640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163015c020 con 0x7f163006d400 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f162effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16300aec40 0x7f1630145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f162effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16300aec40 0x7f1630145430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39552/0 (socket says 172.21.15.27:39552) 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f162effd640 1 -- 172.21.15.27:0/1073679001 learned_addr learned my addr 172.21.15.27:0/1073679001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.646+0000 7f162f7fe640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163006d400 0x7f1630142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f162ffff640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f163015bac0 0x7f1630145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f162effd640 1 -- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163006d400 msgr2=0x7f1630142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f162effd640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163006d400 0x7f1630142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f162effd640 1 -- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f163015bac0 msgr2=0x7f1630145970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f162effd640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f163015bac0 0x7f1630145970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f162effd640 1 -- 172.21.15.27:0/1073679001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16301441f0 con 0x7f16300aec40 2024-03-06T07:31:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f162effd640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16300aec40 0x7f1630145430 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f1624000c00 tx=0x7f16240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f1617fff640 1 -- 172.21.15.27:0/1073679001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624041020 con 0x7f16300aec40 2024-03-06T07:31:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f1617fff640 1 -- 172.21.15.27:0/1073679001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162403c870 con 0x7f16300aec40 2024-03-06T07:31:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f1617fff640 1 -- 172.21.15.27:0/1073679001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624037040 con 0x7f16300aec40 2024-03-06T07:31:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16301444d0 con 0x7f16300aec40 2024-03-06T07:31:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.647+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1630144a90 con 0x7f16300aec40 2024-03-06T07:31:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.648+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1630067ff0 con 0x7f16300aec40 2024-03-06T07:31:39.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.649+0000 7f1617fff640 1 -- 172.21.15.27:0/1073679001 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1624044020 con 0x7f16300aec40 2024-03-06T07:31:39.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.649+0000 7f1617fff640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f161c06b4d0 0x7f161c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.649+0000 7f1617fff640 1 -- 172.21.15.27:0/1073679001 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f16240ab7e0 con 0x7f16300aec40 2024-03-06T07:31:39.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.649+0000 7f162f7fe640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f161c06b4d0 0x7f161c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.650+0000 7f162f7fe640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f161c06b4d0 0x7f161c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1620009a30 tx=0x7f1620008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.653+0000 7f1617fff640 1 -- 172.21.15.27:0/1073679001 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f162404b030 con 0x7f16300aec40 2024-03-06T07:31:39.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.872+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f163015c3b0 con 0x7f16300aec40 2024-03-06T07:31:39.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.873+0000 7f1617fff640 1 -- 172.21.15.27:0/1073679001 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+2120 (secure 0 0 0) 0x7f162407a9f0 con 0x7f16300aec40 2024-03-06T07:31:39.875 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:39.875 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":105,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9060,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3131328741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3131328741},{"type":"v1","addr":"172.21.15.27:6834","nonce":3131328741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":9201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2307392027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2307392027},{"type":"v1","addr":"172.21.15.27:6835","nonce":2307392027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[]} 2024-03-06T07:31:39.876 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 105 2024-03-06T07:31:39.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.874+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f161c06b4d0 msgr2=0x7f161c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.874+0000 7f1635b47640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f161c06b4d0 0x7f161c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1620009a30 tx=0x7f1620008040 comp rx=0 tx=0).stop 2024-03-06T07:31:39.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16300aec40 msgr2=0x7f1630145430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16300aec40 0x7f1630145430 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f1624000c00 tx=0x7f16240046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 shutdown_connections 2024-03-06T07:31:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f161c06b4d0 0x7f161c06d990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f163015bac0 0x7f1630145970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16300aec40 0x7f1630145430 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 --2- 172.21.15.27:0/1073679001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f163006d400 0x7f1630142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 >> 172.21.15.27:0/1073679001 conn(0x7f1630066d00 msgr2=0x7f163015de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 shutdown_connections 2024-03-06T07:31:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:39.875+0000 7f1635b47640 1 -- 172.21.15.27:0/1073679001 wait complete. 2024-03-06T07:31:39.888 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:31:39.888 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:31:39.889 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:31:39.889 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:39.889 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:31:39.889 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:31:39.889 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:31:39.889 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:39.889 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:31:39.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:31:39.890 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:31:39.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:31:39.892 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:31:39.893 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:39.893 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:31:39.893 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:39.898 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:31:39.898 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:31:39.899 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:31:39.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:31:39.940 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:31:39.940 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:39.941 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:31:39.941 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:39.941 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:31:39.978 INFO:tasks.ceph.mds.b.smithi069.stdout:starting mds.b at 2024-03-06T07:31:39.978 INFO:tasks.ceph.mds.d.smithi069.stdout:starting mds.d at 2024-03-06T07:31:40.024 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:31:40.024 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-06T07:31:40.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.051+0000 7f762f7bc640 1 Processor -- start 2024-03-06T07:31:40.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.051+0000 7f762f7bc640 1 -- start start 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762f7bc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76280b95a0 0x7f76280b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762f7bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 0x7f76280b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762f7bc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76280cea30 con 0x7f762815df00 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762f7bc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76280cf3c0 con 0x7f76280b95a0 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762cd57640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76280b95a0 0x7f76280b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762f7bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76280cf0e0 con 0x7f76280b9a10 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762cd57640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76280b95a0 0x7f76280b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39572/0 (socket says 172.21.15.27:39572) 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762cd57640 1 -- 172.21.15.27:0/1794749257 learned_addr learned my addr 172.21.15.27:0/1794749257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762dd59640 1 --2- 172.21.15.27:0/1794749257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 0x7f76280b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762cd57640 1 -- 172.21.15.27:0/1794749257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 msgr2=0x7f76280b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762cd57640 1 --2- 172.21.15.27:0/1794749257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 0x7f76280b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762cd57640 1 -- 172.21.15.27:0/1794749257 >> v1:172.21.15.69:6789/0 conn(0x7f762815df00 legacy=0x7f76280d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.052+0000 7f762cd57640 1 -- 172.21.15.27:0/1794749257 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76280ce470 con 0x7f76280b95a0 2024-03-06T07:31:40.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.053+0000 7f762cd57640 1 --2- 172.21.15.27:0/1794749257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 0x7f76280b5700 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f7618009870 tx=0x7f761802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de5650b070643af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.053+0000 7f761ffff640 1 -- 172.21.15.27:0/1794749257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618034070 con 0x7f76280b95a0 2024-03-06T07:31:40.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.053+0000 7f761ffff640 1 -- 172.21.15.27:0/1794749257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7618030840 con 0x7f76280b95a0 2024-03-06T07:31:40.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.053+0000 7f761ffff640 1 -- 172.21.15.27:0/1794749257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618030c10 con 0x7f76280b95a0 2024-03-06T07:31:40.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.054+0000 7f762f7bc640 1 -- 172.21.15.27:0/1794749257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 msgr2=0x7f76280b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.054+0000 7f762f7bc640 1 --2- 172.21.15.27:0/1794749257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 0x7f76280b5700 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f7618009870 tx=0x7f761802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:40.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.054+0000 7f762f7bc640 1 -- 172.21.15.27:0/1794749257 shutdown_connections 2024-03-06T07:31:40.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.054+0000 7f762f7bc640 1 --2- 172.21.15.27:0/1794749257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 0x7f76280b5700 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.054+0000 7f762f7bc640 1 --2- 172.21.15.27:0/1794749257 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 0x7f76280b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.054+0000 7f762f7bc640 1 -- 172.21.15.27:0/1794749257 >> 172.21.15.27:0/1794749257 conn(0x7f7628071960 msgr2=0x7f7628071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.054+0000 7f762f7bc640 1 -- 172.21.15.27:0/1794749257 shutdown_connections 2024-03-06T07:31:40.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.054+0000 7f762f7bc640 1 -- 172.21.15.27:0/1794749257 wait complete. 2024-03-06T07:31:40.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.055+0000 7f762f7bc640 1 Processor -- start 2024-03-06T07:31:40.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.056+0000 7f762f7bc640 1 -- start start 2024-03-06T07:31:40.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.056+0000 7f762f7bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 0x7f76280c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.056+0000 7f762f7bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 0x7f7628158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.056+0000 7f762f7bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f762815df00 0x7f76280ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.056+0000 7f762f7bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280cedf0 con 0x7f76280b95a0 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762f7bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280cec70 con 0x7f762815df00 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762d558640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 0x7f76280c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762f7bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76280d3220 con 0x7f76280b9a10 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762d558640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 0x7f76280c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39578/0 (socket says 172.21.15.27:39578) 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762d558640 1 -- 172.21.15.27:0/4005198980 learned_addr learned my addr 172.21.15.27:0/4005198980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762cd57640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 0x7f7628158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762d558640 1 -- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 msgr2=0x7f7628158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762d558640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 0x7f7628158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762d558640 1 -- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f762815df00 msgr2=0x7f76280ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762d558640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f762815df00 0x7f76280ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762d558640 1 -- 172.21.15.27:0/4005198980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76280cc040 con 0x7f76280b95a0 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762cd57640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 0x7f7628158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762d558640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 0x7f76280c8030 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f762400c400 tx=0x7f762400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f761dffb640 1 -- 172.21.15.27:0/4005198980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7624017070 con 0x7f76280b95a0 2024-03-06T07:31:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f761dffb640 1 -- 172.21.15.27:0/4005198980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76240052f0 con 0x7f76280b95a0 2024-03-06T07:31:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f761dffb640 1 -- 172.21.15.27:0/4005198980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7624003830 con 0x7f76280b95a0 2024-03-06T07:31:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.057+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76280cc320 con 0x7f76280b95a0 2024-03-06T07:31:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.058+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76280cc8e0 con 0x7f76280b95a0 2024-03-06T07:31:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.058+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7628067ff0 con 0x7f76280b95a0 2024-03-06T07:31:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.059+0000 7f761dffb640 1 -- 172.21.15.27:0/4005198980 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f762401a070 con 0x7f76280b95a0 2024-03-06T07:31:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.059+0000 7f761dffb640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f760c06b4d0 0x7f760c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.059+0000 7f761dffb640 1 -- 172.21.15.27:0/4005198980 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7624089190 con 0x7f76280b95a0 2024-03-06T07:31:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.059+0000 7f762cd57640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f760c06b4d0 0x7f760c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.060+0000 7f762cd57640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f760c06b4d0 0x7f760c06d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7618036000 tx=0x7f761803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.063+0000 7f761dffb640 1 -- 172.21.15.27:0/4005198980 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76240583a0 con 0x7f76280b95a0 2024-03-06T07:31:40.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:40.257+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f76280b7f40 con 0x7f76280b95a0 2024-03-06T07:31:41.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.000+0000 7f761dffb640 1 -- 172.21.15.27:0/4005198980 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v141) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7624057ca0 con 0x7f76280b95a0 2024-03-06T07:31:41.002 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-03-06T07:31:41.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f760c06b4d0 msgr2=0x7f760c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f760c06b4d0 0x7f760c06d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7618036000 tx=0x7f761803b040 comp rx=0 tx=0).stop 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 msgr2=0x7f76280c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 0x7f76280c8030 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f762400c400 tx=0x7f762400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 shutdown_connections 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f760c06b4d0 0x7f760c06d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f762815df00 0x7f76280ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f76280b9a10 0x7f7628158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 --2- 172.21.15.27:0/4005198980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76280b95a0 0x7f76280c8030 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 >> 172.21.15.27:0/4005198980 conn(0x7f7628071960 msgr2=0x7f76280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 shutdown_connections 2024-03-06T07:31:41.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.003+0000 7f762f7bc640 1 -- 172.21.15.27:0/4005198980 wait complete. 2024-03-06T07:31:41.018 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:31:41.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.129+0000 7f6fce513640 1 Processor -- start 2024-03-06T07:31:41.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.129+0000 7f6fce513640 1 -- start start 2024-03-06T07:31:41.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.129+0000 7f6fce513640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6fc815df00 0x7f6fc80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.129+0000 7f6fce513640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc80b9a30 0x7f6fc80d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.129+0000 7f6fce513640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc80cea30 con 0x7f6fc80b95a0 2024-03-06T07:31:41.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.129+0000 7f6fce513640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc80d2af0 con 0x7f6fc815df00 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.129+0000 7f6fce513640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc80cf3c0 con 0x7f6fc80b9a30 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.129+0000 7f6fc7fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6fc815df00 0x7f6fc80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.130+0000 7f6fc7fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6fc815df00 0x7f6fc80d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39582/0 (socket says 172.21.15.27:39582) 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.130+0000 7f6fc7fff640 1 -- 172.21.15.27:0/3268303629 learned_addr learned my addr 172.21.15.27:0/3268303629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.130+0000 7f6fccab0640 1 --2- 172.21.15.27:0/3268303629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc80b9a30 0x7f6fc80d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.130+0000 7f6fc7fff640 1 -- 172.21.15.27:0/3268303629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc80b9a30 msgr2=0x7f6fc80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.130+0000 7f6fc7fff640 1 --2- 172.21.15.27:0/3268303629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc80b9a30 0x7f6fc80d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.130+0000 7f6fc7fff640 1 -- 172.21.15.27:0/3268303629 >> v1:172.21.15.69:6789/0 conn(0x7f6fc80b95a0 legacy=0x7f6fc80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.130+0000 7f6fc7fff640 1 -- 172.21.15.27:0/3268303629 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc80cf0e0 con 0x7f6fc815df00 2024-03-06T07:31:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.130+0000 7f6fccab0640 1 --2- 172.21.15.27:0/3268303629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc80b9a30 0x7f6fc80d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fc7fff640 1 --2- 172.21.15.27:0/3268303629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc815df00 0x7f6fc80d25b0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009870 tx=0x7f6fb802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d1f48ddcd71f39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fc6ffd640 1 -- 172.21.15.27:0/3268303629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8034070 con 0x7f6fc815df00 2024-03-06T07:31:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fc6ffd640 1 -- 172.21.15.27:0/3268303629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb8030840 con 0x7f6fc815df00 2024-03-06T07:31:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fc6ffd640 1 -- 172.21.15.27:0/3268303629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8030c10 con 0x7f6fc815df00 2024-03-06T07:31:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fce513640 1 -- 172.21.15.27:0/3268303629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc815df00 msgr2=0x7f6fc80d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fce513640 1 --2- 172.21.15.27:0/3268303629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc815df00 0x7f6fc80d25b0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009870 tx=0x7f6fb802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fce513640 1 -- 172.21.15.27:0/3268303629 shutdown_connections 2024-03-06T07:31:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fce513640 1 --2- 172.21.15.27:0/3268303629 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc80b9a30 0x7f6fc80d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fce513640 1 --2- 172.21.15.27:0/3268303629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc815df00 0x7f6fc80d25b0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fce513640 1 -- 172.21.15.27:0/3268303629 >> 172.21.15.27:0/3268303629 conn(0x7f6fc8071960 msgr2=0x7f6fc8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fce513640 1 -- 172.21.15.27:0/3268303629 shutdown_connections 2024-03-06T07:31:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.131+0000 7f6fce513640 1 -- 172.21.15.27:0/3268303629 wait complete. 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.133+0000 7f6fce513640 1 Processor -- start 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.133+0000 7f6fce513640 1 -- start start 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.133+0000 7f6fce513640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc80b95a0 0x7f6fc81f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.133+0000 7f6fce513640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc815df00 0x7f6fc81f0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.133+0000 7f6fce513640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc80d3000 0x7f6fc80d3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.133+0000 7f6fce513640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc80cec70 con 0x7f6fc80d3000 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.133+0000 7f6fce513640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc80ce750 con 0x7f6fc80b95a0 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.133+0000 7f6fce513640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc80cedf0 con 0x7f6fc815df00 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fccab0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc80d3000 0x7f6fc80d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fccab0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc80d3000 0x7f6fc80d3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39598/0 (socket says 172.21.15.27:39598) 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fccab0640 1 -- 172.21.15.27:0/2909680099 learned_addr learned my addr 172.21.15.27:0/2909680099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fc7fff640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc80b95a0 0x7f6fc81f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fccab0640 1 -- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc815df00 msgr2=0x7f6fc81f0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fccab0640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc815df00 0x7f6fc81f0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fccab0640 1 -- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc80b95a0 msgr2=0x7f6fc81f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fccab0640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc80b95a0 0x7f6fc81f4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fccab0640 1 -- 172.21.15.27:0/2909680099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc81f7d10 con 0x7f6fc80d3000 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fccab0640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc80d3000 0x7f6fc80d3490 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009fd0 tx=0x7f6fc000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fc4ff9640 1 -- 172.21.15.27:0/2909680099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc001a070 con 0x7f6fc80d3000 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fc4ff9640 1 -- 172.21.15.27:0/2909680099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc0003bb0 con 0x7f6fc80d3000 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc815f9b0 con 0x7f6fc80d3000 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fc4ff9640 1 -- 172.21.15.27:0/2909680099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0004030 con 0x7f6fc80d3000 2024-03-06T07:31:41.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.134+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc81f12f0 con 0x7f6fc80d3000 2024-03-06T07:31:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.135+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc815f7a0 con 0x7f6fc80d3000 2024-03-06T07:31:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.136+0000 7f6fc4ff9640 1 -- 172.21.15.27:0/2909680099 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fc001d070 con 0x7f6fc80d3000 2024-03-06T07:31:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.136+0000 7f6fc4ff9640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fa806b4d0 0x7f6fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.137+0000 7f6fc4ff9640 1 -- 172.21.15.27:0/2909680099 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6fc008a160 con 0x7f6fc80d3000 2024-03-06T07:31:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.137+0000 7f6fc7fff640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fa806b4d0 0x7f6fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.137+0000 7f6fc7fff640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fa806b4d0 0x7f6fa806d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8036000 tx=0x7f6fb803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.141+0000 7f6fc4ff9640 1 -- 172.21.15.27:0/2909680099 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fc0010070 con 0x7f6fc80d3000 2024-03-06T07:31:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.394+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6fc8067ff0 con 0x7f6fc80d3000 2024-03-06T07:31:41.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.394+0000 7f6fc4ff9640 1 -- 172.21.15.27:0/2909680099 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6fc00593c0 con 0x7f6fc80d3000 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlzA/HMBAAoLUXO7QApFa2yIHe07zR7g==", 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehldPyWAhAA8BlZ71uwnReX1JoA7uKiyQ==", 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-03-06T07:31:41.398 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlz3/sOBAA88VmYATyhlf7qsVyYxiaMg==", 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlL5/3CRAAhCOeraRvPbN0Hf9lKYtJIQ==", 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:31:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRGehl4rb9LRAAXEMD5e+slkPTdhdXNJrFdw==", 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-03-06T07:31:41.400 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCUGehl5PI1LRAA7/q2Tr0cx/JKlB3UiuR4wA==", 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXGehlOGQILBAApO8k5wBbmn+yHgSPpC1hrA==", 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:31:41.401 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaGehl5KMxKxAArqxg+90PZmMpm3iNto0lCg==", 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgGehl+NncMRAA66lI1IdCOGN4feBZV8gm2A==", 2024-03-06T07:31:41.402 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCjGehlloJqMBAAlr3KdXwkT1cKYkzL9AE0nw==", 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmGehlBPWMLhAAvB2FAbsYZfbD9urHVd4XxA==", 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCpGehlZ7GQLBAA5EaT77JjtfSj6MpAqvOQ2A==", 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.404 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA==", 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlygPzFRAA1CVNJ9TWUhBxtrEYf7TcQQ==", 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlZ5xaEhAAKZJlq/z19EfhJWPhw946VQ==", 2024-03-06T07:31:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOGehlDPrLGxAAGMDzmgGJVjFPcuoI9tpwPg==", 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-03-06T07:31:41.407 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlkc2iEhAA6BuYba0FzXeVjGPOP8sgyQ==", 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehlydHsLBAACuGy++3DAD/qKMDKYIEkUw==", 2024-03-06T07:31:41.408 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNGehl4uDmKBAAvPodjm5FqL1dKrvGlWLxSg==", 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-03-06T07:31:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-03-06T07:31:41.410 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:31:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fa806b4d0 msgr2=0x7f6fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fa806b4d0 0x7f6fa806d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8036000 tx=0x7f6fb803b040 comp rx=0 tx=0).stop 2024-03-06T07:31:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc80d3000 msgr2=0x7f6fc80d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc80d3000 0x7f6fc80d3490 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009fd0 tx=0x7f6fc000d560 comp rx=0 tx=0).stop 2024-03-06T07:31:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 shutdown_connections 2024-03-06T07:31:41.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6fa806b4d0 0x7f6fa806d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc80d3000 0x7f6fc80d3490 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6fc815df00 0x7f6fc81f0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 --2- 172.21.15.27:0/2909680099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fc80b95a0 0x7f6fc81f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 >> 172.21.15.27:0/2909680099 conn(0x7f6fc8071960 msgr2=0x7f6fc80b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 shutdown_connections 2024-03-06T07:31:41.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.396+0000 7f6fce513640 1 -- 172.21.15.27:0/2909680099 wait complete. 2024-03-06T07:31:41.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.517+0000 7f25067f7640 1 Processor -- start 2024-03-06T07:31:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.517+0000 7f25067f7640 1 -- start start 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.518+0000 7f25067f7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2500072660 0x7f250015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.518+0000 7f25067f7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25000b4160 con 0x7f250006e700 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.518+0000 7f25067f7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25000b5b40 con 0x7f25000b55d0 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.518+0000 7f25067f7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25000b3ba0 con 0x7f2500072660 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.518+0000 7f24fffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f250006e700 0x7f250006d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53186/0 (socket says 172.21.15.27:53186) 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.518+0000 7f24fffff640 1 -- 172.21.15.27:0/600760008 learned_addr learned my addr 172.21.15.27:0/600760008 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.518+0000 7f24ff7fe640 1 --2- 172.21.15.27:0/600760008 >> v2:172.21.15.69:3300/0 conn(0x7f2500072660 0x7f250015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.518+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150379987 0 0) 0x7f25000b4160 con 0x7f250006e700 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.518+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e8003520 con 0x7f250006e700 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1450271135 0 0) 0x7f24e8003520 con 0x7f250006e700 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 >> v2:172.21.15.69:3300/0 conn(0x7f2500072660 msgr2=0x7f250015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 --2- 172.21.15.27:0/600760008 >> v2:172.21.15.69:3300/0 conn(0x7f2500072660 0x7f250015ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 >> v1:172.21.15.69:6789/0 conn(0x7f25000b55d0 legacy=0x7f2500071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25000b3e80 con 0x7f250006e700 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f24f4001670 con 0x7f250006e700 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 >> v1:172.21.15.27:6789/0 conn(0x7f250006e700 legacy=0x7f250006d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24e8004800 0x7f24e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80051c0 0x7f24e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25000b3ba0 con 0x7f24e80039b0 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e800c060 con 0x7f24e8004800 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e800c260 con 0x7f24e80051c0 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24fffff640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24ff7fe640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24e8004800 0x7f24e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f2504d94640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80051c0 0x7f24e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24fffff640 1 -- 172.21.15.27:0/600760008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80051c0 msgr2=0x7f24e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24fffff640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80051c0 0x7f24e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24fffff640 1 -- 172.21.15.27:0/600760008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24e8004800 msgr2=0x7f24e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24fffff640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24e8004800 0x7f24e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f24fffff640 1 -- 172.21.15.27:0/600760008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e8003520 con 0x7f24e80039b0 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f2504d94640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80051c0 0x7f24e800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.519+0000 7f2504d94640 1 -- 172.21.15.27:0/600760008 reap_dead start 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.520+0000 7f24fffff640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f24f400c240 tx=0x7f24e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb3e6a85b0cdc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.520+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4007d10 con 0x7f24e80039b0 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.520+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f4030070 con 0x7f24e80039b0 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.520+0000 7f24feffd640 1 -- 172.21.15.27:0/600760008 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f402cdc0 con 0x7f24e80039b0 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.521+0000 7f25067f7640 1 -- 172.21.15.27:0/600760008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80039b0 msgr2=0x7f24e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.521+0000 7f25067f7640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f24f400c240 tx=0x7f24e8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.521+0000 7f25067f7640 1 -- 172.21.15.27:0/600760008 shutdown_connections 2024-03-06T07:31:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.521+0000 7f25067f7640 1 --2- 172.21.15.27:0/600760008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.521+0000 7f25067f7640 1 -- 172.21.15.27:0/600760008 >> 172.21.15.27:0/600760008 conn(0x7f25000b2720 msgr2=0x7f25000b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.521+0000 7f25067f7640 1 -- 172.21.15.27:0/600760008 shutdown_connections 2024-03-06T07:31:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.521+0000 7f25067f7640 1 -- 172.21.15.27:0/600760008 wait complete. 2024-03-06T07:31:41.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.522+0000 7f25067f7640 1 Processor -- start 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.522+0000 7f25067f7640 1 -- start start 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f25067f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80039b0 0x7f25001f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f25067f7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25000b1300 0x7f25000b1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f25067f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2500146f40 0x7f25001473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f25067f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250015f100 con 0x7f2500146f40 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f25067f7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250015ef80 con 0x7f25000b1300 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f25067f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250015f280 con 0x7f24e80039b0 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f2504d94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2500146f40 0x7f25001473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f2504d94640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2500146f40 0x7f25001473a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39624/0 (socket says 172.21.15.27:39624) 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f2504d94640 1 -- 172.21.15.27:0/1898157790 learned_addr learned my addr 172.21.15.27:0/1898157790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f24ff7fe640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25000b1300 0x7f25000b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f24fffff640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80039b0 0x7f25001f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f2504d94640 1 -- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80039b0 msgr2=0x7f25001f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f2504d94640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80039b0 0x7f25001f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f2504d94640 1 -- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25000b1300 msgr2=0x7f25000b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f2504d94640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25000b1300 0x7f25000b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f2504d94640 1 -- 172.21.15.27:0/1898157790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25000ae3f0 con 0x7f2500146f40 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f24ff7fe640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25000b1300 0x7f25000b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f24fffff640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80039b0 0x7f25001f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.523+0000 7f2504d94640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2500146f40 0x7f25001473a0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f25000b37e0 tx=0x7f24f800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.524+0000 7f24fcff9640 1 -- 172.21.15.27:0/1898157790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8003970 con 0x7f2500146f40 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.524+0000 7f24fcff9640 1 -- 172.21.15.27:0/1898157790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f8018070 con 0x7f2500146f40 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.524+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25000ae690 con 0x7f2500146f40 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.524+0000 7f24fcff9640 1 -- 172.21.15.27:0/1898157790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8003d50 con 0x7f2500146f40 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.524+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2500158d10 con 0x7f2500146f40 2024-03-06T07:31:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.525+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2500067ff0 con 0x7f2500146f40 2024-03-06T07:31:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.525+0000 7f24fcff9640 1 -- 172.21.15.27:0/1898157790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24f80050d0 con 0x7f2500146f40 2024-03-06T07:31:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.525+0000 7f24fcff9640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f24ec06b4d0 0x7f24ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.525+0000 7f24fcff9640 1 -- 172.21.15.27:0/1898157790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f24f80897c0 con 0x7f2500146f40 2024-03-06T07:31:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.526+0000 7f24fffff640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f24ec06b4d0 0x7f24ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.526+0000 7f24fffff640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f24ec06b4d0 0x7f24ec06d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f24f4034000 tx=0x7f24f4039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.530+0000 7f24fcff9640 1 -- 172.21.15.27:0/1898157790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24f8022020 con 0x7f2500146f40 2024-03-06T07:31:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.710+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2500072c30 con 0x7f2500146f40 2024-03-06T07:31:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.710+0000 7f24fcff9640 1 -- 172.21.15.27:0/1898157790 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+12829 (secure 0 0 0) 0x7f24f8058a20 con 0x7f2500146f40 2024-03-06T07:31:41.713 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:41.713 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":141,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:31:40.999092+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:41.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f24ec06b4d0 msgr2=0x7f24ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f24ec06b4d0 0x7f24ec06d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f24f4034000 tx=0x7f24f4039040 comp rx=0 tx=0).stop 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2500146f40 msgr2=0x7f25001473a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2500146f40 0x7f25001473a0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f25000b37e0 tx=0x7f24f800eb90 comp rx=0 tx=0).stop 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 shutdown_connections 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f24ec06b4d0 0x7f24ec06d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2500146f40 0x7f25001473a0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25000b1300 0x7f25000b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 --2- 172.21.15.27:0/1898157790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24e80039b0 0x7f25001f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.712+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 >> 172.21.15.27:0/1898157790 conn(0x7f25000b2720 msgr2=0x7f250006d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.713+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 shutdown_connections 2024-03-06T07:31:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.713+0000 7f25067f7640 1 -- 172.21.15.27:0/1898157790 wait complete. 2024-03-06T07:31:41.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:41.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:41.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.834+0000 7fdc251ac640 1 Processor -- start 2024-03-06T07:31:41.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.834+0000 7fdc251ac640 1 -- start start 2024-03-06T07:31:41.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc251ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc20072a30 0x7fdc201624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc251ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc200d07c0 0x7fdc2015d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc251ac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2006d4d0 con 0x7fdc200cf9f0 2024-03-06T07:31:41.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc251ac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2006dba0 con 0x7fdc200d07c0 2024-03-06T07:31:41.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc251ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc200b4950 con 0x7fdc20072a30 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc1f59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc200d07c0 0x7fdc2015d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc1f59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc200d07c0 0x7fdc2015d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39634/0 (socket says 172.21.15.27:39634) 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc17fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdc200cf9f0 0x7fdc200d00b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53194/0 (socket says 172.21.15.27:53194) 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc1f59e640 1 -- 172.21.15.27:0/8546097 learned_addr learned my addr 172.21.15.27:0/8546097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc1f59e640 1 -- 172.21.15.27:0/8546097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc20072a30 msgr2=0x7fdc201624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc1ed9d640 1 --2- 172.21.15.27:0/8546097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc20072a30 0x7fdc201624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc1f59e640 1 --2- 172.21.15.27:0/8546097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc20072a30 0x7fdc201624c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc1f59e640 1 -- 172.21.15.27:0/8546097 >> v1:172.21.15.27:6789/0 conn(0x7fdc200cf9f0 legacy=0x7fdc200d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc1f59e640 1 -- 172.21.15.27:0/8546097 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2006d7b0 con 0x7fdc200d07c0 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.835+0000 7fdc1e59c640 1 -- 172.21.15.27:0/8546097 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359511954 0 0) 0x7fdc2006d4d0 con 0x7fdc200cf9f0 2024-03-06T07:31:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.836+0000 7fdc1f59e640 1 --2- 172.21.15.27:0/8546097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc200d07c0 0x7fdc2015d660 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009e50 tx=0x7fdc1802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef64617b910556a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.836+0000 7fdc1e59c640 1 -- 172.21.15.27:0/8546097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1802c9e0 con 0x7fdc200d07c0 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.836+0000 7fdc1e59c640 1 -- 172.21.15.27:0/8546097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1802cb80 con 0x7fdc200d07c0 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.836+0000 7fdc1e59c640 1 -- 172.21.15.27:0/8546097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc180343f0 con 0x7fdc200d07c0 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.837+0000 7fdc251ac640 1 -- 172.21.15.27:0/8546097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc200d07c0 msgr2=0x7fdc2015d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.837+0000 7fdc251ac640 1 --2- 172.21.15.27:0/8546097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc200d07c0 0x7fdc2015d660 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009e50 tx=0x7fdc1802be40 comp rx=0 tx=0).stop 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.837+0000 7fdc251ac640 1 -- 172.21.15.27:0/8546097 shutdown_connections 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.837+0000 7fdc251ac640 1 --2- 172.21.15.27:0/8546097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc200d07c0 0x7fdc2015d660 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.837+0000 7fdc251ac640 1 --2- 172.21.15.27:0/8546097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc20072a30 0x7fdc201624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.837+0000 7fdc251ac640 1 -- 172.21.15.27:0/8546097 >> 172.21.15.27:0/8546097 conn(0x7fdc2006af20 msgr2=0x7fdc2006b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.837+0000 7fdc251ac640 1 -- 172.21.15.27:0/8546097 shutdown_connections 2024-03-06T07:31:41.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.837+0000 7fdc251ac640 1 -- 172.21.15.27:0/8546097 wait complete. 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.838+0000 7fdc251ac640 1 Processor -- start 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.838+0000 7fdc251ac640 1 -- start start 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc251ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc20072a30 0x7fdc201594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc251ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc200cf9f0 0x7fdc2014edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc251ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc200d07c0 0x7fdc20160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc251ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc20072e10 con 0x7fdc20072a30 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc251ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc20071130 con 0x7fdc200cf9f0 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc251ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc20072f90 con 0x7fdc200d07c0 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc20072a30 0x7fdc201594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc20072a30 0x7fdc201594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39644/0 (socket says 172.21.15.27:39644) 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1ed9d640 1 -- 172.21.15.27:0/1190405644 learned_addr learned my addr 172.21.15.27:0/1190405644 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1f59e640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc200d07c0 0x7fdc20160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc17fff640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc200cf9f0 0x7fdc2014edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1ed9d640 1 -- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc200d07c0 msgr2=0x7fdc20160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1ed9d640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc200d07c0 0x7fdc20160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1ed9d640 1 -- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc200cf9f0 msgr2=0x7fdc2014edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1ed9d640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc200cf9f0 0x7fdc2014edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1ed9d640 1 -- 172.21.15.27:0/1190405644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2015da80 con 0x7fdc20072a30 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc1ed9d640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc20072a30 0x7fdc201594f0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000c820 tx=0x7fdc1000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc177fe640 1 -- 172.21.15.27:0/1190405644 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10015070 con 0x7fdc20072a30 2024-03-06T07:31:41.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.839+0000 7fdc177fe640 1 -- 172.21.15.27:0/1190405644 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc10004020 con 0x7fdc20072a30 2024-03-06T07:31:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.840+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2009d5f0 con 0x7fdc20072a30 2024-03-06T07:31:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.840+0000 7fdc177fe640 1 -- 172.21.15.27:0/1190405644 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc100043f0 con 0x7fdc20072a30 2024-03-06T07:31:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.840+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2009dbb0 con 0x7fdc20072a30 2024-03-06T07:31:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.841+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2009e320 con 0x7fdc20072a30 2024-03-06T07:31:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.841+0000 7fdc177fe640 1 -- 172.21.15.27:0/1190405644 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc10007830 con 0x7fdc20072a30 2024-03-06T07:31:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.842+0000 7fdc177fe640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b500 0x7fdc0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.842+0000 7fdc177fe640 1 -- 172.21.15.27:0/1190405644 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdc10089d00 con 0x7fdc20072a30 2024-03-06T07:31:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.842+0000 7fdc17fff640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b500 0x7fdc0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.842+0000 7fdc17fff640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b500 0x7fdc0006d9c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fdc200b1f20 tx=0x7fdc08009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:41.846+0000 7fdc177fe640 1 -- 172.21.15.27:0/1190405644 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc10058e40 con 0x7fdc20072a30 2024-03-06T07:31:42.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.027+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdc2004f410 con 0x7fdc20072a30 2024-03-06T07:31:42.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.028+0000 7fdc177fe640 1 -- 172.21.15.27:0/1190405644 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdc10058740 con 0x7fdc20072a30 2024-03-06T07:31:42.030 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b500 msgr2=0x7fdc0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b500 0x7fdc0006d9c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fdc200b1f20 tx=0x7fdc08009040 comp rx=0 tx=0).stop 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc20072a30 msgr2=0x7fdc201594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc20072a30 0x7fdc201594f0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000c820 tx=0x7fdc1000cce0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 shutdown_connections 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b500 0x7fdc0006d9c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc200d07c0 0x7fdc20160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc200cf9f0 0x7fdc2014edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 --2- 172.21.15.27:0/1190405644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc20072a30 0x7fdc201594f0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 >> 172.21.15.27:0/1190405644 conn(0x7fdc2006af20 msgr2=0x7fdc200acfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 shutdown_connections 2024-03-06T07:31:42.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.030+0000 7fdc251ac640 1 -- 172.21.15.27:0/1190405644 wait complete. 2024-03-06T07:31:42.044 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:31:42.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.149+0000 7efe285d6640 1 Processor -- start 2024-03-06T07:31:42.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.149+0000 7efe285d6640 1 -- start start 2024-03-06T07:31:42.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe285d6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efe2006fe20 0x7efe200cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe285d6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200d0dd0 con 0x7efe2015e060 2024-03-06T07:31:42.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe285d6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200d10b0 con 0x7efe200d0880 2024-03-06T07:31:42.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe285d6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe200cc0e0 con 0x7efe2006fe20 2024-03-06T07:31:42.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe26372640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efe2015e060 0x7efe200ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53206/0 (socket says 172.21.15.27:53206) 2024-03-06T07:31:42.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe26372640 1 -- 172.21.15.27:0/3583491111 learned_addr learned my addr 172.21.15.27:0/3583491111 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:42.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071051617 0 0) 0x7efe200d0dd0 con 0x7efe2015e060 2024-03-06T07:31:42.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25b71640 1 --2- 172.21.15.27:0/3583491111 >> v2:172.21.15.69:3300/0 conn(0x7efe2006fe20 0x7efe200cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c003520 con 0x7efe2015e060 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1736358919 0 0) 0x7efe0c003520 con 0x7efe2015e060 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 >> v2:172.21.15.69:3300/0 conn(0x7efe2006fe20 msgr2=0x7efe200cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 --2- 172.21.15.27:0/3583491111 >> v2:172.21.15.69:3300/0 conn(0x7efe2006fe20 0x7efe200cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 >> v1:172.21.15.69:6789/0 conn(0x7efe200d0880 legacy=0x7efe200ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe200cc3c0 con 0x7efe2015e060 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7efe1c001670 con 0x7efe2015e060 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 >> v1:172.21.15.27:6789/0 conn(0x7efe2015e060 legacy=0x7efe200ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c0039b0 0x7efe0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c0051c0 0x7efe0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe200cc0e0 con 0x7efe0c004800 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c00c110 con 0x7efe0c0051c0 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c00c310 con 0x7efe0c0039b0 2024-03-06T07:31:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25b71640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe26372640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c0039b0 0x7efe0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe26b73640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c0051c0 0x7efe0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25b71640 1 -- 172.21.15.27:0/3583491111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c0039b0 msgr2=0x7efe0c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25b71640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe0c0039b0 0x7efe0c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25b71640 1 -- 172.21.15.27:0/3583491111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c0051c0 msgr2=0x7efe0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25b71640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c0051c0 0x7efe0c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe25b71640 1 -- 172.21.15.27:0/3583491111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c003520 con 0x7efe0c004800 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe26b73640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c0051c0 0x7efe0c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.151+0000 7efe26b73640 1 -- 172.21.15.27:0/3583491111 reap_dead start 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.152+0000 7efe25b71640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe0c004c80 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7efe1000ba00 tx=0x7efe0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86b0c29962e9172 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.152+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe10036020 con 0x7efe0c004800 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.152+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe10032620 con 0x7efe0c004800 2024-03-06T07:31:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.152+0000 7efe25370640 1 -- 172.21.15.27:0/3583491111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1000b600 con 0x7efe0c004800 2024-03-06T07:31:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.152+0000 7efe285d6640 1 -- 172.21.15.27:0/3583491111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 msgr2=0x7efe0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.152+0000 7efe285d6640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe0c004c80 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7efe1000ba00 tx=0x7efe0c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.152+0000 7efe285d6640 1 -- 172.21.15.27:0/3583491111 shutdown_connections 2024-03-06T07:31:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.152+0000 7efe285d6640 1 --2- 172.21.15.27:0/3583491111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe0c004c80 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.152+0000 7efe285d6640 1 -- 172.21.15.27:0/3583491111 >> 172.21.15.27:0/3583491111 conn(0x7efe200d1a90 msgr2=0x7efe200d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.153+0000 7efe285d6640 1 -- 172.21.15.27:0/3583491111 shutdown_connections 2024-03-06T07:31:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.153+0000 7efe285d6640 1 -- 172.21.15.27:0/3583491111 wait complete. 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe285d6640 1 Processor -- start 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe285d6640 1 -- start start 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe285d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe200c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe285d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe200cb620 0x7efe2014dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe285d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe200c9d30 0x7efe2006f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe285d6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe200ce310 con 0x7efe0c004800 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe285d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2015e440 con 0x7efe200cb620 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe285d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe200ce490 con 0x7efe200c9d30 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe26372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe200c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe26372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe200c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39660/0 (socket says 172.21.15.27:39660) 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.154+0000 7efe26372640 1 -- 172.21.15.27:0/2621981605 learned_addr learned my addr 172.21.15.27:0/2621981605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe26372640 1 -- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe200c9d30 msgr2=0x7efe2006f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe25b71640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe200cb620 0x7efe2014dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe26b73640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe200c9d30 0x7efe2006f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe26372640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe200c9d30 0x7efe2006f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe26372640 1 -- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe200cb620 msgr2=0x7efe2014dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe26372640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe200cb620 0x7efe2014dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe26372640 1 -- 172.21.15.27:0/2621981605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe200ca580 con 0x7efe0c004800 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe25b71640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe200cb620 0x7efe2014dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe26b73640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe200c9d30 0x7efe2006f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe26372640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe200c7c70 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0072c0 tx=0x7efe1c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe16ffd640 1 -- 172.21.15.27:0/2621981605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c01a070 con 0x7efe0c004800 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe16ffd640 1 -- 172.21.15.27:0/2621981605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c00fa20 con 0x7efe0c004800 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe16ffd640 1 -- 172.21.15.27:0/2621981605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c00fe10 con 0x7efe0c004800 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe200ca860 con 0x7efe0c004800 2024-03-06T07:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.155+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe200cae20 con 0x7efe0c004800 2024-03-06T07:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.156+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe20067ff0 con 0x7efe0c004800 2024-03-06T07:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.157+0000 7efe16ffd640 1 -- 172.21.15.27:0/2621981605 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe1c01d360 con 0x7efe0c004800 2024-03-06T07:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.157+0000 7efe16ffd640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe0406b4d0 0x7efe0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.157+0000 7efe16ffd640 1 -- 172.21.15.27:0/2621981605 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efe1c0898d0 con 0x7efe0c004800 2024-03-06T07:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.157+0000 7efe25b71640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe0406b4d0 0x7efe0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.158+0000 7efe25b71640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe0406b4d0 0x7efe0406d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7efe20070270 tx=0x7efe10030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.162+0000 7efe16ffd640 1 -- 172.21.15.27:0/2621981605 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe1c058aa0 con 0x7efe0c004800 2024-03-06T07:31:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.344+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efe2015d6a0 con 0x7efe0c004800 2024-03-06T07:31:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.345+0000 7efe16ffd640 1 -- 172.21.15.27:0/2621981605 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v141) v1 ==== 84+0+65 (secure 0 0 0) 0x7efe1c0583a0 con 0x7efe0c004800 2024-03-06T07:31:42.347 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:42.347 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-06T07:31:42.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-06T07:31:42.347 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-06T07:31:42.347 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-06T07:31:42.348 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:42.348 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-06T07:31:42.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.346+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe0406b4d0 msgr2=0x7efe0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.346+0000 7efe285d6640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe0406b4d0 0x7efe0406d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7efe20070270 tx=0x7efe10030000 comp rx=0 tx=0).stop 2024-03-06T07:31:42.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.346+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 msgr2=0x7efe200c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.346+0000 7efe285d6640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe200c7c70 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0072c0 tx=0x7efe1c007780 comp rx=0 tx=0).stop 2024-03-06T07:31:42.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.346+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 shutdown_connections 2024-03-06T07:31:42.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.346+0000 7efe285d6640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7efe0406b4d0 0x7efe0406d990 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.346+0000 7efe285d6640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe200c9d30 0x7efe2006f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.346+0000 7efe285d6640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe200cb620 0x7efe2014dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.346+0000 7efe285d6640 1 --2- 172.21.15.27:0/2621981605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe0c004800 0x7efe200c7c70 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.347+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 >> 172.21.15.27:0/2621981605 conn(0x7efe200d1a90 msgr2=0x7efe2006c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.347+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 shutdown_connections 2024-03-06T07:31:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.347+0000 7efe285d6640 1 -- 172.21.15.27:0/2621981605 wait complete. 2024-03-06T07:31:42.360 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:31:42.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:31:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.468+0000 7f1bd5d19640 1 Processor -- start 2024-03-06T07:31:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.468+0000 7f1bd5d19640 1 -- start start 2024-03-06T07:31:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.469+0000 7f1bd5d19640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1bd00cead0 0x7f1bd0168e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.469+0000 7f1bd5d19640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd015e6d0 con 0x7f1bd00b41f0 2024-03-06T07:31:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.469+0000 7f1bd5d19640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd00ccd00 con 0x7f1bd015e2f0 2024-03-06T07:31:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.469+0000 7f1bd5d19640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd00d0a10 con 0x7f1bd00cead0 2024-03-06T07:31:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.469+0000 7f1bcffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1bd00cead0 0x7f1bd0168e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.469+0000 7f1bceffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1bd00b41f0 0x7f1bd006bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53216/0 (socket says 172.21.15.27:53216) 2024-03-06T07:31:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.469+0000 7f1bceffd640 1 -- 172.21.15.27:0/379008019 learned_addr learned my addr 172.21.15.27:0/379008019 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.470+0000 7f1bcffff640 1 -- 172.21.15.27:0/379008019 >> v1:172.21.15.69:6789/0 conn(0x7f1bd015e2f0 legacy=0x7f1bd006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.470+0000 7f1bcffff640 1 -- 172.21.15.27:0/379008019 >> v1:172.21.15.27:6789/0 conn(0x7f1bd00b41f0 legacy=0x7f1bd006bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.470+0000 7f1bce7fc640 1 -- 172.21.15.27:0/379008019 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019842573 0 0) 0x7f1bd015e6d0 con 0x7f1bd00b41f0 2024-03-06T07:31:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.470+0000 7f1bcffff640 1 -- 172.21.15.27:0/379008019 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd00ccfe0 con 0x7f1bd00cead0 2024-03-06T07:31:42.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.470+0000 7f1bcffff640 1 --2- 172.21.15.27:0/379008019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00cead0 0x7f1bd0168e30 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009d90 tx=0x7f1bc802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca66f46cb0d98cf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bce7fc640 1 -- 172.21.15.27:0/379008019 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8039070 con 0x7f1bd00cead0 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bce7fc640 1 -- 172.21.15.27:0/379008019 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc802cc60 con 0x7f1bd00cead0 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bce7fc640 1 -- 172.21.15.27:0/379008019 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8034660 con 0x7f1bd00cead0 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bd5d19640 1 -- 172.21.15.27:0/379008019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00cead0 msgr2=0x7f1bd0168e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bd5d19640 1 --2- 172.21.15.27:0/379008019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00cead0 0x7f1bd0168e30 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009d90 tx=0x7f1bc802beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bd5d19640 1 -- 172.21.15.27:0/379008019 shutdown_connections 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bd5d19640 1 --2- 172.21.15.27:0/379008019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00cead0 0x7f1bd0168e30 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bd5d19640 1 -- 172.21.15.27:0/379008019 >> 172.21.15.27:0/379008019 conn(0x7f1bd00b36d0 msgr2=0x7f1bd00b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bd5d19640 1 -- 172.21.15.27:0/379008019 shutdown_connections 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.471+0000 7f1bd5d19640 1 -- 172.21.15.27:0/379008019 wait complete. 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.472+0000 7f1bd5d19640 1 Processor -- start 2024-03-06T07:31:42.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.472+0000 7f1bd5d19640 1 -- start start 2024-03-06T07:31:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.473+0000 7f1bd5d19640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00b41f0 0x7f1bd014c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.473+0000 7f1bd5d19640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd00cead0 0x7f1bd0157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.473+0000 7f1bd5d19640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bd015e2f0 0x7f1bd006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.473+0000 7f1bd5d19640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd006c300 con 0x7f1bd00b41f0 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.473+0000 7f1bd5d19640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd00cd5a0 con 0x7f1bd00cead0 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.473+0000 7f1bd5d19640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0169370 con 0x7f1bd015e2f0 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.473+0000 7f1bcf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00b41f0 0x7f1bd014c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.473+0000 7f1bcf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00b41f0 0x7f1bd014c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39682/0 (socket says 172.21.15.27:39682) 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.473+0000 7f1bcf7fe640 1 -- 172.21.15.27:0/3865010226 learned_addr learned my addr 172.21.15.27:0/3865010226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bceffd640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd00cead0 0x7f1bd0157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bcf7fe640 1 -- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bd015e2f0 msgr2=0x7f1bd006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bcffff640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bd015e2f0 0x7f1bd006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bcf7fe640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bd015e2f0 0x7f1bd006b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bcf7fe640 1 -- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd00cead0 msgr2=0x7f1bd0157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bcf7fe640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd00cead0 0x7f1bd0157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bcf7fe640 1 -- 172.21.15.27:0/3865010226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd014d1a0 con 0x7f1bd00b41f0 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bceffd640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd00cead0 0x7f1bd0157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bcf7fe640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00b41f0 0x7f1bd014c7b0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000c430 tx=0x7f1bc000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bb7fff640 1 -- 172.21.15.27:0/3865010226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0017070 con 0x7f1bd00b41f0 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bb7fff640 1 -- 172.21.15.27:0/3865010226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc00052d0 con 0x7f1bd00b41f0 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd0156460 con 0x7f1bd00b41f0 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bb7fff640 1 -- 172.21.15.27:0/3865010226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0003830 con 0x7f1bd00b41f0 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.474+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd0156700 con 0x7f1bd00b41f0 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.475+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd009fd30 con 0x7f1bd00b41f0 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.475+0000 7f1bb7fff640 1 -- 172.21.15.27:0/3865010226 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bc001a070 con 0x7f1bd00b41f0 2024-03-06T07:31:42.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.476+0000 7f1bb7fff640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.476+0000 7f1bb7fff640 1 -- 172.21.15.27:0/3865010226 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1bc00891e0 con 0x7f1bd00b41f0 2024-03-06T07:31:42.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.476+0000 7f1bceffd640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.477+0000 7f1bceffd640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1bd015c8a0 tx=0x7f1bc4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.480+0000 7f1bb7fff640 1 -- 172.21.15.27:0/3865010226 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1bc00583b0 con 0x7f1bd00b41f0 2024-03-06T07:31:42.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:42.660+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1bd006cb30 con 0x7f1bd00b41f0 2024-03-06T07:31:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.006+0000 7f1bb7fff640 1 -- 172.21.15.27:0/3865010226 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v142) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1bc0057cb0 con 0x7f1bd00b41f0 2024-03-06T07:31:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1bbc06b4d0 msgr2=0x7f1bbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1bd015c8a0 tx=0x7f1bc4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00b41f0 msgr2=0x7f1bd014c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00b41f0 0x7f1bd014c7b0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000c430 tx=0x7f1bc000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 shutdown_connections 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1bd015e2f0 0x7f1bd006b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd00cead0 0x7f1bd0157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 --2- 172.21.15.27:0/3865010226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bd00b41f0 0x7f1bd014c7b0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 >> 172.21.15.27:0/3865010226 conn(0x7f1bd00b36d0 msgr2=0x7f1bd00cef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 shutdown_connections 2024-03-06T07:31:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.009+0000 7f1bd5d19640 1 -- 172.21.15.27:0/3865010226 wait complete. 2024-03-06T07:31:43.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:31:43.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.140+0000 7f2ebcc50640 1 Processor -- start 2024-03-06T07:31:43.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.140+0000 7f2ebcc50640 1 -- start start 2024-03-06T07:31:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2ebcc50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb8144640 0x7f2eb806e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2ebcc50640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2eb80ce900 0x7f2eb80cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2ebcc50640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb815bc40 con 0x7f2eb806e940 2024-03-06T07:31:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2ebcc50640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb806df70 con 0x7f2eb80ce900 2024-03-06T07:31:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2ebcc50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb815bf20 con 0x7f2eb8144640 2024-03-06T07:31:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eaffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2eb806e940 0x7f2eb815b130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53220/0 (socket says 172.21.15.27:53220) 2024-03-06T07:31:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eaffff640 1 -- 172.21.15.27:0/4008774356 learned_addr learned my addr 172.21.15.27:0/4008774356 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb6d9d640 1 --2- 172.21.15.27:0/4008774356 >> v2:172.21.15.69:3300/0 conn(0x7f2eb80ce900 0x7f2eb80cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943716854 0 0) 0x7f2eb815bc40 con 0x7f2eb806e940 2024-03-06T07:31:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb659c640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb8144640 0x7f2eb806e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea4003520 con 0x7f2eb806e940 2024-03-06T07:31:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2503908574 0 0) 0x7f2ea4003520 con 0x7f2eb806e940 2024-03-06T07:31:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb8144640 msgr2=0x7f2eb806e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb5d9b640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb8144640 0x7f2eb806e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 >> v2:172.21.15.69:3300/0 conn(0x7f2eb80ce900 msgr2=0x7f2eb80cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb5d9b640 1 --2- 172.21.15.27:0/4008774356 >> v2:172.21.15.69:3300/0 conn(0x7f2eb80ce900 0x7f2eb80cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb806dc00 con 0x7f2eb806e940 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.141+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f2ea0001670 con 0x7f2eb806e940 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 >> v1:172.21.15.27:6789/0 conn(0x7f2eb806e940 legacy=0x7f2eb815b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb5d9b640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb8144640 0x7f2ea40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb5d9b640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea4003f30 0x7f2ea4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb5d9b640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4004d40 0x7f2ea40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb806df70 con 0x7f2ea4003f30 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb815bf20 con 0x7f2ea4004d40 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea400b9c0 con 0x7f2eb8144640 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eaffff640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea4003f30 0x7f2ea4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb659c640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb8144640 0x7f2ea40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb6d9d640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4004d40 0x7f2ea40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eaffff640 1 -- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb8144640 msgr2=0x7f2ea40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eaffff640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb8144640 0x7f2ea40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eaffff640 1 -- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4004d40 msgr2=0x7f2ea40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eaffff640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4004d40 0x7f2ea40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eaffff640 1 -- 172.21.15.27:0/4008774356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea4003520 con 0x7f2ea4003f30 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.142+0000 7f2eb6d9d640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4004d40 0x7f2ea40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.143+0000 7f2eaffff640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea4003f30 0x7f2ea4004800 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806eda0 tx=0x7f2ea4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24fb9de972bfe08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.143+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0007d50 con 0x7f2ea4003f30 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.143+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0030070 con 0x7f2ea4003f30 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.143+0000 7f2eb5d9b640 1 -- 172.21.15.27:0/4008774356 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0005d10 con 0x7f2ea4003f30 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.143+0000 7f2ebcc50640 1 -- 172.21.15.27:0/4008774356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea4003f30 msgr2=0x7f2ea4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.143+0000 7f2ebcc50640 1 --2- 172.21.15.27:0/4008774356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea4003f30 0x7f2ea4004800 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806eda0 tx=0x7f2ea4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.143+0000 7f2eb6d9d640 1 -- 172.21.15.27:0/4008774356 reap_dead start 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.144+0000 7f2ebcc50640 1 -- 172.21.15.27:0/4008774356 shutdown_connections 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.144+0000 7f2ebcc50640 1 -- 172.21.15.27:0/4008774356 >> 172.21.15.27:0/4008774356 conn(0x7f2eb8066d00 msgr2=0x7f2eb8067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.144+0000 7f2ebcc50640 1 -- 172.21.15.27:0/4008774356 shutdown_connections 2024-03-06T07:31:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.144+0000 7f2ebcc50640 1 -- 172.21.15.27:0/4008774356 wait complete. 2024-03-06T07:31:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.145+0000 7f2ebcc50640 1 Processor -- start 2024-03-06T07:31:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.145+0000 7f2ebcc50640 1 -- start start 2024-03-06T07:31:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2ebcc50640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4003f30 0x7f2eb8159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2ebcc50640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eb815a760 0x7f2eb81e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2ebcc50640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb81e6fc0 0x7f2eb81f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2ebcc50640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb815b9c0 con 0x7f2eb815a760 2024-03-06T07:31:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2ebcc50640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb815b840 con 0x7f2ea4003f30 2024-03-06T07:31:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2ebcc50640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb806e400 con 0x7f2eb81e6fc0 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eb815a760 0x7f2eb81e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eb815a760 0x7f2eb81e6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39710/0 (socket says 172.21.15.27:39710) 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaffff640 1 -- 172.21.15.27:0/3400174385 learned_addr learned my addr 172.21.15.27:0/3400174385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eb659c640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4003f30 0x7f2eb8159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eb6d9d640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb81e6fc0 0x7f2eb81f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaffff640 1 -- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb81e6fc0 msgr2=0x7f2eb81f16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaffff640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb81e6fc0 0x7f2eb81f16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaffff640 1 -- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4003f30 msgr2=0x7f2eb8159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaffff640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4003f30 0x7f2eb8159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaffff640 1 -- 172.21.15.27:0/3400174385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb81e4d90 con 0x7f2eb815a760 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaffff640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eb815a760 0x7f2eb81e6a80 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0007fd0 tx=0x7f2eb8144a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaeffd640 1 -- 172.21.15.27:0/3400174385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0045070 con 0x7f2eb815a760 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2eaeffd640 1 -- 172.21.15.27:0/3400174385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0004bf0 con 0x7f2eb815a760 2024-03-06T07:31:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.146+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb81e5070 con 0x7f2eb815a760 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.147+0000 7f2eaeffd640 1 -- 172.21.15.27:0/3400174385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0007070 con 0x7f2eb815a760 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.147+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb81e5630 con 0x7f2eb815a760 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.148+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb80733c0 con 0x7f2eb815a760 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.148+0000 7f2eaeffd640 1 -- 172.21.15.27:0/3400174385 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ea0007210 con 0x7f2eb815a760 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.149+0000 7f2eaeffd640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e9806b4d0 0x7f2e9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.149+0000 7f2eaeffd640 1 -- 172.21.15.27:0/3400174385 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f2ea00ab120 con 0x7f2eb815a760 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.149+0000 7f2eb659c640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e9806b4d0 0x7f2e9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.149+0000 7f2eb659c640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e9806b4d0 0x7f2e9806d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009a20 tx=0x7f2ea8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.153+0000 7f2eaeffd640 1 -- 172.21.15.27:0/3400174385 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ea007a2f0 con 0x7f2eb815a760 2024-03-06T07:31:43.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:43.336+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2eb804f410 con 0x7f2eb815a760 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.009+0000 7f2eaeffd640 1 -- 172.21.15.27:0/3400174385 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v143) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2ea0079bf0 con 0x7f2eb815a760 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.012+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e9806b4d0 msgr2=0x7f2e9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.013+0000 7f2ebcc50640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e9806b4d0 0x7f2e9806d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009a20 tx=0x7f2ea8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.013+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eb815a760 msgr2=0x7f2eb81e6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.013+0000 7f2ebcc50640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eb815a760 0x7f2eb81e6a80 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0007fd0 tx=0x7f2eb8144a20 comp rx=0 tx=0).stop 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.014+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 shutdown_connections 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.014+0000 7f2ebcc50640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2e9806b4d0 0x7f2e9806d990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.014+0000 7f2ebcc50640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2eb81e6fc0 0x7f2eb81f16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.014+0000 7f2ebcc50640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eb815a760 0x7f2eb81e6a80 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.014+0000 7f2ebcc50640 1 --2- 172.21.15.27:0/3400174385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ea4003f30 0x7f2eb8159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.014+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 >> 172.21.15.27:0/3400174385 conn(0x7f2eb8066d00 msgr2=0x7f2eb815d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.015+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 shutdown_connections 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.015+0000 7f2ebcc50640 1 -- 172.21.15.27:0/3400174385 wait complete. 2024-03-06T07:31:44.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.154+0000 7f9a32556640 1 Processor -- start 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.154+0000 7f9a32556640 1 -- start start 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.154+0000 7f9a32556640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a2c15e150 0x7f9a2c0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.154+0000 7f9a32556640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 0x7f9a2c161bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.154+0000 7f9a32556640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c0ce590 con 0x7f9a2c162130 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a32556640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2c1625a0 con 0x7f9a2c15e150 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a32556640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c15e530 con 0x7f9a2c162790 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a2bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a2c15e150 0x7f9a2c0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a2bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a2c15e150 0x7f9a2c0b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36488/0 (socket says 172.21.15.27:36488) 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a2bfff640 1 -- 172.21.15.27:0/3426721283 learned_addr learned my addr 172.21.15.27:0/3426721283 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a2b7fe640 1 --2- 172.21.15.27:0/3426721283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 0x7f9a2c161bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a2bfff640 1 -- 172.21.15.27:0/3426721283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 msgr2=0x7f9a2c161bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a2bfff640 1 --2- 172.21.15.27:0/3426721283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 0x7f9a2c161bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a2bfff640 1 -- 172.21.15.27:0/3426721283 >> v1:172.21.15.69:6789/0 conn(0x7f9a2c162130 legacy=0x7f9a2c0718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a2bfff640 1 -- 172.21.15.27:0/3426721283 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c0ce870 con 0x7f9a2c15e150 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.155+0000 7f9a2b7fe640 1 --2- 172.21.15.27:0/3426721283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 0x7f9a2c161bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.156+0000 7f9a2bfff640 1 --2- 172.21.15.27:0/3426721283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c15e150 0x7f9a2c0b3130 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0098a0 tx=0x7f9a1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84974571bc7947e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.156+0000 7f9a2affd640 1 -- 172.21.15.27:0/3426721283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c035020 con 0x7f9a2c15e150 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.156+0000 7f9a2affd640 1 -- 172.21.15.27:0/3426721283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1c031640 con 0x7f9a2c15e150 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.156+0000 7f9a2affd640 1 -- 172.21.15.27:0/3426721283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c02d040 con 0x7f9a2c15e150 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.156+0000 7f9a32556640 1 -- 172.21.15.27:0/3426721283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c15e150 msgr2=0x7f9a2c0b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.156+0000 7f9a32556640 1 --2- 172.21.15.27:0/3426721283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c15e150 0x7f9a2c0b3130 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0098a0 tx=0x7f9a1c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.157+0000 7f9a32556640 1 -- 172.21.15.27:0/3426721283 shutdown_connections 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.157+0000 7f9a32556640 1 --2- 172.21.15.27:0/3426721283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 0x7f9a2c161bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.157+0000 7f9a32556640 1 --2- 172.21.15.27:0/3426721283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c15e150 0x7f9a2c0b3130 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.157+0000 7f9a32556640 1 -- 172.21.15.27:0/3426721283 >> 172.21.15.27:0/3426721283 conn(0x7f9a2c0d40c0 msgr2=0x7f9a2c0d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.157+0000 7f9a32556640 1 -- 172.21.15.27:0/3426721283 shutdown_connections 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.157+0000 7f9a32556640 1 -- 172.21.15.27:0/3426721283 wait complete. 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.158+0000 7f9a32556640 1 Processor -- start 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.158+0000 7f9a32556640 1 -- start start 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a32556640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a2c15e150 0x7f9a2c151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a32556640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c162130 0x7f9a2c0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a32556640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 0x7f9a2c0b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a32556640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c0cf3d0 con 0x7f9a2c162130 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a32556640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c0cf0f0 con 0x7f9a2c15e150 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a32556640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c0cf550 con 0x7f9a2c162790 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c162130 0x7f9a2c0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c162130 0x7f9a2c0cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36492/0 (socket says 172.21.15.27:36492) 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2b7fe640 1 -- 172.21.15.27:0/1722569847 learned_addr learned my addr 172.21.15.27:0/1722569847 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2bfff640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a2c15e150 0x7f9a2c151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a30af3640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 0x7f9a2c0b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2b7fe640 1 -- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 msgr2=0x7f9a2c0b7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2b7fe640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 0x7f9a2c0b7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2b7fe640 1 -- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a2c15e150 msgr2=0x7f9a2c151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2b7fe640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a2c15e150 0x7f9a2c151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2b7fe640 1 -- 172.21.15.27:0/1722569847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c0cb5c0 con 0x7f9a2c162130 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a2b7fe640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c162130 0x7f9a2c0cad70 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000e7e0 tx=0x7f9a2000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a28ff9640 1 -- 172.21.15.27:0/1722569847 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2000f040 con 0x7f9a2c162130 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.159+0000 7f9a28ff9640 1 -- 172.21.15.27:0/1722569847 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a20004030 con 0x7f9a2c162130 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.160+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c0cb8a0 con 0x7f9a2c162130 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.160+0000 7f9a28ff9640 1 -- 172.21.15.27:0/1722569847 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20004400 con 0x7f9a2c162130 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.160+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c152330 con 0x7f9a2c162130 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.161+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2c06cdb0 con 0x7f9a2c162130 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.161+0000 7f9a28ff9640 1 -- 172.21.15.27:0/1722569847 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a2000c6d0 con 0x7f9a2c162130 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.161+0000 7f9a28ff9640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1806b4d0 0x7f9a1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.162+0000 7f9a28ff9640 1 -- 172.21.15.27:0/1722569847 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9a20089ec0 con 0x7f9a2c162130 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.162+0000 7f9a2bfff640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1806b4d0 0x7f9a1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.162+0000 7f9a2bfff640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1806b4d0 0x7f9a1806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c036000 tx=0x7f9a1c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.166+0000 7f9a28ff9640 1 -- 172.21.15.27:0/1722569847 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a20017020 con 0x7f9a2c162130 2024-03-06T07:31:44.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:44.384+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9a2c04f410 con 0x7f9a2c162130 2024-03-06T07:31:45.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.036+0000 7f9a28ff9640 1 -- 172.21.15.27:0/1722569847 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v108) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9a200587e0 con 0x7f9a2c162130 2024-03-06T07:31:45.038 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 20 and data pool 21 2024-03-06T07:31:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.044+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1806b4d0 msgr2=0x7f9a1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.045+0000 7f9a32556640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1806b4d0 0x7f9a1806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c036000 tx=0x7f9a1c03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.045+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c162130 msgr2=0x7f9a2c0cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.045+0000 7f9a32556640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c162130 0x7f9a2c0cad70 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000e7e0 tx=0x7f9a2000eca0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.045+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 shutdown_connections 2024-03-06T07:31:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.045+0000 7f9a32556640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a1806b4d0 0x7f9a1806d990 secure :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c036000 tx=0x7f9a1c03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.045+0000 7f9a32556640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a2c162790 0x7f9a2c0b7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.045+0000 7f9a32556640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a2c162130 0x7f9a2c0cad70 secure :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000e7e0 tx=0x7f9a2000eca0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.045+0000 7f9a32556640 1 --2- 172.21.15.27:0/1722569847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a2c15e150 0x7f9a2c151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.045+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 >> 172.21.15.27:0/1722569847 conn(0x7f9a2c0d40c0 msgr2=0x7f9a2c072690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.046+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 shutdown_connections 2024-03-06T07:31:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.046+0000 7f9a32556640 1 -- 172.21.15.27:0/1722569847 wait complete. 2024-03-06T07:31:45.065 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:45.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.172+0000 7f2dd5415640 1 Processor -- start 2024-03-06T07:31:45.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.172+0000 7f2dd5415640 1 -- start start 2024-03-06T07:31:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dd5415640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2dd015e060 0x7f2dd00ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dd5415640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2dd006fe20 0x7f2dd00ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dd5415640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd00d0dd0 con 0x7f2dd00d0880 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dd5415640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd00d10b0 con 0x7f2dd015e060 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dd5415640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd00cc0e0 con 0x7f2dd006fe20 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dceffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2dd015e060 0x7f2dd00ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dceffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2dd015e060 0x7f2dd00ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36498/0 (socket says 172.21.15.27:36498) 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dceffd640 1 -- 172.21.15.27:0/1242208436 learned_addr learned my addr 172.21.15.27:0/1242208436 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dce7fc640 1 --2- 172.21.15.27:0/1242208436 >> v2:172.21.15.69:3300/0 conn(0x7f2dd006fe20 0x7f2dd00ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dceffd640 1 -- 172.21.15.27:0/1242208436 >> v2:172.21.15.69:3300/0 conn(0x7f2dd006fe20 msgr2=0x7f2dd00ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dceffd640 1 --2- 172.21.15.27:0/1242208436 >> v2:172.21.15.69:3300/0 conn(0x7f2dd006fe20 0x7f2dd00ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dceffd640 1 -- 172.21.15.27:0/1242208436 >> v1:172.21.15.69:6789/0 conn(0x7f2dd00d0880 legacy=0x7f2dd006ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.173+0000 7f2dceffd640 1 -- 172.21.15.27:0/1242208436 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd00cc3c0 con 0x7f2dd015e060 2024-03-06T07:31:45.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.174+0000 7f2dceffd640 1 --2- 172.21.15.27:0/1242208436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 0x7f2dd00ce310 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0068cd0 tx=0x7f2dc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826da9e651a61fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.174+0000 7f2dcdffb640 1 -- 172.21.15.27:0/1242208436 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0035020 con 0x7f2dd015e060 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.174+0000 7f2dcdffb640 1 -- 172.21.15.27:0/1242208436 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc0031870 con 0x7f2dd015e060 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.174+0000 7f2dcdffb640 1 -- 172.21.15.27:0/1242208436 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0031c40 con 0x7f2dd015e060 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.175+0000 7f2dd5415640 1 -- 172.21.15.27:0/1242208436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 msgr2=0x7f2dd00ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.175+0000 7f2dd5415640 1 --2- 172.21.15.27:0/1242208436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 0x7f2dd00ce310 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0068cd0 tx=0x7f2dc002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.175+0000 7f2dd5415640 1 -- 172.21.15.27:0/1242208436 shutdown_connections 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.175+0000 7f2dd5415640 1 --2- 172.21.15.27:0/1242208436 >> v2:172.21.15.69:3300/0 conn(0x7f2dd006fe20 0x7f2dd00ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.175+0000 7f2dd5415640 1 --2- 172.21.15.27:0/1242208436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 0x7f2dd00ce310 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.175+0000 7f2dd5415640 1 -- 172.21.15.27:0/1242208436 >> 172.21.15.27:0/1242208436 conn(0x7f2dd00d1a90 msgr2=0x7f2dd00d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.175+0000 7f2dd5415640 1 -- 172.21.15.27:0/1242208436 shutdown_connections 2024-03-06T07:31:45.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.175+0000 7f2dd5415640 1 -- 172.21.15.27:0/1242208436 wait complete. 2024-03-06T07:31:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.176+0000 7f2dd5415640 1 Processor -- start 2024-03-06T07:31:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.176+0000 7f2dd5415640 1 -- start start 2024-03-06T07:31:45.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dd5415640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dd006fe20 0x7f2dd00aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dd5415640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dd00d0880 0x7f2dd014bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dd5415640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 0x7f2dd00b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dd5415640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00cdbd0 con 0x7f2dd015e060 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dd5415640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd015e440 con 0x7f2dd006fe20 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dd5415640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00cdd50 con 0x7f2dd00d0880 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dcf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 0x7f2dd00b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dcf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 0x7f2dd00b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36512/0 (socket says 172.21.15.27:36512) 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dcf7fe640 1 -- 172.21.15.27:0/2112128293 learned_addr learned my addr 172.21.15.27:0/2112128293 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dceffd640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dd006fe20 0x7f2dd00aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dce7fc640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dd00d0880 0x7f2dd014bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dcf7fe640 1 -- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dd00d0880 msgr2=0x7f2dd014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dcf7fe640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dd00d0880 0x7f2dd014bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dcf7fe640 1 -- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dd006fe20 msgr2=0x7f2dd00aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.177+0000 7f2dcf7fe640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dd006fe20 0x7f2dd00aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.178+0000 7f2dcf7fe640 1 -- 172.21.15.27:0/2112128293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd014c410 con 0x7f2dd015e060 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.178+0000 7f2dcf7fe640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 0x7f2dd00b2150 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800c780 tx=0x7f2dc800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.178+0000 7f2db77fe640 1 -- 172.21.15.27:0/2112128293 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8017070 con 0x7f2dd015e060 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.178+0000 7f2db77fe640 1 -- 172.21.15.27:0/2112128293 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc8003da0 con 0x7f2dd015e060 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.178+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd00b4980 con 0x7f2dd015e060 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.178+0000 7f2db77fe640 1 -- 172.21.15.27:0/2112128293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80052b0 con 0x7f2dd015e060 2024-03-06T07:31:45.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.178+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd00b4f40 con 0x7f2dd015e060 2024-03-06T07:31:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.179+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd0067ff0 con 0x7f2dd015e060 2024-03-06T07:31:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.179+0000 7f2db77fe640 1 -- 172.21.15.27:0/2112128293 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dc801a070 con 0x7f2dd015e060 2024-03-06T07:31:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.180+0000 7f2db77fe640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.180+0000 7f2db77fe640 1 -- 172.21.15.27:0/2112128293 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dc80897d0 con 0x7f2dd015e060 2024-03-06T07:31:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.180+0000 7f2dceffd640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.180+0000 7f2dceffd640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0036000 tx=0x7f2dc003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.184+0000 7f2db77fe640 1 -- 172.21.15.27:0/2112128293 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dd014c410 con 0x7f2dd015e060 2024-03-06T07:31:45.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.371+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dd00b7910 con 0x7f2dd015e060 2024-03-06T07:31:45.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.372+0000 7f2db77fe640 1 -- 172.21.15.27:0/2112128293 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f2dc8058000 con 0x7f2dd015e060 2024-03-06T07:31:45.373 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:45.374 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":144,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:31:44.385062+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:45.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dbc06b4d0 msgr2=0x7f2dbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0036000 tx=0x7f2dc003c040 comp rx=0 tx=0).stop 2024-03-06T07:31:45.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 msgr2=0x7f2dd00b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 0x7f2dd00b2150 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800c780 tx=0x7f2dc800cc40 comp rx=0 tx=0).stop 2024-03-06T07:31:45.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 shutdown_connections 2024-03-06T07:31:45.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2dd015e060 0x7f2dd00b2150 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2dd00d0880 0x7f2dd014bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 --2- 172.21.15.27:0/2112128293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2dd006fe20 0x7f2dd00aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 >> 172.21.15.27:0/2112128293 conn(0x7f2dd00d1a90 msgr2=0x7f2dd006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 shutdown_connections 2024-03-06T07:31:45.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.374+0000 7f2dd5415640 1 -- 172.21.15.27:0/2112128293 wait complete. 2024-03-06T07:31:45.387 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:45.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.492+0000 7f0aa3ac9640 1 Processor -- start 2024-03-06T07:31:45.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.492+0000 7f0aa3ac9640 1 -- start start 2024-03-06T07:31:45.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.493+0000 7f0aa3ac9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c071250 0x7f0a9c15b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.493+0000 7f0aa3ac9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c06c7b0 con 0x7f0a9c157f60 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.493+0000 7f0aa3ac9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c06ca90 con 0x7f0a9c157b80 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.493+0000 7f0aa3ac9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c06cd70 con 0x7f0a9c071250 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.493+0000 7f0aa1064640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0a9c157f60 0x7f0a9c070b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52936/0 (socket says 172.21.15.27:52936) 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.493+0000 7f0aa1064640 1 -- 172.21.15.27:0/3926633820 learned_addr learned my addr 172.21.15.27:0/3926633820 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.493+0000 7f0aa2066640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c071250 0x7f0a9c15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.493+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149288215 0 0) 0x7f0a9c06c7b0 con 0x7f0a9c157f60 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.493+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a84003520 con 0x7f0a9c157f60 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041281546 0 0) 0x7f0a84003520 con 0x7f0a9c157f60 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c071250 msgr2=0x7f0a9c15b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c071250 0x7f0a9c15b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 >> v1:172.21.15.69:6789/0 conn(0x7f0a9c157b80 legacy=0x7f0a9c15b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c06d050 con 0x7f0a9c157f60 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f0a90001670 con 0x7f0a9c157f60 2024-03-06T07:31:45.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 >> v1:172.21.15.27:6789/0 conn(0x7f0a9c157f60 legacy=0x7f0a9c070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a840039e0 0x7f0a84003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c071250 0x7f0a84004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a84004d40 0x7f0a840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c06cd70 con 0x7f0a84004d40 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8400b9c0 con 0x7f0a840039e0 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8400bbc0 con 0x7f0a9c071250 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa2066640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a84004d40 0x7f0a840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa1865640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a840039e0 0x7f0a84003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa1064640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c071250 0x7f0a84004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa2066640 1 -- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c071250 msgr2=0x7f0a84004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa2066640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c071250 0x7f0a84004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa2066640 1 -- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a840039e0 msgr2=0x7f0a84003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa2066640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a840039e0 0x7f0a84003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.494+0000 7f0aa2066640 1 -- 172.21.15.27:0/3926633820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a84003520 con 0x7f0a84004d40 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.495+0000 7f0aa2066640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a84004d40 0x7f0a840051c0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009f90 tx=0x7f0a84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88bed9e630ae064b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.495+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c039070 con 0x7f0a84004d40 2024-03-06T07:31:45.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.495+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c02cbd0 con 0x7f0a84004d40 2024-03-06T07:31:45.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.495+0000 7f0aa0863640 1 -- 172.21.15.27:0/3926633820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c009c10 con 0x7f0a84004d40 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.496+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/3926633820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a84004d40 msgr2=0x7f0a840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.496+0000 7f0aa3ac9640 1 --2- 172.21.15.27:0/3926633820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a84004d40 0x7f0a840051c0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009f90 tx=0x7f0a84005830 comp rx=0 tx=0).stop 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.496+0000 7f0aa2066640 1 -- 172.21.15.27:0/3926633820 reap_dead start 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.496+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/3926633820 shutdown_connections 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.496+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/3926633820 >> 172.21.15.27:0/3926633820 conn(0x7f0a9c066d00 msgr2=0x7f0a9c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.496+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/3926633820 shutdown_connections 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.496+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/3926633820 wait complete. 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.497+0000 7f0aa3ac9640 1 Processor -- start 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.497+0000 7f0aa3ac9640 1 -- start start 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa3ac9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c137df0 0x7f0a9c0cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa3ac9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a9c1381d0 0x7f0a9c0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa3ac9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c0ccef0 0x7f0a9c13bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa3ac9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0cbab0 con 0x7f0a9c1381d0 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa3ac9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0cc260 con 0x7f0a9c0ccef0 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa3ac9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0cbc30 con 0x7f0a9c137df0 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1064640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a9c1381d0 0x7f0a9c0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1064640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a9c1381d0 0x7f0a9c0b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36526/0 (socket says 172.21.15.27:36526) 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1064640 1 -- 172.21.15.27:0/840429676 learned_addr learned my addr 172.21.15.27:0/840429676 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1865640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c137df0 0x7f0a9c0cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa2066640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c0ccef0 0x7f0a9c13bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1064640 1 -- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c137df0 msgr2=0x7f0a9c0cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1064640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c137df0 0x7f0a9c0cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1064640 1 -- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c0ccef0 msgr2=0x7f0a9c13bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1064640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c0ccef0 0x7f0a9c13bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1064640 1 -- 172.21.15.27:0/840429676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c0ce3f0 con 0x7f0a9c1381d0 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1865640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c137df0 0x7f0a9c0cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.498+0000 7f0aa1064640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a9c1381d0 0x7f0a9c0b3020 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f0a90007290 tx=0x7f0a90007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.499+0000 7f0a9a7fc640 1 -- 172.21.15.27:0/840429676 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9000f960 con 0x7f0a9c1381d0 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.499+0000 7f0a9a7fc640 1 -- 172.21.15.27:0/840429676 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a90018070 con 0x7f0a9c1381d0 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.499+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9c0ce6d0 con 0x7f0a9c1381d0 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.499+0000 7f0a9a7fc640 1 -- 172.21.15.27:0/840429676 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9000fd60 con 0x7f0a9c1381d0 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.499+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9c0cc6f0 con 0x7f0a9c1381d0 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.500+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9c067ff0 con 0x7f0a9c1381d0 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.500+0000 7f0a9a7fc640 1 -- 172.21.15.27:0/840429676 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a9001d460 con 0x7f0a9c1381d0 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.500+0000 7f0a9a7fc640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8806b4d0 0x7f0a8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.501+0000 7f0a9a7fc640 1 -- 172.21.15.27:0/840429676 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a9008a360 con 0x7f0a9c1381d0 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.501+0000 7f0aa1865640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8806b4d0 0x7f0a8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.501+0000 7f0aa1865640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8806b4d0 0x7f0a8806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0a94003700 tx=0x7f0a94009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.505+0000 7f0a9a7fc640 1 -- 172.21.15.27:0/840429676 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a900592a0 con 0x7f0a9c1381d0 2024-03-06T07:31:45.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.685+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a9c06d930 con 0x7f0a9c1381d0 2024-03-06T07:31:45.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.686+0000 7f0a9a7fc640 1 -- 172.21.15.27:0/840429676 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f0a90058ba0 con 0x7f0a9c1381d0 2024-03-06T07:31:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:45.688 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":144,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:31:44.385062+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8806b4d0 msgr2=0x7f0a8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8806b4d0 0x7f0a8806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0a94003700 tx=0x7f0a94009040 comp rx=0 tx=0).stop 2024-03-06T07:31:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a9c1381d0 msgr2=0x7f0a9c0b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a9c1381d0 0x7f0a9c0b3020 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f0a90007290 tx=0x7f0a90007750 comp rx=0 tx=0).stop 2024-03-06T07:31:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 shutdown_connections 2024-03-06T07:31:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8806b4d0 0x7f0a8806d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a9c0ccef0 0x7f0a9c13bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a9c1381d0 0x7f0a9c0b3020 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 --2- 172.21.15.27:0/840429676 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0a9c137df0 0x7f0a9c0cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 >> 172.21.15.27:0/840429676 conn(0x7f0a9c066d00 msgr2=0x7f0a9c158400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 shutdown_connections 2024-03-06T07:31:45.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.688+0000 7f0aa3ac9640 1 -- 172.21.15.27:0/840429676 wait complete. 2024-03-06T07:31:45.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:31:45.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.810+0000 7f672b62a640 1 Processor -- start 2024-03-06T07:31:45.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.810+0000 7f672b62a640 1 -- start start 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f672b62a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67240ce960 0x7f67240d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f672b62a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672415f730 con 0x7f67241618e0 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f672b62a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672415ebb0 con 0x7f67240ce4f0 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f672b62a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67240cfce0 con 0x7f67240ce960 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f67293c6640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f67241618e0 0x7f672406b130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52946/0 (socket says 172.21.15.27:52946) 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f67293c6640 1 -- 172.21.15.27:0/2205457459 learned_addr learned my addr 172.21.15.27:0/2205457459 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f6729bc7640 1 --2- 172.21.15.27:0/2205457459 >> v2:172.21.15.69:3300/0 conn(0x7f67240ce960 0x7f67240d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f671bfff640 1 -- 172.21.15.27:0/2205457459 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570792720 0 0) 0x7f672415f730 con 0x7f67241618e0 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f671bfff640 1 -- 172.21.15.27:0/2205457459 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6710003520 con 0x7f67241618e0 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f6729bc7640 1 -- 172.21.15.27:0/2205457459 >> v1:172.21.15.69:6789/0 conn(0x7f67240ce4f0 legacy=0x7f67240d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f6729bc7640 1 -- 172.21.15.27:0/2205457459 >> v1:172.21.15.27:6789/0 conn(0x7f67241618e0 legacy=0x7f672406b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f671bfff640 1 -- 172.21.15.27:0/2205457459 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 241853423 0 0) 0x7f6710003520 con 0x7f67241618e0 2024-03-06T07:31:45.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.811+0000 7f6729bc7640 1 -- 172.21.15.27:0/2205457459 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672415ee90 con 0x7f67240ce960 2024-03-06T07:31:45.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.812+0000 7f6729bc7640 1 --2- 172.21.15.27:0/2205457459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67240ce960 0x7f67240d0860 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f671c009d90 tx=0x7f671c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42091017b9dbf7a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.812+0000 7f671bfff640 1 -- 172.21.15.27:0/2205457459 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c039070 con 0x7f67240ce960 2024-03-06T07:31:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.812+0000 7f671bfff640 1 -- 172.21.15.27:0/2205457459 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671c02c9d0 con 0x7f67240ce960 2024-03-06T07:31:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.812+0000 7f671bfff640 1 -- 172.21.15.27:0/2205457459 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720001670 con 0x7f67240ce960 2024-03-06T07:31:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.813+0000 7f672b62a640 1 -- 172.21.15.27:0/2205457459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67240ce960 msgr2=0x7f67240d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.813+0000 7f672b62a640 1 --2- 172.21.15.27:0/2205457459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67240ce960 0x7f67240d0860 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f671c009d90 tx=0x7f671c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.813+0000 7f672b62a640 1 -- 172.21.15.27:0/2205457459 shutdown_connections 2024-03-06T07:31:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.813+0000 7f672b62a640 1 --2- 172.21.15.27:0/2205457459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67240ce960 0x7f67240d0860 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.813+0000 7f672b62a640 1 -- 172.21.15.27:0/2205457459 >> 172.21.15.27:0/2205457459 conn(0x7f67240b3660 msgr2=0x7f67240b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.813+0000 7f672b62a640 1 -- 172.21.15.27:0/2205457459 shutdown_connections 2024-03-06T07:31:45.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.813+0000 7f672b62a640 1 -- 172.21.15.27:0/2205457459 wait complete. 2024-03-06T07:31:45.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.814+0000 7f672b62a640 1 Processor -- start 2024-03-06T07:31:45.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.814+0000 7f672b62a640 1 -- start start 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f672b62a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240ce4f0 0x7f6724154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f672b62a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67240ce960 0x7f67240c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f672b62a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240ca180 0x7f67240b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f672b62a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67240cedb0 con 0x7f67240ce4f0 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f672b62a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672415f450 con 0x7f67240ce960 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f672b62a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672415e7d0 con 0x7f67240ca180 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f67293c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240ce4f0 0x7f6724154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f67293c6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240ce4f0 0x7f6724154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36538/0 (socket says 172.21.15.27:36538) 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f67293c6640 1 -- 172.21.15.27:0/1279810048 learned_addr learned my addr 172.21.15.27:0/1279810048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f6729bc7640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240ca180 0x7f67240b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f6728bc5640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67240ce960 0x7f67240c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f67293c6640 1 -- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240ca180 msgr2=0x7f67240b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f67293c6640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240ca180 0x7f67240b4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f67293c6640 1 -- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67240ce960 msgr2=0x7f67240c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f67293c6640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67240ce960 0x7f67240c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.815+0000 7f67293c6640 1 -- 172.21.15.27:0/1279810048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67240ca960 con 0x7f67240ce4f0 2024-03-06T07:31:45.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.816+0000 7f67293c6640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240ce4f0 0x7f6724154560 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f672000c520 tx=0x7f672000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.816+0000 7f6719ffb640 1 -- 172.21.15.27:0/1279810048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720017070 con 0x7f67240ce4f0 2024-03-06T07:31:45.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.816+0000 7f6719ffb640 1 -- 172.21.15.27:0/1279810048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720005d00 con 0x7f67240ce4f0 2024-03-06T07:31:45.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.816+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6724158210 con 0x7f67240ce4f0 2024-03-06T07:31:45.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.816+0000 7f6719ffb640 1 -- 172.21.15.27:0/1279810048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67200072b0 con 0x7f67240ce4f0 2024-03-06T07:31:45.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.816+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67241587d0 con 0x7f67240ce4f0 2024-03-06T07:31:45.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.817+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6724068e20 con 0x7f67240ce4f0 2024-03-06T07:31:45.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.817+0000 7f6719ffb640 1 -- 172.21.15.27:0/1279810048 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f672001a070 con 0x7f67240ce4f0 2024-03-06T07:31:45.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.818+0000 7f6719ffb640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670806b4d0 0x7f670806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.818+0000 7f6719ffb640 1 -- 172.21.15.27:0/1279810048 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6720089660 con 0x7f67240ce4f0 2024-03-06T07:31:45.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.818+0000 7f6728bc5640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670806b4d0 0x7f670806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.818+0000 7f6728bc5640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670806b4d0 0x7f670806d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f672415cdd0 tx=0x7f6714009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:45.822+0000 7f6719ffb640 1 -- 172.21.15.27:0/1279810048 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6724068e20 con 0x7f67240ce4f0 2024-03-06T07:31:46.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:46.054+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f67241589a0 con 0x7f67240ce4f0 2024-03-06T07:31:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.045+0000 7f6719ffb640 1 -- 172.21.15.27:0/1279810048 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v111) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6720057e90 con 0x7f67240ce4f0 2024-03-06T07:31:47.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670806b4d0 msgr2=0x7f670806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670806b4d0 0x7f670806d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f672415cdd0 tx=0x7f6714009040 comp rx=0 tx=0).stop 2024-03-06T07:31:47.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240ce4f0 msgr2=0x7f6724154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240ce4f0 0x7f6724154560 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f672000c520 tx=0x7f672000c9e0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 shutdown_connections 2024-03-06T07:31:47.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f670806b4d0 0x7f670806d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67240ca180 0x7f67240b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67240ce960 0x7f67240c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 --2- 172.21.15.27:0/1279810048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67240ce4f0 0x7f6724154560 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 >> 172.21.15.27:0/1279810048 conn(0x7f67240b3660 msgr2=0x7f672406b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 shutdown_connections 2024-03-06T07:31:47.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.047+0000 7f672b62a640 1 -- 172.21.15.27:0/1279810048 wait complete. 2024-03-06T07:31:47.060 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:31:47.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:31:47.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.168+0000 7feaffe6b640 1 Processor -- start 2024-03-06T07:31:47.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.168+0000 7feaffe6b640 1 -- start start 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.169+0000 7feaffe6b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf80b7420 0x7feaf80d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.169+0000 7feaffe6b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf80d3740 con 0x7feaf815dd90 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.169+0000 7feaffe6b640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf80d3120 con 0x7feaf80ceea0 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.169+0000 7feaffe6b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf80d3400 con 0x7feaf80b7420 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.169+0000 7feafdc07640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feaf815dd90 0x7feaf80d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52958/0 (socket says 172.21.15.27:52958) 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.169+0000 7feafdc07640 1 -- 172.21.15.27:0/456698114 learned_addr learned my addr 172.21.15.27:0/456698114 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafe408640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf80b7420 0x7feaf80d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525407820 0 0) 0x7feaf80d3740 con 0x7feaf815dd90 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feae4003520 con 0x7feaf815dd90 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3240179491 0 0) 0x7feae4003520 con 0x7feaf815dd90 2024-03-06T07:31:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf80b7420 msgr2=0x7feaf80d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf80b7420 0x7feaf80d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 >> v1:172.21.15.69:6789/0 conn(0x7feaf80ceea0 legacy=0x7feaf80b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80d39e0 con 0x7feaf815dd90 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7feaf4001670 con 0x7feaf815dd90 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 >> v1:172.21.15.27:6789/0 conn(0x7feaf815dd90 legacy=0x7feaf80d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feae4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae4004390 0x7feae4004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae4005010 0x7feae400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf80d3400 con 0x7feae40039e0 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae400bf00 con 0x7feae4005010 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae400c080 con 0x7feae4004390 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafe408640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf80b7420 0x7feaf80d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafdc07640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feae4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafd406640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae4004390 0x7feae4004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.170+0000 7feafe408640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae4005010 0x7feae400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.171+0000 7feafdc07640 1 -- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae4004390 msgr2=0x7feae4004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.171+0000 7feafdc07640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae4004390 0x7feae4004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.171+0000 7feafdc07640 1 -- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae4005010 msgr2=0x7feae400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.171+0000 7feafdc07640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feae4005010 0x7feae400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.171+0000 7feafdc07640 1 -- 172.21.15.27:0/456698114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae4003520 con 0x7feae40039e0 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.171+0000 7feafd406640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feae4004390 0x7feae4004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.171+0000 7feafdc07640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feae4003dc0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7feaf400c210 tx=0x7feae4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3045a639986223 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.171+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4007ce0 con 0x7feae40039e0 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.172+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf4030070 con 0x7feae40039e0 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.172+0000 7feafcc05640 1 -- 172.21.15.27:0/456698114 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4005d30 con 0x7feae40039e0 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.172+0000 7feaffe6b640 1 -- 172.21.15.27:0/456698114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 msgr2=0x7feae4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.172+0000 7feaffe6b640 1 --2- 172.21.15.27:0/456698114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feae4003dc0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7feaf400c210 tx=0x7feae4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:47.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.172+0000 7feafe408640 1 -- 172.21.15.27:0/456698114 reap_dead start 2024-03-06T07:31:47.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.172+0000 7feaffe6b640 1 -- 172.21.15.27:0/456698114 shutdown_connections 2024-03-06T07:31:47.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.172+0000 7feaffe6b640 1 -- 172.21.15.27:0/456698114 >> 172.21.15.27:0/456698114 conn(0x7feaf8071960 msgr2=0x7feaf8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.172+0000 7feaffe6b640 1 -- 172.21.15.27:0/456698114 shutdown_connections 2024-03-06T07:31:47.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.172+0000 7feaffe6b640 1 -- 172.21.15.27:0/456698114 wait complete. 2024-03-06T07:31:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.173+0000 7feaffe6b640 1 Processor -- start 2024-03-06T07:31:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.173+0000 7feaffe6b640 1 -- start start 2024-03-06T07:31:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feaffe6b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feaf815cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feaffe6b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf80c8800 0x7feaf814f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feaffe6b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf814fb40 0x7feaf80c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feaffe6b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf80d48f0 con 0x7feae40039e0 2024-03-06T07:31:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feaffe6b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf815e170 con 0x7feaf80c8800 2024-03-06T07:31:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feaffe6b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf80d4a70 con 0x7feaf814fb40 2024-03-06T07:31:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feafdc07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feaf815cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feafdc07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feaf815cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36556/0 (socket says 172.21.15.27:36556) 2024-03-06T07:31:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feafdc07640 1 -- 172.21.15.27:0/40331451 learned_addr learned my addr 172.21.15.27:0/40331451 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feafd406640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf80c8800 0x7feaf814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.174+0000 7feafe408640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf814fb40 0x7feaf80c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feafdc07640 1 -- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf814fb40 msgr2=0x7feaf80c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feafdc07640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf814fb40 0x7feaf80c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feafdc07640 1 -- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf80c8800 msgr2=0x7feaf814f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feafdc07640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf80c8800 0x7feaf814f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feafdc07640 1 -- 172.21.15.27:0/40331451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80c8c90 con 0x7feae40039e0 2024-03-06T07:31:47.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feafd406640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf80c8800 0x7feaf814f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:47.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feafdc07640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feaf815cc70 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7feaf402c500 tx=0x7feaf80d3bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feaee7fc640 1 -- 172.21.15.27:0/40331451 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4045070 con 0x7feae40039e0 2024-03-06T07:31:47.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feaee7fc640 1 -- 172.21.15.27:0/40331451 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf4000ed0 con 0x7feae40039e0 2024-03-06T07:31:47.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf80c8f70 con 0x7feae40039e0 2024-03-06T07:31:47.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feaee7fc640 1 -- 172.21.15.27:0/40331451 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4007ce0 con 0x7feae40039e0 2024-03-06T07:31:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.175+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf80c9530 con 0x7feae40039e0 2024-03-06T07:31:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.176+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf8072a30 con 0x7feae40039e0 2024-03-06T07:31:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.177+0000 7feaee7fc640 1 -- 172.21.15.27:0/40331451 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feaf4007070 con 0x7feae40039e0 2024-03-06T07:31:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.178+0000 7feaee7fc640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feadc06b500 0x7feadc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.178+0000 7feaee7fc640 1 -- 172.21.15.27:0/40331451 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feaf40ab140 con 0x7feae40039e0 2024-03-06T07:31:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.178+0000 7feafd406640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feadc06b500 0x7feadc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.179+0000 7feafd406640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feadc06b500 0x7feadc06d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7feaf80b5330 tx=0x7feae8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.182+0000 7feaee7fc640 1 -- 172.21.15.27:0/40331451 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feaf407a130 con 0x7feae40039e0 2024-03-06T07:31:47.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:47.403+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7feaf804f410 con 0x7feae40039e0 2024-03-06T07:31:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.046+0000 7feaee7fc640 1 -- 172.21.15.27:0/40331451 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v112) v1 ==== 122+0+0 (secure 0 0 0) 0x7feaf4079a30 con 0x7feae40039e0 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feadc06b500 msgr2=0x7feadc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feadc06b500 0x7feadc06d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7feaf80b5330 tx=0x7feae8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 msgr2=0x7feaf815cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feaf815cc70 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7feaf402c500 tx=0x7feaf80d3bb0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 shutdown_connections 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feadc06b500 0x7feadc06d9c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7feaf814fb40 0x7feaf80c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feaf80c8800 0x7feaf814f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 --2- 172.21.15.27:0/40331451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feae40039e0 0x7feaf815cc70 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 >> 172.21.15.27:0/40331451 conn(0x7feaf8071960 msgr2=0x7feaf80d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 shutdown_connections 2024-03-06T07:31:48.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.048+0000 7feaffe6b640 1 -- 172.21.15.27:0/40331451 wait complete. 2024-03-06T07:31:48.061 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:31:48.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:48.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.166+0000 7f2fefdd0640 1 Processor -- start 2024-03-06T07:31:48.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.166+0000 7f2fefdd0640 1 -- start start 2024-03-06T07:31:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.167+0000 7f2fefdd0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2fe815df00 0x7f2fe80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.167+0000 7f2fefdd0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fe80b9a30 0x7f2fe80d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.167+0000 7f2fefdd0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe80cea30 con 0x7f2fe80b95a0 2024-03-06T07:31:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.167+0000 7f2fefdd0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80d2af0 con 0x7f2fe80b9a30 2024-03-06T07:31:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.167+0000 7f2fefdd0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80cf3c0 con 0x7f2fe815df00 2024-03-06T07:31:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.167+0000 7f2fee36d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fe80b9a30 0x7f2fe80d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.167+0000 7f2fee36d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fe80b9a30 0x7f2fe80d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36558/0 (socket says 172.21.15.27:36558) 2024-03-06T07:31:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.167+0000 7f2fee36d640 1 -- 172.21.15.27:0/3044620430 learned_addr learned my addr 172.21.15.27:0/3044620430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.167+0000 7f2fedb6c640 1 --2- 172.21.15.27:0/3044620430 >> v2:172.21.15.69:3300/0 conn(0x7f2fe815df00 0x7f2fe80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.168+0000 7f2fee36d640 1 -- 172.21.15.27:0/3044620430 >> v2:172.21.15.69:3300/0 conn(0x7f2fe815df00 msgr2=0x7f2fe80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.168+0000 7f2fee36d640 1 --2- 172.21.15.27:0/3044620430 >> v2:172.21.15.69:3300/0 conn(0x7f2fe815df00 0x7f2fe80d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.168+0000 7f2fee36d640 1 -- 172.21.15.27:0/3044620430 >> v1:172.21.15.69:6789/0 conn(0x7f2fe80b95a0 legacy=0x7f2fe80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.168+0000 7f2fee36d640 1 -- 172.21.15.27:0/3044620430 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe80cf0e0 con 0x7f2fe80b9a30 2024-03-06T07:31:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.168+0000 7f2fee36d640 1 --2- 172.21.15.27:0/3044620430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b9a30 0x7f2fe80d3220 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80d11b0 tx=0x7f2fe002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee93e956a765eea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fecb6a640 1 -- 172.21.15.27:0/3044620430 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe002cb20 con 0x7f2fe80b9a30 2024-03-06T07:31:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fecb6a640 1 -- 172.21.15.27:0/3044620430 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe002ccc0 con 0x7f2fe80b9a30 2024-03-06T07:31:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fecb6a640 1 -- 172.21.15.27:0/3044620430 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0034610 con 0x7f2fe80b9a30 2024-03-06T07:31:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fefdd0640 1 -- 172.21.15.27:0/3044620430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b9a30 msgr2=0x7f2fe80d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fefdd0640 1 --2- 172.21.15.27:0/3044620430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b9a30 0x7f2fe80d3220 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80d11b0 tx=0x7f2fe002bf80 comp rx=0 tx=0).stop 2024-03-06T07:31:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fefdd0640 1 -- 172.21.15.27:0/3044620430 shutdown_connections 2024-03-06T07:31:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fefdd0640 1 --2- 172.21.15.27:0/3044620430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b9a30 0x7f2fe80d3220 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fefdd0640 1 --2- 172.21.15.27:0/3044620430 >> v2:172.21.15.69:3300/0 conn(0x7f2fe815df00 0x7f2fe80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fefdd0640 1 -- 172.21.15.27:0/3044620430 >> 172.21.15.27:0/3044620430 conn(0x7f2fe8071960 msgr2=0x7f2fe8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fefdd0640 1 -- 172.21.15.27:0/3044620430 shutdown_connections 2024-03-06T07:31:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.169+0000 7f2fefdd0640 1 -- 172.21.15.27:0/3044620430 wait complete. 2024-03-06T07:31:48.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fefdd0640 1 Processor -- start 2024-03-06T07:31:48.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fefdd0640 1 -- start start 2024-03-06T07:31:48.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fefdd0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b95a0 0x7f2fe8154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fefdd0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fe815df00 0x7f2fe80b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fefdd0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fe8155d50 0x7f2fe80b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fefdd0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80cec70 con 0x7f2fe80b95a0 2024-03-06T07:31:48.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fefdd0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80ce750 con 0x7f2fe8155d50 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fefdd0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80cedf0 con 0x7f2fe815df00 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fedb6c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b95a0 0x7f2fe8154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fedb6c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b95a0 0x7f2fe8154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36570/0 (socket says 172.21.15.27:36570) 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.171+0000 7f2fedb6c640 1 -- 172.21.15.27:0/988959105 learned_addr learned my addr 172.21.15.27:0/988959105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fed36b640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fe815df00 0x7f2fe80b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fee36d640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fe8155d50 0x7f2fe80b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fedb6c640 1 -- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fe815df00 msgr2=0x7f2fe80b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fedb6c640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fe815df00 0x7f2fe80b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fedb6c640 1 -- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fe8155d50 msgr2=0x7f2fe80b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fedb6c640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fe8155d50 0x7f2fe80b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fedb6c640 1 -- 172.21.15.27:0/988959105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe815e2e0 con 0x7f2fe80b95a0 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fedb6c640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b95a0 0x7f2fe8154120 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800e870 tx=0x7f2fd800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fde7fc640 1 -- 172.21.15.27:0/988959105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8017070 con 0x7f2fe80b95a0 2024-03-06T07:31:48.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fde7fc640 1 -- 172.21.15.27:0/988959105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd800c940 con 0x7f2fe80b95a0 2024-03-06T07:31:48.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe815e5c0 con 0x7f2fe80b95a0 2024-03-06T07:31:48.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fde7fc640 1 -- 172.21.15.27:0/988959105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd800cd30 con 0x7f2fe80b95a0 2024-03-06T07:31:48.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.172+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe815eb80 con 0x7f2fe80b95a0 2024-03-06T07:31:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.173+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe8067ff0 con 0x7f2fe80b95a0 2024-03-06T07:31:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.174+0000 7f2fde7fc640 1 -- 172.21.15.27:0/988959105 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fd801a070 con 0x7f2fe80b95a0 2024-03-06T07:31:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.174+0000 7f2fde7fc640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fcc06b500 0x7f2fcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.174+0000 7f2fde7fc640 1 -- 172.21.15.27:0/988959105 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fd808a3c0 con 0x7f2fe80b95a0 2024-03-06T07:31:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.174+0000 7f2fed36b640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fcc06b500 0x7f2fcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.175+0000 7f2fed36b640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fcc06b500 0x7f2fcc06d9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8155440 tx=0x7f2fe4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.184+0000 7f2fde7fc640 1 -- 172.21.15.27:0/988959105 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fd8058b60 con 0x7f2fe80b95a0 2024-03-06T07:31:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.401+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fe80d2650 con 0x7f2fe80b95a0 2024-03-06T07:31:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.402+0000 7f2fde7fc640 1 -- 172.21.15.27:0/988959105 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4984 (secure 0 0 0) 0x7f2fd805cc00 con 0x7f2fe80b95a0 2024-03-06T07:31:48.403 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:48.404 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1243350995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1243350995},{"type":"v1","addr":"172.21.15.69:6836","nonce":1243350995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":8257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/470342234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":470342234},{"type":"v1","addr":"172.21.15.27:6835","nonce":470342234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":9270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:31:47.404035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:31:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-06T07:31:48.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fcc06b500 msgr2=0x7f2fcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fcc06b500 0x7f2fcc06d9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8155440 tx=0x7f2fe4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b95a0 msgr2=0x7f2fe8154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b95a0 0x7f2fe8154120 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800e870 tx=0x7f2fd800ed30 comp rx=0 tx=0).stop 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 shutdown_connections 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2fcc06b500 0x7f2fcc06d9c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fe8155d50 0x7f2fe80b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2fe815df00 0x7f2fe80b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 --2- 172.21.15.27:0/988959105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe80b95a0 0x7f2fe8154120 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 >> 172.21.15.27:0/988959105 conn(0x7f2fe8071960 msgr2=0x7f2fe80b8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 shutdown_connections 2024-03-06T07:31:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.404+0000 7f2fefdd0640 1 -- 172.21.15.27:0/988959105 wait complete. 2024-03-06T07:31:48.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.522+0000 7f9663081640 1 Processor -- start 2024-03-06T07:31:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.522+0000 7f9663081640 1 -- start start 2024-03-06T07:31:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f9663081640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0d3400 0x7f965c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f9663081640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f965c0b2b50 0x7f965c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f9663081640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c0ce610 con 0x7f965c071540 2024-03-06T07:31:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f9663081640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965c0d2f70 con 0x7f965c0b2b50 2024-03-06T07:31:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f9663081640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c0d4be0 con 0x7f965c0d3400 2024-03-06T07:31:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f966161e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f965c0b2b50 0x7f965c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f966161e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f965c0b2b50 0x7f965c0b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36586/0 (socket says 172.21.15.27:36586) 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f9660e1d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f965c071540 0x7f965c071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52966/0 (socket says 172.21.15.27:52966) 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f966161e640 1 -- 172.21.15.27:0/1787149374 learned_addr learned my addr 172.21.15.27:0/1787149374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f965bfff640 1 --2- 172.21.15.27:0/1787149374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0d3400 0x7f965c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f966161e640 1 -- 172.21.15.27:0/1787149374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0d3400 msgr2=0x7f965c0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f966161e640 1 --2- 172.21.15.27:0/1787149374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0d3400 0x7f965c0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f966161e640 1 -- 172.21.15.27:0/1787149374 >> v1:172.21.15.27:6789/0 conn(0x7f965c071540 legacy=0x7f965c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f966161e640 1 -- 172.21.15.27:0/1787149374 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c0d2920 con 0x7f965c0b2b50 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f965b7fe640 1 -- 172.21.15.27:0/1787149374 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121334948 0 0) 0x7f965c0ce610 con 0x7f965c071540 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.523+0000 7f965bfff640 1 --2- 172.21.15.27:0/1787149374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0d3400 0x7f965c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:48.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.524+0000 7f966161e640 1 --2- 172.21.15.27:0/1787149374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c0b2b50 0x7f965c0b6e20 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f9654009e50 tx=0x7f965402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a83c10a15de47bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.524+0000 7f965b7fe640 1 -- 172.21.15.27:0/1787149374 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965402cb00 con 0x7f965c0b2b50 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.524+0000 7f965b7fe640 1 -- 172.21.15.27:0/1787149374 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965402cca0 con 0x7f965c0b2b50 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.524+0000 7f965b7fe640 1 -- 172.21.15.27:0/1787149374 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9654034610 con 0x7f965c0b2b50 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.525+0000 7f9663081640 1 -- 172.21.15.27:0/1787149374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c0b2b50 msgr2=0x7f965c0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.525+0000 7f9663081640 1 --2- 172.21.15.27:0/1787149374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c0b2b50 0x7f965c0b6e20 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f9654009e50 tx=0x7f965402bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.525+0000 7f9663081640 1 -- 172.21.15.27:0/1787149374 shutdown_connections 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.525+0000 7f9663081640 1 --2- 172.21.15.27:0/1787149374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c0b2b50 0x7f965c0b6e20 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.525+0000 7f9663081640 1 --2- 172.21.15.27:0/1787149374 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0d3400 0x7f965c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.525+0000 7f9663081640 1 -- 172.21.15.27:0/1787149374 >> 172.21.15.27:0/1787149374 conn(0x7f965c0d3ca0 msgr2=0x7f965c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.525+0000 7f9663081640 1 -- 172.21.15.27:0/1787149374 shutdown_connections 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.525+0000 7f9663081640 1 -- 172.21.15.27:0/1787149374 wait complete. 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.526+0000 7f9663081640 1 Processor -- start 2024-03-06T07:31:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.526+0000 7f9663081640 1 -- start start 2024-03-06T07:31:48.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f9663081640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c071540 0x7f965c14d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f9663081640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0b2b50 0x7f965c0aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f9663081640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c0d3400 0x7f965c152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f9663081640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c15e370 con 0x7f965c071540 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f9663081640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c0ce8f0 con 0x7f965c0d3400 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f9663081640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c1518b0 con 0x7f965c0b2b50 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f9660e1d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c071540 0x7f965c14d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f9660e1d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c071540 0x7f965c14d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36602/0 (socket says 172.21.15.27:36602) 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f9660e1d640 1 -- 172.21.15.27:0/2234910490 learned_addr learned my addr 172.21.15.27:0/2234910490 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f965bfff640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0b2b50 0x7f965c0aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.527+0000 7f966161e640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c0d3400 0x7f965c152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f9660e1d640 1 -- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0b2b50 msgr2=0x7f965c0aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f9660e1d640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0b2b50 0x7f965c0aa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f9660e1d640 1 -- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c0d3400 msgr2=0x7f965c152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f9660e1d640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c0d3400 0x7f965c152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f9660e1d640 1 -- 172.21.15.27:0/2234910490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c151b90 con 0x7f965c071540 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f966161e640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c0d3400 0x7f965c152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f9660e1d640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c071540 0x7f965c14d140 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f965000c500 tx=0x7f965000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f96597fa640 1 -- 172.21.15.27:0/2234910490 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9650018070 con 0x7f965c071540 2024-03-06T07:31:48.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f96597fa640 1 -- 172.21.15.27:0/2234910490 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9650007580 con 0x7f965c071540 2024-03-06T07:31:48.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965c151e70 con 0x7f965c071540 2024-03-06T07:31:48.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f96597fa640 1 -- 172.21.15.27:0/2234910490 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9650004a20 con 0x7f965c071540 2024-03-06T07:31:48.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.528+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c152430 con 0x7f965c071540 2024-03-06T07:31:48.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.529+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f965c068700 con 0x7f965c071540 2024-03-06T07:31:48.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.530+0000 7f96597fa640 1 -- 172.21.15.27:0/2234910490 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f965001b070 con 0x7f965c071540 2024-03-06T07:31:48.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.531+0000 7f96597fa640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964006b4d0 0x7f964006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.531+0000 7f96597fa640 1 -- 172.21.15.27:0/2234910490 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96500894a0 con 0x7f965c071540 2024-03-06T07:31:48.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.531+0000 7f965bfff640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964006b4d0 0x7f964006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.531+0000 7f965bfff640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964006b4d0 0x7f964006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f965c0b6030 tx=0x7f9648008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.535+0000 7f96597fa640 1 -- 172.21.15.27:0/2234910490 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96500583b0 con 0x7f965c071540 2024-03-06T07:31:48.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.722+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f965c0cbe20 con 0x7f965c071540 2024-03-06T07:31:48.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.723+0000 7f96597fa640 1 -- 172.21.15.27:0/2234910490 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+16607 (secure 0 0 0) 0x7f9650057cb0 con 0x7f965c071540 2024-03-06T07:31:48.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:48.725 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":145,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:31:46.023504+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:48.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964006b4d0 msgr2=0x7f964006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964006b4d0 0x7f964006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f965c0b6030 tx=0x7f9648008040 comp rx=0 tx=0).stop 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c071540 msgr2=0x7f965c14d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c071540 0x7f965c14d140 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f965000c500 tx=0x7f965000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 shutdown_connections 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f964006b4d0 0x7f964006d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965c0d3400 0x7f965c152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f965c0b2b50 0x7f965c0aa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 --2- 172.21.15.27:0/2234910490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f965c071540 0x7f965c14d140 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 >> 172.21.15.27:0/2234910490 conn(0x7f965c0d3ca0 msgr2=0x7f965c0b8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 shutdown_connections 2024-03-06T07:31:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.725+0000 7f9663081640 1 -- 172.21.15.27:0/2234910490 wait complete. 2024-03-06T07:31:48.738 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:31:48.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:48.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.844+0000 7fe2e3c6b640 1 Processor -- start 2024-03-06T07:31:48.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.845+0000 7fe2e3c6b640 1 -- start start 2024-03-06T07:31:48.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.845+0000 7fe2e3c6b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc071540 0x7fe2dc0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.845+0000 7fe2e3c6b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2dc0b2b50 0x7fe2dc0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.845+0000 7fe2e3c6b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0ce610 con 0x7fe2dc0d3400 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.845+0000 7fe2e3c6b640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc0d2f70 con 0x7fe2dc0b2b50 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.845+0000 7fe2e3c6b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0d4be0 con 0x7fe2dc071540 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.846+0000 7fe2e2208640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2dc0b2b50 0x7fe2dc0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.846+0000 7fe2e2208640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2dc0b2b50 0x7fe2dc0b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36606/0 (socket says 172.21.15.27:36606) 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.846+0000 7fe2e2208640 1 -- 172.21.15.27:0/458287146 learned_addr learned my addr 172.21.15.27:0/458287146 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.846+0000 7fe2e1a07640 1 --2- 172.21.15.27:0/458287146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc071540 0x7fe2dc0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.846+0000 7fe2e2208640 1 -- 172.21.15.27:0/458287146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc071540 msgr2=0x7fe2dc0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.846+0000 7fe2e2208640 1 --2- 172.21.15.27:0/458287146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc071540 0x7fe2dc0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.846+0000 7fe2e2208640 1 -- 172.21.15.27:0/458287146 >> v1:172.21.15.27:6789/0 conn(0x7fe2dc0d3400 legacy=0x7fe2dc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.846+0000 7fe2e2208640 1 -- 172.21.15.27:0/458287146 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0d2920 con 0x7fe2dc0b2b50 2024-03-06T07:31:48.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.846+0000 7fe2e1a07640 1 --2- 172.21.15.27:0/458287146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc071540 0x7fe2dc0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e2208640 1 --2- 172.21.15.27:0/458287146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc0b2b50 0x7fe2dc0b6e20 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009e50 tx=0x7fe2d402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a679ffd18d2edf03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e0a05640 1 -- 172.21.15.27:0/458287146 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d402cb00 con 0x7fe2dc0b2b50 2024-03-06T07:31:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e0a05640 1 -- 172.21.15.27:0/458287146 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d402cca0 con 0x7fe2dc0b2b50 2024-03-06T07:31:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e0a05640 1 -- 172.21.15.27:0/458287146 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4034610 con 0x7fe2dc0b2b50 2024-03-06T07:31:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/458287146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc0b2b50 msgr2=0x7fe2dc0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e3c6b640 1 --2- 172.21.15.27:0/458287146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc0b2b50 0x7fe2dc0b6e20 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009e50 tx=0x7fe2d402bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/458287146 shutdown_connections 2024-03-06T07:31:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e3c6b640 1 --2- 172.21.15.27:0/458287146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc0b2b50 0x7fe2dc0b6e20 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e3c6b640 1 --2- 172.21.15.27:0/458287146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc071540 0x7fe2dc0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.847+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/458287146 >> 172.21.15.27:0/458287146 conn(0x7fe2dc0d3ca0 msgr2=0x7fe2dc0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.848+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/458287146 shutdown_connections 2024-03-06T07:31:48.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.848+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/458287146 wait complete. 2024-03-06T07:31:48.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.849+0000 7fe2e3c6b640 1 Processor -- start 2024-03-06T07:31:48.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.849+0000 7fe2e3c6b640 1 -- start start 2024-03-06T07:31:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.849+0000 7fe2e3c6b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc071540 0x7fe2dc152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.849+0000 7fe2e3c6b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0b2b50 0x7fe2dc0c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.849+0000 7fe2e3c6b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0d3400 0x7fe2dc0c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.849+0000 7fe2e3c6b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc15e370 con 0x7fe2dc071540 2024-03-06T07:31:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.849+0000 7fe2e3c6b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0ce8f0 con 0x7fe2dc0d3400 2024-03-06T07:31:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.849+0000 7fe2e3c6b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0c8130 con 0x7fe2dc0b2b50 2024-03-06T07:31:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1a07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc071540 0x7fe2dc152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1a07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc071540 0x7fe2dc152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36608/0 (socket says 172.21.15.27:36608) 2024-03-06T07:31:48.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1a07640 1 -- 172.21.15.27:0/1789166602 learned_addr learned my addr 172.21.15.27:0/1789166602 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1206640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0b2b50 0x7fe2dc0c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e2208640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0d3400 0x7fe2dc0c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1a07640 1 -- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0b2b50 msgr2=0x7fe2dc0c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1a07640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0b2b50 0x7fe2dc0c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1a07640 1 -- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0d3400 msgr2=0x7fe2dc0c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1a07640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0d3400 0x7fe2dc0c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1a07640 1 -- 172.21.15.27:0/1789166602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0c8410 con 0x7fe2dc071540 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e1a07640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc071540 0x7fe2dc152260 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800e7e0 tx=0x7fe2d800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2d27fc640 1 -- 172.21.15.27:0/1789166602 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d800f040 con 0x7fe2dc071540 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2d27fc640 1 -- 172.21.15.27:0/1789166602 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d8004030 con 0x7fe2dc071540 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc0c86f0 con 0x7fe2dc071540 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2d27fc640 1 -- 172.21.15.27:0/1789166602 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8004400 con 0x7fe2dc071540 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.850+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc0c7e00 con 0x7fe2dc071540 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.851+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc068700 con 0x7fe2dc071540 2024-03-06T07:31:48.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.852+0000 7fe2d27fc640 1 -- 172.21.15.27:0/1789166602 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2d800c6d0 con 0x7fe2dc071540 2024-03-06T07:31:48.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.852+0000 7fe2d27fc640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2c006b4d0 0x7fe2c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.852+0000 7fe2d27fc640 1 -- 172.21.15.27:0/1789166602 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2d8089f70 con 0x7fe2dc071540 2024-03-06T07:31:48.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.852+0000 7fe2e1206640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2c006b4d0 0x7fe2c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.853+0000 7fe2e1206640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2c006b4d0 0x7fe2c006d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc007fd0 tx=0x7fe2cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:48.856+0000 7fe2d27fc640 1 -- 172.21.15.27:0/1789166602 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2d8017020 con 0x7fe2dc071540 2024-03-06T07:31:49.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.035+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2dc0cf040 con 0x7fe2c006b4d0 2024-03-06T07:31:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.041+0000 7fe2d27fc640 1 -- 172.21.15.27:0/1789166602 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251666 (secure 0 0 0) 0x7fe2dc0cf040 con 0x7fe2c006b4d0 2024-03-06T07:31:49.046 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:49.047 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:31:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2c006b4d0 msgr2=0x7fe2c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2c006b4d0 0x7fe2c006d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc007fd0 tx=0x7fe2cc009040 comp rx=0 tx=0).stop 2024-03-06T07:31:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc071540 msgr2=0x7fe2dc152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc071540 0x7fe2dc152260 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800e7e0 tx=0x7fe2d800eca0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 shutdown_connections 2024-03-06T07:31:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe2c006b4d0 0x7fe2c006d990 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2dc0d3400 0x7fe2dc0c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2dc0b2b50 0x7fe2dc0c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 --2- 172.21.15.27:0/1789166602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2dc071540 0x7fe2dc152260 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 >> 172.21.15.27:0/1789166602 conn(0x7fe2dc0d3ca0 msgr2=0x7fe2dc0b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 shutdown_connections 2024-03-06T07:31:49.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.044+0000 7fe2e3c6b640 1 -- 172.21.15.27:0/1789166602 wait complete. 2024-03-06T07:31:49.057 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-03-06T07:31:48.119006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":166,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331160,"kb_used_data":15144,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643560,"statfs":{"total":773094113280,"available":772755005440,"internally_reserved":0,"allocated":15507456,"data_stored":7433560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.109582"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046476+0000","last_change":"2024-03-06T07:31:45.046476+0000","last_active":"2024-03-06T07:31:45.046476+0000","last_peered":"2024-03-06T07:31:45.046476+0000","last_clean":"2024-03-06T07:31:45.046476+0000","last_became_active":"2024-03-06T07:31:45.046179+0000","last_became_peered":"2024-03-06T07:31:45.046179+0000","last_unstale":"2024-03-06T07:31:45.046476+0000","last_undegraded":"2024-03-06T07:31:45.046476+0000","last_fullsized":"2024-03-06T07:31:45.046476+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:28.329642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041710+0000","last_change":"2024-03-06T07:31:45.041710+0000","last_active":"2024-03-06T07:31:45.041710+0000","last_peered":"2024-03-06T07:31:45.041710+0000","last_clean":"2024-03-06T07:31:45.041710+0000","last_became_active":"2024-03-06T07:31:45.041413+0000","last_became_peered":"2024-03-06T07:31:45.041413+0000","last_unstale":"2024-03-06T07:31:45.041710+0000","last_undegraded":"2024-03-06T07:31:45.041710+0000","last_fullsized":"2024-03-06T07:31:45.041710+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:53.307481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.048666+0000","last_change":"2024-03-06T07:31:45.048666+0000","last_active":"2024-03-06T07:31:45.048666+0000","last_peered":"2024-03-06T07:31:45.048666+0000","last_clean":"2024-03-06T07:31:45.048666+0000","last_became_active":"2024-03-06T07:31:45.044404+0000","last_became_peered":"2024-03-06T07:31:45.044404+0000","last_unstale":"2024-03-06T07:31:45.048666+0000","last_undegraded":"2024-03-06T07:31:45.048666+0000","last_fullsized":"2024-03-06T07:31:45.048666+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:55.936260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046599+0000","last_change":"2024-03-06T07:31:45.046599+0000","last_active":"2024-03-06T07:31:45.046599+0000","last_peered":"2024-03-06T07:31:45.046599+0000","last_clean":"2024-03-06T07:31:45.046599+0000","last_became_active":"2024-03-06T07:31:45.046251+0000","last_became_peered":"2024-03-06T07:31:45.046251+0000","last_unstale":"2024-03-06T07:31:45.046599+0000","last_undegraded":"2024-03-06T07:31:45.046599+0000","last_fullsized":"2024-03-06T07:31:45.046599+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:13.567550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050368+0000","last_change":"2024-03-06T07:31:45.048513+0000","last_active":"2024-03-06T07:31:45.050368+0000","last_peered":"2024-03-06T07:31:45.050368+0000","last_clean":"2024-03-06T07:31:45.050368+0000","last_became_active":"2024-03-06T07:31:45.048251+0000","last_became_peered":"2024-03-06T07:31:45.048251+0000","last_unstale":"2024-03-06T07:31:45.050368+0000","last_undegraded":"2024-03-06T07:31:45.050368+0000","last_fullsized":"2024-03-06T07:31:45.050368+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:04.808270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.029936+0000","last_change":"2024-03-06T07:31:45.029936+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.029936+0000","last_undegraded":"2024-03-06T07:31:45.029936+0000","last_fullsized":"2024-03-06T07:31:45.029936+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040291+0000","last_change":"2024-03-06T07:31:45.035876+0000","last_active":"2024-03-06T07:31:45.040291+0000","last_peered":"2024-03-06T07:31:45.040291+0000","last_clean":"2024-03-06T07:31:45.040291+0000","last_became_active":"2024-03-06T07:31:45.035604+0000","last_became_peered":"2024-03-06T07:31:45.035604+0000","last_unstale":"2024-03-06T07:31:45.040291+0000","last_undegraded":"2024-03-06T07:31:45.040291+0000","last_fullsized":"2024-03-06T07:31:45.040291+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:28.333006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037397+0000","last_change":"2024-03-06T07:31:45.035300+0000","last_active":"2024-03-06T07:31:45.037397+0000","last_peered":"2024-03-06T07:31:45.037397+0000","last_clean":"2024-03-06T07:31:45.037397+0000","last_became_active":"2024-03-06T07:31:45.034964+0000","last_became_peered":"2024-03-06T07:31:45.034964+0000","last_unstale":"2024-03-06T07:31:45.037397+0000","last_undegraded":"2024-03-06T07:31:45.037397+0000","last_fullsized":"2024-03-06T07:31:45.037397+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:43.063911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.039202+0000","last_change":"2024-03-06T07:31:45.039202+0000","last_active":"2024-03-06T07:31:45.039202+0000","last_peered":"2024-03-06T07:31:45.039202+0000","last_clean":"2024-03-06T07:31:45.039202+0000","last_became_active":"2024-03-06T07:31:45.038902+0000","last_became_peered":"2024-03-06T07:31:45.038902+0000","last_unstale":"2024-03-06T07:31:45.039202+0000","last_undegraded":"2024-03-06T07:31:45.039202+0000","last_fullsized":"2024-03-06T07:31:45.039202+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:44.736535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050634+0000","last_change":"2024-03-06T07:31:45.047007+0000","last_active":"2024-03-06T07:31:45.050634+0000","last_peered":"2024-03-06T07:31:45.050634+0000","last_clean":"2024-03-06T07:31:45.050634+0000","last_became_active":"2024-03-06T07:31:45.046533+0000","last_became_peered":"2024-03-06T07:31:45.046533+0000","last_unstale":"2024-03-06T07:31:45.050634+0000","last_undegraded":"2024-03-06T07:31:45.050634+0000","last_fullsized":"2024-03-06T07:31:45.050634+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:11.401916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037764+0000","last_change":"2024-03-06T07:31:45.036443+0000","last_active":"2024-03-06T07:31:45.037764+0000","last_peered":"2024-03-06T07:31:45.037764+0000","last_clean":"2024-03-06T07:31:45.037764+0000","last_became_active":"2024-03-06T07:31:45.035986+0000","last_became_peered":"2024-03-06T07:31:45.035986+0000","last_unstale":"2024-03-06T07:31:45.037764+0000","last_undegraded":"2024-03-06T07:31:45.037764+0000","last_fullsized":"2024-03-06T07:31:45.037764+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.690561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.051142+0000","last_change":"2024-03-06T07:31:45.049645+0000","last_active":"2024-03-06T07:31:45.051142+0000","last_peered":"2024-03-06T07:31:45.051142+0000","last_clean":"2024-03-06T07:31:45.051142+0000","last_became_active":"2024-03-06T07:31:45.048865+0000","last_became_peered":"2024-03-06T07:31:45.048865+0000","last_unstale":"2024-03-06T07:31:45.051142+0000","last_undegraded":"2024-03-06T07:31:45.051142+0000","last_fullsized":"2024-03-06T07:31:45.051142+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:36.725280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050346+0000","last_change":"2024-03-06T07:31:45.048451+0000","last_active":"2024-03-06T07:31:45.050346+0000","last_peered":"2024-03-06T07:31:45.050346+0000","last_clean":"2024-03-06T07:31:45.050346+0000","last_became_active":"2024-03-06T07:31:45.048188+0000","last_became_peered":"2024-03-06T07:31:45.048188+0000","last_unstale":"2024-03-06T07:31:45.050346+0000","last_undegraded":"2024-03-06T07:31:45.050346+0000","last_fullsized":"2024-03-06T07:31:45.050346+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:17.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044378+0000","last_change":"2024-03-06T07:31:45.042722+0000","last_active":"2024-03-06T07:31:45.044378+0000","last_peered":"2024-03-06T07:31:45.044378+0000","last_clean":"2024-03-06T07:31:45.044378+0000","last_became_active":"2024-03-06T07:31:45.042456+0000","last_became_peered":"2024-03-06T07:31:45.042456+0000","last_unstale":"2024-03-06T07:31:45.044378+0000","last_undegraded":"2024-03-06T07:31:45.044378+0000","last_fullsized":"2024-03-06T07:31:45.044378+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:52.796540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046120+0000","last_change":"2024-03-06T07:31:45.046120+0000","last_active":"2024-03-06T07:31:45.046120+0000","last_peered":"2024-03-06T07:31:45.046120+0000","last_clean":"2024-03-06T07:31:45.046120+0000","last_became_active":"2024-03-06T07:31:45.045855+0000","last_became_peered":"2024-03-06T07:31:45.045855+0000","last_unstale":"2024-03-06T07:31:45.046120+0000","last_undegraded":"2024-03-06T07:31:45.046120+0000","last_fullsized":"2024-03-06T07:31:45.046120+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:21.522756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042236+0000","last_change":"2024-03-06T07:31:45.039353+0000","last_active":"2024-03-06T07:31:45.042236+0000","last_peered":"2024-03-06T07:31:45.042236+0000","last_clean":"2024-03-06T07:31:45.042236+0000","last_became_active":"2024-03-06T07:31:45.039090+0000","last_became_peered":"2024-03-06T07:31:45.039090+0000","last_unstale":"2024-03-06T07:31:45.042236+0000","last_undegraded":"2024-03-06T07:31:45.042236+0000","last_fullsized":"2024-03-06T07:31:45.042236+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:13.414833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.042465+0000","last_change":"2024-03-06T07:31:45.042465+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.042465+0000","last_undegraded":"2024-03-06T07:31:45.042465+0000","last_fullsized":"2024-03-06T07:31:45.042465+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046112+0000","last_change":"2024-03-06T07:31:45.046112+0000","last_active":"2024-03-06T07:31:45.046112+0000","last_peered":"2024-03-06T07:31:45.046112+0000","last_clean":"2024-03-06T07:31:45.046112+0000","last_became_active":"2024-03-06T07:31:45.043698+0000","last_became_peered":"2024-03-06T07:31:45.043698+0000","last_unstale":"2024-03-06T07:31:45.046112+0000","last_undegraded":"2024-03-06T07:31:45.046112+0000","last_fullsized":"2024-03-06T07:31:45.046112+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:14.807136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.043083+0000","last_change":"2024-03-06T07:31:45.038374+0000","last_active":"2024-03-06T07:31:45.043083+0000","last_peered":"2024-03-06T07:31:45.043083+0000","last_clean":"2024-03-06T07:31:45.043083+0000","last_became_active":"2024-03-06T07:31:45.038088+0000","last_became_peered":"2024-03-06T07:31:45.038088+0000","last_unstale":"2024-03-06T07:31:45.043083+0000","last_undegraded":"2024-03-06T07:31:45.043083+0000","last_fullsized":"2024-03-06T07:31:45.043083+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:04.808270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034187+0000","last_change":"2024-03-06T07:31:45.032953+0000","last_active":"2024-03-06T07:31:45.034187+0000","last_peered":"2024-03-06T07:31:45.034187+0000","last_clean":"2024-03-06T07:31:45.034187+0000","last_became_active":"2024-03-06T07:31:45.032688+0000","last_became_peered":"2024-03-06T07:31:45.032688+0000","last_unstale":"2024-03-06T07:31:45.034187+0000","last_undegraded":"2024-03-06T07:31:45.034187+0000","last_fullsized":"2024-03-06T07:31:45.034187+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:02.496094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.049212+0000","last_change":"2024-03-06T07:31:45.049212+0000","last_active":"2024-03-06T07:31:45.049212+0000","last_peered":"2024-03-06T07:31:45.049212+0000","last_clean":"2024-03-06T07:31:45.049212+0000","last_became_active":"2024-03-06T07:31:45.047359+0000","last_became_peered":"2024-03-06T07:31:45.047359+0000","last_unstale":"2024-03-06T07:31:45.049212+0000","last_undegraded":"2024-03-06T07:31:45.049212+0000","last_fullsized":"2024-03-06T07:31:45.049212+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:05.783188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.038255+0000","last_change":"2024-03-06T07:31:45.035442+0000","last_active":"2024-03-06T07:31:45.038255+0000","last_peered":"2024-03-06T07:31:45.038255+0000","last_clean":"2024-03-06T07:31:45.038255+0000","last_became_active":"2024-03-06T07:31:45.035154+0000","last_became_peered":"2024-03-06T07:31:45.035154+0000","last_unstale":"2024-03-06T07:31:45.038255+0000","last_undegraded":"2024-03-06T07:31:45.038255+0000","last_fullsized":"2024-03-06T07:31:45.038255+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:27.623139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041778+0000","last_change":"2024-03-06T07:31:45.039599+0000","last_active":"2024-03-06T07:31:45.041778+0000","last_peered":"2024-03-06T07:31:45.041778+0000","last_clean":"2024-03-06T07:31:45.041778+0000","last_became_active":"2024-03-06T07:31:45.039297+0000","last_became_peered":"2024-03-06T07:31:45.039297+0000","last_unstale":"2024-03-06T07:31:45.041778+0000","last_undegraded":"2024-03-06T07:31:45.041778+0000","last_fullsized":"2024-03-06T07:31:45.041778+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:33.886298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":6,"reported_epoch":144,"state":"active","last_fresh":"2024-03-06T07:31:45.047191+0000","last_change":"2024-03-06T07:31:45.047127+0000","last_active":"2024-03-06T07:31:45.047191+0000","last_peered":"2024-03-06T07:31:45.047191+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"2024-03-06T07:31:45.047127+0000","last_became_peered":"2024-03-06T07:31:45.047127+0000","last_unstale":"2024-03-06T07:31:45.047191+0000","last_undegraded":"2024-03-06T07:31:45.047191+0000","last_fullsized":"2024-03-06T07:31:45.047191+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042283+0000","last_change":"2024-03-06T07:31:45.040040+0000","last_active":"2024-03-06T07:31:45.042283+0000","last_peered":"2024-03-06T07:31:45.042283+0000","last_clean":"2024-03-06T07:31:45.042283+0000","last_became_active":"2024-03-06T07:31:45.039753+0000","last_became_peered":"2024-03-06T07:31:45.039753+0000","last_unstale":"2024-03-06T07:31:45.042283+0000","last_undegraded":"2024-03-06T07:31:45.042283+0000","last_fullsized":"2024-03-06T07:31:45.042283+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:27.525036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042445+0000","last_change":"2024-03-06T07:31:45.037506+0000","last_active":"2024-03-06T07:31:45.042445+0000","last_peered":"2024-03-06T07:31:45.042445+0000","last_clean":"2024-03-06T07:31:45.042445+0000","last_became_active":"2024-03-06T07:31:45.036896+0000","last_became_peered":"2024-03-06T07:31:45.036896+0000","last_unstale":"2024-03-06T07:31:45.042445+0000","last_undegraded":"2024-03-06T07:31:45.042445+0000","last_fullsized":"2024-03-06T07:31:45.042445+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:17.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.025450+0000","last_change":"2024-03-06T07:31:45.025450+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.025450+0000","last_undegraded":"2024-03-06T07:31:45.025450+0000","last_fullsized":"2024-03-06T07:31:45.025450+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.028666+0000","last_change":"2024-03-06T07:31:45.028666+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.028666+0000","last_undegraded":"2024-03-06T07:31:45.028666+0000","last_fullsized":"2024-03-06T07:31:45.028666+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045636+0000","last_change":"2024-03-06T07:31:45.045636+0000","last_active":"2024-03-06T07:31:45.045636+0000","last_peered":"2024-03-06T07:31:45.045636+0000","last_clean":"2024-03-06T07:31:45.045636+0000","last_became_active":"2024-03-06T07:31:45.045309+0000","last_became_peered":"2024-03-06T07:31:45.045309+0000","last_unstale":"2024-03-06T07:31:45.045636+0000","last_undegraded":"2024-03-06T07:31:45.045636+0000","last_fullsized":"2024-03-06T07:31:45.045636+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:36.113249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044357+0000","last_change":"2024-03-06T07:31:45.041617+0000","last_active":"2024-03-06T07:31:45.044357+0000","last_peered":"2024-03-06T07:31:45.044357+0000","last_clean":"2024-03-06T07:31:45.044357+0000","last_became_active":"2024-03-06T07:31:45.040818+0000","last_became_peered":"2024-03-06T07:31:45.040818+0000","last_unstale":"2024-03-06T07:31:45.044357+0000","last_undegraded":"2024-03-06T07:31:45.044357+0000","last_fullsized":"2024-03-06T07:31:45.044357+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:00.686030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.029698+0000","last_change":"2024-03-06T07:31:45.029698+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.029698+0000","last_undegraded":"2024-03-06T07:31:45.029698+0000","last_fullsized":"2024-03-06T07:31:45.029698+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034138+0000","last_change":"2024-03-06T07:31:45.032208+0000","last_active":"2024-03-06T07:31:45.034138+0000","last_peered":"2024-03-06T07:31:45.034138+0000","last_clean":"2024-03-06T07:31:45.034138+0000","last_became_active":"2024-03-06T07:31:45.031935+0000","last_became_peered":"2024-03-06T07:31:45.031935+0000","last_unstale":"2024-03-06T07:31:45.034138+0000","last_undegraded":"2024-03-06T07:31:45.034138+0000","last_fullsized":"2024-03-06T07:31:45.034138+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:11.848515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050610+0000","last_change":"2024-03-06T07:31:45.046451+0000","last_active":"2024-03-06T07:31:45.050610+0000","last_peered":"2024-03-06T07:31:45.050610+0000","last_clean":"2024-03-06T07:31:45.050610+0000","last_became_active":"2024-03-06T07:31:45.045791+0000","last_became_peered":"2024-03-06T07:31:45.045791+0000","last_unstale":"2024-03-06T07:31:45.050610+0000","last_undegraded":"2024-03-06T07:31:45.050610+0000","last_fullsized":"2024-03-06T07:31:45.050610+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:53.307481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041764+0000","last_change":"2024-03-06T07:31:44.033293+0000","last_active":"2024-03-06T07:31:45.041764+0000","last_peered":"2024-03-06T07:31:45.041764+0000","last_clean":"2024-03-06T07:31:45.041764+0000","last_became_active":"2024-03-06T07:31:44.032831+0000","last_became_peered":"2024-03-06T07:31:44.032831+0000","last_unstale":"2024-03-06T07:31:45.041764+0000","last_undegraded":"2024-03-06T07:31:45.041764+0000","last_fullsized":"2024-03-06T07:31:45.041764+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:43.731816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040911+0000","last_change":"2024-03-06T07:31:45.040911+0000","last_active":"2024-03-06T07:31:45.040911+0000","last_peered":"2024-03-06T07:31:45.040911+0000","last_clean":"2024-03-06T07:31:45.040911+0000","last_became_active":"2024-03-06T07:31:45.040632+0000","last_became_peered":"2024-03-06T07:31:45.040632+0000","last_unstale":"2024-03-06T07:31:45.040911+0000","last_undegraded":"2024-03-06T07:31:45.040911+0000","last_fullsized":"2024-03-06T07:31:45.040911+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:21.522756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031190+0000","last_change":"2024-03-06T07:31:44.027157+0000","last_active":"2024-03-06T07:31:45.031190+0000","last_peered":"2024-03-06T07:31:45.031190+0000","last_clean":"2024-03-06T07:31:45.031190+0000","last_became_active":"2024-03-06T07:31:44.026865+0000","last_became_peered":"2024-03-06T07:31:44.026865+0000","last_unstale":"2024-03-06T07:31:45.031190+0000","last_undegraded":"2024-03-06T07:31:45.031190+0000","last_fullsized":"2024-03-06T07:31:45.031190+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042098+0000","last_change":"2024-03-06T07:31:45.042098+0000","last_active":"2024-03-06T07:31:45.042098+0000","last_peered":"2024-03-06T07:31:45.042098+0000","last_clean":"2024-03-06T07:31:45.042098+0000","last_became_active":"2024-03-06T07:31:45.041801+0000","last_became_peered":"2024-03-06T07:31:45.041801+0000","last_unstale":"2024-03-06T07:31:45.042098+0000","last_undegraded":"2024-03-06T07:31:45.042098+0000","last_fullsized":"2024-03-06T07:31:45.042098+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:48.091616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031296+0000","last_change":"2024-03-06T07:31:44.023697+0000","last_active":"2024-03-06T07:31:45.031296+0000","last_peered":"2024-03-06T07:31:45.031296+0000","last_clean":"2024-03-06T07:31:45.031296+0000","last_became_active":"2024-03-06T07:31:44.023399+0000","last_became_peered":"2024-03-06T07:31:44.023399+0000","last_unstale":"2024-03-06T07:31:45.031296+0000","last_undegraded":"2024-03-06T07:31:45.031296+0000","last_fullsized":"2024-03-06T07:31:45.031296+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:19.201492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050493+0000","last_change":"2024-03-06T07:31:45.046580+0000","last_active":"2024-03-06T07:31:45.050493+0000","last_peered":"2024-03-06T07:31:45.050493+0000","last_clean":"2024-03-06T07:31:45.050493+0000","last_became_active":"2024-03-06T07:31:45.046312+0000","last_became_peered":"2024-03-06T07:31:45.046312+0000","last_unstale":"2024-03-06T07:31:45.050493+0000","last_undegraded":"2024-03-06T07:31:45.050493+0000","last_fullsized":"2024-03-06T07:31:45.050493+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:20.046408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031181+0000","last_change":"2024-03-06T07:31:44.020513+0000","last_active":"2024-03-06T07:31:45.031181+0000","last_peered":"2024-03-06T07:31:45.031181+0000","last_clean":"2024-03-06T07:31:45.031181+0000","last_became_active":"2024-03-06T07:31:44.020196+0000","last_became_peered":"2024-03-06T07:31:44.020196+0000","last_unstale":"2024-03-06T07:31:45.031181+0000","last_undegraded":"2024-03-06T07:31:45.031181+0000","last_fullsized":"2024-03-06T07:31:45.031181+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:47.307708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.030083+0000","last_change":"2024-03-06T07:31:45.030083+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.030083+0000","last_undegraded":"2024-03-06T07:31:45.030083+0000","last_fullsized":"2024-03-06T07:31:45.030083+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025657+0000","last_change":"2024-03-06T07:31:44.033751+0000","last_active":"2024-03-06T07:31:45.025657+0000","last_peered":"2024-03-06T07:31:45.025657+0000","last_clean":"2024-03-06T07:31:45.025657+0000","last_became_active":"2024-03-06T07:31:44.033386+0000","last_became_peered":"2024-03-06T07:31:44.033386+0000","last_unstale":"2024-03-06T07:31:45.025657+0000","last_undegraded":"2024-03-06T07:31:45.025657+0000","last_fullsized":"2024-03-06T07:31:45.025657+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:50.356019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045822+0000","last_change":"2024-03-06T07:31:45.045822+0000","last_active":"2024-03-06T07:31:45.045822+0000","last_peered":"2024-03-06T07:31:45.045822+0000","last_clean":"2024-03-06T07:31:45.045822+0000","last_became_active":"2024-03-06T07:31:45.045552+0000","last_became_peered":"2024-03-06T07:31:45.045552+0000","last_unstale":"2024-03-06T07:31:45.045822+0000","last_undegraded":"2024-03-06T07:31:45.045822+0000","last_fullsized":"2024-03-06T07:31:45.045822+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:08.039630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"144'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.028727+0000","last_change":"2024-03-06T07:31:44.028730+0000","last_active":"2024-03-06T07:31:45.028727+0000","last_peered":"2024-03-06T07:31:45.028727+0000","last_clean":"2024-03-06T07:31:45.028727+0000","last_became_active":"2024-03-06T07:31:44.028446+0000","last_became_peered":"2024-03-06T07:31:44.028446+0000","last_unstale":"2024-03-06T07:31:45.028727+0000","last_undegraded":"2024-03-06T07:31:45.028727+0000","last_fullsized":"2024-03-06T07:31:45.028727+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:38.934563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.035147+0000","last_change":"2024-03-06T07:31:45.033665+0000","last_active":"2024-03-06T07:31:45.035147+0000","last_peered":"2024-03-06T07:31:45.035147+0000","last_clean":"2024-03-06T07:31:45.035147+0000","last_became_active":"2024-03-06T07:31:45.033369+0000","last_became_peered":"2024-03-06T07:31:45.033369+0000","last_unstale":"2024-03-06T07:31:45.035147+0000","last_undegraded":"2024-03-06T07:31:45.035147+0000","last_fullsized":"2024-03-06T07:31:45.035147+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:45.323683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025219+0000","last_change":"2024-03-06T07:31:44.024794+0000","last_active":"2024-03-06T07:31:45.025219+0000","last_peered":"2024-03-06T07:31:45.025219+0000","last_clean":"2024-03-06T07:31:45.025219+0000","last_became_active":"2024-03-06T07:31:44.024499+0000","last_became_peered":"2024-03-06T07:31:44.024499+0000","last_unstale":"2024-03-06T07:31:45.025219+0000","last_undegraded":"2024-03-06T07:31:45.025219+0000","last_fullsized":"2024-03-06T07:31:45.025219+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:01.912662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042985+0000","last_change":"2024-03-06T07:31:45.040904+0000","last_active":"2024-03-06T07:31:45.042985+0000","last_peered":"2024-03-06T07:31:45.042985+0000","last_clean":"2024-03-06T07:31:45.042985+0000","last_became_active":"2024-03-06T07:31:45.040606+0000","last_became_peered":"2024-03-06T07:31:45.040606+0000","last_unstale":"2024-03-06T07:31:45.042985+0000","last_undegraded":"2024-03-06T07:31:45.042985+0000","last_fullsized":"2024-03-06T07:31:45.042985+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:18.363428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031359+0000","last_change":"2024-03-06T07:31:44.022269+0000","last_active":"2024-03-06T07:31:45.031359+0000","last_peered":"2024-03-06T07:31:45.031359+0000","last_clean":"2024-03-06T07:31:45.031359+0000","last_became_active":"2024-03-06T07:31:44.021982+0000","last_became_peered":"2024-03-06T07:31:44.021982+0000","last_unstale":"2024-03-06T07:31:45.031359+0000","last_undegraded":"2024-03-06T07:31:45.031359+0000","last_fullsized":"2024-03-06T07:31:45.031359+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:51.573386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.039591+0000","last_change":"2024-03-06T07:31:45.037434+0000","last_active":"2024-03-06T07:31:45.039591+0000","last_peered":"2024-03-06T07:31:45.039591+0000","last_clean":"2024-03-06T07:31:45.039591+0000","last_became_active":"2024-03-06T07:31:45.037148+0000","last_became_peered":"2024-03-06T07:31:45.037148+0000","last_unstale":"2024-03-06T07:31:45.039591+0000","last_undegraded":"2024-03-06T07:31:45.039591+0000","last_fullsized":"2024-03-06T07:31:45.039591+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:40.063434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031021+0000","last_change":"2024-03-06T07:31:44.019467+0000","last_active":"2024-03-06T07:31:45.031021+0000","last_peered":"2024-03-06T07:31:45.031021+0000","last_clean":"2024-03-06T07:31:45.031021+0000","last_became_active":"2024-03-06T07:31:44.019183+0000","last_became_peered":"2024-03-06T07:31:44.019183+0000","last_unstale":"2024-03-06T07:31:45.031021+0000","last_undegraded":"2024-03-06T07:31:45.031021+0000","last_fullsized":"2024-03-06T07:31:45.031021+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:22.325138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":6,"reported_epoch":144,"state":"active","last_fresh":"2024-03-06T07:31:45.052295+0000","last_change":"2024-03-06T07:31:45.052223+0000","last_active":"2024-03-06T07:31:45.052295+0000","last_peered":"2024-03-06T07:31:45.052295+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"2024-03-06T07:31:45.052223+0000","last_became_peered":"2024-03-06T07:31:45.052223+0000","last_unstale":"2024-03-06T07:31:45.052295+0000","last_undegraded":"2024-03-06T07:31:45.052295+0000","last_fullsized":"2024-03-06T07:31:45.052295+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041417+0000","last_change":"2024-03-06T07:31:44.031262+0000","last_active":"2024-03-06T07:31:45.041417+0000","last_peered":"2024-03-06T07:31:45.041417+0000","last_clean":"2024-03-06T07:31:45.041417+0000","last_became_active":"2024-03-06T07:31:44.030401+0000","last_became_peered":"2024-03-06T07:31:44.030401+0000","last_unstale":"2024-03-06T07:31:45.041417+0000","last_undegraded":"2024-03-06T07:31:45.041417+0000","last_fullsized":"2024-03-06T07:31:45.041417+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:44.318964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.035328+0000","last_change":"2024-03-06T07:31:45.033444+0000","last_active":"2024-03-06T07:31:45.035328+0000","last_peered":"2024-03-06T07:31:45.035328+0000","last_clean":"2024-03-06T07:31:45.035328+0000","last_became_active":"2024-03-06T07:31:45.033109+0000","last_became_peered":"2024-03-06T07:31:45.033109+0000","last_unstale":"2024-03-06T07:31:45.035328+0000","last_undegraded":"2024-03-06T07:31:45.035328+0000","last_fullsized":"2024-03-06T07:31:45.035328+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:45.246725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"144'5","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029309+0000","last_change":"2024-03-06T07:31:44.026781+0000","last_active":"2024-03-06T07:31:45.029309+0000","last_peered":"2024-03-06T07:31:45.029309+0000","last_clean":"2024-03-06T07:31:45.029309+0000","last_became_active":"2024-03-06T07:31:44.026495+0000","last_became_peered":"2024-03-06T07:31:44.026495+0000","last_unstale":"2024-03-06T07:31:45.029309+0000","last_undegraded":"2024-03-06T07:31:45.029309+0000","last_fullsized":"2024-03-06T07:31:45.029309+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:35.090123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.043899+0000","last_change":"2024-03-06T07:31:45.043899+0000","last_active":"2024-03-06T07:31:45.043899+0000","last_peered":"2024-03-06T07:31:45.043899+0000","last_clean":"2024-03-06T07:31:45.043899+0000","last_became_active":"2024-03-06T07:31:45.043131+0000","last_became_peered":"2024-03-06T07:31:45.043131+0000","last_unstale":"2024-03-06T07:31:45.043899+0000","last_undegraded":"2024-03-06T07:31:45.043899+0000","last_fullsized":"2024-03-06T07:31:45.043899+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:13.567550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.035787+0000","last_change":"2024-03-06T07:31:44.026616+0000","last_active":"2024-03-06T07:31:45.035787+0000","last_peered":"2024-03-06T07:31:45.035787+0000","last_clean":"2024-03-06T07:31:45.035787+0000","last_became_active":"2024-03-06T07:31:44.025510+0000","last_became_peered":"2024-03-06T07:31:44.025510+0000","last_unstale":"2024-03-06T07:31:45.035787+0000","last_undegraded":"2024-03-06T07:31:45.035787+0000","last_fullsized":"2024-03-06T07:31:45.035787+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:13.314073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041996+0000","last_change":"2024-03-06T07:31:45.041996+0000","last_active":"2024-03-06T07:31:45.041996+0000","last_peered":"2024-03-06T07:31:45.041996+0000","last_clean":"2024-03-06T07:31:45.041996+0000","last_became_active":"2024-03-06T07:31:45.041403+0000","last_became_peered":"2024-03-06T07:31:45.041403+0000","last_unstale":"2024-03-06T07:31:45.041996+0000","last_undegraded":"2024-03-06T07:31:45.041996+0000","last_fullsized":"2024-03-06T07:31:45.041996+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:20.046408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.032932+0000","last_change":"2024-03-06T07:31:44.022857+0000","last_active":"2024-03-06T07:31:45.032932+0000","last_peered":"2024-03-06T07:31:45.032932+0000","last_clean":"2024-03-06T07:31:45.032932+0000","last_became_active":"2024-03-06T07:31:44.022400+0000","last_became_peered":"2024-03-06T07:31:44.022400+0000","last_unstale":"2024-03-06T07:31:45.032932+0000","last_undegraded":"2024-03-06T07:31:45.032932+0000","last_fullsized":"2024-03-06T07:31:45.032932+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:55.478147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044454+0000","last_change":"2024-03-06T07:31:45.042751+0000","last_active":"2024-03-06T07:31:45.044454+0000","last_peered":"2024-03-06T07:31:45.044454+0000","last_clean":"2024-03-06T07:31:45.044454+0000","last_became_active":"2024-03-06T07:31:45.042477+0000","last_became_peered":"2024-03-06T07:31:45.042477+0000","last_unstale":"2024-03-06T07:31:45.044454+0000","last_undegraded":"2024-03-06T07:31:45.044454+0000","last_fullsized":"2024-03-06T07:31:45.044454+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:43.063911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029362+0000","last_change":"2024-03-06T07:31:44.023105+0000","last_active":"2024-03-06T07:31:45.029362+0000","last_peered":"2024-03-06T07:31:45.029362+0000","last_clean":"2024-03-06T07:31:45.029362+0000","last_became_active":"2024-03-06T07:31:44.022808+0000","last_became_peered":"2024-03-06T07:31:44.022808+0000","last_unstale":"2024-03-06T07:31:45.029362+0000","last_undegraded":"2024-03-06T07:31:45.029362+0000","last_fullsized":"2024-03-06T07:31:45.029362+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.047987+0000","last_change":"2024-03-06T07:31:45.047987+0000","last_active":"2024-03-06T07:31:45.047987+0000","last_peered":"2024-03-06T07:31:45.047987+0000","last_clean":"2024-03-06T07:31:45.047987+0000","last_became_active":"2024-03-06T07:31:45.045672+0000","last_became_peered":"2024-03-06T07:31:45.045672+0000","last_unstale":"2024-03-06T07:31:45.047987+0000","last_undegraded":"2024-03-06T07:31:45.047987+0000","last_fullsized":"2024-03-06T07:31:45.047987+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:38.092312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025614+0000","last_change":"2024-03-06T07:31:44.024742+0000","last_active":"2024-03-06T07:31:45.025614+0000","last_peered":"2024-03-06T07:31:45.025614+0000","last_clean":"2024-03-06T07:31:45.025614+0000","last_became_active":"2024-03-06T07:31:44.024445+0000","last_became_peered":"2024-03-06T07:31:44.024445+0000","last_unstale":"2024-03-06T07:31:45.025614+0000","last_undegraded":"2024-03-06T07:31:45.025614+0000","last_fullsized":"2024-03-06T07:31:45.025614+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:00.838367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":6,"reported_epoch":144,"state":"active","last_fresh":"2024-03-06T07:31:45.051855+0000","last_change":"2024-03-06T07:31:45.051795+0000","last_active":"2024-03-06T07:31:45.051855+0000","last_peered":"2024-03-06T07:31:45.051855+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"2024-03-06T07:31:45.051795+0000","last_became_peered":"2024-03-06T07:31:45.051795+0000","last_unstale":"2024-03-06T07:31:45.051855+0000","last_undegraded":"2024-03-06T07:31:45.051855+0000","last_fullsized":"2024-03-06T07:31:45.051855+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"144'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029593+0000","last_change":"2024-03-06T07:31:44.023389+0000","last_active":"2024-03-06T07:31:45.029593+0000","last_peered":"2024-03-06T07:31:45.029593+0000","last_clean":"2024-03-06T07:31:45.029593+0000","last_became_active":"2024-03-06T07:31:44.023124+0000","last_became_peered":"2024-03-06T07:31:44.023124+0000","last_unstale":"2024-03-06T07:31:45.029593+0000","last_undegraded":"2024-03-06T07:31:45.029593+0000","last_fullsized":"2024-03-06T07:31:45.029593+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044088+0000","last_change":"2024-03-06T07:31:45.044088+0000","last_active":"2024-03-06T07:31:45.044088+0000","last_peered":"2024-03-06T07:31:45.044088+0000","last_clean":"2024-03-06T07:31:45.044088+0000","last_became_active":"2024-03-06T07:31:45.043830+0000","last_became_peered":"2024-03-06T07:31:45.043830+0000","last_unstale":"2024-03-06T07:31:45.044088+0000","last_undegraded":"2024-03-06T07:31:45.044088+0000","last_fullsized":"2024-03-06T07:31:45.044088+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:40.063434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031115+0000","last_change":"2024-03-06T07:31:44.024253+0000","last_active":"2024-03-06T07:31:45.031115+0000","last_peered":"2024-03-06T07:31:45.031115+0000","last_clean":"2024-03-06T07:31:45.031115+0000","last_became_active":"2024-03-06T07:31:44.023842+0000","last_became_peered":"2024-03-06T07:31:44.023842+0000","last_unstale":"2024-03-06T07:31:45.031115+0000","last_undegraded":"2024-03-06T07:31:45.031115+0000","last_fullsized":"2024-03-06T07:31:45.031115+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045694+0000","last_change":"2024-03-06T07:31:45.045694+0000","last_active":"2024-03-06T07:31:45.045694+0000","last_peered":"2024-03-06T07:31:45.045694+0000","last_clean":"2024-03-06T07:31:45.045694+0000","last_became_active":"2024-03-06T07:31:45.044744+0000","last_became_peered":"2024-03-06T07:31:45.044744+0000","last_unstale":"2024-03-06T07:31:45.045694+0000","last_undegraded":"2024-03-06T07:31:45.045694+0000","last_fullsized":"2024-03-06T07:31:45.045694+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:42.160169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025821+0000","last_change":"2024-03-06T07:31:44.035542+0000","last_active":"2024-03-06T07:31:45.025821+0000","last_peered":"2024-03-06T07:31:45.025821+0000","last_clean":"2024-03-06T07:31:45.025821+0000","last_became_active":"2024-03-06T07:31:44.035318+0000","last_became_peered":"2024-03-06T07:31:44.035318+0000","last_unstale":"2024-03-06T07:31:45.025821+0000","last_undegraded":"2024-03-06T07:31:45.025821+0000","last_fullsized":"2024-03-06T07:31:45.025821+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:03.773449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.029060+0000","last_change":"2024-03-06T07:31:45.029060+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.029060+0000","last_undegraded":"2024-03-06T07:31:45.029060+0000","last_fullsized":"2024-03-06T07:31:45.029060+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.030954+0000","last_change":"2024-03-06T07:31:44.028091+0000","last_active":"2024-03-06T07:31:45.030954+0000","last_peered":"2024-03-06T07:31:45.030954+0000","last_clean":"2024-03-06T07:31:45.030954+0000","last_became_active":"2024-03-06T07:31:44.027829+0000","last_became_peered":"2024-03-06T07:31:44.027829+0000","last_unstale":"2024-03-06T07:31:45.030954+0000","last_undegraded":"2024-03-06T07:31:45.030954+0000","last_fullsized":"2024-03-06T07:31:45.030954+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:08.912427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.043408+0000","last_change":"2024-03-06T07:31:45.039966+0000","last_active":"2024-03-06T07:31:45.043408+0000","last_peered":"2024-03-06T07:31:45.043408+0000","last_clean":"2024-03-06T07:31:45.043408+0000","last_became_active":"2024-03-06T07:31:45.039079+0000","last_became_peered":"2024-03-06T07:31:45.039079+0000","last_unstale":"2024-03-06T07:31:45.043408+0000","last_undegraded":"2024-03-06T07:31:45.043408+0000","last_fullsized":"2024-03-06T07:31:45.043408+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:56.482865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029577+0000","last_change":"2024-03-06T07:31:44.024320+0000","last_active":"2024-03-06T07:31:45.029577+0000","last_peered":"2024-03-06T07:31:45.029577+0000","last_clean":"2024-03-06T07:31:45.029577+0000","last_became_active":"2024-03-06T07:31:44.024051+0000","last_became_peered":"2024-03-06T07:31:44.024051+0000","last_unstale":"2024-03-06T07:31:45.029577+0000","last_undegraded":"2024-03-06T07:31:45.029577+0000","last_fullsized":"2024-03-06T07:31:45.029577+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:59.753052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040276+0000","last_change":"2024-03-06T07:31:45.035119+0000","last_active":"2024-03-06T07:31:45.040276+0000","last_peered":"2024-03-06T07:31:45.040276+0000","last_clean":"2024-03-06T07:31:45.040276+0000","last_became_active":"2024-03-06T07:31:45.034850+0000","last_became_peered":"2024-03-06T07:31:45.034850+0000","last_unstale":"2024-03-06T07:31:45.040276+0000","last_undegraded":"2024-03-06T07:31:45.040276+0000","last_fullsized":"2024-03-06T07:31:45.040276+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:34.247852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041190+0000","last_change":"2024-03-06T07:31:44.031164+0000","last_active":"2024-03-06T07:31:45.041190+0000","last_peered":"2024-03-06T07:31:45.041190+0000","last_clean":"2024-03-06T07:31:45.041190+0000","last_became_active":"2024-03-06T07:31:44.030649+0000","last_became_peered":"2024-03-06T07:31:44.030649+0000","last_unstale":"2024-03-06T07:31:45.041190+0000","last_undegraded":"2024-03-06T07:31:45.041190+0000","last_fullsized":"2024-03-06T07:31:45.041190+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.047269+0000","last_change":"2024-03-06T07:31:45.043562+0000","last_active":"2024-03-06T07:31:45.047269+0000","last_peered":"2024-03-06T07:31:45.047269+0000","last_clean":"2024-03-06T07:31:45.047269+0000","last_became_active":"2024-03-06T07:31:45.043293+0000","last_became_peered":"2024-03-06T07:31:45.043293+0000","last_unstale":"2024-03-06T07:31:45.047269+0000","last_undegraded":"2024-03-06T07:31:45.047269+0000","last_fullsized":"2024-03-06T07:31:45.047269+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:14.318792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.038382+0000","last_change":"2024-03-06T07:31:44.032582+0000","last_active":"2024-03-06T07:31:45.038382+0000","last_peered":"2024-03-06T07:31:45.038382+0000","last_clean":"2024-03-06T07:31:45.038382+0000","last_became_active":"2024-03-06T07:31:44.032075+0000","last_became_peered":"2024-03-06T07:31:44.032075+0000","last_unstale":"2024-03-06T07:31:45.038382+0000","last_undegraded":"2024-03-06T07:31:45.038382+0000","last_fullsized":"2024-03-06T07:31:45.038382+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040239+0000","last_change":"2024-03-06T07:31:45.035199+0000","last_active":"2024-03-06T07:31:45.040239+0000","last_peered":"2024-03-06T07:31:45.040239+0000","last_clean":"2024-03-06T07:31:45.040239+0000","last_became_active":"2024-03-06T07:31:45.034925+0000","last_became_peered":"2024-03-06T07:31:45.034925+0000","last_unstale":"2024-03-06T07:31:45.040239+0000","last_undegraded":"2024-03-06T07:31:45.040239+0000","last_fullsized":"2024-03-06T07:31:45.040239+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034897+0000","last_change":"2024-03-06T07:31:44.031018+0000","last_active":"2024-03-06T07:31:45.034897+0000","last_peered":"2024-03-06T07:31:45.034897+0000","last_clean":"2024-03-06T07:31:45.034897+0000","last_became_active":"2024-03-06T07:31:44.030695+0000","last_became_peered":"2024-03-06T07:31:44.030695+0000","last_unstale":"2024-03-06T07:31:45.034897+0000","last_undegraded":"2024-03-06T07:31:45.034897+0000","last_fullsized":"2024-03-06T07:31:45.034897+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:51.791822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.047228+0000","last_change":"2024-03-06T07:31:45.043749+0000","last_active":"2024-03-06T07:31:45.047228+0000","last_peered":"2024-03-06T07:31:45.047228+0000","last_clean":"2024-03-06T07:31:45.047228+0000","last_became_active":"2024-03-06T07:31:45.043501+0000","last_became_peered":"2024-03-06T07:31:45.043501+0000","last_unstale":"2024-03-06T07:31:45.047228+0000","last_undegraded":"2024-03-06T07:31:45.047228+0000","last_fullsized":"2024-03-06T07:31:45.047228+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.690561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041947+0000","last_change":"2024-03-06T07:31:44.033647+0000","last_active":"2024-03-06T07:31:45.041947+0000","last_peered":"2024-03-06T07:31:45.041947+0000","last_clean":"2024-03-06T07:31:45.041947+0000","last_became_active":"2024-03-06T07:31:44.033393+0000","last_became_peered":"2024-03-06T07:31:44.033393+0000","last_unstale":"2024-03-06T07:31:45.041947+0000","last_undegraded":"2024-03-06T07:31:45.041947+0000","last_fullsized":"2024-03-06T07:31:45.041947+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:35.425689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045750+0000","last_change":"2024-03-06T07:31:45.045750+0000","last_active":"2024-03-06T07:31:45.045750+0000","last_peered":"2024-03-06T07:31:45.045750+0000","last_clean":"2024-03-06T07:31:45.045750+0000","last_became_active":"2024-03-06T07:31:45.044153+0000","last_became_peered":"2024-03-06T07:31:45.044153+0000","last_unstale":"2024-03-06T07:31:45.045750+0000","last_undegraded":"2024-03-06T07:31:45.045750+0000","last_fullsized":"2024-03-06T07:31:45.045750+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:26.528565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.030989+0000","last_change":"2024-03-06T07:31:44.028776+0000","last_active":"2024-03-06T07:31:45.030989+0000","last_peered":"2024-03-06T07:31:45.030989+0000","last_clean":"2024-03-06T07:31:45.030989+0000","last_became_active":"2024-03-06T07:31:44.028504+0000","last_became_peered":"2024-03-06T07:31:44.028504+0000","last_unstale":"2024-03-06T07:31:45.030989+0000","last_undegraded":"2024-03-06T07:31:45.030989+0000","last_fullsized":"2024-03-06T07:31:45.030989+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:17.358709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040051+0000","last_change":"2024-03-06T07:31:45.040051+0000","last_active":"2024-03-06T07:31:45.040051+0000","last_peered":"2024-03-06T07:31:45.040051+0000","last_clean":"2024-03-06T07:31:45.040051+0000","last_became_active":"2024-03-06T07:31:45.039760+0000","last_became_peered":"2024-03-06T07:31:45.039760+0000","last_unstale":"2024-03-06T07:31:45.040051+0000","last_undegraded":"2024-03-06T07:31:45.040051+0000","last_fullsized":"2024-03-06T07:31:45.040051+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:36.430408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029750+0000","last_change":"2024-03-06T07:31:44.027749+0000","last_active":"2024-03-06T07:31:45.029750+0000","last_peered":"2024-03-06T07:31:45.029750+0000","last_clean":"2024-03-06T07:31:45.029750+0000","last_became_active":"2024-03-06T07:31:44.027413+0000","last_became_peered":"2024-03-06T07:31:44.027413+0000","last_unstale":"2024-03-06T07:31:45.029750+0000","last_undegraded":"2024-03-06T07:31:45.029750+0000","last_fullsized":"2024-03-06T07:31:45.029750+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:26.520317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037784+0000","last_change":"2024-03-06T07:31:45.036246+0000","last_active":"2024-03-06T07:31:45.037784+0000","last_peered":"2024-03-06T07:31:45.037784+0000","last_clean":"2024-03-06T07:31:45.037784+0000","last_became_active":"2024-03-06T07:31:45.035684+0000","last_became_peered":"2024-03-06T07:31:45.035684+0000","last_unstale":"2024-03-06T07:31:45.037784+0000","last_undegraded":"2024-03-06T07:31:45.037784+0000","last_fullsized":"2024-03-06T07:31:45.037784+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:19.547257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.030746+0000","last_change":"2024-03-06T07:31:44.027642+0000","last_active":"2024-03-06T07:31:45.030746+0000","last_peered":"2024-03-06T07:31:45.030746+0000","last_clean":"2024-03-06T07:31:45.030746+0000","last_became_active":"2024-03-06T07:31:44.027355+0000","last_became_peered":"2024-03-06T07:31:44.027355+0000","last_unstale":"2024-03-06T07:31:45.030746+0000","last_undegraded":"2024-03-06T07:31:45.030746+0000","last_fullsized":"2024-03-06T07:31:45.030746+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.058715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037724+0000","last_change":"2024-03-06T07:31:45.034848+0000","last_active":"2024-03-06T07:31:45.037724+0000","last_peered":"2024-03-06T07:31:45.037724+0000","last_clean":"2024-03-06T07:31:45.037724+0000","last_became_active":"2024-03-06T07:31:45.034565+0000","last_became_peered":"2024-03-06T07:31:45.034565+0000","last_unstale":"2024-03-06T07:31:45.037724+0000","last_undegraded":"2024-03-06T07:31:45.037724+0000","last_fullsized":"2024-03-06T07:31:45.037724+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:39.939282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029755+0000","last_change":"2024-03-06T07:31:44.025967+0000","last_active":"2024-03-06T07:31:45.029755+0000","last_peered":"2024-03-06T07:31:45.029755+0000","last_clean":"2024-03-06T07:31:45.029755+0000","last_became_active":"2024-03-06T07:31:44.025640+0000","last_became_peered":"2024-03-06T07:31:44.025640+0000","last_unstale":"2024-03-06T07:31:45.029755+0000","last_undegraded":"2024-03-06T07:31:45.029755+0000","last_fullsized":"2024-03-06T07:31:45.029755+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:48.746990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.048574+0000","last_change":"2024-03-06T07:31:45.045298+0000","last_active":"2024-03-06T07:31:45.048574+0000","last_peered":"2024-03-06T07:31:45.048574+0000","last_clean":"2024-03-06T07:31:45.048574+0000","last_became_active":"2024-03-06T07:31:45.045033+0000","last_became_peered":"2024-03-06T07:31:45.045033+0000","last_unstale":"2024-03-06T07:31:45.048574+0000","last_undegraded":"2024-03-06T07:31:45.048574+0000","last_fullsized":"2024-03-06T07:31:45.048574+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:19.547257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040670+0000","last_change":"2024-03-06T07:31:44.032074+0000","last_active":"2024-03-06T07:31:45.040670+0000","last_peered":"2024-03-06T07:31:45.040670+0000","last_clean":"2024-03-06T07:31:45.040670+0000","last_became_active":"2024-03-06T07:31:44.031715+0000","last_became_peered":"2024-03-06T07:31:44.031715+0000","last_unstale":"2024-03-06T07:31:45.040670+0000","last_undegraded":"2024-03-06T07:31:45.040670+0000","last_fullsized":"2024-03-06T07:31:45.040670+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.040107+0000","last_change":"2024-03-06T07:31:45.040107+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.040107+0000","last_undegraded":"2024-03-06T07:31:45.040107+0000","last_fullsized":"2024-03-06T07:31:45.040107+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034395+0000","last_change":"2024-03-06T07:31:44.025352+0000","last_active":"2024-03-06T07:31:45.034395+0000","last_peered":"2024-03-06T07:31:45.034395+0000","last_clean":"2024-03-06T07:31:45.034395+0000","last_became_active":"2024-03-06T07:31:44.025043+0000","last_became_peered":"2024-03-06T07:31:44.025043+0000","last_unstale":"2024-03-06T07:31:45.034395+0000","last_undegraded":"2024-03-06T07:31:45.034395+0000","last_fullsized":"2024-03-06T07:31:45.034395+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:59.681311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037968+0000","last_change":"2024-03-06T07:31:45.035827+0000","last_active":"2024-03-06T07:31:45.037968+0000","last_peered":"2024-03-06T07:31:45.037968+0000","last_clean":"2024-03-06T07:31:45.037968+0000","last_became_active":"2024-03-06T07:31:45.035484+0000","last_became_peered":"2024-03-06T07:31:45.035484+0000","last_unstale":"2024-03-06T07:31:45.037968+0000","last_undegraded":"2024-03-06T07:31:45.037968+0000","last_fullsized":"2024-03-06T07:31:45.037968+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:27.525036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031547+0000","last_change":"2024-03-06T07:31:44.027942+0000","last_active":"2024-03-06T07:31:45.031547+0000","last_peered":"2024-03-06T07:31:45.031547+0000","last_clean":"2024-03-06T07:31:45.031547+0000","last_became_active":"2024-03-06T07:31:44.027666+0000","last_became_peered":"2024-03-06T07:31:44.027666+0000","last_unstale":"2024-03-06T07:31:45.031547+0000","last_undegraded":"2024-03-06T07:31:45.031547+0000","last_fullsized":"2024-03-06T07:31:45.031547+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040316+0000","last_change":"2024-03-06T07:31:45.035716+0000","last_active":"2024-03-06T07:31:45.040316+0000","last_peered":"2024-03-06T07:31:45.040316+0000","last_clean":"2024-03-06T07:31:45.040316+0000","last_became_active":"2024-03-06T07:31:45.035437+0000","last_became_peered":"2024-03-06T07:31:45.035437+0000","last_unstale":"2024-03-06T07:31:45.040316+0000","last_undegraded":"2024-03-06T07:31:45.040316+0000","last_fullsized":"2024-03-06T07:31:45.040316+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:20.551707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.038657+0000","last_change":"2024-03-06T07:31:44.031865+0000","last_active":"2024-03-06T07:31:45.038657+0000","last_peered":"2024-03-06T07:31:45.038657+0000","last_clean":"2024-03-06T07:31:45.038657+0000","last_became_active":"2024-03-06T07:31:44.030846+0000","last_became_peered":"2024-03-06T07:31:44.030846+0000","last_unstale":"2024-03-06T07:31:45.038657+0000","last_undegraded":"2024-03-06T07:31:45.038657+0000","last_fullsized":"2024-03-06T07:31:45.038657+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.034911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":407,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040477+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:31:45.040477+0000","last_peered":"2024-03-06T07:31:45.040477+0000","last_clean":"2024-03-06T07:31:45.040477+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:31:45.040477+0000","last_undegraded":"2024-03-06T07:31:45.040477+0000","last_fullsized":"2024-03-06T07:31:45.040477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574961,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41832,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330008,"statfs":{"total":96636764160,"available":96593928192,"internally_reserved":0,"allocated":2383872,"data_stored":1371971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574962,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574961,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41252,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330588,"statfs":{"total":96636764160,"available":96594522112,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574962,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41836,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330004,"statfs":{"total":96636764160,"available":96593924096,"internally_reserved":0,"allocated":2383872,"data_stored":1371971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:49.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.220+0000 7f65cfe64640 1 Processor -- start 2024-03-06T07:31:49.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.220+0000 7f65cfe64640 1 -- start start 2024-03-06T07:31:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cfe64640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c80b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cfe64640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65c815bd70 0x7f65c80b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cfe64640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c80b3600 con 0x7f65c80b2290 2024-03-06T07:31:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cfe64640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c80b26e0 con 0x7f65c815bd70 2024-03-06T07:31:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cfe64640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b38b0 con 0x7f65c8158630 2024-03-06T07:31:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cd3ff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65c815bd70 0x7f65c80b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cdc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c80b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cdc00640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c80b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:54718/0 (socket says 172.21.15.27:54718) 2024-03-06T07:31:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cdc00640 1 -- 172.21.15.27:0/2947163028 learned_addr learned my addr 172.21.15.27:0/2947163028 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cd3ff640 1 --2- 172.21.15.27:0/2947163028 >> v2:172.21.15.69:3300/0 conn(0x7f65c815bd70 0x7f65c80b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:37412/0 (socket says 172.21.15.27:37412) 2024-03-06T07:31:49.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65ce401640 1 --1- 172.21.15.27:0/2947163028 >> v1:172.21.15.69:6789/0 conn(0x7f65c80b2290 0x7f65c815e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:38238/0 (socket says 172.21.15.27:38238) 2024-03-06T07:31:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65ccbfe640 1 -- 172.21.15.27:0/2947163028 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248249852 0 0) 0x7f65c80b3600 con 0x7f65c80b2290 2024-03-06T07:31:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65ccbfe640 1 -- 172.21.15.27:0/2947163028 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b4003520 con 0x7f65c80b2290 2024-03-06T07:31:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cdc00640 1 -- 172.21.15.27:0/2947163028 >> v2:172.21.15.69:3300/0 conn(0x7f65c815bd70 msgr2=0x7f65c80b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cdc00640 1 --2- 172.21.15.27:0/2947163028 >> v2:172.21.15.69:3300/0 conn(0x7f65c815bd70 0x7f65c80b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.222+0000 7f65cdc00640 1 -- 172.21.15.27:0/2947163028 >> v1:172.21.15.69:6789/0 conn(0x7f65c80b2290 legacy=0x7f65c815e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.223+0000 7f65cdc00640 1 -- 172.21.15.27:0/2947163028 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80b3b70 con 0x7f65c8158630 2024-03-06T07:31:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.223+0000 7f65cd3ff640 1 --2- 172.21.15.27:0/2947163028 >> v2:172.21.15.69:3300/0 conn(0x7f65c815bd70 0x7f65c80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.223+0000 7f65cdc00640 1 --2- 172.21.15.27:0/2947163028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c80b30c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009870 tx=0x7f65b802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80d7826d860c0bdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.223+0000 7f65ccbfe640 1 -- 172.21.15.27:0/2947163028 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8035020 con 0x7f65c8158630 2024-03-06T07:31:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.224+0000 7f65ccbfe640 1 -- 172.21.15.27:0/2947163028 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b8031640 con 0x7f65c8158630 2024-03-06T07:31:49.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.224+0000 7f65ccbfe640 1 -- 172.21.15.27:0/2947163028 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b802d040 con 0x7f65c8158630 2024-03-06T07:31:49.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.224+0000 7f65cfe64640 1 -- 172.21.15.27:0/2947163028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 msgr2=0x7f65c80b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.224+0000 7f65cfe64640 1 --2- 172.21.15.27:0/2947163028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c80b30c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009870 tx=0x7f65b802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.224+0000 7f65cfe64640 1 -- 172.21.15.27:0/2947163028 shutdown_connections 2024-03-06T07:31:49.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.224+0000 7f65cfe64640 1 --2- 172.21.15.27:0/2947163028 >> v2:172.21.15.69:3300/0 conn(0x7f65c815bd70 0x7f65c80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.224+0000 7f65cfe64640 1 --2- 172.21.15.27:0/2947163028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c80b30c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.224+0000 7f65cfe64640 1 -- 172.21.15.27:0/2947163028 >> 172.21.15.27:0/2947163028 conn(0x7f65c80b55e0 msgr2=0x7f65c80b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.225+0000 7f65cfe64640 1 -- 172.21.15.27:0/2947163028 shutdown_connections 2024-03-06T07:31:49.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.225+0000 7f65cfe64640 1 -- 172.21.15.27:0/2947163028 wait complete. 2024-03-06T07:31:49.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.227+0000 7f65cfe64640 1 Processor -- start 2024-03-06T07:31:49.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.227+0000 7f65cfe64640 1 -- start start 2024-03-06T07:31:49.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65cfe64640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65c80b2290 0x7f65c814ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65cfe64640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c8151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65cfe64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c815bd70 0x7f65c8151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65cfe64640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b4410 con 0x7f65c815bd70 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65cfe64640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b4130 con 0x7f65c80b2290 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65cfe64640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c8158a10 con 0x7f65c8158630 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65ce401640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c815bd70 0x7f65c8151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65ce401640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c815bd70 0x7f65c8151560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36618/0 (socket says 172.21.15.27:36618) 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65ce401640 1 -- 172.21.15.27:0/3912756198 learned_addr learned my addr 172.21.15.27:0/3912756198 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65cdc00640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65c80b2290 0x7f65c814ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65cd3ff640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c8151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65ce401640 1 -- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 msgr2=0x7f65c8151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65ce401640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c8151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65ce401640 1 -- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65c80b2290 msgr2=0x7f65c814ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65ce401640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65c80b2290 0x7f65c814ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.228+0000 7f65ce401640 1 -- 172.21.15.27:0/3912756198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c81f5380 con 0x7f65c815bd70 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.229+0000 7f65ce401640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c815bd70 0x7f65c8151560 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f65c000c7f0 tx=0x7f65c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.229+0000 7f65be7fc640 1 -- 172.21.15.27:0/3912756198 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0017070 con 0x7f65c815bd70 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.229+0000 7f65be7fc640 1 -- 172.21.15.27:0/3912756198 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c000a640 con 0x7f65c815bd70 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.229+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c81f5660 con 0x7f65c815bd70 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.229+0000 7f65be7fc640 1 -- 172.21.15.27:0/3912756198 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c000aa10 con 0x7f65c815bd70 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.229+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c81f5c20 con 0x7f65c815bd70 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.230+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c8067ff0 con 0x7f65c815bd70 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.231+0000 7f65be7fc640 1 -- 172.21.15.27:0/3912756198 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65c001a070 con 0x7f65c815bd70 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.231+0000 7f65be7fc640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65ac06b4d0 0x7f65ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.231+0000 7f65be7fc640 1 -- 172.21.15.27:0/3912756198 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65c0089820 con 0x7f65c815bd70 2024-03-06T07:31:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.231+0000 7f65cdc00640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65ac06b4d0 0x7f65ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.232+0000 7f65cdc00640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65ac06b4d0 0x7f65ac06d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f65b8036000 tx=0x7f65b803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.240+0000 7f65be7fc640 1 -- 172.21.15.27:0/3912756198 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65c0058060 con 0x7f65c815bd70 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.422+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65c8068700 con 0x7f65ac06b4d0 2024-03-06T07:31:49.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.428+0000 7f65be7fc640 1 -- 172.21.15.27:0/3912756198 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251666 (secure 0 0 0) 0x7f65b8085730 con 0x7f65ac06b4d0 2024-03-06T07:31:49.432 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:49.434 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:31:49.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65ac06b4d0 msgr2=0x7f65ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65ac06b4d0 0x7f65ac06d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f65b8036000 tx=0x7f65b803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:49.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c815bd70 msgr2=0x7f65c8151560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c815bd70 0x7f65c8151560 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f65c000c7f0 tx=0x7f65c000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 shutdown_connections 2024-03-06T07:31:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f65ac06b4d0 0x7f65ac06d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65c815bd70 0x7f65c8151560 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65c8158630 0x7f65c8151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 --2- 172.21.15.27:0/3912756198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65c80b2290 0x7f65c814ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 >> 172.21.15.27:0/3912756198 conn(0x7f65c80b55e0 msgr2=0x7f65c8159550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 shutdown_connections 2024-03-06T07:31:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.431+0000 7f65cfe64640 1 -- 172.21.15.27:0/3912756198 wait complete. 2024-03-06T07:31:49.468 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-03-06T07:31:48.119006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":166,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331160,"kb_used_data":15144,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643560,"statfs":{"total":773094113280,"available":772755005440,"internally_reserved":0,"allocated":15507456,"data_stored":7433560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.109582"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046476+0000","last_change":"2024-03-06T07:31:45.046476+0000","last_active":"2024-03-06T07:31:45.046476+0000","last_peered":"2024-03-06T07:31:45.046476+0000","last_clean":"2024-03-06T07:31:45.046476+0000","last_became_active":"2024-03-06T07:31:45.046179+0000","last_became_peered":"2024-03-06T07:31:45.046179+0000","last_unstale":"2024-03-06T07:31:45.046476+0000","last_undegraded":"2024-03-06T07:31:45.046476+0000","last_fullsized":"2024-03-06T07:31:45.046476+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:28.329642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041710+0000","last_change":"2024-03-06T07:31:45.041710+0000","last_active":"2024-03-06T07:31:45.041710+0000","last_peered":"2024-03-06T07:31:45.041710+0000","last_clean":"2024-03-06T07:31:45.041710+0000","last_became_active":"2024-03-06T07:31:45.041413+0000","last_became_peered":"2024-03-06T07:31:45.041413+0000","last_unstale":"2024-03-06T07:31:45.041710+0000","last_undegraded":"2024-03-06T07:31:45.041710+0000","last_fullsized":"2024-03-06T07:31:45.041710+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:53.307481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.048666+0000","last_change":"2024-03-06T07:31:45.048666+0000","last_active":"2024-03-06T07:31:45.048666+0000","last_peered":"2024-03-06T07:31:45.048666+0000","last_clean":"2024-03-06T07:31:45.048666+0000","last_became_active":"2024-03-06T07:31:45.044404+0000","last_became_peered":"2024-03-06T07:31:45.044404+0000","last_unstale":"2024-03-06T07:31:45.048666+0000","last_undegraded":"2024-03-06T07:31:45.048666+0000","last_fullsized":"2024-03-06T07:31:45.048666+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:55.936260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046599+0000","last_change":"2024-03-06T07:31:45.046599+0000","last_active":"2024-03-06T07:31:45.046599+0000","last_peered":"2024-03-06T07:31:45.046599+0000","last_clean":"2024-03-06T07:31:45.046599+0000","last_became_active":"2024-03-06T07:31:45.046251+0000","last_became_peered":"2024-03-06T07:31:45.046251+0000","last_unstale":"2024-03-06T07:31:45.046599+0000","last_undegraded":"2024-03-06T07:31:45.046599+0000","last_fullsized":"2024-03-06T07:31:45.046599+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:13.567550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050368+0000","last_change":"2024-03-06T07:31:45.048513+0000","last_active":"2024-03-06T07:31:45.050368+0000","last_peered":"2024-03-06T07:31:45.050368+0000","last_clean":"2024-03-06T07:31:45.050368+0000","last_became_active":"2024-03-06T07:31:45.048251+0000","last_became_peered":"2024-03-06T07:31:45.048251+0000","last_unstale":"2024-03-06T07:31:45.050368+0000","last_undegraded":"2024-03-06T07:31:45.050368+0000","last_fullsized":"2024-03-06T07:31:45.050368+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:04.808270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.029936+0000","last_change":"2024-03-06T07:31:45.029936+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.029936+0000","last_undegraded":"2024-03-06T07:31:45.029936+0000","last_fullsized":"2024-03-06T07:31:45.029936+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040291+0000","last_change":"2024-03-06T07:31:45.035876+0000","last_active":"2024-03-06T07:31:45.040291+0000","last_peered":"2024-03-06T07:31:45.040291+0000","last_clean":"2024-03-06T07:31:45.040291+0000","last_became_active":"2024-03-06T07:31:45.035604+0000","last_became_peered":"2024-03-06T07:31:45.035604+0000","last_unstale":"2024-03-06T07:31:45.040291+0000","last_undegraded":"2024-03-06T07:31:45.040291+0000","last_fullsized":"2024-03-06T07:31:45.040291+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:28.333006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037397+0000","last_change":"2024-03-06T07:31:45.035300+0000","last_active":"2024-03-06T07:31:45.037397+0000","last_peered":"2024-03-06T07:31:45.037397+0000","last_clean":"2024-03-06T07:31:45.037397+0000","last_became_active":"2024-03-06T07:31:45.034964+0000","last_became_peered":"2024-03-06T07:31:45.034964+0000","last_unstale":"2024-03-06T07:31:45.037397+0000","last_undegraded":"2024-03-06T07:31:45.037397+0000","last_fullsized":"2024-03-06T07:31:45.037397+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:43.063911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.039202+0000","last_change":"2024-03-06T07:31:45.039202+0000","last_active":"2024-03-06T07:31:45.039202+0000","last_peered":"2024-03-06T07:31:45.039202+0000","last_clean":"2024-03-06T07:31:45.039202+0000","last_became_active":"2024-03-06T07:31:45.038902+0000","last_became_peered":"2024-03-06T07:31:45.038902+0000","last_unstale":"2024-03-06T07:31:45.039202+0000","last_undegraded":"2024-03-06T07:31:45.039202+0000","last_fullsized":"2024-03-06T07:31:45.039202+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:44.736535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050634+0000","last_change":"2024-03-06T07:31:45.047007+0000","last_active":"2024-03-06T07:31:45.050634+0000","last_peered":"2024-03-06T07:31:45.050634+0000","last_clean":"2024-03-06T07:31:45.050634+0000","last_became_active":"2024-03-06T07:31:45.046533+0000","last_became_peered":"2024-03-06T07:31:45.046533+0000","last_unstale":"2024-03-06T07:31:45.050634+0000","last_undegraded":"2024-03-06T07:31:45.050634+0000","last_fullsized":"2024-03-06T07:31:45.050634+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:11.401916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037764+0000","last_change":"2024-03-06T07:31:45.036443+0000","last_active":"2024-03-06T07:31:45.037764+0000","last_peered":"2024-03-06T07:31:45.037764+0000","last_clean":"2024-03-06T07:31:45.037764+0000","last_became_active":"2024-03-06T07:31:45.035986+0000","last_became_peered":"2024-03-06T07:31:45.035986+0000","last_unstale":"2024-03-06T07:31:45.037764+0000","last_undegraded":"2024-03-06T07:31:45.037764+0000","last_fullsized":"2024-03-06T07:31:45.037764+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.690561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.051142+0000","last_change":"2024-03-06T07:31:45.049645+0000","last_active":"2024-03-06T07:31:45.051142+0000","last_peered":"2024-03-06T07:31:45.051142+0000","last_clean":"2024-03-06T07:31:45.051142+0000","last_became_active":"2024-03-06T07:31:45.048865+0000","last_became_peered":"2024-03-06T07:31:45.048865+0000","last_unstale":"2024-03-06T07:31:45.051142+0000","last_undegraded":"2024-03-06T07:31:45.051142+0000","last_fullsized":"2024-03-06T07:31:45.051142+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:36.725280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050346+0000","last_change":"2024-03-06T07:31:45.048451+0000","last_active":"2024-03-06T07:31:45.050346+0000","last_peered":"2024-03-06T07:31:45.050346+0000","last_clean":"2024-03-06T07:31:45.050346+0000","last_became_active":"2024-03-06T07:31:45.048188+0000","last_became_peered":"2024-03-06T07:31:45.048188+0000","last_unstale":"2024-03-06T07:31:45.050346+0000","last_undegraded":"2024-03-06T07:31:45.050346+0000","last_fullsized":"2024-03-06T07:31:45.050346+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:17.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044378+0000","last_change":"2024-03-06T07:31:45.042722+0000","last_active":"2024-03-06T07:31:45.044378+0000","last_peered":"2024-03-06T07:31:45.044378+0000","last_clean":"2024-03-06T07:31:45.044378+0000","last_became_active":"2024-03-06T07:31:45.042456+0000","last_became_peered":"2024-03-06T07:31:45.042456+0000","last_unstale":"2024-03-06T07:31:45.044378+0000","last_undegraded":"2024-03-06T07:31:45.044378+0000","last_fullsized":"2024-03-06T07:31:45.044378+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:52.796540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046120+0000","last_change":"2024-03-06T07:31:45.046120+0000","last_active":"2024-03-06T07:31:45.046120+0000","last_peered":"2024-03-06T07:31:45.046120+0000","last_clean":"2024-03-06T07:31:45.046120+0000","last_became_active":"2024-03-06T07:31:45.045855+0000","last_became_peered":"2024-03-06T07:31:45.045855+0000","last_unstale":"2024-03-06T07:31:45.046120+0000","last_undegraded":"2024-03-06T07:31:45.046120+0000","last_fullsized":"2024-03-06T07:31:45.046120+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:21.522756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042236+0000","last_change":"2024-03-06T07:31:45.039353+0000","last_active":"2024-03-06T07:31:45.042236+0000","last_peered":"2024-03-06T07:31:45.042236+0000","last_clean":"2024-03-06T07:31:45.042236+0000","last_became_active":"2024-03-06T07:31:45.039090+0000","last_became_peered":"2024-03-06T07:31:45.039090+0000","last_unstale":"2024-03-06T07:31:45.042236+0000","last_undegraded":"2024-03-06T07:31:45.042236+0000","last_fullsized":"2024-03-06T07:31:45.042236+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:13.414833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.042465+0000","last_change":"2024-03-06T07:31:45.042465+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.042465+0000","last_undegraded":"2024-03-06T07:31:45.042465+0000","last_fullsized":"2024-03-06T07:31:45.042465+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046112+0000","last_change":"2024-03-06T07:31:45.046112+0000","last_active":"2024-03-06T07:31:45.046112+0000","last_peered":"2024-03-06T07:31:45.046112+0000","last_clean":"2024-03-06T07:31:45.046112+0000","last_became_active":"2024-03-06T07:31:45.043698+0000","last_became_peered":"2024-03-06T07:31:45.043698+0000","last_unstale":"2024-03-06T07:31:45.046112+0000","last_undegraded":"2024-03-06T07:31:45.046112+0000","last_fullsized":"2024-03-06T07:31:45.046112+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:14.807136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.043083+0000","last_change":"2024-03-06T07:31:45.038374+0000","last_active":"2024-03-06T07:31:45.043083+0000","last_peered":"2024-03-06T07:31:45.043083+0000","last_clean":"2024-03-06T07:31:45.043083+0000","last_became_active":"2024-03-06T07:31:45.038088+0000","last_became_peered":"2024-03-06T07:31:45.038088+0000","last_unstale":"2024-03-06T07:31:45.043083+0000","last_undegraded":"2024-03-06T07:31:45.043083+0000","last_fullsized":"2024-03-06T07:31:45.043083+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:04.808270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034187+0000","last_change":"2024-03-06T07:31:45.032953+0000","last_active":"2024-03-06T07:31:45.034187+0000","last_peered":"2024-03-06T07:31:45.034187+0000","last_clean":"2024-03-06T07:31:45.034187+0000","last_became_active":"2024-03-06T07:31:45.032688+0000","last_became_peered":"2024-03-06T07:31:45.032688+0000","last_unstale":"2024-03-06T07:31:45.034187+0000","last_undegraded":"2024-03-06T07:31:45.034187+0000","last_fullsized":"2024-03-06T07:31:45.034187+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:02.496094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.049212+0000","last_change":"2024-03-06T07:31:45.049212+0000","last_active":"2024-03-06T07:31:45.049212+0000","last_peered":"2024-03-06T07:31:45.049212+0000","last_clean":"2024-03-06T07:31:45.049212+0000","last_became_active":"2024-03-06T07:31:45.047359+0000","last_became_peered":"2024-03-06T07:31:45.047359+0000","last_unstale":"2024-03-06T07:31:45.049212+0000","last_undegraded":"2024-03-06T07:31:45.049212+0000","last_fullsized":"2024-03-06T07:31:45.049212+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:05.783188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.038255+0000","last_change":"2024-03-06T07:31:45.035442+0000","last_active":"2024-03-06T07:31:45.038255+0000","last_peered":"2024-03-06T07:31:45.038255+0000","last_clean":"2024-03-06T07:31:45.038255+0000","last_became_active":"2024-03-06T07:31:45.035154+0000","last_became_peered":"2024-03-06T07:31:45.035154+0000","last_unstale":"2024-03-06T07:31:45.038255+0000","last_undegraded":"2024-03-06T07:31:45.038255+0000","last_fullsized":"2024-03-06T07:31:45.038255+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:27.623139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041778+0000","last_change":"2024-03-06T07:31:45.039599+0000","last_active":"2024-03-06T07:31:45.041778+0000","last_peered":"2024-03-06T07:31:45.041778+0000","last_clean":"2024-03-06T07:31:45.041778+0000","last_became_active":"2024-03-06T07:31:45.039297+0000","last_became_peered":"2024-03-06T07:31:45.039297+0000","last_unstale":"2024-03-06T07:31:45.041778+0000","last_undegraded":"2024-03-06T07:31:45.041778+0000","last_fullsized":"2024-03-06T07:31:45.041778+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:33.886298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":6,"reported_epoch":144,"state":"active","last_fresh":"2024-03-06T07:31:45.047191+0000","last_change":"2024-03-06T07:31:45.047127+0000","last_active":"2024-03-06T07:31:45.047191+0000","last_peered":"2024-03-06T07:31:45.047191+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"2024-03-06T07:31:45.047127+0000","last_became_peered":"2024-03-06T07:31:45.047127+0000","last_unstale":"2024-03-06T07:31:45.047191+0000","last_undegraded":"2024-03-06T07:31:45.047191+0000","last_fullsized":"2024-03-06T07:31:45.047191+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042283+0000","last_change":"2024-03-06T07:31:45.040040+0000","last_active":"2024-03-06T07:31:45.042283+0000","last_peered":"2024-03-06T07:31:45.042283+0000","last_clean":"2024-03-06T07:31:45.042283+0000","last_became_active":"2024-03-06T07:31:45.039753+0000","last_became_peered":"2024-03-06T07:31:45.039753+0000","last_unstale":"2024-03-06T07:31:45.042283+0000","last_undegraded":"2024-03-06T07:31:45.042283+0000","last_fullsized":"2024-03-06T07:31:45.042283+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:27.525036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042445+0000","last_change":"2024-03-06T07:31:45.037506+0000","last_active":"2024-03-06T07:31:45.042445+0000","last_peered":"2024-03-06T07:31:45.042445+0000","last_clean":"2024-03-06T07:31:45.042445+0000","last_became_active":"2024-03-06T07:31:45.036896+0000","last_became_peered":"2024-03-06T07:31:45.036896+0000","last_unstale":"2024-03-06T07:31:45.042445+0000","last_undegraded":"2024-03-06T07:31:45.042445+0000","last_fullsized":"2024-03-06T07:31:45.042445+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:17.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.025450+0000","last_change":"2024-03-06T07:31:45.025450+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.025450+0000","last_undegraded":"2024-03-06T07:31:45.025450+0000","last_fullsized":"2024-03-06T07:31:45.025450+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.028666+0000","last_change":"2024-03-06T07:31:45.028666+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.028666+0000","last_undegraded":"2024-03-06T07:31:45.028666+0000","last_fullsized":"2024-03-06T07:31:45.028666+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045636+0000","last_change":"2024-03-06T07:31:45.045636+0000","last_active":"2024-03-06T07:31:45.045636+0000","last_peered":"2024-03-06T07:31:45.045636+0000","last_clean":"2024-03-06T07:31:45.045636+0000","last_became_active":"2024-03-06T07:31:45.045309+0000","last_became_peered":"2024-03-06T07:31:45.045309+0000","last_unstale":"2024-03-06T07:31:45.045636+0000","last_undegraded":"2024-03-06T07:31:45.045636+0000","last_fullsized":"2024-03-06T07:31:45.045636+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:36.113249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044357+0000","last_change":"2024-03-06T07:31:45.041617+0000","last_active":"2024-03-06T07:31:45.044357+0000","last_peered":"2024-03-06T07:31:45.044357+0000","last_clean":"2024-03-06T07:31:45.044357+0000","last_became_active":"2024-03-06T07:31:45.040818+0000","last_became_peered":"2024-03-06T07:31:45.040818+0000","last_unstale":"2024-03-06T07:31:45.044357+0000","last_undegraded":"2024-03-06T07:31:45.044357+0000","last_fullsized":"2024-03-06T07:31:45.044357+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:00.686030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.029698+0000","last_change":"2024-03-06T07:31:45.029698+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.029698+0000","last_undegraded":"2024-03-06T07:31:45.029698+0000","last_fullsized":"2024-03-06T07:31:45.029698+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034138+0000","last_change":"2024-03-06T07:31:45.032208+0000","last_active":"2024-03-06T07:31:45.034138+0000","last_peered":"2024-03-06T07:31:45.034138+0000","last_clean":"2024-03-06T07:31:45.034138+0000","last_became_active":"2024-03-06T07:31:45.031935+0000","last_became_peered":"2024-03-06T07:31:45.031935+0000","last_unstale":"2024-03-06T07:31:45.034138+0000","last_undegraded":"2024-03-06T07:31:45.034138+0000","last_fullsized":"2024-03-06T07:31:45.034138+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:11.848515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050610+0000","last_change":"2024-03-06T07:31:45.046451+0000","last_active":"2024-03-06T07:31:45.050610+0000","last_peered":"2024-03-06T07:31:45.050610+0000","last_clean":"2024-03-06T07:31:45.050610+0000","last_became_active":"2024-03-06T07:31:45.045791+0000","last_became_peered":"2024-03-06T07:31:45.045791+0000","last_unstale":"2024-03-06T07:31:45.050610+0000","last_undegraded":"2024-03-06T07:31:45.050610+0000","last_fullsized":"2024-03-06T07:31:45.050610+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:53.307481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041764+0000","last_change":"2024-03-06T07:31:44.033293+0000","last_active":"2024-03-06T07:31:45.041764+0000","last_peered":"2024-03-06T07:31:45.041764+0000","last_clean":"2024-03-06T07:31:45.041764+0000","last_became_active":"2024-03-06T07:31:44.032831+0000","last_became_peered":"2024-03-06T07:31:44.032831+0000","last_unstale":"2024-03-06T07:31:45.041764+0000","last_undegraded":"2024-03-06T07:31:45.041764+0000","last_fullsized":"2024-03-06T07:31:45.041764+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:43.731816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040911+0000","last_change":"2024-03-06T07:31:45.040911+0000","last_active":"2024-03-06T07:31:45.040911+0000","last_peered":"2024-03-06T07:31:45.040911+0000","last_clean":"2024-03-06T07:31:45.040911+0000","last_became_active":"2024-03-06T07:31:45.040632+0000","last_became_peered":"2024-03-06T07:31:45.040632+0000","last_unstale":"2024-03-06T07:31:45.040911+0000","last_undegraded":"2024-03-06T07:31:45.040911+0000","last_fullsized":"2024-03-06T07:31:45.040911+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:21.522756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031190+0000","last_change":"2024-03-06T07:31:44.027157+0000","last_active":"2024-03-06T07:31:45.031190+0000","last_peered":"2024-03-06T07:31:45.031190+0000","last_clean":"2024-03-06T07:31:45.031190+0000","last_became_active":"2024-03-06T07:31:44.026865+0000","last_became_peered":"2024-03-06T07:31:44.026865+0000","last_unstale":"2024-03-06T07:31:45.031190+0000","last_undegraded":"2024-03-06T07:31:45.031190+0000","last_fullsized":"2024-03-06T07:31:45.031190+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042098+0000","last_change":"2024-03-06T07:31:45.042098+0000","last_active":"2024-03-06T07:31:45.042098+0000","last_peered":"2024-03-06T07:31:45.042098+0000","last_clean":"2024-03-06T07:31:45.042098+0000","last_became_active":"2024-03-06T07:31:45.041801+0000","last_became_peered":"2024-03-06T07:31:45.041801+0000","last_unstale":"2024-03-06T07:31:45.042098+0000","last_undegraded":"2024-03-06T07:31:45.042098+0000","last_fullsized":"2024-03-06T07:31:45.042098+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:48.091616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031296+0000","last_change":"2024-03-06T07:31:44.023697+0000","last_active":"2024-03-06T07:31:45.031296+0000","last_peered":"2024-03-06T07:31:45.031296+0000","last_clean":"2024-03-06T07:31:45.031296+0000","last_became_active":"2024-03-06T07:31:44.023399+0000","last_became_peered":"2024-03-06T07:31:44.023399+0000","last_unstale":"2024-03-06T07:31:45.031296+0000","last_undegraded":"2024-03-06T07:31:45.031296+0000","last_fullsized":"2024-03-06T07:31:45.031296+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:19.201492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050493+0000","last_change":"2024-03-06T07:31:45.046580+0000","last_active":"2024-03-06T07:31:45.050493+0000","last_peered":"2024-03-06T07:31:45.050493+0000","last_clean":"2024-03-06T07:31:45.050493+0000","last_became_active":"2024-03-06T07:31:45.046312+0000","last_became_peered":"2024-03-06T07:31:45.046312+0000","last_unstale":"2024-03-06T07:31:45.050493+0000","last_undegraded":"2024-03-06T07:31:45.050493+0000","last_fullsized":"2024-03-06T07:31:45.050493+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:20.046408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031181+0000","last_change":"2024-03-06T07:31:44.020513+0000","last_active":"2024-03-06T07:31:45.031181+0000","last_peered":"2024-03-06T07:31:45.031181+0000","last_clean":"2024-03-06T07:31:45.031181+0000","last_became_active":"2024-03-06T07:31:44.020196+0000","last_became_peered":"2024-03-06T07:31:44.020196+0000","last_unstale":"2024-03-06T07:31:45.031181+0000","last_undegraded":"2024-03-06T07:31:45.031181+0000","last_fullsized":"2024-03-06T07:31:45.031181+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:47.307708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.030083+0000","last_change":"2024-03-06T07:31:45.030083+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.030083+0000","last_undegraded":"2024-03-06T07:31:45.030083+0000","last_fullsized":"2024-03-06T07:31:45.030083+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025657+0000","last_change":"2024-03-06T07:31:44.033751+0000","last_active":"2024-03-06T07:31:45.025657+0000","last_peered":"2024-03-06T07:31:45.025657+0000","last_clean":"2024-03-06T07:31:45.025657+0000","last_became_active":"2024-03-06T07:31:44.033386+0000","last_became_peered":"2024-03-06T07:31:44.033386+0000","last_unstale":"2024-03-06T07:31:45.025657+0000","last_undegraded":"2024-03-06T07:31:45.025657+0000","last_fullsized":"2024-03-06T07:31:45.025657+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:50.356019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045822+0000","last_change":"2024-03-06T07:31:45.045822+0000","last_active":"2024-03-06T07:31:45.045822+0000","last_peered":"2024-03-06T07:31:45.045822+0000","last_clean":"2024-03-06T07:31:45.045822+0000","last_became_active":"2024-03-06T07:31:45.045552+0000","last_became_peered":"2024-03-06T07:31:45.045552+0000","last_unstale":"2024-03-06T07:31:45.045822+0000","last_undegraded":"2024-03-06T07:31:45.045822+0000","last_fullsized":"2024-03-06T07:31:45.045822+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:08.039630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"144'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.028727+0000","last_change":"2024-03-06T07:31:44.028730+0000","last_active":"2024-03-06T07:31:45.028727+0000","last_peered":"2024-03-06T07:31:45.028727+0000","last_clean":"2024-03-06T07:31:45.028727+0000","last_became_active":"2024-03-06T07:31:44.028446+0000","last_became_peered":"2024-03-06T07:31:44.028446+0000","last_unstale":"2024-03-06T07:31:45.028727+0000","last_undegraded":"2024-03-06T07:31:45.028727+0000","last_fullsized":"2024-03-06T07:31:45.028727+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:38.934563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.035147+0000","last_change":"2024-03-06T07:31:45.033665+0000","last_active":"2024-03-06T07:31:45.035147+0000","last_peered":"2024-03-06T07:31:45.035147+0000","last_clean":"2024-03-06T07:31:45.035147+0000","last_became_active":"2024-03-06T07:31:45.033369+0000","last_became_peered":"2024-03-06T07:31:45.033369+0000","last_unstale":"2024-03-06T07:31:45.035147+0000","last_undegraded":"2024-03-06T07:31:45.035147+0000","last_fullsized":"2024-03-06T07:31:45.035147+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:45.323683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025219+0000","last_change":"2024-03-06T07:31:44.024794+0000","last_active":"2024-03-06T07:31:45.025219+0000","last_peered":"2024-03-06T07:31:45.025219+0000","last_clean":"2024-03-06T07:31:45.025219+0000","last_became_active":"2024-03-06T07:31:44.024499+0000","last_became_peered":"2024-03-06T07:31:44.024499+0000","last_unstale":"2024-03-06T07:31:45.025219+0000","last_undegraded":"2024-03-06T07:31:45.025219+0000","last_fullsized":"2024-03-06T07:31:45.025219+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:01.912662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042985+0000","last_change":"2024-03-06T07:31:45.040904+0000","last_active":"2024-03-06T07:31:45.042985+0000","last_peered":"2024-03-06T07:31:45.042985+0000","last_clean":"2024-03-06T07:31:45.042985+0000","last_became_active":"2024-03-06T07:31:45.040606+0000","last_became_peered":"2024-03-06T07:31:45.040606+0000","last_unstale":"2024-03-06T07:31:45.042985+0000","last_undegraded":"2024-03-06T07:31:45.042985+0000","last_fullsized":"2024-03-06T07:31:45.042985+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:18.363428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031359+0000","last_change":"2024-03-06T07:31:44.022269+0000","last_active":"2024-03-06T07:31:45.031359+0000","last_peered":"2024-03-06T07:31:45.031359+0000","last_clean":"2024-03-06T07:31:45.031359+0000","last_became_active":"2024-03-06T07:31:44.021982+0000","last_became_peered":"2024-03-06T07:31:44.021982+0000","last_unstale":"2024-03-06T07:31:45.031359+0000","last_undegraded":"2024-03-06T07:31:45.031359+0000","last_fullsized":"2024-03-06T07:31:45.031359+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:51.573386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.039591+0000","last_change":"2024-03-06T07:31:45.037434+0000","last_active":"2024-03-06T07:31:45.039591+0000","last_peered":"2024-03-06T07:31:45.039591+0000","last_clean":"2024-03-06T07:31:45.039591+0000","last_became_active":"2024-03-06T07:31:45.037148+0000","last_became_peered":"2024-03-06T07:31:45.037148+0000","last_unstale":"2024-03-06T07:31:45.039591+0000","last_undegraded":"2024-03-06T07:31:45.039591+0000","last_fullsized":"2024-03-06T07:31:45.039591+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:40.063434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031021+0000","last_change":"2024-03-06T07:31:44.019467+0000","last_active":"2024-03-06T07:31:45.031021+0000","last_peered":"2024-03-06T07:31:45.031021+0000","last_clean":"2024-03-06T07:31:45.031021+0000","last_became_active":"2024-03-06T07:31:44.019183+0000","last_became_peered":"2024-03-06T07:31:44.019183+0000","last_unstale":"2024-03-06T07:31:45.031021+0000","last_undegraded":"2024-03-06T07:31:45.031021+0000","last_fullsized":"2024-03-06T07:31:45.031021+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:22.325138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":6,"reported_epoch":144,"state":"active","last_fresh":"2024-03-06T07:31:45.052295+0000","last_change":"2024-03-06T07:31:45.052223+0000","last_active":"2024-03-06T07:31:45.052295+0000","last_peered":"2024-03-06T07:31:45.052295+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"2024-03-06T07:31:45.052223+0000","last_became_peered":"2024-03-06T07:31:45.052223+0000","last_unstale":"2024-03-06T07:31:45.052295+0000","last_undegraded":"2024-03-06T07:31:45.052295+0000","last_fullsized":"2024-03-06T07:31:45.052295+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041417+0000","last_change":"2024-03-06T07:31:44.031262+0000","last_active":"2024-03-06T07:31:45.041417+0000","last_peered":"2024-03-06T07:31:45.041417+0000","last_clean":"2024-03-06T07:31:45.041417+0000","last_became_active":"2024-03-06T07:31:44.030401+0000","last_became_peered":"2024-03-06T07:31:44.030401+0000","last_unstale":"2024-03-06T07:31:45.041417+0000","last_undegraded":"2024-03-06T07:31:45.041417+0000","last_fullsized":"2024-03-06T07:31:45.041417+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:44.318964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.035328+0000","last_change":"2024-03-06T07:31:45.033444+0000","last_active":"2024-03-06T07:31:45.035328+0000","last_peered":"2024-03-06T07:31:45.035328+0000","last_clean":"2024-03-06T07:31:45.035328+0000","last_became_active":"2024-03-06T07:31:45.033109+0000","last_became_peered":"2024-03-06T07:31:45.033109+0000","last_unstale":"2024-03-06T07:31:45.035328+0000","last_undegraded":"2024-03-06T07:31:45.035328+0000","last_fullsized":"2024-03-06T07:31:45.035328+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:45.246725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"144'5","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029309+0000","last_change":"2024-03-06T07:31:44.026781+0000","last_active":"2024-03-06T07:31:45.029309+0000","last_peered":"2024-03-06T07:31:45.029309+0000","last_clean":"2024-03-06T07:31:45.029309+0000","last_became_active":"2024-03-06T07:31:44.026495+0000","last_became_peered":"2024-03-06T07:31:44.026495+0000","last_unstale":"2024-03-06T07:31:45.029309+0000","last_undegraded":"2024-03-06T07:31:45.029309+0000","last_fullsized":"2024-03-06T07:31:45.029309+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:35.090123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.043899+0000","last_change":"2024-03-06T07:31:45.043899+0000","last_active":"2024-03-06T07:31:45.043899+0000","last_peered":"2024-03-06T07:31:45.043899+0000","last_clean":"2024-03-06T07:31:45.043899+0000","last_became_active":"2024-03-06T07:31:45.043131+0000","last_became_peered":"2024-03-06T07:31:45.043131+0000","last_unstale":"2024-03-06T07:31:45.043899+0000","last_undegraded":"2024-03-06T07:31:45.043899+0000","last_fullsized":"2024-03-06T07:31:45.043899+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:13.567550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.035787+0000","last_change":"2024-03-06T07:31:44.026616+0000","last_active":"2024-03-06T07:31:45.035787+0000","last_peered":"2024-03-06T07:31:45.035787+0000","last_clean":"2024-03-06T07:31:45.035787+0000","last_became_active":"2024-03-06T07:31:44.025510+0000","last_became_peered":"2024-03-06T07:31:44.025510+0000","last_unstale":"2024-03-06T07:31:45.035787+0000","last_undegraded":"2024-03-06T07:31:45.035787+0000","last_fullsized":"2024-03-06T07:31:45.035787+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:13.314073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041996+0000","last_change":"2024-03-06T07:31:45.041996+0000","last_active":"2024-03-06T07:31:45.041996+0000","last_peered":"2024-03-06T07:31:45.041996+0000","last_clean":"2024-03-06T07:31:45.041996+0000","last_became_active":"2024-03-06T07:31:45.041403+0000","last_became_peered":"2024-03-06T07:31:45.041403+0000","last_unstale":"2024-03-06T07:31:45.041996+0000","last_undegraded":"2024-03-06T07:31:45.041996+0000","last_fullsized":"2024-03-06T07:31:45.041996+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:20.046408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.032932+0000","last_change":"2024-03-06T07:31:44.022857+0000","last_active":"2024-03-06T07:31:45.032932+0000","last_peered":"2024-03-06T07:31:45.032932+0000","last_clean":"2024-03-06T07:31:45.032932+0000","last_became_active":"2024-03-06T07:31:44.022400+0000","last_became_peered":"2024-03-06T07:31:44.022400+0000","last_unstale":"2024-03-06T07:31:45.032932+0000","last_undegraded":"2024-03-06T07:31:45.032932+0000","last_fullsized":"2024-03-06T07:31:45.032932+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:55.478147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044454+0000","last_change":"2024-03-06T07:31:45.042751+0000","last_active":"2024-03-06T07:31:45.044454+0000","last_peered":"2024-03-06T07:31:45.044454+0000","last_clean":"2024-03-06T07:31:45.044454+0000","last_became_active":"2024-03-06T07:31:45.042477+0000","last_became_peered":"2024-03-06T07:31:45.042477+0000","last_unstale":"2024-03-06T07:31:45.044454+0000","last_undegraded":"2024-03-06T07:31:45.044454+0000","last_fullsized":"2024-03-06T07:31:45.044454+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:43.063911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029362+0000","last_change":"2024-03-06T07:31:44.023105+0000","last_active":"2024-03-06T07:31:45.029362+0000","last_peered":"2024-03-06T07:31:45.029362+0000","last_clean":"2024-03-06T07:31:45.029362+0000","last_became_active":"2024-03-06T07:31:44.022808+0000","last_became_peered":"2024-03-06T07:31:44.022808+0000","last_unstale":"2024-03-06T07:31:45.029362+0000","last_undegraded":"2024-03-06T07:31:45.029362+0000","last_fullsized":"2024-03-06T07:31:45.029362+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.047987+0000","last_change":"2024-03-06T07:31:45.047987+0000","last_active":"2024-03-06T07:31:45.047987+0000","last_peered":"2024-03-06T07:31:45.047987+0000","last_clean":"2024-03-06T07:31:45.047987+0000","last_became_active":"2024-03-06T07:31:45.045672+0000","last_became_peered":"2024-03-06T07:31:45.045672+0000","last_unstale":"2024-03-06T07:31:45.047987+0000","last_undegraded":"2024-03-06T07:31:45.047987+0000","last_fullsized":"2024-03-06T07:31:45.047987+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:38.092312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025614+0000","last_change":"2024-03-06T07:31:44.024742+0000","last_active":"2024-03-06T07:31:45.025614+0000","last_peered":"2024-03-06T07:31:45.025614+0000","last_clean":"2024-03-06T07:31:45.025614+0000","last_became_active":"2024-03-06T07:31:44.024445+0000","last_became_peered":"2024-03-06T07:31:44.024445+0000","last_unstale":"2024-03-06T07:31:45.025614+0000","last_undegraded":"2024-03-06T07:31:45.025614+0000","last_fullsized":"2024-03-06T07:31:45.025614+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:00.838367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":6,"reported_epoch":144,"state":"active","last_fresh":"2024-03-06T07:31:45.051855+0000","last_change":"2024-03-06T07:31:45.051795+0000","last_active":"2024-03-06T07:31:45.051855+0000","last_peered":"2024-03-06T07:31:45.051855+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"2024-03-06T07:31:45.051795+0000","last_became_peered":"2024-03-06T07:31:45.051795+0000","last_unstale":"2024-03-06T07:31:45.051855+0000","last_undegraded":"2024-03-06T07:31:45.051855+0000","last_fullsized":"2024-03-06T07:31:45.051855+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"144'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029593+0000","last_change":"2024-03-06T07:31:44.023389+0000","last_active":"2024-03-06T07:31:45.029593+0000","last_peered":"2024-03-06T07:31:45.029593+0000","last_clean":"2024-03-06T07:31:45.029593+0000","last_became_active":"2024-03-06T07:31:44.023124+0000","last_became_peered":"2024-03-06T07:31:44.023124+0000","last_unstale":"2024-03-06T07:31:45.029593+0000","last_undegraded":"2024-03-06T07:31:45.029593+0000","last_fullsized":"2024-03-06T07:31:45.029593+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044088+0000","last_change":"2024-03-06T07:31:45.044088+0000","last_active":"2024-03-06T07:31:45.044088+0000","last_peered":"2024-03-06T07:31:45.044088+0000","last_clean":"2024-03-06T07:31:45.044088+0000","last_became_active":"2024-03-06T07:31:45.043830+0000","last_became_peered":"2024-03-06T07:31:45.043830+0000","last_unstale":"2024-03-06T07:31:45.044088+0000","last_undegraded":"2024-03-06T07:31:45.044088+0000","last_fullsized":"2024-03-06T07:31:45.044088+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:40.063434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031115+0000","last_change":"2024-03-06T07:31:44.024253+0000","last_active":"2024-03-06T07:31:45.031115+0000","last_peered":"2024-03-06T07:31:45.031115+0000","last_clean":"2024-03-06T07:31:45.031115+0000","last_became_active":"2024-03-06T07:31:44.023842+0000","last_became_peered":"2024-03-06T07:31:44.023842+0000","last_unstale":"2024-03-06T07:31:45.031115+0000","last_undegraded":"2024-03-06T07:31:45.031115+0000","last_fullsized":"2024-03-06T07:31:45.031115+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045694+0000","last_change":"2024-03-06T07:31:45.045694+0000","last_active":"2024-03-06T07:31:45.045694+0000","last_peered":"2024-03-06T07:31:45.045694+0000","last_clean":"2024-03-06T07:31:45.045694+0000","last_became_active":"2024-03-06T07:31:45.044744+0000","last_became_peered":"2024-03-06T07:31:45.044744+0000","last_unstale":"2024-03-06T07:31:45.045694+0000","last_undegraded":"2024-03-06T07:31:45.045694+0000","last_fullsized":"2024-03-06T07:31:45.045694+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:42.160169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025821+0000","last_change":"2024-03-06T07:31:44.035542+0000","last_active":"2024-03-06T07:31:45.025821+0000","last_peered":"2024-03-06T07:31:45.025821+0000","last_clean":"2024-03-06T07:31:45.025821+0000","last_became_active":"2024-03-06T07:31:44.035318+0000","last_became_peered":"2024-03-06T07:31:44.035318+0000","last_unstale":"2024-03-06T07:31:45.025821+0000","last_undegraded":"2024-03-06T07:31:45.025821+0000","last_fullsized":"2024-03-06T07:31:45.025821+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:03.773449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.029060+0000","last_change":"2024-03-06T07:31:45.029060+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.029060+0000","last_undegraded":"2024-03-06T07:31:45.029060+0000","last_fullsized":"2024-03-06T07:31:45.029060+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.030954+0000","last_change":"2024-03-06T07:31:44.028091+0000","last_active":"2024-03-06T07:31:45.030954+0000","last_peered":"2024-03-06T07:31:45.030954+0000","last_clean":"2024-03-06T07:31:45.030954+0000","last_became_active":"2024-03-06T07:31:44.027829+0000","last_became_peered":"2024-03-06T07:31:44.027829+0000","last_unstale":"2024-03-06T07:31:45.030954+0000","last_undegraded":"2024-03-06T07:31:45.030954+0000","last_fullsized":"2024-03-06T07:31:45.030954+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:08.912427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.043408+0000","last_change":"2024-03-06T07:31:45.039966+0000","last_active":"2024-03-06T07:31:45.043408+0000","last_peered":"2024-03-06T07:31:45.043408+0000","last_clean":"2024-03-06T07:31:45.043408+0000","last_became_active":"2024-03-06T07:31:45.039079+0000","last_became_peered":"2024-03-06T07:31:45.039079+0000","last_unstale":"2024-03-06T07:31:45.043408+0000","last_undegraded":"2024-03-06T07:31:45.043408+0000","last_fullsized":"2024-03-06T07:31:45.043408+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:56.482865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029577+0000","last_change":"2024-03-06T07:31:44.024320+0000","last_active":"2024-03-06T07:31:45.029577+0000","last_peered":"2024-03-06T07:31:45.029577+0000","last_clean":"2024-03-06T07:31:45.029577+0000","last_became_active":"2024-03-06T07:31:44.024051+0000","last_became_peered":"2024-03-06T07:31:44.024051+0000","last_unstale":"2024-03-06T07:31:45.029577+0000","last_undegraded":"2024-03-06T07:31:45.029577+0000","last_fullsized":"2024-03-06T07:31:45.029577+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:59.753052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040276+0000","last_change":"2024-03-06T07:31:45.035119+0000","last_active":"2024-03-06T07:31:45.040276+0000","last_peered":"2024-03-06T07:31:45.040276+0000","last_clean":"2024-03-06T07:31:45.040276+0000","last_became_active":"2024-03-06T07:31:45.034850+0000","last_became_peered":"2024-03-06T07:31:45.034850+0000","last_unstale":"2024-03-06T07:31:45.040276+0000","last_undegraded":"2024-03-06T07:31:45.040276+0000","last_fullsized":"2024-03-06T07:31:45.040276+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:34.247852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041190+0000","last_change":"2024-03-06T07:31:44.031164+0000","last_active":"2024-03-06T07:31:45.041190+0000","last_peered":"2024-03-06T07:31:45.041190+0000","last_clean":"2024-03-06T07:31:45.041190+0000","last_became_active":"2024-03-06T07:31:44.030649+0000","last_became_peered":"2024-03-06T07:31:44.030649+0000","last_unstale":"2024-03-06T07:31:45.041190+0000","last_undegraded":"2024-03-06T07:31:45.041190+0000","last_fullsized":"2024-03-06T07:31:45.041190+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.047269+0000","last_change":"2024-03-06T07:31:45.043562+0000","last_active":"2024-03-06T07:31:45.047269+0000","last_peered":"2024-03-06T07:31:45.047269+0000","last_clean":"2024-03-06T07:31:45.047269+0000","last_became_active":"2024-03-06T07:31:45.043293+0000","last_became_peered":"2024-03-06T07:31:45.043293+0000","last_unstale":"2024-03-06T07:31:45.047269+0000","last_undegraded":"2024-03-06T07:31:45.047269+0000","last_fullsized":"2024-03-06T07:31:45.047269+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:14.318792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.038382+0000","last_change":"2024-03-06T07:31:44.032582+0000","last_active":"2024-03-06T07:31:45.038382+0000","last_peered":"2024-03-06T07:31:45.038382+0000","last_clean":"2024-03-06T07:31:45.038382+0000","last_became_active":"2024-03-06T07:31:44.032075+0000","last_became_peered":"2024-03-06T07:31:44.032075+0000","last_unstale":"2024-03-06T07:31:45.038382+0000","last_undegraded":"2024-03-06T07:31:45.038382+0000","last_fullsized":"2024-03-06T07:31:45.038382+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040239+0000","last_change":"2024-03-06T07:31:45.035199+0000","last_active":"2024-03-06T07:31:45.040239+0000","last_peered":"2024-03-06T07:31:45.040239+0000","last_clean":"2024-03-06T07:31:45.040239+0000","last_became_active":"2024-03-06T07:31:45.034925+0000","last_became_peered":"2024-03-06T07:31:45.034925+0000","last_unstale":"2024-03-06T07:31:45.040239+0000","last_undegraded":"2024-03-06T07:31:45.040239+0000","last_fullsized":"2024-03-06T07:31:45.040239+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034897+0000","last_change":"2024-03-06T07:31:44.031018+0000","last_active":"2024-03-06T07:31:45.034897+0000","last_peered":"2024-03-06T07:31:45.034897+0000","last_clean":"2024-03-06T07:31:45.034897+0000","last_became_active":"2024-03-06T07:31:44.030695+0000","last_became_peered":"2024-03-06T07:31:44.030695+0000","last_unstale":"2024-03-06T07:31:45.034897+0000","last_undegraded":"2024-03-06T07:31:45.034897+0000","last_fullsized":"2024-03-06T07:31:45.034897+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:51.791822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.047228+0000","last_change":"2024-03-06T07:31:45.043749+0000","last_active":"2024-03-06T07:31:45.047228+0000","last_peered":"2024-03-06T07:31:45.047228+0000","last_clean":"2024-03-06T07:31:45.047228+0000","last_became_active":"2024-03-06T07:31:45.043501+0000","last_became_peered":"2024-03-06T07:31:45.043501+0000","last_unstale":"2024-03-06T07:31:45.047228+0000","last_undegraded":"2024-03-06T07:31:45.047228+0000","last_fullsized":"2024-03-06T07:31:45.047228+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.690561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041947+0000","last_change":"2024-03-06T07:31:44.033647+0000","last_active":"2024-03-06T07:31:45.041947+0000","last_peered":"2024-03-06T07:31:45.041947+0000","last_clean":"2024-03-06T07:31:45.041947+0000","last_became_active":"2024-03-06T07:31:44.033393+0000","last_became_peered":"2024-03-06T07:31:44.033393+0000","last_unstale":"2024-03-06T07:31:45.041947+0000","last_undegraded":"2024-03-06T07:31:45.041947+0000","last_fullsized":"2024-03-06T07:31:45.041947+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:35.425689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045750+0000","last_change":"2024-03-06T07:31:45.045750+0000","last_active":"2024-03-06T07:31:45.045750+0000","last_peered":"2024-03-06T07:31:45.045750+0000","last_clean":"2024-03-06T07:31:45.045750+0000","last_became_active":"2024-03-06T07:31:45.044153+0000","last_became_peered":"2024-03-06T07:31:45.044153+0000","last_unstale":"2024-03-06T07:31:45.045750+0000","last_undegraded":"2024-03-06T07:31:45.045750+0000","last_fullsized":"2024-03-06T07:31:45.045750+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:26.528565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.030989+0000","last_change":"2024-03-06T07:31:44.028776+0000","last_active":"2024-03-06T07:31:45.030989+0000","last_peered":"2024-03-06T07:31:45.030989+0000","last_clean":"2024-03-06T07:31:45.030989+0000","last_became_active":"2024-03-06T07:31:44.028504+0000","last_became_peered":"2024-03-06T07:31:44.028504+0000","last_unstale":"2024-03-06T07:31:45.030989+0000","last_undegraded":"2024-03-06T07:31:45.030989+0000","last_fullsized":"2024-03-06T07:31:45.030989+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:17.358709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040051+0000","last_change":"2024-03-06T07:31:45.040051+0000","last_active":"2024-03-06T07:31:45.040051+0000","last_peered":"2024-03-06T07:31:45.040051+0000","last_clean":"2024-03-06T07:31:45.040051+0000","last_became_active":"2024-03-06T07:31:45.039760+0000","last_became_peered":"2024-03-06T07:31:45.039760+0000","last_unstale":"2024-03-06T07:31:45.040051+0000","last_undegraded":"2024-03-06T07:31:45.040051+0000","last_fullsized":"2024-03-06T07:31:45.040051+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:36.430408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029750+0000","last_change":"2024-03-06T07:31:44.027749+0000","last_active":"2024-03-06T07:31:45.029750+0000","last_peered":"2024-03-06T07:31:45.029750+0000","last_clean":"2024-03-06T07:31:45.029750+0000","last_became_active":"2024-03-06T07:31:44.027413+0000","last_became_peered":"2024-03-06T07:31:44.027413+0000","last_unstale":"2024-03-06T07:31:45.029750+0000","last_undegraded":"2024-03-06T07:31:45.029750+0000","last_fullsized":"2024-03-06T07:31:45.029750+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:26.520317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037784+0000","last_change":"2024-03-06T07:31:45.036246+0000","last_active":"2024-03-06T07:31:45.037784+0000","last_peered":"2024-03-06T07:31:45.037784+0000","last_clean":"2024-03-06T07:31:45.037784+0000","last_became_active":"2024-03-06T07:31:45.035684+0000","last_became_peered":"2024-03-06T07:31:45.035684+0000","last_unstale":"2024-03-06T07:31:45.037784+0000","last_undegraded":"2024-03-06T07:31:45.037784+0000","last_fullsized":"2024-03-06T07:31:45.037784+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:19.547257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.030746+0000","last_change":"2024-03-06T07:31:44.027642+0000","last_active":"2024-03-06T07:31:45.030746+0000","last_peered":"2024-03-06T07:31:45.030746+0000","last_clean":"2024-03-06T07:31:45.030746+0000","last_became_active":"2024-03-06T07:31:44.027355+0000","last_became_peered":"2024-03-06T07:31:44.027355+0000","last_unstale":"2024-03-06T07:31:45.030746+0000","last_undegraded":"2024-03-06T07:31:45.030746+0000","last_fullsized":"2024-03-06T07:31:45.030746+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.058715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037724+0000","last_change":"2024-03-06T07:31:45.034848+0000","last_active":"2024-03-06T07:31:45.037724+0000","last_peered":"2024-03-06T07:31:45.037724+0000","last_clean":"2024-03-06T07:31:45.037724+0000","last_became_active":"2024-03-06T07:31:45.034565+0000","last_became_peered":"2024-03-06T07:31:45.034565+0000","last_unstale":"2024-03-06T07:31:45.037724+0000","last_undegraded":"2024-03-06T07:31:45.037724+0000","last_fullsized":"2024-03-06T07:31:45.037724+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:39.939282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029755+0000","last_change":"2024-03-06T07:31:44.025967+0000","last_active":"2024-03-06T07:31:45.029755+0000","last_peered":"2024-03-06T07:31:45.029755+0000","last_clean":"2024-03-06T07:31:45.029755+0000","last_became_active":"2024-03-06T07:31:44.025640+0000","last_became_peered":"2024-03-06T07:31:44.025640+0000","last_unstale":"2024-03-06T07:31:45.029755+0000","last_undegraded":"2024-03-06T07:31:45.029755+0000","last_fullsized":"2024-03-06T07:31:45.029755+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:48.746990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.048574+0000","last_change":"2024-03-06T07:31:45.045298+0000","last_active":"2024-03-06T07:31:45.048574+0000","last_peered":"2024-03-06T07:31:45.048574+0000","last_clean":"2024-03-06T07:31:45.048574+0000","last_became_active":"2024-03-06T07:31:45.045033+0000","last_became_peered":"2024-03-06T07:31:45.045033+0000","last_unstale":"2024-03-06T07:31:45.048574+0000","last_undegraded":"2024-03-06T07:31:45.048574+0000","last_fullsized":"2024-03-06T07:31:45.048574+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:19.547257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040670+0000","last_change":"2024-03-06T07:31:44.032074+0000","last_active":"2024-03-06T07:31:45.040670+0000","last_peered":"2024-03-06T07:31:45.040670+0000","last_clean":"2024-03-06T07:31:45.040670+0000","last_became_active":"2024-03-06T07:31:44.031715+0000","last_became_peered":"2024-03-06T07:31:44.031715+0000","last_unstale":"2024-03-06T07:31:45.040670+0000","last_undegraded":"2024-03-06T07:31:45.040670+0000","last_fullsized":"2024-03-06T07:31:45.040670+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.040107+0000","last_change":"2024-03-06T07:31:45.040107+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.040107+0000","last_undegraded":"2024-03-06T07:31:45.040107+0000","last_fullsized":"2024-03-06T07:31:45.040107+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034395+0000","last_change":"2024-03-06T07:31:44.025352+0000","last_active":"2024-03-06T07:31:45.034395+0000","last_peered":"2024-03-06T07:31:45.034395+0000","last_clean":"2024-03-06T07:31:45.034395+0000","last_became_active":"2024-03-06T07:31:44.025043+0000","last_became_peered":"2024-03-06T07:31:44.025043+0000","last_unstale":"2024-03-06T07:31:45.034395+0000","last_undegraded":"2024-03-06T07:31:45.034395+0000","last_fullsized":"2024-03-06T07:31:45.034395+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:59.681311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037968+0000","last_change":"2024-03-06T07:31:45.035827+0000","last_active":"2024-03-06T07:31:45.037968+0000","last_peered":"2024-03-06T07:31:45.037968+0000","last_clean":"2024-03-06T07:31:45.037968+0000","last_became_active":"2024-03-06T07:31:45.035484+0000","last_became_peered":"2024-03-06T07:31:45.035484+0000","last_unstale":"2024-03-06T07:31:45.037968+0000","last_undegraded":"2024-03-06T07:31:45.037968+0000","last_fullsized":"2024-03-06T07:31:45.037968+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:27.525036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031547+0000","last_change":"2024-03-06T07:31:44.027942+0000","last_active":"2024-03-06T07:31:45.031547+0000","last_peered":"2024-03-06T07:31:45.031547+0000","last_clean":"2024-03-06T07:31:45.031547+0000","last_became_active":"2024-03-06T07:31:44.027666+0000","last_became_peered":"2024-03-06T07:31:44.027666+0000","last_unstale":"2024-03-06T07:31:45.031547+0000","last_undegraded":"2024-03-06T07:31:45.031547+0000","last_fullsized":"2024-03-06T07:31:45.031547+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040316+0000","last_change":"2024-03-06T07:31:45.035716+0000","last_active":"2024-03-06T07:31:45.040316+0000","last_peered":"2024-03-06T07:31:45.040316+0000","last_clean":"2024-03-06T07:31:45.040316+0000","last_became_active":"2024-03-06T07:31:45.035437+0000","last_became_peered":"2024-03-06T07:31:45.035437+0000","last_unstale":"2024-03-06T07:31:45.040316+0000","last_undegraded":"2024-03-06T07:31:45.040316+0000","last_fullsized":"2024-03-06T07:31:45.040316+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:20.551707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.038657+0000","last_change":"2024-03-06T07:31:44.031865+0000","last_active":"2024-03-06T07:31:45.038657+0000","last_peered":"2024-03-06T07:31:45.038657+0000","last_clean":"2024-03-06T07:31:45.038657+0000","last_became_active":"2024-03-06T07:31:44.030846+0000","last_became_peered":"2024-03-06T07:31:44.030846+0000","last_unstale":"2024-03-06T07:31:45.038657+0000","last_undegraded":"2024-03-06T07:31:45.038657+0000","last_fullsized":"2024-03-06T07:31:45.038657+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.034911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":407,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040477+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:31:45.040477+0000","last_peered":"2024-03-06T07:31:45.040477+0000","last_clean":"2024-03-06T07:31:45.040477+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:31:45.040477+0000","last_undegraded":"2024-03-06T07:31:45.040477+0000","last_fullsized":"2024-03-06T07:31:45.040477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574961,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41832,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330008,"statfs":{"total":96636764160,"available":96593928192,"internally_reserved":0,"allocated":2383872,"data_stored":1371971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574962,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574961,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41252,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330588,"statfs":{"total":96636764160,"available":96594522112,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574962,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41836,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330004,"statfs":{"total":96636764160,"available":96593924096,"internally_reserved":0,"allocated":2383872,"data_stored":1371971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:49.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:49.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.614+0000 7fa8c9be2640 1 Processor -- start 2024-03-06T07:31:49.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.614+0000 7fa8c9be2640 1 -- start start 2024-03-06T07:31:49.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.614+0000 7fa8c9be2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8c415df00 0x7fa8c40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.614+0000 7fa8c9be2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa8c40b9a30 0x7fa8c40d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.614+0000 7fa8c9be2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c40cea30 con 0x7fa8c40b95a0 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.614+0000 7fa8c9be2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c40d2af0 con 0x7fa8c415df00 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.614+0000 7fa8c9be2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c40cf3c0 con 0x7fa8c40b9a30 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.615+0000 7fa8c37fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8c415df00 0x7fa8c40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.615+0000 7fa8c37fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8c415df00 0x7fa8c40d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36628/0 (socket says 172.21.15.27:36628) 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.615+0000 7fa8c37fe640 1 -- 172.21.15.27:0/919347632 learned_addr learned my addr 172.21.15.27:0/919347632 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.615+0000 7fa8c3fff640 1 --2- 172.21.15.27:0/919347632 >> v2:172.21.15.69:3300/0 conn(0x7fa8c40b9a30 0x7fa8c40d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.615+0000 7fa8c37fe640 1 -- 172.21.15.27:0/919347632 >> v2:172.21.15.69:3300/0 conn(0x7fa8c40b9a30 msgr2=0x7fa8c40d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.615+0000 7fa8c37fe640 1 --2- 172.21.15.27:0/919347632 >> v2:172.21.15.69:3300/0 conn(0x7fa8c40b9a30 0x7fa8c40d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.615+0000 7fa8c37fe640 1 -- 172.21.15.27:0/919347632 >> v1:172.21.15.69:6789/0 conn(0x7fa8c40b95a0 legacy=0x7fa8c40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.615+0000 7fa8c37fe640 1 -- 172.21.15.27:0/919347632 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c40cf0e0 con 0x7fa8c415df00 2024-03-06T07:31:49.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.615+0000 7fa8c3fff640 1 --2- 172.21.15.27:0/919347632 >> v2:172.21.15.69:3300/0 conn(0x7fa8c40b9a30 0x7fa8c40d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:49.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c37fe640 1 --2- 172.21.15.27:0/919347632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c415df00 0x7fa8c40d25b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40d4d20 tx=0x7fa8b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28015ecec3e9bf86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c27fc640 1 -- 172.21.15.27:0/919347632 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4034070 con 0x7fa8c415df00 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c27fc640 1 -- 172.21.15.27:0/919347632 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b4030860 con 0x7fa8c415df00 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c27fc640 1 -- 172.21.15.27:0/919347632 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4030c30 con 0x7fa8c415df00 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c9be2640 1 -- 172.21.15.27:0/919347632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c415df00 msgr2=0x7fa8c40d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c9be2640 1 --2- 172.21.15.27:0/919347632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c415df00 0x7fa8c40d25b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40d4d20 tx=0x7fa8b402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c9be2640 1 -- 172.21.15.27:0/919347632 shutdown_connections 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c9be2640 1 --2- 172.21.15.27:0/919347632 >> v2:172.21.15.69:3300/0 conn(0x7fa8c40b9a30 0x7fa8c40d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c9be2640 1 --2- 172.21.15.27:0/919347632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c415df00 0x7fa8c40d25b0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c9be2640 1 -- 172.21.15.27:0/919347632 >> 172.21.15.27:0/919347632 conn(0x7fa8c4071960 msgr2=0x7fa8c4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.616+0000 7fa8c9be2640 1 -- 172.21.15.27:0/919347632 shutdown_connections 2024-03-06T07:31:49.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.617+0000 7fa8c9be2640 1 -- 172.21.15.27:0/919347632 wait complete. 2024-03-06T07:31:49.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.618+0000 7fa8c9be2640 1 Processor -- start 2024-03-06T07:31:49.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.619+0000 7fa8c9be2640 1 -- start start 2024-03-06T07:31:49.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c9be2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c40b95a0 0x7fa8c41f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c9be2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c415df00 0x7fa8c41f4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c9be2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8c40c9420 0x7fa8c40c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c9be2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40cec70 con 0x7fa8c40b95a0 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c9be2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40ce750 con 0x7fa8c415df00 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c9be2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40cedf0 con 0x7fa8c40c9420 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c40b95a0 0x7fa8c41f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c40b95a0 0x7fa8c41f68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36632/0 (socket says 172.21.15.27:36632) 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c37fe640 1 -- 172.21.15.27:0/280348879 learned_addr learned my addr 172.21.15.27:0/280348879 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c2ffd640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c415df00 0x7fa8c41f4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c3fff640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8c40c9420 0x7fa8c40c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c37fe640 1 -- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8c40c9420 msgr2=0x7fa8c40c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c37fe640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8c40c9420 0x7fa8c40c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c37fe640 1 -- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c415df00 msgr2=0x7fa8c41f4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c37fe640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c415df00 0x7fa8c41f4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.620+0000 7fa8c37fe640 1 -- 172.21.15.27:0/280348879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c40c9ee0 con 0x7fa8c40b95a0 2024-03-06T07:31:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.621+0000 7fa8c37fe640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c40b95a0 0x7fa8c41f68b0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fa8b402b9e0 tx=0x7fa8b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.621+0000 7fa8abfff640 1 -- 172.21.15.27:0/280348879 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4034040 con 0x7fa8c40b95a0 2024-03-06T07:31:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.621+0000 7fa8abfff640 1 -- 172.21.15.27:0/280348879 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b4039d90 con 0x7fa8c40b95a0 2024-03-06T07:31:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.621+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c40cadf0 con 0x7fa8c40b95a0 2024-03-06T07:31:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.621+0000 7fa8abfff640 1 -- 172.21.15.27:0/280348879 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4043610 con 0x7fa8c40b95a0 2024-03-06T07:31:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.621+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c40cb3b0 con 0x7fa8c40b95a0 2024-03-06T07:31:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.622+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c4067ff0 con 0x7fa8c40b95a0 2024-03-06T07:31:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.622+0000 7fa8abfff640 1 -- 172.21.15.27:0/280348879 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8b404a060 con 0x7fa8c40b95a0 2024-03-06T07:31:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.623+0000 7fa8abfff640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8b006b4d0 0x7fa8b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.623+0000 7fa8abfff640 1 -- 172.21.15.27:0/280348879 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8b40ac1f0 con 0x7fa8c40b95a0 2024-03-06T07:31:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.623+0000 7fa8c2ffd640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8b006b4d0 0x7fa8b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.624+0000 7fa8c2ffd640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8b006b4d0 0x7fa8b006d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40cdd50 tx=0x7fa8b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.627+0000 7fa8abfff640 1 -- 172.21.15.27:0/280348879 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8b403e070 con 0x7fa8c40b95a0 2024-03-06T07:31:49.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:49.679 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.851+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa8c415c9e0 con 0x7fa8b006b4d0 2024-03-06T07:31:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.857+0000 7fa8abfff640 1 -- 172.21.15.27:0/280348879 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251666 (secure 0 0 0) 0x7fa8c415c9e0 con 0x7fa8b006b4d0 2024-03-06T07:31:49.861 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:49.862 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:31:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8b006b4d0 msgr2=0x7fa8b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8b006b4d0 0x7fa8b006d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40cdd50 tx=0x7fa8b8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c40b95a0 msgr2=0x7fa8c41f68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c40b95a0 0x7fa8c41f68b0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fa8b402b9e0 tx=0x7fa8b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 shutdown_connections 2024-03-06T07:31:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa8b006b4d0 0x7fa8b006d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa8c40c9420 0x7fa8c40c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c415df00 0x7fa8c41f4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 --2- 172.21.15.27:0/280348879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c40b95a0 0x7fa8c41f68b0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 >> 172.21.15.27:0/280348879 conn(0x7fa8c4071960 msgr2=0x7fa8c40b8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 shutdown_connections 2024-03-06T07:31:49.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:49.860+0000 7fa8c9be2640 1 -- 172.21.15.27:0/280348879 wait complete. 2024-03-06T07:31:49.873 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-03-06T07:31:48.119006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":166,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331160,"kb_used_data":15144,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643560,"statfs":{"total":773094113280,"available":772755005440,"internally_reserved":0,"allocated":15507456,"data_stored":7433560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.109582"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046476+0000","last_change":"2024-03-06T07:31:45.046476+0000","last_active":"2024-03-06T07:31:45.046476+0000","last_peered":"2024-03-06T07:31:45.046476+0000","last_clean":"2024-03-06T07:31:45.046476+0000","last_became_active":"2024-03-06T07:31:45.046179+0000","last_became_peered":"2024-03-06T07:31:45.046179+0000","last_unstale":"2024-03-06T07:31:45.046476+0000","last_undegraded":"2024-03-06T07:31:45.046476+0000","last_fullsized":"2024-03-06T07:31:45.046476+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:28.329642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041710+0000","last_change":"2024-03-06T07:31:45.041710+0000","last_active":"2024-03-06T07:31:45.041710+0000","last_peered":"2024-03-06T07:31:45.041710+0000","last_clean":"2024-03-06T07:31:45.041710+0000","last_became_active":"2024-03-06T07:31:45.041413+0000","last_became_peered":"2024-03-06T07:31:45.041413+0000","last_unstale":"2024-03-06T07:31:45.041710+0000","last_undegraded":"2024-03-06T07:31:45.041710+0000","last_fullsized":"2024-03-06T07:31:45.041710+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:53.307481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.048666+0000","last_change":"2024-03-06T07:31:45.048666+0000","last_active":"2024-03-06T07:31:45.048666+0000","last_peered":"2024-03-06T07:31:45.048666+0000","last_clean":"2024-03-06T07:31:45.048666+0000","last_became_active":"2024-03-06T07:31:45.044404+0000","last_became_peered":"2024-03-06T07:31:45.044404+0000","last_unstale":"2024-03-06T07:31:45.048666+0000","last_undegraded":"2024-03-06T07:31:45.048666+0000","last_fullsized":"2024-03-06T07:31:45.048666+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:55.936260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046599+0000","last_change":"2024-03-06T07:31:45.046599+0000","last_active":"2024-03-06T07:31:45.046599+0000","last_peered":"2024-03-06T07:31:45.046599+0000","last_clean":"2024-03-06T07:31:45.046599+0000","last_became_active":"2024-03-06T07:31:45.046251+0000","last_became_peered":"2024-03-06T07:31:45.046251+0000","last_unstale":"2024-03-06T07:31:45.046599+0000","last_undegraded":"2024-03-06T07:31:45.046599+0000","last_fullsized":"2024-03-06T07:31:45.046599+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:13.567550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050368+0000","last_change":"2024-03-06T07:31:45.048513+0000","last_active":"2024-03-06T07:31:45.050368+0000","last_peered":"2024-03-06T07:31:45.050368+0000","last_clean":"2024-03-06T07:31:45.050368+0000","last_became_active":"2024-03-06T07:31:45.048251+0000","last_became_peered":"2024-03-06T07:31:45.048251+0000","last_unstale":"2024-03-06T07:31:45.050368+0000","last_undegraded":"2024-03-06T07:31:45.050368+0000","last_fullsized":"2024-03-06T07:31:45.050368+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:04.808270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.029936+0000","last_change":"2024-03-06T07:31:45.029936+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.029936+0000","last_undegraded":"2024-03-06T07:31:45.029936+0000","last_fullsized":"2024-03-06T07:31:45.029936+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040291+0000","last_change":"2024-03-06T07:31:45.035876+0000","last_active":"2024-03-06T07:31:45.040291+0000","last_peered":"2024-03-06T07:31:45.040291+0000","last_clean":"2024-03-06T07:31:45.040291+0000","last_became_active":"2024-03-06T07:31:45.035604+0000","last_became_peered":"2024-03-06T07:31:45.035604+0000","last_unstale":"2024-03-06T07:31:45.040291+0000","last_undegraded":"2024-03-06T07:31:45.040291+0000","last_fullsized":"2024-03-06T07:31:45.040291+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:28.333006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037397+0000","last_change":"2024-03-06T07:31:45.035300+0000","last_active":"2024-03-06T07:31:45.037397+0000","last_peered":"2024-03-06T07:31:45.037397+0000","last_clean":"2024-03-06T07:31:45.037397+0000","last_became_active":"2024-03-06T07:31:45.034964+0000","last_became_peered":"2024-03-06T07:31:45.034964+0000","last_unstale":"2024-03-06T07:31:45.037397+0000","last_undegraded":"2024-03-06T07:31:45.037397+0000","last_fullsized":"2024-03-06T07:31:45.037397+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:43.063911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.039202+0000","last_change":"2024-03-06T07:31:45.039202+0000","last_active":"2024-03-06T07:31:45.039202+0000","last_peered":"2024-03-06T07:31:45.039202+0000","last_clean":"2024-03-06T07:31:45.039202+0000","last_became_active":"2024-03-06T07:31:45.038902+0000","last_became_peered":"2024-03-06T07:31:45.038902+0000","last_unstale":"2024-03-06T07:31:45.039202+0000","last_undegraded":"2024-03-06T07:31:45.039202+0000","last_fullsized":"2024-03-06T07:31:45.039202+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:44.736535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050634+0000","last_change":"2024-03-06T07:31:45.047007+0000","last_active":"2024-03-06T07:31:45.050634+0000","last_peered":"2024-03-06T07:31:45.050634+0000","last_clean":"2024-03-06T07:31:45.050634+0000","last_became_active":"2024-03-06T07:31:45.046533+0000","last_became_peered":"2024-03-06T07:31:45.046533+0000","last_unstale":"2024-03-06T07:31:45.050634+0000","last_undegraded":"2024-03-06T07:31:45.050634+0000","last_fullsized":"2024-03-06T07:31:45.050634+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:11.401916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037764+0000","last_change":"2024-03-06T07:31:45.036443+0000","last_active":"2024-03-06T07:31:45.037764+0000","last_peered":"2024-03-06T07:31:45.037764+0000","last_clean":"2024-03-06T07:31:45.037764+0000","last_became_active":"2024-03-06T07:31:45.035986+0000","last_became_peered":"2024-03-06T07:31:45.035986+0000","last_unstale":"2024-03-06T07:31:45.037764+0000","last_undegraded":"2024-03-06T07:31:45.037764+0000","last_fullsized":"2024-03-06T07:31:45.037764+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.690561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.051142+0000","last_change":"2024-03-06T07:31:45.049645+0000","last_active":"2024-03-06T07:31:45.051142+0000","last_peered":"2024-03-06T07:31:45.051142+0000","last_clean":"2024-03-06T07:31:45.051142+0000","last_became_active":"2024-03-06T07:31:45.048865+0000","last_became_peered":"2024-03-06T07:31:45.048865+0000","last_unstale":"2024-03-06T07:31:45.051142+0000","last_undegraded":"2024-03-06T07:31:45.051142+0000","last_fullsized":"2024-03-06T07:31:45.051142+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:36.725280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050346+0000","last_change":"2024-03-06T07:31:45.048451+0000","last_active":"2024-03-06T07:31:45.050346+0000","last_peered":"2024-03-06T07:31:45.050346+0000","last_clean":"2024-03-06T07:31:45.050346+0000","last_became_active":"2024-03-06T07:31:45.048188+0000","last_became_peered":"2024-03-06T07:31:45.048188+0000","last_unstale":"2024-03-06T07:31:45.050346+0000","last_undegraded":"2024-03-06T07:31:45.050346+0000","last_fullsized":"2024-03-06T07:31:45.050346+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:17.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044378+0000","last_change":"2024-03-06T07:31:45.042722+0000","last_active":"2024-03-06T07:31:45.044378+0000","last_peered":"2024-03-06T07:31:45.044378+0000","last_clean":"2024-03-06T07:31:45.044378+0000","last_became_active":"2024-03-06T07:31:45.042456+0000","last_became_peered":"2024-03-06T07:31:45.042456+0000","last_unstale":"2024-03-06T07:31:45.044378+0000","last_undegraded":"2024-03-06T07:31:45.044378+0000","last_fullsized":"2024-03-06T07:31:45.044378+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:52.796540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046120+0000","last_change":"2024-03-06T07:31:45.046120+0000","last_active":"2024-03-06T07:31:45.046120+0000","last_peered":"2024-03-06T07:31:45.046120+0000","last_clean":"2024-03-06T07:31:45.046120+0000","last_became_active":"2024-03-06T07:31:45.045855+0000","last_became_peered":"2024-03-06T07:31:45.045855+0000","last_unstale":"2024-03-06T07:31:45.046120+0000","last_undegraded":"2024-03-06T07:31:45.046120+0000","last_fullsized":"2024-03-06T07:31:45.046120+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:21.522756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042236+0000","last_change":"2024-03-06T07:31:45.039353+0000","last_active":"2024-03-06T07:31:45.042236+0000","last_peered":"2024-03-06T07:31:45.042236+0000","last_clean":"2024-03-06T07:31:45.042236+0000","last_became_active":"2024-03-06T07:31:45.039090+0000","last_became_peered":"2024-03-06T07:31:45.039090+0000","last_unstale":"2024-03-06T07:31:45.042236+0000","last_undegraded":"2024-03-06T07:31:45.042236+0000","last_fullsized":"2024-03-06T07:31:45.042236+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:13.414833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.042465+0000","last_change":"2024-03-06T07:31:45.042465+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.042465+0000","last_undegraded":"2024-03-06T07:31:45.042465+0000","last_fullsized":"2024-03-06T07:31:45.042465+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.046112+0000","last_change":"2024-03-06T07:31:45.046112+0000","last_active":"2024-03-06T07:31:45.046112+0000","last_peered":"2024-03-06T07:31:45.046112+0000","last_clean":"2024-03-06T07:31:45.046112+0000","last_became_active":"2024-03-06T07:31:45.043698+0000","last_became_peered":"2024-03-06T07:31:45.043698+0000","last_unstale":"2024-03-06T07:31:45.046112+0000","last_undegraded":"2024-03-06T07:31:45.046112+0000","last_fullsized":"2024-03-06T07:31:45.046112+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:14.807136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.043083+0000","last_change":"2024-03-06T07:31:45.038374+0000","last_active":"2024-03-06T07:31:45.043083+0000","last_peered":"2024-03-06T07:31:45.043083+0000","last_clean":"2024-03-06T07:31:45.043083+0000","last_became_active":"2024-03-06T07:31:45.038088+0000","last_became_peered":"2024-03-06T07:31:45.038088+0000","last_unstale":"2024-03-06T07:31:45.043083+0000","last_undegraded":"2024-03-06T07:31:45.043083+0000","last_fullsized":"2024-03-06T07:31:45.043083+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:04.808270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034187+0000","last_change":"2024-03-06T07:31:45.032953+0000","last_active":"2024-03-06T07:31:45.034187+0000","last_peered":"2024-03-06T07:31:45.034187+0000","last_clean":"2024-03-06T07:31:45.034187+0000","last_became_active":"2024-03-06T07:31:45.032688+0000","last_became_peered":"2024-03-06T07:31:45.032688+0000","last_unstale":"2024-03-06T07:31:45.034187+0000","last_undegraded":"2024-03-06T07:31:45.034187+0000","last_fullsized":"2024-03-06T07:31:45.034187+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:02.496094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.049212+0000","last_change":"2024-03-06T07:31:45.049212+0000","last_active":"2024-03-06T07:31:45.049212+0000","last_peered":"2024-03-06T07:31:45.049212+0000","last_clean":"2024-03-06T07:31:45.049212+0000","last_became_active":"2024-03-06T07:31:45.047359+0000","last_became_peered":"2024-03-06T07:31:45.047359+0000","last_unstale":"2024-03-06T07:31:45.049212+0000","last_undegraded":"2024-03-06T07:31:45.049212+0000","last_fullsized":"2024-03-06T07:31:45.049212+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:05.783188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.038255+0000","last_change":"2024-03-06T07:31:45.035442+0000","last_active":"2024-03-06T07:31:45.038255+0000","last_peered":"2024-03-06T07:31:45.038255+0000","last_clean":"2024-03-06T07:31:45.038255+0000","last_became_active":"2024-03-06T07:31:45.035154+0000","last_became_peered":"2024-03-06T07:31:45.035154+0000","last_unstale":"2024-03-06T07:31:45.038255+0000","last_undegraded":"2024-03-06T07:31:45.038255+0000","last_fullsized":"2024-03-06T07:31:45.038255+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:27.623139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041778+0000","last_change":"2024-03-06T07:31:45.039599+0000","last_active":"2024-03-06T07:31:45.041778+0000","last_peered":"2024-03-06T07:31:45.041778+0000","last_clean":"2024-03-06T07:31:45.041778+0000","last_became_active":"2024-03-06T07:31:45.039297+0000","last_became_peered":"2024-03-06T07:31:45.039297+0000","last_unstale":"2024-03-06T07:31:45.041778+0000","last_undegraded":"2024-03-06T07:31:45.041778+0000","last_fullsized":"2024-03-06T07:31:45.041778+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:33.886298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":6,"reported_epoch":144,"state":"active","last_fresh":"2024-03-06T07:31:45.047191+0000","last_change":"2024-03-06T07:31:45.047127+0000","last_active":"2024-03-06T07:31:45.047191+0000","last_peered":"2024-03-06T07:31:45.047191+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"2024-03-06T07:31:45.047127+0000","last_became_peered":"2024-03-06T07:31:45.047127+0000","last_unstale":"2024-03-06T07:31:45.047191+0000","last_undegraded":"2024-03-06T07:31:45.047191+0000","last_fullsized":"2024-03-06T07:31:45.047191+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042283+0000","last_change":"2024-03-06T07:31:45.040040+0000","last_active":"2024-03-06T07:31:45.042283+0000","last_peered":"2024-03-06T07:31:45.042283+0000","last_clean":"2024-03-06T07:31:45.042283+0000","last_became_active":"2024-03-06T07:31:45.039753+0000","last_became_peered":"2024-03-06T07:31:45.039753+0000","last_unstale":"2024-03-06T07:31:45.042283+0000","last_undegraded":"2024-03-06T07:31:45.042283+0000","last_fullsized":"2024-03-06T07:31:45.042283+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:27.525036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042445+0000","last_change":"2024-03-06T07:31:45.037506+0000","last_active":"2024-03-06T07:31:45.042445+0000","last_peered":"2024-03-06T07:31:45.042445+0000","last_clean":"2024-03-06T07:31:45.042445+0000","last_became_active":"2024-03-06T07:31:45.036896+0000","last_became_peered":"2024-03-06T07:31:45.036896+0000","last_unstale":"2024-03-06T07:31:45.042445+0000","last_undegraded":"2024-03-06T07:31:45.042445+0000","last_fullsized":"2024-03-06T07:31:45.042445+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:17.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.025450+0000","last_change":"2024-03-06T07:31:45.025450+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.025450+0000","last_undegraded":"2024-03-06T07:31:45.025450+0000","last_fullsized":"2024-03-06T07:31:45.025450+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.028666+0000","last_change":"2024-03-06T07:31:45.028666+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.028666+0000","last_undegraded":"2024-03-06T07:31:45.028666+0000","last_fullsized":"2024-03-06T07:31:45.028666+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045636+0000","last_change":"2024-03-06T07:31:45.045636+0000","last_active":"2024-03-06T07:31:45.045636+0000","last_peered":"2024-03-06T07:31:45.045636+0000","last_clean":"2024-03-06T07:31:45.045636+0000","last_became_active":"2024-03-06T07:31:45.045309+0000","last_became_peered":"2024-03-06T07:31:45.045309+0000","last_unstale":"2024-03-06T07:31:45.045636+0000","last_undegraded":"2024-03-06T07:31:45.045636+0000","last_fullsized":"2024-03-06T07:31:45.045636+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:36.113249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044357+0000","last_change":"2024-03-06T07:31:45.041617+0000","last_active":"2024-03-06T07:31:45.044357+0000","last_peered":"2024-03-06T07:31:45.044357+0000","last_clean":"2024-03-06T07:31:45.044357+0000","last_became_active":"2024-03-06T07:31:45.040818+0000","last_became_peered":"2024-03-06T07:31:45.040818+0000","last_unstale":"2024-03-06T07:31:45.044357+0000","last_undegraded":"2024-03-06T07:31:45.044357+0000","last_fullsized":"2024-03-06T07:31:45.044357+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:00.686030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.029698+0000","last_change":"2024-03-06T07:31:45.029698+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.029698+0000","last_undegraded":"2024-03-06T07:31:45.029698+0000","last_fullsized":"2024-03-06T07:31:45.029698+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034138+0000","last_change":"2024-03-06T07:31:45.032208+0000","last_active":"2024-03-06T07:31:45.034138+0000","last_peered":"2024-03-06T07:31:45.034138+0000","last_clean":"2024-03-06T07:31:45.034138+0000","last_became_active":"2024-03-06T07:31:45.031935+0000","last_became_peered":"2024-03-06T07:31:45.031935+0000","last_unstale":"2024-03-06T07:31:45.034138+0000","last_undegraded":"2024-03-06T07:31:45.034138+0000","last_fullsized":"2024-03-06T07:31:45.034138+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:11.848515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050610+0000","last_change":"2024-03-06T07:31:45.046451+0000","last_active":"2024-03-06T07:31:45.050610+0000","last_peered":"2024-03-06T07:31:45.050610+0000","last_clean":"2024-03-06T07:31:45.050610+0000","last_became_active":"2024-03-06T07:31:45.045791+0000","last_became_peered":"2024-03-06T07:31:45.045791+0000","last_unstale":"2024-03-06T07:31:45.050610+0000","last_undegraded":"2024-03-06T07:31:45.050610+0000","last_fullsized":"2024-03-06T07:31:45.050610+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:53.307481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041764+0000","last_change":"2024-03-06T07:31:44.033293+0000","last_active":"2024-03-06T07:31:45.041764+0000","last_peered":"2024-03-06T07:31:45.041764+0000","last_clean":"2024-03-06T07:31:45.041764+0000","last_became_active":"2024-03-06T07:31:44.032831+0000","last_became_peered":"2024-03-06T07:31:44.032831+0000","last_unstale":"2024-03-06T07:31:45.041764+0000","last_undegraded":"2024-03-06T07:31:45.041764+0000","last_fullsized":"2024-03-06T07:31:45.041764+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:43.731816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040911+0000","last_change":"2024-03-06T07:31:45.040911+0000","last_active":"2024-03-06T07:31:45.040911+0000","last_peered":"2024-03-06T07:31:45.040911+0000","last_clean":"2024-03-06T07:31:45.040911+0000","last_became_active":"2024-03-06T07:31:45.040632+0000","last_became_peered":"2024-03-06T07:31:45.040632+0000","last_unstale":"2024-03-06T07:31:45.040911+0000","last_undegraded":"2024-03-06T07:31:45.040911+0000","last_fullsized":"2024-03-06T07:31:45.040911+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:21.522756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031190+0000","last_change":"2024-03-06T07:31:44.027157+0000","last_active":"2024-03-06T07:31:45.031190+0000","last_peered":"2024-03-06T07:31:45.031190+0000","last_clean":"2024-03-06T07:31:45.031190+0000","last_became_active":"2024-03-06T07:31:44.026865+0000","last_became_peered":"2024-03-06T07:31:44.026865+0000","last_unstale":"2024-03-06T07:31:45.031190+0000","last_undegraded":"2024-03-06T07:31:45.031190+0000","last_fullsized":"2024-03-06T07:31:45.031190+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042098+0000","last_change":"2024-03-06T07:31:45.042098+0000","last_active":"2024-03-06T07:31:45.042098+0000","last_peered":"2024-03-06T07:31:45.042098+0000","last_clean":"2024-03-06T07:31:45.042098+0000","last_became_active":"2024-03-06T07:31:45.041801+0000","last_became_peered":"2024-03-06T07:31:45.041801+0000","last_unstale":"2024-03-06T07:31:45.042098+0000","last_undegraded":"2024-03-06T07:31:45.042098+0000","last_fullsized":"2024-03-06T07:31:45.042098+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:48.091616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031296+0000","last_change":"2024-03-06T07:31:44.023697+0000","last_active":"2024-03-06T07:31:45.031296+0000","last_peered":"2024-03-06T07:31:45.031296+0000","last_clean":"2024-03-06T07:31:45.031296+0000","last_became_active":"2024-03-06T07:31:44.023399+0000","last_became_peered":"2024-03-06T07:31:44.023399+0000","last_unstale":"2024-03-06T07:31:45.031296+0000","last_undegraded":"2024-03-06T07:31:45.031296+0000","last_fullsized":"2024-03-06T07:31:45.031296+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:19.201492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.050493+0000","last_change":"2024-03-06T07:31:45.046580+0000","last_active":"2024-03-06T07:31:45.050493+0000","last_peered":"2024-03-06T07:31:45.050493+0000","last_clean":"2024-03-06T07:31:45.050493+0000","last_became_active":"2024-03-06T07:31:45.046312+0000","last_became_peered":"2024-03-06T07:31:45.046312+0000","last_unstale":"2024-03-06T07:31:45.050493+0000","last_undegraded":"2024-03-06T07:31:45.050493+0000","last_fullsized":"2024-03-06T07:31:45.050493+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:20.046408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031181+0000","last_change":"2024-03-06T07:31:44.020513+0000","last_active":"2024-03-06T07:31:45.031181+0000","last_peered":"2024-03-06T07:31:45.031181+0000","last_clean":"2024-03-06T07:31:45.031181+0000","last_became_active":"2024-03-06T07:31:44.020196+0000","last_became_peered":"2024-03-06T07:31:44.020196+0000","last_unstale":"2024-03-06T07:31:45.031181+0000","last_undegraded":"2024-03-06T07:31:45.031181+0000","last_fullsized":"2024-03-06T07:31:45.031181+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:47.307708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.030083+0000","last_change":"2024-03-06T07:31:45.030083+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.030083+0000","last_undegraded":"2024-03-06T07:31:45.030083+0000","last_fullsized":"2024-03-06T07:31:45.030083+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025657+0000","last_change":"2024-03-06T07:31:44.033751+0000","last_active":"2024-03-06T07:31:45.025657+0000","last_peered":"2024-03-06T07:31:45.025657+0000","last_clean":"2024-03-06T07:31:45.025657+0000","last_became_active":"2024-03-06T07:31:44.033386+0000","last_became_peered":"2024-03-06T07:31:44.033386+0000","last_unstale":"2024-03-06T07:31:45.025657+0000","last_undegraded":"2024-03-06T07:31:45.025657+0000","last_fullsized":"2024-03-06T07:31:45.025657+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:50.356019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045822+0000","last_change":"2024-03-06T07:31:45.045822+0000","last_active":"2024-03-06T07:31:45.045822+0000","last_peered":"2024-03-06T07:31:45.045822+0000","last_clean":"2024-03-06T07:31:45.045822+0000","last_became_active":"2024-03-06T07:31:45.045552+0000","last_became_peered":"2024-03-06T07:31:45.045552+0000","last_unstale":"2024-03-06T07:31:45.045822+0000","last_undegraded":"2024-03-06T07:31:45.045822+0000","last_fullsized":"2024-03-06T07:31:45.045822+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:08.039630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"144'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.028727+0000","last_change":"2024-03-06T07:31:44.028730+0000","last_active":"2024-03-06T07:31:45.028727+0000","last_peered":"2024-03-06T07:31:45.028727+0000","last_clean":"2024-03-06T07:31:45.028727+0000","last_became_active":"2024-03-06T07:31:44.028446+0000","last_became_peered":"2024-03-06T07:31:44.028446+0000","last_unstale":"2024-03-06T07:31:45.028727+0000","last_undegraded":"2024-03-06T07:31:45.028727+0000","last_fullsized":"2024-03-06T07:31:45.028727+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:38.934563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.035147+0000","last_change":"2024-03-06T07:31:45.033665+0000","last_active":"2024-03-06T07:31:45.035147+0000","last_peered":"2024-03-06T07:31:45.035147+0000","last_clean":"2024-03-06T07:31:45.035147+0000","last_became_active":"2024-03-06T07:31:45.033369+0000","last_became_peered":"2024-03-06T07:31:45.033369+0000","last_unstale":"2024-03-06T07:31:45.035147+0000","last_undegraded":"2024-03-06T07:31:45.035147+0000","last_fullsized":"2024-03-06T07:31:45.035147+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:45.323683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025219+0000","last_change":"2024-03-06T07:31:44.024794+0000","last_active":"2024-03-06T07:31:45.025219+0000","last_peered":"2024-03-06T07:31:45.025219+0000","last_clean":"2024-03-06T07:31:45.025219+0000","last_became_active":"2024-03-06T07:31:44.024499+0000","last_became_peered":"2024-03-06T07:31:44.024499+0000","last_unstale":"2024-03-06T07:31:45.025219+0000","last_undegraded":"2024-03-06T07:31:45.025219+0000","last_fullsized":"2024-03-06T07:31:45.025219+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:01.912662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.042985+0000","last_change":"2024-03-06T07:31:45.040904+0000","last_active":"2024-03-06T07:31:45.042985+0000","last_peered":"2024-03-06T07:31:45.042985+0000","last_clean":"2024-03-06T07:31:45.042985+0000","last_became_active":"2024-03-06T07:31:45.040606+0000","last_became_peered":"2024-03-06T07:31:45.040606+0000","last_unstale":"2024-03-06T07:31:45.042985+0000","last_undegraded":"2024-03-06T07:31:45.042985+0000","last_fullsized":"2024-03-06T07:31:45.042985+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:18.363428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031359+0000","last_change":"2024-03-06T07:31:44.022269+0000","last_active":"2024-03-06T07:31:45.031359+0000","last_peered":"2024-03-06T07:31:45.031359+0000","last_clean":"2024-03-06T07:31:45.031359+0000","last_became_active":"2024-03-06T07:31:44.021982+0000","last_became_peered":"2024-03-06T07:31:44.021982+0000","last_unstale":"2024-03-06T07:31:45.031359+0000","last_undegraded":"2024-03-06T07:31:45.031359+0000","last_fullsized":"2024-03-06T07:31:45.031359+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:51.573386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.039591+0000","last_change":"2024-03-06T07:31:45.037434+0000","last_active":"2024-03-06T07:31:45.039591+0000","last_peered":"2024-03-06T07:31:45.039591+0000","last_clean":"2024-03-06T07:31:45.039591+0000","last_became_active":"2024-03-06T07:31:45.037148+0000","last_became_peered":"2024-03-06T07:31:45.037148+0000","last_unstale":"2024-03-06T07:31:45.039591+0000","last_undegraded":"2024-03-06T07:31:45.039591+0000","last_fullsized":"2024-03-06T07:31:45.039591+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:40.063434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031021+0000","last_change":"2024-03-06T07:31:44.019467+0000","last_active":"2024-03-06T07:31:45.031021+0000","last_peered":"2024-03-06T07:31:45.031021+0000","last_clean":"2024-03-06T07:31:45.031021+0000","last_became_active":"2024-03-06T07:31:44.019183+0000","last_became_peered":"2024-03-06T07:31:44.019183+0000","last_unstale":"2024-03-06T07:31:45.031021+0000","last_undegraded":"2024-03-06T07:31:45.031021+0000","last_fullsized":"2024-03-06T07:31:45.031021+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:22.325138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":6,"reported_epoch":144,"state":"active","last_fresh":"2024-03-06T07:31:45.052295+0000","last_change":"2024-03-06T07:31:45.052223+0000","last_active":"2024-03-06T07:31:45.052295+0000","last_peered":"2024-03-06T07:31:45.052295+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"2024-03-06T07:31:45.052223+0000","last_became_peered":"2024-03-06T07:31:45.052223+0000","last_unstale":"2024-03-06T07:31:45.052295+0000","last_undegraded":"2024-03-06T07:31:45.052295+0000","last_fullsized":"2024-03-06T07:31:45.052295+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041417+0000","last_change":"2024-03-06T07:31:44.031262+0000","last_active":"2024-03-06T07:31:45.041417+0000","last_peered":"2024-03-06T07:31:45.041417+0000","last_clean":"2024-03-06T07:31:45.041417+0000","last_became_active":"2024-03-06T07:31:44.030401+0000","last_became_peered":"2024-03-06T07:31:44.030401+0000","last_unstale":"2024-03-06T07:31:45.041417+0000","last_undegraded":"2024-03-06T07:31:45.041417+0000","last_fullsized":"2024-03-06T07:31:45.041417+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:44.318964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.035328+0000","last_change":"2024-03-06T07:31:45.033444+0000","last_active":"2024-03-06T07:31:45.035328+0000","last_peered":"2024-03-06T07:31:45.035328+0000","last_clean":"2024-03-06T07:31:45.035328+0000","last_became_active":"2024-03-06T07:31:45.033109+0000","last_became_peered":"2024-03-06T07:31:45.033109+0000","last_unstale":"2024-03-06T07:31:45.035328+0000","last_undegraded":"2024-03-06T07:31:45.035328+0000","last_fullsized":"2024-03-06T07:31:45.035328+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:45.246725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"144'5","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029309+0000","last_change":"2024-03-06T07:31:44.026781+0000","last_active":"2024-03-06T07:31:45.029309+0000","last_peered":"2024-03-06T07:31:45.029309+0000","last_clean":"2024-03-06T07:31:45.029309+0000","last_became_active":"2024-03-06T07:31:44.026495+0000","last_became_peered":"2024-03-06T07:31:44.026495+0000","last_unstale":"2024-03-06T07:31:45.029309+0000","last_undegraded":"2024-03-06T07:31:45.029309+0000","last_fullsized":"2024-03-06T07:31:45.029309+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:35.090123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.043899+0000","last_change":"2024-03-06T07:31:45.043899+0000","last_active":"2024-03-06T07:31:45.043899+0000","last_peered":"2024-03-06T07:31:45.043899+0000","last_clean":"2024-03-06T07:31:45.043899+0000","last_became_active":"2024-03-06T07:31:45.043131+0000","last_became_peered":"2024-03-06T07:31:45.043131+0000","last_unstale":"2024-03-06T07:31:45.043899+0000","last_undegraded":"2024-03-06T07:31:45.043899+0000","last_fullsized":"2024-03-06T07:31:45.043899+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:13.567550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.035787+0000","last_change":"2024-03-06T07:31:44.026616+0000","last_active":"2024-03-06T07:31:45.035787+0000","last_peered":"2024-03-06T07:31:45.035787+0000","last_clean":"2024-03-06T07:31:45.035787+0000","last_became_active":"2024-03-06T07:31:44.025510+0000","last_became_peered":"2024-03-06T07:31:44.025510+0000","last_unstale":"2024-03-06T07:31:45.035787+0000","last_undegraded":"2024-03-06T07:31:45.035787+0000","last_fullsized":"2024-03-06T07:31:45.035787+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:13.314073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041996+0000","last_change":"2024-03-06T07:31:45.041996+0000","last_active":"2024-03-06T07:31:45.041996+0000","last_peered":"2024-03-06T07:31:45.041996+0000","last_clean":"2024-03-06T07:31:45.041996+0000","last_became_active":"2024-03-06T07:31:45.041403+0000","last_became_peered":"2024-03-06T07:31:45.041403+0000","last_unstale":"2024-03-06T07:31:45.041996+0000","last_undegraded":"2024-03-06T07:31:45.041996+0000","last_fullsized":"2024-03-06T07:31:45.041996+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:20.046408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.032932+0000","last_change":"2024-03-06T07:31:44.022857+0000","last_active":"2024-03-06T07:31:45.032932+0000","last_peered":"2024-03-06T07:31:45.032932+0000","last_clean":"2024-03-06T07:31:45.032932+0000","last_became_active":"2024-03-06T07:31:44.022400+0000","last_became_peered":"2024-03-06T07:31:44.022400+0000","last_unstale":"2024-03-06T07:31:45.032932+0000","last_undegraded":"2024-03-06T07:31:45.032932+0000","last_fullsized":"2024-03-06T07:31:45.032932+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:55.478147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044454+0000","last_change":"2024-03-06T07:31:45.042751+0000","last_active":"2024-03-06T07:31:45.044454+0000","last_peered":"2024-03-06T07:31:45.044454+0000","last_clean":"2024-03-06T07:31:45.044454+0000","last_became_active":"2024-03-06T07:31:45.042477+0000","last_became_peered":"2024-03-06T07:31:45.042477+0000","last_unstale":"2024-03-06T07:31:45.044454+0000","last_undegraded":"2024-03-06T07:31:45.044454+0000","last_fullsized":"2024-03-06T07:31:45.044454+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:43.063911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029362+0000","last_change":"2024-03-06T07:31:44.023105+0000","last_active":"2024-03-06T07:31:45.029362+0000","last_peered":"2024-03-06T07:31:45.029362+0000","last_clean":"2024-03-06T07:31:45.029362+0000","last_became_active":"2024-03-06T07:31:44.022808+0000","last_became_peered":"2024-03-06T07:31:44.022808+0000","last_unstale":"2024-03-06T07:31:45.029362+0000","last_undegraded":"2024-03-06T07:31:45.029362+0000","last_fullsized":"2024-03-06T07:31:45.029362+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.047987+0000","last_change":"2024-03-06T07:31:45.047987+0000","last_active":"2024-03-06T07:31:45.047987+0000","last_peered":"2024-03-06T07:31:45.047987+0000","last_clean":"2024-03-06T07:31:45.047987+0000","last_became_active":"2024-03-06T07:31:45.045672+0000","last_became_peered":"2024-03-06T07:31:45.045672+0000","last_unstale":"2024-03-06T07:31:45.047987+0000","last_undegraded":"2024-03-06T07:31:45.047987+0000","last_fullsized":"2024-03-06T07:31:45.047987+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:38.092312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025614+0000","last_change":"2024-03-06T07:31:44.024742+0000","last_active":"2024-03-06T07:31:45.025614+0000","last_peered":"2024-03-06T07:31:45.025614+0000","last_clean":"2024-03-06T07:31:45.025614+0000","last_became_active":"2024-03-06T07:31:44.024445+0000","last_became_peered":"2024-03-06T07:31:44.024445+0000","last_unstale":"2024-03-06T07:31:45.025614+0000","last_undegraded":"2024-03-06T07:31:45.025614+0000","last_fullsized":"2024-03-06T07:31:45.025614+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:00.838367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":6,"reported_epoch":144,"state":"active","last_fresh":"2024-03-06T07:31:45.051855+0000","last_change":"2024-03-06T07:31:45.051795+0000","last_active":"2024-03-06T07:31:45.051855+0000","last_peered":"2024-03-06T07:31:45.051855+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"2024-03-06T07:31:45.051795+0000","last_became_peered":"2024-03-06T07:31:45.051795+0000","last_unstale":"2024-03-06T07:31:45.051855+0000","last_undegraded":"2024-03-06T07:31:45.051855+0000","last_fullsized":"2024-03-06T07:31:45.051855+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"144'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029593+0000","last_change":"2024-03-06T07:31:44.023389+0000","last_active":"2024-03-06T07:31:45.029593+0000","last_peered":"2024-03-06T07:31:45.029593+0000","last_clean":"2024-03-06T07:31:45.029593+0000","last_became_active":"2024-03-06T07:31:44.023124+0000","last_became_peered":"2024-03-06T07:31:44.023124+0000","last_unstale":"2024-03-06T07:31:45.029593+0000","last_undegraded":"2024-03-06T07:31:45.029593+0000","last_fullsized":"2024-03-06T07:31:45.029593+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.044088+0000","last_change":"2024-03-06T07:31:45.044088+0000","last_active":"2024-03-06T07:31:45.044088+0000","last_peered":"2024-03-06T07:31:45.044088+0000","last_clean":"2024-03-06T07:31:45.044088+0000","last_became_active":"2024-03-06T07:31:45.043830+0000","last_became_peered":"2024-03-06T07:31:45.043830+0000","last_unstale":"2024-03-06T07:31:45.044088+0000","last_undegraded":"2024-03-06T07:31:45.044088+0000","last_fullsized":"2024-03-06T07:31:45.044088+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:40.063434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031115+0000","last_change":"2024-03-06T07:31:44.024253+0000","last_active":"2024-03-06T07:31:45.031115+0000","last_peered":"2024-03-06T07:31:45.031115+0000","last_clean":"2024-03-06T07:31:45.031115+0000","last_became_active":"2024-03-06T07:31:44.023842+0000","last_became_peered":"2024-03-06T07:31:44.023842+0000","last_unstale":"2024-03-06T07:31:45.031115+0000","last_undegraded":"2024-03-06T07:31:45.031115+0000","last_fullsized":"2024-03-06T07:31:45.031115+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045694+0000","last_change":"2024-03-06T07:31:45.045694+0000","last_active":"2024-03-06T07:31:45.045694+0000","last_peered":"2024-03-06T07:31:45.045694+0000","last_clean":"2024-03-06T07:31:45.045694+0000","last_became_active":"2024-03-06T07:31:45.044744+0000","last_became_peered":"2024-03-06T07:31:45.044744+0000","last_unstale":"2024-03-06T07:31:45.045694+0000","last_undegraded":"2024-03-06T07:31:45.045694+0000","last_fullsized":"2024-03-06T07:31:45.045694+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:42.160169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.025821+0000","last_change":"2024-03-06T07:31:44.035542+0000","last_active":"2024-03-06T07:31:45.025821+0000","last_peered":"2024-03-06T07:31:45.025821+0000","last_clean":"2024-03-06T07:31:45.025821+0000","last_became_active":"2024-03-06T07:31:44.035318+0000","last_became_peered":"2024-03-06T07:31:44.035318+0000","last_unstale":"2024-03-06T07:31:45.025821+0000","last_undegraded":"2024-03-06T07:31:45.025821+0000","last_fullsized":"2024-03-06T07:31:45.025821+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:03.773449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.029060+0000","last_change":"2024-03-06T07:31:45.029060+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.029060+0000","last_undegraded":"2024-03-06T07:31:45.029060+0000","last_fullsized":"2024-03-06T07:31:45.029060+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.030954+0000","last_change":"2024-03-06T07:31:44.028091+0000","last_active":"2024-03-06T07:31:45.030954+0000","last_peered":"2024-03-06T07:31:45.030954+0000","last_clean":"2024-03-06T07:31:45.030954+0000","last_became_active":"2024-03-06T07:31:44.027829+0000","last_became_peered":"2024-03-06T07:31:44.027829+0000","last_unstale":"2024-03-06T07:31:45.030954+0000","last_undegraded":"2024-03-06T07:31:45.030954+0000","last_fullsized":"2024-03-06T07:31:45.030954+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:08.912427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.043408+0000","last_change":"2024-03-06T07:31:45.039966+0000","last_active":"2024-03-06T07:31:45.043408+0000","last_peered":"2024-03-06T07:31:45.043408+0000","last_clean":"2024-03-06T07:31:45.043408+0000","last_became_active":"2024-03-06T07:31:45.039079+0000","last_became_peered":"2024-03-06T07:31:45.039079+0000","last_unstale":"2024-03-06T07:31:45.043408+0000","last_undegraded":"2024-03-06T07:31:45.043408+0000","last_fullsized":"2024-03-06T07:31:45.043408+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:56.482865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029577+0000","last_change":"2024-03-06T07:31:44.024320+0000","last_active":"2024-03-06T07:31:45.029577+0000","last_peered":"2024-03-06T07:31:45.029577+0000","last_clean":"2024-03-06T07:31:45.029577+0000","last_became_active":"2024-03-06T07:31:44.024051+0000","last_became_peered":"2024-03-06T07:31:44.024051+0000","last_unstale":"2024-03-06T07:31:45.029577+0000","last_undegraded":"2024-03-06T07:31:45.029577+0000","last_fullsized":"2024-03-06T07:31:45.029577+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:59.753052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040276+0000","last_change":"2024-03-06T07:31:45.035119+0000","last_active":"2024-03-06T07:31:45.040276+0000","last_peered":"2024-03-06T07:31:45.040276+0000","last_clean":"2024-03-06T07:31:45.040276+0000","last_became_active":"2024-03-06T07:31:45.034850+0000","last_became_peered":"2024-03-06T07:31:45.034850+0000","last_unstale":"2024-03-06T07:31:45.040276+0000","last_undegraded":"2024-03-06T07:31:45.040276+0000","last_fullsized":"2024-03-06T07:31:45.040276+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:34.247852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041190+0000","last_change":"2024-03-06T07:31:44.031164+0000","last_active":"2024-03-06T07:31:45.041190+0000","last_peered":"2024-03-06T07:31:45.041190+0000","last_clean":"2024-03-06T07:31:45.041190+0000","last_became_active":"2024-03-06T07:31:44.030649+0000","last_became_peered":"2024-03-06T07:31:44.030649+0000","last_unstale":"2024-03-06T07:31:45.041190+0000","last_undegraded":"2024-03-06T07:31:45.041190+0000","last_fullsized":"2024-03-06T07:31:45.041190+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.047269+0000","last_change":"2024-03-06T07:31:45.043562+0000","last_active":"2024-03-06T07:31:45.047269+0000","last_peered":"2024-03-06T07:31:45.047269+0000","last_clean":"2024-03-06T07:31:45.047269+0000","last_became_active":"2024-03-06T07:31:45.043293+0000","last_became_peered":"2024-03-06T07:31:45.043293+0000","last_unstale":"2024-03-06T07:31:45.047269+0000","last_undegraded":"2024-03-06T07:31:45.047269+0000","last_fullsized":"2024-03-06T07:31:45.047269+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:14.318792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.038382+0000","last_change":"2024-03-06T07:31:44.032582+0000","last_active":"2024-03-06T07:31:45.038382+0000","last_peered":"2024-03-06T07:31:45.038382+0000","last_clean":"2024-03-06T07:31:45.038382+0000","last_became_active":"2024-03-06T07:31:44.032075+0000","last_became_peered":"2024-03-06T07:31:44.032075+0000","last_unstale":"2024-03-06T07:31:45.038382+0000","last_undegraded":"2024-03-06T07:31:45.038382+0000","last_fullsized":"2024-03-06T07:31:45.038382+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040239+0000","last_change":"2024-03-06T07:31:45.035199+0000","last_active":"2024-03-06T07:31:45.040239+0000","last_peered":"2024-03-06T07:31:45.040239+0000","last_clean":"2024-03-06T07:31:45.040239+0000","last_became_active":"2024-03-06T07:31:45.034925+0000","last_became_peered":"2024-03-06T07:31:45.034925+0000","last_unstale":"2024-03-06T07:31:45.040239+0000","last_undegraded":"2024-03-06T07:31:45.040239+0000","last_fullsized":"2024-03-06T07:31:45.040239+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034897+0000","last_change":"2024-03-06T07:31:44.031018+0000","last_active":"2024-03-06T07:31:45.034897+0000","last_peered":"2024-03-06T07:31:45.034897+0000","last_clean":"2024-03-06T07:31:45.034897+0000","last_became_active":"2024-03-06T07:31:44.030695+0000","last_became_peered":"2024-03-06T07:31:44.030695+0000","last_unstale":"2024-03-06T07:31:45.034897+0000","last_undegraded":"2024-03-06T07:31:45.034897+0000","last_fullsized":"2024-03-06T07:31:45.034897+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:51.791822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.047228+0000","last_change":"2024-03-06T07:31:45.043749+0000","last_active":"2024-03-06T07:31:45.047228+0000","last_peered":"2024-03-06T07:31:45.047228+0000","last_clean":"2024-03-06T07:31:45.047228+0000","last_became_active":"2024-03-06T07:31:45.043501+0000","last_became_peered":"2024-03-06T07:31:45.043501+0000","last_unstale":"2024-03-06T07:31:45.047228+0000","last_undegraded":"2024-03-06T07:31:45.047228+0000","last_fullsized":"2024-03-06T07:31:45.047228+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.690561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.041947+0000","last_change":"2024-03-06T07:31:44.033647+0000","last_active":"2024-03-06T07:31:45.041947+0000","last_peered":"2024-03-06T07:31:45.041947+0000","last_clean":"2024-03-06T07:31:45.041947+0000","last_became_active":"2024-03-06T07:31:44.033393+0000","last_became_peered":"2024-03-06T07:31:44.033393+0000","last_unstale":"2024-03-06T07:31:45.041947+0000","last_undegraded":"2024-03-06T07:31:45.041947+0000","last_fullsized":"2024-03-06T07:31:45.041947+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:35.425689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.045750+0000","last_change":"2024-03-06T07:31:45.045750+0000","last_active":"2024-03-06T07:31:45.045750+0000","last_peered":"2024-03-06T07:31:45.045750+0000","last_clean":"2024-03-06T07:31:45.045750+0000","last_became_active":"2024-03-06T07:31:45.044153+0000","last_became_peered":"2024-03-06T07:31:45.044153+0000","last_unstale":"2024-03-06T07:31:45.045750+0000","last_undegraded":"2024-03-06T07:31:45.045750+0000","last_fullsized":"2024-03-06T07:31:45.045750+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:26.528565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.030989+0000","last_change":"2024-03-06T07:31:44.028776+0000","last_active":"2024-03-06T07:31:45.030989+0000","last_peered":"2024-03-06T07:31:45.030989+0000","last_clean":"2024-03-06T07:31:45.030989+0000","last_became_active":"2024-03-06T07:31:44.028504+0000","last_became_peered":"2024-03-06T07:31:44.028504+0000","last_unstale":"2024-03-06T07:31:45.030989+0000","last_undegraded":"2024-03-06T07:31:45.030989+0000","last_fullsized":"2024-03-06T07:31:45.030989+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:17.358709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040051+0000","last_change":"2024-03-06T07:31:45.040051+0000","last_active":"2024-03-06T07:31:45.040051+0000","last_peered":"2024-03-06T07:31:45.040051+0000","last_clean":"2024-03-06T07:31:45.040051+0000","last_became_active":"2024-03-06T07:31:45.039760+0000","last_became_peered":"2024-03-06T07:31:45.039760+0000","last_unstale":"2024-03-06T07:31:45.040051+0000","last_undegraded":"2024-03-06T07:31:45.040051+0000","last_fullsized":"2024-03-06T07:31:45.040051+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:36.430408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029750+0000","last_change":"2024-03-06T07:31:44.027749+0000","last_active":"2024-03-06T07:31:45.029750+0000","last_peered":"2024-03-06T07:31:45.029750+0000","last_clean":"2024-03-06T07:31:45.029750+0000","last_became_active":"2024-03-06T07:31:44.027413+0000","last_became_peered":"2024-03-06T07:31:44.027413+0000","last_unstale":"2024-03-06T07:31:45.029750+0000","last_undegraded":"2024-03-06T07:31:45.029750+0000","last_fullsized":"2024-03-06T07:31:45.029750+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:26.520317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037784+0000","last_change":"2024-03-06T07:31:45.036246+0000","last_active":"2024-03-06T07:31:45.037784+0000","last_peered":"2024-03-06T07:31:45.037784+0000","last_clean":"2024-03-06T07:31:45.037784+0000","last_became_active":"2024-03-06T07:31:45.035684+0000","last_became_peered":"2024-03-06T07:31:45.035684+0000","last_unstale":"2024-03-06T07:31:45.037784+0000","last_undegraded":"2024-03-06T07:31:45.037784+0000","last_fullsized":"2024-03-06T07:31:45.037784+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:19.547257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.030746+0000","last_change":"2024-03-06T07:31:44.027642+0000","last_active":"2024-03-06T07:31:45.030746+0000","last_peered":"2024-03-06T07:31:45.030746+0000","last_clean":"2024-03-06T07:31:45.030746+0000","last_became_active":"2024-03-06T07:31:44.027355+0000","last_became_peered":"2024-03-06T07:31:44.027355+0000","last_unstale":"2024-03-06T07:31:45.030746+0000","last_undegraded":"2024-03-06T07:31:45.030746+0000","last_fullsized":"2024-03-06T07:31:45.030746+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.058715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037724+0000","last_change":"2024-03-06T07:31:45.034848+0000","last_active":"2024-03-06T07:31:45.037724+0000","last_peered":"2024-03-06T07:31:45.037724+0000","last_clean":"2024-03-06T07:31:45.037724+0000","last_became_active":"2024-03-06T07:31:45.034565+0000","last_became_peered":"2024-03-06T07:31:45.034565+0000","last_unstale":"2024-03-06T07:31:45.037724+0000","last_undegraded":"2024-03-06T07:31:45.037724+0000","last_fullsized":"2024-03-06T07:31:45.037724+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:39.939282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.029755+0000","last_change":"2024-03-06T07:31:44.025967+0000","last_active":"2024-03-06T07:31:45.029755+0000","last_peered":"2024-03-06T07:31:45.029755+0000","last_clean":"2024-03-06T07:31:45.029755+0000","last_became_active":"2024-03-06T07:31:44.025640+0000","last_became_peered":"2024-03-06T07:31:44.025640+0000","last_unstale":"2024-03-06T07:31:45.029755+0000","last_undegraded":"2024-03-06T07:31:45.029755+0000","last_fullsized":"2024-03-06T07:31:45.029755+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:48.746990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.048574+0000","last_change":"2024-03-06T07:31:45.045298+0000","last_active":"2024-03-06T07:31:45.048574+0000","last_peered":"2024-03-06T07:31:45.048574+0000","last_clean":"2024-03-06T07:31:45.048574+0000","last_became_active":"2024-03-06T07:31:45.045033+0000","last_became_peered":"2024-03-06T07:31:45.045033+0000","last_unstale":"2024-03-06T07:31:45.048574+0000","last_undegraded":"2024-03-06T07:31:45.048574+0000","last_fullsized":"2024-03-06T07:31:45.048574+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:19.547257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040670+0000","last_change":"2024-03-06T07:31:44.032074+0000","last_active":"2024-03-06T07:31:45.040670+0000","last_peered":"2024-03-06T07:31:45.040670+0000","last_clean":"2024-03-06T07:31:45.040670+0000","last_became_active":"2024-03-06T07:31:44.031715+0000","last_became_peered":"2024-03-06T07:31:44.031715+0000","last_unstale":"2024-03-06T07:31:45.040670+0000","last_undegraded":"2024-03-06T07:31:45.040670+0000","last_fullsized":"2024-03-06T07:31:45.040670+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":3,"reported_epoch":144,"state":"creating+activating","last_fresh":"2024-03-06T07:31:45.040107+0000","last_change":"2024-03-06T07:31:45.040107+0000","last_active":"2024-03-06T07:31:44.007138+0000","last_peered":"2024-03-06T07:31:44.007138+0000","last_clean":"2024-03-06T07:31:44.007138+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:45.040107+0000","last_undegraded":"2024-03-06T07:31:45.040107+0000","last_fullsized":"2024-03-06T07:31:45.040107+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.034395+0000","last_change":"2024-03-06T07:31:44.025352+0000","last_active":"2024-03-06T07:31:45.034395+0000","last_peered":"2024-03-06T07:31:45.034395+0000","last_clean":"2024-03-06T07:31:45.034395+0000","last_became_active":"2024-03-06T07:31:44.025043+0000","last_became_peered":"2024-03-06T07:31:44.025043+0000","last_unstale":"2024-03-06T07:31:45.034395+0000","last_undegraded":"2024-03-06T07:31:45.034395+0000","last_fullsized":"2024-03-06T07:31:45.034395+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:59.681311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.037968+0000","last_change":"2024-03-06T07:31:45.035827+0000","last_active":"2024-03-06T07:31:45.037968+0000","last_peered":"2024-03-06T07:31:45.037968+0000","last_clean":"2024-03-06T07:31:45.037968+0000","last_became_active":"2024-03-06T07:31:45.035484+0000","last_became_peered":"2024-03-06T07:31:45.035484+0000","last_unstale":"2024-03-06T07:31:45.037968+0000","last_undegraded":"2024-03-06T07:31:45.037968+0000","last_fullsized":"2024-03-06T07:31:45.037968+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:27.525036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.031547+0000","last_change":"2024-03-06T07:31:44.027942+0000","last_active":"2024-03-06T07:31:45.031547+0000","last_peered":"2024-03-06T07:31:45.031547+0000","last_clean":"2024-03-06T07:31:45.031547+0000","last_became_active":"2024-03-06T07:31:44.027666+0000","last_became_peered":"2024-03-06T07:31:44.027666+0000","last_unstale":"2024-03-06T07:31:45.031547+0000","last_undegraded":"2024-03-06T07:31:45.031547+0000","last_fullsized":"2024-03-06T07:31:45.031547+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040316+0000","last_change":"2024-03-06T07:31:45.035716+0000","last_active":"2024-03-06T07:31:45.040316+0000","last_peered":"2024-03-06T07:31:45.040316+0000","last_clean":"2024-03-06T07:31:45.040316+0000","last_became_active":"2024-03-06T07:31:45.035437+0000","last_became_peered":"2024-03-06T07:31:45.035437+0000","last_unstale":"2024-03-06T07:31:45.040316+0000","last_undegraded":"2024-03-06T07:31:45.040316+0000","last_fullsized":"2024-03-06T07:31:45.040316+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:20.551707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.038657+0000","last_change":"2024-03-06T07:31:44.031865+0000","last_active":"2024-03-06T07:31:45.038657+0000","last_peered":"2024-03-06T07:31:45.038657+0000","last_clean":"2024-03-06T07:31:45.038657+0000","last_became_active":"2024-03-06T07:31:44.030846+0000","last_became_peered":"2024-03-06T07:31:44.030846+0000","last_unstale":"2024-03-06T07:31:45.038657+0000","last_undegraded":"2024-03-06T07:31:45.038657+0000","last_fullsized":"2024-03-06T07:31:45.038657+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.034911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":407,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:31:45.040477+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:31:45.040477+0000","last_peered":"2024-03-06T07:31:45.040477+0000","last_clean":"2024-03-06T07:31:45.040477+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:31:45.040477+0000","last_undegraded":"2024-03-06T07:31:45.040477+0000","last_fullsized":"2024-03-06T07:31:45.040477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574961,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41832,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330008,"statfs":{"total":96636764160,"available":96593928192,"internally_reserved":0,"allocated":2383872,"data_stored":1371971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574962,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574961,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41252,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330588,"statfs":{"total":96636764160,"available":96594522112,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41248,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330592,"statfs":{"total":96636764160,"available":96594526208,"internally_reserved":0,"allocated":1789952,"data_stored":781603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574962,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41836,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330004,"statfs":{"total":96636764160,"available":96593924096,"internally_reserved":0,"allocated":2383872,"data_stored":1371971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.3a is not active+clean 2024-03-06T07:31:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.3a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 144, 'state': 'creating+activating', 'last_fresh': '2024-03-06T07:31:45.029936+0000', 'last_change': '2024-03-06T07:31:45.029936+0000', 'last_active': '2024-03-06T07:31:44.007138+0000', 'last_peered': '2024-03-06T07:31:44.007138+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:31:45.029936+0000', 'last_undegraded': '2024-03-06T07:31:45.029936+0000', 'last_fullsized': '2024-03-06T07:31:45.029936+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': ['1', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:31:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.2f is not active+clean 2024-03-06T07:31:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.2f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 144, 'state': 'creating+activating', 'last_fresh': '2024-03-06T07:31:45.042465+0000', 'last_change': '2024-03-06T07:31:45.042465+0000', 'last_active': '2024-03-06T07:31:44.007138+0000', 'last_peered': '2024-03-06T07:31:44.007138+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:31:45.042465+0000', 'last_undegraded': '2024-03-06T07:31:45.042465+0000', 'last_fullsized': '2024-03-06T07:31:45.042465+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': ['7', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:31:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.28 is not active+clean 2024-03-06T07:31:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.28', 'version': "0'0", 'reported_seq': 6, 'reported_epoch': 144, 'state': 'active', 'last_fresh': '2024-03-06T07:31:45.047191+0000', 'last_change': '2024-03-06T07:31:45.047127+0000', 'last_active': '2024-03-06T07:31:45.047191+0000', 'last_peered': '2024-03-06T07:31:45.047191+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '2024-03-06T07:31:45.047127+0000', 'last_became_peered': '2024-03-06T07:31:45.047127+0000', 'last_unstale': '2024-03-06T07:31:45.047191+0000', 'last_undegraded': '2024-03-06T07:31:45.047191+0000', 'last_fullsized': '2024-03-06T07:31:45.047191+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': ['7', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:31:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.25 is not active+clean 2024-03-06T07:31:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.25', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 144, 'state': 'creating+activating', 'last_fresh': '2024-03-06T07:31:45.025450+0000', 'last_change': '2024-03-06T07:31:45.025450+0000', 'last_active': '2024-03-06T07:31:44.007138+0000', 'last_peered': '2024-03-06T07:31:44.007138+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:31:45.025450+0000', 'last_undegraded': '2024-03-06T07:31:45.025450+0000', 'last_fullsized': '2024-03-06T07:31:45.025450+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': ['2', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.24 is not active+clean 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.24', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 144, 'state': 'creating+activating', 'last_fresh': '2024-03-06T07:31:45.028666+0000', 'last_change': '2024-03-06T07:31:45.028666+0000', 'last_active': '2024-03-06T07:31:44.007138+0000', 'last_peered': '2024-03-06T07:31:44.007138+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:31:45.028666+0000', 'last_undegraded': '2024-03-06T07:31:45.028666+0000', 'last_fullsized': '2024-03-06T07:31:45.028666+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': ['0', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.21 is not active+clean 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.21', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 144, 'state': 'creating+activating', 'last_fresh': '2024-03-06T07:31:45.029698+0000', 'last_change': '2024-03-06T07:31:45.029698+0000', 'last_active': '2024-03-06T07:31:44.007138+0000', 'last_peered': '2024-03-06T07:31:44.007138+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:31:45.029698+0000', 'last_undegraded': '2024-03-06T07:31:45.029698+0000', 'last_fullsized': '2024-03-06T07:31:45.029698+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': ['3', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.1a is not active+clean 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.1a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 144, 'state': 'creating+activating', 'last_fresh': '2024-03-06T07:31:45.030083+0000', 'last_change': '2024-03-06T07:31:45.030083+0000', 'last_active': '2024-03-06T07:31:44.007138+0000', 'last_peered': '2024-03-06T07:31:44.007138+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:31:45.030083+0000', 'last_undegraded': '2024-03-06T07:31:45.030083+0000', 'last_fullsized': '2024-03-06T07:31:45.030083+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': ['3', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.17 is not active+clean 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.17', 'version': "0'0", 'reported_seq': 6, 'reported_epoch': 144, 'state': 'active', 'last_fresh': '2024-03-06T07:31:45.052295+0000', 'last_change': '2024-03-06T07:31:45.052223+0000', 'last_active': '2024-03-06T07:31:45.052295+0000', 'last_peered': '2024-03-06T07:31:45.052295+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '2024-03-06T07:31:45.052223+0000', 'last_became_peered': '2024-03-06T07:31:45.052223+0000', 'last_unstale': '2024-03-06T07:31:45.052295+0000', 'last_undegraded': '2024-03-06T07:31:45.052295+0000', 'last_fullsized': '2024-03-06T07:31:45.052295+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': ['7', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.e is not active+clean 2024-03-06T07:31:49.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.e', 'version': "0'0", 'reported_seq': 6, 'reported_epoch': 144, 'state': 'active', 'last_fresh': '2024-03-06T07:31:45.051855+0000', 'last_change': '2024-03-06T07:31:45.051795+0000', 'last_active': '2024-03-06T07:31:45.051855+0000', 'last_peered': '2024-03-06T07:31:45.051855+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '2024-03-06T07:31:45.051795+0000', 'last_became_peered': '2024-03-06T07:31:45.051795+0000', 'last_unstale': '2024-03-06T07:31:45.051855+0000', 'last_undegraded': '2024-03-06T07:31:45.051855+0000', 'last_fullsized': '2024-03-06T07:31:45.051855+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': ['7', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:31:49.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.d is not active+clean 2024-03-06T07:31:49.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.d', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 144, 'state': 'creating+activating', 'last_fresh': '2024-03-06T07:31:45.029060+0000', 'last_change': '2024-03-06T07:31:45.029060+0000', 'last_active': '2024-03-06T07:31:44.007138+0000', 'last_peered': '2024-03-06T07:31:44.007138+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:31:45.029060+0000', 'last_undegraded': '2024-03-06T07:31:45.029060+0000', 'last_fullsized': '2024-03-06T07:31:45.029060+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': ['1', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:31:49.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 21.0 is not active+clean 2024-03-06T07:31:49.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '21.0', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 144, 'state': 'creating+activating', 'last_fresh': '2024-03-06T07:31:45.040107+0000', 'last_change': '2024-03-06T07:31:45.040107+0000', 'last_active': '2024-03-06T07:31:44.007138+0000', 'last_peered': '2024-03-06T07:31:44.007138+0000', 'last_clean': '2024-03-06T07:31:44.007138+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:31:45.040107+0000', 'last_undegraded': '2024-03-06T07:31:45.040107+0000', 'last_fullsized': '2024-03-06T07:31:45.040107+0000', 'mapping_epoch': 143, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 143, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'last_clean_scrub_stamp': '2024-03-06T07:31:44.007138+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': ['7', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:31:49.879 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:31:50.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.026+0000 7f20610eb640 1 Processor -- start 2024-03-06T07:31:50.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.026+0000 7f20610eb640 1 -- start start 2024-03-06T07:31:50.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.026+0000 7f20610eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f205c0b95a0 0x7f205c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.026+0000 7f20610eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b9a10 0x7f205c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.026+0000 7f20610eb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205c0cea30 con 0x7f205c15df00 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.026+0000 7f20610eb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205c0cf3c0 con 0x7f205c0b95a0 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.026+0000 7f20610eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205c0cf0e0 con 0x7f205c0b9a10 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.027+0000 7f205a59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f205c0b95a0 0x7f205c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.027+0000 7f205a59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f205c0b95a0 0x7f205c0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36636/0 (socket says 172.21.15.27:36636) 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.027+0000 7f205a59c640 1 -- 172.21.15.27:0/2174406910 learned_addr learned my addr 172.21.15.27:0/2174406910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.027+0000 7f205b59e640 1 --2- 172.21.15.27:0/2174406910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b9a10 0x7f205c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.027+0000 7f205a59c640 1 -- 172.21.15.27:0/2174406910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b9a10 msgr2=0x7f205c0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.027+0000 7f205a59c640 1 --2- 172.21.15.27:0/2174406910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b9a10 0x7f205c0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.027+0000 7f205a59c640 1 -- 172.21.15.27:0/2174406910 >> v1:172.21.15.69:6789/0 conn(0x7f205c15df00 legacy=0x7f205c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.027+0000 7f205a59c640 1 -- 172.21.15.27:0/2174406910 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c0ce470 con 0x7f205c0b95a0 2024-03-06T07:31:50.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f205a59c640 1 --2- 172.21.15.27:0/2174406910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b95a0 0x7f205c0b5700 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f2050009870 tx=0x7f205002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8578e831f3c708e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f2059d9b640 1 -- 172.21.15.27:0/2174406910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050034070 con 0x7f205c0b95a0 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f2059d9b640 1 -- 172.21.15.27:0/2174406910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2050030840 con 0x7f205c0b95a0 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f2059d9b640 1 -- 172.21.15.27:0/2174406910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050030c10 con 0x7f205c0b95a0 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f20610eb640 1 -- 172.21.15.27:0/2174406910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b95a0 msgr2=0x7f205c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f20610eb640 1 --2- 172.21.15.27:0/2174406910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b95a0 0x7f205c0b5700 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f2050009870 tx=0x7f205002b980 comp rx=0 tx=0).stop 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f20610eb640 1 -- 172.21.15.27:0/2174406910 shutdown_connections 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f20610eb640 1 --2- 172.21.15.27:0/2174406910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b95a0 0x7f205c0b5700 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f20610eb640 1 --2- 172.21.15.27:0/2174406910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b9a10 0x7f205c0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f20610eb640 1 -- 172.21.15.27:0/2174406910 >> 172.21.15.27:0/2174406910 conn(0x7f205c071960 msgr2=0x7f205c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f20610eb640 1 -- 172.21.15.27:0/2174406910 shutdown_connections 2024-03-06T07:31:50.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.028+0000 7f20610eb640 1 -- 172.21.15.27:0/2174406910 wait complete. 2024-03-06T07:31:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.030+0000 7f20610eb640 1 Processor -- start 2024-03-06T07:31:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.030+0000 7f20610eb640 1 -- start start 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f20610eb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b95a0 0x7f205c1509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f20610eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b9a10 0x7f205c14d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f20610eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c15df00 0x7f205c0cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f20610eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c0cedf0 con 0x7f205c0b9a10 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f20610eb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c0cec70 con 0x7f205c15df00 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f20610eb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205c0d3220 con 0x7f205c0b95a0 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b9a10 0x7f205c14d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b9a10 0x7f205c14d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36650/0 (socket says 172.21.15.27:36650) 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205a59c640 1 -- 172.21.15.27:0/649497787 learned_addr learned my addr 172.21.15.27:0/649497787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205ad9d640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b95a0 0x7f205c1509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205b59e640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c15df00 0x7f205c0cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205a59c640 1 -- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b95a0 msgr2=0x7f205c1509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205a59c640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b95a0 0x7f205c1509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205a59c640 1 -- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c15df00 msgr2=0x7f205c0cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205a59c640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c15df00 0x7f205c0cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205a59c640 1 -- 172.21.15.27:0/649497787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c0c9b80 con 0x7f205c0b9a10 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f205a59c640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b9a10 0x7f205c14d2b0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f2050000c00 tx=0x7f2050004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f20437fe640 1 -- 172.21.15.27:0/649497787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050048070 con 0x7f205c0b9a10 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.031+0000 7f20437fe640 1 -- 172.21.15.27:0/649497787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2050030640 con 0x7f205c0b9a10 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.032+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205c0c9e60 con 0x7f205c0b9a10 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.032+0000 7f20437fe640 1 -- 172.21.15.27:0/649497787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050034040 con 0x7f205c0b9a10 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.032+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205c0ca420 con 0x7f205c0b9a10 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.033+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205c067ff0 con 0x7f205c0b9a10 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.033+0000 7f20437fe640 1 -- 172.21.15.27:0/649497787 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f205004b070 con 0x7f205c0b9a10 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.034+0000 7f20437fe640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f204806b500 0x7f204806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.034+0000 7f20437fe640 1 -- 172.21.15.27:0/649497787 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20500abf50 con 0x7f205c0b9a10 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.034+0000 7f205ad9d640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f204806b500 0x7f204806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.035+0000 7f205ad9d640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f204806b500 0x7f204806d9c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f204c0016e0 tx=0x7f204c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.038+0000 7f20437fe640 1 -- 172.21.15.27:0/649497787 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f205007a810 con 0x7f205c0b9a10 2024-03-06T07:31:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.297+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f205c15cc60 con 0x7f205c0b9a10 2024-03-06T07:31:50.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.297+0000 7f20437fe640 1 -- 172.21.15.27:0/649497787 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1300 (secure 0 0 0) 0x7f205007e8b0 con 0x7f205c0b9a10 2024-03-06T07:31:50.299 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:50.299 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":566,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":145,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709754,"num_in_osds":8,"osd_in_since":1709709745,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":86},{"state_name":"creating+activating","count":8},{"state_name":"active","count":3}],"num_pgs":97,"num_pools":3,"num_objects":7,"data_bytes":591062,"bytes_used":339107840,"bytes_avail":772755005440,"bytes_total":773094113280,"inactive_pgs_ratio":0.082474224269390106,"write_bytes_sec":601,"read_op_per_sec":0,"write_op_per_sec":1},"fsmap":{"epoch":112,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"a","status":"up:active","gid":9282}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-03-06T07:31:28.114576+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:31:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f204806b500 msgr2=0x7f204806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f204806b500 0x7f204806d9c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f204c0016e0 tx=0x7f204c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:50.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b9a10 msgr2=0x7f205c14d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b9a10 0x7f205c14d2b0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f2050000c00 tx=0x7f2050004730 comp rx=0 tx=0).stop 2024-03-06T07:31:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 shutdown_connections 2024-03-06T07:31:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f204806b500 0x7f204806d9c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c15df00 0x7f205c0cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f205c0b9a10 0x7f205c14d2b0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 --2- 172.21.15.27:0/649497787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f205c0b95a0 0x7f205c1509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.299+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 >> 172.21.15.27:0/649497787 conn(0x7f205c071960 msgr2=0x7f205c0d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.300+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 shutdown_connections 2024-03-06T07:31:50.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.300+0000 7f20610eb640 1 -- 172.21.15.27:0/649497787 wait complete. 2024-03-06T07:31:50.312 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:31:50.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:50.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.417+0000 7fb038823640 1 Processor -- start 2024-03-06T07:31:50.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.418+0000 7fb038823640 1 -- start start 2024-03-06T07:31:50.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.418+0000 7fb038823640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb03415df00 0x7fb0340d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.418+0000 7fb038823640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0340b95a0 0x7fb0340b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.418+0000 7fb038823640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0340cea30 con 0x7fb0340b9a30 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.418+0000 7fb038823640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0340d2af0 con 0x7fb03415df00 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.419+0000 7fb038823640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0340cf3c0 con 0x7fb0340b95a0 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.419+0000 7fb03259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb03415df00 0x7fb0340d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.419+0000 7fb03259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb03415df00 0x7fb0340d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36656/0 (socket says 172.21.15.27:36656) 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.419+0000 7fb03259c640 1 -- 172.21.15.27:0/2104795941 learned_addr learned my addr 172.21.15.27:0/2104795941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.419+0000 7fb031d9b640 1 --2- 172.21.15.27:0/2104795941 >> v2:172.21.15.69:3300/0 conn(0x7fb0340b95a0 0x7fb0340b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.419+0000 7fb03259c640 1 -- 172.21.15.27:0/2104795941 >> v2:172.21.15.69:3300/0 conn(0x7fb0340b95a0 msgr2=0x7fb0340b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.419+0000 7fb03259c640 1 --2- 172.21.15.27:0/2104795941 >> v2:172.21.15.69:3300/0 conn(0x7fb0340b95a0 0x7fb0340b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.419+0000 7fb03259c640 1 -- 172.21.15.27:0/2104795941 >> v1:172.21.15.69:6789/0 conn(0x7fb0340b9a30 legacy=0x7fb0340d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.419+0000 7fb03259c640 1 -- 172.21.15.27:0/2104795941 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0340cf0e0 con 0x7fb03415df00 2024-03-06T07:31:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb03259c640 1 --2- 172.21.15.27:0/2104795941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb03415df00 0x7fb0340d25b0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fb0340d1760 tx=0x7fb02402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6716e7fa857e746 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb03159a640 1 -- 172.21.15.27:0/2104795941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024034070 con 0x7fb03415df00 2024-03-06T07:31:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb03159a640 1 -- 172.21.15.27:0/2104795941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb024030860 con 0x7fb03415df00 2024-03-06T07:31:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb03159a640 1 -- 172.21.15.27:0/2104795941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024030c30 con 0x7fb03415df00 2024-03-06T07:31:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb038823640 1 -- 172.21.15.27:0/2104795941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb03415df00 msgr2=0x7fb0340d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb038823640 1 --2- 172.21.15.27:0/2104795941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb03415df00 0x7fb0340d25b0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fb0340d1760 tx=0x7fb02402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb038823640 1 -- 172.21.15.27:0/2104795941 shutdown_connections 2024-03-06T07:31:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb038823640 1 --2- 172.21.15.27:0/2104795941 >> v2:172.21.15.69:3300/0 conn(0x7fb0340b95a0 0x7fb0340b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb038823640 1 --2- 172.21.15.27:0/2104795941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb03415df00 0x7fb0340d25b0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.420+0000 7fb038823640 1 -- 172.21.15.27:0/2104795941 >> 172.21.15.27:0/2104795941 conn(0x7fb034071960 msgr2=0x7fb034071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.421+0000 7fb038823640 1 -- 172.21.15.27:0/2104795941 shutdown_connections 2024-03-06T07:31:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.421+0000 7fb038823640 1 -- 172.21.15.27:0/2104795941 wait complete. 2024-03-06T07:31:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.422+0000 7fb038823640 1 Processor -- start 2024-03-06T07:31:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.422+0000 7fb038823640 1 -- start start 2024-03-06T07:31:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb038823640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0340b95a0 0x7fb0340cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb038823640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb03415df00 0x7fb0340b5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb038823640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0340cd5f0 0x7fb0341549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb038823640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0340cec70 con 0x7fb0340b95a0 2024-03-06T07:31:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb038823640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0340ce750 con 0x7fb03415df00 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb038823640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0340cedf0 con 0x7fb0340cd5f0 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb03259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0340b95a0 0x7fb0340cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb03259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0340b95a0 0x7fb0340cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36666/0 (socket says 172.21.15.27:36666) 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb03259c640 1 -- 172.21.15.27:0/1284302443 learned_addr learned my addr 172.21.15.27:0/1284302443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb031d9b640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb03415df00 0x7fb0340b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb032d9d640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0340cd5f0 0x7fb0341549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb03259c640 1 -- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0340cd5f0 msgr2=0x7fb0341549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb03259c640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0340cd5f0 0x7fb0341549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb03259c640 1 -- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb03415df00 msgr2=0x7fb0340b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb03259c640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb03415df00 0x7fb0340b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb03259c640 1 -- 172.21.15.27:0/1284302443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034155890 con 0x7fb0340b95a0 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb03259c640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0340b95a0 0x7fb0340cca10 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fb02402b9e0 tx=0x7fb0240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb022ffd640 1 -- 172.21.15.27:0/1284302443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024034040 con 0x7fb0340b95a0 2024-03-06T07:31:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb022ffd640 1 -- 172.21.15.27:0/1284302443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb024039d90 con 0x7fb0340b95a0 2024-03-06T07:31:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.423+0000 7fb022ffd640 1 -- 172.21.15.27:0/1284302443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024043610 con 0x7fb0340b95a0 2024-03-06T07:31:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.424+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0340cd0a0 con 0x7fb0340b95a0 2024-03-06T07:31:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.424+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0341f7920 con 0x7fb0340b95a0 2024-03-06T07:31:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.425+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb034067ff0 con 0x7fb0340b95a0 2024-03-06T07:31:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.425+0000 7fb022ffd640 1 -- 172.21.15.27:0/1284302443 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb02404a060 con 0x7fb0340b95a0 2024-03-06T07:31:50.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.425+0000 7fb022ffd640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.426+0000 7fb022ffd640 1 -- 172.21.15.27:0/1284302443 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0240ac2a0 con 0x7fb0340b95a0 2024-03-06T07:31:50.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.426+0000 7fb031d9b640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.426+0000 7fb031d9b640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb0341f75a0 tx=0x7fb028008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.430+0000 7fb022ffd640 1 -- 172.21.15.27:0/1284302443 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb02403e070 con 0x7fb0340b95a0 2024-03-06T07:31:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.610+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0340b3050 con 0x7fb01406b4d0 2024-03-06T07:31:50.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.616+0000 7fb022ffd640 1 -- 172.21.15.27:0/1284302443 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252476 (secure 0 0 0) 0x7fb0340b3050 con 0x7fb01406b4d0 2024-03-06T07:31:50.620 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:50.622 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:31:50.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 msgr2=0x7fb01406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb0341f75a0 tx=0x7fb028008040 comp rx=0 tx=0).stop 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0340b95a0 msgr2=0x7fb0340cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0340b95a0 0x7fb0340cca10 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fb02402b9e0 tx=0x7fb0240046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 shutdown_connections 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb01406b4d0 0x7fb01406d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0340cd5f0 0x7fb0341549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb03415df00 0x7fb0340b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 --2- 172.21.15.27:0/1284302443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0340b95a0 0x7fb0340cca10 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 >> 172.21.15.27:0/1284302443 conn(0x7fb034071960 msgr2=0x7fb0340b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 shutdown_connections 2024-03-06T07:31:50.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:50.620+0000 7fb038823640 1 -- 172.21.15.27:0/1284302443 wait complete. 2024-03-06T07:31:50.632 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-03-06T07:31:50.119330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331436,"kb_used_data":15400,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643284,"statfs":{"total":773094113280,"available":772754722816,"internally_reserved":0,"allocated":15769600,"data_stored":7542980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.108346"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044905+0000","last_change":"2024-03-06T07:31:45.046476+0000","last_active":"2024-03-06T07:31:46.044905+0000","last_peered":"2024-03-06T07:31:46.044905+0000","last_clean":"2024-03-06T07:31:46.044905+0000","last_became_active":"2024-03-06T07:31:45.046179+0000","last_became_peered":"2024-03-06T07:31:45.046179+0000","last_unstale":"2024-03-06T07:31:46.044905+0000","last_undegraded":"2024-03-06T07:31:46.044905+0000","last_fullsized":"2024-03-06T07:31:46.044905+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:52.796540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028753+0000","last_change":"2024-03-06T07:31:45.041710+0000","last_active":"2024-03-06T07:31:46.028753+0000","last_peered":"2024-03-06T07:31:46.028753+0000","last_clean":"2024-03-06T07:31:46.028753+0000","last_became_active":"2024-03-06T07:31:45.041413+0000","last_became_peered":"2024-03-06T07:31:45.041413+0000","last_unstale":"2024-03-06T07:31:46.028753+0000","last_undegraded":"2024-03-06T07:31:46.028753+0000","last_fullsized":"2024-03-06T07:31:46.028753+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:20.551707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.197070+0000","last_change":"2024-03-06T07:31:45.048666+0000","last_active":"2024-03-06T07:31:47.197070+0000","last_peered":"2024-03-06T07:31:47.197070+0000","last_clean":"2024-03-06T07:31:47.197070+0000","last_became_active":"2024-03-06T07:31:45.044404+0000","last_became_peered":"2024-03-06T07:31:45.044404+0000","last_unstale":"2024-03-06T07:31:47.197070+0000","last_undegraded":"2024-03-06T07:31:47.197070+0000","last_fullsized":"2024-03-06T07:31:47.197070+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:31.223766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995846+0000","last_change":"2024-03-06T07:31:45.046599+0000","last_active":"2024-03-06T07:31:46.995846+0000","last_peered":"2024-03-06T07:31:46.995846+0000","last_clean":"2024-03-06T07:31:46.995846+0000","last_became_active":"2024-03-06T07:31:45.046251+0000","last_became_peered":"2024-03-06T07:31:45.046251+0000","last_unstale":"2024-03-06T07:31:46.995846+0000","last_undegraded":"2024-03-06T07:31:46.995846+0000","last_fullsized":"2024-03-06T07:31:46.995846+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:34.148588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028806+0000","last_change":"2024-03-06T07:31:45.048513+0000","last_active":"2024-03-06T07:31:46.028806+0000","last_peered":"2024-03-06T07:31:46.028806+0000","last_clean":"2024-03-06T07:31:46.028806+0000","last_became_active":"2024-03-06T07:31:45.048251+0000","last_became_peered":"2024-03-06T07:31:45.048251+0000","last_unstale":"2024-03-06T07:31:46.028806+0000","last_undegraded":"2024-03-06T07:31:46.028806+0000","last_fullsized":"2024-03-06T07:31:46.028806+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:31.902903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044418+0000","last_change":"2024-03-06T07:31:45.049835+0000","last_active":"2024-03-06T07:31:46.044418+0000","last_peered":"2024-03-06T07:31:46.044418+0000","last_clean":"2024-03-06T07:31:46.044418+0000","last_became_active":"2024-03-06T07:31:45.049479+0000","last_became_peered":"2024-03-06T07:31:45.049479+0000","last_unstale":"2024-03-06T07:31:46.044418+0000","last_undegraded":"2024-03-06T07:31:46.044418+0000","last_fullsized":"2024-03-06T07:31:46.044418+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:52.578105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029386+0000","last_change":"2024-03-06T07:31:45.035876+0000","last_active":"2024-03-06T07:31:46.029386+0000","last_peered":"2024-03-06T07:31:46.029386+0000","last_clean":"2024-03-06T07:31:46.029386+0000","last_became_active":"2024-03-06T07:31:45.035604+0000","last_became_peered":"2024-03-06T07:31:45.035604+0000","last_unstale":"2024-03-06T07:31:46.029386+0000","last_undegraded":"2024-03-06T07:31:46.029386+0000","last_fullsized":"2024-03-06T07:31:46.029386+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:42.299308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044198+0000","last_change":"2024-03-06T07:31:45.035300+0000","last_active":"2024-03-06T07:31:46.044198+0000","last_peered":"2024-03-06T07:31:46.044198+0000","last_clean":"2024-03-06T07:31:46.044198+0000","last_became_active":"2024-03-06T07:31:45.034964+0000","last_became_peered":"2024-03-06T07:31:45.034964+0000","last_unstale":"2024-03-06T07:31:46.044198+0000","last_undegraded":"2024-03-06T07:31:46.044198+0000","last_fullsized":"2024-03-06T07:31:46.044198+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:16.375176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028276+0000","last_change":"2024-03-06T07:31:45.039202+0000","last_active":"2024-03-06T07:31:46.028276+0000","last_peered":"2024-03-06T07:31:46.028276+0000","last_clean":"2024-03-06T07:31:46.028276+0000","last_became_active":"2024-03-06T07:31:45.038902+0000","last_became_peered":"2024-03-06T07:31:45.038902+0000","last_unstale":"2024-03-06T07:31:46.028276+0000","last_undegraded":"2024-03-06T07:31:46.028276+0000","last_fullsized":"2024-03-06T07:31:46.028276+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:36.094842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047002+0000","last_change":"2024-03-06T07:31:45.047007+0000","last_active":"2024-03-06T07:31:46.047002+0000","last_peered":"2024-03-06T07:31:46.047002+0000","last_clean":"2024-03-06T07:31:46.047002+0000","last_became_active":"2024-03-06T07:31:45.046533+0000","last_became_peered":"2024-03-06T07:31:45.046533+0000","last_unstale":"2024-03-06T07:31:46.047002+0000","last_undegraded":"2024-03-06T07:31:46.047002+0000","last_fullsized":"2024-03-06T07:31:46.047002+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:34.869582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044541+0000","last_change":"2024-03-06T07:31:45.036443+0000","last_active":"2024-03-06T07:31:46.044541+0000","last_peered":"2024-03-06T07:31:46.044541+0000","last_clean":"2024-03-06T07:31:46.044541+0000","last_became_active":"2024-03-06T07:31:45.035986+0000","last_became_peered":"2024-03-06T07:31:45.035986+0000","last_unstale":"2024-03-06T07:31:46.044541+0000","last_undegraded":"2024-03-06T07:31:46.044541+0000","last_fullsized":"2024-03-06T07:31:46.044541+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:27.623139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047015+0000","last_change":"2024-03-06T07:31:45.049645+0000","last_active":"2024-03-06T07:31:46.047015+0000","last_peered":"2024-03-06T07:31:46.047015+0000","last_clean":"2024-03-06T07:31:46.047015+0000","last_became_active":"2024-03-06T07:31:45.048865+0000","last_became_peered":"2024-03-06T07:31:45.048865+0000","last_unstale":"2024-03-06T07:31:46.047015+0000","last_undegraded":"2024-03-06T07:31:46.047015+0000","last_fullsized":"2024-03-06T07:31:46.047015+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:18.474456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028966+0000","last_change":"2024-03-06T07:31:45.048451+0000","last_active":"2024-03-06T07:31:46.028966+0000","last_peered":"2024-03-06T07:31:46.028966+0000","last_clean":"2024-03-06T07:31:46.028966+0000","last_became_active":"2024-03-06T07:31:45.048188+0000","last_became_peered":"2024-03-06T07:31:45.048188+0000","last_unstale":"2024-03-06T07:31:46.028966+0000","last_undegraded":"2024-03-06T07:31:46.028966+0000","last_fullsized":"2024-03-06T07:31:46.028966+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:52.802401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995643+0000","last_change":"2024-03-06T07:31:45.042722+0000","last_active":"2024-03-06T07:31:46.995643+0000","last_peered":"2024-03-06T07:31:46.995643+0000","last_clean":"2024-03-06T07:31:46.995643+0000","last_became_active":"2024-03-06T07:31:45.042456+0000","last_became_peered":"2024-03-06T07:31:45.042456+0000","last_unstale":"2024-03-06T07:31:46.995643+0000","last_undegraded":"2024-03-06T07:31:46.995643+0000","last_fullsized":"2024-03-06T07:31:46.995643+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:58.325034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046758+0000","last_change":"2024-03-06T07:31:45.046120+0000","last_active":"2024-03-06T07:31:46.046758+0000","last_peered":"2024-03-06T07:31:46.046758+0000","last_clean":"2024-03-06T07:31:46.046758+0000","last_became_active":"2024-03-06T07:31:45.045855+0000","last_became_peered":"2024-03-06T07:31:45.045855+0000","last_unstale":"2024-03-06T07:31:46.046758+0000","last_undegraded":"2024-03-06T07:31:46.046758+0000","last_fullsized":"2024-03-06T07:31:46.046758+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:28.333006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438443+0000","last_change":"2024-03-06T07:31:45.039353+0000","last_active":"2024-03-06T07:31:46.438443+0000","last_peered":"2024-03-06T07:31:46.438443+0000","last_clean":"2024-03-06T07:31:46.438443+0000","last_became_active":"2024-03-06T07:31:45.039090+0000","last_became_peered":"2024-03-06T07:31:45.039090+0000","last_unstale":"2024-03-06T07:31:46.438443+0000","last_undegraded":"2024-03-06T07:31:46.438443+0000","last_fullsized":"2024-03-06T07:31:46.438443+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:04.808270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047413+0000","last_change":"2024-03-06T07:31:45.055564+0000","last_active":"2024-03-06T07:31:46.047413+0000","last_peered":"2024-03-06T07:31:46.047413+0000","last_clean":"2024-03-06T07:31:46.047413+0000","last_became_active":"2024-03-06T07:31:45.055141+0000","last_became_peered":"2024-03-06T07:31:45.055141+0000","last_unstale":"2024-03-06T07:31:46.047413+0000","last_undegraded":"2024-03-06T07:31:46.047413+0000","last_fullsized":"2024-03-06T07:31:46.047413+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:35.460837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196909+0000","last_change":"2024-03-06T07:31:45.046112+0000","last_active":"2024-03-06T07:31:47.196909+0000","last_peered":"2024-03-06T07:31:47.196909+0000","last_clean":"2024-03-06T07:31:47.196909+0000","last_became_active":"2024-03-06T07:31:45.043698+0000","last_became_peered":"2024-03-06T07:31:45.043698+0000","last_unstale":"2024-03-06T07:31:47.196909+0000","last_undegraded":"2024-03-06T07:31:47.196909+0000","last_fullsized":"2024-03-06T07:31:47.196909+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:57.381806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044618+0000","last_change":"2024-03-06T07:31:45.038374+0000","last_active":"2024-03-06T07:31:46.044618+0000","last_peered":"2024-03-06T07:31:46.044618+0000","last_clean":"2024-03-06T07:31:46.044618+0000","last_became_active":"2024-03-06T07:31:45.038088+0000","last_became_peered":"2024-03-06T07:31:45.038088+0000","last_unstale":"2024-03-06T07:31:46.044618+0000","last_undegraded":"2024-03-06T07:31:46.044618+0000","last_fullsized":"2024-03-06T07:31:46.044618+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:56.482865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438523+0000","last_change":"2024-03-06T07:31:45.032953+0000","last_active":"2024-03-06T07:31:46.438523+0000","last_peered":"2024-03-06T07:31:46.438523+0000","last_clean":"2024-03-06T07:31:46.438523+0000","last_became_active":"2024-03-06T07:31:45.032688+0000","last_became_peered":"2024-03-06T07:31:45.032688+0000","last_unstale":"2024-03-06T07:31:46.438523+0000","last_undegraded":"2024-03-06T07:31:46.438523+0000","last_fullsized":"2024-03-06T07:31:46.438523+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:19.547257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047108+0000","last_change":"2024-03-06T07:31:45.049212+0000","last_active":"2024-03-06T07:31:46.047108+0000","last_peered":"2024-03-06T07:31:46.047108+0000","last_clean":"2024-03-06T07:31:46.047108+0000","last_became_active":"2024-03-06T07:31:45.047359+0000","last_became_peered":"2024-03-06T07:31:45.047359+0000","last_unstale":"2024-03-06T07:31:46.047108+0000","last_undegraded":"2024-03-06T07:31:46.047108+0000","last_fullsized":"2024-03-06T07:31:46.047108+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:48.244230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995618+0000","last_change":"2024-03-06T07:31:45.035442+0000","last_active":"2024-03-06T07:31:46.995618+0000","last_peered":"2024-03-06T07:31:46.995618+0000","last_clean":"2024-03-06T07:31:46.995618+0000","last_became_active":"2024-03-06T07:31:45.035154+0000","last_became_peered":"2024-03-06T07:31:45.035154+0000","last_unstale":"2024-03-06T07:31:46.995618+0000","last_undegraded":"2024-03-06T07:31:46.995618+0000","last_fullsized":"2024-03-06T07:31:46.995618+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:38.092312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995455+0000","last_change":"2024-03-06T07:31:45.039599+0000","last_active":"2024-03-06T07:31:46.995455+0000","last_peered":"2024-03-06T07:31:46.995455+0000","last_clean":"2024-03-06T07:31:46.995455+0000","last_became_active":"2024-03-06T07:31:45.039297+0000","last_became_peered":"2024-03-06T07:31:45.039297+0000","last_unstale":"2024-03-06T07:31:46.995455+0000","last_undegraded":"2024-03-06T07:31:46.995455+0000","last_fullsized":"2024-03-06T07:31:46.995455+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:14.807136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046685+0000","last_change":"2024-03-06T07:31:45.050761+0000","last_active":"2024-03-06T07:31:46.046685+0000","last_peered":"2024-03-06T07:31:46.046685+0000","last_clean":"2024-03-06T07:31:46.046685+0000","last_became_active":"2024-03-06T07:31:45.047127+0000","last_became_peered":"2024-03-06T07:31:45.047127+0000","last_unstale":"2024-03-06T07:31:46.046685+0000","last_undegraded":"2024-03-06T07:31:46.046685+0000","last_fullsized":"2024-03-06T07:31:46.046685+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438703+0000","last_change":"2024-03-06T07:31:45.040040+0000","last_active":"2024-03-06T07:31:46.438703+0000","last_peered":"2024-03-06T07:31:46.438703+0000","last_clean":"2024-03-06T07:31:46.438703+0000","last_became_active":"2024-03-06T07:31:45.039753+0000","last_became_peered":"2024-03-06T07:31:45.039753+0000","last_unstale":"2024-03-06T07:31:46.438703+0000","last_undegraded":"2024-03-06T07:31:46.438703+0000","last_fullsized":"2024-03-06T07:31:46.438703+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:48.091616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044791+0000","last_change":"2024-03-06T07:31:45.037506+0000","last_active":"2024-03-06T07:31:46.044791+0000","last_peered":"2024-03-06T07:31:46.044791+0000","last_clean":"2024-03-06T07:31:46.044791+0000","last_became_active":"2024-03-06T07:31:45.036896+0000","last_became_peered":"2024-03-06T07:31:45.036896+0000","last_unstale":"2024-03-06T07:31:46.044791+0000","last_undegraded":"2024-03-06T07:31:46.044791+0000","last_fullsized":"2024-03-06T07:31:46.044791+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:00.686030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438729+0000","last_change":"2024-03-06T07:31:45.047399+0000","last_active":"2024-03-06T07:31:46.438729+0000","last_peered":"2024-03-06T07:31:46.438729+0000","last_clean":"2024-03-06T07:31:46.438729+0000","last_became_active":"2024-03-06T07:31:45.047116+0000","last_became_peered":"2024-03-06T07:31:45.047116+0000","last_unstale":"2024-03-06T07:31:46.438729+0000","last_undegraded":"2024-03-06T07:31:46.438729+0000","last_fullsized":"2024-03-06T07:31:46.438729+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:28.329642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029193+0000","last_change":"2024-03-06T07:31:45.048645+0000","last_active":"2024-03-06T07:31:46.029193+0000","last_peered":"2024-03-06T07:31:46.029193+0000","last_clean":"2024-03-06T07:31:46.029193+0000","last_became_active":"2024-03-06T07:31:45.048362+0000","last_became_peered":"2024-03-06T07:31:45.048362+0000","last_unstale":"2024-03-06T07:31:46.029193+0000","last_undegraded":"2024-03-06T07:31:46.029193+0000","last_fullsized":"2024-03-06T07:31:46.029193+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:41.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044368+0000","last_change":"2024-03-06T07:31:45.045636+0000","last_active":"2024-03-06T07:31:46.044368+0000","last_peered":"2024-03-06T07:31:46.044368+0000","last_clean":"2024-03-06T07:31:46.044368+0000","last_became_active":"2024-03-06T07:31:45.045309+0000","last_became_peered":"2024-03-06T07:31:45.045309+0000","last_unstale":"2024-03-06T07:31:46.044368+0000","last_undegraded":"2024-03-06T07:31:46.044368+0000","last_fullsized":"2024-03-06T07:31:46.044368+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:30.930763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995863+0000","last_change":"2024-03-06T07:31:45.041617+0000","last_active":"2024-03-06T07:31:46.995863+0000","last_peered":"2024-03-06T07:31:46.995863+0000","last_clean":"2024-03-06T07:31:46.995863+0000","last_became_active":"2024-03-06T07:31:45.040818+0000","last_became_peered":"2024-03-06T07:31:45.040818+0000","last_unstale":"2024-03-06T07:31:46.995863+0000","last_undegraded":"2024-03-06T07:31:46.995863+0000","last_fullsized":"2024-03-06T07:31:46.995863+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:30.292549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028323+0000","last_change":"2024-03-06T07:31:45.054122+0000","last_active":"2024-03-06T07:31:46.028323+0000","last_peered":"2024-03-06T07:31:46.028323+0000","last_clean":"2024-03-06T07:31:46.028323+0000","last_became_active":"2024-03-06T07:31:45.053807+0000","last_became_peered":"2024-03-06T07:31:45.053807+0000","last_unstale":"2024-03-06T07:31:46.028323+0000","last_undegraded":"2024-03-06T07:31:46.028323+0000","last_fullsized":"2024-03-06T07:31:46.028323+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:00.757771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438516+0000","last_change":"2024-03-06T07:31:45.032208+0000","last_active":"2024-03-06T07:31:46.438516+0000","last_peered":"2024-03-06T07:31:46.438516+0000","last_clean":"2024-03-06T07:31:46.438516+0000","last_became_active":"2024-03-06T07:31:45.031935+0000","last_became_peered":"2024-03-06T07:31:45.031935+0000","last_unstale":"2024-03-06T07:31:46.438516+0000","last_undegraded":"2024-03-06T07:31:46.438516+0000","last_fullsized":"2024-03-06T07:31:46.438516+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:17.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047087+0000","last_change":"2024-03-06T07:31:45.046451+0000","last_active":"2024-03-06T07:31:46.047087+0000","last_peered":"2024-03-06T07:31:46.047087+0000","last_clean":"2024-03-06T07:31:46.047087+0000","last_became_active":"2024-03-06T07:31:45.045791+0000","last_became_peered":"2024-03-06T07:31:45.045791+0000","last_unstale":"2024-03-06T07:31:46.047087+0000","last_undegraded":"2024-03-06T07:31:46.047087+0000","last_fullsized":"2024-03-06T07:31:46.047087+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:41.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"144'4","reported_seq":18,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047262+0000","last_change":"2024-03-06T07:31:44.033293+0000","last_active":"2024-03-06T07:31:46.047262+0000","last_peered":"2024-03-06T07:31:46.047262+0000","last_clean":"2024-03-06T07:31:46.047262+0000","last_became_active":"2024-03-06T07:31:44.032831+0000","last_became_peered":"2024-03-06T07:31:44.032831+0000","last_unstale":"2024-03-06T07:31:46.047262+0000","last_undegraded":"2024-03-06T07:31:46.047262+0000","last_fullsized":"2024-03-06T07:31:46.047262+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:43.731816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028425+0000","last_change":"2024-03-06T07:31:45.040911+0000","last_active":"2024-03-06T07:31:46.028425+0000","last_peered":"2024-03-06T07:31:46.028425+0000","last_clean":"2024-03-06T07:31:46.028425+0000","last_became_active":"2024-03-06T07:31:45.040632+0000","last_became_peered":"2024-03-06T07:31:45.040632+0000","last_unstale":"2024-03-06T07:31:46.028425+0000","last_undegraded":"2024-03-06T07:31:46.028425+0000","last_fullsized":"2024-03-06T07:31:46.028425+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:34.247852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028531+0000","last_change":"2024-03-06T07:31:44.027157+0000","last_active":"2024-03-06T07:31:46.028531+0000","last_peered":"2024-03-06T07:31:46.028531+0000","last_clean":"2024-03-06T07:31:46.028531+0000","last_became_active":"2024-03-06T07:31:44.026865+0000","last_became_peered":"2024-03-06T07:31:44.026865+0000","last_unstale":"2024-03-06T07:31:46.028531+0000","last_undegraded":"2024-03-06T07:31:46.028531+0000","last_fullsized":"2024-03-06T07:31:46.028531+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044375+0000","last_change":"2024-03-06T07:31:45.042098+0000","last_active":"2024-03-06T07:31:46.044375+0000","last_peered":"2024-03-06T07:31:46.044375+0000","last_clean":"2024-03-06T07:31:46.044375+0000","last_became_active":"2024-03-06T07:31:45.041801+0000","last_became_peered":"2024-03-06T07:31:45.041801+0000","last_unstale":"2024-03-06T07:31:46.044375+0000","last_undegraded":"2024-03-06T07:31:46.044375+0000","last_fullsized":"2024-03-06T07:31:46.044375+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:48.312427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995745+0000","last_change":"2024-03-06T07:31:44.023697+0000","last_active":"2024-03-06T07:31:46.995745+0000","last_peered":"2024-03-06T07:31:46.995745+0000","last_clean":"2024-03-06T07:31:46.995745+0000","last_became_active":"2024-03-06T07:31:44.023399+0000","last_became_peered":"2024-03-06T07:31:44.023399+0000","last_unstale":"2024-03-06T07:31:46.995745+0000","last_undegraded":"2024-03-06T07:31:46.995745+0000","last_fullsized":"2024-03-06T07:31:46.995745+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:19.201492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046990+0000","last_change":"2024-03-06T07:31:45.046580+0000","last_active":"2024-03-06T07:31:46.046990+0000","last_peered":"2024-03-06T07:31:46.046990+0000","last_clean":"2024-03-06T07:31:46.046990+0000","last_became_active":"2024-03-06T07:31:45.046312+0000","last_became_peered":"2024-03-06T07:31:45.046312+0000","last_unstale":"2024-03-06T07:31:46.046990+0000","last_undegraded":"2024-03-06T07:31:46.046990+0000","last_fullsized":"2024-03-06T07:31:46.046990+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:55.208319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995866+0000","last_change":"2024-03-06T07:31:44.020513+0000","last_active":"2024-03-06T07:31:46.995866+0000","last_peered":"2024-03-06T07:31:46.995866+0000","last_clean":"2024-03-06T07:31:46.995866+0000","last_became_active":"2024-03-06T07:31:44.020196+0000","last_became_peered":"2024-03-06T07:31:44.020196+0000","last_unstale":"2024-03-06T07:31:46.995866+0000","last_undegraded":"2024-03-06T07:31:46.995866+0000","last_fullsized":"2024-03-06T07:31:46.995866+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:47.307708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028263+0000","last_change":"2024-03-06T07:31:45.045575+0000","last_active":"2024-03-06T07:31:46.028263+0000","last_peered":"2024-03-06T07:31:46.028263+0000","last_clean":"2024-03-06T07:31:46.028263+0000","last_became_active":"2024-03-06T07:31:45.045312+0000","last_became_peered":"2024-03-06T07:31:45.045312+0000","last_unstale":"2024-03-06T07:31:46.028263+0000","last_undegraded":"2024-03-06T07:31:46.028263+0000","last_fullsized":"2024-03-06T07:31:46.028263+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:20.191298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438495+0000","last_change":"2024-03-06T07:31:44.033751+0000","last_active":"2024-03-06T07:31:46.438495+0000","last_peered":"2024-03-06T07:31:46.438495+0000","last_clean":"2024-03-06T07:31:46.438495+0000","last_became_active":"2024-03-06T07:31:44.033386+0000","last_became_peered":"2024-03-06T07:31:44.033386+0000","last_unstale":"2024-03-06T07:31:46.438495+0000","last_undegraded":"2024-03-06T07:31:46.438495+0000","last_fullsized":"2024-03-06T07:31:46.438495+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:50.356019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995664+0000","last_change":"2024-03-06T07:31:45.045822+0000","last_active":"2024-03-06T07:31:46.995664+0000","last_peered":"2024-03-06T07:31:46.995664+0000","last_clean":"2024-03-06T07:31:46.995664+0000","last_became_active":"2024-03-06T07:31:45.045552+0000","last_became_peered":"2024-03-06T07:31:45.045552+0000","last_unstale":"2024-03-06T07:31:46.995664+0000","last_undegraded":"2024-03-06T07:31:46.995664+0000","last_fullsized":"2024-03-06T07:31:46.995664+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:55.936260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028694+0000","last_change":"2024-03-06T07:31:44.028730+0000","last_active":"2024-03-06T07:31:46.028694+0000","last_peered":"2024-03-06T07:31:46.028694+0000","last_clean":"2024-03-06T07:31:46.028694+0000","last_became_active":"2024-03-06T07:31:44.028446+0000","last_became_peered":"2024-03-06T07:31:44.028446+0000","last_unstale":"2024-03-06T07:31:46.028694+0000","last_undegraded":"2024-03-06T07:31:46.028694+0000","last_fullsized":"2024-03-06T07:31:46.028694+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:38.934563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028146+0000","last_change":"2024-03-06T07:31:45.033665+0000","last_active":"2024-03-06T07:31:46.028146+0000","last_peered":"2024-03-06T07:31:46.028146+0000","last_clean":"2024-03-06T07:31:46.028146+0000","last_became_active":"2024-03-06T07:31:45.033369+0000","last_became_peered":"2024-03-06T07:31:45.033369+0000","last_unstale":"2024-03-06T07:31:46.028146+0000","last_undegraded":"2024-03-06T07:31:46.028146+0000","last_fullsized":"2024-03-06T07:31:46.028146+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:36.725280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438556+0000","last_change":"2024-03-06T07:31:44.024794+0000","last_active":"2024-03-06T07:31:46.438556+0000","last_peered":"2024-03-06T07:31:46.438556+0000","last_clean":"2024-03-06T07:31:46.438556+0000","last_became_active":"2024-03-06T07:31:44.024499+0000","last_became_peered":"2024-03-06T07:31:44.024499+0000","last_unstale":"2024-03-06T07:31:46.438556+0000","last_undegraded":"2024-03-06T07:31:46.438556+0000","last_fullsized":"2024-03-06T07:31:46.438556+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:01.912662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028691+0000","last_change":"2024-03-06T07:31:45.040904+0000","last_active":"2024-03-06T07:31:46.028691+0000","last_peered":"2024-03-06T07:31:46.028691+0000","last_clean":"2024-03-06T07:31:46.028691+0000","last_became_active":"2024-03-06T07:31:45.040606+0000","last_became_peered":"2024-03-06T07:31:45.040606+0000","last_unstale":"2024-03-06T07:31:46.028691+0000","last_undegraded":"2024-03-06T07:31:46.028691+0000","last_fullsized":"2024-03-06T07:31:46.028691+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:48.091616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995364+0000","last_change":"2024-03-06T07:31:44.022269+0000","last_active":"2024-03-06T07:31:46.995364+0000","last_peered":"2024-03-06T07:31:46.995364+0000","last_clean":"2024-03-06T07:31:46.995364+0000","last_became_active":"2024-03-06T07:31:44.021982+0000","last_became_peered":"2024-03-06T07:31:44.021982+0000","last_unstale":"2024-03-06T07:31:46.995364+0000","last_undegraded":"2024-03-06T07:31:46.995364+0000","last_fullsized":"2024-03-06T07:31:46.995364+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:51.573386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029104+0000","last_change":"2024-03-06T07:31:45.037434+0000","last_active":"2024-03-06T07:31:46.029104+0000","last_peered":"2024-03-06T07:31:46.029104+0000","last_clean":"2024-03-06T07:31:46.029104+0000","last_became_active":"2024-03-06T07:31:45.037148+0000","last_became_peered":"2024-03-06T07:31:45.037148+0000","last_unstale":"2024-03-06T07:31:46.029104+0000","last_undegraded":"2024-03-06T07:31:46.029104+0000","last_fullsized":"2024-03-06T07:31:46.029104+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:48.312427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028537+0000","last_change":"2024-03-06T07:31:44.019467+0000","last_active":"2024-03-06T07:31:46.028537+0000","last_peered":"2024-03-06T07:31:46.028537+0000","last_clean":"2024-03-06T07:31:46.028537+0000","last_became_active":"2024-03-06T07:31:44.019183+0000","last_became_peered":"2024-03-06T07:31:44.019183+0000","last_unstale":"2024-03-06T07:31:46.028537+0000","last_undegraded":"2024-03-06T07:31:46.028537+0000","last_fullsized":"2024-03-06T07:31:46.028537+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:22.325138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047308+0000","last_change":"2024-03-06T07:31:45.052902+0000","last_active":"2024-03-06T07:31:46.047308+0000","last_peered":"2024-03-06T07:31:46.047308+0000","last_clean":"2024-03-06T07:31:46.047308+0000","last_became_active":"2024-03-06T07:31:45.052223+0000","last_became_peered":"2024-03-06T07:31:45.052223+0000","last_unstale":"2024-03-06T07:31:46.047308+0000","last_undegraded":"2024-03-06T07:31:46.047308+0000","last_fullsized":"2024-03-06T07:31:46.047308+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:03.319261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047254+0000","last_change":"2024-03-06T07:31:44.031262+0000","last_active":"2024-03-06T07:31:46.047254+0000","last_peered":"2024-03-06T07:31:46.047254+0000","last_clean":"2024-03-06T07:31:46.047254+0000","last_became_active":"2024-03-06T07:31:44.030401+0000","last_became_peered":"2024-03-06T07:31:44.030401+0000","last_unstale":"2024-03-06T07:31:46.047254+0000","last_undegraded":"2024-03-06T07:31:46.047254+0000","last_fullsized":"2024-03-06T07:31:46.047254+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:44.318964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438725+0000","last_change":"2024-03-06T07:31:45.033444+0000","last_active":"2024-03-06T07:31:46.438725+0000","last_peered":"2024-03-06T07:31:46.438725+0000","last_clean":"2024-03-06T07:31:46.438725+0000","last_became_active":"2024-03-06T07:31:45.033109+0000","last_became_peered":"2024-03-06T07:31:45.033109+0000","last_unstale":"2024-03-06T07:31:46.438725+0000","last_undegraded":"2024-03-06T07:31:46.438725+0000","last_fullsized":"2024-03-06T07:31:46.438725+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:36.113249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"144'6","reported_seq":20,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029449+0000","last_change":"2024-03-06T07:31:44.026781+0000","last_active":"2024-03-06T07:31:46.029449+0000","last_peered":"2024-03-06T07:31:46.029449+0000","last_clean":"2024-03-06T07:31:46.029449+0000","last_became_active":"2024-03-06T07:31:44.026495+0000","last_became_peered":"2024-03-06T07:31:44.026495+0000","last_unstale":"2024-03-06T07:31:46.029449+0000","last_undegraded":"2024-03-06T07:31:46.029449+0000","last_fullsized":"2024-03-06T07:31:46.029449+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:35.090123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196757+0000","last_change":"2024-03-06T07:31:45.043899+0000","last_active":"2024-03-06T07:31:47.196757+0000","last_peered":"2024-03-06T07:31:47.196757+0000","last_clean":"2024-03-06T07:31:47.196757+0000","last_became_active":"2024-03-06T07:31:45.043131+0000","last_became_peered":"2024-03-06T07:31:45.043131+0000","last_unstale":"2024-03-06T07:31:47.196757+0000","last_undegraded":"2024-03-06T07:31:47.196757+0000","last_fullsized":"2024-03-06T07:31:47.196757+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:30.292549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196947+0000","last_change":"2024-03-06T07:31:44.026616+0000","last_active":"2024-03-06T07:31:47.196947+0000","last_peered":"2024-03-06T07:31:47.196947+0000","last_clean":"2024-03-06T07:31:47.196947+0000","last_became_active":"2024-03-06T07:31:44.025510+0000","last_became_peered":"2024-03-06T07:31:44.025510+0000","last_unstale":"2024-03-06T07:31:47.196947+0000","last_undegraded":"2024-03-06T07:31:47.196947+0000","last_fullsized":"2024-03-06T07:31:47.196947+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:13.314073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028194+0000","last_change":"2024-03-06T07:31:45.041996+0000","last_active":"2024-03-06T07:31:46.028194+0000","last_peered":"2024-03-06T07:31:46.028194+0000","last_clean":"2024-03-06T07:31:46.028194+0000","last_became_active":"2024-03-06T07:31:45.041403+0000","last_became_peered":"2024-03-06T07:31:45.041403+0000","last_unstale":"2024-03-06T07:31:46.028194+0000","last_undegraded":"2024-03-06T07:31:46.028194+0000","last_fullsized":"2024-03-06T07:31:46.028194+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:49.751709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196762+0000","last_change":"2024-03-06T07:31:44.022857+0000","last_active":"2024-03-06T07:31:47.196762+0000","last_peered":"2024-03-06T07:31:47.196762+0000","last_clean":"2024-03-06T07:31:47.196762+0000","last_became_active":"2024-03-06T07:31:44.022400+0000","last_became_peered":"2024-03-06T07:31:44.022400+0000","last_unstale":"2024-03-06T07:31:47.196762+0000","last_undegraded":"2024-03-06T07:31:47.196762+0000","last_fullsized":"2024-03-06T07:31:47.196762+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:55.478147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028725+0000","last_change":"2024-03-06T07:31:45.042751+0000","last_active":"2024-03-06T07:31:46.028725+0000","last_peered":"2024-03-06T07:31:46.028725+0000","last_clean":"2024-03-06T07:31:46.028725+0000","last_became_active":"2024-03-06T07:31:45.042477+0000","last_became_peered":"2024-03-06T07:31:45.042477+0000","last_unstale":"2024-03-06T07:31:46.028725+0000","last_undegraded":"2024-03-06T07:31:46.028725+0000","last_fullsized":"2024-03-06T07:31:46.028725+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:36.113249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044190+0000","last_change":"2024-03-06T07:31:44.023105+0000","last_active":"2024-03-06T07:31:46.044190+0000","last_peered":"2024-03-06T07:31:46.044190+0000","last_clean":"2024-03-06T07:31:46.044190+0000","last_became_active":"2024-03-06T07:31:44.022808+0000","last_became_peered":"2024-03-06T07:31:44.022808+0000","last_unstale":"2024-03-06T07:31:46.044190+0000","last_undegraded":"2024-03-06T07:31:46.044190+0000","last_fullsized":"2024-03-06T07:31:46.044190+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196592+0000","last_change":"2024-03-06T07:31:45.047987+0000","last_active":"2024-03-06T07:31:47.196592+0000","last_peered":"2024-03-06T07:31:47.196592+0000","last_clean":"2024-03-06T07:31:47.196592+0000","last_became_active":"2024-03-06T07:31:45.045672+0000","last_became_peered":"2024-03-06T07:31:45.045672+0000","last_unstale":"2024-03-06T07:31:47.196592+0000","last_undegraded":"2024-03-06T07:31:47.196592+0000","last_fullsized":"2024-03-06T07:31:47.196592+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:34.148588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"144'8","reported_seq":22,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438641+0000","last_change":"2024-03-06T07:31:44.024742+0000","last_active":"2024-03-06T07:31:46.438641+0000","last_peered":"2024-03-06T07:31:46.438641+0000","last_clean":"2024-03-06T07:31:46.438641+0000","last_became_active":"2024-03-06T07:31:44.024445+0000","last_became_peered":"2024-03-06T07:31:44.024445+0000","last_unstale":"2024-03-06T07:31:46.438641+0000","last_undegraded":"2024-03-06T07:31:46.438641+0000","last_fullsized":"2024-03-06T07:31:46.438641+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:00.838367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047192+0000","last_change":"2024-03-06T07:31:45.052132+0000","last_active":"2024-03-06T07:31:46.047192+0000","last_peered":"2024-03-06T07:31:46.047192+0000","last_clean":"2024-03-06T07:31:46.047192+0000","last_became_active":"2024-03-06T07:31:45.051795+0000","last_became_peered":"2024-03-06T07:31:45.051795+0000","last_unstale":"2024-03-06T07:31:46.047192+0000","last_undegraded":"2024-03-06T07:31:46.047192+0000","last_fullsized":"2024-03-06T07:31:46.047192+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:42.299308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029041+0000","last_change":"2024-03-06T07:31:44.023389+0000","last_active":"2024-03-06T07:31:46.029041+0000","last_peered":"2024-03-06T07:31:46.029041+0000","last_clean":"2024-03-06T07:31:46.029041+0000","last_became_active":"2024-03-06T07:31:44.023124+0000","last_became_peered":"2024-03-06T07:31:44.023124+0000","last_unstale":"2024-03-06T07:31:46.029041+0000","last_undegraded":"2024-03-06T07:31:46.029041+0000","last_fullsized":"2024-03-06T07:31:46.029041+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438787+0000","last_change":"2024-03-06T07:31:45.044088+0000","last_active":"2024-03-06T07:31:46.438787+0000","last_peered":"2024-03-06T07:31:46.438787+0000","last_clean":"2024-03-06T07:31:46.438787+0000","last_became_active":"2024-03-06T07:31:45.043830+0000","last_became_peered":"2024-03-06T07:31:45.043830+0000","last_unstale":"2024-03-06T07:31:46.438787+0000","last_undegraded":"2024-03-06T07:31:46.438787+0000","last_fullsized":"2024-03-06T07:31:46.438787+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:31.902903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028699+0000","last_change":"2024-03-06T07:31:44.024253+0000","last_active":"2024-03-06T07:31:46.028699+0000","last_peered":"2024-03-06T07:31:46.028699+0000","last_clean":"2024-03-06T07:31:46.028699+0000","last_became_active":"2024-03-06T07:31:44.023842+0000","last_became_peered":"2024-03-06T07:31:44.023842+0000","last_unstale":"2024-03-06T07:31:46.028699+0000","last_undegraded":"2024-03-06T07:31:46.028699+0000","last_fullsized":"2024-03-06T07:31:46.028699+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196985+0000","last_change":"2024-03-06T07:31:45.045694+0000","last_active":"2024-03-06T07:31:47.196985+0000","last_peered":"2024-03-06T07:31:47.196985+0000","last_clean":"2024-03-06T07:31:47.196985+0000","last_became_active":"2024-03-06T07:31:45.044744+0000","last_became_peered":"2024-03-06T07:31:45.044744+0000","last_unstale":"2024-03-06T07:31:47.196985+0000","last_undegraded":"2024-03-06T07:31:47.196985+0000","last_fullsized":"2024-03-06T07:31:47.196985+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:05.832668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438842+0000","last_change":"2024-03-06T07:31:44.035542+0000","last_active":"2024-03-06T07:31:46.438842+0000","last_peered":"2024-03-06T07:31:46.438842+0000","last_clean":"2024-03-06T07:31:46.438842+0000","last_became_active":"2024-03-06T07:31:44.035318+0000","last_became_peered":"2024-03-06T07:31:44.035318+0000","last_unstale":"2024-03-06T07:31:46.438842+0000","last_undegraded":"2024-03-06T07:31:46.438842+0000","last_fullsized":"2024-03-06T07:31:46.438842+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:03.773449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044394+0000","last_change":"2024-03-06T07:31:45.055140+0000","last_active":"2024-03-06T07:31:46.044394+0000","last_peered":"2024-03-06T07:31:46.044394+0000","last_clean":"2024-03-06T07:31:46.044394+0000","last_became_active":"2024-03-06T07:31:45.054859+0000","last_became_peered":"2024-03-06T07:31:45.054859+0000","last_unstale":"2024-03-06T07:31:46.044394+0000","last_undegraded":"2024-03-06T07:31:46.044394+0000","last_fullsized":"2024-03-06T07:31:46.044394+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:20.206211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028592+0000","last_change":"2024-03-06T07:31:44.028091+0000","last_active":"2024-03-06T07:31:46.028592+0000","last_peered":"2024-03-06T07:31:46.028592+0000","last_clean":"2024-03-06T07:31:46.028592+0000","last_became_active":"2024-03-06T07:31:44.027829+0000","last_became_peered":"2024-03-06T07:31:44.027829+0000","last_unstale":"2024-03-06T07:31:46.028592+0000","last_undegraded":"2024-03-06T07:31:46.028592+0000","last_fullsized":"2024-03-06T07:31:46.028592+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:08.912427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995910+0000","last_change":"2024-03-06T07:31:45.039966+0000","last_active":"2024-03-06T07:31:46.995910+0000","last_peered":"2024-03-06T07:31:46.995910+0000","last_clean":"2024-03-06T07:31:46.995910+0000","last_became_active":"2024-03-06T07:31:45.039079+0000","last_became_peered":"2024-03-06T07:31:45.039079+0000","last_unstale":"2024-03-06T07:31:46.995910+0000","last_undegraded":"2024-03-06T07:31:46.995910+0000","last_fullsized":"2024-03-06T07:31:46.995910+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:57.381806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029266+0000","last_change":"2024-03-06T07:31:44.024320+0000","last_active":"2024-03-06T07:31:46.029266+0000","last_peered":"2024-03-06T07:31:46.029266+0000","last_clean":"2024-03-06T07:31:46.029266+0000","last_became_active":"2024-03-06T07:31:44.024051+0000","last_became_peered":"2024-03-06T07:31:44.024051+0000","last_unstale":"2024-03-06T07:31:46.029266+0000","last_undegraded":"2024-03-06T07:31:46.029266+0000","last_fullsized":"2024-03-06T07:31:46.029266+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:59.753052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029288+0000","last_change":"2024-03-06T07:31:45.035119+0000","last_active":"2024-03-06T07:31:46.029288+0000","last_peered":"2024-03-06T07:31:46.029288+0000","last_clean":"2024-03-06T07:31:46.029288+0000","last_became_active":"2024-03-06T07:31:45.034850+0000","last_became_peered":"2024-03-06T07:31:45.034850+0000","last_unstale":"2024-03-06T07:31:46.029288+0000","last_undegraded":"2024-03-06T07:31:46.029288+0000","last_fullsized":"2024-03-06T07:31:46.029288+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:03.319261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046997+0000","last_change":"2024-03-06T07:31:44.031164+0000","last_active":"2024-03-06T07:31:46.046997+0000","last_peered":"2024-03-06T07:31:46.046997+0000","last_clean":"2024-03-06T07:31:46.046997+0000","last_became_active":"2024-03-06T07:31:44.030649+0000","last_became_peered":"2024-03-06T07:31:44.030649+0000","last_unstale":"2024-03-06T07:31:46.046997+0000","last_undegraded":"2024-03-06T07:31:46.046997+0000","last_fullsized":"2024-03-06T07:31:46.046997+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995324+0000","last_change":"2024-03-06T07:31:45.043562+0000","last_active":"2024-03-06T07:31:46.995324+0000","last_peered":"2024-03-06T07:31:46.995324+0000","last_clean":"2024-03-06T07:31:46.995324+0000","last_became_active":"2024-03-06T07:31:45.043293+0000","last_became_peered":"2024-03-06T07:31:45.043293+0000","last_unstale":"2024-03-06T07:31:46.995324+0000","last_undegraded":"2024-03-06T07:31:46.995324+0000","last_fullsized":"2024-03-06T07:31:46.995324+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:26.528565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046682+0000","last_change":"2024-03-06T07:31:44.032582+0000","last_active":"2024-03-06T07:31:46.046682+0000","last_peered":"2024-03-06T07:31:46.046682+0000","last_clean":"2024-03-06T07:31:46.046682+0000","last_became_active":"2024-03-06T07:31:44.032075+0000","last_became_peered":"2024-03-06T07:31:44.032075+0000","last_unstale":"2024-03-06T07:31:46.046682+0000","last_undegraded":"2024-03-06T07:31:46.046682+0000","last_fullsized":"2024-03-06T07:31:46.046682+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029119+0000","last_change":"2024-03-06T07:31:45.035199+0000","last_active":"2024-03-06T07:31:46.029119+0000","last_peered":"2024-03-06T07:31:46.029119+0000","last_clean":"2024-03-06T07:31:46.029119+0000","last_became_active":"2024-03-06T07:31:45.034925+0000","last_became_peered":"2024-03-06T07:31:45.034925+0000","last_unstale":"2024-03-06T07:31:46.029119+0000","last_undegraded":"2024-03-06T07:31:46.029119+0000","last_fullsized":"2024-03-06T07:31:46.029119+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:18.474456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196488+0000","last_change":"2024-03-06T07:31:44.031018+0000","last_active":"2024-03-06T07:31:47.196488+0000","last_peered":"2024-03-06T07:31:47.196488+0000","last_clean":"2024-03-06T07:31:47.196488+0000","last_became_active":"2024-03-06T07:31:44.030695+0000","last_became_peered":"2024-03-06T07:31:44.030695+0000","last_unstale":"2024-03-06T07:31:47.196488+0000","last_undegraded":"2024-03-06T07:31:47.196488+0000","last_fullsized":"2024-03-06T07:31:47.196488+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:51.791822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029090+0000","last_change":"2024-03-06T07:31:45.043749+0000","last_active":"2024-03-06T07:31:46.029090+0000","last_peered":"2024-03-06T07:31:46.029090+0000","last_clean":"2024-03-06T07:31:46.029090+0000","last_became_active":"2024-03-06T07:31:45.043501+0000","last_became_peered":"2024-03-06T07:31:45.043501+0000","last_unstale":"2024-03-06T07:31:46.029090+0000","last_undegraded":"2024-03-06T07:31:46.029090+0000","last_fullsized":"2024-03-06T07:31:46.029090+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:30.930763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047400+0000","last_change":"2024-03-06T07:31:44.033647+0000","last_active":"2024-03-06T07:31:46.047400+0000","last_peered":"2024-03-06T07:31:46.047400+0000","last_clean":"2024-03-06T07:31:46.047400+0000","last_became_active":"2024-03-06T07:31:44.033393+0000","last_became_peered":"2024-03-06T07:31:44.033393+0000","last_unstale":"2024-03-06T07:31:46.047400+0000","last_undegraded":"2024-03-06T07:31:46.047400+0000","last_fullsized":"2024-03-06T07:31:46.047400+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:35.425689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.197000+0000","last_change":"2024-03-06T07:31:45.045750+0000","last_active":"2024-03-06T07:31:47.197000+0000","last_peered":"2024-03-06T07:31:47.197000+0000","last_clean":"2024-03-06T07:31:47.197000+0000","last_became_active":"2024-03-06T07:31:45.044153+0000","last_became_peered":"2024-03-06T07:31:45.044153+0000","last_unstale":"2024-03-06T07:31:47.197000+0000","last_undegraded":"2024-03-06T07:31:47.197000+0000","last_fullsized":"2024-03-06T07:31:47.197000+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:07.832011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"145'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044808+0000","last_change":"2024-03-06T07:31:44.028776+0000","last_active":"2024-03-06T07:31:46.044808+0000","last_peered":"2024-03-06T07:31:46.044808+0000","last_clean":"2024-03-06T07:31:46.044808+0000","last_became_active":"2024-03-06T07:31:44.028504+0000","last_became_peered":"2024-03-06T07:31:44.028504+0000","last_unstale":"2024-03-06T07:31:46.044808+0000","last_undegraded":"2024-03-06T07:31:46.044808+0000","last_fullsized":"2024-03-06T07:31:46.044808+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:17.358709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028585+0000","last_change":"2024-03-06T07:31:45.040051+0000","last_active":"2024-03-06T07:31:46.028585+0000","last_peered":"2024-03-06T07:31:46.028585+0000","last_clean":"2024-03-06T07:31:46.028585+0000","last_became_active":"2024-03-06T07:31:45.039760+0000","last_became_peered":"2024-03-06T07:31:45.039760+0000","last_unstale":"2024-03-06T07:31:46.028585+0000","last_undegraded":"2024-03-06T07:31:46.028585+0000","last_fullsized":"2024-03-06T07:31:46.028585+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044645+0000","last_change":"2024-03-06T07:31:44.027749+0000","last_active":"2024-03-06T07:31:46.044645+0000","last_peered":"2024-03-06T07:31:46.044645+0000","last_clean":"2024-03-06T07:31:46.044645+0000","last_became_active":"2024-03-06T07:31:44.027413+0000","last_became_peered":"2024-03-06T07:31:44.027413+0000","last_unstale":"2024-03-06T07:31:46.044645+0000","last_undegraded":"2024-03-06T07:31:46.044645+0000","last_fullsized":"2024-03-06T07:31:46.044645+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:26.520317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044559+0000","last_change":"2024-03-06T07:31:45.036246+0000","last_active":"2024-03-06T07:31:46.044559+0000","last_peered":"2024-03-06T07:31:46.044559+0000","last_clean":"2024-03-06T07:31:46.044559+0000","last_became_active":"2024-03-06T07:31:45.035684+0000","last_became_peered":"2024-03-06T07:31:45.035684+0000","last_unstale":"2024-03-06T07:31:46.044559+0000","last_undegraded":"2024-03-06T07:31:46.044559+0000","last_fullsized":"2024-03-06T07:31:46.044559+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:33.886298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044417+0000","last_change":"2024-03-06T07:31:44.027642+0000","last_active":"2024-03-06T07:31:46.044417+0000","last_peered":"2024-03-06T07:31:46.044417+0000","last_clean":"2024-03-06T07:31:46.044417+0000","last_became_active":"2024-03-06T07:31:44.027355+0000","last_became_peered":"2024-03-06T07:31:44.027355+0000","last_unstale":"2024-03-06T07:31:46.044417+0000","last_undegraded":"2024-03-06T07:31:46.044417+0000","last_fullsized":"2024-03-06T07:31:46.044417+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.058715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438511+0000","last_change":"2024-03-06T07:31:45.034848+0000","last_active":"2024-03-06T07:31:46.438511+0000","last_peered":"2024-03-06T07:31:46.438511+0000","last_clean":"2024-03-06T07:31:46.438511+0000","last_became_active":"2024-03-06T07:31:45.034565+0000","last_became_peered":"2024-03-06T07:31:45.034565+0000","last_unstale":"2024-03-06T07:31:46.438511+0000","last_undegraded":"2024-03-06T07:31:46.438511+0000","last_fullsized":"2024-03-06T07:31:46.438511+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.690561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029380+0000","last_change":"2024-03-06T07:31:44.025967+0000","last_active":"2024-03-06T07:31:46.029380+0000","last_peered":"2024-03-06T07:31:46.029380+0000","last_clean":"2024-03-06T07:31:46.029380+0000","last_became_active":"2024-03-06T07:31:44.025640+0000","last_became_peered":"2024-03-06T07:31:44.025640+0000","last_unstale":"2024-03-06T07:31:46.029380+0000","last_undegraded":"2024-03-06T07:31:46.029380+0000","last_fullsized":"2024-03-06T07:31:46.029380+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:48.746990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029016+0000","last_change":"2024-03-06T07:31:45.045298+0000","last_active":"2024-03-06T07:31:46.029016+0000","last_peered":"2024-03-06T07:31:46.029016+0000","last_clean":"2024-03-06T07:31:46.029016+0000","last_became_active":"2024-03-06T07:31:45.045033+0000","last_became_peered":"2024-03-06T07:31:45.045033+0000","last_unstale":"2024-03-06T07:31:46.029016+0000","last_undegraded":"2024-03-06T07:31:46.029016+0000","last_fullsized":"2024-03-06T07:31:46.029016+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:16.375176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046828+0000","last_change":"2024-03-06T07:31:44.032074+0000","last_active":"2024-03-06T07:31:46.046828+0000","last_peered":"2024-03-06T07:31:46.046828+0000","last_clean":"2024-03-06T07:31:46.046828+0000","last_became_active":"2024-03-06T07:31:44.031715+0000","last_became_peered":"2024-03-06T07:31:44.031715+0000","last_unstale":"2024-03-06T07:31:46.046828+0000","last_undegraded":"2024-03-06T07:31:46.046828+0000","last_fullsized":"2024-03-06T07:31:46.046828+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046731+0000","last_change":"2024-03-06T07:31:45.052042+0000","last_active":"2024-03-06T07:31:46.046731+0000","last_peered":"2024-03-06T07:31:46.046731+0000","last_clean":"2024-03-06T07:31:46.046731+0000","last_became_active":"2024-03-06T07:31:45.051442+0000","last_became_peered":"2024-03-06T07:31:45.051442+0000","last_unstale":"2024-03-06T07:31:46.046731+0000","last_undegraded":"2024-03-06T07:31:46.046731+0000","last_fullsized":"2024-03-06T07:31:46.046731+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:20.551707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196627+0000","last_change":"2024-03-06T07:31:44.025352+0000","last_active":"2024-03-06T07:31:47.196627+0000","last_peered":"2024-03-06T07:31:47.196627+0000","last_clean":"2024-03-06T07:31:47.196627+0000","last_became_active":"2024-03-06T07:31:44.025043+0000","last_became_peered":"2024-03-06T07:31:44.025043+0000","last_unstale":"2024-03-06T07:31:47.196627+0000","last_undegraded":"2024-03-06T07:31:47.196627+0000","last_fullsized":"2024-03-06T07:31:47.196627+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:59.681311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028726+0000","last_change":"2024-03-06T07:31:45.035827+0000","last_active":"2024-03-06T07:31:46.028726+0000","last_peered":"2024-03-06T07:31:46.028726+0000","last_clean":"2024-03-06T07:31:46.028726+0000","last_became_active":"2024-03-06T07:31:45.035484+0000","last_became_peered":"2024-03-06T07:31:45.035484+0000","last_unstale":"2024-03-06T07:31:46.028726+0000","last_undegraded":"2024-03-06T07:31:46.028726+0000","last_fullsized":"2024-03-06T07:31:46.028726+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:28.329642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028297+0000","last_change":"2024-03-06T07:31:44.027942+0000","last_active":"2024-03-06T07:31:46.028297+0000","last_peered":"2024-03-06T07:31:46.028297+0000","last_clean":"2024-03-06T07:31:46.028297+0000","last_became_active":"2024-03-06T07:31:44.027666+0000","last_became_peered":"2024-03-06T07:31:44.027666+0000","last_unstale":"2024-03-06T07:31:46.028297+0000","last_undegraded":"2024-03-06T07:31:46.028297+0000","last_fullsized":"2024-03-06T07:31:46.028297+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029356+0000","last_change":"2024-03-06T07:31:45.035716+0000","last_active":"2024-03-06T07:31:46.029356+0000","last_peered":"2024-03-06T07:31:46.029356+0000","last_clean":"2024-03-06T07:31:46.029356+0000","last_became_active":"2024-03-06T07:31:45.035437+0000","last_became_peered":"2024-03-06T07:31:45.035437+0000","last_unstale":"2024-03-06T07:31:46.029356+0000","last_undegraded":"2024-03-06T07:31:46.029356+0000","last_fullsized":"2024-03-06T07:31:46.029356+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:48.244230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"144'3","reported_seq":17,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.197151+0000","last_change":"2024-03-06T07:31:44.031865+0000","last_active":"2024-03-06T07:31:47.197151+0000","last_peered":"2024-03-06T07:31:47.197151+0000","last_clean":"2024-03-06T07:31:47.197151+0000","last_became_active":"2024-03-06T07:31:44.030846+0000","last_became_peered":"2024-03-06T07:31:44.030846+0000","last_unstale":"2024-03-06T07:31:47.197151+0000","last_undegraded":"2024-03-06T07:31:47.197151+0000","last_fullsized":"2024-03-06T07:31:47.197151+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.034911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":409,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047049+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:31:46.047049+0000","last_peered":"2024-03-06T07:31:46.047049+0000","last_clean":"2024-03-06T07:31:46.047049+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:31:46.047049+0000","last_undegraded":"2024-03-06T07:31:46.047049+0000","last_fullsized":"2024-03-06T07:31:46.047049+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574962,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41864,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329976,"statfs":{"total":96636764160,"available":96593895424,"internally_reserved":0,"allocated":2416640,"data_stored":1385698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574963,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":1826816,"data_stored":794824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":1818624,"data_stored":794700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574962,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1814528,"data_stored":794678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":1818624,"data_stored":794768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574962,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41288,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330552,"statfs":{"total":96636764160,"available":96594485248,"internally_reserved":0,"allocated":1830912,"data_stored":796360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":1818624,"data_stored":795260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574963,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2424832,"data_stored":1386692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:51.004 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:31:51.003+0000 7f440e023640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:31:53.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:53.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.744+0000 7f0d23829640 1 Processor -- start 2024-03-06T07:31:53.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.744+0000 7f0d23829640 1 -- start start 2024-03-06T07:31:53.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.744+0000 7f0d23829640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d1c0d02e0 0x7f0d1c0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d23829640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0d06c0 0x7f0d1c0b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d23829640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c0b0dd0 con 0x7f0d1c0b4260 2024-03-06T07:31:53.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d23829640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c0b0b80 con 0x7f0d1c0d02e0 2024-03-06T07:31:53.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d23829640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0aeec0 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d21dc6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0d06c0 0x7f0d1c0b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d215c5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d1c0d02e0 0x7f0d1c0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d215c5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d1c0d02e0 0x7f0d1c0cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:46556/0 (socket says 172.21.15.27:46556) 2024-03-06T07:31:53.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d20dc4640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0d1c0b4260 0x7f0d1c0b1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:41940/0 (socket says 172.21.15.27:41940) 2024-03-06T07:31:53.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d215c5640 1 -- 172.21.15.27:0/3088440226 learned_addr learned my addr 172.21.15.27:0/3088440226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:53.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d215c5640 1 -- 172.21.15.27:0/3088440226 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0d06c0 msgr2=0x7f0d1c0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d215c5640 1 --2- 172.21.15.27:0/3088440226 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0d06c0 0x7f0d1c0b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d215c5640 1 -- 172.21.15.27:0/3088440226 >> v1:172.21.15.69:6789/0 conn(0x7f0d1c0b4260 legacy=0x7f0d1c0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d13fff640 1 -- 172.21.15.27:0/3088440226 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634877968 0 0) 0x7f0d1c0b0dd0 con 0x7f0d1c0b4260 2024-03-06T07:31:53.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.745+0000 7f0d215c5640 1 -- 172.21.15.27:0/3088440226 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c0b10b0 con 0x7f0d1c0d02e0 2024-03-06T07:31:53.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.746+0000 7f0d21dc6640 1 --2- 172.21.15.27:0/3088440226 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0d06c0 0x7f0d1c0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:53.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.747+0000 7f0d215c5640 1 --2- 172.21.15.27:0/3088440226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 0x7f0d1c0cfa00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009870 tx=0x7f0d1802ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9807196d943e61e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.747+0000 7f0d13fff640 1 -- 172.21.15.27:0/3088440226 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1802f840 con 0x7f0d1c0d02e0 2024-03-06T07:31:53.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.747+0000 7f0d13fff640 1 -- 172.21.15.27:0/3088440226 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1802f9e0 con 0x7f0d1c0d02e0 2024-03-06T07:31:53.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.747+0000 7f0d13fff640 1 -- 172.21.15.27:0/3088440226 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1802fdb0 con 0x7f0d1c0d02e0 2024-03-06T07:31:53.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.747+0000 7f0d23829640 1 -- 172.21.15.27:0/3088440226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 msgr2=0x7f0d1c0cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.748+0000 7f0d23829640 1 --2- 172.21.15.27:0/3088440226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 0x7f0d1c0cfa00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009870 tx=0x7f0d1802ba20 comp rx=0 tx=0).stop 2024-03-06T07:31:53.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.748+0000 7f0d23829640 1 -- 172.21.15.27:0/3088440226 shutdown_connections 2024-03-06T07:31:53.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.748+0000 7f0d23829640 1 --2- 172.21.15.27:0/3088440226 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0d06c0 0x7f0d1c0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.748+0000 7f0d23829640 1 --2- 172.21.15.27:0/3088440226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 0x7f0d1c0cfa00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.748+0000 7f0d23829640 1 -- 172.21.15.27:0/3088440226 >> 172.21.15.27:0/3088440226 conn(0x7f0d1c0b36d0 msgr2=0x7f0d1c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.748+0000 7f0d23829640 1 -- 172.21.15.27:0/3088440226 shutdown_connections 2024-03-06T07:31:53.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.748+0000 7f0d23829640 1 -- 172.21.15.27:0/3088440226 wait complete. 2024-03-06T07:31:53.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.749+0000 7f0d23829640 1 Processor -- start 2024-03-06T07:31:53.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.749+0000 7f0d23829640 1 -- start start 2024-03-06T07:31:53.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d23829640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0b4260 0x7f0d1c150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d23829640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 0x7f0d1c1548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d23829640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d1c0d06c0 0x7f0d1c1565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d23829640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0b22d0 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d23829640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0b2150 con 0x7f0d1c0d02e0 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d23829640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0af350 con 0x7f0d1c0b4260 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d21dc6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d1c0d06c0 0x7f0d1c1565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d21dc6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d1c0d06c0 0x7f0d1c1565c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58598/0 (socket says 172.21.15.27:58598) 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d21dc6640 1 -- 172.21.15.27:0/3774404343 learned_addr learned my addr 172.21.15.27:0/3774404343 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d20dc4640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 0x7f0d1c1548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d215c5640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0b4260 0x7f0d1c150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d21dc6640 1 -- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0b4260 msgr2=0x7f0d1c150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d21dc6640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0b4260 0x7f0d1c150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d21dc6640 1 -- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 msgr2=0x7f0d1c1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d21dc6640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 0x7f0d1c1548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d21dc6640 1 -- 172.21.15.27:0/3774404343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c1505a0 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d20dc4640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 0x7f0d1c1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d21dc6640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d1c0d06c0 0x7f0d1c1565c0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009e60 tx=0x7f0d1400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d11ffb640 1 -- 172.21.15.27:0/3774404343 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14018030 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.750+0000 7f0d11ffb640 1 -- 172.21.15.27:0/3774404343 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1400cdf0 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.751+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c0c8c70 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.751+0000 7f0d11ffb640 1 -- 172.21.15.27:0/3774404343 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14003bb0 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.751+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c0c9230 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.752+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c068700 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.752+0000 7f0d11ffb640 1 -- 172.21.15.27:0/3774404343 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d1401b070 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.753+0000 7f0d11ffb640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d0006b4d0 0x7f0d0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.753+0000 7f0d11ffb640 1 -- 172.21.15.27:0/3774404343 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d1408abc0 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.753+0000 7f0d215c5640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d0006b4d0 0x7f0d0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.753+0000 7f0d215c5640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d0006b4d0 0x7f0d0006d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f0d18036000 tx=0x7f0d18008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.762+0000 7f0d11ffb640 1 -- 172.21.15.27:0/3774404343 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d14010040 con 0x7f0d1c0d06c0 2024-03-06T07:31:53.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.938+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d1c0cbec0 con 0x7f0d0006b4d0 2024-03-06T07:31:53.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.944+0000 7f0d11ffb640 1 -- 172.21.15.27:0/3774404343 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252476 (secure 0 0 0) 0x7f0d1c0cbec0 con 0x7f0d0006b4d0 2024-03-06T07:31:53.948 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d0006b4d0 msgr2=0x7f0d0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d0006b4d0 0x7f0d0006d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f0d18036000 tx=0x7f0d18008030 comp rx=0 tx=0).stop 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d1c0d06c0 msgr2=0x7f0d1c1565c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d1c0d06c0 0x7f0d1c1565c0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009e60 tx=0x7f0d1400c490 comp rx=0 tx=0).stop 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 shutdown_connections 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d0006b4d0 0x7f0d0006d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d1c0d06c0 0x7f0d1c1565c0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d1c0d02e0 0x7f0d1c1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 --2- 172.21.15.27:0/3774404343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0d1c0b4260 0x7f0d1c150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 >> 172.21.15.27:0/3774404343 conn(0x7f0d1c0b36d0 msgr2=0x7f0d1c151e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 shutdown_connections 2024-03-06T07:31:53.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:53.947+0000 7f0d23829640 1 -- 172.21.15.27:0/3774404343 wait complete. 2024-03-06T07:31:53.959 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-03-06T07:31:52.119877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331436,"kb_used_data":15400,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643284,"statfs":{"total":773094113280,"available":772754722816,"internally_reserved":0,"allocated":15769600,"data_stored":7542980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001655"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044905+0000","last_change":"2024-03-06T07:31:45.046476+0000","last_active":"2024-03-06T07:31:46.044905+0000","last_peered":"2024-03-06T07:31:46.044905+0000","last_clean":"2024-03-06T07:31:46.044905+0000","last_became_active":"2024-03-06T07:31:45.046179+0000","last_became_peered":"2024-03-06T07:31:45.046179+0000","last_unstale":"2024-03-06T07:31:46.044905+0000","last_undegraded":"2024-03-06T07:31:46.044905+0000","last_fullsized":"2024-03-06T07:31:46.044905+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:52.796540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028753+0000","last_change":"2024-03-06T07:31:45.041710+0000","last_active":"2024-03-06T07:31:46.028753+0000","last_peered":"2024-03-06T07:31:46.028753+0000","last_clean":"2024-03-06T07:31:46.028753+0000","last_became_active":"2024-03-06T07:31:45.041413+0000","last_became_peered":"2024-03-06T07:31:45.041413+0000","last_unstale":"2024-03-06T07:31:46.028753+0000","last_undegraded":"2024-03-06T07:31:46.028753+0000","last_fullsized":"2024-03-06T07:31:46.028753+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:20.551707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.197070+0000","last_change":"2024-03-06T07:31:45.048666+0000","last_active":"2024-03-06T07:31:47.197070+0000","last_peered":"2024-03-06T07:31:47.197070+0000","last_clean":"2024-03-06T07:31:47.197070+0000","last_became_active":"2024-03-06T07:31:45.044404+0000","last_became_peered":"2024-03-06T07:31:45.044404+0000","last_unstale":"2024-03-06T07:31:47.197070+0000","last_undegraded":"2024-03-06T07:31:47.197070+0000","last_fullsized":"2024-03-06T07:31:47.197070+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:31.223766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995846+0000","last_change":"2024-03-06T07:31:45.046599+0000","last_active":"2024-03-06T07:31:46.995846+0000","last_peered":"2024-03-06T07:31:46.995846+0000","last_clean":"2024-03-06T07:31:46.995846+0000","last_became_active":"2024-03-06T07:31:45.046251+0000","last_became_peered":"2024-03-06T07:31:45.046251+0000","last_unstale":"2024-03-06T07:31:46.995846+0000","last_undegraded":"2024-03-06T07:31:46.995846+0000","last_fullsized":"2024-03-06T07:31:46.995846+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:34.148588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028806+0000","last_change":"2024-03-06T07:31:45.048513+0000","last_active":"2024-03-06T07:31:46.028806+0000","last_peered":"2024-03-06T07:31:46.028806+0000","last_clean":"2024-03-06T07:31:46.028806+0000","last_became_active":"2024-03-06T07:31:45.048251+0000","last_became_peered":"2024-03-06T07:31:45.048251+0000","last_unstale":"2024-03-06T07:31:46.028806+0000","last_undegraded":"2024-03-06T07:31:46.028806+0000","last_fullsized":"2024-03-06T07:31:46.028806+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:31.902903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044418+0000","last_change":"2024-03-06T07:31:45.049835+0000","last_active":"2024-03-06T07:31:46.044418+0000","last_peered":"2024-03-06T07:31:46.044418+0000","last_clean":"2024-03-06T07:31:46.044418+0000","last_became_active":"2024-03-06T07:31:45.049479+0000","last_became_peered":"2024-03-06T07:31:45.049479+0000","last_unstale":"2024-03-06T07:31:46.044418+0000","last_undegraded":"2024-03-06T07:31:46.044418+0000","last_fullsized":"2024-03-06T07:31:46.044418+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:52.578105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029386+0000","last_change":"2024-03-06T07:31:45.035876+0000","last_active":"2024-03-06T07:31:46.029386+0000","last_peered":"2024-03-06T07:31:46.029386+0000","last_clean":"2024-03-06T07:31:46.029386+0000","last_became_active":"2024-03-06T07:31:45.035604+0000","last_became_peered":"2024-03-06T07:31:45.035604+0000","last_unstale":"2024-03-06T07:31:46.029386+0000","last_undegraded":"2024-03-06T07:31:46.029386+0000","last_fullsized":"2024-03-06T07:31:46.029386+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:42.299308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044198+0000","last_change":"2024-03-06T07:31:45.035300+0000","last_active":"2024-03-06T07:31:46.044198+0000","last_peered":"2024-03-06T07:31:46.044198+0000","last_clean":"2024-03-06T07:31:46.044198+0000","last_became_active":"2024-03-06T07:31:45.034964+0000","last_became_peered":"2024-03-06T07:31:45.034964+0000","last_unstale":"2024-03-06T07:31:46.044198+0000","last_undegraded":"2024-03-06T07:31:46.044198+0000","last_fullsized":"2024-03-06T07:31:46.044198+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:16.375176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028276+0000","last_change":"2024-03-06T07:31:45.039202+0000","last_active":"2024-03-06T07:31:46.028276+0000","last_peered":"2024-03-06T07:31:46.028276+0000","last_clean":"2024-03-06T07:31:46.028276+0000","last_became_active":"2024-03-06T07:31:45.038902+0000","last_became_peered":"2024-03-06T07:31:45.038902+0000","last_unstale":"2024-03-06T07:31:46.028276+0000","last_undegraded":"2024-03-06T07:31:46.028276+0000","last_fullsized":"2024-03-06T07:31:46.028276+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:36.094842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047002+0000","last_change":"2024-03-06T07:31:45.047007+0000","last_active":"2024-03-06T07:31:46.047002+0000","last_peered":"2024-03-06T07:31:46.047002+0000","last_clean":"2024-03-06T07:31:46.047002+0000","last_became_active":"2024-03-06T07:31:45.046533+0000","last_became_peered":"2024-03-06T07:31:45.046533+0000","last_unstale":"2024-03-06T07:31:46.047002+0000","last_undegraded":"2024-03-06T07:31:46.047002+0000","last_fullsized":"2024-03-06T07:31:46.047002+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:46:34.869582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044541+0000","last_change":"2024-03-06T07:31:45.036443+0000","last_active":"2024-03-06T07:31:46.044541+0000","last_peered":"2024-03-06T07:31:46.044541+0000","last_clean":"2024-03-06T07:31:46.044541+0000","last_became_active":"2024-03-06T07:31:45.035986+0000","last_became_peered":"2024-03-06T07:31:45.035986+0000","last_unstale":"2024-03-06T07:31:46.044541+0000","last_undegraded":"2024-03-06T07:31:46.044541+0000","last_fullsized":"2024-03-06T07:31:46.044541+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:27.623139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047015+0000","last_change":"2024-03-06T07:31:45.049645+0000","last_active":"2024-03-06T07:31:46.047015+0000","last_peered":"2024-03-06T07:31:46.047015+0000","last_clean":"2024-03-06T07:31:46.047015+0000","last_became_active":"2024-03-06T07:31:45.048865+0000","last_became_peered":"2024-03-06T07:31:45.048865+0000","last_unstale":"2024-03-06T07:31:46.047015+0000","last_undegraded":"2024-03-06T07:31:46.047015+0000","last_fullsized":"2024-03-06T07:31:46.047015+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:18.474456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028966+0000","last_change":"2024-03-06T07:31:45.048451+0000","last_active":"2024-03-06T07:31:46.028966+0000","last_peered":"2024-03-06T07:31:46.028966+0000","last_clean":"2024-03-06T07:31:46.028966+0000","last_became_active":"2024-03-06T07:31:45.048188+0000","last_became_peered":"2024-03-06T07:31:45.048188+0000","last_unstale":"2024-03-06T07:31:46.028966+0000","last_undegraded":"2024-03-06T07:31:46.028966+0000","last_fullsized":"2024-03-06T07:31:46.028966+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:52.802401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995643+0000","last_change":"2024-03-06T07:31:45.042722+0000","last_active":"2024-03-06T07:31:46.995643+0000","last_peered":"2024-03-06T07:31:46.995643+0000","last_clean":"2024-03-06T07:31:46.995643+0000","last_became_active":"2024-03-06T07:31:45.042456+0000","last_became_peered":"2024-03-06T07:31:45.042456+0000","last_unstale":"2024-03-06T07:31:46.995643+0000","last_undegraded":"2024-03-06T07:31:46.995643+0000","last_fullsized":"2024-03-06T07:31:46.995643+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:58.325034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046758+0000","last_change":"2024-03-06T07:31:45.046120+0000","last_active":"2024-03-06T07:31:46.046758+0000","last_peered":"2024-03-06T07:31:46.046758+0000","last_clean":"2024-03-06T07:31:46.046758+0000","last_became_active":"2024-03-06T07:31:45.045855+0000","last_became_peered":"2024-03-06T07:31:45.045855+0000","last_unstale":"2024-03-06T07:31:46.046758+0000","last_undegraded":"2024-03-06T07:31:46.046758+0000","last_fullsized":"2024-03-06T07:31:46.046758+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:28.333006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438443+0000","last_change":"2024-03-06T07:31:45.039353+0000","last_active":"2024-03-06T07:31:46.438443+0000","last_peered":"2024-03-06T07:31:46.438443+0000","last_clean":"2024-03-06T07:31:46.438443+0000","last_became_active":"2024-03-06T07:31:45.039090+0000","last_became_peered":"2024-03-06T07:31:45.039090+0000","last_unstale":"2024-03-06T07:31:46.438443+0000","last_undegraded":"2024-03-06T07:31:46.438443+0000","last_fullsized":"2024-03-06T07:31:46.438443+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:04.808270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047413+0000","last_change":"2024-03-06T07:31:45.055564+0000","last_active":"2024-03-06T07:31:46.047413+0000","last_peered":"2024-03-06T07:31:46.047413+0000","last_clean":"2024-03-06T07:31:46.047413+0000","last_became_active":"2024-03-06T07:31:45.055141+0000","last_became_peered":"2024-03-06T07:31:45.055141+0000","last_unstale":"2024-03-06T07:31:46.047413+0000","last_undegraded":"2024-03-06T07:31:46.047413+0000","last_fullsized":"2024-03-06T07:31:46.047413+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:35.460837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196909+0000","last_change":"2024-03-06T07:31:45.046112+0000","last_active":"2024-03-06T07:31:47.196909+0000","last_peered":"2024-03-06T07:31:47.196909+0000","last_clean":"2024-03-06T07:31:47.196909+0000","last_became_active":"2024-03-06T07:31:45.043698+0000","last_became_peered":"2024-03-06T07:31:45.043698+0000","last_unstale":"2024-03-06T07:31:47.196909+0000","last_undegraded":"2024-03-06T07:31:47.196909+0000","last_fullsized":"2024-03-06T07:31:47.196909+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:57.381806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044618+0000","last_change":"2024-03-06T07:31:45.038374+0000","last_active":"2024-03-06T07:31:46.044618+0000","last_peered":"2024-03-06T07:31:46.044618+0000","last_clean":"2024-03-06T07:31:46.044618+0000","last_became_active":"2024-03-06T07:31:45.038088+0000","last_became_peered":"2024-03-06T07:31:45.038088+0000","last_unstale":"2024-03-06T07:31:46.044618+0000","last_undegraded":"2024-03-06T07:31:46.044618+0000","last_fullsized":"2024-03-06T07:31:46.044618+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:56.482865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438523+0000","last_change":"2024-03-06T07:31:45.032953+0000","last_active":"2024-03-06T07:31:46.438523+0000","last_peered":"2024-03-06T07:31:46.438523+0000","last_clean":"2024-03-06T07:31:46.438523+0000","last_became_active":"2024-03-06T07:31:45.032688+0000","last_became_peered":"2024-03-06T07:31:45.032688+0000","last_unstale":"2024-03-06T07:31:46.438523+0000","last_undegraded":"2024-03-06T07:31:46.438523+0000","last_fullsized":"2024-03-06T07:31:46.438523+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:19.547257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047108+0000","last_change":"2024-03-06T07:31:45.049212+0000","last_active":"2024-03-06T07:31:46.047108+0000","last_peered":"2024-03-06T07:31:46.047108+0000","last_clean":"2024-03-06T07:31:46.047108+0000","last_became_active":"2024-03-06T07:31:45.047359+0000","last_became_peered":"2024-03-06T07:31:45.047359+0000","last_unstale":"2024-03-06T07:31:46.047108+0000","last_undegraded":"2024-03-06T07:31:46.047108+0000","last_fullsized":"2024-03-06T07:31:46.047108+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:48.244230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995618+0000","last_change":"2024-03-06T07:31:45.035442+0000","last_active":"2024-03-06T07:31:46.995618+0000","last_peered":"2024-03-06T07:31:46.995618+0000","last_clean":"2024-03-06T07:31:46.995618+0000","last_became_active":"2024-03-06T07:31:45.035154+0000","last_became_peered":"2024-03-06T07:31:45.035154+0000","last_unstale":"2024-03-06T07:31:46.995618+0000","last_undegraded":"2024-03-06T07:31:46.995618+0000","last_fullsized":"2024-03-06T07:31:46.995618+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:38.092312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995455+0000","last_change":"2024-03-06T07:31:45.039599+0000","last_active":"2024-03-06T07:31:46.995455+0000","last_peered":"2024-03-06T07:31:46.995455+0000","last_clean":"2024-03-06T07:31:46.995455+0000","last_became_active":"2024-03-06T07:31:45.039297+0000","last_became_peered":"2024-03-06T07:31:45.039297+0000","last_unstale":"2024-03-06T07:31:46.995455+0000","last_undegraded":"2024-03-06T07:31:46.995455+0000","last_fullsized":"2024-03-06T07:31:46.995455+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:14.807136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046685+0000","last_change":"2024-03-06T07:31:45.050761+0000","last_active":"2024-03-06T07:31:46.046685+0000","last_peered":"2024-03-06T07:31:46.046685+0000","last_clean":"2024-03-06T07:31:46.046685+0000","last_became_active":"2024-03-06T07:31:45.047127+0000","last_became_peered":"2024-03-06T07:31:45.047127+0000","last_unstale":"2024-03-06T07:31:46.046685+0000","last_undegraded":"2024-03-06T07:31:46.046685+0000","last_fullsized":"2024-03-06T07:31:46.046685+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438703+0000","last_change":"2024-03-06T07:31:45.040040+0000","last_active":"2024-03-06T07:31:46.438703+0000","last_peered":"2024-03-06T07:31:46.438703+0000","last_clean":"2024-03-06T07:31:46.438703+0000","last_became_active":"2024-03-06T07:31:45.039753+0000","last_became_peered":"2024-03-06T07:31:45.039753+0000","last_unstale":"2024-03-06T07:31:46.438703+0000","last_undegraded":"2024-03-06T07:31:46.438703+0000","last_fullsized":"2024-03-06T07:31:46.438703+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:48.091616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044791+0000","last_change":"2024-03-06T07:31:45.037506+0000","last_active":"2024-03-06T07:31:46.044791+0000","last_peered":"2024-03-06T07:31:46.044791+0000","last_clean":"2024-03-06T07:31:46.044791+0000","last_became_active":"2024-03-06T07:31:45.036896+0000","last_became_peered":"2024-03-06T07:31:45.036896+0000","last_unstale":"2024-03-06T07:31:46.044791+0000","last_undegraded":"2024-03-06T07:31:46.044791+0000","last_fullsized":"2024-03-06T07:31:46.044791+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:00.686030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438729+0000","last_change":"2024-03-06T07:31:45.047399+0000","last_active":"2024-03-06T07:31:46.438729+0000","last_peered":"2024-03-06T07:31:46.438729+0000","last_clean":"2024-03-06T07:31:46.438729+0000","last_became_active":"2024-03-06T07:31:45.047116+0000","last_became_peered":"2024-03-06T07:31:45.047116+0000","last_unstale":"2024-03-06T07:31:46.438729+0000","last_undegraded":"2024-03-06T07:31:46.438729+0000","last_fullsized":"2024-03-06T07:31:46.438729+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:28.329642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029193+0000","last_change":"2024-03-06T07:31:45.048645+0000","last_active":"2024-03-06T07:31:46.029193+0000","last_peered":"2024-03-06T07:31:46.029193+0000","last_clean":"2024-03-06T07:31:46.029193+0000","last_became_active":"2024-03-06T07:31:45.048362+0000","last_became_peered":"2024-03-06T07:31:45.048362+0000","last_unstale":"2024-03-06T07:31:46.029193+0000","last_undegraded":"2024-03-06T07:31:46.029193+0000","last_fullsized":"2024-03-06T07:31:46.029193+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:41.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044368+0000","last_change":"2024-03-06T07:31:45.045636+0000","last_active":"2024-03-06T07:31:46.044368+0000","last_peered":"2024-03-06T07:31:46.044368+0000","last_clean":"2024-03-06T07:31:46.044368+0000","last_became_active":"2024-03-06T07:31:45.045309+0000","last_became_peered":"2024-03-06T07:31:45.045309+0000","last_unstale":"2024-03-06T07:31:46.044368+0000","last_undegraded":"2024-03-06T07:31:46.044368+0000","last_fullsized":"2024-03-06T07:31:46.044368+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:30.930763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995863+0000","last_change":"2024-03-06T07:31:45.041617+0000","last_active":"2024-03-06T07:31:46.995863+0000","last_peered":"2024-03-06T07:31:46.995863+0000","last_clean":"2024-03-06T07:31:46.995863+0000","last_became_active":"2024-03-06T07:31:45.040818+0000","last_became_peered":"2024-03-06T07:31:45.040818+0000","last_unstale":"2024-03-06T07:31:46.995863+0000","last_undegraded":"2024-03-06T07:31:46.995863+0000","last_fullsized":"2024-03-06T07:31:46.995863+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:30.292549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028323+0000","last_change":"2024-03-06T07:31:45.054122+0000","last_active":"2024-03-06T07:31:46.028323+0000","last_peered":"2024-03-06T07:31:46.028323+0000","last_clean":"2024-03-06T07:31:46.028323+0000","last_became_active":"2024-03-06T07:31:45.053807+0000","last_became_peered":"2024-03-06T07:31:45.053807+0000","last_unstale":"2024-03-06T07:31:46.028323+0000","last_undegraded":"2024-03-06T07:31:46.028323+0000","last_fullsized":"2024-03-06T07:31:46.028323+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:00.757771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438516+0000","last_change":"2024-03-06T07:31:45.032208+0000","last_active":"2024-03-06T07:31:46.438516+0000","last_peered":"2024-03-06T07:31:46.438516+0000","last_clean":"2024-03-06T07:31:46.438516+0000","last_became_active":"2024-03-06T07:31:45.031935+0000","last_became_peered":"2024-03-06T07:31:45.031935+0000","last_unstale":"2024-03-06T07:31:46.438516+0000","last_undegraded":"2024-03-06T07:31:46.438516+0000","last_fullsized":"2024-03-06T07:31:46.438516+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:17.849098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047087+0000","last_change":"2024-03-06T07:31:45.046451+0000","last_active":"2024-03-06T07:31:46.047087+0000","last_peered":"2024-03-06T07:31:46.047087+0000","last_clean":"2024-03-06T07:31:46.047087+0000","last_became_active":"2024-03-06T07:31:45.045791+0000","last_became_peered":"2024-03-06T07:31:45.045791+0000","last_unstale":"2024-03-06T07:31:46.047087+0000","last_undegraded":"2024-03-06T07:31:46.047087+0000","last_fullsized":"2024-03-06T07:31:46.047087+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:41.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"144'4","reported_seq":18,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047262+0000","last_change":"2024-03-06T07:31:44.033293+0000","last_active":"2024-03-06T07:31:46.047262+0000","last_peered":"2024-03-06T07:31:46.047262+0000","last_clean":"2024-03-06T07:31:46.047262+0000","last_became_active":"2024-03-06T07:31:44.032831+0000","last_became_peered":"2024-03-06T07:31:44.032831+0000","last_unstale":"2024-03-06T07:31:46.047262+0000","last_undegraded":"2024-03-06T07:31:46.047262+0000","last_fullsized":"2024-03-06T07:31:46.047262+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:43.731816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028425+0000","last_change":"2024-03-06T07:31:45.040911+0000","last_active":"2024-03-06T07:31:46.028425+0000","last_peered":"2024-03-06T07:31:46.028425+0000","last_clean":"2024-03-06T07:31:46.028425+0000","last_became_active":"2024-03-06T07:31:45.040632+0000","last_became_peered":"2024-03-06T07:31:45.040632+0000","last_unstale":"2024-03-06T07:31:46.028425+0000","last_undegraded":"2024-03-06T07:31:46.028425+0000","last_fullsized":"2024-03-06T07:31:46.028425+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:34.247852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028531+0000","last_change":"2024-03-06T07:31:44.027157+0000","last_active":"2024-03-06T07:31:46.028531+0000","last_peered":"2024-03-06T07:31:46.028531+0000","last_clean":"2024-03-06T07:31:46.028531+0000","last_became_active":"2024-03-06T07:31:44.026865+0000","last_became_peered":"2024-03-06T07:31:44.026865+0000","last_unstale":"2024-03-06T07:31:46.028531+0000","last_undegraded":"2024-03-06T07:31:46.028531+0000","last_fullsized":"2024-03-06T07:31:46.028531+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044375+0000","last_change":"2024-03-06T07:31:45.042098+0000","last_active":"2024-03-06T07:31:46.044375+0000","last_peered":"2024-03-06T07:31:46.044375+0000","last_clean":"2024-03-06T07:31:46.044375+0000","last_became_active":"2024-03-06T07:31:45.041801+0000","last_became_peered":"2024-03-06T07:31:45.041801+0000","last_unstale":"2024-03-06T07:31:46.044375+0000","last_undegraded":"2024-03-06T07:31:46.044375+0000","last_fullsized":"2024-03-06T07:31:46.044375+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:48.312427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995745+0000","last_change":"2024-03-06T07:31:44.023697+0000","last_active":"2024-03-06T07:31:46.995745+0000","last_peered":"2024-03-06T07:31:46.995745+0000","last_clean":"2024-03-06T07:31:46.995745+0000","last_became_active":"2024-03-06T07:31:44.023399+0000","last_became_peered":"2024-03-06T07:31:44.023399+0000","last_unstale":"2024-03-06T07:31:46.995745+0000","last_undegraded":"2024-03-06T07:31:46.995745+0000","last_fullsized":"2024-03-06T07:31:46.995745+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:19.201492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046990+0000","last_change":"2024-03-06T07:31:45.046580+0000","last_active":"2024-03-06T07:31:46.046990+0000","last_peered":"2024-03-06T07:31:46.046990+0000","last_clean":"2024-03-06T07:31:46.046990+0000","last_became_active":"2024-03-06T07:31:45.046312+0000","last_became_peered":"2024-03-06T07:31:45.046312+0000","last_unstale":"2024-03-06T07:31:46.046990+0000","last_undegraded":"2024-03-06T07:31:46.046990+0000","last_fullsized":"2024-03-06T07:31:46.046990+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:55.208319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995866+0000","last_change":"2024-03-06T07:31:44.020513+0000","last_active":"2024-03-06T07:31:46.995866+0000","last_peered":"2024-03-06T07:31:46.995866+0000","last_clean":"2024-03-06T07:31:46.995866+0000","last_became_active":"2024-03-06T07:31:44.020196+0000","last_became_peered":"2024-03-06T07:31:44.020196+0000","last_unstale":"2024-03-06T07:31:46.995866+0000","last_undegraded":"2024-03-06T07:31:46.995866+0000","last_fullsized":"2024-03-06T07:31:46.995866+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:47.307708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028263+0000","last_change":"2024-03-06T07:31:45.045575+0000","last_active":"2024-03-06T07:31:46.028263+0000","last_peered":"2024-03-06T07:31:46.028263+0000","last_clean":"2024-03-06T07:31:46.028263+0000","last_became_active":"2024-03-06T07:31:45.045312+0000","last_became_peered":"2024-03-06T07:31:45.045312+0000","last_unstale":"2024-03-06T07:31:46.028263+0000","last_undegraded":"2024-03-06T07:31:46.028263+0000","last_fullsized":"2024-03-06T07:31:46.028263+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:20.191298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438495+0000","last_change":"2024-03-06T07:31:44.033751+0000","last_active":"2024-03-06T07:31:46.438495+0000","last_peered":"2024-03-06T07:31:46.438495+0000","last_clean":"2024-03-06T07:31:46.438495+0000","last_became_active":"2024-03-06T07:31:44.033386+0000","last_became_peered":"2024-03-06T07:31:44.033386+0000","last_unstale":"2024-03-06T07:31:46.438495+0000","last_undegraded":"2024-03-06T07:31:46.438495+0000","last_fullsized":"2024-03-06T07:31:46.438495+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:50.356019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995664+0000","last_change":"2024-03-06T07:31:45.045822+0000","last_active":"2024-03-06T07:31:46.995664+0000","last_peered":"2024-03-06T07:31:46.995664+0000","last_clean":"2024-03-06T07:31:46.995664+0000","last_became_active":"2024-03-06T07:31:45.045552+0000","last_became_peered":"2024-03-06T07:31:45.045552+0000","last_unstale":"2024-03-06T07:31:46.995664+0000","last_undegraded":"2024-03-06T07:31:46.995664+0000","last_fullsized":"2024-03-06T07:31:46.995664+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:55.936260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028694+0000","last_change":"2024-03-06T07:31:44.028730+0000","last_active":"2024-03-06T07:31:46.028694+0000","last_peered":"2024-03-06T07:31:46.028694+0000","last_clean":"2024-03-06T07:31:46.028694+0000","last_became_active":"2024-03-06T07:31:44.028446+0000","last_became_peered":"2024-03-06T07:31:44.028446+0000","last_unstale":"2024-03-06T07:31:46.028694+0000","last_undegraded":"2024-03-06T07:31:46.028694+0000","last_fullsized":"2024-03-06T07:31:46.028694+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:38.934563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028146+0000","last_change":"2024-03-06T07:31:45.033665+0000","last_active":"2024-03-06T07:31:46.028146+0000","last_peered":"2024-03-06T07:31:46.028146+0000","last_clean":"2024-03-06T07:31:46.028146+0000","last_became_active":"2024-03-06T07:31:45.033369+0000","last_became_peered":"2024-03-06T07:31:45.033369+0000","last_unstale":"2024-03-06T07:31:46.028146+0000","last_undegraded":"2024-03-06T07:31:46.028146+0000","last_fullsized":"2024-03-06T07:31:46.028146+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:36.725280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438556+0000","last_change":"2024-03-06T07:31:44.024794+0000","last_active":"2024-03-06T07:31:46.438556+0000","last_peered":"2024-03-06T07:31:46.438556+0000","last_clean":"2024-03-06T07:31:46.438556+0000","last_became_active":"2024-03-06T07:31:44.024499+0000","last_became_peered":"2024-03-06T07:31:44.024499+0000","last_unstale":"2024-03-06T07:31:46.438556+0000","last_undegraded":"2024-03-06T07:31:46.438556+0000","last_fullsized":"2024-03-06T07:31:46.438556+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:01.912662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028691+0000","last_change":"2024-03-06T07:31:45.040904+0000","last_active":"2024-03-06T07:31:46.028691+0000","last_peered":"2024-03-06T07:31:46.028691+0000","last_clean":"2024-03-06T07:31:46.028691+0000","last_became_active":"2024-03-06T07:31:45.040606+0000","last_became_peered":"2024-03-06T07:31:45.040606+0000","last_unstale":"2024-03-06T07:31:46.028691+0000","last_undegraded":"2024-03-06T07:31:46.028691+0000","last_fullsized":"2024-03-06T07:31:46.028691+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:48.091616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995364+0000","last_change":"2024-03-06T07:31:44.022269+0000","last_active":"2024-03-06T07:31:46.995364+0000","last_peered":"2024-03-06T07:31:46.995364+0000","last_clean":"2024-03-06T07:31:46.995364+0000","last_became_active":"2024-03-06T07:31:44.021982+0000","last_became_peered":"2024-03-06T07:31:44.021982+0000","last_unstale":"2024-03-06T07:31:46.995364+0000","last_undegraded":"2024-03-06T07:31:46.995364+0000","last_fullsized":"2024-03-06T07:31:46.995364+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:51.573386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029104+0000","last_change":"2024-03-06T07:31:45.037434+0000","last_active":"2024-03-06T07:31:46.029104+0000","last_peered":"2024-03-06T07:31:46.029104+0000","last_clean":"2024-03-06T07:31:46.029104+0000","last_became_active":"2024-03-06T07:31:45.037148+0000","last_became_peered":"2024-03-06T07:31:45.037148+0000","last_unstale":"2024-03-06T07:31:46.029104+0000","last_undegraded":"2024-03-06T07:31:46.029104+0000","last_fullsized":"2024-03-06T07:31:46.029104+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:48.312427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028537+0000","last_change":"2024-03-06T07:31:44.019467+0000","last_active":"2024-03-06T07:31:46.028537+0000","last_peered":"2024-03-06T07:31:46.028537+0000","last_clean":"2024-03-06T07:31:46.028537+0000","last_became_active":"2024-03-06T07:31:44.019183+0000","last_became_peered":"2024-03-06T07:31:44.019183+0000","last_unstale":"2024-03-06T07:31:46.028537+0000","last_undegraded":"2024-03-06T07:31:46.028537+0000","last_fullsized":"2024-03-06T07:31:46.028537+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:22.325138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047308+0000","last_change":"2024-03-06T07:31:45.052902+0000","last_active":"2024-03-06T07:31:46.047308+0000","last_peered":"2024-03-06T07:31:46.047308+0000","last_clean":"2024-03-06T07:31:46.047308+0000","last_became_active":"2024-03-06T07:31:45.052223+0000","last_became_peered":"2024-03-06T07:31:45.052223+0000","last_unstale":"2024-03-06T07:31:46.047308+0000","last_undegraded":"2024-03-06T07:31:46.047308+0000","last_fullsized":"2024-03-06T07:31:46.047308+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:03.319261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047254+0000","last_change":"2024-03-06T07:31:44.031262+0000","last_active":"2024-03-06T07:31:46.047254+0000","last_peered":"2024-03-06T07:31:46.047254+0000","last_clean":"2024-03-06T07:31:46.047254+0000","last_became_active":"2024-03-06T07:31:44.030401+0000","last_became_peered":"2024-03-06T07:31:44.030401+0000","last_unstale":"2024-03-06T07:31:46.047254+0000","last_undegraded":"2024-03-06T07:31:46.047254+0000","last_fullsized":"2024-03-06T07:31:46.047254+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:44.318964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438725+0000","last_change":"2024-03-06T07:31:45.033444+0000","last_active":"2024-03-06T07:31:46.438725+0000","last_peered":"2024-03-06T07:31:46.438725+0000","last_clean":"2024-03-06T07:31:46.438725+0000","last_became_active":"2024-03-06T07:31:45.033109+0000","last_became_peered":"2024-03-06T07:31:45.033109+0000","last_unstale":"2024-03-06T07:31:46.438725+0000","last_undegraded":"2024-03-06T07:31:46.438725+0000","last_fullsized":"2024-03-06T07:31:46.438725+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:36.113249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"144'6","reported_seq":20,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029449+0000","last_change":"2024-03-06T07:31:44.026781+0000","last_active":"2024-03-06T07:31:46.029449+0000","last_peered":"2024-03-06T07:31:46.029449+0000","last_clean":"2024-03-06T07:31:46.029449+0000","last_became_active":"2024-03-06T07:31:44.026495+0000","last_became_peered":"2024-03-06T07:31:44.026495+0000","last_unstale":"2024-03-06T07:31:46.029449+0000","last_undegraded":"2024-03-06T07:31:46.029449+0000","last_fullsized":"2024-03-06T07:31:46.029449+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:35.090123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196757+0000","last_change":"2024-03-06T07:31:45.043899+0000","last_active":"2024-03-06T07:31:47.196757+0000","last_peered":"2024-03-06T07:31:47.196757+0000","last_clean":"2024-03-06T07:31:47.196757+0000","last_became_active":"2024-03-06T07:31:45.043131+0000","last_became_peered":"2024-03-06T07:31:45.043131+0000","last_unstale":"2024-03-06T07:31:47.196757+0000","last_undegraded":"2024-03-06T07:31:47.196757+0000","last_fullsized":"2024-03-06T07:31:47.196757+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:30.292549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196947+0000","last_change":"2024-03-06T07:31:44.026616+0000","last_active":"2024-03-06T07:31:47.196947+0000","last_peered":"2024-03-06T07:31:47.196947+0000","last_clean":"2024-03-06T07:31:47.196947+0000","last_became_active":"2024-03-06T07:31:44.025510+0000","last_became_peered":"2024-03-06T07:31:44.025510+0000","last_unstale":"2024-03-06T07:31:47.196947+0000","last_undegraded":"2024-03-06T07:31:47.196947+0000","last_fullsized":"2024-03-06T07:31:47.196947+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:13.314073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028194+0000","last_change":"2024-03-06T07:31:45.041996+0000","last_active":"2024-03-06T07:31:46.028194+0000","last_peered":"2024-03-06T07:31:46.028194+0000","last_clean":"2024-03-06T07:31:46.028194+0000","last_became_active":"2024-03-06T07:31:45.041403+0000","last_became_peered":"2024-03-06T07:31:45.041403+0000","last_unstale":"2024-03-06T07:31:46.028194+0000","last_undegraded":"2024-03-06T07:31:46.028194+0000","last_fullsized":"2024-03-06T07:31:46.028194+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:49.751709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196762+0000","last_change":"2024-03-06T07:31:44.022857+0000","last_active":"2024-03-06T07:31:47.196762+0000","last_peered":"2024-03-06T07:31:47.196762+0000","last_clean":"2024-03-06T07:31:47.196762+0000","last_became_active":"2024-03-06T07:31:44.022400+0000","last_became_peered":"2024-03-06T07:31:44.022400+0000","last_unstale":"2024-03-06T07:31:47.196762+0000","last_undegraded":"2024-03-06T07:31:47.196762+0000","last_fullsized":"2024-03-06T07:31:47.196762+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:55.478147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028725+0000","last_change":"2024-03-06T07:31:45.042751+0000","last_active":"2024-03-06T07:31:46.028725+0000","last_peered":"2024-03-06T07:31:46.028725+0000","last_clean":"2024-03-06T07:31:46.028725+0000","last_became_active":"2024-03-06T07:31:45.042477+0000","last_became_peered":"2024-03-06T07:31:45.042477+0000","last_unstale":"2024-03-06T07:31:46.028725+0000","last_undegraded":"2024-03-06T07:31:46.028725+0000","last_fullsized":"2024-03-06T07:31:46.028725+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:36.113249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044190+0000","last_change":"2024-03-06T07:31:44.023105+0000","last_active":"2024-03-06T07:31:46.044190+0000","last_peered":"2024-03-06T07:31:46.044190+0000","last_clean":"2024-03-06T07:31:46.044190+0000","last_became_active":"2024-03-06T07:31:44.022808+0000","last_became_peered":"2024-03-06T07:31:44.022808+0000","last_unstale":"2024-03-06T07:31:46.044190+0000","last_undegraded":"2024-03-06T07:31:46.044190+0000","last_fullsized":"2024-03-06T07:31:46.044190+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196592+0000","last_change":"2024-03-06T07:31:45.047987+0000","last_active":"2024-03-06T07:31:47.196592+0000","last_peered":"2024-03-06T07:31:47.196592+0000","last_clean":"2024-03-06T07:31:47.196592+0000","last_became_active":"2024-03-06T07:31:45.045672+0000","last_became_peered":"2024-03-06T07:31:45.045672+0000","last_unstale":"2024-03-06T07:31:47.196592+0000","last_undegraded":"2024-03-06T07:31:47.196592+0000","last_fullsized":"2024-03-06T07:31:47.196592+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:34.148588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"144'8","reported_seq":22,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438641+0000","last_change":"2024-03-06T07:31:44.024742+0000","last_active":"2024-03-06T07:31:46.438641+0000","last_peered":"2024-03-06T07:31:46.438641+0000","last_clean":"2024-03-06T07:31:46.438641+0000","last_became_active":"2024-03-06T07:31:44.024445+0000","last_became_peered":"2024-03-06T07:31:44.024445+0000","last_unstale":"2024-03-06T07:31:46.438641+0000","last_undegraded":"2024-03-06T07:31:46.438641+0000","last_fullsized":"2024-03-06T07:31:46.438641+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:00.838367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047192+0000","last_change":"2024-03-06T07:31:45.052132+0000","last_active":"2024-03-06T07:31:46.047192+0000","last_peered":"2024-03-06T07:31:46.047192+0000","last_clean":"2024-03-06T07:31:46.047192+0000","last_became_active":"2024-03-06T07:31:45.051795+0000","last_became_peered":"2024-03-06T07:31:45.051795+0000","last_unstale":"2024-03-06T07:31:46.047192+0000","last_undegraded":"2024-03-06T07:31:46.047192+0000","last_fullsized":"2024-03-06T07:31:46.047192+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:42.299308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029041+0000","last_change":"2024-03-06T07:31:44.023389+0000","last_active":"2024-03-06T07:31:46.029041+0000","last_peered":"2024-03-06T07:31:46.029041+0000","last_clean":"2024-03-06T07:31:46.029041+0000","last_became_active":"2024-03-06T07:31:44.023124+0000","last_became_peered":"2024-03-06T07:31:44.023124+0000","last_unstale":"2024-03-06T07:31:46.029041+0000","last_undegraded":"2024-03-06T07:31:46.029041+0000","last_fullsized":"2024-03-06T07:31:46.029041+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:12.410114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438787+0000","last_change":"2024-03-06T07:31:45.044088+0000","last_active":"2024-03-06T07:31:46.438787+0000","last_peered":"2024-03-06T07:31:46.438787+0000","last_clean":"2024-03-06T07:31:46.438787+0000","last_became_active":"2024-03-06T07:31:45.043830+0000","last_became_peered":"2024-03-06T07:31:45.043830+0000","last_unstale":"2024-03-06T07:31:46.438787+0000","last_undegraded":"2024-03-06T07:31:46.438787+0000","last_fullsized":"2024-03-06T07:31:46.438787+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:31.902903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028699+0000","last_change":"2024-03-06T07:31:44.024253+0000","last_active":"2024-03-06T07:31:46.028699+0000","last_peered":"2024-03-06T07:31:46.028699+0000","last_clean":"2024-03-06T07:31:46.028699+0000","last_became_active":"2024-03-06T07:31:44.023842+0000","last_became_peered":"2024-03-06T07:31:44.023842+0000","last_unstale":"2024-03-06T07:31:46.028699+0000","last_undegraded":"2024-03-06T07:31:46.028699+0000","last_fullsized":"2024-03-06T07:31:46.028699+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196985+0000","last_change":"2024-03-06T07:31:45.045694+0000","last_active":"2024-03-06T07:31:47.196985+0000","last_peered":"2024-03-06T07:31:47.196985+0000","last_clean":"2024-03-06T07:31:47.196985+0000","last_became_active":"2024-03-06T07:31:45.044744+0000","last_became_peered":"2024-03-06T07:31:45.044744+0000","last_unstale":"2024-03-06T07:31:47.196985+0000","last_undegraded":"2024-03-06T07:31:47.196985+0000","last_fullsized":"2024-03-06T07:31:47.196985+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:05.832668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"144'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438842+0000","last_change":"2024-03-06T07:31:44.035542+0000","last_active":"2024-03-06T07:31:46.438842+0000","last_peered":"2024-03-06T07:31:46.438842+0000","last_clean":"2024-03-06T07:31:46.438842+0000","last_became_active":"2024-03-06T07:31:44.035318+0000","last_became_peered":"2024-03-06T07:31:44.035318+0000","last_unstale":"2024-03-06T07:31:46.438842+0000","last_undegraded":"2024-03-06T07:31:46.438842+0000","last_fullsized":"2024-03-06T07:31:46.438842+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:03.773449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044394+0000","last_change":"2024-03-06T07:31:45.055140+0000","last_active":"2024-03-06T07:31:46.044394+0000","last_peered":"2024-03-06T07:31:46.044394+0000","last_clean":"2024-03-06T07:31:46.044394+0000","last_became_active":"2024-03-06T07:31:45.054859+0000","last_became_peered":"2024-03-06T07:31:45.054859+0000","last_unstale":"2024-03-06T07:31:46.044394+0000","last_undegraded":"2024-03-06T07:31:46.044394+0000","last_fullsized":"2024-03-06T07:31:46.044394+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:20.206211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028592+0000","last_change":"2024-03-06T07:31:44.028091+0000","last_active":"2024-03-06T07:31:46.028592+0000","last_peered":"2024-03-06T07:31:46.028592+0000","last_clean":"2024-03-06T07:31:46.028592+0000","last_became_active":"2024-03-06T07:31:44.027829+0000","last_became_peered":"2024-03-06T07:31:44.027829+0000","last_unstale":"2024-03-06T07:31:46.028592+0000","last_undegraded":"2024-03-06T07:31:46.028592+0000","last_fullsized":"2024-03-06T07:31:46.028592+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:08.912427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995910+0000","last_change":"2024-03-06T07:31:45.039966+0000","last_active":"2024-03-06T07:31:46.995910+0000","last_peered":"2024-03-06T07:31:46.995910+0000","last_clean":"2024-03-06T07:31:46.995910+0000","last_became_active":"2024-03-06T07:31:45.039079+0000","last_became_peered":"2024-03-06T07:31:45.039079+0000","last_unstale":"2024-03-06T07:31:46.995910+0000","last_undegraded":"2024-03-06T07:31:46.995910+0000","last_fullsized":"2024-03-06T07:31:46.995910+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:57.381806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029266+0000","last_change":"2024-03-06T07:31:44.024320+0000","last_active":"2024-03-06T07:31:46.029266+0000","last_peered":"2024-03-06T07:31:46.029266+0000","last_clean":"2024-03-06T07:31:46.029266+0000","last_became_active":"2024-03-06T07:31:44.024051+0000","last_became_peered":"2024-03-06T07:31:44.024051+0000","last_unstale":"2024-03-06T07:31:46.029266+0000","last_undegraded":"2024-03-06T07:31:46.029266+0000","last_fullsized":"2024-03-06T07:31:46.029266+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:59.753052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029288+0000","last_change":"2024-03-06T07:31:45.035119+0000","last_active":"2024-03-06T07:31:46.029288+0000","last_peered":"2024-03-06T07:31:46.029288+0000","last_clean":"2024-03-06T07:31:46.029288+0000","last_became_active":"2024-03-06T07:31:45.034850+0000","last_became_peered":"2024-03-06T07:31:45.034850+0000","last_unstale":"2024-03-06T07:31:46.029288+0000","last_undegraded":"2024-03-06T07:31:46.029288+0000","last_fullsized":"2024-03-06T07:31:46.029288+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:03.319261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046997+0000","last_change":"2024-03-06T07:31:44.031164+0000","last_active":"2024-03-06T07:31:46.046997+0000","last_peered":"2024-03-06T07:31:46.046997+0000","last_clean":"2024-03-06T07:31:46.046997+0000","last_became_active":"2024-03-06T07:31:44.030649+0000","last_became_peered":"2024-03-06T07:31:44.030649+0000","last_unstale":"2024-03-06T07:31:46.046997+0000","last_undegraded":"2024-03-06T07:31:46.046997+0000","last_fullsized":"2024-03-06T07:31:46.046997+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.995324+0000","last_change":"2024-03-06T07:31:45.043562+0000","last_active":"2024-03-06T07:31:46.995324+0000","last_peered":"2024-03-06T07:31:46.995324+0000","last_clean":"2024-03-06T07:31:46.995324+0000","last_became_active":"2024-03-06T07:31:45.043293+0000","last_became_peered":"2024-03-06T07:31:45.043293+0000","last_unstale":"2024-03-06T07:31:46.995324+0000","last_undegraded":"2024-03-06T07:31:46.995324+0000","last_fullsized":"2024-03-06T07:31:46.995324+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:26.528565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046682+0000","last_change":"2024-03-06T07:31:44.032582+0000","last_active":"2024-03-06T07:31:46.046682+0000","last_peered":"2024-03-06T07:31:46.046682+0000","last_clean":"2024-03-06T07:31:46.046682+0000","last_became_active":"2024-03-06T07:31:44.032075+0000","last_became_peered":"2024-03-06T07:31:44.032075+0000","last_unstale":"2024-03-06T07:31:46.046682+0000","last_undegraded":"2024-03-06T07:31:46.046682+0000","last_fullsized":"2024-03-06T07:31:46.046682+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.950927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029119+0000","last_change":"2024-03-06T07:31:45.035199+0000","last_active":"2024-03-06T07:31:46.029119+0000","last_peered":"2024-03-06T07:31:46.029119+0000","last_clean":"2024-03-06T07:31:46.029119+0000","last_became_active":"2024-03-06T07:31:45.034925+0000","last_became_peered":"2024-03-06T07:31:45.034925+0000","last_unstale":"2024-03-06T07:31:46.029119+0000","last_undegraded":"2024-03-06T07:31:46.029119+0000","last_fullsized":"2024-03-06T07:31:46.029119+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:18.474456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196488+0000","last_change":"2024-03-06T07:31:44.031018+0000","last_active":"2024-03-06T07:31:47.196488+0000","last_peered":"2024-03-06T07:31:47.196488+0000","last_clean":"2024-03-06T07:31:47.196488+0000","last_became_active":"2024-03-06T07:31:44.030695+0000","last_became_peered":"2024-03-06T07:31:44.030695+0000","last_unstale":"2024-03-06T07:31:47.196488+0000","last_undegraded":"2024-03-06T07:31:47.196488+0000","last_fullsized":"2024-03-06T07:31:47.196488+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:51.791822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029090+0000","last_change":"2024-03-06T07:31:45.043749+0000","last_active":"2024-03-06T07:31:46.029090+0000","last_peered":"2024-03-06T07:31:46.029090+0000","last_clean":"2024-03-06T07:31:46.029090+0000","last_became_active":"2024-03-06T07:31:45.043501+0000","last_became_peered":"2024-03-06T07:31:45.043501+0000","last_unstale":"2024-03-06T07:31:46.029090+0000","last_undegraded":"2024-03-06T07:31:46.029090+0000","last_fullsized":"2024-03-06T07:31:46.029090+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:30.930763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"144'2","reported_seq":16,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047400+0000","last_change":"2024-03-06T07:31:44.033647+0000","last_active":"2024-03-06T07:31:46.047400+0000","last_peered":"2024-03-06T07:31:46.047400+0000","last_clean":"2024-03-06T07:31:46.047400+0000","last_became_active":"2024-03-06T07:31:44.033393+0000","last_became_peered":"2024-03-06T07:31:44.033393+0000","last_unstale":"2024-03-06T07:31:46.047400+0000","last_undegraded":"2024-03-06T07:31:46.047400+0000","last_fullsized":"2024-03-06T07:31:46.047400+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:35.425689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.197000+0000","last_change":"2024-03-06T07:31:45.045750+0000","last_active":"2024-03-06T07:31:47.197000+0000","last_peered":"2024-03-06T07:31:47.197000+0000","last_clean":"2024-03-06T07:31:47.197000+0000","last_became_active":"2024-03-06T07:31:45.044153+0000","last_became_peered":"2024-03-06T07:31:45.044153+0000","last_unstale":"2024-03-06T07:31:47.197000+0000","last_undegraded":"2024-03-06T07:31:47.197000+0000","last_fullsized":"2024-03-06T07:31:47.197000+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:07.832011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"145'1","reported_seq":15,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044808+0000","last_change":"2024-03-06T07:31:44.028776+0000","last_active":"2024-03-06T07:31:46.044808+0000","last_peered":"2024-03-06T07:31:46.044808+0000","last_clean":"2024-03-06T07:31:46.044808+0000","last_became_active":"2024-03-06T07:31:44.028504+0000","last_became_peered":"2024-03-06T07:31:44.028504+0000","last_unstale":"2024-03-06T07:31:46.044808+0000","last_undegraded":"2024-03-06T07:31:46.044808+0000","last_fullsized":"2024-03-06T07:31:46.044808+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:17.358709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028585+0000","last_change":"2024-03-06T07:31:45.040051+0000","last_active":"2024-03-06T07:31:46.028585+0000","last_peered":"2024-03-06T07:31:46.028585+0000","last_clean":"2024-03-06T07:31:46.028585+0000","last_became_active":"2024-03-06T07:31:45.039760+0000","last_became_peered":"2024-03-06T07:31:45.039760+0000","last_unstale":"2024-03-06T07:31:46.028585+0000","last_undegraded":"2024-03-06T07:31:46.028585+0000","last_fullsized":"2024-03-06T07:31:46.028585+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044645+0000","last_change":"2024-03-06T07:31:44.027749+0000","last_active":"2024-03-06T07:31:46.044645+0000","last_peered":"2024-03-06T07:31:46.044645+0000","last_clean":"2024-03-06T07:31:46.044645+0000","last_became_active":"2024-03-06T07:31:44.027413+0000","last_became_peered":"2024-03-06T07:31:44.027413+0000","last_unstale":"2024-03-06T07:31:46.044645+0000","last_undegraded":"2024-03-06T07:31:46.044645+0000","last_fullsized":"2024-03-06T07:31:46.044645+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:26.520317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044559+0000","last_change":"2024-03-06T07:31:45.036246+0000","last_active":"2024-03-06T07:31:46.044559+0000","last_peered":"2024-03-06T07:31:46.044559+0000","last_clean":"2024-03-06T07:31:46.044559+0000","last_became_active":"2024-03-06T07:31:45.035684+0000","last_became_peered":"2024-03-06T07:31:45.035684+0000","last_unstale":"2024-03-06T07:31:46.044559+0000","last_undegraded":"2024-03-06T07:31:46.044559+0000","last_fullsized":"2024-03-06T07:31:46.044559+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:33.886298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.044417+0000","last_change":"2024-03-06T07:31:44.027642+0000","last_active":"2024-03-06T07:31:46.044417+0000","last_peered":"2024-03-06T07:31:46.044417+0000","last_clean":"2024-03-06T07:31:46.044417+0000","last_became_active":"2024-03-06T07:31:44.027355+0000","last_became_peered":"2024-03-06T07:31:44.027355+0000","last_unstale":"2024-03-06T07:31:46.044417+0000","last_undegraded":"2024-03-06T07:31:46.044417+0000","last_fullsized":"2024-03-06T07:31:46.044417+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:39.058715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.438511+0000","last_change":"2024-03-06T07:31:45.034848+0000","last_active":"2024-03-06T07:31:46.438511+0000","last_peered":"2024-03-06T07:31:46.438511+0000","last_clean":"2024-03-06T07:31:46.438511+0000","last_became_active":"2024-03-06T07:31:45.034565+0000","last_became_peered":"2024-03-06T07:31:45.034565+0000","last_unstale":"2024-03-06T07:31:46.438511+0000","last_undegraded":"2024-03-06T07:31:46.438511+0000","last_fullsized":"2024-03-06T07:31:46.438511+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:17.690561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029380+0000","last_change":"2024-03-06T07:31:44.025967+0000","last_active":"2024-03-06T07:31:46.029380+0000","last_peered":"2024-03-06T07:31:46.029380+0000","last_clean":"2024-03-06T07:31:46.029380+0000","last_became_active":"2024-03-06T07:31:44.025640+0000","last_became_peered":"2024-03-06T07:31:44.025640+0000","last_unstale":"2024-03-06T07:31:46.029380+0000","last_undegraded":"2024-03-06T07:31:46.029380+0000","last_fullsized":"2024-03-06T07:31:46.029380+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:48.746990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029016+0000","last_change":"2024-03-06T07:31:45.045298+0000","last_active":"2024-03-06T07:31:46.029016+0000","last_peered":"2024-03-06T07:31:46.029016+0000","last_clean":"2024-03-06T07:31:46.029016+0000","last_became_active":"2024-03-06T07:31:45.045033+0000","last_became_peered":"2024-03-06T07:31:45.045033+0000","last_unstale":"2024-03-06T07:31:46.029016+0000","last_undegraded":"2024-03-06T07:31:46.029016+0000","last_fullsized":"2024-03-06T07:31:46.029016+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:16.375176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046828+0000","last_change":"2024-03-06T07:31:44.032074+0000","last_active":"2024-03-06T07:31:46.046828+0000","last_peered":"2024-03-06T07:31:46.046828+0000","last_clean":"2024-03-06T07:31:46.046828+0000","last_became_active":"2024-03-06T07:31:44.031715+0000","last_became_peered":"2024-03-06T07:31:44.031715+0000","last_unstale":"2024-03-06T07:31:46.046828+0000","last_undegraded":"2024-03-06T07:31:46.046828+0000","last_fullsized":"2024-03-06T07:31:46.046828+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:56.808191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.046731+0000","last_change":"2024-03-06T07:31:45.052042+0000","last_active":"2024-03-06T07:31:46.046731+0000","last_peered":"2024-03-06T07:31:46.046731+0000","last_clean":"2024-03-06T07:31:46.046731+0000","last_became_active":"2024-03-06T07:31:45.051442+0000","last_became_peered":"2024-03-06T07:31:45.051442+0000","last_unstale":"2024-03-06T07:31:46.046731+0000","last_undegraded":"2024-03-06T07:31:46.046731+0000","last_fullsized":"2024-03-06T07:31:46.046731+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:20.551707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.196627+0000","last_change":"2024-03-06T07:31:44.025352+0000","last_active":"2024-03-06T07:31:47.196627+0000","last_peered":"2024-03-06T07:31:47.196627+0000","last_clean":"2024-03-06T07:31:47.196627+0000","last_became_active":"2024-03-06T07:31:44.025043+0000","last_became_peered":"2024-03-06T07:31:44.025043+0000","last_unstale":"2024-03-06T07:31:47.196627+0000","last_undegraded":"2024-03-06T07:31:47.196627+0000","last_fullsized":"2024-03-06T07:31:47.196627+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:59.681311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028726+0000","last_change":"2024-03-06T07:31:45.035827+0000","last_active":"2024-03-06T07:31:46.028726+0000","last_peered":"2024-03-06T07:31:46.028726+0000","last_clean":"2024-03-06T07:31:46.028726+0000","last_became_active":"2024-03-06T07:31:45.035484+0000","last_became_peered":"2024-03-06T07:31:45.035484+0000","last_unstale":"2024-03-06T07:31:46.028726+0000","last_undegraded":"2024-03-06T07:31:46.028726+0000","last_fullsized":"2024-03-06T07:31:46.028726+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:28.329642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.028297+0000","last_change":"2024-03-06T07:31:44.027942+0000","last_active":"2024-03-06T07:31:46.028297+0000","last_peered":"2024-03-06T07:31:46.028297+0000","last_clean":"2024-03-06T07:31:46.028297+0000","last_became_active":"2024-03-06T07:31:44.027666+0000","last_became_peered":"2024-03-06T07:31:44.027666+0000","last_unstale":"2024-03-06T07:31:46.028297+0000","last_undegraded":"2024-03-06T07:31:46.028297+0000","last_fullsized":"2024-03-06T07:31:46.028297+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.120420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.029356+0000","last_change":"2024-03-06T07:31:45.035716+0000","last_active":"2024-03-06T07:31:46.029356+0000","last_peered":"2024-03-06T07:31:46.029356+0000","last_clean":"2024-03-06T07:31:46.029356+0000","last_became_active":"2024-03-06T07:31:45.035437+0000","last_became_peered":"2024-03-06T07:31:45.035437+0000","last_unstale":"2024-03-06T07:31:46.029356+0000","last_undegraded":"2024-03-06T07:31:46.029356+0000","last_fullsized":"2024-03-06T07:31:46.029356+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:44.007138+0000","last_clean_scrub_stamp":"2024-03-06T07:31:44.007138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:48.244230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"144'3","reported_seq":17,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:47.197151+0000","last_change":"2024-03-06T07:31:44.031865+0000","last_active":"2024-03-06T07:31:47.197151+0000","last_peered":"2024-03-06T07:31:47.197151+0000","last_clean":"2024-03-06T07:31:47.197151+0000","last_became_active":"2024-03-06T07:31:44.030846+0000","last_became_peered":"2024-03-06T07:31:44.030846+0000","last_unstale":"2024-03-06T07:31:47.197151+0000","last_undegraded":"2024-03-06T07:31:47.197151+0000","last_fullsized":"2024-03-06T07:31:47.197151+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:43.002419+0000","last_clean_scrub_stamp":"2024-03-06T07:31:43.002419+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.034911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":409,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-06T07:31:46.047049+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:31:46.047049+0000","last_peered":"2024-03-06T07:31:46.047049+0000","last_clean":"2024-03-06T07:31:46.047049+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:31:46.047049+0000","last_undegraded":"2024-03-06T07:31:46.047049+0000","last_fullsized":"2024-03-06T07:31:46.047049+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574962,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41864,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329976,"statfs":{"total":96636764160,"available":96593895424,"internally_reserved":0,"allocated":2416640,"data_stored":1385698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574963,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":1826816,"data_stored":794824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":1818624,"data_stored":794700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574962,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1814528,"data_stored":794678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":1818624,"data_stored":794768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574962,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41288,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330552,"statfs":{"total":96636764160,"available":96594485248,"internally_reserved":0,"allocated":1830912,"data_stored":796360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":1818624,"data_stored":795260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574963,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2424832,"data_stored":1386692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:31:53.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:31:54.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.112+0000 7f3fb0a5c640 1 Processor -- start 2024-03-06T07:31:54.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.112+0000 7f3fb0a5c640 1 -- start start 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3fb0a5c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fac06def0 0x7f3fac071e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3fb0a5c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fac145910 con 0x7f3fac066d00 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3fb0a5c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fac15c6f0 con 0x7f3fac15c180 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3fb0a5c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fac15c9d0 con 0x7f3fac06def0 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3faad9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fac06def0 0x7f3fac071e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3faad9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fac06def0 0x7f3fac071e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58600/0 (socket says 172.21.15.27:58600) 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3faad9d640 1 -- 172.21.15.27:0/959440619 learned_addr learned my addr 172.21.15.27:0/959440619 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3faa59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3fac066d00 0x7f3fac160180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58174/0 (socket says 172.21.15.27:58174) 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3faad9d640 1 -- 172.21.15.27:0/959440619 >> v1:172.21.15.69:6789/0 conn(0x7f3fac15c180 legacy=0x7f3fac0670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3faad9d640 1 -- 172.21.15.27:0/959440619 >> v1:172.21.15.27:6789/0 conn(0x7f3fac066d00 legacy=0x7f3fac160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3faad9d640 1 -- 172.21.15.27:0/959440619 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac15ccb0 con 0x7f3fac06def0 2024-03-06T07:31:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.113+0000 7f3fa959a640 1 -- 172.21.15.27:0/959440619 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125139408 0 0) 0x7f3fac145910 con 0x7f3fac066d00 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.114+0000 7f3faad9d640 1 --2- 172.21.15.27:0/959440619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 0x7f3fac071e70 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009e50 tx=0x7f3fa402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f9676c3c05d6fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.114+0000 7f3fa959a640 1 -- 172.21.15.27:0/959440619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa402cbb0 con 0x7f3fac06def0 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.114+0000 7f3fa959a640 1 -- 172.21.15.27:0/959440619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa402cd50 con 0x7f3fac06def0 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.114+0000 7f3fa959a640 1 -- 172.21.15.27:0/959440619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4034610 con 0x7f3fac06def0 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.115+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/959440619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 msgr2=0x7f3fac071e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.115+0000 7f3fb0a5c640 1 --2- 172.21.15.27:0/959440619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 0x7f3fac071e70 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009e50 tx=0x7f3fa402c010 comp rx=0 tx=0).stop 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.115+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/959440619 shutdown_connections 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.115+0000 7f3fb0a5c640 1 --2- 172.21.15.27:0/959440619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 0x7f3fac071e70 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.115+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/959440619 >> 172.21.15.27:0/959440619 conn(0x7f3fac0d09c0 msgr2=0x7f3fac0d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.115+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/959440619 shutdown_connections 2024-03-06T07:31:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.115+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/959440619 wait complete. 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.116+0000 7f3fb0a5c640 1 Processor -- start 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.116+0000 7f3fb0a5c640 1 -- start start 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fb0a5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fac066d00 0x7f3fac15b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fb0a5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 0x7f3fac0ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fb0a5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fac15c180 0x7f3fac13f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fb0a5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac06e500 con 0x7f3fac06def0 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fb0a5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac06e380 con 0x7f3fac15c180 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fb0a5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0723b0 con 0x7f3fac066d00 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fa9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 0x7f3fac0ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fa9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 0x7f3fac0ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58612/0 (socket says 172.21.15.27:58612) 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fa9d9b640 1 -- 172.21.15.27:0/3152554555 learned_addr learned my addr 172.21.15.27:0/3152554555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3faa59c640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fac066d00 0x7f3fac15b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3faad9d640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fac15c180 0x7f3fac13f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fa9d9b640 1 -- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fac066d00 msgr2=0x7f3fac15b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fa9d9b640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fac066d00 0x7f3fac15b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fa9d9b640 1 -- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fac15c180 msgr2=0x7f3fac13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fa9d9b640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fac15c180 0x7f3fac13f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.117+0000 7f3fa9d9b640 1 -- 172.21.15.27:0/3152554555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac0af610 con 0x7f3fac06def0 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.118+0000 7f3fa9d9b640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 0x7f3fac0ae770 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000c430 tx=0x7f3fa000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.118+0000 7f3f9affd640 1 -- 172.21.15.27:0/3152554555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0017070 con 0x7f3fac06def0 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.118+0000 7f3f9affd640 1 -- 172.21.15.27:0/3152554555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa00052d0 con 0x7f3fac06def0 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.118+0000 7f3f9affd640 1 -- 172.21.15.27:0/3152554555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0003830 con 0x7f3fac06def0 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.118+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac0af8f0 con 0x7f3fac06def0 2024-03-06T07:31:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.118+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fac0afeb0 con 0x7f3fac06def0 2024-03-06T07:31:54.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.119+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fac0733d0 con 0x7f3fac06def0 2024-03-06T07:31:54.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.120+0000 7f3f9affd640 1 -- 172.21.15.27:0/3152554555 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fa001a070 con 0x7f3fac06def0 2024-03-06T07:31:54.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.120+0000 7f3f9affd640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f8c06b4d0 0x7f3f8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.121+0000 7f3f9affd640 1 -- 172.21.15.27:0/3152554555 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fa00893f0 con 0x7f3fac06def0 2024-03-06T07:31:54.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.121+0000 7f3faa59c640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f8c06b4d0 0x7f3f8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.121+0000 7f3faa59c640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f8c06b4d0 0x7f3f8c06d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c007f70 tx=0x7f3f9c007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.125+0000 7f3f9affd640 1 -- 172.21.15.27:0/3152554555 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fa0057cb0 con 0x7f3fac06def0 2024-03-06T07:31:54.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.377+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3fac0b1390 con 0x7f3fac06def0 2024-03-06T07:31:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.379+0000 7f3f9affd640 1 -- 172.21.15.27:0/3152554555 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3fa005bd50 con 0x7f3fac06def0 2024-03-06T07:31:54.380 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-06T07:31:54.380 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOGehlhOh+DhAAEXz3c00VueLr74H8/jIPhA== 2024-03-06T07:31:54.380 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-03-06T07:31:54.380 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-03-06T07:31:54.381 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-03-06T07:31:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f8c06b4d0 msgr2=0x7f3f8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f8c06b4d0 0x7f3f8c06d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c007f70 tx=0x7f3f9c007f00 comp rx=0 tx=0).stop 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 msgr2=0x7f3fac0ae770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 0x7f3fac0ae770 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000c430 tx=0x7f3fa000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 shutdown_connections 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f8c06b4d0 0x7f3f8c06d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3fac15c180 0x7f3fac13f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fac06def0 0x7f3fac0ae770 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 --2- 172.21.15.27:0/3152554555 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3fac066d00 0x7f3fac15b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 >> 172.21.15.27:0/3152554555 conn(0x7f3fac0d09c0 msgr2=0x7f3fac068e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 shutdown_connections 2024-03-06T07:31:54.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.381+0000 7f3fb0a5c640 1 -- 172.21.15.27:0/3152554555 wait complete. 2024-03-06T07:31:54.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.498+0000 7f7b6e513640 1 Processor -- start 2024-03-06T07:31:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.498+0000 7f7b6e513640 1 -- start start 2024-03-06T07:31:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.499+0000 7f7b6e513640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b68071540 0x7f7b680b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.499+0000 7f7b6e513640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d3400 0x7f7b68071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.499+0000 7f7b6e513640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b68071e60 con 0x7f7b680b2b50 2024-03-06T07:31:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.499+0000 7f7b6e513640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b680d2f70 con 0x7f7b68071540 2024-03-06T07:31:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.499+0000 7f7b6e513640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680d4be0 con 0x7f7b680d3400 2024-03-06T07:31:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.499+0000 7f7b677fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d3400 0x7f7b68071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.499+0000 7f7b67fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7b68071540 0x7f7b680b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.499+0000 7f7b6cab0640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7b680b2b50 0x7f7b680d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:41966/0 (socket says 172.21.15.27:41966) 2024-03-06T07:31:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.499+0000 7f7b6cab0640 1 -- 172.21.15.27:0/4202434136 learned_addr learned my addr 172.21.15.27:0/4202434136 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.500+0000 7f7b67fff640 1 -- 172.21.15.27:0/4202434136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d3400 msgr2=0x7f7b68071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.500+0000 7f7b67fff640 1 --2- 172.21.15.27:0/4202434136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d3400 0x7f7b68071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.500+0000 7f7b67fff640 1 -- 172.21.15.27:0/4202434136 >> v1:172.21.15.69:6789/0 conn(0x7f7b680b2b50 legacy=0x7f7b680d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.500+0000 7f7b67fff640 1 -- 172.21.15.27:0/4202434136 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b680ce610 con 0x7f7b68071540 2024-03-06T07:31:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.500+0000 7f7b66ffd640 1 -- 172.21.15.27:0/4202434136 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957471148 0 0) 0x7f7b68071e60 con 0x7f7b680b2b50 2024-03-06T07:31:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b67fff640 1 --2- 172.21.15.27:0/4202434136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b68071540 0x7f7b680b30a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0097a0 tx=0x7f7b5c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3f4b5139d690e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b66ffd640 1 -- 172.21.15.27:0/4202434136 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c035020 con 0x7f7b68071540 2024-03-06T07:31:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b66ffd640 1 -- 172.21.15.27:0/4202434136 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5c031830 con 0x7f7b68071540 2024-03-06T07:31:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b66ffd640 1 -- 172.21.15.27:0/4202434136 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c031c00 con 0x7f7b68071540 2024-03-06T07:31:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b6e513640 1 -- 172.21.15.27:0/4202434136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b68071540 msgr2=0x7f7b680b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b6e513640 1 --2- 172.21.15.27:0/4202434136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b68071540 0x7f7b680b30a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0097a0 tx=0x7f7b5c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b6e513640 1 -- 172.21.15.27:0/4202434136 shutdown_connections 2024-03-06T07:31:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b6e513640 1 --2- 172.21.15.27:0/4202434136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680d3400 0x7f7b68071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b6e513640 1 --2- 172.21.15.27:0/4202434136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b68071540 0x7f7b680b30a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b6e513640 1 -- 172.21.15.27:0/4202434136 >> 172.21.15.27:0/4202434136 conn(0x7f7b680d3ca0 msgr2=0x7f7b680d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b6e513640 1 -- 172.21.15.27:0/4202434136 shutdown_connections 2024-03-06T07:31:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.501+0000 7f7b6e513640 1 -- 172.21.15.27:0/4202434136 wait complete. 2024-03-06T07:31:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.502+0000 7f7b6e513640 1 Processor -- start 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.503+0000 7f7b6e513640 1 -- start start 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.503+0000 7f7b6e513640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b68071540 0x7f7b680ab770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.503+0000 7f7b6e513640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680b2b50 0x7f7b6815d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.503+0000 7f7b6e513640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d3400 0x7f7b680ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.503+0000 7f7b6e513640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680ce8f0 con 0x7f7b68071540 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.503+0000 7f7b6e513640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680d2c00 con 0x7f7b680d3400 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.503+0000 7f7b6e513640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6815e370 con 0x7f7b680b2b50 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b67fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b68071540 0x7f7b680ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b67fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b68071540 0x7f7b680ab770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58622/0 (socket says 172.21.15.27:58622) 2024-03-06T07:31:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b67fff640 1 -- 172.21.15.27:0/576432332 learned_addr learned my addr 172.21.15.27:0/576432332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b677fe640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680b2b50 0x7f7b6815d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b6cab0640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d3400 0x7f7b680ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b67fff640 1 -- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680b2b50 msgr2=0x7f7b6815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b67fff640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680b2b50 0x7f7b6815d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b67fff640 1 -- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d3400 msgr2=0x7f7b680ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b67fff640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d3400 0x7f7b680ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b67fff640 1 -- 172.21.15.27:0/576432332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b680ace90 con 0x7f7b68071540 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b67fff640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b68071540 0x7f7b680ab770 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0097a0 tx=0x7f7b5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b64ff9640 1 -- 172.21.15.27:0/576432332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c035070 con 0x7f7b68071540 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b680cc5c0 con 0x7f7b68071540 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b64ff9640 1 -- 172.21.15.27:0/576432332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5c004230 con 0x7f7b68071540 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b64ff9640 1 -- 172.21.15.27:0/576432332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c03a630 con 0x7f7b68071540 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.504+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b680ccb80 con 0x7f7b68071540 2024-03-06T07:31:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.505+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6815d5c0 con 0x7f7b68071540 2024-03-06T07:31:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.506+0000 7f7b64ff9640 1 -- 172.21.15.27:0/576432332 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b5c038020 con 0x7f7b68071540 2024-03-06T07:31:54.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.507+0000 7f7b64ff9640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b5406b4d0 0x7f7b5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.507+0000 7f7b64ff9640 1 -- 172.21.15.27:0/576432332 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b5c0abca0 con 0x7f7b68071540 2024-03-06T07:31:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.507+0000 7f7b677fe640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b5406b4d0 0x7f7b5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.508+0000 7f7b677fe640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b5406b4d0 0x7f7b5406d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7b681f6e10 tx=0x7f7b58009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.516+0000 7f7b64ff9640 1 -- 172.21.15.27:0/576432332 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b5c040070 con 0x7f7b68071540 2024-03-06T07:31:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.736+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b68068700 con 0x7f7b68071540 2024-03-06T07:31:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.737+0000 7f7b64ff9640 1 -- 172.21.15.27:0/576432332 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4984 (secure 0 0 0) 0x7f7b5c07a4e0 con 0x7f7b68071540 2024-03-06T07:31:54.738 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:54.739 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1243350995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1243350995},{"type":"v1","addr":"172.21.15.69:6836","nonce":1243350995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":8257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/470342234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":470342234},{"type":"v1","addr":"172.21.15.27:6835","nonce":470342234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":9270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:31:47.404035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:31:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-06T07:31:54.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b5406b4d0 msgr2=0x7f7b5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b5406b4d0 0x7f7b5406d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7b681f6e10 tx=0x7f7b58009190 comp rx=0 tx=0).stop 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b68071540 msgr2=0x7f7b680ab770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b68071540 0x7f7b680ab770 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0097a0 tx=0x7f7b5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 shutdown_connections 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7b5406b4d0 0x7f7b5406d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b680d3400 0x7f7b680ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7b680b2b50 0x7f7b6815d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 --2- 172.21.15.27:0/576432332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b68071540 0x7f7b680ab770 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 >> 172.21.15.27:0/576432332 conn(0x7f7b680d3ca0 msgr2=0x7f7b6815f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 shutdown_connections 2024-03-06T07:31:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:54.739+0000 7f7b6e513640 1 -- 172.21.15.27:0/576432332 wait complete. 2024-03-06T07:31:55.155 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:31:47.404035+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:31:55.155 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:31:55.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:31:55.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.262+0000 7fc8af9c9640 1 Processor -- start 2024-03-06T07:31:55.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.263+0000 7fc8af9c9640 1 -- start start 2024-03-06T07:31:55.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8af9c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a815df00 0x7fc8a80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8af9c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8a80b9a30 0x7fc8a80d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8af9c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a80cea30 con 0x7fc8a80b95a0 2024-03-06T07:31:55.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8af9c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a80d2af0 con 0x7fc8a80b9a30 2024-03-06T07:31:55.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8af9c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a80cf3c0 con 0x7fc8a815df00 2024-03-06T07:31:55.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8adf66640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8a80b9a30 0x7fc8a80d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8adf66640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8a80b9a30 0x7fc8a80d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58628/0 (socket says 172.21.15.27:58628) 2024-03-06T07:31:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8adf66640 1 -- 172.21.15.27:0/3437788331 learned_addr learned my addr 172.21.15.27:0/3437788331 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8ad765640 1 --2- 172.21.15.27:0/3437788331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a815df00 0x7fc8a80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8adf66640 1 -- 172.21.15.27:0/3437788331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a815df00 msgr2=0x7fc8a80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8adf66640 1 --2- 172.21.15.27:0/3437788331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a815df00 0x7fc8a80d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8adf66640 1 -- 172.21.15.27:0/3437788331 >> v1:172.21.15.69:6789/0 conn(0x7fc8a80b95a0 legacy=0x7fc8a80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.264+0000 7fc8adf66640 1 -- 172.21.15.27:0/3437788331 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a80cf0e0 con 0x7fc8a80b9a30 2024-03-06T07:31:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.266+0000 7fc8adf66640 1 --2- 172.21.15.27:0/3437788331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a80b9a30 0x7fc8a80d3220 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009e50 tx=0x7fc8a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17b2a9654691cd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.266+0000 7fc89ffff640 1 -- 172.21.15.27:0/3437788331 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a002ca70 con 0x7fc8a80b9a30 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.266+0000 7fc89ffff640 1 -- 172.21.15.27:0/3437788331 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a002cc10 con 0x7fc8a80b9a30 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.266+0000 7fc89ffff640 1 -- 172.21.15.27:0/3437788331 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00343f0 con 0x7fc8a80b9a30 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.267+0000 7fc8af9c9640 1 -- 172.21.15.27:0/3437788331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a80b9a30 msgr2=0x7fc8a80d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.267+0000 7fc8af9c9640 1 --2- 172.21.15.27:0/3437788331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a80b9a30 0x7fc8a80d3220 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009e50 tx=0x7fc8a002bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.267+0000 7fc8af9c9640 1 -- 172.21.15.27:0/3437788331 shutdown_connections 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.267+0000 7fc8af9c9640 1 --2- 172.21.15.27:0/3437788331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a80b9a30 0x7fc8a80d3220 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.267+0000 7fc8af9c9640 1 --2- 172.21.15.27:0/3437788331 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a815df00 0x7fc8a80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.267+0000 7fc8af9c9640 1 -- 172.21.15.27:0/3437788331 >> 172.21.15.27:0/3437788331 conn(0x7fc8a8071960 msgr2=0x7fc8a8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.267+0000 7fc8af9c9640 1 -- 172.21.15.27:0/3437788331 shutdown_connections 2024-03-06T07:31:55.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.267+0000 7fc8af9c9640 1 -- 172.21.15.27:0/3437788331 wait complete. 2024-03-06T07:31:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.268+0000 7fc8af9c9640 1 Processor -- start 2024-03-06T07:31:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.268+0000 7fc8af9c9640 1 -- start start 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8af9c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a80b95a0 0x7fc8a81f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8af9c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a815df00 0x7fc8a815af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8af9c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8a815b4b0 0x7fc8a80ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8af9c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a80cec70 con 0x7fc8a815df00 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8af9c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a80ce750 con 0x7fc8a815b4b0 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8af9c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a80cedf0 con 0x7fc8a80b95a0 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8acf64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a815df00 0x7fc8a815af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8acf64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a815df00 0x7fc8a815af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58630/0 (socket says 172.21.15.27:58630) 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8acf64640 1 -- 172.21.15.27:0/990331949 learned_addr learned my addr 172.21.15.27:0/990331949 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8adf66640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8a815b4b0 0x7fc8a80ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8ad765640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a80b95a0 0x7fc8a81f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8acf64640 1 -- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a80b95a0 msgr2=0x7fc8a81f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8acf64640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a80b95a0 0x7fc8a81f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8acf64640 1 -- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8a815b4b0 msgr2=0x7fc8a80ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8acf64640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8a815b4b0 0x7fc8a80ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8acf64640 1 -- 172.21.15.27:0/990331949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a81f3a20 con 0x7fc8a815df00 2024-03-06T07:31:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc8acf64640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a815df00 0x7fc8a815af70 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fc89800b3f0 tx=0x7fc89800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.269+0000 7fc89dffb640 1 -- 172.21.15.27:0/990331949 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc898015020 con 0x7fc8a815df00 2024-03-06T07:31:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.270+0000 7fc89dffb640 1 -- 172.21.15.27:0/990331949 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc898011830 con 0x7fc8a815df00 2024-03-06T07:31:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.270+0000 7fc89dffb640 1 -- 172.21.15.27:0/990331949 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc898011c20 con 0x7fc8a815df00 2024-03-06T07:31:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.270+0000 7fc8af9c9640 1 -- 172.21.15.27:0/990331949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a81f3d00 con 0x7fc8a815df00 2024-03-06T07:31:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.270+0000 7fc8af9c9640 1 -- 172.21.15.27:0/990331949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a81f42c0 con 0x7fc8a815df00 2024-03-06T07:31:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.271+0000 7fc89dffb640 1 -- 172.21.15.27:0/990331949 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc898018020 con 0x7fc8a815df00 2024-03-06T07:31:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.272+0000 7fc89dffb640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc88c06b4d0 0x7fc88c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.272+0000 7fc89dffb640 1 -- 172.21.15.27:0/990331949 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc898089cf0 con 0x7fc8a815df00 2024-03-06T07:31:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.272+0000 7fc8ad765640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc88c06b4d0 0x7fc88c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.273+0000 7fc8ad765640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc88c06b4d0 0x7fc88c06d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80cdd50 tx=0x7fc8a4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.273+0000 7fc8b25b1740 1 Processor -- start 2024-03-06T07:31:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.273+0000 7fc8b25b1740 1 -- start start 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e014c870 0x55f1e014cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01508c0 0x55f1e0152360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 0x55f1e017ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1e00b66b0 con 0x55f1e01508c0 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1e005fa60 con 0x55f1e017c600 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1e00b2750 con 0x55f1e014c870 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8acf64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01508c0 0x55f1e0152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8acf64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01508c0 0x55f1e0152360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58632/0 (socket says 172.21.15.27:58632) 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8acf64640 1 -- 172.21.15.27:0/3933511594 learned_addr learned my addr 172.21.15.27:0/3933511594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8ad765640 1 --2- 172.21.15.27:0/3933511594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e014c870 0x55f1e014cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8adf66640 1 --2- 172.21.15.27:0/3933511594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 0x55f1e017ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8acf64640 1 -- 172.21.15.27:0/3933511594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e014c870 msgr2=0x55f1e014cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8acf64640 1 --2- 172.21.15.27:0/3933511594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e014c870 0x55f1e014cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8acf64640 1 -- 172.21.15.27:0/3933511594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 msgr2=0x55f1e017ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8acf64640 1 --2- 172.21.15.27:0/3933511594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 0x55f1e017ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8acf64640 1 -- 172.21.15.27:0/3933511594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1e007df30 con 0x55f1e01508c0 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc8acf64640 1 --2- 172.21.15.27:0/3933511594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01508c0 0x55f1e0152360 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fc898088190 tx=0x7fc89805c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9887254d3f60b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.274+0000 7fc88affd640 1 -- 172.21.15.27:0/3933511594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc898086070 con 0x55f1e01508c0 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc88affd640 1 -- 172.21.15.27:0/3933511594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc898054b90 con 0x55f1e01508c0 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc88affd640 1 -- 172.21.15.27:0/3933511594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89805c370 con 0x55f1e01508c0 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc8b25b1740 1 -- 172.21.15.27:0/3933511594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01508c0 msgr2=0x55f1e0152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/3933511594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01508c0 0x55f1e0152360 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fc898088190 tx=0x7fc89805c740 comp rx=0 tx=0).stop 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc8b25b1740 1 -- 172.21.15.27:0/3933511594 shutdown_connections 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/3933511594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 0x55f1e017ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/3933511594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01508c0 0x55f1e0152360 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/3933511594 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e014c870 0x55f1e014cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc8b25b1740 1 -- 172.21.15.27:0/3933511594 >> 172.21.15.27:0/3933511594 conn(0x55f1e014e8f0 msgr2=0x55f1e01501b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc8b25b1740 1 -- 172.21.15.27:0/3933511594 shutdown_connections 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.275+0000 7fc8b25b1740 1 -- 172.21.15.27:0/3933511594 wait complete. 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.276+0000 7fc8b25b1740 1 -- start start 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.276+0000 7fc8b25b1740 1 Processor -- start 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e017c600 0x55f1e017fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 0x55f1e0171d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e01802d0 0x55f1e0195160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1e0150d50 con 0x55f1e017c600 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1e0072ea0 con 0x55f1e01802d0 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1e0151030 con 0x55f1e0175220 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8b25b1740 10 client.0 _ll_register_callbacks cb 0x55f1e0072870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8ad765640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e017c600 0x55f1e017fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8ad765640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e017c600 0x55f1e017fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58642/0 (socket says 172.21.15.27:58642) 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8ad765640 1 -- 172.21.15.27:0/638717801 learned_addr learned my addr 172.21.15.27:0/638717801 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8acf64640 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 0x55f1e0171d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8adf66640 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e01802d0 0x55f1e0195160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8ad765640 1 -- 172.21.15.27:0/638717801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 msgr2=0x55f1e0171d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8ad765640 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 0x55f1e0171d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8ad765640 1 -- 172.21.15.27:0/638717801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e01802d0 msgr2=0x55f1e0195160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8ad765640 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e01802d0 0x55f1e0195160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8ad765640 1 -- 172.21.15.27:0/638717801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1e0073270 con 0x55f1e017c600 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc8ad765640 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e017c600 0x55f1e017fcd0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400ff70 tx=0x7fc8a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3599dfc93ccd73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc883fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.277+0000 7fc883fff640 1 -- 172.21.15.27:0/638717801 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a4025070 con 0x55f1e017c600 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc883fff640 1 -- 172.21.15.27:0/638717801 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8a40159e0 con 0x55f1e017c600 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8b25b1740 1 -- 172.21.15.27:0/638717801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc898054b90 con 0x55f1e017c600 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc883fff640 1 -- 172.21.15.27:0/638717801 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8a4030050 con 0x55f1e017c600 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8b25b1740 10 client.9435 fetch_fsmap learned FSMap version 112 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8b25b1740 1 -- 172.21.15.27:0/638717801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x55f1dfdcb2d0 con 0x55f1e017c600 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc883fff640 1 -- 172.21.15.27:0/638717801 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc898054b90 con 0x55f1e017c600 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc883fff640 1 -- 172.21.15.27:0/638717801 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 112) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc8a4033020 con 0x55f1e017c600 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8b25b1740 10 client.9435 fetch_fsmap finished waiting for FSMap version 112 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8b25b1740 10 client.9435 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8b25b1740 20 client.9435 populate_metadata read hostname 'smithi027' 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] conn(0x55f1e017e000 0x55f1e01b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8b25b1740 4 client.9435 mds_command: new command op to 9282 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8b25b1740 1 -- 172.21.15.27:0/638717801 --> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1e00c3bf0 con 0x55f1e017e000 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.278+0000 7fc8acf64640 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] conn(0x55f1e017e000 0x55f1e01b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.279+0000 7fc8acf64640 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] conn(0x55f1e017e000 0x55f1e01b6810 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd3854dbc5b88f00 server_cookie=e4279da9b9250ba9 in_seq=0 out_seq=0 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.279+0000 7fc883fff640 10 client.9435 ms_handle_connect on v2:172.21.15.27:6832/2910933327 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.279+0000 7fc883fff640 1 -- 172.21.15.27:0/638717801 <== mds.0 v2:172.21.15.27:6832/2910933327 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f1e00c3bf0 con 0x55f1e017e000 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.279+0000 7fc883fff640 10 client.9435 handle_command_reply: tid=0 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 client.9435 shutdown 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 -- 172.21.15.27:0/638717801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e017c600 msgr2=0x55f1e017fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e017c600 0x55f1e017fcd0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400ff70 tx=0x7fc8a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 -- 172.21.15.27:0/638717801 shutdown_connections 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] conn(0x55f1e017e000 0x55f1e01b6810 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e01802d0 0x55f1e0195160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 0x55f1e0171d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/638717801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e017c600 0x55f1e017fcd0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 -- 172.21.15.27:0/638717801 >> 172.21.15.27:0/638717801 conn(0x55f1e014c870 msgr2=0x55f1e014b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc883fff640 0 client.9435 ms_handle_reset on v2:172.21.15.27:6832/2910933327 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 -- 172.21.15.27:0/638717801 shutdown_connections 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 1 -- 172.21.15.27:0/638717801 wait complete. 2024-03-06T07:31:55.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.280+0000 7fc8b25b1740 20 client.9435 trim_cache size 0 max 16384 2024-03-06T07:31:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.297+0000 7fc8b25b1740 1 Processor -- start 2024-03-06T07:31:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.297+0000 7fc8b25b1740 1 -- start start 2024-03-06T07:31:55.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.297+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e0175220 0x55f1e0179c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.297+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e017c600 0x55f1e014b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.297+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e014ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.297+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a81ff3a0 con 0x55f1e01802d0 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.297+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a81f5730 con 0x55f1e0175220 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.297+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a81f5d30 con 0x55f1e017c600 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8adf66640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e014ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8adf66640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e014ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58656/0 (socket says 172.21.15.27:58656) 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8adf66640 1 -- 172.21.15.27:0/607042481 learned_addr learned my addr 172.21.15.27:0/607042481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8adf66640 1 -- 172.21.15.27:0/607042481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e017c600 msgr2=0x55f1e014b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8adf66640 1 --2- 172.21.15.27:0/607042481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e017c600 0x55f1e014b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8adf66640 1 -- 172.21.15.27:0/607042481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e0175220 msgr2=0x55f1e0179c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8ad765640 1 --2- 172.21.15.27:0/607042481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e0175220 0x55f1e0179c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8adf66640 1 --2- 172.21.15.27:0/607042481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e0175220 0x55f1e0179c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8adf66640 1 -- 172.21.15.27:0/607042481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1dfe956b0 con 0x55f1e01802d0 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc8adf66640 1 --2- 172.21.15.27:0/607042481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e014ba50 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fc8a002bf10 tx=0x7fc8a002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ff753cdcd8d633 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc88a7fc640 1 -- 172.21.15.27:0/607042481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a004e070 con 0x55f1e01802d0 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc88a7fc640 1 -- 172.21.15.27:0/607042481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a002cb70 con 0x55f1e01802d0 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.298+0000 7fc88a7fc640 1 -- 172.21.15.27:0/607042481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0049430 con 0x55f1e01802d0 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.299+0000 7fc8b25b1740 1 -- 172.21.15.27:0/607042481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 msgr2=0x55f1e014ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.299+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/607042481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e014ba50 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fc8a002bf10 tx=0x7fc8a002e000 comp rx=0 tx=0).stop 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.299+0000 7fc8b25b1740 1 -- 172.21.15.27:0/607042481 shutdown_connections 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.299+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/607042481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e014ba50 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.299+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/607042481 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e017c600 0x55f1e014b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.299+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/607042481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e0175220 0x55f1e0179c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.299+0000 7fc8b25b1740 1 -- 172.21.15.27:0/607042481 >> 172.21.15.27:0/607042481 conn(0x55f1e014c870 msgr2=0x55f1e017d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.299+0000 7fc8b25b1740 1 -- 172.21.15.27:0/607042481 shutdown_connections 2024-03-06T07:31:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.299+0000 7fc8b25b1740 1 -- 172.21.15.27:0/607042481 wait complete. 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.300+0000 7fc8b25b1740 1 -- start start 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.300+0000 7fc8b25b1740 1 Processor -- start 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 0x55f1e01fe390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 0x55f1e016fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8b25b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e020b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1e0195c30 con 0x55f1e01802d0 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1e017f1f0 con 0x55f1e017c600 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8b25b1740 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1e01791c0 con 0x55f1e0175220 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8b25b1740 10 client.0 _ll_register_callbacks cb 0x7fc8a400fb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8adf66640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e020b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8adf66640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e020b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58658/0 (socket says 172.21.15.27:58658) 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8adf66640 1 -- 172.21.15.27:0/1474187438 learned_addr learned my addr 172.21.15.27:0/1474187438 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8ad765640 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 0x55f1e01fe390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8adf66640 1 -- 172.21.15.27:0/1474187438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 msgr2=0x55f1e01fe390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8adf66640 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 0x55f1e01fe390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8acf64640 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 0x55f1e016fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8adf66640 1 -- 172.21.15.27:0/1474187438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 msgr2=0x55f1e016fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8adf66640 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 0x55f1e016fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8adf66640 1 -- 172.21.15.27:0/1474187438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1e020afa0 con 0x55f1e01802d0 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8adf66640 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e020b760 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fc8a002c4e0 tx=0x7fc8a0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2bf0f80d02ec2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc883fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc883fff640 1 -- 172.21.15.27:0/1474187438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a004e070 con 0x55f1e01802d0 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc883fff640 1 -- 172.21.15.27:0/1474187438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8a0052450 con 0x55f1e01802d0 2024-03-06T07:31:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.302+0000 7fc8b25b1740 1 -- 172.21.15.27:0/1474187438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc8a002cb70 con 0x55f1e01802d0 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc883fff640 1 -- 172.21.15.27:0/1474187438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8a0052690 con 0x55f1e01802d0 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8b25b1740 10 client.9441 fetch_fsmap learned FSMap version 112 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8b25b1740 1 -- 172.21.15.27:0/1474187438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x55f1e01704a0 con 0x55f1e01802d0 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc883fff640 1 -- 172.21.15.27:0/1474187438 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a002cb70 con 0x55f1e01802d0 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc883fff640 1 -- 172.21.15.27:0/1474187438 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 112) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc8a005d080 con 0x55f1e01802d0 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8b25b1740 10 client.9441 fetch_fsmap finished waiting for FSMap version 112 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8b25b1740 10 client.9441 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8b25b1740 20 client.9441 populate_metadata read hostname 'smithi027' 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] conn(0x55f1e0176f30 0x55f1e01aa070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8b25b1740 4 client.9441 mds_command: new command op to 9282 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8b25b1740 1 -- 172.21.15.27:0/1474187438 --> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f1e0174d90 con 0x55f1e0176f30 2024-03-06T07:31:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8ad765640 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] conn(0x55f1e0176f30 0x55f1e01aa070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc883fff640 10 client.9441 ms_handle_connect on v2:172.21.15.27:6832/2910933327 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.303+0000 7fc8ad765640 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] conn(0x55f1e0176f30 0x55f1e01aa070 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1941e2520406909 server_cookie=880458528a761669 in_seq=0 out_seq=0 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc883fff640 1 -- 172.21.15.27:0/1474187438 <== mds.0 v2:172.21.15.27:6832/2910933327 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55f1e0174d90 con 0x55f1e0176f30 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc883fff640 10 client.9441 handle_command_reply: tid=0 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 client.9441 shutdown 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 -- 172.21.15.27:0/1474187438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 msgr2=0x55f1e020b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e020b760 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fc8a002c4e0 tx=0x7fc8a0004ab0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 -- 172.21.15.27:0/1474187438 shutdown_connections 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.27:6832/2910933327,v1:172.21.15.27:6834/2910933327] conn(0x55f1e0176f30 0x55f1e01aa070 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f1e01802d0 0x55f1e020b760 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f1e017c600 0x55f1e016fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 --2- 172.21.15.27:0/1474187438 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x55f1e0175220 0x55f1e01fe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 -- 172.21.15.27:0/1474187438 >> 172.21.15.27:0/1474187438 conn(0x55f1e014c870 msgr2=0x55f1e0171140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc883fff640 0 client.9441 ms_handle_reset on v2:172.21.15.27:6832/2910933327 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 -- 172.21.15.27:0/1474187438 shutdown_connections 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 1 -- 172.21.15.27:0/1474187438 wait complete. 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.304+0000 7fc8b25b1740 20 client.9441 trim_cache size 0 max 16384 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3", 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 9282, 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-06T07:31:55.311 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-03-06T07:31:55.312 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.268913169999999, 2024-03-06T07:31:55.312 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 112, 2024-03-06T07:31:55.312 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 145, 2024-03-06T07:31:55.312 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 145, 2024-03-06T07:31:55.312 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 15.279012071 2024-03-06T07:31:55.312 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 -- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc88c06b4d0 msgr2=0x7fc88c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc88c06b4d0 0x7fc88c06d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80cdd50 tx=0x7fc8a4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 -- 172.21.15.27:0/990331949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a815df00 msgr2=0x7fc8a815af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a815df00 0x7fc8a815af70 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fc89800b3f0 tx=0x7fc89800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 -- 172.21.15.27:0/990331949 shutdown_connections 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc88c06b4d0 0x7fc88c06d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8a815b4b0 0x7fc8a80ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8a815df00 0x7fc8a815af70 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 --2- 172.21.15.27:0/990331949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc8a80b95a0 0x7fc8a81f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 -- 172.21.15.27:0/990331949 >> 172.21.15.27:0/990331949 conn(0x7fc8a8071960 msgr2=0x7fc8a80d35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 -- 172.21.15.27:0/990331949 shutdown_connections 2024-03-06T07:31:55.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.305+0000 7fc8af9c9640 1 -- 172.21.15.27:0/990331949 wait complete. 2024-03-06T07:31:55.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:31:55.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.446+0000 7f0ab2ac9640 1 Processor -- start 2024-03-06T07:31:55.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.446+0000 7f0ab2ac9640 1 -- start start 2024-03-06T07:31:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.446+0000 7f0ab2ac9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac15df00 0x7f0aac0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.446+0000 7f0ab2ac9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0aac0b95a0 0x7f0aac0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.446+0000 7f0ab2ac9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aac0cea30 con 0x7f0aac0b9a30 2024-03-06T07:31:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.446+0000 7f0ab2ac9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac0d2af0 con 0x7f0aac0b95a0 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.446+0000 7f0ab2ac9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0cf3c0 con 0x7f0aac15df00 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0ab1066640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0aac0b9a30 0x7f0aac0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58180/0 (socket says 172.21.15.27:58180) 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0ab1066640 1 -- 172.21.15.27:0/2644606258 learned_addr learned my addr 172.21.15.27:0/2644606258 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0aabfff640 1 --2- 172.21.15.27:0/2644606258 >> v2:172.21.15.69:3300/0 conn(0x7f0aac0b95a0 0x7f0aac0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0ab0865640 1 --2- 172.21.15.27:0/2644606258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac15df00 0x7f0aac0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0aab7fe640 1 -- 172.21.15.27:0/2644606258 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490722551 0 0) 0x7f0aac0cea30 con 0x7f0aac0b9a30 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0aab7fe640 1 -- 172.21.15.27:0/2644606258 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a98003520 con 0x7f0aac0b9a30 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0aabfff640 1 -- 172.21.15.27:0/2644606258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac15df00 msgr2=0x7f0aac0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0aabfff640 1 --2- 172.21.15.27:0/2644606258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac15df00 0x7f0aac0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0aabfff640 1 -- 172.21.15.27:0/2644606258 >> v1:172.21.15.27:6789/0 conn(0x7f0aac0b9a30 legacy=0x7f0aac0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.447+0000 7f0aabfff640 1 -- 172.21.15.27:0/2644606258 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac0cf0e0 con 0x7f0aac0b95a0 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.448+0000 7f0aabfff640 1 --2- 172.21.15.27:0/2644606258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aac0b95a0 0x7f0aac0b5700 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c009d90 tx=0x7f0a9c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef00f0a5116dc5fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.448+0000 7f0aab7fe640 1 -- 172.21.15.27:0/2644606258 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c039070 con 0x7f0aac0b95a0 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.448+0000 7f0aab7fe640 1 -- 172.21.15.27:0/2644606258 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c02cc60 con 0x7f0aac0b95a0 2024-03-06T07:31:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.448+0000 7f0aab7fe640 1 -- 172.21.15.27:0/2644606258 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c034660 con 0x7f0aac0b95a0 2024-03-06T07:31:55.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.449+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2644606258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aac0b95a0 msgr2=0x7f0aac0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.449+0000 7f0ab2ac9640 1 --2- 172.21.15.27:0/2644606258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aac0b95a0 0x7f0aac0b5700 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c009d90 tx=0x7f0a9c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.449+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2644606258 shutdown_connections 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.449+0000 7f0ab2ac9640 1 --2- 172.21.15.27:0/2644606258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aac0b95a0 0x7f0aac0b5700 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.449+0000 7f0ab2ac9640 1 --2- 172.21.15.27:0/2644606258 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac15df00 0x7f0aac0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.449+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2644606258 >> 172.21.15.27:0/2644606258 conn(0x7f0aac071960 msgr2=0x7f0aac071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.449+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2644606258 shutdown_connections 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.449+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2644606258 wait complete. 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.450+0000 7f0ab2ac9640 1 Processor -- start 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.450+0000 7f0ab2ac9640 1 -- start start 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0ab2ac9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aac0b95a0 0x7f0aac0b5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0ab2ac9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aac15df00 0x7f0aac0b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0ab2ac9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac1f4300 0x7f0aac15a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0ab2ac9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0cec70 con 0x7f0aac15df00 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0ab2ac9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0ce750 con 0x7f0aac0b95a0 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0ab2ac9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0cedf0 con 0x7f0aac1f4300 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0aabfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aac15df00 0x7f0aac0b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0aabfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aac15df00 0x7f0aac0b4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58662/0 (socket says 172.21.15.27:58662) 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0aabfff640 1 -- 172.21.15.27:0/2884999603 learned_addr learned my addr 172.21.15.27:0/2884999603 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0ab1066640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac1f4300 0x7f0aac15a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0ab0865640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aac0b95a0 0x7f0aac0b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0aabfff640 1 -- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac1f4300 msgr2=0x7f0aac15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0aabfff640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac1f4300 0x7f0aac15a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0aabfff640 1 -- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aac0b95a0 msgr2=0x7f0aac0b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0aabfff640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aac0b95a0 0x7f0aac0b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.451+0000 7f0aabfff640 1 -- 172.21.15.27:0/2884999603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac1f47b0 con 0x7f0aac15df00 2024-03-06T07:31:55.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.452+0000 7f0aabfff640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aac15df00 0x7f0aac0b4230 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c009d90 tx=0x7f0a9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.452+0000 7f0aa97fa640 1 -- 172.21.15.27:0/2884999603 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c039050 con 0x7f0aac15df00 2024-03-06T07:31:55.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.452+0000 7f0aa97fa640 1 -- 172.21.15.27:0/2884999603 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c045070 con 0x7f0aac15df00 2024-03-06T07:31:55.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.452+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aac0b48a0 con 0x7f0aac15df00 2024-03-06T07:31:55.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.452+0000 7f0aa97fa640 1 -- 172.21.15.27:0/2884999603 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c009450 con 0x7f0aac15df00 2024-03-06T07:31:55.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.452+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac0b4e60 con 0x7f0aac15df00 2024-03-06T07:31:55.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.453+0000 7f0aa97fa640 1 -- 172.21.15.27:0/2884999603 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a9c03e850 con 0x7f0aac15df00 2024-03-06T07:31:55.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.453+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aac067ff0 con 0x7f0aac15df00 2024-03-06T07:31:55.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.454+0000 7f0aa97fa640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.454+0000 7f0aa97fa640 1 -- 172.21.15.27:0/2884999603 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a9c0ace40 con 0x7f0aac15df00 2024-03-06T07:31:55.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.454+0000 7f0ab0865640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.454+0000 7f0ab0865640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0aac15ebd0 tx=0x7f0a94009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.458+0000 7f0aa97fa640 1 -- 172.21.15.27:0/2884999603 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a9c07b8e0 con 0x7f0aac15df00 2024-03-06T07:31:55.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.677+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0aac0b7290 con 0x7f0aac15df00 2024-03-06T07:31:55.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.677+0000 7f0aa97fa640 1 -- 172.21.15.27:0/2884999603 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v112) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0a9c07f980 con 0x7f0aac15df00 2024-03-06T07:31:55.679 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:31:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8c06b4d0 msgr2=0x7f0a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0aac15ebd0 tx=0x7f0a94009190 comp rx=0 tx=0).stop 2024-03-06T07:31:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aac15df00 msgr2=0x7f0aac0b4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aac15df00 0x7f0aac0b4230 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c009d90 tx=0x7f0a9c004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 shutdown_connections 2024-03-06T07:31:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0a8c06b4d0 0x7f0a8c06d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0aac1f4300 0x7f0aac15a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aac15df00 0x7f0aac0b4230 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 --2- 172.21.15.27:0/2884999603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aac0b95a0 0x7f0aac0b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 >> 172.21.15.27:0/2884999603 conn(0x7f0aac071960 msgr2=0x7f0aac0d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 shutdown_connections 2024-03-06T07:31:55.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:55.679+0000 7f0ab2ac9640 1 -- 172.21.15.27:0/2884999603 wait complete. 2024-03-06T07:31:55.692 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:31:55.692 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:31:55.692 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:31:55.692 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:31:55.692 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:55.692 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:31:55.692 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:31:55.692 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:31:55.692 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:55.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:55.693 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:31:55.708 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:31:55.708 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:31:55.709 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42174sec preferred_lft 42174sec 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:31:55.710 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:55.711 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:55.711 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:55.711 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:31:55.711 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:55.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:55.711 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:31:55.778 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:31:55.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:55.779 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:31:55.834 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:31:55.835 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:31:55.835 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:31:55.835 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:31:55.835 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:31:55.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:55.835 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:31:55.835 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:31:55.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:31:55.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:31:55.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:31:55.836 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:31:55.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:55.941 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:31:55.956 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:31:55.956 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:31:55.957 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:55.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:55.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:31:56.027 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:31:56.027 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:56.027 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:56.082 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:56.082 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:56.082 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:56.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:31:56.214 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:31:56.270 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:31:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:31:56.272 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:31:56.272 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:31:56.272 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:31:56.272 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:31:56.272 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:56.272 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:56.272 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:56.272 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:56.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:56.272 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:31:56.325 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:31:56.325 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:31:56.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:31:56.403 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:31:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:31:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:31:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:31:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:31:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:31:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:31:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:31:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:31:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:31:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:56.462 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:56.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:56.464 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:31:56.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:31:56.461+0000 7f3dcd0f4480 -1 init, newargv = 0x556f6afec0f0 newargc=15 2024-03-06T07:31:56.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[83552]: starting ceph client 2024-03-06T07:31:56.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[83552]: starting fuse 2024-03-06T07:31:56.481 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:31:56.481 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:31:56.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:31:56.481 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:31:56.482 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:31:56.483 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:31:56.607 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.83552.asok 2024-03-06T07:31:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:31:56.618 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.83552.asok 2024-03-06T07:31:56.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:56.618 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.83552.asok status 2024-03-06T07:31:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:31:56.760 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:31:56.760 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:31:56.760 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:31:56.761 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:31:56.761 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:31:56.761 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:31:56.761 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "83552", 2024-03-06T07:31:56.761 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:31:56.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:56.761 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 9456, 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 9456 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4036622824 2024-03-06T07:31:56.762 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4036622824 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.9456 192.168.0.1:0/4036622824", 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/4036622824", 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 112, 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 145, 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:31:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:31:56.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:56.769 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:31:56.769 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:31:56.785 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:56.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:56.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:56.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:31:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.005+0000 7f5cacab5640 1 Processor -- start 2024-03-06T07:31:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.005+0000 7f5cacab5640 1 -- start start 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.006+0000 7f5cacab5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca8071540 0x7f5ca8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.006+0000 7f5cacab5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ca80b2d30 0x7f5ca80ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.006+0000 7f5cacab5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ca80cecb0 0x7f5ca80cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.006+0000 7f5cacab5640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca80d3230 con 0x7f5ca80b2d30 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.006+0000 7f5cacab5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca8071e60 con 0x7f5ca80cecb0 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.006+0000 7f5cacab5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca80d4960 con 0x7f5ca8071540 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca5d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ca80b2d30 0x7f5ca80ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca5d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ca80b2d30 0x7f5ca80ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58672/0 (socket says 172.21.15.27:58672) 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca5d9b640 1 -- 172.21.15.27:0/4194843984 learned_addr learned my addr 172.21.15.27:0/4194843984 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca6d9d640 1 --2- 172.21.15.27:0/4194843984 >> v2:172.21.15.69:3300/0 conn(0x7f5ca80cecb0 0x7f5ca80cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca659c640 1 --2- 172.21.15.27:0/4194843984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca8071540 0x7f5ca8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca5d9b640 1 -- 172.21.15.27:0/4194843984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca8071540 msgr2=0x7f5ca8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca5d9b640 1 --2- 172.21.15.27:0/4194843984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca8071540 0x7f5ca8071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca5d9b640 1 -- 172.21.15.27:0/4194843984 >> v2:172.21.15.69:3300/0 conn(0x7f5ca80cecb0 msgr2=0x7f5ca80cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca5d9b640 1 --2- 172.21.15.27:0/4194843984 >> v2:172.21.15.69:3300/0 conn(0x7f5ca80cecb0 0x7f5ca80cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.007+0000 7f5ca5d9b640 1 -- 172.21.15.27:0/4194843984 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca80d36c0 con 0x7f5ca80b2d30 2024-03-06T07:31:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5ca5d9b640 1 --2- 172.21.15.27:0/4194843984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80b2d30 0x7f5ca80ce770 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8068380 tx=0x7f5c9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64b4b781965f70c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5ca559a640 1 -- 172.21.15.27:0/4194843984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c035020 con 0x7f5ca80b2d30 2024-03-06T07:31:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5ca559a640 1 -- 172.21.15.27:0/4194843984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c031870 con 0x7f5ca80b2d30 2024-03-06T07:31:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5ca559a640 1 -- 172.21.15.27:0/4194843984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c031c40 con 0x7f5ca80b2d30 2024-03-06T07:31:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5cacab5640 1 -- 172.21.15.27:0/4194843984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80b2d30 msgr2=0x7f5ca80ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5cacab5640 1 --2- 172.21.15.27:0/4194843984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80b2d30 0x7f5ca80ce770 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8068380 tx=0x7f5c9c02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5cacab5640 1 -- 172.21.15.27:0/4194843984 shutdown_connections 2024-03-06T07:31:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5cacab5640 1 --2- 172.21.15.27:0/4194843984 >> v2:172.21.15.69:3300/0 conn(0x7f5ca80cecb0 0x7f5ca80cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5cacab5640 1 --2- 172.21.15.27:0/4194843984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80b2d30 0x7f5ca80ce770 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5cacab5640 1 --2- 172.21.15.27:0/4194843984 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca8071540 0x7f5ca8071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.008+0000 7f5cacab5640 1 -- 172.21.15.27:0/4194843984 >> 172.21.15.27:0/4194843984 conn(0x7f5ca80d3d60 msgr2=0x7f5ca80d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.009+0000 7f5cacab5640 1 -- 172.21.15.27:0/4194843984 shutdown_connections 2024-03-06T07:31:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.009+0000 7f5cacab5640 1 -- 172.21.15.27:0/4194843984 wait complete. 2024-03-06T07:31:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.010+0000 7f5cacab5640 1 Processor -- start 2024-03-06T07:31:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.010+0000 7f5cacab5640 1 -- start start 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5cacab5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ca8071540 0x7f5ca81f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5cacab5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca80b2d30 0x7f5ca80caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5cacab5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80cecb0 0x7f5ca80d1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5cacab5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca80d2be0 con 0x7f5ca80cecb0 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5cacab5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca80d2900 con 0x7f5ca8071540 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5cacab5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca80d2ec0 con 0x7f5ca80b2d30 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80cecb0 0x7f5ca80d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80cecb0 0x7f5ca80d1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58686/0 (socket says 172.21.15.27:58686) 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca6d9d640 1 -- 172.21.15.27:0/443373743 learned_addr learned my addr 172.21.15.27:0/443373743 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:57.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca659c640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ca8071540 0x7f5ca81f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca6d9d640 1 -- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca80b2d30 msgr2=0x7f5ca80caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca5d9b640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca80b2d30 0x7f5ca80caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca6d9d640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca80b2d30 0x7f5ca80caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca6d9d640 1 -- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ca8071540 msgr2=0x7f5ca81f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca6d9d640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ca8071540 0x7f5ca81f38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca6d9d640 1 -- 172.21.15.27:0/443373743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca80cb910 con 0x7f5ca80cecb0 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5ca6d9d640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80cecb0 0x7f5ca80d1910 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0007e40 tx=0x7f5ca000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5c96ffd640 1 -- 172.21.15.27:0/443373743 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca0014620 con 0x7f5ca80cecb0 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5c96ffd640 1 -- 172.21.15.27:0/443373743 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca0019070 con 0x7f5ca80cecb0 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca80cb420 con 0x7f5ca80cecb0 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.011+0000 7f5c96ffd640 1 -- 172.21.15.27:0/443373743 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca0014a20 con 0x7f5ca80cecb0 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.012+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca81569a0 con 0x7f5ca80cecb0 2024-03-06T07:31:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.013+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca8072070 con 0x7f5ca80cecb0 2024-03-06T07:31:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.013+0000 7f5c96ffd640 1 -- 172.21.15.27:0/443373743 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ca0014bc0 con 0x7f5ca80cecb0 2024-03-06T07:31:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.014+0000 7f5c96ffd640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c8806b4d0 0x7f5c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.014+0000 7f5c96ffd640 1 -- 172.21.15.27:0/443373743 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ca008ab40 con 0x7f5ca80cecb0 2024-03-06T07:31:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.014+0000 7f5ca659c640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c8806b4d0 0x7f5c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.015+0000 7f5ca659c640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c8806b4d0 0x7f5c8806d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f5c98006fd0 tx=0x7f5c98008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.018+0000 7f5c96ffd640 1 -- 172.21.15.27:0/443373743 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ca00599c0 con 0x7f5ca80cecb0 2024-03-06T07:31:57.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:57.237+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f5ca804f410 con 0x7f5ca80cecb0 2024-03-06T07:31:58.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.192+0000 7f5c96ffd640 1 -- 172.21.15.27:0/443373743 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v113) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5ca00592c0 con 0x7f5ca80cecb0 2024-03-06T07:31:58.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c8806b4d0 msgr2=0x7f5c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c8806b4d0 0x7f5c8806d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f5c98006fd0 tx=0x7f5c98008040 comp rx=0 tx=0).stop 2024-03-06T07:31:58.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80cecb0 msgr2=0x7f5ca80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80cecb0 0x7f5ca80d1910 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0007e40 tx=0x7f5ca000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 shutdown_connections 2024-03-06T07:31:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5c8806b4d0 0x7f5c8806d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ca80cecb0 0x7f5ca80d1910 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ca80b2d30 0x7f5ca80caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 --2- 172.21.15.27:0/443373743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ca8071540 0x7f5ca81f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 >> 172.21.15.27:0/443373743 conn(0x7f5ca80d3d60 msgr2=0x7f5ca815b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 shutdown_connections 2024-03-06T07:31:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.195+0000 7f5cacab5640 1 -- 172.21.15.27:0/443373743 wait complete. 2024-03-06T07:31:58.213 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.318+0000 7fbc76b1e640 1 Processor -- start 2024-03-06T07:31:58.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.318+0000 7fbc76b1e640 1 -- start start 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc76b1e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc7015df00 0x7fbc700d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc76b1e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc700cea30 con 0x7fbc700b95a0 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc76b1e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc700d2af0 con 0x7fbc700b9a30 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc76b1e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc700cf3c0 con 0x7fbc7015df00 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc748ba640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc7015df00 0x7fbc700d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc6ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbc700b95a0 0x7fbc700b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58192/0 (socket says 172.21.15.27:58192) 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc748ba640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc7015df00 0x7fbc700d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58702/0 (socket says 172.21.15.27:58702) 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc6ffff640 1 -- 172.21.15.27:0/2095362062 learned_addr learned my addr 172.21.15.27:0/2095362062 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:31:58.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc748ba640 1 -- 172.21.15.27:0/2095362062 >> v1:172.21.15.69:6789/0 conn(0x7fbc700b9a30 legacy=0x7fbc700d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc748ba640 1 -- 172.21.15.27:0/2095362062 >> v1:172.21.15.27:6789/0 conn(0x7fbc700b95a0 legacy=0x7fbc700b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc748ba640 1 -- 172.21.15.27:0/2095362062 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700cf0e0 con 0x7fbc7015df00 2024-03-06T07:31:58.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.319+0000 7fbc6f7fe640 1 -- 172.21.15.27:0/2095362062 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579205025 0 0) 0x7fbc700cea30 con 0x7fbc700b95a0 2024-03-06T07:31:58.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.320+0000 7fbc748ba640 1 --2- 172.21.15.27:0/2095362062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc7015df00 0x7fbc700d25b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fbc640097a0 tx=0x7fbc6402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103099ab0caf3eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.320+0000 7fbc6f7fe640 1 -- 172.21.15.27:0/2095362062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64034070 con 0x7fbc7015df00 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.320+0000 7fbc6f7fe640 1 -- 172.21.15.27:0/2095362062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc64030610 con 0x7fbc7015df00 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.320+0000 7fbc6f7fe640 1 -- 172.21.15.27:0/2095362062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc640309e0 con 0x7fbc7015df00 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.321+0000 7fbc76b1e640 1 -- 172.21.15.27:0/2095362062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc7015df00 msgr2=0x7fbc700d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.321+0000 7fbc76b1e640 1 --2- 172.21.15.27:0/2095362062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc7015df00 0x7fbc700d25b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fbc640097a0 tx=0x7fbc6402b960 comp rx=0 tx=0).stop 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.321+0000 7fbc76b1e640 1 -- 172.21.15.27:0/2095362062 shutdown_connections 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.321+0000 7fbc76b1e640 1 --2- 172.21.15.27:0/2095362062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc7015df00 0x7fbc700d25b0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.321+0000 7fbc76b1e640 1 -- 172.21.15.27:0/2095362062 >> 172.21.15.27:0/2095362062 conn(0x7fbc70071960 msgr2=0x7fbc70071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.321+0000 7fbc76b1e640 1 -- 172.21.15.27:0/2095362062 shutdown_connections 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.321+0000 7fbc76b1e640 1 -- 172.21.15.27:0/2095362062 wait complete. 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.322+0000 7fbc76b1e640 1 Processor -- start 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.322+0000 7fbc76b1e640 1 -- start start 2024-03-06T07:31:58.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc76b1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc700b95a0 0x7fbc700d3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc76b1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7015df00 0x7fbc701f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc76b1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc701f6770 0x7fbc700cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc76b1e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700cec70 con 0x7fbc700b95a0 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc76b1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700ce750 con 0x7fbc7015df00 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc76b1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700cedf0 con 0x7fbc701f6770 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc748ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc700b95a0 0x7fbc700d3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc748ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc700b95a0 0x7fbc700d3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58718/0 (socket says 172.21.15.27:58718) 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc748ba640 1 -- 172.21.15.27:0/1516001532 learned_addr learned my addr 172.21.15.27:0/1516001532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc6ffff640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7015df00 0x7fbc701f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc750bb640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc701f6770 0x7fbc700cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc748ba640 1 -- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc701f6770 msgr2=0x7fbc700cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc748ba640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc701f6770 0x7fbc700cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc748ba640 1 -- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7015df00 msgr2=0x7fbc701f6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc748ba640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7015df00 0x7fbc701f6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc748ba640 1 -- 172.21.15.27:0/1516001532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc701f5470 con 0x7fbc700b95a0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc750bb640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc701f6770 0x7fbc700cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.323+0000 7fbc6ffff640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7015df00 0x7fbc701f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.324+0000 7fbc748ba640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc700b95a0 0x7fbc700d3e20 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fbc640098c0 tx=0x7fbc640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.324+0000 7fbc6d7fa640 1 -- 172.21.15.27:0/1516001532 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64034040 con 0x7fbc700b95a0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.324+0000 7fbc6d7fa640 1 -- 172.21.15.27:0/1516001532 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc64039a10 con 0x7fbc700b95a0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.324+0000 7fbc6d7fa640 1 -- 172.21.15.27:0/1516001532 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64039de0 con 0x7fbc700b95a0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.324+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc701f5750 con 0x7fbc700b95a0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.324+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc701f5d10 con 0x7fbc700b95a0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.325+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc70067ff0 con 0x7fbc700b95a0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.326+0000 7fbc6d7fa640 1 -- 172.21.15.27:0/1516001532 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc64037070 con 0x7fbc700b95a0 2024-03-06T07:31:58.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.326+0000 7fbc6d7fa640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc5406b4d0 0x7fbc5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.326+0000 7fbc6d7fa640 1 -- 172.21.15.27:0/1516001532 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc640ac310 con 0x7fbc700b95a0 2024-03-06T07:31:58.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.326+0000 7fbc6ffff640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc5406b4d0 0x7fbc5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.327+0000 7fbc6ffff640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc5406b4d0 0x7fbc5406d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fbc700b7800 tx=0x7fbc5c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.330+0000 7fbc6d7fa640 1 -- 172.21.15.27:0/1516001532 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc6407b190 con 0x7fbc700b95a0 2024-03-06T07:31:58.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.549+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc700c7560 con 0x7fbc700b95a0 2024-03-06T07:31:58.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.549+0000 7fbc6d7fa640 1 -- 172.21.15.27:0/1516001532 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5000 (secure 0 0 0) 0x7fbc6407aa90 con 0x7fbc700b95a0 2024-03-06T07:31:58.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:31:58.550 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":114,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1243350995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1243350995},{"type":"v1","addr":"172.21.15.69:6836","nonce":1243350995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":8257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/470342234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":470342234},{"type":"v1","addr":"172.21.15.27:6835","nonce":470342234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:31:58.196130+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:creating","state_seq":1,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:31:58.551 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 114 2024-03-06T07:31:58.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc5406b4d0 msgr2=0x7fbc5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc5406b4d0 0x7fbc5406d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fbc700b7800 tx=0x7fbc5c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc700b95a0 msgr2=0x7fbc700d3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc700b95a0 0x7fbc700d3e20 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fbc640098c0 tx=0x7fbc640046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 shutdown_connections 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbc5406b4d0 0x7fbc5406d990 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbc701f6770 0x7fbc700cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbc7015df00 0x7fbc701f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 --2- 172.21.15.27:0/1516001532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc700b95a0 0x7fbc700d3e20 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 >> 172.21.15.27:0/1516001532 conn(0x7fbc70071960 msgr2=0x7fbc701618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 shutdown_connections 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:31:58.551+0000 7fbc76b1e640 1 -- 172.21.15.27:0/1516001532 wait complete. 2024-03-06T07:31:58.563 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:32:01.005 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-06T07:32:01.003+0000 7f440e023640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:32:03.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.669+0000 7fb9f889a640 1 Processor -- start 2024-03-06T07:32:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.669+0000 7fb9f889a640 1 -- start start 2024-03-06T07:32:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f889a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9f415df00 0x7fb9f40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f889a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb9f40b95a0 0x7fb9f40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f889a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40cea30 con 0x7fb9f40b9a30 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f889a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f40d2af0 con 0x7fb9f415df00 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f889a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f40cf3c0 con 0x7fb9f40b95a0 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9f415df00 0x7fb9f40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9f415df00 0x7fb9f40d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36004/0 (socket says 172.21.15.27:36004) 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f259c640 1 -- 172.21.15.27:0/3367412097 learned_addr learned my addr 172.21.15.27:0/3367412097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f1d9b640 1 --2- 172.21.15.27:0/3367412097 >> v2:172.21.15.69:3300/0 conn(0x7fb9f40b95a0 0x7fb9f40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f259c640 1 -- 172.21.15.27:0/3367412097 >> v2:172.21.15.69:3300/0 conn(0x7fb9f40b95a0 msgr2=0x7fb9f40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f259c640 1 --2- 172.21.15.27:0/3367412097 >> v2:172.21.15.69:3300/0 conn(0x7fb9f40b95a0 0x7fb9f40b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f259c640 1 -- 172.21.15.27:0/3367412097 >> v1:172.21.15.69:6789/0 conn(0x7fb9f40b9a30 legacy=0x7fb9f40d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.670+0000 7fb9f259c640 1 -- 172.21.15.27:0/3367412097 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f40cf0e0 con 0x7fb9f415df00 2024-03-06T07:32:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.671+0000 7fb9f259c640 1 --2- 172.21.15.27:0/3367412097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f415df00 0x7fb9f40d25b0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40d1760 tx=0x7fb9e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7049ec48497df58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.671+0000 7fb9f159a640 1 -- 172.21.15.27:0/3367412097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4034070 con 0x7fb9f415df00 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.671+0000 7fb9f159a640 1 -- 172.21.15.27:0/3367412097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e4030860 con 0x7fb9f415df00 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.671+0000 7fb9f159a640 1 -- 172.21.15.27:0/3367412097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4030c30 con 0x7fb9f415df00 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.672+0000 7fb9f889a640 1 -- 172.21.15.27:0/3367412097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f415df00 msgr2=0x7fb9f40d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.672+0000 7fb9f889a640 1 --2- 172.21.15.27:0/3367412097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f415df00 0x7fb9f40d25b0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40d1760 tx=0x7fb9e402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.672+0000 7fb9f889a640 1 -- 172.21.15.27:0/3367412097 shutdown_connections 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.672+0000 7fb9f889a640 1 --2- 172.21.15.27:0/3367412097 >> v2:172.21.15.69:3300/0 conn(0x7fb9f40b95a0 0x7fb9f40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.672+0000 7fb9f889a640 1 --2- 172.21.15.27:0/3367412097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f415df00 0x7fb9f40d25b0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.672+0000 7fb9f889a640 1 -- 172.21.15.27:0/3367412097 >> 172.21.15.27:0/3367412097 conn(0x7fb9f4071960 msgr2=0x7fb9f4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.672+0000 7fb9f889a640 1 -- 172.21.15.27:0/3367412097 shutdown_connections 2024-03-06T07:32:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.672+0000 7fb9f889a640 1 -- 172.21.15.27:0/3367412097 wait complete. 2024-03-06T07:32:03.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.673+0000 7fb9f889a640 1 Processor -- start 2024-03-06T07:32:03.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.673+0000 7fb9f889a640 1 -- start start 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f889a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f40b95a0 0x7fb9f40cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f889a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9f415df00 0x7fb9f40c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f889a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f40cb760 0x7fb9f40b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f889a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f40cec70 con 0x7fb9f40cb760 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f889a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f40ce750 con 0x7fb9f40b95a0 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f889a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f40cedf0 con 0x7fb9f415df00 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f40cb760 0x7fb9f40b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f40b95a0 0x7fb9f40cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f40cb760 0x7fb9f40b4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36006/0 (socket says 172.21.15.27:36006) 2024-03-06T07:32:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f2d9d640 1 -- 172.21.15.27:0/489412017 learned_addr learned my addr 172.21.15.27:0/489412017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f1d9b640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9f415df00 0x7fb9f40c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f259c640 1 -- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9f415df00 msgr2=0x7fb9f40c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f259c640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9f415df00 0x7fb9f40c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f259c640 1 -- 172.21.15.27:0/489412017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f40cb760 msgr2=0x7fb9f40b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f259c640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f40cb760 0x7fb9f40b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f259c640 1 -- 172.21.15.27:0/489412017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f40c9260 con 0x7fb9f40b95a0 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f2d9d640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f40cb760 0x7fb9f40b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.674+0000 7fb9f1d9b640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9f415df00 0x7fb9f40c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.675+0000 7fb9f259c640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f40b95a0 0x7fb9f40cd6a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4033000 tx=0x7fb9e40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.675+0000 7fb9e2ffd640 1 -- 172.21.15.27:0/489412017 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4034040 con 0x7fb9f40b95a0 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.675+0000 7fb9e2ffd640 1 -- 172.21.15.27:0/489412017 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e4039d90 con 0x7fb9f40b95a0 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.675+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f40ccf30 con 0x7fb9f40b95a0 2024-03-06T07:32:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.675+0000 7fb9e2ffd640 1 -- 172.21.15.27:0/489412017 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4043650 con 0x7fb9f40b95a0 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.675+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f40c7f60 con 0x7fb9f40b95a0 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.676+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f4067ff0 con 0x7fb9f40b95a0 2024-03-06T07:32:03.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.677+0000 7fb9e2ffd640 1 -- 172.21.15.27:0/489412017 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9e4037070 con 0x7fb9f40b95a0 2024-03-06T07:32:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.677+0000 7fb9e2ffd640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb9d406b4d0 0x7fb9d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.677+0000 7fb9e2ffd640 1 -- 172.21.15.27:0/489412017 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9e40ac130 con 0x7fb9f40b95a0 2024-03-06T07:32:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.677+0000 7fb9f1d9b640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb9d406b4d0 0x7fb9d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.678+0000 7fb9f1d9b640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb9d406b4d0 0x7fb9d406d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb9f41610b0 tx=0x7fb9e8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.686+0000 7fb9e2ffd640 1 -- 172.21.15.27:0/489412017 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9f40c9260 con 0x7fb9f40b95a0 2024-03-06T07:32:03.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.904+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9f40d30e0 con 0x7fb9f40b95a0 2024-03-06T07:32:03.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.905+0000 7fb9e2ffd640 1 -- 172.21.15.27:0/489412017 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5003 (secure 0 0 0) 0x7fb9e403e070 con 0x7fb9f40b95a0 2024-03-06T07:32:03.907 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:03.908 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1243350995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1243350995},{"type":"v1","addr":"172.21.15.69:6836","nonce":1243350995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":8257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/470342234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":470342234},{"type":"v1","addr":"172.21.15.27:6835","nonce":470342234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:31:59.200887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9270,9282]},"id":10}]} 2024-03-06T07:32:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 115 2024-03-06T07:32:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb9d406b4d0 msgr2=0x7fb9d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb9d406b4d0 0x7fb9d406d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb9f41610b0 tx=0x7fb9e8009190 comp rx=0 tx=0).stop 2024-03-06T07:32:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f40b95a0 msgr2=0x7fb9f40cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f40b95a0 0x7fb9f40cd6a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4033000 tx=0x7fb9e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 shutdown_connections 2024-03-06T07:32:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb9d406b4d0 0x7fb9d406d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9f40cb760 0x7fb9f40b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb9f415df00 0x7fb9f40c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 --2- 172.21.15.27:0/489412017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9f40b95a0 0x7fb9f40cd6a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 >> 172.21.15.27:0/489412017 conn(0x7fb9f4071960 msgr2=0x7fb9f415bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 shutdown_connections 2024-03-06T07:32:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:03.907+0000 7fb9f889a640 1 -- 172.21.15.27:0/489412017 wait complete. 2024-03-06T07:32:03.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:04.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.047+0000 7fbbdeb5e640 1 Processor -- start 2024-03-06T07:32:04.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.047+0000 7fbbdeb5e640 1 -- start start 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdeb5e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbbd8071540 0x7fbbd80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdeb5e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdeb5e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd80ce610 con 0x7fbbd80d3400 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdeb5e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd80d2f70 con 0x7fbbd8071540 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdeb5e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd80d4be0 con 0x7fbbd80b2b50 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdc8fa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbbd8071540 0x7fbbd80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbd7fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbbd80d3400 0x7fbbd8071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59570/0 (socket says 172.21.15.27:59570) 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbd7fff640 1 -- 172.21.15.27:0/3025290880 learned_addr learned my addr 172.21.15.27:0/3025290880 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:04.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdd0fb640 1 --2- 172.21.15.27:0/3025290880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdc8fa640 1 -- 172.21.15.27:0/3025290880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 msgr2=0x7fbbd80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdc8fa640 1 --2- 172.21.15.27:0/3025290880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdc8fa640 1 -- 172.21.15.27:0/3025290880 >> v1:172.21.15.27:6789/0 conn(0x7fbbd80d3400 legacy=0x7fbbd8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdc8fa640 1 -- 172.21.15.27:0/3025290880 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd80d2920 con 0x7fbbd8071540 2024-03-06T07:32:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbd77fe640 1 -- 172.21.15.27:0/3025290880 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700322379 0 0) 0x7fbbd80ce610 con 0x7fbbd80d3400 2024-03-06T07:32:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.048+0000 7fbbdd0fb640 1 --2- 172.21.15.27:0/3025290880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.049+0000 7fbbdc8fa640 1 --2- 172.21.15.27:0/3025290880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8071540 0x7fbbd80b30a0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0097a0 tx=0x7fbbcc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edfa1f59c85be442 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.049+0000 7fbbd77fe640 1 -- 172.21.15.27:0/3025290880 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc035020 con 0x7fbbd8071540 2024-03-06T07:32:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.049+0000 7fbbd77fe640 1 -- 172.21.15.27:0/3025290880 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbcc031830 con 0x7fbbd8071540 2024-03-06T07:32:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.049+0000 7fbbd77fe640 1 -- 172.21.15.27:0/3025290880 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc031c00 con 0x7fbbd8071540 2024-03-06T07:32:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.050+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/3025290880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8071540 msgr2=0x7fbbd80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.050+0000 7fbbdeb5e640 1 --2- 172.21.15.27:0/3025290880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8071540 0x7fbbd80b30a0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0097a0 tx=0x7fbbcc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.050+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/3025290880 shutdown_connections 2024-03-06T07:32:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.050+0000 7fbbdeb5e640 1 --2- 172.21.15.27:0/3025290880 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.050+0000 7fbbdeb5e640 1 --2- 172.21.15.27:0/3025290880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8071540 0x7fbbd80b30a0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.050+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/3025290880 >> 172.21.15.27:0/3025290880 conn(0x7fbbd80d3ca0 msgr2=0x7fbbd80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.050+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/3025290880 shutdown_connections 2024-03-06T07:32:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.050+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/3025290880 wait complete. 2024-03-06T07:32:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.051+0000 7fbbdeb5e640 1 Processor -- start 2024-03-06T07:32:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.051+0000 7fbbdeb5e640 1 -- start start 2024-03-06T07:32:04.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdeb5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbd8071540 0x7fbbd81e7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdeb5e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdeb5e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd80d3400 0x7fbbd80cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdeb5e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd815e370 con 0x7fbbd80d3400 2024-03-06T07:32:04.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdeb5e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd80ce8f0 con 0x7fbbd8071540 2024-03-06T07:32:04.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdeb5e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd81e7ce0 con 0x7fbbd80b2b50 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdd0fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd80d3400 0x7fbbd80cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdd0fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd80d3400 0x7fbbd80cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36022/0 (socket says 172.21.15.27:36022) 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbd7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdd0fb640 1 -- 172.21.15.27:0/104865364 learned_addr learned my addr 172.21.15.27:0/104865364 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbdc8fa640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbd8071540 0x7fbbd81e7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbd7fff640 1 -- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbd8071540 msgr2=0x7fbbd81e7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbd7fff640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbd8071540 0x7fbbd81e7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbd7fff640 1 -- 172.21.15.27:0/104865364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd80d3400 msgr2=0x7fbbd80cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbd7fff640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd80d3400 0x7fbbd80cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.052+0000 7fbbd7fff640 1 -- 172.21.15.27:0/104865364 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd81e7fc0 con 0x7fbbd80b2b50 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.053+0000 7fbbd7fff640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80cc940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400c410 tx=0x7fbbc400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.053+0000 7fbbd57fa640 1 -- 172.21.15.27:0/104865364 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc40074e0 con 0x7fbbd80b2b50 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.053+0000 7fbbd57fa640 1 -- 172.21.15.27:0/104865364 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc40047e0 con 0x7fbbd80b2b50 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.053+0000 7fbbd57fa640 1 -- 172.21.15.27:0/104865364 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4004bd0 con 0x7fbbd80b2b50 2024-03-06T07:32:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.053+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd80cbe60 con 0x7fbbd80b2b50 2024-03-06T07:32:04.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.053+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd80cc420 con 0x7fbbd80b2b50 2024-03-06T07:32:04.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.054+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd80b94c0 con 0x7fbbd80b2b50 2024-03-06T07:32:04.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.054+0000 7fbbd57fa640 1 -- 172.21.15.27:0/104865364 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbc4011070 con 0x7fbbd80b2b50 2024-03-06T07:32:04.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.055+0000 7fbbd57fa640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.055+0000 7fbbd57fa640 1 -- 172.21.15.27:0/104865364 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbc4089c70 con 0x7fbbd80b2b50 2024-03-06T07:32:04.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.055+0000 7fbbdc8fa640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.055+0000 7fbbdc8fa640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc036000 tx=0x7fbbcc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.059+0000 7fbbd57fa640 1 -- 172.21.15.27:0/104865364 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbd80b94c0 con 0x7fbbd80b2b50 2024-03-06T07:32:04.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.305+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbd8068700 con 0x7fbbd80b2b50 2024-03-06T07:32:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.306+0000 7fbbd57fa640 1 -- 172.21.15.27:0/104865364 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5003 (secure 0 0 0) 0x7fbbc4058560 con 0x7fbbd80b2b50 2024-03-06T07:32:04.308 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:04.308 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":115,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7202,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6836/1243350995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1243350995},{"type":"v1","addr":"172.21.15.69:6836","nonce":1243350995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":8257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/470342234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":470342234},{"type":"v1","addr":"172.21.15.27:6835","nonce":470342234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:31:59.200887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9270,9282]},"id":10}]} 2024-03-06T07:32:04.308 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 115 2024-03-06T07:32:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.307+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 msgr2=0x7fbbbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.307+0000 7fbbdeb5e640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc036000 tx=0x7fbbcc03d040 comp rx=0 tx=0).stop 2024-03-06T07:32:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.307+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 msgr2=0x7fbbd80cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.307+0000 7fbbdeb5e640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80cc940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400c410 tx=0x7fbbc400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.308+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 shutdown_connections 2024-03-06T07:32:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.308+0000 7fbbdeb5e640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.308+0000 7fbbdeb5e640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd80d3400 0x7fbbd80cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.308+0000 7fbbdeb5e640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80b2b50 0x7fbbd80cc940 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.308+0000 7fbbdeb5e640 1 --2- 172.21.15.27:0/104865364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbd8071540 0x7fbbd81e7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.308+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 >> 172.21.15.27:0/104865364 conn(0x7fbbd80d3ca0 msgr2=0x7fbbd8071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.308+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 shutdown_connections 2024-03-06T07:32:04.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.308+0000 7fbbdeb5e640 1 -- 172.21.15.27:0/104865364 wait complete. 2024-03-06T07:32:04.321 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and d 2024-03-06T07:32:04.321 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:04.321 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:04.329 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:04.330 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:32:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.475+0000 7f3c52a88640 1 Processor -- start 2024-03-06T07:32:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.475+0000 7f3c52a88640 1 -- start start 2024-03-06T07:32:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c52a88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c071540 0x7f3c4c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c52a88640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3c4c0d3400 0x7f3c4c071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c52a88640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c4c0b2b50 0x7f3c4c0d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c52a88640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4c0d2f70 con 0x7f3c4c0b2b50 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c52a88640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4c0d4be0 con 0x7f3c4c0d3400 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c52a88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c071e60 con 0x7f3c4c071540 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c51025640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c4c0b2b50 0x7f3c4c0d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c50824640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c071540 0x7f3c4c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c4bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3c4c0d3400 0x7f3c4c071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c51025640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c4c0b2b50 0x7f3c4c0d1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36026/0 (socket says 172.21.15.27:36026) 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c50824640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c071540 0x7f3c4c0b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:34240/0 (socket says 172.21.15.27:34240) 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.476+0000 7f3c51025640 1 -- 172.21.15.27:0/217624722 learned_addr learned my addr 172.21.15.27:0/217624722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.477+0000 7f3c50824640 1 -- 172.21.15.27:0/217624722 >> v2:172.21.15.69:3300/0 conn(0x7f3c4c0d3400 msgr2=0x7f3c4c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.477+0000 7f3c50824640 1 --2- 172.21.15.27:0/217624722 >> v2:172.21.15.69:3300/0 conn(0x7f3c4c0d3400 0x7f3c4c071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.477+0000 7f3c50824640 1 -- 172.21.15.27:0/217624722 >> v2:172.21.15.27:3300/0 conn(0x7f3c4c0b2b50 msgr2=0x7f3c4c0d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.477+0000 7f3c50824640 1 --2- 172.21.15.27:0/217624722 >> v2:172.21.15.27:3300/0 conn(0x7f3c4c0b2b50 0x7f3c4c0d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.477+0000 7f3c50824640 1 -- 172.21.15.27:0/217624722 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c0ce610 con 0x7f3c4c071540 2024-03-06T07:32:04.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.477+0000 7f3c51025640 1 --2- 172.21.15.27:0/217624722 >> v2:172.21.15.27:3300/0 conn(0x7f3c4c0b2b50 0x7f3c4c0d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:04.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.477+0000 7f3c50824640 1 --2- 172.21.15.27:0/217624722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c071540 0x7f3c4c0b30a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c08c810 tx=0x7f3c4002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91815bc5415cf863 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c4b7fe640 1 -- 172.21.15.27:0/217624722 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c400315a0 con 0x7f3c4c071540 2024-03-06T07:32:04.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c4b7fe640 1 -- 172.21.15.27:0/217624722 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c40037070 con 0x7f3c4c071540 2024-03-06T07:32:04.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c4b7fe640 1 -- 172.21.15.27:0/217624722 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40031960 con 0x7f3c4c071540 2024-03-06T07:32:04.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c52a88640 1 -- 172.21.15.27:0/217624722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c071540 msgr2=0x7f3c4c0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c52a88640 1 --2- 172.21.15.27:0/217624722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c071540 0x7f3c4c0b30a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c08c810 tx=0x7f3c4002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c52a88640 1 -- 172.21.15.27:0/217624722 shutdown_connections 2024-03-06T07:32:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c52a88640 1 --2- 172.21.15.27:0/217624722 >> v2:172.21.15.27:3300/0 conn(0x7f3c4c0b2b50 0x7f3c4c0d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c52a88640 1 --2- 172.21.15.27:0/217624722 >> v2:172.21.15.69:3300/0 conn(0x7f3c4c0d3400 0x7f3c4c071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c52a88640 1 --2- 172.21.15.27:0/217624722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c071540 0x7f3c4c0b30a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c52a88640 1 -- 172.21.15.27:0/217624722 >> 172.21.15.27:0/217624722 conn(0x7f3c4c0d3ca0 msgr2=0x7f3c4c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c52a88640 1 -- 172.21.15.27:0/217624722 shutdown_connections 2024-03-06T07:32:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.478+0000 7f3c52a88640 1 -- 172.21.15.27:0/217624722 wait complete. 2024-03-06T07:32:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.479+0000 7f3c52a88640 1 Processor -- start 2024-03-06T07:32:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.479+0000 7f3c52a88640 1 -- start start 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c52a88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4c071540 0x7f3c4c1f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c52a88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c4c0b2b50 0x7f3c4c1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c52a88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c0d3400 0x7f3c4c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c52a88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c0ce8f0 con 0x7f3c4c071540 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c52a88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c0d2c00 con 0x7f3c4c0b2b50 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c52a88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c15e370 con 0x7f3c4c0d3400 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c50824640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4c071540 0x7f3c4c1f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c50824640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4c071540 0x7f3c4c1f46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36040/0 (socket says 172.21.15.27:36040) 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c50824640 1 -- 172.21.15.27:0/3545028646 learned_addr learned my addr 172.21.15.27:0/3545028646 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.480+0000 7f3c4bfff640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c4c0b2b50 0x7f3c4c1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c51025640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c0d3400 0x7f3c4c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c50824640 1 -- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c0d3400 msgr2=0x7f3c4c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c50824640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c0d3400 0x7f3c4c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c50824640 1 -- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c4c0b2b50 msgr2=0x7f3c4c1581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c50824640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c4c0b2b50 0x7f3c4c1581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c50824640 1 -- 172.21.15.27:0/3545028646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c1f4bf0 con 0x7f3c4c071540 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c50824640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4c071540 0x7f3c4c1f46b0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f3c400096f0 tx=0x7f3c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c497fa640 1 -- 172.21.15.27:0/3545028646 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40032630 con 0x7f3c4c071540 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c497fa640 1 -- 172.21.15.27:0/3545028646 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c400327d0 con 0x7f3c4c071540 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4c1590f0 con 0x7f3c4c071540 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c497fa640 1 -- 172.21.15.27:0/3545028646 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40032bc0 con 0x7f3c4c071540 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.481+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4c1596b0 con 0x7f3c4c071540 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.482+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4c068700 con 0x7f3c4c071540 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.483+0000 7f3c497fa640 1 -- 172.21.15.27:0/3545028646 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c4004e430 con 0x7f3c4c071540 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.483+0000 7f3c497fa640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.483+0000 7f3c497fa640 1 -- 172.21.15.27:0/3545028646 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c4002f030 con 0x7f3c4c071540 2024-03-06T07:32:04.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.483+0000 7f3c4bfff640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.484+0000 7f3c4bfff640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c3006b4d0 0x7f3c3006d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c0b6660 tx=0x7f3c38009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.487+0000 7f3c497fa640 1 -- 172.21.15.27:0/3545028646 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c40040070 con 0x7f3c4c071540 2024-03-06T07:32:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:04.739+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f3c4c160d60 con 0x7f3c4c071540 2024-03-06T07:32:05.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.225+0000 7f3c497fa640 1 -- 172.21.15.27:0/3545028646 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 8257 v116) v1 ==== 95+0+0 (secure 0 0 0) 0x7f3c4007a6d0 con 0x7f3c4c071540 2024-03-06T07:32:05.227 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 8257 2024-03-06T07:32:05.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c3006b4d0 msgr2=0x7f3c3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c3006b4d0 0x7f3c3006d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c0b6660 tx=0x7f3c38009190 comp rx=0 tx=0).stop 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4c071540 msgr2=0x7f3c4c1f46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4c071540 0x7f3c4c1f46b0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f3c400096f0 tx=0x7f3c400046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 shutdown_connections 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3c4c0d3400 0x7f3c4c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c4c0b2b50 0x7f3c4c1581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 --2- 172.21.15.27:0/3545028646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c4c071540 0x7f3c4c1f46b0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 >> 172.21.15.27:0/3545028646 conn(0x7f3c4c0d3ca0 msgr2=0x7f3c4c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 shutdown_connections 2024-03-06T07:32:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.228+0000 7f3c52a88640 1 -- 172.21.15.27:0/3545028646 wait complete. 2024-03-06T07:32:05.241 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:32:05.242 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:05.251 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:32:05.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:32:05.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.356+0000 7f604fecf640 1 Processor -- start 2024-03-06T07:32:05.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.356+0000 7f604fecf640 1 -- start start 2024-03-06T07:32:05.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.357+0000 7f604fecf640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f60480b95a0 0x7f60480b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.357+0000 7f604fecf640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60480cea30 con 0x7f604815df00 2024-03-06T07:32:05.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.357+0000 7f604fecf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60480cf3c0 con 0x7f60480b9a10 2024-03-06T07:32:05.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.357+0000 7f604fecf640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60480cf0e0 con 0x7f60480b95a0 2024-03-06T07:32:05.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.357+0000 7f604dc6b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f604815df00 0x7f60480d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59586/0 (socket says 172.21.15.27:59586) 2024-03-06T07:32:05.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.357+0000 7f604dc6b640 1 -- 172.21.15.27:0/1481408549 learned_addr learned my addr 172.21.15.27:0/1481408549 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:05.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.357+0000 7f604d46a640 1 --2- 172.21.15.27:0/1481408549 >> v2:172.21.15.69:3300/0 conn(0x7f60480b95a0 0x7f60480b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.358+0000 7f604cc69640 1 -- 172.21.15.27:0/1481408549 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468961315 0 0) 0x7f60480cea30 con 0x7f604815df00 2024-03-06T07:32:05.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.358+0000 7f604cc69640 1 -- 172.21.15.27:0/1481408549 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6034003520 con 0x7f604815df00 2024-03-06T07:32:05.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.358+0000 7f604d46a640 1 -- 172.21.15.27:0/1481408549 >> v1:172.21.15.69:6789/0 conn(0x7f60480b9a10 legacy=0x7f60480d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.358+0000 7f604d46a640 1 -- 172.21.15.27:0/1481408549 >> v1:172.21.15.27:6789/0 conn(0x7f604815df00 legacy=0x7f60480d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.358+0000 7f604d46a640 1 -- 172.21.15.27:0/1481408549 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60480ce470 con 0x7f60480b95a0 2024-03-06T07:32:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604d46a640 1 --2- 172.21.15.27:0/1481408549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60480b95a0 0x7f60480b5700 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f60480cf590 tx=0x7f603802b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d3ed42b3abc0723 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604cc69640 1 -- 172.21.15.27:0/1481408549 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038034070 con 0x7f60480b95a0 2024-03-06T07:32:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604cc69640 1 -- 172.21.15.27:0/1481408549 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6038030800 con 0x7f60480b95a0 2024-03-06T07:32:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604cc69640 1 -- 172.21.15.27:0/1481408549 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038030bb0 con 0x7f60480b95a0 2024-03-06T07:32:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604fecf640 1 -- 172.21.15.27:0/1481408549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60480b95a0 msgr2=0x7f60480b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604fecf640 1 --2- 172.21.15.27:0/1481408549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60480b95a0 0x7f60480b5700 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f60480cf590 tx=0x7f603802b940 comp rx=0 tx=0).stop 2024-03-06T07:32:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604fecf640 1 -- 172.21.15.27:0/1481408549 shutdown_connections 2024-03-06T07:32:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604fecf640 1 --2- 172.21.15.27:0/1481408549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60480b95a0 0x7f60480b5700 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604fecf640 1 -- 172.21.15.27:0/1481408549 >> 172.21.15.27:0/1481408549 conn(0x7f6048071960 msgr2=0x7f6048071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604fecf640 1 -- 172.21.15.27:0/1481408549 shutdown_connections 2024-03-06T07:32:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.359+0000 7f604fecf640 1 -- 172.21.15.27:0/1481408549 wait complete. 2024-03-06T07:32:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604fecf640 1 Processor -- start 2024-03-06T07:32:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604fecf640 1 -- start start 2024-03-06T07:32:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604fecf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60480b95a0 0x7f60480c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604fecf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480b9a10 0x7f60481605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604fecf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f604815df00 0x7f60480ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604fecf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60480cedf0 con 0x7f60480b9a10 2024-03-06T07:32:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604fecf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60480cec70 con 0x7f604815df00 2024-03-06T07:32:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604fecf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60480b5c40 con 0x7f60480b95a0 2024-03-06T07:32:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604d46a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480b9a10 0x7f60481605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604d46a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480b9a10 0x7f60481605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36050/0 (socket says 172.21.15.27:36050) 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604d46a640 1 -- 172.21.15.27:0/1509263764 learned_addr learned my addr 172.21.15.27:0/1509263764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.361+0000 7f604dc6b640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60480b95a0 0x7f60480c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f604e46c640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f604815df00 0x7f60480ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f604d46a640 1 -- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60480b95a0 msgr2=0x7f60480c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f604d46a640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60480b95a0 0x7f60480c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f604d46a640 1 -- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f604815df00 msgr2=0x7f60480ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f604d46a640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f604815df00 0x7f60480ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f604d46a640 1 -- 172.21.15.27:0/1509263764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60480c7d90 con 0x7f60480b9a10 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f604d46a640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480b9a10 0x7f60481605a0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f6038000c00 tx=0x7f60380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f603e7fc640 1 -- 172.21.15.27:0/1509263764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038034040 con 0x7f60480b9a10 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f603e7fc640 1 -- 172.21.15.27:0/1509263764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6038030600 con 0x7f60480b9a10 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60480c8070 con 0x7f60480b9a10 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f603e7fc640 1 -- 172.21.15.27:0/1509263764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038006e10 con 0x7f60480b9a10 2024-03-06T07:32:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.362+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60480cbda0 con 0x7f60480b9a10 2024-03-06T07:32:05.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.363+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6048067ff0 con 0x7f60480b9a10 2024-03-06T07:32:05.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.364+0000 7f603e7fc640 1 -- 172.21.15.27:0/1509263764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6038037070 con 0x7f60480b9a10 2024-03-06T07:32:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.364+0000 7f603e7fc640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f602c06b500 0x7f602c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.364+0000 7f603e7fc640 1 -- 172.21.15.27:0/1509263764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60380abdc0 con 0x7f60480b9a10 2024-03-06T07:32:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.364+0000 7f604dc6b640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f602c06b500 0x7f602c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.365+0000 7f604dc6b640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f602c06b500 0x7f602c06d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f60440011e0 tx=0x7f6044007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.373+0000 7f603e7fc640 1 -- 172.21.15.27:0/1509263764 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f603807a5f0 con 0x7f60480b9a10 2024-03-06T07:32:05.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:05.627+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f60480b8ed0 con 0x7f60480b9a10 2024-03-06T07:32:06.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.233+0000 7f603e7fc640 1 -- 172.21.15.27:0/1509263764 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7202 v117) v1 ==== 95+0+0 (secure 0 0 0) 0x7f603807e690 con 0x7f60480b9a10 2024-03-06T07:32:06.235 INFO:teuthology.orchestra.run.smithi027.stderr:failed mds gid 7202 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f602c06b500 msgr2=0x7f602c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f602c06b500 0x7f602c06d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f60440011e0 tx=0x7f6044007d30 comp rx=0 tx=0).stop 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480b9a10 msgr2=0x7f60481605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480b9a10 0x7f60481605a0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f6038000c00 tx=0x7f60380046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 shutdown_connections 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f602c06b500 0x7f602c06d9c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f604815df00 0x7f60480ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480b9a10 0x7f60481605a0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 --2- 172.21.15.27:0/1509263764 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f60480b95a0 0x7f60480c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 >> 172.21.15.27:0/1509263764 conn(0x7f6048071960 msgr2=0x7f60480b7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 shutdown_connections 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.235+0000 7f604fecf640 1 -- 172.21.15.27:0/1509263764 wait complete. 2024-03-06T07:32:06.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:06.249 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:32:06.250 DEBUG:teuthology.orchestra.run.smithi027:> mkdir dir_1 dir_2 2024-03-06T07:32:06.250 DEBUG:teuthology.orchestra.run.smithi027:> touch dir_1/original 2024-03-06T07:32:06.250 DEBUG:teuthology.orchestra.run.smithi027:> ln dir_1/original dir_2/linkto 2024-03-06T07:32:06.250 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:32:06.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:06.282 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 dir_1) 2024-03-06T07:32:06.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.485+0000 7f5c02e83640 1 Processor -- start 2024-03-06T07:32:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.485+0000 7f5c02e83640 1 -- start start 2024-03-06T07:32:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5c02e83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc15df00 0x7f5bfc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5c02e83640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5bfc0b9a30 0x7f5bfc0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5c02e83640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc0cea30 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5c02e83640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc0d2af0 con 0x7f5bfc0b9a30 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5c02e83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0cf3c0 con 0x7f5bfc15df00 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5c01420640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5bfc0b9a30 0x7f5bfc0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5bfbfff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5bfc0b95a0 0x7f5bfc0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:54900/0 (socket says 172.21.15.27:54900) 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5c01420640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5bfc0b9a30 0x7f5bfc0d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36062/0 (socket says 172.21.15.27:36062) 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5bfbfff640 1 -- 172.21.15.27:0/2390680769 learned_addr learned my addr 172.21.15.27:0/2390680769 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.486+0000 7f5c00c1f640 1 --2- 172.21.15.27:0/2390680769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc15df00 0x7f5bfc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.487+0000 7f5c01420640 1 -- 172.21.15.27:0/2390680769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc15df00 msgr2=0x7f5bfc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.487+0000 7f5c01420640 1 --2- 172.21.15.27:0/2390680769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc15df00 0x7f5bfc0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.487+0000 7f5c01420640 1 -- 172.21.15.27:0/2390680769 >> v1:172.21.15.69:6789/0 conn(0x7f5bfc0b95a0 legacy=0x7f5bfc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.487+0000 7f5c01420640 1 -- 172.21.15.27:0/2390680769 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0cf0e0 con 0x7f5bfc0b9a30 2024-03-06T07:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.487+0000 7f5c01420640 1 --2- 172.21.15.27:0/2390680769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b9a30 0x7f5bfc0d3220 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009d90 tx=0x7f5bf402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c649bbfd0c8f656 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5bfb7fe640 1 -- 172.21.15.27:0/2390680769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4039070 con 0x7f5bfc0b9a30 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5bfb7fe640 1 -- 172.21.15.27:0/2390680769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf402cc50 con 0x7f5bfc0b9a30 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5bfb7fe640 1 -- 172.21.15.27:0/2390680769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4034440 con 0x7f5bfc0b9a30 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5c02e83640 1 -- 172.21.15.27:0/2390680769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b9a30 msgr2=0x7f5bfc0d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5c02e83640 1 --2- 172.21.15.27:0/2390680769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b9a30 0x7f5bfc0d3220 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009d90 tx=0x7f5bf402bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5c02e83640 1 -- 172.21.15.27:0/2390680769 shutdown_connections 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5c02e83640 1 --2- 172.21.15.27:0/2390680769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b9a30 0x7f5bfc0d3220 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5c02e83640 1 --2- 172.21.15.27:0/2390680769 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc15df00 0x7f5bfc0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5c02e83640 1 -- 172.21.15.27:0/2390680769 >> 172.21.15.27:0/2390680769 conn(0x7f5bfc071960 msgr2=0x7f5bfc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5c02e83640 1 -- 172.21.15.27:0/2390680769 shutdown_connections 2024-03-06T07:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.488+0000 7f5c02e83640 1 -- 172.21.15.27:0/2390680769 wait complete. 2024-03-06T07:32:06.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.490+0000 7f5c02e83640 1 Processor -- start 2024-03-06T07:32:06.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.490+0000 7f5c02e83640 1 -- start start 2024-03-06T07:32:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c02e83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc0b95a0 0x7f5bfc158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c02e83640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc15df00 0x7f5bfc0b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c02e83640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b5be0 0x7f5bfc160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c02e83640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0cec70 con 0x7f5bfc0b5be0 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c02e83640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0ce750 con 0x7f5bfc15df00 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c02e83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0cedf0 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c01420640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b5be0 0x7f5bfc160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c00c1f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc0b95a0 0x7f5bfc158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5bfbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc15df00 0x7f5bfc0b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c00c1f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc0b95a0 0x7f5bfc158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:34274/0 (socket says 172.21.15.27:34274) 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c01420640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b5be0 0x7f5bfc160230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36064/0 (socket says 172.21.15.27:36064) 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c00c1f640 1 -- 172.21.15.27:0/2988513346 learned_addr learned my addr 172.21.15.27:0/2988513346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c00c1f640 1 -- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc15df00 msgr2=0x7f5bfc0b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c00c1f640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc15df00 0x7f5bfc0b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c00c1f640 1 -- 172.21.15.27:0/2988513346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b5be0 msgr2=0x7f5bfc160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c00c1f640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b5be0 0x7f5bfc160230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c00c1f640 1 -- 172.21.15.27:0/2988513346 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc15e7b0 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.492+0000 7f5c01420640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b5be0 0x7f5bfc160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.493+0000 7f5c00c1f640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc0b95a0 0x7f5bfc158d90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000e8a0 tx=0x7f5bf000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.493+0000 7f5bf97fa640 1 -- 172.21.15.27:0/2988513346 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0017070 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.493+0000 7f5bf97fa640 1 -- 172.21.15.27:0/2988513346 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf000c940 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.493+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc15ea90 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.493+0000 7f5bf97fa640 1 -- 172.21.15.27:0/2988513346 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf000cd30 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.493+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc15f050 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.494+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc0b5ff0 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.494+0000 7f5bf97fa640 1 -- 172.21.15.27:0/2988513346 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bf001a070 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.495+0000 7f5bf97fa640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.495+0000 7f5bf97fa640 1 -- 172.21.15.27:0/2988513346 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5bf0089f60 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.495+0000 7f5bfbfff640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.496+0000 7f5bfbfff640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0b5460 tx=0x7f5be8007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.500+0000 7f5bf97fa640 1 -- 172.21.15.27:0/2988513346 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bfc0b5ff0 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.721+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bfc0d0420 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.722+0000 7f5bf97fa640 1 -- 172.21.15.27:0/2988513346 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3423 (secure 0 0 0) 0x7f5bf0058820 con 0x7f5bfc0b95a0 2024-03-06T07:32:06.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:06.724 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:31:59.200887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9270,9282]},"id":10}]} 2024-03-06T07:32:06.725 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-03-06T07:32:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 msgr2=0x7f5be006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0b5460 tx=0x7f5be8007e80 comp rx=0 tx=0).stop 2024-03-06T07:32:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc0b95a0 msgr2=0x7f5bfc158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc0b95a0 0x7f5bfc158d90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000e8a0 tx=0x7f5bf000ed60 comp rx=0 tx=0).stop 2024-03-06T07:32:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 shutdown_connections 2024-03-06T07:32:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5be006b4d0 0x7f5be006d990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bfc0b5be0 0x7f5bfc160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bfc15df00 0x7f5bfc0b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 --2- 172.21.15.27:0/2988513346 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bfc0b95a0 0x7f5bfc158d90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 >> 172.21.15.27:0/2988513346 conn(0x7f5bfc071960 msgr2=0x7f5bfc0b8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 shutdown_connections 2024-03-06T07:32:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:06.724+0000 7f5c02e83640 1 -- 172.21.15.27:0/2988513346 wait complete. 2024-03-06T07:32:06.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-06T07:32:06.867 INFO:teuthology.orchestra.run.smithi069.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:32:06.867 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:32:06.867 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:32:08.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.977+0000 7f0f89cc9640 1 Processor -- start 2024-03-06T07:32:08.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.977+0000 7f0f89cc9640 1 -- start start 2024-03-06T07:32:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.978+0000 7f0f89cc9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0f840b9a10 0x7f0f840d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.978+0000 7f0f89cc9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f840cea30 con 0x7f0f840b95a0 2024-03-06T07:32:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.978+0000 7f0f89cc9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f840cf3c0 con 0x7f0f8415df00 2024-03-06T07:32:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.978+0000 7f0f89cc9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f840cf0e0 con 0x7f0f840b9a10 2024-03-06T07:32:08.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.978+0000 7f0f82ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0f840b95a0 0x7f0f840b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59590/0 (socket says 172.21.15.27:59590) 2024-03-06T07:32:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.978+0000 7f0f82ffd640 1 -- 172.21.15.27:0/2983522945 learned_addr learned my addr 172.21.15.27:0/2983522945 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.978+0000 7f0f83fff640 1 --2- 172.21.15.27:0/2983522945 >> v2:172.21.15.69:3300/0 conn(0x7f0f840b9a10 0x7f0f840d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.979+0000 7f0f827fc640 1 -- 172.21.15.27:0/2983522945 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889091155 0 0) 0x7f0f840cea30 con 0x7f0f840b95a0 2024-03-06T07:32:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.979+0000 7f0f827fc640 1 -- 172.21.15.27:0/2983522945 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f70003520 con 0x7f0f840b95a0 2024-03-06T07:32:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.979+0000 7f0f827fc640 1 -- 172.21.15.27:0/2983522945 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973994382 0 0) 0x7f0f840cf3c0 con 0x7f0f8415df00 2024-03-06T07:32:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.979+0000 7f0f827fc640 1 -- 172.21.15.27:0/2983522945 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f840cea30 con 0x7f0f8415df00 2024-03-06T07:32:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.979+0000 7f0f83fff640 1 -- 172.21.15.27:0/2983522945 >> v1:172.21.15.69:6789/0 conn(0x7f0f8415df00 legacy=0x7f0f840d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.979+0000 7f0f83fff640 1 -- 172.21.15.27:0/2983522945 >> v1:172.21.15.27:6789/0 conn(0x7f0f840b95a0 legacy=0x7f0f840b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.979+0000 7f0f83fff640 1 -- 172.21.15.27:0/2983522945 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f840ce470 con 0x7f0f840b9a10 2024-03-06T07:32:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.979+0000 7f0f83fff640 1 --2- 172.21.15.27:0/2983522945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f840b9a10 0x7f0f840d3220 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0f840cf590 tx=0x7f0f7c02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10d59aefac07aaa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f827fc640 1 -- 172.21.15.27:0/2983522945 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c039070 con 0x7f0f840b9a10 2024-03-06T07:32:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f827fc640 1 -- 172.21.15.27:0/2983522945 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c02cce0 con 0x7f0f840b9a10 2024-03-06T07:32:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f827fc640 1 -- 172.21.15.27:0/2983522945 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c034660 con 0x7f0f840b9a10 2024-03-06T07:32:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f89cc9640 1 -- 172.21.15.27:0/2983522945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f840b9a10 msgr2=0x7f0f840d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f89cc9640 1 --2- 172.21.15.27:0/2983522945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f840b9a10 0x7f0f840d3220 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0f840cf590 tx=0x7f0f7c02bf30 comp rx=0 tx=0).stop 2024-03-06T07:32:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f89cc9640 1 -- 172.21.15.27:0/2983522945 shutdown_connections 2024-03-06T07:32:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f89cc9640 1 --2- 172.21.15.27:0/2983522945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f840b9a10 0x7f0f840d3220 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f89cc9640 1 -- 172.21.15.27:0/2983522945 >> 172.21.15.27:0/2983522945 conn(0x7f0f84071960 msgr2=0x7f0f84071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f89cc9640 1 -- 172.21.15.27:0/2983522945 shutdown_connections 2024-03-06T07:32:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.980+0000 7f0f89cc9640 1 -- 172.21.15.27:0/2983522945 wait complete. 2024-03-06T07:32:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.981+0000 7f0f89cc9640 1 Processor -- start 2024-03-06T07:32:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.982+0000 7f0f89cc9640 1 -- start start 2024-03-06T07:32:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.982+0000 7f0f89cc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f840b95a0 0x7f0f84155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.982+0000 7f0f89cc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f840b9a10 0x7f0f84159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.982+0000 7f0f89cc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f8415df00 0x7f0f8415a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.982+0000 7f0f89cc9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840cedf0 con 0x7f0f840b95a0 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.982+0000 7f0f89cc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840cec70 con 0x7f0f8415df00 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.982+0000 7f0f89cc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840b5e10 con 0x7f0f840b9a10 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.982+0000 7f0f837fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f840b95a0 0x7f0f84155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f837fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f840b95a0 0x7f0f84155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36072/0 (socket says 172.21.15.27:36072) 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f837fe640 1 -- 172.21.15.27:0/1715666709 learned_addr learned my addr 172.21.15.27:0/1715666709 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f83fff640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f8415df00 0x7f0f8415a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f82ffd640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f840b9a10 0x7f0f84159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f837fe640 1 -- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f840b9a10 msgr2=0x7f0f84159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f837fe640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f840b9a10 0x7f0f84159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f837fe640 1 -- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f8415df00 msgr2=0x7f0f8415a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f837fe640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f8415df00 0x7f0f8415a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f837fe640 1 -- 172.21.15.27:0/1715666709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f84155750 con 0x7f0f840b95a0 2024-03-06T07:32:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f83fff640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f8415df00 0x7f0f8415a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f837fe640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f840b95a0 0x7f0f84155210 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00c430 tx=0x7f0f6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f7bfff640 1 -- 172.21.15.27:0/1715666709 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c017070 con 0x7f0f840b95a0 2024-03-06T07:32:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f7bfff640 1 -- 172.21.15.27:0/1715666709 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c0052f0 con 0x7f0f840b95a0 2024-03-06T07:32:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8415a8f0 con 0x7f0f840b95a0 2024-03-06T07:32:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f7bfff640 1 -- 172.21.15.27:0/1715666709 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c003830 con 0x7f0f840b95a0 2024-03-06T07:32:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.983+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f840cbdb0 con 0x7f0f840b95a0 2024-03-06T07:32:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.984+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8415ae10 con 0x7f0f840b95a0 2024-03-06T07:32:08.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.985+0000 7f0f7bfff640 1 -- 172.21.15.27:0/1715666709 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f6c01a070 con 0x7f0f840b95a0 2024-03-06T07:32:08.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.985+0000 7f0f7bfff640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f6406b4d0 0x7f0f6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.986+0000 7f0f7bfff640 1 -- 172.21.15.27:0/1715666709 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f6c089470 con 0x7f0f840b95a0 2024-03-06T07:32:08.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.986+0000 7f0f82ffd640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f6406b4d0 0x7f0f6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.986+0000 7f0f82ffd640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f6406b4d0 0x7f0f6406d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0f840cdbb0 tx=0x7f0f74007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:08.990+0000 7f0f7bfff640 1 -- 172.21.15.27:0/1715666709 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f8415ae10 con 0x7f0f840b95a0 2024-03-06T07:32:09.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.213+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f84067ff0 con 0x7f0f840b95a0 2024-03-06T07:32:09.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.213+0000 7f0f7bfff640 1 -- 172.21.15.27:0/1715666709 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3423 (secure 0 0 0) 0x7f0f84155750 con 0x7f0f840b95a0 2024-03-06T07:32:09.216 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:09.216 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:31:59.200887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9270,9282]},"id":10}]} 2024-03-06T07:32:09.217 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-03-06T07:32:09.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.216+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f6406b4d0 msgr2=0x7f0f6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f6406b4d0 0x7f0f6406d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0f840cdbb0 tx=0x7f0f74007d30 comp rx=0 tx=0).stop 2024-03-06T07:32:09.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f840b95a0 msgr2=0x7f0f84155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f840b95a0 0x7f0f84155210 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00c430 tx=0x7f0f6c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 shutdown_connections 2024-03-06T07:32:09.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0f6406b4d0 0x7f0f6406d990 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f8415df00 0x7f0f8415a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f840b9a10 0x7f0f84159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 --2- 172.21.15.27:0/1715666709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f840b95a0 0x7f0f84155210 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 >> 172.21.15.27:0/1715666709 conn(0x7f0f84071960 msgr2=0x7f0f8415d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 shutdown_connections 2024-03-06T07:32:09.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:09.217+0000 7f0f89cc9640 1 -- 172.21.15.27:0/1715666709 wait complete. 2024-03-06T07:32:09.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-06T07:32:09.351 INFO:teuthology.orchestra.run.smithi069.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:32:09.352 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:32:09.352 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:32:11.353 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.470+0000 7f6f0e0a4640 1 Processor -- start 2024-03-06T07:32:11.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.470+0000 7f6f0e0a4640 1 -- start start 2024-03-06T07:32:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.470+0000 7f6f0e0a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f080b9a10 0x7f6f080d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.470+0000 7f6f0e0a4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080cea30 con 0x7f6f0815df00 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.470+0000 7f6f0e0a4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080cf3c0 con 0x7f6f080b95a0 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.470+0000 7f6f0e0a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f080cf0e0 con 0x7f6f080b9a10 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.470+0000 7f6f077fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6f0815df00 0x7f6f080d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59602/0 (socket says 172.21.15.27:59602) 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.471+0000 7f6f077fe640 1 -- 172.21.15.27:0/1548333975 learned_addr learned my addr 172.21.15.27:0/1548333975 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.471+0000 7f6f07fff640 1 --2- 172.21.15.27:0/1548333975 >> v2:172.21.15.69:3300/0 conn(0x7f6f080b9a10 0x7f6f080d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.471+0000 7f6f067fc640 1 -- 172.21.15.27:0/1548333975 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697833270 0 0) 0x7f6f080cea30 con 0x7f6f0815df00 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.471+0000 7f6f067fc640 1 -- 172.21.15.27:0/1548333975 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef4003520 con 0x7f6f0815df00 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.471+0000 7f6f067fc640 1 -- 172.21.15.27:0/1548333975 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565054968 0 0) 0x7f6f080cf3c0 con 0x7f6f080b95a0 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.471+0000 7f6f067fc640 1 -- 172.21.15.27:0/1548333975 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f080cea30 con 0x7f6f080b95a0 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.471+0000 7f6f07fff640 1 -- 172.21.15.27:0/1548333975 >> v1:172.21.15.69:6789/0 conn(0x7f6f080b95a0 legacy=0x7f6f080b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.471+0000 7f6f07fff640 1 -- 172.21.15.27:0/1548333975 >> v1:172.21.15.27:6789/0 conn(0x7f6f0815df00 legacy=0x7f6f080d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.471+0000 7f6f07fff640 1 -- 172.21.15.27:0/1548333975 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f080ce470 con 0x7f6f080b9a10 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.472+0000 7f6f07fff640 1 --2- 172.21.15.27:0/1548333975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f080b9a10 0x7f6f080d3220 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6f080cf590 tx=0x7f6f0002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34d0dfd5a6ebadb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.472+0000 7f6f067fc640 1 -- 172.21.15.27:0/1548333975 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00039070 con 0x7f6f080b9a10 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.472+0000 7f6f067fc640 1 -- 172.21.15.27:0/1548333975 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0002cce0 con 0x7f6f080b9a10 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.472+0000 7f6f067fc640 1 -- 172.21.15.27:0/1548333975 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00034660 con 0x7f6f080b9a10 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.472+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/1548333975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f080b9a10 msgr2=0x7f6f080d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.472+0000 7f6f0e0a4640 1 --2- 172.21.15.27:0/1548333975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f080b9a10 0x7f6f080d3220 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6f080cf590 tx=0x7f6f0002bf30 comp rx=0 tx=0).stop 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.472+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/1548333975 shutdown_connections 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.472+0000 7f6f0e0a4640 1 --2- 172.21.15.27:0/1548333975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f080b9a10 0x7f6f080d3220 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.472+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/1548333975 >> 172.21.15.27:0/1548333975 conn(0x7f6f08071960 msgr2=0x7f6f08071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.473+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/1548333975 shutdown_connections 2024-03-06T07:32:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.473+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/1548333975 wait complete. 2024-03-06T07:32:11.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.474+0000 7f6f0e0a4640 1 Processor -- start 2024-03-06T07:32:11.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.474+0000 7f6f0e0a4640 1 -- start start 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.474+0000 7f6f0e0a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f080b95a0 0x7f6f08159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.474+0000 7f6f0e0a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f080b9a10 0x7f6f080c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f0e0a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f0815df00 0x7f6f080cc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f0e0a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f080cedf0 con 0x7f6f080b95a0 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f0e0a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f080cec70 con 0x7f6f080b9a10 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f0e0a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f080b5e10 con 0x7f6f0815df00 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f080b95a0 0x7f6f08159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f077fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f080b95a0 0x7f6f08159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36074/0 (socket says 172.21.15.27:36074) 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f077fe640 1 -- 172.21.15.27:0/2673496080 learned_addr learned my addr 172.21.15.27:0/2673496080 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f06ffd640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f080b9a10 0x7f6f080c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f07fff640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f0815df00 0x7f6f080cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f077fe640 1 -- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f0815df00 msgr2=0x7f6f080cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f077fe640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f0815df00 0x7f6f080cc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f077fe640 1 -- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f080b9a10 msgr2=0x7f6f080c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f077fe640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f080b9a10 0x7f6f080c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f077fe640 1 -- 172.21.15.27:0/2673496080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f080c84c0 con 0x7f6f080b95a0 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f077fe640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f080b95a0 0x7f6f08159c80 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000c440 tx=0x7f6ef000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6efffff640 1 -- 172.21.15.27:0/2673496080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0017070 con 0x7f6f080b95a0 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6efffff640 1 -- 172.21.15.27:0/2673496080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef0005ce0 con 0x7f6f080b95a0 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f080c87a0 con 0x7f6f080b95a0 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6efffff640 1 -- 172.21.15.27:0/2673496080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0007270 con 0x7f6f080b95a0 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.475+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0815a1c0 con 0x7f6f080b95a0 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.476+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f08067ff0 con 0x7f6f080b95a0 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.477+0000 7f6efffff640 1 -- 172.21.15.27:0/2673496080 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ef001a070 con 0x7f6f080b95a0 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.477+0000 7f6efffff640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ee806b500 0x7f6ee806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.477+0000 7f6efffff640 1 -- 172.21.15.27:0/2673496080 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ef00893c0 con 0x7f6f080b95a0 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.477+0000 7f6f06ffd640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ee806b500 0x7f6ee806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.478+0000 7f6f06ffd640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ee806b500 0x7f6ee806d9c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6f08161670 tx=0x7f6ef8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.482+0000 7f6efffff640 1 -- 172.21.15.27:0/2673496080 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ef0057cb0 con 0x7f6f080b95a0 2024-03-06T07:32:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.748+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f080b5560 con 0x7f6f080b95a0 2024-03-06T07:32:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.748+0000 7f6efffff640 1 -- 172.21.15.27:0/2673496080 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3423 (secure 0 0 0) 0x7f6ef005bd50 con 0x7f6f080b95a0 2024-03-06T07:32:11.751 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:11.751 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:31:59.200887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9270,9282]},"id":10}]} 2024-03-06T07:32:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ee806b500 msgr2=0x7f6ee806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ee806b500 0x7f6ee806d9c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6f08161670 tx=0x7f6ef8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f080b95a0 msgr2=0x7f6f08159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f080b95a0 0x7f6f08159c80 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000c440 tx=0x7f6ef000c900 comp rx=0 tx=0).stop 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 shutdown_connections 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6ee806b500 0x7f6ee806d9c0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f0815df00 0x7f6f080cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f080b9a10 0x7f6f080c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 --2- 172.21.15.27:0/2673496080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f080b95a0 0x7f6f08159c80 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 >> 172.21.15.27:0/2673496080 conn(0x7f6f08071960 msgr2=0x7f6f080b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 shutdown_connections 2024-03-06T07:32:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:11.751+0000 7f6f0e0a4640 1 -- 172.21.15.27:0/2673496080 wait complete. 2024-03-06T07:32:11.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-06T07:32:11.881 INFO:teuthology.orchestra.run.smithi069.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:32:11.882 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:32:11.882 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:32:13.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:13.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.994+0000 7ff367713640 1 Processor -- start 2024-03-06T07:32:13.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.995+0000 7ff367713640 1 -- start start 2024-03-06T07:32:13.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.995+0000 7ff367713640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff3600cfe40 0x7ff3600d0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.995+0000 7ff367713640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36006c040 con 0x7ff3600ce3c0 2024-03-06T07:32:13.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.995+0000 7ff367713640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3600d0760 con 0x7ff3600b48c0 2024-03-06T07:32:13.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.995+0000 7ff367713640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3600b33b0 con 0x7ff3600cfe40 2024-03-06T07:32:13.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff365cb0640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff3600ce3c0 0x7ff3600ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34676/0 (socket says 172.21.15.27:34676) 2024-03-06T07:32:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff365cb0640 1 -- 172.21.15.27:0/1977416714 learned_addr learned my addr 172.21.15.27:0/1977416714 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff3654af640 1 --2- 172.21.15.27:0/1977416714 >> v2:172.21.15.69:3300/0 conn(0x7ff3600cfe40 0x7ff3600d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff35ffff640 1 -- 172.21.15.27:0/1977416714 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881292535 0 0) 0x7ff36006c040 con 0x7ff3600ce3c0 2024-03-06T07:32:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff35ffff640 1 -- 172.21.15.27:0/1977416714 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff348003520 con 0x7ff3600ce3c0 2024-03-06T07:32:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff35ffff640 1 -- 172.21.15.27:0/1977416714 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868810390 0 0) 0x7ff3600d0760 con 0x7ff3600b48c0 2024-03-06T07:32:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff35ffff640 1 -- 172.21.15.27:0/1977416714 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff36006c040 con 0x7ff3600b48c0 2024-03-06T07:32:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff3654af640 1 -- 172.21.15.27:0/1977416714 >> v1:172.21.15.69:6789/0 conn(0x7ff3600b48c0 legacy=0x7ff36006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff3654af640 1 -- 172.21.15.27:0/1977416714 >> v1:172.21.15.27:6789/0 conn(0x7ff3600ce3c0 legacy=0x7ff3600ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.996+0000 7ff3654af640 1 -- 172.21.15.27:0/1977416714 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3600cd320 con 0x7ff3600cfe40 2024-03-06T07:32:14.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.997+0000 7ff3654af640 1 --2- 172.21.15.27:0/1977416714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600cfe40 0x7ff3600d0220 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff350009dc0 tx=0x7ff35002bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a32d66d166e2c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.997+0000 7ff35ffff640 1 -- 172.21.15.27:0/1977416714 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350039070 con 0x7ff3600cfe40 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.997+0000 7ff35ffff640 1 -- 172.21.15.27:0/1977416714 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35002ccd0 con 0x7ff3600cfe40 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.997+0000 7ff35ffff640 1 -- 172.21.15.27:0/1977416714 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350034640 con 0x7ff3600cfe40 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.998+0000 7ff367713640 1 -- 172.21.15.27:0/1977416714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600cfe40 msgr2=0x7ff3600d0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.998+0000 7ff367713640 1 --2- 172.21.15.27:0/1977416714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600cfe40 0x7ff3600d0220 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff350009dc0 tx=0x7ff35002bf20 comp rx=0 tx=0).stop 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.998+0000 7ff367713640 1 -- 172.21.15.27:0/1977416714 shutdown_connections 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.998+0000 7ff367713640 1 --2- 172.21.15.27:0/1977416714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600cfe40 0x7ff3600d0220 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.998+0000 7ff367713640 1 -- 172.21.15.27:0/1977416714 >> 172.21.15.27:0/1977416714 conn(0x7ff3600d1a70 msgr2=0x7ff3600d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.998+0000 7ff367713640 1 -- 172.21.15.27:0/1977416714 shutdown_connections 2024-03-06T07:32:14.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.998+0000 7ff367713640 1 -- 172.21.15.27:0/1977416714 wait complete. 2024-03-06T07:32:14.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.999+0000 7ff367713640 1 Processor -- start 2024-03-06T07:32:14.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:13.999+0000 7ff367713640 1 -- start start 2024-03-06T07:32:14.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff367713640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3600b48c0 0x7ff360155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff367713640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600ce3c0 0x7ff3600b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff367713640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3600cfe40 0x7ff3600b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff367713640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600cd8e0 con 0x7ff3600cfe40 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff367713640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600cd0d0 con 0x7ff3600ce3c0 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff367713640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600cdbc0 con 0x7ff3600b48c0 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff365cb0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3600cfe40 0x7ff3600b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff365cb0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3600cfe40 0x7ff3600b4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60558/0 (socket says 172.21.15.27:60558) 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff365cb0640 1 -- 172.21.15.27:0/3832406250 learned_addr learned my addr 172.21.15.27:0/3832406250 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff3654af640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3600b48c0 0x7ff360155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff364cae640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600ce3c0 0x7ff3600b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff365cb0640 1 -- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3600b48c0 msgr2=0x7ff360155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff365cb0640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3600b48c0 0x7ff360155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff365cb0640 1 -- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600ce3c0 msgr2=0x7ff3600b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff365cb0640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600ce3c0 0x7ff3600b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff365cb0640 1 -- 172.21.15.27:0/3832406250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36009e010 con 0x7ff3600cfe40 2024-03-06T07:32:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff3654af640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3600b48c0 0x7ff360155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:14.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.000+0000 7ff364cae640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600ce3c0 0x7ff3600b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:14.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.001+0000 7ff365cb0640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3600cfe40 0x7ff3600b4cf0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7ff35800c430 tx=0x7ff35800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.001+0000 7ff35dffb640 1 -- 172.21.15.27:0/3832406250 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff358017070 con 0x7ff3600cfe40 2024-03-06T07:32:14.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.001+0000 7ff35dffb640 1 -- 172.21.15.27:0/3832406250 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff358005ce0 con 0x7ff3600cfe40 2024-03-06T07:32:14.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.001+0000 7ff35dffb640 1 -- 172.21.15.27:0/3832406250 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3580072f0 con 0x7ff3600cfe40 2024-03-06T07:32:14.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.001+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3601f8000 con 0x7ff3600cfe40 2024-03-06T07:32:14.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.001+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36009e590 con 0x7ff3600cfe40 2024-03-06T07:32:14.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.002+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff360067ff0 con 0x7ff3600cfe40 2024-03-06T07:32:14.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.002+0000 7ff35dffb640 1 -- 172.21.15.27:0/3832406250 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff35801a070 con 0x7ff3600cfe40 2024-03-06T07:32:14.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.003+0000 7ff35dffb640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff34c06b500 0x7ff34c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.003+0000 7ff35dffb640 1 -- 172.21.15.27:0/3832406250 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff358089590 con 0x7ff3600cfe40 2024-03-06T07:32:14.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.003+0000 7ff3654af640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff34c06b500 0x7ff34c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.004+0000 7ff3654af640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff34c06b500 0x7ff34c06d9c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff350036000 tx=0x7ff350030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.008+0000 7ff35dffb640 1 -- 172.21.15.27:0/3832406250 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff358057dc0 con 0x7ff3600cfe40 2024-03-06T07:32:14.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.291+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3600cde90 con 0x7ff3600cfe40 2024-03-06T07:32:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.292+0000 7ff35dffb640 1 -- 172.21.15.27:0/3832406250 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3423 (secure 0 0 0) 0x7ff35805be60 con 0x7ff3600cfe40 2024-03-06T07:32:14.294 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:14.294 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:31:59.200887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9270,9282]},"id":10}]} 2024-03-06T07:32:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-03-06T07:32:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.293+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff34c06b500 msgr2=0x7ff34c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.293+0000 7ff367713640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff34c06b500 0x7ff34c06d9c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff350036000 tx=0x7ff350030000 comp rx=0 tx=0).stop 2024-03-06T07:32:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.293+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3600cfe40 msgr2=0x7ff3600b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.293+0000 7ff367713640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3600cfe40 0x7ff3600b4cf0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7ff35800c430 tx=0x7ff35800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.294+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 shutdown_connections 2024-03-06T07:32:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.294+0000 7ff367713640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff34c06b500 0x7ff34c06d9c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.294+0000 7ff367713640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3600cfe40 0x7ff3600b4cf0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.294+0000 7ff367713640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3600ce3c0 0x7ff3600b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.294+0000 7ff367713640 1 --2- 172.21.15.27:0/3832406250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3600b48c0 0x7ff360155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.294+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 >> 172.21.15.27:0/3832406250 conn(0x7ff3600d1a70 msgr2=0x7ff36006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.294+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 shutdown_connections 2024-03-06T07:32:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:14.294+0000 7ff367713640 1 -- 172.21.15.27:0/3832406250 wait complete. 2024-03-06T07:32:14.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-06T07:32:14.453 INFO:teuthology.orchestra.run.smithi069.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:32:14.453 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:32:14.453 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:32:16.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:16.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.558+0000 7fa22538a640 1 Processor -- start 2024-03-06T07:32:16.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.559+0000 7fa22538a640 1 -- start start 2024-03-06T07:32:16.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.559+0000 7fa22538a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa22015df00 0x7fa2200d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.559+0000 7fa22538a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa2200b95a0 0x7fa2200b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.559+0000 7fa22538a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2200cea30 con 0x7fa2200b9a30 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.559+0000 7fa22538a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2200d2af0 con 0x7fa22015df00 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.559+0000 7fa22538a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2200cf3c0 con 0x7fa2200b95a0 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.559+0000 7fa21effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa22015df00 0x7fa2200d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.560+0000 7fa21effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa22015df00 0x7fa2200d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60574/0 (socket says 172.21.15.27:60574) 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.560+0000 7fa21effd640 1 -- 172.21.15.27:0/2732157517 learned_addr learned my addr 172.21.15.27:0/2732157517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.560+0000 7fa21e7fc640 1 --2- 172.21.15.27:0/2732157517 >> v2:172.21.15.69:3300/0 conn(0x7fa2200b95a0 0x7fa2200b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.560+0000 7fa21effd640 1 -- 172.21.15.27:0/2732157517 >> v2:172.21.15.69:3300/0 conn(0x7fa2200b95a0 msgr2=0x7fa2200b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.560+0000 7fa21effd640 1 --2- 172.21.15.27:0/2732157517 >> v2:172.21.15.69:3300/0 conn(0x7fa2200b95a0 0x7fa2200b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.560+0000 7fa21effd640 1 -- 172.21.15.27:0/2732157517 >> v1:172.21.15.69:6789/0 conn(0x7fa2200b9a30 legacy=0x7fa2200d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.560+0000 7fa21dffb640 1 -- 172.21.15.27:0/2732157517 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515586954 0 0) 0x7fa2200cea30 con 0x7fa2200b9a30 2024-03-06T07:32:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.560+0000 7fa21effd640 1 -- 172.21.15.27:0/2732157517 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2200cf0e0 con 0x7fa22015df00 2024-03-06T07:32:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa21effd640 1 --2- 172.21.15.27:0/2732157517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22015df00 0x7fa2200d25b0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fa2200d1760 tx=0x7fa21002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67504d6ff4d86498 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa21dffb640 1 -- 172.21.15.27:0/2732157517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210034070 con 0x7fa22015df00 2024-03-06T07:32:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa21dffb640 1 -- 172.21.15.27:0/2732157517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa210030860 con 0x7fa22015df00 2024-03-06T07:32:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa21dffb640 1 -- 172.21.15.27:0/2732157517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210030c30 con 0x7fa22015df00 2024-03-06T07:32:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa22538a640 1 -- 172.21.15.27:0/2732157517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22015df00 msgr2=0x7fa2200d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa22538a640 1 --2- 172.21.15.27:0/2732157517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22015df00 0x7fa2200d25b0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fa2200d1760 tx=0x7fa21002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa22538a640 1 -- 172.21.15.27:0/2732157517 shutdown_connections 2024-03-06T07:32:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa22538a640 1 --2- 172.21.15.27:0/2732157517 >> v2:172.21.15.69:3300/0 conn(0x7fa2200b95a0 0x7fa2200b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa22538a640 1 --2- 172.21.15.27:0/2732157517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa22015df00 0x7fa2200d25b0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa22538a640 1 -- 172.21.15.27:0/2732157517 >> 172.21.15.27:0/2732157517 conn(0x7fa220071960 msgr2=0x7fa220071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa22538a640 1 -- 172.21.15.27:0/2732157517 shutdown_connections 2024-03-06T07:32:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.561+0000 7fa22538a640 1 -- 172.21.15.27:0/2732157517 wait complete. 2024-03-06T07:32:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.563+0000 7fa22538a640 1 Processor -- start 2024-03-06T07:32:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.563+0000 7fa22538a640 1 -- start start 2024-03-06T07:32:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.563+0000 7fa22538a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2200b95a0 0x7fa220152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.563+0000 7fa22538a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa22015df00 0x7fa2201541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.563+0000 7fa22538a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa220154710 0x7fa2200cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.563+0000 7fa22538a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200cec70 con 0x7fa220154710 2024-03-06T07:32:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.563+0000 7fa22538a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200ce750 con 0x7fa22015df00 2024-03-06T07:32:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.563+0000 7fa22538a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200cedf0 con 0x7fa2200b95a0 2024-03-06T07:32:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa220154710 0x7fa2200cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa220154710 0x7fa2200cc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60580/0 (socket says 172.21.15.27:60580) 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21f7fe640 1 -- 172.21.15.27:0/734208479 learned_addr learned my addr 172.21.15.27:0/734208479 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21effd640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2200b95a0 0x7fa220152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21e7fc640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa22015df00 0x7fa2201541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21f7fe640 1 -- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2200b95a0 msgr2=0x7fa220152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21f7fe640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2200b95a0 0x7fa220152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21f7fe640 1 -- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa22015df00 msgr2=0x7fa2201541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21f7fe640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa22015df00 0x7fa2201541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21f7fe640 1 -- 172.21.15.27:0/734208479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2201f7210 con 0x7fa220154710 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa21f7fe640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa220154710 0x7fa2200cc7f0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c760 tx=0x7fa21800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa2077fe640 1 -- 172.21.15.27:0/734208479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218003da0 con 0x7fa220154710 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa2077fe640 1 -- 172.21.15.27:0/734208479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa218005070 con 0x7fa220154710 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa220154d20 con 0x7fa220154710 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa2077fe640 1 -- 172.21.15.27:0/734208479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218005460 con 0x7fa220154710 2024-03-06T07:32:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.564+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2201552e0 con 0x7fa220154710 2024-03-06T07:32:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.566+0000 7fa2077fe640 1 -- 172.21.15.27:0/734208479 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa218005600 con 0x7fa220154710 2024-03-06T07:32:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.566+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa220067ff0 con 0x7fa220154710 2024-03-06T07:32:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.566+0000 7fa2077fe640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa20c06b4d0 0x7fa20c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.566+0000 7fa2077fe640 1 -- 172.21.15.27:0/734208479 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa218089fa0 con 0x7fa220154710 2024-03-06T07:32:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.566+0000 7fa21effd640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa20c06b4d0 0x7fa20c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.567+0000 7fa21effd640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa20c06b4d0 0x7fa20c06d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa210036000 tx=0x7fa21003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.571+0000 7fa2077fe640 1 -- 172.21.15.27:0/734208479 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa218058860 con 0x7fa220154710 2024-03-06T07:32:16.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.848+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2200b6ee0 con 0x7fa220154710 2024-03-06T07:32:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.848+0000 7fa2077fe640 1 -- 172.21.15.27:0/734208479 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+3424 (secure 0 0 0) 0x7fa21805c900 con 0x7fa220154710 2024-03-06T07:32:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:16.851 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":118,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:15.028959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282,9270]},"id":10}]} 2024-03-06T07:32:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 118 2024-03-06T07:32:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa20c06b4d0 msgr2=0x7fa20c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa20c06b4d0 0x7fa20c06d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa210036000 tx=0x7fa21003c040 comp rx=0 tx=0).stop 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa220154710 msgr2=0x7fa2200cc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa220154710 0x7fa2200cc7f0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c760 tx=0x7fa21800cc20 comp rx=0 tx=0).stop 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 shutdown_connections 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa20c06b4d0 0x7fa20c06d990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa220154710 0x7fa2200cc7f0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa22015df00 0x7fa2201541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 --2- 172.21.15.27:0/734208479 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa2200b95a0 0x7fa220152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 >> 172.21.15.27:0/734208479 conn(0x7fa220071960 msgr2=0x7fa2200d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 shutdown_connections 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:16.850+0000 7fa22538a640 1 -- 172.21.15.27:0/734208479 wait complete. 2024-03-06T07:32:16.864 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-06T07:32:16.980 INFO:teuthology.orchestra.run.smithi069.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"16","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:32:16.981 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/dir_1", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:32:16.981 INFO:tasks.cephfs.cephfs_test_case:[('/dir_1', 1)] =?= [('/dir_1', 1)] 2024-03-06T07:32:16.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:16.981 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 dir_2) 2024-03-06T07:32:16.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:17.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.143+0000 7f1cc1725640 1 Processor -- start 2024-03-06T07:32:17.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.143+0000 7f1cc1725640 1 -- start start 2024-03-06T07:32:17.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cc1725640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cc1725640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cbc0ae9b0 0x7f1cbc0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cc1725640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cbc0b3340 con 0x7f1cbc0b0e90 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cc1725640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cbc0cfa00 con 0x7f1cbc0ae9b0 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cc1725640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc0cee90 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cbb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cbc0ae9b0 0x7f1cbc0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cbb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cbc0ae9b0 0x7f1cbc0d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60582/0 (socket says 172.21.15.27:60582) 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cba7fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1cbc0b0e90 0x7f1cbc06b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:35132/0 (socket says 172.21.15.27:35132) 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cbb7fe640 1 -- 172.21.15.27:0/289566065 learned_addr learned my addr 172.21.15.27:0/289566065 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.144+0000 7f1cbaffd640 1 --2- 172.21.15.27:0/289566065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.145+0000 7f1cbb7fe640 1 -- 172.21.15.27:0/289566065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc06b5c0 msgr2=0x7f1cbc0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.145+0000 7f1cbb7fe640 1 --2- 172.21.15.27:0/289566065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.145+0000 7f1cbb7fe640 1 -- 172.21.15.27:0/289566065 >> v1:172.21.15.69:6789/0 conn(0x7f1cbc0b0e90 legacy=0x7f1cbc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.145+0000 7f1cb9ffb640 1 -- 172.21.15.27:0/289566065 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075019993 0 0) 0x7f1cbc0b3340 con 0x7f1cbc0b0e90 2024-03-06T07:32:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.145+0000 7f1cbb7fe640 1 -- 172.21.15.27:0/289566065 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc0cfc50 con 0x7f1cbc0ae9b0 2024-03-06T07:32:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.146+0000 7f1cbb7fe640 1 --2- 172.21.15.27:0/289566065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc0ae9b0 0x7f1cbc0d03a0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009e50 tx=0x7f1cb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b10c0abc5a8af1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.146+0000 7f1cb9ffb640 1 -- 172.21.15.27:0/289566065 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb402ca70 con 0x7f1cbc0ae9b0 2024-03-06T07:32:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.146+0000 7f1cb9ffb640 1 -- 172.21.15.27:0/289566065 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb402cc10 con 0x7f1cbc0ae9b0 2024-03-06T07:32:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.146+0000 7f1cb9ffb640 1 -- 172.21.15.27:0/289566065 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb40343f0 con 0x7f1cbc0ae9b0 2024-03-06T07:32:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.147+0000 7f1cc1725640 1 -- 172.21.15.27:0/289566065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc0ae9b0 msgr2=0x7f1cbc0d03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.147+0000 7f1cc1725640 1 --2- 172.21.15.27:0/289566065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc0ae9b0 0x7f1cbc0d03a0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009e50 tx=0x7f1cb402bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.147+0000 7f1cc1725640 1 -- 172.21.15.27:0/289566065 shutdown_connections 2024-03-06T07:32:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.147+0000 7f1cc1725640 1 --2- 172.21.15.27:0/289566065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc0ae9b0 0x7f1cbc0d03a0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.147+0000 7f1cc1725640 1 --2- 172.21.15.27:0/289566065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.147+0000 7f1cc1725640 1 -- 172.21.15.27:0/289566065 >> 172.21.15.27:0/289566065 conn(0x7f1cbc0b36d0 msgr2=0x7f1cbc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.147+0000 7f1cc1725640 1 -- 172.21.15.27:0/289566065 shutdown_connections 2024-03-06T07:32:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.147+0000 7f1cc1725640 1 -- 172.21.15.27:0/289566065 wait complete. 2024-03-06T07:32:17.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.149+0000 7f1cc1725640 1 Processor -- start 2024-03-06T07:32:17.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.149+0000 7f1cc1725640 1 -- start start 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cc1725640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cc1725640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cbc0ae9b0 0x7f1cbc0c5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cc1725640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc0b0e90 0x7f1cbc0c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cc1725640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc06c910 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cc1725640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc06c790 con 0x7f1cbc0ae9b0 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cc1725640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc0aee00 con 0x7f1cbc0b0e90 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60594/0 (socket says 172.21.15.27:60594) 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cba7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cbc0ae9b0 0x7f1cbc0c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbaffd640 1 -- 172.21.15.27:0/2331023952 learned_addr learned my addr 172.21.15.27:0/2331023952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbb7fe640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc0b0e90 0x7f1cbc0c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbaffd640 1 -- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc0b0e90 msgr2=0x7f1cbc0c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbaffd640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc0b0e90 0x7f1cbc0c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbaffd640 1 -- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cbc0ae9b0 msgr2=0x7f1cbc0c5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbaffd640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cbc0ae9b0 0x7f1cbc0c5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbaffd640 1 -- 172.21.15.27:0/2331023952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc0cbf30 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.151+0000 7f1cbaffd640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b3d20 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00e9b0 tx=0x7f1cac00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.152+0000 7f1cb37fe640 1 -- 172.21.15.27:0/2331023952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac017070 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.152+0000 7f1cb37fe640 1 -- 172.21.15.27:0/2331023952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac00c9a0 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.152+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cbc1f72b0 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.152+0000 7f1cb37fe640 1 -- 172.21.15.27:0/2331023952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac00cd90 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.152+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cbc1f7870 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.153+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cbc068700 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.153+0000 7f1cb37fe640 1 -- 172.21.15.27:0/2331023952 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cac01a070 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.154+0000 7f1cb37fe640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c9c06b4d0 0x7f1c9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.154+0000 7f1cb37fe640 1 -- 172.21.15.27:0/2331023952 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cac08a450 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.154+0000 7f1cba7fc640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c9c06b4d0 0x7f1c9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.155+0000 7f1cba7fc640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c9c06b4d0 0x7f1c9c06d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc1d2a30 tx=0x7f1ca4007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.158+0000 7f1cb37fe640 1 -- 172.21.15.27:0/2331023952 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1cac058d10 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.378+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cbc1e5bf0 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.378+0000 7f1cb37fe640 1 -- 172.21.15.27:0/2331023952 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+3424 (secure 0 0 0) 0x7f1cac05cdb0 con 0x7f1cbc06b5c0 2024-03-06T07:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:17.381 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":118,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:15.028959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282,9270]},"id":10}]} 2024-03-06T07:32:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 118 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c9c06b4d0 msgr2=0x7f1c9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c9c06b4d0 0x7f1c9c06d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc1d2a30 tx=0x7f1ca4007de0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc06b5c0 msgr2=0x7f1cbc0b3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b3d20 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00e9b0 tx=0x7f1cac00ee70 comp rx=0 tx=0).stop 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 shutdown_connections 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1c9c06b4d0 0x7f1c9c06d990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1cbc0b0e90 0x7f1cbc0c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cbc0ae9b0 0x7f1cbc0c5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 --2- 172.21.15.27:0/2331023952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cbc06b5c0 0x7f1cbc0b3d20 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 >> 172.21.15.27:0/2331023952 conn(0x7f1cbc0b36d0 msgr2=0x7f1cbc1df6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 shutdown_connections 2024-03-06T07:32:17.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:17.380+0000 7f1cc1725640 1 -- 172.21.15.27:0/2331023952 wait complete. 2024-03-06T07:32:17.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:32:17.516 INFO:teuthology.orchestra.run.smithi027.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:32:17.517 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:32:17.517 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:32:19.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.637+0000 7f85fbc08640 1 Processor -- start 2024-03-06T07:32:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.637+0000 7f85fbc08640 1 -- start start 2024-03-06T07:32:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.638+0000 7f85fbc08640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f85f406ee50 0x7f85f406f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.638+0000 7f85fbc08640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f406fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.638+0000 7f85fbc08640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f40d0e00 con 0x7f85f406ca00 2024-03-06T07:32:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.638+0000 7f85fbc08640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f40ccba0 con 0x7f85f406ee50 2024-03-06T07:32:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.638+0000 7f85fbc08640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f40cc270 con 0x7f85f406af20 2024-03-06T07:32:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.638+0000 7f85f99a4640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f85f406ca00 0x7f85f40d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34684/0 (socket says 172.21.15.27:34684) 2024-03-06T07:32:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.638+0000 7f85f99a4640 1 -- 172.21.15.27:0/3087689130 learned_addr learned my addr 172.21.15.27:0/3087689130 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.638+0000 7f85fa1a5640 1 --2- 172.21.15.27:0/3087689130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f406fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.638+0000 7f85f91a3640 1 --2- 172.21.15.27:0/3087689130 >> v2:172.21.15.69:3300/0 conn(0x7f85f406ee50 0x7f85f406f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.639+0000 7f85f89a2640 1 -- 172.21.15.27:0/3087689130 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116905581 0 0) 0x7f85f40d0e00 con 0x7f85f406ca00 2024-03-06T07:32:19.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.639+0000 7f85f89a2640 1 -- 172.21.15.27:0/3087689130 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85dc003520 con 0x7f85f406ca00 2024-03-06T07:32:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.639+0000 7f85f91a3640 1 -- 172.21.15.27:0/3087689130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 msgr2=0x7f85f406fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.639+0000 7f85f91a3640 1 --2- 172.21.15.27:0/3087689130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f406fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.639+0000 7f85f91a3640 1 -- 172.21.15.27:0/3087689130 >> v1:172.21.15.27:6789/0 conn(0x7f85f406ca00 legacy=0x7f85f40d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.639+0000 7f85f91a3640 1 -- 172.21.15.27:0/3087689130 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f40cc550 con 0x7f85f406ee50 2024-03-06T07:32:19.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.639+0000 7f85fa1a5640 1 --2- 172.21.15.27:0/3087689130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f406fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:19.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85f91a3640 1 --2- 172.21.15.27:0/3087689130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85f406ee50 0x7f85f406f570 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f85e80097a0 tx=0x7f85e802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cac8973d436d911 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85f89a2640 1 -- 172.21.15.27:0/3087689130 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8034070 con 0x7f85f406ee50 2024-03-06T07:32:19.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85f89a2640 1 -- 172.21.15.27:0/3087689130 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e8030610 con 0x7f85f406ee50 2024-03-06T07:32:19.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85f89a2640 1 -- 172.21.15.27:0/3087689130 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e80309e0 con 0x7f85f406ee50 2024-03-06T07:32:19.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85fbc08640 1 -- 172.21.15.27:0/3087689130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85f406ee50 msgr2=0x7f85f406f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85fbc08640 1 --2- 172.21.15.27:0/3087689130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85f406ee50 0x7f85f406f570 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f85e80097a0 tx=0x7f85e802b960 comp rx=0 tx=0).stop 2024-03-06T07:32:19.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85fbc08640 1 -- 172.21.15.27:0/3087689130 shutdown_connections 2024-03-06T07:32:19.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85fbc08640 1 --2- 172.21.15.27:0/3087689130 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f406fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85fbc08640 1 --2- 172.21.15.27:0/3087689130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85f406ee50 0x7f85f406f570 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85fbc08640 1 -- 172.21.15.27:0/3087689130 >> 172.21.15.27:0/3087689130 conn(0x7f85f40b35d0 msgr2=0x7f85f40b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85fbc08640 1 -- 172.21.15.27:0/3087689130 shutdown_connections 2024-03-06T07:32:19.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.640+0000 7f85fbc08640 1 -- 172.21.15.27:0/3087689130 wait complete. 2024-03-06T07:32:19.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fbc08640 1 Processor -- start 2024-03-06T07:32:19.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fbc08640 1 -- start start 2024-03-06T07:32:19.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fbc08640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f40cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fbc08640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85f406ca00 0x7f85f41f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fbc08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f406ee50 0x7f85f40b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fbc08640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f415d6f0 con 0x7f85f406ee50 2024-03-06T07:32:19.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fbc08640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f406cde0 con 0x7f85f406ca00 2024-03-06T07:32:19.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fbc08640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f415d870 con 0x7f85f406af20 2024-03-06T07:32:19.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fa1a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f406ee50 0x7f85f40b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85f99a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f40cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85f91a3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85f406ca00 0x7f85f41f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fa1a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f406ee50 0x7f85f40b5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60606/0 (socket says 172.21.15.27:60606) 2024-03-06T07:32:19.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.642+0000 7f85fa1a5640 1 -- 172.21.15.27:0/1384989239 learned_addr learned my addr 172.21.15.27:0/1384989239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:19.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85fa1a5640 1 -- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 msgr2=0x7f85f40cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85fa1a5640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f40cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85fa1a5640 1 -- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85f406ca00 msgr2=0x7f85f41f7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85fa1a5640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85f406ca00 0x7f85f41f7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85fa1a5640 1 -- 172.21.15.27:0/1384989239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f40cb990 con 0x7f85f406ee50 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85f99a4640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f40cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85fa1a5640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f406ee50 0x7f85f40b5bf0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f85e400e8c0 tx=0x7f85e400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85f27fc640 1 -- 172.21.15.27:0/1384989239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e4017070 con 0x7f85f406ee50 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85f27fc640 1 -- 172.21.15.27:0/1384989239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e400c940 con 0x7f85f406ee50 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85f27fc640 1 -- 172.21.15.27:0/1384989239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e400cd10 con 0x7f85f406ee50 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f40c8c80 con 0x7f85f406ee50 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.643+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f40c8f20 con 0x7f85f406ee50 2024-03-06T07:32:19.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.644+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f40b3350 con 0x7f85f406ee50 2024-03-06T07:32:19.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.645+0000 7f85f27fc640 1 -- 172.21.15.27:0/1384989239 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85e401a070 con 0x7f85f406ee50 2024-03-06T07:32:19.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.646+0000 7f85f27fc640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.646+0000 7f85f27fc640 1 -- 172.21.15.27:0/1384989239 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85e408a000 con 0x7f85f406ee50 2024-03-06T07:32:19.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.646+0000 7f85f99a4640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.647+0000 7f85f99a4640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85e006b4d0 0x7f85e006d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f85ec005c00 tx=0x7f85ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.651+0000 7f85f27fc640 1 -- 172.21.15.27:0/1384989239 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85e40588c0 con 0x7f85f406ee50 2024-03-06T07:32:19.744 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:19.746 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:19.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.873+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85f4068700 con 0x7f85f406ee50 2024-03-06T07:32:19.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.873+0000 7f85f27fc640 1 -- 172.21.15.27:0/1384989239 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+3424 (secure 0 0 0) 0x7f85e405c960 con 0x7f85f406ee50 2024-03-06T07:32:19.876 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:19.876 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":118,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:15.028959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282,9270]},"id":10}]} 2024-03-06T07:32:19.876 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 118 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85e006b4d0 msgr2=0x7f85e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85e006b4d0 0x7f85e006d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f85ec005c00 tx=0x7f85ec009040 comp rx=0 tx=0).stop 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f406ee50 msgr2=0x7f85f40b5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f406ee50 0x7f85f40b5bf0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f85e400e8c0 tx=0x7f85e400ed80 comp rx=0 tx=0).stop 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 shutdown_connections 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f406ee50 0x7f85f40b5bf0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85f406ca00 0x7f85f41f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 --2- 172.21.15.27:0/1384989239 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85f406af20 0x7f85f40cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 >> 172.21.15.27:0/1384989239 conn(0x7f85f40b35d0 msgr2=0x7f85f415cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 shutdown_connections 2024-03-06T07:32:19.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:19.875+0000 7f85fbc08640 1 -- 172.21.15.27:0/1384989239 wait complete. 2024-03-06T07:32:19.888 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:32:20.010 INFO:teuthology.orchestra.run.smithi027.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:32:20.010 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:32:20.010 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:32:22.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.135+0000 7ff5fb684640 1 Processor -- start 2024-03-06T07:32:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.135+0000 7ff5fb684640 1 -- start start 2024-03-06T07:32:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5fb684640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f40b1d50 0x7ff5f40b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5fb684640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff5f406d210 0x7ff5f406d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5fb684640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f415e410 con 0x7ff5f415e030 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5fb684640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f40b26f0 con 0x7ff5f406d210 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5fb684640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f4162700 con 0x7ff5f40b1d50 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5f9c21640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff5f406d210 0x7ff5f406d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5f9420640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff5f415e030 0x7ff5f406ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34694/0 (socket says 172.21.15.27:34694) 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5f9420640 1 -- 172.21.15.27:0/2612885495 learned_addr learned my addr 172.21.15.27:0/2612885495 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5f9c21640 1 --2- 172.21.15.27:0/2612885495 >> v2:172.21.15.27:3300/0 conn(0x7ff5f406d210 0x7ff5f406d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60610/0 (socket says 172.21.15.27:60610) 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5f8c1f640 1 --2- 172.21.15.27:0/2612885495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f40b1d50 0x7ff5f40b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5f9c21640 1 -- 172.21.15.27:0/2612885495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f40b1d50 msgr2=0x7ff5f40b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5f9c21640 1 --2- 172.21.15.27:0/2612885495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f40b1d50 0x7ff5f40b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5f9c21640 1 -- 172.21.15.27:0/2612885495 >> v1:172.21.15.27:6789/0 conn(0x7ff5f415e030 legacy=0x7ff5f406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5f9c21640 1 -- 172.21.15.27:0/2612885495 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f406f140 con 0x7ff5f406d210 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.136+0000 7ff5ebfff640 1 -- 172.21.15.27:0/2612885495 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450813613 0 0) 0x7ff5f0001110 con 0x7ff5f415e030 2024-03-06T07:32:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.137+0000 7ff5f9c21640 1 --2- 172.21.15.27:0/2612885495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f406d210 0x7ff5f406d690 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009d90 tx=0x7ff5ec02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b07fb8806070358 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.137+0000 7ff5ebfff640 1 -- 172.21.15.27:0/2612885495 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec039070 con 0x7ff5f406d210 2024-03-06T07:32:22.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.137+0000 7ff5ebfff640 1 -- 172.21.15.27:0/2612885495 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec02cc50 con 0x7ff5f406d210 2024-03-06T07:32:22.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.137+0000 7ff5ebfff640 1 -- 172.21.15.27:0/2612885495 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec034440 con 0x7ff5f406d210 2024-03-06T07:32:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.138+0000 7ff5fb684640 1 -- 172.21.15.27:0/2612885495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f406d210 msgr2=0x7ff5f406d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.138+0000 7ff5fb684640 1 --2- 172.21.15.27:0/2612885495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f406d210 0x7ff5f406d690 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009d90 tx=0x7ff5ec02bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.138+0000 7ff5fb684640 1 -- 172.21.15.27:0/2612885495 shutdown_connections 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.138+0000 7ff5fb684640 1 --2- 172.21.15.27:0/2612885495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f406d210 0x7ff5f406d690 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.138+0000 7ff5fb684640 1 --2- 172.21.15.27:0/2612885495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f40b1d50 0x7ff5f40b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.138+0000 7ff5fb684640 1 -- 172.21.15.27:0/2612885495 >> 172.21.15.27:0/2612885495 conn(0x7ff5f40b55e0 msgr2=0x7ff5f40b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.138+0000 7ff5fb684640 1 -- 172.21.15.27:0/2612885495 shutdown_connections 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.138+0000 7ff5fb684640 1 -- 172.21.15.27:0/2612885495 wait complete. 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.139+0000 7ff5fb684640 1 Processor -- start 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.139+0000 7ff5fb684640 1 -- start start 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5fb684640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f406d210 0x7ff5f4155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5fb684640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f40b1d50 0x7ff5f415add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5fb684640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f415e030 0x7ff5f415b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5fb684640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f41620b0 con 0x7ff5f40b1d50 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5fb684640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f4161dd0 con 0x7ff5f406d210 2024-03-06T07:32:22.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5fb684640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f4162390 con 0x7ff5f415e030 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f8c1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f40b1d50 0x7ff5f415add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f8c1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f40b1d50 0x7ff5f415add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60626/0 (socket says 172.21.15.27:60626) 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f8c1f640 1 -- 172.21.15.27:0/3810864244 learned_addr learned my addr 172.21.15.27:0/3810864244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f9420640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f406d210 0x7ff5f4155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f9c21640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f415e030 0x7ff5f415b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f8c1f640 1 -- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f415e030 msgr2=0x7ff5f415b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f8c1f640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f415e030 0x7ff5f415b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f8c1f640 1 -- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f406d210 msgr2=0x7ff5f4155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f8c1f640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f406d210 0x7ff5f4155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f8c1f640 1 -- 172.21.15.27:0/3810864244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f4156040 con 0x7ff5f40b1d50 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.140+0000 7ff5f8c1f640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f40b1d50 0x7ff5f415add0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400e9b0 tx=0x7ff5e400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.141+0000 7ff5e9ffb640 1 -- 172.21.15.27:0/3810864244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4017070 con 0x7ff5f40b1d50 2024-03-06T07:32:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.141+0000 7ff5e9ffb640 1 -- 172.21.15.27:0/3810864244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e400c9a0 con 0x7ff5f40b1d50 2024-03-06T07:32:22.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.141+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f415c5d0 con 0x7ff5f40b1d50 2024-03-06T07:32:22.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.141+0000 7ff5e9ffb640 1 -- 172.21.15.27:0/3810864244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e400cd70 con 0x7ff5f40b1d50 2024-03-06T07:32:22.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.141+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f415bd30 con 0x7ff5f40b1d50 2024-03-06T07:32:22.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.142+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f4067ff0 con 0x7ff5f40b1d50 2024-03-06T07:32:22.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.142+0000 7ff5e9ffb640 1 -- 172.21.15.27:0/3810864244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5e401a070 con 0x7ff5f40b1d50 2024-03-06T07:32:22.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.143+0000 7ff5e9ffb640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5d806b4d0 0x7ff5d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.143+0000 7ff5e9ffb640 1 -- 172.21.15.27:0/3810864244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5e408a1a0 con 0x7ff5f40b1d50 2024-03-06T07:32:22.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.143+0000 7ff5f9420640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5d806b4d0 0x7ff5d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.143+0000 7ff5f9420640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5d806b4d0 0x7ff5d806d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0000f60 tx=0x7ff5f0007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.147+0000 7ff5e9ffb640 1 -- 172.21.15.27:0/3810864244 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5e40589d0 con 0x7ff5f40b1d50 2024-03-06T07:32:22.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.366+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5f40b1030 con 0x7ff5f40b1d50 2024-03-06T07:32:22.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.366+0000 7ff5e9ffb640 1 -- 172.21.15.27:0/3810864244 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+3424 (secure 0 0 0) 0x7ff5e405ca70 con 0x7ff5f40b1d50 2024-03-06T07:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:22.369 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":118,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:15.028959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282,9270]},"id":10}]} 2024-03-06T07:32:22.369 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 118 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5d806b4d0 msgr2=0x7ff5d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5d806b4d0 0x7ff5d806d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0000f60 tx=0x7ff5f0007b50 comp rx=0 tx=0).stop 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f40b1d50 msgr2=0x7ff5f415add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f40b1d50 0x7ff5f415add0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400e9b0 tx=0x7ff5e400ee70 comp rx=0 tx=0).stop 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 shutdown_connections 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff5d806b4d0 0x7ff5d806d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff5f415e030 0x7ff5f415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f40b1d50 0x7ff5f415add0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 --2- 172.21.15.27:0/3810864244 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5f406d210 0x7ff5f4155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 >> 172.21.15.27:0/3810864244 conn(0x7ff5f40b55e0 msgr2=0x7ff5f40ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 shutdown_connections 2024-03-06T07:32:22.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:22.368+0000 7ff5fb684640 1 -- 172.21.15.27:0/3810864244 wait complete. 2024-03-06T07:32:22.382 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:32:22.540 INFO:teuthology.orchestra.run.smithi027.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:32:22.540 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:32:22.540 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:32:24.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:24.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.677+0000 7f683b932640 1 Processor -- start 2024-03-06T07:32:24.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.677+0000 7f683b932640 1 -- start start 2024-03-06T07:32:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.678+0000 7f683b932640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f683415df00 0x7f68340d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.678+0000 7f683b932640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68340b95a0 0x7f68340b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.678+0000 7f683b932640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68340cea30 con 0x7f68340b9a30 2024-03-06T07:32:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.678+0000 7f683b932640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68340d2af0 con 0x7f68340b95a0 2024-03-06T07:32:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.678+0000 7f683b932640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68340cf3c0 con 0x7f683415df00 2024-03-06T07:32:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.678+0000 7f6839ecf640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f68340b9a30 0x7f68340d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53482/0 (socket says 172.21.15.27:53482) 2024-03-06T07:32:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.678+0000 7f6839ecf640 1 -- 172.21.15.27:0/2311964982 learned_addr learned my addr 172.21.15.27:0/2311964982 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:24.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.678+0000 7f6838ecd640 1 --2- 172.21.15.27:0/2311964982 >> v2:172.21.15.69:3300/0 conn(0x7f68340b95a0 0x7f68340b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.679+0000 7f68396ce640 1 --2- 172.21.15.27:0/2311964982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f683415df00 0x7f68340d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.679+0000 7f682bfff640 1 -- 172.21.15.27:0/2311964982 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534084464 0 0) 0x7f68340cea30 con 0x7f68340b9a30 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.679+0000 7f682bfff640 1 -- 172.21.15.27:0/2311964982 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6820003520 con 0x7f68340b9a30 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.679+0000 7f6838ecd640 1 -- 172.21.15.27:0/2311964982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f683415df00 msgr2=0x7f68340d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.679+0000 7f6838ecd640 1 --2- 172.21.15.27:0/2311964982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f683415df00 0x7f68340d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.679+0000 7f6838ecd640 1 -- 172.21.15.27:0/2311964982 >> v1:172.21.15.27:6789/0 conn(0x7f68340b9a30 legacy=0x7f68340d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.679+0000 7f6838ecd640 1 -- 172.21.15.27:0/2311964982 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68340cf0e0 con 0x7f68340b95a0 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f6838ecd640 1 --2- 172.21.15.27:0/2311964982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 0x7f68340b5700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f68300097a0 tx=0x7f683002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86743999571446af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f682bfff640 1 -- 172.21.15.27:0/2311964982 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830035020 con 0x7f68340b95a0 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f682bfff640 1 -- 172.21.15.27:0/2311964982 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6830031830 con 0x7f68340b95a0 2024-03-06T07:32:24.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f682bfff640 1 -- 172.21.15.27:0/2311964982 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830031c00 con 0x7f68340b95a0 2024-03-06T07:32:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f683b932640 1 -- 172.21.15.27:0/2311964982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 msgr2=0x7f68340b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f683b932640 1 --2- 172.21.15.27:0/2311964982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 0x7f68340b5700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f68300097a0 tx=0x7f683002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f683b932640 1 -- 172.21.15.27:0/2311964982 shutdown_connections 2024-03-06T07:32:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f683b932640 1 --2- 172.21.15.27:0/2311964982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 0x7f68340b5700 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f683b932640 1 --2- 172.21.15.27:0/2311964982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f683415df00 0x7f68340d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.680+0000 7f683b932640 1 -- 172.21.15.27:0/2311964982 >> 172.21.15.27:0/2311964982 conn(0x7f6834071960 msgr2=0x7f6834071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.681+0000 7f683b932640 1 -- 172.21.15.27:0/2311964982 shutdown_connections 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.681+0000 7f683b932640 1 -- 172.21.15.27:0/2311964982 wait complete. 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.682+0000 7f683b932640 1 Processor -- start 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.682+0000 7f683b932640 1 -- start start 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.682+0000 7f683b932640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 0x7f6834150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.682+0000 7f683b932640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683415df00 0x7f6834160ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.682+0000 7f683b932640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68340c9020 0x7f68340d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.682+0000 7f683b932640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68340cec70 con 0x7f683415df00 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.682+0000 7f683b932640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68340ce750 con 0x7f68340b95a0 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.682+0000 7f683b932640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68340cedf0 con 0x7f68340c9020 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.682+0000 7f6838ecd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683415df00 0x7f6834160ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6838ecd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683415df00 0x7f6834160ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46870/0 (socket says 172.21.15.27:46870) 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6838ecd640 1 -- 172.21.15.27:0/3996312947 learned_addr learned my addr 172.21.15.27:0/3996312947 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f68396ce640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 0x7f6834150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6839ecf640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68340c9020 0x7f68340d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6838ecd640 1 -- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68340c9020 msgr2=0x7f68340d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6838ecd640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68340c9020 0x7f68340d39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6838ecd640 1 -- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 msgr2=0x7f6834150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6838ecd640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 0x7f6834150fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6838ecd640 1 -- 172.21.15.27:0/3996312947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6834151500 con 0x7f683415df00 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f68396ce640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 0x7f6834150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6838ecd640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683415df00 0x7f6834160ed0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f683002bee0 tx=0x7f68300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6829ffb640 1 -- 172.21.15.27:0/3996312947 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830035040 con 0x7f683415df00 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6829ffb640 1 -- 172.21.15.27:0/3996312947 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683003b920 con 0x7f683415df00 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68340c7cb0 con 0x7f683415df00 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.683+0000 7f6829ffb640 1 -- 172.21.15.27:0/3996312947 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683003bd10 con 0x7f683415df00 2024-03-06T07:32:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.684+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68341603f0 con 0x7f683415df00 2024-03-06T07:32:24.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.684+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6834067ff0 con 0x7f683415df00 2024-03-06T07:32:24.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.685+0000 7f6829ffb640 1 -- 172.21.15.27:0/3996312947 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6830038020 con 0x7f683415df00 2024-03-06T07:32:24.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.685+0000 7f6829ffb640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f681806b4d0 0x7f681806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.686+0000 7f6829ffb640 1 -- 172.21.15.27:0/3996312947 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68300ac1a0 con 0x7f683415df00 2024-03-06T07:32:24.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.686+0000 7f68396ce640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f681806b4d0 0x7f681806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.686+0000 7f68396ce640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f681806b4d0 0x7f681806d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6824009e90 tx=0x7f6824009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.690+0000 7f6829ffb640 1 -- 172.21.15.27:0/3996312947 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f683007a9d0 con 0x7f683415df00 2024-03-06T07:32:24.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.908+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68340cd740 con 0x7f683415df00 2024-03-06T07:32:24.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.909+0000 7f6829ffb640 1 -- 172.21.15.27:0/3996312947 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+3424 (secure 0 0 0) 0x7f683007ea70 con 0x7f683415df00 2024-03-06T07:32:24.910 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:24.910 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":118,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:15.028959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282,9270]},"id":10}]} 2024-03-06T07:32:24.911 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 118 2024-03-06T07:32:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.911+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f681806b4d0 msgr2=0x7f681806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.911+0000 7f683b932640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f681806b4d0 0x7f681806d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6824009e90 tx=0x7f6824009190 comp rx=0 tx=0).stop 2024-03-06T07:32:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.911+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683415df00 msgr2=0x7f6834160ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.911+0000 7f683b932640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683415df00 0x7f6834160ed0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f683002bee0 tx=0x7f68300046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.912+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 shutdown_connections 2024-03-06T07:32:24.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.912+0000 7f683b932640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f681806b4d0 0x7f681806d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.912+0000 7f683b932640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f68340c9020 0x7f68340d39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.912+0000 7f683b932640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683415df00 0x7f6834160ed0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.912+0000 7f683b932640 1 --2- 172.21.15.27:0/3996312947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68340b95a0 0x7f6834150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.912+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 >> 172.21.15.27:0/3996312947 conn(0x7f6834071960 msgr2=0x7f683415bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.912+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 shutdown_connections 2024-03-06T07:32:24.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:24.912+0000 7f683b932640 1 -- 172.21.15.27:0/3996312947 wait complete. 2024-03-06T07:32:24.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:32:25.055 INFO:teuthology.orchestra.run.smithi027.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/dir_2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-06T07:32:25.056 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/dir_2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:32:25.056 INFO:tasks.cephfs.cephfs_test_case:[('/dir_2', 0)] =?= [('/dir_2', 0)] 2024-03-06T07:32:25.056 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:32:25.056 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:25.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:25.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.216+0000 7f4a73b81640 1 Processor -- start 2024-03-06T07:32:25.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.217+0000 7f4a73b81640 1 -- start start 2024-03-06T07:32:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.217+0000 7f4a73b81640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a6c0d3400 0x7f4a6c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.217+0000 7f4a73b81640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4a6c0b2b50 0x7f4a6c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.217+0000 7f4a73b81640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6c0ce610 con 0x7f4a6c071540 2024-03-06T07:32:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.217+0000 7f4a73b81640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6c0d2f70 con 0x7f4a6c0d3400 2024-03-06T07:32:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.217+0000 7f4a73b81640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6c0d4be0 con 0x7f4a6c0b2b50 2024-03-06T07:32:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.218+0000 7f4a7111c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a6c0d3400 0x7f4a6c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.218+0000 7f4a7191d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a6c071540 0x7f4a6c071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53486/0 (socket says 172.21.15.27:53486) 2024-03-06T07:32:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.218+0000 7f4a7191d640 1 -- 172.21.15.27:0/2907135266 learned_addr learned my addr 172.21.15.27:0/2907135266 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.218+0000 7f4a7211e640 1 --2- 172.21.15.27:0/2907135266 >> v2:172.21.15.69:3300/0 conn(0x7f4a6c0b2b50 0x7f4a6c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.218+0000 7f4a7111c640 1 -- 172.21.15.27:0/2907135266 >> v2:172.21.15.69:3300/0 conn(0x7f4a6c0b2b50 msgr2=0x7f4a6c0b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.218+0000 7f4a7111c640 1 --2- 172.21.15.27:0/2907135266 >> v2:172.21.15.69:3300/0 conn(0x7f4a6c0b2b50 0x7f4a6c0b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.218+0000 7f4a7111c640 1 -- 172.21.15.27:0/2907135266 >> v1:172.21.15.27:6789/0 conn(0x7f4a6c071540 legacy=0x7f4a6c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.218+0000 7f4a7111c640 1 -- 172.21.15.27:0/2907135266 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c0d2920 con 0x7f4a6c0d3400 2024-03-06T07:32:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a7111c640 1 --2- 172.21.15.27:0/2907135266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 0x7f4a6c0b30a0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0683b0 tx=0x7f4a5c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16f9e3d483fa76a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a7091b640 1 -- 172.21.15.27:0/2907135266 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c035020 con 0x7f4a6c0d3400 2024-03-06T07:32:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a7091b640 1 -- 172.21.15.27:0/2907135266 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5c031830 con 0x7f4a6c0d3400 2024-03-06T07:32:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a7091b640 1 -- 172.21.15.27:0/2907135266 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c031c00 con 0x7f4a6c0d3400 2024-03-06T07:32:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a73b81640 1 -- 172.21.15.27:0/2907135266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 msgr2=0x7f4a6c0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a73b81640 1 --2- 172.21.15.27:0/2907135266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 0x7f4a6c0b30a0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0683b0 tx=0x7f4a5c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a73b81640 1 -- 172.21.15.27:0/2907135266 shutdown_connections 2024-03-06T07:32:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a73b81640 1 --2- 172.21.15.27:0/2907135266 >> v2:172.21.15.69:3300/0 conn(0x7f4a6c0b2b50 0x7f4a6c0b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a73b81640 1 --2- 172.21.15.27:0/2907135266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 0x7f4a6c0b30a0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.219+0000 7f4a73b81640 1 -- 172.21.15.27:0/2907135266 >> 172.21.15.27:0/2907135266 conn(0x7f4a6c0d3ca0 msgr2=0x7f4a6c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.220+0000 7f4a73b81640 1 -- 172.21.15.27:0/2907135266 shutdown_connections 2024-03-06T07:32:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.220+0000 7f4a73b81640 1 -- 172.21.15.27:0/2907135266 wait complete. 2024-03-06T07:32:25.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.221+0000 7f4a73b81640 1 Processor -- start 2024-03-06T07:32:25.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.221+0000 7f4a73b81640 1 -- start start 2024-03-06T07:32:25.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.221+0000 7f4a73b81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a6c071540 0x7f4a6c0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.221+0000 7f4a73b81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6c0b2b50 0x7f4a6c14e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.221+0000 7f4a73b81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 0x7f4a6c1f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.221+0000 7f4a73b81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c15e370 con 0x7f4a6c0d3400 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.221+0000 7f4a73b81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0ce8f0 con 0x7f4a6c0b2b50 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.221+0000 7f4a73b81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c14e720 con 0x7f4a6c071540 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7211e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 0x7f4a6c1f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7211e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 0x7f4a6c1f3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46888/0 (socket says 172.21.15.27:46888) 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7211e640 1 -- 172.21.15.27:0/3132755933 learned_addr learned my addr 172.21.15.27:0/3132755933 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7111c640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6c0b2b50 0x7f4a6c14e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7191d640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a6c071540 0x7f4a6c0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7211e640 1 -- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a6c071540 msgr2=0x7f4a6c0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7211e640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a6c071540 0x7f4a6c0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7211e640 1 -- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6c0b2b50 msgr2=0x7f4a6c14e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7211e640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6c0b2b50 0x7f4a6c14e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7211e640 1 -- 172.21.15.27:0/3132755933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c0cb400 con 0x7f4a6c0d3400 2024-03-06T07:32:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7111c640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6c0b2b50 0x7f4a6c14e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a7211e640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 0x7f4a6c1f3df0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400efc0 tx=0x7f4a6400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a627fc640 1 -- 172.21.15.27:0/3132755933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64018020 con 0x7f4a6c0d3400 2024-03-06T07:32:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a627fc640 1 -- 172.21.15.27:0/3132755933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a64014480 con 0x7f4a6c0d3400 2024-03-06T07:32:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c0cb6e0 con 0x7f4a6c0d3400 2024-03-06T07:32:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a627fc640 1 -- 172.21.15.27:0/3132755933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64014870 con 0x7f4a6c0d3400 2024-03-06T07:32:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.222+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c0cbca0 con 0x7f4a6c0d3400 2024-03-06T07:32:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.223+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c072140 con 0x7f4a6c0d3400 2024-03-06T07:32:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.224+0000 7f4a627fc640 1 -- 172.21.15.27:0/3132755933 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a64005d10 con 0x7f4a6c0d3400 2024-03-06T07:32:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.224+0000 7f4a627fc640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a5006b500 0x7f4a5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.224+0000 7f4a627fc640 1 -- 172.21.15.27:0/3132755933 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a6408aac0 con 0x7f4a6c0d3400 2024-03-06T07:32:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.224+0000 7f4a7191d640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a5006b500 0x7f4a5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.225+0000 7f4a7191d640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a5006b500 0x7f4a5006d9c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f4a68007fd0 tx=0x7f4a68007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.229+0000 7f4a627fc640 1 -- 172.21.15.27:0/3132755933 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a6c072140 con 0x7f4a6c0d3400 2024-03-06T07:32:25.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.474+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a6c071d70 con 0x7f4a6c0d3400 2024-03-06T07:32:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.475+0000 7f4a627fc640 1 -- 172.21.15.27:0/3132755933 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+16607 (secure 0 0 0) 0x7f4a6c071d70 con 0x7f4a6c0d3400 2024-03-06T07:32:25.476 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:25.477 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":147,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:32:05.629029+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a5006b500 msgr2=0x7f4a5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a5006b500 0x7f4a5006d9c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f4a68007fd0 tx=0x7f4a68007e80 comp rx=0 tx=0).stop 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 msgr2=0x7f4a6c1f3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 0x7f4a6c1f3df0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400efc0 tx=0x7f4a6400c490 comp rx=0 tx=0).stop 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 shutdown_connections 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4a5006b500 0x7f4a5006d9c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a6c0d3400 0x7f4a6c1f3df0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a6c0b2b50 0x7f4a6c14e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 --2- 172.21.15.27:0/3132755933 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4a6c071540 0x7f4a6c0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 >> 172.21.15.27:0/3132755933 conn(0x7f4a6c0d3ca0 msgr2=0x7f4a6c0d4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 shutdown_connections 2024-03-06T07:32:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.477+0000 7f4a73b81640 1 -- 172.21.15.27:0/3132755933 wait complete. 2024-03-06T07:32:25.489 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:42.692383+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:43.367948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:25.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.598+0000 7f50a98f4640 1 Processor -- start 2024-03-06T07:32:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.598+0000 7f50a98f4640 1 -- start start 2024-03-06T07:32:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.599+0000 7f50a98f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4071540 0x7f50a4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.599+0000 7f50a98f4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50a40b2d30 0x7f50a40ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.599+0000 7f50a98f4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a4071e60 con 0x7f50a40cecb0 2024-03-06T07:32:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.599+0000 7f50a98f4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a40d3230 con 0x7f50a40b2d30 2024-03-06T07:32:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.599+0000 7f50a98f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a40d4960 con 0x7f50a4071540 2024-03-06T07:32:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.599+0000 7f50a27fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50a40b2d30 0x7f50a40ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.599+0000 7f50a37fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f50a40cecb0 0x7f50a40b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53500/0 (socket says 172.21.15.27:53500) 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.599+0000 7f50a37fe640 1 -- 172.21.15.27:0/406048973 learned_addr learned my addr 172.21.15.27:0/406048973 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.599+0000 7f50a2ffd640 1 --2- 172.21.15.27:0/406048973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4071540 0x7f50a4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.600+0000 7f50a27fc640 1 -- 172.21.15.27:0/406048973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4071540 msgr2=0x7f50a4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.600+0000 7f50a27fc640 1 --2- 172.21.15.27:0/406048973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4071540 0x7f50a4071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.600+0000 7f50a27fc640 1 -- 172.21.15.27:0/406048973 >> v1:172.21.15.27:6789/0 conn(0x7f50a40cecb0 legacy=0x7f50a40b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.600+0000 7f50a27fc640 1 -- 172.21.15.27:0/406048973 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a40d36c0 con 0x7f50a40b2d30 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.600+0000 7f50a1ffb640 1 -- 172.21.15.27:0/406048973 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646054344 0 0) 0x7f50a4071e60 con 0x7f50a40cecb0 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.600+0000 7f50a2ffd640 1 --2- 172.21.15.27:0/406048973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4071540 0x7f50a4071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.600+0000 7f50a27fc640 1 --2- 172.21.15.27:0/406048973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 0x7f50a40ce770 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f50940097a0 tx=0x7f509402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe52f2e1f67f581 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a1ffb640 1 -- 172.21.15.27:0/406048973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094035020 con 0x7f50a40b2d30 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a1ffb640 1 -- 172.21.15.27:0/406048973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5094031810 con 0x7f50a40b2d30 2024-03-06T07:32:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a1ffb640 1 -- 172.21.15.27:0/406048973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094031be0 con 0x7f50a40b2d30 2024-03-06T07:32:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a98f4640 1 -- 172.21.15.27:0/406048973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 msgr2=0x7f50a40ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a98f4640 1 --2- 172.21.15.27:0/406048973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 0x7f50a40ce770 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f50940097a0 tx=0x7f509402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a98f4640 1 -- 172.21.15.27:0/406048973 shutdown_connections 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a98f4640 1 --2- 172.21.15.27:0/406048973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 0x7f50a40ce770 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a98f4640 1 --2- 172.21.15.27:0/406048973 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a4071540 0x7f50a4071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a98f4640 1 -- 172.21.15.27:0/406048973 >> 172.21.15.27:0/406048973 conn(0x7f50a40d3d60 msgr2=0x7f50a40d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a98f4640 1 -- 172.21.15.27:0/406048973 shutdown_connections 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.601+0000 7f50a98f4640 1 -- 172.21.15.27:0/406048973 wait complete. 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.602+0000 7f50a98f4640 1 Processor -- start 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a98f4640 1 -- start start 2024-03-06T07:32:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a98f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a4071540 0x7f50a4156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a98f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 0x7f50a414e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a98f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a40cecb0 0x7f50a4158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a98f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a40d2be0 con 0x7f50a40b2d30 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a98f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a40d2900 con 0x7f50a4071540 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a98f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a40d2ec0 con 0x7f50a40cecb0 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 0x7f50a414e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 0x7f50a414e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46908/0 (socket says 172.21.15.27:46908) 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.603+0000 7f50a27fc640 1 -- 172.21.15.27:0/676504658 learned_addr learned my addr 172.21.15.27:0/676504658 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a2ffd640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a4071540 0x7f50a4156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a37fe640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a40cecb0 0x7f50a4158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a27fc640 1 -- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a40cecb0 msgr2=0x7f50a4158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a27fc640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a40cecb0 0x7f50a4158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a27fc640 1 -- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a4071540 msgr2=0x7f50a4156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a27fc640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a4071540 0x7f50a4156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a27fc640 1 -- 172.21.15.27:0/676504658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a4157f70 con 0x7f50a40b2d30 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a2ffd640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a4071540 0x7f50a4156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a27fc640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 0x7f50a414e1a0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f5094000c00 tx=0x7f50940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f508bfff640 1 -- 172.21.15.27:0/676504658 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094035050 con 0x7f50a40b2d30 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f508bfff640 1 -- 172.21.15.27:0/676504658 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509403bcf0 con 0x7f50a40b2d30 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f508bfff640 1 -- 172.21.15.27:0/676504658 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094006e10 con 0x7f50a40b2d30 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a4158250 con 0x7f50a40b2d30 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.604+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a40cb2e0 con 0x7f50a40b2d30 2024-03-06T07:32:25.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.605+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a40cb0e0 con 0x7f50a40b2d30 2024-03-06T07:32:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.606+0000 7f508bfff640 1 -- 172.21.15.27:0/676504658 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5094038020 con 0x7f50a40b2d30 2024-03-06T07:32:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.606+0000 7f508bfff640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f509006b500 0x7f509006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.606+0000 7f508bfff640 1 -- 172.21.15.27:0/676504658 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50940ac0d0 con 0x7f50a40b2d30 2024-03-06T07:32:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.607+0000 7f50a2ffd640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f509006b500 0x7f509006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.607+0000 7f50a2ffd640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f509006b500 0x7f509006d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5098009560 tx=0x7f5098009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.616+0000 7f508bfff640 1 -- 172.21.15.27:0/676504658 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5094040070 con 0x7f50a40b2d30 2024-03-06T07:32:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.797+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f50a4072070 con 0x7f50a40b2d30 2024-03-06T07:32:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.797+0000 7f508bfff640 1 -- 172.21.15.27:0/676504658 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) v1 ==== 93+0+10 (secure 0 0 0) 0x7f509407a900 con 0x7f50a40b2d30 2024-03-06T07:32:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f509006b500 msgr2=0x7f509006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f509006b500 0x7f509006d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5098009560 tx=0x7f5098009190 comp rx=0 tx=0).stop 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 msgr2=0x7f50a414e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 0x7f50a414e1a0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f5094000c00 tx=0x7f50940046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 shutdown_connections 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f509006b500 0x7f509006d9c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f50a40cecb0 0x7f50a4158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50a40b2d30 0x7f50a414e1a0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 --2- 172.21.15.27:0/676504658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50a4071540 0x7f50a4156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 >> 172.21.15.27:0/676504658 conn(0x7f50a40d3d60 msgr2=0x7f50a415cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 shutdown_connections 2024-03-06T07:32:25.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.800+0000 7f50a98f4640 1 -- 172.21.15.27:0/676504658 wait complete. 2024-03-06T07:32:25.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.926+0000 7f0896681640 1 Processor -- start 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.926+0000 7f0896681640 1 -- start start 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.927+0000 7f0896681640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f089015dd90 0x7f08900ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.927+0000 7f0896681640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08900b6d10 0x7f08900b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.927+0000 7f0896681640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08900d3740 con 0x7f08900d29d0 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.927+0000 7f0896681640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08900d3400 con 0x7f089015dd90 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.927+0000 7f0896681640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08900d3120 con 0x7f08900b6d10 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.927+0000 7f0894c1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08900b6d10 0x7f08900b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.927+0000 7f088f7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f08900d29d0 0x7f08900d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:50548/0 (socket says 172.21.15.27:50548) 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.927+0000 7f088ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f089015dd90 0x7f08900ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.927+0000 7f088f7fe640 1 -- 172.21.15.27:0/1254531435 learned_addr learned my addr 172.21.15.27:0/1254531435 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.928+0000 7f088effd640 1 -- 172.21.15.27:0/1254531435 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653375749 0 0) 0x7f08900d3740 con 0x7f08900d29d0 2024-03-06T07:32:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.928+0000 7f0894c1e640 1 -- 172.21.15.27:0/1254531435 >> v2:172.21.15.69:3300/0 conn(0x7f089015dd90 msgr2=0x7f08900ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.928+0000 7f0894c1e640 1 --2- 172.21.15.27:0/1254531435 >> v2:172.21.15.69:3300/0 conn(0x7f089015dd90 0x7f08900ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.928+0000 7f0894c1e640 1 -- 172.21.15.27:0/1254531435 >> v1:172.21.15.69:6789/0 conn(0x7f08900d29d0 legacy=0x7f08900d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.928+0000 7f0894c1e640 1 -- 172.21.15.27:0/1254531435 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08900d39e0 con 0x7f08900b6d10 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.928+0000 7f0894c1e640 1 --2- 172.21.15.27:0/1254531435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08900b6d10 0x7f08900b71a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0888009d90 tx=0x7f088802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=923b6e78aeb7f9df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f088effd640 1 -- 172.21.15.27:0/1254531435 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888039070 con 0x7f08900b6d10 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f088effd640 1 -- 172.21.15.27:0/1254531435 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088802cce0 con 0x7f08900b6d10 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f088effd640 1 -- 172.21.15.27:0/1254531435 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888034660 con 0x7f08900b6d10 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f0896681640 1 -- 172.21.15.27:0/1254531435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08900b6d10 msgr2=0x7f08900b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f0896681640 1 --2- 172.21.15.27:0/1254531435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08900b6d10 0x7f08900b71a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0888009d90 tx=0x7f088802bf30 comp rx=0 tx=0).stop 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f0896681640 1 -- 172.21.15.27:0/1254531435 shutdown_connections 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f0896681640 1 --2- 172.21.15.27:0/1254531435 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f08900b6d10 0x7f08900b71a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f0896681640 1 --2- 172.21.15.27:0/1254531435 >> v2:172.21.15.69:3300/0 conn(0x7f089015dd90 0x7f08900ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f0896681640 1 -- 172.21.15.27:0/1254531435 >> 172.21.15.27:0/1254531435 conn(0x7f0890071960 msgr2=0x7f0890071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f0896681640 1 -- 172.21.15.27:0/1254531435 shutdown_connections 2024-03-06T07:32:25.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.929+0000 7f0896681640 1 -- 172.21.15.27:0/1254531435 wait complete. 2024-03-06T07:32:25.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.930+0000 7f0896681640 1 Processor -- start 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.930+0000 7f0896681640 1 -- start start 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f0896681640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08900b6d10 0x7f08901e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f0896681640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08900d29d0 0x7f08901e87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f0896681640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089015dd90 0x7f089014b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f0896681640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08900d2df0 con 0x7f08900d29d0 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f0896681640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089015e170 con 0x7f08900b6d10 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f0896681640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08900d48f0 con 0x7f089015dd90 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f088f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08900d29d0 0x7f08901e87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f0894c1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089015dd90 0x7f089014b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f088f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08900d29d0 0x7f08901e87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46922/0 (socket says 172.21.15.27:46922) 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f088ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08900b6d10 0x7f08901e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.931+0000 7f088f7fe640 1 -- 172.21.15.27:0/696006315 learned_addr learned my addr 172.21.15.27:0/696006315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f0894c1e640 1 -- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08900b6d10 msgr2=0x7f08901e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f0894c1e640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08900b6d10 0x7f08901e8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f0894c1e640 1 -- 172.21.15.27:0/696006315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08900d29d0 msgr2=0x7f08901e87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f0894c1e640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08900d29d0 0x7f08901e87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f0894c1e640 1 -- 172.21.15.27:0/696006315 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08901e9030 con 0x7f089015dd90 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f088ffff640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08900b6d10 0x7f08901e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f088f7fe640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08900d29d0 0x7f08901e87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f0894c1e640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089015dd90 0x7f089014b940 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0888033000 tx=0x7f0888004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f088cff9640 1 -- 172.21.15.27:0/696006315 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888044020 con 0x7f089015dd90 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f088cff9640 1 -- 172.21.15.27:0/696006315 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088803a040 con 0x7f089015dd90 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f088cff9640 1 -- 172.21.15.27:0/696006315 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888039050 con 0x7f089015dd90 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0890149b00 con 0x7f089015dd90 2024-03-06T07:32:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.932+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089014a0c0 con 0x7f089015dd90 2024-03-06T07:32:25.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.934+0000 7f088cff9640 1 -- 172.21.15.27:0/696006315 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0888003bd0 con 0x7f089015dd90 2024-03-06T07:32:25.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.934+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0890072a30 con 0x7f089015dd90 2024-03-06T07:32:25.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.934+0000 7f088cff9640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f087006b4d0 0x7f087006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.934+0000 7f088cff9640 1 -- 172.21.15.27:0/696006315 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08880ac570 con 0x7f089015dd90 2024-03-06T07:32:25.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.934+0000 7f088ffff640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f087006b4d0 0x7f087006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.935+0000 7f088ffff640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f087006b4d0 0x7f087006d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f0880009e90 tx=0x7f0880009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:25.939+0000 7f088cff9640 1 -- 172.21.15.27:0/696006315 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f088804e060 con 0x7f089015dd90 2024-03-06T07:32:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.190+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f089004f410 con 0x7f089015dd90 2024-03-06T07:32:26.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.191+0000 7f088cff9640 1 -- 172.21.15.27:0/696006315 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v147) v1 ==== 104+0+11 (secure 0 0 0) 0x7f088807ae30 con 0x7f089015dd90 2024-03-06T07:32:26.192 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:32:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f087006b4d0 msgr2=0x7f087006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f087006b4d0 0x7f087006d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f0880009e90 tx=0x7f0880009190 comp rx=0 tx=0).stop 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089015dd90 msgr2=0x7f089014b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089015dd90 0x7f089014b940 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0888033000 tx=0x7f0888004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 shutdown_connections 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f087006b4d0 0x7f087006d990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f089015dd90 0x7f089014b940 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08900d29d0 0x7f08901e87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 --2- 172.21.15.27:0/696006315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08900b6d10 0x7f08901e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 >> 172.21.15.27:0/696006315 conn(0x7f0890071960 msgr2=0x7f08900b3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 shutdown_connections 2024-03-06T07:32:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.193+0000 7f0896681640 1 -- 172.21.15.27:0/696006315 wait complete. 2024-03-06T07:32:26.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.352+0000 7f30c19d7640 1 Processor -- start 2024-03-06T07:32:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.352+0000 7f30c19d7640 1 -- start start 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30c19d7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30bc071540 0x7f30bc071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30c19d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 0x7f30bc0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30c19d7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f30bc0cecb0 0x7f30bc0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30c19d7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30bc0d3230 con 0x7f30bc071540 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30c19d7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30bc071e60 con 0x7f30bc0cecb0 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30c19d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0d4960 con 0x7f30bc0b2d30 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30baffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30bc071540 0x7f30bc071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30baffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30bc071540 0x7f30bc071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46930/0 (socket says 172.21.15.27:46930) 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30ba7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 0x7f30bc0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30baffd640 1 -- 172.21.15.27:0/2061797503 learned_addr learned my addr 172.21.15.27:0/2061797503 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30bb7fe640 1 --2- 172.21.15.27:0/2061797503 >> v2:172.21.15.69:3300/0 conn(0x7f30bc0cecb0 0x7f30bc0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30baffd640 1 -- 172.21.15.27:0/2061797503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 msgr2=0x7f30bc0ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30baffd640 1 --2- 172.21.15.27:0/2061797503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 0x7f30bc0ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30baffd640 1 -- 172.21.15.27:0/2061797503 >> v2:172.21.15.69:3300/0 conn(0x7f30bc0cecb0 msgr2=0x7f30bc0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30baffd640 1 --2- 172.21.15.27:0/2061797503 >> v2:172.21.15.69:3300/0 conn(0x7f30bc0cecb0 0x7f30bc0cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30baffd640 1 -- 172.21.15.27:0/2061797503 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30bc0d36c0 con 0x7f30bc071540 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30bb7fe640 1 --2- 172.21.15.27:0/2061797503 >> v2:172.21.15.69:3300/0 conn(0x7f30bc0cecb0 0x7f30bc0cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.353+0000 7f30ba7fc640 1 --2- 172.21.15.27:0/2061797503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 0x7f30bc0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.354+0000 7f30baffd640 1 --2- 172.21.15.27:0/2061797503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc071540 0x7f30bc071920 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0b68c0 tx=0x7f30ac02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b563c6387d6ca24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.354+0000 7f30b9ffb640 1 -- 172.21.15.27:0/2061797503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac035020 con 0x7f30bc071540 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.354+0000 7f30b9ffb640 1 -- 172.21.15.27:0/2061797503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ac02d040 con 0x7f30bc071540 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.354+0000 7f30b9ffb640 1 -- 172.21.15.27:0/2061797503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac031a80 con 0x7f30bc071540 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.355+0000 7f30c19d7640 1 -- 172.21.15.27:0/2061797503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc071540 msgr2=0x7f30bc071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.355+0000 7f30c19d7640 1 --2- 172.21.15.27:0/2061797503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc071540 0x7f30bc071920 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0b68c0 tx=0x7f30ac02b900 comp rx=0 tx=0).stop 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.355+0000 7f30c19d7640 1 -- 172.21.15.27:0/2061797503 shutdown_connections 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.355+0000 7f30c19d7640 1 --2- 172.21.15.27:0/2061797503 >> v2:172.21.15.69:3300/0 conn(0x7f30bc0cecb0 0x7f30bc0cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.355+0000 7f30c19d7640 1 --2- 172.21.15.27:0/2061797503 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 0x7f30bc0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.355+0000 7f30c19d7640 1 --2- 172.21.15.27:0/2061797503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc071540 0x7f30bc071920 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.355+0000 7f30c19d7640 1 -- 172.21.15.27:0/2061797503 >> 172.21.15.27:0/2061797503 conn(0x7f30bc0d3d60 msgr2=0x7f30bc0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.355+0000 7f30c19d7640 1 -- 172.21.15.27:0/2061797503 shutdown_connections 2024-03-06T07:32:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.355+0000 7f30c19d7640 1 -- 172.21.15.27:0/2061797503 wait complete. 2024-03-06T07:32:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.356+0000 7f30c19d7640 1 Processor -- start 2024-03-06T07:32:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.356+0000 7f30c19d7640 1 -- start start 2024-03-06T07:32:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30c19d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30bc071540 0x7f30bc0aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30c19d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 0x7f30bc0aac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30c19d7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc0cecb0 0x7f30bc0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30c19d7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0d2be0 con 0x7f30bc0cecb0 2024-03-06T07:32:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30c19d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0d2900 con 0x7f30bc071540 2024-03-06T07:32:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30c19d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0d2ec0 con 0x7f30bc0b2d30 2024-03-06T07:32:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30bb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc0cecb0 0x7f30bc0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30bb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc0cecb0 0x7f30bc0cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46934/0 (socket says 172.21.15.27:46934) 2024-03-06T07:32:26.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30baffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30bc071540 0x7f30bc0aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30bb7fe640 1 -- 172.21.15.27:0/523065703 learned_addr learned my addr 172.21.15.27:0/523065703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30ba7fc640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 0x7f30bc0aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30bb7fe640 1 -- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 msgr2=0x7f30bc0aac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30bb7fe640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 0x7f30bc0aac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30bb7fe640 1 -- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30bc071540 msgr2=0x7f30bc0aa6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30bb7fe640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30bc071540 0x7f30bc0aa6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30bb7fe640 1 -- 172.21.15.27:0/523065703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30bc0ab240 con 0x7f30bc0cecb0 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.357+0000 7f30baffd640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30bc071540 0x7f30bc0aa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.358+0000 7f30bb7fe640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc0cecb0 0x7f30bc0cc9f0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f30b400efc0 tx=0x7f30b4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.358+0000 7f30a3fff640 1 -- 172.21.15.27:0/523065703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b4018020 con 0x7f30bc0cecb0 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.358+0000 7f30a3fff640 1 -- 172.21.15.27:0/523065703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b4014480 con 0x7f30bc0cecb0 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.358+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30bc0ab520 con 0x7f30bc0cecb0 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.358+0000 7f30a3fff640 1 -- 172.21.15.27:0/523065703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b4014870 con 0x7f30bc0cecb0 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.358+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30bc0c8e00 con 0x7f30bc0cecb0 2024-03-06T07:32:26.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.359+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30bc068e20 con 0x7f30bc0cecb0 2024-03-06T07:32:26.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.359+0000 7f30a3fff640 1 -- 172.21.15.27:0/523065703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30b401b020 con 0x7f30bc0cecb0 2024-03-06T07:32:26.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.360+0000 7f30a3fff640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30a806b4d0 0x7f30a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.360+0000 7f30a3fff640 1 -- 172.21.15.27:0/523065703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30b408ae50 con 0x7f30bc0cecb0 2024-03-06T07:32:26.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.360+0000 7f30baffd640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30a806b4d0 0x7f30a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.360+0000 7f30baffd640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30a806b4d0 0x7f30a806d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f30ac036000 tx=0x7f30ac03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.369+0000 7f30a3fff640 1 -- 172.21.15.27:0/523065703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30bc068e20 con 0x7f30bc0cecb0 2024-03-06T07:32:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.553+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f30bc0c8fd0 con 0x7f30bc0cecb0 2024-03-06T07:32:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.554+0000 7f30a3fff640 1 -- 172.21.15.27:0/523065703 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v147) v1 ==== 100+0+11 (secure 0 0 0) 0x7f30b4022020 con 0x7f30bc0cecb0 2024-03-06T07:32:26.555 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:32:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.556+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30a806b4d0 msgr2=0x7f30a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.556+0000 7f30c19d7640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30a806b4d0 0x7f30a806d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f30ac036000 tx=0x7f30ac03c040 comp rx=0 tx=0).stop 2024-03-06T07:32:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.556+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc0cecb0 msgr2=0x7f30bc0cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.556+0000 7f30c19d7640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc0cecb0 0x7f30bc0cc9f0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f30b400efc0 tx=0x7f30b4009f90 comp rx=0 tx=0).stop 2024-03-06T07:32:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.557+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 shutdown_connections 2024-03-06T07:32:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.557+0000 7f30c19d7640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30a806b4d0 0x7f30a806d990 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.557+0000 7f30c19d7640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30bc0cecb0 0x7f30bc0cc9f0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.557+0000 7f30c19d7640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f30bc0b2d30 0x7f30bc0aac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.557+0000 7f30c19d7640 1 --2- 172.21.15.27:0/523065703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30bc071540 0x7f30bc0aa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.557+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 >> 172.21.15.27:0/523065703 conn(0x7f30bc0d3d60 msgr2=0x7f30bc15b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.557+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 shutdown_connections 2024-03-06T07:32:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.557+0000 7f30c19d7640 1 -- 172.21.15.27:0/523065703 wait complete. 2024-03-06T07:32:26.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.675+0000 7f9a4f30d640 1 Processor -- start 2024-03-06T07:32:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.675+0000 7f9a4f30d640 1 -- start start 2024-03-06T07:32:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4f30d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 0x7f9a480d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4f30d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a480b95a0 0x7f9a480b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4f30d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a480b9a30 0x7f9a480b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4f30d640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a480cea30 con 0x7f9a480b9a30 2024-03-06T07:32:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4f30d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a480d2af0 con 0x7f9a480b95a0 2024-03-06T07:32:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4f30d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480cf3c0 con 0x7f9a4815df00 2024-03-06T07:32:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4d8aa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a480b9a30 0x7f9a480b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4d8aa640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a480b9a30 0x7f9a480b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46944/0 (socket says 172.21.15.27:46944) 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4d8aa640 1 -- 172.21.15.27:0/3447728703 learned_addr learned my addr 172.21.15.27:0/3447728703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4d0a9640 1 --2- 172.21.15.27:0/3447728703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 0x7f9a480d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.676+0000 7f9a4c8a8640 1 --2- 172.21.15.27:0/3447728703 >> v2:172.21.15.69:3300/0 conn(0x7f9a480b95a0 0x7f9a480b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.677+0000 7f9a4d8aa640 1 -- 172.21.15.27:0/3447728703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 msgr2=0x7f9a480d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.677+0000 7f9a4d8aa640 1 --2- 172.21.15.27:0/3447728703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 0x7f9a480d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.677+0000 7f9a4d8aa640 1 -- 172.21.15.27:0/3447728703 >> v2:172.21.15.69:3300/0 conn(0x7f9a480b95a0 msgr2=0x7f9a480b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.677+0000 7f9a4d8aa640 1 --2- 172.21.15.27:0/3447728703 >> v2:172.21.15.69:3300/0 conn(0x7f9a480b95a0 0x7f9a480b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.677+0000 7f9a4d8aa640 1 -- 172.21.15.27:0/3447728703 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a480cf0e0 con 0x7f9a480b9a30 2024-03-06T07:32:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.677+0000 7f9a4c8a8640 1 --2- 172.21.15.27:0/3447728703 >> v2:172.21.15.69:3300/0 conn(0x7f9a480b95a0 0x7f9a480b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.677+0000 7f9a4d8aa640 1 --2- 172.21.15.27:0/3447728703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a480b9a30 0x7f9a480b5c40 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f9a4808cbc0 tx=0x7f9a4002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48455ecc63b91ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a3bfff640 1 -- 172.21.15.27:0/3447728703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4002ca70 con 0x7f9a480b9a30 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a3bfff640 1 -- 172.21.15.27:0/3447728703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4002cc10 con 0x7f9a480b9a30 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a3bfff640 1 -- 172.21.15.27:0/3447728703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a400343f0 con 0x7f9a480b9a30 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a4f30d640 1 -- 172.21.15.27:0/3447728703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a480b9a30 msgr2=0x7f9a480b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/3447728703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a480b9a30 0x7f9a480b5c40 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f9a4808cbc0 tx=0x7f9a4002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a4f30d640 1 -- 172.21.15.27:0/3447728703 shutdown_connections 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/3447728703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a480b9a30 0x7f9a480b5c40 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/3447728703 >> v2:172.21.15.69:3300/0 conn(0x7f9a480b95a0 0x7f9a480b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/3447728703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 0x7f9a480d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a4f30d640 1 -- 172.21.15.27:0/3447728703 >> 172.21.15.27:0/3447728703 conn(0x7f9a48071960 msgr2=0x7f9a48071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a4f30d640 1 -- 172.21.15.27:0/3447728703 shutdown_connections 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.678+0000 7f9a4f30d640 1 -- 172.21.15.27:0/3447728703 wait complete. 2024-03-06T07:32:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.679+0000 7f9a4f30d640 1 Processor -- start 2024-03-06T07:32:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4f30d640 1 -- start start 2024-03-06T07:32:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4f30d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480b95a0 0x7f9a4815ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4f30d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 0x7f9a4815fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4f30d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a48161280 0x7f9a480b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4f30d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480cec70 con 0x7f9a48161280 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4f30d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480ce750 con 0x7f9a480b95a0 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4f30d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480cedf0 con 0x7f9a4815df00 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4d8aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a48161280 0x7f9a480b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4d8aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a48161280 0x7f9a480b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46952/0 (socket says 172.21.15.27:46952) 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4d8aa640 1 -- 172.21.15.27:0/4064981485 learned_addr learned my addr 172.21.15.27:0/4064981485 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4c8a8640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 0x7f9a4815fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.680+0000 7f9a4d0a9640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480b95a0 0x7f9a4815ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4d0a9640 1 -- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 msgr2=0x7f9a4815fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4d0a9640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 0x7f9a4815fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4d0a9640 1 -- 172.21.15.27:0/4064981485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a48161280 msgr2=0x7f9a480b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4d0a9640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a48161280 0x7f9a480b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4d0a9640 1 -- 172.21.15.27:0/4064981485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a481601e0 con 0x7f9a480b95a0 2024-03-06T07:32:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4c8a8640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 0x7f9a4815fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4d8aa640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a48161280 0x7f9a480b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4d0a9640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480b95a0 0x7f9a4815ed60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400e880 tx=0x7f9a4400ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a39ffb640 1 -- 172.21.15.27:0/4064981485 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44017070 con 0x7f9a480b95a0 2024-03-06T07:32:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a39ffb640 1 -- 172.21.15.27:0/4064981485 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4400c940 con 0x7f9a480b95a0 2024-03-06T07:32:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a481604c0 con 0x7f9a480b95a0 2024-03-06T07:32:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a39ffb640 1 -- 172.21.15.27:0/4064981485 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4400cd30 con 0x7f9a480b95a0 2024-03-06T07:32:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.681+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a48160a80 con 0x7f9a480b95a0 2024-03-06T07:32:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.682+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a48067ff0 con 0x7f9a480b95a0 2024-03-06T07:32:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.683+0000 7f9a39ffb640 1 -- 172.21.15.27:0/4064981485 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a4401a070 con 0x7f9a480b95a0 2024-03-06T07:32:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.683+0000 7f9a39ffb640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a2c06b4d0 0x7f9a2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.683+0000 7f9a4c8a8640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a2c06b4d0 0x7f9a2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.684+0000 7f9a39ffb640 1 -- 172.21.15.27:0/4064981485 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a4408a450 con 0x7f9a480b95a0 2024-03-06T07:32:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.684+0000 7f9a4c8a8640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a2c06b4d0 0x7f9a2c06d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9a480b8780 tx=0x7f9a3c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.688+0000 7f9a39ffb640 1 -- 172.21.15.27:0/4064981485 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a44058c70 con 0x7f9a480b95a0 2024-03-06T07:32:26.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.881+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9a48071ff0 con 0x7f9a480b95a0 2024-03-06T07:32:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.882+0000 7f9a39ffb640 1 -- 172.21.15.27:0/4064981485 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v147) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9a4405cd10 con 0x7f9a480b95a0 2024-03-06T07:32:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a2c06b4d0 msgr2=0x7f9a2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a2c06b4d0 0x7f9a2c06d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9a480b8780 tx=0x7f9a3c009190 comp rx=0 tx=0).stop 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480b95a0 msgr2=0x7f9a4815ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480b95a0 0x7f9a4815ed60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400e880 tx=0x7f9a4400ed40 comp rx=0 tx=0).stop 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 shutdown_connections 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a2c06b4d0 0x7f9a2c06d990 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a48161280 0x7f9a480b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a4815df00 0x7f9a4815fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 --2- 172.21.15.27:0/4064981485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a480b95a0 0x7f9a4815ed60 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 >> 172.21.15.27:0/4064981485 conn(0x7f9a48071960 msgr2=0x7f9a480d3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 shutdown_connections 2024-03-06T07:32:26.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:26.885+0000 7f9a4f30d640 1 -- 172.21.15.27:0/4064981485 wait complete. 2024-03-06T07:32:26.898 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:32:26.898 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:26.913 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:27.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.058+0000 7f00589f7640 1 Processor -- start 2024-03-06T07:32:27.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.058+0000 7f00589f7640 1 -- start start 2024-03-06T07:32:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.059+0000 7f00589f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f00540b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.059+0000 7f00589f7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00540d3400 0x7f0054071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.059+0000 7f00589f7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0054071e60 con 0x7f00540b2b50 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.059+0000 7f00589f7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00540d2f70 con 0x7f00540d3400 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.059+0000 7f00589f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00540d4be0 con 0x7f0054071540 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.059+0000 7f0051d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00540d3400 0x7f0054071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.059+0000 7f0052d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f00540b2b50 0x7f00540d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53516/0 (socket says 172.21.15.27:53516) 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.059+0000 7f0052d9d640 1 -- 172.21.15.27:0/2735552330 learned_addr learned my addr 172.21.15.27:0/2735552330 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.059+0000 7f005259c640 1 --2- 172.21.15.27:0/2735552330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f00540b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.060+0000 7f0051d9b640 1 -- 172.21.15.27:0/2735552330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 msgr2=0x7f00540b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.060+0000 7f0051d9b640 1 --2- 172.21.15.27:0/2735552330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f00540b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.060+0000 7f0051d9b640 1 -- 172.21.15.27:0/2735552330 >> v1:172.21.15.27:6789/0 conn(0x7f00540b2b50 legacy=0x7f00540d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.060+0000 7f0051d9b640 1 -- 172.21.15.27:0/2735552330 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00540ce610 con 0x7f00540d3400 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.060+0000 7f005159a640 1 -- 172.21.15.27:0/2735552330 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282008319 0 0) 0x7f0054071e60 con 0x7f00540b2b50 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.060+0000 7f005259c640 1 --2- 172.21.15.27:0/2735552330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f00540b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.060+0000 7f0051d9b640 1 --2- 172.21.15.27:0/2735552330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 0x7f0054071920 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f00440097a0 tx=0x7f004402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da62180752bc6e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.061+0000 7f005159a640 1 -- 172.21.15.27:0/2735552330 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0044035020 con 0x7f00540d3400 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.061+0000 7f005159a640 1 -- 172.21.15.27:0/2735552330 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0044031830 con 0x7f00540d3400 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.061+0000 7f005159a640 1 -- 172.21.15.27:0/2735552330 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0044031c00 con 0x7f00540d3400 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.061+0000 7f00589f7640 1 -- 172.21.15.27:0/2735552330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 msgr2=0x7f0054071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.061+0000 7f00589f7640 1 --2- 172.21.15.27:0/2735552330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 0x7f0054071920 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f00440097a0 tx=0x7f004402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.061+0000 7f00589f7640 1 -- 172.21.15.27:0/2735552330 shutdown_connections 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.061+0000 7f00589f7640 1 --2- 172.21.15.27:0/2735552330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 0x7f0054071920 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.061+0000 7f00589f7640 1 --2- 172.21.15.27:0/2735552330 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f00540b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.061+0000 7f00589f7640 1 -- 172.21.15.27:0/2735552330 >> 172.21.15.27:0/2735552330 conn(0x7f00540d3ca0 msgr2=0x7f00540d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.062+0000 7f00589f7640 1 -- 172.21.15.27:0/2735552330 shutdown_connections 2024-03-06T07:32:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.062+0000 7f00589f7640 1 -- 172.21.15.27:0/2735552330 wait complete. 2024-03-06T07:32:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.064+0000 7f00589f7640 1 Processor -- start 2024-03-06T07:32:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.064+0000 7f00589f7640 1 -- start start 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f00589f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f0054150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f00589f7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00540b2b50 0x7f00540cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f00589f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 0x7f00540cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f00589f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00540ce8f0 con 0x7f00540d3400 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f00589f7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00540d2c00 con 0x7f00540b2b50 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f00589f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005415e370 con 0x7f0054071540 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f005259c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f0054150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f0052d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 0x7f00540cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f005259c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f0054150190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:60266/0 (socket says 172.21.15.27:60266) 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f005259c640 1 -- 172.21.15.27:0/2962164465 learned_addr learned my addr 172.21.15.27:0/2962164465 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.065+0000 7f0051d9b640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00540b2b50 0x7f00540cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0052d9d640 1 -- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 msgr2=0x7f0054150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0052d9d640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f0054150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0052d9d640 1 -- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00540b2b50 msgr2=0x7f00540cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0052d9d640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00540b2b50 0x7f00540cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0052d9d640 1 -- 172.21.15.27:0/2962164465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00540ccd80 con 0x7f00540d3400 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0051d9b640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00540b2b50 0x7f00540cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f005259c640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f0054150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0052d9d640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 0x7f00540cb2b0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f004c00c6d0 tx=0x7f004c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0042ffd640 1 -- 172.21.15.27:0/2962164465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c018070 con 0x7f00540d3400 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0042ffd640 1 -- 172.21.15.27:0/2962164465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004c004d60 con 0x7f00540d3400 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f0042ffd640 1 -- 172.21.15.27:0/2962164465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c00a880 con 0x7f00540d3400 2024-03-06T07:32:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00540cd060 con 0x7f00540d3400 2024-03-06T07:32:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.066+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00540c75e0 con 0x7f00540d3400 2024-03-06T07:32:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.067+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00540cd580 con 0x7f00540d3400 2024-03-06T07:32:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.068+0000 7f0042ffd640 1 -- 172.21.15.27:0/2962164465 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f004c00ac30 con 0x7f00540d3400 2024-03-06T07:32:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.069+0000 7f0042ffd640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003406b4d0 0x7f003406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.069+0000 7f0042ffd640 1 -- 172.21.15.27:0/2962164465 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f004c089940 con 0x7f00540d3400 2024-03-06T07:32:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.069+0000 7f005259c640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003406b4d0 0x7f003406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.069+0000 7f005259c640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003406b4d0 0x7f003406d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f0048009560 tx=0x7f0048009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.073+0000 7f0042ffd640 1 -- 172.21.15.27:0/2962164465 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00540cd580 con 0x7f00540d3400 2024-03-06T07:32:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.257+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0054068700 con 0x7f00540d3400 2024-03-06T07:32:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.258+0000 7f0042ffd640 1 -- 172.21.15.27:0/2962164465 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+16607 (secure 0 0 0) 0x7f004c058170 con 0x7f00540d3400 2024-03-06T07:32:27.260 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:27.260 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":147,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:32:05.629029+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003406b4d0 msgr2=0x7f003406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003406b4d0 0x7f003406d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f0048009560 tx=0x7f0048009190 comp rx=0 tx=0).stop 2024-03-06T07:32:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 msgr2=0x7f00540cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 0x7f00540cb2b0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f004c00c6d0 tx=0x7f004c00cb90 comp rx=0 tx=0).stop 2024-03-06T07:32:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 shutdown_connections 2024-03-06T07:32:27.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f003406b4d0 0x7f003406d990 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00540d3400 0x7f00540cb2b0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00540b2b50 0x7f00540cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 --2- 172.21.15.27:0/2962164465 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0054071540 0x7f0054150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.260+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 >> 172.21.15.27:0/2962164465 conn(0x7f00540d3ca0 msgr2=0x7f0054071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.261+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 shutdown_connections 2024-03-06T07:32:27.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.261+0000 7f00589f7640 1 -- 172.21.15.27:0/2962164465 wait complete. 2024-03-06T07:32:27.273 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:42.692383+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:43.367948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:27.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:27.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.399+0000 7f88ee283640 1 Processor -- start 2024-03-06T07:32:27.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.399+0000 7f88ee283640 1 -- start start 2024-03-06T07:32:27.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.400+0000 7f88ee283640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e80b7420 0x7f88e80d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.400+0000 7f88ee283640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e80d3740 con 0x7f88e815dd90 2024-03-06T07:32:27.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.400+0000 7f88ee283640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e80d3120 con 0x7f88e80ceea0 2024-03-06T07:32:27.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.400+0000 7f88ee283640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80d3400 con 0x7f88e80b7420 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.400+0000 7f88e7fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f88e815dd90 0x7f88e80d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53524/0 (socket says 172.21.15.27:53524) 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.400+0000 7f88e7fff640 1 -- 172.21.15.27:0/2504121621 learned_addr learned my addr 172.21.15.27:0/2504121621 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.400+0000 7f88ec820640 1 --2- 172.21.15.27:0/2504121621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e80b7420 0x7f88e80d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.401+0000 7f88e6ffd640 1 -- 172.21.15.27:0/2504121621 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237087865 0 0) 0x7f88e80d3740 con 0x7f88e815dd90 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.401+0000 7f88e6ffd640 1 -- 172.21.15.27:0/2504121621 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d0003520 con 0x7f88e815dd90 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.401+0000 7f88e6ffd640 1 -- 172.21.15.27:0/2504121621 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787441780 0 0) 0x7f88e80d3120 con 0x7f88e80ceea0 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.401+0000 7f88e6ffd640 1 -- 172.21.15.27:0/2504121621 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e80d3740 con 0x7f88e80ceea0 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.401+0000 7f88ec820640 1 -- 172.21.15.27:0/2504121621 >> v1:172.21.15.69:6789/0 conn(0x7f88e80ceea0 legacy=0x7f88e80b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.401+0000 7f88ec820640 1 -- 172.21.15.27:0/2504121621 >> v1:172.21.15.27:6789/0 conn(0x7f88e815dd90 legacy=0x7f88e80d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.401+0000 7f88ec820640 1 -- 172.21.15.27:0/2504121621 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e80d39e0 con 0x7f88e80b7420 2024-03-06T07:32:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88ec820640 1 --2- 172.21.15.27:0/2504121621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e80b7420 0x7f88e80d29d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f88e0009dc0 tx=0x7f88e002bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86f13f5a7529f59f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88e6ffd640 1 -- 172.21.15.27:0/2504121621 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0039070 con 0x7f88e80b7420 2024-03-06T07:32:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88e6ffd640 1 -- 172.21.15.27:0/2504121621 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e002cd50 con 0x7f88e80b7420 2024-03-06T07:32:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88e6ffd640 1 -- 172.21.15.27:0/2504121621 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0034660 con 0x7f88e80b7420 2024-03-06T07:32:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88ee283640 1 -- 172.21.15.27:0/2504121621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e80b7420 msgr2=0x7f88e80d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88ee283640 1 --2- 172.21.15.27:0/2504121621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e80b7420 0x7f88e80d29d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f88e0009dc0 tx=0x7f88e002bfa0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88ee283640 1 -- 172.21.15.27:0/2504121621 shutdown_connections 2024-03-06T07:32:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88ee283640 1 --2- 172.21.15.27:0/2504121621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e80b7420 0x7f88e80d29d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88ee283640 1 -- 172.21.15.27:0/2504121621 >> 172.21.15.27:0/2504121621 conn(0x7f88e8071960 msgr2=0x7f88e8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88ee283640 1 -- 172.21.15.27:0/2504121621 shutdown_connections 2024-03-06T07:32:27.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.402+0000 7f88ee283640 1 -- 172.21.15.27:0/2504121621 wait complete. 2024-03-06T07:32:27.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.404+0000 7f88ee283640 1 Processor -- start 2024-03-06T07:32:27.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.404+0000 7f88ee283640 1 -- start start 2024-03-06T07:32:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88ee283640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88e80ceea0 0x7f88e8153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88ee283640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e815dd90 0x7f88e8158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88ee283640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e8160bb0 0x7f88e81f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88ee283640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80d48f0 con 0x7f88e80ceea0 2024-03-06T07:32:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88ee283640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e815e170 con 0x7f88e815dd90 2024-03-06T07:32:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88ee283640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80d4a70 con 0x7f88e8160bb0 2024-03-06T07:32:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88e80ceea0 0x7f88e8153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88e80ceea0 0x7f88e8153060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46974/0 (socket says 172.21.15.27:46974) 2024-03-06T07:32:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e7fff640 1 -- 172.21.15.27:0/3592266351 learned_addr learned my addr 172.21.15.27:0/3592266351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88ec820640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e8160bb0 0x7f88e81f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e77fe640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e815dd90 0x7f88e8158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e7fff640 1 -- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e8160bb0 msgr2=0x7f88e81f71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e7fff640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e8160bb0 0x7f88e81f71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e7fff640 1 -- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e815dd90 msgr2=0x7f88e8158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e7fff640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e815dd90 0x7f88e8158400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e7fff640 1 -- 172.21.15.27:0/3592266351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e815e620 con 0x7f88e80ceea0 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e7fff640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88e80ceea0 0x7f88e8153060 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00c410 tx=0x7f88dc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e4ff9640 1 -- 172.21.15.27:0/3592266351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc017070 con 0x7f88e80ceea0 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e4ff9640 1 -- 172.21.15.27:0/3592266351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc005ce0 con 0x7f88e80ceea0 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e8157080 con 0x7f88e80ceea0 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.405+0000 7f88e4ff9640 1 -- 172.21.15.27:0/3592266351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc0072d0 con 0x7f88e80ceea0 2024-03-06T07:32:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.406+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e8157640 con 0x7f88e80ceea0 2024-03-06T07:32:27.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.407+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e8072a30 con 0x7f88e80ceea0 2024-03-06T07:32:27.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.407+0000 7f88e4ff9640 1 -- 172.21.15.27:0/3592266351 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88dc01a070 con 0x7f88e80ceea0 2024-03-06T07:32:27.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.407+0000 7f88e4ff9640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f88d406b4d0 0x7f88d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.408+0000 7f88e4ff9640 1 -- 172.21.15.27:0/3592266351 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88dc089490 con 0x7f88e80ceea0 2024-03-06T07:32:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.408+0000 7f88e77fe640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f88d406b4d0 0x7f88d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.408+0000 7f88e77fe640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f88d406b4d0 0x7f88d406d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f88d8007fd0 tx=0x7f88d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.412+0000 7f88e4ff9640 1 -- 172.21.15.27:0/3592266351 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88dc057cc0 con 0x7f88e80ceea0 2024-03-06T07:32:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.592+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88e804f410 con 0x7f88e80ceea0 2024-03-06T07:32:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.592+0000 7f88e4ff9640 1 -- 172.21.15.27:0/3592266351 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88dc05bd60 con 0x7f88e80ceea0 2024-03-06T07:32:27.594 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:32:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f88d406b4d0 msgr2=0x7f88d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f88d406b4d0 0x7f88d406d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f88d8007fd0 tx=0x7f88d8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88e80ceea0 msgr2=0x7f88e8153060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88e80ceea0 0x7f88e8153060 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00c410 tx=0x7f88dc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 shutdown_connections 2024-03-06T07:32:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f88d406b4d0 0x7f88d406d990 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f88e8160bb0 0x7f88e81f71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e815dd90 0x7f88e8158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 --2- 172.21.15.27:0/3592266351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88e80ceea0 0x7f88e8153060 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.594+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 >> 172.21.15.27:0/3592266351 conn(0x7f88e8071960 msgr2=0x7f88e80b3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.595+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 shutdown_connections 2024-03-06T07:32:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.595+0000 7f88ee283640 1 -- 172.21.15.27:0/3592266351 wait complete. 2024-03-06T07:32:27.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:27.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.715+0000 7fcf2a86d640 1 Processor -- start 2024-03-06T07:32:27.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.716+0000 7fcf2a86d640 1 -- start start 2024-03-06T07:32:27.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.716+0000 7fcf2a86d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf2415dd90 0x7fcf240ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.716+0000 7fcf2a86d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcf240b6d10 0x7fcf240b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.716+0000 7fcf2a86d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf240d3740 con 0x7fcf240d29d0 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.716+0000 7fcf2a86d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf240d3400 con 0x7fcf240b6d10 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.716+0000 7fcf2a86d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240d3120 con 0x7fcf2415dd90 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.717+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf2415dd90 0x7fcf240ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.717+0000 7fcf28e0a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcf240b6d10 0x7fcf240b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.717+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf2415dd90 0x7fcf240ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:60298/0 (socket says 172.21.15.27:60298) 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.717+0000 7fcf23fff640 1 -- 172.21.15.27:0/1046206640 learned_addr learned my addr 172.21.15.27:0/1046206640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.717+0000 7fcf237fe640 1 --1- 172.21.15.27:0/1046206640 >> v1:172.21.15.69:6789/0 conn(0x7fcf240d29d0 0x7fcf240d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:50566/0 (socket says 172.21.15.27:50566) 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.717+0000 7fcf23fff640 1 -- 172.21.15.27:0/1046206640 >> v2:172.21.15.69:3300/0 conn(0x7fcf240b6d10 msgr2=0x7fcf240b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.717+0000 7fcf23fff640 1 --2- 172.21.15.27:0/1046206640 >> v2:172.21.15.69:3300/0 conn(0x7fcf240b6d10 0x7fcf240b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.717+0000 7fcf23fff640 1 -- 172.21.15.27:0/1046206640 >> v1:172.21.15.69:6789/0 conn(0x7fcf240d29d0 legacy=0x7fcf240d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.717+0000 7fcf23fff640 1 -- 172.21.15.27:0/1046206640 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240d39e0 con 0x7fcf2415dd90 2024-03-06T07:32:27.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf23fff640 1 --2- 172.21.15.27:0/1046206640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf2415dd90 0x7fcf240ceea0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009870 tx=0x7fcf1402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68881e95df014cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf22ffd640 1 -- 172.21.15.27:0/1046206640 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1402f840 con 0x7fcf2415dd90 2024-03-06T07:32:27.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf22ffd640 1 -- 172.21.15.27:0/1046206640 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1402f9e0 con 0x7fcf2415dd90 2024-03-06T07:32:27.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf22ffd640 1 -- 172.21.15.27:0/1046206640 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1402fdb0 con 0x7fcf2415dd90 2024-03-06T07:32:27.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf2a86d640 1 -- 172.21.15.27:0/1046206640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf2415dd90 msgr2=0x7fcf240ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf2a86d640 1 --2- 172.21.15.27:0/1046206640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf2415dd90 0x7fcf240ceea0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009870 tx=0x7fcf1402ba20 comp rx=0 tx=0).stop 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf2a86d640 1 -- 172.21.15.27:0/1046206640 shutdown_connections 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf2a86d640 1 --2- 172.21.15.27:0/1046206640 >> v2:172.21.15.69:3300/0 conn(0x7fcf240b6d10 0x7fcf240b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf2a86d640 1 --2- 172.21.15.27:0/1046206640 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf2415dd90 0x7fcf240ceea0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf2a86d640 1 -- 172.21.15.27:0/1046206640 >> 172.21.15.27:0/1046206640 conn(0x7fcf24071960 msgr2=0x7fcf24071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf2a86d640 1 -- 172.21.15.27:0/1046206640 shutdown_connections 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.718+0000 7fcf2a86d640 1 -- 172.21.15.27:0/1046206640 wait complete. 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.720+0000 7fcf2a86d640 1 Processor -- start 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.720+0000 7fcf2a86d640 1 -- start start 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.720+0000 7fcf2a86d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf240b6d10 0x7fcf241f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.720+0000 7fcf2a86d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf240d29d0 0x7fcf2415ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.720+0000 7fcf2a86d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf2415dd90 0x7fcf24148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.720+0000 7fcf2a86d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240d2df0 con 0x7fcf2415dd90 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.720+0000 7fcf2a86d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2415e170 con 0x7fcf240d29d0 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.720+0000 7fcf2a86d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240d48f0 con 0x7fcf240b6d10 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf28e0a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf2415dd90 0x7fcf24148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf237fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf240d29d0 0x7fcf2415ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf237fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf240d29d0 0x7fcf2415ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:35204/0 (socket says 172.21.15.27:35204) 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf237fe640 1 -- 172.21.15.27:0/2073832069 learned_addr learned my addr 172.21.15.27:0/2073832069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf28e0a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf2415dd90 0x7fcf24148810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46986/0 (socket says 172.21.15.27:46986) 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf23fff640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf240b6d10 0x7fcf241f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf237fe640 1 -- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf240b6d10 msgr2=0x7fcf241f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf237fe640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf240b6d10 0x7fcf241f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf237fe640 1 -- 172.21.15.27:0/2073832069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf2415dd90 msgr2=0x7fcf24148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf237fe640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf2415dd90 0x7fcf24148810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf237fe640 1 -- 172.21.15.27:0/2073832069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2415f550 con 0x7fcf240d29d0 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf237fe640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf240d29d0 0x7fcf2415ed00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c400 tx=0x7fcf1800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf20ff9640 1 -- 172.21.15.27:0/2073832069 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf180074e0 con 0x7fcf240d29d0 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf20ff9640 1 -- 172.21.15.27:0/2073832069 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf18007680 con 0x7fcf240d29d0 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2415f830 con 0x7fcf240d29d0 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf20ff9640 1 -- 172.21.15.27:0/2073832069 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18004a00 con 0x7fcf240d29d0 2024-03-06T07:32:27.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.721+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2415fdf0 con 0x7fcf240d29d0 2024-03-06T07:32:27.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.723+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf24072a30 con 0x7fcf240d29d0 2024-03-06T07:32:27.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.723+0000 7fcf20ff9640 1 -- 172.21.15.27:0/2073832069 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf1801a020 con 0x7fcf240d29d0 2024-03-06T07:32:27.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.723+0000 7fcf20ff9640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf1006b4d0 0x7fcf1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.723+0000 7fcf20ff9640 1 -- 172.21.15.27:0/2073832069 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf18089860 con 0x7fcf240d29d0 2024-03-06T07:32:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.724+0000 7fcf23fff640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf1006b4d0 0x7fcf1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.724+0000 7fcf23fff640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf1006b4d0 0x7fcf1006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fcf14036000 tx=0x7fcf14008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.728+0000 7fcf20ff9640 1 -- 172.21.15.27:0/2073832069 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf18058120 con 0x7fcf240d29d0 2024-03-06T07:32:27.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.908+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcf24160190 con 0x7fcf240d29d0 2024-03-06T07:32:27.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.908+0000 7fcf20ff9640 1 -- 172.21.15.27:0/2073832069 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v147) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcf180890a0 con 0x7fcf240d29d0 2024-03-06T07:32:27.910 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:32:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf1006b4d0 msgr2=0x7fcf1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf1006b4d0 0x7fcf1006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fcf14036000 tx=0x7fcf14008030 comp rx=0 tx=0).stop 2024-03-06T07:32:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf240d29d0 msgr2=0x7fcf2415ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf240d29d0 0x7fcf2415ed00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c400 tx=0x7fcf1800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 shutdown_connections 2024-03-06T07:32:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf1006b4d0 0x7fcf1006d990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf2415dd90 0x7fcf24148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf240d29d0 0x7fcf2415ed00 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 --2- 172.21.15.27:0/2073832069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf240b6d10 0x7fcf241f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 >> 172.21.15.27:0/2073832069 conn(0x7fcf24071960 msgr2=0x7fcf2415c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 shutdown_connections 2024-03-06T07:32:27.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:27.910+0000 7fcf2a86d640 1 -- 172.21.15.27:0/2073832069 wait complete. 2024-03-06T07:32:27.927 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:28.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.035+0000 7fd92c70c640 1 Processor -- start 2024-03-06T07:32:28.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.035+0000 7fd92c70c640 1 -- start start 2024-03-06T07:32:28.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.036+0000 7fd92c70c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd92415dd90 0x7fd9240ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.036+0000 7fd92c70c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240d3740 con 0x7fd9240d29d0 2024-03-06T07:32:28.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.036+0000 7fd92c70c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240d3120 con 0x7fd9240b6d10 2024-03-06T07:32:28.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.036+0000 7fd92c70c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9240d3400 con 0x7fd92415dd90 2024-03-06T07:32:28.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.037+0000 7fd929ca7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd9240d29d0 0x7fd9240d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53530/0 (socket says 172.21.15.27:53530) 2024-03-06T07:32:28.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.037+0000 7fd929ca7640 1 -- 172.21.15.27:0/990888833 learned_addr learned my addr 172.21.15.27:0/990888833 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:28.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.037+0000 7fd92a4a8640 1 --2- 172.21.15.27:0/990888833 >> v2:172.21.15.69:3300/0 conn(0x7fd92415dd90 0x7fd9240ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.037+0000 7fd92a4a8640 1 -- 172.21.15.27:0/990888833 >> v1:172.21.15.69:6789/0 conn(0x7fd9240b6d10 legacy=0x7fd9240d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.037+0000 7fd92a4a8640 1 -- 172.21.15.27:0/990888833 >> v1:172.21.15.27:6789/0 conn(0x7fd9240d29d0 legacy=0x7fd9240d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.037+0000 7fd9294a6640 1 -- 172.21.15.27:0/990888833 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985456128 0 0) 0x7fd9240d3740 con 0x7fd9240d29d0 2024-03-06T07:32:28.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.037+0000 7fd92a4a8640 1 -- 172.21.15.27:0/990888833 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240d39e0 con 0x7fd92415dd90 2024-03-06T07:32:28.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.038+0000 7fd92a4a8640 1 --2- 172.21.15.27:0/990888833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd92415dd90 0x7fd9240ceea0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd920009720 tx=0x7fd92002b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=961d0474bdad8e58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.038+0000 7fd9294a6640 1 -- 172.21.15.27:0/990888833 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920038070 con 0x7fd92415dd90 2024-03-06T07:32:28.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.038+0000 7fd9294a6640 1 -- 172.21.15.27:0/990888833 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd920031520 con 0x7fd92415dd90 2024-03-06T07:32:28.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.038+0000 7fd9294a6640 1 -- 172.21.15.27:0/990888833 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200318f0 con 0x7fd92415dd90 2024-03-06T07:32:28.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.038+0000 7fd92c70c640 1 -- 172.21.15.27:0/990888833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd92415dd90 msgr2=0x7fd9240ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.038+0000 7fd92c70c640 1 --2- 172.21.15.27:0/990888833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd92415dd90 0x7fd9240ceea0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd920009720 tx=0x7fd92002b7c0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.039+0000 7fd92c70c640 1 -- 172.21.15.27:0/990888833 shutdown_connections 2024-03-06T07:32:28.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.039+0000 7fd92c70c640 1 --2- 172.21.15.27:0/990888833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd92415dd90 0x7fd9240ceea0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.039+0000 7fd92c70c640 1 -- 172.21.15.27:0/990888833 >> 172.21.15.27:0/990888833 conn(0x7fd924071960 msgr2=0x7fd924071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.039+0000 7fd92c70c640 1 -- 172.21.15.27:0/990888833 shutdown_connections 2024-03-06T07:32:28.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.039+0000 7fd92c70c640 1 -- 172.21.15.27:0/990888833 wait complete. 2024-03-06T07:32:28.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.041+0000 7fd92c70c640 1 Processor -- start 2024-03-06T07:32:28.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.041+0000 7fd92c70c640 1 -- start start 2024-03-06T07:32:28.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd92c70c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9240b6d10 0x7fd9241f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd92c70c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9240d29d0 0x7fd9241588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd92c70c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd92415dd90 0x7fd9241617b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd92c70c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240d2df0 con 0x7fd9240d29d0 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd92c70c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92415e170 con 0x7fd92415dd90 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd92c70c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240d48f0 con 0x7fd9240b6d10 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd929ca7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9240d29d0 0x7fd9241588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd929ca7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9240d29d0 0x7fd9241588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46988/0 (socket says 172.21.15.27:46988) 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd929ca7640 1 -- 172.21.15.27:0/253354915 learned_addr learned my addr 172.21.15.27:0/253354915 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd92a4a8640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9240b6d10 0x7fd9241f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd92aca9640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd92415dd90 0x7fd9241617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd929ca7640 1 -- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9240b6d10 msgr2=0x7fd9241f47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd929ca7640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9240b6d10 0x7fd9241f47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd929ca7640 1 -- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd92415dd90 msgr2=0x7fd9241617b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd929ca7640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd92415dd90 0x7fd9241617b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.042+0000 7fd929ca7640 1 -- 172.21.15.27:0/253354915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240c8c90 con 0x7fd9240d29d0 2024-03-06T07:32:28.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.043+0000 7fd929ca7640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9240d29d0 0x7fd9241588a0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c4a0 tx=0x7fd91400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.043+0000 7fd91affd640 1 -- 172.21.15.27:0/253354915 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914018070 con 0x7fd9240d29d0 2024-03-06T07:32:28.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.043+0000 7fd91affd640 1 -- 172.21.15.27:0/253354915 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9140074e0 con 0x7fd9240d29d0 2024-03-06T07:32:28.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.043+0000 7fd91affd640 1 -- 172.21.15.27:0/253354915 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914004a20 con 0x7fd9240d29d0 2024-03-06T07:32:28.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.043+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9240c8f70 con 0x7fd9240d29d0 2024-03-06T07:32:28.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.043+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9240cc240 con 0x7fd9240d29d0 2024-03-06T07:32:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.044+0000 7fd91affd640 1 -- 172.21.15.27:0/253354915 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd91401b070 con 0x7fd9240d29d0 2024-03-06T07:32:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.044+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9240c9360 con 0x7fd9240d29d0 2024-03-06T07:32:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.045+0000 7fd91affd640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd90806b4d0 0x7fd90806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.045+0000 7fd91affd640 1 -- 172.21.15.27:0/253354915 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9140894c0 con 0x7fd9240d29d0 2024-03-06T07:32:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.045+0000 7fd92a4a8640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd90806b4d0 0x7fd90806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.045+0000 7fd92a4a8640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd90806b4d0 0x7fd90806d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd920035000 tx=0x7fd920030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.049+0000 7fd91affd640 1 -- 172.21.15.27:0/253354915 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd914057cf0 con 0x7fd9240d29d0 2024-03-06T07:32:28.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.229+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd924072a30 con 0x7fd9240d29d0 2024-03-06T07:32:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.230+0000 7fd91affd640 1 -- 172.21.15.27:0/253354915 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v147) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd91405bd90 con 0x7fd9240d29d0 2024-03-06T07:32:28.231 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:32:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.231+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd90806b4d0 msgr2=0x7fd90806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.231+0000 7fd92c70c640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd90806b4d0 0x7fd90806d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd920035000 tx=0x7fd920030000 comp rx=0 tx=0).stop 2024-03-06T07:32:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.231+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9240d29d0 msgr2=0x7fd9241588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.231+0000 7fd92c70c640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9240d29d0 0x7fd9241588a0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c4a0 tx=0x7fd91400c960 comp rx=0 tx=0).stop 2024-03-06T07:32:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.231+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 shutdown_connections 2024-03-06T07:32:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.232+0000 7fd92c70c640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd90806b4d0 0x7fd90806d990 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.232+0000 7fd92c70c640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd92415dd90 0x7fd9241617b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.232+0000 7fd92c70c640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9240d29d0 0x7fd9241588a0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.232+0000 7fd92c70c640 1 --2- 172.21.15.27:0/253354915 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd9240b6d10 0x7fd9241f47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.232+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 >> 172.21.15.27:0/253354915 conn(0x7fd924071960 msgr2=0x7fd9240b7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.232+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 shutdown_connections 2024-03-06T07:32:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.232+0000 7fd92c70c640 1 -- 172.21.15.27:0/253354915 wait complete. 2024-03-06T07:32:28.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:28.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.352+0000 7fda53c40640 1 Processor -- start 2024-03-06T07:32:28.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.352+0000 7fda53c40640 1 -- start start 2024-03-06T07:32:28.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda53c40640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda4c0b95a0 0x7fda4c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda53c40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b9a10 0x7fda4c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda53c40640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4c0cea30 con 0x7fda4c15df00 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda53c40640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4c0cf3c0 con 0x7fda4c0b95a0 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda53c40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0cf0e0 con 0x7fda4c0b9a10 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda511db640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda4c0b95a0 0x7fda4c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda519dc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fda4c15df00 0x7fda4c0d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53536/0 (socket says 172.21.15.27:53536) 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda519dc640 1 -- 172.21.15.27:0/141815696 learned_addr learned my addr 172.21.15.27:0/141815696 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda521dd640 1 --2- 172.21.15.27:0/141815696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b9a10 0x7fda4c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda511db640 1 -- 172.21.15.27:0/141815696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b9a10 msgr2=0x7fda4c0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda511db640 1 --2- 172.21.15.27:0/141815696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b9a10 0x7fda4c0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda511db640 1 -- 172.21.15.27:0/141815696 >> v1:172.21.15.27:6789/0 conn(0x7fda4c15df00 legacy=0x7fda4c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda509da640 1 -- 172.21.15.27:0/141815696 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530757696 0 0) 0x7fda4c0cea30 con 0x7fda4c15df00 2024-03-06T07:32:28.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.353+0000 7fda511db640 1 -- 172.21.15.27:0/141815696 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c0ce470 con 0x7fda4c0b95a0 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.354+0000 7fda511db640 1 --2- 172.21.15.27:0/141815696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b95a0 0x7fda4c0b5700 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526198474123f645 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.354+0000 7fda509da640 1 -- 172.21.15.27:0/141815696 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c035020 con 0x7fda4c0b95a0 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.354+0000 7fda509da640 1 -- 172.21.15.27:0/141815696 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c031830 con 0x7fda4c0b95a0 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.354+0000 7fda509da640 1 -- 172.21.15.27:0/141815696 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c031c00 con 0x7fda4c0b95a0 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.355+0000 7fda53c40640 1 -- 172.21.15.27:0/141815696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b95a0 msgr2=0x7fda4c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.355+0000 7fda53c40640 1 --2- 172.21.15.27:0/141815696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b95a0 0x7fda4c0b5700 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.355+0000 7fda53c40640 1 -- 172.21.15.27:0/141815696 shutdown_connections 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.355+0000 7fda53c40640 1 --2- 172.21.15.27:0/141815696 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b9a10 0x7fda4c0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.355+0000 7fda53c40640 1 --2- 172.21.15.27:0/141815696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b95a0 0x7fda4c0b5700 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.355+0000 7fda53c40640 1 -- 172.21.15.27:0/141815696 >> 172.21.15.27:0/141815696 conn(0x7fda4c071960 msgr2=0x7fda4c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.355+0000 7fda53c40640 1 -- 172.21.15.27:0/141815696 shutdown_connections 2024-03-06T07:32:28.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.355+0000 7fda53c40640 1 -- 172.21.15.27:0/141815696 wait complete. 2024-03-06T07:32:28.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.356+0000 7fda53c40640 1 Processor -- start 2024-03-06T07:32:28.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.356+0000 7fda53c40640 1 -- start start 2024-03-06T07:32:28.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda53c40640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b95a0 0x7fda4c0b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda53c40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b9a10 0x7fda4c1f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda53c40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4c15df00 0x7fda4c1f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda53c40640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0cedf0 con 0x7fda4c0b9a10 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda53c40640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0cec70 con 0x7fda4c15df00 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda53c40640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0d3220 con 0x7fda4c0b95a0 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda511db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b9a10 0x7fda4c1f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda511db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b9a10 0x7fda4c1f3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47014/0 (socket says 172.21.15.27:47014) 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda511db640 1 -- 172.21.15.27:0/4107680023 learned_addr learned my addr 172.21.15.27:0/4107680023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda521dd640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4c15df00 0x7fda4c1f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda519dc640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b95a0 0x7fda4c0b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda511db640 1 -- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b95a0 msgr2=0x7fda4c0b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda511db640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b95a0 0x7fda4c0b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda511db640 1 -- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4c15df00 msgr2=0x7fda4c1f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.357+0000 7fda511db640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4c15df00 0x7fda4c1f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.358+0000 7fda511db640 1 -- 172.21.15.27:0/4107680023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c1f48b0 con 0x7fda4c0b9a10 2024-03-06T07:32:28.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.358+0000 7fda511db640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b9a10 0x7fda4c1f3b80 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.358+0000 7fda427fc640 1 -- 172.21.15.27:0/4107680023 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c035070 con 0x7fda4c0b9a10 2024-03-06T07:32:28.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.358+0000 7fda427fc640 1 -- 172.21.15.27:0/4107680023 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c004230 con 0x7fda4c0b9a10 2024-03-06T07:32:28.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.358+0000 7fda427fc640 1 -- 172.21.15.27:0/4107680023 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c03a630 con 0x7fda4c0b9a10 2024-03-06T07:32:28.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.358+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c1f4600 con 0x7fda4c0b9a10 2024-03-06T07:32:28.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.358+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c0c9e80 con 0x7fda4c0b9a10 2024-03-06T07:32:28.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.359+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c067ff0 con 0x7fda4c0b9a10 2024-03-06T07:32:28.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.359+0000 7fda427fc640 1 -- 172.21.15.27:0/4107680023 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda3c038020 con 0x7fda4c0b9a10 2024-03-06T07:32:28.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.360+0000 7fda427fc640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3006b4d0 0x7fda3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.360+0000 7fda427fc640 1 -- 172.21.15.27:0/4107680023 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda3c0abcb0 con 0x7fda4c0b9a10 2024-03-06T07:32:28.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.360+0000 7fda519dc640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3006b4d0 0x7fda3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.361+0000 7fda519dc640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3006b4d0 0x7fda3006d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fda48007fd0 tx=0x7fda48007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.364+0000 7fda427fc640 1 -- 172.21.15.27:0/4107680023 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda3c040070 con 0x7fda4c0b9a10 2024-03-06T07:32:28.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.580+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fda4c06b780 con 0x7fda4c0b9a10 2024-03-06T07:32:28.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.580+0000 7fda427fc640 1 -- 172.21.15.27:0/4107680023 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v147) v1 ==== 80+0+0 (secure 0 0 0) 0x7fda3c07a510 con 0x7fda4c0b9a10 2024-03-06T07:32:28.582 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3006b4d0 msgr2=0x7fda3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3006b4d0 0x7fda3006d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fda48007fd0 tx=0x7fda48007de0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b9a10 msgr2=0x7fda4c1f3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b9a10 0x7fda4c1f3b80 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 shutdown_connections 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda3006b4d0 0x7fda3006d990 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda4c15df00 0x7fda4c1f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda4c0b9a10 0x7fda4c1f3b80 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 --2- 172.21.15.27:0/4107680023 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda4c0b95a0 0x7fda4c0b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 >> 172.21.15.27:0/4107680023 conn(0x7fda4c071960 msgr2=0x7fda4c0ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 shutdown_connections 2024-03-06T07:32:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:28.582+0000 7fda53c40640 1 -- 172.21.15.27:0/4107680023 wait complete. 2024-03-06T07:32:28.595 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:32:28.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:28.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:28.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[83552]: fuse finished with error 0 and tester_r 0 2024-03-06T07:32:28.626 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:28.683 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:32:28.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:28.683 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:32:28.699 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-03-06T07:32:28.831 INFO:teuthology.orchestra.run.smithi069.stdout:{"message":"","return_code":0} 2024-03-06T07:32:28.831 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:32:28.831 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json cache drop 2024-03-06T07:32:28.950 INFO:teuthology.orchestra.run.smithi069.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":241,"bytes":58528}},"duration":0.0038950389999999999} 2024-03-06T07:32:28.950 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 58528, "items": 241 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.003895039, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:32:28.950 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:32:29.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.057+0000 7f7e6a1e4640 1 Processor -- start 2024-03-06T07:32:29.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.057+0000 7f7e6a1e4640 1 -- start start 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e6a1e4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e64071540 0x7f7e640b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e6a1e4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e640d3400 0x7f7e64071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e6a1e4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e64071e60 con 0x7f7e640b2b50 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e6a1e4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e640d2f70 con 0x7f7e640d3400 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e6a1e4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e640d4be0 con 0x7f7e64071540 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e5bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e640d3400 0x7f7e64071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e63fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e640b2b50 0x7f7e640d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53548/0 (socket says 172.21.15.27:53548) 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e63fff640 1 -- 172.21.15.27:0/820515187 learned_addr learned my addr 172.21.15.27:0/820515187 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e637fe640 1 --2- 172.21.15.27:0/820515187 >> v2:172.21.15.69:3300/0 conn(0x7f7e64071540 0x7f7e640b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e5bfff640 1 -- 172.21.15.27:0/820515187 >> v2:172.21.15.69:3300/0 conn(0x7f7e64071540 msgr2=0x7f7e640b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e5bfff640 1 --2- 172.21.15.27:0/820515187 >> v2:172.21.15.69:3300/0 conn(0x7f7e64071540 0x7f7e640b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e5bfff640 1 -- 172.21.15.27:0/820515187 >> v1:172.21.15.27:6789/0 conn(0x7f7e640b2b50 legacy=0x7f7e640d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e62ffd640 1 -- 172.21.15.27:0/820515187 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514834324 0 0) 0x7f7e64071e60 con 0x7f7e640b2b50 2024-03-06T07:32:29.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.058+0000 7f7e5bfff640 1 -- 172.21.15.27:0/820515187 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e640ce610 con 0x7f7e640d3400 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.059+0000 7f7e5bfff640 1 --2- 172.21.15.27:0/820515187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640d3400 0x7f7e64071920 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7e640d4da0 tx=0x7f7e4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e321e3fa02e1bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.059+0000 7f7e62ffd640 1 -- 172.21.15.27:0/820515187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c035020 con 0x7f7e640d3400 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.059+0000 7f7e62ffd640 1 -- 172.21.15.27:0/820515187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c031830 con 0x7f7e640d3400 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.059+0000 7f7e62ffd640 1 -- 172.21.15.27:0/820515187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c031c00 con 0x7f7e640d3400 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.060+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/820515187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640d3400 msgr2=0x7f7e64071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.060+0000 7f7e6a1e4640 1 --2- 172.21.15.27:0/820515187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640d3400 0x7f7e64071920 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7e640d4da0 tx=0x7f7e4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.060+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/820515187 shutdown_connections 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.060+0000 7f7e6a1e4640 1 --2- 172.21.15.27:0/820515187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640d3400 0x7f7e64071920 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.060+0000 7f7e6a1e4640 1 --2- 172.21.15.27:0/820515187 >> v2:172.21.15.69:3300/0 conn(0x7f7e64071540 0x7f7e640b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.060+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/820515187 >> 172.21.15.27:0/820515187 conn(0x7f7e640d3ca0 msgr2=0x7f7e640d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.060+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/820515187 shutdown_connections 2024-03-06T07:32:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.060+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/820515187 wait complete. 2024-03-06T07:32:29.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.061+0000 7f7e6a1e4640 1 Processor -- start 2024-03-06T07:32:29.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.061+0000 7f7e6a1e4640 1 -- start start 2024-03-06T07:32:29.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e6a1e4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e64071540 0x7f7e641534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e6a1e4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640b2b50 0x7f7e640cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e6a1e4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e640d3400 0x7f7e6415bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e6a1e4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640ce8f0 con 0x7f7e640b2b50 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e6a1e4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640d2c00 con 0x7f7e640d3400 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e6a1e4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6415e370 con 0x7f7e64071540 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e5bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640b2b50 0x7f7e640cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e5bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640b2b50 0x7f7e640cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47032/0 (socket says 172.21.15.27:47032) 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e5bfff640 1 -- 172.21.15.27:0/4090898600 learned_addr learned my addr 172.21.15.27:0/4090898600 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e63fff640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e640d3400 0x7f7e6415bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e637fe640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e64071540 0x7f7e641534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e5bfff640 1 -- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e64071540 msgr2=0x7f7e641534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e5bfff640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e64071540 0x7f7e641534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e5bfff640 1 -- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e640d3400 msgr2=0x7f7e6415bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e5bfff640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e640d3400 0x7f7e6415bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e5bfff640 1 -- 172.21.15.27:0/4090898600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e640cbca0 con 0x7f7e640b2b50 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.062+0000 7f7e5bfff640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640b2b50 0x7f7e640cb760 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009890 tx=0x7f7e4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.063+0000 7f7e60ff9640 1 -- 172.21.15.27:0/4090898600 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c040070 con 0x7f7e640b2b50 2024-03-06T07:32:29.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.063+0000 7f7e60ff9640 1 -- 172.21.15.27:0/4090898600 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c03e070 con 0x7f7e640b2b50 2024-03-06T07:32:29.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.063+0000 7f7e60ff9640 1 -- 172.21.15.27:0/4090898600 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c035040 con 0x7f7e640b2b50 2024-03-06T07:32:29.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.063+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e640b40a0 con 0x7f7e640b2b50 2024-03-06T07:32:29.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.063+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e640b4660 con 0x7f7e640b2b50 2024-03-06T07:32:29.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.064+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e64068700 con 0x7f7e640b2b50 2024-03-06T07:32:29.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.064+0000 7f7e60ff9640 1 -- 172.21.15.27:0/4090898600 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e4c038020 con 0x7f7e640b2b50 2024-03-06T07:32:29.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.065+0000 7f7e60ff9640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e4406b4d0 0x7f7e4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.065+0000 7f7e637fe640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e4406b4d0 0x7f7e4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.065+0000 7f7e60ff9640 1 -- 172.21.15.27:0/4090898600 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e4c0aba50 con 0x7f7e640b2b50 2024-03-06T07:32:29.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.065+0000 7f7e637fe640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e4406b4d0 0x7f7e4406d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7e640d0530 tx=0x7f7e54009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.070+0000 7f7e60ff9640 1 -- 172.21.15.27:0/4090898600 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e4c031800 con 0x7f7e640b2b50 2024-03-06T07:32:29.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.375+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7e6406b8e0 con 0x7f7e640b2b50 2024-03-06T07:32:29.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.375+0000 7f7e60ff9640 1 -- 172.21.15.27:0/4090898600 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v118) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7e4c07a280 con 0x7f7e640b2b50 2024-03-06T07:32:29.376 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:32:29.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e4406b4d0 msgr2=0x7f7e4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e4406b4d0 0x7f7e4406d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7e640d0530 tx=0x7f7e54009190 comp rx=0 tx=0).stop 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640b2b50 msgr2=0x7f7e640cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640b2b50 0x7f7e640cb760 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009890 tx=0x7f7e4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 shutdown_connections 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e4406b4d0 0x7f7e4406d990 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e640d3400 0x7f7e6415bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e640b2b50 0x7f7e640cb760 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 --2- 172.21.15.27:0/4090898600 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e64071540 0x7f7e641534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 >> 172.21.15.27:0/4090898600 conn(0x7f7e640d3ca0 msgr2=0x7f7e64160f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 shutdown_connections 2024-03-06T07:32:29.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:29.377+0000 7f7e6a1e4640 1 -- 172.21.15.27:0/4090898600 wait complete. 2024-03-06T07:32:29.389 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:32:29.389 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:32:29.389 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-06T07:32:29.389 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:32:29.389 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:29.389 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:32:29.389 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:32:29.389 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:32:29.389 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:29.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:29.390 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42140sec preferred_lft 42140sec 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::747e:b1ff:fe5b:5bd1/64 scope link 2024-03-06T07:32:29.407 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:29.408 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:29.408 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:20:64:bf:23:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:29.408 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7420:64ff:febf:23ba/64 scope link 2024-03-06T07:32:29.408 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:29.408 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:29.408 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether be:9e:a0:f0:d9:c0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:29.408 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::bc9e:a0ff:fef0:d9c0/64 scope link 2024-03-06T07:32:29.408 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:29.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:29.408 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:32:29.475 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-06T07:32:29.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:29.477 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-06T07:32:29.533 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-06T07:32:29.533 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:32:29.533 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:32:29.533 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:32:29.534 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:32:29.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:29.534 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-06T07:32:29.534 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-06T07:32:29.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:32:29.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:32:29.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:32:29.534 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-06T07:32:29.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:29.640 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:32:29.655 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:29.656 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:29.656 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:29.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:29.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-06T07:32:29.727 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:32:29.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:29.727 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:29.782 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:29.782 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:29.782 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:29.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:32:29.909 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:32:29.965 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:32:29.966 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:32:29.967 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:32:29.967 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:32:29.967 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:32:29.967 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:32:29.967 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:29.967 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:29.967 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:29.967 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:29.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:29.967 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:32:30.020 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:32:30.020 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:32:30.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-06T07:32:30.097 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:30.157 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:30.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.157 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-06T07:32:30.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-06T07:32:30.158+0000 7fbb6ed83480 -1 init, newargv = 0x562bce40dbe0 newargc=15 2024-03-06T07:32:30.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[85237]: starting ceph client 2024-03-06T07:32:30.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[85237]: starting fuse 2024-03-06T07:32:30.176 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-03-06T07:32:30.176 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-06T07:32:30.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:32:30.176 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-06T07:32:30.176 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-06T07:32:30.176 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-06T07:32:30.176 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-06T07:32:30.176 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:32:30.176 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-06T07:32:30.176 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:32:30.177 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-06T07:32:30.178 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-06T07:32:30.178 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-06T07:32:30.312 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.85237.asok 2024-03-06T07:32:30.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:32:30.326 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.85237.asok 2024-03-06T07:32:30.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.326 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.85237.asok status 2024-03-06T07:32:30.468 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:32:30.468 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-06T07:32:30.468 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:32:30.468 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:32:30.468 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-06T07:32:30.468 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "85237", 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 9636, 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 9636 2024-03-06T07:32:30.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:32:30.470 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:32:30.470 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3891956506 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3891956506 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.9636 192.168.0.1:0/3891956506", 2024-03-06T07:32:30.471 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/3891956506", 2024-03-06T07:32:30.472 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-06T07:32:30.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 118, 2024-03-06T07:32:30.472 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 147, 2024-03-06T07:32:30.472 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:32:30.472 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-06T07:32:30.472 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-03-06T07:32:30.472 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:32:30.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.476 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-06T07:32:30.476 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:30.491 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-06T07:32:30.492 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:30.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:30.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.567 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/original) 2024-03-06T07:32:30.627 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:32:30.628 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:30.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.828+0000 7ff87808d640 1 Processor -- start 2024-03-06T07:32:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.828+0000 7ff87808d640 1 -- start start 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff87808d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 0x7ff87006fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff87808d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff87015e080 0x7ff870066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff87808d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff870161c20 con 0x7ff8700b00c0 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff87808d640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8701619d0 con 0x7ff87015e080 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff87808d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff870161f00 con 0x7ff87015dca0 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff875628640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff87015e080 0x7ff870066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff875628640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff87015e080 0x7ff870066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47046/0 (socket says 172.21.15.27:47046) 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff875628640 1 -- 172.21.15.27:0/1194776288 learned_addr learned my addr 172.21.15.27:0/1194776288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff875e29640 1 --2- 172.21.15.27:0/1194776288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 0x7ff87006fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff875628640 1 -- 172.21.15.27:0/1194776288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 msgr2=0x7ff87006fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff875628640 1 --2- 172.21.15.27:0/1194776288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 0x7ff87006fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff875628640 1 -- 172.21.15.27:0/1194776288 >> v1:172.21.15.69:6789/0 conn(0x7ff8700b00c0 legacy=0x7ff8700719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.829+0000 7ff875628640 1 -- 172.21.15.27:0/1194776288 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8701621e0 con 0x7ff87015e080 2024-03-06T07:32:30.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.830+0000 7ff875628640 1 --2- 172.21.15.27:0/1194776288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff87015e080 0x7ff870066d00 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7ff860009870 tx=0x7ff86002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53c8ed6c09bc42f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff874e27640 1 -- 172.21.15.27:0/1194776288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860034070 con 0x7ff87015e080 2024-03-06T07:32:30.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff874e27640 1 -- 172.21.15.27:0/1194776288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff860030840 con 0x7ff87015e080 2024-03-06T07:32:30.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff874e27640 1 -- 172.21.15.27:0/1194776288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860030c10 con 0x7ff87015e080 2024-03-06T07:32:30.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff87808d640 1 -- 172.21.15.27:0/1194776288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff87015e080 msgr2=0x7ff870066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff87808d640 1 --2- 172.21.15.27:0/1194776288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff87015e080 0x7ff870066d00 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7ff860009870 tx=0x7ff86002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:30.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff87808d640 1 -- 172.21.15.27:0/1194776288 shutdown_connections 2024-03-06T07:32:30.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff87808d640 1 --2- 172.21.15.27:0/1194776288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff87015e080 0x7ff870066d00 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff87808d640 1 --2- 172.21.15.27:0/1194776288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 0x7ff87006fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff87808d640 1 -- 172.21.15.27:0/1194776288 >> 172.21.15.27:0/1194776288 conn(0x7ff8700d0990 msgr2=0x7ff8700d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff87808d640 1 -- 172.21.15.27:0/1194776288 shutdown_connections 2024-03-06T07:32:30.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.831+0000 7ff87808d640 1 -- 172.21.15.27:0/1194776288 wait complete. 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.832+0000 7ff87808d640 1 Processor -- start 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff87808d640 1 -- start start 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff87808d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8700b00c0 0x7ff8701f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff87808d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 0x7ff870160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff87808d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff870153b40 0x7ff870160b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff87808d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700703d0 con 0x7ff8700b00c0 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff87808d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff870070250 con 0x7ff870153b40 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff87808d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700b4cb0 con 0x7ff87015dca0 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff875e29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8700b00c0 0x7ff8701f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff875e29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8700b00c0 0x7ff8701f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47052/0 (socket says 172.21.15.27:47052) 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.833+0000 7ff875e29640 1 -- 172.21.15.27:0/3197973790 learned_addr learned my addr 172.21.15.27:0/3197973790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff875628640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 0x7ff870160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff87662a640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff870153b40 0x7ff870160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff875e29640 1 -- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 msgr2=0x7ff870160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff875e29640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 0x7ff870160640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff875e29640 1 -- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff870153b40 msgr2=0x7ff870160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff875e29640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff870153b40 0x7ff870160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff875e29640 1 -- 172.21.15.27:0/3197973790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87015e510 con 0x7ff8700b00c0 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff87662a640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff870153b40 0x7ff870160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:30.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff875e29640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8700b00c0 0x7ff8701f2f90 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00e9b0 tx=0x7ff86c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff8667fc640 1 -- 172.21.15.27:0/3197973790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c017070 con 0x7ff8700b00c0 2024-03-06T07:32:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff8667fc640 1 -- 172.21.15.27:0/3197973790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c00c9a0 con 0x7ff8700b00c0 2024-03-06T07:32:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87015e780 con 0x7ff8700b00c0 2024-03-06T07:32:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff8667fc640 1 -- 172.21.15.27:0/3197973790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c00cd90 con 0x7ff8700b00c0 2024-03-06T07:32:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.834+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8701f34d0 con 0x7ff8700b00c0 2024-03-06T07:32:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.835+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff870068700 con 0x7ff8700b00c0 2024-03-06T07:32:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.836+0000 7ff8667fc640 1 -- 172.21.15.27:0/3197973790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff86c01a070 con 0x7ff8700b00c0 2024-03-06T07:32:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.836+0000 7ff8667fc640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff85406b4d0 0x7ff85406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.836+0000 7ff8667fc640 1 -- 172.21.15.27:0/3197973790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff86c08a220 con 0x7ff8700b00c0 2024-03-06T07:32:30.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.836+0000 7ff875628640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff85406b4d0 0x7ff85406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.837+0000 7ff875628640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff85406b4d0 0x7ff85406d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff860036000 tx=0x7ff86003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:30.840+0000 7ff8667fc640 1 -- 172.21.15.27:0/3197973790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff86c0589b0 con 0x7ff8700b00c0 2024-03-06T07:32:31.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.022+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff87015ee40 con 0x7ff8700b00c0 2024-03-06T07:32:31.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.023+0000 7ff8667fc640 1 -- 172.21.15.27:0/3197973790 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16657 (secure 0 0 0) 0x7ff86c05ca50 con 0x7ff8700b00c0 2024-03-06T07:32:31.025 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:31.025 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":148,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:32:30.273740+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:31.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff85406b4d0 msgr2=0x7ff85406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff85406b4d0 0x7ff85406d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff860036000 tx=0x7ff86003b040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8700b00c0 msgr2=0x7ff8701f2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8700b00c0 0x7ff8701f2f90 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00e9b0 tx=0x7ff86c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 shutdown_connections 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff85406b4d0 0x7ff85406d990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff870153b40 0x7ff870160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff87015dca0 0x7ff870160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 --2- 172.21.15.27:0/3197973790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8700b00c0 0x7ff8701f2f90 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 >> 172.21.15.27:0/3197973790 conn(0x7ff8700d0990 msgr2=0x7ff870072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 shutdown_connections 2024-03-06T07:32:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.025+0000 7ff87808d640 1 -- 172.21.15.27:0/3197973790 wait complete. 2024-03-06T07:32:31.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:42.692383+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:43.367948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:31.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:31.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.146+0000 7f007a281640 1 Processor -- start 2024-03-06T07:32:31.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.146+0000 7f007a281640 1 -- start start 2024-03-06T07:32:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.147+0000 7f007a281640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f007415dbd0 0x7f007415ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.147+0000 7f007a281640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007415f3b0 con 0x7f00740b40c0 2024-03-06T07:32:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.147+0000 7f007a281640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007406f070 con 0x7f00740b5840 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.147+0000 7f007a281640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0074071940 con 0x7f007415dbd0 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.147+0000 7f007881e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f00740b40c0 0x7f0074066d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53560/0 (socket says 172.21.15.27:53560) 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.147+0000 7f007881e640 1 -- 172.21.15.27:0/3925957530 learned_addr learned my addr 172.21.15.27:0/3925957530 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.147+0000 7f0073fff640 1 --2- 172.21.15.27:0/3925957530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f007415dbd0 0x7f007415ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.148+0000 7f0072ffd640 1 -- 172.21.15.27:0/3925957530 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36667342 0 0) 0x7f007415f3b0 con 0x7f00740b40c0 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.148+0000 7f0072ffd640 1 -- 172.21.15.27:0/3925957530 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f005c003520 con 0x7f00740b40c0 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.148+0000 7f0072ffd640 1 -- 172.21.15.27:0/3925957530 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496531 0 0) 0x7f007406f070 con 0x7f00740b5840 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.148+0000 7f0072ffd640 1 -- 172.21.15.27:0/3925957530 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f007415f3b0 con 0x7f00740b5840 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.148+0000 7f0073fff640 1 -- 172.21.15.27:0/3925957530 >> v1:172.21.15.69:6789/0 conn(0x7f00740b5840 legacy=0x7f00740b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.148+0000 7f0073fff640 1 -- 172.21.15.27:0/3925957530 >> v1:172.21.15.27:6789/0 conn(0x7f00740b40c0 legacy=0x7f0074066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.148+0000 7f0073fff640 1 -- 172.21.15.27:0/3925957530 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00740b5db0 con 0x7f007415dbd0 2024-03-06T07:32:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f0073fff640 1 --2- 172.21.15.27:0/3925957530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f007415dbd0 0x7f007415ee70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f00640097d0 tx=0x7f006402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae91c5f78e73a04c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f0072ffd640 1 -- 172.21.15.27:0/3925957530 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064035020 con 0x7f007415dbd0 2024-03-06T07:32:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f0072ffd640 1 -- 172.21.15.27:0/3925957530 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064031890 con 0x7f007415dbd0 2024-03-06T07:32:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f0072ffd640 1 -- 172.21.15.27:0/3925957530 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064031c60 con 0x7f007415dbd0 2024-03-06T07:32:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f007a281640 1 -- 172.21.15.27:0/3925957530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f007415dbd0 msgr2=0x7f007415ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f007a281640 1 --2- 172.21.15.27:0/3925957530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f007415dbd0 0x7f007415ee70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f00640097d0 tx=0x7f006402b930 comp rx=0 tx=0).stop 2024-03-06T07:32:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f007a281640 1 -- 172.21.15.27:0/3925957530 shutdown_connections 2024-03-06T07:32:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f007a281640 1 --2- 172.21.15.27:0/3925957530 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f007415dbd0 0x7f007415ee70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f007a281640 1 -- 172.21.15.27:0/3925957530 >> 172.21.15.27:0/3925957530 conn(0x7f00740b26b0 msgr2=0x7f00740b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f007a281640 1 -- 172.21.15.27:0/3925957530 shutdown_connections 2024-03-06T07:32:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.149+0000 7f007a281640 1 -- 172.21.15.27:0/3925957530 wait complete. 2024-03-06T07:32:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.150+0000 7f007a281640 1 Processor -- start 2024-03-06T07:32:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007a281640 1 -- start start 2024-03-06T07:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007a281640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00740b40c0 0x7f00740b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007a281640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00740b5840 0x7f00740b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007a281640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007415dbd0 0x7f00740ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007a281640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00740721c0 con 0x7f007415dbd0 2024-03-06T07:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007a281640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0074071ee0 con 0x7f00740b40c0 2024-03-06T07:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007a281640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00740726e0 con 0x7f00740b5840 2024-03-06T07:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007881e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007415dbd0 0x7f00740ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007881e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007415dbd0 0x7f00740ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47062/0 (socket says 172.21.15.27:47062) 2024-03-06T07:32:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f0073fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00740b40c0 0x7f00740b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.151+0000 7f007881e640 1 -- 172.21.15.27:0/194196039 learned_addr learned my addr 172.21.15.27:0/194196039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f00737fe640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00740b5840 0x7f00740b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f0073fff640 1 -- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00740b5840 msgr2=0x7f00740b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f0073fff640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00740b5840 0x7f00740b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f0073fff640 1 -- 172.21.15.27:0/194196039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007415dbd0 msgr2=0x7f00740ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f0073fff640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007415dbd0 0x7f00740ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f0073fff640 1 -- 172.21.15.27:0/194196039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00740b0b90 con 0x7f00740b40c0 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f007881e640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007415dbd0 0x7f00740ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f0073fff640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00740b40c0 0x7f00740b05b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0064033000 tx=0x7f00640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f0070ff9640 1 -- 172.21.15.27:0/194196039 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064035040 con 0x7f00740b40c0 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f0070ff9640 1 -- 172.21.15.27:0/194196039 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064030b10 con 0x7f00740b40c0 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f0070ff9640 1 -- 172.21.15.27:0/194196039 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064044430 con 0x7f00740b40c0 2024-03-06T07:32:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00740b0e70 con 0x7f00740b40c0 2024-03-06T07:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.152+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00740b1430 con 0x7f00740b40c0 2024-03-06T07:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.153+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007404f410 con 0x7f00740b40c0 2024-03-06T07:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.154+0000 7f0070ff9640 1 -- 172.21.15.27:0/194196039 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0064038020 con 0x7f00740b40c0 2024-03-06T07:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.155+0000 7f0070ff9640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f006006b4d0 0x7f006006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.155+0000 7f0070ff9640 1 -- 172.21.15.27:0/194196039 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f00640ac540 con 0x7f00740b40c0 2024-03-06T07:32:31.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.155+0000 7f00737fe640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f006006b4d0 0x7f006006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.155+0000 7f00737fe640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f006006b4d0 0x7f006006d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0068007fd0 tx=0x7f0068009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.159+0000 7f0070ff9640 1 -- 172.21.15.27:0/194196039 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f006402d040 con 0x7f00740b40c0 2024-03-06T07:32:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.365+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0074068830 con 0x7f00740b40c0 2024-03-06T07:32:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.366+0000 7f0070ff9640 1 -- 172.21.15.27:0/194196039 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) v1 ==== 93+0+10 (secure 0 0 0) 0x7f006407acc0 con 0x7f00740b40c0 2024-03-06T07:32:31.367 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f006006b4d0 msgr2=0x7f006006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f006006b4d0 0x7f006006d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0068007fd0 tx=0x7f0068009040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00740b40c0 msgr2=0x7f00740b05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00740b40c0 0x7f00740b05b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0064033000 tx=0x7f00640046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 shutdown_connections 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f006006b4d0 0x7f006006d990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007415dbd0 0x7f00740ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f00740b5840 0x7f00740b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 --2- 172.21.15.27:0/194196039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00740b40c0 0x7f00740b05b0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 >> 172.21.15.27:0/194196039 conn(0x7f00740b26b0 msgr2=0x7f007415e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 shutdown_connections 2024-03-06T07:32:31.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.369+0000 7f007a281640 1 -- 172.21.15.27:0/194196039 wait complete. 2024-03-06T07:32:31.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.495+0000 7f5ec3e39640 1 Processor -- start 2024-03-06T07:32:31.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.495+0000 7f5ec3e39640 1 -- start start 2024-03-06T07:32:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.495+0000 7f5ec3e39640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc071540 0x7f5ebc071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec3e39640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ebc0b2d30 0x7f5ebc0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec3e39640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc071e60 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec3e39640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ebc0d3230 con 0x7f5ebc0b2d30 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec3e39640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc0d4960 con 0x7f5ebc071540 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec13d4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ebc0b2d30 0x7f5ebc0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec23d6640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5ebc0cecb0 0x7f5ebc0b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:50612/0 (socket says 172.21.15.27:50612) 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec23d6640 1 -- 172.21.15.27:0/2570625389 learned_addr learned my addr 172.21.15.27:0/2570625389 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec1bd5640 1 --2- 172.21.15.27:0/2570625389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc071540 0x7f5ebc071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec0bd3640 1 -- 172.21.15.27:0/2570625389 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982076174 0 0) 0x7f5ebc071e60 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec0bd3640 1 -- 172.21.15.27:0/2570625389 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea8003520 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec13d4640 1 -- 172.21.15.27:0/2570625389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc071540 msgr2=0x7f5ebc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec13d4640 1 --2- 172.21.15.27:0/2570625389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc071540 0x7f5ebc071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec13d4640 1 -- 172.21.15.27:0/2570625389 >> v1:172.21.15.69:6789/0 conn(0x7f5ebc0cecb0 legacy=0x7f5ebc0b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec13d4640 1 -- 172.21.15.27:0/2570625389 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc0d36c0 con 0x7f5ebc0b2d30 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.496+0000 7f5ec1bd5640 1 --2- 172.21.15.27:0/2570625389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc071540 0x7f5ebc071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.497+0000 7f5ec13d4640 1 --2- 172.21.15.27:0/2570625389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0b2d30 0x7f5ebc0ce770 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0097a0 tx=0x7f5eac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47bedf2c6961988e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.497+0000 7f5ec0bd3640 1 -- 172.21.15.27:0/2570625389 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac035020 con 0x7f5ebc0b2d30 2024-03-06T07:32:31.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.497+0000 7f5ec0bd3640 1 -- 172.21.15.27:0/2570625389 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eac031830 con 0x7f5ebc0b2d30 2024-03-06T07:32:31.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.497+0000 7f5ec0bd3640 1 -- 172.21.15.27:0/2570625389 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac031c00 con 0x7f5ebc0b2d30 2024-03-06T07:32:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.498+0000 7f5ec3e39640 1 -- 172.21.15.27:0/2570625389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0b2d30 msgr2=0x7f5ebc0ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.498+0000 7f5ec3e39640 1 --2- 172.21.15.27:0/2570625389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0b2d30 0x7f5ebc0ce770 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0097a0 tx=0x7f5eac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.498+0000 7f5ec3e39640 1 -- 172.21.15.27:0/2570625389 shutdown_connections 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.498+0000 7f5ec3e39640 1 --2- 172.21.15.27:0/2570625389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0b2d30 0x7f5ebc0ce770 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.498+0000 7f5ec3e39640 1 --2- 172.21.15.27:0/2570625389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc071540 0x7f5ebc071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.498+0000 7f5ec3e39640 1 -- 172.21.15.27:0/2570625389 >> 172.21.15.27:0/2570625389 conn(0x7f5ebc0d3d60 msgr2=0x7f5ebc0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.498+0000 7f5ec3e39640 1 -- 172.21.15.27:0/2570625389 shutdown_connections 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.498+0000 7f5ec3e39640 1 -- 172.21.15.27:0/2570625389 wait complete. 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.499+0000 7f5ec3e39640 1 Processor -- start 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.499+0000 7f5ec3e39640 1 -- start start 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec3e39640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ebc071540 0x7f5ebc1e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec3e39640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc0b2d30 0x7f5ebc0cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec3e39640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0cecb0 0x7f5ebc1f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec3e39640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc0d2be0 con 0x7f5ebc071540 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec3e39640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc0d2900 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec3e39640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc0d2ec0 con 0x7f5ebc0b2d30 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec1bd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ebc071540 0x7f5ebc1e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec1bd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ebc071540 0x7f5ebc1e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47078/0 (socket says 172.21.15.27:47078) 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec1bd5640 1 -- 172.21.15.27:0/1593485275 learned_addr learned my addr 172.21.15.27:0/1593485275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec23d6640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0cecb0 0x7f5ebc1f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec13d4640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc0b2d30 0x7f5ebc0cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec23d6640 1 -- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc0b2d30 msgr2=0x7f5ebc0cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec23d6640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc0b2d30 0x7f5ebc0cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec23d6640 1 -- 172.21.15.27:0/1593485275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ebc071540 msgr2=0x7f5ebc1e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec23d6640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ebc071540 0x7f5ebc1e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec23d6640 1 -- 172.21.15.27:0/1593485275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc1f50f0 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec1bd5640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ebc071540 0x7f5ebc1e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.500+0000 7f5ec23d6640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0cecb0 0x7f5ebc1f4bb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400cab0 tx=0x7f5eb400cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.501+0000 7f5eb27fc640 1 -- 172.21.15.27:0/1593485275 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4017070 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.501+0000 7f5eb27fc640 1 -- 172.21.15.27:0/1593485275 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb400ac30 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.501+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ebc0cb280 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.501+0000 7f5eb27fc640 1 -- 172.21.15.27:0/1593485275 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4012600 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.501+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ebc0cb840 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.502+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ebc074c70 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.502+0000 7f5eb27fc640 1 -- 172.21.15.27:0/1593485275 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5eb401a070 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.502+0000 7f5eb27fc640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ea006b4d0 0x7f5ea006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.502+0000 7f5eb27fc640 1 -- 172.21.15.27:0/1593485275 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5eb4089f20 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.502+0000 7f5ec1bd5640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ea006b4d0 0x7f5ea006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.503+0000 7f5ec1bd5640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ea006b4d0 0x7f5ea006d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5eb80095e0 tx=0x7f5eb8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.507+0000 7f5eb27fc640 1 -- 172.21.15.27:0/1593485275 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ebc074c70 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.689+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ebc0c6820 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.689+0000 7f5eb27fc640 1 -- 172.21.15.27:0/1593485275 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5ebc0c6820 con 0x7f5ebc0cecb0 2024-03-06T07:32:31.691 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:32:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ea006b4d0 msgr2=0x7f5ea006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ea006b4d0 0x7f5ea006d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5eb80095e0 tx=0x7f5eb8009190 comp rx=0 tx=0).stop 2024-03-06T07:32:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0cecb0 msgr2=0x7f5ebc1f4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0cecb0 0x7f5ebc1f4bb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400cab0 tx=0x7f5eb400cf70 comp rx=0 tx=0).stop 2024-03-06T07:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 shutdown_connections 2024-03-06T07:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ea006b4d0 0x7f5ea006d990 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ebc0cecb0 0x7f5ebc1f4bb0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ebc0b2d30 0x7f5ebc0cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 --2- 172.21.15.27:0/1593485275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ebc071540 0x7f5ebc1e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 >> 172.21.15.27:0/1593485275 conn(0x7f5ebc0d3d60 msgr2=0x7f5ebc15bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 shutdown_connections 2024-03-06T07:32:31.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.692+0000 7f5ec3e39640 1 -- 172.21.15.27:0/1593485275 wait complete. 2024-03-06T07:32:31.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:31.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.812+0000 7f04f1e83640 1 Processor -- start 2024-03-06T07:32:31.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.812+0000 7f04f1e83640 1 -- start start 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04f1e83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 0x7f04ec071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04f1e83640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04ec0b2d30 0x7f04ec0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04f1e83640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04ec0cecb0 0x7f04ec0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04f1e83640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec0d3230 con 0x7f04ec0b2d30 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04f1e83640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec071e60 con 0x7f04ec0cecb0 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04f1e83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0d4960 con 0x7f04ec071540 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04eaffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04ec0b2d30 0x7f04ec0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04eaffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04ec0b2d30 0x7f04ec0ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47094/0 (socket says 172.21.15.27:47094) 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04eaffd640 1 -- 172.21.15.27:0/4063082498 learned_addr learned my addr 172.21.15.27:0/4063082498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04ebfff640 1 --2- 172.21.15.27:0/4063082498 >> v2:172.21.15.69:3300/0 conn(0x7f04ec0cecb0 0x7f04ec0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04eb7fe640 1 --2- 172.21.15.27:0/4063082498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 0x7f04ec071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04eaffd640 1 -- 172.21.15.27:0/4063082498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 msgr2=0x7f04ec071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04eaffd640 1 --2- 172.21.15.27:0/4063082498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 0x7f04ec071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04eaffd640 1 -- 172.21.15.27:0/4063082498 >> v2:172.21.15.69:3300/0 conn(0x7f04ec0cecb0 msgr2=0x7f04ec0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04eaffd640 1 --2- 172.21.15.27:0/4063082498 >> v2:172.21.15.69:3300/0 conn(0x7f04ec0cecb0 0x7f04ec0cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.813+0000 7f04eaffd640 1 -- 172.21.15.27:0/4063082498 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec0d36c0 con 0x7f04ec0b2d30 2024-03-06T07:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.814+0000 7f04eaffd640 1 --2- 172.21.15.27:0/4063082498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 0x7f04ec0ce770 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f04ec068380 tx=0x7f04e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07d9d3c2bc7d928 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.814+0000 7f04ea7fc640 1 -- 172.21.15.27:0/4063082498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0035020 con 0x7f04ec0b2d30 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.814+0000 7f04ea7fc640 1 -- 172.21.15.27:0/4063082498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e0031870 con 0x7f04ec0b2d30 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.814+0000 7f04ea7fc640 1 -- 172.21.15.27:0/4063082498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0031c40 con 0x7f04ec0b2d30 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.815+0000 7f04f1e83640 1 -- 172.21.15.27:0/4063082498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 msgr2=0x7f04ec0ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.815+0000 7f04f1e83640 1 --2- 172.21.15.27:0/4063082498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 0x7f04ec0ce770 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f04ec068380 tx=0x7f04e002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.815+0000 7f04f1e83640 1 -- 172.21.15.27:0/4063082498 shutdown_connections 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.815+0000 7f04f1e83640 1 --2- 172.21.15.27:0/4063082498 >> v2:172.21.15.69:3300/0 conn(0x7f04ec0cecb0 0x7f04ec0cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.815+0000 7f04f1e83640 1 --2- 172.21.15.27:0/4063082498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 0x7f04ec0ce770 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.815+0000 7f04f1e83640 1 --2- 172.21.15.27:0/4063082498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 0x7f04ec071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.815+0000 7f04f1e83640 1 -- 172.21.15.27:0/4063082498 >> 172.21.15.27:0/4063082498 conn(0x7f04ec0d3d60 msgr2=0x7f04ec0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.815+0000 7f04f1e83640 1 -- 172.21.15.27:0/4063082498 shutdown_connections 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.815+0000 7f04f1e83640 1 -- 172.21.15.27:0/4063082498 wait complete. 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.816+0000 7f04f1e83640 1 Processor -- start 2024-03-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.816+0000 7f04f1e83640 1 -- start start 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04f1e83640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 0x7f04ec1f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04f1e83640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 0x7f04ec0caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04f1e83640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec0cecb0 0x7f04ec0d1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04f1e83640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0d2be0 con 0x7f04ec0b2d30 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04f1e83640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0d2900 con 0x7f04ec0cecb0 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04f1e83640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0d2ec0 con 0x7f04ec071540 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 0x7f04ec0caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 0x7f04ec0caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47096/0 (socket says 172.21.15.27:47096) 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04eaffd640 1 -- 172.21.15.27:0/1443561316 learned_addr learned my addr 172.21.15.27:0/1443561316 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04eb7fe640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 0x7f04ec1f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04ebfff640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec0cecb0 0x7f04ec0d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.817+0000 7f04eaffd640 1 -- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 msgr2=0x7f04ec1f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04eaffd640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 0x7f04ec1f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04eaffd640 1 -- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec0cecb0 msgr2=0x7f04ec0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04eaffd640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec0cecb0 0x7f04ec0d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04eaffd640 1 -- 172.21.15.27:0/1443561316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec0cb910 con 0x7f04ec0b2d30 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04eb7fe640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 0x7f04ec1f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04eaffd640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 0x7f04ec0caee0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f04e002b950 tx=0x7f04e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04d3fff640 1 -- 172.21.15.27:0/1443561316 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0040070 con 0x7f04ec0b2d30 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04d3fff640 1 -- 172.21.15.27:0/1443561316 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e0031e10 con 0x7f04ec0b2d30 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04d3fff640 1 -- 172.21.15.27:0/1443561316 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0035040 con 0x7f04ec0b2d30 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec0cb420 con 0x7f04ec0b2d30 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.818+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec1569a0 con 0x7f04ec0b2d30 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.819+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec072070 con 0x7f04ec0b2d30 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.819+0000 7f04d3fff640 1 -- 172.21.15.27:0/1443561316 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04e004b060 con 0x7f04ec0b2d30 2024-03-06T07:32:31.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.820+0000 7f04d3fff640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f04d806b4d0 0x7f04d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.820+0000 7f04d3fff640 1 -- 172.21.15.27:0/1443561316 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f04e00abd70 con 0x7f04ec0b2d30 2024-03-06T07:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.820+0000 7f04eb7fe640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f04d806b4d0 0x7f04d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.820+0000 7f04eb7fe640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f04d806b4d0 0x7f04d806d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009e10 tx=0x7f04dc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:31.824+0000 7f04d3fff640 1 -- 172.21.15.27:0/1443561316 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04e007a580 con 0x7f04ec0b2d30 2024-03-06T07:32:32.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.004+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f04ec04f410 con 0x7f04ec0b2d30 2024-03-06T07:32:32.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.004+0000 7f04d3fff640 1 -- 172.21.15.27:0/1443561316 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+11 (secure 0 0 0) 0x7f04e007e620 con 0x7f04ec0b2d30 2024-03-06T07:32:32.006 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:32:32.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f04d806b4d0 msgr2=0x7f04d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f04d806b4d0 0x7f04d806d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009e10 tx=0x7f04dc009190 comp rx=0 tx=0).stop 2024-03-06T07:32:32.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 msgr2=0x7f04ec0caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 0x7f04ec0caee0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f04e002b950 tx=0x7f04e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 shutdown_connections 2024-03-06T07:32:32.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f04d806b4d0 0x7f04d806d990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec0cecb0 0x7f04ec0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04ec0b2d30 0x7f04ec0caee0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 --2- 172.21.15.27:0/1443561316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f04ec071540 0x7f04ec1f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.006+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 >> 172.21.15.27:0/1443561316 conn(0x7f04ec0d3d60 msgr2=0x7f04ec15b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.007+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 shutdown_connections 2024-03-06T07:32:32.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.007+0000 7f04f1e83640 1 -- 172.21.15.27:0/1443561316 wait complete. 2024-03-06T07:32:32.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.126+0000 7f8f853bc640 1 Processor -- start 2024-03-06T07:32:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.126+0000 7f8f853bc640 1 -- start start 2024-03-06T07:32:32.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.127+0000 7f8f853bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f8015df70 0x7f8f800d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.127+0000 7f8f853bc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8f80071c50 0x7f8f800b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.127+0000 7f8f853bc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f800b6f20 con 0x7f8f800d18c0 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.127+0000 7f8f853bc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800d3030 con 0x7f8f80071c50 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.127+0000 7f8f853bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800d3310 con 0x7f8f8015df70 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8f80071c50 0x7f8f800b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f8015df70 0x7f8f800d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f8015df70 0x7f8f800d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:60452/0 (socket says 172.21.15.27:60452) 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7e7fc640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8f800d18c0 0x7f8f80071540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:50618/0 (socket says 172.21.15.27:50618) 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7f7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8f80071c50 0x7f8f800b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:35314/0 (socket says 172.21.15.27:35314) 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7effd640 1 -- 172.21.15.27:0/3230818818 learned_addr learned my addr 172.21.15.27:0/3230818818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:32.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7f7fe640 1 -- 172.21.15.27:0/3230818818 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f8015df70 msgr2=0x7f8f800d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7f7fe640 1 --2- 172.21.15.27:0/3230818818 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f8015df70 0x7f8f800d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7f7fe640 1 -- 172.21.15.27:0/3230818818 >> v1:172.21.15.69:6789/0 conn(0x7f8f800d18c0 legacy=0x7f8f80071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7f7fe640 1 -- 172.21.15.27:0/3230818818 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f800d35f0 con 0x7f8f80071c50 2024-03-06T07:32:32.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.128+0000 7f8f7dffb640 1 -- 172.21.15.27:0/3230818818 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722512799 0 0) 0x7f8f800b6f20 con 0x7f8f800d18c0 2024-03-06T07:32:32.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.129+0000 7f8f7f7fe640 1 --2- 172.21.15.27:0/3230818818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f80071c50 0x7f8f800b2700 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009d90 tx=0x7f8f7802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5e33cfbdbe75c9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.129+0000 7f8f7dffb640 1 -- 172.21.15.27:0/3230818818 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f78039070 con 0x7f8f80071c50 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.129+0000 7f8f7dffb640 1 -- 172.21.15.27:0/3230818818 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7802cce0 con 0x7f8f80071c50 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.129+0000 7f8f7dffb640 1 -- 172.21.15.27:0/3230818818 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f78034660 con 0x7f8f80071c50 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.129+0000 7f8f853bc640 1 -- 172.21.15.27:0/3230818818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f80071c50 msgr2=0x7f8f800b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.129+0000 7f8f853bc640 1 --2- 172.21.15.27:0/3230818818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f80071c50 0x7f8f800b2700 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009d90 tx=0x7f8f7802bf30 comp rx=0 tx=0).stop 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.129+0000 7f8f853bc640 1 -- 172.21.15.27:0/3230818818 shutdown_connections 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.129+0000 7f8f853bc640 1 --2- 172.21.15.27:0/3230818818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f80071c50 0x7f8f800b2700 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.129+0000 7f8f853bc640 1 --2- 172.21.15.27:0/3230818818 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f8015df70 0x7f8f800d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.130+0000 7f8f853bc640 1 -- 172.21.15.27:0/3230818818 >> 172.21.15.27:0/3230818818 conn(0x7f8f800d3cf0 msgr2=0x7f8f800d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.130+0000 7f8f853bc640 1 -- 172.21.15.27:0/3230818818 shutdown_connections 2024-03-06T07:32:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.130+0000 7f8f853bc640 1 -- 172.21.15.27:0/3230818818 wait complete. 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.131+0000 7f8f853bc640 1 Processor -- start 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.131+0000 7f8f853bc640 1 -- start start 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f853bc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80071c50 0x7f8f80150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f853bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f800d18c0 0x7f8f800acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f853bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8015df70 0x7f8f800ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f853bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8015e350 con 0x7f8f800d18c0 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f853bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800d2cf0 con 0x7f8f8015df70 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f853bc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8015e4d0 con 0x7f8f80071c50 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f800d18c0 0x7f8f800acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80071c50 0x7f8f80150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f800d18c0 0x7f8f800acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47110/0 (socket says 172.21.15.27:47110) 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7e7fc640 1 -- 172.21.15.27:0/3919447106 learned_addr learned my addr 172.21.15.27:0/3919447106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7f7fe640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8015df70 0x7f8f800ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7effd640 1 -- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8015df70 msgr2=0x7f8f800ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7effd640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8015df70 0x7f8f800ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7effd640 1 -- 172.21.15.27:0/3919447106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f800d18c0 msgr2=0x7f8f800acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7effd640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f800d18c0 0x7f8f800acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7effd640 1 -- 172.21.15.27:0/3919447106 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f80150690 con 0x7f8f80071c50 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7e7fc640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f800d18c0 0x7f8f800acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.132+0000 7f8f7effd640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80071c50 0x7f8f80150150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000e8a0 tx=0x7f8f7000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.133+0000 7f8f677fe640 1 -- 172.21.15.27:0/3919447106 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70017070 con 0x7f8f80071c50 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.133+0000 7f8f677fe640 1 -- 172.21.15.27:0/3919447106 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7000c940 con 0x7f8f80071c50 2024-03-06T07:32:32.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.133+0000 7f8f677fe640 1 -- 172.21.15.27:0/3919447106 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70009e60 con 0x7f8f80071c50 2024-03-06T07:32:32.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.133+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f800cd170 con 0x7f8f80071c50 2024-03-06T07:32:32.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.133+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f800cd730 con 0x7f8f80071c50 2024-03-06T07:32:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.134+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f80071770 con 0x7f8f80071c50 2024-03-06T07:32:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.134+0000 7f8f677fe640 1 -- 172.21.15.27:0/3919447106 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f7001a070 con 0x7f8f80071c50 2024-03-06T07:32:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.135+0000 7f8f677fe640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f6c06b4d0 0x7f8f6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.135+0000 7f8f677fe640 1 -- 172.21.15.27:0/3919447106 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8f70089fb0 con 0x7f8f80071c50 2024-03-06T07:32:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.135+0000 7f8f7e7fc640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f6c06b4d0 0x7f8f6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.135+0000 7f8f7e7fc640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f6c06b4d0 0x7f8f6c06d990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8f800b2760 tx=0x7f8f74007e10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.139+0000 7f8f677fe640 1 -- 172.21.15.27:0/3919447106 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f80071770 con 0x7f8f80071c50 2024-03-06T07:32:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.336+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8f80067ff0 con 0x7f8f80071c50 2024-03-06T07:32:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.337+0000 7f8f677fe640 1 -- 172.21.15.27:0/3919447106 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v149) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8f80150690 con 0x7f8f80071c50 2024-03-06T07:32:32.338 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:32:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f6c06b4d0 msgr2=0x7f8f6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f6c06b4d0 0x7f8f6c06d990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8f800b2760 tx=0x7f8f74007e10 comp rx=0 tx=0).stop 2024-03-06T07:32:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80071c50 msgr2=0x7f8f80150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80071c50 0x7f8f80150150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000e8a0 tx=0x7f8f7000ed60 comp rx=0 tx=0).stop 2024-03-06T07:32:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 shutdown_connections 2024-03-06T07:32:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8f6c06b4d0 0x7f8f6c06d990 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f8015df70 0x7f8f800ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f800d18c0 0x7f8f800acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 --2- 172.21.15.27:0/3919447106 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8f80071c50 0x7f8f80150150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 >> 172.21.15.27:0/3919447106 conn(0x7f8f800d3cf0 msgr2=0x7f8f800b5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 shutdown_connections 2024-03-06T07:32:32.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.339+0000 7f8f853bc640 1 -- 172.21.15.27:0/3919447106 wait complete. 2024-03-06T07:32:32.355 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:32:32.355 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:32.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:32.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.516+0000 7fc396488640 1 Processor -- start 2024-03-06T07:32:32.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.516+0000 7fc396488640 1 -- start start 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc396488640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc39015df00 0x7fc3900d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc396488640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3900b9a30 0x7fc3900d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc396488640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3900cea30 con 0x7fc3900b95a0 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc396488640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3900d2af0 con 0x7fc3900b9a30 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc396488640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3900cf3c0 con 0x7fc39015df00 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc394a25640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3900b9a30 0x7fc3900d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc38ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc39015df00 0x7fc3900d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc38ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc39015df00 0x7fc3900d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:35326/0 (socket says 172.21.15.27:35326) 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc394a25640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3900b9a30 0x7fc3900d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47112/0 (socket says 172.21.15.27:47112) 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc38ffff640 1 -- 172.21.15.27:0/2419600425 learned_addr learned my addr 172.21.15.27:0/2419600425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc38f7fe640 1 --1- 172.21.15.27:0/2419600425 >> v1:172.21.15.69:6789/0 conn(0x7fc3900b95a0 0x7fc3900b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:50628/0 (socket says 172.21.15.27:50628) 2024-03-06T07:32:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc38ffff640 1 -- 172.21.15.27:0/2419600425 >> v2:172.21.15.27:3300/0 conn(0x7fc3900b9a30 msgr2=0x7fc3900d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc38ffff640 1 --2- 172.21.15.27:0/2419600425 >> v2:172.21.15.27:3300/0 conn(0x7fc3900b9a30 0x7fc3900d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc38ffff640 1 -- 172.21.15.27:0/2419600425 >> v1:172.21.15.69:6789/0 conn(0x7fc3900b95a0 legacy=0x7fc3900b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc38ffff640 1 -- 172.21.15.27:0/2419600425 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3900cf0e0 con 0x7fc39015df00 2024-03-06T07:32:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.517+0000 7fc38effd640 1 -- 172.21.15.27:0/2419600425 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375380327 0 0) 0x7fc3900cea30 con 0x7fc3900b95a0 2024-03-06T07:32:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.518+0000 7fc38ffff640 1 --2- 172.21.15.27:0/2419600425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015df00 0x7fc3900d25b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc3900d11b0 tx=0x7fc37802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=284db8396bbd59c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.518+0000 7fc38effd640 1 -- 172.21.15.27:0/2419600425 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc378034070 con 0x7fc39015df00 2024-03-06T07:32:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.518+0000 7fc38effd640 1 -- 172.21.15.27:0/2419600425 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc378030860 con 0x7fc39015df00 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.518+0000 7fc38effd640 1 -- 172.21.15.27:0/2419600425 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc378030c30 con 0x7fc39015df00 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.519+0000 7fc396488640 1 -- 172.21.15.27:0/2419600425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015df00 msgr2=0x7fc3900d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.519+0000 7fc396488640 1 --2- 172.21.15.27:0/2419600425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015df00 0x7fc3900d25b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc3900d11b0 tx=0x7fc37802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.519+0000 7fc396488640 1 -- 172.21.15.27:0/2419600425 shutdown_connections 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.519+0000 7fc396488640 1 --2- 172.21.15.27:0/2419600425 >> v2:172.21.15.27:3300/0 conn(0x7fc3900b9a30 0x7fc3900d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.519+0000 7fc396488640 1 --2- 172.21.15.27:0/2419600425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015df00 0x7fc3900d25b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.519+0000 7fc396488640 1 -- 172.21.15.27:0/2419600425 >> 172.21.15.27:0/2419600425 conn(0x7fc390071960 msgr2=0x7fc390071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.519+0000 7fc396488640 1 -- 172.21.15.27:0/2419600425 shutdown_connections 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.519+0000 7fc396488640 1 -- 172.21.15.27:0/2419600425 wait complete. 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.520+0000 7fc396488640 1 Processor -- start 2024-03-06T07:32:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.520+0000 7fc396488640 1 -- start start 2024-03-06T07:32:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc396488640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3900b95a0 0x7fc39014e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc396488640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015df00 0x7fc3901f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc396488640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390161280 0x7fc3900ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc396488640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900cec70 con 0x7fc390161280 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc396488640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900ce750 con 0x7fc3900b95a0 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc396488640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900cedf0 con 0x7fc39015df00 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc394a25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390161280 0x7fc3900ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc38ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3900b95a0 0x7fc39014e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc394a25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390161280 0x7fc3900ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47116/0 (socket says 172.21.15.27:47116) 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc394a25640 1 -- 172.21.15.27:0/1343635809 learned_addr learned my addr 172.21.15.27:0/1343635809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc38f7fe640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015df00 0x7fc3901f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc394a25640 1 -- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015df00 msgr2=0x7fc3901f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc394a25640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015df00 0x7fc3901f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc394a25640 1 -- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3900b95a0 msgr2=0x7fc39014e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.521+0000 7fc394a25640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3900b95a0 0x7fc39014e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.522+0000 7fc394a25640 1 -- 172.21.15.27:0/1343635809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3901601e0 con 0x7fc390161280 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.522+0000 7fc394a25640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390161280 0x7fc3900ccf20 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fc388009dd0 tx=0x7fc38800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.522+0000 7fc38cff9640 1 -- 172.21.15.27:0/1343635809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388018020 con 0x7fc390161280 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.522+0000 7fc38cff9640 1 -- 172.21.15.27:0/1343635809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc388014480 con 0x7fc390161280 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.522+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3901604c0 con 0x7fc390161280 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.522+0000 7fc38cff9640 1 -- 172.21.15.27:0/1343635809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388014870 con 0x7fc390161280 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.522+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc390160a80 con 0x7fc390161280 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.523+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc390067ff0 con 0x7fc390161280 2024-03-06T07:32:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.523+0000 7fc38cff9640 1 -- 172.21.15.27:0/1343635809 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc388005d10 con 0x7fc390161280 2024-03-06T07:32:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.524+0000 7fc38cff9640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b500 0x7fc37006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.524+0000 7fc38cff9640 1 -- 172.21.15.27:0/1343635809 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc38808ab50 con 0x7fc390161280 2024-03-06T07:32:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.524+0000 7fc38ffff640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b500 0x7fc37006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.525+0000 7fc38ffff640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b500 0x7fc37006d9c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc378036000 tx=0x7fc37803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.528+0000 7fc38cff9640 1 -- 172.21.15.27:0/1343635809 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc388022070 con 0x7fc390161280 2024-03-06T07:32:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.709+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc39015d260 con 0x7fc390161280 2024-03-06T07:32:32.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.710+0000 7fc38cff9640 1 -- 172.21.15.27:0/1343635809 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16657 (secure 0 0 0) 0x7fc388010040 con 0x7fc390161280 2024-03-06T07:32:32.712 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:32.712 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":149,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:32:31.280155+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:32.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.712+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b500 msgr2=0x7fc37006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.712+0000 7fc396488640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b500 0x7fc37006d9c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc378036000 tx=0x7fc37803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.712+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390161280 msgr2=0x7fc3900ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.712+0000 7fc396488640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390161280 0x7fc3900ccf20 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fc388009dd0 tx=0x7fc38800c490 comp rx=0 tx=0).stop 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.712+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 shutdown_connections 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.712+0000 7fc396488640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37006b500 0x7fc37006d9c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.713+0000 7fc396488640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc390161280 0x7fc3900ccf20 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.713+0000 7fc396488640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc39015df00 0x7fc3901f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.713+0000 7fc396488640 1 --2- 172.21.15.27:0/1343635809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3900b95a0 0x7fc39014e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.713+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 >> 172.21.15.27:0/1343635809 conn(0x7fc390071960 msgr2=0x7fc3900d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.713+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 shutdown_connections 2024-03-06T07:32:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.713+0000 7fc396488640 1 -- 172.21.15.27:0/1343635809 wait complete. 2024-03-06T07:32:32.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:42.692383+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:43.367948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:32.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.859+0000 7f3a91264640 1 Processor -- start 2024-03-06T07:32:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.859+0000 7f3a91264640 1 -- start start 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a91264640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a8c0b2cd0 0x7f3a8c0b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a91264640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 0x7f3a8c06b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a91264640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8c161a40 con 0x7f3a8c0cdcd0 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a91264640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a8c0d11e0 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a91264640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c0b2a20 con 0x7f3a8c06b2c0 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a83fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a8c0b2cd0 0x7f3a8c0b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a8affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3a8c0cdcd0 0x7f3a8c0b1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53574/0 (socket says 172.21.15.27:53574) 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a8affd640 1 -- 172.21.15.27:0/2303139792 learned_addr learned my addr 172.21.15.27:0/2303139792 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a8b7fe640 1 --2- 172.21.15.27:0/2303139792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 0x7f3a8c06b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a83fff640 1 -- 172.21.15.27:0/2303139792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 msgr2=0x7f3a8c06b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a83fff640 1 --2- 172.21.15.27:0/2303139792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 0x7f3a8c06b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a83fff640 1 -- 172.21.15.27:0/2303139792 >> v1:172.21.15.27:6789/0 conn(0x7f3a8c0cdcd0 legacy=0x7f3a8c0b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a83fff640 1 -- 172.21.15.27:0/2303139792 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c0d0940 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.860+0000 7f3a8a7fc640 1 -- 172.21.15.27:0/2303139792 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465793926 0 0) 0x7f3a8c161a40 con 0x7f3a8c0cdcd0 2024-03-06T07:32:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.861+0000 7f3a83fff640 1 --2- 172.21.15.27:0/2303139792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 0x7f3a8c0b1c60 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f3a740097a0 tx=0x7f3a7402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d23e7a12c23603 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.861+0000 7f3a8a7fc640 1 -- 172.21.15.27:0/2303139792 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a740315a0 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.861+0000 7f3a8a7fc640 1 -- 172.21.15.27:0/2303139792 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a74037070 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.861+0000 7f3a8a7fc640 1 -- 172.21.15.27:0/2303139792 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74031960 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.862+0000 7f3a91264640 1 -- 172.21.15.27:0/2303139792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 msgr2=0x7f3a8c0b1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.862+0000 7f3a91264640 1 --2- 172.21.15.27:0/2303139792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 0x7f3a8c0b1c60 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f3a740097a0 tx=0x7f3a7402b840 comp rx=0 tx=0).stop 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.862+0000 7f3a91264640 1 -- 172.21.15.27:0/2303139792 shutdown_connections 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.862+0000 7f3a91264640 1 --2- 172.21.15.27:0/2303139792 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 0x7f3a8c06b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.862+0000 7f3a91264640 1 --2- 172.21.15.27:0/2303139792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 0x7f3a8c0b1c60 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.862+0000 7f3a91264640 1 -- 172.21.15.27:0/2303139792 >> 172.21.15.27:0/2303139792 conn(0x7f3a8c0b3700 msgr2=0x7f3a8c0b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.862+0000 7f3a91264640 1 -- 172.21.15.27:0/2303139792 shutdown_connections 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.862+0000 7f3a91264640 1 -- 172.21.15.27:0/2303139792 wait complete. 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.863+0000 7f3a91264640 1 Processor -- start 2024-03-06T07:32:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.863+0000 7f3a91264640 1 -- start start 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a91264640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 0x7f3a8c0c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a91264640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 0x7f3a8c0cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a91264640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a8c0cdcd0 0x7f3a8c0caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a91264640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c0cc1c0 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a91264640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c0d0f00 con 0x7f3a8c0cdcd0 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a91264640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c0cc340 con 0x7f3a8c06b2c0 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 0x7f3a8c0cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a83fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 0x7f3a8c0cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47138/0 (socket says 172.21.15.27:47138) 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a83fff640 1 -- 172.21.15.27:0/130416419 learned_addr learned my addr 172.21.15.27:0/130416419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a8affd640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 0x7f3a8c0c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a8b7fe640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a8c0cdcd0 0x7f3a8c0caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a83fff640 1 -- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 msgr2=0x7f3a8c0c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a83fff640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 0x7f3a8c0c55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a83fff640 1 -- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a8c0cdcd0 msgr2=0x7f3a8c0caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a83fff640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a8c0cdcd0 0x7f3a8c0caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a83fff640 1 -- 172.21.15.27:0/130416419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c15ec50 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a83fff640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 0x7f3a8c0cb2e0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f3a74034040 tx=0x7f3a740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a837fe640 1 -- 172.21.15.27:0/130416419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74032630 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a837fe640 1 -- 172.21.15.27:0/130416419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a740327f0 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8c0cb970 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a837fe640 1 -- 172.21.15.27:0/130416419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74032be0 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.864+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8c15f1f0 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.865+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8c09fcf0 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.866+0000 7f3a837fe640 1 -- 172.21.15.27:0/130416419 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a7404e430 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.866+0000 7f3a837fe640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6c06b4d0 0x7f3a6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.866+0000 7f3a837fe640 1 -- 172.21.15.27:0/130416419 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3a7402f030 con 0x7f3a8c0b2cd0 2024-03-06T07:32:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.866+0000 7f3a8affd640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6c06b4d0 0x7f3a6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.867+0000 7f3a8affd640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6c06b4d0 0x7f3a6c06d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c007fd0 tx=0x7f3a7c007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:32.871+0000 7f3a837fe640 1 -- 172.21.15.27:0/130416419 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a74040070 con 0x7f3a8c0b2cd0 2024-03-06T07:32:33.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.056+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a8c0c9e60 con 0x7f3a8c0b2cd0 2024-03-06T07:32:33.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.056+0000 7f3a837fe640 1 -- 172.21.15.27:0/130416419 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a7407a790 con 0x7f3a8c0b2cd0 2024-03-06T07:32:33.058 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:32:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6c06b4d0 msgr2=0x7f3a6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6c06b4d0 0x7f3a6c06d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c007fd0 tx=0x7f3a7c007de0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 msgr2=0x7f3a8c0cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 0x7f3a8c0cb2e0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f3a74034040 tx=0x7f3a740046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 shutdown_connections 2024-03-06T07:32:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3a6c06b4d0 0x7f3a6c06d990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a8c0cdcd0 0x7f3a8c0caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8c0b2cd0 0x7f3a8c0cb2e0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 --2- 172.21.15.27:0/130416419 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3a8c06b2c0 0x7f3a8c0c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 >> 172.21.15.27:0/130416419 conn(0x7f3a8c0b3700 msgr2=0x7f3a8c06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 shutdown_connections 2024-03-06T07:32:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.058+0000 7f3a91264640 1 -- 172.21.15.27:0/130416419 wait complete. 2024-03-06T07:32:33.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.179+0000 7fda192d4640 1 Processor -- start 2024-03-06T07:32:33.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.179+0000 7fda192d4640 1 -- start start 2024-03-06T07:32:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda192d4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda140b79f0 0x7fda140b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda192d4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda140d4960 con 0x7fda14071540 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda192d4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda140d36c0 con 0x7fda140b2d30 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda192d4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda140d3230 con 0x7fda140b79f0 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda137fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda140b79f0 0x7fda140b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda12ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fda14071540 0x7fda14071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53582/0 (socket says 172.21.15.27:53582) 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda12ffd640 1 -- 172.21.15.27:0/618965154 learned_addr learned my addr 172.21.15.27:0/618965154 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda137fe640 1 -- 172.21.15.27:0/618965154 >> v1:172.21.15.69:6789/0 conn(0x7fda140b2d30 legacy=0x7fda140b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda137fe640 1 -- 172.21.15.27:0/618965154 >> v1:172.21.15.27:6789/0 conn(0x7fda14071540 legacy=0x7fda14071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda11ffb640 1 -- 172.21.15.27:0/618965154 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827973210 0 0) 0x7fda140d4960 con 0x7fda14071540 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.180+0000 7fda137fe640 1 -- 172.21.15.27:0/618965154 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda140d2620 con 0x7fda140b79f0 2024-03-06T07:32:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.181+0000 7fda137fe640 1 --2- 172.21.15.27:0/618965154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda140b79f0 0x7fda140b7ea0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009d90 tx=0x7fda0c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1e8c5aa0a784e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.181+0000 7fda11ffb640 1 -- 172.21.15.27:0/618965154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c039070 con 0x7fda140b79f0 2024-03-06T07:32:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.181+0000 7fda11ffb640 1 -- 172.21.15.27:0/618965154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c02cd00 con 0x7fda140b79f0 2024-03-06T07:32:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.181+0000 7fda11ffb640 1 -- 172.21.15.27:0/618965154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c034660 con 0x7fda140b79f0 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.182+0000 7fda192d4640 1 -- 172.21.15.27:0/618965154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda140b79f0 msgr2=0x7fda140b7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.182+0000 7fda192d4640 1 --2- 172.21.15.27:0/618965154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda140b79f0 0x7fda140b7ea0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009d90 tx=0x7fda0c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.182+0000 7fda192d4640 1 -- 172.21.15.27:0/618965154 shutdown_connections 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.182+0000 7fda192d4640 1 --2- 172.21.15.27:0/618965154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda140b79f0 0x7fda140b7ea0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.182+0000 7fda192d4640 1 -- 172.21.15.27:0/618965154 >> 172.21.15.27:0/618965154 conn(0x7fda140d3d60 msgr2=0x7fda140d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.182+0000 7fda192d4640 1 -- 172.21.15.27:0/618965154 shutdown_connections 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.182+0000 7fda192d4640 1 -- 172.21.15.27:0/618965154 wait complete. 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.183+0000 7fda192d4640 1 Processor -- start 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.183+0000 7fda192d4640 1 -- start start 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda192d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda14071540 0x7fda141f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda192d4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140b2d30 0x7fda140c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda192d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140b79f0 0x7fda140c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda192d4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda140d2be0 con 0x7fda14071540 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda192d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda140b83e0 con 0x7fda140b79f0 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda192d4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda140d2ec0 con 0x7fda140b2d30 2024-03-06T07:32:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda12ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda14071540 0x7fda141f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda12ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda14071540 0x7fda141f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47164/0 (socket says 172.21.15.27:47164) 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda12ffd640 1 -- 172.21.15.27:0/3994454963 learned_addr learned my addr 172.21.15.27:0/3994454963 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda137fe640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140b79f0 0x7fda140c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda127fc640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140b2d30 0x7fda140c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda12ffd640 1 -- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140b2d30 msgr2=0x7fda140c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda12ffd640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140b2d30 0x7fda140c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda12ffd640 1 -- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140b79f0 msgr2=0x7fda140c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda12ffd640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140b79f0 0x7fda140c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.184+0000 7fda12ffd640 1 -- 172.21.15.27:0/3994454963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1414bce0 con 0x7fda14071540 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.185+0000 7fda12ffd640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda14071540 0x7fda141f4050 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fda0400c3f0 tx=0x7fda0400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.185+0000 7fd9fb7fe640 1 -- 172.21.15.27:0/3994454963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda040074e0 con 0x7fda14071540 2024-03-06T07:32:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.185+0000 7fd9fb7fe640 1 -- 172.21.15.27:0/3994454963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda040047e0 con 0x7fda14071540 2024-03-06T07:32:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.185+0000 7fd9fb7fe640 1 -- 172.21.15.27:0/3994454963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04004c10 con 0x7fda14071540 2024-03-06T07:32:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.185+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1414bfc0 con 0x7fda14071540 2024-03-06T07:32:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.185+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1414c580 con 0x7fda14071540 2024-03-06T07:32:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.186+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda140d4c70 con 0x7fda14071540 2024-03-06T07:32:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.186+0000 7fd9fb7fe640 1 -- 172.21.15.27:0/3994454963 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda04011070 con 0x7fda14071540 2024-03-06T07:32:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.187+0000 7fd9fb7fe640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0006b500 0x7fda0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.187+0000 7fd9fb7fe640 1 -- 172.21.15.27:0/3994454963 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fda04089cb0 con 0x7fda14071540 2024-03-06T07:32:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.187+0000 7fda127fc640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0006b500 0x7fda0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.187+0000 7fda127fc640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0006b500 0x7fda0006d9c0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fda141f7ad0 tx=0x7fda08009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.191+0000 7fd9fb7fe640 1 -- 172.21.15.27:0/3994454963 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda140d4c70 con 0x7fda14071540 2024-03-06T07:32:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.371+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fda140c76a0 con 0x7fda14071540 2024-03-06T07:32:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.371+0000 7fd9fb7fe640 1 -- 172.21.15.27:0/3994454963 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+11 (secure 0 0 0) 0x7fda040584c0 con 0x7fda14071540 2024-03-06T07:32:33.373 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0006b500 msgr2=0x7fda0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0006b500 0x7fda0006d9c0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fda141f7ad0 tx=0x7fda08009040 comp rx=0 tx=0).stop 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda14071540 msgr2=0x7fda141f4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda14071540 0x7fda141f4050 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fda0400c3f0 tx=0x7fda0400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 shutdown_connections 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fda0006b500 0x7fda0006d9c0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda140b79f0 0x7fda140c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fda140b2d30 0x7fda140c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 --2- 172.21.15.27:0/3994454963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda14071540 0x7fda141f4050 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 >> 172.21.15.27:0/3994454963 conn(0x7fda140d3d60 msgr2=0x7fda140b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 shutdown_connections 2024-03-06T07:32:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.373+0000 7fda192d4640 1 -- 172.21.15.27:0/3994454963 wait complete. 2024-03-06T07:32:33.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:33.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.494+0000 7fb6a5d65640 1 Processor -- start 2024-03-06T07:32:33.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.495+0000 7fb6a5d65640 1 -- start start 2024-03-06T07:32:33.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.495+0000 7fb6a5d65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 0x7fb6a0071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.495+0000 7fb6a5d65640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6a00cee80 0x7fb6a0168900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.495+0000 7fb6a5d65640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a0071e60 con 0x7fb6a00b2d30 2024-03-06T07:32:33.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.495+0000 7fb6a5d65640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a00d3230 con 0x7fb6a00cee80 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.495+0000 7fb6a5d65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00d4960 con 0x7fb6a0071540 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.495+0000 7fb69ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6a00cee80 0x7fb6a0168900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.495+0000 7fb69ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6a00cee80 0x7fb6a0168900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47174/0 (socket says 172.21.15.27:47174) 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.495+0000 7fb69ffff640 1 -- 172.21.15.27:0/324748448 learned_addr learned my addr 172.21.15.27:0/324748448 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.496+0000 7fb69f7fe640 1 --2- 172.21.15.27:0/324748448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 0x7fb6a0071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.496+0000 7fb69ffff640 1 -- 172.21.15.27:0/324748448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 msgr2=0x7fb6a0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.496+0000 7fb69ffff640 1 --2- 172.21.15.27:0/324748448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 0x7fb6a0071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.496+0000 7fb69ffff640 1 -- 172.21.15.27:0/324748448 >> v1:172.21.15.69:6789/0 conn(0x7fb6a00b2d30 legacy=0x7fb6a00ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.496+0000 7fb69e7fc640 1 -- 172.21.15.27:0/324748448 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341290821 0 0) 0x7fb6a0071e60 con 0x7fb6a00b2d30 2024-03-06T07:32:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.496+0000 7fb69ffff640 1 -- 172.21.15.27:0/324748448 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a00d36c0 con 0x7fb6a00cee80 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb69ffff640 1 --2- 172.21.15.27:0/324748448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 0x7fb6a0168900 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fb698009e50 tx=0x7fb69802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3a7fb94447a456 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb69e7fc640 1 -- 172.21.15.27:0/324748448 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69802cb00 con 0x7fb6a00cee80 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb69e7fc640 1 -- 172.21.15.27:0/324748448 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69802cca0 con 0x7fb6a00cee80 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb69e7fc640 1 -- 172.21.15.27:0/324748448 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698034610 con 0x7fb6a00cee80 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb6a5d65640 1 -- 172.21.15.27:0/324748448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 msgr2=0x7fb6a0168900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb6a5d65640 1 --2- 172.21.15.27:0/324748448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 0x7fb6a0168900 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fb698009e50 tx=0x7fb69802bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb6a5d65640 1 -- 172.21.15.27:0/324748448 shutdown_connections 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb6a5d65640 1 --2- 172.21.15.27:0/324748448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 0x7fb6a0168900 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb6a5d65640 1 --2- 172.21.15.27:0/324748448 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 0x7fb6a0071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb6a5d65640 1 -- 172.21.15.27:0/324748448 >> 172.21.15.27:0/324748448 conn(0x7fb6a00d3d60 msgr2=0x7fb6a00d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb6a5d65640 1 -- 172.21.15.27:0/324748448 shutdown_connections 2024-03-06T07:32:33.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.497+0000 7fb6a5d65640 1 -- 172.21.15.27:0/324748448 wait complete. 2024-03-06T07:32:33.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.499+0000 7fb6a5d65640 1 Processor -- start 2024-03-06T07:32:33.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.499+0000 7fb6a5d65640 1 -- start start 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb6a5d65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 0x7fb6a00cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb6a5d65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6a00b2d30 0x7fb6a01e4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb6a5d65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 0x7fb6a01e7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb6a5d65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00d2be0 con 0x7fb6a00cee80 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb6a5d65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00d2900 con 0x7fb6a00b2d30 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb6a5d65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00d2ec0 con 0x7fb6a0071540 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 0x7fb6a01e7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 0x7fb6a01e7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47182/0 (socket says 172.21.15.27:47182) 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69ffff640 1 -- 172.21.15.27:0/369724429 learned_addr learned my addr 172.21.15.27:0/369724429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69f7fe640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 0x7fb6a00cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69effd640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6a00b2d30 0x7fb6a01e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69ffff640 1 -- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 msgr2=0x7fb6a00cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69ffff640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 0x7fb6a00cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69ffff640 1 -- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6a00b2d30 msgr2=0x7fb6a01e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69ffff640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6a00b2d30 0x7fb6a01e4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69ffff640 1 -- 172.21.15.27:0/369724429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a00cc3f0 con 0x7fb6a00cee80 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb69ffff640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 0x7fb6a01e7bb0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fb698036040 tx=0x7fb698004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb687fff640 1 -- 172.21.15.27:0/369724429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698044020 con 0x7fb6a00cee80 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb687fff640 1 -- 172.21.15.27:0/369724429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69802ce50 con 0x7fb6a00cee80 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a00cc6d0 con 0x7fb6a00cee80 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb687fff640 1 -- 172.21.15.27:0/369724429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698009510 con 0x7fb6a00cee80 2024-03-06T07:32:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.500+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a00ccc90 con 0x7fb6a00cee80 2024-03-06T07:32:33.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.501+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a00ca4d0 con 0x7fb6a00cee80 2024-03-06T07:32:33.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.502+0000 7fb687fff640 1 -- 172.21.15.27:0/369724429 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb698047020 con 0x7fb6a00cee80 2024-03-06T07:32:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.502+0000 7fb687fff640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb68c06b4d0 0x7fb68c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.502+0000 7fb687fff640 1 -- 172.21.15.27:0/369724429 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb6980ac8c0 con 0x7fb6a00cee80 2024-03-06T07:32:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.503+0000 7fb69f7fe640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb68c06b4d0 0x7fb68c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.503+0000 7fb69f7fe640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb68c06b4d0 0x7fb68c06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb6a015cf10 tx=0x7fb694009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.512+0000 7fb687fff640 1 -- 172.21.15.27:0/369724429 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb69804e030 con 0x7fb6a00cee80 2024-03-06T07:32:33.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.759+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb6a0072070 con 0x7fb6a00cee80 2024-03-06T07:32:33.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.760+0000 7fb687fff640 1 -- 172.21.15.27:0/369724429 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb69807b160 con 0x7fb6a00cee80 2024-03-06T07:32:33.761 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:32:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb68c06b4d0 msgr2=0x7fb68c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb68c06b4d0 0x7fb68c06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb6a015cf10 tx=0x7fb694009190 comp rx=0 tx=0).stop 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 msgr2=0x7fb6a01e7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 0x7fb6a01e7bb0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fb698036040 tx=0x7fb698004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 shutdown_connections 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb68c06b4d0 0x7fb68c06d990 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a00cee80 0x7fb6a01e7bb0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6a00b2d30 0x7fb6a01e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 --2- 172.21.15.27:0/369724429 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6a0071540 0x7fb6a00cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 >> 172.21.15.27:0/369724429 conn(0x7fb6a00d3d60 msgr2=0x7fb6a00ce560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 shutdown_connections 2024-03-06T07:32:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.762+0000 7fb6a5d65640 1 -- 172.21.15.27:0/369724429 wait complete. 2024-03-06T07:32:33.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.880+0000 7f1a9ba6d640 1 Processor -- start 2024-03-06T07:32:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.881+0000 7f1a9ba6d640 1 -- start start 2024-03-06T07:32:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.881+0000 7f1a9ba6d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a9415df00 0x7f1a940d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.881+0000 7f1a9ba6d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a940b9a30 0x7f1a940d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.881+0000 7f1a9ba6d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a940cea30 con 0x7f1a940b95a0 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.881+0000 7f1a9ba6d640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a940d2af0 con 0x7f1a9415df00 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.881+0000 7f1a9ba6d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940cf3c0 con 0x7f1a940b9a30 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.881+0000 7f1a99809640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a9415df00 0x7f1a940d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.882+0000 7f1a99008640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1a940b95a0 0x7f1a940b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37828/0 (socket says 172.21.15.27:37828) 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.882+0000 7f1a99008640 1 -- 172.21.15.27:0/3035393896 learned_addr learned my addr 172.21.15.27:0/3035393896 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.882+0000 7f1a9a00a640 1 --2- 172.21.15.27:0/3035393896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a940b9a30 0x7f1a940d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.882+0000 7f1a99809640 1 -- 172.21.15.27:0/3035393896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a940b9a30 msgr2=0x7f1a940d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.882+0000 7f1a99809640 1 --2- 172.21.15.27:0/3035393896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a940b9a30 0x7f1a940d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.882+0000 7f1a99809640 1 -- 172.21.15.27:0/3035393896 >> v1:172.21.15.27:6789/0 conn(0x7f1a940b95a0 legacy=0x7f1a940b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.882+0000 7f1a98807640 1 -- 172.21.15.27:0/3035393896 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953974406 0 0) 0x7f1a940cea30 con 0x7f1a940b95a0 2024-03-06T07:32:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.882+0000 7f1a99809640 1 -- 172.21.15.27:0/3035393896 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a940cf0e0 con 0x7f1a9415df00 2024-03-06T07:32:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.882+0000 7f1a9a00a640 1 --2- 172.21.15.27:0/3035393896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a940b9a30 0x7f1a940d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:33.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.883+0000 7f1a99809640 1 --2- 172.21.15.27:0/3035393896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9415df00 0x7f1a940d25b0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f1a900097a0 tx=0x7f1a9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d822925e2c047e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.883+0000 7f1a98807640 1 -- 172.21.15.27:0/3035393896 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90035020 con 0x7f1a9415df00 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.883+0000 7f1a98807640 1 -- 172.21.15.27:0/3035393896 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a90031830 con 0x7f1a9415df00 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.883+0000 7f1a98807640 1 -- 172.21.15.27:0/3035393896 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90031c00 con 0x7f1a9415df00 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.884+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3035393896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9415df00 msgr2=0x7f1a940d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.884+0000 7f1a9ba6d640 1 --2- 172.21.15.27:0/3035393896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9415df00 0x7f1a940d25b0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f1a900097a0 tx=0x7f1a9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.884+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3035393896 shutdown_connections 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.884+0000 7f1a9ba6d640 1 --2- 172.21.15.27:0/3035393896 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a940b9a30 0x7f1a940d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.884+0000 7f1a9ba6d640 1 --2- 172.21.15.27:0/3035393896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9415df00 0x7f1a940d25b0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.884+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3035393896 >> 172.21.15.27:0/3035393896 conn(0x7f1a94071960 msgr2=0x7f1a94071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.884+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3035393896 shutdown_connections 2024-03-06T07:32:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.884+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3035393896 wait complete. 2024-03-06T07:32:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.885+0000 7f1a9ba6d640 1 Processor -- start 2024-03-06T07:32:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.885+0000 7f1a9ba6d640 1 -- start start 2024-03-06T07:32:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a9ba6d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940b95a0 0x7f1a940c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a9ba6d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a9415df00 0x7f1a9415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a9ba6d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a9415e820 0x7f1a941f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a9ba6d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940cec70 con 0x7f1a940b95a0 2024-03-06T07:32:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a9ba6d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940ce750 con 0x7f1a9415e820 2024-03-06T07:32:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a9ba6d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940cedf0 con 0x7f1a9415df00 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a99809640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940b95a0 0x7f1a940c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a99809640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940b95a0 0x7f1a940c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47388/0 (socket says 172.21.15.27:47388) 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a99809640 1 -- 172.21.15.27:0/3063745454 learned_addr learned my addr 172.21.15.27:0/3063745454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a99008640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a9415df00 0x7f1a9415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a9a00a640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a9415e820 0x7f1a941f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a99809640 1 -- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a9415df00 msgr2=0x7f1a9415e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a99809640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a9415df00 0x7f1a9415e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a99809640 1 -- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a9415e820 msgr2=0x7f1a941f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a99809640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a9415e820 0x7f1a941f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.886+0000 7f1a99809640 1 -- 172.21.15.27:0/3063745454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a94160db0 con 0x7f1a940b95a0 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.887+0000 7f1a99809640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940b95a0 0x7f1a940c93a0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f1a9002bee0 tx=0x7f1a900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.887+0000 7f1a8a7fc640 1 -- 172.21.15.27:0/3063745454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90035040 con 0x7f1a940b95a0 2024-03-06T07:32:33.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.887+0000 7f1a8a7fc640 1 -- 172.21.15.27:0/3063745454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9003b860 con 0x7f1a940b95a0 2024-03-06T07:32:33.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.887+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a94161090 con 0x7f1a940b95a0 2024-03-06T07:32:33.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.887+0000 7f1a8a7fc640 1 -- 172.21.15.27:0/3063745454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9003bc50 con 0x7f1a940b95a0 2024-03-06T07:32:33.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.887+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a940cb520 con 0x7f1a940b95a0 2024-03-06T07:32:33.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.888+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a94067ff0 con 0x7f1a940b95a0 2024-03-06T07:32:33.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.889+0000 7f1a8a7fc640 1 -- 172.21.15.27:0/3063745454 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a90038020 con 0x7f1a940b95a0 2024-03-06T07:32:33.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.889+0000 7f1a8a7fc640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a7806b4d0 0x7f1a7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.889+0000 7f1a8a7fc640 1 -- 172.21.15.27:0/3063745454 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1a900ac2f0 con 0x7f1a940b95a0 2024-03-06T07:32:33.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.889+0000 7f1a99008640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a7806b4d0 0x7f1a7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.890+0000 7f1a99008640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a7806b4d0 0x7f1a7806d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f1a94068c00 tx=0x7f1a84007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:33.899+0000 7f1a8a7fc640 1 -- 172.21.15.27:0/3063745454 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a90031800 con 0x7f1a940b95a0 2024-03-06T07:32:34.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.093+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1a940cb760 con 0x7f1a940b95a0 2024-03-06T07:32:34.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.093+0000 7f1a8a7fc640 1 -- 172.21.15.27:0/3063745454 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v149) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1a9007b160 con 0x7f1a940b95a0 2024-03-06T07:32:34.095 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a7806b4d0 msgr2=0x7f1a7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a7806b4d0 0x7f1a7806d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f1a94068c00 tx=0x7f1a84007de0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940b95a0 msgr2=0x7f1a940c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940b95a0 0x7f1a940c93a0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f1a9002bee0 tx=0x7f1a900046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 shutdown_connections 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a7806b4d0 0x7f1a7806d990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a9415e820 0x7f1a941f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a9415df00 0x7f1a9415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 --2- 172.21.15.27:0/3063745454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940b95a0 0x7f1a940c93a0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 >> 172.21.15.27:0/3063745454 conn(0x7f1a94071960 msgr2=0x7f1a940b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 shutdown_connections 2024-03-06T07:32:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:34.095+0000 7f1a9ba6d640 1 -- 172.21.15.27:0/3063745454 wait complete. 2024-03-06T07:32:34.107 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-03-06T07:32:34.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:34.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:34.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[85237]: fuse finished with error 0 and tester_r 0 2024-03-06T07:32:34.138 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:40.239 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:32:40.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:40.240 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:32:40.255 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:32:40.373 INFO:teuthology.orchestra.run.smithi069.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:32:40.373 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:32:40.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:32:40.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.481+0000 7fb1404e0640 1 Processor -- start 2024-03-06T07:32:40.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.481+0000 7fb1404e0640 1 -- start start 2024-03-06T07:32:40.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.482+0000 7fb1404e0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb13815dd90 0x7fb1380ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.482+0000 7fb1404e0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 0x7fb1380d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.482+0000 7fb1404e0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1380d3740 con 0x7fb1380d4720 2024-03-06T07:32:40.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.482+0000 7fb1404e0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1380d3400 con 0x7fb13815dd90 2024-03-06T07:32:40.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.482+0000 7fb1404e0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380d3120 con 0x7fb1380d29d0 2024-03-06T07:32:40.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.482+0000 7fb13e27c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb13815dd90 0x7fb1380ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.482+0000 7fb13da7b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 0x7fb1380d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.482+0000 7fb13ea7d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb1380d4720 0x7fb1380b6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:50102/0 (socket says 172.21.15.27:50102) 2024-03-06T07:32:40.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.482+0000 7fb13ea7d640 1 -- 172.21.15.27:0/1984119616 learned_addr learned my addr 172.21.15.27:0/1984119616 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:40.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.483+0000 7fb13e27c640 1 -- 172.21.15.27:0/1984119616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 msgr2=0x7fb1380d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.483+0000 7fb13e27c640 1 --2- 172.21.15.27:0/1984119616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 0x7fb1380d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.483+0000 7fb13e27c640 1 -- 172.21.15.27:0/1984119616 >> v1:172.21.15.69:6789/0 conn(0x7fb1380d4720 legacy=0x7fb1380b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.483+0000 7fb13e27c640 1 -- 172.21.15.27:0/1984119616 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1380d39e0 con 0x7fb13815dd90 2024-03-06T07:32:40.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.484+0000 7fb13e27c640 1 --2- 172.21.15.27:0/1984119616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 0x7fb1380ceea0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb1280097a0 tx=0x7fb12802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4362a84af6ccd6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.484+0000 7fb13d27a640 1 -- 172.21.15.27:0/1984119616 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128037070 con 0x7fb13815dd90 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.484+0000 7fb13d27a640 1 -- 172.21.15.27:0/1984119616 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb128030710 con 0x7fb13815dd90 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.484+0000 7fb13d27a640 1 -- 172.21.15.27:0/1984119616 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128030ae0 con 0x7fb13815dd90 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.485+0000 7fb1404e0640 1 -- 172.21.15.27:0/1984119616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 msgr2=0x7fb1380ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.485+0000 7fb1404e0640 1 --2- 172.21.15.27:0/1984119616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 0x7fb1380ceea0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb1280097a0 tx=0x7fb12802b790 comp rx=0 tx=0).stop 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.485+0000 7fb1404e0640 1 -- 172.21.15.27:0/1984119616 shutdown_connections 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.485+0000 7fb1404e0640 1 --2- 172.21.15.27:0/1984119616 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 0x7fb1380d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.485+0000 7fb1404e0640 1 --2- 172.21.15.27:0/1984119616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 0x7fb1380ceea0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.485+0000 7fb1404e0640 1 -- 172.21.15.27:0/1984119616 >> 172.21.15.27:0/1984119616 conn(0x7fb138071960 msgr2=0x7fb138071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.485+0000 7fb1404e0640 1 -- 172.21.15.27:0/1984119616 shutdown_connections 2024-03-06T07:32:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.485+0000 7fb1404e0640 1 -- 172.21.15.27:0/1984119616 wait complete. 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.486+0000 7fb1404e0640 1 Processor -- start 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.486+0000 7fb1404e0640 1 -- start start 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb1404e0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 0x7fb138156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb1404e0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1380d4720 0x7fb1380b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb1404e0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 0x7fb138156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb1404e0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380d2df0 con 0x7fb1380d4720 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb1404e0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13815e170 con 0x7fb13815dd90 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb1404e0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380b7420 con 0x7fb1380d29d0 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13da7b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1380d4720 0x7fb1380b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13ea7d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 0x7fb138156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13ea7d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 0x7fb138156630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:52848/0 (socket says 172.21.15.27:52848) 2024-03-06T07:32:40.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13ea7d640 1 -- 172.21.15.27:0/306426799 learned_addr learned my addr 172.21.15.27:0/306426799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13e27c640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 0x7fb138156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13ea7d640 1 -- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 msgr2=0x7fb138156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13ea7d640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 0x7fb138156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13ea7d640 1 -- 172.21.15.27:0/306426799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1380d4720 msgr2=0x7fb1380b8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13ea7d640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1380d4720 0x7fb1380b8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13ea7d640 1 -- 172.21.15.27:0/306426799 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb138156d60 con 0x7fb13815dd90 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.487+0000 7fb13ea7d640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 0x7fb138156630 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb13000caf0 tx=0x7fb1300020f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.488+0000 7fb12effd640 1 -- 172.21.15.27:0/306426799 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130018070 con 0x7fb13815dd90 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.488+0000 7fb12effd640 1 -- 172.21.15.27:0/306426799 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13000a760 con 0x7fb13815dd90 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.488+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb138157040 con 0x7fb13815dd90 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.488+0000 7fb12effd640 1 -- 172.21.15.27:0/306426799 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13000ab50 con 0x7fb13815dd90 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.488+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb138157600 con 0x7fb13815dd90 2024-03-06T07:32:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.489+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb138072a30 con 0x7fb13815dd90 2024-03-06T07:32:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.489+0000 7fb12effd640 1 -- 172.21.15.27:0/306426799 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb13001b070 con 0x7fb13815dd90 2024-03-06T07:32:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.489+0000 7fb12effd640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11c06b4d0 0x7fb11c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.490+0000 7fb12effd640 1 -- 172.21.15.27:0/306426799 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb1300899c0 con 0x7fb13815dd90 2024-03-06T07:32:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.490+0000 7fb13e27c640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11c06b4d0 0x7fb11c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.490+0000 7fb13e27c640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11c06b4d0 0x7fb11c06d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fb128034000 tx=0x7fb12802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.498+0000 7fb12effd640 1 -- 172.21.15.27:0/306426799 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1300581b0 con 0x7fb13815dd90 2024-03-06T07:32:40.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:40.719+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fb13804f410 con 0x7fb13815dd90 2024-03-06T07:32:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.326+0000 7fb12effd640 1 -- 172.21.15.27:0/306426799 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v120) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb13005c250 con 0x7fb13815dd90 2024-03-06T07:32:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11c06b4d0 msgr2=0x7fb11c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11c06b4d0 0x7fb11c06d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fb128034000 tx=0x7fb12802f000 comp rx=0 tx=0).stop 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 msgr2=0x7fb138156630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 0x7fb138156630 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb13000caf0 tx=0x7fb1300020f0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 shutdown_connections 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb11c06b4d0 0x7fb11c06d990 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13815dd90 0x7fb138156630 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1380d4720 0x7fb1380b8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 --2- 172.21.15.27:0/306426799 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb1380d29d0 0x7fb138156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 >> 172.21.15.27:0/306426799 conn(0x7fb138071960 msgr2=0x7fb1380b40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 shutdown_connections 2024-03-06T07:32:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.329+0000 7fb1404e0640 1 -- 172.21.15.27:0/306426799 wait complete. 2024-03-06T07:32:41.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:41.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.447+0000 7f558dfd0640 1 Processor -- start 2024-03-06T07:32:41.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.447+0000 7f558dfd0640 1 -- start start 2024-03-06T07:32:41.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.447+0000 7f558dfd0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f558815df00 0x7f55880d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.447+0000 7f558dfd0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880cea30 con 0x7f55880b9a30 2024-03-06T07:32:41.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.447+0000 7f558dfd0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880d2af0 con 0x7f55880b95a0 2024-03-06T07:32:41.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.447+0000 7f558dfd0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55880cf3c0 con 0x7f558815df00 2024-03-06T07:32:41.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.447+0000 7f55877fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f558815df00 0x7f55880d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.448+0000 7f55877fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f558815df00 0x7f55880d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47410/0 (socket says 172.21.15.27:47410) 2024-03-06T07:32:41.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.448+0000 7f55877fe640 1 -- 172.21.15.27:0/1469723072 learned_addr learned my addr 172.21.15.27:0/1469723072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:41.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.448+0000 7f55877fe640 1 -- 172.21.15.27:0/1469723072 >> v1:172.21.15.69:6789/0 conn(0x7f55880b95a0 legacy=0x7f55880b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.448+0000 7f55877fe640 1 -- 172.21.15.27:0/1469723072 >> v1:172.21.15.27:6789/0 conn(0x7f55880b9a30 legacy=0x7f55880d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.448+0000 7f55867fc640 1 -- 172.21.15.27:0/1469723072 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942643426 0 0) 0x7f55880cea30 con 0x7f55880b9a30 2024-03-06T07:32:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.448+0000 7f55877fe640 1 -- 172.21.15.27:0/1469723072 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55880cf0e0 con 0x7f558815df00 2024-03-06T07:32:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f55877fe640 1 --2- 172.21.15.27:0/1469723072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 0x7f55880d25b0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f5578009870 tx=0x7f557802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4d563d217f86af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f55867fc640 1 -- 172.21.15.27:0/1469723072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578034070 con 0x7f558815df00 2024-03-06T07:32:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f55867fc640 1 -- 172.21.15.27:0/1469723072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5578030860 con 0x7f558815df00 2024-03-06T07:32:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f55867fc640 1 -- 172.21.15.27:0/1469723072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578030c30 con 0x7f558815df00 2024-03-06T07:32:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f558dfd0640 1 -- 172.21.15.27:0/1469723072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 msgr2=0x7f55880d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f558dfd0640 1 --2- 172.21.15.27:0/1469723072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 0x7f55880d25b0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f5578009870 tx=0x7f557802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f558dfd0640 1 -- 172.21.15.27:0/1469723072 shutdown_connections 2024-03-06T07:32:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f558dfd0640 1 --2- 172.21.15.27:0/1469723072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 0x7f55880d25b0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f558dfd0640 1 -- 172.21.15.27:0/1469723072 >> 172.21.15.27:0/1469723072 conn(0x7f5588071960 msgr2=0x7f5588071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f558dfd0640 1 -- 172.21.15.27:0/1469723072 shutdown_connections 2024-03-06T07:32:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.449+0000 7f558dfd0640 1 -- 172.21.15.27:0/1469723072 wait complete. 2024-03-06T07:32:41.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.450+0000 7f558dfd0640 1 Processor -- start 2024-03-06T07:32:41.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.451+0000 7f558dfd0640 1 -- start start 2024-03-06T07:32:41.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.451+0000 7f558dfd0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55880b95a0 0x7f558815ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.451+0000 7f558dfd0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 0x7f558815a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.451+0000 7f558dfd0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f558815b240 0x7f55880d3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.451+0000 7f558dfd0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880cec70 con 0x7f558815df00 2024-03-06T07:32:41.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.451+0000 7f558dfd0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880ce750 con 0x7f558815b240 2024-03-06T07:32:41.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.451+0000 7f558dfd0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880cedf0 con 0x7f55880b95a0 2024-03-06T07:32:41.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.451+0000 7f5586ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 0x7f558815a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5586ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 0x7f558815a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47424/0 (socket says 172.21.15.27:47424) 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5586ffd640 1 -- 172.21.15.27:0/3436894455 learned_addr learned my addr 172.21.15.27:0/3436894455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5587fff640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f558815b240 0x7f55880d3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f55877fe640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55880b95a0 0x7f558815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5586ffd640 1 -- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55880b95a0 msgr2=0x7f558815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5586ffd640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55880b95a0 0x7f558815ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5586ffd640 1 -- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f558815b240 msgr2=0x7f55880d3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5586ffd640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f558815b240 0x7f55880d3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5586ffd640 1 -- 172.21.15.27:0/3436894455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558815e6c0 con 0x7f558815df00 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5587fff640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f558815b240 0x7f55880d3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f55877fe640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55880b95a0 0x7f558815ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f5586ffd640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 0x7f558815a5a0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f557000b420 tx=0x7f557000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f557ffff640 1 -- 172.21.15.27:0/3436894455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570011830 con 0x7f558815df00 2024-03-06T07:32:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f557ffff640 1 -- 172.21.15.27:0/3436894455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5570016070 con 0x7f558815df00 2024-03-06T07:32:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558815e9a0 con 0x7f558815df00 2024-03-06T07:32:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f557ffff640 1 -- 172.21.15.27:0/3436894455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570011c10 con 0x7f558815df00 2024-03-06T07:32:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.452+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558815ef60 con 0x7f558815df00 2024-03-06T07:32:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.453+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5588067ff0 con 0x7f558815df00 2024-03-06T07:32:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.454+0000 7f557ffff640 1 -- 172.21.15.27:0/3436894455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f557001b460 con 0x7f558815df00 2024-03-06T07:32:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.455+0000 7f557ffff640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f556806b4d0 0x7f556806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.455+0000 7f557ffff640 1 -- 172.21.15.27:0/3436894455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5570089f20 con 0x7f558815df00 2024-03-06T07:32:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.455+0000 7f55877fe640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f556806b4d0 0x7f556806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.456+0000 7f55877fe640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f556806b4d0 0x7f556806d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5578036000 tx=0x7f557803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.459+0000 7f557ffff640 1 -- 172.21.15.27:0/3436894455 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5570058e10 con 0x7f558815df00 2024-03-06T07:32:41.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.816+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f558806b780 con 0x7f558815df00 2024-03-06T07:32:41.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.817+0000 7f557ffff640 1 -- 172.21.15.27:0/3436894455 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f5570058710 con 0x7f558815df00 2024-03-06T07:32:41.819 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:41.819 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:41.819 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f556806b4d0 msgr2=0x7f556806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f556806b4d0 0x7f556806d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5578036000 tx=0x7f557803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 msgr2=0x7f558815a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 0x7f558815a5a0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f557000b420 tx=0x7f557000b8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 shutdown_connections 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f556806b4d0 0x7f556806d990 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f558815b240 0x7f55880d3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558815df00 0x7f558815a5a0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 --2- 172.21.15.27:0/3436894455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55880b95a0 0x7f558815ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 >> 172.21.15.27:0/3436894455 conn(0x7f5588071960 msgr2=0x7f55880cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 shutdown_connections 2024-03-06T07:32:41.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:41.819+0000 7f558dfd0640 1 -- 172.21.15.27:0/3436894455 wait complete. 2024-03-06T07:32:42.032 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:42.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:43.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.140+0000 7fdd7bc78640 1 Processor -- start 2024-03-06T07:32:43.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.140+0000 7fdd7bc78640 1 -- start start 2024-03-06T07:32:43.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd7bc78640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd7bc78640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd740b9a30 0x7fdd740d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd7bc78640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd740cea30 con 0x7fdd740b95a0 2024-03-06T07:32:43.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd7bc78640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd740d2af0 con 0x7fdd740b9a30 2024-03-06T07:32:43.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd7bc78640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740cf3c0 con 0x7fdd7415df00 2024-03-06T07:32:43.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd79213640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdd740b95a0 0x7fdd740b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37858/0 (socket says 172.21.15.27:37858) 2024-03-06T07:32:43.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd79213640 1 -- 172.21.15.27:0/3925759831 learned_addr learned my addr 172.21.15.27:0/3925759831 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:43.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd79a14640 1 --2- 172.21.15.27:0/3925759831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd7a215640 1 --2- 172.21.15.27:0/3925759831 >> v2:172.21.15.69:3300/0 conn(0x7fdd740b9a30 0x7fdd740d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd78a12640 1 -- 172.21.15.27:0/3925759831 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455370150 0 0) 0x7fdd740cea30 con 0x7fdd740b95a0 2024-03-06T07:32:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd78a12640 1 -- 172.21.15.27:0/3925759831 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd60003520 con 0x7fdd740b95a0 2024-03-06T07:32:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd79a14640 1 -- 172.21.15.27:0/3925759831 >> v2:172.21.15.69:3300/0 conn(0x7fdd740b9a30 msgr2=0x7fdd740d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd79a14640 1 --2- 172.21.15.27:0/3925759831 >> v2:172.21.15.69:3300/0 conn(0x7fdd740b9a30 0x7fdd740d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd79a14640 1 -- 172.21.15.27:0/3925759831 >> v1:172.21.15.27:6789/0 conn(0x7fdd740b95a0 legacy=0x7fdd740b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd78a12640 1 -- 172.21.15.27:0/3925759831 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4160104688 0 0) 0x7fdd60003520 con 0x7fdd740b95a0 2024-03-06T07:32:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.141+0000 7fdd79a14640 1 -- 172.21.15.27:0/3925759831 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd740cf0e0 con 0x7fdd7415df00 2024-03-06T07:32:43.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.142+0000 7fdd7a215640 1 --2- 172.21.15.27:0/3925759831 >> v2:172.21.15.69:3300/0 conn(0x7fdd740b9a30 0x7fdd740d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.142+0000 7fdd79a14640 1 --2- 172.21.15.27:0/3925759831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740d25b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdd700097a0 tx=0x7fdd7002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=822b0070a1aa0315 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.142+0000 7fdd78a12640 1 -- 172.21.15.27:0/3925759831 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd70037070 con 0x7fdd7415df00 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.142+0000 7fdd78a12640 1 -- 172.21.15.27:0/3925759831 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd70030510 con 0x7fdd7415df00 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.142+0000 7fdd78a12640 1 -- 172.21.15.27:0/3925759831 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64001670 con 0x7fdd7415df00 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.143+0000 7fdd7bc78640 1 -- 172.21.15.27:0/3925759831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 msgr2=0x7fdd740d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.143+0000 7fdd7bc78640 1 --2- 172.21.15.27:0/3925759831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740d25b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdd700097a0 tx=0x7fdd7002b790 comp rx=0 tx=0).stop 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.143+0000 7fdd7bc78640 1 -- 172.21.15.27:0/3925759831 shutdown_connections 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.143+0000 7fdd7bc78640 1 --2- 172.21.15.27:0/3925759831 >> v2:172.21.15.69:3300/0 conn(0x7fdd740b9a30 0x7fdd740d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.143+0000 7fdd7bc78640 1 --2- 172.21.15.27:0/3925759831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740d25b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.143+0000 7fdd7bc78640 1 -- 172.21.15.27:0/3925759831 >> 172.21.15.27:0/3925759831 conn(0x7fdd74071960 msgr2=0x7fdd74071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.143+0000 7fdd7bc78640 1 -- 172.21.15.27:0/3925759831 shutdown_connections 2024-03-06T07:32:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.143+0000 7fdd7bc78640 1 -- 172.21.15.27:0/3925759831 wait complete. 2024-03-06T07:32:43.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.144+0000 7fdd7bc78640 1 Processor -- start 2024-03-06T07:32:43.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.144+0000 7fdd7bc78640 1 -- start start 2024-03-06T07:32:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd7bc78640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740b95a0 0x7fdd7415a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd7bc78640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd7bc78640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd740b1f40 0x7fdd740b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd7bc78640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740cec70 con 0x7fdd740b95a0 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd7bc78640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740ce750 con 0x7fdd740b1f40 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd7bc78640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740cedf0 con 0x7fdd7415df00 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd79a14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740b95a0 0x7fdd7415a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd79a14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740b95a0 0x7fdd7415a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47430/0 (socket says 172.21.15.27:47430) 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd79a14640 1 -- 172.21.15.27:0/4167520388 learned_addr learned my addr 172.21.15.27:0/4167520388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd7a215640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd740b1f40 0x7fdd740b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.145+0000 7fdd79213640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd79a14640 1 -- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 msgr2=0x7fdd740cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd79a14640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd79a14640 1 -- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd740b1f40 msgr2=0x7fdd740b23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd79a14640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd740b1f40 0x7fdd740b23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd79a14640 1 -- 172.21.15.27:0/4167520388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd740cbff0 con 0x7fdd740b95a0 2024-03-06T07:32:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd79213640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd79a14640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740b95a0 0x7fdd7415a7b0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fdd70000c00 tx=0x7fdd700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd6a7fc640 1 -- 172.21.15.27:0/4167520388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd70041020 con 0x7fdd740b95a0 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd6a7fc640 1 -- 172.21.15.27:0/4167520388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd70032700 con 0x7fdd740b95a0 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd6a7fc640 1 -- 172.21.15.27:0/4167520388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd70037040 con 0x7fdd740b95a0 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd740cc2d0 con 0x7fdd740b95a0 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.146+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd740cc890 con 0x7fdd740b95a0 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.147+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd74067ff0 con 0x7fdd740b95a0 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.148+0000 7fdd6a7fc640 1 -- 172.21.15.27:0/4167520388 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd70044020 con 0x7fdd740b95a0 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.148+0000 7fdd6a7fc640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd5806b500 0x7fdd5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.148+0000 7fdd6a7fc640 1 -- 172.21.15.27:0/4167520388 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdd700ab9a0 con 0x7fdd740b95a0 2024-03-06T07:32:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.148+0000 7fdd79213640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd5806b500 0x7fdd5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.149+0000 7fdd79213640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd5806b500 0x7fdd5806d9c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdd741625e0 tx=0x7fdd64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.152+0000 7fdd6a7fc640 1 -- 172.21.15.27:0/4167520388 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd7004b030 con 0x7fdd740b95a0 2024-03-06T07:32:43.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.466+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd740b7150 con 0x7fdd740b95a0 2024-03-06T07:32:43.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.466+0000 7fdd6a7fc640 1 -- 172.21.15.27:0/4167520388 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7fdd7007a800 con 0x7fdd740b95a0 2024-03-06T07:32:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:43.469 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:43.469 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd5806b500 msgr2=0x7fdd5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd5806b500 0x7fdd5806d9c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdd741625e0 tx=0x7fdd64009040 comp rx=0 tx=0).stop 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740b95a0 msgr2=0x7fdd7415a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740b95a0 0x7fdd7415a7b0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fdd70000c00 tx=0x7fdd700046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 shutdown_connections 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdd5806b500 0x7fdd5806d9c0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd740b1f40 0x7fdd740b23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd7415df00 0x7fdd740cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 --2- 172.21.15.27:0/4167520388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740b95a0 0x7fdd7415a7b0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 >> 172.21.15.27:0/4167520388 conn(0x7fdd74071960 msgr2=0x7fdd740b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 shutdown_connections 2024-03-06T07:32:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:43.468+0000 7fdd7bc78640 1 -- 172.21.15.27:0/4167520388 wait complete. 2024-03-06T07:32:43.682 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:43.682 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:44.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:44.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.824+0000 7f7c60b3e640 1 Processor -- start 2024-03-06T07:32:44.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.824+0000 7f7c60b3e640 1 -- start start 2024-03-06T07:32:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.825+0000 7f7c60b3e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7c5c0cfb50 0x7f7c5c166890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.825+0000 7f7c60b3e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c0cc7b0 con 0x7f7c5c0d0030 2024-03-06T07:32:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.825+0000 7f7c60b3e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c0cca90 con 0x7f7c5c15bf40 2024-03-06T07:32:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.825+0000 7f7c60b3e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c0cd3c0 con 0x7f7c5c0cfb50 2024-03-06T07:32:44.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.825+0000 7f7c5a59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7c5c0d0030 0x7f7c5c0d0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44682/0 (socket says 172.21.15.27:44682) 2024-03-06T07:32:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.825+0000 7f7c5a59c640 1 -- 172.21.15.27:0/3980434603 learned_addr learned my addr 172.21.15.27:0/3980434603 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.825+0000 7f7c5ad9d640 1 --2- 172.21.15.27:0/3980434603 >> v2:172.21.15.69:3300/0 conn(0x7f7c5c0cfb50 0x7f7c5c166890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.826+0000 7f7c5959a640 1 -- 172.21.15.27:0/3980434603 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353842821 0 0) 0x7f7c5c0cc7b0 con 0x7f7c5c0d0030 2024-03-06T07:32:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.826+0000 7f7c5959a640 1 -- 172.21.15.27:0/3980434603 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c44003520 con 0x7f7c5c0d0030 2024-03-06T07:32:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.826+0000 7f7c5ad9d640 1 -- 172.21.15.27:0/3980434603 >> v1:172.21.15.69:6789/0 conn(0x7f7c5c15bf40 legacy=0x7f7c5c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.826+0000 7f7c5ad9d640 1 -- 172.21.15.27:0/3980434603 >> v1:172.21.15.27:6789/0 conn(0x7f7c5c0d0030 legacy=0x7f7c5c0d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.826+0000 7f7c5959a640 1 -- 172.21.15.27:0/3980434603 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2273927144 0 0) 0x7f7c44003520 con 0x7f7c5c0d0030 2024-03-06T07:32:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.826+0000 7f7c5ad9d640 1 -- 172.21.15.27:0/3980434603 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c0ccd70 con 0x7f7c5c0cfb50 2024-03-06T07:32:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.827+0000 7f7c5ad9d640 1 --2- 172.21.15.27:0/3980434603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0cfb50 0x7f7c5c166890 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009d90 tx=0x7f7c5402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97d4e35553e0de80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.827+0000 7f7c5959a640 1 -- 172.21.15.27:0/3980434603 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54039070 con 0x7f7c5c0cfb50 2024-03-06T07:32:44.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.827+0000 7f7c5959a640 1 -- 172.21.15.27:0/3980434603 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5402c940 con 0x7f7c5c0cfb50 2024-03-06T07:32:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.827+0000 7f7c5959a640 1 -- 172.21.15.27:0/3980434603 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c001670 con 0x7f7c5c0cfb50 2024-03-06T07:32:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.827+0000 7f7c60b3e640 1 -- 172.21.15.27:0/3980434603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0cfb50 msgr2=0x7f7c5c166890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.827+0000 7f7c60b3e640 1 --2- 172.21.15.27:0/3980434603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0cfb50 0x7f7c5c166890 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009d90 tx=0x7f7c5402be10 comp rx=0 tx=0).stop 2024-03-06T07:32:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.827+0000 7f7c60b3e640 1 -- 172.21.15.27:0/3980434603 shutdown_connections 2024-03-06T07:32:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.827+0000 7f7c60b3e640 1 --2- 172.21.15.27:0/3980434603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0cfb50 0x7f7c5c166890 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.827+0000 7f7c60b3e640 1 -- 172.21.15.27:0/3980434603 >> 172.21.15.27:0/3980434603 conn(0x7f7c5c0d1a70 msgr2=0x7f7c5c0d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.828+0000 7f7c60b3e640 1 -- 172.21.15.27:0/3980434603 shutdown_connections 2024-03-06T07:32:44.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.828+0000 7f7c60b3e640 1 -- 172.21.15.27:0/3980434603 wait complete. 2024-03-06T07:32:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.829+0000 7f7c60b3e640 1 Processor -- start 2024-03-06T07:32:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.829+0000 7f7c60b3e640 1 -- start start 2024-03-06T07:32:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c60b3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5c0cfb50 0x7f7c5c0cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c60b3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0d0030 0x7f7c5c15aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c60b3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c15bf40 0x7f7c5c09d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c60b3e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0cd850 con 0x7f7c5c15bf40 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c60b3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0b58d0 con 0x7f7c5c0d0030 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c60b3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c09cd80 con 0x7f7c5c0cfb50 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c15bf40 0x7f7c5c09d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c15bf40 0x7f7c5c09d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50896/0 (socket says 172.21.15.27:50896) 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5ad9d640 1 -- 172.21.15.27:0/1259260840 learned_addr learned my addr 172.21.15.27:0/1259260840 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5a59c640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5c0cfb50 0x7f7c5c0cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c59d9b640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0d0030 0x7f7c5c15aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5ad9d640 1 -- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5c0cfb50 msgr2=0x7f7c5c0cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5ad9d640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5c0cfb50 0x7f7c5c0cabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5ad9d640 1 -- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0d0030 msgr2=0x7f7c5c15aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5ad9d640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0d0030 0x7f7c5c15aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5ad9d640 1 -- 172.21.15.27:0/1259260840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c0c9760 con 0x7f7c5c15bf40 2024-03-06T07:32:44.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c59d9b640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0d0030 0x7f7c5c15aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:44.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c5ad9d640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c15bf40 0x7f7c5c09d280 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f7c54037040 tx=0x7f7c54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.830+0000 7f7c4affd640 1 -- 172.21.15.27:0/1259260840 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54039040 con 0x7f7c5c15bf40 2024-03-06T07:32:44.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.831+0000 7f7c4affd640 1 -- 172.21.15.27:0/1259260840 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c54034d10 con 0x7f7c5c15bf40 2024-03-06T07:32:44.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.831+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c0c9a40 con 0x7f7c5c15bf40 2024-03-06T07:32:44.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.831+0000 7f7c4affd640 1 -- 172.21.15.27:0/1259260840 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54009450 con 0x7f7c5c15bf40 2024-03-06T07:32:44.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.831+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c0cb120 con 0x7f7c5c15bf40 2024-03-06T07:32:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.832+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c0c9550 con 0x7f7c5c15bf40 2024-03-06T07:32:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.832+0000 7f7c4affd640 1 -- 172.21.15.27:0/1259260840 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c54003bd0 con 0x7f7c5c15bf40 2024-03-06T07:32:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.832+0000 7f7c4affd640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c3c06b4d0 0x7f7c3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.833+0000 7f7c4affd640 1 -- 172.21.15.27:0/1259260840 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7c540aceb0 con 0x7f7c5c15bf40 2024-03-06T07:32:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.833+0000 7f7c5a59c640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c3c06b4d0 0x7f7c3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.833+0000 7f7c5a59c640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c3c06b4d0 0x7f7c3c06d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c005880 tx=0x7f7c4c0057f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:44.837+0000 7f7c4affd640 1 -- 172.21.15.27:0/1259260840 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c5407be30 con 0x7f7c5c15bf40 2024-03-06T07:32:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.181+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c5c067ff0 con 0x7f7c5c15bf40 2024-03-06T07:32:45.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.181+0000 7f7c4affd640 1 -- 172.21.15.27:0/1259260840 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f7c5407b730 con 0x7f7c5c15bf40 2024-03-06T07:32:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:45.184 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:45.184 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c3c06b4d0 msgr2=0x7f7c3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c3c06b4d0 0x7f7c3c06d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c005880 tx=0x7f7c4c0057f0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c15bf40 msgr2=0x7f7c5c09d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c15bf40 0x7f7c5c09d280 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f7c54037040 tx=0x7f7c54004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 shutdown_connections 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7c3c06b4d0 0x7f7c3c06d990 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c15bf40 0x7f7c5c09d280 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c5c0d0030 0x7f7c5c15aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 --2- 172.21.15.27:0/1259260840 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7c5c0cfb50 0x7f7c5c0cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 >> 172.21.15.27:0/1259260840 conn(0x7f7c5c0d1a70 msgr2=0x7f7c5c0b5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 shutdown_connections 2024-03-06T07:32:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:45.183+0000 7f7c60b3e640 1 -- 172.21.15.27:0/1259260840 wait complete. 2024-03-06T07:32:45.398 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:45.398 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:46.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.505+0000 7f770c581640 1 Processor -- start 2024-03-06T07:32:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.505+0000 7f770c581640 1 -- start start 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.506+0000 7f770c581640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7704071540 0x7f77040b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.506+0000 7f770c581640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77040ce610 con 0x7f77040d3400 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.506+0000 7f770c581640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77040d2920 con 0x7f77040b2b50 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.506+0000 7f770c581640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77040d2f70 con 0x7f7704071540 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.506+0000 7f770a31d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7704071540 0x7f77040b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.507+0000 7f7709b1c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f77040d3400 0x7f7704071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44694/0 (socket says 172.21.15.27:44694) 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.507+0000 7f7709b1c640 1 -- 172.21.15.27:0/2642157516 learned_addr learned my addr 172.21.15.27:0/2642157516 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.507+0000 7f770a31d640 1 -- 172.21.15.27:0/2642157516 >> v1:172.21.15.69:6789/0 conn(0x7f77040b2b50 legacy=0x7f77040b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.507+0000 7f770a31d640 1 -- 172.21.15.27:0/2642157516 >> v1:172.21.15.27:6789/0 conn(0x7f77040d3400 legacy=0x7f7704071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.507+0000 7f770a31d640 1 -- 172.21.15.27:0/2642157516 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77040d2c00 con 0x7f7704071540 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770a31d640 1 --2- 172.21.15.27:0/2642157516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7704071540 0x7f77040b30a0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f76f80097a0 tx=0x7f76f802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edda4055f00660f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770931b640 1 -- 172.21.15.27:0/2642157516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8034070 con 0x7f7704071540 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770931b640 1 -- 172.21.15.27:0/2642157516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f8030610 con 0x7f7704071540 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770931b640 1 -- 172.21.15.27:0/2642157516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80309e0 con 0x7f7704071540 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770c581640 1 -- 172.21.15.27:0/2642157516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7704071540 msgr2=0x7f77040b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770c581640 1 --2- 172.21.15.27:0/2642157516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7704071540 0x7f77040b30a0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f76f80097a0 tx=0x7f76f802b960 comp rx=0 tx=0).stop 2024-03-06T07:32:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770c581640 1 -- 172.21.15.27:0/2642157516 shutdown_connections 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770c581640 1 --2- 172.21.15.27:0/2642157516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7704071540 0x7f77040b30a0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770c581640 1 -- 172.21.15.27:0/2642157516 >> 172.21.15.27:0/2642157516 conn(0x7f77040d3ca0 msgr2=0x7f77040d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770c581640 1 -- 172.21.15.27:0/2642157516 shutdown_connections 2024-03-06T07:32:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.508+0000 7f770c581640 1 -- 172.21.15.27:0/2642157516 wait complete. 2024-03-06T07:32:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f770c581640 1 Processor -- start 2024-03-06T07:32:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f770c581640 1 -- start start 2024-03-06T07:32:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f770c581640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7704071540 0x7f77040cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f770c581640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77040b2b50 0x7f77040cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f770c581640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77040d3400 0x7f77041f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f770c581640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770415e370 con 0x7f77040b2b50 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f770c581640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77040d4be0 con 0x7f7704071540 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f770c581640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77040caf80 con 0x7f77040d3400 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f7709b1c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77040b2b50 0x7f77040cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f7709b1c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77040b2b50 0x7f77040cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50920/0 (socket says 172.21.15.27:50920) 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.510+0000 7f7709b1c640 1 -- 172.21.15.27:0/1360595659 learned_addr learned my addr 172.21.15.27:0/1360595659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f770ab1e640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77040d3400 0x7f77041f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f770a31d640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7704071540 0x7f77040cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f7709b1c640 1 -- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77040d3400 msgr2=0x7f77041f62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f7709b1c640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77040d3400 0x7f77041f62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f7709b1c640 1 -- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7704071540 msgr2=0x7f77040cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f7709b1c640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7704071540 0x7f77040cbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f7709b1c640 1 -- 172.21.15.27:0/1360595659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77040cb260 con 0x7f77040b2b50 2024-03-06T07:32:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f7709b1c640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77040b2b50 0x7f77040cc890 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f770000c3c0 tx=0x7f770000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f76f6ffd640 1 -- 172.21.15.27:0/1360595659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700017070 con 0x7f77040b2b50 2024-03-06T07:32:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f76f6ffd640 1 -- 172.21.15.27:0/1360595659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7700005ce0 con 0x7f77040b2b50 2024-03-06T07:32:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77040cb540 con 0x7f77040b2b50 2024-03-06T07:32:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f76f6ffd640 1 -- 172.21.15.27:0/1360595659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77000072d0 con 0x7f77040b2b50 2024-03-06T07:32:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.511+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77040ccdd0 con 0x7f77040b2b50 2024-03-06T07:32:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.512+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7704068700 con 0x7f77040b2b50 2024-03-06T07:32:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.513+0000 7f76f6ffd640 1 -- 172.21.15.27:0/1360595659 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f770001a070 con 0x7f77040b2b50 2024-03-06T07:32:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.513+0000 7f76f6ffd640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f76e806b4d0 0x7f76e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.513+0000 7f76f6ffd640 1 -- 172.21.15.27:0/1360595659 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f77000893a0 con 0x7f77040b2b50 2024-03-06T07:32:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.513+0000 7f770a31d640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f76e806b4d0 0x7f76e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.514+0000 7f770a31d640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f76e806b4d0 0x7f76e806d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f76f8036000 tx=0x7f76f803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.522+0000 7f76f6ffd640 1 -- 172.21.15.27:0/1360595659 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7700057c40 con 0x7f77040b2b50 2024-03-06T07:32:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.744+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77040b91d0 con 0x7f77040b2b50 2024-03-06T07:32:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.744+0000 7f76f6ffd640 1 -- 172.21.15.27:0/1360595659 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f770005bce0 con 0x7f77040b2b50 2024-03-06T07:32:46.747 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:46.747 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f76e806b4d0 msgr2=0x7f76e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f76e806b4d0 0x7f76e806d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f76f8036000 tx=0x7f76f803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77040b2b50 msgr2=0x7f77040cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77040b2b50 0x7f77040cc890 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f770000c3c0 tx=0x7f770000c880 comp rx=0 tx=0).stop 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 shutdown_connections 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f76e806b4d0 0x7f76e806d990 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f77040d3400 0x7f77041f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77040b2b50 0x7f77040cc890 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 --2- 172.21.15.27:0/1360595659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7704071540 0x7f77040cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 >> 172.21.15.27:0/1360595659 conn(0x7f77040d3ca0 msgr2=0x7f77040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 shutdown_connections 2024-03-06T07:32:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:46.746+0000 7f770c581640 1 -- 172.21.15.27:0/1360595659 wait complete. 2024-03-06T07:32:46.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:46.965 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:47.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:48.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.071+0000 7f225a888640 1 Processor -- start 2024-03-06T07:32:48.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.071+0000 7f225a888640 1 -- start start 2024-03-06T07:32:48.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.072+0000 7f225a888640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2254071540 0x7f22540b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.072+0000 7f225a888640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f22540d3400 0x7f2254071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.072+0000 7f225a888640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 0x7f22540d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.072+0000 7f225a888640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22540d2f70 con 0x7f2254071540 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.072+0000 7f225a888640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22540d4be0 con 0x7f22540d3400 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.072+0000 7f225a888640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2254071e60 con 0x7f22540b2b50 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.072+0000 7f2253fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2254071540 0x7f22540b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.072+0000 7f2253fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2254071540 0x7f22540b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50936/0 (socket says 172.21.15.27:50936) 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.072+0000 7f2253fff640 1 -- 172.21.15.27:0/2575728996 learned_addr learned my addr 172.21.15.27:0/2575728996 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.073+0000 7f22537fe640 1 --2- 172.21.15.27:0/2575728996 >> v2:172.21.15.69:3300/0 conn(0x7f22540d3400 0x7f2254071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.073+0000 7f2258e25640 1 --2- 172.21.15.27:0/2575728996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 0x7f22540d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.073+0000 7f2253fff640 1 -- 172.21.15.27:0/2575728996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 msgr2=0x7f22540d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.073+0000 7f2253fff640 1 --2- 172.21.15.27:0/2575728996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 0x7f22540d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.073+0000 7f2253fff640 1 -- 172.21.15.27:0/2575728996 >> v2:172.21.15.69:3300/0 conn(0x7f22540d3400 msgr2=0x7f2254071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.073+0000 7f2253fff640 1 --2- 172.21.15.27:0/2575728996 >> v2:172.21.15.69:3300/0 conn(0x7f22540d3400 0x7f2254071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.073+0000 7f2253fff640 1 -- 172.21.15.27:0/2575728996 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540ce610 con 0x7f2254071540 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.074+0000 7f2253fff640 1 --2- 172.21.15.27:0/2575728996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2254071540 0x7f22540b30a0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f22540b6860 tx=0x7f224402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e7266369770ddd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.074+0000 7f2252ffd640 1 -- 172.21.15.27:0/2575728996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244035020 con 0x7f2254071540 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.074+0000 7f2252ffd640 1 -- 172.21.15.27:0/2575728996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2244031870 con 0x7f2254071540 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.074+0000 7f2252ffd640 1 -- 172.21.15.27:0/2575728996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244031c40 con 0x7f2254071540 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.075+0000 7f225a888640 1 -- 172.21.15.27:0/2575728996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2254071540 msgr2=0x7f22540b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.075+0000 7f225a888640 1 --2- 172.21.15.27:0/2575728996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2254071540 0x7f22540b30a0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f22540b6860 tx=0x7f224402b910 comp rx=0 tx=0).stop 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.075+0000 7f225a888640 1 -- 172.21.15.27:0/2575728996 shutdown_connections 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.075+0000 7f225a888640 1 --2- 172.21.15.27:0/2575728996 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 0x7f22540d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.075+0000 7f225a888640 1 --2- 172.21.15.27:0/2575728996 >> v2:172.21.15.69:3300/0 conn(0x7f22540d3400 0x7f2254071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.075+0000 7f225a888640 1 --2- 172.21.15.27:0/2575728996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2254071540 0x7f22540b30a0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.075+0000 7f225a888640 1 -- 172.21.15.27:0/2575728996 >> 172.21.15.27:0/2575728996 conn(0x7f22540d3ca0 msgr2=0x7f22540d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.075+0000 7f225a888640 1 -- 172.21.15.27:0/2575728996 shutdown_connections 2024-03-06T07:32:48.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.075+0000 7f225a888640 1 -- 172.21.15.27:0/2575728996 wait complete. 2024-03-06T07:32:48.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.076+0000 7f225a888640 1 Processor -- start 2024-03-06T07:32:48.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.076+0000 7f225a888640 1 -- start start 2024-03-06T07:32:48.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f225a888640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254071540 0x7f22541e9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f225a888640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 0x7f22540b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f225a888640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22540d3400 0x7f22540b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f225a888640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22540ce8f0 con 0x7f22540d3400 2024-03-06T07:32:48.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f225a888640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22540d2c00 con 0x7f2254071540 2024-03-06T07:32:48.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f225a888640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225415e370 con 0x7f22540b2b50 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2258e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22540d3400 0x7f22540b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2258e25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22540d3400 0x7f22540b8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50946/0 (socket says 172.21.15.27:50946) 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2258e25640 1 -- 172.21.15.27:0/2955018074 learned_addr learned my addr 172.21.15.27:0/2955018074 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f22537fe640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 0x7f22540b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2253fff640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254071540 0x7f22541e9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2258e25640 1 -- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 msgr2=0x7f22540b88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2258e25640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 0x7f22540b88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2258e25640 1 -- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254071540 msgr2=0x7f22541e9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2258e25640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254071540 0x7f22541e9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2258e25640 1 -- 172.21.15.27:0/2955018074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22541495c0 con 0x7f22540d3400 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.077+0000 7f2258e25640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22540d3400 0x7f22540b8df0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f224c007e60 tx=0x7f224c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.078+0000 7f2250ff9640 1 -- 172.21.15.27:0/2955018074 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c014620 con 0x7f22540d3400 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.078+0000 7f2250ff9640 1 -- 172.21.15.27:0/2955018074 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c019070 con 0x7f22540d3400 2024-03-06T07:32:48.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.078+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22541498a0 con 0x7f22540d3400 2024-03-06T07:32:48.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.078+0000 7f2250ff9640 1 -- 172.21.15.27:0/2955018074 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c014a20 con 0x7f22540d3400 2024-03-06T07:32:48.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.078+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2254149e60 con 0x7f22540d3400 2024-03-06T07:32:48.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.079+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225414a060 con 0x7f22540d3400 2024-03-06T07:32:48.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.079+0000 7f2250ff9640 1 -- 172.21.15.27:0/2955018074 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f224c014bc0 con 0x7f22540d3400 2024-03-06T07:32:48.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.080+0000 7f2250ff9640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f224006b500 0x7f224006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.080+0000 7f2250ff9640 1 -- 172.21.15.27:0/2955018074 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f224c08a880 con 0x7f22540d3400 2024-03-06T07:32:48.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.080+0000 7f2253fff640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f224006b500 0x7f224006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.081+0000 7f2253fff640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f224006b500 0x7f224006d9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f2244036000 tx=0x7f224403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.084+0000 7f2250ff9640 1 -- 172.21.15.27:0/2955018074 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f225414a060 con 0x7f22540d3400 2024-03-06T07:32:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.305+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2254068700 con 0x7f22540d3400 2024-03-06T07:32:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.306+0000 7f2250ff9640 1 -- 172.21.15.27:0/2955018074 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f224c059100 con 0x7f22540d3400 2024-03-06T07:32:48.309 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:48.309 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f224006b500 msgr2=0x7f224006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f224006b500 0x7f224006d9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f2244036000 tx=0x7f224403c040 comp rx=0 tx=0).stop 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22540d3400 msgr2=0x7f22540b8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22540d3400 0x7f22540b8df0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f224c007e60 tx=0x7f224c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 shutdown_connections 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f224006b500 0x7f224006d9c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22540d3400 0x7f22540b8df0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22540b2b50 0x7f22540b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 --2- 172.21.15.27:0/2955018074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2254071540 0x7f22541e9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 >> 172.21.15.27:0/2955018074 conn(0x7f22540d3ca0 msgr2=0x7f225415fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 shutdown_connections 2024-03-06T07:32:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:48.308+0000 7f225a888640 1 -- 172.21.15.27:0/2955018074 wait complete. 2024-03-06T07:32:48.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:48.523 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:49.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:49.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.629+0000 7fb784cc2640 1 Processor -- start 2024-03-06T07:32:49.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.629+0000 7fb784cc2640 1 -- start start 2024-03-06T07:32:49.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.630+0000 7fb784cc2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb780158520 0x7fb78006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.630+0000 7fb784cc2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7800b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.630+0000 7fb784cc2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb780071240 con 0x7fb78006dd60 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.630+0000 7fb784cc2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb780071b70 con 0x7fb780158520 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.630+0000 7fb784cc2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb780071520 con 0x7fb7800aee10 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77e59c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb780158520 0x7fb78006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77dd9b640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb78006dd60 0x7fb7800ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:47194/0 (socket says 172.21.15.27:47194) 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77ed9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7800b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77dd9b640 1 -- 172.21.15.27:0/768931148 learned_addr learned my addr 172.21.15.27:0/768931148 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77d59a640 1 -- 172.21.15.27:0/768931148 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213602257 0 0) 0x7fb780071240 con 0x7fb78006dd60 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77d59a640 1 -- 172.21.15.27:0/768931148 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb768005730 con 0x7fb78006dd60 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77ed9d640 1 -- 172.21.15.27:0/768931148 >> v2:172.21.15.69:3300/0 conn(0x7fb780158520 msgr2=0x7fb78006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77ed9d640 1 --2- 172.21.15.27:0/768931148 >> v2:172.21.15.69:3300/0 conn(0x7fb780158520 0x7fb78006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77ed9d640 1 -- 172.21.15.27:0/768931148 >> v1:172.21.15.69:6789/0 conn(0x7fb78006dd60 legacy=0x7fb7800ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77ed9d640 1 -- 172.21.15.27:0/768931148 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb780071800 con 0x7fb7800aee10 2024-03-06T07:32:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.631+0000 7fb77e59c640 1 --2- 172.21.15.27:0/768931148 >> v2:172.21.15.69:3300/0 conn(0x7fb780158520 0x7fb78006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.632+0000 7fb77ed9d640 1 --2- 172.21.15.27:0/768931148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7800b3640 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb778009d90 tx=0x7fb77802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82709cf9d4e35df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.632+0000 7fb77d59a640 1 -- 172.21.15.27:0/768931148 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778039070 con 0x7fb7800aee10 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.632+0000 7fb77d59a640 1 -- 172.21.15.27:0/768931148 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77802cc50 con 0x7fb7800aee10 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.632+0000 7fb77d59a640 1 -- 172.21.15.27:0/768931148 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778034440 con 0x7fb7800aee10 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.633+0000 7fb784cc2640 1 -- 172.21.15.27:0/768931148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 msgr2=0x7fb7800b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.633+0000 7fb784cc2640 1 --2- 172.21.15.27:0/768931148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7800b3640 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb778009d90 tx=0x7fb77802bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.633+0000 7fb784cc2640 1 -- 172.21.15.27:0/768931148 shutdown_connections 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.633+0000 7fb784cc2640 1 --2- 172.21.15.27:0/768931148 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7800b3640 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.633+0000 7fb784cc2640 1 --2- 172.21.15.27:0/768931148 >> v2:172.21.15.69:3300/0 conn(0x7fb780158520 0x7fb78006d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.633+0000 7fb784cc2640 1 -- 172.21.15.27:0/768931148 >> 172.21.15.27:0/768931148 conn(0x7fb780066d00 msgr2=0x7fb7800670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.633+0000 7fb784cc2640 1 -- 172.21.15.27:0/768931148 shutdown_connections 2024-03-06T07:32:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.633+0000 7fb784cc2640 1 -- 172.21.15.27:0/768931148 wait complete. 2024-03-06T07:32:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.635+0000 7fb784cc2640 1 Processor -- start 2024-03-06T07:32:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.635+0000 7fb784cc2640 1 -- start start 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb784cc2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78006dd60 0x7fb780146320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb784cc2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7801566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb784cc2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb780158520 0x7fb7800cf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb784cc2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb780158a80 con 0x7fb78006dd60 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb784cc2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb780158900 con 0x7fb780158520 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb784cc2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7800b3b80 con 0x7fb7800aee10 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78006dd60 0x7fb780146320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78006dd60 0x7fb780146320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50960/0 (socket says 172.21.15.27:50960) 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77e59c640 1 -- 172.21.15.27:0/160484596 learned_addr learned my addr 172.21.15.27:0/160484596 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77dd9b640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7801566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77ed9d640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb780158520 0x7fb7800cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77e59c640 1 -- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 msgr2=0x7fb7801566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77e59c640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7801566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77e59c640 1 -- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb780158520 msgr2=0x7fb7800cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77e59c640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb780158520 0x7fb7800cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77e59c640 1 -- 172.21.15.27:0/160484596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb780155610 con 0x7fb78006dd60 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77ed9d640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb780158520 0x7fb7800cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77dd9b640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7801566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb77e59c640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78006dd60 0x7fb780146320 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fb77000e960 tx=0x7fb77000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.637+0000 7fb76effd640 1 -- 172.21.15.27:0/160484596 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770017070 con 0x7fb78006dd60 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.638+0000 7fb76effd640 1 -- 172.21.15.27:0/160484596 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77000ca00 con 0x7fb78006dd60 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.638+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7801558f0 con 0x7fb78006dd60 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.638+0000 7fb76effd640 1 -- 172.21.15.27:0/160484596 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77000cdd0 con 0x7fb78006dd60 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.638+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb780155eb0 con 0x7fb78006dd60 2024-03-06T07:32:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.638+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7801562f0 con 0x7fb78006dd60 2024-03-06T07:32:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.639+0000 7fb76effd640 1 -- 172.21.15.27:0/160484596 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb77001a070 con 0x7fb78006dd60 2024-03-06T07:32:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.639+0000 7fb76effd640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76006b4d0 0x7fb76006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.640+0000 7fb76effd640 1 -- 172.21.15.27:0/160484596 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb77008a490 con 0x7fb78006dd60 2024-03-06T07:32:49.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.640+0000 7fb77dd9b640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76006b4d0 0x7fb76006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.640+0000 7fb77dd9b640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76006b4d0 0x7fb76006d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb7801f2de0 tx=0x7fb774009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.649+0000 7fb76effd640 1 -- 172.21.15.27:0/160484596 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb770059300 con 0x7fb78006dd60 2024-03-06T07:32:49.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:49.806 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.869+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7800b31f0 con 0x7fb78006dd60 2024-03-06T07:32:49.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.869+0000 7fb76effd640 1 -- 172.21.15.27:0/160484596 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7fb770058c00 con 0x7fb78006dd60 2024-03-06T07:32:49.872 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:49.872 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:49.872 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76006b4d0 msgr2=0x7fb76006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76006b4d0 0x7fb76006d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb7801f2de0 tx=0x7fb774009040 comp rx=0 tx=0).stop 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78006dd60 msgr2=0x7fb780146320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78006dd60 0x7fb780146320 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fb77000e960 tx=0x7fb77000ee20 comp rx=0 tx=0).stop 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 shutdown_connections 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76006b4d0 0x7fb76006d990 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb780158520 0x7fb7800cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7800aee10 0x7fb7801566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 --2- 172.21.15.27:0/160484596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78006dd60 0x7fb780146320 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 >> 172.21.15.27:0/160484596 conn(0x7fb780066d00 msgr2=0x7fb780159d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 shutdown_connections 2024-03-06T07:32:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:49.871+0000 7fb784cc2640 1 -- 172.21.15.27:0/160484596 wait complete. 2024-03-06T07:32:50.086 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:50.086 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:51.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:51.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.192+0000 7f5515cd4640 1 Processor -- start 2024-03-06T07:32:51.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.192+0000 7f5515cd4640 1 -- start start 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f5515cd4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f551015df00 0x7f55100d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f5515cd4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f55100b9a30 0x7f55100d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f5515cd4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55100cea30 con 0x7f55100b95a0 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f5515cd4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55100d2af0 con 0x7f551015df00 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f5515cd4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55100cf3c0 con 0x7f55100b9a30 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f550f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f551015df00 0x7f55100d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f550effd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f55100b95a0 0x7f55100b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44700/0 (socket says 172.21.15.27:44700) 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f550effd640 1 -- 172.21.15.27:0/3232012449 learned_addr learned my addr 172.21.15.27:0/3232012449 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f550ffff640 1 --2- 172.21.15.27:0/3232012449 >> v2:172.21.15.69:3300/0 conn(0x7f55100b9a30 0x7f55100d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f550f7fe640 1 -- 172.21.15.27:0/3232012449 >> v2:172.21.15.69:3300/0 conn(0x7f55100b9a30 msgr2=0x7f55100d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f550f7fe640 1 --2- 172.21.15.27:0/3232012449 >> v2:172.21.15.69:3300/0 conn(0x7f55100b9a30 0x7f55100d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f550f7fe640 1 -- 172.21.15.27:0/3232012449 >> v1:172.21.15.27:6789/0 conn(0x7f55100b95a0 legacy=0x7f55100b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f550f7fe640 1 -- 172.21.15.27:0/3232012449 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100cf0e0 con 0x7f551015df00 2024-03-06T07:32:51.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.193+0000 7f550e7fc640 1 -- 172.21.15.27:0/3232012449 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950246552 0 0) 0x7f55100cea30 con 0x7f55100b95a0 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.194+0000 7f550f7fe640 1 --2- 172.21.15.27:0/3232012449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f551015df00 0x7f55100d25b0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f55100d4d20 tx=0x7f550002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c8157d0cd89bfc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.194+0000 7f550e7fc640 1 -- 172.21.15.27:0/3232012449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500035020 con 0x7f551015df00 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.194+0000 7f550e7fc640 1 -- 172.21.15.27:0/3232012449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5500031830 con 0x7f551015df00 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.194+0000 7f550e7fc640 1 -- 172.21.15.27:0/3232012449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500031c00 con 0x7f551015df00 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.195+0000 7f5515cd4640 1 -- 172.21.15.27:0/3232012449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f551015df00 msgr2=0x7f55100d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.195+0000 7f5515cd4640 1 --2- 172.21.15.27:0/3232012449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f551015df00 0x7f55100d25b0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f55100d4d20 tx=0x7f550002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.195+0000 7f5515cd4640 1 -- 172.21.15.27:0/3232012449 shutdown_connections 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.195+0000 7f5515cd4640 1 --2- 172.21.15.27:0/3232012449 >> v2:172.21.15.69:3300/0 conn(0x7f55100b9a30 0x7f55100d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.195+0000 7f5515cd4640 1 --2- 172.21.15.27:0/3232012449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f551015df00 0x7f55100d25b0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.195+0000 7f5515cd4640 1 -- 172.21.15.27:0/3232012449 >> 172.21.15.27:0/3232012449 conn(0x7f5510071960 msgr2=0x7f5510071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.195+0000 7f5515cd4640 1 -- 172.21.15.27:0/3232012449 shutdown_connections 2024-03-06T07:32:51.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.195+0000 7f5515cd4640 1 -- 172.21.15.27:0/3232012449 wait complete. 2024-03-06T07:32:51.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.196+0000 7f5515cd4640 1 Processor -- start 2024-03-06T07:32:51.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.196+0000 7f5515cd4640 1 -- start start 2024-03-06T07:32:51.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f5515cd4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55100b95a0 0x7f55100c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f5515cd4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f551015df00 0x7f5510157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f5515cd4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55100c8580 0x7f55100b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f5515cd4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55100cec70 con 0x7f55100c8580 2024-03-06T07:32:51.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f5515cd4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55100ce750 con 0x7f551015df00 2024-03-06T07:32:51.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f5515cd4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55100cedf0 con 0x7f55100b95a0 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55100c8580 0x7f55100b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55100c8580 0x7f55100b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50984/0 (socket says 172.21.15.27:50984) 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550ffff640 1 -- 172.21.15.27:0/3860616250 learned_addr learned my addr 172.21.15.27:0/3860616250 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550effd640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f551015df00 0x7f5510157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550f7fe640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55100b95a0 0x7f55100c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550ffff640 1 -- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55100b95a0 msgr2=0x7f55100c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550ffff640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55100b95a0 0x7f55100c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550ffff640 1 -- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f551015df00 msgr2=0x7f5510157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550ffff640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f551015df00 0x7f5510157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550ffff640 1 -- 172.21.15.27:0/3860616250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100c90e0 con 0x7f55100c8580 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.197+0000 7f550effd640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f551015df00 0x7f5510157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.198+0000 7f550ffff640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55100c8580 0x7f55100b49a0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f5508007e40 tx=0x7f550800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.198+0000 7f54f7fff640 1 -- 172.21.15.27:0/3860616250 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508014620 con 0x7f55100c8580 2024-03-06T07:32:51.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.198+0000 7f54f7fff640 1 -- 172.21.15.27:0/3860616250 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5508019070 con 0x7f55100c8580 2024-03-06T07:32:51.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.198+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551015e2e0 con 0x7f55100c8580 2024-03-06T07:32:51.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.198+0000 7f54f7fff640 1 -- 172.21.15.27:0/3860616250 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508014a20 con 0x7f55100c8580 2024-03-06T07:32:51.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.198+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55100c8e00 con 0x7f55100c8580 2024-03-06T07:32:51.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.199+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5510067ff0 con 0x7f55100c8580 2024-03-06T07:32:51.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.199+0000 7f54f7fff640 1 -- 172.21.15.27:0/3860616250 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5508014bc0 con 0x7f55100c8580 2024-03-06T07:32:51.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.200+0000 7f54f7fff640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f54fc06b4d0 0x7f54fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.200+0000 7f54f7fff640 1 -- 172.21.15.27:0/3860616250 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f550808a9b0 con 0x7f55100c8580 2024-03-06T07:32:51.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.200+0000 7f550f7fe640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f54fc06b4d0 0x7f54fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.201+0000 7f550f7fe640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f54fc06b4d0 0x7f54fc06d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5500036000 tx=0x7f550003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.204+0000 7f54f7fff640 1 -- 172.21.15.27:0/3860616250 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5508059810 con 0x7f55100c8580 2024-03-06T07:32:51.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.423+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55100c7d80 con 0x7f55100c8580 2024-03-06T07:32:51.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.423+0000 7f54f7fff640 1 -- 172.21.15.27:0/3860616250 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f5508059110 con 0x7f55100c8580 2024-03-06T07:32:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:51.426 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:51.426 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:51.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f54fc06b4d0 msgr2=0x7f54fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f54fc06b4d0 0x7f54fc06d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5500036000 tx=0x7f550003c040 comp rx=0 tx=0).stop 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55100c8580 msgr2=0x7f55100b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55100c8580 0x7f55100b49a0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f5508007e40 tx=0x7f550800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 shutdown_connections 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f54fc06b4d0 0x7f54fc06d990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55100c8580 0x7f55100b49a0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f551015df00 0x7f5510157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 --2- 172.21.15.27:0/3860616250 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f55100b95a0 0x7f55100c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 >> 172.21.15.27:0/3860616250 conn(0x7f5510071960 msgr2=0x7f55101618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 shutdown_connections 2024-03-06T07:32:51.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:51.425+0000 7f5515cd4640 1 -- 172.21.15.27:0/3860616250 wait complete. 2024-03-06T07:32:51.640 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:51.640 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:52.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.745+0000 7fdd024c2640 1 Processor -- start 2024-03-06T07:32:52.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.745+0000 7fdd024c2640 1 -- start start 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.746+0000 7fdd024c2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcfc0b4260 0x7fdcfc0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.746+0000 7fdd024c2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc0b2320 con 0x7fdcfc0d06c0 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.746+0000 7fdd024c2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc0b0dd0 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.746+0000 7fdd024c2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcfc0b0b80 con 0x7fdcfc0b4260 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.746+0000 7fdcfb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcfc0b4260 0x7fdcfc0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.746+0000 7fdd00a5f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdcfc0d06c0 0x7fdcfc0b1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44714/0 (socket says 172.21.15.27:44714) 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.746+0000 7fdd00a5f640 1 -- 172.21.15.27:0/3400793655 learned_addr learned my addr 172.21.15.27:0/3400793655 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.747+0000 7fdcfb7fe640 1 -- 172.21.15.27:0/3400793655 >> v1:172.21.15.69:6789/0 conn(0x7fdcfc0d02e0 legacy=0x7fdcfc0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.747+0000 7fdcfb7fe640 1 -- 172.21.15.27:0/3400793655 >> v1:172.21.15.27:6789/0 conn(0x7fdcfc0d06c0 legacy=0x7fdcfc0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.747+0000 7fdcfaffd640 1 -- 172.21.15.27:0/3400793655 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231437100 0 0) 0x7fdcfc0b2320 con 0x7fdcfc0d06c0 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.747+0000 7fdcfb7fe640 1 -- 172.21.15.27:0/3400793655 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc0b10b0 con 0x7fdcfc0b4260 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdcfb7fe640 1 --2- 172.21.15.27:0/3400793655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0b4260 0x7fdcfc0cfa00 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00097a0 tx=0x7fdcf002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e98ca4ec7b1ab58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdcfaffd640 1 -- 172.21.15.27:0/3400793655 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0034070 con 0x7fdcfc0b4260 2024-03-06T07:32:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdcfaffd640 1 -- 172.21.15.27:0/3400793655 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf0030610 con 0x7fdcfc0b4260 2024-03-06T07:32:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdcfaffd640 1 -- 172.21.15.27:0/3400793655 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf00309e0 con 0x7fdcfc0b4260 2024-03-06T07:32:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdd024c2640 1 -- 172.21.15.27:0/3400793655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0b4260 msgr2=0x7fdcfc0cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdd024c2640 1 --2- 172.21.15.27:0/3400793655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0b4260 0x7fdcfc0cfa00 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00097a0 tx=0x7fdcf002b960 comp rx=0 tx=0).stop 2024-03-06T07:32:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdd024c2640 1 -- 172.21.15.27:0/3400793655 shutdown_connections 2024-03-06T07:32:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdd024c2640 1 --2- 172.21.15.27:0/3400793655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0b4260 0x7fdcfc0cfa00 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdd024c2640 1 -- 172.21.15.27:0/3400793655 >> 172.21.15.27:0/3400793655 conn(0x7fdcfc0b36d0 msgr2=0x7fdcfc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdd024c2640 1 -- 172.21.15.27:0/3400793655 shutdown_connections 2024-03-06T07:32:52.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.748+0000 7fdd024c2640 1 -- 172.21.15.27:0/3400793655 wait complete. 2024-03-06T07:32:52.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.749+0000 7fdd024c2640 1 Processor -- start 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.750+0000 7fdd024c2640 1 -- start start 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.750+0000 7fdd024c2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcfc0b4260 0x7fdcfc144a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.750+0000 7fdd024c2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0d02e0 0x7fdcfc0c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.750+0000 7fdd024c2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcfc0d06c0 0x7fdcfc1f5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.750+0000 7fdd024c2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc0b0360 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.750+0000 7fdd024c2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc0aeec0 con 0x7fdcfc0d06c0 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.750+0000 7fdd024c2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc0b04e0 con 0x7fdcfc0b4260 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0d02e0 0x7fdcfc0c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0d02e0 0x7fdcfc0c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50990/0 (socket says 172.21.15.27:50990) 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfb7fe640 1 -- 172.21.15.27:0/677347525 learned_addr learned my addr 172.21.15.27:0/677347525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdd00a5f640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcfc0d06c0 0x7fdcfc1f5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfbfff640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcfc0b4260 0x7fdcfc144a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfb7fe640 1 -- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcfc0b4260 msgr2=0x7fdcfc144a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfb7fe640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcfc0b4260 0x7fdcfc144a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfb7fe640 1 -- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcfc0d06c0 msgr2=0x7fdcfc1f5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfb7fe640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcfc0d06c0 0x7fdcfc1f5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfb7fe640 1 -- 172.21.15.27:0/677347525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc0c9e30 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcfb7fe640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0d02e0 0x7fdcfc0c8150 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0000c00 tx=0x7fdcf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcf8ff9640 1 -- 172.21.15.27:0/677347525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0047070 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcf8ff9640 1 -- 172.21.15.27:0/677347525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf0039a10 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcfc1420c0 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdcf8ff9640 1 -- 172.21.15.27:0/677347525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0034040 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.751+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc142680 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.752+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcfc0ca000 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.753+0000 7fdcf8ff9640 1 -- 172.21.15.27:0/677347525 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcf004a070 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.754+0000 7fdcf8ff9640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdce806b4d0 0x7fdce806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.754+0000 7fdcf8ff9640 1 -- 172.21.15.27:0/677347525 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdcf00abc40 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.754+0000 7fdcfbfff640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdce806b4d0 0x7fdce806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.754+0000 7fdcfbfff640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdce806b4d0 0x7fdce806d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fdcec00a750 tx=0x7fdcec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.758+0000 7fdcf8ff9640 1 -- 172.21.15.27:0/677347525 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdcf007ab30 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.978+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcfc068700 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.978+0000 7fdcf8ff9640 1 -- 172.21.15.27:0/677347525 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7fdcf007a430 con 0x7fdcfc0d02e0 2024-03-06T07:32:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:52.981 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:52.981 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdce806b4d0 msgr2=0x7fdce806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdce806b4d0 0x7fdce806d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fdcec00a750 tx=0x7fdcec008040 comp rx=0 tx=0).stop 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0d02e0 msgr2=0x7fdcfc0c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0d02e0 0x7fdcfc0c8150 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0000c00 tx=0x7fdcf00046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 shutdown_connections 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdce806b4d0 0x7fdce806d990 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcfc0d06c0 0x7fdcfc1f5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcfc0d02e0 0x7fdcfc0c8150 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 --2- 172.21.15.27:0/677347525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdcfc0b4260 0x7fdcfc144a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 >> 172.21.15.27:0/677347525 conn(0x7fdcfc0b36d0 msgr2=0x7fdcfc0b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 shutdown_connections 2024-03-06T07:32:52.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:52.980+0000 7fdd024c2640 1 -- 172.21.15.27:0/677347525 wait complete. 2024-03-06T07:32:53.195 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:53.195 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:54.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:54.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.301+0000 7f5ed4488640 1 Processor -- start 2024-03-06T07:32:54.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.301+0000 7f5ed4488640 1 -- start start 2024-03-06T07:32:54.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.302+0000 7f5ed4488640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ecc15df00 0x7f5ecc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.302+0000 7f5ed4488640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.302+0000 7f5ed4488640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc0cea30 con 0x7f5ecc0b9a30 2024-03-06T07:32:54.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.302+0000 7f5ed4488640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc0d2af0 con 0x7f5ecc15df00 2024-03-06T07:32:54.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.302+0000 7f5ed4488640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0cf3c0 con 0x7f5ecc0b95a0 2024-03-06T07:32:54.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed1a23640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed2224640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ecc15df00 0x7f5ecc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed2a25640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5ecc0b9a30 0x7f5ecc0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:40880/0 (socket says 172.21.15.27:40880) 2024-03-06T07:32:54.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed1a23640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:33598/0 (socket says 172.21.15.27:33598) 2024-03-06T07:32:54.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed2224640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ecc15df00 0x7f5ecc0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:36096/0 (socket says 172.21.15.27:36096) 2024-03-06T07:32:54.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed1a23640 1 -- 172.21.15.27:0/3928874495 learned_addr learned my addr 172.21.15.27:0/3928874495 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed1222640 1 -- 172.21.15.27:0/3928874495 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130290008 0 0) 0x7f5ecc0cea30 con 0x7f5ecc0b9a30 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed1222640 1 -- 172.21.15.27:0/3928874495 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8003520 con 0x7f5ecc0b9a30 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed2224640 1 -- 172.21.15.27:0/3928874495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 msgr2=0x7f5ecc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed2224640 1 --2- 172.21.15.27:0/3928874495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed2224640 1 -- 172.21.15.27:0/3928874495 >> v1:172.21.15.69:6789/0 conn(0x7f5ecc0b9a30 legacy=0x7f5ecc0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed2224640 1 -- 172.21.15.27:0/3928874495 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0cf0e0 con 0x7f5ecc15df00 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.303+0000 7f5ed1a23640 1 --2- 172.21.15.27:0/3928874495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.304+0000 7f5ed2224640 1 --2- 172.21.15.27:0/3928874495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ecc15df00 0x7f5ecc0d25b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80097a0 tx=0x7f5ec802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57d4399dfcd5e25c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.304+0000 7f5ed1222640 1 -- 172.21.15.27:0/3928874495 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8035020 con 0x7f5ecc15df00 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.304+0000 7f5ed1222640 1 -- 172.21.15.27:0/3928874495 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec8031830 con 0x7f5ecc15df00 2024-03-06T07:32:54.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.304+0000 7f5ed1222640 1 -- 172.21.15.27:0/3928874495 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8031be0 con 0x7f5ecc15df00 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.304+0000 7f5ed4488640 1 -- 172.21.15.27:0/3928874495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ecc15df00 msgr2=0x7f5ecc0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.304+0000 7f5ed4488640 1 --2- 172.21.15.27:0/3928874495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ecc15df00 0x7f5ecc0d25b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80097a0 tx=0x7f5ec802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.305+0000 7f5ed4488640 1 -- 172.21.15.27:0/3928874495 shutdown_connections 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.305+0000 7f5ed4488640 1 --2- 172.21.15.27:0/3928874495 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.305+0000 7f5ed4488640 1 --2- 172.21.15.27:0/3928874495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ecc15df00 0x7f5ecc0d25b0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.305+0000 7f5ed4488640 1 -- 172.21.15.27:0/3928874495 >> 172.21.15.27:0/3928874495 conn(0x7f5ecc071960 msgr2=0x7f5ecc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.305+0000 7f5ed4488640 1 -- 172.21.15.27:0/3928874495 shutdown_connections 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.305+0000 7f5ed4488640 1 -- 172.21.15.27:0/3928874495 wait complete. 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.306+0000 7f5ed4488640 1 Processor -- start 2024-03-06T07:32:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.306+0000 7f5ed4488640 1 -- start start 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.306+0000 7f5ed4488640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.306+0000 7f5ed4488640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ecc15df00 0x7f5ecc0b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.306+0000 7f5ed4488640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ecc0b2d70 0x7f5ecc152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.306+0000 7f5ed4488640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0cec70 con 0x7f5ecc15df00 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.306+0000 7f5ed4488640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0ce750 con 0x7f5ecc0b2d70 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.306+0000 7f5ed4488640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0cedf0 con 0x7f5ecc0b95a0 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed1a23640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ecc15df00 0x7f5ecc0b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed1a23640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ecc15df00 0x7f5ecc0b5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42254/0 (socket says 172.21.15.27:42254) 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed1a23640 1 -- 172.21.15.27:0/1269290831 learned_addr learned my addr 172.21.15.27:0/1269290831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed2a25640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ecc0b2d70 0x7f5ecc152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed2224640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed1a23640 1 -- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 msgr2=0x7f5ecc0ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed1a23640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0ca090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed1a23640 1 -- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ecc0b2d70 msgr2=0x7f5ecc152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed1a23640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ecc0b2d70 0x7f5ecc152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed1a23640 1 -- 172.21.15.27:0/1269290831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0c8ff0 con 0x7f5ecc15df00 2024-03-06T07:32:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed1a23640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ecc15df00 0x7f5ecc0b5880 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000e9d0 tx=0x7f5ec000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ebeffd640 1 -- 172.21.15.27:0/1269290831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0017070 con 0x7f5ecc15df00 2024-03-06T07:32:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ebeffd640 1 -- 172.21.15.27:0/1269290831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec000ca60 con 0x7f5ecc15df00 2024-03-06T07:32:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc0c92d0 con 0x7f5ecc15df00 2024-03-06T07:32:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ebeffd640 1 -- 172.21.15.27:0/1269290831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec000ce10 con 0x7f5ecc15df00 2024-03-06T07:32:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.307+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc0c9890 con 0x7f5ecc15df00 2024-03-06T07:32:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.309+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc067ff0 con 0x7f5ecc15df00 2024-03-06T07:32:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.309+0000 7f5ebeffd640 1 -- 172.21.15.27:0/1269290831 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ec001a070 con 0x7f5ecc15df00 2024-03-06T07:32:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.309+0000 7f5ebeffd640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5eb006b4d0 0x7f5eb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.310+0000 7f5ebeffd640 1 -- 172.21.15.27:0/1269290831 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5ec008a580 con 0x7f5ecc15df00 2024-03-06T07:32:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.310+0000 7f5ed2224640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5eb006b4d0 0x7f5eb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.310+0000 7f5ed2224640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5eb006b4d0 0x7f5eb006d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8036000 tx=0x7f5ec8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.314+0000 7f5ebeffd640 1 -- 172.21.15.27:0/1269290831 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ec0059360 con 0x7f5ecc15df00 2024-03-06T07:32:54.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.619+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ecc0b7330 con 0x7f5ecc15df00 2024-03-06T07:32:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.619+0000 7f5ebeffd640 1 -- 172.21.15.27:0/1269290831 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f5ec0058c60 con 0x7f5ecc15df00 2024-03-06T07:32:54.622 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:54.622 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:54.623 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:54.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5eb006b4d0 msgr2=0x7f5eb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5eb006b4d0 0x7f5eb006d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8036000 tx=0x7f5ec8008030 comp rx=0 tx=0).stop 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ecc15df00 msgr2=0x7f5ecc0b5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ecc15df00 0x7f5ecc0b5880 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000e9d0 tx=0x7f5ec000ee90 comp rx=0 tx=0).stop 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 shutdown_connections 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5eb006b4d0 0x7f5eb006d990 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ecc0b2d70 0x7f5ecc152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ecc15df00 0x7f5ecc0b5880 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 --2- 172.21.15.27:0/1269290831 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ecc0b95a0 0x7f5ecc0ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 >> 172.21.15.27:0/1269290831 conn(0x7f5ecc071960 msgr2=0x7f5ecc0cba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 shutdown_connections 2024-03-06T07:32:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:54.623+0000 7f5ed4488640 1 -- 172.21.15.27:0/1269290831 wait complete. 2024-03-06T07:32:54.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:54.838 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:55.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:55.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.945+0000 7fd2ed5ac640 1 Processor -- start 2024-03-06T07:32:55.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.945+0000 7fd2ed5ac640 1 -- start start 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2ed5ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2e8072660 0x7fd2e815ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2ed5ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2e80b55d0 0x7fd2e8071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2ed5ac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80b4160 con 0x7fd2e806e700 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2ed5ac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e80b5b40 con 0x7fd2e8072660 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2ed5ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80b3ba0 con 0x7fd2e80b55d0 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2e67fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2e8072660 0x7fd2e815ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2e67fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2e8072660 0x7fd2e815ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42262/0 (socket says 172.21.15.27:42262) 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2e67fc640 1 -- 172.21.15.27:0/1970248876 learned_addr learned my addr 172.21.15.27:0/1970248876 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2e77fe640 1 --2- 172.21.15.27:0/1970248876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2e80b55d0 0x7fd2e8071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2e67fc640 1 -- 172.21.15.27:0/1970248876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2e80b55d0 msgr2=0x7fd2e8071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2e67fc640 1 --2- 172.21.15.27:0/1970248876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2e80b55d0 0x7fd2e8071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2e67fc640 1 -- 172.21.15.27:0/1970248876 >> v1:172.21.15.27:6789/0 conn(0x7fd2e806e700 legacy=0x7fd2e806d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2e67fc640 1 -- 172.21.15.27:0/1970248876 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80b3e80 con 0x7fd2e8072660 2024-03-06T07:32:55.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.946+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/1970248876 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294475365 0 0) 0x7fd2e80b4160 con 0x7fd2e806e700 2024-03-06T07:32:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.947+0000 7fd2e67fc640 1 --2- 172.21.15.27:0/1970248876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 0x7fd2e815ef80 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009870 tx=0x7fd2dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86f15bfa8415a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.947+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/1970248876 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc034070 con 0x7fd2e8072660 2024-03-06T07:32:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.947+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/1970248876 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc030840 con 0x7fd2e8072660 2024-03-06T07:32:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.947+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/1970248876 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc030c10 con 0x7fd2e8072660 2024-03-06T07:32:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.947+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/1970248876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 msgr2=0x7fd2e815ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.947+0000 7fd2ed5ac640 1 --2- 172.21.15.27:0/1970248876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 0x7fd2e815ef80 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009870 tx=0x7fd2dc02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.948+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/1970248876 shutdown_connections 2024-03-06T07:32:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.948+0000 7fd2ed5ac640 1 --2- 172.21.15.27:0/1970248876 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2e80b55d0 0x7fd2e8071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.948+0000 7fd2ed5ac640 1 --2- 172.21.15.27:0/1970248876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 0x7fd2e815ef80 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.948+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/1970248876 >> 172.21.15.27:0/1970248876 conn(0x7fd2e80b2720 msgr2=0x7fd2e80b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.948+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/1970248876 shutdown_connections 2024-03-06T07:32:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.948+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/1970248876 wait complete. 2024-03-06T07:32:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.949+0000 7fd2ed5ac640 1 Processor -- start 2024-03-06T07:32:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.949+0000 7fd2ed5ac640 1 -- start start 2024-03-06T07:32:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2ed5ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2e806e700 0x7fd2e8149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2ed5ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 0x7fd2e80ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2ed5ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e80b55d0 0x7fd2e80cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2ed5ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e8072040 con 0x7fd2e8072660 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2ed5ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e8071ec0 con 0x7fd2e80b55d0 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2ed5ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80721c0 con 0x7fd2e806e700 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2e67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 0x7fd2e80ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2e67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 0x7fd2e80ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42276/0 (socket says 172.21.15.27:42276) 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2e67fc640 1 -- 172.21.15.27:0/3050075836 learned_addr learned my addr 172.21.15.27:0/3050075836 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2e67fc640 1 -- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2e806e700 msgr2=0x7fd2e8149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2e67fc640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2e806e700 0x7fd2e8149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2e67fc640 1 -- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e80b55d0 msgr2=0x7fd2e80cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2e67fc640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e80b55d0 0x7fd2e80cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2e67fc640 1 -- 172.21.15.27:0/3050075836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e81499c0 con 0x7fd2e8072660 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2e67fc640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 0x7fd2e80ae5d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc000c00 tx=0x7fd2dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2cf7fe640 1 -- 172.21.15.27:0/3050075836 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc048070 con 0x7fd2e8072660 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2cf7fe640 1 -- 172.21.15.27:0/3050075836 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc030640 con 0x7fd2e8072660 2024-03-06T07:32:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e8158c60 con 0x7fd2e8072660 2024-03-06T07:32:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.950+0000 7fd2cf7fe640 1 -- 172.21.15.27:0/3050075836 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc034040 con 0x7fd2e8072660 2024-03-06T07:32:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.951+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e8159220 con 0x7fd2e8072660 2024-03-06T07:32:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.951+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e8067ff0 con 0x7fd2e8072660 2024-03-06T07:32:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.952+0000 7fd2cf7fe640 1 -- 172.21.15.27:0/3050075836 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2dc04b070 con 0x7fd2e8072660 2024-03-06T07:32:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.953+0000 7fd2cf7fe640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.953+0000 7fd2cf7fe640 1 -- 172.21.15.27:0/3050075836 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd2dc0ac0e0 con 0x7fd2e8072660 2024-03-06T07:32:55.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.953+0000 7fd2e6ffd640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.953+0000 7fd2e6ffd640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8001600 tx=0x7fd2d8007910 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:55.961+0000 7fd2cf7fe640 1 -- 172.21.15.27:0/3050075836 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2dc07afd0 con 0x7fd2e8072660 2024-03-06T07:32:56.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.181+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2e815c570 con 0x7fd2e8072660 2024-03-06T07:32:56.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.182+0000 7fd2cf7fe640 1 -- 172.21.15.27:0/3050075836 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7fd2dc07a8d0 con 0x7fd2e8072660 2024-03-06T07:32:56.184 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:56.184 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 msgr2=0x7fd2d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8001600 tx=0x7fd2d8007910 comp rx=0 tx=0).stop 2024-03-06T07:32:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 msgr2=0x7fd2e80ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 0x7fd2e80ae5d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc000c00 tx=0x7fd2dc004730 comp rx=0 tx=0).stop 2024-03-06T07:32:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 shutdown_connections 2024-03-06T07:32:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2e80b55d0 0x7fd2e80cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e8072660 0x7fd2e80ae5d0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 --2- 172.21.15.27:0/3050075836 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2e806e700 0x7fd2e8149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 >> 172.21.15.27:0/3050075836 conn(0x7fd2e80b2720 msgr2=0x7fd2e80b1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 shutdown_connections 2024-03-06T07:32:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:56.184+0000 7fd2ed5ac640 1 -- 172.21.15.27:0/3050075836 wait complete. 2024-03-06T07:32:56.398 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:56.398 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:57.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:57.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.505+0000 7f331f05c640 1 Processor -- start 2024-03-06T07:32:57.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.505+0000 7f331f05c640 1 -- start start 2024-03-06T07:32:57.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f331f05c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f331815df00 0x7f33180d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f331f05c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180b9a30 0x7f33180d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f331f05c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33180cea30 con 0x7f33180b95a0 2024-03-06T07:32:57.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f331f05c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33180d2af0 con 0x7f331815df00 2024-03-06T07:32:57.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f331f05c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33180cf3c0 con 0x7f33180b9a30 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f331cdf8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f331815df00 0x7f33180d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f331d5f9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180b9a30 0x7f33180d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f331cdf8640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f331815df00 0x7f33180d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:36140/0 (socket says 172.21.15.27:36140) 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f3317fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f33180b95a0 0x7f33180b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:40888/0 (socket says 172.21.15.27:40888) 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.506+0000 7f331cdf8640 1 -- 172.21.15.27:0/2324458519 learned_addr learned my addr 172.21.15.27:0/2324458519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.507+0000 7f331d5f9640 1 -- 172.21.15.27:0/2324458519 >> v2:172.21.15.69:3300/0 conn(0x7f331815df00 msgr2=0x7f33180d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.507+0000 7f331d5f9640 1 --2- 172.21.15.27:0/2324458519 >> v2:172.21.15.69:3300/0 conn(0x7f331815df00 0x7f33180d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.507+0000 7f331d5f9640 1 -- 172.21.15.27:0/2324458519 >> v1:172.21.15.69:6789/0 conn(0x7f33180b95a0 legacy=0x7f33180b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.507+0000 7f331d5f9640 1 -- 172.21.15.27:0/2324458519 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180cf0e0 con 0x7f33180b9a30 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.507+0000 7f33177fe640 1 -- 172.21.15.27:0/2324458519 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044272043 0 0) 0x7f33180cea30 con 0x7f33180b95a0 2024-03-06T07:32:57.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.507+0000 7f331cdf8640 1 --2- 172.21.15.27:0/2324458519 >> v2:172.21.15.69:3300/0 conn(0x7f331815df00 0x7f33180d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f331d5f9640 1 --2- 172.21.15.27:0/2324458519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180b9a30 0x7f33180d3220 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3310009d90 tx=0x7f331002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=621e89f87d6fb839 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f33177fe640 1 -- 172.21.15.27:0/2324458519 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310039070 con 0x7f33180b9a30 2024-03-06T07:32:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f33177fe640 1 -- 172.21.15.27:0/2324458519 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331002cc50 con 0x7f33180b9a30 2024-03-06T07:32:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f33177fe640 1 -- 172.21.15.27:0/2324458519 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310034440 con 0x7f33180b9a30 2024-03-06T07:32:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f331f05c640 1 -- 172.21.15.27:0/2324458519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180b9a30 msgr2=0x7f33180d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f331f05c640 1 --2- 172.21.15.27:0/2324458519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180b9a30 0x7f33180d3220 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3310009d90 tx=0x7f331002bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f331f05c640 1 -- 172.21.15.27:0/2324458519 shutdown_connections 2024-03-06T07:32:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f331f05c640 1 --2- 172.21.15.27:0/2324458519 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180b9a30 0x7f33180d3220 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f331f05c640 1 --2- 172.21.15.27:0/2324458519 >> v2:172.21.15.69:3300/0 conn(0x7f331815df00 0x7f33180d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f331f05c640 1 -- 172.21.15.27:0/2324458519 >> 172.21.15.27:0/2324458519 conn(0x7f3318071960 msgr2=0x7f3318071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f331f05c640 1 -- 172.21.15.27:0/2324458519 shutdown_connections 2024-03-06T07:32:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.508+0000 7f331f05c640 1 -- 172.21.15.27:0/2324458519 wait complete. 2024-03-06T07:32:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.510+0000 7f331f05c640 1 Processor -- start 2024-03-06T07:32:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.510+0000 7f331f05c640 1 -- start start 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.510+0000 7f331f05c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33180b95a0 0x7f3318158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.510+0000 7f331f05c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f331815df00 0x7f33180cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.510+0000 7f331f05c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180cd900 0x7f33180b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.510+0000 7f331f05c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33180cec70 con 0x7f33180b95a0 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.510+0000 7f331f05c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33180ce750 con 0x7f331815df00 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.510+0000 7f331f05c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33180cedf0 con 0x7f33180cd900 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331cdf8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33180b95a0 0x7f3318158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331cdf8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33180b95a0 0x7f3318158380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42292/0 (socket says 172.21.15.27:42292) 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331cdf8640 1 -- 172.21.15.27:0/3434922119 learned_addr learned my addr 172.21.15.27:0/3434922119 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f3317fff640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f331815df00 0x7f33180cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331d5f9640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180cd900 0x7f33180b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331cdf8640 1 -- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180cd900 msgr2=0x7f33180b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331cdf8640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180cd900 0x7f33180b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331cdf8640 1 -- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f331815df00 msgr2=0x7f33180cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331cdf8640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f331815df00 0x7f33180cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331cdf8640 1 -- 172.21.15.27:0/3434922119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331815ef40 con 0x7f33180b95a0 2024-03-06T07:32:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331d5f9640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180cd900 0x7f33180b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f3317fff640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f331815df00 0x7f33180cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331cdf8640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33180b95a0 0x7f3318158380 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f330c00e9b0 tx=0x7f330c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f33157fa640 1 -- 172.21.15.27:0/3434922119 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c017070 con 0x7f33180b95a0 2024-03-06T07:32:57.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f33157fa640 1 -- 172.21.15.27:0/3434922119 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c00c9a0 con 0x7f33180b95a0 2024-03-06T07:32:57.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331815f220 con 0x7f33180b95a0 2024-03-06T07:32:57.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.511+0000 7f33157fa640 1 -- 172.21.15.27:0/3434922119 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c00cd50 con 0x7f33180b95a0 2024-03-06T07:32:57.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.512+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331815f7e0 con 0x7f33180b95a0 2024-03-06T07:32:57.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.512+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331804f420 con 0x7f33180b95a0 2024-03-06T07:32:57.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.513+0000 7f33157fa640 1 -- 172.21.15.27:0/3434922119 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f330c01a070 con 0x7f33180b95a0 2024-03-06T07:32:57.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.514+0000 7f33157fa640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32fc06b4d0 0x7f32fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.514+0000 7f33157fa640 1 -- 172.21.15.27:0/3434922119 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f330c08a470 con 0x7f33180b95a0 2024-03-06T07:32:57.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.514+0000 7f3317fff640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32fc06b4d0 0x7f32fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.515+0000 7f3317fff640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32fc06b4d0 0x7f32fc06d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f33180cc850 tx=0x7f3304007c10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.518+0000 7f33157fa640 1 -- 172.21.15.27:0/3434922119 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f331804f420 con 0x7f33180b95a0 2024-03-06T07:32:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.737+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f331815b720 con 0x7f33180b95a0 2024-03-06T07:32:57.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.737+0000 7f33157fa640 1 -- 172.21.15.27:0/3434922119 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f330c0592d0 con 0x7f33180b95a0 2024-03-06T07:32:57.740 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:57.740 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:57.740 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32fc06b4d0 msgr2=0x7f32fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32fc06b4d0 0x7f32fc06d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f33180cc850 tx=0x7f3304007c10 comp rx=0 tx=0).stop 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33180b95a0 msgr2=0x7f3318158380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33180b95a0 0x7f3318158380 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f330c00e9b0 tx=0x7f330c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 shutdown_connections 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f32fc06b4d0 0x7f32fc06d990 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33180cd900 0x7f33180b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f331815df00 0x7f33180cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 --2- 172.21.15.27:0/3434922119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33180b95a0 0x7f3318158380 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 >> 172.21.15.27:0/3434922119 conn(0x7f3318071960 msgr2=0x7f33180b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 shutdown_connections 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:57.739+0000 7f331f05c640 1 -- 172.21.15.27:0/3434922119 wait complete. 2024-03-06T07:32:57.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:57.954 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:32:58.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:59.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.060+0000 7f25b77ac640 1 Processor -- start 2024-03-06T07:32:59.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.060+0000 7f25b77ac640 1 -- start start 2024-03-06T07:32:59.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b77ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 0x7f25b00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b77ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25b00b2b50 0x7f25b00b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b77ac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b00ce610 con 0x7f25b0071540 2024-03-06T07:32:59.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b77ac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b00d2f70 con 0x7f25b00b2b50 2024-03-06T07:32:59.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b77ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00d4be0 con 0x7f25b00d3400 2024-03-06T07:32:59.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b5d49640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25b00b2b50 0x7f25b00b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b5548640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f25b0071540 0x7f25b0071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47382/0 (socket says 172.21.15.27:47382) 2024-03-06T07:32:59.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b5548640 1 -- 172.21.15.27:0/1052456210 learned_addr learned my addr 172.21.15.27:0/1052456210 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:32:59.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b5d49640 1 -- 172.21.15.27:0/1052456210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 msgr2=0x7f25b00b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b4d47640 1 --2- 172.21.15.27:0/1052456210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 0x7f25b00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b5d49640 1 --2- 172.21.15.27:0/1052456210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 0x7f25b00b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25b5d49640 1 -- 172.21.15.27:0/1052456210 >> v1:172.21.15.27:6789/0 conn(0x7f25b0071540 legacy=0x7f25b0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.062+0000 7f25b5d49640 1 -- 172.21.15.27:0/1052456210 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00d2920 con 0x7f25b00b2b50 2024-03-06T07:32:59.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.061+0000 7f25a7fff640 1 -- 172.21.15.27:0/1052456210 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706350120 0 0) 0x7f25b00ce610 con 0x7f25b0071540 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.062+0000 7f25b5d49640 1 --2- 172.21.15.27:0/1052456210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 0x7f25b00b6e20 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009d90 tx=0x7f25ac02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e17a7bd4dd518a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.062+0000 7f25a7fff640 1 -- 172.21.15.27:0/1052456210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac039070 con 0x7f25b00b2b50 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.062+0000 7f25a7fff640 1 -- 172.21.15.27:0/1052456210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac02cc50 con 0x7f25b00b2b50 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.062+0000 7f25a7fff640 1 -- 172.21.15.27:0/1052456210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac034440 con 0x7f25b00b2b50 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.063+0000 7f25b77ac640 1 -- 172.21.15.27:0/1052456210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 msgr2=0x7f25b00b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.063+0000 7f25b77ac640 1 --2- 172.21.15.27:0/1052456210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 0x7f25b00b6e20 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009d90 tx=0x7f25ac02bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.063+0000 7f25b77ac640 1 -- 172.21.15.27:0/1052456210 shutdown_connections 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.063+0000 7f25b77ac640 1 --2- 172.21.15.27:0/1052456210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 0x7f25b00b6e20 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.063+0000 7f25b77ac640 1 --2- 172.21.15.27:0/1052456210 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 0x7f25b00b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.063+0000 7f25b77ac640 1 -- 172.21.15.27:0/1052456210 >> 172.21.15.27:0/1052456210 conn(0x7f25b00d3ca0 msgr2=0x7f25b00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.063+0000 7f25b77ac640 1 -- 172.21.15.27:0/1052456210 shutdown_connections 2024-03-06T07:32:59.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.063+0000 7f25b77ac640 1 -- 172.21.15.27:0/1052456210 wait complete. 2024-03-06T07:32:59.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.064+0000 7f25b77ac640 1 Processor -- start 2024-03-06T07:32:59.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.064+0000 7f25b77ac640 1 -- start start 2024-03-06T07:32:59.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b77ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b0071540 0x7f25b00b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b77ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 0x7f25b014d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b77ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 0x7f25b00b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b77ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b015e370 con 0x7f25b00b2b50 2024-03-06T07:32:59.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b77ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00ce8f0 con 0x7f25b0071540 2024-03-06T07:32:59.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b77ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b014d660 con 0x7f25b00d3400 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b4d47640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 0x7f25b014d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b4d47640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 0x7f25b014d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42310/0 (socket says 172.21.15.27:42310) 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b4d47640 1 -- 172.21.15.27:0/1831300297 learned_addr learned my addr 172.21.15.27:0/1831300297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b5d49640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 0x7f25b00b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.065+0000 7f25b5548640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b0071540 0x7f25b00b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25b4d47640 1 -- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 msgr2=0x7f25b00b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25b4d47640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 0x7f25b00b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25b4d47640 1 -- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b0071540 msgr2=0x7f25b00b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25b4d47640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b0071540 0x7f25b00b4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25b4d47640 1 -- 172.21.15.27:0/1831300297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00b3ec0 con 0x7f25b00b2b50 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25b4d47640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 0x7f25b014d120 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f25a000c7f0 tx=0x7f25a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25a5ffb640 1 -- 172.21.15.27:0/1831300297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0004230 con 0x7f25b00b2b50 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25a5ffb640 1 -- 172.21.15.27:0/1831300297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a00043d0 con 0x7f25b00b2b50 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b00b41a0 con 0x7f25b00b2b50 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25a5ffb640 1 -- 172.21.15.27:0/1831300297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0007860 con 0x7f25b00b2b50 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.066+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b00b4760 con 0x7f25b00b2b50 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.067+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b0068700 con 0x7f25b00b2b50 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.068+0000 7f25a5ffb640 1 -- 172.21.15.27:0/1831300297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25a0007c10 con 0x7f25b00b2b50 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.068+0000 7f25a5ffb640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f259406b4d0 0x7f259406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.068+0000 7f25a5ffb640 1 -- 172.21.15.27:0/1831300297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f25a008a320 con 0x7f25b00b2b50 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.068+0000 7f25b5548640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f259406b4d0 0x7f259406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.069+0000 7f25b5548640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f259406b4d0 0x7f259406d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f25a8000f60 tx=0x7f25a8007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.073+0000 7f25a5ffb640 1 -- 172.21.15.27:0/1831300297 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25a0015020 con 0x7f25b00b2b50 2024-03-06T07:32:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.410+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25b00b7ed0 con 0x7f25b00b2b50 2024-03-06T07:32:59.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.411+0000 7f25a5ffb640 1 -- 172.21.15.27:0/1831300297 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f25a0058b30 con 0x7f25b00b2b50 2024-03-06T07:32:59.414 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:32:59.414 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:32:59.414 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f259406b4d0 msgr2=0x7f259406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f259406b4d0 0x7f259406d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f25a8000f60 tx=0x7f25a8007d50 comp rx=0 tx=0).stop 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 msgr2=0x7f25b014d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 0x7f25b014d120 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f25a000c7f0 tx=0x7f25a000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 shutdown_connections 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f259406b4d0 0x7f259406d990 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25b00d3400 0x7f25b00b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00b2b50 0x7f25b014d120 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 --2- 172.21.15.27:0/1831300297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b0071540 0x7f25b00b4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 >> 172.21.15.27:0/1831300297 conn(0x7f25b00d3ca0 msgr2=0x7f25b015e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 shutdown_connections 2024-03-06T07:32:59.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:32:59.413+0000 7f25b77ac640 1 -- 172.21.15.27:0/1831300297 wait complete. 2024-03-06T07:32:59.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:32:59.627 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:00.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:00.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.745+0000 7fc42b2a3640 1 Processor -- start 2024-03-06T07:33:00.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.746+0000 7fc42b2a3640 1 -- start start 2024-03-06T07:33:00.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.746+0000 7fc42b2a3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4240b9a10 0x7fc4240d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.746+0000 7fc42b2a3640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4240cea30 con 0x7fc4240b95a0 2024-03-06T07:33:00.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.746+0000 7fc42b2a3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4240cf3c0 con 0x7fc42415df00 2024-03-06T07:33:00.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.746+0000 7fc42b2a3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240cf0e0 con 0x7fc4240b9a10 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc42883e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc4240b95a0 0x7fc4240b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47392/0 (socket says 172.21.15.27:47392) 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc42883e640 1 -- 172.21.15.27:0/753061923 learned_addr learned my addr 172.21.15.27:0/753061923 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc429840640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4240b9a10 0x7fc4240d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513186219 0 0) 0x7fc4240cea30 con 0x7fc4240b95a0 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc40c003520 con 0x7fc4240b95a0 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922374451 0 0) 0x7fc4240cf3c0 con 0x7fc42415df00 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4240cea30 con 0x7fc42415df00 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 115273243 0 0) 0x7fc40c003520 con 0x7fc4240b95a0 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4240b9a10 msgr2=0x7fc4240d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4240b9a10 0x7fc4240d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 >> v1:172.21.15.69:6789/0 conn(0x7fc42415df00 legacy=0x7fc4240d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4240ce470 con 0x7fc4240b95a0 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fc418001670 con 0x7fc4240b95a0 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 >> v1:172.21.15.27:6789/0 conn(0x7fc4240b95a0 legacy=0x7fc4240b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc40c003b70 0x7fc40c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc40c004800 0x7fc40c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4240b9a10 0x7fc40c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240cf0e0 con 0x7fc40c003b70 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c00ba70 con 0x7fc40c004800 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c00bc40 con 0x7fc4240b9a10 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.747+0000 7fc42903f640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc40c003b70 0x7fc40c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.748+0000 7fc429840640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4240b9a10 0x7fc40c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.748+0000 7fc42883e640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc40c004800 0x7fc40c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.748+0000 7fc42903f640 1 -- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4240b9a10 msgr2=0x7fc40c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.748+0000 7fc42903f640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4240b9a10 0x7fc40c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.748+0000 7fc42903f640 1 -- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc40c004800 msgr2=0x7fc40c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:00.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.748+0000 7fc42903f640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc40c004800 0x7fc40c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.748+0000 7fc42903f640 1 -- 172.21.15.27:0/753061923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c003520 con 0x7fc40c003b70 2024-03-06T07:33:00.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.749+0000 7fc42903f640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc40c003b70 0x7fc40c003f70 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0018d0 tx=0x7fc40c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d291194f1dfba3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.749+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c0014f0 con 0x7fc40c003b70 2024-03-06T07:33:00.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.749+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c02f050 con 0x7fc40c003b70 2024-03-06T07:33:00.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.749+0000 7fc423fff640 1 -- 172.21.15.27:0/753061923 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c0035f0 con 0x7fc40c003b70 2024-03-06T07:33:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.750+0000 7fc42b2a3640 1 -- 172.21.15.27:0/753061923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc40c003b70 msgr2=0x7fc40c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.750+0000 7fc42b2a3640 1 --2- 172.21.15.27:0/753061923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc40c003b70 0x7fc40c003f70 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0018d0 tx=0x7fc40c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.750+0000 7fc429840640 1 -- 172.21.15.27:0/753061923 reap_dead start 2024-03-06T07:33:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.750+0000 7fc42b2a3640 1 -- 172.21.15.27:0/753061923 shutdown_connections 2024-03-06T07:33:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.750+0000 7fc42b2a3640 1 -- 172.21.15.27:0/753061923 >> 172.21.15.27:0/753061923 conn(0x7fc424071960 msgr2=0x7fc424071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.750+0000 7fc42b2a3640 1 -- 172.21.15.27:0/753061923 shutdown_connections 2024-03-06T07:33:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.750+0000 7fc42b2a3640 1 -- 172.21.15.27:0/753061923 wait complete. 2024-03-06T07:33:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.752+0000 7fc42b2a3640 1 Processor -- start 2024-03-06T07:33:00.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.752+0000 7fc42b2a3640 1 -- start start 2024-03-06T07:33:00.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.753+0000 7fc42b2a3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc40c003b70 0x7fc424160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.753+0000 7fc42b2a3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4240cad50 0x7fc4240b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.753+0000 7fc42b2a3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4240cb1d0 0x7fc4240b8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.753+0000 7fc42b2a3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240cedf0 con 0x7fc4240cad50 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42b2a3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240cec70 con 0x7fc4240cb1d0 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42b2a3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240b5e10 con 0x7fc40c003b70 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42883e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4240cad50 0x7fc4240b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42883e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4240cad50 0x7fc4240b7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42328/0 (socket says 172.21.15.27:42328) 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42883e640 1 -- 172.21.15.27:0/3113884930 learned_addr learned my addr 172.21.15.27:0/3113884930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc429840640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4240cb1d0 0x7fc4240b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42903f640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc40c003b70 0x7fc424160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42883e640 1 -- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc40c003b70 msgr2=0x7fc424160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42883e640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc40c003b70 0x7fc424160bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42883e640 1 -- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4240cb1d0 msgr2=0x7fc4240b8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42883e640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4240cb1d0 0x7fc4240b8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42883e640 1 -- 172.21.15.27:0/3113884930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4240cb7a0 con 0x7fc4240cad50 2024-03-06T07:33:00.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc42883e640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4240cad50 0x7fc4240b7ef0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fc418004f90 tx=0x7fc4180074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc421ffb640 1 -- 172.21.15.27:0/3113884930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418007740 con 0x7fc4240cad50 2024-03-06T07:33:00.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.754+0000 7fc421ffb640 1 -- 172.21.15.27:0/3113884930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418018070 con 0x7fc4240cad50 2024-03-06T07:33:00.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.755+0000 7fc421ffb640 1 -- 172.21.15.27:0/3113884930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41800fbc0 con 0x7fc4240cad50 2024-03-06T07:33:00.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.755+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4240cba80 con 0x7fc4240cad50 2024-03-06T07:33:00.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.755+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4240cc040 con 0x7fc4240cad50 2024-03-06T07:33:00.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.756+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc424067ff0 con 0x7fc4240cad50 2024-03-06T07:33:00.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.756+0000 7fc421ffb640 1 -- 172.21.15.27:0/3113884930 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc41801d460 con 0x7fc4240cad50 2024-03-06T07:33:00.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.757+0000 7fc421ffb640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc41006b670 0x7fc41006db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.757+0000 7fc421ffb640 1 -- 172.21.15.27:0/3113884930 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc41808a140 con 0x7fc4240cad50 2024-03-06T07:33:00.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.757+0000 7fc42903f640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc41006b670 0x7fc41006db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.758+0000 7fc42903f640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc41006b670 0x7fc41006db30 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc41c034000 tx=0x7fc41c031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.761+0000 7fc421ffb640 1 -- 172.21.15.27:0/3113884930 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc418058950 con 0x7fc4240cad50 2024-03-06T07:33:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.981+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc42406b780 con 0x7fc4240cad50 2024-03-06T07:33:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.981+0000 7fc421ffb640 1 -- 172.21.15.27:0/3113884930 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7fc41805c9f0 con 0x7fc4240cad50 2024-03-06T07:33:00.984 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:00.984 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc41006b670 msgr2=0x7fc41006db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc41006b670 0x7fc41006db30 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc41c034000 tx=0x7fc41c031000 comp rx=0 tx=0).stop 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4240cad50 msgr2=0x7fc4240b7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4240cad50 0x7fc4240b7ef0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fc418004f90 tx=0x7fc4180074f0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 shutdown_connections 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc41006b670 0x7fc41006db30 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4240cb1d0 0x7fc4240b8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4240cad50 0x7fc4240b7ef0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 --2- 172.21.15.27:0/3113884930 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc40c003b70 0x7fc424160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 >> 172.21.15.27:0/3113884930 conn(0x7fc424071960 msgr2=0x7fc4240b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 shutdown_connections 2024-03-06T07:33:00.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:00.983+0000 7fc42b2a3640 1 -- 172.21.15.27:0/3113884930 wait complete. 2024-03-06T07:33:01.198 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:01.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:02.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.304+0000 7fe9e8781640 1 Processor -- start 2024-03-06T07:33:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.304+0000 7fe9e8781640 1 -- start start 2024-03-06T07:33:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e8781640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9e015df00 0x7fe9e00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e8781640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e00cea30 con 0x7fe9e00b95a0 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e8781640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e00d2af0 con 0x7fe9e00b9a30 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e8781640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e00cf3c0 con 0x7fe9e015df00 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e651d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9e015df00 0x7fe9e00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e651d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9e015df00 0x7fe9e00d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42336/0 (socket says 172.21.15.27:42336) 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e5d1c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe9e00b95a0 0x7fe9e00b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47394/0 (socket says 172.21.15.27:47394) 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e651d640 1 -- 172.21.15.27:0/4070830664 learned_addr learned my addr 172.21.15.27:0/4070830664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e651d640 1 -- 172.21.15.27:0/4070830664 >> v1:172.21.15.69:6789/0 conn(0x7fe9e00b9a30 legacy=0x7fe9e00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e651d640 1 -- 172.21.15.27:0/4070830664 >> v1:172.21.15.27:6789/0 conn(0x7fe9e00b95a0 legacy=0x7fe9e00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e551b640 1 -- 172.21.15.27:0/4070830664 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252601493 0 0) 0x7fe9e00cea30 con 0x7fe9e00b95a0 2024-03-06T07:33:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.305+0000 7fe9e651d640 1 -- 172.21.15.27:0/4070830664 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e00cf0e0 con 0x7fe9e015df00 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.306+0000 7fe9e651d640 1 --2- 172.21.15.27:0/4070830664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e015df00 0x7fe9e00d25b0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009870 tx=0x7fe9dc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ddc3c289473219b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.306+0000 7fe9e551b640 1 -- 172.21.15.27:0/4070830664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc034070 con 0x7fe9e015df00 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.306+0000 7fe9e551b640 1 -- 172.21.15.27:0/4070830664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc030860 con 0x7fe9e015df00 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.306+0000 7fe9e551b640 1 -- 172.21.15.27:0/4070830664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc030c30 con 0x7fe9e015df00 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.307+0000 7fe9e8781640 1 -- 172.21.15.27:0/4070830664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e015df00 msgr2=0x7fe9e00d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.307+0000 7fe9e8781640 1 --2- 172.21.15.27:0/4070830664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e015df00 0x7fe9e00d25b0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009870 tx=0x7fe9dc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.307+0000 7fe9e8781640 1 -- 172.21.15.27:0/4070830664 shutdown_connections 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.307+0000 7fe9e8781640 1 --2- 172.21.15.27:0/4070830664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e015df00 0x7fe9e00d25b0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.307+0000 7fe9e8781640 1 -- 172.21.15.27:0/4070830664 >> 172.21.15.27:0/4070830664 conn(0x7fe9e0071960 msgr2=0x7fe9e0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.307+0000 7fe9e8781640 1 -- 172.21.15.27:0/4070830664 shutdown_connections 2024-03-06T07:33:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.307+0000 7fe9e8781640 1 -- 172.21.15.27:0/4070830664 wait complete. 2024-03-06T07:33:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.308+0000 7fe9e8781640 1 Processor -- start 2024-03-06T07:33:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.308+0000 7fe9e8781640 1 -- start start 2024-03-06T07:33:02.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.309+0000 7fe9e8781640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9e00b95a0 0x7fe9e0155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.309+0000 7fe9e8781640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9e015df00 0x7fe9e00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.309+0000 7fe9e8781640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e00b53f0 0x7fe9e015e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.309+0000 7fe9e8781640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00cec70 con 0x7fe9e00b53f0 2024-03-06T07:33:02.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.309+0000 7fe9e8781640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00ce750 con 0x7fe9e015df00 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.309+0000 7fe9e8781640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00cedf0 con 0x7fe9e00b95a0 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e6d1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e00b53f0 0x7fe9e015e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e6d1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e00b53f0 0x7fe9e015e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42344/0 (socket says 172.21.15.27:42344) 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e6d1e640 1 -- 172.21.15.27:0/1804998466 learned_addr learned my addr 172.21.15.27:0/1804998466 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e5d1c640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9e015df00 0x7fe9e00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e651d640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9e00b95a0 0x7fe9e0155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e6d1e640 1 -- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9e00b95a0 msgr2=0x7fe9e0155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e6d1e640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9e00b95a0 0x7fe9e0155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e6d1e640 1 -- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9e015df00 msgr2=0x7fe9e00b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e6d1e640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9e015df00 0x7fe9e00b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e6d1e640 1 -- 172.21.15.27:0/1804998466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e00c92c0 con 0x7fe9e00b53f0 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e6d1e640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e00b53f0 0x7fe9e015e2e0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800c780 tx=0x7fe9d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9d6ffd640 1 -- 172.21.15.27:0/1804998466 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8004b60 con 0x7fe9e00b53f0 2024-03-06T07:33:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9d6ffd640 1 -- 172.21.15.27:0/1804998466 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d8004d00 con 0x7fe9e00b53f0 2024-03-06T07:33:02.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9d6ffd640 1 -- 172.21.15.27:0/1804998466 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d800aa90 con 0x7fe9e00b53f0 2024-03-06T07:33:02.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e00c95a0 con 0x7fe9e00b53f0 2024-03-06T07:33:02.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.310+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e00c9b60 con 0x7fe9e00b53f0 2024-03-06T07:33:02.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.311+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e0067ff0 con 0x7fe9e00b53f0 2024-03-06T07:33:02.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.312+0000 7fe9d6ffd640 1 -- 172.21.15.27:0/1804998466 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9d800ac30 con 0x7fe9e00b53f0 2024-03-06T07:33:02.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.312+0000 7fe9d6ffd640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9c406b4d0 0x7fe9c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.312+0000 7fe9d6ffd640 1 -- 172.21.15.27:0/1804998466 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe9d808a400 con 0x7fe9e00b53f0 2024-03-06T07:33:02.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.312+0000 7fe9e651d640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9c406b4d0 0x7fe9c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.313+0000 7fe9e651d640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9c406b4d0 0x7fe9c406d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc036000 tx=0x7fe9dc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.317+0000 7fe9d6ffd640 1 -- 172.21.15.27:0/1804998466 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9d8020070 con 0x7fe9e00b53f0 2024-03-06T07:33:02.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.539+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9e0155890 con 0x7fe9e00b53f0 2024-03-06T07:33:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.539+0000 7fe9d6ffd640 1 -- 172.21.15.27:0/1804998466 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7fe9d8059260 con 0x7fe9e00b53f0 2024-03-06T07:33:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:02.542 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:33:02.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.541+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9c406b4d0 msgr2=0x7fe9c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.541+0000 7fe9e8781640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9c406b4d0 0x7fe9c406d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc036000 tx=0x7fe9dc03b040 comp rx=0 tx=0).stop 2024-03-06T07:33:02.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.541+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e00b53f0 msgr2=0x7fe9e015e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.541+0000 7fe9e8781640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e00b53f0 0x7fe9e015e2e0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800c780 tx=0x7fe9d800cc40 comp rx=0 tx=0).stop 2024-03-06T07:33:02.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.542+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 shutdown_connections 2024-03-06T07:33:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.542+0000 7fe9e8781640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9c406b4d0 0x7fe9c406d990 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.542+0000 7fe9e8781640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9e00b53f0 0x7fe9e015e2e0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.542+0000 7fe9e8781640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9e015df00 0x7fe9e00b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.542+0000 7fe9e8781640 1 --2- 172.21.15.27:0/1804998466 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9e00b95a0 0x7fe9e0155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.542+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 >> 172.21.15.27:0/1804998466 conn(0x7fe9e0071960 msgr2=0x7fe9e00d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.542+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 shutdown_connections 2024-03-06T07:33:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:02.542+0000 7fe9e8781640 1 -- 172.21.15.27:0/1804998466 wait complete. 2024-03-06T07:33:02.756 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:02.756 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:03.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.863+0000 7f2e0e639640 1 Processor -- start 2024-03-06T07:33:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.863+0000 7f2e0e639640 1 -- start start 2024-03-06T07:33:03.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.864+0000 7f2e0e639640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080b2b50 0x7f2e080b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.864+0000 7f2e0e639640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e080ce610 con 0x7f2e08071540 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.864+0000 7f2e0e639640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e080d2920 con 0x7f2e080d3400 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.864+0000 7f2e0e639640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080d2f70 con 0x7f2e080b2b50 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.864+0000 7f2e07fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2e08071540 0x7f2e08071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51094/0 (socket says 172.21.15.27:51094) 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.864+0000 7f2e07fff640 1 -- 172.21.15.27:0/3573219290 learned_addr learned my addr 172.21.15.27:0/3573219290 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.864+0000 7f2e0cbd6640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080b2b50 0x7f2e080b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114794687 0 0) 0x7f2e080ce610 con 0x7f2e08071540 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df0003520 con 0x7f2e08071540 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413584515 0 0) 0x7f2e080d2920 con 0x7f2e080d3400 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e080ce610 con 0x7f2e080d3400 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1212796032 0 0) 0x7f2df0003520 con 0x7f2e08071540 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080b2b50 msgr2=0x7f2e080b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080b2b50 0x7f2e080b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 >> v1:172.21.15.69:6789/0 conn(0x7f2e080d3400 legacy=0x7f2e080b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e080d2c00 con 0x7f2e08071540 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f2df8001670 con 0x7f2e08071540 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e0cbd6640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080b2b50 0x7f2e080b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 >> v1:172.21.15.27:6789/0 conn(0x7f2e08071540 legacy=0x7f2e08071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2df0003b70 0x7f2df0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080b2b50 0x7f2df0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2df0004d40 0x7f2df00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080d2f70 con 0x7f2df0004d40 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df000ba70 con 0x7f2df0003b70 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df000bc20 con 0x7f2e080b2b50 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e0cbd6640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2df0004d40 0x7f2df00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e07fff640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2df0003b70 0x7f2df0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e077fe640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080b2b50 0x7f2df0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e0cbd6640 1 -- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080b2b50 msgr2=0x7f2df0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e0cbd6640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080b2b50 0x7f2df0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e0cbd6640 1 -- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2df0003b70 msgr2=0x7f2df0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e0cbd6640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2df0003b70 0x7f2df0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.865+0000 7f2e0cbd6640 1 -- 172.21.15.27:0/3573219290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df0003520 con 0x7f2df0004d40 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.866+0000 7f2e0cbd6640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2df0004d40 0x7f2df00051c0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f2e00016b60 tx=0x7f2df0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd537a4ff1bc211c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.866+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00009de0 con 0x7f2df0004d40 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.866+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e00040070 con 0x7f2df0004d40 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.866+0000 7f2e06ffd640 1 -- 172.21.15.27:0/3573219290 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0003b860 con 0x7f2df0004d40 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.867+0000 7f2e0e639640 1 -- 172.21.15.27:0/3573219290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2df0004d40 msgr2=0x7f2df00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.867+0000 7f2e0e639640 1 --2- 172.21.15.27:0/3573219290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2df0004d40 0x7f2df00051c0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f2e00016b60 tx=0x7f2df0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.867+0000 7f2e0cbd6640 1 -- 172.21.15.27:0/3573219290 reap_dead start 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.867+0000 7f2e0e639640 1 -- 172.21.15.27:0/3573219290 shutdown_connections 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.867+0000 7f2e0e639640 1 -- 172.21.15.27:0/3573219290 >> 172.21.15.27:0/3573219290 conn(0x7f2e080d3ca0 msgr2=0x7f2e080d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.867+0000 7f2e0e639640 1 -- 172.21.15.27:0/3573219290 shutdown_connections 2024-03-06T07:33:03.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.867+0000 7f2e0e639640 1 -- 172.21.15.27:0/3573219290 wait complete. 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.869+0000 7f2e0e639640 1 Processor -- start 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.869+0000 7f2e0e639640 1 -- start start 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.870+0000 7f2e0e639640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e080c9980 0x7f2e080cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.870+0000 7f2e0e639640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080c9d60 0x7f2e080cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.870+0000 7f2e0e639640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e080ccb30 0x7f2e08158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.870+0000 7f2e0e639640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0815e370 con 0x7f2e080c9980 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.870+0000 7f2e0e639640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080d4be0 con 0x7f2e080ccb30 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.870+0000 7f2e0e639640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080cc160 con 0x7f2e080c9d60 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e07fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e080c9980 0x7f2e080cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e07fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e080c9980 0x7f2e080cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56496/0 (socket says 172.21.15.27:56496) 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e07fff640 1 -- 172.21.15.27:0/3023583733 learned_addr learned my addr 172.21.15.27:0/3023583733 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e077fe640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080c9d60 0x7f2e080cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e07fff640 1 -- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080c9d60 msgr2=0x7f2e080cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e0cbd6640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e080ccb30 0x7f2e08158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e07fff640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080c9d60 0x7f2e080cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e07fff640 1 -- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e080ccb30 msgr2=0x7f2e08158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e07fff640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e080ccb30 0x7f2e08158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e07fff640 1 -- 172.21.15.27:0/3023583733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e080cb2b0 con 0x7f2e080c9980 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e0cbd6640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e080ccb30 0x7f2e08158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e077fe640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080c9d60 0x7f2e080cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e07fff640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e080c9980 0x7f2e080cad70 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f2df80011e0 tx=0x7f2df8007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e04ff9640 1 -- 172.21.15.27:0/3023583733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df800f940 con 0x7f2e080c9980 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e04ff9640 1 -- 172.21.15.27:0/3023583733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df8018070 con 0x7f2e080c9980 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e080cb550 con 0x7f2e080c9980 2024-03-06T07:33:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e04ff9640 1 -- 172.21.15.27:0/3023583733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df800fd40 con 0x7f2e080c9980 2024-03-06T07:33:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.871+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e080cbb10 con 0x7f2e080c9980 2024-03-06T07:33:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.873+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e08068700 con 0x7f2e080c9980 2024-03-06T07:33:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.873+0000 7f2e04ff9640 1 -- 172.21.15.27:0/3023583733 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2df801d460 con 0x7f2e080c9980 2024-03-06T07:33:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.873+0000 7f2e04ff9640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2df406b4d0 0x7f2df406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.873+0000 7f2e04ff9640 1 -- 172.21.15.27:0/3023583733 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2df808a460 con 0x7f2e080c9980 2024-03-06T07:33:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.874+0000 7f2e077fe640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2df406b4d0 0x7f2df406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.874+0000 7f2e077fe640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2df406b4d0 0x7f2df406d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2e081f3000 tx=0x7f2dfc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:03.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:03.878+0000 7f2e04ff9640 1 -- 172.21.15.27:0/3023583733 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2df8059230 con 0x7f2e080c9980 2024-03-06T07:33:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.265+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e0815f530 con 0x7f2e080c9980 2024-03-06T07:33:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.265+0000 7f2e04ff9640 1 -- 172.21.15.27:0/3023583733 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f2df8058b30 con 0x7f2e080c9980 2024-03-06T07:33:04.268 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:04.268 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:33:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2df406b4d0 msgr2=0x7f2df406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2df406b4d0 0x7f2df406d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2e081f3000 tx=0x7f2dfc009040 comp rx=0 tx=0).stop 2024-03-06T07:33:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e080c9980 msgr2=0x7f2e080cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e080c9980 0x7f2e080cad70 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f2df80011e0 tx=0x7f2df8007720 comp rx=0 tx=0).stop 2024-03-06T07:33:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 shutdown_connections 2024-03-06T07:33:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2df406b4d0 0x7f2df406d990 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e080ccb30 0x7f2e08158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2e080c9d60 0x7f2e080cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 --2- 172.21.15.27:0/3023583733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e080c9980 0x7f2e080cad70 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 >> 172.21.15.27:0/3023583733 conn(0x7f2e080d3ca0 msgr2=0x7f2e08160ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 shutdown_connections 2024-03-06T07:33:04.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:04.268+0000 7f2e0e639640 1 -- 172.21.15.27:0/3023583733 wait complete. 2024-03-06T07:33:04.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:04.485 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:05.486 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.592+0000 7f5721efb640 1 Processor -- start 2024-03-06T07:33:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.592+0000 7f5721efb640 1 -- start start 2024-03-06T07:33:05.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f5721efb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f571c15df00 0x7f571c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f5721efb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c0b9a30 0x7f571c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f5721efb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c0cea30 con 0x7f571c0b95a0 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f5721efb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c0d2af0 con 0x7f571c15df00 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f5721efb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0cf3c0 con 0x7f571c0b9a30 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f571c15df00 0x7f571c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f571c0b95a0 0x7f571c0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51100/0 (socket says 172.21.15.27:51100) 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571b7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f571c15df00 0x7f571c0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56500/0 (socket says 172.21.15.27:56500) 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571affd640 1 -- 172.21.15.27:0/478307736 learned_addr learned my addr 172.21.15.27:0/478307736 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571bfff640 1 --2- 172.21.15.27:0/478307736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c0b9a30 0x7f571c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571b7fe640 1 -- 172.21.15.27:0/478307736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c0b9a30 msgr2=0x7f571c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571b7fe640 1 --2- 172.21.15.27:0/478307736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c0b9a30 0x7f571c0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571b7fe640 1 -- 172.21.15.27:0/478307736 >> v1:172.21.15.27:6789/0 conn(0x7f571c0b95a0 legacy=0x7f571c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571b7fe640 1 -- 172.21.15.27:0/478307736 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c0cf0e0 con 0x7f571c15df00 2024-03-06T07:33:05.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571a7fc640 1 -- 172.21.15.27:0/478307736 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242938068 0 0) 0x7f571c0cea30 con 0x7f571c0b95a0 2024-03-06T07:33:05.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.593+0000 7f571bfff640 1 --2- 172.21.15.27:0/478307736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c0b9a30 0x7f571c0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:05.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.594+0000 7f571b7fe640 1 --2- 172.21.15.27:0/478307736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c15df00 0x7f571c0d25b0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f570c0097a0 tx=0x7f570c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72a2bfe66b7b252 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.594+0000 7f571a7fc640 1 -- 172.21.15.27:0/478307736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c035020 con 0x7f571c15df00 2024-03-06T07:33:05.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.594+0000 7f571a7fc640 1 -- 172.21.15.27:0/478307736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c031830 con 0x7f571c15df00 2024-03-06T07:33:05.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.594+0000 7f571a7fc640 1 -- 172.21.15.27:0/478307736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c031c00 con 0x7f571c15df00 2024-03-06T07:33:05.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.595+0000 7f5721efb640 1 -- 172.21.15.27:0/478307736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c15df00 msgr2=0x7f571c0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.595+0000 7f5721efb640 1 --2- 172.21.15.27:0/478307736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c15df00 0x7f571c0d25b0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f570c0097a0 tx=0x7f570c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.595+0000 7f5721efb640 1 -- 172.21.15.27:0/478307736 shutdown_connections 2024-03-06T07:33:05.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.595+0000 7f5721efb640 1 --2- 172.21.15.27:0/478307736 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c0b9a30 0x7f571c0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.595+0000 7f5721efb640 1 --2- 172.21.15.27:0/478307736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c15df00 0x7f571c0d25b0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.595+0000 7f5721efb640 1 -- 172.21.15.27:0/478307736 >> 172.21.15.27:0/478307736 conn(0x7f571c071960 msgr2=0x7f571c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.595+0000 7f5721efb640 1 -- 172.21.15.27:0/478307736 shutdown_connections 2024-03-06T07:33:05.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.595+0000 7f5721efb640 1 -- 172.21.15.27:0/478307736 wait complete. 2024-03-06T07:33:05.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.596+0000 7f5721efb640 1 Processor -- start 2024-03-06T07:33:05.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.596+0000 7f5721efb640 1 -- start start 2024-03-06T07:33:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f5721efb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f571c0b95a0 0x7f571c0cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f5721efb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c15df00 0x7f571c0cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f5721efb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c1f3010 0x7f571c0b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f5721efb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0cec70 con 0x7f571c1f3010 2024-03-06T07:33:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f5721efb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0ce750 con 0x7f571c0b95a0 2024-03-06T07:33:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f5721efb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0cedf0 con 0x7f571c15df00 2024-03-06T07:33:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c1f3010 0x7f571c0b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c1f3010 0x7f571c0b54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56510/0 (socket says 172.21.15.27:56510) 2024-03-06T07:33:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571bfff640 1 -- 172.21.15.27:0/361226077 learned_addr learned my addr 172.21.15.27:0/361226077 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571affd640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c15df00 0x7f571c0cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571b7fe640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f571c0b95a0 0x7f571c0cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571bfff640 1 -- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c15df00 msgr2=0x7f571c0cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571bfff640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c15df00 0x7f571c0cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571bfff640 1 -- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f571c0b95a0 msgr2=0x7f571c0cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571bfff640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f571c0b95a0 0x7f571c0cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.597+0000 7f571bfff640 1 -- 172.21.15.27:0/361226077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c0cdb50 con 0x7f571c1f3010 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.598+0000 7f571bfff640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c1f3010 0x7f571c0b54f0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f571400cf20 tx=0x7f571400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.598+0000 7f5713fff640 1 -- 172.21.15.27:0/361226077 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571401a070 con 0x7f571c1f3010 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.598+0000 7f5713fff640 1 -- 172.21.15.27:0/361226077 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571400d840 con 0x7f571c1f3010 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.598+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c0cbc90 con 0x7f571c1f3010 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.598+0000 7f5713fff640 1 -- 172.21.15.27:0/361226077 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571400dc30 con 0x7f571c1f3010 2024-03-06T07:33:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.598+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c0c86b0 con 0x7f571c1f3010 2024-03-06T07:33:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.599+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c0ce000 con 0x7f571c1f3010 2024-03-06T07:33:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.599+0000 7f5713fff640 1 -- 172.21.15.27:0/361226077 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f571401d070 con 0x7f571c1f3010 2024-03-06T07:33:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.600+0000 7f5713fff640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56fc06b4d0 0x7f56fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.600+0000 7f5713fff640 1 -- 172.21.15.27:0/361226077 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f571408aa30 con 0x7f571c1f3010 2024-03-06T07:33:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.600+0000 7f571b7fe640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56fc06b4d0 0x7f56fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.601+0000 7f571b7fe640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56fc06b4d0 0x7f56fc06d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f570c036000 tx=0x7f570c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.609+0000 7f5713fff640 1 -- 172.21.15.27:0/361226077 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f571c0ce000 con 0x7f571c1f3010 2024-03-06T07:33:05.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.828+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f571c067ff0 con 0x7f571c1f3010 2024-03-06T07:33:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.828+0000 7f5713fff640 1 -- 172.21.15.27:0/361226077 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f5714059240 con 0x7f571c1f3010 2024-03-06T07:33:05.830 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:05.830 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:33:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.830+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56fc06b4d0 msgr2=0x7f56fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.830+0000 7f5721efb640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56fc06b4d0 0x7f56fc06d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f570c036000 tx=0x7f570c03c040 comp rx=0 tx=0).stop 2024-03-06T07:33:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c1f3010 msgr2=0x7f571c0b54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c1f3010 0x7f571c0b54f0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f571400cf20 tx=0x7f571400a7d0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 shutdown_connections 2024-03-06T07:33:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f56fc06b4d0 0x7f56fc06d990 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f571c1f3010 0x7f571c0b54f0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f571c15df00 0x7f571c0cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 --2- 172.21.15.27:0/361226077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f571c0b95a0 0x7f571c0cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 >> 172.21.15.27:0/361226077 conn(0x7f571c071960 msgr2=0x7f571c0b76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 shutdown_connections 2024-03-06T07:33:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:05.831+0000 7f5721efb640 1 -- 172.21.15.27:0/361226077 wait complete. 2024-03-06T07:33:06.044 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:06.044 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:07.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:07.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.152+0000 7f44d51f6640 1 Processor -- start 2024-03-06T07:33:07.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.152+0000 7f44d51f6640 1 -- start start 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44d51f6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f44d015df00 0x7f44d00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44d51f6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44d00b9a30 0x7f44d00d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44d51f6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d00cea30 con 0x7f44d00b95a0 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44d51f6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d00d2af0 con 0x7f44d00b9a30 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44d51f6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d00cf3c0 con 0x7f44d015df00 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44cf59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44d00b9a30 0x7f44d00d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44cf59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44d00b9a30 0x7f44d00d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56518/0 (socket says 172.21.15.27:56518) 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44cf59e640 1 -- 172.21.15.27:0/804834776 learned_addr learned my addr 172.21.15.27:0/804834776 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44ced9d640 1 --2- 172.21.15.27:0/804834776 >> v2:172.21.15.69:3300/0 conn(0x7f44d015df00 0x7f44d00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44cf59e640 1 -- 172.21.15.27:0/804834776 >> v2:172.21.15.69:3300/0 conn(0x7f44d015df00 msgr2=0x7f44d00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44cf59e640 1 --2- 172.21.15.27:0/804834776 >> v2:172.21.15.69:3300/0 conn(0x7f44d015df00 0x7f44d00d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44cf59e640 1 -- 172.21.15.27:0/804834776 >> v1:172.21.15.69:6789/0 conn(0x7f44d00b95a0 legacy=0x7f44d00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44cf59e640 1 -- 172.21.15.27:0/804834776 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d00cf0e0 con 0x7f44d00b9a30 2024-03-06T07:33:07.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.153+0000 7f44ced9d640 1 --2- 172.21.15.27:0/804834776 >> v2:172.21.15.69:3300/0 conn(0x7f44d015df00 0x7f44d00d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.154+0000 7f44cf59e640 1 --2- 172.21.15.27:0/804834776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d00b9a30 0x7f44d00d3220 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f44d00d11b0 tx=0x7f44c802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aac7056537afd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44cdd9b640 1 -- 172.21.15.27:0/804834776 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c802ca80 con 0x7f44d00b9a30 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44cdd9b640 1 -- 172.21.15.27:0/804834776 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c802cc20 con 0x7f44d00b9a30 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44cdd9b640 1 -- 172.21.15.27:0/804834776 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c80343f0 con 0x7f44d00b9a30 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44d51f6640 1 -- 172.21.15.27:0/804834776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d00b9a30 msgr2=0x7f44d00d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44d51f6640 1 --2- 172.21.15.27:0/804834776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d00b9a30 0x7f44d00d3220 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f44d00d11b0 tx=0x7f44c802bee0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44d51f6640 1 -- 172.21.15.27:0/804834776 shutdown_connections 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44d51f6640 1 --2- 172.21.15.27:0/804834776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d00b9a30 0x7f44d00d3220 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44d51f6640 1 --2- 172.21.15.27:0/804834776 >> v2:172.21.15.69:3300/0 conn(0x7f44d015df00 0x7f44d00d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44d51f6640 1 -- 172.21.15.27:0/804834776 >> 172.21.15.27:0/804834776 conn(0x7f44d0071960 msgr2=0x7f44d0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44d51f6640 1 -- 172.21.15.27:0/804834776 shutdown_connections 2024-03-06T07:33:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.155+0000 7f44d51f6640 1 -- 172.21.15.27:0/804834776 wait complete. 2024-03-06T07:33:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.157+0000 7f44d51f6640 1 Processor -- start 2024-03-06T07:33:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.157+0000 7f44d51f6640 1 -- start start 2024-03-06T07:33:07.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.157+0000 7f44d51f6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d00b95a0 0x7f44d0155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.157+0000 7f44d51f6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44d015df00 0x7f44d00cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.157+0000 7f44d51f6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d015e2e0 0x7f44d01560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.157+0000 7f44d51f6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00cec70 con 0x7f44d015e2e0 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.157+0000 7f44d51f6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00ce750 con 0x7f44d00b95a0 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.157+0000 7f44d51f6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00cedf0 con 0x7f44d015df00 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44cf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d015e2e0 0x7f44d01560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44cf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d015e2e0 0x7f44d01560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56530/0 (socket says 172.21.15.27:56530) 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44cf59e640 1 -- 172.21.15.27:0/1930947644 learned_addr learned my addr 172.21.15.27:0/1930947644 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44ce59c640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44d015df00 0x7f44d00cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44ced9d640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d00b95a0 0x7f44d0155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44cf59e640 1 -- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44d015df00 msgr2=0x7f44d00cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44cf59e640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44d015df00 0x7f44d00cb7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44cf59e640 1 -- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d00b95a0 msgr2=0x7f44d0155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44cf59e640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d00b95a0 0x7f44d0155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44cf59e640 1 -- 172.21.15.27:0/1930947644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d015e880 con 0x7f44d015e2e0 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44cf59e640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d015e2e0 0x7f44d01560c0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f44c802c4f0 tx=0x7f44c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44b77fe640 1 -- 172.21.15.27:0/1930947644 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c8043020 con 0x7f44d015e2e0 2024-03-06T07:33:07.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44b77fe640 1 -- 172.21.15.27:0/1930947644 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c803e450 con 0x7f44d015e2e0 2024-03-06T07:33:07.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d015eb60 con 0x7f44d015e2e0 2024-03-06T07:33:07.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44b77fe640 1 -- 172.21.15.27:0/1930947644 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c803e860 con 0x7f44d015e2e0 2024-03-06T07:33:07.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.158+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d015f120 con 0x7f44d015e2e0 2024-03-06T07:33:07.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.159+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0067ff0 con 0x7f44d015e2e0 2024-03-06T07:33:07.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.160+0000 7f44b77fe640 1 -- 172.21.15.27:0/1930947644 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44c8046020 con 0x7f44d015e2e0 2024-03-06T07:33:07.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.160+0000 7f44b77fe640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44bc06b4d0 0x7f44bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.160+0000 7f44b77fe640 1 -- 172.21.15.27:0/1930947644 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f44c80acb50 con 0x7f44d015e2e0 2024-03-06T07:33:07.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.160+0000 7f44ced9d640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44bc06b4d0 0x7f44bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.161+0000 7f44ced9d640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44bc06b4d0 0x7f44bc06d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f44c000a950 tx=0x7f44c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.170+0000 7f44b77fe640 1 -- 172.21.15.27:0/1930947644 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44c804d030 con 0x7f44d015e2e0 2024-03-06T07:33:07.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.431+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44d015c9d0 con 0x7f44d015e2e0 2024-03-06T07:33:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.431+0000 7f44b77fe640 1 -- 172.21.15.27:0/1930947644 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f44c807ba40 con 0x7f44d015e2e0 2024-03-06T07:33:07.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:07.433 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:33:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44bc06b4d0 msgr2=0x7f44bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44bc06b4d0 0x7f44bc06d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f44c000a950 tx=0x7f44c0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d015e2e0 msgr2=0x7f44d01560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d015e2e0 0x7f44d01560c0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f44c802c4f0 tx=0x7f44c8004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 shutdown_connections 2024-03-06T07:33:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44bc06b4d0 0x7f44bc06d990 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44d015e2e0 0x7f44d01560c0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44d015df00 0x7f44d00cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 --2- 172.21.15.27:0/1930947644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d00b95a0 0x7f44d0155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 >> 172.21.15.27:0/1930947644 conn(0x7f44d0071960 msgr2=0x7f44d00b7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 shutdown_connections 2024-03-06T07:33:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:07.434+0000 7f44d51f6640 1 -- 172.21.15.27:0/1930947644 wait complete. 2024-03-06T07:33:07.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:07.647 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:08.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.759+0000 7f5ef7f5c640 1 Processor -- start 2024-03-06T07:33:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.759+0000 7f5ef7f5c640 1 -- start start 2024-03-06T07:33:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.760+0000 7f5ef7f5c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ef00b2b50 0x7f5ef00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.760+0000 7f5ef7f5c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef00ce610 con 0x7f5ef00d3400 2024-03-06T07:33:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.760+0000 7f5ef7f5c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef00d2920 con 0x7f5ef0071540 2024-03-06T07:33:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.760+0000 7f5ef7f5c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef00d2f70 con 0x7f5ef00b2b50 2024-03-06T07:33:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.760+0000 7f5ef54f7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5ef00d3400 0x7f5ef00b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51104/0 (socket says 172.21.15.27:51104) 2024-03-06T07:33:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.760+0000 7f5ef54f7640 1 -- 172.21.15.27:0/1793296743 learned_addr learned my addr 172.21.15.27:0/1793296743 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.760+0000 7f5ef64f9640 1 --2- 172.21.15.27:0/1793296743 >> v2:172.21.15.69:3300/0 conn(0x7f5ef00b2b50 0x7f5ef00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.760+0000 7f5ef4cf6640 1 -- 172.21.15.27:0/1793296743 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222604211 0 0) 0x7f5ef00ce610 con 0x7f5ef00d3400 2024-03-06T07:33:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.760+0000 7f5ef4cf6640 1 -- 172.21.15.27:0/1793296743 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5edc003520 con 0x7f5ef00d3400 2024-03-06T07:33:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.761+0000 7f5ef64f9640 1 -- 172.21.15.27:0/1793296743 >> v1:172.21.15.69:6789/0 conn(0x7f5ef0071540 legacy=0x7f5ef0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.761+0000 7f5ef64f9640 1 -- 172.21.15.27:0/1793296743 >> v1:172.21.15.27:6789/0 conn(0x7f5ef00d3400 legacy=0x7f5ef00b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.761+0000 7f5ef64f9640 1 -- 172.21.15.27:0/1793296743 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef00d2c00 con 0x7f5ef00b2b50 2024-03-06T07:33:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.761+0000 7f5ef64f9640 1 --2- 172.21.15.27:0/1793296743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00b2b50 0x7f5ef00b30a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d90 tx=0x7f5ee802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ff56105ad963b4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef4cf6640 1 -- 172.21.15.27:0/1793296743 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8039070 con 0x7f5ef00b2b50 2024-03-06T07:33:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef4cf6640 1 -- 172.21.15.27:0/1793296743 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee802cd00 con 0x7f5ef00b2b50 2024-03-06T07:33:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef4cf6640 1 -- 172.21.15.27:0/1793296743 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8034660 con 0x7f5ef00b2b50 2024-03-06T07:33:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/1793296743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00b2b50 msgr2=0x7f5ef00b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef7f5c640 1 --2- 172.21.15.27:0/1793296743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00b2b50 0x7f5ef00b30a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d90 tx=0x7f5ee802bf50 comp rx=0 tx=0).stop 2024-03-06T07:33:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/1793296743 shutdown_connections 2024-03-06T07:33:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef7f5c640 1 --2- 172.21.15.27:0/1793296743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef00b2b50 0x7f5ef00b30a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/1793296743 >> 172.21.15.27:0/1793296743 conn(0x7f5ef00d3ca0 msgr2=0x7f5ef00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/1793296743 shutdown_connections 2024-03-06T07:33:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.762+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/1793296743 wait complete. 2024-03-06T07:33:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.763+0000 7f5ef7f5c640 1 Processor -- start 2024-03-06T07:33:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef7f5c640 1 -- start start 2024-03-06T07:33:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef7f5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef0071540 0x7f5ef00b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef7f5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00b2b50 0x7f5ef00c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef7f5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef00d3400 0x7f5ef00ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef7f5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef015e370 con 0x7f5ef00d3400 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef7f5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00d4be0 con 0x7f5ef0071540 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef7f5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00c8bf0 con 0x7f5ef00b2b50 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef64f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef00d3400 0x7f5ef00ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef64f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef00d3400 0x7f5ef00ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56532/0 (socket says 172.21.15.27:56532) 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.764+0000 7f5ef64f9640 1 -- 172.21.15.27:0/2091993001 learned_addr learned my addr 172.21.15.27:0/2091993001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef5cf8640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef0071540 0x7f5ef00b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef54f7640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00b2b50 0x7f5ef00c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef64f9640 1 -- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00b2b50 msgr2=0x7f5ef00c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef64f9640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00b2b50 0x7f5ef00c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef64f9640 1 -- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef0071540 msgr2=0x7f5ef00b5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef64f9640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef0071540 0x7f5ef00b5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef64f9640 1 -- 172.21.15.27:0/2091993001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef00c8910 con 0x7f5ef00d3400 2024-03-06T07:33:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef64f9640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef00d3400 0x7f5ef00ccf40 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d90 tx=0x7f5ee8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ee67fc640 1 -- 172.21.15.27:0/2091993001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8039050 con 0x7f5ef00d3400 2024-03-06T07:33:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ee67fc640 1 -- 172.21.15.27:0/2091993001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee8045070 con 0x7f5ef00d3400 2024-03-06T07:33:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef014caa0 con 0x7f5ef00d3400 2024-03-06T07:33:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ee67fc640 1 -- 172.21.15.27:0/2091993001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8009450 con 0x7f5ef00d3400 2024-03-06T07:33:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.765+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef014d060 con 0x7f5ef00d3400 2024-03-06T07:33:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.766+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef0068700 con 0x7f5ef00d3400 2024-03-06T07:33:08.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.767+0000 7f5ee67fc640 1 -- 172.21.15.27:0/2091993001 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ee803e850 con 0x7f5ef00d3400 2024-03-06T07:33:08.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.767+0000 7f5ee67fc640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed406b4d0 0x7f5ed406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.767+0000 7f5ee67fc640 1 -- 172.21.15.27:0/2091993001 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5ee80ad110 con 0x7f5ef00d3400 2024-03-06T07:33:08.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.767+0000 7f5ef5cf8640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed406b4d0 0x7f5ed406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.768+0000 7f5ef5cf8640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed406b4d0 0x7f5ed406d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5eec007fd0 tx=0x7f5eec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.771+0000 7f5ee67fc640 1 -- 172.21.15.27:0/2091993001 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ee8032e20 con 0x7f5ef00d3400 2024-03-06T07:33:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.991+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ef00d3890 con 0x7f5ef00d3400 2024-03-06T07:33:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.992+0000 7f5ee67fc640 1 -- 172.21.15.27:0/2091993001 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7f5ee807c090 con 0x7f5ef00d3400 2024-03-06T07:33:08.994 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:08.994 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:08.994 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:33:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed406b4d0 msgr2=0x7f5ed406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed406b4d0 0x7f5ed406d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5eec007fd0 tx=0x7f5eec009040 comp rx=0 tx=0).stop 2024-03-06T07:33:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef00d3400 msgr2=0x7f5ef00ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef00d3400 0x7f5ef00ccf40 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d90 tx=0x7f5ee8004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 shutdown_connections 2024-03-06T07:33:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed406b4d0 0x7f5ed406d990 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef00d3400 0x7f5ef00ccf40 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ef00b2b50 0x7f5ef00c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 --2- 172.21.15.27:0/2091993001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef0071540 0x7f5ef00b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 >> 172.21.15.27:0/2091993001 conn(0x7f5ef00d3ca0 msgr2=0x7f5ef015f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 shutdown_connections 2024-03-06T07:33:08.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:08.994+0000 7f5ef7f5c640 1 -- 172.21.15.27:0/2091993001 wait complete. 2024-03-06T07:33:09.207 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:09.207 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:10.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:10.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.313+0000 7fa3103c7640 1 Processor -- start 2024-03-06T07:33:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.313+0000 7fa3103c7640 1 -- start start 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa3103c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa3080b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa3103c7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3080b9a10 0x7fa3080b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa3103c7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3080cea30 con 0x7fa30815df00 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa3103c7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3080cf3c0 con 0x7fa3080b9a10 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa3103c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080cf0e0 con 0x7fa3080b95a0 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa30e964640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3080b9a10 0x7fa3080b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa30e964640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3080b9a10 0x7fa3080b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56548/0 (socket says 172.21.15.27:56548) 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa30e964640 1 -- 172.21.15.27:0/1498217432 learned_addr learned my addr 172.21.15.27:0/1498217432 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa30d962640 1 --2- 172.21.15.27:0/1498217432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa3080b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.314+0000 7fa30e964640 1 -- 172.21.15.27:0/1498217432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 msgr2=0x7fa3080b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.315+0000 7fa30e964640 1 --2- 172.21.15.27:0/1498217432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa3080b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.315+0000 7fa30e964640 1 -- 172.21.15.27:0/1498217432 >> v1:172.21.15.69:6789/0 conn(0x7fa30815df00 legacy=0x7fa3080d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.315+0000 7fa30e964640 1 -- 172.21.15.27:0/1498217432 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3080ce470 con 0x7fa3080b9a10 2024-03-06T07:33:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.315+0000 7fa30d962640 1 --2- 172.21.15.27:0/1498217432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa3080b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.315+0000 7fa30e964640 1 --2- 172.21.15.27:0/1498217432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 0x7fa3080b5c40 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fa300009870 tx=0x7fa30002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6341367b0fbc5e9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.315+0000 7fa30d161640 1 -- 172.21.15.27:0/1498217432 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300035020 con 0x7fa3080b9a10 2024-03-06T07:33:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.315+0000 7fa30d161640 1 -- 172.21.15.27:0/1498217432 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa300031640 con 0x7fa3080b9a10 2024-03-06T07:33:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.316+0000 7fa30d161640 1 -- 172.21.15.27:0/1498217432 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30002d040 con 0x7fa3080b9a10 2024-03-06T07:33:10.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.316+0000 7fa3103c7640 1 -- 172.21.15.27:0/1498217432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 msgr2=0x7fa3080b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.316+0000 7fa3103c7640 1 --2- 172.21.15.27:0/1498217432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 0x7fa3080b5c40 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fa300009870 tx=0x7fa30002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.316+0000 7fa3103c7640 1 -- 172.21.15.27:0/1498217432 shutdown_connections 2024-03-06T07:33:10.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.316+0000 7fa3103c7640 1 --2- 172.21.15.27:0/1498217432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 0x7fa3080b5c40 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.316+0000 7fa3103c7640 1 --2- 172.21.15.27:0/1498217432 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa3080b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.316+0000 7fa3103c7640 1 -- 172.21.15.27:0/1498217432 >> 172.21.15.27:0/1498217432 conn(0x7fa308071960 msgr2=0x7fa308071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.316+0000 7fa3103c7640 1 -- 172.21.15.27:0/1498217432 shutdown_connections 2024-03-06T07:33:10.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.316+0000 7fa3103c7640 1 -- 172.21.15.27:0/1498217432 wait complete. 2024-03-06T07:33:10.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.318+0000 7fa3103c7640 1 Processor -- start 2024-03-06T07:33:10.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.318+0000 7fa3103c7640 1 -- start start 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa3103c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa30814a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa3103c7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 0x7fa308150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa3103c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa30815df00 0x7fa3080c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa3103c7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080cedf0 con 0x7fa3080b9a10 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa3103c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080cec70 con 0x7fa30815df00 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa3103c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080d3220 con 0x7fa3080b95a0 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30d962640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 0x7fa308150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30d962640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 0x7fa308150360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56554/0 (socket says 172.21.15.27:56554) 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30d962640 1 -- 172.21.15.27:0/2438119859 learned_addr learned my addr 172.21.15.27:0/2438119859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:10.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30e964640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa30815df00 0x7fa3080c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30e163640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa30814a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30d962640 1 -- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 msgr2=0x7fa30814a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30d962640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa30814a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30d962640 1 -- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa30815df00 msgr2=0x7fa3080c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30d962640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa30815df00 0x7fa3080c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30d962640 1 -- 172.21.15.27:0/2438119859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308150a40 con 0x7fa3080b9a10 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30e163640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa30814a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30e964640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa30815df00 0x7fa3080c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.320+0000 7fa30d962640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 0x7fa308150360 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800c7f0 tx=0x7fa2f800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.321+0000 7fa2feffd640 1 -- 172.21.15.27:0/2438119859 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8015070 con 0x7fa3080b9a10 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.321+0000 7fa2feffd640 1 -- 172.21.15.27:0/2438119859 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f8007a30 con 0x7fa3080b9a10 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.321+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3080cbf80 con 0x7fa3080b9a10 2024-03-06T07:33:10.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.321+0000 7fa2feffd640 1 -- 172.21.15.27:0/2438119859 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8007e00 con 0x7fa3080b9a10 2024-03-06T07:33:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.321+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3080cc540 con 0x7fa3080b9a10 2024-03-06T07:33:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.322+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa308067ff0 con 0x7fa3080b9a10 2024-03-06T07:33:10.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.322+0000 7fa2feffd640 1 -- 172.21.15.27:0/2438119859 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2f8018070 con 0x7fa3080b9a10 2024-03-06T07:33:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.323+0000 7fa2feffd640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2f406b4d0 0x7fa2f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.323+0000 7fa2feffd640 1 -- 172.21.15.27:0/2438119859 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa2f808a1d0 con 0x7fa3080b9a10 2024-03-06T07:33:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.323+0000 7fa30e163640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2f406b4d0 0x7fa2f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.324+0000 7fa30e163640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2f406b4d0 0x7fa2f406d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa3080b60f0 tx=0x7fa3040073c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.327+0000 7fa2feffd640 1 -- 172.21.15.27:0/2438119859 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2f8059030 con 0x7fa3080b9a10 2024-03-06T07:33:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.552+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3080cc710 con 0x7fa3080b9a10 2024-03-06T07:33:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.552+0000 7fa2feffd640 1 -- 172.21.15.27:0/2438119859 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7fa2f8058930 con 0x7fa3080b9a10 2024-03-06T07:33:10.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:10.554 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:33:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2f406b4d0 msgr2=0x7fa2f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2f406b4d0 0x7fa2f406d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa3080b60f0 tx=0x7fa3040073c0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 msgr2=0x7fa308150360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 0x7fa308150360 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800c7f0 tx=0x7fa2f800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 shutdown_connections 2024-03-06T07:33:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa2f406b4d0 0x7fa2f406d990 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa30815df00 0x7fa3080c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3080b9a10 0x7fa308150360 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 --2- 172.21.15.27:0/2438119859 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3080b95a0 0x7fa30814a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 >> 172.21.15.27:0/2438119859 conn(0x7fa308071960 msgr2=0x7fa308071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 shutdown_connections 2024-03-06T07:33:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:10.554+0000 7fa3103c7640 1 -- 172.21.15.27:0/2438119859 wait complete. 2024-03-06T07:33:10.767 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:10.767 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:11.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:11.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.875+0000 7ff3f51a0640 1 Processor -- start 2024-03-06T07:33:11.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.875+0000 7ff3f51a0640 1 -- start start 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.876+0000 7ff3f51a0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff3f00ce9f0 0x7ff3f00d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.876+0000 7ff3f51a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f006bb40 0x7ff3f006bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.876+0000 7ff3f51a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f00cfea0 con 0x7ff3f01619c0 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.876+0000 7ff3f51a0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f00d0180 con 0x7ff3f00ce9f0 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.876+0000 7ff3f51a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00d0460 con 0x7ff3f006bb40 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3effff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff3f00ce9f0 0x7ff3f00d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3f499f640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff3f01619c0 0x7ff3f015f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:54420/0 (socket says 172.21.15.27:54420) 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3f499f640 1 -- 172.21.15.27:0/2912689047 learned_addr learned my addr 172.21.15.27:0/2912689047 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3ef7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f006bb40 0x7ff3f006bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3eeffd640 1 -- 172.21.15.27:0/2912689047 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638683 0 0) 0x7ff3f00cfea0 con 0x7ff3f01619c0 2024-03-06T07:33:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3eeffd640 1 -- 172.21.15.27:0/2912689047 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc003520 con 0x7ff3f01619c0 2024-03-06T07:33:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3effff640 1 -- 172.21.15.27:0/2912689047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f006bb40 msgr2=0x7ff3f006bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3effff640 1 --2- 172.21.15.27:0/2912689047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f006bb40 0x7ff3f006bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3effff640 1 -- 172.21.15.27:0/2912689047 >> v1:172.21.15.69:6789/0 conn(0x7ff3f01619c0 legacy=0x7ff3f015f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3effff640 1 -- 172.21.15.27:0/2912689047 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f00d0740 con 0x7ff3f00ce9f0 2024-03-06T07:33:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.877+0000 7ff3ef7fe640 1 --2- 172.21.15.27:0/2912689047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f006bb40 0x7ff3f006bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:11.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.878+0000 7ff3effff640 1 --2- 172.21.15.27:0/2912689047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3f00ce9f0 0x7ff3f00d0f40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00097a0 tx=0x7ff3e002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb4074e73c7760bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.878+0000 7ff3eeffd640 1 -- 172.21.15.27:0/2912689047 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0034070 con 0x7ff3f00ce9f0 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.878+0000 7ff3eeffd640 1 -- 172.21.15.27:0/2912689047 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e0030610 con 0x7ff3f00ce9f0 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.878+0000 7ff3eeffd640 1 -- 172.21.15.27:0/2912689047 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e00309e0 con 0x7ff3f00ce9f0 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.879+0000 7ff3f51a0640 1 -- 172.21.15.27:0/2912689047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3f00ce9f0 msgr2=0x7ff3f00d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.879+0000 7ff3f51a0640 1 --2- 172.21.15.27:0/2912689047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3f00ce9f0 0x7ff3f00d0f40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00097a0 tx=0x7ff3e002b960 comp rx=0 tx=0).stop 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.879+0000 7ff3f51a0640 1 -- 172.21.15.27:0/2912689047 shutdown_connections 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.879+0000 7ff3f51a0640 1 --2- 172.21.15.27:0/2912689047 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f006bb40 0x7ff3f006bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.879+0000 7ff3f51a0640 1 --2- 172.21.15.27:0/2912689047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3f00ce9f0 0x7ff3f00d0f40 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.879+0000 7ff3f51a0640 1 -- 172.21.15.27:0/2912689047 >> 172.21.15.27:0/2912689047 conn(0x7ff3f00b3430 msgr2=0x7ff3f00b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.879+0000 7ff3f51a0640 1 -- 172.21.15.27:0/2912689047 shutdown_connections 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.879+0000 7ff3f51a0640 1 -- 172.21.15.27:0/2912689047 wait complete. 2024-03-06T07:33:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.880+0000 7ff3f51a0640 1 Processor -- start 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.880+0000 7ff3f51a0640 1 -- start start 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f51a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3f006bb40 0x7ff3f00b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f51a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f00ce9f0 0x7ff3f00b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f51a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f01619c0 0x7ff3f00b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f51a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0161e50 con 0x7ff3f01619c0 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f51a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0162720 con 0x7ff3f006bb40 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f51a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0161fd0 con 0x7ff3f00ce9f0 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f499f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f01619c0 0x7ff3f00b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f499f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f01619c0 0x7ff3f00b5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56558/0 (socket says 172.21.15.27:56558) 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f499f640 1 -- 172.21.15.27:0/1171588293 learned_addr learned my addr 172.21.15.27:0/1171588293 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3ef7fe640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f00ce9f0 0x7ff3f00b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3effff640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3f006bb40 0x7ff3f00b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f499f640 1 -- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f00ce9f0 msgr2=0x7ff3f00b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f499f640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f00ce9f0 0x7ff3f00b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f499f640 1 -- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3f006bb40 msgr2=0x7ff3f00b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f499f640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3f006bb40 0x7ff3f00b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f499f640 1 -- 172.21.15.27:0/1171588293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f01e8760 con 0x7ff3f01619c0 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.881+0000 7ff3f499f640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f01619c0 0x7ff3f00b5fa0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800caa0 tx=0x7ff3e800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.882+0000 7ff3ecff9640 1 -- 172.21.15.27:0/1171588293 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8017070 con 0x7ff3f01619c0 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.882+0000 7ff3ecff9640 1 -- 172.21.15.27:0/1171588293 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e800aba0 con 0x7ff3f01619c0 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.882+0000 7ff3ecff9640 1 -- 172.21.15.27:0/1171588293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8012400 con 0x7ff3f01619c0 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.882+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f00b4f70 con 0x7ff3f01619c0 2024-03-06T07:33:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.882+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f00b5530 con 0x7ff3f01619c0 2024-03-06T07:33:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.883+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f004f330 con 0x7ff3f01619c0 2024-03-06T07:33:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.883+0000 7ff3ecff9640 1 -- 172.21.15.27:0/1171588293 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3e801a070 con 0x7ff3f01619c0 2024-03-06T07:33:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.884+0000 7ff3ecff9640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3d006b500 0x7ff3d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.884+0000 7ff3ecff9640 1 -- 172.21.15.27:0/1171588293 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff3e8089e80 con 0x7ff3f01619c0 2024-03-06T07:33:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.884+0000 7ff3effff640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3d006b500 0x7ff3d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.884+0000 7ff3effff640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3d006b500 0x7ff3d006d9c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0036000 tx=0x7ff3e003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:11.888+0000 7ff3ecff9640 1 -- 172.21.15.27:0/1171588293 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3e8058690 con 0x7ff3f01619c0 2024-03-06T07:33:12.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.111+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3f00b6600 con 0x7ff3f01619c0 2024-03-06T07:33:12.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.111+0000 7ff3ecff9640 1 -- 172.21.15.27:0/1171588293 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3422 (secure 0 0 0) 0x7ff3e805c730 con 0x7ff3f01619c0 2024-03-06T07:33:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:12.113 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:32:41.322202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:12.113 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-06T07:33:12.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3d006b500 msgr2=0x7ff3d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3d006b500 0x7ff3d006d9c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0036000 tx=0x7ff3e003b040 comp rx=0 tx=0).stop 2024-03-06T07:33:12.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f01619c0 msgr2=0x7ff3f00b5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f01619c0 0x7ff3f00b5fa0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800caa0 tx=0x7ff3e800cf60 comp rx=0 tx=0).stop 2024-03-06T07:33:12.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 shutdown_connections 2024-03-06T07:33:12.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff3d006b500 0x7ff3d006d9c0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f01619c0 0x7ff3f00b5fa0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3f00ce9f0 0x7ff3f00b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 --2- 172.21.15.27:0/1171588293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3f006bb40 0x7ff3f00b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.113+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 >> 172.21.15.27:0/1171588293 conn(0x7ff3f00b3430 msgr2=0x7ff3f006af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.114+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 shutdown_connections 2024-03-06T07:33:12.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:12.114+0000 7ff3f51a0640 1 -- 172.21.15.27:0/1171588293 wait complete. 2024-03-06T07:33:12.327 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:32:41.322202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:12.327 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:13.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.434+0000 7f65695f4640 1 Processor -- start 2024-03-06T07:33:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.434+0000 7f65695f4640 1 -- start start 2024-03-06T07:33:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f65695f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6564071540 0x7f65640b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f65695f4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f65640d3400 0x7f6564071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f65695f4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65640b2b50 0x7f65640d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f65695f4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65640d2f70 con 0x7f65640b2b50 2024-03-06T07:33:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f65695f4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65640d4be0 con 0x7f65640d3400 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f65695f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6564071e60 con 0x7f6564071540 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f65637fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65640b2b50 0x7f65640d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f65637fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65640b2b50 0x7f65640d1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56560/0 (socket says 172.21.15.27:56560) 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f65637fe640 1 -- 172.21.15.27:0/2318986550 learned_addr learned my addr 172.21.15.27:0/2318986550 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f655a7fc640 1 --2- 172.21.15.27:0/2318986550 >> v2:172.21.15.69:3300/0 conn(0x7f65640d3400 0x7f6564071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.435+0000 7f6562ffd640 1 --2- 172.21.15.27:0/2318986550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6564071540 0x7f65640b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.436+0000 7f65637fe640 1 -- 172.21.15.27:0/2318986550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6564071540 msgr2=0x7f65640b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.436+0000 7f65637fe640 1 --2- 172.21.15.27:0/2318986550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6564071540 0x7f65640b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.436+0000 7f65637fe640 1 -- 172.21.15.27:0/2318986550 >> v2:172.21.15.69:3300/0 conn(0x7f65640d3400 msgr2=0x7f6564071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.436+0000 7f65637fe640 1 --2- 172.21.15.27:0/2318986550 >> v2:172.21.15.69:3300/0 conn(0x7f65640d3400 0x7f6564071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.436+0000 7f65637fe640 1 -- 172.21.15.27:0/2318986550 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65640ce610 con 0x7f65640b2b50 2024-03-06T07:33:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65637fe640 1 --2- 172.21.15.27:0/2318986550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 0x7f65640d1870 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f656408c810 tx=0x7f655c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3602ae1d7be027ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65627fc640 1 -- 172.21.15.27:0/2318986550 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c02ca70 con 0x7f65640b2b50 2024-03-06T07:33:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65627fc640 1 -- 172.21.15.27:0/2318986550 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c02cc10 con 0x7f65640b2b50 2024-03-06T07:33:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65627fc640 1 -- 172.21.15.27:0/2318986550 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c0343f0 con 0x7f65640b2b50 2024-03-06T07:33:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65695f4640 1 -- 172.21.15.27:0/2318986550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 msgr2=0x7f65640d1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65695f4640 1 --2- 172.21.15.27:0/2318986550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 0x7f65640d1870 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f656408c810 tx=0x7f655c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65695f4640 1 -- 172.21.15.27:0/2318986550 shutdown_connections 2024-03-06T07:33:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65695f4640 1 --2- 172.21.15.27:0/2318986550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 0x7f65640d1870 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65695f4640 1 --2- 172.21.15.27:0/2318986550 >> v2:172.21.15.69:3300/0 conn(0x7f65640d3400 0x7f6564071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65695f4640 1 --2- 172.21.15.27:0/2318986550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6564071540 0x7f65640b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65695f4640 1 -- 172.21.15.27:0/2318986550 >> 172.21.15.27:0/2318986550 conn(0x7f65640d3ca0 msgr2=0x7f65640d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65695f4640 1 -- 172.21.15.27:0/2318986550 shutdown_connections 2024-03-06T07:33:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.437+0000 7f65695f4640 1 -- 172.21.15.27:0/2318986550 wait complete. 2024-03-06T07:33:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.438+0000 7f65695f4640 1 Processor -- start 2024-03-06T07:33:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.439+0000 7f65695f4640 1 -- start start 2024-03-06T07:33:13.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.439+0000 7f65695f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564071540 0x7f65640b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.439+0000 7f65695f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 0x7f65641f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.439+0000 7f65695f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65640d3400 0x7f65640cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.439+0000 7f65695f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65640ce8f0 con 0x7f65640b2b50 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.439+0000 7f65695f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65640d2c00 con 0x7f6564071540 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.439+0000 7f65695f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656415e370 con 0x7f65640d3400 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.439+0000 7f655a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 0x7f65641f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 0x7f65641f0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56566/0 (socket says 172.21.15.27:56566) 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655a7fc640 1 -- 172.21.15.27:0/1503719635 learned_addr learned my addr 172.21.15.27:0/1503719635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f65637fe640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65640d3400 0x7f65640cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f6562ffd640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564071540 0x7f65640b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655a7fc640 1 -- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65640d3400 msgr2=0x7f65640cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655a7fc640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65640d3400 0x7f65640cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655a7fc640 1 -- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564071540 msgr2=0x7f65640b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655a7fc640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564071540 0x7f65640b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655a7fc640 1 -- 172.21.15.27:0/1503719635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65640b5c80 con 0x7f65640b2b50 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655a7fc640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 0x7f65641f0ae0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f654c00e840 tx=0x7f654c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655bfff640 1 -- 172.21.15.27:0/1503719635 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c017070 con 0x7f65640b2b50 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655bfff640 1 -- 172.21.15.27:0/1503719635 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654c00c940 con 0x7f65640b2b50 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656414c350 con 0x7f65640b2b50 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f655bfff640 1 -- 172.21.15.27:0/1503719635 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c00cd10 con 0x7f65640b2b50 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.440+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656414c910 con 0x7f65640b2b50 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.441+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6564068700 con 0x7f65640b2b50 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.442+0000 7f655bfff640 1 -- 172.21.15.27:0/1503719635 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f654c01a070 con 0x7f65640b2b50 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.442+0000 7f655bfff640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f654406b4d0 0x7f654406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.442+0000 7f655bfff640 1 -- 172.21.15.27:0/1503719635 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f654c08a4c0 con 0x7f65640b2b50 2024-03-06T07:33:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.442+0000 7f6562ffd640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f654406b4d0 0x7f654406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.443+0000 7f6562ffd640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f654406b4d0 0x7f654406d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6554009a50 tx=0x7f6554008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.446+0000 7f655bfff640 1 -- 172.21.15.27:0/1503719635 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f654c059320 con 0x7f65640b2b50 2024-03-06T07:33:13.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.665+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65640b9a60 con 0x7f65640b2b50 2024-03-06T07:33:13.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.666+0000 7f655bfff640 1 -- 172.21.15.27:0/1503719635 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f654c058c20 con 0x7f65640b2b50 2024-03-06T07:33:13.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:13.668 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f654406b4d0 msgr2=0x7f654406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f654406b4d0 0x7f654406d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6554009a50 tx=0x7f6554008040 comp rx=0 tx=0).stop 2024-03-06T07:33:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 msgr2=0x7f65641f0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 0x7f65641f0ae0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f654c00e840 tx=0x7f654c00ed00 comp rx=0 tx=0).stop 2024-03-06T07:33:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 shutdown_connections 2024-03-06T07:33:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f654406b4d0 0x7f654406d990 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f65640d3400 0x7f65640cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65640b2b50 0x7f65641f0ae0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 --2- 172.21.15.27:0/1503719635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564071540 0x7f65640b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 >> 172.21.15.27:0/1503719635 conn(0x7f65640d3ca0 msgr2=0x7f65640ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 shutdown_connections 2024-03-06T07:33:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:13.669+0000 7f65695f4640 1 -- 172.21.15.27:0/1503719635 wait complete. 2024-03-06T07:33:13.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:13.883 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:14.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.990+0000 7f3f4fff6640 1 Processor -- start 2024-03-06T07:33:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.990+0000 7f3f4fff6640 1 -- start start 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.991+0000 7f3f4fff6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f48158520 0x7f3f4806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.991+0000 7f3f4fff6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f480aee10 0x7f3f480b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.991+0000 7f3f4fff6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f48071240 con 0x7f3f4806dd60 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.991+0000 7f3f4fff6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4806e1c0 con 0x7f3f480aee10 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.991+0000 7f3f4fff6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f48071b70 con 0x7f3f48158520 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.991+0000 7f3f4e593640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f480aee10 0x7f3f480b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.991+0000 7f3f4e593640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f480aee10 0x7f3f480b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37842/0 (socket says 172.21.15.27:37842) 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.991+0000 7f3f4e593640 1 -- 172.21.15.27:0/2320802249 learned_addr learned my addr 172.21.15.27:0/2320802249 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.991+0000 7f3f4d591640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3f4806dd60 0x7f3f480ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60218/0 (socket says 172.21.15.27:60218) 2024-03-06T07:33:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.992+0000 7f3f4dd92640 1 --2- 172.21.15.27:0/2320802249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f48158520 0x7f3f4806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.992+0000 7f3f4e593640 1 -- 172.21.15.27:0/2320802249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f48158520 msgr2=0x7f3f4806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.992+0000 7f3f4e593640 1 --2- 172.21.15.27:0/2320802249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f48158520 0x7f3f4806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.992+0000 7f3f4e593640 1 -- 172.21.15.27:0/2320802249 >> v1:172.21.15.27:6789/0 conn(0x7f3f4806dd60 legacy=0x7f3f480ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.992+0000 7f3f4e593640 1 -- 172.21.15.27:0/2320802249 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f48071520 con 0x7f3f480aee10 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.992+0000 7f3f4cd90640 1 -- 172.21.15.27:0/2320802249 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298275047 0 0) 0x7f3f48071240 con 0x7f3f4806dd60 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.992+0000 7f3f4dd92640 1 --2- 172.21.15.27:0/2320802249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f48158520 0x7f3f4806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.992+0000 7f3f4e593640 1 --2- 172.21.15.27:0/2320802249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f480aee10 0x7f3f480b3640 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009e50 tx=0x7f3f3802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d19f6a552b9b47f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4cd90640 1 -- 172.21.15.27:0/2320802249 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3802caf0 con 0x7f3f480aee10 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4cd90640 1 -- 172.21.15.27:0/2320802249 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3802cc90 con 0x7f3f480aee10 2024-03-06T07:33:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4cd90640 1 -- 172.21.15.27:0/2320802249 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38034610 con 0x7f3f480aee10 2024-03-06T07:33:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4fff6640 1 -- 172.21.15.27:0/2320802249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f480aee10 msgr2=0x7f3f480b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4fff6640 1 --2- 172.21.15.27:0/2320802249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f480aee10 0x7f3f480b3640 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009e50 tx=0x7f3f3802bf50 comp rx=0 tx=0).stop 2024-03-06T07:33:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4fff6640 1 -- 172.21.15.27:0/2320802249 shutdown_connections 2024-03-06T07:33:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4fff6640 1 --2- 172.21.15.27:0/2320802249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f480aee10 0x7f3f480b3640 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4fff6640 1 --2- 172.21.15.27:0/2320802249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f48158520 0x7f3f4806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4fff6640 1 -- 172.21.15.27:0/2320802249 >> 172.21.15.27:0/2320802249 conn(0x7f3f48066d00 msgr2=0x7f3f480670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4fff6640 1 -- 172.21.15.27:0/2320802249 shutdown_connections 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.993+0000 7f3f4fff6640 1 -- 172.21.15.27:0/2320802249 wait complete. 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.994+0000 7f3f4fff6640 1 Processor -- start 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.994+0000 7f3f4fff6640 1 -- start start 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4fff6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f4806dd60 0x7f3f481441b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4fff6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f480aee10 0x7f3f481f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4fff6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48158520 0x7f3f48154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4fff6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f48158900 con 0x7f3f48158520 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4fff6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f48072000 con 0x7f3f4806dd60 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4fff6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f48158a80 con 0x7f3f480aee10 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4e593640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48158520 0x7f3f48154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4e593640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48158520 0x7f3f48154310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37858/0 (socket says 172.21.15.27:37858) 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4e593640 1 -- 172.21.15.27:0/3854613290 learned_addr learned my addr 172.21.15.27:0/3854613290 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4dd92640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f4806dd60 0x7f3f481441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.995+0000 7f3f4d591640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f480aee10 0x7f3f481f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f4e593640 1 -- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f480aee10 msgr2=0x7f3f481f2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f4e593640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f480aee10 0x7f3f481f2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f4e593640 1 -- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f4806dd60 msgr2=0x7f3f481441b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f4e593640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f4806dd60 0x7f3f481441b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f4e593640 1 -- 172.21.15.27:0/3854613290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f481f3420 con 0x7f3f48158520 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f4dd92640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f4806dd60 0x7f3f481441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f4e593640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48158520 0x7f3f48154310 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009e50 tx=0x7f3f38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f467fc640 1 -- 172.21.15.27:0/3854613290 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38043030 con 0x7f3f48158520 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f467fc640 1 -- 172.21.15.27:0/3854613290 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3803e450 con 0x7f3f48158520 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f481f3700 con 0x7f3f48158520 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f467fc640 1 -- 172.21.15.27:0/3854613290 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3803e620 con 0x7f3f48158520 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.996+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f480b1240 con 0x7f3f48158520 2024-03-06T07:33:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.997+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f48067ff0 con 0x7f3f48158520 2024-03-06T07:33:15.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.997+0000 7f3f467fc640 1 -- 172.21.15.27:0/3854613290 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f38046070 con 0x7f3f48158520 2024-03-06T07:33:15.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.998+0000 7f3f467fc640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f3406b4d0 0x7f3f3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.998+0000 7f3f467fc640 1 -- 172.21.15.27:0/3854613290 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3f380ac870 con 0x7f3f48158520 2024-03-06T07:33:15.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.998+0000 7f3f4dd92640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f3406b4d0 0x7f3f3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:14.999+0000 7f3f4dd92640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f3406b4d0 0x7f3f3406d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f3f480b08c0 tx=0x7f3f40008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.002+0000 7f3f467fc640 1 -- 172.21.15.27:0/3854613290 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f3804d070 con 0x7f3f48158520 2024-03-06T07:33:15.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.273+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f480cbc80 con 0x7f3f48158520 2024-03-06T07:33:15.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.273+0000 7f3f467fc640 1 -- 172.21.15.27:0/3854613290 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f3f3807b760 con 0x7f3f48158520 2024-03-06T07:33:15.275 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:15.275 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:15.275 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f3406b4d0 msgr2=0x7f3f3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f3406b4d0 0x7f3f3406d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f3f480b08c0 tx=0x7f3f40008040 comp rx=0 tx=0).stop 2024-03-06T07:33:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48158520 msgr2=0x7f3f48154310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48158520 0x7f3f48154310 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009e50 tx=0x7f3f38004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 shutdown_connections 2024-03-06T07:33:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3f3406b4d0 0x7f3f3406d990 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f48158520 0x7f3f48154310 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3f480aee10 0x7f3f481f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 --2- 172.21.15.27:0/3854613290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f4806dd60 0x7f3f481441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 >> 172.21.15.27:0/3854613290 conn(0x7f3f48066d00 msgr2=0x7f3f480670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 shutdown_connections 2024-03-06T07:33:15.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:15.275+0000 7f3f4fff6640 1 -- 172.21.15.27:0/3854613290 wait complete. 2024-03-06T07:33:15.488 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:15.489 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:16.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:16.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.595+0000 7f3d376fb640 1 Processor -- start 2024-03-06T07:33:16.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.596+0000 7f3d376fb640 1 -- start start 2024-03-06T07:33:16.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.596+0000 7f3d376fb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d3006df30 0x7f3d300ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.596+0000 7f3d376fb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d300aec40 0x7f3d300b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.596+0000 7f3d376fb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d30071240 con 0x7f3d30158520 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.596+0000 7f3d376fb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d300af0f0 con 0x7f3d300aec40 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.596+0000 7f3d376fb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d30071b70 con 0x7f3d3006df30 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.597+0000 7f3d35c98640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d300aec40 0x7f3d300b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.597+0000 7f3d35497640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d30158520 0x7f3d3006d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60222/0 (socket says 172.21.15.27:60222) 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.597+0000 7f3d35c98640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d300aec40 0x7f3d300b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37872/0 (socket says 172.21.15.27:37872) 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.597+0000 7f3d35497640 1 -- 172.21.15.27:0/3199675814 learned_addr learned my addr 172.21.15.27:0/3199675814 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.597+0000 7f3d34c96640 1 --2- 172.21.15.27:0/3199675814 >> v2:172.21.15.69:3300/0 conn(0x7f3d3006df30 0x7f3d300ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.597+0000 7f3d35c98640 1 -- 172.21.15.27:0/3199675814 >> v2:172.21.15.69:3300/0 conn(0x7f3d3006df30 msgr2=0x7f3d300ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.597+0000 7f3d35c98640 1 --2- 172.21.15.27:0/3199675814 >> v2:172.21.15.69:3300/0 conn(0x7f3d3006df30 0x7f3d300ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.597+0000 7f3d35c98640 1 -- 172.21.15.27:0/3199675814 >> v1:172.21.15.27:6789/0 conn(0x7f3d30158520 legacy=0x7f3d3006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.597+0000 7f3d35c98640 1 -- 172.21.15.27:0/3199675814 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d30071520 con 0x7f3d300aec40 2024-03-06T07:33:16.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d35c98640 1 --2- 172.21.15.27:0/3199675814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d300aec40 0x7f3d300b3640 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009d90 tx=0x7f3d2802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541dbbd507b7a773 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d23fff640 1 -- 172.21.15.27:0/3199675814 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28039070 con 0x7f3d300aec40 2024-03-06T07:33:16.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d23fff640 1 -- 172.21.15.27:0/3199675814 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2802cc60 con 0x7f3d300aec40 2024-03-06T07:33:16.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d23fff640 1 -- 172.21.15.27:0/3199675814 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28034660 con 0x7f3d300aec40 2024-03-06T07:33:16.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d376fb640 1 -- 172.21.15.27:0/3199675814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d300aec40 msgr2=0x7f3d300b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d376fb640 1 --2- 172.21.15.27:0/3199675814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d300aec40 0x7f3d300b3640 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009d90 tx=0x7f3d2802beb0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d376fb640 1 -- 172.21.15.27:0/3199675814 shutdown_connections 2024-03-06T07:33:16.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d376fb640 1 --2- 172.21.15.27:0/3199675814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d300aec40 0x7f3d300b3640 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d376fb640 1 --2- 172.21.15.27:0/3199675814 >> v2:172.21.15.69:3300/0 conn(0x7f3d3006df30 0x7f3d300ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.598+0000 7f3d376fb640 1 -- 172.21.15.27:0/3199675814 >> 172.21.15.27:0/3199675814 conn(0x7f3d30066d00 msgr2=0x7f3d300670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.599+0000 7f3d376fb640 1 -- 172.21.15.27:0/3199675814 shutdown_connections 2024-03-06T07:33:16.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.599+0000 7f3d376fb640 1 -- 172.21.15.27:0/3199675814 wait complete. 2024-03-06T07:33:16.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.600+0000 7f3d376fb640 1 Processor -- start 2024-03-06T07:33:16.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.600+0000 7f3d376fb640 1 -- start start 2024-03-06T07:33:16.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.600+0000 7f3d376fb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d3006df30 0x7f3d301e3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.600+0000 7f3d376fb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d300aec40 0x7f3d301f1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.600+0000 7f3d376fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d30158520 0x7f3d301f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.600+0000 7f3d376fb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30158900 con 0x7f3d30158520 2024-03-06T07:33:16.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d376fb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30072000 con 0x7f3d300aec40 2024-03-06T07:33:16.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d376fb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30158a80 con 0x7f3d3006df30 2024-03-06T07:33:16.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35c98640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d30158520 0x7f3d301f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35c98640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d30158520 0x7f3d301f4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37888/0 (socket says 172.21.15.27:37888) 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35c98640 1 -- 172.21.15.27:0/3915921999 learned_addr learned my addr 172.21.15.27:0/3915921999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35497640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d3006df30 0x7f3d301e3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d34c96640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d300aec40 0x7f3d301f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35c98640 1 -- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d3006df30 msgr2=0x7f3d301e3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35c98640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d3006df30 0x7f3d301e3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35c98640 1 -- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d300aec40 msgr2=0x7f3d301f1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35c98640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d300aec40 0x7f3d301f1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35c98640 1 -- 172.21.15.27:0/3915921999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d30155a30 con 0x7f3d30158520 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d35c98640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d30158520 0x7f3d301f4e70 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009d90 tx=0x7f3d28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.601+0000 7f3d21ffb640 1 -- 172.21.15.27:0/3915921999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28039050 con 0x7f3d30158520 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.602+0000 7f3d21ffb640 1 -- 172.21.15.27:0/3915921999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d28045070 con 0x7f3d30158520 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.602+0000 7f3d21ffb640 1 -- 172.21.15.27:0/3915921999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2803e830 con 0x7f3d30158520 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.602+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d30155d10 con 0x7f3d30158520 2024-03-06T07:33:16.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.602+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d301e4370 con 0x7f3d30158520 2024-03-06T07:33:16.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.603+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d30067ff0 con 0x7f3d30158520 2024-03-06T07:33:16.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.603+0000 7f3d21ffb640 1 -- 172.21.15.27:0/3915921999 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d2803e9d0 con 0x7f3d30158520 2024-03-06T07:33:16.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.604+0000 7f3d21ffb640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1406b4d0 0x7f3d1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.604+0000 7f3d21ffb640 1 -- 172.21.15.27:0/3915921999 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3d280acf60 con 0x7f3d30158520 2024-03-06T07:33:16.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.604+0000 7f3d35497640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1406b4d0 0x7f3d1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.605+0000 7f3d35497640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1406b4d0 0x7f3d1406d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f3d24007fd0 tx=0x7f3d24007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.609+0000 7f3d21ffb640 1 -- 172.21.15.27:0/3915921999 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d2807b890 con 0x7f3d30158520 2024-03-06T07:33:16.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.905+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d300b31f0 con 0x7f3d30158520 2024-03-06T07:33:16.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.905+0000 7f3d21ffb640 1 -- 172.21.15.27:0/3915921999 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f3d2807f930 con 0x7f3d30158520 2024-03-06T07:33:16.907 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:16.907 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:16.907 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1406b4d0 msgr2=0x7f3d1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1406b4d0 0x7f3d1406d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f3d24007fd0 tx=0x7f3d24007e80 comp rx=0 tx=0).stop 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d30158520 msgr2=0x7f3d301f4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d30158520 0x7f3d301f4e70 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009d90 tx=0x7f3d28004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 shutdown_connections 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f3d1406b4d0 0x7f3d1406d990 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d30158520 0x7f3d301f4e70 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d300aec40 0x7f3d301f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 --2- 172.21.15.27:0/3915921999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3d3006df30 0x7f3d301e3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 >> 172.21.15.27:0/3915921999 conn(0x7f3d30066d00 msgr2=0x7f3d30159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 shutdown_connections 2024-03-06T07:33:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:16.907+0000 7f3d376fb640 1 -- 172.21.15.27:0/3915921999 wait complete. 2024-03-06T07:33:17.121 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:17.121 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:18.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:18.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.229+0000 7f6a6cc9a640 1 Processor -- start 2024-03-06T07:33:18.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.229+0000 7f6a6cc9a640 1 -- start start 2024-03-06T07:33:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a6cc9a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6a680b0e90 0x7f6a6806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a6cc9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a680ae9b0 0x7f6a680d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a6cc9a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680cee90 con 0x7f6a6806b5c0 2024-03-06T07:33:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a6cc9a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a680cfa00 con 0x7f6a680b0e90 2024-03-06T07:33:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a6cc9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680cfc50 con 0x7f6a680ae9b0 2024-03-06T07:33:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a65d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6a680b0e90 0x7f6a6806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a6659c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6a6806b5c0 0x7f6a680b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60230/0 (socket says 172.21.15.27:60230) 2024-03-06T07:33:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a65d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6a680b0e90 0x7f6a6806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37892/0 (socket says 172.21.15.27:37892) 2024-03-06T07:33:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a6659c640 1 -- 172.21.15.27:0/1881993059 learned_addr learned my addr 172.21.15.27:0/1881993059 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a66d9d640 1 --2- 172.21.15.27:0/1881993059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a680ae9b0 0x7f6a680d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a65d9b640 1 -- 172.21.15.27:0/1881993059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a680ae9b0 msgr2=0x7f6a680d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a65d9b640 1 --2- 172.21.15.27:0/1881993059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a680ae9b0 0x7f6a680d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a65d9b640 1 -- 172.21.15.27:0/1881993059 >> v1:172.21.15.27:6789/0 conn(0x7f6a6806b5c0 legacy=0x7f6a680b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a65d9b640 1 -- 172.21.15.27:0/1881993059 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680cff30 con 0x7f6a680b0e90 2024-03-06T07:33:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.230+0000 7f6a6559a640 1 -- 172.21.15.27:0/1881993059 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688905417 0 0) 0x7f6a680cee90 con 0x7f6a6806b5c0 2024-03-06T07:33:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.231+0000 7f6a65d9b640 1 --2- 172.21.15.27:0/1881993059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680b0e90 0x7f6a6806b9a0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0097a0 tx=0x7f6a5c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418d48263ec396e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.231+0000 7f6a6559a640 1 -- 172.21.15.27:0/1881993059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0315a0 con 0x7f6a680b0e90 2024-03-06T07:33:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.231+0000 7f6a6559a640 1 -- 172.21.15.27:0/1881993059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c037070 con 0x7f6a680b0e90 2024-03-06T07:33:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.231+0000 7f6a6559a640 1 -- 172.21.15.27:0/1881993059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c031960 con 0x7f6a680b0e90 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.232+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1881993059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680b0e90 msgr2=0x7f6a6806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.232+0000 7f6a6cc9a640 1 --2- 172.21.15.27:0/1881993059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680b0e90 0x7f6a6806b9a0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0097a0 tx=0x7f6a5c02b840 comp rx=0 tx=0).stop 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.232+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1881993059 shutdown_connections 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.232+0000 7f6a6cc9a640 1 --2- 172.21.15.27:0/1881993059 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a680ae9b0 0x7f6a680d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.232+0000 7f6a6cc9a640 1 --2- 172.21.15.27:0/1881993059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680b0e90 0x7f6a6806b9a0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.232+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1881993059 >> 172.21.15.27:0/1881993059 conn(0x7f6a680b36d0 msgr2=0x7f6a680b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.232+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1881993059 shutdown_connections 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.232+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1881993059 wait complete. 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.233+0000 7f6a6cc9a640 1 Processor -- start 2024-03-06T07:33:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.233+0000 7f6a6cc9a640 1 -- start start 2024-03-06T07:33:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a6cc9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a6806b5c0 0x7f6a680c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a6cc9a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680ae9b0 0x7f6a681e1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a6cc9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a680b0e90 0x7f6a680cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a6cc9a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6806c910 con 0x7f6a680ae9b0 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a6cc9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6806c790 con 0x7f6a680b0e90 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a6cc9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680aee00 con 0x7f6a6806b5c0 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a65d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680ae9b0 0x7f6a681e1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a65d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680ae9b0 0x7f6a681e1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37906/0 (socket says 172.21.15.27:37906) 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a65d9b640 1 -- 172.21.15.27:0/1169595246 learned_addr learned my addr 172.21.15.27:0/1169595246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a6659c640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a6806b5c0 0x7f6a680c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a66d9d640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a680b0e90 0x7f6a680cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a65d9b640 1 -- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a6806b5c0 msgr2=0x7f6a680c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a65d9b640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a6806b5c0 0x7f6a680c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a65d9b640 1 -- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a680b0e90 msgr2=0x7f6a680cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a65d9b640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a680b0e90 0x7f6a680cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a65d9b640 1 -- 172.21.15.27:0/1169595246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a681e2230 con 0x7f6a680ae9b0 2024-03-06T07:33:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a6659c640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a6806b5c0 0x7f6a680c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.234+0000 7f6a66d9d640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a680b0e90 0x7f6a680cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.235+0000 7f6a65d9b640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680ae9b0 0x7f6a681e1cf0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0098c0 tx=0x7f6a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.235+0000 7f6a56ffd640 1 -- 172.21.15.27:0/1169595246 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c032630 con 0x7f6a680ae9b0 2024-03-06T07:33:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.235+0000 7f6a56ffd640 1 -- 172.21.15.27:0/1169595246 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c0327d0 con 0x7f6a680ae9b0 2024-03-06T07:33:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.235+0000 7f6a56ffd640 1 -- 172.21.15.27:0/1169595246 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c032ba0 con 0x7f6a680ae9b0 2024-03-06T07:33:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.235+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a680cba00 con 0x7f6a680ae9b0 2024-03-06T07:33:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.235+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a680b49a0 con 0x7f6a680ae9b0 2024-03-06T07:33:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.236+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a68068700 con 0x7f6a680ae9b0 2024-03-06T07:33:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.236+0000 7f6a56ffd640 1 -- 172.21.15.27:0/1169595246 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a5c04e430 con 0x7f6a680ae9b0 2024-03-06T07:33:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.237+0000 7f6a56ffd640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a4806b500 0x7f6a4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.237+0000 7f6a56ffd640 1 -- 172.21.15.27:0/1169595246 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6a5c02f030 con 0x7f6a680ae9b0 2024-03-06T07:33:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.237+0000 7f6a6659c640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a4806b500 0x7f6a4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.238+0000 7f6a6659c640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a4806b500 0x7f6a4806d9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6a58000f60 tx=0x7f6a58009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.241+0000 7f6a56ffd640 1 -- 172.21.15.27:0/1169595246 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a5c040070 con 0x7f6a680ae9b0 2024-03-06T07:33:18.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.459+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a6806daf0 con 0x7f6a680ae9b0 2024-03-06T07:33:18.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.460+0000 7f6a56ffd640 1 -- 172.21.15.27:0/1169595246 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f6a5c07af40 con 0x7f6a680ae9b0 2024-03-06T07:33:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:18.464 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:18.465 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:18.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a4806b500 msgr2=0x7f6a4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a4806b500 0x7f6a4806d9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6a58000f60 tx=0x7f6a58009040 comp rx=0 tx=0).stop 2024-03-06T07:33:18.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680ae9b0 msgr2=0x7f6a681e1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680ae9b0 0x7f6a681e1cf0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0098c0 tx=0x7f6a5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 shutdown_connections 2024-03-06T07:33:18.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a4806b500 0x7f6a4806d9c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a680b0e90 0x7f6a680cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a680ae9b0 0x7f6a681e1cf0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 --2- 172.21.15.27:0/1169595246 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a6806b5c0 0x7f6a680c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 >> 172.21.15.27:0/1169595246 conn(0x7f6a680b36d0 msgr2=0x7f6a681e50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 shutdown_connections 2024-03-06T07:33:18.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:18.462+0000 7f6a6cc9a640 1 -- 172.21.15.27:0/1169595246 wait complete. 2024-03-06T07:33:18.676 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:18.676 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:19.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.782+0000 7ff451e8a640 1 Processor -- start 2024-03-06T07:33:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.782+0000 7ff451e8a640 1 -- start start 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff451e8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 0x7ff44c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff451e8a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff44c0b2b50 0x7ff44c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff451e8a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff44c0ce610 con 0x7ff44c071540 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff451e8a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44c0d2f70 con 0x7ff44c0b2b50 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff451e8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c0d4be0 con 0x7ff44c0d3400 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff44bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff44c0b2b50 0x7ff44c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff44bfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff44c0b2b50 0x7ff44c0b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37910/0 (socket says 172.21.15.27:37910) 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff44bfff640 1 -- 172.21.15.27:0/3177414342 learned_addr learned my addr 172.21.15.27:0/3177414342 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff44affd640 1 --2- 172.21.15.27:0/3177414342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 0x7ff44c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff44bfff640 1 -- 172.21.15.27:0/3177414342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 msgr2=0x7ff44c0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff44bfff640 1 --2- 172.21.15.27:0/3177414342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 0x7ff44c0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff44bfff640 1 -- 172.21.15.27:0/3177414342 >> v1:172.21.15.69:6789/0 conn(0x7ff44c071540 legacy=0x7ff44c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff44bfff640 1 -- 172.21.15.27:0/3177414342 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c0d2920 con 0x7ff44c0b2b50 2024-03-06T07:33:19.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.783+0000 7ff44a7fc640 1 -- 172.21.15.27:0/3177414342 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197063614 0 0) 0x7ff44c0ce610 con 0x7ff44c071540 2024-03-06T07:33:19.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.784+0000 7ff44bfff640 1 --2- 172.21.15.27:0/3177414342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 0x7ff44c0b6e20 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7ff444009e50 tx=0x7ff44402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa0ef90df1fd861 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.784+0000 7ff44a7fc640 1 -- 172.21.15.27:0/3177414342 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44402cb00 con 0x7ff44c0b2b50 2024-03-06T07:33:19.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.784+0000 7ff44a7fc640 1 -- 172.21.15.27:0/3177414342 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44402cca0 con 0x7ff44c0b2b50 2024-03-06T07:33:19.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.784+0000 7ff44a7fc640 1 -- 172.21.15.27:0/3177414342 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444034610 con 0x7ff44c0b2b50 2024-03-06T07:33:19.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.785+0000 7ff451e8a640 1 -- 172.21.15.27:0/3177414342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 msgr2=0x7ff44c0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.785+0000 7ff451e8a640 1 --2- 172.21.15.27:0/3177414342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 0x7ff44c0b6e20 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7ff444009e50 tx=0x7ff44402bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:19.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.785+0000 7ff451e8a640 1 -- 172.21.15.27:0/3177414342 shutdown_connections 2024-03-06T07:33:19.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.785+0000 7ff451e8a640 1 --2- 172.21.15.27:0/3177414342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 0x7ff44c0b6e20 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.785+0000 7ff451e8a640 1 --2- 172.21.15.27:0/3177414342 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 0x7ff44c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.785+0000 7ff451e8a640 1 -- 172.21.15.27:0/3177414342 >> 172.21.15.27:0/3177414342 conn(0x7ff44c0d3ca0 msgr2=0x7ff44c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.785+0000 7ff451e8a640 1 -- 172.21.15.27:0/3177414342 shutdown_connections 2024-03-06T07:33:19.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.785+0000 7ff451e8a640 1 -- 172.21.15.27:0/3177414342 wait complete. 2024-03-06T07:33:19.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.786+0000 7ff451e8a640 1 Processor -- start 2024-03-06T07:33:19.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.786+0000 7ff451e8a640 1 -- start start 2024-03-06T07:33:19.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff451e8a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff44c071540 0x7ff44c0b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff451e8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 0x7ff44c1f4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff451e8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 0x7ff44c15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff451e8a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c15e370 con 0x7ff44c0b2b50 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff451e8a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c0ce8f0 con 0x7ff44c071540 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff451e8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c1f4970 con 0x7ff44c0d3400 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 0x7ff44c1f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 0x7ff44c1f4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37922/0 (socket says 172.21.15.27:37922) 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44affd640 1 -- 172.21.15.27:0/2936068709 learned_addr learned my addr 172.21.15.27:0/2936068709 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44b7fe640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff44c071540 0x7ff44c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44bfff640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 0x7ff44c15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44affd640 1 -- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 msgr2=0x7ff44c15aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44affd640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 0x7ff44c15aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44affd640 1 -- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff44c071540 msgr2=0x7ff44c0b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44affd640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff44c071540 0x7ff44c0b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.787+0000 7ff44affd640 1 -- 172.21.15.27:0/2936068709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c0b5b00 con 0x7ff44c0b2b50 2024-03-06T07:33:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.788+0000 7ff44affd640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 0x7ff44c1f4430 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7ff44000e840 tx=0x7ff44000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.788+0000 7ff433fff640 1 -- 172.21.15.27:0/2936068709 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440017070 con 0x7ff44c0b2b50 2024-03-06T07:33:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.788+0000 7ff433fff640 1 -- 172.21.15.27:0/2936068709 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44000c8e0 con 0x7ff44c0b2b50 2024-03-06T07:33:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.788+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44c0b5de0 con 0x7ff44c0b2b50 2024-03-06T07:33:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.788+0000 7ff433fff640 1 -- 172.21.15.27:0/2936068709 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44000ccb0 con 0x7ff44c0b2b50 2024-03-06T07:33:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.788+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44c159040 con 0x7ff44c0b2b50 2024-03-06T07:33:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.789+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44c068700 con 0x7ff44c0b2b50 2024-03-06T07:33:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.789+0000 7ff433fff640 1 -- 172.21.15.27:0/2936068709 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff44001a070 con 0x7ff44c0b2b50 2024-03-06T07:33:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.790+0000 7ff433fff640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff43806b4d0 0x7ff43806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.790+0000 7ff433fff640 1 -- 172.21.15.27:0/2936068709 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff44008a2c0 con 0x7ff44c0b2b50 2024-03-06T07:33:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.790+0000 7ff44b7fe640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff43806b4d0 0x7ff43806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.791+0000 7ff44b7fe640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff43806b4d0 0x7ff43806d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff43c000f60 tx=0x7ff43c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:19.794+0000 7ff433fff640 1 -- 172.21.15.27:0/2936068709 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff440058b60 con 0x7ff44c0b2b50 2024-03-06T07:33:19.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:19.912 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:20.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.085+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff44c0cd530 con 0x7ff44c0b2b50 2024-03-06T07:33:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.086+0000 7ff433fff640 1 -- 172.21.15.27:0/2936068709 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7ff44005cc00 con 0x7ff44c0b2b50 2024-03-06T07:33:20.087 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:20.087 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:20.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff43806b4d0 msgr2=0x7ff43806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff43806b4d0 0x7ff43806d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff43c000f60 tx=0x7ff43c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:20.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 msgr2=0x7ff44c1f4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 0x7ff44c1f4430 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7ff44000e840 tx=0x7ff44000ed00 comp rx=0 tx=0).stop 2024-03-06T07:33:20.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 shutdown_connections 2024-03-06T07:33:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff43806b4d0 0x7ff43806d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff44c0d3400 0x7ff44c15aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44c0b2b50 0x7ff44c1f4430 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 --2- 172.21.15.27:0/2936068709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff44c071540 0x7ff44c0b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 >> 172.21.15.27:0/2936068709 conn(0x7ff44c0d3ca0 msgr2=0x7ff44c0b8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 shutdown_connections 2024-03-06T07:33:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:20.088+0000 7ff451e8a640 1 -- 172.21.15.27:0/2936068709 wait complete. 2024-03-06T07:33:20.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:20.302 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:21.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:21.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.408+0000 7f2a20d1e640 1 Processor -- start 2024-03-06T07:33:21.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.408+0000 7f2a20d1e640 1 -- start start 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a20d1e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a1c0b95a0 0x7f2a1c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a20d1e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a1c0b9a10 0x7f2a1c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a20d1e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0cea30 con 0x7f2a1c15df00 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a20d1e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1c0cf3c0 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a20d1e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1c0cf0e0 con 0x7f2a1c0b9a10 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a19d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a1c0b95a0 0x7f2a1c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a19d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a1c0b95a0 0x7f2a1c0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37926/0 (socket says 172.21.15.27:37926) 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a19d9b640 1 -- 172.21.15.27:0/2025381928 learned_addr learned my addr 172.21.15.27:0/2025381928 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a1ad9d640 1 --2- 172.21.15.27:0/2025381928 >> v2:172.21.15.69:3300/0 conn(0x7f2a1c0b9a10 0x7f2a1c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a19d9b640 1 -- 172.21.15.27:0/2025381928 >> v2:172.21.15.69:3300/0 conn(0x7f2a1c0b9a10 msgr2=0x7f2a1c0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a19d9b640 1 --2- 172.21.15.27:0/2025381928 >> v2:172.21.15.69:3300/0 conn(0x7f2a1c0b9a10 0x7f2a1c0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a19d9b640 1 -- 172.21.15.27:0/2025381928 >> v1:172.21.15.69:6789/0 conn(0x7f2a1c15df00 legacy=0x7f2a1c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.409+0000 7f2a19d9b640 1 -- 172.21.15.27:0/2025381928 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c0ce470 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.410+0000 7f2a19d9b640 1 --2- 172.21.15.27:0/2025381928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 0x7f2a1c0b5700 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c0d4d20 tx=0x7f2a1002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11bb7a988fedcd6c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.410+0000 7f2a1959a640 1 -- 172.21.15.27:0/2025381928 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10034070 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.410+0000 7f2a1959a640 1 -- 172.21.15.27:0/2025381928 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a10030860 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.410+0000 7f2a1959a640 1 -- 172.21.15.27:0/2025381928 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10030c30 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.411+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2025381928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 msgr2=0x7f2a1c0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.411+0000 7f2a20d1e640 1 --2- 172.21.15.27:0/2025381928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 0x7f2a1c0b5700 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c0d4d20 tx=0x7f2a1002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.411+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2025381928 shutdown_connections 2024-03-06T07:33:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.411+0000 7f2a20d1e640 1 --2- 172.21.15.27:0/2025381928 >> v2:172.21.15.69:3300/0 conn(0x7f2a1c0b9a10 0x7f2a1c0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.411+0000 7f2a20d1e640 1 --2- 172.21.15.27:0/2025381928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 0x7f2a1c0b5700 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.411+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2025381928 >> 172.21.15.27:0/2025381928 conn(0x7f2a1c071960 msgr2=0x7f2a1c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.411+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2025381928 shutdown_connections 2024-03-06T07:33:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.411+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2025381928 wait complete. 2024-03-06T07:33:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.412+0000 7f2a20d1e640 1 Processor -- start 2024-03-06T07:33:21.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.412+0000 7f2a20d1e640 1 -- start start 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a20d1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 0x7f2a1c155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a20d1e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1c0b9a10 0x7f2a1c0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a20d1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a1c15df00 0x7f2a1c1618b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a20d1e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0cedf0 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a20d1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0cec70 con 0x7f2a1c15df00 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a20d1e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0d3220 con 0x7f2a1c0b9a10 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 0x7f2a1c155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 0x7f2a1c155230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37938/0 (socket says 172.21.15.27:37938) 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1a59c640 1 -- 172.21.15.27:0/2163850964 learned_addr learned my addr 172.21.15.27:0/2163850964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a19d9b640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1c0b9a10 0x7f2a1c0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1ad9d640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a1c15df00 0x7f2a1c1618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1a59c640 1 -- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1c0b9a10 msgr2=0x7f2a1c0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1a59c640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1c0b9a10 0x7f2a1c0cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1a59c640 1 -- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a1c15df00 msgr2=0x7f2a1c1618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1a59c640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a1c15df00 0x7f2a1c1618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1a59c640 1 -- 172.21.15.27:0/2163850964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c155770 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a1a59c640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 0x7f2a1c155230 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00c3c0 tx=0x7f2a0c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.413+0000 7f2a0affd640 1 -- 172.21.15.27:0/2163850964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c0052d0 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.414+0000 7f2a0affd640 1 -- 172.21.15.27:0/2163850964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0c005490 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.414+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c0cb7f0 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.414+0000 7f2a0affd640 1 -- 172.21.15.27:0/2163850964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c007260 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.414+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c0cc0d0 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.415+0000 7f2a0affd640 1 -- 172.21.15.27:0/2163850964 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a0c007400 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.415+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1c067ff0 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.415+0000 7f2a0affd640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f29fc06b500 0x7f29fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.415+0000 7f2a0affd640 1 -- 172.21.15.27:0/2163850964 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2a0c0896c0 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.416+0000 7f2a19d9b640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f29fc06b500 0x7f29fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.416+0000 7f2a19d9b640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f29fc06b500 0x7f29fc06d9c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2a10036000 tx=0x7f2a1003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.420+0000 7f2a0affd640 1 -- 172.21.15.27:0/2163850964 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a1c155770 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.806+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a1c0b7d40 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.807+0000 7f2a0affd640 1 -- 172.21.15.27:0/2163850964 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f2a0c058510 con 0x7f2a1c0b95a0 2024-03-06T07:33:21.808 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:21.809 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.808+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f29fc06b500 msgr2=0x7f29fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.808+0000 7f2a20d1e640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f29fc06b500 0x7f29fc06d9c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2a10036000 tx=0x7f2a1003b040 comp rx=0 tx=0).stop 2024-03-06T07:33:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 msgr2=0x7f2a1c155230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 0x7f2a1c155230 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00c3c0 tx=0x7f2a0c00c880 comp rx=0 tx=0).stop 2024-03-06T07:33:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 shutdown_connections 2024-03-06T07:33:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f29fc06b500 0x7f29fc06d9c0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a1c15df00 0x7f2a1c1618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2a1c0b9a10 0x7f2a1c0cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 --2- 172.21.15.27:0/2163850964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1c0b95a0 0x7f2a1c155230 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 >> 172.21.15.27:0/2163850964 conn(0x7f2a1c071960 msgr2=0x7f2a1c15c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 shutdown_connections 2024-03-06T07:33:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:21.809+0000 7f2a20d1e640 1 -- 172.21.15.27:0/2163850964 wait complete. 2024-03-06T07:33:22.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:22.022 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:23.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:23.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.129+0000 7f35910f4640 1 Processor -- start 2024-03-06T07:33:23.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.129+0000 7f35910f4640 1 -- start start 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.129+0000 7f35910f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f358c15df00 0x7f358c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.129+0000 7f35910f4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358c0cea30 con 0x7f358c0b9a30 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.129+0000 7f35910f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358c0d2af0 con 0x7f358c0b95a0 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.129+0000 7f35910f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358c0cf3c0 con 0x7f358c15df00 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f358b59e640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f358c0b9a30 0x7f358c0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60232/0 (socket says 172.21.15.27:60232) 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f358b59e640 1 -- 172.21.15.27:0/69256288 learned_addr learned my addr 172.21.15.27:0/69256288 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f358ad9d640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f358c15df00 0x7f358c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96245875 0 0) 0x7f358c0cea30 con 0x7f358c0b9a30 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3574003520 con 0x7f358c0b9a30 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310636006 0 0) 0x7f358c0d2af0 con 0x7f358c0b95a0 2024-03-06T07:33:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358c0cea30 con 0x7f358c0b95a0 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3442060468 0 0) 0x7f3574003520 con 0x7f358c0b9a30 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f358c15df00 msgr2=0x7f358c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f358c15df00 0x7f358c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 >> v1:172.21.15.69:6789/0 conn(0x7f358c0b95a0 legacy=0x7f358c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0cf0e0 con 0x7f358c0b9a30 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f35840019f0 con 0x7f358c0b9a30 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 >> v1:172.21.15.27:6789/0 conn(0x7f358c0b9a30 legacy=0x7f358c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3574003b70 0x7f3574003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3574004800 0x7f3574004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f358c15df00 0x7f35740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c0cf3c0 con 0x7f3574004800 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357400ba70 con 0x7f3574003b70 2024-03-06T07:33:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357400bc40 con 0x7f358c15df00 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f358a59c640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3574004800 0x7f3574004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.130+0000 7f358ad9d640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3574003b70 0x7f3574003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.131+0000 7f358b59e640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f358c15df00 0x7f35740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.131+0000 7f358a59c640 1 -- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f358c15df00 msgr2=0x7f35740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.131+0000 7f358a59c640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f358c15df00 0x7f35740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.131+0000 7f358a59c640 1 -- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3574003b70 msgr2=0x7f3574003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.131+0000 7f358a59c640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3574003b70 0x7f3574003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.131+0000 7f358a59c640 1 -- 172.21.15.27:0/69256288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3574003520 con 0x7f3574004800 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.131+0000 7f358a59c640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3574004800 0x7f3574004c80 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f35800018d0 tx=0x7f3574005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9dcd67d9bd6f5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35800014f0 con 0x7f3574004800 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358002f050 con 0x7f3574004800 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f3589d9b640 1 -- 172.21.15.27:0/69256288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35800035f0 con 0x7f3574004800 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f35910f4640 1 -- 172.21.15.27:0/69256288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3574004800 msgr2=0x7f3574004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f35910f4640 1 --2- 172.21.15.27:0/69256288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3574004800 0x7f3574004c80 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f35800018d0 tx=0x7f3574005830 comp rx=0 tx=0).stop 2024-03-06T07:33:23.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f358b59e640 1 -- 172.21.15.27:0/69256288 reap_dead start 2024-03-06T07:33:23.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f35910f4640 1 -- 172.21.15.27:0/69256288 shutdown_connections 2024-03-06T07:33:23.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f35910f4640 1 -- 172.21.15.27:0/69256288 >> 172.21.15.27:0/69256288 conn(0x7f358c071960 msgr2=0x7f358c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f35910f4640 1 -- 172.21.15.27:0/69256288 shutdown_connections 2024-03-06T07:33:23.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.132+0000 7f35910f4640 1 -- 172.21.15.27:0/69256288 wait complete. 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.134+0000 7f35910f4640 1 Processor -- start 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.134+0000 7f35910f4640 1 -- start start 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.134+0000 7f35910f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3574004800 0x7f358c15ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.134+0000 7f35910f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f358c14a210 0x7f358c0cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.134+0000 7f35910f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f358c1605b0 0x7f358c160a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.134+0000 7f35910f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c0cec70 con 0x7f358c1605b0 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.134+0000 7f35910f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c0ce750 con 0x7f358c14a210 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.134+0000 7f35910f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358c0cedf0 con 0x7f3574004800 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.134+0000 7f358b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f358c1605b0 0x7f358c160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f358c1605b0 0x7f358c160a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37956/0 (socket says 172.21.15.27:37956) 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358b59e640 1 -- 172.21.15.27:0/1218364931 learned_addr learned my addr 172.21.15.27:0/1218364931 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358ad9d640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3574004800 0x7f358c15ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358a59c640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f358c14a210 0x7f358c0cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358b59e640 1 -- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3574004800 msgr2=0x7f358c15ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358b59e640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3574004800 0x7f358c15ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358b59e640 1 -- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f358c14a210 msgr2=0x7f358c0cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358b59e640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f358c14a210 0x7f358c0cbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358b59e640 1 -- 172.21.15.27:0/1218364931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0cd740 con 0x7f358c1605b0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f358b59e640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f358c1605b0 0x7f358c160a30 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f3584002230 tx=0x7f3584007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f35737fe640 1 -- 172.21.15.27:0/1218364931 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358400ab10 con 0x7f358c1605b0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f35737fe640 1 -- 172.21.15.27:0/1218364931 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3584018070 con 0x7f358c1605b0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358c0cc6a0 con 0x7f358c1605b0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f35737fe640 1 -- 172.21.15.27:0/1218364931 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35840133f0 con 0x7f358c1605b0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.135+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358c0ccc60 con 0x7f358c1605b0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.136+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358c067ff0 con 0x7f358c1605b0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.137+0000 7f35737fe640 1 -- 172.21.15.27:0/1218364931 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3584013590 con 0x7f358c1605b0 2024-03-06T07:33:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.138+0000 7f35737fe640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357806b500 0x7f357806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.138+0000 7f35737fe640 1 -- 172.21.15.27:0/1218364931 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f358408a6e0 con 0x7f358c1605b0 2024-03-06T07:33:23.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.138+0000 7f358ad9d640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357806b500 0x7f357806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.139+0000 7f358ad9d640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357806b500 0x7f357806d9c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f357c009ae0 tx=0x7f357c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.142+0000 7f35737fe640 1 -- 172.21.15.27:0/1218364931 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3584059540 con 0x7f358c1605b0 2024-03-06T07:33:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.361+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f358c0b6080 con 0x7f358c1605b0 2024-03-06T07:33:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.361+0000 7f35737fe640 1 -- 172.21.15.27:0/1218364931 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f3584058e40 con 0x7f358c1605b0 2024-03-06T07:33:23.365 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:23.365 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357806b500 msgr2=0x7f357806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357806b500 0x7f357806d9c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f357c009ae0 tx=0x7f357c009190 comp rx=0 tx=0).stop 2024-03-06T07:33:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f358c1605b0 msgr2=0x7f358c160a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f358c1605b0 0x7f358c160a30 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f3584002230 tx=0x7f3584007b10 comp rx=0 tx=0).stop 2024-03-06T07:33:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 shutdown_connections 2024-03-06T07:33:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357806b500 0x7f357806d9c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f358c1605b0 0x7f358c160a30 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f358c14a210 0x7f358c0cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 --2- 172.21.15.27:0/1218364931 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3574004800 0x7f358c15ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 >> 172.21.15.27:0/1218364931 conn(0x7f358c071960 msgr2=0x7f358c0b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 shutdown_connections 2024-03-06T07:33:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:23.363+0000 7f35910f4640 1 -- 172.21.15.27:0/1218364931 wait complete. 2024-03-06T07:33:23.578 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:23.578 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:24.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.684+0000 7fbf5e37a640 1 Processor -- start 2024-03-06T07:33:24.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.685+0000 7fbf5e37a640 1 -- start start 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.685+0000 7fbf5e37a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf580d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.685+0000 7fbf5e37a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf580cea30 con 0x7fbf580b95a0 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.685+0000 7fbf5e37a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf580d2af0 con 0x7fbf580b9a30 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.685+0000 7fbf5e37a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580cf3c0 con 0x7fbf5815df00 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.685+0000 7fbf577fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbf580b95a0 0x7fbf580b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51996/0 (socket says 172.21.15.27:51996) 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.685+0000 7fbf577fe640 1 -- 172.21.15.27:0/1112442430 learned_addr learned my addr 172.21.15.27:0/1112442430 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.686+0000 7fbf57fff640 1 --2- 172.21.15.27:0/1112442430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf580d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.686+0000 7fbf56ffd640 1 -- 172.21.15.27:0/1112442430 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187572238 0 0) 0x7fbf580cea30 con 0x7fbf580b95a0 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.686+0000 7fbf56ffd640 1 -- 172.21.15.27:0/1112442430 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf40003520 con 0x7fbf580b95a0 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.686+0000 7fbf56ffd640 1 -- 172.21.15.27:0/1112442430 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170189810 0 0) 0x7fbf580d2af0 con 0x7fbf580b9a30 2024-03-06T07:33:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.686+0000 7fbf56ffd640 1 -- 172.21.15.27:0/1112442430 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf580cea30 con 0x7fbf580b9a30 2024-03-06T07:33:24.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.686+0000 7fbf56ffd640 1 -- 172.21.15.27:0/1112442430 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 420756875 0 0) 0x7fbf40003520 con 0x7fbf580b95a0 2024-03-06T07:33:24.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.686+0000 7fbf57fff640 1 -- 172.21.15.27:0/1112442430 >> v1:172.21.15.69:6789/0 conn(0x7fbf580b9a30 legacy=0x7fbf580d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.686+0000 7fbf57fff640 1 -- 172.21.15.27:0/1112442430 >> v1:172.21.15.27:6789/0 conn(0x7fbf580b95a0 legacy=0x7fbf580b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.686+0000 7fbf57fff640 1 -- 172.21.15.27:0/1112442430 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf580cf0e0 con 0x7fbf5815df00 2024-03-06T07:33:24.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.687+0000 7fbf57fff640 1 --2- 172.21.15.27:0/1112442430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf580d25b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbf480097d0 tx=0x7fbf4802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b2c8252018ffc01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf56ffd640 1 -- 172.21.15.27:0/1112442430 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48037070 con 0x7fbf5815df00 2024-03-06T07:33:24.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf56ffd640 1 -- 172.21.15.27:0/1112442430 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf48030510 con 0x7fbf5815df00 2024-03-06T07:33:24.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf56ffd640 1 -- 172.21.15.27:0/1112442430 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c001670 con 0x7fbf5815df00 2024-03-06T07:33:24.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf5e37a640 1 -- 172.21.15.27:0/1112442430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 msgr2=0x7fbf580d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf5e37a640 1 --2- 172.21.15.27:0/1112442430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf580d25b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbf480097d0 tx=0x7fbf4802b800 comp rx=0 tx=0).stop 2024-03-06T07:33:24.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf5e37a640 1 -- 172.21.15.27:0/1112442430 shutdown_connections 2024-03-06T07:33:24.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf5e37a640 1 --2- 172.21.15.27:0/1112442430 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf580d25b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf5e37a640 1 -- 172.21.15.27:0/1112442430 >> 172.21.15.27:0/1112442430 conn(0x7fbf58071960 msgr2=0x7fbf58071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf5e37a640 1 -- 172.21.15.27:0/1112442430 shutdown_connections 2024-03-06T07:33:24.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.688+0000 7fbf5e37a640 1 -- 172.21.15.27:0/1112442430 wait complete. 2024-03-06T07:33:24.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.689+0000 7fbf5e37a640 1 Processor -- start 2024-03-06T07:33:24.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5e37a640 1 -- start start 2024-03-06T07:33:24.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5e37a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf580b95a0 0x7fbf5815ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5e37a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf58155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5e37a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf580cca10 0x7fbf5815a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5e37a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580cec70 con 0x7fbf580cca10 2024-03-06T07:33:24.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5e37a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580ce750 con 0x7fbf580b95a0 2024-03-06T07:33:24.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5e37a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580cedf0 con 0x7fbf5815df00 2024-03-06T07:33:24.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5c917640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf580cca10 0x7fbf5815a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5c917640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf580cca10 0x7fbf5815a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56570/0 (socket says 172.21.15.27:56570) 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf5c917640 1 -- 172.21.15.27:0/31303910 learned_addr learned my addr 172.21.15.27:0/31303910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf577fe640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf58155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.690+0000 7fbf57fff640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf580b95a0 0x7fbf5815ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf5c917640 1 -- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 msgr2=0x7fbf58155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf5c917640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf58155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf5c917640 1 -- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf580b95a0 msgr2=0x7fbf5815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf5c917640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf580b95a0 0x7fbf5815ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf5c917640 1 -- 172.21.15.27:0/31303910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5815f2b0 con 0x7fbf580cca10 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf577fe640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf58155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf5c917640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf580cca10 0x7fbf5815a6c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000c7d0 tx=0x7fbf5000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf54ff9640 1 -- 172.21.15.27:0/31303910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50003b70 con 0x7fbf580cca10 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf54ff9640 1 -- 172.21.15.27:0/31303910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf50003d10 con 0x7fbf580cca10 2024-03-06T07:33:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5815f590 con 0x7fbf580cca10 2024-03-06T07:33:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf54ff9640 1 -- 172.21.15.27:0/31303910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5000aa30 con 0x7fbf580cca10 2024-03-06T07:33:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.691+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5815fb50 con 0x7fbf580cca10 2024-03-06T07:33:24.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.692+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf58067ff0 con 0x7fbf580cca10 2024-03-06T07:33:24.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.693+0000 7fbf54ff9640 1 -- 172.21.15.27:0/31303910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf5000abd0 con 0x7fbf580cca10 2024-03-06T07:33:24.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.693+0000 7fbf54ff9640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf4406b500 0x7fbf4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.693+0000 7fbf54ff9640 1 -- 172.21.15.27:0/31303910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbf50089f50 con 0x7fbf580cca10 2024-03-06T07:33:24.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.693+0000 7fbf57fff640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf4406b500 0x7fbf4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.694+0000 7fbf57fff640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf4406b500 0x7fbf4406d9c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbf48034000 tx=0x7fbf48008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.698+0000 7fbf54ff9640 1 -- 172.21.15.27:0/31303910 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf50058e40 con 0x7fbf580cca10 2024-03-06T07:33:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.917+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf580b2bd0 con 0x7fbf580cca10 2024-03-06T07:33:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.918+0000 7fbf54ff9640 1 -- 172.21.15.27:0/31303910 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fbf50058740 con 0x7fbf580cca10 2024-03-06T07:33:24.920 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:24.921 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:24.921 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf4406b500 msgr2=0x7fbf4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf4406b500 0x7fbf4406d9c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbf48034000 tx=0x7fbf48008030 comp rx=0 tx=0).stop 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf580cca10 msgr2=0x7fbf5815a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf580cca10 0x7fbf5815a6c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000c7d0 tx=0x7fbf5000cc90 comp rx=0 tx=0).stop 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 shutdown_connections 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf4406b500 0x7fbf4406d9c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf580cca10 0x7fbf5815a6c0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf5815df00 0x7fbf58155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 --2- 172.21.15.27:0/31303910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf580b95a0 0x7fbf5815ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 >> 172.21.15.27:0/31303910 conn(0x7fbf58071960 msgr2=0x7fbf581618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 shutdown_connections 2024-03-06T07:33:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:24.920+0000 7fbf5e37a640 1 -- 172.21.15.27:0/31303910 wait complete. 2024-03-06T07:33:25.134 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:25.135 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:26.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.240+0000 7f9b51050640 1 Processor -- start 2024-03-06T07:33:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.240+0000 7f9b51050640 1 -- start start 2024-03-06T07:33:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.241+0000 7f9b51050640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c15be60 0x7f9b4c0719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.241+0000 7f9b51050640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b4c15fcc0 con 0x7f9b4c15f740 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.241+0000 7f9b51050640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b4c15ffa0 con 0x7f9b4c066d00 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.241+0000 7f9b51050640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c160280 con 0x7f9b4c15be60 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.241+0000 7f9b4a59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9b4c15f740 0x7f9b4c06fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52000/0 (socket says 172.21.15.27:52000) 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.241+0000 7f9b4a59c640 1 -- 172.21.15.27:0/3820844136 learned_addr learned my addr 172.21.15.27:0/3820844136 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b4b59e640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c15be60 0x7f9b4c0719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836130097 0 0) 0x7f9b4c15fcc0 con 0x7f9b4c15f740 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b34003520 con 0x7f9b4c15f740 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1229859849 0 0) 0x7f9b34003520 con 0x7f9b4c15f740 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c15be60 msgr2=0x7f9b4c0719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c15be60 0x7f9b4c0719b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 >> v1:172.21.15.69:6789/0 conn(0x7f9b4c066d00 legacy=0x7f9b4c0670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c160560 con 0x7f9b4c15f740 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f9b40001670 con 0x7f9b4c15f740 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 >> v1:172.21.15.27:6789/0 conn(0x7f9b4c15f740 legacy=0x7f9b4c06fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b340039e0 0x7f9b34003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c15be60 0x7f9b34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34004d40 0x7f9b340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c160280 con 0x7f9b34004d40 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3400baa0 con 0x7f9b340039e0 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3400bca0 con 0x7f9b4c15be60 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b4b59e640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34004d40 0x7f9b340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b4ad9d640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b340039e0 0x7f9b34003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.242+0000 7f9b4a59c640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c15be60 0x7f9b34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.243+0000 7f9b4b59e640 1 -- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c15be60 msgr2=0x7f9b34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.243+0000 7f9b4b59e640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c15be60 0x7f9b34004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.243+0000 7f9b4b59e640 1 -- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b340039e0 msgr2=0x7f9b34003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.243+0000 7f9b4b59e640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b340039e0 0x7f9b34003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.243+0000 7f9b4b59e640 1 -- 172.21.15.27:0/3820844136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b34003520 con 0x7f9b34004d40 2024-03-06T07:33:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.243+0000 7f9b4b59e640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34004d40 0x7f9b340051c0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009f90 tx=0x7f9b34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53dfb04435dd5def server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.243+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b44039070 con 0x7f9b34004d40 2024-03-06T07:33:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.243+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4402cbd0 con 0x7f9b34004d40 2024-03-06T07:33:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.243+0000 7f9b49d9b640 1 -- 172.21.15.27:0/3820844136 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b44009c10 con 0x7f9b34004d40 2024-03-06T07:33:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.244+0000 7f9b51050640 1 -- 172.21.15.27:0/3820844136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34004d40 msgr2=0x7f9b340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.244+0000 7f9b51050640 1 --2- 172.21.15.27:0/3820844136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34004d40 0x7f9b340051c0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009f90 tx=0x7f9b34005830 comp rx=0 tx=0).stop 2024-03-06T07:33:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.244+0000 7f9b4b59e640 1 -- 172.21.15.27:0/3820844136 reap_dead start 2024-03-06T07:33:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.244+0000 7f9b51050640 1 -- 172.21.15.27:0/3820844136 shutdown_connections 2024-03-06T07:33:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.244+0000 7f9b51050640 1 -- 172.21.15.27:0/3820844136 >> 172.21.15.27:0/3820844136 conn(0x7f9b4c0d0990 msgr2=0x7f9b4c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.244+0000 7f9b51050640 1 -- 172.21.15.27:0/3820844136 shutdown_connections 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.244+0000 7f9b51050640 1 -- 172.21.15.27:0/3820844136 wait complete. 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.245+0000 7f9b51050640 1 Processor -- start 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.245+0000 7f9b51050640 1 -- start start 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b51050640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4c1f4ad0 0x7f9b4c1e5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b51050640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c1f4eb0 0x7f9b4c1e7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b51050640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b4c153bf0 0x7f9b4c1e79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b51050640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c071d80 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b51050640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c071c00 con 0x7f9b4c153bf0 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b51050640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c071f00 con 0x7f9b4c1f4eb0 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b4ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4c1f4ad0 0x7f9b4c1e5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b4ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4c1f4ad0 0x7f9b4c1e5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56588/0 (socket says 172.21.15.27:56588) 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b4ad9d640 1 -- 172.21.15.27:0/2994538771 learned_addr learned my addr 172.21.15.27:0/2994538771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b4a59c640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c1f4eb0 0x7f9b4c1e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.246+0000 7f9b4b59e640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b4c153bf0 0x7f9b4c1e79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b4ad9d640 1 -- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c1f4eb0 msgr2=0x7f9b4c1e7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b4ad9d640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c1f4eb0 0x7f9b4c1e7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b4ad9d640 1 -- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b4c153bf0 msgr2=0x7f9b4c1e79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b4ad9d640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b4c153bf0 0x7f9b4c1e79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b4ad9d640 1 -- 172.21.15.27:0/2994538771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c1e6840 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b4ad9d640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4c1f4ad0 0x7f9b4c1e5f60 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00f820 tx=0x7f9b3c00fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b337fe640 1 -- 172.21.15.27:0/2994538771 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0050d0 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b337fe640 1 -- 172.21.15.27:0/2994538771 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c018070 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4c151140 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b337fe640 1 -- 172.21.15.27:0/2994538771 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c00d730 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.247+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4c151700 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.248+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4c068700 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.249+0000 7f9b337fe640 1 -- 172.21.15.27:0/2994538771 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b3c00d8d0 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.249+0000 7f9b337fe640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9b3806b4d0 0x7f9b3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.249+0000 7f9b337fe640 1 -- 172.21.15.27:0/2994538771 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9b3c089f70 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.250+0000 7f9b4a59c640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9b3806b4d0 0x7f9b3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.250+0000 7f9b4a59c640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9b3806b4d0 0x7f9b3806d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9b400017e0 tx=0x7f9b400015d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.253+0000 7f9b337fe640 1 -- 172.21.15.27:0/2994538771 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b3c058dd0 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.564+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b4c06dc60 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.565+0000 7f9b337fe640 1 -- 172.21.15.27:0/2994538771 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f9b3c0586d0 con 0x7f9b4c1f4ad0 2024-03-06T07:33:26.567 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:26.568 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:26.568 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9b3806b4d0 msgr2=0x7f9b3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9b3806b4d0 0x7f9b3806d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9b400017e0 tx=0x7f9b400015d0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4c1f4ad0 msgr2=0x7f9b4c1e5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4c1f4ad0 0x7f9b4c1e5f60 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00f820 tx=0x7f9b3c00fc20 comp rx=0 tx=0).stop 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 shutdown_connections 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9b3806b4d0 0x7f9b3806d990 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b4c153bf0 0x7f9b4c1e79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9b4c1f4eb0 0x7f9b4c1e7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 --2- 172.21.15.27:0/2994538771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4c1f4ad0 0x7f9b4c1e5f60 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 >> 172.21.15.27:0/2994538771 conn(0x7f9b4c0d0990 msgr2=0x7f9b4c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 shutdown_connections 2024-03-06T07:33:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:26.568+0000 7f9b51050640 1 -- 172.21.15.27:0/2994538771 wait complete. 2024-03-06T07:33:26.783 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:26.783 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:27.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:27.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.894+0000 7fd702eed640 1 Processor -- start 2024-03-06T07:33:27.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.894+0000 7fd702eed640 1 -- start start 2024-03-06T07:33:27.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.894+0000 7fd702eed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15df00 0x7fd6fc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.894+0000 7fd702eed640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd6fc0b95a0 0x7fd6fc0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.894+0000 7fd702eed640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc0cea30 con 0x7fd6fc0b9a30 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.894+0000 7fd702eed640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6fc0d2af0 con 0x7fd6fc0b95a0 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.894+0000 7fd702eed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0cf3c0 con 0x7fd6fc15df00 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.895+0000 7fd6fbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd6fc0b95a0 0x7fd6fc0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.895+0000 7fd6fbfff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd6fc0b95a0 0x7fd6fc0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56594/0 (socket says 172.21.15.27:56594) 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.895+0000 7fd6fbfff640 1 -- 172.21.15.27:0/3234836598 learned_addr learned my addr 172.21.15.27:0/3234836598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.895+0000 7fd700c89640 1 --2- 172.21.15.27:0/3234836598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15df00 0x7fd6fc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.895+0000 7fd6fbfff640 1 -- 172.21.15.27:0/3234836598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15df00 msgr2=0x7fd6fc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.895+0000 7fd6fbfff640 1 --2- 172.21.15.27:0/3234836598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15df00 0x7fd6fc0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.895+0000 7fd6fbfff640 1 -- 172.21.15.27:0/3234836598 >> v1:172.21.15.69:6789/0 conn(0x7fd6fc0b9a30 legacy=0x7fd6fc0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.895+0000 7fd6fbfff640 1 -- 172.21.15.27:0/3234836598 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc0cf0e0 con 0x7fd6fc0b95a0 2024-03-06T07:33:27.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd6fbfff640 1 --2- 172.21.15.27:0/3234836598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc0b95a0 0x7fd6fc0b5700 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009870 tx=0x7fd6e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7c5eb154698aea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd6fb7fe640 1 -- 172.21.15.27:0/3234836598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8034070 con 0x7fd6fc0b95a0 2024-03-06T07:33:27.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd6fb7fe640 1 -- 172.21.15.27:0/3234836598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e8030840 con 0x7fd6fc0b95a0 2024-03-06T07:33:27.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd6fb7fe640 1 -- 172.21.15.27:0/3234836598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8030c10 con 0x7fd6fc0b95a0 2024-03-06T07:33:27.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd702eed640 1 -- 172.21.15.27:0/3234836598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc0b95a0 msgr2=0x7fd6fc0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd702eed640 1 --2- 172.21.15.27:0/3234836598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc0b95a0 0x7fd6fc0b5700 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009870 tx=0x7fd6e802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:27.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd702eed640 1 -- 172.21.15.27:0/3234836598 shutdown_connections 2024-03-06T07:33:27.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd702eed640 1 --2- 172.21.15.27:0/3234836598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc0b95a0 0x7fd6fc0b5700 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd702eed640 1 --2- 172.21.15.27:0/3234836598 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc15df00 0x7fd6fc0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd702eed640 1 -- 172.21.15.27:0/3234836598 >> 172.21.15.27:0/3234836598 conn(0x7fd6fc071960 msgr2=0x7fd6fc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd702eed640 1 -- 172.21.15.27:0/3234836598 shutdown_connections 2024-03-06T07:33:27.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.896+0000 7fd702eed640 1 -- 172.21.15.27:0/3234836598 wait complete. 2024-03-06T07:33:27.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.898+0000 7fd702eed640 1 Processor -- start 2024-03-06T07:33:27.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.898+0000 7fd702eed640 1 -- start start 2024-03-06T07:33:27.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd702eed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0b95a0 0x7fd6fc160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd702eed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15df00 0x7fd6fc1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd702eed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc15f190 0x7fd6fc152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd702eed640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0cec70 con 0x7fd6fc15df00 2024-03-06T07:33:27.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd702eed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0ce750 con 0x7fd6fc15f190 2024-03-06T07:33:27.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd702eed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0cedf0 con 0x7fd6fc0b95a0 2024-03-06T07:33:27.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15df00 0x7fd6fc1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15df00 0x7fd6fc1554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56604/0 (socket says 172.21.15.27:56604) 2024-03-06T07:33:27.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6fbfff640 1 -- 172.21.15.27:0/1268814782 learned_addr learned my addr 172.21.15.27:0/1268814782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd700c89640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0b95a0 0x7fd6fc160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd70148a640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc15f190 0x7fd6fc152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6fbfff640 1 -- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0b95a0 msgr2=0x7fd6fc160010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6fbfff640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0b95a0 0x7fd6fc160010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6fbfff640 1 -- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc15f190 msgr2=0x7fd6fc152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6fbfff640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc15f190 0x7fd6fc152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6fbfff640 1 -- 172.21.15.27:0/1268814782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc155c20 con 0x7fd6fc15df00 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6fbfff640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15df00 0x7fd6fc1554b0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8000c00 tx=0x7fd6e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6f97fa640 1 -- 172.21.15.27:0/1268814782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8048070 con 0x7fd6fc15df00 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6f97fa640 1 -- 172.21.15.27:0/1268814782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e8030640 con 0x7fd6fc15df00 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd6f97fa640 1 -- 172.21.15.27:0/1268814782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8034040 con 0x7fd6fc15df00 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.899+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc15e410 con 0x7fd6fc15df00 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.900+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc15e990 con 0x7fd6fc15df00 2024-03-06T07:33:27.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.901+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc067ff0 con 0x7fd6fc15df00 2024-03-06T07:33:27.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.902+0000 7fd6f97fa640 1 -- 172.21.15.27:0/1268814782 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6e804b070 con 0x7fd6fc15df00 2024-03-06T07:33:27.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.902+0000 7fd6f97fa640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6e006b4d0 0x7fd6e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.902+0000 7fd6f97fa640 1 -- 172.21.15.27:0/1268814782 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd6e80ac060 con 0x7fd6fc15df00 2024-03-06T07:33:27.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.903+0000 7fd700c89640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6e006b4d0 0x7fd6e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.903+0000 7fd700c89640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6e006b4d0 0x7fd6e006d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009a30 tx=0x7fd6f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:27.907+0000 7fd6f97fa640 1 -- 172.21.15.27:0/1268814782 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6e807b070 con 0x7fd6fc15df00 2024-03-06T07:33:28.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.142+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6fc0b72a0 con 0x7fd6fc15df00 2024-03-06T07:33:28.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.142+0000 7fd6f97fa640 1 -- 172.21.15.27:0/1268814782 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd6e8030810 con 0x7fd6fc15df00 2024-03-06T07:33:28.147 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:28.148 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:28.152 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.144+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6e006b4d0 msgr2=0x7fd6e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.144+0000 7fd702eed640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6e006b4d0 0x7fd6e006d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009a30 tx=0x7fd6f0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.144+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15df00 msgr2=0x7fd6fc1554b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.144+0000 7fd702eed640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15df00 0x7fd6fc1554b0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8000c00 tx=0x7fd6e8004730 comp rx=0 tx=0).stop 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.145+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 shutdown_connections 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.145+0000 7fd702eed640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd6e006b4d0 0x7fd6e006d990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.145+0000 7fd702eed640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6fc15f190 0x7fd6fc152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.145+0000 7fd702eed640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6fc15df00 0x7fd6fc1554b0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.145+0000 7fd702eed640 1 --2- 172.21.15.27:0/1268814782 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6fc0b95a0 0x7fd6fc160010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.145+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 >> 172.21.15.27:0/1268814782 conn(0x7fd6fc071960 msgr2=0x7fd6fc1618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.145+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 shutdown_connections 2024-03-06T07:33:28.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:28.145+0000 7fd702eed640 1 -- 172.21.15.27:0/1268814782 wait complete. 2024-03-06T07:33:28.359 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:28.360 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:29.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:29.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.468+0000 7f0555e47640 1 Processor -- start 2024-03-06T07:33:29.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.468+0000 7f0555e47640 1 -- start start 2024-03-06T07:33:29.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.469+0000 7f0555e47640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 0x7f05500b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.469+0000 7f0555e47640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f05500b9a10 0x7f05500b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.469+0000 7f0555e47640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05500cea30 con 0x7f055015df00 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.469+0000 7f0555e47640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05500cf3c0 con 0x7f05500b9a10 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.469+0000 7f0555e47640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05500cf0e0 con 0x7f05500b95a0 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.469+0000 7f054f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f055015df00 0x7f05500d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52002/0 (socket says 172.21.15.27:52002) 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.469+0000 7f054f7fe640 1 -- 172.21.15.27:0/2186345873 learned_addr learned my addr 172.21.15.27:0/2186345873 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.470+0000 7f054ffff640 1 --2- 172.21.15.27:0/2186345873 >> v2:172.21.15.69:3300/0 conn(0x7f05500b9a10 0x7f05500b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.470+0000 7f054effd640 1 --2- 172.21.15.27:0/2186345873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 0x7f05500b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.470+0000 7f054e7fc640 1 -- 172.21.15.27:0/2186345873 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969635323 0 0) 0x7f05500cea30 con 0x7f055015df00 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.470+0000 7f054e7fc640 1 -- 172.21.15.27:0/2186345873 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053c003520 con 0x7f055015df00 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.470+0000 7f054ffff640 1 -- 172.21.15.27:0/2186345873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 msgr2=0x7f05500b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.470+0000 7f054ffff640 1 --2- 172.21.15.27:0/2186345873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 0x7f05500b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.470+0000 7f054ffff640 1 -- 172.21.15.27:0/2186345873 >> v1:172.21.15.27:6789/0 conn(0x7f055015df00 legacy=0x7f05500d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.470+0000 7f054ffff640 1 -- 172.21.15.27:0/2186345873 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05500ce470 con 0x7f05500b9a10 2024-03-06T07:33:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.470+0000 7f054effd640 1 --2- 172.21.15.27:0/2186345873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 0x7f05500b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:29.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.471+0000 7f054ffff640 1 --2- 172.21.15.27:0/2186345873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05500b9a10 0x7f05500b5c40 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0548009d90 tx=0x7f054802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e0532351633ad7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.471+0000 7f054e7fc640 1 -- 172.21.15.27:0/2186345873 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548039070 con 0x7f05500b9a10 2024-03-06T07:33:29.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.471+0000 7f054e7fc640 1 -- 172.21.15.27:0/2186345873 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054802cc50 con 0x7f05500b9a10 2024-03-06T07:33:29.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.471+0000 7f054e7fc640 1 -- 172.21.15.27:0/2186345873 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548034440 con 0x7f05500b9a10 2024-03-06T07:33:29.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.472+0000 7f0555e47640 1 -- 172.21.15.27:0/2186345873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05500b9a10 msgr2=0x7f05500b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.472+0000 7f0555e47640 1 --2- 172.21.15.27:0/2186345873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05500b9a10 0x7f05500b5c40 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0548009d90 tx=0x7f054802bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.472+0000 7f0555e47640 1 -- 172.21.15.27:0/2186345873 shutdown_connections 2024-03-06T07:33:29.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.472+0000 7f0555e47640 1 --2- 172.21.15.27:0/2186345873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05500b9a10 0x7f05500b5c40 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.472+0000 7f0555e47640 1 --2- 172.21.15.27:0/2186345873 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 0x7f05500b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.472+0000 7f0555e47640 1 -- 172.21.15.27:0/2186345873 >> 172.21.15.27:0/2186345873 conn(0x7f0550071960 msgr2=0x7f0550071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.472+0000 7f0555e47640 1 -- 172.21.15.27:0/2186345873 shutdown_connections 2024-03-06T07:33:29.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.472+0000 7f0555e47640 1 -- 172.21.15.27:0/2186345873 wait complete. 2024-03-06T07:33:29.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.473+0000 7f0555e47640 1 Processor -- start 2024-03-06T07:33:29.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.473+0000 7f0555e47640 1 -- start start 2024-03-06T07:33:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f0555e47640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 0x7f0550155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f0555e47640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500b9a10 0x7f05500ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f0555e47640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f055015df00 0x7f0550160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f0555e47640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05500cedf0 con 0x7f05500b9a10 2024-03-06T07:33:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f0555e47640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05500cec70 con 0x7f055015df00 2024-03-06T07:33:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f0555e47640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05500d3220 con 0x7f05500b95a0 2024-03-06T07:33:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500b9a10 0x7f05500ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500b9a10 0x7f05500ccf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56620/0 (socket says 172.21.15.27:56620) 2024-03-06T07:33:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054effd640 1 -- 172.21.15.27:0/1092444141 learned_addr learned my addr 172.21.15.27:0/1092444141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054f7fe640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 0x7f0550155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054ffff640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f055015df00 0x7f0550160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054effd640 1 -- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 msgr2=0x7f0550155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054effd640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 0x7f0550155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054effd640 1 -- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f055015df00 msgr2=0x7f0550160860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054effd640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f055015df00 0x7f0550160860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054effd640 1 -- 172.21.15.27:0/1092444141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05500c7b90 con 0x7f05500b9a10 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.474+0000 7f054effd640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500b9a10 0x7f05500ccf90 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f054000ebe0 tx=0x7f054000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.475+0000 7f0547fff640 1 -- 172.21.15.27:0/1092444141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540017070 con 0x7f05500b9a10 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.475+0000 7f0547fff640 1 -- 172.21.15.27:0/1092444141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054000cc30 con 0x7f05500b9a10 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.475+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05500c7e70 con 0x7f05500b9a10 2024-03-06T07:33:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.475+0000 7f0547fff640 1 -- 172.21.15.27:0/1092444141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540013470 con 0x7f05500b9a10 2024-03-06T07:33:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.475+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05500c8430 con 0x7f05500b9a10 2024-03-06T07:33:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.476+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f055015d730 con 0x7f05500b9a10 2024-03-06T07:33:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.476+0000 7f0547fff640 1 -- 172.21.15.27:0/1092444141 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f054001a070 con 0x7f05500b9a10 2024-03-06T07:33:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.477+0000 7f0547fff640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f053006b4d0 0x7f053006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.477+0000 7f0547fff640 1 -- 172.21.15.27:0/1092444141 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f054008a510 con 0x7f05500b9a10 2024-03-06T07:33:29.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.477+0000 7f054f7fe640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f053006b4d0 0x7f053006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.477+0000 7f054f7fe640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f053006b4d0 0x7f053006d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f0538005880 tx=0x7f0538009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.481+0000 7f0547fff640 1 -- 172.21.15.27:0/1092444141 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f055015d730 con 0x7f05500b9a10 2024-03-06T07:33:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.700+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05500b60e0 con 0x7f05500b9a10 2024-03-06T07:33:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.700+0000 7f0547fff640 1 -- 172.21.15.27:0/1092444141 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f05500b60e0 con 0x7f05500b9a10 2024-03-06T07:33:29.705 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:29.705 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f053006b4d0 msgr2=0x7f053006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f053006b4d0 0x7f053006d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f0538005880 tx=0x7f0538009040 comp rx=0 tx=0).stop 2024-03-06T07:33:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500b9a10 msgr2=0x7f05500ccf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500b9a10 0x7f05500ccf90 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f054000ebe0 tx=0x7f054000c520 comp rx=0 tx=0).stop 2024-03-06T07:33:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 shutdown_connections 2024-03-06T07:33:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f053006b4d0 0x7f053006d990 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f055015df00 0x7f0550160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500b9a10 0x7f05500ccf90 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 --2- 172.21.15.27:0/1092444141 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f05500b95a0 0x7f0550155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 >> 172.21.15.27:0/1092444141 conn(0x7f0550071960 msgr2=0x7f0550071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 shutdown_connections 2024-03-06T07:33:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:29.702+0000 7f0555e47640 1 -- 172.21.15.27:0/1092444141 wait complete. 2024-03-06T07:33:29.916 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:29.917 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:30.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.022+0000 7fe98b35c640 1 Processor -- start 2024-03-06T07:33:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.022+0000 7fe98b35c640 1 -- start start 2024-03-06T07:33:31.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe98b35c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe98416aa30 0x7fe98416f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe98b35c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840cfd70 0x7fe9840b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe98b35c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe984070870 con 0x7fe9840b4810 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe98b35c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe98416f570 con 0x7fe98416aa30 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe98b35c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe984072470 con 0x7fe9840cfd70 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe9890f8640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe98416aa30 0x7fe98416f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe9888f7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe9840b4810 0x7fe9840cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52016/0 (socket says 172.21.15.27:52016) 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe9888f7640 1 -- 172.21.15.27:0/3961146433 learned_addr learned my addr 172.21.15.27:0/3961146433 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe9898f9640 1 --2- 172.21.15.27:0/3961146433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840cfd70 0x7fe9840b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.023+0000 7fe97bfff640 1 -- 172.21.15.27:0/3961146433 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457866920 0 0) 0x7fe984070870 con 0x7fe9840b4810 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.024+0000 7fe97bfff640 1 -- 172.21.15.27:0/3961146433 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe970003500 con 0x7fe9840b4810 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.025+0000 7fe97bfff640 1 -- 172.21.15.27:0/3961146433 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 998162457 0 0) 0x7fe970003500 con 0x7fe9840b4810 2024-03-06T07:33:31.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.025+0000 7fe9890f8640 1 -- 172.21.15.27:0/3961146433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840cfd70 msgr2=0x7fe9840b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.025+0000 7fe9890f8640 1 --2- 172.21.15.27:0/3961146433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840cfd70 0x7fe9840b2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.025+0000 7fe9890f8640 1 -- 172.21.15.27:0/3961146433 >> v1:172.21.15.27:6789/0 conn(0x7fe9840b4810 legacy=0x7fe9840cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.025+0000 7fe9890f8640 1 -- 172.21.15.27:0/3961146433 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe984072750 con 0x7fe98416aa30 2024-03-06T07:33:31.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.025+0000 7fe9890f8640 1 --2- 172.21.15.27:0/3961146433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 0x7fe98416f030 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fe98000b850 tx=0x7fe98002bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4b9edcc0281618 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe97bfff640 1 -- 172.21.15.27:0/3961146433 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe980039070 con 0x7fe98416aa30 2024-03-06T07:33:31.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe97bfff640 1 -- 172.21.15.27:0/3961146433 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98002c880 con 0x7fe98416aa30 2024-03-06T07:33:31.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe97bfff640 1 -- 172.21.15.27:0/3961146433 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974001670 con 0x7fe98416aa30 2024-03-06T07:33:31.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe98b35c640 1 -- 172.21.15.27:0/3961146433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 msgr2=0x7fe98416f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe98b35c640 1 --2- 172.21.15.27:0/3961146433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 0x7fe98416f030 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fe98000b850 tx=0x7fe98002bd50 comp rx=0 tx=0).stop 2024-03-06T07:33:31.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe98b35c640 1 -- 172.21.15.27:0/3961146433 shutdown_connections 2024-03-06T07:33:31.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe98b35c640 1 --2- 172.21.15.27:0/3961146433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840cfd70 0x7fe9840b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe98b35c640 1 --2- 172.21.15.27:0/3961146433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 0x7fe98416f030 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe98b35c640 1 -- 172.21.15.27:0/3961146433 >> 172.21.15.27:0/3961146433 conn(0x7fe984066d00 msgr2=0x7fe984067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe98b35c640 1 -- 172.21.15.27:0/3961146433 shutdown_connections 2024-03-06T07:33:31.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.026+0000 7fe98b35c640 1 -- 172.21.15.27:0/3961146433 wait complete. 2024-03-06T07:33:31.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.028+0000 7fe98b35c640 1 Processor -- start 2024-03-06T07:33:31.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.028+0000 7fe98b35c640 1 -- start start 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe98b35c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840b4810 0x7fe984200750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe98b35c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840cfd70 0x7fe9840cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe98b35c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 0x7fe9840a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe98b35c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98416ae10 con 0x7fe98416aa30 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe98b35c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe984070b10 con 0x7fe9840cfd70 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe98b35c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98416af90 con 0x7fe9840b4810 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9898f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 0x7fe9840a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9898f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 0x7fe9840a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56644/0 (socket says 172.21.15.27:56644) 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9898f9640 1 -- 172.21.15.27:0/3225210411 learned_addr learned my addr 172.21.15.27:0/3225210411 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:31.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9888f7640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840cfd70 0x7fe9840cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9890f8640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840b4810 0x7fe984200750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9898f9640 1 -- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840b4810 msgr2=0x7fe984200750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9898f9640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840b4810 0x7fe984200750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9898f9640 1 -- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840cfd70 msgr2=0x7fe9840cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9898f9640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840cfd70 0x7fe9840cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9898f9640 1 -- 172.21.15.27:0/3225210411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe984200ea0 con 0x7fe98416aa30 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.030+0000 7fe9898f9640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 0x7fe9840a94a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00fb30 tx=0x7fe97c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.031+0000 7fe979ffb640 1 -- 172.21.15.27:0/3225210411 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c018020 con 0x7fe98416aa30 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.031+0000 7fe979ffb640 1 -- 172.21.15.27:0/3225210411 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c00daf0 con 0x7fe98416aa30 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.031+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe984201180 con 0x7fe98416aa30 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.031+0000 7fe979ffb640 1 -- 172.21.15.27:0/3225210411 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c014470 con 0x7fe98416aa30 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.031+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe984167960 con 0x7fe98416aa30 2024-03-06T07:33:31.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.032+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe98416e7f0 con 0x7fe98416aa30 2024-03-06T07:33:31.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.032+0000 7fe979ffb640 1 -- 172.21.15.27:0/3225210411 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe97c01b020 con 0x7fe98416aa30 2024-03-06T07:33:31.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.033+0000 7fe979ffb640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe96806b4d0 0x7fe96806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.033+0000 7fe979ffb640 1 -- 172.21.15.27:0/3225210411 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe97c08a6d0 con 0x7fe98416aa30 2024-03-06T07:33:31.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.033+0000 7fe9890f8640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe96806b4d0 0x7fe96806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.033+0000 7fe9890f8640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe96806b4d0 0x7fe96806d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe980036000 tx=0x7fe980030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.037+0000 7fe979ffb640 1 -- 172.21.15.27:0/3225210411 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe98416e7f0 con 0x7fe98416aa30 2024-03-06T07:33:31.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.258+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe984067ff0 con 0x7fe98416aa30 2024-03-06T07:33:31.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.259+0000 7fe979ffb640 1 -- 172.21.15.27:0/3225210411 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fe97c022030 con 0x7fe98416aa30 2024-03-06T07:33:31.262 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:31.262 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:31.262 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe96806b4d0 msgr2=0x7fe96806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe96806b4d0 0x7fe96806d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe980036000 tx=0x7fe980030000 comp rx=0 tx=0).stop 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 msgr2=0x7fe9840a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 0x7fe9840a94a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00fb30 tx=0x7fe97c007f60 comp rx=0 tx=0).stop 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 shutdown_connections 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe96806b4d0 0x7fe96806d990 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe98416aa30 0x7fe9840a94a0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9840cfd70 0x7fe9840cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 --2- 172.21.15.27:0/3225210411 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9840b4810 0x7fe984200750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 >> 172.21.15.27:0/3225210411 conn(0x7fe984066d00 msgr2=0x7fe98416dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 shutdown_connections 2024-03-06T07:33:31.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:31.261+0000 7fe98b35c640 1 -- 172.21.15.27:0/3225210411 wait complete. 2024-03-06T07:33:31.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:31.476 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:32.477 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.600+0000 7f06de039640 1 Processor -- start 2024-03-06T07:33:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.600+0000 7f06de039640 1 -- start start 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.600+0000 7f06de039640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d815df00 0x7f06d80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.600+0000 7f06de039640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f06d80b95a0 0x7f06d80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.600+0000 7f06de039640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d80cea30 con 0x7f06d80b9a30 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.600+0000 7f06de039640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d80d2af0 con 0x7f06d80b95a0 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.600+0000 7f06de039640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d80cf3c0 con 0x7f06d815df00 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d7fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f06d80b9a30 0x7f06d80d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52018/0 (socket says 172.21.15.27:52018) 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d7fff640 1 -- 172.21.15.27:0/3299535316 learned_addr learned my addr 172.21.15.27:0/3299535316 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d6ffd640 1 --2- 172.21.15.27:0/3299535316 >> v2:172.21.15.69:3300/0 conn(0x7f06d80b95a0 0x7f06d80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d77fe640 1 --2- 172.21.15.27:0/3299535316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d815df00 0x7f06d80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d67fc640 1 -- 172.21.15.27:0/3299535316 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80334878 0 0) 0x7f06d80cea30 con 0x7f06d80b9a30 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d67fc640 1 -- 172.21.15.27:0/3299535316 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06c0003520 con 0x7f06d80b9a30 2024-03-06T07:33:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d6ffd640 1 -- 172.21.15.27:0/3299535316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d815df00 msgr2=0x7f06d80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d6ffd640 1 --2- 172.21.15.27:0/3299535316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d815df00 0x7f06d80d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d6ffd640 1 -- 172.21.15.27:0/3299535316 >> v1:172.21.15.27:6789/0 conn(0x7f06d80b9a30 legacy=0x7f06d80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d6ffd640 1 -- 172.21.15.27:0/3299535316 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d80cf0e0 con 0x7f06d80b95a0 2024-03-06T07:33:32.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.601+0000 7f06d77fe640 1 --2- 172.21.15.27:0/3299535316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d815df00 0x7f06d80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.602+0000 7f06d6ffd640 1 --2- 172.21.15.27:0/3299535316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d80b95a0 0x7f06d80b5700 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0097a0 tx=0x7f06cc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61f38b3e70a8a6a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.602+0000 7f06d67fc640 1 -- 172.21.15.27:0/3299535316 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc035020 con 0x7f06d80b95a0 2024-03-06T07:33:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.602+0000 7f06d67fc640 1 -- 172.21.15.27:0/3299535316 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc031830 con 0x7f06d80b95a0 2024-03-06T07:33:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.602+0000 7f06d67fc640 1 -- 172.21.15.27:0/3299535316 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc031be0 con 0x7f06d80b95a0 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.603+0000 7f06de039640 1 -- 172.21.15.27:0/3299535316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d80b95a0 msgr2=0x7f06d80b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.603+0000 7f06de039640 1 --2- 172.21.15.27:0/3299535316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d80b95a0 0x7f06d80b5700 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0097a0 tx=0x7f06cc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.603+0000 7f06de039640 1 -- 172.21.15.27:0/3299535316 shutdown_connections 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.603+0000 7f06de039640 1 --2- 172.21.15.27:0/3299535316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d80b95a0 0x7f06d80b5700 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.603+0000 7f06de039640 1 --2- 172.21.15.27:0/3299535316 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d815df00 0x7f06d80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.603+0000 7f06de039640 1 -- 172.21.15.27:0/3299535316 >> 172.21.15.27:0/3299535316 conn(0x7f06d8071960 msgr2=0x7f06d8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.603+0000 7f06de039640 1 -- 172.21.15.27:0/3299535316 shutdown_connections 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.603+0000 7f06de039640 1 -- 172.21.15.27:0/3299535316 wait complete. 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.604+0000 7f06de039640 1 Processor -- start 2024-03-06T07:33:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.604+0000 7f06de039640 1 -- start start 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06de039640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d80b95a0 0x7f06d8154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06de039640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d815df00 0x7f06d8157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06de039640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d81f6bc0 0x7f06d81f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06de039640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d80cec70 con 0x7f06d815df00 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06de039640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d80ce750 con 0x7f06d81f6bc0 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06de039640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d80cedf0 con 0x7f06d80b95a0 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d815df00 0x7f06d8157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d815df00 0x7f06d8157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56646/0 (socket says 172.21.15.27:56646) 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d6ffd640 1 -- 172.21.15.27:0/2734293877 learned_addr learned my addr 172.21.15.27:0/2734293877 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d77fe640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d80b95a0 0x7f06d8154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d7fff640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d81f6bc0 0x7f06d81f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d6ffd640 1 -- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d80b95a0 msgr2=0x7f06d8154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d6ffd640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d80b95a0 0x7f06d8154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d6ffd640 1 -- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d81f6bc0 msgr2=0x7f06d81f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d6ffd640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d81f6bc0 0x7f06d81f3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d6ffd640 1 -- 172.21.15.27:0/2734293877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d8158640 con 0x7f06d815df00 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d77fe640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d80b95a0 0x7f06d8154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d7fff640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d81f6bc0 0x7f06d81f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.605+0000 7f06d6ffd640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d815df00 0x7f06d8157b00 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f06cc02b910 tx=0x7f06cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.606+0000 7f06bffff640 1 -- 172.21.15.27:0/2734293877 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc040070 con 0x7f06d815df00 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.606+0000 7f06bffff640 1 -- 172.21.15.27:0/2734293877 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc03a910 con 0x7f06d815df00 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.606+0000 7f06bffff640 1 -- 172.21.15.27:0/2734293877 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc035040 con 0x7f06d815df00 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.606+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d8158920 con 0x7f06d815df00 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.606+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d81f3700 con 0x7f06d815df00 2024-03-06T07:33:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.607+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d8067ff0 con 0x7f06d815df00 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.607+0000 7f06bffff640 1 -- 172.21.15.27:0/2734293877 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06cc04b060 con 0x7f06d815df00 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.608+0000 7f06bffff640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f06c406b4d0 0x7f06c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.608+0000 7f06bffff640 1 -- 172.21.15.27:0/2734293877 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f06cc0ab980 con 0x7f06d815df00 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.608+0000 7f06d77fe640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f06c406b4d0 0x7f06c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.609+0000 7f06d77fe640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f06c406b4d0 0x7f06c406d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009ba0 tx=0x7f06c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.612+0000 7f06bffff640 1 -- 172.21.15.27:0/2734293877 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06cc02f2e0 con 0x7f06d815df00 2024-03-06T07:33:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.834+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06d80688a0 con 0x7f06d815df00 2024-03-06T07:33:32.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.835+0000 7f06bffff640 1 -- 172.21.15.27:0/2734293877 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f06cc07a7e0 con 0x7f06d815df00 2024-03-06T07:33:32.837 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:32.837 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f06c406b4d0 msgr2=0x7f06c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f06c406b4d0 0x7f06c406d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009ba0 tx=0x7f06c8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d815df00 msgr2=0x7f06d8157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d815df00 0x7f06d8157b00 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f06cc02b910 tx=0x7f06cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 shutdown_connections 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f06c406b4d0 0x7f06c406d990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06d81f6bc0 0x7f06d81f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d815df00 0x7f06d8157b00 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 --2- 172.21.15.27:0/2734293877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f06d80b95a0 0x7f06d8154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 >> 172.21.15.27:0/2734293877 conn(0x7f06d8071960 msgr2=0x7f06d80cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 shutdown_connections 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:32.837+0000 7f06de039640 1 -- 172.21.15.27:0/2734293877 wait complete. 2024-03-06T07:33:33.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:33.052 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:34.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.157+0000 7f302edbc640 1 Processor -- start 2024-03-06T07:33:34.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.157+0000 7f302edbc640 1 -- start start 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302edbc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3028071540 0x7f30280b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302edbc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30280b2b50 0x7f30280b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302edbc640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30280ce610 con 0x7f30280d3400 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302edbc640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30280d2f70 con 0x7f30280b2b50 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302edbc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30280d4be0 con 0x7f3028071540 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302d359640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30280b2b50 0x7f30280b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f3027fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f30280d3400 0x7f3028071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38568/0 (socket says 172.21.15.27:38568) 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f3027fff640 1 -- 172.21.15.27:0/3749570258 learned_addr learned my addr 172.21.15.27:0/3749570258 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302cb58640 1 --2- 172.21.15.27:0/3749570258 >> v2:172.21.15.69:3300/0 conn(0x7f3028071540 0x7f30280b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302d359640 1 -- 172.21.15.27:0/3749570258 >> v2:172.21.15.69:3300/0 conn(0x7f3028071540 msgr2=0x7f30280b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302d359640 1 --2- 172.21.15.27:0/3749570258 >> v2:172.21.15.69:3300/0 conn(0x7f3028071540 0x7f30280b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302d359640 1 -- 172.21.15.27:0/3749570258 >> v1:172.21.15.27:6789/0 conn(0x7f30280d3400 legacy=0x7f3028071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f30277fe640 1 -- 172.21.15.27:0/3749570258 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465537799 0 0) 0x7f30280ce610 con 0x7f30280d3400 2024-03-06T07:33:34.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302d359640 1 -- 172.21.15.27:0/3749570258 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30280d2920 con 0x7f30280b2b50 2024-03-06T07:33:34.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.158+0000 7f302cb58640 1 --2- 172.21.15.27:0/3749570258 >> v2:172.21.15.69:3300/0 conn(0x7f3028071540 0x7f30280b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.159+0000 7f302d359640 1 --2- 172.21.15.27:0/3749570258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280b2b50 0x7f30280b6e20 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f30280d4da0 tx=0x7f302002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c4a1412b5d93f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.159+0000 7f30277fe640 1 -- 172.21.15.27:0/3749570258 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3020039070 con 0x7f30280b2b50 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.159+0000 7f30277fe640 1 -- 172.21.15.27:0/3749570258 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302002cc80 con 0x7f30280b2b50 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.159+0000 7f30277fe640 1 -- 172.21.15.27:0/3749570258 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3020034660 con 0x7f30280b2b50 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.160+0000 7f302edbc640 1 -- 172.21.15.27:0/3749570258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280b2b50 msgr2=0x7f30280b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.160+0000 7f302edbc640 1 --2- 172.21.15.27:0/3749570258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280b2b50 0x7f30280b6e20 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f30280d4da0 tx=0x7f302002bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.160+0000 7f302edbc640 1 -- 172.21.15.27:0/3749570258 shutdown_connections 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.160+0000 7f302edbc640 1 --2- 172.21.15.27:0/3749570258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280b2b50 0x7f30280b6e20 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.160+0000 7f302edbc640 1 --2- 172.21.15.27:0/3749570258 >> v2:172.21.15.69:3300/0 conn(0x7f3028071540 0x7f30280b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.160+0000 7f302edbc640 1 -- 172.21.15.27:0/3749570258 >> 172.21.15.27:0/3749570258 conn(0x7f30280d3ca0 msgr2=0x7f30280d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.160+0000 7f302edbc640 1 -- 172.21.15.27:0/3749570258 shutdown_connections 2024-03-06T07:33:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.160+0000 7f302edbc640 1 -- 172.21.15.27:0/3749570258 wait complete. 2024-03-06T07:33:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.161+0000 7f302edbc640 1 Processor -- start 2024-03-06T07:33:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.161+0000 7f302edbc640 1 -- start start 2024-03-06T07:33:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302edbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3028071540 0x7f30280b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302edbc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30280b2b50 0x7f302815f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302edbc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280d3400 0x7f3028155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302edbc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302815e370 con 0x7f30280d3400 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302edbc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30280ce8f0 con 0x7f30280b2b50 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302edbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30280cc5b0 con 0x7f3028071540 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302d359640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280d3400 0x7f3028155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302d359640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280d3400 0x7f3028155250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43302/0 (socket says 172.21.15.27:43302) 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302d359640 1 -- 172.21.15.27:0/1239839639 learned_addr learned my addr 172.21.15.27:0/1239839639 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302cb58640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3028071540 0x7f30280b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f3027fff640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30280b2b50 0x7f302815f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302d359640 1 -- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3028071540 msgr2=0x7f30280b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302d359640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3028071540 0x7f30280b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302d359640 1 -- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30280b2b50 msgr2=0x7f302815f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302d359640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30280b2b50 0x7f302815f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302d359640 1 -- 172.21.15.27:0/1239839639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3028155790 con 0x7f30280d3400 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.162+0000 7f302d359640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280d3400 0x7f3028155250 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f3020009e80 tx=0x7f3020004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.163+0000 7f30257fa640 1 -- 172.21.15.27:0/1239839639 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3020039050 con 0x7f30280d3400 2024-03-06T07:33:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.163+0000 7f30257fa640 1 -- 172.21.15.27:0/1239839639 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302003a040 con 0x7f30280d3400 2024-03-06T07:33:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.163+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30280b4760 con 0x7f30280d3400 2024-03-06T07:33:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.163+0000 7f30257fa640 1 -- 172.21.15.27:0/1239839639 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302003e610 con 0x7f30280d3400 2024-03-06T07:33:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.163+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30280cc240 con 0x7f30280d3400 2024-03-06T07:33:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.164+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3028068700 con 0x7f30280d3400 2024-03-06T07:33:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.164+0000 7f30257fa640 1 -- 172.21.15.27:0/1239839639 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3020047020 con 0x7f30280d3400 2024-03-06T07:33:34.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.165+0000 7f30257fa640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f300c06b4d0 0x7f300c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.165+0000 7f30257fa640 1 -- 172.21.15.27:0/1239839639 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f30200ac680 con 0x7f30280d3400 2024-03-06T07:33:34.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.165+0000 7f302cb58640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f300c06b4d0 0x7f300c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.166+0000 7f302cb58640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f300c06b4d0 0x7f300c06d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f30280b6770 tx=0x7f301c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.169+0000 7f30257fa640 1 -- 172.21.15.27:0/1239839639 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f302004e020 con 0x7f30280d3400 2024-03-06T07:33:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.388+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30280b8790 con 0x7f30280d3400 2024-03-06T07:33:34.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.389+0000 7f30257fa640 1 -- 172.21.15.27:0/1239839639 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f3028155790 con 0x7f30280d3400 2024-03-06T07:33:34.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:34.391 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:34.391 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.390+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f300c06b4d0 msgr2=0x7f300c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.390+0000 7f302edbc640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f300c06b4d0 0x7f300c06d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f30280b6770 tx=0x7f301c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.390+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280d3400 msgr2=0x7f3028155250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.390+0000 7f302edbc640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280d3400 0x7f3028155250 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f3020009e80 tx=0x7f3020004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.390+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 shutdown_connections 2024-03-06T07:33:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.390+0000 7f302edbc640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f300c06b4d0 0x7f300c06d990 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.390+0000 7f302edbc640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280d3400 0x7f3028155250 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.391+0000 7f302edbc640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30280b2b50 0x7f302815f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.391+0000 7f302edbc640 1 --2- 172.21.15.27:0/1239839639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3028071540 0x7f30280b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.391+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 >> 172.21.15.27:0/1239839639 conn(0x7f30280d3ca0 msgr2=0x7f302815ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.391+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 shutdown_connections 2024-03-06T07:33:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:34.391+0000 7f302edbc640 1 -- 172.21.15.27:0/1239839639 wait complete. 2024-03-06T07:33:34.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:34.605 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:35.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.711+0000 7f82a1f3a640 1 Processor -- start 2024-03-06T07:33:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.711+0000 7f82a1f3a640 1 -- start start 2024-03-06T07:33:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f82a1f3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c15df00 0x7f829c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f82a1f3a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c0cea30 con 0x7f829c0b9a30 2024-03-06T07:33:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f82a1f3a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c0d2af0 con 0x7f829c0b95a0 2024-03-06T07:33:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f82a1f3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0cf3c0 con 0x7f829c15df00 2024-03-06T07:33:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829bfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f829c0b9a30 0x7f829c0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38584/0 (socket says 172.21.15.27:38584) 2024-03-06T07:33:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829bfff640 1 -- 172.21.15.27:0/627696767 learned_addr learned my addr 172.21.15.27:0/627696767 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829b7fe640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c15df00 0x7f829c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349341414 0 0) 0x7f829c0cea30 con 0x7f829c0b9a30 2024-03-06T07:33:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8284003520 con 0x7f829c0b9a30 2024-03-06T07:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1618543317 0 0) 0x7f8284003520 con 0x7f829c0b9a30 2024-03-06T07:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c15df00 msgr2=0x7f829c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c15df00 0x7f829c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 >> v1:172.21.15.69:6789/0 conn(0x7f829c0b95a0 legacy=0x7f829c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0cf0e0 con 0x7f829c0b9a30 2024-03-06T07:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f82940019f0 con 0x7f829c0b9a30 2024-03-06T07:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 >> v1:172.21.15.27:6789/0 conn(0x7f829c0b9a30 legacy=0x7f829c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82840039e0 0x7f8284003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.712+0000 7f829a7fc640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8284004390 0x7f8284004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829a7fc640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c15df00 0x7f82840050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0cf3c0 con 0x7f8284004390 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828400b9c0 con 0x7f82840039e0 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828400bbc0 con 0x7f829c15df00 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829affd640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8284004390 0x7f8284004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829b7fe640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82840039e0 0x7f8284003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829bfff640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c15df00 0x7f82840050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829affd640 1 -- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c15df00 msgr2=0x7f82840050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829affd640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c15df00 0x7f82840050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829affd640 1 -- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82840039e0 msgr2=0x7f8284003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829affd640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82840039e0 0x7f8284003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829affd640 1 -- 172.21.15.27:0/627696767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8284003520 con 0x7f8284004390 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.713+0000 7f829b7fe640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f82840039e0 0x7f8284003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f829affd640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8284004390 0x7f8284004860 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f8290001910 tx=0x7f8284005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f23cb5b51873f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82900014e0 con 0x7f8284004390 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8290037070 con 0x7f8284004390 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f829a7fc640 1 -- 172.21.15.27:0/627696767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290032840 con 0x7f8284004390 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f82a1f3a640 1 -- 172.21.15.27:0/627696767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8284004390 msgr2=0x7f8284004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f82a1f3a640 1 --2- 172.21.15.27:0/627696767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8284004390 0x7f8284004860 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f8290001910 tx=0x7f8284005830 comp rx=0 tx=0).stop 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f829bfff640 1 -- 172.21.15.27:0/627696767 reap_dead start 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f82a1f3a640 1 -- 172.21.15.27:0/627696767 shutdown_connections 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f82a1f3a640 1 -- 172.21.15.27:0/627696767 >> 172.21.15.27:0/627696767 conn(0x7f829c071960 msgr2=0x7f829c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f82a1f3a640 1 -- 172.21.15.27:0/627696767 shutdown_connections 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.714+0000 7f82a1f3a640 1 -- 172.21.15.27:0/627696767 wait complete. 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.716+0000 7f82a1f3a640 1 Processor -- start 2024-03-06T07:33:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.716+0000 7f82a1f3a640 1 -- start start 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.716+0000 7f82a1f3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8284004390 0x7f829c15b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.716+0000 7f82a1f3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c0c8f20 0x7f829c0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.716+0000 7f82a1f3a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f829c0cc6a0 0x7f829c0ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.716+0000 7f82a1f3a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0cec70 con 0x7f829c0cc6a0 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.716+0000 7f82a1f3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0ce750 con 0x7f8284004390 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.716+0000 7f82a1f3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0cedf0 con 0x7f829c0c8f20 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f829c0cc6a0 0x7f829c0ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f829c0cc6a0 0x7f829c0ccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43320/0 (socket says 172.21.15.27:43320) 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829bfff640 1 -- 172.21.15.27:0/625650450 learned_addr learned my addr 172.21.15.27:0/625650450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829affd640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c0c8f20 0x7f829c0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829b7fe640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8284004390 0x7f829c15b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829bfff640 1 -- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c0c8f20 msgr2=0x7f829c0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829bfff640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c0c8f20 0x7f829c0c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829bfff640 1 -- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8284004390 msgr2=0x7f829c15b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829bfff640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8284004390 0x7f829c15b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829bfff640 1 -- 172.21.15.27:0/625650450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0c7850 con 0x7f829c0cc6a0 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829affd640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c0c8f20 0x7f829c0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829b7fe640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8284004390 0x7f829c15b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f829bfff640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f829c0cc6a0 0x7f829c0ccb20 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f829c0d1780 tx=0x7f8294007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f8283fff640 1 -- 172.21.15.27:0/625650450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829400ea20 con 0x7f829c0cc6a0 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f8283fff640 1 -- 172.21.15.27:0/625650450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829400ebc0 con 0x7f829c0cc6a0 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f8283fff640 1 -- 172.21.15.27:0/625650450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829400edb0 con 0x7f829c0cc6a0 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c0c85d0 con 0x7f829c0cc6a0 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.717+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c0c7dd0 con 0x7f829c0cc6a0 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.718+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829c068ad0 con 0x7f829c0cc6a0 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.719+0000 7f8283fff640 1 -- 172.21.15.27:0/625650450 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f829401a020 con 0x7f829c0cc6a0 2024-03-06T07:33:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.719+0000 7f8283fff640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828806b500 0x7f828806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.719+0000 7f8283fff640 1 -- 172.21.15.27:0/625650450 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8294089ba0 con 0x7f829c0cc6a0 2024-03-06T07:33:35.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.720+0000 7f829b7fe640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828806b500 0x7f828806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.720+0000 7f829b7fe640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828806b500 0x7f828806d9c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f828c009a80 tx=0x7f828c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.729+0000 7f8283fff640 1 -- 172.21.15.27:0/625650450 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f829c068ad0 con 0x7f829c0cc6a0 2024-03-06T07:33:35.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.947+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f829c067ff0 con 0x7f829c0cc6a0 2024-03-06T07:33:35.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.948+0000 7f8283fff640 1 -- 172.21.15.27:0/625650450 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f8294021020 con 0x7f829c0cc6a0 2024-03-06T07:33:35.950 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:35.950 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:35.950 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828806b500 msgr2=0x7f828806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828806b500 0x7f828806d9c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f828c009a80 tx=0x7f828c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f829c0cc6a0 msgr2=0x7f829c0ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f829c0cc6a0 0x7f829c0ccb20 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f829c0d1780 tx=0x7f8294007bc0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 shutdown_connections 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f828806b500 0x7f828806d9c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f829c0cc6a0 0x7f829c0ccb20 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f829c0c8f20 0x7f829c0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 --2- 172.21.15.27:0/625650450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8284004390 0x7f829c15b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 >> 172.21.15.27:0/625650450 conn(0x7f829c071960 msgr2=0x7f829c0b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 shutdown_connections 2024-03-06T07:33:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:35.950+0000 7f82a1f3a640 1 -- 172.21.15.27:0/625650450 wait complete. 2024-03-06T07:33:36.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:36.164 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:37.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.287+0000 7f2b93aeb640 1 Processor -- start 2024-03-06T07:33:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.287+0000 7f2b93aeb640 1 -- start start 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b93aeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06fe20 0x7f2b8c0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b93aeb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8c0d0dd0 con 0x7f2b8c0d0880 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b93aeb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8c0d10b0 con 0x7f2b8c15e060 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b93aeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0cc0e0 con 0x7f2b8c06fe20 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b92088640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2b8c0d0880 0x7f2b8c0ccbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38586/0 (socket says 172.21.15.27:38586) 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b92088640 1 -- 172.21.15.27:0/1477186547 learned_addr learned my addr 172.21.15.27:0/1477186547 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798904849 0 0) 0x7f2b8c0d0dd0 con 0x7f2b8c0d0880 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b78003520 con 0x7f2b8c0d0880 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b91086640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06fe20 0x7f2b8c0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1204408813 0 0) 0x7f2b78003520 con 0x7f2b8c0d0880 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06fe20 msgr2=0x7f2b8c0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b90885640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06fe20 0x7f2b8c0cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 >> v1:172.21.15.69:6789/0 conn(0x7f2b8c15e060 legacy=0x7f2b8c0ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c0cc3c0 con 0x7f2b8c0d0880 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.288+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f2b840019f0 con 0x7f2b8c0d0880 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 >> v1:172.21.15.27:6789/0 conn(0x7f2b8c0d0880 legacy=0x7f2b8c0ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b90885640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b780039e0 0x7f2b78003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b90885640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b78004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b90885640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06fe20 0x7f2b780050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0cc0e0 con 0x7f2b78004390 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7800b9f0 con 0x7f2b780039e0 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7800bbf0 con 0x7f2b8c06fe20 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b91086640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b78004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b91887640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b780039e0 0x7f2b78003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b92088640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06fe20 0x7f2b780050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b91086640 1 -- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06fe20 msgr2=0x7f2b780050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b91086640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c06fe20 0x7f2b780050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b91086640 1 -- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b780039e0 msgr2=0x7f2b78003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b91086640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b780039e0 0x7f2b78003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.289+0000 7f2b91086640 1 -- 172.21.15.27:0/1477186547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b78003520 con 0x7f2b78004390 2024-03-06T07:33:37.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.290+0000 7f2b91086640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b78004860 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009a90 tx=0x7f2b78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc48670f79ae8de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.290+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c039070 con 0x7f2b78004390 2024-03-06T07:33:37.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.291+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c02c680 con 0x7f2b78004390 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.291+0000 7f2b90885640 1 -- 172.21.15.27:0/1477186547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c009690 con 0x7f2b78004390 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.291+0000 7f2b93aeb640 1 -- 172.21.15.27:0/1477186547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 msgr2=0x7f2b78004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.291+0000 7f2b93aeb640 1 --2- 172.21.15.27:0/1477186547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b78004860 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009a90 tx=0x7f2b78005830 comp rx=0 tx=0).stop 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.291+0000 7f2b92088640 1 -- 172.21.15.27:0/1477186547 reap_dead start 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.291+0000 7f2b93aeb640 1 -- 172.21.15.27:0/1477186547 shutdown_connections 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.291+0000 7f2b93aeb640 1 -- 172.21.15.27:0/1477186547 >> 172.21.15.27:0/1477186547 conn(0x7f2b8c0d1a90 msgr2=0x7f2b8c0d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.291+0000 7f2b93aeb640 1 -- 172.21.15.27:0/1477186547 shutdown_connections 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.291+0000 7f2b93aeb640 1 -- 172.21.15.27:0/1477186547 wait complete. 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b93aeb640 1 Processor -- start 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b93aeb640 1 -- start start 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b93aeb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b8c158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b93aeb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c1587d0 0x7f2b8c0ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b93aeb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c158c50 0x7f2b8c0aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b93aeb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0cea20 con 0x7f2b78004390 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b93aeb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c15e440 con 0x7f2b8c1587d0 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b93aeb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0ceba0 con 0x7f2b8c158c50 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b91887640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b8c158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b91887640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b8c158000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43344/0 (socket says 172.21.15.27:43344) 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.293+0000 7f2b91887640 1 -- 172.21.15.27:0/3122492255 learned_addr learned my addr 172.21.15.27:0/3122492255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b91086640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c1587d0 0x7f2b8c0ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b92088640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c158c50 0x7f2b8c0aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b91887640 1 -- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c158c50 msgr2=0x7f2b8c0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b91887640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c158c50 0x7f2b8c0aed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b91887640 1 -- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c1587d0 msgr2=0x7f2b8c0ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b91887640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c1587d0 0x7f2b8c0ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b91887640 1 -- 172.21.15.27:0/3122492255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c159190 con 0x7f2b78004390 2024-03-06T07:33:37.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b91086640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c1587d0 0x7f2b8c0ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:37.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b91887640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b8c158000 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800f7c0 tx=0x7f2b8800fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b827fc640 1 -- 172.21.15.27:0/3122492255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b880052e0 con 0x7f2b78004390 2024-03-06T07:33:37.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b827fc640 1 -- 172.21.15.27:0/3122492255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b88018070 con 0x7f2b78004390 2024-03-06T07:33:37.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c159470 con 0x7f2b78004390 2024-03-06T07:33:37.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b827fc640 1 -- 172.21.15.27:0/3122492255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8800d970 con 0x7f2b78004390 2024-03-06T07:33:37.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.294+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8c159a30 con 0x7f2b78004390 2024-03-06T07:33:37.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.295+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c067ff0 con 0x7f2b78004390 2024-03-06T07:33:37.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.296+0000 7f2b827fc640 1 -- 172.21.15.27:0/3122492255 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b8800db10 con 0x7f2b78004390 2024-03-06T07:33:37.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.296+0000 7f2b827fc640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b7006b4d0 0x7f2b7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.296+0000 7f2b827fc640 1 -- 172.21.15.27:0/3122492255 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2b8808a220 con 0x7f2b78004390 2024-03-06T07:33:37.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.296+0000 7f2b91086640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b7006b4d0 0x7f2b7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.297+0000 7f2b91086640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b7006b4d0 0x7f2b7006d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c036000 tx=0x7f2b7c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.301+0000 7f2b827fc640 1 -- 172.21.15.27:0/3122492255 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b8801a310 con 0x7f2b78004390 2024-03-06T07:33:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.522+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b8c0ce800 con 0x7f2b78004390 2024-03-06T07:33:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.522+0000 7f2b827fc640 1 -- 172.21.15.27:0/3122492255 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f2b88059080 con 0x7f2b78004390 2024-03-06T07:33:37.525 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:37.525 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b7006b4d0 msgr2=0x7f2b7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b7006b4d0 0x7f2b7006d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c036000 tx=0x7f2b7c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 msgr2=0x7f2b8c158000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b8c158000 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800f7c0 tx=0x7f2b8800fc80 comp rx=0 tx=0).stop 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 shutdown_connections 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2b7006b4d0 0x7f2b7006d990 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2b8c158c50 0x7f2b8c0aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8c1587d0 0x7f2b8c0ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 --2- 172.21.15.27:0/3122492255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b78004390 0x7f2b8c158000 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 >> 172.21.15.27:0/3122492255 conn(0x7f2b8c0d1a90 msgr2=0x7f2b8c06bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 shutdown_connections 2024-03-06T07:33:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:37.524+0000 7f2b93aeb640 1 -- 172.21.15.27:0/3122492255 wait complete. 2024-03-06T07:33:37.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:37.739 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:38.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:38.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.845+0000 7f336bfd0640 1 Processor -- start 2024-03-06T07:33:38.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.845+0000 7f336bfd0640 1 -- start start 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336bfd0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33640b9a10 0x7f33640d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336bfd0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33640cea30 con 0x7f33640b95a0 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336bfd0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33640cf3c0 con 0x7f336415df00 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336bfd0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33640cf0e0 con 0x7f33640b9a10 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336a56d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33640b9a10 0x7f33640d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336956b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f33640b95a0 0x7f33640b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38588/0 (socket says 172.21.15.27:38588) 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336a56d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33640b9a10 0x7f33640d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43348/0 (socket says 172.21.15.27:43348) 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336956b640 1 -- 172.21.15.27:0/1144934639 learned_addr learned my addr 172.21.15.27:0/1144934639 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336a56d640 1 -- 172.21.15.27:0/1144934639 >> v1:172.21.15.69:6789/0 conn(0x7f336415df00 legacy=0x7f33640d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336a56d640 1 -- 172.21.15.27:0/1144934639 >> v1:172.21.15.27:6789/0 conn(0x7f33640b95a0 legacy=0x7f33640b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f336a56d640 1 -- 172.21.15.27:0/1144934639 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33640ce470 con 0x7f33640b9a10 2024-03-06T07:33:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.846+0000 7f3368d6a640 1 -- 172.21.15.27:0/1144934639 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979851986 0 0) 0x7f33640cea30 con 0x7f33640b95a0 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.847+0000 7f336a56d640 1 --2- 172.21.15.27:0/1144934639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33640b9a10 0x7f33640d3220 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f33640cf590 tx=0x7f335c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3cbeb435263dbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.847+0000 7f3368d6a640 1 -- 172.21.15.27:0/1144934639 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c039070 con 0x7f33640b9a10 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.847+0000 7f3368d6a640 1 -- 172.21.15.27:0/1144934639 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c02cc50 con 0x7f33640b9a10 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.847+0000 7f3368d6a640 1 -- 172.21.15.27:0/1144934639 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c034440 con 0x7f33640b9a10 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.848+0000 7f336bfd0640 1 -- 172.21.15.27:0/1144934639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33640b9a10 msgr2=0x7f33640d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.848+0000 7f336bfd0640 1 --2- 172.21.15.27:0/1144934639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33640b9a10 0x7f33640d3220 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f33640cf590 tx=0x7f335c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.848+0000 7f336bfd0640 1 -- 172.21.15.27:0/1144934639 shutdown_connections 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.848+0000 7f336bfd0640 1 --2- 172.21.15.27:0/1144934639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33640b9a10 0x7f33640d3220 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.848+0000 7f336bfd0640 1 -- 172.21.15.27:0/1144934639 >> 172.21.15.27:0/1144934639 conn(0x7f3364071960 msgr2=0x7f3364071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.848+0000 7f336bfd0640 1 -- 172.21.15.27:0/1144934639 shutdown_connections 2024-03-06T07:33:38.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.848+0000 7f336bfd0640 1 -- 172.21.15.27:0/1144934639 wait complete. 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.849+0000 7f336bfd0640 1 Processor -- start 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.849+0000 7f336bfd0640 1 -- start start 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336bfd0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33640b95a0 0x7f33641f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336bfd0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33640b9a10 0x7f33640b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336bfd0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336415df00 0x7f33640b5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336bfd0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33640cedf0 con 0x7f336415df00 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336bfd0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33640cec70 con 0x7f33640b9a10 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336bfd0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33640b5e10 con 0x7f33640b95a0 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336a56d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336415df00 0x7f33640b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336a56d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336415df00 0x7f33640b5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43356/0 (socket says 172.21.15.27:43356) 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336a56d640 1 -- 172.21.15.27:0/2754445017 learned_addr learned my addr 172.21.15.27:0/2754445017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336956b640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33640b9a10 0x7f33640b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f3369d6c640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33640b95a0 0x7f33641f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336a56d640 1 -- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33640b95a0 msgr2=0x7f33641f30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336a56d640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33640b95a0 0x7f33641f30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336a56d640 1 -- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33640b9a10 msgr2=0x7f33640b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336a56d640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33640b9a10 0x7f33640b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336a56d640 1 -- 172.21.15.27:0/2754445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33640c8aa0 con 0x7f336415df00 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f336956b640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33640b9a10 0x7f33640b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.850+0000 7f3369d6c640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33640b95a0 0x7f33641f30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.851+0000 7f336a56d640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336415df00 0x7f33640b5250 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f335c02c4b0 tx=0x7f335c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.851+0000 7f335a7fc640 1 -- 172.21.15.27:0/2754445017 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c044030 con 0x7f336415df00 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.851+0000 7f335a7fc640 1 -- 172.21.15.27:0/2754445017 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c02ca50 con 0x7f336415df00 2024-03-06T07:33:38.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.851+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33640c8d80 con 0x7f336415df00 2024-03-06T07:33:38.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.851+0000 7f335a7fc640 1 -- 172.21.15.27:0/2754445017 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c039040 con 0x7f336415df00 2024-03-06T07:33:38.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.851+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33640c9340 con 0x7f336415df00 2024-03-06T07:33:38.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.852+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3364067ff0 con 0x7f336415df00 2024-03-06T07:33:38.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.852+0000 7f335a7fc640 1 -- 172.21.15.27:0/2754445017 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f335c03f470 con 0x7f336415df00 2024-03-06T07:33:38.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.853+0000 7f335a7fc640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f334806b4d0 0x7f334806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.853+0000 7f335a7fc640 1 -- 172.21.15.27:0/2754445017 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f335c0ac900 con 0x7f336415df00 2024-03-06T07:33:38.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.853+0000 7f3369d6c640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f334806b4d0 0x7f334806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.854+0000 7f3369d6c640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f334806b4d0 0x7f334806d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f336000a830 tx=0x7f3360008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:38.857+0000 7f335a7fc640 1 -- 172.21.15.27:0/2754445017 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f335c04e020 con 0x7f336415df00 2024-03-06T07:33:39.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.078+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f336406b780 con 0x7f336415df00 2024-03-06T07:33:39.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.078+0000 7f335a7fc640 1 -- 172.21.15.27:0/2754445017 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f335c02cc20 con 0x7f336415df00 2024-03-06T07:33:39.081 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:39.081 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:39.081 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f334806b4d0 msgr2=0x7f334806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f334806b4d0 0x7f334806d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f336000a830 tx=0x7f3360008040 comp rx=0 tx=0).stop 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336415df00 msgr2=0x7f33640b5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336415df00 0x7f33640b5250 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f335c02c4b0 tx=0x7f335c004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 shutdown_connections 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f334806b4d0 0x7f334806d990 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f336415df00 0x7f33640b5250 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33640b9a10 0x7f33640b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 --2- 172.21.15.27:0/2754445017 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f33640b95a0 0x7f33641f30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 >> 172.21.15.27:0/2754445017 conn(0x7f3364071960 msgr2=0x7f33640b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 shutdown_connections 2024-03-06T07:33:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:39.081+0000 7f336bfd0640 1 -- 172.21.15.27:0/2754445017 wait complete. 2024-03-06T07:33:39.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:39.296 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:40.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:40.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.429+0000 7fb6bffc9640 1 Processor -- start 2024-03-06T07:33:40.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.429+0000 7fb6bffc9640 1 -- start start 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.429+0000 7fb6bffc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b815df00 0x7fb6b80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bffc9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb6b80b95a0 0x7fb6b80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bffc9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b80cea30 con 0x7fb6b80b9a30 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bffc9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b80d2af0 con 0x7fb6b80b95a0 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bffc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80cf3c0 con 0x7fb6b815df00 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bdd65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b815df00 0x7fb6b80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bd564640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb6b80b95a0 0x7fb6b80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6be566640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb6b80b9a30 0x7fb6b80d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:48658/0 (socket says 172.21.15.27:48658) 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bdd65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b815df00 0x7fb6b80d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:57848/0 (socket says 172.21.15.27:57848) 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6be566640 1 -- 172.21.15.27:0/1177843390 learned_addr learned my addr 172.21.15.27:0/1177843390 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bdd65640 1 -- 172.21.15.27:0/1177843390 >> v2:172.21.15.69:3300/0 conn(0x7fb6b80b95a0 msgr2=0x7fb6b80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bdd65640 1 --2- 172.21.15.27:0/1177843390 >> v2:172.21.15.69:3300/0 conn(0x7fb6b80b95a0 0x7fb6b80b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bdd65640 1 -- 172.21.15.27:0/1177843390 >> v1:172.21.15.69:6789/0 conn(0x7fb6b80b9a30 legacy=0x7fb6b80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.430+0000 7fb6bdd65640 1 -- 172.21.15.27:0/1177843390 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b80cf0e0 con 0x7fb6b815df00 2024-03-06T07:33:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.431+0000 7fb6bdd65640 1 --2- 172.21.15.27:0/1177843390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b815df00 0x7fb6b80d25b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40097a0 tx=0x7fb6b402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a7bc47a3d523a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.431+0000 7fb6bcd63640 1 -- 172.21.15.27:0/1177843390 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4035020 con 0x7fb6b815df00 2024-03-06T07:33:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.431+0000 7fb6bcd63640 1 -- 172.21.15.27:0/1177843390 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b4031830 con 0x7fb6b815df00 2024-03-06T07:33:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.431+0000 7fb6bcd63640 1 -- 172.21.15.27:0/1177843390 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4031c00 con 0x7fb6b815df00 2024-03-06T07:33:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.432+0000 7fb6bffc9640 1 -- 172.21.15.27:0/1177843390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b815df00 msgr2=0x7fb6b80d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.432+0000 7fb6bffc9640 1 --2- 172.21.15.27:0/1177843390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b815df00 0x7fb6b80d25b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40097a0 tx=0x7fb6b402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.432+0000 7fb6bffc9640 1 -- 172.21.15.27:0/1177843390 shutdown_connections 2024-03-06T07:33:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.432+0000 7fb6bffc9640 1 --2- 172.21.15.27:0/1177843390 >> v2:172.21.15.69:3300/0 conn(0x7fb6b80b95a0 0x7fb6b80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.432+0000 7fb6bffc9640 1 --2- 172.21.15.27:0/1177843390 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b815df00 0x7fb6b80d25b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.432+0000 7fb6bffc9640 1 -- 172.21.15.27:0/1177843390 >> 172.21.15.27:0/1177843390 conn(0x7fb6b8071960 msgr2=0x7fb6b8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.432+0000 7fb6bffc9640 1 -- 172.21.15.27:0/1177843390 shutdown_connections 2024-03-06T07:33:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.432+0000 7fb6bffc9640 1 -- 172.21.15.27:0/1177843390 wait complete. 2024-03-06T07:33:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.433+0000 7fb6bffc9640 1 Processor -- start 2024-03-06T07:33:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.433+0000 7fb6bffc9640 1 -- start start 2024-03-06T07:33:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6bffc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b80b95a0 0x7fb6b80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6bffc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6b815df00 0x7fb6b8158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6bffc9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b81610c0 0x7fb6b8158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6bffc9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80cec70 con 0x7fb6b81610c0 2024-03-06T07:33:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6bffc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80ce750 con 0x7fb6b815df00 2024-03-06T07:33:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6bffc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80cedf0 con 0x7fb6b80b95a0 2024-03-06T07:33:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6be566640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b81610c0 0x7fb6b8158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6be566640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b81610c0 0x7fb6b8158a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43370/0 (socket says 172.21.15.27:43370) 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6be566640 1 -- 172.21.15.27:0/4047108254 learned_addr learned my addr 172.21.15.27:0/4047108254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6bd564640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6b815df00 0x7fb6b8158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6be566640 1 -- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b80b95a0 msgr2=0x7fb6b80c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6be566640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b80b95a0 0x7fb6b80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6be566640 1 -- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6b815df00 msgr2=0x7fb6b8158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6be566640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6b815df00 0x7fb6b8158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6be566640 1 -- 172.21.15.27:0/4047108254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b80c8be0 con 0x7fb6b81610c0 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6bd564640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6b815df00 0x7fb6b8158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.434+0000 7fb6be566640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b81610c0 0x7fb6b8158a70 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000caf0 tx=0x7fb6b000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.435+0000 7fb6ae7fc640 1 -- 172.21.15.27:0/4047108254 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0017070 con 0x7fb6b81610c0 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.435+0000 7fb6ae7fc640 1 -- 172.21.15.27:0/4047108254 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b000aca0 con 0x7fb6b81610c0 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.435+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b80c8ec0 con 0x7fb6b81610c0 2024-03-06T07:33:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.435+0000 7fb6ae7fc640 1 -- 172.21.15.27:0/4047108254 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0012600 con 0x7fb6b81610c0 2024-03-06T07:33:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.435+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b80c9480 con 0x7fb6b81610c0 2024-03-06T07:33:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.436+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b8067ff0 con 0x7fb6b81610c0 2024-03-06T07:33:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.436+0000 7fb6ae7fc640 1 -- 172.21.15.27:0/4047108254 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6b001a070 con 0x7fb6b81610c0 2024-03-06T07:33:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.437+0000 7fb6ae7fc640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb69c06b4d0 0x7fb69c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.437+0000 7fb6ae7fc640 1 -- 172.21.15.27:0/4047108254 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb6b0089e60 con 0x7fb6b81610c0 2024-03-06T07:33:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.437+0000 7fb6bdd65640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb69c06b4d0 0x7fb69c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.437+0000 7fb6bdd65640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb69c06b4d0 0x7fb69c06d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4036000 tx=0x7fb6b4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.446+0000 7fb6ae7fc640 1 -- 172.21.15.27:0/4047108254 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6b0058670 con 0x7fb6b81610c0 2024-03-06T07:33:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.664+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6b80cce40 con 0x7fb6b81610c0 2024-03-06T07:33:40.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.665+0000 7fb6ae7fc640 1 -- 172.21.15.27:0/4047108254 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fb6b005c710 con 0x7fb6b81610c0 2024-03-06T07:33:40.667 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:40.667 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:40.667 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb69c06b4d0 msgr2=0x7fb69c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb69c06b4d0 0x7fb69c06d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4036000 tx=0x7fb6b4030000 comp rx=0 tx=0).stop 2024-03-06T07:33:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b81610c0 msgr2=0x7fb6b8158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b81610c0 0x7fb6b8158a70 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000caf0 tx=0x7fb6b000a060 comp rx=0 tx=0).stop 2024-03-06T07:33:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 shutdown_connections 2024-03-06T07:33:40.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb69c06b4d0 0x7fb69c06d990 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b81610c0 0x7fb6b8158a70 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6b815df00 0x7fb6b8158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 --2- 172.21.15.27:0/4047108254 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb6b80b95a0 0x7fb6b80c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 >> 172.21.15.27:0/4047108254 conn(0x7fb6b8071960 msgr2=0x7fb6b80d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 shutdown_connections 2024-03-06T07:33:40.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:40.667+0000 7fb6bffc9640 1 -- 172.21.15.27:0/4047108254 wait complete. 2024-03-06T07:33:40.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:40.882 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:41.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:41.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.988+0000 7fe17ced4640 1 Processor -- start 2024-03-06T07:33:41.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.988+0000 7fe17ced4640 1 -- start start 2024-03-06T07:33:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.989+0000 7fe17ced4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1780d0570 0x7fe17815c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.989+0000 7fe17ced4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe17815b120 0x7fe178158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.989+0000 7fe17ced4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe17806ec10 0x7fe17806f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.989+0000 7fe17ced4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe17815d4d0 con 0x7fe1780d0570 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.989+0000 7fe17ced4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe17815c8c0 con 0x7fe17806ec10 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.989+0000 7fe17ced4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17815cba0 con 0x7fe17815b120 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.989+0000 7fe17659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1780d0570 0x7fe17815c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.989+0000 7fe17659c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1780d0570 0x7fe17815c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43372/0 (socket says 172.21.15.27:43372) 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.989+0000 7fe17659c640 1 -- 172.21.15.27:0/1260983705 learned_addr learned my addr 172.21.15.27:0/1260983705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.990+0000 7fe17659c640 1 -- 172.21.15.27:0/1260983705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe17815b120 msgr2=0x7fe178158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.990+0000 7fe17659c640 1 --2- 172.21.15.27:0/1260983705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe17815b120 0x7fe178158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.990+0000 7fe17659c640 1 -- 172.21.15.27:0/1260983705 >> v2:172.21.15.69:3300/0 conn(0x7fe17806ec10 msgr2=0x7fe17806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.990+0000 7fe17659c640 1 --2- 172.21.15.27:0/1260983705 >> v2:172.21.15.69:3300/0 conn(0x7fe17806ec10 0x7fe17806f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.990+0000 7fe17659c640 1 -- 172.21.15.27:0/1260983705 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17815ce80 con 0x7fe1780d0570 2024-03-06T07:33:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.990+0000 7fe17659c640 1 --2- 172.21.15.27:0/1260983705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1780d0570 0x7fe17815c2e0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fe1780b36b0 tx=0x7fe16802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41626ccf4422aa80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17559a640 1 -- 172.21.15.27:0/1260983705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16803b070 con 0x7fe1780d0570 2024-03-06T07:33:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17559a640 1 -- 172.21.15.27:0/1260983705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe168036440 con 0x7fe1780d0570 2024-03-06T07:33:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17559a640 1 -- 172.21.15.27:0/1260983705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe168036810 con 0x7fe1780d0570 2024-03-06T07:33:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17ced4640 1 -- 172.21.15.27:0/1260983705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1780d0570 msgr2=0x7fe17815c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17ced4640 1 --2- 172.21.15.27:0/1260983705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1780d0570 0x7fe17815c2e0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fe1780b36b0 tx=0x7fe16802e0e0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17ced4640 1 -- 172.21.15.27:0/1260983705 shutdown_connections 2024-03-06T07:33:41.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17ced4640 1 --2- 172.21.15.27:0/1260983705 >> v2:172.21.15.69:3300/0 conn(0x7fe17806ec10 0x7fe17806f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17ced4640 1 --2- 172.21.15.27:0/1260983705 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe17815b120 0x7fe178158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17ced4640 1 --2- 172.21.15.27:0/1260983705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1780d0570 0x7fe17815c2e0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17ced4640 1 -- 172.21.15.27:0/1260983705 >> 172.21.15.27:0/1260983705 conn(0x7fe1780d0a20 msgr2=0x7fe1780d0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17ced4640 1 -- 172.21.15.27:0/1260983705 shutdown_connections 2024-03-06T07:33:41.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.991+0000 7fe17ced4640 1 -- 172.21.15.27:0/1260983705 wait complete. 2024-03-06T07:33:41.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.993+0000 7fe17ced4640 1 Processor -- start 2024-03-06T07:33:41.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.993+0000 7fe17ced4640 1 -- start start 2024-03-06T07:33:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.993+0000 7fe17ced4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe17806ec10 0x7fe1780afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.993+0000 7fe17ced4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1780d0570 0x7fe178147380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.993+0000 7fe17ced4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe17815b120 0x7fe178147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.993+0000 7fe17ced4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe178072bd0 con 0x7fe17806ec10 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.993+0000 7fe17ced4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1781592a0 con 0x7fe17815b120 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.993+0000 7fe17ced4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17815d960 con 0x7fe1780d0570 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.993+0000 7fe17659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe17806ec10 0x7fe1780afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe17806ec10 0x7fe1780afdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43384/0 (socket says 172.21.15.27:43384) 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17659c640 1 -- 172.21.15.27:0/3732344482 learned_addr learned my addr 172.21.15.27:0/3732344482 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe176d9d640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe17815b120 0x7fe178147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe175d9b640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1780d0570 0x7fe178147380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17659c640 1 -- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1780d0570 msgr2=0x7fe178147380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17659c640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1780d0570 0x7fe178147380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17659c640 1 -- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe17815b120 msgr2=0x7fe178147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17659c640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe17815b120 0x7fe178147cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17659c640 1 -- 172.21.15.27:0/3732344482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1780ae260 con 0x7fe17806ec10 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe176d9d640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe17815b120 0x7fe178147cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17659c640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe17806ec10 0x7fe1780afdb0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fe16802e120 tx=0x7fe168004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe166ffd640 1 -- 172.21.15.27:0/3732344482 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe168046070 con 0x7fe17806ec10 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe166ffd640 1 -- 172.21.15.27:0/3732344482 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe168034990 con 0x7fe17806ec10 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1780ae540 con 0x7fe17806ec10 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe166ffd640 1 -- 172.21.15.27:0/3732344482 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16803b070 con 0x7fe17806ec10 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.994+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe178152c70 con 0x7fe17806ec10 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.995+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe178068700 con 0x7fe17806ec10 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.996+0000 7fe166ffd640 1 -- 172.21.15.27:0/3732344482 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe168049070 con 0x7fe17806ec10 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.996+0000 7fe166ffd640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe15806b4d0 0x7fe15806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.996+0000 7fe166ffd640 1 -- 172.21.15.27:0/3732344482 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe1680ac490 con 0x7fe17806ec10 2024-03-06T07:33:42.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.996+0000 7fe175d9b640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe15806b4d0 0x7fe15806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:41.997+0000 7fe175d9b640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe15806b4d0 0x7fe15806d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe1780b5850 tx=0x7fe16c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.001+0000 7fe166ffd640 1 -- 172.21.15.27:0/3732344482 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe16807b2f0 con 0x7fe17806ec10 2024-03-06T07:33:42.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.256+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe17806c9d0 con 0x7fe17806ec10 2024-03-06T07:33:42.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.257+0000 7fe166ffd640 1 -- 172.21.15.27:0/3732344482 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fe16807abf0 con 0x7fe17806ec10 2024-03-06T07:33:42.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:42.259 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:42.260 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe15806b4d0 msgr2=0x7fe15806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe15806b4d0 0x7fe15806d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe1780b5850 tx=0x7fe16c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe17806ec10 msgr2=0x7fe1780afdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe17806ec10 0x7fe1780afdb0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fe16802e120 tx=0x7fe168004730 comp rx=0 tx=0).stop 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 shutdown_connections 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe15806b4d0 0x7fe15806d990 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe17815b120 0x7fe178147cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1780d0570 0x7fe178147380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 --2- 172.21.15.27:0/3732344482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe17806ec10 0x7fe1780afdb0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 >> 172.21.15.27:0/3732344482 conn(0x7fe1780d0a20 msgr2=0x7fe178151720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 shutdown_connections 2024-03-06T07:33:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:42.259+0000 7fe17ced4640 1 -- 172.21.15.27:0/3732344482 wait complete. 2024-03-06T07:33:42.473 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:42.474 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:43.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:43.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.580+0000 7f6a5ea64640 1 Processor -- start 2024-03-06T07:33:43.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.580+0000 7f6a5ea64640 1 -- start start 2024-03-06T07:33:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a5ea64640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b95a0 0x7f6a580b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a5ea64640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a580b9a10 0x7f6a580b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a5ea64640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a580cea30 con 0x7f6a5815df00 2024-03-06T07:33:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a5ea64640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a580cf3c0 con 0x7f6a580b9a10 2024-03-06T07:33:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a5ea64640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a580cf0e0 con 0x7f6a580b95a0 2024-03-06T07:33:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b95a0 0x7f6a580b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a5d001640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6a580b9a10 0x7f6a580b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a5c800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6a5815df00 0x7f6a580d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:42046/0 (socket says 172.21.15.27:42046) 2024-03-06T07:33:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b95a0 0x7f6a580b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:35034/0 (socket says 172.21.15.27:35034) 2024-03-06T07:33:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.581+0000 7f6a5c800640 1 -- 172.21.15.27:0/577581112 learned_addr learned my addr 172.21.15.27:0/577581112 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.582+0000 7f6a57fff640 1 -- 172.21.15.27:0/577581112 >> v2:172.21.15.69:3300/0 conn(0x7f6a580b9a10 msgr2=0x7f6a580b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.582+0000 7f6a57fff640 1 --2- 172.21.15.27:0/577581112 >> v2:172.21.15.69:3300/0 conn(0x7f6a580b9a10 0x7f6a580b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.582+0000 7f6a57fff640 1 -- 172.21.15.27:0/577581112 >> v1:172.21.15.69:6789/0 conn(0x7f6a5815df00 legacy=0x7f6a580d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.582+0000 7f6a57fff640 1 -- 172.21.15.27:0/577581112 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a580ce470 con 0x7f6a580b95a0 2024-03-06T07:33:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a57fff640 1 --2- 172.21.15.27:0/577581112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b95a0 0x7f6a580b5700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c0097a0 tx=0x7f6a4c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48f98dcdcbb4775a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a577fe640 1 -- 172.21.15.27:0/577581112 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c035020 con 0x7f6a580b95a0 2024-03-06T07:33:43.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a577fe640 1 -- 172.21.15.27:0/577581112 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4c031830 con 0x7f6a580b95a0 2024-03-06T07:33:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a577fe640 1 -- 172.21.15.27:0/577581112 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c031c00 con 0x7f6a580b95a0 2024-03-06T07:33:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a5ea64640 1 -- 172.21.15.27:0/577581112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b95a0 msgr2=0x7f6a580b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a5ea64640 1 --2- 172.21.15.27:0/577581112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b95a0 0x7f6a580b5700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c0097a0 tx=0x7f6a4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a5ea64640 1 -- 172.21.15.27:0/577581112 shutdown_connections 2024-03-06T07:33:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a5ea64640 1 --2- 172.21.15.27:0/577581112 >> v2:172.21.15.69:3300/0 conn(0x7f6a580b9a10 0x7f6a580b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a5ea64640 1 --2- 172.21.15.27:0/577581112 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b95a0 0x7f6a580b5700 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a5ea64640 1 -- 172.21.15.27:0/577581112 >> 172.21.15.27:0/577581112 conn(0x7f6a58071960 msgr2=0x7f6a58071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a5ea64640 1 -- 172.21.15.27:0/577581112 shutdown_connections 2024-03-06T07:33:43.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.583+0000 7f6a5ea64640 1 -- 172.21.15.27:0/577581112 wait complete. 2024-03-06T07:33:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.585+0000 7f6a5ea64640 1 Processor -- start 2024-03-06T07:33:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.585+0000 7f6a5ea64640 1 -- start start 2024-03-06T07:33:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.585+0000 7f6a5ea64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a580b95a0 0x7f6a58155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.585+0000 7f6a5ea64640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b9a10 0x7f6a580d3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5ea64640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a5815df00 0x7f6a580d3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5ea64640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a580cedf0 con 0x7f6a580b95a0 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5ea64640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a580cec70 con 0x7f6a5815df00 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5ea64640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a580d3220 con 0x7f6a580b9a10 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5c800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a580b95a0 0x7f6a58155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5c800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a580b95a0 0x7f6a58155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52628/0 (socket says 172.21.15.27:52628) 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5c800640 1 -- 172.21.15.27:0/3587312062 learned_addr learned my addr 172.21.15.27:0/3587312062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a57fff640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b9a10 0x7f6a580d3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5d001640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a5815df00 0x7f6a580d3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5c800640 1 -- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b9a10 msgr2=0x7f6a580d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5c800640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b9a10 0x7f6a580d3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5c800640 1 -- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a5815df00 msgr2=0x7f6a580d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5c800640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a5815df00 0x7f6a580d3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5c800640 1 -- 172.21.15.27:0/3587312062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a580cc280 con 0x7f6a580b95a0 2024-03-06T07:33:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5c800640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a580b95a0 0x7f6a58155310 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f6a4400c3c0 tx=0x7f6a4400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a557fa640 1 -- 172.21.15.27:0/3587312062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a44017070 con 0x7f6a580b95a0 2024-03-06T07:33:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a557fa640 1 -- 172.21.15.27:0/3587312062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a44005ce0 con 0x7f6a580b95a0 2024-03-06T07:33:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5815f360 con 0x7f6a580b95a0 2024-03-06T07:33:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.586+0000 7f6a557fa640 1 -- 172.21.15.27:0/3587312062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a440072f0 con 0x7f6a580b95a0 2024-03-06T07:33:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.587+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5815fc00 con 0x7f6a580b95a0 2024-03-06T07:33:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.587+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a580cc450 con 0x7f6a580b95a0 2024-03-06T07:33:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.588+0000 7f6a557fa640 1 -- 172.21.15.27:0/3587312062 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a4401a070 con 0x7f6a580b95a0 2024-03-06T07:33:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.588+0000 7f6a557fa640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3c06b4d0 0x7f6a3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.589+0000 7f6a557fa640 1 -- 172.21.15.27:0/3587312062 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6a440894d0 con 0x7f6a580b95a0 2024-03-06T07:33:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.589+0000 7f6a57fff640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3c06b4d0 0x7f6a3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.589+0000 7f6a57fff640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3c06b4d0 0x7f6a3c06d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c036000 tx=0x7f6a4c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.598+0000 7f6a557fa640 1 -- 172.21.15.27:0/3587312062 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a44057d70 con 0x7f6a580b95a0 2024-03-06T07:33:43.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.818+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a58067ff0 con 0x7f6a580b95a0 2024-03-06T07:33:43.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.819+0000 7f6a557fa640 1 -- 172.21.15.27:0/3587312062 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f6a4405be10 con 0x7f6a580b95a0 2024-03-06T07:33:43.821 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:43.821 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:43.822 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3c06b4d0 msgr2=0x7f6a3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3c06b4d0 0x7f6a3c06d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c036000 tx=0x7f6a4c008030 comp rx=0 tx=0).stop 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a580b95a0 msgr2=0x7f6a58155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a580b95a0 0x7f6a58155310 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f6a4400c3c0 tx=0x7f6a4400c880 comp rx=0 tx=0).stop 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 shutdown_connections 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6a3c06b4d0 0x7f6a3c06d990 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a5815df00 0x7f6a580d3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6a580b9a10 0x7f6a580d3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 --2- 172.21.15.27:0/3587312062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a580b95a0 0x7f6a58155310 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 >> 172.21.15.27:0/3587312062 conn(0x7f6a58071960 msgr2=0x7f6a58071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 shutdown_connections 2024-03-06T07:33:43.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:43.821+0000 7f6a5ea64640 1 -- 172.21.15.27:0/3587312062 wait complete. 2024-03-06T07:33:44.036 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:44.036 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:45.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:45.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.143+0000 7f99eb09d640 1 Processor -- start 2024-03-06T07:33:45.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.143+0000 7f99eb09d640 1 -- start start 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.144+0000 7f99eb09d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99e415df00 0x7f99e40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.144+0000 7f99eb09d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e40cea30 con 0x7f99e40b95a0 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.144+0000 7f99eb09d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e40d2af0 con 0x7f99e40b9a30 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.144+0000 7f99eb09d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e40cf3c0 con 0x7f99e415df00 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.144+0000 7f99e3fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f99e40b95a0 0x7f99e40b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60866/0 (socket says 172.21.15.27:60866) 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.144+0000 7f99e3fff640 1 -- 172.21.15.27:0/2403798174 learned_addr learned my addr 172.21.15.27:0/2403798174 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e8e39640 1 --2- 172.21.15.27:0/2403798174 >> v2:172.21.15.69:3300/0 conn(0x7f99e415df00 0x7f99e40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13133743 0 0) 0x7f99e40cea30 con 0x7f99e40b95a0 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d4003520 con 0x7f99e40b95a0 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267897514 0 0) 0x7f99e40d2af0 con 0x7f99e40b9a30 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e40cea30 con 0x7f99e40b9a30 2024-03-06T07:33:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 377135045 0 0) 0x7f99d4003520 con 0x7f99e40b95a0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 >> v2:172.21.15.69:3300/0 conn(0x7f99e415df00 msgr2=0x7f99e40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 --2- 172.21.15.27:0/2403798174 >> v2:172.21.15.69:3300/0 conn(0x7f99e415df00 0x7f99e40d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 >> v1:172.21.15.69:6789/0 conn(0x7f99e40b9a30 legacy=0x7f99e40d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e40cf0e0 con 0x7f99e40b95a0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e8e39640 1 --2- 172.21.15.27:0/2403798174 >> v2:172.21.15.69:3300/0 conn(0x7f99e415df00 0x7f99e40d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f99d0001670 con 0x7f99e40b95a0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 >> v1:172.21.15.27:6789/0 conn(0x7f99e40b95a0 legacy=0x7f99e40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4003b40 0x7f99d4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4004800 0x7f99d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d40051c0 0x7f99d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40cf3c0 con 0x7f99d40051c0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d400c040 con 0x7f99d4003b40 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d400c240 con 0x7f99d4004800 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e963a640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d40051c0 0x7f99d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e8e39640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4003b40 0x7f99d4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.145+0000 7f99e3fff640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4004800 0x7f99d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e963a640 1 -- 172.21.15.27:0/2403798174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4004800 msgr2=0x7f99d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e963a640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99d4004800 0x7f99d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e963a640 1 -- 172.21.15.27:0/2403798174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4003b40 msgr2=0x7f99d4003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e963a640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d4003b40 0x7f99d4003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e963a640 1 -- 172.21.15.27:0/2403798174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4003520 con 0x7f99d40051c0 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e963a640 1 -- 172.21.15.27:0/2403798174 reap_dead start 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e963a640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d40051c0 0x7f99d400b9c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f99dc001fd0 tx=0x7f99d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed24f4ce761b14d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc001bf0 con 0x7f99d40051c0 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99dc003b70 con 0x7f99d40051c0 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.146+0000 7f99e37fe640 1 -- 172.21.15.27:0/2403798174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc0050d0 con 0x7f99d40051c0 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.147+0000 7f99eb09d640 1 -- 172.21.15.27:0/2403798174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d40051c0 msgr2=0x7f99d400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.147+0000 7f99eb09d640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d40051c0 0x7f99d400b9c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f99dc001fd0 tx=0x7f99d4005830 comp rx=0 tx=0).stop 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.147+0000 7f99eb09d640 1 -- 172.21.15.27:0/2403798174 shutdown_connections 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.147+0000 7f99eb09d640 1 --2- 172.21.15.27:0/2403798174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d40051c0 0x7f99d400b9c0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.147+0000 7f99eb09d640 1 -- 172.21.15.27:0/2403798174 >> 172.21.15.27:0/2403798174 conn(0x7f99e4071960 msgr2=0x7f99e4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.147+0000 7f99eb09d640 1 -- 172.21.15.27:0/2403798174 shutdown_connections 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.147+0000 7f99eb09d640 1 -- 172.21.15.27:0/2403798174 wait complete. 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.149+0000 7f99eb09d640 1 Processor -- start 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.149+0000 7f99eb09d640 1 -- start start 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99eb09d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d40051c0 0x7f99e40c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99eb09d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99e41f0da0 0x7f99e415e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99eb09d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e415ed50 0x7f99e41608b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99eb09d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40cec70 con 0x7f99e415ed50 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99eb09d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40ce750 con 0x7f99d40051c0 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99eb09d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40cedf0 con 0x7f99e41f0da0 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99e963a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e415ed50 0x7f99e41608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99e963a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e415ed50 0x7f99e41608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52652/0 (socket says 172.21.15.27:52652) 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99e963a640 1 -- 172.21.15.27:0/4294227562 learned_addr learned my addr 172.21.15.27:0/4294227562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99e3fff640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99e41f0da0 0x7f99e415e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.150+0000 7f99e8e39640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d40051c0 0x7f99e40c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99e963a640 1 -- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99e41f0da0 msgr2=0x7f99e415e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99e963a640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99e41f0da0 0x7f99e415e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99e963a640 1 -- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d40051c0 msgr2=0x7f99e40c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99e963a640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d40051c0 0x7f99e40c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99e963a640 1 -- 172.21.15.27:0/4294227562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e40c89e0 con 0x7f99e415ed50 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99e963a640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e415ed50 0x7f99e41608b0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0012e0 tx=0x7f99dc001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99e17fa640 1 -- 172.21.15.27:0/4294227562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc001bf0 con 0x7f99e415ed50 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99e17fa640 1 -- 172.21.15.27:0/4294227562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99dc003970 con 0x7f99e415ed50 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99e17fa640 1 -- 172.21.15.27:0/4294227562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc005810 con 0x7f99e415ed50 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e415f260 con 0x7f99e415ed50 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.151+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e40cc080 con 0x7f99e415ed50 2024-03-06T07:33:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.152+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e4067ff0 con 0x7f99e415ed50 2024-03-06T07:33:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.153+0000 7f99e17fa640 1 -- 172.21.15.27:0/4294227562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99dc02c570 con 0x7f99e415ed50 2024-03-06T07:33:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.153+0000 7f99e17fa640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99c806b4d0 0x7f99c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.153+0000 7f99e17fa640 1 -- 172.21.15.27:0/4294227562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f99dc0aaf90 con 0x7f99e415ed50 2024-03-06T07:33:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.153+0000 7f99e8e39640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99c806b4d0 0x7f99c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.154+0000 7f99e8e39640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99c806b4d0 0x7f99c806d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f99d80093d0 tx=0x7f99d8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.162+0000 7f99e17fa640 1 -- 172.21.15.27:0/4294227562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99dc003b40 con 0x7f99e415ed50 2024-03-06T07:33:45.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.383+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99e40d4570 con 0x7f99e415ed50 2024-03-06T07:33:45.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.384+0000 7f99e17fa640 1 -- 172.21.15.27:0/4294227562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f99dc079df0 con 0x7f99e415ed50 2024-03-06T07:33:45.385 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:45.385 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:45.385 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99c806b4d0 msgr2=0x7f99c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99c806b4d0 0x7f99c806d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f99d80093d0 tx=0x7f99d8009190 comp rx=0 tx=0).stop 2024-03-06T07:33:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e415ed50 msgr2=0x7f99e41608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e415ed50 0x7f99e41608b0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0012e0 tx=0x7f99dc001650 comp rx=0 tx=0).stop 2024-03-06T07:33:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 shutdown_connections 2024-03-06T07:33:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f99c806b4d0 0x7f99c806d990 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e415ed50 0x7f99e41608b0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f99e41f0da0 0x7f99e415e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 --2- 172.21.15.27:0/4294227562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99d40051c0 0x7f99e40c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 >> 172.21.15.27:0/4294227562 conn(0x7f99e4071960 msgr2=0x7f99e40b58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 shutdown_connections 2024-03-06T07:33:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:45.386+0000 7f99eb09d640 1 -- 172.21.15.27:0/4294227562 wait complete. 2024-03-06T07:33:45.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:45.599 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:46.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:46.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.706+0000 7f311864e640 1 Processor -- start 2024-03-06T07:33:46.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.706+0000 7f311864e640 1 -- start start 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f311864e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f311015df00 0x7f31100d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f311864e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f31100b95a0 0x7f31100b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f311864e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100cea30 con 0x7f31100b9a30 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f311864e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31100d2af0 con 0x7f311015df00 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f311864e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31100cf3c0 con 0x7f31100b95a0 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f31163ea640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f311015df00 0x7f31100d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f31163ea640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f311015df00 0x7f31100d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52654/0 (socket says 172.21.15.27:52654) 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f3116beb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f31100b9a30 0x7f31100d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60880/0 (socket says 172.21.15.27:60880) 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f31163ea640 1 -- 172.21.15.27:0/706960769 learned_addr learned my addr 172.21.15.27:0/706960769 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f3115be9640 1 --2- 172.21.15.27:0/706960769 >> v2:172.21.15.69:3300/0 conn(0x7f31100b95a0 0x7f31100b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.707+0000 7f31163ea640 1 -- 172.21.15.27:0/706960769 >> v2:172.21.15.69:3300/0 conn(0x7f31100b95a0 msgr2=0x7f31100b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.708+0000 7f31163ea640 1 --2- 172.21.15.27:0/706960769 >> v2:172.21.15.69:3300/0 conn(0x7f31100b95a0 0x7f31100b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.708+0000 7f31163ea640 1 -- 172.21.15.27:0/706960769 >> v1:172.21.15.27:6789/0 conn(0x7f31100b9a30 legacy=0x7f31100d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.708+0000 7f31153e8640 1 -- 172.21.15.27:0/706960769 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347006444 0 0) 0x7f31100cea30 con 0x7f31100b9a30 2024-03-06T07:33:46.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.708+0000 7f31163ea640 1 -- 172.21.15.27:0/706960769 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100cf0e0 con 0x7f311015df00 2024-03-06T07:33:46.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.709+0000 7f31163ea640 1 --2- 172.21.15.27:0/706960769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311015df00 0x7f31100d25b0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f31100d1760 tx=0x7f310c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8831e964ea4126 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.709+0000 7f31153e8640 1 -- 172.21.15.27:0/706960769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c034070 con 0x7f311015df00 2024-03-06T07:33:46.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.709+0000 7f31153e8640 1 -- 172.21.15.27:0/706960769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c030860 con 0x7f311015df00 2024-03-06T07:33:46.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.709+0000 7f31153e8640 1 -- 172.21.15.27:0/706960769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c030c30 con 0x7f311015df00 2024-03-06T07:33:46.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.709+0000 7f311864e640 1 -- 172.21.15.27:0/706960769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311015df00 msgr2=0x7f31100d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.709+0000 7f311864e640 1 --2- 172.21.15.27:0/706960769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311015df00 0x7f31100d25b0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f31100d1760 tx=0x7f310c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.710+0000 7f311864e640 1 -- 172.21.15.27:0/706960769 shutdown_connections 2024-03-06T07:33:46.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.710+0000 7f311864e640 1 --2- 172.21.15.27:0/706960769 >> v2:172.21.15.69:3300/0 conn(0x7f31100b95a0 0x7f31100b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.710+0000 7f311864e640 1 --2- 172.21.15.27:0/706960769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311015df00 0x7f31100d25b0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.710+0000 7f311864e640 1 -- 172.21.15.27:0/706960769 >> 172.21.15.27:0/706960769 conn(0x7f3110071960 msgr2=0x7f3110071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.710+0000 7f311864e640 1 -- 172.21.15.27:0/706960769 shutdown_connections 2024-03-06T07:33:46.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.710+0000 7f311864e640 1 -- 172.21.15.27:0/706960769 wait complete. 2024-03-06T07:33:46.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.711+0000 7f311864e640 1 Processor -- start 2024-03-06T07:33:46.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.711+0000 7f311864e640 1 -- start start 2024-03-06T07:33:46.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f311864e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100b95a0 0x7f31101f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f311864e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f311015df00 0x7f31101f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f311864e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f311015e2e0 0x7f311015a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f311864e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100cec70 con 0x7f31100b95a0 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f311864e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100ce750 con 0x7f311015e2e0 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f311864e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31100cedf0 con 0x7f311015df00 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f31163ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100b95a0 0x7f31101f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f31163ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100b95a0 0x7f31101f4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52658/0 (socket says 172.21.15.27:52658) 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f31163ea640 1 -- 172.21.15.27:0/2326670193 learned_addr learned my addr 172.21.15.27:0/2326670193 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f3115be9640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f311015df00 0x7f31101f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f3116beb640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f311015e2e0 0x7f311015a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f31163ea640 1 -- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f311015df00 msgr2=0x7f31101f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f31163ea640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f311015df00 0x7f31101f0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f31163ea640 1 -- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f311015e2e0 msgr2=0x7f311015a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f31163ea640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f311015e2e0 0x7f311015a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f31163ea640 1 -- 172.21.15.27:0/2326670193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31101f47d0 con 0x7f31100b95a0 2024-03-06T07:33:46.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.712+0000 7f31163ea640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100b95a0 0x7f31101f4290 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f310c02b9e0 tx=0x7f310c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.713+0000 7f3106ffd640 1 -- 172.21.15.27:0/2326670193 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c034040 con 0x7f31100b95a0 2024-03-06T07:33:46.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.713+0000 7f3106ffd640 1 -- 172.21.15.27:0/2326670193 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c039d90 con 0x7f31100b95a0 2024-03-06T07:33:46.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.713+0000 7f3106ffd640 1 -- 172.21.15.27:0/2326670193 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c043610 con 0x7f31100b95a0 2024-03-06T07:33:46.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.713+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311015e7e0 con 0x7f31100b95a0 2024-03-06T07:33:46.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.713+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311015eda0 con 0x7f31100b95a0 2024-03-06T07:33:46.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.714+0000 7f3106ffd640 1 -- 172.21.15.27:0/2326670193 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f310c04a060 con 0x7f31100b95a0 2024-03-06T07:33:46.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.714+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3110067ff0 con 0x7f31100b95a0 2024-03-06T07:33:46.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.714+0000 7f3106ffd640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30f406b4d0 0x7f30f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.715+0000 7f3106ffd640 1 -- 172.21.15.27:0/2326670193 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f310c0ac1f0 con 0x7f31100b95a0 2024-03-06T07:33:46.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.715+0000 7f3115be9640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30f406b4d0 0x7f30f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.715+0000 7f3115be9640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30f406b4d0 0x7f30f406d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f31101f6d60 tx=0x7f3100008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.719+0000 7f3106ffd640 1 -- 172.21.15.27:0/2326670193 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31101f47d0 con 0x7f31100b95a0 2024-03-06T07:33:46.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.938+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31100689e0 con 0x7f31100b95a0 2024-03-06T07:33:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.938+0000 7f3106ffd640 1 -- 172.21.15.27:0/2326670193 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f310c03e070 con 0x7f31100b95a0 2024-03-06T07:33:46.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:46.940 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:46.940 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.940+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30f406b4d0 msgr2=0x7f30f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.940+0000 7f311864e640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30f406b4d0 0x7f30f406d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f31101f6d60 tx=0x7f3100008040 comp rx=0 tx=0).stop 2024-03-06T07:33:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.940+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100b95a0 msgr2=0x7f31101f4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.940+0000 7f311864e640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100b95a0 0x7f31101f4290 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f310c02b9e0 tx=0x7f310c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.941+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 shutdown_connections 2024-03-06T07:33:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.941+0000 7f311864e640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f30f406b4d0 0x7f30f406d990 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.941+0000 7f311864e640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f311015e2e0 0x7f311015a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.941+0000 7f311864e640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f311015df00 0x7f31101f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.941+0000 7f311864e640 1 --2- 172.21.15.27:0/2326670193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100b95a0 0x7f31101f4290 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.941+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 >> 172.21.15.27:0/2326670193 conn(0x7f3110071960 msgr2=0x7f31100b8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.941+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 shutdown_connections 2024-03-06T07:33:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:46.941+0000 7f311864e640 1 -- 172.21.15.27:0/2326670193 wait complete. 2024-03-06T07:33:47.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:47.154 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:48.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.260+0000 7f9a78a9d640 1 Processor -- start 2024-03-06T07:33:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.261+0000 7f9a78a9d640 1 -- start start 2024-03-06T07:33:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.261+0000 7f9a78a9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a740720c0 0x7f9a74165640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.261+0000 7f9a78a9d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a740b4120 con 0x7f9a740afb30 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.261+0000 7f9a78a9d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a740b4400 con 0x7f9a7415ac30 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.261+0000 7f9a78a9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740b46e0 con 0x7f9a740720c0 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a69d9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9a740afb30 0x7f9a740719b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60892/0 (socket says 172.21.15.27:60892) 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a69d9b640 1 -- 172.21.15.27:0/3624354825 learned_addr learned my addr 172.21.15.27:0/3624354825 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a72d9d640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a740720c0 0x7f9a74165640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173920445 0 0) 0x7f9a740b4120 con 0x7f9a740afb30 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a60003520 con 0x7f9a740afb30 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489855179 0 0) 0x7f9a740b4400 con 0x7f9a7415ac30 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a740b4120 con 0x7f9a7415ac30 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3202065230 0 0) 0x7f9a60003520 con 0x7f9a740afb30 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a740720c0 msgr2=0x7f9a74165640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a740720c0 0x7f9a74165640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 >> v1:172.21.15.69:6789/0 conn(0x7f9a7415ac30 legacy=0x7f9a740b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740b49c0 con 0x7f9a740afb30 2024-03-06T07:33:48.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f9a5c001670 con 0x7f9a740afb30 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 >> v1:172.21.15.27:6789/0 conn(0x7f9a740afb30 legacy=0x7f9a740719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a60003b70 0x7f9a60003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a740720c0 0x7f9a60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740b46e0 con 0x7f9a60004d40 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6000ba70 con 0x7f9a60003b70 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6000bc70 con 0x7f9a740720c0 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a72d9d640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a69d9b640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a740720c0 0x7f9a60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.262+0000 7f9a7259c640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a60003b70 0x7f9a60003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.263+0000 7f9a72d9d640 1 -- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a740720c0 msgr2=0x7f9a60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.263+0000 7f9a72d9d640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a740720c0 0x7f9a60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.263+0000 7f9a72d9d640 1 -- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a60003b70 msgr2=0x7f9a60003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.263+0000 7f9a72d9d640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a60003b70 0x7f9a60003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.263+0000 7f9a72d9d640 1 -- 172.21.15.27:0/3624354825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a60003520 con 0x7f9a60004d40 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.263+0000 7f9a72d9d640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a600051c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009f90 tx=0x7f9a60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270cbb71cd99eea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c039070 con 0x7f9a60004d40 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c02cbd0 con 0x7f9a60004d40 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a71d9b640 1 -- 172.21.15.27:0/3624354825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c009c10 con 0x7f9a60004d40 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a78a9d640 1 -- 172.21.15.27:0/3624354825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 msgr2=0x7f9a600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a78a9d640 1 --2- 172.21.15.27:0/3624354825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a600051c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009f90 tx=0x7f9a60005830 comp rx=0 tx=0).stop 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a72d9d640 1 -- 172.21.15.27:0/3624354825 reap_dead start 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a78a9d640 1 -- 172.21.15.27:0/3624354825 shutdown_connections 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a78a9d640 1 -- 172.21.15.27:0/3624354825 >> 172.21.15.27:0/3624354825 conn(0x7f9a740d0990 msgr2=0x7f9a740d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a78a9d640 1 -- 172.21.15.27:0/3624354825 shutdown_connections 2024-03-06T07:33:48.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.264+0000 7f9a78a9d640 1 -- 172.21.15.27:0/3624354825 wait complete. 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.266+0000 7f9a78a9d640 1 Processor -- start 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.266+0000 7f9a78a9d640 1 -- start start 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.266+0000 7f9a78a9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a740c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.266+0000 7f9a78a9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a741f6f50 0x7f9a74148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.266+0000 7f9a78a9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a7414bc80 0x7f9a740aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.266+0000 7f9a78a9d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740cf120 con 0x7f9a60004d40 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.266+0000 7f9a78a9d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740cefa0 con 0x7f9a741f6f50 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.266+0000 7f9a78a9d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740cf2a0 con 0x7f9a7414bc80 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.266+0000 7f9a7259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a740c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a7259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a740c5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52668/0 (socket says 172.21.15.27:52668) 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a7259c640 1 -- 172.21.15.27:0/1583114317 learned_addr learned my addr 172.21.15.27:0/1583114317 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a7259c640 1 -- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a7414bc80 msgr2=0x7f9a740aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a72d9d640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a7414bc80 0x7f9a740aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a69d9b640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a741f6f50 0x7f9a74148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a7259c640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a7414bc80 0x7f9a740aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a7259c640 1 -- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a741f6f50 msgr2=0x7f9a74148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a7259c640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a741f6f50 0x7f9a74148b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a7259c640 1 -- 172.21.15.27:0/1583114317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7414b4e0 con 0x7f9a60004d40 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a72d9d640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a7414bc80 0x7f9a740aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a69d9b640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a741f6f50 0x7f9a74148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:48.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a7259c640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a740c5bd0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400f7a0 tx=0x7f9a6400fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a6b7fe640 1 -- 172.21.15.27:0/1583114317 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6400fe50 con 0x7f9a60004d40 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a6b7fe640 1 -- 172.21.15.27:0/1583114317 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a64018070 con 0x7f9a60004d40 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a6b7fe640 1 -- 172.21.15.27:0/1583114317 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6400d730 con 0x7f9a60004d40 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a741f76c0 con 0x7f9a60004d40 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.267+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a741f7c80 con 0x7f9a60004d40 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.268+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a74068700 con 0x7f9a60004d40 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.269+0000 7f9a6b7fe640 1 -- 172.21.15.27:0/1583114317 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a6400d8d0 con 0x7f9a60004d40 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.269+0000 7f9a6b7fe640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a5406b500 0x7f9a5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.269+0000 7f9a6b7fe640 1 -- 172.21.15.27:0/1583114317 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9a6408a0f0 con 0x7f9a60004d40 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.270+0000 7f9a69d9b640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a5406b500 0x7f9a5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.270+0000 7f9a69d9b640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a5406b500 0x7f9a5406d9c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c0017e0 tx=0x7f9a5c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.274+0000 7f9a6b7fe640 1 -- 172.21.15.27:0/1583114317 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a7414b4e0 con 0x7f9a60004d40 2024-03-06T07:33:48.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.493+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a74067320 con 0x7f9a60004d40 2024-03-06T07:33:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.494+0000 7f9a6b7fe640 1 -- 172.21.15.27:0/1583114317 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f9a6401a310 con 0x7f9a60004d40 2024-03-06T07:33:48.495 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:48.495 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a5406b500 msgr2=0x7f9a5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a5406b500 0x7f9a5406d9c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c0017e0 tx=0x7f9a5c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 msgr2=0x7f9a740c5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a740c5bd0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400f7a0 tx=0x7f9a6400fc60 comp rx=0 tx=0).stop 2024-03-06T07:33:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 shutdown_connections 2024-03-06T07:33:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9a5406b500 0x7f9a5406d9c0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9a7414bc80 0x7f9a740aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a741f6f50 0x7f9a74148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 --2- 172.21.15.27:0/1583114317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a60004d40 0x7f9a740c5bd0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.495+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 >> 172.21.15.27:0/1583114317 conn(0x7f9a740d0990 msgr2=0x7f9a7406db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.496+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 shutdown_connections 2024-03-06T07:33:48.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:48.496+0000 7f9a78a9d640 1 -- 172.21.15.27:0/1583114317 wait complete. 2024-03-06T07:33:48.710 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:48.711 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:49.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:49.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.818+0000 7f521a95c640 1 Processor -- start 2024-03-06T07:33:49.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.818+0000 7f521a95c640 1 -- start start 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f521a95c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f521415df00 0x7f52140d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f521a95c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52140b95a0 0x7f52140b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f521a95c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52140cea30 con 0x7f52140b9a30 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f521a95c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52140d2af0 con 0x7f52140b95a0 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f521a95c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52140cf3c0 con 0x7f521415df00 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f52137fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52140b95a0 0x7f52140b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f52137fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52140b95a0 0x7f52140b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52678/0 (socket says 172.21.15.27:52678) 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f52137fe640 1 -- 172.21.15.27:0/1117179679 learned_addr learned my addr 172.21.15.27:0/1117179679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f5213fff640 1 --2- 172.21.15.27:0/1117179679 >> v2:172.21.15.69:3300/0 conn(0x7f521415df00 0x7f52140d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f52137fe640 1 -- 172.21.15.27:0/1117179679 >> v2:172.21.15.69:3300/0 conn(0x7f521415df00 msgr2=0x7f52140d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f52137fe640 1 --2- 172.21.15.27:0/1117179679 >> v2:172.21.15.69:3300/0 conn(0x7f521415df00 0x7f52140d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f52137fe640 1 -- 172.21.15.27:0/1117179679 >> v1:172.21.15.27:6789/0 conn(0x7f52140b9a30 legacy=0x7f52140d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f5212ffd640 1 -- 172.21.15.27:0/1117179679 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397818515 0 0) 0x7f52140cea30 con 0x7f52140b9a30 2024-03-06T07:33:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.819+0000 7f52137fe640 1 -- 172.21.15.27:0/1117179679 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52140cf0e0 con 0x7f52140b95a0 2024-03-06T07:33:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.820+0000 7f52137fe640 1 --2- 172.21.15.27:0/1117179679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52140b95a0 0x7f52140b5700 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f52140d11b0 tx=0x7f51fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88267ece0e0668bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.820+0000 7f5212ffd640 1 -- 172.21.15.27:0/1117179679 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc034070 con 0x7f52140b95a0 2024-03-06T07:33:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.820+0000 7f5212ffd640 1 -- 172.21.15.27:0/1117179679 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc030860 con 0x7f52140b95a0 2024-03-06T07:33:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.820+0000 7f5212ffd640 1 -- 172.21.15.27:0/1117179679 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc030c30 con 0x7f52140b95a0 2024-03-06T07:33:49.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.821+0000 7f521a95c640 1 -- 172.21.15.27:0/1117179679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52140b95a0 msgr2=0x7f52140b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.821+0000 7f521a95c640 1 --2- 172.21.15.27:0/1117179679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52140b95a0 0x7f52140b5700 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f52140d11b0 tx=0x7f51fc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.821+0000 7f521a95c640 1 -- 172.21.15.27:0/1117179679 shutdown_connections 2024-03-06T07:33:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.821+0000 7f521a95c640 1 --2- 172.21.15.27:0/1117179679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52140b95a0 0x7f52140b5700 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.821+0000 7f521a95c640 1 --2- 172.21.15.27:0/1117179679 >> v2:172.21.15.69:3300/0 conn(0x7f521415df00 0x7f52140d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.821+0000 7f521a95c640 1 -- 172.21.15.27:0/1117179679 >> 172.21.15.27:0/1117179679 conn(0x7f5214071960 msgr2=0x7f5214071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.821+0000 7f521a95c640 1 -- 172.21.15.27:0/1117179679 shutdown_connections 2024-03-06T07:33:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.821+0000 7f521a95c640 1 -- 172.21.15.27:0/1117179679 wait complete. 2024-03-06T07:33:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.822+0000 7f521a95c640 1 Processor -- start 2024-03-06T07:33:49.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.822+0000 7f521a95c640 1 -- start start 2024-03-06T07:33:49.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f521a95c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52140b95a0 0x7f521415a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f521a95c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f521415df00 0x7f52141f78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f521a95c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f521415ae40 0x7f5214161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f521a95c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52140cec70 con 0x7f521415ae40 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f521a95c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52140ce750 con 0x7f521415df00 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f521a95c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52140cedf0 con 0x7f52140b95a0 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5218ef9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f521415ae40 0x7f5214161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5218ef9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f521415ae40 0x7f5214161060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52694/0 (socket says 172.21.15.27:52694) 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5218ef9640 1 -- 172.21.15.27:0/3746254255 learned_addr learned my addr 172.21.15.27:0/3746254255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f52137fe640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f521415df00 0x7f52141f78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5213fff640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52140b95a0 0x7f521415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5218ef9640 1 -- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52140b95a0 msgr2=0x7f521415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5218ef9640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52140b95a0 0x7f521415a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5218ef9640 1 -- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f521415df00 msgr2=0x7f52141f78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5218ef9640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f521415df00 0x7f52141f78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5218ef9640 1 -- 172.21.15.27:0/3746254255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52141f6e30 con 0x7f521415ae40 2024-03-06T07:33:49.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f52137fe640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f521415df00 0x7f52141f78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.823+0000 7f5218ef9640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f521415ae40 0x7f5214161060 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f520c00c760 tx=0x7f520c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.824+0000 7f5210ff9640 1 -- 172.21.15.27:0/3746254255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c003da0 con 0x7f521415ae40 2024-03-06T07:33:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.824+0000 7f5210ff9640 1 -- 172.21.15.27:0/3746254255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c005070 con 0x7f521415ae40 2024-03-06T07:33:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.824+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52141e8db0 con 0x7f521415ae40 2024-03-06T07:33:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.824+0000 7f5210ff9640 1 -- 172.21.15.27:0/3746254255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c005460 con 0x7f521415ae40 2024-03-06T07:33:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.824+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52141f73b0 con 0x7f521415ae40 2024-03-06T07:33:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.825+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5214067ff0 con 0x7f521415ae40 2024-03-06T07:33:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.825+0000 7f5210ff9640 1 -- 172.21.15.27:0/3746254255 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f520c005600 con 0x7f521415ae40 2024-03-06T07:33:49.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.826+0000 7f5210ff9640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f51f406b4d0 0x7f51f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.826+0000 7f5210ff9640 1 -- 172.21.15.27:0/3746254255 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f520c08a180 con 0x7f521415ae40 2024-03-06T07:33:49.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.826+0000 7f5213fff640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f51f406b4d0 0x7f51f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.826+0000 7f5213fff640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f51f406b4d0 0x7f51f406d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5204006fd0 tx=0x7f5204008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:49.830+0000 7f5210ff9640 1 -- 172.21.15.27:0/3746254255 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f520c058fe0 con 0x7f521415ae40 2024-03-06T07:33:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.053+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52140ce0a0 con 0x7f521415ae40 2024-03-06T07:33:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.054+0000 7f5210ff9640 1 -- 172.21.15.27:0/3746254255 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f520c0588e0 con 0x7f521415ae40 2024-03-06T07:33:50.055 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:50.055 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:50.056 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:50.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:50.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f51f406b4d0 msgr2=0x7f51f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f51f406b4d0 0x7f51f406d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5204006fd0 tx=0x7f5204008040 comp rx=0 tx=0).stop 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f521415ae40 msgr2=0x7f5214161060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f521415ae40 0x7f5214161060 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f520c00c760 tx=0x7f520c00cc20 comp rx=0 tx=0).stop 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 shutdown_connections 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f51f406b4d0 0x7f51f406d990 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f521415ae40 0x7f5214161060 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f521415df00 0x7f52141f78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 --2- 172.21.15.27:0/3746254255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52140b95a0 0x7f521415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 >> 172.21.15.27:0/3746254255 conn(0x7f5214071960 msgr2=0x7f52140b8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 shutdown_connections 2024-03-06T07:33:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:50.056+0000 7f521a95c640 1 -- 172.21.15.27:0/3746254255 wait complete. 2024-03-06T07:33:50.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:50.271 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:50.271 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:51.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:51.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.377+0000 7fea9db5c640 1 Processor -- start 2024-03-06T07:33:51.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.377+0000 7fea9db5c640 1 -- start start 2024-03-06T07:33:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.378+0000 7fea9db5c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fea9815df00 0x7fea980d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.378+0000 7fea9db5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 0x7fea980b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.378+0000 7fea9db5c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea980cea30 con 0x7fea980b9a30 2024-03-06T07:33:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.378+0000 7fea9db5c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea980d2af0 con 0x7fea9815df00 2024-03-06T07:33:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.378+0000 7fea9db5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea980cf3c0 con 0x7fea980b95a0 2024-03-06T07:33:51.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.378+0000 7fea97fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fea980b9a30 0x7fea980d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60898/0 (socket says 172.21.15.27:60898) 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.378+0000 7fea97fff640 1 -- 172.21.15.27:0/648987383 learned_addr learned my addr 172.21.15.27:0/648987383 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.378+0000 7fea977fe640 1 --2- 172.21.15.27:0/648987383 >> v2:172.21.15.69:3300/0 conn(0x7fea9815df00 0x7fea980d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.378+0000 7fea96ffd640 1 --2- 172.21.15.27:0/648987383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 0x7fea980b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.379+0000 7fea967fc640 1 -- 172.21.15.27:0/648987383 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611938363 0 0) 0x7fea980cea30 con 0x7fea980b9a30 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.379+0000 7fea967fc640 1 -- 172.21.15.27:0/648987383 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea80003520 con 0x7fea980b9a30 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.379+0000 7fea977fe640 1 -- 172.21.15.27:0/648987383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 msgr2=0x7fea980b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.379+0000 7fea977fe640 1 --2- 172.21.15.27:0/648987383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 0x7fea980b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.379+0000 7fea977fe640 1 -- 172.21.15.27:0/648987383 >> v1:172.21.15.27:6789/0 conn(0x7fea980b9a30 legacy=0x7fea980d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.379+0000 7fea977fe640 1 -- 172.21.15.27:0/648987383 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea980cf0e0 con 0x7fea9815df00 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.379+0000 7fea96ffd640 1 --2- 172.21.15.27:0/648987383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 0x7fea980b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.380+0000 7fea977fe640 1 --2- 172.21.15.27:0/648987383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea9815df00 0x7fea980d25b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fea880097a0 tx=0x7fea8802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4c42f8c250e2fac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.380+0000 7fea967fc640 1 -- 172.21.15.27:0/648987383 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88035020 con 0x7fea9815df00 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.380+0000 7fea967fc640 1 -- 172.21.15.27:0/648987383 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea88031830 con 0x7fea9815df00 2024-03-06T07:33:51.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.380+0000 7fea967fc640 1 -- 172.21.15.27:0/648987383 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88031be0 con 0x7fea9815df00 2024-03-06T07:33:51.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.381+0000 7fea9db5c640 1 -- 172.21.15.27:0/648987383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea9815df00 msgr2=0x7fea980d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.381+0000 7fea9db5c640 1 --2- 172.21.15.27:0/648987383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea9815df00 0x7fea980d25b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fea880097a0 tx=0x7fea8802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.381+0000 7fea9db5c640 1 -- 172.21.15.27:0/648987383 shutdown_connections 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.381+0000 7fea9db5c640 1 --2- 172.21.15.27:0/648987383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 0x7fea980b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.381+0000 7fea9db5c640 1 --2- 172.21.15.27:0/648987383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea9815df00 0x7fea980d25b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.381+0000 7fea9db5c640 1 -- 172.21.15.27:0/648987383 >> 172.21.15.27:0/648987383 conn(0x7fea98071960 msgr2=0x7fea98071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.381+0000 7fea9db5c640 1 -- 172.21.15.27:0/648987383 shutdown_connections 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.381+0000 7fea9db5c640 1 -- 172.21.15.27:0/648987383 wait complete. 2024-03-06T07:33:51.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.383+0000 7fea9db5c640 1 Processor -- start 2024-03-06T07:33:51.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.383+0000 7fea9db5c640 1 -- start start 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.384+0000 7fea9db5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 0x7fea9815e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.384+0000 7fea9db5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea9815df00 0x7fea981f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.384+0000 7fea9db5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981f4e40 0x7fea981f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.384+0000 7fea9db5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea980cec70 con 0x7fea981f4e40 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.384+0000 7fea9db5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea980ce750 con 0x7fea9815df00 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.384+0000 7fea9db5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea980cedf0 con 0x7fea980b95a0 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.384+0000 7fea97fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981f4e40 0x7fea981f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea97fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981f4e40 0x7fea981f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52704/0 (socket says 172.21.15.27:52704) 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea97fff640 1 -- 172.21.15.27:0/784712999 learned_addr learned my addr 172.21.15.27:0/784712999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea96ffd640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea9815df00 0x7fea981f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea977fe640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 0x7fea9815e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea97fff640 1 -- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 msgr2=0x7fea9815e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea97fff640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 0x7fea9815e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea97fff640 1 -- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea9815df00 msgr2=0x7fea981f6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea97fff640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea9815df00 0x7fea981f6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea97fff640 1 -- 172.21.15.27:0/784712999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea981f48d0 con 0x7fea981f4e40 2024-03-06T07:33:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea97fff640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981f4e40 0x7fea981f6bd0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fea9000c400 tx=0x7fea9000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea7ffff640 1 -- 172.21.15.27:0/784712999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea90017070 con 0x7fea981f4e40 2024-03-06T07:33:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea7ffff640 1 -- 172.21.15.27:0/784712999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea90005ce0 con 0x7fea981f4e40 2024-03-06T07:33:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea981f4bb0 con 0x7fea981f4e40 2024-03-06T07:33:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea7ffff640 1 -- 172.21.15.27:0/784712999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea900072d0 con 0x7fea981f4e40 2024-03-06T07:33:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.385+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea980c8b80 con 0x7fea981f4e40 2024-03-06T07:33:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.386+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea98067ff0 con 0x7fea981f4e40 2024-03-06T07:33:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.387+0000 7fea7ffff640 1 -- 172.21.15.27:0/784712999 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea9001a070 con 0x7fea981f4e40 2024-03-06T07:33:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.387+0000 7fea7ffff640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea8406b4d0 0x7fea8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.388+0000 7fea7ffff640 1 -- 172.21.15.27:0/784712999 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fea900895a0 con 0x7fea981f4e40 2024-03-06T07:33:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.388+0000 7fea977fe640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea8406b4d0 0x7fea8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.388+0000 7fea977fe640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea8406b4d0 0x7fea8406d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fea88036000 tx=0x7fea8803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.392+0000 7fea7ffff640 1 -- 172.21.15.27:0/784712999 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea90057dc0 con 0x7fea981f4e40 2024-03-06T07:33:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.612+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea9815c8b0 con 0x7fea981f4e40 2024-03-06T07:33:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.612+0000 7fea7ffff640 1 -- 172.21.15.27:0/784712999 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fea9005be60 con 0x7fea981f4e40 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:51.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea8406b4d0 msgr2=0x7fea8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea8406b4d0 0x7fea8406d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fea88036000 tx=0x7fea8803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981f4e40 msgr2=0x7fea981f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981f4e40 0x7fea981f6bd0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fea9000c400 tx=0x7fea9000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 shutdown_connections 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fea8406b4d0 0x7fea8406d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981f4e40 0x7fea981f6bd0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea9815df00 0x7fea981f6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 --2- 172.21.15.27:0/784712999 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fea980b95a0 0x7fea9815e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 >> 172.21.15.27:0/784712999 conn(0x7fea98071960 msgr2=0x7fea980cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 shutdown_connections 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:51.614+0000 7fea9db5c640 1 -- 172.21.15.27:0/784712999 wait complete. 2024-03-06T07:33:51.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:51.827 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:52.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.953+0000 7faae4763640 1 Processor -- start 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.953+0000 7faae4763640 1 -- start start 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae4763640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15df00 0x7faadc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae4763640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faadc0b95a0 0x7faadc0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae4763640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc0cea30 con 0x7faadc0b9a30 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae4763640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faadc0d2af0 con 0x7faadc0b95a0 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae4763640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0cf3c0 con 0x7faadc15df00 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae1cfe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faadc0b95a0 0x7faadc0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae1cfe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faadc0b95a0 0x7faadc0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52706/0 (socket says 172.21.15.27:52706) 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae1cfe640 1 -- 172.21.15.27:0/3428476750 learned_addr learned my addr 172.21.15.27:0/3428476750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae24ff640 1 --2- 172.21.15.27:0/3428476750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15df00 0x7faadc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae1cfe640 1 -- 172.21.15.27:0/3428476750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15df00 msgr2=0x7faadc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae1cfe640 1 --2- 172.21.15.27:0/3428476750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15df00 0x7faadc0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae1cfe640 1 -- 172.21.15.27:0/3428476750 >> v1:172.21.15.69:6789/0 conn(0x7faadc0b9a30 legacy=0x7faadc0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.954+0000 7faae1cfe640 1 -- 172.21.15.27:0/3428476750 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc0cf0e0 con 0x7faadc0b95a0 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.955+0000 7faae1cfe640 1 --2- 172.21.15.27:0/3428476750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc0b95a0 0x7faadc0b5700 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7faacc009870 tx=0x7faacc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97dae7aaea0d023 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.955+0000 7faae14fd640 1 -- 172.21.15.27:0/3428476750 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc034070 con 0x7faadc0b95a0 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.955+0000 7faae14fd640 1 -- 172.21.15.27:0/3428476750 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faacc030840 con 0x7faadc0b95a0 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.955+0000 7faae14fd640 1 -- 172.21.15.27:0/3428476750 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc030c10 con 0x7faadc0b95a0 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.956+0000 7faae4763640 1 -- 172.21.15.27:0/3428476750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc0b95a0 msgr2=0x7faadc0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.956+0000 7faae4763640 1 --2- 172.21.15.27:0/3428476750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc0b95a0 0x7faadc0b5700 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7faacc009870 tx=0x7faacc02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.956+0000 7faae4763640 1 -- 172.21.15.27:0/3428476750 shutdown_connections 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.956+0000 7faae4763640 1 --2- 172.21.15.27:0/3428476750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc0b95a0 0x7faadc0b5700 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.956+0000 7faae4763640 1 --2- 172.21.15.27:0/3428476750 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15df00 0x7faadc0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.956+0000 7faae4763640 1 -- 172.21.15.27:0/3428476750 >> 172.21.15.27:0/3428476750 conn(0x7faadc071960 msgr2=0x7faadc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.956+0000 7faae4763640 1 -- 172.21.15.27:0/3428476750 shutdown_connections 2024-03-06T07:33:52.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.956+0000 7faae4763640 1 -- 172.21.15.27:0/3428476750 wait complete. 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.957+0000 7faae4763640 1 Processor -- start 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.957+0000 7faae4763640 1 -- start start 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae4763640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faadc0b95a0 0x7faadc160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae4763640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc15df00 0x7faadc1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae4763640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15f190 0x7faadc152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae4763640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0cec70 con 0x7faadc15df00 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae4763640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0ce750 con 0x7faadc0b95a0 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae4763640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0cedf0 con 0x7faadc15f190 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae1cfe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc15df00 0x7faadc1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae1cfe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc15df00 0x7faadc1554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52716/0 (socket says 172.21.15.27:52716) 2024-03-06T07:33:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae1cfe640 1 -- 172.21.15.27:0/1692858040 learned_addr learned my addr 172.21.15.27:0/1692858040 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae24ff640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faadc0b95a0 0x7faadc160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae2d00640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15f190 0x7faadc152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae1cfe640 1 -- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15f190 msgr2=0x7faadc152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae1cfe640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15f190 0x7faadc152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae1cfe640 1 -- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faadc0b95a0 msgr2=0x7faadc160010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae1cfe640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faadc0b95a0 0x7faadc160010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae1cfe640 1 -- 172.21.15.27:0/1692858040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc155c20 con 0x7faadc15df00 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae24ff640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faadc0b95a0 0x7faadc160010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.958+0000 7faae2d00640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15f190 0x7faadc152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.959+0000 7faae1cfe640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc15df00 0x7faadc1554b0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7faacc000c00 tx=0x7faacc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.959+0000 7faad2ffd640 1 -- 172.21.15.27:0/1692858040 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc048070 con 0x7faadc15df00 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.959+0000 7faad2ffd640 1 -- 172.21.15.27:0/1692858040 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faacc030640 con 0x7faadc15df00 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.959+0000 7faad2ffd640 1 -- 172.21.15.27:0/1692858040 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc034040 con 0x7faadc15df00 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.959+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc15e410 con 0x7faadc15df00 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.959+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc15e990 con 0x7faadc15df00 2024-03-06T07:33:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.960+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc067ff0 con 0x7faadc15df00 2024-03-06T07:33:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.961+0000 7faad2ffd640 1 -- 172.21.15.27:0/1692858040 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faacc037070 con 0x7faadc15df00 2024-03-06T07:33:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.961+0000 7faad2ffd640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faac006b4d0 0x7faac006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.961+0000 7faad2ffd640 1 -- 172.21.15.27:0/1692858040 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faacc0ac300 con 0x7faadc15df00 2024-03-06T07:33:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.962+0000 7faae24ff640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faac006b4d0 0x7faac006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.962+0000 7faae24ff640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faac006b4d0 0x7faac006d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7faad8006fd0 tx=0x7faad8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:52.965+0000 7faad2ffd640 1 -- 172.21.15.27:0/1692858040 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faacc051020 con 0x7faadc15df00 2024-03-06T07:33:53.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.188+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faadc0b72a0 con 0x7faadc15df00 2024-03-06T07:33:53.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.188+0000 7faad2ffd640 1 -- 172.21.15.27:0/1692858040 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7faacc07b3f0 con 0x7faadc15df00 2024-03-06T07:33:53.190 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:53.190 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faac006b4d0 msgr2=0x7faac006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faac006b4d0 0x7faac006d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7faad8006fd0 tx=0x7faad8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc15df00 msgr2=0x7faadc1554b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc15df00 0x7faadc1554b0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7faacc000c00 tx=0x7faacc004730 comp rx=0 tx=0).stop 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 shutdown_connections 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faac006b4d0 0x7faac006d990 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faadc15f190 0x7faadc152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faadc15df00 0x7faadc1554b0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 --2- 172.21.15.27:0/1692858040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faadc0b95a0 0x7faadc160010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 >> 172.21.15.27:0/1692858040 conn(0x7faadc071960 msgr2=0x7faadc1618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 shutdown_connections 2024-03-06T07:33:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:53.190+0000 7faae4763640 1 -- 172.21.15.27:0/1692858040 wait complete. 2024-03-06T07:33:53.406 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:53.406 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:54.407 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.512+0000 7fa703d4b640 1 Processor -- start 2024-03-06T07:33:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.512+0000 7fa703d4b640 1 -- start start 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa703d4b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc0b95a0 0x7fa6fc0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa703d4b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6fc0b9a10 0x7fa6fc0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa703d4b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6fc0cea30 con 0x7fa6fc15df00 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa703d4b640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6fc0cf3c0 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa703d4b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc0cf0e0 con 0x7fa6fc0b95a0 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa7022e8640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6fc0b9a10 0x7fa6fc0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa701ae7640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6fc15df00 0x7fa6fc0d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56712/0 (socket says 172.21.15.27:56712) 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa7022e8640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa6fc0b9a10 0x7fa6fc0b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38018/0 (socket says 172.21.15.27:38018) 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa701ae7640 1 -- 172.21.15.27:0/128941712 learned_addr learned my addr 172.21.15.27:0/128941712 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.513+0000 7fa7012e6640 1 --2- 172.21.15.27:0/128941712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc0b95a0 0x7fa6fc0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.514+0000 7fa7022e8640 1 -- 172.21.15.27:0/128941712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc0b95a0 msgr2=0x7fa6fc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.514+0000 7fa7022e8640 1 --2- 172.21.15.27:0/128941712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc0b95a0 0x7fa6fc0b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.514+0000 7fa7022e8640 1 -- 172.21.15.27:0/128941712 >> v1:172.21.15.27:6789/0 conn(0x7fa6fc15df00 legacy=0x7fa6fc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.514+0000 7fa7022e8640 1 -- 172.21.15.27:0/128941712 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc0ce470 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.514+0000 7fa700ae5640 1 -- 172.21.15.27:0/128941712 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80339778 0 0) 0x7fa6fc0cea30 con 0x7fa6fc15df00 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.514+0000 7fa7022e8640 1 --2- 172.21.15.27:0/128941712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 0x7fa6fc0b5c40 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009d90 tx=0x7fa6f402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675cb11c4f6e385c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa700ae5640 1 -- 172.21.15.27:0/128941712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4039070 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa700ae5640 1 -- 172.21.15.27:0/128941712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f402cc50 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa700ae5640 1 -- 172.21.15.27:0/128941712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4034440 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa703d4b640 1 -- 172.21.15.27:0/128941712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 msgr2=0x7fa6fc0b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa703d4b640 1 --2- 172.21.15.27:0/128941712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 0x7fa6fc0b5c40 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009d90 tx=0x7fa6f402bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa703d4b640 1 -- 172.21.15.27:0/128941712 shutdown_connections 2024-03-06T07:33:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa703d4b640 1 --2- 172.21.15.27:0/128941712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 0x7fa6fc0b5c40 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa703d4b640 1 --2- 172.21.15.27:0/128941712 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc0b95a0 0x7fa6fc0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa703d4b640 1 -- 172.21.15.27:0/128941712 >> 172.21.15.27:0/128941712 conn(0x7fa6fc071960 msgr2=0x7fa6fc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa703d4b640 1 -- 172.21.15.27:0/128941712 shutdown_connections 2024-03-06T07:33:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.515+0000 7fa703d4b640 1 -- 172.21.15.27:0/128941712 wait complete. 2024-03-06T07:33:54.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.516+0000 7fa703d4b640 1 Processor -- start 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.516+0000 7fa703d4b640 1 -- start start 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa703d4b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc0b95a0 0x7fa6fc0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa703d4b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 0x7fa6fc1608e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa703d4b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc15df00 0x7fa6fc160e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa703d4b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc0cedf0 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa703d4b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc0cec70 con 0x7fa6fc0b95a0 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa703d4b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc0d3220 con 0x7fa6fc15df00 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa7012e6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 0x7fa6fc1608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa7012e6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 0x7fa6fc1608e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38022/0 (socket says 172.21.15.27:38022) 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa7012e6640 1 -- 172.21.15.27:0/561920609 learned_addr learned my addr 172.21.15.27:0/561920609 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa701ae7640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc0b95a0 0x7fa6fc0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.517+0000 7fa7022e8640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc15df00 0x7fa6fc160e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa7012e6640 1 -- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc15df00 msgr2=0x7fa6fc160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa7012e6640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc15df00 0x7fa6fc160e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa7012e6640 1 -- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc0b95a0 msgr2=0x7fa6fc0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa7012e6640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc0b95a0 0x7fa6fc0c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa7012e6640 1 -- 172.21.15.27:0/561920609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc0ca6b0 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa7012e6640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 0x7fa6fc1608e0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00d800 tx=0x7fa6ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa6f27fc640 1 -- 172.21.15.27:0/561920609 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec00b6c0 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa6f27fc640 1 -- 172.21.15.27:0/561920609 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec019070 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6fc0ccae0 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa6f27fc640 1 -- 172.21.15.27:0/561920609 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec00bac0 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.518+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6fc0cd0a0 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.519+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6fc067ff0 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.520+0000 7fa6f27fc640 1 -- 172.21.15.27:0/561920609 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6ec00bc60 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.520+0000 7fa6f27fc640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6e006b500 0x7fa6e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.520+0000 7fa6f27fc640 1 -- 172.21.15.27:0/561920609 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa6ec08a5f0 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.520+0000 7fa701ae7640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6e006b500 0x7fa6e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.521+0000 7fa701ae7640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6e006b500 0x7fa6e006d9c0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8007fd0 tx=0x7fa6f8007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.525+0000 7fa6f27fc640 1 -- 172.21.15.27:0/561920609 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6ec058d70 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.743+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6fc0d2a60 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.743+0000 7fa6f27fc640 1 -- 172.21.15.27:0/561920609 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fa6ec05ce10 con 0x7fa6fc0b9a10 2024-03-06T07:33:54.745 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:54.745 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6e006b500 msgr2=0x7fa6e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6e006b500 0x7fa6e006d9c0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8007fd0 tx=0x7fa6f8007de0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 msgr2=0x7fa6fc1608e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 0x7fa6fc1608e0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00d800 tx=0x7fa6ec00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 shutdown_connections 2024-03-06T07:33:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa6e006b500 0x7fa6e006d9c0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa6fc15df00 0x7fa6fc160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6fc0b9a10 0x7fa6fc1608e0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 --2- 172.21.15.27:0/561920609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc0b95a0 0x7fa6fc0c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.745+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 >> 172.21.15.27:0/561920609 conn(0x7fa6fc071960 msgr2=0x7fa6fc071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.746+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 shutdown_connections 2024-03-06T07:33:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:54.746+0000 7fa703d4b640 1 -- 172.21.15.27:0/561920609 wait complete. 2024-03-06T07:33:54.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:54.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:55.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:56.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.065+0000 7f5aecc0e640 1 Processor -- start 2024-03-06T07:33:56.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.065+0000 7f5aecc0e640 1 -- start start 2024-03-06T07:33:56.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.066+0000 7f5aecc0e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ae80b95a0 0x7f5ae80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.066+0000 7f5aecc0e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ae80b9a10 0x7f5ae80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.066+0000 7f5aecc0e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae80cea30 con 0x7f5ae815df00 2024-03-06T07:33:56.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.066+0000 7f5aecc0e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae80cf3c0 con 0x7f5ae80b9a10 2024-03-06T07:33:56.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.066+0000 7f5aecc0e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae80cf0e0 con 0x7f5ae80b95a0 2024-03-06T07:33:56.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.066+0000 7f5ae6d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ae80b9a10 0x7f5ae80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.066+0000 7f5ae659c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5ae815df00 0x7f5ae80d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56728/0 (socket says 172.21.15.27:56728) 2024-03-06T07:33:56.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.066+0000 7f5ae659c640 1 -- 172.21.15.27:0/1610745303 learned_addr learned my addr 172.21.15.27:0/1610745303 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:56.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.067+0000 7f5ae5d9b640 1 --2- 172.21.15.27:0/1610745303 >> v2:172.21.15.69:3300/0 conn(0x7f5ae80b95a0 0x7f5ae80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.067+0000 7f5ae6d9d640 1 -- 172.21.15.27:0/1610745303 >> v2:172.21.15.69:3300/0 conn(0x7f5ae80b95a0 msgr2=0x7f5ae80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.067+0000 7f5ae6d9d640 1 --2- 172.21.15.27:0/1610745303 >> v2:172.21.15.69:3300/0 conn(0x7f5ae80b95a0 0x7f5ae80b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.067+0000 7f5ae6d9d640 1 -- 172.21.15.27:0/1610745303 >> v1:172.21.15.27:6789/0 conn(0x7f5ae815df00 legacy=0x7f5ae80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.067+0000 7f5ae6d9d640 1 -- 172.21.15.27:0/1610745303 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae80ce470 con 0x7f5ae80b9a10 2024-03-06T07:33:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.067+0000 7f5ae6d9d640 1 --2- 172.21.15.27:0/1610745303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b9a10 0x7f5ae80b5c40 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80d1760 tx=0x7f5ae002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c082b5a088bcf7da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5ae559a640 1 -- 172.21.15.27:0/1610745303 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0039070 con 0x7f5ae80b9a10 2024-03-06T07:33:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5ae559a640 1 -- 172.21.15.27:0/1610745303 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae002cc80 con 0x7f5ae80b9a10 2024-03-06T07:33:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5ae559a640 1 -- 172.21.15.27:0/1610745303 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0034660 con 0x7f5ae80b9a10 2024-03-06T07:33:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5aecc0e640 1 -- 172.21.15.27:0/1610745303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b9a10 msgr2=0x7f5ae80b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5aecc0e640 1 --2- 172.21.15.27:0/1610745303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b9a10 0x7f5ae80b5c40 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80d1760 tx=0x7f5ae002bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5aecc0e640 1 -- 172.21.15.27:0/1610745303 shutdown_connections 2024-03-06T07:33:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5aecc0e640 1 --2- 172.21.15.27:0/1610745303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b9a10 0x7f5ae80b5c40 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5aecc0e640 1 --2- 172.21.15.27:0/1610745303 >> v2:172.21.15.69:3300/0 conn(0x7f5ae80b95a0 0x7f5ae80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5aecc0e640 1 -- 172.21.15.27:0/1610745303 >> 172.21.15.27:0/1610745303 conn(0x7f5ae8071960 msgr2=0x7f5ae8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5aecc0e640 1 -- 172.21.15.27:0/1610745303 shutdown_connections 2024-03-06T07:33:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.068+0000 7f5aecc0e640 1 -- 172.21.15.27:0/1610745303 wait complete. 2024-03-06T07:33:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.070+0000 7f5aecc0e640 1 Processor -- start 2024-03-06T07:33:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.070+0000 7f5aecc0e640 1 -- start start 2024-03-06T07:33:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.070+0000 7f5aecc0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b95a0 0x7f5ae81f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.070+0000 7f5aecc0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ae80b9a10 0x7f5ae80cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.070+0000 7f5aecc0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae815df00 0x7f5ae80b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.070+0000 7f5aecc0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae80cedf0 con 0x7f5ae80b95a0 2024-03-06T07:33:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.070+0000 7f5aecc0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae80cec70 con 0x7f5ae815df00 2024-03-06T07:33:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.070+0000 7f5aecc0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae80d3220 con 0x7f5ae80b9a10 2024-03-06T07:33:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.070+0000 7f5ae659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b95a0 0x7f5ae81f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b95a0 0x7f5ae81f31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38032/0 (socket says 172.21.15.27:38032) 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae659c640 1 -- 172.21.15.27:0/2295610439 learned_addr learned my addr 172.21.15.27:0/2295610439 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae6d9d640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae815df00 0x7f5ae80b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae5d9b640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ae80b9a10 0x7f5ae80cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae659c640 1 -- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ae80b9a10 msgr2=0x7f5ae80cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae659c640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ae80b9a10 0x7f5ae80cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae659c640 1 -- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae815df00 msgr2=0x7f5ae80b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae659c640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae815df00 0x7f5ae80b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae659c640 1 -- 172.21.15.27:0/2295610439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae80cc000 con 0x7f5ae80b95a0 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae5d9b640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ae80b9a10 0x7f5ae80cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ae659c640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b95a0 0x7f5ae81f31f0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f5ad800c3c0 tx=0x7f5ad800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ad6ffd640 1 -- 172.21.15.27:0/2295610439 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8017070 con 0x7f5ae80b95a0 2024-03-06T07:33:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ad6ffd640 1 -- 172.21.15.27:0/2295610439 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad8005ce0 con 0x7f5ae80b95a0 2024-03-06T07:33:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5ad6ffd640 1 -- 172.21.15.27:0/2295610439 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad80072f0 con 0x7f5ae80b95a0 2024-03-06T07:33:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae80cc2e0 con 0x7f5ae80b95a0 2024-03-06T07:33:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.071+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae80cc8a0 con 0x7f5ae80b95a0 2024-03-06T07:33:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.073+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae8067ff0 con 0x7f5ae80b95a0 2024-03-06T07:33:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.073+0000 7f5ad6ffd640 1 -- 172.21.15.27:0/2295610439 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ad801a070 con 0x7f5ae80b95a0 2024-03-06T07:33:56.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.073+0000 7f5ad6ffd640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ac806b4d0 0x7f5ac806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.073+0000 7f5ad6ffd640 1 -- 172.21.15.27:0/2295610439 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5ad8089590 con 0x7f5ae80b95a0 2024-03-06T07:33:56.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.074+0000 7f5ae5d9b640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ac806b4d0 0x7f5ac806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.074+0000 7f5ae5d9b640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ac806b4d0 0x7f5ac806d990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5ae81f48c0 tx=0x7f5adc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.078+0000 7f5ad6ffd640 1 -- 172.21.15.27:0/2295610439 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ad8058460 con 0x7f5ae80b95a0 2024-03-06T07:33:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.298+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ae806b780 con 0x7f5ae80b95a0 2024-03-06T07:33:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.299+0000 7f5ad6ffd640 1 -- 172.21.15.27:0/2295610439 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f5ad8057d60 con 0x7f5ae80b95a0 2024-03-06T07:33:56.301 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:56.301 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ac806b4d0 msgr2=0x7f5ac806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ac806b4d0 0x7f5ac806d990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5ae81f48c0 tx=0x7f5adc008040 comp rx=0 tx=0).stop 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b95a0 msgr2=0x7f5ae81f31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b95a0 0x7f5ae81f31f0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f5ad800c3c0 tx=0x7f5ad800c880 comp rx=0 tx=0).stop 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 shutdown_connections 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ac806b4d0 0x7f5ac806d990 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae815df00 0x7f5ae80b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ae80b9a10 0x7f5ae80cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 --2- 172.21.15.27:0/2295610439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ae80b95a0 0x7f5ae81f31f0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 >> 172.21.15.27:0/2295610439 conn(0x7f5ae8071960 msgr2=0x7f5ae8071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 shutdown_connections 2024-03-06T07:33:56.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:56.301+0000 7f5aecc0e640 1 -- 172.21.15.27:0/2295610439 wait complete. 2024-03-06T07:33:56.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:56.515 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:57.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:57.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.623+0000 7faaa32f4640 1 Processor -- start 2024-03-06T07:33:57.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.623+0000 7faaa32f4640 1 -- start start 2024-03-06T07:33:57.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa32f4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faa9c15df00 0x7faa9c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa32f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c0b95a0 0x7faa9c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa32f4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9c0cea30 con 0x7faa9c0b9a30 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa32f4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9c0d2af0 con 0x7faa9c15df00 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa32f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c0cf3c0 con 0x7faa9c0b95a0 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa1090640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faa9c15df00 0x7faa9c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa1090640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faa9c15df00 0x7faa9c0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38034/0 (socket says 172.21.15.27:38034) 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa1090640 1 -- 172.21.15.27:0/4183531678 learned_addr learned my addr 172.21.15.27:0/4183531678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa1090640 1 -- 172.21.15.27:0/4183531678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c0b95a0 msgr2=0x7faa9c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa1090640 1 --2- 172.21.15.27:0/4183531678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c0b95a0 0x7faa9c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa1090640 1 -- 172.21.15.27:0/4183531678 >> v1:172.21.15.69:6789/0 conn(0x7faa9c0b9a30 legacy=0x7faa9c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.624+0000 7faaa1090640 1 -- 172.21.15.27:0/4183531678 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c0cf0e0 con 0x7faa9c15df00 2024-03-06T07:33:57.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.625+0000 7faaa1090640 1 --2- 172.21.15.27:0/4183531678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c15df00 0x7faa9c0d25b0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7faa98009870 tx=0x7faa9802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90ad06a25c9db59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.625+0000 7faa8ffff640 1 -- 172.21.15.27:0/4183531678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98035020 con 0x7faa9c15df00 2024-03-06T07:33:57.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.625+0000 7faa8ffff640 1 -- 172.21.15.27:0/4183531678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa98031640 con 0x7faa9c15df00 2024-03-06T07:33:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.625+0000 7faa8ffff640 1 -- 172.21.15.27:0/4183531678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9802d040 con 0x7faa9c15df00 2024-03-06T07:33:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.626+0000 7faaa32f4640 1 -- 172.21.15.27:0/4183531678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c15df00 msgr2=0x7faa9c0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.626+0000 7faaa32f4640 1 --2- 172.21.15.27:0/4183531678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c15df00 0x7faa9c0d25b0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7faa98009870 tx=0x7faa9802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.626+0000 7faaa32f4640 1 -- 172.21.15.27:0/4183531678 shutdown_connections 2024-03-06T07:33:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.626+0000 7faaa32f4640 1 --2- 172.21.15.27:0/4183531678 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c0b95a0 0x7faa9c0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.626+0000 7faaa32f4640 1 --2- 172.21.15.27:0/4183531678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c15df00 0x7faa9c0d25b0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.626+0000 7faaa32f4640 1 -- 172.21.15.27:0/4183531678 >> 172.21.15.27:0/4183531678 conn(0x7faa9c071960 msgr2=0x7faa9c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.626+0000 7faaa32f4640 1 -- 172.21.15.27:0/4183531678 shutdown_connections 2024-03-06T07:33:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.626+0000 7faaa32f4640 1 -- 172.21.15.27:0/4183531678 wait complete. 2024-03-06T07:33:57.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.627+0000 7faaa32f4640 1 Processor -- start 2024-03-06T07:33:57.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.627+0000 7faaa32f4640 1 -- start start 2024-03-06T07:33:57.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa32f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa9c0b95a0 0x7faa9c15b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa32f4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c15df00 0x7faa9c0cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa32f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c1e7880 0x7faa9c0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa32f4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c0cec70 con 0x7faa9c1e7880 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa32f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c0ce750 con 0x7faa9c0b95a0 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa32f4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c0cedf0 con 0x7faa9c15df00 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1891640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c1e7880 0x7faa9c0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1891640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c1e7880 0x7faa9c0cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38042/0 (socket says 172.21.15.27:38042) 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1891640 1 -- 172.21.15.27:0/2532056621 learned_addr learned my addr 172.21.15.27:0/2532056621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa088f640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c15df00 0x7faa9c0cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1090640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa9c0b95a0 0x7faa9c15b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1891640 1 -- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c15df00 msgr2=0x7faa9c0cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1891640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c15df00 0x7faa9c0cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1891640 1 -- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa9c0b95a0 msgr2=0x7faa9c15b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1891640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa9c0b95a0 0x7faa9c15b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1891640 1 -- 172.21.15.27:0/2532056621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c15bc40 con 0x7faa9c1e7880 2024-03-06T07:33:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa088f640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c15df00 0x7faa9c0cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:57.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faaa1891640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c1e7880 0x7faa9c0cad50 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7faa9400c700 tx=0x7faa9400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.628+0000 7faa8dffb640 1 -- 172.21.15.27:0/2532056621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94016020 con 0x7faa9c1e7880 2024-03-06T07:33:57.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.629+0000 7faa8dffb640 1 -- 172.21.15.27:0/2532056621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9400aa50 con 0x7faa9c1e7880 2024-03-06T07:33:57.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.629+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c15bf80 con 0x7faa9c1e7880 2024-03-06T07:33:57.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.629+0000 7faa8dffb640 1 -- 172.21.15.27:0/2532056621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9400ce10 con 0x7faa9c1e7880 2024-03-06T07:33:57.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.629+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c15c7c0 con 0x7faa9c1e7880 2024-03-06T07:33:57.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.630+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c067ff0 con 0x7faa9c1e7880 2024-03-06T07:33:57.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.631+0000 7faa8dffb640 1 -- 172.21.15.27:0/2532056621 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa94019020 con 0x7faa9c1e7880 2024-03-06T07:33:57.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.631+0000 7faa8dffb640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa8006b4d0 0x7faa8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.631+0000 7faa8dffb640 1 -- 172.21.15.27:0/2532056621 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faa9408a600 con 0x7faa9c1e7880 2024-03-06T07:33:57.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.631+0000 7faaa1090640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa8006b4d0 0x7faa8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.632+0000 7faaa1090640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa8006b4d0 0x7faa8006d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7faa98036000 tx=0x7faa9803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:57.636+0000 7faa8dffb640 1 -- 172.21.15.27:0/2532056621 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa94020020 con 0x7faa9c1e7880 2024-03-06T07:33:58.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.047+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa9c0b4880 con 0x7faa9c1e7880 2024-03-06T07:33:58.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.047+0000 7faa8dffb640 1 -- 172.21.15.27:0/2532056621 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7faa9c15bf80 con 0x7faa9c1e7880 2024-03-06T07:33:58.049 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:58.049 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:58.049 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa8006b4d0 msgr2=0x7faa8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa8006b4d0 0x7faa8006d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7faa98036000 tx=0x7faa9803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c1e7880 msgr2=0x7faa9c0cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c1e7880 0x7faa9c0cad50 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7faa9400c700 tx=0x7faa9400cbc0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 shutdown_connections 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faa8006b4d0 0x7faa8006d990 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa9c1e7880 0x7faa9c0cad50 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faa9c15df00 0x7faa9c0cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 --2- 172.21.15.27:0/2532056621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa9c0b95a0 0x7faa9c15b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 >> 172.21.15.27:0/2532056621 conn(0x7faa9c071960 msgr2=0x7faa9c0b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 shutdown_connections 2024-03-06T07:33:58.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:58.050+0000 7faaa32f4640 1 -- 172.21.15.27:0/2532056621 wait complete. 2024-03-06T07:33:58.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:58.268 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:33:59.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:59.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.375+0000 7f1ee293e640 1 Processor -- start 2024-03-06T07:33:59.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.375+0000 7f1ee293e640 1 -- start start 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1ee293e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1edc0b9a10 0x7f1edc0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1ee293e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc0cea30 con 0x7f1edc0b95a0 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1ee293e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc0cf3c0 con 0x7f1edc15df00 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1ee293e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1edc0cf0e0 con 0x7f1edc0b9a10 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1ee0edb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1edc0b9a10 0x7f1edc0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1edb7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1edc0b95a0 0x7f1edc0b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56744/0 (socket says 172.21.15.27:56744) 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1edb7fe640 1 -- 172.21.15.27:0/2836399190 learned_addr learned my addr 172.21.15.27:0/2836399190 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1ee0edb640 1 -- 172.21.15.27:0/2836399190 >> v1:172.21.15.69:6789/0 conn(0x7f1edc15df00 legacy=0x7f1edc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1ee0edb640 1 -- 172.21.15.27:0/2836399190 >> v1:172.21.15.27:6789/0 conn(0x7f1edc0b95a0 legacy=0x7f1edc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1ee0edb640 1 -- 172.21.15.27:0/2836399190 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc0ce470 con 0x7f1edc0b9a10 2024-03-06T07:33:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.376+0000 7f1edaffd640 1 -- 172.21.15.27:0/2836399190 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080335751 0 0) 0x7f1edc0cea30 con 0x7f1edc0b95a0 2024-03-06T07:33:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1ee0edb640 1 --2- 172.21.15.27:0/2836399190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b9a10 0x7f1edc0d3220 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0cf590 tx=0x7f1ed402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6cd1e07adac438c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1edaffd640 1 -- 172.21.15.27:0/2836399190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4039070 con 0x7f1edc0b9a10 2024-03-06T07:33:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1edaffd640 1 -- 172.21.15.27:0/2836399190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed402cce0 con 0x7f1edc0b9a10 2024-03-06T07:33:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1edaffd640 1 -- 172.21.15.27:0/2836399190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4034660 con 0x7f1edc0b9a10 2024-03-06T07:33:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1ee293e640 1 -- 172.21.15.27:0/2836399190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b9a10 msgr2=0x7f1edc0d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1ee293e640 1 --2- 172.21.15.27:0/2836399190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b9a10 0x7f1edc0d3220 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0cf590 tx=0x7f1ed402bf30 comp rx=0 tx=0).stop 2024-03-06T07:33:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1ee293e640 1 -- 172.21.15.27:0/2836399190 shutdown_connections 2024-03-06T07:33:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1ee293e640 1 --2- 172.21.15.27:0/2836399190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b9a10 0x7f1edc0d3220 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1ee293e640 1 -- 172.21.15.27:0/2836399190 >> 172.21.15.27:0/2836399190 conn(0x7f1edc071960 msgr2=0x7f1edc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1ee293e640 1 -- 172.21.15.27:0/2836399190 shutdown_connections 2024-03-06T07:33:59.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.378+0000 7f1ee293e640 1 -- 172.21.15.27:0/2836399190 wait complete. 2024-03-06T07:33:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.380+0000 7f1ee293e640 1 Processor -- start 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.380+0000 7f1ee293e640 1 -- start start 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.380+0000 7f1ee293e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b95a0 0x7f1edc154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.380+0000 7f1ee293e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1edc0b9a10 0x7f1edc15e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.380+0000 7f1ee293e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1edc15df00 0x7f1edc0cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.380+0000 7f1ee293e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc0cedf0 con 0x7f1edc0b95a0 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.380+0000 7f1ee293e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc0cec70 con 0x7f1edc15df00 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.380+0000 7f1ee293e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc0b5e10 con 0x7f1edc0b9a10 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b95a0 0x7f1edc154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b95a0 0x7f1edc154e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38070/0 (socket says 172.21.15.27:38070) 2024-03-06T07:33:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edbfff640 1 -- 172.21.15.27:0/3220369562 learned_addr learned my addr 172.21.15.27:0/3220369562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1ee0edb640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1edc15df00 0x7f1edc0cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edb7fe640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1edc0b9a10 0x7f1edc15e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edbfff640 1 -- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1edc0b9a10 msgr2=0x7f1edc15e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edbfff640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1edc0b9a10 0x7f1edc15e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edbfff640 1 -- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1edc15df00 msgr2=0x7f1edc0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edbfff640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1edc15df00 0x7f1edc0cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edbfff640 1 -- 172.21.15.27:0/3220369562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc0cc180 con 0x7f1edc0b95a0 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1ee0edb640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1edc15df00 0x7f1edc0cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1edbfff640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b95a0 0x7f1edc154e00 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00c430 tx=0x7f1ecc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1ed8ff9640 1 -- 172.21.15.27:0/3220369562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc017070 con 0x7f1edc0b95a0 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1ed8ff9640 1 -- 172.21.15.27:0/3220369562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc0052d0 con 0x7f1edc0b95a0 2024-03-06T07:33:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc0cc460 con 0x7f1edc0b95a0 2024-03-06T07:33:59.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1ed8ff9640 1 -- 172.21.15.27:0/3220369562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc003830 con 0x7f1edc0b95a0 2024-03-06T07:33:59.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.381+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc0cca20 con 0x7f1edc0b95a0 2024-03-06T07:33:59.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.382+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1edc067ff0 con 0x7f1edc0b95a0 2024-03-06T07:33:59.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.383+0000 7f1ed8ff9640 1 -- 172.21.15.27:0/3220369562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ecc01a070 con 0x7f1edc0b95a0 2024-03-06T07:33:59.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.383+0000 7f1ed8ff9640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ec806b500 0x7f1ec806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.383+0000 7f1ed8ff9640 1 -- 172.21.15.27:0/3220369562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1ecc089510 con 0x7f1edc0b95a0 2024-03-06T07:33:59.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.383+0000 7f1edb7fe640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ec806b500 0x7f1ec806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.384+0000 7f1edb7fe640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ec806b500 0x7f1ec806d9c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0007fd0 tx=0x7f1ed0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.388+0000 7f1ed8ff9640 1 -- 172.21.15.27:0/3220369562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ecc058400 con 0x7f1edc0b95a0 2024-03-06T07:33:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.608+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1edc06b780 con 0x7f1edc0b95a0 2024-03-06T07:33:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.608+0000 7f1ed8ff9640 1 -- 172.21.15.27:0/3220369562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f1ecc057d00 con 0x7f1edc0b95a0 2024-03-06T07:33:59.610 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:33:59.610 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:33:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:33:59.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ec806b500 msgr2=0x7f1ec806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ec806b500 0x7f1ec806d9c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0007fd0 tx=0x7f1ed0009040 comp rx=0 tx=0).stop 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b95a0 msgr2=0x7f1edc154e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b95a0 0x7f1edc154e00 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00c430 tx=0x7f1ecc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 shutdown_connections 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ec806b500 0x7f1ec806d9c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1edc15df00 0x7f1edc0cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1edc0b9a10 0x7f1edc15e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 --2- 172.21.15.27:0/3220369562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1edc0b95a0 0x7f1edc154e00 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 >> 172.21.15.27:0/3220369562 conn(0x7f1edc071960 msgr2=0x7f1edc0b5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 shutdown_connections 2024-03-06T07:33:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:33:59.611+0000 7f1ee293e640 1 -- 172.21.15.27:0/3220369562 wait complete. 2024-03-06T07:33:59.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:33:59.825 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:00.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.932+0000 7fcf83081640 1 Processor -- start 2024-03-06T07:34:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.932+0000 7fcf83081640 1 -- start start 2024-03-06T07:34:00.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf83081640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 0x7fcf7c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf83081640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcf7c0b9a10 0x7fcf7c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf83081640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c0cea30 con 0x7fcf7c15df00 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf83081640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf7c0cf3c0 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf83081640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c0cf0e0 con 0x7fcf7c0b95a0 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf8161e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcf7c0b9a10 0x7fcf7c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf80e1d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcf7c15df00 0x7fcf7c0d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56750/0 (socket says 172.21.15.27:56750) 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf80e1d640 1 -- 172.21.15.27:0/2370077721 learned_addr learned my addr 172.21.15.27:0/2370077721 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf7bfff640 1 --2- 172.21.15.27:0/2370077721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 0x7fcf7c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf8161e640 1 -- 172.21.15.27:0/2370077721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 msgr2=0x7fcf7c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf8161e640 1 --2- 172.21.15.27:0/2370077721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 0x7fcf7c0b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf8161e640 1 -- 172.21.15.27:0/2370077721 >> v1:172.21.15.27:6789/0 conn(0x7fcf7c15df00 legacy=0x7fcf7c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf7b7fe640 1 -- 172.21.15.27:0/2370077721 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417324860 0 0) 0x7fcf7c0cea30 con 0x7fcf7c15df00 2024-03-06T07:34:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.933+0000 7fcf8161e640 1 -- 172.21.15.27:0/2370077721 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c0ce470 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.934+0000 7fcf8161e640 1 --2- 172.21.15.27:0/2370077721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 0x7fcf7c0b5c40 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fcf74009d90 tx=0x7fcf7402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b15792dd138f553 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.934+0000 7fcf7b7fe640 1 -- 172.21.15.27:0/2370077721 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf74039070 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.934+0000 7fcf7b7fe640 1 -- 172.21.15.27:0/2370077721 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7402cc50 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.934+0000 7fcf7b7fe640 1 -- 172.21.15.27:0/2370077721 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf74034440 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.935+0000 7fcf83081640 1 -- 172.21.15.27:0/2370077721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 msgr2=0x7fcf7c0b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.935+0000 7fcf83081640 1 --2- 172.21.15.27:0/2370077721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 0x7fcf7c0b5c40 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fcf74009d90 tx=0x7fcf7402bea0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.935+0000 7fcf83081640 1 -- 172.21.15.27:0/2370077721 shutdown_connections 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.935+0000 7fcf83081640 1 --2- 172.21.15.27:0/2370077721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 0x7fcf7c0b5c40 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.935+0000 7fcf83081640 1 --2- 172.21.15.27:0/2370077721 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 0x7fcf7c0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.935+0000 7fcf83081640 1 -- 172.21.15.27:0/2370077721 >> 172.21.15.27:0/2370077721 conn(0x7fcf7c071960 msgr2=0x7fcf7c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.935+0000 7fcf83081640 1 -- 172.21.15.27:0/2370077721 shutdown_connections 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.935+0000 7fcf83081640 1 -- 172.21.15.27:0/2370077721 wait complete. 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.936+0000 7fcf83081640 1 Processor -- start 2024-03-06T07:34:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.936+0000 7fcf83081640 1 -- start start 2024-03-06T07:34:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf83081640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 0x7fcf7c0c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf83081640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 0x7fcf7c0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf83081640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf7c15df00 0x7fcf7c0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf83081640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c0cedf0 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf83081640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c0cec70 con 0x7fcf7c15df00 2024-03-06T07:34:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf83081640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c0d3220 con 0x7fcf7c0b95a0 2024-03-06T07:34:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 0x7fcf7c0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 0x7fcf7c0ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38086/0 (socket says 172.21.15.27:38086) 2024-03-06T07:34:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf7bfff640 1 -- 172.21.15.27:0/3060165403 learned_addr learned my addr 172.21.15.27:0/3060165403 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf80e1d640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 0x7fcf7c0c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf8161e640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf7c15df00 0x7fcf7c0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf7bfff640 1 -- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 msgr2=0x7fcf7c0c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf7bfff640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 0x7fcf7c0c81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf7bfff640 1 -- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf7c15df00 msgr2=0x7fcf7c0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf7bfff640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf7c15df00 0x7fcf7c0cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf7bfff640 1 -- 172.21.15.27:0/3060165403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c0c8850 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf80e1d640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 0x7fcf7c0c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf8161e640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf7c15df00 0x7fcf7c0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf7bfff640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 0x7fcf7c0ca600 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800d830 tx=0x7fcf6800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.937+0000 7fcf797fa640 1 -- 172.21.15.27:0/3060165403 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6800b6c0 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.938+0000 7fcf797fa640 1 -- 172.21.15.27:0/3060165403 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf68019070 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.938+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7c0c8b30 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.938+0000 7fcf797fa640 1 -- 172.21.15.27:0/3060165403 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6800ba80 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.938+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7c14eb10 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.939+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7c067ff0 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.939+0000 7fcf797fa640 1 -- 172.21.15.27:0/3060165403 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf6800bc20 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.940+0000 7fcf797fa640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf6006b4d0 0x7fcf6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.940+0000 7fcf797fa640 1 -- 172.21.15.27:0/3060165403 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcf6808a670 con 0x7fcf7c0b9a10 2024-03-06T07:34:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.940+0000 7fcf80e1d640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf6006b4d0 0x7fcf6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.940+0000 7fcf80e1d640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf6006b4d0 0x7fcf6006d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fcf70000f60 tx=0x7fcf70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:00.944+0000 7fcf797fa640 1 -- 172.21.15.27:0/3060165403 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf6801c360 con 0x7fcf7c0b9a10 2024-03-06T07:34:01.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.201+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf7c0c8d00 con 0x7fcf7c0b9a10 2024-03-06T07:34:01.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.201+0000 7fcf797fa640 1 -- 172.21.15.27:0/3060165403 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fcf68059560 con 0x7fcf7c0b9a10 2024-03-06T07:34:01.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:01.203 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.203+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf6006b4d0 msgr2=0x7fcf6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.203+0000 7fcf83081640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf6006b4d0 0x7fcf6006d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fcf70000f60 tx=0x7fcf70009040 comp rx=0 tx=0).stop 2024-03-06T07:34:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.203+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 msgr2=0x7fcf7c0ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.203+0000 7fcf83081640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 0x7fcf7c0ca600 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800d830 tx=0x7fcf6800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.204+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 shutdown_connections 2024-03-06T07:34:01.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.204+0000 7fcf83081640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcf6006b4d0 0x7fcf6006d990 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.204+0000 7fcf83081640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf7c15df00 0x7fcf7c0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.204+0000 7fcf83081640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf7c0b9a10 0x7fcf7c0ca600 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.204+0000 7fcf83081640 1 --2- 172.21.15.27:0/3060165403 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcf7c0b95a0 0x7fcf7c0c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.204+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 >> 172.21.15.27:0/3060165403 conn(0x7fcf7c071960 msgr2=0x7fcf7c067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.204+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 shutdown_connections 2024-03-06T07:34:01.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:01.204+0000 7fcf83081640 1 -- 172.21.15.27:0/3060165403 wait complete. 2024-03-06T07:34:01.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:01.417 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:02.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.524+0000 7f7995179640 1 Processor -- start 2024-03-06T07:34:02.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.524+0000 7f7995179640 1 -- start start 2024-03-06T07:34:02.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.525+0000 7f7995179640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79900d2e60 0x7f79900b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.525+0000 7f7995179640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79900b5c40 0x7f79900b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.525+0000 7f7995179640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79900b83c0 con 0x7f799015dd70 2024-03-06T07:34:02.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.525+0000 7f7995179640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79900b8fd0 con 0x7f79900b5c40 2024-03-06T07:34:02.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.525+0000 7f7995179640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79900b86a0 con 0x7f79900d2e60 2024-03-06T07:34:02.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.525+0000 7f798f59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79900b5c40 0x7f79900b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.525+0000 7f798f59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79900b5c40 0x7f79900b6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38094/0 (socket says 172.21.15.27:38094) 2024-03-06T07:34:02.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.525+0000 7f798f59e640 1 -- 172.21.15.27:0/1056770878 learned_addr learned my addr 172.21.15.27:0/1056770878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:02.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.526+0000 7f798e59c640 1 --2- 172.21.15.27:0/1056770878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79900d2e60 0x7f79900b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.526+0000 7f798f59e640 1 -- 172.21.15.27:0/1056770878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79900d2e60 msgr2=0x7f79900b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.526+0000 7f798f59e640 1 --2- 172.21.15.27:0/1056770878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79900d2e60 0x7f79900b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.526+0000 7f798f59e640 1 -- 172.21.15.27:0/1056770878 >> v1:172.21.15.69:6789/0 conn(0x7f799015dd70 legacy=0x7f79900b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.526+0000 7f798f59e640 1 -- 172.21.15.27:0/1056770878 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79900b8980 con 0x7f79900b5c40 2024-03-06T07:34:02.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.526+0000 7f798e59c640 1 --2- 172.21.15.27:0/1056770878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79900d2e60 0x7f79900b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.527+0000 7f798f59e640 1 --2- 172.21.15.27:0/1056770878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900b5c40 0x7f79900b6c60 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f7980009e50 tx=0x7f798002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368082a003fe5d74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.527+0000 7f798dd9b640 1 -- 172.21.15.27:0/1056770878 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798002cb90 con 0x7f79900b5c40 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.527+0000 7f798dd9b640 1 -- 172.21.15.27:0/1056770878 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798002cd30 con 0x7f79900b5c40 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.527+0000 7f798dd9b640 1 -- 172.21.15.27:0/1056770878 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980034610 con 0x7f79900b5c40 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.528+0000 7f7995179640 1 -- 172.21.15.27:0/1056770878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900b5c40 msgr2=0x7f79900b6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.528+0000 7f7995179640 1 --2- 172.21.15.27:0/1056770878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900b5c40 0x7f79900b6c60 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f7980009e50 tx=0x7f798002bff0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.528+0000 7f7995179640 1 -- 172.21.15.27:0/1056770878 shutdown_connections 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.528+0000 7f7995179640 1 --2- 172.21.15.27:0/1056770878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900b5c40 0x7f79900b6c60 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.528+0000 7f7995179640 1 --2- 172.21.15.27:0/1056770878 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f79900d2e60 0x7f79900b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.528+0000 7f7995179640 1 -- 172.21.15.27:0/1056770878 >> 172.21.15.27:0/1056770878 conn(0x7f7990071960 msgr2=0x7f7990071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.528+0000 7f7995179640 1 -- 172.21.15.27:0/1056770878 shutdown_connections 2024-03-06T07:34:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.528+0000 7f7995179640 1 -- 172.21.15.27:0/1056770878 wait complete. 2024-03-06T07:34:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.529+0000 7f7995179640 1 Processor -- start 2024-03-06T07:34:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.529+0000 7f7995179640 1 -- start start 2024-03-06T07:34:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f7995179640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79900b5c40 0x7f79901f7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f7995179640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900d2e60 0x7f7990155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f7995179640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f799015dd70 0x7f7990160310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f7995179640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799015e2d0 con 0x7f79900d2e60 2024-03-06T07:34:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f7995179640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799015e150 con 0x7f79900b5c40 2024-03-06T07:34:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f7995179640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79900b71a0 con 0x7f799015dd70 2024-03-06T07:34:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900d2e60 0x7f7990155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900d2e60 0x7f7990155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38110/0 (socket says 172.21.15.27:38110) 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798e59c640 1 -- 172.21.15.27:0/1053774918 learned_addr learned my addr 172.21.15.27:0/1053774918 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798ed9d640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79900b5c40 0x7f79901f7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798f59e640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f799015dd70 0x7f7990160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798e59c640 1 -- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f799015dd70 msgr2=0x7f7990160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798e59c640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f799015dd70 0x7f7990160310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798e59c640 1 -- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79900b5c40 msgr2=0x7f79901f7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798e59c640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79900b5c40 0x7f79901f7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798e59c640 1 -- 172.21.15.27:0/1053774918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7990158fe0 con 0x7f79900d2e60 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798ed9d640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79900b5c40 0x7f79901f7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.530+0000 7f798f59e640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f799015dd70 0x7f7990160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.531+0000 7f798e59c640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900d2e60 0x7f7990155cb0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f798400e730 tx=0x7f798400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.531+0000 7f79777fe640 1 -- 172.21.15.27:0/1053774918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400f040 con 0x7f79900d2e60 2024-03-06T07:34:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.531+0000 7f79777fe640 1 -- 172.21.15.27:0/1053774918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7984004230 con 0x7f79900d2e60 2024-03-06T07:34:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.531+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79901561f0 con 0x7f79900d2e60 2024-03-06T07:34:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.531+0000 7f79777fe640 1 -- 172.21.15.27:0/1053774918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400ee40 con 0x7f79900d2e60 2024-03-06T07:34:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.531+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7990158d00 con 0x7f79900d2e60 2024-03-06T07:34:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.532+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7990067ff0 con 0x7f79900d2e60 2024-03-06T07:34:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.532+0000 7f79777fe640 1 -- 172.21.15.27:0/1053774918 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f798400c6d0 con 0x7f79900d2e60 2024-03-06T07:34:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.533+0000 7f79777fe640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f797c06b4d0 0x7f797c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.533+0000 7f79777fe640 1 -- 172.21.15.27:0/1053774918 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7984089f50 con 0x7f79900d2e60 2024-03-06T07:34:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.533+0000 7f798ed9d640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f797c06b4d0 0x7f797c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.533+0000 7f798ed9d640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f797c06b4d0 0x7f797c06d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f7988001660 tx=0x7f7988009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.537+0000 7f79777fe640 1 -- 172.21.15.27:0/1053774918 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7984017020 con 0x7f79900d2e60 2024-03-06T07:34:02.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.756+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f799015e8d0 con 0x7f79900d2e60 2024-03-06T07:34:02.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.756+0000 7f79777fe640 1 -- 172.21.15.27:0/1053774918 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f7984058db0 con 0x7f79900d2e60 2024-03-06T07:34:02.758 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:02.758 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:02.758 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f797c06b4d0 msgr2=0x7f797c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f797c06b4d0 0x7f797c06d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f7988001660 tx=0x7f7988009040 comp rx=0 tx=0).stop 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900d2e60 msgr2=0x7f7990155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900d2e60 0x7f7990155cb0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f798400e730 tx=0x7f798400ebf0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 shutdown_connections 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f797c06b4d0 0x7f797c06d990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f799015dd70 0x7f7990160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79900d2e60 0x7f7990155cb0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 --2- 172.21.15.27:0/1053774918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79900b5c40 0x7f79901f7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 >> 172.21.15.27:0/1053774918 conn(0x7f7990071960 msgr2=0x7f79900b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 shutdown_connections 2024-03-06T07:34:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:02.758+0000 7f7995179640 1 -- 172.21.15.27:0/1053774918 wait complete. 2024-03-06T07:34:02.972 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:02.972 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:03.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.078+0000 7f707d6b5640 1 Processor -- start 2024-03-06T07:34:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.078+0000 7f707d6b5640 1 -- start start 2024-03-06T07:34:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.078+0000 7f707d6b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b95a0 0x7f70780b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.078+0000 7f707d6b5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70780b9a10 0x7f70780b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.078+0000 7f707d6b5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70780cea30 con 0x7f707815df00 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.078+0000 7f707d6b5640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70780cf3c0 con 0x7f70780b9a10 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.078+0000 7f707d6b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780cf0e0 con 0x7f70780b95a0 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f706e7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b95a0 0x7f70780b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f70777fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70780b9a10 0x7f70780b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f706e7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b95a0 0x7f70780b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:45282/0 (socket says 172.21.15.27:45282) 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f706e7fc640 1 -- 172.21.15.27:0/4151459830 learned_addr learned my addr 172.21.15.27:0/4151459830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f70777fe640 1 -- 172.21.15.27:0/4151459830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b95a0 msgr2=0x7f70780b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f70777fe640 1 --2- 172.21.15.27:0/4151459830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b95a0 0x7f70780b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f70777fe640 1 -- 172.21.15.27:0/4151459830 >> v1:172.21.15.69:6789/0 conn(0x7f707815df00 legacy=0x7f70780d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f70777fe640 1 -- 172.21.15.27:0/4151459830 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70780ce470 con 0x7f70780b9a10 2024-03-06T07:34:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f70767fc640 1 -- 172.21.15.27:0/4151459830 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629457103 0 0) 0x7f70780cea30 con 0x7f707815df00 2024-03-06T07:34:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.079+0000 7f706e7fc640 1 --2- 172.21.15.27:0/4151459830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b95a0 0x7f70780b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.080+0000 7f70777fe640 1 --2- 172.21.15.27:0/4151459830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780b9a10 0x7f70780b5c40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7070009d90 tx=0x7f707002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98fc78e76adbd1e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.080+0000 7f70767fc640 1 -- 172.21.15.27:0/4151459830 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070039070 con 0x7f70780b9a10 2024-03-06T07:34:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.080+0000 7f70767fc640 1 -- 172.21.15.27:0/4151459830 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707002cc50 con 0x7f70780b9a10 2024-03-06T07:34:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.080+0000 7f70767fc640 1 -- 172.21.15.27:0/4151459830 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070034440 con 0x7f70780b9a10 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.081+0000 7f707d6b5640 1 -- 172.21.15.27:0/4151459830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780b9a10 msgr2=0x7f70780b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.081+0000 7f707d6b5640 1 --2- 172.21.15.27:0/4151459830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780b9a10 0x7f70780b5c40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7070009d90 tx=0x7f707002bea0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.081+0000 7f707d6b5640 1 -- 172.21.15.27:0/4151459830 shutdown_connections 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.081+0000 7f707d6b5640 1 --2- 172.21.15.27:0/4151459830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780b9a10 0x7f70780b5c40 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.081+0000 7f707d6b5640 1 --2- 172.21.15.27:0/4151459830 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f70780b95a0 0x7f70780b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.081+0000 7f707d6b5640 1 -- 172.21.15.27:0/4151459830 >> 172.21.15.27:0/4151459830 conn(0x7f7078071960 msgr2=0x7f7078071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.081+0000 7f707d6b5640 1 -- 172.21.15.27:0/4151459830 shutdown_connections 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.081+0000 7f707d6b5640 1 -- 172.21.15.27:0/4151459830 wait complete. 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.082+0000 7f707d6b5640 1 Processor -- start 2024-03-06T07:34:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.082+0000 7f707d6b5640 1 -- start start 2024-03-06T07:34:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f707d6b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780b95a0 0x7f7078152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f707d6b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780b9a10 0x7f70780c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f707d6b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f707815df00 0x7f70780ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f707d6b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780cedf0 con 0x7f70780b9a10 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f707d6b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780cec70 con 0x7f70780b95a0 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f707d6b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780d3220 con 0x7f707815df00 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f706e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780b9a10 0x7f70780c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f706e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780b9a10 0x7f70780c9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52194/0 (socket says 172.21.15.27:52194) 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f706e7fc640 1 -- 172.21.15.27:0/820436019 learned_addr learned my addr 172.21.15.27:0/820436019 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f70777fe640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f707815df00 0x7f70780ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f7076ffd640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780b95a0 0x7f7078152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f706e7fc640 1 -- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f707815df00 msgr2=0x7f70780ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f706e7fc640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f707815df00 0x7f70780ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f706e7fc640 1 -- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780b95a0 msgr2=0x7f7078152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f706e7fc640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780b95a0 0x7f7078152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.083+0000 7f706e7fc640 1 -- 172.21.15.27:0/820436019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078152640 con 0x7f70780b9a10 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.084+0000 7f706e7fc640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780b9a10 0x7f70780c9740 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f706000e9d0 tx=0x7f706000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.084+0000 7f706ffff640 1 -- 172.21.15.27:0/820436019 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060017070 con 0x7f70780b9a10 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.084+0000 7f706ffff640 1 -- 172.21.15.27:0/820436019 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706000ca60 con 0x7f70780b9a10 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.084+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70780ca1e0 con 0x7f70780b9a10 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.084+0000 7f706ffff640 1 -- 172.21.15.27:0/820436019 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706000ce10 con 0x7f70780b9a10 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.084+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70781f50e0 con 0x7f70780b9a10 2024-03-06T07:34:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.085+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70780cab20 con 0x7f70780b9a10 2024-03-06T07:34:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.086+0000 7f706ffff640 1 -- 172.21.15.27:0/820436019 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f706001a070 con 0x7f70780b9a10 2024-03-06T07:34:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.087+0000 7f706ffff640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f705806b4d0 0x7f705806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.087+0000 7f706ffff640 1 -- 172.21.15.27:0/820436019 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f706008a3d0 con 0x7f70780b9a10 2024-03-06T07:34:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.087+0000 7f7076ffd640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f705806b4d0 0x7f705806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.088+0000 7f7076ffd640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f705806b4d0 0x7f705806d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7068000f60 tx=0x7f7068009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.091+0000 7f706ffff640 1 -- 172.21.15.27:0/820436019 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70780cab20 con 0x7f70780b9a10 2024-03-06T07:34:04.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.313+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70781f4ed0 con 0x7f70780b9a10 2024-03-06T07:34:04.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.313+0000 7f706ffff640 1 -- 172.21.15.27:0/820436019 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f7078152640 con 0x7f70780b9a10 2024-03-06T07:34:04.316 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:04.316 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f705806b4d0 msgr2=0x7f705806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f705806b4d0 0x7f705806d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7068000f60 tx=0x7f7068009040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780b9a10 msgr2=0x7f70780c9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780b9a10 0x7f70780c9740 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f706000e9d0 tx=0x7f706000ee90 comp rx=0 tx=0).stop 2024-03-06T07:34:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 shutdown_connections 2024-03-06T07:34:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f705806b4d0 0x7f705806d990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f707815df00 0x7f70780ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70780b9a10 0x7f70780c9740 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 --2- 172.21.15.27:0/820436019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70780b95a0 0x7f7078152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.315+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 >> 172.21.15.27:0/820436019 conn(0x7f7078071960 msgr2=0x7f70780b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.316+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 shutdown_connections 2024-03-06T07:34:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:04.316+0000 7f707d6b5640 1 -- 172.21.15.27:0/820436019 wait complete. 2024-03-06T07:34:04.530 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:04.530 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:05.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.635+0000 7f0db0088640 1 Processor -- start 2024-03-06T07:34:05.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.636+0000 7f0db0088640 1 -- start start 2024-03-06T07:34:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.636+0000 7f0db0088640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0da815df00 0x7f0da80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.636+0000 7f0db0088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b95a0 0x7f0da80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.636+0000 7f0db0088640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0da80b9a30 0x7f0da80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.636+0000 7f0db0088640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0da80cea30 con 0x7f0da80b9a30 2024-03-06T07:34:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.636+0000 7f0db0088640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0da80d2af0 con 0x7f0da815df00 2024-03-06T07:34:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.636+0000 7f0db0088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da80cf3c0 con 0x7f0da80b95a0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dae625640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0da80b9a30 0x7f0da80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dae625640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0da80b9a30 0x7f0da80b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52202/0 (socket says 172.21.15.27:52202) 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dae625640 1 -- 172.21.15.27:0/2322617669 learned_addr learned my addr 172.21.15.27:0/2322617669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dade24640 1 --2- 172.21.15.27:0/2322617669 >> v2:172.21.15.69:3300/0 conn(0x7f0da815df00 0x7f0da80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dad623640 1 --2- 172.21.15.27:0/2322617669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b95a0 0x7f0da80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dae625640 1 -- 172.21.15.27:0/2322617669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b95a0 msgr2=0x7f0da80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dae625640 1 --2- 172.21.15.27:0/2322617669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b95a0 0x7f0da80b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dae625640 1 -- 172.21.15.27:0/2322617669 >> v2:172.21.15.69:3300/0 conn(0x7f0da815df00 msgr2=0x7f0da80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dae625640 1 --2- 172.21.15.27:0/2322617669 >> v2:172.21.15.69:3300/0 conn(0x7f0da815df00 0x7f0da80d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dae625640 1 -- 172.21.15.27:0/2322617669 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da80cf0e0 con 0x7f0da80b9a30 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dade24640 1 --2- 172.21.15.27:0/2322617669 >> v2:172.21.15.69:3300/0 conn(0x7f0da815df00 0x7f0da80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.637+0000 7f0dad623640 1 --2- 172.21.15.27:0/2322617669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b95a0 0x7f0da80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0dae625640 1 --2- 172.21.15.27:0/2322617669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b9a30 0x7f0da80b5c40 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f0da80d11b0 tx=0x7f0da402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c831e0c5e473a913 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0dace22640 1 -- 172.21.15.27:0/2322617669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da402ca80 con 0x7f0da80b9a30 2024-03-06T07:34:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0dace22640 1 -- 172.21.15.27:0/2322617669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da402cc20 con 0x7f0da80b9a30 2024-03-06T07:34:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0dace22640 1 -- 172.21.15.27:0/2322617669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da40343f0 con 0x7f0da80b9a30 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0db0088640 1 -- 172.21.15.27:0/2322617669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b9a30 msgr2=0x7f0da80b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0db0088640 1 --2- 172.21.15.27:0/2322617669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b9a30 0x7f0da80b5c40 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f0da80d11b0 tx=0x7f0da402bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0db0088640 1 -- 172.21.15.27:0/2322617669 shutdown_connections 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0db0088640 1 --2- 172.21.15.27:0/2322617669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b9a30 0x7f0da80b5c40 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0db0088640 1 --2- 172.21.15.27:0/2322617669 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b95a0 0x7f0da80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0db0088640 1 --2- 172.21.15.27:0/2322617669 >> v2:172.21.15.69:3300/0 conn(0x7f0da815df00 0x7f0da80d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.638+0000 7f0db0088640 1 -- 172.21.15.27:0/2322617669 >> 172.21.15.27:0/2322617669 conn(0x7f0da8071960 msgr2=0x7f0da8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.639+0000 7f0db0088640 1 -- 172.21.15.27:0/2322617669 shutdown_connections 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.639+0000 7f0db0088640 1 -- 172.21.15.27:0/2322617669 wait complete. 2024-03-06T07:34:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.641+0000 7f0db0088640 1 Processor -- start 2024-03-06T07:34:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.641+0000 7f0db0088640 1 -- start start 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0db0088640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b95a0 0x7f0da81f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0db0088640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0da815df00 0x7f0da8151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0db0088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b2df0 0x7f0da80b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0db0088640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da80cec70 con 0x7f0da80b95a0 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0db0088640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da80ce750 con 0x7f0da815df00 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0db0088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da80cedf0 con 0x7f0da80b2df0 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dade24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b95a0 0x7f0da81f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dade24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b95a0 0x7f0da81f4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52218/0 (socket says 172.21.15.27:52218) 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dade24640 1 -- 172.21.15.27:0/3981018229 learned_addr learned my addr 172.21.15.27:0/3981018229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dad623640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0da815df00 0x7f0da8151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dae625640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b2df0 0x7f0da80b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dade24640 1 -- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b2df0 msgr2=0x7f0da80b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dade24640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b2df0 0x7f0da80b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dade24640 1 -- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0da815df00 msgr2=0x7f0da8151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dade24640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0da815df00 0x7f0da8151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dade24640 1 -- 172.21.15.27:0/3981018229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da80b3590 con 0x7f0da80b95a0 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.642+0000 7f0dad623640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0da815df00 0x7f0da8151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.643+0000 7f0dade24640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b95a0 0x7f0da81f4620 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f0da000c7f0 tx=0x7f0da000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.643+0000 7f0d9e7fc640 1 -- 172.21.15.27:0/3981018229 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0015070 con 0x7f0da80b95a0 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.643+0000 7f0d9e7fc640 1 -- 172.21.15.27:0/3981018229 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da0007a30 con 0x7f0da80b95a0 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.643+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da80b3870 con 0x7f0da80b95a0 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.643+0000 7f0d9e7fc640 1 -- 172.21.15.27:0/3981018229 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0007e00 con 0x7f0da80b95a0 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.643+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da80b3e30 con 0x7f0da80b95a0 2024-03-06T07:34:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.644+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da8067ff0 con 0x7f0da80b95a0 2024-03-06T07:34:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.644+0000 7f0d9e7fc640 1 -- 172.21.15.27:0/3981018229 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0da0018070 con 0x7f0da80b95a0 2024-03-06T07:34:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.645+0000 7f0d9e7fc640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.645+0000 7f0d9e7fc640 1 -- 172.21.15.27:0/3981018229 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0da008a170 con 0x7f0da80b95a0 2024-03-06T07:34:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.645+0000 7f0dad623640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.646+0000 7f0dad623640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0da80c7d60 tx=0x7f0d98008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.649+0000 7f0d9e7fc640 1 -- 172.21.15.27:0/3981018229 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0da0058fd0 con 0x7f0da80b95a0 2024-03-06T07:34:05.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.868+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0da80b4b10 con 0x7f0da80b95a0 2024-03-06T07:34:05.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.869+0000 7f0d9e7fc640 1 -- 172.21.15.27:0/3981018229 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f0da00588d0 con 0x7f0da80b95a0 2024-03-06T07:34:05.871 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:05.871 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:05.872 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:05.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.870+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d8c06b4d0 msgr2=0x7f0d8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.870+0000 7f0db0088640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0da80c7d60 tx=0x7f0d98008040 comp rx=0 tx=0).stop 2024-03-06T07:34:05.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.870+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b95a0 msgr2=0x7f0da81f4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.870+0000 7f0db0088640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b95a0 0x7f0da81f4620 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f0da000c7f0 tx=0x7f0da000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.871+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 shutdown_connections 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.871+0000 7f0db0088640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.871+0000 7f0db0088640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0da80b2df0 0x7f0da80b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.871+0000 7f0db0088640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0da815df00 0x7f0da8151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.871+0000 7f0db0088640 1 --2- 172.21.15.27:0/3981018229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0da80b95a0 0x7f0da81f4620 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.871+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 >> 172.21.15.27:0/3981018229 conn(0x7f0da8071960 msgr2=0x7f0da80b87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.871+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 shutdown_connections 2024-03-06T07:34:05.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:05.871+0000 7f0db0088640 1 -- 172.21.15.27:0/3981018229 wait complete. 2024-03-06T07:34:06.085 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:06.086 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:07.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.191+0000 7fd402a0e640 1 Processor -- start 2024-03-06T07:34:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.191+0000 7fd402a0e640 1 -- start start 2024-03-06T07:34:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.192+0000 7fd402a0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc15df00 0x7fd3fc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.192+0000 7fd402a0e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd3fc0b95a0 0x7fd3fc0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.192+0000 7fd402a0e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc0cea30 con 0x7fd3fc0b9a30 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.192+0000 7fd402a0e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3fc0d2af0 con 0x7fd3fc0b95a0 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.192+0000 7fd402a0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0cf3c0 con 0x7fd3fc15df00 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.192+0000 7fd400fab640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd3fc0b9a30 0x7fd3fc0d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49330/0 (socket says 172.21.15.27:49330) 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.192+0000 7fd400fab640 1 -- 172.21.15.27:0/4241208841 learned_addr learned my addr 172.21.15.27:0/4241208841 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.192+0000 7fd3fb7fe640 1 --2- 172.21.15.27:0/4241208841 >> v2:172.21.15.69:3300/0 conn(0x7fd3fc0b95a0 0x7fd3fc0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.192+0000 7fd3fbfff640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc15df00 0x7fd3fc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518412378 0 0) 0x7fd3fc0cea30 con 0x7fd3fc0b9a30 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3e8003520 con 0x7fd3fc0b9a30 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2503564545 0 0) 0x7fd3e8003520 con 0x7fd3fc0b9a30 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc15df00 msgr2=0x7fd3fc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc15df00 0x7fd3fc0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 >> v2:172.21.15.69:3300/0 conn(0x7fd3fc0b95a0 msgr2=0x7fd3fc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 --2- 172.21.15.27:0/4241208841 >> v2:172.21.15.69:3300/0 conn(0x7fd3fc0b95a0 0x7fd3fc0b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc0cf0e0 con 0x7fd3fc0b9a30 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fd3f40019f0 con 0x7fd3fc0b9a30 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 >> v1:172.21.15.27:6789/0 conn(0x7fd3fc0b9a30 legacy=0x7fd3fc0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3e80039b0 0x7fd3e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc15df00 0x7fd3e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0d2af0 con 0x7fd3e8004800 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0cf3c0 con 0x7fd3e80039b0 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e800b9c0 con 0x7fd3fc15df00 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3fb7fe640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3fbfff640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3e80039b0 0x7fd3e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd400fab640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc15df00 0x7fd3e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3fb7fe640 1 -- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc15df00 msgr2=0x7fd3e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3fb7fe640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc15df00 0x7fd3e80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3fb7fe640 1 -- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3e80039b0 msgr2=0x7fd3e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3fb7fe640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3e80039b0 0x7fd3e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.193+0000 7fd3fb7fe640 1 -- 172.21.15.27:0/4241208841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e8003520 con 0x7fd3e8004800 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.194+0000 7fd3fb7fe640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3e8004c80 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009ac0 tx=0x7fd3e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74eacb8df23b3c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.194+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4039070 con 0x7fd3e8004800 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.194+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e402c710 con 0x7fd3e8004800 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.194+0000 7fd3faffd640 1 -- 172.21.15.27:0/4241208841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4009690 con 0x7fd3e8004800 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.195+0000 7fd402a0e640 1 -- 172.21.15.27:0/4241208841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 msgr2=0x7fd3e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.195+0000 7fd402a0e640 1 --2- 172.21.15.27:0/4241208841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3e8004c80 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009ac0 tx=0x7fd3e8005830 comp rx=0 tx=0).stop 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.195+0000 7fd400fab640 1 -- 172.21.15.27:0/4241208841 reap_dead start 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.195+0000 7fd402a0e640 1 -- 172.21.15.27:0/4241208841 shutdown_connections 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.195+0000 7fd402a0e640 1 -- 172.21.15.27:0/4241208841 >> 172.21.15.27:0/4241208841 conn(0x7fd3fc071960 msgr2=0x7fd3fc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.195+0000 7fd402a0e640 1 -- 172.21.15.27:0/4241208841 shutdown_connections 2024-03-06T07:34:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.195+0000 7fd402a0e640 1 -- 172.21.15.27:0/4241208841 wait complete. 2024-03-06T07:34:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.197+0000 7fd402a0e640 1 Processor -- start 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.197+0000 7fd402a0e640 1 -- start start 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.198+0000 7fd402a0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3fc151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.198+0000 7fd402a0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc153a10 0x7fd3fc153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.198+0000 7fd402a0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc156860 0x7fd3fc156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.198+0000 7fd402a0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0cec70 con 0x7fd3e8004800 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.198+0000 7fd402a0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0ce750 con 0x7fd3fc153a10 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.198+0000 7fd402a0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0cedf0 con 0x7fd3fc156860 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3fc151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3fc151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52234/0 (socket says 172.21.15.27:52234) 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fbfff640 1 -- 172.21.15.27:0/1622178813 learned_addr learned my addr 172.21.15.27:0/1622178813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fb7fe640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc153a10 0x7fd3fc153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd400fab640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc156860 0x7fd3fc156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fbfff640 1 -- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc156860 msgr2=0x7fd3fc156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fbfff640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc156860 0x7fd3fc156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fbfff640 1 -- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc153a10 msgr2=0x7fd3fc153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fbfff640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc153a10 0x7fd3fc153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fbfff640 1 -- 172.21.15.27:0/1622178813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc152280 con 0x7fd3e8004800 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fb7fe640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc153a10 0x7fd3fc153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3fbfff640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3fc151d40 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0118e0 tx=0x7fd3ec011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3f8ff9640 1 -- 172.21.15.27:0/1622178813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec00faf0 con 0x7fd3e8004800 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3f8ff9640 1 -- 172.21.15.27:0/1622178813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec00fc90 con 0x7fd3e8004800 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc152c50 con 0x7fd3e8004800 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd3f8ff9640 1 -- 172.21.15.27:0/1622178813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec014440 con 0x7fd3e8004800 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.199+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc153210 con 0x7fd3e8004800 2024-03-06T07:34:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.201+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc067ff0 con 0x7fd3e8004800 2024-03-06T07:34:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.201+0000 7fd3f8ff9640 1 -- 172.21.15.27:0/1622178813 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3ec012040 con 0x7fd3e8004800 2024-03-06T07:34:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.201+0000 7fd3f8ff9640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd3dc06b500 0x7fd3dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.201+0000 7fd3f8ff9640 1 -- 172.21.15.27:0/1622178813 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd3ec08a830 con 0x7fd3e8004800 2024-03-06T07:34:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.202+0000 7fd3fb7fe640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd3dc06b500 0x7fd3dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.202+0000 7fd3fb7fe640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd3dc06b500 0x7fd3dc06d9c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4036000 tx=0x7fd3e4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.211+0000 7fd3f8ff9640 1 -- 172.21.15.27:0/1622178813 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3ec019030 con 0x7fd3e8004800 2024-03-06T07:34:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.432+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3fc072000 con 0x7fd3e8004800 2024-03-06T07:34:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.433+0000 7fd3f8ff9640 1 -- 172.21.15.27:0/1622178813 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd3ec059690 con 0x7fd3e8004800 2024-03-06T07:34:07.436 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:07.436 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:07.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd3dc06b500 msgr2=0x7fd3dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd3dc06b500 0x7fd3dc06d9c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4036000 tx=0x7fd3e4030000 comp rx=0 tx=0).stop 2024-03-06T07:34:07.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 msgr2=0x7fd3fc151d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3fc151d40 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0118e0 tx=0x7fd3ec011da0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 shutdown_connections 2024-03-06T07:34:07.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd3dc06b500 0x7fd3dc06d9c0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3fc156860 0x7fd3fc156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc153a10 0x7fd3fc153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 --2- 172.21.15.27:0/1622178813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3e8004800 0x7fd3fc151d40 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 >> 172.21.15.27:0/1622178813 conn(0x7fd3fc071960 msgr2=0x7fd3fc0b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 shutdown_connections 2024-03-06T07:34:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:07.435+0000 7fd402a0e640 1 -- 172.21.15.27:0/1622178813 wait complete. 2024-03-06T07:34:07.649 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:07.650 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:08.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.755+0000 7f43ed7e2640 1 Processor -- start 2024-03-06T07:34:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.755+0000 7f43ed7e2640 1 -- start start 2024-03-06T07:34:08.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43ed7e2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43e80b2c50 0x7f43e80d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43ed7e2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80ce970 con 0x7f43e815e000 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43ed7e2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80cef30 con 0x7f43e8071c50 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43ed7e2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e80cece0 con 0x7f43e80b2c50 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43e77fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43e80b2c50 0x7f43e80d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43e6ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f43e815e000 0x7f43e8071540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49342/0 (socket says 172.21.15.27:49342) 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43e6ffd640 1 -- 172.21.15.27:0/1711609472 learned_addr learned my addr 172.21.15.27:0/1711609472 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43e77fe640 1 -- 172.21.15.27:0/1711609472 >> v1:172.21.15.69:6789/0 conn(0x7f43e8071c50 legacy=0x7f43e80d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43e5ffb640 1 -- 172.21.15.27:0/1711609472 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378626706 0 0) 0x7f43e80ce970 con 0x7f43e815e000 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43e77fe640 1 -- 172.21.15.27:0/1711609472 >> v1:172.21.15.27:6789/0 conn(0x7f43e815e000 legacy=0x7f43e8071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.756+0000 7f43e77fe640 1 -- 172.21.15.27:0/1711609472 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e80cf210 con 0x7f43e80b2c50 2024-03-06T07:34:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.757+0000 7f43e77fe640 1 --2- 172.21.15.27:0/1711609472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e80b2c50 0x7f43e80d2ce0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009d90 tx=0x7f43e002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162b3c69086989be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.757+0000 7f43e5ffb640 1 -- 172.21.15.27:0/1711609472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0039070 con 0x7f43e80b2c50 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.757+0000 7f43e5ffb640 1 -- 172.21.15.27:0/1711609472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e002cc60 con 0x7f43e80b2c50 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.757+0000 7f43e5ffb640 1 -- 172.21.15.27:0/1711609472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0034660 con 0x7f43e80b2c50 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.758+0000 7f43ed7e2640 1 -- 172.21.15.27:0/1711609472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e80b2c50 msgr2=0x7f43e80d2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.758+0000 7f43ed7e2640 1 --2- 172.21.15.27:0/1711609472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e80b2c50 0x7f43e80d2ce0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009d90 tx=0x7f43e002beb0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.758+0000 7f43ed7e2640 1 -- 172.21.15.27:0/1711609472 shutdown_connections 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.758+0000 7f43ed7e2640 1 --2- 172.21.15.27:0/1711609472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e80b2c50 0x7f43e80d2ce0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.758+0000 7f43ed7e2640 1 -- 172.21.15.27:0/1711609472 >> 172.21.15.27:0/1711609472 conn(0x7f43e80d3c80 msgr2=0x7f43e80d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.758+0000 7f43ed7e2640 1 -- 172.21.15.27:0/1711609472 shutdown_connections 2024-03-06T07:34:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.758+0000 7f43ed7e2640 1 -- 172.21.15.27:0/1711609472 wait complete. 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.759+0000 7f43ed7e2640 1 Processor -- start 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.759+0000 7f43ed7e2640 1 -- start start 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43ed7e2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43e8071c50 0x7f43e814ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43ed7e2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43e80b2c50 0x7f43e81f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43ed7e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e815e000 0x7f43e81f7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43ed7e2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e80b3320 con 0x7f43e815e000 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43ed7e2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e80b31a0 con 0x7f43e80b2c50 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43ed7e2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e815e3e0 con 0x7f43e8071c50 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e815e000 0x7f43e81f7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e815e000 0x7f43e81f7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52246/0 (socket says 172.21.15.27:52246) 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e77fe640 1 -- 172.21.15.27:0/140664463 learned_addr learned my addr 172.21.15.27:0/140664463 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e6ffd640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43e8071c50 0x7f43e814ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e67fc640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43e80b2c50 0x7f43e81f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e77fe640 1 -- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43e8071c50 msgr2=0x7f43e814ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e77fe640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43e8071c50 0x7f43e814ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e77fe640 1 -- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43e80b2c50 msgr2=0x7f43e81f6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e77fe640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43e80b2c50 0x7f43e81f6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e77fe640 1 -- 172.21.15.27:0/140664463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e80cb0e0 con 0x7f43e815e000 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e6ffd640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43e8071c50 0x7f43e814ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.760+0000 7f43e67fc640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43e80b2c50 0x7f43e81f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.761+0000 7f43e77fe640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e815e000 0x7f43e81f7390 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f43e0037040 tx=0x7f43e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.761+0000 7f43cf7fe640 1 -- 172.21.15.27:0/140664463 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0044020 con 0x7f43e815e000 2024-03-06T07:34:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.761+0000 7f43cf7fe640 1 -- 172.21.15.27:0/140664463 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e003a040 con 0x7f43e815e000 2024-03-06T07:34:08.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.761+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e80cb3c0 con 0x7f43e815e000 2024-03-06T07:34:08.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.761+0000 7f43cf7fe640 1 -- 172.21.15.27:0/140664463 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0039050 con 0x7f43e815e000 2024-03-06T07:34:08.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.761+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e80cb980 con 0x7f43e815e000 2024-03-06T07:34:08.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.762+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e8067ff0 con 0x7f43e815e000 2024-03-06T07:34:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.762+0000 7f43cf7fe640 1 -- 172.21.15.27:0/140664463 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43e002cae0 con 0x7f43e815e000 2024-03-06T07:34:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.763+0000 7f43cf7fe640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43d406b4d0 0x7f43d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.763+0000 7f43cf7fe640 1 -- 172.21.15.27:0/140664463 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f43e00ac920 con 0x7f43e815e000 2024-03-06T07:34:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.763+0000 7f43e6ffd640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43d406b4d0 0x7f43d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.764+0000 7f43e6ffd640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43d406b4d0 0x7f43d406d990 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f43dc000f60 tx=0x7f43dc007c50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:08.767+0000 7f43cf7fe640 1 -- 172.21.15.27:0/140664463 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43e004e060 con 0x7f43e815e000 2024-03-06T07:34:09.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.062+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43e80cf7d0 con 0x7f43e815e000 2024-03-06T07:34:09.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.062+0000 7f43cf7fe640 1 -- 172.21.15.27:0/140664463 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f43e007b810 con 0x7f43e815e000 2024-03-06T07:34:09.065 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:09.065 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:09.065 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:09.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43d406b4d0 msgr2=0x7f43d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43d406b4d0 0x7f43d406d990 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f43dc000f60 tx=0x7f43dc007c50 comp rx=0 tx=0).stop 2024-03-06T07:34:09.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e815e000 msgr2=0x7f43e81f7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e815e000 0x7f43e81f7390 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f43e0037040 tx=0x7f43e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:09.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 shutdown_connections 2024-03-06T07:34:09.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f43d406b4d0 0x7f43d406d990 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43e815e000 0x7f43e81f7390 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43e80b2c50 0x7f43e81f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 --2- 172.21.15.27:0/140664463 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f43e8071c50 0x7f43e814ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 >> 172.21.15.27:0/140664463 conn(0x7f43e80d3c80 msgr2=0x7f43e815f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 shutdown_connections 2024-03-06T07:34:09.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:09.064+0000 7f43ed7e2640 1 -- 172.21.15.27:0/140664463 wait complete. 2024-03-06T07:34:09.279 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:09.279 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:10.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:10.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.385+0000 7fedbd78a640 1 Processor -- start 2024-03-06T07:34:10.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.385+0000 7fedbd78a640 1 -- start start 2024-03-06T07:34:10.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedbd78a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7fedb8161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedbd78a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fedb8162000 0x7fedb80ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedbd78a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb806dbe0 con 0x7fedb8147930 2024-03-06T07:34:10.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedbd78a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb8071a50 con 0x7fedb8162000 2024-03-06T07:34:10.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedbd78a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb8071d30 con 0x7fedb80b4e70 2024-03-06T07:34:10.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb6ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fedb8147930 0x7fedb8072530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49346/0 (socket says 172.21.15.27:49346) 2024-03-06T07:34:10.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb6ffd640 1 -- 172.21.15.27:0/3086411423 learned_addr learned my addr 172.21.15.27:0/3086411423 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb77fe640 1 --2- 172.21.15.27:0/3086411423 >> v2:172.21.15.69:3300/0 conn(0x7fedb8162000 0x7fedb80ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb67fc640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7fedb8161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992486801 0 0) 0x7fedb806dbe0 con 0x7fedb8147930 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0003520 con 0x7fedb8147930 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4045941714 0 0) 0x7feda0003520 con 0x7fedb8147930 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 msgr2=0x7fedb8161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7fedb8161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 >> v2:172.21.15.69:3300/0 conn(0x7fedb8162000 msgr2=0x7fedb80ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 --2- 172.21.15.27:0/3086411423 >> v2:172.21.15.69:3300/0 conn(0x7fedb8162000 0x7fedb80ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb8072010 con 0x7fedb8147930 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7feda8001670 con 0x7fedb8147930 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 >> v1:172.21.15.27:6789/0 conn(0x7fedb8147930 legacy=0x7fedb8072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.386+0000 7fedb5ffb640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7feda00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb5ffb640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feda0003f30 0x7feda0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb5ffb640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feda0004d40 0x7feda00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb8071a50 con 0x7feda0003f30 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb8071d30 con 0x7feda0004d40 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda000b9c0 con 0x7fedb80b4e70 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb6ffd640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7feda00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb67fc640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feda0003f30 0x7feda0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb77fe640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feda0004d40 0x7feda00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb6ffd640 1 -- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feda0004d40 msgr2=0x7feda00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb6ffd640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feda0004d40 0x7feda00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb6ffd640 1 -- 172.21.15.27:0/3086411423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feda0003f30 msgr2=0x7feda0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb6ffd640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feda0003f30 0x7feda0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb6ffd640 1 -- 172.21.15.27:0/3086411423 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda0003520 con 0x7fedb80b4e70 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.387+0000 7fedb67fc640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feda0003f30 0x7feda0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.388+0000 7fedb6ffd640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7feda00039b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7feda8007fd0 tx=0x7feda0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52fef627f5aafb4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.388+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda8007d50 con 0x7fedb80b4e70 2024-03-06T07:34:10.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.388+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda8030070 con 0x7fedb80b4e70 2024-03-06T07:34:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.388+0000 7fedb5ffb640 1 -- 172.21.15.27:0/3086411423 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda802cd40 con 0x7fedb80b4e70 2024-03-06T07:34:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.388+0000 7fedbd78a640 1 -- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 msgr2=0x7feda00039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.389+0000 7fedbd78a640 1 --2- 172.21.15.27:0/3086411423 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7feda00039b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7feda8007fd0 tx=0x7feda0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.389+0000 7fedb77fe640 1 -- 172.21.15.27:0/3086411423 reap_dead start 2024-03-06T07:34:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.389+0000 7fedbd78a640 1 -- 172.21.15.27:0/3086411423 shutdown_connections 2024-03-06T07:34:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.389+0000 7fedbd78a640 1 -- 172.21.15.27:0/3086411423 >> 172.21.15.27:0/3086411423 conn(0x7fedb806cd20 msgr2=0x7fedb806d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.389+0000 7fedbd78a640 1 -- 172.21.15.27:0/3086411423 shutdown_connections 2024-03-06T07:34:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.389+0000 7fedbd78a640 1 -- 172.21.15.27:0/3086411423 wait complete. 2024-03-06T07:34:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.390+0000 7fedbd78a640 1 Processor -- start 2024-03-06T07:34:10.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.390+0000 7fedbd78a640 1 -- start start 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedbd78a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7fedb815dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedbd78a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb80cee90 0x7fedb80afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedbd78a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedb80b0050 0x7fedb8143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedbd78a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb806d840 con 0x7fedb80cee90 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedbd78a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb80b52b0 con 0x7fedb80b0050 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedbd78a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb806d9c0 con 0x7fedb80b4e70 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb80cee90 0x7fedb80afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb80cee90 0x7fedb80afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52264/0 (socket says 172.21.15.27:52264) 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb67fc640 1 -- 172.21.15.27:0/2461644969 learned_addr learned my addr 172.21.15.27:0/2461644969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb6ffd640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7fedb815dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb77fe640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedb80b0050 0x7fedb8143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb67fc640 1 -- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 msgr2=0x7fedb815dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb67fc640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7fedb815dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb67fc640 1 -- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedb80b0050 msgr2=0x7fedb8143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb67fc640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedb80b0050 0x7fedb8143a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb67fc640 1 -- 172.21.15.27:0/2461644969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb806eda0 con 0x7fedb80cee90 2024-03-06T07:34:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.391+0000 7fedb67fc640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb80cee90 0x7fedb80afb10 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fedac011ac0 tx=0x7fedac011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.392+0000 7fed9f7fe640 1 -- 172.21.15.27:0/2461644969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac00fb50 con 0x7fedb80cee90 2024-03-06T07:34:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.392+0000 7fed9f7fe640 1 -- 172.21.15.27:0/2461644969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac00fcf0 con 0x7fedb80cee90 2024-03-06T07:34:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.392+0000 7fed9f7fe640 1 -- 172.21.15.27:0/2461644969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac00fea0 con 0x7fedb80cee90 2024-03-06T07:34:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.392+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb81468d0 con 0x7fedb80cee90 2024-03-06T07:34:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.392+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb8145b10 con 0x7fedb80cee90 2024-03-06T07:34:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.393+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb8067ff0 con 0x7fedb80cee90 2024-03-06T07:34:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.393+0000 7fed9f7fe640 1 -- 172.21.15.27:0/2461644969 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedac012040 con 0x7fedb80cee90 2024-03-06T07:34:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.394+0000 7fed9f7fe640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda406b4d0 0x7feda406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.394+0000 7fed9f7fe640 1 -- 172.21.15.27:0/2461644969 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fedac08ac00 con 0x7fedb80cee90 2024-03-06T07:34:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.394+0000 7fedb6ffd640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda406b4d0 0x7feda406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.395+0000 7fedb6ffd640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda406b4d0 0x7feda406d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7feda8034000 tx=0x7feda8007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.398+0000 7fed9f7fe640 1 -- 172.21.15.27:0/2461644969 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedac02c080 con 0x7fedb80cee90 2024-03-06T07:34:10.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.616+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedb80b0430 con 0x7fedb80cee90 2024-03-06T07:34:10.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.617+0000 7fed9f7fe640 1 -- 172.21.15.27:0/2461644969 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fedac019030 con 0x7fedb80cee90 2024-03-06T07:34:10.619 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:10.620 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:10.620 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda406b4d0 msgr2=0x7feda406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda406b4d0 0x7feda406d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7feda8034000 tx=0x7feda8007000 comp rx=0 tx=0).stop 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb80cee90 msgr2=0x7fedb80afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb80cee90 0x7fedb80afb10 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fedac011ac0 tx=0x7fedac011f80 comp rx=0 tx=0).stop 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 shutdown_connections 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7feda406b4d0 0x7feda406d990 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedb80b0050 0x7fedb8143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedb80cee90 0x7fedb80afb10 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 --2- 172.21.15.27:0/2461644969 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fedb80b4e70 0x7fedb815dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 >> 172.21.15.27:0/2461644969 conn(0x7fedb806cd20 msgr2=0x7fedb8070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 shutdown_connections 2024-03-06T07:34:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:10.619+0000 7fedbd78a640 1 -- 172.21.15.27:0/2461644969 wait complete. 2024-03-06T07:34:10.833 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:10.834 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:11.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:11.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.940+0000 7f86aec9a640 1 Processor -- start 2024-03-06T07:34:11.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.940+0000 7f86aec9a640 1 -- start start 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86aec9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a806df30 0x7f86a80ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86aec9a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f86a80aec40 0x7f86a80b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86aec9a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a8071240 con 0x7f86a8158520 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86aec9a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a8071b70 con 0x7f86a80aec40 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86aec9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80af0b0 con 0x7f86a806df30 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86aca36640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f86a8158520 0x7f86a806d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49358/0 (socket says 172.21.15.27:49358) 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86aca36640 1 -- 172.21.15.27:0/1213745199 learned_addr learned my addr 172.21.15.27:0/1213745199 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86ad237640 1 --2- 172.21.15.27:0/1213745199 >> v2:172.21.15.69:3300/0 conn(0x7f86a80aec40 0x7f86a80b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136464885 0 0) 0x7f86a8071240 con 0x7f86a8158520 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86a7fff640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a806df30 0x7f86a80ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.941+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8694003520 con 0x7f86a8158520 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348371260 0 0) 0x7f8694003520 con 0x7f86a8158520 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a806df30 msgr2=0x7f86a80ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a806df30 0x7f86a80ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 >> v2:172.21.15.69:3300/0 conn(0x7f86a80aec40 msgr2=0x7f86a80b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 --2- 172.21.15.27:0/1213745199 >> v2:172.21.15.69:3300/0 conn(0x7f86a80aec40 0x7f86a80b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a8071520 con 0x7f86a8158520 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f8690001670 con 0x7f86a8158520 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 >> v1:172.21.15.27:6789/0 conn(0x7f86a8158520 legacy=0x7f86a806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940039b0 0x7f8694003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a806df30 0x7f8694004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8694004d40 0x7f86940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a8071b70 con 0x7f86940039b0 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80af0b0 con 0x7f8694004d40 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869400b9c0 con 0x7f86a806df30 2024-03-06T07:34:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86aca36640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940039b0 0x7f8694003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86ad237640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8694004d40 0x7f86940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86a7fff640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a806df30 0x7f8694004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86aca36640 1 -- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a806df30 msgr2=0x7f8694004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86aca36640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a806df30 0x7f8694004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86aca36640 1 -- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8694004d40 msgr2=0x7f86940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86aca36640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8694004d40 0x7f86940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86aca36640 1 -- 172.21.15.27:0/1213745199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8694003520 con 0x7f86940039b0 2024-03-06T07:34:11.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.942+0000 7f86ad237640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8694004d40 0x7f86940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.943+0000 7f86aca36640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940039b0 0x7f8694003dd0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f869000c240 tx=0x7f8694005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cd47526ed8291d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690007d60 con 0x7f86940039b0 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8690030070 con 0x7f86940039b0 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86a77fe640 1 -- 172.21.15.27:0/1213745199 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869002cdb0 con 0x7f86940039b0 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86aec9a640 1 -- 172.21.15.27:0/1213745199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940039b0 msgr2=0x7f8694003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86aec9a640 1 --2- 172.21.15.27:0/1213745199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940039b0 0x7f8694003dd0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f869000c240 tx=0x7f8694005830 comp rx=0 tx=0).stop 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86ad237640 1 -- 172.21.15.27:0/1213745199 reap_dead start 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86aec9a640 1 -- 172.21.15.27:0/1213745199 shutdown_connections 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86aec9a640 1 -- 172.21.15.27:0/1213745199 >> 172.21.15.27:0/1213745199 conn(0x7f86a8066d00 msgr2=0x7f86a80670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86aec9a640 1 -- 172.21.15.27:0/1213745199 shutdown_connections 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.944+0000 7f86aec9a640 1 -- 172.21.15.27:0/1213745199 wait complete. 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.945+0000 7f86aec9a640 1 Processor -- start 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.945+0000 7f86aec9a640 1 -- start start 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.946+0000 7f86aec9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86940039b0 0x7f86a8142990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.946+0000 7f86aec9a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a8154840 0x7f86a8154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.946+0000 7f86aec9a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a813f9d0 0x7f86a813fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.946+0000 7f86aec9a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a8158900 con 0x7f86a813f9d0 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.946+0000 7f86aec9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a8072000 con 0x7f86940039b0 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.946+0000 7f86aec9a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a8158a80 con 0x7f86a8154840 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.946+0000 7f86ad237640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a813f9d0 0x7f86a813fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.946+0000 7f86ad237640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a813f9d0 0x7f86a813fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52282/0 (socket says 172.21.15.27:52282) 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.946+0000 7f86ad237640 1 -- 172.21.15.27:0/3381417477 learned_addr learned my addr 172.21.15.27:0/3381417477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86aca36640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86940039b0 0x7f86a8142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86a7fff640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a8154840 0x7f86a8154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86ad237640 1 -- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a8154840 msgr2=0x7f86a8154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86ad237640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a8154840 0x7f86a8154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86ad237640 1 -- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86940039b0 msgr2=0x7f86a8142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86ad237640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86940039b0 0x7f86a8142990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86ad237640 1 -- 172.21.15.27:0/3381417477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a8140390 con 0x7f86a813f9d0 2024-03-06T07:34:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86ad237640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a813f9d0 0x7f86a813fe50 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f86a8067ce0 tx=0x7f86a000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86a57fa640 1 -- 172.21.15.27:0/3381417477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0009c10 con 0x7f86a813f9d0 2024-03-06T07:34:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86a57fa640 1 -- 172.21.15.27:0/3381417477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a000fd90 con 0x7f86a813f9d0 2024-03-06T07:34:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a815c4a0 con 0x7f86a813f9d0 2024-03-06T07:34:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86a57fa640 1 -- 172.21.15.27:0/3381417477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0003bd0 con 0x7f86a813f9d0 2024-03-06T07:34:11.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.947+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a815ca60 con 0x7f86a813f9d0 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.948+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a8067ff0 con 0x7f86a813f9d0 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.949+0000 7f86a57fa640 1 -- 172.21.15.27:0/3381417477 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86a001f020 con 0x7f86a813f9d0 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.949+0000 7f86a57fa640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f868806b500 0x7f868806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.949+0000 7f86a57fa640 1 -- 172.21.15.27:0/3381417477 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f86a008a430 con 0x7f86a813f9d0 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.950+0000 7f86aca36640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f868806b500 0x7f868806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.950+0000 7f86aca36640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f868806b500 0x7f868806d9c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8690034000 tx=0x7f8690007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:11.954+0000 7f86a57fa640 1 -- 172.21.15.27:0/3381417477 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86a0010040 con 0x7f86a813f9d0 2024-03-06T07:34:12.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.172+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86a80b31f0 con 0x7f86a813f9d0 2024-03-06T07:34:12.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.173+0000 7f86a57fa640 1 -- 172.21.15.27:0/3381417477 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f86a0059290 con 0x7f86a813f9d0 2024-03-06T07:34:12.175 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:12.175 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:12.176 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:12.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f868806b500 msgr2=0x7f868806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f868806b500 0x7f868806d9c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8690034000 tx=0x7f8690007000 comp rx=0 tx=0).stop 2024-03-06T07:34:12.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a813f9d0 msgr2=0x7f86a813fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a813f9d0 0x7f86a813fe50 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f86a8067ce0 tx=0x7f86a000f580 comp rx=0 tx=0).stop 2024-03-06T07:34:12.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 shutdown_connections 2024-03-06T07:34:12.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f868806b500 0x7f868806d9c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a813f9d0 0x7f86a813fe50 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86a8154840 0x7f86a8154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 --2- 172.21.15.27:0/3381417477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86940039b0 0x7f86a8142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 >> 172.21.15.27:0/3381417477 conn(0x7f86a8066d00 msgr2=0x7f86a8158d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 shutdown_connections 2024-03-06T07:34:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:12.175+0000 7f86aec9a640 1 -- 172.21.15.27:0/3381417477 wait complete. 2024-03-06T07:34:12.389 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:12.389 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:13.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.494+0000 7fdaee44e640 1 Processor -- start 2024-03-06T07:34:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.494+0000 7fdaee44e640 1 -- start start 2024-03-06T07:34:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.494+0000 7fdaee44e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae80afb30 0x7fdae80719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaee44e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdae8071ef0 0x7fdae80cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaee44e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae80b4120 con 0x7fdae815ac30 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaee44e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae8072340 con 0x7fdae8071ef0 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaee44e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80b4400 con 0x7fdae80afb30 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaec9eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdae8071ef0 0x7fdae80cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaec9eb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdae8071ef0 0x7fdae80cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52284/0 (socket says 172.21.15.27:52284) 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaec9eb640 1 -- 172.21.15.27:0/480070773 learned_addr learned my addr 172.21.15.27:0/480070773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdae77fe640 1 --2- 172.21.15.27:0/480070773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae80afb30 0x7fdae80719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaec9eb640 1 -- 172.21.15.27:0/480070773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae80afb30 msgr2=0x7fdae80719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaec9eb640 1 --2- 172.21.15.27:0/480070773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae80afb30 0x7fdae80719b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaec9eb640 1 -- 172.21.15.27:0/480070773 >> v1:172.21.15.69:6789/0 conn(0x7fdae815ac30 legacy=0x7fdae80b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdaec9eb640 1 -- 172.21.15.27:0/480070773 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae80b46e0 con 0x7fdae8071ef0 2024-03-06T07:34:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.495+0000 7fdae77fe640 1 --2- 172.21.15.27:0/480070773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae80afb30 0x7fdae80719b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.496+0000 7fdaec9eb640 1 --2- 172.21.15.27:0/480070773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae8071ef0 0x7fdae80cdf90 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fdae000d150 tx=0x7fdae002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d8feba631463c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.496+0000 7fdae6ffd640 1 -- 172.21.15.27:0/480070773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae002ea80 con 0x7fdae8071ef0 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.496+0000 7fdae6ffd640 1 -- 172.21.15.27:0/480070773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae002ec20 con 0x7fdae8071ef0 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.496+0000 7fdae6ffd640 1 -- 172.21.15.27:0/480070773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0009600 con 0x7fdae8071ef0 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.496+0000 7fdaee44e640 1 -- 172.21.15.27:0/480070773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae8071ef0 msgr2=0x7fdae80cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.496+0000 7fdaee44e640 1 --2- 172.21.15.27:0/480070773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae8071ef0 0x7fdae80cdf90 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fdae000d150 tx=0x7fdae002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.497+0000 7fdaee44e640 1 -- 172.21.15.27:0/480070773 shutdown_connections 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.497+0000 7fdaee44e640 1 --2- 172.21.15.27:0/480070773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae8071ef0 0x7fdae80cdf90 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.497+0000 7fdaee44e640 1 --2- 172.21.15.27:0/480070773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae80afb30 0x7fdae80719b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.497+0000 7fdaee44e640 1 -- 172.21.15.27:0/480070773 >> 172.21.15.27:0/480070773 conn(0x7fdae80d0990 msgr2=0x7fdae80d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.497+0000 7fdaee44e640 1 -- 172.21.15.27:0/480070773 shutdown_connections 2024-03-06T07:34:13.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.497+0000 7fdaee44e640 1 -- 172.21.15.27:0/480070773 wait complete. 2024-03-06T07:34:13.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.498+0000 7fdaee44e640 1 Processor -- start 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.498+0000 7fdaee44e640 1 -- start start 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdaee44e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdae8071ef0 0x7fdae8155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdaee44e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae80afb30 0x7fdae81e4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdaee44e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae815ac30 0x7fdae8154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdaee44e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80ce4d0 con 0x7fdae80afb30 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdaee44e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80b4ca0 con 0x7fdae8071ef0 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdaee44e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80ce650 con 0x7fdae815ac30 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae80afb30 0x7fdae81e4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae80afb30 0x7fdae81e4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52296/0 (socket says 172.21.15.27:52296) 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae77fe640 1 -- 172.21.15.27:0/614640906 learned_addr learned my addr 172.21.15.27:0/614640906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdaec9eb640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae815ac30 0x7fdae8154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae7fff640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdae8071ef0 0x7fdae8155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae77fe640 1 -- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae815ac30 msgr2=0x7fdae8154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae77fe640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae815ac30 0x7fdae8154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae77fe640 1 -- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdae8071ef0 msgr2=0x7fdae8155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae77fe640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdae8071ef0 0x7fdae8155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae77fe640 1 -- 172.21.15.27:0/614640906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae81e3ad0 con 0x7fdae80afb30 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae77fe640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae80afb30 0x7fdae81e4b70 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00c7f0 tx=0x7fdadc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae4ff9640 1 -- 172.21.15.27:0/614640906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc015070 con 0x7fdae80afb30 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae4ff9640 1 -- 172.21.15.27:0/614640906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc0079d0 con 0x7fdae80afb30 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae81e3db0 con 0x7fdae80afb30 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.499+0000 7fdae4ff9640 1 -- 172.21.15.27:0/614640906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc007da0 con 0x7fdae80afb30 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.500+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae81e4370 con 0x7fdae80afb30 2024-03-06T07:34:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.500+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae8068700 con 0x7fdae80afb30 2024-03-06T07:34:13.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.501+0000 7fdae4ff9640 1 -- 172.21.15.27:0/614640906 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdadc018070 con 0x7fdae80afb30 2024-03-06T07:34:13.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.501+0000 7fdae4ff9640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.502+0000 7fdae4ff9640 1 -- 172.21.15.27:0/614640906 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdadc08a130 con 0x7fdae80afb30 2024-03-06T07:34:13.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.502+0000 7fdae7fff640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.502+0000 7fdae7fff640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fdad8006fd0 tx=0x7fdad8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.510+0000 7fdae4ff9640 1 -- 172.21.15.27:0/614640906 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdadc058f90 con 0x7fdae80afb30 2024-03-06T07:34:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.775+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdae815a430 con 0x7fdae80afb30 2024-03-06T07:34:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.776+0000 7fdae4ff9640 1 -- 172.21.15.27:0/614640906 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fdadc058890 con 0x7fdae80afb30 2024-03-06T07:34:13.778 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:13.778 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 msgr2=0x7fdad406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fdad8006fd0 tx=0x7fdad8008040 comp rx=0 tx=0).stop 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae80afb30 msgr2=0x7fdae81e4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae80afb30 0x7fdae81e4b70 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00c7f0 tx=0x7fdadc00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 shutdown_connections 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdae815ac30 0x7fdae8154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdae80afb30 0x7fdae81e4b70 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 --2- 172.21.15.27:0/614640906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdae8071ef0 0x7fdae8155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 >> 172.21.15.27:0/614640906 conn(0x7fdae80d0990 msgr2=0x7fdae806e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 shutdown_connections 2024-03-06T07:34:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:13.779+0000 7fdaee44e640 1 -- 172.21.15.27:0/614640906 wait complete. 2024-03-06T07:34:13.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:13.994 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:14.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.101+0000 7f9eb539a640 1 Processor -- start 2024-03-06T07:34:15.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.101+0000 7f9eb539a640 1 -- start start 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eb539a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9eb015df00 0x7f9eb00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eb539a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b95a0 0x7f9eb00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eb539a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb00cea30 con 0x7f9eb00b9a30 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eb539a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb00d2af0 con 0x7f9eb015df00 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eb539a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb00cf3c0 con 0x7f9eb00b95a0 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eaeffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9eb015df00 0x7f9eb00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eaeffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9eb015df00 0x7f9eb00d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36868/0 (socket says 172.21.15.27:36868) 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eaf7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9eb00b9a30 0x7f9eb00d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54604/0 (socket says 172.21.15.27:54604) 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eaeffd640 1 -- 172.21.15.27:0/3389493321 learned_addr learned my addr 172.21.15.27:0/3389493321 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9ea7fff640 1 --2- 172.21.15.27:0/3389493321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b95a0 0x7f9eb00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eaeffd640 1 -- 172.21.15.27:0/3389493321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b95a0 msgr2=0x7f9eb00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eaeffd640 1 --2- 172.21.15.27:0/3389493321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b95a0 0x7f9eb00b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eaeffd640 1 -- 172.21.15.27:0/3389493321 >> v1:172.21.15.27:6789/0 conn(0x7f9eb00b9a30 legacy=0x7f9eb00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eaeffd640 1 -- 172.21.15.27:0/3389493321 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb00cf0e0 con 0x7f9eb015df00 2024-03-06T07:34:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.102+0000 7f9eae7fc640 1 -- 172.21.15.27:0/3389493321 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394217181 0 0) 0x7f9eb00cea30 con 0x7f9eb00b9a30 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.103+0000 7f9eaeffd640 1 --2- 172.21.15.27:0/3389493321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 0x7f9eb00d25b0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0009870 tx=0x7f9ea002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8660e0506af66c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eae7fc640 1 -- 172.21.15.27:0/3389493321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0034070 con 0x7f9eb015df00 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eae7fc640 1 -- 172.21.15.27:0/3389493321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea0030840 con 0x7f9eb015df00 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eae7fc640 1 -- 172.21.15.27:0/3389493321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0030c10 con 0x7f9eb015df00 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eb539a640 1 -- 172.21.15.27:0/3389493321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 msgr2=0x7f9eb00d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eb539a640 1 --2- 172.21.15.27:0/3389493321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 0x7f9eb00d25b0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0009870 tx=0x7f9ea002b980 comp rx=0 tx=0).stop 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eb539a640 1 -- 172.21.15.27:0/3389493321 shutdown_connections 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eb539a640 1 --2- 172.21.15.27:0/3389493321 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00b95a0 0x7f9eb00b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eb539a640 1 --2- 172.21.15.27:0/3389493321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 0x7f9eb00d25b0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eb539a640 1 -- 172.21.15.27:0/3389493321 >> 172.21.15.27:0/3389493321 conn(0x7f9eb0071960 msgr2=0x7f9eb0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eb539a640 1 -- 172.21.15.27:0/3389493321 shutdown_connections 2024-03-06T07:34:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.104+0000 7f9eb539a640 1 -- 172.21.15.27:0/3389493321 wait complete. 2024-03-06T07:34:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.105+0000 7f9eb539a640 1 Processor -- start 2024-03-06T07:34:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9eb539a640 1 -- start start 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9eb539a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00b95a0 0x7f9eb00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9eb539a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 0x7f9eb00cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9eb539a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00cd8e0 0x7f9eb0157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9eb539a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb00cec70 con 0x7f9eb015df00 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9eb539a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb00ce750 con 0x7f9eb00b95a0 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9eb539a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb00cedf0 con 0x7f9eb00cd8e0 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9ea7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 0x7f9eb00cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9ea7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 0x7f9eb00cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36870/0 (socket says 172.21.15.27:36870) 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.106+0000 7f9ea7fff640 1 -- 172.21.15.27:0/2965476364 learned_addr learned my addr 172.21.15.27:0/2965476364 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9eaeffd640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00b95a0 0x7f9eb00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9eaf7fe640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00cd8e0 0x7f9eb0157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9ea7fff640 1 -- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00cd8e0 msgr2=0x7f9eb0157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9ea7fff640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00cd8e0 0x7f9eb0157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9ea7fff640 1 -- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00b95a0 msgr2=0x7f9eb00b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9ea7fff640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00b95a0 0x7f9eb00b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9ea7fff640 1 -- 172.21.15.27:0/2965476364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb015e2e0 con 0x7f9eb015df00 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9eaf7fe640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00cd8e0 0x7f9eb0157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9eaeffd640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00b95a0 0x7f9eb00b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9ea7fff640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 0x7f9eb00cc450 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800d830 tx=0x7f9e9800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9ea77fe640 1 -- 172.21.15.27:0/2965476364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9800b6c0 con 0x7f9eb015df00 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9ea77fe640 1 -- 172.21.15.27:0/2965476364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e98019070 con 0x7f9eb015df00 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb015e5c0 con 0x7f9eb015df00 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9ea77fe640 1 -- 172.21.15.27:0/2965476364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9800ba80 con 0x7f9eb015df00 2024-03-06T07:34:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.107+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb015eb80 con 0x7f9eb015df00 2024-03-06T07:34:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.108+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb0067ff0 con 0x7f9eb015df00 2024-03-06T07:34:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.109+0000 7f9ea77fe640 1 -- 172.21.15.27:0/2965476364 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e9801e460 con 0x7f9eb015df00 2024-03-06T07:34:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.109+0000 7f9ea77fe640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e9006b4d0 0x7f9e9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.109+0000 7f9ea77fe640 1 -- 172.21.15.27:0/2965476364 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9e9808a950 con 0x7f9eb015df00 2024-03-06T07:34:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.110+0000 7f9eaeffd640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e9006b4d0 0x7f9e9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.110+0000 7f9eaeffd640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e9006b4d0 0x7f9e9006d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0036000 tx=0x7f9ea003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.114+0000 7f9ea77fe640 1 -- 172.21.15.27:0/2965476364 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e980597b0 con 0x7f9eb015df00 2024-03-06T07:34:15.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.333+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9eb00b8190 con 0x7f9eb015df00 2024-03-06T07:34:15.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.333+0000 7f9ea77fe640 1 -- 172.21.15.27:0/2965476364 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f9e980590b0 con 0x7f9eb015df00 2024-03-06T07:34:15.336 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:15.336 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:15.336 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e9006b4d0 msgr2=0x7f9e9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e9006b4d0 0x7f9e9006d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0036000 tx=0x7f9ea003b040 comp rx=0 tx=0).stop 2024-03-06T07:34:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 msgr2=0x7f9eb00cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 0x7f9eb00cc450 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800d830 tx=0x7f9e9800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 shutdown_connections 2024-03-06T07:34:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9e9006b4d0 0x7f9e9006d990 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9eb00cd8e0 0x7f9eb0157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eb015df00 0x7f9eb00cc450 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 --2- 172.21.15.27:0/2965476364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9eb00b95a0 0x7f9eb00b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 >> 172.21.15.27:0/2965476364 conn(0x7f9eb0071960 msgr2=0x7f9eb015d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 shutdown_connections 2024-03-06T07:34:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:15.335+0000 7f9eb539a640 1 -- 172.21.15.27:0/2965476364 wait complete. 2024-03-06T07:34:15.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:15.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:16.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:16.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.656+0000 7fa3ecfd0640 1 Processor -- start 2024-03-06T07:34:16.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.656+0000 7fa3ecfd0640 1 -- start start 2024-03-06T07:34:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.657+0000 7fa3ecfd0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e815df00 0x7fa3e80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.657+0000 7fa3ecfd0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3e80b95a0 0x7fa3e80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.657+0000 7fa3ecfd0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80cea30 con 0x7fa3e80b9a30 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.657+0000 7fa3ecfd0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e80d2af0 con 0x7fa3e80b95a0 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.657+0000 7fa3ecfd0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80cf3c0 con 0x7fa3e815df00 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e659c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e815df00 0x7fa3e80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e5d9b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3e80b95a0 0x7fa3e80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e6d9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa3e80b9a30 0x7fa3e80d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:41384/0 (socket says 172.21.15.27:41384) 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e6d9d640 1 -- 172.21.15.27:0/860929673 learned_addr learned my addr 172.21.15.27:0/860929673 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e559a640 1 -- 172.21.15.27:0/860929673 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804284324 0 0) 0x7fa3e80cea30 con 0x7fa3e80b9a30 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e559a640 1 -- 172.21.15.27:0/860929673 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d0003520 con 0x7fa3e80b9a30 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e659c640 1 -- 172.21.15.27:0/860929673 >> v2:172.21.15.69:3300/0 conn(0x7fa3e80b95a0 msgr2=0x7fa3e80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e659c640 1 --2- 172.21.15.27:0/860929673 >> v2:172.21.15.69:3300/0 conn(0x7fa3e80b95a0 0x7fa3e80b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e659c640 1 -- 172.21.15.27:0/860929673 >> v1:172.21.15.69:6789/0 conn(0x7fa3e80b9a30 legacy=0x7fa3e80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e659c640 1 -- 172.21.15.27:0/860929673 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e80cf0e0 con 0x7fa3e815df00 2024-03-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.658+0000 7fa3e5d9b640 1 --2- 172.21.15.27:0/860929673 >> v2:172.21.15.69:3300/0 conn(0x7fa3e80b95a0 0x7fa3e80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:16.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.659+0000 7fa3e659c640 1 --2- 172.21.15.27:0/860929673 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e815df00 0x7fa3e80d25b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa3d80097a0 tx=0x7fa3d802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=749faf106cbfcc24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.659+0000 7fa3e559a640 1 -- 172.21.15.27:0/860929673 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8035020 con 0x7fa3e815df00 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.659+0000 7fa3e559a640 1 -- 172.21.15.27:0/860929673 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d8031830 con 0x7fa3e815df00 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.659+0000 7fa3e559a640 1 -- 172.21.15.27:0/860929673 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8031be0 con 0x7fa3e815df00 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.660+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/860929673 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e815df00 msgr2=0x7fa3e80d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.660+0000 7fa3ecfd0640 1 --2- 172.21.15.27:0/860929673 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e815df00 0x7fa3e80d25b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa3d80097a0 tx=0x7fa3d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.660+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/860929673 shutdown_connections 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.660+0000 7fa3ecfd0640 1 --2- 172.21.15.27:0/860929673 >> v2:172.21.15.69:3300/0 conn(0x7fa3e80b95a0 0x7fa3e80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.660+0000 7fa3ecfd0640 1 --2- 172.21.15.27:0/860929673 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e815df00 0x7fa3e80d25b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.660+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/860929673 >> 172.21.15.27:0/860929673 conn(0x7fa3e8071960 msgr2=0x7fa3e8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.660+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/860929673 shutdown_connections 2024-03-06T07:34:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.660+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/860929673 wait complete. 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.661+0000 7fa3ecfd0640 1 Processor -- start 2024-03-06T07:34:16.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.661+0000 7fa3ecfd0640 1 -- start start 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3ecfd0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e80b95a0 0x7fa3e8151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3ecfd0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e815df00 0x7fa3e80b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3ecfd0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3e8152060 0x7fa3e815a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3ecfd0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80cec70 con 0x7fa3e815df00 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3ecfd0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80ce750 con 0x7fa3e8152060 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3ecfd0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80cedf0 con 0x7fa3e80b95a0 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e5d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e815df00 0x7fa3e80b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e5d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e815df00 0x7fa3e80b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36878/0 (socket says 172.21.15.27:36878) 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e5d9b640 1 -- 172.21.15.27:0/3501571932 learned_addr learned my addr 172.21.15.27:0/3501571932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e659c640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e80b95a0 0x7fa3e8151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e6d9d640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3e8152060 0x7fa3e815a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e5d9b640 1 -- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e80b95a0 msgr2=0x7fa3e8151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e5d9b640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e80b95a0 0x7fa3e8151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e5d9b640 1 -- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3e8152060 msgr2=0x7fa3e815a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e5d9b640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3e8152060 0x7fa3e815a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.662+0000 7fa3e5d9b640 1 -- 172.21.15.27:0/3501571932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e80b3e10 con 0x7fa3e815df00 2024-03-06T07:34:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.663+0000 7fa3e5d9b640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e815df00 0x7fa3e80b38d0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc00e960 tx=0x7fa3dc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.663+0000 7fa3d6ffd640 1 -- 172.21.15.27:0/3501571932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc017070 con 0x7fa3e815df00 2024-03-06T07:34:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.663+0000 7fa3d6ffd640 1 -- 172.21.15.27:0/3501571932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3dc00ca00 con 0x7fa3e815df00 2024-03-06T07:34:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.663+0000 7fa3d6ffd640 1 -- 172.21.15.27:0/3501571932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc00cdd0 con 0x7fa3e815df00 2024-03-06T07:34:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.663+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e8160210 con 0x7fa3e815df00 2024-03-06T07:34:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.663+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e80b2df0 con 0x7fa3e815df00 2024-03-06T07:34:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.664+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e8067ff0 con 0x7fa3e815df00 2024-03-06T07:34:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.665+0000 7fa3d6ffd640 1 -- 172.21.15.27:0/3501571932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3dc01a070 con 0x7fa3e815df00 2024-03-06T07:34:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.665+0000 7fa3d6ffd640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3c806b4d0 0x7fa3c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.666+0000 7fa3d6ffd640 1 -- 172.21.15.27:0/3501571932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa3dc08a050 con 0x7fa3e815df00 2024-03-06T07:34:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.666+0000 7fa3e659c640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3c806b4d0 0x7fa3c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.666+0000 7fa3e659c640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3c806b4d0 0x7fa3c806d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8036000 tx=0x7fa3d8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.670+0000 7fa3d6ffd640 1 -- 172.21.15.27:0/3501571932 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3dc058860 con 0x7fa3e815df00 2024-03-06T07:34:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.892+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3e80d3170 con 0x7fa3e815df00 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.893+0000 7fa3d6ffd640 1 -- 172.21.15.27:0/3501571932 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fa3dc05c900 con 0x7fa3e815df00 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:16.896 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3c806b4d0 msgr2=0x7fa3c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3c806b4d0 0x7fa3c806d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8036000 tx=0x7fa3d8008030 comp rx=0 tx=0).stop 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e815df00 msgr2=0x7fa3e80b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e815df00 0x7fa3e80b38d0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc00e960 tx=0x7fa3dc00ee20 comp rx=0 tx=0).stop 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 shutdown_connections 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa3c806b4d0 0x7fa3c806d990 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3e8152060 0x7fa3e815a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e815df00 0x7fa3e80b38d0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 --2- 172.21.15.27:0/3501571932 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa3e80b95a0 0x7fa3e8151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.894+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 >> 172.21.15.27:0/3501571932 conn(0x7fa3e8071960 msgr2=0x7fa3e80b6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.895+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 shutdown_connections 2024-03-06T07:34:16.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:16.895+0000 7fa3ecfd0640 1 -- 172.21.15.27:0/3501571932 wait complete. 2024-03-06T07:34:17.109 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:17.109 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:18.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.215+0000 7f57fffe9640 1 Processor -- start 2024-03-06T07:34:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.215+0000 7f57fffe9640 1 -- start start 2024-03-06T07:34:18.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fffe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f8071540 0x7f57f80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fffe9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f80ce610 con 0x7f57f80d3400 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fffe9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f80d2920 con 0x7f57f80b2b50 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fffe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f80d2f70 con 0x7f57f8071540 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fd584640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f57f80d3400 0x7f57f8071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54616/0 (socket says 172.21.15.27:54616) 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fd584640 1 -- 172.21.15.27:0/3747283825 learned_addr learned my addr 172.21.15.27:0/3747283825 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fdd85640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f8071540 0x7f57f80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717770662 0 0) 0x7f57f80ce610 con 0x7f57f80d3400 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e4003520 con 0x7f57f80d3400 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1035199260 0 0) 0x7f57e4003520 con 0x7f57f80d3400 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f8071540 msgr2=0x7f57f80b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f8071540 0x7f57f80b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 >> v1:172.21.15.69:6789/0 conn(0x7f57f80b2b50 legacy=0x7f57f80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f80d2c00 con 0x7f57f80d3400 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f57e8001670 con 0x7f57f80d3400 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 >> v1:172.21.15.27:6789/0 conn(0x7f57f80d3400 legacy=0x7f57f8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e40039e0 0x7f57e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e4004390 0x7f57e4004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f8071540 0x7f57e40050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f80d2f70 con 0x7f57e40039e0 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e400b9c0 con 0x7f57e4004390 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.216+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e400bbc0 con 0x7f57f8071540 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.217+0000 7f57fdd85640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e40039e0 0x7f57e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.217+0000 7f57fd584640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e4004390 0x7f57e4004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.217+0000 7f57fe586640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f8071540 0x7f57e40050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.217+0000 7f57fdd85640 1 -- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f8071540 msgr2=0x7f57e40050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.217+0000 7f57fdd85640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f8071540 0x7f57e40050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.217+0000 7f57fdd85640 1 -- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e4004390 msgr2=0x7f57e4004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.217+0000 7f57fdd85640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e4004390 0x7f57e4004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.217+0000 7f57fdd85640 1 -- 172.21.15.27:0/3747283825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e4003520 con 0x7f57e40039e0 2024-03-06T07:34:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.218+0000 7f57fdd85640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e40039e0 0x7f57e4003dc0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009ac0 tx=0x7f57e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c7266d6a3c81b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.218+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4039070 con 0x7f57e40039e0 2024-03-06T07:34:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.218+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f402c6b0 con 0x7f57e40039e0 2024-03-06T07:34:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.218+0000 7f57fcd83640 1 -- 172.21.15.27:0/3747283825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4009690 con 0x7f57e40039e0 2024-03-06T07:34:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.219+0000 7f57fffe9640 1 -- 172.21.15.27:0/3747283825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e40039e0 msgr2=0x7f57e4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.219+0000 7f57fffe9640 1 --2- 172.21.15.27:0/3747283825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e40039e0 0x7f57e4003dc0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009ac0 tx=0x7f57e4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.219+0000 7f57fe586640 1 -- 172.21.15.27:0/3747283825 reap_dead start 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.219+0000 7f57fffe9640 1 -- 172.21.15.27:0/3747283825 shutdown_connections 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.219+0000 7f57fffe9640 1 -- 172.21.15.27:0/3747283825 >> 172.21.15.27:0/3747283825 conn(0x7f57f80d3ca0 msgr2=0x7f57f80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.219+0000 7f57fffe9640 1 -- 172.21.15.27:0/3747283825 shutdown_connections 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.219+0000 7f57fffe9640 1 -- 172.21.15.27:0/3747283825 wait complete. 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.220+0000 7f57fffe9640 1 Processor -- start 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.220+0000 7f57fffe9640 1 -- start start 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fffe9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e40039e0 0x7f57f80ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fffe9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57f80ab0b0 0x7f57f80acc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fffe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f80ad1d0 0x7f57f80b8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fffe9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f815e370 con 0x7f57f80ab0b0 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fffe9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f80d4be0 con 0x7f57e40039e0 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fffe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f80ac800 con 0x7f57f80ad1d0 2024-03-06T07:34:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fd584640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57f80ab0b0 0x7f57f80acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fd584640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57f80ab0b0 0x7f57f80acc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36888/0 (socket says 172.21.15.27:36888) 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fd584640 1 -- 172.21.15.27:0/3556499753 learned_addr learned my addr 172.21.15.27:0/3556499753 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fe586640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f80ad1d0 0x7f57f80b8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fdd85640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e40039e0 0x7f57f80ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fd584640 1 -- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f80ad1d0 msgr2=0x7f57f80b8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fd584640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f80ad1d0 0x7f57f80b8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fd584640 1 -- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e40039e0 msgr2=0x7f57f80ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fd584640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e40039e0 0x7f57f80ca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fd584640 1 -- 172.21.15.27:0/3556499753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f80abbf0 con 0x7f57f80ab0b0 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fe586640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f80ad1d0 0x7f57f80b8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fdd85640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e40039e0 0x7f57f80ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.221+0000 7f57fd584640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57f80ab0b0 0x7f57f80acc90 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f57e80072c0 tx=0x7f57e8007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.222+0000 7f57ee7fc640 1 -- 172.21.15.27:0/3556499753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e8001670 con 0x7f57f80ab0b0 2024-03-06T07:34:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.222+0000 7f57ee7fc640 1 -- 172.21.15.27:0/3556499753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e8018070 con 0x7f57f80ab0b0 2024-03-06T07:34:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.222+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f80abed0 con 0x7f57f80ab0b0 2024-03-06T07:34:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.222+0000 7f57ee7fc640 1 -- 172.21.15.27:0/3556499753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e800fc00 con 0x7f57f80ab0b0 2024-03-06T07:34:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.222+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f80ac490 con 0x7f57f80ab0b0 2024-03-06T07:34:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.223+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f8068700 con 0x7f57f80ab0b0 2024-03-06T07:34:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.224+0000 7f57ee7fc640 1 -- 172.21.15.27:0/3556499753 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57e801d460 con 0x7f57f80ab0b0 2024-03-06T07:34:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.224+0000 7f57ee7fc640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57dc06b4d0 0x7f57dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.224+0000 7f57ee7fc640 1 -- 172.21.15.27:0/3556499753 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f57e808a2b0 con 0x7f57f80ab0b0 2024-03-06T07:34:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.224+0000 7f57fdd85640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57dc06b4d0 0x7f57dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.225+0000 7f57fdd85640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57dc06b4d0 0x7f57dc06d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009ac0 tx=0x7f57f4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.229+0000 7f57ee7fc640 1 -- 172.21.15.27:0/3556499753 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57e8059110 con 0x7f57f80ab0b0 2024-03-06T07:34:18.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.453+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57f804f3b0 con 0x7f57f80ab0b0 2024-03-06T07:34:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.453+0000 7f57ee7fc640 1 -- 172.21.15.27:0/3556499753 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f57e808a0d0 con 0x7f57f80ab0b0 2024-03-06T07:34:18.527 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:18.527 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:18.529 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:18.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.455+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57dc06b4d0 msgr2=0x7f57dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57dc06b4d0 0x7f57dc06d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009ac0 tx=0x7f57f4030000 comp rx=0 tx=0).stop 2024-03-06T07:34:18.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57f80ab0b0 msgr2=0x7f57f80acc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57f80ab0b0 0x7f57f80acc90 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f57e80072c0 tx=0x7f57e8007780 comp rx=0 tx=0).stop 2024-03-06T07:34:18.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 shutdown_connections 2024-03-06T07:34:18.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f57dc06b4d0 0x7f57dc06d990 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57f80ad1d0 0x7f57f80b8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57f80ab0b0 0x7f57f80acc90 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 --2- 172.21.15.27:0/3556499753 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57e40039e0 0x7f57f80ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 >> 172.21.15.27:0/3556499753 conn(0x7f57f80d3ca0 msgr2=0x7f57f815f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 shutdown_connections 2024-03-06T07:34:18.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:18.456+0000 7f57fffe9640 1 -- 172.21.15.27:0/3556499753 wait complete. 2024-03-06T07:34:18.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:18.733 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:19.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.861+0000 7f90a2a88640 1 Processor -- start 2024-03-06T07:34:19.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.861+0000 7f90a2a88640 1 -- start start 2024-03-06T07:34:19.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f90a2a88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c072660 0x7f909c15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f90a2a88640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f909c0b55d0 0x7f909c071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f90a2a88640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c0b4160 con 0x7f909c06e700 2024-03-06T07:34:19.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f90a2a88640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c0b5b40 con 0x7f909c0b55d0 2024-03-06T07:34:19.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f90a2a88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c0b3ba0 con 0x7f909c072660 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f90a1025640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f909c0b55d0 0x7f909c071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f90a0824640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f909c06e700 0x7f909c06d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54624/0 (socket says 172.21.15.27:54624) 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f90a0824640 1 -- 172.21.15.27:0/3802349662 learned_addr learned my addr 172.21.15.27:0/3802349662 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f909bfff640 1 --2- 172.21.15.27:0/3802349662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c072660 0x7f909c15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f909b7fe640 1 -- 172.21.15.27:0/3802349662 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685894932 0 0) 0x7f909c0b4160 con 0x7f909c06e700 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.862+0000 7f909b7fe640 1 -- 172.21.15.27:0/3802349662 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9088003520 con 0x7f909c06e700 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.863+0000 7f90a1025640 1 -- 172.21.15.27:0/3802349662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c072660 msgr2=0x7f909c15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.863+0000 7f90a1025640 1 --2- 172.21.15.27:0/3802349662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c072660 0x7f909c15ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.863+0000 7f90a1025640 1 -- 172.21.15.27:0/3802349662 >> v1:172.21.15.27:6789/0 conn(0x7f909c06e700 legacy=0x7f909c06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.863+0000 7f90a1025640 1 -- 172.21.15.27:0/3802349662 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c0b3e80 con 0x7f909c0b55d0 2024-03-06T07:34:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.863+0000 7f909bfff640 1 --2- 172.21.15.27:0/3802349662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c072660 0x7f909c15ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.864+0000 7f90a1025640 1 --2- 172.21.15.27:0/3802349662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c0b55d0 0x7f909c071980 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f90840097a0 tx=0x7f908402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae6457dbc2104ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.864+0000 7f909b7fe640 1 -- 172.21.15.27:0/3802349662 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084035020 con 0x7f909c0b55d0 2024-03-06T07:34:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.864+0000 7f909b7fe640 1 -- 172.21.15.27:0/3802349662 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084031830 con 0x7f909c0b55d0 2024-03-06T07:34:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.864+0000 7f909b7fe640 1 -- 172.21.15.27:0/3802349662 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084031c00 con 0x7f909c0b55d0 2024-03-06T07:34:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.865+0000 7f90a2a88640 1 -- 172.21.15.27:0/3802349662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c0b55d0 msgr2=0x7f909c071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.865+0000 7f90a2a88640 1 --2- 172.21.15.27:0/3802349662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c0b55d0 0x7f909c071980 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f90840097a0 tx=0x7f908402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.865+0000 7f90a2a88640 1 -- 172.21.15.27:0/3802349662 shutdown_connections 2024-03-06T07:34:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.865+0000 7f90a2a88640 1 --2- 172.21.15.27:0/3802349662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c0b55d0 0x7f909c071980 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.865+0000 7f90a2a88640 1 --2- 172.21.15.27:0/3802349662 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c072660 0x7f909c15ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.865+0000 7f90a2a88640 1 -- 172.21.15.27:0/3802349662 >> 172.21.15.27:0/3802349662 conn(0x7f909c0b2720 msgr2=0x7f909c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.865+0000 7f90a2a88640 1 -- 172.21.15.27:0/3802349662 shutdown_connections 2024-03-06T07:34:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.865+0000 7f90a2a88640 1 -- 172.21.15.27:0/3802349662 wait complete. 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.866+0000 7f90a2a88640 1 Processor -- start 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.866+0000 7f90a2a88640 1 -- start start 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a2a88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c06e700 0x7f909c153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a2a88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f909c072660 0x7f909c0b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a2a88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c0b55d0 0x7f909c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a2a88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c072040 con 0x7f909c06e700 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a2a88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c071ec0 con 0x7f909c072660 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a2a88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c0721c0 con 0x7f909c0b55d0 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a0824640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c06e700 0x7f909c153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a0824640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c06e700 0x7f909c153bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36910/0 (socket says 172.21.15.27:36910) 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a0824640 1 -- 172.21.15.27:0/353150332 learned_addr learned my addr 172.21.15.27:0/353150332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a1025640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c0b55d0 0x7f909c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f909bfff640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f909c072660 0x7f909c0b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a0824640 1 -- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c0b55d0 msgr2=0x7f909c0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a0824640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c0b55d0 0x7f909c0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a0824640 1 -- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f909c072660 msgr2=0x7f909c0b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a0824640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f909c072660 0x7f909c0b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a0824640 1 -- 172.21.15.27:0/353150332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c0cfd80 con 0x7f909c06e700 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a1025640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c0b55d0 0x7f909c0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f909bfff640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f909c072660 0x7f909c0b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.867+0000 7f90a0824640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c06e700 0x7f909c153bb0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f909400cb70 tx=0x7f909400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.868+0000 7f90997fa640 1 -- 172.21.15.27:0/353150332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094017070 con 0x7f909c06e700 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.868+0000 7f90997fa640 1 -- 172.21.15.27:0/353150332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909400ad40 con 0x7f909c06e700 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.868+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c0d0060 con 0x7f909c06e700 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.868+0000 7f90997fa640 1 -- 172.21.15.27:0/353150332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094012620 con 0x7f909c06e700 2024-03-06T07:34:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.868+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c159e80 con 0x7f909c06e700 2024-03-06T07:34:19.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.869+0000 7f90997fa640 1 -- 172.21.15.27:0/353150332 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f909401a070 con 0x7f909c06e700 2024-03-06T07:34:19.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.869+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909c06fe60 con 0x7f909c06e700 2024-03-06T07:34:19.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.869+0000 7f90997fa640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907c06b4d0 0x7f907c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.870+0000 7f90997fa640 1 -- 172.21.15.27:0/353150332 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f909408a010 con 0x7f909c06e700 2024-03-06T07:34:19.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.870+0000 7f909bfff640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907c06b4d0 0x7f907c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.870+0000 7f909bfff640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907c06b4d0 0x7f907c06d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f909c15a340 tx=0x7f908c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:19.874+0000 7f90997fa640 1 -- 172.21.15.27:0/353150332 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f909c06fe60 con 0x7f909c06e700 2024-03-06T07:34:20.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.095+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f909c067ff0 con 0x7f909c06e700 2024-03-06T07:34:20.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.095+0000 7f90997fa640 1 -- 172.21.15.27:0/353150332 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f9094058f00 con 0x7f909c06e700 2024-03-06T07:34:20.098 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:20.098 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:33:13.569395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:20.098 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-06T07:34:20.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907c06b4d0 msgr2=0x7f907c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907c06b4d0 0x7f907c06d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f909c15a340 tx=0x7f908c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:20.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c06e700 msgr2=0x7f909c153bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c06e700 0x7f909c153bb0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f909400cb70 tx=0x7f909400a430 comp rx=0 tx=0).stop 2024-03-06T07:34:20.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 shutdown_connections 2024-03-06T07:34:20.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f907c06b4d0 0x7f907c06d990 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f909c0b55d0 0x7f909c0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f909c072660 0x7f909c0b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 --2- 172.21.15.27:0/353150332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c06e700 0x7f909c153bb0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 >> 172.21.15.27:0/353150332 conn(0x7f909c0b2720 msgr2=0x7f909c15e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 shutdown_connections 2024-03-06T07:34:20.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:20.098+0000 7f90a2a88640 1 -- 172.21.15.27:0/353150332 wait complete. 2024-03-06T07:34:20.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:20.117 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:20.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:33:13.569395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:20.312 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:21.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.417+0000 7fc049239640 1 Processor -- start 2024-03-06T07:34:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.418+0000 7fc049239640 1 -- start start 2024-03-06T07:34:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.418+0000 7fc049239640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0440b95a0 0x7fc0440b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.418+0000 7fc049239640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0440b9a10 0x7fc0440b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.418+0000 7fc049239640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0440cea30 con 0x7fc04415df00 2024-03-06T07:34:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.418+0000 7fc049239640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0440cf3c0 con 0x7fc0440b95a0 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.418+0000 7fc049239640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0440cf0e0 con 0x7fc0440b9a10 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.418+0000 7fc04259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0440b95a0 0x7fc0440b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.419+0000 7fc04259c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0440b95a0 0x7fc0440b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36924/0 (socket says 172.21.15.27:36924) 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.419+0000 7fc04259c640 1 -- 172.21.15.27:0/235616932 learned_addr learned my addr 172.21.15.27:0/235616932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.419+0000 7fc042d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc04415df00 0x7fc0440d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54632/0 (socket says 172.21.15.27:54632) 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.419+0000 7fc04359e640 1 --2- 172.21.15.27:0/235616932 >> v2:172.21.15.69:3300/0 conn(0x7fc0440b9a10 0x7fc0440b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.419+0000 7fc04259c640 1 -- 172.21.15.27:0/235616932 >> v2:172.21.15.69:3300/0 conn(0x7fc0440b9a10 msgr2=0x7fc0440b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.419+0000 7fc04259c640 1 --2- 172.21.15.27:0/235616932 >> v2:172.21.15.69:3300/0 conn(0x7fc0440b9a10 0x7fc0440b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.419+0000 7fc04259c640 1 -- 172.21.15.27:0/235616932 >> v1:172.21.15.27:6789/0 conn(0x7fc04415df00 legacy=0x7fc0440d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.419+0000 7fc04259c640 1 -- 172.21.15.27:0/235616932 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440ce470 con 0x7fc0440b95a0 2024-03-06T07:34:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc04259c640 1 --2- 172.21.15.27:0/235616932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 0x7fc0440b5700 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fc0440d4d20 tx=0x7fc03802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477bd77a09110730 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc041d9b640 1 -- 172.21.15.27:0/235616932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038037070 con 0x7fc0440b95a0 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc041d9b640 1 -- 172.21.15.27:0/235616932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc038004230 con 0x7fc0440b95a0 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc041d9b640 1 -- 172.21.15.27:0/235616932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038003610 con 0x7fc0440b95a0 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc049239640 1 -- 172.21.15.27:0/235616932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 msgr2=0x7fc0440b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc049239640 1 --2- 172.21.15.27:0/235616932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 0x7fc0440b5700 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fc0440d4d20 tx=0x7fc03802b7e0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc049239640 1 -- 172.21.15.27:0/235616932 shutdown_connections 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc049239640 1 --2- 172.21.15.27:0/235616932 >> v2:172.21.15.69:3300/0 conn(0x7fc0440b9a10 0x7fc0440b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc049239640 1 --2- 172.21.15.27:0/235616932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 0x7fc0440b5700 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc049239640 1 -- 172.21.15.27:0/235616932 >> 172.21.15.27:0/235616932 conn(0x7fc044071960 msgr2=0x7fc044071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc049239640 1 -- 172.21.15.27:0/235616932 shutdown_connections 2024-03-06T07:34:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.420+0000 7fc049239640 1 -- 172.21.15.27:0/235616932 wait complete. 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.422+0000 7fc049239640 1 Processor -- start 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.422+0000 7fc049239640 1 -- start start 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc049239640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 0x7fc04415af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc049239640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0440b9a10 0x7fc0441e8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc049239640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc04415df00 0x7fc0440b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc049239640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440cedf0 con 0x7fc0440b95a0 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc049239640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440cec70 con 0x7fc0440b9a10 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc049239640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440d3220 con 0x7fc04415df00 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc042d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 0x7fc04415af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc042d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 0x7fc04415af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36936/0 (socket says 172.21.15.27:36936) 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc042d9d640 1 -- 172.21.15.27:0/515376161 learned_addr learned my addr 172.21.15.27:0/515376161 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc04359e640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc04415df00 0x7fc0440b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc04259c640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0440b9a10 0x7fc0441e8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc042d9d640 1 -- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc04415df00 msgr2=0x7fc0440b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc042d9d640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc04415df00 0x7fc0440b56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc042d9d640 1 -- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0440b9a10 msgr2=0x7fc0441e8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc042d9d640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0440b9a10 0x7fc0441e8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc042d9d640 1 -- 172.21.15.27:0/515376161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440c9830 con 0x7fc0440b95a0 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc04359e640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc04415df00 0x7fc0440b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc04259c640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0440b9a10 0x7fc0441e8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.423+0000 7fc042d9d640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 0x7fc04415af60 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc03400c780 tx=0x7fc03400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.424+0000 7fc02b7fe640 1 -- 172.21.15.27:0/515376161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034018070 con 0x7fc0440b95a0 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.424+0000 7fc02b7fe640 1 -- 172.21.15.27:0/515376161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0340072d0 con 0x7fc0440b95a0 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.424+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0440c9b10 con 0x7fc0440b95a0 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.424+0000 7fc02b7fe640 1 -- 172.21.15.27:0/515376161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0340047e0 con 0x7fc0440b95a0 2024-03-06T07:34:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.424+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0440ca0d0 con 0x7fc0440b95a0 2024-03-06T07:34:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.425+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc044067ff0 con 0x7fc0440b95a0 2024-03-06T07:34:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.426+0000 7fc02b7fe640 1 -- 172.21.15.27:0/515376161 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc03401b070 con 0x7fc0440b95a0 2024-03-06T07:34:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.426+0000 7fc02b7fe640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc03006b4d0 0x7fc03006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.426+0000 7fc02b7fe640 1 -- 172.21.15.27:0/515376161 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc034089650 con 0x7fc0440b95a0 2024-03-06T07:34:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.426+0000 7fc04259c640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc03006b4d0 0x7fc03006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.427+0000 7fc04259c640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc03006b4d0 0x7fc03006d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc038034000 tx=0x7fc038008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.431+0000 7fc02b7fe640 1 -- 172.21.15.27:0/515376161 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc034058570 con 0x7fc0440b95a0 2024-03-06T07:34:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.648+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0440c7530 con 0x7fc0440b95a0 2024-03-06T07:34:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.648+0000 7fc02b7fe640 1 -- 172.21.15.27:0/515376161 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fc034006e20 con 0x7fc0440b95a0 2024-03-06T07:34:21.651 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:21.651 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:21.651 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc03006b4d0 msgr2=0x7fc03006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc03006b4d0 0x7fc03006d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc038034000 tx=0x7fc038008030 comp rx=0 tx=0).stop 2024-03-06T07:34:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 msgr2=0x7fc04415af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 0x7fc04415af60 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc03400c780 tx=0x7fc03400cc40 comp rx=0 tx=0).stop 2024-03-06T07:34:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 shutdown_connections 2024-03-06T07:34:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc03006b4d0 0x7fc03006d990 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc04415df00 0x7fc0440b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0440b9a10 0x7fc0441e8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 --2- 172.21.15.27:0/515376161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0440b95a0 0x7fc04415af60 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 >> 172.21.15.27:0/515376161 conn(0x7fc044071960 msgr2=0x7fc04415d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 shutdown_connections 2024-03-06T07:34:21.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:21.650+0000 7fc049239640 1 -- 172.21.15.27:0/515376161 wait complete. 2024-03-06T07:34:21.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:21.865 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:22.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:22.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.976+0000 7f1a5fa65640 1 Processor -- start 2024-03-06T07:34:22.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.976+0000 7f1a5fa65640 1 -- start start 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5fa65640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a580b95a0 0x7f1a580b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5fa65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b9a10 0x7f1a580b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5fa65640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a580cea30 con 0x7f1a5815df00 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5fa65640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a580cf3c0 con 0x7f1a580b95a0 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5fa65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580cf0e0 con 0x7f1a580b9a10 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a580b95a0 0x7f1a580b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a580b95a0 0x7f1a580b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36940/0 (socket says 172.21.15.27:36940) 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5d000640 1 -- 172.21.15.27:0/400998742 learned_addr learned my addr 172.21.15.27:0/400998742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5d000640 1 -- 172.21.15.27:0/400998742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b9a10 msgr2=0x7f1a580b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5e002640 1 --2- 172.21.15.27:0/400998742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b9a10 0x7f1a580b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5d000640 1 --2- 172.21.15.27:0/400998742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b9a10 0x7f1a580b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5d000640 1 -- 172.21.15.27:0/400998742 >> v1:172.21.15.69:6789/0 conn(0x7f1a5815df00 legacy=0x7f1a580d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.977+0000 7f1a5d000640 1 -- 172.21.15.27:0/400998742 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a580ce470 con 0x7f1a580b95a0 2024-03-06T07:34:22.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.979+0000 7f1a5d000640 1 --2- 172.21.15.27:0/400998742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a580b95a0 0x7f1a580b5700 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009870 tx=0x7f1a4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73ae8da9c32ee64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.979+0000 7f1a4ffff640 1 -- 172.21.15.27:0/400998742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48035020 con 0x7f1a580b95a0 2024-03-06T07:34:22.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.979+0000 7f1a4ffff640 1 -- 172.21.15.27:0/400998742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a48031640 con 0x7f1a580b95a0 2024-03-06T07:34:22.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.979+0000 7f1a4ffff640 1 -- 172.21.15.27:0/400998742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4802d040 con 0x7f1a580b95a0 2024-03-06T07:34:22.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.980+0000 7f1a5fa65640 1 -- 172.21.15.27:0/400998742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a580b95a0 msgr2=0x7f1a580b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.980+0000 7f1a5fa65640 1 --2- 172.21.15.27:0/400998742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a580b95a0 0x7f1a580b5700 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009870 tx=0x7f1a4802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.980+0000 7f1a5fa65640 1 -- 172.21.15.27:0/400998742 shutdown_connections 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.980+0000 7f1a5fa65640 1 --2- 172.21.15.27:0/400998742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a580b95a0 0x7f1a580b5700 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.980+0000 7f1a5fa65640 1 --2- 172.21.15.27:0/400998742 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b9a10 0x7f1a580b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.980+0000 7f1a5fa65640 1 -- 172.21.15.27:0/400998742 >> 172.21.15.27:0/400998742 conn(0x7f1a58071960 msgr2=0x7f1a58071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.980+0000 7f1a5fa65640 1 -- 172.21.15.27:0/400998742 shutdown_connections 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.980+0000 7f1a5fa65640 1 -- 172.21.15.27:0/400998742 wait complete. 2024-03-06T07:34:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.981+0000 7f1a5fa65640 1 Processor -- start 2024-03-06T07:34:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.981+0000 7f1a5fa65640 1 -- start start 2024-03-06T07:34:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5fa65640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b95a0 0x7f1a5815fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5fa65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a580b9a10 0x7f1a58155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5fa65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a5815df00 0x7f1a580cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5fa65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580cedf0 con 0x7f1a5815df00 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5fa65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580cec70 con 0x7f1a580b9a10 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5fa65640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580d3220 con 0x7f1a580b95a0 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5e002640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a5815df00 0x7f1a580cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5e002640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a5815df00 0x7f1a580cbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36956/0 (socket says 172.21.15.27:36956) 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5e002640 1 -- 172.21.15.27:0/295284076 learned_addr learned my addr 172.21.15.27:0/295284076 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5d000640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a580b9a10 0x7f1a58155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5d801640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b95a0 0x7f1a5815fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5e002640 1 -- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b95a0 msgr2=0x7f1a5815fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5e002640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b95a0 0x7f1a5815fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5e002640 1 -- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a580b9a10 msgr2=0x7f1a58155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5e002640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a580b9a10 0x7f1a58155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.982+0000 7f1a5e002640 1 -- 172.21.15.27:0/295284076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a580cc5d0 con 0x7f1a5815df00 2024-03-06T07:34:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.983+0000 7f1a5e002640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a5815df00 0x7f1a580cbda0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000cda0 tx=0x7f1a5000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.983+0000 7f1a4dffb640 1 -- 172.21.15.27:0/295284076 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50003b90 con 0x7f1a5815df00 2024-03-06T07:34:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.983+0000 7f1a4dffb640 1 -- 172.21.15.27:0/295284076 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a50003d30 con 0x7f1a5815df00 2024-03-06T07:34:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.983+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a580cc8b0 con 0x7f1a5815df00 2024-03-06T07:34:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.983+0000 7f1a4dffb640 1 -- 172.21.15.27:0/295284076 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50004240 con 0x7f1a5815df00 2024-03-06T07:34:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.983+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a580cc2e0 con 0x7f1a5815df00 2024-03-06T07:34:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.984+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a58067ff0 con 0x7f1a5815df00 2024-03-06T07:34:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.985+0000 7f1a4dffb640 1 -- 172.21.15.27:0/295284076 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a500043e0 con 0x7f1a5815df00 2024-03-06T07:34:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.985+0000 7f1a4dffb640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a3c06b4d0 0x7f1a3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.985+0000 7f1a4dffb640 1 -- 172.21.15.27:0/295284076 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1a5008acb0 con 0x7f1a5815df00 2024-03-06T07:34:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.985+0000 7f1a5d801640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a3c06b4d0 0x7f1a3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.986+0000 7f1a5d801640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a3c06b4d0 0x7f1a3c06d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f1a54006fd0 tx=0x7f1a54008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:22.994+0000 7f1a4dffb640 1 -- 172.21.15.27:0/295284076 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a5001d340 con 0x7f1a5815df00 2024-03-06T07:34:23.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.217+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a580b8df0 con 0x7f1a5815df00 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.217+0000 7f1a4dffb640 1 -- 172.21.15.27:0/295284076 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f1a50059ba0 con 0x7f1a5815df00 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:23.220 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a3c06b4d0 msgr2=0x7f1a3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a3c06b4d0 0x7f1a3c06d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f1a54006fd0 tx=0x7f1a54008040 comp rx=0 tx=0).stop 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a5815df00 msgr2=0x7f1a580cbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a5815df00 0x7f1a580cbda0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000cda0 tx=0x7f1a5000a660 comp rx=0 tx=0).stop 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 shutdown_connections 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1a3c06b4d0 0x7f1a3c06d990 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a5815df00 0x7f1a580cbda0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a580b9a10 0x7f1a58155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 --2- 172.21.15.27:0/295284076 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1a580b95a0 0x7f1a5815fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 >> 172.21.15.27:0/295284076 conn(0x7f1a58071960 msgr2=0x7f1a580b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 shutdown_connections 2024-03-06T07:34:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:23.220+0000 7f1a5fa65640 1 -- 172.21.15.27:0/295284076 wait complete. 2024-03-06T07:34:23.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:23.434 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:24.436 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.541+0000 7f8678801640 1 Processor -- start 2024-03-06T07:34:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.541+0000 7f8678801640 1 -- start start 2024-03-06T07:34:24.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f8678801640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740cfd90 0x7f86740cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f8678801640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867415ce00 con 0x7f867415f540 2024-03-06T07:34:24.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f8678801640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867415f9e0 con 0x7f86740ce880 2024-03-06T07:34:24.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f8678801640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86740b27e0 con 0x7f86740cfd90 2024-03-06T07:34:24.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f866a59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f867415f540 0x7f86740cf680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57844/0 (socket says 172.21.15.27:57844) 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f866a59c640 1 -- 172.21.15.27:0/1763322877 learned_addr learned my addr 172.21.15.27:0/1763322877 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f867359e640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740cfd90 0x7f86740cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181359895 0 0) 0x7f867415ce00 con 0x7f867415f540 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8654003520 con 0x7f867415f540 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2040164828 0 0) 0x7f8654003520 con 0x7f867415f540 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740cfd90 msgr2=0x7f86740cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f867259c640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740cfd90 0x7f86740cd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 >> v1:172.21.15.69:6789/0 conn(0x7f86740ce880 legacy=0x7f86740722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.542+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867415fcc0 con 0x7f867415f540 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f865c001670 con 0x7f867415f540 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 >> v1:172.21.15.27:6789/0 conn(0x7f867415f540 legacy=0x7f86740cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f867259c640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86540039e0 0x7f8654003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f867259c640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f8654004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f867259c640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740cfd90 0x7f86540050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86740b27e0 con 0x7f8654004390 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865400b9f0 con 0x7f86540039e0 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865400bbf0 con 0x7f86740cfd90 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f866a59c640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f8654004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f8672d9d640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86540039e0 0x7f8654003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f867359e640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740cfd90 0x7f86540050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f866a59c640 1 -- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740cfd90 msgr2=0x7f86540050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f866a59c640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740cfd90 0x7f86540050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f866a59c640 1 -- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86540039e0 msgr2=0x7f8654003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f866a59c640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86540039e0 0x7f8654003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f866a59c640 1 -- 172.21.15.27:0/1763322877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8654003520 con 0x7f8654004390 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.543+0000 7f8672d9d640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86540039e0 0x7f8654003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.544+0000 7f866a59c640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f8654004860 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f86740704a0 tx=0x7f8654005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b01f96db152696 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.544+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c007ca0 con 0x7f8654004390 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.544+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c030070 con 0x7f8654004390 2024-03-06T07:34:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.544+0000 7f867259c640 1 -- 172.21.15.27:0/1763322877 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c005d10 con 0x7f8654004390 2024-03-06T07:34:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.545+0000 7f8678801640 1 -- 172.21.15.27:0/1763322877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 msgr2=0x7f8654004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.545+0000 7f8678801640 1 --2- 172.21.15.27:0/1763322877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f8654004860 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f86740704a0 tx=0x7f8654005830 comp rx=0 tx=0).stop 2024-03-06T07:34:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.545+0000 7f867359e640 1 -- 172.21.15.27:0/1763322877 reap_dead start 2024-03-06T07:34:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.545+0000 7f8678801640 1 -- 172.21.15.27:0/1763322877 shutdown_connections 2024-03-06T07:34:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.545+0000 7f8678801640 1 -- 172.21.15.27:0/1763322877 >> 172.21.15.27:0/1763322877 conn(0x7f8674066d00 msgr2=0x7f8674067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.545+0000 7f8678801640 1 -- 172.21.15.27:0/1763322877 shutdown_connections 2024-03-06T07:34:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.545+0000 7f8678801640 1 -- 172.21.15.27:0/1763322877 wait complete. 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.546+0000 7f8678801640 1 Processor -- start 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.546+0000 7f8678801640 1 -- start start 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8678801640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f867414c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8678801640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86740c8190 0x7f86741f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8678801640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740c8610 0x7f86741f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8678801640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8674160280 con 0x7f8654004390 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8678801640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8674072a20 con 0x7f86740c8190 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8678801640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86740cec60 con 0x7f86740c8610 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8672d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f867414c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8672d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f867414c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43042/0 (socket says 172.21.15.27:43042) 2024-03-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8672d9d640 1 -- 172.21.15.27:0/2738853650 learned_addr learned my addr 172.21.15.27:0/2738853650 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f866a59c640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86740c8190 0x7f86741f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f867359e640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740c8610 0x7f86741f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8672d9d640 1 -- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740c8610 msgr2=0x7f86741f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8672d9d640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740c8610 0x7f86741f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8672d9d640 1 -- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86740c8190 msgr2=0x7f86741f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8672d9d640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86740c8190 0x7f86741f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f8672d9d640 1 -- 172.21.15.27:0/2738853650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867414c760 con 0x7f8654004390 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f866a59c640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86740c8190 0x7f86741f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.547+0000 7f867359e640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740c8610 0x7f86741f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.548+0000 7f8672d9d640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f867414c220 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f866400f980 tx=0x7f866400fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.548+0000 7f866bfff640 1 -- 172.21.15.27:0/2738853650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86640052e0 con 0x7f8654004390 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.548+0000 7f866bfff640 1 -- 172.21.15.27:0/2738853650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8664018070 con 0x7f8654004390 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.548+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86740c8ba0 con 0x7f8654004390 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.548+0000 7f866bfff640 1 -- 172.21.15.27:0/2738853650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866400d9f0 con 0x7f8654004390 2024-03-06T07:34:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.548+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86740a9640 con 0x7f8654004390 2024-03-06T07:34:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.549+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86740c90c0 con 0x7f8654004390 2024-03-06T07:34:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.550+0000 7f866bfff640 1 -- 172.21.15.27:0/2738853650 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f866400db90 con 0x7f8654004390 2024-03-06T07:34:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.550+0000 7f866bfff640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f865806b500 0x7f865806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.550+0000 7f866bfff640 1 -- 172.21.15.27:0/2738853650 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f866408a2d0 con 0x7f8654004390 2024-03-06T07:34:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.550+0000 7f866a59c640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f865806b500 0x7f865806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.551+0000 7f866a59c640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f865806b500 0x7f865806d9c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f865c034000 tx=0x7f865c007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.555+0000 7f866bfff640 1 -- 172.21.15.27:0/2738853650 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86740c90c0 con 0x7f8654004390 2024-03-06T07:34:24.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.774+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86740cbae0 con 0x7f8654004390 2024-03-06T07:34:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.774+0000 7f866bfff640 1 -- 172.21.15.27:0/2738853650 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f86740cbae0 con 0x7f8654004390 2024-03-06T07:34:24.777 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:24.777 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.776+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f865806b500 msgr2=0x7f865806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.776+0000 7f8678801640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f865806b500 0x7f865806d9c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f865c034000 tx=0x7f865c007000 comp rx=0 tx=0).stop 2024-03-06T07:34:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.776+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 msgr2=0x7f867414c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.776+0000 7f8678801640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f867414c220 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f866400f980 tx=0x7f866400fe40 comp rx=0 tx=0).stop 2024-03-06T07:34:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.777+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 shutdown_connections 2024-03-06T07:34:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.777+0000 7f8678801640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f865806b500 0x7f865806d9c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.777+0000 7f8678801640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86740c8610 0x7f86741f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.777+0000 7f8678801640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86740c8190 0x7f86741f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.777+0000 7f8678801640 1 --2- 172.21.15.27:0/2738853650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8654004390 0x7f867414c220 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.777+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 >> 172.21.15.27:0/2738853650 conn(0x7f8674066d00 msgr2=0x7f867415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.777+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 shutdown_connections 2024-03-06T07:34:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:24.777+0000 7f8678801640 1 -- 172.21.15.27:0/2738853650 wait complete. 2024-03-06T07:34:24.991 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:24.991 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:25.992 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.099+0000 7fc6c6015640 1 Processor -- start 2024-03-06T07:34:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.099+0000 7fc6c6015640 1 -- start start 2024-03-06T07:34:26.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.100+0000 7fc6c6015640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6c015df00 0x7fc6c00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6c6015640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6c00b9a30 0x7fc6c00d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6c6015640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c00cea30 con 0x7fc6c00b95a0 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6c6015640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c00d2af0 con 0x7fc6c00b9a30 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6c6015640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c00cf3c0 con 0x7fc6c015df00 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6bffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6c00b9a30 0x7fc6c00d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6beffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc6c00b95a0 0x7fc6c00b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57858/0 (socket says 172.21.15.27:57858) 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6beffd640 1 -- 172.21.15.27:0/682679467 learned_addr learned my addr 172.21.15.27:0/682679467 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6bf7fe640 1 --2- 172.21.15.27:0/682679467 >> v2:172.21.15.69:3300/0 conn(0x7fc6c015df00 0x7fc6c00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6bffff640 1 -- 172.21.15.27:0/682679467 >> v2:172.21.15.69:3300/0 conn(0x7fc6c015df00 msgr2=0x7fc6c00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6bffff640 1 --2- 172.21.15.27:0/682679467 >> v2:172.21.15.69:3300/0 conn(0x7fc6c015df00 0x7fc6c00d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6bffff640 1 -- 172.21.15.27:0/682679467 >> v1:172.21.15.27:6789/0 conn(0x7fc6c00b95a0 legacy=0x7fc6c00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6bffff640 1 -- 172.21.15.27:0/682679467 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c00cf0e0 con 0x7fc6c00b9a30 2024-03-06T07:34:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.101+0000 7fc6be7fc640 1 -- 172.21.15.27:0/682679467 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062019667 0 0) 0x7fc6c00cea30 con 0x7fc6c00b95a0 2024-03-06T07:34:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.102+0000 7fc6bffff640 1 --2- 172.21.15.27:0/682679467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c00b9a30 0x7fc6c00d3220 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fc6c00d11b0 tx=0x7fc6b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe629dd07f4726d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.102+0000 7fc6be7fc640 1 -- 172.21.15.27:0/682679467 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8039070 con 0x7fc6c00b9a30 2024-03-06T07:34:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.102+0000 7fc6be7fc640 1 -- 172.21.15.27:0/682679467 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b802cc80 con 0x7fc6c00b9a30 2024-03-06T07:34:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.102+0000 7fc6be7fc640 1 -- 172.21.15.27:0/682679467 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8034660 con 0x7fc6c00b9a30 2024-03-06T07:34:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.103+0000 7fc6c6015640 1 -- 172.21.15.27:0/682679467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c00b9a30 msgr2=0x7fc6c00d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.103+0000 7fc6c6015640 1 --2- 172.21.15.27:0/682679467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c00b9a30 0x7fc6c00d3220 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fc6c00d11b0 tx=0x7fc6b802bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.103+0000 7fc6c6015640 1 -- 172.21.15.27:0/682679467 shutdown_connections 2024-03-06T07:34:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.103+0000 7fc6c6015640 1 --2- 172.21.15.27:0/682679467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c00b9a30 0x7fc6c00d3220 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.103+0000 7fc6c6015640 1 --2- 172.21.15.27:0/682679467 >> v2:172.21.15.69:3300/0 conn(0x7fc6c015df00 0x7fc6c00d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.103+0000 7fc6c6015640 1 -- 172.21.15.27:0/682679467 >> 172.21.15.27:0/682679467 conn(0x7fc6c0071960 msgr2=0x7fc6c0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.103+0000 7fc6c6015640 1 -- 172.21.15.27:0/682679467 shutdown_connections 2024-03-06T07:34:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.103+0000 7fc6c6015640 1 -- 172.21.15.27:0/682679467 wait complete. 2024-03-06T07:34:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.104+0000 7fc6c6015640 1 Processor -- start 2024-03-06T07:34:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.104+0000 7fc6c6015640 1 -- start start 2024-03-06T07:34:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6c6015640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6c00b95a0 0x7fc6c00cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6c6015640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6c015df00 0x7fc6c00b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6c6015640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c01555f0 0x7fc6c00b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6c6015640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c00cec70 con 0x7fc6c01555f0 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6c6015640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c00ce750 con 0x7fc6c00b95a0 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6c6015640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c00cedf0 con 0x7fc6c015df00 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c01555f0 0x7fc6c00b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c01555f0 0x7fc6c00b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43058/0 (socket says 172.21.15.27:43058) 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6bffff640 1 -- 172.21.15.27:0/3449004060 learned_addr learned my addr 172.21.15.27:0/3449004060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6bf7fe640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6c00b95a0 0x7fc6c00cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6beffd640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6c015df00 0x7fc6c00b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6bffff640 1 -- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6c015df00 msgr2=0x7fc6c00b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6bffff640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6c015df00 0x7fc6c00b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6bffff640 1 -- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6c00b95a0 msgr2=0x7fc6c00cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6bffff640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6c00b95a0 0x7fc6c00cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.105+0000 7fc6bffff640 1 -- 172.21.15.27:0/3449004060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c015e2e0 con 0x7fc6c01555f0 2024-03-06T07:34:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.106+0000 7fc6bffff640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c01555f0 0x7fc6c00b4f10 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8037040 tx=0x7fc6b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.106+0000 7fc6b7fff640 1 -- 172.21.15.27:0/3449004060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8044020 con 0x7fc6c01555f0 2024-03-06T07:34:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.106+0000 7fc6b7fff640 1 -- 172.21.15.27:0/3449004060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b803a040 con 0x7fc6c01555f0 2024-03-06T07:34:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.106+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c00cb810 con 0x7fc6c01555f0 2024-03-06T07:34:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.106+0000 7fc6b7fff640 1 -- 172.21.15.27:0/3449004060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8039050 con 0x7fc6c01555f0 2024-03-06T07:34:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.106+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c00cbdd0 con 0x7fc6c01555f0 2024-03-06T07:34:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.107+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c0067ff0 con 0x7fc6c01555f0 2024-03-06T07:34:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.107+0000 7fc6b7fff640 1 -- 172.21.15.27:0/3449004060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6b8003b90 con 0x7fc6c01555f0 2024-03-06T07:34:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.108+0000 7fc6b7fff640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6a006b4d0 0x7fc6a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.108+0000 7fc6b7fff640 1 -- 172.21.15.27:0/3449004060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc6b80aca60 con 0x7fc6c01555f0 2024-03-06T07:34:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.108+0000 7fc6bf7fe640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6a006b4d0 0x7fc6a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.108+0000 7fc6bf7fe640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6a006b4d0 0x7fc6a006d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8006fd0 tx=0x7fc6a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.117+0000 7fc6b7fff640 1 -- 172.21.15.27:0/3449004060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6b804e020 con 0x7fc6c01555f0 2024-03-06T07:34:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.335+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6c006b780 con 0x7fc6c01555f0 2024-03-06T07:34:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.335+0000 7fc6b7fff640 1 -- 172.21.15.27:0/3449004060 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fc6b807b8d0 con 0x7fc6c01555f0 2024-03-06T07:34:26.337 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:26.337 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:26.337 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6a006b4d0 msgr2=0x7fc6a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6a006b4d0 0x7fc6a006d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8006fd0 tx=0x7fc6a8008040 comp rx=0 tx=0).stop 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c01555f0 msgr2=0x7fc6c00b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c01555f0 0x7fc6c00b4f10 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8037040 tx=0x7fc6b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 shutdown_connections 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6a006b4d0 0x7fc6a006d990 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6c01555f0 0x7fc6c00b4f10 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc6c015df00 0x7fc6c00b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 --2- 172.21.15.27:0/3449004060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6c00b95a0 0x7fc6c00cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 >> 172.21.15.27:0/3449004060 conn(0x7fc6c0071960 msgr2=0x7fc6c00b81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 shutdown_connections 2024-03-06T07:34:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:26.338+0000 7fc6c6015640 1 -- 172.21.15.27:0/3449004060 wait complete. 2024-03-06T07:34:26.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:26.552 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:27.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:27.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.660+0000 7f9ee40df640 1 Processor -- start 2024-03-06T07:34:27.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.660+0000 7f9ee40df640 1 -- start start 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee40df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0d3400 0x7f9edc0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee40df640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9edc0b2b50 0x7f9edc0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee40df640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9edc0ce610 con 0x7f9edc071540 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee40df640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc0d2f70 con 0x7f9edc0b2b50 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee40df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0d4be0 con 0x7f9edc0d3400 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee1e7b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9edc071540 0x7f9edc071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57860/0 (socket says 172.21.15.27:57860) 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee1e7b640 1 -- 172.21.15.27:0/3114194418 learned_addr learned my addr 172.21.15.27:0/3114194418 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee167a640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0d3400 0x7f9edc0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee267c640 1 --2- 172.21.15.27:0/3114194418 >> v2:172.21.15.69:3300/0 conn(0x7f9edc0b2b50 0x7f9edc0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619023310 0 0) 0x7f9edc0ce610 con 0x7f9edc071540 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec8003520 con 0x7f9edc071540 2024-03-06T07:34:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789822887 0 0) 0x7f9ec8003520 con 0x7f9edc071540 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0d3400 msgr2=0x7f9edc0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0d3400 0x7f9edc0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 >> v2:172.21.15.69:3300/0 conn(0x7f9edc0b2b50 msgr2=0x7f9edc0b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 --2- 172.21.15.27:0/3114194418 >> v2:172.21.15.69:3300/0 conn(0x7f9edc0b2b50 0x7f9edc0b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc0d2920 con 0x7f9edc071540 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f9ed8001670 con 0x7f9edc071540 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 >> v1:172.21.15.27:6789/0 conn(0x7f9edc071540 legacy=0x7f9edc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.661+0000 7f9ee0e79640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ec80039b0 0x7f9ec8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee0e79640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0d3400 0x7f9ec8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee0e79640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9ec80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0d2f70 con 0x7f9ec8004d40 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0d4be0 con 0x7f9ec80039b0 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec800b9c0 con 0x7f9edc0d3400 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee267c640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9ec80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee1e7b640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ec80039b0 0x7f9ec8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee167a640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0d3400 0x7f9ec8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee267c640 1 -- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0d3400 msgr2=0x7f9ec8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee267c640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0d3400 0x7f9ec8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee267c640 1 -- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ec80039b0 msgr2=0x7f9ec8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee267c640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ec80039b0 0x7f9ec8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee267c640 1 -- 172.21.15.27:0/3114194418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec8003520 con 0x7f9ec8004d40 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.662+0000 7f9ee1e7b640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ec80039b0 0x7f9ec8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.663+0000 7f9ee267c640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9ec80051c0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009fc0 tx=0x7f9ec8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0315f1cd2333e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.663+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4039070 con 0x7f9ec8004d40 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.663+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed402cc60 con 0x7f9ec8004d40 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.663+0000 7f9ee0e79640 1 -- 172.21.15.27:0/3114194418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4009c10 con 0x7f9ec8004d40 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.663+0000 7f9ee40df640 1 -- 172.21.15.27:0/3114194418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 msgr2=0x7f9ec80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.663+0000 7f9ee40df640 1 --2- 172.21.15.27:0/3114194418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9ec80051c0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009fc0 tx=0x7f9ec8005830 comp rx=0 tx=0).stop 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.663+0000 7f9ee267c640 1 -- 172.21.15.27:0/3114194418 reap_dead start 2024-03-06T07:34:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.663+0000 7f9ee40df640 1 -- 172.21.15.27:0/3114194418 shutdown_connections 2024-03-06T07:34:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.663+0000 7f9ee40df640 1 -- 172.21.15.27:0/3114194418 >> 172.21.15.27:0/3114194418 conn(0x7f9edc0d3ca0 msgr2=0x7f9edc0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.664+0000 7f9ee40df640 1 -- 172.21.15.27:0/3114194418 shutdown_connections 2024-03-06T07:34:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.664+0000 7f9ee40df640 1 -- 172.21.15.27:0/3114194418 wait complete. 2024-03-06T07:34:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.665+0000 7f9ee40df640 1 Processor -- start 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.665+0000 7f9ee40df640 1 -- start start 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.665+0000 7f9ee40df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9edc154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.665+0000 7f9ee40df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0cbf70 0x7f9edc155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.665+0000 7f9ee40df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9edc0cc3d0 0x7f9edc156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.665+0000 7f9ee40df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc15e370 con 0x7f9ec8004d40 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.665+0000 7f9ee40df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0ce8f0 con 0x7f9edc0cc3d0 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.665+0000 7f9ee40df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0caf10 con 0x7f9edc0cbf70 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee1e7b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9edc154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee1e7b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9edc154c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43074/0 (socket says 172.21.15.27:43074) 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee1e7b640 1 -- 172.21.15.27:0/74755550 learned_addr learned my addr 172.21.15.27:0/74755550 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee167a640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0cbf70 0x7f9edc155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee267c640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9edc0cc3d0 0x7f9edc156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee1e7b640 1 -- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0cbf70 msgr2=0x7f9edc155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee1e7b640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0cbf70 0x7f9edc155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee1e7b640 1 -- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9edc0cc3d0 msgr2=0x7f9edc156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee1e7b640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9edc0cc3d0 0x7f9edc156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee1e7b640 1 -- 172.21.15.27:0/74755550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc0cb1b0 con 0x7f9ec8004d40 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee267c640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9edc0cc3d0 0x7f9edc156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee1e7b640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9edc154c50 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800e870 tx=0x7f9ed800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ed27fc640 1 -- 172.21.15.27:0/74755550 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed800ee60 con 0x7f9ec8004d40 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ed27fc640 1 -- 172.21.15.27:0/74755550 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed8018070 con 0x7f9ec8004d40 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc0cb490 con 0x7f9ec8004d40 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ed27fc640 1 -- 172.21.15.27:0/74755550 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed800cad0 con 0x7f9ec8004d40 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.666+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc155490 con 0x7f9ec8004d40 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.667+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc0d2640 con 0x7f9ec8004d40 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.668+0000 7f9ed27fc640 1 -- 172.21.15.27:0/74755550 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ed800cc70 con 0x7f9ec8004d40 2024-03-06T07:34:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.668+0000 7f9ed27fc640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ec006b4d0 0x7f9ec006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.668+0000 7f9ed27fc640 1 -- 172.21.15.27:0/74755550 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9ed808a130 con 0x7f9ec8004d40 2024-03-06T07:34:27.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.669+0000 7f9ee167a640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ec006b4d0 0x7f9ec006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.669+0000 7f9ee167a640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ec006b4d0 0x7f9ec006d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0b5fd0 tx=0x7f9ecc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:27.678+0000 7f9ed27fc640 1 -- 172.21.15.27:0/74755550 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9edc0d2640 con 0x7f9ec8004d40 2024-03-06T07:34:28.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.007+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9edc068700 con 0x7f9ec8004d40 2024-03-06T07:34:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.007+0000 7f9ed27fc640 1 -- 172.21.15.27:0/74755550 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f9ed8058f90 con 0x7f9ec8004d40 2024-03-06T07:34:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:28.009 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ec006b4d0 msgr2=0x7f9ec006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ec006b4d0 0x7f9ec006d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0b5fd0 tx=0x7f9ecc009190 comp rx=0 tx=0).stop 2024-03-06T07:34:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 msgr2=0x7f9edc154c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9edc154c50 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800e870 tx=0x7f9ed800ec70 comp rx=0 tx=0).stop 2024-03-06T07:34:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 shutdown_connections 2024-03-06T07:34:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f9ec006b4d0 0x7f9ec006d990 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9edc0cc3d0 0x7f9edc156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9edc0cbf70 0x7f9edc155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 --2- 172.21.15.27:0/74755550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ec8004d40 0x7f9edc154c50 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 >> 172.21.15.27:0/74755550 conn(0x7f9edc0d3ca0 msgr2=0x7f9edc15e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 shutdown_connections 2024-03-06T07:34:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:28.009+0000 7f9ee40df640 1 -- 172.21.15.27:0/74755550 wait complete. 2024-03-06T07:34:28.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:28.223 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:29.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.330+0000 7f5ffd80e640 1 Processor -- start 2024-03-06T07:34:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.330+0000 7f5ffd80e640 1 -- start start 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ffd80e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ff815e030 0x7f5ff806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ffd80e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff806ef70 0x7f5ff80b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ffd80e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff815e410 con 0x7f5ff80b2290 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ffd80e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff8162700 con 0x7f5ff815e030 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ffd80e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff8161af0 con 0x7f5ff806ef70 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff77fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5ff80b2290 0x7f5ff80d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57872/0 (socket says 172.21.15.27:57872) 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff77fe640 1 -- 172.21.15.27:0/310987632 learned_addr learned my addr 172.21.15.27:0/310987632 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff6ffd640 1 --2- 172.21.15.27:0/310987632 >> v2:172.21.15.69:3300/0 conn(0x7f5ff815e030 0x7f5ff806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff67fc640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff806ef70 0x7f5ff80b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102940302 0 0) 0x7f5ff0001490 con 0x7f5ff80b2290 2024-03-06T07:34:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe0003520 con 0x7f5ff80b2290 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952784484 0 0) 0x7f5fe0003520 con 0x7f5ff80b2290 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff806ef70 msgr2=0x7f5ff80b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff5ffb640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff806ef70 0x7f5ff80b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 >> v2:172.21.15.69:3300/0 conn(0x7f5ff815e030 msgr2=0x7f5ff806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff5ffb640 1 --2- 172.21.15.27:0/310987632 >> v2:172.21.15.69:3300/0 conn(0x7f5ff815e030 0x7f5ff806ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff8161dd0 con 0x7f5ff80b2290 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.331+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f5ff0001bc0 con 0x7f5ff80b2290 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 >> v1:172.21.15.27:6789/0 conn(0x7f5ff80b2290 legacy=0x7f5ff80d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff5ffb640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe00039b0 0x7f5fe0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff5ffb640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff806ef70 0x7f5fe0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff5ffb640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe0004d40 0x7f5fe00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff8162700 con 0x7f5fe0004d40 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff8161af0 con 0x7f5fe00039b0 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe000b9c0 con 0x7f5ff806ef70 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff77fe640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe0004d40 0x7f5fe00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff6ffd640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe00039b0 0x7f5fe0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff67fc640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff806ef70 0x7f5fe0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff77fe640 1 -- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff806ef70 msgr2=0x7f5fe0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff77fe640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff806ef70 0x7f5fe0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff77fe640 1 -- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe00039b0 msgr2=0x7f5fe0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff77fe640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fe00039b0 0x7f5fe0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.332+0000 7f5ff77fe640 1 -- 172.21.15.27:0/310987632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe0003520 con 0x7f5fe0004d40 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.333+0000 7f5ff77fe640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe0004d40 0x7f5fe00051c0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000aa40 tx=0x7f5fe0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c760fb1a42a04e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.333+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0038070 con 0x7f5fe0004d40 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.333+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff00070d0 con 0x7f5fe0004d40 2024-03-06T07:34:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.333+0000 7f5ff5ffb640 1 -- 172.21.15.27:0/310987632 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff000a690 con 0x7f5fe0004d40 2024-03-06T07:34:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.333+0000 7f5ffd80e640 1 -- 172.21.15.27:0/310987632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe0004d40 msgr2=0x7f5fe00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.333+0000 7f5ffd80e640 1 --2- 172.21.15.27:0/310987632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe0004d40 0x7f5fe00051c0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000aa40 tx=0x7f5fe0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.333+0000 7f5ff77fe640 1 -- 172.21.15.27:0/310987632 reap_dead start 2024-03-06T07:34:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.333+0000 7f5ffd80e640 1 -- 172.21.15.27:0/310987632 shutdown_connections 2024-03-06T07:34:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.333+0000 7f5ffd80e640 1 -- 172.21.15.27:0/310987632 >> 172.21.15.27:0/310987632 conn(0x7f5ff80b55e0 msgr2=0x7f5ff80b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.334+0000 7f5ffd80e640 1 -- 172.21.15.27:0/310987632 shutdown_connections 2024-03-06T07:34:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.334+0000 7f5ffd80e640 1 -- 172.21.15.27:0/310987632 wait complete. 2024-03-06T07:34:29.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.335+0000 7f5ffd80e640 1 Processor -- start 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.335+0000 7f5ffd80e640 1 -- start start 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ffd80e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ff8157ba0 0x7f5ff814e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ffd80e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff8157f80 0x7f5ff8154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ffd80e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff8155460 0x7f5ff814c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ffd80e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff806e220 con 0x7f5ff8157f80 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ffd80e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff8162390 con 0x7f5ff8157ba0 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ffd80e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff806e3a0 con 0x7f5ff8155460 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff8157f80 0x7f5ff8154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff8157f80 0x7f5ff8154f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43090/0 (socket says 172.21.15.27:43090) 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff67fc640 1 -- 172.21.15.27:0/258610949 learned_addr learned my addr 172.21.15.27:0/258610949 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff6ffd640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ff8157ba0 0x7f5ff814e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff77fe640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff8155460 0x7f5ff814c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff67fc640 1 -- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff8155460 msgr2=0x7f5ff814c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff67fc640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff8155460 0x7f5ff814c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff67fc640 1 -- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ff8157ba0 msgr2=0x7f5ff814e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff67fc640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ff8157ba0 0x7f5ff814e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff67fc640 1 -- 172.21.15.27:0/258610949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff81558e0 con 0x7f5ff8157f80 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ff67fc640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff8157f80 0x7f5ff8154f20 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0119d0 tx=0x7f5fec011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5fdffff640 1 -- 172.21.15.27:0/258610949 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec00fb50 con 0x7f5ff8157f80 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5fdffff640 1 -- 172.21.15.27:0/258610949 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec00fcf0 con 0x7f5ff8157f80 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff8155ba0 con 0x7f5ff8157f80 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5fdffff640 1 -- 172.21.15.27:0/258610949 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec00fea0 con 0x7f5ff8157f80 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.336+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff81530b0 con 0x7f5ff8157f80 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.338+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff8067ff0 con 0x7f5ff8157f80 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.339+0000 7f5fdffff640 1 -- 172.21.15.27:0/258610949 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fec012040 con 0x7f5ff8157f80 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.339+0000 7f5fdffff640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5fe406b500 0x7f5fe406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.339+0000 7f5fdffff640 1 -- 172.21.15.27:0/258610949 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5fec08a870 con 0x7f5ff8157f80 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.339+0000 7f5ff6ffd640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5fe406b500 0x7f5fe406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.340+0000 7f5ff6ffd640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5fe406b500 0x7f5fe406d9c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8009b00 tx=0x7f5fe8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.344+0000 7f5fdffff640 1 -- 172.21.15.27:0/258610949 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fec019030 con 0x7f5ff8157f80 2024-03-06T07:34:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.592+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ff8161130 con 0x7f5ff8157f80 2024-03-06T07:34:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.592+0000 7f5fdffff640 1 -- 172.21.15.27:0/258610949 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f5fec059650 con 0x7f5ff8157f80 2024-03-06T07:34:29.594 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:29.594 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:29.595 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5fe406b500 msgr2=0x7f5fe406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5fe406b500 0x7f5fe406d9c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8009b00 tx=0x7f5fe8008040 comp rx=0 tx=0).stop 2024-03-06T07:34:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff8157f80 msgr2=0x7f5ff8154f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff8157f80 0x7f5ff8154f20 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0119d0 tx=0x7f5fec011e90 comp rx=0 tx=0).stop 2024-03-06T07:34:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 shutdown_connections 2024-03-06T07:34:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5fe406b500 0x7f5fe406d9c0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ff8155460 0x7f5ff814c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff8157f80 0x7f5ff8154f20 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 --2- 172.21.15.27:0/258610949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ff8157ba0 0x7f5ff814e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.594+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 >> 172.21.15.27:0/258610949 conn(0x7f5ff80b55e0 msgr2=0x7f5ff80b49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.595+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 shutdown_connections 2024-03-06T07:34:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:29.595+0000 7f5ffd80e640 1 -- 172.21.15.27:0/258610949 wait complete. 2024-03-06T07:34:29.808 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:29.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:30.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.915+0000 7ff26a8b5640 1 Processor -- start 2024-03-06T07:34:30.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.916+0000 7ff26a8b5640 1 -- start start 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff26a8b5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff264071540 0x7ff2640b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff26a8b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2640d3400 0x7ff264071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff26a8b5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff264071e60 con 0x7ff2640b2b50 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff26a8b5640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2640d2f70 con 0x7ff264071540 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff26a8b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640d4be0 con 0x7ff2640d3400 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff263fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff264071540 0x7ff2640b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff268e52640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff2640b2b50 0x7ff2640d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57874/0 (socket says 172.21.15.27:57874) 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff268e52640 1 -- 172.21.15.27:0/2036741856 learned_addr learned my addr 172.21.15.27:0/2036741856 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff263fff640 1 --2- 172.21.15.27:0/2036741856 >> v2:172.21.15.27:3300/0 conn(0x7ff264071540 0x7ff2640b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43100/0 (socket says 172.21.15.27:43100) 2024-03-06T07:34:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff263fff640 1 -- 172.21.15.27:0/2036741856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2640d3400 msgr2=0x7ff264071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff263fff640 1 --2- 172.21.15.27:0/2036741856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2640d3400 0x7ff264071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff263fff640 1 -- 172.21.15.27:0/2036741856 >> v1:172.21.15.27:6789/0 conn(0x7ff2640b2b50 legacy=0x7ff2640d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff263fff640 1 -- 172.21.15.27:0/2036741856 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2640ce610 con 0x7ff264071540 2024-03-06T07:34:30.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.917+0000 7ff262ffd640 1 -- 172.21.15.27:0/2036741856 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777704050 0 0) 0x7ff264071e60 con 0x7ff2640b2b50 2024-03-06T07:34:30.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.918+0000 7ff263fff640 1 --2- 172.21.15.27:0/2036741856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 0x7ff2640b30a0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0097a0 tx=0x7ff24c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2d9d20fc9ac0e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.918+0000 7ff262ffd640 1 -- 172.21.15.27:0/2036741856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c035020 con 0x7ff264071540 2024-03-06T07:34:30.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.918+0000 7ff262ffd640 1 -- 172.21.15.27:0/2036741856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c031830 con 0x7ff264071540 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.918+0000 7ff262ffd640 1 -- 172.21.15.27:0/2036741856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c031c00 con 0x7ff264071540 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.919+0000 7ff26a8b5640 1 -- 172.21.15.27:0/2036741856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 msgr2=0x7ff2640b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.919+0000 7ff26a8b5640 1 --2- 172.21.15.27:0/2036741856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 0x7ff2640b30a0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0097a0 tx=0x7ff24c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.919+0000 7ff26a8b5640 1 -- 172.21.15.27:0/2036741856 shutdown_connections 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.919+0000 7ff26a8b5640 1 --2- 172.21.15.27:0/2036741856 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2640d3400 0x7ff264071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.919+0000 7ff26a8b5640 1 --2- 172.21.15.27:0/2036741856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 0x7ff2640b30a0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.919+0000 7ff26a8b5640 1 -- 172.21.15.27:0/2036741856 >> 172.21.15.27:0/2036741856 conn(0x7ff2640d3ca0 msgr2=0x7ff2640d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.919+0000 7ff26a8b5640 1 -- 172.21.15.27:0/2036741856 shutdown_connections 2024-03-06T07:34:30.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.919+0000 7ff26a8b5640 1 -- 172.21.15.27:0/2036741856 wait complete. 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.920+0000 7ff26a8b5640 1 Processor -- start 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.921+0000 7ff26a8b5640 1 -- start start 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.921+0000 7ff26a8b5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 0x7ff26414d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.921+0000 7ff26a8b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2640b2b50 0x7ff2640cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.921+0000 7ff26a8b5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2640d3400 0x7ff2640cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.921+0000 7ff26a8b5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640ce8f0 con 0x7ff264071540 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.921+0000 7ff26a8b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640d2c00 con 0x7ff2640b2b50 2024-03-06T07:34:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.921+0000 7ff26a8b5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26415e370 con 0x7ff2640d3400 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.921+0000 7ff263fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 0x7ff26414d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff263fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 0x7ff26414d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43114/0 (socket says 172.21.15.27:43114) 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff263fff640 1 -- 172.21.15.27:0/1664163995 learned_addr learned my addr 172.21.15.27:0/1664163995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff268e52640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2640d3400 0x7ff2640cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff2637fe640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2640b2b50 0x7ff2640cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff263fff640 1 -- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2640d3400 msgr2=0x7ff2640cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff263fff640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2640d3400 0x7ff2640cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff263fff640 1 -- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2640b2b50 msgr2=0x7ff2640cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff263fff640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2640b2b50 0x7ff2640cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff263fff640 1 -- 172.21.15.27:0/1664163995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2640cb5c0 con 0x7ff264071540 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff263fff640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 0x7ff26414d260 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0098c0 tx=0x7ff24c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff260ff9640 1 -- 172.21.15.27:0/1664163995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c03b9c0 con 0x7ff264071540 2024-03-06T07:34:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff260ff9640 1 -- 172.21.15.27:0/1664163995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c03bb60 con 0x7ff264071540 2024-03-06T07:34:30.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2640cb8a0 con 0x7ff264071540 2024-03-06T07:34:30.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff260ff9640 1 -- 172.21.15.27:0/1664163995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0453f0 con 0x7ff264071540 2024-03-06T07:34:30.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.922+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2640cbe60 con 0x7ff264071540 2024-03-06T07:34:30.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.923+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff264068700 con 0x7ff264071540 2024-03-06T07:34:30.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.924+0000 7ff260ff9640 1 -- 172.21.15.27:0/1664163995 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff24c04c060 con 0x7ff264071540 2024-03-06T07:34:30.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.925+0000 7ff260ff9640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff24406b4d0 0x7ff24406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:30.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.925+0000 7ff260ff9640 1 -- 172.21.15.27:0/1664163995 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff24c0abe50 con 0x7ff264071540 2024-03-06T07:34:30.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.925+0000 7ff2637fe640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff24406b4d0 0x7ff24406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:30.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.925+0000 7ff2637fe640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff24406b4d0 0x7ff24406d990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff2641f7020 tx=0x7ff254009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:30.928+0000 7ff260ff9640 1 -- 172.21.15.27:0/1664163995 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff24c040070 con 0x7ff264071540 2024-03-06T07:34:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.151+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff264071e50 con 0x7ff264071540 2024-03-06T07:34:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.152+0000 7ff260ff9640 1 -- 172.21.15.27:0/1664163995 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7ff24c07a860 con 0x7ff264071540 2024-03-06T07:34:31.153 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:31.154 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:31.154 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff24406b4d0 msgr2=0x7ff24406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff24406b4d0 0x7ff24406d990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff2641f7020 tx=0x7ff254009060 comp rx=0 tx=0).stop 2024-03-06T07:34:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 msgr2=0x7ff26414d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 0x7ff26414d260 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0098c0 tx=0x7ff24c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 shutdown_connections 2024-03-06T07:34:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff24406b4d0 0x7ff24406d990 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff2640d3400 0x7ff2640cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2640b2b50 0x7ff2640cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 --2- 172.21.15.27:0/1664163995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264071540 0x7ff26414d260 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 >> 172.21.15.27:0/1664163995 conn(0x7ff2640d3ca0 msgr2=0x7ff26415fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 shutdown_connections 2024-03-06T07:34:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:31.154+0000 7ff26a8b5640 1 -- 172.21.15.27:0/1664163995 wait complete. 2024-03-06T07:34:31.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:31.368 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:32.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:32.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.475+0000 7f0194217640 1 Processor -- start 2024-03-06T07:34:32.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.475+0000 7f0194217640 1 -- start start 2024-03-06T07:34:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.476+0000 7f0194217640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f018c158520 0x7f018c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.476+0000 7f0194217640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f018c0aee10 0x7f018c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.476+0000 7f0194217640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018c071240 con 0x7f018c06dd60 2024-03-06T07:34:32.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.476+0000 7f0194217640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f018c071b70 con 0x7f018c158520 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.476+0000 7f0194217640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c071520 con 0x7f018c0aee10 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.476+0000 7f01917b2640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f018c06dd60 0x7f018c0ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57876/0 (socket says 172.21.15.27:57876) 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.476+0000 7f01917b2640 1 -- 172.21.15.27:0/4082407824 learned_addr learned my addr 172.21.15.27:0/4082407824 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0191fb3640 1 --2- 172.21.15.27:0/4082407824 >> v2:172.21.15.69:3300/0 conn(0x7f018c158520 0x7f018c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552678540 0 0) 0x7f018c071240 con 0x7f018c06dd60 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f01927b4640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f018c0aee10 0x7f018c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0174005730 con 0x7f018c06dd60 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 806670903 0 0) 0x7f0174005730 con 0x7f018c06dd60 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f018c0aee10 msgr2=0x7f018c0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f018c0aee10 0x7f018c0b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 >> v2:172.21.15.69:3300/0 conn(0x7f018c158520 msgr2=0x7f018c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 --2- 172.21.15.27:0/4082407824 >> v2:172.21.15.69:3300/0 conn(0x7f018c158520 0x7f018c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c071800 con 0x7f018c06dd60 2024-03-06T07:34:32.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f0180001670 con 0x7f018c06dd60 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 >> v1:172.21.15.27:6789/0 conn(0x7f018c06dd60 legacy=0x7f018c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0174005bc0 0x7f0174005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f018c0aee10 0x7f0174006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0174006f50 0x7f01740073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c071b70 con 0x7f0174005bc0 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c071520 con 0x7f0174006f50 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017400dbd0 con 0x7f018c0aee10 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0191fb3640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0174005bc0 0x7f0174005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f01917b2640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f018c0aee10 0x7f0174006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f01927b4640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0174006f50 0x7f01740073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.477+0000 7f0191fb3640 1 -- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f018c0aee10 msgr2=0x7f0174006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.478+0000 7f0191fb3640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f018c0aee10 0x7f0174006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.478+0000 7f0191fb3640 1 -- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0174006f50 msgr2=0x7f01740073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:32.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.478+0000 7f0191fb3640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0174006f50 0x7f01740073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.478+0000 7f0191fb3640 1 -- 172.21.15.27:0/4082407824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0174005730 con 0x7f0174005bc0 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.478+0000 7f01927b4640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0174006f50 0x7f01740073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.478+0000 7f0191fb3640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0174005bc0 0x7f0174005fe0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f0184009ac0 tx=0x7f0174007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650c2aa0ef56656f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.478+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184039070 con 0x7f0174005bc0 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.478+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018402c710 con 0x7f0174005bc0 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.478+0000 7f0190fb1640 1 -- 172.21.15.27:0/4082407824 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184009690 con 0x7f0174005bc0 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.479+0000 7f0194217640 1 -- 172.21.15.27:0/4082407824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0174005bc0 msgr2=0x7f0174005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.479+0000 7f0194217640 1 --2- 172.21.15.27:0/4082407824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0174005bc0 0x7f0174005fe0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f0184009ac0 tx=0x7f0174007a40 comp rx=0 tx=0).stop 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.479+0000 7f01927b4640 1 -- 172.21.15.27:0/4082407824 reap_dead start 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.479+0000 7f0194217640 1 -- 172.21.15.27:0/4082407824 shutdown_connections 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.479+0000 7f0194217640 1 -- 172.21.15.27:0/4082407824 >> 172.21.15.27:0/4082407824 conn(0x7f018c066d00 msgr2=0x7f018c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.479+0000 7f0194217640 1 -- 172.21.15.27:0/4082407824 shutdown_connections 2024-03-06T07:34:32.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.479+0000 7f0194217640 1 -- 172.21.15.27:0/4082407824 wait complete. 2024-03-06T07:34:32.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.480+0000 7f0194217640 1 Processor -- start 2024-03-06T07:34:32.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.480+0000 7f0194217640 1 -- start start 2024-03-06T07:34:32.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f0194217640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0174005bc0 0x7f018c152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f0194217640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f018c1597c0 0x7f018c159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f0194217640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c0cec70 0x7f018c0cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f0194217640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c158a80 con 0x7f018c0cec70 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f0194217640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c158900 con 0x7f018c1597c0 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f0194217640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c06c710 con 0x7f0174005bc0 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f01927b4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c0cec70 0x7f018c0cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f01927b4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c0cec70 0x7f018c0cf100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43130/0 (socket says 172.21.15.27:43130) 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f01927b4640 1 -- 172.21.15.27:0/2400415639 learned_addr learned my addr 172.21.15.27:0/2400415639 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f01917b2640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f018c1597c0 0x7f018c159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f0191fb3640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0174005bc0 0x7f018c152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f01927b4640 1 -- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0174005bc0 msgr2=0x7f018c152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f01927b4640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0174005bc0 0x7f018c152550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f01927b4640 1 -- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f018c1597c0 msgr2=0x7f018c159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f01927b4640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f018c1597c0 0x7f018c159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.481+0000 7f01927b4640 1 -- 172.21.15.27:0/2400415639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c0cfa00 con 0x7f018c0cec70 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.482+0000 7f01927b4640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c0cec70 0x7f018c0cf100 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f018c06ec50 tx=0x7f017c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:32.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.482+0000 7f018a7fc640 1 -- 172.21.15.27:0/2400415639 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c01a030 con 0x7f018c0cec70 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.482+0000 7f018a7fc640 1 -- 172.21.15.27:0/2400415639 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c010d80 con 0x7f018c0cec70 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.482+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018c1545c0 con 0x7f018c0cec70 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.482+0000 7f018a7fc640 1 -- 172.21.15.27:0/2400415639 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c003bd0 con 0x7f018c0cec70 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.482+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018c154b80 con 0x7f018c0cec70 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.483+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018c067ff0 con 0x7f018c0cec70 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.483+0000 7f018a7fc640 1 -- 172.21.15.27:0/2400415639 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f017c01d070 con 0x7f018c0cec70 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.484+0000 7f018a7fc640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f017806b4d0 0x7f017806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.484+0000 7f018a7fc640 1 -- 172.21.15.27:0/2400415639 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f017c08a800 con 0x7f018c0cec70 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.484+0000 7f0191fb3640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f017806b4d0 0x7f017806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.485+0000 7f0191fb3640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f017806b4d0 0x7f017806d990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0184036000 tx=0x7f0184030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:32.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.493+0000 7f018a7fc640 1 -- 172.21.15.27:0/2400415639 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f017c0596f0 con 0x7f018c0cec70 2024-03-06T07:34:32.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.713+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f018c154d50 con 0x7f018c0cec70 2024-03-06T07:34:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.714+0000 7f018a7fc640 1 -- 172.21.15.27:0/2400415639 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f017c058ff0 con 0x7f018c0cec70 2024-03-06T07:34:32.715 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:32.715 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.715+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f017806b4d0 msgr2=0x7f017806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.715+0000 7f0194217640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f017806b4d0 0x7f017806d990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0184036000 tx=0x7f0184030000 comp rx=0 tx=0).stop 2024-03-06T07:34:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.715+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c0cec70 msgr2=0x7f018c0cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.715+0000 7f0194217640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c0cec70 0x7f018c0cf100 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f018c06ec50 tx=0x7f017c010590 comp rx=0 tx=0).stop 2024-03-06T07:34:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.716+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 shutdown_connections 2024-03-06T07:34:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.716+0000 7f0194217640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f017806b4d0 0x7f017806d990 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.716+0000 7f0194217640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c0cec70 0x7f018c0cf100 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.716+0000 7f0194217640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f018c1597c0 0x7f018c159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.716+0000 7f0194217640 1 --2- 172.21.15.27:0/2400415639 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0174005bc0 0x7f018c152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.716+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 >> 172.21.15.27:0/2400415639 conn(0x7f018c066d00 msgr2=0x7f018c0af310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.716+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 shutdown_connections 2024-03-06T07:34:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:32.716+0000 7f0194217640 1 -- 172.21.15.27:0/2400415639 wait complete. 2024-03-06T07:34:32.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:32.930 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:33.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.036+0000 7f52d2781640 1 Processor -- start 2024-03-06T07:34:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.036+0000 7f52d2781640 1 -- start start 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52d2781640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52cc0b95a0 0x7f52cc0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52d2781640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b9a10 0x7f52cc0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52d2781640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52cc0cea30 con 0x7f52cc15df00 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52d2781640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52cc0cf3c0 con 0x7f52cc0b95a0 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52d2781640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0cf0e0 con 0x7f52cc0b9a10 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52cb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52cc0b95a0 0x7f52cc0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52cb7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52cc0b95a0 0x7f52cc0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52428/0 (socket says 172.21.15.27:52428) 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52cb7fe640 1 -- 172.21.15.27:0/1401995756 learned_addr learned my addr 172.21.15.27:0/1401995756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52d0d1e640 1 --2- 172.21.15.27:0/1401995756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b9a10 0x7f52cc0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52cb7fe640 1 -- 172.21.15.27:0/1401995756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b9a10 msgr2=0x7f52cc0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52cb7fe640 1 --2- 172.21.15.27:0/1401995756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b9a10 0x7f52cc0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52cb7fe640 1 -- 172.21.15.27:0/1401995756 >> v1:172.21.15.27:6789/0 conn(0x7f52cc15df00 legacy=0x7f52cc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52cb7fe640 1 -- 172.21.15.27:0/1401995756 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc0ce470 con 0x7f52cc0b95a0 2024-03-06T07:34:34.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.037+0000 7f52caffd640 1 -- 172.21.15.27:0/1401995756 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179778284 0 0) 0x7f52cc0cea30 con 0x7f52cc15df00 2024-03-06T07:34:34.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.038+0000 7f52cb7fe640 1 --2- 172.21.15.27:0/1401995756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b95a0 0x7f52cc0b5700 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009870 tx=0x7f52bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f1fe4e7517631c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.038+0000 7f52caffd640 1 -- 172.21.15.27:0/1401995756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc034070 con 0x7f52cc0b95a0 2024-03-06T07:34:34.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.038+0000 7f52caffd640 1 -- 172.21.15.27:0/1401995756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc030840 con 0x7f52cc0b95a0 2024-03-06T07:34:34.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.038+0000 7f52caffd640 1 -- 172.21.15.27:0/1401995756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc030c10 con 0x7f52cc0b95a0 2024-03-06T07:34:34.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.039+0000 7f52d2781640 1 -- 172.21.15.27:0/1401995756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b95a0 msgr2=0x7f52cc0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.039+0000 7f52d2781640 1 --2- 172.21.15.27:0/1401995756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b95a0 0x7f52cc0b5700 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009870 tx=0x7f52bc02b980 comp rx=0 tx=0).stop 2024-03-06T07:34:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.039+0000 7f52d2781640 1 -- 172.21.15.27:0/1401995756 shutdown_connections 2024-03-06T07:34:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.039+0000 7f52d2781640 1 --2- 172.21.15.27:0/1401995756 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b9a10 0x7f52cc0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.039+0000 7f52d2781640 1 --2- 172.21.15.27:0/1401995756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b95a0 0x7f52cc0b5700 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.039+0000 7f52d2781640 1 -- 172.21.15.27:0/1401995756 >> 172.21.15.27:0/1401995756 conn(0x7f52cc071960 msgr2=0x7f52cc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.039+0000 7f52d2781640 1 -- 172.21.15.27:0/1401995756 shutdown_connections 2024-03-06T07:34:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.039+0000 7f52d2781640 1 -- 172.21.15.27:0/1401995756 wait complete. 2024-03-06T07:34:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.040+0000 7f52d2781640 1 Processor -- start 2024-03-06T07:34:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.040+0000 7f52d2781640 1 -- start start 2024-03-06T07:34:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52d2781640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b95a0 0x7f52cc0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52d2781640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b9a10 0x7f52cc14fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52d2781640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc15df00 0x7f52cc0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52d2781640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0cedf0 con 0x7f52cc0b9a10 2024-03-06T07:34:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52d2781640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0cec70 con 0x7f52cc15df00 2024-03-06T07:34:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52d2781640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0d3220 con 0x7f52cc0b95a0 2024-03-06T07:34:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b9a10 0x7f52cc14fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b9a10 0x7f52cc14fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52434/0 (socket says 172.21.15.27:52434) 2024-03-06T07:34:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cb7fe640 1 -- 172.21.15.27:0/547476121 learned_addr learned my addr 172.21.15.27:0/547476121 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52d0d1e640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc15df00 0x7f52cc0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cbfff640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b95a0 0x7f52cc0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cb7fe640 1 -- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b95a0 msgr2=0x7f52cc0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cb7fe640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b95a0 0x7f52cc0c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cb7fe640 1 -- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc15df00 msgr2=0x7f52cc0c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cb7fe640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc15df00 0x7f52cc0c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cb7fe640 1 -- 172.21.15.27:0/547476121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc14ffa0 con 0x7f52cc0b9a10 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.041+0000 7f52cbfff640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b95a0 0x7f52cc0c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.042+0000 7f52cb7fe640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b9a10 0x7f52cc14fa60 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f52bc000c00 tx=0x7f52bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.042+0000 7f52c8ff9640 1 -- 172.21.15.27:0/547476121 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc034040 con 0x7f52cc0b9a10 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.042+0000 7f52c8ff9640 1 -- 172.21.15.27:0/547476121 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc030640 con 0x7f52cc0b9a10 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.042+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52cc0c7f20 con 0x7f52cc0b9a10 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.042+0000 7f52c8ff9640 1 -- 172.21.15.27:0/547476121 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc039d40 con 0x7f52cc0b9a10 2024-03-06T07:34:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.042+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52cc0c84e0 con 0x7f52cc0b9a10 2024-03-06T07:34:34.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.043+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52cc067ff0 con 0x7f52cc0b9a10 2024-03-06T07:34:34.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.043+0000 7f52c8ff9640 1 -- 172.21.15.27:0/547476121 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52bc04b060 con 0x7f52cc0b9a10 2024-03-06T07:34:34.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.044+0000 7f52c8ff9640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f52b806b4d0 0x7f52b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.044+0000 7f52c8ff9640 1 -- 172.21.15.27:0/547476121 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f52bc0abe60 con 0x7f52cc0b9a10 2024-03-06T07:34:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.044+0000 7f52cbfff640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f52b806b4d0 0x7f52b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.045+0000 7f52cbfff640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f52b806b4d0 0x7f52b806d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f52c0000f60 tx=0x7f52c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.048+0000 7f52c8ff9640 1 -- 172.21.15.27:0/547476121 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52bc07ade0 con 0x7f52cc0b9a10 2024-03-06T07:34:34.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.346+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52cc15d470 con 0x7f52cc0b9a10 2024-03-06T07:34:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.347+0000 7f52c8ff9640 1 -- 172.21.15.27:0/547476121 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f52bc07a6e0 con 0x7f52cc0b9a10 2024-03-06T07:34:34.348 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:34.348 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f52b806b4d0 msgr2=0x7f52b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f52b806b4d0 0x7f52b806d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f52c0000f60 tx=0x7f52c0009040 comp rx=0 tx=0).stop 2024-03-06T07:34:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b9a10 msgr2=0x7f52cc14fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b9a10 0x7f52cc14fa60 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f52bc000c00 tx=0x7f52bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 shutdown_connections 2024-03-06T07:34:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f52b806b4d0 0x7f52b806d990 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52cc15df00 0x7f52cc0c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52cc0b9a10 0x7f52cc14fa60 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 --2- 172.21.15.27:0/547476121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f52cc0b95a0 0x7f52cc0c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 >> 172.21.15.27:0/547476121 conn(0x7f52cc071960 msgr2=0x7f52cc0b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 shutdown_connections 2024-03-06T07:34:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:34.349+0000 7f52d2781640 1 -- 172.21.15.27:0/547476121 wait complete. 2024-03-06T07:34:34.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:34.562 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:35.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:35.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.668+0000 7f8b5d4a4640 1 Processor -- start 2024-03-06T07:34:35.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.668+0000 7f8b5d4a4640 1 -- start start 2024-03-06T07:34:35.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.669+0000 7f8b5d4a4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b5815bac0 0x7f8b5815cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.669+0000 7f8b5d4a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b5806d400 0x7f8b580ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.669+0000 7f8b5d4a4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b580af050 con 0x7f8b580aec40 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.669+0000 7f8b5d4a4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5806cf70 con 0x7f8b5815bac0 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.669+0000 7f8b5d4a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b580729b0 con 0x7f8b5806d400 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.669+0000 7f8b56ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b5815bac0 0x7f8b5815cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.669+0000 7f8b56ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b5815bac0 0x7f8b5815cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52438/0 (socket says 172.21.15.27:52438) 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.670+0000 7f8b56ffd640 1 -- 172.21.15.27:0/1691170186 learned_addr learned my addr 172.21.15.27:0/1691170186 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.670+0000 7f8b567fc640 1 --2- 172.21.15.27:0/1691170186 >> v2:172.21.15.69:3300/0 conn(0x7f8b5806d400 0x7f8b580ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.670+0000 7f8b56ffd640 1 -- 172.21.15.27:0/1691170186 >> v2:172.21.15.69:3300/0 conn(0x7f8b5806d400 msgr2=0x7f8b580ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.670+0000 7f8b56ffd640 1 --2- 172.21.15.27:0/1691170186 >> v2:172.21.15.69:3300/0 conn(0x7f8b5806d400 0x7f8b580ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.670+0000 7f8b56ffd640 1 -- 172.21.15.27:0/1691170186 >> v1:172.21.15.69:6789/0 conn(0x7f8b580aec40 legacy=0x7f8b58070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.670+0000 7f8b56ffd640 1 -- 172.21.15.27:0/1691170186 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5806c920 con 0x7f8b5815bac0 2024-03-06T07:34:35.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.671+0000 7f8b56ffd640 1 --2- 172.21.15.27:0/1691170186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b5815bac0 0x7f8b5815cbe0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f8b5806e100 tx=0x7f8b4c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dee20f321b3983f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.671+0000 7f8b55ffb640 1 -- 172.21.15.27:0/1691170186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c034070 con 0x7f8b5815bac0 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.671+0000 7f8b55ffb640 1 -- 172.21.15.27:0/1691170186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c030860 con 0x7f8b5815bac0 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.671+0000 7f8b55ffb640 1 -- 172.21.15.27:0/1691170186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c030c30 con 0x7f8b5815bac0 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.672+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1691170186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b5815bac0 msgr2=0x7f8b5815cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.672+0000 7f8b5d4a4640 1 --2- 172.21.15.27:0/1691170186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b5815bac0 0x7f8b5815cbe0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f8b5806e100 tx=0x7f8b4c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.672+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1691170186 shutdown_connections 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.672+0000 7f8b5d4a4640 1 --2- 172.21.15.27:0/1691170186 >> v2:172.21.15.69:3300/0 conn(0x7f8b5806d400 0x7f8b580ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.672+0000 7f8b5d4a4640 1 --2- 172.21.15.27:0/1691170186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b5815bac0 0x7f8b5815cbe0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.672+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1691170186 >> 172.21.15.27:0/1691170186 conn(0x7f8b58066d00 msgr2=0x7f8b580670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.672+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1691170186 shutdown_connections 2024-03-06T07:34:35.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.672+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1691170186 wait complete. 2024-03-06T07:34:35.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.673+0000 7f8b5d4a4640 1 Processor -- start 2024-03-06T07:34:35.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.673+0000 7f8b5d4a4640 1 -- start start 2024-03-06T07:34:35.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b5d4a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b5806d400 0x7f8b58151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b5d4a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b580aec40 0x7f8b581f1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b5d4a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b5815bac0 0x7f8b581e6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b5d4a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5815bea0 con 0x7f8b580aec40 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b5d4a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580cc490 con 0x7f8b5815bac0 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b5d4a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5815c020 con 0x7f8b5806d400 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b567fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b580aec40 0x7f8b581f1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b567fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b580aec40 0x7f8b581f1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52454/0 (socket says 172.21.15.27:52454) 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b567fc640 1 -- 172.21.15.27:0/1552869168 learned_addr learned my addr 172.21.15.27:0/1552869168 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b56ffd640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b5806d400 0x7f8b58151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b577fe640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b5815bac0 0x7f8b581e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b567fc640 1 -- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b5806d400 msgr2=0x7f8b58151af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b567fc640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b5806d400 0x7f8b58151af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b567fc640 1 -- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b5815bac0 msgr2=0x7f8b581e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b567fc640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b5815bac0 0x7f8b581e6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b567fc640 1 -- 172.21.15.27:0/1552869168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b58152030 con 0x7f8b580aec40 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b567fc640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b580aec40 0x7f8b581f1e90 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800d800 tx=0x7f8b4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.674+0000 7f8b3f7fe640 1 -- 172.21.15.27:0/1552869168 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4800b6c0 con 0x7f8b580aec40 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.675+0000 7f8b3f7fe640 1 -- 172.21.15.27:0/1552869168 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b48019070 con 0x7f8b580aec40 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.675+0000 7f8b3f7fe640 1 -- 172.21.15.27:0/1552869168 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4800ba80 con 0x7f8b580aec40 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.675+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b581f2840 con 0x7f8b580aec40 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.675+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b581f2e00 con 0x7f8b580aec40 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.676+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b58067ff0 con 0x7f8b580aec40 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.676+0000 7f8b3f7fe640 1 -- 172.21.15.27:0/1552869168 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b4800bc20 con 0x7f8b580aec40 2024-03-06T07:34:35.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.676+0000 7f8b3f7fe640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b4406b500 0x7f8b4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.677+0000 7f8b3f7fe640 1 -- 172.21.15.27:0/1552869168 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8b4808a6e0 con 0x7f8b580aec40 2024-03-06T07:34:35.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.677+0000 7f8b56ffd640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b4406b500 0x7f8b4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.677+0000 7f8b56ffd640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b4406b500 0x7f8b4406d9c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c036000 tx=0x7f8b4c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.681+0000 7f8b3f7fe640 1 -- 172.21.15.27:0/1552869168 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b4801c360 con 0x7f8b580aec40 2024-03-06T07:34:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.902+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b5806de50 con 0x7f8b580aec40 2024-03-06T07:34:35.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.902+0000 7f8b3f7fe640 1 -- 172.21.15.27:0/1552869168 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f8b48059540 con 0x7f8b580aec40 2024-03-06T07:34:35.904 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:35.904 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:35.905 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b4406b500 msgr2=0x7f8b4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b4406b500 0x7f8b4406d9c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c036000 tx=0x7f8b4c03b040 comp rx=0 tx=0).stop 2024-03-06T07:34:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b580aec40 msgr2=0x7f8b581f1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b580aec40 0x7f8b581f1e90 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800d800 tx=0x7f8b4800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 shutdown_connections 2024-03-06T07:34:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8b4406b500 0x7f8b4406d9c0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b5815bac0 0x7f8b581e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b580aec40 0x7f8b581f1e90 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 --2- 172.21.15.27:0/1552869168 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8b5806d400 0x7f8b58151af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 >> 172.21.15.27:0/1552869168 conn(0x7f8b58066d00 msgr2=0x7f8b5815dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 shutdown_connections 2024-03-06T07:34:35.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:35.904+0000 7f8b5d4a4640 1 -- 172.21.15.27:0/1552869168 wait complete. 2024-03-06T07:34:36.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:36.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:37.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:37.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.224+0000 7fc70a7d6640 1 Processor -- start 2024-03-06T07:34:37.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.224+0000 7fc70a7d6640 1 -- start start 2024-03-06T07:34:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc70a7d6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7040b95a0 0x7fc7040b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc70a7d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7040b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc70a7d6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040cea30 con 0x7fc70415df00 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc70a7d6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7040cf3c0 con 0x7fc7040b95a0 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc70a7d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040cf0e0 con 0x7fc7040b9a10 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc7037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7040b95a0 0x7fc7040b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc7037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7040b95a0 0x7fc7040b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52466/0 (socket says 172.21.15.27:52466) 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc7037fe640 1 -- 172.21.15.27:0/1426508720 learned_addr learned my addr 172.21.15.27:0/1426508720 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc708d73640 1 --2- 172.21.15.27:0/1426508720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7040b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc7037fe640 1 -- 172.21.15.27:0/1426508720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 msgr2=0x7fc7040b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc7037fe640 1 --2- 172.21.15.27:0/1426508720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7040b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc7037fe640 1 -- 172.21.15.27:0/1426508720 >> v1:172.21.15.69:6789/0 conn(0x7fc70415df00 legacy=0x7fc7040d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc7037fe640 1 -- 172.21.15.27:0/1426508720 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040ce470 con 0x7fc7040b95a0 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.225+0000 7fc708d73640 1 --2- 172.21.15.27:0/1426508720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7040b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.226+0000 7fc7037fe640 1 --2- 172.21.15.27:0/1426508720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7040b95a0 0x7fc7040b5700 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009870 tx=0x7fc6f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63249f6b662b83de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.226+0000 7fc702ffd640 1 -- 172.21.15.27:0/1426508720 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8035020 con 0x7fc7040b95a0 2024-03-06T07:34:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.226+0000 7fc702ffd640 1 -- 172.21.15.27:0/1426508720 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f8031640 con 0x7fc7040b95a0 2024-03-06T07:34:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.226+0000 7fc702ffd640 1 -- 172.21.15.27:0/1426508720 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f802d040 con 0x7fc7040b95a0 2024-03-06T07:34:37.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.227+0000 7fc70a7d6640 1 -- 172.21.15.27:0/1426508720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7040b95a0 msgr2=0x7fc7040b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.227+0000 7fc70a7d6640 1 --2- 172.21.15.27:0/1426508720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7040b95a0 0x7fc7040b5700 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009870 tx=0x7fc6f802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.227+0000 7fc70a7d6640 1 -- 172.21.15.27:0/1426508720 shutdown_connections 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.227+0000 7fc70a7d6640 1 --2- 172.21.15.27:0/1426508720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7040b95a0 0x7fc7040b5700 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.227+0000 7fc70a7d6640 1 --2- 172.21.15.27:0/1426508720 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7040b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.227+0000 7fc70a7d6640 1 -- 172.21.15.27:0/1426508720 >> 172.21.15.27:0/1426508720 conn(0x7fc704071960 msgr2=0x7fc704071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.227+0000 7fc70a7d6640 1 -- 172.21.15.27:0/1426508720 shutdown_connections 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.227+0000 7fc70a7d6640 1 -- 172.21.15.27:0/1426508720 wait complete. 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.228+0000 7fc70a7d6640 1 Processor -- start 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.228+0000 7fc70a7d6640 1 -- start start 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc70a7d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7040b95a0 0x7fc704159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc70a7d6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7041f6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc70a7d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70415df00 0x7fc7040b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc70a7d6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040cedf0 con 0x7fc70415df00 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc70a7d6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040cec70 con 0x7fc7040b95a0 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc70a7d6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040d3220 con 0x7fc7040b9a10 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc708d73640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70415df00 0x7fc7040b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc708d73640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70415df00 0x7fc7040b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52476/0 (socket says 172.21.15.27:52476) 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc708d73640 1 -- 172.21.15.27:0/3373905661 learned_addr learned my addr 172.21.15.27:0/3373905661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc703fff640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7040b95a0 0x7fc704159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc7037fe640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7041f6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc708d73640 1 -- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 msgr2=0x7fc7041f6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc708d73640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7041f6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc708d73640 1 -- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7040b95a0 msgr2=0x7fc704159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc708d73640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7040b95a0 0x7fc704159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc708d73640 1 -- 172.21.15.27:0/3373905661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7041f77a0 con 0x7fc70415df00 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc703fff640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7040b95a0 0x7fc704159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.229+0000 7fc7037fe640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7041f6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.230+0000 7fc708d73640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70415df00 0x7fc7040b4870 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00cef0 tx=0x7fc6fc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.230+0000 7fc700ff9640 1 -- 172.21.15.27:0/3373905661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc01a070 con 0x7fc70415df00 2024-03-06T07:34:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.230+0000 7fc700ff9640 1 -- 172.21.15.27:0/3373905661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc00d860 con 0x7fc70415df00 2024-03-06T07:34:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.230+0000 7fc700ff9640 1 -- 172.21.15.27:0/3373905661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00dc50 con 0x7fc70415df00 2024-03-06T07:34:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.230+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7041f7a80 con 0x7fc70415df00 2024-03-06T07:34:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.230+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7041f7490 con 0x7fc70415df00 2024-03-06T07:34:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.231+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc704067ff0 con 0x7fc70415df00 2024-03-06T07:34:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.231+0000 7fc700ff9640 1 -- 172.21.15.27:0/3373905661 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6fc01d070 con 0x7fc70415df00 2024-03-06T07:34:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.232+0000 7fc700ff9640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6f006b4d0 0x7fc6f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.232+0000 7fc700ff9640 1 -- 172.21.15.27:0/3373905661 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc6fc08aaf0 con 0x7fc70415df00 2024-03-06T07:34:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.232+0000 7fc703fff640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6f006b4d0 0x7fc6f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.233+0000 7fc703fff640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6f006b4d0 0x7fc6f006d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400a830 tx=0x7fc6f4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.236+0000 7fc700ff9640 1 -- 172.21.15.27:0/3373905661 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6fc059950 con 0x7fc70415df00 2024-03-06T07:34:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.454+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7040cdad0 con 0x7fc70415df00 2024-03-06T07:34:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.455+0000 7fc700ff9640 1 -- 172.21.15.27:0/3373905661 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fc6fc059250 con 0x7fc70415df00 2024-03-06T07:34:37.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:37.456 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:37.457 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.456+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6f006b4d0 msgr2=0x7fc6f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.456+0000 7fc70a7d6640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6f006b4d0 0x7fc6f006d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400a830 tx=0x7fc6f4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70415df00 msgr2=0x7fc7040b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70415df00 0x7fc7040b4870 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00cef0 tx=0x7fc6fc00a710 comp rx=0 tx=0).stop 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 shutdown_connections 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc6f006b4d0 0x7fc6f006d990 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70415df00 0x7fc7040b4870 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc7040b9a10 0x7fc7041f6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 --2- 172.21.15.27:0/3373905661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7040b95a0 0x7fc704159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 >> 172.21.15.27:0/3373905661 conn(0x7fc704071960 msgr2=0x7fc7040b84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 shutdown_connections 2024-03-06T07:34:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:37.457+0000 7fc70a7d6640 1 -- 172.21.15.27:0/3373905661 wait complete. 2024-03-06T07:34:37.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:37.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:38.672 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:38.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.778+0000 7f0fd7881640 1 Processor -- start 2024-03-06T07:34:38.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.778+0000 7f0fd7881640 1 -- start start 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd7881640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0fd015df00 0x7f0fd00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd7881640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b9a30 0x7f0fd00d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd7881640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd00cea30 con 0x7f0fd00b95a0 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd7881640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd00d2af0 con 0x7f0fd015df00 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd7881640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00cf3c0 con 0x7f0fd00b9a30 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd561d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0fd015df00 0x7f0fd00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd561d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0fd015df00 0x7f0fd00d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52492/0 (socket says 172.21.15.27:52492) 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd561d640 1 -- 172.21.15.27:0/3990427739 learned_addr learned my addr 172.21.15.27:0/3990427739 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd5e1e640 1 --2- 172.21.15.27:0/3990427739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b9a30 0x7f0fd00d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd561d640 1 -- 172.21.15.27:0/3990427739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b9a30 msgr2=0x7f0fd00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:38.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd561d640 1 --2- 172.21.15.27:0/3990427739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b9a30 0x7f0fd00d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd561d640 1 -- 172.21.15.27:0/3990427739 >> v1:172.21.15.69:6789/0 conn(0x7f0fd00b95a0 legacy=0x7f0fd00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.779+0000 7f0fd561d640 1 -- 172.21.15.27:0/3990427739 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd00cf0e0 con 0x7f0fd015df00 2024-03-06T07:34:38.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.780+0000 7f0fd561d640 1 --2- 172.21.15.27:0/3990427739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 0x7f0fd00d25b0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009870 tx=0x7f0fc002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655ca782b681e370 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:38.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.780+0000 7f0fc7fff640 1 -- 172.21.15.27:0/3990427739 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0034070 con 0x7f0fd015df00 2024-03-06T07:34:38.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.780+0000 7f0fc7fff640 1 -- 172.21.15.27:0/3990427739 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc0030840 con 0x7f0fd015df00 2024-03-06T07:34:38.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.780+0000 7f0fc7fff640 1 -- 172.21.15.27:0/3990427739 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0030c10 con 0x7f0fd015df00 2024-03-06T07:34:38.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.781+0000 7f0fd7881640 1 -- 172.21.15.27:0/3990427739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 msgr2=0x7f0fd00d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.781+0000 7f0fd7881640 1 --2- 172.21.15.27:0/3990427739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 0x7f0fd00d25b0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009870 tx=0x7f0fc002b980 comp rx=0 tx=0).stop 2024-03-06T07:34:38.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.781+0000 7f0fd7881640 1 -- 172.21.15.27:0/3990427739 shutdown_connections 2024-03-06T07:34:38.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.781+0000 7f0fd7881640 1 --2- 172.21.15.27:0/3990427739 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b9a30 0x7f0fd00d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.781+0000 7f0fd7881640 1 --2- 172.21.15.27:0/3990427739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 0x7f0fd00d25b0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.781+0000 7f0fd7881640 1 -- 172.21.15.27:0/3990427739 >> 172.21.15.27:0/3990427739 conn(0x7f0fd0071960 msgr2=0x7f0fd0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:38.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.781+0000 7f0fd7881640 1 -- 172.21.15.27:0/3990427739 shutdown_connections 2024-03-06T07:34:38.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.781+0000 7f0fd7881640 1 -- 172.21.15.27:0/3990427739 wait complete. 2024-03-06T07:34:38.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd7881640 1 Processor -- start 2024-03-06T07:34:38.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd7881640 1 -- start start 2024-03-06T07:34:38.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd7881640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b95a0 0x7f0fd00c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd7881640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 0x7f0fd0150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd7881640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fd00ca4a0 0x7f0fd015e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd7881640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00cec70 con 0x7f0fd015df00 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd7881640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00ce750 con 0x7f0fd00ca4a0 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd7881640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00cedf0 con 0x7f0fd00b95a0 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd4e1c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 0x7f0fd0150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd4e1c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 0x7f0fd0150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52500/0 (socket says 172.21.15.27:52500) 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.783+0000 7f0fd4e1c640 1 -- 172.21.15.27:0/2434936585 learned_addr learned my addr 172.21.15.27:0/2434936585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd561d640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b95a0 0x7f0fd00c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd5e1e640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fd00ca4a0 0x7f0fd015e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd4e1c640 1 -- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b95a0 msgr2=0x7f0fd00c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd4e1c640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b95a0 0x7f0fd00c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd4e1c640 1 -- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fd00ca4a0 msgr2=0x7f0fd015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd4e1c640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fd00ca4a0 0x7f0fd015e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd4e1c640 1 -- 172.21.15.27:0/2434936585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd00c9400 con 0x7f0fd015df00 2024-03-06T07:34:38.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd4e1c640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 0x7f0fd0150fa0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00c430 tx=0x7f0fcc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:38.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fc5ffb640 1 -- 172.21.15.27:0/2434936585 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc0052d0 con 0x7f0fd015df00 2024-03-06T07:34:38.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fc5ffb640 1 -- 172.21.15.27:0/2434936585 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fcc005490 con 0x7f0fd015df00 2024-03-06T07:34:38.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd00c96e0 con 0x7f0fd015df00 2024-03-06T07:34:38.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fc5ffb640 1 -- 172.21.15.27:0/2434936585 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc007260 con 0x7f0fd015df00 2024-03-06T07:34:38.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.784+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd00c9ca0 con 0x7f0fd015df00 2024-03-06T07:34:38.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.785+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd00d4680 con 0x7f0fd015df00 2024-03-06T07:34:38.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.786+0000 7f0fc5ffb640 1 -- 172.21.15.27:0/2434936585 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fcc007400 con 0x7f0fd015df00 2024-03-06T07:34:38.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.786+0000 7f0fc5ffb640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0fb406b4d0 0x7f0fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.786+0000 7f0fc5ffb640 1 -- 172.21.15.27:0/2434936585 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0fcc089660 con 0x7f0fd015df00 2024-03-06T07:34:38.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.786+0000 7f0fd561d640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0fb406b4d0 0x7f0fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.787+0000 7f0fd561d640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0fb406b4d0 0x7f0fb406d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0036000 tx=0x7f0fc003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:38.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:38.791+0000 7f0fc5ffb640 1 -- 172.21.15.27:0/2434936585 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fcc057e70 con 0x7f0fd015df00 2024-03-06T07:34:39.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.009+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fd0067ff0 con 0x7f0fd015df00 2024-03-06T07:34:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.010+0000 7f0fc5ffb640 1 -- 172.21.15.27:0/2434936585 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f0fcc05bf10 con 0x7f0fd015df00 2024-03-06T07:34:39.011 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:39.011 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0fb406b4d0 msgr2=0x7f0fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0fb406b4d0 0x7f0fb406d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0036000 tx=0x7f0fc003b040 comp rx=0 tx=0).stop 2024-03-06T07:34:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 msgr2=0x7f0fd0150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 0x7f0fd0150fa0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00c430 tx=0x7f0fcc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 shutdown_connections 2024-03-06T07:34:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0fb406b4d0 0x7f0fb406d990 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:39.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0fd00ca4a0 0x7f0fd015e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:39.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd015df00 0x7f0fd0150fa0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:39.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 --2- 172.21.15.27:0/2434936585 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0fd00b95a0 0x7f0fd00c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:39.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.011+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 >> 172.21.15.27:0/2434936585 conn(0x7f0fd0071960 msgr2=0x7f0fd00b7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:39.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.012+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 shutdown_connections 2024-03-06T07:34:39.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:39.012+0000 7f0fd7881640 1 -- 172.21.15.27:0/2434936585 wait complete. 2024-03-06T07:34:39.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:39.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:40.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:40.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.332+0000 7fbb067e9640 1 Processor -- start 2024-03-06T07:34:40.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.332+0000 7fbb067e9640 1 -- start start 2024-03-06T07:34:40.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.333+0000 7fbb067e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb00071540 0x7fbb000b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.333+0000 7fbb067e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb000ce610 con 0x7fbb000d3400 2024-03-06T07:34:40.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.333+0000 7fbb067e9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb000d2920 con 0x7fbb000b2b50 2024-03-06T07:34:40.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.333+0000 7fbb067e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000d2f70 con 0x7fbb00071540 2024-03-06T07:34:40.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.333+0000 7fbaff7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbb000d3400 0x7fbb00071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58654/0 (socket says 172.21.15.27:58654) 2024-03-06T07:34:40.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.333+0000 7fbaff7fe640 1 -- 172.21.15.27:0/1275771510 learned_addr learned my addr 172.21.15.27:0/1275771510 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:40.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921478092 0 0) 0x7fbb000ce610 con 0x7fbb000d3400 2024-03-06T07:34:40.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafffff640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb00071540 0x7fbb000b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8003520 con 0x7fbb000d3400 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 565746522 0 0) 0x7fbae8003520 con 0x7fbb000d3400 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb00071540 msgr2=0x7fbb000b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb00071540 0x7fbb000b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 >> v1:172.21.15.69:6789/0 conn(0x7fbb000b2b50 legacy=0x7fbb000b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb000d2c00 con 0x7fbb000d3400 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fbaf4001670 con 0x7fbb000d3400 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 >> v1:172.21.15.27:6789/0 conn(0x7fbb000d3400 legacy=0x7fbb00071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbae8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb00071540 0x7fbae8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae8004d40 0x7fbae80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000d2f70 con 0x7fbae80039e0 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae800b9c0 con 0x7fbae8004d40 2024-03-06T07:34:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae800bbc0 con 0x7fbb00071540 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbafffff640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbae8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbaff7fe640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb00071540 0x7fbae8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.334+0000 7fbb04d86640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae8004d40 0x7fbae80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.335+0000 7fbafffff640 1 -- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb00071540 msgr2=0x7fbae8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.335+0000 7fbafffff640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb00071540 0x7fbae8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.335+0000 7fbafffff640 1 -- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae8004d40 msgr2=0x7fbae80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.335+0000 7fbafffff640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae8004d40 0x7fbae80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.335+0000 7fbafffff640 1 -- 172.21.15.27:0/1275771510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae8003520 con 0x7fbae80039e0 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.335+0000 7fbaff7fe640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb00071540 0x7fbae8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.335+0000 7fbafffff640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbae8003dc0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009ac0 tx=0x7fbae8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39903945f1001e94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.335+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0039070 con 0x7fbae80039e0 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.336+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf002c600 con 0x7fbae80039e0 2024-03-06T07:34:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.336+0000 7fbafeffd640 1 -- 172.21.15.27:0/1275771510 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0009690 con 0x7fbae80039e0 2024-03-06T07:34:40.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.336+0000 7fbb067e9640 1 -- 172.21.15.27:0/1275771510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 msgr2=0x7fbae8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.336+0000 7fbb067e9640 1 --2- 172.21.15.27:0/1275771510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbae8003dc0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009ac0 tx=0x7fbae8005830 comp rx=0 tx=0).stop 2024-03-06T07:34:40.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.336+0000 7fbb04d86640 1 -- 172.21.15.27:0/1275771510 reap_dead start 2024-03-06T07:34:40.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.336+0000 7fbb067e9640 1 -- 172.21.15.27:0/1275771510 shutdown_connections 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.336+0000 7fbb067e9640 1 -- 172.21.15.27:0/1275771510 >> 172.21.15.27:0/1275771510 conn(0x7fbb000d3ca0 msgr2=0x7fbb000d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.336+0000 7fbb067e9640 1 -- 172.21.15.27:0/1275771510 shutdown_connections 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.336+0000 7fbb067e9640 1 -- 172.21.15.27:0/1275771510 wait complete. 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.337+0000 7fbb067e9640 1 Processor -- start 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbb067e9640 1 -- start start 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbb067e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbb000b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbb067e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b4700 0x7fbb0014f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbb067e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb000ac3b0 0x7fbb000ac830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbb067e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0015e370 con 0x7fbae80039e0 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbb067e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000d4be0 con 0x7fbb000b4700 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbb067e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000b4270 con 0x7fbb000ac3b0 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbafffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbb000b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbafffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbb000b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52522/0 (socket says 172.21.15.27:52522) 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.338+0000 7fbafffff640 1 -- 172.21.15.27:0/3374892538 learned_addr learned my addr 172.21.15.27:0/3374892538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbb04d86640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb000ac3b0 0x7fbb000ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbaff7fe640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b4700 0x7fbb0014f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbafffff640 1 -- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb000ac3b0 msgr2=0x7fbb000ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbafffff640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb000ac3b0 0x7fbb000ac830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbafffff640 1 -- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b4700 msgr2=0x7fbb0014f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbafffff640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b4700 0x7fbb0014f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbafffff640 1 -- 172.21.15.27:0/3374892538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0014f630 con 0x7fbae80039e0 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbafffff640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbb000b3080 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0037040 tx=0x7fbaf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbafcff9640 1 -- 172.21.15.27:0/3374892538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0039050 con 0x7fbae80039e0 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbafcff9640 1 -- 172.21.15.27:0/3374892538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf0045070 con 0x7fbae80039e0 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb000b3660 con 0x7fbae80039e0 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbafcff9640 1 -- 172.21.15.27:0/3374892538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf00044e0 con 0x7fbae80039e0 2024-03-06T07:34:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.339+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb000b3c20 con 0x7fbae80039e0 2024-03-06T07:34:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.340+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb00068700 con 0x7fbae80039e0 2024-03-06T07:34:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.341+0000 7fbafcff9640 1 -- 172.21.15.27:0/3374892538 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaf0003830 con 0x7fbae80039e0 2024-03-06T07:34:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.342+0000 7fbafcff9640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b500 0x7fbaec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.342+0000 7fbafcff9640 1 -- 172.21.15.27:0/3374892538 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbaf00abc50 con 0x7fbae80039e0 2024-03-06T07:34:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.342+0000 7fbaff7fe640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b500 0x7fbaec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:40.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.343+0000 7fbaff7fe640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b500 0x7fbaec06d9c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4001050 tx=0x7fbaf4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:40.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.346+0000 7fbafcff9640 1 -- 172.21.15.27:0/3374892538 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaf00b1020 con 0x7fbae80039e0 2024-03-06T07:34:40.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.567+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb000cf680 con 0x7fbae80039e0 2024-03-06T07:34:40.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.568+0000 7fbafcff9640 1 -- 172.21.15.27:0/3374892538 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fbaf004e020 con 0x7fbae80039e0 2024-03-06T07:34:40.569 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:40.569 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:40.570 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:40.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b500 msgr2=0x7fbaec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b500 0x7fbaec06d9c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4001050 tx=0x7fbaf4009040 comp rx=0 tx=0).stop 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 msgr2=0x7fbb000b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbb000b3080 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0037040 tx=0x7fbaf0004730 comp rx=0 tx=0).stop 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 shutdown_connections 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbaec06b500 0x7fbaec06d9c0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbb000ac3b0 0x7fbb000ac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb000b4700 0x7fbb0014f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 --2- 172.21.15.27:0/3374892538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbae80039e0 0x7fbb000b3080 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 >> 172.21.15.27:0/3374892538 conn(0x7fbb000d3ca0 msgr2=0x7fbb0015f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 shutdown_connections 2024-03-06T07:34:40.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:40.570+0000 7fbb067e9640 1 -- 172.21.15.27:0/3374892538 wait complete. 2024-03-06T07:34:40.783 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:40.784 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:41.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.890+0000 7fba747b6640 1 Processor -- start 2024-03-06T07:34:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.890+0000 7fba747b6640 1 -- start start 2024-03-06T07:34:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.891+0000 7fba747b6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fba6c0d3400 0x7fba6c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.891+0000 7fba747b6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba6c0b2b50 0x7fba6c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.891+0000 7fba747b6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c0ce610 con 0x7fba6c071540 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.891+0000 7fba747b6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6c0d2f70 con 0x7fba6c0b2b50 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.891+0000 7fba747b6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6c0d4be0 con 0x7fba6c0d3400 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.892+0000 7fba72d53640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba6c0b2b50 0x7fba6c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.892+0000 7fba72d53640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba6c0b2b50 0x7fba6c0b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52530/0 (socket says 172.21.15.27:52530) 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.892+0000 7fba72d53640 1 -- 172.21.15.27:0/3639454763 learned_addr learned my addr 172.21.15.27:0/3639454763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.892+0000 7fba71d51640 1 --2- 172.21.15.27:0/3639454763 >> v2:172.21.15.69:3300/0 conn(0x7fba6c0d3400 0x7fba6c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.892+0000 7fba72d53640 1 -- 172.21.15.27:0/3639454763 >> v2:172.21.15.69:3300/0 conn(0x7fba6c0d3400 msgr2=0x7fba6c0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.892+0000 7fba72d53640 1 --2- 172.21.15.27:0/3639454763 >> v2:172.21.15.69:3300/0 conn(0x7fba6c0d3400 0x7fba6c0b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.892+0000 7fba72d53640 1 -- 172.21.15.27:0/3639454763 >> v1:172.21.15.27:6789/0 conn(0x7fba6c071540 legacy=0x7fba6c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.892+0000 7fba72d53640 1 -- 172.21.15.27:0/3639454763 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c0d2920 con 0x7fba6c0b2b50 2024-03-06T07:34:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.892+0000 7fba71550640 1 -- 172.21.15.27:0/3639454763 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104760808 0 0) 0x7fba6c0ce610 con 0x7fba6c071540 2024-03-06T07:34:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.893+0000 7fba72d53640 1 --2- 172.21.15.27:0/3639454763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c0b2b50 0x7fba6c0b6e20 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0b6860 tx=0x7fba6402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1244ec8c8cf89a81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.893+0000 7fba71550640 1 -- 172.21.15.27:0/3639454763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6402cb20 con 0x7fba6c0b2b50 2024-03-06T07:34:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.893+0000 7fba71550640 1 -- 172.21.15.27:0/3639454763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6402ccc0 con 0x7fba6c0b2b50 2024-03-06T07:34:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.893+0000 7fba71550640 1 -- 172.21.15.27:0/3639454763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba64034610 con 0x7fba6c0b2b50 2024-03-06T07:34:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.893+0000 7fba747b6640 1 -- 172.21.15.27:0/3639454763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c0b2b50 msgr2=0x7fba6c0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.893+0000 7fba747b6640 1 --2- 172.21.15.27:0/3639454763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c0b2b50 0x7fba6c0b6e20 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0b6860 tx=0x7fba6402bf80 comp rx=0 tx=0).stop 2024-03-06T07:34:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.894+0000 7fba747b6640 1 -- 172.21.15.27:0/3639454763 shutdown_connections 2024-03-06T07:34:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.894+0000 7fba747b6640 1 --2- 172.21.15.27:0/3639454763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c0b2b50 0x7fba6c0b6e20 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:41.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.894+0000 7fba747b6640 1 --2- 172.21.15.27:0/3639454763 >> v2:172.21.15.69:3300/0 conn(0x7fba6c0d3400 0x7fba6c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.894+0000 7fba747b6640 1 -- 172.21.15.27:0/3639454763 >> 172.21.15.27:0/3639454763 conn(0x7fba6c0d3ca0 msgr2=0x7fba6c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.894+0000 7fba747b6640 1 -- 172.21.15.27:0/3639454763 shutdown_connections 2024-03-06T07:34:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.894+0000 7fba747b6640 1 -- 172.21.15.27:0/3639454763 wait complete. 2024-03-06T07:34:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.895+0000 7fba747b6640 1 Processor -- start 2024-03-06T07:34:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.895+0000 7fba747b6640 1 -- start start 2024-03-06T07:34:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.895+0000 7fba747b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c071540 0x7fba6c151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.895+0000 7fba747b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba6c0b2b50 0x7fba6c0b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.895+0000 7fba747b6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba6c0d3400 0x7fba6c1f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.895+0000 7fba747b6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c15e370 con 0x7fba6c071540 2024-03-06T07:34:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.895+0000 7fba747b6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c0ce8f0 con 0x7fba6c0d3400 2024-03-06T07:34:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.895+0000 7fba747b6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c0c7440 con 0x7fba6c0b2b50 2024-03-06T07:34:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72552640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c071540 0x7fba6c151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72552640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c071540 0x7fba6c151210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52542/0 (socket says 172.21.15.27:52542) 2024-03-06T07:34:41.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72552640 1 -- 172.21.15.27:0/3267667048 learned_addr learned my addr 172.21.15.27:0/3267667048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba71d51640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba6c0b2b50 0x7fba6c0b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72d53640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba6c0d3400 0x7fba6c1f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72552640 1 -- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba6c0b2b50 msgr2=0x7fba6c0b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72552640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba6c0b2b50 0x7fba6c0b5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72552640 1 -- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba6c0d3400 msgr2=0x7fba6c1f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72552640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba6c0d3400 0x7fba6c1f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72552640 1 -- 172.21.15.27:0/3267667048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c0c7690 con 0x7fba6c071540 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72d53640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba6c0d3400 0x7fba6c1f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba72552640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c071540 0x7fba6c151210 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fba6800c4e0 tx=0x7fba6800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba62ffd640 1 -- 172.21.15.27:0/3267667048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68017070 con 0x7fba6c071540 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba62ffd640 1 -- 172.21.15.27:0/3267667048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba680035f0 con 0x7fba6c071540 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6c0c7970 con 0x7fba6c071540 2024-03-06T07:34:41.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba62ffd640 1 -- 172.21.15.27:0/3267667048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba680039e0 con 0x7fba6c071540 2024-03-06T07:34:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.896+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6c0c7f30 con 0x7fba6c071540 2024-03-06T07:34:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.898+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6c074c70 con 0x7fba6c071540 2024-03-06T07:34:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.898+0000 7fba62ffd640 1 -- 172.21.15.27:0/3267667048 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba6801a070 con 0x7fba6c071540 2024-03-06T07:34:41.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.898+0000 7fba62ffd640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba5006b4d0 0x7fba5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.898+0000 7fba62ffd640 1 -- 172.21.15.27:0/3267667048 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fba68089420 con 0x7fba6c071540 2024-03-06T07:34:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.898+0000 7fba71d51640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba5006b4d0 0x7fba5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:41.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.899+0000 7fba71d51640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba5006b4d0 0x7fba5006d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fba6c151a00 tx=0x7fba5c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:41.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:41.903+0000 7fba62ffd640 1 -- 172.21.15.27:0/3267667048 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba6c074c70 con 0x7fba6c071540 2024-03-06T07:34:42.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.140+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba6c068700 con 0x7fba6c071540 2024-03-06T07:34:42.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.140+0000 7fba62ffd640 1 -- 172.21.15.27:0/3267667048 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fba68057cc0 con 0x7fba6c071540 2024-03-06T07:34:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:42.142 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:42.142 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba5006b4d0 msgr2=0x7fba5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba5006b4d0 0x7fba5006d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fba6c151a00 tx=0x7fba5c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c071540 msgr2=0x7fba6c151210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c071540 0x7fba6c151210 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fba6800c4e0 tx=0x7fba6800c9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 shutdown_connections 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fba5006b4d0 0x7fba5006d990 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba6c0d3400 0x7fba6c1f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fba6c0b2b50 0x7fba6c0b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 --2- 172.21.15.27:0/3267667048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba6c071540 0x7fba6c151210 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 >> 172.21.15.27:0/3267667048 conn(0x7fba6c0d3ca0 msgr2=0x7fba6c15ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 shutdown_connections 2024-03-06T07:34:42.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:42.142+0000 7fba747b6640 1 -- 172.21.15.27:0/3267667048 wait complete. 2024-03-06T07:34:42.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:42.356 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:43.357 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:43.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.462+0000 7fbf7e088640 1 Processor -- start 2024-03-06T07:34:43.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.462+0000 7fbf7e088640 1 -- start start 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf7e088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 0x7fbf780d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf7e088640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf780b95a0 0x7fbf780b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf7e088640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf780cea30 con 0x7fbf780b9a30 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf7e088640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf780d2af0 con 0x7fbf780b95a0 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf7e088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780cf3c0 con 0x7fbf7815df00 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf76ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf780b95a0 0x7fbf780b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf76ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf780b95a0 0x7fbf780b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52558/0 (socket says 172.21.15.27:52558) 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf76ffd640 1 -- 172.21.15.27:0/2946205684 learned_addr learned my addr 172.21.15.27:0/2946205684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf777fe640 1 --2- 172.21.15.27:0/2946205684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 0x7fbf780d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf76ffd640 1 -- 172.21.15.27:0/2946205684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 msgr2=0x7fbf780d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:43.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.463+0000 7fbf76ffd640 1 --2- 172.21.15.27:0/2946205684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 0x7fbf780d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.464+0000 7fbf76ffd640 1 -- 172.21.15.27:0/2946205684 >> v1:172.21.15.69:6789/0 conn(0x7fbf780b9a30 legacy=0x7fbf780d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:43.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.464+0000 7fbf76ffd640 1 -- 172.21.15.27:0/2946205684 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf780cf0e0 con 0x7fbf780b95a0 2024-03-06T07:34:43.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.464+0000 7fbf76ffd640 1 --2- 172.21.15.27:0/2946205684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 0x7fbf780b5700 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009870 tx=0x7fbf6c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41671d1aabe74af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.464+0000 7fbf767fc640 1 -- 172.21.15.27:0/2946205684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c035020 con 0x7fbf780b95a0 2024-03-06T07:34:43.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.464+0000 7fbf767fc640 1 -- 172.21.15.27:0/2946205684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c031640 con 0x7fbf780b95a0 2024-03-06T07:34:43.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.464+0000 7fbf767fc640 1 -- 172.21.15.27:0/2946205684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c02d040 con 0x7fbf780b95a0 2024-03-06T07:34:43.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.465+0000 7fbf7e088640 1 -- 172.21.15.27:0/2946205684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 msgr2=0x7fbf780b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.465+0000 7fbf7e088640 1 --2- 172.21.15.27:0/2946205684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 0x7fbf780b5700 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009870 tx=0x7fbf6c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.465+0000 7fbf7e088640 1 -- 172.21.15.27:0/2946205684 shutdown_connections 2024-03-06T07:34:43.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.465+0000 7fbf7e088640 1 --2- 172.21.15.27:0/2946205684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 0x7fbf780b5700 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.465+0000 7fbf7e088640 1 --2- 172.21.15.27:0/2946205684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 0x7fbf780d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.465+0000 7fbf7e088640 1 -- 172.21.15.27:0/2946205684 >> 172.21.15.27:0/2946205684 conn(0x7fbf78071960 msgr2=0x7fbf78071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:43.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.465+0000 7fbf7e088640 1 -- 172.21.15.27:0/2946205684 shutdown_connections 2024-03-06T07:34:43.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.465+0000 7fbf7e088640 1 -- 172.21.15.27:0/2946205684 wait complete. 2024-03-06T07:34:43.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.466+0000 7fbf7e088640 1 Processor -- start 2024-03-06T07:34:43.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.466+0000 7fbf7e088640 1 -- start start 2024-03-06T07:34:43.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf7e088640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 0x7fbf7815f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf7e088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 0x7fbf7815bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf7e088640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf7815b2a0 0x7fbf780c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf7e088640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780cec70 con 0x7fbf780b95a0 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf7e088640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780ce750 con 0x7fbf7815b2a0 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf7e088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780cedf0 con 0x7fbf7815df00 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf777fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 0x7fbf7815f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf777fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 0x7fbf7815f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52564/0 (socket says 172.21.15.27:52564) 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf777fe640 1 -- 172.21.15.27:0/1193152726 learned_addr learned my addr 172.21.15.27:0/1193152726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf76ffd640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 0x7fbf7815bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.467+0000 7fbf77fff640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf7815b2a0 0x7fbf780c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf777fe640 1 -- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 msgr2=0x7fbf7815bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf777fe640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 0x7fbf7815bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf777fe640 1 -- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf7815b2a0 msgr2=0x7fbf780c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf777fe640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf7815b2a0 0x7fbf780c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf777fe640 1 -- 172.21.15.27:0/1193152726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf780c8d50 con 0x7fbf780b95a0 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf77fff640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf7815b2a0 0x7fbf780c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf777fe640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 0x7fbf7815f570 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800d620 tx=0x7fbf6800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf5ffff640 1 -- 172.21.15.27:0/1193152726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6800b620 con 0x7fbf780b95a0 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf5ffff640 1 -- 172.21.15.27:0/1193152726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf68019070 con 0x7fbf780b95a0 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf780c9030 con 0x7fbf780b95a0 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf5ffff640 1 -- 172.21.15.27:0/1193152726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6800ba00 con 0x7fbf780b95a0 2024-03-06T07:34:43.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.468+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf780c95f0 con 0x7fbf780b95a0 2024-03-06T07:34:43.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.469+0000 7fbf5ffff640 1 -- 172.21.15.27:0/1193152726 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf6800bba0 con 0x7fbf780b95a0 2024-03-06T07:34:43.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.469+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf78067ff0 con 0x7fbf780b95a0 2024-03-06T07:34:43.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.470+0000 7fbf5ffff640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf6406b4d0 0x7fbf6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.470+0000 7fbf5ffff640 1 -- 172.21.15.27:0/1193152726 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbf6808a590 con 0x7fbf780b95a0 2024-03-06T07:34:43.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.470+0000 7fbf76ffd640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf6406b4d0 0x7fbf6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.471+0000 7fbf76ffd640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf6406b4d0 0x7fbf6406d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c036000 tx=0x7fbf6c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.474+0000 7fbf5ffff640 1 -- 172.21.15.27:0/1193152726 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf68059480 con 0x7fbf780b95a0 2024-03-06T07:34:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.695+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf7815ccb0 con 0x7fbf780b95a0 2024-03-06T07:34:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.696+0000 7fbf5ffff640 1 -- 172.21.15.27:0/1193152726 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fbf68058d80 con 0x7fbf780b95a0 2024-03-06T07:34:43.697 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:43.697 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:43.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf6406b4d0 msgr2=0x7fbf6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf6406b4d0 0x7fbf6406d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c036000 tx=0x7fbf6c03c040 comp rx=0 tx=0).stop 2024-03-06T07:34:43.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 msgr2=0x7fbf7815f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 0x7fbf7815f570 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800d620 tx=0x7fbf6800dae0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 shutdown_connections 2024-03-06T07:34:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbf6406b4d0 0x7fbf6406d990 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf7815b2a0 0x7fbf780c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbf7815df00 0x7fbf7815bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 --2- 172.21.15.27:0/1193152726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf780b95a0 0x7fbf7815f570 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 >> 172.21.15.27:0/1193152726 conn(0x7fbf78071960 msgr2=0x7fbf780d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 shutdown_connections 2024-03-06T07:34:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:43.698+0000 7fbf7e088640 1 -- 172.21.15.27:0/1193152726 wait complete. 2024-03-06T07:34:43.911 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:43.911 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:44.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.017+0000 7f5ee9f7a640 1 Processor -- start 2024-03-06T07:34:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.017+0000 7f5ee9f7a640 1 -- start start 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.017+0000 7f5ee9f7a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ee415df00 0x7f5ee40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.017+0000 7f5ee9f7a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ee40b95a0 0x7f5ee40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.017+0000 7f5ee9f7a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee40cea30 con 0x7f5ee40b9a30 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.017+0000 7f5ee9f7a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee40d2af0 con 0x7f5ee415df00 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.017+0000 7f5ee9f7a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee40cf3c0 con 0x7f5ee40b95a0 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.018+0000 7f5ee37fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ee415df00 0x7f5ee40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.018+0000 7f5ee37fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ee415df00 0x7f5ee40d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52684/0 (socket says 172.21.15.27:52684) 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.018+0000 7f5ee37fe640 1 -- 172.21.15.27:0/2475007168 learned_addr learned my addr 172.21.15.27:0/2475007168 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.018+0000 7f5ee2ffd640 1 --2- 172.21.15.27:0/2475007168 >> v2:172.21.15.69:3300/0 conn(0x7f5ee40b95a0 0x7f5ee40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.018+0000 7f5ee37fe640 1 -- 172.21.15.27:0/2475007168 >> v2:172.21.15.69:3300/0 conn(0x7f5ee40b95a0 msgr2=0x7f5ee40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.018+0000 7f5ee37fe640 1 --2- 172.21.15.27:0/2475007168 >> v2:172.21.15.69:3300/0 conn(0x7f5ee40b95a0 0x7f5ee40b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.018+0000 7f5ee37fe640 1 -- 172.21.15.27:0/2475007168 >> v1:172.21.15.69:6789/0 conn(0x7f5ee40b9a30 legacy=0x7f5ee40d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.018+0000 7f5ee37fe640 1 -- 172.21.15.27:0/2475007168 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee40cf0e0 con 0x7f5ee415df00 2024-03-06T07:34:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee37fe640 1 --2- 172.21.15.27:0/2475007168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 0x7f5ee40d25b0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40d1760 tx=0x7f5ed402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e7f353e6904c080 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee27fc640 1 -- 172.21.15.27:0/2475007168 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4034070 con 0x7f5ee415df00 2024-03-06T07:34:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee27fc640 1 -- 172.21.15.27:0/2475007168 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed4030860 con 0x7f5ee415df00 2024-03-06T07:34:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee27fc640 1 -- 172.21.15.27:0/2475007168 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4030c30 con 0x7f5ee415df00 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2475007168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 msgr2=0x7f5ee40d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee9f7a640 1 --2- 172.21.15.27:0/2475007168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 0x7f5ee40d25b0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40d1760 tx=0x7f5ed402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2475007168 shutdown_connections 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee9f7a640 1 --2- 172.21.15.27:0/2475007168 >> v2:172.21.15.69:3300/0 conn(0x7f5ee40b95a0 0x7f5ee40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee9f7a640 1 --2- 172.21.15.27:0/2475007168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 0x7f5ee40d25b0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.019+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2475007168 >> 172.21.15.27:0/2475007168 conn(0x7f5ee4071960 msgr2=0x7f5ee4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.020+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2475007168 shutdown_connections 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.020+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2475007168 wait complete. 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.021+0000 7f5ee9f7a640 1 Processor -- start 2024-03-06T07:34:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.021+0000 7f5ee9f7a640 1 -- start start 2024-03-06T07:34:45.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee9f7a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ee40b95a0 0x7f5ee41552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee9f7a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 0x7f5ee40cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee9f7a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ee4155b50 0x7f5ee40b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee9f7a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee40cec70 con 0x7f5ee415df00 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee9f7a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee40ce750 con 0x7f5ee4155b50 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee9f7a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee40cedf0 con 0x7f5ee40b95a0 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 0x7f5ee40cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 0x7f5ee40cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52696/0 (socket says 172.21.15.27:52696) 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee2ffd640 1 -- 172.21.15.27:0/2791724850 learned_addr learned my addr 172.21.15.27:0/2791724850 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee3fff640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ee4155b50 0x7f5ee40b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee37fe640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ee40b95a0 0x7f5ee41552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee2ffd640 1 -- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ee40b95a0 msgr2=0x7f5ee41552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee2ffd640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ee40b95a0 0x7f5ee41552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee2ffd640 1 -- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ee4155b50 msgr2=0x7f5ee40b4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee2ffd640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ee4155b50 0x7f5ee40b4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee2ffd640 1 -- 172.21.15.27:0/2791724850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee415e4f0 con 0x7f5ee415df00 2024-03-06T07:34:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee37fe640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ee40b95a0 0x7f5ee41552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee2ffd640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 0x7f5ee40cc7a0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800d800 tx=0x7f5ed800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ecbfff640 1 -- 172.21.15.27:0/2791724850 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed800b6c0 con 0x7f5ee415df00 2024-03-06T07:34:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ecbfff640 1 -- 172.21.15.27:0/2791724850 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed8019070 con 0x7f5ee415df00 2024-03-06T07:34:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee415e7d0 con 0x7f5ee415df00 2024-03-06T07:34:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ecbfff640 1 -- 172.21.15.27:0/2791724850 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed800bac0 con 0x7f5ee415df00 2024-03-06T07:34:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.022+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee415ed90 con 0x7f5ee415df00 2024-03-06T07:34:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.023+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee415ef90 con 0x7f5ee415df00 2024-03-06T07:34:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.024+0000 7f5ecbfff640 1 -- 172.21.15.27:0/2791724850 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ed800bc60 con 0x7f5ee415df00 2024-03-06T07:34:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.025+0000 7f5ecbfff640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed006b4d0 0x7f5ed006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.025+0000 7f5ecbfff640 1 -- 172.21.15.27:0/2791724850 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5ed808a660 con 0x7f5ee415df00 2024-03-06T07:34:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.025+0000 7f5ee37fe640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed006b4d0 0x7f5ed006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.026+0000 7f5ee37fe640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed006b4d0 0x7f5ed006d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4036000 tx=0x7f5ed403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:45.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.029+0000 7f5ecbfff640 1 -- 172.21.15.27:0/2791724850 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ee415ef90 con 0x7f5ee415df00 2024-03-06T07:34:45.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.250+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ee4067ff0 con 0x7f5ee415df00 2024-03-06T07:34:45.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.251+0000 7f5ecbfff640 1 -- 172.21.15.27:0/2791724850 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f5ed80594c0 con 0x7f5ee415df00 2024-03-06T07:34:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:45.254 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:45.254 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:45.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed006b4d0 msgr2=0x7f5ed006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed006b4d0 0x7f5ed006d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4036000 tx=0x7f5ed403b040 comp rx=0 tx=0).stop 2024-03-06T07:34:45.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 msgr2=0x7f5ee40cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 0x7f5ee40cc7a0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800d800 tx=0x7f5ed800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 shutdown_connections 2024-03-06T07:34:45.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f5ed006b4d0 0x7f5ed006d990 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ee4155b50 0x7f5ee40b4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ee415df00 0x7f5ee40cc7a0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 --2- 172.21.15.27:0/2791724850 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5ee40b95a0 0x7f5ee41552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 >> 172.21.15.27:0/2791724850 conn(0x7f5ee4071960 msgr2=0x7f5ee40b87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:45.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 shutdown_connections 2024-03-06T07:34:45.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:45.253+0000 7f5ee9f7a640 1 -- 172.21.15.27:0/2791724850 wait complete. 2024-03-06T07:34:45.468 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:45.468 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:46.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.573+0000 7f1f4da8a640 1 Processor -- start 2024-03-06T07:34:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.573+0000 7f1f4da8a640 1 -- start start 2024-03-06T07:34:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.574+0000 7f1f4da8a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f480b95a0 0x7f1f480b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.574+0000 7f1f4da8a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f480b9a10 0x7f1f480b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.574+0000 7f1f4da8a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f480cea30 con 0x7f1f4815df00 2024-03-06T07:34:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.574+0000 7f1f4da8a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f480cf3c0 con 0x7f1f480b95a0 2024-03-06T07:34:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.574+0000 7f1f4da8a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f480cf0e0 con 0x7f1f480b9a10 2024-03-06T07:34:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.574+0000 7f1f46ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f480b95a0 0x7f1f480b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.574+0000 7f1f477fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1f4815df00 0x7f1f480d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38196/0 (socket says 172.21.15.27:38196) 2024-03-06T07:34:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.575+0000 7f1f46ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f480b95a0 0x7f1f480b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52708/0 (socket says 172.21.15.27:52708) 2024-03-06T07:34:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.575+0000 7f1f477fe640 1 -- 172.21.15.27:0/3571340930 learned_addr learned my addr 172.21.15.27:0/3571340930 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.575+0000 7f1f47fff640 1 --2- 172.21.15.27:0/3571340930 >> v2:172.21.15.69:3300/0 conn(0x7f1f480b9a10 0x7f1f480b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.575+0000 7f1f46ffd640 1 -- 172.21.15.27:0/3571340930 >> v2:172.21.15.69:3300/0 conn(0x7f1f480b9a10 msgr2=0x7f1f480b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.575+0000 7f1f46ffd640 1 --2- 172.21.15.27:0/3571340930 >> v2:172.21.15.69:3300/0 conn(0x7f1f480b9a10 0x7f1f480b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.575+0000 7f1f46ffd640 1 -- 172.21.15.27:0/3571340930 >> v1:172.21.15.27:6789/0 conn(0x7f1f4815df00 legacy=0x7f1f480d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.575+0000 7f1f46ffd640 1 -- 172.21.15.27:0/3571340930 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f480ce470 con 0x7f1f480b95a0 2024-03-06T07:34:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.575+0000 7f1f467fc640 1 -- 172.21.15.27:0/3571340930 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410830233 0 0) 0x7f1f480cea30 con 0x7f1f4815df00 2024-03-06T07:34:46.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.576+0000 7f1f46ffd640 1 --2- 172.21.15.27:0/3571340930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 0x7f1f480b5700 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f1f480d4d20 tx=0x7f1f3802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3afa040e7c59577c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:46.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.576+0000 7f1f467fc640 1 -- 172.21.15.27:0/3571340930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38035020 con 0x7f1f480b95a0 2024-03-06T07:34:46.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.576+0000 7f1f467fc640 1 -- 172.21.15.27:0/3571340930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f38031830 con 0x7f1f480b95a0 2024-03-06T07:34:46.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.576+0000 7f1f467fc640 1 -- 172.21.15.27:0/3571340930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38031c00 con 0x7f1f480b95a0 2024-03-06T07:34:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.577+0000 7f1f4da8a640 1 -- 172.21.15.27:0/3571340930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 msgr2=0x7f1f480b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.577+0000 7f1f4da8a640 1 --2- 172.21.15.27:0/3571340930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 0x7f1f480b5700 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f1f480d4d20 tx=0x7f1f3802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.577+0000 7f1f4da8a640 1 -- 172.21.15.27:0/3571340930 shutdown_connections 2024-03-06T07:34:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.577+0000 7f1f4da8a640 1 --2- 172.21.15.27:0/3571340930 >> v2:172.21.15.69:3300/0 conn(0x7f1f480b9a10 0x7f1f480b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.577+0000 7f1f4da8a640 1 --2- 172.21.15.27:0/3571340930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 0x7f1f480b5700 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.577+0000 7f1f4da8a640 1 -- 172.21.15.27:0/3571340930 >> 172.21.15.27:0/3571340930 conn(0x7f1f48071960 msgr2=0x7f1f48071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.577+0000 7f1f4da8a640 1 -- 172.21.15.27:0/3571340930 shutdown_connections 2024-03-06T07:34:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.577+0000 7f1f4da8a640 1 -- 172.21.15.27:0/3571340930 wait complete. 2024-03-06T07:34:46.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.578+0000 7f1f4da8a640 1 Processor -- start 2024-03-06T07:34:46.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.578+0000 7f1f4da8a640 1 -- start start 2024-03-06T07:34:46.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f4da8a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 0x7f1f480cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f4da8a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f480b9a10 0x7f1f481584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f4da8a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f4815df00 0x7f1f481f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f4da8a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f480cedf0 con 0x7f1f480b95a0 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f4da8a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f480cec70 con 0x7f1f4815df00 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f4da8a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f480d3220 con 0x7f1f480b9a10 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 0x7f1f480cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 0x7f1f480cb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52718/0 (socket says 172.21.15.27:52718) 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f477fe640 1 -- 172.21.15.27:0/2151860805 learned_addr learned my addr 172.21.15.27:0/2151860805 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f47fff640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f4815df00 0x7f1f481f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.579+0000 7f1f46ffd640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f480b9a10 0x7f1f481584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f477fe640 1 -- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f480b9a10 msgr2=0x7f1f481584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f477fe640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f480b9a10 0x7f1f481584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f477fe640 1 -- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f4815df00 msgr2=0x7f1f481f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f477fe640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f4815df00 0x7f1f481f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f477fe640 1 -- 172.21.15.27:0/2151860805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f481f4310 con 0x7f1f480b95a0 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f46ffd640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f480b9a10 0x7f1f481584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:46.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f477fe640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 0x7f1f480cb680 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c430 tx=0x7f1f3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f2ffff640 1 -- 172.21.15.27:0/2151860805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c0074e0 con 0x7f1f480b95a0 2024-03-06T07:34:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f2ffff640 1 -- 172.21.15.27:0/2151860805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c0047e0 con 0x7f1f480b95a0 2024-03-06T07:34:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f481f45f0 con 0x7f1f480b95a0 2024-03-06T07:34:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f2ffff640 1 -- 172.21.15.27:0/2151860805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c004bd0 con 0x7f1f480b95a0 2024-03-06T07:34:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.580+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f480cb160 con 0x7f1f480b95a0 2024-03-06T07:34:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.581+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f480cb360 con 0x7f1f480b95a0 2024-03-06T07:34:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.582+0000 7f1f2ffff640 1 -- 172.21.15.27:0/2151860805 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f3c011070 con 0x7f1f480b95a0 2024-03-06T07:34:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.582+0000 7f1f2ffff640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f3406b4d0 0x7f1f3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.582+0000 7f1f2ffff640 1 -- 172.21.15.27:0/2151860805 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1f3c089d70 con 0x7f1f480b95a0 2024-03-06T07:34:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.582+0000 7f1f46ffd640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f3406b4d0 0x7f1f3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.583+0000 7f1f46ffd640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f3406b4d0 0x7f1f3406d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f1f38036000 tx=0x7f1f38008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:46.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.587+0000 7f1f2ffff640 1 -- 172.21.15.27:0/2151860805 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f3c058c60 con 0x7f1f480b95a0 2024-03-06T07:34:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.806+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f48067ff0 con 0x7f1f480b95a0 2024-03-06T07:34:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.806+0000 7f1f2ffff640 1 -- 172.21.15.27:0/2151860805 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f1f3c058560 con 0x7f1f480b95a0 2024-03-06T07:34:46.808 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:46.809 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:46.809 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f3406b4d0 msgr2=0x7f1f3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f3406b4d0 0x7f1f3406d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f1f38036000 tx=0x7f1f38008030 comp rx=0 tx=0).stop 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 msgr2=0x7f1f480cb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 0x7f1f480cb680 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c430 tx=0x7f1f3c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 shutdown_connections 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f3406b4d0 0x7f1f3406d990 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f4815df00 0x7f1f481f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f480b9a10 0x7f1f481584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 --2- 172.21.15.27:0/2151860805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f480b95a0 0x7f1f480cb680 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 >> 172.21.15.27:0/2151860805 conn(0x7f1f48071960 msgr2=0x7f1f48071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 shutdown_connections 2024-03-06T07:34:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:46.808+0000 7f1f4da8a640 1 -- 172.21.15.27:0/2151860805 wait complete. 2024-03-06T07:34:47.026 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:47.027 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:48.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:48.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.133+0000 7f89f4af4640 1 Processor -- start 2024-03-06T07:34:48.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.133+0000 7f89f4af4640 1 -- start start 2024-03-06T07:34:48.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89f4af4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89f015df00 0x7f89f00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89f4af4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89f00b95a0 0x7f89f00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89f4af4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f00cea30 con 0x7f89f00b9a30 2024-03-06T07:34:48.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89f4af4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f00d2af0 con 0x7f89f015df00 2024-03-06T07:34:48.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89f4af4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f00cf3c0 con 0x7f89f00b95a0 2024-03-06T07:34:48.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89ee59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89f015df00 0x7f89f00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89ee59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89f015df00 0x7f89f00d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52732/0 (socket says 172.21.15.27:52732) 2024-03-06T07:34:48.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89ee59c640 1 -- 172.21.15.27:0/908978367 learned_addr learned my addr 172.21.15.27:0/908978367 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:48.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89edd9b640 1 --2- 172.21.15.27:0/908978367 >> v2:172.21.15.69:3300/0 conn(0x7f89f00b95a0 0x7f89f00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89ee59c640 1 -- 172.21.15.27:0/908978367 >> v2:172.21.15.69:3300/0 conn(0x7f89f00b95a0 msgr2=0x7f89f00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:48.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89ee59c640 1 --2- 172.21.15.27:0/908978367 >> v2:172.21.15.69:3300/0 conn(0x7f89f00b95a0 0x7f89f00b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89ee59c640 1 -- 172.21.15.27:0/908978367 >> v1:172.21.15.69:6789/0 conn(0x7f89f00b9a30 legacy=0x7f89f00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.134+0000 7f89ee59c640 1 -- 172.21.15.27:0/908978367 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f00cf0e0 con 0x7f89f015df00 2024-03-06T07:34:48.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.136+0000 7f89ee59c640 1 --2- 172.21.15.27:0/908978367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f015df00 0x7f89f00d25b0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f89f00d1760 tx=0x7f89d802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9011564e861092 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.136+0000 7f89ed59a640 1 -- 172.21.15.27:0/908978367 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d8034070 con 0x7f89f015df00 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.136+0000 7f89ed59a640 1 -- 172.21.15.27:0/908978367 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d8030860 con 0x7f89f015df00 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.136+0000 7f89ed59a640 1 -- 172.21.15.27:0/908978367 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d8030c30 con 0x7f89f015df00 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.136+0000 7f89f4af4640 1 -- 172.21.15.27:0/908978367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f015df00 msgr2=0x7f89f00d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.136+0000 7f89f4af4640 1 --2- 172.21.15.27:0/908978367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f015df00 0x7f89f00d25b0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f89f00d1760 tx=0x7f89d802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.137+0000 7f89f4af4640 1 -- 172.21.15.27:0/908978367 shutdown_connections 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.137+0000 7f89f4af4640 1 --2- 172.21.15.27:0/908978367 >> v2:172.21.15.69:3300/0 conn(0x7f89f00b95a0 0x7f89f00b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.137+0000 7f89f4af4640 1 --2- 172.21.15.27:0/908978367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f015df00 0x7f89f00d25b0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.137+0000 7f89f4af4640 1 -- 172.21.15.27:0/908978367 >> 172.21.15.27:0/908978367 conn(0x7f89f0071960 msgr2=0x7f89f0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.137+0000 7f89f4af4640 1 -- 172.21.15.27:0/908978367 shutdown_connections 2024-03-06T07:34:48.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.137+0000 7f89f4af4640 1 -- 172.21.15.27:0/908978367 wait complete. 2024-03-06T07:34:48.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.138+0000 7f89f4af4640 1 Processor -- start 2024-03-06T07:34:48.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.138+0000 7f89f4af4640 1 -- start start 2024-03-06T07:34:48.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89f4af4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89f00b95a0 0x7f89f015b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89f4af4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f015df00 0x7f89f00d3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89f4af4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f00ca6e0 0x7f89f00d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89f4af4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f00cec70 con 0x7f89f00ca6e0 2024-03-06T07:34:48.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89f4af4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f00ce750 con 0x7f89f015df00 2024-03-06T07:34:48.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89f4af4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f00cedf0 con 0x7f89f00b95a0 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89eed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f00ca6e0 0x7f89f00d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89eed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f00ca6e0 0x7f89f00d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52746/0 (socket says 172.21.15.27:52746) 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89eed9d640 1 -- 172.21.15.27:0/1285342399 learned_addr learned my addr 172.21.15.27:0/1285342399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89edd9b640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f015df00 0x7f89f00d3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89ee59c640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89f00b95a0 0x7f89f015b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89eed9d640 1 -- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89f00b95a0 msgr2=0x7f89f015b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89eed9d640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89f00b95a0 0x7f89f015b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89eed9d640 1 -- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f015df00 msgr2=0x7f89f00d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89eed9d640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f015df00 0x7f89f00d3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89eed9d640 1 -- 172.21.15.27:0/1285342399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f00b3270 con 0x7f89f00ca6e0 2024-03-06T07:34:48.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89edd9b640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f015df00 0x7f89f00d3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:48.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89ee59c640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89f00b95a0 0x7f89f015b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:48.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.139+0000 7f89eed9d640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f00ca6e0 0x7f89f00d3bd0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f89e800c7a0 tx=0x7f89e800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:48.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.140+0000 7f89e6ffd640 1 -- 172.21.15.27:0/1285342399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e8003b70 con 0x7f89f00ca6e0 2024-03-06T07:34:48.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.140+0000 7f89e6ffd640 1 -- 172.21.15.27:0/1285342399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e8003d30 con 0x7f89f00ca6e0 2024-03-06T07:34:48.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.140+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f00b2f90 con 0x7f89f00ca6e0 2024-03-06T07:34:48.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.140+0000 7f89e6ffd640 1 -- 172.21.15.27:0/1285342399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e800aa90 con 0x7f89f00ca6e0 2024-03-06T07:34:48.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.140+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f00c7bc0 con 0x7f89f00ca6e0 2024-03-06T07:34:48.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.141+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f0067ff0 con 0x7f89f00ca6e0 2024-03-06T07:34:48.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.141+0000 7f89e6ffd640 1 -- 172.21.15.27:0/1285342399 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89e800ac30 con 0x7f89f00ca6e0 2024-03-06T07:34:48.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.142+0000 7f89e6ffd640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89d006b4d0 0x7f89d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.142+0000 7f89e6ffd640 1 -- 172.21.15.27:0/1285342399 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f89e808a0e0 con 0x7f89f00ca6e0 2024-03-06T07:34:48.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.142+0000 7f89ee59c640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89d006b4d0 0x7f89d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.143+0000 7f89ee59c640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89d006b4d0 0x7f89d006d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f89d8036000 tx=0x7f89d803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:48.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.146+0000 7f89e6ffd640 1 -- 172.21.15.27:0/1285342399 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89e8058fd0 con 0x7f89f00ca6e0 2024-03-06T07:34:48.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.364+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89f006b780 con 0x7f89f00ca6e0 2024-03-06T07:34:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.364+0000 7f89e6ffd640 1 -- 172.21.15.27:0/1285342399 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f89e80588d0 con 0x7f89f00ca6e0 2024-03-06T07:34:48.367 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:48.367 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:48.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89d006b4d0 msgr2=0x7f89d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89d006b4d0 0x7f89d006d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f89d8036000 tx=0x7f89d803b040 comp rx=0 tx=0).stop 2024-03-06T07:34:48.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f00ca6e0 msgr2=0x7f89f00d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f00ca6e0 0x7f89f00d3bd0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f89e800c7a0 tx=0x7f89e800cc60 comp rx=0 tx=0).stop 2024-03-06T07:34:48.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 shutdown_connections 2024-03-06T07:34:48.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f89d006b4d0 0x7f89d006d990 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89f00ca6e0 0x7f89f00d3bd0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f015df00 0x7f89f00d3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 --2- 172.21.15.27:0/1285342399 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f89f00b95a0 0x7f89f015b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 >> 172.21.15.27:0/1285342399 conn(0x7f89f0071960 msgr2=0x7f89f00d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 shutdown_connections 2024-03-06T07:34:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:48.366+0000 7f89f4af4640 1 -- 172.21.15.27:0/1285342399 wait complete. 2024-03-06T07:34:48.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:48.581 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:49.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:49.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.687+0000 7fb78a516640 1 Processor -- start 2024-03-06T07:34:49.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.687+0000 7fb78a516640 1 -- start start 2024-03-06T07:34:49.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb78a516640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb7840b95a0 0x7fb7840b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb78a516640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7840b9a10 0x7fb7840b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb78a516640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7840cea30 con 0x7fb78415df00 2024-03-06T07:34:49.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb78a516640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7840cf3c0 con 0x7fb7840b9a10 2024-03-06T07:34:49.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb78a516640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7840cf0e0 con 0x7fb7840b95a0 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb788ab3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7840b9a10 0x7fb7840b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb788ab3640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7840b9a10 0x7fb7840b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52756/0 (socket says 172.21.15.27:52756) 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb788ab3640 1 -- 172.21.15.27:0/288966319 learned_addr learned my addr 172.21.15.27:0/288966319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb783fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb78415df00 0x7fb7840d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38212/0 (socket says 172.21.15.27:38212) 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.688+0000 7fb7837fe640 1 --2- 172.21.15.27:0/288966319 >> v2:172.21.15.69:3300/0 conn(0x7fb7840b95a0 0x7fb7840b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.689+0000 7fb788ab3640 1 -- 172.21.15.27:0/288966319 >> v2:172.21.15.69:3300/0 conn(0x7fb7840b95a0 msgr2=0x7fb7840b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.689+0000 7fb788ab3640 1 --2- 172.21.15.27:0/288966319 >> v2:172.21.15.69:3300/0 conn(0x7fb7840b95a0 0x7fb7840b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.689+0000 7fb788ab3640 1 -- 172.21.15.27:0/288966319 >> v1:172.21.15.27:6789/0 conn(0x7fb78415df00 legacy=0x7fb7840d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.689+0000 7fb788ab3640 1 -- 172.21.15.27:0/288966319 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7840ce470 con 0x7fb7840b9a10 2024-03-06T07:34:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.689+0000 7fb782ffd640 1 -- 172.21.15.27:0/288966319 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218171381 0 0) 0x7fb7840cea30 con 0x7fb78415df00 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.689+0000 7fb788ab3640 1 --2- 172.21.15.27:0/288966319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b9a10 0x7fb7840b5c40 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fb7840d1760 tx=0x7fb77c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5149fcf7dbd761ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb782ffd640 1 -- 172.21.15.27:0/288966319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c02cb20 con 0x7fb7840b9a10 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb782ffd640 1 -- 172.21.15.27:0/288966319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77c02ccc0 con 0x7fb7840b9a10 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb782ffd640 1 -- 172.21.15.27:0/288966319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c034610 con 0x7fb7840b9a10 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb78a516640 1 -- 172.21.15.27:0/288966319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b9a10 msgr2=0x7fb7840b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb78a516640 1 --2- 172.21.15.27:0/288966319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b9a10 0x7fb7840b5c40 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fb7840d1760 tx=0x7fb77c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb78a516640 1 -- 172.21.15.27:0/288966319 shutdown_connections 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb78a516640 1 --2- 172.21.15.27:0/288966319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b9a10 0x7fb7840b5c40 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb78a516640 1 --2- 172.21.15.27:0/288966319 >> v2:172.21.15.69:3300/0 conn(0x7fb7840b95a0 0x7fb7840b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb78a516640 1 -- 172.21.15.27:0/288966319 >> 172.21.15.27:0/288966319 conn(0x7fb784071960 msgr2=0x7fb784071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb78a516640 1 -- 172.21.15.27:0/288966319 shutdown_connections 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.690+0000 7fb78a516640 1 -- 172.21.15.27:0/288966319 wait complete. 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.691+0000 7fb78a516640 1 Processor -- start 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb78a516640 1 -- start start 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb78a516640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b95a0 0x7fb7840cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb78a516640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840b9a10 0x7fb7840c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb78a516640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415df00 0x7fb7840d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb78a516640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7840cedf0 con 0x7fb7840b95a0 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb78a516640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7840cec70 con 0x7fb78415df00 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb78a516640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7840d3220 con 0x7fb7840b9a10 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb783fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b95a0 0x7fb7840cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb783fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b95a0 0x7fb7840cc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52758/0 (socket says 172.21.15.27:52758) 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.692+0000 7fb783fff640 1 -- 172.21.15.27:0/2031576201 learned_addr learned my addr 172.21.15.27:0/2031576201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb788ab3640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415df00 0x7fb7840d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb7837fe640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840b9a10 0x7fb7840c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb783fff640 1 -- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840b9a10 msgr2=0x7fb7840c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb783fff640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840b9a10 0x7fb7840c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb783fff640 1 -- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415df00 msgr2=0x7fb7840d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb783fff640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415df00 0x7fb7840d35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb783fff640 1 -- 172.21.15.27:0/2031576201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb784160550 con 0x7fb7840b95a0 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb788ab3640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415df00 0x7fb7840d35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb783fff640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b95a0 0x7fb7840cc840 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00c430 tx=0x7fb76c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb780ff9640 1 -- 172.21.15.27:0/2031576201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0074e0 con 0x7fb7840b95a0 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb780ff9640 1 -- 172.21.15.27:0/2031576201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c0047e0 con 0x7fb7840b95a0 2024-03-06T07:34:49.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7840ccd80 con 0x7fb7840b95a0 2024-03-06T07:34:49.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb780ff9640 1 -- 172.21.15.27:0/2031576201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c004bd0 con 0x7fb7840b95a0 2024-03-06T07:34:49.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.693+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb784160ad0 con 0x7fb7840b95a0 2024-03-06T07:34:49.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.694+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7841611e0 con 0x7fb7840b95a0 2024-03-06T07:34:49.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.694+0000 7fb780ff9640 1 -- 172.21.15.27:0/2031576201 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb76c011070 con 0x7fb7840b95a0 2024-03-06T07:34:49.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.695+0000 7fb780ff9640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76406b4d0 0x7fb76406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:49.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.695+0000 7fb780ff9640 1 -- 172.21.15.27:0/2031576201 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb76c089ea0 con 0x7fb7840b95a0 2024-03-06T07:34:49.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.695+0000 7fb7837fe640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76406b4d0 0x7fb76406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:49.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.696+0000 7fb7837fe640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76406b4d0 0x7fb76406d990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fb7840cdd50 tx=0x7fb774008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:49.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.699+0000 7fb780ff9640 1 -- 172.21.15.27:0/2031576201 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb76c058d00 con 0x7fb7840b95a0 2024-03-06T07:34:49.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.995+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb78406b780 con 0x7fb7840b95a0 2024-03-06T07:34:49.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.995+0000 7fb780ff9640 1 -- 172.21.15.27:0/2031576201 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fb76c058600 con 0x7fb7840b95a0 2024-03-06T07:34:49.998 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:49.998 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:49.998 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:50.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76406b4d0 msgr2=0x7fb76406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76406b4d0 0x7fb76406d990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fb7840cdd50 tx=0x7fb774008040 comp rx=0 tx=0).stop 2024-03-06T07:34:50.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b95a0 msgr2=0x7fb7840cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:50.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b95a0 0x7fb7840cc840 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00c430 tx=0x7fb76c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 shutdown_connections 2024-03-06T07:34:50.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb76406b4d0 0x7fb76406d990 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb78415df00 0x7fb7840d35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb7840b9a10 0x7fb7840c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 --2- 172.21.15.27:0/2031576201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7840b95a0 0x7fb7840cc840 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:50.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.997+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 >> 172.21.15.27:0/2031576201 conn(0x7fb784071960 msgr2=0x7fb78415d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:50.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.998+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 shutdown_connections 2024-03-06T07:34:50.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:49.998+0000 7fb78a516640 1 -- 172.21.15.27:0/2031576201 wait complete. 2024-03-06T07:34:50.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:50.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:50.217 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:50.217 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:51.218 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:51.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.322+0000 7f4345b3a640 1 Processor -- start 2024-03-06T07:34:51.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.322+0000 7f4345b3a640 1 -- start start 2024-03-06T07:34:51.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.323+0000 7f4345b3a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f43400b9a10 0x7f43400d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.323+0000 7f4345b3a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400cea30 con 0x7f43400b95a0 2024-03-06T07:34:51.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.323+0000 7f4345b3a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400cf3c0 con 0x7f434015df00 2024-03-06T07:34:51.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.323+0000 7f4345b3a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43400cf0e0 con 0x7f43400b9a10 2024-03-06T07:34:51.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.323+0000 7f433effd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f43400b95a0 0x7f43400b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38228/0 (socket says 172.21.15.27:38228) 2024-03-06T07:34:51.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.323+0000 7f433effd640 1 -- 172.21.15.27:0/4209970808 learned_addr learned my addr 172.21.15.27:0/4209970808 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:51.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.324+0000 7f433ffff640 1 --2- 172.21.15.27:0/4209970808 >> v2:172.21.15.69:3300/0 conn(0x7f43400b9a10 0x7f43400d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.324+0000 7f433e7fc640 1 -- 172.21.15.27:0/4209970808 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150929368 0 0) 0x7f43400cea30 con 0x7f43400b95a0 2024-03-06T07:34:51.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.324+0000 7f433e7fc640 1 -- 172.21.15.27:0/4209970808 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f432c003520 con 0x7f43400b95a0 2024-03-06T07:34:51.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.324+0000 7f433e7fc640 1 -- 172.21.15.27:0/4209970808 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781674462 0 0) 0x7f432c003520 con 0x7f43400b95a0 2024-03-06T07:34:51.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.324+0000 7f433ffff640 1 -- 172.21.15.27:0/4209970808 >> v1:172.21.15.69:6789/0 conn(0x7f434015df00 legacy=0x7f43400d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.324+0000 7f433ffff640 1 -- 172.21.15.27:0/4209970808 >> v1:172.21.15.27:6789/0 conn(0x7f43400b95a0 legacy=0x7f43400b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.324+0000 7f433ffff640 1 -- 172.21.15.27:0/4209970808 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400ce470 con 0x7f43400b9a10 2024-03-06T07:34:51.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.325+0000 7f433ffff640 1 --2- 172.21.15.27:0/4209970808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43400b9a10 0x7f43400d3220 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f43400cf590 tx=0x7f433802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=829b4db2b9a91e9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:51.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.325+0000 7f433e7fc640 1 -- 172.21.15.27:0/4209970808 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338039070 con 0x7f43400b9a10 2024-03-06T07:34:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.325+0000 7f433e7fc640 1 -- 172.21.15.27:0/4209970808 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433802c930 con 0x7f43400b9a10 2024-03-06T07:34:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.325+0000 7f433e7fc640 1 -- 172.21.15.27:0/4209970808 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330001670 con 0x7f43400b9a10 2024-03-06T07:34:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.325+0000 7f4345b3a640 1 -- 172.21.15.27:0/4209970808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43400b9a10 msgr2=0x7f43400d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.325+0000 7f4345b3a640 1 --2- 172.21.15.27:0/4209970808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43400b9a10 0x7f43400d3220 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f43400cf590 tx=0x7f433802be00 comp rx=0 tx=0).stop 2024-03-06T07:34:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.325+0000 7f4345b3a640 1 -- 172.21.15.27:0/4209970808 shutdown_connections 2024-03-06T07:34:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.325+0000 7f4345b3a640 1 --2- 172.21.15.27:0/4209970808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43400b9a10 0x7f43400d3220 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.325+0000 7f4345b3a640 1 -- 172.21.15.27:0/4209970808 >> 172.21.15.27:0/4209970808 conn(0x7f4340071960 msgr2=0x7f4340071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.326+0000 7f4345b3a640 1 -- 172.21.15.27:0/4209970808 shutdown_connections 2024-03-06T07:34:51.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.326+0000 7f4345b3a640 1 -- 172.21.15.27:0/4209970808 wait complete. 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.327+0000 7f4345b3a640 1 Processor -- start 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.327+0000 7f4345b3a640 1 -- start start 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4345b3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43400b95a0 0x7f43401576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4345b3a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43400b9a10 0x7f43400cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4345b3a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434015df00 0x7f43400ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4345b3a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43400cedf0 con 0x7f43400b9a10 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4345b3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43400cec70 con 0x7f43400b95a0 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4345b3a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43400b5e10 con 0x7f434015df00 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43400b9a10 0x7f43400cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43400b9a10 0x7f43400cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52766/0 (socket says 172.21.15.27:52766) 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433effd640 1 -- 172.21.15.27:0/1071306207 learned_addr learned my addr 172.21.15.27:0/1071306207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433f7fe640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43400b95a0 0x7f43401576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433ffff640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434015df00 0x7f43400ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433effd640 1 -- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434015df00 msgr2=0x7f43400ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433effd640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434015df00 0x7f43400ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433effd640 1 -- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43400b95a0 msgr2=0x7f43401576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433effd640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43400b95a0 0x7f43401576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433effd640 1 -- 172.21.15.27:0/1071306207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434015e2e0 con 0x7f43400b9a10 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f433effd640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43400b9a10 0x7f43400cc9d0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f433000c730 tx=0x7f433000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4337fff640 1 -- 172.21.15.27:0/1071306207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330017070 con 0x7f43400b9a10 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4337fff640 1 -- 172.21.15.27:0/1071306207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4330005af0 con 0x7f43400b9a10 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434015e5a0 con 0x7f43400b9a10 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.328+0000 7f4337fff640 1 -- 172.21.15.27:0/1071306207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330007070 con 0x7f43400b9a10 2024-03-06T07:34:51.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.329+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434015eb60 con 0x7f43400b9a10 2024-03-06T07:34:51.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.330+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4340067ff0 con 0x7f43400b9a10 2024-03-06T07:34:51.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.330+0000 7f4337fff640 1 -- 172.21.15.27:0/1071306207 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f433001a070 con 0x7f43400b9a10 2024-03-06T07:34:51.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.331+0000 7f4337fff640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f432006b4d0 0x7f432006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:51.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.331+0000 7f4337fff640 1 -- 172.21.15.27:0/1071306207 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4330089690 con 0x7f43400b9a10 2024-03-06T07:34:51.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.331+0000 7f433f7fe640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f432006b4d0 0x7f432006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:51.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.331+0000 7f433f7fe640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f432006b4d0 0x7f432006d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4328000f60 tx=0x7f4328009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:51.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.335+0000 7f4337fff640 1 -- 172.21.15.27:0/1071306207 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4330057ea0 con 0x7f43400b9a10 2024-03-06T07:34:51.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.557+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43400b5a90 con 0x7f43400b9a10 2024-03-06T07:34:51.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.557+0000 7f4337fff640 1 -- 172.21.15.27:0/1071306207 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f433005bf40 con 0x7f43400b9a10 2024-03-06T07:34:51.560 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:51.560 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:51.560 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:51.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f432006b4d0 msgr2=0x7f432006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f432006b4d0 0x7f432006d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4328000f60 tx=0x7f4328009040 comp rx=0 tx=0).stop 2024-03-06T07:34:51.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43400b9a10 msgr2=0x7f43400cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:51.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43400b9a10 0x7f43400cc9d0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f433000c730 tx=0x7f433000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 shutdown_connections 2024-03-06T07:34:51.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f432006b4d0 0x7f432006d990 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f434015df00 0x7f43400ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43400b9a10 0x7f43400cc9d0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 --2- 172.21.15.27:0/1071306207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43400b95a0 0x7f43401576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:51.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.560+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 >> 172.21.15.27:0/1071306207 conn(0x7f4340071960 msgr2=0x7f4340071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:51.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.561+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 shutdown_connections 2024-03-06T07:34:51.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:51.561+0000 7f4345b3a640 1 -- 172.21.15.27:0/1071306207 wait complete. 2024-03-06T07:34:51.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:51.776 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:52.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.882+0000 7fd833c5c640 1 Processor -- start 2024-03-06T07:34:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.882+0000 7fd833c5c640 1 -- start start 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd833c5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c0b95a0 0x7fd82c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd833c5c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd82c0b9a10 0x7fd82c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd833c5c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82c0cea30 con 0x7fd82c15df00 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd833c5c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd82c0cf3c0 con 0x7fd82c0b9a10 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd833c5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c0cf0e0 con 0x7fd82c0b95a0 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd8321f9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd82c0b9a10 0x7fd82c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd8321f9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd82c0b9a10 0x7fd82c0b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52776/0 (socket says 172.21.15.27:52776) 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd8321f9640 1 -- 172.21.15.27:0/3208297483 learned_addr learned my addr 172.21.15.27:0/3208297483 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd8321f9640 1 -- 172.21.15.27:0/3208297483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c0b95a0 msgr2=0x7fd82c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd8321f9640 1 --2- 172.21.15.27:0/3208297483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c0b95a0 0x7fd82c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd8321f9640 1 -- 172.21.15.27:0/3208297483 >> v1:172.21.15.69:6789/0 conn(0x7fd82c15df00 legacy=0x7fd82c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.883+0000 7fd8321f9640 1 -- 172.21.15.27:0/3208297483 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c0ce470 con 0x7fd82c0b9a10 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.884+0000 7fd8321f9640 1 --2- 172.21.15.27:0/3208297483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c0b9a10 0x7fd82c0b5c40 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fd824009e50 tx=0x7fd82402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b53cc2527b1458f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.884+0000 7fd8309f6640 1 -- 172.21.15.27:0/3208297483 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82402ca70 con 0x7fd82c0b9a10 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.884+0000 7fd8309f6640 1 -- 172.21.15.27:0/3208297483 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82402cc10 con 0x7fd82c0b9a10 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.884+0000 7fd8309f6640 1 -- 172.21.15.27:0/3208297483 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240343f0 con 0x7fd82c0b9a10 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.885+0000 7fd833c5c640 1 -- 172.21.15.27:0/3208297483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c0b9a10 msgr2=0x7fd82c0b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.885+0000 7fd833c5c640 1 --2- 172.21.15.27:0/3208297483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c0b9a10 0x7fd82c0b5c40 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fd824009e50 tx=0x7fd82402bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.885+0000 7fd833c5c640 1 -- 172.21.15.27:0/3208297483 shutdown_connections 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.885+0000 7fd833c5c640 1 --2- 172.21.15.27:0/3208297483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c0b9a10 0x7fd82c0b5c40 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.885+0000 7fd833c5c640 1 --2- 172.21.15.27:0/3208297483 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c0b95a0 0x7fd82c0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.885+0000 7fd833c5c640 1 -- 172.21.15.27:0/3208297483 >> 172.21.15.27:0/3208297483 conn(0x7fd82c071960 msgr2=0x7fd82c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.885+0000 7fd833c5c640 1 -- 172.21.15.27:0/3208297483 shutdown_connections 2024-03-06T07:34:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.885+0000 7fd833c5c640 1 -- 172.21.15.27:0/3208297483 wait complete. 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.886+0000 7fd833c5c640 1 Processor -- start 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.886+0000 7fd833c5c640 1 -- start start 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd833c5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c0b95a0 0x7fd82c0c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd833c5c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c0b9a10 0x7fd82c1e9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd833c5c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c15df00 0x7fd82c0ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd833c5c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c0cedf0 con 0x7fd82c15df00 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd833c5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c0cec70 con 0x7fd82c0b95a0 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd833c5c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c0d3220 con 0x7fd82c0b9a10 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8321f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c15df00 0x7fd82c0ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8321f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c15df00 0x7fd82c0ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52792/0 (socket says 172.21.15.27:52792) 2024-03-06T07:34:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8321f9640 1 -- 172.21.15.27:0/2255914145 learned_addr learned my addr 172.21.15.27:0/2255914145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8311f7640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c0b9a10 0x7fd82c1e9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8319f8640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c0b95a0 0x7fd82c0c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8321f9640 1 -- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c0b9a10 msgr2=0x7fd82c1e9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8321f9640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c0b9a10 0x7fd82c1e9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8321f9640 1 -- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c0b95a0 msgr2=0x7fd82c0c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8321f9640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c0b95a0 0x7fd82c0c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.887+0000 7fd8321f9640 1 -- 172.21.15.27:0/2255914145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c1e77f0 con 0x7fd82c15df00 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.888+0000 7fd8321f9640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c15df00 0x7fd82c0ccd30 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fd824036040 tx=0x7fd824004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.888+0000 7fd8227fc640 1 -- 172.21.15.27:0/2255914145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824043020 con 0x7fd82c15df00 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.888+0000 7fd8227fc640 1 -- 172.21.15.27:0/2255914145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82403e450 con 0x7fd82c15df00 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.888+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82c0cd270 con 0x7fd82c15df00 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.888+0000 7fd8227fc640 1 -- 172.21.15.27:0/2255914145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824009510 con 0x7fd82c15df00 2024-03-06T07:34:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.888+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82c0cd7f0 con 0x7fd82c15df00 2024-03-06T07:34:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.889+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82c067ff0 con 0x7fd82c15df00 2024-03-06T07:34:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.890+0000 7fd8227fc640 1 -- 172.21.15.27:0/2255914145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd824046020 con 0x7fd82c15df00 2024-03-06T07:34:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.891+0000 7fd8227fc640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b4d0 0x7fd81006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.891+0000 7fd8227fc640 1 -- 172.21.15.27:0/2255914145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd8240aca20 con 0x7fd82c15df00 2024-03-06T07:34:52.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.891+0000 7fd8319f8640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b4d0 0x7fd81006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.891+0000 7fd8319f8640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b4d0 0x7fd81006d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd828006fd0 tx=0x7fd828009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:52.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:52.895+0000 7fd8227fc640 1 -- 172.21.15.27:0/2255914145 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd82404d030 con 0x7fd82c15df00 2024-03-06T07:34:53.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.114+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd82c0cfae0 con 0x7fd82c15df00 2024-03-06T07:34:53.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.115+0000 7fd8227fc640 1 -- 172.21.15.27:0/2255914145 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fd82407b880 con 0x7fd82c15df00 2024-03-06T07:34:53.117 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:53.117 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:53.118 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:53.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b4d0 msgr2=0x7fd81006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b4d0 0x7fd81006d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd828006fd0 tx=0x7fd828009060 comp rx=0 tx=0).stop 2024-03-06T07:34:53.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c15df00 msgr2=0x7fd82c0ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c15df00 0x7fd82c0ccd30 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fd824036040 tx=0x7fd824004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:53.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 shutdown_connections 2024-03-06T07:34:53.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd81006b4d0 0x7fd81006d990 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd82c15df00 0x7fd82c0ccd30 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd82c0b9a10 0x7fd82c1e9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 --2- 172.21.15.27:0/2255914145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd82c0b95a0 0x7fd82c0c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 >> 172.21.15.27:0/2255914145 conn(0x7fd82c071960 msgr2=0x7fd82c15bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:53.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 shutdown_connections 2024-03-06T07:34:53.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:53.117+0000 7fd833c5c640 1 -- 172.21.15.27:0/2255914145 wait complete. 2024-03-06T07:34:53.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:53.332 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:54.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:54.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.438+0000 7fc0356a4640 1 Processor -- start 2024-03-06T07:34:54.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.438+0000 7fc0356a4640 1 -- start start 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.438+0000 7fc0356a4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0300b95a0 0x7fc0300b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.438+0000 7fc0356a4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0300cea30 con 0x7fc0300b9a10 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.438+0000 7fc0356a4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0300cf3c0 con 0x7fc03015df00 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.438+0000 7fc0356a4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0300cf0e0 con 0x7fc0300b95a0 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc0300b9a10 0x7fc0300d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39876/0 (socket says 172.21.15.27:39876) 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02f7fe640 1 -- 172.21.15.27:0/1618539696 learned_addr learned my addr 172.21.15.27:0/1618539696 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02e7fc640 1 --2- 172.21.15.27:0/1618539696 >> v2:172.21.15.69:3300/0 conn(0x7fc0300b95a0 0x7fc0300b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02dffb640 1 -- 172.21.15.27:0/1618539696 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619003375 0 0) 0x7fc0300cea30 con 0x7fc0300b9a10 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02dffb640 1 -- 172.21.15.27:0/1618539696 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc01c003520 con 0x7fc0300b9a10 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02dffb640 1 -- 172.21.15.27:0/1618539696 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119506151 0 0) 0x7fc0300cf3c0 con 0x7fc03015df00 2024-03-06T07:34:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02dffb640 1 -- 172.21.15.27:0/1618539696 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0300cea30 con 0x7fc03015df00 2024-03-06T07:34:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02dffb640 1 -- 172.21.15.27:0/1618539696 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265518500 0 0) 0x7fc01c003520 con 0x7fc0300b9a10 2024-03-06T07:34:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02e7fc640 1 -- 172.21.15.27:0/1618539696 >> v1:172.21.15.69:6789/0 conn(0x7fc03015df00 legacy=0x7fc0300d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02e7fc640 1 -- 172.21.15.27:0/1618539696 >> v1:172.21.15.27:6789/0 conn(0x7fc0300b9a10 legacy=0x7fc0300d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.439+0000 7fc02e7fc640 1 -- 172.21.15.27:0/1618539696 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300ce470 con 0x7fc0300b95a0 2024-03-06T07:34:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.440+0000 7fc02e7fc640 1 --2- 172.21.15.27:0/1618539696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0300b95a0 0x7fc0300b5700 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc0300cf590 tx=0x7fc01802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6efe7dba501626f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.440+0000 7fc02dffb640 1 -- 172.21.15.27:0/1618539696 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018039070 con 0x7fc0300b95a0 2024-03-06T07:34:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.440+0000 7fc02dffb640 1 -- 172.21.15.27:0/1618539696 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0180314f0 con 0x7fc0300b95a0 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.440+0000 7fc02dffb640 1 -- 172.21.15.27:0/1618539696 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020001670 con 0x7fc0300b95a0 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.441+0000 7fc0356a4640 1 -- 172.21.15.27:0/1618539696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0300b95a0 msgr2=0x7fc0300b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.441+0000 7fc0356a4640 1 --2- 172.21.15.27:0/1618539696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0300b95a0 0x7fc0300b5700 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc0300cf590 tx=0x7fc01802b810 comp rx=0 tx=0).stop 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.441+0000 7fc0356a4640 1 -- 172.21.15.27:0/1618539696 shutdown_connections 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.441+0000 7fc0356a4640 1 --2- 172.21.15.27:0/1618539696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0300b95a0 0x7fc0300b5700 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.441+0000 7fc0356a4640 1 -- 172.21.15.27:0/1618539696 >> 172.21.15.27:0/1618539696 conn(0x7fc030071960 msgr2=0x7fc030071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.441+0000 7fc0356a4640 1 -- 172.21.15.27:0/1618539696 shutdown_connections 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.441+0000 7fc0356a4640 1 -- 172.21.15.27:0/1618539696 wait complete. 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.442+0000 7fc0356a4640 1 Processor -- start 2024-03-06T07:34:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.442+0000 7fc0356a4640 1 -- start start 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc0356a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0300b95a0 0x7fc0300cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc0356a4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0300b9a10 0x7fc030157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc0356a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03015df00 0x7fc0300c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc0356a4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300cedf0 con 0x7fc0300b95a0 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc0356a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300cec70 con 0x7fc03015df00 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc0356a4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300b5c40 con 0x7fc0300b9a10 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0300b95a0 0x7fc0300cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0300b95a0 0x7fc0300cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38554/0 (socket says 172.21.15.27:38554) 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02effd640 1 -- 172.21.15.27:0/2464132912 learned_addr learned my addr 172.21.15.27:0/2464132912 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02f7fe640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03015df00 0x7fc0300c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02effd640 1 -- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0300b9a10 msgr2=0x7fc030157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02effd640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0300b9a10 0x7fc030157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02effd640 1 -- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03015df00 msgr2=0x7fc0300c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02effd640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03015df00 0x7fc0300c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02effd640 1 -- 172.21.15.27:0/2464132912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300cc6a0 con 0x7fc0300b95a0 2024-03-06T07:34:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.443+0000 7fc02effd640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0300b95a0 0x7fc0300cd740 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fc02800c7d0 tx=0x7fc02800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.444+0000 7fc0277fe640 1 -- 172.21.15.27:0/2464132912 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028003b70 con 0x7fc0300b95a0 2024-03-06T07:34:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.444+0000 7fc0277fe640 1 -- 172.21.15.27:0/2464132912 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc028003d30 con 0x7fc0300b95a0 2024-03-06T07:34:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.444+0000 7fc0277fe640 1 -- 172.21.15.27:0/2464132912 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02800aa30 con 0x7fc0300b95a0 2024-03-06T07:34:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.444+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0300cc980 con 0x7fc0300b95a0 2024-03-06T07:34:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.444+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0300ccf40 con 0x7fc0300b95a0 2024-03-06T07:34:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.445+0000 7fc0277fe640 1 -- 172.21.15.27:0/2464132912 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc02800abd0 con 0x7fc0300b95a0 2024-03-06T07:34:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.445+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc030067ff0 con 0x7fc0300b95a0 2024-03-06T07:34:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.446+0000 7fc0277fe640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc01006b4d0 0x7fc01006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.446+0000 7fc0277fe640 1 -- 172.21.15.27:0/2464132912 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc02808a290 con 0x7fc0300b95a0 2024-03-06T07:34:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.446+0000 7fc02e7fc640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc01006b4d0 0x7fc01006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.446+0000 7fc02e7fc640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc01006b4d0 0x7fc01006d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fc018036000 tx=0x7fc018030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.450+0000 7fc0277fe640 1 -- 172.21.15.27:0/2464132912 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc028059050 con 0x7fc0300b95a0 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.759+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0300b2710 con 0x7fc0300b95a0 2024-03-06T07:34:54.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.759+0000 7fc0277fe640 1 -- 172.21.15.27:0/2464132912 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fc028058950 con 0x7fc0300b95a0 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc01006b4d0 msgr2=0x7fc01006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc01006b4d0 0x7fc01006d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fc018036000 tx=0x7fc018030000 comp rx=0 tx=0).stop 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0300b95a0 msgr2=0x7fc0300cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0300b95a0 0x7fc0300cd740 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fc02800c7d0 tx=0x7fc02800cc90 comp rx=0 tx=0).stop 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 shutdown_connections 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc01006b4d0 0x7fc01006d990 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc03015df00 0x7fc0300c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0300b9a10 0x7fc030157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 --2- 172.21.15.27:0/2464132912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0300b95a0 0x7fc0300cd740 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 >> 172.21.15.27:0/2464132912 conn(0x7fc030071960 msgr2=0x7fc0300b6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 shutdown_connections 2024-03-06T07:34:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:54.761+0000 7fc0356a4640 1 -- 172.21.15.27:0/2464132912 wait complete. 2024-03-06T07:34:54.976 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:54.976 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:55.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:56.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.084+0000 7fa98ad79640 1 Processor -- start 2024-03-06T07:34:56.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.084+0000 7fa98ad79640 1 -- start start 2024-03-06T07:34:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa98ad79640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 0x7fa9840d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa98ad79640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9840b95a0 0x7fa9840b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa98ad79640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa9840b9a30 0x7fa9840b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa98ad79640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9840cea30 con 0x7fa9840b95a0 2024-03-06T07:34:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa98ad79640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9840d2af0 con 0x7fa9840b9a30 2024-03-06T07:34:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa98ad79640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840cf3c0 con 0x7fa98415df00 2024-03-06T07:34:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa983fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9840b95a0 0x7fa9840b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa983fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9840b95a0 0x7fa9840b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38562/0 (socket says 172.21.15.27:38562) 2024-03-06T07:34:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa983fff640 1 -- 172.21.15.27:0/3765726842 learned_addr learned my addr 172.21.15.27:0/3765726842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa988b15640 1 --2- 172.21.15.27:0/3765726842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 0x7fa9840d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.085+0000 7fa989316640 1 --2- 172.21.15.27:0/3765726842 >> v2:172.21.15.69:3300/0 conn(0x7fa9840b9a30 0x7fa9840b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.086+0000 7fa983fff640 1 -- 172.21.15.27:0/3765726842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 msgr2=0x7fa9840d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.086+0000 7fa983fff640 1 --2- 172.21.15.27:0/3765726842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 0x7fa9840d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.086+0000 7fa983fff640 1 -- 172.21.15.27:0/3765726842 >> v2:172.21.15.69:3300/0 conn(0x7fa9840b9a30 msgr2=0x7fa9840b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.086+0000 7fa983fff640 1 --2- 172.21.15.27:0/3765726842 >> v2:172.21.15.69:3300/0 conn(0x7fa9840b9a30 0x7fa9840b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.086+0000 7fa983fff640 1 -- 172.21.15.27:0/3765726842 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9840cf0e0 con 0x7fa9840b95a0 2024-03-06T07:34:56.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.086+0000 7fa983fff640 1 --2- 172.21.15.27:0/3765726842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9840b95a0 0x7fa9840b5700 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fa9840d1160 tx=0x7fa97002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c779a6d3e98c1262 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.086+0000 7fa9837fe640 1 -- 172.21.15.27:0/3765726842 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970035020 con 0x7fa9840b95a0 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa9837fe640 1 -- 172.21.15.27:0/3765726842 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa970031870 con 0x7fa9840b95a0 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa9837fe640 1 -- 172.21.15.27:0/3765726842 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970031c40 con 0x7fa9840b95a0 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa98ad79640 1 -- 172.21.15.27:0/3765726842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9840b95a0 msgr2=0x7fa9840b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa98ad79640 1 --2- 172.21.15.27:0/3765726842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9840b95a0 0x7fa9840b5700 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fa9840d1160 tx=0x7fa97002b910 comp rx=0 tx=0).stop 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa98ad79640 1 -- 172.21.15.27:0/3765726842 shutdown_connections 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa98ad79640 1 --2- 172.21.15.27:0/3765726842 >> v2:172.21.15.69:3300/0 conn(0x7fa9840b9a30 0x7fa9840b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa98ad79640 1 --2- 172.21.15.27:0/3765726842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9840b95a0 0x7fa9840b5700 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa98ad79640 1 --2- 172.21.15.27:0/3765726842 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 0x7fa9840d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa98ad79640 1 -- 172.21.15.27:0/3765726842 >> 172.21.15.27:0/3765726842 conn(0x7fa984071960 msgr2=0x7fa984071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa98ad79640 1 -- 172.21.15.27:0/3765726842 shutdown_connections 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.087+0000 7fa98ad79640 1 -- 172.21.15.27:0/3765726842 wait complete. 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.088+0000 7fa98ad79640 1 Processor -- start 2024-03-06T07:34:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.088+0000 7fa98ad79640 1 -- start start 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.089+0000 7fa98ad79640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9840b95a0 0x7fa984160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.089+0000 7fa98ad79640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 0x7fa9840b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.089+0000 7fa98ad79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa98414d8d0 0x7fa98414dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.089+0000 7fa98ad79640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840cec70 con 0x7fa98414d8d0 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.089+0000 7fa98ad79640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840ce750 con 0x7fa9840b95a0 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.089+0000 7fa98ad79640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840cedf0 con 0x7fa98415df00 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.089+0000 7fa989316640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa98414d8d0 0x7fa98414dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.089+0000 7fa989316640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa98414d8d0 0x7fa98414dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38570/0 (socket says 172.21.15.27:38570) 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.089+0000 7fa989316640 1 -- 172.21.15.27:0/936810632 learned_addr learned my addr 172.21.15.27:0/936810632 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa983fff640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 0x7fa9840b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa988b15640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9840b95a0 0x7fa984160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa989316640 1 -- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 msgr2=0x7fa9840b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa989316640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 0x7fa9840b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa989316640 1 -- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9840b95a0 msgr2=0x7fa984160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa989316640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9840b95a0 0x7fa984160b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa989316640 1 -- 172.21.15.27:0/936810632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9840c8cf0 con 0x7fa98414d8d0 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa983fff640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 0x7fa9840b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa989316640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa98414d8d0 0x7fa98414dd60 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007e30 tx=0x7fa97c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa9817fa640 1 -- 172.21.15.27:0/936810632 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c014620 con 0x7fa98414d8d0 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa9817fa640 1 -- 172.21.15.27:0/936810632 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c019070 con 0x7fa98414d8d0 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa9817fa640 1 -- 172.21.15.27:0/936810632 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c014a00 con 0x7fa98414d8d0 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9840c8fd0 con 0x7fa98414d8d0 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.090+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9840c9870 con 0x7fa98414d8d0 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.091+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9840c93c0 con 0x7fa98414d8d0 2024-03-06T07:34:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.091+0000 7fa9817fa640 1 -- 172.21.15.27:0/936810632 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa97c014ba0 con 0x7fa98414d8d0 2024-03-06T07:34:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.092+0000 7fa9817fa640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa96806b4d0 0x7fa96806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.092+0000 7fa9817fa640 1 -- 172.21.15.27:0/936810632 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa97c08aba0 con 0x7fa98414d8d0 2024-03-06T07:34:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.092+0000 7fa988b15640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa96806b4d0 0x7fa96806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.093+0000 7fa988b15640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa96806b4d0 0x7fa96806d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa978006fd0 tx=0x7fa978008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.097+0000 7fa9817fa640 1 -- 172.21.15.27:0/936810632 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa97c059970 con 0x7fa98414d8d0 2024-03-06T07:34:56.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.387+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa984067ff0 con 0x7fa98414d8d0 2024-03-06T07:34:56.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.388+0000 7fa9817fa640 1 -- 172.21.15.27:0/936810632 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fa97c059270 con 0x7fa98414d8d0 2024-03-06T07:34:56.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:56.391 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:56.391 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa96806b4d0 msgr2=0x7fa96806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa96806b4d0 0x7fa96806d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa978006fd0 tx=0x7fa978008040 comp rx=0 tx=0).stop 2024-03-06T07:34:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa98414d8d0 msgr2=0x7fa98414dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa98414d8d0 0x7fa98414dd60 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007e30 tx=0x7fa97c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 shutdown_connections 2024-03-06T07:34:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa96806b4d0 0x7fa96806d990 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa98414d8d0 0x7fa98414dd60 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa98415df00 0x7fa9840b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 --2- 172.21.15.27:0/936810632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9840b95a0 0x7fa984160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 >> 172.21.15.27:0/936810632 conn(0x7fa984071960 msgr2=0x7fa9840cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 shutdown_connections 2024-03-06T07:34:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:56.390+0000 7fa98ad79640 1 -- 172.21.15.27:0/936810632 wait complete. 2024-03-06T07:34:56.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:56.606 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:57.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:57.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.711+0000 7f93f3177640 1 Processor -- start 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.711+0000 7f93f3177640 1 -- start start 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f3177640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f93ec15df00 0x7f93ec0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f3177640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f93ec0b9a30 0x7f93ec0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f3177640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec0cea30 con 0x7f93ec0b95a0 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f3177640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec0d2af0 con 0x7f93ec15df00 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f3177640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec0cf3c0 con 0x7f93ec0b9a30 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f0f13640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f93ec15df00 0x7f93ec0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f0f13640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f93ec15df00 0x7f93ec0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38578/0 (socket says 172.21.15.27:38578) 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f0f13640 1 -- 172.21.15.27:0/2569071817 learned_addr learned my addr 172.21.15.27:0/2569071817 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f1714640 1 --2- 172.21.15.27:0/2569071817 >> v2:172.21.15.69:3300/0 conn(0x7f93ec0b9a30 0x7f93ec0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f0f13640 1 -- 172.21.15.27:0/2569071817 >> v2:172.21.15.69:3300/0 conn(0x7f93ec0b9a30 msgr2=0x7f93ec0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f0f13640 1 --2- 172.21.15.27:0/2569071817 >> v2:172.21.15.69:3300/0 conn(0x7f93ec0b9a30 0x7f93ec0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f0f13640 1 -- 172.21.15.27:0/2569071817 >> v1:172.21.15.69:6789/0 conn(0x7f93ec0b95a0 legacy=0x7f93ec0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.712+0000 7f93f0f13640 1 -- 172.21.15.27:0/2569071817 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec0cf0e0 con 0x7f93ec15df00 2024-03-06T07:34:57.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.713+0000 7f93f0f13640 1 --2- 172.21.15.27:0/2569071817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15df00 0x7f93ec0d25b0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f93ec0d4d20 tx=0x7f93e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddecd3f257039be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.713+0000 7f93eb7fe640 1 -- 172.21.15.27:0/2569071817 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0035020 con 0x7f93ec15df00 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.713+0000 7f93eb7fe640 1 -- 172.21.15.27:0/2569071817 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e0031870 con 0x7f93ec15df00 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.713+0000 7f93eb7fe640 1 -- 172.21.15.27:0/2569071817 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0031c40 con 0x7f93ec15df00 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.714+0000 7f93f3177640 1 -- 172.21.15.27:0/2569071817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15df00 msgr2=0x7f93ec0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.714+0000 7f93f3177640 1 --2- 172.21.15.27:0/2569071817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15df00 0x7f93ec0d25b0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f93ec0d4d20 tx=0x7f93e002b910 comp rx=0 tx=0).stop 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.714+0000 7f93f3177640 1 -- 172.21.15.27:0/2569071817 shutdown_connections 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.714+0000 7f93f3177640 1 --2- 172.21.15.27:0/2569071817 >> v2:172.21.15.69:3300/0 conn(0x7f93ec0b9a30 0x7f93ec0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.714+0000 7f93f3177640 1 --2- 172.21.15.27:0/2569071817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15df00 0x7f93ec0d25b0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.714+0000 7f93f3177640 1 -- 172.21.15.27:0/2569071817 >> 172.21.15.27:0/2569071817 conn(0x7f93ec071960 msgr2=0x7f93ec071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.714+0000 7f93f3177640 1 -- 172.21.15.27:0/2569071817 shutdown_connections 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.714+0000 7f93f3177640 1 -- 172.21.15.27:0/2569071817 wait complete. 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.715+0000 7f93f3177640 1 Processor -- start 2024-03-06T07:34:57.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.715+0000 7f93f3177640 1 -- start start 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f3177640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0b95a0 0x7f93ec15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f3177640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93ec15df00 0x7f93ec0c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f3177640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15f0b0 0x7f93ec0c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f3177640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0cec70 con 0x7f93ec15f0b0 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f3177640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0ce750 con 0x7f93ec0b95a0 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f3177640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0cedf0 con 0x7f93ec15df00 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f1714640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15f0b0 0x7f93ec0c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f1714640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15f0b0 0x7f93ec0c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38586/0 (socket says 172.21.15.27:38586) 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f1714640 1 -- 172.21.15.27:0/1991106681 learned_addr learned my addr 172.21.15.27:0/1991106681 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f0f13640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0b95a0 0x7f93ec15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93ebfff640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93ec15df00 0x7f93ec0c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f1714640 1 -- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93ec15df00 msgr2=0x7f93ec0c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f1714640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93ec15df00 0x7f93ec0c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f1714640 1 -- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0b95a0 msgr2=0x7f93ec15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f1714640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0b95a0 0x7f93ec15aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f1714640 1 -- 172.21.15.27:0/1991106681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec0c7930 con 0x7f93ec15f0b0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93ebfff640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93ec15df00 0x7f93ec0c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.716+0000 7f93f1714640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15f0b0 0x7f93ec0c9c40 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f93e4007e40 tx=0x7f93e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.717+0000 7f93e97fa640 1 -- 172.21.15.27:0/1991106681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4014620 con 0x7f93ec15f0b0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.717+0000 7f93e97fa640 1 -- 172.21.15.27:0/1991106681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e4019070 con 0x7f93ec15f0b0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.717+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec0c7c10 con 0x7f93ec15f0b0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.717+0000 7f93e97fa640 1 -- 172.21.15.27:0/1991106681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4014a20 con 0x7f93ec15f0b0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.717+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec15fc70 con 0x7f93ec15f0b0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.718+0000 7f93e97fa640 1 -- 172.21.15.27:0/1991106681 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93e4014bc0 con 0x7f93ec15f0b0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.718+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec15fe70 con 0x7f93ec15f0b0 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.719+0000 7f93e97fa640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f93d006b4d0 0x7f93d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.719+0000 7f93e97fa640 1 -- 172.21.15.27:0/1991106681 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f93e408ab60 con 0x7f93ec15f0b0 2024-03-06T07:34:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.719+0000 7f93f0f13640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f93d006b4d0 0x7f93d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.720+0000 7f93f0f13640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f93d006b4d0 0x7f93d006d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f93e0036000 tx=0x7f93e003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:57.723+0000 7f93e97fa640 1 -- 172.21.15.27:0/1991106681 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93ec15fe70 con 0x7f93ec15f0b0 2024-03-06T07:34:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.022+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93ec0c7de0 con 0x7f93ec15f0b0 2024-03-06T07:34:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.023+0000 7f93e97fa640 1 -- 172.21.15.27:0/1991106681 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f93ec0c7de0 con 0x7f93ec15f0b0 2024-03-06T07:34:58.025 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:58.025 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f93d006b4d0 msgr2=0x7f93d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f93d006b4d0 0x7f93d006d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f93e0036000 tx=0x7f93e003c040 comp rx=0 tx=0).stop 2024-03-06T07:34:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15f0b0 msgr2=0x7f93ec0c9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15f0b0 0x7f93ec0c9c40 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f93e4007e40 tx=0x7f93e400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 shutdown_connections 2024-03-06T07:34:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f93d006b4d0 0x7f93d006d990 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93ec15f0b0 0x7f93ec0c9c40 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f93ec15df00 0x7f93ec0c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 --2- 172.21.15.27:0/1991106681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0b95a0 0x7f93ec15aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 >> 172.21.15.27:0/1991106681 conn(0x7f93ec071960 msgr2=0x7f93ec15bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 shutdown_connections 2024-03-06T07:34:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:58.025+0000 7f93f3177640 1 -- 172.21.15.27:0/1991106681 wait complete. 2024-03-06T07:34:58.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:58.240 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:34:59.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.346+0000 7ff1923e9640 1 Processor -- start 2024-03-06T07:34:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.346+0000 7ff1923e9640 1 -- start start 2024-03-06T07:34:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.347+0000 7ff1923e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff18c0b9a10 0x7ff18c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.347+0000 7ff1923e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0cea30 con 0x7ff18c15df00 2024-03-06T07:34:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.347+0000 7ff1923e9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0cf3c0 con 0x7ff18c0b95a0 2024-03-06T07:34:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.347+0000 7ff1923e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c0cf0e0 con 0x7ff18c0b9a10 2024-03-06T07:34:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.347+0000 7ff18bfff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff18c15df00 0x7ff18c0d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39886/0 (socket says 172.21.15.27:39886) 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.347+0000 7ff18bfff640 1 -- 172.21.15.27:0/1918425289 learned_addr learned my addr 172.21.15.27:0/1918425289 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.347+0000 7ff190986640 1 --2- 172.21.15.27:0/1918425289 >> v2:172.21.15.69:3300/0 conn(0x7ff18c0b9a10 0x7ff18c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.348+0000 7ff18affd640 1 -- 172.21.15.27:0/1918425289 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910364364 0 0) 0x7ff18c0cea30 con 0x7ff18c15df00 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.348+0000 7ff18affd640 1 -- 172.21.15.27:0/1918425289 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff174003520 con 0x7ff18c15df00 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.348+0000 7ff18affd640 1 -- 172.21.15.27:0/1918425289 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729994423 0 0) 0x7ff18c0cf3c0 con 0x7ff18c0b95a0 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.348+0000 7ff18affd640 1 -- 172.21.15.27:0/1918425289 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff18c0cea30 con 0x7ff18c0b95a0 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.348+0000 7ff190986640 1 -- 172.21.15.27:0/1918425289 >> v1:172.21.15.69:6789/0 conn(0x7ff18c0b95a0 legacy=0x7ff18c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.348+0000 7ff190986640 1 -- 172.21.15.27:0/1918425289 >> v1:172.21.15.27:6789/0 conn(0x7ff18c15df00 legacy=0x7ff18c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.348+0000 7ff18affd640 1 -- 172.21.15.27:0/1918425289 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3640311789 0 0) 0x7ff174003520 con 0x7ff18c15df00 2024-03-06T07:34:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.348+0000 7ff190986640 1 -- 172.21.15.27:0/1918425289 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0ce470 con 0x7ff18c0b9a10 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.349+0000 7ff190986640 1 --2- 172.21.15.27:0/1918425289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff18c0b9a10 0x7ff18c0d3220 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff18c0cf590 tx=0x7ff18402be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff731f334042378d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.349+0000 7ff18affd640 1 -- 172.21.15.27:0/1918425289 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184039070 con 0x7ff18c0b9a10 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.349+0000 7ff18affd640 1 -- 172.21.15.27:0/1918425289 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18402c930 con 0x7ff18c0b9a10 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.349+0000 7ff18affd640 1 -- 172.21.15.27:0/1918425289 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180001670 con 0x7ff18c0b9a10 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.349+0000 7ff1923e9640 1 -- 172.21.15.27:0/1918425289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff18c0b9a10 msgr2=0x7ff18c0d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.349+0000 7ff1923e9640 1 --2- 172.21.15.27:0/1918425289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff18c0b9a10 0x7ff18c0d3220 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff18c0cf590 tx=0x7ff18402be00 comp rx=0 tx=0).stop 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.349+0000 7ff1923e9640 1 -- 172.21.15.27:0/1918425289 shutdown_connections 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.349+0000 7ff1923e9640 1 --2- 172.21.15.27:0/1918425289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff18c0b9a10 0x7ff18c0d3220 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.349+0000 7ff1923e9640 1 -- 172.21.15.27:0/1918425289 >> 172.21.15.27:0/1918425289 conn(0x7ff18c071960 msgr2=0x7ff18c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.350+0000 7ff1923e9640 1 -- 172.21.15.27:0/1918425289 shutdown_connections 2024-03-06T07:34:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.350+0000 7ff1923e9640 1 -- 172.21.15.27:0/1918425289 wait complete. 2024-03-06T07:34:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.351+0000 7ff1923e9640 1 Processor -- start 2024-03-06T07:34:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.351+0000 7ff1923e9640 1 -- start start 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff1923e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff18c0b95a0 0x7ff18c1f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff1923e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18c0b9a10 0x7ff18c1618b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff1923e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff18c15df00 0x7ff18c161df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff1923e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0cedf0 con 0x7ff18c0b9a10 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff1923e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0cec70 con 0x7ff18c15df00 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff1923e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0b5e10 con 0x7ff18c0b95a0 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18c0b9a10 0x7ff18c1618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18c0b9a10 0x7ff18c1618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38600/0 (socket says 172.21.15.27:38600) 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18b7fe640 1 -- 172.21.15.27:0/2819017853 learned_addr learned my addr 172.21.15.27:0/2819017853 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18bfff640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff18c0b95a0 0x7ff18c1f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff190986640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff18c15df00 0x7ff18c161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18b7fe640 1 -- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff18c0b95a0 msgr2=0x7ff18c1f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18b7fe640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff18c0b95a0 0x7ff18c1f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18b7fe640 1 -- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff18c15df00 msgr2=0x7ff18c161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18b7fe640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff18c15df00 0x7ff18c161df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18b7fe640 1 -- 172.21.15.27:0/2819017853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c1f45e0 con 0x7ff18c0b9a10 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff18b7fe640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18c0b9a10 0x7ff18c1618b0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00c4c0 tx=0x7ff17c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff188ff9640 1 -- 172.21.15.27:0/2819017853 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0052d0 con 0x7ff18c0b9a10 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff188ff9640 1 -- 172.21.15.27:0/2819017853 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c005470 con 0x7ff18c0b9a10 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff188ff9640 1 -- 172.21.15.27:0/2819017853 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c012870 con 0x7ff18c0b9a10 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.352+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c1f48c0 con 0x7ff18c0b9a10 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.353+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c0cc6a0 con 0x7ff18c0b9a10 2024-03-06T07:34:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.353+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c067ff0 con 0x7ff18c0b9a10 2024-03-06T07:34:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.354+0000 7ff188ff9640 1 -- 172.21.15.27:0/2819017853 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff17c012a10 con 0x7ff18c0b9a10 2024-03-06T07:34:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.354+0000 7ff188ff9640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff17806b500 0x7ff17806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.354+0000 7ff188ff9640 1 -- 172.21.15.27:0/2819017853 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff17c089ea0 con 0x7ff18c0b9a10 2024-03-06T07:34:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.355+0000 7ff18bfff640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff17806b500 0x7ff17806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.355+0000 7ff18bfff640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff17806b500 0x7ff17806d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff1800011e0 tx=0x7ff180009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:59.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.359+0000 7ff188ff9640 1 -- 172.21.15.27:0/2819017853 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff17c058d00 con 0x7ff18c0b9a10 2024-03-06T07:34:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.578+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff18c15d4d0 con 0x7ff18c0b9a10 2024-03-06T07:34:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.578+0000 7ff188ff9640 1 -- 172.21.15.27:0/2819017853 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7ff17c058600 con 0x7ff18c0b9a10 2024-03-06T07:34:59.581 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:34:59.581 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:34:59.581 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff17806b500 msgr2=0x7ff17806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff17806b500 0x7ff17806d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff1800011e0 tx=0x7ff180009040 comp rx=0 tx=0).stop 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18c0b9a10 msgr2=0x7ff18c1618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18c0b9a10 0x7ff18c1618b0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00c4c0 tx=0x7ff17c00c980 comp rx=0 tx=0).stop 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 shutdown_connections 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff17806b500 0x7ff17806d9c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff18c15df00 0x7ff18c161df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18c0b9a10 0x7ff18c1618b0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 --2- 172.21.15.27:0/2819017853 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff18c0b95a0 0x7ff18c1f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:59.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 >> 172.21.15.27:0/2819017853 conn(0x7ff18c071960 msgr2=0x7ff18c0d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:59.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 shutdown_connections 2024-03-06T07:34:59.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:34:59.580+0000 7ff1923e9640 1 -- 172.21.15.27:0/2819017853 wait complete. 2024-03-06T07:34:59.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:34:59.794 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:00.795 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:00.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.900+0000 7fe229dee640 1 Processor -- start 2024-03-06T07:35:00.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.900+0000 7fe229dee640 1 -- start start 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe229dee640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2240b95a0 0x7fe2240b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe229dee640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2240b9a10 0x7fe2240b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe229dee640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2240cea30 con 0x7fe22415df00 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe229dee640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2240cf3c0 con 0x7fe2240b95a0 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe229dee640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2240cf0e0 con 0x7fe2240b9a10 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe222ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2240b95a0 0x7fe2240b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe2237fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe22415df00 0x7fe2240d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39892/0 (socket says 172.21.15.27:39892) 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe222ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2240b95a0 0x7fe2240b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38604/0 (socket says 172.21.15.27:38604) 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe2237fe640 1 -- 172.21.15.27:0/1287972360 learned_addr learned my addr 172.21.15.27:0/1287972360 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe223fff640 1 --2- 172.21.15.27:0/1287972360 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2240b9a10 0x7fe2240b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe222ffd640 1 -- 172.21.15.27:0/1287972360 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2240b9a10 msgr2=0x7fe2240b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe222ffd640 1 --2- 172.21.15.27:0/1287972360 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2240b9a10 0x7fe2240b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe222ffd640 1 -- 172.21.15.27:0/1287972360 >> v1:172.21.15.27:6789/0 conn(0x7fe22415df00 legacy=0x7fe2240d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.901+0000 7fe222ffd640 1 -- 172.21.15.27:0/1287972360 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2240ce470 con 0x7fe2240b95a0 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.902+0000 7fe222ffd640 1 --2- 172.21.15.27:0/1287972360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b95a0 0x7fe2240b5700 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fe2140097a0 tx=0x7fe21402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fadecd800793ba77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.902+0000 7fe2227fc640 1 -- 172.21.15.27:0/1287972360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214035020 con 0x7fe2240b95a0 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.902+0000 7fe2227fc640 1 -- 172.21.15.27:0/1287972360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214031830 con 0x7fe2240b95a0 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.902+0000 7fe2227fc640 1 -- 172.21.15.27:0/1287972360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214031c00 con 0x7fe2240b95a0 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.903+0000 7fe229dee640 1 -- 172.21.15.27:0/1287972360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b95a0 msgr2=0x7fe2240b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.903+0000 7fe229dee640 1 --2- 172.21.15.27:0/1287972360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b95a0 0x7fe2240b5700 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fe2140097a0 tx=0x7fe21402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.903+0000 7fe229dee640 1 -- 172.21.15.27:0/1287972360 shutdown_connections 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.903+0000 7fe229dee640 1 --2- 172.21.15.27:0/1287972360 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2240b9a10 0x7fe2240b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.903+0000 7fe229dee640 1 --2- 172.21.15.27:0/1287972360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b95a0 0x7fe2240b5700 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.903+0000 7fe229dee640 1 -- 172.21.15.27:0/1287972360 >> 172.21.15.27:0/1287972360 conn(0x7fe224071960 msgr2=0x7fe224071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.903+0000 7fe229dee640 1 -- 172.21.15.27:0/1287972360 shutdown_connections 2024-03-06T07:35:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.903+0000 7fe229dee640 1 -- 172.21.15.27:0/1287972360 wait complete. 2024-03-06T07:35:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.905+0000 7fe229dee640 1 Processor -- start 2024-03-06T07:35:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.905+0000 7fe229dee640 1 -- start start 2024-03-06T07:35:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.906+0000 7fe229dee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2240b95a0 0x7fe22415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.906+0000 7fe229dee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b9a10 0x7fe2240cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.906+0000 7fe229dee640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe22415df00 0x7fe22414b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.906+0000 7fe229dee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2240cedf0 con 0x7fe2240b9a10 2024-03-06T07:35:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.906+0000 7fe229dee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2240cec70 con 0x7fe2240b95a0 2024-03-06T07:35:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.906+0000 7fe229dee640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2240d3220 con 0x7fe22415df00 2024-03-06T07:35:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.906+0000 7fe222ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b9a10 0x7fe2240cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.906+0000 7fe222ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b9a10 0x7fe2240cd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38612/0 (socket says 172.21.15.27:38612) 2024-03-06T07:35:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.906+0000 7fe222ffd640 1 -- 172.21.15.27:0/1170135248 learned_addr learned my addr 172.21.15.27:0/1170135248 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe223fff640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe22415df00 0x7fe22414b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe2237fe640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2240b95a0 0x7fe22415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe222ffd640 1 -- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe22415df00 msgr2=0x7fe22414b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe222ffd640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe22415df00 0x7fe22414b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe222ffd640 1 -- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2240b95a0 msgr2=0x7fe22415e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe222ffd640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2240b95a0 0x7fe22415e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe222ffd640 1 -- 172.21.15.27:0/1170135248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22415ea30 con 0x7fe2240b9a10 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe222ffd640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b9a10 0x7fe2240cd5d0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fe21402bee0 tx=0x7fe2140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe20bfff640 1 -- 172.21.15.27:0/1170135248 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214041070 con 0x7fe2240b9a10 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe20bfff640 1 -- 172.21.15.27:0/1170135248 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21403b920 con 0x7fe2240b9a10 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2240cc810 con 0x7fe2240b9a10 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe20bfff640 1 -- 172.21.15.27:0/1170135248 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214035040 con 0x7fe2240b9a10 2024-03-06T07:35:00.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.907+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2240ccdd0 con 0x7fe2240b9a10 2024-03-06T07:35:00.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.909+0000 7fe20bfff640 1 -- 172.21.15.27:0/1170135248 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe21404c060 con 0x7fe2240b9a10 2024-03-06T07:35:00.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.909+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe224067ff0 con 0x7fe2240b9a10 2024-03-06T07:35:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.909+0000 7fe20bfff640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe21006b500 0x7fe21006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.909+0000 7fe20bfff640 1 -- 172.21.15.27:0/1170135248 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe2140abaf0 con 0x7fe2240b9a10 2024-03-06T07:35:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.909+0000 7fe2237fe640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe21006b500 0x7fe21006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.910+0000 7fe2237fe640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe21006b500 0x7fe21006d9c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe218007fd0 tx=0x7fe218007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:00.918+0000 7fe20bfff640 1 -- 172.21.15.27:0/1170135248 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe21407a310 con 0x7fe2240b9a10 2024-03-06T07:35:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.222+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2240c71e0 con 0x7fe2240b9a10 2024-03-06T07:35:01.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.223+0000 7fe20bfff640 1 -- 172.21.15.27:0/1170135248 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fe21407e3b0 con 0x7fe2240b9a10 2024-03-06T07:35:01.225 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:01.226 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:01.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe21006b500 msgr2=0x7fe21006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe21006b500 0x7fe21006d9c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe218007fd0 tx=0x7fe218007e80 comp rx=0 tx=0).stop 2024-03-06T07:35:01.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b9a10 msgr2=0x7fe2240cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b9a10 0x7fe2240cd5d0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fe21402bee0 tx=0x7fe2140046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 shutdown_connections 2024-03-06T07:35:01.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe21006b500 0x7fe21006d9c0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe22415df00 0x7fe22414b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2240b9a10 0x7fe2240cd5d0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 --2- 172.21.15.27:0/1170135248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2240b95a0 0x7fe22415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.225+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 >> 172.21.15.27:0/1170135248 conn(0x7fe224071960 msgr2=0x7fe2240b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.226+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 shutdown_connections 2024-03-06T07:35:01.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:01.226+0000 7fe229dee640 1 -- 172.21.15.27:0/1170135248 wait complete. 2024-03-06T07:35:01.444 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:01.445 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:02.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:02.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.553+0000 7fe209264640 1 Processor -- start 2024-03-06T07:35:02.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.553+0000 7fe209264640 1 -- start start 2024-03-06T07:35:02.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe209264640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2040b1d50 0x7fe2040b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe209264640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20415e410 con 0x7fe20415e030 2024-03-06T07:35:02.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe209264640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20406f140 con 0x7fe20406d210 2024-03-06T07:35:02.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe209264640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2040b26f0 con 0x7fe2040b1d50 2024-03-06T07:35:02.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe202ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe20415e030 0x7fe20406ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39902/0 (socket says 172.21.15.27:39902) 2024-03-06T07:35:02.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe202ffd640 1 -- 172.21.15.27:0/1984539621 learned_addr learned my addr 172.21.15.27:0/1984539621 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:02.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe1fbfff640 1 --2- 172.21.15.27:0/1984539621 >> v2:172.21.15.69:3300/0 conn(0x7fe2040b1d50 0x7fe2040b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417711899 0 0) 0x7fe1f4001110 con 0x7fe20415e030 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f0003520 con 0x7fe20415e030 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2186289488 0 0) 0x7fe1f0003520 con 0x7fe20415e030 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 >> v2:172.21.15.69:3300/0 conn(0x7fe2040b1d50 msgr2=0x7fe2040b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 --2- 172.21.15.27:0/1984539621 >> v2:172.21.15.69:3300/0 conn(0x7fe2040b1d50 0x7fe2040b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 >> v1:172.21.15.69:6789/0 conn(0x7fe20406d210 legacy=0x7fe20406d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204161af0 con 0x7fe20415e030 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fe1f4001840 con 0x7fe20415e030 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 >> v1:172.21.15.27:6789/0 conn(0x7fe20415e030 legacy=0x7fe20406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1f00039b0 0x7fe1f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.554+0000 7fe2027fc640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f0004800 0x7fe1f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe2027fc640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f00051c0 0x7fe1f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f000c030 con 0x7fe1f0004800 2024-03-06T07:35:02.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f000c1e0 con 0x7fe1f00051c0 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f000c3e0 con 0x7fe1f00039b0 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe1fbfff640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f0004800 0x7fe1f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe202ffd640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1f00039b0 0x7fe1f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe2037fe640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f00051c0 0x7fe1f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe1fbfff640 1 -- 172.21.15.27:0/1984539621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1f00039b0 msgr2=0x7fe1f0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe1fbfff640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1f00039b0 0x7fe1f0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe1fbfff640 1 -- 172.21.15.27:0/1984539621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f00051c0 msgr2=0x7fe1f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe1fbfff640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f00051c0 0x7fe1f000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe1fbfff640 1 -- 172.21.15.27:0/1984539621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f0003520 con 0x7fe1f0004800 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.555+0000 7fe2037fe640 1 -- 172.21.15.27:0/1984539621 reap_dead start 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.556+0000 7fe1fbfff640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f0004800 0x7fe1f0004c80 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009a90 tx=0x7fe1f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968174173fd5fb38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.556+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec039070 con 0x7fe1f0004800 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.556+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec02c680 con 0x7fe1f0004800 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.556+0000 7fe2027fc640 1 -- 172.21.15.27:0/1984539621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec009690 con 0x7fe1f0004800 2024-03-06T07:35:02.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.556+0000 7fe209264640 1 -- 172.21.15.27:0/1984539621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f0004800 msgr2=0x7fe1f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.556+0000 7fe209264640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f0004800 0x7fe1f0004c80 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009a90 tx=0x7fe1f0005830 comp rx=0 tx=0).stop 2024-03-06T07:35:02.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.556+0000 7fe209264640 1 -- 172.21.15.27:0/1984539621 shutdown_connections 2024-03-06T07:35:02.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.557+0000 7fe209264640 1 --2- 172.21.15.27:0/1984539621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1f0004800 0x7fe1f0004c80 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.557+0000 7fe209264640 1 -- 172.21.15.27:0/1984539621 >> 172.21.15.27:0/1984539621 conn(0x7fe2040b55e0 msgr2=0x7fe2040b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.557+0000 7fe209264640 1 -- 172.21.15.27:0/1984539621 shutdown_connections 2024-03-06T07:35:02.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.557+0000 7fe209264640 1 -- 172.21.15.27:0/1984539621 wait complete. 2024-03-06T07:35:02.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.559+0000 7fe209264640 1 Processor -- start 2024-03-06T07:35:02.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.559+0000 7fe209264640 1 -- start start 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe209264640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f0004800 0x7fe20409d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe209264640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe20409c240 0x7fe20409a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe209264640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe20409a860 0x7fe20413ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe209264640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2041620b0 con 0x7fe20409c240 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe209264640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe204162700 con 0x7fe1f0004800 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe209264640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe204162390 con 0x7fe20409a860 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe1fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe20409c240 0x7fe20409a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe1fbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe20409c240 0x7fe20409a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38632/0 (socket says 172.21.15.27:38632) 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe1fbfff640 1 -- 172.21.15.27:0/956224811 learned_addr learned my addr 172.21.15.27:0/956224811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe202ffd640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f0004800 0x7fe20409d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe2037fe640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe20409a860 0x7fe20413ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe1fbfff640 1 -- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe20409a860 msgr2=0x7fe20413ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe1fbfff640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe20409a860 0x7fe20413ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe1fbfff640 1 -- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f0004800 msgr2=0x7fe20409d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe1fbfff640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f0004800 0x7fe20409d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.560+0000 7fe1fbfff640 1 -- 172.21.15.27:0/956224811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20409acc0 con 0x7fe20409c240 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.561+0000 7fe1fbfff640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe20409c240 0x7fe20409a320 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009bb0 tx=0x7fe1ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.561+0000 7fe1fb7fe640 1 -- 172.21.15.27:0/956224811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec042020 con 0x7fe20409c240 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.561+0000 7fe1fb7fe640 1 -- 172.21.15.27:0/956224811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec03a040 con 0x7fe20409c240 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.561+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20409c900 con 0x7fe20409c240 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.561+0000 7fe1fb7fe640 1 -- 172.21.15.27:0/956224811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec039070 con 0x7fe20409c240 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.561+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20409cec0 con 0x7fe20409c240 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.562+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe204067ff0 con 0x7fe20409c240 2024-03-06T07:35:02.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.562+0000 7fe1fb7fe640 1 -- 172.21.15.27:0/956224811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1ec045020 con 0x7fe20409c240 2024-03-06T07:35:02.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.563+0000 7fe1fb7fe640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1e406b500 0x7fe1e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.563+0000 7fe1fb7fe640 1 -- 172.21.15.27:0/956224811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe1ec0ab870 con 0x7fe20409c240 2024-03-06T07:35:02.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.563+0000 7fe202ffd640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1e406b500 0x7fe1e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.564+0000 7fe202ffd640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1e406b500 0x7fe1e406d9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40013d0 tx=0x7fe1f4007ef0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.567+0000 7fe1fb7fe640 1 -- 172.21.15.27:0/956224811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1ec04c030 con 0x7fe20409c240 2024-03-06T07:35:02.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.785+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2040b2910 con 0x7fe20409c240 2024-03-06T07:35:02.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.785+0000 7fe1fb7fe640 1 -- 172.21.15.27:0/956224811 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fe1ec07a110 con 0x7fe20409c240 2024-03-06T07:35:02.788 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:02.788 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:02.788 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1e406b500 msgr2=0x7fe1e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1e406b500 0x7fe1e406d9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40013d0 tx=0x7fe1f4007ef0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe20409c240 msgr2=0x7fe20409a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe20409c240 0x7fe20409a320 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009bb0 tx=0x7fe1ec004730 comp rx=0 tx=0).stop 2024-03-06T07:35:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 shutdown_connections 2024-03-06T07:35:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe1e406b500 0x7fe1e406d9c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe20409a860 0x7fe20413ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe20409c240 0x7fe20409a320 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 --2- 172.21.15.27:0/956224811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1f0004800 0x7fe20409d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 >> 172.21.15.27:0/956224811 conn(0x7fe2040b55e0 msgr2=0x7fe2040ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.787+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 shutdown_connections 2024-03-06T07:35:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:02.788+0000 7fe209264640 1 -- 172.21.15.27:0/956224811 wait complete. 2024-03-06T07:35:03.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:03.002 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:04.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:04.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.110+0000 7fc918120640 1 Processor -- start 2024-03-06T07:35:04.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.110+0000 7fc918120640 1 -- start start 2024-03-06T07:35:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc918120640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc91015df00 0x7fc9100d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc918120640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9100b95a0 0x7fc9100b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc918120640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9100b9a30 0x7fc9100b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc918120640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9100cea30 con 0x7fc9100b9a30 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc918120640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9100d2af0 con 0x7fc9100b95a0 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc918120640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100cf3c0 con 0x7fc91015df00 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc9166bd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9100b9a30 0x7fc9100b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc9166bd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9100b9a30 0x7fc9100b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53284/0 (socket says 172.21.15.27:53284) 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc9166bd640 1 -- 172.21.15.27:0/89239428 learned_addr learned my addr 172.21.15.27:0/89239428 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc915ebc640 1 --2- 172.21.15.27:0/89239428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc91015df00 0x7fc9100d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc9156bb640 1 --2- 172.21.15.27:0/89239428 >> v2:172.21.15.69:3300/0 conn(0x7fc9100b95a0 0x7fc9100b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc9166bd640 1 -- 172.21.15.27:0/89239428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc91015df00 msgr2=0x7fc9100d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc9166bd640 1 --2- 172.21.15.27:0/89239428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc91015df00 0x7fc9100d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc9166bd640 1 -- 172.21.15.27:0/89239428 >> v2:172.21.15.69:3300/0 conn(0x7fc9100b95a0 msgr2=0x7fc9100b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc9166bd640 1 --2- 172.21.15.27:0/89239428 >> v2:172.21.15.69:3300/0 conn(0x7fc9100b95a0 0x7fc9100b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.111+0000 7fc9166bd640 1 -- 172.21.15.27:0/89239428 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100cf0e0 con 0x7fc9100b9a30 2024-03-06T07:35:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.112+0000 7fc9166bd640 1 --2- 172.21.15.27:0/89239428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b9a30 0x7fc9100b5c40 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fc91008cbc0 tx=0x7fc90802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83eba885b988f23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.112+0000 7fc914eba640 1 -- 172.21.15.27:0/89239428 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90802ca70 con 0x7fc9100b9a30 2024-03-06T07:35:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.112+0000 7fc914eba640 1 -- 172.21.15.27:0/89239428 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90802cc10 con 0x7fc9100b9a30 2024-03-06T07:35:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.112+0000 7fc914eba640 1 -- 172.21.15.27:0/89239428 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9080343f0 con 0x7fc9100b9a30 2024-03-06T07:35:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.112+0000 7fc918120640 1 -- 172.21.15.27:0/89239428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b9a30 msgr2=0x7fc9100b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.112+0000 7fc918120640 1 --2- 172.21.15.27:0/89239428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b9a30 0x7fc9100b5c40 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fc91008cbc0 tx=0x7fc90802bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.113+0000 7fc918120640 1 -- 172.21.15.27:0/89239428 shutdown_connections 2024-03-06T07:35:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.113+0000 7fc918120640 1 --2- 172.21.15.27:0/89239428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b9a30 0x7fc9100b5c40 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.113+0000 7fc918120640 1 --2- 172.21.15.27:0/89239428 >> v2:172.21.15.69:3300/0 conn(0x7fc9100b95a0 0x7fc9100b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.113+0000 7fc918120640 1 --2- 172.21.15.27:0/89239428 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc91015df00 0x7fc9100d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.113+0000 7fc918120640 1 -- 172.21.15.27:0/89239428 >> 172.21.15.27:0/89239428 conn(0x7fc910071960 msgr2=0x7fc910071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.113+0000 7fc918120640 1 -- 172.21.15.27:0/89239428 shutdown_connections 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.113+0000 7fc918120640 1 -- 172.21.15.27:0/89239428 wait complete. 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.114+0000 7fc918120640 1 Processor -- start 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.114+0000 7fc918120640 1 -- start start 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.114+0000 7fc918120640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9100b95a0 0x7fc9100b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.114+0000 7fc918120640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc91015df00 0x7fc910157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.114+0000 7fc918120640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b4f20 0x7fc910158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.114+0000 7fc918120640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100cec70 con 0x7fc9100b4f20 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.114+0000 7fc918120640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100ce750 con 0x7fc91015df00 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.114+0000 7fc918120640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100cedf0 con 0x7fc9100b95a0 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9166bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b4f20 0x7fc910158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9166bd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b4f20 0x7fc910158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53296/0 (socket says 172.21.15.27:53296) 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9166bd640 1 -- 172.21.15.27:0/2508187811 learned_addr learned my addr 172.21.15.27:0/2508187811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc915ebc640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9100b95a0 0x7fc9100b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9156bb640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc91015df00 0x7fc910157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9166bd640 1 -- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9100b95a0 msgr2=0x7fc9100b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9166bd640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9100b95a0 0x7fc9100b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9166bd640 1 -- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc91015df00 msgr2=0x7fc910157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9166bd640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc91015df00 0x7fc910157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9166bd640 1 -- 172.21.15.27:0/2508187811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91015e460 con 0x7fc9100b4f20 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9156bb640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc91015df00 0x7fc910157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9166bd640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b4f20 0x7fc910158450 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fc90802c4e0 tx=0x7fc908004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9067fc640 1 -- 172.21.15.27:0/2508187811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908043020 con 0x7fc9100b4f20 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9067fc640 1 -- 172.21.15.27:0/2508187811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90803e450 con 0x7fc9100b4f20 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9100b5430 con 0x7fc9100b4f20 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc9067fc640 1 -- 172.21.15.27:0/2508187811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908009510 con 0x7fc9100b4f20 2024-03-06T07:35:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.115+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9101591e0 con 0x7fc9100b4f20 2024-03-06T07:35:04.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.116+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc910067ff0 con 0x7fc9100b4f20 2024-03-06T07:35:04.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.117+0000 7fc9067fc640 1 -- 172.21.15.27:0/2508187811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc908046020 con 0x7fc9100b4f20 2024-03-06T07:35:04.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.118+0000 7fc9067fc640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc8f406b500 0x7fc8f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.118+0000 7fc9067fc640 1 -- 172.21.15.27:0/2508187811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc9080ac9d0 con 0x7fc9100b4f20 2024-03-06T07:35:04.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.118+0000 7fc915ebc640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc8f406b500 0x7fc8f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.118+0000 7fc915ebc640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc8f406b500 0x7fc8f406d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc90c006fd0 tx=0x7fc90c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.122+0000 7fc9067fc640 1 -- 172.21.15.27:0/2508187811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc90804d030 con 0x7fc9100b4f20 2024-03-06T07:35:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.342+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9101568b0 con 0x7fc9100b4f20 2024-03-06T07:35:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.342+0000 7fc9067fc640 1 -- 172.21.15.27:0/2508187811 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fc90807b830 con 0x7fc9100b4f20 2024-03-06T07:35:04.345 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:04.345 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc8f406b500 msgr2=0x7fc8f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc8f406b500 0x7fc8f406d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc90c006fd0 tx=0x7fc90c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b4f20 msgr2=0x7fc910158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b4f20 0x7fc910158450 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fc90802c4e0 tx=0x7fc908004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 shutdown_connections 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc8f406b500 0x7fc8f406d9c0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9100b4f20 0x7fc910158450 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc91015df00 0x7fc910157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 --2- 172.21.15.27:0/2508187811 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc9100b95a0 0x7fc9100b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 >> 172.21.15.27:0/2508187811 conn(0x7fc910071960 msgr2=0x7fc9101618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 shutdown_connections 2024-03-06T07:35:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:04.344+0000 7fc918120640 1 -- 172.21.15.27:0/2508187811 wait complete. 2024-03-06T07:35:04.559 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:04.559 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:05.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.664+0000 7f4440e55640 1 Processor -- start 2024-03-06T07:35:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.664+0000 7f4440e55640 1 -- start start 2024-03-06T07:35:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f4440e55640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f443c15df00 0x7f443c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f4440e55640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f443c0b95a0 0x7f443c0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f4440e55640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c0cea30 con 0x7f443c0b9a30 2024-03-06T07:35:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f4440e55640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443c0d2af0 con 0x7f443c15df00 2024-03-06T07:35:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f4440e55640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443c0cf3c0 con 0x7f443c0b95a0 2024-03-06T07:35:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f443a59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f443c15df00 0x7f443c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f443a59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f443c15df00 0x7f443c0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53308/0 (socket says 172.21.15.27:53308) 2024-03-06T07:35:05.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f443a59c640 1 -- 172.21.15.27:0/4000336763 learned_addr learned my addr 172.21.15.27:0/4000336763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f4439d9b640 1 --2- 172.21.15.27:0/4000336763 >> v2:172.21.15.69:3300/0 conn(0x7f443c0b95a0 0x7f443c0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f443a59c640 1 -- 172.21.15.27:0/4000336763 >> v2:172.21.15.69:3300/0 conn(0x7f443c0b95a0 msgr2=0x7f443c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f443a59c640 1 --2- 172.21.15.27:0/4000336763 >> v2:172.21.15.69:3300/0 conn(0x7f443c0b95a0 0x7f443c0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.665+0000 7f443a59c640 1 -- 172.21.15.27:0/4000336763 >> v1:172.21.15.69:6789/0 conn(0x7f443c0b9a30 legacy=0x7f443c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.666+0000 7f443a59c640 1 -- 172.21.15.27:0/4000336763 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0cf0e0 con 0x7f443c15df00 2024-03-06T07:35:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.666+0000 7f443a59c640 1 --2- 172.21.15.27:0/4000336763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 0x7f443c0d25b0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f443c0d1760 tx=0x7f442c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554d7d204f6f2afe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.666+0000 7f443959a640 1 -- 172.21.15.27:0/4000336763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c035020 con 0x7f443c15df00 2024-03-06T07:35:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.666+0000 7f443959a640 1 -- 172.21.15.27:0/4000336763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c031870 con 0x7f443c15df00 2024-03-06T07:35:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.666+0000 7f443959a640 1 -- 172.21.15.27:0/4000336763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c031c40 con 0x7f443c15df00 2024-03-06T07:35:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.667+0000 7f4440e55640 1 -- 172.21.15.27:0/4000336763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 msgr2=0x7f443c0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.667+0000 7f4440e55640 1 --2- 172.21.15.27:0/4000336763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 0x7f443c0d25b0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f443c0d1760 tx=0x7f442c02b910 comp rx=0 tx=0).stop 2024-03-06T07:35:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.667+0000 7f4440e55640 1 -- 172.21.15.27:0/4000336763 shutdown_connections 2024-03-06T07:35:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.667+0000 7f4440e55640 1 --2- 172.21.15.27:0/4000336763 >> v2:172.21.15.69:3300/0 conn(0x7f443c0b95a0 0x7f443c0b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.667+0000 7f4440e55640 1 --2- 172.21.15.27:0/4000336763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 0x7f443c0d25b0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.667+0000 7f4440e55640 1 -- 172.21.15.27:0/4000336763 >> 172.21.15.27:0/4000336763 conn(0x7f443c071960 msgr2=0x7f443c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.667+0000 7f4440e55640 1 -- 172.21.15.27:0/4000336763 shutdown_connections 2024-03-06T07:35:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.667+0000 7f4440e55640 1 -- 172.21.15.27:0/4000336763 wait complete. 2024-03-06T07:35:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.668+0000 7f4440e55640 1 Processor -- start 2024-03-06T07:35:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.668+0000 7f4440e55640 1 -- start start 2024-03-06T07:35:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f4440e55640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f443c0b95a0 0x7f443c160f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f4440e55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 0x7f443c0cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f4440e55640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f443c1e89e0 0x7f443c15aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f4440e55640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0cec70 con 0x7f443c15df00 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f4440e55640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0ce750 con 0x7f443c0b95a0 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f4440e55640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0cedf0 con 0x7f443c1e89e0 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f4439d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 0x7f443c0cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f4439d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 0x7f443c0cc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53310/0 (socket says 172.21.15.27:53310) 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f4439d9b640 1 -- 172.21.15.27:0/1396742051 learned_addr learned my addr 172.21.15.27:0/1396742051 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f443a59c640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f443c0b95a0 0x7f443c160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.669+0000 7f443ad9d640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f443c1e89e0 0x7f443c15aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f4439d9b640 1 -- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f443c1e89e0 msgr2=0x7f443c15aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f4439d9b640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f443c1e89e0 0x7f443c15aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f4439d9b640 1 -- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f443c0b95a0 msgr2=0x7f443c160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f4439d9b640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f443c0b95a0 0x7f443c160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f4439d9b640 1 -- 172.21.15.27:0/1396742051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c161450 con 0x7f443c15df00 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f4439d9b640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 0x7f443c0cc5b0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f443000c820 tx=0x7f443000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f442affd640 1 -- 172.21.15.27:0/1396742051 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430015070 con 0x7f443c15df00 2024-03-06T07:35:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f442affd640 1 -- 172.21.15.27:0/1396742051 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4430004220 con 0x7f443c15df00 2024-03-06T07:35:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c1f7070 con 0x7f443c15df00 2024-03-06T07:35:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f442affd640 1 -- 172.21.15.27:0/1396742051 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443000ce10 con 0x7f443c15df00 2024-03-06T07:35:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.670+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c1f7630 con 0x7f443c15df00 2024-03-06T07:35:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.671+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c067ff0 con 0x7f443c15df00 2024-03-06T07:35:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.672+0000 7f442affd640 1 -- 172.21.15.27:0/1396742051 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4430007830 con 0x7f443c15df00 2024-03-06T07:35:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.672+0000 7f442affd640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441c06b4d0 0x7f441c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.672+0000 7f442affd640 1 -- 172.21.15.27:0/1396742051 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4430089fd0 con 0x7f443c15df00 2024-03-06T07:35:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.672+0000 7f443a59c640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441c06b4d0 0x7f441c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.673+0000 7f443a59c640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441c06b4d0 0x7f441c06d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f442c036000 tx=0x7f442c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.681+0000 7f442affd640 1 -- 172.21.15.27:0/1396742051 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4430058870 con 0x7f443c15df00 2024-03-06T07:35:05.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.902+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f443c0b3040 con 0x7f443c15df00 2024-03-06T07:35:05.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.903+0000 7f442affd640 1 -- 172.21.15.27:0/1396742051 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f443005c910 con 0x7f443c15df00 2024-03-06T07:35:05.906 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:05.906 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:05.906 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:05.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441c06b4d0 msgr2=0x7f441c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441c06b4d0 0x7f441c06d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f442c036000 tx=0x7f442c03c040 comp rx=0 tx=0).stop 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 msgr2=0x7f443c0cc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 0x7f443c0cc5b0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f443000c820 tx=0x7f443000cce0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 shutdown_connections 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f441c06b4d0 0x7f441c06d990 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f443c1e89e0 0x7f443c15aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f443c15df00 0x7f443c0cc5b0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 --2- 172.21.15.27:0/1396742051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f443c0b95a0 0x7f443c160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 >> 172.21.15.27:0/1396742051 conn(0x7f443c071960 msgr2=0x7f443c15bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 shutdown_connections 2024-03-06T07:35:05.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:05.906+0000 7f4440e55640 1 -- 172.21.15.27:0/1396742051 wait complete. 2024-03-06T07:35:06.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:06.125 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:07.125 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:07.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.231+0000 7fa458193640 1 Processor -- start 2024-03-06T07:35:07.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.231+0000 7fa458193640 1 -- start start 2024-03-06T07:35:07.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa458193640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4500b0e90 0x7fa45006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa458193640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa4500ae9b0 0x7fa4500d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa458193640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4500cee90 con 0x7fa45006b5c0 2024-03-06T07:35:07.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa458193640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4500cfa00 con 0x7fa4500b0e90 2024-03-06T07:35:07.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa458193640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4500cfc50 con 0x7fa4500ae9b0 2024-03-06T07:35:07.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa45572e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4500b0e90 0x7fa45006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa45572e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4500b0e90 0x7fa45006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53322/0 (socket says 172.21.15.27:53322) 2024-03-06T07:35:07.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa455f2f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa45006b5c0 0x7fa4500b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38338/0 (socket says 172.21.15.27:38338) 2024-03-06T07:35:07.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa45572e640 1 -- 172.21.15.27:0/2065680633 learned_addr learned my addr 172.21.15.27:0/2065680633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:07.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa456730640 1 --2- 172.21.15.27:0/2065680633 >> v2:172.21.15.69:3300/0 conn(0x7fa4500ae9b0 0x7fa4500d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa45572e640 1 -- 172.21.15.27:0/2065680633 >> v2:172.21.15.69:3300/0 conn(0x7fa4500ae9b0 msgr2=0x7fa4500d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:07.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa45572e640 1 --2- 172.21.15.27:0/2065680633 >> v2:172.21.15.69:3300/0 conn(0x7fa4500ae9b0 0x7fa4500d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa454f2d640 1 -- 172.21.15.27:0/2065680633 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126781483 0 0) 0x7fa4500cee90 con 0x7fa45006b5c0 2024-03-06T07:35:07.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa45572e640 1 -- 172.21.15.27:0/2065680633 >> v1:172.21.15.27:6789/0 conn(0x7fa45006b5c0 legacy=0x7fa4500b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa45572e640 1 -- 172.21.15.27:0/2065680633 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4500cff30 con 0x7fa4500b0e90 2024-03-06T07:35:07.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.232+0000 7fa456730640 1 --2- 172.21.15.27:0/2065680633 >> v2:172.21.15.69:3300/0 conn(0x7fa4500ae9b0 0x7fa4500d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:07.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.233+0000 7fa45572e640 1 --2- 172.21.15.27:0/2065680633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 0x7fa45006b9a0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fa450072170 tx=0x7fa44002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494977c847894995 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.234+0000 7fa454f2d640 1 -- 172.21.15.27:0/2065680633 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440035020 con 0x7fa4500b0e90 2024-03-06T07:35:07.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.234+0000 7fa454f2d640 1 -- 172.21.15.27:0/2065680633 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa440031870 con 0x7fa4500b0e90 2024-03-06T07:35:07.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.234+0000 7fa454f2d640 1 -- 172.21.15.27:0/2065680633 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440031c40 con 0x7fa4500b0e90 2024-03-06T07:35:07.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.234+0000 7fa458193640 1 -- 172.21.15.27:0/2065680633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 msgr2=0x7fa45006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.234+0000 7fa458193640 1 --2- 172.21.15.27:0/2065680633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 0x7fa45006b9a0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fa450072170 tx=0x7fa44002b910 comp rx=0 tx=0).stop 2024-03-06T07:35:07.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.235+0000 7fa458193640 1 -- 172.21.15.27:0/2065680633 shutdown_connections 2024-03-06T07:35:07.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.235+0000 7fa458193640 1 --2- 172.21.15.27:0/2065680633 >> v2:172.21.15.69:3300/0 conn(0x7fa4500ae9b0 0x7fa4500d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.235+0000 7fa458193640 1 --2- 172.21.15.27:0/2065680633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 0x7fa45006b9a0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.235+0000 7fa458193640 1 -- 172.21.15.27:0/2065680633 >> 172.21.15.27:0/2065680633 conn(0x7fa4500b36d0 msgr2=0x7fa4500b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:07.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.235+0000 7fa458193640 1 -- 172.21.15.27:0/2065680633 shutdown_connections 2024-03-06T07:35:07.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.235+0000 7fa458193640 1 -- 172.21.15.27:0/2065680633 wait complete. 2024-03-06T07:35:07.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.237+0000 7fa458193640 1 Processor -- start 2024-03-06T07:35:07.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.237+0000 7fa458193640 1 -- start start 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa458193640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa45006b5c0 0x7fa4501d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa458193640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4500ae9b0 0x7fa4501deb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa458193640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 0x7fa4501df040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa458193640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45006c910 con 0x7fa4500b0e90 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa458193640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45006c790 con 0x7fa4500ae9b0 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa458193640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4500aedc0 con 0x7fa45006b5c0 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa456730640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 0x7fa4501df040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa456730640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 0x7fa4501df040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53326/0 (socket says 172.21.15.27:53326) 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa456730640 1 -- 172.21.15.27:0/4114418381 learned_addr learned my addr 172.21.15.27:0/4114418381 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa45572e640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4500ae9b0 0x7fa4501deb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.238+0000 7fa455f2f640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa45006b5c0 0x7fa4501d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa456730640 1 -- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa45006b5c0 msgr2=0x7fa4501d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa456730640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa45006b5c0 0x7fa4501d1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa456730640 1 -- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4500ae9b0 msgr2=0x7fa4501deb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa456730640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4500ae9b0 0x7fa4501deb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa456730640 1 -- 172.21.15.27:0/4114418381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4500cbfd0 con 0x7fa4500b0e90 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa456730640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 0x7fa4501df040 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fa44800cef0 tx=0x7fa44800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa4467fc640 1 -- 172.21.15.27:0/4114418381 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44801a070 con 0x7fa4500b0e90 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa4467fc640 1 -- 172.21.15.27:0/4114418381 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44800d860 con 0x7fa4500b0e90 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa4467fc640 1 -- 172.21.15.27:0/4114418381 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44800dc50 con 0x7fa4500b0e90 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4500cc2b0 con 0x7fa4500b0e90 2024-03-06T07:35:07.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.239+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4500cc870 con 0x7fa4500b0e90 2024-03-06T07:35:07.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.240+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa450068700 con 0x7fa4500b0e90 2024-03-06T07:35:07.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.241+0000 7fa4467fc640 1 -- 172.21.15.27:0/4114418381 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa44801d070 con 0x7fa4500b0e90 2024-03-06T07:35:07.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.241+0000 7fa4467fc640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa43406b4d0 0x7fa43406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.241+0000 7fa4467fc640 1 -- 172.21.15.27:0/4114418381 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa44808aa40 con 0x7fa4500b0e90 2024-03-06T07:35:07.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.241+0000 7fa455f2f640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa43406b4d0 0x7fa43406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.242+0000 7fa455f2f640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa43406b4d0 0x7fa43406d990 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa44c000f60 tx=0x7fa44c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.246+0000 7fa4467fc640 1 -- 172.21.15.27:0/4114418381 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa448059310 con 0x7fa4500b0e90 2024-03-06T07:35:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.462+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa45006e440 con 0x7fa4500b0e90 2024-03-06T07:35:07.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.463+0000 7fa4467fc640 1 -- 172.21.15.27:0/4114418381 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fa44808ae20 con 0x7fa4500b0e90 2024-03-06T07:35:07.464 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:07.464 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:07.465 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.464+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa43406b4d0 msgr2=0x7fa43406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.464+0000 7fa458193640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa43406b4d0 0x7fa43406d990 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa44c000f60 tx=0x7fa44c007d50 comp rx=0 tx=0).stop 2024-03-06T07:35:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.464+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 msgr2=0x7fa4501df040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.465+0000 7fa458193640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 0x7fa4501df040 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fa44800cef0 tx=0x7fa44800a710 comp rx=0 tx=0).stop 2024-03-06T07:35:07.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.465+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 shutdown_connections 2024-03-06T07:35:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.465+0000 7fa458193640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fa43406b4d0 0x7fa43406d990 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.465+0000 7fa458193640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4500b0e90 0x7fa4501df040 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.465+0000 7fa458193640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4500ae9b0 0x7fa4501deb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.465+0000 7fa458193640 1 --2- 172.21.15.27:0/4114418381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa45006b5c0 0x7fa4501d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.465+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 >> 172.21.15.27:0/4114418381 conn(0x7fa4500b36d0 msgr2=0x7fa4500b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.465+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 shutdown_connections 2024-03-06T07:35:07.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:07.465+0000 7fa458193640 1 -- 172.21.15.27:0/4114418381 wait complete. 2024-03-06T07:35:07.678 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:07.678 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:08.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:08.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.784+0000 7f15aac88640 1 Processor -- start 2024-03-06T07:35:08.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.784+0000 7f15aac88640 1 -- start start 2024-03-06T07:35:08.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.785+0000 7f15aac88640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f15a415df00 0x7f15a40d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.785+0000 7f15aac88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 0x7f15a40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.785+0000 7f15aac88640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40cea30 con 0x7f15a40b9a30 2024-03-06T07:35:08.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.785+0000 7f15aac88640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a40d2af0 con 0x7f15a415df00 2024-03-06T07:35:08.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.785+0000 7f15aac88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40cf3c0 con 0x7f15a40b95a0 2024-03-06T07:35:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.785+0000 7f15a8a24640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f15a415df00 0x7f15a40d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.785+0000 7f15a8a24640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f15a415df00 0x7f15a40d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53340/0 (socket says 172.21.15.27:53340) 2024-03-06T07:35:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.785+0000 7f15a8a24640 1 -- 172.21.15.27:0/2959749697 learned_addr learned my addr 172.21.15.27:0/2959749697 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.786+0000 7f15a3fff640 1 --2- 172.21.15.27:0/2959749697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 0x7f15a40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.786+0000 7f15a8a24640 1 -- 172.21.15.27:0/2959749697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 msgr2=0x7f15a40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.786+0000 7f15a8a24640 1 --2- 172.21.15.27:0/2959749697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 0x7f15a40b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.786+0000 7f15a8a24640 1 -- 172.21.15.27:0/2959749697 >> v1:172.21.15.69:6789/0 conn(0x7f15a40b9a30 legacy=0x7f15a40d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.786+0000 7f15a8a24640 1 -- 172.21.15.27:0/2959749697 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40cf0e0 con 0x7f15a415df00 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15a8a24640 1 --2- 172.21.15.27:0/2959749697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 0x7f15a40d25b0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f159c009870 tx=0x7f159c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a21bbbe50d31d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15a37fe640 1 -- 172.21.15.27:0/2959749697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c035020 con 0x7f15a415df00 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15a37fe640 1 -- 172.21.15.27:0/2959749697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c031640 con 0x7f15a415df00 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15a37fe640 1 -- 172.21.15.27:0/2959749697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c02d040 con 0x7f15a415df00 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15aac88640 1 -- 172.21.15.27:0/2959749697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 msgr2=0x7f15a40d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15aac88640 1 --2- 172.21.15.27:0/2959749697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 0x7f15a40d25b0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f159c009870 tx=0x7f159c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15aac88640 1 -- 172.21.15.27:0/2959749697 shutdown_connections 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15aac88640 1 --2- 172.21.15.27:0/2959749697 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 0x7f15a40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15aac88640 1 --2- 172.21.15.27:0/2959749697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 0x7f15a40d25b0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15aac88640 1 -- 172.21.15.27:0/2959749697 >> 172.21.15.27:0/2959749697 conn(0x7f15a4071960 msgr2=0x7f15a4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15aac88640 1 -- 172.21.15.27:0/2959749697 shutdown_connections 2024-03-06T07:35:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.787+0000 7f15aac88640 1 -- 172.21.15.27:0/2959749697 wait complete. 2024-03-06T07:35:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.788+0000 7f15aac88640 1 Processor -- start 2024-03-06T07:35:08.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.789+0000 7f15aac88640 1 -- start start 2024-03-06T07:35:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.789+0000 7f15aac88640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 0x7f15a415a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.789+0000 7f15aac88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 0x7f15a415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.789+0000 7f15aac88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a415abc0 0x7f15a415b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.789+0000 7f15aac88640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40cec70 con 0x7f15a415df00 2024-03-06T07:35:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.789+0000 7f15aac88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40ce750 con 0x7f15a415abc0 2024-03-06T07:35:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.789+0000 7f15aac88640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40cedf0 con 0x7f15a40b95a0 2024-03-06T07:35:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.789+0000 7f15a3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 0x7f15a415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 0x7f15a415e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53350/0 (socket says 172.21.15.27:53350) 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a3fff640 1 -- 172.21.15.27:0/2376573823 learned_addr learned my addr 172.21.15.27:0/2376573823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a8a24640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 0x7f15a415a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a9225640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a415abc0 0x7f15a415b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a3fff640 1 -- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 msgr2=0x7f15a415a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a3fff640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 0x7f15a415a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a3fff640 1 -- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a415abc0 msgr2=0x7f15a415b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a3fff640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a415abc0 0x7f15a415b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a3fff640 1 -- 172.21.15.27:0/2376573823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a415a970 con 0x7f15a415df00 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a3fff640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 0x7f15a415e2e0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f159800c850 tx=0x7f159800cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a17fa640 1 -- 172.21.15.27:0/2376573823 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598015070 con 0x7f15a415df00 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a17fa640 1 -- 172.21.15.27:0/2376573823 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1598004020 con 0x7f15a415df00 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a40d4090 con 0x7f15a415df00 2024-03-06T07:35:08.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15a17fa640 1 -- 172.21.15.27:0/2376573823 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15980043f0 con 0x7f15a415df00 2024-03-06T07:35:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.790+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a40c93b0 con 0x7f15a415df00 2024-03-06T07:35:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.791+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a4067ff0 con 0x7f15a415df00 2024-03-06T07:35:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.792+0000 7f15a17fa640 1 -- 172.21.15.27:0/2376573823 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1598007830 con 0x7f15a415df00 2024-03-06T07:35:08.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.792+0000 7f15a17fa640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159006b4d0 0x7f159006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.792+0000 7f15a17fa640 1 -- 172.21.15.27:0/2376573823 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1598089fa0 con 0x7f15a415df00 2024-03-06T07:35:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.793+0000 7f15a8a24640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159006b4d0 0x7f159006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.793+0000 7f15a8a24640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159006b4d0 0x7f159006d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f159c036000 tx=0x7f159c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:08.797+0000 7f15a17fa640 1 -- 172.21.15.27:0/2376573823 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15980587b0 con 0x7f15a415df00 2024-03-06T07:35:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.014+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15a406b780 con 0x7f15a415df00 2024-03-06T07:35:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.014+0000 7f15a17fa640 1 -- 172.21.15.27:0/2376573823 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f159805c850 con 0x7f15a415df00 2024-03-06T07:35:09.016 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:09.016 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:09.016 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159006b4d0 msgr2=0x7f159006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159006b4d0 0x7f159006d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f159c036000 tx=0x7f159c03c040 comp rx=0 tx=0).stop 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 msgr2=0x7f15a415e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 0x7f15a415e2e0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f159800c850 tx=0x7f159800cd10 comp rx=0 tx=0).stop 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 shutdown_connections 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f159006b4d0 0x7f159006d990 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15a415abc0 0x7f15a415b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15a415df00 0x7f15a415e2e0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 --2- 172.21.15.27:0/2376573823 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f15a40b95a0 0x7f15a415a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 >> 172.21.15.27:0/2376573823 conn(0x7f15a4071960 msgr2=0x7f15a40d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 shutdown_connections 2024-03-06T07:35:09.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:09.017+0000 7f15aac88640 1 -- 172.21.15.27:0/2376573823 wait complete. 2024-03-06T07:35:09.235 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:09.235 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:10.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.343+0000 7fc398637640 1 Processor -- start 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.343+0000 7fc398637640 1 -- start start 2024-03-06T07:35:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.344+0000 7fc398637640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900b95a0 0x7fc3900b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.344+0000 7fc398637640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3900cea30 con 0x7fc39015df00 2024-03-06T07:35:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.344+0000 7fc398637640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3900cf3c0 con 0x7fc3900b9a10 2024-03-06T07:35:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.344+0000 7fc398637640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900cf0e0 con 0x7fc3900b95a0 2024-03-06T07:35:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.344+0000 7fc3963d3640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc39015df00 0x7fc3900d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38348/0 (socket says 172.21.15.27:38348) 2024-03-06T07:35:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.344+0000 7fc3963d3640 1 -- 172.21.15.27:0/706067000 learned_addr learned my addr 172.21.15.27:0/706067000 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.344+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690899211 0 0) 0x7fc3900cea30 con 0x7fc39015df00 2024-03-06T07:35:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c003520 con 0x7fc39015df00 2024-03-06T07:35:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc395bd2640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900b95a0 0x7fc3900b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2526109025 0 0) 0x7fc37c003520 con 0x7fc39015df00 2024-03-06T07:35:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900b95a0 msgr2=0x7fc3900b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900b95a0 0x7fc3900b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 >> v1:172.21.15.69:6789/0 conn(0x7fc3900b9a10 legacy=0x7fc3900d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3900ce470 con 0x7fc39015df00 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fc38c001670 con 0x7fc39015df00 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 >> v1:172.21.15.27:6789/0 conn(0x7fc39015df00 legacy=0x7fc3900d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc37c0039e0 0x7fc37c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc37c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900b95a0 0x7fc37c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900cf0e0 con 0x7fc37c004800 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c00baa0 con 0x7fc37c0039e0 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c00bca0 con 0x7fc3900b95a0 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc395bd2640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc37c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc3963d3640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc37c0039e0 0x7fc37c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc396bd4640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900b95a0 0x7fc37c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc395bd2640 1 -- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900b95a0 msgr2=0x7fc37c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc395bd2640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900b95a0 0x7fc37c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc395bd2640 1 -- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc37c0039e0 msgr2=0x7fc37c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc395bd2640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc37c0039e0 0x7fc37c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.345+0000 7fc395bd2640 1 -- 172.21.15.27:0/706067000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c003520 con 0x7fc37c004800 2024-03-06T07:35:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.347+0000 7fc395bd2640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc37c004c80 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fc384009a90 tx=0x7fc37c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13cf92d4f223c68b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.347+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384039070 con 0x7fc37c004800 2024-03-06T07:35:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.347+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38402c680 con 0x7fc37c004800 2024-03-06T07:35:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.347+0000 7fc3953d1640 1 -- 172.21.15.27:0/706067000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384009690 con 0x7fc37c004800 2024-03-06T07:35:10.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.347+0000 7fc398637640 1 -- 172.21.15.27:0/706067000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 msgr2=0x7fc37c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.347+0000 7fc398637640 1 --2- 172.21.15.27:0/706067000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc37c004c80 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fc384009a90 tx=0x7fc37c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.347+0000 7fc396bd4640 1 -- 172.21.15.27:0/706067000 reap_dead start 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.347+0000 7fc398637640 1 -- 172.21.15.27:0/706067000 shutdown_connections 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.347+0000 7fc398637640 1 -- 172.21.15.27:0/706067000 >> 172.21.15.27:0/706067000 conn(0x7fc390071960 msgr2=0x7fc390071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.348+0000 7fc398637640 1 -- 172.21.15.27:0/706067000 shutdown_connections 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.348+0000 7fc398637640 1 -- 172.21.15.27:0/706067000 wait complete. 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.349+0000 7fc398637640 1 Processor -- start 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.349+0000 7fc398637640 1 -- start start 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.349+0000 7fc398637640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc390158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.349+0000 7fc398637640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015be00 0x7fc39015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.349+0000 7fc398637640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900cc4e0 0x7fc3900cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.349+0000 7fc398637640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900cedf0 con 0x7fc37c004800 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.349+0000 7fc398637640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900cec70 con 0x7fc39015be00 2024-03-06T07:35:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.349+0000 7fc398637640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900d3930 con 0x7fc3900cc4e0 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc3963d3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc390158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc3963d3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc390158580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53376/0 (socket says 172.21.15.27:53376) 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc3963d3640 1 -- 172.21.15.27:0/3253587797 learned_addr learned my addr 172.21.15.27:0/3253587797 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc3963d3640 1 -- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900cc4e0 msgr2=0x7fc3900cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc3963d3640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900cc4e0 0x7fc3900cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc3963d3640 1 -- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015be00 msgr2=0x7fc39015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc395bd2640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015be00 0x7fc39015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc3963d3640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015be00 0x7fc39015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc3963d3640 1 -- 172.21.15.27:0/3253587797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc390158ac0 con 0x7fc37c004800 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc3963d3640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc390158580 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0072c0 tx=0x7fc38c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc382ffd640 1 -- 172.21.15.27:0/3253587797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c01a070 con 0x7fc37c004800 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc382ffd640 1 -- 172.21.15.27:0/3253587797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38c00f8e0 con 0x7fc37c004800 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3900c78e0 con 0x7fc37c004800 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc382ffd640 1 -- 172.21.15.27:0/3253587797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c00fc90 con 0x7fc37c004800 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.350+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3900c7ea0 con 0x7fc37c004800 2024-03-06T07:35:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.351+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39015c7c0 con 0x7fc37c004800 2024-03-06T07:35:10.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.352+0000 7fc382ffd640 1 -- 172.21.15.27:0/3253587797 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc38c01d070 con 0x7fc37c004800 2024-03-06T07:35:10.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.352+0000 7fc382ffd640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37406b4d0 0x7fc37406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.352+0000 7fc382ffd640 1 -- 172.21.15.27:0/3253587797 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc38c089ca0 con 0x7fc37c004800 2024-03-06T07:35:10.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.353+0000 7fc395bd2640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37406b4d0 0x7fc37406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.353+0000 7fc395bd2640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37406b4d0 0x7fc37406d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc384036000 tx=0x7fc384030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.357+0000 7fc382ffd640 1 -- 172.21.15.27:0/3253587797 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc38c058b90 con 0x7fc37c004800 2024-03-06T07:35:10.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.576+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc390067ff0 con 0x7fc37c004800 2024-03-06T07:35:10.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.577+0000 7fc382ffd640 1 -- 172.21.15.27:0/3253587797 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fc38c058490 con 0x7fc37c004800 2024-03-06T07:35:10.579 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:10.580 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:10.580 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:10.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37406b4d0 msgr2=0x7fc37406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37406b4d0 0x7fc37406d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc384036000 tx=0x7fc384030000 comp rx=0 tx=0).stop 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 msgr2=0x7fc390158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc390158580 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0072c0 tx=0x7fc38c007780 comp rx=0 tx=0).stop 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 shutdown_connections 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc37406b4d0 0x7fc37406d990 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc3900cc4e0 0x7fc3900cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc39015be00 0x7fc39015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 --2- 172.21.15.27:0/3253587797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc37c004800 0x7fc390158580 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.578+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 >> 172.21.15.27:0/3253587797 conn(0x7fc390071960 msgr2=0x7fc390068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.579+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 shutdown_connections 2024-03-06T07:35:10.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:10.579+0000 7fc398637640 1 -- 172.21.15.27:0/3253587797 wait complete. 2024-03-06T07:35:10.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:10.792 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:11.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.900+0000 7f43a0c8d640 1 Processor -- start 2024-03-06T07:35:11.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.900+0000 7f43a0c8d640 1 -- start start 2024-03-06T07:35:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.901+0000 7f43a0c8d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f439c0ae9b0 0x7f439c0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.901+0000 7f43a0c8d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439c0cee90 con 0x7f439c06b5c0 2024-03-06T07:35:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.901+0000 7f43a0c8d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439c0cfc50 con 0x7f439c0b0e90 2024-03-06T07:35:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.901+0000 7f43a0c8d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f439c0cfa00 con 0x7f439c0ae9b0 2024-03-06T07:35:11.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.901+0000 7f439a59c640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f439c06b5c0 0x7f439c0b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38356/0 (socket says 172.21.15.27:38356) 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.901+0000 7f439a59c640 1 -- 172.21.15.27:0/2620101253 learned_addr learned my addr 172.21.15.27:0/2620101253 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.901+0000 7f439ad9d640 1 --2- 172.21.15.27:0/2620101253 >> v2:172.21.15.69:3300/0 conn(0x7f439c0ae9b0 0x7f439c0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.901+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717676443 0 0) 0x7f439c0cee90 con 0x7f439c06b5c0 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.901+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4384003520 con 0x7f439c06b5c0 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588856095 0 0) 0x7f439c0cfc50 con 0x7f439c0b0e90 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f439c0cee90 con 0x7f439c0b0e90 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1312232577 0 0) 0x7f4384003520 con 0x7f439c06b5c0 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 >> v2:172.21.15.69:3300/0 conn(0x7f439c0ae9b0 msgr2=0x7f439c0d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 --2- 172.21.15.27:0/2620101253 >> v2:172.21.15.69:3300/0 conn(0x7f439c0ae9b0 0x7f439c0d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 >> v1:172.21.15.69:6789/0 conn(0x7f439c0b0e90 legacy=0x7f439c06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c0cff30 con 0x7f439c06b5c0 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f438c0015f0 con 0x7f439c06b5c0 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 >> v1:172.21.15.27:6789/0 conn(0x7f439c06b5c0 legacy=0x7f439c0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f4384003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4384004800 0x7f4384004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4384005210 0x7f438400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0cfa00 con 0x7f4384003b90 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438400c140 con 0x7f4384005210 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438400c340 con 0x7f4384004800 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439a59c640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f4384003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f4399d9b640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4384004800 0x7f4384004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439ad9d640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4384005210 0x7f438400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439a59c640 1 -- 172.21.15.27:0/2620101253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4384004800 msgr2=0x7f4384004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439a59c640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4384004800 0x7f4384004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439a59c640 1 -- 172.21.15.27:0/2620101253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4384005210 msgr2=0x7f438400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439a59c640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4384005210 0x7f438400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439ad9d640 1 -- 172.21.15.27:0/2620101253 reap_dead start 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.902+0000 7f439a59c640 1 -- 172.21.15.27:0/2620101253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4384003520 con 0x7f4384003b90 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.903+0000 7f439a59c640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f4384003f90 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f438c007f60 tx=0x7f4384005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee810aae79c828b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.903+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c007b80 con 0x7f4384003b90 2024-03-06T07:35:11.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.903+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c030070 con 0x7f4384003b90 2024-03-06T07:35:11.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.903+0000 7f439959a640 1 -- 172.21.15.27:0/2620101253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c005e50 con 0x7f4384003b90 2024-03-06T07:35:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.904+0000 7f43a0c8d640 1 -- 172.21.15.27:0/2620101253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 msgr2=0x7f4384003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.904+0000 7f43a0c8d640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f4384003f90 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f438c007f60 tx=0x7f4384005830 comp rx=0 tx=0).stop 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.904+0000 7f43a0c8d640 1 -- 172.21.15.27:0/2620101253 shutdown_connections 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.904+0000 7f43a0c8d640 1 --2- 172.21.15.27:0/2620101253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f4384003f90 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.904+0000 7f43a0c8d640 1 -- 172.21.15.27:0/2620101253 >> 172.21.15.27:0/2620101253 conn(0x7f439c0b36d0 msgr2=0x7f439c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.904+0000 7f43a0c8d640 1 -- 172.21.15.27:0/2620101253 shutdown_connections 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.904+0000 7f43a0c8d640 1 -- 172.21.15.27:0/2620101253 wait complete. 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.905+0000 7f43a0c8d640 1 Processor -- start 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.905+0000 7f43a0c8d640 1 -- start start 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f43a0c8d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f439c1defc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f43a0c8d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439c0c9df0 0x7f439c0c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f43a0c8d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f439c1e4650 0x7f439c1e4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f43a0c8d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0aef40 con 0x7f4384003b90 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f43a0c8d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0aedc0 con 0x7f439c1e4650 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f43a0c8d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0b2e00 con 0x7f439c0c9df0 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f439a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f439c1defc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f439a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f439c1defc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53400/0 (socket says 172.21.15.27:53400) 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f439a59c640 1 -- 172.21.15.27:0/1885921778 learned_addr learned my addr 172.21.15.27:0/1885921778 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f4399d9b640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439c0c9df0 0x7f439c0c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f439ad9d640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f439c1e4650 0x7f439c1e4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f439a59c640 1 -- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439c0c9df0 msgr2=0x7f439c0c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f439a59c640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439c0c9df0 0x7f439c0c6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f439a59c640 1 -- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f439c1e4650 msgr2=0x7f439c1e4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f439a59c640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f439c1e4650 0x7f439c1e4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.906+0000 7f439a59c640 1 -- 172.21.15.27:0/1885921778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c0c6f90 con 0x7f4384003b90 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.907+0000 7f439a59c640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f439c1defc0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f438c00c1d0 tx=0x7f438c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.907+0000 7f438affd640 1 -- 172.21.15.27:0/1885921778 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c007b80 con 0x7f4384003b90 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.907+0000 7f438affd640 1 -- 172.21.15.27:0/1885921778 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c007480 con 0x7f4384003b90 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.907+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439c0c7270 con 0x7f4384003b90 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.907+0000 7f438affd640 1 -- 172.21.15.27:0/1885921778 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c02c770 con 0x7f4384003b90 2024-03-06T07:35:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.907+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439c0c7830 con 0x7f4384003b90 2024-03-06T07:35:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.908+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439c1e5010 con 0x7f4384003b90 2024-03-06T07:35:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.908+0000 7f438affd640 1 -- 172.21.15.27:0/1885921778 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f438c02c910 con 0x7f4384003b90 2024-03-06T07:35:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.909+0000 7f438affd640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437c06b500 0x7f437c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.909+0000 7f438affd640 1 -- 172.21.15.27:0/1885921778 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f438c0ab670 con 0x7f4384003b90 2024-03-06T07:35:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.909+0000 7f4399d9b640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437c06b500 0x7f437c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.910+0000 7f4399d9b640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437c06b500 0x7f437c06d9c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4390000c00 tx=0x7f4390009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:11.918+0000 7f438affd640 1 -- 172.21.15.27:0/1885921778 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f438c07a4d0 con 0x7f4384003b90 2024-03-06T07:35:12.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.246+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f439c068700 con 0x7f4384003b90 2024-03-06T07:35:12.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.246+0000 7f438affd640 1 -- 172.21.15.27:0/1885921778 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f438c079dd0 con 0x7f4384003b90 2024-03-06T07:35:12.247 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:12.248 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:12.248 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:12.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437c06b500 msgr2=0x7f437c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437c06b500 0x7f437c06d9c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4390000c00 tx=0x7f4390009040 comp rx=0 tx=0).stop 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 msgr2=0x7f439c1defc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f439c1defc0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f438c00c1d0 tx=0x7f438c02e070 comp rx=0 tx=0).stop 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 shutdown_connections 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f437c06b500 0x7f437c06d9c0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f439c1e4650 0x7f439c1e4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f439c0c9df0 0x7f439c0c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 --2- 172.21.15.27:0/1885921778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4384003b90 0x7f439c1defc0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 >> 172.21.15.27:0/1885921778 conn(0x7f439c0b36d0 msgr2=0x7f439c0b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 shutdown_connections 2024-03-06T07:35:12.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:12.248+0000 7f43a0c8d640 1 -- 172.21.15.27:0/1885921778 wait complete. 2024-03-06T07:35:12.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:12.462 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:13.463 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.569+0000 7fe9f1281640 1 Processor -- start 2024-03-06T07:35:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.569+0000 7fe9f1281640 1 -- start start 2024-03-06T07:35:13.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9f1281640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9ec0aee10 0x7fe9ec15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9f1281640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ec0729b0 con 0x7fe9ec06d400 2024-03-06T07:35:13.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9f1281640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ec06c920 con 0x7fe9ec15bac0 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9f1281640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9ec06cf70 con 0x7fe9ec0aee10 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9eb7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe9ec06d400 0x7fe9ec070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39912/0 (socket says 172.21.15.27:39912) 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/505922685 learned_addr learned my addr 172.21.15.27:0/505922685 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9ea7fc640 1 --2- 172.21.15.27:0/505922685 >> v2:172.21.15.69:3300/0 conn(0x7fe9ec0aee10 0x7fe9ec15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274789308 0 0) 0x7fe9ec0729b0 con 0x7fe9ec06d400 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d4003520 con 0x7fe9ec06d400 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3822987734 0 0) 0x7fe9d4003520 con 0x7fe9ec06d400 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 >> v2:172.21.15.69:3300/0 conn(0x7fe9ec0aee10 msgr2=0x7fe9ec15cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9e9ffb640 1 --2- 172.21.15.27:0/505922685 >> v2:172.21.15.69:3300/0 conn(0x7fe9ec0aee10 0x7fe9ec15cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.570+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 >> v1:172.21.15.69:6789/0 conn(0x7fe9ec15bac0 legacy=0x7fe9ec0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec06cc00 con 0x7fe9ec06d400 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fe9e40019f0 con 0x7fe9ec06d400 2024-03-06T07:35:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 >> v1:172.21.15.27:6789/0 conn(0x7fe9ec06d400 legacy=0x7fe9ec070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9e9ffb640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d40039b0 0x7fe9d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9e9ffb640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d4004800 0x7fe9d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9e9ffb640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d40051c0 0x7fe9d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec06cf70 con 0x7fe9d40051c0 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d400c060 con 0x7fe9d4004800 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d400c260 con 0x7fe9d40039b0 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9eb7fe640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d40051c0 0x7fe9d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9ea7fc640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d4004800 0x7fe9d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9eaffd640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d40039b0 0x7fe9d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/505922685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d40039b0 msgr2=0x7fe9d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9eb7fe640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9d40039b0 0x7fe9d4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/505922685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d4004800 msgr2=0x7fe9d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9eb7fe640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d4004800 0x7fe9d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.571+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/505922685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d4003520 con 0x7fe9d40051c0 2024-03-06T07:35:13.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.572+0000 7fe9eb7fe640 1 -- 172.21.15.27:0/505922685 reap_dead start 2024-03-06T07:35:13.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.572+0000 7fe9eb7fe640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d40051c0 0x7fe9d400b9c0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400a840 tx=0x7fe9d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731a83ebd72da488 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.572+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e400a460 con 0x7fe9d40051c0 2024-03-06T07:35:13.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.572+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e4034650 con 0x7fe9d40051c0 2024-03-06T07:35:13.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.572+0000 7fe9e9ffb640 1 -- 172.21.15.27:0/505922685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4034a40 con 0x7fe9d40051c0 2024-03-06T07:35:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.573+0000 7fe9f1281640 1 -- 172.21.15.27:0/505922685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d40051c0 msgr2=0x7fe9d400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.573+0000 7fe9f1281640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d40051c0 0x7fe9d400b9c0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400a840 tx=0x7fe9d4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.573+0000 7fe9f1281640 1 -- 172.21.15.27:0/505922685 shutdown_connections 2024-03-06T07:35:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.573+0000 7fe9f1281640 1 --2- 172.21.15.27:0/505922685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d40051c0 0x7fe9d400b9c0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.573+0000 7fe9f1281640 1 -- 172.21.15.27:0/505922685 >> 172.21.15.27:0/505922685 conn(0x7fe9ec066d00 msgr2=0x7fe9ec0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.573+0000 7fe9f1281640 1 -- 172.21.15.27:0/505922685 shutdown_connections 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.573+0000 7fe9f1281640 1 -- 172.21.15.27:0/505922685 wait complete. 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.574+0000 7fe9f1281640 1 Processor -- start 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.574+0000 7fe9f1281640 1 -- start start 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9f1281640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d40051c0 0x7fe9ec1433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9f1281640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec155bf0 0x7fe9ec143dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9f1281640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9ec144310 0x7fe9ec154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9f1281640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec15c020 con 0x7fe9ec155bf0 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9f1281640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec15bea0 con 0x7fe9d40051c0 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9f1281640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec072580 con 0x7fe9ec144310 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9ea7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec155bf0 0x7fe9ec143dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9ea7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec155bf0 0x7fe9ec143dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42694/0 (socket says 172.21.15.27:42694) 2024-03-06T07:35:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9ea7fc640 1 -- 172.21.15.27:0/1250548709 learned_addr learned my addr 172.21.15.27:0/1250548709 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9eaffd640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d40051c0 0x7fe9ec1433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9eb7fe640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9ec144310 0x7fe9ec154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9ea7fc640 1 -- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9ec144310 msgr2=0x7fe9ec154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9ea7fc640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9ec144310 0x7fe9ec154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9ea7fc640 1 -- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d40051c0 msgr2=0x7fe9ec1433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9ea7fc640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d40051c0 0x7fe9ec1433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9ea7fc640 1 -- 172.21.15.27:0/1250548709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec153ed0 con 0x7fe9ec155bf0 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.575+0000 7fe9ea7fc640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec155bf0 0x7fe9ec143dd0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00118e0 tx=0x7fe9e0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.576+0000 7fe9d37fe640 1 -- 172.21.15.27:0/1250548709 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e000faf0 con 0x7fe9ec155bf0 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.576+0000 7fe9d37fe640 1 -- 172.21.15.27:0/1250548709 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e000fc90 con 0x7fe9ec155bf0 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.576+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec1541b0 con 0x7fe9ec155bf0 2024-03-06T07:35:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.576+0000 7fe9d37fe640 1 -- 172.21.15.27:0/1250548709 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0014440 con 0x7fe9ec155bf0 2024-03-06T07:35:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.576+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec154770 con 0x7fe9ec155bf0 2024-03-06T07:35:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.577+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec067ff0 con 0x7fe9ec155bf0 2024-03-06T07:35:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.577+0000 7fe9d37fe640 1 -- 172.21.15.27:0/1250548709 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9e0012040 con 0x7fe9ec155bf0 2024-03-06T07:35:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.578+0000 7fe9d37fe640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9d806b500 0x7fe9d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.578+0000 7fe9d37fe640 1 -- 172.21.15.27:0/1250548709 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe9e008a8f0 con 0x7fe9ec155bf0 2024-03-06T07:35:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.578+0000 7fe9eaffd640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9d806b500 0x7fe9d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.579+0000 7fe9eaffd640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9d806b500 0x7fe9d806d9c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc000c00 tx=0x7fe9dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.583+0000 7fe9d37fe640 1 -- 172.21.15.27:0/1250548709 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9e0019030 con 0x7fe9ec155bf0 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.800+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9ec06de50 con 0x7fe9ec155bf0 2024-03-06T07:35:13.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.801+0000 7fe9d37fe640 1 -- 172.21.15.27:0/1250548709 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fe9e0059750 con 0x7fe9ec155bf0 2024-03-06T07:35:13.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:13.802 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:13.802 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9d806b500 msgr2=0x7fe9d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9d806b500 0x7fe9d806d9c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc000c00 tx=0x7fe9dc009040 comp rx=0 tx=0).stop 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec155bf0 msgr2=0x7fe9ec143dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec155bf0 0x7fe9ec143dd0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00118e0 tx=0x7fe9e0011da0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 shutdown_connections 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe9d806b500 0x7fe9d806d9c0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe9ec144310 0x7fe9ec154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec155bf0 0x7fe9ec143dd0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 --2- 172.21.15.27:0/1250548709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9d40051c0 0x7fe9ec1433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.802+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 >> 172.21.15.27:0/1250548709 conn(0x7fe9ec066d00 msgr2=0x7fe9ec0cfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.803+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 shutdown_connections 2024-03-06T07:35:13.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:13.803+0000 7fe9f1281640 1 -- 172.21.15.27:0/1250548709 wait complete. 2024-03-06T07:35:14.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:14.016 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:15.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:15.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.124+0000 7f2de3f81640 1 Processor -- start 2024-03-06T07:35:15.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.124+0000 7f2de3f81640 1 -- start start 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de3f81640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ddc15df00 0x7f2ddc0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de3f81640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ddc0b9a30 0x7f2ddc0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de3f81640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc0cea30 con 0x7f2ddc0b95a0 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de3f81640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc0d2af0 con 0x7f2ddc15df00 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de3f81640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc0cf3c0 con 0x7f2ddc0b9a30 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de1d1d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ddc15df00 0x7f2ddc0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de1d1d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ddc15df00 0x7f2ddc0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42700/0 (socket says 172.21.15.27:42700) 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de1d1d640 1 -- 172.21.15.27:0/133925892 learned_addr learned my addr 172.21.15.27:0/133925892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de251e640 1 --2- 172.21.15.27:0/133925892 >> v2:172.21.15.69:3300/0 conn(0x7f2ddc0b9a30 0x7f2ddc0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de1d1d640 1 -- 172.21.15.27:0/133925892 >> v2:172.21.15.69:3300/0 conn(0x7f2ddc0b9a30 msgr2=0x7f2ddc0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de1d1d640 1 --2- 172.21.15.27:0/133925892 >> v2:172.21.15.69:3300/0 conn(0x7f2ddc0b9a30 0x7f2ddc0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de1d1d640 1 -- 172.21.15.27:0/133925892 >> v1:172.21.15.69:6789/0 conn(0x7f2ddc0b95a0 legacy=0x7f2ddc0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.125+0000 7f2de1d1d640 1 -- 172.21.15.27:0/133925892 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc0cf0e0 con 0x7f2ddc15df00 2024-03-06T07:35:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.126+0000 7f2de1d1d640 1 --2- 172.21.15.27:0/133925892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15df00 0x7f2ddc0d25b0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0d4d20 tx=0x7f2dd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4d698929644d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.126+0000 7f2de0d1b640 1 -- 172.21.15.27:0/133925892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8035020 con 0x7f2ddc15df00 2024-03-06T07:35:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.126+0000 7f2de0d1b640 1 -- 172.21.15.27:0/133925892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd8031870 con 0x7f2ddc15df00 2024-03-06T07:35:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.126+0000 7f2de0d1b640 1 -- 172.21.15.27:0/133925892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8031c40 con 0x7f2ddc15df00 2024-03-06T07:35:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.127+0000 7f2de3f81640 1 -- 172.21.15.27:0/133925892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15df00 msgr2=0x7f2ddc0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.127+0000 7f2de3f81640 1 --2- 172.21.15.27:0/133925892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15df00 0x7f2ddc0d25b0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0d4d20 tx=0x7f2dd802b910 comp rx=0 tx=0).stop 2024-03-06T07:35:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.127+0000 7f2de3f81640 1 -- 172.21.15.27:0/133925892 shutdown_connections 2024-03-06T07:35:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.127+0000 7f2de3f81640 1 --2- 172.21.15.27:0/133925892 >> v2:172.21.15.69:3300/0 conn(0x7f2ddc0b9a30 0x7f2ddc0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.127+0000 7f2de3f81640 1 --2- 172.21.15.27:0/133925892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15df00 0x7f2ddc0d25b0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.127+0000 7f2de3f81640 1 -- 172.21.15.27:0/133925892 >> 172.21.15.27:0/133925892 conn(0x7f2ddc071960 msgr2=0x7f2ddc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.127+0000 7f2de3f81640 1 -- 172.21.15.27:0/133925892 shutdown_connections 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.127+0000 7f2de3f81640 1 -- 172.21.15.27:0/133925892 wait complete. 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.128+0000 7f2de3f81640 1 Processor -- start 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.128+0000 7f2de3f81640 1 -- start start 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de3f81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ddc0b95a0 0x7f2ddc15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de3f81640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ddc15df00 0x7f2ddc0c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de3f81640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15f0b0 0x7f2ddc0c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de3f81640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc0cec70 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de3f81640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc0ce750 con 0x7f2ddc0b95a0 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de3f81640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc0cedf0 con 0x7f2ddc15df00 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de251e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15f0b0 0x7f2ddc0c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de251e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15f0b0 0x7f2ddc0c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42712/0 (socket says 172.21.15.27:42712) 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de251e640 1 -- 172.21.15.27:0/2665639872 learned_addr learned my addr 172.21.15.27:0/2665639872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de1d1d640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ddc0b95a0 0x7f2ddc15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de151c640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ddc15df00 0x7f2ddc0c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de251e640 1 -- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ddc15df00 msgr2=0x7f2ddc0c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de251e640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ddc15df00 0x7f2ddc0c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de251e640 1 -- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ddc0b95a0 msgr2=0x7f2ddc15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de251e640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ddc0b95a0 0x7f2ddc15aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de251e640 1 -- 172.21.15.27:0/2665639872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc0c7930 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de1d1d640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ddc0b95a0 0x7f2ddc15aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.129+0000 7f2de251e640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15f0b0 0x7f2ddc0c9c40 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4007e40 tx=0x7f2dd400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.130+0000 7f2dd27fc640 1 -- 172.21.15.27:0/2665639872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4014620 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.130+0000 7f2dd27fc640 1 -- 172.21.15.27:0/2665639872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd4019070 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.130+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc0c7c10 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.130+0000 7f2dd27fc640 1 -- 172.21.15.27:0/2665639872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4014a20 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.130+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc15fc70 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.131+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc15fe70 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.131+0000 7f2dd27fc640 1 -- 172.21.15.27:0/2665639872 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dd4014bc0 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.132+0000 7f2dd27fc640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dc006b4d0 0x7f2dc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.132+0000 7f2dd27fc640 1 -- 172.21.15.27:0/2665639872 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2dd408ab60 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.132+0000 7f2de1d1d640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dc006b4d0 0x7f2dc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.133+0000 7f2de1d1d640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dc006b4d0 0x7f2dc006d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8036000 tx=0x7f2dd803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.136+0000 7f2dd27fc640 1 -- 172.21.15.27:0/2665639872 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ddc15fe70 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.355+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ddc0c7de0 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.355+0000 7f2dd27fc640 1 -- 172.21.15.27:0/2665639872 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f2ddc0c7de0 con 0x7f2ddc15f0b0 2024-03-06T07:35:15.357 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:15.357 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dc006b4d0 msgr2=0x7f2dc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dc006b4d0 0x7f2dc006d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8036000 tx=0x7f2dd803c040 comp rx=0 tx=0).stop 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15f0b0 msgr2=0x7f2ddc0c9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15f0b0 0x7f2ddc0c9c40 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4007e40 tx=0x7f2dd400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 shutdown_connections 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f2dc006b4d0 0x7f2dc006d990 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ddc15f0b0 0x7f2ddc0c9c40 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ddc15df00 0x7f2ddc0c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 --2- 172.21.15.27:0/2665639872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ddc0b95a0 0x7f2ddc15aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 >> 172.21.15.27:0/2665639872 conn(0x7f2ddc071960 msgr2=0x7f2ddc15bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 shutdown_connections 2024-03-06T07:35:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:15.357+0000 7f2de3f81640 1 -- 172.21.15.27:0/2665639872 wait complete. 2024-03-06T07:35:15.570 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:15.571 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:16.572 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.678+0000 7fc775bac640 1 Processor -- start 2024-03-06T07:35:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.678+0000 7fc775bac640 1 -- start start 2024-03-06T07:35:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.679+0000 7fc775bac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc77015df00 0x7fc7700d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.679+0000 7fc775bac640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7700cea30 con 0x7fc7700b95a0 2024-03-06T07:35:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.679+0000 7fc775bac640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7700d2af0 con 0x7fc7700b9a30 2024-03-06T07:35:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.679+0000 7fc775bac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7700cf3c0 con 0x7fc77015df00 2024-03-06T07:35:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.679+0000 7fc76f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc77015df00 0x7fc7700d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.679+0000 7fc76effd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc7700b95a0 0x7fc7700b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39924/0 (socket says 172.21.15.27:39924) 2024-03-06T07:35:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.679+0000 7fc76effd640 1 -- 172.21.15.27:0/1552331408 learned_addr learned my addr 172.21.15.27:0/1552331408 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.680+0000 7fc76f7fe640 1 -- 172.21.15.27:0/1552331408 >> v1:172.21.15.69:6789/0 conn(0x7fc7700b9a30 legacy=0x7fc7700d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.680+0000 7fc76f7fe640 1 -- 172.21.15.27:0/1552331408 >> v1:172.21.15.27:6789/0 conn(0x7fc7700b95a0 legacy=0x7fc7700b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.680+0000 7fc76f7fe640 1 -- 172.21.15.27:0/1552331408 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7700cf0e0 con 0x7fc77015df00 2024-03-06T07:35:16.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.680+0000 7fc76f7fe640 1 --2- 172.21.15.27:0/1552331408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 0x7fc7700d25b0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fc7580097a0 tx=0x7fc75802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c7c2a3f0588559 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.681+0000 7fc76e7fc640 1 -- 172.21.15.27:0/1552331408 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758034070 con 0x7fc77015df00 2024-03-06T07:35:16.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.681+0000 7fc76e7fc640 1 -- 172.21.15.27:0/1552331408 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc758030610 con 0x7fc77015df00 2024-03-06T07:35:16.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.681+0000 7fc76e7fc640 1 -- 172.21.15.27:0/1552331408 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7580309e0 con 0x7fc77015df00 2024-03-06T07:35:16.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.681+0000 7fc775bac640 1 -- 172.21.15.27:0/1552331408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 msgr2=0x7fc7700d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.681+0000 7fc775bac640 1 --2- 172.21.15.27:0/1552331408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 0x7fc7700d25b0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fc7580097a0 tx=0x7fc75802b960 comp rx=0 tx=0).stop 2024-03-06T07:35:16.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.681+0000 7fc775bac640 1 -- 172.21.15.27:0/1552331408 shutdown_connections 2024-03-06T07:35:16.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.681+0000 7fc775bac640 1 --2- 172.21.15.27:0/1552331408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 0x7fc7700d25b0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.681+0000 7fc775bac640 1 -- 172.21.15.27:0/1552331408 >> 172.21.15.27:0/1552331408 conn(0x7fc770071960 msgr2=0x7fc770071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:16.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.682+0000 7fc775bac640 1 -- 172.21.15.27:0/1552331408 shutdown_connections 2024-03-06T07:35:16.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.682+0000 7fc775bac640 1 -- 172.21.15.27:0/1552331408 wait complete. 2024-03-06T07:35:16.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.683+0000 7fc775bac640 1 Processor -- start 2024-03-06T07:35:16.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.684+0000 7fc775bac640 1 -- start start 2024-03-06T07:35:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc775bac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7700b95a0 0x7fc7700cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc775bac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 0x7fc770160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc775bac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc77015b4c0 0x7fc7701f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc775bac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700cec70 con 0x7fc77015df00 2024-03-06T07:35:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc775bac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700ce750 con 0x7fc7700b95a0 2024-03-06T07:35:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc775bac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700cedf0 con 0x7fc77015b4c0 2024-03-06T07:35:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc76effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 0x7fc770160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc76effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 0x7fc770160720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42732/0 (socket says 172.21.15.27:42732) 2024-03-06T07:35:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc76effd640 1 -- 172.21.15.27:0/250901410 learned_addr learned my addr 172.21.15.27:0/250901410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc76f7fe640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7700b95a0 0x7fc7700cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.685+0000 7fc76ffff640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc77015b4c0 0x7fc7701f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc76effd640 1 -- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc77015b4c0 msgr2=0x7fc7701f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc76effd640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc77015b4c0 0x7fc7701f3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc76effd640 1 -- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7700b95a0 msgr2=0x7fc7700cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc76effd640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7700b95a0 0x7fc7700cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc76effd640 1 -- 172.21.15.27:0/250901410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7700cc4a0 con 0x7fc77015df00 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc76effd640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 0x7fc770160720 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fc76000c4b0 tx=0x7fc76000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc767fff640 1 -- 172.21.15.27:0/250901410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760017070 con 0x7fc77015df00 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc767fff640 1 -- 172.21.15.27:0/250901410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc760005ce0 con 0x7fc77015df00 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7700cc780 con 0x7fc77015df00 2024-03-06T07:35:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc767fff640 1 -- 172.21.15.27:0/250901410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760007270 con 0x7fc77015df00 2024-03-06T07:35:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.686+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7700ccd40 con 0x7fc77015df00 2024-03-06T07:35:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.687+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc770067ff0 con 0x7fc77015df00 2024-03-06T07:35:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.688+0000 7fc767fff640 1 -- 172.21.15.27:0/250901410 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc76001a070 con 0x7fc77015df00 2024-03-06T07:35:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.688+0000 7fc767fff640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc75006b4d0 0x7fc75006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.688+0000 7fc767fff640 1 -- 172.21.15.27:0/250901410 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc760089440 con 0x7fc77015df00 2024-03-06T07:35:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.688+0000 7fc76f7fe640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc75006b4d0 0x7fc75006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.689+0000 7fc76f7fe640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc75006b4d0 0x7fc75006d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc758036000 tx=0x7fc75803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.692+0000 7fc767fff640 1 -- 172.21.15.27:0/250901410 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc760057c50 con 0x7fc77015df00 2024-03-06T07:35:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.914+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7700b2bd0 con 0x7fc77015df00 2024-03-06T07:35:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.916+0000 7fc767fff640 1 -- 172.21.15.27:0/250901410 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fc76005bcf0 con 0x7fc77015df00 2024-03-06T07:35:16.917 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:16.917 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc75006b4d0 msgr2=0x7fc75006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc75006b4d0 0x7fc75006d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc758036000 tx=0x7fc75803b040 comp rx=0 tx=0).stop 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 msgr2=0x7fc770160720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 0x7fc770160720 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fc76000c4b0 tx=0x7fc76000c970 comp rx=0 tx=0).stop 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 shutdown_connections 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc75006b4d0 0x7fc75006d990 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc77015b4c0 0x7fc7701f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc77015df00 0x7fc770160720 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 --2- 172.21.15.27:0/250901410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7700b95a0 0x7fc7700cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 >> 172.21.15.27:0/250901410 conn(0x7fc770071960 msgr2=0x7fc7700b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 shutdown_connections 2024-03-06T07:35:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:16.918+0000 7fc775bac640 1 -- 172.21.15.27:0/250901410 wait complete. 2024-03-06T07:35:17.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:17.132 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:18.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.239+0000 7f668277a640 1 Processor -- start 2024-03-06T07:35:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.239+0000 7f668277a640 1 -- start start 2024-03-06T07:35:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f668277a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f668277a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f667c0b2b50 0x7f667c0b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f668277a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c0ce610 con 0x7f667c071540 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f668277a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667c0d2f70 con 0x7f667c0b2b50 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f668277a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c0d4be0 con 0x7f667c0d3400 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f667b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f6680d17640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f667c0b2b50 0x7f667c0b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f6680d17640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f667c0b2b50 0x7f667c0b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:39046/0 (socket says 172.21.15.27:39046) 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f667bfff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f667c071540 0x7f667c071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:38130/0 (socket says 172.21.15.27:38130) 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f6680d17640 1 -- 172.21.15.27:0/2891553146 learned_addr learned my addr 172.21.15.27:0/2891553146 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f6680d17640 1 -- 172.21.15.27:0/2891553146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 msgr2=0x7f667c0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f6680d17640 1 --2- 172.21.15.27:0/2891553146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f6680d17640 1 -- 172.21.15.27:0/2891553146 >> v1:172.21.15.69:6789/0 conn(0x7f667c071540 legacy=0x7f667c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f6680d17640 1 -- 172.21.15.27:0/2891553146 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c0d2920 con 0x7f667c0b2b50 2024-03-06T07:35:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.240+0000 7f667b7fe640 1 --2- 172.21.15.27:0/2891553146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.241+0000 7f6680d17640 1 --2- 172.21.15.27:0/2891553146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 0x7f667c0b6e20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6674009e50 tx=0x7f667402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=859c419514bc88aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.241+0000 7f667affd640 1 -- 172.21.15.27:0/2891553146 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667402cb00 con 0x7f667c0b2b50 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.241+0000 7f667affd640 1 -- 172.21.15.27:0/2891553146 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667402cca0 con 0x7f667c0b2b50 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.241+0000 7f667affd640 1 -- 172.21.15.27:0/2891553146 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6674034610 con 0x7f667c0b2b50 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.242+0000 7f668277a640 1 -- 172.21.15.27:0/2891553146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 msgr2=0x7f667c0b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.242+0000 7f668277a640 1 --2- 172.21.15.27:0/2891553146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 0x7f667c0b6e20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6674009e50 tx=0x7f667402bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.242+0000 7f668277a640 1 -- 172.21.15.27:0/2891553146 shutdown_connections 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.242+0000 7f668277a640 1 --2- 172.21.15.27:0/2891553146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 0x7f667c0b6e20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.242+0000 7f668277a640 1 --2- 172.21.15.27:0/2891553146 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.242+0000 7f668277a640 1 -- 172.21.15.27:0/2891553146 >> 172.21.15.27:0/2891553146 conn(0x7f667c0d3ca0 msgr2=0x7f667c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:18.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.242+0000 7f668277a640 1 -- 172.21.15.27:0/2891553146 shutdown_connections 2024-03-06T07:35:18.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.242+0000 7f668277a640 1 -- 172.21.15.27:0/2891553146 wait complete. 2024-03-06T07:35:18.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.243+0000 7f668277a640 1 Processor -- start 2024-03-06T07:35:18.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.243+0000 7f668277a640 1 -- start start 2024-03-06T07:35:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f668277a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f667c071540 0x7f667c0ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f668277a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 0x7f667c151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f668277a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c1f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f668277a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c15e370 con 0x7f667c071540 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f668277a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c0ce8f0 con 0x7f667c0b2b50 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f668277a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c0aa9f0 con 0x7f667c0d3400 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f667bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f667c071540 0x7f667c0ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f667bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f667c071540 0x7f667c0ac510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42744/0 (socket says 172.21.15.27:42744) 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f667bfff640 1 -- 172.21.15.27:0/2610767627 learned_addr learned my addr 172.21.15.27:0/2610767627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f6680d17640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c1f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.244+0000 7f667b7fe640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 0x7f667c151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f667bfff640 1 -- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 msgr2=0x7f667c1f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f667bfff640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c1f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f667bfff640 1 -- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 msgr2=0x7f667c151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f667bfff640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 0x7f667c151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f667bfff640 1 -- 172.21.15.27:0/2610767627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c0aca80 con 0x7f667c071540 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f6680d17640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c1f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f667b7fe640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 0x7f667c151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f667bfff640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f667c071540 0x7f667c0ac510 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f666c00c430 tx=0x7f666c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f6678ff9640 1 -- 172.21.15.27:0/2610767627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c0074e0 con 0x7f667c071540 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f6678ff9640 1 -- 172.21.15.27:0/2610767627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c007680 con 0x7f667c071540 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f6678ff9640 1 -- 172.21.15.27:0/2610767627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c004a00 con 0x7f667c071540 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667c1f57c0 con 0x7f667c071540 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.245+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667c1f6000 con 0x7f667c071540 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.246+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667c068700 con 0x7f667c071540 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.247+0000 7f6678ff9640 1 -- 172.21.15.27:0/2610767627 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f666c01a020 con 0x7f667c071540 2024-03-06T07:35:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.247+0000 7f6678ff9640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f666806b4d0 0x7f666806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.247+0000 7f6678ff9640 1 -- 172.21.15.27:0/2610767627 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f666c089a30 con 0x7f667c071540 2024-03-06T07:35:18.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.248+0000 7f667b7fe640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f666806b4d0 0x7f666806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.248+0000 7f667b7fe640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f666806b4d0 0x7f666806d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f667c1e7a60 tx=0x7f6670008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.252+0000 7f6678ff9640 1 -- 172.21.15.27:0/2610767627 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f667c1f57c0 con 0x7f667c071540 2024-03-06T07:35:18.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.470+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f667c074400 con 0x7f667c071540 2024-03-06T07:35:18.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.471+0000 7f6678ff9640 1 -- 172.21.15.27:0/2610767627 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f666c058890 con 0x7f667c071540 2024-03-06T07:35:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:18.472 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:18.473 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:18.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.472+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f666806b4d0 msgr2=0x7f666806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.472+0000 7f668277a640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f666806b4d0 0x7f666806d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f667c1e7a60 tx=0x7f6670008040 comp rx=0 tx=0).stop 2024-03-06T07:35:18.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.472+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f667c071540 msgr2=0x7f667c0ac510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.472+0000 7f668277a640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f667c071540 0x7f667c0ac510 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f666c00c430 tx=0x7f666c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.473+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 shutdown_connections 2024-03-06T07:35:18.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.473+0000 7f668277a640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f666806b4d0 0x7f666806d990 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.473+0000 7f668277a640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f667c0d3400 0x7f667c1f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.473+0000 7f668277a640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f667c0b2b50 0x7f667c151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.473+0000 7f668277a640 1 --2- 172.21.15.27:0/2610767627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f667c071540 0x7f667c0ac510 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.473+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 >> 172.21.15.27:0/2610767627 conn(0x7f667c0d3ca0 msgr2=0x7f667c0ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:18.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.473+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 shutdown_connections 2024-03-06T07:35:18.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:18.473+0000 7f668277a640 1 -- 172.21.15.27:0/2610767627 wait complete. 2024-03-06T07:35:18.686 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:18.686 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:19.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.793+0000 7f26bb89d640 1 Processor -- start 2024-03-06T07:35:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.793+0000 7f26bb89d640 1 -- start start 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26bb89d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26b40d3400 0x7f26b40b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26bb89d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b40b2b50 0x7f26b40b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26bb89d640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b40ce610 con 0x7f26b4071540 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26bb89d640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b40d2f70 con 0x7f26b40d3400 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26bb89d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40d4be0 con 0x7f26b40b2b50 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b8e38640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26b40d3400 0x7f26b40b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b8e38640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26b40d3400 0x7f26b40b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42756/0 (socket says 172.21.15.27:42756) 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b8e38640 1 -- 172.21.15.27:0/1310589509 learned_addr learned my addr 172.21.15.27:0/1310589509 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b9639640 1 --1- 172.21.15.27:0/1310589509 >> v1:172.21.15.27:6789/0 conn(0x7f26b4071540 0x7f26b4071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39940/0 (socket says 172.21.15.27:39940) 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b9e3a640 1 --2- 172.21.15.27:0/1310589509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b40b2b50 0x7f26b40b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b8e38640 1 -- 172.21.15.27:0/1310589509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b40b2b50 msgr2=0x7f26b40b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b8e38640 1 --2- 172.21.15.27:0/1310589509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b40b2b50 0x7f26b40b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b8e38640 1 -- 172.21.15.27:0/1310589509 >> v1:172.21.15.27:6789/0 conn(0x7f26b4071540 legacy=0x7f26b4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26abfff640 1 -- 172.21.15.27:0/1310589509 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294303712 0 0) 0x7f26b40ce610 con 0x7f26b4071540 2024-03-06T07:35:19.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b8e38640 1 -- 172.21.15.27:0/1310589509 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b40d2920 con 0x7f26b40d3400 2024-03-06T07:35:19.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.794+0000 7f26b9e3a640 1 --2- 172.21.15.27:0/1310589509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b40b2b50 0x7f26b40b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.795+0000 7f26b8e38640 1 --2- 172.21.15.27:0/1310589509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40d3400 0x7f26b40b30a0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009870 tx=0x7f26a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427932f2bc1c1c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.795+0000 7f26abfff640 1 -- 172.21.15.27:0/1310589509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4034070 con 0x7f26b40d3400 2024-03-06T07:35:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.795+0000 7f26abfff640 1 -- 172.21.15.27:0/1310589509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a4030840 con 0x7f26b40d3400 2024-03-06T07:35:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.795+0000 7f26abfff640 1 -- 172.21.15.27:0/1310589509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4030c10 con 0x7f26b40d3400 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.796+0000 7f26bb89d640 1 -- 172.21.15.27:0/1310589509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40d3400 msgr2=0x7f26b40b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.796+0000 7f26bb89d640 1 --2- 172.21.15.27:0/1310589509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40d3400 0x7f26b40b30a0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009870 tx=0x7f26a402b980 comp rx=0 tx=0).stop 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.796+0000 7f26bb89d640 1 -- 172.21.15.27:0/1310589509 shutdown_connections 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.796+0000 7f26bb89d640 1 --2- 172.21.15.27:0/1310589509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b40b2b50 0x7f26b40b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.796+0000 7f26bb89d640 1 --2- 172.21.15.27:0/1310589509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40d3400 0x7f26b40b30a0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.796+0000 7f26bb89d640 1 -- 172.21.15.27:0/1310589509 >> 172.21.15.27:0/1310589509 conn(0x7f26b40d3ca0 msgr2=0x7f26b40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.796+0000 7f26bb89d640 1 -- 172.21.15.27:0/1310589509 shutdown_connections 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.796+0000 7f26bb89d640 1 -- 172.21.15.27:0/1310589509 wait complete. 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.797+0000 7f26bb89d640 1 Processor -- start 2024-03-06T07:35:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.797+0000 7f26bb89d640 1 -- start start 2024-03-06T07:35:19.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26bb89d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b4071540 0x7f26b4157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26bb89d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40b2b50 0x7f26b4157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26bb89d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b40d3400 0x7f26b40b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26bb89d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b415e370 con 0x7f26b40b2b50 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26bb89d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40ce8f0 con 0x7f26b40d3400 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26bb89d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4158cc0 con 0x7f26b4071540 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b8e38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40b2b50 0x7f26b4157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b8e38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40b2b50 0x7f26b4157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42764/0 (socket says 172.21.15.27:42764) 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b8e38640 1 -- 172.21.15.27:0/55333818 learned_addr learned my addr 172.21.15.27:0/55333818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b8e38640 1 -- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b4071540 msgr2=0x7f26b4157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b9639640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b4071540 0x7f26b4157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b9e3a640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b40d3400 0x7f26b40b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b8e38640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b4071540 0x7f26b4157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b8e38640 1 -- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b40d3400 msgr2=0x7f26b40b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b8e38640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b40d3400 0x7f26b40b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b8e38640 1 -- 172.21.15.27:0/55333818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b4158420 con 0x7f26b40b2b50 2024-03-06T07:35:19.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b9e3a640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b40d3400 0x7f26b40b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26b8e38640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40b2b50 0x7f26b4157950 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f26a40090b0 tx=0x7f26a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.798+0000 7f26a9ffb640 1 -- 172.21.15.27:0/55333818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4034040 con 0x7f26b40b2b50 2024-03-06T07:35:19.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.799+0000 7f26a9ffb640 1 -- 172.21.15.27:0/55333818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a403ad90 con 0x7f26b40b2b50 2024-03-06T07:35:19.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.799+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b4158700 con 0x7f26b40b2b50 2024-03-06T07:35:19.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.799+0000 7f26a9ffb640 1 -- 172.21.15.27:0/55333818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4004270 con 0x7f26b40b2b50 2024-03-06T07:35:19.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.799+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b4157e90 con 0x7f26b40b2b50 2024-03-06T07:35:19.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.800+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b4068700 con 0x7f26b40b2b50 2024-03-06T07:35:19.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.800+0000 7f26a9ffb640 1 -- 172.21.15.27:0/55333818 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26a404b060 con 0x7f26b40b2b50 2024-03-06T07:35:19.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.801+0000 7f26a9ffb640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f269806b4d0 0x7f269806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.801+0000 7f26a9ffb640 1 -- 172.21.15.27:0/55333818 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f26a40ac160 con 0x7f26b40b2b50 2024-03-06T07:35:19.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.801+0000 7f26b9639640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f269806b4d0 0x7f269806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.802+0000 7f26b9639640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f269806b4d0 0x7f269806d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f26b0007fd0 tx=0x7f26b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:19.805+0000 7f26a9ffb640 1 -- 172.21.15.27:0/55333818 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26a403f070 con 0x7f26b40b2b50 2024-03-06T07:35:20.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.023+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26b40caae0 con 0x7f26b40b2b50 2024-03-06T07:35:20.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.024+0000 7f26a9ffb640 1 -- 172.21.15.27:0/55333818 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f26a407b050 con 0x7f26b40b2b50 2024-03-06T07:35:20.025 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:20.025 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:20.025 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:20.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f269806b4d0 msgr2=0x7f269806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f269806b4d0 0x7f269806d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f26b0007fd0 tx=0x7f26b0009040 comp rx=0 tx=0).stop 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40b2b50 msgr2=0x7f26b4157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40b2b50 0x7f26b4157950 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f26a40090b0 tx=0x7f26a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 shutdown_connections 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f269806b4d0 0x7f269806d990 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b40d3400 0x7f26b40b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b40b2b50 0x7f26b4157950 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 --2- 172.21.15.27:0/55333818 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26b4071540 0x7f26b4157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 >> 172.21.15.27:0/55333818 conn(0x7f26b40d3ca0 msgr2=0x7f26b40ced10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 shutdown_connections 2024-03-06T07:35:20.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:20.026+0000 7f26bb89d640 1 -- 172.21.15.27:0/55333818 wait complete. 2024-03-06T07:35:20.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:20.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:20.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:20.241 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:21.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.362+0000 7f26a60fb640 1 Processor -- start 2024-03-06T07:35:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.362+0000 7f26a60fb640 1 -- start start 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f26a60fb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f26a015e030 0x7f26a006ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f26a60fb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26a006ef70 0x7f26a00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f26a60fb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b2290 0x7f26a00d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f26a60fb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a0162700 con 0x7f26a006ef70 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f26a60fb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a015e410 con 0x7f26a015e030 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f26a60fb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0161af0 con 0x7f26a00b2290 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26a006ef70 0x7f26a00b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269effd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26a006ef70 0x7f26a00b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42776/0 (socket says 172.21.15.27:42776) 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269effd640 1 -- 172.21.15.27:0/1386614577 learned_addr learned my addr 172.21.15.27:0/1386614577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:21.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269f7fe640 1 --2- 172.21.15.27:0/1386614577 >> v2:172.21.15.69:3300/0 conn(0x7f26a015e030 0x7f26a006ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269ffff640 1 --2- 172.21.15.27:0/1386614577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b2290 0x7f26a00d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269effd640 1 -- 172.21.15.27:0/1386614577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b2290 msgr2=0x7f26a00d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269effd640 1 --2- 172.21.15.27:0/1386614577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b2290 0x7f26a00d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269effd640 1 -- 172.21.15.27:0/1386614577 >> v2:172.21.15.69:3300/0 conn(0x7f26a015e030 msgr2=0x7f26a006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269effd640 1 --2- 172.21.15.27:0/1386614577 >> v2:172.21.15.69:3300/0 conn(0x7f26a015e030 0x7f26a006ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.363+0000 7f269effd640 1 -- 172.21.15.27:0/1386614577 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0161dd0 con 0x7f26a006ef70 2024-03-06T07:35:21.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.364+0000 7f269effd640 1 --2- 172.21.15.27:0/1386614577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 0x7f26a00b1d50 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f26a0161a50 tx=0x7f269002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17254b66c68f586 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.364+0000 7f269e7fc640 1 -- 172.21.15.27:0/1386614577 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690035020 con 0x7f26a006ef70 2024-03-06T07:35:21.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.364+0000 7f269e7fc640 1 -- 172.21.15.27:0/1386614577 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2690031870 con 0x7f26a006ef70 2024-03-06T07:35:21.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.364+0000 7f269e7fc640 1 -- 172.21.15.27:0/1386614577 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690031c40 con 0x7f26a006ef70 2024-03-06T07:35:21.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.365+0000 7f26a60fb640 1 -- 172.21.15.27:0/1386614577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 msgr2=0x7f26a00b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.365+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1386614577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 0x7f26a00b1d50 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f26a0161a50 tx=0x7f269002b910 comp rx=0 tx=0).stop 2024-03-06T07:35:21.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.365+0000 7f26a60fb640 1 -- 172.21.15.27:0/1386614577 shutdown_connections 2024-03-06T07:35:21.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.365+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1386614577 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a00b2290 0x7f26a00d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.365+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1386614577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 0x7f26a00b1d50 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.365+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1386614577 >> v2:172.21.15.69:3300/0 conn(0x7f26a015e030 0x7f26a006ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.365+0000 7f26a60fb640 1 -- 172.21.15.27:0/1386614577 >> 172.21.15.27:0/1386614577 conn(0x7f26a00b55e0 msgr2=0x7f26a00b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.365+0000 7f26a60fb640 1 -- 172.21.15.27:0/1386614577 shutdown_connections 2024-03-06T07:35:21.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.365+0000 7f26a60fb640 1 -- 172.21.15.27:0/1386614577 wait complete. 2024-03-06T07:35:21.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.366+0000 7f26a60fb640 1 Processor -- start 2024-03-06T07:35:21.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.366+0000 7f26a60fb640 1 -- start start 2024-03-06T07:35:21.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f26a60fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 0x7f26a0146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f26a60fb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a00b2290 0x7f26a015b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f26a60fb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a015e030 0x7f26a015f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f26a60fb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a00b31e0 con 0x7f26a006ef70 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f26a60fb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0162390 con 0x7f26a00b2290 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f26a60fb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a00b3360 con 0x7f26a015e030 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 0x7f26a0146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 0x7f26a0146830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42786/0 (socket says 172.21.15.27:42786) 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269f7fe640 1 -- 172.21.15.27:0/1938654114 learned_addr learned my addr 172.21.15.27:0/1938654114 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269ffff640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a015e030 0x7f26a015f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269effd640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a00b2290 0x7f26a015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269f7fe640 1 -- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a015e030 msgr2=0x7f26a015f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269f7fe640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a015e030 0x7f26a015f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269f7fe640 1 -- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a00b2290 msgr2=0x7f26a015b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269f7fe640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a00b2290 0x7f26a015b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.367+0000 7f269f7fe640 1 -- 172.21.15.27:0/1938654114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a015b8b0 con 0x7f26a006ef70 2024-03-06T07:35:21.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.368+0000 7f269f7fe640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 0x7f26a0146830 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f269400e8a0 tx=0x7f269400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.368+0000 7f2687fff640 1 -- 172.21.15.27:0/1938654114 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694017070 con 0x7f26a006ef70 2024-03-06T07:35:21.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.368+0000 7f2687fff640 1 -- 172.21.15.27:0/1938654114 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269400c940 con 0x7f26a006ef70 2024-03-06T07:35:21.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.368+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a009c280 con 0x7f26a006ef70 2024-03-06T07:35:21.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.368+0000 7f2687fff640 1 -- 172.21.15.27:0/1938654114 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269400cd30 con 0x7f26a006ef70 2024-03-06T07:35:21.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.368+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a0160ae0 con 0x7f26a006ef70 2024-03-06T07:35:21.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.369+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a0160d20 con 0x7f26a006ef70 2024-03-06T07:35:21.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.370+0000 7f2687fff640 1 -- 172.21.15.27:0/1938654114 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f269401a070 con 0x7f26a006ef70 2024-03-06T07:35:21.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.370+0000 7f2687fff640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f268c06b4d0 0x7f268c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.370+0000 7f2687fff640 1 -- 172.21.15.27:0/1938654114 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f269408a670 con 0x7f26a006ef70 2024-03-06T07:35:21.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.370+0000 7f269effd640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f268c06b4d0 0x7f268c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.371+0000 7f269effd640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f268c06b4d0 0x7f268c06d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2690036000 tx=0x7f269003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.379+0000 7f2687fff640 1 -- 172.21.15.27:0/1938654114 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26a0160d20 con 0x7f26a006ef70 2024-03-06T07:35:21.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.597+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26a0067ff0 con 0x7f26a006ef70 2024-03-06T07:35:21.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.598+0000 7f2687fff640 1 -- 172.21.15.27:0/1938654114 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f2694058ff0 con 0x7f26a006ef70 2024-03-06T07:35:21.599 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:21.600 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:21.600 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:21.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f268c06b4d0 msgr2=0x7f268c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f268c06b4d0 0x7f268c06d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2690036000 tx=0x7f269003c040 comp rx=0 tx=0).stop 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 msgr2=0x7f26a0146830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 0x7f26a0146830 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f269400e8a0 tx=0x7f269400ed60 comp rx=0 tx=0).stop 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 shutdown_connections 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f268c06b4d0 0x7f268c06d990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f26a015e030 0x7f26a015f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a00b2290 0x7f26a015b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 --2- 172.21.15.27:0/1938654114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26a006ef70 0x7f26a0146830 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 >> 172.21.15.27:0/1938654114 conn(0x7f26a00b55e0 msgr2=0x7f26a00ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 shutdown_connections 2024-03-06T07:35:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:21.600+0000 7f26a60fb640 1 -- 172.21.15.27:0/1938654114 wait complete. 2024-03-06T07:35:21.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:21.814 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:22.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.941+0000 7f094b846640 1 Processor -- start 2024-03-06T07:35:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.941+0000 7f094b846640 1 -- start start 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f094b846640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094415df00 0x7f09440d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f094b846640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09440b95a0 0x7f09440b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f094b846640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09440cea30 con 0x7f09440b9a30 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f094b846640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09440d2af0 con 0x7f09440b95a0 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f094b846640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09440cf3c0 con 0x7f094415df00 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f0949de3640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f09440b9a30 0x7f09440d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39954/0 (socket says 172.21.15.27:39954) 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f0949de3640 1 -- 172.21.15.27:0/3260125703 learned_addr learned my addr 172.21.15.27:0/3260125703 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f0948de1640 1 --2- 172.21.15.27:0/3260125703 >> v2:172.21.15.69:3300/0 conn(0x7f09440b95a0 0x7f09440b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f09495e2640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094415df00 0x7f09440d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262929285 0 0) 0x7f09440cea30 con 0x7f09440b9a30 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0930003520 con 0x7f09440b9a30 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3539597744 0 0) 0x7f0930003520 con 0x7f09440b9a30 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094415df00 msgr2=0x7f09440d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094415df00 0x7f09440d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 >> v2:172.21.15.69:3300/0 conn(0x7f09440b95a0 msgr2=0x7f09440b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 --2- 172.21.15.27:0/3260125703 >> v2:172.21.15.69:3300/0 conn(0x7f09440b95a0 0x7f09440b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09440cf0e0 con 0x7f09440b9a30 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f093c0019f0 con 0x7f09440b9a30 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 >> v1:172.21.15.27:6789/0 conn(0x7f09440b9a30 legacy=0x7f09440d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09300039b0 0x7f0930003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.942+0000 7f093bfff640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0930004800 0x7f0930004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f093bfff640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094415df00 0x7f09300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09440d2af0 con 0x7f09300039b0 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09440cf3c0 con 0x7f0930004800 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093000b9c0 con 0x7f094415df00 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f09495e2640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09300039b0 0x7f0930003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f0949de3640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094415df00 0x7f09300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f0948de1640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0930004800 0x7f0930004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f09495e2640 1 -- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094415df00 msgr2=0x7f09300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f09495e2640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094415df00 0x7f09300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f09495e2640 1 -- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0930004800 msgr2=0x7f0930004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f09495e2640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0930004800 0x7f0930004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f09495e2640 1 -- 172.21.15.27:0/3260125703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0930003520 con 0x7f09300039b0 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f0949de3640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f094415df00 0x7f09300051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:22.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.943+0000 7f0948de1640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0930004800 0x7f0930004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:22.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.944+0000 7f09495e2640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09300039b0 0x7f0930003dd0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f0940009ac0 tx=0x7f0930005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581232a845303f25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.944+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940039070 con 0x7f09300039b0 2024-03-06T07:35:22.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.944+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094002c6b0 con 0x7f09300039b0 2024-03-06T07:35:22.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.944+0000 7f093bfff640 1 -- 172.21.15.27:0/3260125703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940009690 con 0x7f09300039b0 2024-03-06T07:35:22.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.945+0000 7f094b846640 1 -- 172.21.15.27:0/3260125703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09300039b0 msgr2=0x7f0930003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.945+0000 7f094b846640 1 --2- 172.21.15.27:0/3260125703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09300039b0 0x7f0930003dd0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f0940009ac0 tx=0x7f0930005830 comp rx=0 tx=0).stop 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.945+0000 7f0949de3640 1 -- 172.21.15.27:0/3260125703 reap_dead start 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.945+0000 7f094b846640 1 -- 172.21.15.27:0/3260125703 shutdown_connections 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.945+0000 7f094b846640 1 -- 172.21.15.27:0/3260125703 >> 172.21.15.27:0/3260125703 conn(0x7f0944071960 msgr2=0x7f0944071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.945+0000 7f094b846640 1 -- 172.21.15.27:0/3260125703 shutdown_connections 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.945+0000 7f094b846640 1 -- 172.21.15.27:0/3260125703 wait complete. 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.946+0000 7f094b846640 1 Processor -- start 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.946+0000 7f094b846640 1 -- start start 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f094b846640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09300039b0 0x7f0944160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f094b846640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09440ccc10 0x7f09440b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f094b846640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09440cd050 0x7f09440b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f094b846640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09440cec70 con 0x7f09440cd050 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f094b846640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09440ce750 con 0x7f09300039b0 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f094b846640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09440cedf0 con 0x7f09440ccc10 2024-03-06T07:35:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f0949de3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09440cd050 0x7f09440b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f0949de3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09440cd050 0x7f09440b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42810/0 (socket says 172.21.15.27:42810) 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f0949de3640 1 -- 172.21.15.27:0/2028960240 learned_addr learned my addr 172.21.15.27:0/2028960240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f0948de1640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09440ccc10 0x7f09440b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f09495e2640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09300039b0 0x7f0944160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f0949de3640 1 -- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09440ccc10 msgr2=0x7f09440b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f0949de3640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09440ccc10 0x7f09440b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f0949de3640 1 -- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09300039b0 msgr2=0x7f0944160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f0949de3640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09300039b0 0x7f0944160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.947+0000 7f0949de3640 1 -- 172.21.15.27:0/2028960240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0944161220 con 0x7f09440cd050 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.948+0000 7f0949de3640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09440cd050 0x7f09440b3140 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f09440d1780 tx=0x7f093c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.948+0000 7f0939ffb640 1 -- 172.21.15.27:0/2028960240 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c007a80 con 0x7f09440cd050 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.948+0000 7f0939ffb640 1 -- 172.21.15.27:0/2028960240 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c00edd0 con 0x7f09440cd050 2024-03-06T07:35:22.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.948+0000 7f0939ffb640 1 -- 172.21.15.27:0/2028960240 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c014670 con 0x7f09440cd050 2024-03-06T07:35:22.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.948+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09440b37c0 con 0x7f09440cd050 2024-03-06T07:35:22.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.948+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0944160200 con 0x7f09440cd050 2024-03-06T07:35:22.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.949+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09440b2810 con 0x7f09440cd050 2024-03-06T07:35:22.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.950+0000 7f0939ffb640 1 -- 172.21.15.27:0/2028960240 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f093c014810 con 0x7f09440cd050 2024-03-06T07:35:22.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.950+0000 7f0939ffb640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f092806b4d0 0x7f092806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.951+0000 7f0939ffb640 1 -- 172.21.15.27:0/2028960240 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f093c08a1b0 con 0x7f09440cd050 2024-03-06T07:35:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.951+0000 7f09495e2640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f092806b4d0 0x7f092806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.951+0000 7f09495e2640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f092806b4d0 0x7f092806d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0940036000 tx=0x7f0940030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:22.955+0000 7f0939ffb640 1 -- 172.21.15.27:0/2028960240 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09440b2810 con 0x7f09440cd050 2024-03-06T07:35:23.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.176+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0944067ff0 con 0x7f09440cd050 2024-03-06T07:35:23.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.176+0000 7f0939ffb640 1 -- 172.21.15.27:0/2028960240 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f093c058f80 con 0x7f09440cd050 2024-03-06T07:35:23.178 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:23.178 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:23.178 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:23.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f092806b4d0 msgr2=0x7f092806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f092806b4d0 0x7f092806d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0940036000 tx=0x7f0940030000 comp rx=0 tx=0).stop 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09440cd050 msgr2=0x7f09440b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09440cd050 0x7f09440b3140 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f09440d1780 tx=0x7f093c00e530 comp rx=0 tx=0).stop 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 shutdown_connections 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f092806b4d0 0x7f092806d990 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09440cd050 0x7f09440b3140 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09440ccc10 0x7f09440b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 --2- 172.21.15.27:0/2028960240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09300039b0 0x7f0944160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 >> 172.21.15.27:0/2028960240 conn(0x7f0944071960 msgr2=0x7f09441618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 shutdown_connections 2024-03-06T07:35:23.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:23.178+0000 7f094b846640 1 -- 172.21.15.27:0/2028960240 wait complete. 2024-03-06T07:35:23.391 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:23.391 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:24.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:24.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.498+0000 7f18f1355640 1 Processor -- start 2024-03-06T07:35:24.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.498+0000 7f18f1355640 1 -- start start 2024-03-06T07:35:24.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.499+0000 7f18f1355640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18ec071540 0x7f18ec0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.499+0000 7f18f1355640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18ec0d3400 0x7f18ec071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.499+0000 7f18f1355640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ec071e60 con 0x7f18ec0b2b50 2024-03-06T07:35:24.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.499+0000 7f18f1355640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18ec0d2f70 con 0x7f18ec0d3400 2024-03-06T07:35:24.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.499+0000 7f18f1355640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18ec0d4be0 con 0x7f18ec071540 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.499+0000 7f18ea7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18ec0d3400 0x7f18ec071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.499+0000 7f18ea7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18ec0d3400 0x7f18ec071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54982/0 (socket says 172.21.15.27:54982) 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.499+0000 7f18eb7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f18ec0b2b50 0x7f18ec0d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37124/0 (socket says 172.21.15.27:37124) 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.499+0000 7f18ea7fc640 1 -- 172.21.15.27:0/2914529686 learned_addr learned my addr 172.21.15.27:0/2914529686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.500+0000 7f18eaffd640 1 --2- 172.21.15.27:0/2914529686 >> v2:172.21.15.69:3300/0 conn(0x7f18ec071540 0x7f18ec0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.500+0000 7f18ea7fc640 1 -- 172.21.15.27:0/2914529686 >> v2:172.21.15.69:3300/0 conn(0x7f18ec071540 msgr2=0x7f18ec0b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.500+0000 7f18ea7fc640 1 --2- 172.21.15.27:0/2914529686 >> v2:172.21.15.69:3300/0 conn(0x7f18ec071540 0x7f18ec0b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.500+0000 7f18ea7fc640 1 -- 172.21.15.27:0/2914529686 >> v1:172.21.15.27:6789/0 conn(0x7f18ec0b2b50 legacy=0x7f18ec0d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.500+0000 7f18e9ffb640 1 -- 172.21.15.27:0/2914529686 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020144233 0 0) 0x7f18ec071e60 con 0x7f18ec0b2b50 2024-03-06T07:35:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.500+0000 7f18ea7fc640 1 -- 172.21.15.27:0/2914529686 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec0ce610 con 0x7f18ec0d3400 2024-03-06T07:35:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.501+0000 7f18ea7fc640 1 --2- 172.21.15.27:0/2914529686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 0x7f18ec071920 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0d4da0 tx=0x7f18d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7197c1a17e28a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.501+0000 7f18e9ffb640 1 -- 172.21.15.27:0/2914529686 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4034070 con 0x7f18ec0d3400 2024-03-06T07:35:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.501+0000 7f18e9ffb640 1 -- 172.21.15.27:0/2914529686 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d4030860 con 0x7f18ec0d3400 2024-03-06T07:35:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.501+0000 7f18e9ffb640 1 -- 172.21.15.27:0/2914529686 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4030c30 con 0x7f18ec0d3400 2024-03-06T07:35:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.502+0000 7f18f1355640 1 -- 172.21.15.27:0/2914529686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 msgr2=0x7f18ec071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.502+0000 7f18f1355640 1 --2- 172.21.15.27:0/2914529686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 0x7f18ec071920 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0d4da0 tx=0x7f18d402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.502+0000 7f18f1355640 1 -- 172.21.15.27:0/2914529686 shutdown_connections 2024-03-06T07:35:24.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.502+0000 7f18f1355640 1 --2- 172.21.15.27:0/2914529686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 0x7f18ec071920 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.502+0000 7f18f1355640 1 --2- 172.21.15.27:0/2914529686 >> v2:172.21.15.69:3300/0 conn(0x7f18ec071540 0x7f18ec0b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.502+0000 7f18f1355640 1 -- 172.21.15.27:0/2914529686 >> 172.21.15.27:0/2914529686 conn(0x7f18ec0d3ca0 msgr2=0x7f18ec0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.502+0000 7f18f1355640 1 -- 172.21.15.27:0/2914529686 shutdown_connections 2024-03-06T07:35:24.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.502+0000 7f18f1355640 1 -- 172.21.15.27:0/2914529686 wait complete. 2024-03-06T07:35:24.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.503+0000 7f18f1355640 1 Processor -- start 2024-03-06T07:35:24.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.503+0000 7f18f1355640 1 -- start start 2024-03-06T07:35:24.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18f1355640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18ec071540 0x7f18ec0b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18f1355640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18ec0b2b50 0x7f18ec0c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18f1355640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 0x7f18ec0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18f1355640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec0ce8f0 con 0x7f18ec0d3400 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18f1355640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec0d2c00 con 0x7f18ec0b2b50 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18f1355640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec15e370 con 0x7f18ec071540 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 0x7f18ec0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 0x7f18ec0c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54988/0 (socket says 172.21.15.27:54988) 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18eb7fe640 1 -- 172.21.15.27:0/2512991937 learned_addr learned my addr 172.21.15.27:0/2512991937 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18eaffd640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18ec071540 0x7f18ec0b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.504+0000 7f18ea7fc640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18ec0b2b50 0x7f18ec0c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18eb7fe640 1 -- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18ec071540 msgr2=0x7f18ec0b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18eb7fe640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18ec071540 0x7f18ec0b22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18eb7fe640 1 -- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18ec0b2b50 msgr2=0x7f18ec0c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18eb7fe640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18ec0b2b50 0x7f18ec0c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18eb7fe640 1 -- 172.21.15.27:0/2512991937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec0c7fb0 con 0x7f18ec0d3400 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18eb7fe640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 0x7f18ec0c7980 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f18e400c7d0 tx=0x7f18e400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18e37fe640 1 -- 172.21.15.27:0/2512991937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4018070 con 0x7f18ec0d3400 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18e37fe640 1 -- 172.21.15.27:0/2512991937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e400a640 con 0x7f18ec0d3400 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ec0c8290 con 0x7f18ec0d3400 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18e37fe640 1 -- 172.21.15.27:0/2512991937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e400aa70 con 0x7f18ec0d3400 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.505+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ec0c8850 con 0x7f18ec0d3400 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.506+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ec068700 con 0x7f18ec0d3400 2024-03-06T07:35:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.507+0000 7f18e37fe640 1 -- 172.21.15.27:0/2512991937 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18e401b070 con 0x7f18ec0d3400 2024-03-06T07:35:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.507+0000 7f18e37fe640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18cc06b4d0 0x7f18cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.507+0000 7f18e37fe640 1 -- 172.21.15.27:0/2512991937 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f18e4089790 con 0x7f18ec0d3400 2024-03-06T07:35:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.507+0000 7f18eaffd640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18cc06b4d0 0x7f18cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.508+0000 7f18eaffd640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18cc06b4d0 0x7f18cc06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f18dc006fd0 tx=0x7f18dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.512+0000 7f18e37fe640 1 -- 172.21.15.27:0/2512991937 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18e4057fa0 con 0x7f18ec0d3400 2024-03-06T07:35:24.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.733+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18ec06b8e0 con 0x7f18ec0d3400 2024-03-06T07:35:24.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.733+0000 7f18e37fe640 1 -- 172.21.15.27:0/2512991937 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f18e405c040 con 0x7f18ec0d3400 2024-03-06T07:35:24.734 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:24.734 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:24.735 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18cc06b4d0 msgr2=0x7f18cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18cc06b4d0 0x7f18cc06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f18dc006fd0 tx=0x7f18dc008040 comp rx=0 tx=0).stop 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 msgr2=0x7f18ec0c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 0x7f18ec0c7980 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f18e400c7d0 tx=0x7f18e400cc90 comp rx=0 tx=0).stop 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 shutdown_connections 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f18cc06b4d0 0x7f18cc06d990 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ec0d3400 0x7f18ec0c7980 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18ec0b2b50 0x7f18ec0c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 --2- 172.21.15.27:0/2512991937 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f18ec071540 0x7f18ec0b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 >> 172.21.15.27:0/2512991937 conn(0x7f18ec0d3ca0 msgr2=0x7f18ec160b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 shutdown_connections 2024-03-06T07:35:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:24.735+0000 7f18f1355640 1 -- 172.21.15.27:0/2512991937 wait complete. 2024-03-06T07:35:24.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:24.954 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:25.954 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:26.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.065+0000 7ff9489a0640 1 Processor -- start 2024-03-06T07:35:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.065+0000 7ff9489a0640 1 -- start start 2024-03-06T07:35:26.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.066+0000 7ff9489a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9440ccf30 0x7ff944072010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.066+0000 7ff9489a0640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9440cc540 con 0x7ff94406db00 2024-03-06T07:35:26.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.066+0000 7ff9489a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94406d020 con 0x7ff94415dc10 2024-03-06T07:35:26.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.066+0000 7ff9489a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94406d300 con 0x7ff9440ccf30 2024-03-06T07:35:26.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.066+0000 7ff942ffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff94406db00 0x7ff9440cc820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37130/0 (socket says 172.21.15.27:37130) 2024-03-06T07:35:26.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.066+0000 7ff942ffd640 1 -- 172.21.15.27:0/1374476982 learned_addr learned my addr 172.21.15.27:0/1374476982 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:26.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.066+0000 7ff943fff640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9440ccf30 0x7ff944072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.066+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438934579 0 0) 0x7ff9440cc540 con 0x7ff94406db00 2024-03-06T07:35:26.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.066+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff930003520 con 0x7ff94406db00 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2830433320 0 0) 0x7ff930003520 con 0x7ff94406db00 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9440ccf30 msgr2=0x7ff944072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9440ccf30 0x7ff944072010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 >> v1:172.21.15.69:6789/0 conn(0x7ff94415dc10 legacy=0x7ff9440cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94406d5e0 con 0x7ff94406db00 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7ff934001670 con 0x7ff94406db00 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 >> v1:172.21.15.27:6789/0 conn(0x7ff94406db00 legacy=0x7ff9440cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9300039e0 0x7ff930003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9440ccf30 0x7ff930004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff930004d40 0x7ff9300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94406d300 con 0x7ff9300039e0 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93000ba70 con 0x7ff930004d40 2024-03-06T07:35:26.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93000bc70 con 0x7ff9440ccf30 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9437fe640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9300039e0 0x7ff930003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff942ffd640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9440ccf30 0x7ff930004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff943fff640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff930004d40 0x7ff9300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9437fe640 1 -- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9440ccf30 msgr2=0x7ff930004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9437fe640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9440ccf30 0x7ff930004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9437fe640 1 -- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff930004d40 msgr2=0x7ff9300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9437fe640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff930004d40 0x7ff9300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff9437fe640 1 -- 172.21.15.27:0/1374476982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff930003520 con 0x7ff9300039e0 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff943fff640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff930004d40 0x7ff9300051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.067+0000 7ff942ffd640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9440ccf30 0x7ff930004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.068+0000 7ff9437fe640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9300039e0 0x7ff930003dc0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7ff92c001860 tx=0x7ff930005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fa39d231ee8637 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.068+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c02f050 con 0x7ff9300039e0 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.068+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c033650 con 0x7ff9300039e0 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.068+0000 7ff9427fc640 1 -- 172.21.15.27:0/1374476982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c001430 con 0x7ff9300039e0 2024-03-06T07:35:26.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.069+0000 7ff9489a0640 1 -- 172.21.15.27:0/1374476982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9300039e0 msgr2=0x7ff930003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.069+0000 7ff9489a0640 1 --2- 172.21.15.27:0/1374476982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9300039e0 0x7ff930003dc0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7ff92c001860 tx=0x7ff930005830 comp rx=0 tx=0).stop 2024-03-06T07:35:26.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.069+0000 7ff943fff640 1 -- 172.21.15.27:0/1374476982 reap_dead start 2024-03-06T07:35:26.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.069+0000 7ff9489a0640 1 -- 172.21.15.27:0/1374476982 shutdown_connections 2024-03-06T07:35:26.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.069+0000 7ff9489a0640 1 -- 172.21.15.27:0/1374476982 >> 172.21.15.27:0/1374476982 conn(0x7ff9440b1b70 msgr2=0x7ff9440b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.069+0000 7ff9489a0640 1 -- 172.21.15.27:0/1374476982 shutdown_connections 2024-03-06T07:35:26.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.069+0000 7ff9489a0640 1 -- 172.21.15.27:0/1374476982 wait complete. 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.070+0000 7ff9489a0640 1 Processor -- start 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.070+0000 7ff9489a0640 1 -- start start 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff9489a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9300039e0 0x7ff94409cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff9489a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9440ace50 0x7ff94409d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff9489a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff94409aa40 0x7ff94409aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff9489a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440cc2c0 con 0x7ff94409aa40 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff9489a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440cc140 con 0x7ff9440ace50 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff9489a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94415dff0 con 0x7ff9300039e0 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff943fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff94409aa40 0x7ff94409aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff943fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff94409aa40 0x7ff94409aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55002/0 (socket says 172.21.15.27:55002) 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff943fff640 1 -- 172.21.15.27:0/1121219128 learned_addr learned my addr 172.21.15.27:0/1121219128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff9437fe640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9300039e0 0x7ff94409cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff942ffd640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9440ace50 0x7ff94409d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff943fff640 1 -- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9300039e0 msgr2=0x7ff94409cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff943fff640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9300039e0 0x7ff94409cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff943fff640 1 -- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9440ace50 msgr2=0x7ff94409d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff943fff640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9440ace50 0x7ff94409d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff943fff640 1 -- 172.21.15.27:0/1121219128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440ad680 con 0x7ff94409aa40 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff9437fe640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9300039e0 0x7ff94409cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff942ffd640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9440ace50 0x7ff94409d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.071+0000 7ff943fff640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff94409aa40 0x7ff94409aec0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7ff94406bf50 tx=0x7ff93c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.072+0000 7ff93bfff640 1 -- 172.21.15.27:0/1121219128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c010d80 con 0x7ff94409aa40 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.072+0000 7ff93bfff640 1 -- 172.21.15.27:0/1121219128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93c003990 con 0x7ff94409aa40 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.072+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9440ad960 con 0x7ff94409aa40 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.072+0000 7ff93bfff640 1 -- 172.21.15.27:0/1121219128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c003dc0 con 0x7ff94409aa40 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.072+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9440adf20 con 0x7ff94409aa40 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.073+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94404f330 con 0x7ff94409aa40 2024-03-06T07:35:26.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.073+0000 7ff93bfff640 1 -- 172.21.15.27:0/1121219128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff93c005d10 con 0x7ff94409aa40 2024-03-06T07:35:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.074+0000 7ff93bfff640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff92406b500 0x7ff92406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.074+0000 7ff93bfff640 1 -- 172.21.15.27:0/1121219128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff93c08a820 con 0x7ff94409aa40 2024-03-06T07:35:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.074+0000 7ff9437fe640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff92406b500 0x7ff92406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.074+0000 7ff9437fe640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff92406b500 0x7ff92406d9c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7ff92c036000 tx=0x7ff92c031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.083+0000 7ff93bfff640 1 -- 172.21.15.27:0/1121219128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff93c025020 con 0x7ff94409aa40 2024-03-06T07:35:26.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.302+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff94406dfd0 con 0x7ff94409aa40 2024-03-06T07:35:26.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.303+0000 7ff93bfff640 1 -- 172.21.15.27:0/1121219128 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7ff93c059680 con 0x7ff94409aa40 2024-03-06T07:35:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:26.305 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:26.306 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:26.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff92406b500 msgr2=0x7ff92406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff92406b500 0x7ff92406d9c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7ff92c036000 tx=0x7ff92c031000 comp rx=0 tx=0).stop 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff94409aa40 msgr2=0x7ff94409aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff94409aa40 0x7ff94409aec0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7ff94406bf50 tx=0x7ff93c010590 comp rx=0 tx=0).stop 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 shutdown_connections 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7ff92406b500 0x7ff92406d9c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff94409aa40 0x7ff94409aec0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9440ace50 0x7ff94409d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 --2- 172.21.15.27:0/1121219128 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff9300039e0 0x7ff94409cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 >> 172.21.15.27:0/1121219128 conn(0x7ff9440b1b70 msgr2=0x7ff9440b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 shutdown_connections 2024-03-06T07:35:26.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:26.306+0000 7ff9489a0640 1 -- 172.21.15.27:0/1121219128 wait complete. 2024-03-06T07:35:26.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:26.520 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:27.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:27.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.626+0000 7fd315b02640 1 Processor -- start 2024-03-06T07:35:27.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.627+0000 7fd315b02640 1 -- start start 2024-03-06T07:35:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.627+0000 7fd315b02640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd31015df00 0x7fd3100d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.627+0000 7fd315b02640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd3100b9a30 0x7fd3100d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.627+0000 7fd315b02640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3100cea30 con 0x7fd3100b95a0 2024-03-06T07:35:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.627+0000 7fd315b02640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3100d2af0 con 0x7fd3100b9a30 2024-03-06T07:35:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.627+0000 7fd315b02640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100cf3c0 con 0x7fd31015df00 2024-03-06T07:35:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd31015df00 0x7fd3100d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd3100b9a30 0x7fd3100d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30effd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd3100b95a0 0x7fd3100b5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:48696/0 (socket says 172.21.15.27:48696) 2024-03-06T07:35:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30effd640 1 -- 172.21.15.27:0/672134310 learned_addr learned my addr 172.21.15.27:0/672134310 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30e7fc640 1 -- 172.21.15.27:0/672134310 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879270348 0 0) 0x7fd3100cea30 con 0x7fd3100b95a0 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30e7fc640 1 -- 172.21.15.27:0/672134310 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f8003520 con 0x7fd3100b95a0 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30ffff640 1 -- 172.21.15.27:0/672134310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd31015df00 msgr2=0x7fd3100d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30ffff640 1 --2- 172.21.15.27:0/672134310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd31015df00 0x7fd3100d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30ffff640 1 -- 172.21.15.27:0/672134310 >> v1:172.21.15.69:6789/0 conn(0x7fd3100b95a0 legacy=0x7fd3100b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30ffff640 1 -- 172.21.15.27:0/672134310 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3100cf0e0 con 0x7fd3100b9a30 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.628+0000 7fd30f7fe640 1 --2- 172.21.15.27:0/672134310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd31015df00 0x7fd3100d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.629+0000 7fd30ffff640 1 --2- 172.21.15.27:0/672134310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3100b9a30 0x7fd3100d3220 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd308009d90 tx=0x7fd30802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa20a4d601ccfa90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.629+0000 7fd30e7fc640 1 -- 172.21.15.27:0/672134310 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308039070 con 0x7fd3100b9a30 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.629+0000 7fd30e7fc640 1 -- 172.21.15.27:0/672134310 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30802cc50 con 0x7fd3100b9a30 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.629+0000 7fd30e7fc640 1 -- 172.21.15.27:0/672134310 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308034440 con 0x7fd3100b9a30 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.630+0000 7fd315b02640 1 -- 172.21.15.27:0/672134310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3100b9a30 msgr2=0x7fd3100d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.630+0000 7fd315b02640 1 --2- 172.21.15.27:0/672134310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3100b9a30 0x7fd3100d3220 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd308009d90 tx=0x7fd30802bea0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.630+0000 7fd315b02640 1 -- 172.21.15.27:0/672134310 shutdown_connections 2024-03-06T07:35:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.630+0000 7fd315b02640 1 --2- 172.21.15.27:0/672134310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3100b9a30 0x7fd3100d3220 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.630+0000 7fd315b02640 1 --2- 172.21.15.27:0/672134310 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd31015df00 0x7fd3100d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.630+0000 7fd315b02640 1 -- 172.21.15.27:0/672134310 >> 172.21.15.27:0/672134310 conn(0x7fd310071960 msgr2=0x7fd310071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.630+0000 7fd315b02640 1 -- 172.21.15.27:0/672134310 shutdown_connections 2024-03-06T07:35:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.630+0000 7fd315b02640 1 -- 172.21.15.27:0/672134310 wait complete. 2024-03-06T07:35:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.631+0000 7fd315b02640 1 Processor -- start 2024-03-06T07:35:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.631+0000 7fd315b02640 1 -- start start 2024-03-06T07:35:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd315b02640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3100b95a0 0x7fd3101f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd315b02640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31015df00 0x7fd3100b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd315b02640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3101604b0 0x7fd310160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd315b02640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100cec70 con 0x7fd31015df00 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd315b02640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100ce750 con 0x7fd3100b95a0 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd315b02640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100cedf0 con 0x7fd3101604b0 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31015df00 0x7fd3100b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31015df00 0x7fd3100b1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55016/0 (socket says 172.21.15.27:55016) 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30effd640 1 -- 172.21.15.27:0/1142045795 learned_addr learned my addr 172.21.15.27:0/1142045795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30ffff640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3101604b0 0x7fd310160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30f7fe640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3100b95a0 0x7fd3101f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30effd640 1 -- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3101604b0 msgr2=0x7fd310160960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30effd640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3101604b0 0x7fd310160960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30effd640 1 -- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3100b95a0 msgr2=0x7fd3101f3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30effd640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3100b95a0 0x7fd3101f3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.632+0000 7fd30effd640 1 -- 172.21.15.27:0/1142045795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3101f7be0 con 0x7fd31015df00 2024-03-06T07:35:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.633+0000 7fd30effd640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31015df00 0x7fd3100b1d40 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fd30400c430 tx=0x7fd30400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.633+0000 7fd2f7fff640 1 -- 172.21.15.27:0/1142045795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd304017070 con 0x7fd31015df00 2024-03-06T07:35:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.633+0000 7fd2f7fff640 1 -- 172.21.15.27:0/1142045795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd304005ce0 con 0x7fd31015df00 2024-03-06T07:35:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.633+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3100c7f30 con 0x7fd31015df00 2024-03-06T07:35:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.633+0000 7fd2f7fff640 1 -- 172.21.15.27:0/1142045795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3040072f0 con 0x7fd31015df00 2024-03-06T07:35:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.633+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3100c84f0 con 0x7fd31015df00 2024-03-06T07:35:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.634+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd310067ff0 con 0x7fd31015df00 2024-03-06T07:35:27.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.635+0000 7fd2f7fff640 1 -- 172.21.15.27:0/1142045795 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd30401a070 con 0x7fd31015df00 2024-03-06T07:35:27.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.635+0000 7fd2f7fff640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2fc06b500 0x7fd2fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.635+0000 7fd2f7fff640 1 -- 172.21.15.27:0/1142045795 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd304089590 con 0x7fd31015df00 2024-03-06T07:35:27.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.636+0000 7fd30f7fe640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2fc06b500 0x7fd2fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.636+0000 7fd30f7fe640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2fc06b500 0x7fd2fc06d9c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd300009560 tx=0x7fd300009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.640+0000 7fd2f7fff640 1 -- 172.21.15.27:0/1142045795 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd304057da0 con 0x7fd31015df00 2024-03-06T07:35:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.861+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3100cdb40 con 0x7fd31015df00 2024-03-06T07:35:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.861+0000 7fd2f7fff640 1 -- 172.21.15.27:0/1142045795 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fd30405be40 con 0x7fd31015df00 2024-03-06T07:35:27.864 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:27.864 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2fc06b500 msgr2=0x7fd2fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2fc06b500 0x7fd2fc06d9c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd300009560 tx=0x7fd300009190 comp rx=0 tx=0).stop 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31015df00 msgr2=0x7fd3100b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31015df00 0x7fd3100b1d40 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fd30400c430 tx=0x7fd30400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 shutdown_connections 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2fc06b500 0x7fd2fc06d9c0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd3101604b0 0x7fd310160960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31015df00 0x7fd3100b1d40 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 --2- 172.21.15.27:0/1142045795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3100b95a0 0x7fd3101f3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 >> 172.21.15.27:0/1142045795 conn(0x7fd310071960 msgr2=0x7fd3100d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 shutdown_connections 2024-03-06T07:35:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:27.863+0000 7fd315b02640 1 -- 172.21.15.27:0/1142045795 wait complete. 2024-03-06T07:35:28.082 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:28.082 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:29.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:29.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.192+0000 7f35994df640 1 Processor -- start 2024-03-06T07:35:29.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.192+0000 7f35994df640 1 -- start start 2024-03-06T07:35:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35994df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3594071540 0x7f35940b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35994df640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35940b2b50 0x7f35940b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35994df640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940ce610 con 0x7f35940d3400 2024-03-06T07:35:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35994df640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35940d2f70 con 0x7f35940b2b50 2024-03-06T07:35:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35994df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35940d4be0 con 0x7f3594071540 2024-03-06T07:35:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35937fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35940b2b50 0x7f35940b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35937fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35940b2b50 0x7f35940b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55032/0 (socket says 172.21.15.27:55032) 2024-03-06T07:35:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35937fe640 1 -- 172.21.15.27:0/2146468212 learned_addr learned my addr 172.21.15.27:0/2146468212 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f3592ffd640 1 --2- 172.21.15.27:0/2146468212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3594071540 0x7f35940b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35937fe640 1 -- 172.21.15.27:0/2146468212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3594071540 msgr2=0x7f35940b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35937fe640 1 --2- 172.21.15.27:0/2146468212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3594071540 0x7f35940b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35937fe640 1 -- 172.21.15.27:0/2146468212 >> v1:172.21.15.69:6789/0 conn(0x7f35940d3400 legacy=0x7f3594071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.193+0000 7f35937fe640 1 -- 172.21.15.27:0/2146468212 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940d2920 con 0x7f35940b2b50 2024-03-06T07:35:29.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.194+0000 7f35937fe640 1 --2- 172.21.15.27:0/2146468212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 0x7f35940b6e20 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f358c009e50 tx=0x7f358c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e003820a8ca3180 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.194+0000 7f3591ffb640 1 -- 172.21.15.27:0/2146468212 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c02cb00 con 0x7f35940b2b50 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.194+0000 7f3591ffb640 1 -- 172.21.15.27:0/2146468212 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c02cca0 con 0x7f35940b2b50 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.194+0000 7f3591ffb640 1 -- 172.21.15.27:0/2146468212 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c034610 con 0x7f35940b2b50 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.195+0000 7f35994df640 1 -- 172.21.15.27:0/2146468212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 msgr2=0x7f35940b6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.195+0000 7f35994df640 1 --2- 172.21.15.27:0/2146468212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 0x7f35940b6e20 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f358c009e50 tx=0x7f358c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.195+0000 7f35994df640 1 -- 172.21.15.27:0/2146468212 shutdown_connections 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.195+0000 7f35994df640 1 --2- 172.21.15.27:0/2146468212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 0x7f35940b6e20 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.195+0000 7f35994df640 1 --2- 172.21.15.27:0/2146468212 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3594071540 0x7f35940b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.195+0000 7f35994df640 1 -- 172.21.15.27:0/2146468212 >> 172.21.15.27:0/2146468212 conn(0x7f35940d3ca0 msgr2=0x7f35940d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.195+0000 7f35994df640 1 -- 172.21.15.27:0/2146468212 shutdown_connections 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.195+0000 7f35994df640 1 -- 172.21.15.27:0/2146468212 wait complete. 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.196+0000 7f35994df640 1 Processor -- start 2024-03-06T07:35:29.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.196+0000 7f35994df640 1 -- start start 2024-03-06T07:35:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35994df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3594071540 0x7f35940c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35994df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 0x7f35940cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35994df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35940d3400 0x7f35940cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35994df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359415e370 con 0x7f35940b2b50 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35994df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35940ce8f0 con 0x7f3594071540 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35994df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35940c7440 con 0x7f35940d3400 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 0x7f35940cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35927fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 0x7f35940cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55042/0 (socket says 172.21.15.27:55042) 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35927fc640 1 -- 172.21.15.27:0/2954991100 learned_addr learned my addr 172.21.15.27:0/2954991100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f3592ffd640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3594071540 0x7f35940c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.197+0000 7f35937fe640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35940d3400 0x7f35940cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f35927fc640 1 -- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35940d3400 msgr2=0x7f35940cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f35927fc640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35940d3400 0x7f35940cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f35927fc640 1 -- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3594071540 msgr2=0x7f35940c8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f35927fc640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3594071540 0x7f35940c8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f35927fc640 1 -- 172.21.15.27:0/2954991100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940c76e0 con 0x7f35940b2b50 2024-03-06T07:35:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f35927fc640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 0x7f35940cd380 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f357c00c440 tx=0x7f357c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f358b7fe640 1 -- 172.21.15.27:0/2954991100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c0052d0 con 0x7f35940b2b50 2024-03-06T07:35:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f358b7fe640 1 -- 172.21.15.27:0/2954991100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c005470 con 0x7f35940b2b50 2024-03-06T07:35:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35940c79c0 con 0x7f35940b2b50 2024-03-06T07:35:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f358b7fe640 1 -- 172.21.15.27:0/2954991100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c007260 con 0x7f35940b2b50 2024-03-06T07:35:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.198+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35940c7f80 con 0x7f35940b2b50 2024-03-06T07:35:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.199+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3594068700 con 0x7f35940b2b50 2024-03-06T07:35:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.200+0000 7f358b7fe640 1 -- 172.21.15.27:0/2954991100 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f357c0074c0 con 0x7f35940b2b50 2024-03-06T07:35:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.200+0000 7f358b7fe640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357406b4d0 0x7f357406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.200+0000 7f358b7fe640 1 -- 172.21.15.27:0/2954991100 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f357c089d10 con 0x7f35940b2b50 2024-03-06T07:35:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.200+0000 7f3592ffd640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357406b4d0 0x7f357406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.201+0000 7f3592ffd640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357406b4d0 0x7f357406d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3584006fd0 tx=0x7f3584008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.204+0000 7f358b7fe640 1 -- 172.21.15.27:0/2954991100 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f357c058690 con 0x7f35940b2b50 2024-03-06T07:35:29.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.425+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35940cf680 con 0x7f35940b2b50 2024-03-06T07:35:29.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.425+0000 7f358b7fe640 1 -- 172.21.15.27:0/2954991100 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f357c05c730 con 0x7f35940b2b50 2024-03-06T07:35:29.428 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:29.428 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:29.428 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357406b4d0 msgr2=0x7f357406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357406b4d0 0x7f357406d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3584006fd0 tx=0x7f3584008040 comp rx=0 tx=0).stop 2024-03-06T07:35:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 msgr2=0x7f35940cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 0x7f35940cd380 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f357c00c440 tx=0x7f357c00c900 comp rx=0 tx=0).stop 2024-03-06T07:35:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 shutdown_connections 2024-03-06T07:35:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f357406b4d0 0x7f357406d990 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f35940d3400 0x7f35940cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35940b2b50 0x7f35940cd380 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 --2- 172.21.15.27:0/2954991100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3594071540 0x7f35940c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.427+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 >> 172.21.15.27:0/2954991100 conn(0x7f35940d3ca0 msgr2=0x7f359415f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.428+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 shutdown_connections 2024-03-06T07:35:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:29.428+0000 7f35994df640 1 -- 172.21.15.27:0/2954991100 wait complete. 2024-03-06T07:35:29.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:29.642 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:30.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.747+0000 7f142011e640 1 Processor -- start 2024-03-06T07:35:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.747+0000 7f142011e640 1 -- start start 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f142011e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f141815df00 0x7f14180d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f142011e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f14180b95a0 0x7f14180b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f142011e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180cea30 con 0x7f14180b9a30 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f142011e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14180d2af0 con 0x7f14180b95a0 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f142011e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cf3c0 con 0x7f141815df00 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f141e6bb640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f14180b9a30 0x7f14180d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37144/0 (socket says 172.21.15.27:37144) 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f141e6bb640 1 -- 172.21.15.27:0/3575040765 learned_addr learned my addr 172.21.15.27:0/3575040765 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f141d6b9640 1 --2- 172.21.15.27:0/3575040765 >> v2:172.21.15.69:3300/0 conn(0x7f14180b95a0 0x7f14180b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f141deba640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f141815df00 0x7f14180d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.748+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941467561 0 0) 0x7f14180cea30 con 0x7f14180b9a30 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1404003520 con 0x7f14180b9a30 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 581700684 0 0) 0x7f1404003520 con 0x7f14180b9a30 2024-03-06T07:35:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f141815df00 msgr2=0x7f14180d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f141815df00 0x7f14180d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 >> v2:172.21.15.69:3300/0 conn(0x7f14180b95a0 msgr2=0x7f14180b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 --2- 172.21.15.27:0/3575040765 >> v2:172.21.15.69:3300/0 conn(0x7f14180b95a0 0x7f14180b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14180cf0e0 con 0x7f14180b9a30 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f14100019f0 con 0x7f14180b9a30 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 >> v1:172.21.15.27:6789/0 conn(0x7f14180b9a30 legacy=0x7f14180d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14040039b0 0x7f1404003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f141815df00 0x7f1404004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1404004d40 0x7f14040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14180d2af0 con 0x7f14040039b0 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cf3c0 con 0x7f1404004d40 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140400b9c0 con 0x7f141815df00 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141deba640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14040039b0 0x7f1404003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141e6bb640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1404004d40 0x7f14040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141d6b9640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f141815df00 0x7f1404004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141deba640 1 -- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f141815df00 msgr2=0x7f1404004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141deba640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f141815df00 0x7f1404004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141deba640 1 -- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1404004d40 msgr2=0x7f14040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141deba640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1404004d40 0x7f14040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141deba640 1 -- 172.21.15.27:0/3575040765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1404003520 con 0x7f14040039b0 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.749+0000 7f141e6bb640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1404004d40 0x7f14040051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.750+0000 7f141deba640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14040039b0 0x7f1404003dd0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f140c009ac0 tx=0x7f1404005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ab0b30be2d6439 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.750+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c039070 con 0x7f14040039b0 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.750+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140c02c6b0 con 0x7f14040039b0 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.750+0000 7f141ceb8640 1 -- 172.21.15.27:0/3575040765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c009690 con 0x7f14040039b0 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.751+0000 7f142011e640 1 -- 172.21.15.27:0/3575040765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14040039b0 msgr2=0x7f1404003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.751+0000 7f142011e640 1 --2- 172.21.15.27:0/3575040765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14040039b0 0x7f1404003dd0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f140c009ac0 tx=0x7f1404005830 comp rx=0 tx=0).stop 2024-03-06T07:35:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.751+0000 7f141e6bb640 1 -- 172.21.15.27:0/3575040765 reap_dead start 2024-03-06T07:35:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.751+0000 7f142011e640 1 -- 172.21.15.27:0/3575040765 shutdown_connections 2024-03-06T07:35:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.751+0000 7f142011e640 1 -- 172.21.15.27:0/3575040765 >> 172.21.15.27:0/3575040765 conn(0x7f1418071960 msgr2=0x7f1418071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.751+0000 7f142011e640 1 -- 172.21.15.27:0/3575040765 shutdown_connections 2024-03-06T07:35:30.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.751+0000 7f142011e640 1 -- 172.21.15.27:0/3575040765 wait complete. 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.752+0000 7f142011e640 1 Processor -- start 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.752+0000 7f142011e640 1 -- start start 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f142011e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14040039b0 0x7f14180c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f142011e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14181551f0 0x7f14180ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f142011e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180cd4e0 0x7f14180c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f142011e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cec70 con 0x7f14180cd4e0 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f142011e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14180ce750 con 0x7f14181551f0 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f142011e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cedf0 con 0x7f14040039b0 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141e6bb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180cd4e0 0x7f14180c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141e6bb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180cd4e0 0x7f14180c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55064/0 (socket says 172.21.15.27:55064) 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141e6bb640 1 -- 172.21.15.27:0/2262074441 learned_addr learned my addr 172.21.15.27:0/2262074441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141d6b9640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14181551f0 0x7f14180ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141deba640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14040039b0 0x7f14180c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141e6bb640 1 -- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14040039b0 msgr2=0x7f14180c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141e6bb640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14040039b0 0x7f14180c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141e6bb640 1 -- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14181551f0 msgr2=0x7f14180ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141e6bb640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14181551f0 0x7f14180ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.753+0000 7f141e6bb640 1 -- 172.21.15.27:0/2262074441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14180c9d30 con 0x7f14180cd4e0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.754+0000 7f141e6bb640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180cd4e0 0x7f14180c79e0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f14100013d0 tx=0x7f141000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.754+0000 7f140a7fc640 1 -- 172.21.15.27:0/2262074441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410007a80 con 0x7f14180cd4e0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.754+0000 7f140a7fc640 1 -- 172.21.15.27:0/2262074441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141000eda0 con 0x7f14180cd4e0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.754+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14180c8750 con 0x7f14180cd4e0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.754+0000 7f140a7fc640 1 -- 172.21.15.27:0/2262074441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410014670 con 0x7f14180cd4e0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.754+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14180c8d10 con 0x7f14180cd4e0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.755+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1418067ff0 con 0x7f14180cd4e0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.755+0000 7f140a7fc640 1 -- 172.21.15.27:0/2262074441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1410014810 con 0x7f14180cd4e0 2024-03-06T07:35:30.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.756+0000 7f140a7fc640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f13fc06b4d0 0x7f13fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.756+0000 7f140a7fc640 1 -- 172.21.15.27:0/2262074441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f141008a180 con 0x7f14180cd4e0 2024-03-06T07:35:30.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.756+0000 7f141deba640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f13fc06b4d0 0x7f13fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.757+0000 7f141deba640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f13fc06b4d0 0x7f13fc06d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f140c036000 tx=0x7f140c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.760+0000 7f140a7fc640 1 -- 172.21.15.27:0/2262074441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14180c9d30 con 0x7f14180cd4e0 2024-03-06T07:35:30.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.981+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14180d3170 con 0x7f14180cd4e0 2024-03-06T07:35:30.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.982+0000 7f140a7fc640 1 -- 172.21.15.27:0/2262074441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f1410058fe0 con 0x7f14180cd4e0 2024-03-06T07:35:30.984 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:30.984 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:30.984 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:30.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f13fc06b4d0 msgr2=0x7f13fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f13fc06b4d0 0x7f13fc06d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f140c036000 tx=0x7f140c030000 comp rx=0 tx=0).stop 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180cd4e0 msgr2=0x7f14180c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180cd4e0 0x7f14180c79e0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f14100013d0 tx=0x7f141000e530 comp rx=0 tx=0).stop 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 shutdown_connections 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f13fc06b4d0 0x7f13fc06d990 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180cd4e0 0x7f14180c79e0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14181551f0 0x7f14180ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 --2- 172.21.15.27:0/2262074441 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f14040039b0 0x7f14180c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 >> 172.21.15.27:0/2262074441 conn(0x7f1418071960 msgr2=0x7f14180b8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 shutdown_connections 2024-03-06T07:35:30.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:30.984+0000 7f142011e640 1 -- 172.21.15.27:0/2262074441 wait complete. 2024-03-06T07:35:31.199 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:31.199 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:32.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:32.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.306+0000 7f554e301640 1 Processor -- start 2024-03-06T07:35:32.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.306+0000 7f554e301640 1 -- start start 2024-03-06T07:35:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.307+0000 7f554e301640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5548071540 0x7f55480b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.307+0000 7f554e301640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f55480b2b50 0x7f55480b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.307+0000 7f554e301640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55480ce610 con 0x7f55480d3400 2024-03-06T07:35:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.307+0000 7f554e301640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55480d2f70 con 0x7f5548071540 2024-03-06T07:35:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.307+0000 7f554e301640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55480d4be0 con 0x7f55480b2b50 2024-03-06T07:35:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.307+0000 7f5547fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5548071540 0x7f55480b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.307+0000 7f5547fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5548071540 0x7f55480b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55068/0 (socket says 172.21.15.27:55068) 2024-03-06T07:35:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.307+0000 7f5547fff640 1 -- 172.21.15.27:0/2365725780 learned_addr learned my addr 172.21.15.27:0/2365725780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.307+0000 7f554c89e640 1 --2- 172.21.15.27:0/2365725780 >> v2:172.21.15.69:3300/0 conn(0x7f55480b2b50 0x7f55480b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.308+0000 7f5547fff640 1 -- 172.21.15.27:0/2365725780 >> v2:172.21.15.69:3300/0 conn(0x7f55480b2b50 msgr2=0x7f55480b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:32.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.308+0000 7f5547fff640 1 --2- 172.21.15.27:0/2365725780 >> v2:172.21.15.69:3300/0 conn(0x7f55480b2b50 0x7f55480b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.308+0000 7f5547fff640 1 -- 172.21.15.27:0/2365725780 >> v1:172.21.15.69:6789/0 conn(0x7f55480d3400 legacy=0x7f5548071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:32.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.308+0000 7f5547fff640 1 -- 172.21.15.27:0/2365725780 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55480d2920 con 0x7f5548071540 2024-03-06T07:35:32.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.308+0000 7f554c89e640 1 --2- 172.21.15.27:0/2365725780 >> v2:172.21.15.69:3300/0 conn(0x7f55480b2b50 0x7f55480b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.309+0000 7f5547fff640 1 --2- 172.21.15.27:0/2365725780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5548071540 0x7f55480b30a0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f55480683b0 tx=0x7f553802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ead3518c0785fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.309+0000 7f5546ffd640 1 -- 172.21.15.27:0/2365725780 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538035020 con 0x7f5548071540 2024-03-06T07:35:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.309+0000 7f5546ffd640 1 -- 172.21.15.27:0/2365725780 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5538031890 con 0x7f5548071540 2024-03-06T07:35:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.309+0000 7f5546ffd640 1 -- 172.21.15.27:0/2365725780 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538031c60 con 0x7f5548071540 2024-03-06T07:35:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.310+0000 7f554e301640 1 -- 172.21.15.27:0/2365725780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5548071540 msgr2=0x7f55480b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.310+0000 7f554e301640 1 --2- 172.21.15.27:0/2365725780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5548071540 0x7f55480b30a0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f55480683b0 tx=0x7f553802b930 comp rx=0 tx=0).stop 2024-03-06T07:35:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.310+0000 7f554e301640 1 -- 172.21.15.27:0/2365725780 shutdown_connections 2024-03-06T07:35:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.310+0000 7f554e301640 1 --2- 172.21.15.27:0/2365725780 >> v2:172.21.15.69:3300/0 conn(0x7f55480b2b50 0x7f55480b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.310+0000 7f554e301640 1 --2- 172.21.15.27:0/2365725780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5548071540 0x7f55480b30a0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.310+0000 7f554e301640 1 -- 172.21.15.27:0/2365725780 >> 172.21.15.27:0/2365725780 conn(0x7f55480d3ca0 msgr2=0x7f55480d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.310+0000 7f554e301640 1 -- 172.21.15.27:0/2365725780 shutdown_connections 2024-03-06T07:35:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.310+0000 7f554e301640 1 -- 172.21.15.27:0/2365725780 wait complete. 2024-03-06T07:35:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.311+0000 7f554e301640 1 Processor -- start 2024-03-06T07:35:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.311+0000 7f554e301640 1 -- start start 2024-03-06T07:35:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f554e301640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5548071540 0x7f55481e9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f554e301640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55480b2b50 0x7f55481e8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f554e301640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55480d3400 0x7f55481f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f554e301640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554815e370 con 0x7f55480b2b50 2024-03-06T07:35:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f554e301640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55480ce8f0 con 0x7f55480d3400 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f554e301640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55481e7c50 con 0x7f5548071540 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f55477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55480b2b50 0x7f55481e8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f55477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55480b2b50 0x7f55481e8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55084/0 (socket says 172.21.15.27:55084) 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f55477fe640 1 -- 172.21.15.27:0/1430101863 learned_addr learned my addr 172.21.15.27:0/1430101863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f554c89e640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55480d3400 0x7f55481f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f5547fff640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5548071540 0x7f55481e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f55477fe640 1 -- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5548071540 msgr2=0x7f55481e9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f55477fe640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5548071540 0x7f55481e9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f55477fe640 1 -- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55480d3400 msgr2=0x7f55481f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f55477fe640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55480d3400 0x7f55481f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f55477fe640 1 -- 172.21.15.27:0/1430101863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55481e7f30 con 0x7f55480b2b50 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f554c89e640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55480d3400 0x7f55481f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f5547fff640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5548071540 0x7f55481e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.312+0000 7f55477fe640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55480b2b50 0x7f55481e8cd0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f553c00b3c0 tx=0x7f553c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.313+0000 7f5544ff9640 1 -- 172.21.15.27:0/1430101863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c011830 con 0x7f55480b2b50 2024-03-06T07:35:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.313+0000 7f5544ff9640 1 -- 172.21.15.27:0/1430101863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c016070 con 0x7f55480b2b50 2024-03-06T07:35:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.313+0000 7f5544ff9640 1 -- 172.21.15.27:0/1430101863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c011c50 con 0x7f55480b2b50 2024-03-06T07:35:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.313+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55481e8210 con 0x7f55480b2b50 2024-03-06T07:35:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.313+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55481e9e50 con 0x7f55480b2b50 2024-03-06T07:35:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.314+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5548068700 con 0x7f55480b2b50 2024-03-06T07:35:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.314+0000 7f5544ff9640 1 -- 172.21.15.27:0/1430101863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f553c01b460 con 0x7f55480b2b50 2024-03-06T07:35:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.315+0000 7f5544ff9640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f553406b4d0 0x7f553406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.315+0000 7f5544ff9640 1 -- 172.21.15.27:0/1430101863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f553c089db0 con 0x7f55480b2b50 2024-03-06T07:35:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.315+0000 7f5547fff640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f553406b4d0 0x7f553406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.316+0000 7f5547fff640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f553406b4d0 0x7f553406d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f5538036000 tx=0x7f5538008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.324+0000 7f5544ff9640 1 -- 172.21.15.27:0/1430101863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f553c020020 con 0x7f55480b2b50 2024-03-06T07:35:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.543+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55481e85a0 con 0x7f55480b2b50 2024-03-06T07:35:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.544+0000 7f5544ff9640 1 -- 172.21.15.27:0/1430101863 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f553c058b80 con 0x7f55480b2b50 2024-03-06T07:35:32.546 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:32.546 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:32.547 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f553406b4d0 msgr2=0x7f553406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f553406b4d0 0x7f553406d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f5538036000 tx=0x7f5538008030 comp rx=0 tx=0).stop 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55480b2b50 msgr2=0x7f55481e8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55480b2b50 0x7f55481e8cd0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f553c00b3c0 tx=0x7f553c00b880 comp rx=0 tx=0).stop 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 shutdown_connections 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f553406b4d0 0x7f553406d990 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55480d3400 0x7f55481f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55480b2b50 0x7f55481e8cd0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 --2- 172.21.15.27:0/1430101863 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5548071540 0x7f55481e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 >> 172.21.15.27:0/1430101863 conn(0x7f55480d3ca0 msgr2=0x7f55480b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 shutdown_connections 2024-03-06T07:35:32.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:32.546+0000 7f554e301640 1 -- 172.21.15.27:0/1430101863 wait complete. 2024-03-06T07:35:32.760 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:32.760 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:33.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.900+0000 7f94e56c9640 1 Processor -- start 2024-03-06T07:35:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.900+0000 7f94e56c9640 1 -- start start 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94e56c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f94e0071540 0x7f94e00b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94e56c9640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94e00d3400 0x7f94e0071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94e56c9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e0071e60 con 0x7f94e00b2b50 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94e56c9640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e00d2f70 con 0x7f94e00d3400 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94e56c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e00d4be0 con 0x7f94e0071540 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94de7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94e00d3400 0x7f94e0071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94df7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f94e00b2b50 0x7f94e00d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51392/0 (socket says 172.21.15.27:51392) 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94de7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94e00d3400 0x7f94e0071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60942/0 (socket says 172.21.15.27:60942) 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94df7fe640 1 -- 172.21.15.27:0/2716826564 learned_addr learned my addr 172.21.15.27:0/2716826564 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94de7fc640 1 -- 172.21.15.27:0/2716826564 >> v2:172.21.15.69:3300/0 conn(0x7f94e0071540 msgr2=0x7f94e00b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94deffd640 1 --2- 172.21.15.27:0/2716826564 >> v2:172.21.15.69:3300/0 conn(0x7f94e0071540 0x7f94e00b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94de7fc640 1 --2- 172.21.15.27:0/2716826564 >> v2:172.21.15.69:3300/0 conn(0x7f94e0071540 0x7f94e00b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94ddffb640 1 -- 172.21.15.27:0/2716826564 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175061104 0 0) 0x7f94e0071e60 con 0x7f94e00b2b50 2024-03-06T07:35:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94de7fc640 1 -- 172.21.15.27:0/2716826564 >> v1:172.21.15.27:6789/0 conn(0x7f94e00b2b50 legacy=0x7f94e00d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.902+0000 7f94de7fc640 1 -- 172.21.15.27:0/2716826564 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e00ce610 con 0x7f94e00d3400 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.903+0000 7f94de7fc640 1 --2- 172.21.15.27:0/2716826564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00d3400 0x7f94e0071920 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f94e00d4da0 tx=0x7f94d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd9e48760f03599 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.903+0000 7f94ddffb640 1 -- 172.21.15.27:0/2716826564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0035020 con 0x7f94e00d3400 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.903+0000 7f94ddffb640 1 -- 172.21.15.27:0/2716826564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d0031830 con 0x7f94e00d3400 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.903+0000 7f94ddffb640 1 -- 172.21.15.27:0/2716826564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0031c00 con 0x7f94e00d3400 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.904+0000 7f94e56c9640 1 -- 172.21.15.27:0/2716826564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00d3400 msgr2=0x7f94e0071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.904+0000 7f94e56c9640 1 --2- 172.21.15.27:0/2716826564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00d3400 0x7f94e0071920 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f94e00d4da0 tx=0x7f94d002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.904+0000 7f94e56c9640 1 -- 172.21.15.27:0/2716826564 shutdown_connections 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.904+0000 7f94e56c9640 1 --2- 172.21.15.27:0/2716826564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00d3400 0x7f94e0071920 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.904+0000 7f94e56c9640 1 --2- 172.21.15.27:0/2716826564 >> v2:172.21.15.69:3300/0 conn(0x7f94e0071540 0x7f94e00b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.904+0000 7f94e56c9640 1 -- 172.21.15.27:0/2716826564 >> 172.21.15.27:0/2716826564 conn(0x7f94e00d3ca0 msgr2=0x7f94e00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.904+0000 7f94e56c9640 1 -- 172.21.15.27:0/2716826564 shutdown_connections 2024-03-06T07:35:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.904+0000 7f94e56c9640 1 -- 172.21.15.27:0/2716826564 wait complete. 2024-03-06T07:35:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.905+0000 7f94e56c9640 1 Processor -- start 2024-03-06T07:35:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.905+0000 7f94e56c9640 1 -- start start 2024-03-06T07:35:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94e56c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e0071540 0x7f94e00b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94e56c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00b2b50 0x7f94e00c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94e56c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e00d3400 0x7f94e00c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94e56c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e00ce8f0 con 0x7f94e00b2b50 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94e56c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e00d2c00 con 0x7f94e0071540 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94e56c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e015e370 con 0x7f94e00d3400 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00b2b50 0x7f94e00c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00b2b50 0x7f94e00c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60950/0 (socket says 172.21.15.27:60950) 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94de7fc640 1 -- 172.21.15.27:0/2088275583 learned_addr learned my addr 172.21.15.27:0/2088275583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94deffd640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e0071540 0x7f94e00b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94df7fe640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e00d3400 0x7f94e00c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94de7fc640 1 -- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e00d3400 msgr2=0x7f94e00c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94de7fc640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e00d3400 0x7f94e00c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94de7fc640 1 -- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e0071540 msgr2=0x7f94e00b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94de7fc640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e0071540 0x7f94e00b22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94de7fc640 1 -- 172.21.15.27:0/2088275583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e00c7fb0 con 0x7f94e00b2b50 2024-03-06T07:35:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.906+0000 7f94de7fc640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00b2b50 0x7f94e00c7440 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f94d0034040 tx=0x7f94d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.907+0000 7f94c77fe640 1 -- 172.21.15.27:0/2088275583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0041070 con 0x7f94e00b2b50 2024-03-06T07:35:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.907+0000 7f94c77fe640 1 -- 172.21.15.27:0/2088275583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d0004230 con 0x7f94e00b2b50 2024-03-06T07:35:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.907+0000 7f94c77fe640 1 -- 172.21.15.27:0/2088275583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0035040 con 0x7f94e00b2b50 2024-03-06T07:35:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.907+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e00c8290 con 0x7f94e00b2b50 2024-03-06T07:35:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.907+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e00c8850 con 0x7f94e00b2b50 2024-03-06T07:35:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.908+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e0068700 con 0x7f94e00b2b50 2024-03-06T07:35:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.908+0000 7f94c77fe640 1 -- 172.21.15.27:0/2088275583 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94d0038020 con 0x7f94e00b2b50 2024-03-06T07:35:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.909+0000 7f94c77fe640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94cc06b4d0 0x7f94cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.909+0000 7f94c77fe640 1 -- 172.21.15.27:0/2088275583 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f94d00abdc0 con 0x7f94e00b2b50 2024-03-06T07:35:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.909+0000 7f94deffd640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94cc06b4d0 0x7f94cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.909+0000 7f94deffd640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94cc06b4d0 0x7f94cc06d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f94d400a970 tx=0x7f94d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:33.913+0000 7f94c77fe640 1 -- 172.21.15.27:0/2088275583 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94d007aba0 con 0x7f94e00b2b50 2024-03-06T07:35:34.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.134+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94e006b8e0 con 0x7f94e00b2b50 2024-03-06T07:35:34.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.134+0000 7f94c77fe640 1 -- 172.21.15.27:0/2088275583 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f94d007a4a0 con 0x7f94e00b2b50 2024-03-06T07:35:34.136 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:34.137 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:34.137 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94cc06b4d0 msgr2=0x7f94cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94cc06b4d0 0x7f94cc06d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f94d400a970 tx=0x7f94d4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00b2b50 msgr2=0x7f94e00c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00b2b50 0x7f94e00c7440 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f94d0034040 tx=0x7f94d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 shutdown_connections 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94cc06b4d0 0x7f94cc06d990 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e00d3400 0x7f94e00c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e00b2b50 0x7f94e00c7440 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 --2- 172.21.15.27:0/2088275583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e0071540 0x7f94e00b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 >> 172.21.15.27:0/2088275583 conn(0x7f94e00d3ca0 msgr2=0x7f94e0160b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 shutdown_connections 2024-03-06T07:35:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:34.136+0000 7f94e56c9640 1 -- 172.21.15.27:0/2088275583 wait complete. 2024-03-06T07:35:34.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:34.351 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:35.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:35.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.456+0000 7f4dd9b2b640 1 Processor -- start 2024-03-06T07:35:35.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.456+0000 7f4dd9b2b640 1 -- start start 2024-03-06T07:35:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.457+0000 7f4dd9b2b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dd40b95a0 0x7f4dd40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.457+0000 7f4dd9b2b640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4dd40b9a10 0x7f4dd40b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.457+0000 7f4dd9b2b640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd40cea30 con 0x7f4dd415df00 2024-03-06T07:35:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.457+0000 7f4dd9b2b640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd40cf3c0 con 0x7f4dd40b95a0 2024-03-06T07:35:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.457+0000 7f4dd9b2b640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd40cf0e0 con 0x7f4dd40b9a10 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.457+0000 7f4dd2ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dd40b95a0 0x7f4dd40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.457+0000 7f4dd37fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4dd415df00 0x7f4dd40d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51402/0 (socket says 172.21.15.27:51402) 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.457+0000 7f4dd2ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dd40b95a0 0x7f4dd40b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60954/0 (socket says 172.21.15.27:60954) 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.457+0000 7f4dd37fe640 1 -- 172.21.15.27:0/4181050820 learned_addr learned my addr 172.21.15.27:0/4181050820 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.458+0000 7f4dd3fff640 1 --2- 172.21.15.27:0/4181050820 >> v2:172.21.15.69:3300/0 conn(0x7f4dd40b9a10 0x7f4dd40b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.458+0000 7f4dd2ffd640 1 -- 172.21.15.27:0/4181050820 >> v2:172.21.15.69:3300/0 conn(0x7f4dd40b9a10 msgr2=0x7f4dd40b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.458+0000 7f4dd2ffd640 1 --2- 172.21.15.27:0/4181050820 >> v2:172.21.15.69:3300/0 conn(0x7f4dd40b9a10 0x7f4dd40b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.458+0000 7f4dd2ffd640 1 -- 172.21.15.27:0/4181050820 >> v1:172.21.15.27:6789/0 conn(0x7f4dd415df00 legacy=0x7f4dd40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.458+0000 7f4dd2ffd640 1 -- 172.21.15.27:0/4181050820 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40ce470 con 0x7f4dd40b95a0 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.458+0000 7f4dd27fc640 1 -- 172.21.15.27:0/4181050820 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982275932 0 0) 0x7f4dd40cea30 con 0x7f4dd415df00 2024-03-06T07:35:35.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.458+0000 7f4dd3fff640 1 --2- 172.21.15.27:0/4181050820 >> v2:172.21.15.69:3300/0 conn(0x7f4dd40b9a10 0x7f4dd40b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd2ffd640 1 --2- 172.21.15.27:0/4181050820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd40b95a0 0x7f4dd40b5700 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40d4d20 tx=0x7f4dc802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e4c5467d17e423 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd27fc640 1 -- 172.21.15.27:0/4181050820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8035020 con 0x7f4dd40b95a0 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd27fc640 1 -- 172.21.15.27:0/4181050820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc8031830 con 0x7f4dd40b95a0 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd27fc640 1 -- 172.21.15.27:0/4181050820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8031c00 con 0x7f4dd40b95a0 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4181050820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd40b95a0 msgr2=0x7f4dd40b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd9b2b640 1 --2- 172.21.15.27:0/4181050820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd40b95a0 0x7f4dd40b5700 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40d4d20 tx=0x7f4dc802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4181050820 shutdown_connections 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd9b2b640 1 --2- 172.21.15.27:0/4181050820 >> v2:172.21.15.69:3300/0 conn(0x7f4dd40b9a10 0x7f4dd40b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd9b2b640 1 --2- 172.21.15.27:0/4181050820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd40b95a0 0x7f4dd40b5700 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4181050820 >> 172.21.15.27:0/4181050820 conn(0x7f4dd4071960 msgr2=0x7f4dd4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4181050820 shutdown_connections 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.459+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4181050820 wait complete. 2024-03-06T07:35:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.460+0000 7f4dd9b2b640 1 Processor -- start 2024-03-06T07:35:35.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.461+0000 7f4dd9b2b640 1 -- start start 2024-03-06T07:35:35.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.461+0000 7f4dd9b2b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40b95a0 0x7f4dd41eb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.461+0000 7f4dd9b2b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dd40b9a10 0x7f4dd41605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.461+0000 7f4dd9b2b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415df00 0x7f4dd40c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.461+0000 7f4dd9b2b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40cedf0 con 0x7f4dd415df00 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.461+0000 7f4dd9b2b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40cec70 con 0x7f4dd40b95a0 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.461+0000 7f4dd9b2b640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40d3220 con 0x7f4dd40b9a10 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.461+0000 7f4dd3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415df00 0x7f4dd40c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415df00 0x7f4dd40c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60968/0 (socket says 172.21.15.27:60968) 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd3fff640 1 -- 172.21.15.27:0/4160720767 learned_addr learned my addr 172.21.15.27:0/4160720767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd2ffd640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dd40b9a10 0x7f4dd41605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd37fe640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40b95a0 0x7f4dd41eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd3fff640 1 -- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dd40b9a10 msgr2=0x7f4dd41605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd3fff640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dd40b9a10 0x7f4dd41605a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd3fff640 1 -- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40b95a0 msgr2=0x7f4dd41eb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd3fff640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40b95a0 0x7f4dd41eb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd3fff640 1 -- 172.21.15.27:0/4160720767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40c9e00 con 0x7f4dd415df00 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd2ffd640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dd40b9a10 0x7f4dd41605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:35.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd37fe640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40b95a0 0x7f4dd41eb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:35.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd3fff640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415df00 0x7f4dd40c8190 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009fa0 tx=0x7f4dcc00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dbbfff640 1 -- 172.21.15.27:0/4160720767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc01a070 con 0x7f4dd415df00 2024-03-06T07:35:35.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dbbfff640 1 -- 172.21.15.27:0/4160720767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc003bb0 con 0x7f4dd415df00 2024-03-06T07:35:35.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd40ca660 con 0x7f4dd415df00 2024-03-06T07:35:35.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dbbfff640 1 -- 172.21.15.27:0/4160720767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc004030 con 0x7f4dd415df00 2024-03-06T07:35:35.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.462+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd40ca380 con 0x7f4dd415df00 2024-03-06T07:35:35.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.464+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd4067ff0 con 0x7f4dd415df00 2024-03-06T07:35:35.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.464+0000 7f4dbbfff640 1 -- 172.21.15.27:0/4160720767 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dcc01d070 con 0x7f4dd415df00 2024-03-06T07:35:35.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.464+0000 7f4dbbfff640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4dc006b4d0 0x7f4dc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.464+0000 7f4dbbfff640 1 -- 172.21.15.27:0/4160720767 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4dcc08a560 con 0x7f4dd415df00 2024-03-06T07:35:35.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.464+0000 7f4dd37fe640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4dc006b4d0 0x7f4dc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.465+0000 7f4dd37fe640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4dc006b4d0 0x7f4dc006d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4000f60 tx=0x7f4dc4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.468+0000 7f4dbbfff640 1 -- 172.21.15.27:0/4160720767 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4dcc010070 con 0x7f4dd415df00 2024-03-06T07:35:35.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.687+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dd40c7530 con 0x7f4dd415df00 2024-03-06T07:35:35.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.687+0000 7f4dbbfff640 1 -- 172.21.15.27:0/4160720767 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f4dcc0593c0 con 0x7f4dd415df00 2024-03-06T07:35:35.689 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:35.689 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:35.690 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4dc006b4d0 msgr2=0x7f4dc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4dc006b4d0 0x7f4dc006d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4000f60 tx=0x7f4dc4009040 comp rx=0 tx=0).stop 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415df00 msgr2=0x7f4dd40c8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415df00 0x7f4dd40c8190 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009fa0 tx=0x7f4dcc00d560 comp rx=0 tx=0).stop 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 shutdown_connections 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f4dc006b4d0 0x7f4dc006d990 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415df00 0x7f4dd40c8190 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4dd40b9a10 0x7f4dd41605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 --2- 172.21.15.27:0/4160720767 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40b95a0 0x7f4dd41eb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 >> 172.21.15.27:0/4160720767 conn(0x7f4dd4071960 msgr2=0x7f4dd4071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 shutdown_connections 2024-03-06T07:35:35.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:35.689+0000 7f4dd9b2b640 1 -- 172.21.15.27:0/4160720767 wait complete. 2024-03-06T07:35:35.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:35.904 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:36.904 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.009+0000 7f8d25071640 1 Processor -- start 2024-03-06T07:35:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.009+0000 7f8d25071640 1 -- start start 2024-03-06T07:35:37.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d25071640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d200b0e90 0x7f8d2006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d25071640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d200ae9b0 0x7f8d200d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d25071640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d200cee90 con 0x7f8d2006b5c0 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d25071640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d200cfa00 con 0x7f8d200ae9b0 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d25071640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d200cfc50 con 0x7f8d200b0e90 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1f59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d200ae9b0 0x7f8d200d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1f59e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d200ae9b0 0x7f8d200d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60984/0 (socket says 172.21.15.27:60984) 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1ed9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8d2006b5c0 0x7f8d200b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51410/0 (socket says 172.21.15.27:51410) 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1f59e640 1 -- 172.21.15.27:0/2737291571 learned_addr learned my addr 172.21.15.27:0/2737291571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1e59c640 1 --2- 172.21.15.27:0/2737291571 >> v2:172.21.15.69:3300/0 conn(0x7f8d200b0e90 0x7f8d2006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1f59e640 1 -- 172.21.15.27:0/2737291571 >> v2:172.21.15.69:3300/0 conn(0x7f8d200b0e90 msgr2=0x7f8d2006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1f59e640 1 --2- 172.21.15.27:0/2737291571 >> v2:172.21.15.69:3300/0 conn(0x7f8d200b0e90 0x7f8d2006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1f59e640 1 -- 172.21.15.27:0/2737291571 >> v1:172.21.15.27:6789/0 conn(0x7f8d2006b5c0 legacy=0x7f8d200b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1f59e640 1 -- 172.21.15.27:0/2737291571 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200cff30 con 0x7f8d200ae9b0 2024-03-06T07:35:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1dd9b640 1 -- 172.21.15.27:0/2737291571 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965412235 0 0) 0x7f8d200cee90 con 0x7f8d2006b5c0 2024-03-06T07:35:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.010+0000 7f8d1e59c640 1 --2- 172.21.15.27:0/2737291571 >> v2:172.21.15.69:3300/0 conn(0x7f8d200b0e90 0x7f8d2006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.011+0000 7f8d1f59e640 1 --2- 172.21.15.27:0/2737291571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 0x7f8d200d03a0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f8d200b0e40 tx=0x7f8d1802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7017a818820666e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.011+0000 7f8d1dd9b640 1 -- 172.21.15.27:0/2737291571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1802ca80 con 0x7f8d200ae9b0 2024-03-06T07:35:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.011+0000 7f8d1dd9b640 1 -- 172.21.15.27:0/2737291571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1802cc20 con 0x7f8d200ae9b0 2024-03-06T07:35:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.011+0000 7f8d1dd9b640 1 -- 172.21.15.27:0/2737291571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180343f0 con 0x7f8d200ae9b0 2024-03-06T07:35:37.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.012+0000 7f8d25071640 1 -- 172.21.15.27:0/2737291571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 msgr2=0x7f8d200d03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.012+0000 7f8d25071640 1 --2- 172.21.15.27:0/2737291571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 0x7f8d200d03a0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f8d200b0e40 tx=0x7f8d1802bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.012+0000 7f8d25071640 1 -- 172.21.15.27:0/2737291571 shutdown_connections 2024-03-06T07:35:37.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.012+0000 7f8d25071640 1 --2- 172.21.15.27:0/2737291571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 0x7f8d200d03a0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.012+0000 7f8d25071640 1 --2- 172.21.15.27:0/2737291571 >> v2:172.21.15.69:3300/0 conn(0x7f8d200b0e90 0x7f8d2006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.012+0000 7f8d25071640 1 -- 172.21.15.27:0/2737291571 >> 172.21.15.27:0/2737291571 conn(0x7f8d200b36d0 msgr2=0x7f8d200b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.012+0000 7f8d25071640 1 -- 172.21.15.27:0/2737291571 shutdown_connections 2024-03-06T07:35:37.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.012+0000 7f8d25071640 1 -- 172.21.15.27:0/2737291571 wait complete. 2024-03-06T07:35:37.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.014+0000 7f8d25071640 1 Processor -- start 2024-03-06T07:35:37.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.014+0000 7f8d25071640 1 -- start start 2024-03-06T07:35:37.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.014+0000 7f8d25071640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2006b5c0 0x7f8d201d9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.014+0000 7f8d25071640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 0x7f8d200cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.014+0000 7f8d25071640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d200b0e90 0x7f8d201e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d25071640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2006c910 con 0x7f8d200ae9b0 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d25071640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2006c790 con 0x7f8d2006b5c0 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d25071640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d200aedc0 con 0x7f8d200b0e90 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 0x7f8d200cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 0x7f8d200cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60998/0 (socket says 172.21.15.27:60998) 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1e59c640 1 -- 172.21.15.27:0/2438564597 learned_addr learned my addr 172.21.15.27:0/2438564597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1f59e640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d200b0e90 0x7f8d201e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1ed9d640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2006b5c0 0x7f8d201d9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1e59c640 1 -- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d200b0e90 msgr2=0x7f8d201e82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1e59c640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d200b0e90 0x7f8d201e82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1e59c640 1 -- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2006b5c0 msgr2=0x7f8d201d9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1e59c640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2006b5c0 0x7f8d201d9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1e59c640 1 -- 172.21.15.27:0/2438564597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200cc5a0 con 0x7f8d200ae9b0 2024-03-06T07:35:37.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d1e59c640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 0x7f8d200cd640 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400d7d0 tx=0x7f8d1400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d077fe640 1 -- 172.21.15.27:0/2438564597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14004030 con 0x7f8d200ae9b0 2024-03-06T07:35:37.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d077fe640 1 -- 172.21.15.27:0/2438564597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d140041d0 con 0x7f8d200ae9b0 2024-03-06T07:35:37.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d077fe640 1 -- 172.21.15.27:0/2438564597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140096a0 con 0x7f8d200ae9b0 2024-03-06T07:35:37.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d200cc880 con 0x7f8d200ae9b0 2024-03-06T07:35:37.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.015+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d200cce40 con 0x7f8d200ae9b0 2024-03-06T07:35:37.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.017+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d20068700 con 0x7f8d200ae9b0 2024-03-06T07:35:37.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.017+0000 7f8d077fe640 1 -- 172.21.15.27:0/2438564597 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d14009840 con 0x7f8d200ae9b0 2024-03-06T07:35:37.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.017+0000 7f8d077fe640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d0c06b4d0 0x7f8d0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.017+0000 7f8d077fe640 1 -- 172.21.15.27:0/2438564597 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8d1408a480 con 0x7f8d200ae9b0 2024-03-06T07:35:37.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.017+0000 7f8d1ed9d640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d0c06b4d0 0x7f8d0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.018+0000 7f8d1ed9d640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d0c06b4d0 0x7f8d0c06d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8d10000f60 tx=0x7f8d10007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.022+0000 7f8d077fe640 1 -- 172.21.15.27:0/2438564597 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d14059370 con 0x7f8d200ae9b0 2024-03-06T07:35:37.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.247+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d201e6180 con 0x7f8d200ae9b0 2024-03-06T07:35:37.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.247+0000 7f8d077fe640 1 -- 172.21.15.27:0/2438564597 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f8d14058c70 con 0x7f8d200ae9b0 2024-03-06T07:35:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:37.250 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:37.250 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:37.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.250+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d0c06b4d0 msgr2=0x7f8d0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.250+0000 7f8d25071640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d0c06b4d0 0x7f8d0c06d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8d10000f60 tx=0x7f8d10007b50 comp rx=0 tx=0).stop 2024-03-06T07:35:37.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.250+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 msgr2=0x7f8d200cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.250+0000 7f8d25071640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 0x7f8d200cd640 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400d7d0 tx=0x7f8d1400dc90 comp rx=0 tx=0).stop 2024-03-06T07:35:37.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.251+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 shutdown_connections 2024-03-06T07:35:37.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.251+0000 7f8d25071640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8d0c06b4d0 0x7f8d0c06d990 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.251+0000 7f8d25071640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8d200b0e90 0x7f8d201e82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.251+0000 7f8d25071640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d200ae9b0 0x7f8d200cd640 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.251+0000 7f8d25071640 1 --2- 172.21.15.27:0/2438564597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d2006b5c0 0x7f8d201d9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.251+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 >> 172.21.15.27:0/2438564597 conn(0x7f8d200b36d0 msgr2=0x7f8d201e9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.251+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 shutdown_connections 2024-03-06T07:35:37.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:37.251+0000 7f8d25071640 1 -- 172.21.15.27:0/2438564597 wait complete. 2024-03-06T07:35:37.470 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:37.470 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:38.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:38.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.575+0000 7fdaf7fe2640 1 Processor -- start 2024-03-06T07:35:38.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.575+0000 7fdaf7fe2640 1 -- start start 2024-03-06T07:35:38.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.576+0000 7fdaf7fe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf015df00 0x7fdaf00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.576+0000 7fdaf7fe2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdaf00b95a0 0x7fdaf00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.576+0000 7fdaf7fe2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf00cea30 con 0x7fdaf00b9a30 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.576+0000 7fdaf7fe2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf00d2af0 con 0x7fdaf00b95a0 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.576+0000 7fdaf7fe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00cf3c0 con 0x7fdaf015df00 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.576+0000 7fdaf657f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdaf00b9a30 0x7fdaf00d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51420/0 (socket says 172.21.15.27:51420) 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.576+0000 7fdaf657f640 1 -- 172.21.15.27:0/3128736288 learned_addr learned my addr 172.21.15.27:0/3128736288 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf5d7e640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf015df00 0x7fdaf00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf557d640 1 --2- 172.21.15.27:0/3128736288 >> v2:172.21.15.69:3300/0 conn(0x7fdaf00b95a0 0x7fdaf00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840468923 0 0) 0x7fdaf00cea30 con 0x7fdaf00b9a30 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdadc003520 con 0x7fdaf00b9a30 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3019663630 0 0) 0x7fdadc003520 con 0x7fdaf00b9a30 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf015df00 msgr2=0x7fdaf00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf015df00 0x7fdaf00d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 >> v2:172.21.15.69:3300/0 conn(0x7fdaf00b95a0 msgr2=0x7fdaf00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 --2- 172.21.15.27:0/3128736288 >> v2:172.21.15.69:3300/0 conn(0x7fdaf00b95a0 0x7fdaf00b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf00cf0e0 con 0x7fdaf00b9a30 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fdae8001670 con 0x7fdaf00b9a30 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 >> v1:172.21.15.27:6789/0 conn(0x7fdaf00b9a30 legacy=0x7fdaf00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf015df00 0x7fdadc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdadc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdadc004d40 0x7fdadc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00d2af0 con 0x7fdadc003f30 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00cf3c0 con 0x7fdadc004d40 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc00b9c0 con 0x7fdaf015df00 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf557d640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdadc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf5d7e640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf015df00 0x7fdadc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf657f640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdadc004d40 0x7fdadc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf557d640 1 -- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf015df00 msgr2=0x7fdadc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf557d640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf015df00 0x7fdadc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf557d640 1 -- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdadc004d40 msgr2=0x7fdadc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.577+0000 7fdaf557d640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdadc004d40 0x7fdadc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.578+0000 7fdaf557d640 1 -- 172.21.15.27:0/3128736288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc003520 con 0x7fdadc003f30 2024-03-06T07:35:38.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.578+0000 7fdaf657f640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdadc004d40 0x7fdadc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:38.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.578+0000 7fdaf557d640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdadc004800 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009ac0 tx=0x7fdadc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b808a863a0d496e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.578+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0039070 con 0x7fdadc003f30 2024-03-06T07:35:38.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.578+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae002c6d0 con 0x7fdadc003f30 2024-03-06T07:35:38.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.578+0000 7fdaf4d7c640 1 -- 172.21.15.27:0/3128736288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0009690 con 0x7fdadc003f30 2024-03-06T07:35:38.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.579+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/3128736288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 msgr2=0x7fdadc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.579+0000 7fdaf7fe2640 1 --2- 172.21.15.27:0/3128736288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdadc004800 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009ac0 tx=0x7fdadc005830 comp rx=0 tx=0).stop 2024-03-06T07:35:38.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.579+0000 7fdaf657f640 1 -- 172.21.15.27:0/3128736288 reap_dead start 2024-03-06T07:35:38.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.579+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/3128736288 shutdown_connections 2024-03-06T07:35:38.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.579+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/3128736288 >> 172.21.15.27:0/3128736288 conn(0x7fdaf0071960 msgr2=0x7fdaf0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:38.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.579+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/3128736288 shutdown_connections 2024-03-06T07:35:38.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.579+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/3128736288 wait complete. 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.580+0000 7fdaf7fe2640 1 Processor -- start 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.580+0000 7fdaf7fe2640 1 -- start start 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf7fe2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdaf0151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf7fe2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0157a30 0x7fdaf0157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf7fe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf00b7aa0 0x7fdaf00b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf7fe2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00cec70 con 0x7fdadc003f30 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf7fe2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00ce750 con 0x7fdaf0157a30 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf7fe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00cedf0 con 0x7fdaf00b7aa0 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf5d7e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdaf0151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf5d7e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdaf0151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32786/0 (socket says 172.21.15.27:32786) 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf5d7e640 1 -- 172.21.15.27:0/1483301972 learned_addr learned my addr 172.21.15.27:0/1483301972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf657f640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf00b7aa0 0x7fdaf00b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf557d640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0157a30 0x7fdaf0157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf5d7e640 1 -- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf00b7aa0 msgr2=0x7fdaf00b7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf5d7e640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf00b7aa0 0x7fdaf00b7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf5d7e640 1 -- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0157a30 msgr2=0x7fdaf0157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf5d7e640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0157a30 0x7fdaf0157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf5d7e640 1 -- 172.21.15.27:0/1483301972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf00b35e0 con 0x7fdadc003f30 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.581+0000 7fdaf557d640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0157a30 0x7fdaf0157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.582+0000 7fdaf5d7e640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdaf0151d40 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fdaec011e90 tx=0x7fdaec00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.582+0000 7fdae67fc640 1 -- 172.21.15.27:0/1483301972 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec015630 con 0x7fdadc003f30 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.582+0000 7fdae67fc640 1 -- 172.21.15.27:0/1483301972 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec0157d0 con 0x7fdadc003f30 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.582+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf00b38c0 con 0x7fdadc003f30 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.582+0000 7fdae67fc640 1 -- 172.21.15.27:0/1483301972 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec015ba0 con 0x7fdadc003f30 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.582+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf00b3e80 con 0x7fdadc003f30 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.583+0000 7fdae67fc640 1 -- 172.21.15.27:0/1483301972 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdaec01d400 con 0x7fdadc003f30 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.583+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf0067ff0 con 0x7fdadc003f30 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.584+0000 7fdae67fc640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.584+0000 7fdae67fc640 1 -- 172.21.15.27:0/1483301972 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdaec08b3e0 con 0x7fdadc003f30 2024-03-06T07:35:38.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.584+0000 7fdaf557d640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.585+0000 7fdaf557d640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fdae0036000 tx=0x7fdae0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.588+0000 7fdae67fc640 1 -- 172.21.15.27:0/1483301972 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdaec019030 con 0x7fdadc003f30 2024-03-06T07:35:38.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.808+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaf00b7580 con 0x7fdadc003f30 2024-03-06T07:35:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.809+0000 7fdae67fc640 1 -- 172.21.15.27:0/1483301972 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fdaec010040 con 0x7fdadc003f30 2024-03-06T07:35:38.811 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:38.811 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 msgr2=0x7fdad406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fdae0036000 tx=0x7fdae0030000 comp rx=0 tx=0).stop 2024-03-06T07:35:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 msgr2=0x7fdaf0151d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdaf0151d40 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fdaec011e90 tx=0x7fdaec00f580 comp rx=0 tx=0).stop 2024-03-06T07:35:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 shutdown_connections 2024-03-06T07:35:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdad406b4d0 0x7fdad406d990 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdaf00b7aa0 0x7fdaf00b7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdaf0157a30 0x7fdaf0157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 --2- 172.21.15.27:0/1483301972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdadc003f30 0x7fdaf0151d40 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 >> 172.21.15.27:0/1483301972 conn(0x7fdaf0071960 msgr2=0x7fdaf00d3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 shutdown_connections 2024-03-06T07:35:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:38.811+0000 7fdaf7fe2640 1 -- 172.21.15.27:0/1483301972 wait complete. 2024-03-06T07:35:39.030 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:39.030 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:40.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:40.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.136+0000 7fe846415640 1 Processor -- start 2024-03-06T07:35:40.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.136+0000 7fe846415640 1 -- start start 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.137+0000 7fe846415640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe840158520 0x7fe84006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.137+0000 7fe846415640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe840071240 con 0x7fe84006dd60 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.137+0000 7fe846415640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe840071520 con 0x7fe8400aee10 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.137+0000 7fe846415640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe840071b70 con 0x7fe840158520 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.137+0000 7fe83ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe840158520 0x7fe84006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.137+0000 7fe83ffff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe840158520 0x7fe84006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32798/0 (socket says 172.21.15.27:32798) 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.137+0000 7fe83ffff640 1 -- 172.21.15.27:0/381716321 learned_addr learned my addr 172.21.15.27:0/381716321 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.137+0000 7fe83f7fe640 1 --1- 172.21.15.27:0/381716321 >> v1:172.21.15.27:6789/0 conn(0x7fe84006dd60 0x7fe8400ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51430/0 (socket says 172.21.15.27:51430) 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.138+0000 7fe83ffff640 1 -- 172.21.15.27:0/381716321 >> v1:172.21.15.69:6789/0 conn(0x7fe8400aee10 legacy=0x7fe8400b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.138+0000 7fe83ffff640 1 -- 172.21.15.27:0/381716321 >> v1:172.21.15.27:6789/0 conn(0x7fe84006dd60 legacy=0x7fe8400ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.138+0000 7fe83ffff640 1 -- 172.21.15.27:0/381716321 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe840071800 con 0x7fe840158520 2024-03-06T07:35:40.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.138+0000 7fe83effd640 1 -- 172.21.15.27:0/381716321 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873578506 0 0) 0x7fe840071240 con 0x7fe84006dd60 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.138+0000 7fe83ffff640 1 --2- 172.21.15.27:0/381716321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe840158520 0x7fe84006d820 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fe840071cf0 tx=0x7fe83402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08d802cb18138c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe83effd640 1 -- 172.21.15.27:0/381716321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83402f840 con 0x7fe840158520 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe83effd640 1 -- 172.21.15.27:0/381716321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83402f9e0 con 0x7fe840158520 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe83effd640 1 -- 172.21.15.27:0/381716321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83402fdb0 con 0x7fe840158520 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe846415640 1 -- 172.21.15.27:0/381716321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe840158520 msgr2=0x7fe84006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe846415640 1 --2- 172.21.15.27:0/381716321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe840158520 0x7fe84006d820 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fe840071cf0 tx=0x7fe83402ba20 comp rx=0 tx=0).stop 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe846415640 1 -- 172.21.15.27:0/381716321 shutdown_connections 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe846415640 1 --2- 172.21.15.27:0/381716321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe840158520 0x7fe84006d820 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe846415640 1 -- 172.21.15.27:0/381716321 >> 172.21.15.27:0/381716321 conn(0x7fe840066d00 msgr2=0x7fe8400670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe846415640 1 -- 172.21.15.27:0/381716321 shutdown_connections 2024-03-06T07:35:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.139+0000 7fe846415640 1 -- 172.21.15.27:0/381716321 wait complete. 2024-03-06T07:35:40.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.140+0000 7fe846415640 1 Processor -- start 2024-03-06T07:35:40.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe846415640 1 -- start start 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe846415640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe84006dd60 0x7fe8400adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe846415640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8400aee10 0x7fe8400ce950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe846415640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe840158520 0x7fe8400cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe846415640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe840158a80 con 0x7fe8400aee10 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe846415640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe840158900 con 0x7fe84006dd60 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe846415640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8400b3d50 con 0x7fe840158520 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe83f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8400aee10 0x7fe8400ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe83f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8400aee10 0x7fe8400ce950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32802/0 (socket says 172.21.15.27:32802) 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.141+0000 7fe83f7fe640 1 -- 172.21.15.27:0/2470616829 learned_addr learned my addr 172.21.15.27:0/2470616829 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83ffff640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe84006dd60 0x7fe8400adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe8449b2640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe840158520 0x7fe8400cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83f7fe640 1 -- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe840158520 msgr2=0x7fe8400cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83f7fe640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe840158520 0x7fe8400cee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83f7fe640 1 -- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe84006dd60 msgr2=0x7fe8400adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83f7fe640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe84006dd60 0x7fe8400adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83f7fe640 1 -- 172.21.15.27:0/2470616829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8400af3b0 con 0x7fe8400aee10 2024-03-06T07:35:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83f7fe640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8400aee10 0x7fe8400ce950 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fe83000c4a0 tx=0x7fe83000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83cff9640 1 -- 172.21.15.27:0/2470616829 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830018070 con 0x7fe8400aee10 2024-03-06T07:35:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83cff9640 1 -- 172.21.15.27:0/2470616829 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8300074e0 con 0x7fe8400aee10 2024-03-06T07:35:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8400af650 con 0x7fe8400aee10 2024-03-06T07:35:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe83cff9640 1 -- 172.21.15.27:0/2470616829 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830004a20 con 0x7fe8400aee10 2024-03-06T07:35:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.142+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8400afef0 con 0x7fe8400aee10 2024-03-06T07:35:40.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.143+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8400af820 con 0x7fe8400aee10 2024-03-06T07:35:40.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.144+0000 7fe83cff9640 1 -- 172.21.15.27:0/2470616829 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe83001b070 con 0x7fe8400aee10 2024-03-06T07:35:40.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.144+0000 7fe83cff9640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe82c06b4d0 0x7fe82c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.144+0000 7fe83cff9640 1 -- 172.21.15.27:0/2470616829 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe830089470 con 0x7fe8400aee10 2024-03-06T07:35:40.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.145+0000 7fe83ffff640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe82c06b4d0 0x7fe82c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.145+0000 7fe83ffff640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe82c06b4d0 0x7fe82c06d990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe834036000 tx=0x7fe834008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.149+0000 7fe83cff9640 1 -- 172.21.15.27:0/2470616829 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe830058360 con 0x7fe8400aee10 2024-03-06T07:35:40.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.370+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe840067ff0 con 0x7fe8400aee10 2024-03-06T07:35:40.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.370+0000 7fe83cff9640 1 -- 172.21.15.27:0/2470616829 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fe830006e20 con 0x7fe8400aee10 2024-03-06T07:35:40.373 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:40.373 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:40.373 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe82c06b4d0 msgr2=0x7fe82c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe82c06b4d0 0x7fe82c06d990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe834036000 tx=0x7fe834008030 comp rx=0 tx=0).stop 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8400aee10 msgr2=0x7fe8400ce950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8400aee10 0x7fe8400ce950 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fe83000c4a0 tx=0x7fe83000c960 comp rx=0 tx=0).stop 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 shutdown_connections 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe82c06b4d0 0x7fe82c06d990 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe840158520 0x7fe8400cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8400aee10 0x7fe8400ce950 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 --2- 172.21.15.27:0/2470616829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe84006dd60 0x7fe8400adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 >> 172.21.15.27:0/2470616829 conn(0x7fe840066d00 msgr2=0x7fe8400b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 shutdown_connections 2024-03-06T07:35:40.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:40.372+0000 7fe846415640 1 -- 172.21.15.27:0/2470616829 wait complete. 2024-03-06T07:35:40.586 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:40.587 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:41.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:41.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.692+0000 7f96c7088640 1 Processor -- start 2024-03-06T07:35:41.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.692+0000 7f96c7088640 1 -- start start 2024-03-06T07:35:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.693+0000 7f96c7088640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96c00b95a0 0x7f96c00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.693+0000 7f96c7088640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96c00b9a10 0x7f96c00b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.693+0000 7f96c7088640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c00cea30 con 0x7f96c015df00 2024-03-06T07:35:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.693+0000 7f96c7088640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c00cf3c0 con 0x7f96c00b9a10 2024-03-06T07:35:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.693+0000 7f96c7088640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c00cf0e0 con 0x7f96c00b95a0 2024-03-06T07:35:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.693+0000 7f96c5625640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96c00b9a10 0x7f96c00b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.693+0000 7f96c5625640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96c00b9a10 0x7f96c00b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32804/0 (socket says 172.21.15.27:32804) 2024-03-06T07:35:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.693+0000 7f96c5625640 1 -- 172.21.15.27:0/1387559521 learned_addr learned my addr 172.21.15.27:0/1387559521 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.694+0000 7f96bffff640 1 --2- 172.21.15.27:0/1387559521 >> v2:172.21.15.69:3300/0 conn(0x7f96c00b95a0 0x7f96c00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.694+0000 7f96c5625640 1 -- 172.21.15.27:0/1387559521 >> v2:172.21.15.69:3300/0 conn(0x7f96c00b95a0 msgr2=0x7f96c00b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.694+0000 7f96c5625640 1 --2- 172.21.15.27:0/1387559521 >> v2:172.21.15.69:3300/0 conn(0x7f96c00b95a0 0x7f96c00b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.694+0000 7f96c5625640 1 -- 172.21.15.27:0/1387559521 >> v1:172.21.15.69:6789/0 conn(0x7f96c015df00 legacy=0x7f96c00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.694+0000 7f96c5625640 1 -- 172.21.15.27:0/1387559521 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c00ce470 con 0x7f96c00b9a10 2024-03-06T07:35:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.694+0000 7f96c5625640 1 --2- 172.21.15.27:0/1387559521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c00b9a10 0x7f96c00b5c40 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f96c00d1760 tx=0x7f96b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7937b5100bc34b35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.695+0000 7f96bf7fe640 1 -- 172.21.15.27:0/1387559521 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b0034070 con 0x7f96c00b9a10 2024-03-06T07:35:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.695+0000 7f96bf7fe640 1 -- 172.21.15.27:0/1387559521 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b0030860 con 0x7f96c00b9a10 2024-03-06T07:35:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.695+0000 7f96bf7fe640 1 -- 172.21.15.27:0/1387559521 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b0030c30 con 0x7f96c00b9a10 2024-03-06T07:35:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.695+0000 7f96c7088640 1 -- 172.21.15.27:0/1387559521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c00b9a10 msgr2=0x7f96c00b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.695+0000 7f96c7088640 1 --2- 172.21.15.27:0/1387559521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c00b9a10 0x7f96c00b5c40 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f96c00d1760 tx=0x7f96b002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.695+0000 7f96c7088640 1 -- 172.21.15.27:0/1387559521 shutdown_connections 2024-03-06T07:35:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.696+0000 7f96c7088640 1 --2- 172.21.15.27:0/1387559521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c00b9a10 0x7f96c00b5c40 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.696+0000 7f96c7088640 1 --2- 172.21.15.27:0/1387559521 >> v2:172.21.15.69:3300/0 conn(0x7f96c00b95a0 0x7f96c00b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.696+0000 7f96c7088640 1 -- 172.21.15.27:0/1387559521 >> 172.21.15.27:0/1387559521 conn(0x7f96c0071960 msgr2=0x7f96c0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.696+0000 7f96c7088640 1 -- 172.21.15.27:0/1387559521 shutdown_connections 2024-03-06T07:35:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.696+0000 7f96c7088640 1 -- 172.21.15.27:0/1387559521 wait complete. 2024-03-06T07:35:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.698+0000 7f96c7088640 1 Processor -- start 2024-03-06T07:35:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.698+0000 7f96c7088640 1 -- start start 2024-03-06T07:35:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c7088640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96c00b95a0 0x7f96c00cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c7088640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96c00b9a10 0x7f96c015f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c7088640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c015df00 0x7f96c01e8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c7088640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00cedf0 con 0x7f96c015df00 2024-03-06T07:35:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c7088640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00cec70 con 0x7f96c00b95a0 2024-03-06T07:35:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c7088640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00d3220 con 0x7f96c00b9a10 2024-03-06T07:35:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c5625640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c015df00 0x7f96c01e8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c5625640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c015df00 0x7f96c01e8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32812/0 (socket says 172.21.15.27:32812) 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c5625640 1 -- 172.21.15.27:0/3571641131 learned_addr learned my addr 172.21.15.27:0/3571641131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96c4e24640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96c00b95a0 0x7f96c00cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.699+0000 7f96bffff640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96c00b9a10 0x7f96c015f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96c5625640 1 -- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96c00b9a10 msgr2=0x7f96c015f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96c5625640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96c00b9a10 0x7f96c015f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96c5625640 1 -- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96c00b95a0 msgr2=0x7f96c00cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96c5625640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96c00b95a0 0x7f96c00cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96c5625640 1 -- 172.21.15.27:0/3571641131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c0160c70 con 0x7f96c015df00 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96bffff640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96c00b9a10 0x7f96c015f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96c5625640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c015df00 0x7f96c01e8aa0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f96b002b9e0 tx=0x7f96b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96bd7fa640 1 -- 172.21.15.27:0/3571641131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b0048070 con 0x7f96c015df00 2024-03-06T07:35:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96bd7fa640 1 -- 172.21.15.27:0/3571641131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b0030660 con 0x7f96c015df00 2024-03-06T07:35:41.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c015fbd0 con 0x7f96c015df00 2024-03-06T07:35:41.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96bd7fa640 1 -- 172.21.15.27:0/3571641131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b0034040 con 0x7f96c015df00 2024-03-06T07:35:41.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.700+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c0160190 con 0x7f96c015df00 2024-03-06T07:35:41.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.701+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c00b6d70 con 0x7f96c015df00 2024-03-06T07:35:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.702+0000 7f96bd7fa640 1 -- 172.21.15.27:0/3571641131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96b004b070 con 0x7f96c015df00 2024-03-06T07:35:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.702+0000 7f96bd7fa640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96ac06b500 0x7f96ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.702+0000 7f96bd7fa640 1 -- 172.21.15.27:0/3571641131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f96b00ac1b0 con 0x7f96c015df00 2024-03-06T07:35:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.703+0000 7f96c4e24640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96ac06b500 0x7f96ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.703+0000 7f96c4e24640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96ac06b500 0x7f96ac06d9c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f96b8001ca0 tx=0x7f96b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.707+0000 7f96bd7fa640 1 -- 172.21.15.27:0/3571641131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96c00b6d70 con 0x7f96c015df00 2024-03-06T07:35:41.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.926+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96c0067ff0 con 0x7f96c015df00 2024-03-06T07:35:41.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.926+0000 7f96bd7fa640 1 -- 172.21.15.27:0/3571641131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f96b007b130 con 0x7f96c015df00 2024-03-06T07:35:41.929 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:41.929 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:41.929 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.929+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96ac06b500 msgr2=0x7f96ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.929+0000 7f96c7088640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96ac06b500 0x7f96ac06d9c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f96b8001ca0 tx=0x7f96b8009040 comp rx=0 tx=0).stop 2024-03-06T07:35:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.929+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c015df00 msgr2=0x7f96c01e8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.929+0000 7f96c7088640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c015df00 0x7f96c01e8aa0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f96b002b9e0 tx=0x7f96b0004730 comp rx=0 tx=0).stop 2024-03-06T07:35:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.929+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 shutdown_connections 2024-03-06T07:35:41.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.929+0000 7f96c7088640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f96ac06b500 0x7f96ac06d9c0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.929+0000 7f96c7088640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96c015df00 0x7f96c01e8aa0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.930+0000 7f96c7088640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f96c00b9a10 0x7f96c015f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.930+0000 7f96c7088640 1 --2- 172.21.15.27:0/3571641131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96c00b95a0 0x7f96c00cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.930+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 >> 172.21.15.27:0/3571641131 conn(0x7f96c0071960 msgr2=0x7f96c015d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:41.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.930+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 shutdown_connections 2024-03-06T07:35:41.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:41.930+0000 7f96c7088640 1 -- 172.21.15.27:0/3571641131 wait complete. 2024-03-06T07:35:42.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:42.145 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:43.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:43.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.251+0000 7f1935f3e640 1 Processor -- start 2024-03-06T07:35:43.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.251+0000 7f1935f3e640 1 -- start start 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f1935f3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f193015df00 0x7f19300d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f1935f3e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19300b95a0 0x7f19300b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f1935f3e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19300cea30 con 0x7f19300b9a30 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f1935f3e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19300d2af0 con 0x7f19300b95a0 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f1935f3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19300cf3c0 con 0x7f193015df00 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f192ffff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19300b9a30 0x7f19300d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51434/0 (socket says 172.21.15.27:51434) 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f192ffff640 1 -- 172.21.15.27:0/2779909097 learned_addr learned my addr 172.21.15.27:0/2779909097 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f192f7fe640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f193015df00 0x7f19300d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f192effd640 1 --2- 172.21.15.27:0/2779909097 >> v2:172.21.15.69:3300/0 conn(0x7f19300b95a0 0x7f19300b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685869372 0 0) 0x7f19300cea30 con 0x7f19300b9a30 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f191c003520 con 0x7f19300b9a30 2024-03-06T07:35:43.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.252+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1891540487 0 0) 0x7f191c003520 con 0x7f19300b9a30 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f193015df00 msgr2=0x7f19300d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f193015df00 0x7f19300d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 >> v2:172.21.15.69:3300/0 conn(0x7f19300b95a0 msgr2=0x7f19300b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 --2- 172.21.15.27:0/2779909097 >> v2:172.21.15.69:3300/0 conn(0x7f19300b95a0 0x7f19300b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19300cf0e0 con 0x7f19300b9a30 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f19280019f0 con 0x7f19300b9a30 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 >> v1:172.21.15.27:6789/0 conn(0x7f19300b9a30 legacy=0x7f19300d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0039b0 0x7f191c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f193015df00 0x7f191c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f191c004d40 0x7f191c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300d2af0 con 0x7f191c0039b0 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300cf3c0 con 0x7f191c004d40 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191c00b9c0 con 0x7f193015df00 2024-03-06T07:35:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192f7fe640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0039b0 0x7f191c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192ffff640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f191c004d40 0x7f191c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192effd640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f193015df00 0x7f191c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192f7fe640 1 -- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f193015df00 msgr2=0x7f191c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192f7fe640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f193015df00 0x7f191c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192f7fe640 1 -- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f191c004d40 msgr2=0x7f191c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192f7fe640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f191c004d40 0x7f191c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.253+0000 7f192f7fe640 1 -- 172.21.15.27:0/2779909097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c003520 con 0x7f191c0039b0 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.254+0000 7f192f7fe640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0039b0 0x7f191c003dd0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f1920009ac0 tx=0x7f191c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d50fa2ee96ebe4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1920039070 con 0x7f191c0039b0 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192002c6b0 con 0x7f191c0039b0 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f192e7fc640 1 -- 172.21.15.27:0/2779909097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1920009690 con 0x7f191c0039b0 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f1935f3e640 1 -- 172.21.15.27:0/2779909097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0039b0 msgr2=0x7f191c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f1935f3e640 1 --2- 172.21.15.27:0/2779909097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0039b0 0x7f191c003dd0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f1920009ac0 tx=0x7f191c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:43.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f192ffff640 1 -- 172.21.15.27:0/2779909097 reap_dead start 2024-03-06T07:35:43.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f1935f3e640 1 -- 172.21.15.27:0/2779909097 shutdown_connections 2024-03-06T07:35:43.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f1935f3e640 1 -- 172.21.15.27:0/2779909097 >> 172.21.15.27:0/2779909097 conn(0x7f1930071960 msgr2=0x7f1930071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:43.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f1935f3e640 1 -- 172.21.15.27:0/2779909097 shutdown_connections 2024-03-06T07:35:43.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.255+0000 7f1935f3e640 1 -- 172.21.15.27:0/2779909097 wait complete. 2024-03-06T07:35:43.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.256+0000 7f1935f3e640 1 Processor -- start 2024-03-06T07:35:43.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f1935f3e640 1 -- start start 2024-03-06T07:35:43.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f1935f3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f191c0039b0 0x7f19300ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f1935f3e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193015ee80 0x7f19300b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f1935f3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19301542d0 0x7f1930154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f1935f3e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300cec70 con 0x7f193015ee80 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f1935f3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300ce750 con 0x7f191c0039b0 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f1935f3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19300cedf0 con 0x7f19301542d0 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f192effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193015ee80 0x7f19300b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f192effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193015ee80 0x7f19300b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32834/0 (socket says 172.21.15.27:32834) 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.257+0000 7f192effd640 1 -- 172.21.15.27:0/811656361 learned_addr learned my addr 172.21.15.27:0/811656361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f192f7fe640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f191c0039b0 0x7f19300ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f192ffff640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19301542d0 0x7f1930154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f192effd640 1 -- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19301542d0 msgr2=0x7f1930154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f192effd640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19301542d0 0x7f1930154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f192effd640 1 -- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f191c0039b0 msgr2=0x7f19300ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f192effd640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f191c0039b0 0x7f19300ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f192effd640 1 -- 172.21.15.27:0/811656361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1930154c90 con 0x7f193015ee80 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f192ffff640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19301542d0 0x7f1930154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f192effd640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193015ee80 0x7f19300b4b50 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f1918011aa0 tx=0x7f1918011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f1927fff640 1 -- 172.21.15.27:0/811656361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191800fb80 con 0x7f193015ee80 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f1927fff640 1 -- 172.21.15.27:0/811656361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191800fd20 con 0x7f193015ee80 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193015f3d0 con 0x7f193015ee80 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f1927fff640 1 -- 172.21.15.27:0/811656361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1918014660 con 0x7f193015ee80 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.258+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193015f990 con 0x7f193015ee80 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.259+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193015fbd0 con 0x7f193015ee80 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.260+0000 7f1927fff640 1 -- 172.21.15.27:0/811656361 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1918012040 con 0x7f193015ee80 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.260+0000 7f1927fff640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f191006b500 0x7f191006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.260+0000 7f1927fff640 1 -- 172.21.15.27:0/811656361 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f191808a8e0 con 0x7f193015ee80 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.260+0000 7f192f7fe640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f191006b500 0x7f191006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.261+0000 7f192f7fe640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f191006b500 0x7f191006d9c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1920036000 tx=0x7f1920030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:43.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.264+0000 7f1927fff640 1 -- 172.21.15.27:0/811656361 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1918019030 con 0x7f193015ee80 2024-03-06T07:35:43.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.483+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19300cc130 con 0x7f193015ee80 2024-03-06T07:35:43.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.484+0000 7f1927fff640 1 -- 172.21.15.27:0/811656361 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f19180596b0 con 0x7f193015ee80 2024-03-06T07:35:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:43.486 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:43.487 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:43.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f191006b500 msgr2=0x7f191006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f191006b500 0x7f191006d9c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1920036000 tx=0x7f1920030000 comp rx=0 tx=0).stop 2024-03-06T07:35:43.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193015ee80 msgr2=0x7f19300b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193015ee80 0x7f19300b4b50 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f1918011aa0 tx=0x7f1918011f60 comp rx=0 tx=0).stop 2024-03-06T07:35:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 shutdown_connections 2024-03-06T07:35:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f191006b500 0x7f191006d9c0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f19301542d0 0x7f1930154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193015ee80 0x7f19300b4b50 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 --2- 172.21.15.27:0/811656361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f191c0039b0 0x7f19300ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 >> 172.21.15.27:0/811656361 conn(0x7f1930071960 msgr2=0x7f19300b88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 shutdown_connections 2024-03-06T07:35:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:43.486+0000 7f1935f3e640 1 -- 172.21.15.27:0/811656361 wait complete. 2024-03-06T07:35:43.701 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:43.701 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:44.702 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.807+0000 7f48bdd56640 1 Processor -- start 2024-03-06T07:35:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.807+0000 7f48bdd56640 1 -- start start 2024-03-06T07:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.807+0000 7f48bdd56640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f48b815df00 0x7f48b80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.807+0000 7f48bdd56640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b80cea30 con 0x7f48b80b9a30 2024-03-06T07:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.807+0000 7f48bdd56640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b80d2af0 con 0x7f48b80b95a0 2024-03-06T07:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.807+0000 7f48bdd56640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b80cf3c0 con 0x7f48b815df00 2024-03-06T07:35:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.808+0000 7f48b77fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f48b815df00 0x7f48b80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.808+0000 7f48b7fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f48b80b9a30 0x7f48b80d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42704/0 (socket says 172.21.15.27:42704) 2024-03-06T07:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.808+0000 7f48b7fff640 1 -- 172.21.15.27:0/3137570530 learned_addr learned my addr 172.21.15.27:0/3137570530 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.808+0000 7f48b77fe640 1 -- 172.21.15.27:0/3137570530 >> v1:172.21.15.69:6789/0 conn(0x7f48b80b95a0 legacy=0x7f48b80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.808+0000 7f48b77fe640 1 -- 172.21.15.27:0/3137570530 >> v1:172.21.15.27:6789/0 conn(0x7f48b80b9a30 legacy=0x7f48b80d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.808+0000 7f48b67fc640 1 -- 172.21.15.27:0/3137570530 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335287603 0 0) 0x7f48b80cea30 con 0x7f48b80b9a30 2024-03-06T07:35:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.808+0000 7f48b77fe640 1 -- 172.21.15.27:0/3137570530 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b80cf0e0 con 0x7f48b815df00 2024-03-06T07:35:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.809+0000 7f48b77fe640 1 --2- 172.21.15.27:0/3137570530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b815df00 0x7f48b80d25b0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f48a80097a0 tx=0x7f48a802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86980dbf36ff0bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.809+0000 7f48b67fc640 1 -- 172.21.15.27:0/3137570530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8034070 con 0x7f48b815df00 2024-03-06T07:35:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.809+0000 7f48b67fc640 1 -- 172.21.15.27:0/3137570530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a8030610 con 0x7f48b815df00 2024-03-06T07:35:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.809+0000 7f48b67fc640 1 -- 172.21.15.27:0/3137570530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a80309e0 con 0x7f48b815df00 2024-03-06T07:35:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.810+0000 7f48bdd56640 1 -- 172.21.15.27:0/3137570530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b815df00 msgr2=0x7f48b80d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.810+0000 7f48bdd56640 1 --2- 172.21.15.27:0/3137570530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b815df00 0x7f48b80d25b0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f48a80097a0 tx=0x7f48a802b960 comp rx=0 tx=0).stop 2024-03-06T07:35:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.810+0000 7f48bdd56640 1 -- 172.21.15.27:0/3137570530 shutdown_connections 2024-03-06T07:35:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.810+0000 7f48bdd56640 1 --2- 172.21.15.27:0/3137570530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b815df00 0x7f48b80d25b0 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.810+0000 7f48bdd56640 1 -- 172.21.15.27:0/3137570530 >> 172.21.15.27:0/3137570530 conn(0x7f48b8071960 msgr2=0x7f48b8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.810+0000 7f48bdd56640 1 -- 172.21.15.27:0/3137570530 shutdown_connections 2024-03-06T07:35:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.810+0000 7f48bdd56640 1 -- 172.21.15.27:0/3137570530 wait complete. 2024-03-06T07:35:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.811+0000 7f48bdd56640 1 Processor -- start 2024-03-06T07:35:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.811+0000 7f48bdd56640 1 -- start start 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48bdd56640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b80b95a0 0x7f48b815fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48bdd56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b815df00 0x7f48b80c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48bdd56640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b8151f10 0x7f48b80c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48bdd56640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80cec70 con 0x7f48b8151f10 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48bdd56640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80ce750 con 0x7f48b80b95a0 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48bdd56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80cedf0 con 0x7f48b815df00 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b8151f10 0x7f48b80c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b8151f10 0x7f48b80c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36934/0 (socket says 172.21.15.27:36934) 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48b7fff640 1 -- 172.21.15.27:0/4216530890 learned_addr learned my addr 172.21.15.27:0/4216530890 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.812+0000 7f48b77fe640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b80b95a0 0x7f48b815fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48b6ffd640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b815df00 0x7f48b80c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48b7fff640 1 -- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b815df00 msgr2=0x7f48b80c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48b7fff640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b815df00 0x7f48b80c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48b7fff640 1 -- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b80b95a0 msgr2=0x7f48b815fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48b7fff640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b80b95a0 0x7f48b815fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48b7fff640 1 -- 172.21.15.27:0/4216530890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b80c9380 con 0x7f48b8151f10 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48b77fe640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b80b95a0 0x7f48b815fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48b7fff640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b8151f10 0x7f48b80c7cf0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f48b000c7a0 tx=0x7f48b000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f489ffff640 1 -- 172.21.15.27:0/4216530890 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b0018070 con 0x7f48b8151f10 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f489ffff640 1 -- 172.21.15.27:0/4216530890 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b000a640 con 0x7f48b8151f10 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b8160850 con 0x7f48b8151f10 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f489ffff640 1 -- 172.21.15.27:0/4216530890 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b000aab0 con 0x7f48b8151f10 2024-03-06T07:35:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.813+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b80c9630 con 0x7f48b8151f10 2024-03-06T07:35:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.815+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b8067ff0 con 0x7f48b8151f10 2024-03-06T07:35:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.815+0000 7f489ffff640 1 -- 172.21.15.27:0/4216530890 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48b001b070 con 0x7f48b8151f10 2024-03-06T07:35:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.815+0000 7f489ffff640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f48a406b4d0 0x7f48a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.815+0000 7f489ffff640 1 -- 172.21.15.27:0/4216530890 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f48b00897a0 con 0x7f48b8151f10 2024-03-06T07:35:44.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.815+0000 7f48b77fe640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f48a406b4d0 0x7f48a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.816+0000 7f48b77fe640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f48a406b4d0 0x7f48a406d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f48a8036000 tx=0x7f48a803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:44.820+0000 7f489ffff640 1 -- 172.21.15.27:0/4216530890 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48b0058690 con 0x7f48b8151f10 2024-03-06T07:35:45.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.040+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48b80d3550 con 0x7f48b8151f10 2024-03-06T07:35:45.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.041+0000 7f489ffff640 1 -- 172.21.15.27:0/4216530890 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f48b0057f90 con 0x7f48b8151f10 2024-03-06T07:35:45.043 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:45.043 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:45.043 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f48a406b4d0 msgr2=0x7f48a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f48a406b4d0 0x7f48a406d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f48a8036000 tx=0x7f48a803b040 comp rx=0 tx=0).stop 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b8151f10 msgr2=0x7f48b80c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b8151f10 0x7f48b80c7cf0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f48b000c7a0 tx=0x7f48b000cc60 comp rx=0 tx=0).stop 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 shutdown_connections 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f48a406b4d0 0x7f48a406d990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48b8151f10 0x7f48b80c7cf0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48b815df00 0x7f48b80c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 --2- 172.21.15.27:0/4216530890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48b80b95a0 0x7f48b815fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 >> 172.21.15.27:0/4216530890 conn(0x7f48b8071960 msgr2=0x7f48b80d3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 shutdown_connections 2024-03-06T07:35:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:45.043+0000 7f48bdd56640 1 -- 172.21.15.27:0/4216530890 wait complete. 2024-03-06T07:35:45.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:45.259 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:46.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:46.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.364+0000 7f6d2315e640 1 Processor -- start 2024-03-06T07:35:46.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.364+0000 7f6d2315e640 1 -- start start 2024-03-06T07:35:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.365+0000 7f6d2315e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c15df00 0x7f6d1c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.365+0000 7f6d2315e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6d1c0b9a30 0x7f6d1c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.365+0000 7f6d2315e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1c0cea30 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.365+0000 7f6d2315e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1c0d2af0 con 0x7f6d1c0b9a30 2024-03-06T07:35:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.365+0000 7f6d2315e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0cf3c0 con 0x7f6d1c15df00 2024-03-06T07:35:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.365+0000 7f6d20efa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c15df00 0x7f6d1c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.365+0000 7f6d20efa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c15df00 0x7f6d1c0d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:50856/0 (socket says 172.21.15.27:50856) 2024-03-06T07:35:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.365+0000 7f6d20efa640 1 -- 172.21.15.27:0/4258786400 learned_addr learned my addr 172.21.15.27:0/4258786400 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.365+0000 7f6d216fb640 1 --2- 172.21.15.27:0/4258786400 >> v2:172.21.15.69:3300/0 conn(0x7f6d1c0b9a30 0x7f6d1c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.366+0000 7f6d20efa640 1 -- 172.21.15.27:0/4258786400 >> v2:172.21.15.69:3300/0 conn(0x7f6d1c0b9a30 msgr2=0x7f6d1c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:46.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.366+0000 7f6d20efa640 1 --2- 172.21.15.27:0/4258786400 >> v2:172.21.15.69:3300/0 conn(0x7f6d1c0b9a30 0x7f6d1c0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.366+0000 7f6d20efa640 1 -- 172.21.15.27:0/4258786400 >> v1:172.21.15.69:6789/0 conn(0x7f6d1c0b95a0 legacy=0x7f6d1c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.366+0000 7f6d20efa640 1 -- 172.21.15.27:0/4258786400 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c0cf0e0 con 0x7f6d1c15df00 2024-03-06T07:35:46.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d20efa640 1 --2- 172.21.15.27:0/4258786400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c15df00 0x7f6d1c0d25b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009870 tx=0x7f6d1002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11408ffb8657a48c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d1b7fe640 1 -- 172.21.15.27:0/4258786400 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d10034070 con 0x7f6d1c15df00 2024-03-06T07:35:46.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d1b7fe640 1 -- 172.21.15.27:0/4258786400 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d10030840 con 0x7f6d1c15df00 2024-03-06T07:35:46.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d1b7fe640 1 -- 172.21.15.27:0/4258786400 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d10030c10 con 0x7f6d1c15df00 2024-03-06T07:35:46.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d2315e640 1 -- 172.21.15.27:0/4258786400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c15df00 msgr2=0x7f6d1c0d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d2315e640 1 --2- 172.21.15.27:0/4258786400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c15df00 0x7f6d1c0d25b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009870 tx=0x7f6d1002b980 comp rx=0 tx=0).stop 2024-03-06T07:35:46.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d2315e640 1 -- 172.21.15.27:0/4258786400 shutdown_connections 2024-03-06T07:35:46.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d2315e640 1 --2- 172.21.15.27:0/4258786400 >> v2:172.21.15.69:3300/0 conn(0x7f6d1c0b9a30 0x7f6d1c0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d2315e640 1 --2- 172.21.15.27:0/4258786400 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c15df00 0x7f6d1c0d25b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d2315e640 1 -- 172.21.15.27:0/4258786400 >> 172.21.15.27:0/4258786400 conn(0x7f6d1c071960 msgr2=0x7f6d1c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:46.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d2315e640 1 -- 172.21.15.27:0/4258786400 shutdown_connections 2024-03-06T07:35:46.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.367+0000 7f6d2315e640 1 -- 172.21.15.27:0/4258786400 wait complete. 2024-03-06T07:35:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.368+0000 7f6d2315e640 1 Processor -- start 2024-03-06T07:35:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.369+0000 7f6d2315e640 1 -- start start 2024-03-06T07:35:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.369+0000 7f6d2315e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d1c0b95a0 0x7f6d1c160ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.369+0000 7f6d2315e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d1c15df00 0x7f6d1c14ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.369+0000 7f6d2315e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c0b3df0 0x7f6d1c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.369+0000 7f6d2315e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0cec70 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.369+0000 7f6d2315e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0ce750 con 0x7f6d1c15df00 2024-03-06T07:35:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.369+0000 7f6d2315e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0cedf0 con 0x7f6d1c0b3df0 2024-03-06T07:35:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d20efa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d1c0b95a0 0x7f6d1c160ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d20efa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d1c0b95a0 0x7f6d1c160ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36946/0 (socket says 172.21.15.27:36946) 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d20efa640 1 -- 172.21.15.27:0/3527431731 learned_addr learned my addr 172.21.15.27:0/3527431731 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d216fb640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c0b3df0 0x7f6d1c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d1bfff640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d1c15df00 0x7f6d1c14ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d20efa640 1 -- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c0b3df0 msgr2=0x7f6d1c0b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d20efa640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c0b3df0 0x7f6d1c0b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d20efa640 1 -- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d1c15df00 msgr2=0x7f6d1c14ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d20efa640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d1c15df00 0x7f6d1c14ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d20efa640 1 -- 172.21.15.27:0/3527431731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c15ed90 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d20efa640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d1c0b95a0 0x7f6d1c160ea0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f6d10000c00 tx=0x7f6d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d197fa640 1 -- 172.21.15.27:0/3527431731 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d10034040 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d197fa640 1 -- 172.21.15.27:0/3527431731 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d10039d90 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1c15f070 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d197fa640 1 -- 172.21.15.27:0/3527431731 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d10043630 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.370+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1c15f630 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.371+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1c067ff0 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.372+0000 7f6d197fa640 1 -- 172.21.15.27:0/3527431731 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d1004a060 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.373+0000 7f6d197fa640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d0006b500 0x7f6d0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.373+0000 7f6d197fa640 1 -- 172.21.15.27:0/3527431731 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6d100ac020 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.373+0000 7f6d1bfff640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d0006b500 0x7f6d0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.374+0000 7f6d1bfff640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d0006b500 0x7f6d0006d9c0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0ccc00 tx=0x7f6d08009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.377+0000 7f6d197fa640 1 -- 172.21.15.27:0/3527431731 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d1003e070 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.595+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d1c15caf0 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.596+0000 7f6d197fa640 1 -- 172.21.15.27:0/3527431731 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f6d1007a830 con 0x7f6d1c0b95a0 2024-03-06T07:35:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:46.598 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:46.599 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:46.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d0006b500 msgr2=0x7f6d0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d0006b500 0x7f6d0006d9c0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0ccc00 tx=0x7f6d08009040 comp rx=0 tx=0).stop 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d1c0b95a0 msgr2=0x7f6d1c160ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d1c0b95a0 0x7f6d1c160ea0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f6d10000c00 tx=0x7f6d100046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 shutdown_connections 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d0006b500 0x7f6d0006d9c0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d1c0b3df0 0x7f6d1c0b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d1c15df00 0x7f6d1c14ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 --2- 172.21.15.27:0/3527431731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d1c0b95a0 0x7f6d1c160ea0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 >> 172.21.15.27:0/3527431731 conn(0x7f6d1c071960 msgr2=0x7f6d1c0b8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 shutdown_connections 2024-03-06T07:35:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:46.598+0000 7f6d2315e640 1 -- 172.21.15.27:0/3527431731 wait complete. 2024-03-06T07:35:46.812 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:46.812 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:47.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:47.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.919+0000 7f8615f8c640 1 Processor -- start 2024-03-06T07:35:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.919+0000 7f8615f8c640 1 -- start start 2024-03-06T07:35:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f8615f8c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86100b95a0 0x7f86100b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f8615f8c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86100b9a10 0x7f86100b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f8615f8c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86100cea30 con 0x7f861015df00 2024-03-06T07:35:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f8615f8c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86100cf3c0 con 0x7f86100b9a10 2024-03-06T07:35:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f8615f8c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86100cf0e0 con 0x7f86100b95a0 2024-03-06T07:35:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f860f7fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f861015df00 0x7f86100d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42710/0 (socket says 172.21.15.27:42710) 2024-03-06T07:35:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f860f7fe640 1 -- 172.21.15.27:0/3954379069 learned_addr learned my addr 172.21.15.27:0/3954379069 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f860ffff640 1 --2- 172.21.15.27:0/3954379069 >> v2:172.21.15.27:3300/0 conn(0x7f86100b9a10 0x7f86100b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f860effd640 1 --2- 172.21.15.27:0/3954379069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86100b95a0 0x7f86100b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f860ffff640 1 -- 172.21.15.27:0/3954379069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86100b95a0 msgr2=0x7f86100b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f860ffff640 1 --2- 172.21.15.27:0/3954379069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86100b95a0 0x7f86100b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f860ffff640 1 -- 172.21.15.27:0/3954379069 >> v1:172.21.15.27:6789/0 conn(0x7f861015df00 legacy=0x7f86100d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.920+0000 7f860ffff640 1 -- 172.21.15.27:0/3954379069 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86100ce470 con 0x7f86100b9a10 2024-03-06T07:35:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.921+0000 7f860ffff640 1 --2- 172.21.15.27:0/3954379069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b9a10 0x7f86100b5c40 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f8608009d90 tx=0x7f860802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8cb5f4ffc934e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.921+0000 7f860e7fc640 1 -- 172.21.15.27:0/3954379069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8608039070 con 0x7f86100b9a10 2024-03-06T07:35:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.921+0000 7f860e7fc640 1 -- 172.21.15.27:0/3954379069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860802cc50 con 0x7f86100b9a10 2024-03-06T07:35:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.921+0000 7f860e7fc640 1 -- 172.21.15.27:0/3954379069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8608034440 con 0x7f86100b9a10 2024-03-06T07:35:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.922+0000 7f8615f8c640 1 -- 172.21.15.27:0/3954379069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b9a10 msgr2=0x7f86100b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.922+0000 7f8615f8c640 1 --2- 172.21.15.27:0/3954379069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b9a10 0x7f86100b5c40 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f8608009d90 tx=0x7f860802bea0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.922+0000 7f8615f8c640 1 -- 172.21.15.27:0/3954379069 shutdown_connections 2024-03-06T07:35:47.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.922+0000 7f8615f8c640 1 --2- 172.21.15.27:0/3954379069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b9a10 0x7f86100b5c40 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.922+0000 7f8615f8c640 1 --2- 172.21.15.27:0/3954379069 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f86100b95a0 0x7f86100b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.922+0000 7f8615f8c640 1 -- 172.21.15.27:0/3954379069 >> 172.21.15.27:0/3954379069 conn(0x7f8610071960 msgr2=0x7f8610071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:47.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.922+0000 7f8615f8c640 1 -- 172.21.15.27:0/3954379069 shutdown_connections 2024-03-06T07:35:47.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.922+0000 7f8615f8c640 1 -- 172.21.15.27:0/3954379069 wait complete. 2024-03-06T07:35:47.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.924+0000 7f8615f8c640 1 Processor -- start 2024-03-06T07:35:47.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.924+0000 7f8615f8c640 1 -- start start 2024-03-06T07:35:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.925+0000 7f8615f8c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b95a0 0x7f8610155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.925+0000 7f8615f8c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86100b9a10 0x7f86100cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.925+0000 7f8615f8c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f861015df00 0x7f86100b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.925+0000 7f8615f8c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86100cedf0 con 0x7f86100b95a0 2024-03-06T07:35:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f8615f8c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86100cec70 con 0x7f86100b9a10 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f8615f8c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86100d3220 con 0x7f861015df00 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b95a0 0x7f8610155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b95a0 0x7f8610155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36962/0 (socket says 172.21.15.27:36962) 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860f7fe640 1 -- 172.21.15.27:0/3030520509 learned_addr learned my addr 172.21.15.27:0/3030520509 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860effd640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86100b9a10 0x7f86100cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860ffff640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f861015df00 0x7f86100b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860f7fe640 1 -- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f861015df00 msgr2=0x7f86100b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860f7fe640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f861015df00 0x7f86100b4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860f7fe640 1 -- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86100b9a10 msgr2=0x7f86100cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860f7fe640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86100b9a10 0x7f86100cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860f7fe640 1 -- 172.21.15.27:0/3030520509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86100cdb90 con 0x7f86100b95a0 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860ffff640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f861015df00 0x7f86100b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:47.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f860f7fe640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b95a0 0x7f8610155130 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f860000c3f0 tx=0x7f860000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f8607fff640 1 -- 172.21.15.27:0/3030520509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600017070 con 0x7f86100b95a0 2024-03-06T07:35:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f8607fff640 1 -- 172.21.15.27:0/3030520509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8600005ce0 con 0x7f86100b95a0 2024-03-06T07:35:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86100c78e0 con 0x7f86100b95a0 2024-03-06T07:35:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f8607fff640 1 -- 172.21.15.27:0/3030520509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86000072f0 con 0x7f86100b95a0 2024-03-06T07:35:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.926+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86100cd5d0 con 0x7f86100b95a0 2024-03-06T07:35:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.927+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8610067ff0 con 0x7f86100b95a0 2024-03-06T07:35:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.928+0000 7f8607fff640 1 -- 172.21.15.27:0/3030520509 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f860001a070 con 0x7f86100b95a0 2024-03-06T07:35:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.928+0000 7f8607fff640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.928+0000 7f8607fff640 1 -- 172.21.15.27:0/3030520509 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8600089530 con 0x7f86100b95a0 2024-03-06T07:35:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.929+0000 7f860effd640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.929+0000 7f860effd640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f86101f1910 tx=0x7f85f8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:47.933+0000 7f8607fff640 1 -- 172.21.15.27:0/3030520509 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8600058420 con 0x7f86100b95a0 2024-03-06T07:35:48.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.150+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8610071f90 con 0x7f86100b95a0 2024-03-06T07:35:48.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.151+0000 7f8607fff640 1 -- 172.21.15.27:0/3030520509 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f8600057d20 con 0x7f86100b95a0 2024-03-06T07:35:48.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:48.153 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:48.153 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:48.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 msgr2=0x7f85f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f86101f1910 tx=0x7f85f8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:48.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b95a0 msgr2=0x7f8610155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b95a0 0x7f8610155130 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f860000c3f0 tx=0x7f860000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 shutdown_connections 2024-03-06T07:35:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f85f006b4d0 0x7f85f006d990 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f861015df00 0x7f86100b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f86100b9a10 0x7f86100cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 --2- 172.21.15.27:0/3030520509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86100b95a0 0x7f8610155130 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 >> 172.21.15.27:0/3030520509 conn(0x7f8610071960 msgr2=0x7f86100b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 shutdown_connections 2024-03-06T07:35:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:48.153+0000 7f8615f8c640 1 -- 172.21.15.27:0/3030520509 wait complete. 2024-03-06T07:35:48.367 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:48.367 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:49.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:49.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.474+0000 7f8fd0c84640 1 Processor -- start 2024-03-06T07:35:49.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.474+0000 7f8fd0c84640 1 -- start start 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fd0c84640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fcc071540 0x7f8fcc0b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fd0c84640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 0x7f8fcc071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fd0c84640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8fcc0b2b50 0x7f8fcc0d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fd0c84640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0d2f70 con 0x7f8fcc071540 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fd0c84640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0d4be0 con 0x7f8fcc0b2b50 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fd0c84640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc071e60 con 0x7f8fcc0d3400 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fca59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fcc071540 0x7f8fcc0b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fca59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fcc071540 0x7f8fcc0b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36970/0 (socket says 172.21.15.27:36970) 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fca59c640 1 -- 172.21.15.27:0/4135411499 learned_addr learned my addr 172.21.15.27:0/4135411499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fcad9d640 1 --2- 172.21.15.27:0/4135411499 >> v2:172.21.15.69:3300/0 conn(0x7f8fcc0b2b50 0x7f8fcc0d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fc9d9b640 1 --2- 172.21.15.27:0/4135411499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 0x7f8fcc071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fca59c640 1 -- 172.21.15.27:0/4135411499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 msgr2=0x7f8fcc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fca59c640 1 --2- 172.21.15.27:0/4135411499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 0x7f8fcc071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fca59c640 1 -- 172.21.15.27:0/4135411499 >> v2:172.21.15.69:3300/0 conn(0x7f8fcc0b2b50 msgr2=0x7f8fcc0d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fca59c640 1 --2- 172.21.15.27:0/4135411499 >> v2:172.21.15.69:3300/0 conn(0x7f8fcc0b2b50 0x7f8fcc0d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fca59c640 1 -- 172.21.15.27:0/4135411499 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0ce610 con 0x7f8fcc071540 2024-03-06T07:35:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.475+0000 7f8fc9d9b640 1 --2- 172.21.15.27:0/4135411499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 0x7f8fcc071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.476+0000 7f8fca59c640 1 --2- 172.21.15.27:0/4135411499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 0x7f8fcc0b30a0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc06c230 tx=0x7f8fbc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bdbe811155474fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.476+0000 7f8fc959a640 1 -- 172.21.15.27:0/4135411499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc035020 con 0x7f8fcc071540 2024-03-06T07:35:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.476+0000 7f8fc959a640 1 -- 172.21.15.27:0/4135411499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc02d040 con 0x7f8fcc071540 2024-03-06T07:35:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.476+0000 7f8fc959a640 1 -- 172.21.15.27:0/4135411499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc031a80 con 0x7f8fcc071540 2024-03-06T07:35:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.477+0000 7f8fd0c84640 1 -- 172.21.15.27:0/4135411499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 msgr2=0x7f8fcc0b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.477+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/4135411499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 0x7f8fcc0b30a0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc06c230 tx=0x7f8fbc02b900 comp rx=0 tx=0).stop 2024-03-06T07:35:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.477+0000 7f8fd0c84640 1 -- 172.21.15.27:0/4135411499 shutdown_connections 2024-03-06T07:35:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.477+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/4135411499 >> v2:172.21.15.69:3300/0 conn(0x7f8fcc0b2b50 0x7f8fcc0d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.477+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/4135411499 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 0x7f8fcc071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.477+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/4135411499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 0x7f8fcc0b30a0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.477+0000 7f8fd0c84640 1 -- 172.21.15.27:0/4135411499 >> 172.21.15.27:0/4135411499 conn(0x7f8fcc0d3ca0 msgr2=0x7f8fcc0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.477+0000 7f8fd0c84640 1 -- 172.21.15.27:0/4135411499 shutdown_connections 2024-03-06T07:35:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.477+0000 7f8fd0c84640 1 -- 172.21.15.27:0/4135411499 wait complete. 2024-03-06T07:35:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.478+0000 7f8fd0c84640 1 Processor -- start 2024-03-06T07:35:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.478+0000 7f8fd0c84640 1 -- start start 2024-03-06T07:35:49.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fd0c84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 0x7f8fcc1f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fd0c84640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fcc0b2b50 0x7f8fcc0aca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fd0c84640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 0x7f8fcc1f4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fd0c84640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0ce8f0 con 0x7f8fcc071540 2024-03-06T07:35:49.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fd0c84640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0d2c00 con 0x7f8fcc0b2b50 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fd0c84640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15e370 con 0x7f8fcc0d3400 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fca59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 0x7f8fcc1f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fca59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 0x7f8fcc1f3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36976/0 (socket says 172.21.15.27:36976) 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fca59c640 1 -- 172.21.15.27:0/1618512574 learned_addr learned my addr 172.21.15.27:0/1618512574 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fcad9d640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 0x7f8fcc1f4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fc9d9b640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fcc0b2b50 0x7f8fcc0aca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fca59c640 1 -- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 msgr2=0x7f8fcc1f4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fca59c640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 0x7f8fcc1f4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fca59c640 1 -- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fcc0b2b50 msgr2=0x7f8fcc0aca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fca59c640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fcc0b2b50 0x7f8fcc0aca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fca59c640 1 -- 172.21.15.27:0/1618512574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0acf40 con 0x7f8fcc071540 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fc9d9b640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fcc0b2b50 0x7f8fcc0aca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fca59c640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 0x7f8fcc1f3940 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc000c00 tx=0x7f8fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.479+0000 7f8fbaffd640 1 -- 172.21.15.27:0/1618512574 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc035040 con 0x7f8fcc071540 2024-03-06T07:35:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.480+0000 7f8fbaffd640 1 -- 172.21.15.27:0/1618512574 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc039bf0 con 0x7f8fcc071540 2024-03-06T07:35:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.480+0000 7f8fbaffd640 1 -- 172.21.15.27:0/1618512574 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc045480 con 0x7f8fcc071540 2024-03-06T07:35:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.480+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc1f48c0 con 0x7f8fcc071540 2024-03-06T07:35:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.480+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc1f4300 con 0x7f8fcc071540 2024-03-06T07:35:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.481+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc068700 con 0x7f8fcc071540 2024-03-06T07:35:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.481+0000 7f8fbaffd640 1 -- 172.21.15.27:0/1618512574 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fbc038020 con 0x7f8fcc071540 2024-03-06T07:35:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.482+0000 7f8fbaffd640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fac06b4d0 0x7f8fac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.482+0000 7f8fbaffd640 1 -- 172.21.15.27:0/1618512574 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8fbc0ac3a0 con 0x7f8fcc071540 2024-03-06T07:35:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.482+0000 7f8fc9d9b640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fac06b4d0 0x7f8fac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.482+0000 7f8fc9d9b640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fac06b4d0 0x7f8fac06d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0006fd0 tx=0x7f8fc0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.491+0000 7f8fbaffd640 1 -- 172.21.15.27:0/1618512574 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fbc07b200 con 0x7f8fcc071540 2024-03-06T07:35:49.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.711+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fcc0b63a0 con 0x7f8fcc071540 2024-03-06T07:35:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.712+0000 7f8fbaffd640 1 -- 172.21.15.27:0/1618512574 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f8fbc07ab00 con 0x7f8fcc071540 2024-03-06T07:35:49.713 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:49.713 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:49.713 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fac06b4d0 msgr2=0x7f8fac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fac06b4d0 0x7f8fac06d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0006fd0 tx=0x7f8fc0008040 comp rx=0 tx=0).stop 2024-03-06T07:35:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 msgr2=0x7f8fcc1f3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 0x7f8fcc1f3940 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc000c00 tx=0x7f8fbc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 shutdown_connections 2024-03-06T07:35:49.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8fac06b4d0 0x7f8fac06d990 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8fcc0d3400 0x7f8fcc1f4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fcc0b2b50 0x7f8fcc0aca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 --2- 172.21.15.27:0/1618512574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc071540 0x7f8fcc1f3940 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.713+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 >> 172.21.15.27:0/1618512574 conn(0x7f8fcc0d3ca0 msgr2=0x7f8fcc0b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.714+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 shutdown_connections 2024-03-06T07:35:49.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:49.714+0000 7f8fd0c84640 1 -- 172.21.15.27:0/1618512574 wait complete. 2024-03-06T07:35:49.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:49.928 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:50.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:50.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:50.928 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:51.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.033+0000 7f22fb4ed640 1 Processor -- start 2024-03-06T07:35:51.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.034+0000 7f22fb4ed640 1 -- start start 2024-03-06T07:35:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.034+0000 7f22fb4ed640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22f4071540 0x7f22f40b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.034+0000 7f22fb4ed640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f22f40d3400 0x7f22f4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.034+0000 7f22fb4ed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40b2b50 0x7f22f40d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.034+0000 7f22fb4ed640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f40d2f70 con 0x7f22f4071540 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.034+0000 7f22fb4ed640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f40d4be0 con 0x7f22f40d3400 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.034+0000 7f22fb4ed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f4071e60 con 0x7f22f40b2b50 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f9289640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22f4071540 0x7f22f40b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f9289640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22f4071540 0x7f22f40b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36980/0 (socket says 172.21.15.27:36980) 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f9289640 1 -- 172.21.15.27:0/4239796965 learned_addr learned my addr 172.21.15.27:0/4239796965 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f9a8a640 1 --2- 172.21.15.27:0/4239796965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40b2b50 0x7f22f40d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f8a88640 1 --2- 172.21.15.27:0/4239796965 >> v2:172.21.15.69:3300/0 conn(0x7f22f40d3400 0x7f22f4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f9289640 1 -- 172.21.15.27:0/4239796965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40b2b50 msgr2=0x7f22f40d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f9289640 1 --2- 172.21.15.27:0/4239796965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40b2b50 0x7f22f40d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f9289640 1 -- 172.21.15.27:0/4239796965 >> v2:172.21.15.69:3300/0 conn(0x7f22f40d3400 msgr2=0x7f22f4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f9289640 1 --2- 172.21.15.27:0/4239796965 >> v2:172.21.15.69:3300/0 conn(0x7f22f40d3400 0x7f22f4071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f9289640 1 -- 172.21.15.27:0/4239796965 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f40ce610 con 0x7f22f4071540 2024-03-06T07:35:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.035+0000 7f22f8a88640 1 --2- 172.21.15.27:0/4239796965 >> v2:172.21.15.69:3300/0 conn(0x7f22f40d3400 0x7f22f4071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.036+0000 7f22f9289640 1 --2- 172.21.15.27:0/4239796965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f4071540 0x7f22f40b30a0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f22f40b6860 tx=0x7f22e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e40d89e8d9972a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.036+0000 7f22ebfff640 1 -- 172.21.15.27:0/4239796965 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e4035020 con 0x7f22f4071540 2024-03-06T07:35:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.036+0000 7f22ebfff640 1 -- 172.21.15.27:0/4239796965 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e4031870 con 0x7f22f4071540 2024-03-06T07:35:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.036+0000 7f22ebfff640 1 -- 172.21.15.27:0/4239796965 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e4031c40 con 0x7f22f4071540 2024-03-06T07:35:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.036+0000 7f22fb4ed640 1 -- 172.21.15.27:0/4239796965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f4071540 msgr2=0x7f22f40b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.036+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/4239796965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f4071540 0x7f22f40b30a0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f22f40b6860 tx=0x7f22e402b910 comp rx=0 tx=0).stop 2024-03-06T07:35:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.037+0000 7f22fb4ed640 1 -- 172.21.15.27:0/4239796965 shutdown_connections 2024-03-06T07:35:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.037+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/4239796965 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40b2b50 0x7f22f40d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.037+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/4239796965 >> v2:172.21.15.69:3300/0 conn(0x7f22f40d3400 0x7f22f4071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.037+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/4239796965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f4071540 0x7f22f40b30a0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.037+0000 7f22fb4ed640 1 -- 172.21.15.27:0/4239796965 >> 172.21.15.27:0/4239796965 conn(0x7f22f40d3ca0 msgr2=0x7f22f40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.037+0000 7f22fb4ed640 1 -- 172.21.15.27:0/4239796965 shutdown_connections 2024-03-06T07:35:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.037+0000 7f22fb4ed640 1 -- 172.21.15.27:0/4239796965 wait complete. 2024-03-06T07:35:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.038+0000 7f22fb4ed640 1 Processor -- start 2024-03-06T07:35:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.038+0000 7f22fb4ed640 1 -- start start 2024-03-06T07:35:51.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.038+0000 7f22fb4ed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4071540 0x7f22f41e9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.038+0000 7f22fb4ed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f40b2b50 0x7f22f40b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22fb4ed640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d3400 0x7f22f40b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22fb4ed640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40ce8f0 con 0x7f22f40b2b50 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22fb4ed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f40d2c00 con 0x7f22f4071540 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22fb4ed640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f415e370 con 0x7f22f40d3400 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f8a88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f40b2b50 0x7f22f40b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f8a88640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f40b2b50 0x7f22f40b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36996/0 (socket says 172.21.15.27:36996) 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f8a88640 1 -- 172.21.15.27:0/796952459 learned_addr learned my addr 172.21.15.27:0/796952459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f9a8a640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d3400 0x7f22f40b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f9289640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4071540 0x7f22f41e9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f8a88640 1 -- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d3400 msgr2=0x7f22f40b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f8a88640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d3400 0x7f22f40b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f8a88640 1 -- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4071540 msgr2=0x7f22f41e9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f8a88640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4071540 0x7f22f41e9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f8a88640 1 -- 172.21.15.27:0/796952459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f41495c0 con 0x7f22f40b2b50 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f9a8a640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d3400 0x7f22f40b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f9289640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4071540 0x7f22f41e9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22f8a88640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f40b2b50 0x7f22f40b88b0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f22f000d7d0 tx=0x7f22f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22e9ffb640 1 -- 172.21.15.27:0/796952459 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f00096a0 con 0x7f22f40b2b50 2024-03-06T07:35:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22e9ffb640 1 -- 172.21.15.27:0/796952459 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f0019070 con 0x7f22f40b2b50 2024-03-06T07:35:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f41498a0 con 0x7f22f40b2b50 2024-03-06T07:35:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.039+0000 7f22e9ffb640 1 -- 172.21.15.27:0/796952459 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0009a80 con 0x7f22f40b2b50 2024-03-06T07:35:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.040+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f4149e60 con 0x7f22f40b2b50 2024-03-06T07:35:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.041+0000 7f22e9ffb640 1 -- 172.21.15.27:0/796952459 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22f0009c20 con 0x7f22f40b2b50 2024-03-06T07:35:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.041+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f4068700 con 0x7f22f40b2b50 2024-03-06T07:35:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.041+0000 7f22e9ffb640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f22d806b500 0x7f22d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.041+0000 7f22e9ffb640 1 -- 172.21.15.27:0/796952459 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f22f008a580 con 0x7f22f40b2b50 2024-03-06T07:35:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.042+0000 7f22f9289640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f22d806b500 0x7f22d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.042+0000 7f22f9289640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f22d806b500 0x7f22d806d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f22e4036000 tx=0x7f22e403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.046+0000 7f22e9ffb640 1 -- 172.21.15.27:0/796952459 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22f0059470 con 0x7f22f40b2b50 2024-03-06T07:35:51.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.265+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22f406b8e0 con 0x7f22f40b2b50 2024-03-06T07:35:51.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.266+0000 7f22e9ffb640 1 -- 172.21.15.27:0/796952459 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f22f0058d70 con 0x7f22f40b2b50 2024-03-06T07:35:51.268 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:51.268 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:51.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f22d806b500 msgr2=0x7f22d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f22d806b500 0x7f22d806d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f22e4036000 tx=0x7f22e403c040 comp rx=0 tx=0).stop 2024-03-06T07:35:51.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f40b2b50 msgr2=0x7f22f40b88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f40b2b50 0x7f22f40b88b0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f22f000d7d0 tx=0x7f22f000dc90 comp rx=0 tx=0).stop 2024-03-06T07:35:51.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 shutdown_connections 2024-03-06T07:35:51.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f22d806b500 0x7f22d806d9c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f22f40d3400 0x7f22f40b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22f40b2b50 0x7f22f40b88b0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 --2- 172.21.15.27:0/796952459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4071540 0x7f22f41e9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 >> 172.21.15.27:0/796952459 conn(0x7f22f40d3ca0 msgr2=0x7f22f415fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:51.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 shutdown_connections 2024-03-06T07:35:51.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:51.268+0000 7f22fb4ed640 1 -- 172.21.15.27:0/796952459 wait complete. 2024-03-06T07:35:51.482 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:44.385375+0000', 'modified': '2024-03-06T07:34:20.030594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9282, 'mds_1': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 1, 'incarnation': 114, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.69:6837/1115582329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6835', 'nonce': 1115582329}, {'type': 'v1', 'addr': '172.21.15.69:6837', 'nonce': 1115582329}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:33:13.569395+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9282': {'gid': 9282, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2910933327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2910933327}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2910933327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9282, 'qdb_cluster': [9282]} 2024-03-06T07:35:51.482 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9270:up:stopping 2024-03-06T07:35:52.483 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 123, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:31:44.385375+0000", "damaged": [], "data_pools": [ 21 ], "enabled": true, "epoch": 123, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9270": { "addr": "172.21.15.69:6837/1115582329", "addrs": { "addrvec": [ { "addr": "172.21.15.69:6835", "nonce": 1115582329, "type": "v2" }, { "addr": "172.21.15.69:6837", "nonce": 1115582329, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540701547738038271, "flags": 0, "gid": 9270, "incarnation": 114, "join_fscid": -1, "laggy_since": "2024-03-06T07:33:13.569395+0000", "name": "d", "rank": 1, "state": "up:stopping", "state_seq": 6 }, "gid_9282": { "addr": "172.21.15.27:6834/2910933327", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6832", "nonce": 2910933327, "type": "v2" }, { "addr": "172.21.15.27:6834", "nonce": 2910933327, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9282, "incarnation": 108, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 20, "modified": "2024-03-06T07:34:20.030594+0000", "qdb_cluster": [ 9282 ], "qdb_leader": 9282, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9282, "mds_1": 9270 } } } ], "standbys": [] } 2024-03-06T07:35:52.484 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:35:52.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:52.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:52.588 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:35:52.588 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:35:52.589 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:52.604 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:52.604 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:35:52.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.604 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:35:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-06T07:35:52.659 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:35:52.659 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:52.676 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:52.677 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:35:52.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.677 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:35:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:52.733 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:35:52.733 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:35:52.734 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:52.751 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:52.751 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:35:52.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.751 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:35:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:52.806 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:35:52.806 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:35:52.806 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:52.823 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:52.824 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:35:52.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.824 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:35:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:52.880 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:35:52.880 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:52.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.991+0000 7faeed9c7640 1 Processor -- start 2024-03-06T07:35:52.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.991+0000 7faeed9c7640 1 -- start start 2024-03-06T07:35:52.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.992+0000 7faeed9c7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faee815df00 0x7faee80d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.992+0000 7faeed9c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee80b95a0 0x7faee80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.992+0000 7faeed9c7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faee80b9a30 0x7faee80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.992+0000 7faeed9c7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee80cea30 con 0x7faee815df00 2024-03-06T07:35:52.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.992+0000 7faeed9c7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee80d2af0 con 0x7faee80b9a30 2024-03-06T07:35:52.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.992+0000 7faeed9c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee80cf3c0 con 0x7faee80b95a0 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.992+0000 7faee6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faee815df00 0x7faee80d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.992+0000 7faee6ffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faee815df00 0x7faee80d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37004/0 (socket says 172.21.15.27:37004) 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.992+0000 7faee6ffd640 1 -- 172.21.15.27:0/779086104 learned_addr learned my addr 172.21.15.27:0/779086104 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.993+0000 7faee77fe640 1 --2- 172.21.15.27:0/779086104 >> v2:172.21.15.69:3300/0 conn(0x7faee80b9a30 0x7faee80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.993+0000 7faee67fc640 1 --2- 172.21.15.27:0/779086104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee80b95a0 0x7faee80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.993+0000 7faee6ffd640 1 -- 172.21.15.27:0/779086104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee80b95a0 msgr2=0x7faee80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.993+0000 7faee6ffd640 1 --2- 172.21.15.27:0/779086104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee80b95a0 0x7faee80b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.993+0000 7faee6ffd640 1 -- 172.21.15.27:0/779086104 >> v2:172.21.15.69:3300/0 conn(0x7faee80b9a30 msgr2=0x7faee80b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.993+0000 7faee6ffd640 1 --2- 172.21.15.27:0/779086104 >> v2:172.21.15.69:3300/0 conn(0x7faee80b9a30 0x7faee80b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.993+0000 7faee6ffd640 1 -- 172.21.15.27:0/779086104 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee80cf0e0 con 0x7faee815df00 2024-03-06T07:35:52.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.994+0000 7faee6ffd640 1 --2- 172.21.15.27:0/779086104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee815df00 0x7faee80d25b0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7faee80d1160 tx=0x7faed802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c00fb9ef942896 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.994+0000 7faee5ffb640 1 -- 172.21.15.27:0/779086104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed802ca70 con 0x7faee815df00 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.994+0000 7faee5ffb640 1 -- 172.21.15.27:0/779086104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed802cc10 con 0x7faee815df00 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.994+0000 7faee5ffb640 1 -- 172.21.15.27:0/779086104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed80343f0 con 0x7faee815df00 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.995+0000 7faeed9c7640 1 -- 172.21.15.27:0/779086104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee815df00 msgr2=0x7faee80d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.995+0000 7faeed9c7640 1 --2- 172.21.15.27:0/779086104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee815df00 0x7faee80d25b0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7faee80d1160 tx=0x7faed802bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.995+0000 7faeed9c7640 1 -- 172.21.15.27:0/779086104 shutdown_connections 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.995+0000 7faeed9c7640 1 --2- 172.21.15.27:0/779086104 >> v2:172.21.15.69:3300/0 conn(0x7faee80b9a30 0x7faee80b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.995+0000 7faeed9c7640 1 --2- 172.21.15.27:0/779086104 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee80b95a0 0x7faee80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.995+0000 7faeed9c7640 1 --2- 172.21.15.27:0/779086104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee815df00 0x7faee80d25b0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.995+0000 7faeed9c7640 1 -- 172.21.15.27:0/779086104 >> 172.21.15.27:0/779086104 conn(0x7faee8071960 msgr2=0x7faee8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.995+0000 7faeed9c7640 1 -- 172.21.15.27:0/779086104 shutdown_connections 2024-03-06T07:35:52.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.995+0000 7faeed9c7640 1 -- 172.21.15.27:0/779086104 wait complete. 2024-03-06T07:35:53.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.996+0000 7faeed9c7640 1 Processor -- start 2024-03-06T07:35:53.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.996+0000 7faeed9c7640 1 -- start start 2024-03-06T07:35:53.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faeed9c7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee80b95a0 0x7faee80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faeed9c7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee815df00 0x7faee80b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faeed9c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faee80c9a40 0x7faee80c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faeed9c7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee80cec70 con 0x7faee80b95a0 2024-03-06T07:35:53.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faeed9c7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee80ce750 con 0x7faee80c9a40 2024-03-06T07:35:53.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faeed9c7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee80cedf0 con 0x7faee815df00 2024-03-06T07:35:53.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee80b95a0 0x7faee80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee80b95a0 0x7faee80c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37012/0 (socket says 172.21.15.27:37012) 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee6ffd640 1 -- 172.21.15.27:0/1880226381 learned_addr learned my addr 172.21.15.27:0/1880226381 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee67fc640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee815df00 0x7faee80b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee77fe640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faee80c9a40 0x7faee80c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee6ffd640 1 -- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee815df00 msgr2=0x7faee80b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee6ffd640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee815df00 0x7faee80b6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee6ffd640 1 -- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faee80c9a40 msgr2=0x7faee80c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee6ffd640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faee80c9a40 0x7faee80c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee6ffd640 1 -- 172.21.15.27:0/1880226381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee80b74c0 con 0x7faee80b95a0 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee67fc640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee815df00 0x7faee80b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee77fe640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faee80c9a40 0x7faee80c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.997+0000 7faee6ffd640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee80b95a0 0x7faee80c8170 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7faed8000f80 tx=0x7faed8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.998+0000 7faedffff640 1 -- 172.21.15.27:0/1880226381 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8043020 con 0x7faee80b95a0 2024-03-06T07:35:53.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.998+0000 7faedffff640 1 -- 172.21.15.27:0/1880226381 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed803e450 con 0x7faee80b95a0 2024-03-06T07:35:53.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.998+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee80b77a0 con 0x7faee80b95a0 2024-03-06T07:35:53.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.998+0000 7faedffff640 1 -- 172.21.15.27:0/1880226381 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8009510 con 0x7faee80b95a0 2024-03-06T07:35:53.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.998+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee80b7d60 con 0x7faee80b95a0 2024-03-06T07:35:53.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.999+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee8067ff0 con 0x7faee80b95a0 2024-03-06T07:35:53.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:52.999+0000 7faedffff640 1 -- 172.21.15.27:0/1880226381 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faed8046020 con 0x7faee80b95a0 2024-03-06T07:35:53.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.000+0000 7faedffff640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faec806b4d0 0x7faec806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.000+0000 7faedffff640 1 -- 172.21.15.27:0/1880226381 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faed80ac9c0 con 0x7faee80b95a0 2024-03-06T07:35:53.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.000+0000 7faee67fc640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faec806b4d0 0x7faec806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.001+0000 7faee67fc640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faec806b4d0 0x7faec806d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7faed0006fd0 tx=0x7faed0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.004+0000 7faedffff640 1 -- 172.21.15.27:0/1880226381 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faed80aee20 con 0x7faee80b95a0 2024-03-06T07:35:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.242+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faee81625a0 con 0x7faee80b95a0 2024-03-06T07:35:53.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.242+0000 7faedffff640 1 -- 172.21.15.27:0/1880226381 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7faed804d030 con 0x7faee80b95a0 2024-03-06T07:35:53.243 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:53.244 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:53.244 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:53.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faec806b4d0 msgr2=0x7faec806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faec806b4d0 0x7faec806d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7faed0006fd0 tx=0x7faed0008040 comp rx=0 tx=0).stop 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee80b95a0 msgr2=0x7faee80c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee80b95a0 0x7faee80c8170 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7faed8000f80 tx=0x7faed8004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 shutdown_connections 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7faec806b4d0 0x7faec806d990 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faee80c9a40 0x7faee80c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faee815df00 0x7faee80b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 --2- 172.21.15.27:0/1880226381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee80b95a0 0x7faee80c8170 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 >> 172.21.15.27:0/1880226381 conn(0x7faee8071960 msgr2=0x7faee815c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 shutdown_connections 2024-03-06T07:35:53.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.244+0000 7faeed9c7640 1 -- 172.21.15.27:0/1880226381 wait complete. 2024-03-06T07:35:53.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:53.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.362+0000 7f61f215e640 1 Processor -- start 2024-03-06T07:35:53.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.362+0000 7f61f215e640 1 -- start start 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61f215e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f61ec157b80 0x7f61ec15b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61f215e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec0cbab0 0x7f61ec070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61f215e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ec06c7b0 con 0x7f61ec157f60 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61f215e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61ec06ca90 con 0x7f61ec157b80 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61f215e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec06cd70 con 0x7f61ec0cbab0 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61eaffd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f61ec157f60 0x7f61ec15b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42718/0 (socket says 172.21.15.27:42718) 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61eaffd640 1 -- 172.21.15.27:0/3458436590 learned_addr learned my addr 172.21.15.27:0/3458436590 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ebfff640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec0cbab0 0x7f61ec070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61eb7fe640 1 --2- 172.21.15.27:0/3458436590 >> v2:172.21.15.69:3300/0 conn(0x7f61ec157b80 0x7f61ec15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225034198 0 0) 0x7f61ec06c7b0 con 0x7f61ec157f60 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61d4003520 con 0x7f61ec157f60 2024-03-06T07:35:53.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3554397441 0 0) 0x7f61d4003520 con 0x7f61ec157f60 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec0cbab0 msgr2=0x7f61ec070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec0cbab0 0x7f61ec070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 >> v2:172.21.15.69:3300/0 conn(0x7f61ec157b80 msgr2=0x7f61ec15b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 --2- 172.21.15.27:0/3458436590 >> v2:172.21.15.69:3300/0 conn(0x7f61ec157b80 0x7f61ec15b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec06d050 con 0x7f61ec157f60 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f61e0001670 con 0x7f61ec157f60 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 >> v1:172.21.15.27:6789/0 conn(0x7f61ec157f60 legacy=0x7f61ec15b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61d40039b0 0x7f61d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec0cbab0 0x7f61d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec06ca90 con 0x7f61d4004800 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec06cd70 con 0x7f61d40039b0 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.363+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d400cad0 con 0x7f61ec0cbab0 2024-03-06T07:35:53.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.364+0000 7f61eaffd640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.364+0000 7f61eb7fe640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61d40039b0 0x7f61d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.364+0000 7f61ebfff640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec0cbab0 0x7f61d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.364+0000 7f61eaffd640 1 -- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec0cbab0 msgr2=0x7f61d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:53.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.364+0000 7f61eaffd640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec0cbab0 0x7f61d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.364+0000 7f61eaffd640 1 -- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61d40039b0 msgr2=0x7f61d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:53.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.364+0000 7f61eaffd640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61d40039b0 0x7f61d4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.364+0000 7f61eaffd640 1 -- 172.21.15.27:0/3458436590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d4003520 con 0x7f61d4004800 2024-03-06T07:35:53.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.365+0000 7f61eaffd640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61d4004c80 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f61e000c210 tx=0x7f61d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d2f2c64336ca60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.365+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e0007d30 con 0x7f61d4004800 2024-03-06T07:35:53.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.365+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e0030070 con 0x7f61d4004800 2024-03-06T07:35:53.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.365+0000 7f61ea7fc640 1 -- 172.21.15.27:0/3458436590 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e0005d30 con 0x7f61d4004800 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.366+0000 7f61f215e640 1 -- 172.21.15.27:0/3458436590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 msgr2=0x7f61d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.366+0000 7f61f215e640 1 --2- 172.21.15.27:0/3458436590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61d4004c80 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f61e000c210 tx=0x7f61d4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.366+0000 7f61ebfff640 1 -- 172.21.15.27:0/3458436590 reap_dead start 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.366+0000 7f61f215e640 1 -- 172.21.15.27:0/3458436590 shutdown_connections 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.366+0000 7f61f215e640 1 -- 172.21.15.27:0/3458436590 >> 172.21.15.27:0/3458436590 conn(0x7f61ec066d00 msgr2=0x7f61ec0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.366+0000 7f61f215e640 1 -- 172.21.15.27:0/3458436590 shutdown_connections 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.366+0000 7f61f215e640 1 -- 172.21.15.27:0/3458436590 wait complete. 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.367+0000 7f61f215e640 1 Processor -- start 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.367+0000 7f61f215e640 1 -- start start 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61f215e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61ec141460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61f215e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec156c40 0x7f61ec151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61f215e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61ec1570c0 0x7f61ec151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61f215e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec0cf890 con 0x7f61d4004800 2024-03-06T07:35:53.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61f215e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec0cc260 con 0x7f61ec1570c0 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61f215e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec0cfa10 con 0x7f61ec156c40 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61ec141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61eb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61ec141460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37018/0 (socket says 172.21.15.27:37018) 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61eb7fe640 1 -- 172.21.15.27:0/1264448626 learned_addr learned my addr 172.21.15.27:0/1264448626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61ebfff640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61ec1570c0 0x7f61ec151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.368+0000 7f61eaffd640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec156c40 0x7f61ec151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61eb7fe640 1 -- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec156c40 msgr2=0x7f61ec151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61eb7fe640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec156c40 0x7f61ec151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61eb7fe640 1 -- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61ec1570c0 msgr2=0x7f61ec151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61eb7fe640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61ec1570c0 0x7f61ec151860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61eb7fe640 1 -- 172.21.15.27:0/1264448626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec142170 con 0x7f61d4004800 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61ebfff640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61ec1570c0 0x7f61ec151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61eb7fe640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61ec141460 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f61dc0119c0 tx=0x7f61dc011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61d3fff640 1 -- 172.21.15.27:0/1264448626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc00faf0 con 0x7f61d4004800 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61d3fff640 1 -- 172.21.15.27:0/1264448626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc00fc90 con 0x7f61d4004800 2024-03-06T07:35:53.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ec142450 con 0x7f61d4004800 2024-03-06T07:35:53.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61d3fff640 1 -- 172.21.15.27:0/1264448626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc014440 con 0x7f61d4004800 2024-03-06T07:35:53.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.369+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec142a10 con 0x7f61d4004800 2024-03-06T07:35:53.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.370+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ec067ff0 con 0x7f61d4004800 2024-03-06T07:35:53.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.371+0000 7f61d3fff640 1 -- 172.21.15.27:0/1264448626 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61dc012040 con 0x7f61d4004800 2024-03-06T07:35:53.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.371+0000 7f61d3fff640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61d806b500 0x7f61d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.372+0000 7f61d3fff640 1 -- 172.21.15.27:0/1264448626 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f61dc08a8a0 con 0x7f61d4004800 2024-03-06T07:35:53.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.372+0000 7f61eaffd640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61d806b500 0x7f61d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.372+0000 7f61eaffd640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61d806b500 0x7f61d806d9c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f61e0034000 tx=0x7f61e0007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.376+0000 7f61d3fff640 1 -- 172.21.15.27:0/1264448626 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61dc019030 con 0x7f61d4004800 2024-03-06T07:35:53.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.623+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61ec0d07a0 con 0x7f61d4004800 2024-03-06T07:35:53.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.624+0000 7f61d3fff640 1 -- 172.21.15.27:0/1264448626 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f61dc059700 con 0x7f61d4004800 2024-03-06T07:35:53.627 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:53.627 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":149,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:32:31.280155+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61d806b500 msgr2=0x7f61d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61d806b500 0x7f61d806d9c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f61e0034000 tx=0x7f61e0007000 comp rx=0 tx=0).stop 2024-03-06T07:35:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 msgr2=0x7f61ec141460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61ec141460 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f61dc0119c0 tx=0x7f61dc011e80 comp rx=0 tx=0).stop 2024-03-06T07:35:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 shutdown_connections 2024-03-06T07:35:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f61d806b500 0x7f61d806d9c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61ec1570c0 0x7f61ec151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f61ec156c40 0x7f61ec151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 --2- 172.21.15.27:0/1264448626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61d4004800 0x7f61ec141460 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.627+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 >> 172.21.15.27:0/1264448626 conn(0x7f61ec066d00 msgr2=0x7f61ec158400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.628+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 shutdown_connections 2024-03-06T07:35:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.628+0000 7f61f215e640 1 -- 172.21.15.27:0/1264448626 wait complete. 2024-03-06T07:35:53.641 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:22:36.043110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:42.692383+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:43.367948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:53.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:53.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.750+0000 7f75d37a0640 1 Processor -- start 2024-03-06T07:35:53.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.750+0000 7f75d37a0640 1 -- start start 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d37a0640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75cc15ae50 0x7f75cc0708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d37a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 0x7f75cc071160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d37a0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc070e20 con 0x7f75cc15b690 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d37a0640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75cc0b2e30 con 0x7f75cc15ae50 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d37a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc06dfe0 con 0x7f75cc15b230 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d153c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75cc15ae50 0x7f75cc0708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d153c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75cc15ae50 0x7f75cc0708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41810/0 (socket says 172.21.15.27:41810) 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d153c640 1 -- 172.21.15.27:0/2714175910 learned_addr learned my addr 172.21.15.27:0/2714175910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d0d3b640 1 --2- 172.21.15.27:0/2714175910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 0x7f75cc071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d153c640 1 -- 172.21.15.27:0/2714175910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 msgr2=0x7f75cc071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d153c640 1 --2- 172.21.15.27:0/2714175910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 0x7f75cc071160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d153c640 1 -- 172.21.15.27:0/2714175910 >> v1:172.21.15.69:6789/0 conn(0x7f75cc15b690 legacy=0x7f75cc150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d153c640 1 -- 172.21.15.27:0/2714175910 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc0b30b0 con 0x7f75cc15ae50 2024-03-06T07:35:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.751+0000 7f75d0d3b640 1 --2- 172.21.15.27:0/2714175910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 0x7f75cc071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:53.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.752+0000 7f75d153c640 1 --2- 172.21.15.27:0/2714175910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15ae50 0x7f75cc0708e0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009870 tx=0x7f75c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c8e33c2e5cbdaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.752+0000 7f75c3fff640 1 -- 172.21.15.27:0/2714175910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c8035020 con 0x7f75cc15ae50 2024-03-06T07:35:53.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.752+0000 7f75c3fff640 1 -- 172.21.15.27:0/2714175910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c8031640 con 0x7f75cc15ae50 2024-03-06T07:35:53.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.752+0000 7f75c3fff640 1 -- 172.21.15.27:0/2714175910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c802d040 con 0x7f75cc15ae50 2024-03-06T07:35:53.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.752+0000 7f75d37a0640 1 -- 172.21.15.27:0/2714175910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15ae50 msgr2=0x7f75cc0708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.752+0000 7f75d37a0640 1 --2- 172.21.15.27:0/2714175910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15ae50 0x7f75cc0708e0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009870 tx=0x7f75c802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.753+0000 7f75d37a0640 1 -- 172.21.15.27:0/2714175910 shutdown_connections 2024-03-06T07:35:53.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.753+0000 7f75d37a0640 1 --2- 172.21.15.27:0/2714175910 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 0x7f75cc071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.753+0000 7f75d37a0640 1 --2- 172.21.15.27:0/2714175910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15ae50 0x7f75cc0708e0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.753+0000 7f75d37a0640 1 -- 172.21.15.27:0/2714175910 >> 172.21.15.27:0/2714175910 conn(0x7f75cc066d00 msgr2=0x7f75cc067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.753+0000 7f75d37a0640 1 -- 172.21.15.27:0/2714175910 shutdown_connections 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.753+0000 7f75d37a0640 1 -- 172.21.15.27:0/2714175910 wait complete. 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.754+0000 7f75d37a0640 1 Processor -- start 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.754+0000 7f75d37a0640 1 -- start start 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.754+0000 7f75d37a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc15ae50 0x7f75cc143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.754+0000 7f75d37a0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 0x7f75cc159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.754+0000 7f75d37a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15b690 0x7f75cc1598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.754+0000 7f75d37a0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0b3950 con 0x7f75cc15b690 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d37a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0b3670 con 0x7f75cc15ae50 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d37a0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0b4150 con 0x7f75cc15b230 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d1d3d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15b690 0x7f75cc1598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d1d3d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15b690 0x7f75cc1598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41814/0 (socket says 172.21.15.27:41814) 2024-03-06T07:35:53.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d1d3d640 1 -- 172.21.15.27:0/4036973671 learned_addr learned my addr 172.21.15.27:0/4036973671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d153c640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc15ae50 0x7f75cc143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d0d3b640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 0x7f75cc159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d1d3d640 1 -- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 msgr2=0x7f75cc159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d1d3d640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 0x7f75cc159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d1d3d640 1 -- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc15ae50 msgr2=0x7f75cc143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d1d3d640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc15ae50 0x7f75cc143260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d1d3d640 1 -- 172.21.15.27:0/4036973671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc1314d0 con 0x7f75cc15b690 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.755+0000 7f75d1d3d640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15b690 0x7f75cc1598c0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f75c400c730 tx=0x7f75c400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.756+0000 7f75c1ffb640 1 -- 172.21.15.27:0/4036973671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4003b70 con 0x7f75cc15b690 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.756+0000 7f75c1ffb640 1 -- 172.21.15.27:0/4036973671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c4003d10 con 0x7f75cc15b690 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.756+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc1317b0 con 0x7f75cc15b690 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.756+0000 7f75c1ffb640 1 -- 172.21.15.27:0/4036973671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c400aa90 con 0x7f75cc15b690 2024-03-06T07:35:53.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.756+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc1437a0 con 0x7f75cc15b690 2024-03-06T07:35:53.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.757+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc0733c0 con 0x7f75cc15b690 2024-03-06T07:35:53.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.757+0000 7f75c1ffb640 1 -- 172.21.15.27:0/4036973671 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75c400ac30 con 0x7f75cc15b690 2024-03-06T07:35:53.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.758+0000 7f75c1ffb640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f75b006b4d0 0x7f75b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.758+0000 7f75c1ffb640 1 -- 172.21.15.27:0/4036973671 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f75c408a0d0 con 0x7f75cc15b690 2024-03-06T07:35:53.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.758+0000 7f75d153c640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f75b006b4d0 0x7f75b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.759+0000 7f75d153c640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f75b006b4d0 0x7f75b006d990 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f75c8036000 tx=0x7f75c803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.762+0000 7f75c1ffb640 1 -- 172.21.15.27:0/4036973671 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75c4058f30 con 0x7f75cc15b690 2024-03-06T07:35:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.945+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75cc0cce50 con 0x7f75cc15b690 2024-03-06T07:35:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.945+0000 7f75c1ffb640 1 -- 172.21.15.27:0/4036973671 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75c4058830 con 0x7f75cc15b690 2024-03-06T07:35:53.946 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f75b006b4d0 msgr2=0x7f75b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f75b006b4d0 0x7f75b006d990 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f75c8036000 tx=0x7f75c803c040 comp rx=0 tx=0).stop 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15b690 msgr2=0x7f75cc1598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15b690 0x7f75cc1598c0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f75c400c730 tx=0x7f75c400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 shutdown_connections 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f75b006b4d0 0x7f75b006d990 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75cc15b690 0x7f75cc1598c0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f75cc15b230 0x7f75cc159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 --2- 172.21.15.27:0/4036973671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc15ae50 0x7f75cc143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 >> 172.21.15.27:0/4036973671 conn(0x7f75cc066d00 msgr2=0x7f75cc0ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 shutdown_connections 2024-03-06T07:35:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:53.947+0000 7f75d37a0640 1 -- 172.21.15.27:0/4036973671 wait complete. 2024-03-06T07:35:53.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:35:54.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.068+0000 7f1f2c1f7640 1 Processor -- start 2024-03-06T07:35:54.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.068+0000 7f1f2c1f7640 1 -- start start 2024-03-06T07:35:54.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.068+0000 7f1f2c1f7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f24071540 0x7f1f24071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.068+0000 7f1f2c1f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240b2d30 0x7f1f240ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f2c1f7640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f24071e60 con 0x7f1f240cecb0 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f2c1f7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f240d3230 con 0x7f1f24071540 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f2c1f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240d4960 con 0x7f1f240b2d30 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f29f93640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f24071540 0x7f1f24071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f29f93640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f24071540 0x7f1f24071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41824/0 (socket says 172.21.15.27:41824) 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f29f93640 1 -- 172.21.15.27:0/3524092373 learned_addr learned my addr 172.21.15.27:0/3524092373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f29792640 1 --2- 172.21.15.27:0/3524092373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240b2d30 0x7f1f240ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f29f93640 1 -- 172.21.15.27:0/3524092373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240b2d30 msgr2=0x7f1f240ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f29f93640 1 --2- 172.21.15.27:0/3524092373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240b2d30 0x7f1f240ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f29f93640 1 -- 172.21.15.27:0/3524092373 >> v1:172.21.15.69:6789/0 conn(0x7f1f240cecb0 legacy=0x7f1f240b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.069+0000 7f1f29f93640 1 -- 172.21.15.27:0/3524092373 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240d36c0 con 0x7f1f24071540 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f29f93640 1 --2- 172.21.15.27:0/3524092373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 0x7f1f24071920 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f1f18009870 tx=0x7f1f1802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801fa4a13a664881 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f28f91640 1 -- 172.21.15.27:0/3524092373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18034070 con 0x7f1f24071540 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f28f91640 1 -- 172.21.15.27:0/3524092373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f18030840 con 0x7f1f24071540 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f28f91640 1 -- 172.21.15.27:0/3524092373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18030c10 con 0x7f1f24071540 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3524092373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 msgr2=0x7f1f24071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f2c1f7640 1 --2- 172.21.15.27:0/3524092373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 0x7f1f24071920 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f1f18009870 tx=0x7f1f1802b980 comp rx=0 tx=0).stop 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3524092373 shutdown_connections 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f2c1f7640 1 --2- 172.21.15.27:0/3524092373 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240b2d30 0x7f1f240ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f2c1f7640 1 --2- 172.21.15.27:0/3524092373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 0x7f1f24071920 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.070+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3524092373 >> 172.21.15.27:0/3524092373 conn(0x7f1f240d3d60 msgr2=0x7f1f240d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.071+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3524092373 shutdown_connections 2024-03-06T07:35:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.071+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3524092373 wait complete. 2024-03-06T07:35:54.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.072+0000 7f1f2c1f7640 1 Processor -- start 2024-03-06T07:35:54.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.072+0000 7f1f2c1f7640 1 -- start start 2024-03-06T07:35:54.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.072+0000 7f1f2c1f7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 0x7f1f2415a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.072+0000 7f1f2c1f7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f240b2d30 0x7f1f240aa760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.072+0000 7f1f2c1f7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240cecb0 0x7f1f241f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.072+0000 7f1f2c1f7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240d2be0 con 0x7f1f24071540 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.072+0000 7f1f2c1f7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240d2900 con 0x7f1f240b2d30 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.072+0000 7f1f2c1f7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240d2ec0 con 0x7f1f240cecb0 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29f93640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 0x7f1f2415a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29f93640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 0x7f1f2415a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41830/0 (socket says 172.21.15.27:41830) 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29f93640 1 -- 172.21.15.27:0/3985813531 learned_addr learned my addr 172.21.15.27:0/3985813531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29792640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f240b2d30 0x7f1f240aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f2a794640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240cecb0 0x7f1f241f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29f93640 1 -- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240cecb0 msgr2=0x7f1f241f3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29f93640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240cecb0 0x7f1f241f3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29f93640 1 -- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f240b2d30 msgr2=0x7f1f240aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29f93640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f240b2d30 0x7f1f240aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29f93640 1 -- 172.21.15.27:0/3985813531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240c85e0 con 0x7f1f24071540 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29792640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f240b2d30 0x7f1f240aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f29f93640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 0x7f1f2415a1f0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f1f1802b9c0 tx=0x7f1f180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f167fc640 1 -- 172.21.15.27:0/3985813531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18034040 con 0x7f1f24071540 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f167fc640 1 -- 172.21.15.27:0/3985813531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f18039d90 con 0x7f1f24071540 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2414d850 con 0x7f1f24071540 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f167fc640 1 -- 172.21.15.27:0/3985813531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18043610 con 0x7f1f24071540 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.073+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2414daf0 con 0x7f1f24071540 2024-03-06T07:35:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.074+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f240ce120 con 0x7f1f24071540 2024-03-06T07:35:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.075+0000 7f1f167fc640 1 -- 172.21.15.27:0/3985813531 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f1804a060 con 0x7f1f24071540 2024-03-06T07:35:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.075+0000 7f1f167fc640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0806b4d0 0x7f1f0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.075+0000 7f1f167fc640 1 -- 172.21.15.27:0/3985813531 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1f180ac250 con 0x7f1f24071540 2024-03-06T07:35:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.075+0000 7f1f29792640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0806b4d0 0x7f1f0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.076+0000 7f1f29792640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0806b4d0 0x7f1f0806d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1f24153e30 tx=0x7f1f20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.085+0000 7f1f167fc640 1 -- 172.21.15.27:0/3985813531 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f1803e070 con 0x7f1f24071540 2024-03-06T07:35:54.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.266+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1f2404f410 con 0x7f1f24071540 2024-03-06T07:35:54.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.267+0000 7f1f167fc640 1 -- 172.21.15.27:0/3985813531 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1f1802d330 con 0x7f1f24071540 2024-03-06T07:35:54.268 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-06T07:35:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0806b4d0 msgr2=0x7f1f0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0806b4d0 0x7f1f0806d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1f24153e30 tx=0x7f1f20008040 comp rx=0 tx=0).stop 2024-03-06T07:35:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 msgr2=0x7f1f2415a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 0x7f1f2415a1f0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f1f1802b9c0 tx=0x7f1f180046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 shutdown_connections 2024-03-06T07:35:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1f0806b4d0 0x7f1f0806d990 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f240cecb0 0x7f1f241f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f240b2d30 0x7f1f240aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 --2- 172.21.15.27:0/3985813531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f24071540 0x7f1f2415a1f0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 >> 172.21.15.27:0/3985813531 conn(0x7f1f240d3d60 msgr2=0x7f1f2415b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.268+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 shutdown_connections 2024-03-06T07:35:54.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.269+0000 7f1f2c1f7640 1 -- 172.21.15.27:0/3985813531 wait complete. 2024-03-06T07:35:54.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:35:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.390+0000 7f014a04e640 1 Processor -- start 2024-03-06T07:35:54.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.390+0000 7f014a04e640 1 -- start start 2024-03-06T07:35:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.391+0000 7f014a04e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01440b7420 0x7f01440d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.391+0000 7f014a04e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01440d3740 con 0x7f014415dd90 2024-03-06T07:35:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.391+0000 7f014a04e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01440d3120 con 0x7f01440ceea0 2024-03-06T07:35:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.391+0000 7f014a04e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01440d3400 con 0x7f01440b7420 2024-03-06T07:35:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.391+0000 7f01437fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f014415dd90 0x7f01440d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51850/0 (socket says 172.21.15.27:51850) 2024-03-06T07:35:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.391+0000 7f01437fe640 1 -- 172.21.15.27:0/442426806 learned_addr learned my addr 172.21.15.27:0/442426806 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.391+0000 7f0143fff640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01440b7420 0x7f01440d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.391+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055036707 0 0) 0x7f01440d3740 con 0x7f014415dd90 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0130003520 con 0x7f014415dd90 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195601359 0 0) 0x7f01440d3120 con 0x7f01440ceea0 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01440d3740 con 0x7f01440ceea0 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1349010275 0 0) 0x7f0130003520 con 0x7f014415dd90 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01440b7420 msgr2=0x7f01440d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01440b7420 0x7f01440d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 >> v1:172.21.15.69:6789/0 conn(0x7f01440ceea0 legacy=0x7f01440b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01440d39e0 con 0x7f014415dd90 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f0134001670 con 0x7f014415dd90 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f0143fff640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01440b7420 0x7f01440d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 >> v1:172.21.15.27:6789/0 conn(0x7f014415dd90 legacy=0x7f01440d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0130003b70 0x7f0130003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0130004800 0x7f0130004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01300051c0 0x7f013000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01440d3400 con 0x7f01300051c0 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013000bf00 con 0x7f0130004800 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013000c080 con 0x7f0130003b70 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f0143fff640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01300051c0 0x7f013000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f0142ffd640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0130004800 0x7f0130004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f01437fe640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0130003b70 0x7f0130003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f0143fff640 1 -- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0130003b70 msgr2=0x7f0130003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f0143fff640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0130003b70 0x7f0130003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f0143fff640 1 -- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0130004800 msgr2=0x7f0130004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f0143fff640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0130004800 0x7f0130004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.392+0000 7f0143fff640 1 -- 172.21.15.27:0/442426806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0130003520 con 0x7f01300051c0 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.393+0000 7f0143fff640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01300051c0 0x7f013000b9c0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f013c00e320 tx=0x7f0130005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6b36997fefb837 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.393+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c009dc0 con 0x7f01300051c0 2024-03-06T07:35:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.393+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013c02cd30 con 0x7f01300051c0 2024-03-06T07:35:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.393+0000 7f01427fc640 1 -- 172.21.15.27:0/442426806 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c003b90 con 0x7f01300051c0 2024-03-06T07:35:54.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.394+0000 7f014a04e640 1 -- 172.21.15.27:0/442426806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01300051c0 msgr2=0x7f013000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.394+0000 7f014a04e640 1 --2- 172.21.15.27:0/442426806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01300051c0 0x7f013000b9c0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f013c00e320 tx=0x7f0130005830 comp rx=0 tx=0).stop 2024-03-06T07:35:54.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.394+0000 7f0143fff640 1 -- 172.21.15.27:0/442426806 reap_dead start 2024-03-06T07:35:54.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.394+0000 7f014a04e640 1 -- 172.21.15.27:0/442426806 shutdown_connections 2024-03-06T07:35:54.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.394+0000 7f014a04e640 1 -- 172.21.15.27:0/442426806 >> 172.21.15.27:0/442426806 conn(0x7f0144071960 msgr2=0x7f0144071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.394+0000 7f014a04e640 1 -- 172.21.15.27:0/442426806 shutdown_connections 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.394+0000 7f014a04e640 1 -- 172.21.15.27:0/442426806 wait complete. 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.395+0000 7f014a04e640 1 Processor -- start 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f014a04e640 1 -- start start 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f014a04e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415c8e0 0x7f01440cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f014a04e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01441f42d0 0x7f014415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f014a04e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440aab40 0x7f01440aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f014a04e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01440d48f0 con 0x7f01441f42d0 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f014a04e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014415e170 con 0x7f01440aab40 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f014a04e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01440d4a70 con 0x7f014415c8e0 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f0142ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01441f42d0 0x7f014415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f0142ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01441f42d0 0x7f014415ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41840/0 (socket says 172.21.15.27:41840) 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f0142ffd640 1 -- 172.21.15.27:0/3735110243 learned_addr learned my addr 172.21.15.27:0/3735110243 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f01437fe640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415c8e0 0x7f01440cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.396+0000 7f0143fff640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440aab40 0x7f01440aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f0142ffd640 1 -- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415c8e0 msgr2=0x7f01440cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f0142ffd640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415c8e0 0x7f01440cdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f0142ffd640 1 -- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440aab40 msgr2=0x7f01440aafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f0142ffd640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440aab40 0x7f01440aafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f0142ffd640 1 -- 172.21.15.27:0/3735110243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01441f4730 con 0x7f01441f42d0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f0142ffd640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01441f42d0 0x7f014415ccc0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f012c00f770 tx=0x7f012c00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f013bfff640 1 -- 172.21.15.27:0/3735110243 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c00fe80 con 0x7f01441f42d0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f013bfff640 1 -- 172.21.15.27:0/3735110243 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c018070 con 0x7f01441f42d0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01440ad590 con 0x7f01441f42d0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f013bfff640 1 -- 172.21.15.27:0/3735110243 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c00d730 con 0x7f01441f42d0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.397+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01440ac7d0 con 0x7f01441f42d0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.398+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0144072a30 con 0x7f01441f42d0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.399+0000 7f013bfff640 1 -- 172.21.15.27:0/3735110243 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f012c00d8d0 con 0x7f01441f42d0 2024-03-06T07:35:54.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.400+0000 7f013bfff640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b4d0 0x7f012406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.400+0000 7f013bfff640 1 -- 172.21.15.27:0/3735110243 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f012c08a0b0 con 0x7f01441f42d0 2024-03-06T07:35:54.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.400+0000 7f01437fe640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b4d0 0x7f012406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.400+0000 7f01437fe640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b4d0 0x7f012406d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f0134007ce0 tx=0x7f0134001e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.404+0000 7f013bfff640 1 -- 172.21.15.27:0/3735110243 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f012c0588c0 con 0x7f01441f42d0 2024-03-06T07:35:54.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.585+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f014404f410 con 0x7f01441f42d0 2024-03-06T07:35:54.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.586+0000 7f013bfff640 1 -- 172.21.15.27:0/3735110243 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+11 (secure 0 0 0) 0x7f012c05c960 con 0x7f01441f42d0 2024-03-06T07:35:54.587 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-06T07:35:54.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b4d0 msgr2=0x7f012406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b4d0 0x7f012406d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f0134007ce0 tx=0x7f0134001e30 comp rx=0 tx=0).stop 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01441f42d0 msgr2=0x7f014415ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01441f42d0 0x7f014415ccc0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f012c00f770 tx=0x7f012c00fc30 comp rx=0 tx=0).stop 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 shutdown_connections 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f012406b4d0 0x7f012406d990 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01440aab40 0x7f01440aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01441f42d0 0x7f014415ccc0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 --2- 172.21.15.27:0/3735110243 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f014415c8e0 0x7f01440cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 >> 172.21.15.27:0/3735110243 conn(0x7f0144071960 msgr2=0x7f01440d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 shutdown_connections 2024-03-06T07:35:54.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.588+0000 7f014a04e640 1 -- 172.21.15.27:0/3735110243 wait complete. 2024-03-06T07:35:54.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:54.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.705+0000 7f0e7015c640 1 Processor -- start 2024-03-06T07:35:54.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.705+0000 7f0e7015c640 1 -- start start 2024-03-06T07:35:54.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e7015c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e6815df00 0x7f0e680d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e7015c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e680b95a0 0x7f0e680b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e7015c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e680cea30 con 0x7f0e680b9a30 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e7015c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e680d2af0 con 0x7f0e680b95a0 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e7015c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e680cf3c0 con 0x7f0e6815df00 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e6d6f7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e680b95a0 0x7f0e680b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e6d6f7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e680b95a0 0x7f0e680b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41848/0 (socket says 172.21.15.27:41848) 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e6d6f7640 1 -- 172.21.15.27:0/3045810681 learned_addr learned my addr 172.21.15.27:0/3045810681 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e6def8640 1 --2- 172.21.15.27:0/3045810681 >> v2:172.21.15.69:3300/0 conn(0x7f0e6815df00 0x7f0e680d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e6d6f7640 1 -- 172.21.15.27:0/3045810681 >> v2:172.21.15.69:3300/0 conn(0x7f0e6815df00 msgr2=0x7f0e680d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e6d6f7640 1 --2- 172.21.15.27:0/3045810681 >> v2:172.21.15.69:3300/0 conn(0x7f0e6815df00 0x7f0e680d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e6d6f7640 1 -- 172.21.15.27:0/3045810681 >> v1:172.21.15.27:6789/0 conn(0x7f0e680b9a30 legacy=0x7f0e680d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.706+0000 7f0e6d6f7640 1 -- 172.21.15.27:0/3045810681 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e680cf0e0 con 0x7f0e680b95a0 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.707+0000 7f0e6d6f7640 1 --2- 172.21.15.27:0/3045810681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680b95a0 0x7f0e680b5700 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f0e680d11b0 tx=0x7f0e5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0335c527731202 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.707+0000 7f0e6cef6640 1 -- 172.21.15.27:0/3045810681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e58034070 con 0x7f0e680b95a0 2024-03-06T07:35:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.707+0000 7f0e6cef6640 1 -- 172.21.15.27:0/3045810681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e58030860 con 0x7f0e680b95a0 2024-03-06T07:35:54.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.707+0000 7f0e6cef6640 1 -- 172.21.15.27:0/3045810681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e58030c30 con 0x7f0e680b95a0 2024-03-06T07:35:54.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.708+0000 7f0e7015c640 1 -- 172.21.15.27:0/3045810681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680b95a0 msgr2=0x7f0e680b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.708+0000 7f0e7015c640 1 --2- 172.21.15.27:0/3045810681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680b95a0 0x7f0e680b5700 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f0e680d11b0 tx=0x7f0e5802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.708+0000 7f0e7015c640 1 -- 172.21.15.27:0/3045810681 shutdown_connections 2024-03-06T07:35:54.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.708+0000 7f0e7015c640 1 --2- 172.21.15.27:0/3045810681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680b95a0 0x7f0e680b5700 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.708+0000 7f0e7015c640 1 --2- 172.21.15.27:0/3045810681 >> v2:172.21.15.69:3300/0 conn(0x7f0e6815df00 0x7f0e680d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.708+0000 7f0e7015c640 1 -- 172.21.15.27:0/3045810681 >> 172.21.15.27:0/3045810681 conn(0x7f0e68071960 msgr2=0x7f0e68071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.708+0000 7f0e7015c640 1 -- 172.21.15.27:0/3045810681 shutdown_connections 2024-03-06T07:35:54.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.708+0000 7f0e7015c640 1 -- 172.21.15.27:0/3045810681 wait complete. 2024-03-06T07:35:54.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.710+0000 7f0e7015c640 1 Processor -- start 2024-03-06T07:35:54.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.710+0000 7f0e7015c640 1 -- start start 2024-03-06T07:35:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e7015c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e680b95a0 0x7f0e681553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e7015c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6815df00 0x7f0e680cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e7015c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680cc960 0x7f0e681f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e7015c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e680cec70 con 0x7f0e680cc960 2024-03-06T07:35:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e7015c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e680ce750 con 0x7f0e680b95a0 2024-03-06T07:35:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e7015c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e680cedf0 con 0x7f0e6815df00 2024-03-06T07:35:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6e6f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680cc960 0x7f0e681f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6e6f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680cc960 0x7f0e681f3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41864/0 (socket says 172.21.15.27:41864) 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6e6f9640 1 -- 172.21.15.27:0/1444915731 learned_addr learned my addr 172.21.15.27:0/1444915731 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6def8640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e680b95a0 0x7f0e681553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6d6f7640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6815df00 0x7f0e680cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6e6f9640 1 -- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6815df00 msgr2=0x7f0e680cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6e6f9640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6815df00 0x7f0e680cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6e6f9640 1 -- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e680b95a0 msgr2=0x7f0e681553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6e6f9640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e680b95a0 0x7f0e681553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6e6f9640 1 -- 172.21.15.27:0/1444915731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e680cae80 con 0x7f0e680cc960 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.711+0000 7f0e6d6f7640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6815df00 0x7f0e680cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.712+0000 7f0e6e6f9640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680cc960 0x7f0e681f3930 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c7a0 tx=0x7f0e6000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.712+0000 7f0e5e7fc640 1 -- 172.21.15.27:0/1444915731 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e60003b70 con 0x7f0e680cc960 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.712+0000 7f0e5e7fc640 1 -- 172.21.15.27:0/1444915731 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e60003d30 con 0x7f0e680cc960 2024-03-06T07:35:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.712+0000 7f0e5e7fc640 1 -- 172.21.15.27:0/1444915731 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6000aa90 con 0x7f0e680cc960 2024-03-06T07:35:54.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.712+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e681f48a0 con 0x7f0e680cc960 2024-03-06T07:35:54.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.712+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e681f45c0 con 0x7f0e680cc960 2024-03-06T07:35:54.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.713+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e68067ff0 con 0x7f0e680cc960 2024-03-06T07:35:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.713+0000 7f0e5e7fc640 1 -- 172.21.15.27:0/1444915731 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e6000ac30 con 0x7f0e680cc960 2024-03-06T07:35:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.714+0000 7f0e5e7fc640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4c06b4d0 0x7f0e4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.714+0000 7f0e5e7fc640 1 -- 172.21.15.27:0/1444915731 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0e6008a040 con 0x7f0e680cc960 2024-03-06T07:35:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.714+0000 7f0e6def8640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4c06b4d0 0x7f0e4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.715+0000 7f0e6def8640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4c06b4d0 0x7f0e4c06d990 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0e640079a0 tx=0x7f0e64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.718+0000 7f0e5e7fc640 1 -- 172.21.15.27:0/1444915731 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e60058f30 con 0x7f0e680cc960 2024-03-06T07:35:54.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.938+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e680b7cf0 con 0x7f0e680cc960 2024-03-06T07:35:54.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.939+0000 7f0e5e7fc640 1 -- 172.21.15.27:0/1444915731 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f0e60058830 con 0x7f0e680cc960 2024-03-06T07:35:54.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:54.940 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:54.941 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4c06b4d0 msgr2=0x7f0e4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4c06b4d0 0x7f0e4c06d990 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0e640079a0 tx=0x7f0e64008040 comp rx=0 tx=0).stop 2024-03-06T07:35:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680cc960 msgr2=0x7f0e681f3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680cc960 0x7f0e681f3930 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c7a0 tx=0x7f0e6000cc60 comp rx=0 tx=0).stop 2024-03-06T07:35:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 shutdown_connections 2024-03-06T07:35:54.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0e4c06b4d0 0x7f0e4c06d990 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e680cc960 0x7f0e681f3930 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e6815df00 0x7f0e680cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 --2- 172.21.15.27:0/1444915731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e680b95a0 0x7f0e681553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 >> 172.21.15.27:0/1444915731 conn(0x7f0e68071960 msgr2=0x7f0e680d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 shutdown_connections 2024-03-06T07:35:54.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:54.941+0000 7f0e7015c640 1 -- 172.21.15.27:0/1444915731 wait complete. 2024-03-06T07:35:54.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:55.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.059+0000 7f30319e9640 1 Processor -- start 2024-03-06T07:35:55.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.059+0000 7f30319e9640 1 -- start start 2024-03-06T07:35:55.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.059+0000 7f30319e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c06df30 0x7f302c0ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.059+0000 7f30319e9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f302c0aec40 0x7f302c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.059+0000 7f30319e9640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c071240 con 0x7f302c158520 2024-03-06T07:35:55.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.059+0000 7f30319e9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c071b70 con 0x7f302c0aec40 2024-03-06T07:35:55.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.059+0000 7f30319e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0af0b0 con 0x7f302c06df30 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f302affd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f302c158520 0x7f302c06d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51876/0 (socket says 172.21.15.27:51876) 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f302affd640 1 -- 172.21.15.27:0/98295153 learned_addr learned my addr 172.21.15.27:0/98295153 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f302a7fc640 1 --2- 172.21.15.27:0/98295153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c06df30 0x7f302c0ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f302b7fe640 1 --2- 172.21.15.27:0/98295153 >> v2:172.21.15.69:3300/0 conn(0x7f302c0aec40 0x7f302c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f3029ffb640 1 -- 172.21.15.27:0/98295153 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722380252 0 0) 0x7f302c071240 con 0x7f302c158520 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f3029ffb640 1 -- 172.21.15.27:0/98295153 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3014003520 con 0x7f302c158520 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f302b7fe640 1 -- 172.21.15.27:0/98295153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c06df30 msgr2=0x7f302c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f302b7fe640 1 --2- 172.21.15.27:0/98295153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c06df30 0x7f302c0ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f302b7fe640 1 -- 172.21.15.27:0/98295153 >> v1:172.21.15.27:6789/0 conn(0x7f302c158520 legacy=0x7f302c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.060+0000 7f302b7fe640 1 -- 172.21.15.27:0/98295153 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c071520 con 0x7f302c0aec40 2024-03-06T07:35:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.061+0000 7f302b7fe640 1 --2- 172.21.15.27:0/98295153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c0aec40 0x7f302c0b3640 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3024009d90 tx=0x7f302402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4578870cc22be60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.061+0000 7f3029ffb640 1 -- 172.21.15.27:0/98295153 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3024039070 con 0x7f302c0aec40 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.061+0000 7f3029ffb640 1 -- 172.21.15.27:0/98295153 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302402cc50 con 0x7f302c0aec40 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.061+0000 7f3029ffb640 1 -- 172.21.15.27:0/98295153 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3024034440 con 0x7f302c0aec40 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.062+0000 7f30319e9640 1 -- 172.21.15.27:0/98295153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c0aec40 msgr2=0x7f302c0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.062+0000 7f30319e9640 1 --2- 172.21.15.27:0/98295153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c0aec40 0x7f302c0b3640 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3024009d90 tx=0x7f302402bea0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.062+0000 7f30319e9640 1 -- 172.21.15.27:0/98295153 shutdown_connections 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.062+0000 7f30319e9640 1 --2- 172.21.15.27:0/98295153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c0aec40 0x7f302c0b3640 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.062+0000 7f30319e9640 1 --2- 172.21.15.27:0/98295153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c06df30 0x7f302c0ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.062+0000 7f30319e9640 1 -- 172.21.15.27:0/98295153 >> 172.21.15.27:0/98295153 conn(0x7f302c066d00 msgr2=0x7f302c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.062+0000 7f30319e9640 1 -- 172.21.15.27:0/98295153 shutdown_connections 2024-03-06T07:35:55.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.062+0000 7f30319e9640 1 -- 172.21.15.27:0/98295153 wait complete. 2024-03-06T07:35:55.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.064+0000 7f30319e9640 1 Processor -- start 2024-03-06T07:35:55.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.064+0000 7f30319e9640 1 -- start start 2024-03-06T07:35:55.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f30319e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c06df30 0x7f302c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f30319e9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0aec40 0x7f302c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f30319e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f302c158520 0x7f302c0addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f30319e9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c158900 con 0x7f302c158520 2024-03-06T07:35:55.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f30319e9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c072000 con 0x7f302c06df30 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f30319e9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c158a80 con 0x7f302c0aec40 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f302c158520 0x7f302c0addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f302c158520 0x7f302c0addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41870/0 (socket says 172.21.15.27:41870) 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302b7fe640 1 -- 172.21.15.27:0/2618089993 learned_addr learned my addr 172.21.15.27:0/2618089993 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302a7fc640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0aec40 0x7f302c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302affd640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c06df30 0x7f302c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302b7fe640 1 -- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0aec40 msgr2=0x7f302c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302b7fe640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0aec40 0x7f302c0ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302b7fe640 1 -- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c06df30 msgr2=0x7f302c152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302b7fe640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c06df30 0x7f302c152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302b7fe640 1 -- 172.21.15.27:0/2618089993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c155150 con 0x7f302c158520 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f302b7fe640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f302c158520 0x7f302c0addf0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f3024037040 tx=0x7f3024004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.066+0000 7f3013fff640 1 -- 172.21.15.27:0/2618089993 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3024042030 con 0x7f302c158520 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.067+0000 7f3013fff640 1 -- 172.21.15.27:0/2618089993 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302403a040 con 0x7f302c158520 2024-03-06T07:35:55.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.067+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c155430 con 0x7f302c158520 2024-03-06T07:35:55.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.067+0000 7f3013fff640 1 -- 172.21.15.27:0/2618089993 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3024039070 con 0x7f302c158520 2024-03-06T07:35:55.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.067+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c1559f0 con 0x7f302c158520 2024-03-06T07:35:55.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.068+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c067ff0 con 0x7f302c158520 2024-03-06T07:35:55.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.068+0000 7f3013fff640 1 -- 172.21.15.27:0/2618089993 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3024045070 con 0x7f302c158520 2024-03-06T07:35:55.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.069+0000 7f3013fff640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f301806b4d0 0x7f301806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.069+0000 7f3013fff640 1 -- 172.21.15.27:0/2618089993 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f30240abce0 con 0x7f302c158520 2024-03-06T07:35:55.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.069+0000 7f302affd640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f301806b4d0 0x7f301806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.069+0000 7f302affd640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f301806b4d0 0x7f301806d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f301c005880 tx=0x7f301c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.073+0000 7f3013fff640 1 -- 172.21.15.27:0/2618089993 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f302404c070 con 0x7f302c158520 2024-03-06T07:35:55.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.276+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f302c15e3e0 con 0x7f302c158520 2024-03-06T07:35:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.277+0000 7f3013fff640 1 -- 172.21.15.27:0/2618089993 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f302407a640 con 0x7f302c158520 2024-03-06T07:35:55.279 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:55.279 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":149,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:32:31.280155+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:55.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f301806b4d0 msgr2=0x7f301806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f301806b4d0 0x7f301806d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f301c005880 tx=0x7f301c009040 comp rx=0 tx=0).stop 2024-03-06T07:35:55.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f302c158520 msgr2=0x7f302c0addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f302c158520 0x7f302c0addf0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f3024037040 tx=0x7f3024004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 shutdown_connections 2024-03-06T07:35:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f301806b4d0 0x7f301806d990 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f302c158520 0x7f302c0addf0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f302c0aec40 0x7f302c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 --2- 172.21.15.27:0/2618089993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f302c06df30 0x7f302c152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 >> 172.21.15.27:0/2618089993 conn(0x7f302c066d00 msgr2=0x7f302c0b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 shutdown_connections 2024-03-06T07:35:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.280+0000 7f30319e9640 1 -- 172.21.15.27:0/2618089993 wait complete. 2024-03-06T07:35:55.292 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:35:55.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:55.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.411+0000 7f91881bf640 1 Processor -- start 2024-03-06T07:35:55.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.411+0000 7f91881bf640 1 -- start start 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f91881bf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9180157b80 0x7f918015b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f91881bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 0x7f9180070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f91881bf640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918006c7b0 con 0x7f9180157f60 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f91881bf640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918006ca90 con 0x7f9180157b80 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f91881bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918006cd70 con 0x7f91800cbab0 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f9185f5b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9180157b80 0x7f918015b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f918575a640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9180157f60 0x7f918015b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51886/0 (socket says 172.21.15.27:51886) 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f918575a640 1 -- 172.21.15.27:0/985002420 learned_addr learned my addr 172.21.15.27:0/985002420 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f9185f5b640 1 -- 172.21.15.27:0/985002420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 msgr2=0x7f9180070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:55.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f918675c640 1 --2- 172.21.15.27:0/985002420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 0x7f9180070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f9184f59640 1 -- 172.21.15.27:0/985002420 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534626938 0 0) 0x7f918006c7b0 con 0x7f9180157f60 2024-03-06T07:35:55.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f9185f5b640 1 --2- 172.21.15.27:0/985002420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 0x7f9180070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f9185f5b640 1 -- 172.21.15.27:0/985002420 >> v1:172.21.15.27:6789/0 conn(0x7f9180157f60 legacy=0x7f918015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f9185f5b640 1 -- 172.21.15.27:0/985002420 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918006d050 con 0x7f9180157b80 2024-03-06T07:35:55.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.412+0000 7f918675c640 1 --2- 172.21.15.27:0/985002420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 0x7f9180070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:55.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.413+0000 7f9185f5b640 1 --2- 172.21.15.27:0/985002420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 0x7f918015b260 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f917c0097a0 tx=0x7f917c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a72a1dc8df75e7d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.413+0000 7f9184f59640 1 -- 172.21.15.27:0/985002420 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c035020 con 0x7f9180157b80 2024-03-06T07:35:55.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.413+0000 7f9184f59640 1 -- 172.21.15.27:0/985002420 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c031830 con 0x7f9180157b80 2024-03-06T07:35:55.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.413+0000 7f9184f59640 1 -- 172.21.15.27:0/985002420 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c031c00 con 0x7f9180157b80 2024-03-06T07:35:55.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.414+0000 7f91881bf640 1 -- 172.21.15.27:0/985002420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 msgr2=0x7f918015b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.414+0000 7f91881bf640 1 --2- 172.21.15.27:0/985002420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 0x7f918015b260 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f917c0097a0 tx=0x7f917c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.414+0000 7f91881bf640 1 -- 172.21.15.27:0/985002420 shutdown_connections 2024-03-06T07:35:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.414+0000 7f91881bf640 1 --2- 172.21.15.27:0/985002420 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 0x7f9180070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.414+0000 7f91881bf640 1 --2- 172.21.15.27:0/985002420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 0x7f918015b260 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.414+0000 7f91881bf640 1 -- 172.21.15.27:0/985002420 >> 172.21.15.27:0/985002420 conn(0x7f9180066d00 msgr2=0x7f91800670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.414+0000 7f91881bf640 1 -- 172.21.15.27:0/985002420 shutdown_connections 2024-03-06T07:35:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.414+0000 7f91881bf640 1 -- 172.21.15.27:0/985002420 wait complete. 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.416+0000 7f91881bf640 1 Processor -- start 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.416+0000 7f91881bf640 1 -- start start 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f91881bf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 0x7f9180155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f91881bf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 0x7f9180154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f91881bf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9180157f60 0x7f91800b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f91881bf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9180071080 con 0x7f9180157b80 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f91881bf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91800cc260 con 0x7f9180157f60 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f91881bf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9180071200 con 0x7f91800cbab0 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918575a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 0x7f9180154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918575a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 0x7f9180154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41886/0 (socket says 172.21.15.27:41886) 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918575a640 1 -- 172.21.15.27:0/223592245 learned_addr learned my addr 172.21.15.27:0/223592245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:55.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f9185f5b640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 0x7f9180155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918675c640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9180157f60 0x7f91800b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918575a640 1 -- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 msgr2=0x7f9180155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918575a640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 0x7f9180155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918575a640 1 -- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9180157f60 msgr2=0x7f91800b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918575a640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9180157f60 0x7f91800b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918575a640 1 -- 172.21.15.27:0/223592245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91801566d0 con 0x7f9180157b80 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.417+0000 7f918675c640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9180157f60 0x7f91800b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.418+0000 7f918575a640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 0x7f9180154500 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f917000e6b0 tx=0x7f917000eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.418+0000 7f91767fc640 1 -- 172.21.15.27:0/223592245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91700070d0 con 0x7f9180157b80 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.418+0000 7f91767fc640 1 -- 172.21.15.27:0/223592245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9170016020 con 0x7f9180157b80 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.418+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91801569b0 con 0x7f9180157b80 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.418+0000 7f91767fc640 1 -- 172.21.15.27:0/223592245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91700074f0 con 0x7f9180157b80 2024-03-06T07:35:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.418+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9180150a60 con 0x7f9180157b80 2024-03-06T07:35:55.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.419+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9180067ff0 con 0x7f9180157b80 2024-03-06T07:35:55.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.419+0000 7f91767fc640 1 -- 172.21.15.27:0/223592245 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9170001730 con 0x7f9180157b80 2024-03-06T07:35:55.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.420+0000 7f91767fc640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f916406b4d0 0x7f916406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.420+0000 7f91767fc640 1 -- 172.21.15.27:0/223592245 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f917008c350 con 0x7f9180157b80 2024-03-06T07:35:55.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.420+0000 7f9185f5b640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f916406b4d0 0x7f916406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.421+0000 7f9185f5b640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f916406b4d0 0x7f916406d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f917c036000 tx=0x7f917c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.424+0000 7f91767fc640 1 -- 172.21.15.27:0/223592245 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9170020030 con 0x7f9180157b80 2024-03-06T07:35:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.642+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91800d07a0 con 0x7f9180157b80 2024-03-06T07:35:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.642+0000 7f91767fc640 1 -- 172.21.15.27:0/223592245 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7f917005b240 con 0x7f9180157b80 2024-03-06T07:35:55.644 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:55.644 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f916406b4d0 msgr2=0x7f916406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f916406b4d0 0x7f916406d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f917c036000 tx=0x7f917c03d040 comp rx=0 tx=0).stop 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 msgr2=0x7f9180154500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 0x7f9180154500 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f917000e6b0 tx=0x7f917000eb70 comp rx=0 tx=0).stop 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 shutdown_connections 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f916406b4d0 0x7f916406d990 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9180157f60 0x7f91800b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9180157b80 0x7f9180154500 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 --2- 172.21.15.27:0/223592245 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f91800cbab0 0x7f9180155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 >> 172.21.15.27:0/223592245 conn(0x7f9180066d00 msgr2=0x7f91801586a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 shutdown_connections 2024-03-06T07:35:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.644+0000 7f91881bf640 1 -- 172.21.15.27:0/223592245 wait complete. 2024-03-06T07:35:55.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:55.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.761+0000 7fcc5485c640 1 Processor -- start 2024-03-06T07:35:55.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.761+0000 7fcc5485c640 1 -- start start 2024-03-06T07:35:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.762+0000 7fcc5485c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc5015df00 0x7fcc500d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.762+0000 7fcc5485c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc500b95a0 0x7fcc500b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.762+0000 7fcc5485c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc500cea30 con 0x7fcc500b9a30 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.762+0000 7fcc5485c640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc500d2af0 con 0x7fcc5015df00 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.762+0000 7fcc5485c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500cf3c0 con 0x7fcc500b95a0 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4e59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc5015df00 0x7fcc500d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4e59c640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc5015df00 0x7fcc500d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41888/0 (socket says 172.21.15.27:41888) 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4ed9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcc500b9a30 0x7fcc500d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51896/0 (socket says 172.21.15.27:51896) 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4e59c640 1 -- 172.21.15.27:0/2342031086 learned_addr learned my addr 172.21.15.27:0/2342031086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4dd9b640 1 --2- 172.21.15.27:0/2342031086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc500b95a0 0x7fcc500b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4e59c640 1 -- 172.21.15.27:0/2342031086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc500b95a0 msgr2=0x7fcc500b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4e59c640 1 --2- 172.21.15.27:0/2342031086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc500b95a0 0x7fcc500b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4d59a640 1 -- 172.21.15.27:0/2342031086 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878200566 0 0) 0x7fcc500cea30 con 0x7fcc500b9a30 2024-03-06T07:35:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4e59c640 1 -- 172.21.15.27:0/2342031086 >> v1:172.21.15.27:6789/0 conn(0x7fcc500b9a30 legacy=0x7fcc500d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.763+0000 7fcc4e59c640 1 -- 172.21.15.27:0/2342031086 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500cf0e0 con 0x7fcc5015df00 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.764+0000 7fcc4e59c640 1 --2- 172.21.15.27:0/2342031086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc5015df00 0x7fcc500d25b0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009870 tx=0x7fcc4002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a04f2b34db8174 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.764+0000 7fcc4d59a640 1 -- 172.21.15.27:0/2342031086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40034070 con 0x7fcc5015df00 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.764+0000 7fcc4d59a640 1 -- 172.21.15.27:0/2342031086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc40030840 con 0x7fcc5015df00 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.764+0000 7fcc4d59a640 1 -- 172.21.15.27:0/2342031086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40030c10 con 0x7fcc5015df00 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.765+0000 7fcc5485c640 1 -- 172.21.15.27:0/2342031086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc5015df00 msgr2=0x7fcc500d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.765+0000 7fcc5485c640 1 --2- 172.21.15.27:0/2342031086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc5015df00 0x7fcc500d25b0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009870 tx=0x7fcc4002b980 comp rx=0 tx=0).stop 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.765+0000 7fcc5485c640 1 -- 172.21.15.27:0/2342031086 shutdown_connections 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.765+0000 7fcc5485c640 1 --2- 172.21.15.27:0/2342031086 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc500b95a0 0x7fcc500b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.765+0000 7fcc5485c640 1 --2- 172.21.15.27:0/2342031086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc5015df00 0x7fcc500d25b0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.765+0000 7fcc5485c640 1 -- 172.21.15.27:0/2342031086 >> 172.21.15.27:0/2342031086 conn(0x7fcc50071960 msgr2=0x7fcc50071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.765+0000 7fcc5485c640 1 -- 172.21.15.27:0/2342031086 shutdown_connections 2024-03-06T07:35:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.765+0000 7fcc5485c640 1 -- 172.21.15.27:0/2342031086 wait complete. 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.766+0000 7fcc5485c640 1 Processor -- start 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.766+0000 7fcc5485c640 1 -- start start 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc5485c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc500b95a0 0x7fcc5014e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc5485c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc5015df00 0x7fcc500c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc5485c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501f3c60 0x7fcc500b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc5485c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500cec70 con 0x7fcc501f3c60 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc5485c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500ce750 con 0x7fcc500b95a0 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc5485c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500cedf0 con 0x7fcc5015df00 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc4ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501f3c60 0x7fcc500b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc4ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501f3c60 0x7fcc500b5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41896/0 (socket says 172.21.15.27:41896) 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc4ed9d640 1 -- 172.21.15.27:0/1753641450 learned_addr learned my addr 172.21.15.27:0/1753641450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc4dd9b640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc5015df00 0x7fcc500c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.767+0000 7fcc4e59c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc500b95a0 0x7fcc5014e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc4ed9d640 1 -- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc5015df00 msgr2=0x7fcc500c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc4ed9d640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc5015df00 0x7fcc500c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc4ed9d640 1 -- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc500b95a0 msgr2=0x7fcc5014e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc4ed9d640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc500b95a0 0x7fcc5014e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc4ed9d640 1 -- 172.21.15.27:0/1753641450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5014e670 con 0x7fcc501f3c60 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc4e59c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc500b95a0 0x7fcc5014e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc4dd9b640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc5015df00 0x7fcc500c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc4ed9d640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501f3c60 0x7fcc500b5210 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800c780 tx=0x7fcc4800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc3effd640 1 -- 172.21.15.27:0/1753641450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48018070 con 0x7fcc501f3c60 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc3effd640 1 -- 172.21.15.27:0/1753641450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4800a640 con 0x7fcc501f3c60 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc500cc140 con 0x7fcc501f3c60 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc3effd640 1 -- 172.21.15.27:0/1753641450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4800aab0 con 0x7fcc501f3c60 2024-03-06T07:35:55.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.768+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc500cc700 con 0x7fcc501f3c60 2024-03-06T07:35:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.769+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc500cced0 con 0x7fcc501f3c60 2024-03-06T07:35:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.770+0000 7fcc3effd640 1 -- 172.21.15.27:0/1753641450 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc4801b070 con 0x7fcc501f3c60 2024-03-06T07:35:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.770+0000 7fcc3effd640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc3006b4d0 0x7fcc3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.770+0000 7fcc3effd640 1 -- 172.21.15.27:0/1753641450 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcc48089770 con 0x7fcc501f3c60 2024-03-06T07:35:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.770+0000 7fcc4e59c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc3006b4d0 0x7fcc3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.771+0000 7fcc4e59c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc3006b4d0 0x7fcc3006d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fcc40036000 tx=0x7fcc40008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.779+0000 7fcc3effd640 1 -- 172.21.15.27:0/1753641450 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc5014e670 con 0x7fcc501f3c60 2024-03-06T07:35:55.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.997+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc500c7230 con 0x7fcc501f3c60 2024-03-06T07:35:55.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.998+0000 7fcc3effd640 1 -- 172.21.15.27:0/1753641450 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3469 (secure 0 0 0) 0x7fcc48058660 con 0x7fcc501f3c60 2024-03-06T07:35:55.999 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:55.999 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:44.385375+0000","modified":"2024-03-06T07:34:20.030594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9282,"mds_1":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":1,"incarnation":114,"state":"up:stopping","state_seq":6,"addr":"172.21.15.69:6837/1115582329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6835","nonce":1115582329},{"type":"v1","addr":"172.21.15.69:6837","nonce":1115582329}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:33:13.569395+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9282":{"gid":9282,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2910933327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2910933327},{"type":"v1","addr":"172.21.15.27:6834","nonce":2910933327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9282,"qdb_cluster":[9282]},"id":10}]} 2024-03-06T07:35:55.999 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-06T07:35:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.999+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc3006b4d0 msgr2=0x7fcc3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.999+0000 7fcc5485c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc3006b4d0 0x7fcc3006d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fcc40036000 tx=0x7fcc40008030 comp rx=0 tx=0).stop 2024-03-06T07:35:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.999+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501f3c60 msgr2=0x7fcc500b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.999+0000 7fcc5485c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501f3c60 0x7fcc500b5210 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800c780 tx=0x7fcc4800cc40 comp rx=0 tx=0).stop 2024-03-06T07:35:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:55.999+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 shutdown_connections 2024-03-06T07:35:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.000+0000 7fcc5485c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcc3006b4d0 0x7fcc3006d990 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.000+0000 7fcc5485c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501f3c60 0x7fcc500b5210 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.000+0000 7fcc5485c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fcc5015df00 0x7fcc500c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.000+0000 7fcc5485c640 1 --2- 172.21.15.27:0/1753641450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc500b95a0 0x7fcc5014e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.000+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 >> 172.21.15.27:0/1753641450 conn(0x7fcc50071960 msgr2=0x7fcc500b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.000+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 shutdown_connections 2024-03-06T07:35:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.000+0000 7fcc5485c640 1 -- 172.21.15.27:0/1753641450 wait complete. 2024-03-06T07:35:56.012 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:56.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.159+0000 7fe3c579a640 1 Processor -- start 2024-03-06T07:35:56.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.159+0000 7fe3c579a640 1 -- start start 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.160+0000 7fe3c579a640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3c015df70 0x7fe3c00d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.160+0000 7fe3c579a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c00b6f20 con 0x7fe3c00d18c0 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.160+0000 7fe3c579a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c00d3030 con 0x7fe3c0071c50 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.160+0000 7fe3c579a640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c00d3310 con 0x7fe3c015df70 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.160+0000 7fe3beffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3c015df70 0x7fe3c00d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.160+0000 7fe3be7fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe3c00d18c0 0x7fe3c0071540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51908/0 (socket says 172.21.15.27:51908) 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.160+0000 7fe3beffd640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3c015df70 0x7fe3c00d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41902/0 (socket says 172.21.15.27:41902) 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.160+0000 7fe3be7fc640 1 -- 172.21.15.27:0/412637546 learned_addr learned my addr 172.21.15.27:0/412637546 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.160+0000 7fe3beffd640 1 -- 172.21.15.27:0/412637546 >> v1:172.21.15.69:6789/0 conn(0x7fe3c0071c50 legacy=0x7fe3c00b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.161+0000 7fe3beffd640 1 -- 172.21.15.27:0/412637546 >> v1:172.21.15.27:6789/0 conn(0x7fe3c00d18c0 legacy=0x7fe3c0071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.161+0000 7fe3bdffb640 1 -- 172.21.15.27:0/412637546 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755661096 0 0) 0x7fe3c00b6f20 con 0x7fe3c00d18c0 2024-03-06T07:35:56.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.161+0000 7fe3beffd640 1 -- 172.21.15.27:0/412637546 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c00d35f0 con 0x7fe3c015df70 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.161+0000 7fe3beffd640 1 --2- 172.21.15.27:0/412637546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 0x7fe3c00d27b0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009d90 tx=0x7fe3b002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b815e0ee68bf9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.161+0000 7fe3bdffb640 1 -- 172.21.15.27:0/412637546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0039070 con 0x7fe3c015df70 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.161+0000 7fe3bdffb640 1 -- 172.21.15.27:0/412637546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b002cd00 con 0x7fe3c015df70 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.161+0000 7fe3bdffb640 1 -- 172.21.15.27:0/412637546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0034660 con 0x7fe3c015df70 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.162+0000 7fe3c579a640 1 -- 172.21.15.27:0/412637546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 msgr2=0x7fe3c00d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.162+0000 7fe3c579a640 1 --2- 172.21.15.27:0/412637546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 0x7fe3c00d27b0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009d90 tx=0x7fe3b002bf50 comp rx=0 tx=0).stop 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.162+0000 7fe3c579a640 1 -- 172.21.15.27:0/412637546 shutdown_connections 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.162+0000 7fe3c579a640 1 --2- 172.21.15.27:0/412637546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 0x7fe3c00d27b0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.162+0000 7fe3c579a640 1 -- 172.21.15.27:0/412637546 >> 172.21.15.27:0/412637546 conn(0x7fe3c00d3cf0 msgr2=0x7fe3c00d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.162+0000 7fe3c579a640 1 -- 172.21.15.27:0/412637546 shutdown_connections 2024-03-06T07:35:56.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.162+0000 7fe3c579a640 1 -- 172.21.15.27:0/412637546 wait complete. 2024-03-06T07:35:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.164+0000 7fe3c579a640 1 Processor -- start 2024-03-06T07:35:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.164+0000 7fe3c579a640 1 -- start start 2024-03-06T07:35:56.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3c579a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3c0071c50 0x7fe3c0151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3c579a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3c00d18c0 0x7fe3c00cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3c579a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 0x7fe3c00cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3c579a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c015e350 con 0x7fe3c015df70 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3c579a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00d2cf0 con 0x7fe3c00d18c0 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3c579a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c015e4d0 con 0x7fe3c0071c50 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 0x7fe3c00cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 0x7fe3c00cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41908/0 (socket says 172.21.15.27:41908) 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3bf7fe640 1 -- 172.21.15.27:0/511860192 learned_addr learned my addr 172.21.15.27:0/511860192 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3be7fc640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3c00d18c0 0x7fe3c00cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3beffd640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3c0071c50 0x7fe3c0151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3bf7fe640 1 -- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3c0071c50 msgr2=0x7fe3c0151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3bf7fe640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3c0071c50 0x7fe3c0151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3bf7fe640 1 -- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3c00d18c0 msgr2=0x7fe3c00cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3bf7fe640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3c00d18c0 0x7fe3c00cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3bf7fe640 1 -- 172.21.15.27:0/511860192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c00cc500 con 0x7fe3c015df70 2024-03-06T07:35:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3be7fc640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3c00d18c0 0x7fe3c00cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3bf7fe640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 0x7fe3c00cda40 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800c3e0 tx=0x7fe3b800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3b77fe640 1 -- 172.21.15.27:0/511860192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8011790 con 0x7fe3c015df70 2024-03-06T07:35:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3b77fe640 1 -- 172.21.15.27:0/511860192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b8017070 con 0x7fe3c015df70 2024-03-06T07:35:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c00cc7e0 con 0x7fe3c015df70 2024-03-06T07:35:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.166+0000 7fe3b77fe640 1 -- 172.21.15.27:0/511860192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8011bb0 con 0x7fe3c015df70 2024-03-06T07:35:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.167+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c00c8af0 con 0x7fe3c015df70 2024-03-06T07:35:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.168+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c0067ff0 con 0x7fe3c015df70 2024-03-06T07:35:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.168+0000 7fe3b77fe640 1 -- 172.21.15.27:0/511860192 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3b801c460 con 0x7fe3c015df70 2024-03-06T07:35:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.168+0000 7fe3b77fe640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe3a006b500 0x7fe3a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.169+0000 7fe3b77fe640 1 -- 172.21.15.27:0/511860192 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe3b8089ed0 con 0x7fe3c015df70 2024-03-06T07:35:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.169+0000 7fe3beffd640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe3a006b500 0x7fe3a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.169+0000 7fe3beffd640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe3a006b500 0x7fe3a006d9c0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0036000 tx=0x7fe3b0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.173+0000 7fe3b77fe640 1 -- 172.21.15.27:0/511860192 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3b8058dc0 con 0x7fe3c015df70 2024-03-06T07:35:56.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.474+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c00caa00 con 0x7fe3c015df70 2024-03-06T07:35:56.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.475+0000 7fe3b77fe640 1 -- 172.21.15.27:0/511860192 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16657 (secure 0 0 0) 0x7fe3b80586c0 con 0x7fe3c015df70 2024-03-06T07:35:56.477 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:35:56.477 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":149,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:32:31.280155+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:42.692383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:43.367948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:56.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.478+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe3a006b500 msgr2=0x7fe3a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.478+0000 7fe3c579a640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe3a006b500 0x7fe3a006d9c0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0036000 tx=0x7fe3b0030000 comp rx=0 tx=0).stop 2024-03-06T07:35:56.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.478+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 msgr2=0x7fe3c00cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.478+0000 7fe3c579a640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 0x7fe3c00cda40 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800c3e0 tx=0x7fe3b800c8a0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.479+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 shutdown_connections 2024-03-06T07:35:56.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.479+0000 7fe3c579a640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe3a006b500 0x7fe3a006d9c0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.479+0000 7fe3c579a640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3c015df70 0x7fe3c00cda40 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.479+0000 7fe3c579a640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3c00d18c0 0x7fe3c00cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.479+0000 7fe3c579a640 1 --2- 172.21.15.27:0/511860192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3c0071c50 0x7fe3c0151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.479+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 >> 172.21.15.27:0/511860192 conn(0x7fe3c00d3cf0 msgr2=0x7fe3c00d1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.479+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 shutdown_connections 2024-03-06T07:35:56.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.479+0000 7fe3c579a640 1 -- 172.21.15.27:0/511860192 wait complete. 2024-03-06T07:35:56.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.602+0000 7f8e1838c640 1 Processor -- start 2024-03-06T07:35:56.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.602+0000 7f8e1838c640 1 -- start start 2024-03-06T07:35:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.603+0000 7f8e1838c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8e100aee10 0x7f8e10165a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.603+0000 7f8e1838c640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e10071240 con 0x7f8e1006df30 2024-03-06T07:35:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.603+0000 7f8e1838c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e10071520 con 0x7f8e10158520 2024-03-06T07:35:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.603+0000 7f8e1838c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e10071b70 con 0x7f8e100aee10 2024-03-06T07:35:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.603+0000 7f8e15927640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8e1006df30 0x7f8e100ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51910/0 (socket says 172.21.15.27:51910) 2024-03-06T07:35:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.603+0000 7f8e15927640 1 -- 172.21.15.27:0/4069770571 learned_addr learned my addr 172.21.15.27:0/4069770571 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.603+0000 7f8e16929640 1 --2- 172.21.15.27:0/4069770571 >> v2:172.21.15.69:3300/0 conn(0x7f8e100aee10 0x7f8e10165a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.604+0000 7f8e15126640 1 -- 172.21.15.27:0/4069770571 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997649699 0 0) 0x7f8e10071240 con 0x7f8e1006df30 2024-03-06T07:35:56.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.604+0000 7f8e15126640 1 -- 172.21.15.27:0/4069770571 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dfc003520 con 0x7f8e1006df30 2024-03-06T07:35:56.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.604+0000 7f8e16929640 1 -- 172.21.15.27:0/4069770571 >> v1:172.21.15.69:6789/0 conn(0x7f8e10158520 legacy=0x7f8e1006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.604+0000 7f8e16929640 1 -- 172.21.15.27:0/4069770571 >> v1:172.21.15.27:6789/0 conn(0x7f8e1006df30 legacy=0x7f8e100ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.604+0000 7f8e16929640 1 -- 172.21.15.27:0/4069770571 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e10071800 con 0x7f8e100aee10 2024-03-06T07:35:56.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e16929640 1 --2- 172.21.15.27:0/4069770571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 0x7f8e10165a00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800bfa0 tx=0x7f8e0802e0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90d1c66fb530cad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e15126640 1 -- 172.21.15.27:0/4069770571 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0802ea50 con 0x7f8e100aee10 2024-03-06T07:35:56.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e15126640 1 -- 172.21.15.27:0/4069770571 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0802ebf0 con 0x7f8e100aee10 2024-03-06T07:35:56.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e15126640 1 -- 172.21.15.27:0/4069770571 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e08009540 con 0x7f8e100aee10 2024-03-06T07:35:56.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e1838c640 1 -- 172.21.15.27:0/4069770571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 msgr2=0x7f8e10165a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e1838c640 1 --2- 172.21.15.27:0/4069770571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 0x7f8e10165a00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800bfa0 tx=0x7f8e0802e0b0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e1838c640 1 -- 172.21.15.27:0/4069770571 shutdown_connections 2024-03-06T07:35:56.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e1838c640 1 --2- 172.21.15.27:0/4069770571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 0x7f8e10165a00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e1838c640 1 -- 172.21.15.27:0/4069770571 >> 172.21.15.27:0/4069770571 conn(0x7f8e10066d00 msgr2=0x7f8e100670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:56.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e1838c640 1 -- 172.21.15.27:0/4069770571 shutdown_connections 2024-03-06T07:35:56.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.605+0000 7f8e1838c640 1 -- 172.21.15.27:0/4069770571 wait complete. 2024-03-06T07:35:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.606+0000 7f8e1838c640 1 Processor -- start 2024-03-06T07:35:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e1838c640 1 -- start start 2024-03-06T07:35:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e1838c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e1006df30 0x7f8e1014f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e1838c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 0x7f8e100ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e1838c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10158520 0x7f8e100ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e1838c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e10158900 con 0x7f8e10158520 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e1838c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e10072000 con 0x7f8e100aee10 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e1838c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e10158a80 con 0x7f8e1006df30 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e16929640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10158520 0x7f8e100ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e16929640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10158520 0x7f8e100ce570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41920/0 (socket says 172.21.15.27:41920) 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.607+0000 7f8e16929640 1 -- 172.21.15.27:0/3158162897 learned_addr learned my addr 172.21.15.27:0/3158162897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e15927640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 0x7f8e100ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e16128640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e1006df30 0x7f8e1014f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e16929640 1 -- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e1006df30 msgr2=0x7f8e1014f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e16929640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e1006df30 0x7f8e1014f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e16929640 1 -- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 msgr2=0x7f8e100ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e16929640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 0x7f8e100ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e16929640 1 -- 172.21.15.27:0/3158162897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1014fdb0 con 0x7f8e10158520 2024-03-06T07:35:56.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e15927640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 0x7f8e100ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:56.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e16929640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10158520 0x7f8e100ce570 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800d110 tx=0x7f8e08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e06ffd640 1 -- 172.21.15.27:0/3158162897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e08040620 con 0x7f8e10158520 2024-03-06T07:35:56.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e06ffd640 1 -- 172.21.15.27:0/3158162897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e08045070 con 0x7f8e10158520 2024-03-06T07:35:56.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e06ffd640 1 -- 172.21.15.27:0/3158162897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e08040a20 con 0x7f8e10158520 2024-03-06T07:35:56.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e100cd4d0 con 0x7f8e10158520 2024-03-06T07:35:56.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.608+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e100cdd70 con 0x7f8e10158520 2024-03-06T07:35:56.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.609+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e10067ff0 con 0x7f8e10158520 2024-03-06T07:35:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.610+0000 7f8e06ffd640 1 -- 172.21.15.27:0/3158162897 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e08040bc0 con 0x7f8e10158520 2024-03-06T07:35:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.610+0000 7f8e06ffd640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b500 0x7f8df406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.610+0000 7f8e06ffd640 1 -- 172.21.15.27:0/3158162897 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8e080ad0a0 con 0x7f8e10158520 2024-03-06T07:35:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.610+0000 7f8e16128640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b500 0x7f8df406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.611+0000 7f8e16128640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b500 0x7f8df406d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c000f60 tx=0x7f8e0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:56.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:56.614+0000 7f8e06ffd640 1 -- 172.21.15.27:0/3158162897 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e1014fdb0 con 0x7f8e10158520 2024-03-06T07:35:57.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.035+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8e100b1330 con 0x7f8e10158520 2024-03-06T07:35:57.037 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-03-06T07:35:57.036+0000 7fa8a9b06640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:35:57.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.037+0000 7f8e06ffd640 1 -- 172.21.15.27:0/3158162897 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v124) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8e08035e40 con 0x7f8e10158520 2024-03-06T07:35:57.038 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b500 msgr2=0x7f8df406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b500 0x7f8df406d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c000f60 tx=0x7f8e0c009040 comp rx=0 tx=0).stop 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10158520 msgr2=0x7f8e100ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10158520 0x7f8e100ce570 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800d110 tx=0x7f8e08004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 shutdown_connections 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f8df406b500 0x7f8df406d9c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10158520 0x7f8e100ce570 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e100aee10 0x7f8e100ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 --2- 172.21.15.27:0/3158162897 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8e1006df30 0x7f8e1014f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 >> 172.21.15.27:0/3158162897 conn(0x7f8e10066d00 msgr2=0x7f8e10158c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:57.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.039+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 shutdown_connections 2024-03-06T07:35:57.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.040+0000 7f8e1838c640 1 -- 172.21.15.27:0/3158162897 wait complete. 2024-03-06T07:35:57.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:35:57.081 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-06T07:35:57.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.164+0000 7fde5fcc3640 1 Processor -- start 2024-03-06T07:35:57.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.164+0000 7fde5fcc3640 1 -- start start 2024-03-06T07:35:57.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5fcc3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815ae50 0x7fde580708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5fcc3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde5815b230 0x7fde58071160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5fcc3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde58070e20 con 0x7fde5815b690 2024-03-06T07:35:57.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5fcc3640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde580b2e30 con 0x7fde5815b230 2024-03-06T07:35:57.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5fcc3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5806dfe0 con 0x7fde5815ae50 2024-03-06T07:35:57.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5da5f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815ae50 0x7fde580708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5d25e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fde5815b230 0x7fde58071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5e260640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fde5815b690 0x7fde58150a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:35806/0 (socket says 172.21.15.27:35806) 2024-03-06T07:35:57.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5e260640 1 -- 172.21.15.27:0/2971440038 learned_addr learned my addr 172.21.15.27:0/2971440038 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:57.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5ca5d640 1 -- 172.21.15.27:0/2971440038 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372349436 0 0) 0x7fde58070e20 con 0x7fde5815b690 2024-03-06T07:35:57.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5da5f640 1 -- 172.21.15.27:0/2971440038 >> v2:172.21.15.69:3300/0 conn(0x7fde5815b230 msgr2=0x7fde58071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5da5f640 1 --2- 172.21.15.27:0/2971440038 >> v2:172.21.15.69:3300/0 conn(0x7fde5815b230 0x7fde58071160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5da5f640 1 -- 172.21.15.27:0/2971440038 >> v1:172.21.15.69:6789/0 conn(0x7fde5815b690 legacy=0x7fde58150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.165+0000 7fde5da5f640 1 -- 172.21.15.27:0/2971440038 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580b30b0 con 0x7fde5815ae50 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.166+0000 7fde5da5f640 1 --2- 172.21.15.27:0/2971440038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815ae50 0x7fde580708e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde540097a0 tx=0x7fde5402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6c6bbb0878161c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.166+0000 7fde5ca5d640 1 -- 172.21.15.27:0/2971440038 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde540315a0 con 0x7fde5815ae50 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.166+0000 7fde5ca5d640 1 -- 172.21.15.27:0/2971440038 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde54037070 con 0x7fde5815ae50 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.166+0000 7fde5ca5d640 1 -- 172.21.15.27:0/2971440038 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54031960 con 0x7fde5815ae50 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.167+0000 7fde5fcc3640 1 -- 172.21.15.27:0/2971440038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815ae50 msgr2=0x7fde580708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.167+0000 7fde5fcc3640 1 --2- 172.21.15.27:0/2971440038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815ae50 0x7fde580708e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde540097a0 tx=0x7fde5402b840 comp rx=0 tx=0).stop 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.167+0000 7fde5fcc3640 1 -- 172.21.15.27:0/2971440038 shutdown_connections 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.167+0000 7fde5fcc3640 1 --2- 172.21.15.27:0/2971440038 >> v2:172.21.15.69:3300/0 conn(0x7fde5815b230 0x7fde58071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.167+0000 7fde5fcc3640 1 --2- 172.21.15.27:0/2971440038 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815ae50 0x7fde580708e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.167+0000 7fde5fcc3640 1 -- 172.21.15.27:0/2971440038 >> 172.21.15.27:0/2971440038 conn(0x7fde58066d00 msgr2=0x7fde58067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.167+0000 7fde5fcc3640 1 -- 172.21.15.27:0/2971440038 shutdown_connections 2024-03-06T07:35:57.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.167+0000 7fde5fcc3640 1 -- 172.21.15.27:0/2971440038 wait complete. 2024-03-06T07:35:57.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.168+0000 7fde5fcc3640 1 Processor -- start 2024-03-06T07:35:57.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.168+0000 7fde5fcc3640 1 -- start start 2024-03-06T07:35:57.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5fcc3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde5815ae50 0x7fde581f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5fcc3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815b230 0x7fde58154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5fcc3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde5815b690 0x7fde580ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5fcc3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde580b3950 con 0x7fde5815ae50 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5fcc3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde580b3670 con 0x7fde5815b690 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5fcc3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde580b4150 con 0x7fde5815b230 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5da5f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde5815ae50 0x7fde581f5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5da5f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde5815ae50 0x7fde581f5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41950/0 (socket says 172.21.15.27:41950) 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5da5f640 1 -- 172.21.15.27:0/4261795624 learned_addr learned my addr 172.21.15.27:0/4261795624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5e260640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde5815b690 0x7fde580ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5d25e640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815b230 0x7fde58154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5da5f640 1 -- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815b230 msgr2=0x7fde58154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5da5f640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815b230 0x7fde58154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5da5f640 1 -- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde5815b690 msgr2=0x7fde580ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5da5f640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde5815b690 0x7fde580ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5da5f640 1 -- 172.21.15.27:0/4261795624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde581f21e0 con 0x7fde5815ae50 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.169+0000 7fde5e260640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde5815b690 0x7fde580ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.170+0000 7fde5da5f640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde5815ae50 0x7fde581f5d50 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fde54034040 tx=0x7fde540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.170+0000 7fde4e7fc640 1 -- 172.21.15.27:0/4261795624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54004230 con 0x7fde5815ae50 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.170+0000 7fde4e7fc640 1 -- 172.21.15.27:0/4261795624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde540043d0 con 0x7fde5815ae50 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.170+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde581f24c0 con 0x7fde5815ae50 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.170+0000 7fde4e7fc640 1 -- 172.21.15.27:0/4261795624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54003850 con 0x7fde5815ae50 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.170+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde581f2a80 con 0x7fde5815ae50 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.171+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde580733c0 con 0x7fde5815ae50 2024-03-06T07:35:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.171+0000 7fde4e7fc640 1 -- 172.21.15.27:0/4261795624 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde540039f0 con 0x7fde5815ae50 2024-03-06T07:35:57.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.172+0000 7fde4e7fc640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde3c06b500 0x7fde3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:57.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.172+0000 7fde4e7fc640 1 -- 172.21.15.27:0/4261795624 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fde5402f030 con 0x7fde5815ae50 2024-03-06T07:35:57.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.172+0000 7fde5d25e640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde3c06b500 0x7fde3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:57.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.172+0000 7fde5d25e640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde3c06b500 0x7fde3c06d9c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fde580cde20 tx=0x7fde48008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.176+0000 7fde4e7fc640 1 -- 172.21.15.27:0/4261795624 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde54041070 con 0x7fde5815ae50 2024-03-06T07:35:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:57.401+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fde5804f410 con 0x7fde5815ae50 2024-03-06T07:35:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.038+0000 7fde4e7fc640 1 -- 172.21.15.27:0/4261795624 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v125) v1 ==== 104+0+0 (secure 0 0 0) 0x7fde5407b1f0 con 0x7fde5815ae50 2024-03-06T07:35:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde3c06b500 msgr2=0x7fde3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde3c06b500 0x7fde3c06d9c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fde580cde20 tx=0x7fde48008040 comp rx=0 tx=0).stop 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde5815ae50 msgr2=0x7fde581f5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde5815ae50 0x7fde581f5d50 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fde54034040 tx=0x7fde540046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 shutdown_connections 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fde3c06b500 0x7fde3c06d9c0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde5815b690 0x7fde580ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fde5815b230 0x7fde58154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 --2- 172.21.15.27:0/4261795624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde5815ae50 0x7fde581f5d50 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 >> 172.21.15.27:0/4261795624 conn(0x7fde58066d00 msgr2=0x7fde580670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 shutdown_connections 2024-03-06T07:35:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.042+0000 7fde5fcc3640 1 -- 172.21.15.27:0/4261795624 wait complete. 2024-03-06T07:35:58.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:35:58.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.210+0000 7f63a9dc7640 1 Processor -- start 2024-03-06T07:35:58.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.210+0000 7f63a9dc7640 1 -- start start 2024-03-06T07:35:58.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.210+0000 7f63a9dc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63a415b130 0x7f63a415b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.210+0000 7f63a9dc7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63a40ce900 0x7f63a40ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.210+0000 7f63a9dc7640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a415bc40 con 0x7f63a4144640 2024-03-06T07:35:58.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a9dc7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a406df70 con 0x7f63a40ce900 2024-03-06T07:35:58.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a9dc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a415bf20 con 0x7f63a415b130 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a37fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f63a4144640 0x7f63a406e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51912/0 (socket says 172.21.15.27:51912) 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a37fe640 1 -- 172.21.15.27:0/1620952874 learned_addr learned my addr 172.21.15.27:0/1620952874 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a3fff640 1 --2- 172.21.15.27:0/1620952874 >> v2:172.21.15.27:3300/0 conn(0x7f63a40ce900 0x7f63a40ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a2ffd640 1 --2- 172.21.15.27:0/1620952874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63a415b130 0x7f63a415b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a27fc640 1 -- 172.21.15.27:0/1620952874 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620834108 0 0) 0x7f63a415bc40 con 0x7f63a4144640 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a27fc640 1 -- 172.21.15.27:0/1620952874 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c003520 con 0x7f63a4144640 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a3fff640 1 -- 172.21.15.27:0/1620952874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63a415b130 msgr2=0x7f63a415b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a3fff640 1 --2- 172.21.15.27:0/1620952874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63a415b130 0x7f63a415b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a3fff640 1 -- 172.21.15.27:0/1620952874 >> v1:172.21.15.27:6789/0 conn(0x7f63a4144640 legacy=0x7f63a406e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.211+0000 7f63a3fff640 1 -- 172.21.15.27:0/1620952874 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a406dc00 con 0x7f63a40ce900 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a3fff640 1 --2- 172.21.15.27:0/1620952874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a40ce900 0x7f63a40ced90 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f639c009d90 tx=0x7f639c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c027451a23b99d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a27fc640 1 -- 172.21.15.27:0/1620952874 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c039070 con 0x7f63a40ce900 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a27fc640 1 -- 172.21.15.27:0/1620952874 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c02cc50 con 0x7f63a40ce900 2024-03-06T07:35:58.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a27fc640 1 -- 172.21.15.27:0/1620952874 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c034440 con 0x7f63a40ce900 2024-03-06T07:35:58.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a9dc7640 1 -- 172.21.15.27:0/1620952874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a40ce900 msgr2=0x7f63a40ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a9dc7640 1 --2- 172.21.15.27:0/1620952874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a40ce900 0x7f63a40ced90 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f639c009d90 tx=0x7f639c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a9dc7640 1 -- 172.21.15.27:0/1620952874 shutdown_connections 2024-03-06T07:35:58.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a9dc7640 1 --2- 172.21.15.27:0/1620952874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a40ce900 0x7f63a40ced90 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a9dc7640 1 --2- 172.21.15.27:0/1620952874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63a415b130 0x7f63a415b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.212+0000 7f63a9dc7640 1 -- 172.21.15.27:0/1620952874 >> 172.21.15.27:0/1620952874 conn(0x7f63a4066d00 msgr2=0x7f63a4067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:58.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.213+0000 7f63a9dc7640 1 -- 172.21.15.27:0/1620952874 shutdown_connections 2024-03-06T07:35:58.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.213+0000 7f63a9dc7640 1 -- 172.21.15.27:0/1620952874 wait complete. 2024-03-06T07:35:58.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.214+0000 7f63a9dc7640 1 Processor -- start 2024-03-06T07:35:58.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.214+0000 7f63a9dc7640 1 -- start start 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a9dc7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63a40ce900 0x7f63a40ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a9dc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a4144640 0x7f63a40ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a9dc7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63a415b130 0x7f63a413bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a9dc7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a4070c20 con 0x7f63a4144640 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a9dc7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a4070aa0 con 0x7f63a40ce900 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a9dc7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40b36d0 con 0x7f63a415b130 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a4144640 0x7f63a40ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a4144640 0x7f63a40ae5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41956/0 (socket says 172.21.15.27:41956) 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2288534413 learned_addr learned my addr 172.21.15.27:0/2288534413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a37fe640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63a40ce900 0x7f63a40ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63a415b130 msgr2=0x7f63a413bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a2ffd640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63a415b130 0x7f63a413bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63a40ce900 msgr2=0x7f63a40ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a2ffd640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63a40ce900 0x7f63a40ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a2ffd640 1 -- 172.21.15.27:0/2288534413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40aeb30 con 0x7f63a4144640 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a2ffd640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a4144640 0x7f63a40ae5f0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f639800d5f0 tx=0x7f639800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f638bfff640 1 -- 172.21.15.27:0/2288534413 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63980125f0 con 0x7f63a4144640 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f638bfff640 1 -- 172.21.15.27:0/2288534413 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6398019070 con 0x7f63a4144640 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a41f45c0 con 0x7f63a4144640 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f638bfff640 1 -- 172.21.15.27:0/2288534413 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63980129d0 con 0x7f63a4144640 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.215+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a41f4b80 con 0x7f63a4144640 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.216+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a41441a0 con 0x7f63a4144640 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.217+0000 7f638bfff640 1 -- 172.21.15.27:0/2288534413 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6398012b70 con 0x7f63a4144640 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.217+0000 7f638bfff640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f639006b500 0x7f639006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.217+0000 7f638bfff640 1 -- 172.21.15.27:0/2288534413 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f639808a6a0 con 0x7f63a4144640 2024-03-06T07:35:58.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.217+0000 7f63a37fe640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f639006b500 0x7f639006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.218+0000 7f63a37fe640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f639006b500 0x7f639006d9c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f63a4143990 tx=0x7f6394009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:58.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.221+0000 7f638bfff640 1 -- 172.21.15.27:0/2288534413 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63a41441a0 con 0x7f63a4144640 2024-03-06T07:35:58.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:58.403+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f63a40b2800 con 0x7f63a4144640 2024-03-06T07:35:59.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.329+0000 7f638bfff640 1 -- 172.21.15.27:0/2288534413 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v151) v1 ==== 181+0+0 (secure 0 0 0) 0x7f63a40b2800 con 0x7f63a4144640 2024-03-06T07:35:59.331 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:35:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.332+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f639006b500 msgr2=0x7f639006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.332+0000 7f63a9dc7640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f639006b500 0x7f639006d9c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f63a4143990 tx=0x7f6394009040 comp rx=0 tx=0).stop 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.332+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a4144640 msgr2=0x7f63a40ae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.332+0000 7f63a9dc7640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a4144640 0x7f63a40ae5f0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f639800d5f0 tx=0x7f639800dab0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.333+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 shutdown_connections 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.333+0000 7f63a9dc7640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f639006b500 0x7f639006d9c0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.333+0000 7f63a9dc7640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63a415b130 0x7f63a413bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.333+0000 7f63a9dc7640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a4144640 0x7f63a40ae5f0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.333+0000 7f63a9dc7640 1 --2- 172.21.15.27:0/2288534413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63a40ce900 0x7f63a40ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.333+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 >> 172.21.15.27:0/2288534413 conn(0x7f63a4066d00 msgr2=0x7f63a406e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.333+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 shutdown_connections 2024-03-06T07:35:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.333+0000 7f63a9dc7640 1 -- 172.21.15.27:0/2288534413 wait complete. 2024-03-06T07:35:59.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:35:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.466+0000 7f09c56c9640 1 Processor -- start 2024-03-06T07:35:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.466+0000 7f09c56c9640 1 -- start start 2024-03-06T07:35:59.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09c56c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c00b5840 0x7f09c00b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09c56c9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09c015dfb0 0x7f09c00b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09c56c9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c006f070 con 0x7f09c015dbd0 2024-03-06T07:35:59.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09c56c9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c0071940 con 0x7f09c015dfb0 2024-03-06T07:35:59.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09c56c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c00b5db0 con 0x7f09c00b5840 2024-03-06T07:35:59.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09bf7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09c015dfb0 0x7f09c00b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09b7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c00b5840 0x7f09c00b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09beffd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f09c015dbd0 0x7f09c015ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:35810/0 (socket says 172.21.15.27:35810) 2024-03-06T07:35:59.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09beffd640 1 -- 172.21.15.27:0/2348380832 learned_addr learned my addr 172.21.15.27:0/2348380832 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:35:59.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09be7fc640 1 -- 172.21.15.27:0/2348380832 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939123753 0 0) 0x7f09c006f070 con 0x7f09c015dbd0 2024-03-06T07:35:59.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09be7fc640 1 -- 172.21.15.27:0/2348380832 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ac003520 con 0x7f09c015dbd0 2024-03-06T07:35:59.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09b7fff640 1 -- 172.21.15.27:0/2348380832 >> v2:172.21.15.69:3300/0 conn(0x7f09c015dfb0 msgr2=0x7f09c00b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:59.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09b7fff640 1 --2- 172.21.15.27:0/2348380832 >> v2:172.21.15.69:3300/0 conn(0x7f09c015dfb0 0x7f09c00b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09b7fff640 1 -- 172.21.15.27:0/2348380832 >> v1:172.21.15.69:6789/0 conn(0x7f09c015dbd0 legacy=0x7f09c015ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09b7fff640 1 -- 172.21.15.27:0/2348380832 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0071c00 con 0x7f09c00b5840 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.467+0000 7f09bf7fe640 1 --2- 172.21.15.27:0/2348380832 >> v2:172.21.15.69:3300/0 conn(0x7f09c015dfb0 0x7f09c00b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.468+0000 7f09b7fff640 1 --2- 172.21.15.27:0/2348380832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c00b5840 0x7f09c00b39b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f09a80097a0 tx=0x7f09a802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7e9ce9d4e531bde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.468+0000 7f09be7fc640 1 -- 172.21.15.27:0/2348380832 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8035020 con 0x7f09c00b5840 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09be7fc640 1 -- 172.21.15.27:0/2348380832 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8031810 con 0x7f09c00b5840 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09be7fc640 1 -- 172.21.15.27:0/2348380832 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8031bc0 con 0x7f09c00b5840 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09c56c9640 1 -- 172.21.15.27:0/2348380832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c00b5840 msgr2=0x7f09c00b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09c56c9640 1 --2- 172.21.15.27:0/2348380832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c00b5840 0x7f09c00b39b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f09a80097a0 tx=0x7f09a802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09c56c9640 1 -- 172.21.15.27:0/2348380832 shutdown_connections 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09c56c9640 1 --2- 172.21.15.27:0/2348380832 >> v2:172.21.15.69:3300/0 conn(0x7f09c015dfb0 0x7f09c00b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09c56c9640 1 --2- 172.21.15.27:0/2348380832 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c00b5840 0x7f09c00b39b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09c56c9640 1 -- 172.21.15.27:0/2348380832 >> 172.21.15.27:0/2348380832 conn(0x7f09c00b26b0 msgr2=0x7f09c00b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09c56c9640 1 -- 172.21.15.27:0/2348380832 shutdown_connections 2024-03-06T07:35:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.469+0000 7f09c56c9640 1 -- 172.21.15.27:0/2348380832 wait complete. 2024-03-06T07:35:59.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.470+0000 7f09c56c9640 1 Processor -- start 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.470+0000 7f09c56c9640 1 -- start start 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09c56c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c00b5840 0x7f09c0156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09c56c9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c015dbd0 0x7f09c01575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09c56c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c015dfb0 0x7f09c015ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09c56c9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c00726e0 con 0x7f09c015dfb0 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09c56c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c00721c0 con 0x7f09c00b5840 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09c56c9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0072860 con 0x7f09c015dbd0 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c015dfb0 0x7f09c015ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09bf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c015dfb0 0x7f09c015ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41960/0 (socket says 172.21.15.27:41960) 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09bf7fe640 1 -- 172.21.15.27:0/3272221493 learned_addr learned my addr 172.21.15.27:0/3272221493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09beffd640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c00b5840 0x7f09c0156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09b7fff640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c015dbd0 0x7f09c01575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09bf7fe640 1 -- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c015dbd0 msgr2=0x7f09c01575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09bf7fe640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c015dbd0 0x7f09c01575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09bf7fe640 1 -- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c00b5840 msgr2=0x7f09c0156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.471+0000 7f09bf7fe640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c00b5840 0x7f09c0156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.472+0000 7f09bf7fe640 1 -- 172.21.15.27:0/3272221493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c015fb10 con 0x7f09c015dfb0 2024-03-06T07:35:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.472+0000 7f09bf7fe640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c015dfb0 0x7f09c015ee20 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f09b800efc0 tx=0x7f09b8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.472+0000 7f09b77fe640 1 -- 172.21.15.27:0/3272221493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8018030 con 0x7f09c015dfb0 2024-03-06T07:35:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.472+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c0156170 con 0x7f09c015dfb0 2024-03-06T07:35:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.472+0000 7f09b77fe640 1 -- 172.21.15.27:0/3272221493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b800cde0 con 0x7f09c015dfb0 2024-03-06T07:35:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.472+0000 7f09b77fe640 1 -- 172.21.15.27:0/3272221493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8003bb0 con 0x7f09c015dfb0 2024-03-06T07:35:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.472+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c0156730 con 0x7f09c015dfb0 2024-03-06T07:35:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.473+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c0073380 con 0x7f09c015dfb0 2024-03-06T07:35:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.473+0000 7f09b77fe640 1 -- 172.21.15.27:0/3272221493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09b801b070 con 0x7f09c015dfb0 2024-03-06T07:35:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.474+0000 7f09b77fe640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09a006b4d0 0x7f09a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.474+0000 7f09b77fe640 1 -- 172.21.15.27:0/3272221493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5472+0+0 (secure 0 0 0) 0x7f09b808abe0 con 0x7f09c015dfb0 2024-03-06T07:35:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.474+0000 7f09beffd640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09a006b4d0 0x7f09a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.474+0000 7f09beffd640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09a006b4d0 0x7f09a006d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f09b0007fd0 tx=0x7f09b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:59.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.478+0000 7f09b77fe640 1 -- 172.21.15.27:0/3272221493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09b8010040 con 0x7f09c015dfb0 2024-03-06T07:35:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:35:59.659+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f09c004f410 con 0x7f09c015dfb0 2024-03-06T07:36:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.336+0000 7f09b77fe640 1 -- 172.21.15.27:0/3272221493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v152) v1 ==== 169+0+0 (secure 0 0 0) 0x7f09b8059ad0 con 0x7f09c015dfb0 2024-03-06T07:36:00.338 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-03-06T07:36:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.338+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09a006b4d0 msgr2=0x7f09a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.338+0000 7f09c56c9640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09a006b4d0 0x7f09a006d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f09b0007fd0 tx=0x7f09b0009040 comp rx=0 tx=0).stop 2024-03-06T07:36:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.338+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c015dfb0 msgr2=0x7f09c015ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.338+0000 7f09c56c9640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c015dfb0 0x7f09c015ee20 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f09b800efc0 tx=0x7f09b8009f90 comp rx=0 tx=0).stop 2024-03-06T07:36:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.339+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 shutdown_connections 2024-03-06T07:36:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.339+0000 7f09c56c9640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f09a006b4d0 0x7f09a006d990 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.339+0000 7f09c56c9640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09c015dfb0 0x7f09c015ee20 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.339+0000 7f09c56c9640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f09c015dbd0 0x7f09c01575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.339+0000 7f09c56c9640 1 --2- 172.21.15.27:0/3272221493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c00b5840 0x7f09c0156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.339+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 >> 172.21.15.27:0/3272221493 conn(0x7f09c00b26b0 msgr2=0x7f09c006e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.339+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 shutdown_connections 2024-03-06T07:36:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.339+0000 7f09c56c9640 1 -- 172.21.15.27:0/3272221493 wait complete. 2024-03-06T07:36:00.357 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:36:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.467+0000 7f0c49deb640 1 Processor -- start 2024-03-06T07:36:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.467+0000 7f0c49deb640 1 -- start start 2024-03-06T07:36:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c49deb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0c440b41f0 0x7f0c4406bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c49deb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4406c130 0x7f0c440cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c49deb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4415e6d0 con 0x7f0c4415e2f0 2024-03-06T07:36:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c49deb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c440d0a10 con 0x7f0c440b41f0 2024-03-06T07:36:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c49deb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c440ccd00 con 0x7f0c4406c130 2024-03-06T07:36:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c437fe640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0c4415e2f0 0x7f0c4406b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51924/0 (socket says 172.21.15.27:51924) 2024-03-06T07:36:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c437fe640 1 -- 172.21.15.27:0/3119491770 learned_addr learned my addr 172.21.15.27:0/3119491770 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c42ffd640 1 --2- 172.21.15.27:0/3119491770 >> v2:172.21.15.69:3300/0 conn(0x7f0c440b41f0 0x7f0c4406bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c43fff640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4406c130 0x7f0c440cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985379500 0 0) 0x7f0c4415e6d0 con 0x7f0c4415e2f0 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.468+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2c003520 con 0x7f0c4415e2f0 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899859422 0 0) 0x7f0c2c003520 con 0x7f0c4415e2f0 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4406c130 msgr2=0x7f0c440cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4406c130 0x7f0c440cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 >> v2:172.21.15.69:3300/0 conn(0x7f0c440b41f0 msgr2=0x7f0c4406bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 --2- 172.21.15.27:0/3119491770 >> v2:172.21.15.69:3300/0 conn(0x7f0c440b41f0 0x7f0c4406bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c440ccfe0 con 0x7f0c4415e2f0 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f0c34001670 con 0x7f0c4415e2f0 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 >> v1:172.21.15.27:6789/0 conn(0x7f0c4415e2f0 legacy=0x7f0c4406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c2c0039b0 0x7f0c2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4406c130 0x7f0c2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c2c004d40 0x7f0c2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c440d0a10 con 0x7f0c2c0039b0 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c440ccd00 con 0x7f0c2c004d40 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c00b9c0 con 0x7f0c4406c130 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c437fe640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c2c0039b0 0x7f0c2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c42ffd640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4406c130 0x7f0c2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c43fff640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c2c004d40 0x7f0c2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c437fe640 1 -- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4406c130 msgr2=0x7f0c2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c437fe640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c4406c130 0x7f0c2c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c437fe640 1 -- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c2c004d40 msgr2=0x7f0c2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c437fe640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c2c004d40 0x7f0c2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.469+0000 7f0c437fe640 1 -- 172.21.15.27:0/3119491770 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c003520 con 0x7f0c2c0039b0 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.470+0000 7f0c437fe640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c2c0039b0 0x7f0c2c003dd0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c210 tx=0x7f0c2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c414dae95cd3a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.470+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34007d50 con 0x7f0c2c0039b0 2024-03-06T07:36:00.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.470+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c34030070 con 0x7f0c2c0039b0 2024-03-06T07:36:00.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.470+0000 7f0c427fc640 1 -- 172.21.15.27:0/3119491770 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3402cd80 con 0x7f0c2c0039b0 2024-03-06T07:36:00.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.471+0000 7f0c49deb640 1 -- 172.21.15.27:0/3119491770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c2c0039b0 msgr2=0x7f0c2c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.471+0000 7f0c49deb640 1 --2- 172.21.15.27:0/3119491770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c2c0039b0 0x7f0c2c003dd0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c210 tx=0x7f0c2c005830 comp rx=0 tx=0).stop 2024-03-06T07:36:00.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.471+0000 7f0c43fff640 1 -- 172.21.15.27:0/3119491770 reap_dead start 2024-03-06T07:36:00.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.471+0000 7f0c49deb640 1 -- 172.21.15.27:0/3119491770 shutdown_connections 2024-03-06T07:36:00.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.471+0000 7f0c49deb640 1 -- 172.21.15.27:0/3119491770 >> 172.21.15.27:0/3119491770 conn(0x7f0c440b36d0 msgr2=0x7f0c440b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:00.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.471+0000 7f0c49deb640 1 -- 172.21.15.27:0/3119491770 shutdown_connections 2024-03-06T07:36:00.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.471+0000 7f0c49deb640 1 -- 172.21.15.27:0/3119491770 wait complete. 2024-03-06T07:36:00.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.472+0000 7f0c49deb640 1 Processor -- start 2024-03-06T07:36:00.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.472+0000 7f0c49deb640 1 -- start start 2024-03-06T07:36:00.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c49deb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c44157990 0x7f0c440b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c49deb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c440c7050 0x7f0c440c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c49deb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c440cabd0 0x7f0c440cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c49deb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c440cdac0 con 0x7f0c44157990 2024-03-06T07:36:00.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c49deb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c440cd5a0 con 0x7f0c440c7050 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c49deb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c440cdc40 con 0x7f0c440cabd0 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c44157990 0x7f0c440b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c437fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c44157990 0x7f0c440b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41974/0 (socket says 172.21.15.27:41974) 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c437fe640 1 -- 172.21.15.27:0/1431012746 learned_addr learned my addr 172.21.15.27:0/1431012746 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c43fff640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c440cabd0 0x7f0c440cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c42ffd640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c440c7050 0x7f0c440c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c437fe640 1 -- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c440cabd0 msgr2=0x7f0c440cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c437fe640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c440cabd0 0x7f0c440cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c437fe640 1 -- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c440c7050 msgr2=0x7f0c440c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c437fe640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c440c7050 0x7f0c440c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c437fe640 1 -- 172.21.15.27:0/1431012746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c440c74b0 con 0x7f0c44157990 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c42ffd640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c440c7050 0x7f0c440c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c43fff640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c440cabd0 0x7f0c440cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.473+0000 7f0c437fe640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c44157990 0x7f0c440b45d0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f0c3402c510 tx=0x7f0c34001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.474+0000 7f0c2bfff640 1 -- 172.21.15.27:0/1431012746 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34001c90 con 0x7f0c44157990 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.474+0000 7f0c2bfff640 1 -- 172.21.15.27:0/1431012746 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c34004cb0 con 0x7f0c44157990 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.474+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c440c8bd0 con 0x7f0c44157990 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.474+0000 7f0c2bfff640 1 -- 172.21.15.27:0/1431012746 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34007d50 con 0x7f0c44157990 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.474+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c440c7a30 con 0x7f0c44157990 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.475+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4409fd30 con 0x7f0c44157990 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.475+0000 7f0c2bfff640 1 -- 172.21.15.27:0/1431012746 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c34048020 con 0x7f0c44157990 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.476+0000 7f0c2bfff640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3006b500 0x7f0c3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.476+0000 7f0c2bfff640 1 -- 172.21.15.27:0/1431012746 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0c340ab300 con 0x7f0c44157990 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.476+0000 7f0c42ffd640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3006b500 0x7f0c3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.476+0000 7f0c42ffd640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3006b500 0x7f0c3006d9c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0c44074420 tx=0x7f0c38008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.480+0000 7f0c2bfff640 1 -- 172.21.15.27:0/1431012746 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c3407a420 con 0x7f0c44157990 2024-03-06T07:36:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.662+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0c4404f410 con 0x7f0c44157990 2024-03-06T07:36:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.666+0000 7f0c2bfff640 1 -- 172.21.15.27:0/1431012746 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0c34079d20 con 0x7f0c44157990 2024-03-06T07:36:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3006b500 msgr2=0x7f0c3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3006b500 0x7f0c3006d9c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0c44074420 tx=0x7f0c38008040 comp rx=0 tx=0).stop 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c44157990 msgr2=0x7f0c440b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c44157990 0x7f0c440b45d0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f0c3402c510 tx=0x7f0c34001ab0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 shutdown_connections 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f0c3006b500 0x7f0c3006d9c0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0c440cabd0 0x7f0c440cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c440c7050 0x7f0c440c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 --2- 172.21.15.27:0/1431012746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c44157990 0x7f0c440b45d0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 >> 172.21.15.27:0/1431012746 conn(0x7f0c440b36d0 msgr2=0x7f0c440cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 shutdown_connections 2024-03-06T07:36:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.668+0000 7f0c49deb640 1 -- 172.21.15.27:0/1431012746 wait complete. 2024-03-06T07:36:00.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:36:00.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.786+0000 7fc798bf7640 1 Processor -- start 2024-03-06T07:36:00.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.786+0000 7fc798bf7640 1 -- start start 2024-03-06T07:36:00.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.787+0000 7fc798bf7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 0x7fc7940b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.787+0000 7fc798bf7640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7940d3400 0x7fc794071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.787+0000 7fc798bf7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7940b2b50 0x7fc7940d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.787+0000 7fc798bf7640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7940d2f70 con 0x7fc7940d3400 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.787+0000 7fc798bf7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7940d4be0 con 0x7fc7940b2b50 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.787+0000 7fc798bf7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc794071e60 con 0x7fc794071540 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc791d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7940d3400 0x7fc794071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc791d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7940d3400 0x7fc794071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41990/0 (socket says 172.21.15.27:41990) 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc791d9b640 1 -- 172.21.15.27:0/3795040309 learned_addr learned my addr 172.21.15.27:0/3795040309 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc79259c640 1 --2- 172.21.15.27:0/3795040309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 0x7fc7940b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc792d9d640 1 --2- 172.21.15.27:0/3795040309 >> v2:172.21.15.69:3300/0 conn(0x7fc7940b2b50 0x7fc7940d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc791d9b640 1 -- 172.21.15.27:0/3795040309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 msgr2=0x7fc7940b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc791d9b640 1 --2- 172.21.15.27:0/3795040309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 0x7fc7940b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc791d9b640 1 -- 172.21.15.27:0/3795040309 >> v2:172.21.15.69:3300/0 conn(0x7fc7940b2b50 msgr2=0x7fc7940d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc791d9b640 1 --2- 172.21.15.27:0/3795040309 >> v2:172.21.15.69:3300/0 conn(0x7fc7940b2b50 0x7fc7940d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.788+0000 7fc791d9b640 1 -- 172.21.15.27:0/3795040309 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940ce610 con 0x7fc7940d3400 2024-03-06T07:36:00.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc791d9b640 1 --2- 172.21.15.27:0/3795040309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 0x7fc794071920 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fc79406c230 tx=0x7fc77c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0a512f4287d92e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc79159a640 1 -- 172.21.15.27:0/3795040309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c035020 con 0x7fc7940d3400 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc79159a640 1 -- 172.21.15.27:0/3795040309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77c031870 con 0x7fc7940d3400 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc79159a640 1 -- 172.21.15.27:0/3795040309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c031c40 con 0x7fc7940d3400 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc798bf7640 1 -- 172.21.15.27:0/3795040309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 msgr2=0x7fc794071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc798bf7640 1 --2- 172.21.15.27:0/3795040309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 0x7fc794071920 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fc79406c230 tx=0x7fc77c02b910 comp rx=0 tx=0).stop 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc798bf7640 1 -- 172.21.15.27:0/3795040309 shutdown_connections 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc798bf7640 1 --2- 172.21.15.27:0/3795040309 >> v2:172.21.15.69:3300/0 conn(0x7fc7940b2b50 0x7fc7940d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc798bf7640 1 --2- 172.21.15.27:0/3795040309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 0x7fc794071920 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc798bf7640 1 --2- 172.21.15.27:0/3795040309 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 0x7fc7940b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc798bf7640 1 -- 172.21.15.27:0/3795040309 >> 172.21.15.27:0/3795040309 conn(0x7fc7940d3ca0 msgr2=0x7fc7940d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.789+0000 7fc798bf7640 1 -- 172.21.15.27:0/3795040309 shutdown_connections 2024-03-06T07:36:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.790+0000 7fc798bf7640 1 -- 172.21.15.27:0/3795040309 wait complete. 2024-03-06T07:36:00.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.791+0000 7fc798bf7640 1 Processor -- start 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.791+0000 7fc798bf7640 1 -- start start 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc798bf7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 0x7fc79414c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc798bf7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7940b2b50 0x7fc7940b5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc798bf7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 0x7fc79415ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc798bf7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940ce8f0 con 0x7fc7940d3400 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc798bf7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940d2c00 con 0x7fc7940b2b50 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc798bf7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79415e370 con 0x7fc794071540 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc792d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 0x7fc79415ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc792d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 0x7fc79415ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41998/0 (socket says 172.21.15.27:41998) 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc792d9d640 1 -- 172.21.15.27:0/2391061806 learned_addr learned my addr 172.21.15.27:0/2391061806 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc79259c640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 0x7fc79414c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc791d9b640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7940b2b50 0x7fc7940b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc792d9d640 1 -- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 msgr2=0x7fc79414c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc792d9d640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 0x7fc79414c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc792d9d640 1 -- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7940b2b50 msgr2=0x7fc7940b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc792d9d640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7940b2b50 0x7fc7940b5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc792d9d640 1 -- 172.21.15.27:0/2391061806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940b3f60 con 0x7fc7940d3400 2024-03-06T07:36:00.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc792d9d640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 0x7fc79415ac80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fc78400d800 tx=0x7fc78400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc78affd640 1 -- 172.21.15.27:0/2391061806 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78400b6c0 con 0x7fc7940d3400 2024-03-06T07:36:00.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc78affd640 1 -- 172.21.15.27:0/2391061806 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc784019070 con 0x7fc7940d3400 2024-03-06T07:36:00.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc78affd640 1 -- 172.21.15.27:0/2391061806 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78400bac0 con 0x7fc7940d3400 2024-03-06T07:36:00.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.792+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7940b4240 con 0x7fc7940d3400 2024-03-06T07:36:00.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.793+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7940b4800 con 0x7fc7940d3400 2024-03-06T07:36:00.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.794+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7940cfcf0 con 0x7fc7940d3400 2024-03-06T07:36:00.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.794+0000 7fc78affd640 1 -- 172.21.15.27:0/2391061806 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc78400bc60 con 0x7fc7940d3400 2024-03-06T07:36:00.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.794+0000 7fc78affd640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc77406b4d0 0x7fc77406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.795+0000 7fc78affd640 1 -- 172.21.15.27:0/2391061806 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc78408a1f0 con 0x7fc7940d3400 2024-03-06T07:36:00.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.795+0000 7fc79259c640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc77406b4d0 0x7fc77406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.795+0000 7fc79259c640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc77406b4d0 0x7fc77406d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc78c003aa0 tx=0x7fc78c009830 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:00.799+0000 7fc78affd640 1 -- 172.21.15.27:0/2391061806 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7940cfcf0 con 0x7fc7940d3400 2024-03-06T07:36:01.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.054+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7fc794068700 con 0x7fc7940d3400 2024-03-06T07:36:01.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.337+0000 7fc78affd640 1 -- 172.21.15.27:0/2391061806 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v699) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc784058d50 con 0x7fc7940d3400 2024-03-06T07:36:01.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.339+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc77406b4d0 msgr2=0x7fc77406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.339+0000 7fc798bf7640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc77406b4d0 0x7fc77406d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc78c003aa0 tx=0x7fc78c009830 comp rx=0 tx=0).stop 2024-03-06T07:36:01.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.339+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 msgr2=0x7fc79415ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.339+0000 7fc798bf7640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 0x7fc79415ac80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fc78400d800 tx=0x7fc78400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.340+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 shutdown_connections 2024-03-06T07:36:01.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.340+0000 7fc798bf7640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc77406b4d0 0x7fc77406d990 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.340+0000 7fc798bf7640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7940d3400 0x7fc79415ac80 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.340+0000 7fc798bf7640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7940b2b50 0x7fc7940b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.340+0000 7fc798bf7640 1 --2- 172.21.15.27:0/2391061806 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc794071540 0x7fc79414c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.340+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 >> 172.21.15.27:0/2391061806 conn(0x7fc7940d3ca0 msgr2=0x7fc7940ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:01.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.340+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 shutdown_connections 2024-03-06T07:36:01.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:01.340+0000 7fc798bf7640 1 -- 172.21.15.27:0/2391061806 wait complete. 2024-03-06T07:36:01.353 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays ... ERROR 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner: 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:36:01.354 INFO:tasks.cephfs_test_runner: 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner:Ran 9 tests in 764.438s 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner: 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner: 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:36:01.355 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:36:01.356 INFO:tasks.cephfs_test_runner: 2024-03-06T07:36:01.356 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:36:01.547 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=87083e1b34594069b6395246351d2017 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:36:01.548 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-06T07:36:01.582 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-06T07:36:01.590 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-06T07:36:01.591 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:36:01.591 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:01.611 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:36:01.611 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-06T07:36:01.611 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:01.667 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=309 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:36:01.667 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:36:01.667 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:01.686 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:36:01.686 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-06T07:36:01.686 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:01.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=984 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:36:01.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:01.744 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-03-06T07:36:01.801 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-06T07:36:01.801 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:36:01.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:01.802 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:36:01.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:01.871 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:36:01.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:01.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-03-06T07:36:02.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:02.040 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-06T07:36:02.056 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:36:02.057 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:36:02.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:02.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:36:02.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:02.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:36:02.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:02.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-03-06T07:36:02.305 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-06T07:36:02.315 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:36:02.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:36:02.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.423+0000 7f46acc95640 1 Processor -- start 2024-03-06T07:36:02.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.423+0000 7f46acc95640 1 -- start start 2024-03-06T07:36:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46acc95640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f46a8158520 0x7f46a806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46acc95640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 0x7f46a80ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46acc95640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a80af080 con 0x7f46a80aec40 2024-03-06T07:36:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46acc95640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a806e1c0 con 0x7f46a8158520 2024-03-06T07:36:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46acc95640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8071b70 con 0x7f46a806dd60 2024-03-06T07:36:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46a6d9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f46a80aec40 0x7f46a815e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51932/0 (socket says 172.21.15.27:51932) 2024-03-06T07:36:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46a6d9d640 1 -- 172.21.15.27:0/2352458276 learned_addr learned my addr 172.21.15.27:0/2352458276 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46a659c640 1 --2- 172.21.15.27:0/2352458276 >> v2:172.21.15.69:3300/0 conn(0x7f46a8158520 0x7f46a806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f469ffff640 1 --2- 172.21.15.27:0/2352458276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 0x7f46a80ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46a5d9b640 1 -- 172.21.15.27:0/2352458276 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864396666 0 0) 0x7f46a80af080 con 0x7f46a80aec40 2024-03-06T07:36:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46a5d9b640 1 -- 172.21.15.27:0/2352458276 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4694003520 con 0x7f46a80aec40 2024-03-06T07:36:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46a659c640 1 -- 172.21.15.27:0/2352458276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 msgr2=0x7f46a80ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46a659c640 1 --2- 172.21.15.27:0/2352458276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 0x7f46a80ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46a659c640 1 -- 172.21.15.27:0/2352458276 >> v1:172.21.15.27:6789/0 conn(0x7f46a80aec40 legacy=0x7f46a815e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f46a659c640 1 -- 172.21.15.27:0/2352458276 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a8071240 con 0x7f46a8158520 2024-03-06T07:36:02.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.424+0000 7f469ffff640 1 --2- 172.21.15.27:0/2352458276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 0x7f46a80ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:02.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.425+0000 7f46a659c640 1 --2- 172.21.15.27:0/2352458276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46a8158520 0x7f46a806d820 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f46980097a0 tx=0x7f469802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c54819406bc6a8ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.425+0000 7f46a5d9b640 1 -- 172.21.15.27:0/2352458276 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4698035020 con 0x7f46a8158520 2024-03-06T07:36:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.425+0000 7f46a5d9b640 1 -- 172.21.15.27:0/2352458276 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4698031830 con 0x7f46a8158520 2024-03-06T07:36:02.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.425+0000 7f46a5d9b640 1 -- 172.21.15.27:0/2352458276 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4698031be0 con 0x7f46a8158520 2024-03-06T07:36:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.426+0000 7f46acc95640 1 -- 172.21.15.27:0/2352458276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46a8158520 msgr2=0x7f46a806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.426+0000 7f46acc95640 1 --2- 172.21.15.27:0/2352458276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46a8158520 0x7f46a806d820 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f46980097a0 tx=0x7f469802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.426+0000 7f46acc95640 1 -- 172.21.15.27:0/2352458276 shutdown_connections 2024-03-06T07:36:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.426+0000 7f46acc95640 1 --2- 172.21.15.27:0/2352458276 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 0x7f46a80ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.426+0000 7f46acc95640 1 --2- 172.21.15.27:0/2352458276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46a8158520 0x7f46a806d820 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.426+0000 7f46acc95640 1 -- 172.21.15.27:0/2352458276 >> 172.21.15.27:0/2352458276 conn(0x7f46a8066d00 msgr2=0x7f46a80670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.426+0000 7f46acc95640 1 -- 172.21.15.27:0/2352458276 shutdown_connections 2024-03-06T07:36:02.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.426+0000 7f46acc95640 1 -- 172.21.15.27:0/2352458276 wait complete. 2024-03-06T07:36:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.427+0000 7f46acc95640 1 Processor -- start 2024-03-06T07:36:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.427+0000 7f46acc95640 1 -- start start 2024-03-06T07:36:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f46acc95640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 0x7f46a81f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f46acc95640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a80aec40 0x7f46a8152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f46acc95640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46a8158520 0x7f46a815d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f46acc95640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a80b4650 con 0x7f46a80aec40 2024-03-06T07:36:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f46acc95640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8071800 con 0x7f46a8158520 2024-03-06T07:36:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f46acc95640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8072000 con 0x7f46a806dd60 2024-03-06T07:36:02.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f469ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a80aec40 0x7f46a8152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f469ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a80aec40 0x7f46a8152730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42006/0 (socket says 172.21.15.27:42006) 2024-03-06T07:36:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f469ffff640 1 -- 172.21.15.27:0/2649162882 learned_addr learned my addr 172.21.15.27:0/2649162882 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f46a6d9d640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46a8158520 0x7f46a815d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f46a659c640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 0x7f46a81f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f469ffff640 1 -- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 msgr2=0x7f46a81f56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.428+0000 7f469ffff640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 0x7f46a81f56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.429+0000 7f469ffff640 1 -- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46a8158520 msgr2=0x7f46a815d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.429+0000 7f469ffff640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46a8158520 0x7f46a815d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.429+0000 7f469ffff640 1 -- 172.21.15.27:0/2649162882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a81f6030 con 0x7f46a80aec40 2024-03-06T07:36:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.429+0000 7f469ffff640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a80aec40 0x7f46a8152730 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f469000ec10 tx=0x7f469000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.429+0000 7f469effd640 1 -- 172.21.15.27:0/2649162882 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690017070 con 0x7f46a80aec40 2024-03-06T07:36:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.429+0000 7f469effd640 1 -- 172.21.15.27:0/2649162882 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469000cc30 con 0x7f46a80aec40 2024-03-06T07:36:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.429+0000 7f469effd640 1 -- 172.21.15.27:0/2649162882 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690013470 con 0x7f46a80aec40 2024-03-06T07:36:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.429+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a81f6310 con 0x7f46a80aec40 2024-03-06T07:36:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.429+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a8152210 con 0x7f46a80aec40 2024-03-06T07:36:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.430+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a8067ff0 con 0x7f46a80aec40 2024-03-06T07:36:02.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.430+0000 7f469effd640 1 -- 172.21.15.27:0/2649162882 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f469001a070 con 0x7f46a80aec40 2024-03-06T07:36:02.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.431+0000 7f469effd640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f468806b4d0 0x7f468806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.431+0000 7f469effd640 1 -- 172.21.15.27:0/2649162882 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4690089da0 con 0x7f46a80aec40 2024-03-06T07:36:02.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.431+0000 7f46a659c640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f468806b4d0 0x7f468806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.432+0000 7f46a659c640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f468806b4d0 0x7f468806d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4698036000 tx=0x7f4698008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.435+0000 7f469effd640 1 -- 172.21.15.27:0/2649162882 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4690058900 con 0x7f46a80aec40 2024-03-06T07:36:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.614+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46a80b31f0 con 0x7f468806b4d0 2024-03-06T07:36:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.615+0000 7f469effd640 1 -- 172.21.15.27:0/2649162882 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13421 (secure 0 0 0) 0x7f469804c480 con 0x7f468806b4d0 2024-03-06T07:36:02.617 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:36:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:36:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f468806b4d0 msgr2=0x7f468806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f468806b4d0 0x7f468806d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4698036000 tx=0x7f4698008030 comp rx=0 tx=0).stop 2024-03-06T07:36:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a80aec40 msgr2=0x7f46a8152730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a80aec40 0x7f46a8152730 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f469000ec10 tx=0x7f469000c520 comp rx=0 tx=0).stop 2024-03-06T07:36:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 shutdown_connections 2024-03-06T07:36:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f468806b4d0 0x7f468806d990 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46a8158520 0x7f46a815d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a80aec40 0x7f46a8152730 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 --2- 172.21.15.27:0/2649162882 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46a806dd60 0x7f46a81f56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 >> 172.21.15.27:0/2649162882 conn(0x7f46a8066d00 msgr2=0x7f46a80af310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 shutdown_connections 2024-03-06T07:36:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.617+0000 7f46acc95640 1 -- 172.21.15.27:0/2649162882 wait complete. 2024-03-06T07:36:02.630 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":549,"stamp":"2024-03-06T07:36:02.180870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331964,"kb_used_data":15960,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754642756,"statfs":{"total":773094113280,"available":772754182144,"internally_reserved":0,"allocated":16343040,"data_stored":7834476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-37242,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-139,"num_write_kb":-84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001595"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":423,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-06T07:35:59.681896+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:35:59.681896+0000","last_peered":"2024-03-06T07:35:59.681896+0000","last_clean":"2024-03-06T07:35:59.681896+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:35:59.681896+0000","last_undegraded":"2024-03-06T07:35:59.681896+0000","last_fullsized":"2024-03-06T07:35:59.681896+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575012,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41944,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329896,"statfs":{"total":96636764160,"available":96593813504,"internally_reserved":0,"allocated":2502656,"data_stored":1428473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575013,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41348,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330492,"statfs":{"total":96636764160,"available":96594423808,"internally_reserved":0,"allocated":1892352,"data_stored":825419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41352,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330488,"statfs":{"total":96636764160,"available":96594419712,"internally_reserved":0,"allocated":1892352,"data_stored":830809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575012,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41332,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330508,"statfs":{"total":96636764160,"available":96594440192,"internally_reserved":0,"allocated":1875968,"data_stored":824691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41348,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330492,"statfs":{"total":96636764160,"available":96594423808,"internally_reserved":0,"allocated":1892352,"data_stored":830809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575012,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41348,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330492,"statfs":{"total":96636764160,"available":96594423808,"internally_reserved":0,"allocated":1892352,"data_stored":830809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41344,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330496,"statfs":{"total":96636764160,"available":96594427904,"internally_reserved":0,"allocated":1888256,"data_stored":825385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575013,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41948,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329892,"statfs":{"total":96636764160,"available":96593809408,"internally_reserved":0,"allocated":2506752,"data_stored":1438081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:36:02.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:36:02.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.737+0000 7fb0f3be4640 1 Processor -- start 2024-03-06T07:36:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.737+0000 7fb0f3be4640 1 -- start start 2024-03-06T07:36:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.738+0000 7fb0f3be4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0ec15df00 0x7fb0ec0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.738+0000 7fb0f3be4640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0ec0b95a0 0x7fb0ec0b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.738+0000 7fb0f3be4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec0b9a30 0x7fb0ec0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.738+0000 7fb0f3be4640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ec0cea30 con 0x7fb0ec0b95a0 2024-03-06T07:36:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.738+0000 7fb0f3be4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ec0d2af0 con 0x7fb0ec15df00 2024-03-06T07:36:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.738+0000 7fb0f3be4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec0cf3c0 con 0x7fb0ec0b9a30 2024-03-06T07:36:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.738+0000 7fb0f117f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0ec0b95a0 0x7fb0ec0b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.738+0000 7fb0f117f640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0ec0b95a0 0x7fb0ec0b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42008/0 (socket says 172.21.15.27:42008) 2024-03-06T07:36:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.738+0000 7fb0f117f640 1 -- 172.21.15.27:0/2863109371 learned_addr learned my addr 172.21.15.27:0/2863109371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.739+0000 7fb0f2181640 1 --2- 172.21.15.27:0/2863109371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec0b9a30 0x7fb0ec0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.739+0000 7fb0f1980640 1 --2- 172.21.15.27:0/2863109371 >> v2:172.21.15.69:3300/0 conn(0x7fb0ec15df00 0x7fb0ec0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.739+0000 7fb0f117f640 1 -- 172.21.15.27:0/2863109371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec0b9a30 msgr2=0x7fb0ec0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.739+0000 7fb0f117f640 1 --2- 172.21.15.27:0/2863109371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec0b9a30 0x7fb0ec0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.739+0000 7fb0f117f640 1 -- 172.21.15.27:0/2863109371 >> v2:172.21.15.69:3300/0 conn(0x7fb0ec15df00 msgr2=0x7fb0ec0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.739+0000 7fb0f117f640 1 --2- 172.21.15.27:0/2863109371 >> v2:172.21.15.69:3300/0 conn(0x7fb0ec15df00 0x7fb0ec0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.739+0000 7fb0f117f640 1 -- 172.21.15.27:0/2863109371 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec0cf0e0 con 0x7fb0ec0b95a0 2024-03-06T07:36:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f117f640 1 --2- 172.21.15.27:0/2863109371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec0b95a0 0x7fb0ec0b5700 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0d11b0 tx=0x7fb0e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922bcf832bdc6658 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f097e640 1 -- 172.21.15.27:0/2863109371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0035020 con 0x7fb0ec0b95a0 2024-03-06T07:36:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f097e640 1 -- 172.21.15.27:0/2863109371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e0031870 con 0x7fb0ec0b95a0 2024-03-06T07:36:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f097e640 1 -- 172.21.15.27:0/2863109371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0031c40 con 0x7fb0ec0b95a0 2024-03-06T07:36:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f3be4640 1 -- 172.21.15.27:0/2863109371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec0b95a0 msgr2=0x7fb0ec0b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/2863109371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec0b95a0 0x7fb0ec0b5700 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0d11b0 tx=0x7fb0e002b910 comp rx=0 tx=0).stop 2024-03-06T07:36:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f3be4640 1 -- 172.21.15.27:0/2863109371 shutdown_connections 2024-03-06T07:36:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/2863109371 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec0b9a30 0x7fb0ec0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/2863109371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec0b95a0 0x7fb0ec0b5700 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/2863109371 >> v2:172.21.15.69:3300/0 conn(0x7fb0ec15df00 0x7fb0ec0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f3be4640 1 -- 172.21.15.27:0/2863109371 >> 172.21.15.27:0/2863109371 conn(0x7fb0ec071960 msgr2=0x7fb0ec071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f3be4640 1 -- 172.21.15.27:0/2863109371 shutdown_connections 2024-03-06T07:36:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.740+0000 7fb0f3be4640 1 -- 172.21.15.27:0/2863109371 wait complete. 2024-03-06T07:36:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.742+0000 7fb0f3be4640 1 Processor -- start 2024-03-06T07:36:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.742+0000 7fb0f3be4640 1 -- start start 2024-03-06T07:36:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.742+0000 7fb0f3be4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0ec0b95a0 0x7fb0ec151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.742+0000 7fb0f3be4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec15df00 0x7fb0ec160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.742+0000 7fb0f3be4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec160bc0 0x7fb0ec0ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.742+0000 7fb0f3be4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec0cec70 con 0x7fb0ec160bc0 2024-03-06T07:36:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.742+0000 7fb0f3be4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec0ce750 con 0x7fb0ec0b95a0 2024-03-06T07:36:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.742+0000 7fb0f3be4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec0cedf0 con 0x7fb0ec15df00 2024-03-06T07:36:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f2181640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec160bc0 0x7fb0ec0ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f2181640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec160bc0 0x7fb0ec0ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42024/0 (socket says 172.21.15.27:42024) 2024-03-06T07:36:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f2181640 1 -- 172.21.15.27:0/45747846 learned_addr learned my addr 172.21.15.27:0/45747846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f117f640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec15df00 0x7fb0ec160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f1980640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0ec0b95a0 0x7fb0ec151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f2181640 1 -- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec15df00 msgr2=0x7fb0ec160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f2181640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec15df00 0x7fb0ec160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f2181640 1 -- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0ec0b95a0 msgr2=0x7fb0ec151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f2181640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0ec0b95a0 0x7fb0ec151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f2181640 1 -- 172.21.15.27:0/45747846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec161060 con 0x7fb0ec160bc0 2024-03-06T07:36:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f117f640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec15df00 0x7fb0ec160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f1980640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0ec0b95a0 0x7fb0ec151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f2181640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec160bc0 0x7fb0ec0ccaa0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009dd0 tx=0x7fb0dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0ea7fc640 1 -- 172.21.15.27:0/45747846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc018020 con 0x7fb0ec160bc0 2024-03-06T07:36:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0ea7fc640 1 -- 172.21.15.27:0/45747846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0dc014480 con 0x7fb0ec160bc0 2024-03-06T07:36:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0ea7fc640 1 -- 172.21.15.27:0/45747846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc014870 con 0x7fb0ec160bc0 2024-03-06T07:36:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec161340 con 0x7fb0ec160bc0 2024-03-06T07:36:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.743+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec0c7ce0 con 0x7fb0ec160bc0 2024-03-06T07:36:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.744+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ec068ad0 con 0x7fb0ec160bc0 2024-03-06T07:36:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.745+0000 7fb0ea7fc640 1 -- 172.21.15.27:0/45747846 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0dc005d10 con 0x7fb0ec160bc0 2024-03-06T07:36:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.745+0000 7fb0ea7fc640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb0d806b4d0 0x7fb0d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.745+0000 7fb0ea7fc640 1 -- 172.21.15.27:0/45747846 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb0dc08ab10 con 0x7fb0ec160bc0 2024-03-06T07:36:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.746+0000 7fb0f1980640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb0d806b4d0 0x7fb0d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.746+0000 7fb0f1980640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb0d806b4d0 0x7fb0d806d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40099a0 tx=0x7fb0e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.750+0000 7fb0ea7fc640 1 -- 172.21.15.27:0/45747846 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0ec068ad0 con 0x7fb0ec160bc0 2024-03-06T07:36:02.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.929+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0ec067ff0 con 0x7fb0d806b4d0 2024-03-06T07:36:02.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.930+0000 7fb0ea7fc640 1 -- 172.21.15.27:0/45747846 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13421 (secure 0 0 0) 0x7fb0ec067ff0 con 0x7fb0d806b4d0 2024-03-06T07:36:02.932 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:36:02.932 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:36:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb0d806b4d0 msgr2=0x7fb0d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb0d806b4d0 0x7fb0d806d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40099a0 tx=0x7fb0e4008040 comp rx=0 tx=0).stop 2024-03-06T07:36:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec160bc0 msgr2=0x7fb0ec0ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec160bc0 0x7fb0ec0ccaa0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009dd0 tx=0x7fb0dc00c490 comp rx=0 tx=0).stop 2024-03-06T07:36:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 shutdown_connections 2024-03-06T07:36:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fb0d806b4d0 0x7fb0d806d990 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0ec160bc0 0x7fb0ec0ccaa0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb0ec15df00 0x7fb0ec160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 --2- 172.21.15.27:0/45747846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0ec0b95a0 0x7fb0ec151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 >> 172.21.15.27:0/45747846 conn(0x7fb0ec071960 msgr2=0x7fb0ec0b85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:02.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 shutdown_connections 2024-03-06T07:36:02.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:02.932+0000 7fb0f3be4640 1 -- 172.21.15.27:0/45747846 wait complete. 2024-03-06T07:36:02.944 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":549,"stamp":"2024-03-06T07:36:02.180870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331964,"kb_used_data":15960,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754642756,"statfs":{"total":773094113280,"available":772754182144,"internally_reserved":0,"allocated":16343040,"data_stored":7834476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-37242,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-139,"num_write_kb":-84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001595"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":423,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-06T07:35:59.681896+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:35:59.681896+0000","last_peered":"2024-03-06T07:35:59.681896+0000","last_clean":"2024-03-06T07:35:59.681896+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:35:59.681896+0000","last_undegraded":"2024-03-06T07:35:59.681896+0000","last_fullsized":"2024-03-06T07:35:59.681896+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575012,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41944,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329896,"statfs":{"total":96636764160,"available":96593813504,"internally_reserved":0,"allocated":2502656,"data_stored":1428473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575013,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41348,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330492,"statfs":{"total":96636764160,"available":96594423808,"internally_reserved":0,"allocated":1892352,"data_stored":825419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41352,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330488,"statfs":{"total":96636764160,"available":96594419712,"internally_reserved":0,"allocated":1892352,"data_stored":830809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575012,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41332,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330508,"statfs":{"total":96636764160,"available":96594440192,"internally_reserved":0,"allocated":1875968,"data_stored":824691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41348,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330492,"statfs":{"total":96636764160,"available":96594423808,"internally_reserved":0,"allocated":1892352,"data_stored":830809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575012,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41348,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330492,"statfs":{"total":96636764160,"available":96594423808,"internally_reserved":0,"allocated":1892352,"data_stored":830809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41344,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330496,"statfs":{"total":96636764160,"available":96594427904,"internally_reserved":0,"allocated":1888256,"data_stored":825385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575013,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41948,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329892,"statfs":{"total":96636764160,"available":96593809408,"internally_reserved":0,"allocated":2506752,"data_stored":1438081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:36:02.945 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:36:02.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:36:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.050+0000 7f01cfa0e640 1 Processor -- start 2024-03-06T07:36:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.050+0000 7f01cfa0e640 1 -- start start 2024-03-06T07:36:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.051+0000 7f01cfa0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c80d3400 0x7f01c80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.051+0000 7f01cfa0e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c80ce610 con 0x7f01c80b2b50 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.051+0000 7f01cfa0e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c80d2920 con 0x7f01c8071540 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.051+0000 7f01cfa0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80d2f70 con 0x7f01c80d3400 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.051+0000 7f01cdfab640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f01c80b2b50 0x7f01c80b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51948/0 (socket says 172.21.15.27:51948) 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.051+0000 7f01cdfab640 1 -- 172.21.15.27:0/3078232124 learned_addr learned my addr 172.21.15.27:0/3078232124 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.052+0000 7f01ccfa9640 1 --2- 172.21.15.27:0/3078232124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c80d3400 0x7f01c80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.052+0000 7f01c7fff640 1 -- 172.21.15.27:0/3078232124 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278898771 0 0) 0x7f01c80ce610 con 0x7f01c80b2b50 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.052+0000 7f01c7fff640 1 -- 172.21.15.27:0/3078232124 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b0003520 con 0x7f01c80b2b50 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.052+0000 7f01c7fff640 1 -- 172.21.15.27:0/3078232124 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148568185 0 0) 0x7f01c80d2920 con 0x7f01c8071540 2024-03-06T07:36:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.052+0000 7f01c7fff640 1 -- 172.21.15.27:0/3078232124 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c80ce610 con 0x7f01c8071540 2024-03-06T07:36:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.052+0000 7f01ccfa9640 1 -- 172.21.15.27:0/3078232124 >> v1:172.21.15.69:6789/0 conn(0x7f01c8071540 legacy=0x7f01c8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.052+0000 7f01ccfa9640 1 -- 172.21.15.27:0/3078232124 >> v1:172.21.15.27:6789/0 conn(0x7f01c80b2b50 legacy=0x7f01c80b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.052+0000 7f01ccfa9640 1 -- 172.21.15.27:0/3078232124 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c80d2c00 con 0x7f01c80d3400 2024-03-06T07:36:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.053+0000 7f01ccfa9640 1 --2- 172.21.15.27:0/3078232124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c80d3400 0x7f01c80b30a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0097d0 tx=0x7f01bc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3deb2b77bd39bb80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.053+0000 7f01c7fff640 1 -- 172.21.15.27:0/3078232124 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc035020 con 0x7f01c80d3400 2024-03-06T07:36:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.053+0000 7f01c7fff640 1 -- 172.21.15.27:0/3078232124 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01bc031890 con 0x7f01c80d3400 2024-03-06T07:36:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.053+0000 7f01c7fff640 1 -- 172.21.15.27:0/3078232124 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc031c60 con 0x7f01c80d3400 2024-03-06T07:36:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.053+0000 7f01cfa0e640 1 -- 172.21.15.27:0/3078232124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c80d3400 msgr2=0x7f01c80b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.053+0000 7f01cfa0e640 1 --2- 172.21.15.27:0/3078232124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c80d3400 0x7f01c80b30a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0097d0 tx=0x7f01bc02b930 comp rx=0 tx=0).stop 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.053+0000 7f01cfa0e640 1 -- 172.21.15.27:0/3078232124 shutdown_connections 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.053+0000 7f01cfa0e640 1 --2- 172.21.15.27:0/3078232124 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c80d3400 0x7f01c80b30a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.053+0000 7f01cfa0e640 1 -- 172.21.15.27:0/3078232124 >> 172.21.15.27:0/3078232124 conn(0x7f01c80d3ca0 msgr2=0x7f01c80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.054+0000 7f01cfa0e640 1 -- 172.21.15.27:0/3078232124 shutdown_connections 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.054+0000 7f01cfa0e640 1 -- 172.21.15.27:0/3078232124 wait complete. 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.055+0000 7f01cfa0e640 1 Processor -- start 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.055+0000 7f01cfa0e640 1 -- start start 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.055+0000 7f01cfa0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c8071540 0x7f01c80b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.055+0000 7f01cfa0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01c80b2b50 0x7f01c80b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.055+0000 7f01cfa0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c80d3400 0x7f01c80aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.055+0000 7f01cfa0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c815e370 con 0x7f01c80d3400 2024-03-06T07:36:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.055+0000 7f01cfa0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80d4be0 con 0x7f01c80b2b50 2024-03-06T07:36:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.055+0000 7f01cfa0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80b57c0 con 0x7f01c8071540 2024-03-06T07:36:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cdfab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c80d3400 0x7f01c80aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cdfab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c80d3400 0x7f01c80aa6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42028/0 (socket says 172.21.15.27:42028) 2024-03-06T07:36:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cdfab640 1 -- 172.21.15.27:0/1195899765 learned_addr learned my addr 172.21.15.27:0/1195899765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01ccfa9640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01c80b2b50 0x7f01c80b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cd7aa640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c8071540 0x7f01c80b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cdfab640 1 -- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c8071540 msgr2=0x7f01c80b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cdfab640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c8071540 0x7f01c80b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cdfab640 1 -- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01c80b2b50 msgr2=0x7f01c80b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cdfab640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01c80b2b50 0x7f01c80b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cdfab640 1 -- 172.21.15.27:0/1195899765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c80b4bb0 con 0x7f01c80d3400 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cdfab640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c80d3400 0x7f01c80aa6a0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f01b800cba0 tx=0x7f01b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01c5ffb640 1 -- 172.21.15.27:0/1195899765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8017070 con 0x7f01c80d3400 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01c5ffb640 1 -- 172.21.15.27:0/1195899765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b800ace0 con 0x7f01c80d3400 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c80b4e90 con 0x7f01c80d3400 2024-03-06T07:36:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.056+0000 7f01c5ffb640 1 -- 172.21.15.27:0/1195899765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8012600 con 0x7f01c80d3400 2024-03-06T07:36:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.057+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c80b5450 con 0x7f01c80d3400 2024-03-06T07:36:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.057+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c8068700 con 0x7f01c80d3400 2024-03-06T07:36:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.058+0000 7f01c5ffb640 1 -- 172.21.15.27:0/1195899765 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01b801a070 con 0x7f01c80d3400 2024-03-06T07:36:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.058+0000 7f01c5ffb640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01b406b4d0 0x7f01b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.058+0000 7f01c5ffb640 1 -- 172.21.15.27:0/1195899765 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f01b8089cc0 con 0x7f01c80d3400 2024-03-06T07:36:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.059+0000 7f01cd7aa640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01b406b4d0 0x7f01b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.059+0000 7f01cd7aa640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01b406b4d0 0x7f01b406d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f01c00011e0 tx=0x7f01c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.063+0000 7f01c5ffb640 1 -- 172.21.15.27:0/1195899765 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01b80587a0 con 0x7f01c80d3400 2024-03-06T07:36:03.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.241+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01c80b92b0 con 0x7f01b406b4d0 2024-03-06T07:36:03.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.242+0000 7f01c5ffb640 1 -- 172.21.15.27:0/1195899765 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13421 (secure 0 0 0) 0x7f01c80b92b0 con 0x7f01b406b4d0 2024-03-06T07:36:03.244 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:36:03.244 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:36:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01b406b4d0 msgr2=0x7f01b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01b406b4d0 0x7f01b406d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f01c00011e0 tx=0x7f01c0009040 comp rx=0 tx=0).stop 2024-03-06T07:36:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c80d3400 msgr2=0x7f01c80aa6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c80d3400 0x7f01c80aa6a0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f01b800cba0 tx=0x7f01b800a430 comp rx=0 tx=0).stop 2024-03-06T07:36:03.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 shutdown_connections 2024-03-06T07:36:03.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f01b406b4d0 0x7f01b406d990 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c80d3400 0x7f01c80aa6a0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01c80b2b50 0x7f01c80b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 --2- 172.21.15.27:0/1195899765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f01c8071540 0x7f01c80b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 >> 172.21.15.27:0/1195899765 conn(0x7f01c80d3ca0 msgr2=0x7f01c80c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:03.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 shutdown_connections 2024-03-06T07:36:03.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.244+0000 7f01cfa0e640 1 -- 172.21.15.27:0/1195899765 wait complete. 2024-03-06T07:36:03.256 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":549,"stamp":"2024-03-06T07:36:02.180870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331964,"kb_used_data":15960,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754642756,"statfs":{"total":773094113280,"available":772754182144,"internally_reserved":0,"allocated":16343040,"data_stored":7834476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-37242,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-139,"num_write_kb":-84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001595"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":423,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-06T07:35:59.681896+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:35:59.681896+0000","last_peered":"2024-03-06T07:35:59.681896+0000","last_clean":"2024-03-06T07:35:59.681896+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:35:59.681896+0000","last_undegraded":"2024-03-06T07:35:59.681896+0000","last_fullsized":"2024-03-06T07:35:59.681896+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575012,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41944,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329896,"statfs":{"total":96636764160,"available":96593813504,"internally_reserved":0,"allocated":2502656,"data_stored":1428473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575013,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41348,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330492,"statfs":{"total":96636764160,"available":96594423808,"internally_reserved":0,"allocated":1892352,"data_stored":825419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41352,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330488,"statfs":{"total":96636764160,"available":96594419712,"internally_reserved":0,"allocated":1892352,"data_stored":830809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575012,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41332,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330508,"statfs":{"total":96636764160,"available":96594440192,"internally_reserved":0,"allocated":1875968,"data_stored":824691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41348,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330492,"statfs":{"total":96636764160,"available":96594423808,"internally_reserved":0,"allocated":1892352,"data_stored":830809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575012,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41348,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330492,"statfs":{"total":96636764160,"available":96594423808,"internally_reserved":0,"allocated":1892352,"data_stored":830809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41344,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330496,"statfs":{"total":96636764160,"available":96594427904,"internally_reserved":0,"allocated":1888256,"data_stored":825385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575013,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41948,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329892,"statfs":{"total":96636764160,"available":96593809408,"internally_reserved":0,"allocated":2506752,"data_stored":1438081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:36:03.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:36:03.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.363+0000 7f6357fbc640 1 Processor -- start 2024-03-06T07:36:03.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.363+0000 7f6357fbc640 1 -- start start 2024-03-06T07:36:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6357fbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6350071c50 0x7f63500b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6357fbc640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63500b6f20 con 0x7f635015df70 2024-03-06T07:36:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6357fbc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63500d3030 con 0x7f63500d18c0 2024-03-06T07:36:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6357fbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63500d3310 con 0x7f6350071c50 2024-03-06T07:36:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6355d58640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f635015df70 0x7f63500d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51960/0 (socket says 172.21.15.27:51960) 2024-03-06T07:36:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6355d58640 1 -- 172.21.15.27:0/4014893693 learned_addr learned my addr 172.21.15.27:0/4014893693 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6356559640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6350071c50 0x7f63500b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924586381 0 0) 0x7f63500b6f20 con 0x7f635015df70 2024-03-06T07:36:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f633c003520 con 0x7f635015df70 2024-03-06T07:36:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256255242 0 0) 0x7f63500d3030 con 0x7f63500d18c0 2024-03-06T07:36:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63500b6f20 con 0x7f63500d18c0 2024-03-06T07:36:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2116405026 0 0) 0x7f633c003520 con 0x7f635015df70 2024-03-06T07:36:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6350071c50 msgr2=0x7f63500b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:03.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6350071c50 0x7f63500b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 >> v1:172.21.15.69:6789/0 conn(0x7f63500d18c0 legacy=0x7f6350071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63500d35f0 con 0x7f635015df70 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7f634c001670 con 0x7f635015df70 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6356559640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6350071c50 0x7f63500b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.364+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 >> v1:172.21.15.27:6789/0 conn(0x7f635015df70 legacy=0x7f63500d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6354d56640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6350071c50 0x7f633c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6354d56640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f633c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6354d56640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f633c004d40 0x7f633c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63500d3310 con 0x7f633c004100 2024-03-06T07:36:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c00bad0 con 0x7f633c004d40 2024-03-06T07:36:03.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c00bcd0 con 0x7f6350071c50 2024-03-06T07:36:03.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6355557640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f633c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6355d58640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6350071c50 0x7f633c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6356559640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f633c004d40 0x7f633c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6355557640 1 -- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6350071c50 msgr2=0x7f633c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:03.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6355557640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6350071c50 0x7f633c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6355557640 1 -- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f633c004d40 msgr2=0x7f633c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:03.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6355557640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f633c004d40 0x7f633c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.365+0000 7f6355557640 1 -- 172.21.15.27:0/4014893693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c003520 con 0x7f633c004100 2024-03-06T07:36:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.366+0000 7f6355557640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f633c004800 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f63400018a0 tx=0x7f633c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1d6b5dba749d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.366+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340037070 con 0x7f633c004100 2024-03-06T07:36:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.366+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63400052d0 con 0x7f633c004100 2024-03-06T07:36:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.366+0000 7f6354d56640 1 -- 172.21.15.27:0/4014893693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63400014f0 con 0x7f633c004100 2024-03-06T07:36:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.366+0000 7f6357fbc640 1 -- 172.21.15.27:0/4014893693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 msgr2=0x7f633c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.366+0000 7f6357fbc640 1 --2- 172.21.15.27:0/4014893693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f633c004800 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f63400018a0 tx=0x7f633c005830 comp rx=0 tx=0).stop 2024-03-06T07:36:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.366+0000 7f6356559640 1 -- 172.21.15.27:0/4014893693 reap_dead start 2024-03-06T07:36:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.366+0000 7f6357fbc640 1 -- 172.21.15.27:0/4014893693 shutdown_connections 2024-03-06T07:36:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.366+0000 7f6357fbc640 1 -- 172.21.15.27:0/4014893693 >> 172.21.15.27:0/4014893693 conn(0x7f63500d3cf0 msgr2=0x7f63500d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:03.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.367+0000 7f6357fbc640 1 -- 172.21.15.27:0/4014893693 shutdown_connections 2024-03-06T07:36:03.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.367+0000 7f6357fbc640 1 -- 172.21.15.27:0/4014893693 wait complete. 2024-03-06T07:36:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6357fbc640 1 Processor -- start 2024-03-06T07:36:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6357fbc640 1 -- start start 2024-03-06T07:36:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6357fbc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f63500cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6357fbc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635014b820 0x7f63500ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6357fbc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f635014bca0 0x7f63500ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6357fbc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635015e4d0 con 0x7f633c004100 2024-03-06T07:36:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6357fbc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635015e350 con 0x7f635014bca0 2024-03-06T07:36:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6357fbc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63500b8410 con 0x7f635014b820 2024-03-06T07:36:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6355d58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f63500cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.368+0000 7f6355d58640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f63500cb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42034/0 (socket says 172.21.15.27:42034) 2024-03-06T07:36:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6355d58640 1 -- 172.21.15.27:0/2601152143 learned_addr learned my addr 172.21.15.27:0/2601152143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6355557640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635014b820 0x7f63500ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6356559640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f635014bca0 0x7f63500ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6355d58640 1 -- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635014b820 msgr2=0x7f63500ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6355d58640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635014b820 0x7f63500ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6355d58640 1 -- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f635014bca0 msgr2=0x7f63500ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6355d58640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f635014bca0 0x7f63500ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6355d58640 1 -- 172.21.15.27:0/2601152143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635014c3d0 con 0x7f633c004100 2024-03-06T07:36:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6355d58640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f63500cb6c0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f634c007290 tx=0x7f634c007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f63467fc640 1 -- 172.21.15.27:0/2601152143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c001640 con 0x7f633c004100 2024-03-06T07:36:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f63467fc640 1 -- 172.21.15.27:0/2601152143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c018070 con 0x7f633c004100 2024-03-06T07:36:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635014c6b0 con 0x7f633c004100 2024-03-06T07:36:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f63467fc640 1 -- 172.21.15.27:0/2601152143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c00fc00 con 0x7f633c004100 2024-03-06T07:36:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.369+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635014cc70 con 0x7f633c004100 2024-03-06T07:36:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.370+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6350067ff0 con 0x7f633c004100 2024-03-06T07:36:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.371+0000 7f63467fc640 1 -- 172.21.15.27:0/2601152143 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f634c01d460 con 0x7f633c004100 2024-03-06T07:36:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.371+0000 7f63467fc640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633406b4d0 0x7f633406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.371+0000 7f63467fc640 1 -- 172.21.15.27:0/2601152143 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f634c089f70 con 0x7f633c004100 2024-03-06T07:36:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.371+0000 7f6355557640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633406b4d0 0x7f633406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.372+0000 7f6355557640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633406b4d0 0x7f633406d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f6340034000 tx=0x7f6340031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:03.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.375+0000 7f63467fc640 1 -- 172.21.15.27:0/2601152143 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f634c059090 con 0x7f633c004100 2024-03-06T07:36:03.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.558+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63500d2ac0 con 0x7f633c004100 2024-03-06T07:36:03.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.559+0000 7f63467fc640 1 -- 172.21.15.27:0/2601152143 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+13088 (secure 0 0 0) 0x7f634c058990 con 0x7f633c004100 2024-03-06T07:36:03.560 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:36:03.560 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":152,"fsid":"ad4c3e9b-9a91-4ae2-99c9-4cb584977bc3","created":"2024-03-06T07:22:23.740158+0000","modified":"2024-03-06T07:36:00.330398+0000","last_up_change":"2024-03-06T07:22:34.942546+0000","last_in_change":"2024-03-06T07:22:25.997497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:22:36.043110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a4d6578-2fd9-4b76-99c2-17fb01916001","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6801","nonce":3926946944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6803","nonce":3926946944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6807","nonce":3926946944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3926946944},{"type":"v1","addr":"172.21.15.27:6805","nonce":3926946944}]},"public_addr":"172.21.15.27:6801/3926946944","cluster_addr":"172.21.15.27:6803/3926946944","heartbeat_back_addr":"172.21.15.27:6807/3926946944","heartbeat_front_addr":"172.21.15.27:6805/3926946944","state":["exists","up"]},{"osd":1,"uuid":"69fd2f9d-6cff-4219-9c13-3a1b08bd1649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6811","nonce":746548704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6815","nonce":746548704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6823","nonce":746548704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":746548704},{"type":"v1","addr":"172.21.15.27:6820","nonce":746548704}]},"public_addr":"172.21.15.27:6811/746548704","cluster_addr":"172.21.15.27:6815/746548704","heartbeat_back_addr":"172.21.15.27:6823/746548704","heartbeat_front_addr":"172.21.15.27:6820/746548704","state":["exists","up"]},{"osd":2,"uuid":"5bd533bb-ebc0-4066-b0ff-6cf9fad0ca59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6810","nonce":3892975520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6814","nonce":3892975520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6821","nonce":3892975520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3892975520},{"type":"v1","addr":"172.21.15.27:6817","nonce":3892975520}]},"public_addr":"172.21.15.27:6810/3892975520","cluster_addr":"172.21.15.27:6814/3892975520","heartbeat_back_addr":"172.21.15.27:6821/3892975520","heartbeat_front_addr":"172.21.15.27:6817/3892975520","state":["exists","up"]},{"osd":3,"uuid":"be254672-63a5-4e78-a2f8-596fe2afa104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6825","nonce":4277055095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6827","nonce":4277055095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6831","nonce":4277055095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4277055095},{"type":"v1","addr":"172.21.15.27:6829","nonce":4277055095}]},"public_addr":"172.21.15.27:6825/4277055095","cluster_addr":"172.21.15.27:6827/4277055095","heartbeat_back_addr":"172.21.15.27:6831/4277055095","heartbeat_front_addr":"172.21.15.27:6829/4277055095","state":["exists","up"]},{"osd":4,"uuid":"76d7303e-19c5-4bbe-bd4f-0efdaa3c0cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6809","nonce":1626703870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6811","nonce":1626703870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6815","nonce":1626703870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1626703870},{"type":"v1","addr":"172.21.15.69:6813","nonce":1626703870}]},"public_addr":"172.21.15.69:6809/1626703870","cluster_addr":"172.21.15.69:6811/1626703870","heartbeat_back_addr":"172.21.15.69:6815/1626703870","heartbeat_front_addr":"172.21.15.69:6813/1626703870","state":["exists","up"]},{"osd":5,"uuid":"ef198e8f-3cec-4506-bc07-dd1a4f70aceb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6821","nonce":715204514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6823","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6825","nonce":715204514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6831","nonce":715204514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6827","nonce":715204514},{"type":"v1","addr":"172.21.15.69:6829","nonce":715204514}]},"public_addr":"172.21.15.69:6821/715204514","cluster_addr":"172.21.15.69:6825/715204514","heartbeat_back_addr":"172.21.15.69:6831/715204514","heartbeat_front_addr":"172.21.15.69:6829/715204514","state":["exists","up"]},{"osd":6,"uuid":"93b9645a-7585-49e2-9c6d-bc4d715946d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6801","nonce":288062332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6803","nonce":288062332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6807","nonce":288062332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":288062332},{"type":"v1","addr":"172.21.15.69:6805","nonce":288062332}]},"public_addr":"172.21.15.69:6801/288062332","cluster_addr":"172.21.15.69:6803/288062332","heartbeat_back_addr":"172.21.15.69:6807/288062332","heartbeat_front_addr":"172.21.15.69:6805/288062332","state":["exists","up"]},{"osd":7,"uuid":"f0686243-4ac5-48eb-97aa-b9b9345b3615","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6817","nonce":1392567864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6820","nonce":1392567864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6828","nonce":1392567864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1392567864},{"type":"v1","addr":"172.21.15.69:6824","nonce":1392567864}]},"public_addr":"172.21.15.69:6817/1392567864","cluster_addr":"172.21.15.69:6820/1392567864","heartbeat_back_addr":"172.21.15.69:6828/1392567864","heartbeat_front_addr":"172.21.15.69:6824/1392567864","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.125473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.159364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.172526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.157306+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.113631+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.155345+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.163327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:22:31.195480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6837/1115582329":"2024-03-07T07:35:57.036354+0000","172.21.15.69:6835/1115582329":"2024-03-07T07:35:57.036354+0000","172.21.15.27:6834/2910933327":"2024-03-07T07:35:57.036371+0000","172.21.15.27:6832/2910933327":"2024-03-07T07:35:57.036371+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:36:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.560+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633406b4d0 msgr2=0x7f633406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.560+0000 7f6357fbc640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633406b4d0 0x7f633406d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f6340034000 tx=0x7f6340031000 comp rx=0 tx=0).stop 2024-03-06T07:36:03.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.560+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 msgr2=0x7f63500cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.560+0000 7f6357fbc640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f63500cb6c0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f634c007290 tx=0x7f634c007750 comp rx=0 tx=0).stop 2024-03-06T07:36:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.561+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 shutdown_connections 2024-03-06T07:36:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.561+0000 7f6357fbc640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f633406b4d0 0x7f633406d990 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.561+0000 7f6357fbc640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f635014bca0 0x7f63500ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.561+0000 7f6357fbc640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f635014b820 0x7f63500ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.561+0000 7f6357fbc640 1 --2- 172.21.15.27:0/2601152143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f633c004100 0x7f63500cb6c0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.561+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 >> 172.21.15.27:0/2601152143 conn(0x7f63500d3cf0 msgr2=0x7f63500b87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.561+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 shutdown_connections 2024-03-06T07:36:03.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:03.561+0000 7f6357fbc640 1 -- 172.21.15.27:0/2601152143 wait complete. 2024-03-06T07:36:04.574 INFO:tasks.ceph:Scrubbing osd.0 2024-03-06T07:36:04.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:36:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.697+0000 7f363565c640 1 Processor -- start 2024-03-06T07:36:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.697+0000 7f363565c640 1 -- start start 2024-03-06T07:36:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.698+0000 7f363565c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36300b2d30 0x7f36300b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.698+0000 7f363565c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b7820 0x7f36300b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.698+0000 7f363565c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300d4960 con 0x7f3630071540 2024-03-06T07:36:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.698+0000 7f363565c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36300d3230 con 0x7f36300b2d30 2024-03-06T07:36:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.698+0000 7f363565c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36300d36c0 con 0x7f36300b7820 2024-03-06T07:36:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362e7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36300b2d30 0x7f36300b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b7820 0x7f36300b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362e7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36300b2d30 0x7f36300b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:39170/0 (socket says 172.21.15.27:39170) 2024-03-06T07:36:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b7820 0x7f36300b7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.27:35780/0 (socket says 172.21.15.27:35780) 2024-03-06T07:36:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362effd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3630071540 0x7f3630071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:48492/0 (socket says 172.21.15.27:48492) 2024-03-06T07:36:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362e7fc640 1 -- 172.21.15.27:0/1093957162 learned_addr learned my addr 172.21.15.27:0/1093957162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362f7fe640 1 -- 172.21.15.27:0/1093957162 >> v2:172.21.15.69:3300/0 conn(0x7f36300b2d30 msgr2=0x7f36300b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362f7fe640 1 --2- 172.21.15.27:0/1093957162 >> v2:172.21.15.69:3300/0 conn(0x7f36300b2d30 0x7f36300b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362f7fe640 1 -- 172.21.15.27:0/1093957162 >> v1:172.21.15.69:6789/0 conn(0x7f3630071540 legacy=0x7f3630071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362f7fe640 1 -- 172.21.15.27:0/1093957162 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36300d2620 con 0x7f36300b7820 2024-03-06T07:36:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.699+0000 7f362e7fc640 1 --2- 172.21.15.27:0/1093957162 >> v2:172.21.15.69:3300/0 conn(0x7f36300b2d30 0x7f36300b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.700+0000 7f362f7fe640 1 --2- 172.21.15.27:0/1093957162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b7820 0x7f36300b7ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3628009d90 tx=0x7f362802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c28e67e11cadba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.700+0000 7f362dffb640 1 -- 172.21.15.27:0/1093957162 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628039070 con 0x7f36300b7820 2024-03-06T07:36:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.700+0000 7f362dffb640 1 -- 172.21.15.27:0/1093957162 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362802cc50 con 0x7f36300b7820 2024-03-06T07:36:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.700+0000 7f362dffb640 1 -- 172.21.15.27:0/1093957162 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628034440 con 0x7f36300b7820 2024-03-06T07:36:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.701+0000 7f363565c640 1 -- 172.21.15.27:0/1093957162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b7820 msgr2=0x7f36300b7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.701+0000 7f363565c640 1 --2- 172.21.15.27:0/1093957162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b7820 0x7f36300b7ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3628009d90 tx=0x7f362802bea0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.701+0000 7f363565c640 1 -- 172.21.15.27:0/1093957162 shutdown_connections 2024-03-06T07:36:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.701+0000 7f363565c640 1 --2- 172.21.15.27:0/1093957162 >> v2:172.21.15.69:3300/0 conn(0x7f36300b2d30 0x7f36300b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.701+0000 7f363565c640 1 --2- 172.21.15.27:0/1093957162 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b7820 0x7f36300b7ca0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.701+0000 7f363565c640 1 -- 172.21.15.27:0/1093957162 >> 172.21.15.27:0/1093957162 conn(0x7f36300d3d60 msgr2=0x7f36300d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.701+0000 7f363565c640 1 -- 172.21.15.27:0/1093957162 shutdown_connections 2024-03-06T07:36:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.701+0000 7f363565c640 1 -- 172.21.15.27:0/1093957162 wait complete. 2024-03-06T07:36:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.702+0000 7f363565c640 1 Processor -- start 2024-03-06T07:36:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.702+0000 7f363565c640 1 -- start start 2024-03-06T07:36:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f363565c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3630071540 0x7f36300ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f363565c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b2d30 0x7f36300c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f363565c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36300b7820 0x7f3630152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f363565c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36300d2ec0 con 0x7f3630071540 2024-03-06T07:36:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f363565c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36300d2be0 con 0x7f36300b7820 2024-03-06T07:36:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f363565c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36300d22f0 con 0x7f36300b2d30 2024-03-06T07:36:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3630071540 0x7f36300ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3630071540 0x7f36300ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38454/0 (socket says 172.21.15.27:38454) 2024-03-06T07:36:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362effd640 1 -- 172.21.15.27:0/3408605333 learned_addr learned my addr 172.21.15.27:0/3408605333 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362f7fe640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36300b7820 0x7f3630152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362e7fc640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b2d30 0x7f36300c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362effd640 1 -- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b2d30 msgr2=0x7f36300c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362effd640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b2d30 0x7f36300c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362effd640 1 -- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36300b7820 msgr2=0x7f3630152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362effd640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36300b7820 0x7f3630152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362effd640 1 -- 172.21.15.27:0/3408605333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36300cabd0 con 0x7f3630071540 2024-03-06T07:36:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362e7fc640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f36300b2d30 0x7f36300c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.703+0000 7f362f7fe640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36300b7820 0x7f3630152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.704+0000 7f362effd640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3630071540 0x7f36300ca690 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f362400c3c0 tx=0x7f362400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.704+0000 7f36177fe640 1 -- 172.21.15.27:0/3408605333 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36240074e0 con 0x7f3630071540 2024-03-06T07:36:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.704+0000 7f36177fe640 1 -- 172.21.15.27:0/3408605333 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3624007680 con 0x7f3630071540 2024-03-06T07:36:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.704+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3630151ea0 con 0x7f3630071540 2024-03-06T07:36:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.704+0000 7f36177fe640 1 -- 172.21.15.27:0/3408605333 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624004a00 con 0x7f3630071540 2024-03-06T07:36:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.704+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3630152460 con 0x7f3630071540 2024-03-06T07:36:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.705+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3630072070 con 0x7f3630071540 2024-03-06T07:36:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.705+0000 7f36177fe640 1 -- 172.21.15.27:0/3408605333 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f362401a020 con 0x7f3630071540 2024-03-06T07:36:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.706+0000 7f36177fe640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f361c06b4d0 0x7f361c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.706+0000 7f36177fe640 1 -- 172.21.15.27:0/3408605333 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f36240896d0 con 0x7f3630071540 2024-03-06T07:36:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.706+0000 7f36177fe640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f361c071430 0x7f361c073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.706+0000 7f36177fe640 1 -- 172.21.15.27:0/3408605333 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3624011e50 con 0x7f361c071430 2024-03-06T07:36:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.706+0000 7f36177fe640 1 -- 172.21.15.27:0/3408605333 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3624090050 con 0x7f3630071540 2024-03-06T07:36:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.706+0000 7f362e7fc640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f361c06b4d0 0x7f361c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.706+0000 7f362f7fe640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f361c071430 0x7f361c073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.707+0000 7f362f7fe640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f361c071430 0x7f361c073850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.707+0000 7f362e7fc640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f361c06b4d0 0x7f361c06d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f36300b9940 tx=0x7f3620008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.707+0000 7f36177fe640 1 -- 172.21.15.27:0/3408605333 <== osd.0 v2:172.21.15.27:6800/3926946944 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3624011e50 con 0x7f361c071430 2024-03-06T07:36:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 --> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f363006ce20 con 0x7f361c071430 2024-03-06T07:36:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f36177fe640 1 -- 172.21.15.27:0/3408605333 <== osd.0 v2:172.21.15.27:6800/3926946944 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f363006ce20 con 0x7f361c071430 2024-03-06T07:36:04.722 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:36:04.723 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:36:04.723 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:36:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f361c071430 msgr2=0x7f361c073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.27:6800/3926946944,v1:172.21.15.27:6801/3926946944] conn(0x7f361c071430 0x7f361c073850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f361c06b4d0 msgr2=0x7f361c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f361c06b4d0 0x7f361c06d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f36300b9940 tx=0x7f3620008040 comp rx=0 tx=0).stop 2024-03-06T07:36:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3630071540 msgr2=0x7f36300ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 --2- 172.21.15.27:0/3408605333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3630071540 0x7f36300ca690 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f362400c3c0 tx=0x7f362400c880 comp rx=0 tx=0).stop 2024-03-06T07:36:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f362f7fe640 1 -- 172.21.15.27:0/3408605333 reap_dead start 2024-03-06T07:36:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 shutdown_connections 2024-03-06T07:36:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 >> 172.21.15.27:0/3408605333 conn(0x7f36300d3d60 msgr2=0x7f363015b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.721+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 shutdown_connections 2024-03-06T07:36:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.722+0000 7f363565c640 1 -- 172.21.15.27:0/3408605333 wait complete. 2024-03-06T07:36:04.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-06T07:36:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.842+0000 7f947eaa4640 1 Processor -- start 2024-03-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.842+0000 7f947eaa4640 1 -- start start 2024-03-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f947eaa4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072390 0x7f94780708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f947eaa4640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947815e690 con 0x7f9478072770 2024-03-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f947eaa4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947815e970 con 0x7f9478157e10 2024-03-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f947eaa4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94780af300 con 0x7f9478072390 2024-03-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f9477fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9478072770 0x7f94780cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49494/0 (socket says 172.21.15.27:49494) 2024-03-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f9477fff640 1 -- 172.21.15.27:0/3702629538 learned_addr learned my addr 172.21.15.27:0/3702629538 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f947c840640 1 --2- 172.21.15.27:0/3702629538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072390 0x7f94780708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f94777fe640 1 -- 172.21.15.27:0/3702629538 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248739210 0 0) 0x7f947815e690 con 0x7f9478072770 2024-03-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f94777fe640 1 -- 172.21.15.27:0/3702629538 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9468003520 con 0x7f9478072770 2024-03-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f94777fe640 1 -- 172.21.15.27:0/3702629538 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908581425 0 0) 0x7f947815e970 con 0x7f9478157e10 2024-03-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f94777fe640 1 -- 172.21.15.27:0/3702629538 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f947815e690 con 0x7f9478157e10 2024-03-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f947c840640 1 -- 172.21.15.27:0/3702629538 >> v1:172.21.15.69:6789/0 conn(0x7f9478157e10 legacy=0x7f9478161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f947c840640 1 -- 172.21.15.27:0/3702629538 >> v1:172.21.15.27:6789/0 conn(0x7f9478072770 legacy=0x7f94780cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.843+0000 7f947c840640 1 -- 172.21.15.27:0/3702629538 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947815ec50 con 0x7f9478072390 2024-03-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.844+0000 7f947c840640 1 --2- 172.21.15.27:0/3702629538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072390 0x7f94780708b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f946c0097d0 tx=0x7f946c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb90368f039dbfeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.844+0000 7f94777fe640 1 -- 172.21.15.27:0/3702629538 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c035020 con 0x7f9478072390 2024-03-06T07:36:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.844+0000 7f94777fe640 1 -- 172.21.15.27:0/3702629538 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c031890 con 0x7f9478072390 2024-03-06T07:36:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.844+0000 7f94777fe640 1 -- 172.21.15.27:0/3702629538 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c031c60 con 0x7f9478072390 2024-03-06T07:36:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.845+0000 7f947eaa4640 1 -- 172.21.15.27:0/3702629538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072390 msgr2=0x7f94780708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.845+0000 7f947eaa4640 1 --2- 172.21.15.27:0/3702629538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072390 0x7f94780708b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f946c0097d0 tx=0x7f946c02b930 comp rx=0 tx=0).stop 2024-03-06T07:36:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.845+0000 7f947eaa4640 1 -- 172.21.15.27:0/3702629538 shutdown_connections 2024-03-06T07:36:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.845+0000 7f947eaa4640 1 --2- 172.21.15.27:0/3702629538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072390 0x7f94780708b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.845+0000 7f947eaa4640 1 -- 172.21.15.27:0/3702629538 >> 172.21.15.27:0/3702629538 conn(0x7f9478066d00 msgr2=0x7f94780670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.845+0000 7f947eaa4640 1 -- 172.21.15.27:0/3702629538 shutdown_connections 2024-03-06T07:36:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.845+0000 7f947eaa4640 1 -- 172.21.15.27:0/3702629538 wait complete. 2024-03-06T07:36:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.846+0000 7f947eaa4640 1 Processor -- start 2024-03-06T07:36:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.846+0000 7f947eaa4640 1 -- start start 2024-03-06T07:36:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.847+0000 7f947eaa4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9478072390 0x7f9478152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.847+0000 7f947eaa4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072770 0x7f94780b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.847+0000 7f947eaa4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9478157e10 0x7f9478160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.847+0000 7f947eaa4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947815f210 con 0x7f9478157e10 2024-03-06T07:36:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.847+0000 7f947eaa4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94780d0130 con 0x7f9478072390 2024-03-06T07:36:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.847+0000 7f947eaa4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9478070df0 con 0x7f9478072770 2024-03-06T07:36:04.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.847+0000 7f947d041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9478157e10 0x7f9478160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.847+0000 7f947d041640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9478157e10 0x7f9478160960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38462/0 (socket says 172.21.15.27:38462) 2024-03-06T07:36:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.847+0000 7f947d041640 1 -- 172.21.15.27:0/176392795 learned_addr learned my addr 172.21.15.27:0/176392795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f9477fff640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072770 0x7f94780b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f947c840640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9478072390 0x7f9478152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f947d041640 1 -- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072770 msgr2=0x7f94780b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f947d041640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072770 0x7f94780b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f947d041640 1 -- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9478072390 msgr2=0x7f9478152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f947d041640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9478072390 0x7f9478152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f947d041640 1 -- 172.21.15.27:0/176392795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94781f7260 con 0x7f9478157e10 2024-03-06T07:36:04.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f947d041640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9478157e10 0x7f9478160960 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f947000c6e0 tx=0x7f947000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f94757fa640 1 -- 172.21.15.27:0/176392795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9470003b70 con 0x7f9478157e10 2024-03-06T07:36:04.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f94757fa640 1 -- 172.21.15.27:0/176392795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9470003d10 con 0x7f9478157e10 2024-03-06T07:36:04.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94781f7540 con 0x7f9478157e10 2024-03-06T07:36:04.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f94757fa640 1 -- 172.21.15.27:0/176392795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94700052d0 con 0x7f9478157e10 2024-03-06T07:36:04.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.848+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94781f7b00 con 0x7f9478157e10 2024-03-06T07:36:04.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.849+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9478067ff0 con 0x7f9478157e10 2024-03-06T07:36:04.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.849+0000 7f94757fa640 1 -- 172.21.15.27:0/176392795 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9470005470 con 0x7f9478157e10 2024-03-06T07:36:04.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.850+0000 7f94757fa640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f945c06b4d0 0x7f945c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.850+0000 7f94757fa640 1 -- 172.21.15.27:0/176392795 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9470089de0 con 0x7f9478157e10 2024-03-06T07:36:04.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.850+0000 7f947c840640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f945c06b4d0 0x7f945c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.851+0000 7f947c840640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f945c06b4d0 0x7f945c06d990 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f946c036000 tx=0x7f946c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:04.855+0000 7f94757fa640 1 -- 172.21.15.27:0/176392795 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9470058f90 con 0x7f9478157e10 2024-03-06T07:36:05.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.041+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f94781f7010 con 0x7f945c06b4d0 2024-03-06T07:36:05.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.042+0000 7f94757fa640 1 -- 172.21.15.27:0/176392795 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f94781f7010 con 0x7f945c06b4d0 2024-03-06T07:36:05.043 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 0 to deep-scrub 2024-03-06T07:36:05.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f945c06b4d0 msgr2=0x7f945c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f945c06b4d0 0x7f945c06d990 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f946c036000 tx=0x7f946c008030 comp rx=0 tx=0).stop 2024-03-06T07:36:05.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9478157e10 msgr2=0x7f9478160960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9478157e10 0x7f9478160960 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f947000c6e0 tx=0x7f947000cba0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 shutdown_connections 2024-03-06T07:36:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f945c06b4d0 0x7f945c06d990 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9478157e10 0x7f9478160960 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9478072770 0x7f94780b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 --2- 172.21.15.27:0/176392795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9478072390 0x7f9478152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 >> 172.21.15.27:0/176392795 conn(0x7f9478066d00 msgr2=0x7f9478156080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 shutdown_connections 2024-03-06T07:36:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.044+0000 7f947eaa4640 1 -- 172.21.15.27:0/176392795 wait complete. 2024-03-06T07:36:05.057 INFO:tasks.ceph:Scrubbing osd.1 2024-03-06T07:36:05.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:36:05.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.168+0000 7f7e3cb19640 1 Processor -- start 2024-03-06T07:36:05.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.168+0000 7f7e3cb19640 1 -- start start 2024-03-06T07:36:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.169+0000 7f7e3cb19640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e3815dd90 0x7f7e380ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.169+0000 7f7e3cb19640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e380d29d0 0x7f7e380d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.169+0000 7f7e3cb19640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e380d4720 0x7f7e380b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.169+0000 7f7e3cb19640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e380d3400 con 0x7f7e380d29d0 2024-03-06T07:36:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.169+0000 7f7e3cb19640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e380d3740 con 0x7f7e380d4720 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.169+0000 7f7e3cb19640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e380d3120 con 0x7f7e3815dd90 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.169+0000 7f7e35d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e380d29d0 0x7f7e380d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.169+0000 7f7e35d9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e380d29d0 0x7f7e380d41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38474/0 (socket says 172.21.15.27:38474) 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.169+0000 7f7e35d9b640 1 -- 172.21.15.27:0/1161328487 learned_addr learned my addr 172.21.15.27:0/1161328487 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e36d9d640 1 --2- 172.21.15.27:0/1161328487 >> v2:172.21.15.69:3300/0 conn(0x7f7e380d4720 0x7f7e380b6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e3659c640 1 --2- 172.21.15.27:0/1161328487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e3815dd90 0x7f7e380ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e35d9b640 1 -- 172.21.15.27:0/1161328487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e3815dd90 msgr2=0x7f7e380ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e35d9b640 1 --2- 172.21.15.27:0/1161328487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e3815dd90 0x7f7e380ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e35d9b640 1 -- 172.21.15.27:0/1161328487 >> v2:172.21.15.69:3300/0 conn(0x7f7e380d4720 msgr2=0x7f7e380b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e35d9b640 1 --2- 172.21.15.27:0/1161328487 >> v2:172.21.15.69:3300/0 conn(0x7f7e380d4720 0x7f7e380b6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e35d9b640 1 -- 172.21.15.27:0/1161328487 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e380d39e0 con 0x7f7e380d29d0 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e3659c640 1 --2- 172.21.15.27:0/1161328487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e3815dd90 0x7f7e380ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:05.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e36d9d640 1 --2- 172.21.15.27:0/1161328487 >> v2:172.21.15.69:3300/0 conn(0x7f7e380d4720 0x7f7e380b6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.170+0000 7f7e35d9b640 1 --2- 172.21.15.27:0/1161328487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e380d29d0 0x7f7e380d41e0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f7e380b6c60 tx=0x7f7e2802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f34453ce48aee5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3559a640 1 -- 172.21.15.27:0/1161328487 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28034070 con 0x7f7e380d29d0 2024-03-06T07:36:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3559a640 1 -- 172.21.15.27:0/1161328487 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e28030850 con 0x7f7e380d29d0 2024-03-06T07:36:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3559a640 1 -- 172.21.15.27:0/1161328487 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28030c20 con 0x7f7e380d29d0 2024-03-06T07:36:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3cb19640 1 -- 172.21.15.27:0/1161328487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e380d29d0 msgr2=0x7f7e380d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3cb19640 1 --2- 172.21.15.27:0/1161328487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e380d29d0 0x7f7e380d41e0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f7e380b6c60 tx=0x7f7e2802b990 comp rx=0 tx=0).stop 2024-03-06T07:36:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3cb19640 1 -- 172.21.15.27:0/1161328487 shutdown_connections 2024-03-06T07:36:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3cb19640 1 --2- 172.21.15.27:0/1161328487 >> v2:172.21.15.69:3300/0 conn(0x7f7e380d4720 0x7f7e380b6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3cb19640 1 --2- 172.21.15.27:0/1161328487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e380d29d0 0x7f7e380d41e0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3cb19640 1 --2- 172.21.15.27:0/1161328487 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e3815dd90 0x7f7e380ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3cb19640 1 -- 172.21.15.27:0/1161328487 >> 172.21.15.27:0/1161328487 conn(0x7f7e38071960 msgr2=0x7f7e38071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3cb19640 1 -- 172.21.15.27:0/1161328487 shutdown_connections 2024-03-06T07:36:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.171+0000 7f7e3cb19640 1 -- 172.21.15.27:0/1161328487 wait complete. 2024-03-06T07:36:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.173+0000 7f7e3cb19640 1 Processor -- start 2024-03-06T07:36:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.173+0000 7f7e3cb19640 1 -- start start 2024-03-06T07:36:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.173+0000 7f7e3cb19640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e380d29d0 0x7f7e381492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.173+0000 7f7e3cb19640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380d4720 0x7f7e38146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.173+0000 7f7e3cb19640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3815dd90 0x7f7e3815ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e3cb19640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e380d2df0 con 0x7f7e3815dd90 2024-03-06T07:36:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e3cb19640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3815e170 con 0x7f7e380d29d0 2024-03-06T07:36:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e3cb19640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e380b7250 con 0x7f7e380d4720 2024-03-06T07:36:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e36d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3815dd90 0x7f7e3815ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e36d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3815dd90 0x7f7e3815ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38482/0 (socket says 172.21.15.27:38482) 2024-03-06T07:36:05.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e36d9d640 1 -- 172.21.15.27:0/2753604283 learned_addr learned my addr 172.21.15.27:0/2753604283 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:05.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e3659c640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e380d29d0 0x7f7e381492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e35d9b640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380d4720 0x7f7e38146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e36d9d640 1 -- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380d4720 msgr2=0x7f7e38146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e36d9d640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380d4720 0x7f7e38146880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e36d9d640 1 -- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e380d29d0 msgr2=0x7f7e381492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e36d9d640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e380d29d0 0x7f7e381492c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e36d9d640 1 -- 172.21.15.27:0/2753604283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3815e420 con 0x7f7e3815dd90 2024-03-06T07:36:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e3659c640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e380d29d0 0x7f7e381492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e35d9b640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7e380d4720 0x7f7e38146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e36d9d640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3815dd90 0x7f7e3815ae90 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000cef0 tx=0x7f7e3000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e26ffd640 1 -- 172.21.15.27:0/2753604283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3001a070 con 0x7f7e3815dd90 2024-03-06T07:36:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e26ffd640 1 -- 172.21.15.27:0/2753604283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3000d840 con 0x7f7e3815dd90 2024-03-06T07:36:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3815e6c0 con 0x7f7e3815dd90 2024-03-06T07:36:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e26ffd640 1 -- 172.21.15.27:0/2753604283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3000dc30 con 0x7f7e3815dd90 2024-03-06T07:36:05.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.174+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3815ec80 con 0x7f7e3815dd90 2024-03-06T07:36:05.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.175+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e38072a30 con 0x7f7e3815dd90 2024-03-06T07:36:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.176+0000 7f7e26ffd640 1 -- 172.21.15.27:0/2753604283 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e3001d070 con 0x7f7e3815dd90 2024-03-06T07:36:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.176+0000 7f7e26ffd640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e1806b4d0 0x7f7e1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.176+0000 7f7e26ffd640 1 -- 172.21.15.27:0/2753604283 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7e3008a760 con 0x7f7e3815dd90 2024-03-06T07:36:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.177+0000 7f7e26ffd640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7e18071430 0x7f7e18073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.177+0000 7f7e26ffd640 1 -- 172.21.15.27:0/2753604283 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e30018e70 con 0x7f7e18071430 2024-03-06T07:36:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.177+0000 7f7e26ffd640 1 -- 172.21.15.27:0/2753604283 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e30091050 con 0x7f7e3815dd90 2024-03-06T07:36:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.177+0000 7f7e35d9b640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7e18071430 0x7f7e18073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.177+0000 7f7e3659c640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e1806b4d0 0x7f7e1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.177+0000 7f7e35d9b640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7e18071430 0x7f7e18073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.177+0000 7f7e3659c640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e1806b4d0 0x7f7e1806d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c006fd0 tx=0x7f7e2c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.178+0000 7f7e26ffd640 1 -- 172.21.15.27:0/2753604283 <== osd.1 v2:172.21.15.27:6809/746548704 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7e30018e70 con 0x7f7e18071430 2024-03-06T07:36:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.191+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 --> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7e3804f410 con 0x7f7e18071430 2024-03-06T07:36:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.191+0000 7f7e26ffd640 1 -- 172.21.15.27:0/2753604283 <== osd.1 v2:172.21.15.27:6809/746548704 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7e3804f410 con 0x7f7e18071430 2024-03-06T07:36:05.193 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:36:05.193 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:36:05.193 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:36:05.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.191+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7e18071430 msgr2=0x7f7e18073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.191+0000 7f7e3cb19640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.27:6809/746548704,v1:172.21.15.27:6811/746548704] conn(0x7f7e18071430 0x7f7e18073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.191+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e1806b4d0 msgr2=0x7f7e1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.191+0000 7f7e3cb19640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f7e1806b4d0 0x7f7e1806d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c006fd0 tx=0x7f7e2c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.191+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3815dd90 msgr2=0x7f7e3815ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.191+0000 7f7e3cb19640 1 --2- 172.21.15.27:0/2753604283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3815dd90 0x7f7e3815ae90 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000cef0 tx=0x7f7e3000a710 comp rx=0 tx=0).stop 2024-03-06T07:36:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.191+0000 7f7e36d9d640 1 -- 172.21.15.27:0/2753604283 reap_dead start 2024-03-06T07:36:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.192+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 shutdown_connections 2024-03-06T07:36:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.192+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 >> 172.21.15.27:0/2753604283 conn(0x7f7e38071960 msgr2=0x7f7e381616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.192+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 shutdown_connections 2024-03-06T07:36:05.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.192+0000 7f7e3cb19640 1 -- 172.21.15.27:0/2753604283 wait complete. 2024-03-06T07:36:05.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-06T07:36:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.311+0000 7fd2f7681640 1 Processor -- start 2024-03-06T07:36:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.312+0000 7fd2f7681640 1 -- start start 2024-03-06T07:36:05.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.313+0000 7fd2f7681640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f015df00 0x7fd2f00d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.313+0000 7fd2f7681640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2f00b95a0 0x7fd2f00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.313+0000 7fd2f7681640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f00cea30 con 0x7fd2f00b9a30 2024-03-06T07:36:05.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.313+0000 7fd2f7681640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f00d2af0 con 0x7fd2f00b95a0 2024-03-06T07:36:05.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.313+0000 7fd2f7681640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00cf3c0 con 0x7fd2f015df00 2024-03-06T07:36:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.313+0000 7fd2f4c1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2f00b95a0 0x7fd2f00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.314+0000 7fd2f541d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f015df00 0x7fd2f00d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.314+0000 7fd2f4c1c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2f00b95a0 0x7fd2f00b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:39212/0 (socket says 172.21.15.27:39212) 2024-03-06T07:36:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.314+0000 7fd2f5c1e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd2f00b9a30 0x7fd2f00d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:48512/0 (socket says 172.21.15.27:48512) 2024-03-06T07:36:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.314+0000 7fd2f4c1c640 1 -- 172.21.15.27:0/3987811953 learned_addr learned my addr 172.21.15.27:0/3987811953 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.314+0000 7fd2f4c1c640 1 -- 172.21.15.27:0/3987811953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f015df00 msgr2=0x7fd2f00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.314+0000 7fd2f4c1c640 1 --2- 172.21.15.27:0/3987811953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f015df00 0x7fd2f00d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.314+0000 7fd2f4c1c640 1 -- 172.21.15.27:0/3987811953 >> v1:172.21.15.69:6789/0 conn(0x7fd2f00b9a30 legacy=0x7fd2f00d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.314+0000 7fd2f4c1c640 1 -- 172.21.15.27:0/3987811953 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f00cf0e0 con 0x7fd2f00b95a0 2024-03-06T07:36:05.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.314+0000 7fd2f541d640 1 --2- 172.21.15.27:0/3987811953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f015df00 0x7fd2f00d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.315+0000 7fd2f4c1c640 1 --2- 172.21.15.27:0/3987811953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2f00b95a0 0x7fd2f00b5700 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009870 tx=0x7fd2ec02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afef968a754dea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.315+0000 7fd2e7fff640 1 -- 172.21.15.27:0/3987811953 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec034070 con 0x7fd2f00b95a0 2024-03-06T07:36:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.315+0000 7fd2e7fff640 1 -- 172.21.15.27:0/3987811953 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ec030840 con 0x7fd2f00b95a0 2024-03-06T07:36:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.315+0000 7fd2e7fff640 1 -- 172.21.15.27:0/3987811953 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec030c10 con 0x7fd2f00b95a0 2024-03-06T07:36:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.316+0000 7fd2f7681640 1 -- 172.21.15.27:0/3987811953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2f00b95a0 msgr2=0x7fd2f00b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.316+0000 7fd2f7681640 1 --2- 172.21.15.27:0/3987811953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2f00b95a0 0x7fd2f00b5700 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009870 tx=0x7fd2ec02b980 comp rx=0 tx=0).stop 2024-03-06T07:36:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.316+0000 7fd2f7681640 1 -- 172.21.15.27:0/3987811953 shutdown_connections 2024-03-06T07:36:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.316+0000 7fd2f7681640 1 --2- 172.21.15.27:0/3987811953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2f00b95a0 0x7fd2f00b5700 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.316+0000 7fd2f7681640 1 --2- 172.21.15.27:0/3987811953 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f015df00 0x7fd2f00d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.316+0000 7fd2f7681640 1 -- 172.21.15.27:0/3987811953 >> 172.21.15.27:0/3987811953 conn(0x7fd2f0071960 msgr2=0x7fd2f0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.316+0000 7fd2f7681640 1 -- 172.21.15.27:0/3987811953 shutdown_connections 2024-03-06T07:36:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.316+0000 7fd2f7681640 1 -- 172.21.15.27:0/3987811953 wait complete. 2024-03-06T07:36:05.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.318+0000 7fd2f7681640 1 Processor -- start 2024-03-06T07:36:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.318+0000 7fd2f7681640 1 -- start start 2024-03-06T07:36:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f7681640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f00b95a0 0x7fd2f014abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f7681640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2f015df00 0x7fd2f00cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f7681640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f0149ce0 0x7fd2f014a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f7681640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00cec70 con 0x7fd2f00b95a0 2024-03-06T07:36:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f7681640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00ce750 con 0x7fd2f015df00 2024-03-06T07:36:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f7681640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00cedf0 con 0x7fd2f0149ce0 2024-03-06T07:36:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f541d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f00b95a0 0x7fd2f014abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f541d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f00b95a0 0x7fd2f014abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38496/0 (socket says 172.21.15.27:38496) 2024-03-06T07:36:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f541d640 1 -- 172.21.15.27:0/618714012 learned_addr learned my addr 172.21.15.27:0/618714012 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f4c1c640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2f015df00 0x7fd2f00cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f5c1e640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f0149ce0 0x7fd2f014a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f541d640 1 -- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f0149ce0 msgr2=0x7fd2f014a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f541d640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f0149ce0 0x7fd2f014a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f541d640 1 -- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2f015df00 msgr2=0x7fd2f00cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f541d640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2f015df00 0x7fd2f00cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.319+0000 7fd2f541d640 1 -- 172.21.15.27:0/618714012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f014a6b0 con 0x7fd2f00b95a0 2024-03-06T07:36:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.320+0000 7fd2f541d640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f00b95a0 0x7fd2f014abf0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000e960 tx=0x7fd2e000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.320+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/618714012 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0017070 con 0x7fd2f00b95a0 2024-03-06T07:36:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.320+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/618714012 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e000ca00 con 0x7fd2f00b95a0 2024-03-06T07:36:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.320+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/618714012 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e000cdd0 con 0x7fd2f00b95a0 2024-03-06T07:36:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.320+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f00c8c50 con 0x7fd2f00b95a0 2024-03-06T07:36:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.320+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f00c9210 con 0x7fd2f00b95a0 2024-03-06T07:36:05.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.321+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f0067ff0 con 0x7fd2f00b95a0 2024-03-06T07:36:05.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.321+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/618714012 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2e001a070 con 0x7fd2f00b95a0 2024-03-06T07:36:05.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.322+0000 7fd2e5ffb640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.322+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/618714012 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd2e008a120 con 0x7fd2f00b95a0 2024-03-06T07:36:05.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.322+0000 7fd2f4c1c640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.323+0000 7fd2f4c1c640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec036000 tx=0x7fd2ec008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.326+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/618714012 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2e0058bf0 con 0x7fd2f00b95a0 2024-03-06T07:36:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.514+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2f00c7310 con 0x7fd2d406b4d0 2024-03-06T07:36:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.515+0000 7fd2e5ffb640 1 -- 172.21.15.27:0/618714012 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd2f00c7310 con 0x7fd2d406b4d0 2024-03-06T07:36:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 1 to deep-scrub 2024-03-06T07:36:05.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 msgr2=0x7fd2d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec036000 tx=0x7fd2ec008030 comp rx=0 tx=0).stop 2024-03-06T07:36:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f00b95a0 msgr2=0x7fd2f014abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f00b95a0 0x7fd2f014abf0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000e960 tx=0x7fd2e000ee20 comp rx=0 tx=0).stop 2024-03-06T07:36:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 shutdown_connections 2024-03-06T07:36:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fd2d406b4d0 0x7fd2d406d990 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd2f0149ce0 0x7fd2f014a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2f015df00 0x7fd2f00cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 --2- 172.21.15.27:0/618714012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f00b95a0 0x7fd2f014abf0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 >> 172.21.15.27:0/618714012 conn(0x7fd2f0071960 msgr2=0x7fd2f00b8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 shutdown_connections 2024-03-06T07:36:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.517+0000 7fd2f7681640 1 -- 172.21.15.27:0/618714012 wait complete. 2024-03-06T07:36:05.531 INFO:tasks.ceph:Scrubbing osd.2 2024-03-06T07:36:05.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:36:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.641+0000 7fe8e7be2640 1 Processor -- start 2024-03-06T07:36:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.641+0000 7fe8e7be2640 1 -- start start 2024-03-06T07:36:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e7be2640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8e0144640 0x7fe8e006e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e7be2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 0x7fe8e015b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e7be2640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e015bc40 con 0x7fe8e015b670 2024-03-06T07:36:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e7be2640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e006df70 con 0x7fe8e0144640 2024-03-06T07:36:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e7be2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e015bf20 con 0x7fe8e006e940 2024-03-06T07:36:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e597e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8e0144640 0x7fe8e006e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e617f640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe8e015b670 0x7fe8e00ce900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49498/0 (socket says 172.21.15.27:49498) 2024-03-06T07:36:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e597e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8e0144640 0x7fe8e006e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38510/0 (socket says 172.21.15.27:38510) 2024-03-06T07:36:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e617f640 1 -- 172.21.15.27:0/400372192 learned_addr learned my addr 172.21.15.27:0/400372192 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e597e640 1 -- 172.21.15.27:0/400372192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 msgr2=0x7fe8e015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.642+0000 7fe8e517d640 1 --2- 172.21.15.27:0/400372192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 0x7fe8e015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.643+0000 7fe8e597e640 1 --2- 172.21.15.27:0/400372192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 0x7fe8e015b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.643+0000 7fe8e597e640 1 -- 172.21.15.27:0/400372192 >> v1:172.21.15.27:6789/0 conn(0x7fe8e015b670 legacy=0x7fe8e00ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.643+0000 7fe8e497c640 1 -- 172.21.15.27:0/400372192 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252072371 0 0) 0x7fe8e015bc40 con 0x7fe8e015b670 2024-03-06T07:36:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.643+0000 7fe8e597e640 1 -- 172.21.15.27:0/400372192 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e006dc00 con 0x7fe8e0144640 2024-03-06T07:36:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.643+0000 7fe8e517d640 1 --2- 172.21.15.27:0/400372192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 0x7fe8e015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.643+0000 7fe8e597e640 1 --2- 172.21.15.27:0/400372192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 0x7fe8e006e400 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000ba40 tx=0x7fe8d002e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8422a9a6eed790b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.643+0000 7fe8e497c640 1 -- 172.21.15.27:0/400372192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d003b070 con 0x7fe8e0144640 2024-03-06T07:36:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.643+0000 7fe8e497c640 1 -- 172.21.15.27:0/400372192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d002ee30 con 0x7fe8e0144640 2024-03-06T07:36:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.643+0000 7fe8e497c640 1 -- 172.21.15.27:0/400372192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0036640 con 0x7fe8e0144640 2024-03-06T07:36:05.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.644+0000 7fe8e7be2640 1 -- 172.21.15.27:0/400372192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 msgr2=0x7fe8e006e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.644+0000 7fe8e7be2640 1 --2- 172.21.15.27:0/400372192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 0x7fe8e006e400 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000ba40 tx=0x7fe8d002e080 comp rx=0 tx=0).stop 2024-03-06T07:36:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.644+0000 7fe8e7be2640 1 -- 172.21.15.27:0/400372192 shutdown_connections 2024-03-06T07:36:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.644+0000 7fe8e7be2640 1 --2- 172.21.15.27:0/400372192 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 0x7fe8e015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.644+0000 7fe8e7be2640 1 --2- 172.21.15.27:0/400372192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 0x7fe8e006e400 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.644+0000 7fe8e7be2640 1 -- 172.21.15.27:0/400372192 >> 172.21.15.27:0/400372192 conn(0x7fe8e0066d00 msgr2=0x7fe8e0067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.644+0000 7fe8e7be2640 1 -- 172.21.15.27:0/400372192 shutdown_connections 2024-03-06T07:36:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.644+0000 7fe8e7be2640 1 -- 172.21.15.27:0/400372192 wait complete. 2024-03-06T07:36:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.645+0000 7fe8e7be2640 1 Processor -- start 2024-03-06T07:36:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.645+0000 7fe8e7be2640 1 -- start start 2024-03-06T07:36:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e7be2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 0x7fe8e0141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e7be2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 0x7fe8e013c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e7be2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e015b670 0x7fe8e0138cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e7be2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00cf190 con 0x7fe8e0144640 2024-03-06T07:36:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e7be2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00cf010 con 0x7fe8e015b670 2024-03-06T07:36:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e7be2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00b36d0 con 0x7fe8e006e940 2024-03-06T07:36:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e517d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 0x7fe8e013c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e517d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 0x7fe8e013c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38524/0 (socket says 172.21.15.27:38524) 2024-03-06T07:36:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e517d640 1 -- 172.21.15.27:0/1814311125 learned_addr learned my addr 172.21.15.27:0/1814311125 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e597e640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 0x7fe8e0141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e617f640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e015b670 0x7fe8e0138cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e517d640 1 -- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 msgr2=0x7fe8e0141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e517d640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe8e006e940 0x7fe8e0141ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e517d640 1 -- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e015b670 msgr2=0x7fe8e0138cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e517d640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8e015b670 0x7fe8e0138cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.646+0000 7fe8e517d640 1 -- 172.21.15.27:0/1814311125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e01f3a70 con 0x7fe8e0144640 2024-03-06T07:36:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.647+0000 7fe8e517d640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 0x7fe8e013c7f0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00c7f0 tx=0x7fe8dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.647+0000 7fe8d67fc640 1 -- 172.21.15.27:0/1814311125 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc004230 con 0x7fe8e0144640 2024-03-06T07:36:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.647+0000 7fe8d67fc640 1 -- 172.21.15.27:0/1814311125 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc0043d0 con 0x7fe8e0144640 2024-03-06T07:36:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.647+0000 7fe8d67fc640 1 -- 172.21.15.27:0/1814311125 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc007a70 con 0x7fe8e0144640 2024-03-06T07:36:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.647+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e0143830 con 0x7fe8e0144640 2024-03-06T07:36:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.647+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e01f77a0 con 0x7fe8e0144640 2024-03-06T07:36:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.648+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe8e00733c0 con 0x7fe8e0144640 2024-03-06T07:36:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.648+0000 7fe8d67fc640 1 -- 172.21.15.27:0/1814311125 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8dc007c10 con 0x7fe8e0144640 2024-03-06T07:36:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.649+0000 7fe8d67fc640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe8c406b4d0 0x7fe8c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.649+0000 7fe8d67fc640 1 -- 172.21.15.27:0/1814311125 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe8dc089f60 con 0x7fe8e0144640 2024-03-06T07:36:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.649+0000 7fe8d67fc640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fe8c4071430 0x7fe8c4073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.649+0000 7fe8d67fc640 1 -- 172.21.15.27:0/1814311125 --> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8dc018ec0 con 0x7fe8c4071430 2024-03-06T07:36:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.649+0000 7fe8d67fc640 1 -- 172.21.15.27:0/1814311125 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8dc08c600 con 0x7fe8e0144640 2024-03-06T07:36:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.649+0000 7fe8e597e640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe8c406b4d0 0x7fe8c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.649+0000 7fe8e617f640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fe8c4071430 0x7fe8c4073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.649+0000 7fe8e617f640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fe8c4071430 0x7fe8c4073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.649+0000 7fe8e597e640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe8c406b4d0 0x7fe8c406d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0038000 tx=0x7fe8d0032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.650+0000 7fe8d67fc640 1 -- 172.21.15.27:0/1814311125 <== osd.2 v2:172.21.15.27:6808/3892975520 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe8dc018ec0 con 0x7fe8c4071430 2024-03-06T07:36:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.676+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 --> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe8e004f410 con 0x7fe8c4071430 2024-03-06T07:36:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.676+0000 7fe8d67fc640 1 -- 172.21.15.27:0/1814311125 <== osd.2 v2:172.21.15.27:6808/3892975520 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe8e004f410 con 0x7fe8c4071430 2024-03-06T07:36:05.678 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:36:05.678 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:36:05.678 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:36:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.676+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fe8c4071430 msgr2=0x7fe8c4073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.676+0000 7fe8e7be2640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.27:6808/3892975520,v1:172.21.15.27:6810/3892975520] conn(0x7fe8c4071430 0x7fe8c4073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.676+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe8c406b4d0 msgr2=0x7fe8c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.676+0000 7fe8e7be2640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe8c406b4d0 0x7fe8c406d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0038000 tx=0x7fe8d0032000 comp rx=0 tx=0).stop 2024-03-06T07:36:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.677+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 msgr2=0x7fe8e013c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.677+0000 7fe8e7be2640 1 --2- 172.21.15.27:0/1814311125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e0144640 0x7fe8e013c7f0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00c7f0 tx=0x7fe8dc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.677+0000 7fe8e617f640 1 -- 172.21.15.27:0/1814311125 reap_dead start 2024-03-06T07:36:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.677+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 shutdown_connections 2024-03-06T07:36:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.677+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 >> 172.21.15.27:0/1814311125 conn(0x7fe8e0066d00 msgr2=0x7fe8e00ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.677+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 shutdown_connections 2024-03-06T07:36:05.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.677+0000 7fe8e7be2640 1 -- 172.21.15.27:0/1814311125 wait complete. 2024-03-06T07:36:05.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-06T07:36:05.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.797+0000 7f45142fa640 1 Processor -- start 2024-03-06T07:36:05.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.797+0000 7f45142fa640 1 -- start start 2024-03-06T07:36:05.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.798+0000 7f45142fa640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f450c06b5c0 0x7f450c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.798+0000 7f45142fa640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c0b3340 con 0x7f450c0ae9b0 2024-03-06T07:36:05.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.798+0000 7f45142fa640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c0cee90 con 0x7f450c0b0e90 2024-03-06T07:36:05.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.798+0000 7f45142fa640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c0cfa00 con 0x7f450c06b5c0 2024-03-06T07:36:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.798+0000 7f4512897640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f450c0ae9b0 0x7f450c0d03a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49506/0 (socket says 172.21.15.27:49506) 2024-03-06T07:36:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.798+0000 7f4512897640 1 -- 172.21.15.27:0/2804052911 learned_addr learned my addr 172.21.15.27:0/2804052911 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.799+0000 7f4512096640 1 --2- 172.21.15.27:0/2804052911 >> v2:172.21.15.69:3300/0 conn(0x7f450c06b5c0 0x7f450c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.799+0000 7f4511094640 1 -- 172.21.15.27:0/2804052911 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476847393 0 0) 0x7f450c0b3340 con 0x7f450c0ae9b0 2024-03-06T07:36:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.799+0000 7f4511094640 1 -- 172.21.15.27:0/2804052911 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f8003520 con 0x7f450c0ae9b0 2024-03-06T07:36:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.799+0000 7f4511094640 1 -- 172.21.15.27:0/2804052911 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738267818 0 0) 0x7f450c0cee90 con 0x7f450c0b0e90 2024-03-06T07:36:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.799+0000 7f4511094640 1 -- 172.21.15.27:0/2804052911 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450c0b3340 con 0x7f450c0b0e90 2024-03-06T07:36:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.799+0000 7f4512096640 1 -- 172.21.15.27:0/2804052911 >> v1:172.21.15.69:6789/0 conn(0x7f450c0b0e90 legacy=0x7f450c06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.799+0000 7f4511094640 1 -- 172.21.15.27:0/2804052911 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2301307512 0 0) 0x7f44f8003520 con 0x7f450c0ae9b0 2024-03-06T07:36:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.799+0000 7f4512096640 1 -- 172.21.15.27:0/2804052911 >> v1:172.21.15.27:6789/0 conn(0x7f450c0ae9b0 legacy=0x7f450c0d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.799+0000 7f4512096640 1 -- 172.21.15.27:0/2804052911 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c0cfc50 con 0x7f450c06b5c0 2024-03-06T07:36:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.800+0000 7f4512096640 1 --2- 172.21.15.27:0/2804052911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c06b5c0 0x7f450c0b2e00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0097a0 tx=0x7f44fc02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8fcc660b29773ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.800+0000 7f4511094640 1 -- 172.21.15.27:0/2804052911 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc037070 con 0x7f450c06b5c0 2024-03-06T07:36:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.800+0000 7f4511094640 1 -- 172.21.15.27:0/2804052911 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc030510 con 0x7f450c06b5c0 2024-03-06T07:36:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.800+0000 7f4511094640 1 -- 172.21.15.27:0/2804052911 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504001a10 con 0x7f450c06b5c0 2024-03-06T07:36:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.800+0000 7f45142fa640 1 -- 172.21.15.27:0/2804052911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c06b5c0 msgr2=0x7f450c0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.800+0000 7f45142fa640 1 --2- 172.21.15.27:0/2804052911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c06b5c0 0x7f450c0b2e00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0097a0 tx=0x7f44fc02b790 comp rx=0 tx=0).stop 2024-03-06T07:36:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.801+0000 7f45142fa640 1 -- 172.21.15.27:0/2804052911 shutdown_connections 2024-03-06T07:36:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.801+0000 7f45142fa640 1 --2- 172.21.15.27:0/2804052911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c06b5c0 0x7f450c0b2e00 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.801+0000 7f45142fa640 1 -- 172.21.15.27:0/2804052911 >> 172.21.15.27:0/2804052911 conn(0x7f450c0b36d0 msgr2=0x7f450c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.801+0000 7f45142fa640 1 -- 172.21.15.27:0/2804052911 shutdown_connections 2024-03-06T07:36:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.801+0000 7f45142fa640 1 -- 172.21.15.27:0/2804052911 wait complete. 2024-03-06T07:36:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.802+0000 7f45142fa640 1 Processor -- start 2024-03-06T07:36:05.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.802+0000 7f45142fa640 1 -- start start 2024-03-06T07:36:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.802+0000 7f45142fa640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f450c06b5c0 0x7f450c0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.802+0000 7f45142fa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450c0ae9b0 0x7f450c0cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.802+0000 7f45142fa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0b0e90 0x7f450c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.802+0000 7f45142fa640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c06c910 con 0x7f450c0ae9b0 2024-03-06T07:36:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.802+0000 7f45142fa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c06c790 con 0x7f450c0b0e90 2024-03-06T07:36:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.802+0000 7f45142fa640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c0d0ab0 con 0x7f450c06b5c0 2024-03-06T07:36:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4511895640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450c0ae9b0 0x7f450c0cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4511895640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450c0ae9b0 0x7f450c0cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38530/0 (socket says 172.21.15.27:38530) 2024-03-06T07:36:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4511895640 1 -- 172.21.15.27:0/2530574716 learned_addr learned my addr 172.21.15.27:0/2530574716 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4512096640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f450c06b5c0 0x7f450c0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4512897640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0b0e90 0x7f450c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4511895640 1 -- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f450c06b5c0 msgr2=0x7f450c0c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4511895640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f450c06b5c0 0x7f450c0c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4511895640 1 -- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0b0e90 msgr2=0x7f450c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4511895640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0b0e90 0x7f450c0cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4511895640 1 -- 172.21.15.27:0/2530574716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c1f3e00 con 0x7f450c0ae9b0 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4511895640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450c0ae9b0 0x7f450c0cae30 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f450800c500 tx=0x7f450800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4502ffd640 1 -- 172.21.15.27:0/2530574716 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45080052d0 con 0x7f450c0ae9b0 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4502ffd640 1 -- 172.21.15.27:0/2530574716 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4508005490 con 0x7f450c0ae9b0 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f4502ffd640 1 -- 172.21.15.27:0/2530574716 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508012870 con 0x7f450c0ae9b0 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c1f40e0 con 0x7f450c0ae9b0 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.803+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c1f46a0 con 0x7f450c0ae9b0 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.804+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c068700 con 0x7f450c0ae9b0 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.805+0000 7f4502ffd640 1 -- 172.21.15.27:0/2530574716 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4508012a10 con 0x7f450c0ae9b0 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.805+0000 7f4502ffd640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44f006b500 0x7f44f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.806+0000 7f4502ffd640 1 -- 172.21.15.27:0/2530574716 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4508089b00 con 0x7f450c0ae9b0 2024-03-06T07:36:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.806+0000 7f4512096640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44f006b500 0x7f44f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.806+0000 7f4512096640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44f006b500 0x7f44f006d9c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f44fc034000 tx=0x7f44fc02f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.810+0000 7f4502ffd640 1 -- 172.21.15.27:0/2530574716 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45080585e0 con 0x7f450c0ae9b0 2024-03-06T07:36:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.992+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f450c0cdd10 con 0x7f44f006b500 2024-03-06T07:36:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.993+0000 7f4502ffd640 1 -- 172.21.15.27:0/2530574716 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f450c0cdd10 con 0x7f44f006b500 2024-03-06T07:36:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 2 to deep-scrub 2024-03-06T07:36:05.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.994+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44f006b500 msgr2=0x7f44f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44f006b500 0x7f44f006d9c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f44fc034000 tx=0x7f44fc02f000 comp rx=0 tx=0).stop 2024-03-06T07:36:05.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450c0ae9b0 msgr2=0x7f450c0cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450c0ae9b0 0x7f450c0cae30 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f450800c500 tx=0x7f450800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 shutdown_connections 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f44f006b500 0x7f44f006d9c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0b0e90 0x7f450c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450c0ae9b0 0x7f450c0cae30 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 --2- 172.21.15.27:0/2530574716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f450c06b5c0 0x7f450c0c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 >> 172.21.15.27:0/2530574716 conn(0x7f450c0b36d0 msgr2=0x7f450c15e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 shutdown_connections 2024-03-06T07:36:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:05.995+0000 7f45142fa640 1 -- 172.21.15.27:0/2530574716 wait complete. 2024-03-06T07:36:06.006 INFO:tasks.ceph:Scrubbing osd.3 2024-03-06T07:36:06.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:36:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.115+0000 7f24bcd79640 1 Processor -- start 2024-03-06T07:36:06.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.115+0000 7f24bcd79640 1 -- start start 2024-03-06T07:36:06.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.116+0000 7f24bcd79640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b8071540 0x7f24b8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.116+0000 7f24bcd79640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24b80cee80 0x7f24b8168900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.116+0000 7f24bcd79640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b8071e60 con 0x7f24b80b2d30 2024-03-06T07:36:06.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.116+0000 7f24bcd79640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b80d3230 con 0x7f24b80cee80 2024-03-06T07:36:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.116+0000 7f24bcd79640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b80d4960 con 0x7f24b8071540 2024-03-06T07:36:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.116+0000 7f24b6d9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24b80cee80 0x7f24b8168900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.116+0000 7f24b5d9b640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f24b80b2d30 0x7f24b80ce770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49508/0 (socket says 172.21.15.27:49508) 2024-03-06T07:36:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.116+0000 7f24b5d9b640 1 -- 172.21.15.27:0/1697290942 learned_addr learned my addr 172.21.15.27:0/1697290942 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.117+0000 7f24b659c640 1 --2- 172.21.15.27:0/1697290942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b8071540 0x7f24b8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.117+0000 7f24b6d9d640 1 -- 172.21.15.27:0/1697290942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b8071540 msgr2=0x7f24b8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.117+0000 7f24b6d9d640 1 --2- 172.21.15.27:0/1697290942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b8071540 0x7f24b8071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.117+0000 7f24b6d9d640 1 -- 172.21.15.27:0/1697290942 >> v1:172.21.15.27:6789/0 conn(0x7f24b80b2d30 legacy=0x7f24b80ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.117+0000 7f24b559a640 1 -- 172.21.15.27:0/1697290942 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942963148 0 0) 0x7f24b8071e60 con 0x7f24b80b2d30 2024-03-06T07:36:06.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.117+0000 7f24b6d9d640 1 -- 172.21.15.27:0/1697290942 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b80d36c0 con 0x7f24b80cee80 2024-03-06T07:36:06.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24b6d9d640 1 --2- 172.21.15.27:0/1697290942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 0x7f24b8168900 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009d90 tx=0x7f24a802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3976b8fddcfc31aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24b559a640 1 -- 172.21.15.27:0/1697290942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8039070 con 0x7f24b80cee80 2024-03-06T07:36:06.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24b559a640 1 -- 172.21.15.27:0/1697290942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a802cc50 con 0x7f24b80cee80 2024-03-06T07:36:06.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24b559a640 1 -- 172.21.15.27:0/1697290942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8034440 con 0x7f24b80cee80 2024-03-06T07:36:06.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24bcd79640 1 -- 172.21.15.27:0/1697290942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 msgr2=0x7f24b8168900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24bcd79640 1 --2- 172.21.15.27:0/1697290942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 0x7f24b8168900 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009d90 tx=0x7f24a802bea0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24bcd79640 1 -- 172.21.15.27:0/1697290942 shutdown_connections 2024-03-06T07:36:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24bcd79640 1 --2- 172.21.15.27:0/1697290942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 0x7f24b8168900 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24bcd79640 1 --2- 172.21.15.27:0/1697290942 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b8071540 0x7f24b8071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24bcd79640 1 -- 172.21.15.27:0/1697290942 >> 172.21.15.27:0/1697290942 conn(0x7f24b80d3d60 msgr2=0x7f24b80d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24bcd79640 1 -- 172.21.15.27:0/1697290942 shutdown_connections 2024-03-06T07:36:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.118+0000 7f24bcd79640 1 -- 172.21.15.27:0/1697290942 wait complete. 2024-03-06T07:36:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.120+0000 7f24bcd79640 1 Processor -- start 2024-03-06T07:36:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.120+0000 7f24bcd79640 1 -- start start 2024-03-06T07:36:06.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.120+0000 7f24bcd79640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b8071540 0x7f24b814e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.120+0000 7f24bcd79640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b80b2d30 0x7f24b80cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.120+0000 7f24bcd79640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 0x7f24b80cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.120+0000 7f24bcd79640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b80d2be0 con 0x7f24b80cee80 2024-03-06T07:36:06.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.120+0000 7f24bcd79640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b80d2900 con 0x7f24b8071540 2024-03-06T07:36:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.120+0000 7f24bcd79640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b80d2ec0 con 0x7f24b80b2d30 2024-03-06T07:36:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.120+0000 7f24b6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 0x7f24b80cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 0x7f24b80cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38544/0 (socket says 172.21.15.27:38544) 2024-03-06T07:36:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b6d9d640 1 -- 172.21.15.27:0/3815998798 learned_addr learned my addr 172.21.15.27:0/3815998798 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b659c640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b8071540 0x7f24b814e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b5d9b640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b80b2d30 0x7f24b80cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b6d9d640 1 -- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b80b2d30 msgr2=0x7f24b80cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b6d9d640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f24b80b2d30 0x7f24b80cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b6d9d640 1 -- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b8071540 msgr2=0x7f24b814e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b6d9d640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b8071540 0x7f24b814e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b6d9d640 1 -- 172.21.15.27:0/3815998798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b80ccd50 con 0x7f24b80cee80 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b659c640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24b8071540 0x7f24b814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24b6d9d640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 0x7f24b80cc6d0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f24a8037040 tx=0x7f24a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24a6ffd640 1 -- 172.21.15.27:0/3815998798 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8039050 con 0x7f24b80cee80 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24a6ffd640 1 -- 172.21.15.27:0/3815998798 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a803a040 con 0x7f24b80cee80 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24a6ffd640 1 -- 172.21.15.27:0/3815998798 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a802ca50 con 0x7f24b80cee80 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b8156c90 con 0x7f24b80cee80 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.121+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b8157250 con 0x7f24b80cee80 2024-03-06T07:36:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.122+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f24b8072070 con 0x7f24b80cee80 2024-03-06T07:36:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.123+0000 7f24a6ffd640 1 -- 172.21.15.27:0/3815998798 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24a8047030 con 0x7f24b80cee80 2024-03-06T07:36:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.123+0000 7f24a6ffd640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f249806b4d0 0x7f249806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.123+0000 7f24a6ffd640 1 -- 172.21.15.27:0/3815998798 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f24a80abdd0 con 0x7f24b80cee80 2024-03-06T07:36:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.123+0000 7f24a6ffd640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f2498071430 0x7f2498073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.123+0000 7f24a6ffd640 1 -- 172.21.15.27:0/3815998798 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24a80091a0 con 0x7f2498071430 2024-03-06T07:36:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.123+0000 7f24a6ffd640 1 -- 172.21.15.27:0/3815998798 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24a80ae4d0 con 0x7f24b80cee80 2024-03-06T07:36:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.123+0000 7f24b659c640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f249806b4d0 0x7f249806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.123+0000 7f24b5d9b640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f2498071430 0x7f2498073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.124+0000 7f24b5d9b640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f2498071430 0x7f2498073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.124+0000 7f24b659c640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f249806b4d0 0x7f249806d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f24b0006fd0 tx=0x7f24b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.124+0000 7f24a6ffd640 1 -- 172.21.15.27:0/3815998798 <== osd.3 v2:172.21.15.27:6824/4277055095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f24a80091a0 con 0x7f2498071430 2024-03-06T07:36:06.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.137+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 --> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f24b804f410 con 0x7f2498071430 2024-03-06T07:36:06.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24a6ffd640 1 -- 172.21.15.27:0/3815998798 <== osd.3 v2:172.21.15.27:6824/4277055095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f24b804f410 con 0x7f2498071430 2024-03-06T07:36:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:36:06.140 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:36:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:36:06.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f2498071430 msgr2=0x7f2498073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.27:6824/4277055095,v1:172.21.15.27:6825/4277055095] conn(0x7f2498071430 0x7f2498073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f249806b4d0 msgr2=0x7f249806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f249806b4d0 0x7f249806d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f24b0006fd0 tx=0x7f24b0008040 comp rx=0 tx=0).stop 2024-03-06T07:36:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 msgr2=0x7f24b80cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 --2- 172.21.15.27:0/3815998798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24b80cee80 0x7f24b80cc6d0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f24a8037040 tx=0x7f24a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24b6d9d640 1 -- 172.21.15.27:0/3815998798 reap_dead start 2024-03-06T07:36:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 shutdown_connections 2024-03-06T07:36:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 >> 172.21.15.27:0/3815998798 conn(0x7f24b80d3d60 msgr2=0x7f24b80d1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 shutdown_connections 2024-03-06T07:36:06.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.138+0000 7f24bcd79640 1 -- 172.21.15.27:0/3815998798 wait complete. 2024-03-06T07:36:06.151 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-06T07:36:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.256+0000 7fdc1a7b6640 1 Processor -- start 2024-03-06T07:36:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.256+0000 7fdc1a7b6640 1 -- start start 2024-03-06T07:36:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc1a7b6640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc1406ed10 0x7fdc1406af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc1a7b6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdc140cd890 0x7fdc1415d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc1a7b6640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc140cc0e0 con 0x7fdc140b1020 2024-03-06T07:36:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc1a7b6640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc140d28b0 con 0x7fdc1406ed10 2024-03-06T07:36:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc1a7b6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc140cf320 con 0x7fdc140cd890 2024-03-06T07:36:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc137fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc1406ed10 0x7fdc1406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc137fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc1406ed10 0x7fdc1406af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38550/0 (socket says 172.21.15.27:38550) 2024-03-06T07:36:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc137fe640 1 -- 172.21.15.27:0/3247123176 learned_addr learned my addr 172.21.15.27:0/3247123176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc13fff640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdc140b1020 0x7fdc140cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49510/0 (socket says 172.21.15.27:49510) 2024-03-06T07:36:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc18d53640 1 --2- 172.21.15.27:0/3247123176 >> v2:172.21.15.69:3300/0 conn(0x7fdc140cd890 0x7fdc1415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc137fe640 1 -- 172.21.15.27:0/3247123176 >> v2:172.21.15.69:3300/0 conn(0x7fdc140cd890 msgr2=0x7fdc1415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc137fe640 1 --2- 172.21.15.27:0/3247123176 >> v2:172.21.15.69:3300/0 conn(0x7fdc140cd890 0x7fdc1415d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc137fe640 1 -- 172.21.15.27:0/3247123176 >> v1:172.21.15.27:6789/0 conn(0x7fdc140b1020 legacy=0x7fdc140cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.257+0000 7fdc137fe640 1 -- 172.21.15.27:0/3247123176 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc140cc3c0 con 0x7fdc1406ed10 2024-03-06T07:36:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.258+0000 7fdc137fe640 1 --2- 172.21.15.27:0/3247123176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc1406ed10 0x7fdc1406af20 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fdc140cc070 tx=0x7fdc0802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8724d9066bb7658c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.258+0000 7fdc12ffd640 1 -- 172.21.15.27:0/3247123176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08034070 con 0x7fdc1406ed10 2024-03-06T07:36:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.258+0000 7fdc12ffd640 1 -- 172.21.15.27:0/3247123176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc08030860 con 0x7fdc1406ed10 2024-03-06T07:36:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.258+0000 7fdc12ffd640 1 -- 172.21.15.27:0/3247123176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08030c30 con 0x7fdc1406ed10 2024-03-06T07:36:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.258+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3247123176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc1406ed10 msgr2=0x7fdc1406af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.258+0000 7fdc1a7b6640 1 --2- 172.21.15.27:0/3247123176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc1406ed10 0x7fdc1406af20 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fdc140cc070 tx=0x7fdc0802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.259+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3247123176 shutdown_connections 2024-03-06T07:36:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.259+0000 7fdc1a7b6640 1 --2- 172.21.15.27:0/3247123176 >> v2:172.21.15.69:3300/0 conn(0x7fdc140cd890 0x7fdc1415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.259+0000 7fdc1a7b6640 1 --2- 172.21.15.27:0/3247123176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc1406ed10 0x7fdc1406af20 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.259+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3247123176 >> 172.21.15.27:0/3247123176 conn(0x7fdc140d1a90 msgr2=0x7fdc140d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.259+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3247123176 shutdown_connections 2024-03-06T07:36:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.259+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3247123176 wait complete. 2024-03-06T07:36:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.260+0000 7fdc1a7b6640 1 Processor -- start 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.260+0000 7fdc1a7b6640 1 -- start start 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc1a7b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc1406ed10 0x7fdc14154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc1a7b6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc140b1020 0x7fdc140c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc1a7b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc140cd890 0x7fdc140c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc1a7b6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc140ccc60 con 0x7fdc140cd890 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc1a7b6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc140cc980 con 0x7fdc140b1020 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc1a7b6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc140cdca0 con 0x7fdc1406ed10 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc18d53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc140cd890 0x7fdc140c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc18d53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc140cd890 0x7fdc140c5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38562/0 (socket says 172.21.15.27:38562) 2024-03-06T07:36:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc18d53640 1 -- 172.21.15.27:0/3357799005 learned_addr learned my addr 172.21.15.27:0/3357799005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:06.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc13fff640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc1406ed10 0x7fdc14154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc137fe640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc140b1020 0x7fdc140c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc18d53640 1 -- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc1406ed10 msgr2=0x7fdc14154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc18d53640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc1406ed10 0x7fdc14154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc18d53640 1 -- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc140b1020 msgr2=0x7fdc140c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc18d53640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc140b1020 0x7fdc140c55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc18d53640 1 -- 172.21.15.27:0/3357799005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1415e3d0 con 0x7fdc140cd890 2024-03-06T07:36:06.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc18d53640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc140cd890 0x7fdc140c5b00 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c007e60 tx=0x7fdc0c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc10ff9640 1 -- 172.21.15.27:0/3357799005 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c014420 con 0x7fdc140cd890 2024-03-06T07:36:06.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc10ff9640 1 -- 172.21.15.27:0/3357799005 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c019070 con 0x7fdc140cd890 2024-03-06T07:36:06.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc140c6110 con 0x7fdc140cd890 2024-03-06T07:36:06.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.261+0000 7fdc10ff9640 1 -- 172.21.15.27:0/3357799005 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c014820 con 0x7fdc140cd890 2024-03-06T07:36:06.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.262+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc140c66d0 con 0x7fdc140cd890 2024-03-06T07:36:06.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.262+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc14067ff0 con 0x7fdc140cd890 2024-03-06T07:36:06.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.263+0000 7fdc10ff9640 1 -- 172.21.15.27:0/3357799005 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc0c0149c0 con 0x7fdc140cd890 2024-03-06T07:36:06.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.263+0000 7fdc10ff9640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b4d0 0x7fdc0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.264+0000 7fdc10ff9640 1 -- 172.21.15.27:0/3357799005 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdc0c08a740 con 0x7fdc140cd890 2024-03-06T07:36:06.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.264+0000 7fdc13fff640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b4d0 0x7fdc0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.264+0000 7fdc13fff640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b4d0 0x7fdc0006d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fdc04000f60 tx=0x7fdc04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.268+0000 7fdc10ff9640 1 -- 172.21.15.27:0/3357799005 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc0c0598f0 con 0x7fdc140cd890 2024-03-06T07:36:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.449+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc1406f2f0 con 0x7fdc0006b4d0 2024-03-06T07:36:06.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.449+0000 7fdc10ff9640 1 -- 172.21.15.27:0/3357799005 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdc1406f2f0 con 0x7fdc0006b4d0 2024-03-06T07:36:06.452 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 3 to deep-scrub 2024-03-06T07:36:06.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b4d0 msgr2=0x7fdc0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b4d0 0x7fdc0006d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fdc04000f60 tx=0x7fdc04009040 comp rx=0 tx=0).stop 2024-03-06T07:36:06.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc140cd890 msgr2=0x7fdc140c5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc140cd890 0x7fdc140c5b00 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c007e60 tx=0x7fdc0c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 shutdown_connections 2024-03-06T07:36:06.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdc0006b4d0 0x7fdc0006d990 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc140cd890 0x7fdc140c5b00 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc140b1020 0x7fdc140c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 --2- 172.21.15.27:0/3357799005 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc1406ed10 0x7fdc14154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 >> 172.21.15.27:0/3357799005 conn(0x7fdc140d1a90 msgr2=0x7fdc140ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 shutdown_connections 2024-03-06T07:36:06.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.451+0000 7fdc1a7b6640 1 -- 172.21.15.27:0/3357799005 wait complete. 2024-03-06T07:36:06.464 INFO:tasks.ceph:Scrubbing osd.4 2024-03-06T07:36:06.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:36:06.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.573+0000 7f6d84a0e640 1 Processor -- start 2024-03-06T07:36:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.573+0000 7f6d84a0e640 1 -- start start 2024-03-06T07:36:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d84a0e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6d8015f5c0 0x7f6d800cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d84a0e640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d800cff40 0x7f6d800d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d84a0e640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d800b33b0 con 0x7f6d800b1470 2024-03-06T07:36:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d84a0e640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d800b0af0 con 0x7f6d800cff40 2024-03-06T07:36:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d84a0e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d800b0d40 con 0x7f6d8015f5c0 2024-03-06T07:36:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d77fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d800cff40 0x7f6d800d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d77fff640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d800cff40 0x7f6d800d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38572/0 (socket says 172.21.15.27:38572) 2024-03-06T07:36:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d77fff640 1 -- 172.21.15.27:0/233264131 learned_addr learned my addr 172.21.15.27:0/233264131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d7ed9d640 1 --1- 172.21.15.27:0/233264131 >> v1:172.21.15.27:6789/0 conn(0x7f6d800b1470 0x7f6d800b18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49514/0 (socket says 172.21.15.27:49514) 2024-03-06T07:36:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d7e59c640 1 --2- 172.21.15.27:0/233264131 >> v2:172.21.15.69:3300/0 conn(0x7f6d8015f5c0 0x7f6d800cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d77fff640 1 -- 172.21.15.27:0/233264131 >> v2:172.21.15.69:3300/0 conn(0x7f6d8015f5c0 msgr2=0x7f6d800cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d77fff640 1 --2- 172.21.15.27:0/233264131 >> v2:172.21.15.69:3300/0 conn(0x7f6d8015f5c0 0x7f6d800cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d77fff640 1 -- 172.21.15.27:0/233264131 >> v1:172.21.15.27:6789/0 conn(0x7f6d800b1470 legacy=0x7f6d800b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d7dd9b640 1 -- 172.21.15.27:0/233264131 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233357677 0 0) 0x7f6d800b33b0 con 0x7f6d800b1470 2024-03-06T07:36:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d77fff640 1 -- 172.21.15.27:0/233264131 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d800b1020 con 0x7f6d800cff40 2024-03-06T07:36:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.574+0000 7f6d7e59c640 1 --2- 172.21.15.27:0/233264131 >> v2:172.21.15.69:3300/0 conn(0x7f6d8015f5c0 0x7f6d800cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.575+0000 7f6d77fff640 1 --2- 172.21.15.27:0/233264131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d800cff40 0x7f6d800d03d0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f6d80071e70 tx=0x7f6d6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de706183ce65ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.575+0000 7f6d7dd9b640 1 -- 172.21.15.27:0/233264131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d68035020 con 0x7f6d800cff40 2024-03-06T07:36:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.575+0000 7f6d7dd9b640 1 -- 172.21.15.27:0/233264131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d68031870 con 0x7f6d800cff40 2024-03-06T07:36:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.575+0000 7f6d7dd9b640 1 -- 172.21.15.27:0/233264131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d68031c40 con 0x7f6d800cff40 2024-03-06T07:36:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.576+0000 7f6d84a0e640 1 -- 172.21.15.27:0/233264131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d800cff40 msgr2=0x7f6d800d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.576+0000 7f6d84a0e640 1 --2- 172.21.15.27:0/233264131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d800cff40 0x7f6d800d03d0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f6d80071e70 tx=0x7f6d6802b910 comp rx=0 tx=0).stop 2024-03-06T07:36:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.576+0000 7f6d84a0e640 1 -- 172.21.15.27:0/233264131 shutdown_connections 2024-03-06T07:36:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.576+0000 7f6d84a0e640 1 --2- 172.21.15.27:0/233264131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d800cff40 0x7f6d800d03d0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.576+0000 7f6d84a0e640 1 --2- 172.21.15.27:0/233264131 >> v2:172.21.15.69:3300/0 conn(0x7f6d8015f5c0 0x7f6d800cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.576+0000 7f6d84a0e640 1 -- 172.21.15.27:0/233264131 >> 172.21.15.27:0/233264131 conn(0x7f6d800b36d0 msgr2=0x7f6d800b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.576+0000 7f6d84a0e640 1 -- 172.21.15.27:0/233264131 shutdown_connections 2024-03-06T07:36:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.576+0000 7f6d84a0e640 1 -- 172.21.15.27:0/233264131 wait complete. 2024-03-06T07:36:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.577+0000 7f6d84a0e640 1 Processor -- start 2024-03-06T07:36:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.577+0000 7f6d84a0e640 1 -- start start 2024-03-06T07:36:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d84a0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d800b1470 0x7f6d8015b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d84a0e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d800cff40 0x7f6d800c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d84a0e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d8015f5c0 0x7f6d8015dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d84a0e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8015f9a0 con 0x7f6d8015f5c0 2024-03-06T07:36:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d84a0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800ced80 con 0x7f6d800b1470 2024-03-06T07:36:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d84a0e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800cf2a0 con 0x7f6d800cff40 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d8015f5c0 0x7f6d8015dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d8015f5c0 0x7f6d8015dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38588/0 (socket says 172.21.15.27:38588) 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7ed9d640 1 -- 172.21.15.27:0/225902654 learned_addr learned my addr 172.21.15.27:0/225902654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7e59c640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d800b1470 0x7f6d8015b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d77fff640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d800cff40 0x7f6d800c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7ed9d640 1 -- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d800cff40 msgr2=0x7f6d800c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7ed9d640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d800cff40 0x7f6d800c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7ed9d640 1 -- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d800b1470 msgr2=0x7f6d8015b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7ed9d640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d800b1470 0x7f6d8015b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7ed9d640 1 -- 172.21.15.27:0/225902654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8015b770 con 0x7f6d8015f5c0 2024-03-06T07:36:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7e59c640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d800b1470 0x7f6d8015b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d77fff640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6d800cff40 0x7f6d800c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.578+0000 7f6d7ed9d640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d8015f5c0 0x7f6d8015dbc0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800c780 tx=0x7f6d7800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.579+0000 7f6d76ffd640 1 -- 172.21.15.27:0/225902654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78018070 con 0x7f6d8015f5c0 2024-03-06T07:36:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.579+0000 7f6d76ffd640 1 -- 172.21.15.27:0/225902654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d78004d60 con 0x7f6d8015f5c0 2024-03-06T07:36:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.579+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d800c9690 con 0x7f6d8015f5c0 2024-03-06T07:36:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.579+0000 7f6d76ffd640 1 -- 172.21.15.27:0/225902654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7800a860 con 0x7f6d8015f5c0 2024-03-06T07:36:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.579+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d800c9c50 con 0x7f6d8015f5c0 2024-03-06T07:36:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.580+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6d8009fd30 con 0x7f6d8015f5c0 2024-03-06T07:36:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.580+0000 7f6d76ffd640 1 -- 172.21.15.27:0/225902654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d7801b070 con 0x7f6d8015f5c0 2024-03-06T07:36:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.580+0000 7f6d76ffd640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d6006b4d0 0x7f6d6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.581+0000 7f6d76ffd640 1 -- 172.21.15.27:0/225902654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6d78089340 con 0x7f6d8015f5c0 2024-03-06T07:36:06.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.581+0000 7f6d76ffd640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f6d60071430 0x7f6d60073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.581+0000 7f6d76ffd640 1 -- 172.21.15.27:0/225902654 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d78017ec0 con 0x7f6d60071430 2024-03-06T07:36:06.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.581+0000 7f6d76ffd640 1 -- 172.21.15.27:0/225902654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d78090050 con 0x7f6d8015f5c0 2024-03-06T07:36:06.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.581+0000 7f6d7e59c640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d6006b4d0 0x7f6d6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.581+0000 7f6d77fff640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f6d60071430 0x7f6d60073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.581+0000 7f6d7e59c640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d6006b4d0 0x7f6d6006d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000aa30 tx=0x7f6d70008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.581+0000 7f6d77fff640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f6d60071430 0x7f6d60073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.582+0000 7f6d76ffd640 1 -- 172.21.15.27:0/225902654 <== osd.4 v2:172.21.15.69:6808/1626703870 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6d78017ec0 con 0x7f6d60071430 2024-03-06T07:36:06.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.595+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 --> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6d8004f410 con 0x7f6d60071430 2024-03-06T07:36:06.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d76ffd640 1 -- 172.21.15.27:0/225902654 <== osd.4 v2:172.21.15.69:6808/1626703870 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6d8004f410 con 0x7f6d60071430 2024-03-06T07:36:06.628 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:36:06.628 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:36:06.628 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:36:06.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f6d60071430 msgr2=0x7f6d60073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6808/1626703870,v1:172.21.15.69:6809/1626703870] conn(0x7f6d60071430 0x7f6d60073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d6006b4d0 msgr2=0x7f6d6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6d6006b4d0 0x7f6d6006d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000aa30 tx=0x7f6d70008040 comp rx=0 tx=0).stop 2024-03-06T07:36:06.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d8015f5c0 msgr2=0x7f6d8015dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 --2- 172.21.15.27:0/225902654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d8015f5c0 0x7f6d8015dbc0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800c780 tx=0x7f6d7800cb80 comp rx=0 tx=0).stop 2024-03-06T07:36:06.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d7ed9d640 1 -- 172.21.15.27:0/225902654 reap_dead start 2024-03-06T07:36:06.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 shutdown_connections 2024-03-06T07:36:06.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 >> 172.21.15.27:0/225902654 conn(0x7f6d800b36d0 msgr2=0x7f6d8015e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 shutdown_connections 2024-03-06T07:36:06.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.596+0000 7f6d84a0e640 1 -- 172.21.15.27:0/225902654 wait complete. 2024-03-06T07:36:06.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-06T07:36:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.736+0000 7fd0096fb640 1 Processor -- start 2024-03-06T07:36:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.736+0000 7fd0096fb640 1 -- start start 2024-03-06T07:36:06.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0096fb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd004158520 0x7fd00406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0096fb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0040aee10 0x7fd0040b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0096fb640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd004071240 con 0x7fd00406dd60 2024-03-06T07:36:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0096fb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00406e1c0 con 0x7fd0040aee10 2024-03-06T07:36:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0096fb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd004071b70 con 0x7fd004158520 2024-03-06T07:36:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0040aee10 0x7fd0040b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0027fc640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd00406dd60 0x7fd0040ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49516/0 (socket says 172.21.15.27:49516) 2024-03-06T07:36:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0040aee10 0x7fd0040b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38594/0 (socket says 172.21.15.27:38594) 2024-03-06T07:36:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0027fc640 1 -- 172.21.15.27:0/2086465028 learned_addr learned my addr 172.21.15.27:0/2086465028 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd002ffd640 1 --2- 172.21.15.27:0/2086465028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd004158520 0x7fd00406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0037fe640 1 -- 172.21.15.27:0/2086465028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd004158520 msgr2=0x7fd00406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0037fe640 1 --2- 172.21.15.27:0/2086465028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd004158520 0x7fd00406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0037fe640 1 -- 172.21.15.27:0/2086465028 >> v1:172.21.15.27:6789/0 conn(0x7fd00406dd60 legacy=0x7fd0040ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd001ffb640 1 -- 172.21.15.27:0/2086465028 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172708360 0 0) 0x7fd004071240 con 0x7fd00406dd60 2024-03-06T07:36:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.737+0000 7fd0037fe640 1 -- 172.21.15.27:0/2086465028 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd004071520 con 0x7fd0040aee10 2024-03-06T07:36:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.738+0000 7fd0037fe640 1 --2- 172.21.15.27:0/2086465028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0040aee10 0x7fd0040b3640 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009d90 tx=0x7fcffc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8b0d24e7ec1aa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.738+0000 7fd001ffb640 1 -- 172.21.15.27:0/2086465028 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc039070 con 0x7fd0040aee10 2024-03-06T07:36:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.738+0000 7fd001ffb640 1 -- 172.21.15.27:0/2086465028 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc02cc50 con 0x7fd0040aee10 2024-03-06T07:36:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.738+0000 7fd001ffb640 1 -- 172.21.15.27:0/2086465028 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc034440 con 0x7fd0040aee10 2024-03-06T07:36:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.739+0000 7fd0096fb640 1 -- 172.21.15.27:0/2086465028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0040aee10 msgr2=0x7fd0040b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.739+0000 7fd0096fb640 1 --2- 172.21.15.27:0/2086465028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0040aee10 0x7fd0040b3640 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009d90 tx=0x7fcffc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.739+0000 7fd0096fb640 1 -- 172.21.15.27:0/2086465028 shutdown_connections 2024-03-06T07:36:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.739+0000 7fd0096fb640 1 --2- 172.21.15.27:0/2086465028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0040aee10 0x7fd0040b3640 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.739+0000 7fd0096fb640 1 --2- 172.21.15.27:0/2086465028 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd004158520 0x7fd00406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.739+0000 7fd0096fb640 1 -- 172.21.15.27:0/2086465028 >> 172.21.15.27:0/2086465028 conn(0x7fd004066d00 msgr2=0x7fd0040670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.739+0000 7fd0096fb640 1 -- 172.21.15.27:0/2086465028 shutdown_connections 2024-03-06T07:36:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.739+0000 7fd0096fb640 1 -- 172.21.15.27:0/2086465028 wait complete. 2024-03-06T07:36:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.740+0000 7fd0096fb640 1 Processor -- start 2024-03-06T07:36:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.740+0000 7fd0096fb640 1 -- start start 2024-03-06T07:36:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd0096fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00406dd60 0x7fd0040af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd0096fb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0040aee10 0x7fd004154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd0096fb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd004158520 0x7fd004148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd0096fb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd004158900 con 0x7fd00406dd60 2024-03-06T07:36:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd0096fb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd004072000 con 0x7fd004158520 2024-03-06T07:36:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd0096fb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd004158a80 con 0x7fd0040aee10 2024-03-06T07:36:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd002ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00406dd60 0x7fd0040af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd002ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00406dd60 0x7fd0040af580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38610/0 (socket says 172.21.15.27:38610) 2024-03-06T07:36:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd002ffd640 1 -- 172.21.15.27:0/1563498838 learned_addr learned my addr 172.21.15.27:0/1563498838 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd0037fe640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd004158520 0x7fd004148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.741+0000 7fd0027fc640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0040aee10 0x7fd004154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fd002ffd640 1 -- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0040aee10 msgr2=0x7fd004154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fd002ffd640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0040aee10 0x7fd004154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fd002ffd640 1 -- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd004158520 msgr2=0x7fd004148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fd002ffd640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd004158520 0x7fd004148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fd002ffd640 1 -- 172.21.15.27:0/1563498838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd004155cc0 con 0x7fd00406dd60 2024-03-06T07:36:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fd002ffd640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00406dd60 0x7fd0040af580 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fcff400c7f0 tx=0x7fcff400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fcffb7fe640 1 -- 172.21.15.27:0/1563498838 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4015070 con 0x7fd00406dd60 2024-03-06T07:36:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fcffb7fe640 1 -- 172.21.15.27:0/1563498838 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff4004220 con 0x7fd00406dd60 2024-03-06T07:36:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd004154c20 con 0x7fd00406dd60 2024-03-06T07:36:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fcffb7fe640 1 -- 172.21.15.27:0/1563498838 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4007620 con 0x7fd00406dd60 2024-03-06T07:36:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.742+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0041554c0 con 0x7fd00406dd60 2024-03-06T07:36:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.743+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd004067ff0 con 0x7fd00406dd60 2024-03-06T07:36:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.744+0000 7fcffb7fe640 1 -- 172.21.15.27:0/1563498838 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcff40079d0 con 0x7fd00406dd60 2024-03-06T07:36:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.744+0000 7fcffb7fe640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfe406b500 0x7fcfe406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.744+0000 7fcffb7fe640 1 -- 172.21.15.27:0/1563498838 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fcff4089e50 con 0x7fd00406dd60 2024-03-06T07:36:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.744+0000 7fd0027fc640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfe406b500 0x7fcfe406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.745+0000 7fd0027fc640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfe406b500 0x7fcfe406d9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd0040d0230 tx=0x7fcfec007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.749+0000 7fcffb7fe640 1 -- 172.21.15.27:0/1563498838 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcff4058930 con 0x7fd00406dd60 2024-03-06T07:36:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.930+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0040b31f0 con 0x7fcfe406b500 2024-03-06T07:36:06.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.930+0000 7fcffb7fe640 1 -- 172.21.15.27:0/1563498838 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd0040b31f0 con 0x7fcfe406b500 2024-03-06T07:36:06.933 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 4 to deep-scrub 2024-03-06T07:36:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfe406b500 msgr2=0x7fcfe406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfe406b500 0x7fcfe406d9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd0040d0230 tx=0x7fcfec007d50 comp rx=0 tx=0).stop 2024-03-06T07:36:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00406dd60 msgr2=0x7fd0040af580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00406dd60 0x7fd0040af580 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fcff400c7f0 tx=0x7fcff400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 shutdown_connections 2024-03-06T07:36:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fcfe406b500 0x7fcfe406d9c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd004158520 0x7fd004148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd0040aee10 0x7fd004154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 --2- 172.21.15.27:0/1563498838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00406dd60 0x7fd0040af580 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 >> 172.21.15.27:0/1563498838 conn(0x7fd004066d00 msgr2=0x7fd00406c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 shutdown_connections 2024-03-06T07:36:06.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:06.932+0000 7fd0096fb640 1 -- 172.21.15.27:0/1563498838 wait complete. 2024-03-06T07:36:06.945 INFO:tasks.ceph:Scrubbing osd.5 2024-03-06T07:36:06.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:36:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.054+0000 7fc085fb5640 1 Processor -- start 2024-03-06T07:36:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.054+0000 7fc085fb5640 1 -- start start 2024-03-06T07:36:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc085fb5640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc080144640 0x7fc08006e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc085fb5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0800ce900 0x7fc0800ced70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc085fb5640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08015bc40 con 0x7fc08006e940 2024-03-06T07:36:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc085fb5640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0800cf2b0 con 0x7fc080144640 2024-03-06T07:36:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc085fb5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08006df70 con 0x7fc0800ce900 2024-03-06T07:36:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc07f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc080144640 0x7fc08006e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc07effd640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc08006e940 0x7fc08015b130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49530/0 (socket says 172.21.15.27:49530) 2024-03-06T07:36:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc07f7fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc080144640 0x7fc08006e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38626/0 (socket says 172.21.15.27:38626) 2024-03-06T07:36:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc07effd640 1 -- 172.21.15.27:0/2274302035 learned_addr learned my addr 172.21.15.27:0/2274302035 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc07ffff640 1 --2- 172.21.15.27:0/2274302035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0800ce900 0x7fc0800ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc07f7fe640 1 -- 172.21.15.27:0/2274302035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0800ce900 msgr2=0x7fc0800ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc07f7fe640 1 --2- 172.21.15.27:0/2274302035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0800ce900 0x7fc0800ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc07f7fe640 1 -- 172.21.15.27:0/2274302035 >> v1:172.21.15.27:6789/0 conn(0x7fc08006e940 legacy=0x7fc08015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.055+0000 7fc07f7fe640 1 -- 172.21.15.27:0/2274302035 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08015bf20 con 0x7fc080144640 2024-03-06T07:36:07.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.056+0000 7fc07f7fe640 1 --2- 172.21.15.27:0/2274302035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc080144640 0x7fc08006e400 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fc0700097a0 tx=0x7fc07002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1f2b04b4066d5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.056+0000 7fc07e7fc640 1 -- 172.21.15.27:0/2274302035 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070035020 con 0x7fc080144640 2024-03-06T07:36:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.056+0000 7fc07e7fc640 1 -- 172.21.15.27:0/2274302035 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc070031830 con 0x7fc080144640 2024-03-06T07:36:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.056+0000 7fc07e7fc640 1 -- 172.21.15.27:0/2274302035 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070031c00 con 0x7fc080144640 2024-03-06T07:36:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.057+0000 7fc085fb5640 1 -- 172.21.15.27:0/2274302035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc080144640 msgr2=0x7fc08006e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.057+0000 7fc085fb5640 1 --2- 172.21.15.27:0/2274302035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc080144640 0x7fc08006e400 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fc0700097a0 tx=0x7fc07002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.057+0000 7fc085fb5640 1 -- 172.21.15.27:0/2274302035 shutdown_connections 2024-03-06T07:36:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.057+0000 7fc085fb5640 1 --2- 172.21.15.27:0/2274302035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc0800ce900 0x7fc0800ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.057+0000 7fc085fb5640 1 --2- 172.21.15.27:0/2274302035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc080144640 0x7fc08006e400 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.057+0000 7fc085fb5640 1 -- 172.21.15.27:0/2274302035 >> 172.21.15.27:0/2274302035 conn(0x7fc080066d00 msgr2=0x7fc080067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.057+0000 7fc085fb5640 1 -- 172.21.15.27:0/2274302035 shutdown_connections 2024-03-06T07:36:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.057+0000 7fc085fb5640 1 -- 172.21.15.27:0/2274302035 wait complete. 2024-03-06T07:36:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.058+0000 7fc085fb5640 1 Processor -- start 2024-03-06T07:36:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.058+0000 7fc085fb5640 1 -- start start 2024-03-06T07:36:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc085fb5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc08006e940 0x7fc08015a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc085fb5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0800ce900 0x7fc0801e6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc085fb5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc080144640 0x7fc0801e7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc085fb5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc080070c20 con 0x7fc08006e940 2024-03-06T07:36:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc085fb5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc080070aa0 con 0x7fc0800ce900 2024-03-06T07:36:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc085fb5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08015b840 con 0x7fc080144640 2024-03-06T07:36:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc08006e940 0x7fc08015a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc08006e940 0x7fc08015a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38634/0 (socket says 172.21.15.27:38634) 2024-03-06T07:36:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07f7fe640 1 -- 172.21.15.27:0/2939779525 learned_addr learned my addr 172.21.15.27:0/2939779525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07effd640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0800ce900 0x7fc0801e6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07ffff640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc080144640 0x7fc0801e7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07f7fe640 1 -- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc080144640 msgr2=0x7fc0801e7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07f7fe640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc080144640 0x7fc0801e7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07f7fe640 1 -- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0800ce900 msgr2=0x7fc0801e6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07f7fe640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0800ce900 0x7fc0801e6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07f7fe640 1 -- 172.21.15.27:0/2939779525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08013d800 con 0x7fc08006e940 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07effd640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0800ce900 0x7fc0801e6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc07f7fe640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc08006e940 0x7fc08015a440 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fc070034040 tx=0x7fc0700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.059+0000 7fc067fff640 1 -- 172.21.15.27:0/2939779525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070040070 con 0x7fc08006e940 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.060+0000 7fc067fff640 1 -- 172.21.15.27:0/2939779525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0700399a0 con 0x7fc08006e940 2024-03-06T07:36:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.060+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08013dae0 con 0x7fc08006e940 2024-03-06T07:36:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.060+0000 7fc067fff640 1 -- 172.21.15.27:0/2939779525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070035040 con 0x7fc08006e940 2024-03-06T07:36:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.060+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08013e0a0 con 0x7fc08006e940 2024-03-06T07:36:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.060+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0800733c0 con 0x7fc08006e940 2024-03-06T07:36:07.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.061+0000 7fc067fff640 1 -- 172.21.15.27:0/2939779525 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc070038020 con 0x7fc08006e940 2024-03-06T07:36:07.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.061+0000 7fc067fff640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc06c06b4d0 0x7fc06c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.061+0000 7fc067fff640 1 -- 172.21.15.27:0/2939779525 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc0700ab8d0 con 0x7fc08006e940 2024-03-06T07:36:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.062+0000 7fc067fff640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fc06c071430 0x7fc06c073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.062+0000 7fc067fff640 1 -- 172.21.15.27:0/2939779525 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc070033e90 con 0x7fc06c071430 2024-03-06T07:36:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.062+0000 7fc067fff640 1 -- 172.21.15.27:0/2939779525 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0700b02c0 con 0x7fc08006e940 2024-03-06T07:36:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.062+0000 7fc07effd640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc06c06b4d0 0x7fc06c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.062+0000 7fc07ffff640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fc06c071430 0x7fc06c073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.062+0000 7fc07effd640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc06c06b4d0 0x7fc06c06d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc08013b560 tx=0x7fc074009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.062+0000 7fc07ffff640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fc06c071430 0x7fc06c073850 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.063+0000 7fc067fff640 1 -- 172.21.15.27:0/2939779525 <== osd.5 v2:172.21.15.69:6819/715204514 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc070033e90 con 0x7fc06c071430 2024-03-06T07:36:07.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.076+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 --> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc080141880 con 0x7fc06c071430 2024-03-06T07:36:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc067fff640 1 -- 172.21.15.27:0/2939779525 <== osd.5 v2:172.21.15.69:6819/715204514 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc080141880 con 0x7fc06c071430 2024-03-06T07:36:07.079 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:36:07.079 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:36:07.079 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:36:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fc06c071430 msgr2=0x7fc06c073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6819/715204514,v1:172.21.15.69:6821/715204514] conn(0x7fc06c071430 0x7fc06c073850 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc06c06b4d0 msgr2=0x7fc06c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fc06c06b4d0 0x7fc06c06d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc08013b560 tx=0x7fc074009040 comp rx=0 tx=0).stop 2024-03-06T07:36:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc08006e940 msgr2=0x7fc08015a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 --2- 172.21.15.27:0/2939779525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc08006e940 0x7fc08015a440 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fc070034040 tx=0x7fc0700046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc07ffff640 1 -- 172.21.15.27:0/2939779525 reap_dead start 2024-03-06T07:36:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 shutdown_connections 2024-03-06T07:36:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 >> 172.21.15.27:0/2939779525 conn(0x7fc080066d00 msgr2=0x7fc0800670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 shutdown_connections 2024-03-06T07:36:07.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.077+0000 7fc085fb5640 1 -- 172.21.15.27:0/2939779525 wait complete. 2024-03-06T07:36:07.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-06T07:36:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.195+0000 7fbbe058a640 1 Processor -- start 2024-03-06T07:36:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.195+0000 7fbbe058a640 1 -- start start 2024-03-06T07:36:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.196+0000 7fbbe058a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd8157f60 0x7fbbd815b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.196+0000 7fbbe058a640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd806c7b0 con 0x7fbbd8157b80 2024-03-06T07:36:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.196+0000 7fbbe058a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd806ca90 con 0x7fbbd80cbab0 2024-03-06T07:36:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.196+0000 7fbbe058a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd806cd70 con 0x7fbbd8157f60 2024-03-06T07:36:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.196+0000 7fbbde326640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbbd8157b80 0x7fbbd815b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49546/0 (socket says 172.21.15.27:49546) 2024-03-06T07:36:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.196+0000 7fbbde326640 1 -- 172.21.15.27:0/3516342388 learned_addr learned my addr 172.21.15.27:0/3516342388 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbddb25640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd8157f60 0x7fbbd815b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039948527 0 0) 0x7fbbd806c7b0 con 0x7fbbd8157b80 2024-03-06T07:36:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbc4003440 con 0x7fbbd8157b80 2024-03-06T07:36:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 255974236 0 0) 0x7fbbc4003440 con 0x7fbbd8157b80 2024-03-06T07:36:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd8157f60 msgr2=0x7fbbd815b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd8157f60 0x7fbbd815b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 >> v1:172.21.15.69:6789/0 conn(0x7fbbd80cbab0 legacy=0x7fbbd8070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd806d050 con 0x7fbbd8157b80 2024-03-06T07:36:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fbbd4001670 con 0x7fbbd8157b80 2024-03-06T07:36:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 >> v1:172.21.15.27:6789/0 conn(0x7fbbd8157b80 legacy=0x7fbbd815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd8157f60 0x7fbbc4003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc4004010 0x7fbbc400a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbc400af80 0x7fbbc400d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd806cd70 con 0x7fbbc4004010 2024-03-06T07:36:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc400dc30 con 0x7fbbc400af80 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc400de30 con 0x7fbbd8157f60 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbddb25640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc4004010 0x7fbbc400a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbde326640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd8157f60 0x7fbbc4003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdeb27640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbc400af80 0x7fbbc400d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbddb25640 1 -- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd8157f60 msgr2=0x7fbbc4003ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbddb25640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd8157f60 0x7fbbc4003ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbddb25640 1 -- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbc400af80 msgr2=0x7fbbc400d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbddb25640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbc400af80 0x7fbbc400d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbddb25640 1 -- 172.21.15.27:0/3516342388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc4003440 con 0x7fbbc4004010 2024-03-06T07:36:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.197+0000 7fbbdeb27640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbc400af80 0x7fbbc400d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.198+0000 7fbbddb25640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc4004010 0x7fbbc400a860 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc009ac0 tx=0x7fbbc4003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ce1435abecf6d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.198+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc039070 con 0x7fbbc4004010 2024-03-06T07:36:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.198+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbcc02c6b0 con 0x7fbbc4004010 2024-03-06T07:36:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.198+0000 7fbbdd324640 1 -- 172.21.15.27:0/3516342388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc009690 con 0x7fbbc4004010 2024-03-06T07:36:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.199+0000 7fbbe058a640 1 -- 172.21.15.27:0/3516342388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc4004010 msgr2=0x7fbbc400a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.199+0000 7fbbe058a640 1 --2- 172.21.15.27:0/3516342388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc4004010 0x7fbbc400a860 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc009ac0 tx=0x7fbbc4003730 comp rx=0 tx=0).stop 2024-03-06T07:36:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.199+0000 7fbbdeb27640 1 -- 172.21.15.27:0/3516342388 reap_dead start 2024-03-06T07:36:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.199+0000 7fbbe058a640 1 -- 172.21.15.27:0/3516342388 shutdown_connections 2024-03-06T07:36:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.199+0000 7fbbe058a640 1 -- 172.21.15.27:0/3516342388 >> 172.21.15.27:0/3516342388 conn(0x7fbbd8066d00 msgr2=0x7fbbd80670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.199+0000 7fbbe058a640 1 -- 172.21.15.27:0/3516342388 shutdown_connections 2024-03-06T07:36:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.199+0000 7fbbe058a640 1 -- 172.21.15.27:0/3516342388 wait complete. 2024-03-06T07:36:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.200+0000 7fbbe058a640 1 Processor -- start 2024-03-06T07:36:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.200+0000 7fbbe058a640 1 -- start start 2024-03-06T07:36:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbe058a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbc4004010 0x7fbbd81551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbe058a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8155b10 0x7fbbd814dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbe058a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80cf890 0x7fbbd80cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbe058a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd8071250 con 0x7fbbd8155b10 2024-03-06T07:36:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbe058a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd80cc260 con 0x7fbbc4004010 2024-03-06T07:36:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbe058a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd80713d0 con 0x7fbbd80cf890 2024-03-06T07:36:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbddb25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8155b10 0x7fbbd814dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbddb25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8155b10 0x7fbbd814dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38652/0 (socket says 172.21.15.27:38652) 2024-03-06T07:36:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbddb25640 1 -- 172.21.15.27:0/3792405819 learned_addr learned my addr 172.21.15.27:0/3792405819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbddb25640 1 -- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80cf890 msgr2=0x7fbbd80cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbddb25640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80cf890 0x7fbbd80cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbddb25640 1 -- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbc4004010 msgr2=0x7fbbd81551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbddb25640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbc4004010 0x7fbbd81551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbddb25640 1 -- 172.21.15.27:0/3792405819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd8156360 con 0x7fbbd8155b10 2024-03-06T07:36:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbddb25640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8155b10 0x7fbbd814dc00 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc009ac0 tx=0x7fbbcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbcaffd640 1 -- 172.21.15.27:0/3792405819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc039050 con 0x7fbbd8155b10 2024-03-06T07:36:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbcaffd640 1 -- 172.21.15.27:0/3792405819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbcc045070 con 0x7fbbd8155b10 2024-03-06T07:36:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd8156080 con 0x7fbbd8155b10 2024-03-06T07:36:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.201+0000 7fbbcaffd640 1 -- 172.21.15.27:0/3792405819 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc0044e0 con 0x7fbbd8155b10 2024-03-06T07:36:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.202+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd814ff20 con 0x7fbbd8155b10 2024-03-06T07:36:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.202+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd814fa70 con 0x7fbbd8155b10 2024-03-06T07:36:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.203+0000 7fbbcaffd640 1 -- 172.21.15.27:0/3792405819 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbcc003850 con 0x7fbbd8155b10 2024-03-06T07:36:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.203+0000 7fbbcaffd640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.203+0000 7fbbcaffd640 1 -- 172.21.15.27:0/3792405819 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fbbcc0aba20 con 0x7fbbd8155b10 2024-03-06T07:36:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.204+0000 7fbbde326640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.204+0000 7fbbde326640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4001050 tx=0x7fbbd4001af0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.208+0000 7fbbcaffd640 1 -- 172.21.15.27:0/3792405819 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbcc04e020 con 0x7fbbd8155b10 2024-03-06T07:36:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.389+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbd8067ff0 con 0x7fbbbc06b4d0 2024-03-06T07:36:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.389+0000 7fbbcaffd640 1 -- 172.21.15.27:0/3792405819 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbbd8067ff0 con 0x7fbbbc06b4d0 2024-03-06T07:36:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 5 to deep-scrub 2024-03-06T07:36:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 msgr2=0x7fbbbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4001050 tx=0x7fbbd4001af0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8155b10 msgr2=0x7fbbd814dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8155b10 0x7fbbd814dc00 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc009ac0 tx=0x7fbbcc004730 comp rx=0 tx=0).stop 2024-03-06T07:36:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 shutdown_connections 2024-03-06T07:36:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fbbbc06b4d0 0x7fbbbc06d990 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fbbd80cf890 0x7fbbd80cfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbd8155b10 0x7fbbd814dc00 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 --2- 172.21.15.27:0/3792405819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbc4004010 0x7fbbd81551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 >> 172.21.15.27:0/3792405819 conn(0x7fbbd8066d00 msgr2=0x7fbbd81589e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 shutdown_connections 2024-03-06T07:36:07.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.391+0000 7fbbe058a640 1 -- 172.21.15.27:0/3792405819 wait complete. 2024-03-06T07:36:07.404 INFO:tasks.ceph:Scrubbing osd.6 2024-03-06T07:36:07.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:36:07.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.514+0000 7f94eb1ac640 1 Processor -- start 2024-03-06T07:36:07.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.514+0000 7f94eb1ac640 1 -- start start 2024-03-06T07:36:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.514+0000 7f94eb1ac640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94e4071540 0x7f94e4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.514+0000 7f94eb1ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e40cee80 0x7f94e4168900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.514+0000 7f94eb1ac640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e4071e60 con 0x7f94e40b2d30 2024-03-06T07:36:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.514+0000 7f94eb1ac640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e40d3230 con 0x7f94e4071540 2024-03-06T07:36:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.514+0000 7f94eb1ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e40d4960 con 0x7f94e40cee80 2024-03-06T07:36:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.515+0000 7f94e8f48640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94e4071540 0x7f94e4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.515+0000 7f94e8f48640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94e4071540 0x7f94e4071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38654/0 (socket says 172.21.15.27:38654) 2024-03-06T07:36:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.515+0000 7f94e8f48640 1 -- 172.21.15.27:0/1165316779 learned_addr learned my addr 172.21.15.27:0/1165316779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.515+0000 7f94e9749640 1 --2- 172.21.15.27:0/1165316779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e40cee80 0x7f94e4168900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.515+0000 7f94e8f48640 1 -- 172.21.15.27:0/1165316779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e40cee80 msgr2=0x7f94e4168900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.515+0000 7f94e8f48640 1 --2- 172.21.15.27:0/1165316779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e40cee80 0x7f94e4168900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.515+0000 7f94e8f48640 1 -- 172.21.15.27:0/1165316779 >> v1:172.21.15.69:6789/0 conn(0x7f94e40b2d30 legacy=0x7f94e40ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.515+0000 7f94e8f48640 1 -- 172.21.15.27:0/1165316779 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e40d36c0 con 0x7f94e4071540 2024-03-06T07:36:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94e8f48640 1 --2- 172.21.15.27:0/1165316779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e4071540 0x7f94e4071920 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009e50 tx=0x7f94dc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc0a123e2256625 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94e37fe640 1 -- 172.21.15.27:0/1165316779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc02cb00 con 0x7f94e4071540 2024-03-06T07:36:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94e37fe640 1 -- 172.21.15.27:0/1165316779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc02cca0 con 0x7f94e4071540 2024-03-06T07:36:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94e37fe640 1 -- 172.21.15.27:0/1165316779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc034610 con 0x7f94e4071540 2024-03-06T07:36:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94eb1ac640 1 -- 172.21.15.27:0/1165316779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e4071540 msgr2=0x7f94e4071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94eb1ac640 1 --2- 172.21.15.27:0/1165316779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e4071540 0x7f94e4071920 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009e50 tx=0x7f94dc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:36:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94eb1ac640 1 -- 172.21.15.27:0/1165316779 shutdown_connections 2024-03-06T07:36:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94eb1ac640 1 --2- 172.21.15.27:0/1165316779 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e40cee80 0x7f94e4168900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94eb1ac640 1 --2- 172.21.15.27:0/1165316779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e4071540 0x7f94e4071920 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94eb1ac640 1 -- 172.21.15.27:0/1165316779 >> 172.21.15.27:0/1165316779 conn(0x7f94e40d3d60 msgr2=0x7f94e40d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94eb1ac640 1 -- 172.21.15.27:0/1165316779 shutdown_connections 2024-03-06T07:36:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.516+0000 7f94eb1ac640 1 -- 172.21.15.27:0/1165316779 wait complete. 2024-03-06T07:36:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.518+0000 7f94eb1ac640 1 Processor -- start 2024-03-06T07:36:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.518+0000 7f94eb1ac640 1 -- start start 2024-03-06T07:36:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.518+0000 7f94eb1ac640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e4071540 0x7f94e41f7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.518+0000 7f94eb1ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e40b2d30 0x7f94e40c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.518+0000 7f94eb1ac640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e40cee80 0x7f94e40c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.518+0000 7f94eb1ac640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e40d2be0 con 0x7f94e40cee80 2024-03-06T07:36:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.518+0000 7f94eb1ac640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e40d2900 con 0x7f94e40b2d30 2024-03-06T07:36:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.518+0000 7f94eb1ac640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e40d2ec0 con 0x7f94e4071540 2024-03-06T07:36:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e9749640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e40cee80 0x7f94e40c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e9749640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e40cee80 0x7f94e40c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38664/0 (socket says 172.21.15.27:38664) 2024-03-06T07:36:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e9749640 1 -- 172.21.15.27:0/3649691656 learned_addr learned my addr 172.21.15.27:0/3649691656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e8f48640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e4071540 0x7f94e41f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e3fff640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e40b2d30 0x7f94e40c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e9749640 1 -- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e4071540 msgr2=0x7f94e41f7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e9749640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94e4071540 0x7f94e41f7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e9749640 1 -- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e40b2d30 msgr2=0x7f94e40c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e9749640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e40b2d30 0x7f94e40c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e9749640 1 -- 172.21.15.27:0/3649691656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e40cce70 con 0x7f94e40cee80 2024-03-06T07:36:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e9749640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e40cee80 0x7f94e40c5ba0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00d830 tx=0x7f94cc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e17fa640 1 -- 172.21.15.27:0/3649691656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc00b660 con 0x7f94e40cee80 2024-03-06T07:36:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e17fa640 1 -- 172.21.15.27:0/3649691656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94cc019070 con 0x7f94e40cee80 2024-03-06T07:36:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e40cd150 con 0x7f94e40cee80 2024-03-06T07:36:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94e17fa640 1 -- 172.21.15.27:0/3649691656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc00ba60 con 0x7f94e40cee80 2024-03-06T07:36:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.519+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e40cd710 con 0x7f94e40cee80 2024-03-06T07:36:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.520+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94e40c6a20 con 0x7f94e40cee80 2024-03-06T07:36:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.521+0000 7f94e17fa640 1 -- 172.21.15.27:0/3649691656 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94cc01e460 con 0x7f94e40cee80 2024-03-06T07:36:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.521+0000 7f94e17fa640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94c406b4d0 0x7f94c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.521+0000 7f94e17fa640 1 -- 172.21.15.27:0/3649691656 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f94cc08ae70 con 0x7f94e40cee80 2024-03-06T07:36:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.521+0000 7f94e17fa640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f94c4071430 0x7f94c4073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.521+0000 7f94e17fa640 1 -- 172.21.15.27:0/3649691656 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94c4073f00 con 0x7f94c4071430 2024-03-06T07:36:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.521+0000 7f94e17fa640 1 -- 172.21.15.27:0/3649691656 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94cc08b300 con 0x7f94e40cee80 2024-03-06T07:36:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.522+0000 7f94e8f48640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94c406b4d0 0x7f94c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.522+0000 7f94e3fff640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f94c4071430 0x7f94c4073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.522+0000 7f94e8f48640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94c406b4d0 0x7f94c406d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f94dc035000 tx=0x7f94dc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.522+0000 7f94e3fff640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f94c4071430 0x7f94c4073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.523+0000 7f94e17fa640 1 -- 172.21.15.27:0/3649691656 <== osd.6 v2:172.21.15.69:6800/288062332 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f94c4073f00 con 0x7f94c4071430 2024-03-06T07:36:07.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.536+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 --> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f94e4072070 con 0x7f94c4071430 2024-03-06T07:36:07.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94e17fa640 1 -- 172.21.15.27:0/3649691656 <== osd.6 v2:172.21.15.69:6800/288062332 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f94e4072070 con 0x7f94c4071430 2024-03-06T07:36:07.539 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:36:07.539 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:36:07.539 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:36:07.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f94c4071430 msgr2=0x7f94c4073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6800/288062332,v1:172.21.15.69:6801/288062332] conn(0x7f94c4071430 0x7f94c4073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94c406b4d0 msgr2=0x7f94c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f94c406b4d0 0x7f94c406d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f94dc035000 tx=0x7f94dc030000 comp rx=0 tx=0).stop 2024-03-06T07:36:07.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e40cee80 msgr2=0x7f94e40c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 --2- 172.21.15.27:0/3649691656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94e40cee80 0x7f94e40c5ba0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00d830 tx=0x7f94cc00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94e9749640 1 -- 172.21.15.27:0/3649691656 reap_dead start 2024-03-06T07:36:07.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 shutdown_connections 2024-03-06T07:36:07.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 >> 172.21.15.27:0/3649691656 conn(0x7f94e40d3d60 msgr2=0x7f94e415bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 shutdown_connections 2024-03-06T07:36:07.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.537+0000 7f94eb1ac640 1 -- 172.21.15.27:0/3649691656 wait complete. 2024-03-06T07:36:07.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-06T07:36:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.654+0000 7f2094edf640 1 Processor -- start 2024-03-06T07:36:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.654+0000 7f2094edf640 1 -- start start 2024-03-06T07:36:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.655+0000 7f2094edf640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f20900b95a0 0x7f20900b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.655+0000 7f2094edf640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20900cea30 con 0x7f20900b9a10 2024-03-06T07:36:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.655+0000 7f2094edf640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20900cf3c0 con 0x7f209015df00 2024-03-06T07:36:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.655+0000 7f2094edf640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20900cf0e0 con 0x7f20900b95a0 2024-03-06T07:36:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.655+0000 7f208dd9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f20900b95a0 0x7f20900b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.655+0000 7f208ed9d640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f20900b9a10 0x7f20900d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49552/0 (socket says 172.21.15.27:49552) 2024-03-06T07:36:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.655+0000 7f208dd9b640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f20900b95a0 0x7f20900b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38672/0 (socket says 172.21.15.27:38672) 2024-03-06T07:36:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.655+0000 7f208ed9d640 1 -- 172.21.15.27:0/1679227309 learned_addr learned my addr 172.21.15.27:0/1679227309 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.656+0000 7f208dd9b640 1 -- 172.21.15.27:0/1679227309 >> v1:172.21.15.69:6789/0 conn(0x7f209015df00 legacy=0x7f20900d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.656+0000 7f208dd9b640 1 -- 172.21.15.27:0/1679227309 >> v1:172.21.15.27:6789/0 conn(0x7f20900b9a10 legacy=0x7f20900d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.656+0000 7f208d59a640 1 -- 172.21.15.27:0/1679227309 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566812620 0 0) 0x7f20900cea30 con 0x7f20900b9a10 2024-03-06T07:36:07.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.656+0000 7f208dd9b640 1 -- 172.21.15.27:0/1679227309 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20900ce470 con 0x7f20900b95a0 2024-03-06T07:36:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.656+0000 7f208dd9b640 1 --2- 172.21.15.27:0/1679227309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20900b95a0 0x7f20900b5700 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f20900cf590 tx=0x7f207802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d06f0045101a34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f208d59a640 1 -- 172.21.15.27:0/1679227309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2078035020 con 0x7f20900b95a0 2024-03-06T07:36:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f208d59a640 1 -- 172.21.15.27:0/1679227309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2078031830 con 0x7f20900b95a0 2024-03-06T07:36:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f208d59a640 1 -- 172.21.15.27:0/1679227309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2078031c00 con 0x7f20900b95a0 2024-03-06T07:36:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f2094edf640 1 -- 172.21.15.27:0/1679227309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20900b95a0 msgr2=0x7f20900b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f2094edf640 1 --2- 172.21.15.27:0/1679227309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20900b95a0 0x7f20900b5700 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f20900cf590 tx=0x7f207802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f2094edf640 1 -- 172.21.15.27:0/1679227309 shutdown_connections 2024-03-06T07:36:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f2094edf640 1 --2- 172.21.15.27:0/1679227309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20900b95a0 0x7f20900b5700 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f2094edf640 1 -- 172.21.15.27:0/1679227309 >> 172.21.15.27:0/1679227309 conn(0x7f2090071960 msgr2=0x7f2090071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f2094edf640 1 -- 172.21.15.27:0/1679227309 shutdown_connections 2024-03-06T07:36:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.657+0000 7f2094edf640 1 -- 172.21.15.27:0/1679227309 wait complete. 2024-03-06T07:36:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f2094edf640 1 Processor -- start 2024-03-06T07:36:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f2094edf640 1 -- start start 2024-03-06T07:36:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f2094edf640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900b95a0 0x7f20900cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f2094edf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900b9a10 0x7f20900c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f2094edf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f209015df00 0x7f20900d2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f2094edf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20900cedf0 con 0x7f209015df00 2024-03-06T07:36:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f2094edf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20900cec70 con 0x7f20900b9a10 2024-03-06T07:36:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f2094edf640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20900b5c40 con 0x7f20900b95a0 2024-03-06T07:36:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f208ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f209015df00 0x7f20900d2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f208ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f209015df00 0x7f20900d2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38688/0 (socket says 172.21.15.27:38688) 2024-03-06T07:36:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.659+0000 7f208ed9d640 1 -- 172.21.15.27:0/2790849129 learned_addr learned my addr 172.21.15.27:0/2790849129 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208dd9b640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900b9a10 0x7f20900c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208ed9d640 1 -- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900b95a0 msgr2=0x7f20900cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208e59c640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900b95a0 0x7f20900cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208ed9d640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900b95a0 0x7f20900cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208ed9d640 1 -- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900b9a10 msgr2=0x7f20900c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208ed9d640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900b9a10 0x7f20900c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208ed9d640 1 -- 172.21.15.27:0/2790849129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20900c7d00 con 0x7f209015df00 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208dd9b640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900b9a10 0x7f20900c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208e59c640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900b95a0 0x7f20900cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f208ed9d640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f209015df00 0x7f20900d2d60 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f208000c7a0 tx=0x7f208000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f2086ffd640 1 -- 172.21.15.27:0/2790849129 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080018070 con 0x7f209015df00 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f2086ffd640 1 -- 172.21.15.27:0/2790849129 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208000a640 con 0x7f209015df00 2024-03-06T07:36:07.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20900c7fe0 con 0x7f209015df00 2024-03-06T07:36:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f2086ffd640 1 -- 172.21.15.27:0/2790849129 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208000aab0 con 0x7f209015df00 2024-03-06T07:36:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.660+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20900c85a0 con 0x7f209015df00 2024-03-06T07:36:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.661+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2090067ff0 con 0x7f209015df00 2024-03-06T07:36:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.662+0000 7f2086ffd640 1 -- 172.21.15.27:0/2790849129 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f208001b070 con 0x7f209015df00 2024-03-06T07:36:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.662+0000 7f2086ffd640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207006b4d0 0x7f207006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.662+0000 7f2086ffd640 1 -- 172.21.15.27:0/2790849129 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2080089590 con 0x7f209015df00 2024-03-06T07:36:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.662+0000 7f208e59c640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207006b4d0 0x7f207006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.663+0000 7f208e59c640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207006b4d0 0x7f207006d990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f208800a830 tx=0x7f2088008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.666+0000 7f2086ffd640 1 -- 172.21.15.27:0/2790849129 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20800586b0 con 0x7f209015df00 2024-03-06T07:36:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.852+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f20900d4110 con 0x7f207006b4d0 2024-03-06T07:36:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.852+0000 7f2086ffd640 1 -- 172.21.15.27:0/2790849129 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f20900d4110 con 0x7f207006b4d0 2024-03-06T07:36:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 6 to deep-scrub 2024-03-06T07:36:07.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.854+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207006b4d0 msgr2=0x7f207006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.854+0000 7f2094edf640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207006b4d0 0x7f207006d990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f208800a830 tx=0x7f2088008040 comp rx=0 tx=0).stop 2024-03-06T07:36:07.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.854+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f209015df00 msgr2=0x7f20900d2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.854+0000 7f2094edf640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f209015df00 0x7f20900d2d60 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f208000c7a0 tx=0x7f208000cc60 comp rx=0 tx=0).stop 2024-03-06T07:36:07.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.855+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 shutdown_connections 2024-03-06T07:36:07.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.855+0000 7f2094edf640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f207006b4d0 0x7f207006d990 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.855+0000 7f2094edf640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f209015df00 0x7f20900d2d60 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.855+0000 7f2094edf640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20900b9a10 0x7f20900c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.855+0000 7f2094edf640 1 --2- 172.21.15.27:0/2790849129 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f20900b95a0 0x7f20900cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.855+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 >> 172.21.15.27:0/2790849129 conn(0x7f2090071960 msgr2=0x7f209015ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.855+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 shutdown_connections 2024-03-06T07:36:07.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.855+0000 7f2094edf640 1 -- 172.21.15.27:0/2790849129 wait complete. 2024-03-06T07:36:07.868 INFO:tasks.ceph:Scrubbing osd.7 2024-03-06T07:36:07.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:36:07.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.976+0000 7f10fd4df640 1 Processor -- start 2024-03-06T07:36:07.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.976+0000 7f10fd4df640 1 -- start start 2024-03-06T07:36:07.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.977+0000 7f10fd4df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f8071540 0x7f10f8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.977+0000 7f10fd4df640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10f80b2d30 0x7f10f80ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.977+0000 7f10fd4df640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f8071e60 con 0x7f10f80cecb0 2024-03-06T07:36:07.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.977+0000 7f10fd4df640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f80d3230 con 0x7f10f80b2d30 2024-03-06T07:36:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.977+0000 7f10fd4df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80d4960 con 0x7f10f8071540 2024-03-06T07:36:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.977+0000 7f10f67fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10f80b2d30 0x7f10f80ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.978+0000 7f10f6ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f8071540 0x7f10f8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.978+0000 7f10f67fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10f80b2d30 0x7f10f80ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.27:39356/0 (socket says 172.21.15.27:39356) 2024-03-06T07:36:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.978+0000 7f10f77fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f10f80cecb0 0x7f10f80b72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.27:48554/0 (socket says 172.21.15.27:48554) 2024-03-06T07:36:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.978+0000 7f10f67fc640 1 -- 172.21.15.27:0/348231523 learned_addr learned my addr 172.21.15.27:0/348231523 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.978+0000 7f10f67fc640 1 -- 172.21.15.27:0/348231523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f8071540 msgr2=0x7f10f8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.978+0000 7f10f67fc640 1 --2- 172.21.15.27:0/348231523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f8071540 0x7f10f8071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.978+0000 7f10f67fc640 1 -- 172.21.15.27:0/348231523 >> v1:172.21.15.69:6789/0 conn(0x7f10f80cecb0 legacy=0x7f10f80b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.978+0000 7f10f67fc640 1 -- 172.21.15.27:0/348231523 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80d36c0 con 0x7f10f80b2d30 2024-03-06T07:36:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.978+0000 7f10f6ffd640 1 --2- 172.21.15.27:0/348231523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f8071540 0x7f10f8071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10f67fc640 1 --2- 172.21.15.27:0/348231523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f80b2d30 0x7f10f80ce770 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009870 tx=0x7f10e002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e2c93efa8980afc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10f5ffb640 1 -- 172.21.15.27:0/348231523 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0034070 con 0x7f10f80b2d30 2024-03-06T07:36:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10f5ffb640 1 -- 172.21.15.27:0/348231523 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e0030840 con 0x7f10f80b2d30 2024-03-06T07:36:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10f5ffb640 1 -- 172.21.15.27:0/348231523 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0030c10 con 0x7f10f80b2d30 2024-03-06T07:36:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10fd4df640 1 -- 172.21.15.27:0/348231523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f80b2d30 msgr2=0x7f10f80ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10fd4df640 1 --2- 172.21.15.27:0/348231523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f80b2d30 0x7f10f80ce770 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009870 tx=0x7f10e002b980 comp rx=0 tx=0).stop 2024-03-06T07:36:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10fd4df640 1 -- 172.21.15.27:0/348231523 shutdown_connections 2024-03-06T07:36:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10fd4df640 1 --2- 172.21.15.27:0/348231523 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f80b2d30 0x7f10f80ce770 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10fd4df640 1 --2- 172.21.15.27:0/348231523 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f8071540 0x7f10f8071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10fd4df640 1 -- 172.21.15.27:0/348231523 >> 172.21.15.27:0/348231523 conn(0x7f10f80d3d60 msgr2=0x7f10f80d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10fd4df640 1 -- 172.21.15.27:0/348231523 shutdown_connections 2024-03-06T07:36:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.979+0000 7f10fd4df640 1 -- 172.21.15.27:0/348231523 wait complete. 2024-03-06T07:36:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.981+0000 7f10fd4df640 1 Processor -- start 2024-03-06T07:36:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.981+0000 7f10fd4df640 1 -- start start 2024-03-06T07:36:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.981+0000 7f10fd4df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f8071540 0x7f10f81f5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.981+0000 7f10fd4df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f80b2d30 0x7f10f80ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.981+0000 7f10fd4df640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80cecb0 0x7f10f80acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.981+0000 7f10fd4df640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80d2be0 con 0x7f10f80b2d30 2024-03-06T07:36:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.981+0000 7f10fd4df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80d2900 con 0x7f10f8071540 2024-03-06T07:36:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.981+0000 7f10fd4df640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80d2ec0 con 0x7f10f80cecb0 2024-03-06T07:36:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f80b2d30 0x7f10f80ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f80b2d30 0x7f10f80ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38704/0 (socket says 172.21.15.27:38704) 2024-03-06T07:36:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f67fc640 1 -- 172.21.15.27:0/4011641535 learned_addr learned my addr 172.21.15.27:0/4011641535 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f77fe640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80cecb0 0x7f10f80acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f6ffd640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f8071540 0x7f10f81f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f67fc640 1 -- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80cecb0 msgr2=0x7f10f80acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f67fc640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80cecb0 0x7f10f80acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f67fc640 1 -- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f8071540 msgr2=0x7f10f81f5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f67fc640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f8071540 0x7f10f81f5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f67fc640 1 -- 172.21.15.27:0/4011641535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80cbf20 con 0x7f10f80b2d30 2024-03-06T07:36:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f77fe640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f10f80cecb0 0x7f10f80acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f6ffd640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10f8071540 0x7f10f81f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10f67fc640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f80b2d30 0x7f10f80ac710 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f10e0000c00 tx=0x7f10e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10ef7fe640 1 -- 172.21.15.27:0/4011641535 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0048070 con 0x7f10f80b2d30 2024-03-06T07:36:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.982+0000 7f10ef7fe640 1 -- 172.21.15.27:0/4011641535 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e0030640 con 0x7f10f80b2d30 2024-03-06T07:36:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.983+0000 7f10fd4df640 1 -- 172.21.15.27:0/4011641535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f80cc200 con 0x7f10f80b2d30 2024-03-06T07:36:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.983+0000 7f10ef7fe640 1 -- 172.21.15.27:0/4011641535 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0034040 con 0x7f10f80b2d30 2024-03-06T07:36:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.983+0000 7f10fd4df640 1 -- 172.21.15.27:0/4011641535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f80cc7c0 con 0x7f10f80b2d30 2024-03-06T07:36:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.983+0000 7f10ed7fa640 1 -- 172.21.15.27:0/4011641535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10d0001030 con 0x7f10f80b2d30 2024-03-06T07:36:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.984+0000 7f10ef7fe640 1 -- 172.21.15.27:0/4011641535 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10e004b070 con 0x7f10f80b2d30 2024-03-06T07:36:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.984+0000 7f10ef7fe640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 0x7f10d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.984+0000 7f10ef7fe640 1 -- 172.21.15.27:0/4011641535 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f10e00abdd0 con 0x7f10f80b2d30 2024-03-06T07:36:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.985+0000 7f10ef7fe640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7f10d8071430 0x7f10d8073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.985+0000 7f10ef7fe640 1 -- 172.21.15.27:0/4011641535 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10d8073f00 con 0x7f10d8071430 2024-03-06T07:36:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.985+0000 7f10ef7fe640 1 -- 172.21.15.27:0/4011641535 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10e00ae4d0 con 0x7f10f80b2d30 2024-03-06T07:36:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.985+0000 7f10f6ffd640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 0x7f10d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.985+0000 7f10f77fe640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7f10d8071430 0x7f10d8073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:07.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.985+0000 7f10f77fe640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7f10d8071430 0x7f10d8073850 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.985+0000 7f10f6ffd640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 0x7f10d806d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009af0 tx=0x7f10f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:07.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.986+0000 7f10ef7fe640 1 -- 172.21.15.27:0/4011641535 <== osd.7 v2:172.21.15.69:6816/1392567864 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f10d8073f00 con 0x7f10d8071430 2024-03-06T07:36:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:07.999+0000 7f10ed7fa640 1 -- 172.21.15.27:0/4011641535 --> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f10d0002e30 con 0x7f10d8071430 2024-03-06T07:36:08.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10ef7fe640 1 -- 172.21.15.27:0/4011641535 <== osd.7 v2:172.21.15.69:6816/1392567864 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f10d0002e30 con 0x7f10d8071430 2024-03-06T07:36:08.002 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-06T07:36:08.003 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:36:08.003 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-06T07:36:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10ed7fa640 1 -- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7f10d8071430 msgr2=0x7f10d8073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10ed7fa640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6816/1392567864,v1:172.21.15.69:6817/1392567864] conn(0x7f10d8071430 0x7f10d8073850 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10ed7fa640 1 -- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 msgr2=0x7f10d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10ed7fa640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f10d806b4d0 0x7f10d806d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009af0 tx=0x7f10f0008040 comp rx=0 tx=0).stop 2024-03-06T07:36:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10ed7fa640 1 -- 172.21.15.27:0/4011641535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f80b2d30 msgr2=0x7f10f80ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10ed7fa640 1 --2- 172.21.15.27:0/4011641535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10f80b2d30 0x7f10f80ac710 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f10e0000c00 tx=0x7f10e0004730 comp rx=0 tx=0).stop 2024-03-06T07:36:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10f77fe640 1 -- 172.21.15.27:0/4011641535 reap_dead start 2024-03-06T07:36:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10ed7fa640 1 -- 172.21.15.27:0/4011641535 shutdown_connections 2024-03-06T07:36:08.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.000+0000 7f10ed7fa640 1 -- 172.21.15.27:0/4011641535 >> 172.21.15.27:0/4011641535 conn(0x7f10f80d3d60 msgr2=0x7f10f815b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.001+0000 7f10ed7fa640 1 -- 172.21.15.27:0/4011641535 shutdown_connections 2024-03-06T07:36:08.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.001+0000 7f10ed7fa640 1 -- 172.21.15.27:0/4011641535 wait complete. 2024-03-06T07:36:08.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-06T07:36:08.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.118+0000 7f1f09372640 1 Processor -- start 2024-03-06T07:36:08.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.118+0000 7f1f09372640 1 -- start start 2024-03-06T07:36:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f09372640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040b1020 0x7f1f0406af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f09372640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f040cd890 0x7f1f0415d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f09372640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f040cc0e0 con 0x7f1f0406ed10 2024-03-06T07:36:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f09372640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f040d28b0 con 0x7f1f040cd890 2024-03-06T07:36:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f09372640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040cf320 con 0x7f1f040b1020 2024-03-06T07:36:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f040cd890 0x7f1f0415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f037fe640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f040cd890 0x7f1f0415d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38716/0 (socket says 172.21.15.27:38716) 2024-03-06T07:36:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f037fe640 1 -- 172.21.15.27:0/1255438344 learned_addr learned my addr 172.21.15.27:0/1255438344 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f02ffd640 1 --2- 172.21.15.27:0/1255438344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040b1020 0x7f1f0406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f037fe640 1 -- 172.21.15.27:0/1255438344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040b1020 msgr2=0x7f1f0406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f037fe640 1 --2- 172.21.15.27:0/1255438344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040b1020 0x7f1f0406af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f037fe640 1 -- 172.21.15.27:0/1255438344 >> v1:172.21.15.69:6789/0 conn(0x7f1f0406ed10 legacy=0x7f1f040cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.119+0000 7f1f037fe640 1 -- 172.21.15.27:0/1255438344 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f040cc3c0 con 0x7f1f040cd890 2024-03-06T07:36:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.120+0000 7f1f037fe640 1 --2- 172.21.15.27:0/1255438344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f040cd890 0x7f1f0415d700 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009e50 tx=0x7f1efc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2579a28416104b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.120+0000 7f1f01ffb640 1 -- 172.21.15.27:0/1255438344 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc02ca70 con 0x7f1f040cd890 2024-03-06T07:36:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.120+0000 7f1f01ffb640 1 -- 172.21.15.27:0/1255438344 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc02cc10 con 0x7f1f040cd890 2024-03-06T07:36:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.120+0000 7f1f01ffb640 1 -- 172.21.15.27:0/1255438344 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc0343f0 con 0x7f1f040cd890 2024-03-06T07:36:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.121+0000 7f1f09372640 1 -- 172.21.15.27:0/1255438344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f040cd890 msgr2=0x7f1f0415d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.121+0000 7f1f09372640 1 --2- 172.21.15.27:0/1255438344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f040cd890 0x7f1f0415d700 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009e50 tx=0x7f1efc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.121+0000 7f1f09372640 1 -- 172.21.15.27:0/1255438344 shutdown_connections 2024-03-06T07:36:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.121+0000 7f1f09372640 1 --2- 172.21.15.27:0/1255438344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f040cd890 0x7f1f0415d700 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.121+0000 7f1f09372640 1 --2- 172.21.15.27:0/1255438344 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040b1020 0x7f1f0406af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.121+0000 7f1f09372640 1 -- 172.21.15.27:0/1255438344 >> 172.21.15.27:0/1255438344 conn(0x7f1f040d1a90 msgr2=0x7f1f040d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.121+0000 7f1f09372640 1 -- 172.21.15.27:0/1255438344 shutdown_connections 2024-03-06T07:36:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.121+0000 7f1f09372640 1 -- 172.21.15.27:0/1255438344 wait complete. 2024-03-06T07:36:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.122+0000 7f1f09372640 1 Processor -- start 2024-03-06T07:36:08.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.122+0000 7f1f09372640 1 -- start start 2024-03-06T07:36:08.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f09372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0406ed10 0x7f1f04155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f09372640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f040b1020 0x7f1f040b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f09372640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040cd890 0x7f1f040cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f09372640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040ccc60 con 0x7f1f0406ed10 2024-03-06T07:36:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f09372640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040cc980 con 0x7f1f040b1020 2024-03-06T07:36:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f09372640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040cdcc0 con 0x7f1f040cd890 2024-03-06T07:36:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f02ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0406ed10 0x7f1f04155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f02ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0406ed10 0x7f1f04155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38720/0 (socket says 172.21.15.27:38720) 2024-03-06T07:36:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f02ffd640 1 -- 172.21.15.27:0/1414147274 learned_addr learned my addr 172.21.15.27:0/1414147274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:08.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f037fe640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040cd890 0x7f1f040cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f027fc640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f040b1020 0x7f1f040b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f02ffd640 1 -- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040cd890 msgr2=0x7f1f040cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f02ffd640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040cd890 0x7f1f040cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f02ffd640 1 -- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f040b1020 msgr2=0x7f1f040b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f02ffd640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f040b1020 0x7f1f040b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.123+0000 7f1f02ffd640 1 -- 172.21.15.27:0/1414147274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f040b5030 con 0x7f1f0406ed10 2024-03-06T07:36:08.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.124+0000 7f1f027fc640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f040b1020 0x7f1f040b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:08.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.124+0000 7f1f02ffd640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0406ed10 0x7f1f04155210 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400d800 tx=0x7f1ef400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.124+0000 7f1eeb7fe640 1 -- 172.21.15.27:0/1414147274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef400b6c0 con 0x7f1f0406ed10 2024-03-06T07:36:08.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.124+0000 7f1eeb7fe640 1 -- 172.21.15.27:0/1414147274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef4019070 con 0x7f1f0406ed10 2024-03-06T07:36:08.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.124+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f040b5310 con 0x7f1f0406ed10 2024-03-06T07:36:08.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.124+0000 7f1eeb7fe640 1 -- 172.21.15.27:0/1414147274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef400ba80 con 0x7f1f0406ed10 2024-03-06T07:36:08.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.124+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f040b58d0 con 0x7f1f0406ed10 2024-03-06T07:36:08.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.125+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f04067ff0 con 0x7f1f0406ed10 2024-03-06T07:36:08.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.125+0000 7f1eeb7fe640 1 -- 172.21.15.27:0/1414147274 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ef400bc20 con 0x7f1f0406ed10 2024-03-06T07:36:08.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.126+0000 7f1eeb7fe640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ef006b4d0 0x7f1ef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.126+0000 7f1eeb7fe640 1 -- 172.21.15.27:0/1414147274 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1ef408a410 con 0x7f1f0406ed10 2024-03-06T07:36:08.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.126+0000 7f1f027fc640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ef006b4d0 0x7f1ef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.126+0000 7f1f027fc640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ef006b4d0 0x7f1ef006d990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1f0415e7e0 tx=0x7f1ef8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.130+0000 7f1eeb7fe640 1 -- 172.21.15.27:0/1414147274 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ef4059530 con 0x7f1f0406ed10 2024-03-06T07:36:08.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.311+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f040d1410 con 0x7f1ef006b4d0 2024-03-06T07:36:08.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.311+0000 7f1eeb7fe640 1 -- 172.21.15.27:0/1414147274 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1f040d1410 con 0x7f1ef006b4d0 2024-03-06T07:36:08.314 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 7 to deep-scrub 2024-03-06T07:36:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ef006b4d0 msgr2=0x7f1ef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ef006b4d0 0x7f1ef006d990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1f0415e7e0 tx=0x7f1ef8009040 comp rx=0 tx=0).stop 2024-03-06T07:36:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0406ed10 msgr2=0x7f1f04155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0406ed10 0x7f1f04155210 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400d800 tx=0x7f1ef400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 shutdown_connections 2024-03-06T07:36:08.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f1ef006b4d0 0x7f1ef006d990 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f1f040cd890 0x7f1f040cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f040b1020 0x7f1f040b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 --2- 172.21.15.27:0/1414147274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f0406ed10 0x7f1f04155210 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 >> 172.21.15.27:0/1414147274 conn(0x7f1f040d1a90 msgr2=0x7f1f040b3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 shutdown_connections 2024-03-06T07:36:08.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.314+0000 7f1f09372640 1 -- 172.21.15.27:0/1414147274 wait complete. 2024-03-06T07:36:08.327 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:36:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.431+0000 7fdf33356640 1 Processor -- start 2024-03-06T07:36:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.431+0000 7fdf33356640 1 -- start start 2024-03-06T07:36:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.432+0000 7fdf33356640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c0d0880 0x7fdf2c168c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.432+0000 7fdf33356640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c0cf130 con 0x7fdf2c06fe20 2024-03-06T07:36:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.432+0000 7fdf33356640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c0d0dd0 con 0x7fdf2c15e060 2024-03-06T07:36:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.432+0000 7fdf33356640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0d10b0 con 0x7fdf2c0d0880 2024-03-06T07:36:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.432+0000 7fdf308f1640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdf2c06fe20 0x7fdf2c0cea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49566/0 (socket says 172.21.15.27:49566) 2024-03-06T07:36:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.432+0000 7fdf308f1640 1 -- 172.21.15.27:0/70594538 learned_addr learned my addr 172.21.15.27:0/70594538 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-06T07:36:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf318f3640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c0d0880 0x7fdf2c168c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389074498 0 0) 0x7fdf2c0cf130 con 0x7fdf2c06fe20 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf14003520 con 0x7fdf2c06fe20 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 <== mon.1 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126870263 0 0) 0x7fdf2c0d0dd0 con 0x7fdf2c15e060 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf2c0cf130 con 0x7fdf2c15e060 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1610210879 0 0) 0x7fdf14003520 con 0x7fdf2c06fe20 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c0d0880 msgr2=0x7fdf2c168c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c0d0880 0x7fdf2c168c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 >> v1:172.21.15.69:6789/0 conn(0x7fdf2c15e060 legacy=0x7fdf2c0ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c0cc0e0 con 0x7fdf2c06fe20 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3666889202 0 0) 0x7fdf200015f0 con 0x7fdf2c06fe20 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 >> v1:172.21.15.27:6789/0 conn(0x7fdf2c06fe20 legacy=0x7fdf2c0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf14003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf14004800 0x7fdf14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c0d0880 0x7fdf140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0d10b0 con 0x7fdf14003b70 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1400ba70 con 0x7fdf14004800 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1400bc70 con 0x7fdf2c0d0880 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf310f2640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf14003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf318f3640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c0d0880 0x7fdf140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.433+0000 7fdf308f1640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf14004800 0x7fdf14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.434+0000 7fdf310f2640 1 -- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c0d0880 msgr2=0x7fdf140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.434+0000 7fdf310f2640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c0d0880 0x7fdf140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.434+0000 7fdf310f2640 1 -- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf14004800 msgr2=0x7fdf14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.434+0000 7fdf310f2640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf14004800 0x7fdf14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.434+0000 7fdf310f2640 1 -- 172.21.15.27:0/70594538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf14003520 con 0x7fdf14003b70 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.434+0000 7fdf310f2640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf14003f70 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fdf240018d0 tx=0x7fdf14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c91953c819b585 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.434+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24037070 con 0x7fdf14003b70 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.434+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf240052d0 con 0x7fdf14003b70 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.434+0000 7fdf2bfff640 1 -- 172.21.15.27:0/70594538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf240014f0 con 0x7fdf14003b70 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.435+0000 7fdf33356640 1 -- 172.21.15.27:0/70594538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 msgr2=0x7fdf14003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.435+0000 7fdf33356640 1 --2- 172.21.15.27:0/70594538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf14003f70 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fdf240018d0 tx=0x7fdf14005830 comp rx=0 tx=0).stop 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.435+0000 7fdf318f3640 1 -- 172.21.15.27:0/70594538 reap_dead start 2024-03-06T07:36:08.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.435+0000 7fdf33356640 1 -- 172.21.15.27:0/70594538 shutdown_connections 2024-03-06T07:36:08.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.435+0000 7fdf33356640 1 -- 172.21.15.27:0/70594538 >> 172.21.15.27:0/70594538 conn(0x7fdf2c0d1a90 msgr2=0x7fdf2c0d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.435+0000 7fdf33356640 1 -- 172.21.15.27:0/70594538 shutdown_connections 2024-03-06T07:36:08.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.435+0000 7fdf33356640 1 -- 172.21.15.27:0/70594538 wait complete. 2024-03-06T07:36:08.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.436+0000 7fdf33356640 1 Processor -- start 2024-03-06T07:36:08.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.436+0000 7fdf33356640 1 -- start start 2024-03-06T07:36:08.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf33356640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf2c153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf33356640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2c153390 0x7fdf2c0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf33356640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c155130 0x7fdf2c1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf33356640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c15e440 con 0x7fdf14003b70 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf33356640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0cc6a0 con 0x7fdf2c153390 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf33356640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0cdbd0 con 0x7fdf2c155130 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf310f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf2c153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf310f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf2c153a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38738/0 (socket says 172.21.15.27:38738) 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf310f2640 1 -- 172.21.15.27:0/1734802433 learned_addr learned my addr 172.21.15.27:0/1734802433 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf318f3640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c155130 0x7fdf2c1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf308f1640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2c153390 0x7fdf2c0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf310f2640 1 -- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c155130 msgr2=0x7fdf2c1555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf310f2640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c155130 0x7fdf2c1555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf310f2640 1 -- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2c153390 msgr2=0x7fdf2c0c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf310f2640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2c153390 0x7fdf2c0c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.437+0000 7fdf310f2640 1 -- 172.21.15.27:0/1734802433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c153810 con 0x7fdf14003b70 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.438+0000 7fdf310f2640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf2c153a40 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fdf240019f0 tx=0x7fdf240012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.438+0000 7fdf29ffb640 1 -- 172.21.15.27:0/1734802433 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24037070 con 0x7fdf14003b70 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.438+0000 7fdf29ffb640 1 -- 172.21.15.27:0/1734802433 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf24038040 con 0x7fdf14003b70 2024-03-06T07:36:08.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.438+0000 7fdf29ffb640 1 -- 172.21.15.27:0/1734802433 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf240014f0 con 0x7fdf14003b70 2024-03-06T07:36:08.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.438+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2c154090 con 0x7fdf14003b70 2024-03-06T07:36:08.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.438+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2c154650 con 0x7fdf14003b70 2024-03-06T07:36:08.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.439+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c067ff0 con 0x7fdf14003b70 2024-03-06T07:36:08.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.439+0000 7fdf29ffb640 1 -- 172.21.15.27:0/1734802433 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf240052d0 con 0x7fdf14003b70 2024-03-06T07:36:08.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.440+0000 7fdf29ffb640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf1806b500 0x7fdf1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.440+0000 7fdf29ffb640 1 -- 172.21.15.27:0/1734802433 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdf240ab560 con 0x7fdf14003b70 2024-03-06T07:36:08.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.440+0000 7fdf308f1640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf1806b500 0x7fdf1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.441+0000 7fdf308f1640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf1806b500 0x7fdf1806d9c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fdf20001b20 tx=0x7fdf200019f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.444+0000 7fdf29ffb640 1 -- 172.21.15.27:0/1734802433 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf2407a1e0 con 0x7fdf14003b70 2024-03-06T07:36:08.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.622+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf2c0b0d70 con 0x7fdf1806b500 2024-03-06T07:36:08.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.623+0000 7fdf29ffb640 1 -- 172.21.15.27:0/1734802433 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13464 (secure 0 0 0) 0x7fdf2c0b0d70 con 0x7fdf1806b500 2024-03-06T07:36:08.626 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:36:08.626 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:36:08.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf1806b500 msgr2=0x7fdf1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf1806b500 0x7fdf1806d9c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fdf20001b20 tx=0x7fdf200019f0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 msgr2=0x7fdf2c153a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf2c153a40 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fdf240019f0 tx=0x7fdf240012d0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 shutdown_connections 2024-03-06T07:36:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fdf1806b500 0x7fdf1806d9c0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdf2c155130 0x7fdf2c1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf2c153390 0x7fdf2c0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 --2- 172.21.15.27:0/1734802433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf14003b70 0x7fdf2c153a40 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 >> 172.21.15.27:0/1734802433 conn(0x7fdf2c0d1a90 msgr2=0x7fdf2c15d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 shutdown_connections 2024-03-06T07:36:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:08.626+0000 7fdf33356640 1 -- 172.21.15.27:0/1734802433 wait complete. 2024-03-06T07:36:08.639 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2024-03-06T07:36:08.182034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":332096,"kb_used_data":16072,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754642624,"statfs":{"total":773094113280,"available":772754046976,"internally_reserved":0,"allocated":16457728,"data_stored":7872552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.849658"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":425,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:36:01.128915+0000","last_change":"2024-03-06T07:22:38.064821+0000","last_active":"2024-03-06T07:36:01.128915+0000","last_peered":"2024-03-06T07:36:01.128915+0000","last_clean":"2024-03-06T07:36:01.128915+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:36:01.128915+0000","last_undegraded":"2024-03-06T07:36:01.128915+0000","last_fullsized":"2024-03-06T07:36:01.128915+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:37.007126+0000","last_clean_scrub_stamp":"2024-03-06T07:22:37.007126+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:19.168175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575013,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41944,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329896,"statfs":{"total":96636764160,"available":96593813504,"internally_reserved":0,"allocated":2502656,"data_stored":1426845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575014,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575013,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41364,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330476,"statfs":{"total":96636764160,"available":96594407424,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41948,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329892,"statfs":{"total":96636764160,"available":96593809408,"internally_reserved":0,"allocated":2502656,"data_stored":1426845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:36:08.642 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-06T07:22:37.007126+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=22, tm_sec=37, tm_wday=2, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=36, tm_sec=3, tm_wday=2, tm_yday=66, tm_isdst=0) 2024-03-06T07:36:08.642 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-06T07:36:20.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:20.654 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:28.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:36:28.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.760+0000 7fe770765640 1 Processor -- start 2024-03-06T07:36:28.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.760+0000 7fe770765640 1 -- start start 2024-03-06T07:36:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.761+0000 7fe770765640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7680b95a0 0x7fe7680b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.761+0000 7fe770765640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 0x7fe7680b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.761+0000 7fe770765640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680cea30 con 0x7fe76815df00 2024-03-06T07:36:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.761+0000 7fe770765640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7680cf3c0 con 0x7fe7680b95a0 2024-03-06T07:36:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.761+0000 7fe770765640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680cf0e0 con 0x7fe7680b9a10 2024-03-06T07:36:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.762+0000 7fe76dd00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7680b95a0 0x7fe7680b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.762+0000 7fe76dd00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7680b95a0 0x7fe7680b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37800/0 (socket says 172.21.15.27:37800) 2024-03-06T07:36:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.762+0000 7fe76dd00640 1 -- 172.21.15.27:0/3643111397 learned_addr learned my addr 172.21.15.27:0/3643111397 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.762+0000 7fe76ed02640 1 --2- 172.21.15.27:0/3643111397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 0x7fe7680b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.762+0000 7fe76dd00640 1 -- 172.21.15.27:0/3643111397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 msgr2=0x7fe7680b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.762+0000 7fe76dd00640 1 --2- 172.21.15.27:0/3643111397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 0x7fe7680b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.762+0000 7fe76dd00640 1 -- 172.21.15.27:0/3643111397 >> v1:172.21.15.69:6789/0 conn(0x7fe76815df00 legacy=0x7fe7680d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.762+0000 7fe76dd00640 1 -- 172.21.15.27:0/3643111397 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680ce470 con 0x7fe7680b95a0 2024-03-06T07:36:28.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.762+0000 7fe76ed02640 1 --2- 172.21.15.27:0/3643111397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 0x7fe7680b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.763+0000 7fe76dd00640 1 --2- 172.21.15.27:0/3643111397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 0x7fe7680b5700 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fe758009870 tx=0x7fe75802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a4a6f8510b291f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.763+0000 7fe76d4ff640 1 -- 172.21.15.27:0/3643111397 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758034070 con 0x7fe7680b95a0 2024-03-06T07:36:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.763+0000 7fe76d4ff640 1 -- 172.21.15.27:0/3643111397 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758030840 con 0x7fe7680b95a0 2024-03-06T07:36:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.763+0000 7fe76d4ff640 1 -- 172.21.15.27:0/3643111397 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758030c10 con 0x7fe7680b95a0 2024-03-06T07:36:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.763+0000 7fe770765640 1 -- 172.21.15.27:0/3643111397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 msgr2=0x7fe7680b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.763+0000 7fe770765640 1 --2- 172.21.15.27:0/3643111397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 0x7fe7680b5700 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fe758009870 tx=0x7fe75802b980 comp rx=0 tx=0).stop 2024-03-06T07:36:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.764+0000 7fe770765640 1 -- 172.21.15.27:0/3643111397 shutdown_connections 2024-03-06T07:36:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.764+0000 7fe770765640 1 --2- 172.21.15.27:0/3643111397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 0x7fe7680b5700 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.764+0000 7fe770765640 1 --2- 172.21.15.27:0/3643111397 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 0x7fe7680b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.764+0000 7fe770765640 1 -- 172.21.15.27:0/3643111397 >> 172.21.15.27:0/3643111397 conn(0x7fe768071960 msgr2=0x7fe768071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.764+0000 7fe770765640 1 -- 172.21.15.27:0/3643111397 shutdown_connections 2024-03-06T07:36:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.764+0000 7fe770765640 1 -- 172.21.15.27:0/3643111397 wait complete. 2024-03-06T07:36:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.765+0000 7fe770765640 1 Processor -- start 2024-03-06T07:36:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.765+0000 7fe770765640 1 -- start start 2024-03-06T07:36:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe770765640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 0x7fe7680cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe770765640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 0x7fe768151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe770765640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe76815df00 0x7fe768160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe770765640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680cedf0 con 0x7fe7680b95a0 2024-03-06T07:36:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe770765640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680cec70 con 0x7fe76815df00 2024-03-06T07:36:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe770765640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680d3220 con 0x7fe7680b9a10 2024-03-06T07:36:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76e501640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 0x7fe7680cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76e501640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 0x7fe7680cc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37814/0 (socket says 172.21.15.27:37814) 2024-03-06T07:36:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76e501640 1 -- 172.21.15.27:0/2982748874 learned_addr learned my addr 172.21.15.27:0/2982748874 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76ed02640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe76815df00 0x7fe768160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76dd00640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 0x7fe768151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76e501640 1 -- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 msgr2=0x7fe768151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76e501640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 0x7fe768151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76e501640 1 -- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe76815df00 msgr2=0x7fe768160bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76e501640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe76815df00 0x7fe768160bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76e501640 1 -- 172.21.15.27:0/2982748874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680c77b0 con 0x7fe7680b95a0 2024-03-06T07:36:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe76e501640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 0x7fe7680cc8c0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fe76400c3c0 tx=0x7fe76400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe75effd640 1 -- 172.21.15.27:0/2982748874 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7640052d0 con 0x7fe7680b95a0 2024-03-06T07:36:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe75effd640 1 -- 172.21.15.27:0/2982748874 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe764005490 con 0x7fe7680b95a0 2024-03-06T07:36:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe768152690 con 0x7fe7680b95a0 2024-03-06T07:36:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe75effd640 1 -- 172.21.15.27:0/2982748874 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764007260 con 0x7fe7680b95a0 2024-03-06T07:36:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.766+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7680c7d30 con 0x7fe7680b95a0 2024-03-06T07:36:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.767+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe768067ff0 con 0x7fe7680b95a0 2024-03-06T07:36:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.768+0000 7fe75effd640 1 -- 172.21.15.27:0/2982748874 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe764007400 con 0x7fe7680b95a0 2024-03-06T07:36:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.768+0000 7fe75effd640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74c06b4d0 0x7fe74c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.768+0000 7fe75effd640 1 -- 172.21.15.27:0/2982748874 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe764089300 con 0x7fe7680b95a0 2024-03-06T07:36:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.769+0000 7fe76dd00640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74c06b4d0 0x7fe74c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.769+0000 7fe76dd00640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74c06b4d0 0x7fe74c06d990 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe758036000 tx=0x7fe758008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.773+0000 7fe75effd640 1 -- 172.21.15.27:0/2982748874 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe764057e60 con 0x7fe7680b95a0 2024-03-06T07:36:28.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.951+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 --> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7680689a0 con 0x7fe74c06b4d0 2024-03-06T07:36:28.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.952+0000 7fe75effd640 1 -- 172.21.15.27:0/2982748874 <== mgr.4106 v2:172.21.15.69:6832/2286301988 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13379 (secure 0 0 0) 0x7fe7680689a0 con 0x7fe74c06b4d0 2024-03-06T07:36:28.954 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:36:28.954 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-06T07:36:28.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74c06b4d0 msgr2=0x7fe74c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74c06b4d0 0x7fe74c06d990 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe758036000 tx=0x7fe758008030 comp rx=0 tx=0).stop 2024-03-06T07:36:28.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 msgr2=0x7fe7680cc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 0x7fe7680cc8c0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fe76400c3c0 tx=0x7fe76400c880 comp rx=0 tx=0).stop 2024-03-06T07:36:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 shutdown_connections 2024-03-06T07:36:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7fe74c06b4d0 0x7fe74c06d990 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe76815df00 0x7fe768160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe7680b9a10 0x7fe768151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 --2- 172.21.15.27:0/2982748874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7680b95a0 0x7fe7680cc8c0 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 >> 172.21.15.27:0/2982748874 conn(0x7fe768071960 msgr2=0x7fe768067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:28.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 shutdown_connections 2024-03-06T07:36:28.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:28.955+0000 7fe770765640 1 -- 172.21.15.27:0/2982748874 wait complete. 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":562,"stamp":"2024-03-06T07:36:28.186065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":332084,"kb_used_data":16072,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754642636,"statfs":{"total":773094113280,"available":772754059264,"internally_reserved":0,"allocated":16457728,"data_stored":7872552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323472946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002482"},"pg_stats":[{"pgid":"1.0","version":"152'119","reported_seq":436,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:36:09.226454+0000","last_change":"2024-03-06T07:36:09.226511+0000","last_active":"2024-03-06T07:36:09.226454+0000","last_peered":"2024-03-06T07:36:09.226454+0000","last_clean":"2024-03-06T07:36:09.226454+0000","last_became_active":"2024-03-06T07:22:38.063714+0000","last_became_peered":"2024-03-06T07:22:38.063714+0000","last_unstale":"2024-03-06T07:36:09.226454+0000","last_undegraded":"2024-03-06T07:36:09.226454+0000","last_fullsized":"2024-03-06T07:36:09.226454+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'119","last_scrub_stamp":"2024-03-06T07:36:09.226336+0000","last_deep_scrub":"152'119","last_deep_scrub_stamp":"2024-03-06T07:36:09.226336+0000","last_clean_scrub_stamp":"2024-03-06T07:36:09.226336+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:55.950671+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41940,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329900,"statfs":{"total":96636764160,"available":96593817600,"internally_reserved":0,"allocated":2502656,"data_stored":1426845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41364,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330476,"statfs":{"total":96636764160,"available":96594407424,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41368,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1908736,"data_stored":836477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575018,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41940,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329900,"statfs":{"total":96636764160,"available":96593817600,"internally_reserved":0,"allocated":2502656,"data_stored":1426845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:36:28.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-06T07:36:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.074+0000 7f6f550fb640 1 Processor -- start 2024-03-06T07:36:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.074+0000 7f6f550fb640 1 -- start start 2024-03-06T07:36:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.075+0000 7f6f550fb640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6f5015dd90 0x7f6f500ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.075+0000 7f6f550fb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d29d0 0x7f6f500d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.075+0000 7f6f550fb640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f500d3740 con 0x7f6f500d4720 2024-03-06T07:36:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.075+0000 7f6f550fb640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f500d3400 con 0x7f6f5015dd90 2024-03-06T07:36:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.075+0000 7f6f550fb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500d3120 con 0x7f6f500d29d0 2024-03-06T07:36:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.075+0000 7f6f4ed9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6f5015dd90 0x7f6f500ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.075+0000 7f6f4ed9d640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6f5015dd90 0x7f6f500ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37818/0 (socket says 172.21.15.27:37818) 2024-03-06T07:36:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.075+0000 7f6f4ed9d640 1 -- 172.21.15.27:0/4209176216 learned_addr learned my addr 172.21.15.27:0/4209176216 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.075+0000 7f6f4e59c640 1 --2- 172.21.15.27:0/4209176216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d29d0 0x7f6f500d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.076+0000 7f6f4ed9d640 1 -- 172.21.15.27:0/4209176216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d29d0 msgr2=0x7f6f500d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.076+0000 7f6f4ed9d640 1 --2- 172.21.15.27:0/4209176216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d29d0 0x7f6f500d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.076+0000 7f6f4ed9d640 1 -- 172.21.15.27:0/4209176216 >> v1:172.21.15.69:6789/0 conn(0x7f6f500d4720 legacy=0x7f6f500b6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.076+0000 7f6f4ed9d640 1 -- 172.21.15.27:0/4209176216 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f500d39e0 con 0x7f6f5015dd90 2024-03-06T07:36:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.076+0000 7f6f4ed9d640 1 --2- 172.21.15.27:0/4209176216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f5015dd90 0x7f6f500ceea0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009870 tx=0x7f6f4802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a52675086b1d47c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f4dd9b640 1 -- 172.21.15.27:0/4209176216 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4802f840 con 0x7f6f5015dd90 2024-03-06T07:36:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f4dd9b640 1 -- 172.21.15.27:0/4209176216 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4802f9e0 con 0x7f6f5015dd90 2024-03-06T07:36:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f4dd9b640 1 -- 172.21.15.27:0/4209176216 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4802fdb0 con 0x7f6f5015dd90 2024-03-06T07:36:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f550fb640 1 -- 172.21.15.27:0/4209176216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f5015dd90 msgr2=0x7f6f500ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f550fb640 1 --2- 172.21.15.27:0/4209176216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f5015dd90 0x7f6f500ceea0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009870 tx=0x7f6f4802ba20 comp rx=0 tx=0).stop 2024-03-06T07:36:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f550fb640 1 -- 172.21.15.27:0/4209176216 shutdown_connections 2024-03-06T07:36:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f550fb640 1 --2- 172.21.15.27:0/4209176216 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d29d0 0x7f6f500d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f550fb640 1 --2- 172.21.15.27:0/4209176216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f5015dd90 0x7f6f500ceea0 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f550fb640 1 -- 172.21.15.27:0/4209176216 >> 172.21.15.27:0/4209176216 conn(0x7f6f50071960 msgr2=0x7f6f50071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f550fb640 1 -- 172.21.15.27:0/4209176216 shutdown_connections 2024-03-06T07:36:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.077+0000 7f6f550fb640 1 -- 172.21.15.27:0/4209176216 wait complete. 2024-03-06T07:36:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.079+0000 7f6f550fb640 1 Processor -- start 2024-03-06T07:36:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.079+0000 7f6f550fb640 1 -- start start 2024-03-06T07:36:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.079+0000 7f6f550fb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f500d29d0 0x7f6f50160bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.079+0000 7f6f550fb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d4720 0x7f6f5015ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.079+0000 7f6f550fb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f5015dd90 0x7f6f5015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.079+0000 7f6f550fb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500d2df0 con 0x7f6f500d29d0 2024-03-06T07:36:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.079+0000 7f6f550fb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5015e170 con 0x7f6f5015dd90 2024-03-06T07:36:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.079+0000 7f6f550fb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500b7420 con 0x7f6f500d4720 2024-03-06T07:36:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.079+0000 7f6f4ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f500d29d0 0x7f6f50160bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f500d29d0 0x7f6f50160bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37830/0 (socket says 172.21.15.27:37830) 2024-03-06T07:36:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4ed9d640 1 -- 172.21.15.27:0/1143043142 learned_addr learned my addr 172.21.15.27:0/1143043142 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4e59c640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d4720 0x7f6f5015ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4f59e640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f5015dd90 0x7f6f5015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4ed9d640 1 -- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d4720 msgr2=0x7f6f5015ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4ed9d640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d4720 0x7f6f5015ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4ed9d640 1 -- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f5015dd90 msgr2=0x7f6f5015e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4ed9d640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f5015dd90 0x7f6f5015e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4ed9d640 1 -- 172.21.15.27:0/1143043142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5015eed0 con 0x7f6f500d29d0 2024-03-06T07:36:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4e59c640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d4720 0x7f6f5015ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4f59e640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f5015dd90 0x7f6f5015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f4ed9d640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f500d29d0 0x7f6f50160bb0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f6f48000c00 tx=0x7f6f48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f477fe640 1 -- 172.21.15.27:0/1143043142 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48037bc0 con 0x7f6f500d29d0 2024-03-06T07:36:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f477fe640 1 -- 172.21.15.27:0/1143043142 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f48046070 con 0x7f6f500d29d0 2024-03-06T07:36:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5015f1b0 con 0x7f6f500d29d0 2024-03-06T07:36:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f477fe640 1 -- 172.21.15.27:0/1143043142 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48041400 con 0x7f6f500d29d0 2024-03-06T07:36:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.080+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5015f770 con 0x7f6f500d29d0 2024-03-06T07:36:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.081+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f50072a30 con 0x7f6f500d29d0 2024-03-06T07:36:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.082+0000 7f6f477fe640 1 -- 172.21.15.27:0/1143043142 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f48040070 con 0x7f6f500d29d0 2024-03-06T07:36:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.082+0000 7f6f477fe640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f3006b4d0 0x7f6f3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.082+0000 7f6f477fe640 1 -- 172.21.15.27:0/1143043142 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6f480ac550 con 0x7f6f500d29d0 2024-03-06T07:36:29.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.082+0000 7f6f4e59c640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f3006b4d0 0x7f6f3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.083+0000 7f6f4e59c640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f3006b4d0 0x7f6f3006d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f6f500ca640 tx=0x7f6f38008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.087+0000 7f6f477fe640 1 -- 172.21.15.27:0/1143043142 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f48033030 con 0x7f6f500d29d0 2024-03-06T07:36:29.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.269+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f6f50159890 con 0x7f6f500d29d0 2024-03-06T07:36:29.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.272+0000 7f6f477fe640 1 -- 172.21.15.27:0/1143043142 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v36)=0 v36) v1 ==== 123+0+0 (secure 0 0 0) 0x7f6f4807b680 con 0x7f6f500d29d0 2024-03-06T07:36:29.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.273+0000 7f6f477fe640 1 -- 172.21.15.27:0/1143043142 <== mon.0 v2:172.21.15.27:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f6f4807af80 con 0x7f6f500d29d0 2024-03-06T07:36:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f3006b4d0 msgr2=0x7f6f3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f3006b4d0 0x7f6f3006d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f6f500ca640 tx=0x7f6f38008040 comp rx=0 tx=0).stop 2024-03-06T07:36:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f500d29d0 msgr2=0x7f6f50160bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f500d29d0 0x7f6f50160bb0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f6f48000c00 tx=0x7f6f48004730 comp rx=0 tx=0).stop 2024-03-06T07:36:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 shutdown_connections 2024-03-06T07:36:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f6f3006b4d0 0x7f6f3006d990 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f5015dd90 0x7f6f5015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f500d4720 0x7f6f5015ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 --2- 172.21.15.27:0/1143043142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f500d29d0 0x7f6f50160bb0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 >> 172.21.15.27:0/1143043142 conn(0x7f6f50071960 msgr2=0x7f6f500b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 shutdown_connections 2024-03-06T07:36:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.275+0000 7f6f550fb640 1 -- 172.21.15.27:0/1143043142 wait complete. 2024-03-06T07:36:29.287 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:36:29.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.392+0000 7f47c1325640 1 Processor -- start 2024-03-06T07:36:29.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.393+0000 7f47c1325640 1 -- start start 2024-03-06T07:36:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.393+0000 7f47c1325640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f47bc158520 0x7f47bc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.393+0000 7f47c1325640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47bc06dd60 0x7f47bc0ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.393+0000 7f47c1325640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47bc071240 con 0x7f47bc0aec40 2024-03-06T07:36:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.393+0000 7f47c1325640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47bc071b70 con 0x7f47bc06dd60 2024-03-06T07:36:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.393+0000 7f47c1325640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47bc0af0b0 con 0x7f47bc158520 2024-03-06T07:36:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.394+0000 7f47ba7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47bc06dd60 0x7f47bc0ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.394+0000 7f47ba7fc640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47bc06dd60 0x7f47bc0ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37832/0 (socket says 172.21.15.27:37832) 2024-03-06T07:36:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.394+0000 7f47ba7fc640 1 -- 172.21.15.27:0/4156103679 learned_addr learned my addr 172.21.15.27:0/4156103679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.394+0000 7f47baffd640 1 --2- 172.21.15.27:0/4156103679 >> v2:172.21.15.69:3300/0 conn(0x7f47bc158520 0x7f47bc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.394+0000 7f47ba7fc640 1 -- 172.21.15.27:0/4156103679 >> v2:172.21.15.69:3300/0 conn(0x7f47bc158520 msgr2=0x7f47bc06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:29.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.394+0000 7f47ba7fc640 1 --2- 172.21.15.27:0/4156103679 >> v2:172.21.15.69:3300/0 conn(0x7f47bc158520 0x7f47bc06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.394+0000 7f47ba7fc640 1 -- 172.21.15.27:0/4156103679 >> v1:172.21.15.69:6789/0 conn(0x7f47bc0aec40 legacy=0x7f47bc15e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.394+0000 7f47ba7fc640 1 -- 172.21.15.27:0/4156103679 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc071520 con 0x7f47bc06dd60 2024-03-06T07:36:29.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.395+0000 7f47ba7fc640 1 --2- 172.21.15.27:0/4156103679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc06dd60 0x7f47bc0ae700 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f47bc06e900 tx=0x7f47b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6de02ebb7e4e35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47b9ffb640 1 -- 172.21.15.27:0/4156103679 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0034070 con 0x7f47bc06dd60 2024-03-06T07:36:29.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47b9ffb640 1 -- 172.21.15.27:0/4156103679 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f47b0030860 con 0x7f47bc06dd60 2024-03-06T07:36:29.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47b9ffb640 1 -- 172.21.15.27:0/4156103679 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0030c50 con 0x7f47bc06dd60 2024-03-06T07:36:29.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47c1325640 1 -- 172.21.15.27:0/4156103679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc06dd60 msgr2=0x7f47bc0ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47c1325640 1 --2- 172.21.15.27:0/4156103679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc06dd60 0x7f47bc0ae700 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f47bc06e900 tx=0x7f47b002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47c1325640 1 -- 172.21.15.27:0/4156103679 shutdown_connections 2024-03-06T07:36:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47c1325640 1 --2- 172.21.15.27:0/4156103679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc06dd60 0x7f47bc0ae700 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47c1325640 1 --2- 172.21.15.27:0/4156103679 >> v2:172.21.15.69:3300/0 conn(0x7f47bc158520 0x7f47bc06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47c1325640 1 -- 172.21.15.27:0/4156103679 >> 172.21.15.27:0/4156103679 conn(0x7f47bc066d00 msgr2=0x7f47bc0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47c1325640 1 -- 172.21.15.27:0/4156103679 shutdown_connections 2024-03-06T07:36:29.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.396+0000 7f47c1325640 1 -- 172.21.15.27:0/4156103679 wait complete. 2024-03-06T07:36:29.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.398+0000 7f47c1325640 1 Processor -- start 2024-03-06T07:36:29.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.398+0000 7f47c1325640 1 -- start start 2024-03-06T07:36:29.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.398+0000 7f47c1325640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47bc06dd60 0x7f47bc14d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.398+0000 7f47c1325640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc0aec40 0x7f47bc0ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.398+0000 7f47c1325640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47bc158520 0x7f47bc145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.398+0000 7f47c1325640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc072000 con 0x7f47bc0aec40 2024-03-06T07:36:29.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.398+0000 7f47c1325640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc0b4650 con 0x7f47bc158520 2024-03-06T07:36:29.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.398+0000 7f47c1325640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc158900 con 0x7f47bc06dd60 2024-03-06T07:36:29.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47ba7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc0aec40 0x7f47bc0ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47ba7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc0aec40 0x7f47bc0ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37834/0 (socket says 172.21.15.27:37834) 2024-03-06T07:36:29.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47ba7fc640 1 -- 172.21.15.27:0/290123773 learned_addr learned my addr 172.21.15.27:0/290123773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-06T07:36:29.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47baffd640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47bc06dd60 0x7f47bc14d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47bb7fe640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47bc158520 0x7f47bc145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47ba7fc640 1 -- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47bc06dd60 msgr2=0x7f47bc14d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47ba7fc640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47bc06dd60 0x7f47bc14d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47ba7fc640 1 -- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47bc158520 msgr2=0x7f47bc145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47ba7fc640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47bc158520 0x7f47bc145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47ba7fc640 1 -- 172.21.15.27:0/290123773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc145f30 con 0x7f47bc0aec40 2024-03-06T07:36:29.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47bb7fe640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47bc158520 0x7f47bc145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:29.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47ba7fc640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc0aec40 0x7f47bc0ccc40 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f47b002b9e0 tx=0x7f47b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47a37fe640 1 -- 172.21.15.27:0/290123773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0034040 con 0x7f47bc0aec40 2024-03-06T07:36:29.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47a37fe640 1 -- 172.21.15.27:0/290123773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f47b0039d90 con 0x7f47bc0aec40 2024-03-06T07:36:29.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc1489c0 con 0x7f47bc0aec40 2024-03-06T07:36:29.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47a37fe640 1 -- 172.21.15.27:0/290123773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0043650 con 0x7f47bc0aec40 2024-03-06T07:36:29.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.399+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc149260 con 0x7f47bc0aec40 2024-03-06T07:36:29.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.400+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc067ff0 con 0x7f47bc0aec40 2024-03-06T07:36:29.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.401+0000 7f47a37fe640 1 -- 172.21.15.27:0/290123773 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47b004a060 con 0x7f47bc0aec40 2024-03-06T07:36:29.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.401+0000 7f47a37fe640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f47a806b4d0 0x7f47a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:29.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.401+0000 7f47a37fe640 1 -- 172.21.15.27:0/290123773 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f47b00ac050 con 0x7f47bc0aec40 2024-03-06T07:36:29.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.401+0000 7f47baffd640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f47a806b4d0 0x7f47a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:29.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.402+0000 7f47baffd640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f47a806b4d0 0x7f47a806d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009b40 tx=0x7f47ac009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:29.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.406+0000 7f47a37fe640 1 -- 172.21.15.27:0/290123773 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47b003e070 con 0x7f47bc0aec40 2024-03-06T07:36:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.624+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47bc0b31f0 con 0x7f47bc0aec40 2024-03-06T07:36:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.624+0000 7f47a37fe640 1 -- 172.21.15.27:0/290123773 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+1329 (secure 0 0 0) 0x7f47b007b0e0 con 0x7f47bc0aec40 2024-03-06T07:36:29.626 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-06T07:36:29.626 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":125,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/3844031010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3844031010},{"type":"v1","addr":"172.21.15.27:6833","nonce":3844031010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[]} 2024-03-06T07:36:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 125 2024-03-06T07:36:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f47a806b4d0 msgr2=0x7f47a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f47a806b4d0 0x7f47a806d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009b40 tx=0x7f47ac009190 comp rx=0 tx=0).stop 2024-03-06T07:36:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc0aec40 msgr2=0x7f47bc0ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc0aec40 0x7f47bc0ccc40 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f47b002b9e0 tx=0x7f47b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 shutdown_connections 2024-03-06T07:36:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:6832/2286301988,v1:172.21.15.69:6833/2286301988] conn(0x7f47a806b4d0 0x7f47a806d990 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47bc158520 0x7f47bc145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47bc0aec40 0x7f47bc0ccc40 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 --2- 172.21.15.27:0/290123773 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47bc06dd60 0x7f47bc14d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 >> 172.21.15.27:0/290123773 conn(0x7f47bc066d00 msgr2=0x7f47bc06c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 shutdown_connections 2024-03-06T07:36:29.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-06T07:36:29.626+0000 7f47c1325640 1 -- 172.21.15.27:0/290123773 wait complete. 2024-03-06T07:36:29.638 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-06T07:36:29.638 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-06T07:36:29.639 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:36:29.639 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:29.647 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:36:29.648 ERROR:tasks.ceph.mds.c:tried to stop a non-running daemon 2024-03-06T07:36:29.650 ERROR:tasks.ceph.mds.b:tried to stop a non-running daemon 2024-03-06T07:36:29.653 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:36:29.653 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:29.665 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:36:29.665 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-06T07:36:29.666 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-06T07:36:29.666 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:29.717 INFO:tasks.ceph.osd.0:Stopped 2024-03-06T07:36:29.717 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-06T07:36:29.717 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:29.790 INFO:tasks.ceph.osd.1:Stopped 2024-03-06T07:36:29.791 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-06T07:36:29.791 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:29.861 INFO:tasks.ceph.osd.2:Stopped 2024-03-06T07:36:29.861 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-06T07:36:29.862 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:29.931 INFO:tasks.ceph.osd.3:Stopped 2024-03-06T07:36:29.931 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-06T07:36:29.931 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:29.966 INFO:tasks.ceph.osd.4:Stopped 2024-03-06T07:36:29.967 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-06T07:36:29.967 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:30.037 INFO:tasks.ceph.osd.5:Stopped 2024-03-06T07:36:30.038 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-06T07:36:30.038 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:30.117 INFO:tasks.ceph.osd.6:Stopped 2024-03-06T07:36:30.117 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-06T07:36:30.117 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:30.189 INFO:tasks.ceph.osd.7:Stopped 2024-03-06T07:36:30.190 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-06T07:36:30.190 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-06T07:36:30.190 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:30.214 INFO:tasks.ceph.mgr.y:Stopped 2024-03-06T07:36:30.214 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-06T07:36:30.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:30.259 INFO:tasks.ceph.mgr.x:Stopped 2024-03-06T07:36:30.260 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-06T07:36:30.260 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-06T07:36:30.260 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:30.271 INFO:tasks.ceph.mon.a:Stopped 2024-03-06T07:36:30.272 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-06T07:36:30.272 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:30.313 INFO:tasks.ceph.mon.b:Stopped 2024-03-06T07:36:30.313 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-06T07:36:30.313 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:30.367 INFO:tasks.ceph.mon.c:Stopped 2024-03-06T07:36:30.367 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-06T07:36:30.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-06T07:36:30.402 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:36:30.402 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-06T07:36:30.613 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:36:30.613 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-06T07:36:30.704 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:36:30.704 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-06T07:36:30.760 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:36:30.760 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-06T07:36:30.811 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:36:30.811 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-06T07:36:30.951 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:36:30.951 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-06T07:36:31.063 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:36:31.064 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-06T07:36:31.123 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:36:31.123 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-06T07:36:31.188 INFO:tasks.ceph:Archiving mon data... 2024-03-06T07:36:31.189 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/data/mon.a.tgz 2024-03-06T07:36:31.190 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-03-06T07:36:31.206 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.U0G2ecmQxT 2024-03-06T07:36:31.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.U0G2ecmQxT 2024-03-06T07:36:31.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.U0G2ecmQxT 2024-03-06T07:36:31.450 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.U0G2ecmQxT is 406KB 2024-03-06T07:36:31.547 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.U0G2ecmQxT 2024-03-06T07:36:31.561 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/data/mon.b.tgz 2024-03-06T07:36:31.562 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-03-06T07:36:31.579 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.nGVcIFwDBn 2024-03-06T07:36:31.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.nGVcIFwDBn 2024-03-06T07:36:31.746 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.nGVcIFwDBn 2024-03-06T07:36:31.831 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.nGVcIFwDBn is 412KB 2024-03-06T07:36:31.929 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.nGVcIFwDBn 2024-03-06T07:36:31.943 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/data/mon.c.tgz 2024-03-06T07:36:31.943 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-03-06T07:36:31.997 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.uFpcUny3gk 2024-03-06T07:36:31.997 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.uFpcUny3gk 2024-03-06T07:36:32.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.uFpcUny3gk 2024-03-06T07:36:32.224 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.uFpcUny3gk is 412KB 2024-03-06T07:36:32.290 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.uFpcUny3gk 2024-03-06T07:36:32.305 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-06T07:36:32.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:36:32.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:36:32.396 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:36:32.425 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:36:32.425 INFO:tasks.ceph:Archiving crash dumps... 2024-03-06T07:36:32.427 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/remote/smithi027/crash 2024-03-06T07:36:32.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:36:32.469 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/remote/smithi069/crash 2024-03-06T07:36:32.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:36:32.496 INFO:tasks.ceph:Shutting down logrotate 2024-03-06T07:36:32.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:32.513 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:32.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:32.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:32.647 INFO:tasks.ceph:Compressing logs... 2024-03-06T07:36:32.648 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:36:32.664 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:36:32.739 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-06T07:36:32.740 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37824.log 2024-03-06T07:36:32.740 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37864.log 2024-03-06T07:36:32.740 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.37824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44642.log 2024-03-06T07:36:32.740 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37864.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph.audit.log 2024-03-06T07:36:32.740 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37824.log.gz 2024-03-06T07:36:32.740 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37864.log.gz 2024-03-06T07:36:32.740 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-06T07:36:32.740 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.3.44642.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38024.log 2024-03-06T07:36:32.741 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-06T07:36:32.741 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37944.log 2024-03-06T07:36:32.741 INFO:teuthology.orchestra.run.smithi069.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.3.44642.log.gz 2024-03-06T07:36:32.741 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38024.log.gz 2024-03-06T07:36:32.741 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-06T07:36:32.741 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37944.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-06T07:36:32.741 INFO:teuthology.orchestra.run.smithi069.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37944.log.gz 2024-03-06T07:36:32.741 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38064.log 2024-03-06T07:36:32.741 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44190.log 2024-03-06T07:36:32.742 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37984.log 2024-03-06T07:36:32.742 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38064.log.gz 2024-03-06T07:36:32.742 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.1.44190.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-06T07:36:32.742 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37984.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37984.log.gz 2024-03-06T07:36:32.742 INFO:teuthology.orchestra.run.smithi069.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.1.44190.log.gz 2024-03-06T07:36:32.742 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:36:32.742 INFO:teuthology.orchestra.run.smithi069.stderr: 93.1%/var/log/ceph/ceph-mgr.x.log: -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-06T07:36:32.742 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37784.log 2024-03-06T07:36:32.742 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-06T07:36:32.743 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37784.log: /var/log/ceph/ceph.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37784.log.gz 2024-03-06T07:36:32.743 INFO:teuthology.orchestra.run.smithi069.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:36:32.743 INFO:teuthology.orchestra.run.smithi069.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:36:32.743 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-06T07:36:32.743 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37904.log 2024-03-06T07:36:32.743 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-06T07:36:32.743 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37904.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37904.log.gz 2024-03-06T07:36:32.746 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-06T07:36:32.908 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-06T07:36:33.023 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76922.log 2024-03-06T07:36:33.023 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78861.log 2024-03-06T07:36:33.023 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92851.log 2024-03-06T07:36:33.023 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.78861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63195.log 2024-03-06T07:36:33.023 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76922.log.gz 2024-03-06T07:36:33.024 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54873.log 2024-03-06T07:36:33.024 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63195.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92851.log.gz 2024-03-06T07:36:33.024 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84674.log 2024-03-06T07:36:33.024 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 87.6% --verbose -- /var/log/ceph/ceph-client.admin.71023.log -- replaced with /var/log/ceph/ceph-client.admin.63195.log.gz 2024-03-06T07:36:33.024 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.024 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54873.log: /var/log/ceph/ceph-client.admin.84674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66978.log 2024-03-06T07:36:33.024 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54873.log.gz 2024-03-06T07:36:33.024 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85821.log 2024-03-06T07:36:33.025 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.84674.log.gz 2024-03-06T07:36:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67884.log 2024-03-06T07:36:33.025 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71023.log.gz 2024-03-06T07:36:33.025 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66978.log.gz 2024-03-06T07:36:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51500.log 2024-03-06T07:36:33.025 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85821.log.gz 2024-03-06T07:36:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82747.log 2024-03-06T07:36:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49737.log 2024-03-06T07:36:33.025 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67884.log.gz 2024-03-06T07:36:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82747.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51500.log.gz 2024-03-06T07:36:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45338.log 2024-03-06T07:36:33.026 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82747.log.gz 2024-03-06T07:36:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48154.log 2024-03-06T07:36:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45338.log: gzip -5 87.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.49737.log.gz 92.8% 2024-03-06T07:36:33.026 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.80186.log -- replaced with /var/log/ceph/ceph-client.0.78861.log.gz 2024-03-06T07:36:33.026 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.026 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45338.log.gz 2024-03-06T07:36:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72283.log 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80186.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48154.log.gz 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72283.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80186.log.gz 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72637.log 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72283.log.gz 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49621.log 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62579.log 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49621.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72637.log.gz 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59611.log 2024-03-06T07:36:33.027 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49621.log.gz 2024-03-06T07:36:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62387.log 2024-03-06T07:36:33.028 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62579.log.gz 2024-03-06T07:36:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72111.log 2024-03-06T07:36:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62387.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59611.log.gz 2024-03-06T07:36:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58317.log 2024-03-06T07:36:33.028 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62387.log.gz 2024-03-06T07:36:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87363.log 2024-03-06T07:36:33.028 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72111.log.gz 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91615.log 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85983.log 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58317.log.gz 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91615.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87363.log.gz 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59059.log 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91615.log.gz 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55167.log 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57311.log 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85983.log.gz 2024-03-06T07:36:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55167.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59059.log.gz 2024-03-06T07:36:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86025.log 2024-03-06T07:36:33.030 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55167.log.gz 2024-03-06T07:36:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88599.log 2024-03-06T07:36:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50587.log 2024-03-06T07:36:33.030 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57311.log.gz 2024-03-06T07:36:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88599.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86025.log.gz 2024-03-06T07:36:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87027.log 2024-03-06T07:36:33.030 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88599.log.gz 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61003.log 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87027.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50587.log.gz 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51752.log 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87027.log.gz 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70939.log 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61003.log.gz 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82957.log 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71462.log 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51752.log.gz 2024-03-06T07:36:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82957.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70939.log.gz 2024-03-06T07:36:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86463.log 2024-03-06T07:36:33.032 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82957.log.gz 2024-03-06T07:36:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61638.log 2024-03-06T07:36:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70687.log 2024-03-06T07:36:33.032 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71462.log.gz 2024-03-06T07:36:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61638.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86463.log.gz 2024-03-06T07:36:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54259.log 2024-03-06T07:36:33.032 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61638.log.gz 2024-03-06T07:36:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64990.log 2024-03-06T07:36:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53320.log 2024-03-06T07:36:33.033 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70687.log.gz 2024-03-06T07:36:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64990.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54259.log.gz 2024-03-06T07:36:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44988.log 2024-03-06T07:36:33.033 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64990.log.gz 2024-03-06T07:36:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43728.log 2024-03-06T07:36:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75985.log 2024-03-06T07:36:33.033 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53320.log.gz 2024-03-06T07:36:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43728.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44988.log.gz 2024-03-06T07:36:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55346.log 2024-03-06T07:36:33.034 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43728.log.gz 2024-03-06T07:36:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50209.log 2024-03-06T07:36:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81801.log 2024-03-06T07:36:33.034 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75985.log.gz 2024-03-06T07:36:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50209.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55346.log.gz 2024-03-06T07:36:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88305.log 2024-03-06T07:36:33.034 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50209.log.gz 2024-03-06T07:36:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75587.log 2024-03-06T07:36:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88305.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.81801.log.gz 2024-03-06T07:36:33.035 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87885.log 2024-03-06T07:36:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75587.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88305.log.gz 2024-03-06T07:36:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73579.log 2024-03-06T07:36:33.035 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75587.log.gz 2024-03-06T07:36:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88515.log 2024-03-06T07:36:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73579.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.87885.log.gz -5 2024-03-06T07:36:33.035 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69690.log 2024-03-06T07:36:33.036 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73579.log.gz 2024-03-06T07:36:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64168.log 2024-03-06T07:36:33.036 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88515.log.gz 2024-03-06T07:36:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50335.log 2024-03-06T07:36:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77530.log 2024-03-06T07:36:33.036 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69690.log.gz 2024-03-06T07:36:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50335.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64168.log.gz 2024-03-06T07:36:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60732.log 2024-03-06T07:36:33.036 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50335.log.gz 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43905.log 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63553.log 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77530.log.gz 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43905.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60732.log.gz 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70771.log 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43905.log.gz 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49695.log 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53770.log 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63553.log.gz 2024-03-06T07:36:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49695.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70771.log.gz 2024-03-06T07:36:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68098.log 2024-03-06T07:36:33.038 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49695.log.gz 2024-03-06T07:36:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83751.log 2024-03-06T07:36:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68098.log: gzip 88.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.53770.log.gz -- 2024-03-06T07:36:33.038 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.46080.log 2024-03-06T07:36:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83751.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68098.log.gz 2024-03-06T07:36:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92809.log 2024-03-06T07:36:33.038 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83751.log.gz 2024-03-06T07:36:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53425.log 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70645.log 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46080.log.gz 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53425.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92809.log.gz 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76752.log 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53425.log.gz 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80666.log 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56956.log 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70645.log.gz 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76752.log.gz 2024-03-06T07:36:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75649.log 2024-03-06T07:36:33.040 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80666.log.gz 2024-03-06T07:36:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54793.log 2024-03-06T07:36:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75649.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56956.log.gz 2024-03-06T07:36:33.040 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75649.log.gz 2024-03-06T07:36:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61339.log 2024-03-06T07:36:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92302.log 2024-03-06T07:36:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61339.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54793.log.gz 2024-03-06T07:36:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68010.log 2024-03-06T07:36:33.040 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61339.log.gz 2024-03-06T07:36:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67680.log 2024-03-06T07:36:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72721.log 2024-03-06T07:36:33.041 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.92302.log.gz 2024-03-06T07:36:33.041 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68010.log.gz 2024-03-06T07:36:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91405.log 2024-03-06T07:36:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76795.log 2024-03-06T07:36:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72721.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67680.log.gz 2024-03-06T07:36:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45331.log 2024-03-06T07:36:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76795.log: gzip 91.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.72721.log.gz --verbose 2024-03-06T07:36:33.042 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.46164.log 2024-03-06T07:36:33.042 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91405.log.gz 2024-03-06T07:36:33.042 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76795.log.gz 2024-03-06T07:36:33.042 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71168.log 2024-03-06T07:36:33.042 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45331.log.gz 2024-03-06T07:36:33.042 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48070.log 2024-03-06T07:36:33.042 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71168.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46164.log.gz 2024-03-06T07:36:33.042 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73763.log 2024-03-06T07:36:33.042 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71168.log.gz 2024-03-06T07:36:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50965.log 2024-03-06T07:36:33.043 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48070.log.gz 2024-03-06T07:36:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50167.log 2024-03-06T07:36:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50965.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.78452.log 2024-03-06T07:36:33.043 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73763.log.gz 2024-03-06T07:36:33.043 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50965.log.gz 2024-03-06T07:36:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45605.log 2024-03-06T07:36:33.043 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50167.log.gz 2024-03-06T07:36:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52021.log 2024-03-06T07:36:33.044 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74699.log 2024-03-06T07:36:33.044 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78452.log.gz 2024-03-06T07:36:33.044 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45605.log.gz 2024-03-06T07:36:33.044 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73365.log 2024-03-06T07:36:33.044 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52021.log: /var/log/ceph/ceph-client.0.74699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73453.log 2024-03-06T07:36:33.044 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52021.log.gz 2024-03-06T07:36:33.044 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56662.log 2024-03-06T07:36:33.045 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86715.log 2024-03-06T07:36:33.045 INFO:teuthology.orchestra.run.smithi027.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.73365.log.gz 2024-03-06T07:36:33.045 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73453.log.gz 2024-03-06T07:36:33.045 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88095.log 2024-03-06T07:36:33.045 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56662.log.gz 2024-03-06T07:36:33.045 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48897.log 2024-03-06T07:36:33.045 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92083.log 2024-03-06T07:36:33.045 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86715.log.gz 2024-03-06T07:36:33.045 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48897.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88095.log.gz 2024-03-06T07:36:33.046 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61434.log 2024-03-06T07:36:33.046 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56186.log 2024-03-06T07:36:33.046 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92083.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48897.log.gz 2024-03-06T07:36:33.046 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61434.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.92083.log.gz 2024-03-06T07:36:33.046 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90837.log 2024-03-06T07:36:33.046 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61434.log.gz 2024-03-06T07:36:33.046 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57437.log 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56186.log: /var/log/ceph/ceph-client.admin.90837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80396.log 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56186.log.gz 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90837.log.gz 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67926.log 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57437.log.gz 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66894.log 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80852.log 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67926.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80396.log.gz 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66894.log: gzip -5 --verbose 89.3% -- -- replaced with /var/log/ceph/ceph-client.admin.67926.log.gz /var/log/ceph/ceph-client.admin.80810.log 2024-03-06T07:36:33.047 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.048 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66894.log.gz 2024-03-06T07:36:33.048 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63275.log 2024-03-06T07:36:33.048 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80852.log: /var/log/ceph/ceph-client.admin.80810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55081.log 2024-03-06T07:36:33.048 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63275.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80810.log.gz 2024-03-06T07:36:33.048 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68550.log 2024-03-06T07:36:33.048 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.80852.log.gz 2024-03-06T07:36:33.048 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63275.log.gz 2024-03-06T07:36:33.048 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55081.log: 93.5% -- replaced with /var/log/ceph/ceph-client.0.74699.log.gz 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54996.log 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55081.log.gz 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73847.log 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68550.log: /var/log/ceph/ceph-client.admin.54996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74440.log 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68550.log.gz 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54996.log.gz 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71065.log 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61129.log 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73847.log.gz 2024-03-06T07:36:33.049 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71065.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74440.log.gz 2024-03-06T07:36:33.050 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44990.log 2024-03-06T07:36:33.050 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71065.log.gz 2024-03-06T07:36:33.050 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.85237.log 2024-03-06T07:36:33.050 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44990.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61129.log.gz 2024-03-06T07:36:33.050 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54217.log 2024-03-06T07:36:33.050 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.85237.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44990.log.gz 2024-03-06T07:36:33.050 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88911.log 2024-03-06T07:36:33.050 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81572.log 2024-03-06T07:36:33.051 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88911.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54217.log.gz 2024-03-06T07:36:33.051 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52983.log 2024-03-06T07:36:33.051 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88911.log.gz 2024-03-06T07:36:33.051 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85595.log 2024-03-06T07:36:33.051 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81572.log: /var/log/ceph/ceph-client.admin.52983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47986.log 2024-03-06T07:36:33.051 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81572.log.gz 2024-03-06T07:36:33.051 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85595.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52983.log.gz 2024-03-06T07:36:33.051 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82831.log 2024-03-06T07:36:33.052 INFO:teuthology.orchestra.run.smithi027.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.85237.log.gz 2024-03-06T07:36:33.052 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85595.log.gz 2024-03-06T07:36:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92557.log 2024-03-06T07:36:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47986.log: /var/log/ceph/ceph-client.admin.82831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63832.log 2024-03-06T07:36:33.052 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47986.log.gz 2024-03-06T07:36:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92557.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82831.log.gz 2024-03-06T07:36:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44396.log 2024-03-06T07:36:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67842.log 2024-03-06T07:36:33.052 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92557.log.gz 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44396.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63832.log.gz 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45870.log 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44396.log.gz 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52357.log 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67842.log.gz 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54656.log 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45870.log: /var/log/ceph/ceph-client.admin.52357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77278.log 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45870.log.gz 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52357.log.gz 2024-03-06T07:36:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54954.log 2024-03-06T07:36:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77278.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.54656.log.gz -5 2024-03-06T07:36:33.054 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.82522.log 2024-03-06T07:36:33.054 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77278.log.gz 2024-03-06T07:36:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86631.log 2024-03-06T07:36:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82522.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54954.log.gz 2024-03-06T07:36:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89415.log 2024-03-06T07:36:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86631.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82522.log.gz 2024-03-06T07:36:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75055.log 2024-03-06T07:36:33.054 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86631.log.gz 2024-03-06T07:36:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.89415.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44820.log 2024-03-06T07:36:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70266.log 2024-03-06T07:36:33.055 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89415.log.gz 2024-03-06T07:36:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44820.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75055.log.gz 2024-03-06T07:36:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51584.log 2024-03-06T07:36:33.055 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44820.log.gz 2024-03-06T07:36:33.055 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45333.log 2024-03-06T07:36:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51584.log: gzip 88.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.70266.log.gz -- 2024-03-06T07:36:33.056 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.72451.log 2024-03-06T07:36:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45333.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51584.log.gz 2024-03-06T07:36:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88137.log 2024-03-06T07:36:33.056 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45333.log.gz 2024-03-06T07:36:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56788.log 2024-03-06T07:36:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60774.log 2024-03-06T07:36:33.056 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72451.log.gz 2024-03-06T07:36:33.056 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56788.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88137.log.gz 2024-03-06T07:36:33.057 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90105.log 2024-03-06T07:36:33.057 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56788.log.gz 2024-03-06T07:36:33.057 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44652.log 2024-03-06T07:36:33.057 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90105.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60774.log.gz 2024-03-06T07:36:33.057 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55452.log 2024-03-06T07:36:33.057 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90105.log.gz 2024-03-06T07:36:33.057 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90291.log 2024-03-06T07:36:33.057 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44652.log.gz 2024-03-06T07:36:33.057 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48028.log 2024-03-06T07:36:33.058 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59411.log 2024-03-06T07:36:33.058 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55452.log.gz 2024-03-06T07:36:33.058 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48028.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.90291.log.gz 2024-03-06T07:36:33.058 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58967.log 2024-03-06T07:36:33.058 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48028.log.gz 2024-03-06T07:36:33.058 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72787.log 2024-03-06T07:36:33.058 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62995.log 2024-03-06T07:36:33.058 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59411.log.gz 2024-03-06T07:36:33.058 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72787.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58967.log.gz 2024-03-06T07:36:33.059 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80102.log 2024-03-06T07:36:33.059 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62995.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72787.log.gz 2024-03-06T07:36:33.059 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91237.log 2024-03-06T07:36:33.059 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80102.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62995.log.gz 2024-03-06T07:36:33.059 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45954.log 2024-03-06T07:36:33.059 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80102.log.gz 2024-03-06T07:36:33.059 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90501.log 2024-03-06T07:36:33.059 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84506.log 2024-03-06T07:36:33.059 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.91237.log.gz 2024-03-06T07:36:33.060 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45954.log.gz 2024-03-06T07:36:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87615.log 2024-03-06T07:36:33.060 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90501.log.gz 2024-03-06T07:36:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56746.log 2024-03-06T07:36:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76473.log 88.3% 2024-03-06T07:36:33.060 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84506.log.gz 2024-03-06T07:36:33.060 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87615.log.gz 2024-03-06T07:36:33.060 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81320.log 2024-03-06T07:36:33.061 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56746.log.gz 2024-03-06T07:36:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87447.log 2024-03-06T07:36:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81320.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76473.log.gz 2024-03-06T07:36:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85695.log 2024-03-06T07:36:33.061 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3%/var/log/ceph/ceph-client.admin.87447.log: -- replaced with /var/log/ceph/ceph-client.admin.81320.log.gz 2024-03-06T07:36:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50545.log 2024-03-06T07:36:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64355.log/var/log/ceph/ceph-client.admin.85695.log: 2024-03-06T07:36:33.061 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87447.log.gz 2024-03-06T07:36:33.061 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86109.log 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85695.log.gz 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64355.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50545.log.gz 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64702.log 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64355.log.gz 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52315.log 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76431.log 89.3% 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86109.log.gz 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64702.log.gz 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79748.log 2024-03-06T07:36:33.062 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52315.log.gz 2024-03-06T07:36:33.063 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79850.log 2024-03-06T07:36:33.063 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80957.log 2024-03-06T07:36:33.063 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76431.log.gz 2024-03-06T07:36:33.063 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79850.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79748.log.gz 2024-03-06T07:36:33.063 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79198.log 2024-03-06T07:36:33.063 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79850.log.gz 2024-03-06T07:36:33.063 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72595.log 2024-03-06T07:36:33.063 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48550.log 2024-03-06T07:36:33.064 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80957.log.gz 2024-03-06T07:36:33.064 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79198.log.gz 2024-03-06T07:36:33.064 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57185.log 2024-03-06T07:36:33.064 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72595.log.gz 2024-03-06T07:36:33.064 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65255.log 2024-03-06T07:36:33.064 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66410.log 2024-03-06T07:36:33.064 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48550.log.gz 2024-03-06T07:36:33.064 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65255.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57185.log.gz 2024-03-06T07:36:33.065 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66852.log 2024-03-06T07:36:33.065 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65255.log.gz 2024-03-06T07:36:33.065 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44996.log 2024-03-06T07:36:33.065 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66410.log.gz 2024-03-06T07:36:33.065 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62135.log 2024-03-06T07:36:33.065 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61554.log 2024-03-06T07:36:33.065 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66852.log.gz 2024-03-06T07:36:33.065 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62135.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44996.log.gz 2024-03-06T07:36:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55620.log 2024-03-06T07:36:33.066 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62135.log.gz 2024-03-06T07:36:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59549.log 2024-03-06T07:36:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69876.log 2024-03-06T07:36:33.066 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61554.log.gz 2024-03-06T07:36:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59549.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55620.log.gz 2024-03-06T07:36:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89121.log 2024-03-06T07:36:33.066 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59549.log.gz 2024-03-06T07:36:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53602.log 2024-03-06T07:36:33.067 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44071.log 2024-03-06T07:36:33.067 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69876.log.gz 2024-03-06T07:36:33.067 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53602.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89121.log.gz 2024-03-06T07:36:33.067 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84081.log 2024-03-06T07:36:33.067 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53602.log.gz 2024-03-06T07:36:33.067 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63874.log 2024-03-06T07:36:33.067 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73495.log 2024-03-06T07:36:33.067 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44071.log.gz 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63874.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84081.log.gz 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64607.log 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63874.log.gz 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87069.log 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64607.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73495.log.gz 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45335.log 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64607.log.gz/var/log/ceph/ceph-client.admin.87069.log: 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65171.log 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45335.log: gzip -5 88.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.87069.log.gz 2024-03-06T07:36:33.068 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.61297.log 2024-03-06T07:36:33.069 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4%/var/log/ceph/ceph-client.admin.65171.log: -- replaced with /var/log/ceph/ceph-client.admin.45335.log.gz 2024-03-06T07:36:33.069 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38067.log 2024-03-06T07:36:33.069 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65171.log.gz 2024-03-06T07:36:33.069 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81656.log 2024-03-06T07:36:33.069 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61297.log.gz 2024-03-06T07:36:33.069 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49537.log 2024-03-06T07:36:33.069 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81656.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.49963.log 2024-03-06T07:36:33.069 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38067.log.gz 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49537.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81656.log.gz 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60690.log 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49537.log.gz 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89769.log 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47129.log 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49963.log.gz 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89769.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60690.log.gz 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81404.log 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89769.log.gz 2024-03-06T07:36:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81362.log 2024-03-06T07:36:33.071 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80060.log 2024-03-06T07:36:33.071 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47129.log.gz 2024-03-06T07:36:33.071 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81362.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81404.log.gz 2024-03-06T07:36:33.071 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65739.log 2024-03-06T07:36:33.071 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81362.log.gz 2024-03-06T07:36:33.071 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64000.log 2024-03-06T07:36:33.071 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54530.log 2024-03-06T07:36:33.071 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80060.log.gz 2024-03-06T07:36:33.072 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64000.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65739.log.gz 2024-03-06T07:36:33.072 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77235.log 2024-03-06T07:36:33.072 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64000.log.gz 2024-03-06T07:36:33.072 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89331.log 2024-03-06T07:36:33.072 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76515.log 2024-03-06T07:36:33.072 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54530.log.gz 2024-03-06T07:36:33.072 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89331.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77235.log.gz 2024-03-06T07:36:33.072 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91279.log 2024-03-06T07:36:33.072 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89331.log.gz 2024-03-06T07:36:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84774.log 2024-03-06T07:36:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90063.log 2024-03-06T07:36:33.073 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76515.log.gz 2024-03-06T07:36:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84774.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91279.log.gz 2024-03-06T07:36:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77320.log 2024-03-06T07:36:33.073 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84774.log.gz 2024-03-06T07:36:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82617.log 2024-03-06T07:36:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88557.log 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90063.log.gz 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82617.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77320.log.gz 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76109.log 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82617.log.gz 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50377.log 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61213.log 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88557.log.gz 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50377.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76109.log.gz 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92210.log 2024-03-06T07:36:33.074 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50377.log.gz 2024-03-06T07:36:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68658.log 2024-03-06T07:36:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45912.log 2024-03-06T07:36:33.075 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61213.log.gz 2024-03-06T07:36:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68658.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92210.log.gz 2024-03-06T07:36:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88431.log 2024-03-06T07:36:33.075 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68658.log.gz 2024-03-06T07:36:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62513.log 2024-03-06T07:36:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88431.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45912.log.gz 2024-03-06T07:36:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91321.log 2024-03-06T07:36:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62513.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88431.log.gz 2024-03-06T07:36:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69938.log 2024-03-06T07:36:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49411.log 2024-03-06T07:36:33.076 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62513.log.gz 2024-03-06T07:36:33.076 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91321.log.gz 2024-03-06T07:36:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52441.log 2024-03-06T07:36:33.076 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69938.log.gz 2024-03-06T07:36:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90879.log 2024-03-06T07:36:33.077 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52441.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49411.log.gz 2024-03-06T07:36:33.077 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78410.log 2024-03-06T07:36:33.077 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52441.log.gz 2024-03-06T07:36:33.077 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44736.log 2024-03-06T07:36:33.077 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44345.log 2024-03-06T07:36:33.077 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90879.log.gz 2024-03-06T07:36:33.077 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78410.log.gz 2024-03-06T07:36:33.077 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77446.log 2024-03-06T07:36:33.077 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44736.log.gz 2024-03-06T07:36:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57269.log 2024-03-06T07:36:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51891.log 2024-03-06T07:36:33.078 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44345.log.gz 2024-03-06T07:36:33.078 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77446.log.gz 2024-03-06T07:36:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84428.log 2024-03-06T07:36:33.078 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57269.log.gz 2024-03-06T07:36:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50005.log 2024-03-06T07:36:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91802.log 2024-03-06T07:36:33.078 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51891.log.gz 2024-03-06T07:36:33.079 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84428.log.gz 2024-03-06T07:36:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56998.log 2024-03-06T07:36:33.079 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50005.log.gz 2024-03-06T07:36:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67188.log 2024-03-06T07:36:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51152.log 2024-03-06T07:36:33.079 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91802.log.gz 2024-03-06T07:36:33.079 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56998.log.gz 2024-03-06T07:36:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80480.log 2024-03-06T07:36:33.079 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67188.log.gz 2024-03-06T07:36:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44998.log 2024-03-06T07:36:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66348.log 2024-03-06T07:36:33.080 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51152.log.gz 2024-03-06T07:36:33.080 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80480.log.gz 2024-03-06T07:36:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52549.log 2024-03-06T07:36:33.080 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44998.log.gz 2024-03-06T07:36:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81969.log 2024-03-06T07:36:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50881.log 2024-03-06T07:36:33.080 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66348.log.gz 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52549.log.gz 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54446.log 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81969.log.gz 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83835.log 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54446.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50881.log.gz 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68507.log 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54446.log.gz 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75816.log 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83835.log.gz 2024-03-06T07:36:33.081 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86295.log 2024-03-06T07:36:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84165.log 2024-03-06T07:36:33.082 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68507.log.gz 2024-03-06T07:36:33.082 INFO:teuthology.orchestra.run.smithi027.stderr: 89.8%/var/log/ceph/ceph-client.admin.86295.log: -- replaced with /var/log/ceph/ceph-client.admin.75816.log.gz 2024-03-06T07:36:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83059.log 2024-03-06T07:36:33.082 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86295.log.gz 2024-03-06T07:36:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80144.log 2024-03-06T07:36:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64948.log 2024-03-06T07:36:33.082 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84165.log.gz 2024-03-06T07:36:33.082 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83059.log.gz 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67968.log 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80144.log.gz 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76557.log 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67968.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64948.log.gzgzip 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87531.log 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67968.log.gz 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46374.log 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76557.log.gz 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86421.log 2024-03-06T07:36:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46374.log: gzip 88.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.87531.log.gz -- 2024-03-06T07:36:33.084 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.67291.log 2024-03-06T07:36:33.084 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46374.log.gz 2024-03-06T07:36:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76270.log 2024-03-06T07:36:33.084 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86421.log.gz 2024-03-06T07:36:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83919.log 2024-03-06T07:36:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88389.log 2024-03-06T07:36:33.084 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67291.log.gz 2024-03-06T07:36:33.084 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76270.log.gz 2024-03-06T07:36:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55494.log 2024-03-06T07:36:33.085 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83919.log.gz 2024-03-06T07:36:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70534.log 2024-03-06T07:36:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77404.log 2024-03-06T07:36:33.085 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88389.log.gz 2024-03-06T07:36:33.085 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55494.log.gz 2024-03-06T07:36:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70366.log 2024-03-06T07:36:33.085 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70534.log.gz 2024-03-06T07:36:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90333.log 2024-03-06T07:36:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70366.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77404.log.gz 2024-03-06T07:36:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81083.log 2024-03-06T07:36:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45812.log 2024-03-06T07:36:33.086 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70366.log.gz 2024-03-06T07:36:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81083.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.90333.log.gz 2024-03-06T07:36:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50797.log 2024-03-06T07:36:33.086 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81083.log.gz 2024-03-06T07:36:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58925.log 2024-03-06T07:36:33.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83877.log 2024-03-06T07:36:33.087 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45812.log.gz 2024-03-06T07:36:33.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58925.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50797.log.gz 2024-03-06T07:36:33.087 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46248.log 2024-03-06T07:36:33.087 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58925.log.gz 2024-03-06T07:36:33.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65469.log 2024-03-06T07:36:33.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89727.log 2024-03-06T07:36:33.087 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83877.log.gz 2024-03-06T07:36:33.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65469.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46248.log.gz 2024-03-06T07:36:33.088 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84366.log 2024-03-06T07:36:33.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89727.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65469.log.gz 2024-03-06T07:36:33.088 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57101.log 2024-03-06T07:36:33.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84366.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89727.log.gz 2024-03-06T07:36:33.088 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89685.log 2024-03-06T07:36:33.088 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84366.log.gz 2024-03-06T07:36:33.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59285.log 2024-03-06T07:36:33.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78284.log 2024-03-06T07:36:33.089 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57101.log.gz 2024-03-06T07:36:33.089 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89685.log.gz 2024-03-06T07:36:33.089 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92515.log 2024-03-06T07:36:33.089 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59285.log.gz 2024-03-06T07:36:33.089 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64228.log 2024-03-06T07:36:33.089 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54404.log 2024-03-06T07:36:33.089 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78284.log.gz 2024-03-06T07:36:33.089 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92515.log.gz 2024-03-06T07:36:33.090 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51796.log 2024-03-06T07:36:33.090 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64228.log.gz 2024-03-06T07:36:33.090 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55211.log 2024-03-06T07:36:33.090 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82351.log 2024-03-06T07:36:33.090 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54404.log.gz 2024-03-06T07:36:33.090 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51796.log.gz 2024-03-06T07:36:33.090 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50461.log 2024-03-06T07:36:33.090 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55211.log.gz 2024-03-06T07:36:33.090 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44239.log 2024-03-06T07:36:33.091 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77006.log 2024-03-06T07:36:33.091 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82351.log.gz 2024-03-06T07:36:33.091 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50461.log.gz 2024-03-06T07:36:33.091 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44239.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31611.log 2024-03-06T07:36:33.091 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4%/var/log/ceph/ceph-client.admin.77006.log: -- replaced with /var/log/ceph/ceph-client.admin.44239.log.gz 2024-03-06T07:36:33.091 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90459.log 2024-03-06T07:36:33.091 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.tmp-client.admin.31611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56425.log 2024-03-06T07:36:33.091 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77006.log.gz 2024-03-06T07:36:33.091 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31611.log.gz 2024-03-06T07:36:33.092 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78196.log 2024-03-06T07:36:33.092 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90459.log.gz 2024-03-06T07:36:33.092 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45000.log 2024-03-06T07:36:33.092 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8%/var/log/ceph/ceph-client.admin.78196.log: -- replaced with /var/log/ceph/ceph-client.admin.56425.log.gz 2024-03-06T07:36:33.092 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44778.log 2024-03-06T07:36:33.092 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45000.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78196.log.gz 2024-03-06T07:36:33.092 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65032.log 2024-03-06T07:36:33.092 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6%/var/log/ceph/ceph-client.admin.44778.log: -- replaced with /var/log/ceph/ceph-client.admin.45000.log.gz 2024-03-06T07:36:33.092 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51007.log 2024-03-06T07:36:33.093 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44778.log.gz 2024-03-06T07:36:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86379.log 2024-03-06T07:36:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77048.log 2024-03-06T07:36:33.093 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65032.log.gz 2024-03-06T07:36:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86379.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51007.log.gz 2024-03-06T07:36:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74046.log 2024-03-06T07:36:33.093 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86379.log.gz 2024-03-06T07:36:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81236.log 2024-03-06T07:36:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64481.log 2024-03-06T07:36:33.094 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77048.log.gz 2024-03-06T07:36:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81236.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74046.log.gz 2024-03-06T07:36:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:36:33.094 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81236.log.gz 2024-03-06T07:36:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83997.log 2024-03-06T07:36:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.audit.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64481.log.gz 2024-03-06T07:36:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83185.log 2024-03-06T07:36:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64313.log 2024-03-06T07:36:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83185.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83997.log.gz 2024-03-06T07:36:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65427.log 2024-03-06T07:36:33.095 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83185.log.gz 2024-03-06T07:36:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44568.log 2024-03-06T07:36:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64313.log: /var/log/ceph/ceph-client.admin.65427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88995.log 2024-03-06T07:36:33.095 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64313.log.gz 2024-03-06T07:36:33.095 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65427.log.gz 2024-03-06T07:36:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57058.log 2024-03-06T07:36:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87405.log 2024-03-06T07:36:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88995.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44568.log.gz 2024-03-06T07:36:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82701.log 2024-03-06T07:36:33.096 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88995.log.gz 2024-03-06T07:36:33.096 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57058.log.gz 2024-03-06T07:36:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84224.log 2024-03-06T07:36:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87405.log: /var/log/ceph/ceph-client.admin.82701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88785.log 2024-03-06T07:36:33.096 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.87405.log.gz 2024-03-06T07:36:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84224.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82701.log.gz 2024-03-06T07:36:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.83552.log 2024-03-06T07:36:33.097 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84224.log.gz 2024-03-06T07:36:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85653.log 2024-03-06T07:36:33.097 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88785.log.gz 2024-03-06T07:36:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- 89.4% /var/log/ceph/ceph-client.admin.91657.log 2024-03-06T07:36:33.097 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:36:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.83552.log: /var/log/ceph/ceph-client.admin.85653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89373.log 2024-03-06T07:36:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91657.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85653.log.gz 2024-03-06T07:36:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85941.log 2024-03-06T07:36:33.097 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91657.log.gz 2024-03-06T07:36:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52808.log 2024-03-06T07:36:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89373.log: /var/log/ceph/ceph-client.admin.85941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71722.log 2024-03-06T07:36:33.098 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89373.log.gz 2024-03-06T07:36:33.098 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85941.log.gz 2024-03-06T07:36:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.52808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44480.log 2024-03-06T07:36:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46038.log 2024-03-06T07:36:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71722.log: /var/log/ceph/ceph-client.admin.44480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79503.log 2024-03-06T07:36:33.098 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71722.log.gz 2024-03-06T07:36:33.098 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44480.log.gz 2024-03-06T07:36:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82659.log 2024-03-06T07:36:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53965.log 2024-03-06T07:36:33.099 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46038.log.gz 2024-03-06T07:36:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79503.log: /var/log/ceph/ceph-client.admin.82659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63511.log 2024-03-06T07:36:33.099 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79503.log.gz 2024-03-06T07:36:33.099 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82659.log.gz 2024-03-06T07:36:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85469.log 2024-03-06T07:36:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90753.log 2024-03-06T07:36:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63511.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53965.log.gz 2024-03-06T07:36:33.100 INFO:teuthology.orchestra.run.smithi027.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.0.83552.log.gz/var/log/ceph/ceph-client.admin.85469.log: 2024-03-06T07:36:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73925.log 2024-03-06T07:36:33.100 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63511.log.gz 2024-03-06T07:36:33.100 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85469.log.gz 2024-03-06T07:36:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51458.log 2024-03-06T07:36:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37695.log 2024-03-06T07:36:33.100 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90753.log.gz 2024-03-06T07:36:33.100 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73925.log.gz 2024-03-06T07:36:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86547.log 2024-03-06T07:36:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91363.log 2024-03-06T07:36:33.101 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2%/var/log/ceph/ceph-client.admin.37695.log: -- replaced with /var/log/ceph/ceph-client.admin.51458.log.gz 2024-03-06T07:36:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75438.log 2024-03-06T07:36:33.101 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37695.log.gz 2024-03-06T07:36:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60910.log 2024-03-06T07:36:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91363.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86547.log.gz 2024-03-06T07:36:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75438.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.91363.log.gz -5 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.92344.log 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92388.log 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60910.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75438.log.gz 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92344.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60910.log.gz 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80018.log 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68339.log 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92388.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92344.log.gz 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49821.log 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.92388.log.gz 2024-03-06T07:36:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56578.log 2024-03-06T07:36:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68339.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80018.log.gz 2024-03-06T07:36:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49821.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68339.log.gz 2024-03-06T07:36:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59453.log 2024-03-06T07:36:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80270.log 2024-03-06T07:36:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56578.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49821.log.gz 2024-03-06T07:36:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59453.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.56578.log.gz 2024-03-06T07:36:33.103 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61087.log 2024-03-06T07:36:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67020.log 2024-03-06T07:36:33.103 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59453.log.gz 2024-03-06T07:36:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80270.log: /var/log/ceph/ceph-client.admin.61087.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80270.log.gz 2024-03-06T07:36:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46122.log 2024-03-06T07:36:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92041.log 2024-03-06T07:36:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67020.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61087.log.gz 2024-03-06T07:36:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81614.log 2024-03-06T07:36:33.104 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67020.log.gz 2024-03-06T07:36:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91047.log 2024-03-06T07:36:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92041.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46122.log.gz 2024-03-06T07:36:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52231.log 2024-03-06T07:36:33.105 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92041.log.gz 2024-03-06T07:36:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92994.log 2024-03-06T07:36:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91047.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81614.log.gz 2024-03-06T07:36:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51626.log 2024-03-06T07:36:33.105 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.91047.log.gz 2024-03-06T07:36:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84548.log 2024-03-06T07:36:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92994.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52231.log.gz 2024-03-06T07:36:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51626.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92994.log.gz 2024-03-06T07:36:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67062.log 2024-03-06T07:36:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90669.log 2024-03-06T07:36:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84548.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51626.log.gz 2024-03-06T07:36:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67062.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.84548.log.gz 2024-03-06T07:36:33.106 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.92893.log 2024-03-06T07:36:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48939.log 2024-03-06T07:36:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90669.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67062.log.gz 2024-03-06T07:36:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69752.log 2024-03-06T07:36:33.106 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90669.log.gz 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45534.log 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48939.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92893.log.gz 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89979.log 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48939.log.gz 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81125.log 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45534.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69752.log.gz 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89979.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45534.log.gz 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44283.log 2024-03-06T07:36:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81446.log 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81125.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89979.log.gz 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44283.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81125.log.gz 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91107.log 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74224.log 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44283.log.gz 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81446.log: /var/log/ceph/ceph-client.admin.91107.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81446.log.gz 2024-03-06T07:36:33.108 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55704.log 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83709.log 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.91107.log.gz 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74224.log: /var/log/ceph/ceph-client.admin.55704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46458.log 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74224.log.gz 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57532.log 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83709.log: /var/log/ceph/ceph-client.admin.46458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75711.log 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83709.log.gz 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.55704.log.gz 2024-03-06T07:36:33.109 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46458.log.gz 2024-03-06T07:36:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.57532.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50251.log 2024-03-06T07:36:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75711.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57532.log.gz 2024-03-06T07:36:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55123.log 2024-03-06T07:36:33.110 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75711.log.gz 2024-03-06T07:36:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75120.log 2024-03-06T07:36:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50251.log: /var/log/ceph/ceph-client.admin.55123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87237.log 2024-03-06T07:36:33.110 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50251.log.gz 2024-03-06T07:36:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75120.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55123.log.gz 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74044.log 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75120.log.gz 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67104.log 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45324.log 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74044.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87237.log.gz 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67104.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74044.log.gzgzip 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62345.log 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67104.log.gz 2024-03-06T07:36:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65120.log 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45324.log: /var/log/ceph/ceph-client.admin.62345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92125.log 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45324.log.gz 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62345.log.gz 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51416.log 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65120.log.gz 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69260.log 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77090.log 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92125.log.gz 2024-03-06T07:36:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51416.log: /var/log/ceph/ceph-client.admin.69260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72060.log 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51416.log.gz 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69260.log.gz 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48448.log 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77090.log: /var/log/ceph/ceph-client.admin.72060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91447.log 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77090.log.gz 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72060.log.gz 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57632.log 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48448.log.gz 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55536.log 2024-03-06T07:36:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54488.log 2024-03-06T07:36:33.114 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91447.log.gz 2024-03-06T07:36:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57632.log: /var/log/ceph/ceph-client.admin.55536.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57632.log.gz 2024-03-06T07:36:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67501.log 2024-03-06T07:36:33.114 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55536.log.gz 2024-03-06T07:36:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67375.log 2024-03-06T07:36:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54488.log: /var/log/ceph/ceph-client.admin.67501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85553.log 2024-03-06T07:36:33.114 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54488.log.gz 2024-03-06T07:36:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67375.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67501.log.gz 2024-03-06T07:36:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75773.log 2024-03-06T07:36:33.115 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67375.log.gz 2024-03-06T07:36:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58509.log 2024-03-06T07:36:33.115 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.85553.log.gz 2024-03-06T07:36:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62838.log 2024-03-06T07:36:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75773.log: /var/log/ceph/ceph-client.admin.58509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84732.log 2024-03-06T07:36:33.115 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75773.log.gz 2024-03-06T07:36:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.62838.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58509.log.gz 2024-03-06T07:36:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54301.log 2024-03-06T07:36:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.47501.log 2024-03-06T07:36:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84732.log: /var/log/ceph/ceph-client.admin.54301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55662.log 2024-03-06T07:36:33.116 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84732.log.gz 2024-03-06T07:36:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.2.47501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50419.log 2024-03-06T07:36:33.116 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54301.log.gz 2024-03-06T07:36:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60961.log 2024-03-06T07:36:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50419.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.55662.log.gz 2024-03-06T07:36:33.116 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80438.log 2024-03-06T07:36:33.116 INFO:teuthology.orchestra.run.smithi027.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.2.47501.log.gz 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr:gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50419.log.gz 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose --/var/log/ceph/ceph-client.admin.60961.log: /var/log/ceph/ceph-client.admin.92767.log 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80438.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60961.log.gz 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52189.log 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80438.log.gz 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57880.log 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92767.log: /var/log/ceph/ceph-client.admin.52189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79892.log 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92767.log.gz 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56467.log 2024-03-06T07:36:33.117 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52189.log.gz 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43809.log 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57880.log.gz 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56467.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79892.log.gz 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47665.log 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6%/var/log/ceph/ceph-client.admin.43809.log: -- replaced with /var/log/ceph/ceph-client.admin.56467.log.gz 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92252.log 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47665.log: gzip 88.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.43809.log.gz -- 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.44694.log 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47665.log.gz 2024-03-06T07:36:33.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72535.log 2024-03-06T07:36:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44610.log 2024-03-06T07:36:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44694.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92252.log.gz 2024-03-06T07:36:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84286.log 2024-03-06T07:36:33.119 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44694.log.gz 2024-03-06T07:36:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:gzip 87.9% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.72535.log.gz 2024-03-06T07:36:33.119 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.81927.log 2024-03-06T07:36:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44610.log: /var/log/ceph/ceph-client.admin.84286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50923.log 2024-03-06T07:36:33.119 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44610.log.gz 2024-03-06T07:36:33.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81927.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84286.log.gz 2024-03-06T07:36:33.120 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66106.log 2024-03-06T07:36:33.120 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46416.log 2024-03-06T07:36:33.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50923.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81927.log.gz 2024-03-06T07:36:33.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.66106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92473.log 2024-03-06T07:36:33.120 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50923.log.gz 2024-03-06T07:36:33.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84039.log 2024-03-06T07:36:33.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92473.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60522.log 2024-03-06T07:36:33.120 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46416.log.gz 2024-03-06T07:36:33.121 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92473.log.gz 2024-03-06T07:36:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74936.log 2024-03-06T07:36:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57143.log 2024-03-06T07:36:33.121 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84039.log.gz 2024-03-06T07:36:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60522.log: /var/log/ceph/ceph-client.admin.74936.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-06T07:36:33.121 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50503.log 2024-03-06T07:36:33.121 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74936.log.gz 2024-03-06T07:36:33.121 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60522.log.gz 2024-03-06T07:36:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57143.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59101.log 2024-03-06T07:36:33.122 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57143.log.gz 2024-03-06T07:36:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62177.log 2024-03-06T07:36:33.122 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50503.log.gz 2024-03-06T07:36:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70142.log 2024-03-06T07:36:33.122 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59101.log.gz 2024-03-06T07:36:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90417.log 2024-03-06T07:36:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62177.log: /var/log/ceph/ceph-client.admin.70142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50671.log 2024-03-06T07:36:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90417.log: 88.5% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62177.log.gz 2024-03-06T07:36:33.123 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70142.log.gz 2024-03-06T07:36:33.123 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.90417.log.gz 2024-03-06T07:36:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87279.log 2024-03-06T07:36:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75481.log 2024-03-06T07:36:33.123 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0%/var/log/ceph/ceph-client.admin.87279.log: -- replaced with /var/log/ceph/ceph-client.admin.50671.log.gz 2024-03-06T07:36:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66657.log 2024-03-06T07:36:33.123 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87279.log.gz 2024-03-06T07:36:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47944.log 2024-03-06T07:36:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75481.log: /var/log/ceph/ceph-client.admin.66657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50089.log 2024-03-06T07:36:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47944.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66657.log.gz 89.8% -- replaced with /var/log/ceph/ceph-client.admin.75481.log.gz 2024-03-06T07:36:33.124 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.124 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47944.log.gz 2024-03-06T07:36:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64782.log 2024-03-06T07:36:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68381.log 2024-03-06T07:36:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61171.log 2024-03-06T07:36:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64782.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50089.log.gz 2024-03-06T07:36:33.124 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68381.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64782.log.gz 2024-03-06T07:36:33.124 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68381.log.gz 2024-03-06T07:36:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63637.log 2024-03-06T07:36:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71888.log 2024-03-06T07:36:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63637.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61171.log.gz 2024-03-06T07:36:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64864.log 2024-03-06T07:36:33.125 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63637.log.gz 2024-03-06T07:36:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45612.log 2024-03-06T07:36:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71888.log: /var/log/ceph/ceph-client.admin.64864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78242.log 2024-03-06T07:36:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45612.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71888.log.gz 2024-03-06T07:36:33.126 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64864.log.gz 2024-03-06T07:36:33.126 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45612.log.gz 2024-03-06T07:36:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69542.log 2024-03-06T07:36:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80312.log 2024-03-06T07:36:33.126 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78242.log.gz 2024-03-06T07:36:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83793.log 2024-03-06T07:36:33.126 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69542.log.gz 2024-03-06T07:36:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54049.log 2024-03-06T07:36:33.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68465.log 2024-03-06T07:36:33.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83793.log: 89.1%/var/log/ceph/ceph-client.admin.54049.log: -- replaced with /var/log/ceph/ceph-client.admin.80312.log.gz 2024-03-06T07:36:33.127 INFO:teuthology.orchestra.run.smithi027.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.0.62838.log.gz 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83793.log.gz 2024-03-06T07:36:33.127 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.127 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54049.log.gz 2024-03-06T07:36:33.127 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80564.log 2024-03-06T07:36:33.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87843.log 2024-03-06T07:36:33.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80564.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68465.log.gz 2024-03-06T07:36:33.127 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91886.log 2024-03-06T07:36:33.128 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80564.log.gz 2024-03-06T07:36:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58233.log 2024-03-06T07:36:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90711.log 2024-03-06T07:36:33.128 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87843.log.gz 2024-03-06T07:36:33.128 INFO:teuthology.orchestra.run.smithi027.stderr: 95.5%/var/log/ceph/ceph-client.admin.58233.log: gzip -- replaced with /var/log/ceph/ceph-client.0.66106.log.gz 2024-03-06T07:36:33.128 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47902.log 2024-03-06T07:36:33.128 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91886.log.gz 2024-03-06T07:36:33.128 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58233.log.gz 2024-03-06T07:36:33.128 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91844.log 2024-03-06T07:36:33.129 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90711.log: /var/log/ceph/ceph-client.admin.47902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77132.log 2024-03-06T07:36:33.129 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90711.log.gz 2024-03-06T07:36:33.129 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47902.log.gz 2024-03-06T07:36:33.129 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73679.log 2024-03-06T07:36:33.129 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78017.log 2024-03-06T07:36:33.129 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77132.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91844.log.gz 2024-03-06T07:36:33.129 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67585.log 2024-03-06T07:36:33.129 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77132.log.gz 2024-03-06T07:36:33.130 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88827.log 2024-03-06T07:36:33.130 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73679.log.gz 2024-03-06T07:36:33.130 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60438.log 2024-03-06T07:36:33.130 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78017.log.gz 2024-03-06T07:36:33.130 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74374.log 2024-03-06T07:36:33.130 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2%/var/log/ceph/ceph-client.admin.88827.log: -- replaced with /var/log/ceph/ceph-client.admin.67585.log.gz 2024-03-06T07:36:33.130 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88179.log 2024-03-06T07:36:33.130 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88827.log.gz 2024-03-06T07:36:33.130 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60438.log.gz 2024-03-06T07:36:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87969.log 2024-03-06T07:36:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.88179.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.70020.log 2024-03-06T07:36:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87969.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88179.log.gz 2024-03-06T07:36:33.131 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61255.log 2024-03-06T07:36:33.131 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.74374.log.gz 2024-03-06T07:36:33.131 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87969.log.gz 2024-03-06T07:36:33.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54361.log 2024-03-06T07:36:33.131 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70020.log.gz 2024-03-06T07:36:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90189.log 2024-03-06T07:36:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61255.log: /var/log/ceph/ceph-client.admin.54361.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61255.log.gz 2024-03-06T07:36:33.132 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.92167.log 2024-03-06T07:36:33.132 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54361.log.gz 2024-03-06T07:36:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52483.log 2024-03-06T07:36:33.132 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90189.log.gz 2024-03-06T07:36:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63037.log 2024-03-06T07:36:33.132 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92167.log.gz 2024-03-06T07:36:33.132 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71804.log 2024-03-06T07:36:33.133 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.63037.log: --verbose -- /var/log/ceph/ceph-client.admin.69814.log 2024-03-06T07:36:33.133 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71804.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63037.log.gz 2024-03-06T07:36:33.133 INFO:teuthology.orchestra.run.smithi027.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.52483.log.gz 2024-03-06T07:36:33.133 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79976.log 2024-03-06T07:36:33.133 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71804.log.gz 2024-03-06T07:36:33.133 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69406.log 2024-03-06T07:36:33.133 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79976.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69814.log.gz 2024-03-06T07:36:33.133 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77829.log 2024-03-06T07:36:33.133 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79976.log.gz 2024-03-06T07:36:33.134 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89601.log 2024-03-06T07:36:33.134 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89811.log 2024-03-06T07:36:33.134 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77829.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69406.log.gz 2024-03-06T07:36:33.134 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70408.log 2024-03-06T07:36:33.134 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77829.log.gz 2024-03-06T07:36:33.134 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58401.log 2024-03-06T07:36:33.134 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89601.log.gz 2024-03-06T07:36:33.134 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89811.log.gz 2024-03-06T07:36:33.135 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54175.log 2024-03-06T07:36:33.135 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70000.log 2024-03-06T07:36:33.135 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70408.log.gz 2024-03-06T07:36:33.135 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58401.log: /var/log/ceph/ceph-client.admin.54175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91573.log 2024-03-06T07:36:33.135 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58401.log.gz 2024-03-06T07:36:33.135 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3%/var/log/ceph/ceph-client.admin.70000.log: -- replaced with /var/log/ceph/ceph-client.admin.54175.log.gzgzip 2024-03-06T07:36:33.135 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57395.log 2024-03-06T07:36:33.135 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62471.log 89.3% 2024-03-06T07:36:33.135 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70000.log.gz 2024-03-06T07:36:33.136 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91573.log.gz/var/log/ceph/ceph-client.admin.57395.log: 2024-03-06T07:36:33.136 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74047.log 2024-03-06T07:36:33.136 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62471.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57395.log.gz 2024-03-06T07:36:33.136 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74332.log 2024-03-06T07:36:33.136 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62471.log.gz 2024-03-06T07:36:33.136 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49579.log 2024-03-06T07:36:33.136 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74332.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74047.log.gz 2024-03-06T07:36:33.136 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 87.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.74332.log.gz /var/log/ceph/ceph-client.admin.79934.log 2024-03-06T07:36:33.136 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70450.log 2024-03-06T07:36:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49579.log: /var/log/ceph/ceph-client.admin.79934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64523.log 2024-03-06T07:36:33.137 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49579.log.gz 2024-03-06T07:36:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56872.log 2024-03-06T07:36:33.137 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79934.log.gz 2024-03-06T07:36:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72679.log 2024-03-06T07:36:33.137 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70450.log.gz 2024-03-06T07:36:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89163.log 2024-03-06T07:36:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56872.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64523.log.gz 2024-03-06T07:36:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62005.log 2024-03-06T07:36:33.138 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56872.log.gz 2024-03-06T07:36:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86589.log 2024-03-06T07:36:33.138 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72679.log.gz 2024-03-06T07:36:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77931.log 2024-03-06T07:36:33.138 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89163.log.gz 2024-03-06T07:36:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81488.log 2024-03-06T07:36:33.138 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62005.log.gz 2024-03-06T07:36:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54007.log 2024-03-06T07:36:33.139 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86589.log.gz 2024-03-06T07:36:33.139 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0%/var/log/ceph/ceph-client.admin.81488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89289.log 2024-03-06T07:36:33.139 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77931.log.gz 2024-03-06T07:36:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54007.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.81488.log.gz --verbose 2024-03-06T07:36:33.139 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.92641.log 2024-03-06T07:36:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89289.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54007.log.gz 2024-03-06T07:36:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57838.log 2024-03-06T07:36:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92641.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.86967.log 2024-03-06T07:36:33.140 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89289.log.gz 2024-03-06T07:36:33.140 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.92641.log.gz 2024-03-06T07:36:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88473.log 2024-03-06T07:36:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61764.log 2024-03-06T07:36:33.140 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57838.log.gz 2024-03-06T07:36:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.88473.log: --verbose -- /var/log/ceph/ceph-client.admin.69258.log 2024-03-06T07:36:33.140 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86967.log.gz 2024-03-06T07:36:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61764.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88473.log.gzgzip 2024-03-06T07:36:33.141 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45332.log 2024-03-06T07:36:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67459.log 2024-03-06T07:36:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69258.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61764.log.gz 2024-03-06T07:36:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79156.log 2024-03-06T07:36:33.141 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69258.log.gz 2024-03-06T07:36:33.141 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45332.log.gz 2024-03-06T07:36:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92954.log 2024-03-06T07:36:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59327.log 2024-03-06T07:36:33.141 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67459.log.gz 2024-03-06T07:36:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79156.log: /var/log/ceph/ceph-client.admin.92954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87489.log 2024-03-06T07:36:33.142 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79156.log.gz 2024-03-06T07:36:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59327.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92954.log.gz 2024-03-06T07:36:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64565.log 2024-03-06T07:36:33.142 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59327.log.gz 2024-03-06T07:36:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76332.log 2024-03-06T07:36:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87489.log: /var/log/ceph/ceph-client.admin.64565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50047.log 2024-03-06T07:36:33.142 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87489.log.gz 2024-03-06T07:36:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76332.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64565.log.gz 2024-03-06T07:36:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67146.log 2024-03-06T07:36:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50047.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76332.log.gz 2024-03-06T07:36:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76880.log 2024-03-06T07:36:33.143 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50047.log.gz 2024-03-06T07:36:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84900.log 2024-03-06T07:36:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64042.log 2024-03-06T07:36:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76880.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67146.log.gz 2024-03-06T07:36:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58061.log 2024-03-06T07:36:33.144 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76880.log.gz 2024-03-06T07:36:33.144 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84900.log.gz 2024-03-06T07:36:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79018.log 2024-03-06T07:36:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86757.log 2024-03-06T07:36:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58061.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64042.log.gz 2024-03-06T07:36:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79018.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58061.log.gz 2024-03-06T07:36:33.145 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.83101.log 2024-03-06T07:36:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86757.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79018.log.gz 2024-03-06T07:36:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91489.log 2024-03-06T07:36:33.145 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86757.log.gz 2024-03-06T07:36:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65385.log 2024-03-06T07:36:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83101.log: /var/log/ceph/ceph-client.admin.91489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90585.log 2024-03-06T07:36:33.145 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83101.log.gz 2024-03-06T07:36:33.145 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91489.log.gz 2024-03-06T07:36:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83293.log 2024-03-06T07:36:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55262.log 2024-03-06T07:36:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90585.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65385.log.gz 2024-03-06T07:36:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83293.log: gzip -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90585.log.gz 2024-03-06T07:36:33.146 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.68167.log 2024-03-06T07:36:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55262.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83293.log.gz 2024-03-06T07:36:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78154.log 2024-03-06T07:36:33.146 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55262.log.gz 2024-03-06T07:36:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53881.log 2024-03-06T07:36:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68167.log: /var/log/ceph/ceph-client.admin.78154.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68167.log.gz 2024-03-06T07:36:33.147 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.61921.log 2024-03-06T07:36:33.147 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78154.log.gz 2024-03-06T07:36:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89037.log 2024-03-06T07:36:33.147 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53881.log.gz 2024-03-06T07:36:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60816.log 2024-03-06T07:36:33.147 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61921.log.gz 2024-03-06T07:36:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76599.log 2024-03-06T07:36:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85511.log 2024-03-06T07:36:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60816.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89037.log.gz 2024-03-06T07:36:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76599.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-06T07:36:33.148 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60816.log.gz 2024-03-06T07:36:33.148 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76599.log.gz 2024-03-06T07:36:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79114.log 2024-03-06T07:36:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81041.log 2024-03-06T07:36:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.3.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85511.log.gz 2024-03-06T07:36:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71642.log 2024-03-06T07:36:33.149 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79114.log.gz/var/log/ceph/ceph-client.admin.81041.log: 2024-03-06T07:36:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51110.log 2024-03-06T07:36:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71642.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81041.log.gz 2024-03-06T07:36:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73048.log 2024-03-06T07:36:33.149 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71642.log.gz 2024-03-06T07:36:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49123.log 2024-03-06T07:36:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51110.log: /var/log/ceph/ceph-client.0.73048.log: gzip -5 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51110.log.gz 2024-03-06T07:36:33.149 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49327.log 2024-03-06T07:36:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55410.log 2024-03-06T07:36:33.150 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49123.log.gz 2024-03-06T07:36:33.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44197.log 2024-03-06T07:36:33.150 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49327.log.gz/var/log/ceph/ceph-client.admin.55410.log: 2024-03-06T07:36:33.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56704.log 2024-03-06T07:36:33.150 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55410.log.gz 2024-03-06T07:36:33.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77745.log 2024-03-06T07:36:33.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44197.log: /var/log/ceph/ceph-client.admin.56704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44027.log 2024-03-06T07:36:33.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77745.log: 88.8% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44197.log.gz 2024-03-06T07:36:33.150 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56704.log.gz 2024-03-06T07:36:33.151 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77745.log.gz 2024-03-06T07:36:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69500.log 2024-03-06T07:36:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86151.log 2024-03-06T07:36:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69500.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44027.log.gz 2024-03-06T07:36:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92430.log 2024-03-06T07:36:33.151 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69500.log.gz 2024-03-06T07:36:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51668.log 2024-03-06T07:36:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86151.log: /var/log/ceph/ceph-client.admin.92430.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86151.log.gz 2024-03-06T07:36:33.152 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92430.log.gz 2024-03-06T07:36:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58103.log 2024-03-06T07:36:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89895.log 2024-03-06T07:36:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58103.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51668.log.gz 2024-03-06T07:36:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88701.log 2024-03-06T07:36:33.152 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58103.log.gz 2024-03-06T07:36:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46602.log 2024-03-06T07:36:33.152 INFO:teuthology.orchestra.run.smithi027.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.0.73048.log.gz 2024-03-06T07:36:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89895.log: /var/log/ceph/ceph-client.admin.88701.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89895.log.gz 2024-03-06T07:36:33.153 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88701.log.gz 2024-03-06T07:36:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64271.log 2024-03-06T07:36:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47623.log 2024-03-06T07:36:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64271.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46602.log.gz 2024-03-06T07:36:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77625.log 2024-03-06T07:36:33.153 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64271.log.gz 2024-03-06T07:36:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88743.log 2024-03-06T07:36:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77625.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47623.log.gz 2024-03-06T07:36:33.154 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.65595.log 2024-03-06T07:36:33.154 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77625.log.gz 2024-03-06T07:36:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60202.log 2024-03-06T07:36:33.154 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88743.log.gz 2024-03-06T07:36:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62303.log 2024-03-06T07:36:33.154 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1%/var/log/ceph/ceph-client.admin.60202.log: -- replaced with /var/log/ceph/ceph-client.admin.65595.log.gz 2024-03-06T07:36:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48490.log 2024-03-06T07:36:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62303.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60202.log.gz 2024-03-06T07:36:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62303.log.gz 2024-03-06T07:36:33.155 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.68209.log 2024-03-06T07:36:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74874.log 2024-03-06T07:36:33.155 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48490.log.gz 2024-03-06T07:36:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72409.log 2024-03-06T07:36:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74874.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68209.log.gz 2024-03-06T07:36:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45325.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74874.log.gz 2024-03-06T07:36:33.155 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54091.log 2024-03-06T07:36:33.156 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72409.log.gz 2024-03-06T07:36:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81759.log 2024-03-06T07:36:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54091.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45325.log.gz 2024-03-06T07:36:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54091.log.gz 2024-03-06T07:36:33.156 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.51252.log 2024-03-06T07:36:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62429.log 2024-03-06T07:36:33.156 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81759.log.gz 2024-03-06T07:36:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46500.log 2024-03-06T07:36:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62429.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51252.log.gz 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62429.log.gz 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.67333.log 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93036.log 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82789.log 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr: 91.2%/var/log/ceph/ceph-client.admin.93036.log: -- replaced with /var/log/ceph/ceph-client.admin.46500.log.gz 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67333.log.gz 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56914.log 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93036.log.gz 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65781.log 2024-03-06T07:36:33.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82789.log: /var/log/ceph/ceph-client.admin.56914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44113.log 2024-03-06T07:36:33.158 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82789.log.gz 2024-03-06T07:36:33.158 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86211.log 2024-03-06T07:36:33.158 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56914.log.gz 2024-03-06T07:36:33.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65781.log: /var/log/ceph/ceph-client.admin.44113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50839.log 2024-03-06T07:36:33.158 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45613.log 2024-03-06T07:36:33.158 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44113.log.gz 2024-03-06T07:36:33.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86211.log: 91.1%gzip/var/log/ceph/ceph-client.admin.50839.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.87741.log 2024-03-06T07:36:33.158 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65781.log.gz 2024-03-06T07:36:33.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45613.log: 88.1% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86211.log.gz 2024-03-06T07:36:33.159 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50839.log.gz 2024-03-06T07:36:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61870.log 2024-03-06T07:36:33.159 INFO:teuthology.orchestra.run.smithi027.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45613.log.gz 2024-03-06T07:36:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86925.log 2024-03-06T07:36:33.159 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.87741.log.gz 2024-03-06T07:36:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72195.log 2024-03-06T07:36:33.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61870.log: /var/log/ceph/ceph-client.admin.86925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63748.log 2024-03-06T07:36:33.159 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4%/var/log/ceph/ceph-client.admin.72195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51933.log 2024-03-06T07:36:33.159 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86925.log.gz 2024-03-06T07:36:33.160 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61870.log.gz 2024-03-06T07:36:33.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63748.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72195.log.gz 2024-03-06T07:36:33.160 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63748.log.gz 2024-03-06T07:36:33.160 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82148.log 2024-03-06T07:36:33.160 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88869.log 2024-03-06T07:36:33.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51933.log: /var/log/ceph/ceph-client.admin.82148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54572.log 2024-03-06T07:36:33.160 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51933.log.gz 2024-03-06T07:36:33.160 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82148.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59227.log 2024-03-06T07:36:33.160 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88869.log: /var/log/ceph/ceph-client.admin.54572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63958.log 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59227.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54572.log.gz 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49039.log 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88869.log.gz 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89079.log 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59227.log.gz 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr:gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63958.log.gz 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr: -5/var/log/ceph/ceph-client.admin.49039.log: --verbose -- /var/log/ceph/ceph-client.admin.75314.log 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77787.log 2024-03-06T07:36:33.161 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49039.log.gz 2024-03-06T07:36:33.162 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89079.log.gz 2024-03-06T07:36:33.162 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61963.log 2024-03-06T07:36:33.162 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75314.log.gz 2024-03-06T07:36:33.162 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63595.log 2024-03-06T07:36:33.162 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61963.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77787.log.gz 2024-03-06T07:36:33.162 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72153.log 2024-03-06T07:36:33.162 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63595.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61963.log.gz 2024-03-06T07:36:33.162 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49495.log 2024-03-06T07:36:33.163 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63595.log.gz 2024-03-06T07:36:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85779.log 2024-03-06T07:36:33.163 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72153.log.gz 2024-03-06T07:36:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45655.log 2024-03-06T07:36:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52399.log 2024-03-06T07:36:33.163 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49495.log.gz 2024-03-06T07:36:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71930.log 2024-03-06T07:36:33.163 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85779.log.gz 2024-03-06T07:36:33.163 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52399.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45655.log.gz 2024-03-06T07:36:33.164 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.74045.log 2024-03-06T07:36:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71930.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52399.log.gz 2024-03-06T07:36:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.0.68919.log 2024-03-06T07:36:33.164 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71930.log.gz 2024-03-06T07:36:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63469.log 2024-03-06T07:36:33.164 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74045.log.gz 2024-03-06T07:36:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.68919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60480.log 2024-03-06T07:36:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46206.log 2024-03-06T07:36:33.164 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.60480.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56830.log 2024-03-06T07:36:33.165 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63469.log.gz 2024-03-06T07:36:33.165 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6%/var/log/ceph/ceph-client.admin.46206.log: -- replaced with /var/log/ceph/ceph-client.admin.60480.log.gz 2024-03-06T07:36:33.165 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83143.log 2024-03-06T07:36:33.165 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90231.log 2024-03-06T07:36:33.165 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46206.log.gz 2024-03-06T07:36:33.165 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83143.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56830.log.gz 2024-03-06T07:36:33.165 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71420.log 2024-03-06T07:36:33.165 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83143.log.gz 2024-03-06T07:36:33.166 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70897.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90231.log.gz 2024-03-06T07:36:33.166 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.166 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48855.log 2024-03-06T07:36:33.166 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2%/var/log/ceph/ceph-client.admin.70897.log: -- replaced with /var/log/ceph/ceph-client.admin.71420.log.gz 2024-03-06T07:36:33.166 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59673.log 2024-03-06T07:36:33.166 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70897.log.gz 2024-03-06T07:36:33.166 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81843.log 2024-03-06T07:36:33.166 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86337.log 2024-03-06T07:36:33.166 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6%/var/log/ceph/ceph-client.admin.59673.log: -- replaced with /var/log/ceph/ceph-client.admin.48855.log.gz 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62219.log 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59673.log.gz 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81843.log.gz 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91531.log 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86337.log.gz 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68297.log 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9%/var/log/ceph/ceph-client.admin.91531.log: -- replaced with /var/log/ceph/ceph-client.admin.62219.log.gz 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80750.log 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91531.log.gz 2024-03-06T07:36:33.167 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64439.log 2024-03-06T07:36:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68297.log: /var/log/ceph/ceph-client.admin.80750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51710.log 2024-03-06T07:36:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64439.log: 88.1% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68297.log.gz 2024-03-06T07:36:33.168 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80750.log.gz 2024-03-06T07:36:33.168 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64439.log.gz 2024-03-06T07:36:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82228.log 2024-03-06T07:36:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78602.log 2024-03-06T07:36:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51710.log: /var/log/ceph/ceph-client.admin.82228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55578.log 2024-03-06T07:36:33.168 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78602.log: 88.7% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82228.log.gz 2024-03-06T07:36:33.168 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51710.log.gz 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78602.log.gz 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80708.log 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65301.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55578.log.gz 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80708.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80708.log.gz 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52147.log 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.65301.log: --verbose -- /var/log/ceph/ceph-client.admin.58443.log 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84816.log 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65301.log.gz 2024-03-06T07:36:33.169 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52147.log.gz/var/log/ceph/ceph-client.admin.58443.log: 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48196.log 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75905.log 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr: 91.2%/var/log/ceph/ceph-client.admin.48196.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84816.log.gz 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58443.log.gz 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48196.log.gz 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44999.log 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61826.log 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45654.log 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75905.log.gz 2024-03-06T07:36:33.170 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61826.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-06T07:36:33.171 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44999.log.gz 2024-03-06T07:36:33.171 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53240.log 2024-03-06T07:36:33.171 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61826.log.gz 2024-03-06T07:36:33.171 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mgr.y.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45654.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45002.log 2024-03-06T07:36:33.171 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.171 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43985.log 2024-03-06T07:36:33.171 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53240.log: /var/log/ceph/ceph-client.admin.45002.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53240.log.gz 2024-03-06T07:36:33.171 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66936.log 2024-03-06T07:36:33.171 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45002.log.gz 2024-03-06T07:36:33.172 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65655.log 2024-03-06T07:36:33.172 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66936.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43985.log.gz 2024-03-06T07:36:33.172 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66936.log.gz 2024-03-06T07:36:33.172 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63916.log 2024-03-06T07:36:33.172 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82478.log 2024-03-06T07:36:33.172 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65655.log.gz 2024-03-06T07:36:33.172 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54698.log 2024-03-06T07:36:33.172 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5%/var/log/ceph/ceph-client.admin.82478.log: -- replaced with /var/log/ceph/ceph-client.admin.63916.log.gz 2024-03-06T07:36:33.172 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53178.log 2024-03-06T07:36:33.173 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82478.log.gz 2024-03-06T07:36:33.173 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60648.log 2024-03-06T07:36:33.173 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54698.log: /var/log/ceph/ceph-client.admin.53178.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54698.log.gz 2024-03-06T07:36:33.173 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81278.log 2024-03-06T07:36:33.173 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53178.log.gz 2024-03-06T07:36:33.173 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87783.log 2024-03-06T07:36:33.173 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81278.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60648.log.gz 2024-03-06T07:36:33.173 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81278.log.gz 2024-03-06T07:36:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73721.log 2024-03-06T07:36:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89475.log 2024-03-06T07:36:33.174 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87783.log.gz 2024-03-06T07:36:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82915.log 2024-03-06T07:36:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89475.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73721.log.gz 2024-03-06T07:36:33.174 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89475.log.gz 2024-03-06T07:36:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77488.log 2024-03-06T07:36:33.174 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87195.log 2024-03-06T07:36:33.174 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82915.log.gz 2024-03-06T07:36:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58019.log 2024-03-06T07:36:33.175 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1%/var/log/ceph/ceph-client.admin.87195.log: -- replaced with /var/log/ceph/ceph-client.admin.77488.log.gz 2024-03-06T07:36:33.175 INFO:teuthology.orchestra.run.smithi027.stderr: 94.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-06T07:36:33.175 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87195.log.gz 2024-03-06T07:36:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66615.log 2024-03-06T07:36:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.58019.log: -- /var/log/ceph/ceph-client.admin.87657.log 2024-03-06T07:36:33.175 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66615.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58019.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89853.log 2024-03-06T07:36:33.175 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.176 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66615.log.gz 2024-03-06T07:36:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49453.log 2024-03-06T07:36:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87657.log: /var/log/ceph/ceph-client.admin.89853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49921.log 2024-03-06T07:36:33.176 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.87657.log.gz 2024-03-06T07:36:33.176 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89853.log.gz 2024-03-06T07:36:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49453.log: gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49453.log.gz 2024-03-06T07:36:33.176 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.89205.log 2024-03-06T07:36:33.176 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73322.log 2024-03-06T07:36:33.176 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49921.log.gz 2024-03-06T07:36:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92725.log 88.5% 2024-03-06T07:36:33.177 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89205.log.gz 2024-03-06T07:36:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47776.log 2024-03-06T07:36:33.177 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73322.log.gz/var/log/ceph/ceph-client.admin.92725.log: 2024-03-06T07:36:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86253.log 2024-03-06T07:36:33.177 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92725.log.gz 2024-03-06T07:36:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90375.log 2024-03-06T07:36:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47776.log: /var/log/ceph/ceph-client.admin.86253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57353.log 2024-03-06T07:36:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90375.log: 87.5% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47776.log.gz 2024-03-06T07:36:33.178 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86253.log.gz 2024-03-06T07:36:33.178 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90375.log.gz 2024-03-06T07:36:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64084.log 2024-03-06T07:36:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63337.log 2024-03-06T07:36:33.178 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7%/var/log/ceph/ceph-client.admin.64084.log: -- replaced with /var/log/ceph/ceph-client.admin.57353.log.gz 2024-03-06T07:36:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91717.log 2024-03-06T07:36:33.178 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64084.log.gz 2024-03-06T07:36:33.178 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88953.log 2024-03-06T07:36:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63337.log: /var/log/ceph/ceph-client.admin.91717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79069.log 2024-03-06T07:36:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88953.log: 88.0% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91717.log.gz 2024-03-06T07:36:33.179 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63337.log.gz 2024-03-06T07:36:33.179 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88953.log.gz 2024-03-06T07:36:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53644.log 2024-03-06T07:36:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57968.log 2024-03-06T07:36:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89247.log 2024-03-06T07:36:33.179 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53644.log: 88.9% 88.1%/var/log/ceph/ceph-client.admin.57968.log: -- replaced with /var/log/ceph/ceph-client.admin.53644.log.gz 2024-03-06T07:36:33.180 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79069.log.gz 2024-03-06T07:36:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48322.log 2024-03-06T07:36:33.180 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57968.log.gz/var/log/ceph/ceph-client.admin.89247.log: 2024-03-06T07:36:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77192.log 2024-03-06T07:36:33.180 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89247.log.gz 2024-03-06T07:36:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90795.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48322.log.gz 2024-03-06T07:36:33.180 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.180 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71336.log 2024-03-06T07:36:33.180 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77192.log.gz/var/log/ceph/ceph-client.admin.90795.log: 2024-03-06T07:36:33.181 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60606.log 2024-03-06T07:36:33.181 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90795.log.gz 2024-03-06T07:36:33.181 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61596.log 2024-03-06T07:36:33.181 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71336.log: gzip -5 89.4% --verbose -- /var/log/ceph/ceph-client.admin.79808.log 2024-03-06T07:36:33.181 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60606.log: -- replaced with /var/log/ceph/ceph-client.admin.71336.log.gz 2024-03-06T07:36:33.181 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1%/var/log/ceph/ceph-client.admin.61596.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.60606.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.53382.log 2024-03-06T07:36:33.181 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.181 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53728.log 2024-03-06T07:36:33.182 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61596.log.gz 2024-03-06T07:36:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53382.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79808.log.gz 2024-03-06T07:36:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74286.log 2024-03-06T07:36:33.182 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53382.log.gz 2024-03-06T07:36:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44438.log 2024-03-06T07:36:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58149.log 2024-03-06T07:36:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74286.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53728.log.gz 2024-03-06T07:36:33.182 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44438.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74286.log.gz 2024-03-06T07:36:33.183 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44438.log.gz 2024-03-06T07:36:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52273.log 2024-03-06T07:36:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47818.log 2024-03-06T07:36:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58149.log: /var/log/ceph/ceph-client.admin.52273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75163.log 2024-03-06T07:36:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47818.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58149.log.gz 2024-03-06T07:36:33.183 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52273.log.gz 2024-03-06T07:36:33.183 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72325.log 2024-03-06T07:36:33.183 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47818.log.gz 2024-03-06T07:36:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88011.log 2024-03-06T07:36:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75163.log: /var/log/ceph/ceph-client.admin.72325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84858.log 2024-03-06T07:36:33.184 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75163.log.gz/var/log/ceph/ceph-client.admin.88011.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72325.log.gz 2024-03-06T07:36:33.184 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88011.log.gz 2024-03-06T07:36:33.184 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80606.log 2024-03-06T07:36:33.184 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56033.log 2024-03-06T07:36:33.184 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84858.log.gz 2024-03-06T07:36:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72367.log 2024-03-06T07:36:33.185 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46332.log 2024-03-06T07:36:33.185 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80606.log.gz 2024-03-06T07:36:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.56033.log: /var/log/ceph/ceph-client.admin.72367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60160.log 2024-03-06T07:36:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46332.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72367.log.gz 2024-03-06T07:36:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58768.log 2024-03-06T07:36:33.185 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46332.log.gz 2024-03-06T07:36:33.185 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60160.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60160.log.gz 2024-03-06T07:36:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56383.log 2024-03-06T07:36:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.58768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65697.log 2024-03-06T07:36:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56383.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56383.log.gz 2024-03-06T07:36:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90147.log 2024-03-06T07:36:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65697.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65697.log.gz 2024-03-06T07:36:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89559.log 2024-03-06T07:36:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90147.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90147.log.gz 2024-03-06T07:36:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50713.log 2024-03-06T07:36:33.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89559.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89559.log.gz 2024-03-06T07:36:33.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57712.log 2024-03-06T07:36:33.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50713.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50713.log.gz 2024-03-06T07:36:33.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82011.log 2024-03-06T07:36:33.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57712.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57712.log.gz 2024-03-06T07:36:33.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61680.log 2024-03-06T07:36:33.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82011.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82011.log.gz 2024-03-06T07:36:33.187 INFO:teuthology.orchestra.run.smithi027.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.56033.log.gz 2024-03-06T07:36:33.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71972.log 2024-03-06T07:36:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52105.log 2024-03-06T07:36:33.188 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61680.log.gz 2024-03-06T07:36:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48771.log 2024-03-06T07:36:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52105.log: 93.3% -- replaced with /var/log/ceph/ceph-client.0.58768.log.gz 2024-03-06T07:36:33.188 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71972.log.gz 2024-03-06T07:36:33.188 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68054.log 2024-03-06T07:36:33.188 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52105.log.gz 2024-03-06T07:36:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60858.log 2024-03-06T07:36:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48771.log: /var/log/ceph/ceph-client.admin.68054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89517.log 2024-03-06T07:36:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60858.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48771.log.gz 2024-03-06T07:36:33.189 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68054.log.gz 2024-03-06T07:36:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip 87.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.60858.log.gz --verbose 2024-03-06T07:36:33.189 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-osd.0.log 2024-03-06T07:36:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63790.log 2024-03-06T07:36:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89517.log: gzip/var/log/ceph/ceph-osd.0.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.86799.log 2024-03-06T07:36:33.189 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89517.log.gz 2024-03-06T07:36:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47025.log 2024-03-06T07:36:33.190 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63790.log.gz 2024-03-06T07:36:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68251.log 2024-03-06T07:36:33.190 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3%/var/log/ceph/ceph-client.0.47025.log: -- replaced with /var/log/ceph/ceph-client.admin.86799.log.gz 2024-03-06T07:36:33.190 INFO:teuthology.orchestra.run.smithi027.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.0.47025.log.gz 2024-03-06T07:36:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67248.log 2024-03-06T07:36:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45996.log 2024-03-06T07:36:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68251.log: /var/log/ceph/ceph-client.admin.67248.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68251.log.gz 2024-03-06T07:36:33.191 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67248.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49285.log 2024-03-06T07:36:33.191 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78326.log 2024-03-06T07:36:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45996.log: /var/log/ceph/ceph-client.admin.49285.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49285.log.gz 2024-03-06T07:36:33.191 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45996.log.gz 2024-03-06T07:36:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80522.log 2024-03-06T07:36:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65076.log 2024-03-06T07:36:33.191 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78326.log.gz 2024-03-06T07:36:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59185.log 2024-03-06T07:36:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65076.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80522.log.gz 2024-03-06T07:36:33.192 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65076.log.gz 2024-03-06T07:36:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87321.log 2024-03-06T07:36:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51840.log 2024-03-06T07:36:33.192 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59185.log.gz 2024-03-06T07:36:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70204.log 2024-03-06T07:36:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51840.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87321.log.gz 2024-03-06T07:36:33.192 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51840.log.gz 2024-03-06T07:36:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85899.log 2024-03-06T07:36:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61722.log 2024-03-06T07:36:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70204.log: /var/log/ceph/ceph-client.admin.85899.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70204.log.gz 2024-03-06T07:36:33.193 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85899.log.gz 2024-03-06T07:36:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81716.log 2024-03-06T07:36:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67543.log 2024-03-06T07:36:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81716.log: /var/log/ceph/ceph-client.admin.61722.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61722.log.gz 2024-03-06T07:36:33.193 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81716.log.gz 2024-03-06T07:36:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69200.log 2024-03-06T07:36:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47860.log 2024-03-06T07:36:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67543.log: /var/log/ceph/ceph-client.admin.69200.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67543.log.gz 2024-03-06T07:36:33.194 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69200.log.gz 2024-03-06T07:36:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70729.log 2024-03-06T07:36:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78061.log 2024-03-06T07:36:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70729.log: /var/log/ceph/ceph-client.admin.47860.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47860.log.gz 2024-03-06T07:36:33.194 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70729.log.gz 2024-03-06T07:36:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50293.log 2024-03-06T07:36:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.78061.log: --verbose -- /var/log/ceph/ceph-client.admin.74998.log 2024-03-06T07:36:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50293.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78061.log.gz 2024-03-06T07:36:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88053.log 2024-03-06T07:36:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74998.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50293.log.gz 2024-03-06T07:36:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80228.log 2024-03-06T07:36:33.195 INFO:teuthology.orchestra.run.smithi027.stderr: 92.5%/var/log/ceph/ceph-client.admin.88053.log: -- replaced with /var/log/ceph/ceph-client.admin.74998.log.gz 2024-03-06T07:36:33.195 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44155.log 2024-03-06T07:36:33.196 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88053.log.gz/var/log/ceph/ceph-client.admin.80228.log: 2024-03-06T07:36:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80228.log.gz 2024-03-06T07:36:33.196 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.76838.log 2024-03-06T07:36:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44526.log 2024-03-06T07:36:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44155.log: /var/log/ceph/ceph-client.admin.76838.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44155.log.gz 2024-03-06T07:36:33.196 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76838.log.gz 2024-03-06T07:36:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70981.log 2024-03-06T07:36:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68423.log 2024-03-06T07:36:33.196 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44526.log.gz 2024-03-06T07:36:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64906.log 2024-03-06T07:36:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68423.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70981.log.gz 2024-03-06T07:36:33.197 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68423.log.gz 2024-03-06T07:36:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92599.log 2024-03-06T07:36:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64906.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64906.log.gz 2024-03-06T07:36:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48112.log 2024-03-06T07:36:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57924.log 2024-03-06T07:36:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92599.log: /var/log/ceph/ceph-client.admin.48112.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92599.log.gz 2024-03-06T07:36:33.198 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48112.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64126.log 2024-03-06T07:36:33.198 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82309.log 2024-03-06T07:36:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57924.log: /var/log/ceph/ceph-client.admin.64126.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57924.log.gz 2024-03-06T07:36:33.198 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64126.log.gz 2024-03-06T07:36:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81885.log 2024-03-06T07:36:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45614.log 2024-03-06T07:36:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82309.log: /var/log/ceph/ceph-client.admin.81885.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82309.log.gz 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90627.log 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81885.log.gz 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69130.log 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90627.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45614.log.gz 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90627.log.gz 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82873.log 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75376.log 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69130.log: /var/log/ceph/ceph-client.admin.82873.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69130.log.gz 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57796.log 2024-03-06T07:36:33.199 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82873.log.gz 2024-03-06T07:36:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54614.log 2024-03-06T07:36:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75376.log: /var/log/ceph/ceph-client.admin.57796.log: 88.2% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75376.log.gz 2024-03-06T07:36:33.200 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57796.log.gz 2024-03-06T07:36:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:36:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54614.log: /var/log/ceph/ceph-client.admin.73805.log 2024-03-06T07:36:33.200 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54614.log.gz 2024-03-06T07:36:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88347.log 2024-03-06T07:36:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73805.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73805.log.gz 2024-03-06T07:36:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90963.log 2024-03-06T07:36:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88347.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88347.log.gz 2024-03-06T07:36:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37948.log 2024-03-06T07:36:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90963.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90963.log.gz 2024-03-06T07:36:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91005.log 2024-03-06T07:36:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37948.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37948.log.gz 2024-03-06T07:36:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78112.log 2024-03-06T07:36:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91005.log: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:36:33.201 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91005.log.gz 2024-03-06T07:36:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86883.log 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85737.log 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78112.log: /var/log/ceph/ceph-client.admin.86883.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78112.log.gz 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76710.log 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86883.log.gz 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81530.log 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76710.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85737.log.gz 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76710.log.gz 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71378.log 2024-03-06T07:36:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81530.log: /var/log/ceph/ceph-osd.2.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81530.log.gz 2024-03-06T07:36:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55770.log 2024-03-06T07:36:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71378.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71378.log.gz 2024-03-06T07:36:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90543.log 2024-03-06T07:36:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55770.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55770.log.gz 2024-03-06T07:36:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80999.log 2024-03-06T07:36:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90543.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90543.log.gz 2024-03-06T07:36:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53043.log 2024-03-06T07:36:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80999.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80999.log.gz 2024-03-06T07:36:33.204 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48676.log 2024-03-06T07:36:33.204 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53043.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53043.log.gz 2024-03-06T07:36:33.204 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48997.log 2024-03-06T07:36:33.204 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48676.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48676.log.gz 2024-03-06T07:36:33.204 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58191.log 2024-03-06T07:36:33.205 INFO:teuthology.orchestra.run.smithi069.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-06T07:36:33.225 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48997.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48997.log.gz 2024-03-06T07:36:33.225 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89643.log 2024-03-06T07:36:33.225 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58191.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.58191.log.gz 2024-03-06T07:36:33.225 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-06T07:36:33.225 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89643.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89643.log.gz 2024-03-06T07:36:33.225 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79240.log 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51067.log 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79240.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79240.log.gz 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62093.log 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51067.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51067.log.gz 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65511.log 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66810.log 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65511.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62093.log.gz 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65511.log.gz 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48280.log 2024-03-06T07:36:33.226 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66810.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66810.log.gz 2024-03-06T07:36:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66531.log 2024-03-06T07:36:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48280.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48280.log.gz 2024-03-06T07:36:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88221.log 2024-03-06T07:36:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66531.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66531.log.gz 2024-03-06T07:36:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67800.log 2024-03-06T07:36:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88221.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.88221.log.gz 2024-03-06T07:36:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49165.log 2024-03-06T07:36:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76964.log 2024-03-06T07:36:33.227 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49165.log: 88.9% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67800.log.gz 2024-03-06T07:36:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61045.log 2024-03-06T07:36:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76964.log: -- replaced with /var/log/ceph/ceph-client.admin.49165.log.gz 2024-03-06T07:36:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48592.log 2024-03-06T07:36:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61045.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61045.log.gz 2024-03-06T07:36:33.228 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76964.log.gz 2024-03-06T07:36:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59932.log 2024-03-06T07:36:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82436.log 2024-03-06T07:36:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.59932.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48592.log.gz 2024-03-06T07:36:33.228 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63133.log 2024-03-06T07:36:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73537.log 2024-03-06T07:36:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63133.log: 89.0% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82436.log.gz 2024-03-06T07:36:33.229 INFO:teuthology.orchestra.run.smithi027.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-06T07:36:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48634.log 2024-03-06T07:36:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73537.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73537.log.gz 2024-03-06T07:36:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87927.log 2024-03-06T07:36:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48634.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48634.log.gz 2024-03-06T07:36:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51975.log 2024-03-06T07:36:33.229 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87927.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87927.log.gz 2024-03-06T07:36:33.230 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59143.log 2024-03-06T07:36:33.230 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51975.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51975.log.gz 2024-03-06T07:36:33.230 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48364.log 2024-03-06T07:36:33.230 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59143.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59143.log.gz 2024-03-06T07:36:33.230 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63133.log.gz 2024-03-06T07:36:33.230 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77973.log 2024-03-06T07:36:33.230 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48364.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48364.log.gz 2024-03-06T07:36:33.230 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66573.log 2024-03-06T07:36:33.230 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77973.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77973.log.gz 2024-03-06T07:36:33.231 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87111.log 2024-03-06T07:36:33.231 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66573.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66573.log.gz 2024-03-06T07:36:33.231 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82053.log 2024-03-06T07:36:33.231 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87111.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87111.log.gz 2024-03-06T07:36:33.231 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90921.log 2024-03-06T07:36:33.231 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82053.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82053.log.gz 2024-03-06T07:36:33.231 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46290.log 2024-03-06T07:36:33.231 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90921.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90921.log.gz 2024-03-06T07:36:33.244 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76171.log 2024-03-06T07:36:33.244 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46290.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46290.log.gz 2024-03-06T07:36:33.244 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52063.log 2024-03-06T07:36:33.244 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76171.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76171.log.gz 2024-03-06T07:36:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58275.log 2024-03-06T07:36:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52063.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52063.log.gz 2024-03-06T07:36:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70855.log 2024-03-06T07:36:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58275.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58275.log.gz 2024-03-06T07:36:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86841.log 2024-03-06T07:36:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70855.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70855.log.gz 2024-03-06T07:36:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51374.log 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86841.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86841.log.gz 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56620.log 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51374.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51374.log.gz 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57227.log 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56620.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56620.log.gz 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49369.log 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57227.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57227.log.gz 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71125.log 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49369.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49369.log.gz 2024-03-06T07:36:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87699.log 2024-03-06T07:36:33.247 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71125.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71125.log.gz 2024-03-06T07:36:33.247 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78368.log 2024-03-06T07:36:33.247 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87699.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87699.log.gz 2024-03-06T07:36:33.247 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57754.log 2024-03-06T07:36:33.247 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78368.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78368.log.gz 2024-03-06T07:36:33.247 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53923.log 2024-03-06T07:36:33.247 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57754.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57754.log.gz 2024-03-06T07:36:33.252 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53116.log 2024-03-06T07:36:33.252 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53923.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53923.log.gz 2024-03-06T07:36:33.252 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66489.log 2024-03-06T07:36:33.252 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53116.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53116.log.gz 2024-03-06T07:36:33.252 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83227.log 2024-03-06T07:36:33.253 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66489.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66489.log.gz 2024-03-06T07:36:33.253 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76214.log 2024-03-06T07:36:33.253 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83227.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.83227.log.gz 2024-03-06T07:36:33.253 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49779.log 2024-03-06T07:36:33.253 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76214.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.76214.log.gz 2024-03-06T07:36:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77362.log 2024-03-06T07:36:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49779.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49779.log.gz 2024-03-06T07:36:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51542.log 2024-03-06T07:36:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77362.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77362.log.gz 2024-03-06T07:36:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83017.log 2024-03-06T07:36:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51542.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51542.log.gz 2024-03-06T07:36:33.258 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50755.log 2024-03-06T07:36:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83017.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83017.log.gz 2024-03-06T07:36:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84590.log 2024-03-06T07:36:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50755.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50755.log.gz 2024-03-06T07:36:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69259.log 2024-03-06T07:36:33.262 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84590.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84590.log.gz 2024-03-06T07:36:33.262 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60328.log 2024-03-06T07:36:33.262 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69259.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69259.log.gz 2024-03-06T07:36:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77871.log 2024-03-06T07:36:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60328.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60328.log.gz 2024-03-06T07:36:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55038.log 2024-03-06T07:36:33.265 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-06T07:36:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77871.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77871.log.gz 2024-03-06T07:36:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56299.log 2024-03-06T07:36:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55038.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55038.log.gz 2024-03-06T07:36:33.267 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65847.log 2024-03-06T07:36:33.268 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56299.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56299.log.gz 2024-03-06T07:36:33.268 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72016.log 2024-03-06T07:36:33.270 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65847.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65847.log.gz 2024-03-06T07:36:33.270 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64397.log 2024-03-06T07:36:33.270 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72016.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72016.log.gz 2024-03-06T07:36:33.272 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82566.log 2024-03-06T07:36:33.272 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64397.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64397.log.gz 2024-03-06T07:36:33.272 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79441.log 2024-03-06T07:36:33.273 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82566.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82566.log.gz 2024-03-06T07:36:33.274 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88263.log 2024-03-06T07:36:33.275 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79441.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79441.log.gz 2024-03-06T07:36:33.275 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70492.log 2024-03-06T07:36:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88263.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88263.log.gz 2024-03-06T07:36:33.276 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82393.log 2024-03-06T07:36:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70492.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70492.log.gz 2024-03-06T07:36:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48813.log 2024-03-06T07:36:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82393.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82393.log.gz 2024-03-06T07:36:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76047.log 2024-03-06T07:36:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48813.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48813.log.gz 2024-03-06T07:36:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49863.log 2024-03-06T07:36:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76047.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76047.log.gz 2024-03-06T07:36:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65343.log 2024-03-06T07:36:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49863.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49863.log.gz 2024-03-06T07:36:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60286.log 2024-03-06T07:36:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65343.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65343.log.gz 2024-03-06T07:36:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49243.log 2024-03-06T07:36:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60286.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60286.log.gz 2024-03-06T07:36:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86067.log 2024-03-06T07:36:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49243.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49243.log.gz 2024-03-06T07:36:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79379.log 2024-03-06T07:36:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86067.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86067.log.gz 2024-03-06T07:36:33.289 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80354.log 2024-03-06T07:36:33.290 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79379.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79379.log.gz 2024-03-06T07:36:33.290 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48238.log 2024-03-06T07:36:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80354.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80354.log.gz 2024-03-06T07:36:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65553.log 2024-03-06T07:36:33.293 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48238.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48238.log.gz 2024-03-06T07:36:33.293 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66768.log 2024-03-06T07:36:33.294 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65553.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65553.log.gz 2024-03-06T07:36:33.294 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55304.log 2024-03-06T07:36:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66768.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66768.log.gz 2024-03-06T07:36:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87153.log 2024-03-06T07:36:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55304.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55304.log.gz 2024-03-06T07:36:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86505.log 2024-03-06T07:36:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87153.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87153.log.gz 2024-03-06T07:36:33.299 INFO:teuthology.orchestra.run.smithi027.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.59932.log.gz 2024-03-06T07:36:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58359.log 2024-03-06T07:36:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86505.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86505.log.gz 2024-03-06T07:36:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85427.log 2024-03-06T07:36:33.301 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58359.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58359.log.gz 2024-03-06T07:36:33.302 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45334.log 2024-03-06T07:36:33.302 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85427.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85427.log.gz 2024-03-06T07:36:33.303 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84632.log 2024-03-06T07:36:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45334.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45334.log.gz 2024-03-06T07:36:33.304 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92683.log 2024-03-06T07:36:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84632.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.84632.log.gz 2024-03-06T07:36:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67417.log 2024-03-06T07:36:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92683.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92683.log.gz 2024-03-06T07:36:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44986.log 2024-03-06T07:36:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67417.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67417.log.gz 2024-03-06T07:36:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51332.log 2024-03-06T07:36:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44986.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44986.log.gz 2024-03-06T07:36:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65213.log 2024-03-06T07:36:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51332.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51332.log.gz 2024-03-06T07:36:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78494.log 2024-03-06T07:36:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.65213.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65213.log.gz 2024-03-06T07:36:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71546.log 2024-03-06T07:36:33.314 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78494.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78494.log.gz 2024-03-06T07:36:33.314 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79317.log 2024-03-06T07:36:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71546.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71546.log.gz 2024-03-06T07:36:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53523.log 2024-03-06T07:36:33.317 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79317.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79317.log.gz 2024-03-06T07:36:33.317 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54133.log 2024-03-06T07:36:33.317 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53523.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53523.log.gz 2024-03-06T07:36:33.319 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-06T07:36:33.319 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54133.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54133.log.gz 2024-03-06T07:36:33.319 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72493.log 2024-03-06T07:36:33.332 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62051.log 2024-03-06T07:36:33.333 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72493.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72493.log.gz 2024-03-06T07:36:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62261.log 2024-03-06T07:36:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62051.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62051.log.gz 2024-03-06T07:36:33.353 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71210.log 2024-03-06T07:36:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62261.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62261.log.gz 2024-03-06T07:36:33.362 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49081.log 2024-03-06T07:36:33.363 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71210.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71210.log.gz 2024-03-06T07:36:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84123.log 2024-03-06T07:36:33.378 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49081.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49081.log.gz 2024-03-06T07:36:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87573.log 2024-03-06T07:36:33.387 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84123.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84123.log.gz 2024-03-06T07:36:33.402 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50629.log 2024-03-06T07:36:33.402 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87573.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.87573.log.gz 2024-03-06T07:36:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70813.log 2024-03-06T07:36:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50629.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50629.log.gz 2024-03-06T07:36:33.428 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89937.log 2024-03-06T07:36:33.429 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70813.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70813.log.gz 2024-03-06T07:36:33.443 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86673.log 2024-03-06T07:36:33.443 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.89937.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89937.log.gz 2024-03-06T07:36:33.454 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59369.log 2024-03-06T07:36:33.455 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86673.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86673.log.gz 2024-03-06T07:36:33.469 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53686.log 2024-03-06T07:36:33.470 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59369.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59369.log.gz 2024-03-06T07:36:33.483 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78536.log 2024-03-06T07:36:33.484 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53686.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53686.log.gz 2024-03-06T07:36:33.501 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73621.log 2024-03-06T07:36:33.502 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78536.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.78536.log.gz 2024-03-06T07:36:33.512 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72241.log 2024-03-06T07:36:33.512 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73621.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73621.log.gz 2024-03-06T07:36:33.520 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56341.log 2024-03-06T07:36:33.523 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72241.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72241.log.gz 2024-03-06T07:36:33.529 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84978.log 2024-03-06T07:36:33.539 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88659.log 2024-03-06T07:36:33.542 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84978.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56341.log.gz 2024-03-06T07:36:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60564.log 2024-03-06T07:36:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.88659.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84978.log.gz 2024-03-06T07:36:33.560 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90021.log 2024-03-06T07:36:33.560 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88659.log.gz 2024-03-06T07:36:33.561 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60564.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60564.log.gz 2024-03-06T07:36:33.569 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53481.log 2024-03-06T07:36:33.578 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48406.log 2024-03-06T07:36:33.582 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53481.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.90021.log.gz 2024-03-06T07:36:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71504.log 2024-03-06T07:36:33.591 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48406.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53481.log.gz 2024-03-06T07:36:33.596 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91760.log 2024-03-06T07:36:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71504.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48406.log.gz 2024-03-06T07:36:33.609 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60244.log 2024-03-06T07:36:33.609 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71504.log.gz 2024-03-06T07:36:33.610 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91760.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91760.log.gz 2024-03-06T07:36:33.618 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71846.log 2024-03-06T07:36:33.627 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75235.log 2024-03-06T07:36:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71846.log: 87.8% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60244.log.gz 2024-03-06T07:36:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69589.log 2024-03-06T07:36:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75235.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75235.log.gz 2024-03-06T07:36:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68592.log 2024-03-06T07:36:33.639 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71846.log.gz 2024-03-06T07:36:33.640 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69589.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.69589.log.gz 2024-03-06T07:36:33.648 INFO:teuthology.orchestra.run.smithi069.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-06T07:36:33.653 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68592.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.68592.log.gz 2024-03-06T07:36:33.693 INFO:teuthology.orchestra.run.smithi027.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.0.52808.log.gz 2024-03-06T07:36:33.808 INFO:teuthology.orchestra.run.smithi069.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-06T07:36:33.856 INFO:teuthology.orchestra.run.smithi027.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-06T07:36:33.879 INFO:teuthology.orchestra.run.smithi069.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-06T07:36:33.970 INFO:teuthology.orchestra.run.smithi069.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-06T07:36:33.972 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-06T07:36:33.972 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m1.271s 2024-03-06T07:36:33.972 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m5.407s 2024-03-06T07:36:33.973 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m0.229s 2024-03-06T07:36:34.322 INFO:teuthology.orchestra.run.smithi027.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-06T07:36:34.510 INFO:teuthology.orchestra.run.smithi027.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-06T07:36:34.559 INFO:teuthology.orchestra.run.smithi027.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-06T07:36:34.639 INFO:teuthology.orchestra.run.smithi027.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-06T07:36:35.221 INFO:teuthology.orchestra.run.smithi027.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.0.68919.log.gz 2024-03-06T07:36:42.619 INFO:teuthology.orchestra.run.smithi027.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-06T07:36:42.621 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-03-06T07:36:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m9.944s 2024-03-06T07:36:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m20.098s 2024-03-06T07:36:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m1.125s 2024-03-06T07:36:42.621 INFO:tasks.ceph:Archiving logs... 2024-03-06T07:36:42.622 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/remote/smithi027/log 2024-03-06T07:36:42.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:36:45.750 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/remote/smithi069/log 2024-03-06T07:36:45.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:36:46.466 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-06T07:36:46.475 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-06T07:36:46.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:36:46.478 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:36:46.561 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-06T07:36:46.569 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-06T07:36:46.569 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:36:46.572 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:36:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:36:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:36:46.592 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:36:46.592 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-03-06T07:36:46.592 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +1842us[+1921us] +/- 85ms 2024-03-06T07:36:46.592 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -1072us[ -993us] +/- 56ms 2024-03-06T07:36:46.592 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 38 -31us[ -31us] +/- 92ms 2024-03-06T07:36:46.592 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:36:46.593 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:36:46.593 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-03-06T07:36:46.594 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +1912us[+1912us] +/- 85ms 2024-03-06T07:36:46.594 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -1015us[-1001us] +/- 56ms 2024-03-06T07:36:46.594 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +37us[ +51us] +/- 92ms 2024-03-06T07:36:46.594 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:36:46.594 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-06T07:36:46.603 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-06T07:36:46.603 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-06T07:36:46.612 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:36:46.740 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:36:46.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:36:46.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:36:46.850 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:36:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:36:46.882 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-06T07:36:46.891 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-06T07:36:46.899 INFO:teuthology.task.internal:Duration was 1277.734835 seconds 2024-03-06T07:36:46.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-06T07:36:46.909 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-06T07:36:46.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:36:46.911 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:36:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:36:46.968 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:36:47.373 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-06T07:36:47.373 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-03-06T07:36:47.373 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:36:47.393 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-03-06T07:36:47.394 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:36:47.435 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-06T07:36:47.436 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:36:47.438 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:36:47.505 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-06T07:36:47.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:36:47.511 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:36:47.655 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-06T07:36:47.665 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-06T07:36:47.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:36:47.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:36:47.723 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-06T07:36:47.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:36:47.739 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:36:47.766 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-03-06T07:36:47.788 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-03-06T07:36:47.804 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:36:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:36:47.840 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:36:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:36:47.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-06T07:36:47.868 INFO:teuthology.task.internal:Transferring archived files... 2024-03-06T07:36:47.869 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/remote/smithi027 2024-03-06T07:36:47.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:36:47.923 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058/remote/smithi069 2024-03-06T07:36:47.923 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:36:47.958 INFO:teuthology.task.internal:Removing archive directory... 2024-03-06T07:36:47.958 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:36:47.960 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:36:48.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-06T07:36:48.019 INFO:teuthology.task.internal:Not uploading archives. 2024-03-06T07:36:48.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-06T07:36:48.028 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-06T07:36:48.028 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:36:48.030 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:36:48.043 INFO:teuthology.orchestra.run.smithi027.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:36 /home/ubuntu/cephtest 2024-03-06T07:36:48.063 INFO:teuthology.orchestra.run.smithi069.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:36 /home/ubuntu/cephtest 2024-03-06T07:36:48.064 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-06T07:36:48.074 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-06T07:36:48.094 INFO:teuthology.nuke:Checking targets against current locks 2024-03-06T07:36:48.120 DEBUG:teuthology.nuke:shortname: smithi027 2024-03-06T07:36:48.120 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:36:48.144 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:39.801121', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJ3vhpohM9xB5ux8f7iuiyN+Bf6kEPwaNo6LDrS7aDwVfwCdJ+zvK44YvoLEqEN6iG62xG3g1AQhcguxwafKWc='} 2024-03-06T07:36:48.147 DEBUG:teuthology.nuke:shortname: smithi069 2024-03-06T07:36:48.147 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:36:48.171 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583058', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:39.802255', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVpYiu/YoCYgaV/eQmFohRMqb3IJ5fc5iJmrQsU0co4UgQoF2P0s0/28GIdJ3ak/3iXpY9jZANDyCOyCMhbsro='} 2024-03-06T07:36:48.203 INFO:teuthology.orchestra.console.smithi027:Power off 2024-03-06T07:36:48.203 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:36:48.225 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-03-06T07:36:48.226 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:36:48.240 INFO:teuthology.orchestra.console.smithi069:Power off 2024-03-06T07:36:48.240 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:36:48.265 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-03-06T07:36:48.265 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:36:48.275 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-03-06T07:36:48.287 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-06T07:36:52.276 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:36:52.288 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:36:52.386 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-03-06T07:36:52.397 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-06T07:36:56.388 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:36:56.398 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:36:56.500 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-03-06T07:36:56.508 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-06T07:37:00.501 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:37:00.509 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:37:00.611 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-03-06T07:37:00.618 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-03-06T07:37:00.713 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-03-06T07:37:00.719 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-03-06T07:37:00.823 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} duration: 1277.7348353862762 failure_reason: 'Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=87083e1b34594069b6395246351d2017 status: fail success: false 2024-03-06T07:37:00.824 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:37:00.917 INFO:teuthology.run:FAIL